2022-09-12T18:35:13.195 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-12T18:35:13.197 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2022-09-11_20:31:56-rgw-main-distro-default-smithi/7026931 --name yuriw-2022-09-11_20:31:56-rgw-main-distro-default-smithi --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi} --verbose --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2022-09-11_20:31:56-rgw-main-distro-default-smithi/7026931/orig.config.yaml 2022-09-12T18:35:13.245 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-09-12T18:35:13.320 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-09-11_20:31:56-rgw-main-distro-default-smithi/7026931 branch: main description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi} email: ceph-qa@ceph.io first_in_suite: false job_id: '7026931' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2022-09-11_20:31:56-rgw-main-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: main ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 10f7d25d6a0f340676cd251eab5a787ba01ace95 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 10f7d25d6a0f340676cd251eab5a787ba01ace95 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 rgw-multisite-tests: args: - tests.py - tests_ps.py selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: main sha1: 10f7d25d6a0f340676cd251eab5a787ba01ace95 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 10f7d25d6a0f340676cd251eab5a787ba01ace95 sleep_before_teardown: 0 suite: rgw suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_10f7d25d6a0f340676cd251eab5a787ba01ace95/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 10f7d25d6a0f340676cd251eab5a787ba01ace95 targets: smithi035.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDhDcIgHo8yPwbEK2iHZHrqUzB4APgwzUj7VVWbHPtCAGblmY3il1ZTa06qXVADIwzGqmBDnClFbNh9Xl9ezouikXdGUn6gH+058Uc5DmuDveEWjx9ZG8ypl7K1NI6mA2ZF4ZhavIk1Mece12gvaQfSuc3aaBvpGU5M5Vzoe24AXLIZ9eOtsHpWz7KJ3tCLONBRtUUzCF4sj2xG1cclnzJEmD4DjOXa6CL0plEsq6miQhtw+c9AsYS/ZIZwGHIVOFuLdIASDixTFS74uuJaVt6hzwpRbjdRvfhRB4EeiiOFF9HRleD6wEIBxyzbNjqBeKeP6AYr74v+0E9K68iOMp54q2xxkmWFnhvsuVYTyYmU+A1VqJMRLuNfJBYyB0C2MxvcuXzFweggp7mbybUuhOKpgNAzBliyM7GY/VDZheHJInzFHLRTVY9XsJklqPugCYycox6C/YAlh0Q6pwQ0Sr2WaEMz6PAyG9t+eK+pZAaTXwux1fGSsfpah46Vmy9ngOM= smithi138.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpUYYo//MHC5osesSZpvrMfoP5AJD18lKMj30OB4QvSUyQC7Nwq1VJzA3Q71fceJFqHa2/LJAxGfhT297bQk8IsAh6ewsfysQhYwYOC4DhWDuuP3m6O0v15ME6VtShlwvk9sowvjf1f/DcyDWr2jVtN91ZeHVCjSyFMU5Qvr7BnKk/rDp/AnJwxKrKHE5OhJLicQ/OWLs6Bc4lreV5M0zIxxJMalQYeL5VnLzDaPYQ6iuDbp/AYbhIPcjrQfp/QFcreVGJxsW3mRwbOpo9/2YYTOUisbXmboaxY+FU4M81ehTFntB9X5PxcMrOe3PX6MGzn9XSiRB/sTkH63R6TY5MRC/qd6J2XymC9/llkLMhNiIoKMPwm6zAOEUfDQc8U2gDDjPcqeIXMAfswyMo4M6iAyM/Wglt5+CE3olTSfUfdRiTaBIYH5NJPsQLYSdwYlUwBVLJau4+gdzdIfJwP3n5Oihatqx2aOOmiy8FIMHnsrekeQcZXS+/vxqNnFcc2OE= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b58f2c18636eb10faa77ed3614abd00cb85dfc2c timestamp: 2022-09-11_20:31:56 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.14413 2022-09-12T18:35:13.320 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_10f7d25d6a0f340676cd251eab5a787ba01ace95/qa; will attempt to use it 2022-09-12T18:35:13.321 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_10f7d25d6a0f340676cd251eab5a787ba01ace95/qa/tasks 2022-09-12T18:35:13.322 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-09-12T18:35:13.324 INFO:teuthology.task.internal:Checking packages... 2022-09-12T18:35:13.362 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '10f7d25d6a0f340676cd251eab5a787ba01ace95' 2022-09-12T18:35:13.363 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-09-12T18:35:13.364 INFO:teuthology.packaging:ref: None 2022-09-12T18:35:13.364 INFO:teuthology.packaging:tag: None 2022-09-12T18:35:13.365 INFO:teuthology.packaging:branch: main 2022-09-12T18:35:13.365 INFO:teuthology.packaging:sha1: 10f7d25d6a0f340676cd251eab5a787ba01ace95 2022-09-12T18:35:13.366 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=main 2022-09-12T18:35:13.629 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-14885.g6ef176b6 2022-09-12T18:35:13.630 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-09-12T18:35:13.641 INFO:teuthology.task.internal:no buildpackages task found 2022-09-12T18:35:13.641 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-09-12T18:35:13.649 INFO:teuthology.task.internal:Saving configuration 2022-09-12T18:35:13.672 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-09-12T18:35:13.680 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-09-12T18:35:13.714 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi035.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-09-11_20:31:56-rgw-main-distro-default-smithi/7026931', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-09-12 18:30:28.502861', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFgxovX6W1Yhs4o9ookuxz38zHKaH0UxKEpOXJcnX01GWpe4TwiNbGBMxD5covqmhm5zyZ6JKVo5oS44CFzQQPo='} 2022-09-12T18:35:13.744 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi138.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-09-11_20:31:56-rgw-main-distro-default-smithi/7026931', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-09-12 18:30:28.501284', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMQqsC39OVtcQog4ZqWLLSH7lxswRdlY3x9iCRrPd3oyitBnAFZhoveKqZyFBE3kM6d7OPqdwQADLmcGoSIMHSU='} 2022-09-12T18:35:13.744 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-09-12T18:35:13.755 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2022-09-12T18:35:13.756 INFO:teuthology.task.internal:roles: ubuntu@smithi138.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2022-09-12T18:35:13.756 INFO:teuthology.run_tasks:Running task console_log... 2022-09-12T18:35:13.901 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdba33bb2f0>, signals=[15]) 2022-09-12T18:35:13.902 INFO:teuthology.run_tasks:Running task internal.connect... 2022-09-12T18:35:13.909 INFO:teuthology.task.internal:Opening connections... 2022-09-12T18:35:13.909 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi035.front.sepia.ceph.com 2022-09-12T18:35:13.912 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi035.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-12T18:35:14.002 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi138.front.sepia.ceph.com 2022-09-12T18:35:14.003 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-12T18:35:14.125 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-09-12T18:35:14.132 DEBUG:teuthology.orchestra.run.smithi035:> uname -m 2022-09-12T18:35:14.153 INFO:teuthology.orchestra.run.smithi035.stdout:x86_64 2022-09-12T18:35:14.154 DEBUG:teuthology.orchestra.run.smithi035:> cat /etc/os-release 2022-09-12T18:35:14.213 INFO:teuthology.orchestra.run.smithi035.stdout:NAME="CentOS Stream" 2022-09-12T18:35:14.214 INFO:teuthology.orchestra.run.smithi035.stdout:VERSION="8" 2022-09-12T18:35:14.214 INFO:teuthology.orchestra.run.smithi035.stdout:ID="centos" 2022-09-12T18:35:14.214 INFO:teuthology.orchestra.run.smithi035.stdout:ID_LIKE="rhel fedora" 2022-09-12T18:35:14.214 INFO:teuthology.orchestra.run.smithi035.stdout:VERSION_ID="8" 2022-09-12T18:35:14.215 INFO:teuthology.orchestra.run.smithi035.stdout:PLATFORM_ID="platform:el8" 2022-09-12T18:35:14.215 INFO:teuthology.orchestra.run.smithi035.stdout:PRETTY_NAME="CentOS Stream 8" 2022-09-12T18:35:14.215 INFO:teuthology.orchestra.run.smithi035.stdout:ANSI_COLOR="0;31" 2022-09-12T18:35:14.215 INFO:teuthology.orchestra.run.smithi035.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-09-12T18:35:14.216 INFO:teuthology.orchestra.run.smithi035.stdout:HOME_URL="https://centos.org/" 2022-09-12T18:35:14.216 INFO:teuthology.orchestra.run.smithi035.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-12T18:35:14.216 INFO:teuthology.orchestra.run.smithi035.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-12T18:35:14.216 INFO:teuthology.orchestra.run.smithi035.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-09-12T18:35:14.217 INFO:teuthology.lock.ops:Updating smithi035.front.sepia.ceph.com on lock server 2022-09-12T18:35:14.239 DEBUG:teuthology.orchestra.run.smithi138:> uname -m 2022-09-12T18:35:14.258 INFO:teuthology.orchestra.run.smithi138.stdout:x86_64 2022-09-12T18:35:14.259 DEBUG:teuthology.orchestra.run.smithi138:> cat /etc/os-release 2022-09-12T18:35:14.331 INFO:teuthology.orchestra.run.smithi138.stdout:NAME="CentOS Stream" 2022-09-12T18:35:14.331 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION="8" 2022-09-12T18:35:14.331 INFO:teuthology.orchestra.run.smithi138.stdout:ID="centos" 2022-09-12T18:35:14.332 INFO:teuthology.orchestra.run.smithi138.stdout:ID_LIKE="rhel fedora" 2022-09-12T18:35:14.332 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION_ID="8" 2022-09-12T18:35:14.332 INFO:teuthology.orchestra.run.smithi138.stdout:PLATFORM_ID="platform:el8" 2022-09-12T18:35:14.332 INFO:teuthology.orchestra.run.smithi138.stdout:PRETTY_NAME="CentOS Stream 8" 2022-09-12T18:35:14.333 INFO:teuthology.orchestra.run.smithi138.stdout:ANSI_COLOR="0;31" 2022-09-12T18:35:14.333 INFO:teuthology.orchestra.run.smithi138.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-09-12T18:35:14.333 INFO:teuthology.orchestra.run.smithi138.stdout:HOME_URL="https://centos.org/" 2022-09-12T18:35:14.334 INFO:teuthology.orchestra.run.smithi138.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-12T18:35:14.334 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-12T18:35:14.334 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-09-12T18:35:14.334 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server 2022-09-12T18:35:14.365 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-09-12T18:35:14.384 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-09-12T18:35:14.401 INFO:teuthology.task.internal:Checking for old test directory... 2022-09-12T18:35:14.402 DEBUG:teuthology.orchestra.run.smithi035:> test '!' -e /home/ubuntu/cephtest 2022-09-12T18:35:14.411 DEBUG:teuthology.orchestra.run.smithi138:> test '!' -e /home/ubuntu/cephtest 2022-09-12T18:35:14.430 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-09-12T18:35:14.437 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-09-12T18:35:14.438 DEBUG:teuthology.orchestra.run.smithi035:> test -z $(ls -A /var/lib/ceph) 2022-09-12T18:35:14.471 DEBUG:teuthology.orchestra.run.smithi138:> test -z $(ls -A /var/lib/ceph) 2022-09-12T18:35:14.507 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-09-12T18:35:14.570 INFO:teuthology.run_tasks:Running task kernel... 2022-09-12T18:35:14.597 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-09-12T18:35:14.598 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2022-09-12T18:35:14.599 DEBUG:teuthology.orchestra.run.smithi035:> test -f /run/.containerenv -o -f /.dockerenv 2022-09-12T18:35:14.601 DEBUG:teuthology.orchestra.run.smithi138:> test -f /run/.containerenv -o -f /.dockerenv 2022-09-12T18:35:14.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-12T18:35:14.619 DEBUG:teuthology.orchestra.run.smithi138:> uname -r 2022-09-12T18:35:14.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-12T18:35:14.623 DEBUG:teuthology.orchestra.run.smithi035:> uname -r 2022-09-12T18:35:14.677 INFO:teuthology.orchestra.run.smithi138.stdout:4.18.0-408.el8.x86_64 2022-09-12T18:35:14.678 INFO:teuthology.task.kernel:Running kernel on smithi138: 4.18.0-408.el8.x86_64 2022-09-12T18:35:14.679 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum install -y kernel 2022-09-12T18:35:14.684 INFO:teuthology.orchestra.run.smithi035.stdout:4.18.0-408.el8.x86_64 2022-09-12T18:35:14.685 INFO:teuthology.task.kernel:Running kernel on smithi035: 4.18.0-408.el8.x86_64 2022-09-12T18:35:14.686 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum install -y kernel 2022-09-12T18:35:17.934 INFO:teuthology.orchestra.run.smithi035.stdout:CentOS-8 - AppStream 28 MB/s | 25 MB 00:00 2022-09-12T18:35:17.935 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-8 - AppStream 29 MB/s | 25 MB 00:00 2022-09-12T18:35:22.810 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-8 - BaseOS 44 MB/s | 25 MB 00:00 2022-09-12T18:35:23.226 INFO:teuthology.orchestra.run.smithi035.stdout:CentOS-8 - BaseOS 41 MB/s | 25 MB 00:00 2022-09-12T18:35:26.617 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-8 - Extras 191 kB/s | 18 kB 00:00 2022-09-12T18:35:26.986 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-8 - PowerTools 20 MB/s | 5.0 MB 00:00 2022-09-12T18:35:27.353 INFO:teuthology.orchestra.run.smithi035.stdout:CentOS-8 - Extras 104 kB/s | 18 kB 00:00 2022-09-12T18:35:27.744 INFO:teuthology.orchestra.run.smithi035.stdout:CentOS-8 - PowerTools 19 MB/s | 5.0 MB 00:00 2022-09-12T18:35:30.343 INFO:teuthology.orchestra.run.smithi138.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2022-09-12T18:35:30.751 INFO:teuthology.orchestra.run.smithi035.stdout:Copr repo for python3-asyncssh owned by ceph 1.8 kB/s | 3.5 kB 00:01 2022-09-12T18:35:31.477 INFO:teuthology.orchestra.run.smithi138.stdout:Extra Packages for Enterprise Linux 13 MB/s | 13 MB 00:01 2022-09-12T18:35:31.852 INFO:teuthology.orchestra.run.smithi035.stdout:Extra Packages for Enterprise Linux 13 MB/s | 13 MB 00:00 2022-09-12T18:35:34.552 INFO:teuthology.orchestra.run.smithi138.stdout:lab-extras 461 kB/s | 24 kB 00:00 2022-09-12T18:35:35.060 INFO:teuthology.orchestra.run.smithi035.stdout:lab-extras 634 kB/s | 24 kB 00:00 2022-09-12T18:35:37.358 INFO:teuthology.orchestra.run.smithi138.stdout:Package kernel-4.18.0-408.el8.x86_64 is already installed. 2022-09-12T18:35:37.359 INFO:teuthology.orchestra.run.smithi138.stdout:Package kernel-4.18.0-305.3.1.el8.x86_64 is already installed. 2022-09-12T18:35:37.474 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:35:37.476 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2022-09-12T18:35:37.476 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:35:37.581 DEBUG:teuthology.orchestra.run.smithi138:> echo no | sudo yum reinstall kernel || true 2022-09-12T18:35:38.050 INFO:teuthology.orchestra.run.smithi035.stdout:Package kernel-4.18.0-408.el8.x86_64 is already installed. 2022-09-12T18:35:38.051 INFO:teuthology.orchestra.run.smithi035.stdout:Package kernel-4.18.0-305.3.1.el8.x86_64 is already installed. 2022-09-12T18:35:38.148 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:00:04 ago on Mon 12 Sep 2022 06:35:34 PM UTC. 2022-09-12T18:35:38.169 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:35:38.170 INFO:teuthology.orchestra.run.smithi035.stdout:Nothing to do. 2022-09-12T18:35:38.171 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:35:38.284 DEBUG:teuthology.orchestra.run.smithi035:> echo no | sudo yum reinstall kernel || true 2022-09-12T18:35:38.882 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:00:03 ago on Mon 12 Sep 2022 06:35:35 PM UTC. 2022-09-12T18:35:39.149 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:35:39.150 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:35:39.151 INFO:teuthology.orchestra.run.smithi138.stdout: Package Architecture Version Repository Size 2022-09-12T18:35:39.151 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:35:39.151 INFO:teuthology.orchestra.run.smithi138.stdout:Reinstalling: 2022-09-12T18:35:39.151 INFO:teuthology.orchestra.run.smithi138.stdout: kernel x86_64 4.18.0-408.el8 CentOS-BaseOS 8.5 M 2022-09-12T18:35:39.152 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:35:39.152 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-09-12T18:35:39.152 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:35:39.152 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:35:39.153 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 8.5 M 2022-09-12T18:35:39.153 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 0 2022-09-12T18:35:39.154 INFO:teuthology.orchestra.run.smithi138.stderr:Operation aborted. 2022-09-12T18:35:39.231 INFO:teuthology.orchestra.run.smithi138.stdout:Is this ok [y/N]: 2022-09-12T18:35:39.231 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum reinstall -y kernel || true 2022-09-12T18:35:39.794 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:00:05 ago on Mon 12 Sep 2022 06:35:34 PM UTC. 2022-09-12T18:35:39.970 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:35:39.971 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:35:39.972 INFO:teuthology.orchestra.run.smithi035.stdout: Package Architecture Version Repository Size 2022-09-12T18:35:39.972 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:35:39.972 INFO:teuthology.orchestra.run.smithi035.stdout:Reinstalling: 2022-09-12T18:35:39.972 INFO:teuthology.orchestra.run.smithi035.stdout: kernel x86_64 4.18.0-408.el8 CentOS-BaseOS 8.5 M 2022-09-12T18:35:39.973 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:35:39.973 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2022-09-12T18:35:39.973 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:35:39.973 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:35:39.974 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 8.5 M 2022-09-12T18:35:39.974 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 0 2022-09-12T18:35:39.974 INFO:teuthology.orchestra.run.smithi035.stderr:Operation aborted. 2022-09-12T18:35:40.038 INFO:teuthology.orchestra.run.smithi035.stdout:Is this ok [y/N]: 2022-09-12T18:35:40.038 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum reinstall -y kernel || true 2022-09-12T18:35:40.631 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:00:05 ago on Mon 12 Sep 2022 06:35:35 PM UTC. 2022-09-12T18:35:40.805 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:35:40.806 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:35:40.807 INFO:teuthology.orchestra.run.smithi138.stdout: Package Architecture Version Repository Size 2022-09-12T18:35:40.807 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:35:40.807 INFO:teuthology.orchestra.run.smithi138.stdout:Reinstalling: 2022-09-12T18:35:40.808 INFO:teuthology.orchestra.run.smithi138.stdout: kernel x86_64 4.18.0-408.el8 CentOS-BaseOS 8.5 M 2022-09-12T18:35:40.808 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:35:40.808 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-09-12T18:35:40.809 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:35:40.809 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:35:40.809 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 8.5 M 2022-09-12T18:35:40.810 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 0 2022-09-12T18:35:40.810 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-09-12T18:35:41.067 INFO:teuthology.orchestra.run.smithi138.stdout:kernel-4.18.0-408.el8.x86_64.rpm 33 MB/s | 8.5 MB 00:00 2022-09-12T18:35:41.067 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:35:41.068 INFO:teuthology.orchestra.run.smithi138.stdout:Total 33 MB/s | 8.5 MB 00:00 2022-09-12T18:35:41.169 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-09-12T18:35:41.552 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-09-12T18:35:41.552 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-09-12T18:35:41.703 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:35:41.704 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:35:41.704 INFO:teuthology.orchestra.run.smithi035.stdout: Package Architecture Version Repository Size 2022-09-12T18:35:41.705 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:35:41.705 INFO:teuthology.orchestra.run.smithi035.stdout:Reinstalling: 2022-09-12T18:35:41.705 INFO:teuthology.orchestra.run.smithi035.stdout: kernel x86_64 4.18.0-408.el8 CentOS-BaseOS 8.5 M 2022-09-12T18:35:41.705 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:35:41.706 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2022-09-12T18:35:41.706 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:35:41.706 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:35:41.707 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 8.5 M 2022-09-12T18:35:41.707 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 0 2022-09-12T18:35:41.708 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2022-09-12T18:35:41.721 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-09-12T18:35:41.733 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-09-12T18:35:41.938 INFO:teuthology.orchestra.run.smithi035.stdout:kernel-4.18.0-408.el8.x86_64.rpm 37 MB/s | 8.5 MB 00:00 2022-09-12T18:35:41.939 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:35:41.940 INFO:teuthology.orchestra.run.smithi035.stdout:Total 37 MB/s | 8.5 MB 00:00 2022-09-12T18:35:42.024 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-09-12T18:35:42.036 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2022-09-12T18:35:42.214 INFO:teuthology.orchestra.run.smithi138.stdout: Reinstalling : kernel-4.18.0-408.el8.x86_64 1/2 2022-09-12T18:35:42.214 INFO:teuthology.orchestra.run.smithi138.stdout: Cleanup : kernel-4.18.0-408.el8.x86_64 2/2 2022-09-12T18:35:42.440 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2022-09-12T18:35:42.441 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2022-09-12T18:35:42.614 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-4.18.0-408.el8.x86_64 2/2 2022-09-12T18:35:42.615 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : kernel-4.18.0-408.el8.x86_64 1/2 2022-09-12T18:35:42.628 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2022-09-12T18:35:42.647 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2022-09-12T18:35:42.949 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2022-09-12T18:35:43.086 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : kernel-4.18.0-408.el8.x86_64 2/2 2022-09-12T18:35:43.087 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:35:43.087 INFO:teuthology.orchestra.run.smithi138.stdout:Reinstalled: 2022-09-12T18:35:43.087 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-4.18.0-408.el8.x86_64 2022-09-12T18:35:43.087 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:35:43.088 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:35:43.169 DEBUG:teuthology.orchestra.run.smithi138:> rpm -q kernel --last | head -n 1 2022-09-12T18:35:43.177 INFO:teuthology.orchestra.run.smithi035.stdout: Reinstalling : kernel-4.18.0-408.el8.x86_64 1/2 2022-09-12T18:35:43.182 INFO:teuthology.orchestra.run.smithi035.stdout: Cleanup : kernel-4.18.0-408.el8.x86_64 2/2 2022-09-12T18:35:43.297 INFO:teuthology.orchestra.run.smithi138.stdout:kernel-4.18.0-408.el8.x86_64 Mon 12 Sep 2022 06:35:42 PM UTC 2022-09-12T18:35:43.299 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-408.el8.x86_64 2022-09-12T18:35:43.299 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-09-12T18:35:43.299 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2022-09-12T18:35:43.300 DEBUG:teuthology.orchestra.run.smithi138:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-09-12T18:35:43.349 INFO:teuthology.orchestra.run.smithi138.stdout:ttyS1 2022-09-12T18:35:43.363 DEBUG:teuthology.parallel:result is None 2022-09-12T18:35:43.587 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: kernel-4.18.0-408.el8.x86_64 2/2 2022-09-12T18:35:43.587 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : kernel-4.18.0-408.el8.x86_64 1/2 2022-09-12T18:35:44.171 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : kernel-4.18.0-408.el8.x86_64 2/2 2022-09-12T18:35:44.172 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:35:44.172 INFO:teuthology.orchestra.run.smithi035.stdout:Reinstalled: 2022-09-12T18:35:44.173 INFO:teuthology.orchestra.run.smithi035.stdout: kernel-4.18.0-408.el8.x86_64 2022-09-12T18:35:44.173 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:35:44.173 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:35:44.254 DEBUG:teuthology.orchestra.run.smithi035:> rpm -q kernel --last | head -n 1 2022-09-12T18:35:44.427 INFO:teuthology.orchestra.run.smithi035.stdout:kernel-4.18.0-408.el8.x86_64 Mon 12 Sep 2022 06:35:42 PM UTC 2022-09-12T18:35:44.429 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-408.el8.x86_64 2022-09-12T18:35:44.430 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-09-12T18:35:44.431 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2022-09-12T18:35:44.431 DEBUG:teuthology.orchestra.run.smithi035:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-09-12T18:35:44.466 INFO:teuthology.orchestra.run.smithi035.stdout:ttyS1 2022-09-12T18:35:44.481 DEBUG:teuthology.parallel:result is None 2022-09-12T18:35:44.482 INFO:teuthology.run_tasks:Running task internal.base... 2022-09-12T18:35:44.503 INFO:teuthology.task.internal:Creating test directory... 2022-09-12T18:35:44.504 DEBUG:teuthology.orchestra.run.smithi035:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-09-12T18:35:44.524 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-09-12T18:35:44.544 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-09-12T18:35:44.556 INFO:teuthology.run_tasks:Running task internal.archive... 2022-09-12T18:35:44.567 INFO:teuthology.task.internal:Creating archive directory... 2022-09-12T18:35:44.567 DEBUG:teuthology.orchestra.run.smithi035:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-09-12T18:35:44.583 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-09-12T18:35:44.619 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-09-12T18:35:44.628 INFO:teuthology.task.internal:Enabling coredump saving... 2022-09-12T18:35:44.629 DEBUG:teuthology.orchestra.run.smithi035:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-09-12T18:35:44.654 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-09-12T18:35:44.683 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-12T18:35:44.690 INFO:teuthology.orchestra.run.smithi035.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-12T18:35:44.692 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-12T18:35:44.699 INFO:teuthology.orchestra.run.smithi035.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-12T18:35:44.701 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-09-12T18:35:44.712 INFO:teuthology.task.internal:Configuring sudo... 2022-09-12T18:35:44.713 DEBUG:teuthology.orchestra.run.smithi035:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-09-12T18:35:44.744 DEBUG:teuthology.orchestra.run.smithi138:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-09-12T18:35:44.772 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-09-12T18:35:44.785 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-09-12T18:35:44.786 DEBUG:teuthology.orchestra.run.smithi035:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-09-12T18:35:44.820 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-09-12T18:35:44.839 DEBUG:teuthology.orchestra.run.smithi035:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-12T18:35:44.927 DEBUG:teuthology.orchestra.run.smithi035:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-12T18:35:44.974 DEBUG:teuthology.orchestra.run.smithi035:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-12T18:35:45.070 DEBUG:teuthology.orchestra.run.smithi035:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-12T18:35:45.103 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2022-09-12T18:35:45.103 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-09-12T18:35:45.171 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-12T18:35:45.218 DEBUG:teuthology.orchestra.run.smithi138:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-12T18:35:45.292 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-12T18:35:45.343 DEBUG:teuthology.orchestra.run.smithi138:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-12T18:35:45.383 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-09-12T18:35:45.387 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-09-12T18:35:45.450 DEBUG:teuthology.orchestra.run.smithi035:> sudo service rsyslog restart 2022-09-12T18:35:45.453 DEBUG:teuthology.orchestra.run.smithi138:> sudo service rsyslog restart 2022-09-12T18:35:45.516 INFO:teuthology.orchestra.run.smithi035.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-09-12T18:35:45.552 INFO:teuthology.orchestra.run.smithi138.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-09-12T18:35:46.009 INFO:teuthology.run_tasks:Running task internal.timer... 2022-09-12T18:35:46.019 INFO:teuthology.task.internal:Starting timer... 2022-09-12T18:35:46.020 INFO:teuthology.run_tasks:Running task pcp... 2022-09-12T18:35:46.030 INFO:teuthology.run_tasks:Running task selinux... 2022-09-12T18:35:46.047 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2022-09-12T18:35:46.048 DEBUG:teuthology.orchestra.run.smithi035:> sudo service auditd rotate 2022-09-12T18:35:46.125 INFO:teuthology.orchestra.run.smithi035.stdout:Rotating logs: 2022-09-12T18:35:46.127 DEBUG:teuthology.orchestra.run.smithi138:> sudo service auditd rotate 2022-09-12T18:35:46.221 INFO:teuthology.orchestra.run.smithi138.stdout:Rotating logs: 2022-09-12T18:35:46.223 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-09-12T18:35:46.224 DEBUG:teuthology.orchestra.run.smithi035:> /usr/sbin/getenforce 2022-09-12T18:35:46.246 INFO:teuthology.orchestra.run.smithi035.stdout:Permissive 2022-09-12T18:35:46.247 DEBUG:teuthology.orchestra.run.smithi138:> /usr/sbin/getenforce 2022-09-12T18:35:46.276 INFO:teuthology.orchestra.run.smithi138.stdout:Permissive 2022-09-12T18:35:46.277 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi035.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi138.front.sepia.ceph.com': 'permissive'} 2022-09-12T18:35:46.277 DEBUG:teuthology.orchestra.run.smithi035:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2022-09-12T18:35:46.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-12T18:35:46.315 DEBUG:teuthology.orchestra.run.smithi138:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2022-09-12T18:35:46.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-12T18:35:46.344 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-09-12T18:35:46.345 DEBUG:teuthology.orchestra.run.smithi035:> sudo /usr/sbin/setenforce permissive 2022-09-12T18:35:46.392 DEBUG:teuthology.orchestra.run.smithi138:> sudo /usr/sbin/setenforce permissive 2022-09-12T18:35:46.427 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-09-12T18:35:46.446 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2022-09-12T18:35:46.447 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main to origin/main 2022-09-12T18:35:46.472 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-09-12T18:35:46.475 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi035.front.sepia.ceph.com,smithi138.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main/cephlab.yml 2022-09-12T18:41:53.542 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi035.front.sepia.ceph.com'), Remote(name='ubuntu@smithi138.front.sepia.ceph.com')] 2022-09-12T18:41:53.546 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi035.front.sepia.ceph.com' 2022-09-12T18:41:53.548 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi035.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-12T18:41:53.635 DEBUG:teuthology.orchestra.run.smithi035:> true 2022-09-12T18:41:53.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2022-09-12T18:41:53.705 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-12T18:41:53.801 DEBUG:teuthology.orchestra.run.smithi138:> true 2022-09-12T18:41:53.872 INFO:teuthology.run_tasks:Running task clock... 2022-09-12T18:41:53.886 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-09-12T18:41:53.887 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-09-12T18:41:53.889 DEBUG:teuthology.orchestra.run.smithi035:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-09-12T18:41:53.892 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-09-12T18:41:53.892 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-09-12T18:41:53.923 INFO:teuthology.orchestra.run.smithi035.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-09-12T18:41:53.938 INFO:teuthology.orchestra.run.smithi035.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-09-12T18:41:53.943 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-09-12T18:41:53.958 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-09-12T18:41:54.004 INFO:teuthology.orchestra.run.smithi035.stderr:sudo: ntpd: command not found 2022-09-12T18:41:54.015 INFO:teuthology.orchestra.run.smithi035.stdout:506 Cannot talk to daemon 2022-09-12T18:41:54.029 INFO:teuthology.orchestra.run.smithi035.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-09-12T18:41:54.030 INFO:teuthology.orchestra.run.smithi138.stderr:sudo: ntpd: command not found 2022-09-12T18:41:54.041 INFO:teuthology.orchestra.run.smithi138.stdout:506 Cannot talk to daemon 2022-09-12T18:41:54.044 INFO:teuthology.orchestra.run.smithi035.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-09-12T18:41:54.054 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-09-12T18:41:54.067 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-09-12T18:41:54.124 INFO:teuthology.orchestra.run.smithi035.stderr:bash: ntpq: command not found 2022-09-12T18:41:54.128 INFO:teuthology.orchestra.run.smithi035.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-09-12T18:41:54.129 INFO:teuthology.orchestra.run.smithi035.stdout:=============================================================================== 2022-09-12T18:41:54.129 INFO:teuthology.orchestra.run.smithi035.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-12T18:41:54.129 INFO:teuthology.orchestra.run.smithi035.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-12T18:41:54.129 INFO:teuthology.orchestra.run.smithi035.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-12T18:41:54.130 INFO:teuthology.orchestra.run.smithi035.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-12T18:41:54.135 INFO:teuthology.orchestra.run.smithi138.stderr:bash: ntpq: command not found 2022-09-12T18:41:54.139 INFO:teuthology.orchestra.run.smithi138.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-09-12T18:41:54.139 INFO:teuthology.orchestra.run.smithi138.stdout:=============================================================================== 2022-09-12T18:41:54.139 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-12T18:41:54.139 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-12T18:41:54.140 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-12T18:41:54.140 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-12T18:41:54.141 INFO:teuthology.run_tasks:Running task install... 2022-09-12T18:41:54.151 DEBUG:teuthology.task.install:project ceph 2022-09-12T18:41:54.152 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '10f7d25d6a0f340676cd251eab5a787ba01ace95'}} 2022-09-12T18:41:54.152 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '10f7d25d6a0f340676cd251eab5a787ba01ace95'} 2022-09-12T18:41:54.153 INFO:teuthology.task.install:Using flavor: default 2022-09-12T18:41:54.160 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-09-12T18:41:54.160 INFO:teuthology.task.install:extra packages: [] 2022-09-12T18:41:54.161 DEBUG:teuthology.orchestra.run.smithi035:> sudo lsb_release -is 2022-09-12T18:41:54.208 INFO:teuthology.orchestra.run.smithi035.stdout:CentOSStream 2022-09-12T18:41:54.210 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-09-12T18:41:54.211 DEBUG:teuthology.orchestra.run.smithi138:> sudo lsb_release -is 2022-09-12T18:41:54.212 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '10f7d25d6a0f340676cd251eab5a787ba01ace95', 'tag': None, 'wait_for_package': False} 2022-09-12T18:41:54.213 DEBUG:teuthology.orchestra.run.smithi035:> sudo lsb_release -is 2022-09-12T18:41:54.249 INFO:teuthology.orchestra.run.smithi138.stdout:CentOSStream 2022-09-12T18:41:54.250 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-09-12T18:41:54.251 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '10f7d25d6a0f340676cd251eab5a787ba01ace95', 'tag': None, 'wait_for_package': False} 2022-09-12T18:41:54.251 DEBUG:teuthology.orchestra.run.smithi138:> sudo lsb_release -is 2022-09-12T18:41:54.289 INFO:teuthology.orchestra.run.smithi035.stdout:CentOSStream 2022-09-12T18:41:54.292 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-09-12T18:41:54.293 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=10f7d25d6a0f340676cd251eab5a787ba01ace95 2022-09-12T18:41:54.338 INFO:teuthology.orchestra.run.smithi138.stdout:CentOSStream 2022-09-12T18:41:54.338 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-09-12T18:41:54.339 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=10f7d25d6a0f340676cd251eab5a787ba01ace95 2022-09-12T18:41:54.455 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/main/10f7d25d6a0f340676cd251eab5a787ba01ace95/centos/8/flavors/default/ 2022-09-12T18:41:54.456 INFO:teuthology.task.install.rpm:Package version is 17.0.0-14883.g10f7d25d 2022-09-12T18:41:54.498 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/main/10f7d25d6a0f340676cd251eab5a787ba01ace95/centos/8/flavors/default/ 2022-09-12T18:41:54.499 INFO:teuthology.task.install.rpm:Package version is 17.0.0-14883.g10f7d25d 2022-09-12T18:41:54.580 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/main/10f7d25d6a0f340676cd251eab5a787ba01ace95/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/main/10f7d25d6a0f340676cd251eab5a787ba01ace95/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/main/10f7d25d6a0f340676cd251eab5a787ba01ace95/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-09-12T18:41:54.581 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2022-09-12T18:41:54.581 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-09-12T18:41:54.610 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2022-09-12T18:41:54.611 DEBUG:teuthology.orchestra.run.smithi035:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/10f7d25d6a0f340676cd251eab5a787ba01ace95/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-09-12T18:41:54.619 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/main/10f7d25d6a0f340676cd251eab5a787ba01ace95/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/main/10f7d25d6a0f340676cd251eab5a787ba01ace95/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/main/10f7d25d6a0f340676cd251eab5a787ba01ace95/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-09-12T18:41:54.620 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-09-12T18:41:54.621 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-09-12T18:41:54.648 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2022-09-12T18:41:54.649 DEBUG:teuthology.orchestra.run.smithi138:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/10f7d25d6a0f340676cd251eab5a787ba01ace95/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-09-12T18:41:54.680 DEBUG:teuthology.orchestra.run.smithi035:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-09-12T18:41:54.717 DEBUG:teuthology.orchestra.run.smithi138:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-09-12T18:41:54.757 DEBUG:teuthology.orchestra.run.smithi035:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-09-12T18:41:54.795 DEBUG:teuthology.orchestra.run.smithi138:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-09-12T18:41:54.825 INFO:teuthology.orchestra.run.smithi035.stdout:check_obsoletes = 1 2022-09-12T18:41:54.827 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum clean all 2022-09-12T18:41:54.862 INFO:teuthology.orchestra.run.smithi138.stdout:check_obsoletes = 1 2022-09-12T18:41:54.863 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum clean all 2022-09-12T18:41:55.084 INFO:teuthology.orchestra.run.smithi035.stdout:55 files removed 2022-09-12T18:41:55.110 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install ceph-radosgw 2022-09-12T18:41:55.111 INFO:teuthology.orchestra.run.smithi138.stdout:55 files removed 2022-09-12T18:41:55.134 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-radosgw 2022-09-12T18:41:55.811 INFO:teuthology.orchestra.run.smithi138.stdout:ceph packages for x86_64 266 kB/s | 81 kB 00:00 2022-09-12T18:41:55.850 INFO:teuthology.orchestra.run.smithi035.stdout:ceph packages for x86_64 217 kB/s | 81 kB 00:00 2022-09-12T18:41:56.094 INFO:teuthology.orchestra.run.smithi138.stdout:ceph noarch packages 38 kB/s | 9.4 kB 00:00 2022-09-12T18:41:56.151 INFO:teuthology.orchestra.run.smithi035.stdout:ceph noarch packages 36 kB/s | 9.4 kB 00:00 2022-09-12T18:41:56.370 INFO:teuthology.orchestra.run.smithi138.stdout:ceph source packages 6.0 kB/s | 1.5 kB 00:00 2022-09-12T18:41:56.409 INFO:teuthology.orchestra.run.smithi035.stdout:ceph source packages 6.5 kB/s | 1.5 kB 00:00 2022-09-12T18:41:57.189 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-8 - AppStream 31 MB/s | 25 MB 00:00 2022-09-12T18:41:57.204 INFO:teuthology.orchestra.run.smithi035.stdout:CentOS-8 - AppStream 32 MB/s | 25 MB 00:00 2022-09-12T18:42:02.131 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-8 - BaseOS 44 MB/s | 25 MB 00:00 2022-09-12T18:42:02.596 INFO:teuthology.orchestra.run.smithi035.stdout:CentOS-8 - BaseOS 36 MB/s | 25 MB 00:00 2022-09-12T18:42:07.131 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-8 - Extras 14 kB/s | 18 kB 00:01 2022-09-12T18:42:07.210 INFO:teuthology.orchestra.run.smithi035.stdout:CentOS-8 - Extras 26 kB/s | 18 kB 00:00 2022-09-12T18:42:07.501 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-8 - PowerTools 20 MB/s | 5.0 MB 00:00 2022-09-12T18:42:07.640 INFO:teuthology.orchestra.run.smithi035.stdout:CentOS-8 - PowerTools 17 MB/s | 5.0 MB 00:00 2022-09-12T18:42:10.906 INFO:teuthology.orchestra.run.smithi138.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2022-09-12T18:42:10.930 INFO:teuthology.orchestra.run.smithi035.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2022-09-12T18:42:12.021 INFO:teuthology.orchestra.run.smithi035.stdout:Extra Packages for Enterprise Linux 13 MB/s | 13 MB 00:00 2022-09-12T18:42:12.048 INFO:teuthology.orchestra.run.smithi138.stdout:Extra Packages for Enterprise Linux 13 MB/s | 13 MB 00:01 2022-09-12T18:42:15.057 INFO:teuthology.orchestra.run.smithi138.stdout:lab-extras 531 kB/s | 24 kB 00:00 2022-09-12T18:42:15.222 INFO:teuthology.orchestra.run.smithi035.stdout:lab-extras 283 kB/s | 24 kB 00:00 2022-09-12T18:42:18.121 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:42:18.123 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:42:18.123 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2022-09-12T18:42:18.123 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:42:18.124 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-09-12T18:42:18.124 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-radosgw x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 18 M 2022-09-12T18:42:18.124 INFO:teuthology.orchestra.run.smithi138.stdout:Upgrading: 2022-09-12T18:42:18.124 INFO:teuthology.orchestra.run.smithi138.stdout: librados2 x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 3.6 M 2022-09-12T18:42:18.125 INFO:teuthology.orchestra.run.smithi138.stdout: librbd1 x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 3.6 M 2022-09-12T18:42:18.125 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2022-09-12T18:42:18.125 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-base x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 5.2 M 2022-09-12T18:42:18.125 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-common x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 24 M 2022-09-12T18:42:18.126 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-selinux x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 24 k 2022-09-12T18:42:18.126 INFO:teuthology.orchestra.run.smithi138.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-09-12T18:42:18.126 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs2 x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 766 k 2022-09-12T18:42:18.126 INFO:teuthology.orchestra.run.smithi138.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-09-12T18:42:18.127 INFO:teuthology.orchestra.run.smithi138.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-BaseOS 47 k 2022-09-12T18:42:18.127 INFO:teuthology.orchestra.run.smithi138.stdout: libradosstriper1 x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 473 k 2022-09-12T18:42:18.127 INFO:teuthology.orchestra.run.smithi138.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2022-09-12T18:42:18.127 INFO:teuthology.orchestra.run.smithi138.stdout: librgw2 x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 5.8 M 2022-09-12T18:42:18.128 INFO:teuthology.orchestra.run.smithi138.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-09-12T18:42:18.128 INFO:teuthology.orchestra.run.smithi138.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2022-09-12T18:42:18.128 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-argparse 2022-09-12T18:42:18.128 INFO:teuthology.orchestra.run.smithi138.stdout: x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 45 k 2022-09-12T18:42:18.128 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-common x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 109 k 2022-09-12T18:42:18.129 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cephfs x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 190 k 2022-09-12T18:42:18.129 INFO:teuthology.orchestra.run.smithi138.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2022-09-12T18:42:18.129 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rados x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 370 k 2022-09-12T18:42:18.129 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rbd x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 357 k 2022-09-12T18:42:18.130 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rgw x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 107 k 2022-09-12T18:42:18.130 INFO:teuthology.orchestra.run.smithi138.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-09-12T18:42:18.130 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:42:18.130 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-09-12T18:42:18.131 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:42:18.131 INFO:teuthology.orchestra.run.smithi138.stdout:Install 21 Packages 2022-09-12T18:42:18.131 INFO:teuthology.orchestra.run.smithi138.stdout:Upgrade 2 Packages 2022-09-12T18:42:18.131 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:42:18.132 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 65 M 2022-09-12T18:42:18.132 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-09-12T18:42:18.487 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:42:18.489 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:42:18.490 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repository Size 2022-09-12T18:42:18.490 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:42:18.491 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2022-09-12T18:42:18.491 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-radosgw x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 18 M 2022-09-12T18:42:18.491 INFO:teuthology.orchestra.run.smithi035.stdout:Upgrading: 2022-09-12T18:42:18.492 INFO:teuthology.orchestra.run.smithi035.stdout: librados2 x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 3.6 M 2022-09-12T18:42:18.492 INFO:teuthology.orchestra.run.smithi035.stdout: librbd1 x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 3.6 M 2022-09-12T18:42:18.493 INFO:teuthology.orchestra.run.smithi035.stdout:Installing dependencies: 2022-09-12T18:42:18.493 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-base x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 5.2 M 2022-09-12T18:42:18.494 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-common x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 24 M 2022-09-12T18:42:18.494 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-selinux x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 24 k 2022-09-12T18:42:18.494 INFO:teuthology.orchestra.run.smithi035.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-09-12T18:42:18.494 INFO:teuthology.orchestra.run.smithi035.stdout: libcephfs2 x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 766 k 2022-09-12T18:42:18.495 INFO:teuthology.orchestra.run.smithi035.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-09-12T18:42:18.495 INFO:teuthology.orchestra.run.smithi035.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-BaseOS 47 k 2022-09-12T18:42:18.495 INFO:teuthology.orchestra.run.smithi035.stdout: libradosstriper1 x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 473 k 2022-09-12T18:42:18.495 INFO:teuthology.orchestra.run.smithi035.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2022-09-12T18:42:18.496 INFO:teuthology.orchestra.run.smithi035.stdout: librgw2 x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 5.8 M 2022-09-12T18:42:18.496 INFO:teuthology.orchestra.run.smithi035.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-09-12T18:42:18.496 INFO:teuthology.orchestra.run.smithi035.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2022-09-12T18:42:18.497 INFO:teuthology.orchestra.run.smithi035.stdout: python3-ceph-argparse 2022-09-12T18:42:18.497 INFO:teuthology.orchestra.run.smithi035.stdout: x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 45 k 2022-09-12T18:42:18.497 INFO:teuthology.orchestra.run.smithi035.stdout: python3-ceph-common x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 109 k 2022-09-12T18:42:18.498 INFO:teuthology.orchestra.run.smithi035.stdout: python3-cephfs x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 190 k 2022-09-12T18:42:18.498 INFO:teuthology.orchestra.run.smithi035.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2022-09-12T18:42:18.498 INFO:teuthology.orchestra.run.smithi035.stdout: python3-rados x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 370 k 2022-09-12T18:42:18.498 INFO:teuthology.orchestra.run.smithi035.stdout: python3-rbd x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 357 k 2022-09-12T18:42:18.499 INFO:teuthology.orchestra.run.smithi035.stdout: python3-rgw x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 107 k 2022-09-12T18:42:18.499 INFO:teuthology.orchestra.run.smithi035.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-09-12T18:42:18.499 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:42:18.499 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2022-09-12T18:42:18.500 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:42:18.500 INFO:teuthology.orchestra.run.smithi035.stdout:Install 21 Packages 2022-09-12T18:42:18.501 INFO:teuthology.orchestra.run.smithi035.stdout:Upgrade 2 Packages 2022-09-12T18:42:18.501 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:42:18.502 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 65 M 2022-09-12T18:42:18.502 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2022-09-12T18:42:19.691 INFO:teuthology.orchestra.run.smithi138.stdout:(1/23): ceph-base-17.0.0-14883.g10f7d25d.el8.x8 3.3 MB/s | 5.2 MB 00:01 2022-09-12T18:42:19.775 INFO:teuthology.orchestra.run.smithi138.stdout:(2/23): ceph-selinux-17.0.0-14883.g10f7d25d.el8 291 kB/s | 24 kB 00:00 2022-09-12T18:42:20.366 INFO:teuthology.orchestra.run.smithi138.stdout:(3/23): libcephfs2-17.0.0-14883.g10f7d25d.el8.x 1.3 MB/s | 766 kB 00:00 2022-09-12T18:42:20.491 INFO:teuthology.orchestra.run.smithi138.stdout:(4/23): libradosstriper1-17.0.0-14883.g10f7d25d 3.7 MB/s | 473 kB 00:00 2022-09-12T18:42:20.573 INFO:teuthology.orchestra.run.smithi035.stdout:(1/23): ceph-base-17.0.0-14883.g10f7d25d.el8.x8 2.5 MB/s | 5.2 MB 00:02 2022-09-12T18:42:20.640 INFO:teuthology.orchestra.run.smithi035.stdout:(2/23): ceph-selinux-17.0.0-14883.g10f7d25d.el8 364 kB/s | 24 kB 00:00 2022-09-12T18:42:20.823 INFO:teuthology.orchestra.run.smithi035.stdout:(3/23): libcephfs2-17.0.0-14883.g10f7d25d.el8.x 4.1 MB/s | 766 kB 00:00 2022-09-12T18:42:20.949 INFO:teuthology.orchestra.run.smithi035.stdout:(4/23): libradosstriper1-17.0.0-14883.g10f7d25d 3.7 MB/s | 473 kB 00:00 2022-09-12T18:42:21.555 INFO:teuthology.orchestra.run.smithi138.stdout:(5/23): librgw2-17.0.0-14883.g10f7d25d.el8.x86_ 5.5 MB/s | 5.8 MB 00:01 2022-09-12T18:42:21.638 INFO:teuthology.orchestra.run.smithi138.stdout:(6/23): python3-ceph-argparse-17.0.0-14883.g10f 567 kB/s | 45 kB 00:00 2022-09-12T18:42:21.657 INFO:teuthology.orchestra.run.smithi035.stdout:(5/23): librgw2-17.0.0-14883.g10f7d25d.el8.x86_ 8.2 MB/s | 5.8 MB 00:00 2022-09-12T18:42:21.701 INFO:teuthology.orchestra.run.smithi138.stdout:(7/23): python3-ceph-common-17.0.0-14883.g10f7d 1.6 MB/s | 109 kB 00:00 2022-09-12T18:42:21.749 INFO:teuthology.orchestra.run.smithi035.stdout:(6/23): python3-ceph-argparse-17.0.0-14883.g10f 487 kB/s | 45 kB 00:00 2022-09-12T18:42:21.776 INFO:teuthology.orchestra.run.smithi138.stdout:(8/23): python3-cephfs-17.0.0-14883.g10f7d25d.e 2.5 MB/s | 190 kB 00:00 2022-09-12T18:42:21.837 INFO:teuthology.orchestra.run.smithi035.stdout:(7/23): python3-ceph-common-17.0.0-14883.g10f7d 1.3 MB/s | 109 kB 00:00 2022-09-12T18:42:21.901 INFO:teuthology.orchestra.run.smithi138.stdout:(9/23): python3-rados-17.0.0-14883.g10f7d25d.el 2.9 MB/s | 370 kB 00:00 2022-09-12T18:42:21.908 INFO:teuthology.orchestra.run.smithi035.stdout:(8/23): python3-cephfs-17.0.0-14883.g10f7d25d.e 2.5 MB/s | 190 kB 00:00 2022-09-12T18:42:22.008 INFO:teuthology.orchestra.run.smithi035.stdout:(9/23): python3-rados-17.0.0-14883.g10f7d25d.el 3.6 MB/s | 370 kB 00:00 2022-09-12T18:42:22.027 INFO:teuthology.orchestra.run.smithi138.stdout:(10/23): python3-rbd-17.0.0-14883.g10f7d25d.el8 2.8 MB/s | 357 kB 00:00 2022-09-12T18:42:22.093 INFO:teuthology.orchestra.run.smithi035.stdout:(10/23): python3-rbd-17.0.0-14883.g10f7d25d.el8 4.2 MB/s | 357 kB 00:00 2022-09-12T18:42:22.102 INFO:teuthology.orchestra.run.smithi138.stdout:(11/23): python3-rgw-17.0.0-14883.g10f7d25d.el8 1.4 MB/s | 107 kB 00:00 2022-09-12T18:42:22.159 INFO:teuthology.orchestra.run.smithi035.stdout:(11/23): python3-rgw-17.0.0-14883.g10f7d25d.el8 1.6 MB/s | 107 kB 00:00 2022-09-12T18:42:22.161 INFO:teuthology.orchestra.run.smithi138.stdout:(12/23): librdkafka-0.11.4-3.el8.x86_64.rpm 5.9 MB/s | 354 kB 00:00 2022-09-12T18:42:22.187 INFO:teuthology.orchestra.run.smithi138.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 10 MB/s | 259 kB 00:00 2022-09-12T18:42:22.208 INFO:teuthology.orchestra.run.smithi138.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 2.6 MB/s | 44 kB 00:00 2022-09-12T18:42:22.218 INFO:teuthology.orchestra.run.smithi035.stdout:(12/23): librdkafka-0.11.4-3.el8.x86_64.rpm 5.9 MB/s | 354 kB 00:00 2022-09-12T18:42:22.220 INFO:teuthology.orchestra.run.smithi138.stdout:(15/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 2.8 MB/s | 47 kB 00:00 2022-09-12T18:42:22.245 INFO:teuthology.orchestra.run.smithi035.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 10 MB/s | 259 kB 00:00 2022-09-12T18:42:22.274 INFO:teuthology.orchestra.run.smithi035.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 1.7 MB/s | 44 kB 00:00 2022-09-12T18:42:22.294 INFO:teuthology.orchestra.run.smithi035.stdout:(15/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.8 MB/s | 47 kB 00:00 2022-09-12T18:42:24.019 INFO:teuthology.orchestra.run.smithi035.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 177 kB/s | 306 kB 00:01 2022-09-12T18:42:24.020 INFO:teuthology.orchestra.run.smithi138.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 170 kB/s | 306 kB 00:01 2022-09-12T18:42:24.070 INFO:teuthology.orchestra.run.smithi138.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 1.2 MB/s | 59 kB 00:00 2022-09-12T18:42:24.077 INFO:teuthology.orchestra.run.smithi035.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2022-09-12T18:42:24.129 INFO:teuthology.orchestra.run.smithi138.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 2022-09-12T18:42:24.144 INFO:teuthology.orchestra.run.smithi035.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2022-09-12T18:42:24.421 INFO:teuthology.orchestra.run.smithi138.stdout:(19/23): thrift-0.13.0-2.el8.x86_64.rpm 5.9 MB/s | 1.7 MB 00:00 2022-09-12T18:42:24.445 INFO:teuthology.orchestra.run.smithi035.stdout:(19/23): thrift-0.13.0-2.el8.x86_64.rpm 5.8 MB/s | 1.7 MB 00:00 2022-09-12T18:42:24.953 INFO:teuthology.orchestra.run.smithi035.stdout:(20/23): librados2-17.0.0-14883.g10f7d25d.el8.x 7.1 MB/s | 3.6 MB 00:00 2022-09-12T18:42:25.063 INFO:teuthology.orchestra.run.smithi138.stdout:(20/23): librados2-17.0.0-14883.g10f7d25d.el8.x 5.6 MB/s | 3.6 MB 00:00 2022-09-12T18:42:25.331 INFO:teuthology.orchestra.run.smithi138.stdout:(21/23): ceph-common-17.0.0-14883.g10f7d25d.el8 3.3 MB/s | 24 MB 00:07 2022-09-12T18:42:25.388 INFO:teuthology.orchestra.run.smithi035.stdout:(21/23): ceph-common-17.0.0-14883.g10f7d25d.el8 3.5 MB/s | 24 MB 00:06 2022-09-12T18:42:25.445 INFO:teuthology.orchestra.run.smithi035.stdout:(22/23): librbd1-17.0.0-14883.g10f7d25d.el8.x86 7.3 MB/s | 3.6 MB 00:00 2022-09-12T18:42:25.655 INFO:teuthology.orchestra.run.smithi138.stdout:(22/23): ceph-radosgw-17.0.0-14883.g10f7d25d.el 2.4 MB/s | 18 MB 00:07 2022-09-12T18:42:25.738 INFO:teuthology.orchestra.run.smithi138.stdout:(23/23): librbd1-17.0.0-14883.g10f7d25d.el8.x86 5.3 MB/s | 3.6 MB 00:00 2022-09-12T18:42:25.741 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:42:25.743 INFO:teuthology.orchestra.run.smithi138.stdout:Total 8.6 MB/s | 65 MB 00:07 2022-09-12T18:42:25.778 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-09-12T18:42:25.831 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-09-12T18:42:25.831 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-09-12T18:42:26.549 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-09-12T18:42:26.552 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-09-12T18:42:27.354 INFO:teuthology.orchestra.run.smithi035.stdout:(23/23): ceph-radosgw-17.0.0-14883.g10f7d25d.el 2.0 MB/s | 18 MB 00:08 2022-09-12T18:42:27.358 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:42:27.358 INFO:teuthology.orchestra.run.smithi035.stdout:Total 7.4 MB/s | 65 MB 00:08 2022-09-12T18:42:27.403 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2022-09-12T18:42:27.459 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2022-09-12T18:42:27.460 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2022-09-12T18:42:27.897 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-09-12T18:42:28.078 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2022-09-12T18:42:28.127 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2022-09-12T18:42:28.131 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2022-09-12T18:42:28.325 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2022-09-12T18:42:28.513 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2022-09-12T18:42:28.940 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2022-09-12T18:42:29.171 INFO:teuthology.orchestra.run.smithi138.stdout: Upgrading : librados2-2:17.0.0-14883.g10f7d25d.el8.x86_64 4/25 2022-09-12T18:42:29.273 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librados2-2:17.0.0-14883.g10f7d25d.el8.x86_64 4/25 2022-09-12T18:42:29.375 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2022-09-12T18:42:29.544 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rados-2:17.0.0-14883.g10f7d25d.el8.x86_64 5/25 2022-09-12T18:42:29.685 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2022-09-12T18:42:29.805 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephfs2-2:17.0.0-14883.g10f7d25d.el8.x86_64 6/25 2022-09-12T18:42:29.924 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2022-09-12T18:42:30.127 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2022-09-12T18:42:30.184 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libcephfs2-2:17.0.0-14883.g10f7d25d.el8.x86_64 6/25 2022-09-12T18:42:30.388 INFO:teuthology.orchestra.run.smithi138.stdout: Upgrading : librbd1-2:17.0.0-14883.g10f7d25d.el8.x86_64 7/25 2022-09-12T18:42:30.453 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librbd1-2:17.0.0-14883.g10f7d25d.el8.x86_64 7/25 2022-09-12T18:42:30.541 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2022-09-12T18:42:30.694 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2022-09-12T18:42:30.778 INFO:teuthology.orchestra.run.smithi035.stdout: Upgrading : librados2-2:17.0.0-14883.g10f7d25d.el8.x86_64 4/25 2022-09-12T18:42:30.846 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2022-09-12T18:42:30.865 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: librados2-2:17.0.0-14883.g10f7d25d.el8.x86_64 4/25 2022-09-12T18:42:30.915 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2022-09-12T18:42:31.107 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-ceph-argparse-2:17.0.0-14883.g10f7d25d.el8 10/25 2022-09-12T18:42:31.122 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-rados-2:17.0.0-14883.g10f7d25d.el8.x86_64 5/25 2022-09-12T18:42:31.291 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cephfs-2:17.0.0-14883.g10f7d25d.el8.x86_64 11/25 2022-09-12T18:42:31.345 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : libcephfs2-2:17.0.0-14883.g10f7d25d.el8.x86_64 6/25 2022-09-12T18:42:31.731 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rbd-2:17.0.0-14883.g10f7d25d.el8.x86_64 12/25 2022-09-12T18:42:31.755 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: libcephfs2-2:17.0.0-14883.g10f7d25d.el8.x86_64 6/25 2022-09-12T18:42:31.917 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libradosstriper1-2:17.0.0-14883.g10f7d25d.el8.x86_ 13/25 2022-09-12T18:42:31.961 INFO:teuthology.orchestra.run.smithi035.stdout: Upgrading : librbd1-2:17.0.0-14883.g10f7d25d.el8.x86_64 7/25 2022-09-12T18:42:31.972 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libradosstriper1-2:17.0.0-14883.g10f7d25d.el8.x86_ 13/25 2022-09-12T18:42:32.027 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: librbd1-2:17.0.0-14883.g10f7d25d.el8.x86_64 7/25 2022-09-12T18:42:32.186 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2022-09-12T18:42:32.207 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2022-09-12T18:42:32.378 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2022-09-12T18:42:32.448 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2022-09-12T18:42:32.624 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-ceph-argparse-2:17.0.0-14883.g10f7d25d.el8 10/25 2022-09-12T18:42:32.852 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-cephfs-2:17.0.0-14883.g10f7d25d.el8.x86_64 11/25 2022-09-12T18:42:32.912 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2022-09-12T18:42:33.074 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-rbd-2:17.0.0-14883.g10f7d25d.el8.x86_64 12/25 2022-09-12T18:42:33.079 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librgw2-2:17.0.0-14883.g10f7d25d.el8.x86_64 16/25 2022-09-12T18:42:33.186 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librgw2-2:17.0.0-14883.g10f7d25d.el8.x86_64 16/25 2022-09-12T18:42:33.275 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : libradosstriper1-2:17.0.0-14883.g10f7d25d.el8.x86_ 13/25 2022-09-12T18:42:33.331 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: libradosstriper1-2:17.0.0-14883.g10f7d25d.el8.x86_ 13/25 2022-09-12T18:42:33.354 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rgw-2:17.0.0-14883.g10f7d25d.el8.x86_64 17/25 2022-09-12T18:42:33.527 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2022-09-12T18:42:33.575 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2022-09-12T18:42:33.721 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-ceph-common-2:17.0.0-14883.g10f7d25d.el8.x 19/25 2022-09-12T18:42:34.314 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2022-09-12T18:42:34.544 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : librgw2-2:17.0.0-14883.g10f7d25d.el8.x86_64 16/25 2022-09-12T18:42:34.651 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: librgw2-2:17.0.0-14883.g10f7d25d.el8.x86_64 16/25 2022-09-12T18:42:34.812 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-rgw-2:17.0.0-14883.g10f7d25d.el8.x86_64 17/25 2022-09-12T18:42:35.023 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2022-09-12T18:42:35.186 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-ceph-common-2:17.0.0-14883.g10f7d25d.el8.x 19/25 2022-09-12T18:42:37.474 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-common-2:17.0.0-14883.g10f7d25d.el8.x86_64 20/25 2022-09-12T18:42:37.721 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-common-2:17.0.0-14883.g10f7d25d.el8.x86_64 20/25 2022-09-12T18:42:38.308 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-common-2:17.0.0-14883.g10f7d25d.el8.x86_64 20/25 2022-09-12T18:42:38.613 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-base-2:17.0.0-14883.g10f7d25d.el8.x86_64 21/25 2022-09-12T18:42:38.707 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-base-2:17.0.0-14883.g10f7d25d.el8.x86_64 21/25 2022-09-12T18:42:38.837 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-selinux-2:17.0.0-14883.g10f7d25d.el8.x86_64 22/25 2022-09-12T18:42:39.220 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-common-2:17.0.0-14883.g10f7d25d.el8.x86_64 20/25 2022-09-12T18:42:39.436 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-common-2:17.0.0-14883.g10f7d25d.el8.x86_64 20/25 2022-09-12T18:42:40.048 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-common-2:17.0.0-14883.g10f7d25d.el8.x86_64 20/25 2022-09-12T18:42:40.423 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-base-2:17.0.0-14883.g10f7d25d.el8.x86_64 21/25 2022-09-12T18:42:40.520 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-base-2:17.0.0-14883.g10f7d25d.el8.x86_64 21/25 2022-09-12T18:42:40.677 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-selinux-2:17.0.0-14883.g10f7d25d.el8.x86_64 22/25 2022-09-12T18:42:58.263 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-selinux-2:17.0.0-14883.g10f7d25d.el8.x86_64 22/25 2022-09-12T18:42:58.263 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /sys 2022-09-12T18:42:58.264 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /proc 2022-09-12T18:42:58.264 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /dev 2022-09-12T18:42:58.264 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /run 2022-09-12T18:42:58.264 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /mnt 2022-09-12T18:42:58.265 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /var/tmp 2022-09-12T18:42:58.265 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /home 2022-09-12T18:42:58.265 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /tmp 2022-09-12T18:42:58.265 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /dev 2022-09-12T18:42:58.266 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:43:00.034 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-radosgw-2:17.0.0-14883.g10f7d25d.el8.x86_64 23/25 2022-09-12T18:43:00.054 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-14883.g10f7d25d.el8.x86_64 23/25 2022-09-12T18:43:00.055 INFO:teuthology.orchestra.run.smithi138.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-09-12T18:43:00.229 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-09-12T18:43:00.229 INFO:teuthology.orchestra.run.smithi138.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2022-09-12T18:43:00.369 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-selinux-2:17.0.0-14883.g10f7d25d.el8.x86_64 22/25 2022-09-12T18:43:00.370 INFO:teuthology.orchestra.run.smithi035.stdout:skipping the directory /sys 2022-09-12T18:43:00.370 INFO:teuthology.orchestra.run.smithi035.stdout:skipping the directory /proc 2022-09-12T18:43:00.371 INFO:teuthology.orchestra.run.smithi035.stdout:skipping the directory /dev 2022-09-12T18:43:00.371 INFO:teuthology.orchestra.run.smithi035.stdout:skipping the directory /run 2022-09-12T18:43:00.371 INFO:teuthology.orchestra.run.smithi035.stdout:skipping the directory /mnt 2022-09-12T18:43:00.372 INFO:teuthology.orchestra.run.smithi035.stdout:skipping the directory /var/tmp 2022-09-12T18:43:00.372 INFO:teuthology.orchestra.run.smithi035.stdout:skipping the directory /home 2022-09-12T18:43:00.372 INFO:teuthology.orchestra.run.smithi035.stdout:skipping the directory /tmp 2022-09-12T18:43:00.373 INFO:teuthology.orchestra.run.smithi035.stdout:skipping the directory /dev 2022-09-12T18:43:00.373 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:43:00.886 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2022-09-12T18:43:00.886 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-base-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/25 2022-09-12T18:43:00.887 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-common-2:17.0.0-14883.g10f7d25d.el8.x86_64 2/25 2022-09-12T18:43:00.887 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-radosgw-2:17.0.0-14883.g10f7d25d.el8.x86_64 3/25 2022-09-12T18:43:00.888 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-selinux-2:17.0.0-14883.g10f7d25d.el8.x86_64 4/25 2022-09-12T18:43:00.888 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephfs2-2:17.0.0-14883.g10f7d25d.el8.x86_64 5/25 2022-09-12T18:43:00.888 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libradosstriper1-2:17.0.0-14883.g10f7d25d.el8.x86_ 6/25 2022-09-12T18:43:00.889 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librgw2-2:17.0.0-14883.g10f7d25d.el8.x86_64 7/25 2022-09-12T18:43:00.890 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-ceph-argparse-2:17.0.0-14883.g10f7d25d.el8 8/25 2022-09-12T18:43:00.891 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-ceph-common-2:17.0.0-14883.g10f7d25d.el8.x 9/25 2022-09-12T18:43:00.891 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cephfs-2:17.0.0-14883.g10f7d25d.el8.x86_64 10/25 2022-09-12T18:43:00.891 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rados-2:17.0.0-14883.g10f7d25d.el8.x86_64 11/25 2022-09-12T18:43:00.892 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rbd-2:17.0.0-14883.g10f7d25d.el8.x86_64 12/25 2022-09-12T18:43:00.892 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rgw-2:17.0.0-14883.g10f7d25d.el8.x86_64 13/25 2022-09-12T18:43:00.892 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2022-09-12T18:43:00.892 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2022-09-12T18:43:00.894 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2022-09-12T18:43:00.894 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2022-09-12T18:43:00.894 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2022-09-12T18:43:00.895 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2022-09-12T18:43:00.895 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2022-09-12T18:43:00.895 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2022-09-12T18:43:00.896 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados2-2:17.0.0-14883.g10f7d25d.el8.x86_64 22/25 2022-09-12T18:43:00.896 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2022-09-12T18:43:00.896 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librbd1-2:17.0.0-14883.g10f7d25d.el8.x86_64 24/25 2022-09-12T18:43:01.698 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2022-09-12T18:43:01.698 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:43:01.698 INFO:teuthology.orchestra.run.smithi138.stdout:Upgraded: 2022-09-12T18:43:01.699 INFO:teuthology.orchestra.run.smithi138.stdout: librados2-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:01.699 INFO:teuthology.orchestra.run.smithi138.stdout: librbd1-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:01.699 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-09-12T18:43:01.699 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-base-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:01.700 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-common-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:01.700 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-radosgw-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:01.700 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-selinux-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:01.700 INFO:teuthology.orchestra.run.smithi138.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-09-12T18:43:01.701 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs2-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:01.701 INFO:teuthology.orchestra.run.smithi138.stdout: liboath-2.6.2-3.el8.x86_64 2022-09-12T18:43:01.701 INFO:teuthology.orchestra.run.smithi138.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-09-12T18:43:01.701 INFO:teuthology.orchestra.run.smithi138.stdout: libradosstriper1-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:01.702 INFO:teuthology.orchestra.run.smithi138.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-09-12T18:43:01.702 INFO:teuthology.orchestra.run.smithi138.stdout: librgw2-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:01.702 INFO:teuthology.orchestra.run.smithi138.stdout: libunwind-1.3.1-3.el8.x86_64 2022-09-12T18:43:01.703 INFO:teuthology.orchestra.run.smithi138.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-09-12T18:43:01.703 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-argparse-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:01.704 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-common-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:01.704 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cephfs-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:01.704 INFO:teuthology.orchestra.run.smithi138.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-09-12T18:43:01.704 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rados-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:01.705 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rbd-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:01.705 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rgw-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:01.705 INFO:teuthology.orchestra.run.smithi138.stdout: thrift-0.13.0-2.el8.x86_64 2022-09-12T18:43:01.705 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:43:01.705 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:43:01.905 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-test 2022-09-12T18:43:02.262 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-radosgw-2:17.0.0-14883.g10f7d25d.el8.x86_64 23/25 2022-09-12T18:43:02.284 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-14883.g10f7d25d.el8.x86_64 23/25 2022-09-12T18:43:02.284 INFO:teuthology.orchestra.run.smithi035.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-09-12T18:43:02.514 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:00:47 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:43:02.518 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-09-12T18:43:02.518 INFO:teuthology.orchestra.run.smithi035.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2022-09-12T18:43:03.189 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2022-09-12T18:43:03.190 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-base-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/25 2022-09-12T18:43:03.190 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-common-2:17.0.0-14883.g10f7d25d.el8.x86_64 2/25 2022-09-12T18:43:03.190 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-radosgw-2:17.0.0-14883.g10f7d25d.el8.x86_64 3/25 2022-09-12T18:43:03.191 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-selinux-2:17.0.0-14883.g10f7d25d.el8.x86_64 4/25 2022-09-12T18:43:03.191 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : libcephfs2-2:17.0.0-14883.g10f7d25d.el8.x86_64 5/25 2022-09-12T18:43:03.191 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : libradosstriper1-2:17.0.0-14883.g10f7d25d.el8.x86_ 6/25 2022-09-12T18:43:03.191 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : librgw2-2:17.0.0-14883.g10f7d25d.el8.x86_64 7/25 2022-09-12T18:43:03.191 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-ceph-argparse-2:17.0.0-14883.g10f7d25d.el8 8/25 2022-09-12T18:43:03.192 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-ceph-common-2:17.0.0-14883.g10f7d25d.el8.x 9/25 2022-09-12T18:43:03.192 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-cephfs-2:17.0.0-14883.g10f7d25d.el8.x86_64 10/25 2022-09-12T18:43:03.192 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-rados-2:17.0.0-14883.g10f7d25d.el8.x86_64 11/25 2022-09-12T18:43:03.192 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-rbd-2:17.0.0-14883.g10f7d25d.el8.x86_64 12/25 2022-09-12T18:43:03.193 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-rgw-2:17.0.0-14883.g10f7d25d.el8.x86_64 13/25 2022-09-12T18:43:03.193 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2022-09-12T18:43:03.193 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2022-09-12T18:43:03.193 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2022-09-12T18:43:03.194 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2022-09-12T18:43:03.194 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2022-09-12T18:43:03.194 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2022-09-12T18:43:03.194 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2022-09-12T18:43:03.195 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2022-09-12T18:43:03.195 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : librados2-2:17.0.0-14883.g10f7d25d.el8.x86_64 22/25 2022-09-12T18:43:03.195 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2022-09-12T18:43:03.195 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : librbd1-2:17.0.0-14883.g10f7d25d.el8.x86_64 24/25 2022-09-12T18:43:03.539 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:43:03.540 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:43:03.541 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2022-09-12T18:43:03.541 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:43:03.541 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-09-12T18:43:03.542 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-test x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 54 M 2022-09-12T18:43:03.542 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2022-09-12T18:43:03.542 INFO:teuthology.orchestra.run.smithi138.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2022-09-12T18:43:03.542 INFO:teuthology.orchestra.run.smithi138.stdout: libcephsqlite x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 137 k 2022-09-12T18:43:03.543 INFO:teuthology.orchestra.run.smithi138.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2022-09-12T18:43:03.543 INFO:teuthology.orchestra.run.smithi138.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2022-09-12T18:43:03.543 INFO:teuthology.orchestra.run.smithi138.stdout: xmlstarlet x86_64 1.6.1-20.el8 CentOS-PowerTools 71 k 2022-09-12T18:43:03.543 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:43:03.544 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-09-12T18:43:03.544 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:43:03.544 INFO:teuthology.orchestra.run.smithi138.stdout:Install 6 Packages 2022-09-12T18:43:03.544 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:43:03.545 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 55 M 2022-09-12T18:43:03.545 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 238 M 2022-09-12T18:43:03.546 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-09-12T18:43:03.643 INFO:teuthology.orchestra.run.smithi138.stdout:(1/6): jq-1.6-3.el8.x86_64.rpm 2.0 MB/s | 202 kB 00:00 2022-09-12T18:43:03.728 INFO:teuthology.orchestra.run.smithi138.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 2.2 MB/s | 187 kB 00:00 2022-09-12T18:43:03.770 INFO:teuthology.orchestra.run.smithi138.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 7.5 MB/s | 323 kB 00:00 2022-09-12T18:43:03.798 INFO:teuthology.orchestra.run.smithi138.stdout:(4/6): libcephsqlite-17.0.0-14883.g10f7d25d.el8 543 kB/s | 137 kB 00:00 2022-09-12T18:43:03.837 INFO:teuthology.orchestra.run.smithi138.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.0 MB/s | 71 kB 00:00 2022-09-12T18:43:04.004 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2022-09-12T18:43:04.004 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:43:04.005 INFO:teuthology.orchestra.run.smithi035.stdout:Upgraded: 2022-09-12T18:43:04.005 INFO:teuthology.orchestra.run.smithi035.stdout: librados2-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:04.006 INFO:teuthology.orchestra.run.smithi035.stdout: librbd1-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:04.006 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2022-09-12T18:43:04.007 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-base-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:04.007 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-common-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:04.008 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-radosgw-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:04.008 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-selinux-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:04.008 INFO:teuthology.orchestra.run.smithi035.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-09-12T18:43:04.009 INFO:teuthology.orchestra.run.smithi035.stdout: libcephfs2-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:04.009 INFO:teuthology.orchestra.run.smithi035.stdout: liboath-2.6.2-3.el8.x86_64 2022-09-12T18:43:04.010 INFO:teuthology.orchestra.run.smithi035.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-09-12T18:43:04.010 INFO:teuthology.orchestra.run.smithi035.stdout: libradosstriper1-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:04.010 INFO:teuthology.orchestra.run.smithi035.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-09-12T18:43:04.011 INFO:teuthology.orchestra.run.smithi035.stdout: librgw2-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:04.011 INFO:teuthology.orchestra.run.smithi035.stdout: libunwind-1.3.1-3.el8.x86_64 2022-09-12T18:43:04.012 INFO:teuthology.orchestra.run.smithi035.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-09-12T18:43:04.012 INFO:teuthology.orchestra.run.smithi035.stdout: python3-ceph-argparse-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:04.013 INFO:teuthology.orchestra.run.smithi035.stdout: python3-ceph-common-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:04.013 INFO:teuthology.orchestra.run.smithi035.stdout: python3-cephfs-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:04.014 INFO:teuthology.orchestra.run.smithi035.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-09-12T18:43:04.014 INFO:teuthology.orchestra.run.smithi035.stdout: python3-rados-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:04.014 INFO:teuthology.orchestra.run.smithi035.stdout: python3-rbd-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:04.014 INFO:teuthology.orchestra.run.smithi035.stdout: python3-rgw-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:04.015 INFO:teuthology.orchestra.run.smithi035.stdout: thrift-0.13.0-2.el8.x86_64 2022-09-12T18:43:04.015 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:43:04.015 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:43:04.172 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install ceph-test 2022-09-12T18:43:04.777 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:00:49 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:43:05.882 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:43:05.883 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:43:05.883 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repository Size 2022-09-12T18:43:05.883 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:43:05.884 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2022-09-12T18:43:05.884 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-test x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 54 M 2022-09-12T18:43:05.884 INFO:teuthology.orchestra.run.smithi035.stdout:Installing dependencies: 2022-09-12T18:43:05.884 INFO:teuthology.orchestra.run.smithi035.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2022-09-12T18:43:05.885 INFO:teuthology.orchestra.run.smithi035.stdout: libcephsqlite x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 137 k 2022-09-12T18:43:05.885 INFO:teuthology.orchestra.run.smithi035.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2022-09-12T18:43:05.885 INFO:teuthology.orchestra.run.smithi035.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2022-09-12T18:43:05.885 INFO:teuthology.orchestra.run.smithi035.stdout: xmlstarlet x86_64 1.6.1-20.el8 CentOS-PowerTools 71 k 2022-09-12T18:43:05.886 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:43:05.886 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2022-09-12T18:43:05.887 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:43:05.887 INFO:teuthology.orchestra.run.smithi035.stdout:Install 6 Packages 2022-09-12T18:43:05.887 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:43:05.890 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 55 M 2022-09-12T18:43:05.890 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 238 M 2022-09-12T18:43:05.890 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2022-09-12T18:43:05.958 INFO:teuthology.orchestra.run.smithi035.stdout:(1/6): jq-1.6-3.el8.x86_64.rpm 2.8 MB/s | 202 kB 00:00 2022-09-12T18:43:06.041 INFO:teuthology.orchestra.run.smithi035.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 2.5 MB/s | 187 kB 00:00 2022-09-12T18:43:06.100 INFO:teuthology.orchestra.run.smithi035.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 5.5 MB/s | 323 kB 00:00 2022-09-12T18:43:06.134 INFO:teuthology.orchestra.run.smithi035.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.0 MB/s | 71 kB 00:00 2022-09-12T18:43:06.495 INFO:teuthology.orchestra.run.smithi035.stdout:(5/6): libcephsqlite-17.0.0-14883.g10f7d25d.el8 224 kB/s | 137 kB 00:00 2022-09-12T18:43:07.077 INFO:teuthology.orchestra.run.smithi138.stdout:(6/6): ceph-test-17.0.0-14883.g10f7d25d.el8.x86 15 MB/s | 54 MB 00:03 2022-09-12T18:43:07.079 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:43:07.080 INFO:teuthology.orchestra.run.smithi138.stdout:Total 16 MB/s | 55 MB 00:03 2022-09-12T18:43:07.122 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-09-12T18:43:07.143 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-09-12T18:43:07.143 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-09-12T18:43:07.451 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-09-12T18:43:07.454 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-09-12T18:43:07.963 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-09-12T18:43:08.160 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2022-09-12T18:43:08.364 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2022-09-12T18:43:08.490 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2022-09-12T18:43:08.572 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2022-09-12T18:43:08.769 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : jq-1.6-3.el8.x86_64 4/6 2022-09-12T18:43:08.948 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephsqlite-2:17.0.0-14883.g10f7d25d.el8.x86_64 5/6 2022-09-12T18:43:09.974 INFO:teuthology.orchestra.run.smithi035.stdout:(6/6): ceph-test-17.0.0-14883.g10f7d25d.el8.x86 13 MB/s | 54 MB 00:04 2022-09-12T18:43:09.977 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:43:09.977 INFO:teuthology.orchestra.run.smithi035.stdout:Total 13 MB/s | 55 MB 00:04 2022-09-12T18:43:10.020 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2022-09-12T18:43:10.043 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2022-09-12T18:43:10.043 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2022-09-12T18:43:10.376 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2022-09-12T18:43:10.379 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2022-09-12T18:43:10.907 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2022-09-12T18:43:11.088 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2022-09-12T18:43:11.277 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2022-09-12T18:43:11.448 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2022-09-12T18:43:11.512 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2022-09-12T18:43:11.685 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : jq-1.6-3.el8.x86_64 4/6 2022-09-12T18:43:11.881 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : libcephsqlite-2:17.0.0-14883.g10f7d25d.el8.x86_64 5/6 2022-09-12T18:43:14.066 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libcephsqlite-2:17.0.0-14883.g10f7d25d.el8.x86_64 5/6 2022-09-12T18:43:14.279 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-test-2:17.0.0-14883.g10f7d25d.el8.x86_64 6/6 2022-09-12T18:43:14.541 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-test-2:17.0.0-14883.g10f7d25d.el8.x86_64 6/6 2022-09-12T18:43:14.542 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-test-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/6 2022-09-12T18:43:14.542 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephsqlite-2:17.0.0-14883.g10f7d25d.el8.x86_64 2/6 2022-09-12T18:43:14.542 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : jq-1.6-3.el8.x86_64 3/6 2022-09-12T18:43:14.542 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2022-09-12T18:43:14.543 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2022-09-12T18:43:14.968 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2022-09-12T18:43:14.969 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:43:14.970 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-09-12T18:43:14.970 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-test-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:14.971 INFO:teuthology.orchestra.run.smithi138.stdout: jq-1.6-3.el8.x86_64 2022-09-12T18:43:14.971 INFO:teuthology.orchestra.run.smithi138.stdout: libcephsqlite-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:14.971 INFO:teuthology.orchestra.run.smithi138.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-09-12T18:43:14.972 INFO:teuthology.orchestra.run.smithi138.stdout: socat-1.7.4.1-1.el8.x86_64 2022-09-12T18:43:14.972 INFO:teuthology.orchestra.run.smithi138.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2022-09-12T18:43:14.973 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:43:14.973 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:43:15.182 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph 2022-09-12T18:43:15.755 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:00 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:43:16.773 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:43:16.776 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================== 2022-09-12T18:43:16.776 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2022-09-12T18:43:16.777 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================== 2022-09-12T18:43:16.777 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-09-12T18:43:16.777 INFO:teuthology.orchestra.run.smithi138.stdout: ceph x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 6.4 k 2022-09-12T18:43:16.778 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2022-09-12T18:43:16.778 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mds x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 2.3 M 2022-09-12T18:43:16.778 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 1.5 M 2022-09-12T18:43:16.779 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-modules-core noarch 2:17.0.0-14883.g10f7d25d.el8 ceph-noarch 233 k 2022-09-12T18:43:16.779 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mon x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 3.9 M 2022-09-12T18:43:16.779 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-osd x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 17 M 2022-09-12T18:43:16.780 INFO:teuthology.orchestra.run.smithi138.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-09-12T18:43:16.780 INFO:teuthology.orchestra.run.smithi138.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-09-12T18:43:16.781 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-BaseOS 237 k 2022-09-12T18:43:16.781 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-09-12T18:43:16.781 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-09-12T18:43:16.782 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-BaseOS 559 k 2022-09-12T18:43:16.782 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-09-12T18:43:16.783 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-09-12T18:43:16.783 INFO:teuthology.orchestra.run.smithi138.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-09-12T18:43:16.784 INFO:teuthology.orchestra.run.smithi138.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2022-09-12T18:43:16.784 INFO:teuthology.orchestra.run.smithi138.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2022-09-12T18:43:16.784 INFO:teuthology.orchestra.run.smithi138.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-09-12T18:43:16.785 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-09-12T18:43:16.785 INFO:teuthology.orchestra.run.smithi138.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-09-12T18:43:16.785 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2022-09-12T18:43:16.785 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-BaseOS 109 k 2022-09-12T18:43:16.785 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2022-09-12T18:43:16.786 INFO:teuthology.orchestra.run.smithi138.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-09-12T18:43:16.786 INFO:teuthology.orchestra.run.smithi138.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-09-12T18:43:16.786 INFO:teuthology.orchestra.run.smithi138.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-09-12T18:43:16.786 INFO:teuthology.orchestra.run.smithi138.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-09-12T18:43:16.787 INFO:teuthology.orchestra.run.smithi138.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-09-12T18:43:16.787 INFO:teuthology.orchestra.run.smithi138.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-09-12T18:43:16.787 INFO:teuthology.orchestra.run.smithi138.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-09-12T18:43:16.787 INFO:teuthology.orchestra.run.smithi138.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2022-09-12T18:43:16.788 INFO:teuthology.orchestra.run.smithi138.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-09-12T18:43:16.788 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:43:16.788 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-09-12T18:43:16.788 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================== 2022-09-12T18:43:16.788 INFO:teuthology.orchestra.run.smithi138.stdout:Install 32 Packages 2022-09-12T18:43:16.789 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:43:16.790 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 28 M 2022-09-12T18:43:16.790 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 110 M 2022-09-12T18:43:16.790 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-09-12T18:43:16.922 INFO:teuthology.orchestra.run.smithi138.stdout:(1/32): ceph-17.0.0-14883.g10f7d25d.el8.x86_64. 46 kB/s | 6.4 kB 00:00 2022-09-12T18:43:17.168 INFO:teuthology.orchestra.run.smithi138.stdout:(2/32): ceph-mgr-17.0.0-14883.g10f7d25d.el8.x86 4.0 MB/s | 1.5 MB 00:00 2022-09-12T18:43:17.206 INFO:teuthology.orchestra.run.smithi138.stdout:(3/32): ceph-mds-17.0.0-14883.g10f7d25d.el8.x86 5.3 MB/s | 2.3 MB 00:00 2022-09-12T18:43:17.273 INFO:teuthology.orchestra.run.smithi138.stdout:(4/32): ceph-mgr-modules-core-17.0.0-14883.g10f 3.4 MB/s | 233 kB 00:00 2022-09-12T18:43:17.324 INFO:teuthology.orchestra.run.smithi138.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 3.1 MB/s | 157 kB 00:00 2022-09-12T18:43:17.349 INFO:teuthology.orchestra.run.smithi138.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 1.5 MB/s | 39 kB 00:00 2022-09-12T18:43:17.374 INFO:teuthology.orchestra.run.smithi138.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 4.0 MB/s | 103 kB 00:00 2022-09-12T18:43:17.382 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: libcephsqlite-2:17.0.0-14883.g10f7d25d.el8.x86_64 5/6 2022-09-12T18:43:17.399 INFO:teuthology.orchestra.run.smithi138.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 2.1 MB/s | 54 kB 00:00 2022-09-12T18:43:17.450 INFO:teuthology.orchestra.run.smithi138.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 8.9 MB/s | 457 kB 00:00 2022-09-12T18:43:17.475 INFO:teuthology.orchestra.run.smithi138.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 9.2 MB/s | 237 kB 00:00 2022-09-12T18:43:17.509 INFO:teuthology.orchestra.run.smithi138.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 16 MB/s | 559 kB 00:00 2022-09-12T18:43:17.560 INFO:teuthology.orchestra.run.smithi138.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 2.1 MB/s | 109 kB 00:00 2022-09-12T18:43:17.595 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-test-2:17.0.0-14883.g10f7d25d.el8.x86_64 6/6 2022-09-12T18:43:17.619 INFO:teuthology.orchestra.run.smithi138.stdout:(13/32): ceph-mon-17.0.0-14883.g10f7d25d.el8.x8 5.5 MB/s | 3.9 MB 00:00 2022-09-12T18:43:17.727 INFO:teuthology.orchestra.run.smithi138.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 261 kB/s | 44 kB 00:00 2022-09-12T18:43:17.835 INFO:teuthology.orchestra.run.smithi138.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 853 kB/s | 185 kB 00:00 2022-09-12T18:43:17.861 INFO:teuthology.orchestra.run.smithi138.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.3 MB/s | 173 kB 00:00 2022-09-12T18:43:17.880 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-test-2:17.0.0-14883.g10f7d25d.el8.x86_64 6/6 2022-09-12T18:43:17.881 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-test-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/6 2022-09-12T18:43:17.881 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : libcephsqlite-2:17.0.0-14883.g10f7d25d.el8.x86_64 2/6 2022-09-12T18:43:17.881 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : jq-1.6-3.el8.x86_64 3/6 2022-09-12T18:43:17.882 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2022-09-12T18:43:17.882 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2022-09-12T18:43:17.911 INFO:teuthology.orchestra.run.smithi138.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 222 kB/s | 11 kB 00:00 2022-09-12T18:43:18.103 INFO:teuthology.orchestra.run.smithi138.stdout:(18/32): ceph-osd-17.0.0-14883.g10f7d25d.el8.x8 18 MB/s | 17 MB 00:00 2022-09-12T18:43:18.129 INFO:teuthology.orchestra.run.smithi138.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 81 kB/s | 18 kB 00:00 2022-09-12T18:43:18.154 INFO:teuthology.orchestra.run.smithi138.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.2 MB/s | 384 kB 00:00 2022-09-12T18:43:18.238 INFO:teuthology.orchestra.run.smithi138.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2022-09-12T18:43:18.239 INFO:teuthology.orchestra.run.smithi138.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 312 kB/s | 16 kB 00:00 2022-09-12T18:43:18.263 INFO:teuthology.orchestra.run.smithi138.stdout:(23/32): python3-logutils-0.3.5-11.el8.noarch.r 305 kB/s | 49 kB 00:00 2022-09-12T18:43:18.288 INFO:teuthology.orchestra.run.smithi138.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 319 kB/s | 19 kB 00:00 2022-09-12T18:43:18.314 INFO:teuthology.orchestra.run.smithi138.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.7 MB/s | 283 kB 00:00 2022-09-12T18:43:18.358 INFO:teuthology.orchestra.run.smithi138.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 252 kB/s | 24 kB 00:00 2022-09-12T18:43:18.381 INFO:teuthology.orchestra.run.smithi138.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 315 kB/s | 29 kB 00:00 2022-09-12T18:43:18.400 INFO:teuthology.orchestra.run.smithi138.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 328 kB/s | 27 kB 00:00 2022-09-12T18:43:18.443 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2022-09-12T18:43:18.443 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:43:18.444 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2022-09-12T18:43:18.444 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-test-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:18.444 INFO:teuthology.orchestra.run.smithi035.stdout: jq-1.6-3.el8.x86_64 2022-09-12T18:43:18.444 INFO:teuthology.orchestra.run.smithi035.stdout: libcephsqlite-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:18.445 INFO:teuthology.orchestra.run.smithi035.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-09-12T18:43:18.445 INFO:teuthology.orchestra.run.smithi035.stdout: socat-1.7.4.1-1.el8.x86_64 2022-09-12T18:43:18.445 INFO:teuthology.orchestra.run.smithi035.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2022-09-12T18:43:18.445 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:43:18.445 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:43:18.482 INFO:teuthology.orchestra.run.smithi138.stdout:(29/32): python3-waitress-1.2.1-2.el8.1.noarch. 1.9 MB/s | 241 kB 00:00 2022-09-12T18:43:18.508 INFO:teuthology.orchestra.run.smithi138.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 786 kB/s | 85 kB 00:00 2022-09-12T18:43:18.532 INFO:teuthology.orchestra.run.smithi138.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.8 MB/s | 251 kB 00:00 2022-09-12T18:43:18.549 INFO:teuthology.orchestra.run.smithi138.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 341 kB/s | 23 kB 00:00 2022-09-12T18:43:18.552 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:43:18.553 INFO:teuthology.orchestra.run.smithi138.stdout:Total 16 MB/s | 28 MB 00:01 2022-09-12T18:43:18.623 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install ceph 2022-09-12T18:43:18.631 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-09-12T18:43:18.669 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-09-12T18:43:18.670 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-09-12T18:43:18.985 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-09-12T18:43:18.988 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-09-12T18:43:19.227 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:01:04 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:43:20.161 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-09-12T18:43:20.322 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:43:20.325 INFO:teuthology.orchestra.run.smithi035.stdout:======================================================================================== 2022-09-12T18:43:20.325 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repository Size 2022-09-12T18:43:20.326 INFO:teuthology.orchestra.run.smithi035.stdout:======================================================================================== 2022-09-12T18:43:20.326 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2022-09-12T18:43:20.326 INFO:teuthology.orchestra.run.smithi035.stdout: ceph x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 6.4 k 2022-09-12T18:43:20.326 INFO:teuthology.orchestra.run.smithi035.stdout:Installing dependencies: 2022-09-12T18:43:20.327 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mds x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 2.3 M 2022-09-12T18:43:20.327 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mgr x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 1.5 M 2022-09-12T18:43:20.327 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mgr-modules-core noarch 2:17.0.0-14883.g10f7d25d.el8 ceph-noarch 233 k 2022-09-12T18:43:20.327 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mon x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 3.9 M 2022-09-12T18:43:20.328 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-osd x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 17 M 2022-09-12T18:43:20.328 INFO:teuthology.orchestra.run.smithi035.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-09-12T18:43:20.328 INFO:teuthology.orchestra.run.smithi035.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-09-12T18:43:20.328 INFO:teuthology.orchestra.run.smithi035.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-BaseOS 237 k 2022-09-12T18:43:20.329 INFO:teuthology.orchestra.run.smithi035.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-09-12T18:43:20.329 INFO:teuthology.orchestra.run.smithi035.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-09-12T18:43:20.329 INFO:teuthology.orchestra.run.smithi035.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-BaseOS 559 k 2022-09-12T18:43:20.329 INFO:teuthology.orchestra.run.smithi035.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-09-12T18:43:20.330 INFO:teuthology.orchestra.run.smithi035.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-09-12T18:43:20.330 INFO:teuthology.orchestra.run.smithi035.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-09-12T18:43:20.330 INFO:teuthology.orchestra.run.smithi035.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2022-09-12T18:43:20.331 INFO:teuthology.orchestra.run.smithi035.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2022-09-12T18:43:20.331 INFO:teuthology.orchestra.run.smithi035.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-09-12T18:43:20.331 INFO:teuthology.orchestra.run.smithi035.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-09-12T18:43:20.331 INFO:teuthology.orchestra.run.smithi035.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-09-12T18:43:20.332 INFO:teuthology.orchestra.run.smithi035.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2022-09-12T18:43:20.332 INFO:teuthology.orchestra.run.smithi035.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-BaseOS 109 k 2022-09-12T18:43:20.332 INFO:teuthology.orchestra.run.smithi035.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2022-09-12T18:43:20.333 INFO:teuthology.orchestra.run.smithi035.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-09-12T18:43:20.334 INFO:teuthology.orchestra.run.smithi035.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-09-12T18:43:20.334 INFO:teuthology.orchestra.run.smithi035.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-09-12T18:43:20.334 INFO:teuthology.orchestra.run.smithi035.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-09-12T18:43:20.335 INFO:teuthology.orchestra.run.smithi035.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-09-12T18:43:20.335 INFO:teuthology.orchestra.run.smithi035.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-09-12T18:43:20.336 INFO:teuthology.orchestra.run.smithi035.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-09-12T18:43:20.336 INFO:teuthology.orchestra.run.smithi035.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2022-09-12T18:43:20.337 INFO:teuthology.orchestra.run.smithi035.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-09-12T18:43:20.337 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:43:20.338 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2022-09-12T18:43:20.338 INFO:teuthology.orchestra.run.smithi035.stdout:======================================================================================== 2022-09-12T18:43:20.338 INFO:teuthology.orchestra.run.smithi035.stdout:Install 32 Packages 2022-09-12T18:43:20.338 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:43:20.339 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 28 M 2022-09-12T18:43:20.340 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 110 M 2022-09-12T18:43:20.340 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2022-09-12T18:43:20.394 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2022-09-12T18:43:20.541 INFO:teuthology.orchestra.run.smithi035.stdout:(1/32): ceph-17.0.0-14883.g10f7d25d.el8.x86_64. 31 kB/s | 6.4 kB 00:00 2022-09-12T18:43:20.596 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2022-09-12T18:43:20.868 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2022-09-12T18:43:21.086 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2022-09-12T18:43:21.284 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2022-09-12T18:43:21.456 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2022-09-12T18:43:21.691 INFO:teuthology.orchestra.run.smithi035.stdout:(2/32): ceph-mgr-17.0.0-14883.g10f7d25d.el8.x86 1.1 MB/s | 1.5 MB 00:01 2022-09-12T18:43:21.707 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2022-09-12T18:43:21.733 INFO:teuthology.orchestra.run.smithi035.stdout:(3/32): ceph-mds-17.0.0-14883.g10f7d25d.el8.x86 1.6 MB/s | 2.3 MB 00:01 2022-09-12T18:43:21.817 INFO:teuthology.orchestra.run.smithi035.stdout:(4/32): ceph-mgr-modules-core-17.0.0-14883.g10f 2.7 MB/s | 233 kB 00:00 2022-09-12T18:43:21.885 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2022-09-12T18:43:21.909 INFO:teuthology.orchestra.run.smithi035.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 1.7 MB/s | 157 kB 00:00 2022-09-12T18:43:21.934 INFO:teuthology.orchestra.run.smithi035.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 1.5 MB/s | 39 kB 00:00 2022-09-12T18:43:21.959 INFO:teuthology.orchestra.run.smithi035.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 4.0 MB/s | 103 kB 00:00 2022-09-12T18:43:21.985 INFO:teuthology.orchestra.run.smithi035.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 2.1 MB/s | 54 kB 00:00 2022-09-12T18:43:22.043 INFO:teuthology.orchestra.run.smithi035.stdout:(9/32): ceph-mon-17.0.0-14883.g10f7d25d.el8.x86 2.6 MB/s | 3.9 MB 00:01 2022-09-12T18:43:22.077 INFO:teuthology.orchestra.run.smithi035.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 4.9 MB/s | 457 kB 00:00 2022-09-12T18:43:22.109 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2022-09-12T18:43:22.136 INFO:teuthology.orchestra.run.smithi035.stdout:(11/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.5 MB/s | 237 kB 00:00 2022-09-12T18:43:22.169 INFO:teuthology.orchestra.run.smithi035.stdout:(12/32): python3-cryptography-3.2.1-5.el8.x86_6 6.0 MB/s | 559 kB 00:00 2022-09-12T18:43:22.221 INFO:teuthology.orchestra.run.smithi035.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.2 MB/s | 109 kB 00:00 2022-09-12T18:43:22.320 INFO:teuthology.orchestra.run.smithi035.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 290 kB/s | 44 kB 00:00 2022-09-12T18:43:22.337 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2022-09-12T18:43:22.387 INFO:teuthology.orchestra.run.smithi035.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.1 MB/s | 185 kB 00:00 2022-09-12T18:43:22.437 INFO:teuthology.orchestra.run.smithi035.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.4 MB/s | 173 kB 00:00 2022-09-12T18:43:22.471 INFO:teuthology.orchestra.run.smithi035.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 4.5 MB/s | 384 kB 00:00 2022-09-12T18:43:22.517 INFO:teuthology.orchestra.run.smithi035.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 140 kB/s | 11 kB 00:00 2022-09-12T18:43:22.538 INFO:teuthology.orchestra.run.smithi035.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 262 kB/s | 18 kB 00:00 2022-09-12T18:43:22.563 INFO:teuthology.orchestra.run.smithi035.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 1.0 MB/s | 49 kB 00:00 2022-09-12T18:43:22.584 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2022-09-12T18:43:22.589 INFO:teuthology.orchestra.run.smithi035.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2022-09-12T18:43:22.630 INFO:teuthology.orchestra.run.smithi035.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 374 kB/s | 16 kB 00:00 2022-09-12T18:43:22.656 INFO:teuthology.orchestra.run.smithi035.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2022-09-12T18:43:22.681 INFO:teuthology.orchestra.run.smithi035.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 371 kB/s | 19 kB 00:00 2022-09-12T18:43:22.706 INFO:teuthology.orchestra.run.smithi035.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 476 kB/s | 24 kB 00:00 2022-09-12T18:43:22.731 INFO:teuthology.orchestra.run.smithi035.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 584 kB/s | 29 kB 00:00 2022-09-12T18:43:22.757 INFO:teuthology.orchestra.run.smithi035.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 546 kB/s | 27 kB 00:00 2022-09-12T18:43:22.800 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2022-09-12T18:43:22.833 INFO:teuthology.orchestra.run.smithi035.stdout:(28/32): python3-waitress-1.2.1-2.el8.1.noarch. 2.3 MB/s | 241 kB 00:00 2022-09-12T18:43:22.858 INFO:teuthology.orchestra.run.smithi035.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.5 MB/s | 251 kB 00:00 2022-09-12T18:43:22.883 INFO:teuthology.orchestra.run.smithi035.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2022-09-12T18:43:22.908 INFO:teuthology.orchestra.run.smithi035.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 457 kB/s | 23 kB 00:00 2022-09-12T18:43:23.212 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2022-09-12T18:43:23.284 INFO:teuthology.orchestra.run.smithi035.stdout:(32/32): ceph-osd-17.0.0-14883.g10f7d25d.el8.x8 11 MB/s | 17 MB 00:01 2022-09-12T18:43:23.287 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:43:23.287 INFO:teuthology.orchestra.run.smithi035.stdout:Total 9.5 MB/s | 28 MB 00:02 2022-09-12T18:43:23.363 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2022-09-12T18:43:23.402 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2022-09-12T18:43:23.403 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2022-09-12T18:43:23.484 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2022-09-12T18:43:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2022-09-12T18:43:23.733 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2022-09-12T18:43:23.736 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2022-09-12T18:43:23.928 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2022-09-12T18:43:24.163 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-09-12T18:43:24.430 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2022-09-12T18:43:24.643 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2022-09-12T18:43:24.815 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2022-09-12T18:43:25.009 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2022-09-12T18:43:25.017 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2022-09-12T18:43:25.263 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2022-09-12T18:43:25.267 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2022-09-12T18:43:25.438 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2022-09-12T18:43:25.481 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2022-09-12T18:43:25.680 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2022-09-12T18:43:25.722 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2022-09-12T18:43:25.953 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2022-09-12T18:43:25.987 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2022-09-12T18:43:26.126 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2022-09-12T18:43:26.283 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2022-09-12T18:43:26.333 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2022-09-12T18:43:26.508 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2022-09-12T18:43:26.702 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-14883.g10f7d25d.el8 27/32 2022-09-12T18:43:26.720 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2022-09-12T18:43:26.863 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-2:17.0.0-14883.g10f7d25d.el8.x86_64 28/32 2022-09-12T18:43:26.922 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2022-09-12T18:43:27.166 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2022-09-12T18:43:27.405 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2022-09-12T18:43:27.654 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2022-09-12T18:43:28.079 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2022-09-12T18:43:28.342 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2022-09-12T18:43:28.365 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-2:17.0.0-14883.g10f7d25d.el8.x86_64 28/32 2022-09-12T18:43:28.538 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2022-09-12T18:43:28.596 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-osd-2:17.0.0-14883.g10f7d25d.el8.x86_64 29/32 2022-09-12T18:43:28.766 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2022-09-12T18:43:28.986 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-osd-2:17.0.0-14883.g10f7d25d.el8.x86_64 29/32 2022-09-12T18:43:29.022 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-09-12T18:43:29.146 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mon-2:17.0.0-14883.g10f7d25d.el8.x86_64 30/32 2022-09-12T18:43:29.312 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2022-09-12T18:43:29.359 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mon-2:17.0.0-14883.g10f7d25d.el8.x86_64 30/32 2022-09-12T18:43:29.512 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mds-2:17.0.0-14883.g10f7d25d.el8.x86_64 31/32 2022-09-12T18:43:29.527 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2022-09-12T18:43:29.530 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mds-2:17.0.0-14883.g10f7d25d.el8.x86_64 31/32 2022-09-12T18:43:29.644 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-2:17.0.0-14883.g10f7d25d.el8.x86_64 32/32 2022-09-12T18:43:29.732 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2022-09-12T18:43:29.901 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2022-09-12T18:43:30.045 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-2:17.0.0-14883.g10f7d25d.el8.x86_64 32/32 2022-09-12T18:43:30.046 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/32 2022-09-12T18:43:30.046 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mds-2:17.0.0-14883.g10f7d25d.el8.x86_64 2/32 2022-09-12T18:43:30.046 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-2:17.0.0-14883.g10f7d25d.el8.x86_64 3/32 2022-09-12T18:43:30.046 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mon-2:17.0.0-14883.g10f7d25d.el8.x86_64 4/32 2022-09-12T18:43:30.047 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-osd-2:17.0.0-14883.g10f7d25d.el8.x86_64 5/32 2022-09-12T18:43:30.047 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-14883.g10f7d25d.el8 6/32 2022-09-12T18:43:30.047 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2022-09-12T18:43:30.047 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2022-09-12T18:43:30.048 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2022-09-12T18:43:30.048 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2022-09-12T18:43:30.048 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2022-09-12T18:43:30.048 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2022-09-12T18:43:30.048 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2022-09-12T18:43:30.049 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2022-09-12T18:43:30.049 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2022-09-12T18:43:30.049 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2022-09-12T18:43:30.049 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-09-12T18:43:30.050 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2022-09-12T18:43:30.050 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2022-09-12T18:43:30.050 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2022-09-12T18:43:30.050 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2022-09-12T18:43:30.050 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2022-09-12T18:43:30.051 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2022-09-12T18:43:30.051 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2022-09-12T18:43:30.051 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2022-09-12T18:43:30.052 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2022-09-12T18:43:30.052 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2022-09-12T18:43:30.052 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2022-09-12T18:43:30.053 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2022-09-12T18:43:30.053 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2022-09-12T18:43:30.053 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2022-09-12T18:43:30.217 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2022-09-12T18:43:30.452 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2022-09-12T18:43:30.687 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2022-09-12T18:43:31.039 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2022-09-12T18:43:31.138 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2022-09-12T18:43:31.138 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:43:31.138 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-09-12T18:43:31.139 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:31.139 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mds-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:31.139 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:31.140 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-modules-core-2:17.0.0-14883.g10f7d25d.el8.noarch 2022-09-12T18:43:31.140 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mon-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:31.141 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-osd-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:31.141 INFO:teuthology.orchestra.run.smithi138.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-09-12T18:43:31.141 INFO:teuthology.orchestra.run.smithi138.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-09-12T18:43:31.142 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-09-12T18:43:31.142 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-09-12T18:43:31.142 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-09-12T18:43:31.143 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-09-12T18:43:31.143 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-6.2-6.el8.noarch 2022-09-12T18:43:31.143 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-09-12T18:43:31.143 INFO:teuthology.orchestra.run.smithi138.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-09-12T18:43:31.144 INFO:teuthology.orchestra.run.smithi138.stdout: python3-mako-1.0.6-13.el8.noarch 2022-09-12T18:43:31.144 INFO:teuthology.orchestra.run.smithi138.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-09-12T18:43:31.144 INFO:teuthology.orchestra.run.smithi138.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-09-12T18:43:31.144 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-09-12T18:43:31.144 INFO:teuthology.orchestra.run.smithi138.stdout: python3-portend-2.6-1.el8.noarch 2022-09-12T18:43:31.145 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-09-12T18:43:31.145 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pycparser-2.14-14.el8.noarch 2022-09-12T18:43:31.145 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pytz-2017.2-9.el8.noarch 2022-09-12T18:43:31.145 INFO:teuthology.orchestra.run.smithi138.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-09-12T18:43:31.146 INFO:teuthology.orchestra.run.smithi138.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-09-12T18:43:31.146 INFO:teuthology.orchestra.run.smithi138.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-09-12T18:43:31.146 INFO:teuthology.orchestra.run.smithi138.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-09-12T18:43:31.146 INFO:teuthology.orchestra.run.smithi138.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-09-12T18:43:31.147 INFO:teuthology.orchestra.run.smithi138.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-09-12T18:43:31.147 INFO:teuthology.orchestra.run.smithi138.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-09-12T18:43:31.147 INFO:teuthology.orchestra.run.smithi138.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-09-12T18:43:31.147 INFO:teuthology.orchestra.run.smithi138.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-09-12T18:43:31.148 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:43:31.148 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:43:31.363 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2022-09-12T18:43:31.448 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-base 2022-09-12T18:43:31.731 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-14883.g10f7d25d.el8 27/32 2022-09-12T18:43:31.917 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-mgr-2:17.0.0-14883.g10f7d25d.el8.x86_64 28/32 2022-09-12T18:43:32.019 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:17 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:43:32.956 INFO:teuthology.orchestra.run.smithi138.stdout:Package ceph-base-2:17.0.0-14883.g10f7d25d.el8.x86_64 is already installed. 2022-09-12T18:43:33.030 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:43:33.031 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2022-09-12T18:43:33.031 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:43:33.112 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install cephadm 2022-09-12T18:43:33.497 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-mgr-2:17.0.0-14883.g10f7d25d.el8.x86_64 28/32 2022-09-12T18:43:33.681 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:18 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:43:33.725 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-osd-2:17.0.0-14883.g10f7d25d.el8.x86_64 29/32 2022-09-12T18:43:34.139 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-osd-2:17.0.0-14883.g10f7d25d.el8.x86_64 29/32 2022-09-12T18:43:34.312 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-mon-2:17.0.0-14883.g10f7d25d.el8.x86_64 30/32 2022-09-12T18:43:34.538 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-mon-2:17.0.0-14883.g10f7d25d.el8.x86_64 30/32 2022-09-12T18:43:34.679 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-mds-2:17.0.0-14883.g10f7d25d.el8.x86_64 31/32 2022-09-12T18:43:34.684 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:43:34.685 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:43:34.685 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2022-09-12T18:43:34.685 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:43:34.686 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-09-12T18:43:34.687 INFO:teuthology.orchestra.run.smithi138.stdout: cephadm noarch 2:17.0.0-14883.g10f7d25d.el8 ceph-noarch 81 k 2022-09-12T18:43:34.687 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:43:34.688 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-09-12T18:43:34.688 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:43:34.688 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2022-09-12T18:43:34.688 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:43:34.689 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 81 k 2022-09-12T18:43:34.689 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 367 k 2022-09-12T18:43:34.690 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-09-12T18:43:34.697 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-mds-2:17.0.0-14883.g10f7d25d.el8.x86_64 31/32 2022-09-12T18:43:34.836 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-2:17.0.0-14883.g10f7d25d.el8.x86_64 32/32 2022-09-12T18:43:35.049 INFO:teuthology.orchestra.run.smithi138.stdout:cephadm-17.0.0-14883.g10f7d25d.el8.noarch.rpm 223 kB/s | 81 kB 00:00 2022-09-12T18:43:35.051 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:43:35.051 INFO:teuthology.orchestra.run.smithi138.stdout:Total 222 kB/s | 81 kB 00:00 2022-09-12T18:43:35.052 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-09-12T18:43:35.058 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-09-12T18:43:35.058 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-09-12T18:43:35.060 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-09-12T18:43:35.063 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-09-12T18:43:35.115 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-09-12T18:43:35.276 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-2:17.0.0-14883.g10f7d25d.el8.x86_64 32/32 2022-09-12T18:43:35.276 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/32 2022-09-12T18:43:35.276 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-mds-2:17.0.0-14883.g10f7d25d.el8.x86_64 2/32 2022-09-12T18:43:35.277 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-mgr-2:17.0.0-14883.g10f7d25d.el8.x86_64 3/32 2022-09-12T18:43:35.277 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-mon-2:17.0.0-14883.g10f7d25d.el8.x86_64 4/32 2022-09-12T18:43:35.277 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-osd-2:17.0.0-14883.g10f7d25d.el8.x86_64 5/32 2022-09-12T18:43:35.277 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-14883.g10f7d25d.el8 6/32 2022-09-12T18:43:35.278 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2022-09-12T18:43:35.278 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2022-09-12T18:43:35.278 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2022-09-12T18:43:35.278 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2022-09-12T18:43:35.279 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2022-09-12T18:43:35.279 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2022-09-12T18:43:35.279 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2022-09-12T18:43:35.279 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2022-09-12T18:43:35.280 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2022-09-12T18:43:35.280 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2022-09-12T18:43:35.280 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-09-12T18:43:35.281 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2022-09-12T18:43:35.281 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2022-09-12T18:43:35.281 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2022-09-12T18:43:35.281 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2022-09-12T18:43:35.281 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2022-09-12T18:43:35.282 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2022-09-12T18:43:35.282 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2022-09-12T18:43:35.282 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2022-09-12T18:43:35.283 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2022-09-12T18:43:35.284 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2022-09-12T18:43:35.284 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2022-09-12T18:43:35.284 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2022-09-12T18:43:35.284 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2022-09-12T18:43:35.285 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2022-09-12T18:43:36.655 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: cephadm-2:17.0.0-14883.g10f7d25d.el8.noarch 1/1 2022-09-12T18:43:36.857 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : cephadm-2:17.0.0-14883.g10f7d25d.el8.noarch 1/1 2022-09-12T18:43:36.953 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2022-09-12T18:43:36.953 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:43:36.954 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2022-09-12T18:43:36.954 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:36.954 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mds-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:36.954 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mgr-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:36.955 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mgr-modules-core-2:17.0.0-14883.g10f7d25d.el8.noarch 2022-09-12T18:43:36.955 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mon-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:36.955 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-osd-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:36.956 INFO:teuthology.orchestra.run.smithi035.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-09-12T18:43:36.956 INFO:teuthology.orchestra.run.smithi035.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-09-12T18:43:36.956 INFO:teuthology.orchestra.run.smithi035.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-09-12T18:43:36.957 INFO:teuthology.orchestra.run.smithi035.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-09-12T18:43:36.957 INFO:teuthology.orchestra.run.smithi035.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-09-12T18:43:36.957 INFO:teuthology.orchestra.run.smithi035.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-09-12T18:43:36.958 INFO:teuthology.orchestra.run.smithi035.stdout: python3-jaraco-6.2-6.el8.noarch 2022-09-12T18:43:36.958 INFO:teuthology.orchestra.run.smithi035.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-09-12T18:43:36.958 INFO:teuthology.orchestra.run.smithi035.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-09-12T18:43:36.959 INFO:teuthology.orchestra.run.smithi035.stdout: python3-mako-1.0.6-13.el8.noarch 2022-09-12T18:43:36.960 INFO:teuthology.orchestra.run.smithi035.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-09-12T18:43:36.961 INFO:teuthology.orchestra.run.smithi035.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-09-12T18:43:36.961 INFO:teuthology.orchestra.run.smithi035.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-09-12T18:43:36.961 INFO:teuthology.orchestra.run.smithi035.stdout: python3-portend-2.6-1.el8.noarch 2022-09-12T18:43:36.962 INFO:teuthology.orchestra.run.smithi035.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-09-12T18:43:36.962 INFO:teuthology.orchestra.run.smithi035.stdout: python3-pycparser-2.14-14.el8.noarch 2022-09-12T18:43:36.962 INFO:teuthology.orchestra.run.smithi035.stdout: python3-pytz-2017.2-9.el8.noarch 2022-09-12T18:43:36.962 INFO:teuthology.orchestra.run.smithi035.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-09-12T18:43:36.963 INFO:teuthology.orchestra.run.smithi035.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-09-12T18:43:36.963 INFO:teuthology.orchestra.run.smithi035.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-09-12T18:43:36.963 INFO:teuthology.orchestra.run.smithi035.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-09-12T18:43:36.963 INFO:teuthology.orchestra.run.smithi035.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-09-12T18:43:36.964 INFO:teuthology.orchestra.run.smithi035.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-09-12T18:43:36.964 INFO:teuthology.orchestra.run.smithi035.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-09-12T18:43:36.965 INFO:teuthology.orchestra.run.smithi035.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-09-12T18:43:36.965 INFO:teuthology.orchestra.run.smithi035.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-09-12T18:43:36.966 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:43:36.966 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:43:37.046 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: cephadm-2:17.0.0-14883.g10f7d25d.el8.noarch 1/1 2022-09-12T18:43:37.218 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install ceph-base 2022-09-12T18:43:37.435 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : cephadm-2:17.0.0-14883.g10f7d25d.el8.noarch 1/1 2022-09-12T18:43:37.436 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:43:37.436 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-09-12T18:43:37.436 INFO:teuthology.orchestra.run.smithi138.stdout: cephadm-2:17.0.0-14883.g10f7d25d.el8.noarch 2022-09-12T18:43:37.436 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:43:37.437 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:43:37.684 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-immutable-object-cache 2022-09-12T18:43:37.821 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:01:22 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:43:38.261 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:23 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:43:38.818 INFO:teuthology.orchestra.run.smithi035.stdout:Package ceph-base-2:17.0.0-14883.g10f7d25d.el8.x86_64 is already installed. 2022-09-12T18:43:38.901 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:43:38.902 INFO:teuthology.orchestra.run.smithi035.stdout:Nothing to do. 2022-09-12T18:43:38.903 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:43:38.986 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install cephadm 2022-09-12T18:43:39.271 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:43:39.272 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:43:39.273 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2022-09-12T18:43:39.273 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:43:39.273 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-09-12T18:43:39.274 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 165 k 2022-09-12T18:43:39.274 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:43:39.275 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-09-12T18:43:39.275 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:43:39.275 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2022-09-12T18:43:39.276 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:43:39.276 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 165 k 2022-09-12T18:43:39.276 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 503 k 2022-09-12T18:43:39.276 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-09-12T18:43:39.579 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:01:24 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:43:39.799 INFO:teuthology.orchestra.run.smithi138.stdout:ceph-immutable-object-cache-17.0.0-14883.g10f7d 314 kB/s | 165 kB 00:00 2022-09-12T18:43:39.800 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:43:39.800 INFO:teuthology.orchestra.run.smithi138.stdout:Total 313 kB/s | 165 kB 00:00 2022-09-12T18:43:39.800 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-09-12T18:43:39.811 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-09-12T18:43:39.812 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-09-12T18:43:39.860 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-09-12T18:43:39.863 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-09-12T18:43:39.988 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-09-12T18:43:40.152 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-14883.g10f7d25d 1/1 2022-09-12T18:43:40.542 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-14883.g10f7d25d 1/1 2022-09-12T18:43:40.669 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:43:40.671 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:43:40.671 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repository Size 2022-09-12T18:43:40.671 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:43:40.671 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2022-09-12T18:43:40.672 INFO:teuthology.orchestra.run.smithi035.stdout: cephadm noarch 2:17.0.0-14883.g10f7d25d.el8 ceph-noarch 81 k 2022-09-12T18:43:40.672 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:43:40.672 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2022-09-12T18:43:40.672 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:43:40.672 INFO:teuthology.orchestra.run.smithi035.stdout:Install 1 Package 2022-09-12T18:43:40.673 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:43:40.673 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 81 k 2022-09-12T18:43:40.673 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 367 k 2022-09-12T18:43:40.673 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2022-09-12T18:43:40.889 INFO:teuthology.orchestra.run.smithi035.stdout:cephadm-17.0.0-14883.g10f7d25d.el8.noarch.rpm 373 kB/s | 81 kB 00:00 2022-09-12T18:43:40.889 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:43:40.890 INFO:teuthology.orchestra.run.smithi035.stdout:Total 371 kB/s | 81 kB 00:00 2022-09-12T18:43:40.890 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2022-09-12T18:43:40.898 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2022-09-12T18:43:40.899 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2022-09-12T18:43:40.900 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2022-09-12T18:43:40.903 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2022-09-12T18:43:40.966 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-14883.g10f7d25d 1/1 2022-09-12T18:43:40.966 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:43:40.967 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-09-12T18:43:40.967 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-immutable-object-cache-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:40.967 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:43:40.967 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:43:41.050 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2022-09-12T18:43:41.134 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr 2022-09-12T18:43:41.713 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:26 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:43:42.647 INFO:teuthology.orchestra.run.smithi138.stdout:Package ceph-mgr-2:17.0.0-14883.g10f7d25d.el8.x86_64 is already installed. 2022-09-12T18:43:42.651 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: cephadm-2:17.0.0-14883.g10f7d25d.el8.noarch 1/1 2022-09-12T18:43:42.729 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:43:42.729 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2022-09-12T18:43:42.730 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:43:42.815 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-dashboard 2022-09-12T18:43:42.845 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : cephadm-2:17.0.0-14883.g10f7d25d.el8.noarch 1/1 2022-09-12T18:43:43.040 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: cephadm-2:17.0.0-14883.g10f7d25d.el8.noarch 1/1 2022-09-12T18:43:43.383 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:28 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:43:43.478 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : cephadm-2:17.0.0-14883.g10f7d25d.el8.noarch 1/1 2022-09-12T18:43:43.479 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:43:43.479 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2022-09-12T18:43:43.479 INFO:teuthology.orchestra.run.smithi035.stdout: cephadm-2:17.0.0-14883.g10f7d25d.el8.noarch 2022-09-12T18:43:43.479 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:43:43.480 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:43:43.770 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install ceph-immutable-object-cache 2022-09-12T18:43:44.370 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:01:29 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:43:44.387 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:43:44.388 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:43:44.389 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2022-09-12T18:43:44.389 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:43:44.389 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-09-12T18:43:44.390 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-dashboard noarch 2:17.0.0-14883.g10f7d25d.el8 ceph-noarch 1.6 M 2022-09-12T18:43:44.390 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2022-09-12T18:43:44.391 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-grafana-dashboards 2022-09-12T18:43:44.391 INFO:teuthology.orchestra.run.smithi138.stdout: noarch 2:17.0.0-14883.g10f7d25d.el8 ceph-noarch 23 k 2022-09-12T18:43:44.391 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-prometheus-alerts noarch 2:17.0.0-14883.g10f7d25d.el8 ceph-noarch 14 k 2022-09-12T18:43:44.392 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-BaseOS 43 k 2022-09-12T18:43:44.392 INFO:teuthology.orchestra.run.smithi138.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-09-12T18:43:44.393 INFO:teuthology.orchestra.run.smithi138.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-09-12T18:43:44.393 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:43:44.394 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-09-12T18:43:44.394 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:43:44.394 INFO:teuthology.orchestra.run.smithi138.stdout:Install 6 Packages 2022-09-12T18:43:44.395 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:43:44.396 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 1.9 M 2022-09-12T18:43:44.396 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 7.8 M 2022-09-12T18:43:44.396 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-09-12T18:43:44.553 INFO:teuthology.orchestra.run.smithi138.stdout:(1/6): ceph-grafana-dashboards-17.0.0-14883.g10 140 kB/s | 23 kB 00:00 2022-09-12T18:43:44.579 INFO:teuthology.orchestra.run.smithi138.stdout:(2/6): ceph-prometheus-alerts-17.0.0-14883.g10f 77 kB/s | 14 kB 00:00 2022-09-12T18:43:44.638 INFO:teuthology.orchestra.run.smithi138.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 508 kB/s | 43 kB 00:00 2022-09-12T18:43:44.696 INFO:teuthology.orchestra.run.smithi138.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 287 kB/s | 34 kB 00:00 2022-09-12T18:43:44.747 INFO:teuthology.orchestra.run.smithi138.stdout:(5/6): ceph-mgr-dashboard-17.0.0-14883.g10f7d25 4.6 MB/s | 1.6 MB 00:00 2022-09-12T18:43:44.855 INFO:teuthology.orchestra.run.smithi138.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 905 kB/s | 196 kB 00:00 2022-09-12T18:43:44.857 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:43:44.857 INFO:teuthology.orchestra.run.smithi138.stdout:Total 4.2 MB/s | 1.9 MB 00:00 2022-09-12T18:43:44.867 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-09-12T18:43:44.876 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-09-12T18:43:44.876 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-09-12T18:43:44.953 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-09-12T18:43:44.956 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-09-12T18:43:45.457 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:43:45.458 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:43:45.458 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repo Size 2022-09-12T18:43:45.459 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:43:45.459 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2022-09-12T18:43:45.459 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 165 k 2022-09-12T18:43:45.460 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:43:45.460 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2022-09-12T18:43:45.460 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:43:45.461 INFO:teuthology.orchestra.run.smithi035.stdout:Install 1 Package 2022-09-12T18:43:45.461 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:43:45.461 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 165 k 2022-09-12T18:43:45.461 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 503 k 2022-09-12T18:43:45.461 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2022-09-12T18:43:45.700 INFO:teuthology.orchestra.run.smithi035.stdout:ceph-immutable-object-cache-17.0.0-14883.g10f7d 681 kB/s | 165 kB 00:00 2022-09-12T18:43:45.701 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:43:45.702 INFO:teuthology.orchestra.run.smithi035.stdout:Total 678 kB/s | 165 kB 00:00 2022-09-12T18:43:45.702 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2022-09-12T18:43:45.713 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2022-09-12T18:43:45.714 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2022-09-12T18:43:45.765 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2022-09-12T18:43:45.768 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2022-09-12T18:43:45.901 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2022-09-12T18:43:46.077 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-14883.g10f7d25d 1/1 2022-09-12T18:43:46.474 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-14883.g10f7d25d 1/1 2022-09-12T18:43:47.057 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-09-12T18:43:47.290 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2022-09-12T18:43:47.484 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2022-09-12T18:43:47.605 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-14883.g10f7d25d 1/1 2022-09-12T18:43:47.605 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:43:47.605 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2022-09-12T18:43:47.606 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-immutable-object-cache-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:43:47.606 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:43:47.606 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:43:47.679 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2022-09-12T18:43:47.807 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-14883.g10f7d25d.el8. 4/6 2022-09-12T18:43:48.118 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-14883.g10f7d25d.el8 5/6 2022-09-12T18:43:48.326 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-14883.g10f7d25d.el8.noar 6/6 2022-09-12T18:43:48.601 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-14883.g10f7d25d.el8.noar 6/6 2022-09-12T18:43:48.601 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-14883.g10f7d25d.el8 1/6 2022-09-12T18:43:48.601 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-14883.g10f7d25d.el8.noar 2/6 2022-09-12T18:43:48.602 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-14883.g10f7d25d.el8. 3/6 2022-09-12T18:43:48.602 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2022-09-12T18:43:48.602 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2022-09-12T18:43:48.985 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install ceph-mgr 2022-09-12T18:43:49.014 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2022-09-12T18:43:49.015 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:43:49.015 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-09-12T18:43:49.015 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-grafana-dashboards-2:17.0.0-14883.g10f7d25d.el8.noarch 2022-09-12T18:43:49.015 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-dashboard-2:17.0.0-14883.g10f7d25d.el8.noarch 2022-09-12T18:43:49.016 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-prometheus-alerts-2:17.0.0-14883.g10f7d25d.el8.noarch 2022-09-12T18:43:49.016 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-09-12T18:43:49.016 INFO:teuthology.orchestra.run.smithi138.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-09-12T18:43:49.016 INFO:teuthology.orchestra.run.smithi138.stdout: python3-routes-2.4.1-12.el8.noarch 2022-09-12T18:43:49.016 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:43:49.017 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:43:49.114 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-diskprediction-local 2022-09-12T18:43:49.585 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:01:34 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:43:49.680 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:34 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:43:50.583 INFO:teuthology.orchestra.run.smithi035.stdout:Package ceph-mgr-2:17.0.0-14883.g10f7d25d.el8.x86_64 is already installed. 2022-09-12T18:43:50.684 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:43:50.685 INFO:teuthology.orchestra.run.smithi138.stdout:===================================================================================================== 2022-09-12T18:43:50.685 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2022-09-12T18:43:50.686 INFO:teuthology.orchestra.run.smithi138.stdout:===================================================================================================== 2022-09-12T18:43:50.686 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-09-12T18:43:50.686 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-14883.g10f7d25d.el8 ceph-noarch 7.4 M 2022-09-12T18:43:50.686 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2022-09-12T18:43:50.687 INFO:teuthology.orchestra.run.smithi138.stdout: libgfortran x86_64 8.5.0-15.el8 CentOS-BaseOS 644 k 2022-09-12T18:43:50.687 INFO:teuthology.orchestra.run.smithi138.stdout: libquadmath x86_64 8.5.0-15.el8 CentOS-BaseOS 172 k 2022-09-12T18:43:50.687 INFO:teuthology.orchestra.run.smithi138.stdout: openblas x86_64 0.3.15-4.el8 CentOS-AppStream 4.7 M 2022-09-12T18:43:50.687 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-threads x86_64 0.3.15-4.el8 CentOS-AppStream 4.9 M 2022-09-12T18:43:50.688 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2022-09-12T18:43:50.688 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2022-09-12T18:43:50.688 INFO:teuthology.orchestra.run.smithi138.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2022-09-12T18:43:50.688 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:43:50.689 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-09-12T18:43:50.689 INFO:teuthology.orchestra.run.smithi138.stdout:===================================================================================================== 2022-09-12T18:43:50.689 INFO:teuthology.orchestra.run.smithi138.stdout:Install 8 Packages 2022-09-12T18:43:50.689 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:43:50.690 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:43:50.691 INFO:teuthology.orchestra.run.smithi035.stdout:Nothing to do. 2022-09-12T18:43:50.691 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:43:50.691 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 35 M 2022-09-12T18:43:50.691 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 210 M 2022-09-12T18:43:50.692 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-09-12T18:43:50.768 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install ceph-mgr-dashboard 2022-09-12T18:43:50.888 INFO:teuthology.orchestra.run.smithi138.stdout:(1/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 24 MB/s | 4.9 MB 00:00 2022-09-12T18:43:51.013 INFO:teuthology.orchestra.run.smithi138.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 30 MB/s | 3.7 MB 00:00 2022-09-12T18:43:51.080 INFO:teuthology.orchestra.run.smithi138.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2022-09-12T18:43:51.106 INFO:teuthology.orchestra.run.smithi138.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2022-09-12T18:43:51.148 INFO:teuthology.orchestra.run.smithi138.stdout:(5/8): libgfortran-8.5.0-15.el8.x86_64.rpm 15 MB/s | 644 kB 00:00 2022-09-12T18:43:51.181 INFO:teuthology.orchestra.run.smithi138.stdout:(6/8): libquadmath-8.5.0-15.el8.x86_64.rpm 5.0 MB/s | 172 kB 00:00 2022-09-12T18:43:51.367 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:01:36 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:43:51.394 INFO:teuthology.orchestra.run.smithi138.stdout:(7/8): ceph-mgr-diskprediction-local-17.0.0-148 11 MB/s | 7.4 MB 00:00 2022-09-12T18:43:51.582 INFO:teuthology.orchestra.run.smithi138.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 27 MB/s | 14 MB 00:00 2022-09-12T18:43:51.585 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:43:51.585 INFO:teuthology.orchestra.run.smithi138.stdout:Total 39 MB/s | 35 MB 00:00 2022-09-12T18:43:51.815 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-09-12T18:43:51.833 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-09-12T18:43:51.834 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-09-12T18:43:52.204 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-09-12T18:43:52.207 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-09-12T18:43:52.445 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:43:52.446 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:43:52.447 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repository Size 2022-09-12T18:43:52.447 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:43:52.447 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2022-09-12T18:43:52.447 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mgr-dashboard noarch 2:17.0.0-14883.g10f7d25d.el8 ceph-noarch 1.6 M 2022-09-12T18:43:52.448 INFO:teuthology.orchestra.run.smithi035.stdout:Installing dependencies: 2022-09-12T18:43:52.448 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-grafana-dashboards 2022-09-12T18:43:52.448 INFO:teuthology.orchestra.run.smithi035.stdout: noarch 2:17.0.0-14883.g10f7d25d.el8 ceph-noarch 23 k 2022-09-12T18:43:52.448 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-prometheus-alerts noarch 2:17.0.0-14883.g10f7d25d.el8 ceph-noarch 14 k 2022-09-12T18:43:52.449 INFO:teuthology.orchestra.run.smithi035.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-BaseOS 43 k 2022-09-12T18:43:52.449 INFO:teuthology.orchestra.run.smithi035.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-09-12T18:43:52.449 INFO:teuthology.orchestra.run.smithi035.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-09-12T18:43:52.449 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:43:52.450 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2022-09-12T18:43:52.450 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:43:52.450 INFO:teuthology.orchestra.run.smithi035.stdout:Install 6 Packages 2022-09-12T18:43:52.450 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:43:52.451 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 1.9 M 2022-09-12T18:43:52.451 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 7.8 M 2022-09-12T18:43:52.452 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2022-09-12T18:43:52.605 INFO:teuthology.orchestra.run.smithi035.stdout:(1/6): ceph-prometheus-alerts-17.0.0-14883.g10f 93 kB/s | 14 kB 00:00 2022-09-12T18:43:52.622 INFO:teuthology.orchestra.run.smithi035.stdout:(2/6): ceph-grafana-dashboards-17.0.0-14883.g10 132 kB/s | 23 kB 00:00 2022-09-12T18:43:52.674 INFO:teuthology.orchestra.run.smithi035.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 636 kB/s | 43 kB 00:00 2022-09-12T18:43:52.723 INFO:teuthology.orchestra.run.smithi035.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 336 kB/s | 34 kB 00:00 2022-09-12T18:43:52.781 INFO:teuthology.orchestra.run.smithi035.stdout:(5/6): ceph-mgr-dashboard-17.0.0-14883.g10f7d25 5.0 MB/s | 1.6 MB 00:00 2022-09-12T18:43:52.831 INFO:teuthology.orchestra.run.smithi035.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.2 MB/s | 196 kB 00:00 2022-09-12T18:43:52.833 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:43:52.834 INFO:teuthology.orchestra.run.smithi035.stdout:Total 5.1 MB/s | 1.9 MB 00:00 2022-09-12T18:43:52.844 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2022-09-12T18:43:52.845 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-09-12T18:43:52.853 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2022-09-12T18:43:52.853 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2022-09-12T18:43:52.933 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2022-09-12T18:43:52.937 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2022-09-12T18:43:52.975 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libquadmath-8.5.0-15.el8.x86_64 1/8 2022-09-12T18:43:53.102 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libquadmath-8.5.0-15.el8.x86_64 1/8 2022-09-12T18:43:53.274 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2022-09-12T18:43:53.274 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libgfortran-8.5.0-15.el8.x86_64 2/8 2022-09-12T18:43:53.510 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2022-09-12T18:43:53.728 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2022-09-12T18:43:53.876 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libgfortran-8.5.0-15.el8.x86_64 2/8 2022-09-12T18:43:53.903 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2022-09-12T18:43:54.066 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-14883.g10f7d25d.el8. 4/6 2022-09-12T18:43:54.067 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2022-09-12T18:43:54.382 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-14883.g10f7d25d.el8 5/6 2022-09-12T18:43:54.630 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-14883.g10f7d25d.el8.noar 6/6 2022-09-12T18:43:54.696 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2022-09-12T18:43:54.871 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-14883.g10f7d25d.el8.noar 6/6 2022-09-12T18:43:54.872 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-14883.g10f7d25d.el8 1/6 2022-09-12T18:43:54.872 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-14883.g10f7d25d.el8.noar 2/6 2022-09-12T18:43:54.872 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-14883.g10f7d25d.el8. 3/6 2022-09-12T18:43:54.873 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2022-09-12T18:43:54.873 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2022-09-12T18:43:54.885 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2022-09-12T18:43:55.312 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2022-09-12T18:43:55.312 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:43:55.313 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2022-09-12T18:43:55.313 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-grafana-dashboards-2:17.0.0-14883.g10f7d25d.el8.noarch 2022-09-12T18:43:55.314 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mgr-dashboard-2:17.0.0-14883.g10f7d25d.el8.noarch 2022-09-12T18:43:55.314 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-prometheus-alerts-2:17.0.0-14883.g10f7d25d.el8.noarch 2022-09-12T18:43:55.315 INFO:teuthology.orchestra.run.smithi035.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-09-12T18:43:55.315 INFO:teuthology.orchestra.run.smithi035.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-09-12T18:43:55.316 INFO:teuthology.orchestra.run.smithi035.stdout: python3-routes-2.4.1-12.el8.noarch 2022-09-12T18:43:55.316 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:43:55.316 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:43:55.413 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install ceph-mgr-diskprediction-local 2022-09-12T18:43:55.588 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2022-09-12T18:43:55.911 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2022-09-12T18:43:56.016 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:01:41 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:43:56.157 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-09-12T18:43:57.095 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:43:57.096 INFO:teuthology.orchestra.run.smithi035.stdout:===================================================================================================== 2022-09-12T18:43:57.096 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repository Size 2022-09-12T18:43:57.097 INFO:teuthology.orchestra.run.smithi035.stdout:===================================================================================================== 2022-09-12T18:43:57.097 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2022-09-12T18:43:57.097 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-14883.g10f7d25d.el8 ceph-noarch 7.4 M 2022-09-12T18:43:57.097 INFO:teuthology.orchestra.run.smithi035.stdout:Installing dependencies: 2022-09-12T18:43:57.098 INFO:teuthology.orchestra.run.smithi035.stdout: libgfortran x86_64 8.5.0-15.el8 CentOS-BaseOS 644 k 2022-09-12T18:43:57.098 INFO:teuthology.orchestra.run.smithi035.stdout: libquadmath x86_64 8.5.0-15.el8 CentOS-BaseOS 172 k 2022-09-12T18:43:57.098 INFO:teuthology.orchestra.run.smithi035.stdout: openblas x86_64 0.3.15-4.el8 CentOS-AppStream 4.7 M 2022-09-12T18:43:57.098 INFO:teuthology.orchestra.run.smithi035.stdout: openblas-threads x86_64 0.3.15-4.el8 CentOS-AppStream 4.9 M 2022-09-12T18:43:57.099 INFO:teuthology.orchestra.run.smithi035.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2022-09-12T18:43:57.099 INFO:teuthology.orchestra.run.smithi035.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2022-09-12T18:43:57.099 INFO:teuthology.orchestra.run.smithi035.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2022-09-12T18:43:57.099 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:43:57.100 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2022-09-12T18:43:57.100 INFO:teuthology.orchestra.run.smithi035.stdout:===================================================================================================== 2022-09-12T18:43:57.101 INFO:teuthology.orchestra.run.smithi035.stdout:Install 8 Packages 2022-09-12T18:43:57.101 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:43:57.103 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 35 M 2022-09-12T18:43:57.103 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 210 M 2022-09-12T18:43:57.103 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2022-09-12T18:43:57.280 INFO:teuthology.orchestra.run.smithi035.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2022-09-12T18:43:57.397 INFO:teuthology.orchestra.run.smithi035.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 32 MB/s | 3.7 MB 00:00 2022-09-12T18:43:57.431 INFO:teuthology.orchestra.run.smithi035.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 6.6 MB/s | 225 kB 00:00 2022-09-12T18:43:57.540 INFO:teuthology.orchestra.run.smithi035.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 11 MB/s | 4.9 MB 00:00 2022-09-12T18:43:57.590 INFO:teuthology.orchestra.run.smithi035.stdout:(5/8): libgfortran-8.5.0-15.el8.x86_64.rpm 13 MB/s | 644 kB 00:00 2022-09-12T18:43:57.624 INFO:teuthology.orchestra.run.smithi035.stdout:(6/8): libquadmath-8.5.0-15.el8.x86_64.rpm 5.1 MB/s | 172 kB 00:00 2022-09-12T18:43:57.774 INFO:teuthology.orchestra.run.smithi035.stdout:(7/8): ceph-mgr-diskprediction-local-17.0.0-148 11 MB/s | 7.4 MB 00:00 2022-09-12T18:43:57.917 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-09-12T18:43:57.932 INFO:teuthology.orchestra.run.smithi035.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 27 MB/s | 14 MB 00:00 2022-09-12T18:43:57.934 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:43:57.935 INFO:teuthology.orchestra.run.smithi035.stdout:Total 42 MB/s | 35 MB 00:00 2022-09-12T18:43:58.171 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2022-09-12T18:43:58.190 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2022-09-12T18:43:58.190 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2022-09-12T18:43:58.561 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2022-09-12T18:43:58.565 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2022-09-12T18:43:59.153 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2022-09-12T18:43:59.156 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2022-09-12T18:43:59.349 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-14883.g10f7d2 8/8 2022-09-12T18:43:59.626 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-14883.g10f7d2 8/8 2022-09-12T18:43:59.627 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-14883.g10f7d2 1/8 2022-09-12T18:43:59.627 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2022-09-12T18:43:59.627 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2022-09-12T18:43:59.628 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2022-09-12T18:43:59.628 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2022-09-12T18:43:59.628 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2022-09-12T18:43:59.628 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libgfortran-8.5.0-15.el8.x86_64 7/8 2022-09-12T18:43:59.698 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : libquadmath-8.5.0-15.el8.x86_64 1/8 2022-09-12T18:44:00.056 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: libquadmath-8.5.0-15.el8.x86_64 1/8 2022-09-12T18:44:00.086 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libquadmath-8.5.0-15.el8.x86_64 8/8 2022-09-12T18:44:00.086 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:00.087 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-09-12T18:44:00.087 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-diskprediction-local-2:17.0.0-14883.g10f7d25d.el8.noarch 2022-09-12T18:44:00.087 INFO:teuthology.orchestra.run.smithi138.stdout: libgfortran-8.5.0-15.el8.x86_64 2022-09-12T18:44:00.088 INFO:teuthology.orchestra.run.smithi138.stdout: libquadmath-8.5.0-15.el8.x86_64 2022-09-12T18:44:00.088 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-0.3.15-4.el8.x86_64 2022-09-12T18:44:00.088 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-threads-0.3.15-4.el8.x86_64 2022-09-12T18:44:00.088 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-09-12T18:44:00.089 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-09-12T18:44:00.089 INFO:teuthology.orchestra.run.smithi138.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2022-09-12T18:44:00.089 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:00.089 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:44:00.183 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : libgfortran-8.5.0-15.el8.x86_64 2/8 2022-09-12T18:44:00.210 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-rook 2022-09-12T18:44:00.786 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:45 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:00.853 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: libgfortran-8.5.0-15.el8.x86_64 2/8 2022-09-12T18:44:01.009 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2022-09-12T18:44:01.656 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2022-09-12T18:44:01.788 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:44:01.790 INFO:teuthology.orchestra.run.smithi138.stdout:========================================================================================= 2022-09-12T18:44:01.791 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2022-09-12T18:44:01.791 INFO:teuthology.orchestra.run.smithi138.stdout:========================================================================================= 2022-09-12T18:44:01.791 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-09-12T18:44:01.791 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-rook noarch 2:17.0.0-14883.g10f7d25d.el8 ceph-noarch 47 k 2022-09-12T18:44:01.792 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2022-09-12T18:44:01.792 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-09-12T18:44:01.792 INFO:teuthology.orchestra.run.smithi138.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-09-12T18:44:01.793 INFO:teuthology.orchestra.run.smithi138.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-09-12T18:44:01.793 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2022-09-12T18:44:01.793 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2022-09-12T18:44:01.793 INFO:teuthology.orchestra.run.smithi138.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-09-12T18:44:01.793 INFO:teuthology.orchestra.run.smithi138.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-BaseOS 155 k 2022-09-12T18:44:01.794 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2022-09-12T18:44:01.794 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2022-09-12T18:44:01.794 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-BaseOS 43 k 2022-09-12T18:44:01.795 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-09-12T18:44:01.795 INFO:teuthology.orchestra.run.smithi138.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-09-12T18:44:01.795 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:01.795 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-09-12T18:44:01.795 INFO:teuthology.orchestra.run.smithi138.stdout:========================================================================================= 2022-09-12T18:44:01.796 INFO:teuthology.orchestra.run.smithi138.stdout:Install 13 Packages 2022-09-12T18:44:01.796 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:01.797 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 1.9 M 2022-09-12T18:44:01.797 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 25 M 2022-09-12T18:44:01.797 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-09-12T18:44:01.833 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2022-09-12T18:44:01.863 INFO:teuthology.orchestra.run.smithi138.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 381 kB/s | 27 kB 00:00 2022-09-12T18:44:01.888 INFO:teuthology.orchestra.run.smithi138.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 5.1 MB/s | 126 kB 00:00 2022-09-12T18:44:01.913 INFO:teuthology.orchestra.run.smithi138.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 166 kB/s | 20 kB 00:00 2022-09-12T18:44:01.938 INFO:teuthology.orchestra.run.smithi138.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.1 MB/s | 110 kB 00:00 2022-09-12T18:44:01.965 INFO:teuthology.orchestra.run.smithi138.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.7 MB/s | 43 kB 00:00 2022-09-12T18:44:01.989 INFO:teuthology.orchestra.run.smithi138.stdout:(6/13): ceph-mgr-rook-17.0.0-14883.g10f7d25d.el 239 kB/s | 47 kB 00:00 2022-09-12T18:44:02.015 INFO:teuthology.orchestra.run.smithi138.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.5 MB/s | 155 kB 00:00 2022-09-12T18:44:02.107 INFO:teuthology.orchestra.run.smithi138.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 231 kB/s | 33 kB 00:00 2022-09-12T18:44:02.124 INFO:teuthology.orchestra.run.smithi138.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 121 kB/s | 16 kB 00:00 2022-09-12T18:44:02.174 INFO:teuthology.orchestra.run.smithi138.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 511 kB/s | 81 kB 00:00 2022-09-12T18:44:02.208 INFO:teuthology.orchestra.run.smithi138.stdout:(11/13): python3-rsa-4.8-1.el8.noarch.rpm 733 kB/s | 61 kB 00:00 2022-09-12T18:44:02.225 INFO:teuthology.orchestra.run.smithi138.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.2 MB/s | 61 kB 00:00 2022-09-12T18:44:02.350 INFO:teuthology.orchestra.run.smithi138.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 4.8 MB/s | 1.2 MB 00:00 2022-09-12T18:44:02.351 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:44:02.351 INFO:teuthology.orchestra.run.smithi138.stdout:Total 3.4 MB/s | 1.9 MB 00:00 2022-09-12T18:44:02.403 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-09-12T18:44:02.418 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-09-12T18:44:02.419 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-09-12T18:44:02.509 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-09-12T18:44:02.512 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-09-12T18:44:02.568 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2022-09-12T18:44:02.891 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2022-09-12T18:44:03.006 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-09-12T18:44:03.141 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-09-12T18:44:03.705 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2022-09-12T18:44:03.894 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2022-09-12T18:44:04.102 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2022-09-12T18:44:04.264 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2022-09-12T18:44:04.463 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2022-09-12T18:44:04.663 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2022-09-12T18:44:04.947 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2022-09-12T18:44:05.002 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-09-12T18:44:05.187 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2022-09-12T18:44:06.212 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2022-09-12T18:44:06.277 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2022-09-12T18:44:06.446 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2022-09-12T18:44:06.458 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-14883.g10f7d2 8/8 2022-09-12T18:44:06.588 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2022-09-12T18:44:06.743 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-14883.g10f7d2 8/8 2022-09-12T18:44:06.743 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-14883.g10f7d2 1/8 2022-09-12T18:44:06.743 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2022-09-12T18:44:06.744 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2022-09-12T18:44:06.744 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2022-09-12T18:44:06.744 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2022-09-12T18:44:06.744 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2022-09-12T18:44:06.744 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : libgfortran-8.5.0-15.el8.x86_64 7/8 2022-09-12T18:44:06.790 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2022-09-12T18:44:06.914 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-rook-2:17.0.0-14883.g10f7d25d.el8.noarch 13/13 2022-09-12T18:44:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-14883.g10f7d25d.el8.noarch 13/13 2022-09-12T18:44:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-rook-2:17.0.0-14883.g10f7d25d.el8.noarch 1/13 2022-09-12T18:44:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2022-09-12T18:44:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2022-09-12T18:44:07.167 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2022-09-12T18:44:07.167 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2022-09-12T18:44:07.168 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2022-09-12T18:44:07.168 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2022-09-12T18:44:07.169 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2022-09-12T18:44:07.169 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2022-09-12T18:44:07.170 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2022-09-12T18:44:07.171 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2022-09-12T18:44:07.171 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2022-09-12T18:44:07.191 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : libquadmath-8.5.0-15.el8.x86_64 8/8 2022-09-12T18:44:07.192 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:07.192 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2022-09-12T18:44:07.192 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mgr-diskprediction-local-2:17.0.0-14883.g10f7d25d.el8.noarch 2022-09-12T18:44:07.193 INFO:teuthology.orchestra.run.smithi035.stdout: libgfortran-8.5.0-15.el8.x86_64 2022-09-12T18:44:07.193 INFO:teuthology.orchestra.run.smithi035.stdout: libquadmath-8.5.0-15.el8.x86_64 2022-09-12T18:44:07.194 INFO:teuthology.orchestra.run.smithi035.stdout: openblas-0.3.15-4.el8.x86_64 2022-09-12T18:44:07.194 INFO:teuthology.orchestra.run.smithi035.stdout: openblas-threads-0.3.15-4.el8.x86_64 2022-09-12T18:44:07.194 INFO:teuthology.orchestra.run.smithi035.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-09-12T18:44:07.194 INFO:teuthology.orchestra.run.smithi035.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-09-12T18:44:07.195 INFO:teuthology.orchestra.run.smithi035.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2022-09-12T18:44:07.195 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:07.195 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:44:07.293 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install ceph-mgr-rook 2022-09-12T18:44:07.631 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2022-09-12T18:44:07.632 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:07.632 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-09-12T18:44:07.632 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-rook-2:17.0.0-14883.g10f7d25d.el8.noarch 2022-09-12T18:44:07.633 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-09-12T18:44:07.633 INFO:teuthology.orchestra.run.smithi138.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-09-12T18:44:07.633 INFO:teuthology.orchestra.run.smithi138.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-09-12T18:44:07.633 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-09-12T18:44:07.634 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-09-12T18:44:07.634 INFO:teuthology.orchestra.run.smithi138.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-09-12T18:44:07.634 INFO:teuthology.orchestra.run.smithi138.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-09-12T18:44:07.634 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-09-12T18:44:07.635 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-09-12T18:44:07.635 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-09-12T18:44:07.635 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rsa-4.8-1.el8.noarch 2022-09-12T18:44:07.635 INFO:teuthology.orchestra.run.smithi138.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-09-12T18:44:07.636 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:07.636 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:44:07.754 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-cephadm 2022-09-12T18:44:07.884 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:01:52 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:08.326 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:53 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:08.969 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:44:08.970 INFO:teuthology.orchestra.run.smithi035.stdout:========================================================================================= 2022-09-12T18:44:08.971 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repository Size 2022-09-12T18:44:08.971 INFO:teuthology.orchestra.run.smithi035.stdout:========================================================================================= 2022-09-12T18:44:08.971 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2022-09-12T18:44:08.972 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mgr-rook noarch 2:17.0.0-14883.g10f7d25d.el8 ceph-noarch 47 k 2022-09-12T18:44:08.972 INFO:teuthology.orchestra.run.smithi035.stdout:Installing dependencies: 2022-09-12T18:44:08.972 INFO:teuthology.orchestra.run.smithi035.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-09-12T18:44:08.973 INFO:teuthology.orchestra.run.smithi035.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-09-12T18:44:08.973 INFO:teuthology.orchestra.run.smithi035.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-09-12T18:44:08.973 INFO:teuthology.orchestra.run.smithi035.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2022-09-12T18:44:08.974 INFO:teuthology.orchestra.run.smithi035.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2022-09-12T18:44:08.974 INFO:teuthology.orchestra.run.smithi035.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-09-12T18:44:08.975 INFO:teuthology.orchestra.run.smithi035.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-BaseOS 155 k 2022-09-12T18:44:08.975 INFO:teuthology.orchestra.run.smithi035.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2022-09-12T18:44:08.975 INFO:teuthology.orchestra.run.smithi035.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2022-09-12T18:44:08.976 INFO:teuthology.orchestra.run.smithi035.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-BaseOS 43 k 2022-09-12T18:44:08.976 INFO:teuthology.orchestra.run.smithi035.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-09-12T18:44:08.976 INFO:teuthology.orchestra.run.smithi035.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-09-12T18:44:08.977 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:08.977 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2022-09-12T18:44:08.977 INFO:teuthology.orchestra.run.smithi035.stdout:========================================================================================= 2022-09-12T18:44:08.978 INFO:teuthology.orchestra.run.smithi035.stdout:Install 13 Packages 2022-09-12T18:44:08.978 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:08.979 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 1.9 M 2022-09-12T18:44:08.980 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 25 M 2022-09-12T18:44:08.980 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2022-09-12T18:44:09.034 INFO:teuthology.orchestra.run.smithi035.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 444 kB/s | 27 kB 00:00 2022-09-12T18:44:09.068 INFO:teuthology.orchestra.run.smithi035.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 4.4 MB/s | 126 kB 00:00 2022-09-12T18:44:09.093 INFO:teuthology.orchestra.run.smithi035.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 168 kB/s | 20 kB 00:00 2022-09-12T18:44:09.118 INFO:teuthology.orchestra.run.smithi035.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.1 MB/s | 110 kB 00:00 2022-09-12T18:44:09.145 INFO:teuthology.orchestra.run.smithi035.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.7 MB/s | 43 kB 00:00 2022-09-12T18:44:09.169 INFO:teuthology.orchestra.run.smithi035.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 2.0 MB/s | 155 kB 00:00 2022-09-12T18:44:09.294 INFO:teuthology.orchestra.run.smithi035.stdout:(7/13): ceph-mgr-rook-17.0.0-14883.g10f7d25d.el 146 kB/s | 47 kB 00:00 2022-09-12T18:44:09.320 INFO:teuthology.orchestra.run.smithi035.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 108 kB/s | 16 kB 00:00 2022-09-12T18:44:09.342 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:44:09.343 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:44:09.343 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2022-09-12T18:44:09.343 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:44:09.344 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-09-12T18:44:09.344 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-cephadm noarch 2:17.0.0-14883.g10f7d25d.el8 ceph-noarch 115 k 2022-09-12T18:44:09.344 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2022-09-12T18:44:09.344 INFO:teuthology.orchestra.run.smithi138.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-09-12T18:44:09.345 INFO:teuthology.orchestra.run.smithi138.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2022-09-12T18:44:09.345 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2022-09-12T18:44:09.345 INFO:teuthology.orchestra.run.smithi138.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-09-12T18:44:09.345 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:09.345 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-09-12T18:44:09.346 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:44:09.346 INFO:teuthology.orchestra.run.smithi138.stdout:Install 5 Packages 2022-09-12T18:44:09.346 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:09.347 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 5.9 M 2022-09-12T18:44:09.347 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 26 M 2022-09-12T18:44:09.348 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-09-12T18:44:09.353 INFO:teuthology.orchestra.run.smithi035.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 157 kB/s | 33 kB 00:00 2022-09-12T18:44:09.396 INFO:teuthology.orchestra.run.smithi035.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 807 kB/s | 81 kB 00:00 2022-09-12T18:44:09.435 INFO:teuthology.orchestra.run.smithi138.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 5.9 MB/s | 538 kB 00:00 2022-09-12T18:44:09.446 INFO:teuthology.orchestra.run.smithi035.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.2 MB/s | 61 kB 00:00 2022-09-12T18:44:09.496 INFO:teuthology.orchestra.run.smithi035.stdout:(12/13): python3-rsa-4.8-1.el8.noarch.rpm 434 kB/s | 61 kB 00:00 2022-09-12T18:44:09.560 INFO:teuthology.orchestra.run.smithi138.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 22 MB/s | 4.8 MB 00:00 2022-09-12T18:44:09.572 INFO:teuthology.orchestra.run.smithi035.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 4.7 MB/s | 1.2 MB 00:00 2022-09-12T18:44:09.573 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:44:09.573 INFO:teuthology.orchestra.run.smithi035.stdout:Total 3.2 MB/s | 1.9 MB 00:00 2022-09-12T18:44:09.627 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2022-09-12T18:44:09.649 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2022-09-12T18:44:09.649 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2022-09-12T18:44:09.661 INFO:teuthology.orchestra.run.smithi138.stdout:(3/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 2.1 MB/s | 476 kB 00:00 2022-09-12T18:44:09.678 INFO:teuthology.orchestra.run.smithi138.stdout:(4/5): ceph-mgr-cephadm-17.0.0-14883.g10f7d25d. 346 kB/s | 115 kB 00:00 2022-09-12T18:44:09.703 INFO:teuthology.orchestra.run.smithi138.stdout:(5/5): python3-natsort-7.1.1-2.el8.noarch.rpm 424 kB/s | 60 kB 00:00 2022-09-12T18:44:09.704 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:44:09.705 INFO:teuthology.orchestra.run.smithi138.stdout:Total 16 MB/s | 5.9 MB 00:00 2022-09-12T18:44:09.727 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2022-09-12T18:44:09.730 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2022-09-12T18:44:09.756 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-09-12T18:44:09.767 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-09-12T18:44:09.767 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-09-12T18:44:09.857 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-09-12T18:44:09.860 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-09-12T18:44:10.142 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-09-12T18:44:10.252 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2022-09-12T18:44:10.426 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2022-09-12T18:44:10.446 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2022-09-12T18:44:10.643 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2022-09-12T18:44:10.844 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2022-09-12T18:44:11.016 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2022-09-12T18:44:11.190 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2022-09-12T18:44:11.324 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2022-09-12T18:44:11.382 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2022-09-12T18:44:11.617 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2022-09-12T18:44:11.649 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2022-09-12T18:44:11.847 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2022-09-12T18:44:11.874 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2022-09-12T18:44:12.072 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-14883.g10f7d25d.el8.noarch 5/5 2022-09-12T18:44:12.320 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-14883.g10f7d25d.el8.noarch 5/5 2022-09-12T18:44:12.320 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-14883.g10f7d25d.el8.noarch 1/5 2022-09-12T18:44:12.321 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2022-09-12T18:44:12.321 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2022-09-12T18:44:12.321 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2022-09-12T18:44:12.720 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2022-09-12T18:44:12.721 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:12.721 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-09-12T18:44:12.721 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-cephadm-2:17.0.0-14883.g10f7d25d.el8.noarch 2022-09-12T18:44:12.721 INFO:teuthology.orchestra.run.smithi138.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-09-12T18:44:12.722 INFO:teuthology.orchestra.run.smithi138.stdout: python3-babel-2.5.1-7.el8.noarch 2022-09-12T18:44:12.722 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-09-12T18:44:12.722 INFO:teuthology.orchestra.run.smithi138.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-09-12T18:44:12.723 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:12.723 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:44:12.825 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-fuse 2022-09-12T18:44:12.972 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2022-09-12T18:44:13.205 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2022-09-12T18:44:13.347 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2022-09-12T18:44:13.392 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:58 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:13.551 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2022-09-12T18:44:13.683 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-mgr-rook-2:17.0.0-14883.g10f7d25d.el8.noarch 13/13 2022-09-12T18:44:13.974 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-14883.g10f7d25d.el8.noarch 13/13 2022-09-12T18:44:13.975 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-mgr-rook-2:17.0.0-14883.g10f7d25d.el8.noarch 1/13 2022-09-12T18:44:13.975 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2022-09-12T18:44:13.975 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2022-09-12T18:44:13.976 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2022-09-12T18:44:13.976 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2022-09-12T18:44:13.977 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2022-09-12T18:44:13.977 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2022-09-12T18:44:13.978 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2022-09-12T18:44:13.978 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2022-09-12T18:44:13.979 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2022-09-12T18:44:13.979 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2022-09-12T18:44:13.980 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2022-09-12T18:44:14.418 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:44:14.420 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:44:14.420 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2022-09-12T18:44:14.421 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:44:14.421 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-09-12T18:44:14.422 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-fuse x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 844 k 2022-09-12T18:44:14.422 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2022-09-12T18:44:14.423 INFO:teuthology.orchestra.run.smithi138.stdout: fuse x86_64 2.9.7-16.el8 CentOS-BaseOS 83 k 2022-09-12T18:44:14.423 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:14.424 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-09-12T18:44:14.424 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:44:14.425 INFO:teuthology.orchestra.run.smithi138.stdout:Install 2 Packages 2022-09-12T18:44:14.425 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:14.426 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 927 k 2022-09-12T18:44:14.426 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 2.9 M 2022-09-12T18:44:14.427 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-09-12T18:44:14.471 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2022-09-12T18:44:14.471 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:14.472 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2022-09-12T18:44:14.472 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mgr-rook-2:17.0.0-14883.g10f7d25d.el8.noarch 2022-09-12T18:44:14.473 INFO:teuthology.orchestra.run.smithi035.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-09-12T18:44:14.473 INFO:teuthology.orchestra.run.smithi035.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-09-12T18:44:14.473 INFO:teuthology.orchestra.run.smithi035.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-09-12T18:44:14.474 INFO:teuthology.orchestra.run.smithi035.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-09-12T18:44:14.474 INFO:teuthology.orchestra.run.smithi035.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-09-12T18:44:14.475 INFO:teuthology.orchestra.run.smithi035.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-09-12T18:44:14.475 INFO:teuthology.orchestra.run.smithi035.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-09-12T18:44:14.475 INFO:teuthology.orchestra.run.smithi035.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-09-12T18:44:14.476 INFO:teuthology.orchestra.run.smithi035.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-09-12T18:44:14.476 INFO:teuthology.orchestra.run.smithi035.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-09-12T18:44:14.476 INFO:teuthology.orchestra.run.smithi035.stdout: python3-rsa-4.8-1.el8.noarch 2022-09-12T18:44:14.477 INFO:teuthology.orchestra.run.smithi035.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-09-12T18:44:14.552 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:14.552 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:44:14.553 INFO:teuthology.orchestra.run.smithi138.stdout:(1/2): fuse-2.9.7-16.el8.x86_64.rpm 1.1 MB/s | 83 kB 00:00 2022-09-12T18:44:14.579 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install ceph-mgr-cephadm 2022-09-12T18:44:15.191 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:02:00 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:15.195 INFO:teuthology.orchestra.run.smithi138.stdout:(2/2): ceph-fuse-17.0.0-14883.g10f7d25d.el8.x86 1.1 MB/s | 844 kB 00:00 2022-09-12T18:44:15.197 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:44:15.197 INFO:teuthology.orchestra.run.smithi138.stdout:Total 1.2 MB/s | 927 kB 00:00 2022-09-12T18:44:15.207 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-09-12T18:44:15.229 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-09-12T18:44:15.229 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-09-12T18:44:15.302 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-09-12T18:44:15.305 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-09-12T18:44:15.466 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-09-12T18:44:15.709 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : fuse-2.9.7-16.el8.x86_64 1/2 2022-09-12T18:44:15.862 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-fuse-2:17.0.0-14883.g10f7d25d.el8.x86_64 2/2 2022-09-12T18:44:16.218 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-fuse-2:17.0.0-14883.g10f7d25d.el8.x86_64 2/2 2022-09-12T18:44:16.219 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-fuse-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/2 2022-09-12T18:44:16.287 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:44:16.288 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:44:16.288 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repository Size 2022-09-12T18:44:16.288 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:44:16.288 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2022-09-12T18:44:16.289 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mgr-cephadm noarch 2:17.0.0-14883.g10f7d25d.el8 ceph-noarch 115 k 2022-09-12T18:44:16.289 INFO:teuthology.orchestra.run.smithi035.stdout:Installing dependencies: 2022-09-12T18:44:16.289 INFO:teuthology.orchestra.run.smithi035.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-09-12T18:44:16.289 INFO:teuthology.orchestra.run.smithi035.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2022-09-12T18:44:16.290 INFO:teuthology.orchestra.run.smithi035.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2022-09-12T18:44:16.290 INFO:teuthology.orchestra.run.smithi035.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-09-12T18:44:16.291 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:16.292 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2022-09-12T18:44:16.292 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:44:16.293 INFO:teuthology.orchestra.run.smithi035.stdout:Install 5 Packages 2022-09-12T18:44:16.293 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:16.295 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 5.9 M 2022-09-12T18:44:16.295 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 26 M 2022-09-12T18:44:16.295 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2022-09-12T18:44:16.470 INFO:teuthology.orchestra.run.smithi035.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.9 MB/s | 538 kB 00:00 2022-09-12T18:44:16.579 INFO:teuthology.orchestra.run.smithi035.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 17 MB/s | 4.8 MB 00:00 2022-09-12T18:44:16.621 INFO:teuthology.orchestra.run.smithi035.stdout:(3/5): ceph-mgr-cephadm-17.0.0-14883.g10f7d25d. 348 kB/s | 115 kB 00:00 2022-09-12T18:44:16.685 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : fuse-2.9.7-16.el8.x86_64 2/2 2022-09-12T18:44:16.686 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:16.686 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-09-12T18:44:16.686 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-fuse-2:17.0.0-14883.g10f7d25d.el8.x86_64 fuse-2.9.7-16.el8.x86_64 2022-09-12T18:44:16.687 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:16.687 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:44:16.705 INFO:teuthology.orchestra.run.smithi035.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 2.0 MB/s | 476 kB 00:00 2022-09-12T18:44:16.730 INFO:teuthology.orchestra.run.smithi035.stdout:(5/5): python3-natsort-7.1.1-2.el8.noarch.rpm 400 kB/s | 60 kB 00:00 2022-09-12T18:44:16.731 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:44:16.732 INFO:teuthology.orchestra.run.smithi035.stdout:Total 13 MB/s | 5.9 MB 00:00 2022-09-12T18:44:16.785 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2022-09-12T18:44:16.797 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2022-09-12T18:44:16.797 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2022-09-12T18:44:16.894 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-volume 2022-09-12T18:44:16.895 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2022-09-12T18:44:16.897 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2022-09-12T18:44:17.151 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2022-09-12T18:44:17.429 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2022-09-12T18:44:17.480 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:02 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:18.372 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2022-09-12T18:44:18.483 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:44:18.484 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:44:18.484 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2022-09-12T18:44:18.484 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:44:18.485 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-09-12T18:44:18.485 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-volume noarch 2:17.0.0-14883.g10f7d25d.el8 ceph-noarch 263 k 2022-09-12T18:44:18.485 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:18.485 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-09-12T18:44:18.486 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:44:18.486 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2022-09-12T18:44:18.486 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:18.486 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 263 k 2022-09-12T18:44:18.487 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 1.3 M 2022-09-12T18:44:18.487 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-09-12T18:44:18.727 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2022-09-12T18:44:18.794 INFO:teuthology.orchestra.run.smithi138.stdout:ceph-volume-17.0.0-14883.g10f7d25d.el8.noarch.r 851 kB/s | 263 kB 00:00 2022-09-12T18:44:18.795 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:44:18.795 INFO:teuthology.orchestra.run.smithi138.stdout:Total 848 kB/s | 263 kB 00:00 2022-09-12T18:44:18.795 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-09-12T18:44:18.804 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-09-12T18:44:18.805 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-09-12T18:44:18.828 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-09-12T18:44:18.829 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-09-12T18:44:18.969 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2022-09-12T18:44:18.997 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-09-12T18:44:19.124 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-14883.g10f7d25d.el8.noarch 5/5 2022-09-12T18:44:19.213 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-volume-2:17.0.0-14883.g10f7d25d.el8.noarch 1/1 2022-09-12T18:44:19.404 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-14883.g10f7d25d.el8.noarch 5/5 2022-09-12T18:44:19.405 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-14883.g10f7d25d.el8.noarch 1/5 2022-09-12T18:44:19.405 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2022-09-12T18:44:19.406 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2022-09-12T18:44:19.406 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2022-09-12T18:44:19.684 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-volume-2:17.0.0-14883.g10f7d25d.el8.noarch 1/1 2022-09-12T18:44:19.846 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2022-09-12T18:44:19.847 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:19.847 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2022-09-12T18:44:19.847 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-mgr-cephadm-2:17.0.0-14883.g10f7d25d.el8.noarch 2022-09-12T18:44:19.848 INFO:teuthology.orchestra.run.smithi035.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-09-12T18:44:19.848 INFO:teuthology.orchestra.run.smithi035.stdout: python3-babel-2.5.1-7.el8.noarch 2022-09-12T18:44:19.848 INFO:teuthology.orchestra.run.smithi035.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-09-12T18:44:19.848 INFO:teuthology.orchestra.run.smithi035.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-09-12T18:44:19.849 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:19.849 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:44:19.956 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install ceph-fuse 2022-09-12T18:44:20.119 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-volume-2:17.0.0-14883.g10f7d25d.el8.noarch 1/1 2022-09-12T18:44:20.120 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:20.120 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-09-12T18:44:20.121 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-volume-2:17.0.0-14883.g10f7d25d.el8.noarch 2022-09-12T18:44:20.121 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:20.121 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:44:20.287 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install librados-devel 2022-09-12T18:44:20.557 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:02:05 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:20.869 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:05 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:21.644 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:44:21.646 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:44:21.646 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repository Size 2022-09-12T18:44:21.646 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:44:21.647 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2022-09-12T18:44:21.647 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-fuse x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 844 k 2022-09-12T18:44:21.647 INFO:teuthology.orchestra.run.smithi035.stdout:Installing dependencies: 2022-09-12T18:44:21.647 INFO:teuthology.orchestra.run.smithi035.stdout: fuse x86_64 2.9.7-16.el8 CentOS-BaseOS 83 k 2022-09-12T18:44:21.648 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:21.648 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2022-09-12T18:44:21.648 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:44:21.648 INFO:teuthology.orchestra.run.smithi035.stdout:Install 2 Packages 2022-09-12T18:44:21.649 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:21.649 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 927 k 2022-09-12T18:44:21.649 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 2.9 M 2022-09-12T18:44:21.649 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2022-09-12T18:44:21.722 INFO:teuthology.orchestra.run.smithi035.stdout:(1/2): fuse-2.9.7-16.el8.x86_64.rpm 1.1 MB/s | 83 kB 00:00 2022-09-12T18:44:21.878 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:44:21.880 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:44:21.880 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2022-09-12T18:44:21.880 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:44:21.881 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-09-12T18:44:21.881 INFO:teuthology.orchestra.run.smithi138.stdout: librados-devel x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 139 k 2022-09-12T18:44:21.881 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:21.881 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-09-12T18:44:21.882 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:44:21.882 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2022-09-12T18:44:21.883 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:21.883 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 139 k 2022-09-12T18:44:21.884 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 512 k 2022-09-12T18:44:21.884 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-09-12T18:44:22.083 INFO:teuthology.orchestra.run.smithi138.stdout:librados-devel-17.0.0-14883.g10f7d25d.el8.x86_6 682 kB/s | 139 kB 00:00 2022-09-12T18:44:22.085 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:44:22.085 INFO:teuthology.orchestra.run.smithi138.stdout:Total 678 kB/s | 139 kB 00:00 2022-09-12T18:44:22.085 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-09-12T18:44:22.089 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-09-12T18:44:22.090 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-09-12T18:44:22.141 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-09-12T18:44:22.144 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-09-12T18:44:22.247 INFO:teuthology.orchestra.run.smithi035.stdout:(2/2): ceph-fuse-17.0.0-14883.g10f7d25d.el8.x86 1.4 MB/s | 844 kB 00:00 2022-09-12T18:44:22.249 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:44:22.249 INFO:teuthology.orchestra.run.smithi035.stdout:Total 1.5 MB/s | 927 kB 00:00 2022-09-12T18:44:22.256 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-09-12T18:44:22.259 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2022-09-12T18:44:22.282 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2022-09-12T18:44:22.282 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2022-09-12T18:44:22.362 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2022-09-12T18:44:22.365 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2022-09-12T18:44:22.468 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librados-devel-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:22.561 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2022-09-12T18:44:22.742 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librados-devel-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:22.842 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : fuse-2.9.7-16.el8.x86_64 1/2 2022-09-12T18:44:22.963 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-fuse-2:17.0.0-14883.g10f7d25d.el8.x86_64 2/2 2022-09-12T18:44:23.144 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados-devel-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:23.144 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:23.144 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-09-12T18:44:23.145 INFO:teuthology.orchestra.run.smithi138.stdout: librados-devel-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:44:23.145 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:23.145 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:44:23.354 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-fuse-2:17.0.0-14883.g10f7d25d.el8.x86_64 2/2 2022-09-12T18:44:23.355 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-fuse-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/2 2022-09-12T18:44:23.357 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install libcephfs2 2022-09-12T18:44:23.803 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : fuse-2.9.7-16.el8.x86_64 2/2 2022-09-12T18:44:23.803 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:23.804 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2022-09-12T18:44:23.804 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-fuse-2:17.0.0-14883.g10f7d25d.el8.x86_64 fuse-2.9.7-16.el8.x86_64 2022-09-12T18:44:23.804 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:23.804 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:44:23.942 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:08 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:24.049 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install ceph-volume 2022-09-12T18:44:24.658 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:02:09 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:24.875 INFO:teuthology.orchestra.run.smithi138.stdout:Package libcephfs2-2:17.0.0-14883.g10f7d25d.el8.x86_64 is already installed. 2022-09-12T18:44:24.950 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:44:24.951 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2022-09-12T18:44:24.951 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:44:25.153 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install libcephfs-devel 2022-09-12T18:44:25.721 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:10 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:25.750 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:44:25.751 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:44:25.751 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repository Size 2022-09-12T18:44:25.751 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:44:25.752 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2022-09-12T18:44:25.752 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-volume noarch 2:17.0.0-14883.g10f7d25d.el8 ceph-noarch 263 k 2022-09-12T18:44:25.752 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:25.752 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2022-09-12T18:44:25.753 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:44:25.753 INFO:teuthology.orchestra.run.smithi035.stdout:Install 1 Package 2022-09-12T18:44:25.754 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:25.754 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 263 k 2022-09-12T18:44:25.754 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 1.3 M 2022-09-12T18:44:25.755 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2022-09-12T18:44:26.034 INFO:teuthology.orchestra.run.smithi035.stdout:ceph-volume-17.0.0-14883.g10f7d25d.el8.noarch.r 930 kB/s | 263 kB 00:00 2022-09-12T18:44:26.035 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:44:26.035 INFO:teuthology.orchestra.run.smithi035.stdout:Total 926 kB/s | 263 kB 00:00 2022-09-12T18:44:26.036 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2022-09-12T18:44:26.047 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2022-09-12T18:44:26.047 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2022-09-12T18:44:26.073 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2022-09-12T18:44:26.076 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2022-09-12T18:44:26.250 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2022-09-12T18:44:26.498 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : ceph-volume-2:17.0.0-14883.g10f7d25d.el8.noarch 1/1 2022-09-12T18:44:26.723 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:44:26.724 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:44:26.724 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2022-09-12T18:44:26.725 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:44:26.725 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-09-12T18:44:26.726 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs-devel x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 30 k 2022-09-12T18:44:26.726 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:26.726 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-09-12T18:44:26.727 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:44:26.727 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2022-09-12T18:44:26.728 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:26.728 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 30 k 2022-09-12T18:44:26.728 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 137 k 2022-09-12T18:44:26.729 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-09-12T18:44:26.881 INFO:teuthology.orchestra.run.smithi138.stdout:libcephfs-devel-17.0.0-14883.g10f7d25d.el8.x86_ 195 kB/s | 30 kB 00:00 2022-09-12T18:44:26.882 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:44:26.883 INFO:teuthology.orchestra.run.smithi138.stdout:Total 193 kB/s | 30 kB 00:00 2022-09-12T18:44:26.883 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-09-12T18:44:26.885 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-09-12T18:44:26.886 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-09-12T18:44:26.914 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-09-12T18:44:26.917 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-09-12T18:44:26.921 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: ceph-volume-2:17.0.0-14883.g10f7d25d.el8.noarch 1/1 2022-09-12T18:44:26.974 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-09-12T18:44:27.142 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephfs-devel-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:27.368 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:27.440 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : ceph-volume-2:17.0.0-14883.g10f7d25d.el8.noarch 1/1 2022-09-12T18:44:27.441 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:27.441 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2022-09-12T18:44:27.441 INFO:teuthology.orchestra.run.smithi035.stdout: ceph-volume-2:17.0.0-14883.g10f7d25d.el8.noarch 2022-09-12T18:44:27.442 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:27.442 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:44:27.657 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install librados-devel 2022-09-12T18:44:27.727 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephfs-devel-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:27.728 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:27.728 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-09-12T18:44:27.729 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs-devel-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:44:27.729 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:27.730 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:44:27.839 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install librados2 2022-09-12T18:44:28.262 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:02:13 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:28.412 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:13 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:29.347 INFO:teuthology.orchestra.run.smithi138.stdout:Package librados2-2:17.0.0-14883.g10f7d25d.el8.x86_64 is already installed. 2022-09-12T18:44:29.351 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:44:29.352 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:44:29.352 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repo Size 2022-09-12T18:44:29.352 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:44:29.352 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2022-09-12T18:44:29.353 INFO:teuthology.orchestra.run.smithi035.stdout: librados-devel x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 139 k 2022-09-12T18:44:29.353 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:29.353 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2022-09-12T18:44:29.353 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:44:29.354 INFO:teuthology.orchestra.run.smithi035.stdout:Install 1 Package 2022-09-12T18:44:29.354 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:29.354 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 139 k 2022-09-12T18:44:29.354 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 512 k 2022-09-12T18:44:29.355 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2022-09-12T18:44:29.422 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:44:29.423 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2022-09-12T18:44:29.423 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:44:29.507 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install librbd1 2022-09-12T18:44:29.589 INFO:teuthology.orchestra.run.smithi035.stdout:librados-devel-17.0.0-14883.g10f7d25d.el8.x86_6 587 kB/s | 139 kB 00:00 2022-09-12T18:44:29.590 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:44:29.590 INFO:teuthology.orchestra.run.smithi035.stdout:Total 584 kB/s | 139 kB 00:00 2022-09-12T18:44:29.591 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2022-09-12T18:44:29.596 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2022-09-12T18:44:29.596 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2022-09-12T18:44:29.655 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2022-09-12T18:44:29.658 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2022-09-12T18:44:29.774 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2022-09-12T18:44:29.987 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : librados-devel-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:30.080 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:15 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:30.251 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: librados-devel-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:30.691 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : librados-devel-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:30.692 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:30.692 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2022-09-12T18:44:30.692 INFO:teuthology.orchestra.run.smithi035.stdout: librados-devel-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:44:30.693 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:30.693 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:44:30.931 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install libcephfs2 2022-09-12T18:44:31.011 INFO:teuthology.orchestra.run.smithi138.stdout:Package librbd1-2:17.0.0-14883.g10f7d25d.el8.x86_64 is already installed. 2022-09-12T18:44:31.085 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:44:31.086 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2022-09-12T18:44:31.087 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:44:31.361 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-rados 2022-09-12T18:44:31.539 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:02:16 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:31.930 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:16 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:32.527 INFO:teuthology.orchestra.run.smithi035.stdout:Package libcephfs2-2:17.0.0-14883.g10f7d25d.el8.x86_64 is already installed. 2022-09-12T18:44:32.610 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:44:32.611 INFO:teuthology.orchestra.run.smithi035.stdout:Nothing to do. 2022-09-12T18:44:32.611 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:44:32.858 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-rados-2:17.0.0-14883.g10f7d25d.el8.x86_64 is already installed. 2022-09-12T18:44:32.932 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:44:32.933 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2022-09-12T18:44:32.933 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:44:33.011 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-rgw 2022-09-12T18:44:33.099 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install libcephfs-devel 2022-09-12T18:44:33.578 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:18 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:33.702 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:02:18 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:34.526 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-rgw-2:17.0.0-14883.g10f7d25d.el8.x86_64 is already installed. 2022-09-12T18:44:34.600 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:44:34.601 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2022-09-12T18:44:34.601 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:44:34.693 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-cephfs 2022-09-12T18:44:34.792 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:44:34.793 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:44:34.794 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repo Size 2022-09-12T18:44:34.794 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:44:34.794 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2022-09-12T18:44:34.795 INFO:teuthology.orchestra.run.smithi035.stdout: libcephfs-devel x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 30 k 2022-09-12T18:44:34.795 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:34.796 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2022-09-12T18:44:34.796 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:44:34.796 INFO:teuthology.orchestra.run.smithi035.stdout:Install 1 Package 2022-09-12T18:44:34.797 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:34.797 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 30 k 2022-09-12T18:44:34.797 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 137 k 2022-09-12T18:44:34.798 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2022-09-12T18:44:34.978 INFO:teuthology.orchestra.run.smithi035.stdout:libcephfs-devel-17.0.0-14883.g10f7d25d.el8.x86_ 165 kB/s | 30 kB 00:00 2022-09-12T18:44:34.979 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:44:34.979 INFO:teuthology.orchestra.run.smithi035.stdout:Total 164 kB/s | 30 kB 00:00 2022-09-12T18:44:34.980 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2022-09-12T18:44:34.983 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2022-09-12T18:44:34.983 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2022-09-12T18:44:35.021 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2022-09-12T18:44:35.024 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2022-09-12T18:44:35.119 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2022-09-12T18:44:35.252 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : libcephfs-devel-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:35.262 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:20 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:35.490 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:35.874 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : libcephfs-devel-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:35.875 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:35.875 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2022-09-12T18:44:35.875 INFO:teuthology.orchestra.run.smithi035.stdout: libcephfs-devel-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:44:35.875 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:35.876 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:44:35.973 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install librados2 2022-09-12T18:44:36.196 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-cephfs-2:17.0.0-14883.g10f7d25d.el8.x86_64 is already installed. 2022-09-12T18:44:36.270 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:44:36.271 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2022-09-12T18:44:36.272 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:44:36.485 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-rbd 2022-09-12T18:44:36.578 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:02:21 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:37.058 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:22 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:37.594 INFO:teuthology.orchestra.run.smithi035.stdout:Package librados2-2:17.0.0-14883.g10f7d25d.el8.x86_64 is already installed. 2022-09-12T18:44:37.677 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:44:37.678 INFO:teuthology.orchestra.run.smithi035.stdout:Nothing to do. 2022-09-12T18:44:37.678 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:44:37.757 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install librbd1 2022-09-12T18:44:37.995 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-rbd-2:17.0.0-14883.g10f7d25d.el8.x86_64 is already installed. 2022-09-12T18:44:38.070 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:44:38.071 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2022-09-12T18:44:38.071 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:44:38.164 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install rbd-fuse 2022-09-12T18:44:38.360 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:02:23 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:38.730 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:23 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:39.369 INFO:teuthology.orchestra.run.smithi035.stdout:Package librbd1-2:17.0.0-14883.g10f7d25d.el8.x86_64 is already installed. 2022-09-12T18:44:39.454 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:44:39.455 INFO:teuthology.orchestra.run.smithi035.stdout:Nothing to do. 2022-09-12T18:44:39.455 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:44:39.532 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install python3-rados 2022-09-12T18:44:39.740 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:44:39.741 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:44:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2022-09-12T18:44:39.741 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:44:39.742 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-09-12T18:44:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-fuse x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 94 k 2022-09-12T18:44:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:39.742 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-09-12T18:44:39.743 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:44:39.743 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2022-09-12T18:44:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:39.743 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 94 k 2022-09-12T18:44:39.743 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 280 k 2022-09-12T18:44:39.744 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-09-12T18:44:39.958 INFO:teuthology.orchestra.run.smithi138.stdout:rbd-fuse-17.0.0-14883.g10f7d25d.el8.x86_64.rpm 435 kB/s | 94 kB 00:00 2022-09-12T18:44:39.959 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:44:39.959 INFO:teuthology.orchestra.run.smithi138.stdout:Total 433 kB/s | 94 kB 00:00 2022-09-12T18:44:39.959 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-09-12T18:44:39.969 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-09-12T18:44:39.969 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-09-12T18:44:40.018 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-09-12T18:44:40.021 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-09-12T18:44:40.123 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-09-12T18:44:40.129 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:02:25 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:40.307 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-fuse-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:40.557 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: rbd-fuse-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:40.948 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-fuse-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:40.949 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:40.949 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-09-12T18:44:40.949 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-fuse-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:44:40.950 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:40.950 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:44:41.131 INFO:teuthology.orchestra.run.smithi035.stdout:Package python3-rados-2:17.0.0-14883.g10f7d25d.el8.x86_64 is already installed. 2022-09-12T18:44:41.136 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install rbd-mirror 2022-09-12T18:44:41.215 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:44:41.216 INFO:teuthology.orchestra.run.smithi035.stdout:Nothing to do. 2022-09-12T18:44:41.217 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:44:41.293 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install python3-rgw 2022-09-12T18:44:41.716 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:26 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:41.890 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:02:26 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:42.731 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:44:42.733 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:44:42.733 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2022-09-12T18:44:42.733 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:44:42.734 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-09-12T18:44:42.734 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-mirror x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 3.5 M 2022-09-12T18:44:42.734 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:42.734 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-09-12T18:44:42.735 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:44:42.735 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2022-09-12T18:44:42.735 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:42.735 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 3.5 M 2022-09-12T18:44:42.735 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 16 M 2022-09-12T18:44:42.736 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-09-12T18:44:42.903 INFO:teuthology.orchestra.run.smithi035.stdout:Package python3-rgw-2:17.0.0-14883.g10f7d25d.el8.x86_64 is already installed. 2022-09-12T18:44:42.988 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:44:42.989 INFO:teuthology.orchestra.run.smithi035.stdout:Nothing to do. 2022-09-12T18:44:42.989 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:44:43.073 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install python3-cephfs 2022-09-12T18:44:43.671 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:02:28 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:43.948 INFO:teuthology.orchestra.run.smithi138.stdout:rbd-mirror-17.0.0-14883.g10f7d25d.el8.x86_64.rp 2.9 MB/s | 3.5 MB 00:01 2022-09-12T18:44:43.949 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:44:43.949 INFO:teuthology.orchestra.run.smithi138.stdout:Total 2.9 MB/s | 3.5 MB 00:01 2022-09-12T18:44:43.950 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-09-12T18:44:43.961 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-09-12T18:44:43.961 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-09-12T18:44:44.022 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-09-12T18:44:44.026 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-09-12T18:44:44.466 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-09-12T18:44:44.659 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-mirror-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:44.668 INFO:teuthology.orchestra.run.smithi035.stdout:Package python3-cephfs-2:17.0.0-14883.g10f7d25d.el8.x86_64 is already installed. 2022-09-12T18:44:44.752 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:44:44.754 INFO:teuthology.orchestra.run.smithi035.stdout:Nothing to do. 2022-09-12T18:44:44.754 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:44:44.825 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install python3-rbd 2022-09-12T18:44:45.039 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: rbd-mirror-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:45.423 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:02:30 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:45.423 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-mirror-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:45.424 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:45.424 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-09-12T18:44:45.424 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-mirror-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:44:45.425 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:45.425 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:44:45.611 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install rbd-nbd 2022-09-12T18:44:46.194 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:31 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:46.419 INFO:teuthology.orchestra.run.smithi035.stdout:Package python3-rbd-2:17.0.0-14883.g10f7d25d.el8.x86_64 is already installed. 2022-09-12T18:44:46.503 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:44:46.504 INFO:teuthology.orchestra.run.smithi035.stdout:Nothing to do. 2022-09-12T18:44:46.504 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:44:46.576 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install rbd-fuse 2022-09-12T18:44:47.177 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:02:32 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:47.216 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2022-09-12T18:44:47.217 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:44:47.217 INFO:teuthology.orchestra.run.smithi138.stdout: Package Architecture Version Repository Size 2022-09-12T18:44:47.218 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:44:47.218 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2022-09-12T18:44:47.218 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-nbd x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 175 k 2022-09-12T18:44:47.219 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:47.219 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2022-09-12T18:44:47.219 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2022-09-12T18:44:47.219 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2022-09-12T18:44:47.220 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:47.220 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 175 k 2022-09-12T18:44:47.220 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 556 k 2022-09-12T18:44:47.220 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2022-09-12T18:44:47.445 INFO:teuthology.orchestra.run.smithi138.stdout:rbd-nbd-17.0.0-14883.g10f7d25d.el8.x86_64.rpm 770 kB/s | 175 kB 00:00 2022-09-12T18:44:47.446 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:44:47.446 INFO:teuthology.orchestra.run.smithi138.stdout:Total 766 kB/s | 175 kB 00:00 2022-09-12T18:44:47.446 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2022-09-12T18:44:47.456 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2022-09-12T18:44:47.457 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2022-09-12T18:44:47.504 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2022-09-12T18:44:47.507 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2022-09-12T18:44:47.639 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2022-09-12T18:44:47.840 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-nbd-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:48.094 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: rbd-nbd-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:48.275 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:44:48.276 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:44:48.276 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repo Size 2022-09-12T18:44:48.276 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:44:48.277 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2022-09-12T18:44:48.277 INFO:teuthology.orchestra.run.smithi035.stdout: rbd-fuse x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 94 k 2022-09-12T18:44:48.277 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:48.277 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2022-09-12T18:44:48.277 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:44:48.278 INFO:teuthology.orchestra.run.smithi035.stdout:Install 1 Package 2022-09-12T18:44:48.279 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:48.279 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 94 k 2022-09-12T18:44:48.279 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 280 k 2022-09-12T18:44:48.280 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2022-09-12T18:44:48.465 INFO:teuthology.orchestra.run.smithi035.stdout:rbd-fuse-17.0.0-14883.g10f7d25d.el8.x86_64.rpm 498 kB/s | 94 kB 00:00 2022-09-12T18:44:48.466 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:44:48.467 INFO:teuthology.orchestra.run.smithi035.stdout:Total 495 kB/s | 94 kB 00:00 2022-09-12T18:44:48.467 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2022-09-12T18:44:48.478 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2022-09-12T18:44:48.478 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2022-09-12T18:44:48.516 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-nbd-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:48.516 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:48.517 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2022-09-12T18:44:48.517 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-nbd-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:44:48.517 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:48.517 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2022-09-12T18:44:48.531 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2022-09-12T18:44:48.534 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2022-09-12T18:44:48.648 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2022-09-12T18:44:48.673 DEBUG:teuthology.parallel:result is None 2022-09-12T18:44:48.809 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : rbd-fuse-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:49.067 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: rbd-fuse-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:49.521 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : rbd-fuse-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:49.521 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:49.521 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2022-09-12T18:44:49.522 INFO:teuthology.orchestra.run.smithi035.stdout: rbd-fuse-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:44:49.522 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:49.522 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:44:49.726 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install rbd-mirror 2022-09-12T18:44:50.334 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:02:35 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:51.413 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:44:51.414 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:44:51.415 INFO:teuthology.orchestra.run.smithi035.stdout: Package Arch Version Repo Size 2022-09-12T18:44:51.415 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:44:51.415 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2022-09-12T18:44:51.415 INFO:teuthology.orchestra.run.smithi035.stdout: rbd-mirror x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 3.5 M 2022-09-12T18:44:51.416 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:51.416 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2022-09-12T18:44:51.416 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:44:51.416 INFO:teuthology.orchestra.run.smithi035.stdout:Install 1 Package 2022-09-12T18:44:51.417 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:51.417 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 3.5 M 2022-09-12T18:44:51.417 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 16 M 2022-09-12T18:44:51.417 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2022-09-12T18:44:52.100 INFO:teuthology.orchestra.run.smithi035.stdout:rbd-mirror-17.0.0-14883.g10f7d25d.el8.x86_64.rp 5.1 MB/s | 3.5 MB 00:00 2022-09-12T18:44:52.101 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:44:52.101 INFO:teuthology.orchestra.run.smithi035.stdout:Total 5.1 MB/s | 3.5 MB 00:00 2022-09-12T18:44:52.101 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2022-09-12T18:44:52.115 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2022-09-12T18:44:52.115 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2022-09-12T18:44:52.179 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2022-09-12T18:44:52.182 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2022-09-12T18:44:52.643 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2022-09-12T18:44:52.844 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : rbd-mirror-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:53.231 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: rbd-mirror-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:53.619 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : rbd-mirror-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:53.619 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:53.620 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2022-09-12T18:44:53.620 INFO:teuthology.orchestra.run.smithi035.stdout: rbd-mirror-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:44:53.621 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:53.621 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:44:53.841 DEBUG:teuthology.orchestra.run.smithi035:> sudo yum -y install rbd-nbd 2022-09-12T18:44:54.445 INFO:teuthology.orchestra.run.smithi035.stdout:Last metadata expiration check: 0:02:39 ago on Mon 12 Sep 2022 06:42:15 PM UTC. 2022-09-12T18:44:55.537 INFO:teuthology.orchestra.run.smithi035.stdout:Dependencies resolved. 2022-09-12T18:44:55.537 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:44:55.538 INFO:teuthology.orchestra.run.smithi035.stdout: Package Architecture Version Repository Size 2022-09-12T18:44:55.538 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:44:55.538 INFO:teuthology.orchestra.run.smithi035.stdout:Installing: 2022-09-12T18:44:55.538 INFO:teuthology.orchestra.run.smithi035.stdout: rbd-nbd x86_64 2:17.0.0-14883.g10f7d25d.el8 ceph 175 k 2022-09-12T18:44:55.539 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:55.539 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction Summary 2022-09-12T18:44:55.539 INFO:teuthology.orchestra.run.smithi035.stdout:================================================================================ 2022-09-12T18:44:55.539 INFO:teuthology.orchestra.run.smithi035.stdout:Install 1 Package 2022-09-12T18:44:55.539 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:55.540 INFO:teuthology.orchestra.run.smithi035.stdout:Total download size: 175 k 2022-09-12T18:44:55.540 INFO:teuthology.orchestra.run.smithi035.stdout:Installed size: 556 k 2022-09-12T18:44:55.540 INFO:teuthology.orchestra.run.smithi035.stdout:Downloading Packages: 2022-09-12T18:44:55.757 INFO:teuthology.orchestra.run.smithi035.stdout:rbd-nbd-17.0.0-14883.g10f7d25d.el8.x86_64.rpm 795 kB/s | 175 kB 00:00 2022-09-12T18:44:55.758 INFO:teuthology.orchestra.run.smithi035.stdout:-------------------------------------------------------------------------------- 2022-09-12T18:44:55.758 INFO:teuthology.orchestra.run.smithi035.stdout:Total 790 kB/s | 175 kB 00:00 2022-09-12T18:44:55.759 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction check 2022-09-12T18:44:55.770 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction check succeeded. 2022-09-12T18:44:55.771 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction test 2022-09-12T18:44:55.822 INFO:teuthology.orchestra.run.smithi035.stdout:Transaction test succeeded. 2022-09-12T18:44:55.826 INFO:teuthology.orchestra.run.smithi035.stdout:Running transaction 2022-09-12T18:44:55.948 INFO:teuthology.orchestra.run.smithi035.stdout: Preparing : 1/1 2022-09-12T18:44:56.150 INFO:teuthology.orchestra.run.smithi035.stdout: Installing : rbd-nbd-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:56.420 INFO:teuthology.orchestra.run.smithi035.stdout: Running scriptlet: rbd-nbd-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:56.853 INFO:teuthology.orchestra.run.smithi035.stdout: Verifying : rbd-nbd-2:17.0.0-14883.g10f7d25d.el8.x86_64 1/1 2022-09-12T18:44:56.854 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:56.854 INFO:teuthology.orchestra.run.smithi035.stdout:Installed: 2022-09-12T18:44:56.854 INFO:teuthology.orchestra.run.smithi035.stdout: rbd-nbd-2:17.0.0-14883.g10f7d25d.el8.x86_64 2022-09-12T18:44:56.854 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:56.855 INFO:teuthology.orchestra.run.smithi035.stdout:Complete! 2022-09-12T18:44:57.090 DEBUG:teuthology.parallel:result is None 2022-09-12T18:44:57.091 DEBUG:teuthology.orchestra.run.smithi035:> sudo lsb_release -is 2022-09-12T18:44:57.129 INFO:teuthology.orchestra.run.smithi035.stdout:CentOSStream 2022-09-12T18:44:57.131 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-09-12T18:44:57.132 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=10f7d25d6a0f340676cd251eab5a787ba01ace95 2022-09-12T18:44:57.300 DEBUG:teuthology.orchestra.run.smithi035:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-09-12T18:44:57.323 INFO:teuthology.orchestra.run.smithi035.stdout:17.0.0-14883.g10f7d25d.el8 2022-09-12T18:44:57.324 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-14883.g10f7d25d.el8 2022-09-12T18:44:57.325 INFO:teuthology.task.install:The correct ceph version 17.0.0-14883.g10f7d25d is installed. 2022-09-12T18:44:57.325 DEBUG:teuthology.orchestra.run.smithi138:> sudo lsb_release -is 2022-09-12T18:44:57.364 INFO:teuthology.orchestra.run.smithi138.stdout:CentOSStream 2022-09-12T18:44:57.365 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-09-12T18:44:57.366 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=10f7d25d6a0f340676cd251eab5a787ba01ace95 2022-09-12T18:44:57.515 DEBUG:teuthology.orchestra.run.smithi138:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-09-12T18:44:57.538 INFO:teuthology.orchestra.run.smithi138.stdout:17.0.0-14883.g10f7d25d.el8 2022-09-12T18:44:57.538 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-14883.g10f7d25d.el8 2022-09-12T18:44:57.539 INFO:teuthology.task.install:The correct ceph version 17.0.0-14883.g10f7d25d is installed. 2022-09-12T18:44:57.539 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-09-12T18:44:57.540 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2022-09-12T18:44:57.540 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-09-12T18:44:57.570 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-09-12T18:44:57.570 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-09-12T18:44:57.605 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-09-12T18:44:57.606 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2022-09-12T18:44:57.607 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd of=/usr/bin/daemon-helper 2022-09-12T18:44:57.637 DEBUG:teuthology.orchestra.run.smithi035:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-09-12T18:44:57.706 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-09-12T18:44:57.707 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/daemon-helper 2022-09-12T18:44:57.735 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-09-12T18:44:57.801 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-09-12T18:44:57.802 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2022-09-12T18:44:57.802 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd of=/usr/bin/adjust-ulimits 2022-09-12T18:44:57.834 DEBUG:teuthology.orchestra.run.smithi035:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-09-12T18:44:57.901 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-09-12T18:44:57.902 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/adjust-ulimits 2022-09-12T18:44:57.930 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-09-12T18:44:57.998 INFO:teuthology.run_tasks:Running task ceph... 2022-09-12T18:44:58.085 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2022-09-12T18:44:58.086 DEBUG:teuthology.orchestra.run.smithi035:> sudo chmod 777 /var/log/ceph 2022-09-12T18:44:58.088 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 777 /var/log/ceph 2022-09-12T18:44:58.115 INFO:tasks.ceph:Disabling ceph logrotate... 2022-09-12T18:44:58.116 DEBUG:teuthology.orchestra.run.smithi035:> sudo rm -f -- /etc/logrotate.d/ceph 2022-09-12T18:44:58.156 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /etc/logrotate.d/ceph 2022-09-12T18:44:58.184 INFO:tasks.ceph:Creating extra log directories... 2022-09-12T18:44:58.184 DEBUG:teuthology.orchestra.run.smithi035:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-09-12T18:44:58.224 DEBUG:teuthology.orchestra.run.smithi138:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-09-12T18:44:58.264 INFO:tasks.ceph:Creating ceph cluster c1... 2022-09-12T18:44:58.264 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2022-09-12T18:44:58.265 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2022-09-11_20:31:56-rgw-main-distro-default-smithi/7026931', 'branch': 'main', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi}', 'email': 'ceph-qa@ceph.io', 'first_in_suite': False, 'job_id': '7026931', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2022-09-11_20:31:56-rgw-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '10f7d25d6a0f340676cd251eab5a787ba01ace95', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '10f7d25d6a0f340676cd251eab5a787ba01ace95'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py', 'tests_ps.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'main', 'sha1': '10f7d25d6a0f340676cd251eab5a787ba01ace95'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 71, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '10f7d25d6a0f340676cd251eab5a787ba01ace95', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_10f7d25d6a0f340676cd251eab5a787ba01ace95/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '10f7d25d6a0f340676cd251eab5a787ba01ace95', 'targets': {'smithi035.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDhDcIgHo8yPwbEK2iHZHrqUzB4APgwzUj7VVWbHPtCAGblmY3il1ZTa06qXVADIwzGqmBDnClFbNh9Xl9ezouikXdGUn6gH+058Uc5DmuDveEWjx9ZG8ypl7K1NI6mA2ZF4ZhavIk1Mece12gvaQfSuc3aaBvpGU5M5Vzoe24AXLIZ9eOtsHpWz7KJ3tCLONBRtUUzCF4sj2xG1cclnzJEmD4DjOXa6CL0plEsq6miQhtw+c9AsYS/ZIZwGHIVOFuLdIASDixTFS74uuJaVt6hzwpRbjdRvfhRB4EeiiOFF9HRleD6wEIBxyzbNjqBeKeP6AYr74v+0E9K68iOMp54q2xxkmWFnhvsuVYTyYmU+A1VqJMRLuNfJBYyB0C2MxvcuXzFweggp7mbybUuhOKpgNAzBliyM7GY/VDZheHJInzFHLRTVY9XsJklqPugCYycox6C/YAlh0Q6pwQ0Sr2WaEMz6PAyG9t+eK+pZAaTXwux1fGSsfpah46Vmy9ngOM=', 'smithi138.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpUYYo//MHC5osesSZpvrMfoP5AJD18lKMj30OB4QvSUyQC7Nwq1VJzA3Q71fceJFqHa2/LJAxGfhT297bQk8IsAh6ewsfysQhYwYOC4DhWDuuP3m6O0v15ME6VtShlwvk9sowvjf1f/DcyDWr2jVtN91ZeHVCjSyFMU5Qvr7BnKk/rDp/AnJwxKrKHE5OhJLicQ/OWLs6Bc4lreV5M0zIxxJMalQYeL5VnLzDaPYQ6iuDbp/AYbhIPcjrQfp/QFcreVGJxsW3mRwbOpo9/2YYTOUisbXmboaxY+FU4M81ehTFntB9X5PxcMrOe3PX6MGzn9XSiRB/sTkH63R6TY5MRC/qd6J2XymC9/llkLMhNiIoKMPwm6zAOEUfDQc8U2gDDjPcqeIXMAfswyMo4M6iAyM/Wglt5+CE3olTSfUfdRiTaBIYH5NJPsQLYSdwYlUwBVLJau4+gdzdIfJwP3n5Oihatqx2aOOmiy8FIMHnsrekeQcZXS+/vxqNnFcc2OE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '10f7d25d6a0f340676cd251eab5a787ba01ace95', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b58f2c18636eb10faa77ed3614abd00cb85dfc2c', 'timestamp': '2022-09-11_20:31:56', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.14413'} 2022-09-12T18:44:58.266 DEBUG:teuthology.orchestra.run.smithi035:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2022-09-12T18:44:58.293 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2022-09-12T18:44:58.311 DEBUG:teuthology.orchestra.run.smithi035:> sudo install -d -m0777 -- /var/run/ceph 2022-09-12T18:44:58.351 DEBUG:teuthology.orchestra.run.smithi138:> sudo install -d -m0777 -- /var/run/ceph 2022-09-12T18:44:58.379 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2022-09-12T18:44:58.380 DEBUG:teuthology.orchestra.run.smithi035:> dd if=/scratch_devs of=/dev/stdout 2022-09-12T18:44:58.436 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-09-12T18:44:58.436 DEBUG:teuthology.orchestra.run.smithi035:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-09-12T18:44:58.493 INFO:teuthology.orchestra.run.smithi035.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-09-12T18:44:58.494 INFO:teuthology.orchestra.run.smithi035.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-12T18:44:58.494 INFO:teuthology.orchestra.run.smithi035.stdout:Device: 6h/6d Inode: 351920 Links: 1 2022-09-12T18:44:58.494 INFO:teuthology.orchestra.run.smithi035.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-12T18:44:58.494 INFO:teuthology.orchestra.run.smithi035.stdout:Context: system_u:object_r:device_t:s0 2022-09-12T18:44:58.495 INFO:teuthology.orchestra.run.smithi035.stdout:Access: 2022-09-12 18:44:53.009137989 +0000 2022-09-12T18:44:58.495 INFO:teuthology.orchestra.run.smithi035.stdout:Modify: 2022-09-12 18:41:23.967610807 +0000 2022-09-12T18:44:58.495 INFO:teuthology.orchestra.run.smithi035.stdout:Change: 2022-09-12 18:41:23.967610807 +0000 2022-09-12T18:44:58.495 INFO:teuthology.orchestra.run.smithi035.stdout: Birth: - 2022-09-12T18:44:58.503 INFO:teuthology.orchestra.run.smithi035.stderr:1+0 records in 2022-09-12T18:44:58.504 INFO:teuthology.orchestra.run.smithi035.stderr:1+0 records out 2022-09-12T18:44:58.504 INFO:teuthology.orchestra.run.smithi035.stderr:512 bytes copied, 0.000243709 s, 2.1 MB/s 2022-09-12T18:44:58.506 DEBUG:teuthology.orchestra.run.smithi035:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-09-12T18:44:58.564 INFO:teuthology.orchestra.run.smithi035.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-09-12T18:44:58.565 INFO:teuthology.orchestra.run.smithi035.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-12T18:44:58.565 INFO:teuthology.orchestra.run.smithi035.stdout:Device: 6h/6d Inode: 366840 Links: 1 2022-09-12T18:44:58.565 INFO:teuthology.orchestra.run.smithi035.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-12T18:44:58.565 INFO:teuthology.orchestra.run.smithi035.stdout:Context: system_u:object_r:device_t:s0 2022-09-12T18:44:58.566 INFO:teuthology.orchestra.run.smithi035.stdout:Access: 2022-09-12 18:44:53.009137989 +0000 2022-09-12T18:44:58.566 INFO:teuthology.orchestra.run.smithi035.stdout:Modify: 2022-09-12 18:41:24.528599356 +0000 2022-09-12T18:44:58.566 INFO:teuthology.orchestra.run.smithi035.stdout:Change: 2022-09-12 18:41:24.528599356 +0000 2022-09-12T18:44:58.566 INFO:teuthology.orchestra.run.smithi035.stdout: Birth: - 2022-09-12T18:44:58.574 INFO:teuthology.orchestra.run.smithi035.stderr:1+0 records in 2022-09-12T18:44:58.575 INFO:teuthology.orchestra.run.smithi035.stderr:1+0 records out 2022-09-12T18:44:58.575 INFO:teuthology.orchestra.run.smithi035.stderr:512 bytes copied, 0.000226591 s, 2.3 MB/s 2022-09-12T18:44:58.577 DEBUG:teuthology.orchestra.run.smithi035:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-09-12T18:44:58.635 INFO:teuthology.orchestra.run.smithi035.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-09-12T18:44:58.636 INFO:teuthology.orchestra.run.smithi035.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-12T18:44:58.637 INFO:teuthology.orchestra.run.smithi035.stdout:Device: 6h/6d Inode: 363008 Links: 1 2022-09-12T18:44:58.637 INFO:teuthology.orchestra.run.smithi035.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-12T18:44:58.637 INFO:teuthology.orchestra.run.smithi035.stdout:Context: system_u:object_r:device_t:s0 2022-09-12T18:44:58.638 INFO:teuthology.orchestra.run.smithi035.stdout:Access: 2022-09-12 18:44:53.009137989 +0000 2022-09-12T18:44:58.638 INFO:teuthology.orchestra.run.smithi035.stdout:Modify: 2022-09-12 18:41:25.090587886 +0000 2022-09-12T18:44:58.639 INFO:teuthology.orchestra.run.smithi035.stdout:Change: 2022-09-12 18:41:25.090587886 +0000 2022-09-12T18:44:58.639 INFO:teuthology.orchestra.run.smithi035.stdout: Birth: - 2022-09-12T18:44:58.645 INFO:teuthology.orchestra.run.smithi035.stderr:1+0 records in 2022-09-12T18:44:58.646 INFO:teuthology.orchestra.run.smithi035.stderr:1+0 records out 2022-09-12T18:44:58.646 INFO:teuthology.orchestra.run.smithi035.stderr:512 bytes copied, 0.000220593 s, 2.3 MB/s 2022-09-12T18:44:58.648 DEBUG:teuthology.orchestra.run.smithi035:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-09-12T18:44:58.706 INFO:teuthology.orchestra.run.smithi035.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-09-12T18:44:58.707 INFO:teuthology.orchestra.run.smithi035.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-12T18:44:58.707 INFO:teuthology.orchestra.run.smithi035.stdout:Device: 6h/6d Inode: 357315 Links: 1 2022-09-12T18:44:58.707 INFO:teuthology.orchestra.run.smithi035.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-12T18:44:58.708 INFO:teuthology.orchestra.run.smithi035.stdout:Context: system_u:object_r:device_t:s0 2022-09-12T18:44:58.708 INFO:teuthology.orchestra.run.smithi035.stdout:Access: 2022-09-12 18:44:53.009137989 +0000 2022-09-12T18:44:58.708 INFO:teuthology.orchestra.run.smithi035.stdout:Modify: 2022-09-12 18:41:25.635576762 +0000 2022-09-12T18:44:58.708 INFO:teuthology.orchestra.run.smithi035.stdout:Change: 2022-09-12 18:41:25.635576762 +0000 2022-09-12T18:44:58.708 INFO:teuthology.orchestra.run.smithi035.stdout: Birth: - 2022-09-12T18:44:58.716 INFO:teuthology.orchestra.run.smithi035.stderr:1+0 records in 2022-09-12T18:44:58.716 INFO:teuthology.orchestra.run.smithi035.stderr:1+0 records out 2022-09-12T18:44:58.717 INFO:teuthology.orchestra.run.smithi035.stderr:512 bytes copied, 0.000311345 s, 1.6 MB/s 2022-09-12T18:44:58.719 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-12T18:44:58.719 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi035.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2022-09-12T18:44:58.720 INFO:tasks.ceph:Generating config... 2022-09-12T18:44:58.721 INFO:tasks.ceph:[client] debug rgw = 20 2022-09-12T18:44:58.722 INFO:tasks.ceph:[client] debug rgw sync = 20 2022-09-12T18:44:58.722 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2022-09-12T18:44:58.722 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2022-09-12T18:44:58.723 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-09-12T18:44:58.723 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2022-09-12T18:44:58.723 INFO:tasks.ceph:[client] rgw data log num shards = 4 2022-09-12T18:44:58.723 INFO:tasks.ceph:[client] rgw md log max shards = 4 2022-09-12T18:44:58.724 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2022-09-12T18:44:58.724 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2022-09-12T18:44:58.724 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2022-09-12T18:44:58.725 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2022-09-12T18:44:58.725 INFO:tasks.ceph:[client] setgroup = ceph 2022-09-12T18:44:58.725 INFO:tasks.ceph:[client] setuser = ceph 2022-09-12T18:44:58.726 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2022-09-12T18:44:58.726 INFO:tasks.ceph:[mgr] debug mgr = 20 2022-09-12T18:44:58.726 INFO:tasks.ceph:[mgr] debug ms = 1 2022-09-12T18:44:58.727 INFO:tasks.ceph:[mon] debug mon = 20 2022-09-12T18:44:58.727 INFO:tasks.ceph:[mon] debug ms = 1 2022-09-12T18:44:58.727 INFO:tasks.ceph:[mon] debug paxos = 20 2022-09-12T18:44:58.727 INFO:tasks.ceph:[osd] debug ms = 1 2022-09-12T18:44:58.728 INFO:tasks.ceph:[osd] debug osd = 20 2022-09-12T18:44:58.728 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2022-09-12T18:44:58.728 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2022-09-12T18:44:58.729 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2022-09-12T18:44:58.729 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2022-09-12T18:44:58.729 INFO:tasks.ceph:Setting up c1.mon.a... 2022-09-12T18:44:58.730 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2022-09-12T18:44:58.817 INFO:teuthology.orchestra.run.smithi035.stdout:creating /etc/ceph/c1.keyring 2022-09-12T18:44:58.819 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2022-09-12T18:44:58.865 DEBUG:teuthology.orchestra.run.smithi035:> sudo chmod 0644 /etc/ceph/c1.keyring 2022-09-12T18:44:58.894 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.35')] 2022-09-12T18:44:58.895 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.35'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2022-09-12T18:44:58.896 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2022-09-12T18:44:58.897 DEBUG:teuthology.orchestra.run.smithi035:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2022-09-12T18:44:58.953 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.35 --print /home/ubuntu/cephtest/c1.monmap 2022-09-12T18:44:59.035 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:44:59.035 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:44:59.036 INFO:teuthology.orchestra.run.smithi035.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2022-09-12T18:44:59.037 INFO:teuthology.orchestra.run.smithi035.stdout:monmaptool: generated fsid 33adcfa0-1bd1-4a60-abd9-678f3cd4b239 2022-09-12T18:44:59.037 INFO:teuthology.orchestra.run.smithi035.stdout:setting min_mon_release = octopus 2022-09-12T18:44:59.037 INFO:teuthology.orchestra.run.smithi035.stdout:epoch 0 2022-09-12T18:44:59.037 INFO:teuthology.orchestra.run.smithi035.stdout:fsid 33adcfa0-1bd1-4a60-abd9-678f3cd4b239 2022-09-12T18:44:59.038 INFO:teuthology.orchestra.run.smithi035.stdout:last_changed 2022-09-12T18:44:59.032838+0000 2022-09-12T18:44:59.038 INFO:teuthology.orchestra.run.smithi035.stdout:created 2022-09-12T18:44:59.032838+0000 2022-09-12T18:44:59.038 INFO:teuthology.orchestra.run.smithi035.stdout:min_mon_release 15 (octopus) 2022-09-12T18:44:59.038 INFO:teuthology.orchestra.run.smithi035.stdout:election_strategy: 1 2022-09-12T18:44:59.039 INFO:teuthology.orchestra.run.smithi035.stdout:0: [v2:172.21.15.35:3300/0,v1:172.21.15.35:6789/0] mon.a 2022-09-12T18:44:59.039 INFO:teuthology.orchestra.run.smithi035.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2022-09-12T18:44:59.040 DEBUG:teuthology.orchestra.run.smithi035:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2022-09-12T18:44:59.095 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 33adcfa0-1bd1-4a60-abd9-678f3cd4b239... 2022-09-12T18:44:59.096 DEBUG:teuthology.orchestra.run.smithi035:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2022-09-12T18:44:59.137 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2022-09-12T18:44:59.180 INFO:teuthology.orchestra.run.smithi035.stdout:[global] 2022-09-12T18:44:59.181 INFO:teuthology.orchestra.run.smithi035.stdout: chdir = "" 2022-09-12T18:44:59.181 INFO:teuthology.orchestra.run.smithi035.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-09-12T18:44:59.181 INFO:teuthology.orchestra.run.smithi035.stdout: auth supported = cephx 2022-09-12T18:44:59.181 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.182 INFO:teuthology.orchestra.run.smithi035.stdout: filestore xattr use omap = true 2022-09-12T18:44:59.182 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.182 INFO:teuthology.orchestra.run.smithi035.stdout: mon clock drift allowed = 1.000 2022-09-12T18:44:59.182 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.183 INFO:teuthology.orchestra.run.smithi035.stdout: osd crush chooseleaf type = 0 2022-09-12T18:44:59.183 INFO:teuthology.orchestra.run.smithi035.stdout: auth debug = true 2022-09-12T18:44:59.183 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.183 INFO:teuthology.orchestra.run.smithi035.stdout: ms die on old message = true 2022-09-12T18:44:59.184 INFO:teuthology.orchestra.run.smithi035.stdout: ms die on bug = true 2022-09-12T18:44:59.184 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.184 INFO:teuthology.orchestra.run.smithi035.stdout: mon max pg per osd = 10000 # >= luminous 2022-09-12T18:44:59.184 INFO:teuthology.orchestra.run.smithi035.stdout: mon pg warn max object skew = 0 2022-09-12T18:44:59.184 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.185 INFO:teuthology.orchestra.run.smithi035.stdout: # disable pg_autoscaler by default for new pools 2022-09-12T18:44:59.185 INFO:teuthology.orchestra.run.smithi035.stdout: osd_pool_default_pg_autoscale_mode = off 2022-09-12T18:44:59.185 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.185 INFO:teuthology.orchestra.run.smithi035.stdout: osd pool default size = 2 2022-09-12T18:44:59.185 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.186 INFO:teuthology.orchestra.run.smithi035.stdout: mon osd allow primary affinity = true 2022-09-12T18:44:59.186 INFO:teuthology.orchestra.run.smithi035.stdout: mon osd allow pg remap = true 2022-09-12T18:44:59.186 INFO:teuthology.orchestra.run.smithi035.stdout: mon warn on legacy crush tunables = false 2022-09-12T18:44:59.186 INFO:teuthology.orchestra.run.smithi035.stdout: mon warn on crush straw calc version zero = false 2022-09-12T18:44:59.187 INFO:teuthology.orchestra.run.smithi035.stdout: mon warn on no sortbitwise = false 2022-09-12T18:44:59.187 INFO:teuthology.orchestra.run.smithi035.stdout: mon warn on osd down out interval zero = false 2022-09-12T18:44:59.187 INFO:teuthology.orchestra.run.smithi035.stdout: mon warn on too few osds = false 2022-09-12T18:44:59.187 INFO:teuthology.orchestra.run.smithi035.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-09-12T18:44:59.188 INFO:teuthology.orchestra.run.smithi035.stdout: mon_warn_on_pool_no_redundancy = false 2022-09-12T18:44:59.188 INFO:teuthology.orchestra.run.smithi035.stdout: mon_allow_pool_size_one = true 2022-09-12T18:44:59.189 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.189 INFO:teuthology.orchestra.run.smithi035.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-09-12T18:44:59.189 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.189 INFO:teuthology.orchestra.run.smithi035.stdout: osd default data pool replay window = 5 2022-09-12T18:44:59.189 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.190 INFO:teuthology.orchestra.run.smithi035.stdout: mon allow pool delete = true 2022-09-12T18:44:59.190 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.190 INFO:teuthology.orchestra.run.smithi035.stdout: mon cluster log file level = debug 2022-09-12T18:44:59.190 INFO:teuthology.orchestra.run.smithi035.stdout: debug asserts on shutdown = true 2022-09-12T18:44:59.191 INFO:teuthology.orchestra.run.smithi035.stdout: mon health detail to clog = false 2022-09-12T18:44:59.191 INFO:teuthology.orchestra.run.smithi035.stdout: mon host = 172.21.15.35 2022-09-12T18:44:59.191 INFO:teuthology.orchestra.run.smithi035.stdout: fsid = 33adcfa0-1bd1-4a60-abd9-678f3cd4b239 2022-09-12T18:44:59.191 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.192 INFO:teuthology.orchestra.run.smithi035.stdout:[osd] 2022-09-12T18:44:59.192 INFO:teuthology.orchestra.run.smithi035.stdout: osd journal size = 100 2022-09-12T18:44:59.192 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.192 INFO:teuthology.orchestra.run.smithi035.stdout: osd scrub load threshold = 5.0 2022-09-12T18:44:59.193 INFO:teuthology.orchestra.run.smithi035.stdout: osd scrub max interval = 600 2022-09-12T18:44:59.193 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.193 INFO:teuthology.orchestra.run.smithi035.stdout: osd recover clone overlap = true 2022-09-12T18:44:59.193 INFO:teuthology.orchestra.run.smithi035.stdout: osd recovery max chunk = 1048576 2022-09-12T18:44:59.193 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.194 INFO:teuthology.orchestra.run.smithi035.stdout: osd debug shutdown = true 2022-09-12T18:44:59.194 INFO:teuthology.orchestra.run.smithi035.stdout: osd debug op order = true 2022-09-12T18:44:59.194 INFO:teuthology.orchestra.run.smithi035.stdout: osd debug verify stray on activate = true 2022-09-12T18:44:59.194 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.195 INFO:teuthology.orchestra.run.smithi035.stdout: osd open classes on start = true 2022-09-12T18:44:59.195 INFO:teuthology.orchestra.run.smithi035.stdout: osd debug pg log writeout = true 2022-09-12T18:44:59.195 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.195 INFO:teuthology.orchestra.run.smithi035.stdout: osd deep scrub update digest min age = 30 2022-09-12T18:44:59.195 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.196 INFO:teuthology.orchestra.run.smithi035.stdout: osd map max advance = 10 2022-09-12T18:44:59.196 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.196 INFO:teuthology.orchestra.run.smithi035.stdout: journal zero on create = true 2022-09-12T18:44:59.196 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.197 INFO:teuthology.orchestra.run.smithi035.stdout: filestore ondisk finisher threads = 3 2022-09-12T18:44:59.197 INFO:teuthology.orchestra.run.smithi035.stdout: filestore apply finisher threads = 3 2022-09-12T18:44:59.197 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.197 INFO:teuthology.orchestra.run.smithi035.stdout: bdev debug aio = true 2022-09-12T18:44:59.197 INFO:teuthology.orchestra.run.smithi035.stdout: osd debug misdirected ops = true 2022-09-12T18:44:59.198 INFO:teuthology.orchestra.run.smithi035.stdout: debug ms = 1 2022-09-12T18:44:59.198 INFO:teuthology.orchestra.run.smithi035.stdout: debug osd = 20 2022-09-12T18:44:59.198 INFO:teuthology.orchestra.run.smithi035.stdout: rocksdb delete range threshold = 0 2022-09-12T18:44:59.198 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.199 INFO:teuthology.orchestra.run.smithi035.stdout:[mgr] 2022-09-12T18:44:59.199 INFO:teuthology.orchestra.run.smithi035.stdout: debug ms = 1 2022-09-12T18:44:59.199 INFO:teuthology.orchestra.run.smithi035.stdout: debug mgr = 20 2022-09-12T18:44:59.199 INFO:teuthology.orchestra.run.smithi035.stdout: debug mon = 20 2022-09-12T18:44:59.200 INFO:teuthology.orchestra.run.smithi035.stdout: debug auth = 20 2022-09-12T18:44:59.200 INFO:teuthology.orchestra.run.smithi035.stdout: mon reweight min pgs per osd = 4 2022-09-12T18:44:59.200 INFO:teuthology.orchestra.run.smithi035.stdout: mon reweight min bytes per osd = 10 2022-09-12T18:44:59.200 INFO:teuthology.orchestra.run.smithi035.stdout: mgr/telemetry/nag = false 2022-09-12T18:44:59.201 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.201 INFO:teuthology.orchestra.run.smithi035.stdout:[mon] 2022-09-12T18:44:59.201 INFO:teuthology.orchestra.run.smithi035.stdout: debug ms = 1 2022-09-12T18:44:59.201 INFO:teuthology.orchestra.run.smithi035.stdout: debug mon = 20 2022-09-12T18:44:59.201 INFO:teuthology.orchestra.run.smithi035.stdout: debug paxos = 20 2022-09-12T18:44:59.202 INFO:teuthology.orchestra.run.smithi035.stdout: debug auth = 20 2022-09-12T18:44:59.202 INFO:teuthology.orchestra.run.smithi035.stdout: mon data avail warn = 5 2022-09-12T18:44:59.202 INFO:teuthology.orchestra.run.smithi035.stdout: mon mgr mkfs grace = 240 2022-09-12T18:44:59.203 INFO:teuthology.orchestra.run.smithi035.stdout: mon reweight min pgs per osd = 4 2022-09-12T18:44:59.203 INFO:teuthology.orchestra.run.smithi035.stdout: mon osd reporter subtree level = osd 2022-09-12T18:44:59.203 INFO:teuthology.orchestra.run.smithi035.stdout: mon osd prime pg temp = true 2022-09-12T18:44:59.203 INFO:teuthology.orchestra.run.smithi035.stdout: mon reweight min bytes per osd = 10 2022-09-12T18:44:59.204 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.204 INFO:teuthology.orchestra.run.smithi035.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-09-12T18:44:59.204 INFO:teuthology.orchestra.run.smithi035.stdout: auth mon ticket ttl = 660 # 11m 2022-09-12T18:44:59.204 INFO:teuthology.orchestra.run.smithi035.stdout: auth service ticket ttl = 240 # 4m 2022-09-12T18:44:59.205 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.205 INFO:teuthology.orchestra.run.smithi035.stdout: # don't complain about insecure global_id in the test suite 2022-09-12T18:44:59.205 INFO:teuthology.orchestra.run.smithi035.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-09-12T18:44:59.205 INFO:teuthology.orchestra.run.smithi035.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-09-12T18:44:59.205 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.206 INFO:teuthology.orchestra.run.smithi035.stdout: # 1m isn't quite enough 2022-09-12T18:44:59.206 INFO:teuthology.orchestra.run.smithi035.stdout: mon_down_mkfs_grace = 2m 2022-09-12T18:44:59.206 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.206 INFO:teuthology.orchestra.run.smithi035.stdout: mon_warn_on_filestore_osds = false 2022-09-12T18:44:59.207 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:44:59.207 INFO:teuthology.orchestra.run.smithi035.stdout:[client] 2022-09-12T18:44:59.207 INFO:teuthology.orchestra.run.smithi035.stdout: rgw cache enabled = true 2022-09-12T18:44:59.207 INFO:teuthology.orchestra.run.smithi035.stdout: rgw enable ops log = true 2022-09-12T18:44:59.207 INFO:teuthology.orchestra.run.smithi035.stdout: rgw enable usage log = true 2022-09-12T18:44:59.208 INFO:teuthology.orchestra.run.smithi035.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-09-12T18:44:59.208 INFO:teuthology.orchestra.run.smithi035.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-09-12T18:44:59.208 INFO:teuthology.orchestra.run.smithi035.stdout: debug rgw = 20 2022-09-12T18:44:59.208 INFO:teuthology.orchestra.run.smithi035.stdout: debug rgw sync = 20 2022-09-12T18:44:59.209 INFO:teuthology.orchestra.run.smithi035.stdout: rgw crypt require ssl = False 2022-09-12T18:44:59.209 INFO:teuthology.orchestra.run.smithi035.stdout: rgw crypt s3 kms backend = testing 2022-09-12T18:44:59.209 INFO:teuthology.orchestra.run.smithi035.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-09-12T18:44:59.209 INFO:teuthology.orchestra.run.smithi035.stdout: rgw curl low speed time = 300 2022-09-12T18:44:59.210 INFO:teuthology.orchestra.run.smithi035.stdout: rgw data log num shards = 4 2022-09-12T18:44:59.210 INFO:teuthology.orchestra.run.smithi035.stdout: rgw md log max shards = 4 2022-09-12T18:44:59.210 INFO:teuthology.orchestra.run.smithi035.stdout: rgw sync data inject err probability = 0.1 2022-09-12T18:44:59.210 INFO:teuthology.orchestra.run.smithi035.stdout: rgw sync log trim interval = 0 2022-09-12T18:44:59.210 INFO:teuthology.orchestra.run.smithi035.stdout: rgw sync meta inject err probability = 0.1 2022-09-12T18:44:59.211 INFO:teuthology.orchestra.run.smithi035.stdout: rgw sync obj etag verify = True 2022-09-12T18:44:59.211 INFO:teuthology.orchestra.run.smithi035.stdout: setgroup = ceph 2022-09-12T18:44:59.211 INFO:teuthology.orchestra.run.smithi035.stdout: setuser = ceph 2022-09-12T18:44:59.211 INFO:teuthology.orchestra.run.smithi035.stdout:[mon.a] 2022-09-12T18:44:59.212 INFO:teuthology.orchestra.run.smithi035.stdout:[client.0] 2022-09-12T18:44:59.212 INFO:teuthology.orchestra.run.smithi035.stdout: rgw data notify interval msec = 0 2022-09-12T18:44:59.212 INFO:teuthology.orchestra.run.smithi035.stdout:[osd.0] 2022-09-12T18:44:59.212 INFO:teuthology.orchestra.run.smithi035.stdout: osd_max_omap_entries_per_request = 10 2022-09-12T18:44:59.213 INFO:teuthology.orchestra.run.smithi035.stdout:[osd.1] 2022-09-12T18:44:59.213 INFO:teuthology.orchestra.run.smithi035.stdout: osd_max_omap_entries_per_request = 1000 2022-09-12T18:44:59.213 INFO:teuthology.orchestra.run.smithi035.stdout:[osd.2] 2022-09-12T18:44:59.213 INFO:teuthology.orchestra.run.smithi035.stdout: osd_max_omap_entries_per_request = 10000 2022-09-12T18:44:59.215 INFO:teuthology.orchestra.run.smithi138.stdout:[global] 2022-09-12T18:44:59.216 INFO:teuthology.orchestra.run.smithi138.stdout: chdir = "" 2022-09-12T18:44:59.216 INFO:teuthology.orchestra.run.smithi138.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-09-12T18:44:59.216 INFO:teuthology.orchestra.run.smithi138.stdout: auth supported = cephx 2022-09-12T18:44:59.216 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.217 INFO:teuthology.orchestra.run.smithi138.stdout: filestore xattr use omap = true 2022-09-12T18:44:59.217 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.217 INFO:teuthology.orchestra.run.smithi138.stdout: mon clock drift allowed = 1.000 2022-09-12T18:44:59.217 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.217 INFO:teuthology.orchestra.run.smithi138.stdout: osd crush chooseleaf type = 0 2022-09-12T18:44:59.218 INFO:teuthology.orchestra.run.smithi138.stdout: auth debug = true 2022-09-12T18:44:59.218 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.218 INFO:teuthology.orchestra.run.smithi138.stdout: ms die on old message = true 2022-09-12T18:44:59.218 INFO:teuthology.orchestra.run.smithi138.stdout: ms die on bug = true 2022-09-12T18:44:59.219 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.219 INFO:teuthology.orchestra.run.smithi138.stdout: mon max pg per osd = 10000 # >= luminous 2022-09-12T18:44:59.219 INFO:teuthology.orchestra.run.smithi138.stdout: mon pg warn max object skew = 0 2022-09-12T18:44:59.219 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.220 INFO:teuthology.orchestra.run.smithi138.stdout: # disable pg_autoscaler by default for new pools 2022-09-12T18:44:59.220 INFO:teuthology.orchestra.run.smithi138.stdout: osd_pool_default_pg_autoscale_mode = off 2022-09-12T18:44:59.220 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.220 INFO:teuthology.orchestra.run.smithi138.stdout: osd pool default size = 2 2022-09-12T18:44:59.221 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.221 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd allow primary affinity = true 2022-09-12T18:44:59.221 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd allow pg remap = true 2022-09-12T18:44:59.221 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on legacy crush tunables = false 2022-09-12T18:44:59.221 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on crush straw calc version zero = false 2022-09-12T18:44:59.222 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on no sortbitwise = false 2022-09-12T18:44:59.222 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on osd down out interval zero = false 2022-09-12T18:44:59.222 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on too few osds = false 2022-09-12T18:44:59.222 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-09-12T18:44:59.223 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_pool_no_redundancy = false 2022-09-12T18:44:59.223 INFO:teuthology.orchestra.run.smithi138.stdout: mon_allow_pool_size_one = true 2022-09-12T18:44:59.223 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.223 INFO:teuthology.orchestra.run.smithi138.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-09-12T18:44:59.223 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.224 INFO:teuthology.orchestra.run.smithi138.stdout: osd default data pool replay window = 5 2022-09-12T18:44:59.224 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.224 INFO:teuthology.orchestra.run.smithi138.stdout: mon allow pool delete = true 2022-09-12T18:44:59.224 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.225 INFO:teuthology.orchestra.run.smithi138.stdout: mon cluster log file level = debug 2022-09-12T18:44:59.225 INFO:teuthology.orchestra.run.smithi138.stdout: debug asserts on shutdown = true 2022-09-12T18:44:59.225 INFO:teuthology.orchestra.run.smithi138.stdout: mon health detail to clog = false 2022-09-12T18:44:59.225 INFO:teuthology.orchestra.run.smithi138.stdout: mon host = 172.21.15.35 2022-09-12T18:44:59.225 INFO:teuthology.orchestra.run.smithi138.stdout: fsid = 33adcfa0-1bd1-4a60-abd9-678f3cd4b239 2022-09-12T18:44:59.226 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.235 INFO:teuthology.orchestra.run.smithi138.stdout:[osd] 2022-09-12T18:44:59.235 INFO:teuthology.orchestra.run.smithi138.stdout: osd journal size = 100 2022-09-12T18:44:59.235 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.236 INFO:teuthology.orchestra.run.smithi138.stdout: osd scrub load threshold = 5.0 2022-09-12T18:44:59.236 INFO:teuthology.orchestra.run.smithi138.stdout: osd scrub max interval = 600 2022-09-12T18:44:59.236 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.236 INFO:teuthology.orchestra.run.smithi138.stdout: osd recover clone overlap = true 2022-09-12T18:44:59.237 INFO:teuthology.orchestra.run.smithi138.stdout: osd recovery max chunk = 1048576 2022-09-12T18:44:59.237 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.237 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug shutdown = true 2022-09-12T18:44:59.237 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug op order = true 2022-09-12T18:44:59.238 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug verify stray on activate = true 2022-09-12T18:44:59.238 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.238 INFO:teuthology.orchestra.run.smithi138.stdout: osd open classes on start = true 2022-09-12T18:44:59.238 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug pg log writeout = true 2022-09-12T18:44:59.239 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.239 INFO:teuthology.orchestra.run.smithi138.stdout: osd deep scrub update digest min age = 30 2022-09-12T18:44:59.239 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.239 INFO:teuthology.orchestra.run.smithi138.stdout: osd map max advance = 10 2022-09-12T18:44:59.240 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.240 INFO:teuthology.orchestra.run.smithi138.stdout: journal zero on create = true 2022-09-12T18:44:59.240 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.240 INFO:teuthology.orchestra.run.smithi138.stdout: filestore ondisk finisher threads = 3 2022-09-12T18:44:59.240 INFO:teuthology.orchestra.run.smithi138.stdout: filestore apply finisher threads = 3 2022-09-12T18:44:59.241 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.241 INFO:teuthology.orchestra.run.smithi138.stdout: bdev debug aio = true 2022-09-12T18:44:59.241 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug misdirected ops = true 2022-09-12T18:44:59.241 INFO:teuthology.orchestra.run.smithi138.stdout: debug ms = 1 2022-09-12T18:44:59.242 INFO:teuthology.orchestra.run.smithi138.stdout: debug osd = 20 2022-09-12T18:44:59.242 INFO:teuthology.orchestra.run.smithi138.stdout: rocksdb delete range threshold = 0 2022-09-12T18:44:59.242 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.242 INFO:teuthology.orchestra.run.smithi138.stdout:[mgr] 2022-09-12T18:44:59.242 INFO:teuthology.orchestra.run.smithi138.stdout: debug ms = 1 2022-09-12T18:44:59.243 INFO:teuthology.orchestra.run.smithi138.stdout: debug mgr = 20 2022-09-12T18:44:59.243 INFO:teuthology.orchestra.run.smithi138.stdout: debug mon = 20 2022-09-12T18:44:59.243 INFO:teuthology.orchestra.run.smithi138.stdout: debug auth = 20 2022-09-12T18:44:59.243 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min pgs per osd = 4 2022-09-12T18:44:59.244 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min bytes per osd = 10 2022-09-12T18:44:59.244 INFO:teuthology.orchestra.run.smithi138.stdout: mgr/telemetry/nag = false 2022-09-12T18:44:59.244 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.244 INFO:teuthology.orchestra.run.smithi138.stdout:[mon] 2022-09-12T18:44:59.244 INFO:teuthology.orchestra.run.smithi138.stdout: debug ms = 1 2022-09-12T18:44:59.245 INFO:teuthology.orchestra.run.smithi138.stdout: debug mon = 20 2022-09-12T18:44:59.245 INFO:teuthology.orchestra.run.smithi138.stdout: debug paxos = 20 2022-09-12T18:44:59.245 INFO:teuthology.orchestra.run.smithi138.stdout: debug auth = 20 2022-09-12T18:44:59.245 INFO:teuthology.orchestra.run.smithi138.stdout: mon data avail warn = 5 2022-09-12T18:44:59.246 INFO:teuthology.orchestra.run.smithi138.stdout: mon mgr mkfs grace = 240 2022-09-12T18:44:59.246 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min pgs per osd = 4 2022-09-12T18:44:59.246 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd reporter subtree level = osd 2022-09-12T18:44:59.246 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd prime pg temp = true 2022-09-12T18:44:59.247 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min bytes per osd = 10 2022-09-12T18:44:59.247 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.247 INFO:teuthology.orchestra.run.smithi138.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-09-12T18:44:59.247 INFO:teuthology.orchestra.run.smithi138.stdout: auth mon ticket ttl = 660 # 11m 2022-09-12T18:44:59.248 INFO:teuthology.orchestra.run.smithi138.stdout: auth service ticket ttl = 240 # 4m 2022-09-12T18:44:59.248 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.248 INFO:teuthology.orchestra.run.smithi138.stdout: # don't complain about insecure global_id in the test suite 2022-09-12T18:44:59.248 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-09-12T18:44:59.249 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-09-12T18:44:59.249 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.249 INFO:teuthology.orchestra.run.smithi138.stdout: # 1m isn't quite enough 2022-09-12T18:44:59.249 INFO:teuthology.orchestra.run.smithi138.stdout: mon_down_mkfs_grace = 2m 2022-09-12T18:44:59.250 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.250 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_filestore_osds = false 2022-09-12T18:44:59.250 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:44:59.250 INFO:teuthology.orchestra.run.smithi138.stdout:[client] 2022-09-12T18:44:59.251 INFO:teuthology.orchestra.run.smithi138.stdout: rgw cache enabled = true 2022-09-12T18:44:59.251 INFO:teuthology.orchestra.run.smithi138.stdout: rgw enable ops log = true 2022-09-12T18:44:59.251 INFO:teuthology.orchestra.run.smithi138.stdout: rgw enable usage log = true 2022-09-12T18:44:59.251 INFO:teuthology.orchestra.run.smithi138.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-09-12T18:44:59.252 INFO:teuthology.orchestra.run.smithi138.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-09-12T18:44:59.252 INFO:teuthology.orchestra.run.smithi138.stdout: debug rgw = 20 2022-09-12T18:44:59.252 INFO:teuthology.orchestra.run.smithi138.stdout: debug rgw sync = 20 2022-09-12T18:44:59.252 INFO:teuthology.orchestra.run.smithi138.stdout: rgw crypt require ssl = False 2022-09-12T18:44:59.252 INFO:teuthology.orchestra.run.smithi138.stdout: rgw crypt s3 kms backend = testing 2022-09-12T18:44:59.253 INFO:teuthology.orchestra.run.smithi138.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-09-12T18:44:59.253 INFO:teuthology.orchestra.run.smithi138.stdout: rgw curl low speed time = 300 2022-09-12T18:44:59.253 INFO:teuthology.orchestra.run.smithi138.stdout: rgw data log num shards = 4 2022-09-12T18:44:59.253 INFO:teuthology.orchestra.run.smithi138.stdout: rgw md log max shards = 4 2022-09-12T18:44:59.254 INFO:teuthology.orchestra.run.smithi138.stdout: rgw sync data inject err probability = 0.1 2022-09-12T18:44:59.254 INFO:teuthology.orchestra.run.smithi138.stdout: rgw sync log trim interval = 0 2022-09-12T18:44:59.254 INFO:teuthology.orchestra.run.smithi138.stdout: rgw sync meta inject err probability = 0.1 2022-09-12T18:44:59.254 INFO:teuthology.orchestra.run.smithi138.stdout: rgw sync obj etag verify = True 2022-09-12T18:44:59.254 INFO:teuthology.orchestra.run.smithi138.stdout: setgroup = ceph 2022-09-12T18:44:59.255 INFO:teuthology.orchestra.run.smithi138.stdout: setuser = ceph 2022-09-12T18:44:59.255 INFO:teuthology.orchestra.run.smithi138.stdout:[mon.a] 2022-09-12T18:44:59.255 INFO:teuthology.orchestra.run.smithi138.stdout:[client.0] 2022-09-12T18:44:59.255 INFO:teuthology.orchestra.run.smithi138.stdout: rgw data notify interval msec = 0 2022-09-12T18:44:59.256 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.0] 2022-09-12T18:44:59.256 INFO:teuthology.orchestra.run.smithi138.stdout: osd_max_omap_entries_per_request = 10 2022-09-12T18:44:59.256 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.1] 2022-09-12T18:44:59.256 INFO:teuthology.orchestra.run.smithi138.stdout: osd_max_omap_entries_per_request = 1000 2022-09-12T18:44:59.257 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.2] 2022-09-12T18:44:59.257 INFO:teuthology.orchestra.run.smithi138.stdout: osd_max_omap_entries_per_request = 10000 2022-09-12T18:44:59.257 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2022-09-12T18:44:59.258 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2022-09-12T18:44:59.317 INFO:tasks.ceph:Copying monmap to all nodes... 2022-09-12T18:44:59.318 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2022-09-12T18:44:59.319 DEBUG:teuthology.orchestra.run.smithi035:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2022-09-12T18:44:59.338 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2022-09-12T18:44:59.339 DEBUG:teuthology.orchestra.run.smithi035:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2022-09-12T18:44:59.397 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi035.front.sepia.ceph.com 2022-09-12T18:44:59.398 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2022-09-12T18:44:59.398 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd of=/etc/ceph/c1.keyring 2022-09-12T18:44:59.398 DEBUG:teuthology.orchestra.run.smithi035:> sudo chmod 0644 /etc/ceph/c1.keyring 2022-09-12T18:44:59.474 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2022-09-12T18:44:59.475 DEBUG:teuthology.orchestra.run.smithi035:> dd of=/home/ubuntu/cephtest/c1.monmap 2022-09-12T18:44:59.533 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi138.front.sepia.ceph.com 2022-09-12T18:44:59.534 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-09-12T18:44:59.535 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/ceph/c1.keyring 2022-09-12T18:44:59.536 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0644 /etc/ceph/c1.keyring 2022-09-12T18:44:59.575 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-09-12T18:44:59.575 DEBUG:teuthology.orchestra.run.smithi138:> dd of=/home/ubuntu/cephtest/c1.monmap 2022-09-12T18:44:59.633 INFO:tasks.ceph:Setting up mon nodes... 2022-09-12T18:44:59.633 INFO:tasks.ceph:Setting up mgr nodes... 2022-09-12T18:44:59.634 DEBUG:teuthology.orchestra.run.smithi035:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2022-09-12T18:44:59.689 INFO:teuthology.orchestra.run.smithi035.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2022-09-12T18:44:59.692 INFO:tasks.ceph:Setting up mds nodes... 2022-09-12T18:44:59.692 INFO:tasks.ceph_client:Setting up client nodes... 2022-09-12T18:44:59.693 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2022-09-12T18:44:59.738 INFO:teuthology.orchestra.run.smithi035.stdout:creating /etc/ceph/c1.client.0.keyring 2022-09-12T18:44:59.750 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2022-09-12T18:44:59.794 INFO:teuthology.orchestra.run.smithi035.stdout:creating /etc/ceph/c1.client.1.keyring 2022-09-12T18:44:59.808 INFO:tasks.ceph:Running mkfs on osd nodes... 2022-09-12T18:44:59.808 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi035.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2022-09-12T18:44:59.809 DEBUG:teuthology.orchestra.run.smithi035:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2022-09-12T18:44:59.837 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-12T18:44:59.838 INFO:tasks.ceph:role: c1.osd.0 2022-09-12T18:44:59.838 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi035.front.sepia.ceph.com 2022-09-12T18:44:59.839 DEBUG:teuthology.orchestra.run.smithi035:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-09-12T18:44:59.913 INFO:teuthology.orchestra.run.smithi035.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-09-12T18:44:59.914 INFO:teuthology.orchestra.run.smithi035.stdout: = sectsz=512 attr=2, projid32bit=1 2022-09-12T18:44:59.914 INFO:teuthology.orchestra.run.smithi035.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-09-12T18:44:59.915 INFO:teuthology.orchestra.run.smithi035.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-09-12T18:44:59.915 INFO:teuthology.orchestra.run.smithi035.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-09-12T18:44:59.915 INFO:teuthology.orchestra.run.smithi035.stdout: = sunit=0 swidth=0 blks 2022-09-12T18:44:59.916 INFO:teuthology.orchestra.run.smithi035.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-09-12T18:44:59.916 INFO:teuthology.orchestra.run.smithi035.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-09-12T18:44:59.916 INFO:teuthology.orchestra.run.smithi035.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-09-12T18:44:59.916 INFO:teuthology.orchestra.run.smithi035.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-09-12T18:45:00.363 INFO:teuthology.orchestra.run.smithi035.stdout:Discarding blocks...Done. 2022-09-12T18:45:00.365 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi035.front.sepia.ceph.com -o noatime 2022-09-12T18:45:00.366 DEBUG:teuthology.orchestra.run.smithi035:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2022-09-12T18:45:00.486 DEBUG:teuthology.orchestra.run.smithi035:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2022-09-12T18:45:00.520 DEBUG:teuthology.orchestra.run.smithi035:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2022-09-12T18:45:00.589 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-12T18:45:00.589 INFO:tasks.ceph:role: c1.osd.1 2022-09-12T18:45:00.590 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi035.front.sepia.ceph.com 2022-09-12T18:45:00.590 DEBUG:teuthology.orchestra.run.smithi035:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-09-12T18:45:00.665 INFO:teuthology.orchestra.run.smithi035.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-09-12T18:45:00.666 INFO:teuthology.orchestra.run.smithi035.stdout: = sectsz=512 attr=2, projid32bit=1 2022-09-12T18:45:00.666 INFO:teuthology.orchestra.run.smithi035.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-09-12T18:45:00.666 INFO:teuthology.orchestra.run.smithi035.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-09-12T18:45:00.667 INFO:teuthology.orchestra.run.smithi035.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-09-12T18:45:00.667 INFO:teuthology.orchestra.run.smithi035.stdout: = sunit=0 swidth=0 blks 2022-09-12T18:45:00.667 INFO:teuthology.orchestra.run.smithi035.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-09-12T18:45:00.667 INFO:teuthology.orchestra.run.smithi035.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-09-12T18:45:00.668 INFO:teuthology.orchestra.run.smithi035.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-09-12T18:45:00.668 INFO:teuthology.orchestra.run.smithi035.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-09-12T18:45:01.110 INFO:teuthology.orchestra.run.smithi035.stdout:Discarding blocks...Done. 2022-09-12T18:45:01.111 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi035.front.sepia.ceph.com -o noatime 2022-09-12T18:45:01.112 DEBUG:teuthology.orchestra.run.smithi035:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2022-09-12T18:45:01.200 DEBUG:teuthology.orchestra.run.smithi035:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2022-09-12T18:45:01.234 DEBUG:teuthology.orchestra.run.smithi035:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2022-09-12T18:45:01.303 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-12T18:45:01.304 INFO:tasks.ceph:role: c1.osd.2 2022-09-12T18:45:01.304 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi035.front.sepia.ceph.com 2022-09-12T18:45:01.304 DEBUG:teuthology.orchestra.run.smithi035:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-09-12T18:45:01.379 INFO:teuthology.orchestra.run.smithi035.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-09-12T18:45:01.380 INFO:teuthology.orchestra.run.smithi035.stdout: = sectsz=512 attr=2, projid32bit=1 2022-09-12T18:45:01.380 INFO:teuthology.orchestra.run.smithi035.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-09-12T18:45:01.380 INFO:teuthology.orchestra.run.smithi035.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-09-12T18:45:01.380 INFO:teuthology.orchestra.run.smithi035.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-09-12T18:45:01.381 INFO:teuthology.orchestra.run.smithi035.stdout: = sunit=0 swidth=0 blks 2022-09-12T18:45:01.381 INFO:teuthology.orchestra.run.smithi035.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-09-12T18:45:01.381 INFO:teuthology.orchestra.run.smithi035.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-09-12T18:45:01.381 INFO:teuthology.orchestra.run.smithi035.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-09-12T18:45:01.382 INFO:teuthology.orchestra.run.smithi035.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-09-12T18:45:01.824 INFO:teuthology.orchestra.run.smithi035.stdout:Discarding blocks...Done. 2022-09-12T18:45:01.826 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi035.front.sepia.ceph.com -o noatime 2022-09-12T18:45:01.827 DEBUG:teuthology.orchestra.run.smithi035:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2022-09-12T18:45:01.942 DEBUG:teuthology.orchestra.run.smithi035:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2022-09-12T18:45:01.975 DEBUG:teuthology.orchestra.run.smithi035:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2022-09-12T18:45:02.062 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:45:02.058+0000 7f72fc690440 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2022-09-12T18:45:02.063 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:45:02.058+0000 7f72fc690440 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2022-09-12T18:45:02.063 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:45:02.058+0000 7f72fc690440 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2022-09-12T18:45:02.063 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:45:02.058+0000 7f72fc690440 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2022-09-12T18:45:02.063 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:45:02.058+0000 7f72fc690440 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2022-09-12T18:45:02.064 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:45:02.059+0000 7f72fc690440 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2022-09-12T18:45:04.148 DEBUG:teuthology.orchestra.run.smithi035:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2022-09-12T18:45:04.179 DEBUG:teuthology.orchestra.run.smithi035:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2022-09-12T18:45:04.269 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:45:04.265+0000 7f8a9eaa2440 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2022-09-12T18:45:04.270 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:45:04.265+0000 7f8a9eaa2440 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2022-09-12T18:45:04.270 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:45:04.265+0000 7f8a9eaa2440 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2022-09-12T18:45:04.270 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:45:04.265+0000 7f8a9eaa2440 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2022-09-12T18:45:04.270 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:45:04.265+0000 7f8a9eaa2440 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2022-09-12T18:45:04.271 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:45:04.265+0000 7f8a9eaa2440 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2022-09-12T18:45:06.337 DEBUG:teuthology.orchestra.run.smithi035:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2022-09-12T18:45:06.370 DEBUG:teuthology.orchestra.run.smithi035:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2022-09-12T18:45:06.463 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:45:06.458+0000 7f75b8dfa440 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2022-09-12T18:45:06.464 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:45:06.459+0000 7f75b8dfa440 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2022-09-12T18:45:06.464 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:45:06.459+0000 7f75b8dfa440 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2022-09-12T18:45:06.464 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:45:06.459+0000 7f75b8dfa440 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2022-09-12T18:45:06.464 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:45:06.459+0000 7f75b8dfa440 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2022-09-12T18:45:06.465 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:45:06.459+0000 7f75b8dfa440 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2022-09-12T18:45:08.533 DEBUG:teuthology.orchestra.run.smithi035:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2022-09-12T18:45:08.565 INFO:tasks.ceph:Reading keys from all nodes... 2022-09-12T18:45:08.566 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2022-09-12T18:45:08.566 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2022-09-12T18:45:08.637 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2022-09-12T18:45:08.638 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2022-09-12T18:45:08.708 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2022-09-12T18:45:08.709 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2022-09-12T18:45:08.778 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2022-09-12T18:45:08.778 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2022-09-12T18:45:08.848 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2022-09-12T18:45:08.849 DEBUG:teuthology.orchestra.run.smithi035:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2022-09-12T18:45:08.907 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2022-09-12T18:45:08.908 DEBUG:teuthology.orchestra.run.smithi035:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2022-09-12T18:45:08.967 INFO:tasks.ceph:Adding keys to all mons... 2022-09-12T18:45:08.967 DEBUG:teuthology.orchestra.run.smithi035:> sudo tee -a /etc/ceph/c1.keyring 2022-09-12T18:45:09.035 INFO:teuthology.orchestra.run.smithi035.stdout:[mgr.x] 2022-09-12T18:45:09.035 INFO:teuthology.orchestra.run.smithi035.stdout: key = AQArfh9jDy72KBAA+dmnuZR0SwB+YgqdLRw4EA== 2022-09-12T18:45:09.036 INFO:teuthology.orchestra.run.smithi035.stdout:[osd.0] 2022-09-12T18:45:09.036 INFO:teuthology.orchestra.run.smithi035.stdout: key = AQAufh9jR4qMAxAA02LQS8Q+Q8BASFpOZ3rcuw== 2022-09-12T18:45:09.036 INFO:teuthology.orchestra.run.smithi035.stdout:[osd.1] 2022-09-12T18:45:09.036 INFO:teuthology.orchestra.run.smithi035.stdout: key = AQAwfh9jwVHgDxAATGlhphBYfxtIR9tLuRoxqw== 2022-09-12T18:45:09.037 INFO:teuthology.orchestra.run.smithi035.stdout:[osd.2] 2022-09-12T18:45:09.037 INFO:teuthology.orchestra.run.smithi035.stdout: key = AQAyfh9jxwdvGxAAPmCEHORoMw8SrTwSz9DdWw== 2022-09-12T18:45:09.037 INFO:teuthology.orchestra.run.smithi035.stdout:[client.0] 2022-09-12T18:45:09.037 INFO:teuthology.orchestra.run.smithi035.stdout: key = AQArfh9jmxrVKxAATym51plmJ5hPc9e6qpkWAg== 2022-09-12T18:45:09.037 INFO:teuthology.orchestra.run.smithi035.stdout:[client.1] 2022-09-12T18:45:09.038 INFO:teuthology.orchestra.run.smithi035.stdout: key = AQArfh9jbp01LxAAW9z2yeU0mevXTsIis0j8hA== 2022-09-12T18:45:09.039 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-09-12T18:45:09.135 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-09-12T18:45:09.190 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-09-12T18:45:09.245 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-09-12T18:45:09.299 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-09-12T18:45:09.356 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-09-12T18:45:09.413 INFO:tasks.ceph:Running mkfs on mon nodes... 2022-09-12T18:45:09.414 DEBUG:teuthology.orchestra.run.smithi035:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2022-09-12T18:45:09.442 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2022-09-12T18:45:09.540 DEBUG:teuthology.orchestra.run.smithi035:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2022-09-12T18:45:09.568 DEBUG:teuthology.orchestra.run.smithi035:> rm -- /home/ubuntu/cephtest/c1.monmap 2022-09-12T18:45:09.625 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2022-09-12T18:45:09.626 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2022-09-12T18:45:09.627 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2022-09-12T18:45:09.668 INFO:tasks.ceph.c1.mon.a:Started 2022-09-12T18:45:09.668 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2022-09-12T18:45:09.669 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2022-09-12T18:45:09.669 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2022-09-12T18:45:09.671 INFO:tasks.ceph.c1.mgr.x:Started 2022-09-12T18:45:09.672 INFO:tasks.ceph:Setting crush tunables to default 2022-09-12T18:45:09.672 DEBUG:teuthology.orchestra.run.smithi035:> sudo ceph --cluster c1 osd crush tunables default 2022-09-12T18:45:09.883 INFO:teuthology.orchestra.run.smithi035.stderr:adjusted tunables profile to default 2022-09-12T18:45:09.894 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2022-09-12T18:45:09.895 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2022-09-12T18:45:09.896 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2022-09-12T18:45:09.924 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2022-09-12T18:45:09.924 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2022-09-12T18:45:09.991 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2022-09-12T18:45:09.992 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2022-09-12T18:45:10.061 DEBUG:teuthology.orchestra.run.smithi035:> sudo ceph --cluster c1 osd new af61f01a-7923-4ed3-9e95-2c979be1814b 0 2022-09-12T18:45:10.065 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T18:45:10.061+0000 7f267da24080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-09-12T18:45:10.221 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T18:45:10.217+0000 7f267da24080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-09-12T18:45:10.282 INFO:teuthology.orchestra.run.smithi035.stdout:0 2022-09-12T18:45:10.295 DEBUG:teuthology.orchestra.run.smithi035:> sudo ceph --cluster c1 osd new c37ec252-4e98-404a-a901-e30c04703802 1 2022-09-12T18:45:10.510 INFO:teuthology.orchestra.run.smithi035.stdout:1 2022-09-12T18:45:10.522 DEBUG:teuthology.orchestra.run.smithi035:> sudo ceph --cluster c1 osd new c0054d52-2fa8-45fc-ba5f-f344332b9a26 2 2022-09-12T18:45:10.782 INFO:teuthology.orchestra.run.smithi035.stdout:2 2022-09-12T18:45:10.783 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T18:45:10.779+0000 7f267da24080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-09-12T18:45:10.795 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2022-09-12T18:45:10.795 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2022-09-12T18:45:10.797 INFO:tasks.ceph.c1.osd.0:Started 2022-09-12T18:45:10.798 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2022-09-12T18:45:10.798 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2022-09-12T18:45:10.800 INFO:tasks.ceph.c1.osd.1:Started 2022-09-12T18:45:10.801 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2022-09-12T18:45:10.801 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2022-09-12T18:45:10.803 INFO:tasks.ceph.c1.osd.2:Started 2022-09-12T18:45:10.804 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2022-09-12T18:45:11.038 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:11.039 INFO:teuthology.orchestra.run.smithi035.stdout:{"epoch":5,"fsid":"33adcfa0-1bd1-4a60-abd9-678f3cd4b239","created":"2022-09-12T18:45:09.785367+0000","modified":"2022-09-12T18:45:10.777794+0000","last_up_change":"0.000000","last_in_change":"2022-09-12T18:45:10.777794+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"af61f01a-7923-4ed3-9e95-2c979be1814b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c37ec252-4e98-404a-a901-e30c04703802","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c0054d52-2fa8-45fc-ba5f-f344332b9a26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-12T18:45:11.052 INFO:tasks.ceph.ceph_manager.c1:[] 2022-09-12T18:45:11.052 INFO:tasks.ceph:Waiting for OSDs to come up 2022-09-12T18:45:11.193 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T18:45:11.189+0000 7f267da24080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-09-12T18:45:11.294 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T18:45:11.290+0000 7f267da24080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-09-12T18:45:11.356 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2022-09-12T18:45:11.402 INFO:tasks.ceph.c1.osd.0.smithi035.stderr:2022-09-12T18:45:11.398+0000 7f45b780a440 -1 Falling back to public interface 2022-09-12T18:45:11.426 INFO:tasks.ceph.c1.osd.1.smithi035.stderr:2022-09-12T18:45:11.422+0000 7f4c905ab440 -1 Falling back to public interface 2022-09-12T18:45:11.429 INFO:tasks.ceph.c1.osd.2.smithi035.stderr:2022-09-12T18:45:11.426+0000 7f638f0dc440 -1 Falling back to public interface 2022-09-12T18:45:11.506 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T18:45:11.502+0000 7f267da24080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-09-12T18:45:11.553 INFO:teuthology.misc.health.smithi035.stdout: 2022-09-12T18:45:11.554 INFO:teuthology.misc.health.smithi035.stdout:{"epoch":5,"fsid":"33adcfa0-1bd1-4a60-abd9-678f3cd4b239","created":"2022-09-12T18:45:09.785367+0000","modified":"2022-09-12T18:45:10.777794+0000","last_up_change":"0.000000","last_in_change":"2022-09-12T18:45:10.777794+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"af61f01a-7923-4ed3-9e95-2c979be1814b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c37ec252-4e98-404a-a901-e30c04703802","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c0054d52-2fa8-45fc-ba5f-f344332b9a26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-12T18:45:11.567 DEBUG:teuthology.misc:0 of 3 OSDs are up 2022-09-12T18:45:12.504 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T18:45:12.500+0000 7f267da24080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-09-12T18:45:12.583 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T18:45:12.579+0000 7f267da24080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-09-12T18:45:12.681 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T18:45:12.677+0000 7f267da24080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-09-12T18:45:12.763 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T18:45:12.758+0000 7f267da24080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-09-12T18:45:13.064 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T18:45:13.060+0000 7f267da24080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-09-12T18:45:13.139 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T18:45:13.135+0000 7f267da24080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-09-12T18:45:13.214 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T18:45:13.210+0000 7f267da24080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-09-12T18:45:13.867 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T18:45:13.863+0000 7f267da24080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-09-12T18:45:14.010 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T18:45:14.006+0000 7f267da24080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-09-12T18:45:14.122 INFO:tasks.ceph.c1.osd.0.smithi035.stderr:2022-09-12T18:45:14.118+0000 7f45b780a440 -1 osd.0 0 log_to_monitors true 2022-09-12T18:45:14.137 INFO:tasks.ceph.c1.osd.1.smithi035.stderr:2022-09-12T18:45:14.133+0000 7f4c905ab440 -1 osd.1 0 log_to_monitors true 2022-09-12T18:45:14.160 INFO:tasks.ceph.c1.osd.2.smithi035.stderr:2022-09-12T18:45:14.156+0000 7f638f0dc440 -1 osd.2 0 log_to_monitors true 2022-09-12T18:45:14.174 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T18:45:14.170+0000 7f267da24080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-09-12T18:45:14.250 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T18:45:14.246+0000 7f267da24080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-09-12T18:45:14.387 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T18:45:14.383+0000 7f267da24080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-09-12T18:45:14.487 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T18:45:14.483+0000 7f267da24080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-09-12T18:45:14.746 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T18:45:14.742+0000 7f267da24080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-09-12T18:45:15.026 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T18:45:15.022+0000 7f267da24080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-09-12T18:45:15.111 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T18:45:15.107+0000 7f267da24080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-09-12T18:45:15.308 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T18:45:15.304+0000 7f267da24080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-09-12T18:45:15.497 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T18:45:15.493+0000 7f267da24080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-09-12T18:45:16.312 INFO:tasks.ceph.c1.osd.2.smithi035.stderr:2022-09-12T18:45:16.308+0000 7f6382e89700 -1 osd.2 0 waiting for initial osdmap 2022-09-12T18:45:16.322 INFO:tasks.ceph.c1.osd.2.smithi035.stderr:2022-09-12T18:45:16.318+0000 7f637b264700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-12T18:45:16.323 INFO:tasks.ceph.c1.osd.1.smithi035.stderr:2022-09-12T18:45:16.319+0000 7f4c84358700 -1 osd.1 0 waiting for initial osdmap 2022-09-12T18:45:16.326 INFO:tasks.ceph.c1.osd.0.smithi035.stderr:2022-09-12T18:45:16.322+0000 7f45ab5b7700 -1 osd.0 0 waiting for initial osdmap 2022-09-12T18:45:16.333 INFO:tasks.ceph.c1.osd.1.smithi035.stderr:2022-09-12T18:45:16.329+0000 7f4c7c733700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-12T18:45:16.337 INFO:tasks.ceph.c1.osd.0.smithi035.stderr:2022-09-12T18:45:16.333+0000 7f45a3992700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-12T18:45:16.598 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T18:45:16.594+0000 7f263c7dd700 -1 mgr.server handle_report got status from non-daemon mon.a 2022-09-12T18:45:17.870 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2022-09-12T18:45:18.195 INFO:teuthology.misc.health.smithi035.stdout: 2022-09-12T18:45:18.195 INFO:teuthology.misc.health.smithi035.stdout:{"epoch":8,"fsid":"33adcfa0-1bd1-4a60-abd9-678f3cd4b239","created":"2022-09-12T18:45:09.785367+0000","modified":"2022-09-12T18:45:17.324284+0000","last_up_change":"2022-09-12T18:45:17.324284+0000","last_in_change":"2022-09-12T18:45:10.777794+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"af61f01a-7923-4ed3-9e95-2c979be1814b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6800","nonce":101700},{"type":"v1","addr":"172.21.15.35:6801","nonce":101700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6802","nonce":101700},{"type":"v1","addr":"172.21.15.35:6803","nonce":101700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6806","nonce":101700},{"type":"v1","addr":"172.21.15.35:6807","nonce":101700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6804","nonce":101700},{"type":"v1","addr":"172.21.15.35:6805","nonce":101700}]},"public_addr":"172.21.15.35:6801/101700","cluster_addr":"172.21.15.35:6803/101700","heartbeat_back_addr":"172.21.15.35:6807/101700","heartbeat_front_addr":"172.21.15.35:6805/101700","state":["exists","up"]},{"osd":1,"uuid":"c37ec252-4e98-404a-a901-e30c04703802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6808","nonce":101701},{"type":"v1","addr":"172.21.15.35:6809","nonce":101701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6810","nonce":101701},{"type":"v1","addr":"172.21.15.35:6811","nonce":101701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6814","nonce":101701},{"type":"v1","addr":"172.21.15.35:6815","nonce":101701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6812","nonce":101701},{"type":"v1","addr":"172.21.15.35:6813","nonce":101701}]},"public_addr":"172.21.15.35:6809/101701","cluster_addr":"172.21.15.35:6811/101701","heartbeat_back_addr":"172.21.15.35:6815/101701","heartbeat_front_addr":"172.21.15.35:6813/101701","state":["exists","up"]},{"osd":2,"uuid":"c0054d52-2fa8-45fc-ba5f-f344332b9a26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6816","nonce":101716},{"type":"v1","addr":"172.21.15.35:6817","nonce":101716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6818","nonce":101716},{"type":"v1","addr":"172.21.15.35:6819","nonce":101716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6822","nonce":101716},{"type":"v1","addr":"172.21.15.35:6823","nonce":101716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6820","nonce":101716},{"type":"v1","addr":"172.21.15.35:6821","nonce":101716}]},"public_addr":"172.21.15.35:6817/101716","cluster_addr":"172.21.15.35:6819/101716","heartbeat_back_addr":"172.21.15.35:6823/101716","heartbeat_front_addr":"172.21.15.35:6821/101716","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-12T18:45:18.208 DEBUG:teuthology.misc:3 of 3 OSDs are up 2022-09-12T18:45:18.208 INFO:tasks.ceph:Creating RBD pool 2022-09-12T18:45:18.209 DEBUG:teuthology.orchestra.run.smithi035:> sudo ceph --cluster c1 osd pool create rbd 8 2022-09-12T18:45:18.604 INFO:teuthology.orchestra.run.smithi035.stderr:pool 'rbd' created 2022-09-12T18:45:18.617 DEBUG:teuthology.orchestra.run.smithi035:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2022-09-12T18:45:19.606 INFO:teuthology.orchestra.run.smithi035.stderr:enabled application 'rbd' on pool 'rbd' 2022-09-12T18:45:19.623 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2022-09-12T18:45:19.623 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2022-09-12T18:45:19.624 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2022-09-12T18:45:19.624 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2022-09-12T18:45:19.625 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2022-09-12T18:45:20.029 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:20.043 INFO:teuthology.orchestra.run.smithi035.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6824","nonce":101285},{"type":"v1","addr":"172.21.15.35:6825","nonce":101285}]},"active_addr":"172.21.15.35:6825/101285","active_change":"2022-09-12T18:45:15.507261+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.35:0","nonce":617558040}]},{"addrvec":[{"type":"v2","addr":"172.21.15.35:0","nonce":2774318486}]},{"addrvec":[{"type":"v2","addr":"172.21.15.35:0","nonce":257585633}]}]}} 2022-09-12T18:45:20.051 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2022-09-12T18:45:20.052 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2022-09-12T18:45:20.053 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2022-09-12T18:45:20.379 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:20.380 INFO:teuthology.orchestra.run.smithi035.stdout:{"epoch":10,"fsid":"33adcfa0-1bd1-4a60-abd9-678f3cd4b239","created":"2022-09-12T18:45:09.785367+0000","modified":"2022-09-12T18:45:19.595773+0000","last_up_change":"2022-09-12T18:45:17.324284+0000","last_in_change":"2022-09-12T18:45:10.777794+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-12T18:45:17.648292+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-09-12T18:45:18.554385+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"af61f01a-7923-4ed3-9e95-2c979be1814b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6800","nonce":101700},{"type":"v1","addr":"172.21.15.35:6801","nonce":101700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6802","nonce":101700},{"type":"v1","addr":"172.21.15.35:6803","nonce":101700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6806","nonce":101700},{"type":"v1","addr":"172.21.15.35:6807","nonce":101700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6804","nonce":101700},{"type":"v1","addr":"172.21.15.35:6805","nonce":101700}]},"public_addr":"172.21.15.35:6801/101700","cluster_addr":"172.21.15.35:6803/101700","heartbeat_back_addr":"172.21.15.35:6807/101700","heartbeat_front_addr":"172.21.15.35:6805/101700","state":["exists","up"]},{"osd":1,"uuid":"c37ec252-4e98-404a-a901-e30c04703802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6808","nonce":101701},{"type":"v1","addr":"172.21.15.35:6809","nonce":101701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6810","nonce":101701},{"type":"v1","addr":"172.21.15.35:6811","nonce":101701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6814","nonce":101701},{"type":"v1","addr":"172.21.15.35:6815","nonce":101701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6812","nonce":101701},{"type":"v1","addr":"172.21.15.35:6813","nonce":101701}]},"public_addr":"172.21.15.35:6809/101701","cluster_addr":"172.21.15.35:6811/101701","heartbeat_back_addr":"172.21.15.35:6815/101701","heartbeat_front_addr":"172.21.15.35:6813/101701","state":["exists","up"]},{"osd":2,"uuid":"c0054d52-2fa8-45fc-ba5f-f344332b9a26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6816","nonce":101716},{"type":"v1","addr":"172.21.15.35:6817","nonce":101716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6818","nonce":101716},{"type":"v1","addr":"172.21.15.35:6819","nonce":101716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6822","nonce":101716},{"type":"v1","addr":"172.21.15.35:6823","nonce":101716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6820","nonce":101716},{"type":"v1","addr":"172.21.15.35:6821","nonce":101716}]},"public_addr":"172.21.15.35:6817/101716","cluster_addr":"172.21.15.35:6819/101716","heartbeat_back_addr":"172.21.15.35:6823/101716","heartbeat_front_addr":"172.21.15.35:6821/101716","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-12T18:45:15.096877+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-12T18:45:15.147187+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-12T18:45:15.120712+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-12T18:45:20.394 INFO:tasks.ceph.ceph_manager.c1:all up! 2022-09-12T18:45:20.395 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2022-09-12T18:45:20.719 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:20.719 INFO:teuthology.orchestra.run.smithi035.stdout:{"epoch":11,"fsid":"33adcfa0-1bd1-4a60-abd9-678f3cd4b239","created":"2022-09-12T18:45:09.785367+0000","modified":"2022-09-12T18:45:20.596788+0000","last_up_change":"2022-09-12T18:45:17.324284+0000","last_in_change":"2022-09-12T18:45:10.777794+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-12T18:45:17.648292+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-09-12T18:45:18.554385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"af61f01a-7923-4ed3-9e95-2c979be1814b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6800","nonce":101700},{"type":"v1","addr":"172.21.15.35:6801","nonce":101700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6802","nonce":101700},{"type":"v1","addr":"172.21.15.35:6803","nonce":101700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6806","nonce":101700},{"type":"v1","addr":"172.21.15.35:6807","nonce":101700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6804","nonce":101700},{"type":"v1","addr":"172.21.15.35:6805","nonce":101700}]},"public_addr":"172.21.15.35:6801/101700","cluster_addr":"172.21.15.35:6803/101700","heartbeat_back_addr":"172.21.15.35:6807/101700","heartbeat_front_addr":"172.21.15.35:6805/101700","state":["exists","up"]},{"osd":1,"uuid":"c37ec252-4e98-404a-a901-e30c04703802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6808","nonce":101701},{"type":"v1","addr":"172.21.15.35:6809","nonce":101701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6810","nonce":101701},{"type":"v1","addr":"172.21.15.35:6811","nonce":101701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6814","nonce":101701},{"type":"v1","addr":"172.21.15.35:6815","nonce":101701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6812","nonce":101701},{"type":"v1","addr":"172.21.15.35:6813","nonce":101701}]},"public_addr":"172.21.15.35:6809/101701","cluster_addr":"172.21.15.35:6811/101701","heartbeat_back_addr":"172.21.15.35:6815/101701","heartbeat_front_addr":"172.21.15.35:6813/101701","state":["exists","up"]},{"osd":2,"uuid":"c0054d52-2fa8-45fc-ba5f-f344332b9a26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6816","nonce":101716},{"type":"v1","addr":"172.21.15.35:6817","nonce":101716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6818","nonce":101716},{"type":"v1","addr":"172.21.15.35:6819","nonce":101716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6822","nonce":101716},{"type":"v1","addr":"172.21.15.35:6823","nonce":101716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.35:6820","nonce":101716},{"type":"v1","addr":"172.21.15.35:6821","nonce":101716}]},"public_addr":"172.21.15.35:6817/101716","cluster_addr":"172.21.15.35:6819/101716","heartbeat_back_addr":"172.21.15.35:6823/101716","heartbeat_front_addr":"172.21.15.35:6821/101716","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-12T18:45:15.096877+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-12T18:45:15.147187+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-12T18:45:15.120712+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-12T18:45:20.733 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2022-09-12T18:45:20.734 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2022-09-12T18:45:20.735 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2022-09-12T18:45:20.903 INFO:teuthology.orchestra.run.smithi035.stdout:34359738370 2022-09-12T18:45:20.903 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2022-09-12T18:45:20.904 INFO:teuthology.orchestra.run.smithi035.stdout:34359738370 2022-09-12T18:45:20.905 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2022-09-12T18:45:20.906 INFO:teuthology.orchestra.run.smithi035.stdout:34359738370 2022-09-12T18:45:20.907 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2022-09-12T18:45:21.271 INFO:teuthology.orchestra.run.smithi035.stdout:34359738369 2022-09-12T18:45:21.286 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.2 2022-09-12T18:45:21.288 INFO:teuthology.orchestra.run.smithi035.stdout:34359738369 2022-09-12T18:45:21.297 INFO:teuthology.orchestra.run.smithi035.stdout:34359738369 2022-09-12T18:45:21.302 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.1 2022-09-12T18:45:21.311 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.0 2022-09-12T18:45:22.287 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2022-09-12T18:45:22.303 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2022-09-12T18:45:22.312 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2022-09-12T18:45:22.651 INFO:teuthology.orchestra.run.smithi035.stdout:34359738370 2022-09-12T18:45:22.670 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.2 2022-09-12T18:45:22.670 DEBUG:teuthology.parallel:result is None 2022-09-12T18:45:22.683 INFO:teuthology.orchestra.run.smithi035.stdout:34359738370 2022-09-12T18:45:22.686 INFO:teuthology.orchestra.run.smithi035.stdout:34359738370 2022-09-12T18:45:22.698 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.0 2022-09-12T18:45:22.698 DEBUG:teuthology.parallel:result is None 2022-09-12T18:45:22.701 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.1 2022-09-12T18:45:22.702 DEBUG:teuthology.parallel:result is None 2022-09-12T18:45:22.702 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2022-09-12T18:45:22.702 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2022-09-12T18:45:23.057 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:23.059 INFO:teuthology.orchestra.run.smithi035.stderr:dumped all 2022-09-12T18:45:23.072 INFO:teuthology.orchestra.run.smithi035.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2022-09-12T18:45:21.593099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":25556,"kb_used_data":1416,"kb_used_omap":0,"kb_used_meta":23296,"kb_avail":314547244,"statfs":{"total":322122547200,"available":322096377856,"internally_reserved":0,"allocated":1449984,"data_stored":1245886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23855104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-12T18:45:20.601696+0000","last_change":"2022-09-12T18:45:19.605796+0000","last_active":"2022-09-12T18:45:20.601696+0000","last_peered":"2022-09-12T18:45:20.601696+0000","last_clean":"2022-09-12T18:45:20.601696+0000","last_became_active":"2022-09-12T18:45:19.605539+0000","last_became_peered":"2022-09-12T18:45:19.605539+0000","last_unstale":"2022-09-12T18:45:20.601696+0000","last_undegraded":"2022-09-12T18:45:20.601696+0000","last_fullsized":"2022-09-12T18:45:20.601696+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_clean_scrub_stamp":"2022-09-12T18:45:18.595121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T00:21:13.771585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-12T18:45:20.601566+0000","last_change":"2022-09-12T18:45:19.607654+0000","last_active":"2022-09-12T18:45:20.601566+0000","last_peered":"2022-09-12T18:45:20.601566+0000","last_clean":"2022-09-12T18:45:20.601566+0000","last_became_active":"2022-09-12T18:45:19.607389+0000","last_became_peered":"2022-09-12T18:45:19.607389+0000","last_unstale":"2022-09-12T18:45:20.601566+0000","last_undegraded":"2022-09-12T18:45:20.601566+0000","last_fullsized":"2022-09-12T18:45:20.601566+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_clean_scrub_stamp":"2022-09-12T18:45:18.595121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T04:08:17.404998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-12T18:45:20.601696+0000","last_change":"2022-09-12T18:45:19.607667+0000","last_active":"2022-09-12T18:45:20.601696+0000","last_peered":"2022-09-12T18:45:20.601696+0000","last_clean":"2022-09-12T18:45:20.601696+0000","last_became_active":"2022-09-12T18:45:19.607389+0000","last_became_peered":"2022-09-12T18:45:19.607389+0000","last_unstale":"2022-09-12T18:45:20.601696+0000","last_undegraded":"2022-09-12T18:45:20.601696+0000","last_fullsized":"2022-09-12T18:45:20.601696+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_clean_scrub_stamp":"2022-09-12T18:45:18.595121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T03:00:00.631844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2022-09-12T18:45:19.608151+0000","last_change":"2022-09-12T18:45:19.603619+0000","last_active":"2022-09-12T18:45:19.608151+0000","last_peered":"2022-09-12T18:45:19.608151+0000","last_clean":"2022-09-12T18:45:19.608151+0000","last_became_active":"2022-09-12T18:45:19.603099+0000","last_became_peered":"2022-09-12T18:45:19.603099+0000","last_unstale":"2022-09-12T18:45:19.608151+0000","last_undegraded":"2022-09-12T18:45:19.608151+0000","last_fullsized":"2022-09-12T18:45:19.608151+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_clean_scrub_stamp":"2022-09-12T18:45:18.595121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-13T21:23:19.506927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-12T18:45:20.601441+0000","last_change":"2022-09-12T18:45:19.606756+0000","last_active":"2022-09-12T18:45:20.601441+0000","last_peered":"2022-09-12T18:45:20.601441+0000","last_clean":"2022-09-12T18:45:20.601441+0000","last_became_active":"2022-09-12T18:45:19.605457+0000","last_became_peered":"2022-09-12T18:45:19.605457+0000","last_unstale":"2022-09-12T18:45:20.601441+0000","last_undegraded":"2022-09-12T18:45:20.601441+0000","last_fullsized":"2022-09-12T18:45:20.601441+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_clean_scrub_stamp":"2022-09-12T18:45:18.595121+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-13T20:47:34.099210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-12T18:45:20.601450+0000","last_change":"2022-09-12T18:45:19.606646+0000","last_active":"2022-09-12T18:45:20.601450+0000","last_peered":"2022-09-12T18:45:20.601450+0000","last_clean":"2022-09-12T18:45:20.601450+0000","last_became_active":"2022-09-12T18:45:19.606342+0000","last_became_peered":"2022-09-12T18:45:19.606342+0000","last_unstale":"2022-09-12T18:45:20.601450+0000","last_undegraded":"2022-09-12T18:45:20.601450+0000","last_fullsized":"2022-09-12T18:45:20.601450+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_clean_scrub_stamp":"2022-09-12T18:45:18.595121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T02:15:12.380381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2022-09-12T18:45:19.610084+0000","last_change":"2022-09-12T18:45:19.605797+0000","last_active":"2022-09-12T18:45:19.610084+0000","last_peered":"2022-09-12T18:45:19.610084+0000","last_clean":"2022-09-12T18:45:19.610084+0000","last_became_active":"2022-09-12T18:45:19.605531+0000","last_became_peered":"2022-09-12T18:45:19.605531+0000","last_unstale":"2022-09-12T18:45:19.610084+0000","last_undegraded":"2022-09-12T18:45:19.610084+0000","last_fullsized":"2022-09-12T18:45:19.610084+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_clean_scrub_stamp":"2022-09-12T18:45:18.595121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-13T20:39:07.630865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2022-09-12T18:45:19.608008+0000","last_change":"2022-09-12T18:45:19.603318+0000","last_active":"2022-09-12T18:45:19.608008+0000","last_peered":"2022-09-12T18:45:19.608008+0000","last_clean":"2022-09-12T18:45:19.608008+0000","last_became_active":"2022-09-12T18:45:19.603046+0000","last_became_peered":"2022-09-12T18:45:19.603046+0000","last_unstale":"2022-09-12T18:45:19.608008+0000","last_undegraded":"2022-09-12T18:45:19.608008+0000","last_fullsized":"2022-09-12T18:45:19.608008+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_clean_scrub_stamp":"2022-09-12T18:45:18.595121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T00:42:06.817595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-12T18:45:20.601744+0000","last_change":"2022-09-12T18:45:19.605791+0000","last_active":"2022-09-12T18:45:20.601744+0000","last_peered":"2022-09-12T18:45:20.601744+0000","last_clean":"2022-09-12T18:45:20.601744+0000","last_became_active":"2022-09-12T18:45:19.605487+0000","last_became_peered":"2022-09-12T18:45:19.605487+0000","last_unstale":"2022-09-12T18:45:20.601744+0000","last_undegraded":"2022-09-12T18:45:20.601744+0000","last_fullsized":"2022-09-12T18:45:20.601744+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_clean_scrub_stamp":"2022-09-12T18:45:18.595121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-13T19:22:14.136836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7580,"kb_used_data":84,"kb_used_omap":0,"kb_used_meta":7488,"kb_avail":104850020,"statfs":{"total":107374182400,"available":107366420480,"internally_reserved":0,"allocated":86016,"data_stored":21349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7667712},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":9112,"kb_used_data":664,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848488,"statfs":{"total":107374182400,"available":107364851712,"internally_reserved":0,"allocated":679936,"data_stored":611717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8864,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104848736,"statfs":{"total":107374182400,"available":107365105664,"internally_reserved":0,"allocated":684032,"data_stored":612820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-09-12T18:45:23.073 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2022-09-12T18:45:23.389 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:23.392 INFO:teuthology.orchestra.run.smithi035.stderr:dumped all 2022-09-12T18:45:23.403 INFO:teuthology.orchestra.run.smithi035.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2022-09-12T18:45:21.593099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":25556,"kb_used_data":1416,"kb_used_omap":0,"kb_used_meta":23296,"kb_avail":314547244,"statfs":{"total":322122547200,"available":322096377856,"internally_reserved":0,"allocated":1449984,"data_stored":1245886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23855104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-12T18:45:20.601696+0000","last_change":"2022-09-12T18:45:19.605796+0000","last_active":"2022-09-12T18:45:20.601696+0000","last_peered":"2022-09-12T18:45:20.601696+0000","last_clean":"2022-09-12T18:45:20.601696+0000","last_became_active":"2022-09-12T18:45:19.605539+0000","last_became_peered":"2022-09-12T18:45:19.605539+0000","last_unstale":"2022-09-12T18:45:20.601696+0000","last_undegraded":"2022-09-12T18:45:20.601696+0000","last_fullsized":"2022-09-12T18:45:20.601696+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_clean_scrub_stamp":"2022-09-12T18:45:18.595121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T00:21:13.771585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-12T18:45:20.601566+0000","last_change":"2022-09-12T18:45:19.607654+0000","last_active":"2022-09-12T18:45:20.601566+0000","last_peered":"2022-09-12T18:45:20.601566+0000","last_clean":"2022-09-12T18:45:20.601566+0000","last_became_active":"2022-09-12T18:45:19.607389+0000","last_became_peered":"2022-09-12T18:45:19.607389+0000","last_unstale":"2022-09-12T18:45:20.601566+0000","last_undegraded":"2022-09-12T18:45:20.601566+0000","last_fullsized":"2022-09-12T18:45:20.601566+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_clean_scrub_stamp":"2022-09-12T18:45:18.595121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T04:08:17.404998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-12T18:45:20.601696+0000","last_change":"2022-09-12T18:45:19.607667+0000","last_active":"2022-09-12T18:45:20.601696+0000","last_peered":"2022-09-12T18:45:20.601696+0000","last_clean":"2022-09-12T18:45:20.601696+0000","last_became_active":"2022-09-12T18:45:19.607389+0000","last_became_peered":"2022-09-12T18:45:19.607389+0000","last_unstale":"2022-09-12T18:45:20.601696+0000","last_undegraded":"2022-09-12T18:45:20.601696+0000","last_fullsized":"2022-09-12T18:45:20.601696+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_clean_scrub_stamp":"2022-09-12T18:45:18.595121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T03:00:00.631844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2022-09-12T18:45:19.608151+0000","last_change":"2022-09-12T18:45:19.603619+0000","last_active":"2022-09-12T18:45:19.608151+0000","last_peered":"2022-09-12T18:45:19.608151+0000","last_clean":"2022-09-12T18:45:19.608151+0000","last_became_active":"2022-09-12T18:45:19.603099+0000","last_became_peered":"2022-09-12T18:45:19.603099+0000","last_unstale":"2022-09-12T18:45:19.608151+0000","last_undegraded":"2022-09-12T18:45:19.608151+0000","last_fullsized":"2022-09-12T18:45:19.608151+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_clean_scrub_stamp":"2022-09-12T18:45:18.595121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-13T21:23:19.506927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-12T18:45:20.601441+0000","last_change":"2022-09-12T18:45:19.606756+0000","last_active":"2022-09-12T18:45:20.601441+0000","last_peered":"2022-09-12T18:45:20.601441+0000","last_clean":"2022-09-12T18:45:20.601441+0000","last_became_active":"2022-09-12T18:45:19.605457+0000","last_became_peered":"2022-09-12T18:45:19.605457+0000","last_unstale":"2022-09-12T18:45:20.601441+0000","last_undegraded":"2022-09-12T18:45:20.601441+0000","last_fullsized":"2022-09-12T18:45:20.601441+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_clean_scrub_stamp":"2022-09-12T18:45:18.595121+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-13T20:47:34.099210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-12T18:45:20.601450+0000","last_change":"2022-09-12T18:45:19.606646+0000","last_active":"2022-09-12T18:45:20.601450+0000","last_peered":"2022-09-12T18:45:20.601450+0000","last_clean":"2022-09-12T18:45:20.601450+0000","last_became_active":"2022-09-12T18:45:19.606342+0000","last_became_peered":"2022-09-12T18:45:19.606342+0000","last_unstale":"2022-09-12T18:45:20.601450+0000","last_undegraded":"2022-09-12T18:45:20.601450+0000","last_fullsized":"2022-09-12T18:45:20.601450+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_clean_scrub_stamp":"2022-09-12T18:45:18.595121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T02:15:12.380381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2022-09-12T18:45:19.610084+0000","last_change":"2022-09-12T18:45:19.605797+0000","last_active":"2022-09-12T18:45:19.610084+0000","last_peered":"2022-09-12T18:45:19.610084+0000","last_clean":"2022-09-12T18:45:19.610084+0000","last_became_active":"2022-09-12T18:45:19.605531+0000","last_became_peered":"2022-09-12T18:45:19.605531+0000","last_unstale":"2022-09-12T18:45:19.610084+0000","last_undegraded":"2022-09-12T18:45:19.610084+0000","last_fullsized":"2022-09-12T18:45:19.610084+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_clean_scrub_stamp":"2022-09-12T18:45:18.595121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-13T20:39:07.630865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2022-09-12T18:45:19.608008+0000","last_change":"2022-09-12T18:45:19.603318+0000","last_active":"2022-09-12T18:45:19.608008+0000","last_peered":"2022-09-12T18:45:19.608008+0000","last_clean":"2022-09-12T18:45:19.608008+0000","last_became_active":"2022-09-12T18:45:19.603046+0000","last_became_peered":"2022-09-12T18:45:19.603046+0000","last_unstale":"2022-09-12T18:45:19.608008+0000","last_undegraded":"2022-09-12T18:45:19.608008+0000","last_fullsized":"2022-09-12T18:45:19.608008+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_clean_scrub_stamp":"2022-09-12T18:45:18.595121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T00:42:06.817595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-12T18:45:20.601744+0000","last_change":"2022-09-12T18:45:19.605791+0000","last_active":"2022-09-12T18:45:20.601744+0000","last_peered":"2022-09-12T18:45:20.601744+0000","last_clean":"2022-09-12T18:45:20.601744+0000","last_became_active":"2022-09-12T18:45:19.605487+0000","last_became_peered":"2022-09-12T18:45:19.605487+0000","last_unstale":"2022-09-12T18:45:20.601744+0000","last_undegraded":"2022-09-12T18:45:20.601744+0000","last_fullsized":"2022-09-12T18:45:20.601744+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:18.595121+0000","last_clean_scrub_stamp":"2022-09-12T18:45:18.595121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-13T19:22:14.136836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7580,"kb_used_data":84,"kb_used_omap":0,"kb_used_meta":7488,"kb_avail":104850020,"statfs":{"total":107374182400,"available":107366420480,"internally_reserved":0,"allocated":86016,"data_stored":21349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7667712},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":9112,"kb_used_data":664,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848488,"statfs":{"total":107374182400,"available":107364851712,"internally_reserved":0,"allocated":679936,"data_stored":611717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8864,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104848736,"statfs":{"total":107374182400,"available":107365105664,"internally_reserved":0,"allocated":684032,"data_stored":612820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-09-12T18:45:23.404 INFO:tasks.ceph.ceph_manager.c1:clean! 2022-09-12T18:45:23.404 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2022-09-12T18:45:23.405 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2022-09-12T18:45:23.405 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2022-09-12T18:45:23.766 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:23.767 INFO:teuthology.orchestra.run.smithi035.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-09-12T18:45:23.780 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2022-09-12T18:45:23.780 INFO:teuthology.run_tasks:Running task ceph... 2022-09-12T18:45:23.789 INFO:tasks.ceph:Creating ceph cluster c2... 2022-09-12T18:45:23.790 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2022-09-12T18:45:23.790 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2022-09-11_20:31:56-rgw-main-distro-default-smithi/7026931', 'branch': 'main', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi}', 'email': 'ceph-qa@ceph.io', 'first_in_suite': False, 'job_id': '7026931', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2022-09-11_20:31:56-rgw-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '10f7d25d6a0f340676cd251eab5a787ba01ace95', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '10f7d25d6a0f340676cd251eab5a787ba01ace95'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py', 'tests_ps.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'main', 'sha1': '10f7d25d6a0f340676cd251eab5a787ba01ace95'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 71, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '10f7d25d6a0f340676cd251eab5a787ba01ace95', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_10f7d25d6a0f340676cd251eab5a787ba01ace95/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '10f7d25d6a0f340676cd251eab5a787ba01ace95', 'targets': {'smithi035.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDhDcIgHo8yPwbEK2iHZHrqUzB4APgwzUj7VVWbHPtCAGblmY3il1ZTa06qXVADIwzGqmBDnClFbNh9Xl9ezouikXdGUn6gH+058Uc5DmuDveEWjx9ZG8ypl7K1NI6mA2ZF4ZhavIk1Mece12gvaQfSuc3aaBvpGU5M5Vzoe24AXLIZ9eOtsHpWz7KJ3tCLONBRtUUzCF4sj2xG1cclnzJEmD4DjOXa6CL0plEsq6miQhtw+c9AsYS/ZIZwGHIVOFuLdIASDixTFS74uuJaVt6hzwpRbjdRvfhRB4EeiiOFF9HRleD6wEIBxyzbNjqBeKeP6AYr74v+0E9K68iOMp54q2xxkmWFnhvsuVYTyYmU+A1VqJMRLuNfJBYyB0C2MxvcuXzFweggp7mbybUuhOKpgNAzBliyM7GY/VDZheHJInzFHLRTVY9XsJklqPugCYycox6C/YAlh0Q6pwQ0Sr2WaEMz6PAyG9t+eK+pZAaTXwux1fGSsfpah46Vmy9ngOM=', 'smithi138.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpUYYo//MHC5osesSZpvrMfoP5AJD18lKMj30OB4QvSUyQC7Nwq1VJzA3Q71fceJFqHa2/LJAxGfhT297bQk8IsAh6ewsfysQhYwYOC4DhWDuuP3m6O0v15ME6VtShlwvk9sowvjf1f/DcyDWr2jVtN91ZeHVCjSyFMU5Qvr7BnKk/rDp/AnJwxKrKHE5OhJLicQ/OWLs6Bc4lreV5M0zIxxJMalQYeL5VnLzDaPYQ6iuDbp/AYbhIPcjrQfp/QFcreVGJxsW3mRwbOpo9/2YYTOUisbXmboaxY+FU4M81ehTFntB9X5PxcMrOe3PX6MGzn9XSiRB/sTkH63R6TY5MRC/qd6J2XymC9/llkLMhNiIoKMPwm6zAOEUfDQc8U2gDDjPcqeIXMAfswyMo4M6iAyM/Wglt5+CE3olTSfUfdRiTaBIYH5NJPsQLYSdwYlUwBVLJau4+gdzdIfJwP3n5Oihatqx2aOOmiy8FIMHnsrekeQcZXS+/vxqNnFcc2OE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '10f7d25d6a0f340676cd251eab5a787ba01ace95', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '10f7d25d6a0f340676cd251eab5a787ba01ace95', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b58f2c18636eb10faa77ed3614abd00cb85dfc2c', 'timestamp': '2022-09-11_20:31:56', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.14413'} 2022-09-12T18:45:23.791 DEBUG:teuthology.orchestra.run.smithi035:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2022-09-12T18:45:23.793 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2022-09-12T18:45:23.811 DEBUG:teuthology.orchestra.run.smithi035:> sudo install -d -m0777 -- /var/run/ceph 2022-09-12T18:45:23.851 DEBUG:teuthology.orchestra.run.smithi138:> sudo install -d -m0777 -- /var/run/ceph 2022-09-12T18:45:23.878 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-09-12T18:45:23.879 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/scratch_devs of=/dev/stdout 2022-09-12T18:45:23.937 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-09-12T18:45:23.938 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-09-12T18:45:23.995 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-09-12T18:45:23.995 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-12T18:45:23.996 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 6h/6d Inode: 360906 Links: 1 2022-09-12T18:45:23.996 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-12T18:45:23.996 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2022-09-12T18:45:23.996 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2022-09-12 18:44:44.826681137 +0000 2022-09-12T18:45:23.997 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2022-09-12 18:41:18.928651368 +0000 2022-09-12T18:45:23.997 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2022-09-12 18:41:18.928651368 +0000 2022-09-12T18:45:23.997 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2022-09-12T18:45:24.004 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2022-09-12T18:45:24.005 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2022-09-12T18:45:24.005 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000253159 s, 2.0 MB/s 2022-09-12T18:45:24.007 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-09-12T18:45:24.065 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-09-12T18:45:24.065 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-12T18:45:24.065 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 6h/6d Inode: 366645 Links: 1 2022-09-12T18:45:24.066 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-12T18:45:24.066 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2022-09-12T18:45:24.066 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2022-09-12 18:44:44.826681137 +0000 2022-09-12T18:45:24.066 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2022-09-12 18:41:19.474639234 +0000 2022-09-12T18:45:24.067 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2022-09-12 18:41:19.474639234 +0000 2022-09-12T18:45:24.067 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2022-09-12T18:45:24.074 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2022-09-12T18:45:24.074 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2022-09-12T18:45:24.075 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000295133 s, 1.7 MB/s 2022-09-12T18:45:24.077 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-09-12T18:45:24.135 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-09-12T18:45:24.135 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-12T18:45:24.136 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 6h/6d Inode: 354086 Links: 1 2022-09-12T18:45:24.136 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-12T18:45:24.136 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2022-09-12T18:45:24.137 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2022-09-12 18:44:44.826681137 +0000 2022-09-12T18:45:24.137 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2022-09-12 18:41:20.044626566 +0000 2022-09-12T18:45:24.137 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2022-09-12 18:41:20.044626566 +0000 2022-09-12T18:45:24.137 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2022-09-12T18:45:24.144 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2022-09-12T18:45:24.145 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2022-09-12T18:45:24.145 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000194106 s, 2.6 MB/s 2022-09-12T18:45:24.147 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-09-12T18:45:24.205 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-09-12T18:45:24.205 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-12T18:45:24.205 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 6h/6d Inode: 368679 Links: 1 2022-09-12T18:45:24.205 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-12T18:45:24.206 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2022-09-12T18:45:24.206 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2022-09-12 18:44:44.826681137 +0000 2022-09-12T18:45:24.206 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2022-09-12 18:41:20.584614565 +0000 2022-09-12T18:45:24.206 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2022-09-12 18:41:20.584614565 +0000 2022-09-12T18:45:24.207 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2022-09-12T18:45:24.214 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2022-09-12T18:45:24.214 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2022-09-12T18:45:24.215 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000193467 s, 2.6 MB/s 2022-09-12T18:45:24.217 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-12T18:45:24.217 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi138.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2022-09-12T18:45:24.217 INFO:tasks.ceph:Generating config... 2022-09-12T18:45:24.219 INFO:tasks.ceph:[client] debug rgw = 20 2022-09-12T18:45:24.219 INFO:tasks.ceph:[client] debug rgw sync = 20 2022-09-12T18:45:24.219 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2022-09-12T18:45:24.220 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2022-09-12T18:45:24.220 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-09-12T18:45:24.220 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2022-09-12T18:45:24.221 INFO:tasks.ceph:[client] rgw data log num shards = 4 2022-09-12T18:45:24.221 INFO:tasks.ceph:[client] rgw md log max shards = 4 2022-09-12T18:45:24.221 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2022-09-12T18:45:24.222 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2022-09-12T18:45:24.222 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2022-09-12T18:45:24.222 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2022-09-12T18:45:24.222 INFO:tasks.ceph:[client] setgroup = ceph 2022-09-12T18:45:24.223 INFO:tasks.ceph:[client] setuser = ceph 2022-09-12T18:45:24.223 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2022-09-12T18:45:24.223 INFO:tasks.ceph:[mgr] debug mgr = 20 2022-09-12T18:45:24.224 INFO:tasks.ceph:[mgr] debug ms = 1 2022-09-12T18:45:24.224 INFO:tasks.ceph:[mon] debug mon = 20 2022-09-12T18:45:24.224 INFO:tasks.ceph:[mon] debug ms = 1 2022-09-12T18:45:24.225 INFO:tasks.ceph:[mon] debug paxos = 20 2022-09-12T18:45:24.225 INFO:tasks.ceph:[osd] debug ms = 1 2022-09-12T18:45:24.225 INFO:tasks.ceph:[osd] debug osd = 20 2022-09-12T18:45:24.225 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2022-09-12T18:45:24.226 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2022-09-12T18:45:24.226 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2022-09-12T18:45:24.226 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2022-09-12T18:45:24.227 INFO:tasks.ceph:Setting up c2.mon.a... 2022-09-12T18:45:24.227 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2022-09-12T18:45:24.319 INFO:teuthology.orchestra.run.smithi138.stdout:creating /etc/ceph/c2.keyring 2022-09-12T18:45:24.321 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2022-09-12T18:45:24.368 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0644 /etc/ceph/c2.keyring 2022-09-12T18:45:24.394 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.138')] 2022-09-12T18:45:24.395 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.138'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2022-09-12T18:45:24.396 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-09-12T18:45:24.397 DEBUG:teuthology.orchestra.run.smithi138:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2022-09-12T18:45:24.452 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.138 --print /home/ubuntu/cephtest/c2.monmap 2022-09-12T18:45:24.533 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:45:24.534 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:45:24.535 INFO:teuthology.orchestra.run.smithi138.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2022-09-12T18:45:24.535 INFO:teuthology.orchestra.run.smithi138.stdout:monmaptool: generated fsid 27700e6f-3189-467a-882c-63786db1882c 2022-09-12T18:45:24.536 INFO:teuthology.orchestra.run.smithi138.stdout:setting min_mon_release = octopus 2022-09-12T18:45:24.536 INFO:teuthology.orchestra.run.smithi138.stdout:epoch 0 2022-09-12T18:45:24.536 INFO:teuthology.orchestra.run.smithi138.stdout:fsid 27700e6f-3189-467a-882c-63786db1882c 2022-09-12T18:45:24.536 INFO:teuthology.orchestra.run.smithi138.stdout:last_changed 2022-09-12T18:45:24.531591+0000 2022-09-12T18:45:24.537 INFO:teuthology.orchestra.run.smithi138.stdout:created 2022-09-12T18:45:24.531591+0000 2022-09-12T18:45:24.537 INFO:teuthology.orchestra.run.smithi138.stdout:min_mon_release 15 (octopus) 2022-09-12T18:45:24.537 INFO:teuthology.orchestra.run.smithi138.stdout:election_strategy: 1 2022-09-12T18:45:24.537 INFO:teuthology.orchestra.run.smithi138.stdout:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2022-09-12T18:45:24.537 INFO:teuthology.orchestra.run.smithi138.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2022-09-12T18:45:24.539 DEBUG:teuthology.orchestra.run.smithi138:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2022-09-12T18:45:24.593 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 27700e6f-3189-467a-882c-63786db1882c... 2022-09-12T18:45:24.595 DEBUG:teuthology.orchestra.run.smithi035:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2022-09-12T18:45:24.597 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2022-09-12T18:45:24.646 INFO:teuthology.orchestra.run.smithi035.stdout:[global] 2022-09-12T18:45:24.646 INFO:teuthology.orchestra.run.smithi035.stdout: chdir = "" 2022-09-12T18:45:24.647 INFO:teuthology.orchestra.run.smithi035.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-09-12T18:45:24.647 INFO:teuthology.orchestra.run.smithi035.stdout: auth supported = cephx 2022-09-12T18:45:24.647 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.647 INFO:teuthology.orchestra.run.smithi035.stdout: filestore xattr use omap = true 2022-09-12T18:45:24.648 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.648 INFO:teuthology.orchestra.run.smithi035.stdout: mon clock drift allowed = 1.000 2022-09-12T18:45:24.648 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.648 INFO:teuthology.orchestra.run.smithi035.stdout: osd crush chooseleaf type = 0 2022-09-12T18:45:24.649 INFO:teuthology.orchestra.run.smithi035.stdout: auth debug = true 2022-09-12T18:45:24.649 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.649 INFO:teuthology.orchestra.run.smithi035.stdout: ms die on old message = true 2022-09-12T18:45:24.649 INFO:teuthology.orchestra.run.smithi035.stdout: ms die on bug = true 2022-09-12T18:45:24.649 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.651 INFO:teuthology.orchestra.run.smithi035.stdout: mon max pg per osd = 10000 # >= luminous 2022-09-12T18:45:24.651 INFO:teuthology.orchestra.run.smithi035.stdout: mon pg warn max object skew = 0 2022-09-12T18:45:24.651 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.651 INFO:teuthology.orchestra.run.smithi035.stdout: # disable pg_autoscaler by default for new pools 2022-09-12T18:45:24.652 INFO:teuthology.orchestra.run.smithi035.stdout: osd_pool_default_pg_autoscale_mode = off 2022-09-12T18:45:24.652 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.652 INFO:teuthology.orchestra.run.smithi035.stdout: osd pool default size = 2 2022-09-12T18:45:24.652 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.653 INFO:teuthology.orchestra.run.smithi035.stdout: mon osd allow primary affinity = true 2022-09-12T18:45:24.653 INFO:teuthology.orchestra.run.smithi035.stdout: mon osd allow pg remap = true 2022-09-12T18:45:24.653 INFO:teuthology.orchestra.run.smithi035.stdout: mon warn on legacy crush tunables = false 2022-09-12T18:45:24.653 INFO:teuthology.orchestra.run.smithi035.stdout: mon warn on crush straw calc version zero = false 2022-09-12T18:45:24.654 INFO:teuthology.orchestra.run.smithi035.stdout: mon warn on no sortbitwise = false 2022-09-12T18:45:24.654 INFO:teuthology.orchestra.run.smithi035.stdout: mon warn on osd down out interval zero = false 2022-09-12T18:45:24.654 INFO:teuthology.orchestra.run.smithi035.stdout: mon warn on too few osds = false 2022-09-12T18:45:24.654 INFO:teuthology.orchestra.run.smithi035.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-09-12T18:45:24.654 INFO:teuthology.orchestra.run.smithi035.stdout: mon_warn_on_pool_no_redundancy = false 2022-09-12T18:45:24.655 INFO:teuthology.orchestra.run.smithi035.stdout: mon_allow_pool_size_one = true 2022-09-12T18:45:24.655 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.655 INFO:teuthology.orchestra.run.smithi035.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-09-12T18:45:24.656 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.656 INFO:teuthology.orchestra.run.smithi035.stdout: osd default data pool replay window = 5 2022-09-12T18:45:24.656 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.656 INFO:teuthology.orchestra.run.smithi035.stdout: mon allow pool delete = true 2022-09-12T18:45:24.657 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.657 INFO:teuthology.orchestra.run.smithi035.stdout: mon cluster log file level = debug 2022-09-12T18:45:24.658 INFO:teuthology.orchestra.run.smithi035.stdout: debug asserts on shutdown = true 2022-09-12T18:45:24.658 INFO:teuthology.orchestra.run.smithi035.stdout: mon health detail to clog = false 2022-09-12T18:45:24.658 INFO:teuthology.orchestra.run.smithi035.stdout: mon host = 172.21.15.138 2022-09-12T18:45:24.658 INFO:teuthology.orchestra.run.smithi035.stdout: fsid = 27700e6f-3189-467a-882c-63786db1882c 2022-09-12T18:45:24.659 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.659 INFO:teuthology.orchestra.run.smithi035.stdout:[osd] 2022-09-12T18:45:24.659 INFO:teuthology.orchestra.run.smithi035.stdout: osd journal size = 100 2022-09-12T18:45:24.660 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.660 INFO:teuthology.orchestra.run.smithi035.stdout: osd scrub load threshold = 5.0 2022-09-12T18:45:24.660 INFO:teuthology.orchestra.run.smithi035.stdout: osd scrub max interval = 600 2022-09-12T18:45:24.660 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.660 INFO:teuthology.orchestra.run.smithi035.stdout: osd recover clone overlap = true 2022-09-12T18:45:24.661 INFO:teuthology.orchestra.run.smithi035.stdout: osd recovery max chunk = 1048576 2022-09-12T18:45:24.661 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.661 INFO:teuthology.orchestra.run.smithi035.stdout: osd debug shutdown = true 2022-09-12T18:45:24.661 INFO:teuthology.orchestra.run.smithi035.stdout: osd debug op order = true 2022-09-12T18:45:24.661 INFO:teuthology.orchestra.run.smithi035.stdout: osd debug verify stray on activate = true 2022-09-12T18:45:24.662 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.662 INFO:teuthology.orchestra.run.smithi035.stdout: osd open classes on start = true 2022-09-12T18:45:24.662 INFO:teuthology.orchestra.run.smithi035.stdout: osd debug pg log writeout = true 2022-09-12T18:45:24.662 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.663 INFO:teuthology.orchestra.run.smithi035.stdout: osd deep scrub update digest min age = 30 2022-09-12T18:45:24.663 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.663 INFO:teuthology.orchestra.run.smithi035.stdout: osd map max advance = 10 2022-09-12T18:45:24.663 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.663 INFO:teuthology.orchestra.run.smithi035.stdout: journal zero on create = true 2022-09-12T18:45:24.664 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.664 INFO:teuthology.orchestra.run.smithi035.stdout: filestore ondisk finisher threads = 3 2022-09-12T18:45:24.664 INFO:teuthology.orchestra.run.smithi035.stdout: filestore apply finisher threads = 3 2022-09-12T18:45:24.664 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.665 INFO:teuthology.orchestra.run.smithi035.stdout: bdev debug aio = true 2022-09-12T18:45:24.665 INFO:teuthology.orchestra.run.smithi035.stdout: osd debug misdirected ops = true 2022-09-12T18:45:24.665 INFO:teuthology.orchestra.run.smithi035.stdout: debug ms = 1 2022-09-12T18:45:24.665 INFO:teuthology.orchestra.run.smithi035.stdout: debug osd = 20 2022-09-12T18:45:24.666 INFO:teuthology.orchestra.run.smithi035.stdout: rocksdb delete range threshold = 0 2022-09-12T18:45:24.666 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.666 INFO:teuthology.orchestra.run.smithi035.stdout:[mgr] 2022-09-12T18:45:24.666 INFO:teuthology.orchestra.run.smithi035.stdout: debug ms = 1 2022-09-12T18:45:24.667 INFO:teuthology.orchestra.run.smithi035.stdout: debug mgr = 20 2022-09-12T18:45:24.667 INFO:teuthology.orchestra.run.smithi035.stdout: debug mon = 20 2022-09-12T18:45:24.667 INFO:teuthology.orchestra.run.smithi035.stdout: debug auth = 20 2022-09-12T18:45:24.668 INFO:teuthology.orchestra.run.smithi035.stdout: mon reweight min pgs per osd = 4 2022-09-12T18:45:24.668 INFO:teuthology.orchestra.run.smithi035.stdout: mon reweight min bytes per osd = 10 2022-09-12T18:45:24.668 INFO:teuthology.orchestra.run.smithi035.stdout: mgr/telemetry/nag = false 2022-09-12T18:45:24.668 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.669 INFO:teuthology.orchestra.run.smithi035.stdout:[mon] 2022-09-12T18:45:24.669 INFO:teuthology.orchestra.run.smithi035.stdout: debug ms = 1 2022-09-12T18:45:24.669 INFO:teuthology.orchestra.run.smithi035.stdout: debug mon = 20 2022-09-12T18:45:24.669 INFO:teuthology.orchestra.run.smithi035.stdout: debug paxos = 20 2022-09-12T18:45:24.669 INFO:teuthology.orchestra.run.smithi035.stdout: debug auth = 20 2022-09-12T18:45:24.670 INFO:teuthology.orchestra.run.smithi035.stdout: mon data avail warn = 5 2022-09-12T18:45:24.670 INFO:teuthology.orchestra.run.smithi035.stdout: mon mgr mkfs grace = 240 2022-09-12T18:45:24.670 INFO:teuthology.orchestra.run.smithi035.stdout: mon reweight min pgs per osd = 4 2022-09-12T18:45:24.670 INFO:teuthology.orchestra.run.smithi035.stdout: mon osd reporter subtree level = osd 2022-09-12T18:45:24.671 INFO:teuthology.orchestra.run.smithi035.stdout: mon osd prime pg temp = true 2022-09-12T18:45:24.671 INFO:teuthology.orchestra.run.smithi035.stdout: mon reweight min bytes per osd = 10 2022-09-12T18:45:24.672 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.672 INFO:teuthology.orchestra.run.smithi035.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-09-12T18:45:24.672 INFO:teuthology.orchestra.run.smithi035.stdout: auth mon ticket ttl = 660 # 11m 2022-09-12T18:45:24.672 INFO:teuthology.orchestra.run.smithi035.stdout: auth service ticket ttl = 240 # 4m 2022-09-12T18:45:24.673 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.673 INFO:teuthology.orchestra.run.smithi035.stdout: # don't complain about insecure global_id in the test suite 2022-09-12T18:45:24.673 INFO:teuthology.orchestra.run.smithi035.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-09-12T18:45:24.673 INFO:teuthology.orchestra.run.smithi035.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-09-12T18:45:24.673 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.674 INFO:teuthology.orchestra.run.smithi035.stdout: # 1m isn't quite enough 2022-09-12T18:45:24.674 INFO:teuthology.orchestra.run.smithi035.stdout: mon_down_mkfs_grace = 2m 2022-09-12T18:45:24.674 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.674 INFO:teuthology.orchestra.run.smithi035.stdout: mon_warn_on_filestore_osds = false 2022-09-12T18:45:24.675 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:45:24.675 INFO:teuthology.orchestra.run.smithi035.stdout:[client] 2022-09-12T18:45:24.675 INFO:teuthology.orchestra.run.smithi035.stdout: rgw cache enabled = true 2022-09-12T18:45:24.675 INFO:teuthology.orchestra.run.smithi035.stdout: rgw enable ops log = true 2022-09-12T18:45:24.675 INFO:teuthology.orchestra.run.smithi035.stdout: rgw enable usage log = true 2022-09-12T18:45:24.676 INFO:teuthology.orchestra.run.smithi035.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-09-12T18:45:24.676 INFO:teuthology.orchestra.run.smithi035.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-09-12T18:45:24.676 INFO:teuthology.orchestra.run.smithi035.stdout: debug rgw = 20 2022-09-12T18:45:24.677 INFO:teuthology.orchestra.run.smithi035.stdout: debug rgw sync = 20 2022-09-12T18:45:24.677 INFO:teuthology.orchestra.run.smithi035.stdout: rgw crypt require ssl = False 2022-09-12T18:45:24.677 INFO:teuthology.orchestra.run.smithi035.stdout: rgw crypt s3 kms backend = testing 2022-09-12T18:45:24.678 INFO:teuthology.orchestra.run.smithi035.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-09-12T18:45:24.678 INFO:teuthology.orchestra.run.smithi035.stdout: rgw curl low speed time = 300 2022-09-12T18:45:24.678 INFO:teuthology.orchestra.run.smithi035.stdout: rgw data log num shards = 4 2022-09-12T18:45:24.679 INFO:teuthology.orchestra.run.smithi035.stdout: rgw md log max shards = 4 2022-09-12T18:45:24.679 INFO:teuthology.orchestra.run.smithi035.stdout: rgw sync data inject err probability = 0.1 2022-09-12T18:45:24.679 INFO:teuthology.orchestra.run.smithi035.stdout: rgw sync log trim interval = 0 2022-09-12T18:45:24.679 INFO:teuthology.orchestra.run.smithi035.stdout: rgw sync meta inject err probability = 0.1 2022-09-12T18:45:24.680 INFO:teuthology.orchestra.run.smithi035.stdout: rgw sync obj etag verify = True 2022-09-12T18:45:24.680 INFO:teuthology.orchestra.run.smithi035.stdout: setgroup = ceph 2022-09-12T18:45:24.680 INFO:teuthology.orchestra.run.smithi035.stdout: setuser = ceph 2022-09-12T18:45:24.680 INFO:teuthology.orchestra.run.smithi035.stdout:[mon.a] 2022-09-12T18:45:24.680 INFO:teuthology.orchestra.run.smithi035.stdout:[client.0] 2022-09-12T18:45:24.681 INFO:teuthology.orchestra.run.smithi035.stdout: rgw data notify interval msec = 0 2022-09-12T18:45:24.681 INFO:teuthology.orchestra.run.smithi035.stdout:[osd.0] 2022-09-12T18:45:24.682 INFO:teuthology.orchestra.run.smithi035.stdout: osd_max_omap_entries_per_request = 10 2022-09-12T18:45:24.682 INFO:teuthology.orchestra.run.smithi035.stdout:[osd.1] 2022-09-12T18:45:24.682 INFO:teuthology.orchestra.run.smithi035.stdout: osd_max_omap_entries_per_request = 1000 2022-09-12T18:45:24.682 INFO:teuthology.orchestra.run.smithi035.stdout:[osd.2] 2022-09-12T18:45:24.683 INFO:teuthology.orchestra.run.smithi035.stdout: osd_max_omap_entries_per_request = 10000 2022-09-12T18:45:24.684 INFO:teuthology.orchestra.run.smithi138.stdout:[global] 2022-09-12T18:45:24.684 INFO:teuthology.orchestra.run.smithi138.stdout: chdir = "" 2022-09-12T18:45:24.684 INFO:teuthology.orchestra.run.smithi138.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-09-12T18:45:24.685 INFO:teuthology.orchestra.run.smithi138.stdout: auth supported = cephx 2022-09-12T18:45:24.685 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.685 INFO:teuthology.orchestra.run.smithi138.stdout: filestore xattr use omap = true 2022-09-12T18:45:24.685 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.686 INFO:teuthology.orchestra.run.smithi138.stdout: mon clock drift allowed = 1.000 2022-09-12T18:45:24.686 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.686 INFO:teuthology.orchestra.run.smithi138.stdout: osd crush chooseleaf type = 0 2022-09-12T18:45:24.686 INFO:teuthology.orchestra.run.smithi138.stdout: auth debug = true 2022-09-12T18:45:24.687 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.687 INFO:teuthology.orchestra.run.smithi138.stdout: ms die on old message = true 2022-09-12T18:45:24.687 INFO:teuthology.orchestra.run.smithi138.stdout: ms die on bug = true 2022-09-12T18:45:24.688 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.688 INFO:teuthology.orchestra.run.smithi138.stdout: mon max pg per osd = 10000 # >= luminous 2022-09-12T18:45:24.688 INFO:teuthology.orchestra.run.smithi138.stdout: mon pg warn max object skew = 0 2022-09-12T18:45:24.688 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.689 INFO:teuthology.orchestra.run.smithi138.stdout: # disable pg_autoscaler by default for new pools 2022-09-12T18:45:24.689 INFO:teuthology.orchestra.run.smithi138.stdout: osd_pool_default_pg_autoscale_mode = off 2022-09-12T18:45:24.689 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.690 INFO:teuthology.orchestra.run.smithi138.stdout: osd pool default size = 2 2022-09-12T18:45:24.690 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.690 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd allow primary affinity = true 2022-09-12T18:45:24.691 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd allow pg remap = true 2022-09-12T18:45:24.691 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on legacy crush tunables = false 2022-09-12T18:45:24.691 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on crush straw calc version zero = false 2022-09-12T18:45:24.692 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on no sortbitwise = false 2022-09-12T18:45:24.692 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on osd down out interval zero = false 2022-09-12T18:45:24.692 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on too few osds = false 2022-09-12T18:45:24.693 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-09-12T18:45:24.693 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_pool_no_redundancy = false 2022-09-12T18:45:24.693 INFO:teuthology.orchestra.run.smithi138.stdout: mon_allow_pool_size_one = true 2022-09-12T18:45:24.693 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.694 INFO:teuthology.orchestra.run.smithi138.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-09-12T18:45:24.694 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.694 INFO:teuthology.orchestra.run.smithi138.stdout: osd default data pool replay window = 5 2022-09-12T18:45:24.694 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.695 INFO:teuthology.orchestra.run.smithi138.stdout: mon allow pool delete = true 2022-09-12T18:45:24.695 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.695 INFO:teuthology.orchestra.run.smithi138.stdout: mon cluster log file level = debug 2022-09-12T18:45:24.695 INFO:teuthology.orchestra.run.smithi138.stdout: debug asserts on shutdown = true 2022-09-12T18:45:24.696 INFO:teuthology.orchestra.run.smithi138.stdout: mon health detail to clog = false 2022-09-12T18:45:24.696 INFO:teuthology.orchestra.run.smithi138.stdout: mon host = 172.21.15.138 2022-09-12T18:45:24.696 INFO:teuthology.orchestra.run.smithi138.stdout: fsid = 27700e6f-3189-467a-882c-63786db1882c 2022-09-12T18:45:24.696 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.696 INFO:teuthology.orchestra.run.smithi138.stdout:[osd] 2022-09-12T18:45:24.697 INFO:teuthology.orchestra.run.smithi138.stdout: osd journal size = 100 2022-09-12T18:45:24.697 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.697 INFO:teuthology.orchestra.run.smithi138.stdout: osd scrub load threshold = 5.0 2022-09-12T18:45:24.697 INFO:teuthology.orchestra.run.smithi138.stdout: osd scrub max interval = 600 2022-09-12T18:45:24.698 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.698 INFO:teuthology.orchestra.run.smithi138.stdout: osd recover clone overlap = true 2022-09-12T18:45:24.698 INFO:teuthology.orchestra.run.smithi138.stdout: osd recovery max chunk = 1048576 2022-09-12T18:45:24.698 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.699 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug shutdown = true 2022-09-12T18:45:24.699 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug op order = true 2022-09-12T18:45:24.699 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug verify stray on activate = true 2022-09-12T18:45:24.699 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.699 INFO:teuthology.orchestra.run.smithi138.stdout: osd open classes on start = true 2022-09-12T18:45:24.700 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug pg log writeout = true 2022-09-12T18:45:24.700 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.700 INFO:teuthology.orchestra.run.smithi138.stdout: osd deep scrub update digest min age = 30 2022-09-12T18:45:24.701 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.701 INFO:teuthology.orchestra.run.smithi138.stdout: osd map max advance = 10 2022-09-12T18:45:24.701 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.702 INFO:teuthology.orchestra.run.smithi138.stdout: journal zero on create = true 2022-09-12T18:45:24.702 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.703 INFO:teuthology.orchestra.run.smithi138.stdout: filestore ondisk finisher threads = 3 2022-09-12T18:45:24.703 INFO:teuthology.orchestra.run.smithi138.stdout: filestore apply finisher threads = 3 2022-09-12T18:45:24.704 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.704 INFO:teuthology.orchestra.run.smithi138.stdout: bdev debug aio = true 2022-09-12T18:45:24.704 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug misdirected ops = true 2022-09-12T18:45:24.705 INFO:teuthology.orchestra.run.smithi138.stdout: debug ms = 1 2022-09-12T18:45:24.705 INFO:teuthology.orchestra.run.smithi138.stdout: debug osd = 20 2022-09-12T18:45:24.706 INFO:teuthology.orchestra.run.smithi138.stdout: rocksdb delete range threshold = 0 2022-09-12T18:45:24.706 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.706 INFO:teuthology.orchestra.run.smithi138.stdout:[mgr] 2022-09-12T18:45:24.707 INFO:teuthology.orchestra.run.smithi138.stdout: debug ms = 1 2022-09-12T18:45:24.707 INFO:teuthology.orchestra.run.smithi138.stdout: debug mgr = 20 2022-09-12T18:45:24.708 INFO:teuthology.orchestra.run.smithi138.stdout: debug mon = 20 2022-09-12T18:45:24.708 INFO:teuthology.orchestra.run.smithi138.stdout: debug auth = 20 2022-09-12T18:45:24.708 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min pgs per osd = 4 2022-09-12T18:45:24.709 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min bytes per osd = 10 2022-09-12T18:45:24.709 INFO:teuthology.orchestra.run.smithi138.stdout: mgr/telemetry/nag = false 2022-09-12T18:45:24.709 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.709 INFO:teuthology.orchestra.run.smithi138.stdout:[mon] 2022-09-12T18:45:24.709 INFO:teuthology.orchestra.run.smithi138.stdout: debug ms = 1 2022-09-12T18:45:24.710 INFO:teuthology.orchestra.run.smithi138.stdout: debug mon = 20 2022-09-12T18:45:24.710 INFO:teuthology.orchestra.run.smithi138.stdout: debug paxos = 20 2022-09-12T18:45:24.710 INFO:teuthology.orchestra.run.smithi138.stdout: debug auth = 20 2022-09-12T18:45:24.710 INFO:teuthology.orchestra.run.smithi138.stdout: mon data avail warn = 5 2022-09-12T18:45:24.710 INFO:teuthology.orchestra.run.smithi138.stdout: mon mgr mkfs grace = 240 2022-09-12T18:45:24.711 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min pgs per osd = 4 2022-09-12T18:45:24.711 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd reporter subtree level = osd 2022-09-12T18:45:24.711 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd prime pg temp = true 2022-09-12T18:45:24.711 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min bytes per osd = 10 2022-09-12T18:45:24.712 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.712 INFO:teuthology.orchestra.run.smithi138.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-09-12T18:45:24.712 INFO:teuthology.orchestra.run.smithi138.stdout: auth mon ticket ttl = 660 # 11m 2022-09-12T18:45:24.712 INFO:teuthology.orchestra.run.smithi138.stdout: auth service ticket ttl = 240 # 4m 2022-09-12T18:45:24.712 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.713 INFO:teuthology.orchestra.run.smithi138.stdout: # don't complain about insecure global_id in the test suite 2022-09-12T18:45:24.713 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-09-12T18:45:24.713 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-09-12T18:45:24.713 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.714 INFO:teuthology.orchestra.run.smithi138.stdout: # 1m isn't quite enough 2022-09-12T18:45:24.714 INFO:teuthology.orchestra.run.smithi138.stdout: mon_down_mkfs_grace = 2m 2022-09-12T18:45:24.714 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.714 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_filestore_osds = false 2022-09-12T18:45:24.715 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:24.715 INFO:teuthology.orchestra.run.smithi138.stdout:[client] 2022-09-12T18:45:24.715 INFO:teuthology.orchestra.run.smithi138.stdout: rgw cache enabled = true 2022-09-12T18:45:24.715 INFO:teuthology.orchestra.run.smithi138.stdout: rgw enable ops log = true 2022-09-12T18:45:24.716 INFO:teuthology.orchestra.run.smithi138.stdout: rgw enable usage log = true 2022-09-12T18:45:24.716 INFO:teuthology.orchestra.run.smithi138.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-09-12T18:45:24.716 INFO:teuthology.orchestra.run.smithi138.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-09-12T18:45:24.716 INFO:teuthology.orchestra.run.smithi138.stdout: debug rgw = 20 2022-09-12T18:45:24.716 INFO:teuthology.orchestra.run.smithi138.stdout: debug rgw sync = 20 2022-09-12T18:45:24.717 INFO:teuthology.orchestra.run.smithi138.stdout: rgw crypt require ssl = False 2022-09-12T18:45:24.717 INFO:teuthology.orchestra.run.smithi138.stdout: rgw crypt s3 kms backend = testing 2022-09-12T18:45:24.717 INFO:teuthology.orchestra.run.smithi138.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-09-12T18:45:24.717 INFO:teuthology.orchestra.run.smithi138.stdout: rgw curl low speed time = 300 2022-09-12T18:45:24.717 INFO:teuthology.orchestra.run.smithi138.stdout: rgw data log num shards = 4 2022-09-12T18:45:24.718 INFO:teuthology.orchestra.run.smithi138.stdout: rgw md log max shards = 4 2022-09-12T18:45:24.718 INFO:teuthology.orchestra.run.smithi138.stdout: rgw sync data inject err probability = 0.1 2022-09-12T18:45:24.718 INFO:teuthology.orchestra.run.smithi138.stdout: rgw sync log trim interval = 0 2022-09-12T18:45:24.718 INFO:teuthology.orchestra.run.smithi138.stdout: rgw sync meta inject err probability = 0.1 2022-09-12T18:45:24.719 INFO:teuthology.orchestra.run.smithi138.stdout: rgw sync obj etag verify = True 2022-09-12T18:45:24.719 INFO:teuthology.orchestra.run.smithi138.stdout: setgroup = ceph 2022-09-12T18:45:24.719 INFO:teuthology.orchestra.run.smithi138.stdout: setuser = ceph 2022-09-12T18:45:24.719 INFO:teuthology.orchestra.run.smithi138.stdout:[mon.a] 2022-09-12T18:45:24.719 INFO:teuthology.orchestra.run.smithi138.stdout:[client.0] 2022-09-12T18:45:24.720 INFO:teuthology.orchestra.run.smithi138.stdout: rgw data notify interval msec = 0 2022-09-12T18:45:24.720 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.0] 2022-09-12T18:45:24.720 INFO:teuthology.orchestra.run.smithi138.stdout: osd_max_omap_entries_per_request = 10 2022-09-12T18:45:24.720 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.1] 2022-09-12T18:45:24.720 INFO:teuthology.orchestra.run.smithi138.stdout: osd_max_omap_entries_per_request = 1000 2022-09-12T18:45:24.721 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.2] 2022-09-12T18:45:24.721 INFO:teuthology.orchestra.run.smithi138.stdout: osd_max_omap_entries_per_request = 10000 2022-09-12T18:45:24.723 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2022-09-12T18:45:24.723 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2022-09-12T18:45:24.818 INFO:tasks.ceph:Copying monmap to all nodes... 2022-09-12T18:45:24.819 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-09-12T18:45:24.819 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2022-09-12T18:45:24.837 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-09-12T18:45:24.838 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2022-09-12T18:45:24.895 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi035.front.sepia.ceph.com 2022-09-12T18:45:24.896 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2022-09-12T18:45:24.896 DEBUG:teuthology.orchestra.run.smithi035:> sudo dd of=/etc/ceph/c2.keyring 2022-09-12T18:45:24.897 DEBUG:teuthology.orchestra.run.smithi035:> sudo chmod 0644 /etc/ceph/c2.keyring 2022-09-12T18:45:24.939 DEBUG:teuthology.orchestra.run.smithi035:> set -ex 2022-09-12T18:45:24.939 DEBUG:teuthology.orchestra.run.smithi035:> dd of=/home/ubuntu/cephtest/c2.monmap 2022-09-12T18:45:24.998 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi138.front.sepia.ceph.com 2022-09-12T18:45:24.999 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-09-12T18:45:24.999 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/ceph/c2.keyring 2022-09-12T18:45:24.999 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0644 /etc/ceph/c2.keyring 2022-09-12T18:45:25.035 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-09-12T18:45:25.036 DEBUG:teuthology.orchestra.run.smithi138:> dd of=/home/ubuntu/cephtest/c2.monmap 2022-09-12T18:45:25.093 INFO:tasks.ceph:Setting up mon nodes... 2022-09-12T18:45:25.094 INFO:tasks.ceph:Setting up mgr nodes... 2022-09-12T18:45:25.094 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2022-09-12T18:45:25.187 INFO:teuthology.orchestra.run.smithi138.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2022-09-12T18:45:25.189 INFO:tasks.ceph:Setting up mds nodes... 2022-09-12T18:45:25.190 INFO:tasks.ceph_client:Setting up client nodes... 2022-09-12T18:45:25.191 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2022-09-12T18:45:25.235 INFO:teuthology.orchestra.run.smithi138.stdout:creating /etc/ceph/c2.client.0.keyring 2022-09-12T18:45:25.246 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2022-09-12T18:45:25.330 INFO:teuthology.orchestra.run.smithi138.stdout:creating /etc/ceph/c2.client.1.keyring 2022-09-12T18:45:25.343 INFO:tasks.ceph:Running mkfs on osd nodes... 2022-09-12T18:45:25.344 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi035.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi138.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2022-09-12T18:45:25.345 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2022-09-12T18:45:25.411 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-12T18:45:25.411 INFO:tasks.ceph:role: c2.osd.0 2022-09-12T18:45:25.411 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi138.front.sepia.ceph.com 2022-09-12T18:45:25.412 DEBUG:teuthology.orchestra.run.smithi138:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-09-12T18:45:25.486 INFO:teuthology.orchestra.run.smithi138.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-09-12T18:45:25.487 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 attr=2, projid32bit=1 2022-09-12T18:45:25.487 INFO:teuthology.orchestra.run.smithi138.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-09-12T18:45:25.487 INFO:teuthology.orchestra.run.smithi138.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-09-12T18:45:25.488 INFO:teuthology.orchestra.run.smithi138.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-09-12T18:45:25.488 INFO:teuthology.orchestra.run.smithi138.stdout: = sunit=0 swidth=0 blks 2022-09-12T18:45:25.489 INFO:teuthology.orchestra.run.smithi138.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-09-12T18:45:25.489 INFO:teuthology.orchestra.run.smithi138.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-09-12T18:45:25.490 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-09-12T18:45:25.490 INFO:teuthology.orchestra.run.smithi138.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-09-12T18:45:25.913 INFO:teuthology.orchestra.run.smithi138.stdout:Discarding blocks...Done. 2022-09-12T18:45:25.915 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi138.front.sepia.ceph.com -o noatime 2022-09-12T18:45:25.915 DEBUG:teuthology.orchestra.run.smithi138:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2022-09-12T18:45:26.094 DEBUG:teuthology.orchestra.run.smithi138:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2022-09-12T18:45:26.128 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2022-09-12T18:45:26.195 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-12T18:45:26.196 INFO:tasks.ceph:role: c2.osd.1 2022-09-12T18:45:26.196 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi138.front.sepia.ceph.com 2022-09-12T18:45:26.197 DEBUG:teuthology.orchestra.run.smithi138:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-09-12T18:45:26.270 INFO:teuthology.orchestra.run.smithi138.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-09-12T18:45:26.271 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 attr=2, projid32bit=1 2022-09-12T18:45:26.271 INFO:teuthology.orchestra.run.smithi138.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-09-12T18:45:26.274 INFO:teuthology.orchestra.run.smithi138.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-09-12T18:45:26.274 INFO:teuthology.orchestra.run.smithi138.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-09-12T18:45:26.274 INFO:teuthology.orchestra.run.smithi138.stdout: = sunit=0 swidth=0 blks 2022-09-12T18:45:26.274 INFO:teuthology.orchestra.run.smithi138.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-09-12T18:45:26.275 INFO:teuthology.orchestra.run.smithi138.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-09-12T18:45:26.275 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-09-12T18:45:26.275 INFO:teuthology.orchestra.run.smithi138.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-09-12T18:45:26.691 INFO:teuthology.orchestra.run.smithi138.stdout:Discarding blocks...Done. 2022-09-12T18:45:26.692 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi138.front.sepia.ceph.com -o noatime 2022-09-12T18:45:26.693 DEBUG:teuthology.orchestra.run.smithi138:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2022-09-12T18:45:26.804 DEBUG:teuthology.orchestra.run.smithi138:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2022-09-12T18:45:26.835 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2022-09-12T18:45:26.902 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-12T18:45:26.903 INFO:tasks.ceph:role: c2.osd.2 2022-09-12T18:45:26.903 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi138.front.sepia.ceph.com 2022-09-12T18:45:26.904 DEBUG:teuthology.orchestra.run.smithi138:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-09-12T18:45:26.977 INFO:teuthology.orchestra.run.smithi138.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-09-12T18:45:26.977 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 attr=2, projid32bit=1 2022-09-12T18:45:26.977 INFO:teuthology.orchestra.run.smithi138.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-09-12T18:45:26.978 INFO:teuthology.orchestra.run.smithi138.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-09-12T18:45:26.978 INFO:teuthology.orchestra.run.smithi138.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-09-12T18:45:26.978 INFO:teuthology.orchestra.run.smithi138.stdout: = sunit=0 swidth=0 blks 2022-09-12T18:45:26.978 INFO:teuthology.orchestra.run.smithi138.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-09-12T18:45:26.979 INFO:teuthology.orchestra.run.smithi138.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-09-12T18:45:26.979 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-09-12T18:45:26.979 INFO:teuthology.orchestra.run.smithi138.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-09-12T18:45:27.411 INFO:teuthology.orchestra.run.smithi138.stdout:Discarding blocks...Done. 2022-09-12T18:45:27.413 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi138.front.sepia.ceph.com -o noatime 2022-09-12T18:45:27.413 DEBUG:teuthology.orchestra.run.smithi138:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2022-09-12T18:45:27.565 DEBUG:teuthology.orchestra.run.smithi138:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2022-09-12T18:45:27.598 DEBUG:teuthology.orchestra.run.smithi138:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2022-09-12T18:45:27.684 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:45:27.679+0000 7f832c46c440 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2022-09-12T18:45:27.685 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:45:27.680+0000 7f832c46c440 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2022-09-12T18:45:27.685 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:45:27.680+0000 7f832c46c440 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2022-09-12T18:45:27.685 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:45:27.680+0000 7f832c46c440 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2022-09-12T18:45:27.685 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:45:27.681+0000 7f832c46c440 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2022-09-12T18:45:27.689 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:45:27.685+0000 7f832c46c440 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2022-09-12T18:45:29.760 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2022-09-12T18:45:29.791 DEBUG:teuthology.orchestra.run.smithi138:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2022-09-12T18:45:29.880 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:45:29.875+0000 7ff9651f6440 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2022-09-12T18:45:29.880 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:45:29.876+0000 7ff9651f6440 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2022-09-12T18:45:29.881 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:45:29.876+0000 7ff9651f6440 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2022-09-12T18:45:29.881 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:45:29.876+0000 7ff9651f6440 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2022-09-12T18:45:29.881 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:45:29.876+0000 7ff9651f6440 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2022-09-12T18:45:29.881 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:45:29.876+0000 7ff9651f6440 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2022-09-12T18:45:31.952 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2022-09-12T18:45:31.981 DEBUG:teuthology.orchestra.run.smithi138:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2022-09-12T18:45:32.068 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:45:32.063+0000 7fa286bfc440 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2022-09-12T18:45:32.069 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:45:32.063+0000 7fa286bfc440 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2022-09-12T18:45:32.069 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:45:32.063+0000 7fa286bfc440 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2022-09-12T18:45:32.069 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:45:32.063+0000 7fa286bfc440 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2022-09-12T18:45:32.070 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:45:32.063+0000 7fa286bfc440 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2022-09-12T18:45:32.070 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:45:32.064+0000 7fa286bfc440 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2022-09-12T18:45:34.140 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2022-09-12T18:45:34.168 INFO:tasks.ceph:Reading keys from all nodes... 2022-09-12T18:45:34.169 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-09-12T18:45:34.169 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2022-09-12T18:45:34.236 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-09-12T18:45:34.236 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2022-09-12T18:45:34.302 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-09-12T18:45:34.303 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2022-09-12T18:45:34.370 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-09-12T18:45:34.370 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2022-09-12T18:45:34.437 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-09-12T18:45:34.437 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2022-09-12T18:45:34.496 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-09-12T18:45:34.497 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2022-09-12T18:45:34.554 INFO:tasks.ceph:Adding keys to all mons... 2022-09-12T18:45:34.555 DEBUG:teuthology.orchestra.run.smithi138:> sudo tee -a /etc/ceph/c2.keyring 2022-09-12T18:45:34.620 INFO:teuthology.orchestra.run.smithi138.stdout:[mgr.x] 2022-09-12T18:45:34.620 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQBFfh9j0YH1ChAAtFFu1XlTg6VOxx30/VLQgQ== 2022-09-12T18:45:34.621 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.0] 2022-09-12T18:45:34.621 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQBHfh9jGweaKBAAQTXRXnptf+ZBs1YCl2pu0w== 2022-09-12T18:45:34.621 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.1] 2022-09-12T18:45:34.622 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQBJfh9j781GNBAAa42m4WtVABx8OmwVqNDMEw== 2022-09-12T18:45:34.622 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.2] 2022-09-12T18:45:34.622 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQBMfh9jbUHaAxAABO2H06BlWrHXva7C1noGOw== 2022-09-12T18:45:34.622 INFO:teuthology.orchestra.run.smithi138.stdout:[client.0] 2022-09-12T18:45:34.623 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQBFfh9jPCDUDRAAxnHBzJQLMDvZ5Yo6IcbUcw== 2022-09-12T18:45:34.623 INFO:teuthology.orchestra.run.smithi138.stdout:[client.1] 2022-09-12T18:45:34.623 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQBFfh9jvPCHExAAFGUFPMgVleoQutzJBd0jsQ== 2022-09-12T18:45:34.624 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-09-12T18:45:34.717 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-09-12T18:45:34.771 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-09-12T18:45:34.824 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-09-12T18:45:34.877 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-09-12T18:45:34.933 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-09-12T18:45:34.990 INFO:tasks.ceph:Running mkfs on mon nodes... 2022-09-12T18:45:34.990 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2022-09-12T18:45:35.019 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2022-09-12T18:45:35.116 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2022-09-12T18:45:35.143 DEBUG:teuthology.orchestra.run.smithi138:> rm -- /home/ubuntu/cephtest/c2.monmap 2022-09-12T18:45:35.200 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2022-09-12T18:45:35.201 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2022-09-12T18:45:35.201 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2022-09-12T18:45:35.243 INFO:tasks.ceph.c2.mon.a:Started 2022-09-12T18:45:35.243 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2022-09-12T18:45:35.244 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2022-09-12T18:45:35.244 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2022-09-12T18:45:35.247 INFO:tasks.ceph.c2.mgr.x:Started 2022-09-12T18:45:35.247 INFO:tasks.ceph:Setting crush tunables to default 2022-09-12T18:45:35.247 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster c2 osd crush tunables default 2022-09-12T18:45:35.450 INFO:teuthology.orchestra.run.smithi138.stderr:adjusted tunables profile to default 2022-09-12T18:45:35.460 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2022-09-12T18:45:35.461 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-09-12T18:45:35.461 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2022-09-12T18:45:35.488 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-09-12T18:45:35.488 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2022-09-12T18:45:35.555 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-09-12T18:45:35.555 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2022-09-12T18:45:35.622 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster c2 osd new 8c2d9141-4edb-4d32-a6f0-ebbd0b8ab31e 0 2022-09-12T18:45:35.636 INFO:tasks.ceph.c2.mgr.x.smithi138.stderr:2022-09-12T18:45:35.632+0000 7fcb30b35080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-09-12T18:45:35.801 INFO:tasks.ceph.c2.mgr.x.smithi138.stderr:2022-09-12T18:45:35.797+0000 7fcb30b35080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-09-12T18:45:35.863 INFO:teuthology.orchestra.run.smithi138.stdout:0 2022-09-12T18:45:35.877 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster c2 osd new 77e305d5-5ef3-47c2-837f-1a15d8ecb609 1 2022-09-12T18:45:36.091 INFO:teuthology.orchestra.run.smithi138.stdout:1 2022-09-12T18:45:36.105 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster c2 osd new 8d7622b7-3875-4ce4-991e-99547ce3eec1 2 2022-09-12T18:45:36.335 INFO:tasks.ceph.c2.mgr.x.smithi138.stderr:2022-09-12T18:45:36.331+0000 7fcb30b35080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-09-12T18:45:36.343 INFO:teuthology.orchestra.run.smithi138.stdout:2 2022-09-12T18:45:36.355 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2022-09-12T18:45:36.356 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2022-09-12T18:45:36.359 INFO:tasks.ceph.c2.osd.0:Started 2022-09-12T18:45:36.359 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2022-09-12T18:45:36.360 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2022-09-12T18:45:36.362 INFO:tasks.ceph.c2.osd.1:Started 2022-09-12T18:45:36.363 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2022-09-12T18:45:36.363 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2022-09-12T18:45:36.366 INFO:tasks.ceph.c2.osd.2:Started 2022-09-12T18:45:36.366 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2022-09-12T18:45:36.580 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:36.580 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":5,"fsid":"27700e6f-3189-467a-882c-63786db1882c","created":"2022-09-12T18:45:35.355317+0000","modified":"2022-09-12T18:45:36.338538+0000","last_up_change":"0.000000","last_in_change":"2022-09-12T18:45:36.338538+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"8c2d9141-4edb-4d32-a6f0-ebbd0b8ab31e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"77e305d5-5ef3-47c2-837f-1a15d8ecb609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8d7622b7-3875-4ce4-991e-99547ce3eec1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-12T18:45:36.592 INFO:tasks.ceph.ceph_manager.c2:[] 2022-09-12T18:45:36.592 INFO:tasks.ceph:Waiting for OSDs to come up 2022-09-12T18:45:36.701 INFO:tasks.ceph.c2.mgr.x.smithi138.stderr:2022-09-12T18:45:36.698+0000 7fcb30b35080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-09-12T18:45:36.800 INFO:tasks.ceph.c2.mgr.x.smithi138.stderr:2022-09-12T18:45:36.796+0000 7fcb30b35080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-09-12T18:45:36.896 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2022-09-12T18:45:36.965 INFO:tasks.ceph.c2.osd.1.smithi138.stderr:2022-09-12T18:45:36.961+0000 7fdddd00c440 -1 Falling back to public interface 2022-09-12T18:45:36.991 INFO:tasks.ceph.c2.osd.2.smithi138.stderr:2022-09-12T18:45:36.986+0000 7f7b06ad6440 -1 Falling back to public interface 2022-09-12T18:45:36.993 INFO:tasks.ceph.c2.osd.0.smithi138.stderr:2022-09-12T18:45:36.989+0000 7f004d1ca440 -1 Falling back to public interface 2022-09-12T18:45:37.006 INFO:tasks.ceph.c2.mgr.x.smithi138.stderr:2022-09-12T18:45:37.002+0000 7fcb30b35080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-09-12T18:45:37.084 INFO:teuthology.misc.health.smithi138.stdout: 2022-09-12T18:45:37.084 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":5,"fsid":"27700e6f-3189-467a-882c-63786db1882c","created":"2022-09-12T18:45:35.355317+0000","modified":"2022-09-12T18:45:36.338538+0000","last_up_change":"0.000000","last_in_change":"2022-09-12T18:45:36.338538+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"8c2d9141-4edb-4d32-a6f0-ebbd0b8ab31e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"77e305d5-5ef3-47c2-837f-1a15d8ecb609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8d7622b7-3875-4ce4-991e-99547ce3eec1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-12T18:45:37.096 DEBUG:teuthology.misc:0 of 3 OSDs are up 2022-09-12T18:45:37.954 INFO:tasks.ceph.c2.mgr.x.smithi138.stderr:2022-09-12T18:45:37.950+0000 7fcb30b35080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-09-12T18:45:38.029 INFO:tasks.ceph.c2.mgr.x.smithi138.stderr:2022-09-12T18:45:38.025+0000 7fcb30b35080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-09-12T18:45:38.118 INFO:tasks.ceph.c2.mgr.x.smithi138.stderr:2022-09-12T18:45:38.114+0000 7fcb30b35080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-09-12T18:45:38.197 INFO:tasks.ceph.c2.mgr.x.smithi138.stderr:2022-09-12T18:45:38.193+0000 7fcb30b35080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-09-12T18:45:38.487 INFO:tasks.ceph.c2.mgr.x.smithi138.stderr:2022-09-12T18:45:38.483+0000 7fcb30b35080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-09-12T18:45:38.558 INFO:tasks.ceph.c2.mgr.x.smithi138.stderr:2022-09-12T18:45:38.554+0000 7fcb30b35080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-09-12T18:45:38.630 INFO:tasks.ceph.c2.mgr.x.smithi138.stderr:2022-09-12T18:45:38.626+0000 7fcb30b35080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-09-12T18:45:39.258 INFO:tasks.ceph.c2.mgr.x.smithi138.stderr:2022-09-12T18:45:39.255+0000 7fcb30b35080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-09-12T18:45:39.395 INFO:tasks.ceph.c2.mgr.x.smithi138.stderr:2022-09-12T18:45:39.391+0000 7fcb30b35080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-09-12T18:45:39.541 INFO:tasks.ceph.c2.mgr.x.smithi138.stderr:2022-09-12T18:45:39.537+0000 7fcb30b35080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-09-12T18:45:39.614 INFO:tasks.ceph.c2.mgr.x.smithi138.stderr:2022-09-12T18:45:39.609+0000 7fcb30b35080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-09-12T18:45:39.746 INFO:tasks.ceph.c2.mgr.x.smithi138.stderr:2022-09-12T18:45:39.742+0000 7fcb30b35080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-09-12T18:45:39.760 INFO:tasks.ceph.c2.osd.1.smithi138.stderr:2022-09-12T18:45:39.755+0000 7fdddd00c440 -1 osd.1 0 log_to_monitors true 2022-09-12T18:45:39.760 INFO:tasks.ceph.c2.osd.0.smithi138.stderr:2022-09-12T18:45:39.756+0000 7f004d1ca440 -1 osd.0 0 log_to_monitors true 2022-09-12T18:45:39.783 INFO:tasks.ceph.c2.osd.2.smithi138.stderr:2022-09-12T18:45:39.779+0000 7f7b06ad6440 -1 osd.2 0 log_to_monitors true 2022-09-12T18:45:39.846 INFO:tasks.ceph.c2.mgr.x.smithi138.stderr:2022-09-12T18:45:39.842+0000 7fcb30b35080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-09-12T18:45:40.096 INFO:tasks.ceph.c2.mgr.x.smithi138.stderr:2022-09-12T18:45:40.092+0000 7fcb30b35080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-09-12T18:45:40.361 INFO:tasks.ceph.c2.mgr.x.smithi138.stderr:2022-09-12T18:45:40.356+0000 7fcb30b35080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-09-12T18:45:40.441 INFO:tasks.ceph.c2.mgr.x.smithi138.stderr:2022-09-12T18:45:40.437+0000 7fcb30b35080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-09-12T18:45:40.622 INFO:tasks.ceph.c2.mgr.x.smithi138.stderr:2022-09-12T18:45:40.617+0000 7fcb30b35080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-09-12T18:45:40.772 INFO:tasks.ceph.c2.mgr.x.smithi138.stderr:2022-09-12T18:45:40.768+0000 7fcb30b35080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-09-12T18:45:41.837 INFO:tasks.ceph.c2.mgr.x.smithi138.stderr:2022-09-12T18:45:41.831+0000 7fcaef8ee700 -1 mgr.server handle_report got status from non-daemon mon.a 2022-09-12T18:45:41.984 INFO:tasks.ceph.c2.osd.0.smithi138.stderr:2022-09-12T18:45:41.980+0000 7f0040f77700 -1 osd.0 0 waiting for initial osdmap 2022-09-12T18:45:42.002 INFO:tasks.ceph.c2.osd.0.smithi138.stderr:2022-09-12T18:45:41.998+0000 7f0039352700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-12T18:45:42.022 INFO:tasks.ceph.c2.osd.2.smithi138.stderr:2022-09-12T18:45:42.018+0000 7f7afa883700 -1 osd.2 0 waiting for initial osdmap 2022-09-12T18:45:42.026 INFO:tasks.ceph.c2.osd.1.smithi138.stderr:2022-09-12T18:45:42.021+0000 7fddd0db9700 -1 osd.1 0 waiting for initial osdmap 2022-09-12T18:45:42.031 INFO:tasks.ceph.c2.osd.2.smithi138.stderr:2022-09-12T18:45:42.028+0000 7f7af2c5e700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-12T18:45:42.040 INFO:tasks.ceph.c2.osd.1.smithi138.stderr:2022-09-12T18:45:42.036+0000 7fddc9194700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-12T18:45:43.400 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2022-09-12T18:45:43.712 INFO:teuthology.misc.health.smithi138.stdout: 2022-09-12T18:45:43.712 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":8,"fsid":"27700e6f-3189-467a-882c-63786db1882c","created":"2022-09-12T18:45:35.355317+0000","modified":"2022-09-12T18:45:42.982696+0000","last_up_change":"2022-09-12T18:45:42.982696+0000","last_in_change":"2022-09-12T18:45:36.338538+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"8c2d9141-4edb-4d32-a6f0-ebbd0b8ab31e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":101799},{"type":"v1","addr":"172.21.15.138:6817","nonce":101799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":101799},{"type":"v1","addr":"172.21.15.138:6819","nonce":101799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":101799},{"type":"v1","addr":"172.21.15.138:6823","nonce":101799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":101799},{"type":"v1","addr":"172.21.15.138:6821","nonce":101799}]},"public_addr":"172.21.15.138:6817/101799","cluster_addr":"172.21.15.138:6819/101799","heartbeat_back_addr":"172.21.15.138:6823/101799","heartbeat_front_addr":"172.21.15.138:6821/101799","state":["exists","up"]},{"osd":1,"uuid":"77e305d5-5ef3-47c2-837f-1a15d8ecb609","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":101798},{"type":"v1","addr":"172.21.15.138:6801","nonce":101798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":101798},{"type":"v1","addr":"172.21.15.138:6803","nonce":101798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":101798},{"type":"v1","addr":"172.21.15.138:6807","nonce":101798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":101798},{"type":"v1","addr":"172.21.15.138:6805","nonce":101798}]},"public_addr":"172.21.15.138:6801/101798","cluster_addr":"172.21.15.138:6803/101798","heartbeat_back_addr":"172.21.15.138:6807/101798","heartbeat_front_addr":"172.21.15.138:6805/101798","state":["exists","up"]},{"osd":2,"uuid":"8d7622b7-3875-4ce4-991e-99547ce3eec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":101814},{"type":"v1","addr":"172.21.15.138:6809","nonce":101814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":101814},{"type":"v1","addr":"172.21.15.138:6811","nonce":101814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":101814},{"type":"v1","addr":"172.21.15.138:6815","nonce":101814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":101814},{"type":"v1","addr":"172.21.15.138:6813","nonce":101814}]},"public_addr":"172.21.15.138:6809/101814","cluster_addr":"172.21.15.138:6811/101814","heartbeat_back_addr":"172.21.15.138:6815/101814","heartbeat_front_addr":"172.21.15.138:6813/101814","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-12T18:45:43.725 DEBUG:teuthology.misc:3 of 3 OSDs are up 2022-09-12T18:45:43.725 INFO:tasks.ceph:Creating RBD pool 2022-09-12T18:45:43.726 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster c2 osd pool create rbd 8 2022-09-12T18:45:45.037 INFO:teuthology.orchestra.run.smithi138.stderr:pool 'rbd' created 2022-09-12T18:45:45.049 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2022-09-12T18:45:46.041 INFO:teuthology.orchestra.run.smithi138.stderr:enabled application 'rbd' on pool 'rbd' 2022-09-12T18:45:46.054 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2022-09-12T18:45:46.054 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2022-09-12T18:45:46.055 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2022-09-12T18:45:46.055 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2022-09-12T18:45:46.056 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2022-09-12T18:45:46.449 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:46.463 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":101383},{"type":"v1","addr":"172.21.15.138:6825","nonce":101383}]},"active_addr":"172.21.15.138:6825/101383","active_change":"2022-09-12T18:45:40.781905+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.138:0","nonce":3313238799}]},{"addrvec":[{"type":"v2","addr":"172.21.15.138:0","nonce":2184620890}]},{"addrvec":[{"type":"v2","addr":"172.21.15.138:0","nonce":3101653506}]}]}} 2022-09-12T18:45:46.466 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2022-09-12T18:45:46.467 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2022-09-12T18:45:46.467 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2022-09-12T18:45:46.774 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:46.774 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":11,"fsid":"27700e6f-3189-467a-882c-63786db1882c","created":"2022-09-12T18:45:35.355317+0000","modified":"2022-09-12T18:45:46.029423+0000","last_up_change":"2022-09-12T18:45:42.982696+0000","last_in_change":"2022-09-12T18:45:36.338538+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2022-09-12T18:45:44.045182+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2022-09-12T18:45:44.825712+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"8c2d9141-4edb-4d32-a6f0-ebbd0b8ab31e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":101799},{"type":"v1","addr":"172.21.15.138:6817","nonce":101799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":101799},{"type":"v1","addr":"172.21.15.138:6819","nonce":101799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":101799},{"type":"v1","addr":"172.21.15.138:6823","nonce":101799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":101799},{"type":"v1","addr":"172.21.15.138:6821","nonce":101799}]},"public_addr":"172.21.15.138:6817/101799","cluster_addr":"172.21.15.138:6819/101799","heartbeat_back_addr":"172.21.15.138:6823/101799","heartbeat_front_addr":"172.21.15.138:6821/101799","state":["exists","up"]},{"osd":1,"uuid":"77e305d5-5ef3-47c2-837f-1a15d8ecb609","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":101798},{"type":"v1","addr":"172.21.15.138:6801","nonce":101798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":101798},{"type":"v1","addr":"172.21.15.138:6803","nonce":101798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":101798},{"type":"v1","addr":"172.21.15.138:6807","nonce":101798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":101798},{"type":"v1","addr":"172.21.15.138:6805","nonce":101798}]},"public_addr":"172.21.15.138:6801/101798","cluster_addr":"172.21.15.138:6803/101798","heartbeat_back_addr":"172.21.15.138:6807/101798","heartbeat_front_addr":"172.21.15.138:6805/101798","state":["exists","up"]},{"osd":2,"uuid":"8d7622b7-3875-4ce4-991e-99547ce3eec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":101814},{"type":"v1","addr":"172.21.15.138:6809","nonce":101814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":101814},{"type":"v1","addr":"172.21.15.138:6811","nonce":101814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":101814},{"type":"v1","addr":"172.21.15.138:6815","nonce":101814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":101814},{"type":"v1","addr":"172.21.15.138:6813","nonce":101814}]},"public_addr":"172.21.15.138:6809/101814","cluster_addr":"172.21.15.138:6811/101814","heartbeat_back_addr":"172.21.15.138:6815/101814","heartbeat_front_addr":"172.21.15.138:6813/101814","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-12T18:45:40.743106+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-12T18:45:40.777150+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-12T18:45:40.768446+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-12T18:45:46.788 INFO:tasks.ceph.ceph_manager.c2:all up! 2022-09-12T18:45:46.788 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2022-09-12T18:45:47.094 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:47.094 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":12,"fsid":"27700e6f-3189-467a-882c-63786db1882c","created":"2022-09-12T18:45:35.355317+0000","modified":"2022-09-12T18:45:47.032647+0000","last_up_change":"2022-09-12T18:45:42.982696+0000","last_in_change":"2022-09-12T18:45:36.338538+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2022-09-12T18:45:44.045182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2022-09-12T18:45:44.825712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"8c2d9141-4edb-4d32-a6f0-ebbd0b8ab31e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":101799},{"type":"v1","addr":"172.21.15.138:6817","nonce":101799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":101799},{"type":"v1","addr":"172.21.15.138:6819","nonce":101799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":101799},{"type":"v1","addr":"172.21.15.138:6823","nonce":101799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":101799},{"type":"v1","addr":"172.21.15.138:6821","nonce":101799}]},"public_addr":"172.21.15.138:6817/101799","cluster_addr":"172.21.15.138:6819/101799","heartbeat_back_addr":"172.21.15.138:6823/101799","heartbeat_front_addr":"172.21.15.138:6821/101799","state":["exists","up"]},{"osd":1,"uuid":"77e305d5-5ef3-47c2-837f-1a15d8ecb609","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":101798},{"type":"v1","addr":"172.21.15.138:6801","nonce":101798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":101798},{"type":"v1","addr":"172.21.15.138:6803","nonce":101798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":101798},{"type":"v1","addr":"172.21.15.138:6807","nonce":101798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":101798},{"type":"v1","addr":"172.21.15.138:6805","nonce":101798}]},"public_addr":"172.21.15.138:6801/101798","cluster_addr":"172.21.15.138:6803/101798","heartbeat_back_addr":"172.21.15.138:6807/101798","heartbeat_front_addr":"172.21.15.138:6805/101798","state":["exists","up"]},{"osd":2,"uuid":"8d7622b7-3875-4ce4-991e-99547ce3eec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":101814},{"type":"v1","addr":"172.21.15.138:6809","nonce":101814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":101814},{"type":"v1","addr":"172.21.15.138:6811","nonce":101814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":101814},{"type":"v1","addr":"172.21.15.138:6815","nonce":101814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":101814},{"type":"v1","addr":"172.21.15.138:6813","nonce":101814}]},"public_addr":"172.21.15.138:6809/101814","cluster_addr":"172.21.15.138:6811/101814","heartbeat_back_addr":"172.21.15.138:6815/101814","heartbeat_front_addr":"172.21.15.138:6813/101814","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-12T18:45:40.743106+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-12T18:45:40.777150+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-12T18:45:40.768446+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-12T18:45:47.109 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2022-09-12T18:45:47.110 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2022-09-12T18:45:47.111 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2022-09-12T18:45:47.278 INFO:teuthology.orchestra.run.smithi138.stdout:34359738371 2022-09-12T18:45:47.279 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2022-09-12T18:45:47.281 INFO:teuthology.orchestra.run.smithi138.stdout:34359738372 2022-09-12T18:45:47.282 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2022-09-12T18:45:47.283 INFO:teuthology.orchestra.run.smithi138.stdout:34359738371 2022-09-12T18:45:47.284 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2022-09-12T18:45:47.644 INFO:teuthology.orchestra.run.smithi138.stdout:34359738370 2022-09-12T18:45:47.645 INFO:teuthology.orchestra.run.smithi138.stdout:34359738370 2022-09-12T18:45:47.654 INFO:teuthology.orchestra.run.smithi138.stdout:34359738371 2022-09-12T18:45:47.659 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.0 2022-09-12T18:45:47.665 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.2 2022-09-12T18:45:47.671 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738371 for osd.1 2022-09-12T18:45:48.660 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2022-09-12T18:45:48.666 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2022-09-12T18:45:48.672 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2022-09-12T18:45:48.999 INFO:teuthology.orchestra.run.smithi138.stdout:34359738370 2022-09-12T18:45:49.012 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.0 2022-09-12T18:45:49.042 INFO:teuthology.orchestra.run.smithi138.stdout:34359738371 2022-09-12T18:45:49.051 INFO:teuthology.orchestra.run.smithi138.stdout:34359738372 2022-09-12T18:45:49.056 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.2 2022-09-12T18:45:49.057 DEBUG:teuthology.parallel:result is None 2022-09-12T18:45:49.066 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.1 2022-09-12T18:45:49.066 DEBUG:teuthology.parallel:result is None 2022-09-12T18:45:50.014 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2022-09-12T18:45:50.335 INFO:teuthology.orchestra.run.smithi138.stdout:34359738371 2022-09-12T18:45:50.350 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.0 2022-09-12T18:45:50.350 DEBUG:teuthology.parallel:result is None 2022-09-12T18:45:50.351 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2022-09-12T18:45:50.352 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2022-09-12T18:45:50.652 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:50.654 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2022-09-12T18:45:50.665 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2022-09-12T18:45:48.786965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":25336,"kb_used_data":1448,"kb_used_omap":0,"kb_used_meta":23360,"kb_avail":314547464,"statfs":{"total":322122547200,"available":322096603136,"internally_reserved":0,"allocated":1482752,"data_stored":1255259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23920640},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-12T18:45:46.041772+0000","last_change":"2022-09-12T18:45:46.037529+0000","last_active":"2022-09-12T18:45:46.041772+0000","last_peered":"2022-09-12T18:45:46.041772+0000","last_clean":"2022-09-12T18:45:46.041772+0000","last_became_active":"2022-09-12T18:45:46.037270+0000","last_became_peered":"2022-09-12T18:45:46.037270+0000","last_unstale":"2022-09-12T18:45:46.041772+0000","last_undegraded":"2022-09-12T18:45:46.041772+0000","last_fullsized":"2022-09-12T18:45:46.041772+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_clean_scrub_stamp":"2022-09-12T18:45:45.027279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-13T19:52:44.873673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-12T18:45:46.037217+0000","last_change":"2022-09-12T18:45:46.035293+0000","last_active":"2022-09-12T18:45:46.037217+0000","last_peered":"2022-09-12T18:45:46.037217+0000","last_clean":"2022-09-12T18:45:46.037217+0000","last_became_active":"2022-09-12T18:45:46.035030+0000","last_became_peered":"2022-09-12T18:45:46.035030+0000","last_unstale":"2022-09-12T18:45:46.037217+0000","last_undegraded":"2022-09-12T18:45:46.037217+0000","last_fullsized":"2022-09-12T18:45:46.037217+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_clean_scrub_stamp":"2022-09-12T18:45:45.027279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T03:29:55.303055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-12T18:45:47.034776+0000","last_change":"2022-09-12T18:45:46.036676+0000","last_active":"2022-09-12T18:45:47.034776+0000","last_peered":"2022-09-12T18:45:47.034776+0000","last_clean":"2022-09-12T18:45:47.034776+0000","last_became_active":"2022-09-12T18:45:46.036405+0000","last_became_peered":"2022-09-12T18:45:46.036405+0000","last_unstale":"2022-09-12T18:45:47.034776+0000","last_undegraded":"2022-09-12T18:45:47.034776+0000","last_fullsized":"2022-09-12T18:45:47.034776+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_clean_scrub_stamp":"2022-09-12T18:45:45.027279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T01:38:33.118134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-12T18:45:46.037593+0000","last_change":"2022-09-12T18:45:46.036570+0000","last_active":"2022-09-12T18:45:46.037593+0000","last_peered":"2022-09-12T18:45:46.037593+0000","last_clean":"2022-09-12T18:45:46.037593+0000","last_became_active":"2022-09-12T18:45:46.036329+0000","last_became_peered":"2022-09-12T18:45:46.036329+0000","last_unstale":"2022-09-12T18:45:46.037593+0000","last_undegraded":"2022-09-12T18:45:46.037593+0000","last_fullsized":"2022-09-12T18:45:46.037593+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_clean_scrub_stamp":"2022-09-12T18:45:45.027279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-13T21:11:47.753884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-12T18:45:46.039402+0000","last_change":"2022-09-12T18:45:46.035387+0000","last_active":"2022-09-12T18:45:46.039402+0000","last_peered":"2022-09-12T18:45:46.039402+0000","last_clean":"2022-09-12T18:45:46.039402+0000","last_became_active":"2022-09-12T18:45:46.035140+0000","last_became_peered":"2022-09-12T18:45:46.035140+0000","last_unstale":"2022-09-12T18:45:46.039402+0000","last_undegraded":"2022-09-12T18:45:46.039402+0000","last_fullsized":"2022-09-12T18:45:46.039402+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_clean_scrub_stamp":"2022-09-12T18:45:45.027279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-13T21:09:35.621609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":137,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-12T18:45:47.034700+0000","last_change":"2022-09-12T18:45:46.038236+0000","last_active":"2022-09-12T18:45:47.034700+0000","last_peered":"2022-09-12T18:45:47.034700+0000","last_clean":"2022-09-12T18:45:47.034700+0000","last_became_active":"2022-09-12T18:45:46.038038+0000","last_became_peered":"2022-09-12T18:45:46.038038+0000","last_unstale":"2022-09-12T18:45:47.034700+0000","last_undegraded":"2022-09-12T18:45:47.034700+0000","last_fullsized":"2022-09-12T18:45:47.034700+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_clean_scrub_stamp":"2022-09-12T18:45:45.027279+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T06:25:20.133219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-12T18:45:46.041761+0000","last_change":"2022-09-12T18:45:46.037310+0000","last_active":"2022-09-12T18:45:46.041761+0000","last_peered":"2022-09-12T18:45:46.041761+0000","last_clean":"2022-09-12T18:45:46.041761+0000","last_became_active":"2022-09-12T18:45:46.037064+0000","last_became_peered":"2022-09-12T18:45:46.037064+0000","last_unstale":"2022-09-12T18:45:46.041761+0000","last_undegraded":"2022-09-12T18:45:46.041761+0000","last_fullsized":"2022-09-12T18:45:46.041761+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_clean_scrub_stamp":"2022-09-12T18:45:45.027279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T06:09:08.547963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-12T18:45:46.037247+0000","last_change":"2022-09-12T18:45:46.035284+0000","last_active":"2022-09-12T18:45:46.037247+0000","last_peered":"2022-09-12T18:45:46.037247+0000","last_clean":"2022-09-12T18:45:46.037247+0000","last_became_active":"2022-09-12T18:45:46.035018+0000","last_became_peered":"2022-09-12T18:45:46.035018+0000","last_unstale":"2022-09-12T18:45:46.037247+0000","last_undegraded":"2022-09-12T18:45:46.037247+0000","last_fullsized":"2022-09-12T18:45:46.037247+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_clean_scrub_stamp":"2022-09-12T18:45:45.027279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T00:14:56.958234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-12T18:45:47.034744+0000","last_change":"2022-09-12T18:45:46.036689+0000","last_active":"2022-09-12T18:45:47.034744+0000","last_peered":"2022-09-12T18:45:47.034744+0000","last_clean":"2022-09-12T18:45:47.034744+0000","last_became_active":"2022-09-12T18:45:46.036432+0000","last_became_peered":"2022-09-12T18:45:46.036432+0000","last_unstale":"2022-09-12T18:45:47.034744+0000","last_undegraded":"2022-09-12T18:45:47.034744+0000","last_fullsized":"2022-09-12T18:45:47.034744+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_clean_scrub_stamp":"2022-09-12T18:45:45.027279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-13T21:01:47.741105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8556,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104849044,"statfs":{"total":107374182400,"available":107365421056,"internally_reserved":0,"allocated":692224,"data_stored":615209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7656,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849944,"statfs":{"total":107374182400,"available":107366342656,"internally_reserved":0,"allocated":98304,"data_stored":24841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":9124,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848476,"statfs":{"total":107374182400,"available":107364839424,"internally_reserved":0,"allocated":692224,"data_stored":615209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-09-12T18:45:50.667 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2022-09-12T18:45:50.969 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:50.971 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2022-09-12T18:45:50.982 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2022-09-12T18:45:50.787312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":25416,"kb_used_data":1464,"kb_used_omap":0,"kb_used_meta":23424,"kb_avail":314547384,"statfs":{"total":322122547200,"available":322096521216,"internally_reserved":0,"allocated":1499136,"data_stored":1262931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23986176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000347"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-12T18:45:46.041772+0000","last_change":"2022-09-12T18:45:46.037529+0000","last_active":"2022-09-12T18:45:46.041772+0000","last_peered":"2022-09-12T18:45:46.041772+0000","last_clean":"2022-09-12T18:45:46.041772+0000","last_became_active":"2022-09-12T18:45:46.037270+0000","last_became_peered":"2022-09-12T18:45:46.037270+0000","last_unstale":"2022-09-12T18:45:46.041772+0000","last_undegraded":"2022-09-12T18:45:46.041772+0000","last_fullsized":"2022-09-12T18:45:46.041772+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_clean_scrub_stamp":"2022-09-12T18:45:45.027279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-13T19:52:44.873673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-12T18:45:46.037217+0000","last_change":"2022-09-12T18:45:46.035293+0000","last_active":"2022-09-12T18:45:46.037217+0000","last_peered":"2022-09-12T18:45:46.037217+0000","last_clean":"2022-09-12T18:45:46.037217+0000","last_became_active":"2022-09-12T18:45:46.035030+0000","last_became_peered":"2022-09-12T18:45:46.035030+0000","last_unstale":"2022-09-12T18:45:46.037217+0000","last_undegraded":"2022-09-12T18:45:46.037217+0000","last_fullsized":"2022-09-12T18:45:46.037217+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_clean_scrub_stamp":"2022-09-12T18:45:45.027279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T03:29:55.303055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-12T18:45:47.034776+0000","last_change":"2022-09-12T18:45:46.036676+0000","last_active":"2022-09-12T18:45:47.034776+0000","last_peered":"2022-09-12T18:45:47.034776+0000","last_clean":"2022-09-12T18:45:47.034776+0000","last_became_active":"2022-09-12T18:45:46.036405+0000","last_became_peered":"2022-09-12T18:45:46.036405+0000","last_unstale":"2022-09-12T18:45:47.034776+0000","last_undegraded":"2022-09-12T18:45:47.034776+0000","last_fullsized":"2022-09-12T18:45:47.034776+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_clean_scrub_stamp":"2022-09-12T18:45:45.027279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T01:38:33.118134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-12T18:45:46.037593+0000","last_change":"2022-09-12T18:45:46.036570+0000","last_active":"2022-09-12T18:45:46.037593+0000","last_peered":"2022-09-12T18:45:46.037593+0000","last_clean":"2022-09-12T18:45:46.037593+0000","last_became_active":"2022-09-12T18:45:46.036329+0000","last_became_peered":"2022-09-12T18:45:46.036329+0000","last_unstale":"2022-09-12T18:45:46.037593+0000","last_undegraded":"2022-09-12T18:45:46.037593+0000","last_fullsized":"2022-09-12T18:45:46.037593+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_clean_scrub_stamp":"2022-09-12T18:45:45.027279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-13T21:11:47.753884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-12T18:45:47.259422+0000","last_change":"2022-09-12T18:45:46.035387+0000","last_active":"2022-09-12T18:45:47.259422+0000","last_peered":"2022-09-12T18:45:47.259422+0000","last_clean":"2022-09-12T18:45:47.259422+0000","last_became_active":"2022-09-12T18:45:46.035140+0000","last_became_peered":"2022-09-12T18:45:46.035140+0000","last_unstale":"2022-09-12T18:45:47.259422+0000","last_undegraded":"2022-09-12T18:45:47.259422+0000","last_fullsized":"2022-09-12T18:45:47.259422+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_clean_scrub_stamp":"2022-09-12T18:45:45.027279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-13T21:09:35.621609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":137,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-12T18:45:47.034700+0000","last_change":"2022-09-12T18:45:46.038236+0000","last_active":"2022-09-12T18:45:47.034700+0000","last_peered":"2022-09-12T18:45:47.034700+0000","last_clean":"2022-09-12T18:45:47.034700+0000","last_became_active":"2022-09-12T18:45:46.038038+0000","last_became_peered":"2022-09-12T18:45:46.038038+0000","last_unstale":"2022-09-12T18:45:47.034700+0000","last_undegraded":"2022-09-12T18:45:47.034700+0000","last_fullsized":"2022-09-12T18:45:47.034700+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_clean_scrub_stamp":"2022-09-12T18:45:45.027279+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T06:25:20.133219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-12T18:45:46.041761+0000","last_change":"2022-09-12T18:45:46.037310+0000","last_active":"2022-09-12T18:45:46.041761+0000","last_peered":"2022-09-12T18:45:46.041761+0000","last_clean":"2022-09-12T18:45:46.041761+0000","last_became_active":"2022-09-12T18:45:46.037064+0000","last_became_peered":"2022-09-12T18:45:46.037064+0000","last_unstale":"2022-09-12T18:45:46.041761+0000","last_undegraded":"2022-09-12T18:45:46.041761+0000","last_fullsized":"2022-09-12T18:45:46.041761+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_clean_scrub_stamp":"2022-09-12T18:45:45.027279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T06:09:08.547963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-09-12T18:45:46.037247+0000","last_change":"2022-09-12T18:45:46.035284+0000","last_active":"2022-09-12T18:45:46.037247+0000","last_peered":"2022-09-12T18:45:46.037247+0000","last_clean":"2022-09-12T18:45:46.037247+0000","last_became_active":"2022-09-12T18:45:46.035018+0000","last_became_peered":"2022-09-12T18:45:46.035018+0000","last_unstale":"2022-09-12T18:45:46.037247+0000","last_undegraded":"2022-09-12T18:45:46.037247+0000","last_fullsized":"2022-09-12T18:45:46.037247+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_clean_scrub_stamp":"2022-09-12T18:45:45.027279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-14T00:14:56.958234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-12T18:45:47.034744+0000","last_change":"2022-09-12T18:45:46.036689+0000","last_active":"2022-09-12T18:45:47.034744+0000","last_peered":"2022-09-12T18:45:47.034744+0000","last_clean":"2022-09-12T18:45:47.034744+0000","last_became_active":"2022-09-12T18:45:46.036432+0000","last_became_peered":"2022-09-12T18:45:46.036432+0000","last_unstale":"2022-09-12T18:45:47.034744+0000","last_undegraded":"2022-09-12T18:45:47.034744+0000","last_fullsized":"2022-09-12T18:45:47.034744+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-12T18:45:45.027279+0000","last_clean_scrub_stamp":"2022-09-12T18:45:45.027279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-13T21:01:47.741105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8628,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848972,"statfs":{"total":107374182400,"available":107365347328,"internally_reserved":0,"allocated":700416,"data_stored":619045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7664,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849936,"statfs":{"total":107374182400,"available":107366334464,"internally_reserved":0,"allocated":106496,"data_stored":28677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.193}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":9124,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848476,"statfs":{"total":107374182400,"available":107364839424,"internally_reserved":0,"allocated":692224,"data_stored":615209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-09-12T18:45:50.984 INFO:tasks.ceph.ceph_manager.c2:clean! 2022-09-12T18:45:50.984 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2022-09-12T18:45:50.984 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2022-09-12T18:45:50.985 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2022-09-12T18:45:51.328 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T18:45:51.329 INFO:teuthology.orchestra.run.smithi138.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-09-12T18:45:51.343 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2022-09-12T18:45:51.344 INFO:teuthology.run_tasks:Running task rgw... 2022-09-12T18:45:51.365 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2022-09-12T18:45:51.366 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2022-09-12T18:45:51.366 INFO:tasks.rgw:Creating data pools 2022-09-12T18:45:51.367 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2022-09-12T18:45:51.367 DEBUG:teuthology.orchestra.run.smithi035:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2022-09-12T18:45:51.795 INFO:teuthology.orchestra.run.smithi035.stderr:pool 'default.rgw.buckets.data' created 2022-09-12T18:45:51.811 DEBUG:teuthology.orchestra.run.smithi035:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2022-09-12T18:45:52.815 INFO:teuthology.orchestra.run.smithi035.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2022-09-12T18:45:52.836 DEBUG:teuthology.orchestra.run.smithi035:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2022-09-12T18:45:53.813 INFO:teuthology.orchestra.run.smithi035.stderr:pool 'default.rgw.buckets.index' created 2022-09-12T18:45:53.828 DEBUG:teuthology.orchestra.run.smithi035:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2022-09-12T18:45:54.824 INFO:teuthology.orchestra.run.smithi035.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2022-09-12T18:45:54.841 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2022-09-12T18:45:54.842 DEBUG:teuthology.orchestra.run.smithi035:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2022-09-12T18:45:55.171 INFO:teuthology.orchestra.run.smithi035.stderr:pool 'default.rgw.buckets.data' already exists 2022-09-12T18:45:55.183 DEBUG:teuthology.orchestra.run.smithi035:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2022-09-12T18:45:55.813 INFO:teuthology.orchestra.run.smithi035.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2022-09-12T18:45:55.825 DEBUG:teuthology.orchestra.run.smithi035:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2022-09-12T18:45:56.155 INFO:teuthology.orchestra.run.smithi035.stderr:pool 'default.rgw.buckets.index' already exists 2022-09-12T18:45:56.167 DEBUG:teuthology.orchestra.run.smithi035:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2022-09-12T18:45:56.815 INFO:teuthology.orchestra.run.smithi035.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2022-09-12T18:45:56.827 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2022-09-12T18:45:56.827 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2022-09-12T18:45:57.361 INFO:teuthology.orchestra.run.smithi138.stderr:pool 'default.rgw.buckets.data' created 2022-09-12T18:45:57.371 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2022-09-12T18:45:58.368 INFO:teuthology.orchestra.run.smithi138.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2022-09-12T18:45:58.385 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2022-09-12T18:45:59.366 INFO:teuthology.orchestra.run.smithi138.stderr:pool 'default.rgw.buckets.index' created 2022-09-12T18:45:59.379 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2022-09-12T18:46:00.384 INFO:teuthology.orchestra.run.smithi138.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2022-09-12T18:46:00.401 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2022-09-12T18:46:00.402 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2022-09-12T18:46:00.715 INFO:teuthology.orchestra.run.smithi138.stderr:pool 'default.rgw.buckets.data' already exists 2022-09-12T18:46:00.727 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2022-09-12T18:46:01.368 INFO:teuthology.orchestra.run.smithi138.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2022-09-12T18:46:01.380 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2022-09-12T18:46:01.689 INFO:teuthology.orchestra.run.smithi138.stderr:pool 'default.rgw.buckets.index' already exists 2022-09-12T18:46:01.700 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2022-09-12T18:46:02.369 INFO:teuthology.orchestra.run.smithi138.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2022-09-12T18:46:02.381 DEBUG:tasks.rgw:Pools created 2022-09-12T18:46:02.381 INFO:tasks.rgw:Configuring compression type = random 2022-09-12T18:46:02.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2022-09-12T18:46:02.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2022-09-12T18:46:02.383 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2022-09-12T18:46:02.469 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:46:02.469 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:46:02.475 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:02.471+0000 7f60d0df5600 20 rados->read ofs=0 len=0 2022-09-12T18:46:03.848 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.844+0000 7f60d0df5600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:03.849 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.844+0000 7f60d0df5600 20 realm 2022-09-12T18:46:03.849 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.844+0000 7f60d0df5600 20 rados->read ofs=0 len=0 2022-09-12T18:46:03.850 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.845+0000 7f60d0df5600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:03.851 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.845+0000 7f60d0df5600 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-09-12T18:46:03.851 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.845+0000 7f60d0df5600 20 rados->read ofs=0 len=0 2022-09-12T18:46:03.851 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.846+0000 7f60d0df5600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:03.852 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.846+0000 7f60d0df5600 20 rados->read ofs=0 len=0 2022-09-12T18:46:03.852 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.846+0000 7f60d0df5600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:03.852 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.846+0000 7f60d0df5600 20 rados->read ofs=0 len=0 2022-09-12T18:46:03.853 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.847+0000 7f60d0df5600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:03.854 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.847+0000 7f60d0df5600 20 rados->read ofs=0 len=0 2022-09-12T18:46:03.854 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.848+0000 7f60d0df5600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:03.855 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.848+0000 7f60d0df5600 10 Creating default zonegroup 2022-09-12T18:46:03.855 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.849+0000 7f60d0df5600 10 couldn't find old data placement pools config, setting up new ones for the zone 2022-09-12T18:46:03.864 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.860+0000 7f60d0df5600 10 WARNING: store->list_zones() returned r=-2 2022-09-12T18:46:03.865 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.860+0000 7f60d0df5600 20 rados->read ofs=0 len=0 2022-09-12T18:46:03.865 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.860+0000 7f60d0df5600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:03.870 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.865+0000 7f60d0df5600 20 rados->read ofs=0 len=0 2022-09-12T18:46:03.870 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.866+0000 7f60d0df5600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:03.871 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.866+0000 7f60d0df5600 10 could not read realm id: (2) No such file or directory 2022-09-12T18:46:03.871 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.866+0000 7f60d0df5600 10 WARNING: failed to set zone as default, r=-22 2022-09-12T18:46:03.871 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.866+0000 7f60d0df5600 20 rados->read ofs=0 len=0 2022-09-12T18:46:03.872 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.866+0000 7f60d0df5600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:03.875 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.868+0000 7f60d0df5600 20 rados->read ofs=0 len=0 2022-09-12T18:46:03.875 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.869+0000 7f60d0df5600 20 rados_obj.operate() r=0 bl.length=873 2022-09-12T18:46:03.875 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.869+0000 7f60d0df5600 20 rados->read ofs=0 len=0 2022-09-12T18:46:03.876 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.869+0000 7f60d0df5600 20 rados_obj.operate() r=0 bl.length=398 2022-09-12T18:46:03.876 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.869+0000 7f60d0df5600 10 Using default name default 2022-09-12T18:46:03.876 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.869+0000 7f60d0df5600 20 rados->read ofs=0 len=0 2022-09-12T18:46:03.877 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.870+0000 7f60d0df5600 20 rados_obj.operate() r=0 bl.length=46 2022-09-12T18:46:03.877 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.870+0000 7f60d0df5600 20 rados->read ofs=0 len=0 2022-09-12T18:46:03.877 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.870+0000 7f60d0df5600 20 rados_obj.operate() r=0 bl.length=873 2022-09-12T18:46:03.878 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.870+0000 7f60d0df5600 20 zone default found 2022-09-12T18:46:03.878 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.870+0000 7f60d0df5600 4 Realm: () 2022-09-12T18:46:03.878 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.870+0000 7f60d0df5600 4 ZoneGroup: default (1693a4d5-143e-4f6a-acbd-076182241d97) 2022-09-12T18:46:03.879 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.870+0000 7f60d0df5600 4 Zone: default (53c7de48-aca4-45ae-947b-6c71bea55ec9) 2022-09-12T18:46:03.879 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.870+0000 7f60d0df5600 10 cannot find current period zonegroup using local zonegroup configuration 2022-09-12T18:46:03.879 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.870+0000 7f60d0df5600 20 zonegroup default 2022-09-12T18:46:03.880 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.870+0000 7f60d0df5600 20 rados->read ofs=0 len=0 2022-09-12T18:46:03.880 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.870+0000 7f60d0df5600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:03.881 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:03.870+0000 7f60d0df5600 20 rados->read ofs=0 len=0 2022-09-12T18:46:05.861 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:05.857+0000 7f60d0df5600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:05.862 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:05.857+0000 7f60d0df5600 20 rados->read ofs=0 len=0 2022-09-12T18:46:05.863 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:05.858+0000 7f60d0df5600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:05.863 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:05.858+0000 7f60d0df5600 20 started sync module instance, tier type = 2022-09-12T18:46:05.863 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:05.858+0000 7f60d0df5600 20 started zone id=53c7de48-aca4-45ae-947b-6c71bea55ec9 (name=default) with tier type = 2022-09-12T18:46:07.909 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.905+0000 7f60d0df5600 20 add_watcher() i=0 2022-09-12T18:46:07.909 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.905+0000 7f60d0df5600 20 add_watcher() i=1 2022-09-12T18:46:07.910 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.905+0000 7f60d0df5600 20 add_watcher() i=2 2022-09-12T18:46:07.910 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.905+0000 7f60d0df5600 20 add_watcher() i=3 2022-09-12T18:46:07.910 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.905+0000 7f60d0df5600 20 add_watcher() i=4 2022-09-12T18:46:07.910 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.905+0000 7f60d0df5600 20 add_watcher() i=5 2022-09-12T18:46:07.911 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.905+0000 7f60d0df5600 20 add_watcher() i=6 2022-09-12T18:46:07.911 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.907+0000 7f60d0df5600 20 add_watcher() i=7 2022-09-12T18:46:07.911 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.907+0000 7f60d0df5600 2 all 8 watchers are set, enabling cache 2022-09-12T18:46:07.913 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.908+0000 7f60d0df5600 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2022-09-12T18:46:07.913 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.908+0000 7f60d0df5600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2022-09-12T18:46:07.913 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.908+0000 7f60d0df5600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-12T18:46:07.914 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.908+0000 7f60d0df5600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2022-09-12T18:46:07.914 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.908+0000 7f60d0df5600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2022-09-12T18:46:07.919 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.908+0000 7f60d0df5600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-12T18:46:07.920 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.908+0000 7f60d0df5600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2022-09-12T18:46:07.920 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.908+0000 7f60d0df5600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2022-09-12T18:46:07.920 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.908+0000 7f60d0df5600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-12T18:46:07.920 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.908+0000 7f60d0df5600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2022-09-12T18:46:07.921 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.908+0000 7f60d0df5600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2022-09-12T18:46:07.921 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.908+0000 7f60d0df5600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-12T18:46:07.921 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.909+0000 7f60d0df5600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2022-09-12T18:46:07.922 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.909+0000 7f60d0df5600 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2022-09-12T18:46:07.923 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.910+0000 7f60d0df5600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-12T18:46:07.923 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.916+0000 7f60b9576700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-12T18:46:07.923 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.916+0000 7f60d0df5600 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-12T18:46:07.924 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:07.916+0000 7f60d0df5600 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-12T18:46:10.858 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.853+0000 7f60d0df5600 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2022-09-12T18:46:10.858 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.853+0000 7f60d0df5600 5 note: GC not initialized 2022-09-12T18:46:10.859 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.855+0000 7f60b4d6d700 20 reqs_thread_entry: start 2022-09-12T18:46:10.859 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.855+0000 7f60d0df5600 20 init_complete bucket index max shards: 11 2022-09-12T18:46:10.860 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.855+0000 7f60d0df5600 10 rgw notify: Started notification manager with: 1 workers 2022-09-12T18:46:10.860 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.855+0000 7f60b2d69700 20 rgw notify: INFO: next queues processing will happen at: Mon Sep 12 18:46:41 2022 2022-09-12T18:46:10.860 INFO:teuthology.orchestra.run.smithi035.stderr: 2022-09-12T18:46:10.861 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.855+0000 7f60b2547700 20 reqs_thread_entry: start 2022-09-12T18:46:10.881 INFO:teuthology.orchestra.run.smithi035.stdout:[] 2022-09-12T18:46:10.882 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.876+0000 7f60d0df5600 20 remove_watcher() i=0 2022-09-12T18:46:10.883 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.876+0000 7f60d0df5600 2 removed watcher, disabling cache 2022-09-12T18:46:10.883 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.878+0000 7f60d0df5600 20 remove_watcher() i=1 2022-09-12T18:46:10.884 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.878+0000 7f60d0df5600 20 remove_watcher() i=2 2022-09-12T18:46:10.885 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.880+0000 7f60d0df5600 20 remove_watcher() i=3 2022-09-12T18:46:10.885 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.881+0000 7f60d0df5600 20 remove_watcher() i=4 2022-09-12T18:46:10.886 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.882+0000 7f60d0df5600 20 remove_watcher() i=5 2022-09-12T18:46:10.887 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.883+0000 7f60d0df5600 20 remove_watcher() i=6 2022-09-12T18:46:10.888 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.884+0000 7f60d0df5600 20 remove_watcher() i=7 2022-09-12T18:46:10.889 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.885+0000 7f60b9576700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-12T18:46:10.894 DEBUG:tasks.util.rgw: json result: [] 2022-09-12T18:46:10.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-12T18:46:10.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-12T18:46:10.895 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2022-09-12T18:46:10.950 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:46:10.951 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:46:10.956 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.952+0000 7f26bd3ba600 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-12T18:46:10.957 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.952+0000 7f26bd3ba600 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-12T18:46:10.960 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.956+0000 7f26a2334700 20 reqs_thread_entry: start 2022-09-12T18:46:10.960 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.956+0000 7f26bd3ba600 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-09-12T18:46:10.961 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.956+0000 7f26bd3ba600 20 rados->read ofs=0 len=0 2022-09-12T18:46:10.961 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.957+0000 7f26bd3ba600 20 rados_obj.operate() r=0 bl.length=46 2022-09-12T18:46:10.961 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.957+0000 7f26bd3ba600 20 rados->read ofs=0 len=0 2022-09-12T18:46:10.962 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.958+0000 7f26bd3ba600 20 rados_obj.operate() r=0 bl.length=873 2022-09-12T18:46:10.970 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.966+0000 7f26bd3ba600 20 RGWRados::pool_iterate: got zone_info.53c7de48-aca4-45ae-947b-6c71bea55ec9 2022-09-12T18:46:10.970 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.966+0000 7f26bd3ba600 20 RGWRados::pool_iterate: got zonegroup_info.1693a4d5-143e-4f6a-acbd-076182241d97 2022-09-12T18:46:10.971 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.966+0000 7f26bd3ba600 20 RGWRados::pool_iterate: got zone_names.default 2022-09-12T18:46:10.971 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.966+0000 7f26bd3ba600 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-09-12T18:46:10.971 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.966+0000 7f26bd3ba600 20 rados->read ofs=0 len=0 2022-09-12T18:46:10.971 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.966+0000 7f26bd3ba600 20 rados_obj.operate() r=0 bl.length=873 2022-09-12T18:46:10.972 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.966+0000 7f26bd3ba600 20 rados->read ofs=0 len=0 2022-09-12T18:46:10.972 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.966+0000 7f26bd3ba600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:10.972 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.966+0000 7f26bd3ba600 20 rados->read ofs=0 len=0 2022-09-12T18:46:10.972 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.966+0000 7f26bd3ba600 20 rados_obj.operate() r=0 bl.length=46 2022-09-12T18:46:10.972 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.966+0000 7f26bd3ba600 20 rados->read ofs=0 len=0 2022-09-12T18:46:10.973 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:10.967+0000 7f26bd3ba600 20 rados_obj.operate() r=0 bl.length=398 2022-09-12T18:46:10.974 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T18:46:10.974 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "53c7de48-aca4-45ae-947b-6c71bea55ec9", 2022-09-12T18:46:10.975 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "default", 2022-09-12T18:46:10.975 INFO:teuthology.orchestra.run.smithi035.stdout: "domain_root": "default.rgw.meta:root", 2022-09-12T18:46:10.975 INFO:teuthology.orchestra.run.smithi035.stdout: "control_pool": "default.rgw.control", 2022-09-12T18:46:10.975 INFO:teuthology.orchestra.run.smithi035.stdout: "gc_pool": "default.rgw.log:gc", 2022-09-12T18:46:10.975 INFO:teuthology.orchestra.run.smithi035.stdout: "lc_pool": "default.rgw.log:lc", 2022-09-12T18:46:10.976 INFO:teuthology.orchestra.run.smithi035.stdout: "log_pool": "default.rgw.log", 2022-09-12T18:46:10.976 INFO:teuthology.orchestra.run.smithi035.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-09-12T18:46:10.976 INFO:teuthology.orchestra.run.smithi035.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-09-12T18:46:10.976 INFO:teuthology.orchestra.run.smithi035.stdout: "roles_pool": "default.rgw.meta:roles", 2022-09-12T18:46:10.977 INFO:teuthology.orchestra.run.smithi035.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-09-12T18:46:10.977 INFO:teuthology.orchestra.run.smithi035.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-09-12T18:46:10.977 INFO:teuthology.orchestra.run.smithi035.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-09-12T18:46:10.977 INFO:teuthology.orchestra.run.smithi035.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-09-12T18:46:10.977 INFO:teuthology.orchestra.run.smithi035.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-09-12T18:46:10.978 INFO:teuthology.orchestra.run.smithi035.stdout: "otp_pool": "default.rgw.otp", 2022-09-12T18:46:10.978 INFO:teuthology.orchestra.run.smithi035.stdout: "system_key": { 2022-09-12T18:46:10.978 INFO:teuthology.orchestra.run.smithi035.stdout: "access_key": "", 2022-09-12T18:46:10.978 INFO:teuthology.orchestra.run.smithi035.stdout: "secret_key": "" 2022-09-12T18:46:10.979 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:46:10.979 INFO:teuthology.orchestra.run.smithi035.stdout: "placement_pools": [ 2022-09-12T18:46:10.979 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:46:10.979 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "default-placement", 2022-09-12T18:46:10.979 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:46:10.980 INFO:teuthology.orchestra.run.smithi035.stdout: "index_pool": "default.rgw.buckets.index", 2022-09-12T18:46:10.980 INFO:teuthology.orchestra.run.smithi035.stdout: "storage_classes": { 2022-09-12T18:46:10.980 INFO:teuthology.orchestra.run.smithi035.stdout: "STANDARD": { 2022-09-12T18:46:10.980 INFO:teuthology.orchestra.run.smithi035.stdout: "data_pool": "default.rgw.buckets.data", 2022-09-12T18:46:10.980 INFO:teuthology.orchestra.run.smithi035.stdout: "compression_type": "random" 2022-09-12T18:46:10.981 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:46:10.981 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:46:10.981 INFO:teuthology.orchestra.run.smithi035.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-09-12T18:46:10.981 INFO:teuthology.orchestra.run.smithi035.stdout: "index_type": 0 2022-09-12T18:46:10.982 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:46:10.982 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:46:10.982 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T18:46:10.982 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_id": "", 2022-09-12T18:46:10.982 INFO:teuthology.orchestra.run.smithi035.stdout: "notif_pool": "default.rgw.log:notif" 2022-09-12T18:46:10.983 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T18:46:10.984 DEBUG:tasks.util.rgw: json result: {'id': '53c7de48-aca4-45ae-947b-6c71bea55ec9', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-09-12T18:46:10.984 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2022-09-12T18:46:10.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2022-09-12T18:46:10.985 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2022-09-12T18:46:11.031 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:46:11.031 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:46:11.036 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.032+0000 7efcef102600 20 rados->read ofs=0 len=0 2022-09-12T18:46:11.038 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.033+0000 7efcef102600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:11.038 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.033+0000 7efcef102600 20 realm 2022-09-12T18:46:11.038 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.033+0000 7efcef102600 20 rados->read ofs=0 len=0 2022-09-12T18:46:11.039 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.033+0000 7efcef102600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:11.039 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.033+0000 7efcef102600 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-09-12T18:46:11.039 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.033+0000 7efcef102600 20 rados->read ofs=0 len=0 2022-09-12T18:46:11.040 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.033+0000 7efcef102600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:11.040 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.033+0000 7efcef102600 20 rados->read ofs=0 len=0 2022-09-12T18:46:11.040 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.034+0000 7efcef102600 20 rados_obj.operate() r=0 bl.length=46 2022-09-12T18:46:11.040 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.034+0000 7efcef102600 20 rados->read ofs=0 len=0 2022-09-12T18:46:11.041 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.034+0000 7efcef102600 20 rados_obj.operate() r=0 bl.length=889 2022-09-12T18:46:11.041 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.035+0000 7efcef102600 20 searching for the correct realm 2022-09-12T18:46:11.046 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.041+0000 7efcef102600 20 RGWRados::pool_iterate: got zone_info.53c7de48-aca4-45ae-947b-6c71bea55ec9 2022-09-12T18:46:11.046 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.041+0000 7efcef102600 20 RGWRados::pool_iterate: got zonegroup_info.1693a4d5-143e-4f6a-acbd-076182241d97 2022-09-12T18:46:11.046 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.041+0000 7efcef102600 20 RGWRados::pool_iterate: got zone_names.default 2022-09-12T18:46:11.046 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.041+0000 7efcef102600 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-09-12T18:46:11.047 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.041+0000 7efcef102600 20 rados->read ofs=0 len=0 2022-09-12T18:46:11.047 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.042+0000 7efcef102600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:11.047 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.042+0000 7efcef102600 20 rados->read ofs=0 len=0 2022-09-12T18:46:11.047 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.042+0000 7efcef102600 20 rados_obj.operate() r=0 bl.length=46 2022-09-12T18:46:11.048 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.042+0000 7efcef102600 20 rados->read ofs=0 len=0 2022-09-12T18:46:11.049 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.042+0000 7efcef102600 20 rados_obj.operate() r=0 bl.length=398 2022-09-12T18:46:11.049 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.042+0000 7efcef102600 20 zone default found 2022-09-12T18:46:11.050 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.042+0000 7efcef102600 4 Realm: () 2022-09-12T18:46:11.050 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.042+0000 7efcef102600 4 ZoneGroup: default (1693a4d5-143e-4f6a-acbd-076182241d97) 2022-09-12T18:46:11.050 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.042+0000 7efcef102600 4 Zone: default (53c7de48-aca4-45ae-947b-6c71bea55ec9) 2022-09-12T18:46:11.050 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.042+0000 7efcef102600 10 cannot find current period zonegroup using local zonegroup configuration 2022-09-12T18:46:11.051 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.042+0000 7efcef102600 20 zonegroup default 2022-09-12T18:46:11.051 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.042+0000 7efcef102600 20 rados->read ofs=0 len=0 2022-09-12T18:46:11.051 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.042+0000 7efcef102600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:11.051 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.042+0000 7efcef102600 20 rados->read ofs=0 len=0 2022-09-12T18:46:11.052 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.043+0000 7efcef102600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:11.052 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.043+0000 7efcef102600 20 rados->read ofs=0 len=0 2022-09-12T18:46:11.052 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.043+0000 7efcef102600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:11.052 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.043+0000 7efcef102600 20 started sync module instance, tier type = 2022-09-12T18:46:11.052 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.043+0000 7efcef102600 20 started zone id=53c7de48-aca4-45ae-947b-6c71bea55ec9 (name=default) with tier type = 2022-09-12T18:46:11.066 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.062+0000 7efcef102600 20 add_watcher() i=0 2022-09-12T18:46:11.066 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.062+0000 7efcef102600 20 add_watcher() i=1 2022-09-12T18:46:11.067 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.062+0000 7efcef102600 20 add_watcher() i=2 2022-09-12T18:46:11.067 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.062+0000 7efcef102600 20 add_watcher() i=3 2022-09-12T18:46:11.067 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.062+0000 7efcef102600 20 add_watcher() i=4 2022-09-12T18:46:11.067 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.062+0000 7efcef102600 20 add_watcher() i=5 2022-09-12T18:46:11.068 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.064+0000 7efcef102600 20 add_watcher() i=6 2022-09-12T18:46:11.068 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.064+0000 7efcef102600 20 add_watcher() i=7 2022-09-12T18:46:11.069 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.064+0000 7efcef102600 2 all 8 watchers are set, enabling cache 2022-09-12T18:46:11.071 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.067+0000 7efcd7883700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-12T18:46:11.071 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.067+0000 7efcef102600 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-12T18:46:11.071 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.067+0000 7efcef102600 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-12T18:46:11.074 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.071+0000 7efcef102600 5 note: GC not initialized 2022-09-12T18:46:11.076 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.072+0000 7efcd307a700 20 reqs_thread_entry: start 2022-09-12T18:46:11.076 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.072+0000 7efcef102600 20 init_complete bucket index max shards: 11 2022-09-12T18:46:11.076 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.072+0000 7efcef102600 10 rgw notify: Started notification manager with: 1 workers 2022-09-12T18:46:11.077 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.072+0000 7efcd1076700 20 rgw notify: INFO: next queues processing will happen at: Mon Sep 12 18:46:41 2022 2022-09-12T18:46:11.077 INFO:teuthology.orchestra.run.smithi035.stderr: 2022-09-12T18:46:11.077 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.072+0000 7efcd0854700 20 reqs_thread_entry: start 2022-09-12T18:46:11.083 INFO:teuthology.orchestra.run.smithi035.stdout:[] 2022-09-12T18:46:11.084 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.080+0000 7efcef102600 20 remove_watcher() i=0 2022-09-12T18:46:11.084 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.080+0000 7efcef102600 2 removed watcher, disabling cache 2022-09-12T18:46:11.085 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.081+0000 7efcef102600 20 remove_watcher() i=1 2022-09-12T18:46:11.085 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.082+0000 7efcef102600 20 remove_watcher() i=2 2022-09-12T18:46:11.086 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.082+0000 7efcef102600 20 remove_watcher() i=3 2022-09-12T18:46:11.087 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.083+0000 7efcef102600 20 remove_watcher() i=4 2022-09-12T18:46:11.088 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.084+0000 7efcef102600 20 remove_watcher() i=5 2022-09-12T18:46:11.091 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.087+0000 7efcef102600 20 remove_watcher() i=6 2022-09-12T18:46:11.092 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.088+0000 7efcef102600 20 remove_watcher() i=7 2022-09-12T18:46:11.092 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.089+0000 7efcd7883700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-12T18:46:11.097 DEBUG:tasks.util.rgw: json result: [] 2022-09-12T18:46:11.098 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-12T18:46:11.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-12T18:46:11.099 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2022-09-12T18:46:11.150 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:46:11.151 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:46:11.156 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.152+0000 7f610df07600 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-12T18:46:11.156 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.152+0000 7f610df07600 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-12T18:46:11.160 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.156+0000 7f60f2e81700 20 reqs_thread_entry: start 2022-09-12T18:46:11.160 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.156+0000 7f610df07600 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-09-12T18:46:11.160 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.156+0000 7f610df07600 20 rados->read ofs=0 len=0 2022-09-12T18:46:11.161 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.156+0000 7f610df07600 20 rados_obj.operate() r=0 bl.length=46 2022-09-12T18:46:11.162 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.156+0000 7f610df07600 20 rados->read ofs=0 len=0 2022-09-12T18:46:11.162 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.157+0000 7f610df07600 20 rados_obj.operate() r=0 bl.length=889 2022-09-12T18:46:11.169 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.164+0000 7f610df07600 20 RGWRados::pool_iterate: got zone_info.53c7de48-aca4-45ae-947b-6c71bea55ec9 2022-09-12T18:46:11.169 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.164+0000 7f610df07600 20 RGWRados::pool_iterate: got zonegroup_info.1693a4d5-143e-4f6a-acbd-076182241d97 2022-09-12T18:46:11.169 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.164+0000 7f610df07600 20 RGWRados::pool_iterate: got zone_names.default 2022-09-12T18:46:11.169 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.164+0000 7f610df07600 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-09-12T18:46:11.170 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.164+0000 7f610df07600 20 rados->read ofs=0 len=0 2022-09-12T18:46:11.170 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.164+0000 7f610df07600 20 rados_obj.operate() r=0 bl.length=889 2022-09-12T18:46:11.170 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.164+0000 7f610df07600 20 rados->read ofs=0 len=0 2022-09-12T18:46:11.170 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.164+0000 7f610df07600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:11.171 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.164+0000 7f610df07600 20 rados->read ofs=0 len=0 2022-09-12T18:46:11.171 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.165+0000 7f610df07600 20 rados_obj.operate() r=0 bl.length=46 2022-09-12T18:46:11.171 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.165+0000 7f610df07600 20 rados->read ofs=0 len=0 2022-09-12T18:46:11.171 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:11.165+0000 7f610df07600 20 rados_obj.operate() r=0 bl.length=398 2022-09-12T18:46:11.172 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T18:46:11.172 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "53c7de48-aca4-45ae-947b-6c71bea55ec9", 2022-09-12T18:46:11.173 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "default", 2022-09-12T18:46:11.173 INFO:teuthology.orchestra.run.smithi035.stdout: "domain_root": "default.rgw.meta:root", 2022-09-12T18:46:11.173 INFO:teuthology.orchestra.run.smithi035.stdout: "control_pool": "default.rgw.control", 2022-09-12T18:46:11.173 INFO:teuthology.orchestra.run.smithi035.stdout: "gc_pool": "default.rgw.log:gc", 2022-09-12T18:46:11.174 INFO:teuthology.orchestra.run.smithi035.stdout: "lc_pool": "default.rgw.log:lc", 2022-09-12T18:46:11.174 INFO:teuthology.orchestra.run.smithi035.stdout: "log_pool": "default.rgw.log", 2022-09-12T18:46:11.174 INFO:teuthology.orchestra.run.smithi035.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-09-12T18:46:11.174 INFO:teuthology.orchestra.run.smithi035.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-09-12T18:46:11.175 INFO:teuthology.orchestra.run.smithi035.stdout: "roles_pool": "default.rgw.meta:roles", 2022-09-12T18:46:11.175 INFO:teuthology.orchestra.run.smithi035.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-09-12T18:46:11.175 INFO:teuthology.orchestra.run.smithi035.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-09-12T18:46:11.175 INFO:teuthology.orchestra.run.smithi035.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-09-12T18:46:11.176 INFO:teuthology.orchestra.run.smithi035.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-09-12T18:46:11.176 INFO:teuthology.orchestra.run.smithi035.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-09-12T18:46:11.176 INFO:teuthology.orchestra.run.smithi035.stdout: "otp_pool": "default.rgw.otp", 2022-09-12T18:46:11.176 INFO:teuthology.orchestra.run.smithi035.stdout: "system_key": { 2022-09-12T18:46:11.176 INFO:teuthology.orchestra.run.smithi035.stdout: "access_key": "", 2022-09-12T18:46:11.177 INFO:teuthology.orchestra.run.smithi035.stdout: "secret_key": "" 2022-09-12T18:46:11.177 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:46:11.177 INFO:teuthology.orchestra.run.smithi035.stdout: "placement_pools": [ 2022-09-12T18:46:11.177 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:46:11.178 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "default-placement", 2022-09-12T18:46:11.178 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:46:11.178 INFO:teuthology.orchestra.run.smithi035.stdout: "index_pool": "default.rgw.buckets.index", 2022-09-12T18:46:11.178 INFO:teuthology.orchestra.run.smithi035.stdout: "storage_classes": { 2022-09-12T18:46:11.179 INFO:teuthology.orchestra.run.smithi035.stdout: "STANDARD": { 2022-09-12T18:46:11.179 INFO:teuthology.orchestra.run.smithi035.stdout: "data_pool": "default.rgw.buckets.data", 2022-09-12T18:46:11.179 INFO:teuthology.orchestra.run.smithi035.stdout: "compression_type": "random" 2022-09-12T18:46:11.179 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:46:11.180 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:46:11.180 INFO:teuthology.orchestra.run.smithi035.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-09-12T18:46:11.180 INFO:teuthology.orchestra.run.smithi035.stdout: "index_type": 0 2022-09-12T18:46:11.180 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:46:11.180 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:46:11.181 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T18:46:11.181 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_id": "", 2022-09-12T18:46:11.181 INFO:teuthology.orchestra.run.smithi035.stdout: "notif_pool": "default.rgw.log:notif" 2022-09-12T18:46:11.181 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T18:46:11.183 DEBUG:tasks.util.rgw: json result: {'id': '53c7de48-aca4-45ae-947b-6c71bea55ec9', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-09-12T18:46:11.183 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2022-09-12T18:46:11.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2022-09-12T18:46:11.184 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2022-09-12T18:46:11.265 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:46:11.265 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:46:11.270 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:11.266+0000 7faa8a91f600 20 rados->read ofs=0 len=0 2022-09-12T18:46:12.391 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.387+0000 7faa8a91f600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:12.391 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.387+0000 7faa8a91f600 20 realm 2022-09-12T18:46:12.392 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.387+0000 7faa8a91f600 20 rados->read ofs=0 len=0 2022-09-12T18:46:12.395 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.391+0000 7faa8a91f600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:12.395 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.391+0000 7faa8a91f600 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-09-12T18:46:12.395 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.391+0000 7faa8a91f600 20 rados->read ofs=0 len=0 2022-09-12T18:46:12.397 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.391+0000 7faa8a91f600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:12.397 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.391+0000 7faa8a91f600 20 rados->read ofs=0 len=0 2022-09-12T18:46:12.397 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.392+0000 7faa8a91f600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:12.398 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.392+0000 7faa8a91f600 20 rados->read ofs=0 len=0 2022-09-12T18:46:12.399 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.394+0000 7faa8a91f600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:12.400 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.394+0000 7faa8a91f600 20 rados->read ofs=0 len=0 2022-09-12T18:46:12.401 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.395+0000 7faa8a91f600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:12.402 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.395+0000 7faa8a91f600 10 Creating default zonegroup 2022-09-12T18:46:12.403 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.395+0000 7faa8a91f600 10 couldn't find old data placement pools config, setting up new ones for the zone 2022-09-12T18:46:12.409 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.405+0000 7faa8a91f600 10 WARNING: store->list_zones() returned r=-2 2022-09-12T18:46:12.409 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.405+0000 7faa8a91f600 20 rados->read ofs=0 len=0 2022-09-12T18:46:12.409 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.405+0000 7faa8a91f600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:12.414 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.410+0000 7faa8a91f600 20 rados->read ofs=0 len=0 2022-09-12T18:46:12.415 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.410+0000 7faa8a91f600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:12.415 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.410+0000 7faa8a91f600 10 could not read realm id: (2) No such file or directory 2022-09-12T18:46:12.415 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.410+0000 7faa8a91f600 10 WARNING: failed to set zone as default, r=-22 2022-09-12T18:46:12.415 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.410+0000 7faa8a91f600 20 rados->read ofs=0 len=0 2022-09-12T18:46:12.416 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.410+0000 7faa8a91f600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:12.416 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.413+0000 7faa8a91f600 20 rados->read ofs=0 len=0 2022-09-12T18:46:12.417 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.413+0000 7faa8a91f600 20 rados_obj.operate() r=0 bl.length=873 2022-09-12T18:46:12.418 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.413+0000 7faa8a91f600 20 rados->read ofs=0 len=0 2022-09-12T18:46:12.418 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.414+0000 7faa8a91f600 20 rados_obj.operate() r=0 bl.length=398 2022-09-12T18:46:12.418 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.414+0000 7faa8a91f600 10 Using default name default 2022-09-12T18:46:12.418 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.414+0000 7faa8a91f600 20 rados->read ofs=0 len=0 2022-09-12T18:46:12.420 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.414+0000 7faa8a91f600 20 rados_obj.operate() r=0 bl.length=46 2022-09-12T18:46:12.420 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.414+0000 7faa8a91f600 20 rados->read ofs=0 len=0 2022-09-12T18:46:12.420 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.414+0000 7faa8a91f600 20 rados_obj.operate() r=0 bl.length=873 2022-09-12T18:46:12.421 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.414+0000 7faa8a91f600 20 zone default found 2022-09-12T18:46:12.421 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.414+0000 7faa8a91f600 4 Realm: () 2022-09-12T18:46:12.422 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.414+0000 7faa8a91f600 4 ZoneGroup: default (0111c249-2f80-4ca4-bcee-71408e44b2a2) 2022-09-12T18:46:12.422 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.414+0000 7faa8a91f600 4 Zone: default (ca5bb4fd-3e51-40a4-a57b-badff6fd3866) 2022-09-12T18:46:12.423 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.414+0000 7faa8a91f600 10 cannot find current period zonegroup using local zonegroup configuration 2022-09-12T18:46:12.423 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.414+0000 7faa8a91f600 20 zonegroup default 2022-09-12T18:46:12.423 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.414+0000 7faa8a91f600 20 rados->read ofs=0 len=0 2022-09-12T18:46:12.424 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.415+0000 7faa8a91f600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:12.424 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:12.415+0000 7faa8a91f600 20 rados->read ofs=0 len=0 2022-09-12T18:46:14.411 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:14.407+0000 7faa8a91f600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:14.411 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:14.407+0000 7faa8a91f600 20 rados->read ofs=0 len=0 2022-09-12T18:46:14.412 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:14.407+0000 7faa8a91f600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:14.412 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:14.407+0000 7faa8a91f600 20 started sync module instance, tier type = 2022-09-12T18:46:14.412 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:14.407+0000 7faa8a91f600 20 started zone id=ca5bb4fd-3e51-40a4-a57b-badff6fd3866 (name=default) with tier type = 2022-09-12T18:46:16.452 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.447+0000 7faa8a91f600 20 add_watcher() i=0 2022-09-12T18:46:16.452 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.447+0000 7faa8a91f600 20 add_watcher() i=1 2022-09-12T18:46:16.453 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.447+0000 7faa8a91f600 20 add_watcher() i=2 2022-09-12T18:46:16.453 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.447+0000 7faa8a91f600 20 add_watcher() i=3 2022-09-12T18:46:16.453 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.447+0000 7faa8a91f600 20 add_watcher() i=4 2022-09-12T18:46:16.454 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.447+0000 7faa8a91f600 20 add_watcher() i=5 2022-09-12T18:46:16.454 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.447+0000 7faa8a91f600 20 add_watcher() i=6 2022-09-12T18:46:16.457 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.450+0000 7faa8a91f600 20 add_watcher() i=7 2022-09-12T18:46:16.457 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.450+0000 7faa8a91f600 2 all 8 watchers are set, enabling cache 2022-09-12T18:46:16.457 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.450+0000 7faa8a91f600 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2022-09-12T18:46:16.457 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.450+0000 7faa8a91f600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2022-09-12T18:46:16.458 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.450+0000 7faa8a91f600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-12T18:46:16.458 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.450+0000 7faa8a91f600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2022-09-12T18:46:16.459 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.450+0000 7faa8a91f600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2022-09-12T18:46:16.459 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.450+0000 7faa8a91f600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-12T18:46:16.459 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.450+0000 7faa8a91f600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2022-09-12T18:46:16.459 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.450+0000 7faa8a91f600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2022-09-12T18:46:16.460 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.450+0000 7faa8a91f600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-12T18:46:16.460 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.451+0000 7faa8a91f600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2022-09-12T18:46:16.460 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.451+0000 7faa8a91f600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2022-09-12T18:46:16.460 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.451+0000 7faa8a91f600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-12T18:46:16.461 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.451+0000 7faa8a91f600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2022-09-12T18:46:16.461 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.451+0000 7faa8a91f600 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2022-09-12T18:46:16.461 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.452+0000 7faa8a91f600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-12T18:46:16.462 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.457+0000 7faa730a0700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-12T18:46:16.462 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.457+0000 7faa8a91f600 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-12T18:46:16.462 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:16.457+0000 7faa8a91f600 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-12T18:46:19.423 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.419+0000 7faa8a91f600 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2022-09-12T18:46:19.424 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.419+0000 7faa8a91f600 5 note: GC not initialized 2022-09-12T18:46:19.426 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.420+0000 7faa6e897700 20 reqs_thread_entry: start 2022-09-12T18:46:19.427 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.420+0000 7faa8a91f600 20 init_complete bucket index max shards: 11 2022-09-12T18:46:19.427 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.421+0000 7faa8a91f600 10 rgw notify: Started notification manager with: 1 workers 2022-09-12T18:46:19.427 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.421+0000 7faa6c893700 20 rgw notify: INFO: next queues processing will happen at: Mon Sep 12 18:46:49 2022 2022-09-12T18:46:19.427 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-09-12T18:46:19.428 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.421+0000 7faa6c071700 20 reqs_thread_entry: start 2022-09-12T18:46:19.445 INFO:teuthology.orchestra.run.smithi138.stdout:[] 2022-09-12T18:46:19.446 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.441+0000 7faa8a91f600 20 remove_watcher() i=0 2022-09-12T18:46:19.446 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.441+0000 7faa8a91f600 2 removed watcher, disabling cache 2022-09-12T18:46:19.446 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.442+0000 7faa8a91f600 20 remove_watcher() i=1 2022-09-12T18:46:19.447 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.444+0000 7faa8a91f600 20 remove_watcher() i=2 2022-09-12T18:46:19.450 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.446+0000 7faa8a91f600 20 remove_watcher() i=3 2022-09-12T18:46:19.453 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.449+0000 7faa8a91f600 20 remove_watcher() i=4 2022-09-12T18:46:19.454 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.450+0000 7faa8a91f600 20 remove_watcher() i=5 2022-09-12T18:46:19.456 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.453+0000 7faa8a91f600 20 remove_watcher() i=6 2022-09-12T18:46:19.457 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.454+0000 7faa8a91f600 20 remove_watcher() i=7 2022-09-12T18:46:19.458 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.454+0000 7faa730a0700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-12T18:46:19.463 DEBUG:tasks.util.rgw: json result: [] 2022-09-12T18:46:19.464 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-12T18:46:19.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-12T18:46:19.466 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2022-09-12T18:46:19.519 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:46:19.520 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:46:19.525 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.521+0000 7f7c50500600 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-12T18:46:19.526 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.521+0000 7f7c50500600 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-12T18:46:19.529 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.525+0000 7f7c3547a700 20 reqs_thread_entry: start 2022-09-12T18:46:19.529 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.525+0000 7f7c50500600 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-09-12T18:46:19.529 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.525+0000 7f7c50500600 20 rados->read ofs=0 len=0 2022-09-12T18:46:19.531 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.526+0000 7f7c50500600 20 rados_obj.operate() r=0 bl.length=46 2022-09-12T18:46:19.531 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.526+0000 7f7c50500600 20 rados->read ofs=0 len=0 2022-09-12T18:46:19.532 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.526+0000 7f7c50500600 20 rados_obj.operate() r=0 bl.length=873 2022-09-12T18:46:19.538 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.534+0000 7f7c50500600 20 RGWRados::pool_iterate: got zone_info.ca5bb4fd-3e51-40a4-a57b-badff6fd3866 2022-09-12T18:46:19.539 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.534+0000 7f7c50500600 20 RGWRados::pool_iterate: got zone_names.default 2022-09-12T18:46:19.539 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.534+0000 7f7c50500600 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-09-12T18:46:19.539 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.534+0000 7f7c50500600 20 RGWRados::pool_iterate: got zonegroup_info.0111c249-2f80-4ca4-bcee-71408e44b2a2 2022-09-12T18:46:19.540 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.534+0000 7f7c50500600 20 rados->read ofs=0 len=0 2022-09-12T18:46:19.540 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.534+0000 7f7c50500600 20 rados_obj.operate() r=0 bl.length=873 2022-09-12T18:46:19.540 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.534+0000 7f7c50500600 20 rados->read ofs=0 len=0 2022-09-12T18:46:19.540 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.534+0000 7f7c50500600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:19.541 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.534+0000 7f7c50500600 20 rados->read ofs=0 len=0 2022-09-12T18:46:19.541 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.535+0000 7f7c50500600 20 rados_obj.operate() r=0 bl.length=46 2022-09-12T18:46:19.541 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.535+0000 7f7c50500600 20 rados->read ofs=0 len=0 2022-09-12T18:46:19.541 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.535+0000 7f7c50500600 20 rados_obj.operate() r=0 bl.length=398 2022-09-12T18:46:19.542 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:46:19.542 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "ca5bb4fd-3e51-40a4-a57b-badff6fd3866", 2022-09-12T18:46:19.542 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "default", 2022-09-12T18:46:19.543 INFO:teuthology.orchestra.run.smithi138.stdout: "domain_root": "default.rgw.meta:root", 2022-09-12T18:46:19.543 INFO:teuthology.orchestra.run.smithi138.stdout: "control_pool": "default.rgw.control", 2022-09-12T18:46:19.543 INFO:teuthology.orchestra.run.smithi138.stdout: "gc_pool": "default.rgw.log:gc", 2022-09-12T18:46:19.543 INFO:teuthology.orchestra.run.smithi138.stdout: "lc_pool": "default.rgw.log:lc", 2022-09-12T18:46:19.544 INFO:teuthology.orchestra.run.smithi138.stdout: "log_pool": "default.rgw.log", 2022-09-12T18:46:19.544 INFO:teuthology.orchestra.run.smithi138.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-09-12T18:46:19.544 INFO:teuthology.orchestra.run.smithi138.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-09-12T18:46:19.544 INFO:teuthology.orchestra.run.smithi138.stdout: "roles_pool": "default.rgw.meta:roles", 2022-09-12T18:46:19.545 INFO:teuthology.orchestra.run.smithi138.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-09-12T18:46:19.545 INFO:teuthology.orchestra.run.smithi138.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-09-12T18:46:19.545 INFO:teuthology.orchestra.run.smithi138.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-09-12T18:46:19.545 INFO:teuthology.orchestra.run.smithi138.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-09-12T18:46:19.545 INFO:teuthology.orchestra.run.smithi138.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-09-12T18:46:19.546 INFO:teuthology.orchestra.run.smithi138.stdout: "otp_pool": "default.rgw.otp", 2022-09-12T18:46:19.546 INFO:teuthology.orchestra.run.smithi138.stdout: "system_key": { 2022-09-12T18:46:19.546 INFO:teuthology.orchestra.run.smithi138.stdout: "access_key": "", 2022-09-12T18:46:19.546 INFO:teuthology.orchestra.run.smithi138.stdout: "secret_key": "" 2022-09-12T18:46:19.547 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:46:19.547 INFO:teuthology.orchestra.run.smithi138.stdout: "placement_pools": [ 2022-09-12T18:46:19.547 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:46:19.547 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "default-placement", 2022-09-12T18:46:19.548 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:46:19.548 INFO:teuthology.orchestra.run.smithi138.stdout: "index_pool": "default.rgw.buckets.index", 2022-09-12T18:46:19.548 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_classes": { 2022-09-12T18:46:19.548 INFO:teuthology.orchestra.run.smithi138.stdout: "STANDARD": { 2022-09-12T18:46:19.549 INFO:teuthology.orchestra.run.smithi138.stdout: "data_pool": "default.rgw.buckets.data", 2022-09-12T18:46:19.549 INFO:teuthology.orchestra.run.smithi138.stdout: "compression_type": "random" 2022-09-12T18:46:19.549 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:46:19.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:46:19.549 INFO:teuthology.orchestra.run.smithi138.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-09-12T18:46:19.550 INFO:teuthology.orchestra.run.smithi138.stdout: "index_type": 0 2022-09-12T18:46:19.550 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:46:19.550 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:46:19.550 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T18:46:19.551 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "", 2022-09-12T18:46:19.551 INFO:teuthology.orchestra.run.smithi138.stdout: "notif_pool": "default.rgw.log:notif" 2022-09-12T18:46:19.551 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:46:19.552 DEBUG:tasks.util.rgw: json result: {'id': 'ca5bb4fd-3e51-40a4-a57b-badff6fd3866', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-09-12T18:46:19.553 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2022-09-12T18:46:19.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2022-09-12T18:46:19.554 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2022-09-12T18:46:19.598 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:46:19.598 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:46:19.603 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.599+0000 7fd7173b3600 20 rados->read ofs=0 len=0 2022-09-12T18:46:19.605 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.600+0000 7fd7173b3600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:19.605 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.600+0000 7fd7173b3600 20 realm 2022-09-12T18:46:19.605 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.600+0000 7fd7173b3600 20 rados->read ofs=0 len=0 2022-09-12T18:46:19.605 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.600+0000 7fd7173b3600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:19.605 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.600+0000 7fd7173b3600 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-09-12T18:46:19.606 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.600+0000 7fd7173b3600 20 rados->read ofs=0 len=0 2022-09-12T18:46:19.606 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.601+0000 7fd7173b3600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:19.607 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.601+0000 7fd7173b3600 20 rados->read ofs=0 len=0 2022-09-12T18:46:19.607 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.601+0000 7fd7173b3600 20 rados_obj.operate() r=0 bl.length=46 2022-09-12T18:46:19.607 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.601+0000 7fd7173b3600 20 rados->read ofs=0 len=0 2022-09-12T18:46:19.608 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.602+0000 7fd7173b3600 20 rados_obj.operate() r=0 bl.length=889 2022-09-12T18:46:19.608 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.602+0000 7fd7173b3600 20 searching for the correct realm 2022-09-12T18:46:19.614 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.608+0000 7fd7173b3600 20 RGWRados::pool_iterate: got zone_info.ca5bb4fd-3e51-40a4-a57b-badff6fd3866 2022-09-12T18:46:19.614 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.608+0000 7fd7173b3600 20 RGWRados::pool_iterate: got zone_names.default 2022-09-12T18:46:19.615 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.608+0000 7fd7173b3600 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-09-12T18:46:19.615 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.608+0000 7fd7173b3600 20 RGWRados::pool_iterate: got zonegroup_info.0111c249-2f80-4ca4-bcee-71408e44b2a2 2022-09-12T18:46:19.615 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.608+0000 7fd7173b3600 20 rados->read ofs=0 len=0 2022-09-12T18:46:19.615 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.609+0000 7fd7173b3600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:19.616 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.609+0000 7fd7173b3600 20 rados->read ofs=0 len=0 2022-09-12T18:46:19.616 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.609+0000 7fd7173b3600 20 rados_obj.operate() r=0 bl.length=46 2022-09-12T18:46:19.616 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.609+0000 7fd7173b3600 20 rados->read ofs=0 len=0 2022-09-12T18:46:19.616 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.609+0000 7fd7173b3600 20 rados_obj.operate() r=0 bl.length=398 2022-09-12T18:46:19.617 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.609+0000 7fd7173b3600 20 zone default found 2022-09-12T18:46:19.617 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.609+0000 7fd7173b3600 4 Realm: () 2022-09-12T18:46:19.617 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.609+0000 7fd7173b3600 4 ZoneGroup: default (0111c249-2f80-4ca4-bcee-71408e44b2a2) 2022-09-12T18:46:19.617 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.609+0000 7fd7173b3600 4 Zone: default (ca5bb4fd-3e51-40a4-a57b-badff6fd3866) 2022-09-12T18:46:19.617 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.609+0000 7fd7173b3600 10 cannot find current period zonegroup using local zonegroup configuration 2022-09-12T18:46:19.618 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.609+0000 7fd7173b3600 20 zonegroup default 2022-09-12T18:46:19.618 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.609+0000 7fd7173b3600 20 rados->read ofs=0 len=0 2022-09-12T18:46:19.618 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.609+0000 7fd7173b3600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:19.618 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.609+0000 7fd7173b3600 20 rados->read ofs=0 len=0 2022-09-12T18:46:19.619 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.609+0000 7fd7173b3600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:19.619 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.609+0000 7fd7173b3600 20 rados->read ofs=0 len=0 2022-09-12T18:46:19.619 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.610+0000 7fd7173b3600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:19.620 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.610+0000 7fd7173b3600 20 started sync module instance, tier type = 2022-09-12T18:46:19.620 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.610+0000 7fd7173b3600 20 started zone id=ca5bb4fd-3e51-40a4-a57b-badff6fd3866 (name=default) with tier type = 2022-09-12T18:46:19.638 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.634+0000 7fd7173b3600 20 add_watcher() i=0 2022-09-12T18:46:19.638 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.634+0000 7fd7173b3600 20 add_watcher() i=1 2022-09-12T18:46:19.638 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.634+0000 7fd7173b3600 20 add_watcher() i=2 2022-09-12T18:46:19.639 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.634+0000 7fd7173b3600 20 add_watcher() i=3 2022-09-12T18:46:19.639 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.634+0000 7fd7173b3600 20 add_watcher() i=4 2022-09-12T18:46:19.639 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.634+0000 7fd7173b3600 20 add_watcher() i=5 2022-09-12T18:46:19.639 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.634+0000 7fd7173b3600 20 add_watcher() i=6 2022-09-12T18:46:19.640 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.636+0000 7fd7173b3600 20 add_watcher() i=7 2022-09-12T18:46:19.640 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.636+0000 7fd7173b3600 2 all 8 watchers are set, enabling cache 2022-09-12T18:46:19.643 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.639+0000 7fd6ffb34700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-12T18:46:19.643 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.639+0000 7fd7173b3600 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-12T18:46:19.644 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.639+0000 7fd7173b3600 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-12T18:46:19.646 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.643+0000 7fd7173b3600 5 note: GC not initialized 2022-09-12T18:46:19.648 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.644+0000 7fd6fb32b700 20 reqs_thread_entry: start 2022-09-12T18:46:19.648 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.644+0000 7fd7173b3600 20 init_complete bucket index max shards: 11 2022-09-12T18:46:19.648 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.644+0000 7fd7173b3600 10 rgw notify: Started notification manager with: 1 workers 2022-09-12T18:46:19.648 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.644+0000 7fd6f9327700 20 rgw notify: INFO: next queues processing will happen at: Mon Sep 12 18:46:50 2022 2022-09-12T18:46:19.649 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-09-12T18:46:19.649 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.644+0000 7fd6f8b05700 20 reqs_thread_entry: start 2022-09-12T18:46:19.655 INFO:teuthology.orchestra.run.smithi138.stdout:[] 2022-09-12T18:46:19.655 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.651+0000 7fd7173b3600 20 remove_watcher() i=0 2022-09-12T18:46:19.656 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.651+0000 7fd7173b3600 2 removed watcher, disabling cache 2022-09-12T18:46:19.658 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.654+0000 7fd7173b3600 20 remove_watcher() i=1 2022-09-12T18:46:19.660 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.656+0000 7fd7173b3600 20 remove_watcher() i=2 2022-09-12T18:46:19.663 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.659+0000 7fd7173b3600 20 remove_watcher() i=3 2022-09-12T18:46:19.666 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.662+0000 7fd7173b3600 20 remove_watcher() i=4 2022-09-12T18:46:19.668 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.665+0000 7fd7173b3600 20 remove_watcher() i=5 2022-09-12T18:46:19.669 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.665+0000 7fd7173b3600 20 remove_watcher() i=6 2022-09-12T18:46:19.671 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.667+0000 7fd7173b3600 20 remove_watcher() i=7 2022-09-12T18:46:19.672 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.668+0000 7fd6ffb34700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-12T18:46:19.677 DEBUG:tasks.util.rgw: json result: [] 2022-09-12T18:46:19.678 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-12T18:46:19.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-12T18:46:19.678 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2022-09-12T18:46:19.727 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:46:19.728 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:46:19.733 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.729+0000 7ff4582ca600 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-12T18:46:19.734 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.729+0000 7ff4582ca600 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-12T18:46:19.736 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.733+0000 7ff4582ca600 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-09-12T18:46:19.737 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.733+0000 7ff43d244700 20 reqs_thread_entry: start 2022-09-12T18:46:19.737 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.733+0000 7ff4582ca600 20 rados->read ofs=0 len=0 2022-09-12T18:46:19.738 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.733+0000 7ff4582ca600 20 rados_obj.operate() r=0 bl.length=46 2022-09-12T18:46:19.739 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.733+0000 7ff4582ca600 20 rados->read ofs=0 len=0 2022-09-12T18:46:19.739 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.734+0000 7ff4582ca600 20 rados_obj.operate() r=0 bl.length=889 2022-09-12T18:46:19.746 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.741+0000 7ff4582ca600 20 RGWRados::pool_iterate: got zone_info.ca5bb4fd-3e51-40a4-a57b-badff6fd3866 2022-09-12T18:46:19.746 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.741+0000 7ff4582ca600 20 RGWRados::pool_iterate: got zone_names.default 2022-09-12T18:46:19.746 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.741+0000 7ff4582ca600 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-09-12T18:46:19.747 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.741+0000 7ff4582ca600 20 RGWRados::pool_iterate: got zonegroup_info.0111c249-2f80-4ca4-bcee-71408e44b2a2 2022-09-12T18:46:19.747 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.741+0000 7ff4582ca600 20 rados->read ofs=0 len=0 2022-09-12T18:46:19.748 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.741+0000 7ff4582ca600 20 rados_obj.operate() r=0 bl.length=889 2022-09-12T18:46:19.748 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.741+0000 7ff4582ca600 20 rados->read ofs=0 len=0 2022-09-12T18:46:19.748 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.741+0000 7ff4582ca600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-12T18:46:19.749 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.741+0000 7ff4582ca600 20 rados->read ofs=0 len=0 2022-09-12T18:46:19.749 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.741+0000 7ff4582ca600 20 rados_obj.operate() r=0 bl.length=46 2022-09-12T18:46:19.749 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.741+0000 7ff4582ca600 20 rados->read ofs=0 len=0 2022-09-12T18:46:19.750 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:19.741+0000 7ff4582ca600 20 rados_obj.operate() r=0 bl.length=398 2022-09-12T18:46:19.752 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:46:19.752 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "ca5bb4fd-3e51-40a4-a57b-badff6fd3866", 2022-09-12T18:46:19.752 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "default", 2022-09-12T18:46:19.753 INFO:teuthology.orchestra.run.smithi138.stdout: "domain_root": "default.rgw.meta:root", 2022-09-12T18:46:19.753 INFO:teuthology.orchestra.run.smithi138.stdout: "control_pool": "default.rgw.control", 2022-09-12T18:46:19.753 INFO:teuthology.orchestra.run.smithi138.stdout: "gc_pool": "default.rgw.log:gc", 2022-09-12T18:46:19.753 INFO:teuthology.orchestra.run.smithi138.stdout: "lc_pool": "default.rgw.log:lc", 2022-09-12T18:46:19.754 INFO:teuthology.orchestra.run.smithi138.stdout: "log_pool": "default.rgw.log", 2022-09-12T18:46:19.754 INFO:teuthology.orchestra.run.smithi138.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-09-12T18:46:19.754 INFO:teuthology.orchestra.run.smithi138.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-09-12T18:46:19.754 INFO:teuthology.orchestra.run.smithi138.stdout: "roles_pool": "default.rgw.meta:roles", 2022-09-12T18:46:19.755 INFO:teuthology.orchestra.run.smithi138.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-09-12T18:46:19.755 INFO:teuthology.orchestra.run.smithi138.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-09-12T18:46:19.755 INFO:teuthology.orchestra.run.smithi138.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-09-12T18:46:19.755 INFO:teuthology.orchestra.run.smithi138.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-09-12T18:46:19.755 INFO:teuthology.orchestra.run.smithi138.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-09-12T18:46:19.756 INFO:teuthology.orchestra.run.smithi138.stdout: "otp_pool": "default.rgw.otp", 2022-09-12T18:46:19.756 INFO:teuthology.orchestra.run.smithi138.stdout: "system_key": { 2022-09-12T18:46:19.756 INFO:teuthology.orchestra.run.smithi138.stdout: "access_key": "", 2022-09-12T18:46:19.757 INFO:teuthology.orchestra.run.smithi138.stdout: "secret_key": "" 2022-09-12T18:46:19.757 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:46:19.757 INFO:teuthology.orchestra.run.smithi138.stdout: "placement_pools": [ 2022-09-12T18:46:19.757 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:46:19.757 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "default-placement", 2022-09-12T18:46:19.758 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:46:19.758 INFO:teuthology.orchestra.run.smithi138.stdout: "index_pool": "default.rgw.buckets.index", 2022-09-12T18:46:19.758 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_classes": { 2022-09-12T18:46:19.758 INFO:teuthology.orchestra.run.smithi138.stdout: "STANDARD": { 2022-09-12T18:46:19.759 INFO:teuthology.orchestra.run.smithi138.stdout: "data_pool": "default.rgw.buckets.data", 2022-09-12T18:46:19.759 INFO:teuthology.orchestra.run.smithi138.stdout: "compression_type": "random" 2022-09-12T18:46:19.759 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:46:19.759 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:46:19.759 INFO:teuthology.orchestra.run.smithi138.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-09-12T18:46:19.760 INFO:teuthology.orchestra.run.smithi138.stdout: "index_type": 0 2022-09-12T18:46:19.760 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:46:19.760 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:46:19.760 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T18:46:19.761 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "", 2022-09-12T18:46:19.761 INFO:teuthology.orchestra.run.smithi138.stdout: "notif_pool": "default.rgw.log:notif" 2022-09-12T18:46:19.761 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:46:19.761 DEBUG:tasks.util.rgw: json result: {'id': 'ca5bb4fd-3e51-40a4-a57b-badff6fd3866', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-09-12T18:46:19.762 INFO:tasks.rgw:Starting rgw... 2022-09-12T18:46:19.762 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2022-09-12T18:46:19.762 INFO:tasks.rgw:Using beast as radosgw frontend 2022-09-12T18:46:19.763 INFO:tasks.rgw.c1.client.0:Restarting daemon 2022-09-12T18:46:19.764 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2022-09-12T18:46:19.767 INFO:tasks.rgw.c1.client.0:Started 2022-09-12T18:46:19.767 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2022-09-12T18:46:19.767 INFO:tasks.rgw:Using beast as radosgw frontend 2022-09-12T18:46:19.768 INFO:tasks.rgw.c1.client.1:Restarting daemon 2022-09-12T18:46:19.768 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2022-09-12T18:46:19.771 INFO:tasks.rgw.c1.client.1:Started 2022-09-12T18:46:19.771 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2022-09-12T18:46:19.771 INFO:tasks.rgw:Using beast as radosgw frontend 2022-09-12T18:46:19.772 INFO:tasks.rgw.c2.client.0:Restarting daemon 2022-09-12T18:46:19.772 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2022-09-12T18:46:19.774 INFO:tasks.rgw.c2.client.0:Started 2022-09-12T18:46:19.775 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2022-09-12T18:46:19.775 INFO:tasks.rgw:Using beast as radosgw frontend 2022-09-12T18:46:19.775 INFO:tasks.rgw.c2.client.1:Restarting daemon 2022-09-12T18:46:19.776 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2022-09-12T18:46:19.778 INFO:tasks.rgw.c2.client.1:Started 2022-09-12T18:46:19.778 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi035.front.sepia.ceph.com:8000/ 2022-09-12T18:46:19.779 DEBUG:teuthology.orchestra.run.smithi035:> curl http://smithi035.front.sepia.ceph.com:8000/ 2022-09-12T18:46:19.878 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-12T18:46:19.879 INFO:teuthology.orchestra.run.smithi035.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-12T18:46:19.880 INFO:teuthology.orchestra.run.smithi035.stderr: Dload Upload Total Spent Left Speed 2022-09-12T18:46:19.880 INFO:teuthology.orchestra.run.smithi035.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi035.front.sepia.ceph.com port 8000: Connection refused 2022-09-12T18:46:20.881 DEBUG:teuthology.orchestra.run.smithi035:> curl http://smithi035.front.sepia.ceph.com:8000/ 2022-09-12T18:46:20.908 INFO:teuthology.orchestra.run.smithi035.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-12T18:46:20.908 INFO:teuthology.orchestra.run.smithi035.stderr: Dload Upload Total Spent Left Speed 2022-09-12T18:46:20.909 INFO:teuthology.orchestra.run.smithi035.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-12T18:46:20.910 INFO:teuthology.orchestra.run.smithi035.stdout:anonymous 2022-09-12T18:46:20.911 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi035.front.sepia.ceph.com:8001/ 2022-09-12T18:46:20.912 DEBUG:teuthology.orchestra.run.smithi035:> curl http://smithi035.front.sepia.ceph.com:8001/ 2022-09-12T18:46:20.974 INFO:teuthology.orchestra.run.smithi035.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-12T18:46:20.974 INFO:teuthology.orchestra.run.smithi035.stderr: Dload Upload Total Spent Left Speed 2022-09-12T18:46:20.975 INFO:teuthology.orchestra.run.smithi035.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-12T18:46:20.976 INFO:teuthology.orchestra.run.smithi035.stdout:anonymous 2022-09-12T18:46:20.976 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi138.front.sepia.ceph.com:8000/ 2022-09-12T18:46:20.977 DEBUG:teuthology.orchestra.run.smithi138:> curl http://smithi138.front.sepia.ceph.com:8000/ 2022-09-12T18:46:21.012 INFO:teuthology.orchestra.run.smithi138.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-12T18:46:21.013 INFO:teuthology.orchestra.run.smithi138.stderr: Dload Upload Total Spent Left Speed 2022-09-12T18:46:21.013 INFO:teuthology.orchestra.run.smithi138.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-12T18:46:21.015 INFO:teuthology.orchestra.run.smithi138.stdout:anonymous 2022-09-12T18:46:21.016 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi138.front.sepia.ceph.com:8001/ 2022-09-12T18:46:21.016 DEBUG:teuthology.orchestra.run.smithi138:> curl http://smithi138.front.sepia.ceph.com:8001/ 2022-09-12T18:46:21.078 INFO:teuthology.orchestra.run.smithi138.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-12T18:46:21.078 INFO:teuthology.orchestra.run.smithi138.stderr: Dload Upload Total Spent Left Speed 2022-09-12T18:46:21.079 INFO:teuthology.orchestra.run.smithi138.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-12T18:46:21.080 INFO:teuthology.orchestra.run.smithi138.stdout:anonymous 2022-09-12T18:46:21.080 INFO:teuthology.run_tasks:Running task rgw-multisite... 2022-09-12T18:46:21.340 INFO:tasks.rgw_multisite:creating realm.. 2022-09-12T18:46:21.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:21.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:21.341 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T18:46:21.391 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:46:21.392 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:46:21.421 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T18:46:21.422 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T18:46:21.422 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-realm", 2022-09-12T18:46:21.422 INFO:teuthology.orchestra.run.smithi035.stdout: "current_period": "2e627c13-04da-45e8-bcee-0ed7248cedbf", 2022-09-12T18:46:21.423 INFO:teuthology.orchestra.run.smithi035.stdout: "epoch": 1 2022-09-12T18:46:21.423 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T18:46:21.424 INFO:tasks.rgw_multisite:creating master zonegroup.. 2022-09-12T18:46:21.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:21.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:21.426 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T18:46:21.474 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:46:21.475 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:46:21.486 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:21.482+0000 7f890f387600 -1 old format 2022-09-12T18:46:21.491 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T18:46:21.492 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "test-zonegroup", 2022-09-12T18:46:21.492 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zonegroup", 2022-09-12T18:46:21.492 INFO:teuthology.orchestra.run.smithi035.stdout: "api_name": "", 2022-09-12T18:46:21.493 INFO:teuthology.orchestra.run.smithi035.stdout: "is_master": true, 2022-09-12T18:46:21.493 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T18:46:21.494 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T18:46:21.494 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T18:46:21.494 INFO:teuthology.orchestra.run.smithi035.stdout: "hostnames": [], 2022-09-12T18:46:21.495 INFO:teuthology.orchestra.run.smithi035.stdout: "hostnames_s3website": [], 2022-09-12T18:46:21.495 INFO:teuthology.orchestra.run.smithi035.stdout: "master_zone": "", 2022-09-12T18:46:21.495 INFO:teuthology.orchestra.run.smithi035.stdout: "zones": [], 2022-09-12T18:46:21.495 INFO:teuthology.orchestra.run.smithi035.stdout: "placement_targets": [], 2022-09-12T18:46:21.496 INFO:teuthology.orchestra.run.smithi035.stdout: "default_placement": "", 2022-09-12T18:46:21.496 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T18:46:21.496 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_policy": { 2022-09-12T18:46:21.496 INFO:teuthology.orchestra.run.smithi035.stdout: "groups": [] 2022-09-12T18:46:21.497 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:46:21.497 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled_features": [ 2022-09-12T18:46:21.497 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T18:46:21.497 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T18:46:21.498 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T18:46:21.499 INFO:tasks.rgw_multisite:creating master zone.. 2022-09-12T18:46:21.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'RDIEFLSOLOEHUH1F', '--secret', 'LDCKRd1RecZNgErNcVDnUigDhC7QggCQ', '--endpoints', 'http://smithi035.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:21.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'RDIEFLSOLOEHUH1F', '--secret', 'LDCKRd1RecZNgErNcVDnUigDhC7QggCQ', '--endpoints', 'http://smithi035.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:21.500 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key RDIEFLSOLOEHUH1F --secret LDCKRd1RecZNgErNcVDnUigDhC7QggCQ --endpoints http://smithi035.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T18:46:21.595 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:46:21.595 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:46:21.626 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T18:46:21.627 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T18:46:21.627 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone1", 2022-09-12T18:46:21.627 INFO:teuthology.orchestra.run.smithi035.stdout: "domain_root": "test-zone1.rgw.meta:root", 2022-09-12T18:46:21.628 INFO:teuthology.orchestra.run.smithi035.stdout: "control_pool": "test-zone1.rgw.control", 2022-09-12T18:46:21.628 INFO:teuthology.orchestra.run.smithi035.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2022-09-12T18:46:21.628 INFO:teuthology.orchestra.run.smithi035.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2022-09-12T18:46:21.628 INFO:teuthology.orchestra.run.smithi035.stdout: "log_pool": "test-zone1.rgw.log", 2022-09-12T18:46:21.629 INFO:teuthology.orchestra.run.smithi035.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2022-09-12T18:46:21.629 INFO:teuthology.orchestra.run.smithi035.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2022-09-12T18:46:21.629 INFO:teuthology.orchestra.run.smithi035.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2022-09-12T18:46:21.629 INFO:teuthology.orchestra.run.smithi035.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2022-09-12T18:46:21.630 INFO:teuthology.orchestra.run.smithi035.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2022-09-12T18:46:21.630 INFO:teuthology.orchestra.run.smithi035.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2022-09-12T18:46:21.630 INFO:teuthology.orchestra.run.smithi035.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2022-09-12T18:46:21.630 INFO:teuthology.orchestra.run.smithi035.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2022-09-12T18:46:21.630 INFO:teuthology.orchestra.run.smithi035.stdout: "otp_pool": "test-zone1.rgw.otp", 2022-09-12T18:46:21.631 INFO:teuthology.orchestra.run.smithi035.stdout: "system_key": { 2022-09-12T18:46:21.631 INFO:teuthology.orchestra.run.smithi035.stdout: "access_key": "RDIEFLSOLOEHUH1F", 2022-09-12T18:46:21.631 INFO:teuthology.orchestra.run.smithi035.stdout: "secret_key": "LDCKRd1RecZNgErNcVDnUigDhC7QggCQ" 2022-09-12T18:46:21.632 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:46:21.632 INFO:teuthology.orchestra.run.smithi035.stdout: "placement_pools": [ 2022-09-12T18:46:21.632 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:46:21.632 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "default-placement", 2022-09-12T18:46:21.632 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:46:21.633 INFO:teuthology.orchestra.run.smithi035.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2022-09-12T18:46:21.633 INFO:teuthology.orchestra.run.smithi035.stdout: "storage_classes": { 2022-09-12T18:46:21.633 INFO:teuthology.orchestra.run.smithi035.stdout: "STANDARD": { 2022-09-12T18:46:21.633 INFO:teuthology.orchestra.run.smithi035.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2022-09-12T18:46:21.634 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:46:21.634 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:46:21.634 INFO:teuthology.orchestra.run.smithi035.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2022-09-12T18:46:21.634 INFO:teuthology.orchestra.run.smithi035.stdout: "index_type": 0 2022-09-12T18:46:21.635 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:46:21.635 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:46:21.635 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T18:46:21.635 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T18:46:21.635 INFO:teuthology.orchestra.run.smithi035.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2022-09-12T18:46:21.636 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T18:46:21.637 DEBUG:teuthology.orchestra.run.smithi035:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2022-09-12T18:46:22.913 INFO:teuthology.orchestra.run.smithi035.stderr:pool 'test-zone1.rgw.buckets.data' created 2022-09-12T18:46:22.936 DEBUG:teuthology.orchestra.run.smithi035:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2022-09-12T18:46:23.939 INFO:teuthology.orchestra.run.smithi035.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2022-09-12T18:46:23.956 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:23.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:23.957 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T18:46:24.014 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:46:24.015 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:46:24.040 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T18:46:24.040 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T18:46:24.040 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone1", 2022-09-12T18:46:24.040 INFO:teuthology.orchestra.run.smithi035.stdout: "domain_root": "test-zone1.rgw.meta:root", 2022-09-12T18:46:24.041 INFO:teuthology.orchestra.run.smithi035.stdout: "control_pool": "test-zone1.rgw.control", 2022-09-12T18:46:24.041 INFO:teuthology.orchestra.run.smithi035.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2022-09-12T18:46:24.041 INFO:teuthology.orchestra.run.smithi035.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2022-09-12T18:46:24.041 INFO:teuthology.orchestra.run.smithi035.stdout: "log_pool": "test-zone1.rgw.log", 2022-09-12T18:46:24.042 INFO:teuthology.orchestra.run.smithi035.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2022-09-12T18:46:24.042 INFO:teuthology.orchestra.run.smithi035.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2022-09-12T18:46:24.042 INFO:teuthology.orchestra.run.smithi035.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2022-09-12T18:46:24.042 INFO:teuthology.orchestra.run.smithi035.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2022-09-12T18:46:24.043 INFO:teuthology.orchestra.run.smithi035.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2022-09-12T18:46:24.043 INFO:teuthology.orchestra.run.smithi035.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2022-09-12T18:46:24.043 INFO:teuthology.orchestra.run.smithi035.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2022-09-12T18:46:24.043 INFO:teuthology.orchestra.run.smithi035.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2022-09-12T18:46:24.044 INFO:teuthology.orchestra.run.smithi035.stdout: "otp_pool": "test-zone1.rgw.otp", 2022-09-12T18:46:24.044 INFO:teuthology.orchestra.run.smithi035.stdout: "system_key": { 2022-09-12T18:46:24.044 INFO:teuthology.orchestra.run.smithi035.stdout: "access_key": "RDIEFLSOLOEHUH1F", 2022-09-12T18:46:24.044 INFO:teuthology.orchestra.run.smithi035.stdout: "secret_key": "LDCKRd1RecZNgErNcVDnUigDhC7QggCQ" 2022-09-12T18:46:24.044 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:46:24.045 INFO:teuthology.orchestra.run.smithi035.stdout: "placement_pools": [ 2022-09-12T18:46:24.045 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:46:24.045 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "default-placement", 2022-09-12T18:46:24.045 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:46:24.046 INFO:teuthology.orchestra.run.smithi035.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2022-09-12T18:46:24.046 INFO:teuthology.orchestra.run.smithi035.stdout: "storage_classes": { 2022-09-12T18:46:24.047 INFO:teuthology.orchestra.run.smithi035.stdout: "STANDARD": { 2022-09-12T18:46:24.047 INFO:teuthology.orchestra.run.smithi035.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2022-09-12T18:46:24.047 INFO:teuthology.orchestra.run.smithi035.stdout: "compression_type": "random" 2022-09-12T18:46:24.047 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:46:24.048 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:46:24.048 INFO:teuthology.orchestra.run.smithi035.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2022-09-12T18:46:24.048 INFO:teuthology.orchestra.run.smithi035.stdout: "index_type": 0 2022-09-12T18:46:24.048 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:46:24.049 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:46:24.049 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T18:46:24.049 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_id": "", 2022-09-12T18:46:24.049 INFO:teuthology.orchestra.run.smithi035.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2022-09-12T18:46:24.049 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T18:46:24.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:24.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:24.052 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T18:46:24.102 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:46:24.102 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:46:24.111 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:24.107+0000 7f22ad3a9600 0 period (2e627c13-04da-45e8-bcee-0ed7248cedbf does not have zone fb29b4fb-1f78-4524-ab80-6a5e9e01ab47 configured 2022-09-12T18:46:31.007 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T18:46:31.008 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:46:31.008 INFO:teuthology.orchestra.run.smithi035.stdout: "epoch": 1, 2022-09-12T18:46:31.008 INFO:teuthology.orchestra.run.smithi035.stdout: "predecessor_uuid": "2e627c13-04da-45e8-bcee-0ed7248cedbf", 2022-09-12T18:46:31.008 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": [], 2022-09-12T18:46:31.009 INFO:teuthology.orchestra.run.smithi035.stdout: "period_map": { 2022-09-12T18:46:31.009 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:46:31.009 INFO:teuthology.orchestra.run.smithi035.stdout: "zonegroups": [ 2022-09-12T18:46:31.009 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:46:31.010 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "test-zonegroup", 2022-09-12T18:46:31.010 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zonegroup", 2022-09-12T18:46:31.010 INFO:teuthology.orchestra.run.smithi035.stdout: "api_name": "", 2022-09-12T18:46:31.010 INFO:teuthology.orchestra.run.smithi035.stdout: "is_master": true, 2022-09-12T18:46:31.011 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T18:46:31.011 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T18:46:31.011 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T18:46:31.011 INFO:teuthology.orchestra.run.smithi035.stdout: "hostnames": [], 2022-09-12T18:46:31.011 INFO:teuthology.orchestra.run.smithi035.stdout: "hostnames_s3website": [], 2022-09-12T18:46:31.012 INFO:teuthology.orchestra.run.smithi035.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T18:46:31.012 INFO:teuthology.orchestra.run.smithi035.stdout: "zones": [ 2022-09-12T18:46:31.012 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:46:31.012 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T18:46:31.013 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone1", 2022-09-12T18:46:31.013 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T18:46:31.013 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T18:46:31.013 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T18:46:31.014 INFO:teuthology.orchestra.run.smithi035.stdout: "log_meta": false, 2022-09-12T18:46:31.014 INFO:teuthology.orchestra.run.smithi035.stdout: "log_data": false, 2022-09-12T18:46:31.014 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_index_max_shards": 11, 2022-09-12T18:46:31.014 INFO:teuthology.orchestra.run.smithi035.stdout: "read_only": false, 2022-09-12T18:46:31.015 INFO:teuthology.orchestra.run.smithi035.stdout: "tier_type": "", 2022-09-12T18:46:31.015 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from_all": true, 2022-09-12T18:46:31.015 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from": [], 2022-09-12T18:46:31.015 INFO:teuthology.orchestra.run.smithi035.stdout: "redirect_zone": "", 2022-09-12T18:46:31.015 INFO:teuthology.orchestra.run.smithi035.stdout: "supported_features": [ 2022-09-12T18:46:31.016 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T18:46:31.016 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T18:46:31.016 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:46:31.016 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T18:46:31.017 INFO:teuthology.orchestra.run.smithi035.stdout: "placement_targets": [ 2022-09-12T18:46:31.017 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:46:31.017 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "default-placement", 2022-09-12T18:46:31.017 INFO:teuthology.orchestra.run.smithi035.stdout: "tags": [], 2022-09-12T18:46:31.018 INFO:teuthology.orchestra.run.smithi035.stdout: "storage_classes": [ 2022-09-12T18:46:31.018 INFO:teuthology.orchestra.run.smithi035.stdout: "STANDARD" 2022-09-12T18:46:31.018 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T18:46:31.019 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:46:31.019 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T18:46:31.019 INFO:teuthology.orchestra.run.smithi035.stdout: "default_placement": "default-placement", 2022-09-12T18:46:31.019 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T18:46:31.019 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_policy": { 2022-09-12T18:46:31.020 INFO:teuthology.orchestra.run.smithi035.stdout: "groups": [] 2022-09-12T18:46:31.020 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:46:31.020 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled_features": [ 2022-09-12T18:46:31.020 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T18:46:31.021 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T18:46:31.021 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:46:31.021 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T18:46:31.021 INFO:teuthology.orchestra.run.smithi035.stdout: "short_zone_ids": [ 2022-09-12T18:46:31.021 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:46:31.022 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T18:46:31.022 INFO:teuthology.orchestra.run.smithi035.stdout: "val": 422195086 2022-09-12T18:46:31.022 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:46:31.022 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T18:46:31.023 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:46:31.023 INFO:teuthology.orchestra.run.smithi035.stdout: "master_zonegroup": "test-zonegroup", 2022-09-12T18:46:31.023 INFO:teuthology.orchestra.run.smithi035.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T18:46:31.023 INFO:teuthology.orchestra.run.smithi035.stdout: "period_config": { 2022-09-12T18:46:31.024 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_quota": { 2022-09-12T18:46:31.024 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false, 2022-09-12T18:46:31.024 INFO:teuthology.orchestra.run.smithi035.stdout: "check_on_raw": false, 2022-09-12T18:46:31.024 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size": -1, 2022-09-12T18:46:31.025 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size_kb": 0, 2022-09-12T18:46:31.025 INFO:teuthology.orchestra.run.smithi035.stdout: "max_objects": -1 2022-09-12T18:46:31.025 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:46:31.025 INFO:teuthology.orchestra.run.smithi035.stdout: "user_quota": { 2022-09-12T18:46:31.025 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false, 2022-09-12T18:46:31.026 INFO:teuthology.orchestra.run.smithi035.stdout: "check_on_raw": false, 2022-09-12T18:46:31.026 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size": -1, 2022-09-12T18:46:31.026 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size_kb": 0, 2022-09-12T18:46:31.026 INFO:teuthology.orchestra.run.smithi035.stdout: "max_objects": -1 2022-09-12T18:46:31.027 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:46:31.027 INFO:teuthology.orchestra.run.smithi035.stdout: "user_ratelimit": { 2022-09-12T18:46:31.027 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_ops": 0, 2022-09-12T18:46:31.027 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_ops": 0, 2022-09-12T18:46:31.028 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_bytes": 0, 2022-09-12T18:46:31.028 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_bytes": 0, 2022-09-12T18:46:31.028 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false 2022-09-12T18:46:31.028 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:46:31.028 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_ratelimit": { 2022-09-12T18:46:31.029 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_ops": 0, 2022-09-12T18:46:31.029 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_ops": 0, 2022-09-12T18:46:31.029 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_bytes": 0, 2022-09-12T18:46:31.029 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_bytes": 0, 2022-09-12T18:46:31.030 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false 2022-09-12T18:46:31.030 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:46:31.030 INFO:teuthology.orchestra.run.smithi035.stdout: "anonymous_ratelimit": { 2022-09-12T18:46:31.030 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_ops": 0, 2022-09-12T18:46:31.031 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_ops": 0, 2022-09-12T18:46:31.031 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_bytes": 0, 2022-09-12T18:46:31.031 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_bytes": 0, 2022-09-12T18:46:31.031 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false 2022-09-12T18:46:31.031 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:46:31.032 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:46:31.032 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T18:46:31.032 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_name": "test-realm", 2022-09-12T18:46:31.032 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:46:31.033 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T18:46:31.034 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2022-09-12T18:46:31.035 INFO:teuthology.orchestra.run:waiting for 300 2022-09-12T18:46:31.036 INFO:tasks.rgw.c1.client.0.smithi035.stdout:2022-09-12T18:46:31.032+0000 7f7f93477700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 107288) UID: 0 2022-09-12T18:46:31.036 INFO:tasks.rgw.c1.client.0.smithi035.stdout:2022-09-12T18:46:31.032+0000 7f7f976e2600 -1 shutting down 2022-09-12T18:46:32.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2022-09-12T18:46:37.137 INFO:tasks.rgw.c1.client.0:Stopped 2022-09-12T18:46:37.138 INFO:tasks.rgw.c1.client.0:Restarting daemon 2022-09-12T18:46:37.138 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2022-09-12T18:46:37.141 INFO:tasks.rgw.c1.client.0:Started 2022-09-12T18:46:37.142 DEBUG:teuthology.orchestra.run.smithi035:> curl http://smithi035.front.sepia.ceph.com:8000 2022-09-12T18:46:37.169 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-12T18:46:37.170 INFO:teuthology.orchestra.run.smithi035.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-12T18:46:37.170 INFO:teuthology.orchestra.run.smithi035.stderr: Dload Upload Total Spent Left Speed 2022-09-12T18:46:37.171 INFO:teuthology.orchestra.run.smithi035.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi035.front.sepia.ceph.com port 8000: Connection refused 2022-09-12T18:46:37.900 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2022-09-12T18:46:38.172 DEBUG:teuthology.orchestra.run.smithi035:> curl http://smithi035.front.sepia.ceph.com:8000 2022-09-12T18:46:38.196 INFO:teuthology.orchestra.run.smithi035.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-12T18:46:38.197 INFO:teuthology.orchestra.run.smithi035.stderr: Dload Upload Total Spent Left Speed 2022-09-12T18:46:38.198 INFO:teuthology.orchestra.run.smithi035.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-12T18:46:38.199 INFO:teuthology.orchestra.run.smithi035.stdout:anonymous 2022-09-12T18:46:38.199 INFO:tasks.rgw_multisite:creating admin user.. 2022-09-12T18:46:38.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'RDIEFLSOLOEHUH1F', '--secret', 'LDCKRd1RecZNgErNcVDnUigDhC7QggCQ', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:38.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'RDIEFLSOLOEHUH1F', '--secret', 'LDCKRd1RecZNgErNcVDnUigDhC7QggCQ', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:38.201 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key RDIEFLSOLOEHUH1F --secret LDCKRd1RecZNgErNcVDnUigDhC7QggCQ --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T18:46:38.300 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:46:38.300 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:46:38.356 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T18:46:38.357 INFO:teuthology.orchestra.run.smithi035.stdout: "user_id": "realm-admin", 2022-09-12T18:46:38.357 INFO:teuthology.orchestra.run.smithi035.stdout: "display_name": "Realm Admin", 2022-09-12T18:46:38.357 INFO:teuthology.orchestra.run.smithi035.stdout: "email": "", 2022-09-12T18:46:38.357 INFO:teuthology.orchestra.run.smithi035.stdout: "suspended": 0, 2022-09-12T18:46:38.358 INFO:teuthology.orchestra.run.smithi035.stdout: "max_buckets": 1000, 2022-09-12T18:46:38.358 INFO:teuthology.orchestra.run.smithi035.stdout: "subusers": [], 2022-09-12T18:46:38.358 INFO:teuthology.orchestra.run.smithi035.stdout: "keys": [ 2022-09-12T18:46:38.358 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:46:38.359 INFO:teuthology.orchestra.run.smithi035.stdout: "user": "realm-admin", 2022-09-12T18:46:38.359 INFO:teuthology.orchestra.run.smithi035.stdout: "access_key": "RDIEFLSOLOEHUH1F", 2022-09-12T18:46:38.359 INFO:teuthology.orchestra.run.smithi035.stdout: "secret_key": "LDCKRd1RecZNgErNcVDnUigDhC7QggCQ" 2022-09-12T18:46:38.359 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:46:38.360 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T18:46:38.360 INFO:teuthology.orchestra.run.smithi035.stdout: "swift_keys": [], 2022-09-12T18:46:38.360 INFO:teuthology.orchestra.run.smithi035.stdout: "caps": [], 2022-09-12T18:46:38.361 INFO:teuthology.orchestra.run.smithi035.stdout: "op_mask": "read, write, delete", 2022-09-12T18:46:38.361 INFO:teuthology.orchestra.run.smithi035.stdout: "system": true, 2022-09-12T18:46:38.362 INFO:teuthology.orchestra.run.smithi035.stdout: "default_placement": "", 2022-09-12T18:46:38.362 INFO:teuthology.orchestra.run.smithi035.stdout: "default_storage_class": "", 2022-09-12T18:46:38.363 INFO:teuthology.orchestra.run.smithi035.stdout: "placement_tags": [], 2022-09-12T18:46:38.363 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_quota": { 2022-09-12T18:46:38.364 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false, 2022-09-12T18:46:38.364 INFO:teuthology.orchestra.run.smithi035.stdout: "check_on_raw": false, 2022-09-12T18:46:38.364 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size": -1, 2022-09-12T18:46:38.365 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size_kb": 0, 2022-09-12T18:46:38.365 INFO:teuthology.orchestra.run.smithi035.stdout: "max_objects": -1 2022-09-12T18:46:38.366 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:46:38.366 INFO:teuthology.orchestra.run.smithi035.stdout: "user_quota": { 2022-09-12T18:46:38.367 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false, 2022-09-12T18:46:38.367 INFO:teuthology.orchestra.run.smithi035.stdout: "check_on_raw": false, 2022-09-12T18:46:38.368 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size": -1, 2022-09-12T18:46:38.368 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size_kb": 0, 2022-09-12T18:46:38.369 INFO:teuthology.orchestra.run.smithi035.stdout: "max_objects": -1 2022-09-12T18:46:38.369 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:46:38.369 INFO:teuthology.orchestra.run.smithi035.stdout: "temp_url_keys": [], 2022-09-12T18:46:38.370 INFO:teuthology.orchestra.run.smithi035.stdout: "type": "rgw", 2022-09-12T18:46:38.370 INFO:teuthology.orchestra.run.smithi035.stdout: "mfa_ids": [] 2022-09-12T18:46:38.370 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T18:46:38.371 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:46:38.379 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2022-09-12T18:46:38.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi035.front.sepia.ceph.com:8000', '--access-key', 'RDIEFLSOLOEHUH1F', '--secret', 'LDCKRd1RecZNgErNcVDnUigDhC7QggCQ', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:38.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi035.front.sepia.ceph.com:8000', '--access-key', 'RDIEFLSOLOEHUH1F', '--secret', 'LDCKRd1RecZNgErNcVDnUigDhC7QggCQ', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:38.381 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi035.front.sepia.ceph.com:8000 --access-key RDIEFLSOLOEHUH1F --secret LDCKRd1RecZNgErNcVDnUigDhC7QggCQ --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T18:46:38.439 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:46:38.439 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:46:38.475 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:46:38.476 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T18:46:38.476 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-realm", 2022-09-12T18:46:38.476 INFO:teuthology.orchestra.run.smithi138.stdout: "current_period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:46:38.476 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 2 2022-09-12T18:46:38.477 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:46:38.478 INFO:tasks.rgw_multisite:creating zone.. 2022-09-12T18:46:38.478 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'RDIEFLSOLOEHUH1F', '--secret', 'LDCKRd1RecZNgErNcVDnUigDhC7QggCQ', '--endpoints', 'http://smithi138.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:38.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'RDIEFLSOLOEHUH1F', '--secret', 'LDCKRd1RecZNgErNcVDnUigDhC7QggCQ', '--endpoints', 'http://smithi138.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:38.479 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key RDIEFLSOLOEHUH1F --secret LDCKRd1RecZNgErNcVDnUigDhC7QggCQ --endpoints http://smithi138.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T18:46:38.534 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:46:38.534 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:46:38.556 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:38.552+0000 7f550bad1600 0 failed reading obj info from .rgw.root:zone_info.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47: (2) No such file or directory 2022-09-12T18:46:38.556 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:38.552+0000 7f550bad1600 0 WARNING: could not read zone params for zone id=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47 name=test-zone1 2022-09-12T18:46:38.561 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:46:38.562 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T18:46:38.562 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone2", 2022-09-12T18:46:38.562 INFO:teuthology.orchestra.run.smithi138.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-09-12T18:46:38.563 INFO:teuthology.orchestra.run.smithi138.stdout: "control_pool": "test-zone2.rgw.control", 2022-09-12T18:46:38.563 INFO:teuthology.orchestra.run.smithi138.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-09-12T18:46:38.563 INFO:teuthology.orchestra.run.smithi138.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-09-12T18:46:38.564 INFO:teuthology.orchestra.run.smithi138.stdout: "log_pool": "test-zone2.rgw.log", 2022-09-12T18:46:38.564 INFO:teuthology.orchestra.run.smithi138.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-09-12T18:46:38.565 INFO:teuthology.orchestra.run.smithi138.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-09-12T18:46:38.565 INFO:teuthology.orchestra.run.smithi138.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-09-12T18:46:38.565 INFO:teuthology.orchestra.run.smithi138.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-09-12T18:46:38.565 INFO:teuthology.orchestra.run.smithi138.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-09-12T18:46:38.565 INFO:teuthology.orchestra.run.smithi138.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-09-12T18:46:38.566 INFO:teuthology.orchestra.run.smithi138.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-09-12T18:46:38.566 INFO:teuthology.orchestra.run.smithi138.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-09-12T18:46:38.566 INFO:teuthology.orchestra.run.smithi138.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-09-12T18:46:38.566 INFO:teuthology.orchestra.run.smithi138.stdout: "system_key": { 2022-09-12T18:46:38.567 INFO:teuthology.orchestra.run.smithi138.stdout: "access_key": "RDIEFLSOLOEHUH1F", 2022-09-12T18:46:38.567 INFO:teuthology.orchestra.run.smithi138.stdout: "secret_key": "LDCKRd1RecZNgErNcVDnUigDhC7QggCQ" 2022-09-12T18:46:38.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:46:38.567 INFO:teuthology.orchestra.run.smithi138.stdout: "placement_pools": [ 2022-09-12T18:46:38.568 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:46:38.568 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "default-placement", 2022-09-12T18:46:38.568 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:46:38.568 INFO:teuthology.orchestra.run.smithi138.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-09-12T18:46:38.569 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_classes": { 2022-09-12T18:46:38.569 INFO:teuthology.orchestra.run.smithi138.stdout: "STANDARD": { 2022-09-12T18:46:38.569 INFO:teuthology.orchestra.run.smithi138.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2022-09-12T18:46:38.569 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:46:38.569 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:46:38.570 INFO:teuthology.orchestra.run.smithi138.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-09-12T18:46:38.570 INFO:teuthology.orchestra.run.smithi138.stdout: "index_type": 0 2022-09-12T18:46:38.571 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:46:38.571 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:46:38.571 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T18:46:38.572 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T18:46:38.572 INFO:teuthology.orchestra.run.smithi138.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-09-12T18:46:38.572 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:46:38.573 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2022-09-12T18:46:39.443 INFO:teuthology.orchestra.run.smithi138.stderr:pool 'test-zone2.rgw.buckets.data' created 2022-09-12T18:46:39.460 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2022-09-12T18:46:40.454 INFO:teuthology.orchestra.run.smithi138.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2022-09-12T18:46:40.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:40.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:40.474 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T18:46:40.527 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:46:40.527 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:46:40.549 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:46:40.550 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T18:46:40.550 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone2", 2022-09-12T18:46:40.550 INFO:teuthology.orchestra.run.smithi138.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-09-12T18:46:40.551 INFO:teuthology.orchestra.run.smithi138.stdout: "control_pool": "test-zone2.rgw.control", 2022-09-12T18:46:40.551 INFO:teuthology.orchestra.run.smithi138.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-09-12T18:46:40.552 INFO:teuthology.orchestra.run.smithi138.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-09-12T18:46:40.552 INFO:teuthology.orchestra.run.smithi138.stdout: "log_pool": "test-zone2.rgw.log", 2022-09-12T18:46:40.552 INFO:teuthology.orchestra.run.smithi138.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-09-12T18:46:40.552 INFO:teuthology.orchestra.run.smithi138.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-09-12T18:46:40.553 INFO:teuthology.orchestra.run.smithi138.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-09-12T18:46:40.553 INFO:teuthology.orchestra.run.smithi138.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-09-12T18:46:40.553 INFO:teuthology.orchestra.run.smithi138.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-09-12T18:46:40.553 INFO:teuthology.orchestra.run.smithi138.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-09-12T18:46:40.554 INFO:teuthology.orchestra.run.smithi138.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-09-12T18:46:40.554 INFO:teuthology.orchestra.run.smithi138.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-09-12T18:46:40.554 INFO:teuthology.orchestra.run.smithi138.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-09-12T18:46:40.554 INFO:teuthology.orchestra.run.smithi138.stdout: "system_key": { 2022-09-12T18:46:40.555 INFO:teuthology.orchestra.run.smithi138.stdout: "access_key": "RDIEFLSOLOEHUH1F", 2022-09-12T18:46:40.555 INFO:teuthology.orchestra.run.smithi138.stdout: "secret_key": "LDCKRd1RecZNgErNcVDnUigDhC7QggCQ" 2022-09-12T18:46:40.555 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:46:40.555 INFO:teuthology.orchestra.run.smithi138.stdout: "placement_pools": [ 2022-09-12T18:46:40.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:46:40.556 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "default-placement", 2022-09-12T18:46:40.556 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:46:40.556 INFO:teuthology.orchestra.run.smithi138.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-09-12T18:46:40.557 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_classes": { 2022-09-12T18:46:40.557 INFO:teuthology.orchestra.run.smithi138.stdout: "STANDARD": { 2022-09-12T18:46:40.557 INFO:teuthology.orchestra.run.smithi138.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2022-09-12T18:46:40.557 INFO:teuthology.orchestra.run.smithi138.stdout: "compression_type": "random" 2022-09-12T18:46:40.558 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:46:40.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:46:40.559 INFO:teuthology.orchestra.run.smithi138.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-09-12T18:46:40.559 INFO:teuthology.orchestra.run.smithi138.stdout: "index_type": 0 2022-09-12T18:46:40.560 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:46:40.560 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:46:40.560 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T18:46:40.561 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "", 2022-09-12T18:46:40.561 INFO:teuthology.orchestra.run.smithi138.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-09-12T18:46:40.562 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:46:40.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:40.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:40.565 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T18:46:40.614 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:46:40.614 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:46:40.625 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:46:40.622+0000 7fe01eaf9600 0 period (c0f7a512-c940-4d1f-8adf-3484630f876b does not have zone fdf3b910-2ea3-4164-a3f8-b3e146af1c44 configured 2022-09-12T18:46:47.512 INFO:teuthology.orchestra.run.smithi138.stderr:Sending period to new master zone fb29b4fb-1f78-4524-ab80-6a5e9e01ab47 2022-09-12T18:46:47.540 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:46:47.540 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:46:47.541 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 2, 2022-09-12T18:46:47.543 INFO:teuthology.orchestra.run.smithi138.stdout: "predecessor_uuid": "2e627c13-04da-45e8-bcee-0ed7248cedbf", 2022-09-12T18:46:47.544 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": [], 2022-09-12T18:46:47.544 INFO:teuthology.orchestra.run.smithi138.stdout: "period_map": { 2022-09-12T18:46:47.544 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:46:47.545 INFO:teuthology.orchestra.run.smithi138.stdout: "zonegroups": [ 2022-09-12T18:46:47.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:46:47.546 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "test-zonegroup", 2022-09-12T18:46:47.547 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zonegroup", 2022-09-12T18:46:47.547 INFO:teuthology.orchestra.run.smithi138.stdout: "api_name": "", 2022-09-12T18:46:47.548 INFO:teuthology.orchestra.run.smithi138.stdout: "is_master": true, 2022-09-12T18:46:47.548 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T18:46:47.549 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T18:46:47.549 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T18:46:47.550 INFO:teuthology.orchestra.run.smithi138.stdout: "hostnames": [], 2022-09-12T18:46:47.550 INFO:teuthology.orchestra.run.smithi138.stdout: "hostnames_s3website": [], 2022-09-12T18:46:47.551 INFO:teuthology.orchestra.run.smithi138.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T18:46:47.551 INFO:teuthology.orchestra.run.smithi138.stdout: "zones": [ 2022-09-12T18:46:47.552 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:46:47.552 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T18:46:47.552 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone1", 2022-09-12T18:46:47.553 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T18:46:47.553 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T18:46:47.554 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T18:46:47.554 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T18:46:47.555 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T18:46:47.555 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T18:46:47.555 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T18:46:47.555 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "", 2022-09-12T18:46:47.556 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": true, 2022-09-12T18:46:47.556 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [], 2022-09-12T18:46:47.556 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T18:46:47.556 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T18:46:47.557 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T18:46:47.557 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:46:47.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:46:47.557 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:46:47.558 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T18:46:47.558 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone2", 2022-09-12T18:46:47.558 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T18:46:47.559 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi138.front.sepia.ceph.com:8000" 2022-09-12T18:46:47.559 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T18:46:47.560 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T18:46:47.561 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T18:46:47.561 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T18:46:47.562 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T18:46:47.562 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "", 2022-09-12T18:46:47.563 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": true, 2022-09-12T18:46:47.563 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [], 2022-09-12T18:46:47.566 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T18:46:47.566 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T18:46:47.566 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T18:46:47.567 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:46:47.567 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:46:47.567 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T18:46:47.568 INFO:teuthology.orchestra.run.smithi138.stdout: "placement_targets": [ 2022-09-12T18:46:47.568 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:46:47.568 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "default-placement", 2022-09-12T18:46:47.569 INFO:teuthology.orchestra.run.smithi138.stdout: "tags": [], 2022-09-12T18:46:47.569 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_classes": [ 2022-09-12T18:46:47.569 INFO:teuthology.orchestra.run.smithi138.stdout: "STANDARD" 2022-09-12T18:46:47.570 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:46:47.570 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:46:47.570 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T18:46:47.571 INFO:teuthology.orchestra.run.smithi138.stdout: "default_placement": "default-placement", 2022-09-12T18:46:47.571 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T18:46:47.572 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_policy": { 2022-09-12T18:46:47.572 INFO:teuthology.orchestra.run.smithi138.stdout: "groups": [] 2022-09-12T18:46:47.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:46:47.573 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled_features": [ 2022-09-12T18:46:47.574 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T18:46:47.574 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:46:47.574 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:46:47.575 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T18:46:47.575 INFO:teuthology.orchestra.run.smithi138.stdout: "short_zone_ids": [ 2022-09-12T18:46:47.575 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:46:47.576 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T18:46:47.576 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 422195086 2022-09-12T18:46:47.576 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:46:47.577 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:46:47.577 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T18:46:47.578 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 2881234289 2022-09-12T18:46:47.578 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:46:47.578 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:46:47.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:46:47.579 INFO:teuthology.orchestra.run.smithi138.stdout: "master_zonegroup": "test-zonegroup", 2022-09-12T18:46:47.579 INFO:teuthology.orchestra.run.smithi138.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T18:46:47.580 INFO:teuthology.orchestra.run.smithi138.stdout: "period_config": { 2022-09-12T18:46:47.580 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_quota": { 2022-09-12T18:46:47.581 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false, 2022-09-12T18:46:47.581 INFO:teuthology.orchestra.run.smithi138.stdout: "check_on_raw": false, 2022-09-12T18:46:47.581 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size": -1, 2022-09-12T18:46:47.582 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size_kb": 0, 2022-09-12T18:46:47.582 INFO:teuthology.orchestra.run.smithi138.stdout: "max_objects": -1 2022-09-12T18:46:47.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:46:47.583 INFO:teuthology.orchestra.run.smithi138.stdout: "user_quota": { 2022-09-12T18:46:47.583 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false, 2022-09-12T18:46:47.583 INFO:teuthology.orchestra.run.smithi138.stdout: "check_on_raw": false, 2022-09-12T18:46:47.584 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size": -1, 2022-09-12T18:46:47.584 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size_kb": 0, 2022-09-12T18:46:47.585 INFO:teuthology.orchestra.run.smithi138.stdout: "max_objects": -1 2022-09-12T18:46:47.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:46:47.585 INFO:teuthology.orchestra.run.smithi138.stdout: "user_ratelimit": { 2022-09-12T18:46:47.586 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_ops": 0, 2022-09-12T18:46:47.586 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_ops": 0, 2022-09-12T18:46:47.587 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_bytes": 0, 2022-09-12T18:46:47.587 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_bytes": 0, 2022-09-12T18:46:47.587 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false 2022-09-12T18:46:47.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:46:47.588 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_ratelimit": { 2022-09-12T18:46:47.588 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_ops": 0, 2022-09-12T18:46:47.589 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_ops": 0, 2022-09-12T18:46:47.589 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_bytes": 0, 2022-09-12T18:46:47.590 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_bytes": 0, 2022-09-12T18:46:47.590 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false 2022-09-12T18:46:47.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:46:47.593 INFO:teuthology.orchestra.run.smithi138.stdout: "anonymous_ratelimit": { 2022-09-12T18:46:47.593 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_ops": 0, 2022-09-12T18:46:47.593 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_ops": 0, 2022-09-12T18:46:47.594 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_bytes": 0, 2022-09-12T18:46:47.594 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_bytes": 0, 2022-09-12T18:46:47.594 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false 2022-09-12T18:46:47.595 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:46:47.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:46:47.596 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T18:46:47.596 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_name": "test-realm", 2022-09-12T18:46:47.596 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:46:47.597 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:46:47.598 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2022-09-12T18:46:47.599 INFO:teuthology.orchestra.run:waiting for 300 2022-09-12T18:46:47.600 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T18:46:47.596+0000 7fb5b580a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 107175) UID: 0 2022-09-12T18:46:47.600 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T18:46:47.596+0000 7fb5b9a75600 -1 shutting down 2022-09-12T18:46:48.015 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2022-09-12T18:46:53.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~6s 2022-09-12T18:46:53.701 INFO:tasks.rgw.c2.client.0:Stopped 2022-09-12T18:46:53.701 INFO:tasks.rgw.c2.client.0:Restarting daemon 2022-09-12T18:46:53.702 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2022-09-12T18:46:53.705 INFO:tasks.rgw.c2.client.0:Started 2022-09-12T18:46:53.705 DEBUG:teuthology.orchestra.run.smithi138:> curl http://smithi138.front.sepia.ceph.com:8000 2022-09-12T18:46:53.731 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-12T18:46:53.733 INFO:teuthology.orchestra.run.smithi138.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-12T18:46:53.733 INFO:teuthology.orchestra.run.smithi138.stderr: Dload Upload Total Spent Left Speed 2022-09-12T18:46:53.733 INFO:teuthology.orchestra.run.smithi138.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi138.front.sepia.ceph.com port 8000: Connection refused 2022-09-12T18:46:54.734 DEBUG:teuthology.orchestra.run.smithi138:> curl http://smithi138.front.sepia.ceph.com:8000 2022-09-12T18:46:54.758 INFO:teuthology.orchestra.run.smithi138.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-12T18:46:54.759 INFO:teuthology.orchestra.run.smithi138.stderr: Dload Upload Total Spent Left Speed 2022-09-12T18:46:54.759 INFO:teuthology.orchestra.run.smithi138.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-12T18:46:54.761 INFO:teuthology.orchestra.run.smithi138.stdout:anonymous 2022-09-12T18:46:54.761 INFO:tasks.rgw_multisite:creating zone.. 2022-09-12T18:46:54.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'RDIEFLSOLOEHUH1F', '--secret', 'LDCKRd1RecZNgErNcVDnUigDhC7QggCQ', '--endpoints', 'http://smithi035.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:54.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'RDIEFLSOLOEHUH1F', '--secret', 'LDCKRd1RecZNgErNcVDnUigDhC7QggCQ', '--endpoints', 'http://smithi035.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:54.764 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key RDIEFLSOLOEHUH1F --secret LDCKRd1RecZNgErNcVDnUigDhC7QggCQ --endpoints http://smithi035.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T18:46:54.826 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:46:54.826 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:46:54.854 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:54.850+0000 7f4cf1ff9600 0 failed reading obj info from .rgw.root:zone_info.fdf3b910-2ea3-4164-a3f8-b3e146af1c44: (2) No such file or directory 2022-09-12T18:46:54.854 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:54.850+0000 7f4cf1ff9600 0 WARNING: could not read zone params for zone id=fdf3b910-2ea3-4164-a3f8-b3e146af1c44 name=test-zone2 2022-09-12T18:46:54.857 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T18:46:54.857 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T18:46:54.858 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone3", 2022-09-12T18:46:54.858 INFO:teuthology.orchestra.run.smithi035.stdout: "domain_root": "test-zone3.rgw.meta:root", 2022-09-12T18:46:54.858 INFO:teuthology.orchestra.run.smithi035.stdout: "control_pool": "test-zone3.rgw.control", 2022-09-12T18:46:54.858 INFO:teuthology.orchestra.run.smithi035.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2022-09-12T18:46:54.859 INFO:teuthology.orchestra.run.smithi035.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2022-09-12T18:46:54.859 INFO:teuthology.orchestra.run.smithi035.stdout: "log_pool": "test-zone3.rgw.log", 2022-09-12T18:46:54.859 INFO:teuthology.orchestra.run.smithi035.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2022-09-12T18:46:54.859 INFO:teuthology.orchestra.run.smithi035.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2022-09-12T18:46:54.860 INFO:teuthology.orchestra.run.smithi035.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2022-09-12T18:46:54.860 INFO:teuthology.orchestra.run.smithi035.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2022-09-12T18:46:54.860 INFO:teuthology.orchestra.run.smithi035.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2022-09-12T18:46:54.860 INFO:teuthology.orchestra.run.smithi035.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2022-09-12T18:46:54.861 INFO:teuthology.orchestra.run.smithi035.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2022-09-12T18:46:54.861 INFO:teuthology.orchestra.run.smithi035.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2022-09-12T18:46:54.861 INFO:teuthology.orchestra.run.smithi035.stdout: "otp_pool": "test-zone3.rgw.otp", 2022-09-12T18:46:54.861 INFO:teuthology.orchestra.run.smithi035.stdout: "system_key": { 2022-09-12T18:46:54.861 INFO:teuthology.orchestra.run.smithi035.stdout: "access_key": "RDIEFLSOLOEHUH1F", 2022-09-12T18:46:54.862 INFO:teuthology.orchestra.run.smithi035.stdout: "secret_key": "LDCKRd1RecZNgErNcVDnUigDhC7QggCQ" 2022-09-12T18:46:54.862 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:46:54.862 INFO:teuthology.orchestra.run.smithi035.stdout: "placement_pools": [ 2022-09-12T18:46:54.863 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:46:54.863 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "default-placement", 2022-09-12T18:46:54.863 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:46:54.863 INFO:teuthology.orchestra.run.smithi035.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2022-09-12T18:46:54.864 INFO:teuthology.orchestra.run.smithi035.stdout: "storage_classes": { 2022-09-12T18:46:54.864 INFO:teuthology.orchestra.run.smithi035.stdout: "STANDARD": { 2022-09-12T18:46:54.864 INFO:teuthology.orchestra.run.smithi035.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2022-09-12T18:46:54.864 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:46:54.864 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:46:54.865 INFO:teuthology.orchestra.run.smithi035.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2022-09-12T18:46:54.865 INFO:teuthology.orchestra.run.smithi035.stdout: "index_type": 0 2022-09-12T18:46:54.865 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:46:54.865 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:46:54.865 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T18:46:54.866 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T18:46:54.866 INFO:teuthology.orchestra.run.smithi035.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2022-09-12T18:46:54.866 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T18:46:54.868 DEBUG:teuthology.orchestra.run.smithi035:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2022-09-12T18:46:55.984 INFO:teuthology.orchestra.run.smithi035.stderr:pool 'test-zone3.rgw.buckets.data' created 2022-09-12T18:46:56.005 DEBUG:teuthology.orchestra.run.smithi035:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2022-09-12T18:46:56.997 INFO:teuthology.orchestra.run.smithi035.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2022-09-12T18:46:57.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:57.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:57.016 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T18:46:57.072 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:46:57.072 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:46:57.096 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T18:46:57.097 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T18:46:57.097 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone3", 2022-09-12T18:46:57.097 INFO:teuthology.orchestra.run.smithi035.stdout: "domain_root": "test-zone3.rgw.meta:root", 2022-09-12T18:46:57.097 INFO:teuthology.orchestra.run.smithi035.stdout: "control_pool": "test-zone3.rgw.control", 2022-09-12T18:46:57.098 INFO:teuthology.orchestra.run.smithi035.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2022-09-12T18:46:57.098 INFO:teuthology.orchestra.run.smithi035.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2022-09-12T18:46:57.098 INFO:teuthology.orchestra.run.smithi035.stdout: "log_pool": "test-zone3.rgw.log", 2022-09-12T18:46:57.099 INFO:teuthology.orchestra.run.smithi035.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2022-09-12T18:46:57.099 INFO:teuthology.orchestra.run.smithi035.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2022-09-12T18:46:57.099 INFO:teuthology.orchestra.run.smithi035.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2022-09-12T18:46:57.099 INFO:teuthology.orchestra.run.smithi035.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2022-09-12T18:46:57.100 INFO:teuthology.orchestra.run.smithi035.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2022-09-12T18:46:57.100 INFO:teuthology.orchestra.run.smithi035.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2022-09-12T18:46:57.100 INFO:teuthology.orchestra.run.smithi035.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2022-09-12T18:46:57.100 INFO:teuthology.orchestra.run.smithi035.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2022-09-12T18:46:57.101 INFO:teuthology.orchestra.run.smithi035.stdout: "otp_pool": "test-zone3.rgw.otp", 2022-09-12T18:46:57.101 INFO:teuthology.orchestra.run.smithi035.stdout: "system_key": { 2022-09-12T18:46:57.101 INFO:teuthology.orchestra.run.smithi035.stdout: "access_key": "RDIEFLSOLOEHUH1F", 2022-09-12T18:46:57.101 INFO:teuthology.orchestra.run.smithi035.stdout: "secret_key": "LDCKRd1RecZNgErNcVDnUigDhC7QggCQ" 2022-09-12T18:46:57.101 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:46:57.102 INFO:teuthology.orchestra.run.smithi035.stdout: "placement_pools": [ 2022-09-12T18:46:57.102 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:46:57.103 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "default-placement", 2022-09-12T18:46:57.103 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:46:57.103 INFO:teuthology.orchestra.run.smithi035.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2022-09-12T18:46:57.104 INFO:teuthology.orchestra.run.smithi035.stdout: "storage_classes": { 2022-09-12T18:46:57.104 INFO:teuthology.orchestra.run.smithi035.stdout: "STANDARD": { 2022-09-12T18:46:57.104 INFO:teuthology.orchestra.run.smithi035.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2022-09-12T18:46:57.104 INFO:teuthology.orchestra.run.smithi035.stdout: "compression_type": "random" 2022-09-12T18:46:57.105 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:46:57.105 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:46:57.106 INFO:teuthology.orchestra.run.smithi035.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2022-09-12T18:46:57.106 INFO:teuthology.orchestra.run.smithi035.stdout: "index_type": 0 2022-09-12T18:46:57.107 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:46:57.107 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:46:57.108 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T18:46:57.108 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_id": "", 2022-09-12T18:46:57.108 INFO:teuthology.orchestra.run.smithi035.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2022-09-12T18:46:57.109 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T18:46:57.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:57.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:46:57.111 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T18:46:57.161 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:46:57.161 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:46:57.170 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:46:57.166+0000 7f2e98dfe600 0 period (c0f7a512-c940-4d1f-8adf-3484630f876b does not have zone 823fb3a5-4d8a-49dc-ae64-e05804bbea94 configured 2022-09-12T18:46:59.329 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2022-09-12T18:47:04.044 INFO:teuthology.orchestra.run.smithi035.stderr:Sending period to new master zone fb29b4fb-1f78-4524-ab80-6a5e9e01ab47 2022-09-12T18:47:04.079 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T18:47:04.080 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:47:04.080 INFO:teuthology.orchestra.run.smithi035.stdout: "epoch": 3, 2022-09-12T18:47:04.080 INFO:teuthology.orchestra.run.smithi035.stdout: "predecessor_uuid": "2e627c13-04da-45e8-bcee-0ed7248cedbf", 2022-09-12T18:47:04.081 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": [], 2022-09-12T18:47:04.081 INFO:teuthology.orchestra.run.smithi035.stdout: "period_map": { 2022-09-12T18:47:04.082 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:47:04.082 INFO:teuthology.orchestra.run.smithi035.stdout: "zonegroups": [ 2022-09-12T18:47:04.082 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:47:04.083 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "test-zonegroup", 2022-09-12T18:47:04.083 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zonegroup", 2022-09-12T18:47:04.084 INFO:teuthology.orchestra.run.smithi035.stdout: "api_name": "", 2022-09-12T18:47:04.084 INFO:teuthology.orchestra.run.smithi035.stdout: "is_master": true, 2022-09-12T18:47:04.084 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T18:47:04.085 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T18:47:04.085 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T18:47:04.085 INFO:teuthology.orchestra.run.smithi035.stdout: "hostnames": [], 2022-09-12T18:47:04.086 INFO:teuthology.orchestra.run.smithi035.stdout: "hostnames_s3website": [], 2022-09-12T18:47:04.086 INFO:teuthology.orchestra.run.smithi035.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T18:47:04.087 INFO:teuthology.orchestra.run.smithi035.stdout: "zones": [ 2022-09-12T18:47:04.087 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:47:04.087 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T18:47:04.088 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone3", 2022-09-12T18:47:04.088 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T18:47:04.088 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi035.front.sepia.ceph.com:8001" 2022-09-12T18:47:04.089 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T18:47:04.089 INFO:teuthology.orchestra.run.smithi035.stdout: "log_meta": false, 2022-09-12T18:47:04.090 INFO:teuthology.orchestra.run.smithi035.stdout: "log_data": true, 2022-09-12T18:47:04.090 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_index_max_shards": 11, 2022-09-12T18:47:04.090 INFO:teuthology.orchestra.run.smithi035.stdout: "read_only": false, 2022-09-12T18:47:04.090 INFO:teuthology.orchestra.run.smithi035.stdout: "tier_type": "", 2022-09-12T18:47:04.091 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from_all": true, 2022-09-12T18:47:04.091 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from": [], 2022-09-12T18:47:04.091 INFO:teuthology.orchestra.run.smithi035.stdout: "redirect_zone": "", 2022-09-12T18:47:04.091 INFO:teuthology.orchestra.run.smithi035.stdout: "supported_features": [ 2022-09-12T18:47:04.092 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T18:47:04.092 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T18:47:04.093 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:47:04.093 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:47:04.093 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T18:47:04.094 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone1", 2022-09-12T18:47:04.094 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T18:47:04.095 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T18:47:04.095 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T18:47:04.096 INFO:teuthology.orchestra.run.smithi035.stdout: "log_meta": false, 2022-09-12T18:47:04.096 INFO:teuthology.orchestra.run.smithi035.stdout: "log_data": true, 2022-09-12T18:47:04.096 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_index_max_shards": 11, 2022-09-12T18:47:04.096 INFO:teuthology.orchestra.run.smithi035.stdout: "read_only": false, 2022-09-12T18:47:04.097 INFO:teuthology.orchestra.run.smithi035.stdout: "tier_type": "", 2022-09-12T18:47:04.097 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from_all": true, 2022-09-12T18:47:04.097 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from": [], 2022-09-12T18:47:04.097 INFO:teuthology.orchestra.run.smithi035.stdout: "redirect_zone": "", 2022-09-12T18:47:04.098 INFO:teuthology.orchestra.run.smithi035.stdout: "supported_features": [ 2022-09-12T18:47:04.098 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T18:47:04.098 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T18:47:04.098 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:47:04.099 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:47:04.099 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T18:47:04.099 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone2", 2022-09-12T18:47:04.099 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T18:47:04.099 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi138.front.sepia.ceph.com:8000" 2022-09-12T18:47:04.100 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T18:47:04.100 INFO:teuthology.orchestra.run.smithi035.stdout: "log_meta": false, 2022-09-12T18:47:04.100 INFO:teuthology.orchestra.run.smithi035.stdout: "log_data": true, 2022-09-12T18:47:04.100 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_index_max_shards": 11, 2022-09-12T18:47:04.101 INFO:teuthology.orchestra.run.smithi035.stdout: "read_only": false, 2022-09-12T18:47:04.101 INFO:teuthology.orchestra.run.smithi035.stdout: "tier_type": "", 2022-09-12T18:47:04.101 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from_all": true, 2022-09-12T18:47:04.101 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from": [], 2022-09-12T18:47:04.102 INFO:teuthology.orchestra.run.smithi035.stdout: "redirect_zone": "", 2022-09-12T18:47:04.102 INFO:teuthology.orchestra.run.smithi035.stdout: "supported_features": [ 2022-09-12T18:47:04.102 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T18:47:04.102 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T18:47:04.103 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:47:04.103 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T18:47:04.103 INFO:teuthology.orchestra.run.smithi035.stdout: "placement_targets": [ 2022-09-12T18:47:04.103 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:47:04.104 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "default-placement", 2022-09-12T18:47:04.104 INFO:teuthology.orchestra.run.smithi035.stdout: "tags": [], 2022-09-12T18:47:04.104 INFO:teuthology.orchestra.run.smithi035.stdout: "storage_classes": [ 2022-09-12T18:47:04.104 INFO:teuthology.orchestra.run.smithi035.stdout: "STANDARD" 2022-09-12T18:47:04.104 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T18:47:04.105 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:47:04.105 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T18:47:04.105 INFO:teuthology.orchestra.run.smithi035.stdout: "default_placement": "default-placement", 2022-09-12T18:47:04.105 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T18:47:04.106 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_policy": { 2022-09-12T18:47:04.106 INFO:teuthology.orchestra.run.smithi035.stdout: "groups": [] 2022-09-12T18:47:04.106 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:47:04.106 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled_features": [ 2022-09-12T18:47:04.107 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T18:47:04.107 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T18:47:04.107 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:47:04.107 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T18:47:04.107 INFO:teuthology.orchestra.run.smithi035.stdout: "short_zone_ids": [ 2022-09-12T18:47:04.108 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:47:04.108 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T18:47:04.108 INFO:teuthology.orchestra.run.smithi035.stdout: "val": 227168046 2022-09-12T18:47:04.108 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:47:04.109 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:47:04.109 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T18:47:04.109 INFO:teuthology.orchestra.run.smithi035.stdout: "val": 422195086 2022-09-12T18:47:04.109 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:47:04.109 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:47:04.110 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T18:47:04.110 INFO:teuthology.orchestra.run.smithi035.stdout: "val": 2881234289 2022-09-12T18:47:04.110 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:47:04.110 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T18:47:04.111 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:47:04.112 INFO:teuthology.orchestra.run.smithi035.stdout: "master_zonegroup": "test-zonegroup", 2022-09-12T18:47:04.112 INFO:teuthology.orchestra.run.smithi035.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T18:47:04.112 INFO:teuthology.orchestra.run.smithi035.stdout: "period_config": { 2022-09-12T18:47:04.113 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_quota": { 2022-09-12T18:47:04.113 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false, 2022-09-12T18:47:04.113 INFO:teuthology.orchestra.run.smithi035.stdout: "check_on_raw": false, 2022-09-12T18:47:04.113 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size": -1, 2022-09-12T18:47:04.114 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size_kb": 0, 2022-09-12T18:47:04.114 INFO:teuthology.orchestra.run.smithi035.stdout: "max_objects": -1 2022-09-12T18:47:04.114 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:47:04.114 INFO:teuthology.orchestra.run.smithi035.stdout: "user_quota": { 2022-09-12T18:47:04.115 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false, 2022-09-12T18:47:04.115 INFO:teuthology.orchestra.run.smithi035.stdout: "check_on_raw": false, 2022-09-12T18:47:04.115 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size": -1, 2022-09-12T18:47:04.115 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size_kb": 0, 2022-09-12T18:47:04.115 INFO:teuthology.orchestra.run.smithi035.stdout: "max_objects": -1 2022-09-12T18:47:04.116 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:47:04.116 INFO:teuthology.orchestra.run.smithi035.stdout: "user_ratelimit": { 2022-09-12T18:47:04.116 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_ops": 0, 2022-09-12T18:47:04.116 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_ops": 0, 2022-09-12T18:47:04.117 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_bytes": 0, 2022-09-12T18:47:04.117 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_bytes": 0, 2022-09-12T18:47:04.117 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false 2022-09-12T18:47:04.117 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:47:04.117 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_ratelimit": { 2022-09-12T18:47:04.118 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_ops": 0, 2022-09-12T18:47:04.118 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_ops": 0, 2022-09-12T18:47:04.118 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_bytes": 0, 2022-09-12T18:47:04.118 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_bytes": 0, 2022-09-12T18:47:04.119 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false 2022-09-12T18:47:04.119 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:47:04.119 INFO:teuthology.orchestra.run.smithi035.stdout: "anonymous_ratelimit": { 2022-09-12T18:47:04.119 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_ops": 0, 2022-09-12T18:47:04.120 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_ops": 0, 2022-09-12T18:47:04.120 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_bytes": 0, 2022-09-12T18:47:04.120 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_bytes": 0, 2022-09-12T18:47:04.120 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false 2022-09-12T18:47:04.121 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:47:04.121 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:47:04.121 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T18:47:04.121 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_name": "test-realm", 2022-09-12T18:47:04.122 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:47:04.122 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T18:47:04.123 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2022-09-12T18:47:04.123 INFO:teuthology.orchestra.run:waiting for 300 2022-09-12T18:47:04.124 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T18:47:04.119+0000 7fe47af35700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 107295) UID: 0 2022-09-12T18:47:04.124 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T18:47:04.119+0000 7fe47f1a0600 -1 shutting down 2022-09-12T18:47:06.331 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2022-09-12T18:47:10.228 INFO:tasks.rgw.c1.client.1:Stopped 2022-09-12T18:47:10.228 INFO:tasks.rgw.c1.client.1:Restarting daemon 2022-09-12T18:47:10.229 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2022-09-12T18:47:10.232 INFO:tasks.rgw.c1.client.1:Started 2022-09-12T18:47:10.232 DEBUG:teuthology.orchestra.run.smithi035:> curl http://smithi035.front.sepia.ceph.com:8001 2022-09-12T18:47:10.258 INFO:teuthology.orchestra.run.smithi035.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-12T18:47:10.259 INFO:teuthology.orchestra.run.smithi035.stderr: Dload Upload Total Spent Left Speed 2022-09-12T18:47:10.261 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-12T18:47:10.262 INFO:teuthology.orchestra.run.smithi035.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi035.front.sepia.ceph.com port 8001: Connection refused 2022-09-12T18:47:11.263 DEBUG:teuthology.orchestra.run.smithi035:> curl http://smithi035.front.sepia.ceph.com:8001 2022-09-12T18:47:11.292 INFO:teuthology.orchestra.run.smithi035.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-12T18:47:11.292 INFO:teuthology.orchestra.run.smithi035.stderr: Dload Upload Total Spent Left Speed 2022-09-12T18:47:11.293 INFO:teuthology.orchestra.run.smithi035.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-12T18:47:11.295 INFO:teuthology.orchestra.run.smithi035.stdout:anonymous 2022-09-12T18:47:11.295 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2022-09-12T18:47:11.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi035.front.sepia.ceph.com:8000', '--access-key', 'RDIEFLSOLOEHUH1F', '--secret', 'LDCKRd1RecZNgErNcVDnUigDhC7QggCQ', '--url', 'http://smithi035.front.sepia.ceph.com:8000', '--access-key', 'RDIEFLSOLOEHUH1F', '--secret', 'LDCKRd1RecZNgErNcVDnUigDhC7QggCQ', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:47:11.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi035.front.sepia.ceph.com:8000', '--access-key', 'RDIEFLSOLOEHUH1F', '--secret', 'LDCKRd1RecZNgErNcVDnUigDhC7QggCQ', '--url', 'http://smithi035.front.sepia.ceph.com:8000', '--access-key', 'RDIEFLSOLOEHUH1F', '--secret', 'LDCKRd1RecZNgErNcVDnUigDhC7QggCQ', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:47:11.297 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi035.front.sepia.ceph.com:8000 --access-key RDIEFLSOLOEHUH1F --secret LDCKRd1RecZNgErNcVDnUigDhC7QggCQ --url http://smithi035.front.sepia.ceph.com:8000 --access-key RDIEFLSOLOEHUH1F --secret LDCKRd1RecZNgErNcVDnUigDhC7QggCQ --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T18:47:11.363 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:47:11.364 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:47:11.397 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T18:47:11.393+0000 7fa7b265a700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.06f742c1-7106-48ed-a794-ceccfd2207bd.control err=-107 2022-09-12T18:47:11.410 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:47:11.410 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T18:47:11.411 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-realm", 2022-09-12T18:47:11.411 INFO:teuthology.orchestra.run.smithi138.stdout: "current_period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:47:11.411 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 2 2022-09-12T18:47:11.411 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:47:11.413 INFO:tasks.rgw_multisite:creating zone.. 2022-09-12T18:47:11.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'RDIEFLSOLOEHUH1F', '--secret', 'LDCKRd1RecZNgErNcVDnUigDhC7QggCQ', '--endpoints', 'http://smithi138.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:47:11.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'RDIEFLSOLOEHUH1F', '--secret', 'LDCKRd1RecZNgErNcVDnUigDhC7QggCQ', '--endpoints', 'http://smithi138.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:47:11.414 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key RDIEFLSOLOEHUH1F --secret LDCKRd1RecZNgErNcVDnUigDhC7QggCQ --endpoints http://smithi138.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T18:47:11.475 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:47:11.476 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:47:11.498 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:47:11.494+0000 7fed52b8b600 0 failed reading obj info from .rgw.root:zone_info.823fb3a5-4d8a-49dc-ae64-e05804bbea94: (2) No such file or directory 2022-09-12T18:47:11.498 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:47:11.494+0000 7fed52b8b600 0 WARNING: could not read zone params for zone id=823fb3a5-4d8a-49dc-ae64-e05804bbea94 name=test-zone3 2022-09-12T18:47:11.499 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:47:11.494+0000 7fed52b8b600 0 failed reading obj info from .rgw.root:zone_info.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47: (2) No such file or directory 2022-09-12T18:47:11.499 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:47:11.494+0000 7fed52b8b600 0 WARNING: could not read zone params for zone id=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47 name=test-zone1 2022-09-12T18:47:11.501 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:47:11.502 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "eb41eca3-59e3-4448-adac-b7c5067fb406", 2022-09-12T18:47:11.502 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone4", 2022-09-12T18:47:11.502 INFO:teuthology.orchestra.run.smithi138.stdout: "domain_root": "test-zone4.rgw.meta:root", 2022-09-12T18:47:11.503 INFO:teuthology.orchestra.run.smithi138.stdout: "control_pool": "test-zone4.rgw.control", 2022-09-12T18:47:11.503 INFO:teuthology.orchestra.run.smithi138.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2022-09-12T18:47:11.504 INFO:teuthology.orchestra.run.smithi138.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2022-09-12T18:47:11.504 INFO:teuthology.orchestra.run.smithi138.stdout: "log_pool": "test-zone4.rgw.log", 2022-09-12T18:47:11.504 INFO:teuthology.orchestra.run.smithi138.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2022-09-12T18:47:11.505 INFO:teuthology.orchestra.run.smithi138.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2022-09-12T18:47:11.505 INFO:teuthology.orchestra.run.smithi138.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2022-09-12T18:47:11.506 INFO:teuthology.orchestra.run.smithi138.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2022-09-12T18:47:11.506 INFO:teuthology.orchestra.run.smithi138.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2022-09-12T18:47:11.506 INFO:teuthology.orchestra.run.smithi138.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2022-09-12T18:47:11.507 INFO:teuthology.orchestra.run.smithi138.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2022-09-12T18:47:11.507 INFO:teuthology.orchestra.run.smithi138.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2022-09-12T18:47:11.508 INFO:teuthology.orchestra.run.smithi138.stdout: "otp_pool": "test-zone4.rgw.otp", 2022-09-12T18:47:11.508 INFO:teuthology.orchestra.run.smithi138.stdout: "system_key": { 2022-09-12T18:47:11.508 INFO:teuthology.orchestra.run.smithi138.stdout: "access_key": "RDIEFLSOLOEHUH1F", 2022-09-12T18:47:11.509 INFO:teuthology.orchestra.run.smithi138.stdout: "secret_key": "LDCKRd1RecZNgErNcVDnUigDhC7QggCQ" 2022-09-12T18:47:11.509 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:11.510 INFO:teuthology.orchestra.run.smithi138.stdout: "placement_pools": [ 2022-09-12T18:47:11.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:11.510 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "default-placement", 2022-09-12T18:47:11.510 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:11.511 INFO:teuthology.orchestra.run.smithi138.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2022-09-12T18:47:11.511 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_classes": { 2022-09-12T18:47:11.511 INFO:teuthology.orchestra.run.smithi138.stdout: "STANDARD": { 2022-09-12T18:47:11.511 INFO:teuthology.orchestra.run.smithi138.stdout: "data_pool": "test-zone4.rgw.buckets.data" 2022-09-12T18:47:11.512 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:11.512 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:11.512 INFO:teuthology.orchestra.run.smithi138.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2022-09-12T18:47:11.513 INFO:teuthology.orchestra.run.smithi138.stdout: "index_type": 0 2022-09-12T18:47:11.513 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:11.513 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:11.514 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T18:47:11.514 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_config": { 2022-09-12T18:47:11.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event_retention_days": 7, 2022-09-12T18:47:11.514 INFO:teuthology.orchestra.run.smithi138.stdout: "start_with_full_sync": false 2022-09-12T18:47:11.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:11.515 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T18:47:11.515 INFO:teuthology.orchestra.run.smithi138.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2022-09-12T18:47:11.515 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:47:11.517 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2022-09-12T18:47:12.038 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2022-09-12T18:47:12.513 INFO:teuthology.orchestra.run.smithi138.stderr:pool 'test-zone4.rgw.buckets.data' created 2022-09-12T18:47:12.529 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2022-09-12T18:47:13.541 INFO:teuthology.orchestra.run.smithi138.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2022-09-12T18:47:13.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:47:13.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:47:13.554 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T18:47:13.606 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:47:13.607 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:47:13.629 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:47:13.630 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "eb41eca3-59e3-4448-adac-b7c5067fb406", 2022-09-12T18:47:13.630 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone4", 2022-09-12T18:47:13.630 INFO:teuthology.orchestra.run.smithi138.stdout: "domain_root": "test-zone4.rgw.meta:root", 2022-09-12T18:47:13.630 INFO:teuthology.orchestra.run.smithi138.stdout: "control_pool": "test-zone4.rgw.control", 2022-09-12T18:47:13.631 INFO:teuthology.orchestra.run.smithi138.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2022-09-12T18:47:13.631 INFO:teuthology.orchestra.run.smithi138.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2022-09-12T18:47:13.631 INFO:teuthology.orchestra.run.smithi138.stdout: "log_pool": "test-zone4.rgw.log", 2022-09-12T18:47:13.631 INFO:teuthology.orchestra.run.smithi138.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2022-09-12T18:47:13.632 INFO:teuthology.orchestra.run.smithi138.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2022-09-12T18:47:13.632 INFO:teuthology.orchestra.run.smithi138.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2022-09-12T18:47:13.632 INFO:teuthology.orchestra.run.smithi138.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2022-09-12T18:47:13.632 INFO:teuthology.orchestra.run.smithi138.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2022-09-12T18:47:13.633 INFO:teuthology.orchestra.run.smithi138.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2022-09-12T18:47:13.633 INFO:teuthology.orchestra.run.smithi138.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2022-09-12T18:47:13.633 INFO:teuthology.orchestra.run.smithi138.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2022-09-12T18:47:13.634 INFO:teuthology.orchestra.run.smithi138.stdout: "otp_pool": "test-zone4.rgw.otp", 2022-09-12T18:47:13.634 INFO:teuthology.orchestra.run.smithi138.stdout: "system_key": { 2022-09-12T18:47:13.635 INFO:teuthology.orchestra.run.smithi138.stdout: "access_key": "RDIEFLSOLOEHUH1F", 2022-09-12T18:47:13.635 INFO:teuthology.orchestra.run.smithi138.stdout: "secret_key": "LDCKRd1RecZNgErNcVDnUigDhC7QggCQ" 2022-09-12T18:47:13.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:13.636 INFO:teuthology.orchestra.run.smithi138.stdout: "placement_pools": [ 2022-09-12T18:47:13.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:13.637 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "default-placement", 2022-09-12T18:47:13.638 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:13.638 INFO:teuthology.orchestra.run.smithi138.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2022-09-12T18:47:13.639 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_classes": { 2022-09-12T18:47:13.639 INFO:teuthology.orchestra.run.smithi138.stdout: "STANDARD": { 2022-09-12T18:47:13.640 INFO:teuthology.orchestra.run.smithi138.stdout: "data_pool": "test-zone4.rgw.buckets.data", 2022-09-12T18:47:13.640 INFO:teuthology.orchestra.run.smithi138.stdout: "compression_type": "random" 2022-09-12T18:47:13.640 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:13.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:13.641 INFO:teuthology.orchestra.run.smithi138.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2022-09-12T18:47:13.642 INFO:teuthology.orchestra.run.smithi138.stdout: "index_type": 0 2022-09-12T18:47:13.642 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:13.643 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:13.643 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T18:47:13.644 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_config": { 2022-09-12T18:47:13.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event_retention_days": 7, 2022-09-12T18:47:13.644 INFO:teuthology.orchestra.run.smithi138.stdout: "start_with_full_sync": false 2022-09-12T18:47:13.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:13.645 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "", 2022-09-12T18:47:13.645 INFO:teuthology.orchestra.run.smithi138.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2022-09-12T18:47:13.646 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:47:13.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:47:13.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:47:13.648 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T18:47:13.697 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:47:13.697 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:47:13.708 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:47:13.704+0000 7f1ac8c3f600 0 period (c0f7a512-c940-4d1f-8adf-3484630f876b does not have zone eb41eca3-59e3-4448-adac-b7c5067fb406 configured 2022-09-12T18:47:20.565 INFO:teuthology.orchestra.run.smithi138.stderr:Sending period to new master zone fb29b4fb-1f78-4524-ab80-6a5e9e01ab47 2022-09-12T18:47:20.601 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:47:20.602 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:47:20.602 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 4, 2022-09-12T18:47:20.602 INFO:teuthology.orchestra.run.smithi138.stdout: "predecessor_uuid": "2e627c13-04da-45e8-bcee-0ed7248cedbf", 2022-09-12T18:47:20.603 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": [], 2022-09-12T18:47:20.603 INFO:teuthology.orchestra.run.smithi138.stdout: "period_map": { 2022-09-12T18:47:20.603 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:47:20.603 INFO:teuthology.orchestra.run.smithi138.stdout: "zonegroups": [ 2022-09-12T18:47:20.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:20.604 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "test-zonegroup", 2022-09-12T18:47:20.604 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zonegroup", 2022-09-12T18:47:20.604 INFO:teuthology.orchestra.run.smithi138.stdout: "api_name": "", 2022-09-12T18:47:20.604 INFO:teuthology.orchestra.run.smithi138.stdout: "is_master": true, 2022-09-12T18:47:20.605 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T18:47:20.605 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T18:47:20.605 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T18:47:20.605 INFO:teuthology.orchestra.run.smithi138.stdout: "hostnames": [], 2022-09-12T18:47:20.606 INFO:teuthology.orchestra.run.smithi138.stdout: "hostnames_s3website": [], 2022-09-12T18:47:20.606 INFO:teuthology.orchestra.run.smithi138.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T18:47:20.606 INFO:teuthology.orchestra.run.smithi138.stdout: "zones": [ 2022-09-12T18:47:20.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:20.606 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T18:47:20.607 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone3", 2022-09-12T18:47:20.607 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T18:47:20.607 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi035.front.sepia.ceph.com:8001" 2022-09-12T18:47:20.607 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T18:47:20.608 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T18:47:20.608 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T18:47:20.608 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T18:47:20.608 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T18:47:20.608 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "", 2022-09-12T18:47:20.609 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": true, 2022-09-12T18:47:20.609 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [], 2022-09-12T18:47:20.609 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T18:47:20.609 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T18:47:20.610 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T18:47:20.610 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:47:20.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:20.610 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:20.610 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "eb41eca3-59e3-4448-adac-b7c5067fb406", 2022-09-12T18:47:20.611 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone4", 2022-09-12T18:47:20.611 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T18:47:20.611 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi138.front.sepia.ceph.com:8001" 2022-09-12T18:47:20.611 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T18:47:20.611 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T18:47:20.612 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T18:47:20.612 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T18:47:20.612 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T18:47:20.612 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "pubsub", 2022-09-12T18:47:20.613 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": false, 2022-09-12T18:47:20.613 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [ 2022-09-12T18:47:20.613 INFO:teuthology.orchestra.run.smithi138.stdout: "test-zone1" 2022-09-12T18:47:20.613 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T18:47:20.613 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T18:47:20.614 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T18:47:20.614 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T18:47:20.614 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:47:20.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:20.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:20.615 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T18:47:20.615 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone1", 2022-09-12T18:47:20.615 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T18:47:20.615 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T18:47:20.616 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T18:47:20.616 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T18:47:20.616 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T18:47:20.616 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T18:47:20.617 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T18:47:20.617 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "", 2022-09-12T18:47:20.617 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": true, 2022-09-12T18:47:20.617 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [], 2022-09-12T18:47:20.617 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T18:47:20.618 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T18:47:20.618 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T18:47:20.618 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:47:20.618 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:20.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:20.619 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T18:47:20.619 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone2", 2022-09-12T18:47:20.619 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T18:47:20.619 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi138.front.sepia.ceph.com:8000" 2022-09-12T18:47:20.620 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T18:47:20.620 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T18:47:20.620 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T18:47:20.620 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T18:47:20.620 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T18:47:20.621 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "", 2022-09-12T18:47:20.621 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": true, 2022-09-12T18:47:20.621 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [], 2022-09-12T18:47:20.621 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T18:47:20.622 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T18:47:20.622 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T18:47:20.622 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:47:20.622 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:20.622 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T18:47:20.623 INFO:teuthology.orchestra.run.smithi138.stdout: "placement_targets": [ 2022-09-12T18:47:20.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:20.623 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "default-placement", 2022-09-12T18:47:20.623 INFO:teuthology.orchestra.run.smithi138.stdout: "tags": [], 2022-09-12T18:47:20.624 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_classes": [ 2022-09-12T18:47:20.625 INFO:teuthology.orchestra.run.smithi138.stdout: "STANDARD" 2022-09-12T18:47:20.625 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:47:20.625 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:20.626 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T18:47:20.626 INFO:teuthology.orchestra.run.smithi138.stdout: "default_placement": "default-placement", 2022-09-12T18:47:20.626 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T18:47:20.626 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_policy": { 2022-09-12T18:47:20.627 INFO:teuthology.orchestra.run.smithi138.stdout: "groups": [] 2022-09-12T18:47:20.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:20.627 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled_features": [ 2022-09-12T18:47:20.627 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T18:47:20.627 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:47:20.628 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:20.628 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T18:47:20.628 INFO:teuthology.orchestra.run.smithi138.stdout: "short_zone_ids": [ 2022-09-12T18:47:20.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:20.628 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T18:47:20.629 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 227168046 2022-09-12T18:47:20.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:20.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:20.629 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "eb41eca3-59e3-4448-adac-b7c5067fb406", 2022-09-12T18:47:20.630 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 2080702172 2022-09-12T18:47:20.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:20.630 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:20.630 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T18:47:20.630 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 422195086 2022-09-12T18:47:20.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:20.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:20.631 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T18:47:20.631 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 2881234289 2022-09-12T18:47:20.632 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:20.632 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:47:20.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:20.632 INFO:teuthology.orchestra.run.smithi138.stdout: "master_zonegroup": "test-zonegroup", 2022-09-12T18:47:20.632 INFO:teuthology.orchestra.run.smithi138.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T18:47:20.633 INFO:teuthology.orchestra.run.smithi138.stdout: "period_config": { 2022-09-12T18:47:20.633 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_quota": { 2022-09-12T18:47:20.633 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false, 2022-09-12T18:47:20.633 INFO:teuthology.orchestra.run.smithi138.stdout: "check_on_raw": false, 2022-09-12T18:47:20.633 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size": -1, 2022-09-12T18:47:20.634 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size_kb": 0, 2022-09-12T18:47:20.634 INFO:teuthology.orchestra.run.smithi138.stdout: "max_objects": -1 2022-09-12T18:47:20.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:20.634 INFO:teuthology.orchestra.run.smithi138.stdout: "user_quota": { 2022-09-12T18:47:20.635 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false, 2022-09-12T18:47:20.635 INFO:teuthology.orchestra.run.smithi138.stdout: "check_on_raw": false, 2022-09-12T18:47:20.635 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size": -1, 2022-09-12T18:47:20.635 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size_kb": 0, 2022-09-12T18:47:20.635 INFO:teuthology.orchestra.run.smithi138.stdout: "max_objects": -1 2022-09-12T18:47:20.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:20.636 INFO:teuthology.orchestra.run.smithi138.stdout: "user_ratelimit": { 2022-09-12T18:47:20.636 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_ops": 0, 2022-09-12T18:47:20.636 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_ops": 0, 2022-09-12T18:47:20.636 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_bytes": 0, 2022-09-12T18:47:20.637 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_bytes": 0, 2022-09-12T18:47:20.637 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false 2022-09-12T18:47:20.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:20.637 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_ratelimit": { 2022-09-12T18:47:20.638 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_ops": 0, 2022-09-12T18:47:20.638 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_ops": 0, 2022-09-12T18:47:20.638 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_bytes": 0, 2022-09-12T18:47:20.638 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_bytes": 0, 2022-09-12T18:47:20.638 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false 2022-09-12T18:47:20.639 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:20.639 INFO:teuthology.orchestra.run.smithi138.stdout: "anonymous_ratelimit": { 2022-09-12T18:47:20.639 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_ops": 0, 2022-09-12T18:47:20.639 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_ops": 0, 2022-09-12T18:47:20.640 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_bytes": 0, 2022-09-12T18:47:20.640 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_bytes": 0, 2022-09-12T18:47:20.640 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false 2022-09-12T18:47:20.640 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:20.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:20.641 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T18:47:20.641 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_name": "test-realm", 2022-09-12T18:47:20.641 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:20.641 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:47:20.642 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2022-09-12T18:47:20.643 INFO:teuthology.orchestra.run:waiting for 300 2022-09-12T18:47:20.643 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T18:47:20.639+0000 7fe0a9b51700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 107178) UID: 0 2022-09-12T18:47:20.644 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T18:47:20.639+0000 7fe0addbc600 -1 shutting down 2022-09-12T18:47:22.053 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2022-09-12T18:47:26.745 INFO:tasks.rgw.c2.client.1:Stopped 2022-09-12T18:47:26.745 INFO:tasks.rgw.c2.client.1:Restarting daemon 2022-09-12T18:47:26.746 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2022-09-12T18:47:26.749 INFO:tasks.rgw.c2.client.1:Started 2022-09-12T18:47:26.749 DEBUG:teuthology.orchestra.run.smithi138:> curl http://smithi138.front.sepia.ceph.com:8001 2022-09-12T18:47:26.780 INFO:teuthology.orchestra.run.smithi138.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-12T18:47:26.781 INFO:teuthology.orchestra.run.smithi138.stderr: Dload Upload Total Spent Left Speed 2022-09-12T18:47:26.782 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-12T18:47:26.783 INFO:teuthology.orchestra.run.smithi138.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi138.front.sepia.ceph.com port 8001: Connection refused 2022-09-12T18:47:27.761 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2022-09-12T18:47:27.784 DEBUG:teuthology.orchestra.run.smithi138:> curl http://smithi138.front.sepia.ceph.com:8001 2022-09-12T18:47:27.811 INFO:teuthology.orchestra.run.smithi138.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-12T18:47:27.811 INFO:teuthology.orchestra.run.smithi138.stderr: Dload Upload Total Spent Left Speed 2022-09-12T18:47:27.811 INFO:teuthology.orchestra.run.smithi138.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 130k 0 --:--:-- --:--:-- --:--:-- 130k 2022-09-12T18:47:27.814 INFO:teuthology.orchestra.run.smithi138.stdout:{"Code":"MethodNotAllowed","RequestId":"tx00000504eab327859ad11-00631f7ebf-10ea-test-zone4","HostId":"10ea-test-zone4-test-zonegroup"} 2022-09-12T18:47:27.815 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2022-09-12T18:47:27.816 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2022-09-12T18:47:27.833 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/github.com_ceph_ceph-c_10f7d25d6a0f340676cd251eab5a787ba01ace95/qa/../src/test/rgw/rgw_multi 2022-09-12T18:47:27.853 INFO:tasks.rgw_multisite_tests:creating test user.. 2022-09-12T18:47:27.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:47:27.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T18:47:27.854 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T18:47:27.920 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:47:27.921 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:47:27.985 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T18:47:27.986 INFO:teuthology.orchestra.run.smithi035.stdout: "user_id": "rgw-multisite-test-user", 2022-09-12T18:47:27.986 INFO:teuthology.orchestra.run.smithi035.stdout: "display_name": "Multisite Test User", 2022-09-12T18:47:27.986 INFO:teuthology.orchestra.run.smithi035.stdout: "email": "", 2022-09-12T18:47:27.987 INFO:teuthology.orchestra.run.smithi035.stdout: "suspended": 0, 2022-09-12T18:47:27.987 INFO:teuthology.orchestra.run.smithi035.stdout: "max_buckets": 1000, 2022-09-12T18:47:27.987 INFO:teuthology.orchestra.run.smithi035.stdout: "subusers": [], 2022-09-12T18:47:27.987 INFO:teuthology.orchestra.run.smithi035.stdout: "keys": [ 2022-09-12T18:47:27.988 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:47:27.988 INFO:teuthology.orchestra.run.smithi035.stdout: "user": "rgw-multisite-test-user", 2022-09-12T18:47:27.988 INFO:teuthology.orchestra.run.smithi035.stdout: "access_key": "FUCN3RT6SAOJRSRI0ZIX", 2022-09-12T18:47:27.989 INFO:teuthology.orchestra.run.smithi035.stdout: "secret_key": "GiY8Aw7BLTX1KRc2ZO1gRib8SqZfBmHcFnjpBptL" 2022-09-12T18:47:27.989 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:47:27.990 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T18:47:27.990 INFO:teuthology.orchestra.run.smithi035.stdout: "swift_keys": [], 2022-09-12T18:47:27.991 INFO:teuthology.orchestra.run.smithi035.stdout: "caps": [ 2022-09-12T18:47:27.991 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:47:27.991 INFO:teuthology.orchestra.run.smithi035.stdout: "type": "roles", 2022-09-12T18:47:27.992 INFO:teuthology.orchestra.run.smithi035.stdout: "perm": "*" 2022-09-12T18:47:27.992 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:47:27.993 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T18:47:27.993 INFO:teuthology.orchestra.run.smithi035.stdout: "op_mask": "read, write, delete", 2022-09-12T18:47:27.994 INFO:teuthology.orchestra.run.smithi035.stdout: "default_placement": "", 2022-09-12T18:47:27.994 INFO:teuthology.orchestra.run.smithi035.stdout: "default_storage_class": "", 2022-09-12T18:47:27.994 INFO:teuthology.orchestra.run.smithi035.stdout: "placement_tags": [], 2022-09-12T18:47:27.995 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_quota": { 2022-09-12T18:47:27.995 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false, 2022-09-12T18:47:27.995 INFO:teuthology.orchestra.run.smithi035.stdout: "check_on_raw": false, 2022-09-12T18:47:27.996 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size": -1, 2022-09-12T18:47:27.996 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size_kb": 0, 2022-09-12T18:47:27.996 INFO:teuthology.orchestra.run.smithi035.stdout: "max_objects": -1 2022-09-12T18:47:27.996 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:47:27.996 INFO:teuthology.orchestra.run.smithi035.stdout: "user_quota": { 2022-09-12T18:47:27.997 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false, 2022-09-12T18:47:27.997 INFO:teuthology.orchestra.run.smithi035.stdout: "check_on_raw": false, 2022-09-12T18:47:27.997 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size": -1, 2022-09-12T18:47:27.997 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size_kb": 0, 2022-09-12T18:47:27.998 INFO:teuthology.orchestra.run.smithi035.stdout: "max_objects": -1 2022-09-12T18:47:27.998 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:47:27.998 INFO:teuthology.orchestra.run.smithi035.stdout: "temp_url_keys": [], 2022-09-12T18:47:27.998 INFO:teuthology.orchestra.run.smithi035.stdout: "type": "rgw", 2022-09-12T18:47:27.999 INFO:teuthology.orchestra.run.smithi035.stdout: "mfa_ids": [] 2022-09-12T18:47:27.999 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T18:47:27.999 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T18:47:28.012 INFO:rgw_multi.tests:meta checkpoint 2022-09-12T18:47:28.013 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:47:28.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:47:28.014 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:47:28.064 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:47:28.064 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:47:28.102 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=c0f7a512-c940-4d1f-8adf-3484630f876b 2022-09-12T18:47:28.114 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T18:47:28.114 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:47:28.114 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:47:28.115 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:47:27.976965Z" 2022-09-12T18:47:28.115 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:47:28.115 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:47:28.115 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:47:28.116 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T18:47:28.116 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:47:28.116 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:47:28.116 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:47:28.117 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T18:47:28.117 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:47:28.117 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:47:28.118 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:47:28.118 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T18:47:28.118 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:47:28.118 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T18:47:28.120 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T18:47:28.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:47:28.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:47:28.121 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:47:28.177 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:47:28.177 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:47:28.223 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:47:28.220+0000 7fb6816bb600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:47:28.224 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:47:28.221+0000 7fb6816bb600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:47:28.236 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:47:28.236 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:47:28.236 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:47:28.237 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:47:28.237 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:47:28.237 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:47:28.237 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:28.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:28.238 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:47:28.238 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:28.238 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:47:28.239 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:28.240 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:28.240 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:28.240 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:28.240 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:47:28.241 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:28.241 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:28.241 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:28.241 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:28.242 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:28.242 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:28.242 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:47:28.242 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:28.243 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:28.243 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:28.243 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:28.243 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:47:28.244 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:28.244 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:28.244 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:28.244 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:28.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:28.245 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:28.245 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:47:28.245 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:28.245 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:28.246 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:28.246 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:28.246 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:47:28.246 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:28.246 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:28.247 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:28.247 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:28.247 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:28.247 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:28.248 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:47:28.248 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:28.248 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:28.248 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:28.249 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:28.249 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:47:28.249 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:28.249 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:28.249 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:28.250 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:28.250 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:28.250 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:47:28.250 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:28.251 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:47:28.251 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:47:28.251 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:47:28.251 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:28.252 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:47:28Z" 2022-09-12T18:47:28.252 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:47:28.253 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663008447.976965_53.1 target= 2022-09-12T18:47:33.254 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:47:33.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:47:33.255 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:47:33.311 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:47:33.312 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:47:33.350 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:47:33.347+0000 7f9aa13fb600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:47:33.351 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:47:33.348+0000 7f9aa13fb600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:47:33.367 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:47:33.367 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:47:33.368 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:47:33.368 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:47:33.369 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:47:33.369 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:47:33.369 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:33.370 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:33.370 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:47:33.370 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:33.370 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:47:33.371 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:33.371 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:33.371 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:33.371 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:33.372 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:47:33.372 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:33.372 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:33.372 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:33.372 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:33.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:33.373 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:33.373 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:47:33.373 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:33.374 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:33.374 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:33.374 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:33.374 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:47:33.374 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:33.375 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:33.375 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:33.375 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:33.375 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:33.376 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:33.376 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:47:33.376 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:33.376 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:33.376 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:33.377 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:33.377 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:47:33.377 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:33.377 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:33.378 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:33.378 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:33.378 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:33.378 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:33.379 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:47:33.379 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:33.379 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:33.379 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:33.379 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:33.380 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:47:33.380 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:33.380 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:33.380 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:33.381 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:33.381 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:33.381 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:47:33.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:33.382 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:47:33.382 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:47:33.382 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:47:33.383 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:33.383 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:47:33Z" 2022-09-12T18:47:33.383 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:47:33.385 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663008447.976965_53.1 target= 2022-09-12T18:47:38.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:47:38.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:47:38.386 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:47:38.445 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:47:38.445 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:47:38.487 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:47:38.483+0000 7fda8ffab600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:47:38.488 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:47:38.484+0000 7fda8ffab600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:47:38.499 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:47:38.500 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:47:38.500 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:47:38.500 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:47:38.501 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:47:38.501 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:47:38.501 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:38.501 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:38.502 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:47:38.502 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:38.502 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:47:38.502 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:38.503 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:38.503 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:38.503 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:38.503 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:47:38.504 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:38.504 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:38.504 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:38.504 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:38.504 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:38.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:38.505 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:47:38.505 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:38.505 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:38.506 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:38.506 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:38.506 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:47:38.506 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:38.507 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:38.507 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:38.507 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:38.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:38.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:38.508 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:47:38.508 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:38.508 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:38.508 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:38.509 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:38.509 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:47:38.509 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:38.509 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:38.510 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:38.510 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:38.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:38.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:38.510 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:47:38.511 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:38.511 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:38.511 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:38.511 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:38.512 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:47:38.512 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:38.512 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:38.512 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:38.512 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:38.513 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:38.513 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:47:38.513 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:38.513 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:47:38.513 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:47:38.514 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:47:38.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:38.514 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:47:38Z" 2022-09-12T18:47:38.514 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:47:38.516 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663008447.976965_53.1 target= 2022-09-12T18:47:43.517 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:47:43.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:47:43.518 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:47:43.576 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:47:43.577 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:47:43.619 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:47:43.615+0000 7f2541a57600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:47:43.619 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:47:43.616+0000 7f2541a57600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:47:43.635 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:47:43.636 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:47:43.636 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:47:43.636 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:47:43.637 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:47:43.637 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:47:43.637 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:43.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:43.638 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:47:43.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:43.638 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:47:43.638 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:43.639 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:43.639 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:43.639 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:43.639 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:47:43.639 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:43.640 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:43.640 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:43.640 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:43.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:43.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:43.641 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:47:43.641 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:43.641 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:43.642 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:43.642 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:43.642 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:47:43.642 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:43.643 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:43.643 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:43.643 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:43.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:43.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:43.644 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:47:43.644 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:43.644 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:43.644 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:43.645 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:43.645 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:47:43.645 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:43.645 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:43.646 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:43.646 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:43.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:43.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:43.647 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:47:43.647 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:43.647 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:43.648 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:43.648 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:43.648 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:47:43.648 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:43.649 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:43.649 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:43.649 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:43.649 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:43.649 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:47:43.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:43.650 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:47:43.650 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:47:43.650 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:47:43.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:43.651 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:47:43Z" 2022-09-12T18:47:43.651 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:47:43.653 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663008447.976965_53.1 target= 2022-09-12T18:47:48.654 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:47:48.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:47:48.656 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:47:48.713 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:47:48.713 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:47:48.746 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:47:48.742+0000 7f3d8026f600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:47:48.747 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:47:48.743+0000 7f3d8026f600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:47:48.766 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:47:48.766 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:47:48.767 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:47:48.767 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:47:48.767 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:47:48.767 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:47:48.768 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:48.768 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:48.768 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:47:48.768 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:48.769 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:47:48.769 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:48.769 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:48.769 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:48.770 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:48.770 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:47:48.770 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:48.770 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:48.771 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:48.771 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:48.771 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:48.771 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:48.771 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:47:48.772 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:48.772 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:48.772 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:48.772 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:48.773 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:47:48.773 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:48.773 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:48.773 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:48.774 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:48.774 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:48.774 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:48.774 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:47:48.775 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:48.775 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:48.775 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:48.775 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:48.775 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:47:48.776 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:48.776 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:48.776 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:48.776 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:48.777 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:48.777 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:48.778 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:47:48.778 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:48.779 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:48.779 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:48.780 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:48.780 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:47:48.780 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:48.780 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:48.781 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:48.781 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:48.781 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:48.781 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:47:48.781 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:48.782 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:47:48.782 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:47:48.782 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:47:48.782 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:48.783 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:47:48Z" 2022-09-12T18:47:48.783 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:47:48.786 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663008447.976965_53.1 target= 2022-09-12T18:47:53.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:47:53.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:47:53.788 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:47:53.847 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:47:53.847 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:47:53.898 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:47:53.894+0000 7fa5e99bd600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:47:53.899 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:47:53.895+0000 7fa5e99bd600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:47:53.917 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:47:53.918 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:47:53.918 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:47:53.919 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:47:53.920 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:47:53.920 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:47:53.920 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:53.921 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:53.921 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:47:53.921 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:53.922 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:47:53.922 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:53.922 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:53.923 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:53.923 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:53.924 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:47:53.924 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:53.924 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:53.925 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:53.925 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:53.925 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:53.926 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:53.926 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:47:53.926 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:53.927 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:53.927 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:53.928 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:53.928 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:47:53.928 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:53.929 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:53.929 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:53.929 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:53.930 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:53.930 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:53.931 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:47:53.931 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:53.931 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:53.932 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:53.932 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:53.932 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:47:53.933 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:53.933 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:53.933 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:53.934 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:53.934 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:53.935 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:53.935 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:47:53.935 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:53.936 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:53.936 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:53.936 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:53.937 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:47:53.937 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:53.937 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:53.938 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:53.938 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:53.939 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:53.939 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:47:53.939 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:53.940 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:47:53.940 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:47:53.940 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:47:53.941 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:53.941 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:47:53Z" 2022-09-12T18:47:53.941 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:47:53.943 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663008447.976965_53.1 target= 2022-09-12T18:47:58.945 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:47:58.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:47:58.948 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:47:59.005 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:47:59.006 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:47:59.044 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:47:59.040+0000 7ff4e94ae600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:47:59.045 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:47:59.041+0000 7ff4e94ae600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:47:59.057 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:47:59.057 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:47:59.057 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:47:59.058 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:47:59.058 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:47:59.058 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:47:59.059 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:59.059 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:59.059 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:47:59.059 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:59.059 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:47:59.060 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:59.060 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:59.060 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:59.061 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:59.061 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:47:59.061 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:59.061 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:59.062 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:59.062 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:59.062 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:59.062 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:59.063 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:47:59.063 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:59.064 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:59.064 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:59.065 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:59.066 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:47:59.066 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:59.066 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:59.067 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:59.067 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:59.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:59.067 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:59.068 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:47:59.068 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:59.068 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:59.068 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:59.069 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:59.069 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:47:59.069 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:59.070 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:59.070 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:59.070 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:59.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:59.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:47:59.071 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:47:59.071 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:47:59.071 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:47:59.072 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:47:59.072 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:47:59.072 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:47:59.072 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:47:59.073 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:47:59.073 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:47:59.073 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:59.073 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:47:59.073 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:47:59.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:59.074 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:47:59.075 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:47:59.075 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:47:59.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:47:59.076 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:47:59Z" 2022-09-12T18:47:59.076 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:47:59.079 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663008447.976965_53.1 target= 2022-09-12T18:48:04.081 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:48:04.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:48:04.082 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:48:04.138 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:48:04.139 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:48:04.174 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:48:04.171+0000 7faef8197600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:48:04.175 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:48:04.172+0000 7faef8197600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:48:04.199 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:48:04.199 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:48:04.200 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:48:04.200 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:48:04.200 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:48:04.201 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:48:04.201 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:04.201 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:04.201 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:48:04.202 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:04.202 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:48:04.202 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:04.202 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:04.203 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:04.203 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:04.203 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:48:04.204 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:04.204 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:04.204 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:04.204 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:04.204 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:04.205 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:04.205 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:48:04.205 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:04.205 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:04.206 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:04.206 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:04.206 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:04.206 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:04.207 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:04.207 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:04.207 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:04.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:04.208 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:04.208 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:48:04.208 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:04.208 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:04.208 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:04.209 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:04.209 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:04.209 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:04.209 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:04.210 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:04.210 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:04.210 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:04.210 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:04.211 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:48:04.211 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:04.211 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:04.211 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:04.211 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:04.212 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:04.212 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:04.212 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:04.212 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:04.213 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:04.213 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:04.213 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:48:04.213 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:04.214 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:48:04.214 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:48:04.214 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:48:04.215 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:04.215 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:48:04Z" 2022-09-12T18:48:04.215 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:48:04.217 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663008447.976965_53.1 target= 2022-09-12T18:48:09.218 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:48:09.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:48:09.219 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:48:09.275 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:48:09.275 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:48:09.326 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:48:09.322+0000 7fb17e61a600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:48:09.327 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:48:09.323+0000 7fb17e61a600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:48:09.339 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:48:09.339 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:48:09.340 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:48:09.340 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:48:09.341 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:48:09.341 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:48:09.342 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:09.342 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:09.343 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:48:09.344 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:09.344 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:48:09.345 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:09.345 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:09.346 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:09.346 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:09.346 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:48:09.347 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:09.347 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:09.348 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:09.348 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:09.349 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:09.349 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:09.350 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:48:09.350 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:09.351 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:09.351 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:09.352 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:09.352 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:09.353 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:09.353 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:09.354 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:09.354 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:09.355 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:09.355 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:09.356 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:48:09.356 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:09.357 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:09.357 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:09.357 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:09.358 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:09.358 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:09.359 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:09.359 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:09.360 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:09.360 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:09.361 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:09.361 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:48:09.362 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:09.362 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:09.363 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:09.364 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:09.364 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:09.364 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:09.365 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:09.365 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:09.366 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:09.366 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:09.367 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:48:09.367 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:09.368 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:48:09.368 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:48:09.369 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:48:09.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:09.370 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:48:09Z" 2022-09-12T18:48:09.370 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:48:09.373 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663008447.976965_53.1 target= 2022-09-12T18:48:14.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:48:14.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:48:14.376 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:48:14.433 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:48:14.433 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:48:14.463 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:48:14.459+0000 7faefa94f600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:48:14.464 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:48:14.460+0000 7faefa94f600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:48:14.475 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:48:14.476 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:48:14.476 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:48:14.476 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:48:14.476 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:48:14.477 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:48:14.477 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:14.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:14.477 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:48:14.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:14.478 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:48:14.478 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:14.478 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:14.478 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:14.479 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:14.479 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:48:14.479 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:14.479 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:14.480 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:14.480 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:14.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:14.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:14.481 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:48:14.481 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:14.481 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:14.481 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:14.482 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:14.482 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:14.482 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:14.482 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:14.482 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:14.483 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:14.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:14.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:14.483 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:48:14.484 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:14.484 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:14.484 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:14.485 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:14.485 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:14.485 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:14.485 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:14.486 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:14.486 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:14.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:14.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:14.487 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:48:14.487 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:14.487 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:14.488 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:14.488 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:14.488 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:14.489 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:14.489 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:14.489 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:14.489 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:14.490 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:14.490 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:48:14.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:14.490 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:48:14.491 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:48:14.491 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:48:14.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:14.492 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:48:14Z" 2022-09-12T18:48:14.492 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:48:14.494 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663008447.976965_53.1 target= 2022-09-12T18:48:19.494 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:48:19.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:48:19.496 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:48:19.552 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:48:19.553 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:48:19.599 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:48:19.596+0000 7f0a2a052600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:48:19.599 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:48:19.597+0000 7f0a2a052600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:48:19.618 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:48:19.618 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:48:19.619 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:48:19.619 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:48:19.619 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:48:19.619 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:48:19.619 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:19.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:19.620 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:48:19.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:19.620 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:48:19.621 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:19.621 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:19.621 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:19.621 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:19.622 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:48:19.622 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:19.622 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:19.622 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:19.623 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:19.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:19.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:19.623 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:48:19.623 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:19.624 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:19.624 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:19.624 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:19.624 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:19.625 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:19.625 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:19.625 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:19.625 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:19.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:19.626 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:19.626 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:48:19.626 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:19.626 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:19.627 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:19.627 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:19.627 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:19.627 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:19.628 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:19.628 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:19.628 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:19.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:19.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:19.629 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:48:19.629 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:19.629 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:19.629 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:19.630 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:19.630 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:19.630 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:19.630 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:19.631 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:19.631 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:19.631 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:19.631 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:48:19.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:19.632 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:48:19.632 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:48:19.632 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:48:19.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:19.633 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:48:19Z" 2022-09-12T18:48:19.633 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:48:19.634 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663008447.976965_53.1 target= 2022-09-12T18:48:24.635 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:48:24.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:48:24.637 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:48:24.693 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:48:24.694 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:48:24.725 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:48:24.722+0000 7f6b682b4600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:48:24.726 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:48:24.723+0000 7f6b682b4600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:48:24.738 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:48:24.738 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:48:24.739 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:48:24.739 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:48:24.739 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:48:24.739 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:48:24.740 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:24.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:24.740 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:48:24.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:24.741 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:48:24.741 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:24.741 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:24.742 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:24.742 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:24.742 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:48:24.742 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:24.743 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:24.743 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:24.743 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:24.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:24.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:24.744 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:48:24.744 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:24.744 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:24.744 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:24.745 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:24.745 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:24.745 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:24.745 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:24.746 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:24.746 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:24.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:24.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:24.747 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:48:24.747 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:24.747 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:24.748 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:24.748 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:24.748 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:24.748 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:24.748 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:24.749 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:24.749 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:24.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:24.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:24.750 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:48:24.750 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:24.750 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:24.750 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:24.751 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:24.751 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:24.751 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:24.751 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:24.752 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:24.752 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:24.752 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:24.752 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:48:24.752 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:24.753 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:48:24.753 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:48:24.753 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:48:24.753 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:24.754 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:48:24Z" 2022-09-12T18:48:24.754 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:48:24.755 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663008447.976965_53.1 target= 2022-09-12T18:48:29.757 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:48:29.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:48:29.758 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:48:29.815 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:48:29.815 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:48:29.861 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:48:29.857+0000 7f8cacf48600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:48:29.862 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:48:29.859+0000 7f8cacf48600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:48:29.876 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:48:29.876 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:48:29.877 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:48:29.877 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:48:29.877 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:48:29.878 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:48:29.878 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:29.878 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:29.878 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:48:29.879 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:29.879 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:48:29.879 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:29.879 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:29.879 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:29.880 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:29.880 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:48:29.880 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:29.880 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:29.881 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:29.881 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:29.881 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:29.881 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:29.882 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:48:29.882 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:29.882 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:29.882 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:29.883 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:29.883 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:29.883 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:29.883 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:29.883 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:29.884 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:29.884 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:29.884 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:29.884 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:48:29.885 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:29.885 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:29.885 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:29.885 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:29.886 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:29.886 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:29.886 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:29.886 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:29.887 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:29.887 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:29.887 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:29.887 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:48:29.887 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:29.888 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:29.888 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:29.888 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:29.888 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:29.889 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:29.889 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:29.889 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:29.889 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:29.890 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:29.890 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:48:29.890 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:29.890 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:48:29.891 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:48:29.891 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:48:29.891 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:29.891 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:48:29Z" 2022-09-12T18:48:29.891 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:48:29.893 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663008447.976965_53.1 target= 2022-09-12T18:48:34.894 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:48:34.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:48:34.895 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:48:34.952 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:48:34.953 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:48:34.995 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:48:34.991+0000 7fd55ba12600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:48:34.996 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:48:34.992+0000 7fd55ba12600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:48:35.012 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:48:35.012 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:48:35.012 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:48:35.013 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:48:35.013 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:48:35.013 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:48:35.013 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:35.014 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:35.014 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:48:35.014 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:35.014 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:48:35.015 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:35.015 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:35.015 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:35.015 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:35.016 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:48:35.016 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:35.016 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:35.016 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:35.017 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:35.017 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:35.017 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:35.018 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:48:35.018 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:35.018 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:35.018 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:35.018 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:35.019 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:35.019 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:35.019 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:35.019 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:35.020 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:35.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:35.020 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:35.020 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:48:35.021 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:35.021 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:35.021 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:35.022 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:35.022 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:35.022 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:35.023 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:35.023 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:35.023 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:35.023 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:35.023 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:35.024 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:48:35.024 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:35.024 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:35.024 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:35.025 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:35.025 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:35.025 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:35.025 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:35.025 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:35.026 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:35.026 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:35.026 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:48:35.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:35.027 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:48:35.027 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:48:35.027 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:48:35.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:35.028 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:48:34Z" 2022-09-12T18:48:35.028 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:48:35.029 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663008447.976965_53.1 target= 2022-09-12T18:48:40.030 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:48:40.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:48:40.031 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:48:40.087 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:48:40.088 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:48:40.116 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:48:40.112+0000 7f62409eb600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:48:40.117 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:48:40.113+0000 7f62409eb600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:48:40.129 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:48:40.130 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:48:40.131 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:48:40.131 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:48:40.131 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:48:40.132 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:48:40.132 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:40.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:40.134 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:48:40.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:40.135 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:48:40.135 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:40.136 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:40.136 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:40.137 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:40.137 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:48:40.138 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:40.138 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:40.139 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:40.139 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:40.140 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:40.141 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:40.141 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:48:40.142 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:40.142 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:40.143 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:40.143 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:40.144 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:40.144 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:40.145 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:40.145 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:40.146 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:40.146 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:40.147 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:40.147 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:48:40.148 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:40.148 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:40.149 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:40.149 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:40.150 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:40.150 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:40.151 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:40.152 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:40.152 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:40.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:40.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:40.153 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:48:40.154 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:40.155 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:40.155 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:40.155 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:40.156 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:40.156 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:40.157 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:40.157 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:40.158 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:40.158 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:40.160 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:48:40.160 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:40.160 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:48:40.161 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:48:40.161 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:48:40.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:40.161 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:48:40Z" 2022-09-12T18:48:40.161 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:48:40.163 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663008447.976965_53.1 target= 2022-09-12T18:48:45.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:48:45.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:48:45.167 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:48:45.223 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:48:45.224 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:48:45.264 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:48:45.260+0000 7f9b2b21a600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:48:45.265 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:48:45.261+0000 7f9b2b21a600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:48:45.285 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:48:45.286 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:48:45.286 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:48:45.286 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:48:45.286 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:48:45.287 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:48:45.287 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:45.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:45.287 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:48:45.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:45.288 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:48:45.288 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:45.288 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:45.289 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:45.289 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:45.289 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:48:45.290 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:45.290 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:45.290 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:45.290 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:45.291 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:45.291 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:45.291 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:48:45.291 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:45.292 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:45.292 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:45.292 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:45.292 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:45.293 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:45.293 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:45.293 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:45.293 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:45.293 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:45.294 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:45.294 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:48:45.294 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:45.294 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:45.295 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:45.295 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:45.295 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:45.295 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:45.296 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:45.296 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:45.296 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:45.296 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:45.297 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:45.297 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:48:45.297 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:45.297 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:45.297 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:45.298 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:45.298 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:45.298 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:45.298 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:45.299 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:45.299 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:45.299 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:45.300 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:48:45.300 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:45.300 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:48:45.300 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:48:45.300 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:48:45.301 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:45.301 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:48:45Z" 2022-09-12T18:48:45.301 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:48:45.302 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663008447.976965_53.1 target= 2022-09-12T18:48:50.303 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:48:50.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:48:50.304 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:48:50.363 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:48:50.364 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:48:50.402 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:48:50.398+0000 7f54ad4da600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:48:50.403 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:48:50.399+0000 7f54ad4da600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:48:50.415 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:48:50.416 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:48:50.416 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:48:50.417 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:48:50.417 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:48:50.417 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:48:50.417 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:50.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:50.418 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:48:50.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:50.419 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:48:50.419 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:50.419 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:50.420 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:50.420 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:50.420 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:48:50.421 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:50.421 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:50.421 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:50.421 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:50.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:50.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:50.422 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:48:50.423 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:50.423 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:50.423 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:50.424 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:50.424 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:50.424 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:50.425 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:50.425 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:50.425 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:50.426 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:50.426 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:50.427 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:48:50.427 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:50.428 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:50.428 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:50.428 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:50.429 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:50.429 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:50.430 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:50.430 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:50.430 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:50.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:50.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:50.431 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:48:50.431 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:50.432 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:50.432 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:50.432 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:50.433 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:50.433 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:50.433 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:50.434 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:50.434 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:50.434 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:50.435 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:48:50.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:50.435 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:48:50.435 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:48:50.436 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:48:50.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:50.436 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:48:50Z" 2022-09-12T18:48:50.437 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:48:50.438 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663008447.976965_53.1 target= 2022-09-12T18:48:55.439 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:48:55.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:48:55.441 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:48:55.498 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:48:55.499 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:48:55.528 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:48:55.524+0000 7f13e1b61600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:48:55.529 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:48:55.525+0000 7f13e1b61600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:48:55.543 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:48:55.543 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:48:55.543 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:48:55.544 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:48:55.544 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:48:55.544 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:48:55.544 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:55.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:55.545 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:48:55.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:55.545 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:48:55.546 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:55.546 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:55.546 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:55.546 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:55.547 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:48:55.547 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:55.547 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:55.547 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:55.548 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:55.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:55.548 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:55.549 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:48:55.549 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:55.549 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:55.550 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:55.550 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:55.551 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:55.551 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:55.552 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:55.552 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:55.552 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:55.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:55.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:55.553 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:48:55.553 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:55.553 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:55.554 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:55.554 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:55.554 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:55.554 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:55.555 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:55.555 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:55.556 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:55.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:55.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:48:55.557 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:48:55.557 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:48:55.557 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:48:55.558 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:48:55.558 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:48:55.559 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:48:55.559 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:48:55.559 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:48:55.560 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:48:55.560 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:55.561 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:48:55.561 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:48:55.561 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:55.562 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:48:55.562 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:48:55.563 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:48:55.563 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:48:55.563 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:48:55Z" 2022-09-12T18:48:55.564 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:48:55.566 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663008447.976965_53.1 target= 2022-09-12T18:49:00.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:00.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:00.569 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:00.626 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:00.627 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:00.679 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:00.675+0000 7f1bd4295600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:00.680 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:00.676+0000 7f1bd4295600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:00.696 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:49:00.697 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:49:00.697 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:49:00.698 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:49:00.698 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:49:00.699 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:00.699 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:00.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:00.700 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:49:00.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:00.701 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:49:00.701 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:00.702 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:00.702 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:00.703 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:00.703 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:49:00.703 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:00.704 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:00.704 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:00.705 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:00.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:00.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:00.706 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:49:00.707 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:00.707 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:00.707 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:00.708 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:00.708 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:00.709 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:00.709 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:00.710 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:00.710 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:00.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:00.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:00.711 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:49:00.712 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:00.712 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:00.713 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:00.713 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:00.714 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:00.714 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:00.715 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:00.715 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:00.715 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:00.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:00.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:00.717 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:49:00.717 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:00.718 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:00.718 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:00.719 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:00.719 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:00.720 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:00.720 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:00.720 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:00.721 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:00.721 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:00.722 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:49:00.722 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:00.723 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:49:00.723 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:49:00.724 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:49:00.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:00.724 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:49:00Z" 2022-09-12T18:49:00.725 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:49:00.727 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663008447.976965_53.1 target= 2022-09-12T18:49:05.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:05.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:05.731 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:05.789 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:05.789 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:05.820 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:05.817+0000 7fedf5fde600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:05.821 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:05.818+0000 7fedf5fde600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:05.833 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:49:05.833 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:49:05.833 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:49:05.834 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:49:05.834 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:49:05.835 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:05.835 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:05.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:05.836 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:49:05.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:05.837 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:49:05.837 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:05.837 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:05.838 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:05.838 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:05.839 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:49:05.839 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:05.839 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:05.840 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:05.840 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:05.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:05.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:05.841 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:49:05.844 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:05.844 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:05.844 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:05.845 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:05.845 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:05.845 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:05.846 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:05.846 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:05.846 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:05.846 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:05.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:05.847 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:49:05.847 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:05.847 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:05.848 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:05.848 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:05.848 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:05.848 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:05.848 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:05.849 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:05.849 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:05.849 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:05.849 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:05.850 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:49:05.850 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:05.850 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:05.850 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:05.851 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:05.852 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:05.853 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:05.853 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:05.853 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:05.854 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:05.854 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:05.855 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:49:05.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:05.855 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:49:05.855 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:49:05.856 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:49:05.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:05.856 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:49:05Z" 2022-09-12T18:49:05.856 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:49:05.858 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663008447.976965_53.1 target= 2022-09-12T18:49:10.859 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:10.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:10.861 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:10.918 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:10.919 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:10.969 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:10.966+0000 7f871a8e6600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:10.970 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:10.967+0000 7f871a8e6600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:10.988 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:49:10.988 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:49:10.989 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:49:10.989 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:49:10.990 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:49:10.990 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:10.990 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:10.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:10.991 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:49:10.991 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:10.991 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:49:10.991 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:10.992 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:10.992 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:10.992 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:10.992 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:49:10.993 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:10.993 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:10.994 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:10.994 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:10.994 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:10.994 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:10.995 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:49:10.995 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:10.995 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:10.995 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:10.996 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:10.996 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:10.996 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:10.996 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:10.996 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:10.997 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:10.997 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:10.997 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:10.998 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:49:10.998 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:10.998 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:10.998 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:10.999 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:10.999 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:10.999 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:10.999 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:10.999 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:11.000 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:11.000 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:11.000 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:11.000 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:49:11.001 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:11.001 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:11.001 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:11.001 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:11.001 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:11.002 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:11.002 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:11.002 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:11.002 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:11.003 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:11.003 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:49:11.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:11.004 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:49:11.004 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:49:11.004 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:49:11.004 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:11.005 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:49:10Z" 2022-09-12T18:49:11.005 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:49:11.006 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663008447.976965_53.1 target= 2022-09-12T18:49:16.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:16.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:16.008 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:16.065 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:16.066 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:16.115 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:16.111+0000 7f95461dc600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:16.116 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:16.112+0000 7f95461dc600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:16.139 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:49:16.140 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:49:16.140 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:49:16.140 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:49:16.141 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:49:16.141 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:16.141 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:16.142 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:16.142 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:49:16.142 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:16.142 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:49:16.143 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:16.143 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:16.143 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:16.143 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:16.143 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:49:16.144 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:16.144 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:16.144 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:16.144 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:16.145 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:16.145 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:16.145 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:49:16.145 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:16.146 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:16.146 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:16.146 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:16.146 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:16.147 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:16.147 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:16.147 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:16.147 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:16.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:16.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:16.148 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:49:16.148 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:16.149 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:16.149 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:16.149 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:16.149 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:16.149 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:16.150 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:16.150 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:16.150 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:16.151 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:16.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:16.152 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:49:16.152 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:16.153 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:16.153 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:16.153 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:16.154 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:16.154 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:16.155 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:16.155 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:16.156 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:16.156 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:16.156 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:49:16.156 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:16.157 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:49:16.157 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:49:16.157 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:49:16.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:16.158 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:49:16Z" 2022-09-12T18:49:16.158 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:49:16.158 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663008447.976965_53.1 target= 2022-09-12T18:49:21.160 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:21.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:21.161 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:21.218 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:21.219 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:21.249 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:21.245+0000 7f16490c4600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:21.250 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:21.247+0000 7f16490c4600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:21.262 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:49:21.263 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:49:21.263 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:49:21.263 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:49:21.263 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:49:21.264 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:21.264 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:21.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:21.265 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:49:21.265 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:21.265 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:49:21.266 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:21.266 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:21.266 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:21.267 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:21.267 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:49:21.267 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:21.267 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:21.268 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:21.268 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:21.268 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:21.268 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:21.268 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:49:21.269 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:21.269 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:21.269 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:21.269 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:21.270 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:21.270 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:21.270 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:21.270 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:21.270 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:21.271 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:21.271 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:21.271 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:49:21.271 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:21.272 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:21.272 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:21.272 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:21.272 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:21.272 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:21.273 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:21.273 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:21.273 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:21.273 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:21.274 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:21.274 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:49:21.274 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:21.274 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:21.275 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:21.275 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:21.275 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:21.275 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:21.276 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:21.276 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:21.276 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:21.276 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:21.276 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:49:21.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:21.277 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:49:21.277 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:49:21.277 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:49:21.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:21.278 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:49:21Z" 2022-09-12T18:49:21.278 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:49:21.279 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1663008447.976965_53.1 target= 2022-09-12T18:49:26.281 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:26.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:26.282 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:26.340 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:26.340 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:26.371 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:26.368+0000 7f53e65c2600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:26.372 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:26.369+0000 7f53e65c2600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:26.388 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:49:26.388 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:49:26.388 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:49:26.389 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:49:26.389 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:49:26.389 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:26.389 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:26.390 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:26.390 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:49:26.390 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:26.390 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:49:26.391 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:26.391 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:26.391 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:26.392 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:26.392 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:49:26.392 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:26.392 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:26.393 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:26.393 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:26.393 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:26.393 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:26.393 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:49:26.394 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:26.394 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:26.394 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:26.394 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:26.395 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:26.395 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:26.395 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:26.395 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:26.395 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:26.396 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:26.396 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:26.396 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:49:26.396 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:26.397 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:26.397 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:26.397 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:26.397 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:26.397 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:26.398 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:26.398 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:26.398 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:26.398 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:26.399 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:26.399 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:49:26.399 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:26.399 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:26.399 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:26.400 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:26.400 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:26.400 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:26.401 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:26.401 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:26.401 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:26.401 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:26.402 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:49:26.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:26.402 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:49:26.403 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:49:26.403 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:49:26.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:26.404 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:49:26Z" 2022-09-12T18:49:26.404 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:49:26.405 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T18:49:26.406 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T18:49:26.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:26.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:26.407 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:26.466 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:26.466 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:26.498 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:49:26.495+0000 7f7738a9e600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:26.499 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:49:26.496+0000 7f7738a9e600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:26.512 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T18:49:26.512 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T18:49:26.512 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T18:49:26.513 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T18:49:26.513 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T18:49:26.513 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:26.513 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:26.514 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:26.514 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T18:49:26.514 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:26.514 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T18:49:26.515 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:26.515 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:26.515 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:26.515 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:26.515 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T18:49:26.516 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:26.516 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:26.516 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:26.516 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:26.517 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:26.517 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:26.517 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T18:49:26.517 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:26.517 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:26.518 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:26.518 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:26.518 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:49:26.518 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:26.519 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T18:49:26.519 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:26.519 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:26.519 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:26.520 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:26.520 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T18:49:26.520 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:26.520 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:26.520 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:26.521 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:26.521 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:49:26.521 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:26.521 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T18:49:26.522 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:26.522 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:26.522 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:26.522 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:26.523 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T18:49:26.523 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:26.523 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:26.523 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:26.523 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:26.524 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:49:26.524 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:26.524 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T18:49:26.524 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:26.525 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:26.525 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:26.525 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T18:49:26.525 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:26.525 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T18:49:26.526 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T18:49:26.526 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T18:49:26.526 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:26.526 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T18:49:26Z" 2022-09-12T18:49:26.527 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T18:49:26.528 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T18:49:26.528 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T18:49:26.529 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:26.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:26.530 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:26.580 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:26.581 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:26.618 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:26.614+0000 7f90acb74600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:26.619 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:26.615+0000 7f90acb74600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:26.634 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:49:26.634 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:49:26.634 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:49:26.634 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:49:26.635 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:49:26.635 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:26.635 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:26.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:26.636 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:49:26.636 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:26.636 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:49:26.636 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:26.637 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:26.637 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:26.637 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:26.637 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:49:26.637 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:26.638 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:26.638 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:26.638 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:26.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:26.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:26.639 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:49:26.639 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:26.639 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:26.640 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:26.640 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:26.640 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:26.640 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:26.641 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:26.641 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:26.641 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:26.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:26.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:26.642 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:49:26.642 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:26.642 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:26.642 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:26.643 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:26.643 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:26.643 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:26.643 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:26.644 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:26.644 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:26.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:26.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:26.645 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:49:26.645 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:26.645 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:26.645 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:26.646 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:26.646 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:26.646 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:26.646 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:26.647 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:26.647 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:26.647 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:26.647 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:49:26.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:26.648 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:49:26.648 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:49:26.648 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:49:26.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:26.649 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:49:26Z" 2022-09-12T18:49:26.649 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:49:26.650 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T18:49:26.651 INFO:rgw_multi.tests:meta checkpoint 2022-09-12T18:49:26.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:26.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:26.652 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:26.704 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:26.704 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:26.732 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=c0f7a512-c940-4d1f-8adf-3484630f876b 2022-09-12T18:49:26.744 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T18:49:26.744 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:26.744 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:26.745 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:47:27.976965Z" 2022-09-12T18:49:26.745 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:26.745 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:26.745 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:26.746 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T18:49:26.746 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:26.746 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:26.747 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:26.747 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T18:49:26.747 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:26.747 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:26.748 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:26.748 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T18:49:26.748 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:26.748 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T18:49:26.749 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T18:49:26.750 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:26.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:26.751 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:26.800 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:26.801 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:26.852 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:26.848+0000 7f970f6b8600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:26.853 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:26.849+0000 7f970f6b8600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:26.866 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:49:26.866 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:49:26.867 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:49:26.867 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:49:26.867 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:49:26.867 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:26.868 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:26.868 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:26.868 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:49:26.868 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:26.868 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:49:26.869 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:26.869 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:26.869 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:26.869 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:26.870 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:49:26.870 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:26.870 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:26.871 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:26.871 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:26.872 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:26.872 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:26.873 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:49:26.873 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:26.874 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:26.874 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:26.874 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:26.875 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:26.875 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:26.876 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:26.876 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:26.876 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:26.877 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:26.877 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:26.878 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:49:26.878 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:26.878 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:26.878 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:26.879 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:26.879 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:26.879 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:26.879 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:26.880 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:26.880 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:26.880 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:26.880 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:26.880 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:49:26.881 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:26.881 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:26.882 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:26.882 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:26.882 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:26.883 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:26.883 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:26.883 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:26.883 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:26.884 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:26.884 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:49:26.884 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:26.884 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:49:26.885 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:49:26.885 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:49:26.885 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:26.885 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:49:26Z" 2022-09-12T18:49:26.885 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:49:26.887 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T18:49:26.887 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T18:49:26.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:26.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:26.889 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:26.941 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:26.942 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:26.970 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:49:26.966+0000 7fa2e8c0e600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:26.971 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:49:26.967+0000 7fa2e8c0e600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:26.986 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T18:49:26.987 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T18:49:26.987 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T18:49:26.987 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T18:49:26.987 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T18:49:26.988 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:26.988 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:26.988 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:26.988 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T18:49:26.989 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:26.989 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T18:49:26.989 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:26.990 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:26.990 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:26.990 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:26.990 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T18:49:26.991 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:26.991 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:26.991 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:26.991 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:26.992 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:26.992 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:26.992 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T18:49:26.993 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:26.993 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:26.993 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:26.993 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:26.994 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:49:26.994 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:26.994 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T18:49:26.994 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:26.995 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:26.995 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:26.995 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:26.995 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T18:49:26.996 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:26.996 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:26.996 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:26.996 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:26.997 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:49:26.997 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:26.997 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T18:49:26.997 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:26.997 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:26.998 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:26.998 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:26.998 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T18:49:26.998 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:26.999 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:26.999 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:26.999 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:26.999 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:49:26.999 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:27.000 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T18:49:27.000 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:27.000 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:27.000 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:27.001 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T18:49:27.001 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:27.001 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T18:49:27.001 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T18:49:27.001 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T18:49:27.002 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:27.002 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T18:49:26Z" 2022-09-12T18:49:27.002 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T18:49:27.004 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T18:49:27.004 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T18:49:27.004 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:27.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:27.005 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:27.055 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:27.056 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:27.100 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:27.096+0000 7f181f4f2600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:27.101 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:27.097+0000 7f181f4f2600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:27.123 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:49:27.124 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:49:27.124 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:49:27.125 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:49:27.125 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:49:27.126 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:27.127 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:27.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:27.128 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:49:27.128 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:27.129 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:49:27.129 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:27.130 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:27.130 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:27.131 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:27.131 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:49:27.131 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:27.132 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:27.132 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:27.133 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:27.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:27.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:27.134 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:49:27.134 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:27.134 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:27.135 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:27.135 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:27.135 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:27.135 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:27.136 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:27.136 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:27.136 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:27.137 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:27.137 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:27.137 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:49:27.137 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:27.137 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:27.138 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:27.138 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:27.138 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:27.139 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:27.139 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:27.139 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:27.139 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:27.140 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:27.140 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:27.140 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:49:27.140 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:27.141 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:27.141 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:27.141 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:27.141 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:27.142 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:27.142 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:27.142 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:27.142 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:27.143 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:27.143 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:49:27.143 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:27.143 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:49:27.144 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:49:27.144 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:49:27.144 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:27.144 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:49:27Z" 2022-09-12T18:49:27.145 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:49:27.146 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T18:49:27.147 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/github.com_ceph_ceph-c_10f7d25d6a0f340676cd251eab5a787ba01ace95/qa/../src/test/rgw/rgw_multi' with args=['tests.py', 'tests_ps.py'] 2022-09-12T18:49:27.220 DEBUG:boto:Using access key provided by client. 2022-09-12T18:49:27.225 DEBUG:boto:Using secret key provided by client. 2022-09-12T18:49:27.228 DEBUG:boto:Using access key provided by client. 2022-09-12T18:49:27.229 DEBUG:boto:Using secret key provided by client. 2022-09-12T18:49:27.230 DEBUG:boto:Using access key provided by client. 2022-09-12T18:49:27.230 DEBUG:boto:Using secret key provided by client. 2022-09-12T18:49:27.232 DEBUG:boto:Using access key provided by client. 2022-09-12T18:49:27.232 DEBUG:boto:Using secret key provided by client. 2022-09-12T18:49:27.233 DEBUG:boto:Using access key provided by client. 2022-09-12T18:49:27.234 DEBUG:boto:Using secret key provided by client. 2022-09-12T18:49:27.235 DEBUG:boto:Using access key provided by client. 2022-09-12T18:49:27.235 DEBUG:boto:Using secret key provided by client. 2022-09-12T18:49:27.236 DEBUG:boto:Using access key provided by client. 2022-09-12T18:49:27.237 DEBUG:boto:Using secret key provided by client. 2022-09-12T18:49:27.238 DEBUG:boto:Using access key provided by client. 2022-09-12T18:49:27.238 DEBUG:boto:Using secret key provided by client. 2022-09-12T18:49:27.239 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-1 2022-09-12T18:49:27.240 DEBUG:boto:path=/jdpiak-1/ 2022-09-12T18:49:27.240 DEBUG:boto:auth_path=/jdpiak-1/ 2022-09-12T18:49:27.241 DEBUG:boto:Method: PUT 2022-09-12T18:49:27.242 DEBUG:boto:Path: /jdpiak-1/ 2022-09-12T18:49:27.243 DEBUG:boto:Data: 2022-09-12T18:49:27.244 DEBUG:boto:Headers: {} 2022-09-12T18:49:27.245 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:27.246 DEBUG:boto:Port: 8000 2022-09-12T18:49:27.246 DEBUG:boto:Params: {} 2022-09-12T18:49:27.247 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T18:49:27.247 DEBUG:boto:Token: None 2022-09-12T18:49:27.248 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:49:27 GMT /jdpiak-1/ 2022-09-12T18:49:27.249 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:d2Qeoi8YbwW/Wkk0k2R1IIG1JMA= 2022-09-12T18:49:27.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:d2Qeoi8YbwW/Wkk0k2R1IIG1JMA=', 'Content-Length': '0'} 2022-09-12T18:49:29.957 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b7876434556c9e1e-00631f7f37-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:29.959 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jdpiak-2 2022-09-12T18:49:29.960 DEBUG:boto:path=/jdpiak-2/ 2022-09-12T18:49:29.961 DEBUG:boto:auth_path=/jdpiak-2/ 2022-09-12T18:49:29.962 DEBUG:boto:Method: PUT 2022-09-12T18:49:29.963 DEBUG:boto:Path: /jdpiak-2/ 2022-09-12T18:49:29.964 DEBUG:boto:Data: 2022-09-12T18:49:29.965 DEBUG:boto:Headers: {} 2022-09-12T18:49:29.966 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:29.967 DEBUG:boto:Port: 8000 2022-09-12T18:49:29.967 DEBUG:boto:Params: {} 2022-09-12T18:49:29.968 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T18:49:29.969 DEBUG:boto:Token: None 2022-09-12T18:49:29.970 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:49:29 GMT /jdpiak-2/ 2022-09-12T18:49:29.970 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dh3OiGh76DNY4mT0MBVf0LEeWCw= 2022-09-12T18:49:29.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dh3OiGh76DNY4mT0MBVf0LEeWCw=', 'Content-Length': '0'} 2022-09-12T18:49:32.512 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d6abb26269ed64a8-00631f7f39-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:32.513 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jdpiak-3 2022-09-12T18:49:32.514 DEBUG:boto:path=/jdpiak-3/ 2022-09-12T18:49:32.514 DEBUG:boto:auth_path=/jdpiak-3/ 2022-09-12T18:49:32.515 DEBUG:boto:Method: PUT 2022-09-12T18:49:32.516 DEBUG:boto:Path: /jdpiak-3/ 2022-09-12T18:49:32.517 DEBUG:boto:Data: 2022-09-12T18:49:32.518 DEBUG:boto:Headers: {} 2022-09-12T18:49:32.519 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:32.519 DEBUG:boto:Port: 8001 2022-09-12T18:49:32.520 DEBUG:boto:Params: {} 2022-09-12T18:49:32.520 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T18:49:32.521 DEBUG:boto:Token: None 2022-09-12T18:49:32.522 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:49:32 GMT /jdpiak-3/ 2022-09-12T18:49:32.522 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EFsr+gmT1HYYPlxudFUf/VjQJSQ= 2022-09-12T18:49:32.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EFsr+gmT1HYYPlxudFUf/VjQJSQ=', 'Content-Length': '0'} 2022-09-12T18:49:32.555 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006835617d705319e9-00631f7f3c-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:32.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:32.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:32.558 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:32.612 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:32.613 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:32.647 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=c0f7a512-c940-4d1f-8adf-3484630f876b 2022-09-12T18:49:32.659 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T18:49:32.659 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:32.659 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:32.660 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:47:27.976965Z" 2022-09-12T18:49:32.660 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:32.660 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:32.660 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:32.661 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T18:49:32.661 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:32.661 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:32.662 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:32.662 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T18:49:32.662 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:32.663 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:32.663 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008572.542081_50.1", 2022-09-12T18:49:32.663 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:49:32.542081Z" 2022-09-12T18:49:32.663 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:32.664 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T18:49:32.665 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008572.542081_50.1'} 2022-09-12T18:49:32.665 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T18:49:32.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:32.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:32.667 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:32.722 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:32.722 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:32.765 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:32.761+0000 7ffb80904600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:32.766 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:32.763+0000 7ffb80904600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:32.783 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:49:32.783 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:49:32.783 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:49:32.784 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:49:32.784 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:49:32.784 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:32.784 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:32.785 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:32.785 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:49:32.785 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:32.785 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:49:32.786 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:32.786 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:32.786 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:32.787 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:32.787 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:49:32.787 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:32.787 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:32.788 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:32.788 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:32.788 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:32.788 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:32.789 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:49:32.789 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:32.789 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:32.789 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:32.790 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:32.790 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:32.790 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:32.790 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:32.791 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:32.791 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:32.791 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:32.791 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:32.792 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:49:32.792 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:32.792 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:32.792 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:32.793 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:32.793 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:32.793 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:32.793 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:32.794 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:32.794 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:32.794 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:32.795 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:32.795 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:49:32.795 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:32.795 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:32.795 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008569.937424_38.1", 2022-09-12T18:49:32.796 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:32.796 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:32.796 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:32.796 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:29.937424Z", 2022-09-12T18:49:32.797 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:32.797 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:32.797 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:32.797 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:49:32.798 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:32.798 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:49:32.798 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:49:32.799 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:49:32.799 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:32.799 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:49:32Z" 2022-09-12T18:49:32.799 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:49:32.801 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008569.937424_38.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:29.937424Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:49:32Z" } 2022-09-12T18:49:32.801 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008569.937424_38.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:29.937424Z', 'realm_epoch': 2}}] 2022-09-12T18:49:32.802 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008572.542081_50.1'} 2022-09-12T18:49:32.802 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008569.937424_38.1'} 2022-09-12T18:49:32.803 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1663008572.542081_50.1 target=1_1663008569.937424_38.1 2022-09-12T18:49:37.804 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:37.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:37.806 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:37.863 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:37.863 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:37.901 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:37.898+0000 7f33c1b66600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:37.902 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:37.899+0000 7f33c1b66600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:37.915 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:49:37.916 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:49:37.916 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:49:37.917 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:49:37.917 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:49:37.918 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:37.918 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:37.919 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:37.919 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:49:37.919 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:37.919 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:49:37.920 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:37.920 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:37.920 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:37.920 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:37.921 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:49:37.921 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:37.922 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:37.922 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:37.922 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:37.923 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:37.923 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:37.923 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:49:37.923 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:37.924 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:37.924 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:37.924 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:37.924 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:37.925 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:37.925 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:37.925 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:37.925 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:37.926 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:37.926 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:37.926 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:49:37.927 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:37.927 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:37.927 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:37.927 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:37.927 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:37.928 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:37.928 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:37.928 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:37.929 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:37.929 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:37.929 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:37.929 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:49:37.930 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:37.930 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:37.930 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008572.542081_50.1", 2022-09-12T18:49:37.930 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:37.931 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:37.931 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:37.931 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:32.542081Z", 2022-09-12T18:49:37.932 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:37.932 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:37.933 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:37.933 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:49:37.934 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:37.934 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:49:37.934 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:49:37.935 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:49:37.935 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:37.935 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:49:37Z" 2022-09-12T18:49:37.935 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:49:37.937 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008572.542081_50.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:32.542081Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:49:37Z" } 2022-09-12T18:49:37.938 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008572.542081_50.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:32.542081Z', 'realm_epoch': 2}}] 2022-09-12T18:49:37.938 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008572.542081_50.1'} 2022-09-12T18:49:37.939 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008572.542081_50.1'} 2022-09-12T18:49:37.939 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T18:49:37.940 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T18:49:37.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:37.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:37.942 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:38.000 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:38.000 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:38.044 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:49:38.041+0000 7f63121f3600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:38.045 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:49:38.042+0000 7f63121f3600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:38.058 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T18:49:38.058 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T18:49:38.059 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T18:49:38.059 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T18:49:38.059 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T18:49:38.059 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:38.060 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:38.060 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:38.060 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T18:49:38.061 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:38.061 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T18:49:38.061 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:38.061 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:38.062 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:38.062 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:38.062 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T18:49:38.063 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:38.063 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:38.063 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:38.063 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:38.064 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:38.064 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:38.064 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T18:49:38.065 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:38.065 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:38.065 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:38.065 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:38.066 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:49:38.066 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:38.066 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T18:49:38.066 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:38.067 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:38.067 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:38.067 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:38.067 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T18:49:38.068 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:38.068 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:38.069 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:38.069 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:38.069 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:49:38.070 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:38.070 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T18:49:38.070 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:38.070 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:38.071 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:38.071 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:38.071 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T18:49:38.072 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:38.072 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:38.072 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008572.542081_50.1", 2022-09-12T18:49:38.072 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:38.073 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:49:38.073 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:38.073 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:32.542081Z", 2022-09-12T18:49:38.073 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:38.074 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:38.076 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:38.076 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T18:49:38.077 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:38.077 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T18:49:38.077 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T18:49:38.077 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T18:49:38.078 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:38.078 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T18:49:38Z" 2022-09-12T18:49:38.078 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T18:49:38.080 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008572.542081_50.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:32.542081Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:49:38Z" } 2022-09-12T18:49:38.081 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008572.542081_50.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:32.542081Z', 'realm_epoch': 2}}] 2022-09-12T18:49:38.081 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008572.542081_50.1'} 2022-09-12T18:49:38.082 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008572.542081_50.1'} 2022-09-12T18:49:38.082 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T18:49:38.083 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T18:49:38.083 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:38.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:38.084 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:38.137 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:38.138 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:38.164 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:38.160+0000 7f9daa3ef600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:38.165 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:38.162+0000 7f9daa3ef600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:38.179 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:49:38.179 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:49:38.179 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:49:38.180 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:49:38.180 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:49:38.180 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:38.181 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:38.181 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:38.181 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:49:38.182 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:38.182 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:49:38.182 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:38.182 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:38.183 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:38.183 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:38.183 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:49:38.183 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:38.184 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:38.184 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:38.184 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:38.185 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:38.185 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:38.185 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:49:38.185 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:38.186 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:38.186 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:38.186 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:38.187 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:38.187 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:38.187 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:38.187 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:38.188 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:38.188 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:38.188 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:38.188 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:49:38.189 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:38.189 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:38.189 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:38.190 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:38.190 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:38.190 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:38.190 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:38.191 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:38.191 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:38.191 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:38.192 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:38.192 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:49:38.192 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:38.192 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:38.193 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008572.542081_50.1", 2022-09-12T18:49:38.193 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:38.193 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:38.193 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:38.194 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:32.542081Z", 2022-09-12T18:49:38.194 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:38.194 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:38.195 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:38.195 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:49:38.195 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:38.196 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:49:38.196 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:49:38.196 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:49:38.196 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:38.197 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:49:38Z" 2022-09-12T18:49:38.197 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:49:38.204 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008572.542081_50.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:32.542081Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:49:38Z" } 2022-09-12T18:49:38.205 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008572.542081_50.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:32.542081Z', 'realm_epoch': 2}}] 2022-09-12T18:49:38.206 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008572.542081_50.1'} 2022-09-12T18:49:38.206 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008572.542081_50.1'} 2022-09-12T18:49:38.207 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T18:49:38.207 DEBUG:boto:path=/jdpiak-1/ 2022-09-12T18:49:38.208 DEBUG:boto:auth_path=/jdpiak-1/ 2022-09-12T18:49:38.209 DEBUG:boto:Method: HEAD 2022-09-12T18:49:38.209 DEBUG:boto:Path: /jdpiak-1/ 2022-09-12T18:49:38.210 DEBUG:boto:Data: 2022-09-12T18:49:38.210 DEBUG:boto:Headers: {} 2022-09-12T18:49:38.211 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:38.211 DEBUG:boto:Port: 8000 2022-09-12T18:49:38.212 DEBUG:boto:Params: {} 2022-09-12T18:49:38.212 DEBUG:boto:Token: None 2022-09-12T18:49:38.213 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:38 GMT /jdpiak-1/ 2022-09-12T18:49:38.214 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lPgfjeA8rd0WpTf1LV9iIlU5xLU= 2022-09-12T18:49:38.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lPgfjeA8rd0WpTf1LV9iIlU5xLU=', 'Content-Length': '0'} 2022-09-12T18:49:38.217 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000647627ad0076ae4c-00631f7f42-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:38.218 DEBUG:boto:path=/jdpiak-2/ 2022-09-12T18:49:38.218 DEBUG:boto:auth_path=/jdpiak-2/ 2022-09-12T18:49:38.219 DEBUG:boto:Method: HEAD 2022-09-12T18:49:38.219 DEBUG:boto:Path: /jdpiak-2/ 2022-09-12T18:49:38.220 DEBUG:boto:Data: 2022-09-12T18:49:38.220 DEBUG:boto:Headers: {} 2022-09-12T18:49:38.221 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:38.221 DEBUG:boto:Port: 8000 2022-09-12T18:49:38.222 DEBUG:boto:Params: {} 2022-09-12T18:49:38.222 DEBUG:boto:Token: None 2022-09-12T18:49:38.223 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:38 GMT /jdpiak-2/ 2022-09-12T18:49:38.224 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2AzBvNxzYKbtztMiCSJcSgiDOw8= 2022-09-12T18:49:38.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2AzBvNxzYKbtztMiCSJcSgiDOw8=', 'Content-Length': '0'} 2022-09-12T18:49:38.227 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000096aea72b6e09fb2d-00631f7f42-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:38.227 DEBUG:boto:path=/jdpiak-3/ 2022-09-12T18:49:38.228 DEBUG:boto:auth_path=/jdpiak-3/ 2022-09-12T18:49:38.229 DEBUG:boto:Method: HEAD 2022-09-12T18:49:38.229 DEBUG:boto:Path: /jdpiak-3/ 2022-09-12T18:49:38.230 DEBUG:boto:Data: 2022-09-12T18:49:38.230 DEBUG:boto:Headers: {} 2022-09-12T18:49:38.231 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:38.231 DEBUG:boto:Port: 8000 2022-09-12T18:49:38.232 DEBUG:boto:Params: {} 2022-09-12T18:49:38.232 DEBUG:boto:Token: None 2022-09-12T18:49:38.233 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:38 GMT /jdpiak-3/ 2022-09-12T18:49:38.234 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IryVq1MYny3lJ4c8REyhBLKs/5w= 2022-09-12T18:49:38.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IryVq1MYny3lJ4c8REyhBLKs/5w=', 'Content-Length': '0'} 2022-09-12T18:49:38.237 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c77333739aa9364-00631f7f42-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:38.237 DEBUG:boto:path=/jdpiak-1/ 2022-09-12T18:49:38.238 DEBUG:boto:auth_path=/jdpiak-1/ 2022-09-12T18:49:38.239 DEBUG:boto:Method: HEAD 2022-09-12T18:49:38.239 DEBUG:boto:Path: /jdpiak-1/ 2022-09-12T18:49:38.240 DEBUG:boto:Data: 2022-09-12T18:49:38.240 DEBUG:boto:Headers: {} 2022-09-12T18:49:38.241 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:38.241 DEBUG:boto:Port: 8000 2022-09-12T18:49:38.242 DEBUG:boto:Params: {} 2022-09-12T18:49:38.243 DEBUG:boto:Token: None 2022-09-12T18:49:38.243 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:38 GMT /jdpiak-1/ 2022-09-12T18:49:38.244 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lPgfjeA8rd0WpTf1LV9iIlU5xLU= 2022-09-12T18:49:38.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lPgfjeA8rd0WpTf1LV9iIlU5xLU=', 'Content-Length': '0'} 2022-09-12T18:49:38.247 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c73730f6540565f0-00631f7f42-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:38.248 DEBUG:boto:path=/jdpiak-2/ 2022-09-12T18:49:38.249 DEBUG:boto:auth_path=/jdpiak-2/ 2022-09-12T18:49:38.249 DEBUG:boto:Method: HEAD 2022-09-12T18:49:38.250 DEBUG:boto:Path: /jdpiak-2/ 2022-09-12T18:49:38.251 DEBUG:boto:Data: 2022-09-12T18:49:38.251 DEBUG:boto:Headers: {} 2022-09-12T18:49:38.252 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:38.253 DEBUG:boto:Port: 8000 2022-09-12T18:49:38.254 DEBUG:boto:Params: {} 2022-09-12T18:49:38.258 DEBUG:boto:Token: None 2022-09-12T18:49:38.261 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:38 GMT /jdpiak-2/ 2022-09-12T18:49:38.261 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2AzBvNxzYKbtztMiCSJcSgiDOw8= 2022-09-12T18:49:38.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2AzBvNxzYKbtztMiCSJcSgiDOw8=', 'Content-Length': '0'} 2022-09-12T18:49:38.264 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002fc593d969c5bdab-00631f7f42-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:38.266 DEBUG:boto:path=/jdpiak-3/ 2022-09-12T18:49:38.267 DEBUG:boto:auth_path=/jdpiak-3/ 2022-09-12T18:49:38.268 DEBUG:boto:Method: HEAD 2022-09-12T18:49:38.269 DEBUG:boto:Path: /jdpiak-3/ 2022-09-12T18:49:38.270 DEBUG:boto:Data: 2022-09-12T18:49:38.271 DEBUG:boto:Headers: {} 2022-09-12T18:49:38.272 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:38.273 DEBUG:boto:Port: 8000 2022-09-12T18:49:38.274 DEBUG:boto:Params: {} 2022-09-12T18:49:38.276 DEBUG:boto:Token: None 2022-09-12T18:49:38.278 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:38 GMT /jdpiak-3/ 2022-09-12T18:49:38.280 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IryVq1MYny3lJ4c8REyhBLKs/5w= 2022-09-12T18:49:38.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IryVq1MYny3lJ4c8REyhBLKs/5w=', 'Content-Length': '0'} 2022-09-12T18:49:38.284 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087e8deb4734914a0-00631f7f42-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:38.286 DEBUG:boto:path=/jdpiak-1/ 2022-09-12T18:49:38.287 DEBUG:boto:auth_path=/jdpiak-1/ 2022-09-12T18:49:38.288 DEBUG:boto:Method: HEAD 2022-09-12T18:49:38.288 DEBUG:boto:Path: /jdpiak-1/ 2022-09-12T18:49:38.289 DEBUG:boto:Data: 2022-09-12T18:49:38.290 DEBUG:boto:Headers: {} 2022-09-12T18:49:38.290 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:38.291 DEBUG:boto:Port: 8001 2022-09-12T18:49:38.292 DEBUG:boto:Params: {} 2022-09-12T18:49:38.292 DEBUG:boto:Token: None 2022-09-12T18:49:38.293 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:38 GMT /jdpiak-1/ 2022-09-12T18:49:38.293 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lPgfjeA8rd0WpTf1LV9iIlU5xLU= 2022-09-12T18:49:38.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lPgfjeA8rd0WpTf1LV9iIlU5xLU=', 'Content-Length': '0'} 2022-09-12T18:49:38.297 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9780321c64ebd62-00631f7f42-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:38.297 DEBUG:boto:path=/jdpiak-2/ 2022-09-12T18:49:38.298 DEBUG:boto:auth_path=/jdpiak-2/ 2022-09-12T18:49:38.298 DEBUG:boto:Method: HEAD 2022-09-12T18:49:38.299 DEBUG:boto:Path: /jdpiak-2/ 2022-09-12T18:49:38.299 DEBUG:boto:Data: 2022-09-12T18:49:38.300 DEBUG:boto:Headers: {} 2022-09-12T18:49:38.300 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:38.301 DEBUG:boto:Port: 8001 2022-09-12T18:49:38.301 DEBUG:boto:Params: {} 2022-09-12T18:49:38.302 DEBUG:boto:Token: None 2022-09-12T18:49:38.303 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:38 GMT /jdpiak-2/ 2022-09-12T18:49:38.303 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2AzBvNxzYKbtztMiCSJcSgiDOw8= 2022-09-12T18:49:38.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2AzBvNxzYKbtztMiCSJcSgiDOw8=', 'Content-Length': '0'} 2022-09-12T18:49:38.306 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000262713cf066f5144-00631f7f42-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:38.307 DEBUG:boto:path=/jdpiak-3/ 2022-09-12T18:49:38.308 DEBUG:boto:auth_path=/jdpiak-3/ 2022-09-12T18:49:38.308 DEBUG:boto:Method: HEAD 2022-09-12T18:49:38.309 DEBUG:boto:Path: /jdpiak-3/ 2022-09-12T18:49:38.309 DEBUG:boto:Data: 2022-09-12T18:49:38.310 DEBUG:boto:Headers: {} 2022-09-12T18:49:38.310 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:38.311 DEBUG:boto:Port: 8001 2022-09-12T18:49:38.311 DEBUG:boto:Params: {} 2022-09-12T18:49:38.312 DEBUG:boto:Token: None 2022-09-12T18:49:38.312 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:38 GMT /jdpiak-3/ 2022-09-12T18:49:38.313 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IryVq1MYny3lJ4c8REyhBLKs/5w= 2022-09-12T18:49:38.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IryVq1MYny3lJ4c8REyhBLKs/5w=', 'Content-Length': '0'} 2022-09-12T18:49:38.316 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007077a8633dc31138-00631f7f42-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:38.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2022-09-12T18:49:38.318 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-4 2022-09-12T18:49:38.318 DEBUG:boto:path=/jdpiak-4/ 2022-09-12T18:49:38.319 DEBUG:boto:auth_path=/jdpiak-4/ 2022-09-12T18:49:38.319 DEBUG:boto:Method: PUT 2022-09-12T18:49:38.320 DEBUG:boto:Path: /jdpiak-4/ 2022-09-12T18:49:38.320 DEBUG:boto:Data: 2022-09-12T18:49:38.321 DEBUG:boto:Headers: {} 2022-09-12T18:49:38.321 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:38.321 DEBUG:boto:Port: 8000 2022-09-12T18:49:38.322 DEBUG:boto:Params: {} 2022-09-12T18:49:38.323 DEBUG:boto:Token: None 2022-09-12T18:49:38.323 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:49:38 GMT /jdpiak-4/ 2022-09-12T18:49:38.324 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fX4gsJUoZ7xXHeY3w1vvphB3/kM= 2022-09-12T18:49:38.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fX4gsJUoZ7xXHeY3w1vvphB3/kM=', 'Content-Length': '0'} 2022-09-12T18:49:38.339 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000844d8de06e06f5ff-00631f7f42-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:38.340 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jdpiak-5 2022-09-12T18:49:38.341 DEBUG:boto:path=/jdpiak-5/ 2022-09-12T18:49:38.341 DEBUG:boto:auth_path=/jdpiak-5/ 2022-09-12T18:49:38.342 DEBUG:boto:Method: PUT 2022-09-12T18:49:38.343 DEBUG:boto:Path: /jdpiak-5/ 2022-09-12T18:49:38.344 DEBUG:boto:Data: 2022-09-12T18:49:38.345 DEBUG:boto:Headers: {} 2022-09-12T18:49:38.346 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:38.346 DEBUG:boto:Port: 8000 2022-09-12T18:49:38.347 DEBUG:boto:Params: {} 2022-09-12T18:49:38.347 DEBUG:boto:Token: None 2022-09-12T18:49:38.348 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:49:38 GMT /jdpiak-5/ 2022-09-12T18:49:38.348 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sLq9WUFzNNe4YxyadYX9X5qR5B8= 2022-09-12T18:49:38.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sLq9WUFzNNe4YxyadYX9X5qR5B8=', 'Content-Length': '0'} 2022-09-12T18:49:38.380 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000034e48cc4be1a397f-00631f7f42-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:38.381 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jdpiak-6 2022-09-12T18:49:38.381 DEBUG:boto:path=/jdpiak-6/ 2022-09-12T18:49:38.382 DEBUG:boto:auth_path=/jdpiak-6/ 2022-09-12T18:49:38.382 DEBUG:boto:Method: PUT 2022-09-12T18:49:38.383 DEBUG:boto:Path: /jdpiak-6/ 2022-09-12T18:49:38.383 DEBUG:boto:Data: 2022-09-12T18:49:38.384 DEBUG:boto:Headers: {} 2022-09-12T18:49:38.384 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:38.385 DEBUG:boto:Port: 8001 2022-09-12T18:49:38.386 DEBUG:boto:Params: {} 2022-09-12T18:49:38.386 DEBUG:boto:Token: None 2022-09-12T18:49:38.387 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:49:38 GMT /jdpiak-6/ 2022-09-12T18:49:38.388 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ypiS9VQk7h2zLCNnUG6vdvf4FZQ= 2022-09-12T18:49:38.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ypiS9VQk7h2zLCNnUG6vdvf4FZQ=', 'Content-Length': '0'} 2022-09-12T18:49:38.414 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000ef6a47289727198-00631f7f42-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:38.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:38.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:38.417 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:38.477 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:38.478 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:38.529 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=c0f7a512-c940-4d1f-8adf-3484630f876b 2022-09-12T18:49:38.543 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T18:49:38.544 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:38.544 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:38.544 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:47:27.976965Z" 2022-09-12T18:49:38.545 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:38.545 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:38.545 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:38.545 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T18:49:38.546 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:38.546 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:38.546 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:38.546 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T18:49:38.547 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:38.547 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:38.547 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008578.400100_62.1", 2022-09-12T18:49:38.548 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:49:38.400100Z" 2022-09-12T18:49:38.548 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:38.548 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T18:49:38.549 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008578.400100_62.1'} 2022-09-12T18:49:38.550 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T18:49:38.550 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:38.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:38.552 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:38.601 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:38.601 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:38.654 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:38.650+0000 7f4beb50c600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:38.655 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:38.651+0000 7f4beb50c600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:38.671 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:49:38.671 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:49:38.672 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:49:38.672 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:49:38.673 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:49:38.673 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:38.674 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:38.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:38.674 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:49:38.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:38.675 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:49:38.675 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:38.675 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:38.676 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:38.676 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:38.676 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:49:38.676 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:38.677 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:38.677 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:38.677 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:38.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:38.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:38.678 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:49:38.678 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:38.678 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:38.679 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:38.679 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:38.679 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:38.680 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:38.680 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:38.680 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:38.681 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:38.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:38.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:38.682 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:49:38.683 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:38.683 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:38.683 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:38.683 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:38.684 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:38.684 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:38.684 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:38.684 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:38.685 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:38.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:38.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:38.685 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:49:38.686 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:38.686 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:38.686 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008578.400100_62.1", 2022-09-12T18:49:38.686 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:38.687 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:38.687 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:38.687 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.400100Z", 2022-09-12T18:49:38.687 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:38.688 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:38.688 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:38.689 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:49:38.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:38.690 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:49:38.690 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:49:38.691 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:49:38.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:38.691 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:49:38Z" 2022-09-12T18:49:38.692 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:49:38.693 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008578.400100_62.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:38.400100Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:49:38Z" } 2022-09-12T18:49:38.694 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008578.400100_62.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:38.400100Z', 'realm_epoch': 2}}] 2022-09-12T18:49:38.695 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008578.400100_62.1'} 2022-09-12T18:49:38.695 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008578.400100_62.1'} 2022-09-12T18:49:38.695 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T18:49:38.696 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T18:49:38.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:38.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:38.697 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:38.749 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:38.750 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:38.778 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:49:38.774+0000 7f72453bf600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:38.779 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:49:38.775+0000 7f72453bf600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:38.790 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T18:49:38.791 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T18:49:38.791 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T18:49:38.791 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T18:49:38.792 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T18:49:38.792 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:38.792 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:38.793 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:38.793 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T18:49:38.793 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:38.793 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T18:49:38.794 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:38.794 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:38.794 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:38.794 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:38.795 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T18:49:38.795 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:38.795 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:38.795 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:38.796 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:38.796 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:38.796 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:38.796 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T18:49:38.797 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:38.797 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:38.797 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:38.797 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:38.798 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:49:38.798 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:38.798 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T18:49:38.798 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:38.799 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:38.799 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:38.799 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:38.799 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T18:49:38.800 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:38.800 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:38.800 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:38.800 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:38.801 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:49:38.801 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:38.802 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T18:49:38.802 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:38.802 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:38.802 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:38.803 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:38.803 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T18:49:38.803 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:38.803 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:38.804 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008578.400100_62.1", 2022-09-12T18:49:38.804 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:38.804 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:49:38.805 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:38.805 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.400100Z", 2022-09-12T18:49:38.805 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:38.805 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:38.806 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:38.806 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T18:49:38.806 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:38.806 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T18:49:38.807 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T18:49:38.807 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T18:49:38.807 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:38.808 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T18:49:38Z" 2022-09-12T18:49:38.808 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T18:49:38.811 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008578.400100_62.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:38.400100Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:49:38Z" } 2022-09-12T18:49:38.812 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008578.400100_62.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:38.400100Z', 'realm_epoch': 2}}] 2022-09-12T18:49:38.812 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008578.400100_62.1'} 2022-09-12T18:49:38.813 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008578.400100_62.1'} 2022-09-12T18:49:38.813 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T18:49:38.814 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T18:49:38.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:38.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:38.815 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:38.865 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:38.865 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:38.898 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:38.894+0000 7fdcbf5a4600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:38.899 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:38.895+0000 7fdcbf5a4600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:38.912 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:49:38.912 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:49:38.913 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:49:38.913 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:49:38.913 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:49:38.914 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:38.914 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:38.914 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:38.914 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:49:38.915 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:38.915 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:49:38.915 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:38.915 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:38.916 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:38.916 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:38.916 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:49:38.916 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:38.917 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:38.917 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:38.917 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:38.918 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:38.918 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:38.918 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:49:38.918 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:38.919 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:38.919 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:38.919 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:38.919 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:38.920 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:38.920 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:38.920 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:38.920 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:38.921 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:38.921 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:38.921 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:49:38.921 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:38.922 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:38.922 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:38.922 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:38.922 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:38.923 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:38.923 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:38.923 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:38.923 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:38.924 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:38.924 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:38.924 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:49:38.924 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:38.925 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:38.925 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008578.400100_62.1", 2022-09-12T18:49:38.925 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:38.925 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:38.926 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:38.926 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.400100Z", 2022-09-12T18:49:38.926 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:38.926 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:38.927 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:38.927 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:49:38.927 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:38.927 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:49:38.928 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:49:38.928 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:49:38.928 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:38.928 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:49:38Z" 2022-09-12T18:49:38.929 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:49:38.930 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008578.400100_62.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:38.400100Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:49:38Z" } 2022-09-12T18:49:38.931 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008578.400100_62.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:38.400100Z', 'realm_epoch': 2}}] 2022-09-12T18:49:38.931 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008578.400100_62.1'} 2022-09-12T18:49:38.932 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008578.400100_62.1'} 2022-09-12T18:49:38.932 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T18:49:38.933 DEBUG:boto:path=/jdpiak-4/ 2022-09-12T18:49:38.933 DEBUG:boto:auth_path=/jdpiak-4/ 2022-09-12T18:49:38.934 DEBUG:boto:Method: HEAD 2022-09-12T18:49:38.934 DEBUG:boto:Path: /jdpiak-4/ 2022-09-12T18:49:38.935 DEBUG:boto:Data: 2022-09-12T18:49:38.935 DEBUG:boto:Headers: {} 2022-09-12T18:49:38.936 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:38.936 DEBUG:boto:Port: 8000 2022-09-12T18:49:38.937 DEBUG:boto:Params: {} 2022-09-12T18:49:38.937 DEBUG:boto:Token: None 2022-09-12T18:49:38.938 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:38 GMT /jdpiak-4/ 2022-09-12T18:49:38.938 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Tppk0Jz+o5CHv4dSXff7O9ewIZI= 2022-09-12T18:49:38.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Tppk0Jz+o5CHv4dSXff7O9ewIZI=', 'Content-Length': '0'} 2022-09-12T18:49:38.942 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002d62f3bec155e41-00631f7f42-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:38.943 DEBUG:boto:path=/jdpiak-5/ 2022-09-12T18:49:38.943 DEBUG:boto:auth_path=/jdpiak-5/ 2022-09-12T18:49:38.944 DEBUG:boto:Method: HEAD 2022-09-12T18:49:38.944 DEBUG:boto:Path: /jdpiak-5/ 2022-09-12T18:49:38.945 DEBUG:boto:Data: 2022-09-12T18:49:38.945 DEBUG:boto:Headers: {} 2022-09-12T18:49:38.946 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:38.946 DEBUG:boto:Port: 8000 2022-09-12T18:49:38.947 DEBUG:boto:Params: {} 2022-09-12T18:49:38.947 DEBUG:boto:Token: None 2022-09-12T18:49:38.948 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:38 GMT /jdpiak-5/ 2022-09-12T18:49:38.948 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G66x0TBIYf0L/tNzBcGGi5l6UCc= 2022-09-12T18:49:38.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G66x0TBIYf0L/tNzBcGGi5l6UCc=', 'Content-Length': '0'} 2022-09-12T18:49:38.951 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000a6688a57991834-00631f7f42-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:38.952 DEBUG:boto:path=/jdpiak-6/ 2022-09-12T18:49:38.952 DEBUG:boto:auth_path=/jdpiak-6/ 2022-09-12T18:49:38.953 DEBUG:boto:Method: HEAD 2022-09-12T18:49:38.953 DEBUG:boto:Path: /jdpiak-6/ 2022-09-12T18:49:38.954 DEBUG:boto:Data: 2022-09-12T18:49:38.954 DEBUG:boto:Headers: {} 2022-09-12T18:49:38.955 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:38.955 DEBUG:boto:Port: 8000 2022-09-12T18:49:38.956 DEBUG:boto:Params: {} 2022-09-12T18:49:38.957 DEBUG:boto:Token: None 2022-09-12T18:49:38.957 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:38 GMT /jdpiak-6/ 2022-09-12T18:49:38.958 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DhfzkyC8GGoU4atEAWBaDRK8u30= 2022-09-12T18:49:38.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DhfzkyC8GGoU4atEAWBaDRK8u30=', 'Content-Length': '0'} 2022-09-12T18:49:38.961 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001fd6653e69c8a1df-00631f7f42-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:38.961 DEBUG:boto:path=/jdpiak-4/ 2022-09-12T18:49:38.962 DEBUG:boto:auth_path=/jdpiak-4/ 2022-09-12T18:49:38.963 DEBUG:boto:Method: HEAD 2022-09-12T18:49:38.963 DEBUG:boto:Path: /jdpiak-4/ 2022-09-12T18:49:38.964 DEBUG:boto:Data: 2022-09-12T18:49:38.964 DEBUG:boto:Headers: {} 2022-09-12T18:49:38.965 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:38.965 DEBUG:boto:Port: 8000 2022-09-12T18:49:38.965 DEBUG:boto:Params: {} 2022-09-12T18:49:38.966 DEBUG:boto:Token: None 2022-09-12T18:49:38.967 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:38 GMT /jdpiak-4/ 2022-09-12T18:49:38.967 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Tppk0Jz+o5CHv4dSXff7O9ewIZI= 2022-09-12T18:49:38.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Tppk0Jz+o5CHv4dSXff7O9ewIZI=', 'Content-Length': '0'} 2022-09-12T18:49:38.970 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043e065d072f86bfd-00631f7f42-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:38.971 DEBUG:boto:path=/jdpiak-5/ 2022-09-12T18:49:38.971 DEBUG:boto:auth_path=/jdpiak-5/ 2022-09-12T18:49:38.972 DEBUG:boto:Method: HEAD 2022-09-12T18:49:38.973 DEBUG:boto:Path: /jdpiak-5/ 2022-09-12T18:49:38.974 DEBUG:boto:Data: 2022-09-12T18:49:38.975 DEBUG:boto:Headers: {} 2022-09-12T18:49:38.975 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:38.976 DEBUG:boto:Port: 8000 2022-09-12T18:49:38.977 DEBUG:boto:Params: {} 2022-09-12T18:49:38.978 DEBUG:boto:Token: None 2022-09-12T18:49:38.979 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:38 GMT /jdpiak-5/ 2022-09-12T18:49:38.980 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G66x0TBIYf0L/tNzBcGGi5l6UCc= 2022-09-12T18:49:38.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G66x0TBIYf0L/tNzBcGGi5l6UCc=', 'Content-Length': '0'} 2022-09-12T18:49:38.984 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e98eca5014d12cd4-00631f7f42-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:38.985 DEBUG:boto:path=/jdpiak-6/ 2022-09-12T18:49:38.985 DEBUG:boto:auth_path=/jdpiak-6/ 2022-09-12T18:49:38.986 DEBUG:boto:Method: HEAD 2022-09-12T18:49:38.986 DEBUG:boto:Path: /jdpiak-6/ 2022-09-12T18:49:38.987 DEBUG:boto:Data: 2022-09-12T18:49:38.987 DEBUG:boto:Headers: {} 2022-09-12T18:49:38.988 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:38.988 DEBUG:boto:Port: 8000 2022-09-12T18:49:38.989 DEBUG:boto:Params: {} 2022-09-12T18:49:38.989 DEBUG:boto:Token: None 2022-09-12T18:49:38.990 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:38 GMT /jdpiak-6/ 2022-09-12T18:49:38.991 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DhfzkyC8GGoU4atEAWBaDRK8u30= 2022-09-12T18:49:38.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DhfzkyC8GGoU4atEAWBaDRK8u30=', 'Content-Length': '0'} 2022-09-12T18:49:38.994 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bff7d06c4d4c96f6-00631f7f42-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:38.994 DEBUG:boto:path=/jdpiak-4/ 2022-09-12T18:49:38.995 DEBUG:boto:auth_path=/jdpiak-4/ 2022-09-12T18:49:38.995 DEBUG:boto:Method: HEAD 2022-09-12T18:49:38.996 DEBUG:boto:Path: /jdpiak-4/ 2022-09-12T18:49:38.996 DEBUG:boto:Data: 2022-09-12T18:49:38.997 DEBUG:boto:Headers: {} 2022-09-12T18:49:38.997 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:38.998 DEBUG:boto:Port: 8001 2022-09-12T18:49:38.998 DEBUG:boto:Params: {} 2022-09-12T18:49:38.999 DEBUG:boto:Token: None 2022-09-12T18:49:39.000 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:38 GMT /jdpiak-4/ 2022-09-12T18:49:39.000 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Tppk0Jz+o5CHv4dSXff7O9ewIZI= 2022-09-12T18:49:39.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Tppk0Jz+o5CHv4dSXff7O9ewIZI=', 'Content-Length': '0'} 2022-09-12T18:49:39.003 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095cb413dcf25fb1d-00631f7f42-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.004 DEBUG:boto:path=/jdpiak-5/ 2022-09-12T18:49:39.004 DEBUG:boto:auth_path=/jdpiak-5/ 2022-09-12T18:49:39.005 DEBUG:boto:Method: HEAD 2022-09-12T18:49:39.006 DEBUG:boto:Path: /jdpiak-5/ 2022-09-12T18:49:39.007 DEBUG:boto:Data: 2022-09-12T18:49:39.007 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.008 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:39.008 DEBUG:boto:Port: 8001 2022-09-12T18:49:39.009 DEBUG:boto:Params: {} 2022-09-12T18:49:39.009 DEBUG:boto:Token: None 2022-09-12T18:49:39.010 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-5/ 2022-09-12T18:49:39.011 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X+FAu/rl9xygpGrE8Zh6Vljt/ps= 2022-09-12T18:49:39.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X+FAu/rl9xygpGrE8Zh6Vljt/ps=', 'Content-Length': '0'} 2022-09-12T18:49:39.014 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d56ffcc000cbb4e-00631f7f43-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.014 DEBUG:boto:path=/jdpiak-6/ 2022-09-12T18:49:39.015 DEBUG:boto:auth_path=/jdpiak-6/ 2022-09-12T18:49:39.015 DEBUG:boto:Method: HEAD 2022-09-12T18:49:39.016 DEBUG:boto:Path: /jdpiak-6/ 2022-09-12T18:49:39.016 DEBUG:boto:Data: 2022-09-12T18:49:39.017 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.017 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:39.018 DEBUG:boto:Port: 8001 2022-09-12T18:49:39.018 DEBUG:boto:Params: {} 2022-09-12T18:49:39.019 DEBUG:boto:Token: None 2022-09-12T18:49:39.020 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-6/ 2022-09-12T18:49:39.020 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VIJeE0ytt1qEeTtoi97gVTlRCO8= 2022-09-12T18:49:39.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VIJeE0ytt1qEeTtoi97gVTlRCO8=', 'Content-Length': '0'} 2022-09-12T18:49:39.023 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8d760a584ba614f-00631f7f43-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.024 DEBUG:boto:path=/jdpiak-4/ 2022-09-12T18:49:39.024 DEBUG:boto:auth_path=/jdpiak-4/ 2022-09-12T18:49:39.025 DEBUG:boto:Method: PUT 2022-09-12T18:49:39.025 DEBUG:boto:Path: /jdpiak-4/ 2022-09-12T18:49:39.026 DEBUG:boto:Data: 2022-09-12T18:49:39.026 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.027 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:39.027 DEBUG:boto:Port: 8000 2022-09-12T18:49:39.028 DEBUG:boto:Params: {} 2022-09-12T18:49:39.028 DEBUG:boto:Token: None 2022-09-12T18:49:39.029 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-4/ 2022-09-12T18:49:39.030 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IBvLF5mhDAjIl+15h1NxDilqecc= 2022-09-12T18:49:39.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IBvLF5mhDAjIl+15h1NxDilqecc=', 'Content-Length': '0'} 2022-09-12T18:49:39.033 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b865158e0742af60-00631f7f43-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.034 DEBUG:boto:path=/jdpiak-5/ 2022-09-12T18:49:39.034 DEBUG:boto:auth_path=/jdpiak-5/ 2022-09-12T18:49:39.035 DEBUG:boto:Method: PUT 2022-09-12T18:49:39.035 DEBUG:boto:Path: /jdpiak-5/ 2022-09-12T18:49:39.036 DEBUG:boto:Data: 2022-09-12T18:49:39.036 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.037 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:39.037 DEBUG:boto:Port: 8000 2022-09-12T18:49:39.038 DEBUG:boto:Params: {} 2022-09-12T18:49:39.038 DEBUG:boto:Token: None 2022-09-12T18:49:39.039 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-5/ 2022-09-12T18:49:39.040 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hrhrBZzsVLYfafvzjMA+iIfXvKI= 2022-09-12T18:49:39.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hrhrBZzsVLYfafvzjMA+iIfXvKI=', 'Content-Length': '0'} 2022-09-12T18:49:39.043 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004f777d739dbdb5fe-00631f7f43-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.044 DEBUG:boto:path=/jdpiak-6/ 2022-09-12T18:49:39.044 DEBUG:boto:auth_path=/jdpiak-6/ 2022-09-12T18:49:39.045 DEBUG:boto:Method: PUT 2022-09-12T18:49:39.045 DEBUG:boto:Path: /jdpiak-6/ 2022-09-12T18:49:39.046 DEBUG:boto:Data: 2022-09-12T18:49:39.046 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.047 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:39.047 DEBUG:boto:Port: 8000 2022-09-12T18:49:39.048 DEBUG:boto:Params: {} 2022-09-12T18:49:39.048 DEBUG:boto:Token: None 2022-09-12T18:49:39.049 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-6/ 2022-09-12T18:49:39.049 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exIOVzZxl72izmapxF5QunMMdV4= 2022-09-12T18:49:39.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exIOVzZxl72izmapxF5QunMMdV4=', 'Content-Length': '0'} 2022-09-12T18:49:39.053 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008456806f04bc405f-00631f7f43-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.053 DEBUG:boto:path=/jdpiak-4/ 2022-09-12T18:49:39.054 DEBUG:boto:auth_path=/jdpiak-4/ 2022-09-12T18:49:39.054 DEBUG:boto:Method: PUT 2022-09-12T18:49:39.055 DEBUG:boto:Path: /jdpiak-4/ 2022-09-12T18:49:39.055 DEBUG:boto:Data: 2022-09-12T18:49:39.056 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.057 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:39.057 DEBUG:boto:Port: 8000 2022-09-12T18:49:39.058 DEBUG:boto:Params: {} 2022-09-12T18:49:39.058 DEBUG:boto:Token: None 2022-09-12T18:49:39.059 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-4/ 2022-09-12T18:49:39.060 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IBvLF5mhDAjIl+15h1NxDilqecc= 2022-09-12T18:49:39.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IBvLF5mhDAjIl+15h1NxDilqecc=', 'Content-Length': '0'} 2022-09-12T18:49:39.065 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000424b507233f0e18c-00631f7f43-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.066 DEBUG:boto:path=/jdpiak-5/ 2022-09-12T18:49:39.067 DEBUG:boto:auth_path=/jdpiak-5/ 2022-09-12T18:49:39.067 DEBUG:boto:Method: PUT 2022-09-12T18:49:39.068 DEBUG:boto:Path: /jdpiak-5/ 2022-09-12T18:49:39.068 DEBUG:boto:Data: 2022-09-12T18:49:39.069 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.069 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:39.070 DEBUG:boto:Port: 8000 2022-09-12T18:49:39.070 DEBUG:boto:Params: {} 2022-09-12T18:49:39.071 DEBUG:boto:Token: None 2022-09-12T18:49:39.072 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-5/ 2022-09-12T18:49:39.072 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hrhrBZzsVLYfafvzjMA+iIfXvKI= 2022-09-12T18:49:39.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hrhrBZzsVLYfafvzjMA+iIfXvKI=', 'Content-Length': '0'} 2022-09-12T18:49:39.119 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001ed8b80250ced983-00631f7f43-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.120 DEBUG:boto:path=/jdpiak-6/ 2022-09-12T18:49:39.121 DEBUG:boto:auth_path=/jdpiak-6/ 2022-09-12T18:49:39.122 DEBUG:boto:Method: PUT 2022-09-12T18:49:39.122 DEBUG:boto:Path: /jdpiak-6/ 2022-09-12T18:49:39.123 DEBUG:boto:Data: 2022-09-12T18:49:39.123 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.124 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:39.124 DEBUG:boto:Port: 8000 2022-09-12T18:49:39.125 DEBUG:boto:Params: {} 2022-09-12T18:49:39.125 DEBUG:boto:Token: None 2022-09-12T18:49:39.126 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-6/ 2022-09-12T18:49:39.126 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exIOVzZxl72izmapxF5QunMMdV4= 2022-09-12T18:49:39.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exIOVzZxl72izmapxF5QunMMdV4=', 'Content-Length': '0'} 2022-09-12T18:49:39.173 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008ed8a0cda4ea01f7-00631f7f43-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.174 DEBUG:boto:path=/jdpiak-4/ 2022-09-12T18:49:39.175 DEBUG:boto:auth_path=/jdpiak-4/ 2022-09-12T18:49:39.175 DEBUG:boto:Method: PUT 2022-09-12T18:49:39.176 DEBUG:boto:Path: /jdpiak-4/ 2022-09-12T18:49:39.176 DEBUG:boto:Data: 2022-09-12T18:49:39.177 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.177 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:39.178 DEBUG:boto:Port: 8001 2022-09-12T18:49:39.178 DEBUG:boto:Params: {} 2022-09-12T18:49:39.179 DEBUG:boto:Token: None 2022-09-12T18:49:39.179 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-4/ 2022-09-12T18:49:39.180 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IBvLF5mhDAjIl+15h1NxDilqecc= 2022-09-12T18:49:39.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IBvLF5mhDAjIl+15h1NxDilqecc=', 'Content-Length': '0'} 2022-09-12T18:49:39.186 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000021deedbede19da9a-00631f7f43-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.186 DEBUG:boto:path=/jdpiak-5/ 2022-09-12T18:49:39.187 DEBUG:boto:auth_path=/jdpiak-5/ 2022-09-12T18:49:39.187 DEBUG:boto:Method: PUT 2022-09-12T18:49:39.188 DEBUG:boto:Path: /jdpiak-5/ 2022-09-12T18:49:39.188 DEBUG:boto:Data: 2022-09-12T18:49:39.189 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.189 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:39.190 DEBUG:boto:Port: 8001 2022-09-12T18:49:39.190 DEBUG:boto:Params: {} 2022-09-12T18:49:39.191 DEBUG:boto:Token: None 2022-09-12T18:49:39.191 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-5/ 2022-09-12T18:49:39.192 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hrhrBZzsVLYfafvzjMA+iIfXvKI= 2022-09-12T18:49:39.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hrhrBZzsVLYfafvzjMA+iIfXvKI=', 'Content-Length': '0'} 2022-09-12T18:49:39.238 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008ed6b33e27cae7fa-00631f7f43-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.239 DEBUG:boto:path=/jdpiak-6/ 2022-09-12T18:49:39.240 DEBUG:boto:auth_path=/jdpiak-6/ 2022-09-12T18:49:39.240 DEBUG:boto:Method: PUT 2022-09-12T18:49:39.241 DEBUG:boto:Path: /jdpiak-6/ 2022-09-12T18:49:39.241 DEBUG:boto:Data: 2022-09-12T18:49:39.242 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.242 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:39.243 DEBUG:boto:Port: 8001 2022-09-12T18:49:39.244 DEBUG:boto:Params: {} 2022-09-12T18:49:39.245 DEBUG:boto:Token: None 2022-09-12T18:49:39.245 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-6/ 2022-09-12T18:49:39.246 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exIOVzZxl72izmapxF5QunMMdV4= 2022-09-12T18:49:39.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exIOVzZxl72izmapxF5QunMMdV4=', 'Content-Length': '0'} 2022-09-12T18:49:39.293 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e6c9a40ae53ecb0b-00631f7f43-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.295 DEBUG:boto:path=/jdpiak-4/ 2022-09-12T18:49:39.295 DEBUG:boto:auth_path=/jdpiak-4/ 2022-09-12T18:49:39.296 DEBUG:boto:Method: HEAD 2022-09-12T18:49:39.297 DEBUG:boto:Path: /jdpiak-4/ 2022-09-12T18:49:39.297 DEBUG:boto:Data: 2022-09-12T18:49:39.298 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.298 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:39.299 DEBUG:boto:Port: 8000 2022-09-12T18:49:39.300 DEBUG:boto:Params: {} 2022-09-12T18:49:39.300 DEBUG:boto:Token: None 2022-09-12T18:49:39.301 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-4/ 2022-09-12T18:49:39.302 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WHd0Jwp/IHti6tA5LCuLMUvuxQs= 2022-09-12T18:49:39.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WHd0Jwp/IHti6tA5LCuLMUvuxQs=', 'Content-Length': '0'} 2022-09-12T18:49:39.305 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000707a44d99494eefc-00631f7f43-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.305 DEBUG:boto:path=/jdpiak-5/ 2022-09-12T18:49:39.306 DEBUG:boto:auth_path=/jdpiak-5/ 2022-09-12T18:49:39.306 DEBUG:boto:Method: HEAD 2022-09-12T18:49:39.307 DEBUG:boto:Path: /jdpiak-5/ 2022-09-12T18:49:39.308 DEBUG:boto:Data: 2022-09-12T18:49:39.308 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.309 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:39.309 DEBUG:boto:Port: 8000 2022-09-12T18:49:39.310 DEBUG:boto:Params: {} 2022-09-12T18:49:39.310 DEBUG:boto:Token: None 2022-09-12T18:49:39.311 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-5/ 2022-09-12T18:49:39.312 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X+FAu/rl9xygpGrE8Zh6Vljt/ps= 2022-09-12T18:49:39.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X+FAu/rl9xygpGrE8Zh6Vljt/ps=', 'Content-Length': '0'} 2022-09-12T18:49:39.314 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f265ccca2941bdc-00631f7f43-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.315 DEBUG:boto:path=/jdpiak-6/ 2022-09-12T18:49:39.316 DEBUG:boto:auth_path=/jdpiak-6/ 2022-09-12T18:49:39.316 DEBUG:boto:Method: HEAD 2022-09-12T18:49:39.317 DEBUG:boto:Path: /jdpiak-6/ 2022-09-12T18:49:39.317 DEBUG:boto:Data: 2022-09-12T18:49:39.318 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.318 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:39.319 DEBUG:boto:Port: 8000 2022-09-12T18:49:39.319 DEBUG:boto:Params: {} 2022-09-12T18:49:39.320 DEBUG:boto:Token: None 2022-09-12T18:49:39.321 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-6/ 2022-09-12T18:49:39.321 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VIJeE0ytt1qEeTtoi97gVTlRCO8= 2022-09-12T18:49:39.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VIJeE0ytt1qEeTtoi97gVTlRCO8=', 'Content-Length': '0'} 2022-09-12T18:49:39.325 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb1d33ec2806270c-00631f7f43-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.325 DEBUG:boto:path=/jdpiak-4/ 2022-09-12T18:49:39.326 DEBUG:boto:auth_path=/jdpiak-4/ 2022-09-12T18:49:39.326 DEBUG:boto:Method: HEAD 2022-09-12T18:49:39.327 DEBUG:boto:Path: /jdpiak-4/ 2022-09-12T18:49:39.327 DEBUG:boto:Data: 2022-09-12T18:49:39.328 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.328 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:39.329 DEBUG:boto:Port: 8000 2022-09-12T18:49:39.329 DEBUG:boto:Params: {} 2022-09-12T18:49:39.330 DEBUG:boto:Token: None 2022-09-12T18:49:39.331 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-4/ 2022-09-12T18:49:39.331 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WHd0Jwp/IHti6tA5LCuLMUvuxQs= 2022-09-12T18:49:39.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WHd0Jwp/IHti6tA5LCuLMUvuxQs=', 'Content-Length': '0'} 2022-09-12T18:49:39.334 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f183174164cf074c-00631f7f43-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.335 DEBUG:boto:path=/jdpiak-5/ 2022-09-12T18:49:39.335 DEBUG:boto:auth_path=/jdpiak-5/ 2022-09-12T18:49:39.336 DEBUG:boto:Method: HEAD 2022-09-12T18:49:39.336 DEBUG:boto:Path: /jdpiak-5/ 2022-09-12T18:49:39.337 DEBUG:boto:Data: 2022-09-12T18:49:39.337 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.338 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:39.338 DEBUG:boto:Port: 8000 2022-09-12T18:49:39.339 DEBUG:boto:Params: {} 2022-09-12T18:49:39.340 DEBUG:boto:Token: None 2022-09-12T18:49:39.340 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-5/ 2022-09-12T18:49:39.341 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X+FAu/rl9xygpGrE8Zh6Vljt/ps= 2022-09-12T18:49:39.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X+FAu/rl9xygpGrE8Zh6Vljt/ps=', 'Content-Length': '0'} 2022-09-12T18:49:39.344 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000785354376c308235-00631f7f43-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.344 DEBUG:boto:path=/jdpiak-6/ 2022-09-12T18:49:39.345 DEBUG:boto:auth_path=/jdpiak-6/ 2022-09-12T18:49:39.345 DEBUG:boto:Method: HEAD 2022-09-12T18:49:39.346 DEBUG:boto:Path: /jdpiak-6/ 2022-09-12T18:49:39.346 DEBUG:boto:Data: 2022-09-12T18:49:39.347 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.347 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:39.348 DEBUG:boto:Port: 8000 2022-09-12T18:49:39.348 DEBUG:boto:Params: {} 2022-09-12T18:49:39.349 DEBUG:boto:Token: None 2022-09-12T18:49:39.349 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-6/ 2022-09-12T18:49:39.350 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VIJeE0ytt1qEeTtoi97gVTlRCO8= 2022-09-12T18:49:39.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VIJeE0ytt1qEeTtoi97gVTlRCO8=', 'Content-Length': '0'} 2022-09-12T18:49:39.353 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a43489ad59a608a0-00631f7f43-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.354 DEBUG:boto:path=/jdpiak-4/ 2022-09-12T18:49:39.354 DEBUG:boto:auth_path=/jdpiak-4/ 2022-09-12T18:49:39.355 DEBUG:boto:Method: HEAD 2022-09-12T18:49:39.355 DEBUG:boto:Path: /jdpiak-4/ 2022-09-12T18:49:39.356 DEBUG:boto:Data: 2022-09-12T18:49:39.356 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.357 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:39.357 DEBUG:boto:Port: 8001 2022-09-12T18:49:39.358 DEBUG:boto:Params: {} 2022-09-12T18:49:39.358 DEBUG:boto:Token: None 2022-09-12T18:49:39.359 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-4/ 2022-09-12T18:49:39.360 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WHd0Jwp/IHti6tA5LCuLMUvuxQs= 2022-09-12T18:49:39.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WHd0Jwp/IHti6tA5LCuLMUvuxQs=', 'Content-Length': '0'} 2022-09-12T18:49:39.362 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004588be8f130b9041-00631f7f43-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.363 DEBUG:boto:path=/jdpiak-5/ 2022-09-12T18:49:39.364 DEBUG:boto:auth_path=/jdpiak-5/ 2022-09-12T18:49:39.364 DEBUG:boto:Method: HEAD 2022-09-12T18:49:39.365 DEBUG:boto:Path: /jdpiak-5/ 2022-09-12T18:49:39.365 DEBUG:boto:Data: 2022-09-12T18:49:39.366 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.366 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:39.367 DEBUG:boto:Port: 8001 2022-09-12T18:49:39.367 DEBUG:boto:Params: {} 2022-09-12T18:49:39.368 DEBUG:boto:Token: None 2022-09-12T18:49:39.368 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-5/ 2022-09-12T18:49:39.369 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X+FAu/rl9xygpGrE8Zh6Vljt/ps= 2022-09-12T18:49:39.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X+FAu/rl9xygpGrE8Zh6Vljt/ps=', 'Content-Length': '0'} 2022-09-12T18:49:39.372 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e8fc23bc5f329ee-00631f7f43-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.373 DEBUG:boto:path=/jdpiak-6/ 2022-09-12T18:49:39.373 DEBUG:boto:auth_path=/jdpiak-6/ 2022-09-12T18:49:39.374 DEBUG:boto:Method: HEAD 2022-09-12T18:49:39.374 DEBUG:boto:Path: /jdpiak-6/ 2022-09-12T18:49:39.375 DEBUG:boto:Data: 2022-09-12T18:49:39.375 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.376 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:39.376 DEBUG:boto:Port: 8001 2022-09-12T18:49:39.377 DEBUG:boto:Params: {} 2022-09-12T18:49:39.377 DEBUG:boto:Token: None 2022-09-12T18:49:39.378 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-6/ 2022-09-12T18:49:39.378 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VIJeE0ytt1qEeTtoi97gVTlRCO8= 2022-09-12T18:49:39.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VIJeE0ytt1qEeTtoi97gVTlRCO8=', 'Content-Length': '0'} 2022-09-12T18:49:39.381 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4b8f561a30b3d5e-00631f7f43-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:39.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:39.383 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:39.434 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:39.435 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:39.474 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=c0f7a512-c940-4d1f-8adf-3484630f876b 2022-09-12T18:49:39.490 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T18:49:39.491 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:39.491 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:39.491 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:47:27.976965Z" 2022-09-12T18:49:39.492 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:39.492 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:39.492 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:39.492 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T18:49:39.493 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:39.493 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:39.493 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:39.493 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T18:49:39.494 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:39.494 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:39.494 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008578.400100_62.1", 2022-09-12T18:49:39.494 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:49:38.400100Z" 2022-09-12T18:49:39.495 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:39.495 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T18:49:39.496 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008578.400100_62.1'} 2022-09-12T18:49:39.497 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T18:49:39.498 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:39.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:39.499 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:39.550 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:39.550 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:39.600 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:39.596+0000 7fa7589f3600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:39.601 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:39.597+0000 7fa7589f3600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:39.623 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:49:39.624 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:49:39.624 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:49:39.624 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:49:39.625 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:49:39.625 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:39.626 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:39.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:39.626 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:49:39.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:39.627 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:49:39.627 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:39.628 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:39.628 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:39.629 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:39.629 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:49:39.629 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:39.630 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:39.630 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:39.630 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:39.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:39.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:39.631 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:49:39.632 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:39.632 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:39.632 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:39.633 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:39.633 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:39.633 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:39.634 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:39.634 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:39.635 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:39.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:39.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:39.636 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:49:39.636 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:39.637 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:39.637 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:39.637 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:39.638 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:39.638 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:39.639 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:39.640 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:39.640 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:39.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:39.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:39.641 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:49:39.642 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:39.642 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:39.643 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008578.400100_62.1", 2022-09-12T18:49:39.643 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:39.644 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:39.644 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:39.645 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.400100Z", 2022-09-12T18:49:39.645 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:39.646 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:39.646 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:39.647 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:49:39.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:39.648 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:49:39.648 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:49:39.649 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:49:39.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:39.650 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:49:39Z" 2022-09-12T18:49:39.650 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:49:39.652 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008578.400100_62.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:38.400100Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:49:39Z" } 2022-09-12T18:49:39.653 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008578.400100_62.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:38.400100Z', 'realm_epoch': 2}}] 2022-09-12T18:49:39.654 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008578.400100_62.1'} 2022-09-12T18:49:39.655 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008578.400100_62.1'} 2022-09-12T18:49:39.656 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T18:49:39.657 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T18:49:39.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:39.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:39.660 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:39.712 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:39.712 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:39.748 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:49:39.744+0000 7fc1bb827600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:39.749 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:49:39.746+0000 7fc1bb827600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:39.761 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T18:49:39.761 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T18:49:39.762 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T18:49:39.762 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T18:49:39.763 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T18:49:39.763 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:39.764 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:39.764 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:39.765 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T18:49:39.765 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:39.766 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T18:49:39.766 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:39.767 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:39.768 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:39.768 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:39.769 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T18:49:39.769 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:39.770 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:39.770 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:39.771 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:39.771 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:39.772 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:39.772 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T18:49:39.773 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:39.773 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:39.774 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:39.774 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:39.775 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:49:39.775 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:39.776 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T18:49:39.777 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:39.778 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:39.778 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:39.779 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:39.779 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T18:49:39.780 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:39.780 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:39.781 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:39.781 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:39.781 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:49:39.781 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:39.782 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T18:49:39.782 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:39.782 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:39.783 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:39.783 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:39.783 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T18:49:39.783 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:39.784 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:39.784 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008578.400100_62.1", 2022-09-12T18:49:39.784 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:39.784 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:49:39.785 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:39.786 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.400100Z", 2022-09-12T18:49:39.786 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:39.787 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:39.787 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:39.788 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T18:49:39.788 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:39.788 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T18:49:39.789 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T18:49:39.789 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T18:49:39.790 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:39.790 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T18:49:39Z" 2022-09-12T18:49:39.791 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T18:49:39.792 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008578.400100_62.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:38.400100Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:49:39Z" } 2022-09-12T18:49:39.793 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008578.400100_62.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:38.400100Z', 'realm_epoch': 2}}] 2022-09-12T18:49:39.794 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008578.400100_62.1'} 2022-09-12T18:49:39.795 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008578.400100_62.1'} 2022-09-12T18:49:39.796 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T18:49:39.797 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T18:49:39.797 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:39.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:39.799 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:39.849 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:39.849 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:39.893 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:39.889+0000 7efe262af600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:39.893 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:39.890+0000 7efe262af600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:39.909 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:49:39.910 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:49:39.910 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:49:39.911 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:49:39.911 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:49:39.911 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:39.911 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:39.912 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:39.912 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:49:39.913 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:39.913 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:49:39.914 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:39.914 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:39.915 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:39.915 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:39.916 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:49:39.916 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:39.917 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:39.917 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:39.918 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:39.918 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:39.919 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:39.919 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:49:39.920 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:39.920 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:39.921 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:39.921 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:39.922 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:39.922 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:39.923 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:39.923 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:39.924 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:39.924 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:39.925 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:39.925 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:49:39.925 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:39.926 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:39.926 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:39.927 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:39.927 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:39.928 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:39.928 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:39.929 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:39.929 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:39.930 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:39.930 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:39.930 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:49:39.931 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:39.931 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:39.932 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008578.400100_62.1", 2022-09-12T18:49:39.932 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:39.933 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:39.933 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:39.933 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.400100Z", 2022-09-12T18:49:39.934 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:39.934 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:39.934 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:39.934 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:49:39.935 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:39.935 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:49:39.935 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:49:39.935 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:49:39.936 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:39.936 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:49:39Z" 2022-09-12T18:49:39.936 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:49:39.937 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008578.400100_62.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:38.400100Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:49:39Z" } 2022-09-12T18:49:39.938 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008578.400100_62.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:38.400100Z', 'realm_epoch': 2}}] 2022-09-12T18:49:39.939 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008578.400100_62.1'} 2022-09-12T18:49:39.939 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008578.400100_62.1'} 2022-09-12T18:49:39.940 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T18:49:39.940 DEBUG:boto:path=/jdpiak-4/ 2022-09-12T18:49:39.941 DEBUG:boto:auth_path=/jdpiak-4/ 2022-09-12T18:49:39.941 DEBUG:boto:Method: HEAD 2022-09-12T18:49:39.942 DEBUG:boto:Path: /jdpiak-4/ 2022-09-12T18:49:39.942 DEBUG:boto:Data: 2022-09-12T18:49:39.943 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.943 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:39.944 DEBUG:boto:Port: 8000 2022-09-12T18:49:39.944 DEBUG:boto:Params: {} 2022-09-12T18:49:39.945 DEBUG:boto:Token: None 2022-09-12T18:49:39.945 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-4/ 2022-09-12T18:49:39.946 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WHd0Jwp/IHti6tA5LCuLMUvuxQs= 2022-09-12T18:49:39.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WHd0Jwp/IHti6tA5LCuLMUvuxQs=', 'Content-Length': '0'} 2022-09-12T18:49:39.949 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007226ea5d692f54dc-00631f7f43-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.950 DEBUG:boto:path=/jdpiak-5/ 2022-09-12T18:49:39.950 DEBUG:boto:auth_path=/jdpiak-5/ 2022-09-12T18:49:39.951 DEBUG:boto:Method: HEAD 2022-09-12T18:49:39.951 DEBUG:boto:Path: /jdpiak-5/ 2022-09-12T18:49:39.952 DEBUG:boto:Data: 2022-09-12T18:49:39.952 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.953 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:39.953 DEBUG:boto:Port: 8000 2022-09-12T18:49:39.954 DEBUG:boto:Params: {} 2022-09-12T18:49:39.954 DEBUG:boto:Token: None 2022-09-12T18:49:39.955 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-5/ 2022-09-12T18:49:39.956 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X+FAu/rl9xygpGrE8Zh6Vljt/ps= 2022-09-12T18:49:39.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X+FAu/rl9xygpGrE8Zh6Vljt/ps=', 'Content-Length': '0'} 2022-09-12T18:49:39.959 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000024141c98b872f5e2-00631f7f43-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.960 DEBUG:boto:path=/jdpiak-6/ 2022-09-12T18:49:39.960 DEBUG:boto:auth_path=/jdpiak-6/ 2022-09-12T18:49:39.961 DEBUG:boto:Method: HEAD 2022-09-12T18:49:39.962 DEBUG:boto:Path: /jdpiak-6/ 2022-09-12T18:49:39.962 DEBUG:boto:Data: 2022-09-12T18:49:39.963 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.963 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:39.964 DEBUG:boto:Port: 8000 2022-09-12T18:49:39.964 DEBUG:boto:Params: {} 2022-09-12T18:49:39.965 DEBUG:boto:Token: None 2022-09-12T18:49:39.965 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-6/ 2022-09-12T18:49:39.966 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VIJeE0ytt1qEeTtoi97gVTlRCO8= 2022-09-12T18:49:39.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VIJeE0ytt1qEeTtoi97gVTlRCO8=', 'Content-Length': '0'} 2022-09-12T18:49:39.969 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088e6c9a5abe8524e-00631f7f43-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.970 DEBUG:boto:path=/jdpiak-4/ 2022-09-12T18:49:39.971 DEBUG:boto:auth_path=/jdpiak-4/ 2022-09-12T18:49:39.972 DEBUG:boto:Method: HEAD 2022-09-12T18:49:39.973 DEBUG:boto:Path: /jdpiak-4/ 2022-09-12T18:49:39.974 DEBUG:boto:Data: 2022-09-12T18:49:39.975 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.976 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:39.977 DEBUG:boto:Port: 8000 2022-09-12T18:49:39.978 DEBUG:boto:Params: {} 2022-09-12T18:49:39.978 DEBUG:boto:Token: None 2022-09-12T18:49:39.979 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-4/ 2022-09-12T18:49:39.980 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WHd0Jwp/IHti6tA5LCuLMUvuxQs= 2022-09-12T18:49:39.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WHd0Jwp/IHti6tA5LCuLMUvuxQs=', 'Content-Length': '0'} 2022-09-12T18:49:39.984 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea17d7e65e631e62-00631f7f43-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.984 DEBUG:boto:path=/jdpiak-5/ 2022-09-12T18:49:39.985 DEBUG:boto:auth_path=/jdpiak-5/ 2022-09-12T18:49:39.985 DEBUG:boto:Method: HEAD 2022-09-12T18:49:39.986 DEBUG:boto:Path: /jdpiak-5/ 2022-09-12T18:49:39.986 DEBUG:boto:Data: 2022-09-12T18:49:39.987 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.988 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:39.988 DEBUG:boto:Port: 8000 2022-09-12T18:49:39.989 DEBUG:boto:Params: {} 2022-09-12T18:49:39.989 DEBUG:boto:Token: None 2022-09-12T18:49:39.990 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-5/ 2022-09-12T18:49:39.990 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X+FAu/rl9xygpGrE8Zh6Vljt/ps= 2022-09-12T18:49:39.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X+FAu/rl9xygpGrE8Zh6Vljt/ps=', 'Content-Length': '0'} 2022-09-12T18:49:39.993 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b833c161bb04678-00631f7f43-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:39.994 DEBUG:boto:path=/jdpiak-6/ 2022-09-12T18:49:39.994 DEBUG:boto:auth_path=/jdpiak-6/ 2022-09-12T18:49:39.995 DEBUG:boto:Method: HEAD 2022-09-12T18:49:39.996 DEBUG:boto:Path: /jdpiak-6/ 2022-09-12T18:49:39.996 DEBUG:boto:Data: 2022-09-12T18:49:39.997 DEBUG:boto:Headers: {} 2022-09-12T18:49:39.997 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:39.997 DEBUG:boto:Port: 8000 2022-09-12T18:49:39.998 DEBUG:boto:Params: {} 2022-09-12T18:49:39.999 DEBUG:boto:Token: None 2022-09-12T18:49:39.999 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:39 GMT /jdpiak-6/ 2022-09-12T18:49:40.000 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VIJeE0ytt1qEeTtoi97gVTlRCO8= 2022-09-12T18:49:40.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VIJeE0ytt1qEeTtoi97gVTlRCO8=', 'Content-Length': '0'} 2022-09-12T18:49:40.003 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012d6f98ef3d8a54f-00631f7f43-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:40.003 DEBUG:boto:path=/jdpiak-4/ 2022-09-12T18:49:40.004 DEBUG:boto:auth_path=/jdpiak-4/ 2022-09-12T18:49:40.004 DEBUG:boto:Method: HEAD 2022-09-12T18:49:40.005 DEBUG:boto:Path: /jdpiak-4/ 2022-09-12T18:49:40.005 DEBUG:boto:Data: 2022-09-12T18:49:40.006 DEBUG:boto:Headers: {} 2022-09-12T18:49:40.006 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:40.007 DEBUG:boto:Port: 8001 2022-09-12T18:49:40.007 DEBUG:boto:Params: {} 2022-09-12T18:49:40.008 DEBUG:boto:Token: None 2022-09-12T18:49:40.008 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:40 GMT /jdpiak-4/ 2022-09-12T18:49:40.009 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k6ftlE/YFSiTJuK7cZMkNIAt/Ek= 2022-09-12T18:49:40.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k6ftlE/YFSiTJuK7cZMkNIAt/Ek=', 'Content-Length': '0'} 2022-09-12T18:49:40.012 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1dcece8d71b2e84-00631f7f44-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:40.012 DEBUG:boto:path=/jdpiak-5/ 2022-09-12T18:49:40.013 DEBUG:boto:auth_path=/jdpiak-5/ 2022-09-12T18:49:40.013 DEBUG:boto:Method: HEAD 2022-09-12T18:49:40.014 DEBUG:boto:Path: /jdpiak-5/ 2022-09-12T18:49:40.014 DEBUG:boto:Data: 2022-09-12T18:49:40.015 DEBUG:boto:Headers: {} 2022-09-12T18:49:40.015 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:40.016 DEBUG:boto:Port: 8001 2022-09-12T18:49:40.016 DEBUG:boto:Params: {} 2022-09-12T18:49:40.017 DEBUG:boto:Token: None 2022-09-12T18:49:40.017 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:40 GMT /jdpiak-5/ 2022-09-12T18:49:40.018 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:isWGKCp5BRJdOGqQh0Pbtqt2Z3A= 2022-09-12T18:49:40.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:isWGKCp5BRJdOGqQh0Pbtqt2Z3A=', 'Content-Length': '0'} 2022-09-12T18:49:40.021 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000000bed291dceca8c-00631f7f44-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:40.021 DEBUG:boto:path=/jdpiak-6/ 2022-09-12T18:49:40.022 DEBUG:boto:auth_path=/jdpiak-6/ 2022-09-12T18:49:40.023 DEBUG:boto:Method: HEAD 2022-09-12T18:49:40.023 DEBUG:boto:Path: /jdpiak-6/ 2022-09-12T18:49:40.024 DEBUG:boto:Data: 2022-09-12T18:49:40.024 DEBUG:boto:Headers: {} 2022-09-12T18:49:40.024 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:40.025 DEBUG:boto:Port: 8001 2022-09-12T18:49:40.025 DEBUG:boto:Params: {} 2022-09-12T18:49:40.026 DEBUG:boto:Token: None 2022-09-12T18:49:40.027 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:40 GMT /jdpiak-6/ 2022-09-12T18:49:40.027 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ookm6MeCNDAXzy/41kC8tlmGHt0= 2022-09-12T18:49:40.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ookm6MeCNDAXzy/41kC8tlmGHt0=', 'Content-Length': '0'} 2022-09-12T18:49:40.030 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee1057b9ada10e72-00631f7f44-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:40.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2022-09-12T18:49:40.031 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-7 2022-09-12T18:49:40.032 DEBUG:boto:path=/jdpiak-7/ 2022-09-12T18:49:40.033 DEBUG:boto:auth_path=/jdpiak-7/ 2022-09-12T18:49:40.033 DEBUG:boto:Method: PUT 2022-09-12T18:49:40.033 DEBUG:boto:Path: /jdpiak-7/ 2022-09-12T18:49:40.034 DEBUG:boto:Data: 2022-09-12T18:49:40.034 DEBUG:boto:Headers: {} 2022-09-12T18:49:40.035 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:40.035 DEBUG:boto:Port: 8000 2022-09-12T18:49:40.036 DEBUG:boto:Params: {} 2022-09-12T18:49:40.036 DEBUG:boto:Token: None 2022-09-12T18:49:40.037 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:49:40 GMT /jdpiak-7/ 2022-09-12T18:49:40.038 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0764iwkjIeuosLhwOH76E1fx7AE= 2022-09-12T18:49:40.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0764iwkjIeuosLhwOH76E1fx7AE=', 'Content-Length': '0'} 2022-09-12T18:49:40.051 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fa8e376db37d22f0-00631f7f44-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:40.052 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jdpiak-8 2022-09-12T18:49:40.053 DEBUG:boto:path=/jdpiak-8/ 2022-09-12T18:49:40.053 DEBUG:boto:auth_path=/jdpiak-8/ 2022-09-12T18:49:40.054 DEBUG:boto:Method: PUT 2022-09-12T18:49:40.055 DEBUG:boto:Path: /jdpiak-8/ 2022-09-12T18:49:40.056 DEBUG:boto:Data: 2022-09-12T18:49:40.057 DEBUG:boto:Headers: {} 2022-09-12T18:49:40.058 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:40.059 DEBUG:boto:Port: 8000 2022-09-12T18:49:40.060 DEBUG:boto:Params: {} 2022-09-12T18:49:40.061 DEBUG:boto:Token: None 2022-09-12T18:49:40.062 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:49:40 GMT /jdpiak-8/ 2022-09-12T18:49:40.063 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ju/gKhzO/aXpZdEqFikiqu4H71c= 2022-09-12T18:49:40.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ju/gKhzO/aXpZdEqFikiqu4H71c=', 'Content-Length': '0'} 2022-09-12T18:49:40.087 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000444978e9d61dbca2-00631f7f44-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:40.088 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jdpiak-9 2022-09-12T18:49:40.089 DEBUG:boto:path=/jdpiak-9/ 2022-09-12T18:49:40.090 DEBUG:boto:auth_path=/jdpiak-9/ 2022-09-12T18:49:40.091 DEBUG:boto:Method: PUT 2022-09-12T18:49:40.092 DEBUG:boto:Path: /jdpiak-9/ 2022-09-12T18:49:40.093 DEBUG:boto:Data: 2022-09-12T18:49:40.094 DEBUG:boto:Headers: {} 2022-09-12T18:49:40.095 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:40.095 DEBUG:boto:Port: 8001 2022-09-12T18:49:40.096 DEBUG:boto:Params: {} 2022-09-12T18:49:40.097 DEBUG:boto:Token: None 2022-09-12T18:49:40.098 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:49:40 GMT /jdpiak-9/ 2022-09-12T18:49:40.099 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pvyDwgV2qbD9niCtoW8xlm/ejT8= 2022-09-12T18:49:40.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pvyDwgV2qbD9niCtoW8xlm/ejT8=', 'Content-Length': '0'} 2022-09-12T18:49:40.131 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002632819d62ed78db-00631f7f44-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:40.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:40.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:40.133 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:40.184 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:40.185 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:40.222 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=c0f7a512-c940-4d1f-8adf-3484630f876b 2022-09-12T18:49:40.249 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T18:49:40.249 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:40.250 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:40.250 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:47:27.976965Z" 2022-09-12T18:49:40.250 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:40.250 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:40.251 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:40.251 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T18:49:40.251 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:40.252 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:40.252 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:40.252 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T18:49:40.252 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:40.253 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:40.253 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.116857_74.1", 2022-09-12T18:49:40.253 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:49:40.116857Z" 2022-09-12T18:49:40.254 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:40.254 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T18:49:40.256 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008580.116857_74.1'} 2022-09-12T18:49:40.257 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T18:49:40.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:40.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:40.259 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:40.311 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:40.312 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:40.339 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:40.335+0000 7f872ac2f600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:40.340 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:40.336+0000 7f872ac2f600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:40.357 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:49:40.358 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:49:40.358 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:49:40.358 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:49:40.359 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:49:40.359 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:40.359 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:40.360 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:40.360 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:49:40.360 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:40.360 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:49:40.361 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:40.361 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:40.361 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:40.361 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:40.362 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:49:40.362 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:40.362 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:40.362 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:40.363 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:40.363 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:40.363 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:40.363 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:49:40.364 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:40.364 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:40.364 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:40.364 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:40.365 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:40.365 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:40.365 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:40.365 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:40.366 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:40.366 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:40.366 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:40.367 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:49:40.367 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:40.367 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:40.367 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:40.368 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:40.368 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:40.368 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:40.368 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:40.369 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:40.369 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:40.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:40.369 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:40.370 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:49:40.370 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:40.370 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:40.370 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.116857_74.1", 2022-09-12T18:49:40.371 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:40.371 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:40.371 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:40.371 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.116857Z", 2022-09-12T18:49:40.372 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:40.372 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:40.372 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:40.372 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:49:40.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:40.373 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:49:40.373 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:49:40.373 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:49:40.374 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:40.374 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:49:40Z" 2022-09-12T18:49:40.374 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:49:40.376 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.116857_74.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.116857Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:49:40Z" } 2022-09-12T18:49:40.376 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.116857_74.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.116857Z', 'realm_epoch': 2}}] 2022-09-12T18:49:40.377 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008580.116857_74.1'} 2022-09-12T18:49:40.377 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008580.116857_74.1'} 2022-09-12T18:49:40.378 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T18:49:40.378 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T18:49:40.379 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:40.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:40.380 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:40.431 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:40.431 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:40.473 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:49:40.469+0000 7f720bb95600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:40.473 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:49:40.470+0000 7f720bb95600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:40.489 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T18:49:40.489 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T18:49:40.490 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T18:49:40.490 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T18:49:40.491 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T18:49:40.491 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:40.492 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:40.492 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:40.493 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T18:49:40.493 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:40.494 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T18:49:40.494 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:40.495 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:40.495 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:40.496 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:40.497 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T18:49:40.497 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:40.498 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:40.498 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:40.498 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:40.499 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:40.499 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:40.500 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T18:49:40.500 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:40.501 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:40.501 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:40.501 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:40.502 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:49:40.502 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:40.503 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T18:49:40.503 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:40.504 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:40.504 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:40.505 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:40.505 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T18:49:40.506 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:40.506 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:40.507 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:40.507 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:40.508 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:49:40.508 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:40.509 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T18:49:40.509 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:40.510 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:40.510 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:40.511 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:40.511 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T18:49:40.512 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:40.512 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:40.513 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.116857_74.1", 2022-09-12T18:49:40.513 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:40.514 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:49:40.514 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:40.515 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.116857Z", 2022-09-12T18:49:40.515 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:40.515 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:40.516 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:40.516 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T18:49:40.517 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:40.517 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T18:49:40.518 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T18:49:40.518 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T18:49:40.518 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:40.519 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T18:49:40Z" 2022-09-12T18:49:40.519 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T18:49:40.520 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.116857_74.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.116857Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:49:40Z" } 2022-09-12T18:49:40.521 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.116857_74.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.116857Z', 'realm_epoch': 2}}] 2022-09-12T18:49:40.521 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008580.116857_74.1'} 2022-09-12T18:49:40.522 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008580.116857_74.1'} 2022-09-12T18:49:40.522 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T18:49:40.524 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T18:49:40.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:40.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:40.526 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:40.575 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:40.576 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:40.621 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:40.617+0000 7f4e8f863600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:40.622 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:40.618+0000 7f4e8f863600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:40.638 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:49:40.638 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:49:40.638 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:49:40.639 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:49:40.639 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:49:40.639 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:40.639 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:40.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:40.640 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:49:40.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:40.640 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:49:40.641 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:40.641 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:40.641 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:40.642 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:40.642 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:49:40.642 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:40.643 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:40.643 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:40.643 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:40.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:40.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:40.644 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:49:40.644 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:40.644 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:40.644 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:40.645 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:40.645 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:40.645 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:40.646 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:40.646 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:40.646 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:40.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:40.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:40.647 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:49:40.647 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:40.647 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:40.648 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:40.648 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:40.648 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:40.648 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:40.649 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:40.649 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:40.649 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:40.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:40.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:40.650 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:49:40.650 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:40.650 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:40.651 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.116857_74.1", 2022-09-12T18:49:40.651 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:40.651 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:40.651 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:40.652 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.116857Z", 2022-09-12T18:49:40.652 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:40.652 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:40.652 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:40.653 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:49:40.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:40.653 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:49:40.653 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:49:40.654 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:49:40.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:40.654 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:49:40Z" 2022-09-12T18:49:40.654 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:49:40.656 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.116857_74.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.116857Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:49:40Z" } 2022-09-12T18:49:40.656 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.116857_74.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.116857Z', 'realm_epoch': 2}}] 2022-09-12T18:49:40.657 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008580.116857_74.1'} 2022-09-12T18:49:40.657 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008580.116857_74.1'} 2022-09-12T18:49:40.658 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T18:49:40.658 DEBUG:boto:path=/jdpiak-7/ 2022-09-12T18:49:40.659 DEBUG:boto:auth_path=/jdpiak-7/ 2022-09-12T18:49:40.659 DEBUG:boto:Method: HEAD 2022-09-12T18:49:40.660 DEBUG:boto:Path: /jdpiak-7/ 2022-09-12T18:49:40.660 DEBUG:boto:Data: 2022-09-12T18:49:40.661 DEBUG:boto:Headers: {} 2022-09-12T18:49:40.661 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:40.662 DEBUG:boto:Port: 8000 2022-09-12T18:49:40.662 DEBUG:boto:Params: {} 2022-09-12T18:49:40.663 DEBUG:boto:Token: None 2022-09-12T18:49:40.664 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:40 GMT /jdpiak-7/ 2022-09-12T18:49:40.664 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XECctGq+nWLpG4y0Ab5ZARySSa0= 2022-09-12T18:49:40.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XECctGq+nWLpG4y0Ab5ZARySSa0=', 'Content-Length': '0'} 2022-09-12T18:49:40.668 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3c1a5f4ede9d1ed-00631f7f44-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:40.669 DEBUG:boto:path=/jdpiak-8/ 2022-09-12T18:49:40.669 DEBUG:boto:auth_path=/jdpiak-8/ 2022-09-12T18:49:40.670 DEBUG:boto:Method: HEAD 2022-09-12T18:49:40.671 DEBUG:boto:Path: /jdpiak-8/ 2022-09-12T18:49:40.671 DEBUG:boto:Data: 2022-09-12T18:49:40.672 DEBUG:boto:Headers: {} 2022-09-12T18:49:40.672 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:40.673 DEBUG:boto:Port: 8000 2022-09-12T18:49:40.673 DEBUG:boto:Params: {} 2022-09-12T18:49:40.674 DEBUG:boto:Token: None 2022-09-12T18:49:40.674 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:40 GMT /jdpiak-8/ 2022-09-12T18:49:40.675 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/ghfjEhzim6zw9dnE3HAVjmeQX4= 2022-09-12T18:49:40.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/ghfjEhzim6zw9dnE3HAVjmeQX4=', 'Content-Length': '0'} 2022-09-12T18:49:40.678 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e97fdda6ed6648eb-00631f7f44-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:40.679 DEBUG:boto:path=/jdpiak-9/ 2022-09-12T18:49:40.679 DEBUG:boto:auth_path=/jdpiak-9/ 2022-09-12T18:49:40.680 DEBUG:boto:Method: HEAD 2022-09-12T18:49:40.681 DEBUG:boto:Path: /jdpiak-9/ 2022-09-12T18:49:40.681 DEBUG:boto:Data: 2022-09-12T18:49:40.682 DEBUG:boto:Headers: {} 2022-09-12T18:49:40.682 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:40.683 DEBUG:boto:Port: 8000 2022-09-12T18:49:40.683 DEBUG:boto:Params: {} 2022-09-12T18:49:40.684 DEBUG:boto:Token: None 2022-09-12T18:49:40.684 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:40 GMT /jdpiak-9/ 2022-09-12T18:49:40.685 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q+36ffjGKqOZLjVP8dH8/yLro3M= 2022-09-12T18:49:40.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q+36ffjGKqOZLjVP8dH8/yLro3M=', 'Content-Length': '0'} 2022-09-12T18:49:40.688 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6f865927fcb234f-00631f7f44-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:40.689 DEBUG:boto:path=/jdpiak-7/ 2022-09-12T18:49:40.690 DEBUG:boto:auth_path=/jdpiak-7/ 2022-09-12T18:49:40.691 DEBUG:boto:Method: HEAD 2022-09-12T18:49:40.692 DEBUG:boto:Path: /jdpiak-7/ 2022-09-12T18:49:40.693 DEBUG:boto:Data: 2022-09-12T18:49:40.693 DEBUG:boto:Headers: {} 2022-09-12T18:49:40.694 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:40.694 DEBUG:boto:Port: 8000 2022-09-12T18:49:40.695 DEBUG:boto:Params: {} 2022-09-12T18:49:40.695 DEBUG:boto:Token: None 2022-09-12T18:49:40.696 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:40 GMT /jdpiak-7/ 2022-09-12T18:49:40.697 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XECctGq+nWLpG4y0Ab5ZARySSa0= 2022-09-12T18:49:40.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XECctGq+nWLpG4y0Ab5ZARySSa0=', 'Content-Length': '0'} 2022-09-12T18:49:40.700 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbcfa8430d23ba33-00631f7f44-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:40.700 DEBUG:boto:path=/jdpiak-8/ 2022-09-12T18:49:40.701 DEBUG:boto:auth_path=/jdpiak-8/ 2022-09-12T18:49:40.701 DEBUG:boto:Method: HEAD 2022-09-12T18:49:40.702 DEBUG:boto:Path: /jdpiak-8/ 2022-09-12T18:49:40.703 DEBUG:boto:Data: 2022-09-12T18:49:40.704 DEBUG:boto:Headers: {} 2022-09-12T18:49:40.704 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:40.705 DEBUG:boto:Port: 8000 2022-09-12T18:49:40.706 DEBUG:boto:Params: {} 2022-09-12T18:49:40.707 DEBUG:boto:Token: None 2022-09-12T18:49:40.708 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:40 GMT /jdpiak-8/ 2022-09-12T18:49:40.709 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/ghfjEhzim6zw9dnE3HAVjmeQX4= 2022-09-12T18:49:40.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/ghfjEhzim6zw9dnE3HAVjmeQX4=', 'Content-Length': '0'} 2022-09-12T18:49:40.713 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7b882d9064eadf5-00631f7f44-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:40.714 DEBUG:boto:path=/jdpiak-9/ 2022-09-12T18:49:40.715 DEBUG:boto:auth_path=/jdpiak-9/ 2022-09-12T18:49:40.716 DEBUG:boto:Method: HEAD 2022-09-12T18:49:40.717 DEBUG:boto:Path: /jdpiak-9/ 2022-09-12T18:49:40.717 DEBUG:boto:Data: 2022-09-12T18:49:40.718 DEBUG:boto:Headers: {} 2022-09-12T18:49:40.718 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:40.719 DEBUG:boto:Port: 8000 2022-09-12T18:49:40.719 DEBUG:boto:Params: {} 2022-09-12T18:49:40.720 DEBUG:boto:Token: None 2022-09-12T18:49:40.720 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:40 GMT /jdpiak-9/ 2022-09-12T18:49:40.721 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q+36ffjGKqOZLjVP8dH8/yLro3M= 2022-09-12T18:49:40.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q+36ffjGKqOZLjVP8dH8/yLro3M=', 'Content-Length': '0'} 2022-09-12T18:49:40.724 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ba5c5d2b22633f1-00631f7f44-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:40.724 DEBUG:boto:path=/jdpiak-7/ 2022-09-12T18:49:40.725 DEBUG:boto:auth_path=/jdpiak-7/ 2022-09-12T18:49:40.726 DEBUG:boto:Method: HEAD 2022-09-12T18:49:40.726 DEBUG:boto:Path: /jdpiak-7/ 2022-09-12T18:49:40.727 DEBUG:boto:Data: 2022-09-12T18:49:40.727 DEBUG:boto:Headers: {} 2022-09-12T18:49:40.728 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:40.728 DEBUG:boto:Port: 8001 2022-09-12T18:49:40.729 DEBUG:boto:Params: {} 2022-09-12T18:49:40.729 DEBUG:boto:Token: None 2022-09-12T18:49:40.730 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:40 GMT /jdpiak-7/ 2022-09-12T18:49:40.731 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XECctGq+nWLpG4y0Ab5ZARySSa0= 2022-09-12T18:49:40.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XECctGq+nWLpG4y0Ab5ZARySSa0=', 'Content-Length': '0'} 2022-09-12T18:49:40.734 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c11aaa7d76f695b1-00631f7f44-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:40.735 DEBUG:boto:path=/jdpiak-8/ 2022-09-12T18:49:40.736 DEBUG:boto:auth_path=/jdpiak-8/ 2022-09-12T18:49:40.736 DEBUG:boto:Method: HEAD 2022-09-12T18:49:40.737 DEBUG:boto:Path: /jdpiak-8/ 2022-09-12T18:49:40.737 DEBUG:boto:Data: 2022-09-12T18:49:40.738 DEBUG:boto:Headers: {} 2022-09-12T18:49:40.738 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:40.739 DEBUG:boto:Port: 8001 2022-09-12T18:49:40.739 DEBUG:boto:Params: {} 2022-09-12T18:49:40.740 DEBUG:boto:Token: None 2022-09-12T18:49:40.740 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:40 GMT /jdpiak-8/ 2022-09-12T18:49:40.741 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/ghfjEhzim6zw9dnE3HAVjmeQX4= 2022-09-12T18:49:40.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/ghfjEhzim6zw9dnE3HAVjmeQX4=', 'Content-Length': '0'} 2022-09-12T18:49:40.744 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040f79d23fcfe21a4-00631f7f44-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:40.744 DEBUG:boto:path=/jdpiak-9/ 2022-09-12T18:49:40.745 DEBUG:boto:auth_path=/jdpiak-9/ 2022-09-12T18:49:40.745 DEBUG:boto:Method: HEAD 2022-09-12T18:49:40.746 DEBUG:boto:Path: /jdpiak-9/ 2022-09-12T18:49:40.746 DEBUG:boto:Data: 2022-09-12T18:49:40.747 DEBUG:boto:Headers: {} 2022-09-12T18:49:40.747 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:40.748 DEBUG:boto:Port: 8001 2022-09-12T18:49:40.748 DEBUG:boto:Params: {} 2022-09-12T18:49:40.749 DEBUG:boto:Token: None 2022-09-12T18:49:40.750 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:40 GMT /jdpiak-9/ 2022-09-12T18:49:40.750 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q+36ffjGKqOZLjVP8dH8/yLro3M= 2022-09-12T18:49:40.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q+36ffjGKqOZLjVP8dH8/yLro3M=', 'Content-Length': '0'} 2022-09-12T18:49:40.753 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016a2824833a51b53-00631f7f44-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:40.754 DEBUG:boto:path=/jdpiak-7/ 2022-09-12T18:49:40.754 DEBUG:boto:auth_path=/jdpiak-7/ 2022-09-12T18:49:40.755 DEBUG:boto:Method: DELETE 2022-09-12T18:49:40.755 DEBUG:boto:Path: /jdpiak-7/ 2022-09-12T18:49:40.756 DEBUG:boto:Data: 2022-09-12T18:49:40.756 DEBUG:boto:Headers: {} 2022-09-12T18:49:40.757 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:40.757 DEBUG:boto:Port: 8000 2022-09-12T18:49:40.758 DEBUG:boto:Params: {} 2022-09-12T18:49:40.758 DEBUG:boto:Token: None 2022-09-12T18:49:40.759 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 18:49:40 GMT /jdpiak-7/ 2022-09-12T18:49:40.759 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KxTNuyLp/JSAR6vgGE7he3u2zYQ= 2022-09-12T18:49:40.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KxTNuyLp/JSAR6vgGE7he3u2zYQ=', 'Content-Length': '0'} 2022-09-12T18:49:40.793 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d04618d392facdc2-00631f7f44-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 18:49:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:40.794 DEBUG:boto:path=/jdpiak-8/ 2022-09-12T18:49:40.795 DEBUG:boto:auth_path=/jdpiak-8/ 2022-09-12T18:49:40.796 DEBUG:boto:Method: DELETE 2022-09-12T18:49:40.797 DEBUG:boto:Path: /jdpiak-8/ 2022-09-12T18:49:40.798 DEBUG:boto:Data: 2022-09-12T18:49:40.799 DEBUG:boto:Headers: {} 2022-09-12T18:49:40.799 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:40.800 DEBUG:boto:Port: 8000 2022-09-12T18:49:40.801 DEBUG:boto:Params: {} 2022-09-12T18:49:40.801 DEBUG:boto:Token: None 2022-09-12T18:49:40.802 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 18:49:40 GMT /jdpiak-8/ 2022-09-12T18:49:40.802 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4zo+C7kMiNg6+gTY3n4uc5HYL/E= 2022-09-12T18:49:40.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4zo+C7kMiNg6+gTY3n4uc5HYL/E=', 'Content-Length': '0'} 2022-09-12T18:49:40.867 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009751663fe4dcb541-00631f7f44-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 18:49:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:40.868 DEBUG:boto:path=/jdpiak-9/ 2022-09-12T18:49:40.869 DEBUG:boto:auth_path=/jdpiak-9/ 2022-09-12T18:49:40.870 DEBUG:boto:Method: DELETE 2022-09-12T18:49:40.870 DEBUG:boto:Path: /jdpiak-9/ 2022-09-12T18:49:40.871 DEBUG:boto:Data: 2022-09-12T18:49:40.871 DEBUG:boto:Headers: {} 2022-09-12T18:49:40.872 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:40.872 DEBUG:boto:Port: 8001 2022-09-12T18:49:40.873 DEBUG:boto:Params: {} 2022-09-12T18:49:40.873 DEBUG:boto:Token: None 2022-09-12T18:49:40.874 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 18:49:40 GMT /jdpiak-9/ 2022-09-12T18:49:40.875 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2tKDn43HrSyix5dXouUVtziLZnE= 2022-09-12T18:49:40.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2tKDn43HrSyix5dXouUVtziLZnE=', 'Content-Length': '0'} 2022-09-12T18:49:40.931 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000003ecb4ab42e24ef-00631f7f44-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 18:49:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:40.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:40.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:40.933 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:40.982 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:40.983 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:41.018 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=c0f7a512-c940-4d1f-8adf-3484630f876b 2022-09-12T18:49:41.043 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T18:49:41.044 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:41.044 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:41.044 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:47:27.976965Z" 2022-09-12T18:49:41.044 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:41.045 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:41.045 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:41.045 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T18:49:41.045 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:41.046 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:41.046 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:41.046 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T18:49:41.047 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:41.047 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:41.047 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T18:49:41.047 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:49:40.904048Z" 2022-09-12T18:49:41.048 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:41.048 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T18:49:41.049 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:49:41.050 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T18:49:41.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:41.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:41.052 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:41.106 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:41.107 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:41.147 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:41.143+0000 7fbb68a5b600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:41.148 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:41.144+0000 7fbb68a5b600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:41.166 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:49:41.167 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:49:41.167 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:49:41.168 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:49:41.168 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:49:41.169 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:41.169 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:41.169 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:41.170 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:49:41.170 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:41.171 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:49:41.171 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:41.172 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:41.172 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:41.172 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:41.173 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:49:41.173 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:41.174 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:41.174 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:41.174 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:41.175 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:41.175 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:41.176 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:49:41.176 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:41.176 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:41.177 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:41.177 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:41.178 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:41.178 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:41.179 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:41.179 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:41.179 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:41.180 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:41.180 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:41.181 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:49:41.181 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:41.181 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:41.182 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:41.182 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:41.183 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:41.183 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:41.184 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:41.184 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:41.184 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:41.185 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:41.185 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:41.186 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:49:41.186 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:41.186 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:41.187 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.768561_75.1", 2022-09-12T18:49:41.187 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:41.187 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:41.187 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:41.188 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.768561Z", 2022-09-12T18:49:41.188 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:41.188 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:41.188 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:41.189 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:49:41.189 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:41.189 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:49:41.189 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:49:41.190 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:49:41.190 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:41.190 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:49:41Z" 2022-09-12T18:49:41.190 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:49:41.192 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.768561_75.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.768561Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:49:41Z" } 2022-09-12T18:49:41.192 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.768561_75.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.768561Z', 'realm_epoch': 2}}] 2022-09-12T18:49:41.193 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:49:41.193 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008580.768561_75.1'} 2022-09-12T18:49:41.194 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1663008580.904048_86.1 target=1_1663008580.768561_75.1 2022-09-12T18:49:46.195 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:46.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:46.197 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:46.254 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:46.255 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:46.293 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:46.289+0000 7f1d9731a600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:46.294 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:46.291+0000 7f1d9731a600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:46.311 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:49:46.312 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:49:46.312 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:49:46.313 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:49:46.313 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:49:46.313 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:46.313 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:46.314 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:46.314 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:49:46.314 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:46.315 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:49:46.315 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:46.315 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:46.315 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:46.316 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:46.316 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:49:46.316 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:46.317 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:46.317 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:46.317 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:46.318 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:46.318 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:46.318 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:49:46.319 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:46.319 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:46.320 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:46.320 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:46.321 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:46.321 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:46.322 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:46.322 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:46.322 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:46.323 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:46.323 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:46.323 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:49:46.323 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:46.324 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:46.324 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:46.324 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:46.324 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:46.325 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:46.325 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:46.325 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:46.325 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:46.326 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:46.326 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:46.326 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:49:46.326 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:46.327 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:46.327 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T18:49:46.327 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:46.328 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:46.328 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:46.328 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T18:49:46.328 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:46.329 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:46.329 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:46.329 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:49:46.329 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:46.329 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:49:46.330 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:49:46.330 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:49:46.330 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:46.330 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:49:46Z" 2022-09-12T18:49:46.331 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:49:46.332 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:49:46Z" } 2022-09-12T18:49:46.333 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T18:49:46.333 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:49:46.334 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:49:46.334 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T18:49:46.335 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T18:49:46.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:46.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:46.336 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:46.395 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:46.395 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:46.434 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:49:46.430+0000 7f202cb71600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:46.435 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:49:46.432+0000 7f202cb71600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:46.449 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T18:49:46.450 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T18:49:46.450 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T18:49:46.450 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T18:49:46.451 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T18:49:46.451 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:46.451 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:46.452 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:46.452 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T18:49:46.452 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:46.453 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T18:49:46.453 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:46.454 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:46.454 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:46.454 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:46.454 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T18:49:46.455 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:46.455 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:46.455 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:46.455 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:46.456 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:46.456 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:46.457 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T18:49:46.457 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:46.458 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:46.458 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:46.458 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:46.458 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:49:46.459 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:46.459 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T18:49:46.459 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:46.459 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:46.460 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:46.460 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:46.460 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T18:49:46.460 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:46.461 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:46.461 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:46.461 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:46.461 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:49:46.462 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:46.462 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T18:49:46.462 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:46.462 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:46.463 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:46.463 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:46.464 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T18:49:46.464 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:46.465 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:46.465 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T18:49:46.465 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:46.465 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:49:46.466 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:46.466 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T18:49:46.466 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:46.467 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:46.467 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:46.467 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T18:49:46.467 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:46.468 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T18:49:46.468 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T18:49:46.468 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T18:49:46.468 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:46.469 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T18:49:46Z" 2022-09-12T18:49:46.469 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T18:49:46.471 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:49:46Z" } 2022-09-12T18:49:46.472 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T18:49:46.473 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:49:46.473 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:49:46.474 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T18:49:46.474 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T18:49:46.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:46.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:46.476 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:46.527 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:46.527 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:46.574 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:46.571+0000 7fd1803e7600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:46.575 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:46.572+0000 7fd1803e7600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:46.591 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:49:46.592 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:49:46.592 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:49:46.592 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:49:46.593 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:49:46.593 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:46.593 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:46.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:46.594 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:49:46.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:46.595 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:49:46.595 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:46.595 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:46.595 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:46.596 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:46.596 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:49:46.596 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:46.597 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:46.597 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:46.597 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:46.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:46.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:46.598 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:49:46.598 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:46.598 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:46.599 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:46.599 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:46.599 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:46.600 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:46.600 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:46.600 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:46.600 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:46.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:46.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:46.601 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:49:46.601 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:46.602 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:46.602 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:46.602 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:46.602 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:46.603 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:46.603 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:46.603 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:46.604 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:46.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:46.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:46.604 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:49:46.605 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:46.605 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:46.605 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T18:49:46.605 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:46.606 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:46.606 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:46.606 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T18:49:46.607 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:46.607 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:46.607 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:46.607 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:49:46.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:46.608 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:49:46.608 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:49:46.608 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:49:46.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:46.609 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:49:46Z" 2022-09-12T18:49:46.609 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:49:46.610 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:49:46Z" } 2022-09-12T18:49:46.611 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T18:49:46.612 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:49:46.612 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:49:46.613 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T18:49:46.613 DEBUG:boto:path=/jdpiak-7/ 2022-09-12T18:49:46.614 DEBUG:boto:auth_path=/jdpiak-7/ 2022-09-12T18:49:46.614 DEBUG:boto:Method: HEAD 2022-09-12T18:49:46.615 DEBUG:boto:Path: /jdpiak-7/ 2022-09-12T18:49:46.615 DEBUG:boto:Data: 2022-09-12T18:49:46.616 DEBUG:boto:Headers: {} 2022-09-12T18:49:46.617 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:46.617 DEBUG:boto:Port: 8000 2022-09-12T18:49:46.618 DEBUG:boto:Params: {} 2022-09-12T18:49:46.618 DEBUG:boto:Token: None 2022-09-12T18:49:46.619 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:46 GMT /jdpiak-7/ 2022-09-12T18:49:46.620 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YasDpfi31UObDOYr51A+UtAL+90= 2022-09-12T18:49:46.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YasDpfi31UObDOYr51A+UtAL+90=', 'Content-Length': '0'} 2022-09-12T18:49:46.622 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx0000019a657d28906764e-00631f7f4a-10f6-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:46.623 DEBUG:boto:path=/jdpiak-8/ 2022-09-12T18:49:46.623 DEBUG:boto:auth_path=/jdpiak-8/ 2022-09-12T18:49:46.624 DEBUG:boto:Method: HEAD 2022-09-12T18:49:46.625 DEBUG:boto:Path: /jdpiak-8/ 2022-09-12T18:49:46.625 DEBUG:boto:Data: 2022-09-12T18:49:46.626 DEBUG:boto:Headers: {} 2022-09-12T18:49:46.626 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:46.627 DEBUG:boto:Port: 8000 2022-09-12T18:49:46.627 DEBUG:boto:Params: {} 2022-09-12T18:49:46.628 DEBUG:boto:Token: None 2022-09-12T18:49:46.629 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:46 GMT /jdpiak-8/ 2022-09-12T18:49:46.629 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RoT/fFKlf9636dO4MjyPug15jRk= 2022-09-12T18:49:46.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RoT/fFKlf9636dO4MjyPug15jRk=', 'Content-Length': '0'} 2022-09-12T18:49:46.631 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx00000eb2ed62073204a44-00631f7f4a-10f6-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:46.632 DEBUG:boto:path=/jdpiak-9/ 2022-09-12T18:49:46.633 DEBUG:boto:auth_path=/jdpiak-9/ 2022-09-12T18:49:46.633 DEBUG:boto:Method: HEAD 2022-09-12T18:49:46.634 DEBUG:boto:Path: /jdpiak-9/ 2022-09-12T18:49:46.634 DEBUG:boto:Data: 2022-09-12T18:49:46.635 DEBUG:boto:Headers: {} 2022-09-12T18:49:46.635 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:46.636 DEBUG:boto:Port: 8000 2022-09-12T18:49:46.637 DEBUG:boto:Params: {} 2022-09-12T18:49:46.637 DEBUG:boto:Token: None 2022-09-12T18:49:46.638 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:46 GMT /jdpiak-9/ 2022-09-12T18:49:46.638 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7OzRVOOYtg8eqobjmlROC8hb4ho= 2022-09-12T18:49:46.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7OzRVOOYtg8eqobjmlROC8hb4ho=', 'Content-Length': '0'} 2022-09-12T18:49:46.640 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx0000023c6611ddb0ad304-00631f7f4a-10f6-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:46.641 DEBUG:boto:path=/jdpiak-7/ 2022-09-12T18:49:46.641 DEBUG:boto:auth_path=/jdpiak-7/ 2022-09-12T18:49:46.642 DEBUG:boto:Method: HEAD 2022-09-12T18:49:46.642 DEBUG:boto:Path: /jdpiak-7/ 2022-09-12T18:49:46.643 DEBUG:boto:Data: 2022-09-12T18:49:46.643 DEBUG:boto:Headers: {} 2022-09-12T18:49:46.644 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:46.644 DEBUG:boto:Port: 8000 2022-09-12T18:49:46.645 DEBUG:boto:Params: {} 2022-09-12T18:49:46.646 DEBUG:boto:Token: None 2022-09-12T18:49:46.646 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:46 GMT /jdpiak-7/ 2022-09-12T18:49:46.647 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YasDpfi31UObDOYr51A+UtAL+90= 2022-09-12T18:49:46.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YasDpfi31UObDOYr51A+UtAL+90=', 'Content-Length': '0'} 2022-09-12T18:49:46.649 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx00000509bf2619a1d4df6-00631f7f4a-10e4-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:46.649 DEBUG:boto:path=/jdpiak-8/ 2022-09-12T18:49:46.650 DEBUG:boto:auth_path=/jdpiak-8/ 2022-09-12T18:49:46.651 DEBUG:boto:Method: HEAD 2022-09-12T18:49:46.651 DEBUG:boto:Path: /jdpiak-8/ 2022-09-12T18:49:46.652 DEBUG:boto:Data: 2022-09-12T18:49:46.652 DEBUG:boto:Headers: {} 2022-09-12T18:49:46.653 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:46.653 DEBUG:boto:Port: 8000 2022-09-12T18:49:46.654 DEBUG:boto:Params: {} 2022-09-12T18:49:46.654 DEBUG:boto:Token: None 2022-09-12T18:49:46.655 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:46 GMT /jdpiak-8/ 2022-09-12T18:49:46.656 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RoT/fFKlf9636dO4MjyPug15jRk= 2022-09-12T18:49:46.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RoT/fFKlf9636dO4MjyPug15jRk=', 'Content-Length': '0'} 2022-09-12T18:49:46.658 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx00000a6042f1c2153a992-00631f7f4a-10e4-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:46.658 DEBUG:boto:path=/jdpiak-9/ 2022-09-12T18:49:46.659 DEBUG:boto:auth_path=/jdpiak-9/ 2022-09-12T18:49:46.659 DEBUG:boto:Method: HEAD 2022-09-12T18:49:46.660 DEBUG:boto:Path: /jdpiak-9/ 2022-09-12T18:49:46.661 DEBUG:boto:Data: 2022-09-12T18:49:46.661 DEBUG:boto:Headers: {} 2022-09-12T18:49:46.662 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:46.662 DEBUG:boto:Port: 8000 2022-09-12T18:49:46.663 DEBUG:boto:Params: {} 2022-09-12T18:49:46.663 DEBUG:boto:Token: None 2022-09-12T18:49:46.664 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:46 GMT /jdpiak-9/ 2022-09-12T18:49:46.665 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7OzRVOOYtg8eqobjmlROC8hb4ho= 2022-09-12T18:49:46.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7OzRVOOYtg8eqobjmlROC8hb4ho=', 'Content-Length': '0'} 2022-09-12T18:49:46.667 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx0000090654e7b947441da-00631f7f4a-10e4-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:46.668 DEBUG:boto:path=/jdpiak-7/ 2022-09-12T18:49:46.668 DEBUG:boto:auth_path=/jdpiak-7/ 2022-09-12T18:49:46.669 DEBUG:boto:Method: HEAD 2022-09-12T18:49:46.670 DEBUG:boto:Path: /jdpiak-7/ 2022-09-12T18:49:46.670 DEBUG:boto:Data: 2022-09-12T18:49:46.671 DEBUG:boto:Headers: {} 2022-09-12T18:49:46.671 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:46.672 DEBUG:boto:Port: 8001 2022-09-12T18:49:46.672 DEBUG:boto:Params: {} 2022-09-12T18:49:46.673 DEBUG:boto:Token: None 2022-09-12T18:49:46.673 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:46 GMT /jdpiak-7/ 2022-09-12T18:49:46.674 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YasDpfi31UObDOYr51A+UtAL+90= 2022-09-12T18:49:46.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YasDpfi31UObDOYr51A+UtAL+90=', 'Content-Length': '0'} 2022-09-12T18:49:46.676 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx00000060cbc98443093ca-00631f7f4a-10f2-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:46.677 DEBUG:boto:path=/jdpiak-8/ 2022-09-12T18:49:46.678 DEBUG:boto:auth_path=/jdpiak-8/ 2022-09-12T18:49:46.679 DEBUG:boto:Method: HEAD 2022-09-12T18:49:46.680 DEBUG:boto:Path: /jdpiak-8/ 2022-09-12T18:49:46.680 DEBUG:boto:Data: 2022-09-12T18:49:46.681 DEBUG:boto:Headers: {} 2022-09-12T18:49:46.681 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:46.682 DEBUG:boto:Port: 8001 2022-09-12T18:49:46.682 DEBUG:boto:Params: {} 2022-09-12T18:49:46.683 DEBUG:boto:Token: None 2022-09-12T18:49:46.684 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:46 GMT /jdpiak-8/ 2022-09-12T18:49:46.684 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RoT/fFKlf9636dO4MjyPug15jRk= 2022-09-12T18:49:46.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RoT/fFKlf9636dO4MjyPug15jRk=', 'Content-Length': '0'} 2022-09-12T18:49:46.686 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx0000086623035970fa559-00631f7f4a-10f2-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:46.687 DEBUG:boto:path=/jdpiak-9/ 2022-09-12T18:49:46.688 DEBUG:boto:auth_path=/jdpiak-9/ 2022-09-12T18:49:46.688 DEBUG:boto:Method: HEAD 2022-09-12T18:49:46.689 DEBUG:boto:Path: /jdpiak-9/ 2022-09-12T18:49:46.689 DEBUG:boto:Data: 2022-09-12T18:49:46.690 DEBUG:boto:Headers: {} 2022-09-12T18:49:46.690 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:46.691 DEBUG:boto:Port: 8001 2022-09-12T18:49:46.691 DEBUG:boto:Params: {} 2022-09-12T18:49:46.692 DEBUG:boto:Token: None 2022-09-12T18:49:46.692 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:46 GMT /jdpiak-9/ 2022-09-12T18:49:46.693 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7OzRVOOYtg8eqobjmlROC8hb4ho= 2022-09-12T18:49:46.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7OzRVOOYtg8eqobjmlROC8hb4ho=', 'Content-Length': '0'} 2022-09-12T18:49:46.695 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx00000d6f4d64eec73ccd0-00631f7f4a-10f2-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:46.695 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2022-09-12T18:49:46.696 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-10 2022-09-12T18:49:46.697 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:49:46.697 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:49:46.698 DEBUG:boto:Method: PUT 2022-09-12T18:49:46.698 DEBUG:boto:Path: /jdpiak-10/ 2022-09-12T18:49:46.699 DEBUG:boto:Data: 2022-09-12T18:49:46.699 DEBUG:boto:Headers: {} 2022-09-12T18:49:46.700 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:46.700 DEBUG:boto:Port: 8000 2022-09-12T18:49:46.701 DEBUG:boto:Params: {} 2022-09-12T18:49:46.701 DEBUG:boto:Token: None 2022-09-12T18:49:46.702 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:49:46 GMT /jdpiak-10/ 2022-09-12T18:49:46.703 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ni5DxdVRK36xsHslcfGDBiLP5rc= 2022-09-12T18:49:46.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ni5DxdVRK36xsHslcfGDBiLP5rc=', 'Content-Length': '0'} 2022-09-12T18:49:46.724 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000093844bd0d70c4045-00631f7f4a-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:46.724 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jdpiak-11 2022-09-12T18:49:46.725 DEBUG:boto:path=/jdpiak-11/ 2022-09-12T18:49:46.725 DEBUG:boto:auth_path=/jdpiak-11/ 2022-09-12T18:49:46.726 DEBUG:boto:Method: PUT 2022-09-12T18:49:46.726 DEBUG:boto:Path: /jdpiak-11/ 2022-09-12T18:49:46.727 DEBUG:boto:Data: 2022-09-12T18:49:46.727 DEBUG:boto:Headers: {} 2022-09-12T18:49:46.728 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:46.728 DEBUG:boto:Port: 8000 2022-09-12T18:49:46.729 DEBUG:boto:Params: {} 2022-09-12T18:49:46.729 DEBUG:boto:Token: None 2022-09-12T18:49:46.730 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:49:46 GMT /jdpiak-11/ 2022-09-12T18:49:46.730 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KTy0Uc2iN5W1PXWqD/4koHF5b9U= 2022-09-12T18:49:46.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KTy0Uc2iN5W1PXWqD/4koHF5b9U=', 'Content-Length': '0'} 2022-09-12T18:49:46.759 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000318f9dcad14333d0-00631f7f4a-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:46.760 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jdpiak-12 2022-09-12T18:49:46.760 DEBUG:boto:path=/jdpiak-12/ 2022-09-12T18:49:46.761 DEBUG:boto:auth_path=/jdpiak-12/ 2022-09-12T18:49:46.761 DEBUG:boto:Method: PUT 2022-09-12T18:49:46.762 DEBUG:boto:Path: /jdpiak-12/ 2022-09-12T18:49:46.762 DEBUG:boto:Data: 2022-09-12T18:49:46.763 DEBUG:boto:Headers: {} 2022-09-12T18:49:46.763 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:46.764 DEBUG:boto:Port: 8001 2022-09-12T18:49:46.764 DEBUG:boto:Params: {} 2022-09-12T18:49:46.765 DEBUG:boto:Token: None 2022-09-12T18:49:46.765 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:49:46 GMT /jdpiak-12/ 2022-09-12T18:49:46.766 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:djds1s/0XsTsNyOzgHqRWEUP04I= 2022-09-12T18:49:46.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:djds1s/0XsTsNyOzgHqRWEUP04I=', 'Content-Length': '0'} 2022-09-12T18:49:46.826 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e518b1055bbc7792-00631f7f4a-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:46.827 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:49:46.827 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:49:46.828 DEBUG:boto:Method: HEAD 2022-09-12T18:49:46.828 DEBUG:boto:Path: /jdpiak-10/ 2022-09-12T18:49:46.829 DEBUG:boto:Data: 2022-09-12T18:49:46.829 DEBUG:boto:Headers: {} 2022-09-12T18:49:46.829 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:46.830 DEBUG:boto:Port: 8000 2022-09-12T18:49:46.830 DEBUG:boto:Params: {} 2022-09-12T18:49:46.831 DEBUG:boto:Token: None 2022-09-12T18:49:46.832 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:46 GMT /jdpiak-10/ 2022-09-12T18:49:46.832 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uCkMT54DhRU4MV0Id1IbEt90qaY= 2022-09-12T18:49:46.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uCkMT54DhRU4MV0Id1IbEt90qaY=', 'Content-Length': '0'} 2022-09-12T18:49:46.837 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f89ec3e61c2723a3-00631f7f4a-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:46.838 DEBUG:boto:path=/jdpiak-10/myobj 2022-09-12T18:49:46.839 DEBUG:boto:auth_path=/jdpiak-10/myobj 2022-09-12T18:49:46.840 DEBUG:boto:Method: PUT 2022-09-12T18:49:46.840 DEBUG:boto:Path: /jdpiak-10/myobj 2022-09-12T18:49:46.841 DEBUG:boto:Data: 2022-09-12T18:49:46.841 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T18:49:46.842 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:46.842 DEBUG:boto:Port: 8000 2022-09-12T18:49:46.843 DEBUG:boto:Params: {} 2022-09-12T18:49:46.843 DEBUG:boto:Token: None 2022-09-12T18:49:46.844 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 18:49:46 GMT /jdpiak-10/myobj 2022-09-12T18:49:46.844 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:81y6cWa0+rmpDlkqDptex2RhXTA= 2022-09-12T18:49:46.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:49:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:81y6cWa0+rmpDlkqDptex2RhXTA='} 2022-09-12T18:49:46.860 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c25b8d4207c2f2ed-00631f7f4a-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:46.861 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:49:46.862 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:49:46.863 DEBUG:boto:Method: HEAD 2022-09-12T18:49:46.863 DEBUG:boto:Path: /jdpiak-10/ 2022-09-12T18:49:46.864 DEBUG:boto:Data: 2022-09-12T18:49:46.864 DEBUG:boto:Headers: {} 2022-09-12T18:49:46.865 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:46.865 DEBUG:boto:Port: 8000 2022-09-12T18:49:46.866 DEBUG:boto:Params: {} 2022-09-12T18:49:46.866 DEBUG:boto:Token: None 2022-09-12T18:49:46.867 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:46 GMT /jdpiak-10/ 2022-09-12T18:49:46.868 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uCkMT54DhRU4MV0Id1IbEt90qaY= 2022-09-12T18:49:46.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uCkMT54DhRU4MV0Id1IbEt90qaY=', 'Content-Length': '0'} 2022-09-12T18:49:46.871 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fbcb5410d4ced453-00631f7f4a-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:46.871 DEBUG:boto:path=/jdpiak-10/_myobj 2022-09-12T18:49:46.872 DEBUG:boto:auth_path=/jdpiak-10/_myobj 2022-09-12T18:49:46.873 DEBUG:boto:Method: PUT 2022-09-12T18:49:46.873 DEBUG:boto:Path: /jdpiak-10/_myobj 2022-09-12T18:49:46.874 DEBUG:boto:Data: 2022-09-12T18:49:46.874 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T18:49:46.875 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:46.875 DEBUG:boto:Port: 8000 2022-09-12T18:49:46.876 DEBUG:boto:Params: {} 2022-09-12T18:49:46.876 DEBUG:boto:Token: None 2022-09-12T18:49:46.877 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 18:49:46 GMT /jdpiak-10/_myobj 2022-09-12T18:49:46.877 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9iPQaq4ma0lrpqP2Q20ZyKT31Jg= 2022-09-12T18:49:46.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:49:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9iPQaq4ma0lrpqP2Q20ZyKT31Jg='} 2022-09-12T18:49:46.922 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bd78e2d7a108e9fa-00631f7f4a-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:46.923 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:49:46.924 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:49:46.924 DEBUG:boto:Method: HEAD 2022-09-12T18:49:46.925 DEBUG:boto:Path: /jdpiak-10/ 2022-09-12T18:49:46.925 DEBUG:boto:Data: 2022-09-12T18:49:46.926 DEBUG:boto:Headers: {} 2022-09-12T18:49:46.926 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:46.927 DEBUG:boto:Port: 8000 2022-09-12T18:49:46.927 DEBUG:boto:Params: {} 2022-09-12T18:49:46.928 DEBUG:boto:Token: None 2022-09-12T18:49:46.928 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:46 GMT /jdpiak-10/ 2022-09-12T18:49:46.929 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uCkMT54DhRU4MV0Id1IbEt90qaY= 2022-09-12T18:49:46.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uCkMT54DhRU4MV0Id1IbEt90qaY=', 'Content-Length': '0'} 2022-09-12T18:49:46.932 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004edfc16cccaa0802-00631f7f4a-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:46.933 DEBUG:boto:path=/jdpiak-10/%3A 2022-09-12T18:49:46.933 DEBUG:boto:auth_path=/jdpiak-10/%3A 2022-09-12T18:49:46.934 DEBUG:boto:Method: PUT 2022-09-12T18:49:46.934 DEBUG:boto:Path: /jdpiak-10/%3A 2022-09-12T18:49:46.935 DEBUG:boto:Data: 2022-09-12T18:49:46.935 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T18:49:46.936 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:46.936 DEBUG:boto:Port: 8000 2022-09-12T18:49:46.937 DEBUG:boto:Params: {} 2022-09-12T18:49:46.937 DEBUG:boto:Token: None 2022-09-12T18:49:46.938 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 18:49:46 GMT /jdpiak-10/%3A 2022-09-12T18:49:46.939 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qppgQG3s+DM3ThH2eaOYs9s/LPU= 2022-09-12T18:49:46.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:49:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qppgQG3s+DM3ThH2eaOYs9s/LPU='} 2022-09-12T18:49:46.953 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ccd3c0af90eb406-00631f7f4a-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:46.954 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:49:46.955 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:49:46.955 DEBUG:boto:Method: HEAD 2022-09-12T18:49:46.956 DEBUG:boto:Path: /jdpiak-10/ 2022-09-12T18:49:46.956 DEBUG:boto:Data: 2022-09-12T18:49:46.957 DEBUG:boto:Headers: {} 2022-09-12T18:49:46.957 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:46.958 DEBUG:boto:Port: 8000 2022-09-12T18:49:46.958 DEBUG:boto:Params: {} 2022-09-12T18:49:46.959 DEBUG:boto:Token: None 2022-09-12T18:49:46.959 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:46 GMT /jdpiak-10/ 2022-09-12T18:49:46.960 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uCkMT54DhRU4MV0Id1IbEt90qaY= 2022-09-12T18:49:46.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uCkMT54DhRU4MV0Id1IbEt90qaY=', 'Content-Length': '0'} 2022-09-12T18:49:46.963 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006546c890ff19dd0a-00631f7f4a-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:46.964 DEBUG:boto:path=/jdpiak-10/%26 2022-09-12T18:49:46.965 DEBUG:boto:auth_path=/jdpiak-10/%26 2022-09-12T18:49:46.965 DEBUG:boto:Method: PUT 2022-09-12T18:49:46.966 DEBUG:boto:Path: /jdpiak-10/%26 2022-09-12T18:49:46.966 DEBUG:boto:Data: 2022-09-12T18:49:46.967 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T18:49:46.967 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:49:46.968 DEBUG:boto:Port: 8000 2022-09-12T18:49:46.968 DEBUG:boto:Params: {} 2022-09-12T18:49:46.969 DEBUG:boto:Token: None 2022-09-12T18:49:46.969 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 18:49:46 GMT /jdpiak-10/%26 2022-09-12T18:49:46.970 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N6BZ5hk7vNQtTHH4LmCQOvJhDso= 2022-09-12T18:49:46.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:49:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N6BZ5hk7vNQtTHH4LmCQOvJhDso='} 2022-09-12T18:49:47.014 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f9a7e86f167dcf4d-00631f7f4a-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:47.015 DEBUG:boto:path=/jdpiak-11/ 2022-09-12T18:49:47.016 DEBUG:boto:auth_path=/jdpiak-11/ 2022-09-12T18:49:47.016 DEBUG:boto:Method: HEAD 2022-09-12T18:49:47.017 DEBUG:boto:Path: /jdpiak-11/ 2022-09-12T18:49:47.017 DEBUG:boto:Data: 2022-09-12T18:49:47.018 DEBUG:boto:Headers: {} 2022-09-12T18:49:47.019 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:47.019 DEBUG:boto:Port: 8000 2022-09-12T18:49:47.020 DEBUG:boto:Params: {} 2022-09-12T18:49:47.020 DEBUG:boto:Token: None 2022-09-12T18:49:47.021 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:47 GMT /jdpiak-11/ 2022-09-12T18:49:47.022 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7hjeDg41cmJ4iqL5JjjWVR2pwtw= 2022-09-12T18:49:47.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7hjeDg41cmJ4iqL5JjjWVR2pwtw=', 'Content-Length': '0'} 2022-09-12T18:49:47.025 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ac07291b64337e3-00631f7f4b-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:47.026 DEBUG:boto:path=/jdpiak-11/myobj 2022-09-12T18:49:47.027 DEBUG:boto:auth_path=/jdpiak-11/myobj 2022-09-12T18:49:47.027 DEBUG:boto:Method: PUT 2022-09-12T18:49:47.028 DEBUG:boto:Path: /jdpiak-11/myobj 2022-09-12T18:49:47.029 DEBUG:boto:Data: 2022-09-12T18:49:47.030 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T18:49:47.031 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:47.031 DEBUG:boto:Port: 8000 2022-09-12T18:49:47.032 DEBUG:boto:Params: {} 2022-09-12T18:49:47.033 DEBUG:boto:Token: None 2022-09-12T18:49:47.033 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 18:49:47 GMT /jdpiak-11/myobj 2022-09-12T18:49:47.034 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+rR/YGwXL2Tn/akw65vJjbwFKKw= 2022-09-12T18:49:47.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:49:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+rR/YGwXL2Tn/akw65vJjbwFKKw='} 2022-09-12T18:49:47.047 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a6a1126e170055e7-00631f7f4b-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 18:49:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:47.048 DEBUG:boto:path=/jdpiak-11/ 2022-09-12T18:49:47.048 DEBUG:boto:auth_path=/jdpiak-11/ 2022-09-12T18:49:47.049 DEBUG:boto:Method: HEAD 2022-09-12T18:49:47.049 DEBUG:boto:Path: /jdpiak-11/ 2022-09-12T18:49:47.050 DEBUG:boto:Data: 2022-09-12T18:49:47.051 DEBUG:boto:Headers: {} 2022-09-12T18:49:47.051 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:47.052 DEBUG:boto:Port: 8000 2022-09-12T18:49:47.052 DEBUG:boto:Params: {} 2022-09-12T18:49:47.053 DEBUG:boto:Token: None 2022-09-12T18:49:47.053 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:47 GMT /jdpiak-11/ 2022-09-12T18:49:47.055 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7hjeDg41cmJ4iqL5JjjWVR2pwtw= 2022-09-12T18:49:47.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7hjeDg41cmJ4iqL5JjjWVR2pwtw=', 'Content-Length': '0'} 2022-09-12T18:49:47.058 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005debd04c51c2c28c-00631f7f4b-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:47.059 DEBUG:boto:path=/jdpiak-11/_myobj 2022-09-12T18:49:47.059 DEBUG:boto:auth_path=/jdpiak-11/_myobj 2022-09-12T18:49:47.060 DEBUG:boto:Method: PUT 2022-09-12T18:49:47.060 DEBUG:boto:Path: /jdpiak-11/_myobj 2022-09-12T18:49:47.061 DEBUG:boto:Data: 2022-09-12T18:49:47.061 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T18:49:47.062 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:47.063 DEBUG:boto:Port: 8000 2022-09-12T18:49:47.063 DEBUG:boto:Params: {} 2022-09-12T18:49:47.064 DEBUG:boto:Token: None 2022-09-12T18:49:47.064 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 18:49:47 GMT /jdpiak-11/_myobj 2022-09-12T18:49:47.065 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Am2lKwYEQdz41YqFVuQtds7mKOU= 2022-09-12T18:49:47.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:49:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Am2lKwYEQdz41YqFVuQtds7mKOU='} 2022-09-12T18:49:47.083 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008b4e55935a719192-00631f7f4b-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 18:49:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:47.084 DEBUG:boto:path=/jdpiak-11/ 2022-09-12T18:49:47.085 DEBUG:boto:auth_path=/jdpiak-11/ 2022-09-12T18:49:47.085 DEBUG:boto:Method: HEAD 2022-09-12T18:49:47.086 DEBUG:boto:Path: /jdpiak-11/ 2022-09-12T18:49:47.086 DEBUG:boto:Data: 2022-09-12T18:49:47.087 DEBUG:boto:Headers: {} 2022-09-12T18:49:47.088 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:47.088 DEBUG:boto:Port: 8000 2022-09-12T18:49:47.089 DEBUG:boto:Params: {} 2022-09-12T18:49:47.089 DEBUG:boto:Token: None 2022-09-12T18:49:47.090 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:47 GMT /jdpiak-11/ 2022-09-12T18:49:47.091 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7hjeDg41cmJ4iqL5JjjWVR2pwtw= 2022-09-12T18:49:47.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7hjeDg41cmJ4iqL5JjjWVR2pwtw=', 'Content-Length': '0'} 2022-09-12T18:49:47.093 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001dddef21b385e61d-00631f7f4b-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:47.094 DEBUG:boto:path=/jdpiak-11/%3A 2022-09-12T18:49:47.095 DEBUG:boto:auth_path=/jdpiak-11/%3A 2022-09-12T18:49:47.096 DEBUG:boto:Method: PUT 2022-09-12T18:49:47.096 DEBUG:boto:Path: /jdpiak-11/%3A 2022-09-12T18:49:47.097 DEBUG:boto:Data: 2022-09-12T18:49:47.097 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T18:49:47.098 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:47.099 DEBUG:boto:Port: 8000 2022-09-12T18:49:47.099 DEBUG:boto:Params: {} 2022-09-12T18:49:47.100 DEBUG:boto:Token: None 2022-09-12T18:49:47.100 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 18:49:47 GMT /jdpiak-11/%3A 2022-09-12T18:49:47.101 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qm6YKutZH9QGZP0ZJbL8f2y9GpA= 2022-09-12T18:49:47.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:49:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qm6YKutZH9QGZP0ZJbL8f2y9GpA='} 2022-09-12T18:49:47.150 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000085f3046a93841b39-00631f7f4b-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 18:49:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:47.151 DEBUG:boto:path=/jdpiak-11/ 2022-09-12T18:49:47.151 DEBUG:boto:auth_path=/jdpiak-11/ 2022-09-12T18:49:47.152 DEBUG:boto:Method: HEAD 2022-09-12T18:49:47.153 DEBUG:boto:Path: /jdpiak-11/ 2022-09-12T18:49:47.153 DEBUG:boto:Data: 2022-09-12T18:49:47.154 DEBUG:boto:Headers: {} 2022-09-12T18:49:47.154 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:47.155 DEBUG:boto:Port: 8000 2022-09-12T18:49:47.155 DEBUG:boto:Params: {} 2022-09-12T18:49:47.156 DEBUG:boto:Token: None 2022-09-12T18:49:47.158 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:47 GMT /jdpiak-11/ 2022-09-12T18:49:47.158 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7hjeDg41cmJ4iqL5JjjWVR2pwtw= 2022-09-12T18:49:47.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7hjeDg41cmJ4iqL5JjjWVR2pwtw=', 'Content-Length': '0'} 2022-09-12T18:49:47.162 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007523e8bb3c7bffb-00631f7f4b-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:47.163 DEBUG:boto:path=/jdpiak-11/%26 2022-09-12T18:49:47.163 DEBUG:boto:auth_path=/jdpiak-11/%26 2022-09-12T18:49:47.164 DEBUG:boto:Method: PUT 2022-09-12T18:49:47.165 DEBUG:boto:Path: /jdpiak-11/%26 2022-09-12T18:49:47.165 DEBUG:boto:Data: 2022-09-12T18:49:47.166 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T18:49:47.166 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:49:47.167 DEBUG:boto:Port: 8000 2022-09-12T18:49:47.168 DEBUG:boto:Params: {} 2022-09-12T18:49:47.168 DEBUG:boto:Token: None 2022-09-12T18:49:47.169 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 18:49:47 GMT /jdpiak-11/%26 2022-09-12T18:49:47.170 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:j8MhKs2EfGRDZq3njDuYYgLtAwA= 2022-09-12T18:49:47.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:49:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:j8MhKs2EfGRDZq3njDuYYgLtAwA='} 2022-09-12T18:49:47.178 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d22a047dc0752e7c-00631f7f4b-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 18:49:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:47.179 DEBUG:boto:path=/jdpiak-12/ 2022-09-12T18:49:47.180 DEBUG:boto:auth_path=/jdpiak-12/ 2022-09-12T18:49:47.181 DEBUG:boto:Method: HEAD 2022-09-12T18:49:47.182 DEBUG:boto:Path: /jdpiak-12/ 2022-09-12T18:49:47.183 DEBUG:boto:Data: 2022-09-12T18:49:47.184 DEBUG:boto:Headers: {} 2022-09-12T18:49:47.185 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:47.186 DEBUG:boto:Port: 8001 2022-09-12T18:49:47.187 DEBUG:boto:Params: {} 2022-09-12T18:49:47.188 DEBUG:boto:Token: None 2022-09-12T18:49:47.189 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:47 GMT /jdpiak-12/ 2022-09-12T18:49:47.190 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fo+074LxcsJDlwJMWUVhJ54UjkI= 2022-09-12T18:49:47.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fo+074LxcsJDlwJMWUVhJ54UjkI=', 'Content-Length': '0'} 2022-09-12T18:49:47.194 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009066cb6257cc91a2-00631f7f4b-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:47.195 DEBUG:boto:path=/jdpiak-12/myobj 2022-09-12T18:49:47.196 DEBUG:boto:auth_path=/jdpiak-12/myobj 2022-09-12T18:49:47.196 DEBUG:boto:Method: PUT 2022-09-12T18:49:47.197 DEBUG:boto:Path: /jdpiak-12/myobj 2022-09-12T18:49:47.197 DEBUG:boto:Data: 2022-09-12T18:49:47.198 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T18:49:47.198 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:47.199 DEBUG:boto:Port: 8001 2022-09-12T18:49:47.199 DEBUG:boto:Params: {} 2022-09-12T18:49:47.200 DEBUG:boto:Token: None 2022-09-12T18:49:47.200 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 18:49:47 GMT /jdpiak-12/myobj 2022-09-12T18:49:47.201 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zrnpyw1Ayd8rFQL45xTA1vlc5zI= 2022-09-12T18:49:47.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:49:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zrnpyw1Ayd8rFQL45xTA1vlc5zI='} 2022-09-12T18:49:47.246 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000068e4132a07bdbbf9-00631f7f4b-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 18:49:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:47.247 DEBUG:boto:path=/jdpiak-12/ 2022-09-12T18:49:47.248 DEBUG:boto:auth_path=/jdpiak-12/ 2022-09-12T18:49:47.248 DEBUG:boto:Method: HEAD 2022-09-12T18:49:47.249 DEBUG:boto:Path: /jdpiak-12/ 2022-09-12T18:49:47.249 DEBUG:boto:Data: 2022-09-12T18:49:47.250 DEBUG:boto:Headers: {} 2022-09-12T18:49:47.250 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:47.251 DEBUG:boto:Port: 8001 2022-09-12T18:49:47.251 DEBUG:boto:Params: {} 2022-09-12T18:49:47.252 DEBUG:boto:Token: None 2022-09-12T18:49:47.252 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:47 GMT /jdpiak-12/ 2022-09-12T18:49:47.253 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fo+074LxcsJDlwJMWUVhJ54UjkI= 2022-09-12T18:49:47.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fo+074LxcsJDlwJMWUVhJ54UjkI=', 'Content-Length': '0'} 2022-09-12T18:49:47.256 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000deee48b2e01811e6-00631f7f4b-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:47.257 DEBUG:boto:path=/jdpiak-12/_myobj 2022-09-12T18:49:47.257 DEBUG:boto:auth_path=/jdpiak-12/_myobj 2022-09-12T18:49:47.258 DEBUG:boto:Method: PUT 2022-09-12T18:49:47.258 DEBUG:boto:Path: /jdpiak-12/_myobj 2022-09-12T18:49:47.259 DEBUG:boto:Data: 2022-09-12T18:49:47.259 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T18:49:47.260 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:47.260 DEBUG:boto:Port: 8001 2022-09-12T18:49:47.261 DEBUG:boto:Params: {} 2022-09-12T18:49:47.261 DEBUG:boto:Token: None 2022-09-12T18:49:47.262 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 18:49:47 GMT /jdpiak-12/_myobj 2022-09-12T18:49:47.263 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DFmM5kVyEAZ+yC7LoBJnv362S10= 2022-09-12T18:49:47.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:49:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DFmM5kVyEAZ+yC7LoBJnv362S10='} 2022-09-12T18:49:47.279 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000744512117437ed10-00631f7f4b-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 18:49:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:47.280 DEBUG:boto:path=/jdpiak-12/ 2022-09-12T18:49:47.280 DEBUG:boto:auth_path=/jdpiak-12/ 2022-09-12T18:49:47.281 DEBUG:boto:Method: HEAD 2022-09-12T18:49:47.281 DEBUG:boto:Path: /jdpiak-12/ 2022-09-12T18:49:47.282 DEBUG:boto:Data: 2022-09-12T18:49:47.283 DEBUG:boto:Headers: {} 2022-09-12T18:49:47.283 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:47.284 DEBUG:boto:Port: 8001 2022-09-12T18:49:47.284 DEBUG:boto:Params: {} 2022-09-12T18:49:47.285 DEBUG:boto:Token: None 2022-09-12T18:49:47.285 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:47 GMT /jdpiak-12/ 2022-09-12T18:49:47.286 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fo+074LxcsJDlwJMWUVhJ54UjkI= 2022-09-12T18:49:47.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fo+074LxcsJDlwJMWUVhJ54UjkI=', 'Content-Length': '0'} 2022-09-12T18:49:47.289 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009d8c7b8c6dd7094-00631f7f4b-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:47.290 DEBUG:boto:path=/jdpiak-12/%3A 2022-09-12T18:49:47.290 DEBUG:boto:auth_path=/jdpiak-12/%3A 2022-09-12T18:49:47.291 DEBUG:boto:Method: PUT 2022-09-12T18:49:47.291 DEBUG:boto:Path: /jdpiak-12/%3A 2022-09-12T18:49:47.292 DEBUG:boto:Data: 2022-09-12T18:49:47.292 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T18:49:47.293 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:47.294 DEBUG:boto:Port: 8001 2022-09-12T18:49:47.294 DEBUG:boto:Params: {} 2022-09-12T18:49:47.295 DEBUG:boto:Token: None 2022-09-12T18:49:47.296 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 18:49:47 GMT /jdpiak-12/%3A 2022-09-12T18:49:47.296 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+l5SPoVb0ryxZWYj+U+i/yq+Fr0= 2022-09-12T18:49:47.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:49:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+l5SPoVb0ryxZWYj+U+i/yq+Fr0='} 2022-09-12T18:49:47.312 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000367185c9d447c588-00631f7f4b-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 18:49:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:47.313 DEBUG:boto:path=/jdpiak-12/ 2022-09-12T18:49:47.313 DEBUG:boto:auth_path=/jdpiak-12/ 2022-09-12T18:49:47.314 DEBUG:boto:Method: HEAD 2022-09-12T18:49:47.314 DEBUG:boto:Path: /jdpiak-12/ 2022-09-12T18:49:47.315 DEBUG:boto:Data: 2022-09-12T18:49:47.315 DEBUG:boto:Headers: {} 2022-09-12T18:49:47.316 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:47.316 DEBUG:boto:Port: 8001 2022-09-12T18:49:47.316 DEBUG:boto:Params: {} 2022-09-12T18:49:47.317 DEBUG:boto:Token: None 2022-09-12T18:49:47.318 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:49:47 GMT /jdpiak-12/ 2022-09-12T18:49:47.318 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fo+074LxcsJDlwJMWUVhJ54UjkI= 2022-09-12T18:49:47.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:49:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fo+074LxcsJDlwJMWUVhJ54UjkI=', 'Content-Length': '0'} 2022-09-12T18:49:47.322 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c63e385f24f0e28-00631f7f4b-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:49:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:47.323 DEBUG:boto:path=/jdpiak-12/%26 2022-09-12T18:49:47.323 DEBUG:boto:auth_path=/jdpiak-12/%26 2022-09-12T18:49:47.324 DEBUG:boto:Method: PUT 2022-09-12T18:49:47.324 DEBUG:boto:Path: /jdpiak-12/%26 2022-09-12T18:49:47.325 DEBUG:boto:Data: 2022-09-12T18:49:47.325 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T18:49:47.326 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:49:47.326 DEBUG:boto:Port: 8001 2022-09-12T18:49:47.327 DEBUG:boto:Params: {} 2022-09-12T18:49:47.328 DEBUG:boto:Token: None 2022-09-12T18:49:47.329 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 18:49:47 GMT /jdpiak-12/%26 2022-09-12T18:49:47.330 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fIMPDZQXdin2Q/aFA6AdHOHj7X4= 2022-09-12T18:49:47.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:49:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fIMPDZQXdin2Q/aFA6AdHOHj7X4='} 2022-09-12T18:49:47.374 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000071c07170dbaf5a96-00631f7f4b-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 18:49:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:49:47.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:47.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:47.377 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:47.432 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:47.432 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:47.470 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=c0f7a512-c940-4d1f-8adf-3484630f876b 2022-09-12T18:49:47.482 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T18:49:47.482 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:47.483 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:47.483 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:47:27.976965Z" 2022-09-12T18:49:47.483 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:47.484 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:47.484 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:47.484 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T18:49:47.484 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:47.485 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:47.485 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008586.780479_68.1", 2022-09-12T18:49:47.485 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:49:46.780479Z" 2022-09-12T18:49:47.485 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:47.486 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:47.486 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T18:49:47.486 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:49:40.904048Z" 2022-09-12T18:49:47.486 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:47.487 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T18:49:47.488 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663008586.780479_68.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:49:47.488 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T18:49:47.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:47.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:47.490 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:47.540 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:47.540 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:47.574 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:47.570+0000 7f699875b600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:47.575 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:47.571+0000 7f699875b600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:47.586 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:49:47.587 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:49:47.587 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:49:47.587 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:49:47.587 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:49:47.588 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:47.588 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:47.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:47.588 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:49:47.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:47.589 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:49:47.589 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:47.590 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:47.590 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:47.590 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:47.590 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:49:47.591 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:47.591 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:47.591 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:47.592 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:47.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:47.592 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:47.592 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:49:47.593 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:47.593 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:47.593 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:47.594 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:47.594 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:47.594 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:47.595 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:47.595 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:47.595 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:47.596 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:47.596 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:47.596 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:49:47.596 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:47.597 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:47.597 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008586.711103_57.1", 2022-09-12T18:49:47.597 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:47.597 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:47.598 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:47.598 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.711103Z", 2022-09-12T18:49:47.598 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:47.599 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:47.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:47.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:47.599 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:49:47.600 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:47.600 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:47.600 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T18:49:47.600 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:47.601 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:47.601 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:47.601 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T18:49:47.601 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:47.602 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:47.602 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:47.602 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:49:47.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:47.603 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:49:47.603 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:49:47.603 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:49:47.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:47.604 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:49:47Z" 2022-09-12T18:49:47.604 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:49:47.605 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663008586.711103_57.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:46.711103Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:49:47Z" } 2022-09-12T18:49:47.606 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663008586.711103_57.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:46.711103Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T18:49:47.607 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663008586.780479_68.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:49:47.607 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663008586.711103_57.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:49:47.608 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663008586.780479_68.1 target=1_1663008586.711103_57.1 2022-09-12T18:49:52.609 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:52.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:52.611 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:52.668 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:52.668 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:52.710 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:52.708+0000 7f7d25d4c600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:52.711 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:52.709+0000 7f7d25d4c600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:52.728 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:49:52.728 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:49:52.728 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:49:52.728 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:49:52.729 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:49:52.729 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:52.729 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:52.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:52.730 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:49:52.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:52.730 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:49:52.731 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:52.731 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:52.731 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:52.731 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:52.731 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:49:52.732 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:52.732 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:52.732 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:52.732 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:52.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:52.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:52.733 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:49:52.733 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:52.734 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:52.734 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:52.734 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:52.734 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:52.735 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:52.735 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:52.735 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:52.735 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:52.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:52.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:52.736 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:49:52.737 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:52.737 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:52.737 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008586.780479_68.1", 2022-09-12T18:49:52.737 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:52.738 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:52.738 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:52.738 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.780479Z", 2022-09-12T18:49:52.739 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:52.739 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:52.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:52.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:52.740 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:49:52.740 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:52.740 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:52.741 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T18:49:52.741 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:52.741 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:52.741 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:52.742 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T18:49:52.742 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:52.742 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:52.743 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:52.743 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:49:52.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:52.743 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:49:52.744 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:49:52.744 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:49:52.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:52.744 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:49:52Z" 2022-09-12T18:49:52.745 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:49:52.746 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663008586.780479_68.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:46.780479Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:49:52Z" } 2022-09-12T18:49:52.747 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663008586.780479_68.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:46.780479Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T18:49:52.747 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663008586.780479_68.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:49:52.748 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663008586.780479_68.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:49:52.748 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T18:49:52.749 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T18:49:52.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:52.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:52.750 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:52.809 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:52.809 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:52.856 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:49:52.852+0000 7fda1cddd600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:52.857 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:49:52.854+0000 7fda1cddd600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:52.870 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T18:49:52.870 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T18:49:52.871 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T18:49:52.871 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T18:49:52.871 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T18:49:52.872 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:52.872 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:52.872 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:52.873 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T18:49:52.873 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:52.873 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T18:49:52.873 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:52.874 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:52.874 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:52.874 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:52.875 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T18:49:52.875 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:52.875 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:52.875 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:52.876 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:52.876 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:52.877 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:52.877 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T18:49:52.877 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:52.877 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:52.878 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:49:52.878 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:52.879 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:49:52.879 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:52.879 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T18:49:52.880 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:52.880 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:52.881 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:52.881 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:52.882 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T18:49:52.882 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:52.883 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:52.883 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008586.780479_68.1", 2022-09-12T18:49:52.884 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:52.884 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:49:52.885 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:52.885 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:46.780479Z", 2022-09-12T18:49:52.885 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:52.885 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:52.886 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:52.886 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:49:52.887 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T18:49:52.887 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:49:52.888 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:49:52.888 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T18:49:52.888 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:49:52.889 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:49:52.889 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:49:52.890 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T18:49:52.890 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:49:52.891 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:52.891 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:49:52.891 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T18:49:52.892 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:52.893 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T18:49:52.893 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T18:49:52.893 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T18:49:52.893 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:49:52.894 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T18:49:52Z" 2022-09-12T18:49:52.894 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T18:49:52.895 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663008586.780479_68.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:46.780479Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:49:52Z" } 2022-09-12T18:49:52.896 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663008586.780479_68.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:46.780479Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T18:49:52.896 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663008586.780479_68.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:49:52.897 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663008586.780479_68.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:49:52.897 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T18:49:52.898 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T18:49:52.899 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:52.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:49:52.900 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:49:52.951 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:52.952 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:52.979 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:52.976+0000 7fb3b7c0e600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:52.979 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:52.977+0000 7fb3b7c0e600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:49:52.990 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:49:52.991 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:49:52.991 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:49:52.992 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:49:52.992 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:49:52.992 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:49:52.993 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:52.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:52.994 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:49:52.994 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:52.995 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:49:52.995 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:52.995 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:52.996 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:49:52.996 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:52.996 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:49:52.996 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:52.997 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:49:52.997 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:52.997 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:52.998 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:52.998 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:52.998 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:49:52.999 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:52.999 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:52.999 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:49:52.999 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:53.000 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:53.000 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:53.000 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:49:53.000 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:53.001 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:53.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:53.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:53.002 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:49:53.002 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:53.002 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:53.002 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008586.780479_68.1", 2022-09-12T18:49:53.003 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:53.003 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:53.003 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:53.004 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.780479Z", 2022-09-12T18:49:53.004 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:53.004 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:53.005 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:53.005 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:49:53.005 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:49:53.005 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:49:53.006 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:49:53.006 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T18:49:53.006 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:49:53.006 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:49:53.007 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:49:53.007 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T18:49:53.007 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:49:53.008 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:53.008 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:49:53.008 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:49:53.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:53.009 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:49:53.010 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:49:53.010 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:49:53.011 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:49:53.011 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:49:52Z" 2022-09-12T18:49:53.012 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:49:53.014 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663008586.780479_68.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:46.780479Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:49:52Z" } 2022-09-12T18:49:53.015 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663008586.780479_68.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:46.780479Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T18:49:53.016 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663008586.780479_68.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:49:53.017 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663008586.780479_68.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:49:53.018 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T18:49:53.019 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:49:53.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:49:53.020 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T18:49:53.075 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:49:53.075 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:49:53.109 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:53.107+0000 7fd162a1b600 1 waiting to reach incremental sync.. 2022-09-12T18:49:58.110 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:49:58.107+0000 7fd162a1b600 1 waiting to reach incremental sync.. 2022-09-12T18:50:03.112 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:50:03.108+0000 7fd162a1b600 1 bucket sync caught up with source: 2022-09-12T18:50:03.113 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:50:03.113 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:50:03.113 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:50:03.108+0000 7fd162a1b600 0 bucket checkpoint complete 2022-09-12T18:50:03.134 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-10 zones={test-zone2, test-zone1} 2022-09-12T18:50:03.135 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:03.136 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:03.137 DEBUG:boto:Method: HEAD 2022-09-12T18:50:03.138 DEBUG:boto:Path: /jdpiak-10/ 2022-09-12T18:50:03.139 DEBUG:boto:Data: 2022-09-12T18:50:03.139 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.140 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:03.141 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.142 DEBUG:boto:Params: {} 2022-09-12T18:50:03.143 DEBUG:boto:Token: None 2022-09-12T18:50:03.145 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/ 2022-09-12T18:50:03.146 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jA8OAIHBR1EIInzRQmW/Dee4mUE= 2022-09-12T18:50:03.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jA8OAIHBR1EIInzRQmW/Dee4mUE=', 'Content-Length': '0'} 2022-09-12T18:50:03.150 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018d395b772c688a8-00631f7f5b-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.151 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:03.152 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:03.153 DEBUG:boto:Method: HEAD 2022-09-12T18:50:03.154 DEBUG:boto:Path: /jdpiak-10/ 2022-09-12T18:50:03.155 DEBUG:boto:Data: 2022-09-12T18:50:03.156 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.157 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:03.158 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.159 DEBUG:boto:Params: {} 2022-09-12T18:50:03.160 DEBUG:boto:Token: None 2022-09-12T18:50:03.161 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/ 2022-09-12T18:50:03.162 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jA8OAIHBR1EIInzRQmW/Dee4mUE= 2022-09-12T18:50:03.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jA8OAIHBR1EIInzRQmW/Dee4mUE=', 'Content-Length': '0'} 2022-09-12T18:50:03.167 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8f958ea90967e35-00631f7f5b-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.168 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T18:50:03.169 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:03.171 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:03.172 DEBUG:boto:path=/jdpiak-10/?versions&max-keys=999 2022-09-12T18:50:03.173 DEBUG:boto:auth_path=/jdpiak-10/?versions&max-keys=999 2022-09-12T18:50:03.174 DEBUG:boto:Method: GET 2022-09-12T18:50:03.175 DEBUG:boto:Path: /jdpiak-10/?versions&max-keys=999 2022-09-12T18:50:03.176 DEBUG:boto:Data: 2022-09-12T18:50:03.177 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.178 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:03.179 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.180 DEBUG:boto:Params: {} 2022-09-12T18:50:03.181 DEBUG:boto:Token: None 2022-09-12T18:50:03.181 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/?versions 2022-09-12T18:50:03.182 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k5XdEO34CC6Ou4No/lUeNG3+7OI= 2022-09-12T18:50:03.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k5XdEO34CC6Ou4No/lUeNG3+7OI=', 'Content-Length': '0'} 2022-09-12T18:50:03.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7bbba05ac77a163-00631f7f5b-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.186 DEBUG:boto:b'jdpiak-10999false&nulltrue2022-09-12T18:49:46.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-12T18:49:46.939Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-12T18:49:46.878Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-12T18:49:46.847Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:03.188 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-12T18:50:03.189 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-12T18:50:03.189 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-12T18:50:03.190 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-12T18:50:03.190 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T18:50:03.191 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:03.191 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:03.192 DEBUG:boto:path=/jdpiak-10/?versions&max-keys=999 2022-09-12T18:50:03.192 DEBUG:boto:auth_path=/jdpiak-10/?versions&max-keys=999 2022-09-12T18:50:03.193 DEBUG:boto:Method: GET 2022-09-12T18:50:03.194 DEBUG:boto:Path: /jdpiak-10/?versions&max-keys=999 2022-09-12T18:50:03.195 DEBUG:boto:Data: 2022-09-12T18:50:03.196 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.197 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:03.198 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.199 DEBUG:boto:Params: {} 2022-09-12T18:50:03.200 DEBUG:boto:Token: None 2022-09-12T18:50:03.201 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/?versions 2022-09-12T18:50:03.202 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k5XdEO34CC6Ou4No/lUeNG3+7OI= 2022-09-12T18:50:03.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k5XdEO34CC6Ou4No/lUeNG3+7OI=', 'Content-Length': '0'} 2022-09-12T18:50:03.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084d9091b76680842-00631f7f5b-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.207 DEBUG:boto:b'jdpiak-10999false&nulltrue2022-09-12T18:49:46.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-12T18:49:46.939Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-12T18:49:46.878Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-12T18:49:46.847Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:03.209 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-12T18:50:03.209 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-12T18:50:03.210 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-12T18:50:03.211 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-12T18:50:03.212 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:03.213 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:03.214 DEBUG:boto:path=/jdpiak-10/?versions&max-keys=999 2022-09-12T18:50:03.215 DEBUG:boto:auth_path=/jdpiak-10/?versions&max-keys=999 2022-09-12T18:50:03.215 DEBUG:boto:Method: GET 2022-09-12T18:50:03.216 DEBUG:boto:Path: /jdpiak-10/?versions&max-keys=999 2022-09-12T18:50:03.217 DEBUG:boto:Data: 2022-09-12T18:50:03.218 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.219 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:03.220 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.221 DEBUG:boto:Params: {} 2022-09-12T18:50:03.222 DEBUG:boto:Token: None 2022-09-12T18:50:03.223 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/?versions 2022-09-12T18:50:03.224 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k5XdEO34CC6Ou4No/lUeNG3+7OI= 2022-09-12T18:50:03.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k5XdEO34CC6Ou4No/lUeNG3+7OI=', 'Content-Length': '0'} 2022-09-12T18:50:03.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007034b969ea2091fa-00631f7f5b-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.228 DEBUG:boto:b'jdpiak-10999false&nulltrue2022-09-12T18:49:46.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-12T18:49:46.939Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-12T18:49:46.878Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-12T18:49:46.847Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:03.229 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:03.230 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:03.230 DEBUG:boto:path=/jdpiak-10/?versions&max-keys=999 2022-09-12T18:50:03.231 DEBUG:boto:auth_path=/jdpiak-10/?versions&max-keys=999 2022-09-12T18:50:03.241 DEBUG:boto:Method: GET 2022-09-12T18:50:03.242 DEBUG:boto:Path: /jdpiak-10/?versions&max-keys=999 2022-09-12T18:50:03.242 DEBUG:boto:Data: 2022-09-12T18:50:03.243 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.243 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:03.244 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.245 DEBUG:boto:Params: {} 2022-09-12T18:50:03.245 DEBUG:boto:Token: None 2022-09-12T18:50:03.246 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/?versions 2022-09-12T18:50:03.246 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k5XdEO34CC6Ou4No/lUeNG3+7OI= 2022-09-12T18:50:03.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k5XdEO34CC6Ou4No/lUeNG3+7OI=', 'Content-Length': '0'} 2022-09-12T18:50:03.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bee48ad0211ba2c4-00631f7f5b-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.250 DEBUG:boto:b'jdpiak-10999false&nulltrue2022-09-12T18:49:46.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-12T18:49:46.939Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-12T18:49:46.878Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-12T18:49:46.847Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:03.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-12T18:50:03.252 DEBUG:boto:path=/jdpiak-10/%26 2022-09-12T18:50:03.252 DEBUG:boto:auth_path=/jdpiak-10/%26 2022-09-12T18:50:03.253 DEBUG:boto:path=/jdpiak-10/%26?versionId=null 2022-09-12T18:50:03.253 DEBUG:boto:auth_path=/jdpiak-10/%26?versionId=null 2022-09-12T18:50:03.254 DEBUG:boto:Method: GET 2022-09-12T18:50:03.255 DEBUG:boto:Path: /jdpiak-10/%26?versionId=null 2022-09-12T18:50:03.255 DEBUG:boto:Data: 2022-09-12T18:50:03.256 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.256 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:03.257 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.257 DEBUG:boto:Params: {} 2022-09-12T18:50:03.258 DEBUG:boto:Token: None 2022-09-12T18:50:03.259 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/%26?versionId=null 2022-09-12T18:50:03.259 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iAzAPyI85a+qa3XP/rZkRehQors= 2022-09-12T18:50:03.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iAzAPyI85a+qa3XP/rZkRehQors=', 'Content-Length': '0'} 2022-09-12T18:50:03.262 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d85cdf9991479d4a-00631f7f5b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.263 DEBUG:boto:path=/jdpiak-10/%26 2022-09-12T18:50:03.263 DEBUG:boto:auth_path=/jdpiak-10/%26 2022-09-12T18:50:03.264 DEBUG:boto:path=/jdpiak-10/%26?versionId=null 2022-09-12T18:50:03.265 DEBUG:boto:auth_path=/jdpiak-10/%26?versionId=null 2022-09-12T18:50:03.265 DEBUG:boto:Method: GET 2022-09-12T18:50:03.266 DEBUG:boto:Path: /jdpiak-10/%26?versionId=null 2022-09-12T18:50:03.267 DEBUG:boto:Data: 2022-09-12T18:50:03.267 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.268 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:03.268 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.269 DEBUG:boto:Params: {} 2022-09-12T18:50:03.270 DEBUG:boto:Token: None 2022-09-12T18:50:03.270 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/%26?versionId=null 2022-09-12T18:50:03.271 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iAzAPyI85a+qa3XP/rZkRehQors= 2022-09-12T18:50:03.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iAzAPyI85a+qa3XP/rZkRehQors=', 'Content-Length': '0'} 2022-09-12T18:50:03.274 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bfdb6df33eb434df-00631f7f5b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.275 DEBUG:boto:path=/jdpiak-10/%26 2022-09-12T18:50:03.275 DEBUG:boto:auth_path=/jdpiak-10/%26 2022-09-12T18:50:03.276 DEBUG:boto:path=/jdpiak-10/%26?versionId=null 2022-09-12T18:50:03.276 DEBUG:boto:auth_path=/jdpiak-10/%26?versionId=null 2022-09-12T18:50:03.277 DEBUG:boto:Method: HEAD 2022-09-12T18:50:03.277 DEBUG:boto:Path: /jdpiak-10/%26?versionId=null 2022-09-12T18:50:03.278 DEBUG:boto:Data: 2022-09-12T18:50:03.278 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.279 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:03.279 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.280 DEBUG:boto:Params: {} 2022-09-12T18:50:03.281 DEBUG:boto:Token: None 2022-09-12T18:50:03.282 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/%26?versionId=null 2022-09-12T18:50:03.283 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Er0lxhicpsVUlnpWpUfh0J0o9GU= 2022-09-12T18:50:03.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Er0lxhicpsVUlnpWpUfh0J0o9GU=', 'Content-Length': '0'} 2022-09-12T18:50:03.287 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000300d5dfa3af4ccd9-00631f7f5b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.288 DEBUG:boto:path=/jdpiak-10/%26 2022-09-12T18:50:03.289 DEBUG:boto:auth_path=/jdpiak-10/%26 2022-09-12T18:50:03.289 DEBUG:boto:path=/jdpiak-10/%26?versionId=null 2022-09-12T18:50:03.290 DEBUG:boto:auth_path=/jdpiak-10/%26?versionId=null 2022-09-12T18:50:03.290 DEBUG:boto:Method: HEAD 2022-09-12T18:50:03.291 DEBUG:boto:Path: /jdpiak-10/%26?versionId=null 2022-09-12T18:50:03.291 DEBUG:boto:Data: 2022-09-12T18:50:03.292 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.292 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:03.293 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.293 DEBUG:boto:Params: {} 2022-09-12T18:50:03.294 DEBUG:boto:Token: None 2022-09-12T18:50:03.294 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/%26?versionId=null 2022-09-12T18:50:03.295 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Er0lxhicpsVUlnpWpUfh0J0o9GU= 2022-09-12T18:50:03.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Er0lxhicpsVUlnpWpUfh0J0o9GU=', 'Content-Length': '0'} 2022-09-12T18:50:03.298 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000722029adf2a1d39a-00631f7f5b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-12T18:50:03.299 DEBUG:boto:path=/jdpiak-10/%26 2022-09-12T18:50:03.300 DEBUG:boto:auth_path=/jdpiak-10/%26 2022-09-12T18:50:03.300 DEBUG:boto:path=/jdpiak-10/%26?versionId=null 2022-09-12T18:50:03.301 DEBUG:boto:auth_path=/jdpiak-10/%26?versionId=null 2022-09-12T18:50:03.301 DEBUG:boto:Method: GET 2022-09-12T18:50:03.302 DEBUG:boto:Path: /jdpiak-10/%26?versionId=null 2022-09-12T18:50:03.302 DEBUG:boto:Data: 2022-09-12T18:50:03.303 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.303 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:03.304 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.304 DEBUG:boto:Params: {} 2022-09-12T18:50:03.305 DEBUG:boto:Token: None 2022-09-12T18:50:03.306 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/%26?versionId=null 2022-09-12T18:50:03.306 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iAzAPyI85a+qa3XP/rZkRehQors= 2022-09-12T18:50:03.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iAzAPyI85a+qa3XP/rZkRehQors=', 'Content-Length': '0'} 2022-09-12T18:50:03.309 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008fe532e6e14f4aea-00631f7f5b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.354 DEBUG:boto:path=/jdpiak-10/%26 2022-09-12T18:50:03.355 DEBUG:boto:auth_path=/jdpiak-10/%26 2022-09-12T18:50:03.355 DEBUG:boto:path=/jdpiak-10/%26?versionId=null 2022-09-12T18:50:03.356 DEBUG:boto:auth_path=/jdpiak-10/%26?versionId=null 2022-09-12T18:50:03.357 DEBUG:boto:Method: GET 2022-09-12T18:50:03.358 DEBUG:boto:Path: /jdpiak-10/%26?versionId=null 2022-09-12T18:50:03.359 DEBUG:boto:Data: 2022-09-12T18:50:03.360 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.361 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:03.361 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.362 DEBUG:boto:Params: {} 2022-09-12T18:50:03.363 DEBUG:boto:Token: None 2022-09-12T18:50:03.363 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/%26?versionId=null 2022-09-12T18:50:03.364 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iAzAPyI85a+qa3XP/rZkRehQors= 2022-09-12T18:50:03.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iAzAPyI85a+qa3XP/rZkRehQors=', 'Content-Length': '0'} 2022-09-12T18:50:03.367 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7c0017b3e4517cf-00631f7f5b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.368 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:03.369 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:03.369 DEBUG:boto:path=/jdpiak-10/?prefix=%26 2022-09-12T18:50:03.370 DEBUG:boto:auth_path=/jdpiak-10/?prefix=%26 2022-09-12T18:50:03.370 DEBUG:boto:Method: GET 2022-09-12T18:50:03.371 DEBUG:boto:Path: /jdpiak-10/?prefix=%26 2022-09-12T18:50:03.372 DEBUG:boto:Data: 2022-09-12T18:50:03.373 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.374 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:03.375 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.376 DEBUG:boto:Params: {} 2022-09-12T18:50:03.377 DEBUG:boto:Token: None 2022-09-12T18:50:03.377 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/ 2022-09-12T18:50:03.378 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5tQNLH26t1xV3d1kpelDI1MTmT4= 2022-09-12T18:50:03.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5tQNLH26t1xV3d1kpelDI1MTmT4=', 'Content-Length': '0'} 2022-09-12T18:50:03.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003601fcbda0e8c1f8-00631f7f5b-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.382 DEBUG:boto:b'jdpiak-10&1000false&2022-09-12T18:49:46.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:03.383 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:03.384 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:03.384 DEBUG:boto:path=/jdpiak-10/?prefix=%26 2022-09-12T18:50:03.385 DEBUG:boto:auth_path=/jdpiak-10/?prefix=%26 2022-09-12T18:50:03.385 DEBUG:boto:Method: GET 2022-09-12T18:50:03.386 DEBUG:boto:Path: /jdpiak-10/?prefix=%26 2022-09-12T18:50:03.387 DEBUG:boto:Data: 2022-09-12T18:50:03.387 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.388 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:03.388 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.389 DEBUG:boto:Params: {} 2022-09-12T18:50:03.389 DEBUG:boto:Token: None 2022-09-12T18:50:03.390 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/ 2022-09-12T18:50:03.390 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5tQNLH26t1xV3d1kpelDI1MTmT4= 2022-09-12T18:50:03.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5tQNLH26t1xV3d1kpelDI1MTmT4=', 'Content-Length': '0'} 2022-09-12T18:50:03.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b40d40f2d42af97-00631f7f5b-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.394 DEBUG:boto:b'jdpiak-10&1000false&2022-09-12T18:49:46.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:03.395 DEBUG:boto:path=/jdpiak-10/%26 2022-09-12T18:50:03.396 DEBUG:boto:auth_path=/jdpiak-10/%26 2022-09-12T18:50:03.396 DEBUG:boto:Method: HEAD 2022-09-12T18:50:03.397 DEBUG:boto:Path: /jdpiak-10/%26 2022-09-12T18:50:03.397 DEBUG:boto:Data: 2022-09-12T18:50:03.398 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.398 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:03.399 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.399 DEBUG:boto:Params: {} 2022-09-12T18:50:03.400 DEBUG:boto:Token: None 2022-09-12T18:50:03.400 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/%26 2022-09-12T18:50:03.401 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aFJCYw7mFK5jjNDJhf7z2bBhGOE= 2022-09-12T18:50:03.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aFJCYw7mFK5jjNDJhf7z2bBhGOE=', 'Content-Length': '0'} 2022-09-12T18:50:03.404 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f31193de41c36547-00631f7f5b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.404 DEBUG:boto:path=/jdpiak-10/%26 2022-09-12T18:50:03.405 DEBUG:boto:auth_path=/jdpiak-10/%26 2022-09-12T18:50:03.405 DEBUG:boto:Method: HEAD 2022-09-12T18:50:03.406 DEBUG:boto:Path: /jdpiak-10/%26 2022-09-12T18:50:03.407 DEBUG:boto:Data: 2022-09-12T18:50:03.408 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.409 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:03.410 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.411 DEBUG:boto:Params: {} 2022-09-12T18:50:03.412 DEBUG:boto:Token: None 2022-09-12T18:50:03.413 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/%26 2022-09-12T18:50:03.414 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aFJCYw7mFK5jjNDJhf7z2bBhGOE= 2022-09-12T18:50:03.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aFJCYw7mFK5jjNDJhf7z2bBhGOE=', 'Content-Length': '0'} 2022-09-12T18:50:03.418 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c80c949843b1a38-00631f7f5b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-12T18:50:03.419 DEBUG:boto:path=/jdpiak-10/%26 2022-09-12T18:50:03.420 DEBUG:boto:auth_path=/jdpiak-10/%26 2022-09-12T18:50:03.420 DEBUG:boto:Method: GET 2022-09-12T18:50:03.421 DEBUG:boto:Path: /jdpiak-10/%26 2022-09-12T18:50:03.421 DEBUG:boto:Data: 2022-09-12T18:50:03.422 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.422 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:03.423 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.423 DEBUG:boto:Params: {} 2022-09-12T18:50:03.424 DEBUG:boto:Token: None 2022-09-12T18:50:03.425 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/%26 2022-09-12T18:50:03.425 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q7GHHRf3iCUqC62fLRnlyISy8PY= 2022-09-12T18:50:03.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q7GHHRf3iCUqC62fLRnlyISy8PY=', 'Content-Length': '0'} 2022-09-12T18:50:03.428 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ff569fb722e1048-00631f7f5b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.429 DEBUG:boto:path=/jdpiak-10/%26 2022-09-12T18:50:03.429 DEBUG:boto:auth_path=/jdpiak-10/%26 2022-09-12T18:50:03.430 DEBUG:boto:Method: GET 2022-09-12T18:50:03.431 DEBUG:boto:Path: /jdpiak-10/%26 2022-09-12T18:50:03.432 DEBUG:boto:Data: 2022-09-12T18:50:03.433 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.433 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:03.434 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.435 DEBUG:boto:Params: {} 2022-09-12T18:50:03.436 DEBUG:boto:Token: None 2022-09-12T18:50:03.437 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/%26 2022-09-12T18:50:03.437 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q7GHHRf3iCUqC62fLRnlyISy8PY= 2022-09-12T18:50:03.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q7GHHRf3iCUqC62fLRnlyISy8PY=', 'Content-Length': '0'} 2022-09-12T18:50:03.440 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9a1f7554a65aef0-00631f7f5b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.486 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:03.487 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:03.488 DEBUG:boto:path=/jdpiak-10/?prefix=%26 2022-09-12T18:50:03.489 DEBUG:boto:auth_path=/jdpiak-10/?prefix=%26 2022-09-12T18:50:03.490 DEBUG:boto:Method: GET 2022-09-12T18:50:03.491 DEBUG:boto:Path: /jdpiak-10/?prefix=%26 2022-09-12T18:50:03.491 DEBUG:boto:Data: 2022-09-12T18:50:03.492 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.492 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:03.493 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.494 DEBUG:boto:Params: {} 2022-09-12T18:50:03.494 DEBUG:boto:Token: None 2022-09-12T18:50:03.495 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/ 2022-09-12T18:50:03.496 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5tQNLH26t1xV3d1kpelDI1MTmT4= 2022-09-12T18:50:03.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5tQNLH26t1xV3d1kpelDI1MTmT4=', 'Content-Length': '0'} 2022-09-12T18:50:03.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002213991be103fab2-00631f7f5b-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.500 DEBUG:boto:b'jdpiak-10&1000false&2022-09-12T18:49:46.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:03.501 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:03.502 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:03.502 DEBUG:boto:path=/jdpiak-10/?prefix=%26 2022-09-12T18:50:03.503 DEBUG:boto:auth_path=/jdpiak-10/?prefix=%26 2022-09-12T18:50:03.504 DEBUG:boto:Method: GET 2022-09-12T18:50:03.504 DEBUG:boto:Path: /jdpiak-10/?prefix=%26 2022-09-12T18:50:03.505 DEBUG:boto:Data: 2022-09-12T18:50:03.505 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.506 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:03.506 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.507 DEBUG:boto:Params: {} 2022-09-12T18:50:03.507 DEBUG:boto:Token: None 2022-09-12T18:50:03.508 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/ 2022-09-12T18:50:03.509 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5tQNLH26t1xV3d1kpelDI1MTmT4= 2022-09-12T18:50:03.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5tQNLH26t1xV3d1kpelDI1MTmT4=', 'Content-Length': '0'} 2022-09-12T18:50:03.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9a5f117b45a169e-00631f7f5b-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.513 DEBUG:boto:b'jdpiak-10&1000false&2022-09-12T18:49:46.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:03.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-12T18:50:03.515 DEBUG:boto:path=/jdpiak-10/%3A 2022-09-12T18:50:03.515 DEBUG:boto:auth_path=/jdpiak-10/%3A 2022-09-12T18:50:03.516 DEBUG:boto:path=/jdpiak-10/%3A?versionId=null 2022-09-12T18:50:03.516 DEBUG:boto:auth_path=/jdpiak-10/%3A?versionId=null 2022-09-12T18:50:03.517 DEBUG:boto:Method: GET 2022-09-12T18:50:03.517 DEBUG:boto:Path: /jdpiak-10/%3A?versionId=null 2022-09-12T18:50:03.518 DEBUG:boto:Data: 2022-09-12T18:50:03.519 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.519 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:03.520 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.520 DEBUG:boto:Params: {} 2022-09-12T18:50:03.521 DEBUG:boto:Token: None 2022-09-12T18:50:03.521 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/%3A?versionId=null 2022-09-12T18:50:03.522 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fxrMzY10VqjLQWHkO9A31AQ5R7s= 2022-09-12T18:50:03.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fxrMzY10VqjLQWHkO9A31AQ5R7s=', 'Content-Length': '0'} 2022-09-12T18:50:03.525 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b916926a0d81aac-00631f7f5b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.526 DEBUG:boto:path=/jdpiak-10/%3A 2022-09-12T18:50:03.526 DEBUG:boto:auth_path=/jdpiak-10/%3A 2022-09-12T18:50:03.527 DEBUG:boto:path=/jdpiak-10/%3A?versionId=null 2022-09-12T18:50:03.527 DEBUG:boto:auth_path=/jdpiak-10/%3A?versionId=null 2022-09-12T18:50:03.528 DEBUG:boto:Method: GET 2022-09-12T18:50:03.528 DEBUG:boto:Path: /jdpiak-10/%3A?versionId=null 2022-09-12T18:50:03.529 DEBUG:boto:Data: 2022-09-12T18:50:03.529 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.530 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:03.531 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.531 DEBUG:boto:Params: {} 2022-09-12T18:50:03.532 DEBUG:boto:Token: None 2022-09-12T18:50:03.533 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/%3A?versionId=null 2022-09-12T18:50:03.533 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fxrMzY10VqjLQWHkO9A31AQ5R7s= 2022-09-12T18:50:03.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fxrMzY10VqjLQWHkO9A31AQ5R7s=', 'Content-Length': '0'} 2022-09-12T18:50:03.536 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051bc3685f04faf2b-00631f7f5b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.537 DEBUG:boto:path=/jdpiak-10/%3A 2022-09-12T18:50:03.537 DEBUG:boto:auth_path=/jdpiak-10/%3A 2022-09-12T18:50:03.538 DEBUG:boto:path=/jdpiak-10/%3A?versionId=null 2022-09-12T18:50:03.538 DEBUG:boto:auth_path=/jdpiak-10/%3A?versionId=null 2022-09-12T18:50:03.539 DEBUG:boto:Method: HEAD 2022-09-12T18:50:03.539 DEBUG:boto:Path: /jdpiak-10/%3A?versionId=null 2022-09-12T18:50:03.540 DEBUG:boto:Data: 2022-09-12T18:50:03.540 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.541 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:03.541 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.542 DEBUG:boto:Params: {} 2022-09-12T18:50:03.542 DEBUG:boto:Token: None 2022-09-12T18:50:03.543 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/%3A?versionId=null 2022-09-12T18:50:03.543 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jdSy3Q0YjPzMcSwaeTzZYTJ0RLo= 2022-09-12T18:50:03.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jdSy3Q0YjPzMcSwaeTzZYTJ0RLo=', 'Content-Length': '0'} 2022-09-12T18:50:03.546 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005959844da2958acb-00631f7f5b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.547 DEBUG:boto:path=/jdpiak-10/%3A 2022-09-12T18:50:03.547 DEBUG:boto:auth_path=/jdpiak-10/%3A 2022-09-12T18:50:03.548 DEBUG:boto:path=/jdpiak-10/%3A?versionId=null 2022-09-12T18:50:03.548 DEBUG:boto:auth_path=/jdpiak-10/%3A?versionId=null 2022-09-12T18:50:03.549 DEBUG:boto:Method: HEAD 2022-09-12T18:50:03.549 DEBUG:boto:Path: /jdpiak-10/%3A?versionId=null 2022-09-12T18:50:03.550 DEBUG:boto:Data: 2022-09-12T18:50:03.550 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.551 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:03.551 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.552 DEBUG:boto:Params: {} 2022-09-12T18:50:03.552 DEBUG:boto:Token: None 2022-09-12T18:50:03.553 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/%3A?versionId=null 2022-09-12T18:50:03.553 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jdSy3Q0YjPzMcSwaeTzZYTJ0RLo= 2022-09-12T18:50:03.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jdSy3Q0YjPzMcSwaeTzZYTJ0RLo=', 'Content-Length': '0'} 2022-09-12T18:50:03.556 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003586c63c7066fd30-00631f7f5b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-12T18:50:03.558 DEBUG:boto:path=/jdpiak-10/%3A 2022-09-12T18:50:03.559 DEBUG:boto:auth_path=/jdpiak-10/%3A 2022-09-12T18:50:03.559 DEBUG:boto:path=/jdpiak-10/%3A?versionId=null 2022-09-12T18:50:03.560 DEBUG:boto:auth_path=/jdpiak-10/%3A?versionId=null 2022-09-12T18:50:03.562 DEBUG:boto:Method: GET 2022-09-12T18:50:03.562 DEBUG:boto:Path: /jdpiak-10/%3A?versionId=null 2022-09-12T18:50:03.563 DEBUG:boto:Data: 2022-09-12T18:50:03.564 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.564 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:03.565 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.565 DEBUG:boto:Params: {} 2022-09-12T18:50:03.566 DEBUG:boto:Token: None 2022-09-12T18:50:03.567 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/%3A?versionId=null 2022-09-12T18:50:03.567 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fxrMzY10VqjLQWHkO9A31AQ5R7s= 2022-09-12T18:50:03.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fxrMzY10VqjLQWHkO9A31AQ5R7s=', 'Content-Length': '0'} 2022-09-12T18:50:03.570 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091a37b3ac15330fc-00631f7f5b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.571 DEBUG:boto:path=/jdpiak-10/%3A 2022-09-12T18:50:03.571 DEBUG:boto:auth_path=/jdpiak-10/%3A 2022-09-12T18:50:03.572 DEBUG:boto:path=/jdpiak-10/%3A?versionId=null 2022-09-12T18:50:03.572 DEBUG:boto:auth_path=/jdpiak-10/%3A?versionId=null 2022-09-12T18:50:03.573 DEBUG:boto:Method: GET 2022-09-12T18:50:03.573 DEBUG:boto:Path: /jdpiak-10/%3A?versionId=null 2022-09-12T18:50:03.574 DEBUG:boto:Data: 2022-09-12T18:50:03.575 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.575 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:03.576 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.576 DEBUG:boto:Params: {} 2022-09-12T18:50:03.577 DEBUG:boto:Token: None 2022-09-12T18:50:03.578 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/%3A?versionId=null 2022-09-12T18:50:03.578 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fxrMzY10VqjLQWHkO9A31AQ5R7s= 2022-09-12T18:50:03.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fxrMzY10VqjLQWHkO9A31AQ5R7s=', 'Content-Length': '0'} 2022-09-12T18:50:03.581 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf4e2f1feeba6461-00631f7f5b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.583 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:03.583 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:03.584 DEBUG:boto:path=/jdpiak-10/?prefix=%3A 2022-09-12T18:50:03.585 DEBUG:boto:auth_path=/jdpiak-10/?prefix=%3A 2022-09-12T18:50:03.585 DEBUG:boto:Method: GET 2022-09-12T18:50:03.586 DEBUG:boto:Path: /jdpiak-10/?prefix=%3A 2022-09-12T18:50:03.587 DEBUG:boto:Data: 2022-09-12T18:50:03.588 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.589 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:03.590 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.591 DEBUG:boto:Params: {} 2022-09-12T18:50:03.592 DEBUG:boto:Token: None 2022-09-12T18:50:03.592 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/ 2022-09-12T18:50:03.594 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5tQNLH26t1xV3d1kpelDI1MTmT4= 2022-09-12T18:50:03.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5tQNLH26t1xV3d1kpelDI1MTmT4=', 'Content-Length': '0'} 2022-09-12T18:50:03.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000693c12e6290adcab-00631f7f5b-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.651 DEBUG:boto:b'jdpiak-10:1000false:2022-09-12T18:49:46.939Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:03.653 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:03.653 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:03.654 DEBUG:boto:path=/jdpiak-10/?prefix=%3A 2022-09-12T18:50:03.654 DEBUG:boto:auth_path=/jdpiak-10/?prefix=%3A 2022-09-12T18:50:03.655 DEBUG:boto:Method: GET 2022-09-12T18:50:03.655 DEBUG:boto:Path: /jdpiak-10/?prefix=%3A 2022-09-12T18:50:03.656 DEBUG:boto:Data: 2022-09-12T18:50:03.656 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.657 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:03.658 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.658 DEBUG:boto:Params: {} 2022-09-12T18:50:03.659 DEBUG:boto:Token: None 2022-09-12T18:50:03.659 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/ 2022-09-12T18:50:03.660 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5tQNLH26t1xV3d1kpelDI1MTmT4= 2022-09-12T18:50:03.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5tQNLH26t1xV3d1kpelDI1MTmT4=', 'Content-Length': '0'} 2022-09-12T18:50:03.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070a6c916691e5fcc-00631f7f5b-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.664 DEBUG:boto:b'jdpiak-10:1000false:2022-09-12T18:49:46.939Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:03.665 DEBUG:boto:path=/jdpiak-10/%3A 2022-09-12T18:50:03.665 DEBUG:boto:auth_path=/jdpiak-10/%3A 2022-09-12T18:50:03.666 DEBUG:boto:Method: HEAD 2022-09-12T18:50:03.666 DEBUG:boto:Path: /jdpiak-10/%3A 2022-09-12T18:50:03.667 DEBUG:boto:Data: 2022-09-12T18:50:03.667 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.668 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:03.668 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.669 DEBUG:boto:Params: {} 2022-09-12T18:50:03.669 DEBUG:boto:Token: None 2022-09-12T18:50:03.670 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/%3A 2022-09-12T18:50:03.670 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XX4rAVhyqs+Ev8ndYE/3pFCjRy0= 2022-09-12T18:50:03.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XX4rAVhyqs+Ev8ndYE/3pFCjRy0=', 'Content-Length': '0'} 2022-09-12T18:50:03.673 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc977412042f91d6-00631f7f5b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.674 DEBUG:boto:path=/jdpiak-10/%3A 2022-09-12T18:50:03.674 DEBUG:boto:auth_path=/jdpiak-10/%3A 2022-09-12T18:50:03.675 DEBUG:boto:Method: HEAD 2022-09-12T18:50:03.675 DEBUG:boto:Path: /jdpiak-10/%3A 2022-09-12T18:50:03.676 DEBUG:boto:Data: 2022-09-12T18:50:03.676 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.677 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:03.677 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.678 DEBUG:boto:Params: {} 2022-09-12T18:50:03.678 DEBUG:boto:Token: None 2022-09-12T18:50:03.679 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/%3A 2022-09-12T18:50:03.679 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XX4rAVhyqs+Ev8ndYE/3pFCjRy0= 2022-09-12T18:50:03.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XX4rAVhyqs+Ev8ndYE/3pFCjRy0=', 'Content-Length': '0'} 2022-09-12T18:50:03.682 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9fe49b8c33d7463-00631f7f5b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-12T18:50:03.684 DEBUG:boto:path=/jdpiak-10/%3A 2022-09-12T18:50:03.684 DEBUG:boto:auth_path=/jdpiak-10/%3A 2022-09-12T18:50:03.685 DEBUG:boto:Method: GET 2022-09-12T18:50:03.685 DEBUG:boto:Path: /jdpiak-10/%3A 2022-09-12T18:50:03.686 DEBUG:boto:Data: 2022-09-12T18:50:03.686 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.687 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:03.687 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.688 DEBUG:boto:Params: {} 2022-09-12T18:50:03.688 DEBUG:boto:Token: None 2022-09-12T18:50:03.689 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/%3A 2022-09-12T18:50:03.690 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Y3IucJ+pnLT0t5W6TJQmEVfQcA= 2022-09-12T18:50:03.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Y3IucJ+pnLT0t5W6TJQmEVfQcA=', 'Content-Length': '0'} 2022-09-12T18:50:03.692 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019146f6834eb0915-00631f7f5b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.693 DEBUG:boto:path=/jdpiak-10/%3A 2022-09-12T18:50:03.694 DEBUG:boto:auth_path=/jdpiak-10/%3A 2022-09-12T18:50:03.694 DEBUG:boto:Method: GET 2022-09-12T18:50:03.695 DEBUG:boto:Path: /jdpiak-10/%3A 2022-09-12T18:50:03.695 DEBUG:boto:Data: 2022-09-12T18:50:03.696 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.696 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:03.697 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.697 DEBUG:boto:Params: {} 2022-09-12T18:50:03.698 DEBUG:boto:Token: None 2022-09-12T18:50:03.699 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/%3A 2022-09-12T18:50:03.699 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Y3IucJ+pnLT0t5W6TJQmEVfQcA= 2022-09-12T18:50:03.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Y3IucJ+pnLT0t5W6TJQmEVfQcA=', 'Content-Length': '0'} 2022-09-12T18:50:03.702 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7cc68319a6b32b5-00631f7f5b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.746 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:03.747 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:03.748 DEBUG:boto:path=/jdpiak-10/?prefix=%3A 2022-09-12T18:50:03.748 DEBUG:boto:auth_path=/jdpiak-10/?prefix=%3A 2022-09-12T18:50:03.749 DEBUG:boto:Method: GET 2022-09-12T18:50:03.749 DEBUG:boto:Path: /jdpiak-10/?prefix=%3A 2022-09-12T18:50:03.750 DEBUG:boto:Data: 2022-09-12T18:50:03.750 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.751 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:03.751 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.752 DEBUG:boto:Params: {} 2022-09-12T18:50:03.752 DEBUG:boto:Token: None 2022-09-12T18:50:03.753 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/ 2022-09-12T18:50:03.754 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5tQNLH26t1xV3d1kpelDI1MTmT4= 2022-09-12T18:50:03.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5tQNLH26t1xV3d1kpelDI1MTmT4=', 'Content-Length': '0'} 2022-09-12T18:50:03.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000216aabb63eb46a2e-00631f7f5b-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.758 DEBUG:boto:b'jdpiak-10:1000false:2022-09-12T18:49:46.939Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:03.759 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:03.759 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:03.760 DEBUG:boto:path=/jdpiak-10/?prefix=%3A 2022-09-12T18:50:03.760 DEBUG:boto:auth_path=/jdpiak-10/?prefix=%3A 2022-09-12T18:50:03.761 DEBUG:boto:Method: GET 2022-09-12T18:50:03.761 DEBUG:boto:Path: /jdpiak-10/?prefix=%3A 2022-09-12T18:50:03.762 DEBUG:boto:Data: 2022-09-12T18:50:03.762 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.763 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:03.764 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.764 DEBUG:boto:Params: {} 2022-09-12T18:50:03.765 DEBUG:boto:Token: None 2022-09-12T18:50:03.766 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/ 2022-09-12T18:50:03.767 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5tQNLH26t1xV3d1kpelDI1MTmT4= 2022-09-12T18:50:03.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5tQNLH26t1xV3d1kpelDI1MTmT4=', 'Content-Length': '0'} 2022-09-12T18:50:03.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000349680f0378065b-00631f7f5b-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.772 DEBUG:boto:b'jdpiak-10:1000false:2022-09-12T18:49:46.939Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:03.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-12T18:50:03.773 DEBUG:boto:path=/jdpiak-10/_myobj 2022-09-12T18:50:03.774 DEBUG:boto:auth_path=/jdpiak-10/_myobj 2022-09-12T18:50:03.774 DEBUG:boto:path=/jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:03.775 DEBUG:boto:auth_path=/jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:03.776 DEBUG:boto:Method: GET 2022-09-12T18:50:03.776 DEBUG:boto:Path: /jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:03.777 DEBUG:boto:Data: 2022-09-12T18:50:03.777 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.778 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:03.778 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.779 DEBUG:boto:Params: {} 2022-09-12T18:50:03.779 DEBUG:boto:Token: None 2022-09-12T18:50:03.780 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:03.781 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q8aBms1jZcpKBJkPgcKrYa9tlBM= 2022-09-12T18:50:03.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q8aBms1jZcpKBJkPgcKrYa9tlBM=', 'Content-Length': '0'} 2022-09-12T18:50:03.784 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bbc5ab546dd291b8-00631f7f5b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.784 DEBUG:boto:path=/jdpiak-10/_myobj 2022-09-12T18:50:03.785 DEBUG:boto:auth_path=/jdpiak-10/_myobj 2022-09-12T18:50:03.786 DEBUG:boto:path=/jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:03.787 DEBUG:boto:auth_path=/jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:03.787 DEBUG:boto:Method: GET 2022-09-12T18:50:03.788 DEBUG:boto:Path: /jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:03.788 DEBUG:boto:Data: 2022-09-12T18:50:03.789 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.789 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:03.790 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.790 DEBUG:boto:Params: {} 2022-09-12T18:50:03.791 DEBUG:boto:Token: None 2022-09-12T18:50:03.791 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:03.792 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q8aBms1jZcpKBJkPgcKrYa9tlBM= 2022-09-12T18:50:03.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q8aBms1jZcpKBJkPgcKrYa9tlBM=', 'Content-Length': '0'} 2022-09-12T18:50:03.796 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6d38160f45bc7f1-00631f7f5b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.797 DEBUG:boto:path=/jdpiak-10/_myobj 2022-09-12T18:50:03.798 DEBUG:boto:auth_path=/jdpiak-10/_myobj 2022-09-12T18:50:03.799 DEBUG:boto:path=/jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:03.799 DEBUG:boto:auth_path=/jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:03.800 DEBUG:boto:Method: HEAD 2022-09-12T18:50:03.800 DEBUG:boto:Path: /jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:03.801 DEBUG:boto:Data: 2022-09-12T18:50:03.801 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.802 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:03.802 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.803 DEBUG:boto:Params: {} 2022-09-12T18:50:03.804 DEBUG:boto:Token: None 2022-09-12T18:50:03.804 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:03.805 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y8uTQFTWfeBdrukdNsngoA2DZYA= 2022-09-12T18:50:03.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y8uTQFTWfeBdrukdNsngoA2DZYA=', 'Content-Length': '0'} 2022-09-12T18:50:03.808 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b23f5e61739ff0b5-00631f7f5b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.808 DEBUG:boto:path=/jdpiak-10/_myobj 2022-09-12T18:50:03.809 DEBUG:boto:auth_path=/jdpiak-10/_myobj 2022-09-12T18:50:03.809 DEBUG:boto:path=/jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:03.810 DEBUG:boto:auth_path=/jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:03.810 DEBUG:boto:Method: HEAD 2022-09-12T18:50:03.811 DEBUG:boto:Path: /jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:03.811 DEBUG:boto:Data: 2022-09-12T18:50:03.812 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.812 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:03.813 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.813 DEBUG:boto:Params: {} 2022-09-12T18:50:03.814 DEBUG:boto:Token: None 2022-09-12T18:50:03.814 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:03.815 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y8uTQFTWfeBdrukdNsngoA2DZYA= 2022-09-12T18:50:03.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y8uTQFTWfeBdrukdNsngoA2DZYA=', 'Content-Length': '0'} 2022-09-12T18:50:03.818 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000579dadd7038e3cb-00631f7f5b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-12T18:50:03.819 DEBUG:boto:path=/jdpiak-10/_myobj 2022-09-12T18:50:03.819 DEBUG:boto:auth_path=/jdpiak-10/_myobj 2022-09-12T18:50:03.820 DEBUG:boto:path=/jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:03.821 DEBUG:boto:auth_path=/jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:03.821 DEBUG:boto:Method: GET 2022-09-12T18:50:03.822 DEBUG:boto:Path: /jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:03.822 DEBUG:boto:Data: 2022-09-12T18:50:03.823 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.823 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:03.824 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.824 DEBUG:boto:Params: {} 2022-09-12T18:50:03.825 DEBUG:boto:Token: None 2022-09-12T18:50:03.825 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:03.826 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q8aBms1jZcpKBJkPgcKrYa9tlBM= 2022-09-12T18:50:03.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q8aBms1jZcpKBJkPgcKrYa9tlBM=', 'Content-Length': '0'} 2022-09-12T18:50:03.829 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae7d52a591875482-00631f7f5b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.830 DEBUG:boto:path=/jdpiak-10/_myobj 2022-09-12T18:50:03.830 DEBUG:boto:auth_path=/jdpiak-10/_myobj 2022-09-12T18:50:03.831 DEBUG:boto:path=/jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:03.831 DEBUG:boto:auth_path=/jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:03.832 DEBUG:boto:Method: GET 2022-09-12T18:50:03.833 DEBUG:boto:Path: /jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:03.833 DEBUG:boto:Data: 2022-09-12T18:50:03.834 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.834 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:03.835 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.835 DEBUG:boto:Params: {} 2022-09-12T18:50:03.836 DEBUG:boto:Token: None 2022-09-12T18:50:03.836 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:03.837 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q8aBms1jZcpKBJkPgcKrYa9tlBM= 2022-09-12T18:50:03.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q8aBms1jZcpKBJkPgcKrYa9tlBM=', 'Content-Length': '0'} 2022-09-12T18:50:03.840 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032421fa0dd4d3e63-00631f7f5b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.841 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:03.841 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:03.842 DEBUG:boto:path=/jdpiak-10/?prefix=_myobj 2022-09-12T18:50:03.842 DEBUG:boto:auth_path=/jdpiak-10/?prefix=_myobj 2022-09-12T18:50:03.843 DEBUG:boto:Method: GET 2022-09-12T18:50:03.843 DEBUG:boto:Path: /jdpiak-10/?prefix=_myobj 2022-09-12T18:50:03.844 DEBUG:boto:Data: 2022-09-12T18:50:03.844 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.845 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:03.845 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.846 DEBUG:boto:Params: {} 2022-09-12T18:50:03.847 DEBUG:boto:Token: None 2022-09-12T18:50:03.847 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/ 2022-09-12T18:50:03.848 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5tQNLH26t1xV3d1kpelDI1MTmT4= 2022-09-12T18:50:03.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5tQNLH26t1xV3d1kpelDI1MTmT4=', 'Content-Length': '0'} 2022-09-12T18:50:03.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f984afe30676221-00631f7f5b-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.852 DEBUG:boto:b'jdpiak-10_myobj1000false_myobj2022-09-12T18:49:46.878Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:03.852 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:03.853 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:03.854 DEBUG:boto:path=/jdpiak-10/?prefix=_myobj 2022-09-12T18:50:03.854 DEBUG:boto:auth_path=/jdpiak-10/?prefix=_myobj 2022-09-12T18:50:03.855 DEBUG:boto:Method: GET 2022-09-12T18:50:03.855 DEBUG:boto:Path: /jdpiak-10/?prefix=_myobj 2022-09-12T18:50:03.856 DEBUG:boto:Data: 2022-09-12T18:50:03.856 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.857 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:03.857 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.858 DEBUG:boto:Params: {} 2022-09-12T18:50:03.858 DEBUG:boto:Token: None 2022-09-12T18:50:03.859 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/ 2022-09-12T18:50:03.859 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5tQNLH26t1xV3d1kpelDI1MTmT4= 2022-09-12T18:50:03.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5tQNLH26t1xV3d1kpelDI1MTmT4=', 'Content-Length': '0'} 2022-09-12T18:50:03.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c06817ace236ae4-00631f7f5b-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.864 DEBUG:boto:b'jdpiak-10_myobj1000false_myobj2022-09-12T18:49:46.878Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:03.864 DEBUG:boto:path=/jdpiak-10/_myobj 2022-09-12T18:50:03.865 DEBUG:boto:auth_path=/jdpiak-10/_myobj 2022-09-12T18:50:03.865 DEBUG:boto:Method: HEAD 2022-09-12T18:50:03.866 DEBUG:boto:Path: /jdpiak-10/_myobj 2022-09-12T18:50:03.866 DEBUG:boto:Data: 2022-09-12T18:50:03.867 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.867 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:03.868 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.868 DEBUG:boto:Params: {} 2022-09-12T18:50:03.869 DEBUG:boto:Token: None 2022-09-12T18:50:03.870 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/_myobj 2022-09-12T18:50:03.870 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KMpon6aOE4ORQopTiCl9rNLZrEY= 2022-09-12T18:50:03.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KMpon6aOE4ORQopTiCl9rNLZrEY=', 'Content-Length': '0'} 2022-09-12T18:50:03.873 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c67eb65137ef8281-00631f7f5b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.874 DEBUG:boto:path=/jdpiak-10/_myobj 2022-09-12T18:50:03.874 DEBUG:boto:auth_path=/jdpiak-10/_myobj 2022-09-12T18:50:03.875 DEBUG:boto:Method: HEAD 2022-09-12T18:50:03.875 DEBUG:boto:Path: /jdpiak-10/_myobj 2022-09-12T18:50:03.876 DEBUG:boto:Data: 2022-09-12T18:50:03.876 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.877 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:03.877 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.878 DEBUG:boto:Params: {} 2022-09-12T18:50:03.878 DEBUG:boto:Token: None 2022-09-12T18:50:03.879 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/_myobj 2022-09-12T18:50:03.880 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KMpon6aOE4ORQopTiCl9rNLZrEY= 2022-09-12T18:50:03.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KMpon6aOE4ORQopTiCl9rNLZrEY=', 'Content-Length': '0'} 2022-09-12T18:50:03.882 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3ebefe09df5edb9-00631f7f5b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-12T18:50:03.883 DEBUG:boto:path=/jdpiak-10/_myobj 2022-09-12T18:50:03.884 DEBUG:boto:auth_path=/jdpiak-10/_myobj 2022-09-12T18:50:03.884 DEBUG:boto:Method: GET 2022-09-12T18:50:03.885 DEBUG:boto:Path: /jdpiak-10/_myobj 2022-09-12T18:50:03.885 DEBUG:boto:Data: 2022-09-12T18:50:03.886 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.886 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:03.887 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.887 DEBUG:boto:Params: {} 2022-09-12T18:50:03.888 DEBUG:boto:Token: None 2022-09-12T18:50:03.888 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/_myobj 2022-09-12T18:50:03.889 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4MrzQrtd9xCgTasAZY1IM3+kHsU= 2022-09-12T18:50:03.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4MrzQrtd9xCgTasAZY1IM3+kHsU=', 'Content-Length': '0'} 2022-09-12T18:50:03.892 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000dc61b635b48250a-00631f7f5b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.892 DEBUG:boto:path=/jdpiak-10/_myobj 2022-09-12T18:50:03.893 DEBUG:boto:auth_path=/jdpiak-10/_myobj 2022-09-12T18:50:03.894 DEBUG:boto:Method: GET 2022-09-12T18:50:03.894 DEBUG:boto:Path: /jdpiak-10/_myobj 2022-09-12T18:50:03.895 DEBUG:boto:Data: 2022-09-12T18:50:03.895 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.896 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:03.896 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.897 DEBUG:boto:Params: {} 2022-09-12T18:50:03.897 DEBUG:boto:Token: None 2022-09-12T18:50:03.898 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/_myobj 2022-09-12T18:50:03.898 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4MrzQrtd9xCgTasAZY1IM3+kHsU= 2022-09-12T18:50:03.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4MrzQrtd9xCgTasAZY1IM3+kHsU=', 'Content-Length': '0'} 2022-09-12T18:50:03.901 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba9aa82f14c7772e-00631f7f5b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.942 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:03.943 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:03.944 DEBUG:boto:path=/jdpiak-10/?prefix=_myobj 2022-09-12T18:50:03.945 DEBUG:boto:auth_path=/jdpiak-10/?prefix=_myobj 2022-09-12T18:50:03.946 DEBUG:boto:Method: GET 2022-09-12T18:50:03.947 DEBUG:boto:Path: /jdpiak-10/?prefix=_myobj 2022-09-12T18:50:03.948 DEBUG:boto:Data: 2022-09-12T18:50:03.949 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.950 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:03.951 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.952 DEBUG:boto:Params: {} 2022-09-12T18:50:03.954 DEBUG:boto:Token: None 2022-09-12T18:50:03.955 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/ 2022-09-12T18:50:03.955 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5tQNLH26t1xV3d1kpelDI1MTmT4= 2022-09-12T18:50:03.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5tQNLH26t1xV3d1kpelDI1MTmT4=', 'Content-Length': '0'} 2022-09-12T18:50:03.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb5064a0c8bf4d75-00631f7f5b-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.960 DEBUG:boto:b'jdpiak-10_myobj1000false_myobj2022-09-12T18:49:46.878Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:03.961 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:03.961 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:03.962 DEBUG:boto:path=/jdpiak-10/?prefix=_myobj 2022-09-12T18:50:03.962 DEBUG:boto:auth_path=/jdpiak-10/?prefix=_myobj 2022-09-12T18:50:03.963 DEBUG:boto:Method: GET 2022-09-12T18:50:03.964 DEBUG:boto:Path: /jdpiak-10/?prefix=_myobj 2022-09-12T18:50:03.965 DEBUG:boto:Data: 2022-09-12T18:50:03.966 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.967 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:03.967 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.968 DEBUG:boto:Params: {} 2022-09-12T18:50:03.969 DEBUG:boto:Token: None 2022-09-12T18:50:03.971 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/ 2022-09-12T18:50:03.972 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5tQNLH26t1xV3d1kpelDI1MTmT4= 2022-09-12T18:50:03.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5tQNLH26t1xV3d1kpelDI1MTmT4=', 'Content-Length': '0'} 2022-09-12T18:50:03.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000149263c25b5db479-00631f7f5b-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:03.977 DEBUG:boto:b'jdpiak-10_myobj1000false_myobj2022-09-12T18:49:46.878Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:03.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-12T18:50:03.979 DEBUG:boto:path=/jdpiak-10/myobj 2022-09-12T18:50:03.980 DEBUG:boto:auth_path=/jdpiak-10/myobj 2022-09-12T18:50:03.980 DEBUG:boto:path=/jdpiak-10/myobj?versionId=null 2022-09-12T18:50:03.981 DEBUG:boto:auth_path=/jdpiak-10/myobj?versionId=null 2022-09-12T18:50:03.981 DEBUG:boto:Method: GET 2022-09-12T18:50:03.982 DEBUG:boto:Path: /jdpiak-10/myobj?versionId=null 2022-09-12T18:50:03.982 DEBUG:boto:Data: 2022-09-12T18:50:03.983 DEBUG:boto:Headers: {} 2022-09-12T18:50:03.983 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:03.984 DEBUG:boto:Port: 8000 2022-09-12T18:50:03.984 DEBUG:boto:Params: {} 2022-09-12T18:50:03.985 DEBUG:boto:Token: None 2022-09-12T18:50:03.986 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:03 GMT /jdpiak-10/myobj?versionId=null 2022-09-12T18:50:03.986 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:w8SIW8oLJ+kNR/dOrHGdFUL/4s0= 2022-09-12T18:50:03.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:w8SIW8oLJ+kNR/dOrHGdFUL/4s0=', 'Content-Length': '0'} 2022-09-12T18:50:03.989 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005aea7641424b1749-00631f7f5b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.031 DEBUG:boto:path=/jdpiak-10/myobj 2022-09-12T18:50:04.032 DEBUG:boto:auth_path=/jdpiak-10/myobj 2022-09-12T18:50:04.033 DEBUG:boto:path=/jdpiak-10/myobj?versionId=null 2022-09-12T18:50:04.034 DEBUG:boto:auth_path=/jdpiak-10/myobj?versionId=null 2022-09-12T18:50:04.034 DEBUG:boto:Method: GET 2022-09-12T18:50:04.035 DEBUG:boto:Path: /jdpiak-10/myobj?versionId=null 2022-09-12T18:50:04.035 DEBUG:boto:Data: 2022-09-12T18:50:04.036 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.036 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.037 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.038 DEBUG:boto:Params: {} 2022-09-12T18:50:04.039 DEBUG:boto:Token: None 2022-09-12T18:50:04.040 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/myobj?versionId=null 2022-09-12T18:50:04.041 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iufkYLviXACumY8icsXyz4QNZbY= 2022-09-12T18:50:04.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iufkYLviXACumY8icsXyz4QNZbY=', 'Content-Length': '0'} 2022-09-12T18:50:04.044 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f56903212cf8da0c-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.044 DEBUG:boto:path=/jdpiak-10/myobj 2022-09-12T18:50:04.045 DEBUG:boto:auth_path=/jdpiak-10/myobj 2022-09-12T18:50:04.045 DEBUG:boto:path=/jdpiak-10/myobj?versionId=null 2022-09-12T18:50:04.046 DEBUG:boto:auth_path=/jdpiak-10/myobj?versionId=null 2022-09-12T18:50:04.047 DEBUG:boto:Method: HEAD 2022-09-12T18:50:04.047 DEBUG:boto:Path: /jdpiak-10/myobj?versionId=null 2022-09-12T18:50:04.048 DEBUG:boto:Data: 2022-09-12T18:50:04.048 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.049 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:04.049 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.050 DEBUG:boto:Params: {} 2022-09-12T18:50:04.051 DEBUG:boto:Token: None 2022-09-12T18:50:04.052 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/myobj?versionId=null 2022-09-12T18:50:04.053 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX://SGImoOhbd+YFCOpE4DCt7NfMA= 2022-09-12T18:50:04.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX://SGImoOhbd+YFCOpE4DCt7NfMA=', 'Content-Length': '0'} 2022-09-12T18:50:04.057 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a060144d062bf6f3-00631f7f5c-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.058 DEBUG:boto:path=/jdpiak-10/myobj 2022-09-12T18:50:04.059 DEBUG:boto:auth_path=/jdpiak-10/myobj 2022-09-12T18:50:04.060 DEBUG:boto:path=/jdpiak-10/myobj?versionId=null 2022-09-12T18:50:04.060 DEBUG:boto:auth_path=/jdpiak-10/myobj?versionId=null 2022-09-12T18:50:04.061 DEBUG:boto:Method: HEAD 2022-09-12T18:50:04.061 DEBUG:boto:Path: /jdpiak-10/myobj?versionId=null 2022-09-12T18:50:04.062 DEBUG:boto:Data: 2022-09-12T18:50:04.062 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.063 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.064 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.065 DEBUG:boto:Params: {} 2022-09-12T18:50:04.065 DEBUG:boto:Token: None 2022-09-12T18:50:04.066 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/myobj?versionId=null 2022-09-12T18:50:04.067 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX://SGImoOhbd+YFCOpE4DCt7NfMA= 2022-09-12T18:50:04.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX://SGImoOhbd+YFCOpE4DCt7NfMA=', 'Content-Length': '0'} 2022-09-12T18:50:04.070 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec8409e4ef3ecfef-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-12T18:50:04.071 DEBUG:boto:path=/jdpiak-10/myobj 2022-09-12T18:50:04.072 DEBUG:boto:auth_path=/jdpiak-10/myobj 2022-09-12T18:50:04.073 DEBUG:boto:path=/jdpiak-10/myobj?versionId=null 2022-09-12T18:50:04.074 DEBUG:boto:auth_path=/jdpiak-10/myobj?versionId=null 2022-09-12T18:50:04.075 DEBUG:boto:Method: GET 2022-09-12T18:50:04.075 DEBUG:boto:Path: /jdpiak-10/myobj?versionId=null 2022-09-12T18:50:04.076 DEBUG:boto:Data: 2022-09-12T18:50:04.076 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.077 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:04.078 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.078 DEBUG:boto:Params: {} 2022-09-12T18:50:04.079 DEBUG:boto:Token: None 2022-09-12T18:50:04.079 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/myobj?versionId=null 2022-09-12T18:50:04.080 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iufkYLviXACumY8icsXyz4QNZbY= 2022-09-12T18:50:04.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iufkYLviXACumY8icsXyz4QNZbY=', 'Content-Length': '0'} 2022-09-12T18:50:04.083 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a466fc0e38dd674-00631f7f5c-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.084 DEBUG:boto:path=/jdpiak-10/myobj 2022-09-12T18:50:04.084 DEBUG:boto:auth_path=/jdpiak-10/myobj 2022-09-12T18:50:04.085 DEBUG:boto:path=/jdpiak-10/myobj?versionId=null 2022-09-12T18:50:04.085 DEBUG:boto:auth_path=/jdpiak-10/myobj?versionId=null 2022-09-12T18:50:04.086 DEBUG:boto:Method: GET 2022-09-12T18:50:04.087 DEBUG:boto:Path: /jdpiak-10/myobj?versionId=null 2022-09-12T18:50:04.087 DEBUG:boto:Data: 2022-09-12T18:50:04.088 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.088 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.089 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.089 DEBUG:boto:Params: {} 2022-09-12T18:50:04.090 DEBUG:boto:Token: None 2022-09-12T18:50:04.090 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/myobj?versionId=null 2022-09-12T18:50:04.091 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iufkYLviXACumY8icsXyz4QNZbY= 2022-09-12T18:50:04.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iufkYLviXACumY8icsXyz4QNZbY=', 'Content-Length': '0'} 2022-09-12T18:50:04.094 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000511cbc9a26d2ac4f-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.095 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:04.095 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:04.096 DEBUG:boto:path=/jdpiak-10/?prefix=myobj 2022-09-12T18:50:04.096 DEBUG:boto:auth_path=/jdpiak-10/?prefix=myobj 2022-09-12T18:50:04.097 DEBUG:boto:Method: GET 2022-09-12T18:50:04.097 DEBUG:boto:Path: /jdpiak-10/?prefix=myobj 2022-09-12T18:50:04.098 DEBUG:boto:Data: 2022-09-12T18:50:04.098 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.099 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:04.100 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.100 DEBUG:boto:Params: {} 2022-09-12T18:50:04.101 DEBUG:boto:Token: None 2022-09-12T18:50:04.101 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/ 2022-09-12T18:50:04.102 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hDDUcNRlEDmaMg6EFa/fpwenFzQ= 2022-09-12T18:50:04.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hDDUcNRlEDmaMg6EFa/fpwenFzQ=', 'Content-Length': '0'} 2022-09-12T18:50:04.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009112e6ef1ba56e6f-00631f7f5c-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.106 DEBUG:boto:b'jdpiak-10myobj1000falsemyobj2022-09-12T18:49:46.847Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:04.107 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:04.107 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:04.108 DEBUG:boto:path=/jdpiak-10/?prefix=myobj 2022-09-12T18:50:04.109 DEBUG:boto:auth_path=/jdpiak-10/?prefix=myobj 2022-09-12T18:50:04.109 DEBUG:boto:Method: GET 2022-09-12T18:50:04.110 DEBUG:boto:Path: /jdpiak-10/?prefix=myobj 2022-09-12T18:50:04.110 DEBUG:boto:Data: 2022-09-12T18:50:04.111 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.112 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.112 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.113 DEBUG:boto:Params: {} 2022-09-12T18:50:04.113 DEBUG:boto:Token: None 2022-09-12T18:50:04.114 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/ 2022-09-12T18:50:04.115 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hDDUcNRlEDmaMg6EFa/fpwenFzQ= 2022-09-12T18:50:04.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hDDUcNRlEDmaMg6EFa/fpwenFzQ=', 'Content-Length': '0'} 2022-09-12T18:50:04.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e0fe7cc9438b1cf-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.162 DEBUG:boto:b'jdpiak-10myobj1000falsemyobj2022-09-12T18:49:46.847Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:04.163 DEBUG:boto:path=/jdpiak-10/myobj 2022-09-12T18:50:04.164 DEBUG:boto:auth_path=/jdpiak-10/myobj 2022-09-12T18:50:04.164 DEBUG:boto:Method: HEAD 2022-09-12T18:50:04.165 DEBUG:boto:Path: /jdpiak-10/myobj 2022-09-12T18:50:04.165 DEBUG:boto:Data: 2022-09-12T18:50:04.166 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.166 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:04.167 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.168 DEBUG:boto:Params: {} 2022-09-12T18:50:04.168 DEBUG:boto:Token: None 2022-09-12T18:50:04.169 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/myobj 2022-09-12T18:50:04.170 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0/4JfxyaPSs63xG2ocIBAtj8n7Y= 2022-09-12T18:50:04.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0/4JfxyaPSs63xG2ocIBAtj8n7Y=', 'Content-Length': '0'} 2022-09-12T18:50:04.173 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bbc2c3907062470d-00631f7f5c-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.173 DEBUG:boto:path=/jdpiak-10/myobj 2022-09-12T18:50:04.174 DEBUG:boto:auth_path=/jdpiak-10/myobj 2022-09-12T18:50:04.175 DEBUG:boto:Method: HEAD 2022-09-12T18:50:04.175 DEBUG:boto:Path: /jdpiak-10/myobj 2022-09-12T18:50:04.176 DEBUG:boto:Data: 2022-09-12T18:50:04.176 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.177 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.177 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.178 DEBUG:boto:Params: {} 2022-09-12T18:50:04.178 DEBUG:boto:Token: None 2022-09-12T18:50:04.179 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/myobj 2022-09-12T18:50:04.180 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0/4JfxyaPSs63xG2ocIBAtj8n7Y= 2022-09-12T18:50:04.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0/4JfxyaPSs63xG2ocIBAtj8n7Y=', 'Content-Length': '0'} 2022-09-12T18:50:04.182 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7563da1007af2f7-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-12T18:50:04.184 DEBUG:boto:path=/jdpiak-10/myobj 2022-09-12T18:50:04.184 DEBUG:boto:auth_path=/jdpiak-10/myobj 2022-09-12T18:50:04.185 DEBUG:boto:Method: GET 2022-09-12T18:50:04.185 DEBUG:boto:Path: /jdpiak-10/myobj 2022-09-12T18:50:04.186 DEBUG:boto:Data: 2022-09-12T18:50:04.187 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.187 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:04.188 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.188 DEBUG:boto:Params: {} 2022-09-12T18:50:04.189 DEBUG:boto:Token: None 2022-09-12T18:50:04.190 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/myobj 2022-09-12T18:50:04.191 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9mAarG7dO6ZD2AoxUNZdri6iqGs= 2022-09-12T18:50:04.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9mAarG7dO6ZD2AoxUNZdri6iqGs=', 'Content-Length': '0'} 2022-09-12T18:50:04.194 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b05feb31911712e9-00631f7f5c-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.195 DEBUG:boto:path=/jdpiak-10/myobj 2022-09-12T18:50:04.196 DEBUG:boto:auth_path=/jdpiak-10/myobj 2022-09-12T18:50:04.196 DEBUG:boto:Method: GET 2022-09-12T18:50:04.197 DEBUG:boto:Path: /jdpiak-10/myobj 2022-09-12T18:50:04.198 DEBUG:boto:Data: 2022-09-12T18:50:04.198 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.199 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.199 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.200 DEBUG:boto:Params: {} 2022-09-12T18:50:04.201 DEBUG:boto:Token: None 2022-09-12T18:50:04.202 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/myobj 2022-09-12T18:50:04.203 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9mAarG7dO6ZD2AoxUNZdri6iqGs= 2022-09-12T18:50:04.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9mAarG7dO6ZD2AoxUNZdri6iqGs=', 'Content-Length': '0'} 2022-09-12T18:50:04.208 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5b45ba2b10a6239-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.209 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:04.210 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:04.211 DEBUG:boto:path=/jdpiak-10/?prefix=myobj 2022-09-12T18:50:04.212 DEBUG:boto:auth_path=/jdpiak-10/?prefix=myobj 2022-09-12T18:50:04.213 DEBUG:boto:Method: GET 2022-09-12T18:50:04.214 DEBUG:boto:Path: /jdpiak-10/?prefix=myobj 2022-09-12T18:50:04.215 DEBUG:boto:Data: 2022-09-12T18:50:04.215 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.216 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:50:04.217 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.218 DEBUG:boto:Params: {} 2022-09-12T18:50:04.219 DEBUG:boto:Token: None 2022-09-12T18:50:04.221 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/ 2022-09-12T18:50:04.222 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hDDUcNRlEDmaMg6EFa/fpwenFzQ= 2022-09-12T18:50:04.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hDDUcNRlEDmaMg6EFa/fpwenFzQ=', 'Content-Length': '0'} 2022-09-12T18:50:04.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1e2f9dfb17cf078-00631f7f5c-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.270 DEBUG:boto:b'jdpiak-10myobj1000falsemyobj2022-09-12T18:49:46.847Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:04.271 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:04.272 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:04.272 DEBUG:boto:path=/jdpiak-10/?prefix=myobj 2022-09-12T18:50:04.273 DEBUG:boto:auth_path=/jdpiak-10/?prefix=myobj 2022-09-12T18:50:04.273 DEBUG:boto:Method: GET 2022-09-12T18:50:04.274 DEBUG:boto:Path: /jdpiak-10/?prefix=myobj 2022-09-12T18:50:04.275 DEBUG:boto:Data: 2022-09-12T18:50:04.275 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.276 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.276 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.277 DEBUG:boto:Params: {} 2022-09-12T18:50:04.277 DEBUG:boto:Token: None 2022-09-12T18:50:04.278 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/ 2022-09-12T18:50:04.279 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hDDUcNRlEDmaMg6EFa/fpwenFzQ= 2022-09-12T18:50:04.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hDDUcNRlEDmaMg6EFa/fpwenFzQ=', 'Content-Length': '0'} 2022-09-12T18:50:04.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3b30fff72f61a0d-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.283 DEBUG:boto:b'jdpiak-10myobj1000falsemyobj2022-09-12T18:49:46.847Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:04.284 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-10 zones={test-zone2, test-zone1} 2022-09-12T18:50:04.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:50:04.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:50:04.286 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T18:50:04.351 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:50:04.352 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:50:04.398 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:50:04.395+0000 7fee04fae600 1 bucket sync caught up with source: 2022-09-12T18:50:04.398 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:50:04.398 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:50:04.399 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:50:04.395+0000 7fee04fae600 0 bucket checkpoint complete 2022-09-12T18:50:04.412 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-10 zones={test-zone3, test-zone1} 2022-09-12T18:50:04.414 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:04.415 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:04.417 DEBUG:boto:Method: HEAD 2022-09-12T18:50:04.419 DEBUG:boto:Path: /jdpiak-10/ 2022-09-12T18:50:04.419 DEBUG:boto:Data: 2022-09-12T18:50:04.420 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.421 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:04.422 DEBUG:boto:Port: 8001 2022-09-12T18:50:04.422 DEBUG:boto:Params: {} 2022-09-12T18:50:04.423 DEBUG:boto:Token: None 2022-09-12T18:50:04.423 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/ 2022-09-12T18:50:04.424 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uR68xDVbmkrMyU+F17I8ic0nvAA= 2022-09-12T18:50:04.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uR68xDVbmkrMyU+F17I8ic0nvAA=', 'Content-Length': '0'} 2022-09-12T18:50:04.427 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052aa599e111b1d0f-00631f7f5c-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.428 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:04.428 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:04.429 DEBUG:boto:Method: HEAD 2022-09-12T18:50:04.430 DEBUG:boto:Path: /jdpiak-10/ 2022-09-12T18:50:04.430 DEBUG:boto:Data: 2022-09-12T18:50:04.431 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.432 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.433 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.434 DEBUG:boto:Params: {} 2022-09-12T18:50:04.435 DEBUG:boto:Token: None 2022-09-12T18:50:04.436 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/ 2022-09-12T18:50:04.437 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uR68xDVbmkrMyU+F17I8ic0nvAA= 2022-09-12T18:50:04.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uR68xDVbmkrMyU+F17I8ic0nvAA=', 'Content-Length': '0'} 2022-09-12T18:50:04.441 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006007b42db36f0865-00631f7f5c-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.442 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T18:50:04.443 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:04.444 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:04.445 DEBUG:boto:path=/jdpiak-10/?versions&max-keys=999 2022-09-12T18:50:04.446 DEBUG:boto:auth_path=/jdpiak-10/?versions&max-keys=999 2022-09-12T18:50:04.447 DEBUG:boto:Method: GET 2022-09-12T18:50:04.448 DEBUG:boto:Path: /jdpiak-10/?versions&max-keys=999 2022-09-12T18:50:04.449 DEBUG:boto:Data: 2022-09-12T18:50:04.450 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.451 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:04.452 DEBUG:boto:Port: 8001 2022-09-12T18:50:04.453 DEBUG:boto:Params: {} 2022-09-12T18:50:04.453 DEBUG:boto:Token: None 2022-09-12T18:50:04.454 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/?versions 2022-09-12T18:50:04.455 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7XRf/PK0euN1r7z4vQ42AAnuhQA= 2022-09-12T18:50:04.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7XRf/PK0euN1r7z4vQ42AAnuhQA=', 'Content-Length': '0'} 2022-09-12T18:50:04.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059d170cea345eb90-00631f7f5c-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.459 DEBUG:boto:b'jdpiak-10999false&nulltrue2022-09-12T18:49:46.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-12T18:49:46.939Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-12T18:49:46.878Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-12T18:49:46.847Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:04.459 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-12T18:50:04.460 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-12T18:50:04.460 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-12T18:50:04.461 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-12T18:50:04.462 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T18:50:04.463 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:04.464 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:04.465 DEBUG:boto:path=/jdpiak-10/?versions&max-keys=999 2022-09-12T18:50:04.465 DEBUG:boto:auth_path=/jdpiak-10/?versions&max-keys=999 2022-09-12T18:50:04.467 DEBUG:boto:Method: GET 2022-09-12T18:50:04.467 DEBUG:boto:Path: /jdpiak-10/?versions&max-keys=999 2022-09-12T18:50:04.468 DEBUG:boto:Data: 2022-09-12T18:50:04.468 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.469 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.469 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.470 DEBUG:boto:Params: {} 2022-09-12T18:50:04.471 DEBUG:boto:Token: None 2022-09-12T18:50:04.471 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/?versions 2022-09-12T18:50:04.472 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7XRf/PK0euN1r7z4vQ42AAnuhQA= 2022-09-12T18:50:04.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7XRf/PK0euN1r7z4vQ42AAnuhQA=', 'Content-Length': '0'} 2022-09-12T18:50:04.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000788ffd5f08489e5e-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.476 DEBUG:boto:b'jdpiak-10999false&nulltrue2022-09-12T18:49:46.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-12T18:49:46.939Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-12T18:49:46.878Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-12T18:49:46.847Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:04.477 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-12T18:50:04.477 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-12T18:50:04.478 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-12T18:50:04.478 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-12T18:50:04.479 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:04.479 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:04.480 DEBUG:boto:path=/jdpiak-10/?versions&max-keys=999 2022-09-12T18:50:04.481 DEBUG:boto:auth_path=/jdpiak-10/?versions&max-keys=999 2022-09-12T18:50:04.481 DEBUG:boto:Method: GET 2022-09-12T18:50:04.482 DEBUG:boto:Path: /jdpiak-10/?versions&max-keys=999 2022-09-12T18:50:04.482 DEBUG:boto:Data: 2022-09-12T18:50:04.483 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.484 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:04.484 DEBUG:boto:Port: 8001 2022-09-12T18:50:04.485 DEBUG:boto:Params: {} 2022-09-12T18:50:04.485 DEBUG:boto:Token: None 2022-09-12T18:50:04.486 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/?versions 2022-09-12T18:50:04.487 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7XRf/PK0euN1r7z4vQ42AAnuhQA= 2022-09-12T18:50:04.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7XRf/PK0euN1r7z4vQ42AAnuhQA=', 'Content-Length': '0'} 2022-09-12T18:50:04.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cb6929d19cd0712-00631f7f5c-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.491 DEBUG:boto:b'jdpiak-10999false&nulltrue2022-09-12T18:49:46.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-12T18:49:46.939Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-12T18:49:46.878Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-12T18:49:46.847Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:04.492 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:04.493 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:04.493 DEBUG:boto:path=/jdpiak-10/?versions&max-keys=999 2022-09-12T18:50:04.494 DEBUG:boto:auth_path=/jdpiak-10/?versions&max-keys=999 2022-09-12T18:50:04.494 DEBUG:boto:Method: GET 2022-09-12T18:50:04.495 DEBUG:boto:Path: /jdpiak-10/?versions&max-keys=999 2022-09-12T18:50:04.495 DEBUG:boto:Data: 2022-09-12T18:50:04.496 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.496 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.497 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.497 DEBUG:boto:Params: {} 2022-09-12T18:50:04.498 DEBUG:boto:Token: None 2022-09-12T18:50:04.498 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/?versions 2022-09-12T18:50:04.499 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7XRf/PK0euN1r7z4vQ42AAnuhQA= 2022-09-12T18:50:04.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7XRf/PK0euN1r7z4vQ42AAnuhQA=', 'Content-Length': '0'} 2022-09-12T18:50:04.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c008a48beeb64849-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.503 DEBUG:boto:b'jdpiak-10999false&nulltrue2022-09-12T18:49:46.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-12T18:49:46.939Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-12T18:49:46.878Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-12T18:49:46.847Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:04.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-12T18:50:04.505 DEBUG:boto:path=/jdpiak-10/%26 2022-09-12T18:50:04.505 DEBUG:boto:auth_path=/jdpiak-10/%26 2022-09-12T18:50:04.506 DEBUG:boto:path=/jdpiak-10/%26?versionId=null 2022-09-12T18:50:04.507 DEBUG:boto:auth_path=/jdpiak-10/%26?versionId=null 2022-09-12T18:50:04.507 DEBUG:boto:Method: GET 2022-09-12T18:50:04.508 DEBUG:boto:Path: /jdpiak-10/%26?versionId=null 2022-09-12T18:50:04.508 DEBUG:boto:Data: 2022-09-12T18:50:04.509 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.510 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:04.510 DEBUG:boto:Port: 8001 2022-09-12T18:50:04.511 DEBUG:boto:Params: {} 2022-09-12T18:50:04.511 DEBUG:boto:Token: None 2022-09-12T18:50:04.512 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/%26?versionId=null 2022-09-12T18:50:04.513 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q0nktn9VAWaTuFonG6WxzBsdCAU= 2022-09-12T18:50:04.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q0nktn9VAWaTuFonG6WxzBsdCAU=', 'Content-Length': '0'} 2022-09-12T18:50:04.516 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a1d796841a70526-00631f7f5c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.558 DEBUG:boto:path=/jdpiak-10/%26 2022-09-12T18:50:04.559 DEBUG:boto:auth_path=/jdpiak-10/%26 2022-09-12T18:50:04.559 DEBUG:boto:path=/jdpiak-10/%26?versionId=null 2022-09-12T18:50:04.560 DEBUG:boto:auth_path=/jdpiak-10/%26?versionId=null 2022-09-12T18:50:04.560 DEBUG:boto:Method: GET 2022-09-12T18:50:04.561 DEBUG:boto:Path: /jdpiak-10/%26?versionId=null 2022-09-12T18:50:04.561 DEBUG:boto:Data: 2022-09-12T18:50:04.562 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.563 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.563 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.564 DEBUG:boto:Params: {} 2022-09-12T18:50:04.564 DEBUG:boto:Token: None 2022-09-12T18:50:04.565 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/%26?versionId=null 2022-09-12T18:50:04.566 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q0nktn9VAWaTuFonG6WxzBsdCAU= 2022-09-12T18:50:04.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q0nktn9VAWaTuFonG6WxzBsdCAU=', 'Content-Length': '0'} 2022-09-12T18:50:04.569 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000297419c464bfae1e-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.570 DEBUG:boto:path=/jdpiak-10/%26 2022-09-12T18:50:04.571 DEBUG:boto:auth_path=/jdpiak-10/%26 2022-09-12T18:50:04.571 DEBUG:boto:path=/jdpiak-10/%26?versionId=null 2022-09-12T18:50:04.572 DEBUG:boto:auth_path=/jdpiak-10/%26?versionId=null 2022-09-12T18:50:04.573 DEBUG:boto:Method: HEAD 2022-09-12T18:50:04.573 DEBUG:boto:Path: /jdpiak-10/%26?versionId=null 2022-09-12T18:50:04.574 DEBUG:boto:Data: 2022-09-12T18:50:04.574 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.575 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:04.576 DEBUG:boto:Port: 8001 2022-09-12T18:50:04.576 DEBUG:boto:Params: {} 2022-09-12T18:50:04.577 DEBUG:boto:Token: None 2022-09-12T18:50:04.578 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/%26?versionId=null 2022-09-12T18:50:04.579 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:utKPyb+3CPPzLiJOT1Sa9rRDkUk= 2022-09-12T18:50:04.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:utKPyb+3CPPzLiJOT1Sa9rRDkUk=', 'Content-Length': '0'} 2022-09-12T18:50:04.583 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099decae07c39efc4-00631f7f5c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.584 DEBUG:boto:path=/jdpiak-10/%26 2022-09-12T18:50:04.585 DEBUG:boto:auth_path=/jdpiak-10/%26 2022-09-12T18:50:04.586 DEBUG:boto:path=/jdpiak-10/%26?versionId=null 2022-09-12T18:50:04.587 DEBUG:boto:auth_path=/jdpiak-10/%26?versionId=null 2022-09-12T18:50:04.587 DEBUG:boto:Method: HEAD 2022-09-12T18:50:04.588 DEBUG:boto:Path: /jdpiak-10/%26?versionId=null 2022-09-12T18:50:04.589 DEBUG:boto:Data: 2022-09-12T18:50:04.590 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.591 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.591 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.592 DEBUG:boto:Params: {} 2022-09-12T18:50:04.593 DEBUG:boto:Token: None 2022-09-12T18:50:04.594 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/%26?versionId=null 2022-09-12T18:50:04.595 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:utKPyb+3CPPzLiJOT1Sa9rRDkUk= 2022-09-12T18:50:04.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:utKPyb+3CPPzLiJOT1Sa9rRDkUk=', 'Content-Length': '0'} 2022-09-12T18:50:04.599 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000626a43acd35a828d-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-12T18:50:04.601 DEBUG:boto:path=/jdpiak-10/%26 2022-09-12T18:50:04.601 DEBUG:boto:auth_path=/jdpiak-10/%26 2022-09-12T18:50:04.602 DEBUG:boto:path=/jdpiak-10/%26?versionId=null 2022-09-12T18:50:04.603 DEBUG:boto:auth_path=/jdpiak-10/%26?versionId=null 2022-09-12T18:50:04.604 DEBUG:boto:Method: GET 2022-09-12T18:50:04.605 DEBUG:boto:Path: /jdpiak-10/%26?versionId=null 2022-09-12T18:50:04.606 DEBUG:boto:Data: 2022-09-12T18:50:04.607 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.608 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:04.608 DEBUG:boto:Port: 8001 2022-09-12T18:50:04.609 DEBUG:boto:Params: {} 2022-09-12T18:50:04.610 DEBUG:boto:Token: None 2022-09-12T18:50:04.611 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/%26?versionId=null 2022-09-12T18:50:04.612 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q0nktn9VAWaTuFonG6WxzBsdCAU= 2022-09-12T18:50:04.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q0nktn9VAWaTuFonG6WxzBsdCAU=', 'Content-Length': '0'} 2022-09-12T18:50:04.616 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b292d879433b83fb-00631f7f5c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.617 DEBUG:boto:path=/jdpiak-10/%26 2022-09-12T18:50:04.618 DEBUG:boto:auth_path=/jdpiak-10/%26 2022-09-12T18:50:04.619 DEBUG:boto:path=/jdpiak-10/%26?versionId=null 2022-09-12T18:50:04.620 DEBUG:boto:auth_path=/jdpiak-10/%26?versionId=null 2022-09-12T18:50:04.621 DEBUG:boto:Method: GET 2022-09-12T18:50:04.622 DEBUG:boto:Path: /jdpiak-10/%26?versionId=null 2022-09-12T18:50:04.623 DEBUG:boto:Data: 2022-09-12T18:50:04.623 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.624 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.624 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.625 DEBUG:boto:Params: {} 2022-09-12T18:50:04.625 DEBUG:boto:Token: None 2022-09-12T18:50:04.626 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/%26?versionId=null 2022-09-12T18:50:04.627 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q0nktn9VAWaTuFonG6WxzBsdCAU= 2022-09-12T18:50:04.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q0nktn9VAWaTuFonG6WxzBsdCAU=', 'Content-Length': '0'} 2022-09-12T18:50:04.629 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e81a4d610d36c140-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.630 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:04.631 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:04.631 DEBUG:boto:path=/jdpiak-10/?prefix=%26 2022-09-12T18:50:04.632 DEBUG:boto:auth_path=/jdpiak-10/?prefix=%26 2022-09-12T18:50:04.632 DEBUG:boto:Method: GET 2022-09-12T18:50:04.633 DEBUG:boto:Path: /jdpiak-10/?prefix=%26 2022-09-12T18:50:04.633 DEBUG:boto:Data: 2022-09-12T18:50:04.634 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.634 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:04.635 DEBUG:boto:Port: 8001 2022-09-12T18:50:04.636 DEBUG:boto:Params: {} 2022-09-12T18:50:04.636 DEBUG:boto:Token: None 2022-09-12T18:50:04.637 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/ 2022-09-12T18:50:04.637 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hDDUcNRlEDmaMg6EFa/fpwenFzQ= 2022-09-12T18:50:04.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hDDUcNRlEDmaMg6EFa/fpwenFzQ=', 'Content-Length': '0'} 2022-09-12T18:50:04.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e67c00de1471c12-00631f7f5c-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.641 DEBUG:boto:b'jdpiak-10&1000false&2022-09-12T18:49:46.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:04.642 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:04.643 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:04.643 DEBUG:boto:path=/jdpiak-10/?prefix=%26 2022-09-12T18:50:04.644 DEBUG:boto:auth_path=/jdpiak-10/?prefix=%26 2022-09-12T18:50:04.644 DEBUG:boto:Method: GET 2022-09-12T18:50:04.645 DEBUG:boto:Path: /jdpiak-10/?prefix=%26 2022-09-12T18:50:04.645 DEBUG:boto:Data: 2022-09-12T18:50:04.646 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.646 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.647 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.647 DEBUG:boto:Params: {} 2022-09-12T18:50:04.648 DEBUG:boto:Token: None 2022-09-12T18:50:04.649 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/ 2022-09-12T18:50:04.649 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hDDUcNRlEDmaMg6EFa/fpwenFzQ= 2022-09-12T18:50:04.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hDDUcNRlEDmaMg6EFa/fpwenFzQ=', 'Content-Length': '0'} 2022-09-12T18:50:04.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0bce70c40d8f197-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.653 DEBUG:boto:b'jdpiak-10&1000false&2022-09-12T18:49:46.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:04.654 DEBUG:boto:path=/jdpiak-10/%26 2022-09-12T18:50:04.654 DEBUG:boto:auth_path=/jdpiak-10/%26 2022-09-12T18:50:04.655 DEBUG:boto:Method: HEAD 2022-09-12T18:50:04.656 DEBUG:boto:Path: /jdpiak-10/%26 2022-09-12T18:50:04.656 DEBUG:boto:Data: 2022-09-12T18:50:04.657 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.657 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:04.658 DEBUG:boto:Port: 8001 2022-09-12T18:50:04.658 DEBUG:boto:Params: {} 2022-09-12T18:50:04.659 DEBUG:boto:Token: None 2022-09-12T18:50:04.660 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/%26 2022-09-12T18:50:04.660 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2eUCSRl2XZg8ijMlWzVpLV7PMMQ= 2022-09-12T18:50:04.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2eUCSRl2XZg8ijMlWzVpLV7PMMQ=', 'Content-Length': '0'} 2022-09-12T18:50:04.663 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b595effeb729a429-00631f7f5c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.663 DEBUG:boto:path=/jdpiak-10/%26 2022-09-12T18:50:04.664 DEBUG:boto:auth_path=/jdpiak-10/%26 2022-09-12T18:50:04.664 DEBUG:boto:Method: HEAD 2022-09-12T18:50:04.665 DEBUG:boto:Path: /jdpiak-10/%26 2022-09-12T18:50:04.665 DEBUG:boto:Data: 2022-09-12T18:50:04.666 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.666 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.667 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.668 DEBUG:boto:Params: {} 2022-09-12T18:50:04.668 DEBUG:boto:Token: None 2022-09-12T18:50:04.669 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/%26 2022-09-12T18:50:04.669 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2eUCSRl2XZg8ijMlWzVpLV7PMMQ= 2022-09-12T18:50:04.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2eUCSRl2XZg8ijMlWzVpLV7PMMQ=', 'Content-Length': '0'} 2022-09-12T18:50:04.672 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c974b246f30277f-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-12T18:50:04.674 DEBUG:boto:path=/jdpiak-10/%26 2022-09-12T18:50:04.675 DEBUG:boto:auth_path=/jdpiak-10/%26 2022-09-12T18:50:04.675 DEBUG:boto:Method: GET 2022-09-12T18:50:04.676 DEBUG:boto:Path: /jdpiak-10/%26 2022-09-12T18:50:04.676 DEBUG:boto:Data: 2022-09-12T18:50:04.677 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.677 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:04.678 DEBUG:boto:Port: 8001 2022-09-12T18:50:04.679 DEBUG:boto:Params: {} 2022-09-12T18:50:04.679 DEBUG:boto:Token: None 2022-09-12T18:50:04.680 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/%26 2022-09-12T18:50:04.681 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wpzrAj7CJqCuaDmwf4R7OXfX0eU= 2022-09-12T18:50:04.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wpzrAj7CJqCuaDmwf4R7OXfX0eU=', 'Content-Length': '0'} 2022-09-12T18:50:04.683 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb25865b4c9d7cb2-00631f7f5c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.684 DEBUG:boto:path=/jdpiak-10/%26 2022-09-12T18:50:04.685 DEBUG:boto:auth_path=/jdpiak-10/%26 2022-09-12T18:50:04.685 DEBUG:boto:Method: GET 2022-09-12T18:50:04.686 DEBUG:boto:Path: /jdpiak-10/%26 2022-09-12T18:50:04.687 DEBUG:boto:Data: 2022-09-12T18:50:04.687 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.688 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.688 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.689 DEBUG:boto:Params: {} 2022-09-12T18:50:04.689 DEBUG:boto:Token: None 2022-09-12T18:50:04.690 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/%26 2022-09-12T18:50:04.691 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wpzrAj7CJqCuaDmwf4R7OXfX0eU= 2022-09-12T18:50:04.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wpzrAj7CJqCuaDmwf4R7OXfX0eU=', 'Content-Length': '0'} 2022-09-12T18:50:04.693 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ddd53e38be7e9b82-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.694 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:04.694 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:04.695 DEBUG:boto:path=/jdpiak-10/?prefix=%26 2022-09-12T18:50:04.695 DEBUG:boto:auth_path=/jdpiak-10/?prefix=%26 2022-09-12T18:50:04.696 DEBUG:boto:Method: GET 2022-09-12T18:50:04.697 DEBUG:boto:Path: /jdpiak-10/?prefix=%26 2022-09-12T18:50:04.697 DEBUG:boto:Data: 2022-09-12T18:50:04.698 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.698 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:04.699 DEBUG:boto:Port: 8001 2022-09-12T18:50:04.700 DEBUG:boto:Params: {} 2022-09-12T18:50:04.700 DEBUG:boto:Token: None 2022-09-12T18:50:04.701 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/ 2022-09-12T18:50:04.702 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hDDUcNRlEDmaMg6EFa/fpwenFzQ= 2022-09-12T18:50:04.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hDDUcNRlEDmaMg6EFa/fpwenFzQ=', 'Content-Length': '0'} 2022-09-12T18:50:04.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab6201af3133c691-00631f7f5c-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.706 DEBUG:boto:b'jdpiak-10&1000false&2022-09-12T18:49:46.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:04.707 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:04.707 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:04.708 DEBUG:boto:path=/jdpiak-10/?prefix=%26 2022-09-12T18:50:04.708 DEBUG:boto:auth_path=/jdpiak-10/?prefix=%26 2022-09-12T18:50:04.709 DEBUG:boto:Method: GET 2022-09-12T18:50:04.709 DEBUG:boto:Path: /jdpiak-10/?prefix=%26 2022-09-12T18:50:04.710 DEBUG:boto:Data: 2022-09-12T18:50:04.711 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.711 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.712 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.712 DEBUG:boto:Params: {} 2022-09-12T18:50:04.713 DEBUG:boto:Token: None 2022-09-12T18:50:04.713 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/ 2022-09-12T18:50:04.714 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hDDUcNRlEDmaMg6EFa/fpwenFzQ= 2022-09-12T18:50:04.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hDDUcNRlEDmaMg6EFa/fpwenFzQ=', 'Content-Length': '0'} 2022-09-12T18:50:04.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000998e19c91a47f9cc-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.717 DEBUG:boto:b'jdpiak-10&1000false&2022-09-12T18:49:46.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:04.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-12T18:50:04.719 DEBUG:boto:path=/jdpiak-10/%3A 2022-09-12T18:50:04.719 DEBUG:boto:auth_path=/jdpiak-10/%3A 2022-09-12T18:50:04.720 DEBUG:boto:path=/jdpiak-10/%3A?versionId=null 2022-09-12T18:50:04.720 DEBUG:boto:auth_path=/jdpiak-10/%3A?versionId=null 2022-09-12T18:50:04.721 DEBUG:boto:Method: GET 2022-09-12T18:50:04.722 DEBUG:boto:Path: /jdpiak-10/%3A?versionId=null 2022-09-12T18:50:04.722 DEBUG:boto:Data: 2022-09-12T18:50:04.723 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.723 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:04.724 DEBUG:boto:Port: 8001 2022-09-12T18:50:04.724 DEBUG:boto:Params: {} 2022-09-12T18:50:04.725 DEBUG:boto:Token: None 2022-09-12T18:50:04.725 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/%3A?versionId=null 2022-09-12T18:50:04.726 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IqY9SbVIAkq/UHYUkghM+QzqFjU= 2022-09-12T18:50:04.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IqY9SbVIAkq/UHYUkghM+QzqFjU=', 'Content-Length': '0'} 2022-09-12T18:50:04.729 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c13daad56a41b745-00631f7f5c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.730 DEBUG:boto:path=/jdpiak-10/%3A 2022-09-12T18:50:04.730 DEBUG:boto:auth_path=/jdpiak-10/%3A 2022-09-12T18:50:04.731 DEBUG:boto:path=/jdpiak-10/%3A?versionId=null 2022-09-12T18:50:04.732 DEBUG:boto:auth_path=/jdpiak-10/%3A?versionId=null 2022-09-12T18:50:04.732 DEBUG:boto:Method: GET 2022-09-12T18:50:04.733 DEBUG:boto:Path: /jdpiak-10/%3A?versionId=null 2022-09-12T18:50:04.733 DEBUG:boto:Data: 2022-09-12T18:50:04.734 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.734 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.735 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.735 DEBUG:boto:Params: {} 2022-09-12T18:50:04.736 DEBUG:boto:Token: None 2022-09-12T18:50:04.737 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/%3A?versionId=null 2022-09-12T18:50:04.737 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IqY9SbVIAkq/UHYUkghM+QzqFjU= 2022-09-12T18:50:04.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IqY9SbVIAkq/UHYUkghM+QzqFjU=', 'Content-Length': '0'} 2022-09-12T18:50:04.740 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d230c47cb6a7e993-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.740 DEBUG:boto:path=/jdpiak-10/%3A 2022-09-12T18:50:04.741 DEBUG:boto:auth_path=/jdpiak-10/%3A 2022-09-12T18:50:04.741 DEBUG:boto:path=/jdpiak-10/%3A?versionId=null 2022-09-12T18:50:04.742 DEBUG:boto:auth_path=/jdpiak-10/%3A?versionId=null 2022-09-12T18:50:04.742 DEBUG:boto:Method: HEAD 2022-09-12T18:50:04.743 DEBUG:boto:Path: /jdpiak-10/%3A?versionId=null 2022-09-12T18:50:04.744 DEBUG:boto:Data: 2022-09-12T18:50:04.744 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.745 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:04.745 DEBUG:boto:Port: 8001 2022-09-12T18:50:04.746 DEBUG:boto:Params: {} 2022-09-12T18:50:04.746 DEBUG:boto:Token: None 2022-09-12T18:50:04.747 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/%3A?versionId=null 2022-09-12T18:50:04.747 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eZCFrTI6Yl8MvrCC1uCvhMyJkSY= 2022-09-12T18:50:04.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eZCFrTI6Yl8MvrCC1uCvhMyJkSY=', 'Content-Length': '0'} 2022-09-12T18:50:04.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b990dd15c504f5e4-00631f7f5c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.751 DEBUG:boto:path=/jdpiak-10/%3A 2022-09-12T18:50:04.752 DEBUG:boto:auth_path=/jdpiak-10/%3A 2022-09-12T18:50:04.752 DEBUG:boto:path=/jdpiak-10/%3A?versionId=null 2022-09-12T18:50:04.753 DEBUG:boto:auth_path=/jdpiak-10/%3A?versionId=null 2022-09-12T18:50:04.753 DEBUG:boto:Method: HEAD 2022-09-12T18:50:04.754 DEBUG:boto:Path: /jdpiak-10/%3A?versionId=null 2022-09-12T18:50:04.754 DEBUG:boto:Data: 2022-09-12T18:50:04.755 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.755 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.756 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.756 DEBUG:boto:Params: {} 2022-09-12T18:50:04.757 DEBUG:boto:Token: None 2022-09-12T18:50:04.758 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/%3A?versionId=null 2022-09-12T18:50:04.758 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eZCFrTI6Yl8MvrCC1uCvhMyJkSY= 2022-09-12T18:50:04.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eZCFrTI6Yl8MvrCC1uCvhMyJkSY=', 'Content-Length': '0'} 2022-09-12T18:50:04.762 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005037369070e31025-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-12T18:50:04.764 DEBUG:boto:path=/jdpiak-10/%3A 2022-09-12T18:50:04.765 DEBUG:boto:auth_path=/jdpiak-10/%3A 2022-09-12T18:50:04.766 DEBUG:boto:path=/jdpiak-10/%3A?versionId=null 2022-09-12T18:50:04.767 DEBUG:boto:auth_path=/jdpiak-10/%3A?versionId=null 2022-09-12T18:50:04.768 DEBUG:boto:Method: GET 2022-09-12T18:50:04.769 DEBUG:boto:Path: /jdpiak-10/%3A?versionId=null 2022-09-12T18:50:04.770 DEBUG:boto:Data: 2022-09-12T18:50:04.771 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.771 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:04.772 DEBUG:boto:Port: 8001 2022-09-12T18:50:04.772 DEBUG:boto:Params: {} 2022-09-12T18:50:04.773 DEBUG:boto:Token: None 2022-09-12T18:50:04.774 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/%3A?versionId=null 2022-09-12T18:50:04.774 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IqY9SbVIAkq/UHYUkghM+QzqFjU= 2022-09-12T18:50:04.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IqY9SbVIAkq/UHYUkghM+QzqFjU=', 'Content-Length': '0'} 2022-09-12T18:50:04.777 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054ce93883936dc8b-00631f7f5c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.818 DEBUG:boto:path=/jdpiak-10/%3A 2022-09-12T18:50:04.819 DEBUG:boto:auth_path=/jdpiak-10/%3A 2022-09-12T18:50:04.819 DEBUG:boto:path=/jdpiak-10/%3A?versionId=null 2022-09-12T18:50:04.820 DEBUG:boto:auth_path=/jdpiak-10/%3A?versionId=null 2022-09-12T18:50:04.820 DEBUG:boto:Method: GET 2022-09-12T18:50:04.821 DEBUG:boto:Path: /jdpiak-10/%3A?versionId=null 2022-09-12T18:50:04.822 DEBUG:boto:Data: 2022-09-12T18:50:04.822 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.823 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.823 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.824 DEBUG:boto:Params: {} 2022-09-12T18:50:04.825 DEBUG:boto:Token: None 2022-09-12T18:50:04.825 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/%3A?versionId=null 2022-09-12T18:50:04.826 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IqY9SbVIAkq/UHYUkghM+QzqFjU= 2022-09-12T18:50:04.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IqY9SbVIAkq/UHYUkghM+QzqFjU=', 'Content-Length': '0'} 2022-09-12T18:50:04.829 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f25777a3cf854c1f-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.830 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:04.830 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:04.831 DEBUG:boto:path=/jdpiak-10/?prefix=%3A 2022-09-12T18:50:04.831 DEBUG:boto:auth_path=/jdpiak-10/?prefix=%3A 2022-09-12T18:50:04.832 DEBUG:boto:Method: GET 2022-09-12T18:50:04.832 DEBUG:boto:Path: /jdpiak-10/?prefix=%3A 2022-09-12T18:50:04.833 DEBUG:boto:Data: 2022-09-12T18:50:04.833 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.834 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:04.835 DEBUG:boto:Port: 8001 2022-09-12T18:50:04.835 DEBUG:boto:Params: {} 2022-09-12T18:50:04.836 DEBUG:boto:Token: None 2022-09-12T18:50:04.836 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/ 2022-09-12T18:50:04.837 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hDDUcNRlEDmaMg6EFa/fpwenFzQ= 2022-09-12T18:50:04.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hDDUcNRlEDmaMg6EFa/fpwenFzQ=', 'Content-Length': '0'} 2022-09-12T18:50:04.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000199aa4dc8910d926-00631f7f5c-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.841 DEBUG:boto:b'jdpiak-10:1000false:2022-09-12T18:49:46.939Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:04.842 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:04.842 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:04.843 DEBUG:boto:path=/jdpiak-10/?prefix=%3A 2022-09-12T18:50:04.844 DEBUG:boto:auth_path=/jdpiak-10/?prefix=%3A 2022-09-12T18:50:04.844 DEBUG:boto:Method: GET 2022-09-12T18:50:04.845 DEBUG:boto:Path: /jdpiak-10/?prefix=%3A 2022-09-12T18:50:04.845 DEBUG:boto:Data: 2022-09-12T18:50:04.846 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.847 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.847 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.848 DEBUG:boto:Params: {} 2022-09-12T18:50:04.848 DEBUG:boto:Token: None 2022-09-12T18:50:04.849 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/ 2022-09-12T18:50:04.849 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hDDUcNRlEDmaMg6EFa/fpwenFzQ= 2022-09-12T18:50:04.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hDDUcNRlEDmaMg6EFa/fpwenFzQ=', 'Content-Length': '0'} 2022-09-12T18:50:04.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a3f620b457ef8de-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.853 DEBUG:boto:b'jdpiak-10:1000false:2022-09-12T18:49:46.939Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:04.854 DEBUG:boto:path=/jdpiak-10/%3A 2022-09-12T18:50:04.855 DEBUG:boto:auth_path=/jdpiak-10/%3A 2022-09-12T18:50:04.855 DEBUG:boto:Method: HEAD 2022-09-12T18:50:04.856 DEBUG:boto:Path: /jdpiak-10/%3A 2022-09-12T18:50:04.857 DEBUG:boto:Data: 2022-09-12T18:50:04.857 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.858 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:04.859 DEBUG:boto:Port: 8001 2022-09-12T18:50:04.859 DEBUG:boto:Params: {} 2022-09-12T18:50:04.860 DEBUG:boto:Token: None 2022-09-12T18:50:04.860 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/%3A 2022-09-12T18:50:04.861 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XlPPMXh8Jjk8xhbRmHFgFF8PNLM= 2022-09-12T18:50:04.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XlPPMXh8Jjk8xhbRmHFgFF8PNLM=', 'Content-Length': '0'} 2022-09-12T18:50:04.863 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091a50ff910e569e9-00631f7f5c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.864 DEBUG:boto:path=/jdpiak-10/%3A 2022-09-12T18:50:04.864 DEBUG:boto:auth_path=/jdpiak-10/%3A 2022-09-12T18:50:04.865 DEBUG:boto:Method: HEAD 2022-09-12T18:50:04.865 DEBUG:boto:Path: /jdpiak-10/%3A 2022-09-12T18:50:04.866 DEBUG:boto:Data: 2022-09-12T18:50:04.866 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.867 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.867 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.868 DEBUG:boto:Params: {} 2022-09-12T18:50:04.869 DEBUG:boto:Token: None 2022-09-12T18:50:04.869 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/%3A 2022-09-12T18:50:04.870 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XlPPMXh8Jjk8xhbRmHFgFF8PNLM= 2022-09-12T18:50:04.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XlPPMXh8Jjk8xhbRmHFgFF8PNLM=', 'Content-Length': '0'} 2022-09-12T18:50:04.872 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e542191297f37424-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-12T18:50:04.873 DEBUG:boto:path=/jdpiak-10/%3A 2022-09-12T18:50:04.874 DEBUG:boto:auth_path=/jdpiak-10/%3A 2022-09-12T18:50:04.875 DEBUG:boto:Method: GET 2022-09-12T18:50:04.875 DEBUG:boto:Path: /jdpiak-10/%3A 2022-09-12T18:50:04.876 DEBUG:boto:Data: 2022-09-12T18:50:04.877 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.877 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:04.878 DEBUG:boto:Port: 8001 2022-09-12T18:50:04.879 DEBUG:boto:Params: {} 2022-09-12T18:50:04.879 DEBUG:boto:Token: None 2022-09-12T18:50:04.880 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/%3A 2022-09-12T18:50:04.881 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6vfIIKRsUnz6HSxBFi0oze6UUsg= 2022-09-12T18:50:04.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6vfIIKRsUnz6HSxBFi0oze6UUsg=', 'Content-Length': '0'} 2022-09-12T18:50:04.884 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074fa1028468f2570-00631f7f5c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.885 DEBUG:boto:path=/jdpiak-10/%3A 2022-09-12T18:50:04.885 DEBUG:boto:auth_path=/jdpiak-10/%3A 2022-09-12T18:50:04.886 DEBUG:boto:Method: GET 2022-09-12T18:50:04.887 DEBUG:boto:Path: /jdpiak-10/%3A 2022-09-12T18:50:04.887 DEBUG:boto:Data: 2022-09-12T18:50:04.888 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.889 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.889 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.890 DEBUG:boto:Params: {} 2022-09-12T18:50:04.891 DEBUG:boto:Token: None 2022-09-12T18:50:04.892 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/%3A 2022-09-12T18:50:04.892 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6vfIIKRsUnz6HSxBFi0oze6UUsg= 2022-09-12T18:50:04.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6vfIIKRsUnz6HSxBFi0oze6UUsg=', 'Content-Length': '0'} 2022-09-12T18:50:04.895 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000828b8eeea67dade3-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.896 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:04.896 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:04.897 DEBUG:boto:path=/jdpiak-10/?prefix=%3A 2022-09-12T18:50:04.898 DEBUG:boto:auth_path=/jdpiak-10/?prefix=%3A 2022-09-12T18:50:04.898 DEBUG:boto:Method: GET 2022-09-12T18:50:04.899 DEBUG:boto:Path: /jdpiak-10/?prefix=%3A 2022-09-12T18:50:04.899 DEBUG:boto:Data: 2022-09-12T18:50:04.900 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.901 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:04.901 DEBUG:boto:Port: 8001 2022-09-12T18:50:04.902 DEBUG:boto:Params: {} 2022-09-12T18:50:04.902 DEBUG:boto:Token: None 2022-09-12T18:50:04.903 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/ 2022-09-12T18:50:04.904 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hDDUcNRlEDmaMg6EFa/fpwenFzQ= 2022-09-12T18:50:04.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hDDUcNRlEDmaMg6EFa/fpwenFzQ=', 'Content-Length': '0'} 2022-09-12T18:50:04.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007313963b498b9dfb-00631f7f5c-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.908 DEBUG:boto:b'jdpiak-10:1000false:2022-09-12T18:49:46.939Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:04.909 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:04.910 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:04.910 DEBUG:boto:path=/jdpiak-10/?prefix=%3A 2022-09-12T18:50:04.911 DEBUG:boto:auth_path=/jdpiak-10/?prefix=%3A 2022-09-12T18:50:04.912 DEBUG:boto:Method: GET 2022-09-12T18:50:04.912 DEBUG:boto:Path: /jdpiak-10/?prefix=%3A 2022-09-12T18:50:04.913 DEBUG:boto:Data: 2022-09-12T18:50:04.913 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.914 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.915 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.915 DEBUG:boto:Params: {} 2022-09-12T18:50:04.916 DEBUG:boto:Token: None 2022-09-12T18:50:04.917 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/ 2022-09-12T18:50:04.918 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hDDUcNRlEDmaMg6EFa/fpwenFzQ= 2022-09-12T18:50:04.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hDDUcNRlEDmaMg6EFa/fpwenFzQ=', 'Content-Length': '0'} 2022-09-12T18:50:04.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d9ff6814b62dc27-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.922 DEBUG:boto:b'jdpiak-10:1000false:2022-09-12T18:49:46.939Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:04.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-12T18:50:04.923 DEBUG:boto:path=/jdpiak-10/_myobj 2022-09-12T18:50:04.924 DEBUG:boto:auth_path=/jdpiak-10/_myobj 2022-09-12T18:50:04.925 DEBUG:boto:path=/jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:04.926 DEBUG:boto:auth_path=/jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:04.926 DEBUG:boto:Method: GET 2022-09-12T18:50:04.927 DEBUG:boto:Path: /jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:04.928 DEBUG:boto:Data: 2022-09-12T18:50:04.929 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.929 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:04.930 DEBUG:boto:Port: 8001 2022-09-12T18:50:04.931 DEBUG:boto:Params: {} 2022-09-12T18:50:04.932 DEBUG:boto:Token: None 2022-09-12T18:50:04.932 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:04.933 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Xs76EQNC94LRPDUwEb01QVHqTI8= 2022-09-12T18:50:04.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Xs76EQNC94LRPDUwEb01QVHqTI8=', 'Content-Length': '0'} 2022-09-12T18:50:04.936 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a536233675b6c19-00631f7f5c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.937 DEBUG:boto:path=/jdpiak-10/_myobj 2022-09-12T18:50:04.938 DEBUG:boto:auth_path=/jdpiak-10/_myobj 2022-09-12T18:50:04.939 DEBUG:boto:path=/jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:04.939 DEBUG:boto:auth_path=/jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:04.940 DEBUG:boto:Method: GET 2022-09-12T18:50:04.941 DEBUG:boto:Path: /jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:04.942 DEBUG:boto:Data: 2022-09-12T18:50:04.942 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.943 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.943 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.944 DEBUG:boto:Params: {} 2022-09-12T18:50:04.944 DEBUG:boto:Token: None 2022-09-12T18:50:04.945 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:04.946 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Xs76EQNC94LRPDUwEb01QVHqTI8= 2022-09-12T18:50:04.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Xs76EQNC94LRPDUwEb01QVHqTI8=', 'Content-Length': '0'} 2022-09-12T18:50:04.948 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b0b566c7e80cdd7-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.949 DEBUG:boto:path=/jdpiak-10/_myobj 2022-09-12T18:50:04.950 DEBUG:boto:auth_path=/jdpiak-10/_myobj 2022-09-12T18:50:04.950 DEBUG:boto:path=/jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:04.951 DEBUG:boto:auth_path=/jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:04.951 DEBUG:boto:Method: HEAD 2022-09-12T18:50:04.952 DEBUG:boto:Path: /jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:04.952 DEBUG:boto:Data: 2022-09-12T18:50:04.953 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.954 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:04.954 DEBUG:boto:Port: 8001 2022-09-12T18:50:04.955 DEBUG:boto:Params: {} 2022-09-12T18:50:04.956 DEBUG:boto:Token: None 2022-09-12T18:50:04.957 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:04.957 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gvoOtAMf8jAMnkFH8yu2/F/rovE= 2022-09-12T18:50:04.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gvoOtAMf8jAMnkFH8yu2/F/rovE=', 'Content-Length': '0'} 2022-09-12T18:50:04.960 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e30de6333ed45ec6-00631f7f5c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.961 DEBUG:boto:path=/jdpiak-10/_myobj 2022-09-12T18:50:04.962 DEBUG:boto:auth_path=/jdpiak-10/_myobj 2022-09-12T18:50:04.962 DEBUG:boto:path=/jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:04.963 DEBUG:boto:auth_path=/jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:04.963 DEBUG:boto:Method: HEAD 2022-09-12T18:50:04.964 DEBUG:boto:Path: /jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:04.965 DEBUG:boto:Data: 2022-09-12T18:50:04.966 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.966 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.967 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.968 DEBUG:boto:Params: {} 2022-09-12T18:50:04.969 DEBUG:boto:Token: None 2022-09-12T18:50:04.969 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:04.970 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gvoOtAMf8jAMnkFH8yu2/F/rovE= 2022-09-12T18:50:04.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gvoOtAMf8jAMnkFH8yu2/F/rovE=', 'Content-Length': '0'} 2022-09-12T18:50:04.973 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000642268490d512f0e-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-12T18:50:04.975 DEBUG:boto:path=/jdpiak-10/_myobj 2022-09-12T18:50:04.975 DEBUG:boto:auth_path=/jdpiak-10/_myobj 2022-09-12T18:50:04.976 DEBUG:boto:path=/jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:04.976 DEBUG:boto:auth_path=/jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:04.977 DEBUG:boto:Method: GET 2022-09-12T18:50:04.977 DEBUG:boto:Path: /jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:04.978 DEBUG:boto:Data: 2022-09-12T18:50:04.979 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.979 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:04.980 DEBUG:boto:Port: 8001 2022-09-12T18:50:04.981 DEBUG:boto:Params: {} 2022-09-12T18:50:04.981 DEBUG:boto:Token: None 2022-09-12T18:50:04.982 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:04.983 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Xs76EQNC94LRPDUwEb01QVHqTI8= 2022-09-12T18:50:04.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Xs76EQNC94LRPDUwEb01QVHqTI8=', 'Content-Length': '0'} 2022-09-12T18:50:04.986 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1505f0161dbfe00-00631f7f5c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.987 DEBUG:boto:path=/jdpiak-10/_myobj 2022-09-12T18:50:04.987 DEBUG:boto:auth_path=/jdpiak-10/_myobj 2022-09-12T18:50:04.988 DEBUG:boto:path=/jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:04.989 DEBUG:boto:auth_path=/jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:04.989 DEBUG:boto:Method: GET 2022-09-12T18:50:04.990 DEBUG:boto:Path: /jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:04.990 DEBUG:boto:Data: 2022-09-12T18:50:04.991 DEBUG:boto:Headers: {} 2022-09-12T18:50:04.992 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:04.993 DEBUG:boto:Port: 8000 2022-09-12T18:50:04.993 DEBUG:boto:Params: {} 2022-09-12T18:50:04.994 DEBUG:boto:Token: None 2022-09-12T18:50:04.995 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:04 GMT /jdpiak-10/_myobj?versionId=null 2022-09-12T18:50:04.996 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Xs76EQNC94LRPDUwEb01QVHqTI8= 2022-09-12T18:50:04.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Xs76EQNC94LRPDUwEb01QVHqTI8=', 'Content-Length': '0'} 2022-09-12T18:50:04.998 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000116c3af16b72f1d4-00631f7f5c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:04.999 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:05.000 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:05.001 DEBUG:boto:path=/jdpiak-10/?prefix=_myobj 2022-09-12T18:50:05.001 DEBUG:boto:auth_path=/jdpiak-10/?prefix=_myobj 2022-09-12T18:50:05.002 DEBUG:boto:Method: GET 2022-09-12T18:50:05.003 DEBUG:boto:Path: /jdpiak-10/?prefix=_myobj 2022-09-12T18:50:05.003 DEBUG:boto:Data: 2022-09-12T18:50:05.004 DEBUG:boto:Headers: {} 2022-09-12T18:50:05.004 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:05.005 DEBUG:boto:Port: 8001 2022-09-12T18:50:05.006 DEBUG:boto:Params: {} 2022-09-12T18:50:05.006 DEBUG:boto:Token: None 2022-09-12T18:50:05.007 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:05 GMT /jdpiak-10/ 2022-09-12T18:50:05.008 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QneLnL0B/J8DNlTvQmGQVfN79QE= 2022-09-12T18:50:05.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QneLnL0B/J8DNlTvQmGQVfN79QE=', 'Content-Length': '0'} 2022-09-12T18:50:05.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e5fdbb2b52436ad-00631f7f5d-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:05.012 DEBUG:boto:b'jdpiak-10_myobj1000false_myobj2022-09-12T18:49:46.878Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:05.013 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:05.014 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:05.015 DEBUG:boto:path=/jdpiak-10/?prefix=_myobj 2022-09-12T18:50:05.015 DEBUG:boto:auth_path=/jdpiak-10/?prefix=_myobj 2022-09-12T18:50:05.016 DEBUG:boto:Method: GET 2022-09-12T18:50:05.016 DEBUG:boto:Path: /jdpiak-10/?prefix=_myobj 2022-09-12T18:50:05.017 DEBUG:boto:Data: 2022-09-12T18:50:05.018 DEBUG:boto:Headers: {} 2022-09-12T18:50:05.019 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:05.019 DEBUG:boto:Port: 8000 2022-09-12T18:50:05.020 DEBUG:boto:Params: {} 2022-09-12T18:50:05.021 DEBUG:boto:Token: None 2022-09-12T18:50:05.021 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:05 GMT /jdpiak-10/ 2022-09-12T18:50:05.022 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QneLnL0B/J8DNlTvQmGQVfN79QE= 2022-09-12T18:50:05.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QneLnL0B/J8DNlTvQmGQVfN79QE=', 'Content-Length': '0'} 2022-09-12T18:50:05.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b90b3ddc3dea823c-00631f7f5d-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:05.028 DEBUG:boto:b'jdpiak-10_myobj1000false_myobj2022-09-12T18:49:46.878Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:05.030 DEBUG:boto:path=/jdpiak-10/_myobj 2022-09-12T18:50:05.031 DEBUG:boto:auth_path=/jdpiak-10/_myobj 2022-09-12T18:50:05.031 DEBUG:boto:Method: HEAD 2022-09-12T18:50:05.032 DEBUG:boto:Path: /jdpiak-10/_myobj 2022-09-12T18:50:05.032 DEBUG:boto:Data: 2022-09-12T18:50:05.033 DEBUG:boto:Headers: {} 2022-09-12T18:50:05.033 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:05.034 DEBUG:boto:Port: 8001 2022-09-12T18:50:05.034 DEBUG:boto:Params: {} 2022-09-12T18:50:05.035 DEBUG:boto:Token: None 2022-09-12T18:50:05.035 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:05 GMT /jdpiak-10/_myobj 2022-09-12T18:50:05.036 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JsHwsT3BVE92Zwl9UFip9C+Ov6A= 2022-09-12T18:50:05.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JsHwsT3BVE92Zwl9UFip9C+Ov6A=', 'Content-Length': '0'} 2022-09-12T18:50:05.040 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9043ad1a55541ac-00631f7f5d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:05.042 DEBUG:boto:path=/jdpiak-10/_myobj 2022-09-12T18:50:05.043 DEBUG:boto:auth_path=/jdpiak-10/_myobj 2022-09-12T18:50:05.043 DEBUG:boto:Method: HEAD 2022-09-12T18:50:05.044 DEBUG:boto:Path: /jdpiak-10/_myobj 2022-09-12T18:50:05.044 DEBUG:boto:Data: 2022-09-12T18:50:05.045 DEBUG:boto:Headers: {} 2022-09-12T18:50:05.045 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:05.046 DEBUG:boto:Port: 8000 2022-09-12T18:50:05.046 DEBUG:boto:Params: {} 2022-09-12T18:50:05.047 DEBUG:boto:Token: None 2022-09-12T18:50:05.047 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:05 GMT /jdpiak-10/_myobj 2022-09-12T18:50:05.048 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JsHwsT3BVE92Zwl9UFip9C+Ov6A= 2022-09-12T18:50:05.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JsHwsT3BVE92Zwl9UFip9C+Ov6A=', 'Content-Length': '0'} 2022-09-12T18:50:05.051 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2a3e84d2343e565-00631f7f5d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:05.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-12T18:50:05.052 DEBUG:boto:path=/jdpiak-10/_myobj 2022-09-12T18:50:05.052 DEBUG:boto:auth_path=/jdpiak-10/_myobj 2022-09-12T18:50:05.053 DEBUG:boto:Method: GET 2022-09-12T18:50:05.054 DEBUG:boto:Path: /jdpiak-10/_myobj 2022-09-12T18:50:05.054 DEBUG:boto:Data: 2022-09-12T18:50:05.055 DEBUG:boto:Headers: {} 2022-09-12T18:50:05.055 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:05.056 DEBUG:boto:Port: 8001 2022-09-12T18:50:05.056 DEBUG:boto:Params: {} 2022-09-12T18:50:05.057 DEBUG:boto:Token: None 2022-09-12T18:50:05.057 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:05 GMT /jdpiak-10/_myobj 2022-09-12T18:50:05.058 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y3tYFQ7U/4/oyonrfogpzVqJhAI= 2022-09-12T18:50:05.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y3tYFQ7U/4/oyonrfogpzVqJhAI=', 'Content-Length': '0'} 2022-09-12T18:50:05.061 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f69c62e34ff8400-00631f7f5d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:05.102 DEBUG:boto:path=/jdpiak-10/_myobj 2022-09-12T18:50:05.103 DEBUG:boto:auth_path=/jdpiak-10/_myobj 2022-09-12T18:50:05.103 DEBUG:boto:Method: GET 2022-09-12T18:50:05.104 DEBUG:boto:Path: /jdpiak-10/_myobj 2022-09-12T18:50:05.105 DEBUG:boto:Data: 2022-09-12T18:50:05.105 DEBUG:boto:Headers: {} 2022-09-12T18:50:05.106 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:05.106 DEBUG:boto:Port: 8000 2022-09-12T18:50:05.107 DEBUG:boto:Params: {} 2022-09-12T18:50:05.107 DEBUG:boto:Token: None 2022-09-12T18:50:05.108 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:05 GMT /jdpiak-10/_myobj 2022-09-12T18:50:05.109 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y3tYFQ7U/4/oyonrfogpzVqJhAI= 2022-09-12T18:50:05.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y3tYFQ7U/4/oyonrfogpzVqJhAI=', 'Content-Length': '0'} 2022-09-12T18:50:05.111 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008004457a30cbd21e-00631f7f5d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:05.112 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:05.113 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:05.113 DEBUG:boto:path=/jdpiak-10/?prefix=_myobj 2022-09-12T18:50:05.114 DEBUG:boto:auth_path=/jdpiak-10/?prefix=_myobj 2022-09-12T18:50:05.114 DEBUG:boto:Method: GET 2022-09-12T18:50:05.115 DEBUG:boto:Path: /jdpiak-10/?prefix=_myobj 2022-09-12T18:50:05.115 DEBUG:boto:Data: 2022-09-12T18:50:05.116 DEBUG:boto:Headers: {} 2022-09-12T18:50:05.116 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:05.117 DEBUG:boto:Port: 8001 2022-09-12T18:50:05.117 DEBUG:boto:Params: {} 2022-09-12T18:50:05.118 DEBUG:boto:Token: None 2022-09-12T18:50:05.119 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:05 GMT /jdpiak-10/ 2022-09-12T18:50:05.119 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QneLnL0B/J8DNlTvQmGQVfN79QE= 2022-09-12T18:50:05.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QneLnL0B/J8DNlTvQmGQVfN79QE=', 'Content-Length': '0'} 2022-09-12T18:50:05.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a95e691ac515ac13-00631f7f5d-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:05.123 DEBUG:boto:b'jdpiak-10_myobj1000false_myobj2022-09-12T18:49:46.878Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:05.124 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:05.125 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:05.125 DEBUG:boto:path=/jdpiak-10/?prefix=_myobj 2022-09-12T18:50:05.126 DEBUG:boto:auth_path=/jdpiak-10/?prefix=_myobj 2022-09-12T18:50:05.127 DEBUG:boto:Method: GET 2022-09-12T18:50:05.127 DEBUG:boto:Path: /jdpiak-10/?prefix=_myobj 2022-09-12T18:50:05.128 DEBUG:boto:Data: 2022-09-12T18:50:05.128 DEBUG:boto:Headers: {} 2022-09-12T18:50:05.129 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:05.129 DEBUG:boto:Port: 8000 2022-09-12T18:50:05.130 DEBUG:boto:Params: {} 2022-09-12T18:50:05.130 DEBUG:boto:Token: None 2022-09-12T18:50:05.131 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:05 GMT /jdpiak-10/ 2022-09-12T18:50:05.132 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QneLnL0B/J8DNlTvQmGQVfN79QE= 2022-09-12T18:50:05.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QneLnL0B/J8DNlTvQmGQVfN79QE=', 'Content-Length': '0'} 2022-09-12T18:50:05.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083ffb4130112196e-00631f7f5d-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:05.135 DEBUG:boto:b'jdpiak-10_myobj1000false_myobj2022-09-12T18:49:46.878Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:05.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-12T18:50:05.137 DEBUG:boto:path=/jdpiak-10/myobj 2022-09-12T18:50:05.137 DEBUG:boto:auth_path=/jdpiak-10/myobj 2022-09-12T18:50:05.138 DEBUG:boto:path=/jdpiak-10/myobj?versionId=null 2022-09-12T18:50:05.139 DEBUG:boto:auth_path=/jdpiak-10/myobj?versionId=null 2022-09-12T18:50:05.139 DEBUG:boto:Method: GET 2022-09-12T18:50:05.140 DEBUG:boto:Path: /jdpiak-10/myobj?versionId=null 2022-09-12T18:50:05.140 DEBUG:boto:Data: 2022-09-12T18:50:05.141 DEBUG:boto:Headers: {} 2022-09-12T18:50:05.141 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:05.142 DEBUG:boto:Port: 8001 2022-09-12T18:50:05.143 DEBUG:boto:Params: {} 2022-09-12T18:50:05.143 DEBUG:boto:Token: None 2022-09-12T18:50:05.144 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:05 GMT /jdpiak-10/myobj?versionId=null 2022-09-12T18:50:05.145 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KdCnq/vsdu5eultYLjFnTHf/jZ4= 2022-09-12T18:50:05.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KdCnq/vsdu5eultYLjFnTHf/jZ4=', 'Content-Length': '0'} 2022-09-12T18:50:05.147 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a08e04c11581652-00631f7f5d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:05.148 DEBUG:boto:path=/jdpiak-10/myobj 2022-09-12T18:50:05.149 DEBUG:boto:auth_path=/jdpiak-10/myobj 2022-09-12T18:50:05.149 DEBUG:boto:path=/jdpiak-10/myobj?versionId=null 2022-09-12T18:50:05.150 DEBUG:boto:auth_path=/jdpiak-10/myobj?versionId=null 2022-09-12T18:50:05.150 DEBUG:boto:Method: GET 2022-09-12T18:50:05.151 DEBUG:boto:Path: /jdpiak-10/myobj?versionId=null 2022-09-12T18:50:05.152 DEBUG:boto:Data: 2022-09-12T18:50:05.152 DEBUG:boto:Headers: {} 2022-09-12T18:50:05.153 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:05.153 DEBUG:boto:Port: 8000 2022-09-12T18:50:05.154 DEBUG:boto:Params: {} 2022-09-12T18:50:05.155 DEBUG:boto:Token: None 2022-09-12T18:50:05.155 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:05 GMT /jdpiak-10/myobj?versionId=null 2022-09-12T18:50:05.156 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KdCnq/vsdu5eultYLjFnTHf/jZ4= 2022-09-12T18:50:05.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KdCnq/vsdu5eultYLjFnTHf/jZ4=', 'Content-Length': '0'} 2022-09-12T18:50:05.158 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d993d97b8d25f03-00631f7f5d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:05.202 DEBUG:boto:path=/jdpiak-10/myobj 2022-09-12T18:50:05.203 DEBUG:boto:auth_path=/jdpiak-10/myobj 2022-09-12T18:50:05.204 DEBUG:boto:path=/jdpiak-10/myobj?versionId=null 2022-09-12T18:50:05.204 DEBUG:boto:auth_path=/jdpiak-10/myobj?versionId=null 2022-09-12T18:50:05.205 DEBUG:boto:Method: HEAD 2022-09-12T18:50:05.206 DEBUG:boto:Path: /jdpiak-10/myobj?versionId=null 2022-09-12T18:50:05.206 DEBUG:boto:Data: 2022-09-12T18:50:05.207 DEBUG:boto:Headers: {} 2022-09-12T18:50:05.207 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:05.208 DEBUG:boto:Port: 8001 2022-09-12T18:50:05.208 DEBUG:boto:Params: {} 2022-09-12T18:50:05.209 DEBUG:boto:Token: None 2022-09-12T18:50:05.209 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:05 GMT /jdpiak-10/myobj?versionId=null 2022-09-12T18:50:05.210 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pBfAt0kPGYt80dWgqch8j63sOyo= 2022-09-12T18:50:05.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pBfAt0kPGYt80dWgqch8j63sOyo=', 'Content-Length': '0'} 2022-09-12T18:50:05.213 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec72dad19f3039ba-00631f7f5d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:05.213 DEBUG:boto:path=/jdpiak-10/myobj 2022-09-12T18:50:05.214 DEBUG:boto:auth_path=/jdpiak-10/myobj 2022-09-12T18:50:05.214 DEBUG:boto:path=/jdpiak-10/myobj?versionId=null 2022-09-12T18:50:05.215 DEBUG:boto:auth_path=/jdpiak-10/myobj?versionId=null 2022-09-12T18:50:05.215 DEBUG:boto:Method: HEAD 2022-09-12T18:50:05.216 DEBUG:boto:Path: /jdpiak-10/myobj?versionId=null 2022-09-12T18:50:05.216 DEBUG:boto:Data: 2022-09-12T18:50:05.217 DEBUG:boto:Headers: {} 2022-09-12T18:50:05.217 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:05.218 DEBUG:boto:Port: 8000 2022-09-12T18:50:05.218 DEBUG:boto:Params: {} 2022-09-12T18:50:05.219 DEBUG:boto:Token: None 2022-09-12T18:50:05.219 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:05 GMT /jdpiak-10/myobj?versionId=null 2022-09-12T18:50:05.220 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pBfAt0kPGYt80dWgqch8j63sOyo= 2022-09-12T18:50:05.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pBfAt0kPGYt80dWgqch8j63sOyo=', 'Content-Length': '0'} 2022-09-12T18:50:05.222 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea0b540af34bae62-00631f7f5d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:05.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-12T18:50:05.224 DEBUG:boto:path=/jdpiak-10/myobj 2022-09-12T18:50:05.224 DEBUG:boto:auth_path=/jdpiak-10/myobj 2022-09-12T18:50:05.225 DEBUG:boto:path=/jdpiak-10/myobj?versionId=null 2022-09-12T18:50:05.225 DEBUG:boto:auth_path=/jdpiak-10/myobj?versionId=null 2022-09-12T18:50:05.226 DEBUG:boto:Method: GET 2022-09-12T18:50:05.226 DEBUG:boto:Path: /jdpiak-10/myobj?versionId=null 2022-09-12T18:50:05.227 DEBUG:boto:Data: 2022-09-12T18:50:05.227 DEBUG:boto:Headers: {} 2022-09-12T18:50:05.228 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:05.228 DEBUG:boto:Port: 8001 2022-09-12T18:50:05.229 DEBUG:boto:Params: {} 2022-09-12T18:50:05.229 DEBUG:boto:Token: None 2022-09-12T18:50:05.230 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:05 GMT /jdpiak-10/myobj?versionId=null 2022-09-12T18:50:05.231 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KdCnq/vsdu5eultYLjFnTHf/jZ4= 2022-09-12T18:50:05.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KdCnq/vsdu5eultYLjFnTHf/jZ4=', 'Content-Length': '0'} 2022-09-12T18:50:05.233 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d95c927bb3bb3b4a-00631f7f5d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:05.234 DEBUG:boto:path=/jdpiak-10/myobj 2022-09-12T18:50:05.234 DEBUG:boto:auth_path=/jdpiak-10/myobj 2022-09-12T18:50:05.235 DEBUG:boto:path=/jdpiak-10/myobj?versionId=null 2022-09-12T18:50:05.235 DEBUG:boto:auth_path=/jdpiak-10/myobj?versionId=null 2022-09-12T18:50:05.236 DEBUG:boto:Method: GET 2022-09-12T18:50:05.237 DEBUG:boto:Path: /jdpiak-10/myobj?versionId=null 2022-09-12T18:50:05.237 DEBUG:boto:Data: 2022-09-12T18:50:05.238 DEBUG:boto:Headers: {} 2022-09-12T18:50:05.238 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:05.239 DEBUG:boto:Port: 8000 2022-09-12T18:50:05.239 DEBUG:boto:Params: {} 2022-09-12T18:50:05.240 DEBUG:boto:Token: None 2022-09-12T18:50:05.240 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:05 GMT /jdpiak-10/myobj?versionId=null 2022-09-12T18:50:05.241 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KdCnq/vsdu5eultYLjFnTHf/jZ4= 2022-09-12T18:50:05.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KdCnq/vsdu5eultYLjFnTHf/jZ4=', 'Content-Length': '0'} 2022-09-12T18:50:05.243 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ad7bfc629806010-00631f7f5d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:05.244 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:05.245 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:05.245 DEBUG:boto:path=/jdpiak-10/?prefix=myobj 2022-09-12T18:50:05.246 DEBUG:boto:auth_path=/jdpiak-10/?prefix=myobj 2022-09-12T18:50:05.246 DEBUG:boto:Method: GET 2022-09-12T18:50:05.247 DEBUG:boto:Path: /jdpiak-10/?prefix=myobj 2022-09-12T18:50:05.248 DEBUG:boto:Data: 2022-09-12T18:50:05.248 DEBUG:boto:Headers: {} 2022-09-12T18:50:05.249 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:05.249 DEBUG:boto:Port: 8001 2022-09-12T18:50:05.250 DEBUG:boto:Params: {} 2022-09-12T18:50:05.250 DEBUG:boto:Token: None 2022-09-12T18:50:05.251 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:05 GMT /jdpiak-10/ 2022-09-12T18:50:05.252 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QneLnL0B/J8DNlTvQmGQVfN79QE= 2022-09-12T18:50:05.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QneLnL0B/J8DNlTvQmGQVfN79QE=', 'Content-Length': '0'} 2022-09-12T18:50:05.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b33d84614a88edba-00631f7f5d-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:05.256 DEBUG:boto:b'jdpiak-10myobj1000falsemyobj2022-09-12T18:49:46.847Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:05.257 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:05.257 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:05.258 DEBUG:boto:path=/jdpiak-10/?prefix=myobj 2022-09-12T18:50:05.258 DEBUG:boto:auth_path=/jdpiak-10/?prefix=myobj 2022-09-12T18:50:05.259 DEBUG:boto:Method: GET 2022-09-12T18:50:05.259 DEBUG:boto:Path: /jdpiak-10/?prefix=myobj 2022-09-12T18:50:05.260 DEBUG:boto:Data: 2022-09-12T18:50:05.261 DEBUG:boto:Headers: {} 2022-09-12T18:50:05.261 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:05.262 DEBUG:boto:Port: 8000 2022-09-12T18:50:05.262 DEBUG:boto:Params: {} 2022-09-12T18:50:05.263 DEBUG:boto:Token: None 2022-09-12T18:50:05.264 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:05 GMT /jdpiak-10/ 2022-09-12T18:50:05.264 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QneLnL0B/J8DNlTvQmGQVfN79QE= 2022-09-12T18:50:05.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QneLnL0B/J8DNlTvQmGQVfN79QE=', 'Content-Length': '0'} 2022-09-12T18:50:05.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005ffbf1f1aba63c3-00631f7f5d-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:05.268 DEBUG:boto:b'jdpiak-10myobj1000falsemyobj2022-09-12T18:49:46.847Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:05.269 DEBUG:boto:path=/jdpiak-10/myobj 2022-09-12T18:50:05.269 DEBUG:boto:auth_path=/jdpiak-10/myobj 2022-09-12T18:50:05.270 DEBUG:boto:Method: HEAD 2022-09-12T18:50:05.271 DEBUG:boto:Path: /jdpiak-10/myobj 2022-09-12T18:50:05.271 DEBUG:boto:Data: 2022-09-12T18:50:05.272 DEBUG:boto:Headers: {} 2022-09-12T18:50:05.272 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:05.273 DEBUG:boto:Port: 8001 2022-09-12T18:50:05.273 DEBUG:boto:Params: {} 2022-09-12T18:50:05.274 DEBUG:boto:Token: None 2022-09-12T18:50:05.275 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:05 GMT /jdpiak-10/myobj 2022-09-12T18:50:05.275 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MMCZIOBSt7UPP2vsPi7nmCphdY4= 2022-09-12T18:50:05.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MMCZIOBSt7UPP2vsPi7nmCphdY4=', 'Content-Length': '0'} 2022-09-12T18:50:05.278 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000926e54a9c939d8e2-00631f7f5d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:05.278 DEBUG:boto:path=/jdpiak-10/myobj 2022-09-12T18:50:05.279 DEBUG:boto:auth_path=/jdpiak-10/myobj 2022-09-12T18:50:05.280 DEBUG:boto:Method: HEAD 2022-09-12T18:50:05.280 DEBUG:boto:Path: /jdpiak-10/myobj 2022-09-12T18:50:05.281 DEBUG:boto:Data: 2022-09-12T18:50:05.281 DEBUG:boto:Headers: {} 2022-09-12T18:50:05.282 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:05.282 DEBUG:boto:Port: 8000 2022-09-12T18:50:05.283 DEBUG:boto:Params: {} 2022-09-12T18:50:05.283 DEBUG:boto:Token: None 2022-09-12T18:50:05.284 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:50:05 GMT /jdpiak-10/myobj 2022-09-12T18:50:05.284 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MMCZIOBSt7UPP2vsPi7nmCphdY4= 2022-09-12T18:50:05.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MMCZIOBSt7UPP2vsPi7nmCphdY4=', 'Content-Length': '0'} 2022-09-12T18:50:05.287 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090f9fc0c543fa15d-00631f7f5d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:05.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-12T18:50:05.288 DEBUG:boto:path=/jdpiak-10/myobj 2022-09-12T18:50:05.289 DEBUG:boto:auth_path=/jdpiak-10/myobj 2022-09-12T18:50:05.289 DEBUG:boto:Method: GET 2022-09-12T18:50:05.290 DEBUG:boto:Path: /jdpiak-10/myobj 2022-09-12T18:50:05.290 DEBUG:boto:Data: 2022-09-12T18:50:05.291 DEBUG:boto:Headers: {} 2022-09-12T18:50:05.291 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:05.292 DEBUG:boto:Port: 8001 2022-09-12T18:50:05.292 DEBUG:boto:Params: {} 2022-09-12T18:50:05.293 DEBUG:boto:Token: None 2022-09-12T18:50:05.294 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:05 GMT /jdpiak-10/myobj 2022-09-12T18:50:05.294 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:93ZajMOk9zGNjNnfC+0z22YB/K8= 2022-09-12T18:50:05.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:93ZajMOk9zGNjNnfC+0z22YB/K8=', 'Content-Length': '0'} 2022-09-12T18:50:05.297 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0791064d664dc57-00631f7f5d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:05.338 DEBUG:boto:path=/jdpiak-10/myobj 2022-09-12T18:50:05.339 DEBUG:boto:auth_path=/jdpiak-10/myobj 2022-09-12T18:50:05.340 DEBUG:boto:Method: GET 2022-09-12T18:50:05.341 DEBUG:boto:Path: /jdpiak-10/myobj 2022-09-12T18:50:05.341 DEBUG:boto:Data: 2022-09-12T18:50:05.342 DEBUG:boto:Headers: {} 2022-09-12T18:50:05.342 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:05.343 DEBUG:boto:Port: 8000 2022-09-12T18:50:05.343 DEBUG:boto:Params: {} 2022-09-12T18:50:05.344 DEBUG:boto:Token: None 2022-09-12T18:50:05.345 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:05 GMT /jdpiak-10/myobj 2022-09-12T18:50:05.345 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:93ZajMOk9zGNjNnfC+0z22YB/K8= 2022-09-12T18:50:05.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:93ZajMOk9zGNjNnfC+0z22YB/K8=', 'Content-Length': '0'} 2022-09-12T18:50:05.348 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:49:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df523dd2236c0be8-00631f7f5d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:05.349 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:05.349 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:05.350 DEBUG:boto:path=/jdpiak-10/?prefix=myobj 2022-09-12T18:50:05.350 DEBUG:boto:auth_path=/jdpiak-10/?prefix=myobj 2022-09-12T18:50:05.351 DEBUG:boto:Method: GET 2022-09-12T18:50:05.351 DEBUG:boto:Path: /jdpiak-10/?prefix=myobj 2022-09-12T18:50:05.352 DEBUG:boto:Data: 2022-09-12T18:50:05.353 DEBUG:boto:Headers: {} 2022-09-12T18:50:05.353 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:50:05.354 DEBUG:boto:Port: 8001 2022-09-12T18:50:05.354 DEBUG:boto:Params: {} 2022-09-12T18:50:05.355 DEBUG:boto:Token: None 2022-09-12T18:50:05.355 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:05 GMT /jdpiak-10/ 2022-09-12T18:50:05.356 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QneLnL0B/J8DNlTvQmGQVfN79QE= 2022-09-12T18:50:05.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QneLnL0B/J8DNlTvQmGQVfN79QE=', 'Content-Length': '0'} 2022-09-12T18:50:05.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000512636e183229b84-00631f7f5d-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:05.360 DEBUG:boto:b'jdpiak-10myobj1000falsemyobj2022-09-12T18:49:46.847Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:05.361 DEBUG:boto:path=/jdpiak-10/ 2022-09-12T18:50:05.361 DEBUG:boto:auth_path=/jdpiak-10/ 2022-09-12T18:50:05.362 DEBUG:boto:path=/jdpiak-10/?prefix=myobj 2022-09-12T18:50:05.362 DEBUG:boto:auth_path=/jdpiak-10/?prefix=myobj 2022-09-12T18:50:05.363 DEBUG:boto:Method: GET 2022-09-12T18:50:05.364 DEBUG:boto:Path: /jdpiak-10/?prefix=myobj 2022-09-12T18:50:05.364 DEBUG:boto:Data: 2022-09-12T18:50:05.365 DEBUG:boto:Headers: {} 2022-09-12T18:50:05.365 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:50:05.366 DEBUG:boto:Port: 8000 2022-09-12T18:50:05.366 DEBUG:boto:Params: {} 2022-09-12T18:50:05.367 DEBUG:boto:Token: None 2022-09-12T18:50:05.368 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:50:05 GMT /jdpiak-10/ 2022-09-12T18:50:05.368 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QneLnL0B/J8DNlTvQmGQVfN79QE= 2022-09-12T18:50:05.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:50:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QneLnL0B/J8DNlTvQmGQVfN79QE=', 'Content-Length': '0'} 2022-09-12T18:50:05.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edb3165c5a399231-00631f7f5d-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:50:05.372 DEBUG:boto:b'jdpiak-10myobj1000falsemyobj2022-09-12T18:49:46.847Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:50:05.373 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-10 zones={test-zone3, test-zone1} 2022-09-12T18:50:05.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:50:05.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:50:05.375 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T18:50:05.439 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:50:05.440 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:50:05.485 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:50:05.481+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:50:05.485 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , ] 2022-09-12T18:50:05.486 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:50:10.487 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:50:10.483+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:50:10.487 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , ] 2022-09-12T18:50:10.487 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:50:15.488 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:50:15.484+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:50:15.488 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , ] 2022-09-12T18:50:15.489 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:50:20.490 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:50:20.486+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:50:20.490 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , ] 2022-09-12T18:50:20.491 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:50:25.491 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:50:25.487+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:50:25.492 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , ] 2022-09-12T18:50:25.492 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:50:30.492 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:50:30.489+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:50:30.493 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , ] 2022-09-12T18:50:30.493 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:50:35.494 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:50:35.490+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:50:35.494 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , ] 2022-09-12T18:50:35.494 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:50:40.495 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:50:40.492+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:50:40.496 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , ] 2022-09-12T18:50:40.496 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:50:45.497 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:50:45.494+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:50:45.498 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , ] 2022-09-12T18:50:45.498 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:50:50.498 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:50:50.494+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:50:50.499 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:50:50.499 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:50:55.499 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:50:55.496+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:50:55.500 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:50:55.500 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:51:00.501 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:51:00.497+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:51:00.502 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:51:00.502 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:51:05.502 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:51:05.498+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:51:05.503 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:51:05.504 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:51:10.504 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:51:10.500+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:51:10.504 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:51:10.504 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:51:15.505 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:51:15.501+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:51:15.506 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:51:15.506 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:51:20.507 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:51:20.502+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:51:20.508 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:51:20.508 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:51:25.508 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:51:25.504+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:51:25.509 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:51:25.509 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:51:30.509 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:51:30.506+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:51:30.510 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:51:30.511 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:51:35.511 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:51:35.507+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:51:35.511 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:51:35.511 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:51:40.512 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:51:40.508+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:51:40.513 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:51:40.513 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:51:45.514 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:51:45.510+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:51:45.514 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:51:45.515 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:51:50.515 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:51:50.511+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:51:50.515 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:51:50.516 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:51:55.516 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:51:55.512+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:51:55.516 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:51:55.517 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:52:00.517 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:52:00.514+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:52:00.518 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:52:00.518 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:52:05.518 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:52:05.515+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:52:05.618 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:52:05.619 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:52:10.520 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:52:10.516+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:52:10.520 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:52:10.521 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:52:15.521 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:52:15.519+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:52:15.522 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:52:15.522 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:52:20.523 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:52:20.520+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:52:20.523 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:52:20.524 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:52:25.524 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:52:25.521+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:52:25.525 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:52:25.525 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:52:30.526 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:52:30.522+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:52:30.526 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:52:30.526 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:52:35.527 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:52:35.523+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:52:35.528 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:52:35.528 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:52:40.529 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:52:40.525+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:52:40.529 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:52:40.530 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:52:45.530 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:52:45.526+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:52:45.531 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:52:45.532 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:52:50.531 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:52:50.528+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:52:50.532 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:52:50.532 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:52:55.533 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:52:55.529+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:52:55.533 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:52:55.533 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:53:00.534 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:53:00.530+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:53:00.534 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:53:00.535 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:53:05.535 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:53:05.533+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:53:05.536 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:53:05.536 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:53:10.537 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:53:10.533+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:53:10.538 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:53:10.538 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:53:15.538 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:53:15.534+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:53:15.538 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:53:15.539 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:53:20.539 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:53:20.536+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:53:20.540 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:53:20.540 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:53:25.541 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:53:25.537+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:53:25.541 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:53:25.542 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:53:30.542 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:53:30.539+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:53:30.543 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:53:30.543 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:53:35.543 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:53:35.540+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:53:35.544 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:53:35.544 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:53:40.545 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:53:40.542+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:53:40.546 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:53:40.546 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:53:45.547 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:53:45.543+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:53:45.548 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:53:45.548 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:53:50.550 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:53:50.545+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:53:50.550 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:53:50.551 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:53:55.550 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:53:55.546+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:53:55.551 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:53:55.552 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:54:00.551 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:54:00.548+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:54:00.552 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:54:00.552 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:54:05.553 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:54:05.549+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:54:05.553 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:54:05.553 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:54:10.554 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:54:10.550+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:54:10.555 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:54:10.555 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:54:15.556 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:54:15.552+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:54:15.556 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:54:15.557 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:54:20.557 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:54:20.554+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:54:20.558 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:54:20.559 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:54:25.559 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:54:25.555+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:54:25.559 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:54:25.560 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:54:30.560 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:54:30.557+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:54:30.560 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:54:30.561 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:54:35.562 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:54:35.559+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:54:35.562 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:54:35.563 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:54:40.563 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:54:40.560+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:54:40.563 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:54:40.564 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:54:45.564 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:54:45.561+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:54:45.565 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:54:45.565 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:54:50.566 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:54:50.562+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:54:50.566 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:54:50.567 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:54:55.568 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:54:55.564+0000 7f769546a600 1 waiting for incremental sync to catch up: 2022-09-12T18:54:55.568 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.7.5, , 00000000001.4.5, , , , , 00000000001.9.5, , ] 2022-09-12T18:54:55.569 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.7.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.9.5, , ] 2022-09-12T18:55:00.569 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:55:00.566+0000 7f769546a600 0 bucket checkpoint timed out waiting for incremental sync to catch up 2022-09-12T18:55:00.570 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:55:00.566+0000 7f769546a600 0 bucket sync checkpoint failed: (110) Connection timed out 2022-09-12T18:55:00.570 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:55:00.566+0000 7f769546a600 -1 bucket sync checkpoint failed: (110) Connection timed out 2022-09-12T18:55:00.589 DEBUG:teuthology.orchestra.run:got remote process result: 110 2022-09-12T18:55:00.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... FAIL 2022-09-12T18:55:00.599 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-13 2022-09-12T18:55:00.600 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:55:00.601 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:55:00.602 DEBUG:boto:Method: PUT 2022-09-12T18:55:00.602 DEBUG:boto:Path: /jdpiak-13/ 2022-09-12T18:55:00.603 DEBUG:boto:Data: 2022-09-12T18:55:00.603 DEBUG:boto:Headers: {} 2022-09-12T18:55:00.604 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:00.604 DEBUG:boto:Port: 8000 2022-09-12T18:55:00.605 DEBUG:boto:Params: {} 2022-09-12T18:55:00.606 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T18:55:00.607 DEBUG:boto:Token: None 2022-09-12T18:55:00.608 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:55:00 GMT /jdpiak-13/ 2022-09-12T18:55:00.608 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6xKKPSOpMQP29cBsnvGSNx9bzgg= 2022-09-12T18:55:00.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6xKKPSOpMQP29cBsnvGSNx9bzgg=', 'Content-Length': '0'} 2022-09-12T18:55:00.636 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000730523ff60e9e494-00631f8084-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:00.638 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jdpiak-14 2022-09-12T18:55:00.639 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:55:00.639 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:55:00.640 DEBUG:boto:Method: PUT 2022-09-12T18:55:00.641 DEBUG:boto:Path: /jdpiak-14/ 2022-09-12T18:55:00.642 DEBUG:boto:Data: 2022-09-12T18:55:00.643 DEBUG:boto:Headers: {} 2022-09-12T18:55:00.644 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:00.645 DEBUG:boto:Port: 8000 2022-09-12T18:55:00.646 DEBUG:boto:Params: {} 2022-09-12T18:55:00.647 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T18:55:00.648 DEBUG:boto:Token: None 2022-09-12T18:55:00.649 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:55:00 GMT /jdpiak-14/ 2022-09-12T18:55:00.650 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jX0Ux7YiQIkbkYFrsoRaDMneiWY= 2022-09-12T18:55:00.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jX0Ux7YiQIkbkYFrsoRaDMneiWY=', 'Content-Length': '0'} 2022-09-12T18:55:00.693 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000006763ad85e4dab7-00631f8084-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:00.694 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jdpiak-15 2022-09-12T18:55:00.695 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:55:00.695 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:55:00.696 DEBUG:boto:Method: PUT 2022-09-12T18:55:00.696 DEBUG:boto:Path: /jdpiak-15/ 2022-09-12T18:55:00.697 DEBUG:boto:Data: 2022-09-12T18:55:00.697 DEBUG:boto:Headers: {} 2022-09-12T18:55:00.698 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:00.698 DEBUG:boto:Port: 8001 2022-09-12T18:55:00.699 DEBUG:boto:Params: {} 2022-09-12T18:55:00.699 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T18:55:00.700 DEBUG:boto:Token: None 2022-09-12T18:55:00.700 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:55:00 GMT /jdpiak-15/ 2022-09-12T18:55:00.701 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wd/aYmcHZHVTjdUOc9CFvOb8imM= 2022-09-12T18:55:00.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wd/aYmcHZHVTjdUOc9CFvOb8imM=', 'Content-Length': '0'} 2022-09-12T18:55:00.734 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000daeac476ea692070-00631f8084-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:00.735 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:55:00.736 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:55:00.736 DEBUG:boto:Method: HEAD 2022-09-12T18:55:00.737 DEBUG:boto:Path: /jdpiak-13/ 2022-09-12T18:55:00.737 DEBUG:boto:Data: 2022-09-12T18:55:00.738 DEBUG:boto:Headers: {} 2022-09-12T18:55:00.738 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:00.739 DEBUG:boto:Port: 8000 2022-09-12T18:55:00.739 DEBUG:boto:Params: {} 2022-09-12T18:55:00.740 DEBUG:boto:Token: None 2022-09-12T18:55:00.741 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:00 GMT /jdpiak-13/ 2022-09-12T18:55:00.741 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:myfuSnq0HZDgfiDGujYg/AXNlNE= 2022-09-12T18:55:00.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:myfuSnq0HZDgfiDGujYg/AXNlNE=', 'Content-Length': '0'} 2022-09-12T18:55:00.744 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000545cf4fcf43da543-00631f8084-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:00.746 DEBUG:boto:path=/jdpiak-13/myobj 2022-09-12T18:55:00.746 DEBUG:boto:auth_path=/jdpiak-13/myobj 2022-09-12T18:55:00.747 DEBUG:boto:Method: PUT 2022-09-12T18:55:00.747 DEBUG:boto:Path: /jdpiak-13/myobj 2022-09-12T18:55:00.748 DEBUG:boto:Data: 2022-09-12T18:55:00.748 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T18:55:00.749 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:00.749 DEBUG:boto:Port: 8000 2022-09-12T18:55:00.750 DEBUG:boto:Params: {} 2022-09-12T18:55:00.750 DEBUG:boto:Token: None 2022-09-12T18:55:00.751 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 18:55:00 GMT /jdpiak-13/myobj 2022-09-12T18:55:00.752 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vNLEM2bepUtNOLEorBlGqM2v1E0= 2022-09-12T18:55:00.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:55:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vNLEM2bepUtNOLEorBlGqM2v1E0='} 2022-09-12T18:55:00.759 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f583aba3d8e85760-00631f8084-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:00.760 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:55:00.761 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:55:00.761 DEBUG:boto:Method: HEAD 2022-09-12T18:55:00.761 DEBUG:boto:Path: /jdpiak-14/ 2022-09-12T18:55:00.762 DEBUG:boto:Data: 2022-09-12T18:55:00.762 DEBUG:boto:Headers: {} 2022-09-12T18:55:00.763 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:00.763 DEBUG:boto:Port: 8000 2022-09-12T18:55:00.764 DEBUG:boto:Params: {} 2022-09-12T18:55:00.764 DEBUG:boto:Token: None 2022-09-12T18:55:00.765 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:00 GMT /jdpiak-14/ 2022-09-12T18:55:00.766 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oyET9Q9jsyAzJreuX2LkIwN71oA= 2022-09-12T18:55:00.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oyET9Q9jsyAzJreuX2LkIwN71oA=', 'Content-Length': '0'} 2022-09-12T18:55:00.769 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000342736aea451d2bb-00631f8084-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:00.769 DEBUG:boto:path=/jdpiak-14/myobj 2022-09-12T18:55:00.770 DEBUG:boto:auth_path=/jdpiak-14/myobj 2022-09-12T18:55:00.771 DEBUG:boto:Method: PUT 2022-09-12T18:55:00.771 DEBUG:boto:Path: /jdpiak-14/myobj 2022-09-12T18:55:00.772 DEBUG:boto:Data: 2022-09-12T18:55:00.772 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T18:55:00.773 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:00.773 DEBUG:boto:Port: 8000 2022-09-12T18:55:00.774 DEBUG:boto:Params: {} 2022-09-12T18:55:00.774 DEBUG:boto:Token: None 2022-09-12T18:55:00.775 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 18:55:00 GMT /jdpiak-14/myobj 2022-09-12T18:55:00.775 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mTK91ChOb19HAqK9fCn5Z+FiooI= 2022-09-12T18:55:00.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:55:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mTK91ChOb19HAqK9fCn5Z+FiooI='} 2022-09-12T18:55:00.784 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a22b4c7d08af645-00631f8084-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:00.785 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:55:00.785 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:55:00.786 DEBUG:boto:Method: HEAD 2022-09-12T18:55:00.786 DEBUG:boto:Path: /jdpiak-15/ 2022-09-12T18:55:00.787 DEBUG:boto:Data: 2022-09-12T18:55:00.788 DEBUG:boto:Headers: {} 2022-09-12T18:55:00.788 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:00.789 DEBUG:boto:Port: 8001 2022-09-12T18:55:00.789 DEBUG:boto:Params: {} 2022-09-12T18:55:00.790 DEBUG:boto:Token: None 2022-09-12T18:55:00.791 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:00 GMT /jdpiak-15/ 2022-09-12T18:55:00.791 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:26IIARqrkCG+rAM197hq2uYhLVs= 2022-09-12T18:55:00.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:26IIARqrkCG+rAM197hq2uYhLVs=', 'Content-Length': '0'} 2022-09-12T18:55:00.795 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000056bcdf6cb8eebc13-00631f8084-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:00.796 DEBUG:boto:path=/jdpiak-15/myobj 2022-09-12T18:55:00.796 DEBUG:boto:auth_path=/jdpiak-15/myobj 2022-09-12T18:55:00.797 DEBUG:boto:Method: PUT 2022-09-12T18:55:00.797 DEBUG:boto:Path: /jdpiak-15/myobj 2022-09-12T18:55:00.798 DEBUG:boto:Data: 2022-09-12T18:55:00.798 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T18:55:00.799 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:00.799 DEBUG:boto:Port: 8001 2022-09-12T18:55:00.800 DEBUG:boto:Params: {} 2022-09-12T18:55:00.800 DEBUG:boto:Token: None 2022-09-12T18:55:00.801 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 18:55:00 GMT /jdpiak-15/myobj 2022-09-12T18:55:00.801 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fQcTwjyPyiYaQV7BvsbFJyVkWME= 2022-09-12T18:55:00.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:55:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fQcTwjyPyiYaQV7BvsbFJyVkWME='} 2022-09-12T18:55:00.810 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004bb4074901c5ca59-00631f8084-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:00.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:55:00.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:55:00.813 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:55:00.869 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:55:00.870 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:55:00.917 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=c0f7a512-c940-4d1f-8adf-3484630f876b 2022-09-12T18:55:00.940 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T18:55:00.940 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:55:00.941 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:55:00.941 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:47:27.976965Z" 2022-09-12T18:55:00.941 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:55:00.942 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:55:00.942 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:55:00.943 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T18:55:00.943 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:55:00.944 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:55:00.944 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008900.721341_113.1", 2022-09-12T18:55:00.945 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:55:00.721341Z" 2022-09-12T18:55:00.945 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:55:00.945 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:55:00.946 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T18:55:00.946 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:49:40.904048Z" 2022-09-12T18:55:00.947 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:55:00.947 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T18:55:00.949 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663008900.721341_113.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:55:00.950 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T18:55:00.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:55:00.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:55:00.952 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:55:01.003 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:55:01.004 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:55:01.041 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:55:01.037+0000 7f5c063cc600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:55:01.041 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:55:01.038+0000 7f5c063cc600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:55:01.064 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:55:01.065 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:55:01.065 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:55:01.065 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:55:01.066 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:55:01.066 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:55:01.066 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:55:01.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:55:01.067 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:55:01.067 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:55:01.068 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:55:01.068 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:55:01.068 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:55:01.068 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:55:01.069 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:55:01.069 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:55:01.069 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:55:01.069 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:55:01.070 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:55:01.070 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:55:01.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:55:01.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:55:01.071 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:55:01.071 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:55:01.071 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:55:01.072 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:55:01.072 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:55:01.072 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:55:01.073 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:55:01.073 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:55:01.073 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:55:01.073 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:55:01.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:55:01.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:55:01.074 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:55:01.074 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:55:01.075 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:55:01.075 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008900.721341_113.1", 2022-09-12T18:55:01.075 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:55:01.076 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:55:01.076 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:55:01.076 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.721341Z", 2022-09-12T18:55:01.076 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:55:01.077 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:55:01.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:55:01.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:55:01.078 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:55:01.078 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:55:01.078 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:55:01.079 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T18:55:01.079 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:55:01.079 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:55:01.079 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:55:01.080 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T18:55:01.080 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:55:01.080 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:55:01.080 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:55:01.081 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:55:01.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:55:01.081 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:55:01.081 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:55:01.082 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:55:01.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:55:01.082 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:55:01Z" 2022-09-12T18:55:01.082 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:55:01.084 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663008900.721341_113.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:55:00.721341Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:55:01Z" } 2022-09-12T18:55:01.084 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663008900.721341_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:55:00.721341Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T18:55:01.085 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663008900.721341_113.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:55:01.085 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663008900.721341_113.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:55:01.086 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T18:55:01.086 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T18:55:01.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:55:01.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:55:01.088 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:55:01.142 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:55:01.143 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:55:01.195 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:55:01.192+0000 7f79f9a6a600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:55:01.197 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:55:01.193+0000 7f79f9a6a600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:55:01.223 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T18:55:01.223 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T18:55:01.223 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T18:55:01.224 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T18:55:01.224 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T18:55:01.225 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:55:01.225 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:55:01.226 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:55:01.226 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T18:55:01.227 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:55:01.227 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T18:55:01.228 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:55:01.228 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:55:01.229 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:55:01.229 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:55:01.230 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T18:55:01.230 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:55:01.231 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:55:01.231 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:55:01.231 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:55:01.232 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:55:01.232 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:55:01.233 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T18:55:01.233 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:55:01.233 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:55:01.234 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:55:01.234 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:55:01.234 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:55:01.234 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:55:01.235 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T18:55:01.235 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:55:01.236 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:55:01.236 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:55:01.236 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:55:01.237 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T18:55:01.237 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:55:01.237 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:55:01.237 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008900.721341_113.1", 2022-09-12T18:55:01.238 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:55:01.238 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:55:01.238 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:55:01.238 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:55:00.721341Z", 2022-09-12T18:55:01.239 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:55:01.239 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:55:01.239 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:55:01.239 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:55:01.240 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T18:55:01.240 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:55:01.240 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:55:01.240 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T18:55:01.241 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:55:01.241 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:55:01.241 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:55:01.242 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T18:55:01.242 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:55:01.242 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:55:01.243 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:55:01.243 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T18:55:01.243 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:55:01.243 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T18:55:01.244 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T18:55:01.244 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T18:55:01.244 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:55:01.244 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T18:55:01Z" 2022-09-12T18:55:01.245 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T18:55:01.246 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663008900.721341_113.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:55:00.721341Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:55:01Z" } 2022-09-12T18:55:01.247 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663008900.721341_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:55:00.721341Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T18:55:01.247 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663008900.721341_113.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:55:01.248 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663008900.721341_113.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:55:01.248 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T18:55:01.249 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T18:55:01.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:55:01.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:55:01.251 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:55:01.301 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:55:01.302 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:55:01.349 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:55:01.345+0000 7fe3dee40600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:55:01.350 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:55:01.346+0000 7fe3dee40600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:55:01.369 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:55:01.369 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:55:01.370 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:55:01.370 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:55:01.370 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:55:01.370 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:55:01.371 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:55:01.371 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:55:01.371 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:55:01.372 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:55:01.372 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:55:01.372 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:55:01.372 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:55:01.373 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:55:01.374 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:55:01.374 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:55:01.375 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:55:01.375 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:55:01.375 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:55:01.375 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:55:01.376 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:55:01.376 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:55:01.376 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:55:01.376 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:55:01.377 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:55:01.377 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:55:01.377 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:55:01.377 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:55:01.378 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:55:01.378 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:55:01.378 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:55:01.379 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:55:01.379 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:55:01.379 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:55:01.379 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:55:01.380 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:55:01.380 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:55:01.380 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008900.721341_113.1", 2022-09-12T18:55:01.380 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:55:01.381 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:55:01.381 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:55:01.381 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.721341Z", 2022-09-12T18:55:01.382 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:55:01.382 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:55:01.382 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:55:01.382 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:55:01.383 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:55:01.383 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:55:01.383 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:55:01.384 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T18:55:01.384 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:55:01.384 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:55:01.384 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:55:01.385 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T18:55:01.385 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:55:01.385 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:55:01.386 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:55:01.386 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:55:01.387 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:55:01.387 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:55:01.387 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:55:01.388 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:55:01.388 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:55:01.388 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:55:01Z" 2022-09-12T18:55:01.389 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:55:01.390 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663008900.721341_113.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:55:00.721341Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:55:01Z" } 2022-09-12T18:55:01.391 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663008900.721341_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:55:00.721341Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T18:55:01.392 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663008900.721341_113.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:55:01.393 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663008900.721341_113.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:55:01.394 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T18:55:01.395 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:55:01.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:55:01.397 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T18:55:01.453 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:55:01.454 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:55:01.488 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:55:01.485+0000 7ff16ab27600 1 waiting to reach incremental sync.. 2022-09-12T18:55:06.491 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:55:06.487+0000 7ff16ab27600 1 bucket sync caught up with source: 2022-09-12T18:55:06.491 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:55:06.491 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:55:06.492 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:55:06.487+0000 7ff16ab27600 0 bucket checkpoint complete 2022-09-12T18:55:06.507 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-13 zones={test-zone2, test-zone1} 2022-09-12T18:55:06.509 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:55:06.509 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:55:06.510 DEBUG:boto:Method: HEAD 2022-09-12T18:55:06.510 DEBUG:boto:Path: /jdpiak-13/ 2022-09-12T18:55:06.511 DEBUG:boto:Data: 2022-09-12T18:55:06.511 DEBUG:boto:Headers: {} 2022-09-12T18:55:06.512 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:06.512 DEBUG:boto:Port: 8000 2022-09-12T18:55:06.513 DEBUG:boto:Params: {} 2022-09-12T18:55:06.513 DEBUG:boto:Token: None 2022-09-12T18:55:06.514 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:06 GMT /jdpiak-13/ 2022-09-12T18:55:06.515 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rchG4pPhoeG/0J620bFtfx03BNE= 2022-09-12T18:55:06.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rchG4pPhoeG/0J620bFtfx03BNE=', 'Content-Length': '0'} 2022-09-12T18:55:06.519 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a90915098266abd2-00631f808a-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:55:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:06.520 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:55:06.521 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:55:06.522 DEBUG:boto:Method: HEAD 2022-09-12T18:55:06.523 DEBUG:boto:Path: /jdpiak-13/ 2022-09-12T18:55:06.524 DEBUG:boto:Data: 2022-09-12T18:55:06.525 DEBUG:boto:Headers: {} 2022-09-12T18:55:06.526 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:06.526 DEBUG:boto:Port: 8000 2022-09-12T18:55:06.527 DEBUG:boto:Params: {} 2022-09-12T18:55:06.529 DEBUG:boto:Token: None 2022-09-12T18:55:06.530 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:06 GMT /jdpiak-13/ 2022-09-12T18:55:06.530 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rchG4pPhoeG/0J620bFtfx03BNE= 2022-09-12T18:55:06.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rchG4pPhoeG/0J620bFtfx03BNE=', 'Content-Length': '0'} 2022-09-12T18:55:06.534 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000941acbbf060405ab-00631f808a-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:55:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:06.535 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T18:55:06.535 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:55:06.536 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:55:06.536 DEBUG:boto:path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:55:06.537 DEBUG:boto:auth_path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:55:06.537 DEBUG:boto:Method: GET 2022-09-12T18:55:06.538 DEBUG:boto:Path: /jdpiak-13/?versions&max-keys=999 2022-09-12T18:55:06.538 DEBUG:boto:Data: 2022-09-12T18:55:06.539 DEBUG:boto:Headers: {} 2022-09-12T18:55:06.540 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:06.540 DEBUG:boto:Port: 8000 2022-09-12T18:55:06.541 DEBUG:boto:Params: {} 2022-09-12T18:55:06.542 DEBUG:boto:Token: None 2022-09-12T18:55:06.543 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:06 GMT /jdpiak-13/?versions 2022-09-12T18:55:06.544 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vypeR/F/4NIGdirdW6h6rk+PyXE= 2022-09-12T18:55:06.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vypeR/F/4NIGdirdW6h6rk+PyXE=', 'Content-Length': '0'} 2022-09-12T18:55:06.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ff5f256381ac359-00631f808a-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:06.549 DEBUG:boto:b'jdpiak-13999falsemyobjnulltrue2022-09-12T18:55:00.753Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:06.551 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-12T18:55:06.551 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T18:55:06.552 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:55:06.553 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:55:06.554 DEBUG:boto:path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:55:06.554 DEBUG:boto:auth_path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:55:06.555 DEBUG:boto:Method: GET 2022-09-12T18:55:06.556 DEBUG:boto:Path: /jdpiak-13/?versions&max-keys=999 2022-09-12T18:55:06.557 DEBUG:boto:Data: 2022-09-12T18:55:06.558 DEBUG:boto:Headers: {} 2022-09-12T18:55:06.559 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:06.560 DEBUG:boto:Port: 8000 2022-09-12T18:55:06.560 DEBUG:boto:Params: {} 2022-09-12T18:55:06.561 DEBUG:boto:Token: None 2022-09-12T18:55:06.562 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:06 GMT /jdpiak-13/?versions 2022-09-12T18:55:06.563 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vypeR/F/4NIGdirdW6h6rk+PyXE= 2022-09-12T18:55:06.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vypeR/F/4NIGdirdW6h6rk+PyXE=', 'Content-Length': '0'} 2022-09-12T18:55:06.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072cceae98b7321fe-00631f808a-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:06.567 DEBUG:boto:b'jdpiak-13999falsemyobjnulltrue2022-09-12T18:55:00.753Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:06.568 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-12T18:55:06.568 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:55:06.569 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:55:06.569 DEBUG:boto:path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:55:06.570 DEBUG:boto:auth_path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:55:06.571 DEBUG:boto:Method: GET 2022-09-12T18:55:06.572 DEBUG:boto:Path: /jdpiak-13/?versions&max-keys=999 2022-09-12T18:55:06.573 DEBUG:boto:Data: 2022-09-12T18:55:06.574 DEBUG:boto:Headers: {} 2022-09-12T18:55:06.575 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:06.576 DEBUG:boto:Port: 8000 2022-09-12T18:55:06.576 DEBUG:boto:Params: {} 2022-09-12T18:55:06.577 DEBUG:boto:Token: None 2022-09-12T18:55:06.578 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:06 GMT /jdpiak-13/?versions 2022-09-12T18:55:06.579 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vypeR/F/4NIGdirdW6h6rk+PyXE= 2022-09-12T18:55:06.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vypeR/F/4NIGdirdW6h6rk+PyXE=', 'Content-Length': '0'} 2022-09-12T18:55:06.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff41f43e8910af2d-00631f808a-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:06.583 DEBUG:boto:b'jdpiak-13999falsemyobjnulltrue2022-09-12T18:55:00.753Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:06.584 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:55:06.584 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:55:06.585 DEBUG:boto:path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:55:06.585 DEBUG:boto:auth_path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:55:06.586 DEBUG:boto:Method: GET 2022-09-12T18:55:06.587 DEBUG:boto:Path: /jdpiak-13/?versions&max-keys=999 2022-09-12T18:55:06.587 DEBUG:boto:Data: 2022-09-12T18:55:06.588 DEBUG:boto:Headers: {} 2022-09-12T18:55:06.588 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:06.589 DEBUG:boto:Port: 8000 2022-09-12T18:55:06.589 DEBUG:boto:Params: {} 2022-09-12T18:55:06.590 DEBUG:boto:Token: None 2022-09-12T18:55:06.591 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:06 GMT /jdpiak-13/?versions 2022-09-12T18:55:06.591 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vypeR/F/4NIGdirdW6h6rk+PyXE= 2022-09-12T18:55:06.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vypeR/F/4NIGdirdW6h6rk+PyXE=', 'Content-Length': '0'} 2022-09-12T18:55:06.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc3a53ac41b043a0-00631f808a-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:06.595 DEBUG:boto:b'jdpiak-13999falsemyobjnulltrue2022-09-12T18:55:00.753Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:06.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-12T18:55:06.597 DEBUG:boto:path=/jdpiak-13/myobj 2022-09-12T18:55:06.597 DEBUG:boto:auth_path=/jdpiak-13/myobj 2022-09-12T18:55:06.598 DEBUG:boto:path=/jdpiak-13/myobj?versionId=null 2022-09-12T18:55:06.598 DEBUG:boto:auth_path=/jdpiak-13/myobj?versionId=null 2022-09-12T18:55:06.599 DEBUG:boto:Method: GET 2022-09-12T18:55:06.600 DEBUG:boto:Path: /jdpiak-13/myobj?versionId=null 2022-09-12T18:55:06.600 DEBUG:boto:Data: 2022-09-12T18:55:06.601 DEBUG:boto:Headers: {} 2022-09-12T18:55:06.601 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:06.602 DEBUG:boto:Port: 8000 2022-09-12T18:55:06.602 DEBUG:boto:Params: {} 2022-09-12T18:55:06.603 DEBUG:boto:Token: None 2022-09-12T18:55:06.604 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:06 GMT /jdpiak-13/myobj?versionId=null 2022-09-12T18:55:06.605 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3m/hRwD7oUH6WCcdm0eWeCirEvE= 2022-09-12T18:55:06.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3m/hRwD7oUH6WCcdm0eWeCirEvE=', 'Content-Length': '0'} 2022-09-12T18:55:06.609 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000714906461d535ead-00631f808a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:06.609 DEBUG:boto:path=/jdpiak-13/myobj 2022-09-12T18:55:06.610 DEBUG:boto:auth_path=/jdpiak-13/myobj 2022-09-12T18:55:06.611 DEBUG:boto:path=/jdpiak-13/myobj?versionId=null 2022-09-12T18:55:06.611 DEBUG:boto:auth_path=/jdpiak-13/myobj?versionId=null 2022-09-12T18:55:06.612 DEBUG:boto:Method: GET 2022-09-12T18:55:06.612 DEBUG:boto:Path: /jdpiak-13/myobj?versionId=null 2022-09-12T18:55:06.613 DEBUG:boto:Data: 2022-09-12T18:55:06.613 DEBUG:boto:Headers: {} 2022-09-12T18:55:06.614 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:06.615 DEBUG:boto:Port: 8000 2022-09-12T18:55:06.616 DEBUG:boto:Params: {} 2022-09-12T18:55:06.617 DEBUG:boto:Token: None 2022-09-12T18:55:06.618 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:06 GMT /jdpiak-13/myobj?versionId=null 2022-09-12T18:55:06.619 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3m/hRwD7oUH6WCcdm0eWeCirEvE= 2022-09-12T18:55:06.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3m/hRwD7oUH6WCcdm0eWeCirEvE=', 'Content-Length': '0'} 2022-09-12T18:55:06.623 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004be1fc9d9398f275-00631f808a-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:06.625 DEBUG:boto:path=/jdpiak-13/myobj 2022-09-12T18:55:06.625 DEBUG:boto:auth_path=/jdpiak-13/myobj 2022-09-12T18:55:06.626 DEBUG:boto:path=/jdpiak-13/myobj?versionId=null 2022-09-12T18:55:06.627 DEBUG:boto:auth_path=/jdpiak-13/myobj?versionId=null 2022-09-12T18:55:06.627 DEBUG:boto:Method: HEAD 2022-09-12T18:55:06.628 DEBUG:boto:Path: /jdpiak-13/myobj?versionId=null 2022-09-12T18:55:06.628 DEBUG:boto:Data: 2022-09-12T18:55:06.629 DEBUG:boto:Headers: {} 2022-09-12T18:55:06.629 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:06.630 DEBUG:boto:Port: 8000 2022-09-12T18:55:06.630 DEBUG:boto:Params: {} 2022-09-12T18:55:06.631 DEBUG:boto:Token: None 2022-09-12T18:55:06.631 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:06 GMT /jdpiak-13/myobj?versionId=null 2022-09-12T18:55:06.632 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7pcC1xmOur9tBD/jIWQEmI8H9Io= 2022-09-12T18:55:06.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7pcC1xmOur9tBD/jIWQEmI8H9Io=', 'Content-Length': '0'} 2022-09-12T18:55:06.635 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a35ec19236745ef7-00631f808a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:06.636 DEBUG:boto:path=/jdpiak-13/myobj 2022-09-12T18:55:06.637 DEBUG:boto:auth_path=/jdpiak-13/myobj 2022-09-12T18:55:06.637 DEBUG:boto:path=/jdpiak-13/myobj?versionId=null 2022-09-12T18:55:06.638 DEBUG:boto:auth_path=/jdpiak-13/myobj?versionId=null 2022-09-12T18:55:06.639 DEBUG:boto:Method: HEAD 2022-09-12T18:55:06.640 DEBUG:boto:Path: /jdpiak-13/myobj?versionId=null 2022-09-12T18:55:06.641 DEBUG:boto:Data: 2022-09-12T18:55:06.642 DEBUG:boto:Headers: {} 2022-09-12T18:55:06.643 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:06.643 DEBUG:boto:Port: 8000 2022-09-12T18:55:06.644 DEBUG:boto:Params: {} 2022-09-12T18:55:06.645 DEBUG:boto:Token: None 2022-09-12T18:55:06.646 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:06 GMT /jdpiak-13/myobj?versionId=null 2022-09-12T18:55:06.647 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7pcC1xmOur9tBD/jIWQEmI8H9Io= 2022-09-12T18:55:06.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7pcC1xmOur9tBD/jIWQEmI8H9Io=', 'Content-Length': '0'} 2022-09-12T18:55:06.650 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c6ee30e363fd592-00631f808a-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:06.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-12T18:55:06.651 DEBUG:boto:path=/jdpiak-13/myobj 2022-09-12T18:55:06.652 DEBUG:boto:auth_path=/jdpiak-13/myobj 2022-09-12T18:55:06.652 DEBUG:boto:path=/jdpiak-13/myobj?versionId=null 2022-09-12T18:55:06.653 DEBUG:boto:auth_path=/jdpiak-13/myobj?versionId=null 2022-09-12T18:55:06.653 DEBUG:boto:Method: GET 2022-09-12T18:55:06.654 DEBUG:boto:Path: /jdpiak-13/myobj?versionId=null 2022-09-12T18:55:06.654 DEBUG:boto:Data: 2022-09-12T18:55:06.655 DEBUG:boto:Headers: {} 2022-09-12T18:55:06.655 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:06.656 DEBUG:boto:Port: 8000 2022-09-12T18:55:06.656 DEBUG:boto:Params: {} 2022-09-12T18:55:06.657 DEBUG:boto:Token: None 2022-09-12T18:55:06.658 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:06 GMT /jdpiak-13/myobj?versionId=null 2022-09-12T18:55:06.658 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3m/hRwD7oUH6WCcdm0eWeCirEvE= 2022-09-12T18:55:06.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3m/hRwD7oUH6WCcdm0eWeCirEvE=', 'Content-Length': '0'} 2022-09-12T18:55:06.661 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000762289a8526ec5a0-00631f808a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:06.706 DEBUG:boto:path=/jdpiak-13/myobj 2022-09-12T18:55:06.707 DEBUG:boto:auth_path=/jdpiak-13/myobj 2022-09-12T18:55:06.708 DEBUG:boto:path=/jdpiak-13/myobj?versionId=null 2022-09-12T18:55:06.709 DEBUG:boto:auth_path=/jdpiak-13/myobj?versionId=null 2022-09-12T18:55:06.710 DEBUG:boto:Method: GET 2022-09-12T18:55:06.711 DEBUG:boto:Path: /jdpiak-13/myobj?versionId=null 2022-09-12T18:55:06.712 DEBUG:boto:Data: 2022-09-12T18:55:06.713 DEBUG:boto:Headers: {} 2022-09-12T18:55:06.714 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:06.715 DEBUG:boto:Port: 8000 2022-09-12T18:55:06.715 DEBUG:boto:Params: {} 2022-09-12T18:55:06.716 DEBUG:boto:Token: None 2022-09-12T18:55:06.717 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:06 GMT /jdpiak-13/myobj?versionId=null 2022-09-12T18:55:06.717 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3m/hRwD7oUH6WCcdm0eWeCirEvE= 2022-09-12T18:55:06.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3m/hRwD7oUH6WCcdm0eWeCirEvE=', 'Content-Length': '0'} 2022-09-12T18:55:06.721 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf5906d8fe406ae6-00631f808a-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:06.722 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:55:06.723 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:55:06.724 DEBUG:boto:path=/jdpiak-13/?prefix=myobj 2022-09-12T18:55:06.724 DEBUG:boto:auth_path=/jdpiak-13/?prefix=myobj 2022-09-12T18:55:06.725 DEBUG:boto:Method: GET 2022-09-12T18:55:06.725 DEBUG:boto:Path: /jdpiak-13/?prefix=myobj 2022-09-12T18:55:06.726 DEBUG:boto:Data: 2022-09-12T18:55:06.727 DEBUG:boto:Headers: {} 2022-09-12T18:55:06.727 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:06.728 DEBUG:boto:Port: 8000 2022-09-12T18:55:06.728 DEBUG:boto:Params: {} 2022-09-12T18:55:06.729 DEBUG:boto:Token: None 2022-09-12T18:55:06.730 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:06 GMT /jdpiak-13/ 2022-09-12T18:55:06.731 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zuKmt9VXWx8i3HZF0NWo5Vh8Qbc= 2022-09-12T18:55:06.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zuKmt9VXWx8i3HZF0NWo5Vh8Qbc=', 'Content-Length': '0'} 2022-09-12T18:55:06.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000074b4fad258ad823-00631f808a-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:06.735 DEBUG:boto:b'jdpiak-13myobj1000falsemyobj2022-09-12T18:55:00.753Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:06.736 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:55:06.736 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:55:06.737 DEBUG:boto:path=/jdpiak-13/?prefix=myobj 2022-09-12T18:55:06.737 DEBUG:boto:auth_path=/jdpiak-13/?prefix=myobj 2022-09-12T18:55:06.738 DEBUG:boto:Method: GET 2022-09-12T18:55:06.738 DEBUG:boto:Path: /jdpiak-13/?prefix=myobj 2022-09-12T18:55:06.739 DEBUG:boto:Data: 2022-09-12T18:55:06.739 DEBUG:boto:Headers: {} 2022-09-12T18:55:06.740 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:06.740 DEBUG:boto:Port: 8000 2022-09-12T18:55:06.741 DEBUG:boto:Params: {} 2022-09-12T18:55:06.741 DEBUG:boto:Token: None 2022-09-12T18:55:06.742 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:06 GMT /jdpiak-13/ 2022-09-12T18:55:06.743 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zuKmt9VXWx8i3HZF0NWo5Vh8Qbc= 2022-09-12T18:55:06.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zuKmt9VXWx8i3HZF0NWo5Vh8Qbc=', 'Content-Length': '0'} 2022-09-12T18:55:06.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001998ae5507583f79-00631f808a-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:06.747 DEBUG:boto:b'jdpiak-13myobj1000falsemyobj2022-09-12T18:55:00.753Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:06.748 DEBUG:boto:path=/jdpiak-13/myobj 2022-09-12T18:55:06.748 DEBUG:boto:auth_path=/jdpiak-13/myobj 2022-09-12T18:55:06.749 DEBUG:boto:Method: HEAD 2022-09-12T18:55:06.749 DEBUG:boto:Path: /jdpiak-13/myobj 2022-09-12T18:55:06.750 DEBUG:boto:Data: 2022-09-12T18:55:06.750 DEBUG:boto:Headers: {} 2022-09-12T18:55:06.751 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:06.751 DEBUG:boto:Port: 8000 2022-09-12T18:55:06.752 DEBUG:boto:Params: {} 2022-09-12T18:55:06.752 DEBUG:boto:Token: None 2022-09-12T18:55:06.753 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:06 GMT /jdpiak-13/myobj 2022-09-12T18:55:06.753 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0NgMd9kp4cEAuco5ZQk2JZHscLY= 2022-09-12T18:55:06.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0NgMd9kp4cEAuco5ZQk2JZHscLY=', 'Content-Length': '0'} 2022-09-12T18:55:06.756 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014fc0b400a9d57ac-00631f808a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:06.757 DEBUG:boto:path=/jdpiak-13/myobj 2022-09-12T18:55:06.757 DEBUG:boto:auth_path=/jdpiak-13/myobj 2022-09-12T18:55:06.758 DEBUG:boto:Method: HEAD 2022-09-12T18:55:06.759 DEBUG:boto:Path: /jdpiak-13/myobj 2022-09-12T18:55:06.759 DEBUG:boto:Data: 2022-09-12T18:55:06.760 DEBUG:boto:Headers: {} 2022-09-12T18:55:06.760 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:06.761 DEBUG:boto:Port: 8000 2022-09-12T18:55:06.761 DEBUG:boto:Params: {} 2022-09-12T18:55:06.762 DEBUG:boto:Token: None 2022-09-12T18:55:06.762 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:06 GMT /jdpiak-13/myobj 2022-09-12T18:55:06.763 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0NgMd9kp4cEAuco5ZQk2JZHscLY= 2022-09-12T18:55:06.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0NgMd9kp4cEAuco5ZQk2JZHscLY=', 'Content-Length': '0'} 2022-09-12T18:55:06.767 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e712bce862191237-00631f808a-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:06.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-12T18:55:06.769 DEBUG:boto:path=/jdpiak-13/myobj 2022-09-12T18:55:06.770 DEBUG:boto:auth_path=/jdpiak-13/myobj 2022-09-12T18:55:06.771 DEBUG:boto:Method: GET 2022-09-12T18:55:06.772 DEBUG:boto:Path: /jdpiak-13/myobj 2022-09-12T18:55:06.772 DEBUG:boto:Data: 2022-09-12T18:55:06.773 DEBUG:boto:Headers: {} 2022-09-12T18:55:06.773 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:06.774 DEBUG:boto:Port: 8000 2022-09-12T18:55:06.774 DEBUG:boto:Params: {} 2022-09-12T18:55:06.775 DEBUG:boto:Token: None 2022-09-12T18:55:06.775 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:06 GMT /jdpiak-13/myobj 2022-09-12T18:55:06.776 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BuU0u/qTn4YRS/usYNwZRLd/PFU= 2022-09-12T18:55:06.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BuU0u/qTn4YRS/usYNwZRLd/PFU=', 'Content-Length': '0'} 2022-09-12T18:55:06.779 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000078d0066a5f6c1e73-00631f808a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:06.779 DEBUG:boto:path=/jdpiak-13/myobj 2022-09-12T18:55:06.780 DEBUG:boto:auth_path=/jdpiak-13/myobj 2022-09-12T18:55:06.781 DEBUG:boto:Method: GET 2022-09-12T18:55:06.781 DEBUG:boto:Path: /jdpiak-13/myobj 2022-09-12T18:55:06.782 DEBUG:boto:Data: 2022-09-12T18:55:06.782 DEBUG:boto:Headers: {} 2022-09-12T18:55:06.783 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:06.783 DEBUG:boto:Port: 8000 2022-09-12T18:55:06.784 DEBUG:boto:Params: {} 2022-09-12T18:55:06.784 DEBUG:boto:Token: None 2022-09-12T18:55:06.785 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:06 GMT /jdpiak-13/myobj 2022-09-12T18:55:06.785 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BuU0u/qTn4YRS/usYNwZRLd/PFU= 2022-09-12T18:55:06.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BuU0u/qTn4YRS/usYNwZRLd/PFU=', 'Content-Length': '0'} 2022-09-12T18:55:06.788 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000909ecf1f11e71bd8-00631f808a-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:06.789 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:55:06.790 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:55:06.790 DEBUG:boto:path=/jdpiak-13/?prefix=myobj 2022-09-12T18:55:06.791 DEBUG:boto:auth_path=/jdpiak-13/?prefix=myobj 2022-09-12T18:55:06.792 DEBUG:boto:Method: GET 2022-09-12T18:55:06.792 DEBUG:boto:Path: /jdpiak-13/?prefix=myobj 2022-09-12T18:55:06.793 DEBUG:boto:Data: 2022-09-12T18:55:06.794 DEBUG:boto:Headers: {} 2022-09-12T18:55:06.794 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:06.795 DEBUG:boto:Port: 8000 2022-09-12T18:55:06.795 DEBUG:boto:Params: {} 2022-09-12T18:55:06.796 DEBUG:boto:Token: None 2022-09-12T18:55:06.797 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:06 GMT /jdpiak-13/ 2022-09-12T18:55:06.798 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zuKmt9VXWx8i3HZF0NWo5Vh8Qbc= 2022-09-12T18:55:06.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zuKmt9VXWx8i3HZF0NWo5Vh8Qbc=', 'Content-Length': '0'} 2022-09-12T18:55:06.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000789f1434c554ba4a-00631f808a-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:06.842 DEBUG:boto:b'jdpiak-13myobj1000falsemyobj2022-09-12T18:55:00.753Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:06.844 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:55:06.845 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:55:06.846 DEBUG:boto:path=/jdpiak-13/?prefix=myobj 2022-09-12T18:55:06.846 DEBUG:boto:auth_path=/jdpiak-13/?prefix=myobj 2022-09-12T18:55:06.847 DEBUG:boto:Method: GET 2022-09-12T18:55:06.847 DEBUG:boto:Path: /jdpiak-13/?prefix=myobj 2022-09-12T18:55:06.848 DEBUG:boto:Data: 2022-09-12T18:55:06.848 DEBUG:boto:Headers: {} 2022-09-12T18:55:06.849 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:06.850 DEBUG:boto:Port: 8000 2022-09-12T18:55:06.851 DEBUG:boto:Params: {} 2022-09-12T18:55:06.852 DEBUG:boto:Token: None 2022-09-12T18:55:06.853 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:06 GMT /jdpiak-13/ 2022-09-12T18:55:06.853 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zuKmt9VXWx8i3HZF0NWo5Vh8Qbc= 2022-09-12T18:55:06.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zuKmt9VXWx8i3HZF0NWo5Vh8Qbc=', 'Content-Length': '0'} 2022-09-12T18:55:06.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008e25a5c65abbf00-00631f808a-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:06.858 DEBUG:boto:b'jdpiak-13myobj1000falsemyobj2022-09-12T18:55:00.753Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:06.858 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-13 zones={test-zone2, test-zone1} 2022-09-12T18:55:06.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:55:06.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:55:06.860 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T18:55:06.925 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:55:06.926 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:55:06.974 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:55:06.972+0000 7f6a26472600 1 bucket sync caught up with source: 2022-09-12T18:55:06.975 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:55:06.975 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:55:06.976 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:55:06.972+0000 7f6a26472600 0 bucket checkpoint complete 2022-09-12T18:55:06.991 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-13 zones={test-zone3, test-zone1} 2022-09-12T18:55:06.992 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:55:06.993 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:55:06.994 DEBUG:boto:Method: HEAD 2022-09-12T18:55:06.995 DEBUG:boto:Path: /jdpiak-13/ 2022-09-12T18:55:06.995 DEBUG:boto:Data: 2022-09-12T18:55:06.996 DEBUG:boto:Headers: {} 2022-09-12T18:55:06.996 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:06.997 DEBUG:boto:Port: 8001 2022-09-12T18:55:06.998 DEBUG:boto:Params: {} 2022-09-12T18:55:06.999 DEBUG:boto:Token: None 2022-09-12T18:55:07.000 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:07 GMT /jdpiak-13/ 2022-09-12T18:55:07.001 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4jSXbsWmTHDjIaLsApK64CdaxqM= 2022-09-12T18:55:07.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4jSXbsWmTHDjIaLsApK64CdaxqM=', 'Content-Length': '0'} 2022-09-12T18:55:07.005 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000601bb3d6750a28ae-00631f808b-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:55:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:07.005 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:55:07.006 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:55:07.006 DEBUG:boto:Method: HEAD 2022-09-12T18:55:07.007 DEBUG:boto:Path: /jdpiak-13/ 2022-09-12T18:55:07.008 DEBUG:boto:Data: 2022-09-12T18:55:07.009 DEBUG:boto:Headers: {} 2022-09-12T18:55:07.010 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:07.011 DEBUG:boto:Port: 8000 2022-09-12T18:55:07.012 DEBUG:boto:Params: {} 2022-09-12T18:55:07.013 DEBUG:boto:Token: None 2022-09-12T18:55:07.014 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:07 GMT /jdpiak-13/ 2022-09-12T18:55:07.015 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4jSXbsWmTHDjIaLsApK64CdaxqM= 2022-09-12T18:55:07.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4jSXbsWmTHDjIaLsApK64CdaxqM=', 'Content-Length': '0'} 2022-09-12T18:55:07.019 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee818e80bd62b96c-00631f808b-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:55:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:07.020 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T18:55:07.020 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:55:07.021 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:55:07.021 DEBUG:boto:path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:55:07.022 DEBUG:boto:auth_path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:55:07.022 DEBUG:boto:Method: GET 2022-09-12T18:55:07.023 DEBUG:boto:Path: /jdpiak-13/?versions&max-keys=999 2022-09-12T18:55:07.023 DEBUG:boto:Data: 2022-09-12T18:55:07.024 DEBUG:boto:Headers: {} 2022-09-12T18:55:07.024 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:07.025 DEBUG:boto:Port: 8001 2022-09-12T18:55:07.026 DEBUG:boto:Params: {} 2022-09-12T18:55:07.027 DEBUG:boto:Token: None 2022-09-12T18:55:07.028 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:07 GMT /jdpiak-13/?versions 2022-09-12T18:55:07.029 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oeBGSH1YGen9iJQSDqbzXPganu8= 2022-09-12T18:55:07.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oeBGSH1YGen9iJQSDqbzXPganu8=', 'Content-Length': '0'} 2022-09-12T18:55:07.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073329212ad5f6f40-00631f808b-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:07.033 DEBUG:boto:b'jdpiak-13999falsemyobjnulltrue2022-09-12T18:55:00.753Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:07.034 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-12T18:55:07.035 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T18:55:07.036 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:55:07.036 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:55:07.037 DEBUG:boto:path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:55:07.037 DEBUG:boto:auth_path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:55:07.038 DEBUG:boto:Method: GET 2022-09-12T18:55:07.038 DEBUG:boto:Path: /jdpiak-13/?versions&max-keys=999 2022-09-12T18:55:07.039 DEBUG:boto:Data: 2022-09-12T18:55:07.040 DEBUG:boto:Headers: {} 2022-09-12T18:55:07.040 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:07.041 DEBUG:boto:Port: 8000 2022-09-12T18:55:07.041 DEBUG:boto:Params: {} 2022-09-12T18:55:07.042 DEBUG:boto:Token: None 2022-09-12T18:55:07.042 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:07 GMT /jdpiak-13/?versions 2022-09-12T18:55:07.043 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oeBGSH1YGen9iJQSDqbzXPganu8= 2022-09-12T18:55:07.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oeBGSH1YGen9iJQSDqbzXPganu8=', 'Content-Length': '0'} 2022-09-12T18:55:07.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dcc19a26e1de046-00631f808b-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:07.094 DEBUG:boto:b'jdpiak-13999falsemyobjnulltrue2022-09-12T18:55:00.753Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:07.096 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-12T18:55:07.096 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:55:07.097 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:55:07.098 DEBUG:boto:path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:55:07.099 DEBUG:boto:auth_path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:55:07.100 DEBUG:boto:Method: GET 2022-09-12T18:55:07.101 DEBUG:boto:Path: /jdpiak-13/?versions&max-keys=999 2022-09-12T18:55:07.102 DEBUG:boto:Data: 2022-09-12T18:55:07.103 DEBUG:boto:Headers: {} 2022-09-12T18:55:07.104 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:07.105 DEBUG:boto:Port: 8001 2022-09-12T18:55:07.106 DEBUG:boto:Params: {} 2022-09-12T18:55:07.107 DEBUG:boto:Token: None 2022-09-12T18:55:07.108 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:07 GMT /jdpiak-13/?versions 2022-09-12T18:55:07.110 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oeBGSH1YGen9iJQSDqbzXPganu8= 2022-09-12T18:55:07.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oeBGSH1YGen9iJQSDqbzXPganu8=', 'Content-Length': '0'} 2022-09-12T18:55:07.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3adb056717495b5-00631f808b-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:07.115 DEBUG:boto:b'jdpiak-13999falsemyobjnulltrue2022-09-12T18:55:00.753Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:07.117 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:55:07.118 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:55:07.118 DEBUG:boto:path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:55:07.119 DEBUG:boto:auth_path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:55:07.120 DEBUG:boto:Method: GET 2022-09-12T18:55:07.121 DEBUG:boto:Path: /jdpiak-13/?versions&max-keys=999 2022-09-12T18:55:07.122 DEBUG:boto:Data: 2022-09-12T18:55:07.123 DEBUG:boto:Headers: {} 2022-09-12T18:55:07.124 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:07.124 DEBUG:boto:Port: 8000 2022-09-12T18:55:07.125 DEBUG:boto:Params: {} 2022-09-12T18:55:07.126 DEBUG:boto:Token: None 2022-09-12T18:55:07.127 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:07 GMT /jdpiak-13/?versions 2022-09-12T18:55:07.128 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oeBGSH1YGen9iJQSDqbzXPganu8= 2022-09-12T18:55:07.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oeBGSH1YGen9iJQSDqbzXPganu8=', 'Content-Length': '0'} 2022-09-12T18:55:07.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebfa064f27840405-00631f808b-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:07.133 DEBUG:boto:b'jdpiak-13999falsemyobjnulltrue2022-09-12T18:55:00.753Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:07.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-12T18:55:07.135 DEBUG:boto:path=/jdpiak-13/myobj 2022-09-12T18:55:07.136 DEBUG:boto:auth_path=/jdpiak-13/myobj 2022-09-12T18:55:07.137 DEBUG:boto:path=/jdpiak-13/myobj?versionId=null 2022-09-12T18:55:07.138 DEBUG:boto:auth_path=/jdpiak-13/myobj?versionId=null 2022-09-12T18:55:07.139 DEBUG:boto:Method: GET 2022-09-12T18:55:07.139 DEBUG:boto:Path: /jdpiak-13/myobj?versionId=null 2022-09-12T18:55:07.140 DEBUG:boto:Data: 2022-09-12T18:55:07.141 DEBUG:boto:Headers: {} 2022-09-12T18:55:07.142 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:07.143 DEBUG:boto:Port: 8001 2022-09-12T18:55:07.144 DEBUG:boto:Params: {} 2022-09-12T18:55:07.145 DEBUG:boto:Token: None 2022-09-12T18:55:07.146 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:07 GMT /jdpiak-13/myobj?versionId=null 2022-09-12T18:55:07.147 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hFw1vW0bowubhiHvJQQ0nfP/8IE= 2022-09-12T18:55:07.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hFw1vW0bowubhiHvJQQ0nfP/8IE=', 'Content-Length': '0'} 2022-09-12T18:55:07.150 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001edbabf3d47df1d5-00631f808b-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:07.150 DEBUG:boto:path=/jdpiak-13/myobj 2022-09-12T18:55:07.151 DEBUG:boto:auth_path=/jdpiak-13/myobj 2022-09-12T18:55:07.151 DEBUG:boto:path=/jdpiak-13/myobj?versionId=null 2022-09-12T18:55:07.152 DEBUG:boto:auth_path=/jdpiak-13/myobj?versionId=null 2022-09-12T18:55:07.152 DEBUG:boto:Method: GET 2022-09-12T18:55:07.153 DEBUG:boto:Path: /jdpiak-13/myobj?versionId=null 2022-09-12T18:55:07.153 DEBUG:boto:Data: 2022-09-12T18:55:07.154 DEBUG:boto:Headers: {} 2022-09-12T18:55:07.155 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:07.155 DEBUG:boto:Port: 8000 2022-09-12T18:55:07.156 DEBUG:boto:Params: {} 2022-09-12T18:55:07.156 DEBUG:boto:Token: None 2022-09-12T18:55:07.157 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:07 GMT /jdpiak-13/myobj?versionId=null 2022-09-12T18:55:07.157 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hFw1vW0bowubhiHvJQQ0nfP/8IE= 2022-09-12T18:55:07.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hFw1vW0bowubhiHvJQQ0nfP/8IE=', 'Content-Length': '0'} 2022-09-12T18:55:07.160 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e34d506c44889a8-00631f808b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:07.161 DEBUG:boto:path=/jdpiak-13/myobj 2022-09-12T18:55:07.161 DEBUG:boto:auth_path=/jdpiak-13/myobj 2022-09-12T18:55:07.162 DEBUG:boto:path=/jdpiak-13/myobj?versionId=null 2022-09-12T18:55:07.162 DEBUG:boto:auth_path=/jdpiak-13/myobj?versionId=null 2022-09-12T18:55:07.163 DEBUG:boto:Method: HEAD 2022-09-12T18:55:07.163 DEBUG:boto:Path: /jdpiak-13/myobj?versionId=null 2022-09-12T18:55:07.164 DEBUG:boto:Data: 2022-09-12T18:55:07.164 DEBUG:boto:Headers: {} 2022-09-12T18:55:07.165 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:07.165 DEBUG:boto:Port: 8001 2022-09-12T18:55:07.166 DEBUG:boto:Params: {} 2022-09-12T18:55:07.167 DEBUG:boto:Token: None 2022-09-12T18:55:07.167 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:07 GMT /jdpiak-13/myobj?versionId=null 2022-09-12T18:55:07.168 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JRhUfarUf7BBk0IqXr64pPqBnuw= 2022-09-12T18:55:07.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JRhUfarUf7BBk0IqXr64pPqBnuw=', 'Content-Length': '0'} 2022-09-12T18:55:07.170 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a11f8fa9b9b7a7b-00631f808b-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:07.171 DEBUG:boto:path=/jdpiak-13/myobj 2022-09-12T18:55:07.171 DEBUG:boto:auth_path=/jdpiak-13/myobj 2022-09-12T18:55:07.172 DEBUG:boto:path=/jdpiak-13/myobj?versionId=null 2022-09-12T18:55:07.172 DEBUG:boto:auth_path=/jdpiak-13/myobj?versionId=null 2022-09-12T18:55:07.173 DEBUG:boto:Method: HEAD 2022-09-12T18:55:07.173 DEBUG:boto:Path: /jdpiak-13/myobj?versionId=null 2022-09-12T18:55:07.174 DEBUG:boto:Data: 2022-09-12T18:55:07.174 DEBUG:boto:Headers: {} 2022-09-12T18:55:07.175 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:07.176 DEBUG:boto:Port: 8000 2022-09-12T18:55:07.176 DEBUG:boto:Params: {} 2022-09-12T18:55:07.177 DEBUG:boto:Token: None 2022-09-12T18:55:07.177 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:07 GMT /jdpiak-13/myobj?versionId=null 2022-09-12T18:55:07.178 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JRhUfarUf7BBk0IqXr64pPqBnuw= 2022-09-12T18:55:07.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JRhUfarUf7BBk0IqXr64pPqBnuw=', 'Content-Length': '0'} 2022-09-12T18:55:07.181 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028915e809d61bf8d-00631f808b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:07.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-12T18:55:07.182 DEBUG:boto:path=/jdpiak-13/myobj 2022-09-12T18:55:07.183 DEBUG:boto:auth_path=/jdpiak-13/myobj 2022-09-12T18:55:07.184 DEBUG:boto:path=/jdpiak-13/myobj?versionId=null 2022-09-12T18:55:07.186 DEBUG:boto:auth_path=/jdpiak-13/myobj?versionId=null 2022-09-12T18:55:07.187 DEBUG:boto:Method: GET 2022-09-12T18:55:07.188 DEBUG:boto:Path: /jdpiak-13/myobj?versionId=null 2022-09-12T18:55:07.189 DEBUG:boto:Data: 2022-09-12T18:55:07.189 DEBUG:boto:Headers: {} 2022-09-12T18:55:07.190 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:07.190 DEBUG:boto:Port: 8001 2022-09-12T18:55:07.191 DEBUG:boto:Params: {} 2022-09-12T18:55:07.191 DEBUG:boto:Token: None 2022-09-12T18:55:07.192 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:07 GMT /jdpiak-13/myobj?versionId=null 2022-09-12T18:55:07.193 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hFw1vW0bowubhiHvJQQ0nfP/8IE= 2022-09-12T18:55:07.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hFw1vW0bowubhiHvJQQ0nfP/8IE=', 'Content-Length': '0'} 2022-09-12T18:55:07.197 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ef2119b50fe1807-00631f808b-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:07.242 DEBUG:boto:path=/jdpiak-13/myobj 2022-09-12T18:55:07.243 DEBUG:boto:auth_path=/jdpiak-13/myobj 2022-09-12T18:55:07.243 DEBUG:boto:path=/jdpiak-13/myobj?versionId=null 2022-09-12T18:55:07.244 DEBUG:boto:auth_path=/jdpiak-13/myobj?versionId=null 2022-09-12T18:55:07.245 DEBUG:boto:Method: GET 2022-09-12T18:55:07.246 DEBUG:boto:Path: /jdpiak-13/myobj?versionId=null 2022-09-12T18:55:07.247 DEBUG:boto:Data: 2022-09-12T18:55:07.247 DEBUG:boto:Headers: {} 2022-09-12T18:55:07.248 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:07.248 DEBUG:boto:Port: 8000 2022-09-12T18:55:07.249 DEBUG:boto:Params: {} 2022-09-12T18:55:07.249 DEBUG:boto:Token: None 2022-09-12T18:55:07.250 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:07 GMT /jdpiak-13/myobj?versionId=null 2022-09-12T18:55:07.251 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hFw1vW0bowubhiHvJQQ0nfP/8IE= 2022-09-12T18:55:07.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hFw1vW0bowubhiHvJQQ0nfP/8IE=', 'Content-Length': '0'} 2022-09-12T18:55:07.253 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000065003fc652b67113-00631f808b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:07.254 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:55:07.255 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:55:07.255 DEBUG:boto:path=/jdpiak-13/?prefix=myobj 2022-09-12T18:55:07.256 DEBUG:boto:auth_path=/jdpiak-13/?prefix=myobj 2022-09-12T18:55:07.256 DEBUG:boto:Method: GET 2022-09-12T18:55:07.257 DEBUG:boto:Path: /jdpiak-13/?prefix=myobj 2022-09-12T18:55:07.257 DEBUG:boto:Data: 2022-09-12T18:55:07.258 DEBUG:boto:Headers: {} 2022-09-12T18:55:07.259 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:07.259 DEBUG:boto:Port: 8001 2022-09-12T18:55:07.260 DEBUG:boto:Params: {} 2022-09-12T18:55:07.261 DEBUG:boto:Token: None 2022-09-12T18:55:07.261 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:07 GMT /jdpiak-13/ 2022-09-12T18:55:07.262 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Mvcr7F8LC72wv4DWjR7s5+ROzo0= 2022-09-12T18:55:07.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Mvcr7F8LC72wv4DWjR7s5+ROzo0=', 'Content-Length': '0'} 2022-09-12T18:55:07.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044e8a7f4f9343919-00631f808b-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:07.266 DEBUG:boto:b'jdpiak-13myobj1000falsemyobj2022-09-12T18:55:00.753Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:07.267 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:55:07.267 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:55:07.268 DEBUG:boto:path=/jdpiak-13/?prefix=myobj 2022-09-12T18:55:07.268 DEBUG:boto:auth_path=/jdpiak-13/?prefix=myobj 2022-09-12T18:55:07.269 DEBUG:boto:Method: GET 2022-09-12T18:55:07.270 DEBUG:boto:Path: /jdpiak-13/?prefix=myobj 2022-09-12T18:55:07.270 DEBUG:boto:Data: 2022-09-12T18:55:07.271 DEBUG:boto:Headers: {} 2022-09-12T18:55:07.271 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:07.272 DEBUG:boto:Port: 8000 2022-09-12T18:55:07.273 DEBUG:boto:Params: {} 2022-09-12T18:55:07.273 DEBUG:boto:Token: None 2022-09-12T18:55:07.274 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:07 GMT /jdpiak-13/ 2022-09-12T18:55:07.275 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Mvcr7F8LC72wv4DWjR7s5+ROzo0= 2022-09-12T18:55:07.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Mvcr7F8LC72wv4DWjR7s5+ROzo0=', 'Content-Length': '0'} 2022-09-12T18:55:07.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec051b59d67cba3f-00631f808b-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:07.279 DEBUG:boto:b'jdpiak-13myobj1000falsemyobj2022-09-12T18:55:00.753Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:07.279 DEBUG:boto:path=/jdpiak-13/myobj 2022-09-12T18:55:07.280 DEBUG:boto:auth_path=/jdpiak-13/myobj 2022-09-12T18:55:07.280 DEBUG:boto:Method: HEAD 2022-09-12T18:55:07.281 DEBUG:boto:Path: /jdpiak-13/myobj 2022-09-12T18:55:07.282 DEBUG:boto:Data: 2022-09-12T18:55:07.282 DEBUG:boto:Headers: {} 2022-09-12T18:55:07.283 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:07.284 DEBUG:boto:Port: 8001 2022-09-12T18:55:07.284 DEBUG:boto:Params: {} 2022-09-12T18:55:07.285 DEBUG:boto:Token: None 2022-09-12T18:55:07.285 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:07 GMT /jdpiak-13/myobj 2022-09-12T18:55:07.286 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hACrubdyOxjoCYDF4VT+Q5m4Ot0= 2022-09-12T18:55:07.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hACrubdyOxjoCYDF4VT+Q5m4Ot0=', 'Content-Length': '0'} 2022-09-12T18:55:07.289 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e32972d0dfbf9ce9-00631f808b-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:07.289 DEBUG:boto:path=/jdpiak-13/myobj 2022-09-12T18:55:07.290 DEBUG:boto:auth_path=/jdpiak-13/myobj 2022-09-12T18:55:07.291 DEBUG:boto:Method: HEAD 2022-09-12T18:55:07.292 DEBUG:boto:Path: /jdpiak-13/myobj 2022-09-12T18:55:07.292 DEBUG:boto:Data: 2022-09-12T18:55:07.293 DEBUG:boto:Headers: {} 2022-09-12T18:55:07.293 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:07.294 DEBUG:boto:Port: 8000 2022-09-12T18:55:07.294 DEBUG:boto:Params: {} 2022-09-12T18:55:07.295 DEBUG:boto:Token: None 2022-09-12T18:55:07.295 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:07 GMT /jdpiak-13/myobj 2022-09-12T18:55:07.296 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hACrubdyOxjoCYDF4VT+Q5m4Ot0= 2022-09-12T18:55:07.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hACrubdyOxjoCYDF4VT+Q5m4Ot0=', 'Content-Length': '0'} 2022-09-12T18:55:07.300 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1c79dd389799d2e-00631f808b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:07.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-12T18:55:07.302 DEBUG:boto:path=/jdpiak-13/myobj 2022-09-12T18:55:07.303 DEBUG:boto:auth_path=/jdpiak-13/myobj 2022-09-12T18:55:07.304 DEBUG:boto:Method: GET 2022-09-12T18:55:07.305 DEBUG:boto:Path: /jdpiak-13/myobj 2022-09-12T18:55:07.306 DEBUG:boto:Data: 2022-09-12T18:55:07.307 DEBUG:boto:Headers: {} 2022-09-12T18:55:07.308 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:07.309 DEBUG:boto:Port: 8001 2022-09-12T18:55:07.310 DEBUG:boto:Params: {} 2022-09-12T18:55:07.311 DEBUG:boto:Token: None 2022-09-12T18:55:07.312 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:07 GMT /jdpiak-13/myobj 2022-09-12T18:55:07.313 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1OkHPBk1t+eNTDjQ43mVEj30s/Q= 2022-09-12T18:55:07.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1OkHPBk1t+eNTDjQ43mVEj30s/Q=', 'Content-Length': '0'} 2022-09-12T18:55:07.317 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075f5dc9c68f79bfa-00631f808b-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:07.320 DEBUG:boto:path=/jdpiak-13/myobj 2022-09-12T18:55:07.321 DEBUG:boto:auth_path=/jdpiak-13/myobj 2022-09-12T18:55:07.322 DEBUG:boto:Method: GET 2022-09-12T18:55:07.323 DEBUG:boto:Path: /jdpiak-13/myobj 2022-09-12T18:55:07.324 DEBUG:boto:Data: 2022-09-12T18:55:07.325 DEBUG:boto:Headers: {} 2022-09-12T18:55:07.326 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:07.326 DEBUG:boto:Port: 8000 2022-09-12T18:55:07.327 DEBUG:boto:Params: {} 2022-09-12T18:55:07.328 DEBUG:boto:Token: None 2022-09-12T18:55:07.328 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:07 GMT /jdpiak-13/myobj 2022-09-12T18:55:07.329 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1OkHPBk1t+eNTDjQ43mVEj30s/Q= 2022-09-12T18:55:07.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1OkHPBk1t+eNTDjQ43mVEj30s/Q=', 'Content-Length': '0'} 2022-09-12T18:55:07.331 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a6603113510ef6a-00631f808b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:07.378 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:55:07.379 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:55:07.379 DEBUG:boto:path=/jdpiak-13/?prefix=myobj 2022-09-12T18:55:07.380 DEBUG:boto:auth_path=/jdpiak-13/?prefix=myobj 2022-09-12T18:55:07.381 DEBUG:boto:Method: GET 2022-09-12T18:55:07.382 DEBUG:boto:Path: /jdpiak-13/?prefix=myobj 2022-09-12T18:55:07.383 DEBUG:boto:Data: 2022-09-12T18:55:07.384 DEBUG:boto:Headers: {} 2022-09-12T18:55:07.385 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:07.386 DEBUG:boto:Port: 8001 2022-09-12T18:55:07.387 DEBUG:boto:Params: {} 2022-09-12T18:55:07.388 DEBUG:boto:Token: None 2022-09-12T18:55:07.389 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:07 GMT /jdpiak-13/ 2022-09-12T18:55:07.390 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Mvcr7F8LC72wv4DWjR7s5+ROzo0= 2022-09-12T18:55:07.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Mvcr7F8LC72wv4DWjR7s5+ROzo0=', 'Content-Length': '0'} 2022-09-12T18:55:07.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3b2b59344956a97-00631f808b-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:07.394 DEBUG:boto:b'jdpiak-13myobj1000falsemyobj2022-09-12T18:55:00.753Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:07.395 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:55:07.396 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:55:07.396 DEBUG:boto:path=/jdpiak-13/?prefix=myobj 2022-09-12T18:55:07.397 DEBUG:boto:auth_path=/jdpiak-13/?prefix=myobj 2022-09-12T18:55:07.398 DEBUG:boto:Method: GET 2022-09-12T18:55:07.399 DEBUG:boto:Path: /jdpiak-13/?prefix=myobj 2022-09-12T18:55:07.400 DEBUG:boto:Data: 2022-09-12T18:55:07.401 DEBUG:boto:Headers: {} 2022-09-12T18:55:07.402 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:07.402 DEBUG:boto:Port: 8000 2022-09-12T18:55:07.403 DEBUG:boto:Params: {} 2022-09-12T18:55:07.403 DEBUG:boto:Token: None 2022-09-12T18:55:07.404 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:07 GMT /jdpiak-13/ 2022-09-12T18:55:07.405 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Mvcr7F8LC72wv4DWjR7s5+ROzo0= 2022-09-12T18:55:07.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Mvcr7F8LC72wv4DWjR7s5+ROzo0=', 'Content-Length': '0'} 2022-09-12T18:55:07.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9f7871dcbb4e926-00631f808b-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:07.409 DEBUG:boto:b'jdpiak-13myobj1000falsemyobj2022-09-12T18:55:00.753Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:07.411 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-13 zones={test-zone3, test-zone1} 2022-09-12T18:55:07.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:55:07.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:55:07.414 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T18:55:07.475 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:55:07.476 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:55:07.522 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:55:07.519+0000 7f8a81fcf600 1 waiting for incremental sync to catch up: 2022-09-12T18:55:07.523 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , ] 2022-09-12T18:55:07.523 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:55:12.524 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:55:12.521+0000 7f8a81fcf600 1 waiting for incremental sync to catch up: 2022-09-12T18:55:12.524 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , ] 2022-09-12T18:55:12.525 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:55:17.525 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:55:17.522+0000 7f8a81fcf600 1 waiting for incremental sync to catch up: 2022-09-12T18:55:17.526 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , ] 2022-09-12T18:55:17.526 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:55:22.527 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:55:22.524+0000 7f8a81fcf600 1 waiting for incremental sync to catch up: 2022-09-12T18:55:22.527 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , ] 2022-09-12T18:55:22.528 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:55:27.528 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:55:27.525+0000 7f8a81fcf600 1 bucket sync caught up with source: 2022-09-12T18:55:27.529 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:55:27.529 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:55:27.530 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:55:27.525+0000 7f8a81fcf600 0 bucket checkpoint complete 2022-09-12T18:55:27.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:55:27.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:55:27.552 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T18:55:27.617 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:55:27.618 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:55:27.658 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:55:27.654+0000 7f648de2c600 1 bucket sync caught up with source: 2022-09-12T18:55:27.658 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.15.5, , , , , , , , , ] 2022-09-12T18:55:27.659 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000001.15.5, , , , , , , , , ] 2022-09-12T18:55:27.659 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:55:27.654+0000 7f648de2c600 0 bucket checkpoint complete 2022-09-12T18:55:27.671 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-14 zones={test-zone1, test-zone2} 2022-09-12T18:55:27.672 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:55:27.672 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:55:27.673 DEBUG:boto:Method: HEAD 2022-09-12T18:55:27.673 DEBUG:boto:Path: /jdpiak-14/ 2022-09-12T18:55:27.674 DEBUG:boto:Data: 2022-09-12T18:55:27.674 DEBUG:boto:Headers: {} 2022-09-12T18:55:27.675 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:27.675 DEBUG:boto:Port: 8000 2022-09-12T18:55:27.676 DEBUG:boto:Params: {} 2022-09-12T18:55:27.676 DEBUG:boto:Token: None 2022-09-12T18:55:27.677 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:27 GMT /jdpiak-14/ 2022-09-12T18:55:27.678 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AyQ6jZbh7wevaeUyw1zNfuzaOPk= 2022-09-12T18:55:27.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AyQ6jZbh7wevaeUyw1zNfuzaOPk=', 'Content-Length': '0'} 2022-09-12T18:55:27.681 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d48eb3b7773a440f-00631f809f-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:55:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:27.682 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:55:27.682 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:55:27.683 DEBUG:boto:Method: HEAD 2022-09-12T18:55:27.683 DEBUG:boto:Path: /jdpiak-14/ 2022-09-12T18:55:27.684 DEBUG:boto:Data: 2022-09-12T18:55:27.684 DEBUG:boto:Headers: {} 2022-09-12T18:55:27.685 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:27.685 DEBUG:boto:Port: 8000 2022-09-12T18:55:27.686 DEBUG:boto:Params: {} 2022-09-12T18:55:27.686 DEBUG:boto:Token: None 2022-09-12T18:55:27.687 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:27 GMT /jdpiak-14/ 2022-09-12T18:55:27.687 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AyQ6jZbh7wevaeUyw1zNfuzaOPk= 2022-09-12T18:55:27.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AyQ6jZbh7wevaeUyw1zNfuzaOPk=', 'Content-Length': '0'} 2022-09-12T18:55:27.691 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000953938f5eab2bdb7-00631f809f-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:55:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:27.691 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T18:55:27.692 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:55:27.692 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:55:27.693 DEBUG:boto:path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:55:27.693 DEBUG:boto:auth_path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:55:27.694 DEBUG:boto:Method: GET 2022-09-12T18:55:27.694 DEBUG:boto:Path: /jdpiak-14/?versions&max-keys=999 2022-09-12T18:55:27.695 DEBUG:boto:Data: 2022-09-12T18:55:27.695 DEBUG:boto:Headers: {} 2022-09-12T18:55:27.696 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:27.696 DEBUG:boto:Port: 8000 2022-09-12T18:55:27.697 DEBUG:boto:Params: {} 2022-09-12T18:55:27.698 DEBUG:boto:Token: None 2022-09-12T18:55:27.698 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:27 GMT /jdpiak-14/?versions 2022-09-12T18:55:27.699 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P0Pz3Khi4i7b6OysFLBPXuDTBzs= 2022-09-12T18:55:27.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P0Pz3Khi4i7b6OysFLBPXuDTBzs=', 'Content-Length': '0'} 2022-09-12T18:55:27.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000891b05c275c489e6-00631f809f-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:27.703 DEBUG:boto:b'jdpiak-14999falsemyobjnulltrue2022-09-12T18:55:00.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:27.703 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-12T18:55:27.704 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T18:55:27.704 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:55:27.705 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:55:27.705 DEBUG:boto:path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:55:27.706 DEBUG:boto:auth_path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:55:27.706 DEBUG:boto:Method: GET 2022-09-12T18:55:27.707 DEBUG:boto:Path: /jdpiak-14/?versions&max-keys=999 2022-09-12T18:55:27.707 DEBUG:boto:Data: 2022-09-12T18:55:27.708 DEBUG:boto:Headers: {} 2022-09-12T18:55:27.708 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:27.709 DEBUG:boto:Port: 8000 2022-09-12T18:55:27.709 DEBUG:boto:Params: {} 2022-09-12T18:55:27.710 DEBUG:boto:Token: None 2022-09-12T18:55:27.711 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:27 GMT /jdpiak-14/?versions 2022-09-12T18:55:27.711 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P0Pz3Khi4i7b6OysFLBPXuDTBzs= 2022-09-12T18:55:27.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P0Pz3Khi4i7b6OysFLBPXuDTBzs=', 'Content-Length': '0'} 2022-09-12T18:55:27.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007dc8deea2625c27-00631f809f-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:27.715 DEBUG:boto:b'jdpiak-14999falsemyobjnulltrue2022-09-12T18:55:00.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:27.716 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-12T18:55:27.716 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:55:27.717 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:55:27.717 DEBUG:boto:path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:55:27.718 DEBUG:boto:auth_path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:55:27.718 DEBUG:boto:Method: GET 2022-09-12T18:55:27.719 DEBUG:boto:Path: /jdpiak-14/?versions&max-keys=999 2022-09-12T18:55:27.719 DEBUG:boto:Data: 2022-09-12T18:55:27.720 DEBUG:boto:Headers: {} 2022-09-12T18:55:27.720 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:27.721 DEBUG:boto:Port: 8000 2022-09-12T18:55:27.721 DEBUG:boto:Params: {} 2022-09-12T18:55:27.722 DEBUG:boto:Token: None 2022-09-12T18:55:27.723 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:27 GMT /jdpiak-14/?versions 2022-09-12T18:55:27.723 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P0Pz3Khi4i7b6OysFLBPXuDTBzs= 2022-09-12T18:55:27.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P0Pz3Khi4i7b6OysFLBPXuDTBzs=', 'Content-Length': '0'} 2022-09-12T18:55:27.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005766e1b0e93c3069-00631f809f-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:27.727 DEBUG:boto:b'jdpiak-14999falsemyobjnulltrue2022-09-12T18:55:00.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:27.727 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:55:27.728 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:55:27.729 DEBUG:boto:path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:55:27.729 DEBUG:boto:auth_path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:55:27.730 DEBUG:boto:Method: GET 2022-09-12T18:55:27.730 DEBUG:boto:Path: /jdpiak-14/?versions&max-keys=999 2022-09-12T18:55:27.731 DEBUG:boto:Data: 2022-09-12T18:55:27.731 DEBUG:boto:Headers: {} 2022-09-12T18:55:27.732 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:27.732 DEBUG:boto:Port: 8000 2022-09-12T18:55:27.733 DEBUG:boto:Params: {} 2022-09-12T18:55:27.733 DEBUG:boto:Token: None 2022-09-12T18:55:27.734 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:27 GMT /jdpiak-14/?versions 2022-09-12T18:55:27.734 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P0Pz3Khi4i7b6OysFLBPXuDTBzs= 2022-09-12T18:55:27.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P0Pz3Khi4i7b6OysFLBPXuDTBzs=', 'Content-Length': '0'} 2022-09-12T18:55:27.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0e5f5932f25e7b8-00631f809f-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:27.738 DEBUG:boto:b'jdpiak-14999falsemyobjnulltrue2022-09-12T18:55:00.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:27.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-12T18:55:27.739 DEBUG:boto:path=/jdpiak-14/myobj 2022-09-12T18:55:27.740 DEBUG:boto:auth_path=/jdpiak-14/myobj 2022-09-12T18:55:27.741 DEBUG:boto:path=/jdpiak-14/myobj?versionId=null 2022-09-12T18:55:27.741 DEBUG:boto:auth_path=/jdpiak-14/myobj?versionId=null 2022-09-12T18:55:27.742 DEBUG:boto:Method: GET 2022-09-12T18:55:27.742 DEBUG:boto:Path: /jdpiak-14/myobj?versionId=null 2022-09-12T18:55:27.743 DEBUG:boto:Data: 2022-09-12T18:55:27.744 DEBUG:boto:Headers: {} 2022-09-12T18:55:27.744 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:27.745 DEBUG:boto:Port: 8000 2022-09-12T18:55:27.745 DEBUG:boto:Params: {} 2022-09-12T18:55:27.746 DEBUG:boto:Token: None 2022-09-12T18:55:27.747 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:27 GMT /jdpiak-14/myobj?versionId=null 2022-09-12T18:55:27.747 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3pTt9BOvC8ZEeiH/Fw1ED94CvjM= 2022-09-12T18:55:27.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3pTt9BOvC8ZEeiH/Fw1ED94CvjM=', 'Content-Length': '0'} 2022-09-12T18:55:27.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0847379cd639184-00631f809f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:27.794 DEBUG:boto:path=/jdpiak-14/myobj 2022-09-12T18:55:27.795 DEBUG:boto:auth_path=/jdpiak-14/myobj 2022-09-12T18:55:27.796 DEBUG:boto:path=/jdpiak-14/myobj?versionId=null 2022-09-12T18:55:27.797 DEBUG:boto:auth_path=/jdpiak-14/myobj?versionId=null 2022-09-12T18:55:27.798 DEBUG:boto:Method: GET 2022-09-12T18:55:27.799 DEBUG:boto:Path: /jdpiak-14/myobj?versionId=null 2022-09-12T18:55:27.800 DEBUG:boto:Data: 2022-09-12T18:55:27.801 DEBUG:boto:Headers: {} 2022-09-12T18:55:27.801 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:27.802 DEBUG:boto:Port: 8000 2022-09-12T18:55:27.803 DEBUG:boto:Params: {} 2022-09-12T18:55:27.804 DEBUG:boto:Token: None 2022-09-12T18:55:27.805 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:27 GMT /jdpiak-14/myobj?versionId=null 2022-09-12T18:55:27.806 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3pTt9BOvC8ZEeiH/Fw1ED94CvjM= 2022-09-12T18:55:27.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3pTt9BOvC8ZEeiH/Fw1ED94CvjM=', 'Content-Length': '0'} 2022-09-12T18:55:27.810 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077a57c07444f73a3-00631f809f-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:27.811 DEBUG:boto:path=/jdpiak-14/myobj 2022-09-12T18:55:27.813 DEBUG:boto:auth_path=/jdpiak-14/myobj 2022-09-12T18:55:27.814 DEBUG:boto:path=/jdpiak-14/myobj?versionId=null 2022-09-12T18:55:27.815 DEBUG:boto:auth_path=/jdpiak-14/myobj?versionId=null 2022-09-12T18:55:27.816 DEBUG:boto:Method: HEAD 2022-09-12T18:55:27.817 DEBUG:boto:Path: /jdpiak-14/myobj?versionId=null 2022-09-12T18:55:27.818 DEBUG:boto:Data: 2022-09-12T18:55:27.819 DEBUG:boto:Headers: {} 2022-09-12T18:55:27.820 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:27.821 DEBUG:boto:Port: 8000 2022-09-12T18:55:27.822 DEBUG:boto:Params: {} 2022-09-12T18:55:27.823 DEBUG:boto:Token: None 2022-09-12T18:55:27.824 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:27 GMT /jdpiak-14/myobj?versionId=null 2022-09-12T18:55:27.825 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:COnIYRnS2xwXm24IpO98T5e/Des= 2022-09-12T18:55:27.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:COnIYRnS2xwXm24IpO98T5e/Des=', 'Content-Length': '0'} 2022-09-12T18:55:27.828 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df21d8f551d1ccd4-00631f809f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:27.829 DEBUG:boto:path=/jdpiak-14/myobj 2022-09-12T18:55:27.830 DEBUG:boto:auth_path=/jdpiak-14/myobj 2022-09-12T18:55:27.830 DEBUG:boto:path=/jdpiak-14/myobj?versionId=null 2022-09-12T18:55:27.831 DEBUG:boto:auth_path=/jdpiak-14/myobj?versionId=null 2022-09-12T18:55:27.832 DEBUG:boto:Method: HEAD 2022-09-12T18:55:27.832 DEBUG:boto:Path: /jdpiak-14/myobj?versionId=null 2022-09-12T18:55:27.833 DEBUG:boto:Data: 2022-09-12T18:55:27.834 DEBUG:boto:Headers: {} 2022-09-12T18:55:27.834 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:27.835 DEBUG:boto:Port: 8000 2022-09-12T18:55:27.836 DEBUG:boto:Params: {} 2022-09-12T18:55:27.836 DEBUG:boto:Token: None 2022-09-12T18:55:27.837 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:27 GMT /jdpiak-14/myobj?versionId=null 2022-09-12T18:55:27.838 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:COnIYRnS2xwXm24IpO98T5e/Des= 2022-09-12T18:55:27.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:COnIYRnS2xwXm24IpO98T5e/Des=', 'Content-Length': '0'} 2022-09-12T18:55:27.841 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000880222af68602c48-00631f809f-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:27.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-12T18:55:27.842 DEBUG:boto:path=/jdpiak-14/myobj 2022-09-12T18:55:27.843 DEBUG:boto:auth_path=/jdpiak-14/myobj 2022-09-12T18:55:27.844 DEBUG:boto:path=/jdpiak-14/myobj?versionId=null 2022-09-12T18:55:27.845 DEBUG:boto:auth_path=/jdpiak-14/myobj?versionId=null 2022-09-12T18:55:27.845 DEBUG:boto:Method: GET 2022-09-12T18:55:27.846 DEBUG:boto:Path: /jdpiak-14/myobj?versionId=null 2022-09-12T18:55:27.847 DEBUG:boto:Data: 2022-09-12T18:55:27.848 DEBUG:boto:Headers: {} 2022-09-12T18:55:27.848 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:27.849 DEBUG:boto:Port: 8000 2022-09-12T18:55:27.850 DEBUG:boto:Params: {} 2022-09-12T18:55:27.851 DEBUG:boto:Token: None 2022-09-12T18:55:27.852 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:27 GMT /jdpiak-14/myobj?versionId=null 2022-09-12T18:55:27.852 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3pTt9BOvC8ZEeiH/Fw1ED94CvjM= 2022-09-12T18:55:27.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3pTt9BOvC8ZEeiH/Fw1ED94CvjM=', 'Content-Length': '0'} 2022-09-12T18:55:27.856 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095b5bbc645a13ffc-00631f809f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:27.857 DEBUG:boto:path=/jdpiak-14/myobj 2022-09-12T18:55:27.857 DEBUG:boto:auth_path=/jdpiak-14/myobj 2022-09-12T18:55:27.858 DEBUG:boto:path=/jdpiak-14/myobj?versionId=null 2022-09-12T18:55:27.858 DEBUG:boto:auth_path=/jdpiak-14/myobj?versionId=null 2022-09-12T18:55:27.859 DEBUG:boto:Method: GET 2022-09-12T18:55:27.860 DEBUG:boto:Path: /jdpiak-14/myobj?versionId=null 2022-09-12T18:55:27.860 DEBUG:boto:Data: 2022-09-12T18:55:27.861 DEBUG:boto:Headers: {} 2022-09-12T18:55:27.861 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:27.862 DEBUG:boto:Port: 8000 2022-09-12T18:55:27.863 DEBUG:boto:Params: {} 2022-09-12T18:55:27.863 DEBUG:boto:Token: None 2022-09-12T18:55:27.864 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:27 GMT /jdpiak-14/myobj?versionId=null 2022-09-12T18:55:27.865 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3pTt9BOvC8ZEeiH/Fw1ED94CvjM= 2022-09-12T18:55:27.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3pTt9BOvC8ZEeiH/Fw1ED94CvjM=', 'Content-Length': '0'} 2022-09-12T18:55:27.868 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd1a0fba3f986eb1-00631f809f-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:27.910 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:55:27.911 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:55:27.912 DEBUG:boto:path=/jdpiak-14/?prefix=myobj 2022-09-12T18:55:27.913 DEBUG:boto:auth_path=/jdpiak-14/?prefix=myobj 2022-09-12T18:55:27.913 DEBUG:boto:Method: GET 2022-09-12T18:55:27.914 DEBUG:boto:Path: /jdpiak-14/?prefix=myobj 2022-09-12T18:55:27.915 DEBUG:boto:Data: 2022-09-12T18:55:27.915 DEBUG:boto:Headers: {} 2022-09-12T18:55:27.916 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:27.916 DEBUG:boto:Port: 8000 2022-09-12T18:55:27.917 DEBUG:boto:Params: {} 2022-09-12T18:55:27.918 DEBUG:boto:Token: None 2022-09-12T18:55:27.918 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:27 GMT /jdpiak-14/ 2022-09-12T18:55:27.919 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nCaNHAhEdwJNv9cmqh64KughEmw= 2022-09-12T18:55:27.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nCaNHAhEdwJNv9cmqh64KughEmw=', 'Content-Length': '0'} 2022-09-12T18:55:27.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6c262d42cc5c585-00631f809f-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:27.923 DEBUG:boto:b'jdpiak-14myobj1000falsemyobj2022-09-12T18:55:00.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:27.925 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:55:27.925 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:55:27.926 DEBUG:boto:path=/jdpiak-14/?prefix=myobj 2022-09-12T18:55:27.928 DEBUG:boto:auth_path=/jdpiak-14/?prefix=myobj 2022-09-12T18:55:27.929 DEBUG:boto:Method: GET 2022-09-12T18:55:27.930 DEBUG:boto:Path: /jdpiak-14/?prefix=myobj 2022-09-12T18:55:27.931 DEBUG:boto:Data: 2022-09-12T18:55:27.932 DEBUG:boto:Headers: {} 2022-09-12T18:55:27.933 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:27.934 DEBUG:boto:Port: 8000 2022-09-12T18:55:27.935 DEBUG:boto:Params: {} 2022-09-12T18:55:27.936 DEBUG:boto:Token: None 2022-09-12T18:55:27.937 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:27 GMT /jdpiak-14/ 2022-09-12T18:55:27.938 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nCaNHAhEdwJNv9cmqh64KughEmw= 2022-09-12T18:55:27.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nCaNHAhEdwJNv9cmqh64KughEmw=', 'Content-Length': '0'} 2022-09-12T18:55:27.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a6597ecfe1c7dd2-00631f809f-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:27.943 DEBUG:boto:b'jdpiak-14myobj1000falsemyobj2022-09-12T18:55:00.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:27.944 DEBUG:boto:path=/jdpiak-14/myobj 2022-09-12T18:55:27.944 DEBUG:boto:auth_path=/jdpiak-14/myobj 2022-09-12T18:55:27.945 DEBUG:boto:Method: HEAD 2022-09-12T18:55:27.946 DEBUG:boto:Path: /jdpiak-14/myobj 2022-09-12T18:55:27.946 DEBUG:boto:Data: 2022-09-12T18:55:27.947 DEBUG:boto:Headers: {} 2022-09-12T18:55:27.947 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:27.948 DEBUG:boto:Port: 8000 2022-09-12T18:55:27.949 DEBUG:boto:Params: {} 2022-09-12T18:55:27.949 DEBUG:boto:Token: None 2022-09-12T18:55:27.950 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:27 GMT /jdpiak-14/myobj 2022-09-12T18:55:27.951 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3fdahK18SDoGo78oqiahUjGreo8= 2022-09-12T18:55:27.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3fdahK18SDoGo78oqiahUjGreo8=', 'Content-Length': '0'} 2022-09-12T18:55:27.954 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c96c8606c1fac5e-00631f809f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:27.955 DEBUG:boto:path=/jdpiak-14/myobj 2022-09-12T18:55:27.956 DEBUG:boto:auth_path=/jdpiak-14/myobj 2022-09-12T18:55:27.956 DEBUG:boto:Method: HEAD 2022-09-12T18:55:27.957 DEBUG:boto:Path: /jdpiak-14/myobj 2022-09-12T18:55:27.958 DEBUG:boto:Data: 2022-09-12T18:55:27.958 DEBUG:boto:Headers: {} 2022-09-12T18:55:27.959 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:27.959 DEBUG:boto:Port: 8000 2022-09-12T18:55:27.960 DEBUG:boto:Params: {} 2022-09-12T18:55:27.961 DEBUG:boto:Token: None 2022-09-12T18:55:27.961 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:27 GMT /jdpiak-14/myobj 2022-09-12T18:55:27.962 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3fdahK18SDoGo78oqiahUjGreo8= 2022-09-12T18:55:27.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3fdahK18SDoGo78oqiahUjGreo8=', 'Content-Length': '0'} 2022-09-12T18:55:27.965 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070ea9b8db16c1b77-00631f809f-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:27.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-12T18:55:27.967 DEBUG:boto:path=/jdpiak-14/myobj 2022-09-12T18:55:27.967 DEBUG:boto:auth_path=/jdpiak-14/myobj 2022-09-12T18:55:27.968 DEBUG:boto:Method: GET 2022-09-12T18:55:27.968 DEBUG:boto:Path: /jdpiak-14/myobj 2022-09-12T18:55:27.969 DEBUG:boto:Data: 2022-09-12T18:55:27.969 DEBUG:boto:Headers: {} 2022-09-12T18:55:27.970 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:27.970 DEBUG:boto:Port: 8000 2022-09-12T18:55:27.971 DEBUG:boto:Params: {} 2022-09-12T18:55:27.972 DEBUG:boto:Token: None 2022-09-12T18:55:27.973 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:27 GMT /jdpiak-14/myobj 2022-09-12T18:55:27.974 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EjuBLx/WVmUjqgFM20RM9YGVGF0= 2022-09-12T18:55:27.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EjuBLx/WVmUjqgFM20RM9YGVGF0=', 'Content-Length': '0'} 2022-09-12T18:55:27.976 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006bcf285cab8ab710-00631f809f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.018 DEBUG:boto:path=/jdpiak-14/myobj 2022-09-12T18:55:28.019 DEBUG:boto:auth_path=/jdpiak-14/myobj 2022-09-12T18:55:28.020 DEBUG:boto:Method: GET 2022-09-12T18:55:28.021 DEBUG:boto:Path: /jdpiak-14/myobj 2022-09-12T18:55:28.021 DEBUG:boto:Data: 2022-09-12T18:55:28.022 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.023 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:28.023 DEBUG:boto:Port: 8000 2022-09-12T18:55:28.024 DEBUG:boto:Params: {} 2022-09-12T18:55:28.025 DEBUG:boto:Token: None 2022-09-12T18:55:28.025 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-14/myobj 2022-09-12T18:55:28.026 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BixCBMx6Xnk4Er9dtX9hLFXhmhU= 2022-09-12T18:55:28.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BixCBMx6Xnk4Er9dtX9hLFXhmhU=', 'Content-Length': '0'} 2022-09-12T18:55:28.029 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004fa459ef1f88a45f-00631f80a0-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.030 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:55:28.030 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:55:28.031 DEBUG:boto:path=/jdpiak-14/?prefix=myobj 2022-09-12T18:55:28.031 DEBUG:boto:auth_path=/jdpiak-14/?prefix=myobj 2022-09-12T18:55:28.032 DEBUG:boto:Method: GET 2022-09-12T18:55:28.033 DEBUG:boto:Path: /jdpiak-14/?prefix=myobj 2022-09-12T18:55:28.033 DEBUG:boto:Data: 2022-09-12T18:55:28.034 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.034 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:28.035 DEBUG:boto:Port: 8000 2022-09-12T18:55:28.035 DEBUG:boto:Params: {} 2022-09-12T18:55:28.036 DEBUG:boto:Token: None 2022-09-12T18:55:28.037 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-14/ 2022-09-12T18:55:28.037 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DCTIJKD4wB7gSii0eTgitha1Ha4= 2022-09-12T18:55:28.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DCTIJKD4wB7gSii0eTgitha1Ha4=', 'Content-Length': '0'} 2022-09-12T18:55:28.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005131383627ff3b7c-00631f80a0-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.041 DEBUG:boto:b'jdpiak-14myobj1000falsemyobj2022-09-12T18:55:00.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:28.042 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:55:28.043 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:55:28.044 DEBUG:boto:path=/jdpiak-14/?prefix=myobj 2022-09-12T18:55:28.045 DEBUG:boto:auth_path=/jdpiak-14/?prefix=myobj 2022-09-12T18:55:28.046 DEBUG:boto:Method: GET 2022-09-12T18:55:28.046 DEBUG:boto:Path: /jdpiak-14/?prefix=myobj 2022-09-12T18:55:28.047 DEBUG:boto:Data: 2022-09-12T18:55:28.048 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.049 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:28.050 DEBUG:boto:Port: 8000 2022-09-12T18:55:28.051 DEBUG:boto:Params: {} 2022-09-12T18:55:28.051 DEBUG:boto:Token: None 2022-09-12T18:55:28.052 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-14/ 2022-09-12T18:55:28.053 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DCTIJKD4wB7gSii0eTgitha1Ha4= 2022-09-12T18:55:28.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DCTIJKD4wB7gSii0eTgitha1Ha4=', 'Content-Length': '0'} 2022-09-12T18:55:28.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ced01f724859df5-00631f80a0-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.057 DEBUG:boto:b'jdpiak-14myobj1000falsemyobj2022-09-12T18:55:00.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:28.058 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-14 zones={test-zone1, test-zone2} 2022-09-12T18:55:28.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:55:28.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:55:28.060 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T18:55:28.119 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:55:28.120 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:55:28.159 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:55:28.156+0000 7fafe1e22600 1 bucket sync caught up with source: 2022-09-12T18:55:28.160 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.15.5, , , , , , , , , ] 2022-09-12T18:55:28.160 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000001.15.5, , , , , , , , , ] 2022-09-12T18:55:28.160 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:55:28.156+0000 7fafe1e22600 0 bucket checkpoint complete 2022-09-12T18:55:28.173 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-14 zones={test-zone3, test-zone2} 2022-09-12T18:55:28.174 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:55:28.175 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:55:28.176 DEBUG:boto:Method: HEAD 2022-09-12T18:55:28.177 DEBUG:boto:Path: /jdpiak-14/ 2022-09-12T18:55:28.178 DEBUG:boto:Data: 2022-09-12T18:55:28.179 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.180 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:28.182 DEBUG:boto:Port: 8001 2022-09-12T18:55:28.182 DEBUG:boto:Params: {} 2022-09-12T18:55:28.183 DEBUG:boto:Token: None 2022-09-12T18:55:28.184 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-14/ 2022-09-12T18:55:28.184 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DQCwZ7RP25IdR78vKP+eRAUS+kU= 2022-09-12T18:55:28.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DQCwZ7RP25IdR78vKP+eRAUS+kU=', 'Content-Length': '0'} 2022-09-12T18:55:28.187 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d75982910db35e4b-00631f80a0-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.188 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:55:28.188 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:55:28.189 DEBUG:boto:Method: HEAD 2022-09-12T18:55:28.190 DEBUG:boto:Path: /jdpiak-14/ 2022-09-12T18:55:28.190 DEBUG:boto:Data: 2022-09-12T18:55:28.191 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.191 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:28.192 DEBUG:boto:Port: 8000 2022-09-12T18:55:28.192 DEBUG:boto:Params: {} 2022-09-12T18:55:28.193 DEBUG:boto:Token: None 2022-09-12T18:55:28.193 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-14/ 2022-09-12T18:55:28.194 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DQCwZ7RP25IdR78vKP+eRAUS+kU= 2022-09-12T18:55:28.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DQCwZ7RP25IdR78vKP+eRAUS+kU=', 'Content-Length': '0'} 2022-09-12T18:55:28.197 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad18b08aab7ca646-00631f80a0-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.198 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T18:55:28.198 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:55:28.199 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:55:28.199 DEBUG:boto:path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:55:28.200 DEBUG:boto:auth_path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:55:28.200 DEBUG:boto:Method: GET 2022-09-12T18:55:28.201 DEBUG:boto:Path: /jdpiak-14/?versions&max-keys=999 2022-09-12T18:55:28.201 DEBUG:boto:Data: 2022-09-12T18:55:28.202 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.203 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:28.203 DEBUG:boto:Port: 8001 2022-09-12T18:55:28.204 DEBUG:boto:Params: {} 2022-09-12T18:55:28.204 DEBUG:boto:Token: None 2022-09-12T18:55:28.205 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-14/?versions 2022-09-12T18:55:28.205 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:r5eIplVVveUQkUjOEZ/NFHeurBo= 2022-09-12T18:55:28.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:r5eIplVVveUQkUjOEZ/NFHeurBo=', 'Content-Length': '0'} 2022-09-12T18:55:28.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c842f51726c8fb8d-00631f80a0-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.209 DEBUG:boto:b'jdpiak-14999falsemyobjnulltrue2022-09-12T18:55:00.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:28.210 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-12T18:55:28.211 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T18:55:28.211 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:55:28.212 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:55:28.212 DEBUG:boto:path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:55:28.213 DEBUG:boto:auth_path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:55:28.213 DEBUG:boto:Method: GET 2022-09-12T18:55:28.214 DEBUG:boto:Path: /jdpiak-14/?versions&max-keys=999 2022-09-12T18:55:28.214 DEBUG:boto:Data: 2022-09-12T18:55:28.215 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.215 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:28.216 DEBUG:boto:Port: 8000 2022-09-12T18:55:28.216 DEBUG:boto:Params: {} 2022-09-12T18:55:28.217 DEBUG:boto:Token: None 2022-09-12T18:55:28.218 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-14/?versions 2022-09-12T18:55:28.218 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:r5eIplVVveUQkUjOEZ/NFHeurBo= 2022-09-12T18:55:28.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:r5eIplVVveUQkUjOEZ/NFHeurBo=', 'Content-Length': '0'} 2022-09-12T18:55:28.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009716d2245a23d292-00631f80a0-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.222 DEBUG:boto:b'jdpiak-14999falsemyobjnulltrue2022-09-12T18:55:00.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:28.223 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-12T18:55:28.224 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:55:28.224 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:55:28.225 DEBUG:boto:path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:55:28.226 DEBUG:boto:auth_path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:55:28.226 DEBUG:boto:Method: GET 2022-09-12T18:55:28.227 DEBUG:boto:Path: /jdpiak-14/?versions&max-keys=999 2022-09-12T18:55:28.227 DEBUG:boto:Data: 2022-09-12T18:55:28.228 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.228 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:28.229 DEBUG:boto:Port: 8001 2022-09-12T18:55:28.229 DEBUG:boto:Params: {} 2022-09-12T18:55:28.230 DEBUG:boto:Token: None 2022-09-12T18:55:28.230 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-14/?versions 2022-09-12T18:55:28.231 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:r5eIplVVveUQkUjOEZ/NFHeurBo= 2022-09-12T18:55:28.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:r5eIplVVveUQkUjOEZ/NFHeurBo=', 'Content-Length': '0'} 2022-09-12T18:55:28.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bef230f221110ea6-00631f80a0-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.235 DEBUG:boto:b'jdpiak-14999falsemyobjnulltrue2022-09-12T18:55:00.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:28.236 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:55:28.236 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:55:28.237 DEBUG:boto:path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:55:28.237 DEBUG:boto:auth_path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:55:28.238 DEBUG:boto:Method: GET 2022-09-12T18:55:28.238 DEBUG:boto:Path: /jdpiak-14/?versions&max-keys=999 2022-09-12T18:55:28.239 DEBUG:boto:Data: 2022-09-12T18:55:28.239 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.240 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:28.241 DEBUG:boto:Port: 8000 2022-09-12T18:55:28.241 DEBUG:boto:Params: {} 2022-09-12T18:55:28.242 DEBUG:boto:Token: None 2022-09-12T18:55:28.242 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-14/?versions 2022-09-12T18:55:28.243 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:r5eIplVVveUQkUjOEZ/NFHeurBo= 2022-09-12T18:55:28.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:r5eIplVVveUQkUjOEZ/NFHeurBo=', 'Content-Length': '0'} 2022-09-12T18:55:28.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f0f4dc5cf3a3556-00631f80a0-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.290 DEBUG:boto:b'jdpiak-14999falsemyobjnulltrue2022-09-12T18:55:00.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:28.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-12T18:55:28.292 DEBUG:boto:path=/jdpiak-14/myobj 2022-09-12T18:55:28.292 DEBUG:boto:auth_path=/jdpiak-14/myobj 2022-09-12T18:55:28.293 DEBUG:boto:path=/jdpiak-14/myobj?versionId=null 2022-09-12T18:55:28.293 DEBUG:boto:auth_path=/jdpiak-14/myobj?versionId=null 2022-09-12T18:55:28.294 DEBUG:boto:Method: GET 2022-09-12T18:55:28.295 DEBUG:boto:Path: /jdpiak-14/myobj?versionId=null 2022-09-12T18:55:28.295 DEBUG:boto:Data: 2022-09-12T18:55:28.296 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.296 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:28.297 DEBUG:boto:Port: 8001 2022-09-12T18:55:28.297 DEBUG:boto:Params: {} 2022-09-12T18:55:28.298 DEBUG:boto:Token: None 2022-09-12T18:55:28.299 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-14/myobj?versionId=null 2022-09-12T18:55:28.299 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ITSTzE2nbELVxW2kdPcObYPU3Ws= 2022-09-12T18:55:28.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ITSTzE2nbELVxW2kdPcObYPU3Ws=', 'Content-Length': '0'} 2022-09-12T18:55:28.302 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3859f3ca51fb880-00631f80a0-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.303 DEBUG:boto:path=/jdpiak-14/myobj 2022-09-12T18:55:28.304 DEBUG:boto:auth_path=/jdpiak-14/myobj 2022-09-12T18:55:28.304 DEBUG:boto:path=/jdpiak-14/myobj?versionId=null 2022-09-12T18:55:28.305 DEBUG:boto:auth_path=/jdpiak-14/myobj?versionId=null 2022-09-12T18:55:28.305 DEBUG:boto:Method: GET 2022-09-12T18:55:28.306 DEBUG:boto:Path: /jdpiak-14/myobj?versionId=null 2022-09-12T18:55:28.306 DEBUG:boto:Data: 2022-09-12T18:55:28.307 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.308 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:28.308 DEBUG:boto:Port: 8000 2022-09-12T18:55:28.309 DEBUG:boto:Params: {} 2022-09-12T18:55:28.309 DEBUG:boto:Token: None 2022-09-12T18:55:28.310 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-14/myobj?versionId=null 2022-09-12T18:55:28.311 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ITSTzE2nbELVxW2kdPcObYPU3Ws= 2022-09-12T18:55:28.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ITSTzE2nbELVxW2kdPcObYPU3Ws=', 'Content-Length': '0'} 2022-09-12T18:55:28.314 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc732a03e7ce063d-00631f80a0-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.314 DEBUG:boto:path=/jdpiak-14/myobj 2022-09-12T18:55:28.315 DEBUG:boto:auth_path=/jdpiak-14/myobj 2022-09-12T18:55:28.316 DEBUG:boto:path=/jdpiak-14/myobj?versionId=null 2022-09-12T18:55:28.316 DEBUG:boto:auth_path=/jdpiak-14/myobj?versionId=null 2022-09-12T18:55:28.317 DEBUG:boto:Method: HEAD 2022-09-12T18:55:28.317 DEBUG:boto:Path: /jdpiak-14/myobj?versionId=null 2022-09-12T18:55:28.318 DEBUG:boto:Data: 2022-09-12T18:55:28.318 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.319 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:28.319 DEBUG:boto:Port: 8001 2022-09-12T18:55:28.320 DEBUG:boto:Params: {} 2022-09-12T18:55:28.320 DEBUG:boto:Token: None 2022-09-12T18:55:28.321 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-14/myobj?versionId=null 2022-09-12T18:55:28.322 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Umr3lRBicQgIMg9HIv8J46OJak= 2022-09-12T18:55:28.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Umr3lRBicQgIMg9HIv8J46OJak=', 'Content-Length': '0'} 2022-09-12T18:55:28.324 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000bffd50ae1f88423-00631f80a0-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.325 DEBUG:boto:path=/jdpiak-14/myobj 2022-09-12T18:55:28.326 DEBUG:boto:auth_path=/jdpiak-14/myobj 2022-09-12T18:55:28.327 DEBUG:boto:path=/jdpiak-14/myobj?versionId=null 2022-09-12T18:55:28.327 DEBUG:boto:auth_path=/jdpiak-14/myobj?versionId=null 2022-09-12T18:55:28.328 DEBUG:boto:Method: HEAD 2022-09-12T18:55:28.328 DEBUG:boto:Path: /jdpiak-14/myobj?versionId=null 2022-09-12T18:55:28.329 DEBUG:boto:Data: 2022-09-12T18:55:28.329 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.329 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:28.330 DEBUG:boto:Port: 8000 2022-09-12T18:55:28.330 DEBUG:boto:Params: {} 2022-09-12T18:55:28.331 DEBUG:boto:Token: None 2022-09-12T18:55:28.332 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-14/myobj?versionId=null 2022-09-12T18:55:28.332 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Umr3lRBicQgIMg9HIv8J46OJak= 2022-09-12T18:55:28.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Umr3lRBicQgIMg9HIv8J46OJak=', 'Content-Length': '0'} 2022-09-12T18:55:28.335 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f5defea32ec71419-00631f80a0-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-12T18:55:28.337 DEBUG:boto:path=/jdpiak-14/myobj 2022-09-12T18:55:28.338 DEBUG:boto:auth_path=/jdpiak-14/myobj 2022-09-12T18:55:28.339 DEBUG:boto:path=/jdpiak-14/myobj?versionId=null 2022-09-12T18:55:28.340 DEBUG:boto:auth_path=/jdpiak-14/myobj?versionId=null 2022-09-12T18:55:28.341 DEBUG:boto:Method: GET 2022-09-12T18:55:28.342 DEBUG:boto:Path: /jdpiak-14/myobj?versionId=null 2022-09-12T18:55:28.343 DEBUG:boto:Data: 2022-09-12T18:55:28.344 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.345 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:28.346 DEBUG:boto:Port: 8001 2022-09-12T18:55:28.347 DEBUG:boto:Params: {} 2022-09-12T18:55:28.348 DEBUG:boto:Token: None 2022-09-12T18:55:28.349 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-14/myobj?versionId=null 2022-09-12T18:55:28.350 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ITSTzE2nbELVxW2kdPcObYPU3Ws= 2022-09-12T18:55:28.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ITSTzE2nbELVxW2kdPcObYPU3Ws=', 'Content-Length': '0'} 2022-09-12T18:55:28.354 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018af9bdc7f853e35-00631f80a0-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.398 DEBUG:boto:path=/jdpiak-14/myobj 2022-09-12T18:55:28.399 DEBUG:boto:auth_path=/jdpiak-14/myobj 2022-09-12T18:55:28.399 DEBUG:boto:path=/jdpiak-14/myobj?versionId=null 2022-09-12T18:55:28.400 DEBUG:boto:auth_path=/jdpiak-14/myobj?versionId=null 2022-09-12T18:55:28.401 DEBUG:boto:Method: GET 2022-09-12T18:55:28.401 DEBUG:boto:Path: /jdpiak-14/myobj?versionId=null 2022-09-12T18:55:28.402 DEBUG:boto:Data: 2022-09-12T18:55:28.402 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.403 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:28.403 DEBUG:boto:Port: 8000 2022-09-12T18:55:28.404 DEBUG:boto:Params: {} 2022-09-12T18:55:28.404 DEBUG:boto:Token: None 2022-09-12T18:55:28.405 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-14/myobj?versionId=null 2022-09-12T18:55:28.406 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ITSTzE2nbELVxW2kdPcObYPU3Ws= 2022-09-12T18:55:28.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ITSTzE2nbELVxW2kdPcObYPU3Ws=', 'Content-Length': '0'} 2022-09-12T18:55:28.409 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a79b7e0404713244-00631f80a0-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.410 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:55:28.410 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:55:28.411 DEBUG:boto:path=/jdpiak-14/?prefix=myobj 2022-09-12T18:55:28.411 DEBUG:boto:auth_path=/jdpiak-14/?prefix=myobj 2022-09-12T18:55:28.412 DEBUG:boto:Method: GET 2022-09-12T18:55:28.413 DEBUG:boto:Path: /jdpiak-14/?prefix=myobj 2022-09-12T18:55:28.413 DEBUG:boto:Data: 2022-09-12T18:55:28.414 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.414 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:28.415 DEBUG:boto:Port: 8001 2022-09-12T18:55:28.415 DEBUG:boto:Params: {} 2022-09-12T18:55:28.416 DEBUG:boto:Token: None 2022-09-12T18:55:28.416 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-14/ 2022-09-12T18:55:28.417 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DCTIJKD4wB7gSii0eTgitha1Ha4= 2022-09-12T18:55:28.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DCTIJKD4wB7gSii0eTgitha1Ha4=', 'Content-Length': '0'} 2022-09-12T18:55:28.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ec5bf6c762038da-00631f80a0-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.421 DEBUG:boto:b'jdpiak-14myobj1000falsemyobj2022-09-12T18:55:00.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:28.422 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:55:28.422 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:55:28.423 DEBUG:boto:path=/jdpiak-14/?prefix=myobj 2022-09-12T18:55:28.423 DEBUG:boto:auth_path=/jdpiak-14/?prefix=myobj 2022-09-12T18:55:28.424 DEBUG:boto:Method: GET 2022-09-12T18:55:28.425 DEBUG:boto:Path: /jdpiak-14/?prefix=myobj 2022-09-12T18:55:28.425 DEBUG:boto:Data: 2022-09-12T18:55:28.426 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.426 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:28.427 DEBUG:boto:Port: 8000 2022-09-12T18:55:28.427 DEBUG:boto:Params: {} 2022-09-12T18:55:28.428 DEBUG:boto:Token: None 2022-09-12T18:55:28.429 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-14/ 2022-09-12T18:55:28.429 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DCTIJKD4wB7gSii0eTgitha1Ha4= 2022-09-12T18:55:28.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DCTIJKD4wB7gSii0eTgitha1Ha4=', 'Content-Length': '0'} 2022-09-12T18:55:28.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf5a1a9d8badaf34-00631f80a0-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.433 DEBUG:boto:b'jdpiak-14myobj1000falsemyobj2022-09-12T18:55:00.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:28.434 DEBUG:boto:path=/jdpiak-14/myobj 2022-09-12T18:55:28.435 DEBUG:boto:auth_path=/jdpiak-14/myobj 2022-09-12T18:55:28.435 DEBUG:boto:Method: HEAD 2022-09-12T18:55:28.436 DEBUG:boto:Path: /jdpiak-14/myobj 2022-09-12T18:55:28.437 DEBUG:boto:Data: 2022-09-12T18:55:28.437 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.438 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:28.438 DEBUG:boto:Port: 8001 2022-09-12T18:55:28.439 DEBUG:boto:Params: {} 2022-09-12T18:55:28.439 DEBUG:boto:Token: None 2022-09-12T18:55:28.440 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-14/myobj 2022-09-12T18:55:28.440 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gxTXeL8XJRKo2Mg0+IBMDs+/tFw= 2022-09-12T18:55:28.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gxTXeL8XJRKo2Mg0+IBMDs+/tFw=', 'Content-Length': '0'} 2022-09-12T18:55:28.443 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004805b1d13ba4a9e2-00631f80a0-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.444 DEBUG:boto:path=/jdpiak-14/myobj 2022-09-12T18:55:28.445 DEBUG:boto:auth_path=/jdpiak-14/myobj 2022-09-12T18:55:28.445 DEBUG:boto:Method: HEAD 2022-09-12T18:55:28.446 DEBUG:boto:Path: /jdpiak-14/myobj 2022-09-12T18:55:28.446 DEBUG:boto:Data: 2022-09-12T18:55:28.447 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.447 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:28.448 DEBUG:boto:Port: 8000 2022-09-12T18:55:28.448 DEBUG:boto:Params: {} 2022-09-12T18:55:28.449 DEBUG:boto:Token: None 2022-09-12T18:55:28.450 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-14/myobj 2022-09-12T18:55:28.450 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gxTXeL8XJRKo2Mg0+IBMDs+/tFw= 2022-09-12T18:55:28.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gxTXeL8XJRKo2Mg0+IBMDs+/tFw=', 'Content-Length': '0'} 2022-09-12T18:55:28.453 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f06a836a000d0361-00631f80a0-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-12T18:55:28.454 DEBUG:boto:path=/jdpiak-14/myobj 2022-09-12T18:55:28.455 DEBUG:boto:auth_path=/jdpiak-14/myobj 2022-09-12T18:55:28.455 DEBUG:boto:Method: GET 2022-09-12T18:55:28.456 DEBUG:boto:Path: /jdpiak-14/myobj 2022-09-12T18:55:28.456 DEBUG:boto:Data: 2022-09-12T18:55:28.457 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.458 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:28.458 DEBUG:boto:Port: 8001 2022-09-12T18:55:28.459 DEBUG:boto:Params: {} 2022-09-12T18:55:28.459 DEBUG:boto:Token: None 2022-09-12T18:55:28.460 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-14/myobj 2022-09-12T18:55:28.461 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BixCBMx6Xnk4Er9dtX9hLFXhmhU= 2022-09-12T18:55:28.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BixCBMx6Xnk4Er9dtX9hLFXhmhU=', 'Content-Length': '0'} 2022-09-12T18:55:28.463 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000579426d520b398ff-00631f80a0-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.464 DEBUG:boto:path=/jdpiak-14/myobj 2022-09-12T18:55:28.464 DEBUG:boto:auth_path=/jdpiak-14/myobj 2022-09-12T18:55:28.465 DEBUG:boto:Method: GET 2022-09-12T18:55:28.466 DEBUG:boto:Path: /jdpiak-14/myobj 2022-09-12T18:55:28.466 DEBUG:boto:Data: 2022-09-12T18:55:28.467 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.467 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:28.468 DEBUG:boto:Port: 8000 2022-09-12T18:55:28.468 DEBUG:boto:Params: {} 2022-09-12T18:55:28.469 DEBUG:boto:Token: None 2022-09-12T18:55:28.470 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-14/myobj 2022-09-12T18:55:28.470 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BixCBMx6Xnk4Er9dtX9hLFXhmhU= 2022-09-12T18:55:28.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BixCBMx6Xnk4Er9dtX9hLFXhmhU=', 'Content-Length': '0'} 2022-09-12T18:55:28.473 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d42682f5083c213-00631f80a0-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.474 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:55:28.474 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:55:28.475 DEBUG:boto:path=/jdpiak-14/?prefix=myobj 2022-09-12T18:55:28.475 DEBUG:boto:auth_path=/jdpiak-14/?prefix=myobj 2022-09-12T18:55:28.476 DEBUG:boto:Method: GET 2022-09-12T18:55:28.477 DEBUG:boto:Path: /jdpiak-14/?prefix=myobj 2022-09-12T18:55:28.477 DEBUG:boto:Data: 2022-09-12T18:55:28.478 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.478 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:28.479 DEBUG:boto:Port: 8001 2022-09-12T18:55:28.479 DEBUG:boto:Params: {} 2022-09-12T18:55:28.480 DEBUG:boto:Token: None 2022-09-12T18:55:28.480 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-14/ 2022-09-12T18:55:28.481 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DCTIJKD4wB7gSii0eTgitha1Ha4= 2022-09-12T18:55:28.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DCTIJKD4wB7gSii0eTgitha1Ha4=', 'Content-Length': '0'} 2022-09-12T18:55:28.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd05e86329a3ecef-00631f80a0-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.484 DEBUG:boto:b'jdpiak-14myobj1000falsemyobj2022-09-12T18:55:00.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:28.485 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:55:28.486 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:55:28.486 DEBUG:boto:path=/jdpiak-14/?prefix=myobj 2022-09-12T18:55:28.487 DEBUG:boto:auth_path=/jdpiak-14/?prefix=myobj 2022-09-12T18:55:28.487 DEBUG:boto:Method: GET 2022-09-12T18:55:28.488 DEBUG:boto:Path: /jdpiak-14/?prefix=myobj 2022-09-12T18:55:28.488 DEBUG:boto:Data: 2022-09-12T18:55:28.489 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.489 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:28.490 DEBUG:boto:Port: 8000 2022-09-12T18:55:28.491 DEBUG:boto:Params: {} 2022-09-12T18:55:28.491 DEBUG:boto:Token: None 2022-09-12T18:55:28.492 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-14/ 2022-09-12T18:55:28.492 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DCTIJKD4wB7gSii0eTgitha1Ha4= 2022-09-12T18:55:28.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DCTIJKD4wB7gSii0eTgitha1Ha4=', 'Content-Length': '0'} 2022-09-12T18:55:28.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bba3aaf7fdcf2158-00631f80a0-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.496 DEBUG:boto:b'jdpiak-14myobj1000falsemyobj2022-09-12T18:55:00.776Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:28.497 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-14 zones={test-zone3, test-zone2} 2022-09-12T18:55:28.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:55:28.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:55:28.499 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T18:55:28.560 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:55:28.561 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:55:28.605 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:55:28.601+0000 7f5b1da14600 1 bucket sync caught up with source: 2022-09-12T18:55:28.605 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.41.5, , , , , , , , , ] 2022-09-12T18:55:28.606 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000001.41.5, , , , , , , , , ] 2022-09-12T18:55:28.606 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:55:28.601+0000 7f5b1da14600 0 bucket checkpoint complete 2022-09-12T18:55:28.624 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-15 zones={test-zone1, test-zone3} 2022-09-12T18:55:28.625 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:55:28.625 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:55:28.626 DEBUG:boto:Method: HEAD 2022-09-12T18:55:28.626 DEBUG:boto:Path: /jdpiak-15/ 2022-09-12T18:55:28.627 DEBUG:boto:Data: 2022-09-12T18:55:28.627 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.628 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:28.628 DEBUG:boto:Port: 8000 2022-09-12T18:55:28.629 DEBUG:boto:Params: {} 2022-09-12T18:55:28.630 DEBUG:boto:Token: None 2022-09-12T18:55:28.630 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-15/ 2022-09-12T18:55:28.631 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CDYWBcVeu1rG8m5BAHuIZV++f6c= 2022-09-12T18:55:28.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CDYWBcVeu1rG8m5BAHuIZV++f6c=', 'Content-Length': '0'} 2022-09-12T18:55:28.634 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000892d39ad4b988b41-00631f80a0-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.635 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:55:28.635 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:55:28.636 DEBUG:boto:Method: HEAD 2022-09-12T18:55:28.636 DEBUG:boto:Path: /jdpiak-15/ 2022-09-12T18:55:28.637 DEBUG:boto:Data: 2022-09-12T18:55:28.637 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.638 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:28.638 DEBUG:boto:Port: 8001 2022-09-12T18:55:28.639 DEBUG:boto:Params: {} 2022-09-12T18:55:28.639 DEBUG:boto:Token: None 2022-09-12T18:55:28.640 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-15/ 2022-09-12T18:55:28.641 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CDYWBcVeu1rG8m5BAHuIZV++f6c= 2022-09-12T18:55:28.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CDYWBcVeu1rG8m5BAHuIZV++f6c=', 'Content-Length': '0'} 2022-09-12T18:55:28.644 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a02ed94c46bf6576-00631f80a0-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.644 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T18:55:28.645 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:55:28.645 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:55:28.646 DEBUG:boto:path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:55:28.646 DEBUG:boto:auth_path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:55:28.647 DEBUG:boto:Method: GET 2022-09-12T18:55:28.647 DEBUG:boto:Path: /jdpiak-15/?versions&max-keys=999 2022-09-12T18:55:28.648 DEBUG:boto:Data: 2022-09-12T18:55:28.648 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.649 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:28.649 DEBUG:boto:Port: 8000 2022-09-12T18:55:28.650 DEBUG:boto:Params: {} 2022-09-12T18:55:28.650 DEBUG:boto:Token: None 2022-09-12T18:55:28.651 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-15/?versions 2022-09-12T18:55:28.651 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BbOob3ca6O1xgL+iZERU1T+lP5Y= 2022-09-12T18:55:28.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BbOob3ca6O1xgL+iZERU1T+lP5Y=', 'Content-Length': '0'} 2022-09-12T18:55:28.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005af057d1c66b3c94-00631f80a0-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.655 DEBUG:boto:b'jdpiak-15999falsemyobjnulltrue2022-09-12T18:55:00.802Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:28.656 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-12T18:55:28.656 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T18:55:28.657 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:55:28.657 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:55:28.658 DEBUG:boto:path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:55:28.658 DEBUG:boto:auth_path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:55:28.659 DEBUG:boto:Method: GET 2022-09-12T18:55:28.659 DEBUG:boto:Path: /jdpiak-15/?versions&max-keys=999 2022-09-12T18:55:28.660 DEBUG:boto:Data: 2022-09-12T18:55:28.660 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.661 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:28.661 DEBUG:boto:Port: 8001 2022-09-12T18:55:28.662 DEBUG:boto:Params: {} 2022-09-12T18:55:28.662 DEBUG:boto:Token: None 2022-09-12T18:55:28.663 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-15/?versions 2022-09-12T18:55:28.664 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BbOob3ca6O1xgL+iZERU1T+lP5Y= 2022-09-12T18:55:28.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BbOob3ca6O1xgL+iZERU1T+lP5Y=', 'Content-Length': '0'} 2022-09-12T18:55:28.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b4dc380ef86466a-00631f80a0-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.678 DEBUG:boto:b'jdpiak-15999falsemyobjnulltrue2022-09-12T18:55:00.802Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:28.680 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-12T18:55:28.681 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:55:28.681 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:55:28.682 DEBUG:boto:path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:55:28.683 DEBUG:boto:auth_path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:55:28.684 DEBUG:boto:Method: GET 2022-09-12T18:55:28.685 DEBUG:boto:Path: /jdpiak-15/?versions&max-keys=999 2022-09-12T18:55:28.686 DEBUG:boto:Data: 2022-09-12T18:55:28.686 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.687 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:28.687 DEBUG:boto:Port: 8000 2022-09-12T18:55:28.688 DEBUG:boto:Params: {} 2022-09-12T18:55:28.689 DEBUG:boto:Token: None 2022-09-12T18:55:28.690 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-15/?versions 2022-09-12T18:55:28.690 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BbOob3ca6O1xgL+iZERU1T+lP5Y= 2022-09-12T18:55:28.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BbOob3ca6O1xgL+iZERU1T+lP5Y=', 'Content-Length': '0'} 2022-09-12T18:55:28.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d9063c8cecd02aa-00631f80a0-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.734 DEBUG:boto:b'jdpiak-15999falsemyobjnulltrue2022-09-12T18:55:00.802Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:28.735 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:55:28.736 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:55:28.736 DEBUG:boto:path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:55:28.737 DEBUG:boto:auth_path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:55:28.737 DEBUG:boto:Method: GET 2022-09-12T18:55:28.738 DEBUG:boto:Path: /jdpiak-15/?versions&max-keys=999 2022-09-12T18:55:28.739 DEBUG:boto:Data: 2022-09-12T18:55:28.739 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.740 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:28.740 DEBUG:boto:Port: 8001 2022-09-12T18:55:28.741 DEBUG:boto:Params: {} 2022-09-12T18:55:28.741 DEBUG:boto:Token: None 2022-09-12T18:55:28.742 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-15/?versions 2022-09-12T18:55:28.743 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BbOob3ca6O1xgL+iZERU1T+lP5Y= 2022-09-12T18:55:28.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BbOob3ca6O1xgL+iZERU1T+lP5Y=', 'Content-Length': '0'} 2022-09-12T18:55:28.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b7d15717423b257-00631f80a0-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.746 DEBUG:boto:b'jdpiak-15999falsemyobjnulltrue2022-09-12T18:55:00.802Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:28.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-12T18:55:28.748 DEBUG:boto:path=/jdpiak-15/myobj 2022-09-12T18:55:28.749 DEBUG:boto:auth_path=/jdpiak-15/myobj 2022-09-12T18:55:28.749 DEBUG:boto:path=/jdpiak-15/myobj?versionId=null 2022-09-12T18:55:28.750 DEBUG:boto:auth_path=/jdpiak-15/myobj?versionId=null 2022-09-12T18:55:28.751 DEBUG:boto:Method: GET 2022-09-12T18:55:28.751 DEBUG:boto:Path: /jdpiak-15/myobj?versionId=null 2022-09-12T18:55:28.752 DEBUG:boto:Data: 2022-09-12T18:55:28.752 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.753 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:28.753 DEBUG:boto:Port: 8000 2022-09-12T18:55:28.754 DEBUG:boto:Params: {} 2022-09-12T18:55:28.754 DEBUG:boto:Token: None 2022-09-12T18:55:28.755 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-15/myobj?versionId=null 2022-09-12T18:55:28.756 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EY1+VLGRY4WjhPcRDkCun0mYjPg= 2022-09-12T18:55:28.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EY1+VLGRY4WjhPcRDkCun0mYjPg=', 'Content-Length': '0'} 2022-09-12T18:55:28.758 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f10def27cf6dab3-00631f80a0-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.759 DEBUG:boto:path=/jdpiak-15/myobj 2022-09-12T18:55:28.760 DEBUG:boto:auth_path=/jdpiak-15/myobj 2022-09-12T18:55:28.760 DEBUG:boto:path=/jdpiak-15/myobj?versionId=null 2022-09-12T18:55:28.761 DEBUG:boto:auth_path=/jdpiak-15/myobj?versionId=null 2022-09-12T18:55:28.761 DEBUG:boto:Method: GET 2022-09-12T18:55:28.762 DEBUG:boto:Path: /jdpiak-15/myobj?versionId=null 2022-09-12T18:55:28.763 DEBUG:boto:Data: 2022-09-12T18:55:28.764 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.765 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:28.765 DEBUG:boto:Port: 8001 2022-09-12T18:55:28.766 DEBUG:boto:Params: {} 2022-09-12T18:55:28.766 DEBUG:boto:Token: None 2022-09-12T18:55:28.767 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-15/myobj?versionId=null 2022-09-12T18:55:28.768 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EY1+VLGRY4WjhPcRDkCun0mYjPg= 2022-09-12T18:55:28.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EY1+VLGRY4WjhPcRDkCun0mYjPg=', 'Content-Length': '0'} 2022-09-12T18:55:28.770 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000922579b8ec213bed-00631f80a0-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.771 DEBUG:boto:path=/jdpiak-15/myobj 2022-09-12T18:55:28.772 DEBUG:boto:auth_path=/jdpiak-15/myobj 2022-09-12T18:55:28.772 DEBUG:boto:path=/jdpiak-15/myobj?versionId=null 2022-09-12T18:55:28.773 DEBUG:boto:auth_path=/jdpiak-15/myobj?versionId=null 2022-09-12T18:55:28.773 DEBUG:boto:Method: HEAD 2022-09-12T18:55:28.774 DEBUG:boto:Path: /jdpiak-15/myobj?versionId=null 2022-09-12T18:55:28.774 DEBUG:boto:Data: 2022-09-12T18:55:28.775 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.775 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:28.776 DEBUG:boto:Port: 8000 2022-09-12T18:55:28.776 DEBUG:boto:Params: {} 2022-09-12T18:55:28.777 DEBUG:boto:Token: None 2022-09-12T18:55:28.777 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-15/myobj?versionId=null 2022-09-12T18:55:28.778 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P6tFzdRBXJiW7wZL+JR6SHglxYQ= 2022-09-12T18:55:28.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P6tFzdRBXJiW7wZL+JR6SHglxYQ=', 'Content-Length': '0'} 2022-09-12T18:55:28.781 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f13745928297dd68-00631f80a0-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.782 DEBUG:boto:path=/jdpiak-15/myobj 2022-09-12T18:55:28.783 DEBUG:boto:auth_path=/jdpiak-15/myobj 2022-09-12T18:55:28.783 DEBUG:boto:path=/jdpiak-15/myobj?versionId=null 2022-09-12T18:55:28.784 DEBUG:boto:auth_path=/jdpiak-15/myobj?versionId=null 2022-09-12T18:55:28.784 DEBUG:boto:Method: HEAD 2022-09-12T18:55:28.785 DEBUG:boto:Path: /jdpiak-15/myobj?versionId=null 2022-09-12T18:55:28.785 DEBUG:boto:Data: 2022-09-12T18:55:28.786 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.787 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:28.788 DEBUG:boto:Port: 8001 2022-09-12T18:55:28.789 DEBUG:boto:Params: {} 2022-09-12T18:55:28.790 DEBUG:boto:Token: None 2022-09-12T18:55:28.791 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-15/myobj?versionId=null 2022-09-12T18:55:28.791 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P6tFzdRBXJiW7wZL+JR6SHglxYQ= 2022-09-12T18:55:28.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P6tFzdRBXJiW7wZL+JR6SHglxYQ=', 'Content-Length': '0'} 2022-09-12T18:55:28.794 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097d510ed6b42d46f-00631f80a0-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-12T18:55:28.796 DEBUG:boto:path=/jdpiak-15/myobj 2022-09-12T18:55:28.796 DEBUG:boto:auth_path=/jdpiak-15/myobj 2022-09-12T18:55:28.797 DEBUG:boto:path=/jdpiak-15/myobj?versionId=null 2022-09-12T18:55:28.798 DEBUG:boto:auth_path=/jdpiak-15/myobj?versionId=null 2022-09-12T18:55:28.799 DEBUG:boto:Method: GET 2022-09-12T18:55:28.800 DEBUG:boto:Path: /jdpiak-15/myobj?versionId=null 2022-09-12T18:55:28.800 DEBUG:boto:Data: 2022-09-12T18:55:28.801 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.802 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:28.803 DEBUG:boto:Port: 8000 2022-09-12T18:55:28.804 DEBUG:boto:Params: {} 2022-09-12T18:55:28.805 DEBUG:boto:Token: None 2022-09-12T18:55:28.806 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-15/myobj?versionId=null 2022-09-12T18:55:28.808 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EY1+VLGRY4WjhPcRDkCun0mYjPg= 2022-09-12T18:55:28.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EY1+VLGRY4WjhPcRDkCun0mYjPg=', 'Content-Length': '0'} 2022-09-12T18:55:28.811 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000482e11c07067f3c7-00631f80a0-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.812 DEBUG:boto:path=/jdpiak-15/myobj 2022-09-12T18:55:28.813 DEBUG:boto:auth_path=/jdpiak-15/myobj 2022-09-12T18:55:28.814 DEBUG:boto:path=/jdpiak-15/myobj?versionId=null 2022-09-12T18:55:28.815 DEBUG:boto:auth_path=/jdpiak-15/myobj?versionId=null 2022-09-12T18:55:28.815 DEBUG:boto:Method: GET 2022-09-12T18:55:28.816 DEBUG:boto:Path: /jdpiak-15/myobj?versionId=null 2022-09-12T18:55:28.817 DEBUG:boto:Data: 2022-09-12T18:55:28.818 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.819 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:28.820 DEBUG:boto:Port: 8001 2022-09-12T18:55:28.821 DEBUG:boto:Params: {} 2022-09-12T18:55:28.822 DEBUG:boto:Token: None 2022-09-12T18:55:28.823 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-15/myobj?versionId=null 2022-09-12T18:55:28.823 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EY1+VLGRY4WjhPcRDkCun0mYjPg= 2022-09-12T18:55:28.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EY1+VLGRY4WjhPcRDkCun0mYjPg=', 'Content-Length': '0'} 2022-09-12T18:55:28.828 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d449c27d9bdbdb04-00631f80a0-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.874 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:55:28.875 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:55:28.875 DEBUG:boto:path=/jdpiak-15/?prefix=myobj 2022-09-12T18:55:28.876 DEBUG:boto:auth_path=/jdpiak-15/?prefix=myobj 2022-09-12T18:55:28.876 DEBUG:boto:Method: GET 2022-09-12T18:55:28.877 DEBUG:boto:Path: /jdpiak-15/?prefix=myobj 2022-09-12T18:55:28.877 DEBUG:boto:Data: 2022-09-12T18:55:28.878 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.879 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:28.879 DEBUG:boto:Port: 8000 2022-09-12T18:55:28.880 DEBUG:boto:Params: {} 2022-09-12T18:55:28.881 DEBUG:boto:Token: None 2022-09-12T18:55:28.882 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-15/ 2022-09-12T18:55:28.883 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D6yTmIK6vB8d52DURzLaOwX3Jy0= 2022-09-12T18:55:28.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D6yTmIK6vB8d52DURzLaOwX3Jy0=', 'Content-Length': '0'} 2022-09-12T18:55:28.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cec0e81c7a9274e-00631f80a0-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.888 DEBUG:boto:b'jdpiak-15myobj1000falsemyobj2022-09-12T18:55:00.802Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:28.890 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:55:28.891 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:55:28.891 DEBUG:boto:path=/jdpiak-15/?prefix=myobj 2022-09-12T18:55:28.892 DEBUG:boto:auth_path=/jdpiak-15/?prefix=myobj 2022-09-12T18:55:28.893 DEBUG:boto:Method: GET 2022-09-12T18:55:28.894 DEBUG:boto:Path: /jdpiak-15/?prefix=myobj 2022-09-12T18:55:28.895 DEBUG:boto:Data: 2022-09-12T18:55:28.896 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.897 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:28.898 DEBUG:boto:Port: 8001 2022-09-12T18:55:28.898 DEBUG:boto:Params: {} 2022-09-12T18:55:28.899 DEBUG:boto:Token: None 2022-09-12T18:55:28.900 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-15/ 2022-09-12T18:55:28.901 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D6yTmIK6vB8d52DURzLaOwX3Jy0= 2022-09-12T18:55:28.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D6yTmIK6vB8d52DURzLaOwX3Jy0=', 'Content-Length': '0'} 2022-09-12T18:55:28.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054c7c416bb17d135-00631f80a0-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.907 DEBUG:boto:b'jdpiak-15myobj1000falsemyobj2022-09-12T18:55:00.802Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:28.908 DEBUG:boto:path=/jdpiak-15/myobj 2022-09-12T18:55:28.909 DEBUG:boto:auth_path=/jdpiak-15/myobj 2022-09-12T18:55:28.910 DEBUG:boto:Method: HEAD 2022-09-12T18:55:28.911 DEBUG:boto:Path: /jdpiak-15/myobj 2022-09-12T18:55:28.912 DEBUG:boto:Data: 2022-09-12T18:55:28.913 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.913 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:28.914 DEBUG:boto:Port: 8000 2022-09-12T18:55:28.915 DEBUG:boto:Params: {} 2022-09-12T18:55:28.916 DEBUG:boto:Token: None 2022-09-12T18:55:28.917 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-15/myobj 2022-09-12T18:55:28.918 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yTK82gxEctkn0SEDMH5/4R0CNX0= 2022-09-12T18:55:28.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yTK82gxEctkn0SEDMH5/4R0CNX0=', 'Content-Length': '0'} 2022-09-12T18:55:28.921 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6c7fac50b240435-00631f80a0-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.921 DEBUG:boto:path=/jdpiak-15/myobj 2022-09-12T18:55:28.922 DEBUG:boto:auth_path=/jdpiak-15/myobj 2022-09-12T18:55:28.923 DEBUG:boto:Method: HEAD 2022-09-12T18:55:28.924 DEBUG:boto:Path: /jdpiak-15/myobj 2022-09-12T18:55:28.925 DEBUG:boto:Data: 2022-09-12T18:55:28.926 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.927 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:28.928 DEBUG:boto:Port: 8001 2022-09-12T18:55:28.929 DEBUG:boto:Params: {} 2022-09-12T18:55:28.930 DEBUG:boto:Token: None 2022-09-12T18:55:28.931 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-15/myobj 2022-09-12T18:55:28.932 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yTK82gxEctkn0SEDMH5/4R0CNX0= 2022-09-12T18:55:28.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yTK82gxEctkn0SEDMH5/4R0CNX0=', 'Content-Length': '0'} 2022-09-12T18:55:28.935 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000009b8eb55efe1547b-00631f80a0-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-12T18:55:28.937 DEBUG:boto:path=/jdpiak-15/myobj 2022-09-12T18:55:28.939 DEBUG:boto:auth_path=/jdpiak-15/myobj 2022-09-12T18:55:28.940 DEBUG:boto:Method: GET 2022-09-12T18:55:28.941 DEBUG:boto:Path: /jdpiak-15/myobj 2022-09-12T18:55:28.942 DEBUG:boto:Data: 2022-09-12T18:55:28.943 DEBUG:boto:Headers: {} 2022-09-12T18:55:28.944 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:28.945 DEBUG:boto:Port: 8000 2022-09-12T18:55:28.946 DEBUG:boto:Params: {} 2022-09-12T18:55:28.947 DEBUG:boto:Token: None 2022-09-12T18:55:28.948 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:28 GMT /jdpiak-15/myobj 2022-09-12T18:55:28.948 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CokoeUvpjsZzk2MQ8Ev4PZMC08g= 2022-09-12T18:55:28.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CokoeUvpjsZzk2MQ8Ev4PZMC08g=', 'Content-Length': '0'} 2022-09-12T18:55:28.951 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000669b67a5aef757d4-00631f80a0-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:28.998 DEBUG:boto:path=/jdpiak-15/myobj 2022-09-12T18:55:28.999 DEBUG:boto:auth_path=/jdpiak-15/myobj 2022-09-12T18:55:29.000 DEBUG:boto:Method: GET 2022-09-12T18:55:29.000 DEBUG:boto:Path: /jdpiak-15/myobj 2022-09-12T18:55:29.001 DEBUG:boto:Data: 2022-09-12T18:55:29.002 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.002 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:29.003 DEBUG:boto:Port: 8001 2022-09-12T18:55:29.003 DEBUG:boto:Params: {} 2022-09-12T18:55:29.004 DEBUG:boto:Token: None 2022-09-12T18:55:29.005 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-15/myobj 2022-09-12T18:55:29.005 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KhHcWbMgnxFJzUwgvMJnTJdlPsw= 2022-09-12T18:55:29.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KhHcWbMgnxFJzUwgvMJnTJdlPsw=', 'Content-Length': '0'} 2022-09-12T18:55:29.008 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000096fcf24f1521b06-00631f80a1-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.009 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:55:29.010 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:55:29.010 DEBUG:boto:path=/jdpiak-15/?prefix=myobj 2022-09-12T18:55:29.011 DEBUG:boto:auth_path=/jdpiak-15/?prefix=myobj 2022-09-12T18:55:29.011 DEBUG:boto:Method: GET 2022-09-12T18:55:29.012 DEBUG:boto:Path: /jdpiak-15/?prefix=myobj 2022-09-12T18:55:29.012 DEBUG:boto:Data: 2022-09-12T18:55:29.013 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.014 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:29.014 DEBUG:boto:Port: 8000 2022-09-12T18:55:29.015 DEBUG:boto:Params: {} 2022-09-12T18:55:29.015 DEBUG:boto:Token: None 2022-09-12T18:55:29.016 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-15/ 2022-09-12T18:55:29.017 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oTvgSGT9g//ulgoa9qhR/2sOT+o= 2022-09-12T18:55:29.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oTvgSGT9g//ulgoa9qhR/2sOT+o=', 'Content-Length': '0'} 2022-09-12T18:55:29.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e26c567163908a0c-00631f80a1-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.020 DEBUG:boto:b'jdpiak-15myobj1000falsemyobj2022-09-12T18:55:00.802Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:29.021 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:55:29.022 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:55:29.022 DEBUG:boto:path=/jdpiak-15/?prefix=myobj 2022-09-12T18:55:29.023 DEBUG:boto:auth_path=/jdpiak-15/?prefix=myobj 2022-09-12T18:55:29.023 DEBUG:boto:Method: GET 2022-09-12T18:55:29.024 DEBUG:boto:Path: /jdpiak-15/?prefix=myobj 2022-09-12T18:55:29.024 DEBUG:boto:Data: 2022-09-12T18:55:29.025 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.025 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:29.026 DEBUG:boto:Port: 8001 2022-09-12T18:55:29.027 DEBUG:boto:Params: {} 2022-09-12T18:55:29.027 DEBUG:boto:Token: None 2022-09-12T18:55:29.028 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-15/ 2022-09-12T18:55:29.029 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oTvgSGT9g//ulgoa9qhR/2sOT+o= 2022-09-12T18:55:29.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oTvgSGT9g//ulgoa9qhR/2sOT+o=', 'Content-Length': '0'} 2022-09-12T18:55:29.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035b1322731e3704a-00631f80a1-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.033 DEBUG:boto:b'jdpiak-15myobj1000falsemyobj2022-09-12T18:55:00.802Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:29.033 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-15 zones={test-zone1, test-zone3} 2022-09-12T18:55:29.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:55:29.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:55:29.035 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T18:55:29.101 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:55:29.101 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:55:29.153 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:55:29.150+0000 7eff73d0d600 1 bucket sync caught up with source: 2022-09-12T18:55:29.154 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.41.5, , , , , , , , , ] 2022-09-12T18:55:29.154 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.41.5, , , , , , , , , ] 2022-09-12T18:55:29.154 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:55:29.150+0000 7eff73d0d600 0 bucket checkpoint complete 2022-09-12T18:55:29.167 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-15 zones={test-zone2, test-zone3} 2022-09-12T18:55:29.167 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:55:29.168 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:55:29.168 DEBUG:boto:Method: HEAD 2022-09-12T18:55:29.169 DEBUG:boto:Path: /jdpiak-15/ 2022-09-12T18:55:29.170 DEBUG:boto:Data: 2022-09-12T18:55:29.170 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.171 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:29.171 DEBUG:boto:Port: 8000 2022-09-12T18:55:29.172 DEBUG:boto:Params: {} 2022-09-12T18:55:29.172 DEBUG:boto:Token: None 2022-09-12T18:55:29.173 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-15/ 2022-09-12T18:55:29.174 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jRCIn96LHlLXaM3+R4+fVLujlNk= 2022-09-12T18:55:29.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jRCIn96LHlLXaM3+R4+fVLujlNk=', 'Content-Length': '0'} 2022-09-12T18:55:29.177 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a89ae970023e8482-00631f80a1-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.178 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:55:29.178 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:55:29.179 DEBUG:boto:Method: HEAD 2022-09-12T18:55:29.179 DEBUG:boto:Path: /jdpiak-15/ 2022-09-12T18:55:29.180 DEBUG:boto:Data: 2022-09-12T18:55:29.180 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.181 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:29.182 DEBUG:boto:Port: 8001 2022-09-12T18:55:29.183 DEBUG:boto:Params: {} 2022-09-12T18:55:29.183 DEBUG:boto:Token: None 2022-09-12T18:55:29.184 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-15/ 2022-09-12T18:55:29.184 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jRCIn96LHlLXaM3+R4+fVLujlNk= 2022-09-12T18:55:29.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jRCIn96LHlLXaM3+R4+fVLujlNk=', 'Content-Length': '0'} 2022-09-12T18:55:29.188 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036baacced8649c45-00631f80a1-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.189 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T18:55:29.190 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:55:29.190 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:55:29.191 DEBUG:boto:path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:55:29.191 DEBUG:boto:auth_path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:55:29.192 DEBUG:boto:Method: GET 2022-09-12T18:55:29.192 DEBUG:boto:Path: /jdpiak-15/?versions&max-keys=999 2022-09-12T18:55:29.193 DEBUG:boto:Data: 2022-09-12T18:55:29.194 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.195 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:29.195 DEBUG:boto:Port: 8000 2022-09-12T18:55:29.196 DEBUG:boto:Params: {} 2022-09-12T18:55:29.196 DEBUG:boto:Token: None 2022-09-12T18:55:29.197 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-15/?versions 2022-09-12T18:55:29.198 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1dMUvi8kdKDVrHhhxznP9/tXKXE= 2022-09-12T18:55:29.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1dMUvi8kdKDVrHhhxznP9/tXKXE=', 'Content-Length': '0'} 2022-09-12T18:55:29.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a6f537c5be89d51-00631f80a1-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.202 DEBUG:boto:b'jdpiak-15999falsemyobjnulltrue2022-09-12T18:55:00.802Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:29.204 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-12T18:55:29.204 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T18:55:29.205 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:55:29.206 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:55:29.207 DEBUG:boto:path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:55:29.208 DEBUG:boto:auth_path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:55:29.208 DEBUG:boto:Method: GET 2022-09-12T18:55:29.209 DEBUG:boto:Path: /jdpiak-15/?versions&max-keys=999 2022-09-12T18:55:29.210 DEBUG:boto:Data: 2022-09-12T18:55:29.211 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.211 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:29.212 DEBUG:boto:Port: 8001 2022-09-12T18:55:29.212 DEBUG:boto:Params: {} 2022-09-12T18:55:29.213 DEBUG:boto:Token: None 2022-09-12T18:55:29.213 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-15/?versions 2022-09-12T18:55:29.214 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1dMUvi8kdKDVrHhhxznP9/tXKXE= 2022-09-12T18:55:29.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1dMUvi8kdKDVrHhhxznP9/tXKXE=', 'Content-Length': '0'} 2022-09-12T18:55:29.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000753439811a8762c0-00631f80a1-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.219 DEBUG:boto:b'jdpiak-15999falsemyobjnulltrue2022-09-12T18:55:00.802Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:29.220 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-12T18:55:29.220 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:55:29.221 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:55:29.222 DEBUG:boto:path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:55:29.222 DEBUG:boto:auth_path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:55:29.223 DEBUG:boto:Method: GET 2022-09-12T18:55:29.224 DEBUG:boto:Path: /jdpiak-15/?versions&max-keys=999 2022-09-12T18:55:29.225 DEBUG:boto:Data: 2022-09-12T18:55:29.226 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.226 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:29.227 DEBUG:boto:Port: 8000 2022-09-12T18:55:29.227 DEBUG:boto:Params: {} 2022-09-12T18:55:29.228 DEBUG:boto:Token: None 2022-09-12T18:55:29.229 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-15/?versions 2022-09-12T18:55:29.229 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1dMUvi8kdKDVrHhhxznP9/tXKXE= 2022-09-12T18:55:29.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1dMUvi8kdKDVrHhhxznP9/tXKXE=', 'Content-Length': '0'} 2022-09-12T18:55:29.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092831bde712b4ba4-00631f80a1-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.233 DEBUG:boto:b'jdpiak-15999falsemyobjnulltrue2022-09-12T18:55:00.802Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:29.234 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:55:29.235 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:55:29.235 DEBUG:boto:path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:55:29.236 DEBUG:boto:auth_path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:55:29.236 DEBUG:boto:Method: GET 2022-09-12T18:55:29.237 DEBUG:boto:Path: /jdpiak-15/?versions&max-keys=999 2022-09-12T18:55:29.237 DEBUG:boto:Data: 2022-09-12T18:55:29.238 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.238 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:29.239 DEBUG:boto:Port: 8001 2022-09-12T18:55:29.239 DEBUG:boto:Params: {} 2022-09-12T18:55:29.240 DEBUG:boto:Token: None 2022-09-12T18:55:29.240 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-15/?versions 2022-09-12T18:55:29.241 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1dMUvi8kdKDVrHhhxznP9/tXKXE= 2022-09-12T18:55:29.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1dMUvi8kdKDVrHhhxznP9/tXKXE=', 'Content-Length': '0'} 2022-09-12T18:55:29.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f312f1b8f550ede-00631f80a1-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.245 DEBUG:boto:b'jdpiak-15999falsemyobjnulltrue2022-09-12T18:55:00.802Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:29.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-12T18:55:29.246 DEBUG:boto:path=/jdpiak-15/myobj 2022-09-12T18:55:29.247 DEBUG:boto:auth_path=/jdpiak-15/myobj 2022-09-12T18:55:29.247 DEBUG:boto:path=/jdpiak-15/myobj?versionId=null 2022-09-12T18:55:29.248 DEBUG:boto:auth_path=/jdpiak-15/myobj?versionId=null 2022-09-12T18:55:29.248 DEBUG:boto:Method: GET 2022-09-12T18:55:29.249 DEBUG:boto:Path: /jdpiak-15/myobj?versionId=null 2022-09-12T18:55:29.249 DEBUG:boto:Data: 2022-09-12T18:55:29.250 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.250 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:29.251 DEBUG:boto:Port: 8000 2022-09-12T18:55:29.252 DEBUG:boto:Params: {} 2022-09-12T18:55:29.252 DEBUG:boto:Token: None 2022-09-12T18:55:29.253 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-15/myobj?versionId=null 2022-09-12T18:55:29.253 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kf69MHiNSy0C2t9SlbdJdnuydwI= 2022-09-12T18:55:29.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kf69MHiNSy0C2t9SlbdJdnuydwI=', 'Content-Length': '0'} 2022-09-12T18:55:29.256 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bfeaedd8b1a19dc6-00631f80a1-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.257 DEBUG:boto:path=/jdpiak-15/myobj 2022-09-12T18:55:29.257 DEBUG:boto:auth_path=/jdpiak-15/myobj 2022-09-12T18:55:29.258 DEBUG:boto:path=/jdpiak-15/myobj?versionId=null 2022-09-12T18:55:29.259 DEBUG:boto:auth_path=/jdpiak-15/myobj?versionId=null 2022-09-12T18:55:29.259 DEBUG:boto:Method: GET 2022-09-12T18:55:29.260 DEBUG:boto:Path: /jdpiak-15/myobj?versionId=null 2022-09-12T18:55:29.260 DEBUG:boto:Data: 2022-09-12T18:55:29.261 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.261 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:29.262 DEBUG:boto:Port: 8001 2022-09-12T18:55:29.262 DEBUG:boto:Params: {} 2022-09-12T18:55:29.263 DEBUG:boto:Token: None 2022-09-12T18:55:29.263 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-15/myobj?versionId=null 2022-09-12T18:55:29.264 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kf69MHiNSy0C2t9SlbdJdnuydwI= 2022-09-12T18:55:29.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kf69MHiNSy0C2t9SlbdJdnuydwI=', 'Content-Length': '0'} 2022-09-12T18:55:29.267 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8e54b3e67cc1a1f-00631f80a1-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.268 DEBUG:boto:path=/jdpiak-15/myobj 2022-09-12T18:55:29.268 DEBUG:boto:auth_path=/jdpiak-15/myobj 2022-09-12T18:55:29.269 DEBUG:boto:path=/jdpiak-15/myobj?versionId=null 2022-09-12T18:55:29.269 DEBUG:boto:auth_path=/jdpiak-15/myobj?versionId=null 2022-09-12T18:55:29.270 DEBUG:boto:Method: HEAD 2022-09-12T18:55:29.271 DEBUG:boto:Path: /jdpiak-15/myobj?versionId=null 2022-09-12T18:55:29.271 DEBUG:boto:Data: 2022-09-12T18:55:29.272 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.272 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:29.273 DEBUG:boto:Port: 8000 2022-09-12T18:55:29.273 DEBUG:boto:Params: {} 2022-09-12T18:55:29.274 DEBUG:boto:Token: None 2022-09-12T18:55:29.274 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-15/myobj?versionId=null 2022-09-12T18:55:29.275 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gwLxxM59I4Bf2TI20u/90NiUJFQ= 2022-09-12T18:55:29.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gwLxxM59I4Bf2TI20u/90NiUJFQ=', 'Content-Length': '0'} 2022-09-12T18:55:29.278 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067d0b3af4e483050-00631f80a1-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.278 DEBUG:boto:path=/jdpiak-15/myobj 2022-09-12T18:55:29.279 DEBUG:boto:auth_path=/jdpiak-15/myobj 2022-09-12T18:55:29.279 DEBUG:boto:path=/jdpiak-15/myobj?versionId=null 2022-09-12T18:55:29.280 DEBUG:boto:auth_path=/jdpiak-15/myobj?versionId=null 2022-09-12T18:55:29.280 DEBUG:boto:Method: HEAD 2022-09-12T18:55:29.281 DEBUG:boto:Path: /jdpiak-15/myobj?versionId=null 2022-09-12T18:55:29.282 DEBUG:boto:Data: 2022-09-12T18:55:29.282 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.283 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:29.284 DEBUG:boto:Port: 8001 2022-09-12T18:55:29.285 DEBUG:boto:Params: {} 2022-09-12T18:55:29.286 DEBUG:boto:Token: None 2022-09-12T18:55:29.287 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-15/myobj?versionId=null 2022-09-12T18:55:29.288 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gwLxxM59I4Bf2TI20u/90NiUJFQ= 2022-09-12T18:55:29.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gwLxxM59I4Bf2TI20u/90NiUJFQ=', 'Content-Length': '0'} 2022-09-12T18:55:29.291 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a5369488afd4794-00631f80a1-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-12T18:55:29.292 DEBUG:boto:path=/jdpiak-15/myobj 2022-09-12T18:55:29.293 DEBUG:boto:auth_path=/jdpiak-15/myobj 2022-09-12T18:55:29.293 DEBUG:boto:path=/jdpiak-15/myobj?versionId=null 2022-09-12T18:55:29.294 DEBUG:boto:auth_path=/jdpiak-15/myobj?versionId=null 2022-09-12T18:55:29.295 DEBUG:boto:Method: GET 2022-09-12T18:55:29.295 DEBUG:boto:Path: /jdpiak-15/myobj?versionId=null 2022-09-12T18:55:29.296 DEBUG:boto:Data: 2022-09-12T18:55:29.296 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.297 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:29.298 DEBUG:boto:Port: 8000 2022-09-12T18:55:29.298 DEBUG:boto:Params: {} 2022-09-12T18:55:29.299 DEBUG:boto:Token: None 2022-09-12T18:55:29.299 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-15/myobj?versionId=null 2022-09-12T18:55:29.300 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kf69MHiNSy0C2t9SlbdJdnuydwI= 2022-09-12T18:55:29.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kf69MHiNSy0C2t9SlbdJdnuydwI=', 'Content-Length': '0'} 2022-09-12T18:55:29.303 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008dbb8bb970b8ad0b-00631f80a1-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.303 DEBUG:boto:path=/jdpiak-15/myobj 2022-09-12T18:55:29.304 DEBUG:boto:auth_path=/jdpiak-15/myobj 2022-09-12T18:55:29.304 DEBUG:boto:path=/jdpiak-15/myobj?versionId=null 2022-09-12T18:55:29.305 DEBUG:boto:auth_path=/jdpiak-15/myobj?versionId=null 2022-09-12T18:55:29.306 DEBUG:boto:Method: GET 2022-09-12T18:55:29.307 DEBUG:boto:Path: /jdpiak-15/myobj?versionId=null 2022-09-12T18:55:29.308 DEBUG:boto:Data: 2022-09-12T18:55:29.309 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.309 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:29.310 DEBUG:boto:Port: 8001 2022-09-12T18:55:29.311 DEBUG:boto:Params: {} 2022-09-12T18:55:29.312 DEBUG:boto:Token: None 2022-09-12T18:55:29.313 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-15/myobj?versionId=null 2022-09-12T18:55:29.313 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kf69MHiNSy0C2t9SlbdJdnuydwI= 2022-09-12T18:55:29.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kf69MHiNSy0C2t9SlbdJdnuydwI=', 'Content-Length': '0'} 2022-09-12T18:55:29.317 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d877f85d76c9120e-00631f80a1-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.358 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:55:29.359 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:55:29.359 DEBUG:boto:path=/jdpiak-15/?prefix=myobj 2022-09-12T18:55:29.360 DEBUG:boto:auth_path=/jdpiak-15/?prefix=myobj 2022-09-12T18:55:29.361 DEBUG:boto:Method: GET 2022-09-12T18:55:29.362 DEBUG:boto:Path: /jdpiak-15/?prefix=myobj 2022-09-12T18:55:29.363 DEBUG:boto:Data: 2022-09-12T18:55:29.364 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.364 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:29.365 DEBUG:boto:Port: 8000 2022-09-12T18:55:29.365 DEBUG:boto:Params: {} 2022-09-12T18:55:29.366 DEBUG:boto:Token: None 2022-09-12T18:55:29.367 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-15/ 2022-09-12T18:55:29.368 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oTvgSGT9g//ulgoa9qhR/2sOT+o= 2022-09-12T18:55:29.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oTvgSGT9g//ulgoa9qhR/2sOT+o=', 'Content-Length': '0'} 2022-09-12T18:55:29.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002638c02296b9dbf9-00631f80a1-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.372 DEBUG:boto:b'jdpiak-15myobj1000falsemyobj2022-09-12T18:55:00.802Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:29.373 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:55:29.373 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:55:29.374 DEBUG:boto:path=/jdpiak-15/?prefix=myobj 2022-09-12T18:55:29.374 DEBUG:boto:auth_path=/jdpiak-15/?prefix=myobj 2022-09-12T18:55:29.375 DEBUG:boto:Method: GET 2022-09-12T18:55:29.375 DEBUG:boto:Path: /jdpiak-15/?prefix=myobj 2022-09-12T18:55:29.376 DEBUG:boto:Data: 2022-09-12T18:55:29.376 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.377 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:29.377 DEBUG:boto:Port: 8001 2022-09-12T18:55:29.378 DEBUG:boto:Params: {} 2022-09-12T18:55:29.379 DEBUG:boto:Token: None 2022-09-12T18:55:29.380 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-15/ 2022-09-12T18:55:29.381 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oTvgSGT9g//ulgoa9qhR/2sOT+o= 2022-09-12T18:55:29.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oTvgSGT9g//ulgoa9qhR/2sOT+o=', 'Content-Length': '0'} 2022-09-12T18:55:29.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016b26ecb5e468241-00631f80a1-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.386 DEBUG:boto:b'jdpiak-15myobj1000falsemyobj2022-09-12T18:55:00.802Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:29.387 DEBUG:boto:path=/jdpiak-15/myobj 2022-09-12T18:55:29.388 DEBUG:boto:auth_path=/jdpiak-15/myobj 2022-09-12T18:55:29.389 DEBUG:boto:Method: HEAD 2022-09-12T18:55:29.389 DEBUG:boto:Path: /jdpiak-15/myobj 2022-09-12T18:55:29.390 DEBUG:boto:Data: 2022-09-12T18:55:29.390 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.391 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:29.391 DEBUG:boto:Port: 8000 2022-09-12T18:55:29.392 DEBUG:boto:Params: {} 2022-09-12T18:55:29.392 DEBUG:boto:Token: None 2022-09-12T18:55:29.393 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-15/myobj 2022-09-12T18:55:29.394 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XeYjrA4IeOh3siUJ89pom9jjQRE= 2022-09-12T18:55:29.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XeYjrA4IeOh3siUJ89pom9jjQRE=', 'Content-Length': '0'} 2022-09-12T18:55:29.396 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058b6cc1caf4b38fa-00631f80a1-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.397 DEBUG:boto:path=/jdpiak-15/myobj 2022-09-12T18:55:29.397 DEBUG:boto:auth_path=/jdpiak-15/myobj 2022-09-12T18:55:29.398 DEBUG:boto:Method: HEAD 2022-09-12T18:55:29.398 DEBUG:boto:Path: /jdpiak-15/myobj 2022-09-12T18:55:29.399 DEBUG:boto:Data: 2022-09-12T18:55:29.399 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.400 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:29.400 DEBUG:boto:Port: 8001 2022-09-12T18:55:29.401 DEBUG:boto:Params: {} 2022-09-12T18:55:29.402 DEBUG:boto:Token: None 2022-09-12T18:55:29.402 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-15/myobj 2022-09-12T18:55:29.403 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XeYjrA4IeOh3siUJ89pom9jjQRE= 2022-09-12T18:55:29.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XeYjrA4IeOh3siUJ89pom9jjQRE=', 'Content-Length': '0'} 2022-09-12T18:55:29.405 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f16964fab73dbab-00631f80a1-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-12T18:55:29.407 DEBUG:boto:path=/jdpiak-15/myobj 2022-09-12T18:55:29.407 DEBUG:boto:auth_path=/jdpiak-15/myobj 2022-09-12T18:55:29.408 DEBUG:boto:Method: GET 2022-09-12T18:55:29.409 DEBUG:boto:Path: /jdpiak-15/myobj 2022-09-12T18:55:29.409 DEBUG:boto:Data: 2022-09-12T18:55:29.410 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.410 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:29.411 DEBUG:boto:Port: 8000 2022-09-12T18:55:29.412 DEBUG:boto:Params: {} 2022-09-12T18:55:29.412 DEBUG:boto:Token: None 2022-09-12T18:55:29.413 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-15/myobj 2022-09-12T18:55:29.414 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KhHcWbMgnxFJzUwgvMJnTJdlPsw= 2022-09-12T18:55:29.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KhHcWbMgnxFJzUwgvMJnTJdlPsw=', 'Content-Length': '0'} 2022-09-12T18:55:29.417 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a88f49640c11a77-00631f80a1-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.417 DEBUG:boto:path=/jdpiak-15/myobj 2022-09-12T18:55:29.418 DEBUG:boto:auth_path=/jdpiak-15/myobj 2022-09-12T18:55:29.419 DEBUG:boto:Method: GET 2022-09-12T18:55:29.419 DEBUG:boto:Path: /jdpiak-15/myobj 2022-09-12T18:55:29.420 DEBUG:boto:Data: 2022-09-12T18:55:29.420 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.421 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:29.421 DEBUG:boto:Port: 8001 2022-09-12T18:55:29.422 DEBUG:boto:Params: {} 2022-09-12T18:55:29.423 DEBUG:boto:Token: None 2022-09-12T18:55:29.423 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-15/myobj 2022-09-12T18:55:29.424 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KhHcWbMgnxFJzUwgvMJnTJdlPsw= 2022-09-12T18:55:29.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KhHcWbMgnxFJzUwgvMJnTJdlPsw=', 'Content-Length': '0'} 2022-09-12T18:55:29.427 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015d4e694c1893e1d-00631f80a1-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.428 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:55:29.428 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:55:29.429 DEBUG:boto:path=/jdpiak-15/?prefix=myobj 2022-09-12T18:55:29.429 DEBUG:boto:auth_path=/jdpiak-15/?prefix=myobj 2022-09-12T18:55:29.430 DEBUG:boto:Method: GET 2022-09-12T18:55:29.430 DEBUG:boto:Path: /jdpiak-15/?prefix=myobj 2022-09-12T18:55:29.431 DEBUG:boto:Data: 2022-09-12T18:55:29.431 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.432 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:55:29.433 DEBUG:boto:Port: 8000 2022-09-12T18:55:29.433 DEBUG:boto:Params: {} 2022-09-12T18:55:29.435 DEBUG:boto:Token: None 2022-09-12T18:55:29.436 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-15/ 2022-09-12T18:55:29.436 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oTvgSGT9g//ulgoa9qhR/2sOT+o= 2022-09-12T18:55:29.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oTvgSGT9g//ulgoa9qhR/2sOT+o=', 'Content-Length': '0'} 2022-09-12T18:55:29.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000156f4e0fa71ed69d-00631f80a1-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.441 DEBUG:boto:b'jdpiak-15myobj1000falsemyobj2022-09-12T18:55:00.802Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:29.442 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:55:29.443 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:55:29.443 DEBUG:boto:path=/jdpiak-15/?prefix=myobj 2022-09-12T18:55:29.444 DEBUG:boto:auth_path=/jdpiak-15/?prefix=myobj 2022-09-12T18:55:29.444 DEBUG:boto:Method: GET 2022-09-12T18:55:29.445 DEBUG:boto:Path: /jdpiak-15/?prefix=myobj 2022-09-12T18:55:29.446 DEBUG:boto:Data: 2022-09-12T18:55:29.446 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.447 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:55:29.447 DEBUG:boto:Port: 8001 2022-09-12T18:55:29.448 DEBUG:boto:Params: {} 2022-09-12T18:55:29.449 DEBUG:boto:Token: None 2022-09-12T18:55:29.449 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-15/ 2022-09-12T18:55:29.450 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oTvgSGT9g//ulgoa9qhR/2sOT+o= 2022-09-12T18:55:29.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oTvgSGT9g//ulgoa9qhR/2sOT+o=', 'Content-Length': '0'} 2022-09-12T18:55:29.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e171fb34fe8275b4-00631f80a1-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.454 DEBUG:boto:b'jdpiak-15myobj1000falsemyobj2022-09-12T18:55:00.802Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T18:55:29.455 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-15 zones={test-zone2, test-zone3} 2022-09-12T18:55:29.455 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:55:29.456 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:55:29.456 DEBUG:boto:Method: HEAD 2022-09-12T18:55:29.457 DEBUG:boto:Path: /jdpiak-13/ 2022-09-12T18:55:29.457 DEBUG:boto:Data: 2022-09-12T18:55:29.458 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.458 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:29.459 DEBUG:boto:Port: 8000 2022-09-12T18:55:29.459 DEBUG:boto:Params: {} 2022-09-12T18:55:29.460 DEBUG:boto:Token: None 2022-09-12T18:55:29.460 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-13/ 2022-09-12T18:55:29.461 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+q7DMAZsUjHXbEwPCQ1k9k4zGBE= 2022-09-12T18:55:29.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+q7DMAZsUjHXbEwPCQ1k9k4zGBE=', 'Content-Length': '0'} 2022-09-12T18:55:29.464 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa715b65468bd636-00631f80a1-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.464 DEBUG:boto:path=/jdpiak-13/myobj 2022-09-12T18:55:29.465 DEBUG:boto:auth_path=/jdpiak-13/myobj 2022-09-12T18:55:29.466 DEBUG:boto:Method: HEAD 2022-09-12T18:55:29.466 DEBUG:boto:Path: /jdpiak-13/myobj 2022-09-12T18:55:29.467 DEBUG:boto:Data: 2022-09-12T18:55:29.467 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.468 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:29.468 DEBUG:boto:Port: 8000 2022-09-12T18:55:29.469 DEBUG:boto:Params: {} 2022-09-12T18:55:29.469 DEBUG:boto:Token: None 2022-09-12T18:55:29.470 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-13/myobj 2022-09-12T18:55:29.470 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/3qJ1FIK+jrBbGmA4u7wBzBRjes= 2022-09-12T18:55:29.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/3qJ1FIK+jrBbGmA4u7wBzBRjes=', 'Content-Length': '0'} 2022-09-12T18:55:29.473 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5f24da735ca42cd-00631f80a1-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.474 DEBUG:boto:path=/jdpiak-13/myobj 2022-09-12T18:55:29.475 DEBUG:boto:auth_path=/jdpiak-13/myobj 2022-09-12T18:55:29.475 DEBUG:boto:Method: DELETE 2022-09-12T18:55:29.476 DEBUG:boto:Path: /jdpiak-13/myobj 2022-09-12T18:55:29.476 DEBUG:boto:Data: 2022-09-12T18:55:29.477 DEBUG:boto:Headers: {} 2022-09-12T18:55:29.477 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:55:29.478 DEBUG:boto:Port: 8000 2022-09-12T18:55:29.478 DEBUG:boto:Params: {} 2022-09-12T18:55:29.479 DEBUG:boto:Token: None 2022-09-12T18:55:29.479 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 18:55:29 GMT /jdpiak-13/myobj 2022-09-12T18:55:29.480 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iYH0N213zepT92G1Y0nKQPW7d5E= 2022-09-12T18:55:29.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:55:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iYH0N213zepT92G1Y0nKQPW7d5E=', 'Content-Length': '0'} 2022-09-12T18:55:29.485 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000735be0b9dc8fe256-00631f80a1-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 18:55:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:55:29.485 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:55:29.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:55:29.487 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T18:55:29.547 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:55:29.547 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:55:29.582 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:55:29.579+0000 7fd1ef593600 1 waiting for incremental sync to catch up: 2022-09-12T18:55:29.583 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:55:29.583 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000002.18.5, , , , , , , , , ] 2022-09-12T18:55:34.584 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:55:34.580+0000 7fd1ef593600 1 waiting for incremental sync to catch up: 2022-09-12T18:55:34.584 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:55:34.584 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000002.18.5, , , , , , , , , ] 2022-09-12T18:55:39.585 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:55:39.582+0000 7fd1ef593600 1 waiting for incremental sync to catch up: 2022-09-12T18:55:39.586 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:55:39.586 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000002.18.5, , , , , , , , , ] 2022-09-12T18:55:44.586 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:55:44.583+0000 7fd1ef593600 1 waiting for incremental sync to catch up: 2022-09-12T18:55:44.587 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:55:44.587 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000002.18.5, , , , , , , , , ] 2022-09-12T18:55:49.588 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:55:49.584+0000 7fd1ef593600 1 waiting for incremental sync to catch up: 2022-09-12T18:55:49.588 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:55:49.589 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000002.18.5, , , , , , , , , ] 2022-09-12T18:55:54.590 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:55:54.586+0000 7fd1ef593600 1 waiting for incremental sync to catch up: 2022-09-12T18:55:54.590 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:55:54.591 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000002.18.5, , , , , , , , , ] 2022-09-12T18:55:59.591 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:55:59.587+0000 7fd1ef593600 1 waiting for incremental sync to catch up: 2022-09-12T18:55:59.591 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:55:59.592 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000002.18.5, , , , , , , , , ] 2022-09-12T18:56:04.592 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:56:04.589+0000 7fd1ef593600 1 bucket sync caught up with source: 2022-09-12T18:56:04.593 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.18.5, , , , , , , , , ] 2022-09-12T18:56:04.594 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000002.18.5, , , , , , , , , ] 2022-09-12T18:56:04.594 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:56:04.589+0000 7fd1ef593600 0 bucket checkpoint complete 2022-09-12T18:56:04.613 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-13 zones={test-zone2, test-zone1} 2022-09-12T18:56:04.615 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:56:04.616 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:56:04.617 DEBUG:boto:Method: HEAD 2022-09-12T18:56:04.618 DEBUG:boto:Path: /jdpiak-13/ 2022-09-12T18:56:04.619 DEBUG:boto:Data: 2022-09-12T18:56:04.619 DEBUG:boto:Headers: {} 2022-09-12T18:56:04.620 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:56:04.620 DEBUG:boto:Port: 8000 2022-09-12T18:56:04.621 DEBUG:boto:Params: {} 2022-09-12T18:56:04.622 DEBUG:boto:Token: None 2022-09-12T18:56:04.622 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:56:04 GMT /jdpiak-13/ 2022-09-12T18:56:04.623 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iQTJSPGRdkUyY7LWrzfEMhyKUnM= 2022-09-12T18:56:04.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:56:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iQTJSPGRdkUyY7LWrzfEMhyKUnM=', 'Content-Length': '0'} 2022-09-12T18:56:04.627 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000842a46f2d688ecb4-00631f80c4-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:56:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:56:04.627 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:56:04.628 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:56:04.629 DEBUG:boto:Method: HEAD 2022-09-12T18:56:04.629 DEBUG:boto:Path: /jdpiak-13/ 2022-09-12T18:56:04.630 DEBUG:boto:Data: 2022-09-12T18:56:04.630 DEBUG:boto:Headers: {} 2022-09-12T18:56:04.631 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:56:04.631 DEBUG:boto:Port: 8000 2022-09-12T18:56:04.632 DEBUG:boto:Params: {} 2022-09-12T18:56:04.632 DEBUG:boto:Token: None 2022-09-12T18:56:04.633 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:56:04 GMT /jdpiak-13/ 2022-09-12T18:56:04.634 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iQTJSPGRdkUyY7LWrzfEMhyKUnM= 2022-09-12T18:56:04.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:56:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iQTJSPGRdkUyY7LWrzfEMhyKUnM=', 'Content-Length': '0'} 2022-09-12T18:56:04.637 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e34c50c8c169aebd-00631f80c4-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:56:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:56:04.638 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T18:56:04.639 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:56:04.640 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:56:04.641 DEBUG:boto:path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:56:04.641 DEBUG:boto:auth_path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:56:04.642 DEBUG:boto:Method: GET 2022-09-12T18:56:04.643 DEBUG:boto:Path: /jdpiak-13/?versions&max-keys=999 2022-09-12T18:56:04.644 DEBUG:boto:Data: 2022-09-12T18:56:04.645 DEBUG:boto:Headers: {} 2022-09-12T18:56:04.647 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:56:04.648 DEBUG:boto:Port: 8000 2022-09-12T18:56:04.648 DEBUG:boto:Params: {} 2022-09-12T18:56:04.649 DEBUG:boto:Token: None 2022-09-12T18:56:04.650 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:56:04 GMT /jdpiak-13/?versions 2022-09-12T18:56:04.651 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OiWrPBxIAamoY8EXxBGNIn9OkSY= 2022-09-12T18:56:04.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:56:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OiWrPBxIAamoY8EXxBGNIn9OkSY=', 'Content-Length': '0'} 2022-09-12T18:56:04.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000408628afcd9ada02-00631f80c4-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:56:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:56:04.698 DEBUG:boto:b'jdpiak-13999false' 2022-09-12T18:56:04.699 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T18:56:04.700 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:56:04.700 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:56:04.701 DEBUG:boto:path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:56:04.701 DEBUG:boto:auth_path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:56:04.702 DEBUG:boto:Method: GET 2022-09-12T18:56:04.703 DEBUG:boto:Path: /jdpiak-13/?versions&max-keys=999 2022-09-12T18:56:04.703 DEBUG:boto:Data: 2022-09-12T18:56:04.704 DEBUG:boto:Headers: {} 2022-09-12T18:56:04.704 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:56:04.705 DEBUG:boto:Port: 8000 2022-09-12T18:56:04.706 DEBUG:boto:Params: {} 2022-09-12T18:56:04.706 DEBUG:boto:Token: None 2022-09-12T18:56:04.707 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:56:04 GMT /jdpiak-13/?versions 2022-09-12T18:56:04.708 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OiWrPBxIAamoY8EXxBGNIn9OkSY= 2022-09-12T18:56:04.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:56:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OiWrPBxIAamoY8EXxBGNIn9OkSY=', 'Content-Length': '0'} 2022-09-12T18:56:04.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c48fae1a57a2fb9f-00631f80c4-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:56:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:56:04.712 DEBUG:boto:b'jdpiak-13999false' 2022-09-12T18:56:04.713 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:56:04.713 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:56:04.714 DEBUG:boto:path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:56:04.714 DEBUG:boto:auth_path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:56:04.715 DEBUG:boto:Method: GET 2022-09-12T18:56:04.715 DEBUG:boto:Path: /jdpiak-13/?versions&max-keys=999 2022-09-12T18:56:04.716 DEBUG:boto:Data: 2022-09-12T18:56:04.716 DEBUG:boto:Headers: {} 2022-09-12T18:56:04.717 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:56:04.717 DEBUG:boto:Port: 8000 2022-09-12T18:56:04.718 DEBUG:boto:Params: {} 2022-09-12T18:56:04.719 DEBUG:boto:Token: None 2022-09-12T18:56:04.719 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:56:04 GMT /jdpiak-13/?versions 2022-09-12T18:56:04.720 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OiWrPBxIAamoY8EXxBGNIn9OkSY= 2022-09-12T18:56:04.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:56:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OiWrPBxIAamoY8EXxBGNIn9OkSY=', 'Content-Length': '0'} 2022-09-12T18:56:04.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092a9ced049ecdbc0-00631f80c4-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:56:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:56:04.724 DEBUG:boto:b'jdpiak-13999false' 2022-09-12T18:56:04.725 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:56:04.725 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:56:04.726 DEBUG:boto:path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:56:04.726 DEBUG:boto:auth_path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:56:04.727 DEBUG:boto:Method: GET 2022-09-12T18:56:04.727 DEBUG:boto:Path: /jdpiak-13/?versions&max-keys=999 2022-09-12T18:56:04.728 DEBUG:boto:Data: 2022-09-12T18:56:04.728 DEBUG:boto:Headers: {} 2022-09-12T18:56:04.729 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:56:04.729 DEBUG:boto:Port: 8000 2022-09-12T18:56:04.730 DEBUG:boto:Params: {} 2022-09-12T18:56:04.730 DEBUG:boto:Token: None 2022-09-12T18:56:04.731 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:56:04 GMT /jdpiak-13/?versions 2022-09-12T18:56:04.731 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OiWrPBxIAamoY8EXxBGNIn9OkSY= 2022-09-12T18:56:04.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:56:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OiWrPBxIAamoY8EXxBGNIn9OkSY=', 'Content-Length': '0'} 2022-09-12T18:56:04.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fb12e6a73f9a518-00631f80c4-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:56:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:56:04.736 DEBUG:boto:b'jdpiak-13999false' 2022-09-12T18:56:04.736 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-13 zones={test-zone2, test-zone1} 2022-09-12T18:56:04.737 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:56:04.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:56:04.738 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T18:56:04.804 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:56:04.804 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:56:04.856 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:56:04.854+0000 7fb4810ef600 1 waiting for incremental sync to catch up: 2022-09-12T18:56:04.857 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:56:04.857 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.18.5, , , , , , , , , ] 2022-09-12T18:56:09.858 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:56:09.854+0000 7fb4810ef600 1 waiting for incremental sync to catch up: 2022-09-12T18:56:09.858 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:56:09.859 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.18.5, , , , , , , , , ] 2022-09-12T18:56:14.859 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:56:14.856+0000 7fb4810ef600 1 waiting for incremental sync to catch up: 2022-09-12T18:56:14.860 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:56:14.861 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.18.5, , , , , , , , , ] 2022-09-12T18:56:19.861 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:56:19.857+0000 7fb4810ef600 1 waiting for incremental sync to catch up: 2022-09-12T18:56:19.861 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:56:19.862 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.18.5, , , , , , , , , ] 2022-09-12T18:56:24.863 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:56:24.859+0000 7fb4810ef600 1 waiting for incremental sync to catch up: 2022-09-12T18:56:24.863 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:56:24.863 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.18.5, , , , , , , , , ] 2022-09-12T18:56:29.864 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:56:29.860+0000 7fb4810ef600 1 waiting for incremental sync to catch up: 2022-09-12T18:56:29.864 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:56:29.865 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.18.5, , , , , , , , , ] 2022-09-12T18:56:34.865 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:56:34.862+0000 7fb4810ef600 1 waiting for incremental sync to catch up: 2022-09-12T18:56:34.866 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:56:34.866 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.18.5, , , , , , , , , ] 2022-09-12T18:56:39.867 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:56:39.863+0000 7fb4810ef600 1 waiting for incremental sync to catch up: 2022-09-12T18:56:39.867 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:56:39.868 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.18.5, , , , , , , , , ] 2022-09-12T18:56:44.868 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:56:44.865+0000 7fb4810ef600 1 waiting for incremental sync to catch up: 2022-09-12T18:56:44.869 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:56:44.869 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.18.5, , , , , , , , , ] 2022-09-12T18:56:49.870 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:56:49.867+0000 7fb4810ef600 1 waiting for incremental sync to catch up: 2022-09-12T18:56:49.870 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:56:49.870 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.18.5, , , , , , , , , ] 2022-09-12T18:56:54.871 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:56:54.868+0000 7fb4810ef600 1 waiting for incremental sync to catch up: 2022-09-12T18:56:54.872 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:56:54.872 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.18.5, , , , , , , , , ] 2022-09-12T18:56:59.873 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:56:59.869+0000 7fb4810ef600 1 waiting for incremental sync to catch up: 2022-09-12T18:56:59.873 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.16.5, , , , , , , , , ] 2022-09-12T18:56:59.874 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.18.5, , , , , , , , , ] 2022-09-12T18:57:04.874 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:57:04.870+0000 7fb4810ef600 1 bucket sync caught up with source: 2022-09-12T18:57:04.875 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.18.5, , , , , , , , , ] 2022-09-12T18:57:04.875 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.18.5, , , , , , , , , ] 2022-09-12T18:57:04.876 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:57:04.870+0000 7fb4810ef600 0 bucket checkpoint complete 2022-09-12T18:57:04.891 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-13 zones={test-zone3, test-zone1} 2022-09-12T18:57:04.893 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:57:04.893 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:57:04.894 DEBUG:boto:Method: HEAD 2022-09-12T18:57:04.895 DEBUG:boto:Path: /jdpiak-13/ 2022-09-12T18:57:04.895 DEBUG:boto:Data: 2022-09-12T18:57:04.896 DEBUG:boto:Headers: {} 2022-09-12T18:57:04.897 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:57:04.898 DEBUG:boto:Port: 8001 2022-09-12T18:57:04.899 DEBUG:boto:Params: {} 2022-09-12T18:57:04.900 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T18:57:04.902 DEBUG:boto:Token: None 2022-09-12T18:57:04.902 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:57:04 GMT /jdpiak-13/ 2022-09-12T18:57:04.903 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4AJo7ccPqFkcfXSO/dBzAMU+0os= 2022-09-12T18:57:04.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4AJo7ccPqFkcfXSO/dBzAMU+0os=', 'Content-Length': '0'} 2022-09-12T18:57:04.908 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1c40ec8963f9293-00631f8100-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:57:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:04.909 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:57:04.909 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:57:04.910 DEBUG:boto:Method: HEAD 2022-09-12T18:57:04.910 DEBUG:boto:Path: /jdpiak-13/ 2022-09-12T18:57:04.911 DEBUG:boto:Data: 2022-09-12T18:57:04.911 DEBUG:boto:Headers: {} 2022-09-12T18:57:04.912 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:57:04.912 DEBUG:boto:Port: 8000 2022-09-12T18:57:04.913 DEBUG:boto:Params: {} 2022-09-12T18:57:04.913 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T18:57:04.914 DEBUG:boto:Token: None 2022-09-12T18:57:04.915 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:57:04 GMT /jdpiak-13/ 2022-09-12T18:57:04.915 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4AJo7ccPqFkcfXSO/dBzAMU+0os= 2022-09-12T18:57:04.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4AJo7ccPqFkcfXSO/dBzAMU+0os=', 'Content-Length': '0'} 2022-09-12T18:57:04.920 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000510f0a1c208699d5-00631f8100-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:57:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:04.920 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T18:57:04.921 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:57:04.921 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:57:04.922 DEBUG:boto:path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:57:04.923 DEBUG:boto:auth_path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:57:04.923 DEBUG:boto:Method: GET 2022-09-12T18:57:04.924 DEBUG:boto:Path: /jdpiak-13/?versions&max-keys=999 2022-09-12T18:57:04.924 DEBUG:boto:Data: 2022-09-12T18:57:04.925 DEBUG:boto:Headers: {} 2022-09-12T18:57:04.925 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:57:04.926 DEBUG:boto:Port: 8001 2022-09-12T18:57:04.926 DEBUG:boto:Params: {} 2022-09-12T18:57:04.927 DEBUG:boto:Token: None 2022-09-12T18:57:04.927 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:57:04 GMT /jdpiak-13/?versions 2022-09-12T18:57:04.928 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jborfxk8C1q5vsnS8Puu+FblmsU= 2022-09-12T18:57:04.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jborfxk8C1q5vsnS8Puu+FblmsU=', 'Content-Length': '0'} 2022-09-12T18:57:04.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d335617f5e5ed02-00631f8100-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:57:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:04.932 DEBUG:boto:b'jdpiak-13999false' 2022-09-12T18:57:04.933 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T18:57:04.933 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:57:04.934 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:57:04.934 DEBUG:boto:path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:57:04.935 DEBUG:boto:auth_path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:57:04.935 DEBUG:boto:Method: GET 2022-09-12T18:57:04.936 DEBUG:boto:Path: /jdpiak-13/?versions&max-keys=999 2022-09-12T18:57:04.936 DEBUG:boto:Data: 2022-09-12T18:57:04.937 DEBUG:boto:Headers: {} 2022-09-12T18:57:04.937 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:57:04.938 DEBUG:boto:Port: 8000 2022-09-12T18:57:04.939 DEBUG:boto:Params: {} 2022-09-12T18:57:04.939 DEBUG:boto:Token: None 2022-09-12T18:57:04.940 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:57:04 GMT /jdpiak-13/?versions 2022-09-12T18:57:04.941 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jborfxk8C1q5vsnS8Puu+FblmsU= 2022-09-12T18:57:04.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jborfxk8C1q5vsnS8Puu+FblmsU=', 'Content-Length': '0'} 2022-09-12T18:57:04.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3dd87923a6b9b79-00631f8100-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:57:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:04.945 DEBUG:boto:b'jdpiak-13999false' 2022-09-12T18:57:04.945 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:57:04.946 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:57:04.946 DEBUG:boto:path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:57:04.947 DEBUG:boto:auth_path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:57:04.948 DEBUG:boto:Method: GET 2022-09-12T18:57:04.948 DEBUG:boto:Path: /jdpiak-13/?versions&max-keys=999 2022-09-12T18:57:04.949 DEBUG:boto:Data: 2022-09-12T18:57:04.949 DEBUG:boto:Headers: {} 2022-09-12T18:57:04.950 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:57:04.950 DEBUG:boto:Port: 8001 2022-09-12T18:57:04.951 DEBUG:boto:Params: {} 2022-09-12T18:57:04.951 DEBUG:boto:Token: None 2022-09-12T18:57:04.952 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:57:04 GMT /jdpiak-13/?versions 2022-09-12T18:57:04.953 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jborfxk8C1q5vsnS8Puu+FblmsU= 2022-09-12T18:57:04.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jborfxk8C1q5vsnS8Puu+FblmsU=', 'Content-Length': '0'} 2022-09-12T18:57:04.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047108058fc73f437-00631f8100-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:57:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:04.956 DEBUG:boto:b'jdpiak-13999false' 2022-09-12T18:57:04.957 DEBUG:boto:path=/jdpiak-13/ 2022-09-12T18:57:04.958 DEBUG:boto:auth_path=/jdpiak-13/ 2022-09-12T18:57:04.958 DEBUG:boto:path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:57:04.959 DEBUG:boto:auth_path=/jdpiak-13/?versions&max-keys=999 2022-09-12T18:57:04.959 DEBUG:boto:Method: GET 2022-09-12T18:57:04.960 DEBUG:boto:Path: /jdpiak-13/?versions&max-keys=999 2022-09-12T18:57:04.960 DEBUG:boto:Data: 2022-09-12T18:57:04.961 DEBUG:boto:Headers: {} 2022-09-12T18:57:04.961 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:57:04.962 DEBUG:boto:Port: 8000 2022-09-12T18:57:04.962 DEBUG:boto:Params: {} 2022-09-12T18:57:04.963 DEBUG:boto:Token: None 2022-09-12T18:57:04.963 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:57:04 GMT /jdpiak-13/?versions 2022-09-12T18:57:04.964 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jborfxk8C1q5vsnS8Puu+FblmsU= 2022-09-12T18:57:04.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jborfxk8C1q5vsnS8Puu+FblmsU=', 'Content-Length': '0'} 2022-09-12T18:57:04.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c21edff5e98ab511-00631f8100-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:57:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:04.968 DEBUG:boto:b'jdpiak-13999false' 2022-09-12T18:57:04.969 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-13 zones={test-zone3, test-zone1} 2022-09-12T18:57:04.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:57:04.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:57:04.971 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T18:57:05.036 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:57:05.036 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:57:05.093 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:57:05.089+0000 7f9d5dc33600 1 bucket sync caught up with source: 2022-09-12T18:57:05.093 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.18.5, , , , , , , , , ] 2022-09-12T18:57:05.094 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000002.18.5, , , , , , , , , ] 2022-09-12T18:57:05.094 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:57:05.089+0000 7f9d5dc33600 0 bucket checkpoint complete 2022-09-12T18:57:05.113 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:57:05.114 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:57:05.115 DEBUG:boto:Method: HEAD 2022-09-12T18:57:05.116 DEBUG:boto:Path: /jdpiak-14/ 2022-09-12T18:57:05.117 DEBUG:boto:Data: 2022-09-12T18:57:05.118 DEBUG:boto:Headers: {} 2022-09-12T18:57:05.119 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:57:05.120 DEBUG:boto:Port: 8000 2022-09-12T18:57:05.121 DEBUG:boto:Params: {} 2022-09-12T18:57:05.122 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T18:57:05.123 DEBUG:boto:Token: None 2022-09-12T18:57:05.124 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:57:05 GMT /jdpiak-14/ 2022-09-12T18:57:05.125 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:M3EzMuU+l0BiA+Fh0XzWxzm/JY8= 2022-09-12T18:57:05.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:M3EzMuU+l0BiA+Fh0XzWxzm/JY8=', 'Content-Length': '0'} 2022-09-12T18:57:05.133 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6fcc0b9bbb2b33f-00631f8101-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:57:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:05.136 DEBUG:boto:path=/jdpiak-14/myobj 2022-09-12T18:57:05.137 DEBUG:boto:auth_path=/jdpiak-14/myobj 2022-09-12T18:57:05.138 DEBUG:boto:Method: HEAD 2022-09-12T18:57:05.139 DEBUG:boto:Path: /jdpiak-14/myobj 2022-09-12T18:57:05.140 DEBUG:boto:Data: 2022-09-12T18:57:05.141 DEBUG:boto:Headers: {} 2022-09-12T18:57:05.142 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:57:05.143 DEBUG:boto:Port: 8000 2022-09-12T18:57:05.144 DEBUG:boto:Params: {} 2022-09-12T18:57:05.145 DEBUG:boto:Token: None 2022-09-12T18:57:05.146 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:57:05 GMT /jdpiak-14/myobj 2022-09-12T18:57:05.147 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x09F1ggO3ZKAP81VbzXn+Ckr2xI= 2022-09-12T18:57:05.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x09F1ggO3ZKAP81VbzXn+Ckr2xI=', 'Content-Length': '0'} 2022-09-12T18:57:05.151 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8678caccaef01f1-00631f8101-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:57:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:05.152 DEBUG:boto:path=/jdpiak-14/myobj 2022-09-12T18:57:05.153 DEBUG:boto:auth_path=/jdpiak-14/myobj 2022-09-12T18:57:05.155 DEBUG:boto:Method: DELETE 2022-09-12T18:57:05.156 DEBUG:boto:Path: /jdpiak-14/myobj 2022-09-12T18:57:05.157 DEBUG:boto:Data: 2022-09-12T18:57:05.158 DEBUG:boto:Headers: {} 2022-09-12T18:57:05.158 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:57:05.159 DEBUG:boto:Port: 8000 2022-09-12T18:57:05.160 DEBUG:boto:Params: {} 2022-09-12T18:57:05.161 DEBUG:boto:Token: None 2022-09-12T18:57:05.162 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 18:57:05 GMT /jdpiak-14/myobj 2022-09-12T18:57:05.163 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tovBjmbDtdLBgUvhZhhNHhDWpX8= 2022-09-12T18:57:05.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tovBjmbDtdLBgUvhZhhNHhDWpX8=', 'Content-Length': '0'} 2022-09-12T18:57:05.175 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f9d85aee6f1626da-00631f8101-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 18:57:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:05.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:57:05.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:57:05.179 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T18:57:05.246 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:57:05.247 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:57:05.302 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:57:05.299+0000 7f2290859600 1 waiting for incremental sync to catch up: 2022-09-12T18:57:05.303 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.15.5, , , , , , , , , ] 2022-09-12T18:57:05.303 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.19.5, , , , , , , , , ] 2022-09-12T18:57:10.304 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:57:10.300+0000 7f2290859600 1 waiting for incremental sync to catch up: 2022-09-12T18:57:10.304 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.15.5, , , , , , , , , ] 2022-09-12T18:57:10.305 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.19.5, , , , , , , , , ] 2022-09-12T18:57:15.305 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:57:15.302+0000 7f2290859600 1 waiting for incremental sync to catch up: 2022-09-12T18:57:15.306 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.15.5, , , , , , , , , ] 2022-09-12T18:57:15.306 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.19.5, , , , , , , , , ] 2022-09-12T18:57:20.307 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:57:20.303+0000 7f2290859600 1 waiting for incremental sync to catch up: 2022-09-12T18:57:20.308 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.15.5, , , , , , , , , ] 2022-09-12T18:57:20.308 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.19.5, , , , , , , , , ] 2022-09-12T18:57:25.308 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:57:25.305+0000 7f2290859600 1 bucket sync caught up with source: 2022-09-12T18:57:25.309 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.19.5, , , , , , , , , ] 2022-09-12T18:57:25.309 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.19.5, , , , , , , , , ] 2022-09-12T18:57:25.309 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:57:25.305+0000 7f2290859600 0 bucket checkpoint complete 2022-09-12T18:57:25.325 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-14 zones={test-zone1, test-zone2} 2022-09-12T18:57:25.326 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:57:25.327 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:57:25.328 DEBUG:boto:Method: HEAD 2022-09-12T18:57:25.329 DEBUG:boto:Path: /jdpiak-14/ 2022-09-12T18:57:25.330 DEBUG:boto:Data: 2022-09-12T18:57:25.331 DEBUG:boto:Headers: {} 2022-09-12T18:57:25.332 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:57:25.333 DEBUG:boto:Port: 8000 2022-09-12T18:57:25.334 DEBUG:boto:Params: {} 2022-09-12T18:57:25.335 DEBUG:boto:Token: None 2022-09-12T18:57:25.336 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:57:25 GMT /jdpiak-14/ 2022-09-12T18:57:25.337 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RS3VOderzPeoPCAECnAxUdaHfGI= 2022-09-12T18:57:25.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RS3VOderzPeoPCAECnAxUdaHfGI=', 'Content-Length': '0'} 2022-09-12T18:57:25.342 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068536a0b5761fb6e-00631f8115-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:57:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:25.343 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:57:25.344 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:57:25.345 DEBUG:boto:Method: HEAD 2022-09-12T18:57:25.346 DEBUG:boto:Path: /jdpiak-14/ 2022-09-12T18:57:25.347 DEBUG:boto:Data: 2022-09-12T18:57:25.348 DEBUG:boto:Headers: {} 2022-09-12T18:57:25.349 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:57:25.350 DEBUG:boto:Port: 8000 2022-09-12T18:57:25.352 DEBUG:boto:Params: {} 2022-09-12T18:57:25.353 DEBUG:boto:Token: None 2022-09-12T18:57:25.354 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:57:25 GMT /jdpiak-14/ 2022-09-12T18:57:25.355 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RS3VOderzPeoPCAECnAxUdaHfGI= 2022-09-12T18:57:25.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RS3VOderzPeoPCAECnAxUdaHfGI=', 'Content-Length': '0'} 2022-09-12T18:57:25.359 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6da29c71e7444fb-00631f8115-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:57:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:25.361 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T18:57:25.362 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:57:25.363 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:57:25.364 DEBUG:boto:path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:57:25.365 DEBUG:boto:auth_path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:57:25.365 DEBUG:boto:Method: GET 2022-09-12T18:57:25.366 DEBUG:boto:Path: /jdpiak-14/?versions&max-keys=999 2022-09-12T18:57:25.367 DEBUG:boto:Data: 2022-09-12T18:57:25.368 DEBUG:boto:Headers: {} 2022-09-12T18:57:25.369 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:57:25.370 DEBUG:boto:Port: 8000 2022-09-12T18:57:25.371 DEBUG:boto:Params: {} 2022-09-12T18:57:25.372 DEBUG:boto:Token: None 2022-09-12T18:57:25.373 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:57:25 GMT /jdpiak-14/?versions 2022-09-12T18:57:25.375 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:16uGZLnjwI7WaYF2qLmt6RGwCLo= 2022-09-12T18:57:25.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:16uGZLnjwI7WaYF2qLmt6RGwCLo=', 'Content-Length': '0'} 2022-09-12T18:57:25.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007533166c053f7fad-00631f8115-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:57:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:25.380 DEBUG:boto:b'jdpiak-14999false' 2022-09-12T18:57:25.382 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T18:57:25.382 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:57:25.383 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:57:25.383 DEBUG:boto:path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:57:25.385 DEBUG:boto:auth_path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:57:25.385 DEBUG:boto:Method: GET 2022-09-12T18:57:25.386 DEBUG:boto:Path: /jdpiak-14/?versions&max-keys=999 2022-09-12T18:57:25.387 DEBUG:boto:Data: 2022-09-12T18:57:25.387 DEBUG:boto:Headers: {} 2022-09-12T18:57:25.388 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:57:25.389 DEBUG:boto:Port: 8000 2022-09-12T18:57:25.389 DEBUG:boto:Params: {} 2022-09-12T18:57:25.390 DEBUG:boto:Token: None 2022-09-12T18:57:25.391 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:57:25 GMT /jdpiak-14/?versions 2022-09-12T18:57:25.391 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:16uGZLnjwI7WaYF2qLmt6RGwCLo= 2022-09-12T18:57:25.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:16uGZLnjwI7WaYF2qLmt6RGwCLo=', 'Content-Length': '0'} 2022-09-12T18:57:25.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e62de03002ff1c1d-00631f8115-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:57:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:25.442 DEBUG:boto:b'jdpiak-14999false' 2022-09-12T18:57:25.445 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:57:25.446 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:57:25.446 DEBUG:boto:path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:57:25.447 DEBUG:boto:auth_path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:57:25.448 DEBUG:boto:Method: GET 2022-09-12T18:57:25.449 DEBUG:boto:Path: /jdpiak-14/?versions&max-keys=999 2022-09-12T18:57:25.449 DEBUG:boto:Data: 2022-09-12T18:57:25.450 DEBUG:boto:Headers: {} 2022-09-12T18:57:25.451 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:57:25.453 DEBUG:boto:Port: 8000 2022-09-12T18:57:25.454 DEBUG:boto:Params: {} 2022-09-12T18:57:25.455 DEBUG:boto:Token: None 2022-09-12T18:57:25.456 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:57:25 GMT /jdpiak-14/?versions 2022-09-12T18:57:25.457 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:16uGZLnjwI7WaYF2qLmt6RGwCLo= 2022-09-12T18:57:25.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:16uGZLnjwI7WaYF2qLmt6RGwCLo=', 'Content-Length': '0'} 2022-09-12T18:57:25.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009eb14f1a1abddb7b-00631f8115-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:57:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:25.466 DEBUG:boto:b'jdpiak-14999false' 2022-09-12T18:57:25.467 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:57:25.467 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:57:25.468 DEBUG:boto:path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:57:25.468 DEBUG:boto:auth_path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:57:25.469 DEBUG:boto:Method: GET 2022-09-12T18:57:25.469 DEBUG:boto:Path: /jdpiak-14/?versions&max-keys=999 2022-09-12T18:57:25.470 DEBUG:boto:Data: 2022-09-12T18:57:25.470 DEBUG:boto:Headers: {} 2022-09-12T18:57:25.471 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:57:25.471 DEBUG:boto:Port: 8000 2022-09-12T18:57:25.472 DEBUG:boto:Params: {} 2022-09-12T18:57:25.473 DEBUG:boto:Token: None 2022-09-12T18:57:25.473 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:57:25 GMT /jdpiak-14/?versions 2022-09-12T18:57:25.474 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:16uGZLnjwI7WaYF2qLmt6RGwCLo= 2022-09-12T18:57:25.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:16uGZLnjwI7WaYF2qLmt6RGwCLo=', 'Content-Length': '0'} 2022-09-12T18:57:25.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000693c56394b91cba9-00631f8115-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:57:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:25.478 DEBUG:boto:b'jdpiak-14999false' 2022-09-12T18:57:25.478 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-14 zones={test-zone1, test-zone2} 2022-09-12T18:57:25.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:57:25.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:57:25.480 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T18:57:25.547 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:57:25.548 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:57:25.593 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:57:25.589+0000 7fb98caa7600 1 bucket sync caught up with source: 2022-09-12T18:57:25.593 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.19.5, , , , , , , , , ] 2022-09-12T18:57:25.594 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.19.5, , , , , , , , , ] 2022-09-12T18:57:25.594 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:57:25.589+0000 7fb98caa7600 0 bucket checkpoint complete 2022-09-12T18:57:25.611 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-14 zones={test-zone3, test-zone2} 2022-09-12T18:57:25.613 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:57:25.614 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:57:25.615 DEBUG:boto:Method: HEAD 2022-09-12T18:57:25.615 DEBUG:boto:Path: /jdpiak-14/ 2022-09-12T18:57:25.616 DEBUG:boto:Data: 2022-09-12T18:57:25.617 DEBUG:boto:Headers: {} 2022-09-12T18:57:25.618 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:57:25.619 DEBUG:boto:Port: 8001 2022-09-12T18:57:25.620 DEBUG:boto:Params: {} 2022-09-12T18:57:25.621 DEBUG:boto:Token: None 2022-09-12T18:57:25.622 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:57:25 GMT /jdpiak-14/ 2022-09-12T18:57:25.623 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RS3VOderzPeoPCAECnAxUdaHfGI= 2022-09-12T18:57:25.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RS3VOderzPeoPCAECnAxUdaHfGI=', 'Content-Length': '0'} 2022-09-12T18:57:25.628 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4b2809eb36da390-00631f8115-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:57:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:25.629 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:57:25.629 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:57:25.630 DEBUG:boto:Method: HEAD 2022-09-12T18:57:25.631 DEBUG:boto:Path: /jdpiak-14/ 2022-09-12T18:57:25.631 DEBUG:boto:Data: 2022-09-12T18:57:25.632 DEBUG:boto:Headers: {} 2022-09-12T18:57:25.632 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:57:25.633 DEBUG:boto:Port: 8000 2022-09-12T18:57:25.633 DEBUG:boto:Params: {} 2022-09-12T18:57:25.634 DEBUG:boto:Token: None 2022-09-12T18:57:25.636 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:57:25 GMT /jdpiak-14/ 2022-09-12T18:57:25.636 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RS3VOderzPeoPCAECnAxUdaHfGI= 2022-09-12T18:57:25.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RS3VOderzPeoPCAECnAxUdaHfGI=', 'Content-Length': '0'} 2022-09-12T18:57:25.640 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab7a5c7f2eb676ca-00631f8115-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:57:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:25.640 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T18:57:25.641 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:57:25.641 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:57:25.642 DEBUG:boto:path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:57:25.642 DEBUG:boto:auth_path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:57:25.643 DEBUG:boto:Method: GET 2022-09-12T18:57:25.643 DEBUG:boto:Path: /jdpiak-14/?versions&max-keys=999 2022-09-12T18:57:25.644 DEBUG:boto:Data: 2022-09-12T18:57:25.644 DEBUG:boto:Headers: {} 2022-09-12T18:57:25.645 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:57:25.648 DEBUG:boto:Port: 8001 2022-09-12T18:57:25.649 DEBUG:boto:Params: {} 2022-09-12T18:57:25.650 DEBUG:boto:Token: None 2022-09-12T18:57:25.651 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:57:25 GMT /jdpiak-14/?versions 2022-09-12T18:57:25.652 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:16uGZLnjwI7WaYF2qLmt6RGwCLo= 2022-09-12T18:57:25.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:16uGZLnjwI7WaYF2qLmt6RGwCLo=', 'Content-Length': '0'} 2022-09-12T18:57:25.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051b878bb051bf179-00631f8115-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:57:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:25.658 DEBUG:boto:b'jdpiak-14999false' 2022-09-12T18:57:25.659 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T18:57:25.660 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:57:25.660 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:57:25.661 DEBUG:boto:path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:57:25.661 DEBUG:boto:auth_path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:57:25.662 DEBUG:boto:Method: GET 2022-09-12T18:57:25.663 DEBUG:boto:Path: /jdpiak-14/?versions&max-keys=999 2022-09-12T18:57:25.664 DEBUG:boto:Data: 2022-09-12T18:57:25.664 DEBUG:boto:Headers: {} 2022-09-12T18:57:25.665 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:57:25.665 DEBUG:boto:Port: 8000 2022-09-12T18:57:25.666 DEBUG:boto:Params: {} 2022-09-12T18:57:25.666 DEBUG:boto:Token: None 2022-09-12T18:57:25.667 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:57:25 GMT /jdpiak-14/?versions 2022-09-12T18:57:25.668 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:16uGZLnjwI7WaYF2qLmt6RGwCLo= 2022-09-12T18:57:25.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:16uGZLnjwI7WaYF2qLmt6RGwCLo=', 'Content-Length': '0'} 2022-09-12T18:57:25.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be95e2c64a61f18f-00631f8115-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:57:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:25.673 DEBUG:boto:b'jdpiak-14999false' 2022-09-12T18:57:25.674 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:57:25.674 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:57:25.675 DEBUG:boto:path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:57:25.675 DEBUG:boto:auth_path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:57:25.676 DEBUG:boto:Method: GET 2022-09-12T18:57:25.676 DEBUG:boto:Path: /jdpiak-14/?versions&max-keys=999 2022-09-12T18:57:25.677 DEBUG:boto:Data: 2022-09-12T18:57:25.678 DEBUG:boto:Headers: {} 2022-09-12T18:57:25.678 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:57:25.679 DEBUG:boto:Port: 8001 2022-09-12T18:57:25.679 DEBUG:boto:Params: {} 2022-09-12T18:57:25.680 DEBUG:boto:Token: None 2022-09-12T18:57:25.681 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:57:25 GMT /jdpiak-14/?versions 2022-09-12T18:57:25.682 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:16uGZLnjwI7WaYF2qLmt6RGwCLo= 2022-09-12T18:57:25.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:16uGZLnjwI7WaYF2qLmt6RGwCLo=', 'Content-Length': '0'} 2022-09-12T18:57:25.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfeaa64e4cf1b45c-00631f8115-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:57:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:25.686 DEBUG:boto:b'jdpiak-14999false' 2022-09-12T18:57:25.687 DEBUG:boto:path=/jdpiak-14/ 2022-09-12T18:57:25.687 DEBUG:boto:auth_path=/jdpiak-14/ 2022-09-12T18:57:25.688 DEBUG:boto:path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:57:25.688 DEBUG:boto:auth_path=/jdpiak-14/?versions&max-keys=999 2022-09-12T18:57:25.689 DEBUG:boto:Method: GET 2022-09-12T18:57:25.689 DEBUG:boto:Path: /jdpiak-14/?versions&max-keys=999 2022-09-12T18:57:25.690 DEBUG:boto:Data: 2022-09-12T18:57:25.690 DEBUG:boto:Headers: {} 2022-09-12T18:57:25.691 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:57:25.691 DEBUG:boto:Port: 8000 2022-09-12T18:57:25.692 DEBUG:boto:Params: {} 2022-09-12T18:57:25.693 DEBUG:boto:Token: None 2022-09-12T18:57:25.693 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:57:25 GMT /jdpiak-14/?versions 2022-09-12T18:57:25.694 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:16uGZLnjwI7WaYF2qLmt6RGwCLo= 2022-09-12T18:57:25.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:16uGZLnjwI7WaYF2qLmt6RGwCLo=', 'Content-Length': '0'} 2022-09-12T18:57:25.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc6031604d15c7df-00631f8115-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:57:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:25.698 DEBUG:boto:b'jdpiak-14999false' 2022-09-12T18:57:25.699 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-14 zones={test-zone3, test-zone2} 2022-09-12T18:57:25.699 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:57:25.700 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:57:25.700 DEBUG:boto:Method: HEAD 2022-09-12T18:57:25.701 DEBUG:boto:Path: /jdpiak-15/ 2022-09-12T18:57:25.701 DEBUG:boto:Data: 2022-09-12T18:57:25.702 DEBUG:boto:Headers: {} 2022-09-12T18:57:25.703 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:57:25.703 DEBUG:boto:Port: 8001 2022-09-12T18:57:25.704 DEBUG:boto:Params: {} 2022-09-12T18:57:25.704 DEBUG:boto:Token: None 2022-09-12T18:57:25.705 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:57:25 GMT /jdpiak-15/ 2022-09-12T18:57:25.705 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Z5yp7WZnqIRdjLuL6Po+gDOC2yQ= 2022-09-12T18:57:25.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Z5yp7WZnqIRdjLuL6Po+gDOC2yQ=', 'Content-Length': '0'} 2022-09-12T18:57:25.708 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009627bec53e1e9b27-00631f8115-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:57:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:25.709 DEBUG:boto:path=/jdpiak-15/myobj 2022-09-12T18:57:25.710 DEBUG:boto:auth_path=/jdpiak-15/myobj 2022-09-12T18:57:25.711 DEBUG:boto:Method: HEAD 2022-09-12T18:57:25.712 DEBUG:boto:Path: /jdpiak-15/myobj 2022-09-12T18:57:25.713 DEBUG:boto:Data: 2022-09-12T18:57:25.714 DEBUG:boto:Headers: {} 2022-09-12T18:57:25.715 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:57:25.715 DEBUG:boto:Port: 8001 2022-09-12T18:57:25.716 DEBUG:boto:Params: {} 2022-09-12T18:57:25.717 DEBUG:boto:Token: None 2022-09-12T18:57:25.718 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:57:25 GMT /jdpiak-15/myobj 2022-09-12T18:57:25.718 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LPE7NhjuUVJQmBXQ6Tijl3/P0ac= 2022-09-12T18:57:25.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LPE7NhjuUVJQmBXQ6Tijl3/P0ac=', 'Content-Length': '0'} 2022-09-12T18:57:25.721 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:55:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014a02dd1edc00b82-00631f8115-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 18:57:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:25.722 DEBUG:boto:path=/jdpiak-15/myobj 2022-09-12T18:57:25.723 DEBUG:boto:auth_path=/jdpiak-15/myobj 2022-09-12T18:57:25.723 DEBUG:boto:Method: DELETE 2022-09-12T18:57:25.724 DEBUG:boto:Path: /jdpiak-15/myobj 2022-09-12T18:57:25.725 DEBUG:boto:Data: 2022-09-12T18:57:25.726 DEBUG:boto:Headers: {} 2022-09-12T18:57:25.726 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:57:25.727 DEBUG:boto:Port: 8001 2022-09-12T18:57:25.728 DEBUG:boto:Params: {} 2022-09-12T18:57:25.728 DEBUG:boto:Token: None 2022-09-12T18:57:25.729 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 18:57:25 GMT /jdpiak-15/myobj 2022-09-12T18:57:25.730 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4lwMDjrBbVNXB4cYMW88ooQdXS0= 2022-09-12T18:57:25.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4lwMDjrBbVNXB4cYMW88ooQdXS0=', 'Content-Length': '0'} 2022-09-12T18:57:25.739 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000034d1e0382a6831a9-00631f8115-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 18:57:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:25.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:57:25.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:57:25.742 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T18:57:25.803 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:57:25.804 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:57:25.839 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:57:25.836+0000 7ff731295600 1 waiting for incremental sync to catch up: 2022-09-12T18:57:25.840 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.41.5, , , , , , , , , ] 2022-09-12T18:57:25.840 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.43.5, , , , , , , , , ] 2022-09-12T18:57:30.842 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:57:30.838+0000 7ff731295600 1 waiting for incremental sync to catch up: 2022-09-12T18:57:30.843 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.41.5, , , , , , , , , ] 2022-09-12T18:57:30.843 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.43.5, , , , , , , , , ] 2022-09-12T18:57:35.843 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:57:35.839+0000 7ff731295600 1 waiting for incremental sync to catch up: 2022-09-12T18:57:35.844 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.41.5, , , , , , , , , ] 2022-09-12T18:57:35.844 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.43.5, , , , , , , , , ] 2022-09-12T18:57:40.844 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:57:40.841+0000 7ff731295600 1 waiting for incremental sync to catch up: 2022-09-12T18:57:40.844 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.41.5, , , , , , , , , ] 2022-09-12T18:57:40.845 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.43.5, , , , , , , , , ] 2022-09-12T18:57:45.845 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:57:45.842+0000 7ff731295600 1 bucket sync caught up with source: 2022-09-12T18:57:45.845 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.43.5, , , , , , , , , ] 2022-09-12T18:57:45.845 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.43.5, , , , , , , , , ] 2022-09-12T18:57:45.846 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:57:45.842+0000 7ff731295600 0 bucket checkpoint complete 2022-09-12T18:57:45.862 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-15 zones={test-zone1, test-zone3} 2022-09-12T18:57:45.863 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:57:45.864 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:57:45.865 DEBUG:boto:Method: HEAD 2022-09-12T18:57:45.867 DEBUG:boto:Path: /jdpiak-15/ 2022-09-12T18:57:45.867 DEBUG:boto:Data: 2022-09-12T18:57:45.868 DEBUG:boto:Headers: {} 2022-09-12T18:57:45.869 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:57:45.869 DEBUG:boto:Port: 8000 2022-09-12T18:57:45.870 DEBUG:boto:Params: {} 2022-09-12T18:57:45.871 DEBUG:boto:Token: None 2022-09-12T18:57:45.871 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:57:45 GMT /jdpiak-15/ 2022-09-12T18:57:45.872 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:64NmaG/BRiaeXbjDKs4JqL8Y1LE= 2022-09-12T18:57:45.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:64NmaG/BRiaeXbjDKs4JqL8Y1LE=', 'Content-Length': '0'} 2022-09-12T18:57:45.875 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe03fc507747f775-00631f8129-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:57:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:45.876 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:57:45.877 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:57:45.877 DEBUG:boto:Method: HEAD 2022-09-12T18:57:45.878 DEBUG:boto:Path: /jdpiak-15/ 2022-09-12T18:57:45.878 DEBUG:boto:Data: 2022-09-12T18:57:45.879 DEBUG:boto:Headers: {} 2022-09-12T18:57:45.879 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:57:45.880 DEBUG:boto:Port: 8001 2022-09-12T18:57:45.880 DEBUG:boto:Params: {} 2022-09-12T18:57:45.881 DEBUG:boto:Token: None 2022-09-12T18:57:45.881 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:57:45 GMT /jdpiak-15/ 2022-09-12T18:57:45.882 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:64NmaG/BRiaeXbjDKs4JqL8Y1LE= 2022-09-12T18:57:45.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:64NmaG/BRiaeXbjDKs4JqL8Y1LE=', 'Content-Length': '0'} 2022-09-12T18:57:45.885 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054dfd70513bd1419-00631f8129-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:57:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:45.886 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T18:57:45.887 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:57:45.888 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:57:45.888 DEBUG:boto:path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:57:45.889 DEBUG:boto:auth_path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:57:45.890 DEBUG:boto:Method: GET 2022-09-12T18:57:45.890 DEBUG:boto:Path: /jdpiak-15/?versions&max-keys=999 2022-09-12T18:57:45.891 DEBUG:boto:Data: 2022-09-12T18:57:45.891 DEBUG:boto:Headers: {} 2022-09-12T18:57:45.892 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:57:45.892 DEBUG:boto:Port: 8000 2022-09-12T18:57:45.893 DEBUG:boto:Params: {} 2022-09-12T18:57:45.894 DEBUG:boto:Token: None 2022-09-12T18:57:45.895 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:57:45 GMT /jdpiak-15/?versions 2022-09-12T18:57:45.896 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PIA+NcCir7VullrWIP6NdgpW3vE= 2022-09-12T18:57:45.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PIA+NcCir7VullrWIP6NdgpW3vE=', 'Content-Length': '0'} 2022-09-12T18:57:45.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005a033421a89de7f-00631f8129-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:57:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:45.902 DEBUG:boto:b'jdpiak-15999false' 2022-09-12T18:57:45.903 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T18:57:45.904 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:57:45.904 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:57:45.905 DEBUG:boto:path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:57:45.906 DEBUG:boto:auth_path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:57:45.906 DEBUG:boto:Method: GET 2022-09-12T18:57:45.907 DEBUG:boto:Path: /jdpiak-15/?versions&max-keys=999 2022-09-12T18:57:45.908 DEBUG:boto:Data: 2022-09-12T18:57:45.909 DEBUG:boto:Headers: {} 2022-09-12T18:57:45.909 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:57:45.910 DEBUG:boto:Port: 8001 2022-09-12T18:57:45.911 DEBUG:boto:Params: {} 2022-09-12T18:57:45.911 DEBUG:boto:Token: None 2022-09-12T18:57:45.912 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:57:45 GMT /jdpiak-15/?versions 2022-09-12T18:57:45.913 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PIA+NcCir7VullrWIP6NdgpW3vE= 2022-09-12T18:57:45.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PIA+NcCir7VullrWIP6NdgpW3vE=', 'Content-Length': '0'} 2022-09-12T18:57:45.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d62644ebfc581c5e-00631f8129-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:57:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:45.917 DEBUG:boto:b'jdpiak-15999false' 2022-09-12T18:57:45.918 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:57:45.918 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:57:45.919 DEBUG:boto:path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:57:45.920 DEBUG:boto:auth_path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:57:45.920 DEBUG:boto:Method: GET 2022-09-12T18:57:45.921 DEBUG:boto:Path: /jdpiak-15/?versions&max-keys=999 2022-09-12T18:57:45.921 DEBUG:boto:Data: 2022-09-12T18:57:45.922 DEBUG:boto:Headers: {} 2022-09-12T18:57:45.922 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:57:45.923 DEBUG:boto:Port: 8000 2022-09-12T18:57:45.924 DEBUG:boto:Params: {} 2022-09-12T18:57:45.924 DEBUG:boto:Token: None 2022-09-12T18:57:45.925 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:57:45 GMT /jdpiak-15/?versions 2022-09-12T18:57:45.926 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PIA+NcCir7VullrWIP6NdgpW3vE= 2022-09-12T18:57:45.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PIA+NcCir7VullrWIP6NdgpW3vE=', 'Content-Length': '0'} 2022-09-12T18:57:45.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000121750d32919aa32-00631f8129-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:57:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:45.931 DEBUG:boto:b'jdpiak-15999false' 2022-09-12T18:57:45.932 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:57:45.933 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:57:45.934 DEBUG:boto:path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:57:45.935 DEBUG:boto:auth_path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:57:45.936 DEBUG:boto:Method: GET 2022-09-12T18:57:45.937 DEBUG:boto:Path: /jdpiak-15/?versions&max-keys=999 2022-09-12T18:57:45.938 DEBUG:boto:Data: 2022-09-12T18:57:45.939 DEBUG:boto:Headers: {} 2022-09-12T18:57:45.940 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:57:45.941 DEBUG:boto:Port: 8001 2022-09-12T18:57:45.942 DEBUG:boto:Params: {} 2022-09-12T18:57:45.943 DEBUG:boto:Token: None 2022-09-12T18:57:45.944 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:57:45 GMT /jdpiak-15/?versions 2022-09-12T18:57:45.945 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PIA+NcCir7VullrWIP6NdgpW3vE= 2022-09-12T18:57:45.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PIA+NcCir7VullrWIP6NdgpW3vE=', 'Content-Length': '0'} 2022-09-12T18:57:45.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008735bb9d148be89d-00631f8129-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:57:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:45.949 DEBUG:boto:b'jdpiak-15999false' 2022-09-12T18:57:45.950 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-15 zones={test-zone1, test-zone3} 2022-09-12T18:57:45.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:57:45.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:57:45.952 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T18:57:46.015 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:57:46.016 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:57:46.057 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:57:46.054+0000 7fef60cb7600 1 bucket sync caught up with source: 2022-09-12T18:57:46.058 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.43.5, , , , , , , , , ] 2022-09-12T18:57:46.058 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000002.43.5, , , , , , , , , ] 2022-09-12T18:57:46.058 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:57:46.054+0000 7fef60cb7600 0 bucket checkpoint complete 2022-09-12T18:57:46.072 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-15 zones={test-zone2, test-zone3} 2022-09-12T18:57:46.073 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:57:46.073 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:57:46.074 DEBUG:boto:Method: HEAD 2022-09-12T18:57:46.074 DEBUG:boto:Path: /jdpiak-15/ 2022-09-12T18:57:46.075 DEBUG:boto:Data: 2022-09-12T18:57:46.075 DEBUG:boto:Headers: {} 2022-09-12T18:57:46.076 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:57:46.076 DEBUG:boto:Port: 8000 2022-09-12T18:57:46.077 DEBUG:boto:Params: {} 2022-09-12T18:57:46.078 DEBUG:boto:Token: None 2022-09-12T18:57:46.079 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:57:46 GMT /jdpiak-15/ 2022-09-12T18:57:46.079 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+IeH1+rqKVzN036+QwlMuLOlHJk= 2022-09-12T18:57:46.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+IeH1+rqKVzN036+QwlMuLOlHJk=', 'Content-Length': '0'} 2022-09-12T18:57:46.083 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008dd5f5c77ca07f32-00631f812a-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:57:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:46.084 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:57:46.085 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:57:46.085 DEBUG:boto:Method: HEAD 2022-09-12T18:57:46.086 DEBUG:boto:Path: /jdpiak-15/ 2022-09-12T18:57:46.087 DEBUG:boto:Data: 2022-09-12T18:57:46.087 DEBUG:boto:Headers: {} 2022-09-12T18:57:46.088 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:57:46.089 DEBUG:boto:Port: 8001 2022-09-12T18:57:46.089 DEBUG:boto:Params: {} 2022-09-12T18:57:46.090 DEBUG:boto:Token: None 2022-09-12T18:57:46.091 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:57:46 GMT /jdpiak-15/ 2022-09-12T18:57:46.092 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+IeH1+rqKVzN036+QwlMuLOlHJk= 2022-09-12T18:57:46.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+IeH1+rqKVzN036+QwlMuLOlHJk=', 'Content-Length': '0'} 2022-09-12T18:57:46.095 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004712c3bea5986010-00631f812a-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:57:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:46.096 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T18:57:46.096 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:57:46.097 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:57:46.098 DEBUG:boto:path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:57:46.098 DEBUG:boto:auth_path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:57:46.099 DEBUG:boto:Method: GET 2022-09-12T18:57:46.099 DEBUG:boto:Path: /jdpiak-15/?versions&max-keys=999 2022-09-12T18:57:46.100 DEBUG:boto:Data: 2022-09-12T18:57:46.101 DEBUG:boto:Headers: {} 2022-09-12T18:57:46.101 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:57:46.102 DEBUG:boto:Port: 8000 2022-09-12T18:57:46.102 DEBUG:boto:Params: {} 2022-09-12T18:57:46.103 DEBUG:boto:Token: None 2022-09-12T18:57:46.104 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:57:46 GMT /jdpiak-15/?versions 2022-09-12T18:57:46.104 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nk/elaBrOUpRtk5leoVa9Wbn72k= 2022-09-12T18:57:46.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nk/elaBrOUpRtk5leoVa9Wbn72k=', 'Content-Length': '0'} 2022-09-12T18:57:46.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002935483e2d5f497c-00631f812a-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:57:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:46.108 DEBUG:boto:b'jdpiak-15999false' 2022-09-12T18:57:46.109 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T18:57:46.110 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:57:46.110 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:57:46.111 DEBUG:boto:path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:57:46.112 DEBUG:boto:auth_path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:57:46.112 DEBUG:boto:Method: GET 2022-09-12T18:57:46.113 DEBUG:boto:Path: /jdpiak-15/?versions&max-keys=999 2022-09-12T18:57:46.114 DEBUG:boto:Data: 2022-09-12T18:57:46.115 DEBUG:boto:Headers: {} 2022-09-12T18:57:46.116 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:57:46.117 DEBUG:boto:Port: 8001 2022-09-12T18:57:46.118 DEBUG:boto:Params: {} 2022-09-12T18:57:46.119 DEBUG:boto:Token: None 2022-09-12T18:57:46.120 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:57:46 GMT /jdpiak-15/?versions 2022-09-12T18:57:46.120 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nk/elaBrOUpRtk5leoVa9Wbn72k= 2022-09-12T18:57:46.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nk/elaBrOUpRtk5leoVa9Wbn72k=', 'Content-Length': '0'} 2022-09-12T18:57:46.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011613d91c89dd5ae-00631f812a-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:57:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:46.125 DEBUG:boto:b'jdpiak-15999false' 2022-09-12T18:57:46.126 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:57:46.127 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:57:46.128 DEBUG:boto:path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:57:46.128 DEBUG:boto:auth_path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:57:46.129 DEBUG:boto:Method: GET 2022-09-12T18:57:46.129 DEBUG:boto:Path: /jdpiak-15/?versions&max-keys=999 2022-09-12T18:57:46.130 DEBUG:boto:Data: 2022-09-12T18:57:46.131 DEBUG:boto:Headers: {} 2022-09-12T18:57:46.131 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:57:46.132 DEBUG:boto:Port: 8000 2022-09-12T18:57:46.132 DEBUG:boto:Params: {} 2022-09-12T18:57:46.133 DEBUG:boto:Token: None 2022-09-12T18:57:46.134 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:57:46 GMT /jdpiak-15/?versions 2022-09-12T18:57:46.134 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nk/elaBrOUpRtk5leoVa9Wbn72k= 2022-09-12T18:57:46.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nk/elaBrOUpRtk5leoVa9Wbn72k=', 'Content-Length': '0'} 2022-09-12T18:57:46.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d3cbcab681ca87d-00631f812a-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:57:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:46.138 DEBUG:boto:b'jdpiak-15999false' 2022-09-12T18:57:46.139 DEBUG:boto:path=/jdpiak-15/ 2022-09-12T18:57:46.140 DEBUG:boto:auth_path=/jdpiak-15/ 2022-09-12T18:57:46.141 DEBUG:boto:path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:57:46.142 DEBUG:boto:auth_path=/jdpiak-15/?versions&max-keys=999 2022-09-12T18:57:46.143 DEBUG:boto:Method: GET 2022-09-12T18:57:46.144 DEBUG:boto:Path: /jdpiak-15/?versions&max-keys=999 2022-09-12T18:57:46.145 DEBUG:boto:Data: 2022-09-12T18:57:46.146 DEBUG:boto:Headers: {} 2022-09-12T18:57:46.147 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:57:46.148 DEBUG:boto:Port: 8001 2022-09-12T18:57:46.149 DEBUG:boto:Params: {} 2022-09-12T18:57:46.150 DEBUG:boto:Token: None 2022-09-12T18:57:46.151 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 18:57:46 GMT /jdpiak-15/?versions 2022-09-12T18:57:46.151 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nk/elaBrOUpRtk5leoVa9Wbn72k= 2022-09-12T18:57:46.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nk/elaBrOUpRtk5leoVa9Wbn72k=', 'Content-Length': '0'} 2022-09-12T18:57:46.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed95b7f2ca6c4b14-00631f812a-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 18:57:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:46.156 DEBUG:boto:b'jdpiak-15999false' 2022-09-12T18:57:46.157 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-15 zones={test-zone2, test-zone3} 2022-09-12T18:57:46.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2022-09-12T18:57:46.162 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-16 2022-09-12T18:57:46.163 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T18:57:46.164 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T18:57:46.165 DEBUG:boto:Method: PUT 2022-09-12T18:57:46.165 DEBUG:boto:Path: /jdpiak-16/ 2022-09-12T18:57:46.166 DEBUG:boto:Data: 2022-09-12T18:57:46.167 DEBUG:boto:Headers: {} 2022-09-12T18:57:46.168 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:57:46.169 DEBUG:boto:Port: 8000 2022-09-12T18:57:46.170 DEBUG:boto:Params: {} 2022-09-12T18:57:46.171 DEBUG:boto:Token: None 2022-09-12T18:57:46.172 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:57:46 GMT /jdpiak-16/ 2022-09-12T18:57:46.173 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GlD3w9xMxFzXK7UkNrnOXxkjkgk= 2022-09-12T18:57:46.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GlD3w9xMxFzXK7UkNrnOXxkjkgk=', 'Content-Length': '0'} 2022-09-12T18:57:46.197 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a30b7f39836fe78d-00631f812a-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:57:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:46.199 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jdpiak-17 2022-09-12T18:57:46.199 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T18:57:46.200 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T18:57:46.201 DEBUG:boto:Method: PUT 2022-09-12T18:57:46.202 DEBUG:boto:Path: /jdpiak-17/ 2022-09-12T18:57:46.203 DEBUG:boto:Data: 2022-09-12T18:57:46.204 DEBUG:boto:Headers: {} 2022-09-12T18:57:46.205 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:57:46.206 DEBUG:boto:Port: 8000 2022-09-12T18:57:46.207 DEBUG:boto:Params: {} 2022-09-12T18:57:46.208 DEBUG:boto:Token: None 2022-09-12T18:57:46.209 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:57:46 GMT /jdpiak-17/ 2022-09-12T18:57:46.210 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:I+ggsdaJlOYdoECYn+3rmmSm5aA= 2022-09-12T18:57:46.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:I+ggsdaJlOYdoECYn+3rmmSm5aA=', 'Content-Length': '0'} 2022-09-12T18:57:46.246 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003f66b19f6094baef-00631f812a-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:57:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:46.247 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jdpiak-18 2022-09-12T18:57:46.247 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T18:57:46.248 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T18:57:46.249 DEBUG:boto:Method: PUT 2022-09-12T18:57:46.249 DEBUG:boto:Path: /jdpiak-18/ 2022-09-12T18:57:46.250 DEBUG:boto:Data: 2022-09-12T18:57:46.250 DEBUG:boto:Headers: {} 2022-09-12T18:57:46.251 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:57:46.251 DEBUG:boto:Port: 8001 2022-09-12T18:57:46.252 DEBUG:boto:Params: {} 2022-09-12T18:57:46.253 DEBUG:boto:Token: None 2022-09-12T18:57:46.255 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:57:46 GMT /jdpiak-18/ 2022-09-12T18:57:46.256 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J+6vodP3ejQrNLreGkaMlt3JaRk= 2022-09-12T18:57:46.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J+6vodP3ejQrNLreGkaMlt3JaRk=', 'Content-Length': '0'} 2022-09-12T18:57:46.299 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000890e317de170208c-00631f812a-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:57:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:46.300 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T18:57:46.301 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T18:57:46.301 DEBUG:boto:path=/jdpiak-16/?versioning 2022-09-12T18:57:46.302 DEBUG:boto:auth_path=/jdpiak-16/?versioning 2022-09-12T18:57:46.302 DEBUG:boto:Method: PUT 2022-09-12T18:57:46.303 DEBUG:boto:Path: /jdpiak-16/?versioning 2022-09-12T18:57:46.303 DEBUG:boto:Data: Enabled Disabled 2022-09-12T18:57:46.304 DEBUG:boto:Headers: {} 2022-09-12T18:57:46.304 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:57:46.305 DEBUG:boto:Port: 8000 2022-09-12T18:57:46.306 DEBUG:boto:Params: {} 2022-09-12T18:57:46.306 DEBUG:boto:Token: None 2022-09-12T18:57:46.307 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:57:46 GMT /jdpiak-16/?versioning 2022-09-12T18:57:46.307 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:I0EvmOttCXUgXFR0Hua0Cxf85C8= 2022-09-12T18:57:46.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:I0EvmOttCXUgXFR0Hua0Cxf85C8=', 'Content-Length': '228'} 2022-09-12T18:57:46.313 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bce6a87022e110f9-00631f812a-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:57:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:46.314 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T18:57:46.315 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T18:57:46.315 DEBUG:boto:path=/jdpiak-17/?versioning 2022-09-12T18:57:46.316 DEBUG:boto:auth_path=/jdpiak-17/?versioning 2022-09-12T18:57:46.316 DEBUG:boto:Method: PUT 2022-09-12T18:57:46.317 DEBUG:boto:Path: /jdpiak-17/?versioning 2022-09-12T18:57:46.317 DEBUG:boto:Data: Enabled Disabled 2022-09-12T18:57:46.318 DEBUG:boto:Headers: {} 2022-09-12T18:57:46.318 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:57:46.319 DEBUG:boto:Port: 8000 2022-09-12T18:57:46.319 DEBUG:boto:Params: {} 2022-09-12T18:57:46.320 DEBUG:boto:Token: None 2022-09-12T18:57:46.320 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:57:46 GMT /jdpiak-17/?versioning 2022-09-12T18:57:46.321 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BzaYbf9WBM8/FVhvDD5rFzChSwc= 2022-09-12T18:57:46.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BzaYbf9WBM8/FVhvDD5rFzChSwc=', 'Content-Length': '228'} 2022-09-12T18:57:46.334 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e97ecbff610e77ff-00631f812a-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:57:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:46.335 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T18:57:46.335 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T18:57:46.336 DEBUG:boto:path=/jdpiak-18/?versioning 2022-09-12T18:57:46.336 DEBUG:boto:auth_path=/jdpiak-18/?versioning 2022-09-12T18:57:46.337 DEBUG:boto:Method: PUT 2022-09-12T18:57:46.337 DEBUG:boto:Path: /jdpiak-18/?versioning 2022-09-12T18:57:46.338 DEBUG:boto:Data: Enabled Disabled 2022-09-12T18:57:46.338 DEBUG:boto:Headers: {} 2022-09-12T18:57:46.339 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:57:46.339 DEBUG:boto:Port: 8001 2022-09-12T18:57:46.340 DEBUG:boto:Params: {} 2022-09-12T18:57:46.341 DEBUG:boto:Token: None 2022-09-12T18:57:46.341 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 18:57:46 GMT /jdpiak-18/?versioning 2022-09-12T18:57:46.342 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:THREvPY/GRXRzUC9MZyVUPmaIrk= 2022-09-12T18:57:46.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:THREvPY/GRXRzUC9MZyVUPmaIrk=', 'Content-Length': '228'} 2022-09-12T18:57:46.351 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000020d94a282fc28c8d-00631f812a-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:57:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:46.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:57:46.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:57:46.354 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:57:46.408 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:57:46.409 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:57:46.438 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=c0f7a512-c940-4d1f-8adf-3484630f876b 2022-09-12T18:57:46.450 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T18:57:46.451 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:57:46.451 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:57:46.451 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:47:27.976965Z" 2022-09-12T18:57:46.451 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:57:46.452 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:57:46.452 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:57:46.452 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T18:57:46.452 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:57:46.453 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:57:46.453 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663009066.346375_149.1", 2022-09-12T18:57:46.453 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:57:46.346375Z" 2022-09-12T18:57:46.453 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:57:46.454 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:57:46.454 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T18:57:46.454 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:49:40.904048Z" 2022-09-12T18:57:46.454 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:57:46.455 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T18:57:46.456 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009066.346375_149.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:57:46.457 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T18:57:46.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:57:46.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:57:46.458 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:57:46.522 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:57:46.522 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:57:46.572 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:57:46.568+0000 7fad03fbe600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:57:46.573 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:57:46.569+0000 7fad03fbe600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:57:46.584 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:57:46.585 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:57:46.586 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:57:46.586 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:57:46.587 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:57:46.587 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:57:46.588 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:57:46.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:57:46.589 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:57:46.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:57:46.590 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:57:46.590 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:57:46.591 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:57:46.591 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:57:46.592 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:57:46.592 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:57:46.592 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:57:46.593 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:57:46.593 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:57:46.594 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:57:46.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:57:46.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:57:46.595 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:57:46.596 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:57:46.596 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:57:46.597 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:57:46.597 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:57:46.598 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:57:46.598 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:57:46.599 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:57:46.599 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:57:46.600 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:57:46.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:57:46.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:57:46.601 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:57:46.602 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:57:46.602 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:57:46.602 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663009066.268820_140.1", 2022-09-12T18:57:46.602 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:57:46.603 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:57:46.603 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:57:46.603 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.268820Z", 2022-09-12T18:57:46.603 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:57:46.604 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:57:46.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:57:46.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:57:46.604 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:57:46.604 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:57:46.605 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:57:46.605 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T18:57:46.605 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:57:46.605 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:57:46.606 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:57:46.606 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T18:57:46.606 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:57:46.606 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:57:46.607 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:57:46.607 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:57:46.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:57:46.607 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:57:46.608 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:57:46.608 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:57:46.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:57:46.608 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:57:46Z" 2022-09-12T18:57:46.609 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:57:46.610 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009066.268820_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:57:46.268820Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:57:46Z" } 2022-09-12T18:57:46.611 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009066.268820_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:57:46.268820Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T18:57:46.611 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009066.346375_149.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:57:46.612 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009066.268820_140.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:57:46.612 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663009066.346375_149.1 target=1_1663009066.268820_140.1 2022-09-12T18:57:51.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:57:51.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:57:51.616 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:57:51.674 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:57:51.675 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:57:51.710 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:57:51.706+0000 7f295b6dc600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:57:51.711 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:57:51.708+0000 7f295b6dc600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:57:51.733 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:57:51.733 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:57:51.734 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:57:51.734 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:57:51.734 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:57:51.734 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:57:51.735 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:57:51.735 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:57:51.735 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:57:51.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:57:51.736 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:57:51.736 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:57:51.736 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:57:51.736 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:57:51.737 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:57:51.737 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:57:51.737 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:57:51.737 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:57:51.738 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:57:51.738 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:57:51.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:57:51.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:57:51.739 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:57:51.739 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:57:51.739 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:57:51.739 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:57:51.740 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:57:51.740 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:57:51.740 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:57:51.740 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:57:51.741 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:57:51.741 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:57:51.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:57:51.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:57:51.742 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:57:51.742 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:57:51.742 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:57:51.742 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663009066.346375_149.1", 2022-09-12T18:57:51.743 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:57:51.743 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:57:51.743 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:57:51.743 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.346375Z", 2022-09-12T18:57:51.744 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:57:51.744 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:57:51.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:57:51.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:57:51.745 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:57:51.745 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:57:51.745 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:57:51.745 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T18:57:51.746 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:57:51.746 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:57:51.746 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:57:51.746 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T18:57:51.747 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:57:51.747 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:57:51.747 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:57:51.747 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:57:51.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:57:51.748 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:57:51.748 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:57:51.748 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:57:51.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:57:51.749 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:57:51Z" 2022-09-12T18:57:51.749 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:57:51.750 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009066.346375_149.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:57:46.346375Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:57:51Z" } 2022-09-12T18:57:51.751 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009066.346375_149.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:57:46.346375Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T18:57:51.751 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009066.346375_149.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:57:51.752 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009066.346375_149.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:57:51.752 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T18:57:51.753 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T18:57:51.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:57:51.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:57:51.755 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:57:51.813 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:57:51.815 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:57:51.842 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:57:51.839+0000 7fc5f720a600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:57:51.843 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:57:51.840+0000 7fc5f720a600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:57:51.857 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T18:57:51.858 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T18:57:51.858 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T18:57:51.860 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T18:57:51.860 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T18:57:51.861 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:57:51.861 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:57:51.862 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:57:51.862 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T18:57:51.863 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:57:51.863 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T18:57:51.864 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:57:51.864 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:57:51.865 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:57:51.865 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:57:51.867 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T18:57:51.867 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:57:51.868 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:57:51.868 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:57:51.868 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:57:51.868 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:57:51.869 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:57:51.869 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T18:57:51.869 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:57:51.869 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:57:51.870 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T18:57:51.870 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:57:51.870 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:57:51.871 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:57:51.871 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T18:57:51.871 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:57:51.871 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:57:51.872 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:57:51.872 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:57:51.872 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T18:57:51.872 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:57:51.873 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:57:51.873 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663009066.346375_149.1", 2022-09-12T18:57:51.873 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:57:51.873 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:57:51.874 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:57:51.874 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.346375Z", 2022-09-12T18:57:51.874 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:57:51.875 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:57:51.875 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:57:51.875 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T18:57:51.875 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T18:57:51.876 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T18:57:51.876 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T18:57:51.876 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T18:57:51.876 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T18:57:51.877 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T18:57:51.877 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T18:57:51.877 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T18:57:51.877 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T18:57:51.878 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:57:51.878 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T18:57:51.879 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T18:57:51.879 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:57:51.879 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T18:57:51.879 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T18:57:51.880 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T18:57:51.880 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T18:57:51.880 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T18:57:51Z" 2022-09-12T18:57:51.880 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T18:57:51.882 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009066.346375_149.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:57:46.346375Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:57:51Z" } 2022-09-12T18:57:51.883 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009066.346375_149.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:57:46.346375Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T18:57:51.883 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009066.346375_149.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:57:51.884 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009066.346375_149.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:57:51.884 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T18:57:51.885 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T18:57:51.885 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:57:51.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T18:57:51.887 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T18:57:51.937 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:57:51.937 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:57:51.983 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:57:51.980+0000 7fb901914600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:57:51.984 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:57:51.981+0000 7fb901914600 20 RGW-SYNC:meta: read sync status 2022-09-12T18:57:52.001 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T18:57:52.001 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T18:57:52.002 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T18:57:52.002 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T18:57:52.002 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T18:57:52.003 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T18:57:52.003 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:57:52.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:57:52.004 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T18:57:52.004 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:57:52.004 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T18:57:52.005 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:57:52.005 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:57:52.005 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T18:57:52.005 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:57:52.006 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T18:57:52.006 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:57:52.006 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T18:57:52.007 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:57:52.007 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:57:52.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:57:52.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:57:52.008 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T18:57:52.008 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:57:52.008 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:57:52.008 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T18:57:52.009 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:57:52.009 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:57:52.009 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:57:52.010 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T18:57:52.010 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:57:52.010 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:57:52.010 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:57:52.011 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:57:52.011 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T18:57:52.011 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:57:52.011 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:57:52.012 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663009066.346375_149.1", 2022-09-12T18:57:52.012 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:57:52.012 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:57:52.012 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:57:52.013 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.346375Z", 2022-09-12T18:57:52.013 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:57:52.013 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:57:52.013 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:57:52.014 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T18:57:52.014 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T18:57:52.014 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T18:57:52.015 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T18:57:52.015 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T18:57:52.015 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T18:57:52.015 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T18:57:52.016 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T18:57:52.016 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T18:57:52.016 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T18:57:52.017 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:57:52.017 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T18:57:52.017 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T18:57:52.017 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:57:52.018 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T18:57:52.018 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T18:57:52.018 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T18:57:52.019 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T18:57:52.019 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T18:57:51Z" 2022-09-12T18:57:52.019 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T18:57:52.021 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009066.346375_149.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:57:46.346375Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T18:57:51Z" } 2022-09-12T18:57:52.021 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009066.346375_149.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:57:46.346375Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T18:57:52.022 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009066.346375_149.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:57:52.022 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009066.346375_149.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T18:57:52.023 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T18:57:52.023 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T18:57:52.024 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T18:57:52.024 DEBUG:boto:Method: HEAD 2022-09-12T18:57:52.025 DEBUG:boto:Path: /jdpiak-16/ 2022-09-12T18:57:52.026 DEBUG:boto:Data: 2022-09-12T18:57:52.026 DEBUG:boto:Headers: {} 2022-09-12T18:57:52.027 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:57:52.027 DEBUG:boto:Port: 8000 2022-09-12T18:57:52.028 DEBUG:boto:Params: {} 2022-09-12T18:57:52.028 DEBUG:boto:Token: None 2022-09-12T18:57:52.029 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:57:52 GMT /jdpiak-16/ 2022-09-12T18:57:52.030 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mnYnXdJzNEDcwIHVmMIEsnOEPIY= 2022-09-12T18:57:52.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:57:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mnYnXdJzNEDcwIHVmMIEsnOEPIY=', 'Content-Length': '0'} 2022-09-12T18:57:52.033 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ea4857c6bdea2c0-00631f8130-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:52.034 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T18:57:52.035 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T18:57:52.036 DEBUG:boto:Method: PUT 2022-09-12T18:57:52.037 DEBUG:boto:Path: /jdpiak-16/dummy 2022-09-12T18:57:52.038 DEBUG:boto:Data: 2022-09-12T18:57:52.039 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-12T18:57:52.040 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:57:52.041 DEBUG:boto:Port: 8000 2022-09-12T18:57:52.041 DEBUG:boto:Params: {} 2022-09-12T18:57:52.042 DEBUG:boto:Token: None 2022-09-12T18:57:52.043 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 12 Sep 2022 18:57:52 GMT /jdpiak-16/dummy 2022-09-12T18:57:52.043 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zNEspg0njLfRjS5XHdELVrafcuo= 2022-09-12T18:57:52.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:57:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zNEspg0njLfRjS5XHdELVrafcuo='} 2022-09-12T18:57:52.055 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-amz-request-id', 'tx000002b16b0d3910c0721-00631f8130-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:57:52.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:57:52.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:57:52.058 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T18:57:52.115 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:57:52.115 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:57:52.149 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:57:52.146+0000 7f5697ee5600 1 waiting to reach incremental sync.. 2022-09-12T18:57:57.150 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:57:57.146+0000 7f5697ee5600 1 waiting to reach incremental sync.. 2022-09-12T18:58:02.151 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:58:02.147+0000 7f5697ee5600 1 waiting to reach incremental sync.. 2022-09-12T18:58:07.153 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:58:07.149+0000 7f5697ee5600 1 bucket sync caught up with source: 2022-09-12T18:58:07.153 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000002.7.10, , , , , ] 2022-09-12T18:58:07.154 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000002.7.10, , , , , ] 2022-09-12T18:58:07.154 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:58:07.149+0000 7f5697ee5600 0 bucket checkpoint complete 2022-09-12T18:58:07.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:58:07.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:58:07.171 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T18:58:07.236 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:58:07.237 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:58:07.273 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:58:07.270+0000 7fa97e517600 1 bucket sync caught up with source: 2022-09-12T18:58:07.274 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000002.7.10, , , , , ] 2022-09-12T18:58:07.274 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000002.7.10, , , , , ] 2022-09-12T18:58:07.274 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:58:07.270+0000 7fa97e517600 0 bucket checkpoint complete 2022-09-12T18:58:07.288 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:58:07.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:58:07.289 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T18:58:07.352 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:58:07.352 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:58:07.387 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:58:07.384+0000 7f451354b600 1 bucket sync caught up with source: 2022-09-12T18:58:07.388 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000002.7.10, , , , , ] 2022-09-12T18:58:07.388 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000002.7.10, , , , , ] 2022-09-12T18:58:07.388 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:58:07.384+0000 7f451354b600 0 bucket checkpoint complete 2022-09-12T18:58:07.406 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T18:58:07.406 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T18:58:07.407 DEBUG:boto:Method: HEAD 2022-09-12T18:58:07.407 DEBUG:boto:Path: /jdpiak-17/ 2022-09-12T18:58:07.408 DEBUG:boto:Data: 2022-09-12T18:58:07.409 DEBUG:boto:Headers: {} 2022-09-12T18:58:07.409 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:58:07.410 DEBUG:boto:Port: 8000 2022-09-12T18:58:07.410 DEBUG:boto:Params: {} 2022-09-12T18:58:07.411 DEBUG:boto:Token: None 2022-09-12T18:58:07.411 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:58:07 GMT /jdpiak-17/ 2022-09-12T18:58:07.412 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:upBJtnubuLfo4a5yQDliNOOIw9I= 2022-09-12T18:58:07.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:58:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:upBJtnubuLfo4a5yQDliNOOIw9I=', 'Content-Length': '0'} 2022-09-12T18:58:07.415 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d74076ac356000a5-00631f813f-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:58:07.416 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T18:58:07.417 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T18:58:07.417 DEBUG:boto:Method: PUT 2022-09-12T18:58:07.418 DEBUG:boto:Path: /jdpiak-17/dummy 2022-09-12T18:58:07.418 DEBUG:boto:Data: 2022-09-12T18:58:07.419 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-12T18:58:07.419 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:58:07.420 DEBUG:boto:Port: 8000 2022-09-12T18:58:07.420 DEBUG:boto:Params: {} 2022-09-12T18:58:07.421 DEBUG:boto:Token: None 2022-09-12T18:58:07.421 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 12 Sep 2022 18:58:07 GMT /jdpiak-17/dummy 2022-09-12T18:58:07.422 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zOqskt5nczgMvkCam0jQkg/M16s= 2022-09-12T18:58:07.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:58:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zOqskt5nczgMvkCam0jQkg/M16s='} 2022-09-12T18:58:07.470 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-amz-request-id', 'tx00000874ee4f3fbf960dd-00631f813f-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:58:07.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:58:07.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:58:07.472 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T18:58:07.533 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:58:07.534 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:58:07.575 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:58:07.572+0000 7f44e354c600 1 waiting to reach incremental sync.. 2022-09-12T18:58:12.576 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:58:12.573+0000 7f44e354c600 1 waiting to reach incremental sync.. 2022-09-12T18:58:17.577 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:58:17.573+0000 7f44e354c600 1 waiting to reach incremental sync.. 2022-09-12T18:58:22.579 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:58:22.575+0000 7f44e354c600 1 bucket sync caught up with source: 2022-09-12T18:58:22.579 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000002.20.10, , , , , ] 2022-09-12T18:58:22.580 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000002.20.10, , , , , ] 2022-09-12T18:58:22.580 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:58:22.575+0000 7f44e354c600 0 bucket checkpoint complete 2022-09-12T18:58:22.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:58:22.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:58:22.601 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T18:58:22.661 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:58:22.661 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:58:22.706 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:58:22.703+0000 7f749bd5f600 1 bucket sync caught up with source: 2022-09-12T18:58:22.707 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000002.20.10, , , , , ] 2022-09-12T18:58:22.707 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000002.20.10, , , , , ] 2022-09-12T18:58:22.707 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:58:22.703+0000 7f749bd5f600 0 bucket checkpoint complete 2022-09-12T18:58:22.719 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T18:58:22.720 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T18:58:22.720 DEBUG:boto:Method: HEAD 2022-09-12T18:58:22.721 DEBUG:boto:Path: /jdpiak-18/ 2022-09-12T18:58:22.721 DEBUG:boto:Data: 2022-09-12T18:58:22.722 DEBUG:boto:Headers: {} 2022-09-12T18:58:22.722 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:58:22.723 DEBUG:boto:Port: 8001 2022-09-12T18:58:22.723 DEBUG:boto:Params: {} 2022-09-12T18:58:22.724 DEBUG:boto:Token: None 2022-09-12T18:58:22.724 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:58:22 GMT /jdpiak-18/ 2022-09-12T18:58:22.725 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YnGZEerjMtzXIepooD2vb3FNYUA= 2022-09-12T18:58:22.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:58:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YnGZEerjMtzXIepooD2vb3FNYUA=', 'Content-Length': '0'} 2022-09-12T18:58:22.728 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000033ddfb6d95d46ea6-00631f814e-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:58:22.729 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T18:58:22.730 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T18:58:22.730 DEBUG:boto:Method: PUT 2022-09-12T18:58:22.731 DEBUG:boto:Path: /jdpiak-18/dummy 2022-09-12T18:58:22.731 DEBUG:boto:Data: 2022-09-12T18:58:22.732 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-12T18:58:22.732 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:58:22.733 DEBUG:boto:Port: 8001 2022-09-12T18:58:22.733 DEBUG:boto:Params: {} 2022-09-12T18:58:22.734 DEBUG:boto:Token: None 2022-09-12T18:58:22.735 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 12 Sep 2022 18:58:22 GMT /jdpiak-18/dummy 2022-09-12T18:58:22.736 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+3p8K17EBHTHt+o/7yYG2mFj0Ls= 2022-09-12T18:58:22.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:58:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+3p8K17EBHTHt+o/7yYG2mFj0Ls='} 2022-09-12T18:58:22.747 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-amz-request-id', 'tx00000a2d1d9caf31e483b-00631f814e-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:58:22.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:58:22.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:58:22.749 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T18:58:22.806 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:58:22.806 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:58:22.843 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:58:22.840+0000 7f7673a88600 1 waiting to reach incremental sync.. 2022-09-12T18:58:27.844 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:58:27.841+0000 7f7673a88600 1 waiting to reach incremental sync.. 2022-09-12T18:58:32.845 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:58:32.842+0000 7f7673a88600 1 waiting to reach incremental sync.. 2022-09-12T18:58:37.845 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:58:37.842+0000 7f7673a88600 1 waiting to reach incremental sync.. 2022-09-12T18:58:42.847 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:58:42.843+0000 7f7673a88600 1 bucket sync caught up with source: 2022-09-12T18:58:42.847 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000002.25.10, , , , , ] 2022-09-12T18:58:42.848 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000002.25.10, , , , , ] 2022-09-12T18:58:42.848 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:58:42.843+0000 7f7673a88600 0 bucket checkpoint complete 2022-09-12T18:58:42.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:58:42.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:58:42.870 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T18:58:42.933 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:58:42.933 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:58:42.974 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:58:42.970+0000 7f44a4883600 1 waiting for incremental sync to catch up: 2022-09-12T18:58:42.974 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000002.25.10, , , , , ] 2022-09-12T18:58:42.974 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000005.31.9, , , , , ] 2022-09-12T18:58:47.975 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:58:47.972+0000 7f44a4883600 1 waiting for incremental sync to catch up: 2022-09-12T18:58:47.976 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000002.25.10, , , , , ] 2022-09-12T18:58:47.977 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000005.31.9, , , , , ] 2022-09-12T18:58:52.976 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:58:52.973+0000 7f44a4883600 1 waiting for incremental sync to catch up: 2022-09-12T18:58:52.977 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000002.25.10, , , , , ] 2022-09-12T18:58:52.977 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000005.31.9, , , , , ] 2022-09-12T18:58:57.978 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:58:57.975+0000 7f44a4883600 1 waiting for incremental sync to catch up: 2022-09-12T18:58:57.979 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000002.25.10, , , , , ] 2022-09-12T18:58:57.979 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000005.31.9, , , , , ] 2022-09-12T18:59:02.979 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:59:02.976+0000 7f44a4883600 1 waiting for incremental sync to catch up: 2022-09-12T18:59:02.980 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000002.25.10, , , , , ] 2022-09-12T18:59:02.980 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000005.31.9, , , , , ] 2022-09-12T18:59:07.981 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:59:07.978+0000 7f44a4883600 1 waiting for incremental sync to catch up: 2022-09-12T18:59:07.981 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000002.25.10, , , , , ] 2022-09-12T18:59:07.982 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000005.31.9, , , , , ] 2022-09-12T18:59:12.982 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:59:12.980+0000 7f44a4883600 1 waiting for incremental sync to catch up: 2022-09-12T18:59:12.983 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000002.25.10, , , , , ] 2022-09-12T18:59:12.983 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000005.31.9, , , , , ] 2022-09-12T18:59:17.984 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:59:17.980+0000 7f44a4883600 1 waiting for incremental sync to catch up: 2022-09-12T18:59:17.985 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000002.25.10, , , , , ] 2022-09-12T18:59:17.985 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000005.31.9, , , , , ] 2022-09-12T18:59:22.985 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:59:22.982+0000 7f44a4883600 1 bucket sync caught up with source: 2022-09-12T18:59:22.986 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000006.33.9, , , , , ] 2022-09-12T18:59:22.987 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000005.31.9, , , , , ] 2022-09-12T18:59:22.987 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:59:22.982+0000 7f44a4883600 0 bucket checkpoint complete 2022-09-12T18:59:23.003 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T18:59:23.004 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T18:59:23.005 DEBUG:boto:Method: HEAD 2022-09-12T18:59:23.006 DEBUG:boto:Path: /jdpiak-16/ 2022-09-12T18:59:23.007 DEBUG:boto:Data: 2022-09-12T18:59:23.008 DEBUG:boto:Headers: {} 2022-09-12T18:59:23.009 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:59:23.010 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.011 DEBUG:boto:Params: {} 2022-09-12T18:59:23.013 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T18:59:23.013 DEBUG:boto:Token: None 2022-09-12T18:59:23.014 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-16/ 2022-09-12T18:59:23.015 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZlAXkIB5g9m+WEqLW+Di6qkDCXY= 2022-09-12T18:59:23.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZlAXkIB5g9m+WEqLW+Di6qkDCXY=', 'Content-Length': '0'} 2022-09-12T18:59:23.020 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000022e5588ff29daf11-00631f818b-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.021 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T18:59:23.022 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T18:59:23.023 DEBUG:boto:Method: PUT 2022-09-12T18:59:23.023 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1 2022-09-12T18:59:23.024 DEBUG:boto:Data: 2022-09-12T18:59:23.024 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:23.025 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:59:23.025 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.026 DEBUG:boto:Params: {} 2022-09-12T18:59:23.027 DEBUG:boto:Token: None 2022-09-12T18:59:23.027 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-16/obj-test-zone1 2022-09-12T18:59:23.028 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9/rzkvn0pb1di7oijglvleYihZk= 2022-09-12T18:59:23.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9/rzkvn0pb1di7oijglvleYihZk='} 2022-09-12T18:59:23.042 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-amz-request-id', 'tx00000fcb8abc70d4c9388-00631f818b-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.043 DEBUG:rgw_multi.tests:version1 id=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T18:59:23.045 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T18:59:23.046 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T18:59:23.047 DEBUG:boto:Method: PUT 2022-09-12T18:59:23.047 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1 2022-09-12T18:59:23.048 DEBUG:boto:Data: 2022-09-12T18:59:23.049 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:23.049 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:59:23.050 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.050 DEBUG:boto:Params: {} 2022-09-12T18:59:23.051 DEBUG:boto:Token: None 2022-09-12T18:59:23.052 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-16/obj-test-zone1 2022-09-12T18:59:23.052 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lqWKcQx9TbAyEwcaCrNEkYnJoRs= 2022-09-12T18:59:23.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lqWKcQx9TbAyEwcaCrNEkYnJoRs='} 2022-09-12T18:59:23.064 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'PgNWPtamTO8feNnzNSmdx4P7pXXhy2Y'), ('x-amz-request-id', 'tx000004fa8ff9e81a9579f-00631f818b-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.065 DEBUG:rgw_multi.tests:version2 id=PgNWPtamTO8feNnzNSmdx4P7pXXhy2Y 2022-09-12T18:59:23.066 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T18:59:23.067 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T18:59:23.068 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=PgNWPtamTO8feNnzNSmdx4P7pXXhy2Y 2022-09-12T18:59:23.069 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=PgNWPtamTO8feNnzNSmdx4P7pXXhy2Y 2022-09-12T18:59:23.070 DEBUG:boto:Method: DELETE 2022-09-12T18:59:23.071 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=PgNWPtamTO8feNnzNSmdx4P7pXXhy2Y 2022-09-12T18:59:23.071 DEBUG:boto:Data: 2022-09-12T18:59:23.072 DEBUG:boto:Headers: {} 2022-09-12T18:59:23.073 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:59:23.074 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.075 DEBUG:boto:Params: {} 2022-09-12T18:59:23.076 DEBUG:boto:Token: None 2022-09-12T18:59:23.077 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-16/obj-test-zone1?versionId=PgNWPtamTO8feNnzNSmdx4P7pXXhy2Y 2022-09-12T18:59:23.078 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YhQAlkqoxNsgpHqpGNFo5lrQDIo= 2022-09-12T18:59:23.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YhQAlkqoxNsgpHqpGNFo5lrQDIo=', 'Content-Length': '0'} 2022-09-12T18:59:23.090 DEBUG:boto:Response headers: [('x-amz-version-id', 'PgNWPtamTO8feNnzNSmdx4P7pXXhy2Y'), ('x-amz-request-id', 'tx000006460babf0387676a-00631f818b-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.092 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T18:59:23.092 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T18:59:23.093 DEBUG:boto:Method: PUT 2022-09-12T18:59:23.093 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1 2022-09-12T18:59:23.094 DEBUG:boto:Data: 2022-09-12T18:59:23.094 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:23.095 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:59:23.095 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.096 DEBUG:boto:Params: {} 2022-09-12T18:59:23.096 DEBUG:boto:Token: None 2022-09-12T18:59:23.097 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-16/obj-test-zone1 2022-09-12T18:59:23.097 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gkFn3ubs8ytTfyHjNcc892NUAqM= 2022-09-12T18:59:23.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gkFn3ubs8ytTfyHjNcc892NUAqM='} 2022-09-12T18:59:23.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '3Bnp84UR1g5dHGY1hDE9fXfVcgUMrR7'), ('x-amz-request-id', 'tx000007292ba2433abf82b-00631f818b-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.147 DEBUG:rgw_multi.tests:version3 id=3Bnp84UR1g5dHGY1hDE9fXfVcgUMrR7 2022-09-12T18:59:23.148 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T18:59:23.148 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T18:59:23.149 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=3Bnp84UR1g5dHGY1hDE9fXfVcgUMrR7 2022-09-12T18:59:23.149 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=3Bnp84UR1g5dHGY1hDE9fXfVcgUMrR7 2022-09-12T18:59:23.150 DEBUG:boto:Method: DELETE 2022-09-12T18:59:23.150 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=3Bnp84UR1g5dHGY1hDE9fXfVcgUMrR7 2022-09-12T18:59:23.151 DEBUG:boto:Data: 2022-09-12T18:59:23.151 DEBUG:boto:Headers: {} 2022-09-12T18:59:23.152 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:59:23.152 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.153 DEBUG:boto:Params: {} 2022-09-12T18:59:23.153 DEBUG:boto:Token: None 2022-09-12T18:59:23.154 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-16/obj-test-zone1?versionId=3Bnp84UR1g5dHGY1hDE9fXfVcgUMrR7 2022-09-12T18:59:23.155 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RlcWspIikr/ynj5IAcZfN9nhYtE= 2022-09-12T18:59:23.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RlcWspIikr/ynj5IAcZfN9nhYtE=', 'Content-Length': '0'} 2022-09-12T18:59:23.167 DEBUG:boto:Response headers: [('x-amz-version-id', '3Bnp84UR1g5dHGY1hDE9fXfVcgUMrR7'), ('x-amz-request-id', 'tx00000c949572bb6d1d407-00631f818b-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.167 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T18:59:23.168 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T18:59:23.169 DEBUG:boto:Method: HEAD 2022-09-12T18:59:23.170 DEBUG:boto:Path: /jdpiak-16/ 2022-09-12T18:59:23.171 DEBUG:boto:Data: 2022-09-12T18:59:23.172 DEBUG:boto:Headers: {} 2022-09-12T18:59:23.172 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:59:23.173 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.173 DEBUG:boto:Params: {} 2022-09-12T18:59:23.174 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T18:59:23.175 DEBUG:boto:Token: None 2022-09-12T18:59:23.176 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-16/ 2022-09-12T18:59:23.176 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZlAXkIB5g9m+WEqLW+Di6qkDCXY= 2022-09-12T18:59:23.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZlAXkIB5g9m+WEqLW+Di6qkDCXY=', 'Content-Length': '0'} 2022-09-12T18:59:23.180 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5a746339b496ce7-00631f818b-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.181 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T18:59:23.182 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T18:59:23.182 DEBUG:boto:Method: PUT 2022-09-12T18:59:23.183 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2 2022-09-12T18:59:23.183 DEBUG:boto:Data: 2022-09-12T18:59:23.184 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:23.184 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:59:23.185 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.185 DEBUG:boto:Params: {} 2022-09-12T18:59:23.186 DEBUG:boto:Token: None 2022-09-12T18:59:23.187 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-16/obj-test-zone2 2022-09-12T18:59:23.187 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:upZR3JoocVSA6NEs5RgmAS52lJw= 2022-09-12T18:59:23.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:upZR3JoocVSA6NEs5RgmAS52lJw='} 2022-09-12T18:59:23.203 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-amz-request-id', 'tx00000202aa9a9a1276bc9-00631f818b-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.203 DEBUG:rgw_multi.tests:version1 id=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T18:59:23.204 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T18:59:23.205 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T18:59:23.205 DEBUG:boto:Method: PUT 2022-09-12T18:59:23.206 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2 2022-09-12T18:59:23.207 DEBUG:boto:Data: 2022-09-12T18:59:23.208 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:23.209 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:59:23.209 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.210 DEBUG:boto:Params: {} 2022-09-12T18:59:23.211 DEBUG:boto:Token: None 2022-09-12T18:59:23.212 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-16/obj-test-zone2 2022-09-12T18:59:23.212 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UynRoKDjTf4BAMiNA+fGTG9tQNw= 2022-09-12T18:59:23.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UynRoKDjTf4BAMiNA+fGTG9tQNw='} 2022-09-12T18:59:23.224 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'UGZq5HWqzCzomVNe6p867KQPf4aneYO'), ('x-amz-request-id', 'tx000006d8d889317ce6c1b-00631f818b-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.225 DEBUG:rgw_multi.tests:version2 id=UGZq5HWqzCzomVNe6p867KQPf4aneYO 2022-09-12T18:59:23.225 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T18:59:23.226 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T18:59:23.226 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=UGZq5HWqzCzomVNe6p867KQPf4aneYO 2022-09-12T18:59:23.227 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=UGZq5HWqzCzomVNe6p867KQPf4aneYO 2022-09-12T18:59:23.227 DEBUG:boto:Method: DELETE 2022-09-12T18:59:23.228 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=UGZq5HWqzCzomVNe6p867KQPf4aneYO 2022-09-12T18:59:23.228 DEBUG:boto:Data: 2022-09-12T18:59:23.229 DEBUG:boto:Headers: {} 2022-09-12T18:59:23.229 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:59:23.230 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.230 DEBUG:boto:Params: {} 2022-09-12T18:59:23.231 DEBUG:boto:Token: None 2022-09-12T18:59:23.231 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-16/obj-test-zone2?versionId=UGZq5HWqzCzomVNe6p867KQPf4aneYO 2022-09-12T18:59:23.232 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1dHCH1YcPu5STxQRLMNPjjEgtlI= 2022-09-12T18:59:23.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1dHCH1YcPu5STxQRLMNPjjEgtlI=', 'Content-Length': '0'} 2022-09-12T18:59:23.244 DEBUG:boto:Response headers: [('x-amz-version-id', 'UGZq5HWqzCzomVNe6p867KQPf4aneYO'), ('x-amz-request-id', 'tx00000c48a1c83fc87243e-00631f818b-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.245 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T18:59:23.245 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T18:59:23.246 DEBUG:boto:Method: PUT 2022-09-12T18:59:23.246 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2 2022-09-12T18:59:23.247 DEBUG:boto:Data: 2022-09-12T18:59:23.247 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:23.248 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:59:23.248 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.249 DEBUG:boto:Params: {} 2022-09-12T18:59:23.249 DEBUG:boto:Token: None 2022-09-12T18:59:23.250 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-16/obj-test-zone2 2022-09-12T18:59:23.251 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cmmyqXThVfyloA0ODNO3TK2g/Iw= 2022-09-12T18:59:23.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cmmyqXThVfyloA0ODNO3TK2g/Iw='} 2022-09-12T18:59:23.294 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'PSlX-jGVWLhMnFpVEC8iwTg1pdHFZIt'), ('x-amz-request-id', 'tx000001632b541dd69837f-00631f818b-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.296 DEBUG:rgw_multi.tests:version3 id=PSlX-jGVWLhMnFpVEC8iwTg1pdHFZIt 2022-09-12T18:59:23.297 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T18:59:23.298 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T18:59:23.298 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=PSlX-jGVWLhMnFpVEC8iwTg1pdHFZIt 2022-09-12T18:59:23.299 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=PSlX-jGVWLhMnFpVEC8iwTg1pdHFZIt 2022-09-12T18:59:23.299 DEBUG:boto:Method: DELETE 2022-09-12T18:59:23.300 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=PSlX-jGVWLhMnFpVEC8iwTg1pdHFZIt 2022-09-12T18:59:23.300 DEBUG:boto:Data: 2022-09-12T18:59:23.301 DEBUG:boto:Headers: {} 2022-09-12T18:59:23.301 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:59:23.301 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.302 DEBUG:boto:Params: {} 2022-09-12T18:59:23.303 DEBUG:boto:Token: None 2022-09-12T18:59:23.303 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-16/obj-test-zone2?versionId=PSlX-jGVWLhMnFpVEC8iwTg1pdHFZIt 2022-09-12T18:59:23.304 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pI62edKebMGNoxLWM2lrfNcTXZI= 2022-09-12T18:59:23.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pI62edKebMGNoxLWM2lrfNcTXZI=', 'Content-Length': '0'} 2022-09-12T18:59:23.315 DEBUG:boto:Response headers: [('x-amz-version-id', 'PSlX-jGVWLhMnFpVEC8iwTg1pdHFZIt'), ('x-amz-request-id', 'tx00000f4622c0f60f077ca-00631f818b-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.316 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T18:59:23.317 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T18:59:23.317 DEBUG:boto:Method: HEAD 2022-09-12T18:59:23.318 DEBUG:boto:Path: /jdpiak-16/ 2022-09-12T18:59:23.318 DEBUG:boto:Data: 2022-09-12T18:59:23.319 DEBUG:boto:Headers: {} 2022-09-12T18:59:23.319 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:59:23.320 DEBUG:boto:Port: 8001 2022-09-12T18:59:23.320 DEBUG:boto:Params: {} 2022-09-12T18:59:23.321 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T18:59:23.321 DEBUG:boto:Token: None 2022-09-12T18:59:23.322 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-16/ 2022-09-12T18:59:23.323 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZlAXkIB5g9m+WEqLW+Di6qkDCXY= 2022-09-12T18:59:23.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZlAXkIB5g9m+WEqLW+Di6qkDCXY=', 'Content-Length': '0'} 2022-09-12T18:59:23.327 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a71949f43e4fe91-00631f818b-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.327 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T18:59:23.328 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T18:59:23.328 DEBUG:boto:Method: PUT 2022-09-12T18:59:23.329 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3 2022-09-12T18:59:23.329 DEBUG:boto:Data: 2022-09-12T18:59:23.330 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:23.331 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:59:23.331 DEBUG:boto:Port: 8001 2022-09-12T18:59:23.331 DEBUG:boto:Params: {} 2022-09-12T18:59:23.332 DEBUG:boto:Token: None 2022-09-12T18:59:23.333 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-16/obj-test-zone3 2022-09-12T18:59:23.333 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ifUxgJPFDOqCMd0ceEf/LfeqVhU= 2022-09-12T18:59:23.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ifUxgJPFDOqCMd0ceEf/LfeqVhU='} 2022-09-12T18:59:23.346 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-amz-request-id', 'tx00000a7cb655d15ac9d2f-00631f818b-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.347 DEBUG:rgw_multi.tests:version1 id=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T18:59:23.347 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T18:59:23.348 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T18:59:23.348 DEBUG:boto:Method: PUT 2022-09-12T18:59:23.349 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3 2022-09-12T18:59:23.349 DEBUG:boto:Data: 2022-09-12T18:59:23.350 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:23.350 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:59:23.351 DEBUG:boto:Port: 8001 2022-09-12T18:59:23.351 DEBUG:boto:Params: {} 2022-09-12T18:59:23.352 DEBUG:boto:Token: None 2022-09-12T18:59:23.353 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-16/obj-test-zone3 2022-09-12T18:59:23.353 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wLjVtmQFc7k+w72ByU771X53eR4= 2022-09-12T18:59:23.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wLjVtmQFc7k+w72ByU771X53eR4='} 2022-09-12T18:59:23.366 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'MYpWhq7c5jTR9-UBPXMiYZsDfXeSch1'), ('x-amz-request-id', 'tx000006c8a324597add02a-00631f818b-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.367 DEBUG:rgw_multi.tests:version2 id=MYpWhq7c5jTR9-UBPXMiYZsDfXeSch1 2022-09-12T18:59:23.367 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T18:59:23.368 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T18:59:23.368 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=MYpWhq7c5jTR9-UBPXMiYZsDfXeSch1 2022-09-12T18:59:23.369 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=MYpWhq7c5jTR9-UBPXMiYZsDfXeSch1 2022-09-12T18:59:23.369 DEBUG:boto:Method: DELETE 2022-09-12T18:59:23.370 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=MYpWhq7c5jTR9-UBPXMiYZsDfXeSch1 2022-09-12T18:59:23.370 DEBUG:boto:Data: 2022-09-12T18:59:23.371 DEBUG:boto:Headers: {} 2022-09-12T18:59:23.371 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:59:23.372 DEBUG:boto:Port: 8001 2022-09-12T18:59:23.372 DEBUG:boto:Params: {} 2022-09-12T18:59:23.373 DEBUG:boto:Token: None 2022-09-12T18:59:23.374 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-16/obj-test-zone3?versionId=MYpWhq7c5jTR9-UBPXMiYZsDfXeSch1 2022-09-12T18:59:23.374 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hGf3eT2OnwgdzV8oVW3vMhv1AvM= 2022-09-12T18:59:23.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hGf3eT2OnwgdzV8oVW3vMhv1AvM=', 'Content-Length': '0'} 2022-09-12T18:59:23.387 DEBUG:boto:Response headers: [('x-amz-version-id', 'MYpWhq7c5jTR9-UBPXMiYZsDfXeSch1'), ('x-amz-request-id', 'tx000003ebba8e229ab7c9e-00631f818b-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.388 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T18:59:23.388 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T18:59:23.389 DEBUG:boto:Method: PUT 2022-09-12T18:59:23.389 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3 2022-09-12T18:59:23.390 DEBUG:boto:Data: 2022-09-12T18:59:23.390 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:23.391 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:59:23.392 DEBUG:boto:Port: 8001 2022-09-12T18:59:23.393 DEBUG:boto:Params: {} 2022-09-12T18:59:23.394 DEBUG:boto:Token: None 2022-09-12T18:59:23.395 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-16/obj-test-zone3 2022-09-12T18:59:23.396 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q1SPfCht4Cessajsl9/RNxF/j/Y= 2022-09-12T18:59:23.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q1SPfCht4Cessajsl9/RNxF/j/Y='} 2022-09-12T18:59:23.446 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'JJlJuUhDwTbGrsj3n8iYZogZOtRUhoU'), ('x-amz-request-id', 'tx000005a0a54bb5059d3a7-00631f818b-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.447 DEBUG:rgw_multi.tests:version3 id=JJlJuUhDwTbGrsj3n8iYZogZOtRUhoU 2022-09-12T18:59:23.447 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T18:59:23.448 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T18:59:23.448 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=JJlJuUhDwTbGrsj3n8iYZogZOtRUhoU 2022-09-12T18:59:23.449 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=JJlJuUhDwTbGrsj3n8iYZogZOtRUhoU 2022-09-12T18:59:23.450 DEBUG:boto:Method: DELETE 2022-09-12T18:59:23.450 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=JJlJuUhDwTbGrsj3n8iYZogZOtRUhoU 2022-09-12T18:59:23.451 DEBUG:boto:Data: 2022-09-12T18:59:23.451 DEBUG:boto:Headers: {} 2022-09-12T18:59:23.452 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:59:23.452 DEBUG:boto:Port: 8001 2022-09-12T18:59:23.453 DEBUG:boto:Params: {} 2022-09-12T18:59:23.453 DEBUG:boto:Token: None 2022-09-12T18:59:23.454 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-16/obj-test-zone3?versionId=JJlJuUhDwTbGrsj3n8iYZogZOtRUhoU 2022-09-12T18:59:23.454 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V4O0VXpzbWQhok12cnoBSQ3WV/4= 2022-09-12T18:59:23.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V4O0VXpzbWQhok12cnoBSQ3WV/4=', 'Content-Length': '0'} 2022-09-12T18:59:23.467 DEBUG:boto:Response headers: [('x-amz-version-id', 'JJlJuUhDwTbGrsj3n8iYZogZOtRUhoU'), ('x-amz-request-id', 'tx000003a4989457a80fdac-00631f818b-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.468 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T18:59:23.469 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T18:59:23.470 DEBUG:boto:Method: HEAD 2022-09-12T18:59:23.470 DEBUG:boto:Path: /jdpiak-17/ 2022-09-12T18:59:23.471 DEBUG:boto:Data: 2022-09-12T18:59:23.471 DEBUG:boto:Headers: {} 2022-09-12T18:59:23.472 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:59:23.472 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.473 DEBUG:boto:Params: {} 2022-09-12T18:59:23.473 DEBUG:boto:Token: None 2022-09-12T18:59:23.474 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-17/ 2022-09-12T18:59:23.475 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5qhHW9AjHnIe+4uh012uPzq3IcI= 2022-09-12T18:59:23.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5qhHW9AjHnIe+4uh012uPzq3IcI=', 'Content-Length': '0'} 2022-09-12T18:59:23.478 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c0c6ba2a6ab8a45d-00631f818b-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.479 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T18:59:23.479 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T18:59:23.480 DEBUG:boto:Method: PUT 2022-09-12T18:59:23.480 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1 2022-09-12T18:59:23.481 DEBUG:boto:Data: 2022-09-12T18:59:23.481 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:23.482 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:59:23.483 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.484 DEBUG:boto:Params: {} 2022-09-12T18:59:23.485 DEBUG:boto:Token: None 2022-09-12T18:59:23.486 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-17/obj-test-zone1 2022-09-12T18:59:23.487 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NxEA/JlGAX6DXXDCDhXjw0/3ivE= 2022-09-12T18:59:23.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NxEA/JlGAX6DXXDCDhXjw0/3ivE='} 2022-09-12T18:59:23.500 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-amz-request-id', 'tx00000171b6da2370831d3-00631f818b-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.501 DEBUG:rgw_multi.tests:version1 id=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T18:59:23.502 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T18:59:23.503 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T18:59:23.503 DEBUG:boto:Method: PUT 2022-09-12T18:59:23.504 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1 2022-09-12T18:59:23.504 DEBUG:boto:Data: 2022-09-12T18:59:23.505 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:23.505 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:59:23.506 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.506 DEBUG:boto:Params: {} 2022-09-12T18:59:23.507 DEBUG:boto:Token: None 2022-09-12T18:59:23.508 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-17/obj-test-zone1 2022-09-12T18:59:23.508 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gd4uspwG9dRG+zViL6ZRif2z3qA= 2022-09-12T18:59:23.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gd4uspwG9dRG+zViL6ZRif2z3qA='} 2022-09-12T18:59:23.554 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '40wpNRnmfhbf-I03p9vsISzY4oOOJ6E'), ('x-amz-request-id', 'tx00000969d049fe2c39c48-00631f818b-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.555 DEBUG:rgw_multi.tests:version2 id=40wpNRnmfhbf-I03p9vsISzY4oOOJ6E 2022-09-12T18:59:23.556 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T18:59:23.556 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T18:59:23.557 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=40wpNRnmfhbf-I03p9vsISzY4oOOJ6E 2022-09-12T18:59:23.557 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=40wpNRnmfhbf-I03p9vsISzY4oOOJ6E 2022-09-12T18:59:23.557 DEBUG:boto:Method: DELETE 2022-09-12T18:59:23.558 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=40wpNRnmfhbf-I03p9vsISzY4oOOJ6E 2022-09-12T18:59:23.559 DEBUG:boto:Data: 2022-09-12T18:59:23.559 DEBUG:boto:Headers: {} 2022-09-12T18:59:23.560 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:59:23.560 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.561 DEBUG:boto:Params: {} 2022-09-12T18:59:23.561 DEBUG:boto:Token: None 2022-09-12T18:59:23.562 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-17/obj-test-zone1?versionId=40wpNRnmfhbf-I03p9vsISzY4oOOJ6E 2022-09-12T18:59:23.562 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ECNw9KgEQ3JrO+A3I+9817p6MDA= 2022-09-12T18:59:23.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ECNw9KgEQ3JrO+A3I+9817p6MDA=', 'Content-Length': '0'} 2022-09-12T18:59:23.573 DEBUG:boto:Response headers: [('x-amz-version-id', '40wpNRnmfhbf-I03p9vsISzY4oOOJ6E'), ('x-amz-request-id', 'tx00000f23b9dc606fbb7d6-00631f818b-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.574 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T18:59:23.575 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T18:59:23.576 DEBUG:boto:Method: PUT 2022-09-12T18:59:23.577 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1 2022-09-12T18:59:23.578 DEBUG:boto:Data: 2022-09-12T18:59:23.579 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:23.579 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:59:23.580 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.581 DEBUG:boto:Params: {} 2022-09-12T18:59:23.582 DEBUG:boto:Token: None 2022-09-12T18:59:23.583 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-17/obj-test-zone1 2022-09-12T18:59:23.584 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Pr2uMfXV2z31rBGuB5Yrw533AjI= 2022-09-12T18:59:23.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Pr2uMfXV2z31rBGuB5Yrw533AjI='} 2022-09-12T18:59:23.596 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ToSsR711agA0nBr-7mA72fRnpwLUXLO'), ('x-amz-request-id', 'tx00000ff29b03c173fbedb-00631f818b-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.597 DEBUG:rgw_multi.tests:version3 id=ToSsR711agA0nBr-7mA72fRnpwLUXLO 2022-09-12T18:59:23.597 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T18:59:23.598 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T18:59:23.598 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=ToSsR711agA0nBr-7mA72fRnpwLUXLO 2022-09-12T18:59:23.599 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=ToSsR711agA0nBr-7mA72fRnpwLUXLO 2022-09-12T18:59:23.600 DEBUG:boto:Method: DELETE 2022-09-12T18:59:23.601 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=ToSsR711agA0nBr-7mA72fRnpwLUXLO 2022-09-12T18:59:23.602 DEBUG:boto:Data: 2022-09-12T18:59:23.602 DEBUG:boto:Headers: {} 2022-09-12T18:59:23.603 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:59:23.603 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.604 DEBUG:boto:Params: {} 2022-09-12T18:59:23.604 DEBUG:boto:Token: None 2022-09-12T18:59:23.605 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-17/obj-test-zone1?versionId=ToSsR711agA0nBr-7mA72fRnpwLUXLO 2022-09-12T18:59:23.606 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hLFAgATH0PNRgiJePMbkJWIAR30= 2022-09-12T18:59:23.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hLFAgATH0PNRgiJePMbkJWIAR30=', 'Content-Length': '0'} 2022-09-12T18:59:23.619 DEBUG:boto:Response headers: [('x-amz-version-id', 'ToSsR711agA0nBr-7mA72fRnpwLUXLO'), ('x-amz-request-id', 'tx000006dc4b744e4a701c8-00631f818b-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.620 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T18:59:23.621 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T18:59:23.621 DEBUG:boto:Method: HEAD 2022-09-12T18:59:23.622 DEBUG:boto:Path: /jdpiak-17/ 2022-09-12T18:59:23.622 DEBUG:boto:Data: 2022-09-12T18:59:23.623 DEBUG:boto:Headers: {} 2022-09-12T18:59:23.623 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:59:23.624 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.624 DEBUG:boto:Params: {} 2022-09-12T18:59:23.625 DEBUG:boto:Token: None 2022-09-12T18:59:23.625 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-17/ 2022-09-12T18:59:23.626 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5qhHW9AjHnIe+4uh012uPzq3IcI= 2022-09-12T18:59:23.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5qhHW9AjHnIe+4uh012uPzq3IcI=', 'Content-Length': '0'} 2022-09-12T18:59:23.630 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6243d6fb1ecda65-00631f818b-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.630 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T18:59:23.631 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T18:59:23.631 DEBUG:boto:Method: PUT 2022-09-12T18:59:23.632 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2 2022-09-12T18:59:23.632 DEBUG:boto:Data: 2022-09-12T18:59:23.633 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:23.633 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:59:23.634 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.634 DEBUG:boto:Params: {} 2022-09-12T18:59:23.635 DEBUG:boto:Token: None 2022-09-12T18:59:23.636 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-17/obj-test-zone2 2022-09-12T18:59:23.637 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uDcR0uV07CCJBqLErpErKbbhX8Q= 2022-09-12T18:59:23.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uDcR0uV07CCJBqLErpErKbbhX8Q='} 2022-09-12T18:59:23.654 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-amz-request-id', 'tx000003640910bb8e3cc6a-00631f818b-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.654 DEBUG:rgw_multi.tests:version1 id=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T18:59:23.655 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T18:59:23.656 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T18:59:23.656 DEBUG:boto:Method: PUT 2022-09-12T18:59:23.657 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2 2022-09-12T18:59:23.657 DEBUG:boto:Data: 2022-09-12T18:59:23.658 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:23.658 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:59:23.659 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.659 DEBUG:boto:Params: {} 2022-09-12T18:59:23.660 DEBUG:boto:Token: None 2022-09-12T18:59:23.661 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-17/obj-test-zone2 2022-09-12T18:59:23.662 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VZLPc136R+rIn8y3MDrP0KPUMuU= 2022-09-12T18:59:23.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VZLPc136R+rIn8y3MDrP0KPUMuU='} 2022-09-12T18:59:23.674 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'so9vOABdReboUiunl9QN2fQV.-nRxfn'), ('x-amz-request-id', 'tx0000046e5d36c99a0f22e-00631f818b-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.675 DEBUG:rgw_multi.tests:version2 id=so9vOABdReboUiunl9QN2fQV.-nRxfn 2022-09-12T18:59:23.675 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T18:59:23.676 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T18:59:23.676 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=so9vOABdReboUiunl9QN2fQV.-nRxfn 2022-09-12T18:59:23.677 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=so9vOABdReboUiunl9QN2fQV.-nRxfn 2022-09-12T18:59:23.677 DEBUG:boto:Method: DELETE 2022-09-12T18:59:23.679 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=so9vOABdReboUiunl9QN2fQV.-nRxfn 2022-09-12T18:59:23.679 DEBUG:boto:Data: 2022-09-12T18:59:23.680 DEBUG:boto:Headers: {} 2022-09-12T18:59:23.681 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:59:23.682 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.683 DEBUG:boto:Params: {} 2022-09-12T18:59:23.684 DEBUG:boto:Token: None 2022-09-12T18:59:23.685 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-17/obj-test-zone2?versionId=so9vOABdReboUiunl9QN2fQV.-nRxfn 2022-09-12T18:59:23.685 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A+BGkBBSSKl6v6e52yFpXCXss1o= 2022-09-12T18:59:23.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A+BGkBBSSKl6v6e52yFpXCXss1o=', 'Content-Length': '0'} 2022-09-12T18:59:23.704 DEBUG:boto:Response headers: [('x-amz-version-id', 'so9vOABdReboUiunl9QN2fQV.-nRxfn'), ('x-amz-request-id', 'tx00000fd9fe2ac2139a1a7-00631f818b-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.705 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T18:59:23.705 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T18:59:23.706 DEBUG:boto:Method: PUT 2022-09-12T18:59:23.707 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2 2022-09-12T18:59:23.707 DEBUG:boto:Data: 2022-09-12T18:59:23.708 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:23.708 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:59:23.709 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.709 DEBUG:boto:Params: {} 2022-09-12T18:59:23.710 DEBUG:boto:Token: None 2022-09-12T18:59:23.710 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-17/obj-test-zone2 2022-09-12T18:59:23.712 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QfVLWTVSZNguLBesqGd1HhKkvjI= 2022-09-12T18:59:23.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QfVLWTVSZNguLBesqGd1HhKkvjI='} 2022-09-12T18:59:23.762 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'zoRxI8C9pe7.C.exbUZft0gD3EE8Ne1'), ('x-amz-request-id', 'tx00000e42fb4605f1afff1-00631f818b-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.763 DEBUG:rgw_multi.tests:version3 id=zoRxI8C9pe7.C.exbUZft0gD3EE8Ne1 2022-09-12T18:59:23.764 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T18:59:23.764 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T18:59:23.765 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=zoRxI8C9pe7.C.exbUZft0gD3EE8Ne1 2022-09-12T18:59:23.765 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=zoRxI8C9pe7.C.exbUZft0gD3EE8Ne1 2022-09-12T18:59:23.766 DEBUG:boto:Method: DELETE 2022-09-12T18:59:23.766 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=zoRxI8C9pe7.C.exbUZft0gD3EE8Ne1 2022-09-12T18:59:23.767 DEBUG:boto:Data: 2022-09-12T18:59:23.767 DEBUG:boto:Headers: {} 2022-09-12T18:59:23.768 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:59:23.768 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.769 DEBUG:boto:Params: {} 2022-09-12T18:59:23.769 DEBUG:boto:Token: None 2022-09-12T18:59:23.770 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-17/obj-test-zone2?versionId=zoRxI8C9pe7.C.exbUZft0gD3EE8Ne1 2022-09-12T18:59:23.770 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ksfigP5bxOzVWQyOg0lBrsW413k= 2022-09-12T18:59:23.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ksfigP5bxOzVWQyOg0lBrsW413k=', 'Content-Length': '0'} 2022-09-12T18:59:23.783 DEBUG:boto:Response headers: [('x-amz-version-id', 'zoRxI8C9pe7.C.exbUZft0gD3EE8Ne1'), ('x-amz-request-id', 'tx00000236837d910b4d757-00631f818b-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.784 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T18:59:23.785 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T18:59:23.785 DEBUG:boto:Method: HEAD 2022-09-12T18:59:23.786 DEBUG:boto:Path: /jdpiak-17/ 2022-09-12T18:59:23.786 DEBUG:boto:Data: 2022-09-12T18:59:23.787 DEBUG:boto:Headers: {} 2022-09-12T18:59:23.787 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:59:23.788 DEBUG:boto:Port: 8001 2022-09-12T18:59:23.788 DEBUG:boto:Params: {} 2022-09-12T18:59:23.789 DEBUG:boto:Token: None 2022-09-12T18:59:23.789 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-17/ 2022-09-12T18:59:23.790 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5qhHW9AjHnIe+4uh012uPzq3IcI= 2022-09-12T18:59:23.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5qhHW9AjHnIe+4uh012uPzq3IcI=', 'Content-Length': '0'} 2022-09-12T18:59:23.793 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000372689dda9b1300a-00631f818b-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.794 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T18:59:23.794 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T18:59:23.795 DEBUG:boto:Method: PUT 2022-09-12T18:59:23.795 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3 2022-09-12T18:59:23.796 DEBUG:boto:Data: 2022-09-12T18:59:23.796 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:23.797 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:59:23.797 DEBUG:boto:Port: 8001 2022-09-12T18:59:23.798 DEBUG:boto:Params: {} 2022-09-12T18:59:23.799 DEBUG:boto:Token: None 2022-09-12T18:59:23.799 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-17/obj-test-zone3 2022-09-12T18:59:23.800 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b6zN4gn+ZAzaXCfgCEdoNfcNe9E= 2022-09-12T18:59:23.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b6zN4gn+ZAzaXCfgCEdoNfcNe9E='} 2022-09-12T18:59:23.812 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-amz-request-id', 'tx00000c64d7417f4cf32d5-00631f818b-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.813 DEBUG:rgw_multi.tests:version1 id=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T18:59:23.814 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T18:59:23.815 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T18:59:23.815 DEBUG:boto:Method: PUT 2022-09-12T18:59:23.816 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3 2022-09-12T18:59:23.817 DEBUG:boto:Data: 2022-09-12T18:59:23.818 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:23.820 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:59:23.821 DEBUG:boto:Port: 8001 2022-09-12T18:59:23.822 DEBUG:boto:Params: {} 2022-09-12T18:59:23.823 DEBUG:boto:Token: None 2022-09-12T18:59:23.824 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-17/obj-test-zone3 2022-09-12T18:59:23.825 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DMLoeb+JOGYFBUOGRQs3+8EhNsk= 2022-09-12T18:59:23.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DMLoeb+JOGYFBUOGRQs3+8EhNsk='} 2022-09-12T18:59:23.837 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'bxdpcD3ezkWGGo.jKpBPub5G7caw-xz'), ('x-amz-request-id', 'tx000004d17cf88f8166e46-00631f818b-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.839 DEBUG:rgw_multi.tests:version2 id=bxdpcD3ezkWGGo.jKpBPub5G7caw-xz 2022-09-12T18:59:23.840 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T18:59:23.841 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T18:59:23.841 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=bxdpcD3ezkWGGo.jKpBPub5G7caw-xz 2022-09-12T18:59:23.842 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=bxdpcD3ezkWGGo.jKpBPub5G7caw-xz 2022-09-12T18:59:23.842 DEBUG:boto:Method: DELETE 2022-09-12T18:59:23.843 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=bxdpcD3ezkWGGo.jKpBPub5G7caw-xz 2022-09-12T18:59:23.843 DEBUG:boto:Data: 2022-09-12T18:59:23.844 DEBUG:boto:Headers: {} 2022-09-12T18:59:23.844 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:59:23.845 DEBUG:boto:Port: 8001 2022-09-12T18:59:23.846 DEBUG:boto:Params: {} 2022-09-12T18:59:23.846 DEBUG:boto:Token: None 2022-09-12T18:59:23.847 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-17/obj-test-zone3?versionId=bxdpcD3ezkWGGo.jKpBPub5G7caw-xz 2022-09-12T18:59:23.848 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6UiE3OmBljRSaS1wTJ7J0NycpIs= 2022-09-12T18:59:23.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6UiE3OmBljRSaS1wTJ7J0NycpIs=', 'Content-Length': '0'} 2022-09-12T18:59:23.861 DEBUG:boto:Response headers: [('x-amz-version-id', 'bxdpcD3ezkWGGo.jKpBPub5G7caw-xz'), ('x-amz-request-id', 'tx000005da40f2d0531d0a7-00631f818b-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.863 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T18:59:23.863 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T18:59:23.864 DEBUG:boto:Method: PUT 2022-09-12T18:59:23.865 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3 2022-09-12T18:59:23.865 DEBUG:boto:Data: 2022-09-12T18:59:23.866 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:23.866 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:59:23.867 DEBUG:boto:Port: 8001 2022-09-12T18:59:23.874 DEBUG:boto:Params: {} 2022-09-12T18:59:23.875 DEBUG:boto:Token: None 2022-09-12T18:59:23.876 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-17/obj-test-zone3 2022-09-12T18:59:23.877 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jZ4u/ylISxWdBRxYNC4XqfO+bv8= 2022-09-12T18:59:23.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jZ4u/ylISxWdBRxYNC4XqfO+bv8='} 2022-09-12T18:59:23.926 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'aLZzT8yTjXI4Xh442KAU-7ssNWr.SAQ'), ('x-amz-request-id', 'tx0000003a7aa4a5a9b5cc9-00631f818b-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.927 DEBUG:rgw_multi.tests:version3 id=aLZzT8yTjXI4Xh442KAU-7ssNWr.SAQ 2022-09-12T18:59:23.928 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T18:59:23.928 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T18:59:23.929 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=aLZzT8yTjXI4Xh442KAU-7ssNWr.SAQ 2022-09-12T18:59:23.929 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=aLZzT8yTjXI4Xh442KAU-7ssNWr.SAQ 2022-09-12T18:59:23.930 DEBUG:boto:Method: DELETE 2022-09-12T18:59:23.931 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=aLZzT8yTjXI4Xh442KAU-7ssNWr.SAQ 2022-09-12T18:59:23.931 DEBUG:boto:Data: 2022-09-12T18:59:23.932 DEBUG:boto:Headers: {} 2022-09-12T18:59:23.932 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:59:23.933 DEBUG:boto:Port: 8001 2022-09-12T18:59:23.933 DEBUG:boto:Params: {} 2022-09-12T18:59:23.934 DEBUG:boto:Token: None 2022-09-12T18:59:23.934 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-17/obj-test-zone3?versionId=aLZzT8yTjXI4Xh442KAU-7ssNWr.SAQ 2022-09-12T18:59:23.935 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:a8n5vHHoby2+biiwoG9eIuLfL1A= 2022-09-12T18:59:23.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:a8n5vHHoby2+biiwoG9eIuLfL1A=', 'Content-Length': '0'} 2022-09-12T18:59:23.947 DEBUG:boto:Response headers: [('x-amz-version-id', 'aLZzT8yTjXI4Xh442KAU-7ssNWr.SAQ'), ('x-amz-request-id', 'tx00000aa7847a9d3732f8a-00631f818b-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.948 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T18:59:23.948 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T18:59:23.949 DEBUG:boto:Method: HEAD 2022-09-12T18:59:23.949 DEBUG:boto:Path: /jdpiak-18/ 2022-09-12T18:59:23.950 DEBUG:boto:Data: 2022-09-12T18:59:23.950 DEBUG:boto:Headers: {} 2022-09-12T18:59:23.951 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:59:23.951 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.952 DEBUG:boto:Params: {} 2022-09-12T18:59:23.952 DEBUG:boto:Token: None 2022-09-12T18:59:23.953 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-18/ 2022-09-12T18:59:23.954 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mHH2JKd6ah0gE20wGoqJPUnRPeE= 2022-09-12T18:59:23.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mHH2JKd6ah0gE20wGoqJPUnRPeE=', 'Content-Length': '0'} 2022-09-12T18:59:23.957 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1d17d6039b9b50b-00631f818b-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.958 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T18:59:23.958 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T18:59:23.959 DEBUG:boto:Method: PUT 2022-09-12T18:59:23.960 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1 2022-09-12T18:59:23.960 DEBUG:boto:Data: 2022-09-12T18:59:23.961 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:23.961 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:59:23.962 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.962 DEBUG:boto:Params: {} 2022-09-12T18:59:23.963 DEBUG:boto:Token: None 2022-09-12T18:59:23.964 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-18/obj-test-zone1 2022-09-12T18:59:23.965 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WaL5gu06nmSk0yBWLQpwSkKmxFQ= 2022-09-12T18:59:23.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WaL5gu06nmSk0yBWLQpwSkKmxFQ='} 2022-09-12T18:59:23.987 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-amz-request-id', 'tx0000063ca871a7baf4bc3-00631f818b-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:23.988 DEBUG:rgw_multi.tests:version1 id=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T18:59:23.989 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T18:59:23.989 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T18:59:23.990 DEBUG:boto:Method: PUT 2022-09-12T18:59:23.991 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1 2022-09-12T18:59:23.991 DEBUG:boto:Data: 2022-09-12T18:59:23.992 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:23.992 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:59:23.992 DEBUG:boto:Port: 8000 2022-09-12T18:59:23.993 DEBUG:boto:Params: {} 2022-09-12T18:59:23.994 DEBUG:boto:Token: None 2022-09-12T18:59:23.994 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 12 Sep 2022 18:59:23 GMT /jdpiak-18/obj-test-zone1 2022-09-12T18:59:23.995 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J8VX1qnrh4LqkCqlgwf67yZ0rXQ= 2022-09-12T18:59:23.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J8VX1qnrh4LqkCqlgwf67yZ0rXQ='} 2022-09-12T18:59:24.008 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ZEtRiJg5VbxtxVEwbewimuWX6xV0x.b'), ('x-amz-request-id', 'tx0000089e5b826e9a58943-00631f818b-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:24.009 DEBUG:rgw_multi.tests:version2 id=ZEtRiJg5VbxtxVEwbewimuWX6xV0x.b 2022-09-12T18:59:24.009 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T18:59:24.010 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T18:59:24.010 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ZEtRiJg5VbxtxVEwbewimuWX6xV0x.b 2022-09-12T18:59:24.011 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ZEtRiJg5VbxtxVEwbewimuWX6xV0x.b 2022-09-12T18:59:24.011 DEBUG:boto:Method: DELETE 2022-09-12T18:59:24.012 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ZEtRiJg5VbxtxVEwbewimuWX6xV0x.b 2022-09-12T18:59:24.012 DEBUG:boto:Data: 2022-09-12T18:59:24.013 DEBUG:boto:Headers: {} 2022-09-12T18:59:24.013 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:59:24.014 DEBUG:boto:Port: 8000 2022-09-12T18:59:24.015 DEBUG:boto:Params: {} 2022-09-12T18:59:24.015 DEBUG:boto:Token: None 2022-09-12T18:59:24.016 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 18:59:24 GMT /jdpiak-18/obj-test-zone1?versionId=ZEtRiJg5VbxtxVEwbewimuWX6xV0x.b 2022-09-12T18:59:24.017 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dDqxb+4UzBQKccNeTFHtq4vMlw4= 2022-09-12T18:59:24.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dDqxb+4UzBQKccNeTFHtq4vMlw4=', 'Content-Length': '0'} 2022-09-12T18:59:24.029 DEBUG:boto:Response headers: [('x-amz-version-id', 'ZEtRiJg5VbxtxVEwbewimuWX6xV0x.b'), ('x-amz-request-id', 'tx00000f943f7df5580d6ec-00631f818c-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:24.030 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T18:59:24.031 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T18:59:24.032 DEBUG:boto:Method: PUT 2022-09-12T18:59:24.032 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1 2022-09-12T18:59:24.033 DEBUG:boto:Data: 2022-09-12T18:59:24.034 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:24.034 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:59:24.035 DEBUG:boto:Port: 8000 2022-09-12T18:59:24.035 DEBUG:boto:Params: {} 2022-09-12T18:59:24.036 DEBUG:boto:Token: None 2022-09-12T18:59:24.036 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 12 Sep 2022 18:59:24 GMT /jdpiak-18/obj-test-zone1 2022-09-12T18:59:24.037 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DtrRvHZ3efyMx5kpRU+rfNBYW1Y= 2022-09-12T18:59:24.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DtrRvHZ3efyMx5kpRU+rfNBYW1Y='} 2022-09-12T18:59:24.086 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '1QYPuNDbSBiK2lKbzlvvdpJej5Z1h82'), ('x-amz-request-id', 'tx000006479439e18a914f9-00631f818c-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:24.088 DEBUG:rgw_multi.tests:version3 id=1QYPuNDbSBiK2lKbzlvvdpJej5Z1h82 2022-09-12T18:59:24.088 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T18:59:24.089 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T18:59:24.089 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=1QYPuNDbSBiK2lKbzlvvdpJej5Z1h82 2022-09-12T18:59:24.090 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=1QYPuNDbSBiK2lKbzlvvdpJej5Z1h82 2022-09-12T18:59:24.090 DEBUG:boto:Method: DELETE 2022-09-12T18:59:24.091 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=1QYPuNDbSBiK2lKbzlvvdpJej5Z1h82 2022-09-12T18:59:24.091 DEBUG:boto:Data: 2022-09-12T18:59:24.092 DEBUG:boto:Headers: {} 2022-09-12T18:59:24.092 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T18:59:24.093 DEBUG:boto:Port: 8000 2022-09-12T18:59:24.093 DEBUG:boto:Params: {} 2022-09-12T18:59:24.094 DEBUG:boto:Token: None 2022-09-12T18:59:24.095 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 18:59:24 GMT /jdpiak-18/obj-test-zone1?versionId=1QYPuNDbSBiK2lKbzlvvdpJej5Z1h82 2022-09-12T18:59:24.095 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZV6+aZaYfo5fZ5RHB4ZRQUpe4xk= 2022-09-12T18:59:24.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZV6+aZaYfo5fZ5RHB4ZRQUpe4xk=', 'Content-Length': '0'} 2022-09-12T18:59:24.107 DEBUG:boto:Response headers: [('x-amz-version-id', '1QYPuNDbSBiK2lKbzlvvdpJej5Z1h82'), ('x-amz-request-id', 'tx000008737773b780e3ff0-00631f818c-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:24.108 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T18:59:24.108 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T18:59:24.109 DEBUG:boto:Method: HEAD 2022-09-12T18:59:24.109 DEBUG:boto:Path: /jdpiak-18/ 2022-09-12T18:59:24.110 DEBUG:boto:Data: 2022-09-12T18:59:24.110 DEBUG:boto:Headers: {} 2022-09-12T18:59:24.111 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:59:24.111 DEBUG:boto:Port: 8000 2022-09-12T18:59:24.112 DEBUG:boto:Params: {} 2022-09-12T18:59:24.112 DEBUG:boto:Token: None 2022-09-12T18:59:24.113 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:59:24 GMT /jdpiak-18/ 2022-09-12T18:59:24.113 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dWcFFpAJ3eIafT3ryHbn+JasUMQ= 2022-09-12T18:59:24.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dWcFFpAJ3eIafT3ryHbn+JasUMQ=', 'Content-Length': '0'} 2022-09-12T18:59:24.117 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd911da1ce1fea96-00631f818c-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:24.117 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T18:59:24.118 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T18:59:24.118 DEBUG:boto:Method: PUT 2022-09-12T18:59:24.119 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2 2022-09-12T18:59:24.119 DEBUG:boto:Data: 2022-09-12T18:59:24.120 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:24.120 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:59:24.121 DEBUG:boto:Port: 8000 2022-09-12T18:59:24.122 DEBUG:boto:Params: {} 2022-09-12T18:59:24.123 DEBUG:boto:Token: None 2022-09-12T18:59:24.124 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 18:59:24 GMT /jdpiak-18/obj-test-zone2 2022-09-12T18:59:24.125 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:w5PYyujI1mcGKz6ocBOv2YaLmIk= 2022-09-12T18:59:24.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:w5PYyujI1mcGKz6ocBOv2YaLmIk='} 2022-09-12T18:59:24.148 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-amz-request-id', 'tx00000f16f98a3e561d531-00631f818c-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:24.149 DEBUG:rgw_multi.tests:version1 id=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T18:59:24.150 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T18:59:24.151 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T18:59:24.152 DEBUG:boto:Method: PUT 2022-09-12T18:59:24.152 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2 2022-09-12T18:59:24.153 DEBUG:boto:Data: 2022-09-12T18:59:24.153 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:24.154 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:59:24.154 DEBUG:boto:Port: 8000 2022-09-12T18:59:24.155 DEBUG:boto:Params: {} 2022-09-12T18:59:24.156 DEBUG:boto:Token: None 2022-09-12T18:59:24.156 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 12 Sep 2022 18:59:24 GMT /jdpiak-18/obj-test-zone2 2022-09-12T18:59:24.157 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IYN/QG85c11jLpS3pkEbr9bpcCs= 2022-09-12T18:59:24.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IYN/QG85c11jLpS3pkEbr9bpcCs='} 2022-09-12T18:59:24.177 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'wwLoZI2hNG8wMquz24Ta7rrC9DLA4u7'), ('x-amz-request-id', 'tx0000093936598159a88da-00631f818c-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:24.178 DEBUG:rgw_multi.tests:version2 id=wwLoZI2hNG8wMquz24Ta7rrC9DLA4u7 2022-09-12T18:59:24.179 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T18:59:24.180 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T18:59:24.181 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=wwLoZI2hNG8wMquz24Ta7rrC9DLA4u7 2022-09-12T18:59:24.181 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=wwLoZI2hNG8wMquz24Ta7rrC9DLA4u7 2022-09-12T18:59:24.182 DEBUG:boto:Method: DELETE 2022-09-12T18:59:24.183 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=wwLoZI2hNG8wMquz24Ta7rrC9DLA4u7 2022-09-12T18:59:24.183 DEBUG:boto:Data: 2022-09-12T18:59:24.184 DEBUG:boto:Headers: {} 2022-09-12T18:59:24.184 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:59:24.185 DEBUG:boto:Port: 8000 2022-09-12T18:59:24.185 DEBUG:boto:Params: {} 2022-09-12T18:59:24.186 DEBUG:boto:Token: None 2022-09-12T18:59:24.186 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 18:59:24 GMT /jdpiak-18/obj-test-zone2?versionId=wwLoZI2hNG8wMquz24Ta7rrC9DLA4u7 2022-09-12T18:59:24.194 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3aaQ2jtl3OB0nGzl6oRwehE+ZFQ= 2022-09-12T18:59:24.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3aaQ2jtl3OB0nGzl6oRwehE+ZFQ=', 'Content-Length': '0'} 2022-09-12T18:59:24.214 DEBUG:boto:Response headers: [('x-amz-version-id', 'wwLoZI2hNG8wMquz24Ta7rrC9DLA4u7'), ('x-amz-request-id', 'tx0000061b679129eb52351-00631f818c-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:24.215 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T18:59:24.216 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T18:59:24.216 DEBUG:boto:Method: PUT 2022-09-12T18:59:24.217 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2 2022-09-12T18:59:24.217 DEBUG:boto:Data: 2022-09-12T18:59:24.218 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:24.219 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:59:24.219 DEBUG:boto:Port: 8000 2022-09-12T18:59:24.220 DEBUG:boto:Params: {} 2022-09-12T18:59:24.220 DEBUG:boto:Token: None 2022-09-12T18:59:24.221 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 12 Sep 2022 18:59:24 GMT /jdpiak-18/obj-test-zone2 2022-09-12T18:59:24.221 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MRxuuf1JJ1mlfdmDJvppVkKTmtA= 2022-09-12T18:59:24.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MRxuuf1JJ1mlfdmDJvppVkKTmtA='} 2022-09-12T18:59:24.270 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Sad2LmB3b7gJ25YPQDet7j9Fv70ePgA'), ('x-amz-request-id', 'tx00000383df530f661b026-00631f818c-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:24.272 DEBUG:rgw_multi.tests:version3 id=Sad2LmB3b7gJ25YPQDet7j9Fv70ePgA 2022-09-12T18:59:24.273 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T18:59:24.274 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T18:59:24.274 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=Sad2LmB3b7gJ25YPQDet7j9Fv70ePgA 2022-09-12T18:59:24.275 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=Sad2LmB3b7gJ25YPQDet7j9Fv70ePgA 2022-09-12T18:59:24.276 DEBUG:boto:Method: DELETE 2022-09-12T18:59:24.278 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=Sad2LmB3b7gJ25YPQDet7j9Fv70ePgA 2022-09-12T18:59:24.278 DEBUG:boto:Data: 2022-09-12T18:59:24.279 DEBUG:boto:Headers: {} 2022-09-12T18:59:24.280 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T18:59:24.280 DEBUG:boto:Port: 8000 2022-09-12T18:59:24.281 DEBUG:boto:Params: {} 2022-09-12T18:59:24.281 DEBUG:boto:Token: None 2022-09-12T18:59:24.282 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 18:59:24 GMT /jdpiak-18/obj-test-zone2?versionId=Sad2LmB3b7gJ25YPQDet7j9Fv70ePgA 2022-09-12T18:59:24.282 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UhzNhasJ5/hzh3jHWRiGzdpq0es= 2022-09-12T18:59:24.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UhzNhasJ5/hzh3jHWRiGzdpq0es=', 'Content-Length': '0'} 2022-09-12T18:59:24.294 DEBUG:boto:Response headers: [('x-amz-version-id', 'Sad2LmB3b7gJ25YPQDet7j9Fv70ePgA'), ('x-amz-request-id', 'tx000000916defb54b73086-00631f818c-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:24.295 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T18:59:24.295 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T18:59:24.296 DEBUG:boto:Method: HEAD 2022-09-12T18:59:24.297 DEBUG:boto:Path: /jdpiak-18/ 2022-09-12T18:59:24.298 DEBUG:boto:Data: 2022-09-12T18:59:24.299 DEBUG:boto:Headers: {} 2022-09-12T18:59:24.300 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:59:24.301 DEBUG:boto:Port: 8001 2022-09-12T18:59:24.301 DEBUG:boto:Params: {} 2022-09-12T18:59:24.303 DEBUG:boto:Token: None 2022-09-12T18:59:24.304 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 18:59:24 GMT /jdpiak-18/ 2022-09-12T18:59:24.305 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dWcFFpAJ3eIafT3ryHbn+JasUMQ= 2022-09-12T18:59:24.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dWcFFpAJ3eIafT3ryHbn+JasUMQ=', 'Content-Length': '0'} 2022-09-12T18:59:24.309 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021ac9e801101f6d6-00631f818c-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:24.309 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T18:59:24.310 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T18:59:24.311 DEBUG:boto:Method: PUT 2022-09-12T18:59:24.311 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3 2022-09-12T18:59:24.312 DEBUG:boto:Data: 2022-09-12T18:59:24.312 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:24.313 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:59:24.313 DEBUG:boto:Port: 8001 2022-09-12T18:59:24.313 DEBUG:boto:Params: {} 2022-09-12T18:59:24.314 DEBUG:boto:Token: None 2022-09-12T18:59:24.315 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 18:59:24 GMT /jdpiak-18/obj-test-zone3 2022-09-12T18:59:24.315 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xXKA2zfKTyiLn9lf1ubo4yNo6QM= 2022-09-12T18:59:24.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xXKA2zfKTyiLn9lf1ubo4yNo6QM='} 2022-09-12T18:59:24.327 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-amz-request-id', 'tx0000019c35c426aaf0ecc-00631f818c-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:24.328 DEBUG:rgw_multi.tests:version1 id=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T18:59:24.328 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T18:59:24.329 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T18:59:24.330 DEBUG:boto:Method: PUT 2022-09-12T18:59:24.330 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3 2022-09-12T18:59:24.331 DEBUG:boto:Data: 2022-09-12T18:59:24.331 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:24.332 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:59:24.332 DEBUG:boto:Port: 8001 2022-09-12T18:59:24.333 DEBUG:boto:Params: {} 2022-09-12T18:59:24.333 DEBUG:boto:Token: None 2022-09-12T18:59:24.334 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Mon, 12 Sep 2022 18:59:24 GMT /jdpiak-18/obj-test-zone3 2022-09-12T18:59:24.334 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tL9+3VG6niogTW+rMzdQjlM1oC4= 2022-09-12T18:59:24.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tL9+3VG6niogTW+rMzdQjlM1oC4='} 2022-09-12T18:59:24.346 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'pQhsOSowFv842Tq2g-h7bdXv5gCmIpE'), ('x-amz-request-id', 'tx000003562a4d468dd1cfb-00631f818c-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:24.347 DEBUG:rgw_multi.tests:version2 id=pQhsOSowFv842Tq2g-h7bdXv5gCmIpE 2022-09-12T18:59:24.347 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T18:59:24.348 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T18:59:24.348 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=pQhsOSowFv842Tq2g-h7bdXv5gCmIpE 2022-09-12T18:59:24.349 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=pQhsOSowFv842Tq2g-h7bdXv5gCmIpE 2022-09-12T18:59:24.349 DEBUG:boto:Method: DELETE 2022-09-12T18:59:24.350 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=pQhsOSowFv842Tq2g-h7bdXv5gCmIpE 2022-09-12T18:59:24.350 DEBUG:boto:Data: 2022-09-12T18:59:24.351 DEBUG:boto:Headers: {} 2022-09-12T18:59:24.351 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:59:24.351 DEBUG:boto:Port: 8001 2022-09-12T18:59:24.352 DEBUG:boto:Params: {} 2022-09-12T18:59:24.353 DEBUG:boto:Token: None 2022-09-12T18:59:24.353 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 18:59:24 GMT /jdpiak-18/obj-test-zone3?versionId=pQhsOSowFv842Tq2g-h7bdXv5gCmIpE 2022-09-12T18:59:24.354 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fUWHGmW84RYV8bAnYdTLS2BVU0g= 2022-09-12T18:59:24.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fUWHGmW84RYV8bAnYdTLS2BVU0g=', 'Content-Length': '0'} 2022-09-12T18:59:24.365 DEBUG:boto:Response headers: [('x-amz-version-id', 'pQhsOSowFv842Tq2g-h7bdXv5gCmIpE'), ('x-amz-request-id', 'tx000006136f9c2bbb465ae-00631f818c-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:24.366 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T18:59:24.367 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T18:59:24.367 DEBUG:boto:Method: PUT 2022-09-12T18:59:24.368 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3 2022-09-12T18:59:24.368 DEBUG:boto:Data: 2022-09-12T18:59:24.369 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T18:59:24.369 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:59:24.370 DEBUG:boto:Port: 8001 2022-09-12T18:59:24.370 DEBUG:boto:Params: {} 2022-09-12T18:59:24.371 DEBUG:boto:Token: None 2022-09-12T18:59:24.371 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Mon, 12 Sep 2022 18:59:24 GMT /jdpiak-18/obj-test-zone3 2022-09-12T18:59:24.372 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aRnGRXgQxQNAPaSvcg9pzGOKASg= 2022-09-12T18:59:24.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 18:59:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aRnGRXgQxQNAPaSvcg9pzGOKASg='} 2022-09-12T18:59:24.418 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'WyDLPS-mrMDuIS1188Q2PIr3WyQuHfq'), ('x-amz-request-id', 'tx0000060ae6d4fb45cf2e4-00631f818c-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:24.419 DEBUG:rgw_multi.tests:version3 id=WyDLPS-mrMDuIS1188Q2PIr3WyQuHfq 2022-09-12T18:59:24.420 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T18:59:24.420 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T18:59:24.421 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=WyDLPS-mrMDuIS1188Q2PIr3WyQuHfq 2022-09-12T18:59:24.421 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=WyDLPS-mrMDuIS1188Q2PIr3WyQuHfq 2022-09-12T18:59:24.422 DEBUG:boto:Method: DELETE 2022-09-12T18:59:24.422 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=WyDLPS-mrMDuIS1188Q2PIr3WyQuHfq 2022-09-12T18:59:24.423 DEBUG:boto:Data: 2022-09-12T18:59:24.423 DEBUG:boto:Headers: {} 2022-09-12T18:59:24.424 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T18:59:24.424 DEBUG:boto:Port: 8001 2022-09-12T18:59:24.425 DEBUG:boto:Params: {} 2022-09-12T18:59:24.425 DEBUG:boto:Token: None 2022-09-12T18:59:24.426 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 18:59:24 GMT /jdpiak-18/obj-test-zone3?versionId=WyDLPS-mrMDuIS1188Q2PIr3WyQuHfq 2022-09-12T18:59:24.426 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:acLZpDPYZFF9YX5mwFqUgYWBoAI= 2022-09-12T18:59:24.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 18:59:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:acLZpDPYZFF9YX5mwFqUgYWBoAI=', 'Content-Length': '0'} 2022-09-12T18:59:24.438 DEBUG:boto:Response headers: [('x-amz-version-id', 'WyDLPS-mrMDuIS1188Q2PIr3WyQuHfq'), ('x-amz-request-id', 'tx00000ef5ca66c19f526e0-00631f818c-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T18:59:24.439 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-16 2022-09-12T18:59:24.439 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:59:24.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:59:24.441 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T18:59:24.505 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:59:24.506 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:59:24.542 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:59:24.539+0000 7f6fcfe52600 1 waiting for incremental sync to catch up: 2022-09-12T18:59:24.543 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000005.13.9, , , , , ] 2022-09-12T18:59:24.543 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000010.20.5, , , , 00000000005.13.9, , , , , ] 2022-09-12T18:59:29.544 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:59:29.541+0000 7f6fcfe52600 1 waiting for incremental sync to catch up: 2022-09-12T18:59:29.545 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000005.13.9, , , , , ] 2022-09-12T18:59:29.545 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000010.20.5, , , , 00000000005.13.9, , , , , ] 2022-09-12T18:59:34.546 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:59:34.542+0000 7f6fcfe52600 1 waiting for incremental sync to catch up: 2022-09-12T18:59:34.547 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000005.13.9, , , , , ] 2022-09-12T18:59:34.547 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000010.20.5, , , , 00000000005.13.9, , , , , ] 2022-09-12T18:59:39.547 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:59:39.544+0000 7f6fcfe52600 1 waiting for incremental sync to catch up: 2022-09-12T18:59:39.548 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000005.13.9, , , , , ] 2022-09-12T18:59:39.548 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000010.20.5, , , , 00000000005.13.9, , , , , ] 2022-09-12T18:59:44.549 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:59:44.545+0000 7f6fcfe52600 1 bucket sync caught up with source: 2022-09-12T18:59:44.549 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000010.20.5, , , , 00000000007.17.10, , , , , ] 2022-09-12T18:59:44.550 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000010.20.5, , , , 00000000005.13.9, , , , , ] 2022-09-12T18:59:44.550 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T18:59:44.545+0000 7f6fcfe52600 0 bucket checkpoint complete 2022-09-12T18:59:44.569 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-16 2022-09-12T18:59:44.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:59:44.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T18:59:44.572 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T18:59:44.638 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T18:59:44.639 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T18:59:44.679 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:59:44.676+0000 7fb5eca29600 1 waiting for incremental sync to catch up: 2022-09-12T18:59:44.679 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000010.20.5, , , , 00000000008.19.9, , , , , ] 2022-09-12T18:59:44.680 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000010.20.5, , 00000000002.7.10, , 00000000008.19.9, , , , , ] 2022-09-12T18:59:49.681 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:59:49.677+0000 7fb5eca29600 1 waiting for incremental sync to catch up: 2022-09-12T18:59:49.681 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000010.20.5, , , , 00000000008.19.9, , , , , ] 2022-09-12T18:59:49.681 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000010.20.5, , 00000000002.7.10, , 00000000008.19.9, , , , , ] 2022-09-12T18:59:54.682 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:59:54.679+0000 7fb5eca29600 1 waiting for incremental sync to catch up: 2022-09-12T18:59:54.683 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000010.20.5, , , , 00000000008.19.9, , , , , ] 2022-09-12T18:59:54.683 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000010.20.5, , 00000000002.7.10, , 00000000008.19.9, , , , , ] 2022-09-12T18:59:59.683 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T18:59:59.680+0000 7fb5eca29600 1 waiting for incremental sync to catch up: 2022-09-12T18:59:59.684 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000010.20.5, , , , 00000000008.19.9, , , , , ] 2022-09-12T18:59:59.684 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000010.20.5, , 00000000002.7.10, , 00000000008.19.9, , , , , ] 2022-09-12T19:00:04.686 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:00:04.682+0000 7fb5eca29600 1 bucket sync caught up with source: 2022-09-12T19:00:04.686 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000010.20.5, , 00000000003.9.9, , 00000000008.19.9, , , , , ] 2022-09-12T19:00:04.686 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000010.20.5, , 00000000002.7.10, , 00000000008.19.9, , , , , ] 2022-09-12T19:00:04.687 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:00:04.682+0000 7fb5eca29600 0 bucket checkpoint complete 2022-09-12T19:00:04.701 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-16 2022-09-12T19:00:04.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:00:04.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:00:04.703 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:00:04.770 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:00:04.770 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:00:04.806 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:00:04.803+0000 7ff921eb2600 1 waiting for incremental sync to catch up: 2022-09-12T19:00:04.807 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000010.20.5, , 00000000002.7.10, , 00000000008.19.9, , , , , ] 2022-09-12T19:00:04.807 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000010.20.5, , 00000000003.9.9, , 00000000008.19.9, , , , , ] 2022-09-12T19:00:09.808 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:00:09.805+0000 7ff921eb2600 1 bucket sync caught up with source: 2022-09-12T19:00:09.809 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000010.20.5, , 00000000003.9.9, , 00000000008.19.9, , , , , ] 2022-09-12T19:00:09.809 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000010.20.5, , 00000000003.9.9, , 00000000008.19.9, , , , , ] 2022-09-12T19:00:09.810 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:00:09.805+0000 7ff921eb2600 0 bucket checkpoint complete 2022-09-12T19:00:09.831 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-16 2022-09-12T19:00:09.832 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:00:09.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:00:09.833 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:00:09.900 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:00:09.901 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:00:09.957 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:00:09.954+0000 7f6179273600 1 bucket sync caught up with source: 2022-09-12T19:00:09.958 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.10.9, , 00000000010.34.5, , 00000000007.26.9, , , , , ] 2022-09-12T19:00:09.958 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.10.9, , 00000000010.34.5, , 00000000007.26.9, , , , , ] 2022-09-12T19:00:09.958 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:00:09.955+0000 7f6179273600 0 bucket checkpoint complete 2022-09-12T19:00:09.981 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-16 2022-09-12T19:00:09.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:00:09.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:00:09.985 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:00:10.043 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:00:10.044 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:00:10.095 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:00:10.093+0000 7ff66d0ee600 1 bucket sync caught up with source: 2022-09-12T19:00:10.095 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.10.9, , 00000000010.34.5, , 00000000007.26.9, , , , , ] 2022-09-12T19:00:10.096 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.10.9, , 00000000010.34.5, , 00000000007.26.9, , , , , ] 2022-09-12T19:00:10.096 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:00:10.093+0000 7ff66d0ee600 0 bucket checkpoint complete 2022-09-12T19:00:10.107 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-16 2022-09-12T19:00:10.108 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-16 2022-09-12T19:00:10.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:00:10.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:00:10.110 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:00:10.167 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:00:10.168 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:00:10.201 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:00:10.199+0000 7f13d9c32600 1 waiting for incremental sync to catch up: 2022-09-12T19:00:10.201 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.13.10, , 00000000002.20.10, , 00000000014.38.5, , , , , ] 2022-09-12T19:00:10.202 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.15.9, , 00000000003.22.9, , 00000000014.38.5, , , , , ] 2022-09-12T19:00:15.203 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:00:15.200+0000 7f13d9c32600 1 waiting for incremental sync to catch up: 2022-09-12T19:00:15.203 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.13.10, , 00000000002.20.10, , 00000000014.38.5, , , , , ] 2022-09-12T19:00:15.204 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.15.9, , 00000000003.22.9, , 00000000014.38.5, , , , , ] 2022-09-12T19:00:20.204 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:00:20.201+0000 7f13d9c32600 1 waiting for incremental sync to catch up: 2022-09-12T19:00:20.205 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.13.10, , 00000000002.20.10, , 00000000014.38.5, , , , , ] 2022-09-12T19:00:20.205 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.15.9, , 00000000003.22.9, , 00000000014.38.5, , , , , ] 2022-09-12T19:00:25.206 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:00:25.203+0000 7f13d9c32600 1 bucket sync caught up with source: 2022-09-12T19:00:25.207 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.15.9, , 00000000003.22.9, , 00000000014.38.5, , , , , ] 2022-09-12T19:00:25.207 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.15.9, , 00000000003.22.9, , 00000000014.38.5, , , , , ] 2022-09-12T19:00:25.207 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:00:25.203+0000 7f13d9c32600 0 bucket checkpoint complete 2022-09-12T19:00:25.223 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-16 2022-09-12T19:00:25.224 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:00:25.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:00:25.225 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:00:25.289 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:00:25.290 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:00:25.331 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:00:25.327+0000 7ff035e75600 1 bucket sync caught up with source: 2022-09-12T19:00:25.331 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000003.15.9, , 00000000003.22.9, , 00000000014.38.5, , , , , ] 2022-09-12T19:00:25.331 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000003.15.9, , 00000000003.22.9, , 00000000014.38.5, , , , , ] 2022-09-12T19:00:25.332 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:00:25.327+0000 7ff035e75600 0 bucket checkpoint complete 2022-09-12T19:00:25.347 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-16 2022-09-12T19:00:25.348 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-16 zones={test-zone2, test-zone1} 2022-09-12T19:00:25.348 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:25.349 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:25.350 DEBUG:boto:Method: HEAD 2022-09-12T19:00:25.351 DEBUG:boto:Path: /jdpiak-16/ 2022-09-12T19:00:25.351 DEBUG:boto:Data: 2022-09-12T19:00:25.352 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.353 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:25.354 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.355 DEBUG:boto:Params: {} 2022-09-12T19:00:25.357 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:00:25.357 DEBUG:boto:Token: None 2022-09-12T19:00:25.358 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/ 2022-09-12T19:00:25.359 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:my6toDSowL71Y/2c0bKxY5l35Z8= 2022-09-12T19:00:25.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:my6toDSowL71Y/2c0bKxY5l35Z8=', 'Content-Length': '0'} 2022-09-12T19:00:25.364 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000279ba9d01afd4d-00631f81c9-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.365 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:25.366 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:25.366 DEBUG:boto:Method: HEAD 2022-09-12T19:00:25.367 DEBUG:boto:Path: /jdpiak-16/ 2022-09-12T19:00:25.367 DEBUG:boto:Data: 2022-09-12T19:00:25.368 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.368 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:25.369 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.369 DEBUG:boto:Params: {} 2022-09-12T19:00:25.370 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:00:25.371 DEBUG:boto:Token: None 2022-09-12T19:00:25.372 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/ 2022-09-12T19:00:25.373 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:my6toDSowL71Y/2c0bKxY5l35Z8= 2022-09-12T19:00:25.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:my6toDSowL71Y/2c0bKxY5l35Z8=', 'Content-Length': '0'} 2022-09-12T19:00:25.378 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de510b5a3c400c30-00631f81c9-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.379 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:00:25.380 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:25.381 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:25.382 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:25.382 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:25.383 DEBUG:boto:Method: GET 2022-09-12T19:00:25.384 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:25.385 DEBUG:boto:Data: 2022-09-12T19:00:25.385 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.386 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:25.387 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.387 DEBUG:boto:Params: {} 2022-09-12T19:00:25.388 DEBUG:boto:Token: None 2022-09-12T19:00:25.389 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/?versions 2022-09-12T19:00:25.390 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aeUDTs+iIhjn4Ys/qc5+6FDjYpY= 2022-09-12T19:00:25.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aeUDTs+iIhjn4Ys/qc5+6FDjYpY=', 'Content-Length': '0'} 2022-09-12T19:00:25.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bb055c21cdb5db7-00631f81c9-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.395 DEBUG:boto:b'jdpiak-16999falsedummydM1h1LlChQwJOE3oHBYl7Qu-tamZZbJtrue2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1-zshRkRrxKU0EJVPpVBUsfYl8JR7HFRtrue2022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone23vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5true2022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eEtrue2022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:25.397 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:00:25.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:00:25.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:00:25.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:00:25.399 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:00:25.400 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:25.400 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:25.401 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:25.401 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:25.402 DEBUG:boto:Method: GET 2022-09-12T19:00:25.402 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:25.403 DEBUG:boto:Data: 2022-09-12T19:00:25.403 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.404 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:25.405 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.405 DEBUG:boto:Params: {} 2022-09-12T19:00:25.406 DEBUG:boto:Token: None 2022-09-12T19:00:25.407 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/?versions 2022-09-12T19:00:25.408 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aeUDTs+iIhjn4Ys/qc5+6FDjYpY= 2022-09-12T19:00:25.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aeUDTs+iIhjn4Ys/qc5+6FDjYpY=', 'Content-Length': '0'} 2022-09-12T19:00:25.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017d6f5f7e4015bca-00631f81c9-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.412 DEBUG:boto:b'jdpiak-16999falsedummydM1h1LlChQwJOE3oHBYl7Qu-tamZZbJtrue2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1-zshRkRrxKU0EJVPpVBUsfYl8JR7HFRtrue2022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone23vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5true2022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eEtrue2022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:25.413 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:00:25.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:00:25.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:00:25.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:00:25.415 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:25.416 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:25.417 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:25.417 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:25.418 DEBUG:boto:Method: GET 2022-09-12T19:00:25.418 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:25.419 DEBUG:boto:Data: 2022-09-12T19:00:25.419 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.420 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:25.420 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.421 DEBUG:boto:Params: {} 2022-09-12T19:00:25.421 DEBUG:boto:Token: None 2022-09-12T19:00:25.422 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/?versions 2022-09-12T19:00:25.423 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aeUDTs+iIhjn4Ys/qc5+6FDjYpY= 2022-09-12T19:00:25.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aeUDTs+iIhjn4Ys/qc5+6FDjYpY=', 'Content-Length': '0'} 2022-09-12T19:00:25.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053fea7b897714195-00631f81c9-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.427 DEBUG:boto:b'jdpiak-16999falsedummydM1h1LlChQwJOE3oHBYl7Qu-tamZZbJtrue2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1-zshRkRrxKU0EJVPpVBUsfYl8JR7HFRtrue2022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone23vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5true2022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eEtrue2022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:25.428 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:25.428 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:25.429 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:25.429 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:25.430 DEBUG:boto:Method: GET 2022-09-12T19:00:25.430 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:25.431 DEBUG:boto:Data: 2022-09-12T19:00:25.431 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.432 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:25.433 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.433 DEBUG:boto:Params: {} 2022-09-12T19:00:25.434 DEBUG:boto:Token: None 2022-09-12T19:00:25.434 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/?versions 2022-09-12T19:00:25.435 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aeUDTs+iIhjn4Ys/qc5+6FDjYpY= 2022-09-12T19:00:25.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aeUDTs+iIhjn4Ys/qc5+6FDjYpY=', 'Content-Length': '0'} 2022-09-12T19:00:25.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f757a5c0d92b7ca2-00631f81c9-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.439 DEBUG:boto:b'jdpiak-16999falsedummydM1h1LlChQwJOE3oHBYl7Qu-tamZZbJtrue2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1-zshRkRrxKU0EJVPpVBUsfYl8JR7HFRtrue2022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone23vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5true2022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eEtrue2022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:25.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:00:25.441 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:25.442 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:25.442 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.443 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.443 DEBUG:boto:Method: GET 2022-09-12T19:00:25.444 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.444 DEBUG:boto:Data: 2022-09-12T19:00:25.445 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.445 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:25.446 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.447 DEBUG:boto:Params: {} 2022-09-12T19:00:25.447 DEBUG:boto:Token: None 2022-09-12T19:00:25.448 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.448 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nG41b+Wu5aezFvrtI/gkE5wMwOQ= 2022-09-12T19:00:25.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nG41b+Wu5aezFvrtI/gkE5wMwOQ=', 'Content-Length': '0'} 2022-09-12T19:00:25.451 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003e097ebe4a9ae17f-00631f81c9-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.452 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:25.453 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:25.453 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.454 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.454 DEBUG:boto:Method: GET 2022-09-12T19:00:25.455 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.456 DEBUG:boto:Data: 2022-09-12T19:00:25.456 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.457 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:25.457 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.458 DEBUG:boto:Params: {} 2022-09-12T19:00:25.459 DEBUG:boto:Token: None 2022-09-12T19:00:25.459 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.460 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nG41b+Wu5aezFvrtI/gkE5wMwOQ= 2022-09-12T19:00:25.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nG41b+Wu5aezFvrtI/gkE5wMwOQ=', 'Content-Length': '0'} 2022-09-12T19:00:25.463 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ebf62efe2f395397-00631f81c9-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.463 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:25.464 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:25.464 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.465 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.466 DEBUG:boto:Method: HEAD 2022-09-12T19:00:25.466 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.467 DEBUG:boto:Data: 2022-09-12T19:00:25.467 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.468 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:25.468 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.469 DEBUG:boto:Params: {} 2022-09-12T19:00:25.469 DEBUG:boto:Token: None 2022-09-12T19:00:25.470 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.471 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ClPQb33AyDsvRosE+9iz2t18Qbs= 2022-09-12T19:00:25.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ClPQb33AyDsvRosE+9iz2t18Qbs=', 'Content-Length': '0'} 2022-09-12T19:00:25.473 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e87d992379af2f5f-00631f81c9-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.474 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:25.475 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:25.476 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.476 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.477 DEBUG:boto:Method: HEAD 2022-09-12T19:00:25.477 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.478 DEBUG:boto:Data: 2022-09-12T19:00:25.478 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.479 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:25.479 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.480 DEBUG:boto:Params: {} 2022-09-12T19:00:25.480 DEBUG:boto:Token: None 2022-09-12T19:00:25.481 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.482 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ClPQb33AyDsvRosE+9iz2t18Qbs= 2022-09-12T19:00:25.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ClPQb33AyDsvRosE+9iz2t18Qbs=', 'Content-Length': '0'} 2022-09-12T19:00:25.484 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000078e02791250ca8ca-00631f81c9-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:00:25.486 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:25.486 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:25.487 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.487 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.488 DEBUG:boto:Method: GET 2022-09-12T19:00:25.488 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.489 DEBUG:boto:Data: 2022-09-12T19:00:25.490 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.490 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:25.491 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.491 DEBUG:boto:Params: {} 2022-09-12T19:00:25.492 DEBUG:boto:Token: None 2022-09-12T19:00:25.493 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.493 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nG41b+Wu5aezFvrtI/gkE5wMwOQ= 2022-09-12T19:00:25.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nG41b+Wu5aezFvrtI/gkE5wMwOQ=', 'Content-Length': '0'} 2022-09-12T19:00:25.496 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e8d1670af4462b20-00631f81c9-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.497 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:25.497 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:25.498 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.498 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.499 DEBUG:boto:Method: GET 2022-09-12T19:00:25.500 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.500 DEBUG:boto:Data: 2022-09-12T19:00:25.501 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.501 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:25.502 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.502 DEBUG:boto:Params: {} 2022-09-12T19:00:25.503 DEBUG:boto:Token: None 2022-09-12T19:00:25.504 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.504 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nG41b+Wu5aezFvrtI/gkE5wMwOQ= 2022-09-12T19:00:25.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nG41b+Wu5aezFvrtI/gkE5wMwOQ=', 'Content-Length': '0'} 2022-09-12T19:00:25.507 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002b0d57e845d7d084-00631f81c9-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.508 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:25.508 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:25.509 DEBUG:boto:path=/jdpiak-16/?prefix=dummy 2022-09-12T19:00:25.509 DEBUG:boto:auth_path=/jdpiak-16/?prefix=dummy 2022-09-12T19:00:25.510 DEBUG:boto:Method: GET 2022-09-12T19:00:25.510 DEBUG:boto:Path: /jdpiak-16/?prefix=dummy 2022-09-12T19:00:25.511 DEBUG:boto:Data: 2022-09-12T19:00:25.512 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.512 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:25.513 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.513 DEBUG:boto:Params: {} 2022-09-12T19:00:25.514 DEBUG:boto:Token: None 2022-09-12T19:00:25.514 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/ 2022-09-12T19:00:25.515 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ama54L3h2PgaGUT3NBdD89yLtTY= 2022-09-12T19:00:25.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ama54L3h2PgaGUT3NBdD89yLtTY=', 'Content-Length': '0'} 2022-09-12T19:00:25.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a41c6086f59bb18c-00631f81c9-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.519 DEBUG:boto:b'jdpiak-16dummy1000falsedummy2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:25.520 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:25.521 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:25.522 DEBUG:boto:path=/jdpiak-16/?prefix=dummy 2022-09-12T19:00:25.522 DEBUG:boto:auth_path=/jdpiak-16/?prefix=dummy 2022-09-12T19:00:25.523 DEBUG:boto:Method: GET 2022-09-12T19:00:25.523 DEBUG:boto:Path: /jdpiak-16/?prefix=dummy 2022-09-12T19:00:25.524 DEBUG:boto:Data: 2022-09-12T19:00:25.524 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.525 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:25.525 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.526 DEBUG:boto:Params: {} 2022-09-12T19:00:25.527 DEBUG:boto:Token: None 2022-09-12T19:00:25.527 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/ 2022-09-12T19:00:25.528 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ama54L3h2PgaGUT3NBdD89yLtTY= 2022-09-12T19:00:25.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ama54L3h2PgaGUT3NBdD89yLtTY=', 'Content-Length': '0'} 2022-09-12T19:00:25.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d5e2f518f70b16b-00631f81c9-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.574 DEBUG:boto:b'jdpiak-16dummy1000falsedummy2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:25.575 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:25.576 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:25.577 DEBUG:boto:Method: HEAD 2022-09-12T19:00:25.577 DEBUG:boto:Path: /jdpiak-16/dummy 2022-09-12T19:00:25.578 DEBUG:boto:Data: 2022-09-12T19:00:25.578 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.579 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:25.579 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.580 DEBUG:boto:Params: {} 2022-09-12T19:00:25.580 DEBUG:boto:Token: None 2022-09-12T19:00:25.581 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/dummy 2022-09-12T19:00:25.582 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FIci0Go0FTCFeN4zQrt2vcu341c= 2022-09-12T19:00:25.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FIci0Go0FTCFeN4zQrt2vcu341c=', 'Content-Length': '0'} 2022-09-12T19:00:25.585 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002ab932e59e94d5f5-00631f81c9-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.586 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:25.587 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:25.588 DEBUG:boto:Method: HEAD 2022-09-12T19:00:25.589 DEBUG:boto:Path: /jdpiak-16/dummy 2022-09-12T19:00:25.590 DEBUG:boto:Data: 2022-09-12T19:00:25.591 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.591 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:25.592 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.593 DEBUG:boto:Params: {} 2022-09-12T19:00:25.593 DEBUG:boto:Token: None 2022-09-12T19:00:25.594 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/dummy 2022-09-12T19:00:25.595 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FIci0Go0FTCFeN4zQrt2vcu341c= 2022-09-12T19:00:25.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FIci0Go0FTCFeN4zQrt2vcu341c=', 'Content-Length': '0'} 2022-09-12T19:00:25.598 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f11b2b0718e2ce78-00631f81c9-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:00:25.599 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:25.600 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:25.601 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.601 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.602 DEBUG:boto:Method: GET 2022-09-12T19:00:25.602 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.603 DEBUG:boto:Data: 2022-09-12T19:00:25.604 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.604 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:25.605 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.606 DEBUG:boto:Params: {} 2022-09-12T19:00:25.607 DEBUG:boto:Token: None 2022-09-12T19:00:25.608 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.609 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nG41b+Wu5aezFvrtI/gkE5wMwOQ= 2022-09-12T19:00:25.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nG41b+Wu5aezFvrtI/gkE5wMwOQ=', 'Content-Length': '0'} 2022-09-12T19:00:25.612 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009106d96b6d3f3be0-00631f81c9-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.613 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:25.613 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:25.614 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.614 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.615 DEBUG:boto:Method: GET 2022-09-12T19:00:25.616 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.616 DEBUG:boto:Data: 2022-09-12T19:00:25.617 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.618 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:25.618 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.619 DEBUG:boto:Params: {} 2022-09-12T19:00:25.619 DEBUG:boto:Token: None 2022-09-12T19:00:25.620 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:25.621 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nG41b+Wu5aezFvrtI/gkE5wMwOQ= 2022-09-12T19:00:25.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nG41b+Wu5aezFvrtI/gkE5wMwOQ=', 'Content-Length': '0'} 2022-09-12T19:00:25.623 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007a725482664cfb10-00631f81c9-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.624 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:25.625 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:25.625 DEBUG:boto:path=/jdpiak-16/?prefix=dummy 2022-09-12T19:00:25.626 DEBUG:boto:auth_path=/jdpiak-16/?prefix=dummy 2022-09-12T19:00:25.626 DEBUG:boto:Method: GET 2022-09-12T19:00:25.627 DEBUG:boto:Path: /jdpiak-16/?prefix=dummy 2022-09-12T19:00:25.628 DEBUG:boto:Data: 2022-09-12T19:00:25.628 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.629 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:25.629 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.630 DEBUG:boto:Params: {} 2022-09-12T19:00:25.630 DEBUG:boto:Token: None 2022-09-12T19:00:25.631 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/ 2022-09-12T19:00:25.632 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ama54L3h2PgaGUT3NBdD89yLtTY= 2022-09-12T19:00:25.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ama54L3h2PgaGUT3NBdD89yLtTY=', 'Content-Length': '0'} 2022-09-12T19:00:25.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033f1812018e36bd3-00631f81c9-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.635 DEBUG:boto:b'jdpiak-16dummy1000falsedummy2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:25.636 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:25.637 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:25.637 DEBUG:boto:path=/jdpiak-16/?prefix=dummy 2022-09-12T19:00:25.638 DEBUG:boto:auth_path=/jdpiak-16/?prefix=dummy 2022-09-12T19:00:25.638 DEBUG:boto:Method: GET 2022-09-12T19:00:25.639 DEBUG:boto:Path: /jdpiak-16/?prefix=dummy 2022-09-12T19:00:25.640 DEBUG:boto:Data: 2022-09-12T19:00:25.640 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.641 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:25.641 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.642 DEBUG:boto:Params: {} 2022-09-12T19:00:25.642 DEBUG:boto:Token: None 2022-09-12T19:00:25.643 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/ 2022-09-12T19:00:25.643 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ama54L3h2PgaGUT3NBdD89yLtTY= 2022-09-12T19:00:25.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ama54L3h2PgaGUT3NBdD89yLtTY=', 'Content-Length': '0'} 2022-09-12T19:00:25.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcdf201685ee3d0c-00631f81c9-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.647 DEBUG:boto:b'jdpiak-16dummy1000falsedummy2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:25.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:00:25.649 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:25.649 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:25.650 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.650 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.651 DEBUG:boto:Method: GET 2022-09-12T19:00:25.652 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.652 DEBUG:boto:Data: 2022-09-12T19:00:25.653 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.653 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:25.654 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.654 DEBUG:boto:Params: {} 2022-09-12T19:00:25.655 DEBUG:boto:Token: None 2022-09-12T19:00:25.655 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.656 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4AvH0HJs0BWRzWicgdZYoRQZcmQ= 2022-09-12T19:00:25.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4AvH0HJs0BWRzWicgdZYoRQZcmQ=', 'Content-Length': '0'} 2022-09-12T19:00:25.660 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd25db83ac49216d-00631f81c9-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.661 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:25.662 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:25.663 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.663 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.664 DEBUG:boto:Method: GET 2022-09-12T19:00:25.665 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.665 DEBUG:boto:Data: 2022-09-12T19:00:25.666 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.666 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:25.667 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.667 DEBUG:boto:Params: {} 2022-09-12T19:00:25.668 DEBUG:boto:Token: None 2022-09-12T19:00:25.669 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.669 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4AvH0HJs0BWRzWicgdZYoRQZcmQ= 2022-09-12T19:00:25.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4AvH0HJs0BWRzWicgdZYoRQZcmQ=', 'Content-Length': '0'} 2022-09-12T19:00:25.672 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053596cb3da7ae41f-00631f81c9-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.714 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:25.715 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:25.715 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.716 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.716 DEBUG:boto:Method: HEAD 2022-09-12T19:00:25.717 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.717 DEBUG:boto:Data: 2022-09-12T19:00:25.718 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.719 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:25.719 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.720 DEBUG:boto:Params: {} 2022-09-12T19:00:25.721 DEBUG:boto:Token: None 2022-09-12T19:00:25.721 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.722 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lMH8KVO7PK4y6xNZSFxCOLmygnw= 2022-09-12T19:00:25.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lMH8KVO7PK4y6xNZSFxCOLmygnw=', 'Content-Length': '0'} 2022-09-12T19:00:25.725 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001006fd8935c2b6b0-00631f81c9-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.725 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:25.726 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:25.727 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.727 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.728 DEBUG:boto:Method: HEAD 2022-09-12T19:00:25.729 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.729 DEBUG:boto:Data: 2022-09-12T19:00:25.730 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.730 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:25.731 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.731 DEBUG:boto:Params: {} 2022-09-12T19:00:25.732 DEBUG:boto:Token: None 2022-09-12T19:00:25.733 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.733 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lMH8KVO7PK4y6xNZSFxCOLmygnw= 2022-09-12T19:00:25.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lMH8KVO7PK4y6xNZSFxCOLmygnw=', 'Content-Length': '0'} 2022-09-12T19:00:25.736 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003921816a28e9a62f-00631f81c9-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:00:25.738 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:25.738 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:25.739 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.739 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.740 DEBUG:boto:Method: GET 2022-09-12T19:00:25.740 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.741 DEBUG:boto:Data: 2022-09-12T19:00:25.741 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.742 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:25.743 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.743 DEBUG:boto:Params: {} 2022-09-12T19:00:25.744 DEBUG:boto:Token: None 2022-09-12T19:00:25.745 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.745 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4AvH0HJs0BWRzWicgdZYoRQZcmQ= 2022-09-12T19:00:25.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4AvH0HJs0BWRzWicgdZYoRQZcmQ=', 'Content-Length': '0'} 2022-09-12T19:00:25.748 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000edd1ab91896ad2a2-00631f81c9-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.749 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:25.749 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:25.750 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.751 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.751 DEBUG:boto:Method: GET 2022-09-12T19:00:25.752 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.753 DEBUG:boto:Data: 2022-09-12T19:00:25.754 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.755 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:25.755 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.756 DEBUG:boto:Params: {} 2022-09-12T19:00:25.757 DEBUG:boto:Token: None 2022-09-12T19:00:25.758 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.758 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4AvH0HJs0BWRzWicgdZYoRQZcmQ= 2022-09-12T19:00:25.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4AvH0HJs0BWRzWicgdZYoRQZcmQ=', 'Content-Length': '0'} 2022-09-12T19:00:25.761 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007fe117e5264d2441-00631f81c9-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.762 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:25.763 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:25.764 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:25.764 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:25.765 DEBUG:boto:Method: GET 2022-09-12T19:00:25.765 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:25.766 DEBUG:boto:Data: 2022-09-12T19:00:25.766 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.767 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:25.768 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.768 DEBUG:boto:Params: {} 2022-09-12T19:00:25.769 DEBUG:boto:Token: None 2022-09-12T19:00:25.769 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/ 2022-09-12T19:00:25.770 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ama54L3h2PgaGUT3NBdD89yLtTY= 2022-09-12T19:00:25.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ama54L3h2PgaGUT3NBdD89yLtTY=', 'Content-Length': '0'} 2022-09-12T19:00:25.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080d5d29ea8984d02-00631f81c9-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.774 DEBUG:boto:b'jdpiak-16obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:25.775 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:25.776 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:25.776 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:25.777 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:25.777 DEBUG:boto:Method: GET 2022-09-12T19:00:25.778 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:25.779 DEBUG:boto:Data: 2022-09-12T19:00:25.779 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.780 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:25.781 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.781 DEBUG:boto:Params: {} 2022-09-12T19:00:25.782 DEBUG:boto:Token: None 2022-09-12T19:00:25.782 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/ 2022-09-12T19:00:25.783 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ama54L3h2PgaGUT3NBdD89yLtTY= 2022-09-12T19:00:25.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ama54L3h2PgaGUT3NBdD89yLtTY=', 'Content-Length': '0'} 2022-09-12T19:00:25.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000361ec521e3c61ca3-00631f81c9-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.787 DEBUG:boto:b'jdpiak-16obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:25.788 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:25.789 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:25.789 DEBUG:boto:Method: HEAD 2022-09-12T19:00:25.790 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1 2022-09-12T19:00:25.791 DEBUG:boto:Data: 2022-09-12T19:00:25.792 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.792 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:25.793 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.794 DEBUG:boto:Params: {} 2022-09-12T19:00:25.795 DEBUG:boto:Token: None 2022-09-12T19:00:25.796 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/obj-test-zone1 2022-09-12T19:00:25.796 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yL6cbiGQg/SAhY7o3tbaVYVsYP0= 2022-09-12T19:00:25.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yL6cbiGQg/SAhY7o3tbaVYVsYP0=', 'Content-Length': '0'} 2022-09-12T19:00:25.799 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003cca67f2a5b426e5-00631f81c9-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.800 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:25.801 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:25.801 DEBUG:boto:Method: HEAD 2022-09-12T19:00:25.802 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1 2022-09-12T19:00:25.802 DEBUG:boto:Data: 2022-09-12T19:00:25.803 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.804 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:25.804 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.805 DEBUG:boto:Params: {} 2022-09-12T19:00:25.805 DEBUG:boto:Token: None 2022-09-12T19:00:25.806 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/obj-test-zone1 2022-09-12T19:00:25.807 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yL6cbiGQg/SAhY7o3tbaVYVsYP0= 2022-09-12T19:00:25.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yL6cbiGQg/SAhY7o3tbaVYVsYP0=', 'Content-Length': '0'} 2022-09-12T19:00:25.810 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060174bf8b29dca23-00631f81c9-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:00:25.811 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:25.812 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:25.813 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.813 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.814 DEBUG:boto:Method: GET 2022-09-12T19:00:25.815 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.816 DEBUG:boto:Data: 2022-09-12T19:00:25.816 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.817 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:25.817 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.818 DEBUG:boto:Params: {} 2022-09-12T19:00:25.819 DEBUG:boto:Token: None 2022-09-12T19:00:25.819 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.820 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4AvH0HJs0BWRzWicgdZYoRQZcmQ= 2022-09-12T19:00:25.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4AvH0HJs0BWRzWicgdZYoRQZcmQ=', 'Content-Length': '0'} 2022-09-12T19:00:25.823 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093ab25ff43ed1d5e-00631f81c9-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.824 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:25.825 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:25.826 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.827 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.828 DEBUG:boto:Method: GET 2022-09-12T19:00:25.829 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.830 DEBUG:boto:Data: 2022-09-12T19:00:25.831 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.832 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:25.833 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.834 DEBUG:boto:Params: {} 2022-09-12T19:00:25.834 DEBUG:boto:Token: None 2022-09-12T19:00:25.835 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:25.836 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4AvH0HJs0BWRzWicgdZYoRQZcmQ= 2022-09-12T19:00:25.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4AvH0HJs0BWRzWicgdZYoRQZcmQ=', 'Content-Length': '0'} 2022-09-12T19:00:25.839 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9b1acd018ce3661-00631f81c9-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.882 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:25.883 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:25.883 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:25.884 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:25.885 DEBUG:boto:Method: GET 2022-09-12T19:00:25.886 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:25.887 DEBUG:boto:Data: 2022-09-12T19:00:25.888 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.888 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:25.889 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.889 DEBUG:boto:Params: {} 2022-09-12T19:00:25.891 DEBUG:boto:Token: None 2022-09-12T19:00:25.892 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/ 2022-09-12T19:00:25.893 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ama54L3h2PgaGUT3NBdD89yLtTY= 2022-09-12T19:00:25.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ama54L3h2PgaGUT3NBdD89yLtTY=', 'Content-Length': '0'} 2022-09-12T19:00:25.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c03c18b5520718fd-00631f81c9-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.900 DEBUG:boto:b'jdpiak-16obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:25.902 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:25.903 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:25.904 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:25.905 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:25.906 DEBUG:boto:Method: GET 2022-09-12T19:00:25.907 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:25.909 DEBUG:boto:Data: 2022-09-12T19:00:25.909 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.910 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:25.910 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.911 DEBUG:boto:Params: {} 2022-09-12T19:00:25.911 DEBUG:boto:Token: None 2022-09-12T19:00:25.912 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/ 2022-09-12T19:00:25.913 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ama54L3h2PgaGUT3NBdD89yLtTY= 2022-09-12T19:00:25.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ama54L3h2PgaGUT3NBdD89yLtTY=', 'Content-Length': '0'} 2022-09-12T19:00:25.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000967b9ab23fe674ff-00631f81c9-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.917 DEBUG:boto:b'jdpiak-16obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:25.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:00:25.919 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:25.920 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:25.921 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:25.921 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:25.922 DEBUG:boto:Method: GET 2022-09-12T19:00:25.922 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:25.923 DEBUG:boto:Data: 2022-09-12T19:00:25.923 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.924 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:25.924 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.925 DEBUG:boto:Params: {} 2022-09-12T19:00:25.926 DEBUG:boto:Token: None 2022-09-12T19:00:25.927 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:25.928 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DgxbmdLaI+LRtF2LmKChvKbMvGI= 2022-09-12T19:00:25.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DgxbmdLaI+LRtF2LmKChvKbMvGI=', 'Content-Length': '0'} 2022-09-12T19:00:25.932 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a337b3d6af37b65-00631f81c9-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.933 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:25.933 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:25.934 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:25.934 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:25.935 DEBUG:boto:Method: GET 2022-09-12T19:00:25.936 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:25.937 DEBUG:boto:Data: 2022-09-12T19:00:25.937 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.938 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:25.938 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.939 DEBUG:boto:Params: {} 2022-09-12T19:00:25.939 DEBUG:boto:Token: None 2022-09-12T19:00:25.940 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:25.941 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DgxbmdLaI+LRtF2LmKChvKbMvGI= 2022-09-12T19:00:25.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DgxbmdLaI+LRtF2LmKChvKbMvGI=', 'Content-Length': '0'} 2022-09-12T19:00:25.943 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000098fec466982ab56-00631f81c9-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.944 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:25.945 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:25.946 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:25.946 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:25.947 DEBUG:boto:Method: HEAD 2022-09-12T19:00:25.947 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:25.948 DEBUG:boto:Data: 2022-09-12T19:00:25.948 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.949 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:25.949 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.950 DEBUG:boto:Params: {} 2022-09-12T19:00:25.951 DEBUG:boto:Token: None 2022-09-12T19:00:25.951 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:25.952 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GVXLpBrzDFtDonDw4MERTkJE9v4= 2022-09-12T19:00:25.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GVXLpBrzDFtDonDw4MERTkJE9v4=', 'Content-Length': '0'} 2022-09-12T19:00:25.954 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000306864d269c5914c-00631f81c9-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.955 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:25.956 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:25.956 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:25.957 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:25.957 DEBUG:boto:Method: HEAD 2022-09-12T19:00:25.958 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:25.958 DEBUG:boto:Data: 2022-09-12T19:00:25.959 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.959 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:25.960 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.960 DEBUG:boto:Params: {} 2022-09-12T19:00:25.961 DEBUG:boto:Token: None 2022-09-12T19:00:25.961 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:25.962 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GVXLpBrzDFtDonDw4MERTkJE9v4= 2022-09-12T19:00:25.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GVXLpBrzDFtDonDw4MERTkJE9v4=', 'Content-Length': '0'} 2022-09-12T19:00:25.965 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000195ea847457996d7-00631f81c9-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:25.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:00:25.966 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:25.967 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:25.967 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:25.968 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:25.968 DEBUG:boto:Method: GET 2022-09-12T19:00:25.969 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:25.969 DEBUG:boto:Data: 2022-09-12T19:00:25.970 DEBUG:boto:Headers: {} 2022-09-12T19:00:25.970 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:25.971 DEBUG:boto:Port: 8000 2022-09-12T19:00:25.972 DEBUG:boto:Params: {} 2022-09-12T19:00:25.972 DEBUG:boto:Token: None 2022-09-12T19:00:25.973 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:25 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:25.973 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DgxbmdLaI+LRtF2LmKChvKbMvGI= 2022-09-12T19:00:25.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DgxbmdLaI+LRtF2LmKChvKbMvGI=', 'Content-Length': '0'} 2022-09-12T19:00:25.976 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cfbe964393ff5f97-00631f81c9-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.018 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:26.019 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:26.020 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:26.020 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:26.021 DEBUG:boto:Method: GET 2022-09-12T19:00:26.022 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:26.024 DEBUG:boto:Data: 2022-09-12T19:00:26.024 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.025 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.025 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.026 DEBUG:boto:Params: {} 2022-09-12T19:00:26.027 DEBUG:boto:Token: None 2022-09-12T19:00:26.028 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:26.030 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hjhORhUSr6d5cKMFETLUUBW7X2E= 2022-09-12T19:00:26.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hjhORhUSr6d5cKMFETLUUBW7X2E=', 'Content-Length': '0'} 2022-09-12T19:00:26.034 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b8be54c0cc8970f-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.035 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:26.035 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:26.036 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:26.037 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:26.037 DEBUG:boto:Method: GET 2022-09-12T19:00:26.038 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:26.039 DEBUG:boto:Data: 2022-09-12T19:00:26.040 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.041 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:26.042 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.042 DEBUG:boto:Params: {} 2022-09-12T19:00:26.043 DEBUG:boto:Token: None 2022-09-12T19:00:26.044 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/ 2022-09-12T19:00:26.045 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM= 2022-09-12T19:00:26.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM=', 'Content-Length': '0'} 2022-09-12T19:00:26.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fd9ecc419281528-00631f81ca-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.049 DEBUG:boto:b'jdpiak-16obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:26.050 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:26.051 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:26.051 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:26.052 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:26.052 DEBUG:boto:Method: GET 2022-09-12T19:00:26.053 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:26.053 DEBUG:boto:Data: 2022-09-12T19:00:26.054 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.055 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.055 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.056 DEBUG:boto:Params: {} 2022-09-12T19:00:26.056 DEBUG:boto:Token: None 2022-09-12T19:00:26.057 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/ 2022-09-12T19:00:26.058 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM= 2022-09-12T19:00:26.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM=', 'Content-Length': '0'} 2022-09-12T19:00:26.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c91ed9d5ec66d8a5-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.063 DEBUG:boto:b'jdpiak-16obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:26.064 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:26.065 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:26.066 DEBUG:boto:Method: HEAD 2022-09-12T19:00:26.066 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2 2022-09-12T19:00:26.067 DEBUG:boto:Data: 2022-09-12T19:00:26.067 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.068 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:26.068 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.069 DEBUG:boto:Params: {} 2022-09-12T19:00:26.070 DEBUG:boto:Token: None 2022-09-12T19:00:26.071 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/obj-test-zone2 2022-09-12T19:00:26.071 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Lz8k4NC1IA486VVL/2hQomlNgkU= 2022-09-12T19:00:26.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Lz8k4NC1IA486VVL/2hQomlNgkU=', 'Content-Length': '0'} 2022-09-12T19:00:26.075 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000234d73f1d6b8bfd0-00631f81ca-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.076 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:26.077 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:26.077 DEBUG:boto:Method: HEAD 2022-09-12T19:00:26.078 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2 2022-09-12T19:00:26.079 DEBUG:boto:Data: 2022-09-12T19:00:26.079 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.080 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.080 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.081 DEBUG:boto:Params: {} 2022-09-12T19:00:26.082 DEBUG:boto:Token: None 2022-09-12T19:00:26.082 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/obj-test-zone2 2022-09-12T19:00:26.083 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Lz8k4NC1IA486VVL/2hQomlNgkU= 2022-09-12T19:00:26.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Lz8k4NC1IA486VVL/2hQomlNgkU=', 'Content-Length': '0'} 2022-09-12T19:00:26.088 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd4a4dec3d5d5b37-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:00:26.089 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:26.090 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:26.090 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:26.091 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:26.091 DEBUG:boto:Method: GET 2022-09-12T19:00:26.092 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:26.092 DEBUG:boto:Data: 2022-09-12T19:00:26.093 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.094 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:26.094 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.095 DEBUG:boto:Params: {} 2022-09-12T19:00:26.095 DEBUG:boto:Token: None 2022-09-12T19:00:26.096 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:26.097 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hjhORhUSr6d5cKMFETLUUBW7X2E= 2022-09-12T19:00:26.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hjhORhUSr6d5cKMFETLUUBW7X2E=', 'Content-Length': '0'} 2022-09-12T19:00:26.099 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c32c5ec51f558ca1-00631f81ca-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.100 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:26.101 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:26.102 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:26.102 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:26.103 DEBUG:boto:Method: GET 2022-09-12T19:00:26.103 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:26.104 DEBUG:boto:Data: 2022-09-12T19:00:26.104 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.105 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.105 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.106 DEBUG:boto:Params: {} 2022-09-12T19:00:26.107 DEBUG:boto:Token: None 2022-09-12T19:00:26.107 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:26.108 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hjhORhUSr6d5cKMFETLUUBW7X2E= 2022-09-12T19:00:26.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hjhORhUSr6d5cKMFETLUUBW7X2E=', 'Content-Length': '0'} 2022-09-12T19:00:26.111 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9f7d412b32c2480-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.155 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:26.156 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:26.157 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:26.157 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:26.158 DEBUG:boto:Method: GET 2022-09-12T19:00:26.158 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:26.159 DEBUG:boto:Data: 2022-09-12T19:00:26.159 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.160 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:26.161 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.161 DEBUG:boto:Params: {} 2022-09-12T19:00:26.162 DEBUG:boto:Token: None 2022-09-12T19:00:26.162 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/ 2022-09-12T19:00:26.163 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM= 2022-09-12T19:00:26.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM=', 'Content-Length': '0'} 2022-09-12T19:00:26.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb4e55a9ba7e950d-00631f81ca-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.167 DEBUG:boto:b'jdpiak-16obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:26.168 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:26.168 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:26.169 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:26.169 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:26.170 DEBUG:boto:Method: GET 2022-09-12T19:00:26.170 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:26.171 DEBUG:boto:Data: 2022-09-12T19:00:26.173 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.173 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.174 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.174 DEBUG:boto:Params: {} 2022-09-12T19:00:26.175 DEBUG:boto:Token: None 2022-09-12T19:00:26.175 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/ 2022-09-12T19:00:26.176 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM= 2022-09-12T19:00:26.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM=', 'Content-Length': '0'} 2022-09-12T19:00:26.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fe559bff5398374-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.180 DEBUG:boto:b'jdpiak-16obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:26.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:00:26.182 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:26.183 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:26.183 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.184 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.184 DEBUG:boto:Method: GET 2022-09-12T19:00:26.185 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.185 DEBUG:boto:Data: 2022-09-12T19:00:26.186 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.186 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:26.187 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.187 DEBUG:boto:Params: {} 2022-09-12T19:00:26.188 DEBUG:boto:Token: None 2022-09-12T19:00:26.189 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.189 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AKOntJgq5YXHlU3VtdDEJJDKlEo= 2022-09-12T19:00:26.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AKOntJgq5YXHlU3VtdDEJJDKlEo=', 'Content-Length': '0'} 2022-09-12T19:00:26.192 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009859a727ea763160-00631f81ca-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.234 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:26.235 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:26.235 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.236 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.236 DEBUG:boto:Method: GET 2022-09-12T19:00:26.237 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.238 DEBUG:boto:Data: 2022-09-12T19:00:26.238 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.239 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.239 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.240 DEBUG:boto:Params: {} 2022-09-12T19:00:26.241 DEBUG:boto:Token: None 2022-09-12T19:00:26.241 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.242 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AKOntJgq5YXHlU3VtdDEJJDKlEo= 2022-09-12T19:00:26.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AKOntJgq5YXHlU3VtdDEJJDKlEo=', 'Content-Length': '0'} 2022-09-12T19:00:26.245 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e52bda852f82f6a6-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.246 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:26.246 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:26.247 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.247 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.248 DEBUG:boto:Method: HEAD 2022-09-12T19:00:26.248 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.249 DEBUG:boto:Data: 2022-09-12T19:00:26.249 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.250 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:26.250 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.251 DEBUG:boto:Params: {} 2022-09-12T19:00:26.251 DEBUG:boto:Token: None 2022-09-12T19:00:26.252 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.252 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vYSRJacm/3fELfsh9x9ac6MDgZo= 2022-09-12T19:00:26.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vYSRJacm/3fELfsh9x9ac6MDgZo=', 'Content-Length': '0'} 2022-09-12T19:00:26.255 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1f5776d8d1451e1-00631f81ca-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.256 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:26.256 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:26.257 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.257 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.258 DEBUG:boto:Method: HEAD 2022-09-12T19:00:26.259 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.259 DEBUG:boto:Data: 2022-09-12T19:00:26.260 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.260 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.261 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.261 DEBUG:boto:Params: {} 2022-09-12T19:00:26.262 DEBUG:boto:Token: None 2022-09-12T19:00:26.262 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.263 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vYSRJacm/3fELfsh9x9ac6MDgZo= 2022-09-12T19:00:26.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vYSRJacm/3fELfsh9x9ac6MDgZo=', 'Content-Length': '0'} 2022-09-12T19:00:26.266 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000904a4d333afdc949-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:00:26.267 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:26.268 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:26.268 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.269 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.269 DEBUG:boto:Method: GET 2022-09-12T19:00:26.270 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.270 DEBUG:boto:Data: 2022-09-12T19:00:26.271 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.271 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:26.272 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.272 DEBUG:boto:Params: {} 2022-09-12T19:00:26.273 DEBUG:boto:Token: None 2022-09-12T19:00:26.274 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.274 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AKOntJgq5YXHlU3VtdDEJJDKlEo= 2022-09-12T19:00:26.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AKOntJgq5YXHlU3VtdDEJJDKlEo=', 'Content-Length': '0'} 2022-09-12T19:00:26.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006db5a6c8aadc883b-00631f81ca-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.278 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:26.278 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:26.279 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.279 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.280 DEBUG:boto:Method: GET 2022-09-12T19:00:26.281 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.281 DEBUG:boto:Data: 2022-09-12T19:00:26.282 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.282 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.283 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.283 DEBUG:boto:Params: {} 2022-09-12T19:00:26.284 DEBUG:boto:Token: None 2022-09-12T19:00:26.285 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.285 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AKOntJgq5YXHlU3VtdDEJJDKlEo= 2022-09-12T19:00:26.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AKOntJgq5YXHlU3VtdDEJJDKlEo=', 'Content-Length': '0'} 2022-09-12T19:00:26.288 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020758e5589d189ce-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.289 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:26.289 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:26.290 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:26.290 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:26.291 DEBUG:boto:Method: GET 2022-09-12T19:00:26.292 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:26.292 DEBUG:boto:Data: 2022-09-12T19:00:26.293 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.293 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:26.294 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.294 DEBUG:boto:Params: {} 2022-09-12T19:00:26.295 DEBUG:boto:Token: None 2022-09-12T19:00:26.295 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/ 2022-09-12T19:00:26.296 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM= 2022-09-12T19:00:26.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM=', 'Content-Length': '0'} 2022-09-12T19:00:26.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076042222baab886a-00631f81ca-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.300 DEBUG:boto:b'jdpiak-16obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:26.301 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:26.301 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:26.302 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:26.302 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:26.303 DEBUG:boto:Method: GET 2022-09-12T19:00:26.303 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:26.304 DEBUG:boto:Data: 2022-09-12T19:00:26.304 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.305 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.305 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.306 DEBUG:boto:Params: {} 2022-09-12T19:00:26.307 DEBUG:boto:Token: None 2022-09-12T19:00:26.307 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/ 2022-09-12T19:00:26.308 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM= 2022-09-12T19:00:26.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM=', 'Content-Length': '0'} 2022-09-12T19:00:26.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009184521cc646f430-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.312 DEBUG:boto:b'jdpiak-16obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:26.313 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:26.313 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:26.314 DEBUG:boto:Method: HEAD 2022-09-12T19:00:26.314 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3 2022-09-12T19:00:26.315 DEBUG:boto:Data: 2022-09-12T19:00:26.315 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.316 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:26.316 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.317 DEBUG:boto:Params: {} 2022-09-12T19:00:26.317 DEBUG:boto:Token: None 2022-09-12T19:00:26.318 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/obj-test-zone3 2022-09-12T19:00:26.319 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c+H2eerXaiiUo8QPH4m9sovURMQ= 2022-09-12T19:00:26.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c+H2eerXaiiUo8QPH4m9sovURMQ=', 'Content-Length': '0'} 2022-09-12T19:00:26.322 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004221d5796e05ea67-00631f81ca-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.322 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:26.323 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:26.323 DEBUG:boto:Method: HEAD 2022-09-12T19:00:26.324 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3 2022-09-12T19:00:26.324 DEBUG:boto:Data: 2022-09-12T19:00:26.325 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.325 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.326 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.326 DEBUG:boto:Params: {} 2022-09-12T19:00:26.327 DEBUG:boto:Token: None 2022-09-12T19:00:26.328 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/obj-test-zone3 2022-09-12T19:00:26.328 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c+H2eerXaiiUo8QPH4m9sovURMQ= 2022-09-12T19:00:26.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c+H2eerXaiiUo8QPH4m9sovURMQ=', 'Content-Length': '0'} 2022-09-12T19:00:26.331 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058d55ee5dd3e8233-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:00:26.332 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:26.333 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:26.334 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.334 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.335 DEBUG:boto:Method: GET 2022-09-12T19:00:26.335 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.336 DEBUG:boto:Data: 2022-09-12T19:00:26.336 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.337 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:26.337 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.338 DEBUG:boto:Params: {} 2022-09-12T19:00:26.338 DEBUG:boto:Token: None 2022-09-12T19:00:26.339 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.340 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AKOntJgq5YXHlU3VtdDEJJDKlEo= 2022-09-12T19:00:26.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AKOntJgq5YXHlU3VtdDEJJDKlEo=', 'Content-Length': '0'} 2022-09-12T19:00:26.342 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042f23d4a4f63e0a7-00631f81ca-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.343 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:26.344 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:26.344 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.345 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.345 DEBUG:boto:Method: GET 2022-09-12T19:00:26.346 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.346 DEBUG:boto:Data: 2022-09-12T19:00:26.347 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.347 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.348 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.348 DEBUG:boto:Params: {} 2022-09-12T19:00:26.349 DEBUG:boto:Token: None 2022-09-12T19:00:26.350 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:26.350 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AKOntJgq5YXHlU3VtdDEJJDKlEo= 2022-09-12T19:00:26.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AKOntJgq5YXHlU3VtdDEJJDKlEo=', 'Content-Length': '0'} 2022-09-12T19:00:26.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d031e365b5df4d8-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.394 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:26.395 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:26.396 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:26.396 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:26.397 DEBUG:boto:Method: GET 2022-09-12T19:00:26.397 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:26.398 DEBUG:boto:Data: 2022-09-12T19:00:26.399 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.399 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:26.400 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.400 DEBUG:boto:Params: {} 2022-09-12T19:00:26.401 DEBUG:boto:Token: None 2022-09-12T19:00:26.402 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/ 2022-09-12T19:00:26.402 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM= 2022-09-12T19:00:26.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM=', 'Content-Length': '0'} 2022-09-12T19:00:26.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033cab36b326897ff-00631f81ca-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.407 DEBUG:boto:b'jdpiak-16obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:26.408 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:26.409 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:26.410 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:26.410 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:26.411 DEBUG:boto:Method: GET 2022-09-12T19:00:26.411 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:26.412 DEBUG:boto:Data: 2022-09-12T19:00:26.412 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.413 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.413 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.414 DEBUG:boto:Params: {} 2022-09-12T19:00:26.415 DEBUG:boto:Token: None 2022-09-12T19:00:26.415 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/ 2022-09-12T19:00:26.416 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM= 2022-09-12T19:00:26.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM=', 'Content-Length': '0'} 2022-09-12T19:00:26.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abede15aed1772d9-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.421 DEBUG:boto:b'jdpiak-16obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:26.422 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-16 zones={test-zone2, test-zone1} 2022-09-12T19:00:26.423 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-16 zones={test-zone3, test-zone1} 2022-09-12T19:00:26.424 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:26.425 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:26.426 DEBUG:boto:Method: HEAD 2022-09-12T19:00:26.427 DEBUG:boto:Path: /jdpiak-16/ 2022-09-12T19:00:26.428 DEBUG:boto:Data: 2022-09-12T19:00:26.429 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.430 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:26.431 DEBUG:boto:Port: 8001 2022-09-12T19:00:26.432 DEBUG:boto:Params: {} 2022-09-12T19:00:26.433 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T19:00:26.434 DEBUG:boto:Token: None 2022-09-12T19:00:26.435 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/ 2022-09-12T19:00:26.436 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fOYLoWsi46u95ql0cSc6uigVuw4= 2022-09-12T19:00:26.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fOYLoWsi46u95ql0cSc6uigVuw4=', 'Content-Length': '0'} 2022-09-12T19:00:26.442 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029316f96b464f28d-00631f81ca-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.443 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:26.444 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:26.446 DEBUG:boto:Method: HEAD 2022-09-12T19:00:26.447 DEBUG:boto:Path: /jdpiak-16/ 2022-09-12T19:00:26.448 DEBUG:boto:Data: 2022-09-12T19:00:26.449 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.450 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.451 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.452 DEBUG:boto:Params: {} 2022-09-12T19:00:26.453 DEBUG:boto:Token: None 2022-09-12T19:00:26.454 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/ 2022-09-12T19:00:26.455 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fOYLoWsi46u95ql0cSc6uigVuw4= 2022-09-12T19:00:26.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fOYLoWsi46u95ql0cSc6uigVuw4=', 'Content-Length': '0'} 2022-09-12T19:00:26.458 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027c16b6877e031c8-00631f81ca-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.459 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:00:26.459 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:26.460 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:26.460 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:26.461 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:26.461 DEBUG:boto:Method: GET 2022-09-12T19:00:26.462 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:26.462 DEBUG:boto:Data: 2022-09-12T19:00:26.463 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.463 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:26.464 DEBUG:boto:Port: 8001 2022-09-12T19:00:26.464 DEBUG:boto:Params: {} 2022-09-12T19:00:26.466 DEBUG:boto:Token: None 2022-09-12T19:00:26.466 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/?versions 2022-09-12T19:00:26.467 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eiCf+nNxjy/VXYJSYD7eTkBkC2s= 2022-09-12T19:00:26.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eiCf+nNxjy/VXYJSYD7eTkBkC2s=', 'Content-Length': '0'} 2022-09-12T19:00:26.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045fb8661f07a4cb5-00631f81ca-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.472 DEBUG:boto:b'jdpiak-16999falsedummydM1h1LlChQwJOE3oHBYl7Qu-tamZZbJtrue2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1-zshRkRrxKU0EJVPpVBUsfYl8JR7HFRtrue2022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone23vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5true2022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eEtrue2022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:26.473 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:00:26.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:00:26.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:00:26.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:00:26.476 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:00:26.477 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:26.477 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:26.479 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:26.479 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:26.480 DEBUG:boto:Method: GET 2022-09-12T19:00:26.480 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:26.481 DEBUG:boto:Data: 2022-09-12T19:00:26.481 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.482 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.482 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.483 DEBUG:boto:Params: {} 2022-09-12T19:00:26.483 DEBUG:boto:Token: None 2022-09-12T19:00:26.484 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/?versions 2022-09-12T19:00:26.485 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eiCf+nNxjy/VXYJSYD7eTkBkC2s= 2022-09-12T19:00:26.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eiCf+nNxjy/VXYJSYD7eTkBkC2s=', 'Content-Length': '0'} 2022-09-12T19:00:26.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e84b8d25cf2ddf4-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.489 DEBUG:boto:b'jdpiak-16999falsedummydM1h1LlChQwJOE3oHBYl7Qu-tamZZbJtrue2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1-zshRkRrxKU0EJVPpVBUsfYl8JR7HFRtrue2022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone23vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5true2022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eEtrue2022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:26.490 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:00:26.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:00:26.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:00:26.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:00:26.492 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:26.492 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:26.493 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:26.493 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:26.494 DEBUG:boto:Method: GET 2022-09-12T19:00:26.495 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:26.495 DEBUG:boto:Data: 2022-09-12T19:00:26.496 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.496 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:26.497 DEBUG:boto:Port: 8001 2022-09-12T19:00:26.497 DEBUG:boto:Params: {} 2022-09-12T19:00:26.498 DEBUG:boto:Token: None 2022-09-12T19:00:26.498 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/?versions 2022-09-12T19:00:26.499 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eiCf+nNxjy/VXYJSYD7eTkBkC2s= 2022-09-12T19:00:26.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eiCf+nNxjy/VXYJSYD7eTkBkC2s=', 'Content-Length': '0'} 2022-09-12T19:00:26.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089cf947a8efdf3cf-00631f81ca-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.503 DEBUG:boto:b'jdpiak-16999falsedummydM1h1LlChQwJOE3oHBYl7Qu-tamZZbJtrue2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1-zshRkRrxKU0EJVPpVBUsfYl8JR7HFRtrue2022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone23vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5true2022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eEtrue2022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:26.504 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:26.504 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:26.505 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:26.506 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:26.506 DEBUG:boto:Method: GET 2022-09-12T19:00:26.507 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:26.507 DEBUG:boto:Data: 2022-09-12T19:00:26.508 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.509 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.509 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.510 DEBUG:boto:Params: {} 2022-09-12T19:00:26.510 DEBUG:boto:Token: None 2022-09-12T19:00:26.511 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/?versions 2022-09-12T19:00:26.512 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eiCf+nNxjy/VXYJSYD7eTkBkC2s= 2022-09-12T19:00:26.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eiCf+nNxjy/VXYJSYD7eTkBkC2s=', 'Content-Length': '0'} 2022-09-12T19:00:26.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b0e7bd51b9024bf-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.516 DEBUG:boto:b'jdpiak-16999falsedummydM1h1LlChQwJOE3oHBYl7Qu-tamZZbJtrue2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1-zshRkRrxKU0EJVPpVBUsfYl8JR7HFRtrue2022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone23vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5true2022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eEtrue2022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:26.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:00:26.517 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:26.518 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:26.518 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.519 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.520 DEBUG:boto:Method: GET 2022-09-12T19:00:26.520 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.521 DEBUG:boto:Data: 2022-09-12T19:00:26.521 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.522 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:26.522 DEBUG:boto:Port: 8001 2022-09-12T19:00:26.523 DEBUG:boto:Params: {} 2022-09-12T19:00:26.523 DEBUG:boto:Token: None 2022-09-12T19:00:26.524 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.525 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xr5TA42AMiEgPLH10iP4LGDuQgg= 2022-09-12T19:00:26.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xr5TA42AMiEgPLH10iP4LGDuQgg=', 'Content-Length': '0'} 2022-09-12T19:00:26.528 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002470787b962aaedf-00631f81ca-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.528 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:26.529 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:26.530 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.530 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.531 DEBUG:boto:Method: GET 2022-09-12T19:00:26.531 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.532 DEBUG:boto:Data: 2022-09-12T19:00:26.532 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.533 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.533 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.534 DEBUG:boto:Params: {} 2022-09-12T19:00:26.534 DEBUG:boto:Token: None 2022-09-12T19:00:26.535 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.536 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xr5TA42AMiEgPLH10iP4LGDuQgg= 2022-09-12T19:00:26.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xr5TA42AMiEgPLH10iP4LGDuQgg=', 'Content-Length': '0'} 2022-09-12T19:00:26.539 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ab44ce07b7542188-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.539 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:26.540 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:26.540 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.541 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.541 DEBUG:boto:Method: HEAD 2022-09-12T19:00:26.542 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.543 DEBUG:boto:Data: 2022-09-12T19:00:26.543 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.544 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:26.544 DEBUG:boto:Port: 8001 2022-09-12T19:00:26.545 DEBUG:boto:Params: {} 2022-09-12T19:00:26.545 DEBUG:boto:Token: None 2022-09-12T19:00:26.546 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.547 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Huzq5fKaF/2ayKvFn7xMmHhFWDE= 2022-09-12T19:00:26.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Huzq5fKaF/2ayKvFn7xMmHhFWDE=', 'Content-Length': '0'} 2022-09-12T19:00:26.549 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000021b14d5377d278d8-00631f81ca-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.550 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:26.551 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:26.551 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.552 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.552 DEBUG:boto:Method: HEAD 2022-09-12T19:00:26.553 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.553 DEBUG:boto:Data: 2022-09-12T19:00:26.554 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.554 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.555 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.555 DEBUG:boto:Params: {} 2022-09-12T19:00:26.556 DEBUG:boto:Token: None 2022-09-12T19:00:26.556 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.557 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Huzq5fKaF/2ayKvFn7xMmHhFWDE= 2022-09-12T19:00:26.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Huzq5fKaF/2ayKvFn7xMmHhFWDE=', 'Content-Length': '0'} 2022-09-12T19:00:26.559 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005a09c3ab05f47361-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:00:26.561 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:26.561 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:26.562 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.562 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.564 DEBUG:boto:Method: GET 2022-09-12T19:00:26.564 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.565 DEBUG:boto:Data: 2022-09-12T19:00:26.565 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.566 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:26.566 DEBUG:boto:Port: 8001 2022-09-12T19:00:26.567 DEBUG:boto:Params: {} 2022-09-12T19:00:26.567 DEBUG:boto:Token: None 2022-09-12T19:00:26.568 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.569 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xr5TA42AMiEgPLH10iP4LGDuQgg= 2022-09-12T19:00:26.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xr5TA42AMiEgPLH10iP4LGDuQgg=', 'Content-Length': '0'} 2022-09-12T19:00:26.571 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000e660240dca43a35-00631f81ca-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.572 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:26.573 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:26.573 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.574 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.574 DEBUG:boto:Method: GET 2022-09-12T19:00:26.575 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.575 DEBUG:boto:Data: 2022-09-12T19:00:26.576 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.576 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.577 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.577 DEBUG:boto:Params: {} 2022-09-12T19:00:26.578 DEBUG:boto:Token: None 2022-09-12T19:00:26.579 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.580 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xr5TA42AMiEgPLH10iP4LGDuQgg= 2022-09-12T19:00:26.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xr5TA42AMiEgPLH10iP4LGDuQgg=', 'Content-Length': '0'} 2022-09-12T19:00:26.583 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004e9ebc1b7425c9b0-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.584 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:26.585 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:26.586 DEBUG:boto:path=/jdpiak-16/?prefix=dummy 2022-09-12T19:00:26.587 DEBUG:boto:auth_path=/jdpiak-16/?prefix=dummy 2022-09-12T19:00:26.588 DEBUG:boto:Method: GET 2022-09-12T19:00:26.589 DEBUG:boto:Path: /jdpiak-16/?prefix=dummy 2022-09-12T19:00:26.589 DEBUG:boto:Data: 2022-09-12T19:00:26.590 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.591 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:26.592 DEBUG:boto:Port: 8001 2022-09-12T19:00:26.593 DEBUG:boto:Params: {} 2022-09-12T19:00:26.594 DEBUG:boto:Token: None 2022-09-12T19:00:26.595 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/ 2022-09-12T19:00:26.596 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM= 2022-09-12T19:00:26.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM=', 'Content-Length': '0'} 2022-09-12T19:00:26.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f41e5e1e1afcd19-00631f81ca-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.602 DEBUG:boto:b'jdpiak-16dummy1000falsedummy2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:26.603 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:26.604 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:26.605 DEBUG:boto:path=/jdpiak-16/?prefix=dummy 2022-09-12T19:00:26.606 DEBUG:boto:auth_path=/jdpiak-16/?prefix=dummy 2022-09-12T19:00:26.607 DEBUG:boto:Method: GET 2022-09-12T19:00:26.608 DEBUG:boto:Path: /jdpiak-16/?prefix=dummy 2022-09-12T19:00:26.609 DEBUG:boto:Data: 2022-09-12T19:00:26.609 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.610 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.610 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.611 DEBUG:boto:Params: {} 2022-09-12T19:00:26.611 DEBUG:boto:Token: None 2022-09-12T19:00:26.612 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/ 2022-09-12T19:00:26.613 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM= 2022-09-12T19:00:26.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM=', 'Content-Length': '0'} 2022-09-12T19:00:26.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfe28efb06ab2b47-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.658 DEBUG:boto:b'jdpiak-16dummy1000falsedummy2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:26.659 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:26.660 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:26.660 DEBUG:boto:Method: HEAD 2022-09-12T19:00:26.661 DEBUG:boto:Path: /jdpiak-16/dummy 2022-09-12T19:00:26.661 DEBUG:boto:Data: 2022-09-12T19:00:26.662 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.662 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:26.663 DEBUG:boto:Port: 8001 2022-09-12T19:00:26.663 DEBUG:boto:Params: {} 2022-09-12T19:00:26.664 DEBUG:boto:Token: None 2022-09-12T19:00:26.664 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/dummy 2022-09-12T19:00:26.665 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D6ea3ciOEZX+SFZKw42XL4syAqo= 2022-09-12T19:00:26.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D6ea3ciOEZX+SFZKw42XL4syAqo=', 'Content-Length': '0'} 2022-09-12T19:00:26.668 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f1a2f19dbac17baa-00631f81ca-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.669 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:26.670 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:26.670 DEBUG:boto:Method: HEAD 2022-09-12T19:00:26.671 DEBUG:boto:Path: /jdpiak-16/dummy 2022-09-12T19:00:26.672 DEBUG:boto:Data: 2022-09-12T19:00:26.672 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.673 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.673 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.674 DEBUG:boto:Params: {} 2022-09-12T19:00:26.674 DEBUG:boto:Token: None 2022-09-12T19:00:26.675 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/dummy 2022-09-12T19:00:26.675 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D6ea3ciOEZX+SFZKw42XL4syAqo= 2022-09-12T19:00:26.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D6ea3ciOEZX+SFZKw42XL4syAqo=', 'Content-Length': '0'} 2022-09-12T19:00:26.678 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000043f858e63b48b0cc-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:00:26.680 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:26.680 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:26.681 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.682 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.682 DEBUG:boto:Method: GET 2022-09-12T19:00:26.683 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.683 DEBUG:boto:Data: 2022-09-12T19:00:26.684 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.684 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:26.685 DEBUG:boto:Port: 8001 2022-09-12T19:00:26.685 DEBUG:boto:Params: {} 2022-09-12T19:00:26.686 DEBUG:boto:Token: None 2022-09-12T19:00:26.687 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.687 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xr5TA42AMiEgPLH10iP4LGDuQgg= 2022-09-12T19:00:26.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xr5TA42AMiEgPLH10iP4LGDuQgg=', 'Content-Length': '0'} 2022-09-12T19:00:26.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001b6b0fbd3600e30d-00631f81ca-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.691 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:26.691 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:26.692 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.692 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.693 DEBUG:boto:Method: GET 2022-09-12T19:00:26.694 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.694 DEBUG:boto:Data: 2022-09-12T19:00:26.695 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.695 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.696 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.696 DEBUG:boto:Params: {} 2022-09-12T19:00:26.697 DEBUG:boto:Token: None 2022-09-12T19:00:26.698 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:26.699 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xr5TA42AMiEgPLH10iP4LGDuQgg= 2022-09-12T19:00:26.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xr5TA42AMiEgPLH10iP4LGDuQgg=', 'Content-Length': '0'} 2022-09-12T19:00:26.701 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000007f651edd92080f7-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.702 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:26.703 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:26.703 DEBUG:boto:path=/jdpiak-16/?prefix=dummy 2022-09-12T19:00:26.704 DEBUG:boto:auth_path=/jdpiak-16/?prefix=dummy 2022-09-12T19:00:26.704 DEBUG:boto:Method: GET 2022-09-12T19:00:26.705 DEBUG:boto:Path: /jdpiak-16/?prefix=dummy 2022-09-12T19:00:26.706 DEBUG:boto:Data: 2022-09-12T19:00:26.706 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.707 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:26.707 DEBUG:boto:Port: 8001 2022-09-12T19:00:26.708 DEBUG:boto:Params: {} 2022-09-12T19:00:26.708 DEBUG:boto:Token: None 2022-09-12T19:00:26.709 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/ 2022-09-12T19:00:26.710 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM= 2022-09-12T19:00:26.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM=', 'Content-Length': '0'} 2022-09-12T19:00:26.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e96a64cdea5a4f1b-00631f81ca-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.754 DEBUG:boto:b'jdpiak-16dummy1000falsedummy2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:26.755 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:26.756 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:26.757 DEBUG:boto:path=/jdpiak-16/?prefix=dummy 2022-09-12T19:00:26.757 DEBUG:boto:auth_path=/jdpiak-16/?prefix=dummy 2022-09-12T19:00:26.758 DEBUG:boto:Method: GET 2022-09-12T19:00:26.759 DEBUG:boto:Path: /jdpiak-16/?prefix=dummy 2022-09-12T19:00:26.759 DEBUG:boto:Data: 2022-09-12T19:00:26.760 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.760 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.761 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.761 DEBUG:boto:Params: {} 2022-09-12T19:00:26.762 DEBUG:boto:Token: None 2022-09-12T19:00:26.763 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/ 2022-09-12T19:00:26.763 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM= 2022-09-12T19:00:26.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM=', 'Content-Length': '0'} 2022-09-12T19:00:26.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076bad7866e4de268-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.768 DEBUG:boto:b'jdpiak-16dummy1000falsedummy2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:26.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:00:26.770 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:26.771 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:26.771 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.772 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.772 DEBUG:boto:Method: GET 2022-09-12T19:00:26.773 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.773 DEBUG:boto:Data: 2022-09-12T19:00:26.774 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.774 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:26.775 DEBUG:boto:Port: 8001 2022-09-12T19:00:26.775 DEBUG:boto:Params: {} 2022-09-12T19:00:26.776 DEBUG:boto:Token: None 2022-09-12T19:00:26.777 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.777 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IrJg7SPF72O/vLnJgcKZkNYTnrc= 2022-09-12T19:00:26.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IrJg7SPF72O/vLnJgcKZkNYTnrc=', 'Content-Length': '0'} 2022-09-12T19:00:26.780 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2eb8a171fc36a8b-00631f81ca-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.781 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:26.782 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:26.782 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.790 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.790 DEBUG:boto:Method: GET 2022-09-12T19:00:26.791 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.791 DEBUG:boto:Data: 2022-09-12T19:00:26.792 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.792 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.793 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.793 DEBUG:boto:Params: {} 2022-09-12T19:00:26.794 DEBUG:boto:Token: None 2022-09-12T19:00:26.795 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.795 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IrJg7SPF72O/vLnJgcKZkNYTnrc= 2022-09-12T19:00:26.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IrJg7SPF72O/vLnJgcKZkNYTnrc=', 'Content-Length': '0'} 2022-09-12T19:00:26.798 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6dade1fbaf21289-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.846 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:26.847 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:26.847 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.848 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.848 DEBUG:boto:Method: HEAD 2022-09-12T19:00:26.849 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.849 DEBUG:boto:Data: 2022-09-12T19:00:26.850 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.851 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:26.851 DEBUG:boto:Port: 8001 2022-09-12T19:00:26.852 DEBUG:boto:Params: {} 2022-09-12T19:00:26.852 DEBUG:boto:Token: None 2022-09-12T19:00:26.853 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.853 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:60k/z0WZrQg4Ka3aYzmN2MUdvXU= 2022-09-12T19:00:26.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:60k/z0WZrQg4Ka3aYzmN2MUdvXU=', 'Content-Length': '0'} 2022-09-12T19:00:26.856 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e96aef2cb150a9ec-00631f81ca-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.857 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:26.858 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:26.858 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.859 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.859 DEBUG:boto:Method: HEAD 2022-09-12T19:00:26.860 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.860 DEBUG:boto:Data: 2022-09-12T19:00:26.861 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.861 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.862 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.862 DEBUG:boto:Params: {} 2022-09-12T19:00:26.863 DEBUG:boto:Token: None 2022-09-12T19:00:26.863 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.864 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:60k/z0WZrQg4Ka3aYzmN2MUdvXU= 2022-09-12T19:00:26.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:60k/z0WZrQg4Ka3aYzmN2MUdvXU=', 'Content-Length': '0'} 2022-09-12T19:00:26.867 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbd10da3446c7032-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:00:26.868 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:26.869 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:26.869 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.870 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.870 DEBUG:boto:Method: GET 2022-09-12T19:00:26.871 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.872 DEBUG:boto:Data: 2022-09-12T19:00:26.872 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.873 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:26.873 DEBUG:boto:Port: 8001 2022-09-12T19:00:26.874 DEBUG:boto:Params: {} 2022-09-12T19:00:26.874 DEBUG:boto:Token: None 2022-09-12T19:00:26.875 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.876 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IrJg7SPF72O/vLnJgcKZkNYTnrc= 2022-09-12T19:00:26.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IrJg7SPF72O/vLnJgcKZkNYTnrc=', 'Content-Length': '0'} 2022-09-12T19:00:26.878 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b98e74b388edc48-00631f81ca-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.879 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:26.880 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:26.880 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.881 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.881 DEBUG:boto:Method: GET 2022-09-12T19:00:26.882 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.882 DEBUG:boto:Data: 2022-09-12T19:00:26.883 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.883 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.884 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.884 DEBUG:boto:Params: {} 2022-09-12T19:00:26.885 DEBUG:boto:Token: None 2022-09-12T19:00:26.886 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.886 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IrJg7SPF72O/vLnJgcKZkNYTnrc= 2022-09-12T19:00:26.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IrJg7SPF72O/vLnJgcKZkNYTnrc=', 'Content-Length': '0'} 2022-09-12T19:00:26.889 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003dc0752055923f88-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.890 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:26.891 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:26.891 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:26.892 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:26.892 DEBUG:boto:Method: GET 2022-09-12T19:00:26.893 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:26.893 DEBUG:boto:Data: 2022-09-12T19:00:26.894 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.894 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:26.895 DEBUG:boto:Port: 8001 2022-09-12T19:00:26.895 DEBUG:boto:Params: {} 2022-09-12T19:00:26.896 DEBUG:boto:Token: None 2022-09-12T19:00:26.896 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/ 2022-09-12T19:00:26.897 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM= 2022-09-12T19:00:26.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM=', 'Content-Length': '0'} 2022-09-12T19:00:26.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df1187fa73a9f335-00631f81ca-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.902 DEBUG:boto:b'jdpiak-16obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:26.902 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:26.903 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:26.904 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:26.904 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:26.905 DEBUG:boto:Method: GET 2022-09-12T19:00:26.905 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:26.906 DEBUG:boto:Data: 2022-09-12T19:00:26.907 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.907 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.908 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.909 DEBUG:boto:Params: {} 2022-09-12T19:00:26.910 DEBUG:boto:Token: None 2022-09-12T19:00:26.911 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/ 2022-09-12T19:00:26.911 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM= 2022-09-12T19:00:26.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Xdr7Iwu9/kZO7GIcfTfRJdFdvM=', 'Content-Length': '0'} 2022-09-12T19:00:26.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000937cc4b17aba12a2-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.916 DEBUG:boto:b'jdpiak-16obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:26.917 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:26.917 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:26.918 DEBUG:boto:Method: HEAD 2022-09-12T19:00:26.919 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1 2022-09-12T19:00:26.919 DEBUG:boto:Data: 2022-09-12T19:00:26.920 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.920 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:26.921 DEBUG:boto:Port: 8001 2022-09-12T19:00:26.922 DEBUG:boto:Params: {} 2022-09-12T19:00:26.922 DEBUG:boto:Token: None 2022-09-12T19:00:26.923 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/obj-test-zone1 2022-09-12T19:00:26.923 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UvEexTT2j2yT5hcR+PhTWbrKGe0= 2022-09-12T19:00:26.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UvEexTT2j2yT5hcR+PhTWbrKGe0=', 'Content-Length': '0'} 2022-09-12T19:00:26.927 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015da15dbea3a3da8-00631f81ca-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.927 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:26.928 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:26.928 DEBUG:boto:Method: HEAD 2022-09-12T19:00:26.929 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1 2022-09-12T19:00:26.930 DEBUG:boto:Data: 2022-09-12T19:00:26.931 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.931 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:26.932 DEBUG:boto:Port: 8000 2022-09-12T19:00:26.933 DEBUG:boto:Params: {} 2022-09-12T19:00:26.934 DEBUG:boto:Token: None 2022-09-12T19:00:26.935 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/obj-test-zone1 2022-09-12T19:00:26.936 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UvEexTT2j2yT5hcR+PhTWbrKGe0= 2022-09-12T19:00:26.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UvEexTT2j2yT5hcR+PhTWbrKGe0=', 'Content-Length': '0'} 2022-09-12T19:00:26.939 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae3e64364b539b2c-00631f81ca-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:00:26.940 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:26.940 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:26.941 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.942 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.943 DEBUG:boto:Method: GET 2022-09-12T19:00:26.943 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.944 DEBUG:boto:Data: 2022-09-12T19:00:26.944 DEBUG:boto:Headers: {} 2022-09-12T19:00:26.945 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:26.945 DEBUG:boto:Port: 8001 2022-09-12T19:00:26.946 DEBUG:boto:Params: {} 2022-09-12T19:00:26.947 DEBUG:boto:Token: None 2022-09-12T19:00:26.948 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:26 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.948 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IrJg7SPF72O/vLnJgcKZkNYTnrc= 2022-09-12T19:00:26.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IrJg7SPF72O/vLnJgcKZkNYTnrc=', 'Content-Length': '0'} 2022-09-12T19:00:26.951 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f93c0653c2af679-00631f81ca-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:26.994 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:26.995 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:26.996 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.997 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:26.998 DEBUG:boto:Method: GET 2022-09-12T19:00:26.999 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:27.000 DEBUG:boto:Data: 2022-09-12T19:00:27.001 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.002 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:27.003 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.004 DEBUG:boto:Params: {} 2022-09-12T19:00:27.005 DEBUG:boto:Token: None 2022-09-12T19:00:27.005 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:27.006 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DWV9v5NmG7o+n6onyuiV636U2yk= 2022-09-12T19:00:27.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DWV9v5NmG7o+n6onyuiV636U2yk=', 'Content-Length': '0'} 2022-09-12T19:00:27.009 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071d2f05fd7fc58bf-00631f81cb-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.010 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:27.011 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:27.011 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:27.012 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:27.013 DEBUG:boto:Method: GET 2022-09-12T19:00:27.013 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:27.014 DEBUG:boto:Data: 2022-09-12T19:00:27.014 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.015 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.015 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.016 DEBUG:boto:Params: {} 2022-09-12T19:00:27.016 DEBUG:boto:Token: None 2022-09-12T19:00:27.017 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/ 2022-09-12T19:00:27.018 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs= 2022-09-12T19:00:27.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs=', 'Content-Length': '0'} 2022-09-12T19:00:27.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c0eaededfbb6a5c-00631f81cb-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.023 DEBUG:boto:b'jdpiak-16obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:27.024 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:27.025 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:27.026 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:27.027 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:27.028 DEBUG:boto:Method: GET 2022-09-12T19:00:27.029 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:27.029 DEBUG:boto:Data: 2022-09-12T19:00:27.030 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.031 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:27.031 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.032 DEBUG:boto:Params: {} 2022-09-12T19:00:27.033 DEBUG:boto:Token: None 2022-09-12T19:00:27.033 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/ 2022-09-12T19:00:27.034 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs= 2022-09-12T19:00:27.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs=', 'Content-Length': '0'} 2022-09-12T19:00:27.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c399f38d5224e6e1-00631f81cb-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.038 DEBUG:boto:b'jdpiak-16obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:27.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:00:27.040 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:27.040 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:27.041 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.041 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.042 DEBUG:boto:Method: GET 2022-09-12T19:00:27.042 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.043 DEBUG:boto:Data: 2022-09-12T19:00:27.043 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.044 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.044 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.045 DEBUG:boto:Params: {} 2022-09-12T19:00:27.046 DEBUG:boto:Token: None 2022-09-12T19:00:27.047 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.047 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:knfYH5Xu1CvGFCW9wFXd4AOim18= 2022-09-12T19:00:27.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:knfYH5Xu1CvGFCW9wFXd4AOim18=', 'Content-Length': '0'} 2022-09-12T19:00:27.050 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047652d46e96bbb5a-00631f81cb-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.051 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:27.051 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:27.052 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.053 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.053 DEBUG:boto:Method: GET 2022-09-12T19:00:27.054 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.054 DEBUG:boto:Data: 2022-09-12T19:00:27.055 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.056 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:27.057 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.057 DEBUG:boto:Params: {} 2022-09-12T19:00:27.058 DEBUG:boto:Token: None 2022-09-12T19:00:27.059 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.059 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:knfYH5Xu1CvGFCW9wFXd4AOim18= 2022-09-12T19:00:27.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:knfYH5Xu1CvGFCW9wFXd4AOim18=', 'Content-Length': '0'} 2022-09-12T19:00:27.062 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d954d3d83af0757f-00631f81cb-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.063 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:27.064 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:27.065 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.066 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.067 DEBUG:boto:Method: HEAD 2022-09-12T19:00:27.068 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.069 DEBUG:boto:Data: 2022-09-12T19:00:27.069 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.070 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.071 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.072 DEBUG:boto:Params: {} 2022-09-12T19:00:27.073 DEBUG:boto:Token: None 2022-09-12T19:00:27.074 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.075 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BHEXOb2OSUIN4nnlm22s0K0kFyo= 2022-09-12T19:00:27.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BHEXOb2OSUIN4nnlm22s0K0kFyo=', 'Content-Length': '0'} 2022-09-12T19:00:27.079 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c94d6bc2cea5005-00631f81cb-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.079 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:27.080 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:27.081 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.082 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.082 DEBUG:boto:Method: HEAD 2022-09-12T19:00:27.083 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.084 DEBUG:boto:Data: 2022-09-12T19:00:27.085 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.086 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:27.087 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.088 DEBUG:boto:Params: {} 2022-09-12T19:00:27.089 DEBUG:boto:Token: None 2022-09-12T19:00:27.090 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.091 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BHEXOb2OSUIN4nnlm22s0K0kFyo= 2022-09-12T19:00:27.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BHEXOb2OSUIN4nnlm22s0K0kFyo=', 'Content-Length': '0'} 2022-09-12T19:00:27.094 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e2e47ecc8316ee4-00631f81cb-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:00:27.096 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:27.097 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:27.097 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.098 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.099 DEBUG:boto:Method: GET 2022-09-12T19:00:27.099 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.100 DEBUG:boto:Data: 2022-09-12T19:00:27.101 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.102 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.103 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.103 DEBUG:boto:Params: {} 2022-09-12T19:00:27.104 DEBUG:boto:Token: None 2022-09-12T19:00:27.104 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.106 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:knfYH5Xu1CvGFCW9wFXd4AOim18= 2022-09-12T19:00:27.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:knfYH5Xu1CvGFCW9wFXd4AOim18=', 'Content-Length': '0'} 2022-09-12T19:00:27.109 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006949ea69c253733c-00631f81cb-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.154 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:27.155 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:27.155 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.156 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.157 DEBUG:boto:Method: GET 2022-09-12T19:00:27.157 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.158 DEBUG:boto:Data: 2022-09-12T19:00:27.159 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.160 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:27.160 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.161 DEBUG:boto:Params: {} 2022-09-12T19:00:27.162 DEBUG:boto:Token: None 2022-09-12T19:00:27.163 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.163 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:knfYH5Xu1CvGFCW9wFXd4AOim18= 2022-09-12T19:00:27.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:knfYH5Xu1CvGFCW9wFXd4AOim18=', 'Content-Length': '0'} 2022-09-12T19:00:27.166 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b509ea31e7b80f57-00631f81cb-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.167 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:27.168 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:27.168 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:27.169 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:27.170 DEBUG:boto:Method: GET 2022-09-12T19:00:27.170 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:27.171 DEBUG:boto:Data: 2022-09-12T19:00:27.172 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.173 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.173 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.174 DEBUG:boto:Params: {} 2022-09-12T19:00:27.174 DEBUG:boto:Token: None 2022-09-12T19:00:27.175 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/ 2022-09-12T19:00:27.176 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs= 2022-09-12T19:00:27.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs=', 'Content-Length': '0'} 2022-09-12T19:00:27.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004816c9577f9f4a55-00631f81cb-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.182 DEBUG:boto:b'jdpiak-16obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:27.183 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:27.184 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:27.185 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:27.186 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:27.188 DEBUG:boto:Method: GET 2022-09-12T19:00:27.189 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:27.189 DEBUG:boto:Data: 2022-09-12T19:00:27.190 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.191 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:27.191 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.192 DEBUG:boto:Params: {} 2022-09-12T19:00:27.193 DEBUG:boto:Token: None 2022-09-12T19:00:27.194 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/ 2022-09-12T19:00:27.195 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs= 2022-09-12T19:00:27.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs=', 'Content-Length': '0'} 2022-09-12T19:00:27.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc0b62f06eef1d37-00631f81cb-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.200 DEBUG:boto:b'jdpiak-16obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:27.202 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:27.202 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:27.203 DEBUG:boto:Method: HEAD 2022-09-12T19:00:27.204 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2 2022-09-12T19:00:27.204 DEBUG:boto:Data: 2022-09-12T19:00:27.205 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.205 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.206 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.207 DEBUG:boto:Params: {} 2022-09-12T19:00:27.207 DEBUG:boto:Token: None 2022-09-12T19:00:27.208 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/obj-test-zone2 2022-09-12T19:00:27.209 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pc1GmuBGCk6k1Als9qM+YOPtW54= 2022-09-12T19:00:27.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pc1GmuBGCk6k1Als9qM+YOPtW54=', 'Content-Length': '0'} 2022-09-12T19:00:27.212 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000918db75917cbeb91-00631f81cb-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.213 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:27.214 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:27.214 DEBUG:boto:Method: HEAD 2022-09-12T19:00:27.215 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2 2022-09-12T19:00:27.215 DEBUG:boto:Data: 2022-09-12T19:00:27.216 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.217 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:27.217 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.218 DEBUG:boto:Params: {} 2022-09-12T19:00:27.219 DEBUG:boto:Token: None 2022-09-12T19:00:27.219 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/obj-test-zone2 2022-09-12T19:00:27.221 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pc1GmuBGCk6k1Als9qM+YOPtW54= 2022-09-12T19:00:27.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pc1GmuBGCk6k1Als9qM+YOPtW54=', 'Content-Length': '0'} 2022-09-12T19:00:27.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a320147491c03bb0-00631f81cb-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:00:27.227 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:27.228 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:27.229 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.230 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.232 DEBUG:boto:Method: GET 2022-09-12T19:00:27.233 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.235 DEBUG:boto:Data: 2022-09-12T19:00:27.236 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.238 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.239 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.241 DEBUG:boto:Params: {} 2022-09-12T19:00:27.242 DEBUG:boto:Token: None 2022-09-12T19:00:27.243 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.244 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:knfYH5Xu1CvGFCW9wFXd4AOim18= 2022-09-12T19:00:27.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:knfYH5Xu1CvGFCW9wFXd4AOim18=', 'Content-Length': '0'} 2022-09-12T19:00:27.247 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d184823b5ef57f19-00631f81cb-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.248 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:27.249 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:27.250 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.250 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.251 DEBUG:boto:Method: GET 2022-09-12T19:00:27.252 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.253 DEBUG:boto:Data: 2022-09-12T19:00:27.253 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.254 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:27.255 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.256 DEBUG:boto:Params: {} 2022-09-12T19:00:27.256 DEBUG:boto:Token: None 2022-09-12T19:00:27.257 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:27.258 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:knfYH5Xu1CvGFCW9wFXd4AOim18= 2022-09-12T19:00:27.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:knfYH5Xu1CvGFCW9wFXd4AOim18=', 'Content-Length': '0'} 2022-09-12T19:00:27.261 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003cff038320841060-00631f81cb-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.302 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:27.304 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:27.305 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:27.306 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:27.307 DEBUG:boto:Method: GET 2022-09-12T19:00:27.308 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:27.308 DEBUG:boto:Data: 2022-09-12T19:00:27.309 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.310 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.310 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.311 DEBUG:boto:Params: {} 2022-09-12T19:00:27.312 DEBUG:boto:Token: None 2022-09-12T19:00:27.313 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/ 2022-09-12T19:00:27.313 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs= 2022-09-12T19:00:27.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs=', 'Content-Length': '0'} 2022-09-12T19:00:27.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b04e812222c87bdc-00631f81cb-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.319 DEBUG:boto:b'jdpiak-16obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:27.321 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:27.322 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:27.323 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:27.324 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:27.325 DEBUG:boto:Method: GET 2022-09-12T19:00:27.325 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:27.326 DEBUG:boto:Data: 2022-09-12T19:00:27.327 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.327 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:27.328 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.329 DEBUG:boto:Params: {} 2022-09-12T19:00:27.330 DEBUG:boto:Token: None 2022-09-12T19:00:27.330 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/ 2022-09-12T19:00:27.331 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs= 2022-09-12T19:00:27.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs=', 'Content-Length': '0'} 2022-09-12T19:00:27.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ada99219f8ca48d6-00631f81cb-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.336 DEBUG:boto:b'jdpiak-16obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:27.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:00:27.338 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:27.339 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:27.340 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.341 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.342 DEBUG:boto:Method: GET 2022-09-12T19:00:27.343 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.343 DEBUG:boto:Data: 2022-09-12T19:00:27.344 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.345 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.346 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.347 DEBUG:boto:Params: {} 2022-09-12T19:00:27.348 DEBUG:boto:Token: None 2022-09-12T19:00:27.349 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.350 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9wgfF7O8JRMZDG5TpX/4gpARvw8= 2022-09-12T19:00:27.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9wgfF7O8JRMZDG5TpX/4gpARvw8=', 'Content-Length': '0'} 2022-09-12T19:00:27.354 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000015783012b55a450-00631f81cb-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.355 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:27.356 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:27.357 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.358 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.359 DEBUG:boto:Method: GET 2022-09-12T19:00:27.361 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.362 DEBUG:boto:Data: 2022-09-12T19:00:27.363 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.364 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:27.365 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.366 DEBUG:boto:Params: {} 2022-09-12T19:00:27.367 DEBUG:boto:Token: None 2022-09-12T19:00:27.368 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.370 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9wgfF7O8JRMZDG5TpX/4gpARvw8= 2022-09-12T19:00:27.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9wgfF7O8JRMZDG5TpX/4gpARvw8=', 'Content-Length': '0'} 2022-09-12T19:00:27.373 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9d66b8bd1073ad0-00631f81cb-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.375 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:27.375 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:27.376 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.377 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.377 DEBUG:boto:Method: HEAD 2022-09-12T19:00:27.378 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.379 DEBUG:boto:Data: 2022-09-12T19:00:27.380 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.381 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.382 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.383 DEBUG:boto:Params: {} 2022-09-12T19:00:27.384 DEBUG:boto:Token: None 2022-09-12T19:00:27.385 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.386 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kUqalnL19Kl8zffv4f4HTuP8kRk= 2022-09-12T19:00:27.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kUqalnL19Kl8zffv4f4HTuP8kRk=', 'Content-Length': '0'} 2022-09-12T19:00:27.390 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ec86ff4973c340d-00631f81cb-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.391 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:27.392 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:27.393 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.393 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.394 DEBUG:boto:Method: HEAD 2022-09-12T19:00:27.395 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.396 DEBUG:boto:Data: 2022-09-12T19:00:27.397 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.398 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:27.399 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.399 DEBUG:boto:Params: {} 2022-09-12T19:00:27.400 DEBUG:boto:Token: None 2022-09-12T19:00:27.401 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.402 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kUqalnL19Kl8zffv4f4HTuP8kRk= 2022-09-12T19:00:27.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kUqalnL19Kl8zffv4f4HTuP8kRk=', 'Content-Length': '0'} 2022-09-12T19:00:27.406 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d77a36fdd37b4cef-00631f81cb-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:00:27.408 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:27.409 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:27.410 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.411 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.412 DEBUG:boto:Method: GET 2022-09-12T19:00:27.413 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.414 DEBUG:boto:Data: 2022-09-12T19:00:27.415 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.416 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.416 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.417 DEBUG:boto:Params: {} 2022-09-12T19:00:27.418 DEBUG:boto:Token: None 2022-09-12T19:00:27.419 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.420 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9wgfF7O8JRMZDG5TpX/4gpARvw8= 2022-09-12T19:00:27.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9wgfF7O8JRMZDG5TpX/4gpARvw8=', 'Content-Length': '0'} 2022-09-12T19:00:27.424 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068ca46de4cbdc9e6-00631f81cb-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.469 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:27.470 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:27.471 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.472 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.473 DEBUG:boto:Method: GET 2022-09-12T19:00:27.474 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.475 DEBUG:boto:Data: 2022-09-12T19:00:27.476 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.477 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:27.478 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.479 DEBUG:boto:Params: {} 2022-09-12T19:00:27.480 DEBUG:boto:Token: None 2022-09-12T19:00:27.481 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.482 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9wgfF7O8JRMZDG5TpX/4gpARvw8= 2022-09-12T19:00:27.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9wgfF7O8JRMZDG5TpX/4gpARvw8=', 'Content-Length': '0'} 2022-09-12T19:00:27.486 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054cf8379b444ccdb-00631f81cb-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.488 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:27.489 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:27.490 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:27.491 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:27.492 DEBUG:boto:Method: GET 2022-09-12T19:00:27.493 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:27.494 DEBUG:boto:Data: 2022-09-12T19:00:27.495 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.495 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.496 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.497 DEBUG:boto:Params: {} 2022-09-12T19:00:27.499 DEBUG:boto:Token: None 2022-09-12T19:00:27.500 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/ 2022-09-12T19:00:27.501 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs= 2022-09-12T19:00:27.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs=', 'Content-Length': '0'} 2022-09-12T19:00:27.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003cd3edd207f886e-00631f81cb-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.506 DEBUG:boto:b'jdpiak-16obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:27.507 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:27.507 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:27.508 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:27.509 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:27.509 DEBUG:boto:Method: GET 2022-09-12T19:00:27.510 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:27.511 DEBUG:boto:Data: 2022-09-12T19:00:27.511 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.512 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:27.512 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.513 DEBUG:boto:Params: {} 2022-09-12T19:00:27.513 DEBUG:boto:Token: None 2022-09-12T19:00:27.514 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/ 2022-09-12T19:00:27.515 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs= 2022-09-12T19:00:27.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs=', 'Content-Length': '0'} 2022-09-12T19:00:27.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000091de6fe6fd94c11-00631f81cb-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.562 DEBUG:boto:b'jdpiak-16obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:27.563 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:27.564 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:27.564 DEBUG:boto:Method: HEAD 2022-09-12T19:00:27.565 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3 2022-09-12T19:00:27.566 DEBUG:boto:Data: 2022-09-12T19:00:27.567 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.567 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.568 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.570 DEBUG:boto:Params: {} 2022-09-12T19:00:27.571 DEBUG:boto:Token: None 2022-09-12T19:00:27.572 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/obj-test-zone3 2022-09-12T19:00:27.572 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:djllxjf4IX4luUKHu9frfagoazM= 2022-09-12T19:00:27.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:djllxjf4IX4luUKHu9frfagoazM=', 'Content-Length': '0'} 2022-09-12T19:00:27.576 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000040d97d2689ea29be-00631f81cb-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.577 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:27.578 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:27.578 DEBUG:boto:Method: HEAD 2022-09-12T19:00:27.579 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3 2022-09-12T19:00:27.580 DEBUG:boto:Data: 2022-09-12T19:00:27.580 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.581 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:27.581 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.582 DEBUG:boto:Params: {} 2022-09-12T19:00:27.582 DEBUG:boto:Token: None 2022-09-12T19:00:27.583 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/obj-test-zone3 2022-09-12T19:00:27.584 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:djllxjf4IX4luUKHu9frfagoazM= 2022-09-12T19:00:27.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:djllxjf4IX4luUKHu9frfagoazM=', 'Content-Length': '0'} 2022-09-12T19:00:27.587 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002f6a1860ff2876c-00631f81cb-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:00:27.588 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:27.589 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:27.590 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.590 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.591 DEBUG:boto:Method: GET 2022-09-12T19:00:27.592 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.592 DEBUG:boto:Data: 2022-09-12T19:00:27.593 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.593 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.594 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.594 DEBUG:boto:Params: {} 2022-09-12T19:00:27.595 DEBUG:boto:Token: None 2022-09-12T19:00:27.595 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.596 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9wgfF7O8JRMZDG5TpX/4gpARvw8= 2022-09-12T19:00:27.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9wgfF7O8JRMZDG5TpX/4gpARvw8=', 'Content-Length': '0'} 2022-09-12T19:00:27.599 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb9d60b6a3ff1c11-00631f81cb-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.600 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:27.601 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:27.602 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.603 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.603 DEBUG:boto:Method: GET 2022-09-12T19:00:27.604 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.604 DEBUG:boto:Data: 2022-09-12T19:00:27.605 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.605 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:27.606 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.607 DEBUG:boto:Params: {} 2022-09-12T19:00:27.607 DEBUG:boto:Token: None 2022-09-12T19:00:27.608 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:27.608 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9wgfF7O8JRMZDG5TpX/4gpARvw8= 2022-09-12T19:00:27.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9wgfF7O8JRMZDG5TpX/4gpARvw8=', 'Content-Length': '0'} 2022-09-12T19:00:27.612 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014118091c1da3670-00631f81cb-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.613 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:27.614 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:27.615 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:27.616 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:27.616 DEBUG:boto:Method: GET 2022-09-12T19:00:27.617 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:27.617 DEBUG:boto:Data: 2022-09-12T19:00:27.618 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.618 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.619 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.619 DEBUG:boto:Params: {} 2022-09-12T19:00:27.620 DEBUG:boto:Token: None 2022-09-12T19:00:27.620 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/ 2022-09-12T19:00:27.621 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs= 2022-09-12T19:00:27.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs=', 'Content-Length': '0'} 2022-09-12T19:00:27.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4bc454175df3f99-00631f81cb-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.625 DEBUG:boto:b'jdpiak-16obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:27.626 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:27.627 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:27.628 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:27.628 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:27.629 DEBUG:boto:Method: GET 2022-09-12T19:00:27.629 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:27.630 DEBUG:boto:Data: 2022-09-12T19:00:27.631 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.631 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:00:27.632 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.633 DEBUG:boto:Params: {} 2022-09-12T19:00:27.633 DEBUG:boto:Token: None 2022-09-12T19:00:27.634 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/ 2022-09-12T19:00:27.634 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs= 2022-09-12T19:00:27.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs=', 'Content-Length': '0'} 2022-09-12T19:00:27.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047a403f7a3721c1f-00631f81cb-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.638 DEBUG:boto:b'jdpiak-16obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:27.639 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-16 zones={test-zone3, test-zone1} 2022-09-12T19:00:27.640 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-16 zones={test-zone3, test-zone2} 2022-09-12T19:00:27.640 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:27.641 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:27.641 DEBUG:boto:Method: HEAD 2022-09-12T19:00:27.642 DEBUG:boto:Path: /jdpiak-16/ 2022-09-12T19:00:27.642 DEBUG:boto:Data: 2022-09-12T19:00:27.643 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.643 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.644 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.644 DEBUG:boto:Params: {} 2022-09-12T19:00:27.645 DEBUG:boto:Token: None 2022-09-12T19:00:27.646 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/ 2022-09-12T19:00:27.646 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VUtnPMsXrXIyLOpOFcKLBW5ThZ8= 2022-09-12T19:00:27.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VUtnPMsXrXIyLOpOFcKLBW5ThZ8=', 'Content-Length': '0'} 2022-09-12T19:00:27.649 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ce79da1c09f9141-00631f81cb-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.650 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:27.651 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:27.651 DEBUG:boto:Method: HEAD 2022-09-12T19:00:27.652 DEBUG:boto:Path: /jdpiak-16/ 2022-09-12T19:00:27.652 DEBUG:boto:Data: 2022-09-12T19:00:27.653 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.653 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:27.654 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.654 DEBUG:boto:Params: {} 2022-09-12T19:00:27.655 DEBUG:boto:Token: None 2022-09-12T19:00:27.655 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/ 2022-09-12T19:00:27.656 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VUtnPMsXrXIyLOpOFcKLBW5ThZ8= 2022-09-12T19:00:27.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VUtnPMsXrXIyLOpOFcKLBW5ThZ8=', 'Content-Length': '0'} 2022-09-12T19:00:27.659 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a89f6aa651c424f-00631f81cb-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.660 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:00:27.660 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:27.661 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:27.661 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:27.662 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:27.663 DEBUG:boto:Method: GET 2022-09-12T19:00:27.663 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:27.664 DEBUG:boto:Data: 2022-09-12T19:00:27.664 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.665 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.665 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.666 DEBUG:boto:Params: {} 2022-09-12T19:00:27.666 DEBUG:boto:Token: None 2022-09-12T19:00:27.667 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/?versions 2022-09-12T19:00:27.667 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MkkzwfGlOdJ9UlhGhbybs318e3w= 2022-09-12T19:00:27.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MkkzwfGlOdJ9UlhGhbybs318e3w=', 'Content-Length': '0'} 2022-09-12T19:00:27.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003158267bdf2d49c1-00631f81cb-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.672 DEBUG:boto:b'jdpiak-16999falsedummydM1h1LlChQwJOE3oHBYl7Qu-tamZZbJtrue2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1-zshRkRrxKU0EJVPpVBUsfYl8JR7HFRtrue2022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone23vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5true2022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eEtrue2022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:27.674 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:00:27.675 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:00:27.675 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:00:27.676 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:00:27.677 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:00:27.678 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:27.679 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:27.680 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:27.681 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:27.682 DEBUG:boto:Method: GET 2022-09-12T19:00:27.683 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:27.683 DEBUG:boto:Data: 2022-09-12T19:00:27.684 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.684 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:27.685 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.685 DEBUG:boto:Params: {} 2022-09-12T19:00:27.686 DEBUG:boto:Token: None 2022-09-12T19:00:27.686 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/?versions 2022-09-12T19:00:27.687 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MkkzwfGlOdJ9UlhGhbybs318e3w= 2022-09-12T19:00:27.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MkkzwfGlOdJ9UlhGhbybs318e3w=', 'Content-Length': '0'} 2022-09-12T19:00:27.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9d2af4006983ae3-00631f81cb-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.691 DEBUG:boto:b'jdpiak-16999falsedummydM1h1LlChQwJOE3oHBYl7Qu-tamZZbJtrue2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1-zshRkRrxKU0EJVPpVBUsfYl8JR7HFRtrue2022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone23vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5true2022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eEtrue2022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:27.692 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:00:27.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:00:27.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:00:27.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:00:27.695 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:27.695 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:27.696 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:27.697 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:27.698 DEBUG:boto:Method: GET 2022-09-12T19:00:27.699 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:27.699 DEBUG:boto:Data: 2022-09-12T19:00:27.700 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.701 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.701 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.702 DEBUG:boto:Params: {} 2022-09-12T19:00:27.703 DEBUG:boto:Token: None 2022-09-12T19:00:27.703 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/?versions 2022-09-12T19:00:27.704 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MkkzwfGlOdJ9UlhGhbybs318e3w= 2022-09-12T19:00:27.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MkkzwfGlOdJ9UlhGhbybs318e3w=', 'Content-Length': '0'} 2022-09-12T19:00:27.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e13a4ad5ef31536-00631f81cb-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.708 DEBUG:boto:b'jdpiak-16999falsedummydM1h1LlChQwJOE3oHBYl7Qu-tamZZbJtrue2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1-zshRkRrxKU0EJVPpVBUsfYl8JR7HFRtrue2022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone23vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5true2022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eEtrue2022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:27.709 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:27.710 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:27.710 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:27.711 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:27.711 DEBUG:boto:Method: GET 2022-09-12T19:00:27.712 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999 2022-09-12T19:00:27.712 DEBUG:boto:Data: 2022-09-12T19:00:27.713 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.713 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:27.714 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.714 DEBUG:boto:Params: {} 2022-09-12T19:00:27.715 DEBUG:boto:Token: None 2022-09-12T19:00:27.716 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/?versions 2022-09-12T19:00:27.717 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MkkzwfGlOdJ9UlhGhbybs318e3w= 2022-09-12T19:00:27.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MkkzwfGlOdJ9UlhGhbybs318e3w=', 'Content-Length': '0'} 2022-09-12T19:00:27.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000723e7150fa0af1d7-00631f81cb-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.721 DEBUG:boto:b'jdpiak-16999falsedummydM1h1LlChQwJOE3oHBYl7Qu-tamZZbJtrue2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1-zshRkRrxKU0EJVPpVBUsfYl8JR7HFRtrue2022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone23vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5true2022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eEtrue2022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:27.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:00:27.722 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:27.723 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:27.723 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.724 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.724 DEBUG:boto:Method: GET 2022-09-12T19:00:27.725 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.725 DEBUG:boto:Data: 2022-09-12T19:00:27.726 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.727 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.727 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.728 DEBUG:boto:Params: {} 2022-09-12T19:00:27.728 DEBUG:boto:Token: None 2022-09-12T19:00:27.729 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.730 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3IQI+Dx40AfcBIpwk2vVtN/jZ1Y= 2022-09-12T19:00:27.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3IQI+Dx40AfcBIpwk2vVtN/jZ1Y=', 'Content-Length': '0'} 2022-09-12T19:00:27.734 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009ba23a8700da6b34-00631f81cb-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.735 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:27.736 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:27.736 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.737 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.737 DEBUG:boto:Method: GET 2022-09-12T19:00:27.738 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.738 DEBUG:boto:Data: 2022-09-12T19:00:27.739 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.739 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:27.740 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.740 DEBUG:boto:Params: {} 2022-09-12T19:00:27.741 DEBUG:boto:Token: None 2022-09-12T19:00:27.742 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.742 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3IQI+Dx40AfcBIpwk2vVtN/jZ1Y= 2022-09-12T19:00:27.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3IQI+Dx40AfcBIpwk2vVtN/jZ1Y=', 'Content-Length': '0'} 2022-09-12T19:00:27.745 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005058c55ca6a8bb3f-00631f81cb-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.746 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:27.746 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:27.747 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.747 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.748 DEBUG:boto:Method: HEAD 2022-09-12T19:00:27.748 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.749 DEBUG:boto:Data: 2022-09-12T19:00:27.749 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.750 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.750 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.751 DEBUG:boto:Params: {} 2022-09-12T19:00:27.751 DEBUG:boto:Token: None 2022-09-12T19:00:27.752 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.753 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RHKI46Mnp+PPiqdGG4C6r9YYPLA= 2022-09-12T19:00:27.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RHKI46Mnp+PPiqdGG4C6r9YYPLA=', 'Content-Length': '0'} 2022-09-12T19:00:27.755 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000de0d6c818e6b57d6-00631f81cb-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.756 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:27.756 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:27.757 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.757 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.758 DEBUG:boto:Method: HEAD 2022-09-12T19:00:27.758 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.759 DEBUG:boto:Data: 2022-09-12T19:00:27.759 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.760 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:27.760 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.761 DEBUG:boto:Params: {} 2022-09-12T19:00:27.762 DEBUG:boto:Token: None 2022-09-12T19:00:27.762 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.763 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RHKI46Mnp+PPiqdGG4C6r9YYPLA= 2022-09-12T19:00:27.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RHKI46Mnp+PPiqdGG4C6r9YYPLA=', 'Content-Length': '0'} 2022-09-12T19:00:27.765 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bf03e973379a6702-00631f81cb-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:00:27.767 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:27.768 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:27.769 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.770 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.771 DEBUG:boto:Method: GET 2022-09-12T19:00:27.772 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.773 DEBUG:boto:Data: 2022-09-12T19:00:27.774 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.775 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.776 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.777 DEBUG:boto:Params: {} 2022-09-12T19:00:27.778 DEBUG:boto:Token: None 2022-09-12T19:00:27.779 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.780 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3IQI+Dx40AfcBIpwk2vVtN/jZ1Y= 2022-09-12T19:00:27.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3IQI+Dx40AfcBIpwk2vVtN/jZ1Y=', 'Content-Length': '0'} 2022-09-12T19:00:27.783 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d20abdddb646a276-00631f81cb-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.784 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:27.785 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:27.785 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.786 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.786 DEBUG:boto:Method: GET 2022-09-12T19:00:27.787 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.787 DEBUG:boto:Data: 2022-09-12T19:00:27.788 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.789 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:27.790 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.790 DEBUG:boto:Params: {} 2022-09-12T19:00:27.791 DEBUG:boto:Token: None 2022-09-12T19:00:27.791 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.792 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3IQI+Dx40AfcBIpwk2vVtN/jZ1Y= 2022-09-12T19:00:27.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3IQI+Dx40AfcBIpwk2vVtN/jZ1Y=', 'Content-Length': '0'} 2022-09-12T19:00:27.794 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007fbf881cf005faa9-00631f81cb-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.795 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:27.796 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:27.796 DEBUG:boto:path=/jdpiak-16/?prefix=dummy 2022-09-12T19:00:27.797 DEBUG:boto:auth_path=/jdpiak-16/?prefix=dummy 2022-09-12T19:00:27.797 DEBUG:boto:Method: GET 2022-09-12T19:00:27.798 DEBUG:boto:Path: /jdpiak-16/?prefix=dummy 2022-09-12T19:00:27.798 DEBUG:boto:Data: 2022-09-12T19:00:27.799 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.799 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.800 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.800 DEBUG:boto:Params: {} 2022-09-12T19:00:27.801 DEBUG:boto:Token: None 2022-09-12T19:00:27.802 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/ 2022-09-12T19:00:27.802 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs= 2022-09-12T19:00:27.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs=', 'Content-Length': '0'} 2022-09-12T19:00:27.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2f782b483717e26-00631f81cb-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.846 DEBUG:boto:b'jdpiak-16dummy1000falsedummy2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:27.847 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:27.848 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:27.848 DEBUG:boto:path=/jdpiak-16/?prefix=dummy 2022-09-12T19:00:27.849 DEBUG:boto:auth_path=/jdpiak-16/?prefix=dummy 2022-09-12T19:00:27.849 DEBUG:boto:Method: GET 2022-09-12T19:00:27.850 DEBUG:boto:Path: /jdpiak-16/?prefix=dummy 2022-09-12T19:00:27.851 DEBUG:boto:Data: 2022-09-12T19:00:27.851 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.852 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:27.852 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.853 DEBUG:boto:Params: {} 2022-09-12T19:00:27.854 DEBUG:boto:Token: None 2022-09-12T19:00:27.854 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/ 2022-09-12T19:00:27.855 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs= 2022-09-12T19:00:27.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs=', 'Content-Length': '0'} 2022-09-12T19:00:27.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cea7254b448a602-00631f81cb-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.859 DEBUG:boto:b'jdpiak-16dummy1000falsedummy2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:27.860 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:27.861 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:27.861 DEBUG:boto:Method: HEAD 2022-09-12T19:00:27.862 DEBUG:boto:Path: /jdpiak-16/dummy 2022-09-12T19:00:27.863 DEBUG:boto:Data: 2022-09-12T19:00:27.864 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.864 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.865 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.865 DEBUG:boto:Params: {} 2022-09-12T19:00:27.866 DEBUG:boto:Token: None 2022-09-12T19:00:27.866 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/dummy 2022-09-12T19:00:27.867 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VNpbe0Xwxiuq37lBk9vkduiB5SQ= 2022-09-12T19:00:27.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VNpbe0Xwxiuq37lBk9vkduiB5SQ=', 'Content-Length': '0'} 2022-09-12T19:00:27.870 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d9825d5b1577fce6-00631f81cb-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.871 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:27.871 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:27.872 DEBUG:boto:Method: HEAD 2022-09-12T19:00:27.872 DEBUG:boto:Path: /jdpiak-16/dummy 2022-09-12T19:00:27.873 DEBUG:boto:Data: 2022-09-12T19:00:27.873 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.874 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:27.874 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.875 DEBUG:boto:Params: {} 2022-09-12T19:00:27.875 DEBUG:boto:Token: None 2022-09-12T19:00:27.876 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/dummy 2022-09-12T19:00:27.877 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VNpbe0Xwxiuq37lBk9vkduiB5SQ= 2022-09-12T19:00:27.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VNpbe0Xwxiuq37lBk9vkduiB5SQ=', 'Content-Length': '0'} 2022-09-12T19:00:27.880 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000058a65c574158d952-00631f81cb-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:00:27.881 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:27.881 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:27.882 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.882 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.883 DEBUG:boto:Method: GET 2022-09-12T19:00:27.883 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.884 DEBUG:boto:Data: 2022-09-12T19:00:27.884 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.885 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.885 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.886 DEBUG:boto:Params: {} 2022-09-12T19:00:27.887 DEBUG:boto:Token: None 2022-09-12T19:00:27.887 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.888 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3IQI+Dx40AfcBIpwk2vVtN/jZ1Y= 2022-09-12T19:00:27.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3IQI+Dx40AfcBIpwk2vVtN/jZ1Y=', 'Content-Length': '0'} 2022-09-12T19:00:27.890 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001b2b1222e4bb84a2-00631f81cb-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.891 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:00:27.892 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:00:27.892 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.893 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.893 DEBUG:boto:Method: GET 2022-09-12T19:00:27.894 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.894 DEBUG:boto:Data: 2022-09-12T19:00:27.895 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.895 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:27.896 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.896 DEBUG:boto:Params: {} 2022-09-12T19:00:27.897 DEBUG:boto:Token: None 2022-09-12T19:00:27.897 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:00:27.898 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3IQI+Dx40AfcBIpwk2vVtN/jZ1Y= 2022-09-12T19:00:27.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3IQI+Dx40AfcBIpwk2vVtN/jZ1Y=', 'Content-Length': '0'} 2022-09-12T19:00:27.901 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fa12b1e8da757aff-00631f81cb-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.901 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:27.902 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:27.903 DEBUG:boto:path=/jdpiak-16/?prefix=dummy 2022-09-12T19:00:27.903 DEBUG:boto:auth_path=/jdpiak-16/?prefix=dummy 2022-09-12T19:00:27.904 DEBUG:boto:Method: GET 2022-09-12T19:00:27.904 DEBUG:boto:Path: /jdpiak-16/?prefix=dummy 2022-09-12T19:00:27.905 DEBUG:boto:Data: 2022-09-12T19:00:27.905 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.906 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.906 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.907 DEBUG:boto:Params: {} 2022-09-12T19:00:27.907 DEBUG:boto:Token: None 2022-09-12T19:00:27.908 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/ 2022-09-12T19:00:27.908 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs= 2022-09-12T19:00:27.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs=', 'Content-Length': '0'} 2022-09-12T19:00:27.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024fba7617765efea-00631f81cb-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.954 DEBUG:boto:b'jdpiak-16dummy1000falsedummy2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:27.956 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:27.957 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:27.958 DEBUG:boto:path=/jdpiak-16/?prefix=dummy 2022-09-12T19:00:27.959 DEBUG:boto:auth_path=/jdpiak-16/?prefix=dummy 2022-09-12T19:00:27.960 DEBUG:boto:Method: GET 2022-09-12T19:00:27.961 DEBUG:boto:Path: /jdpiak-16/?prefix=dummy 2022-09-12T19:00:27.962 DEBUG:boto:Data: 2022-09-12T19:00:27.963 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.964 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:27.965 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.966 DEBUG:boto:Params: {} 2022-09-12T19:00:27.967 DEBUG:boto:Token: None 2022-09-12T19:00:27.968 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/ 2022-09-12T19:00:27.969 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs= 2022-09-12T19:00:27.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qKsxiIw8q/WLGlL4BClhapFoENs=', 'Content-Length': '0'} 2022-09-12T19:00:27.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005145944618fe083-00631f81cb-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.975 DEBUG:boto:b'jdpiak-16dummy1000falsedummy2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:27.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:00:27.977 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:27.978 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:27.979 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:27.980 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:27.981 DEBUG:boto:Method: GET 2022-09-12T19:00:27.982 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:27.982 DEBUG:boto:Data: 2022-09-12T19:00:27.983 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.984 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:27.984 DEBUG:boto:Port: 8001 2022-09-12T19:00:27.985 DEBUG:boto:Params: {} 2022-09-12T19:00:27.985 DEBUG:boto:Token: None 2022-09-12T19:00:27.986 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:27.986 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DWV9v5NmG7o+n6onyuiV636U2yk= 2022-09-12T19:00:27.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DWV9v5NmG7o+n6onyuiV636U2yk=', 'Content-Length': '0'} 2022-09-12T19:00:27.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c01b91422350189e-00631f81cb-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:27.990 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:27.990 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:27.991 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:27.991 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:27.992 DEBUG:boto:Method: GET 2022-09-12T19:00:27.993 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:27.993 DEBUG:boto:Data: 2022-09-12T19:00:27.994 DEBUG:boto:Headers: {} 2022-09-12T19:00:27.994 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:27.995 DEBUG:boto:Port: 8000 2022-09-12T19:00:27.995 DEBUG:boto:Params: {} 2022-09-12T19:00:27.996 DEBUG:boto:Token: None 2022-09-12T19:00:27.996 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:27 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:27.997 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DWV9v5NmG7o+n6onyuiV636U2yk= 2022-09-12T19:00:27.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DWV9v5NmG7o+n6onyuiV636U2yk=', 'Content-Length': '0'} 2022-09-12T19:00:28.000 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a01fbfba8853795-00631f81cb-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.000 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:28.001 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:28.002 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:28.002 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:28.003 DEBUG:boto:Method: HEAD 2022-09-12T19:00:28.003 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:28.003 DEBUG:boto:Data: 2022-09-12T19:00:28.004 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.004 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:28.005 DEBUG:boto:Port: 8001 2022-09-12T19:00:28.005 DEBUG:boto:Params: {} 2022-09-12T19:00:28.006 DEBUG:boto:Token: None 2022-09-12T19:00:28.007 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:28.007 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9B1fT8YM9tGC4J5Pj+B38gGhAOs= 2022-09-12T19:00:28.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9B1fT8YM9tGC4J5Pj+B38gGhAOs=', 'Content-Length': '0'} 2022-09-12T19:00:28.010 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a44309aa49557ed1-00631f81cc-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.011 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:28.011 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:28.012 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:28.012 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:28.013 DEBUG:boto:Method: HEAD 2022-09-12T19:00:28.013 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:28.014 DEBUG:boto:Data: 2022-09-12T19:00:28.014 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.015 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:28.015 DEBUG:boto:Port: 8000 2022-09-12T19:00:28.016 DEBUG:boto:Params: {} 2022-09-12T19:00:28.016 DEBUG:boto:Token: None 2022-09-12T19:00:28.017 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:28.017 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9B1fT8YM9tGC4J5Pj+B38gGhAOs= 2022-09-12T19:00:28.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9B1fT8YM9tGC4J5Pj+B38gGhAOs=', 'Content-Length': '0'} 2022-09-12T19:00:28.020 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006dd802f8c9f7ffe4-00631f81cc-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:00:28.021 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:28.022 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:28.022 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:28.023 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:28.023 DEBUG:boto:Method: GET 2022-09-12T19:00:28.024 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:28.024 DEBUG:boto:Data: 2022-09-12T19:00:28.025 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.025 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:28.026 DEBUG:boto:Port: 8001 2022-09-12T19:00:28.026 DEBUG:boto:Params: {} 2022-09-12T19:00:28.027 DEBUG:boto:Token: None 2022-09-12T19:00:28.028 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:28.028 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QABbwz8sAarQYl7/PH82UfdxkN0= 2022-09-12T19:00:28.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QABbwz8sAarQYl7/PH82UfdxkN0=', 'Content-Length': '0'} 2022-09-12T19:00:28.031 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f81dc060f235c89-00631f81cc-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.032 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:28.033 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:28.033 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:28.034 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:28.034 DEBUG:boto:Method: GET 2022-09-12T19:00:28.035 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:28.035 DEBUG:boto:Data: 2022-09-12T19:00:28.036 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.036 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:28.037 DEBUG:boto:Port: 8000 2022-09-12T19:00:28.037 DEBUG:boto:Params: {} 2022-09-12T19:00:28.038 DEBUG:boto:Token: None 2022-09-12T19:00:28.038 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:28.039 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QABbwz8sAarQYl7/PH82UfdxkN0= 2022-09-12T19:00:28.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QABbwz8sAarQYl7/PH82UfdxkN0=', 'Content-Length': '0'} 2022-09-12T19:00:28.042 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b56d101f40c98ee4-00631f81cc-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.042 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:28.043 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:28.043 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:28.044 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:28.045 DEBUG:boto:Method: GET 2022-09-12T19:00:28.045 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:28.046 DEBUG:boto:Data: 2022-09-12T19:00:28.046 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.047 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:28.047 DEBUG:boto:Port: 8001 2022-09-12T19:00:28.048 DEBUG:boto:Params: {} 2022-09-12T19:00:28.049 DEBUG:boto:Token: None 2022-09-12T19:00:28.049 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/ 2022-09-12T19:00:28.050 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kPCGBZgRyM7yedUARmZ0Wxo5i4o= 2022-09-12T19:00:28.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kPCGBZgRyM7yedUARmZ0Wxo5i4o=', 'Content-Length': '0'} 2022-09-12T19:00:28.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000356dc4911d51bf03-00631f81cc-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.054 DEBUG:boto:b'jdpiak-16obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:28.054 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:28.055 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:28.055 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:28.056 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:28.057 DEBUG:boto:Method: GET 2022-09-12T19:00:28.057 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:28.058 DEBUG:boto:Data: 2022-09-12T19:00:28.058 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.059 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:28.059 DEBUG:boto:Port: 8000 2022-09-12T19:00:28.060 DEBUG:boto:Params: {} 2022-09-12T19:00:28.060 DEBUG:boto:Token: None 2022-09-12T19:00:28.061 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/ 2022-09-12T19:00:28.061 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kPCGBZgRyM7yedUARmZ0Wxo5i4o= 2022-09-12T19:00:28.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kPCGBZgRyM7yedUARmZ0Wxo5i4o=', 'Content-Length': '0'} 2022-09-12T19:00:28.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076e88d5e5f91fb07-00631f81cc-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.065 DEBUG:boto:b'jdpiak-16obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:28.066 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:28.066 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:28.067 DEBUG:boto:Method: HEAD 2022-09-12T19:00:28.067 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1 2022-09-12T19:00:28.068 DEBUG:boto:Data: 2022-09-12T19:00:28.068 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.069 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:28.069 DEBUG:boto:Port: 8001 2022-09-12T19:00:28.070 DEBUG:boto:Params: {} 2022-09-12T19:00:28.071 DEBUG:boto:Token: None 2022-09-12T19:00:28.071 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone1 2022-09-12T19:00:28.072 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QbV9mBvWdGdWbuWvNSDaE39rw9Q= 2022-09-12T19:00:28.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QbV9mBvWdGdWbuWvNSDaE39rw9Q=', 'Content-Length': '0'} 2022-09-12T19:00:28.075 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000635b541df47358d4-00631f81cc-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.075 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:28.076 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:28.076 DEBUG:boto:Method: HEAD 2022-09-12T19:00:28.077 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1 2022-09-12T19:00:28.077 DEBUG:boto:Data: 2022-09-12T19:00:28.078 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.078 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:28.079 DEBUG:boto:Port: 8000 2022-09-12T19:00:28.079 DEBUG:boto:Params: {} 2022-09-12T19:00:28.080 DEBUG:boto:Token: None 2022-09-12T19:00:28.081 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone1 2022-09-12T19:00:28.081 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QbV9mBvWdGdWbuWvNSDaE39rw9Q= 2022-09-12T19:00:28.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QbV9mBvWdGdWbuWvNSDaE39rw9Q=', 'Content-Length': '0'} 2022-09-12T19:00:28.084 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cfddc0088f6141a9-00631f81cc-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:00:28.086 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:28.087 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:28.087 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:28.088 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:28.089 DEBUG:boto:Method: GET 2022-09-12T19:00:28.090 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:28.090 DEBUG:boto:Data: 2022-09-12T19:00:28.091 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.091 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:28.092 DEBUG:boto:Port: 8001 2022-09-12T19:00:28.092 DEBUG:boto:Params: {} 2022-09-12T19:00:28.093 DEBUG:boto:Token: None 2022-09-12T19:00:28.093 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:28.094 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QABbwz8sAarQYl7/PH82UfdxkN0= 2022-09-12T19:00:28.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QABbwz8sAarQYl7/PH82UfdxkN0=', 'Content-Length': '0'} 2022-09-12T19:00:28.097 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb92b8cb4cfc5aea-00631f81cc-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.138 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:28.139 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:00:28.139 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:28.140 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:28.140 DEBUG:boto:Method: GET 2022-09-12T19:00:28.141 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:28.141 DEBUG:boto:Data: 2022-09-12T19:00:28.142 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.142 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:28.143 DEBUG:boto:Port: 8000 2022-09-12T19:00:28.143 DEBUG:boto:Params: {} 2022-09-12T19:00:28.144 DEBUG:boto:Token: None 2022-09-12T19:00:28.145 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:00:28.145 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QABbwz8sAarQYl7/PH82UfdxkN0= 2022-09-12T19:00:28.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QABbwz8sAarQYl7/PH82UfdxkN0=', 'Content-Length': '0'} 2022-09-12T19:00:28.148 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085bc540e0775cac4-00631f81cc-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.149 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:28.150 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:28.150 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:28.151 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:28.151 DEBUG:boto:Method: GET 2022-09-12T19:00:28.152 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:28.152 DEBUG:boto:Data: 2022-09-12T19:00:28.153 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.153 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:28.154 DEBUG:boto:Port: 8001 2022-09-12T19:00:28.155 DEBUG:boto:Params: {} 2022-09-12T19:00:28.155 DEBUG:boto:Token: None 2022-09-12T19:00:28.156 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/ 2022-09-12T19:00:28.156 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kPCGBZgRyM7yedUARmZ0Wxo5i4o= 2022-09-12T19:00:28.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kPCGBZgRyM7yedUARmZ0Wxo5i4o=', 'Content-Length': '0'} 2022-09-12T19:00:28.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000388f725bc0fcdfe8-00631f81cc-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.160 DEBUG:boto:b'jdpiak-16obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:28.161 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:28.162 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:28.163 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:28.163 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:28.164 DEBUG:boto:Method: GET 2022-09-12T19:00:28.164 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:00:28.165 DEBUG:boto:Data: 2022-09-12T19:00:28.165 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.166 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:28.166 DEBUG:boto:Port: 8000 2022-09-12T19:00:28.167 DEBUG:boto:Params: {} 2022-09-12T19:00:28.167 DEBUG:boto:Token: None 2022-09-12T19:00:28.168 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/ 2022-09-12T19:00:28.168 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kPCGBZgRyM7yedUARmZ0Wxo5i4o= 2022-09-12T19:00:28.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kPCGBZgRyM7yedUARmZ0Wxo5i4o=', 'Content-Length': '0'} 2022-09-12T19:00:28.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfb8cd71fc8f8acc-00631f81cc-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.172 DEBUG:boto:b'jdpiak-16obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:28.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:00:28.173 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:28.174 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:28.174 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.175 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.175 DEBUG:boto:Method: GET 2022-09-12T19:00:28.176 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.176 DEBUG:boto:Data: 2022-09-12T19:00:28.177 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.177 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:28.178 DEBUG:boto:Port: 8001 2022-09-12T19:00:28.179 DEBUG:boto:Params: {} 2022-09-12T19:00:28.179 DEBUG:boto:Token: None 2022-09-12T19:00:28.180 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.180 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YVZCl6NocQD5damZVp1DGPbtrmM= 2022-09-12T19:00:28.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YVZCl6NocQD5damZVp1DGPbtrmM=', 'Content-Length': '0'} 2022-09-12T19:00:28.183 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c9f013ec39b0f39-00631f81cc-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.184 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:28.185 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:28.185 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.186 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.186 DEBUG:boto:Method: GET 2022-09-12T19:00:28.187 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.187 DEBUG:boto:Data: 2022-09-12T19:00:28.188 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.188 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:28.189 DEBUG:boto:Port: 8000 2022-09-12T19:00:28.189 DEBUG:boto:Params: {} 2022-09-12T19:00:28.190 DEBUG:boto:Token: None 2022-09-12T19:00:28.190 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.191 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YVZCl6NocQD5damZVp1DGPbtrmM= 2022-09-12T19:00:28.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YVZCl6NocQD5damZVp1DGPbtrmM=', 'Content-Length': '0'} 2022-09-12T19:00:28.194 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c6b77ec156ebc9b-00631f81cc-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.195 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:28.195 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:28.196 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.196 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.196 DEBUG:boto:Method: HEAD 2022-09-12T19:00:28.197 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.197 DEBUG:boto:Data: 2022-09-12T19:00:28.198 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.198 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:28.199 DEBUG:boto:Port: 8001 2022-09-12T19:00:28.199 DEBUG:boto:Params: {} 2022-09-12T19:00:28.200 DEBUG:boto:Token: None 2022-09-12T19:00:28.201 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.201 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TzdKXDTbDSifq5uifSwyMuG90+U= 2022-09-12T19:00:28.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TzdKXDTbDSifq5uifSwyMuG90+U=', 'Content-Length': '0'} 2022-09-12T19:00:28.204 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e535d5b4cdf1f4b-00631f81cc-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.205 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:28.205 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:28.206 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.206 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.207 DEBUG:boto:Method: HEAD 2022-09-12T19:00:28.207 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.208 DEBUG:boto:Data: 2022-09-12T19:00:28.208 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.209 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:28.209 DEBUG:boto:Port: 8000 2022-09-12T19:00:28.210 DEBUG:boto:Params: {} 2022-09-12T19:00:28.210 DEBUG:boto:Token: None 2022-09-12T19:00:28.211 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.211 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TzdKXDTbDSifq5uifSwyMuG90+U= 2022-09-12T19:00:28.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TzdKXDTbDSifq5uifSwyMuG90+U=', 'Content-Length': '0'} 2022-09-12T19:00:28.214 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084cd18ab56eaf630-00631f81cc-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:00:28.215 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:28.215 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:28.216 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.216 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.217 DEBUG:boto:Method: GET 2022-09-12T19:00:28.217 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.218 DEBUG:boto:Data: 2022-09-12T19:00:28.219 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.219 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:28.220 DEBUG:boto:Port: 8001 2022-09-12T19:00:28.220 DEBUG:boto:Params: {} 2022-09-12T19:00:28.221 DEBUG:boto:Token: None 2022-09-12T19:00:28.221 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.222 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YVZCl6NocQD5damZVp1DGPbtrmM= 2022-09-12T19:00:28.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YVZCl6NocQD5damZVp1DGPbtrmM=', 'Content-Length': '0'} 2022-09-12T19:00:28.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000453888c8f057b2b3-00631f81cc-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.225 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:28.226 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:28.227 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.228 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.228 DEBUG:boto:Method: GET 2022-09-12T19:00:28.229 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.229 DEBUG:boto:Data: 2022-09-12T19:00:28.230 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.230 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:28.231 DEBUG:boto:Port: 8000 2022-09-12T19:00:28.232 DEBUG:boto:Params: {} 2022-09-12T19:00:28.232 DEBUG:boto:Token: None 2022-09-12T19:00:28.233 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.233 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YVZCl6NocQD5damZVp1DGPbtrmM= 2022-09-12T19:00:28.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YVZCl6NocQD5damZVp1DGPbtrmM=', 'Content-Length': '0'} 2022-09-12T19:00:28.236 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049ba1d760ce13854-00631f81cc-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.237 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:28.238 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:28.238 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:28.239 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:28.239 DEBUG:boto:Method: GET 2022-09-12T19:00:28.240 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:28.240 DEBUG:boto:Data: 2022-09-12T19:00:28.241 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.241 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:28.242 DEBUG:boto:Port: 8001 2022-09-12T19:00:28.243 DEBUG:boto:Params: {} 2022-09-12T19:00:28.243 DEBUG:boto:Token: None 2022-09-12T19:00:28.244 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/ 2022-09-12T19:00:28.245 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kPCGBZgRyM7yedUARmZ0Wxo5i4o= 2022-09-12T19:00:28.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kPCGBZgRyM7yedUARmZ0Wxo5i4o=', 'Content-Length': '0'} 2022-09-12T19:00:28.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074ff3192e8244dbe-00631f81cc-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.249 DEBUG:boto:b'jdpiak-16obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:28.250 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:28.250 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:28.251 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:28.251 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:28.252 DEBUG:boto:Method: GET 2022-09-12T19:00:28.252 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:28.253 DEBUG:boto:Data: 2022-09-12T19:00:28.253 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.254 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:28.254 DEBUG:boto:Port: 8000 2022-09-12T19:00:28.255 DEBUG:boto:Params: {} 2022-09-12T19:00:28.256 DEBUG:boto:Token: None 2022-09-12T19:00:28.256 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/ 2022-09-12T19:00:28.257 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kPCGBZgRyM7yedUARmZ0Wxo5i4o= 2022-09-12T19:00:28.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kPCGBZgRyM7yedUARmZ0Wxo5i4o=', 'Content-Length': '0'} 2022-09-12T19:00:28.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a175c930f202ac52-00631f81cc-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.261 DEBUG:boto:b'jdpiak-16obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:28.261 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:28.262 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:28.263 DEBUG:boto:Method: HEAD 2022-09-12T19:00:28.263 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2 2022-09-12T19:00:28.264 DEBUG:boto:Data: 2022-09-12T19:00:28.264 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.265 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:28.265 DEBUG:boto:Port: 8001 2022-09-12T19:00:28.266 DEBUG:boto:Params: {} 2022-09-12T19:00:28.266 DEBUG:boto:Token: None 2022-09-12T19:00:28.267 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone2 2022-09-12T19:00:28.267 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Slz32qiX/bssfCYIKT0/v7OQ9FQ= 2022-09-12T19:00:28.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Slz32qiX/bssfCYIKT0/v7OQ9FQ=', 'Content-Length': '0'} 2022-09-12T19:00:28.271 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f00f8f4d5aead74-00631f81cc-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.271 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:28.272 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:28.272 DEBUG:boto:Method: HEAD 2022-09-12T19:00:28.273 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2 2022-09-12T19:00:28.273 DEBUG:boto:Data: 2022-09-12T19:00:28.274 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.274 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:28.275 DEBUG:boto:Port: 8000 2022-09-12T19:00:28.275 DEBUG:boto:Params: {} 2022-09-12T19:00:28.276 DEBUG:boto:Token: None 2022-09-12T19:00:28.277 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone2 2022-09-12T19:00:28.277 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Slz32qiX/bssfCYIKT0/v7OQ9FQ= 2022-09-12T19:00:28.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Slz32qiX/bssfCYIKT0/v7OQ9FQ=', 'Content-Length': '0'} 2022-09-12T19:00:28.280 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d4337b25aeed0c3-00631f81cc-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:00:28.282 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:28.282 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:28.283 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.283 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.284 DEBUG:boto:Method: GET 2022-09-12T19:00:28.284 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.285 DEBUG:boto:Data: 2022-09-12T19:00:28.285 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.286 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:28.287 DEBUG:boto:Port: 8001 2022-09-12T19:00:28.287 DEBUG:boto:Params: {} 2022-09-12T19:00:28.288 DEBUG:boto:Token: None 2022-09-12T19:00:28.288 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.289 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YVZCl6NocQD5damZVp1DGPbtrmM= 2022-09-12T19:00:28.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YVZCl6NocQD5damZVp1DGPbtrmM=', 'Content-Length': '0'} 2022-09-12T19:00:28.292 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000fcf87dbc484f70-00631f81cc-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.334 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:28.335 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:00:28.335 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.336 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.336 DEBUG:boto:Method: GET 2022-09-12T19:00:28.337 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.337 DEBUG:boto:Data: 2022-09-12T19:00:28.338 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.339 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:28.339 DEBUG:boto:Port: 8000 2022-09-12T19:00:28.340 DEBUG:boto:Params: {} 2022-09-12T19:00:28.340 DEBUG:boto:Token: None 2022-09-12T19:00:28.341 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:00:28.341 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YVZCl6NocQD5damZVp1DGPbtrmM= 2022-09-12T19:00:28.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YVZCl6NocQD5damZVp1DGPbtrmM=', 'Content-Length': '0'} 2022-09-12T19:00:28.344 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a834cd35a46d9ab-00631f81cc-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.345 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:28.345 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:28.346 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:28.347 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:28.347 DEBUG:boto:Method: GET 2022-09-12T19:00:28.348 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:28.348 DEBUG:boto:Data: 2022-09-12T19:00:28.349 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.349 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:28.350 DEBUG:boto:Port: 8001 2022-09-12T19:00:28.350 DEBUG:boto:Params: {} 2022-09-12T19:00:28.351 DEBUG:boto:Token: None 2022-09-12T19:00:28.351 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/ 2022-09-12T19:00:28.352 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kPCGBZgRyM7yedUARmZ0Wxo5i4o= 2022-09-12T19:00:28.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kPCGBZgRyM7yedUARmZ0Wxo5i4o=', 'Content-Length': '0'} 2022-09-12T19:00:28.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032b1d08ea51febfc-00631f81cc-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.356 DEBUG:boto:b'jdpiak-16obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:28.357 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:28.357 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:28.358 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:28.358 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:28.359 DEBUG:boto:Method: GET 2022-09-12T19:00:28.359 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:00:28.360 DEBUG:boto:Data: 2022-09-12T19:00:28.360 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.361 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:28.361 DEBUG:boto:Port: 8000 2022-09-12T19:00:28.362 DEBUG:boto:Params: {} 2022-09-12T19:00:28.363 DEBUG:boto:Token: None 2022-09-12T19:00:28.363 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/ 2022-09-12T19:00:28.364 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kPCGBZgRyM7yedUARmZ0Wxo5i4o= 2022-09-12T19:00:28.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kPCGBZgRyM7yedUARmZ0Wxo5i4o=', 'Content-Length': '0'} 2022-09-12T19:00:28.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c490065e850aaec-00631f81cc-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.367 DEBUG:boto:b'jdpiak-16obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:28.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:00:28.369 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:28.369 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:28.370 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.370 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.371 DEBUG:boto:Method: GET 2022-09-12T19:00:28.371 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.372 DEBUG:boto:Data: 2022-09-12T19:00:28.372 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.373 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:28.373 DEBUG:boto:Port: 8001 2022-09-12T19:00:28.374 DEBUG:boto:Params: {} 2022-09-12T19:00:28.374 DEBUG:boto:Token: None 2022-09-12T19:00:28.375 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.376 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3XVeX/i77t0kvgO/DFDIqL9HiXw= 2022-09-12T19:00:28.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3XVeX/i77t0kvgO/DFDIqL9HiXw=', 'Content-Length': '0'} 2022-09-12T19:00:28.378 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f00c87d5cd394d2-00631f81cc-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.379 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:28.380 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:28.380 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.381 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.381 DEBUG:boto:Method: GET 2022-09-12T19:00:28.382 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.382 DEBUG:boto:Data: 2022-09-12T19:00:28.383 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.383 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:28.384 DEBUG:boto:Port: 8000 2022-09-12T19:00:28.384 DEBUG:boto:Params: {} 2022-09-12T19:00:28.385 DEBUG:boto:Token: None 2022-09-12T19:00:28.386 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.386 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3XVeX/i77t0kvgO/DFDIqL9HiXw= 2022-09-12T19:00:28.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3XVeX/i77t0kvgO/DFDIqL9HiXw=', 'Content-Length': '0'} 2022-09-12T19:00:28.389 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008edaaf60b2f77177-00631f81cc-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.434 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:28.435 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:28.435 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.436 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.436 DEBUG:boto:Method: HEAD 2022-09-12T19:00:28.437 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.437 DEBUG:boto:Data: 2022-09-12T19:00:28.438 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.439 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:28.439 DEBUG:boto:Port: 8001 2022-09-12T19:00:28.440 DEBUG:boto:Params: {} 2022-09-12T19:00:28.440 DEBUG:boto:Token: None 2022-09-12T19:00:28.441 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.441 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FRvr5W2wNXjrLrb1leU9+SYJ+FA= 2022-09-12T19:00:28.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FRvr5W2wNXjrLrb1leU9+SYJ+FA=', 'Content-Length': '0'} 2022-09-12T19:00:28.444 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081dda3ef74f700d5-00631f81cc-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.445 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:28.445 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:28.446 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.446 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.447 DEBUG:boto:Method: HEAD 2022-09-12T19:00:28.447 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.448 DEBUG:boto:Data: 2022-09-12T19:00:28.449 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.449 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:28.450 DEBUG:boto:Port: 8000 2022-09-12T19:00:28.450 DEBUG:boto:Params: {} 2022-09-12T19:00:28.451 DEBUG:boto:Token: None 2022-09-12T19:00:28.451 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.452 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FRvr5W2wNXjrLrb1leU9+SYJ+FA= 2022-09-12T19:00:28.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FRvr5W2wNXjrLrb1leU9+SYJ+FA=', 'Content-Length': '0'} 2022-09-12T19:00:28.455 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc4d52551f7ead9e-00631f81cc-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:00:28.456 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:28.457 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:28.457 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.458 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.459 DEBUG:boto:Method: GET 2022-09-12T19:00:28.459 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.460 DEBUG:boto:Data: 2022-09-12T19:00:28.460 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.461 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:28.461 DEBUG:boto:Port: 8001 2022-09-12T19:00:28.462 DEBUG:boto:Params: {} 2022-09-12T19:00:28.463 DEBUG:boto:Token: None 2022-09-12T19:00:28.463 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.464 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3XVeX/i77t0kvgO/DFDIqL9HiXw= 2022-09-12T19:00:28.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3XVeX/i77t0kvgO/DFDIqL9HiXw=', 'Content-Length': '0'} 2022-09-12T19:00:28.467 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000478501fe3c78dd60-00631f81cc-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.467 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:28.468 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:28.469 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.469 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.470 DEBUG:boto:Method: GET 2022-09-12T19:00:28.470 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.471 DEBUG:boto:Data: 2022-09-12T19:00:28.471 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.472 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:28.472 DEBUG:boto:Port: 8000 2022-09-12T19:00:28.473 DEBUG:boto:Params: {} 2022-09-12T19:00:28.474 DEBUG:boto:Token: None 2022-09-12T19:00:28.475 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.475 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3XVeX/i77t0kvgO/DFDIqL9HiXw= 2022-09-12T19:00:28.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3XVeX/i77t0kvgO/DFDIqL9HiXw=', 'Content-Length': '0'} 2022-09-12T19:00:28.478 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d83cb80113b92194-00631f81cc-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.479 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:28.480 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:28.480 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:28.481 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:28.482 DEBUG:boto:Method: GET 2022-09-12T19:00:28.482 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:28.483 DEBUG:boto:Data: 2022-09-12T19:00:28.483 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.484 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:28.484 DEBUG:boto:Port: 8001 2022-09-12T19:00:28.485 DEBUG:boto:Params: {} 2022-09-12T19:00:28.485 DEBUG:boto:Token: None 2022-09-12T19:00:28.486 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/ 2022-09-12T19:00:28.487 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kPCGBZgRyM7yedUARmZ0Wxo5i4o= 2022-09-12T19:00:28.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kPCGBZgRyM7yedUARmZ0Wxo5i4o=', 'Content-Length': '0'} 2022-09-12T19:00:28.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007048fee84f8c6420-00631f81cc-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.534 DEBUG:boto:b'jdpiak-16obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:28.535 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:28.536 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:28.536 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:28.537 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:28.537 DEBUG:boto:Method: GET 2022-09-12T19:00:28.538 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:28.538 DEBUG:boto:Data: 2022-09-12T19:00:28.539 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.539 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:28.540 DEBUG:boto:Port: 8000 2022-09-12T19:00:28.540 DEBUG:boto:Params: {} 2022-09-12T19:00:28.541 DEBUG:boto:Token: None 2022-09-12T19:00:28.542 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/ 2022-09-12T19:00:28.542 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kPCGBZgRyM7yedUARmZ0Wxo5i4o= 2022-09-12T19:00:28.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kPCGBZgRyM7yedUARmZ0Wxo5i4o=', 'Content-Length': '0'} 2022-09-12T19:00:28.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003eb17331ed3ea60e-00631f81cc-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.546 DEBUG:boto:b'jdpiak-16obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:28.547 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:28.548 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:28.548 DEBUG:boto:Method: HEAD 2022-09-12T19:00:28.549 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3 2022-09-12T19:00:28.549 DEBUG:boto:Data: 2022-09-12T19:00:28.550 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.550 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:28.551 DEBUG:boto:Port: 8001 2022-09-12T19:00:28.551 DEBUG:boto:Params: {} 2022-09-12T19:00:28.552 DEBUG:boto:Token: None 2022-09-12T19:00:28.552 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone3 2022-09-12T19:00:28.553 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:n/YoisuYb0+RqkmJnB2a/4GfxTw= 2022-09-12T19:00:28.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:n/YoisuYb0+RqkmJnB2a/4GfxTw=', 'Content-Length': '0'} 2022-09-12T19:00:28.557 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9f94aeeb36b7692-00631f81cc-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.557 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:28.558 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:28.559 DEBUG:boto:Method: HEAD 2022-09-12T19:00:28.559 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3 2022-09-12T19:00:28.560 DEBUG:boto:Data: 2022-09-12T19:00:28.560 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.561 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:28.562 DEBUG:boto:Port: 8000 2022-09-12T19:00:28.562 DEBUG:boto:Params: {} 2022-09-12T19:00:28.563 DEBUG:boto:Token: None 2022-09-12T19:00:28.564 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone3 2022-09-12T19:00:28.565 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:n/YoisuYb0+RqkmJnB2a/4GfxTw= 2022-09-12T19:00:28.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:n/YoisuYb0+RqkmJnB2a/4GfxTw=', 'Content-Length': '0'} 2022-09-12T19:00:28.569 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000617bf3909b790581-00631f81cc-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:00:28.570 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:28.570 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:28.571 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.571 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.572 DEBUG:boto:Method: GET 2022-09-12T19:00:28.572 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.573 DEBUG:boto:Data: 2022-09-12T19:00:28.574 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.574 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:28.575 DEBUG:boto:Port: 8001 2022-09-12T19:00:28.575 DEBUG:boto:Params: {} 2022-09-12T19:00:28.576 DEBUG:boto:Token: None 2022-09-12T19:00:28.576 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.577 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3XVeX/i77t0kvgO/DFDIqL9HiXw= 2022-09-12T19:00:28.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3XVeX/i77t0kvgO/DFDIqL9HiXw=', 'Content-Length': '0'} 2022-09-12T19:00:28.580 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002717adbb6abfd618-00631f81cc-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.581 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:28.581 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:00:28.582 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.582 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.583 DEBUG:boto:Method: GET 2022-09-12T19:00:28.584 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.584 DEBUG:boto:Data: 2022-09-12T19:00:28.585 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.585 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:28.586 DEBUG:boto:Port: 8000 2022-09-12T19:00:28.586 DEBUG:boto:Params: {} 2022-09-12T19:00:28.587 DEBUG:boto:Token: None 2022-09-12T19:00:28.587 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:00:28.588 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3XVeX/i77t0kvgO/DFDIqL9HiXw= 2022-09-12T19:00:28.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3XVeX/i77t0kvgO/DFDIqL9HiXw=', 'Content-Length': '0'} 2022-09-12T19:00:28.591 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d237343d03bdfaa-00631f81cc-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.592 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:28.592 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:28.593 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:28.593 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:28.594 DEBUG:boto:Method: GET 2022-09-12T19:00:28.595 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:28.595 DEBUG:boto:Data: 2022-09-12T19:00:28.596 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.596 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:00:28.597 DEBUG:boto:Port: 8001 2022-09-12T19:00:28.597 DEBUG:boto:Params: {} 2022-09-12T19:00:28.598 DEBUG:boto:Token: None 2022-09-12T19:00:28.599 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/ 2022-09-12T19:00:28.599 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kPCGBZgRyM7yedUARmZ0Wxo5i4o= 2022-09-12T19:00:28.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kPCGBZgRyM7yedUARmZ0Wxo5i4o=', 'Content-Length': '0'} 2022-09-12T19:00:28.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082beeacb7d5852fa-00631f81cc-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.604 DEBUG:boto:b'jdpiak-16obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:28.605 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:00:28.606 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:00:28.607 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:28.608 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:28.609 DEBUG:boto:Method: GET 2022-09-12T19:00:28.610 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:00:28.611 DEBUG:boto:Data: 2022-09-12T19:00:28.612 DEBUG:boto:Headers: {} 2022-09-12T19:00:28.612 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:00:28.613 DEBUG:boto:Port: 8000 2022-09-12T19:00:28.614 DEBUG:boto:Params: {} 2022-09-12T19:00:28.614 DEBUG:boto:Token: None 2022-09-12T19:00:28.615 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:00:28 GMT /jdpiak-16/ 2022-09-12T19:00:28.615 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kPCGBZgRyM7yedUARmZ0Wxo5i4o= 2022-09-12T19:00:28.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:00:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kPCGBZgRyM7yedUARmZ0Wxo5i4o=', 'Content-Length': '0'} 2022-09-12T19:00:28.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab01839634c887f4-00631f81cc-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:00:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:00:28.662 DEBUG:boto:b'jdpiak-16obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:00:28.663 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-16 zones={test-zone3, test-zone2} 2022-09-12T19:00:28.664 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-17 2022-09-12T19:00:28.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:00:28.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:00:28.667 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:00:28.733 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:00:28.733 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:00:28.772 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:00:28.768+0000 7fa74e93f600 1 bucket sync caught up with source: 2022-09-12T19:00:28.772 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000010.31.5, , 00000000002.8.10, , 00000000007.20.9, , , , , ] 2022-09-12T19:00:28.773 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000010.31.5, , 00000000002.8.10, , 00000000007.20.9, , , , , ] 2022-09-12T19:00:28.773 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:00:28.768+0000 7fa74e93f600 0 bucket checkpoint complete 2022-09-12T19:00:28.786 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-17 2022-09-12T19:00:28.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:00:28.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:00:28.789 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:00:28.856 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:00:28.857 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:00:28.894 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:00:28.891+0000 7f0815222600 1 bucket sync caught up with source: 2022-09-12T19:00:28.895 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000010.31.5, , 00000000002.8.10, , 00000000007.20.9, , , , , ] 2022-09-12T19:00:28.895 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000010.31.5, , 00000000002.8.10, , 00000000007.20.9, , , , , ] 2022-09-12T19:00:28.895 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:00:28.891+0000 7f0815222600 0 bucket checkpoint complete 2022-09-12T19:00:28.908 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-17 2022-09-12T19:00:28.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:00:28.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:00:28.910 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:00:28.966 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:00:28.967 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:00:29.007 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:00:29.004+0000 7f01736fc600 1 bucket sync caught up with source: 2022-09-12T19:00:29.007 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000010.31.5, , 00000000002.8.10, , 00000000007.20.9, , , , , ] 2022-09-12T19:00:29.008 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000010.31.5, , 00000000002.8.10, , 00000000007.20.9, , , , , ] 2022-09-12T19:00:29.008 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:00:29.004+0000 7f01736fc600 0 bucket checkpoint complete 2022-09-12T19:00:29.021 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-17 2022-09-12T19:00:29.022 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:00:29.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:00:29.023 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:00:29.084 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:00:29.084 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:00:29.122 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:00:29.118+0000 7f9ff8db6600 1 bucket sync caught up with source: 2022-09-12T19:00:29.122 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.28.9, , 00000000010.39.5, , 00000000009.34.9, , , , , ] 2022-09-12T19:00:29.122 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.28.9, , 00000000010.39.5, , 00000000009.34.9, , , , , ] 2022-09-12T19:00:29.123 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:00:29.118+0000 7f9ff8db6600 0 bucket checkpoint complete 2022-09-12T19:00:29.135 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-17 2022-09-12T19:00:29.135 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:00:29.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:00:29.136 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:00:29.195 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:00:29.195 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:00:29.229 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:00:29.226+0000 7f79fd2aa600 1 bucket sync caught up with source: 2022-09-12T19:00:29.229 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.28.9, , 00000000010.39.5, , 00000000009.34.9, , , , , ] 2022-09-12T19:00:29.230 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.28.9, , 00000000010.39.5, , 00000000009.34.9, , , , , ] 2022-09-12T19:00:29.230 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:00:29.226+0000 7f79fd2aa600 0 bucket checkpoint complete 2022-09-12T19:00:29.245 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-17 2022-09-12T19:00:29.246 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-17 2022-09-12T19:00:29.247 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:00:29.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:00:29.249 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:00:29.308 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:00:29.308 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:00:29.353 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:00:29.350+0000 7f2833e5c600 1 waiting for incremental sync to catch up: 2022-09-12T19:00:29.353 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.24.9, , , , 00000000014.41.5, , , , , ] 2022-09-12T19:00:29.354 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:00:34.355 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:00:34.351+0000 7f2833e5c600 1 waiting for incremental sync to catch up: 2022-09-12T19:00:34.355 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.24.9, , , , 00000000014.41.5, , , , , ] 2022-09-12T19:00:34.355 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:00:39.357 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:00:39.353+0000 7f2833e5c600 1 waiting for incremental sync to catch up: 2022-09-12T19:00:39.358 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.24.9, , , , 00000000014.41.5, , , , , ] 2022-09-12T19:00:39.359 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:00:44.358 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:00:44.354+0000 7f2833e5c600 1 waiting for incremental sync to catch up: 2022-09-12T19:00:44.359 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.24.9, , , , 00000000014.41.5, , , , , ] 2022-09-12T19:00:44.359 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:00:49.359 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:00:49.356+0000 7f2833e5c600 1 waiting for incremental sync to catch up: 2022-09-12T19:00:49.360 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.24.9, , , , 00000000014.41.5, , , , , ] 2022-09-12T19:00:49.360 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:00:54.361 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:00:54.358+0000 7f2833e5c600 1 waiting for incremental sync to catch up: 2022-09-12T19:00:54.361 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.24.9, , , , 00000000014.41.5, , , , , ] 2022-09-12T19:00:54.362 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:00:59.363 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:00:59.360+0000 7f2833e5c600 1 waiting for incremental sync to catch up: 2022-09-12T19:00:59.363 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.24.9, , , , 00000000014.41.5, , , , , ] 2022-09-12T19:00:59.363 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:01:04.364 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:01:04.362+0000 7f2833e5c600 1 waiting for incremental sync to catch up: 2022-09-12T19:01:04.365 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.24.9, , , , 00000000014.41.5, , , , , ] 2022-09-12T19:01:04.365 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:01:09.366 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:01:09.363+0000 7f2833e5c600 1 waiting for incremental sync to catch up: 2022-09-12T19:01:09.366 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.24.9, , , , 00000000014.41.5, , , , , ] 2022-09-12T19:01:09.366 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:01:14.367 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:01:14.364+0000 7f2833e5c600 1 waiting for incremental sync to catch up: 2022-09-12T19:01:14.368 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.24.9, , , , 00000000014.41.5, , , , , ] 2022-09-12T19:01:14.368 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:01:19.369 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:01:19.366+0000 7f2833e5c600 1 waiting for incremental sync to catch up: 2022-09-12T19:01:19.369 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.24.9, , , , 00000000014.41.5, , , , , ] 2022-09-12T19:01:19.370 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:01:24.370 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:01:24.367+0000 7f2833e5c600 1 waiting for incremental sync to catch up: 2022-09-12T19:01:24.371 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.24.9, , , , 00000000014.41.5, , , , , ] 2022-09-12T19:01:24.371 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:01:29.372 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:01:29.369+0000 7f2833e5c600 1 waiting for incremental sync to catch up: 2022-09-12T19:01:29.373 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.24.9, , , , 00000000014.41.5, , , , , ] 2022-09-12T19:01:29.373 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:01:34.374 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:01:34.370+0000 7f2833e5c600 1 waiting for incremental sync to catch up: 2022-09-12T19:01:34.375 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.24.9, , , , 00000000014.41.5, , , , , ] 2022-09-12T19:01:34.375 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:01:39.375 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:01:39.372+0000 7f2833e5c600 1 waiting for incremental sync to catch up: 2022-09-12T19:01:39.376 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.24.9, , , , 00000000014.41.5, , , , , ] 2022-09-12T19:01:39.376 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:01:44.377 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:01:44.373+0000 7f2833e5c600 1 waiting for incremental sync to catch up: 2022-09-12T19:01:44.377 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.24.9, , , , 00000000014.41.5, , , , , ] 2022-09-12T19:01:44.377 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:01:49.378 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:01:49.376+0000 7f2833e5c600 1 waiting for incremental sync to catch up: 2022-09-12T19:01:49.379 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.24.9, , , , 00000000014.41.5, , , , , ] 2022-09-12T19:01:49.379 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:01:54.380 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:01:54.377+0000 7f2833e5c600 1 waiting for incremental sync to catch up: 2022-09-12T19:01:54.381 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.24.9, , , , 00000000014.41.5, , , , , ] 2022-09-12T19:01:54.381 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:01:59.381 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:01:59.379+0000 7f2833e5c600 1 waiting for incremental sync to catch up: 2022-09-12T19:01:59.382 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.24.9, , , , 00000000014.41.5, , , , , ] 2022-09-12T19:01:59.382 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:02:04.383 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:02:04.380+0000 7f2833e5c600 1 waiting for incremental sync to catch up: 2022-09-12T19:02:04.383 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.24.9, , , , 00000000014.41.5, , , , , ] 2022-09-12T19:02:04.384 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:02:09.385 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:02:09.381+0000 7f2833e5c600 1 waiting for incremental sync to catch up: 2022-09-12T19:02:09.385 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.24.9, , , , 00000000014.41.5, , , , , ] 2022-09-12T19:02:09.385 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:02:14.386 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:02:14.383+0000 7f2833e5c600 1 waiting for incremental sync to catch up: 2022-09-12T19:02:14.387 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.24.9, , , , 00000000014.41.5, , , , , ] 2022-09-12T19:02:14.387 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:02:19.388 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:02:19.385+0000 7f2833e5c600 1 waiting for incremental sync to catch up: 2022-09-12T19:02:19.389 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.24.9, , , , 00000000014.41.5, , , , , ] 2022-09-12T19:02:19.389 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:02:24.390 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:02:24.386+0000 7f2833e5c600 1 bucket sync caught up with source: 2022-09-12T19:02:24.390 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:02:24.391 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:02:24.391 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:02:24.386+0000 7f2833e5c600 0 bucket checkpoint complete 2022-09-12T19:02:24.408 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-17 2022-09-12T19:02:24.409 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:02:24.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:02:24.412 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:02:24.478 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:02:24.479 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:02:24.516 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:02:24.513+0000 7f49779ac600 1 bucket sync caught up with source: 2022-09-12T19:02:24.516 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:02:24.517 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000003.24.9, , 00000000003.27.9, , 00000000014.41.5, , , , , ] 2022-09-12T19:02:24.517 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:02:24.513+0000 7f49779ac600 0 bucket checkpoint complete 2022-09-12T19:02:24.528 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-17 2022-09-12T19:02:24.529 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-17 zones={test-zone2, test-zone1} 2022-09-12T19:02:24.530 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:24.531 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:24.531 DEBUG:boto:Method: HEAD 2022-09-12T19:02:24.532 DEBUG:boto:Path: /jdpiak-17/ 2022-09-12T19:02:24.532 DEBUG:boto:Data: 2022-09-12T19:02:24.533 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.533 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:24.534 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.534 DEBUG:boto:Params: {} 2022-09-12T19:02:24.535 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:02:24.536 DEBUG:boto:Token: None 2022-09-12T19:02:24.537 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/ 2022-09-12T19:02:24.537 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JGjNN8pWcWM79D2Q7aZj9De0+/g= 2022-09-12T19:02:24.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JGjNN8pWcWM79D2Q7aZj9De0+/g=', 'Content-Length': '0'} 2022-09-12T19:02:24.542 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000030e0ad5919aa6ff-00631f8240-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.543 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:24.543 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:24.544 DEBUG:boto:Method: HEAD 2022-09-12T19:02:24.544 DEBUG:boto:Path: /jdpiak-17/ 2022-09-12T19:02:24.545 DEBUG:boto:Data: 2022-09-12T19:02:24.545 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.546 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:24.546 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.547 DEBUG:boto:Params: {} 2022-09-12T19:02:24.548 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:02:24.549 DEBUG:boto:Token: None 2022-09-12T19:02:24.550 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/ 2022-09-12T19:02:24.550 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JGjNN8pWcWM79D2Q7aZj9De0+/g= 2022-09-12T19:02:24.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JGjNN8pWcWM79D2Q7aZj9De0+/g=', 'Content-Length': '0'} 2022-09-12T19:02:24.554 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ddf8f66f9016008d-00631f8240-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.555 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:02:24.556 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:24.556 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:24.557 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:24.557 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:24.558 DEBUG:boto:Method: GET 2022-09-12T19:02:24.558 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:24.559 DEBUG:boto:Data: 2022-09-12T19:02:24.560 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.561 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:24.562 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.562 DEBUG:boto:Params: {} 2022-09-12T19:02:24.563 DEBUG:boto:Token: None 2022-09-12T19:02:24.563 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/?versions 2022-09-12T19:02:24.564 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6PZytPOE22HhAKn6qaRBil++l5o= 2022-09-12T19:02:24.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6PZytPOE22HhAKn6qaRBil++l5o=', 'Content-Length': '0'} 2022-09-12T19:02:24.567 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000159d1ab6a4157412-00631f8240-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.567 DEBUG:boto:b'jdpiak-17999falsedummybSzfRw-Zsh7lYv-ClgdhoG098KzYg.Strue2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpktrue2022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone20f1rsOoGVVMRuyy11YfWbutRvo943lCtrue2022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3V6Ue4vhVY0WQ9FArGixLjY-hmND5K20true2022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:24.569 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:02:24.569 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:02:24.570 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:02:24.570 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:02:24.571 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:02:24.571 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:24.572 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:24.572 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:24.573 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:24.573 DEBUG:boto:Method: GET 2022-09-12T19:02:24.574 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:24.574 DEBUG:boto:Data: 2022-09-12T19:02:24.575 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.575 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:24.576 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.576 DEBUG:boto:Params: {} 2022-09-12T19:02:24.577 DEBUG:boto:Token: None 2022-09-12T19:02:24.577 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/?versions 2022-09-12T19:02:24.578 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6PZytPOE22HhAKn6qaRBil++l5o= 2022-09-12T19:02:24.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6PZytPOE22HhAKn6qaRBil++l5o=', 'Content-Length': '0'} 2022-09-12T19:02:24.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a155df5b1420d3fb-00631f8240-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.582 DEBUG:boto:b'jdpiak-17999falsedummybSzfRw-Zsh7lYv-ClgdhoG098KzYg.Strue2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpktrue2022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone20f1rsOoGVVMRuyy11YfWbutRvo943lCtrue2022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3V6Ue4vhVY0WQ9FArGixLjY-hmND5K20true2022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:24.583 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:02:24.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:02:24.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:02:24.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:02:24.585 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:24.585 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:24.586 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:24.586 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:24.587 DEBUG:boto:Method: GET 2022-09-12T19:02:24.587 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:24.588 DEBUG:boto:Data: 2022-09-12T19:02:24.588 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.589 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:24.589 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.590 DEBUG:boto:Params: {} 2022-09-12T19:02:24.590 DEBUG:boto:Token: None 2022-09-12T19:02:24.591 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/?versions 2022-09-12T19:02:24.591 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6PZytPOE22HhAKn6qaRBil++l5o= 2022-09-12T19:02:24.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6PZytPOE22HhAKn6qaRBil++l5o=', 'Content-Length': '0'} 2022-09-12T19:02:24.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b8cc33a7e97bc51-00631f8240-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.595 DEBUG:boto:b'jdpiak-17999falsedummybSzfRw-Zsh7lYv-ClgdhoG098KzYg.Strue2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpktrue2022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone20f1rsOoGVVMRuyy11YfWbutRvo943lCtrue2022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3V6Ue4vhVY0WQ9FArGixLjY-hmND5K20true2022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:24.596 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:24.596 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:24.597 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:24.597 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:24.598 DEBUG:boto:Method: GET 2022-09-12T19:02:24.598 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:24.599 DEBUG:boto:Data: 2022-09-12T19:02:24.599 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.600 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:24.600 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.601 DEBUG:boto:Params: {} 2022-09-12T19:02:24.601 DEBUG:boto:Token: None 2022-09-12T19:02:24.602 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/?versions 2022-09-12T19:02:24.602 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6PZytPOE22HhAKn6qaRBil++l5o= 2022-09-12T19:02:24.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6PZytPOE22HhAKn6qaRBil++l5o=', 'Content-Length': '0'} 2022-09-12T19:02:24.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dcbc226e1417b14-00631f8240-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.606 DEBUG:boto:b'jdpiak-17999falsedummybSzfRw-Zsh7lYv-ClgdhoG098KzYg.Strue2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpktrue2022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone20f1rsOoGVVMRuyy11YfWbutRvo943lCtrue2022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3V6Ue4vhVY0WQ9FArGixLjY-hmND5K20true2022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:24.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:02:24.608 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:24.609 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:24.609 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.610 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.611 DEBUG:boto:Method: GET 2022-09-12T19:02:24.612 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.613 DEBUG:boto:Data: 2022-09-12T19:02:24.614 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.615 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:24.616 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.616 DEBUG:boto:Params: {} 2022-09-12T19:02:24.617 DEBUG:boto:Token: None 2022-09-12T19:02:24.617 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.618 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mosCBm/vmZN1m+47cA1Auo4r1+U= 2022-09-12T19:02:24.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mosCBm/vmZN1m+47cA1Auo4r1+U=', 'Content-Length': '0'} 2022-09-12T19:02:24.621 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cd74a0eb921269ce-00631f8240-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.621 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:24.622 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:24.623 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.623 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.624 DEBUG:boto:Method: GET 2022-09-12T19:02:24.624 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.625 DEBUG:boto:Data: 2022-09-12T19:02:24.625 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.626 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:24.626 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.627 DEBUG:boto:Params: {} 2022-09-12T19:02:24.628 DEBUG:boto:Token: None 2022-09-12T19:02:24.628 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.629 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mosCBm/vmZN1m+47cA1Auo4r1+U= 2022-09-12T19:02:24.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mosCBm/vmZN1m+47cA1Auo4r1+U=', 'Content-Length': '0'} 2022-09-12T19:02:24.632 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002891a5e041fc3fa8-00631f8240-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.632 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:24.633 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:24.633 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.634 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.634 DEBUG:boto:Method: HEAD 2022-09-12T19:02:24.635 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.635 DEBUG:boto:Data: 2022-09-12T19:02:24.636 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.636 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:24.637 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.638 DEBUG:boto:Params: {} 2022-09-12T19:02:24.638 DEBUG:boto:Token: None 2022-09-12T19:02:24.639 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.639 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:485EdhgbNGK8/6SAoP+cm/XM3hA= 2022-09-12T19:02:24.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:485EdhgbNGK8/6SAoP+cm/XM3hA=', 'Content-Length': '0'} 2022-09-12T19:02:24.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001d908a2102773083-00631f8240-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.643 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:24.643 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:24.644 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.644 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.645 DEBUG:boto:Method: HEAD 2022-09-12T19:02:24.645 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.646 DEBUG:boto:Data: 2022-09-12T19:02:24.646 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.647 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:24.647 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.648 DEBUG:boto:Params: {} 2022-09-12T19:02:24.649 DEBUG:boto:Token: None 2022-09-12T19:02:24.649 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.650 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:485EdhgbNGK8/6SAoP+cm/XM3hA= 2022-09-12T19:02:24.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:485EdhgbNGK8/6SAoP+cm/XM3hA=', 'Content-Length': '0'} 2022-09-12T19:02:24.653 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000657b0422c8ae89ce-00631f8240-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:02:24.654 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:24.655 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:24.655 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.656 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.656 DEBUG:boto:Method: GET 2022-09-12T19:02:24.657 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.657 DEBUG:boto:Data: 2022-09-12T19:02:24.658 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.658 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:24.659 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.659 DEBUG:boto:Params: {} 2022-09-12T19:02:24.660 DEBUG:boto:Token: None 2022-09-12T19:02:24.661 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.661 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mosCBm/vmZN1m+47cA1Auo4r1+U= 2022-09-12T19:02:24.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mosCBm/vmZN1m+47cA1Auo4r1+U=', 'Content-Length': '0'} 2022-09-12T19:02:24.664 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004f1310e81d2af2c4-00631f8240-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.665 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:24.665 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:24.666 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.666 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.667 DEBUG:boto:Method: GET 2022-09-12T19:02:24.667 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.668 DEBUG:boto:Data: 2022-09-12T19:02:24.668 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.669 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:24.669 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.670 DEBUG:boto:Params: {} 2022-09-12T19:02:24.671 DEBUG:boto:Token: None 2022-09-12T19:02:24.671 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.672 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mosCBm/vmZN1m+47cA1Auo4r1+U= 2022-09-12T19:02:24.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mosCBm/vmZN1m+47cA1Auo4r1+U=', 'Content-Length': '0'} 2022-09-12T19:02:24.675 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c60da0d7b942206e-00631f8240-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.676 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:24.676 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:24.677 DEBUG:boto:path=/jdpiak-17/?prefix=dummy 2022-09-12T19:02:24.677 DEBUG:boto:auth_path=/jdpiak-17/?prefix=dummy 2022-09-12T19:02:24.678 DEBUG:boto:Method: GET 2022-09-12T19:02:24.678 DEBUG:boto:Path: /jdpiak-17/?prefix=dummy 2022-09-12T19:02:24.679 DEBUG:boto:Data: 2022-09-12T19:02:24.679 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.680 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:24.680 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.681 DEBUG:boto:Params: {} 2022-09-12T19:02:24.682 DEBUG:boto:Token: None 2022-09-12T19:02:24.682 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/ 2022-09-12T19:02:24.683 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:i1lauI6yFvaaKrB6jyIGIQ/9St8= 2022-09-12T19:02:24.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:i1lauI6yFvaaKrB6jyIGIQ/9St8=', 'Content-Length': '0'} 2022-09-12T19:02:24.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f020a8041f78e92-00631f8240-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.730 DEBUG:boto:b'jdpiak-17dummy1000falsedummy2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:24.731 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:24.732 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:24.733 DEBUG:boto:path=/jdpiak-17/?prefix=dummy 2022-09-12T19:02:24.733 DEBUG:boto:auth_path=/jdpiak-17/?prefix=dummy 2022-09-12T19:02:24.734 DEBUG:boto:Method: GET 2022-09-12T19:02:24.735 DEBUG:boto:Path: /jdpiak-17/?prefix=dummy 2022-09-12T19:02:24.735 DEBUG:boto:Data: 2022-09-12T19:02:24.736 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.736 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:24.737 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.737 DEBUG:boto:Params: {} 2022-09-12T19:02:24.738 DEBUG:boto:Token: None 2022-09-12T19:02:24.739 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/ 2022-09-12T19:02:24.740 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:i1lauI6yFvaaKrB6jyIGIQ/9St8= 2022-09-12T19:02:24.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:i1lauI6yFvaaKrB6jyIGIQ/9St8=', 'Content-Length': '0'} 2022-09-12T19:02:24.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce7ac44d81f3cb2f-00631f8240-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.744 DEBUG:boto:b'jdpiak-17dummy1000falsedummy2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:24.745 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:24.745 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:24.746 DEBUG:boto:Method: HEAD 2022-09-12T19:02:24.746 DEBUG:boto:Path: /jdpiak-17/dummy 2022-09-12T19:02:24.747 DEBUG:boto:Data: 2022-09-12T19:02:24.748 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.748 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:24.749 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.749 DEBUG:boto:Params: {} 2022-09-12T19:02:24.750 DEBUG:boto:Token: None 2022-09-12T19:02:24.751 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/dummy 2022-09-12T19:02:24.751 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q8OFSuwqUFolo21obavSJg17im4= 2022-09-12T19:02:24.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q8OFSuwqUFolo21obavSJg17im4=', 'Content-Length': '0'} 2022-09-12T19:02:24.755 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000038697426c4ec456f-00631f8240-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.756 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:24.756 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:24.757 DEBUG:boto:Method: HEAD 2022-09-12T19:02:24.758 DEBUG:boto:Path: /jdpiak-17/dummy 2022-09-12T19:02:24.758 DEBUG:boto:Data: 2022-09-12T19:02:24.759 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.759 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:24.760 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.760 DEBUG:boto:Params: {} 2022-09-12T19:02:24.761 DEBUG:boto:Token: None 2022-09-12T19:02:24.761 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/dummy 2022-09-12T19:02:24.762 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q8OFSuwqUFolo21obavSJg17im4= 2022-09-12T19:02:24.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q8OFSuwqUFolo21obavSJg17im4=', 'Content-Length': '0'} 2022-09-12T19:02:24.765 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ff0abe63b88c0ca6-00631f8240-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:02:24.767 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:24.768 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:24.769 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.770 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.772 DEBUG:boto:Method: GET 2022-09-12T19:02:24.773 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.774 DEBUG:boto:Data: 2022-09-12T19:02:24.775 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.776 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:24.776 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.777 DEBUG:boto:Params: {} 2022-09-12T19:02:24.778 DEBUG:boto:Token: None 2022-09-12T19:02:24.779 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.780 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mosCBm/vmZN1m+47cA1Auo4r1+U= 2022-09-12T19:02:24.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mosCBm/vmZN1m+47cA1Auo4r1+U=', 'Content-Length': '0'} 2022-09-12T19:02:24.783 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006b0a1a979c88ec54-00631f8240-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.783 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:24.784 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:24.785 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.785 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.786 DEBUG:boto:Method: GET 2022-09-12T19:02:24.786 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.787 DEBUG:boto:Data: 2022-09-12T19:02:24.788 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.789 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:24.789 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.790 DEBUG:boto:Params: {} 2022-09-12T19:02:24.790 DEBUG:boto:Token: None 2022-09-12T19:02:24.791 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:24.791 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mosCBm/vmZN1m+47cA1Auo4r1+U= 2022-09-12T19:02:24.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mosCBm/vmZN1m+47cA1Auo4r1+U=', 'Content-Length': '0'} 2022-09-12T19:02:24.794 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003ae77e517c8ef7c6-00631f8240-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.795 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:24.795 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:24.796 DEBUG:boto:path=/jdpiak-17/?prefix=dummy 2022-09-12T19:02:24.796 DEBUG:boto:auth_path=/jdpiak-17/?prefix=dummy 2022-09-12T19:02:24.797 DEBUG:boto:Method: GET 2022-09-12T19:02:24.798 DEBUG:boto:Path: /jdpiak-17/?prefix=dummy 2022-09-12T19:02:24.798 DEBUG:boto:Data: 2022-09-12T19:02:24.799 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.799 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:24.800 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.800 DEBUG:boto:Params: {} 2022-09-12T19:02:24.801 DEBUG:boto:Token: None 2022-09-12T19:02:24.802 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/ 2022-09-12T19:02:24.802 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:i1lauI6yFvaaKrB6jyIGIQ/9St8= 2022-09-12T19:02:24.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:i1lauI6yFvaaKrB6jyIGIQ/9St8=', 'Content-Length': '0'} 2022-09-12T19:02:24.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059631bd80de6f6b8-00631f8240-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.850 DEBUG:boto:b'jdpiak-17dummy1000falsedummy2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:24.851 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:24.852 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:24.853 DEBUG:boto:path=/jdpiak-17/?prefix=dummy 2022-09-12T19:02:24.853 DEBUG:boto:auth_path=/jdpiak-17/?prefix=dummy 2022-09-12T19:02:24.854 DEBUG:boto:Method: GET 2022-09-12T19:02:24.854 DEBUG:boto:Path: /jdpiak-17/?prefix=dummy 2022-09-12T19:02:24.855 DEBUG:boto:Data: 2022-09-12T19:02:24.855 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.856 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:24.857 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.857 DEBUG:boto:Params: {} 2022-09-12T19:02:24.858 DEBUG:boto:Token: None 2022-09-12T19:02:24.858 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/ 2022-09-12T19:02:24.859 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:i1lauI6yFvaaKrB6jyIGIQ/9St8= 2022-09-12T19:02:24.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:i1lauI6yFvaaKrB6jyIGIQ/9St8=', 'Content-Length': '0'} 2022-09-12T19:02:24.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028c44a6ebad4bc8d-00631f8240-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.864 DEBUG:boto:b'jdpiak-17dummy1000falsedummy2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:24.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:02:24.865 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:24.865 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:24.866 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.867 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.867 DEBUG:boto:Method: GET 2022-09-12T19:02:24.868 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.868 DEBUG:boto:Data: 2022-09-12T19:02:24.869 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.869 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:24.870 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.870 DEBUG:boto:Params: {} 2022-09-12T19:02:24.871 DEBUG:boto:Token: None 2022-09-12T19:02:24.872 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.872 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3TuofdJhx7xhMeJZKLOOutePjEk= 2022-09-12T19:02:24.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3TuofdJhx7xhMeJZKLOOutePjEk=', 'Content-Length': '0'} 2022-09-12T19:02:24.875 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b5eca51caf08a8e-00631f8240-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.876 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:24.877 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:24.877 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.878 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.878 DEBUG:boto:Method: GET 2022-09-12T19:02:24.879 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.879 DEBUG:boto:Data: 2022-09-12T19:02:24.880 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.880 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:24.882 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.883 DEBUG:boto:Params: {} 2022-09-12T19:02:24.883 DEBUG:boto:Token: None 2022-09-12T19:02:24.884 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.885 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3TuofdJhx7xhMeJZKLOOutePjEk= 2022-09-12T19:02:24.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3TuofdJhx7xhMeJZKLOOutePjEk=', 'Content-Length': '0'} 2022-09-12T19:02:24.888 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007dcbe786346c89a8-00631f8240-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.889 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:24.889 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:24.890 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.890 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.891 DEBUG:boto:Method: HEAD 2022-09-12T19:02:24.891 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.892 DEBUG:boto:Data: 2022-09-12T19:02:24.892 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.893 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:24.893 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.894 DEBUG:boto:Params: {} 2022-09-12T19:02:24.894 DEBUG:boto:Token: None 2022-09-12T19:02:24.895 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.895 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GeUbk2ohoiMcAleDFvpL4zeWAAs= 2022-09-12T19:02:24.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GeUbk2ohoiMcAleDFvpL4zeWAAs=', 'Content-Length': '0'} 2022-09-12T19:02:24.898 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea9787655a19d69b-00631f8240-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.898 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:24.899 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:24.899 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.900 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.900 DEBUG:boto:Method: HEAD 2022-09-12T19:02:24.901 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.901 DEBUG:boto:Data: 2022-09-12T19:02:24.902 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.902 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:24.903 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.904 DEBUG:boto:Params: {} 2022-09-12T19:02:24.904 DEBUG:boto:Token: None 2022-09-12T19:02:24.905 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.905 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GeUbk2ohoiMcAleDFvpL4zeWAAs= 2022-09-12T19:02:24.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GeUbk2ohoiMcAleDFvpL4zeWAAs=', 'Content-Length': '0'} 2022-09-12T19:02:24.908 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001542fa85eb8f5431-00631f8240-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:02:24.909 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:24.910 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:24.911 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.911 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.912 DEBUG:boto:Method: GET 2022-09-12T19:02:24.912 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.913 DEBUG:boto:Data: 2022-09-12T19:02:24.913 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.914 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:24.914 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.915 DEBUG:boto:Params: {} 2022-09-12T19:02:24.915 DEBUG:boto:Token: None 2022-09-12T19:02:24.916 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.917 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3TuofdJhx7xhMeJZKLOOutePjEk= 2022-09-12T19:02:24.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3TuofdJhx7xhMeJZKLOOutePjEk=', 'Content-Length': '0'} 2022-09-12T19:02:24.919 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b165acc3ceceffcf-00631f8240-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.920 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:24.920 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:24.921 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.921 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.922 DEBUG:boto:Method: GET 2022-09-12T19:02:24.923 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.923 DEBUG:boto:Data: 2022-09-12T19:02:24.924 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.924 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:24.925 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.925 DEBUG:boto:Params: {} 2022-09-12T19:02:24.926 DEBUG:boto:Token: None 2022-09-12T19:02:24.926 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.927 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3TuofdJhx7xhMeJZKLOOutePjEk= 2022-09-12T19:02:24.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3TuofdJhx7xhMeJZKLOOutePjEk=', 'Content-Length': '0'} 2022-09-12T19:02:24.930 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e6e3cedb50bc7a9-00631f8240-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.931 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:24.931 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:24.932 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:24.932 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:24.933 DEBUG:boto:Method: GET 2022-09-12T19:02:24.933 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:24.934 DEBUG:boto:Data: 2022-09-12T19:02:24.934 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.935 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:24.935 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.936 DEBUG:boto:Params: {} 2022-09-12T19:02:24.936 DEBUG:boto:Token: None 2022-09-12T19:02:24.937 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/ 2022-09-12T19:02:24.938 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:i1lauI6yFvaaKrB6jyIGIQ/9St8= 2022-09-12T19:02:24.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:i1lauI6yFvaaKrB6jyIGIQ/9St8=', 'Content-Length': '0'} 2022-09-12T19:02:24.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ed4844ee9236f94-00631f8240-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.941 DEBUG:boto:b'jdpiak-17obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:24.942 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:24.943 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:24.943 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:24.944 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:24.944 DEBUG:boto:Method: GET 2022-09-12T19:02:24.945 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:24.945 DEBUG:boto:Data: 2022-09-12T19:02:24.946 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.946 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:24.947 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.947 DEBUG:boto:Params: {} 2022-09-12T19:02:24.948 DEBUG:boto:Token: None 2022-09-12T19:02:24.948 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/ 2022-09-12T19:02:24.949 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:i1lauI6yFvaaKrB6jyIGIQ/9St8= 2022-09-12T19:02:24.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:i1lauI6yFvaaKrB6jyIGIQ/9St8=', 'Content-Length': '0'} 2022-09-12T19:02:24.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c796961d405b9898-00631f8240-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.954 DEBUG:boto:b'jdpiak-17obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:24.954 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:24.955 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:24.955 DEBUG:boto:Method: HEAD 2022-09-12T19:02:24.956 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1 2022-09-12T19:02:24.956 DEBUG:boto:Data: 2022-09-12T19:02:24.957 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.957 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:24.958 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.958 DEBUG:boto:Params: {} 2022-09-12T19:02:24.959 DEBUG:boto:Token: None 2022-09-12T19:02:24.959 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/obj-test-zone1 2022-09-12T19:02:24.960 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hcKJppjxi2ygu3GaK0omCbXMduI= 2022-09-12T19:02:24.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hcKJppjxi2ygu3GaK0omCbXMduI=', 'Content-Length': '0'} 2022-09-12T19:02:24.963 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cdc1925307d82888-00631f8240-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.963 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:24.964 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:24.964 DEBUG:boto:Method: HEAD 2022-09-12T19:02:24.965 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1 2022-09-12T19:02:24.965 DEBUG:boto:Data: 2022-09-12T19:02:24.966 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.966 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:24.967 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.967 DEBUG:boto:Params: {} 2022-09-12T19:02:24.968 DEBUG:boto:Token: None 2022-09-12T19:02:24.968 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/obj-test-zone1 2022-09-12T19:02:24.969 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hcKJppjxi2ygu3GaK0omCbXMduI= 2022-09-12T19:02:24.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hcKJppjxi2ygu3GaK0omCbXMduI=', 'Content-Length': '0'} 2022-09-12T19:02:24.972 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f5556e0fa220ae4-00631f8240-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:02:24.973 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:24.974 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:24.975 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.975 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.976 DEBUG:boto:Method: GET 2022-09-12T19:02:24.977 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.977 DEBUG:boto:Data: 2022-09-12T19:02:24.978 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.978 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:24.979 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.979 DEBUG:boto:Params: {} 2022-09-12T19:02:24.980 DEBUG:boto:Token: None 2022-09-12T19:02:24.980 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.981 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3TuofdJhx7xhMeJZKLOOutePjEk= 2022-09-12T19:02:24.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3TuofdJhx7xhMeJZKLOOutePjEk=', 'Content-Length': '0'} 2022-09-12T19:02:24.984 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099387e5032fa1d7a-00631f8240-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.984 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:24.985 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:24.985 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.986 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.987 DEBUG:boto:Method: GET 2022-09-12T19:02:24.987 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.988 DEBUG:boto:Data: 2022-09-12T19:02:24.988 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.989 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:24.989 DEBUG:boto:Port: 8000 2022-09-12T19:02:24.990 DEBUG:boto:Params: {} 2022-09-12T19:02:24.990 DEBUG:boto:Token: None 2022-09-12T19:02:24.991 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:24 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:24.991 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3TuofdJhx7xhMeJZKLOOutePjEk= 2022-09-12T19:02:24.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3TuofdJhx7xhMeJZKLOOutePjEk=', 'Content-Length': '0'} 2022-09-12T19:02:24.994 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c5df6e60882d84b-00631f8240-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:24.995 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:24.995 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:24.996 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:24.996 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:24.997 DEBUG:boto:Method: GET 2022-09-12T19:02:24.997 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:24.998 DEBUG:boto:Data: 2022-09-12T19:02:24.998 DEBUG:boto:Headers: {} 2022-09-12T19:02:24.999 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:24.999 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.000 DEBUG:boto:Params: {} 2022-09-12T19:02:25.000 DEBUG:boto:Token: None 2022-09-12T19:02:25.001 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/ 2022-09-12T19:02:25.002 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y= 2022-09-12T19:02:25.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y=', 'Content-Length': '0'} 2022-09-12T19:02:25.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005402f62b4cd4c7d8-00631f8241-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.006 DEBUG:boto:b'jdpiak-17obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:25.006 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:25.007 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:25.007 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:25.008 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:25.008 DEBUG:boto:Method: GET 2022-09-12T19:02:25.009 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:25.009 DEBUG:boto:Data: 2022-09-12T19:02:25.010 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.010 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.011 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.012 DEBUG:boto:Params: {} 2022-09-12T19:02:25.012 DEBUG:boto:Token: None 2022-09-12T19:02:25.013 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/ 2022-09-12T19:02:25.013 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y= 2022-09-12T19:02:25.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y=', 'Content-Length': '0'} 2022-09-12T19:02:25.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006eb69d00f3229f18-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.058 DEBUG:boto:b'jdpiak-17obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:25.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:02:25.060 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:25.060 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:25.061 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.061 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.062 DEBUG:boto:Method: GET 2022-09-12T19:02:25.062 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.063 DEBUG:boto:Data: 2022-09-12T19:02:25.063 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.064 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:25.064 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.065 DEBUG:boto:Params: {} 2022-09-12T19:02:25.065 DEBUG:boto:Token: None 2022-09-12T19:02:25.066 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.067 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XgyED+6LQ0p9WOwjYvU8tf8IeDg= 2022-09-12T19:02:25.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XgyED+6LQ0p9WOwjYvU8tf8IeDg=', 'Content-Length': '0'} 2022-09-12T19:02:25.070 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c46ec419766115d9-00631f8241-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.070 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:25.071 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:25.071 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.072 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.073 DEBUG:boto:Method: GET 2022-09-12T19:02:25.073 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.074 DEBUG:boto:Data: 2022-09-12T19:02:25.074 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.075 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.075 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.076 DEBUG:boto:Params: {} 2022-09-12T19:02:25.076 DEBUG:boto:Token: None 2022-09-12T19:02:25.077 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.077 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XgyED+6LQ0p9WOwjYvU8tf8IeDg= 2022-09-12T19:02:25.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XgyED+6LQ0p9WOwjYvU8tf8IeDg=', 'Content-Length': '0'} 2022-09-12T19:02:25.080 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2d3c986b15b6cb5-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.081 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:25.082 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:25.082 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.083 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.084 DEBUG:boto:Method: HEAD 2022-09-12T19:02:25.084 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.085 DEBUG:boto:Data: 2022-09-12T19:02:25.085 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.086 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:25.086 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.087 DEBUG:boto:Params: {} 2022-09-12T19:02:25.087 DEBUG:boto:Token: None 2022-09-12T19:02:25.088 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.088 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iu9qhnGtkVKY2dqovUn9aYItywA= 2022-09-12T19:02:25.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iu9qhnGtkVKY2dqovUn9aYItywA=', 'Content-Length': '0'} 2022-09-12T19:02:25.091 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e48d404e58e35ef-00631f8241-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.092 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:25.093 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:25.093 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.094 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.094 DEBUG:boto:Method: HEAD 2022-09-12T19:02:25.095 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.096 DEBUG:boto:Data: 2022-09-12T19:02:25.096 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.097 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.097 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.098 DEBUG:boto:Params: {} 2022-09-12T19:02:25.098 DEBUG:boto:Token: None 2022-09-12T19:02:25.099 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.099 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iu9qhnGtkVKY2dqovUn9aYItywA= 2022-09-12T19:02:25.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iu9qhnGtkVKY2dqovUn9aYItywA=', 'Content-Length': '0'} 2022-09-12T19:02:25.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049577e8b790d48a6-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:02:25.103 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:25.104 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:25.104 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.105 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.105 DEBUG:boto:Method: GET 2022-09-12T19:02:25.106 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.106 DEBUG:boto:Data: 2022-09-12T19:02:25.107 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.108 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:25.108 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.109 DEBUG:boto:Params: {} 2022-09-12T19:02:25.110 DEBUG:boto:Token: None 2022-09-12T19:02:25.110 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.111 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XgyED+6LQ0p9WOwjYvU8tf8IeDg= 2022-09-12T19:02:25.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XgyED+6LQ0p9WOwjYvU8tf8IeDg=', 'Content-Length': '0'} 2022-09-12T19:02:25.114 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080394b67e228117b-00631f8241-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.115 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:25.115 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:25.116 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.117 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.117 DEBUG:boto:Method: GET 2022-09-12T19:02:25.118 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.118 DEBUG:boto:Data: 2022-09-12T19:02:25.119 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.119 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.120 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.120 DEBUG:boto:Params: {} 2022-09-12T19:02:25.121 DEBUG:boto:Token: None 2022-09-12T19:02:25.122 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.122 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XgyED+6LQ0p9WOwjYvU8tf8IeDg= 2022-09-12T19:02:25.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XgyED+6LQ0p9WOwjYvU8tf8IeDg=', 'Content-Length': '0'} 2022-09-12T19:02:25.125 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f38114ef710a67a3-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.126 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:25.127 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:25.127 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:25.128 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:25.128 DEBUG:boto:Method: GET 2022-09-12T19:02:25.129 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:25.129 DEBUG:boto:Data: 2022-09-12T19:02:25.130 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.130 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:25.131 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.131 DEBUG:boto:Params: {} 2022-09-12T19:02:25.132 DEBUG:boto:Token: None 2022-09-12T19:02:25.132 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/ 2022-09-12T19:02:25.133 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y= 2022-09-12T19:02:25.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y=', 'Content-Length': '0'} 2022-09-12T19:02:25.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd2eb77617668197-00631f8241-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.178 DEBUG:boto:b'jdpiak-17obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:25.179 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:25.180 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:25.180 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:25.181 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:25.181 DEBUG:boto:Method: GET 2022-09-12T19:02:25.182 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:25.183 DEBUG:boto:Data: 2022-09-12T19:02:25.183 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.184 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.184 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.185 DEBUG:boto:Params: {} 2022-09-12T19:02:25.185 DEBUG:boto:Token: None 2022-09-12T19:02:25.186 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/ 2022-09-12T19:02:25.187 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y= 2022-09-12T19:02:25.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y=', 'Content-Length': '0'} 2022-09-12T19:02:25.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ad29378025565d9-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.191 DEBUG:boto:b'jdpiak-17obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:25.192 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:25.192 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:25.193 DEBUG:boto:Method: HEAD 2022-09-12T19:02:25.194 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2 2022-09-12T19:02:25.194 DEBUG:boto:Data: 2022-09-12T19:02:25.195 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.195 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:25.196 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.196 DEBUG:boto:Params: {} 2022-09-12T19:02:25.197 DEBUG:boto:Token: None 2022-09-12T19:02:25.197 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone2 2022-09-12T19:02:25.198 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ojSfHozEFVG+QzBwgmeTIJAMDdM= 2022-09-12T19:02:25.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ojSfHozEFVG+QzBwgmeTIJAMDdM=', 'Content-Length': '0'} 2022-09-12T19:02:25.201 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6bb4643e71a8867-00631f8241-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.202 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:25.202 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:25.203 DEBUG:boto:Method: HEAD 2022-09-12T19:02:25.203 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2 2022-09-12T19:02:25.204 DEBUG:boto:Data: 2022-09-12T19:02:25.204 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.205 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.205 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.206 DEBUG:boto:Params: {} 2022-09-12T19:02:25.206 DEBUG:boto:Token: None 2022-09-12T19:02:25.207 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone2 2022-09-12T19:02:25.208 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ojSfHozEFVG+QzBwgmeTIJAMDdM= 2022-09-12T19:02:25.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ojSfHozEFVG+QzBwgmeTIJAMDdM=', 'Content-Length': '0'} 2022-09-12T19:02:25.211 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a651e2f7fadb703-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:02:25.212 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:25.213 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:25.213 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.214 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.215 DEBUG:boto:Method: GET 2022-09-12T19:02:25.215 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.216 DEBUG:boto:Data: 2022-09-12T19:02:25.217 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.217 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:25.218 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.219 DEBUG:boto:Params: {} 2022-09-12T19:02:25.219 DEBUG:boto:Token: None 2022-09-12T19:02:25.220 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.221 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XgyED+6LQ0p9WOwjYvU8tf8IeDg= 2022-09-12T19:02:25.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XgyED+6LQ0p9WOwjYvU8tf8IeDg=', 'Content-Length': '0'} 2022-09-12T19:02:25.224 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007355c877bc63193c-00631f8241-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.225 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:25.225 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:25.226 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.227 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.227 DEBUG:boto:Method: GET 2022-09-12T19:02:25.228 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.228 DEBUG:boto:Data: 2022-09-12T19:02:25.229 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.230 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.230 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.231 DEBUG:boto:Params: {} 2022-09-12T19:02:25.232 DEBUG:boto:Token: None 2022-09-12T19:02:25.232 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:25.233 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XgyED+6LQ0p9WOwjYvU8tf8IeDg= 2022-09-12T19:02:25.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XgyED+6LQ0p9WOwjYvU8tf8IeDg=', 'Content-Length': '0'} 2022-09-12T19:02:25.236 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cddca28468f42b73-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.278 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:25.280 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:25.281 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:25.282 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:25.283 DEBUG:boto:Method: GET 2022-09-12T19:02:25.284 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:25.285 DEBUG:boto:Data: 2022-09-12T19:02:25.286 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.287 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:25.288 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.289 DEBUG:boto:Params: {} 2022-09-12T19:02:25.290 DEBUG:boto:Token: None 2022-09-12T19:02:25.290 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/ 2022-09-12T19:02:25.292 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y= 2022-09-12T19:02:25.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y=', 'Content-Length': '0'} 2022-09-12T19:02:25.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ec07d94c3e79b34-00631f8241-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.301 DEBUG:boto:b'jdpiak-17obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:25.302 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:25.302 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:25.303 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:25.303 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:25.304 DEBUG:boto:Method: GET 2022-09-12T19:02:25.305 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:25.306 DEBUG:boto:Data: 2022-09-12T19:02:25.306 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.307 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.308 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.309 DEBUG:boto:Params: {} 2022-09-12T19:02:25.310 DEBUG:boto:Token: None 2022-09-12T19:02:25.311 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/ 2022-09-12T19:02:25.312 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y= 2022-09-12T19:02:25.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y=', 'Content-Length': '0'} 2022-09-12T19:02:25.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9d858288da994bc-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.316 DEBUG:boto:b'jdpiak-17obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:25.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:02:25.318 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:25.319 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:25.320 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.321 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.322 DEBUG:boto:Method: GET 2022-09-12T19:02:25.323 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.323 DEBUG:boto:Data: 2022-09-12T19:02:25.324 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.325 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:25.326 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.326 DEBUG:boto:Params: {} 2022-09-12T19:02:25.327 DEBUG:boto:Token: None 2022-09-12T19:02:25.328 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.329 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2pQFh6gi6VPorkMvJG4H9ClpIxY= 2022-09-12T19:02:25.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2pQFh6gi6VPorkMvJG4H9ClpIxY=', 'Content-Length': '0'} 2022-09-12T19:02:25.333 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e091f15d98ca569-00631f8241-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.334 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:25.335 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:25.336 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.337 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.338 DEBUG:boto:Method: GET 2022-09-12T19:02:25.339 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.340 DEBUG:boto:Data: 2022-09-12T19:02:25.341 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.342 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.343 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.344 DEBUG:boto:Params: {} 2022-09-12T19:02:25.345 DEBUG:boto:Token: None 2022-09-12T19:02:25.345 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.346 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2pQFh6gi6VPorkMvJG4H9ClpIxY= 2022-09-12T19:02:25.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2pQFh6gi6VPorkMvJG4H9ClpIxY=', 'Content-Length': '0'} 2022-09-12T19:02:25.349 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008005630f43419e31-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.351 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:25.351 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:25.352 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.353 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.353 DEBUG:boto:Method: HEAD 2022-09-12T19:02:25.354 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.355 DEBUG:boto:Data: 2022-09-12T19:02:25.355 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.356 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:25.356 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.357 DEBUG:boto:Params: {} 2022-09-12T19:02:25.358 DEBUG:boto:Token: None 2022-09-12T19:02:25.358 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.359 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mszjlQhmv4scusRxs1TaYuOhj78= 2022-09-12T19:02:25.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mszjlQhmv4scusRxs1TaYuOhj78=', 'Content-Length': '0'} 2022-09-12T19:02:25.362 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000def017cd4d987f11-00631f8241-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.363 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:25.364 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:25.364 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.365 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.366 DEBUG:boto:Method: HEAD 2022-09-12T19:02:25.366 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.367 DEBUG:boto:Data: 2022-09-12T19:02:25.367 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.368 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.368 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.369 DEBUG:boto:Params: {} 2022-09-12T19:02:25.370 DEBUG:boto:Token: None 2022-09-12T19:02:25.370 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.371 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mszjlQhmv4scusRxs1TaYuOhj78= 2022-09-12T19:02:25.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mszjlQhmv4scusRxs1TaYuOhj78=', 'Content-Length': '0'} 2022-09-12T19:02:25.374 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008592bde8b39b3342-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:02:25.375 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:25.376 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:25.376 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.377 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.377 DEBUG:boto:Method: GET 2022-09-12T19:02:25.378 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.378 DEBUG:boto:Data: 2022-09-12T19:02:25.379 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.379 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:25.380 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.381 DEBUG:boto:Params: {} 2022-09-12T19:02:25.381 DEBUG:boto:Token: None 2022-09-12T19:02:25.382 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.383 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2pQFh6gi6VPorkMvJG4H9ClpIxY= 2022-09-12T19:02:25.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2pQFh6gi6VPorkMvJG4H9ClpIxY=', 'Content-Length': '0'} 2022-09-12T19:02:25.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000760fdb693dde2a57-00631f8241-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.387 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:25.387 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:25.388 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.388 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.389 DEBUG:boto:Method: GET 2022-09-12T19:02:25.389 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.390 DEBUG:boto:Data: 2022-09-12T19:02:25.391 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.392 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.392 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.393 DEBUG:boto:Params: {} 2022-09-12T19:02:25.394 DEBUG:boto:Token: None 2022-09-12T19:02:25.395 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.396 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2pQFh6gi6VPorkMvJG4H9ClpIxY= 2022-09-12T19:02:25.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2pQFh6gi6VPorkMvJG4H9ClpIxY=', 'Content-Length': '0'} 2022-09-12T19:02:25.399 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb1c1e06d9820652-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.400 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:25.401 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:25.401 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:25.402 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:25.403 DEBUG:boto:Method: GET 2022-09-12T19:02:25.404 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:25.404 DEBUG:boto:Data: 2022-09-12T19:02:25.405 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.406 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:25.407 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.407 DEBUG:boto:Params: {} 2022-09-12T19:02:25.408 DEBUG:boto:Token: None 2022-09-12T19:02:25.409 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/ 2022-09-12T19:02:25.410 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y= 2022-09-12T19:02:25.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y=', 'Content-Length': '0'} 2022-09-12T19:02:25.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4b3b223b01db78b-00631f8241-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.454 DEBUG:boto:b'jdpiak-17obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:25.455 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:25.456 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:25.456 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:25.457 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:25.457 DEBUG:boto:Method: GET 2022-09-12T19:02:25.458 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:25.458 DEBUG:boto:Data: 2022-09-12T19:02:25.459 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.459 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.460 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.460 DEBUG:boto:Params: {} 2022-09-12T19:02:25.461 DEBUG:boto:Token: None 2022-09-12T19:02:25.462 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/ 2022-09-12T19:02:25.463 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y= 2022-09-12T19:02:25.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y=', 'Content-Length': '0'} 2022-09-12T19:02:25.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec135849a3c8e608-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.467 DEBUG:boto:b'jdpiak-17obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:25.468 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:25.469 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:25.469 DEBUG:boto:Method: HEAD 2022-09-12T19:02:25.470 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3 2022-09-12T19:02:25.470 DEBUG:boto:Data: 2022-09-12T19:02:25.471 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.471 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:25.472 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.472 DEBUG:boto:Params: {} 2022-09-12T19:02:25.473 DEBUG:boto:Token: None 2022-09-12T19:02:25.474 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone3 2022-09-12T19:02:25.474 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BMY4RG9/ng46RLWeLFQIpbMwb/Q= 2022-09-12T19:02:25.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BMY4RG9/ng46RLWeLFQIpbMwb/Q=', 'Content-Length': '0'} 2022-09-12T19:02:25.477 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7d76f0342185fcf-00631f8241-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.478 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:25.478 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:25.479 DEBUG:boto:Method: HEAD 2022-09-12T19:02:25.479 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3 2022-09-12T19:02:25.480 DEBUG:boto:Data: 2022-09-12T19:02:25.480 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.481 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.481 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.482 DEBUG:boto:Params: {} 2022-09-12T19:02:25.482 DEBUG:boto:Token: None 2022-09-12T19:02:25.483 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone3 2022-09-12T19:02:25.484 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BMY4RG9/ng46RLWeLFQIpbMwb/Q= 2022-09-12T19:02:25.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BMY4RG9/ng46RLWeLFQIpbMwb/Q=', 'Content-Length': '0'} 2022-09-12T19:02:25.487 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad9fad84e5f5f6ba-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:02:25.488 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:25.488 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:25.489 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.489 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.490 DEBUG:boto:Method: GET 2022-09-12T19:02:25.491 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.491 DEBUG:boto:Data: 2022-09-12T19:02:25.492 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.492 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:25.493 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.493 DEBUG:boto:Params: {} 2022-09-12T19:02:25.494 DEBUG:boto:Token: None 2022-09-12T19:02:25.494 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.495 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2pQFh6gi6VPorkMvJG4H9ClpIxY= 2022-09-12T19:02:25.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2pQFh6gi6VPorkMvJG4H9ClpIxY=', 'Content-Length': '0'} 2022-09-12T19:02:25.498 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c5c5133703d3220-00631f8241-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.498 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:25.499 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:25.499 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.500 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.500 DEBUG:boto:Method: GET 2022-09-12T19:02:25.501 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.501 DEBUG:boto:Data: 2022-09-12T19:02:25.502 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.503 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.503 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.504 DEBUG:boto:Params: {} 2022-09-12T19:02:25.504 DEBUG:boto:Token: None 2022-09-12T19:02:25.505 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:25.505 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2pQFh6gi6VPorkMvJG4H9ClpIxY= 2022-09-12T19:02:25.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2pQFh6gi6VPorkMvJG4H9ClpIxY=', 'Content-Length': '0'} 2022-09-12T19:02:25.508 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d169dd1734c014f0-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.550 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:25.551 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:25.552 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:25.553 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:25.554 DEBUG:boto:Method: GET 2022-09-12T19:02:25.555 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:25.555 DEBUG:boto:Data: 2022-09-12T19:02:25.556 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.557 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:25.558 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.558 DEBUG:boto:Params: {} 2022-09-12T19:02:25.559 DEBUG:boto:Token: None 2022-09-12T19:02:25.559 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/ 2022-09-12T19:02:25.560 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y= 2022-09-12T19:02:25.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y=', 'Content-Length': '0'} 2022-09-12T19:02:25.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cecf1f5d44ef4e26-00631f8241-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.564 DEBUG:boto:b'jdpiak-17obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:25.565 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:25.566 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:25.567 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:25.567 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:25.568 DEBUG:boto:Method: GET 2022-09-12T19:02:25.568 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:25.569 DEBUG:boto:Data: 2022-09-12T19:02:25.569 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.570 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.570 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.571 DEBUG:boto:Params: {} 2022-09-12T19:02:25.572 DEBUG:boto:Token: None 2022-09-12T19:02:25.572 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/ 2022-09-12T19:02:25.573 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y= 2022-09-12T19:02:25.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y=', 'Content-Length': '0'} 2022-09-12T19:02:25.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039794a2c2e04f49d-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.577 DEBUG:boto:b'jdpiak-17obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:25.578 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-17 zones={test-zone2, test-zone1} 2022-09-12T19:02:25.578 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-17 zones={test-zone3, test-zone1} 2022-09-12T19:02:25.579 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:25.579 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:25.580 DEBUG:boto:Method: HEAD 2022-09-12T19:02:25.580 DEBUG:boto:Path: /jdpiak-17/ 2022-09-12T19:02:25.581 DEBUG:boto:Data: 2022-09-12T19:02:25.581 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.582 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:25.582 DEBUG:boto:Port: 8001 2022-09-12T19:02:25.583 DEBUG:boto:Params: {} 2022-09-12T19:02:25.584 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T19:02:25.584 DEBUG:boto:Token: None 2022-09-12T19:02:25.585 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/ 2022-09-12T19:02:25.585 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bNRDJEyIPercjWCwoWjITYi8GFQ= 2022-09-12T19:02:25.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bNRDJEyIPercjWCwoWjITYi8GFQ=', 'Content-Length': '0'} 2022-09-12T19:02:25.590 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6292f503b20a6cf-00631f8241-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.590 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:25.591 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:25.591 DEBUG:boto:Method: HEAD 2022-09-12T19:02:25.592 DEBUG:boto:Path: /jdpiak-17/ 2022-09-12T19:02:25.592 DEBUG:boto:Data: 2022-09-12T19:02:25.593 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.594 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.594 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.595 DEBUG:boto:Params: {} 2022-09-12T19:02:25.596 DEBUG:boto:Token: None 2022-09-12T19:02:25.596 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/ 2022-09-12T19:02:25.597 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bNRDJEyIPercjWCwoWjITYi8GFQ= 2022-09-12T19:02:25.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bNRDJEyIPercjWCwoWjITYi8GFQ=', 'Content-Length': '0'} 2022-09-12T19:02:25.600 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009fe1d93fc976fc0a-00631f8241-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.601 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:02:25.601 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:25.602 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:25.602 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:25.603 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:25.603 DEBUG:boto:Method: GET 2022-09-12T19:02:25.604 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:25.604 DEBUG:boto:Data: 2022-09-12T19:02:25.605 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.605 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:25.606 DEBUG:boto:Port: 8001 2022-09-12T19:02:25.606 DEBUG:boto:Params: {} 2022-09-12T19:02:25.607 DEBUG:boto:Token: None 2022-09-12T19:02:25.608 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/?versions 2022-09-12T19:02:25.608 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sPT8o6c2psCU2O/YM+0MqVRSH88= 2022-09-12T19:02:25.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sPT8o6c2psCU2O/YM+0MqVRSH88=', 'Content-Length': '0'} 2022-09-12T19:02:25.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081420060a347cd85-00631f8241-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.612 DEBUG:boto:b'jdpiak-17999falsedummybSzfRw-Zsh7lYv-ClgdhoG098KzYg.Strue2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpktrue2022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone20f1rsOoGVVMRuyy11YfWbutRvo943lCtrue2022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3V6Ue4vhVY0WQ9FArGixLjY-hmND5K20true2022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:25.613 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:02:25.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:02:25.615 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:02:25.615 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:02:25.616 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:02:25.616 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:25.617 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:25.617 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:25.618 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:25.618 DEBUG:boto:Method: GET 2022-09-12T19:02:25.619 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:25.619 DEBUG:boto:Data: 2022-09-12T19:02:25.620 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.620 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.621 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.621 DEBUG:boto:Params: {} 2022-09-12T19:02:25.622 DEBUG:boto:Token: None 2022-09-12T19:02:25.623 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/?versions 2022-09-12T19:02:25.623 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sPT8o6c2psCU2O/YM+0MqVRSH88= 2022-09-12T19:02:25.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sPT8o6c2psCU2O/YM+0MqVRSH88=', 'Content-Length': '0'} 2022-09-12T19:02:25.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a3a602a7416eeee-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.627 DEBUG:boto:b'jdpiak-17999falsedummybSzfRw-Zsh7lYv-ClgdhoG098KzYg.Strue2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpktrue2022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone20f1rsOoGVVMRuyy11YfWbutRvo943lCtrue2022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3V6Ue4vhVY0WQ9FArGixLjY-hmND5K20true2022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:25.628 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:02:25.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:02:25.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:02:25.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:02:25.630 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:25.631 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:25.631 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:25.632 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:25.632 DEBUG:boto:Method: GET 2022-09-12T19:02:25.633 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:25.633 DEBUG:boto:Data: 2022-09-12T19:02:25.634 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.634 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:25.635 DEBUG:boto:Port: 8001 2022-09-12T19:02:25.635 DEBUG:boto:Params: {} 2022-09-12T19:02:25.636 DEBUG:boto:Token: None 2022-09-12T19:02:25.637 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/?versions 2022-09-12T19:02:25.637 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sPT8o6c2psCU2O/YM+0MqVRSH88= 2022-09-12T19:02:25.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sPT8o6c2psCU2O/YM+0MqVRSH88=', 'Content-Length': '0'} 2022-09-12T19:02:25.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac3a755832f4b149-00631f8241-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.641 DEBUG:boto:b'jdpiak-17999falsedummybSzfRw-Zsh7lYv-ClgdhoG098KzYg.Strue2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpktrue2022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone20f1rsOoGVVMRuyy11YfWbutRvo943lCtrue2022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3V6Ue4vhVY0WQ9FArGixLjY-hmND5K20true2022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:25.642 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:25.643 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:25.643 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:25.644 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:25.645 DEBUG:boto:Method: GET 2022-09-12T19:02:25.645 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:25.646 DEBUG:boto:Data: 2022-09-12T19:02:25.646 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.647 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.647 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.648 DEBUG:boto:Params: {} 2022-09-12T19:02:25.648 DEBUG:boto:Token: None 2022-09-12T19:02:25.649 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/?versions 2022-09-12T19:02:25.650 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sPT8o6c2psCU2O/YM+0MqVRSH88= 2022-09-12T19:02:25.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sPT8o6c2psCU2O/YM+0MqVRSH88=', 'Content-Length': '0'} 2022-09-12T19:02:25.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007082a64ae0de2e54-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.654 DEBUG:boto:b'jdpiak-17999falsedummybSzfRw-Zsh7lYv-ClgdhoG098KzYg.Strue2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpktrue2022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone20f1rsOoGVVMRuyy11YfWbutRvo943lCtrue2022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3V6Ue4vhVY0WQ9FArGixLjY-hmND5K20true2022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:25.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:02:25.655 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:25.656 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:25.656 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.657 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.657 DEBUG:boto:Method: GET 2022-09-12T19:02:25.658 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.659 DEBUG:boto:Data: 2022-09-12T19:02:25.659 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.660 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:25.660 DEBUG:boto:Port: 8001 2022-09-12T19:02:25.661 DEBUG:boto:Params: {} 2022-09-12T19:02:25.661 DEBUG:boto:Token: None 2022-09-12T19:02:25.662 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.663 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gAXlrd2Fd5jDtcoA+cpdeuBO8pY= 2022-09-12T19:02:25.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gAXlrd2Fd5jDtcoA+cpdeuBO8pY=', 'Content-Length': '0'} 2022-09-12T19:02:25.665 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004215b5e1b65a4210-00631f8241-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.666 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:25.667 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:25.667 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.668 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.668 DEBUG:boto:Method: GET 2022-09-12T19:02:25.669 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.669 DEBUG:boto:Data: 2022-09-12T19:02:25.670 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.670 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.671 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.672 DEBUG:boto:Params: {} 2022-09-12T19:02:25.672 DEBUG:boto:Token: None 2022-09-12T19:02:25.673 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.673 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gAXlrd2Fd5jDtcoA+cpdeuBO8pY= 2022-09-12T19:02:25.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gAXlrd2Fd5jDtcoA+cpdeuBO8pY=', 'Content-Length': '0'} 2022-09-12T19:02:25.676 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bac13d6cde4d5542-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.677 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:25.677 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:25.678 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.678 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.679 DEBUG:boto:Method: HEAD 2022-09-12T19:02:25.680 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.680 DEBUG:boto:Data: 2022-09-12T19:02:25.681 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.681 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:25.682 DEBUG:boto:Port: 8001 2022-09-12T19:02:25.682 DEBUG:boto:Params: {} 2022-09-12T19:02:25.683 DEBUG:boto:Token: None 2022-09-12T19:02:25.683 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.684 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pcM73uqpoc6ykp+eRTTl1DxECqU= 2022-09-12T19:02:25.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pcM73uqpoc6ykp+eRTTl1DxECqU=', 'Content-Length': '0'} 2022-09-12T19:02:25.687 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bd01f21ba1771f28-00631f8241-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.687 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:25.688 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:25.688 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.689 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.689 DEBUG:boto:Method: HEAD 2022-09-12T19:02:25.690 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.691 DEBUG:boto:Data: 2022-09-12T19:02:25.691 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.692 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.692 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.693 DEBUG:boto:Params: {} 2022-09-12T19:02:25.693 DEBUG:boto:Token: None 2022-09-12T19:02:25.694 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.694 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pcM73uqpoc6ykp+eRTTl1DxECqU= 2022-09-12T19:02:25.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pcM73uqpoc6ykp+eRTTl1DxECqU=', 'Content-Length': '0'} 2022-09-12T19:02:25.697 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b57e8d5138e8556a-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:02:25.698 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:25.699 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:25.700 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.700 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.701 DEBUG:boto:Method: GET 2022-09-12T19:02:25.701 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.702 DEBUG:boto:Data: 2022-09-12T19:02:25.702 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.703 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:25.704 DEBUG:boto:Port: 8001 2022-09-12T19:02:25.705 DEBUG:boto:Params: {} 2022-09-12T19:02:25.706 DEBUG:boto:Token: None 2022-09-12T19:02:25.707 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.708 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gAXlrd2Fd5jDtcoA+cpdeuBO8pY= 2022-09-12T19:02:25.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gAXlrd2Fd5jDtcoA+cpdeuBO8pY=', 'Content-Length': '0'} 2022-09-12T19:02:25.711 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e992b118950a7c4d-00631f8241-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.712 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:25.713 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:25.713 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.714 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.715 DEBUG:boto:Method: GET 2022-09-12T19:02:25.716 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.717 DEBUG:boto:Data: 2022-09-12T19:02:25.718 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.718 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.719 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.719 DEBUG:boto:Params: {} 2022-09-12T19:02:25.720 DEBUG:boto:Token: None 2022-09-12T19:02:25.721 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.721 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gAXlrd2Fd5jDtcoA+cpdeuBO8pY= 2022-09-12T19:02:25.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gAXlrd2Fd5jDtcoA+cpdeuBO8pY=', 'Content-Length': '0'} 2022-09-12T19:02:25.725 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000e209327c0b8100e-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.726 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:25.727 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:25.728 DEBUG:boto:path=/jdpiak-17/?prefix=dummy 2022-09-12T19:02:25.728 DEBUG:boto:auth_path=/jdpiak-17/?prefix=dummy 2022-09-12T19:02:25.729 DEBUG:boto:Method: GET 2022-09-12T19:02:25.729 DEBUG:boto:Path: /jdpiak-17/?prefix=dummy 2022-09-12T19:02:25.730 DEBUG:boto:Data: 2022-09-12T19:02:25.730 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.731 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:25.732 DEBUG:boto:Port: 8001 2022-09-12T19:02:25.732 DEBUG:boto:Params: {} 2022-09-12T19:02:25.733 DEBUG:boto:Token: None 2022-09-12T19:02:25.734 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/ 2022-09-12T19:02:25.735 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y= 2022-09-12T19:02:25.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y=', 'Content-Length': '0'} 2022-09-12T19:02:25.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f024ce3a402f54df-00631f8241-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.739 DEBUG:boto:b'jdpiak-17dummy1000falsedummy2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:25.740 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:25.741 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:25.741 DEBUG:boto:path=/jdpiak-17/?prefix=dummy 2022-09-12T19:02:25.742 DEBUG:boto:auth_path=/jdpiak-17/?prefix=dummy 2022-09-12T19:02:25.743 DEBUG:boto:Method: GET 2022-09-12T19:02:25.744 DEBUG:boto:Path: /jdpiak-17/?prefix=dummy 2022-09-12T19:02:25.745 DEBUG:boto:Data: 2022-09-12T19:02:25.746 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.747 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.748 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.749 DEBUG:boto:Params: {} 2022-09-12T19:02:25.750 DEBUG:boto:Token: None 2022-09-12T19:02:25.751 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/ 2022-09-12T19:02:25.752 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y= 2022-09-12T19:02:25.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y=', 'Content-Length': '0'} 2022-09-12T19:02:25.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000845cd7b24538c2c3-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.760 DEBUG:boto:b'jdpiak-17dummy1000falsedummy2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:25.761 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:25.762 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:25.763 DEBUG:boto:Method: HEAD 2022-09-12T19:02:25.763 DEBUG:boto:Path: /jdpiak-17/dummy 2022-09-12T19:02:25.764 DEBUG:boto:Data: 2022-09-12T19:02:25.764 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.765 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:25.766 DEBUG:boto:Port: 8001 2022-09-12T19:02:25.767 DEBUG:boto:Params: {} 2022-09-12T19:02:25.767 DEBUG:boto:Token: None 2022-09-12T19:02:25.768 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/dummy 2022-09-12T19:02:25.769 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aecSMXdxjNqqzTU/MATwqbLoCh0= 2022-09-12T19:02:25.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aecSMXdxjNqqzTU/MATwqbLoCh0=', 'Content-Length': '0'} 2022-09-12T19:02:25.772 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000b4146bd281176d3-00631f8241-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.773 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:25.774 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:25.774 DEBUG:boto:Method: HEAD 2022-09-12T19:02:25.775 DEBUG:boto:Path: /jdpiak-17/dummy 2022-09-12T19:02:25.775 DEBUG:boto:Data: 2022-09-12T19:02:25.776 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.776 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.777 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.777 DEBUG:boto:Params: {} 2022-09-12T19:02:25.778 DEBUG:boto:Token: None 2022-09-12T19:02:25.778 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/dummy 2022-09-12T19:02:25.779 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aecSMXdxjNqqzTU/MATwqbLoCh0= 2022-09-12T19:02:25.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aecSMXdxjNqqzTU/MATwqbLoCh0=', 'Content-Length': '0'} 2022-09-12T19:02:25.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000025ebffd5ba22de68-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:02:25.783 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:25.784 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:25.784 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.785 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.785 DEBUG:boto:Method: GET 2022-09-12T19:02:25.786 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.786 DEBUG:boto:Data: 2022-09-12T19:02:25.787 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.788 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:25.788 DEBUG:boto:Port: 8001 2022-09-12T19:02:25.789 DEBUG:boto:Params: {} 2022-09-12T19:02:25.789 DEBUG:boto:Token: None 2022-09-12T19:02:25.790 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.791 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gAXlrd2Fd5jDtcoA+cpdeuBO8pY= 2022-09-12T19:02:25.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gAXlrd2Fd5jDtcoA+cpdeuBO8pY=', 'Content-Length': '0'} 2022-09-12T19:02:25.793 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000af7add5d98cd96b6-00631f8241-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.794 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:25.795 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:25.795 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.796 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.796 DEBUG:boto:Method: GET 2022-09-12T19:02:25.797 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.797 DEBUG:boto:Data: 2022-09-12T19:02:25.798 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.798 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.799 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.799 DEBUG:boto:Params: {} 2022-09-12T19:02:25.800 DEBUG:boto:Token: None 2022-09-12T19:02:25.801 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:25.801 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gAXlrd2Fd5jDtcoA+cpdeuBO8pY= 2022-09-12T19:02:25.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gAXlrd2Fd5jDtcoA+cpdeuBO8pY=', 'Content-Length': '0'} 2022-09-12T19:02:25.804 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c53370dc53a28e5b-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.805 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:25.805 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:25.806 DEBUG:boto:path=/jdpiak-17/?prefix=dummy 2022-09-12T19:02:25.806 DEBUG:boto:auth_path=/jdpiak-17/?prefix=dummy 2022-09-12T19:02:25.807 DEBUG:boto:Method: GET 2022-09-12T19:02:25.807 DEBUG:boto:Path: /jdpiak-17/?prefix=dummy 2022-09-12T19:02:25.808 DEBUG:boto:Data: 2022-09-12T19:02:25.808 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.809 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:25.809 DEBUG:boto:Port: 8001 2022-09-12T19:02:25.810 DEBUG:boto:Params: {} 2022-09-12T19:02:25.810 DEBUG:boto:Token: None 2022-09-12T19:02:25.811 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/ 2022-09-12T19:02:25.812 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y= 2022-09-12T19:02:25.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y=', 'Content-Length': '0'} 2022-09-12T19:02:25.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c0c59766d59de2c-00631f8241-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.862 DEBUG:boto:b'jdpiak-17dummy1000falsedummy2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:25.863 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:25.864 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:25.865 DEBUG:boto:path=/jdpiak-17/?prefix=dummy 2022-09-12T19:02:25.865 DEBUG:boto:auth_path=/jdpiak-17/?prefix=dummy 2022-09-12T19:02:25.866 DEBUG:boto:Method: GET 2022-09-12T19:02:25.867 DEBUG:boto:Path: /jdpiak-17/?prefix=dummy 2022-09-12T19:02:25.868 DEBUG:boto:Data: 2022-09-12T19:02:25.868 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.869 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.870 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.871 DEBUG:boto:Params: {} 2022-09-12T19:02:25.872 DEBUG:boto:Token: None 2022-09-12T19:02:25.873 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/ 2022-09-12T19:02:25.874 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y= 2022-09-12T19:02:25.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y=', 'Content-Length': '0'} 2022-09-12T19:02:25.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7402d684fcd6975-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.879 DEBUG:boto:b'jdpiak-17dummy1000falsedummy2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:25.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:02:25.881 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:25.882 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:25.883 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:25.884 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:25.885 DEBUG:boto:Method: GET 2022-09-12T19:02:25.886 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:25.887 DEBUG:boto:Data: 2022-09-12T19:02:25.888 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.889 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:25.889 DEBUG:boto:Port: 8001 2022-09-12T19:02:25.890 DEBUG:boto:Params: {} 2022-09-12T19:02:25.891 DEBUG:boto:Token: None 2022-09-12T19:02:25.892 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:25.894 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Egfs5ViAsFvEVYNkqxtzShbxOrg= 2022-09-12T19:02:25.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Egfs5ViAsFvEVYNkqxtzShbxOrg=', 'Content-Length': '0'} 2022-09-12T19:02:25.898 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005dee025e720471c2-00631f8241-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.899 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:25.899 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:25.900 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:25.900 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:25.901 DEBUG:boto:Method: GET 2022-09-12T19:02:25.902 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:25.903 DEBUG:boto:Data: 2022-09-12T19:02:25.904 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.905 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.907 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.907 DEBUG:boto:Params: {} 2022-09-12T19:02:25.909 DEBUG:boto:Token: None 2022-09-12T19:02:25.910 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:25.911 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Egfs5ViAsFvEVYNkqxtzShbxOrg= 2022-09-12T19:02:25.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Egfs5ViAsFvEVYNkqxtzShbxOrg=', 'Content-Length': '0'} 2022-09-12T19:02:25.915 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000031ca7488fe2b94c-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.916 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:25.917 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:25.918 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:25.918 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:25.919 DEBUG:boto:Method: HEAD 2022-09-12T19:02:25.920 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:25.921 DEBUG:boto:Data: 2022-09-12T19:02:25.922 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.923 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:25.924 DEBUG:boto:Port: 8001 2022-09-12T19:02:25.924 DEBUG:boto:Params: {} 2022-09-12T19:02:25.925 DEBUG:boto:Token: None 2022-09-12T19:02:25.926 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:25.926 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GmTt917bXkB/cEpyfWPObVWi3jE= 2022-09-12T19:02:25.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GmTt917bXkB/cEpyfWPObVWi3jE=', 'Content-Length': '0'} 2022-09-12T19:02:25.929 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d6ac913c5c3cc04-00631f8241-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.930 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:25.930 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:25.931 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:25.931 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:25.932 DEBUG:boto:Method: HEAD 2022-09-12T19:02:25.932 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:25.933 DEBUG:boto:Data: 2022-09-12T19:02:25.933 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.934 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.934 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.935 DEBUG:boto:Params: {} 2022-09-12T19:02:25.935 DEBUG:boto:Token: None 2022-09-12T19:02:25.936 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:25.937 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GmTt917bXkB/cEpyfWPObVWi3jE= 2022-09-12T19:02:25.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GmTt917bXkB/cEpyfWPObVWi3jE=', 'Content-Length': '0'} 2022-09-12T19:02:25.939 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000601c04a61e7bf4cf-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:02:25.941 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:25.941 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:25.942 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:25.942 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:25.943 DEBUG:boto:Method: GET 2022-09-12T19:02:25.943 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:25.944 DEBUG:boto:Data: 2022-09-12T19:02:25.944 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.945 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:25.945 DEBUG:boto:Port: 8001 2022-09-12T19:02:25.946 DEBUG:boto:Params: {} 2022-09-12T19:02:25.947 DEBUG:boto:Token: None 2022-09-12T19:02:25.947 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:25.948 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Egfs5ViAsFvEVYNkqxtzShbxOrg= 2022-09-12T19:02:25.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Egfs5ViAsFvEVYNkqxtzShbxOrg=', 'Content-Length': '0'} 2022-09-12T19:02:25.951 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047d00859366944fb-00631f8241-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.952 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:25.952 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:25.953 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:25.953 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:25.954 DEBUG:boto:Method: GET 2022-09-12T19:02:25.955 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:25.955 DEBUG:boto:Data: 2022-09-12T19:02:25.956 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.956 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.957 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.958 DEBUG:boto:Params: {} 2022-09-12T19:02:25.958 DEBUG:boto:Token: None 2022-09-12T19:02:25.959 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:25.960 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Egfs5ViAsFvEVYNkqxtzShbxOrg= 2022-09-12T19:02:25.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Egfs5ViAsFvEVYNkqxtzShbxOrg=', 'Content-Length': '0'} 2022-09-12T19:02:25.963 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce5466abd4f315d9-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.964 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:25.965 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:25.965 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:25.966 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:25.966 DEBUG:boto:Method: GET 2022-09-12T19:02:25.967 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:25.967 DEBUG:boto:Data: 2022-09-12T19:02:25.968 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.968 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:25.969 DEBUG:boto:Port: 8001 2022-09-12T19:02:25.969 DEBUG:boto:Params: {} 2022-09-12T19:02:25.970 DEBUG:boto:Token: None 2022-09-12T19:02:25.971 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/ 2022-09-12T19:02:25.971 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y= 2022-09-12T19:02:25.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y=', 'Content-Length': '0'} 2022-09-12T19:02:25.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b823d3478f7b733c-00631f8241-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.975 DEBUG:boto:b'jdpiak-17obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:25.976 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:25.977 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:25.977 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:25.978 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:25.978 DEBUG:boto:Method: GET 2022-09-12T19:02:25.979 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:25.979 DEBUG:boto:Data: 2022-09-12T19:02:25.980 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.980 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:25.981 DEBUG:boto:Port: 8000 2022-09-12T19:02:25.981 DEBUG:boto:Params: {} 2022-09-12T19:02:25.982 DEBUG:boto:Token: None 2022-09-12T19:02:25.983 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/ 2022-09-12T19:02:25.983 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y= 2022-09-12T19:02:25.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0PxTBOTM1Yj/90rdCBKdYZJEu4Y=', 'Content-Length': '0'} 2022-09-12T19:02:25.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da58066428bab701-00631f8241-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.987 DEBUG:boto:b'jdpiak-17obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:25.988 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:25.989 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:25.989 DEBUG:boto:Method: HEAD 2022-09-12T19:02:25.990 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1 2022-09-12T19:02:25.990 DEBUG:boto:Data: 2022-09-12T19:02:25.991 DEBUG:boto:Headers: {} 2022-09-12T19:02:25.991 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:25.992 DEBUG:boto:Port: 8001 2022-09-12T19:02:25.992 DEBUG:boto:Params: {} 2022-09-12T19:02:25.993 DEBUG:boto:Token: None 2022-09-12T19:02:25.993 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:25 GMT /jdpiak-17/obj-test-zone1 2022-09-12T19:02:25.994 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xQHUb6s+d149yCoPi8p46eu1164= 2022-09-12T19:02:25.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xQHUb6s+d149yCoPi8p46eu1164=', 'Content-Length': '0'} 2022-09-12T19:02:25.997 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009389d23bac6d9a93-00631f8241-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:25.998 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:25.998 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:25.999 DEBUG:boto:Method: HEAD 2022-09-12T19:02:25.999 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1 2022-09-12T19:02:26.000 DEBUG:boto:Data: 2022-09-12T19:02:26.000 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.001 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:26.001 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.002 DEBUG:boto:Params: {} 2022-09-12T19:02:26.002 DEBUG:boto:Token: None 2022-09-12T19:02:26.003 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone1 2022-09-12T19:02:26.003 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DnQ6M2fRPUJvm6V2e38K8+mK10E= 2022-09-12T19:02:26.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DnQ6M2fRPUJvm6V2e38K8+mK10E=', 'Content-Length': '0'} 2022-09-12T19:02:26.006 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001009e8b834f3bf49-00631f8242-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:02:26.008 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:26.008 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:26.009 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:26.009 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:26.010 DEBUG:boto:Method: GET 2022-09-12T19:02:26.010 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:26.011 DEBUG:boto:Data: 2022-09-12T19:02:26.011 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.012 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.012 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.013 DEBUG:boto:Params: {} 2022-09-12T19:02:26.014 DEBUG:boto:Token: None 2022-09-12T19:02:26.014 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:26.015 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OEUqD5cODyOoP4tY4vQAIMuRoH8= 2022-09-12T19:02:26.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OEUqD5cODyOoP4tY4vQAIMuRoH8=', 'Content-Length': '0'} 2022-09-12T19:02:26.017 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a575f7ed31faa86a-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.058 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:26.059 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:26.059 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:26.060 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:26.060 DEBUG:boto:Method: GET 2022-09-12T19:02:26.061 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:26.062 DEBUG:boto:Data: 2022-09-12T19:02:26.062 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.063 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:26.063 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.064 DEBUG:boto:Params: {} 2022-09-12T19:02:26.066 DEBUG:boto:Token: None 2022-09-12T19:02:26.066 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:26.067 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OEUqD5cODyOoP4tY4vQAIMuRoH8= 2022-09-12T19:02:26.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OEUqD5cODyOoP4tY4vQAIMuRoH8=', 'Content-Length': '0'} 2022-09-12T19:02:26.070 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000883e2c64b2b299d7-00631f8242-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.071 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:26.071 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:26.072 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:26.073 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:26.073 DEBUG:boto:Method: GET 2022-09-12T19:02:26.074 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:26.074 DEBUG:boto:Data: 2022-09-12T19:02:26.075 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.075 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.076 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.077 DEBUG:boto:Params: {} 2022-09-12T19:02:26.078 DEBUG:boto:Token: None 2022-09-12T19:02:26.078 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/ 2022-09-12T19:02:26.079 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts= 2022-09-12T19:02:26.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts=', 'Content-Length': '0'} 2022-09-12T19:02:26.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f03d56d7d52f6aff-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.083 DEBUG:boto:b'jdpiak-17obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:26.084 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:26.084 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:26.085 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:26.086 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:26.086 DEBUG:boto:Method: GET 2022-09-12T19:02:26.087 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:26.087 DEBUG:boto:Data: 2022-09-12T19:02:26.088 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.088 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:26.089 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.089 DEBUG:boto:Params: {} 2022-09-12T19:02:26.090 DEBUG:boto:Token: None 2022-09-12T19:02:26.091 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/ 2022-09-12T19:02:26.091 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts= 2022-09-12T19:02:26.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts=', 'Content-Length': '0'} 2022-09-12T19:02:26.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008adbba2213c58dcd-00631f8242-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.096 DEBUG:boto:b'jdpiak-17obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:26.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:02:26.097 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:26.097 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:26.098 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.099 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.099 DEBUG:boto:Method: GET 2022-09-12T19:02:26.100 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.100 DEBUG:boto:Data: 2022-09-12T19:02:26.101 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.101 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.102 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.102 DEBUG:boto:Params: {} 2022-09-12T19:02:26.103 DEBUG:boto:Token: None 2022-09-12T19:02:26.104 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.104 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8ygDj9n7ZiJzf40hgya6ASmiCTg= 2022-09-12T19:02:26.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8ygDj9n7ZiJzf40hgya6ASmiCTg=', 'Content-Length': '0'} 2022-09-12T19:02:26.108 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b5751e2f62521e6-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.109 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:26.109 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:26.110 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.111 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.111 DEBUG:boto:Method: GET 2022-09-12T19:02:26.112 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.112 DEBUG:boto:Data: 2022-09-12T19:02:26.113 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.113 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:26.114 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.114 DEBUG:boto:Params: {} 2022-09-12T19:02:26.115 DEBUG:boto:Token: None 2022-09-12T19:02:26.116 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.116 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8ygDj9n7ZiJzf40hgya6ASmiCTg= 2022-09-12T19:02:26.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8ygDj9n7ZiJzf40hgya6ASmiCTg=', 'Content-Length': '0'} 2022-09-12T19:02:26.119 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea40576ba35909fd-00631f8242-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.120 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:26.121 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:26.122 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.122 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.123 DEBUG:boto:Method: HEAD 2022-09-12T19:02:26.123 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.124 DEBUG:boto:Data: 2022-09-12T19:02:26.124 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.125 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.125 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.126 DEBUG:boto:Params: {} 2022-09-12T19:02:26.127 DEBUG:boto:Token: None 2022-09-12T19:02:26.127 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.128 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S8IjItUFPPM2Vd58ndAoriiLzDc= 2022-09-12T19:02:26.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S8IjItUFPPM2Vd58ndAoriiLzDc=', 'Content-Length': '0'} 2022-09-12T19:02:26.131 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000040f9e1b1a6ba5b0d-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.132 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:26.132 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:26.133 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.133 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.134 DEBUG:boto:Method: HEAD 2022-09-12T19:02:26.134 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.135 DEBUG:boto:Data: 2022-09-12T19:02:26.135 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.136 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:26.136 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.137 DEBUG:boto:Params: {} 2022-09-12T19:02:26.137 DEBUG:boto:Token: None 2022-09-12T19:02:26.138 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.139 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S8IjItUFPPM2Vd58ndAoriiLzDc= 2022-09-12T19:02:26.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S8IjItUFPPM2Vd58ndAoriiLzDc=', 'Content-Length': '0'} 2022-09-12T19:02:26.141 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d89e578aa494dddd-00631f8242-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:02:26.142 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:26.143 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:26.144 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.144 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.145 DEBUG:boto:Method: GET 2022-09-12T19:02:26.145 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.146 DEBUG:boto:Data: 2022-09-12T19:02:26.146 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.147 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.147 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.148 DEBUG:boto:Params: {} 2022-09-12T19:02:26.149 DEBUG:boto:Token: None 2022-09-12T19:02:26.149 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.150 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8ygDj9n7ZiJzf40hgya6ASmiCTg= 2022-09-12T19:02:26.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8ygDj9n7ZiJzf40hgya6ASmiCTg=', 'Content-Length': '0'} 2022-09-12T19:02:26.153 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008320503af0677273-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.194 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:26.195 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:26.196 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.196 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.197 DEBUG:boto:Method: GET 2022-09-12T19:02:26.197 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.198 DEBUG:boto:Data: 2022-09-12T19:02:26.198 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.199 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:26.200 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.201 DEBUG:boto:Params: {} 2022-09-12T19:02:26.202 DEBUG:boto:Token: None 2022-09-12T19:02:26.203 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.203 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8ygDj9n7ZiJzf40hgya6ASmiCTg= 2022-09-12T19:02:26.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8ygDj9n7ZiJzf40hgya6ASmiCTg=', 'Content-Length': '0'} 2022-09-12T19:02:26.206 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000074b6b6eabc6ce0e4-00631f8242-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.207 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:26.208 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:26.208 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:26.209 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:26.209 DEBUG:boto:Method: GET 2022-09-12T19:02:26.210 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:26.211 DEBUG:boto:Data: 2022-09-12T19:02:26.211 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.212 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.212 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.213 DEBUG:boto:Params: {} 2022-09-12T19:02:26.213 DEBUG:boto:Token: None 2022-09-12T19:02:26.214 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/ 2022-09-12T19:02:26.215 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts= 2022-09-12T19:02:26.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts=', 'Content-Length': '0'} 2022-09-12T19:02:26.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000358015c9822d22eb-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.219 DEBUG:boto:b'jdpiak-17obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:26.219 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:26.220 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:26.221 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:26.221 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:26.222 DEBUG:boto:Method: GET 2022-09-12T19:02:26.222 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:26.223 DEBUG:boto:Data: 2022-09-12T19:02:26.223 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.224 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:26.224 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.225 DEBUG:boto:Params: {} 2022-09-12T19:02:26.226 DEBUG:boto:Token: None 2022-09-12T19:02:26.226 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/ 2022-09-12T19:02:26.227 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts= 2022-09-12T19:02:26.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts=', 'Content-Length': '0'} 2022-09-12T19:02:26.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec3ac9e95e549e71-00631f8242-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.278 DEBUG:boto:b'jdpiak-17obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:26.279 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:26.280 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:26.280 DEBUG:boto:Method: HEAD 2022-09-12T19:02:26.281 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2 2022-09-12T19:02:26.281 DEBUG:boto:Data: 2022-09-12T19:02:26.282 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.282 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.283 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.283 DEBUG:boto:Params: {} 2022-09-12T19:02:26.284 DEBUG:boto:Token: None 2022-09-12T19:02:26.284 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone2 2022-09-12T19:02:26.285 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JV4qLJLDsWCBm5m8gJg6E5cHEp8= 2022-09-12T19:02:26.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JV4qLJLDsWCBm5m8gJg6E5cHEp8=', 'Content-Length': '0'} 2022-09-12T19:02:26.288 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b78213f01bd05ccb-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.289 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:26.289 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:26.290 DEBUG:boto:Method: HEAD 2022-09-12T19:02:26.291 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2 2022-09-12T19:02:26.291 DEBUG:boto:Data: 2022-09-12T19:02:26.292 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.292 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:26.293 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.293 DEBUG:boto:Params: {} 2022-09-12T19:02:26.294 DEBUG:boto:Token: None 2022-09-12T19:02:26.294 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone2 2022-09-12T19:02:26.295 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JV4qLJLDsWCBm5m8gJg6E5cHEp8= 2022-09-12T19:02:26.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JV4qLJLDsWCBm5m8gJg6E5cHEp8=', 'Content-Length': '0'} 2022-09-12T19:02:26.298 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1673b1edb1b5b3e-00631f8242-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:02:26.299 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:26.300 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:26.300 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.301 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.301 DEBUG:boto:Method: GET 2022-09-12T19:02:26.302 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.302 DEBUG:boto:Data: 2022-09-12T19:02:26.303 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.303 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.304 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.305 DEBUG:boto:Params: {} 2022-09-12T19:02:26.305 DEBUG:boto:Token: None 2022-09-12T19:02:26.306 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.306 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8ygDj9n7ZiJzf40hgya6ASmiCTg= 2022-09-12T19:02:26.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8ygDj9n7ZiJzf40hgya6ASmiCTg=', 'Content-Length': '0'} 2022-09-12T19:02:26.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000edf1502943c9ed0d-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.310 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:26.311 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:26.311 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.312 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.312 DEBUG:boto:Method: GET 2022-09-12T19:02:26.313 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.313 DEBUG:boto:Data: 2022-09-12T19:02:26.314 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.314 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:26.315 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.315 DEBUG:boto:Params: {} 2022-09-12T19:02:26.316 DEBUG:boto:Token: None 2022-09-12T19:02:26.317 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:26.317 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8ygDj9n7ZiJzf40hgya6ASmiCTg= 2022-09-12T19:02:26.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8ygDj9n7ZiJzf40hgya6ASmiCTg=', 'Content-Length': '0'} 2022-09-12T19:02:26.320 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e09527273e29fec5-00631f8242-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.321 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:26.322 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:26.322 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:26.323 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:26.323 DEBUG:boto:Method: GET 2022-09-12T19:02:26.324 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:26.324 DEBUG:boto:Data: 2022-09-12T19:02:26.325 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.326 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.326 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.327 DEBUG:boto:Params: {} 2022-09-12T19:02:26.327 DEBUG:boto:Token: None 2022-09-12T19:02:26.328 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/ 2022-09-12T19:02:26.328 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts= 2022-09-12T19:02:26.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts=', 'Content-Length': '0'} 2022-09-12T19:02:26.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef55bc650e33ebd3-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.333 DEBUG:boto:b'jdpiak-17obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:26.334 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:26.334 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:26.335 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:26.335 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:26.336 DEBUG:boto:Method: GET 2022-09-12T19:02:26.336 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:26.337 DEBUG:boto:Data: 2022-09-12T19:02:26.337 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.338 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:26.338 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.339 DEBUG:boto:Params: {} 2022-09-12T19:02:26.340 DEBUG:boto:Token: None 2022-09-12T19:02:26.340 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/ 2022-09-12T19:02:26.341 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts= 2022-09-12T19:02:26.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts=', 'Content-Length': '0'} 2022-09-12T19:02:26.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054e83a03e0d22ed8-00631f8242-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.346 DEBUG:boto:b'jdpiak-17obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:26.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:02:26.347 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:26.348 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:26.348 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.349 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.349 DEBUG:boto:Method: GET 2022-09-12T19:02:26.350 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.350 DEBUG:boto:Data: 2022-09-12T19:02:26.351 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.351 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.352 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.352 DEBUG:boto:Params: {} 2022-09-12T19:02:26.353 DEBUG:boto:Token: None 2022-09-12T19:02:26.353 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.354 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8XdT+M0fJ/JnaYefNuN9IXvQohc= 2022-09-12T19:02:26.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8XdT+M0fJ/JnaYefNuN9IXvQohc=', 'Content-Length': '0'} 2022-09-12T19:02:26.357 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b9edd3ac7218209-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.358 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:26.359 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:26.359 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.360 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.360 DEBUG:boto:Method: GET 2022-09-12T19:02:26.361 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.361 DEBUG:boto:Data: 2022-09-12T19:02:26.362 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.362 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:26.363 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.363 DEBUG:boto:Params: {} 2022-09-12T19:02:26.364 DEBUG:boto:Token: None 2022-09-12T19:02:26.365 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.365 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8XdT+M0fJ/JnaYefNuN9IXvQohc= 2022-09-12T19:02:26.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8XdT+M0fJ/JnaYefNuN9IXvQohc=', 'Content-Length': '0'} 2022-09-12T19:02:26.368 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058a652e83c82523c-00631f8242-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.369 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:26.369 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:26.370 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.370 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.371 DEBUG:boto:Method: HEAD 2022-09-12T19:02:26.371 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.372 DEBUG:boto:Data: 2022-09-12T19:02:26.373 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.373 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.374 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.375 DEBUG:boto:Params: {} 2022-09-12T19:02:26.375 DEBUG:boto:Token: None 2022-09-12T19:02:26.376 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.377 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Hoo0Og5dK8A+M2xG3T6pD+TvZ+s= 2022-09-12T19:02:26.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Hoo0Og5dK8A+M2xG3T6pD+TvZ+s=', 'Content-Length': '0'} 2022-09-12T19:02:26.380 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000224b42c96ceecf92-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.380 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:26.381 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:26.381 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.382 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.383 DEBUG:boto:Method: HEAD 2022-09-12T19:02:26.384 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.385 DEBUG:boto:Data: 2022-09-12T19:02:26.385 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.386 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:26.387 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.387 DEBUG:boto:Params: {} 2022-09-12T19:02:26.388 DEBUG:boto:Token: None 2022-09-12T19:02:26.389 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.390 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Hoo0Og5dK8A+M2xG3T6pD+TvZ+s= 2022-09-12T19:02:26.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Hoo0Og5dK8A+M2xG3T6pD+TvZ+s=', 'Content-Length': '0'} 2022-09-12T19:02:26.394 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c4ef85e97ecc726-00631f8242-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:02:26.396 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:26.396 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:26.397 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.398 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.398 DEBUG:boto:Method: GET 2022-09-12T19:02:26.399 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.399 DEBUG:boto:Data: 2022-09-12T19:02:26.400 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.401 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.402 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.402 DEBUG:boto:Params: {} 2022-09-12T19:02:26.403 DEBUG:boto:Token: None 2022-09-12T19:02:26.404 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.404 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8XdT+M0fJ/JnaYefNuN9IXvQohc= 2022-09-12T19:02:26.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8XdT+M0fJ/JnaYefNuN9IXvQohc=', 'Content-Length': '0'} 2022-09-12T19:02:26.407 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ecbbd02509d0dea8-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.408 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:26.409 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:26.410 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.410 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.411 DEBUG:boto:Method: GET 2022-09-12T19:02:26.411 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.412 DEBUG:boto:Data: 2022-09-12T19:02:26.412 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.413 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:26.413 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.414 DEBUG:boto:Params: {} 2022-09-12T19:02:26.415 DEBUG:boto:Token: None 2022-09-12T19:02:26.415 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.416 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8XdT+M0fJ/JnaYefNuN9IXvQohc= 2022-09-12T19:02:26.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8XdT+M0fJ/JnaYefNuN9IXvQohc=', 'Content-Length': '0'} 2022-09-12T19:02:26.419 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004aa19204406d944d-00631f8242-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.462 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:26.463 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:26.464 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:26.464 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:26.465 DEBUG:boto:Method: GET 2022-09-12T19:02:26.465 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:26.466 DEBUG:boto:Data: 2022-09-12T19:02:26.467 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.467 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.468 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.468 DEBUG:boto:Params: {} 2022-09-12T19:02:26.469 DEBUG:boto:Token: None 2022-09-12T19:02:26.470 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/ 2022-09-12T19:02:26.470 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts= 2022-09-12T19:02:26.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts=', 'Content-Length': '0'} 2022-09-12T19:02:26.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f837761ca17ccc20-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.475 DEBUG:boto:b'jdpiak-17obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:26.477 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:26.477 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:26.478 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:26.478 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:26.479 DEBUG:boto:Method: GET 2022-09-12T19:02:26.479 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:26.480 DEBUG:boto:Data: 2022-09-12T19:02:26.480 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.481 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:26.482 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.482 DEBUG:boto:Params: {} 2022-09-12T19:02:26.483 DEBUG:boto:Token: None 2022-09-12T19:02:26.483 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/ 2022-09-12T19:02:26.484 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts= 2022-09-12T19:02:26.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts=', 'Content-Length': '0'} 2022-09-12T19:02:26.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b68a3939ff400131-00631f8242-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.488 DEBUG:boto:b'jdpiak-17obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:26.489 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:26.490 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:26.491 DEBUG:boto:Method: HEAD 2022-09-12T19:02:26.491 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3 2022-09-12T19:02:26.492 DEBUG:boto:Data: 2022-09-12T19:02:26.492 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.493 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.493 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.494 DEBUG:boto:Params: {} 2022-09-12T19:02:26.494 DEBUG:boto:Token: None 2022-09-12T19:02:26.495 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone3 2022-09-12T19:02:26.495 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0UUPe59R4CvXzXKlR465mV1Ih+o= 2022-09-12T19:02:26.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0UUPe59R4CvXzXKlR465mV1Ih+o=', 'Content-Length': '0'} 2022-09-12T19:02:26.499 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee4f511df9739340-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.500 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:26.500 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:26.501 DEBUG:boto:Method: HEAD 2022-09-12T19:02:26.501 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3 2022-09-12T19:02:26.502 DEBUG:boto:Data: 2022-09-12T19:02:26.503 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.503 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:26.504 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.504 DEBUG:boto:Params: {} 2022-09-12T19:02:26.505 DEBUG:boto:Token: None 2022-09-12T19:02:26.505 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone3 2022-09-12T19:02:26.506 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0UUPe59R4CvXzXKlR465mV1Ih+o= 2022-09-12T19:02:26.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0UUPe59R4CvXzXKlR465mV1Ih+o=', 'Content-Length': '0'} 2022-09-12T19:02:26.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007444560793c8742b-00631f8242-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:02:26.511 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:26.512 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:26.512 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.513 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.513 DEBUG:boto:Method: GET 2022-09-12T19:02:26.514 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.515 DEBUG:boto:Data: 2022-09-12T19:02:26.515 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.516 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.516 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.518 DEBUG:boto:Params: {} 2022-09-12T19:02:26.519 DEBUG:boto:Token: None 2022-09-12T19:02:26.520 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.520 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8XdT+M0fJ/JnaYefNuN9IXvQohc= 2022-09-12T19:02:26.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8XdT+M0fJ/JnaYefNuN9IXvQohc=', 'Content-Length': '0'} 2022-09-12T19:02:26.523 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000055107fb8082484d-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.570 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:26.571 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:26.571 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.572 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.573 DEBUG:boto:Method: GET 2022-09-12T19:02:26.573 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.574 DEBUG:boto:Data: 2022-09-12T19:02:26.574 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.575 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:26.575 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.576 DEBUG:boto:Params: {} 2022-09-12T19:02:26.576 DEBUG:boto:Token: None 2022-09-12T19:02:26.577 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:26.578 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8XdT+M0fJ/JnaYefNuN9IXvQohc= 2022-09-12T19:02:26.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8XdT+M0fJ/JnaYefNuN9IXvQohc=', 'Content-Length': '0'} 2022-09-12T19:02:26.581 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb022742a8ba75f9-00631f8242-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.581 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:26.582 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:26.582 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:26.583 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:26.584 DEBUG:boto:Method: GET 2022-09-12T19:02:26.584 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:26.585 DEBUG:boto:Data: 2022-09-12T19:02:26.585 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.586 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.586 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.587 DEBUG:boto:Params: {} 2022-09-12T19:02:26.587 DEBUG:boto:Token: None 2022-09-12T19:02:26.588 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/ 2022-09-12T19:02:26.589 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts= 2022-09-12T19:02:26.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts=', 'Content-Length': '0'} 2022-09-12T19:02:26.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c201014f17f92552-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.593 DEBUG:boto:b'jdpiak-17obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:26.594 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:26.595 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:26.595 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:26.596 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:26.597 DEBUG:boto:Method: GET 2022-09-12T19:02:26.597 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:26.598 DEBUG:boto:Data: 2022-09-12T19:02:26.598 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.599 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:26.599 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.600 DEBUG:boto:Params: {} 2022-09-12T19:02:26.600 DEBUG:boto:Token: None 2022-09-12T19:02:26.601 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/ 2022-09-12T19:02:26.602 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts= 2022-09-12T19:02:26.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts=', 'Content-Length': '0'} 2022-09-12T19:02:26.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4711a4b6928b08a-00631f8242-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.606 DEBUG:boto:b'jdpiak-17obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:26.606 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-17 zones={test-zone3, test-zone1} 2022-09-12T19:02:26.607 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-17 zones={test-zone3, test-zone2} 2022-09-12T19:02:26.607 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:26.608 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:26.608 DEBUG:boto:Method: HEAD 2022-09-12T19:02:26.609 DEBUG:boto:Path: /jdpiak-17/ 2022-09-12T19:02:26.610 DEBUG:boto:Data: 2022-09-12T19:02:26.610 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.611 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.611 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.612 DEBUG:boto:Params: {} 2022-09-12T19:02:26.612 DEBUG:boto:Token: None 2022-09-12T19:02:26.613 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/ 2022-09-12T19:02:26.613 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:noQ0wJkT3yeIuUqd9CKfeoZ7vQY= 2022-09-12T19:02:26.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:noQ0wJkT3yeIuUqd9CKfeoZ7vQY=', 'Content-Length': '0'} 2022-09-12T19:02:26.617 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2e1846dfbf7460f-00631f8242-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.617 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:26.618 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:26.619 DEBUG:boto:Method: HEAD 2022-09-12T19:02:26.619 DEBUG:boto:Path: /jdpiak-17/ 2022-09-12T19:02:26.620 DEBUG:boto:Data: 2022-09-12T19:02:26.620 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.621 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:26.621 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.622 DEBUG:boto:Params: {} 2022-09-12T19:02:26.622 DEBUG:boto:Token: None 2022-09-12T19:02:26.623 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/ 2022-09-12T19:02:26.624 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:noQ0wJkT3yeIuUqd9CKfeoZ7vQY= 2022-09-12T19:02:26.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:noQ0wJkT3yeIuUqd9CKfeoZ7vQY=', 'Content-Length': '0'} 2022-09-12T19:02:26.627 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072b6a3711e6fcf92-00631f8242-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.627 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:02:26.628 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:26.629 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:26.629 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:26.630 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:26.630 DEBUG:boto:Method: GET 2022-09-12T19:02:26.631 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:26.631 DEBUG:boto:Data: 2022-09-12T19:02:26.632 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.632 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.633 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.633 DEBUG:boto:Params: {} 2022-09-12T19:02:26.634 DEBUG:boto:Token: None 2022-09-12T19:02:26.635 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/?versions 2022-09-12T19:02:26.635 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gfRgP+WsL3JQqFjjYfLbuZ1bBAQ= 2022-09-12T19:02:26.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gfRgP+WsL3JQqFjjYfLbuZ1bBAQ=', 'Content-Length': '0'} 2022-09-12T19:02:26.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084122b34f95eef55-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.639 DEBUG:boto:b'jdpiak-17999falsedummybSzfRw-Zsh7lYv-ClgdhoG098KzYg.Strue2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpktrue2022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone20f1rsOoGVVMRuyy11YfWbutRvo943lCtrue2022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3V6Ue4vhVY0WQ9FArGixLjY-hmND5K20true2022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:26.640 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:02:26.641 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:02:26.641 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:02:26.642 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:02:26.642 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:02:26.643 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:26.644 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:26.644 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:26.645 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:26.645 DEBUG:boto:Method: GET 2022-09-12T19:02:26.646 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:26.646 DEBUG:boto:Data: 2022-09-12T19:02:26.647 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.647 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:26.648 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.648 DEBUG:boto:Params: {} 2022-09-12T19:02:26.649 DEBUG:boto:Token: None 2022-09-12T19:02:26.650 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/?versions 2022-09-12T19:02:26.650 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gfRgP+WsL3JQqFjjYfLbuZ1bBAQ= 2022-09-12T19:02:26.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gfRgP+WsL3JQqFjjYfLbuZ1bBAQ=', 'Content-Length': '0'} 2022-09-12T19:02:26.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058febaf7935013ec-00631f8242-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.655 DEBUG:boto:b'jdpiak-17999falsedummybSzfRw-Zsh7lYv-ClgdhoG098KzYg.Strue2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpktrue2022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone20f1rsOoGVVMRuyy11YfWbutRvo943lCtrue2022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3V6Ue4vhVY0WQ9FArGixLjY-hmND5K20true2022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:26.655 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:02:26.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:02:26.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:02:26.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:02:26.658 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:26.658 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:26.659 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:26.659 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:26.660 DEBUG:boto:Method: GET 2022-09-12T19:02:26.660 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:26.661 DEBUG:boto:Data: 2022-09-12T19:02:26.661 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.662 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.662 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.663 DEBUG:boto:Params: {} 2022-09-12T19:02:26.663 DEBUG:boto:Token: None 2022-09-12T19:02:26.664 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/?versions 2022-09-12T19:02:26.665 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gfRgP+WsL3JQqFjjYfLbuZ1bBAQ= 2022-09-12T19:02:26.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gfRgP+WsL3JQqFjjYfLbuZ1bBAQ=', 'Content-Length': '0'} 2022-09-12T19:02:26.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f1236042cea0773-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.669 DEBUG:boto:b'jdpiak-17999falsedummybSzfRw-Zsh7lYv-ClgdhoG098KzYg.Strue2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpktrue2022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone20f1rsOoGVVMRuyy11YfWbutRvo943lCtrue2022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3V6Ue4vhVY0WQ9FArGixLjY-hmND5K20true2022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:26.670 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:26.671 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:26.671 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:26.672 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:26.672 DEBUG:boto:Method: GET 2022-09-12T19:02:26.673 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999 2022-09-12T19:02:26.673 DEBUG:boto:Data: 2022-09-12T19:02:26.674 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.674 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:26.675 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.676 DEBUG:boto:Params: {} 2022-09-12T19:02:26.676 DEBUG:boto:Token: None 2022-09-12T19:02:26.677 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/?versions 2022-09-12T19:02:26.677 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gfRgP+WsL3JQqFjjYfLbuZ1bBAQ= 2022-09-12T19:02:26.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gfRgP+WsL3JQqFjjYfLbuZ1bBAQ=', 'Content-Length': '0'} 2022-09-12T19:02:26.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6b7f530d3a8704f-00631f8242-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.681 DEBUG:boto:b'jdpiak-17999falsedummybSzfRw-Zsh7lYv-ClgdhoG098KzYg.Strue2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpktrue2022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone20f1rsOoGVVMRuyy11YfWbutRvo943lCtrue2022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3V6Ue4vhVY0WQ9FArGixLjY-hmND5K20true2022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:26.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:02:26.683 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:26.684 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:26.684 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.685 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.685 DEBUG:boto:Method: GET 2022-09-12T19:02:26.686 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.686 DEBUG:boto:Data: 2022-09-12T19:02:26.687 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.687 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.688 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.689 DEBUG:boto:Params: {} 2022-09-12T19:02:26.689 DEBUG:boto:Token: None 2022-09-12T19:02:26.690 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.691 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l6NhE9pb/P+sSBZ3ujJ3yWox7no= 2022-09-12T19:02:26.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l6NhE9pb/P+sSBZ3ujJ3yWox7no=', 'Content-Length': '0'} 2022-09-12T19:02:26.694 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c341b9c31311d423-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.694 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:26.695 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:26.695 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.696 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.697 DEBUG:boto:Method: GET 2022-09-12T19:02:26.697 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.698 DEBUG:boto:Data: 2022-09-12T19:02:26.698 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.699 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:26.699 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.700 DEBUG:boto:Params: {} 2022-09-12T19:02:26.701 DEBUG:boto:Token: None 2022-09-12T19:02:26.701 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.702 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l6NhE9pb/P+sSBZ3ujJ3yWox7no= 2022-09-12T19:02:26.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l6NhE9pb/P+sSBZ3ujJ3yWox7no=', 'Content-Length': '0'} 2022-09-12T19:02:26.704 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000615dd9742b68e3a5-00631f8242-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.705 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:26.706 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:26.706 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.707 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.707 DEBUG:boto:Method: HEAD 2022-09-12T19:02:26.708 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.708 DEBUG:boto:Data: 2022-09-12T19:02:26.709 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.709 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.710 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.711 DEBUG:boto:Params: {} 2022-09-12T19:02:26.711 DEBUG:boto:Token: None 2022-09-12T19:02:26.712 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.712 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l8XSWoZWU4oHP+4OpWCy+eFwswI= 2022-09-12T19:02:26.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l8XSWoZWU4oHP+4OpWCy+eFwswI=', 'Content-Length': '0'} 2022-09-12T19:02:26.715 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000089e4df7e7a88af0d-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.716 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:26.716 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:26.717 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.717 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.718 DEBUG:boto:Method: HEAD 2022-09-12T19:02:26.718 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.719 DEBUG:boto:Data: 2022-09-12T19:02:26.719 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.720 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:26.720 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.721 DEBUG:boto:Params: {} 2022-09-12T19:02:26.722 DEBUG:boto:Token: None 2022-09-12T19:02:26.722 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.723 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l8XSWoZWU4oHP+4OpWCy+eFwswI= 2022-09-12T19:02:26.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l8XSWoZWU4oHP+4OpWCy+eFwswI=', 'Content-Length': '0'} 2022-09-12T19:02:26.725 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000032aeabba085e890c-00631f8242-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:02:26.727 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:26.727 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:26.728 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.728 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.729 DEBUG:boto:Method: GET 2022-09-12T19:02:26.729 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.730 DEBUG:boto:Data: 2022-09-12T19:02:26.730 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.731 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.732 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.732 DEBUG:boto:Params: {} 2022-09-12T19:02:26.733 DEBUG:boto:Token: None 2022-09-12T19:02:26.733 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.734 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l6NhE9pb/P+sSBZ3ujJ3yWox7no= 2022-09-12T19:02:26.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l6NhE9pb/P+sSBZ3ujJ3yWox7no=', 'Content-Length': '0'} 2022-09-12T19:02:26.737 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000fc04f2e761c00bf-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.737 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:26.738 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:26.739 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.739 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.740 DEBUG:boto:Method: GET 2022-09-12T19:02:26.740 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.741 DEBUG:boto:Data: 2022-09-12T19:02:26.741 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.742 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:26.742 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.743 DEBUG:boto:Params: {} 2022-09-12T19:02:26.744 DEBUG:boto:Token: None 2022-09-12T19:02:26.744 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.745 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l6NhE9pb/P+sSBZ3ujJ3yWox7no= 2022-09-12T19:02:26.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l6NhE9pb/P+sSBZ3ujJ3yWox7no=', 'Content-Length': '0'} 2022-09-12T19:02:26.748 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000738a46c2a82fba7a-00631f8242-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.748 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:26.749 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:26.750 DEBUG:boto:path=/jdpiak-17/?prefix=dummy 2022-09-12T19:02:26.750 DEBUG:boto:auth_path=/jdpiak-17/?prefix=dummy 2022-09-12T19:02:26.751 DEBUG:boto:Method: GET 2022-09-12T19:02:26.751 DEBUG:boto:Path: /jdpiak-17/?prefix=dummy 2022-09-12T19:02:26.752 DEBUG:boto:Data: 2022-09-12T19:02:26.752 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.753 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.753 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.754 DEBUG:boto:Params: {} 2022-09-12T19:02:26.755 DEBUG:boto:Token: None 2022-09-12T19:02:26.755 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/ 2022-09-12T19:02:26.756 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts= 2022-09-12T19:02:26.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts=', 'Content-Length': '0'} 2022-09-12T19:02:26.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5af9677dec4b4d5-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.760 DEBUG:boto:b'jdpiak-17dummy1000falsedummy2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:26.761 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:26.761 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:26.762 DEBUG:boto:path=/jdpiak-17/?prefix=dummy 2022-09-12T19:02:26.762 DEBUG:boto:auth_path=/jdpiak-17/?prefix=dummy 2022-09-12T19:02:26.763 DEBUG:boto:Method: GET 2022-09-12T19:02:26.764 DEBUG:boto:Path: /jdpiak-17/?prefix=dummy 2022-09-12T19:02:26.764 DEBUG:boto:Data: 2022-09-12T19:02:26.765 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.765 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:26.766 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.766 DEBUG:boto:Params: {} 2022-09-12T19:02:26.767 DEBUG:boto:Token: None 2022-09-12T19:02:26.768 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/ 2022-09-12T19:02:26.768 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts= 2022-09-12T19:02:26.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts=', 'Content-Length': '0'} 2022-09-12T19:02:26.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7a0482739442d86-00631f8242-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.818 DEBUG:boto:b'jdpiak-17dummy1000falsedummy2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:26.819 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:26.820 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:26.820 DEBUG:boto:Method: HEAD 2022-09-12T19:02:26.821 DEBUG:boto:Path: /jdpiak-17/dummy 2022-09-12T19:02:26.821 DEBUG:boto:Data: 2022-09-12T19:02:26.822 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.822 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.823 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.823 DEBUG:boto:Params: {} 2022-09-12T19:02:26.824 DEBUG:boto:Token: None 2022-09-12T19:02:26.825 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/dummy 2022-09-12T19:02:26.825 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TUKLil+KDDRcgC0/TxHkibU6YXI= 2022-09-12T19:02:26.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TUKLil+KDDRcgC0/TxHkibU6YXI=', 'Content-Length': '0'} 2022-09-12T19:02:26.829 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ff1f4fb6c4db87b9-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.829 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:26.830 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:26.831 DEBUG:boto:Method: HEAD 2022-09-12T19:02:26.831 DEBUG:boto:Path: /jdpiak-17/dummy 2022-09-12T19:02:26.832 DEBUG:boto:Data: 2022-09-12T19:02:26.832 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.833 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:26.833 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.834 DEBUG:boto:Params: {} 2022-09-12T19:02:26.834 DEBUG:boto:Token: None 2022-09-12T19:02:26.835 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/dummy 2022-09-12T19:02:26.836 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TUKLil+KDDRcgC0/TxHkibU6YXI= 2022-09-12T19:02:26.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TUKLil+KDDRcgC0/TxHkibU6YXI=', 'Content-Length': '0'} 2022-09-12T19:02:26.839 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f17b30c2d793ab33-00631f8242-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:02:26.840 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:26.841 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:26.841 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.842 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.842 DEBUG:boto:Method: GET 2022-09-12T19:02:26.843 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.844 DEBUG:boto:Data: 2022-09-12T19:02:26.844 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.845 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.845 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.846 DEBUG:boto:Params: {} 2022-09-12T19:02:26.846 DEBUG:boto:Token: None 2022-09-12T19:02:26.847 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.848 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l6NhE9pb/P+sSBZ3ujJ3yWox7no= 2022-09-12T19:02:26.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l6NhE9pb/P+sSBZ3ujJ3yWox7no=', 'Content-Length': '0'} 2022-09-12T19:02:26.851 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fd2b91bdfa3030be-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.851 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:02:26.852 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:02:26.853 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.853 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.854 DEBUG:boto:Method: GET 2022-09-12T19:02:26.855 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.855 DEBUG:boto:Data: 2022-09-12T19:02:26.856 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.856 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:26.857 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.857 DEBUG:boto:Params: {} 2022-09-12T19:02:26.858 DEBUG:boto:Token: None 2022-09-12T19:02:26.858 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:02:26.859 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l6NhE9pb/P+sSBZ3ujJ3yWox7no= 2022-09-12T19:02:26.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l6NhE9pb/P+sSBZ3ujJ3yWox7no=', 'Content-Length': '0'} 2022-09-12T19:02:26.862 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ab22406b7ee05e4d-00631f8242-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.863 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:26.863 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:26.864 DEBUG:boto:path=/jdpiak-17/?prefix=dummy 2022-09-12T19:02:26.865 DEBUG:boto:auth_path=/jdpiak-17/?prefix=dummy 2022-09-12T19:02:26.865 DEBUG:boto:Method: GET 2022-09-12T19:02:26.866 DEBUG:boto:Path: /jdpiak-17/?prefix=dummy 2022-09-12T19:02:26.866 DEBUG:boto:Data: 2022-09-12T19:02:26.867 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.867 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.868 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.868 DEBUG:boto:Params: {} 2022-09-12T19:02:26.869 DEBUG:boto:Token: None 2022-09-12T19:02:26.869 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/ 2022-09-12T19:02:26.870 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts= 2022-09-12T19:02:26.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts=', 'Content-Length': '0'} 2022-09-12T19:02:26.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000301b746ef48529ba-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.914 DEBUG:boto:b'jdpiak-17dummy1000falsedummy2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:26.915 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:26.916 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:26.916 DEBUG:boto:path=/jdpiak-17/?prefix=dummy 2022-09-12T19:02:26.917 DEBUG:boto:auth_path=/jdpiak-17/?prefix=dummy 2022-09-12T19:02:26.917 DEBUG:boto:Method: GET 2022-09-12T19:02:26.918 DEBUG:boto:Path: /jdpiak-17/?prefix=dummy 2022-09-12T19:02:26.919 DEBUG:boto:Data: 2022-09-12T19:02:26.919 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.920 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:26.920 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.921 DEBUG:boto:Params: {} 2022-09-12T19:02:26.921 DEBUG:boto:Token: None 2022-09-12T19:02:26.922 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/ 2022-09-12T19:02:26.922 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts= 2022-09-12T19:02:26.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rog+aVZY23LKUz0UtQitwjiAvts=', 'Content-Length': '0'} 2022-09-12T19:02:26.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e55de183578cf645-00631f8242-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.928 DEBUG:boto:b'jdpiak-17dummy1000falsedummy2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:26.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:02:26.929 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:26.930 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:26.930 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:26.931 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:26.931 DEBUG:boto:Method: GET 2022-09-12T19:02:26.932 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:26.932 DEBUG:boto:Data: 2022-09-12T19:02:26.933 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.933 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.934 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.934 DEBUG:boto:Params: {} 2022-09-12T19:02:26.935 DEBUG:boto:Token: None 2022-09-12T19:02:26.935 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:26.936 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OEUqD5cODyOoP4tY4vQAIMuRoH8= 2022-09-12T19:02:26.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OEUqD5cODyOoP4tY4vQAIMuRoH8=', 'Content-Length': '0'} 2022-09-12T19:02:26.939 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000067fad85d6b3c166d-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.940 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:26.940 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:26.941 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:26.941 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:26.942 DEBUG:boto:Method: GET 2022-09-12T19:02:26.942 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:26.943 DEBUG:boto:Data: 2022-09-12T19:02:26.943 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.944 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:26.944 DEBUG:boto:Port: 8000 2022-09-12T19:02:26.945 DEBUG:boto:Params: {} 2022-09-12T19:02:26.945 DEBUG:boto:Token: None 2022-09-12T19:02:26.946 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:26.947 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OEUqD5cODyOoP4tY4vQAIMuRoH8= 2022-09-12T19:02:26.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OEUqD5cODyOoP4tY4vQAIMuRoH8=', 'Content-Length': '0'} 2022-09-12T19:02:26.949 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049a6deb62e34d4c0-00631f8242-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:26.990 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:26.991 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:26.991 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:26.992 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:26.992 DEBUG:boto:Method: HEAD 2022-09-12T19:02:26.993 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:26.994 DEBUG:boto:Data: 2022-09-12T19:02:26.994 DEBUG:boto:Headers: {} 2022-09-12T19:02:26.995 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:26.995 DEBUG:boto:Port: 8001 2022-09-12T19:02:26.996 DEBUG:boto:Params: {} 2022-09-12T19:02:26.997 DEBUG:boto:Token: None 2022-09-12T19:02:26.997 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:26 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:26.998 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b6njYnegxmclK2hKjf7KRi36Swo= 2022-09-12T19:02:26.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b6njYnegxmclK2hKjf7KRi36Swo=', 'Content-Length': '0'} 2022-09-12T19:02:27.001 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016518b5f2de480a6-00631f8242-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.003 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:27.004 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:27.005 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:27.006 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:27.007 DEBUG:boto:Method: HEAD 2022-09-12T19:02:27.008 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:27.008 DEBUG:boto:Data: 2022-09-12T19:02:27.009 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.010 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:27.011 DEBUG:boto:Port: 8000 2022-09-12T19:02:27.012 DEBUG:boto:Params: {} 2022-09-12T19:02:27.013 DEBUG:boto:Token: None 2022-09-12T19:02:27.014 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:27.016 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8W4+xfMp9gW+pTTDe3livbVwp70= 2022-09-12T19:02:27.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8W4+xfMp9gW+pTTDe3livbVwp70=', 'Content-Length': '0'} 2022-09-12T19:02:27.019 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d7761608b56c6719-00631f8243-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:02:27.022 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:27.023 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:27.024 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:27.025 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:27.026 DEBUG:boto:Method: GET 2022-09-12T19:02:27.027 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:27.028 DEBUG:boto:Data: 2022-09-12T19:02:27.029 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.030 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:27.031 DEBUG:boto:Port: 8001 2022-09-12T19:02:27.032 DEBUG:boto:Params: {} 2022-09-12T19:02:27.033 DEBUG:boto:Token: None 2022-09-12T19:02:27.034 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:27.035 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b7U83Q+fxUscF3yF4Tp7ofEq0rQ= 2022-09-12T19:02:27.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b7U83Q+fxUscF3yF4Tp7ofEq0rQ=', 'Content-Length': '0'} 2022-09-12T19:02:27.039 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000bb6f8728406b643-00631f8243-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.040 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:27.041 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:27.041 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:27.042 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:27.042 DEBUG:boto:Method: GET 2022-09-12T19:02:27.043 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:27.043 DEBUG:boto:Data: 2022-09-12T19:02:27.044 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.045 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:27.045 DEBUG:boto:Port: 8000 2022-09-12T19:02:27.046 DEBUG:boto:Params: {} 2022-09-12T19:02:27.046 DEBUG:boto:Token: None 2022-09-12T19:02:27.047 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:27.047 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b7U83Q+fxUscF3yF4Tp7ofEq0rQ= 2022-09-12T19:02:27.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b7U83Q+fxUscF3yF4Tp7ofEq0rQ=', 'Content-Length': '0'} 2022-09-12T19:02:27.050 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e5c70ccd471a5815-00631f8243-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.051 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:27.052 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:27.052 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:27.053 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:27.054 DEBUG:boto:Method: GET 2022-09-12T19:02:27.054 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:27.055 DEBUG:boto:Data: 2022-09-12T19:02:27.056 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.056 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:27.057 DEBUG:boto:Port: 8001 2022-09-12T19:02:27.057 DEBUG:boto:Params: {} 2022-09-12T19:02:27.058 DEBUG:boto:Token: None 2022-09-12T19:02:27.059 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/ 2022-09-12T19:02:27.059 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sCved7FpXmx+cQcRVY5/XFYLSWM= 2022-09-12T19:02:27.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sCved7FpXmx+cQcRVY5/XFYLSWM=', 'Content-Length': '0'} 2022-09-12T19:02:27.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000337e69eb12be970b-00631f8243-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.063 DEBUG:boto:b'jdpiak-17obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:27.064 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:27.065 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:27.066 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:27.066 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:27.067 DEBUG:boto:Method: GET 2022-09-12T19:02:27.068 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:27.069 DEBUG:boto:Data: 2022-09-12T19:02:27.070 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.070 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:27.071 DEBUG:boto:Port: 8000 2022-09-12T19:02:27.071 DEBUG:boto:Params: {} 2022-09-12T19:02:27.072 DEBUG:boto:Token: None 2022-09-12T19:02:27.073 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/ 2022-09-12T19:02:27.074 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sCved7FpXmx+cQcRVY5/XFYLSWM= 2022-09-12T19:02:27.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sCved7FpXmx+cQcRVY5/XFYLSWM=', 'Content-Length': '0'} 2022-09-12T19:02:27.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000365a74b1ffdd8c80-00631f8243-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.080 DEBUG:boto:b'jdpiak-17obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:27.081 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:27.082 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:27.083 DEBUG:boto:Method: HEAD 2022-09-12T19:02:27.084 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1 2022-09-12T19:02:27.085 DEBUG:boto:Data: 2022-09-12T19:02:27.085 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.086 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:27.087 DEBUG:boto:Port: 8001 2022-09-12T19:02:27.088 DEBUG:boto:Params: {} 2022-09-12T19:02:27.088 DEBUG:boto:Token: None 2022-09-12T19:02:27.089 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone1 2022-09-12T19:02:27.089 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YaCWSpDDDAP3rh494zsNockvzqI= 2022-09-12T19:02:27.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YaCWSpDDDAP3rh494zsNockvzqI=', 'Content-Length': '0'} 2022-09-12T19:02:27.093 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af203c9c1e5819ea-00631f8243-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.093 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:27.094 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:27.095 DEBUG:boto:Method: HEAD 2022-09-12T19:02:27.095 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1 2022-09-12T19:02:27.096 DEBUG:boto:Data: 2022-09-12T19:02:27.096 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.097 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:27.097 DEBUG:boto:Port: 8000 2022-09-12T19:02:27.098 DEBUG:boto:Params: {} 2022-09-12T19:02:27.098 DEBUG:boto:Token: None 2022-09-12T19:02:27.099 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone1 2022-09-12T19:02:27.099 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YaCWSpDDDAP3rh494zsNockvzqI= 2022-09-12T19:02:27.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YaCWSpDDDAP3rh494zsNockvzqI=', 'Content-Length': '0'} 2022-09-12T19:02:27.103 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a2e1bf869a604e6-00631f8243-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:02:27.104 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:27.105 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:27.105 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:27.106 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:27.106 DEBUG:boto:Method: GET 2022-09-12T19:02:27.107 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:27.107 DEBUG:boto:Data: 2022-09-12T19:02:27.108 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.108 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:27.109 DEBUG:boto:Port: 8001 2022-09-12T19:02:27.109 DEBUG:boto:Params: {} 2022-09-12T19:02:27.110 DEBUG:boto:Token: None 2022-09-12T19:02:27.111 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:27.111 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b7U83Q+fxUscF3yF4Tp7ofEq0rQ= 2022-09-12T19:02:27.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b7U83Q+fxUscF3yF4Tp7ofEq0rQ=', 'Content-Length': '0'} 2022-09-12T19:02:27.114 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de1f19579ef8cfc4-00631f8243-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.115 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:27.115 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:27.116 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:27.116 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:27.117 DEBUG:boto:Method: GET 2022-09-12T19:02:27.117 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:27.118 DEBUG:boto:Data: 2022-09-12T19:02:27.118 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.119 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:27.119 DEBUG:boto:Port: 8000 2022-09-12T19:02:27.120 DEBUG:boto:Params: {} 2022-09-12T19:02:27.121 DEBUG:boto:Token: None 2022-09-12T19:02:27.121 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:02:27.122 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b7U83Q+fxUscF3yF4Tp7ofEq0rQ= 2022-09-12T19:02:27.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b7U83Q+fxUscF3yF4Tp7ofEq0rQ=', 'Content-Length': '0'} 2022-09-12T19:02:27.124 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d0477ef2caa9925-00631f8243-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.166 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:27.167 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:27.167 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:27.168 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:27.168 DEBUG:boto:Method: GET 2022-09-12T19:02:27.169 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:27.169 DEBUG:boto:Data: 2022-09-12T19:02:27.170 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.171 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:27.171 DEBUG:boto:Port: 8001 2022-09-12T19:02:27.172 DEBUG:boto:Params: {} 2022-09-12T19:02:27.172 DEBUG:boto:Token: None 2022-09-12T19:02:27.173 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/ 2022-09-12T19:02:27.173 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sCved7FpXmx+cQcRVY5/XFYLSWM= 2022-09-12T19:02:27.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sCved7FpXmx+cQcRVY5/XFYLSWM=', 'Content-Length': '0'} 2022-09-12T19:02:27.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8550307a9227085-00631f8243-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.177 DEBUG:boto:b'jdpiak-17obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:27.178 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:27.179 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:27.179 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:27.180 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:27.180 DEBUG:boto:Method: GET 2022-09-12T19:02:27.181 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:02:27.181 DEBUG:boto:Data: 2022-09-12T19:02:27.182 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.182 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:27.183 DEBUG:boto:Port: 8000 2022-09-12T19:02:27.184 DEBUG:boto:Params: {} 2022-09-12T19:02:27.184 DEBUG:boto:Token: None 2022-09-12T19:02:27.185 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/ 2022-09-12T19:02:27.185 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sCved7FpXmx+cQcRVY5/XFYLSWM= 2022-09-12T19:02:27.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sCved7FpXmx+cQcRVY5/XFYLSWM=', 'Content-Length': '0'} 2022-09-12T19:02:27.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086c9d9e6c0dc45ed-00631f8243-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.190 DEBUG:boto:b'jdpiak-17obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:27.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:02:27.191 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:27.192 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:27.192 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.193 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.194 DEBUG:boto:Method: GET 2022-09-12T19:02:27.194 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.195 DEBUG:boto:Data: 2022-09-12T19:02:27.195 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.196 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:27.196 DEBUG:boto:Port: 8001 2022-09-12T19:02:27.197 DEBUG:boto:Params: {} 2022-09-12T19:02:27.197 DEBUG:boto:Token: None 2022-09-12T19:02:27.198 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.199 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xyfEk9IilAKHIdONJFz7ru67Ucg= 2022-09-12T19:02:27.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xyfEk9IilAKHIdONJFz7ru67Ucg=', 'Content-Length': '0'} 2022-09-12T19:02:27.201 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002daf1999ceb7e6ac-00631f8243-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.202 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:27.203 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:27.203 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.204 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.204 DEBUG:boto:Method: GET 2022-09-12T19:02:27.205 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.205 DEBUG:boto:Data: 2022-09-12T19:02:27.206 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.207 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:27.207 DEBUG:boto:Port: 8000 2022-09-12T19:02:27.208 DEBUG:boto:Params: {} 2022-09-12T19:02:27.208 DEBUG:boto:Token: None 2022-09-12T19:02:27.209 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.209 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xyfEk9IilAKHIdONJFz7ru67Ucg= 2022-09-12T19:02:27.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xyfEk9IilAKHIdONJFz7ru67Ucg=', 'Content-Length': '0'} 2022-09-12T19:02:27.212 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007cdd543e5a181de3-00631f8243-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.213 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:27.214 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:27.214 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.215 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.215 DEBUG:boto:Method: HEAD 2022-09-12T19:02:27.216 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.216 DEBUG:boto:Data: 2022-09-12T19:02:27.217 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.217 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:27.218 DEBUG:boto:Port: 8001 2022-09-12T19:02:27.218 DEBUG:boto:Params: {} 2022-09-12T19:02:27.219 DEBUG:boto:Token: None 2022-09-12T19:02:27.219 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.220 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+B2Ds033TyB1PpFtsw8OrEONNhg= 2022-09-12T19:02:27.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+B2Ds033TyB1PpFtsw8OrEONNhg=', 'Content-Length': '0'} 2022-09-12T19:02:27.222 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa2c4a80daf04808-00631f8243-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.223 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:27.224 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:27.224 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.225 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.225 DEBUG:boto:Method: HEAD 2022-09-12T19:02:27.226 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.226 DEBUG:boto:Data: 2022-09-12T19:02:27.227 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.227 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:27.228 DEBUG:boto:Port: 8000 2022-09-12T19:02:27.228 DEBUG:boto:Params: {} 2022-09-12T19:02:27.229 DEBUG:boto:Token: None 2022-09-12T19:02:27.229 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.230 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+B2Ds033TyB1PpFtsw8OrEONNhg= 2022-09-12T19:02:27.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+B2Ds033TyB1PpFtsw8OrEONNhg=', 'Content-Length': '0'} 2022-09-12T19:02:27.233 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000023db5cd51dea7ca5-00631f8243-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:02:27.234 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:27.235 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:27.235 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.236 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.236 DEBUG:boto:Method: GET 2022-09-12T19:02:27.237 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.237 DEBUG:boto:Data: 2022-09-12T19:02:27.238 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.238 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:27.239 DEBUG:boto:Port: 8001 2022-09-12T19:02:27.239 DEBUG:boto:Params: {} 2022-09-12T19:02:27.240 DEBUG:boto:Token: None 2022-09-12T19:02:27.241 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.241 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xyfEk9IilAKHIdONJFz7ru67Ucg= 2022-09-12T19:02:27.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xyfEk9IilAKHIdONJFz7ru67Ucg=', 'Content-Length': '0'} 2022-09-12T19:02:27.244 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b1dc1fc90dea5f3-00631f8243-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.286 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:27.287 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:27.288 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.288 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.289 DEBUG:boto:Method: GET 2022-09-12T19:02:27.289 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.290 DEBUG:boto:Data: 2022-09-12T19:02:27.290 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.291 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:27.291 DEBUG:boto:Port: 8000 2022-09-12T19:02:27.292 DEBUG:boto:Params: {} 2022-09-12T19:02:27.293 DEBUG:boto:Token: None 2022-09-12T19:02:27.293 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.294 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xyfEk9IilAKHIdONJFz7ru67Ucg= 2022-09-12T19:02:27.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xyfEk9IilAKHIdONJFz7ru67Ucg=', 'Content-Length': '0'} 2022-09-12T19:02:27.297 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004377202c75e75e29-00631f8243-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.298 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:27.298 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:27.299 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:27.299 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:27.300 DEBUG:boto:Method: GET 2022-09-12T19:02:27.300 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:27.301 DEBUG:boto:Data: 2022-09-12T19:02:27.301 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.302 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:27.303 DEBUG:boto:Port: 8001 2022-09-12T19:02:27.303 DEBUG:boto:Params: {} 2022-09-12T19:02:27.304 DEBUG:boto:Token: None 2022-09-12T19:02:27.304 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/ 2022-09-12T19:02:27.305 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sCved7FpXmx+cQcRVY5/XFYLSWM= 2022-09-12T19:02:27.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sCved7FpXmx+cQcRVY5/XFYLSWM=', 'Content-Length': '0'} 2022-09-12T19:02:27.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b09cef055bc46ff-00631f8243-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.310 DEBUG:boto:b'jdpiak-17obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:27.311 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:27.311 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:27.312 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:27.312 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:27.313 DEBUG:boto:Method: GET 2022-09-12T19:02:27.313 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:27.314 DEBUG:boto:Data: 2022-09-12T19:02:27.314 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.315 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:27.315 DEBUG:boto:Port: 8000 2022-09-12T19:02:27.316 DEBUG:boto:Params: {} 2022-09-12T19:02:27.317 DEBUG:boto:Token: None 2022-09-12T19:02:27.317 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/ 2022-09-12T19:02:27.318 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sCved7FpXmx+cQcRVY5/XFYLSWM= 2022-09-12T19:02:27.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sCved7FpXmx+cQcRVY5/XFYLSWM=', 'Content-Length': '0'} 2022-09-12T19:02:27.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036e78bb35b6bf800-00631f8243-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.322 DEBUG:boto:b'jdpiak-17obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:27.322 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:27.323 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:27.324 DEBUG:boto:Method: HEAD 2022-09-12T19:02:27.324 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2 2022-09-12T19:02:27.325 DEBUG:boto:Data: 2022-09-12T19:02:27.325 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.326 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:27.326 DEBUG:boto:Port: 8001 2022-09-12T19:02:27.327 DEBUG:boto:Params: {} 2022-09-12T19:02:27.327 DEBUG:boto:Token: None 2022-09-12T19:02:27.328 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone2 2022-09-12T19:02:27.328 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fJQaYR/cQxmNDkcfePpmDwoSnco= 2022-09-12T19:02:27.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fJQaYR/cQxmNDkcfePpmDwoSnco=', 'Content-Length': '0'} 2022-09-12T19:02:27.331 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f777c467d22c446-00631f8243-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.332 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:27.333 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:27.333 DEBUG:boto:Method: HEAD 2022-09-12T19:02:27.334 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2 2022-09-12T19:02:27.334 DEBUG:boto:Data: 2022-09-12T19:02:27.335 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.335 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:27.336 DEBUG:boto:Port: 8000 2022-09-12T19:02:27.336 DEBUG:boto:Params: {} 2022-09-12T19:02:27.337 DEBUG:boto:Token: None 2022-09-12T19:02:27.338 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone2 2022-09-12T19:02:27.338 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fJQaYR/cQxmNDkcfePpmDwoSnco= 2022-09-12T19:02:27.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fJQaYR/cQxmNDkcfePpmDwoSnco=', 'Content-Length': '0'} 2022-09-12T19:02:27.341 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f033c0006a398a2a-00631f8243-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:02:27.343 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:27.343 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:27.344 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.344 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.345 DEBUG:boto:Method: GET 2022-09-12T19:02:27.345 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.346 DEBUG:boto:Data: 2022-09-12T19:02:27.346 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.347 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:27.347 DEBUG:boto:Port: 8001 2022-09-12T19:02:27.348 DEBUG:boto:Params: {} 2022-09-12T19:02:27.349 DEBUG:boto:Token: None 2022-09-12T19:02:27.350 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.350 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xyfEk9IilAKHIdONJFz7ru67Ucg= 2022-09-12T19:02:27.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xyfEk9IilAKHIdONJFz7ru67Ucg=', 'Content-Length': '0'} 2022-09-12T19:02:27.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000096d3a179e4d1c9c-00631f8243-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.354 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:27.355 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:27.355 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.356 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.356 DEBUG:boto:Method: GET 2022-09-12T19:02:27.357 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.357 DEBUG:boto:Data: 2022-09-12T19:02:27.358 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.358 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:27.359 DEBUG:boto:Port: 8000 2022-09-12T19:02:27.359 DEBUG:boto:Params: {} 2022-09-12T19:02:27.360 DEBUG:boto:Token: None 2022-09-12T19:02:27.361 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:02:27.362 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xyfEk9IilAKHIdONJFz7ru67Ucg= 2022-09-12T19:02:27.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xyfEk9IilAKHIdONJFz7ru67Ucg=', 'Content-Length': '0'} 2022-09-12T19:02:27.365 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dcaeceb13e5006a9-00631f8243-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.366 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:27.366 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:27.367 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:27.368 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:27.368 DEBUG:boto:Method: GET 2022-09-12T19:02:27.369 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:27.369 DEBUG:boto:Data: 2022-09-12T19:02:27.370 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.370 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:27.371 DEBUG:boto:Port: 8001 2022-09-12T19:02:27.372 DEBUG:boto:Params: {} 2022-09-12T19:02:27.372 DEBUG:boto:Token: None 2022-09-12T19:02:27.373 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/ 2022-09-12T19:02:27.373 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sCved7FpXmx+cQcRVY5/XFYLSWM= 2022-09-12T19:02:27.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sCved7FpXmx+cQcRVY5/XFYLSWM=', 'Content-Length': '0'} 2022-09-12T19:02:27.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8d90649dcf7705e-00631f8243-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.378 DEBUG:boto:b'jdpiak-17obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:27.379 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:27.379 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:27.380 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:27.380 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:27.381 DEBUG:boto:Method: GET 2022-09-12T19:02:27.381 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:02:27.382 DEBUG:boto:Data: 2022-09-12T19:02:27.383 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.383 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:27.384 DEBUG:boto:Port: 8000 2022-09-12T19:02:27.385 DEBUG:boto:Params: {} 2022-09-12T19:02:27.386 DEBUG:boto:Token: None 2022-09-12T19:02:27.387 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/ 2022-09-12T19:02:27.388 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sCved7FpXmx+cQcRVY5/XFYLSWM= 2022-09-12T19:02:27.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sCved7FpXmx+cQcRVY5/XFYLSWM=', 'Content-Length': '0'} 2022-09-12T19:02:27.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b21a89a5101dfc92-00631f8243-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.393 DEBUG:boto:b'jdpiak-17obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:27.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:02:27.394 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:27.395 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:27.395 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.396 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.397 DEBUG:boto:Method: GET 2022-09-12T19:02:27.398 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.398 DEBUG:boto:Data: 2022-09-12T19:02:27.399 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.399 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:27.400 DEBUG:boto:Port: 8001 2022-09-12T19:02:27.400 DEBUG:boto:Params: {} 2022-09-12T19:02:27.401 DEBUG:boto:Token: None 2022-09-12T19:02:27.402 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.402 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DZjhJaLP+ghnrW4F+479sxgk+Ow= 2022-09-12T19:02:27.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DZjhJaLP+ghnrW4F+479sxgk+Ow=', 'Content-Length': '0'} 2022-09-12T19:02:27.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005754165645be18ad-00631f8243-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.406 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:27.407 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:27.407 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.408 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.408 DEBUG:boto:Method: GET 2022-09-12T19:02:27.409 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.410 DEBUG:boto:Data: 2022-09-12T19:02:27.410 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.411 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:27.411 DEBUG:boto:Port: 8000 2022-09-12T19:02:27.412 DEBUG:boto:Params: {} 2022-09-12T19:02:27.412 DEBUG:boto:Token: None 2022-09-12T19:02:27.413 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.414 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DZjhJaLP+ghnrW4F+479sxgk+Ow= 2022-09-12T19:02:27.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DZjhJaLP+ghnrW4F+479sxgk+Ow=', 'Content-Length': '0'} 2022-09-12T19:02:27.417 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f055ce3ae15996b6-00631f8243-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.417 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:27.418 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:27.419 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.419 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.420 DEBUG:boto:Method: HEAD 2022-09-12T19:02:27.420 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.421 DEBUG:boto:Data: 2022-09-12T19:02:27.421 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.422 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:27.423 DEBUG:boto:Port: 8001 2022-09-12T19:02:27.423 DEBUG:boto:Params: {} 2022-09-12T19:02:27.424 DEBUG:boto:Token: None 2022-09-12T19:02:27.424 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.425 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vnM15PYqc5/JNn15QHRrBULQJvA= 2022-09-12T19:02:27.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vnM15PYqc5/JNn15QHRrBULQJvA=', 'Content-Length': '0'} 2022-09-12T19:02:27.427 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a33e2c4200a76d4-00631f8243-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.428 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:27.429 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:27.429 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.430 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.431 DEBUG:boto:Method: HEAD 2022-09-12T19:02:27.431 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.432 DEBUG:boto:Data: 2022-09-12T19:02:27.432 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.433 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:27.433 DEBUG:boto:Port: 8000 2022-09-12T19:02:27.434 DEBUG:boto:Params: {} 2022-09-12T19:02:27.434 DEBUG:boto:Token: None 2022-09-12T19:02:27.435 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.436 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vnM15PYqc5/JNn15QHRrBULQJvA= 2022-09-12T19:02:27.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vnM15PYqc5/JNn15QHRrBULQJvA=', 'Content-Length': '0'} 2022-09-12T19:02:27.438 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093ec41e3280497e3-00631f8243-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:02:27.439 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:27.440 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:27.440 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.441 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.441 DEBUG:boto:Method: GET 2022-09-12T19:02:27.442 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.443 DEBUG:boto:Data: 2022-09-12T19:02:27.443 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.444 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:27.444 DEBUG:boto:Port: 8001 2022-09-12T19:02:27.445 DEBUG:boto:Params: {} 2022-09-12T19:02:27.445 DEBUG:boto:Token: None 2022-09-12T19:02:27.446 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.447 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DZjhJaLP+ghnrW4F+479sxgk+Ow= 2022-09-12T19:02:27.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DZjhJaLP+ghnrW4F+479sxgk+Ow=', 'Content-Length': '0'} 2022-09-12T19:02:27.450 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d7d17d5095d4fa7-00631f8243-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.451 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:27.451 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:27.452 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.452 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.453 DEBUG:boto:Method: GET 2022-09-12T19:02:27.454 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.454 DEBUG:boto:Data: 2022-09-12T19:02:27.455 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.455 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:27.456 DEBUG:boto:Port: 8000 2022-09-12T19:02:27.457 DEBUG:boto:Params: {} 2022-09-12T19:02:27.457 DEBUG:boto:Token: None 2022-09-12T19:02:27.458 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.459 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DZjhJaLP+ghnrW4F+479sxgk+Ow= 2022-09-12T19:02:27.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DZjhJaLP+ghnrW4F+479sxgk+Ow=', 'Content-Length': '0'} 2022-09-12T19:02:27.461 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a39475e74141fec0-00631f8243-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.462 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:27.463 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:27.464 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:27.464 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:27.465 DEBUG:boto:Method: GET 2022-09-12T19:02:27.466 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:27.467 DEBUG:boto:Data: 2022-09-12T19:02:27.468 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.469 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:27.469 DEBUG:boto:Port: 8001 2022-09-12T19:02:27.470 DEBUG:boto:Params: {} 2022-09-12T19:02:27.471 DEBUG:boto:Token: None 2022-09-12T19:02:27.472 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/ 2022-09-12T19:02:27.473 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sCved7FpXmx+cQcRVY5/XFYLSWM= 2022-09-12T19:02:27.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sCved7FpXmx+cQcRVY5/XFYLSWM=', 'Content-Length': '0'} 2022-09-12T19:02:27.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1aa3f49199bfd3d-00631f8243-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.477 DEBUG:boto:b'jdpiak-17obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:27.478 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:27.479 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:27.479 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:27.480 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:27.480 DEBUG:boto:Method: GET 2022-09-12T19:02:27.481 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:27.481 DEBUG:boto:Data: 2022-09-12T19:02:27.482 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.482 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:27.483 DEBUG:boto:Port: 8000 2022-09-12T19:02:27.483 DEBUG:boto:Params: {} 2022-09-12T19:02:27.484 DEBUG:boto:Token: None 2022-09-12T19:02:27.485 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/ 2022-09-12T19:02:27.486 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sCved7FpXmx+cQcRVY5/XFYLSWM= 2022-09-12T19:02:27.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sCved7FpXmx+cQcRVY5/XFYLSWM=', 'Content-Length': '0'} 2022-09-12T19:02:27.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000162b69501f70b983-00631f8243-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.493 DEBUG:boto:b'jdpiak-17obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:27.494 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:27.495 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:27.495 DEBUG:boto:Method: HEAD 2022-09-12T19:02:27.496 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3 2022-09-12T19:02:27.497 DEBUG:boto:Data: 2022-09-12T19:02:27.497 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.498 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:27.498 DEBUG:boto:Port: 8001 2022-09-12T19:02:27.499 DEBUG:boto:Params: {} 2022-09-12T19:02:27.499 DEBUG:boto:Token: None 2022-09-12T19:02:27.500 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone3 2022-09-12T19:02:27.500 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JR27H928aIvUn5XTMte2HPxF7fc= 2022-09-12T19:02:27.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JR27H928aIvUn5XTMte2HPxF7fc=', 'Content-Length': '0'} 2022-09-12T19:02:27.504 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000cf023a79fa953d6-00631f8243-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.505 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:27.506 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:27.507 DEBUG:boto:Method: HEAD 2022-09-12T19:02:27.508 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3 2022-09-12T19:02:27.509 DEBUG:boto:Data: 2022-09-12T19:02:27.510 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.511 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:27.512 DEBUG:boto:Port: 8000 2022-09-12T19:02:27.513 DEBUG:boto:Params: {} 2022-09-12T19:02:27.514 DEBUG:boto:Token: None 2022-09-12T19:02:27.515 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone3 2022-09-12T19:02:27.516 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JR27H928aIvUn5XTMte2HPxF7fc= 2022-09-12T19:02:27.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JR27H928aIvUn5XTMte2HPxF7fc=', 'Content-Length': '0'} 2022-09-12T19:02:27.520 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c1c8020f5cfb416-00631f8243-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:02:27.521 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:27.522 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:27.523 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.523 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.524 DEBUG:boto:Method: GET 2022-09-12T19:02:27.525 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.526 DEBUG:boto:Data: 2022-09-12T19:02:27.526 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.527 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:27.528 DEBUG:boto:Port: 8001 2022-09-12T19:02:27.528 DEBUG:boto:Params: {} 2022-09-12T19:02:27.529 DEBUG:boto:Token: None 2022-09-12T19:02:27.530 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.531 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DZjhJaLP+ghnrW4F+479sxgk+Ow= 2022-09-12T19:02:27.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DZjhJaLP+ghnrW4F+479sxgk+Ow=', 'Content-Length': '0'} 2022-09-12T19:02:27.534 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009a4510ddef9e396-00631f8243-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.535 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:27.536 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:27.536 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.537 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.538 DEBUG:boto:Method: GET 2022-09-12T19:02:27.539 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.539 DEBUG:boto:Data: 2022-09-12T19:02:27.540 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.541 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:27.542 DEBUG:boto:Port: 8000 2022-09-12T19:02:27.542 DEBUG:boto:Params: {} 2022-09-12T19:02:27.543 DEBUG:boto:Token: None 2022-09-12T19:02:27.544 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:02:27.545 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DZjhJaLP+ghnrW4F+479sxgk+Ow= 2022-09-12T19:02:27.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DZjhJaLP+ghnrW4F+479sxgk+Ow=', 'Content-Length': '0'} 2022-09-12T19:02:27.548 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a14affbb008d4fb0-00631f8243-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.594 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:27.595 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:27.596 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:27.596 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:27.597 DEBUG:boto:Method: GET 2022-09-12T19:02:27.598 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:27.598 DEBUG:boto:Data: 2022-09-12T19:02:27.599 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.600 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:27.601 DEBUG:boto:Port: 8001 2022-09-12T19:02:27.601 DEBUG:boto:Params: {} 2022-09-12T19:02:27.602 DEBUG:boto:Token: None 2022-09-12T19:02:27.603 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/ 2022-09-12T19:02:27.604 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sCved7FpXmx+cQcRVY5/XFYLSWM= 2022-09-12T19:02:27.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sCved7FpXmx+cQcRVY5/XFYLSWM=', 'Content-Length': '0'} 2022-09-12T19:02:27.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e3a105d7329fac6-00631f8243-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.608 DEBUG:boto:b'jdpiak-17obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:27.609 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:27.610 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:27.611 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:27.611 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:27.612 DEBUG:boto:Method: GET 2022-09-12T19:02:27.613 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:02:27.613 DEBUG:boto:Data: 2022-09-12T19:02:27.615 DEBUG:boto:Headers: {} 2022-09-12T19:02:27.615 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:27.616 DEBUG:boto:Port: 8000 2022-09-12T19:02:27.617 DEBUG:boto:Params: {} 2022-09-12T19:02:27.618 DEBUG:boto:Token: None 2022-09-12T19:02:27.619 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:27 GMT /jdpiak-17/ 2022-09-12T19:02:27.620 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sCved7FpXmx+cQcRVY5/XFYLSWM= 2022-09-12T19:02:27.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sCved7FpXmx+cQcRVY5/XFYLSWM=', 'Content-Length': '0'} 2022-09-12T19:02:27.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c2f7c3a124d4dfa-00631f8243-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:27.624 DEBUG:boto:b'jdpiak-17obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:27.626 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-17 zones={test-zone3, test-zone2} 2022-09-12T19:02:27.626 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-18 2022-09-12T19:02:27.627 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:02:27.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:02:27.628 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:02:27.692 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:02:27.692 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:02:27.755 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:02:27.752+0000 7fae64f46600 1 bucket sync caught up with source: 2022-09-12T19:02:27.756 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000010.31.5, , 00000000003.9.9, , 00000000007.18.9, , , , , ] 2022-09-12T19:02:27.756 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000010.31.5, , 00000000003.9.9, , 00000000007.18.9, , , , , ] 2022-09-12T19:02:27.756 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:02:27.752+0000 7fae64f46600 0 bucket checkpoint complete 2022-09-12T19:02:27.782 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-18 2022-09-12T19:02:27.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:02:27.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:02:27.785 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:02:27.853 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:02:27.853 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:02:27.893 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:02:27.889+0000 7fa1907dd600 1 bucket sync caught up with source: 2022-09-12T19:02:27.894 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000010.31.5, , 00000000003.9.9, , 00000000007.18.9, , , , , ] 2022-09-12T19:02:27.894 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000010.31.5, , 00000000003.9.9, , 00000000007.18.9, , , , , ] 2022-09-12T19:02:27.894 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:02:27.889+0000 7fa1907dd600 0 bucket checkpoint complete 2022-09-12T19:02:27.906 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-18 2022-09-12T19:02:27.907 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:02:27.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:02:27.908 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:02:27.965 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:02:27.965 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:02:28.015 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:02:28.012+0000 7f3332c8d600 1 bucket sync caught up with source: 2022-09-12T19:02:28.016 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000010.31.5, , 00000000003.9.9, , 00000000007.18.9, , , , , ] 2022-09-12T19:02:28.017 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000010.31.5, , 00000000003.9.9, , 00000000007.18.9, , , , , ] 2022-09-12T19:02:28.017 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:02:28.012+0000 7f3332c8d600 0 bucket checkpoint complete 2022-09-12T19:02:28.027 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-18 2022-09-12T19:02:28.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:02:28.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:02:28.029 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:02:28.089 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:02:28.089 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:02:28.127 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:02:28.123+0000 7f05a9208600 1 bucket sync caught up with source: 2022-09-12T19:02:28.127 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.31.9, , 00000000010.34.5, , 00000000007.28.9, , , , , ] 2022-09-12T19:02:28.128 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.31.9, , 00000000010.34.5, , 00000000007.28.9, , , , , ] 2022-09-12T19:02:28.128 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:02:28.123+0000 7f05a9208600 0 bucket checkpoint complete 2022-09-12T19:02:28.139 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-18 2022-09-12T19:02:28.140 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:02:28.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:02:28.141 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:02:28.199 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:02:28.199 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:02:28.253 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:02:28.250+0000 7f896b122600 1 bucket sync caught up with source: 2022-09-12T19:02:28.254 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.31.9, , 00000000010.34.5, , 00000000007.28.9, , , , , ] 2022-09-12T19:02:28.254 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.31.9, , 00000000010.34.5, , 00000000007.28.9, , , , , ] 2022-09-12T19:02:28.254 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:02:28.250+0000 7f896b122600 0 bucket checkpoint complete 2022-09-12T19:02:28.273 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-18 2022-09-12T19:02:28.274 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-18 2022-09-12T19:02:28.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:02:28.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:02:28.276 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:02:28.333 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:02:28.334 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:02:28.385 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:02:28.382+0000 7feb48f43600 1 bucket sync caught up with source: 2022-09-12T19:02:28.385 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.39.9, , 00000000003.19.9, , 00000000016.53.5, , , , , ] 2022-09-12T19:02:28.385 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.39.9, , 00000000003.19.9, , 00000000016.53.5, , , , , ] 2022-09-12T19:02:28.386 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:02:28.382+0000 7feb48f43600 0 bucket checkpoint complete 2022-09-12T19:02:28.407 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-18 2022-09-12T19:02:28.408 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:02:28.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:02:28.409 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:02:28.467 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:02:28.467 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:02:28.503 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:02:28.500+0000 7fcde231b600 1 bucket sync caught up with source: 2022-09-12T19:02:28.503 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000003.39.9, , 00000000003.19.9, , 00000000016.53.5, , , , , ] 2022-09-12T19:02:28.504 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000003.39.9, , 00000000003.19.9, , 00000000016.53.5, , , , , ] 2022-09-12T19:02:28.504 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:02:28.500+0000 7fcde231b600 0 bucket checkpoint complete 2022-09-12T19:02:28.516 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-18 2022-09-12T19:02:28.516 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-18 zones={test-zone2, test-zone1} 2022-09-12T19:02:28.517 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:28.517 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:28.518 DEBUG:boto:Method: HEAD 2022-09-12T19:02:28.518 DEBUG:boto:Path: /jdpiak-18/ 2022-09-12T19:02:28.519 DEBUG:boto:Data: 2022-09-12T19:02:28.519 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.520 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:28.521 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.521 DEBUG:boto:Params: {} 2022-09-12T19:02:28.522 DEBUG:boto:Token: None 2022-09-12T19:02:28.522 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/ 2022-09-12T19:02:28.523 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OqN8mQAbeJg7IbGs0btFSPjyWWs= 2022-09-12T19:02:28.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OqN8mQAbeJg7IbGs0btFSPjyWWs=', 'Content-Length': '0'} 2022-09-12T19:02:28.527 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f43e5054fd22843c-00631f8244-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.528 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:28.528 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:28.529 DEBUG:boto:Method: HEAD 2022-09-12T19:02:28.529 DEBUG:boto:Path: /jdpiak-18/ 2022-09-12T19:02:28.530 DEBUG:boto:Data: 2022-09-12T19:02:28.531 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.532 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:28.533 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.533 DEBUG:boto:Params: {} 2022-09-12T19:02:28.534 DEBUG:boto:Token: None 2022-09-12T19:02:28.535 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/ 2022-09-12T19:02:28.536 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OqN8mQAbeJg7IbGs0btFSPjyWWs= 2022-09-12T19:02:28.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OqN8mQAbeJg7IbGs0btFSPjyWWs=', 'Content-Length': '0'} 2022-09-12T19:02:28.541 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030fed0f30e9574cb-00631f8244-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.542 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:02:28.543 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:28.543 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:28.544 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:28.544 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:28.545 DEBUG:boto:Method: GET 2022-09-12T19:02:28.546 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:28.547 DEBUG:boto:Data: 2022-09-12T19:02:28.547 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.548 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:28.549 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.549 DEBUG:boto:Params: {} 2022-09-12T19:02:28.550 DEBUG:boto:Token: None 2022-09-12T19:02:28.551 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/?versions 2022-09-12T19:02:28.551 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h45RZBJAziyKrorCyTqiuVv0r7Y= 2022-09-12T19:02:28.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h45RZBJAziyKrorCyTqiuVv0r7Y=', 'Content-Length': '0'} 2022-09-12T19:02:28.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7049e6f046c0620-00631f8244-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.555 DEBUG:boto:b'jdpiak-18999falsedummyqUufSwmbKETbT7yNOH8sL0A4tddUbNOtrue2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ToUtj2XoS.tz3goIbvjFwiigMyzGOpLtrue2022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2DPksivOWVYhJDrEwQqHYV7wK1Ul1JkWtrue2022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Ztrue2022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:28.556 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:02:28.557 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:02:28.557 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:02:28.558 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:02:28.558 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:02:28.559 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:28.559 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:28.560 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:28.560 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:28.561 DEBUG:boto:Method: GET 2022-09-12T19:02:28.561 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:28.562 DEBUG:boto:Data: 2022-09-12T19:02:28.562 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.563 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:28.564 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.564 DEBUG:boto:Params: {} 2022-09-12T19:02:28.565 DEBUG:boto:Token: None 2022-09-12T19:02:28.565 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/?versions 2022-09-12T19:02:28.566 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h45RZBJAziyKrorCyTqiuVv0r7Y= 2022-09-12T19:02:28.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h45RZBJAziyKrorCyTqiuVv0r7Y=', 'Content-Length': '0'} 2022-09-12T19:02:28.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d717e7e2e8959dbc-00631f8244-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.570 DEBUG:boto:b'jdpiak-18999falsedummyqUufSwmbKETbT7yNOH8sL0A4tddUbNOtrue2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ToUtj2XoS.tz3goIbvjFwiigMyzGOpLtrue2022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2DPksivOWVYhJDrEwQqHYV7wK1Ul1JkWtrue2022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Ztrue2022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:28.571 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:02:28.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:02:28.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:02:28.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:02:28.574 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:28.575 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:28.575 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:28.576 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:28.576 DEBUG:boto:Method: GET 2022-09-12T19:02:28.577 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:28.578 DEBUG:boto:Data: 2022-09-12T19:02:28.578 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.579 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:28.579 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.580 DEBUG:boto:Params: {} 2022-09-12T19:02:28.581 DEBUG:boto:Token: None 2022-09-12T19:02:28.581 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/?versions 2022-09-12T19:02:28.582 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h45RZBJAziyKrorCyTqiuVv0r7Y= 2022-09-12T19:02:28.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h45RZBJAziyKrorCyTqiuVv0r7Y=', 'Content-Length': '0'} 2022-09-12T19:02:28.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e38a429986100bb-00631f8244-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.585 DEBUG:boto:b'jdpiak-18999falsedummyqUufSwmbKETbT7yNOH8sL0A4tddUbNOtrue2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ToUtj2XoS.tz3goIbvjFwiigMyzGOpLtrue2022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2DPksivOWVYhJDrEwQqHYV7wK1Ul1JkWtrue2022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Ztrue2022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:28.587 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:28.587 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:28.588 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:28.589 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:28.589 DEBUG:boto:Method: GET 2022-09-12T19:02:28.590 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:28.590 DEBUG:boto:Data: 2022-09-12T19:02:28.591 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.591 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:28.592 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.592 DEBUG:boto:Params: {} 2022-09-12T19:02:28.593 DEBUG:boto:Token: None 2022-09-12T19:02:28.593 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/?versions 2022-09-12T19:02:28.594 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h45RZBJAziyKrorCyTqiuVv0r7Y= 2022-09-12T19:02:28.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h45RZBJAziyKrorCyTqiuVv0r7Y=', 'Content-Length': '0'} 2022-09-12T19:02:28.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be22b5f33927031b-00631f8244-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.598 DEBUG:boto:b'jdpiak-18999falsedummyqUufSwmbKETbT7yNOH8sL0A4tddUbNOtrue2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ToUtj2XoS.tz3goIbvjFwiigMyzGOpLtrue2022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2DPksivOWVYhJDrEwQqHYV7wK1Ul1JkWtrue2022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Ztrue2022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:28.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:02:28.599 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:28.600 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:28.601 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.601 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.602 DEBUG:boto:Method: GET 2022-09-12T19:02:28.602 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.603 DEBUG:boto:Data: 2022-09-12T19:02:28.604 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.604 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:28.605 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.605 DEBUG:boto:Params: {} 2022-09-12T19:02:28.606 DEBUG:boto:Token: None 2022-09-12T19:02:28.606 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.607 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OMPrULflRQ5Bn/QFRSdSAeTNF60= 2022-09-12T19:02:28.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OMPrULflRQ5Bn/QFRSdSAeTNF60=', 'Content-Length': '0'} 2022-09-12T19:02:28.611 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000388df9c27fd97707-00631f8244-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.612 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:28.613 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:28.613 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.614 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.615 DEBUG:boto:Method: GET 2022-09-12T19:02:28.616 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.616 DEBUG:boto:Data: 2022-09-12T19:02:28.617 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.617 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:28.618 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.619 DEBUG:boto:Params: {} 2022-09-12T19:02:28.620 DEBUG:boto:Token: None 2022-09-12T19:02:28.621 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.621 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OMPrULflRQ5Bn/QFRSdSAeTNF60= 2022-09-12T19:02:28.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OMPrULflRQ5Bn/QFRSdSAeTNF60=', 'Content-Length': '0'} 2022-09-12T19:02:28.624 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003846f55b14113a53-00631f8244-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.625 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:28.625 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:28.626 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.626 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.627 DEBUG:boto:Method: HEAD 2022-09-12T19:02:28.627 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.628 DEBUG:boto:Data: 2022-09-12T19:02:28.629 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.629 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:28.630 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.631 DEBUG:boto:Params: {} 2022-09-12T19:02:28.631 DEBUG:boto:Token: None 2022-09-12T19:02:28.632 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.633 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:23+SQEcYBcw/4YBXuHhlAluOcz8= 2022-09-12T19:02:28.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:23+SQEcYBcw/4YBXuHhlAluOcz8=', 'Content-Length': '0'} 2022-09-12T19:02:28.637 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d47dc40ac7afb6a1-00631f8244-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.638 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:28.639 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:28.640 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.640 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.641 DEBUG:boto:Method: HEAD 2022-09-12T19:02:28.642 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.642 DEBUG:boto:Data: 2022-09-12T19:02:28.643 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.644 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:28.645 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.646 DEBUG:boto:Params: {} 2022-09-12T19:02:28.647 DEBUG:boto:Token: None 2022-09-12T19:02:28.648 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.649 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:23+SQEcYBcw/4YBXuHhlAluOcz8= 2022-09-12T19:02:28.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:23+SQEcYBcw/4YBXuHhlAluOcz8=', 'Content-Length': '0'} 2022-09-12T19:02:28.652 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008d8e7daad581312c-00631f8244-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:02:28.654 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:28.655 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:28.656 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.656 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.657 DEBUG:boto:Method: GET 2022-09-12T19:02:28.658 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.659 DEBUG:boto:Data: 2022-09-12T19:02:28.660 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.661 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:28.662 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.662 DEBUG:boto:Params: {} 2022-09-12T19:02:28.663 DEBUG:boto:Token: None 2022-09-12T19:02:28.664 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.665 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OMPrULflRQ5Bn/QFRSdSAeTNF60= 2022-09-12T19:02:28.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OMPrULflRQ5Bn/QFRSdSAeTNF60=', 'Content-Length': '0'} 2022-09-12T19:02:28.669 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e8cc43e951cf9d46-00631f8244-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.670 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:28.671 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:28.672 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.672 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.673 DEBUG:boto:Method: GET 2022-09-12T19:02:28.673 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.674 DEBUG:boto:Data: 2022-09-12T19:02:28.675 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.675 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:28.676 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.676 DEBUG:boto:Params: {} 2022-09-12T19:02:28.677 DEBUG:boto:Token: None 2022-09-12T19:02:28.677 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.678 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OMPrULflRQ5Bn/QFRSdSAeTNF60= 2022-09-12T19:02:28.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OMPrULflRQ5Bn/QFRSdSAeTNF60=', 'Content-Length': '0'} 2022-09-12T19:02:28.680 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c01a166ce1075665-00631f8244-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.681 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:28.682 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:28.682 DEBUG:boto:path=/jdpiak-18/?prefix=dummy 2022-09-12T19:02:28.683 DEBUG:boto:auth_path=/jdpiak-18/?prefix=dummy 2022-09-12T19:02:28.683 DEBUG:boto:Method: GET 2022-09-12T19:02:28.684 DEBUG:boto:Path: /jdpiak-18/?prefix=dummy 2022-09-12T19:02:28.684 DEBUG:boto:Data: 2022-09-12T19:02:28.685 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.685 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:28.686 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.686 DEBUG:boto:Params: {} 2022-09-12T19:02:28.687 DEBUG:boto:Token: None 2022-09-12T19:02:28.688 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/ 2022-09-12T19:02:28.688 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XAmyMxVrDT5PQj181TJgF4HtR3s= 2022-09-12T19:02:28.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XAmyMxVrDT5PQj181TJgF4HtR3s=', 'Content-Length': '0'} 2022-09-12T19:02:28.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccb383bb08502783-00631f8244-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.734 DEBUG:boto:b'jdpiak-18dummy1000falsedummy2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:28.736 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:28.737 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:28.738 DEBUG:boto:path=/jdpiak-18/?prefix=dummy 2022-09-12T19:02:28.739 DEBUG:boto:auth_path=/jdpiak-18/?prefix=dummy 2022-09-12T19:02:28.740 DEBUG:boto:Method: GET 2022-09-12T19:02:28.741 DEBUG:boto:Path: /jdpiak-18/?prefix=dummy 2022-09-12T19:02:28.742 DEBUG:boto:Data: 2022-09-12T19:02:28.743 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.744 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:28.745 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.746 DEBUG:boto:Params: {} 2022-09-12T19:02:28.747 DEBUG:boto:Token: None 2022-09-12T19:02:28.748 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/ 2022-09-12T19:02:28.749 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XAmyMxVrDT5PQj181TJgF4HtR3s= 2022-09-12T19:02:28.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XAmyMxVrDT5PQj181TJgF4HtR3s=', 'Content-Length': '0'} 2022-09-12T19:02:28.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba62dc167735ee60-00631f8244-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.753 DEBUG:boto:b'jdpiak-18dummy1000falsedummy2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:28.753 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:28.754 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:28.755 DEBUG:boto:Method: HEAD 2022-09-12T19:02:28.755 DEBUG:boto:Path: /jdpiak-18/dummy 2022-09-12T19:02:28.756 DEBUG:boto:Data: 2022-09-12T19:02:28.756 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.757 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:28.758 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.759 DEBUG:boto:Params: {} 2022-09-12T19:02:28.760 DEBUG:boto:Token: None 2022-09-12T19:02:28.760 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/dummy 2022-09-12T19:02:28.761 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3FBV3a0nipEhUMI5hYLih24jJ0Y= 2022-09-12T19:02:28.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3FBV3a0nipEhUMI5hYLih24jJ0Y=', 'Content-Length': '0'} 2022-09-12T19:02:28.764 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000040a429d5b0dda11d-00631f8244-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.765 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:28.766 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:28.767 DEBUG:boto:Method: HEAD 2022-09-12T19:02:28.767 DEBUG:boto:Path: /jdpiak-18/dummy 2022-09-12T19:02:28.768 DEBUG:boto:Data: 2022-09-12T19:02:28.768 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.769 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:28.769 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.770 DEBUG:boto:Params: {} 2022-09-12T19:02:28.771 DEBUG:boto:Token: None 2022-09-12T19:02:28.771 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/dummy 2022-09-12T19:02:28.772 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3FBV3a0nipEhUMI5hYLih24jJ0Y= 2022-09-12T19:02:28.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3FBV3a0nipEhUMI5hYLih24jJ0Y=', 'Content-Length': '0'} 2022-09-12T19:02:28.775 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007bfcdb093b2ef4ab-00631f8244-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:02:28.777 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:28.778 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:28.779 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.780 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.780 DEBUG:boto:Method: GET 2022-09-12T19:02:28.781 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.782 DEBUG:boto:Data: 2022-09-12T19:02:28.783 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.784 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:28.784 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.785 DEBUG:boto:Params: {} 2022-09-12T19:02:28.786 DEBUG:boto:Token: None 2022-09-12T19:02:28.786 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.787 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OMPrULflRQ5Bn/QFRSdSAeTNF60= 2022-09-12T19:02:28.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OMPrULflRQ5Bn/QFRSdSAeTNF60=', 'Content-Length': '0'} 2022-09-12T19:02:28.790 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e5229e47f30455fb-00631f8244-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.790 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:28.791 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:28.792 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.792 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.793 DEBUG:boto:Method: GET 2022-09-12T19:02:28.793 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.794 DEBUG:boto:Data: 2022-09-12T19:02:28.794 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.795 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:28.795 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.796 DEBUG:boto:Params: {} 2022-09-12T19:02:28.797 DEBUG:boto:Token: None 2022-09-12T19:02:28.797 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:28.798 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OMPrULflRQ5Bn/QFRSdSAeTNF60= 2022-09-12T19:02:28.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OMPrULflRQ5Bn/QFRSdSAeTNF60=', 'Content-Length': '0'} 2022-09-12T19:02:28.801 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000057683c411af80e0a-00631f8244-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.803 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:28.803 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:28.804 DEBUG:boto:path=/jdpiak-18/?prefix=dummy 2022-09-12T19:02:28.804 DEBUG:boto:auth_path=/jdpiak-18/?prefix=dummy 2022-09-12T19:02:28.805 DEBUG:boto:Method: GET 2022-09-12T19:02:28.806 DEBUG:boto:Path: /jdpiak-18/?prefix=dummy 2022-09-12T19:02:28.806 DEBUG:boto:Data: 2022-09-12T19:02:28.807 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.807 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:28.808 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.808 DEBUG:boto:Params: {} 2022-09-12T19:02:28.809 DEBUG:boto:Token: None 2022-09-12T19:02:28.809 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/ 2022-09-12T19:02:28.810 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XAmyMxVrDT5PQj181TJgF4HtR3s= 2022-09-12T19:02:28.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XAmyMxVrDT5PQj181TJgF4HtR3s=', 'Content-Length': '0'} 2022-09-12T19:02:28.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039a8834665e3c451-00631f8244-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.814 DEBUG:boto:b'jdpiak-18dummy1000falsedummy2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:28.815 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:28.815 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:28.816 DEBUG:boto:path=/jdpiak-18/?prefix=dummy 2022-09-12T19:02:28.816 DEBUG:boto:auth_path=/jdpiak-18/?prefix=dummy 2022-09-12T19:02:28.817 DEBUG:boto:Method: GET 2022-09-12T19:02:28.818 DEBUG:boto:Path: /jdpiak-18/?prefix=dummy 2022-09-12T19:02:28.818 DEBUG:boto:Data: 2022-09-12T19:02:28.819 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.819 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:28.820 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.820 DEBUG:boto:Params: {} 2022-09-12T19:02:28.821 DEBUG:boto:Token: None 2022-09-12T19:02:28.822 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/ 2022-09-12T19:02:28.822 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XAmyMxVrDT5PQj181TJgF4HtR3s= 2022-09-12T19:02:28.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XAmyMxVrDT5PQj181TJgF4HtR3s=', 'Content-Length': '0'} 2022-09-12T19:02:28.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000286157f2fc2571f5-00631f8244-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.866 DEBUG:boto:b'jdpiak-18dummy1000falsedummy2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:28.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:02:28.868 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:28.868 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:28.869 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.869 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.870 DEBUG:boto:Method: GET 2022-09-12T19:02:28.870 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.871 DEBUG:boto:Data: 2022-09-12T19:02:28.871 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.872 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:28.872 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.873 DEBUG:boto:Params: {} 2022-09-12T19:02:28.874 DEBUG:boto:Token: None 2022-09-12T19:02:28.874 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.875 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+R8drOs2yUmxlsuTU6BJZFEVkuM= 2022-09-12T19:02:28.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+R8drOs2yUmxlsuTU6BJZFEVkuM=', 'Content-Length': '0'} 2022-09-12T19:02:28.878 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097d542ac04c5b100-00631f8244-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.879 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:28.879 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:28.880 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.880 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.881 DEBUG:boto:Method: GET 2022-09-12T19:02:28.882 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.882 DEBUG:boto:Data: 2022-09-12T19:02:28.883 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.883 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:28.884 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.884 DEBUG:boto:Params: {} 2022-09-12T19:02:28.885 DEBUG:boto:Token: None 2022-09-12T19:02:28.885 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.886 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+R8drOs2yUmxlsuTU6BJZFEVkuM= 2022-09-12T19:02:28.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+R8drOs2yUmxlsuTU6BJZFEVkuM=', 'Content-Length': '0'} 2022-09-12T19:02:28.889 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9e434de0307ae09-00631f8244-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.890 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:28.890 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:28.891 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.891 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.892 DEBUG:boto:Method: HEAD 2022-09-12T19:02:28.892 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.893 DEBUG:boto:Data: 2022-09-12T19:02:28.893 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.894 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:28.894 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.895 DEBUG:boto:Params: {} 2022-09-12T19:02:28.895 DEBUG:boto:Token: None 2022-09-12T19:02:28.896 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.896 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8NX1MblmWpBZ3r8K1DV4mm2OPmQ= 2022-09-12T19:02:28.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8NX1MblmWpBZ3r8K1DV4mm2OPmQ=', 'Content-Length': '0'} 2022-09-12T19:02:28.899 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa68af0f53da0351-00631f8244-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.900 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:28.900 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:28.901 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.901 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.902 DEBUG:boto:Method: HEAD 2022-09-12T19:02:28.902 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.903 DEBUG:boto:Data: 2022-09-12T19:02:28.903 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.904 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:28.904 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.905 DEBUG:boto:Params: {} 2022-09-12T19:02:28.906 DEBUG:boto:Token: None 2022-09-12T19:02:28.906 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.907 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8NX1MblmWpBZ3r8K1DV4mm2OPmQ= 2022-09-12T19:02:28.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8NX1MblmWpBZ3r8K1DV4mm2OPmQ=', 'Content-Length': '0'} 2022-09-12T19:02:28.909 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000477b010a9b17fc73-00631f8244-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:02:28.911 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:28.911 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:28.912 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.912 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.913 DEBUG:boto:Method: GET 2022-09-12T19:02:28.913 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.914 DEBUG:boto:Data: 2022-09-12T19:02:28.914 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.915 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:28.916 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.916 DEBUG:boto:Params: {} 2022-09-12T19:02:28.917 DEBUG:boto:Token: None 2022-09-12T19:02:28.917 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.918 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+R8drOs2yUmxlsuTU6BJZFEVkuM= 2022-09-12T19:02:28.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+R8drOs2yUmxlsuTU6BJZFEVkuM=', 'Content-Length': '0'} 2022-09-12T19:02:28.920 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000658eadd9de363dc8-00631f8244-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.921 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:28.922 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:28.923 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.923 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.924 DEBUG:boto:Method: GET 2022-09-12T19:02:28.924 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.925 DEBUG:boto:Data: 2022-09-12T19:02:28.925 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.926 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:28.927 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.927 DEBUG:boto:Params: {} 2022-09-12T19:02:28.928 DEBUG:boto:Token: None 2022-09-12T19:02:28.929 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.929 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+R8drOs2yUmxlsuTU6BJZFEVkuM= 2022-09-12T19:02:28.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+R8drOs2yUmxlsuTU6BJZFEVkuM=', 'Content-Length': '0'} 2022-09-12T19:02:28.932 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ae5214fe647ddf3-00631f8244-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.933 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:28.934 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:28.934 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:28.935 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:28.935 DEBUG:boto:Method: GET 2022-09-12T19:02:28.936 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:28.936 DEBUG:boto:Data: 2022-09-12T19:02:28.937 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.937 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:28.938 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.939 DEBUG:boto:Params: {} 2022-09-12T19:02:28.940 DEBUG:boto:Token: None 2022-09-12T19:02:28.941 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/ 2022-09-12T19:02:28.942 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XAmyMxVrDT5PQj181TJgF4HtR3s= 2022-09-12T19:02:28.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XAmyMxVrDT5PQj181TJgF4HtR3s=', 'Content-Length': '0'} 2022-09-12T19:02:28.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072594509bc23d06d-00631f8244-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.947 DEBUG:boto:b'jdpiak-18obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:28.949 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:28.949 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:28.950 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:28.951 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:28.951 DEBUG:boto:Method: GET 2022-09-12T19:02:28.952 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:28.952 DEBUG:boto:Data: 2022-09-12T19:02:28.953 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.953 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:28.954 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.954 DEBUG:boto:Params: {} 2022-09-12T19:02:28.955 DEBUG:boto:Token: None 2022-09-12T19:02:28.956 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/ 2022-09-12T19:02:28.956 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XAmyMxVrDT5PQj181TJgF4HtR3s= 2022-09-12T19:02:28.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XAmyMxVrDT5PQj181TJgF4HtR3s=', 'Content-Length': '0'} 2022-09-12T19:02:28.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000153d2dc8721d2ce4-00631f8244-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.960 DEBUG:boto:b'jdpiak-18obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:28.961 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:28.962 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:28.962 DEBUG:boto:Method: HEAD 2022-09-12T19:02:28.963 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1 2022-09-12T19:02:28.963 DEBUG:boto:Data: 2022-09-12T19:02:28.964 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.964 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:28.965 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.965 DEBUG:boto:Params: {} 2022-09-12T19:02:28.966 DEBUG:boto:Token: None 2022-09-12T19:02:28.966 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/obj-test-zone1 2022-09-12T19:02:28.967 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SXEBgUZbp1g9L/oAYeZGFBOWfGo= 2022-09-12T19:02:28.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SXEBgUZbp1g9L/oAYeZGFBOWfGo=', 'Content-Length': '0'} 2022-09-12T19:02:28.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046307742970acf50-00631f8244-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.971 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:28.972 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:28.972 DEBUG:boto:Method: HEAD 2022-09-12T19:02:28.973 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1 2022-09-12T19:02:28.973 DEBUG:boto:Data: 2022-09-12T19:02:28.974 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.974 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:28.975 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.976 DEBUG:boto:Params: {} 2022-09-12T19:02:28.976 DEBUG:boto:Token: None 2022-09-12T19:02:28.977 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/obj-test-zone1 2022-09-12T19:02:28.978 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SXEBgUZbp1g9L/oAYeZGFBOWfGo= 2022-09-12T19:02:28.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SXEBgUZbp1g9L/oAYeZGFBOWfGo=', 'Content-Length': '0'} 2022-09-12T19:02:28.981 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d0991de5a6927b4b-00631f8244-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:28.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:02:28.982 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:28.983 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:28.983 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.984 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.984 DEBUG:boto:Method: GET 2022-09-12T19:02:28.985 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.985 DEBUG:boto:Data: 2022-09-12T19:02:28.986 DEBUG:boto:Headers: {} 2022-09-12T19:02:28.986 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:28.987 DEBUG:boto:Port: 8000 2022-09-12T19:02:28.988 DEBUG:boto:Params: {} 2022-09-12T19:02:28.988 DEBUG:boto:Token: None 2022-09-12T19:02:28.989 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:28 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:28.989 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+R8drOs2yUmxlsuTU6BJZFEVkuM= 2022-09-12T19:02:28.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+R8drOs2yUmxlsuTU6BJZFEVkuM=', 'Content-Length': '0'} 2022-09-12T19:02:28.992 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093fc0684be3254c8-00631f8244-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.034 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:29.035 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:29.036 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.037 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.038 DEBUG:boto:Method: GET 2022-09-12T19:02:29.038 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.039 DEBUG:boto:Data: 2022-09-12T19:02:29.039 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.040 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.041 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.041 DEBUG:boto:Params: {} 2022-09-12T19:02:29.042 DEBUG:boto:Token: None 2022-09-12T19:02:29.043 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.043 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sOR1P/IGEW94ovSyOXkI19xRrmg= 2022-09-12T19:02:29.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sOR1P/IGEW94ovSyOXkI19xRrmg=', 'Content-Length': '0'} 2022-09-12T19:02:29.046 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be80117a19f98d81-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.047 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:29.048 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:29.048 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:29.049 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:29.049 DEBUG:boto:Method: GET 2022-09-12T19:02:29.050 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:29.050 DEBUG:boto:Data: 2022-09-12T19:02:29.051 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.051 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:29.052 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.052 DEBUG:boto:Params: {} 2022-09-12T19:02:29.053 DEBUG:boto:Token: None 2022-09-12T19:02:29.054 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/ 2022-09-12T19:02:29.054 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA= 2022-09-12T19:02:29.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA=', 'Content-Length': '0'} 2022-09-12T19:02:29.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b58e932bf088d623-00631f8245-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.058 DEBUG:boto:b'jdpiak-18obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:29.059 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:29.060 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:29.060 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:29.061 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:29.061 DEBUG:boto:Method: GET 2022-09-12T19:02:29.062 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:29.062 DEBUG:boto:Data: 2022-09-12T19:02:29.063 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.063 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.064 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.064 DEBUG:boto:Params: {} 2022-09-12T19:02:29.065 DEBUG:boto:Token: None 2022-09-12T19:02:29.065 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/ 2022-09-12T19:02:29.066 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA= 2022-09-12T19:02:29.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA=', 'Content-Length': '0'} 2022-09-12T19:02:29.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000457fa8edea59ecae-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.114 DEBUG:boto:b'jdpiak-18obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:29.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:02:29.116 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:29.117 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:29.117 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.118 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.118 DEBUG:boto:Method: GET 2022-09-12T19:02:29.119 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.120 DEBUG:boto:Data: 2022-09-12T19:02:29.120 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.121 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:29.121 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.122 DEBUG:boto:Params: {} 2022-09-12T19:02:29.122 DEBUG:boto:Token: None 2022-09-12T19:02:29.123 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.124 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rrHwVhXht+e9ENnH1iUzI3AydgE= 2022-09-12T19:02:29.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rrHwVhXht+e9ENnH1iUzI3AydgE=', 'Content-Length': '0'} 2022-09-12T19:02:29.126 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e548d8a68df19e2-00631f8245-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.128 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:29.128 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:29.129 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.129 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.130 DEBUG:boto:Method: GET 2022-09-12T19:02:29.130 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.131 DEBUG:boto:Data: 2022-09-12T19:02:29.131 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.132 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.133 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.133 DEBUG:boto:Params: {} 2022-09-12T19:02:29.134 DEBUG:boto:Token: None 2022-09-12T19:02:29.134 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.135 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rrHwVhXht+e9ENnH1iUzI3AydgE= 2022-09-12T19:02:29.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rrHwVhXht+e9ENnH1iUzI3AydgE=', 'Content-Length': '0'} 2022-09-12T19:02:29.138 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a99d42fc80bf0c0-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.139 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:29.139 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:29.140 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.140 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.141 DEBUG:boto:Method: HEAD 2022-09-12T19:02:29.141 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.142 DEBUG:boto:Data: 2022-09-12T19:02:29.142 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.143 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:29.143 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.144 DEBUG:boto:Params: {} 2022-09-12T19:02:29.144 DEBUG:boto:Token: None 2022-09-12T19:02:29.145 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.146 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:no4G0K9E5xMg4LSS4brSeyoxZB4= 2022-09-12T19:02:29.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:no4G0K9E5xMg4LSS4brSeyoxZB4=', 'Content-Length': '0'} 2022-09-12T19:02:29.148 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e81d5fdea090520a-00631f8245-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.149 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:29.150 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:29.150 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.151 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.152 DEBUG:boto:Method: HEAD 2022-09-12T19:02:29.153 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.154 DEBUG:boto:Data: 2022-09-12T19:02:29.154 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.155 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.155 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.156 DEBUG:boto:Params: {} 2022-09-12T19:02:29.156 DEBUG:boto:Token: None 2022-09-12T19:02:29.157 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.157 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:no4G0K9E5xMg4LSS4brSeyoxZB4= 2022-09-12T19:02:29.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:no4G0K9E5xMg4LSS4brSeyoxZB4=', 'Content-Length': '0'} 2022-09-12T19:02:29.160 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd94528bd320e3d4-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:02:29.161 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:29.162 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:29.162 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.163 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.163 DEBUG:boto:Method: GET 2022-09-12T19:02:29.164 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.164 DEBUG:boto:Data: 2022-09-12T19:02:29.165 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.165 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:29.166 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.167 DEBUG:boto:Params: {} 2022-09-12T19:02:29.167 DEBUG:boto:Token: None 2022-09-12T19:02:29.168 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.168 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rrHwVhXht+e9ENnH1iUzI3AydgE= 2022-09-12T19:02:29.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rrHwVhXht+e9ENnH1iUzI3AydgE=', 'Content-Length': '0'} 2022-09-12T19:02:29.171 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057bbd126f852cadc-00631f8245-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.173 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:29.174 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:29.174 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.175 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.176 DEBUG:boto:Method: GET 2022-09-12T19:02:29.177 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.177 DEBUG:boto:Data: 2022-09-12T19:02:29.178 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.179 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.179 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.180 DEBUG:boto:Params: {} 2022-09-12T19:02:29.180 DEBUG:boto:Token: None 2022-09-12T19:02:29.181 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.182 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rrHwVhXht+e9ENnH1iUzI3AydgE= 2022-09-12T19:02:29.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rrHwVhXht+e9ENnH1iUzI3AydgE=', 'Content-Length': '0'} 2022-09-12T19:02:29.184 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cbf1071bca528220-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.185 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:29.186 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:29.186 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:29.187 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:29.187 DEBUG:boto:Method: GET 2022-09-12T19:02:29.188 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:29.188 DEBUG:boto:Data: 2022-09-12T19:02:29.189 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.189 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:29.190 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.190 DEBUG:boto:Params: {} 2022-09-12T19:02:29.191 DEBUG:boto:Token: None 2022-09-12T19:02:29.192 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/ 2022-09-12T19:02:29.192 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA= 2022-09-12T19:02:29.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA=', 'Content-Length': '0'} 2022-09-12T19:02:29.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000675e62ca2dd79ea9-00631f8245-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.196 DEBUG:boto:b'jdpiak-18obj-test-zone21000falseobj-test-zone22022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:29.197 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:29.197 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:29.198 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:29.199 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:29.199 DEBUG:boto:Method: GET 2022-09-12T19:02:29.200 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:29.201 DEBUG:boto:Data: 2022-09-12T19:02:29.202 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.203 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.204 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.204 DEBUG:boto:Params: {} 2022-09-12T19:02:29.205 DEBUG:boto:Token: None 2022-09-12T19:02:29.206 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/ 2022-09-12T19:02:29.207 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA= 2022-09-12T19:02:29.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA=', 'Content-Length': '0'} 2022-09-12T19:02:29.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f17bdb1a2f05397-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.212 DEBUG:boto:b'jdpiak-18obj-test-zone21000falseobj-test-zone22022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:29.213 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:29.213 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:29.214 DEBUG:boto:Method: HEAD 2022-09-12T19:02:29.214 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2 2022-09-12T19:02:29.215 DEBUG:boto:Data: 2022-09-12T19:02:29.215 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.216 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:29.216 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.217 DEBUG:boto:Params: {} 2022-09-12T19:02:29.217 DEBUG:boto:Token: None 2022-09-12T19:02:29.218 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone2 2022-09-12T19:02:29.219 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4miT9AKqR39G8QW+KWmmuCVrM6U= 2022-09-12T19:02:29.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4miT9AKqR39G8QW+KWmmuCVrM6U=', 'Content-Length': '0'} 2022-09-12T19:02:29.221 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3f08e2dbc4dbb1f-00631f8245-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.222 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:29.223 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:29.224 DEBUG:boto:Method: HEAD 2022-09-12T19:02:29.225 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2 2022-09-12T19:02:29.225 DEBUG:boto:Data: 2022-09-12T19:02:29.226 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.226 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.227 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.227 DEBUG:boto:Params: {} 2022-09-12T19:02:29.228 DEBUG:boto:Token: None 2022-09-12T19:02:29.228 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone2 2022-09-12T19:02:29.229 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4miT9AKqR39G8QW+KWmmuCVrM6U= 2022-09-12T19:02:29.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4miT9AKqR39G8QW+KWmmuCVrM6U=', 'Content-Length': '0'} 2022-09-12T19:02:29.232 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006493e65cc1a41ef-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:02:29.233 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:29.234 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:29.234 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.235 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.235 DEBUG:boto:Method: GET 2022-09-12T19:02:29.236 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.236 DEBUG:boto:Data: 2022-09-12T19:02:29.237 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.238 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:29.238 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.239 DEBUG:boto:Params: {} 2022-09-12T19:02:29.239 DEBUG:boto:Token: None 2022-09-12T19:02:29.240 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.240 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rrHwVhXht+e9ENnH1iUzI3AydgE= 2022-09-12T19:02:29.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rrHwVhXht+e9ENnH1iUzI3AydgE=', 'Content-Length': '0'} 2022-09-12T19:02:29.243 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000548717fb1053bcda-00631f8245-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.244 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:29.244 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:29.245 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.245 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.246 DEBUG:boto:Method: GET 2022-09-12T19:02:29.246 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.247 DEBUG:boto:Data: 2022-09-12T19:02:29.247 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.248 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.248 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.249 DEBUG:boto:Params: {} 2022-09-12T19:02:29.250 DEBUG:boto:Token: None 2022-09-12T19:02:29.250 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:29.251 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rrHwVhXht+e9ENnH1iUzI3AydgE= 2022-09-12T19:02:29.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rrHwVhXht+e9ENnH1iUzI3AydgE=', 'Content-Length': '0'} 2022-09-12T19:02:29.254 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dfe69fb7bc68c957-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.296 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:29.297 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:29.298 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:29.299 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:29.299 DEBUG:boto:Method: GET 2022-09-12T19:02:29.300 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:29.300 DEBUG:boto:Data: 2022-09-12T19:02:29.301 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.301 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:29.302 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.302 DEBUG:boto:Params: {} 2022-09-12T19:02:29.303 DEBUG:boto:Token: None 2022-09-12T19:02:29.304 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/ 2022-09-12T19:02:29.304 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA= 2022-09-12T19:02:29.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA=', 'Content-Length': '0'} 2022-09-12T19:02:29.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c132983ebc9f132-00631f8245-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.309 DEBUG:boto:b'jdpiak-18obj-test-zone21000falseobj-test-zone22022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:29.310 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:29.311 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:29.312 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:29.313 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:29.314 DEBUG:boto:Method: GET 2022-09-12T19:02:29.317 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:29.317 DEBUG:boto:Data: 2022-09-12T19:02:29.318 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.318 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.319 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.319 DEBUG:boto:Params: {} 2022-09-12T19:02:29.320 DEBUG:boto:Token: None 2022-09-12T19:02:29.321 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/ 2022-09-12T19:02:29.321 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA= 2022-09-12T19:02:29.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA=', 'Content-Length': '0'} 2022-09-12T19:02:29.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061a6539856dc9e57-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.326 DEBUG:boto:b'jdpiak-18obj-test-zone21000falseobj-test-zone22022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:29.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:02:29.327 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:29.328 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:29.329 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.329 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.330 DEBUG:boto:Method: GET 2022-09-12T19:02:29.331 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.331 DEBUG:boto:Data: 2022-09-12T19:02:29.332 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.332 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:29.333 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.333 DEBUG:boto:Params: {} 2022-09-12T19:02:29.334 DEBUG:boto:Token: None 2022-09-12T19:02:29.334 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.335 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ufhmX3GGC9EaT6A5+yj+sOelAkY= 2022-09-12T19:02:29.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ufhmX3GGC9EaT6A5+yj+sOelAkY=', 'Content-Length': '0'} 2022-09-12T19:02:29.338 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e5906417bef8b2e6-00631f8245-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.339 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:29.340 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:29.340 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.341 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.341 DEBUG:boto:Method: GET 2022-09-12T19:02:29.342 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.343 DEBUG:boto:Data: 2022-09-12T19:02:29.343 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.344 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.344 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.345 DEBUG:boto:Params: {} 2022-09-12T19:02:29.345 DEBUG:boto:Token: None 2022-09-12T19:02:29.346 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.347 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ufhmX3GGC9EaT6A5+yj+sOelAkY= 2022-09-12T19:02:29.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ufhmX3GGC9EaT6A5+yj+sOelAkY=', 'Content-Length': '0'} 2022-09-12T19:02:29.350 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a4c54ba296c4a09-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.351 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:29.351 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:29.352 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.352 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.353 DEBUG:boto:Method: HEAD 2022-09-12T19:02:29.353 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.354 DEBUG:boto:Data: 2022-09-12T19:02:29.354 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.355 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:29.355 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.356 DEBUG:boto:Params: {} 2022-09-12T19:02:29.356 DEBUG:boto:Token: None 2022-09-12T19:02:29.357 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.358 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dE6j9bgmAD5eCZfz3hZmco5qEn8= 2022-09-12T19:02:29.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dE6j9bgmAD5eCZfz3hZmco5qEn8=', 'Content-Length': '0'} 2022-09-12T19:02:29.360 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d824cc514f343241-00631f8245-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.361 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:29.362 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:29.362 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.363 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.363 DEBUG:boto:Method: HEAD 2022-09-12T19:02:29.364 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.364 DEBUG:boto:Data: 2022-09-12T19:02:29.365 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.365 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.366 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.366 DEBUG:boto:Params: {} 2022-09-12T19:02:29.367 DEBUG:boto:Token: None 2022-09-12T19:02:29.368 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.368 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dE6j9bgmAD5eCZfz3hZmco5qEn8= 2022-09-12T19:02:29.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dE6j9bgmAD5eCZfz3hZmco5qEn8=', 'Content-Length': '0'} 2022-09-12T19:02:29.371 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2be263118857568-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:02:29.373 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:29.374 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:29.374 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.375 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.375 DEBUG:boto:Method: GET 2022-09-12T19:02:29.376 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.376 DEBUG:boto:Data: 2022-09-12T19:02:29.377 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.377 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:29.378 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.378 DEBUG:boto:Params: {} 2022-09-12T19:02:29.379 DEBUG:boto:Token: None 2022-09-12T19:02:29.380 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.381 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ufhmX3GGC9EaT6A5+yj+sOelAkY= 2022-09-12T19:02:29.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ufhmX3GGC9EaT6A5+yj+sOelAkY=', 'Content-Length': '0'} 2022-09-12T19:02:29.384 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076137ac599764ef2-00631f8245-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.426 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:29.427 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:29.427 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.428 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.428 DEBUG:boto:Method: GET 2022-09-12T19:02:29.429 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.429 DEBUG:boto:Data: 2022-09-12T19:02:29.430 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.431 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.431 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.432 DEBUG:boto:Params: {} 2022-09-12T19:02:29.432 DEBUG:boto:Token: None 2022-09-12T19:02:29.433 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.433 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ufhmX3GGC9EaT6A5+yj+sOelAkY= 2022-09-12T19:02:29.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ufhmX3GGC9EaT6A5+yj+sOelAkY=', 'Content-Length': '0'} 2022-09-12T19:02:29.436 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f45d21ddcb4430f6-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.437 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:29.438 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:29.438 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:29.439 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:29.440 DEBUG:boto:Method: GET 2022-09-12T19:02:29.440 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:29.441 DEBUG:boto:Data: 2022-09-12T19:02:29.441 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.442 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:29.442 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.443 DEBUG:boto:Params: {} 2022-09-12T19:02:29.443 DEBUG:boto:Token: None 2022-09-12T19:02:29.444 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/ 2022-09-12T19:02:29.445 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA= 2022-09-12T19:02:29.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA=', 'Content-Length': '0'} 2022-09-12T19:02:29.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013e9b01256a6b2ec-00631f8245-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.450 DEBUG:boto:b'jdpiak-18obj-test-zone31000falseobj-test-zone32022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:29.451 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:29.452 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:29.453 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:29.454 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:29.455 DEBUG:boto:Method: GET 2022-09-12T19:02:29.456 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:29.457 DEBUG:boto:Data: 2022-09-12T19:02:29.458 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.459 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.460 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.461 DEBUG:boto:Params: {} 2022-09-12T19:02:29.462 DEBUG:boto:Token: None 2022-09-12T19:02:29.463 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/ 2022-09-12T19:02:29.464 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA= 2022-09-12T19:02:29.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA=', 'Content-Length': '0'} 2022-09-12T19:02:29.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ea210cee9737bf7-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.469 DEBUG:boto:b'jdpiak-18obj-test-zone31000falseobj-test-zone32022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:29.471 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:29.472 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:29.472 DEBUG:boto:Method: HEAD 2022-09-12T19:02:29.473 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3 2022-09-12T19:02:29.473 DEBUG:boto:Data: 2022-09-12T19:02:29.474 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.474 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:29.475 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.475 DEBUG:boto:Params: {} 2022-09-12T19:02:29.476 DEBUG:boto:Token: None 2022-09-12T19:02:29.477 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone3 2022-09-12T19:02:29.477 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RG1ozpW/CxT9XkqEH/KoAentkRg= 2022-09-12T19:02:29.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RG1ozpW/CxT9XkqEH/KoAentkRg=', 'Content-Length': '0'} 2022-09-12T19:02:29.480 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000298c6facdfbe7cec-00631f8245-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.481 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:29.481 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:29.482 DEBUG:boto:Method: HEAD 2022-09-12T19:02:29.482 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3 2022-09-12T19:02:29.483 DEBUG:boto:Data: 2022-09-12T19:02:29.483 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.484 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.484 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.485 DEBUG:boto:Params: {} 2022-09-12T19:02:29.485 DEBUG:boto:Token: None 2022-09-12T19:02:29.486 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone3 2022-09-12T19:02:29.487 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RG1ozpW/CxT9XkqEH/KoAentkRg= 2022-09-12T19:02:29.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RG1ozpW/CxT9XkqEH/KoAentkRg=', 'Content-Length': '0'} 2022-09-12T19:02:29.489 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078959880e849dc38-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:02:29.491 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:29.491 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:29.492 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.492 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.493 DEBUG:boto:Method: GET 2022-09-12T19:02:29.493 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.494 DEBUG:boto:Data: 2022-09-12T19:02:29.494 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.495 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:29.495 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.496 DEBUG:boto:Params: {} 2022-09-12T19:02:29.497 DEBUG:boto:Token: None 2022-09-12T19:02:29.497 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.498 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ufhmX3GGC9EaT6A5+yj+sOelAkY= 2022-09-12T19:02:29.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ufhmX3GGC9EaT6A5+yj+sOelAkY=', 'Content-Length': '0'} 2022-09-12T19:02:29.500 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093f1e6958ced8b02-00631f8245-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.502 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:29.503 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:29.504 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.505 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.506 DEBUG:boto:Method: GET 2022-09-12T19:02:29.507 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.508 DEBUG:boto:Data: 2022-09-12T19:02:29.508 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.509 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.510 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.511 DEBUG:boto:Params: {} 2022-09-12T19:02:29.512 DEBUG:boto:Token: None 2022-09-12T19:02:29.513 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:29.514 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ufhmX3GGC9EaT6A5+yj+sOelAkY= 2022-09-12T19:02:29.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ufhmX3GGC9EaT6A5+yj+sOelAkY=', 'Content-Length': '0'} 2022-09-12T19:02:29.516 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085e9c23fd3faf9c1-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.517 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:29.518 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:29.518 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:29.519 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:29.520 DEBUG:boto:Method: GET 2022-09-12T19:02:29.520 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:29.521 DEBUG:boto:Data: 2022-09-12T19:02:29.521 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.522 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:29.522 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.523 DEBUG:boto:Params: {} 2022-09-12T19:02:29.523 DEBUG:boto:Token: None 2022-09-12T19:02:29.524 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/ 2022-09-12T19:02:29.525 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA= 2022-09-12T19:02:29.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA=', 'Content-Length': '0'} 2022-09-12T19:02:29.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e2d127e29802fcc-00631f8245-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.529 DEBUG:boto:b'jdpiak-18obj-test-zone31000falseobj-test-zone32022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:29.529 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:29.530 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:29.531 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:29.531 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:29.532 DEBUG:boto:Method: GET 2022-09-12T19:02:29.532 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:29.533 DEBUG:boto:Data: 2022-09-12T19:02:29.533 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.534 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.534 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.535 DEBUG:boto:Params: {} 2022-09-12T19:02:29.535 DEBUG:boto:Token: None 2022-09-12T19:02:29.536 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/ 2022-09-12T19:02:29.537 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA= 2022-09-12T19:02:29.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA=', 'Content-Length': '0'} 2022-09-12T19:02:29.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010bbbdb5685909c7-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.541 DEBUG:boto:b'jdpiak-18obj-test-zone31000falseobj-test-zone32022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:29.542 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-18 zones={test-zone2, test-zone1} 2022-09-12T19:02:29.543 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-18 zones={test-zone3, test-zone1} 2022-09-12T19:02:29.543 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:29.544 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:29.544 DEBUG:boto:Method: HEAD 2022-09-12T19:02:29.545 DEBUG:boto:Path: /jdpiak-18/ 2022-09-12T19:02:29.545 DEBUG:boto:Data: 2022-09-12T19:02:29.546 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.546 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:29.547 DEBUG:boto:Port: 8001 2022-09-12T19:02:29.547 DEBUG:boto:Params: {} 2022-09-12T19:02:29.548 DEBUG:boto:Token: None 2022-09-12T19:02:29.549 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/ 2022-09-12T19:02:29.549 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kI6ZziLcyM0wYhSWH635rvPqQek= 2022-09-12T19:02:29.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kI6ZziLcyM0wYhSWH635rvPqQek=', 'Content-Length': '0'} 2022-09-12T19:02:29.553 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aaef8e0cfef27348-00631f8245-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.553 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:29.554 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:29.554 DEBUG:boto:Method: HEAD 2022-09-12T19:02:29.555 DEBUG:boto:Path: /jdpiak-18/ 2022-09-12T19:02:29.555 DEBUG:boto:Data: 2022-09-12T19:02:29.556 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.556 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.557 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.557 DEBUG:boto:Params: {} 2022-09-12T19:02:29.558 DEBUG:boto:Token: None 2022-09-12T19:02:29.559 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/ 2022-09-12T19:02:29.559 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kI6ZziLcyM0wYhSWH635rvPqQek= 2022-09-12T19:02:29.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kI6ZziLcyM0wYhSWH635rvPqQek=', 'Content-Length': '0'} 2022-09-12T19:02:29.563 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072b43c8ae3165f14-00631f8245-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.563 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:02:29.564 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:29.564 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:29.565 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:29.565 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:29.566 DEBUG:boto:Method: GET 2022-09-12T19:02:29.566 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:29.567 DEBUG:boto:Data: 2022-09-12T19:02:29.567 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.568 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:29.568 DEBUG:boto:Port: 8001 2022-09-12T19:02:29.569 DEBUG:boto:Params: {} 2022-09-12T19:02:29.570 DEBUG:boto:Token: None 2022-09-12T19:02:29.570 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/?versions 2022-09-12T19:02:29.571 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N1XMTJyw5ZtZsXCk/AapmKcVuNY= 2022-09-12T19:02:29.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N1XMTJyw5ZtZsXCk/AapmKcVuNY=', 'Content-Length': '0'} 2022-09-12T19:02:29.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ba3dc83b7376349-00631f8245-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.575 DEBUG:boto:b'jdpiak-18999falsedummyqUufSwmbKETbT7yNOH8sL0A4tddUbNOtrue2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ToUtj2XoS.tz3goIbvjFwiigMyzGOpLtrue2022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2DPksivOWVYhJDrEwQqHYV7wK1Ul1JkWtrue2022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Ztrue2022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:29.576 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:02:29.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:02:29.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:02:29.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:02:29.578 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:02:29.579 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:29.579 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:29.580 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:29.580 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:29.581 DEBUG:boto:Method: GET 2022-09-12T19:02:29.581 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:29.582 DEBUG:boto:Data: 2022-09-12T19:02:29.582 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.583 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.583 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.584 DEBUG:boto:Params: {} 2022-09-12T19:02:29.584 DEBUG:boto:Token: None 2022-09-12T19:02:29.585 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/?versions 2022-09-12T19:02:29.586 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N1XMTJyw5ZtZsXCk/AapmKcVuNY= 2022-09-12T19:02:29.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N1XMTJyw5ZtZsXCk/AapmKcVuNY=', 'Content-Length': '0'} 2022-09-12T19:02:29.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075db9dfa73549b0b-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.590 DEBUG:boto:b'jdpiak-18999falsedummyqUufSwmbKETbT7yNOH8sL0A4tddUbNOtrue2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ToUtj2XoS.tz3goIbvjFwiigMyzGOpLtrue2022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2DPksivOWVYhJDrEwQqHYV7wK1Ul1JkWtrue2022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Ztrue2022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:29.591 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:02:29.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:02:29.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:02:29.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:02:29.593 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:29.594 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:29.594 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:29.595 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:29.595 DEBUG:boto:Method: GET 2022-09-12T19:02:29.596 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:29.596 DEBUG:boto:Data: 2022-09-12T19:02:29.597 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.597 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:29.598 DEBUG:boto:Port: 8001 2022-09-12T19:02:29.598 DEBUG:boto:Params: {} 2022-09-12T19:02:29.599 DEBUG:boto:Token: None 2022-09-12T19:02:29.599 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/?versions 2022-09-12T19:02:29.600 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N1XMTJyw5ZtZsXCk/AapmKcVuNY= 2022-09-12T19:02:29.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N1XMTJyw5ZtZsXCk/AapmKcVuNY=', 'Content-Length': '0'} 2022-09-12T19:02:29.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1fdc8c9d11f32bf-00631f8245-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.605 DEBUG:boto:b'jdpiak-18999falsedummyqUufSwmbKETbT7yNOH8sL0A4tddUbNOtrue2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ToUtj2XoS.tz3goIbvjFwiigMyzGOpLtrue2022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2DPksivOWVYhJDrEwQqHYV7wK1Ul1JkWtrue2022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Ztrue2022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:29.605 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:29.606 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:29.607 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:29.607 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:29.608 DEBUG:boto:Method: GET 2022-09-12T19:02:29.608 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:29.609 DEBUG:boto:Data: 2022-09-12T19:02:29.609 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.610 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.610 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.611 DEBUG:boto:Params: {} 2022-09-12T19:02:29.612 DEBUG:boto:Token: None 2022-09-12T19:02:29.612 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/?versions 2022-09-12T19:02:29.613 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N1XMTJyw5ZtZsXCk/AapmKcVuNY= 2022-09-12T19:02:29.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N1XMTJyw5ZtZsXCk/AapmKcVuNY=', 'Content-Length': '0'} 2022-09-12T19:02:29.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d9461c3baed57b6-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.617 DEBUG:boto:b'jdpiak-18999falsedummyqUufSwmbKETbT7yNOH8sL0A4tddUbNOtrue2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ToUtj2XoS.tz3goIbvjFwiigMyzGOpLtrue2022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2DPksivOWVYhJDrEwQqHYV7wK1Ul1JkWtrue2022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Ztrue2022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:29.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:02:29.619 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:29.619 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:29.620 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.621 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.621 DEBUG:boto:Method: GET 2022-09-12T19:02:29.622 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.622 DEBUG:boto:Data: 2022-09-12T19:02:29.623 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.624 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:29.624 DEBUG:boto:Port: 8001 2022-09-12T19:02:29.625 DEBUG:boto:Params: {} 2022-09-12T19:02:29.625 DEBUG:boto:Token: None 2022-09-12T19:02:29.626 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.626 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HGv0JbtxFpe2Bn1DfnfkB4VyuZo= 2022-09-12T19:02:29.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HGv0JbtxFpe2Bn1DfnfkB4VyuZo=', 'Content-Length': '0'} 2022-09-12T19:02:29.629 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006c58a4d3adf99703-00631f8245-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.630 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:29.631 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:29.632 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.633 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.633 DEBUG:boto:Method: GET 2022-09-12T19:02:29.634 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.634 DEBUG:boto:Data: 2022-09-12T19:02:29.635 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.636 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.636 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.637 DEBUG:boto:Params: {} 2022-09-12T19:02:29.637 DEBUG:boto:Token: None 2022-09-12T19:02:29.638 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.638 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HGv0JbtxFpe2Bn1DfnfkB4VyuZo= 2022-09-12T19:02:29.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HGv0JbtxFpe2Bn1DfnfkB4VyuZo=', 'Content-Length': '0'} 2022-09-12T19:02:29.641 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000001a6576998d08456-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.642 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:29.643 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:29.643 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.644 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.644 DEBUG:boto:Method: HEAD 2022-09-12T19:02:29.645 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.645 DEBUG:boto:Data: 2022-09-12T19:02:29.646 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.646 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:29.647 DEBUG:boto:Port: 8001 2022-09-12T19:02:29.647 DEBUG:boto:Params: {} 2022-09-12T19:02:29.648 DEBUG:boto:Token: None 2022-09-12T19:02:29.648 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.649 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9JpQn5jSIAUzP/xOaT3kRX3WaW0= 2022-09-12T19:02:29.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9JpQn5jSIAUzP/xOaT3kRX3WaW0=', 'Content-Length': '0'} 2022-09-12T19:02:29.652 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000658e586ab265beb2-00631f8245-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.653 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:29.653 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:29.654 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.654 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.655 DEBUG:boto:Method: HEAD 2022-09-12T19:02:29.655 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.656 DEBUG:boto:Data: 2022-09-12T19:02:29.656 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.657 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.657 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.658 DEBUG:boto:Params: {} 2022-09-12T19:02:29.658 DEBUG:boto:Token: None 2022-09-12T19:02:29.659 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.659 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9JpQn5jSIAUzP/xOaT3kRX3WaW0= 2022-09-12T19:02:29.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9JpQn5jSIAUzP/xOaT3kRX3WaW0=', 'Content-Length': '0'} 2022-09-12T19:02:29.662 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e207ef0be5a1b5ce-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:02:29.663 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:29.664 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:29.665 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.665 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.666 DEBUG:boto:Method: GET 2022-09-12T19:02:29.666 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.667 DEBUG:boto:Data: 2022-09-12T19:02:29.667 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.668 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:29.668 DEBUG:boto:Port: 8001 2022-09-12T19:02:29.669 DEBUG:boto:Params: {} 2022-09-12T19:02:29.669 DEBUG:boto:Token: None 2022-09-12T19:02:29.670 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.671 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HGv0JbtxFpe2Bn1DfnfkB4VyuZo= 2022-09-12T19:02:29.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HGv0JbtxFpe2Bn1DfnfkB4VyuZo=', 'Content-Length': '0'} 2022-09-12T19:02:29.673 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000244d4c408c5db4ad-00631f8245-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.674 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:29.675 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:29.675 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.676 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.677 DEBUG:boto:Method: GET 2022-09-12T19:02:29.678 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.679 DEBUG:boto:Data: 2022-09-12T19:02:29.679 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.680 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.680 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.681 DEBUG:boto:Params: {} 2022-09-12T19:02:29.681 DEBUG:boto:Token: None 2022-09-12T19:02:29.682 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.683 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HGv0JbtxFpe2Bn1DfnfkB4VyuZo= 2022-09-12T19:02:29.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HGv0JbtxFpe2Bn1DfnfkB4VyuZo=', 'Content-Length': '0'} 2022-09-12T19:02:29.685 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000052496725d9019dfc-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.686 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:29.687 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:29.687 DEBUG:boto:path=/jdpiak-18/?prefix=dummy 2022-09-12T19:02:29.688 DEBUG:boto:auth_path=/jdpiak-18/?prefix=dummy 2022-09-12T19:02:29.689 DEBUG:boto:Method: GET 2022-09-12T19:02:29.690 DEBUG:boto:Path: /jdpiak-18/?prefix=dummy 2022-09-12T19:02:29.691 DEBUG:boto:Data: 2022-09-12T19:02:29.692 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.693 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:29.694 DEBUG:boto:Port: 8001 2022-09-12T19:02:29.695 DEBUG:boto:Params: {} 2022-09-12T19:02:29.696 DEBUG:boto:Token: None 2022-09-12T19:02:29.697 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/ 2022-09-12T19:02:29.698 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA= 2022-09-12T19:02:29.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA=', 'Content-Length': '0'} 2022-09-12T19:02:29.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3a74c362a5f02c9-00631f8245-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.746 DEBUG:boto:b'jdpiak-18dummy1000falsedummy2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:29.748 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:29.748 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:29.749 DEBUG:boto:path=/jdpiak-18/?prefix=dummy 2022-09-12T19:02:29.749 DEBUG:boto:auth_path=/jdpiak-18/?prefix=dummy 2022-09-12T19:02:29.750 DEBUG:boto:Method: GET 2022-09-12T19:02:29.750 DEBUG:boto:Path: /jdpiak-18/?prefix=dummy 2022-09-12T19:02:29.751 DEBUG:boto:Data: 2022-09-12T19:02:29.751 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.752 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.753 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.754 DEBUG:boto:Params: {} 2022-09-12T19:02:29.755 DEBUG:boto:Token: None 2022-09-12T19:02:29.756 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/ 2022-09-12T19:02:29.757 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA= 2022-09-12T19:02:29.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA=', 'Content-Length': '0'} 2022-09-12T19:02:29.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047e8951d4d8a41bc-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.762 DEBUG:boto:b'jdpiak-18dummy1000falsedummy2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:29.763 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:29.763 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:29.764 DEBUG:boto:Method: HEAD 2022-09-12T19:02:29.765 DEBUG:boto:Path: /jdpiak-18/dummy 2022-09-12T19:02:29.765 DEBUG:boto:Data: 2022-09-12T19:02:29.766 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.766 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:29.767 DEBUG:boto:Port: 8001 2022-09-12T19:02:29.767 DEBUG:boto:Params: {} 2022-09-12T19:02:29.768 DEBUG:boto:Token: None 2022-09-12T19:02:29.768 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/dummy 2022-09-12T19:02:29.769 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:st49YCcdQTRsj2iLcSaNKneDULY= 2022-09-12T19:02:29.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:st49YCcdQTRsj2iLcSaNKneDULY=', 'Content-Length': '0'} 2022-09-12T19:02:29.772 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004b86b2024207cea4-00631f8245-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.773 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:29.774 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:29.774 DEBUG:boto:Method: HEAD 2022-09-12T19:02:29.775 DEBUG:boto:Path: /jdpiak-18/dummy 2022-09-12T19:02:29.775 DEBUG:boto:Data: 2022-09-12T19:02:29.776 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.776 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.777 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.777 DEBUG:boto:Params: {} 2022-09-12T19:02:29.778 DEBUG:boto:Token: None 2022-09-12T19:02:29.778 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/dummy 2022-09-12T19:02:29.779 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:st49YCcdQTRsj2iLcSaNKneDULY= 2022-09-12T19:02:29.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:st49YCcdQTRsj2iLcSaNKneDULY=', 'Content-Length': '0'} 2022-09-12T19:02:29.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009c2e819c71c6f87b-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:02:29.784 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:29.785 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:29.785 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.786 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.786 DEBUG:boto:Method: GET 2022-09-12T19:02:29.787 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.787 DEBUG:boto:Data: 2022-09-12T19:02:29.788 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.789 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:29.789 DEBUG:boto:Port: 8001 2022-09-12T19:02:29.790 DEBUG:boto:Params: {} 2022-09-12T19:02:29.798 DEBUG:boto:Token: None 2022-09-12T19:02:29.799 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.800 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HGv0JbtxFpe2Bn1DfnfkB4VyuZo= 2022-09-12T19:02:29.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HGv0JbtxFpe2Bn1DfnfkB4VyuZo=', 'Content-Length': '0'} 2022-09-12T19:02:29.803 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000500867fdefa6621c-00631f8245-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.804 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:29.804 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:29.805 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.805 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.806 DEBUG:boto:Method: GET 2022-09-12T19:02:29.806 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.807 DEBUG:boto:Data: 2022-09-12T19:02:29.807 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.808 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.808 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.809 DEBUG:boto:Params: {} 2022-09-12T19:02:29.810 DEBUG:boto:Token: None 2022-09-12T19:02:29.811 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:29.811 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HGv0JbtxFpe2Bn1DfnfkB4VyuZo= 2022-09-12T19:02:29.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HGv0JbtxFpe2Bn1DfnfkB4VyuZo=', 'Content-Length': '0'} 2022-09-12T19:02:29.814 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000e9533a5908eeef2-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.815 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:29.816 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:29.817 DEBUG:boto:path=/jdpiak-18/?prefix=dummy 2022-09-12T19:02:29.818 DEBUG:boto:auth_path=/jdpiak-18/?prefix=dummy 2022-09-12T19:02:29.818 DEBUG:boto:Method: GET 2022-09-12T19:02:29.819 DEBUG:boto:Path: /jdpiak-18/?prefix=dummy 2022-09-12T19:02:29.820 DEBUG:boto:Data: 2022-09-12T19:02:29.821 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.822 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:29.823 DEBUG:boto:Port: 8001 2022-09-12T19:02:29.823 DEBUG:boto:Params: {} 2022-09-12T19:02:29.824 DEBUG:boto:Token: None 2022-09-12T19:02:29.824 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/ 2022-09-12T19:02:29.825 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA= 2022-09-12T19:02:29.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA=', 'Content-Length': '0'} 2022-09-12T19:02:29.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000707080b59263d08a-00631f8245-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.829 DEBUG:boto:b'jdpiak-18dummy1000falsedummy2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:29.830 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:29.831 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:29.832 DEBUG:boto:path=/jdpiak-18/?prefix=dummy 2022-09-12T19:02:29.832 DEBUG:boto:auth_path=/jdpiak-18/?prefix=dummy 2022-09-12T19:02:29.833 DEBUG:boto:Method: GET 2022-09-12T19:02:29.833 DEBUG:boto:Path: /jdpiak-18/?prefix=dummy 2022-09-12T19:02:29.834 DEBUG:boto:Data: 2022-09-12T19:02:29.835 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.835 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.836 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.836 DEBUG:boto:Params: {} 2022-09-12T19:02:29.837 DEBUG:boto:Token: None 2022-09-12T19:02:29.837 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/ 2022-09-12T19:02:29.838 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA= 2022-09-12T19:02:29.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA=', 'Content-Length': '0'} 2022-09-12T19:02:29.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004198b0c7189a3867-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.843 DEBUG:boto:b'jdpiak-18dummy1000falsedummy2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:29.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:02:29.845 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:29.846 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:29.847 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.848 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.849 DEBUG:boto:Method: GET 2022-09-12T19:02:29.850 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.851 DEBUG:boto:Data: 2022-09-12T19:02:29.851 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.852 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:29.852 DEBUG:boto:Port: 8001 2022-09-12T19:02:29.853 DEBUG:boto:Params: {} 2022-09-12T19:02:29.854 DEBUG:boto:Token: None 2022-09-12T19:02:29.854 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.855 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sOR1P/IGEW94ovSyOXkI19xRrmg= 2022-09-12T19:02:29.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sOR1P/IGEW94ovSyOXkI19xRrmg=', 'Content-Length': '0'} 2022-09-12T19:02:29.858 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d04534719ddaa36d-00631f8245-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.859 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:29.860 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:29.860 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.861 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.861 DEBUG:boto:Method: GET 2022-09-12T19:02:29.862 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.862 DEBUG:boto:Data: 2022-09-12T19:02:29.863 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.864 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.864 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.865 DEBUG:boto:Params: {} 2022-09-12T19:02:29.865 DEBUG:boto:Token: None 2022-09-12T19:02:29.866 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.867 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sOR1P/IGEW94ovSyOXkI19xRrmg= 2022-09-12T19:02:29.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sOR1P/IGEW94ovSyOXkI19xRrmg=', 'Content-Length': '0'} 2022-09-12T19:02:29.869 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6fec7af65d7e3d6-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.870 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:29.871 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:29.871 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.872 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.872 DEBUG:boto:Method: HEAD 2022-09-12T19:02:29.873 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.873 DEBUG:boto:Data: 2022-09-12T19:02:29.874 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.874 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:29.875 DEBUG:boto:Port: 8001 2022-09-12T19:02:29.875 DEBUG:boto:Params: {} 2022-09-12T19:02:29.876 DEBUG:boto:Token: None 2022-09-12T19:02:29.877 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.878 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6fNgwOXm/lNqUjLLXtKQqgK7GmM= 2022-09-12T19:02:29.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6fNgwOXm/lNqUjLLXtKQqgK7GmM=', 'Content-Length': '0'} 2022-09-12T19:02:29.881 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079dde8a0c83166c9-00631f8245-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.881 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:29.882 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:29.883 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.883 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.884 DEBUG:boto:Method: HEAD 2022-09-12T19:02:29.884 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.885 DEBUG:boto:Data: 2022-09-12T19:02:29.885 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.886 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.886 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.887 DEBUG:boto:Params: {} 2022-09-12T19:02:29.887 DEBUG:boto:Token: None 2022-09-12T19:02:29.888 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.889 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6fNgwOXm/lNqUjLLXtKQqgK7GmM= 2022-09-12T19:02:29.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6fNgwOXm/lNqUjLLXtKQqgK7GmM=', 'Content-Length': '0'} 2022-09-12T19:02:29.891 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053b23e44e764ab64-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:02:29.892 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:29.893 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:29.894 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.894 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.895 DEBUG:boto:Method: GET 2022-09-12T19:02:29.895 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.896 DEBUG:boto:Data: 2022-09-12T19:02:29.896 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.897 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:29.897 DEBUG:boto:Port: 8001 2022-09-12T19:02:29.898 DEBUG:boto:Params: {} 2022-09-12T19:02:29.898 DEBUG:boto:Token: None 2022-09-12T19:02:29.899 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.900 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sOR1P/IGEW94ovSyOXkI19xRrmg= 2022-09-12T19:02:29.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sOR1P/IGEW94ovSyOXkI19xRrmg=', 'Content-Length': '0'} 2022-09-12T19:02:29.903 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5518e052542639e-00631f8245-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.946 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:29.947 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:29.947 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.948 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.948 DEBUG:boto:Method: GET 2022-09-12T19:02:29.949 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.949 DEBUG:boto:Data: 2022-09-12T19:02:29.950 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.951 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.951 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.952 DEBUG:boto:Params: {} 2022-09-12T19:02:29.953 DEBUG:boto:Token: None 2022-09-12T19:02:29.954 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:29.955 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sOR1P/IGEW94ovSyOXkI19xRrmg= 2022-09-12T19:02:29.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sOR1P/IGEW94ovSyOXkI19xRrmg=', 'Content-Length': '0'} 2022-09-12T19:02:29.958 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a65778eed8308a6-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.959 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:29.960 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:29.960 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:29.961 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:29.961 DEBUG:boto:Method: GET 2022-09-12T19:02:29.962 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:29.962 DEBUG:boto:Data: 2022-09-12T19:02:29.963 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.963 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:29.964 DEBUG:boto:Port: 8001 2022-09-12T19:02:29.964 DEBUG:boto:Params: {} 2022-09-12T19:02:29.965 DEBUG:boto:Token: None 2022-09-12T19:02:29.966 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/ 2022-09-12T19:02:29.967 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA= 2022-09-12T19:02:29.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA=', 'Content-Length': '0'} 2022-09-12T19:02:29.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ada2ceaa80f0330-00631f8245-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:29.973 DEBUG:boto:b'jdpiak-18obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:29.973 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:29.974 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:29.975 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:29.975 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:29.976 DEBUG:boto:Method: GET 2022-09-12T19:02:29.976 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:29.977 DEBUG:boto:Data: 2022-09-12T19:02:29.977 DEBUG:boto:Headers: {} 2022-09-12T19:02:29.978 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:29.978 DEBUG:boto:Port: 8000 2022-09-12T19:02:29.979 DEBUG:boto:Params: {} 2022-09-12T19:02:29.979 DEBUG:boto:Token: None 2022-09-12T19:02:29.980 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:29 GMT /jdpiak-18/ 2022-09-12T19:02:29.981 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA= 2022-09-12T19:02:29.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qrrrYVKN6nggCkQP0dyMuoc+4YA=', 'Content-Length': '0'} 2022-09-12T19:02:29.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005747316fa8e1fa7f-00631f8245-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.030 DEBUG:boto:b'jdpiak-18obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:30.031 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:30.032 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:30.032 DEBUG:boto:Method: HEAD 2022-09-12T19:02:30.033 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1 2022-09-12T19:02:30.033 DEBUG:boto:Data: 2022-09-12T19:02:30.034 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.034 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.035 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.035 DEBUG:boto:Params: {} 2022-09-12T19:02:30.036 DEBUG:boto:Token: None 2022-09-12T19:02:30.036 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone1 2022-09-12T19:02:30.037 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1xAelslwhiVLC+vh4DgjccfxPEQ= 2022-09-12T19:02:30.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1xAelslwhiVLC+vh4DgjccfxPEQ=', 'Content-Length': '0'} 2022-09-12T19:02:30.040 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000164e116a657ae0d0-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.041 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:30.042 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:30.042 DEBUG:boto:Method: HEAD 2022-09-12T19:02:30.043 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1 2022-09-12T19:02:30.043 DEBUG:boto:Data: 2022-09-12T19:02:30.044 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.044 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:30.045 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.045 DEBUG:boto:Params: {} 2022-09-12T19:02:30.046 DEBUG:boto:Token: None 2022-09-12T19:02:30.046 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone1 2022-09-12T19:02:30.047 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1xAelslwhiVLC+vh4DgjccfxPEQ= 2022-09-12T19:02:30.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1xAelslwhiVLC+vh4DgjccfxPEQ=', 'Content-Length': '0'} 2022-09-12T19:02:30.050 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000043121d30a82d8ce9-00631f8246-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:02:30.051 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:30.052 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:30.052 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.053 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.053 DEBUG:boto:Method: GET 2022-09-12T19:02:30.054 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.054 DEBUG:boto:Data: 2022-09-12T19:02:30.055 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.056 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.056 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.057 DEBUG:boto:Params: {} 2022-09-12T19:02:30.058 DEBUG:boto:Token: None 2022-09-12T19:02:30.058 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.059 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SDz86yz6sFNflBlibiNeAEY1bzk= 2022-09-12T19:02:30.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SDz86yz6sFNflBlibiNeAEY1bzk=', 'Content-Length': '0'} 2022-09-12T19:02:30.062 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095594e775efa39c7-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.063 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:30.064 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:30.065 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.065 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.066 DEBUG:boto:Method: GET 2022-09-12T19:02:30.066 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.067 DEBUG:boto:Data: 2022-09-12T19:02:30.067 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.068 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:30.068 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.069 DEBUG:boto:Params: {} 2022-09-12T19:02:30.070 DEBUG:boto:Token: None 2022-09-12T19:02:30.070 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.071 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SDz86yz6sFNflBlibiNeAEY1bzk= 2022-09-12T19:02:30.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SDz86yz6sFNflBlibiNeAEY1bzk=', 'Content-Length': '0'} 2022-09-12T19:02:30.074 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f549bf69a4481ebb-00631f8246-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.075 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:30.075 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:30.076 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:30.076 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:30.077 DEBUG:boto:Method: GET 2022-09-12T19:02:30.077 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:30.078 DEBUG:boto:Data: 2022-09-12T19:02:30.078 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.079 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.079 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.080 DEBUG:boto:Params: {} 2022-09-12T19:02:30.080 DEBUG:boto:Token: None 2022-09-12T19:02:30.081 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/ 2022-09-12T19:02:30.082 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM= 2022-09-12T19:02:30.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM=', 'Content-Length': '0'} 2022-09-12T19:02:30.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff29db042d3074ab-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.086 DEBUG:boto:b'jdpiak-18obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:30.087 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:30.087 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:30.088 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:30.088 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:30.089 DEBUG:boto:Method: GET 2022-09-12T19:02:30.089 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:30.090 DEBUG:boto:Data: 2022-09-12T19:02:30.090 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.091 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:30.091 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.092 DEBUG:boto:Params: {} 2022-09-12T19:02:30.093 DEBUG:boto:Token: None 2022-09-12T19:02:30.093 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/ 2022-09-12T19:02:30.094 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM= 2022-09-12T19:02:30.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM=', 'Content-Length': '0'} 2022-09-12T19:02:30.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024ac06bfa12e9ac4-00631f8246-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.098 DEBUG:boto:b'jdpiak-18obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:30.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:02:30.099 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:30.100 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:30.101 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.101 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.102 DEBUG:boto:Method: GET 2022-09-12T19:02:30.102 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.103 DEBUG:boto:Data: 2022-09-12T19:02:30.103 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.104 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.104 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.105 DEBUG:boto:Params: {} 2022-09-12T19:02:30.105 DEBUG:boto:Token: None 2022-09-12T19:02:30.106 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.107 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ouJtJvBrIXTPvln9qO/BkW1Cgus= 2022-09-12T19:02:30.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ouJtJvBrIXTPvln9qO/BkW1Cgus=', 'Content-Length': '0'} 2022-09-12T19:02:30.110 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a04824253f711a7-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.110 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:30.111 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:30.112 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.112 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.113 DEBUG:boto:Method: GET 2022-09-12T19:02:30.113 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.114 DEBUG:boto:Data: 2022-09-12T19:02:30.114 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.115 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:30.115 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.116 DEBUG:boto:Params: {} 2022-09-12T19:02:30.116 DEBUG:boto:Token: None 2022-09-12T19:02:30.117 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.118 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ouJtJvBrIXTPvln9qO/BkW1Cgus= 2022-09-12T19:02:30.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ouJtJvBrIXTPvln9qO/BkW1Cgus=', 'Content-Length': '0'} 2022-09-12T19:02:30.121 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8a56cfa06993f72-00631f8246-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.121 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:30.122 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:30.122 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.123 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.123 DEBUG:boto:Method: HEAD 2022-09-12T19:02:30.124 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.124 DEBUG:boto:Data: 2022-09-12T19:02:30.125 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.125 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.126 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.126 DEBUG:boto:Params: {} 2022-09-12T19:02:30.127 DEBUG:boto:Token: None 2022-09-12T19:02:30.128 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.128 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UEmF9xwHYAVPh0urS3hy3G3OXmE= 2022-09-12T19:02:30.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UEmF9xwHYAVPh0urS3hy3G3OXmE=', 'Content-Length': '0'} 2022-09-12T19:02:30.131 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f35c162477f6dfcf-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.131 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:30.132 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:30.133 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.133 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.134 DEBUG:boto:Method: HEAD 2022-09-12T19:02:30.134 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.135 DEBUG:boto:Data: 2022-09-12T19:02:30.135 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.136 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:30.136 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.137 DEBUG:boto:Params: {} 2022-09-12T19:02:30.137 DEBUG:boto:Token: None 2022-09-12T19:02:30.138 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.138 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UEmF9xwHYAVPh0urS3hy3G3OXmE= 2022-09-12T19:02:30.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UEmF9xwHYAVPh0urS3hy3G3OXmE=', 'Content-Length': '0'} 2022-09-12T19:02:30.141 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ff249aff7c0285b-00631f8246-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:02:30.142 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:30.143 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:30.143 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.144 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.144 DEBUG:boto:Method: GET 2022-09-12T19:02:30.145 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.145 DEBUG:boto:Data: 2022-09-12T19:02:30.146 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.147 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.147 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.148 DEBUG:boto:Params: {} 2022-09-12T19:02:30.148 DEBUG:boto:Token: None 2022-09-12T19:02:30.149 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.149 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ouJtJvBrIXTPvln9qO/BkW1Cgus= 2022-09-12T19:02:30.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ouJtJvBrIXTPvln9qO/BkW1Cgus=', 'Content-Length': '0'} 2022-09-12T19:02:30.152 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027ed3281eb69cbd4-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.194 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:30.195 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:30.195 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.196 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.197 DEBUG:boto:Method: GET 2022-09-12T19:02:30.197 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.198 DEBUG:boto:Data: 2022-09-12T19:02:30.198 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.199 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:30.199 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.200 DEBUG:boto:Params: {} 2022-09-12T19:02:30.200 DEBUG:boto:Token: None 2022-09-12T19:02:30.201 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.202 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ouJtJvBrIXTPvln9qO/BkW1Cgus= 2022-09-12T19:02:30.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ouJtJvBrIXTPvln9qO/BkW1Cgus=', 'Content-Length': '0'} 2022-09-12T19:02:30.205 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000cc40d90039cea5e-00631f8246-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.206 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:30.207 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:30.208 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:30.209 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:30.210 DEBUG:boto:Method: GET 2022-09-12T19:02:30.211 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:30.211 DEBUG:boto:Data: 2022-09-12T19:02:30.212 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.213 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.214 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.215 DEBUG:boto:Params: {} 2022-09-12T19:02:30.216 DEBUG:boto:Token: None 2022-09-12T19:02:30.217 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/ 2022-09-12T19:02:30.218 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM= 2022-09-12T19:02:30.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM=', 'Content-Length': '0'} 2022-09-12T19:02:30.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062a4f6046d94d694-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.223 DEBUG:boto:b'jdpiak-18obj-test-zone21000falseobj-test-zone22022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:30.225 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:30.226 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:30.226 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:30.227 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:30.227 DEBUG:boto:Method: GET 2022-09-12T19:02:30.228 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:30.228 DEBUG:boto:Data: 2022-09-12T19:02:30.229 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.229 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:30.230 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.230 DEBUG:boto:Params: {} 2022-09-12T19:02:30.231 DEBUG:boto:Token: None 2022-09-12T19:02:30.232 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/ 2022-09-12T19:02:30.232 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM= 2022-09-12T19:02:30.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM=', 'Content-Length': '0'} 2022-09-12T19:02:30.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0e20ec84eeca8ad-00631f8246-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.237 DEBUG:boto:b'jdpiak-18obj-test-zone21000falseobj-test-zone22022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:30.237 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:30.238 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:30.239 DEBUG:boto:Method: HEAD 2022-09-12T19:02:30.239 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2 2022-09-12T19:02:30.240 DEBUG:boto:Data: 2022-09-12T19:02:30.240 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.240 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.241 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.241 DEBUG:boto:Params: {} 2022-09-12T19:02:30.242 DEBUG:boto:Token: None 2022-09-12T19:02:30.243 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone2 2022-09-12T19:02:30.243 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0S7adQfYJxEnCKoTv1KbZ1XDR3E= 2022-09-12T19:02:30.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0S7adQfYJxEnCKoTv1KbZ1XDR3E=', 'Content-Length': '0'} 2022-09-12T19:02:30.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b1273a8a484f536-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.247 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:30.248 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:30.248 DEBUG:boto:Method: HEAD 2022-09-12T19:02:30.249 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2 2022-09-12T19:02:30.249 DEBUG:boto:Data: 2022-09-12T19:02:30.250 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.250 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:30.251 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.251 DEBUG:boto:Params: {} 2022-09-12T19:02:30.252 DEBUG:boto:Token: None 2022-09-12T19:02:30.252 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone2 2022-09-12T19:02:30.253 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0S7adQfYJxEnCKoTv1KbZ1XDR3E= 2022-09-12T19:02:30.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0S7adQfYJxEnCKoTv1KbZ1XDR3E=', 'Content-Length': '0'} 2022-09-12T19:02:30.256 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a166ee9053ead5a-00631f8246-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:02:30.257 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:30.257 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:30.258 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.259 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.259 DEBUG:boto:Method: GET 2022-09-12T19:02:30.260 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.260 DEBUG:boto:Data: 2022-09-12T19:02:30.261 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.261 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.262 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.262 DEBUG:boto:Params: {} 2022-09-12T19:02:30.263 DEBUG:boto:Token: None 2022-09-12T19:02:30.263 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.264 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ouJtJvBrIXTPvln9qO/BkW1Cgus= 2022-09-12T19:02:30.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ouJtJvBrIXTPvln9qO/BkW1Cgus=', 'Content-Length': '0'} 2022-09-12T19:02:30.267 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000febdc63bbf4fd8c7-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.310 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:30.311 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:30.311 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.312 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.312 DEBUG:boto:Method: GET 2022-09-12T19:02:30.313 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.313 DEBUG:boto:Data: 2022-09-12T19:02:30.314 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.315 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:30.315 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.316 DEBUG:boto:Params: {} 2022-09-12T19:02:30.317 DEBUG:boto:Token: None 2022-09-12T19:02:30.317 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:30.318 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ouJtJvBrIXTPvln9qO/BkW1Cgus= 2022-09-12T19:02:30.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ouJtJvBrIXTPvln9qO/BkW1Cgus=', 'Content-Length': '0'} 2022-09-12T19:02:30.321 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000867d9d5ea7d92e4f-00631f8246-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.322 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:30.322 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:30.323 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:30.323 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:30.324 DEBUG:boto:Method: GET 2022-09-12T19:02:30.324 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:30.325 DEBUG:boto:Data: 2022-09-12T19:02:30.325 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.326 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.326 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.327 DEBUG:boto:Params: {} 2022-09-12T19:02:30.328 DEBUG:boto:Token: None 2022-09-12T19:02:30.328 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/ 2022-09-12T19:02:30.329 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM= 2022-09-12T19:02:30.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM=', 'Content-Length': '0'} 2022-09-12T19:02:30.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ffc41aef4883e1a-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.333 DEBUG:boto:b'jdpiak-18obj-test-zone21000falseobj-test-zone22022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:30.334 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:30.334 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:30.335 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:30.335 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:30.336 DEBUG:boto:Method: GET 2022-09-12T19:02:30.336 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:30.337 DEBUG:boto:Data: 2022-09-12T19:02:30.337 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.338 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:30.338 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.339 DEBUG:boto:Params: {} 2022-09-12T19:02:30.339 DEBUG:boto:Token: None 2022-09-12T19:02:30.340 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/ 2022-09-12T19:02:30.341 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM= 2022-09-12T19:02:30.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM=', 'Content-Length': '0'} 2022-09-12T19:02:30.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9e26be4d8accf2d-00631f8246-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.345 DEBUG:boto:b'jdpiak-18obj-test-zone21000falseobj-test-zone22022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:30.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:02:30.346 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:30.346 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:30.347 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.348 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.348 DEBUG:boto:Method: GET 2022-09-12T19:02:30.349 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.349 DEBUG:boto:Data: 2022-09-12T19:02:30.350 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.350 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.351 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.351 DEBUG:boto:Params: {} 2022-09-12T19:02:30.352 DEBUG:boto:Token: None 2022-09-12T19:02:30.353 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.353 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iM+1DgK4TLr6wwktji7pptgrtYI= 2022-09-12T19:02:30.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iM+1DgK4TLr6wwktji7pptgrtYI=', 'Content-Length': '0'} 2022-09-12T19:02:30.356 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af81db9b2eec8d04-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.357 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:30.357 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:30.358 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.358 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.359 DEBUG:boto:Method: GET 2022-09-12T19:02:30.360 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.360 DEBUG:boto:Data: 2022-09-12T19:02:30.361 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.361 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:30.362 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.362 DEBUG:boto:Params: {} 2022-09-12T19:02:30.363 DEBUG:boto:Token: None 2022-09-12T19:02:30.363 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.364 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iM+1DgK4TLr6wwktji7pptgrtYI= 2022-09-12T19:02:30.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iM+1DgK4TLr6wwktji7pptgrtYI=', 'Content-Length': '0'} 2022-09-12T19:02:30.367 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016af8f1464f73258-00631f8246-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.367 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:30.368 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:30.368 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.369 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.369 DEBUG:boto:Method: HEAD 2022-09-12T19:02:30.370 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.370 DEBUG:boto:Data: 2022-09-12T19:02:30.371 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.371 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.372 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.372 DEBUG:boto:Params: {} 2022-09-12T19:02:30.373 DEBUG:boto:Token: None 2022-09-12T19:02:30.373 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.374 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XHU+FspJ0eX5w4I//SmntZYIXug= 2022-09-12T19:02:30.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XHU+FspJ0eX5w4I//SmntZYIXug=', 'Content-Length': '0'} 2022-09-12T19:02:30.376 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007dec8811700212f7-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.377 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:30.378 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:30.378 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.379 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.379 DEBUG:boto:Method: HEAD 2022-09-12T19:02:30.380 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.380 DEBUG:boto:Data: 2022-09-12T19:02:30.381 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.381 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:30.382 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.382 DEBUG:boto:Params: {} 2022-09-12T19:02:30.383 DEBUG:boto:Token: None 2022-09-12T19:02:30.383 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.384 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XHU+FspJ0eX5w4I//SmntZYIXug= 2022-09-12T19:02:30.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XHU+FspJ0eX5w4I//SmntZYIXug=', 'Content-Length': '0'} 2022-09-12T19:02:30.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000926537249c7bffc1-00631f8246-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:02:30.388 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:30.388 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:30.389 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.389 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.390 DEBUG:boto:Method: GET 2022-09-12T19:02:30.390 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.391 DEBUG:boto:Data: 2022-09-12T19:02:30.391 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.392 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.392 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.393 DEBUG:boto:Params: {} 2022-09-12T19:02:30.393 DEBUG:boto:Token: None 2022-09-12T19:02:30.394 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.395 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iM+1DgK4TLr6wwktji7pptgrtYI= 2022-09-12T19:02:30.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iM+1DgK4TLr6wwktji7pptgrtYI=', 'Content-Length': '0'} 2022-09-12T19:02:30.397 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017e32f6828ee87dc-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.398 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:30.399 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:30.399 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.400 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.400 DEBUG:boto:Method: GET 2022-09-12T19:02:30.401 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.401 DEBUG:boto:Data: 2022-09-12T19:02:30.402 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.402 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:30.403 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.403 DEBUG:boto:Params: {} 2022-09-12T19:02:30.404 DEBUG:boto:Token: None 2022-09-12T19:02:30.405 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.405 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iM+1DgK4TLr6wwktji7pptgrtYI= 2022-09-12T19:02:30.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iM+1DgK4TLr6wwktji7pptgrtYI=', 'Content-Length': '0'} 2022-09-12T19:02:30.408 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000587ec6cb7460fab1-00631f8246-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.450 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:30.451 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:30.452 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:30.453 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:30.454 DEBUG:boto:Method: GET 2022-09-12T19:02:30.454 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:30.455 DEBUG:boto:Data: 2022-09-12T19:02:30.455 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.456 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.457 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.457 DEBUG:boto:Params: {} 2022-09-12T19:02:30.458 DEBUG:boto:Token: None 2022-09-12T19:02:30.458 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/ 2022-09-12T19:02:30.459 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM= 2022-09-12T19:02:30.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM=', 'Content-Length': '0'} 2022-09-12T19:02:30.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4b63911dbe329b2-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.463 DEBUG:boto:b'jdpiak-18obj-test-zone31000falseobj-test-zone32022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:30.464 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:30.464 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:30.465 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:30.466 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:30.467 DEBUG:boto:Method: GET 2022-09-12T19:02:30.467 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:30.468 DEBUG:boto:Data: 2022-09-12T19:02:30.469 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.469 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:30.470 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.471 DEBUG:boto:Params: {} 2022-09-12T19:02:30.471 DEBUG:boto:Token: None 2022-09-12T19:02:30.472 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/ 2022-09-12T19:02:30.472 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM= 2022-09-12T19:02:30.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM=', 'Content-Length': '0'} 2022-09-12T19:02:30.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdf5e4bedd253ff6-00631f8246-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.478 DEBUG:boto:b'jdpiak-18obj-test-zone31000falseobj-test-zone32022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:30.479 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:30.479 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:30.480 DEBUG:boto:Method: HEAD 2022-09-12T19:02:30.480 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3 2022-09-12T19:02:30.481 DEBUG:boto:Data: 2022-09-12T19:02:30.481 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.482 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.482 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.483 DEBUG:boto:Params: {} 2022-09-12T19:02:30.483 DEBUG:boto:Token: None 2022-09-12T19:02:30.484 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone3 2022-09-12T19:02:30.485 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Vw64qUES1K9XQAv4kN7/2t8VVNg= 2022-09-12T19:02:30.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Vw64qUES1K9XQAv4kN7/2t8VVNg=', 'Content-Length': '0'} 2022-09-12T19:02:30.487 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5c25da4801e0d45-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.488 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:30.489 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:30.489 DEBUG:boto:Method: HEAD 2022-09-12T19:02:30.490 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3 2022-09-12T19:02:30.491 DEBUG:boto:Data: 2022-09-12T19:02:30.491 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.492 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:30.492 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.493 DEBUG:boto:Params: {} 2022-09-12T19:02:30.493 DEBUG:boto:Token: None 2022-09-12T19:02:30.494 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone3 2022-09-12T19:02:30.494 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Vw64qUES1K9XQAv4kN7/2t8VVNg= 2022-09-12T19:02:30.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Vw64qUES1K9XQAv4kN7/2t8VVNg=', 'Content-Length': '0'} 2022-09-12T19:02:30.497 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094bf9c3f0c59cb54-00631f8246-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:02:30.499 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:30.499 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:30.500 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.500 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.501 DEBUG:boto:Method: GET 2022-09-12T19:02:30.501 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.502 DEBUG:boto:Data: 2022-09-12T19:02:30.502 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.503 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.503 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.504 DEBUG:boto:Params: {} 2022-09-12T19:02:30.505 DEBUG:boto:Token: None 2022-09-12T19:02:30.505 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.506 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iM+1DgK4TLr6wwktji7pptgrtYI= 2022-09-12T19:02:30.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iM+1DgK4TLr6wwktji7pptgrtYI=', 'Content-Length': '0'} 2022-09-12T19:02:30.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ccacae8387516e5-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.510 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:30.510 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:30.511 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.511 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.512 DEBUG:boto:Method: GET 2022-09-12T19:02:30.512 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.513 DEBUG:boto:Data: 2022-09-12T19:02:30.513 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.514 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:30.514 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.515 DEBUG:boto:Params: {} 2022-09-12T19:02:30.516 DEBUG:boto:Token: None 2022-09-12T19:02:30.516 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:30.517 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iM+1DgK4TLr6wwktji7pptgrtYI= 2022-09-12T19:02:30.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iM+1DgK4TLr6wwktji7pptgrtYI=', 'Content-Length': '0'} 2022-09-12T19:02:30.520 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007aaf15ae0eb61461-00631f8246-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.520 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:30.521 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:30.521 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:30.522 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:30.523 DEBUG:boto:Method: GET 2022-09-12T19:02:30.523 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:30.524 DEBUG:boto:Data: 2022-09-12T19:02:30.524 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.525 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.525 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.526 DEBUG:boto:Params: {} 2022-09-12T19:02:30.526 DEBUG:boto:Token: None 2022-09-12T19:02:30.527 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/ 2022-09-12T19:02:30.528 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM= 2022-09-12T19:02:30.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM=', 'Content-Length': '0'} 2022-09-12T19:02:30.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d1f1740d3e5687f-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.532 DEBUG:boto:b'jdpiak-18obj-test-zone31000falseobj-test-zone32022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:30.532 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:30.533 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:30.533 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:30.534 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:30.535 DEBUG:boto:Method: GET 2022-09-12T19:02:30.535 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:30.536 DEBUG:boto:Data: 2022-09-12T19:02:30.536 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.537 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:30.537 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.538 DEBUG:boto:Params: {} 2022-09-12T19:02:30.538 DEBUG:boto:Token: None 2022-09-12T19:02:30.539 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/ 2022-09-12T19:02:30.540 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM= 2022-09-12T19:02:30.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM=', 'Content-Length': '0'} 2022-09-12T19:02:30.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d612c1eafaaabb00-00631f8246-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.543 DEBUG:boto:b'jdpiak-18obj-test-zone31000falseobj-test-zone32022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:30.544 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-18 zones={test-zone3, test-zone1} 2022-09-12T19:02:30.545 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-18 zones={test-zone3, test-zone2} 2022-09-12T19:02:30.545 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:30.546 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:30.546 DEBUG:boto:Method: HEAD 2022-09-12T19:02:30.547 DEBUG:boto:Path: /jdpiak-18/ 2022-09-12T19:02:30.547 DEBUG:boto:Data: 2022-09-12T19:02:30.548 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.548 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.549 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.549 DEBUG:boto:Params: {} 2022-09-12T19:02:30.550 DEBUG:boto:Token: None 2022-09-12T19:02:30.550 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/ 2022-09-12T19:02:30.551 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EuLX+2hmhyObBxSGV556XHaRMx8= 2022-09-12T19:02:30.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EuLX+2hmhyObBxSGV556XHaRMx8=', 'Content-Length': '0'} 2022-09-12T19:02:30.554 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ef2b99d4725f37b-00631f8246-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.555 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:30.556 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:30.557 DEBUG:boto:Method: HEAD 2022-09-12T19:02:30.557 DEBUG:boto:Path: /jdpiak-18/ 2022-09-12T19:02:30.558 DEBUG:boto:Data: 2022-09-12T19:02:30.559 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.560 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:30.560 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.561 DEBUG:boto:Params: {} 2022-09-12T19:02:30.562 DEBUG:boto:Token: None 2022-09-12T19:02:30.562 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/ 2022-09-12T19:02:30.563 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EuLX+2hmhyObBxSGV556XHaRMx8= 2022-09-12T19:02:30.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EuLX+2hmhyObBxSGV556XHaRMx8=', 'Content-Length': '0'} 2022-09-12T19:02:30.566 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d36c57d18e4bece-00631f8246-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.567 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:02:30.567 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:30.568 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:30.568 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:30.569 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:30.569 DEBUG:boto:Method: GET 2022-09-12T19:02:30.570 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:30.570 DEBUG:boto:Data: 2022-09-12T19:02:30.571 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.571 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.572 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.572 DEBUG:boto:Params: {} 2022-09-12T19:02:30.573 DEBUG:boto:Token: None 2022-09-12T19:02:30.574 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/?versions 2022-09-12T19:02:30.574 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:taPrrj4CQDVPWBoQZSuPMWavpvQ= 2022-09-12T19:02:30.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:taPrrj4CQDVPWBoQZSuPMWavpvQ=', 'Content-Length': '0'} 2022-09-12T19:02:30.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da615a6571698254-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.578 DEBUG:boto:b'jdpiak-18999falsedummyqUufSwmbKETbT7yNOH8sL0A4tddUbNOtrue2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ToUtj2XoS.tz3goIbvjFwiigMyzGOpLtrue2022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2DPksivOWVYhJDrEwQqHYV7wK1Ul1JkWtrue2022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Ztrue2022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:30.579 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:02:30.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:02:30.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:02:30.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:02:30.581 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:02:30.582 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:30.582 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:30.583 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:30.583 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:30.584 DEBUG:boto:Method: GET 2022-09-12T19:02:30.584 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:30.585 DEBUG:boto:Data: 2022-09-12T19:02:30.585 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.586 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:30.586 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.587 DEBUG:boto:Params: {} 2022-09-12T19:02:30.587 DEBUG:boto:Token: None 2022-09-12T19:02:30.588 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/?versions 2022-09-12T19:02:30.589 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:taPrrj4CQDVPWBoQZSuPMWavpvQ= 2022-09-12T19:02:30.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:taPrrj4CQDVPWBoQZSuPMWavpvQ=', 'Content-Length': '0'} 2022-09-12T19:02:30.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bff2b3bf2e25c5ef-00631f8246-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.593 DEBUG:boto:b'jdpiak-18999falsedummyqUufSwmbKETbT7yNOH8sL0A4tddUbNOtrue2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ToUtj2XoS.tz3goIbvjFwiigMyzGOpLtrue2022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2DPksivOWVYhJDrEwQqHYV7wK1Ul1JkWtrue2022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Ztrue2022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:30.594 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:02:30.594 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:02:30.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:02:30.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:02:30.596 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:30.596 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:30.597 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:30.597 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:30.598 DEBUG:boto:Method: GET 2022-09-12T19:02:30.598 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:30.599 DEBUG:boto:Data: 2022-09-12T19:02:30.599 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.600 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.600 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.601 DEBUG:boto:Params: {} 2022-09-12T19:02:30.601 DEBUG:boto:Token: None 2022-09-12T19:02:30.602 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/?versions 2022-09-12T19:02:30.602 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:taPrrj4CQDVPWBoQZSuPMWavpvQ= 2022-09-12T19:02:30.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:taPrrj4CQDVPWBoQZSuPMWavpvQ=', 'Content-Length': '0'} 2022-09-12T19:02:30.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000083bc15b7538010b-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.607 DEBUG:boto:b'jdpiak-18999falsedummyqUufSwmbKETbT7yNOH8sL0A4tddUbNOtrue2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ToUtj2XoS.tz3goIbvjFwiigMyzGOpLtrue2022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2DPksivOWVYhJDrEwQqHYV7wK1Ul1JkWtrue2022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Ztrue2022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:30.608 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:30.608 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:30.609 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:30.609 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:30.610 DEBUG:boto:Method: GET 2022-09-12T19:02:30.610 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999 2022-09-12T19:02:30.611 DEBUG:boto:Data: 2022-09-12T19:02:30.611 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.612 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:30.612 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.613 DEBUG:boto:Params: {} 2022-09-12T19:02:30.613 DEBUG:boto:Token: None 2022-09-12T19:02:30.614 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/?versions 2022-09-12T19:02:30.615 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:taPrrj4CQDVPWBoQZSuPMWavpvQ= 2022-09-12T19:02:30.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:taPrrj4CQDVPWBoQZSuPMWavpvQ=', 'Content-Length': '0'} 2022-09-12T19:02:30.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d31b7cec96e28ae4-00631f8246-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.618 DEBUG:boto:b'jdpiak-18999falsedummyqUufSwmbKETbT7yNOH8sL0A4tddUbNOtrue2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ToUtj2XoS.tz3goIbvjFwiigMyzGOpLtrue2022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2DPksivOWVYhJDrEwQqHYV7wK1Ul1JkWtrue2022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Ztrue2022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:30.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:02:30.620 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:30.620 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:30.621 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.622 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.622 DEBUG:boto:Method: GET 2022-09-12T19:02:30.623 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.624 DEBUG:boto:Data: 2022-09-12T19:02:30.624 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.625 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.625 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.626 DEBUG:boto:Params: {} 2022-09-12T19:02:30.627 DEBUG:boto:Token: None 2022-09-12T19:02:30.627 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.628 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kFFkOtU3xio6k93bxKnvHHfX8U0= 2022-09-12T19:02:30.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kFFkOtU3xio6k93bxKnvHHfX8U0=', 'Content-Length': '0'} 2022-09-12T19:02:30.631 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d9477a61624cb40b-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.632 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:30.633 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:30.634 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.635 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.635 DEBUG:boto:Method: GET 2022-09-12T19:02:30.636 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.636 DEBUG:boto:Data: 2022-09-12T19:02:30.637 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.639 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:30.640 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.641 DEBUG:boto:Params: {} 2022-09-12T19:02:30.642 DEBUG:boto:Token: None 2022-09-12T19:02:30.643 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.645 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kFFkOtU3xio6k93bxKnvHHfX8U0= 2022-09-12T19:02:30.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kFFkOtU3xio6k93bxKnvHHfX8U0=', 'Content-Length': '0'} 2022-09-12T19:02:30.649 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cdcec2d0cb78a702-00631f8246-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.650 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:30.650 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:30.651 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.652 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.652 DEBUG:boto:Method: HEAD 2022-09-12T19:02:30.653 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.654 DEBUG:boto:Data: 2022-09-12T19:02:30.655 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.655 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.656 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.656 DEBUG:boto:Params: {} 2022-09-12T19:02:30.657 DEBUG:boto:Token: None 2022-09-12T19:02:30.658 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.659 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nOAT6ZJ/vbRIr6m5Y6NqD6bKXbU= 2022-09-12T19:02:30.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nOAT6ZJ/vbRIr6m5Y6NqD6bKXbU=', 'Content-Length': '0'} 2022-09-12T19:02:30.661 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eefa07d1864a3e52-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.662 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:30.663 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:30.664 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.664 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.665 DEBUG:boto:Method: HEAD 2022-09-12T19:02:30.666 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.666 DEBUG:boto:Data: 2022-09-12T19:02:30.667 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.668 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:30.669 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.670 DEBUG:boto:Params: {} 2022-09-12T19:02:30.671 DEBUG:boto:Token: None 2022-09-12T19:02:30.671 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.672 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nOAT6ZJ/vbRIr6m5Y6NqD6bKXbU= 2022-09-12T19:02:30.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nOAT6ZJ/vbRIr6m5Y6NqD6bKXbU=', 'Content-Length': '0'} 2022-09-12T19:02:30.675 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000abd04002baf685a5-00631f8246-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:02:30.677 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:30.679 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:30.680 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.681 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.682 DEBUG:boto:Method: GET 2022-09-12T19:02:30.683 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.684 DEBUG:boto:Data: 2022-09-12T19:02:30.685 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.686 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.687 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.687 DEBUG:boto:Params: {} 2022-09-12T19:02:30.688 DEBUG:boto:Token: None 2022-09-12T19:02:30.689 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.690 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kFFkOtU3xio6k93bxKnvHHfX8U0= 2022-09-12T19:02:30.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kFFkOtU3xio6k93bxKnvHHfX8U0=', 'Content-Length': '0'} 2022-09-12T19:02:30.694 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004160a4bdda3de0ee-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.695 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:30.697 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:30.698 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.699 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.700 DEBUG:boto:Method: GET 2022-09-12T19:02:30.700 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.701 DEBUG:boto:Data: 2022-09-12T19:02:30.702 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.703 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:30.704 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.704 DEBUG:boto:Params: {} 2022-09-12T19:02:30.705 DEBUG:boto:Token: None 2022-09-12T19:02:30.707 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.709 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kFFkOtU3xio6k93bxKnvHHfX8U0= 2022-09-12T19:02:30.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kFFkOtU3xio6k93bxKnvHHfX8U0=', 'Content-Length': '0'} 2022-09-12T19:02:30.713 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000083a0d66f5a3b1ba4-00631f8246-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.715 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:30.716 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:30.717 DEBUG:boto:path=/jdpiak-18/?prefix=dummy 2022-09-12T19:02:30.717 DEBUG:boto:auth_path=/jdpiak-18/?prefix=dummy 2022-09-12T19:02:30.718 DEBUG:boto:Method: GET 2022-09-12T19:02:30.719 DEBUG:boto:Path: /jdpiak-18/?prefix=dummy 2022-09-12T19:02:30.720 DEBUG:boto:Data: 2022-09-12T19:02:30.721 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.722 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.723 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.724 DEBUG:boto:Params: {} 2022-09-12T19:02:30.726 DEBUG:boto:Token: None 2022-09-12T19:02:30.727 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/ 2022-09-12T19:02:30.728 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM= 2022-09-12T19:02:30.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM=', 'Content-Length': '0'} 2022-09-12T19:02:30.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ca6c323fe102308-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.774 DEBUG:boto:b'jdpiak-18dummy1000falsedummy2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:30.775 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:30.776 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:30.776 DEBUG:boto:path=/jdpiak-18/?prefix=dummy 2022-09-12T19:02:30.777 DEBUG:boto:auth_path=/jdpiak-18/?prefix=dummy 2022-09-12T19:02:30.777 DEBUG:boto:Method: GET 2022-09-12T19:02:30.778 DEBUG:boto:Path: /jdpiak-18/?prefix=dummy 2022-09-12T19:02:30.779 DEBUG:boto:Data: 2022-09-12T19:02:30.780 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.781 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:30.781 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.782 DEBUG:boto:Params: {} 2022-09-12T19:02:30.783 DEBUG:boto:Token: None 2022-09-12T19:02:30.784 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/ 2022-09-12T19:02:30.785 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM= 2022-09-12T19:02:30.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM=', 'Content-Length': '0'} 2022-09-12T19:02:30.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c74ec8deed95979-00631f8246-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.791 DEBUG:boto:b'jdpiak-18dummy1000falsedummy2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:30.793 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:30.794 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:30.795 DEBUG:boto:Method: HEAD 2022-09-12T19:02:30.797 DEBUG:boto:Path: /jdpiak-18/dummy 2022-09-12T19:02:30.798 DEBUG:boto:Data: 2022-09-12T19:02:30.800 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.801 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.802 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.804 DEBUG:boto:Params: {} 2022-09-12T19:02:30.805 DEBUG:boto:Token: None 2022-09-12T19:02:30.807 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/dummy 2022-09-12T19:02:30.809 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GkPhLDqMGRzOJjCrUUquGS3Saes= 2022-09-12T19:02:30.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GkPhLDqMGRzOJjCrUUquGS3Saes=', 'Content-Length': '0'} 2022-09-12T19:02:30.814 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c8b2e05e0c885386-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.816 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:30.817 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:30.819 DEBUG:boto:Method: HEAD 2022-09-12T19:02:30.820 DEBUG:boto:Path: /jdpiak-18/dummy 2022-09-12T19:02:30.821 DEBUG:boto:Data: 2022-09-12T19:02:30.822 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.823 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:30.824 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.825 DEBUG:boto:Params: {} 2022-09-12T19:02:30.826 DEBUG:boto:Token: None 2022-09-12T19:02:30.828 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/dummy 2022-09-12T19:02:30.829 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GkPhLDqMGRzOJjCrUUquGS3Saes= 2022-09-12T19:02:30.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GkPhLDqMGRzOJjCrUUquGS3Saes=', 'Content-Length': '0'} 2022-09-12T19:02:30.833 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000084c02b4c35b6bbad-00631f8246-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:02:30.836 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:30.837 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:30.838 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.839 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.840 DEBUG:boto:Method: GET 2022-09-12T19:02:30.841 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.842 DEBUG:boto:Data: 2022-09-12T19:02:30.843 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.843 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.844 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.844 DEBUG:boto:Params: {} 2022-09-12T19:02:30.845 DEBUG:boto:Token: None 2022-09-12T19:02:30.846 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.847 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kFFkOtU3xio6k93bxKnvHHfX8U0= 2022-09-12T19:02:30.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kFFkOtU3xio6k93bxKnvHHfX8U0=', 'Content-Length': '0'} 2022-09-12T19:02:30.850 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000df98c6f74ffd9fe2-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.851 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:02:30.851 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:02:30.852 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.852 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.853 DEBUG:boto:Method: GET 2022-09-12T19:02:30.853 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.854 DEBUG:boto:Data: 2022-09-12T19:02:30.854 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.855 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:30.855 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.856 DEBUG:boto:Params: {} 2022-09-12T19:02:30.856 DEBUG:boto:Token: None 2022-09-12T19:02:30.857 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:02:30.857 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kFFkOtU3xio6k93bxKnvHHfX8U0= 2022-09-12T19:02:30.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kFFkOtU3xio6k93bxKnvHHfX8U0=', 'Content-Length': '0'} 2022-09-12T19:02:30.860 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000050bdac2965ee59e9-00631f8246-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.861 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:30.861 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:30.862 DEBUG:boto:path=/jdpiak-18/?prefix=dummy 2022-09-12T19:02:30.863 DEBUG:boto:auth_path=/jdpiak-18/?prefix=dummy 2022-09-12T19:02:30.863 DEBUG:boto:Method: GET 2022-09-12T19:02:30.864 DEBUG:boto:Path: /jdpiak-18/?prefix=dummy 2022-09-12T19:02:30.864 DEBUG:boto:Data: 2022-09-12T19:02:30.865 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.865 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.866 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.866 DEBUG:boto:Params: {} 2022-09-12T19:02:30.867 DEBUG:boto:Token: None 2022-09-12T19:02:30.867 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/ 2022-09-12T19:02:30.868 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM= 2022-09-12T19:02:30.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM=', 'Content-Length': '0'} 2022-09-12T19:02:30.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b4929d8b39135e6-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.872 DEBUG:boto:b'jdpiak-18dummy1000falsedummy2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:30.873 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:30.873 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:30.874 DEBUG:boto:path=/jdpiak-18/?prefix=dummy 2022-09-12T19:02:30.874 DEBUG:boto:auth_path=/jdpiak-18/?prefix=dummy 2022-09-12T19:02:30.875 DEBUG:boto:Method: GET 2022-09-12T19:02:30.875 DEBUG:boto:Path: /jdpiak-18/?prefix=dummy 2022-09-12T19:02:30.876 DEBUG:boto:Data: 2022-09-12T19:02:30.876 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.877 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:30.877 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.878 DEBUG:boto:Params: {} 2022-09-12T19:02:30.879 DEBUG:boto:Token: None 2022-09-12T19:02:30.879 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/ 2022-09-12T19:02:30.880 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM= 2022-09-12T19:02:30.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gWxIiiBHnWkfUYRCzXSqvY/rtHM=', 'Content-Length': '0'} 2022-09-12T19:02:30.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003222cc334c8e181c-00631f8246-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.884 DEBUG:boto:b'jdpiak-18dummy1000falsedummy2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:30.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:02:30.885 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:30.886 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:30.886 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.887 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.888 DEBUG:boto:Method: GET 2022-09-12T19:02:30.888 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.889 DEBUG:boto:Data: 2022-09-12T19:02:30.889 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.890 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.890 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.891 DEBUG:boto:Params: {} 2022-09-12T19:02:30.891 DEBUG:boto:Token: None 2022-09-12T19:02:30.892 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.893 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SDz86yz6sFNflBlibiNeAEY1bzk= 2022-09-12T19:02:30.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SDz86yz6sFNflBlibiNeAEY1bzk=', 'Content-Length': '0'} 2022-09-12T19:02:30.896 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa84d795c4ba8dcf-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.896 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:30.897 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:30.897 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.898 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.899 DEBUG:boto:Method: GET 2022-09-12T19:02:30.899 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.900 DEBUG:boto:Data: 2022-09-12T19:02:30.900 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.901 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:30.901 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.902 DEBUG:boto:Params: {} 2022-09-12T19:02:30.902 DEBUG:boto:Token: None 2022-09-12T19:02:30.903 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.904 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SDz86yz6sFNflBlibiNeAEY1bzk= 2022-09-12T19:02:30.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SDz86yz6sFNflBlibiNeAEY1bzk=', 'Content-Length': '0'} 2022-09-12T19:02:30.906 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9c1f038efff6734-00631f8246-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.957 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:30.958 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:30.959 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.959 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.960 DEBUG:boto:Method: HEAD 2022-09-12T19:02:30.960 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.961 DEBUG:boto:Data: 2022-09-12T19:02:30.961 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.962 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.962 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.963 DEBUG:boto:Params: {} 2022-09-12T19:02:30.964 DEBUG:boto:Token: None 2022-09-12T19:02:30.964 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.965 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nx2CQDHL5KvHKjSBRQDxVE70vtI= 2022-09-12T19:02:30.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nx2CQDHL5KvHKjSBRQDxVE70vtI=', 'Content-Length': '0'} 2022-09-12T19:02:30.969 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d794a2df6052e0b9-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.970 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:30.971 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:30.972 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.973 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.974 DEBUG:boto:Method: HEAD 2022-09-12T19:02:30.975 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.975 DEBUG:boto:Data: 2022-09-12T19:02:30.976 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.976 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:30.977 DEBUG:boto:Port: 8000 2022-09-12T19:02:30.977 DEBUG:boto:Params: {} 2022-09-12T19:02:30.978 DEBUG:boto:Token: None 2022-09-12T19:02:30.978 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.979 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nx2CQDHL5KvHKjSBRQDxVE70vtI= 2022-09-12T19:02:30.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nx2CQDHL5KvHKjSBRQDxVE70vtI=', 'Content-Length': '0'} 2022-09-12T19:02:30.982 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003eda3a37d6e03ff4-00631f8246-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:02:30.983 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:30.984 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:30.984 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.985 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.985 DEBUG:boto:Method: GET 2022-09-12T19:02:30.986 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.986 DEBUG:boto:Data: 2022-09-12T19:02:30.987 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.987 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:30.988 DEBUG:boto:Port: 8001 2022-09-12T19:02:30.988 DEBUG:boto:Params: {} 2022-09-12T19:02:30.989 DEBUG:boto:Token: None 2022-09-12T19:02:30.990 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:30 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.990 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SDz86yz6sFNflBlibiNeAEY1bzk= 2022-09-12T19:02:30.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SDz86yz6sFNflBlibiNeAEY1bzk=', 'Content-Length': '0'} 2022-09-12T19:02:30.993 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce706f53357b0b3b-00631f8246-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:30.994 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:30.994 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:30.995 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.995 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.996 DEBUG:boto:Method: GET 2022-09-12T19:02:30.997 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:30.997 DEBUG:boto:Data: 2022-09-12T19:02:30.998 DEBUG:boto:Headers: {} 2022-09-12T19:02:30.998 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:30.999 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.000 DEBUG:boto:Params: {} 2022-09-12T19:02:31.000 DEBUG:boto:Token: None 2022-09-12T19:02:31.001 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:31.002 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZGXS4wMDP8+nQ74pSnERkYzk+jk= 2022-09-12T19:02:31.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZGXS4wMDP8+nQ74pSnERkYzk+jk=', 'Content-Length': '0'} 2022-09-12T19:02:31.004 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ed76cf3e7071e49-00631f8247-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.005 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:31.006 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:31.006 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:31.007 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:31.008 DEBUG:boto:Method: GET 2022-09-12T19:02:31.008 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:31.009 DEBUG:boto:Data: 2022-09-12T19:02:31.009 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.010 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:31.011 DEBUG:boto:Port: 8001 2022-09-12T19:02:31.011 DEBUG:boto:Params: {} 2022-09-12T19:02:31.012 DEBUG:boto:Token: None 2022-09-12T19:02:31.012 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/ 2022-09-12T19:02:31.013 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6PbxEADVy5wv3N0+4XxJtG0P048= 2022-09-12T19:02:31.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6PbxEADVy5wv3N0+4XxJtG0P048=', 'Content-Length': '0'} 2022-09-12T19:02:31.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005192aa42a18698fd-00631f8247-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.017 DEBUG:boto:b'jdpiak-18obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:31.019 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:31.020 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:31.021 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:31.022 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:31.023 DEBUG:boto:Method: GET 2022-09-12T19:02:31.024 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:31.025 DEBUG:boto:Data: 2022-09-12T19:02:31.026 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.026 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.027 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.028 DEBUG:boto:Params: {} 2022-09-12T19:02:31.028 DEBUG:boto:Token: None 2022-09-12T19:02:31.029 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/ 2022-09-12T19:02:31.029 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6PbxEADVy5wv3N0+4XxJtG0P048= 2022-09-12T19:02:31.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6PbxEADVy5wv3N0+4XxJtG0P048=', 'Content-Length': '0'} 2022-09-12T19:02:31.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000343bf2e3469f5923-00631f8247-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.033 DEBUG:boto:b'jdpiak-18obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:31.034 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:31.035 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:31.035 DEBUG:boto:Method: HEAD 2022-09-12T19:02:31.036 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1 2022-09-12T19:02:31.036 DEBUG:boto:Data: 2022-09-12T19:02:31.037 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.037 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:31.038 DEBUG:boto:Port: 8001 2022-09-12T19:02:31.039 DEBUG:boto:Params: {} 2022-09-12T19:02:31.039 DEBUG:boto:Token: None 2022-09-12T19:02:31.040 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/obj-test-zone1 2022-09-12T19:02:31.040 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y8wE0ubcRn0uAOVjojju1QYsaIs= 2022-09-12T19:02:31.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y8wE0ubcRn0uAOVjojju1QYsaIs=', 'Content-Length': '0'} 2022-09-12T19:02:31.044 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f13de2ec670fa2c-00631f8247-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.045 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:31.046 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:31.046 DEBUG:boto:Method: HEAD 2022-09-12T19:02:31.047 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1 2022-09-12T19:02:31.047 DEBUG:boto:Data: 2022-09-12T19:02:31.048 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.048 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.049 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.049 DEBUG:boto:Params: {} 2022-09-12T19:02:31.050 DEBUG:boto:Token: None 2022-09-12T19:02:31.052 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/obj-test-zone1 2022-09-12T19:02:31.053 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y8wE0ubcRn0uAOVjojju1QYsaIs= 2022-09-12T19:02:31.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y8wE0ubcRn0uAOVjojju1QYsaIs=', 'Content-Length': '0'} 2022-09-12T19:02:31.056 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bda6b4c320b31abc-00631f8247-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:02:31.057 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:31.058 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:31.058 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:31.059 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:31.059 DEBUG:boto:Method: GET 2022-09-12T19:02:31.060 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:31.061 DEBUG:boto:Data: 2022-09-12T19:02:31.061 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.062 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:31.062 DEBUG:boto:Port: 8001 2022-09-12T19:02:31.063 DEBUG:boto:Params: {} 2022-09-12T19:02:31.063 DEBUG:boto:Token: None 2022-09-12T19:02:31.064 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:31.064 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZGXS4wMDP8+nQ74pSnERkYzk+jk= 2022-09-12T19:02:31.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZGXS4wMDP8+nQ74pSnERkYzk+jk=', 'Content-Length': '0'} 2022-09-12T19:02:31.067 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d66db1e79dc24fda-00631f8247-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.114 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:31.115 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:31.115 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:31.116 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:31.116 DEBUG:boto:Method: GET 2022-09-12T19:02:31.117 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:31.118 DEBUG:boto:Data: 2022-09-12T19:02:31.118 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.119 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.119 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.120 DEBUG:boto:Params: {} 2022-09-12T19:02:31.121 DEBUG:boto:Token: None 2022-09-12T19:02:31.121 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:02:31.122 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZGXS4wMDP8+nQ74pSnERkYzk+jk= 2022-09-12T19:02:31.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZGXS4wMDP8+nQ74pSnERkYzk+jk=', 'Content-Length': '0'} 2022-09-12T19:02:31.125 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:23 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090ccfb2abfc9fb80-00631f8247-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.125 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:31.126 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:31.127 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:31.127 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:31.128 DEBUG:boto:Method: GET 2022-09-12T19:02:31.128 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:31.129 DEBUG:boto:Data: 2022-09-12T19:02:31.129 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.130 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:31.130 DEBUG:boto:Port: 8001 2022-09-12T19:02:31.131 DEBUG:boto:Params: {} 2022-09-12T19:02:31.132 DEBUG:boto:Token: None 2022-09-12T19:02:31.132 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/ 2022-09-12T19:02:31.133 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6PbxEADVy5wv3N0+4XxJtG0P048= 2022-09-12T19:02:31.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6PbxEADVy5wv3N0+4XxJtG0P048=', 'Content-Length': '0'} 2022-09-12T19:02:31.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad018d9c4decc363-00631f8247-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.137 DEBUG:boto:b'jdpiak-18obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:31.138 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:31.138 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:31.139 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:31.139 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:31.140 DEBUG:boto:Method: GET 2022-09-12T19:02:31.141 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:02:31.141 DEBUG:boto:Data: 2022-09-12T19:02:31.142 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.142 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.143 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.143 DEBUG:boto:Params: {} 2022-09-12T19:02:31.144 DEBUG:boto:Token: None 2022-09-12T19:02:31.144 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/ 2022-09-12T19:02:31.145 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6PbxEADVy5wv3N0+4XxJtG0P048= 2022-09-12T19:02:31.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6PbxEADVy5wv3N0+4XxJtG0P048=', 'Content-Length': '0'} 2022-09-12T19:02:31.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007de127b49a68e2af-00631f8247-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.149 DEBUG:boto:b'jdpiak-18obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:31.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:02:31.151 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:31.152 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:31.153 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.153 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.154 DEBUG:boto:Method: GET 2022-09-12T19:02:31.155 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.155 DEBUG:boto:Data: 2022-09-12T19:02:31.156 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.156 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:31.157 DEBUG:boto:Port: 8001 2022-09-12T19:02:31.157 DEBUG:boto:Params: {} 2022-09-12T19:02:31.158 DEBUG:boto:Token: None 2022-09-12T19:02:31.159 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.159 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PGB8lbkbZeruPV7wcEDyomKcm08= 2022-09-12T19:02:31.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PGB8lbkbZeruPV7wcEDyomKcm08=', 'Content-Length': '0'} 2022-09-12T19:02:31.162 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea732d61b587c54d-00631f8247-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.163 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:31.163 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:31.164 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.164 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.165 DEBUG:boto:Method: GET 2022-09-12T19:02:31.165 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.166 DEBUG:boto:Data: 2022-09-12T19:02:31.166 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.167 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.168 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.168 DEBUG:boto:Params: {} 2022-09-12T19:02:31.169 DEBUG:boto:Token: None 2022-09-12T19:02:31.170 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.170 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PGB8lbkbZeruPV7wcEDyomKcm08= 2022-09-12T19:02:31.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PGB8lbkbZeruPV7wcEDyomKcm08=', 'Content-Length': '0'} 2022-09-12T19:02:31.173 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9d666fdceb0392c-00631f8247-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.174 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:31.175 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:31.176 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.177 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.178 DEBUG:boto:Method: HEAD 2022-09-12T19:02:31.179 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.180 DEBUG:boto:Data: 2022-09-12T19:02:31.180 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.181 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:31.182 DEBUG:boto:Port: 8001 2022-09-12T19:02:31.183 DEBUG:boto:Params: {} 2022-09-12T19:02:31.184 DEBUG:boto:Token: None 2022-09-12T19:02:31.185 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.186 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gMSe27r5N5I/MTkHqllYIKAZ8Oc= 2022-09-12T19:02:31.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gMSe27r5N5I/MTkHqllYIKAZ8Oc=', 'Content-Length': '0'} 2022-09-12T19:02:31.189 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046ba5ff7d0fcb32f-00631f8247-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.190 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:31.191 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:31.191 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.192 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.192 DEBUG:boto:Method: HEAD 2022-09-12T19:02:31.193 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.193 DEBUG:boto:Data: 2022-09-12T19:02:31.194 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.195 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.195 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.196 DEBUG:boto:Params: {} 2022-09-12T19:02:31.196 DEBUG:boto:Token: None 2022-09-12T19:02:31.198 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.198 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gMSe27r5N5I/MTkHqllYIKAZ8Oc= 2022-09-12T19:02:31.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gMSe27r5N5I/MTkHqllYIKAZ8Oc=', 'Content-Length': '0'} 2022-09-12T19:02:31.202 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001410e921f430200f-00631f8247-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:02:31.204 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:31.205 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:31.206 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.206 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.207 DEBUG:boto:Method: GET 2022-09-12T19:02:31.208 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.208 DEBUG:boto:Data: 2022-09-12T19:02:31.209 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.209 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:31.210 DEBUG:boto:Port: 8001 2022-09-12T19:02:31.210 DEBUG:boto:Params: {} 2022-09-12T19:02:31.211 DEBUG:boto:Token: None 2022-09-12T19:02:31.212 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.212 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PGB8lbkbZeruPV7wcEDyomKcm08= 2022-09-12T19:02:31.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PGB8lbkbZeruPV7wcEDyomKcm08=', 'Content-Length': '0'} 2022-09-12T19:02:31.215 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a3f285247e4f074-00631f8247-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.216 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:31.216 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:31.217 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.217 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.218 DEBUG:boto:Method: GET 2022-09-12T19:02:31.219 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.219 DEBUG:boto:Data: 2022-09-12T19:02:31.220 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.220 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.221 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.221 DEBUG:boto:Params: {} 2022-09-12T19:02:31.222 DEBUG:boto:Token: None 2022-09-12T19:02:31.222 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.223 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PGB8lbkbZeruPV7wcEDyomKcm08= 2022-09-12T19:02:31.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PGB8lbkbZeruPV7wcEDyomKcm08=', 'Content-Length': '0'} 2022-09-12T19:02:31.226 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046997c01f2d42d97-00631f8247-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.227 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:31.227 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:31.228 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:31.229 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:31.229 DEBUG:boto:Method: GET 2022-09-12T19:02:31.230 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:31.230 DEBUG:boto:Data: 2022-09-12T19:02:31.231 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.231 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:31.232 DEBUG:boto:Port: 8001 2022-09-12T19:02:31.232 DEBUG:boto:Params: {} 2022-09-12T19:02:31.233 DEBUG:boto:Token: None 2022-09-12T19:02:31.234 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/ 2022-09-12T19:02:31.234 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6PbxEADVy5wv3N0+4XxJtG0P048= 2022-09-12T19:02:31.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6PbxEADVy5wv3N0+4XxJtG0P048=', 'Content-Length': '0'} 2022-09-12T19:02:31.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068cd6865e7ce133a-00631f8247-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.238 DEBUG:boto:b'jdpiak-18obj-test-zone21000falseobj-test-zone22022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:31.239 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:31.240 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:31.240 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:31.241 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:31.241 DEBUG:boto:Method: GET 2022-09-12T19:02:31.242 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:31.243 DEBUG:boto:Data: 2022-09-12T19:02:31.243 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.244 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.244 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.245 DEBUG:boto:Params: {} 2022-09-12T19:02:31.245 DEBUG:boto:Token: None 2022-09-12T19:02:31.246 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/ 2022-09-12T19:02:31.247 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6PbxEADVy5wv3N0+4XxJtG0P048= 2022-09-12T19:02:31.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6PbxEADVy5wv3N0+4XxJtG0P048=', 'Content-Length': '0'} 2022-09-12T19:02:31.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e93fca6b111f54b-00631f8247-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.298 DEBUG:boto:b'jdpiak-18obj-test-zone21000falseobj-test-zone22022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:31.299 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:31.300 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:31.300 DEBUG:boto:Method: HEAD 2022-09-12T19:02:31.301 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2 2022-09-12T19:02:31.301 DEBUG:boto:Data: 2022-09-12T19:02:31.302 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.302 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:31.303 DEBUG:boto:Port: 8001 2022-09-12T19:02:31.303 DEBUG:boto:Params: {} 2022-09-12T19:02:31.304 DEBUG:boto:Token: None 2022-09-12T19:02:31.304 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/obj-test-zone2 2022-09-12T19:02:31.305 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uJXoPsmkuhC/jLhBG9FSiqFRNVs= 2022-09-12T19:02:31.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uJXoPsmkuhC/jLhBG9FSiqFRNVs=', 'Content-Length': '0'} 2022-09-12T19:02:31.308 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ef3998a99fef1c4-00631f8247-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.309 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:31.310 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:31.310 DEBUG:boto:Method: HEAD 2022-09-12T19:02:31.311 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2 2022-09-12T19:02:31.311 DEBUG:boto:Data: 2022-09-12T19:02:31.312 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.313 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.314 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.314 DEBUG:boto:Params: {} 2022-09-12T19:02:31.315 DEBUG:boto:Token: None 2022-09-12T19:02:31.315 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/obj-test-zone2 2022-09-12T19:02:31.316 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uJXoPsmkuhC/jLhBG9FSiqFRNVs= 2022-09-12T19:02:31.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uJXoPsmkuhC/jLhBG9FSiqFRNVs=', 'Content-Length': '0'} 2022-09-12T19:02:31.319 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030a4ea03f36c6659-00631f8247-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:02:31.321 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:31.322 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:31.322 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.323 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.323 DEBUG:boto:Method: GET 2022-09-12T19:02:31.324 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.324 DEBUG:boto:Data: 2022-09-12T19:02:31.325 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.325 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:31.326 DEBUG:boto:Port: 8001 2022-09-12T19:02:31.326 DEBUG:boto:Params: {} 2022-09-12T19:02:31.327 DEBUG:boto:Token: None 2022-09-12T19:02:31.328 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.328 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PGB8lbkbZeruPV7wcEDyomKcm08= 2022-09-12T19:02:31.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PGB8lbkbZeruPV7wcEDyomKcm08=', 'Content-Length': '0'} 2022-09-12T19:02:31.331 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000904f0a9b2948c15e-00631f8247-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.332 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:31.332 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:31.333 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.333 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.334 DEBUG:boto:Method: GET 2022-09-12T19:02:31.334 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.335 DEBUG:boto:Data: 2022-09-12T19:02:31.335 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.336 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.337 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.338 DEBUG:boto:Params: {} 2022-09-12T19:02:31.339 DEBUG:boto:Token: None 2022-09-12T19:02:31.340 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:02:31.341 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PGB8lbkbZeruPV7wcEDyomKcm08= 2022-09-12T19:02:31.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PGB8lbkbZeruPV7wcEDyomKcm08=', 'Content-Length': '0'} 2022-09-12T19:02:31.344 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d286af88c6c54109-00631f8247-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.346 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:31.347 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:31.347 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:31.348 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:31.349 DEBUG:boto:Method: GET 2022-09-12T19:02:31.350 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:31.351 DEBUG:boto:Data: 2022-09-12T19:02:31.352 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.353 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:31.354 DEBUG:boto:Port: 8001 2022-09-12T19:02:31.355 DEBUG:boto:Params: {} 2022-09-12T19:02:31.355 DEBUG:boto:Token: None 2022-09-12T19:02:31.356 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/ 2022-09-12T19:02:31.357 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6PbxEADVy5wv3N0+4XxJtG0P048= 2022-09-12T19:02:31.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6PbxEADVy5wv3N0+4XxJtG0P048=', 'Content-Length': '0'} 2022-09-12T19:02:31.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0b4c9c197841f78-00631f8247-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.402 DEBUG:boto:b'jdpiak-18obj-test-zone21000falseobj-test-zone22022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:31.403 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:31.404 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:31.405 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:31.405 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:31.406 DEBUG:boto:Method: GET 2022-09-12T19:02:31.406 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:02:31.407 DEBUG:boto:Data: 2022-09-12T19:02:31.408 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.409 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.410 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.411 DEBUG:boto:Params: {} 2022-09-12T19:02:31.412 DEBUG:boto:Token: None 2022-09-12T19:02:31.412 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/ 2022-09-12T19:02:31.413 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6PbxEADVy5wv3N0+4XxJtG0P048= 2022-09-12T19:02:31.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6PbxEADVy5wv3N0+4XxJtG0P048=', 'Content-Length': '0'} 2022-09-12T19:02:31.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d209492d773f415-00631f8247-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.418 DEBUG:boto:b'jdpiak-18obj-test-zone21000falseobj-test-zone22022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:31.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:02:31.420 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:31.421 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:31.422 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.422 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.423 DEBUG:boto:Method: GET 2022-09-12T19:02:31.423 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.424 DEBUG:boto:Data: 2022-09-12T19:02:31.424 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.425 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:31.426 DEBUG:boto:Port: 8001 2022-09-12T19:02:31.426 DEBUG:boto:Params: {} 2022-09-12T19:02:31.427 DEBUG:boto:Token: None 2022-09-12T19:02:31.427 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.428 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:05mR0w8/rKR9csxHQb3FAwaj4qg= 2022-09-12T19:02:31.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:05mR0w8/rKR9csxHQb3FAwaj4qg=', 'Content-Length': '0'} 2022-09-12T19:02:31.431 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000049aba0581eadc76-00631f8247-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.432 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:31.433 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:31.434 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.435 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.436 DEBUG:boto:Method: GET 2022-09-12T19:02:31.437 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.438 DEBUG:boto:Data: 2022-09-12T19:02:31.439 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.440 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.441 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.442 DEBUG:boto:Params: {} 2022-09-12T19:02:31.444 DEBUG:boto:Token: None 2022-09-12T19:02:31.445 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.446 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:05mR0w8/rKR9csxHQb3FAwaj4qg= 2022-09-12T19:02:31.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:05mR0w8/rKR9csxHQb3FAwaj4qg=', 'Content-Length': '0'} 2022-09-12T19:02:31.450 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de7f4a85f3f19172-00631f8247-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.451 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:31.451 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:31.452 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.452 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.453 DEBUG:boto:Method: HEAD 2022-09-12T19:02:31.453 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.454 DEBUG:boto:Data: 2022-09-12T19:02:31.454 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.455 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:31.455 DEBUG:boto:Port: 8001 2022-09-12T19:02:31.456 DEBUG:boto:Params: {} 2022-09-12T19:02:31.456 DEBUG:boto:Token: None 2022-09-12T19:02:31.457 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.457 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vVgB2vUpyOHlqL2LGuRVV48WRLU= 2022-09-12T19:02:31.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vVgB2vUpyOHlqL2LGuRVV48WRLU=', 'Content-Length': '0'} 2022-09-12T19:02:31.460 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d979a9ba1f08eef9-00631f8247-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.461 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:31.461 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:31.462 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.462 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.463 DEBUG:boto:Method: HEAD 2022-09-12T19:02:31.463 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.464 DEBUG:boto:Data: 2022-09-12T19:02:31.464 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.465 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.465 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.466 DEBUG:boto:Params: {} 2022-09-12T19:02:31.467 DEBUG:boto:Token: None 2022-09-12T19:02:31.467 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.468 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vVgB2vUpyOHlqL2LGuRVV48WRLU= 2022-09-12T19:02:31.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vVgB2vUpyOHlqL2LGuRVV48WRLU=', 'Content-Length': '0'} 2022-09-12T19:02:31.471 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e55535fe654652ca-00631f8247-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:02:31.472 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:31.473 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:31.473 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.474 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.474 DEBUG:boto:Method: GET 2022-09-12T19:02:31.475 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.475 DEBUG:boto:Data: 2022-09-12T19:02:31.476 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.476 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:31.477 DEBUG:boto:Port: 8001 2022-09-12T19:02:31.477 DEBUG:boto:Params: {} 2022-09-12T19:02:31.478 DEBUG:boto:Token: None 2022-09-12T19:02:31.479 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.479 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:05mR0w8/rKR9csxHQb3FAwaj4qg= 2022-09-12T19:02:31.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:05mR0w8/rKR9csxHQb3FAwaj4qg=', 'Content-Length': '0'} 2022-09-12T19:02:31.482 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d94ae02e2069854-00631f8247-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.483 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:31.484 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:31.484 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.485 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.485 DEBUG:boto:Method: GET 2022-09-12T19:02:31.486 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.487 DEBUG:boto:Data: 2022-09-12T19:02:31.487 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.488 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.488 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.489 DEBUG:boto:Params: {} 2022-09-12T19:02:31.489 DEBUG:boto:Token: None 2022-09-12T19:02:31.490 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.491 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:05mR0w8/rKR9csxHQb3FAwaj4qg= 2022-09-12T19:02:31.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:05mR0w8/rKR9csxHQb3FAwaj4qg=', 'Content-Length': '0'} 2022-09-12T19:02:31.493 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f879e8b6b556bae8-00631f8247-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.534 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:31.535 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:31.535 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:31.536 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:31.536 DEBUG:boto:Method: GET 2022-09-12T19:02:31.537 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:31.538 DEBUG:boto:Data: 2022-09-12T19:02:31.538 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.539 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:31.539 DEBUG:boto:Port: 8001 2022-09-12T19:02:31.540 DEBUG:boto:Params: {} 2022-09-12T19:02:31.541 DEBUG:boto:Token: None 2022-09-12T19:02:31.541 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/ 2022-09-12T19:02:31.542 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6PbxEADVy5wv3N0+4XxJtG0P048= 2022-09-12T19:02:31.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6PbxEADVy5wv3N0+4XxJtG0P048=', 'Content-Length': '0'} 2022-09-12T19:02:31.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002745a1557bb00ecb-00631f8247-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.546 DEBUG:boto:b'jdpiak-18obj-test-zone31000falseobj-test-zone32022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:31.547 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:31.548 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:31.549 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:31.549 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:31.550 DEBUG:boto:Method: GET 2022-09-12T19:02:31.550 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:31.551 DEBUG:boto:Data: 2022-09-12T19:02:31.552 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.552 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.553 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.553 DEBUG:boto:Params: {} 2022-09-12T19:02:31.554 DEBUG:boto:Token: None 2022-09-12T19:02:31.555 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/ 2022-09-12T19:02:31.555 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6PbxEADVy5wv3N0+4XxJtG0P048= 2022-09-12T19:02:31.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6PbxEADVy5wv3N0+4XxJtG0P048=', 'Content-Length': '0'} 2022-09-12T19:02:31.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001bb37b94da209e6-00631f8247-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.559 DEBUG:boto:b'jdpiak-18obj-test-zone31000falseobj-test-zone32022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:31.560 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:31.561 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:31.561 DEBUG:boto:Method: HEAD 2022-09-12T19:02:31.562 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3 2022-09-12T19:02:31.562 DEBUG:boto:Data: 2022-09-12T19:02:31.563 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.563 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:31.564 DEBUG:boto:Port: 8001 2022-09-12T19:02:31.564 DEBUG:boto:Params: {} 2022-09-12T19:02:31.565 DEBUG:boto:Token: None 2022-09-12T19:02:31.565 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/obj-test-zone3 2022-09-12T19:02:31.566 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XGDFJE5imcuRiKXMTAu+wJlfdi8= 2022-09-12T19:02:31.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XGDFJE5imcuRiKXMTAu+wJlfdi8=', 'Content-Length': '0'} 2022-09-12T19:02:31.570 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024d1e57c0d1b72e8-00631f8247-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.571 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:31.572 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:31.573 DEBUG:boto:Method: HEAD 2022-09-12T19:02:31.574 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3 2022-09-12T19:02:31.575 DEBUG:boto:Data: 2022-09-12T19:02:31.575 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.576 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.577 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.578 DEBUG:boto:Params: {} 2022-09-12T19:02:31.579 DEBUG:boto:Token: None 2022-09-12T19:02:31.580 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/obj-test-zone3 2022-09-12T19:02:31.581 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XGDFJE5imcuRiKXMTAu+wJlfdi8= 2022-09-12T19:02:31.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XGDFJE5imcuRiKXMTAu+wJlfdi8=', 'Content-Length': '0'} 2022-09-12T19:02:31.585 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff2b996e1221bd51-00631f8247-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:02:31.586 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:31.587 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:31.587 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.588 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.588 DEBUG:boto:Method: GET 2022-09-12T19:02:31.589 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.590 DEBUG:boto:Data: 2022-09-12T19:02:31.590 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.591 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:31.592 DEBUG:boto:Port: 8001 2022-09-12T19:02:31.593 DEBUG:boto:Params: {} 2022-09-12T19:02:31.593 DEBUG:boto:Token: None 2022-09-12T19:02:31.594 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.594 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:05mR0w8/rKR9csxHQb3FAwaj4qg= 2022-09-12T19:02:31.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:05mR0w8/rKR9csxHQb3FAwaj4qg=', 'Content-Length': '0'} 2022-09-12T19:02:31.597 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003ba12d5cb6cf7af-00631f8247-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.638 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:31.639 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:31.639 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.640 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.640 DEBUG:boto:Method: GET 2022-09-12T19:02:31.641 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.642 DEBUG:boto:Data: 2022-09-12T19:02:31.642 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.643 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.643 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.644 DEBUG:boto:Params: {} 2022-09-12T19:02:31.644 DEBUG:boto:Token: None 2022-09-12T19:02:31.645 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:02:31.646 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:05mR0w8/rKR9csxHQb3FAwaj4qg= 2022-09-12T19:02:31.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:05mR0w8/rKR9csxHQb3FAwaj4qg=', 'Content-Length': '0'} 2022-09-12T19:02:31.648 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:59:24 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9d0ff45490d7871-00631f8247-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.649 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:31.650 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:31.650 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:31.651 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:31.651 DEBUG:boto:Method: GET 2022-09-12T19:02:31.652 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:31.652 DEBUG:boto:Data: 2022-09-12T19:02:31.653 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.653 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:31.654 DEBUG:boto:Port: 8001 2022-09-12T19:02:31.655 DEBUG:boto:Params: {} 2022-09-12T19:02:31.655 DEBUG:boto:Token: None 2022-09-12T19:02:31.656 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/ 2022-09-12T19:02:31.656 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6PbxEADVy5wv3N0+4XxJtG0P048= 2022-09-12T19:02:31.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6PbxEADVy5wv3N0+4XxJtG0P048=', 'Content-Length': '0'} 2022-09-12T19:02:31.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000706847535aa6142e-00631f8247-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.660 DEBUG:boto:b'jdpiak-18obj-test-zone31000falseobj-test-zone32022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:31.661 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:31.662 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:31.662 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:31.663 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:31.664 DEBUG:boto:Method: GET 2022-09-12T19:02:31.664 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:02:31.665 DEBUG:boto:Data: 2022-09-12T19:02:31.665 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.666 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.666 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.667 DEBUG:boto:Params: {} 2022-09-12T19:02:31.668 DEBUG:boto:Token: None 2022-09-12T19:02:31.668 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/ 2022-09-12T19:02:31.669 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6PbxEADVy5wv3N0+4XxJtG0P048= 2022-09-12T19:02:31.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6PbxEADVy5wv3N0+4XxJtG0P048=', 'Content-Length': '0'} 2022-09-12T19:02:31.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037d8746de826ae60-00631f8247-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.673 DEBUG:boto:b'jdpiak-18obj-test-zone31000falseobj-test-zone32022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:31.674 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-18 zones={test-zone3, test-zone2} 2022-09-12T19:02:31.674 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:02:31.675 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:02:31.675 DEBUG:boto:Method: HEAD 2022-09-12T19:02:31.676 DEBUG:boto:Path: /jdpiak-16/ 2022-09-12T19:02:31.676 DEBUG:boto:Data: 2022-09-12T19:02:31.677 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.677 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:31.678 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.678 DEBUG:boto:Params: {} 2022-09-12T19:02:31.679 DEBUG:boto:Token: None 2022-09-12T19:02:31.679 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-16/ 2022-09-12T19:02:31.680 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AISraSmCEf404cVM2YN1qWNHCwI= 2022-09-12T19:02:31.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AISraSmCEf404cVM2YN1qWNHCwI=', 'Content-Length': '0'} 2022-09-12T19:02:31.683 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064ec290a8545ec4c-00631f8247-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.684 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:02:31.685 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:02:31.685 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-12T19:02:31.686 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-12T19:02:31.686 DEBUG:boto:Method: GET 2022-09-12T19:02:31.687 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-12T19:02:31.687 DEBUG:boto:Data: 2022-09-12T19:02:31.688 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.688 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:31.689 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.689 DEBUG:boto:Params: {} 2022-09-12T19:02:31.690 DEBUG:boto:Token: None 2022-09-12T19:02:31.690 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-16/?versions 2022-09-12T19:02:31.691 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PrA2+6I0sf7KrRHqV5dWCq3ro58= 2022-09-12T19:02:31.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PrA2+6I0sf7KrRHqV5dWCq3ro58=', 'Content-Length': '0'} 2022-09-12T19:02:31.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfc53852fad82699-00631f8247-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.695 DEBUG:boto:b'jdpiak-16obj-test-zone1999falseobj-test-zone1-zshRkRrxKU0EJVPpVBUsfYl8JR7HFRtrue2022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:31.696 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:02:31.696 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:02:31.697 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?acl 2022-09-12T19:02:31.697 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?acl 2022-09-12T19:02:31.698 DEBUG:boto:Method: PUT 2022-09-12T19:02:31.698 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?acl 2022-09-12T19:02:31.699 DEBUG:boto:Data: 2022-09-12T19:02:31.699 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-12T19:02:31.700 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:31.700 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.701 DEBUG:boto:Params: {} 2022-09-12T19:02:31.701 DEBUG:boto:Token: None 2022-09-12T19:02:31.702 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:02:31 GMT x-amz-acl:public-read /jdpiak-16/obj-test-zone1?acl 2022-09-12T19:02:31.702 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/ZYbhB4ulsqjTskKdOqRrSE16Uc= 2022-09-12T19:02:31.703 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/ZYbhB4ulsqjTskKdOqRrSE16Uc=', 'Content-Length': '0'} 2022-09-12T19:02:31.711 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e68a5ee272911395-00631f8247-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.712 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:02:31.712 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:02:31.713 DEBUG:boto:Method: HEAD 2022-09-12T19:02:31.713 DEBUG:boto:Path: /jdpiak-16/ 2022-09-12T19:02:31.714 DEBUG:boto:Data: 2022-09-12T19:02:31.714 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.715 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.715 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.716 DEBUG:boto:Params: {} 2022-09-12T19:02:31.716 DEBUG:boto:Token: None 2022-09-12T19:02:31.717 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-16/ 2022-09-12T19:02:31.717 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AISraSmCEf404cVM2YN1qWNHCwI= 2022-09-12T19:02:31.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AISraSmCEf404cVM2YN1qWNHCwI=', 'Content-Length': '0'} 2022-09-12T19:02:31.720 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb4e6c88a638fc74-00631f8247-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.721 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:02:31.722 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:02:31.722 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-12T19:02:31.723 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-12T19:02:31.723 DEBUG:boto:Method: GET 2022-09-12T19:02:31.724 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-12T19:02:31.724 DEBUG:boto:Data: 2022-09-12T19:02:31.725 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.725 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.726 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.726 DEBUG:boto:Params: {} 2022-09-12T19:02:31.727 DEBUG:boto:Token: None 2022-09-12T19:02:31.727 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-16/?versions 2022-09-12T19:02:31.728 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PrA2+6I0sf7KrRHqV5dWCq3ro58= 2022-09-12T19:02:31.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PrA2+6I0sf7KrRHqV5dWCq3ro58=', 'Content-Length': '0'} 2022-09-12T19:02:31.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030e03d82e04c864c-00631f8247-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.732 DEBUG:boto:b'jdpiak-16obj-test-zone2999falseobj-test-zone23vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5true2022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:31.734 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:02:31.735 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:02:31.735 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?acl 2022-09-12T19:02:31.736 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?acl 2022-09-12T19:02:31.737 DEBUG:boto:Method: PUT 2022-09-12T19:02:31.738 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?acl 2022-09-12T19:02:31.738 DEBUG:boto:Data: 2022-09-12T19:02:31.739 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-12T19:02:31.739 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.740 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.740 DEBUG:boto:Params: {} 2022-09-12T19:02:31.741 DEBUG:boto:Token: None 2022-09-12T19:02:31.741 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:02:31 GMT x-amz-acl:public-read /jdpiak-16/obj-test-zone2?acl 2022-09-12T19:02:31.742 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m1cE06K05P47yaS571GEUf7CNok= 2022-09-12T19:02:31.743 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m1cE06K05P47yaS571GEUf7CNok=', 'Content-Length': '0'} 2022-09-12T19:02:31.755 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b5fb61343d6abe88-00631f8247-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.755 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:02:31.756 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:02:31.757 DEBUG:boto:Method: HEAD 2022-09-12T19:02:31.757 DEBUG:boto:Path: /jdpiak-16/ 2022-09-12T19:02:31.758 DEBUG:boto:Data: 2022-09-12T19:02:31.758 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.759 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:31.759 DEBUG:boto:Port: 8001 2022-09-12T19:02:31.760 DEBUG:boto:Params: {} 2022-09-12T19:02:31.760 DEBUG:boto:Token: None 2022-09-12T19:02:31.761 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-16/ 2022-09-12T19:02:31.761 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AISraSmCEf404cVM2YN1qWNHCwI= 2022-09-12T19:02:31.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AISraSmCEf404cVM2YN1qWNHCwI=', 'Content-Length': '0'} 2022-09-12T19:02:31.765 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e19f3d9ab49197b9-00631f8247-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.765 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:02:31.766 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:02:31.766 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-12T19:02:31.767 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-12T19:02:31.767 DEBUG:boto:Method: GET 2022-09-12T19:02:31.768 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-12T19:02:31.768 DEBUG:boto:Data: 2022-09-12T19:02:31.769 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.769 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:31.770 DEBUG:boto:Port: 8001 2022-09-12T19:02:31.770 DEBUG:boto:Params: {} 2022-09-12T19:02:31.771 DEBUG:boto:Token: None 2022-09-12T19:02:31.772 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-16/?versions 2022-09-12T19:02:31.772 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PrA2+6I0sf7KrRHqV5dWCq3ro58= 2022-09-12T19:02:31.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PrA2+6I0sf7KrRHqV5dWCq3ro58=', 'Content-Length': '0'} 2022-09-12T19:02:31.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e98ee275a14d804-00631f8247-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.776 DEBUG:boto:b'jdpiak-16obj-test-zone3999falseobj-test-zone3RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eEtrue2022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:31.777 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:02:31.777 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:02:31.778 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?acl 2022-09-12T19:02:31.778 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?acl 2022-09-12T19:02:31.779 DEBUG:boto:Method: PUT 2022-09-12T19:02:31.779 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?acl 2022-09-12T19:02:31.779 DEBUG:boto:Data: 2022-09-12T19:02:31.780 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-12T19:02:31.780 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:31.781 DEBUG:boto:Port: 8001 2022-09-12T19:02:31.781 DEBUG:boto:Params: {} 2022-09-12T19:02:31.782 DEBUG:boto:Token: None 2022-09-12T19:02:31.783 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:02:31 GMT x-amz-acl:public-read /jdpiak-16/obj-test-zone3?acl 2022-09-12T19:02:31.783 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CUtYVQ8gKiorWJ8exBlTxitTejI= 2022-09-12T19:02:31.784 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CUtYVQ8gKiorWJ8exBlTxitTejI=', 'Content-Length': '0'} 2022-09-12T19:02:31.791 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a7e9f280c0ba9411-00631f8247-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.792 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:31.792 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:31.793 DEBUG:boto:Method: HEAD 2022-09-12T19:02:31.793 DEBUG:boto:Path: /jdpiak-17/ 2022-09-12T19:02:31.794 DEBUG:boto:Data: 2022-09-12T19:02:31.794 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.795 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:31.795 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.796 DEBUG:boto:Params: {} 2022-09-12T19:02:31.796 DEBUG:boto:Token: None 2022-09-12T19:02:31.797 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-17/ 2022-09-12T19:02:31.797 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pH+Vhysh/GhcyFn0xIifYFi++IA= 2022-09-12T19:02:31.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pH+Vhysh/GhcyFn0xIifYFi++IA=', 'Content-Length': '0'} 2022-09-12T19:02:31.801 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004af3f09cfea537dc-00631f8247-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.801 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:31.802 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:31.802 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-12T19:02:31.803 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-12T19:02:31.810 DEBUG:boto:Method: GET 2022-09-12T19:02:31.811 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-12T19:02:31.811 DEBUG:boto:Data: 2022-09-12T19:02:31.812 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.812 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:31.813 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.813 DEBUG:boto:Params: {} 2022-09-12T19:02:31.814 DEBUG:boto:Token: None 2022-09-12T19:02:31.815 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-17/?versions 2022-09-12T19:02:31.815 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LFoT1r59b10pfZIYuMBJug+mKt4= 2022-09-12T19:02:31.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LFoT1r59b10pfZIYuMBJug+mKt4=', 'Content-Length': '0'} 2022-09-12T19:02:31.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002686c9af1c2b8466-00631f8247-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.819 DEBUG:boto:b'jdpiak-17obj-test-zone1999falseobj-test-zone1Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpktrue2022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:31.821 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:31.821 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:02:31.822 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?acl 2022-09-12T19:02:31.823 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?acl 2022-09-12T19:02:31.824 DEBUG:boto:Method: PUT 2022-09-12T19:02:31.825 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?acl 2022-09-12T19:02:31.826 DEBUG:boto:Data: 2022-09-12T19:02:31.826 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-12T19:02:31.827 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:31.827 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.827 DEBUG:boto:Params: {} 2022-09-12T19:02:31.828 DEBUG:boto:Token: None 2022-09-12T19:02:31.829 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:02:31 GMT x-amz-acl:public-read /jdpiak-17/obj-test-zone1?acl 2022-09-12T19:02:31.829 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0h68NGHGHwKWwhb2hhS8JSN5GcA= 2022-09-12T19:02:31.830 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0h68NGHGHwKWwhb2hhS8JSN5GcA=', 'Content-Length': '0'} 2022-09-12T19:02:31.837 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000da96cb75d9b7a2d0-00631f8247-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.838 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:31.839 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:31.839 DEBUG:boto:Method: HEAD 2022-09-12T19:02:31.839 DEBUG:boto:Path: /jdpiak-17/ 2022-09-12T19:02:31.840 DEBUG:boto:Data: 2022-09-12T19:02:31.840 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.841 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.841 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.842 DEBUG:boto:Params: {} 2022-09-12T19:02:31.842 DEBUG:boto:Token: None 2022-09-12T19:02:31.843 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-17/ 2022-09-12T19:02:31.844 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pH+Vhysh/GhcyFn0xIifYFi++IA= 2022-09-12T19:02:31.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pH+Vhysh/GhcyFn0xIifYFi++IA=', 'Content-Length': '0'} 2022-09-12T19:02:31.846 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005cca4dead85b2cb4-00631f8247-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.847 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:31.848 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:31.848 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-12T19:02:31.849 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-12T19:02:31.849 DEBUG:boto:Method: GET 2022-09-12T19:02:31.850 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-12T19:02:31.850 DEBUG:boto:Data: 2022-09-12T19:02:31.851 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.851 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.852 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.853 DEBUG:boto:Params: {} 2022-09-12T19:02:31.853 DEBUG:boto:Token: None 2022-09-12T19:02:31.854 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-17/?versions 2022-09-12T19:02:31.854 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LFoT1r59b10pfZIYuMBJug+mKt4= 2022-09-12T19:02:31.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LFoT1r59b10pfZIYuMBJug+mKt4=', 'Content-Length': '0'} 2022-09-12T19:02:31.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060e0f3f51dde8f6f-00631f8247-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.860 DEBUG:boto:b'jdpiak-17obj-test-zone2999falseobj-test-zone20f1rsOoGVVMRuyy11YfWbutRvo943lCtrue2022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:31.861 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:31.862 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:02:31.863 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?acl 2022-09-12T19:02:31.863 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?acl 2022-09-12T19:02:31.864 DEBUG:boto:Method: PUT 2022-09-12T19:02:31.865 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?acl 2022-09-12T19:02:31.866 DEBUG:boto:Data: 2022-09-12T19:02:31.866 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-12T19:02:31.867 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.867 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.868 DEBUG:boto:Params: {} 2022-09-12T19:02:31.868 DEBUG:boto:Token: None 2022-09-12T19:02:31.869 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:02:31 GMT x-amz-acl:public-read /jdpiak-17/obj-test-zone2?acl 2022-09-12T19:02:31.869 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KEm+pCNnHFWVXLxgx2Da+XRsBHg= 2022-09-12T19:02:31.870 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KEm+pCNnHFWVXLxgx2Da+XRsBHg=', 'Content-Length': '0'} 2022-09-12T19:02:31.879 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000081224e69620cb559-00631f8247-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.880 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:31.881 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:31.881 DEBUG:boto:Method: HEAD 2022-09-12T19:02:31.882 DEBUG:boto:Path: /jdpiak-17/ 2022-09-12T19:02:31.882 DEBUG:boto:Data: 2022-09-12T19:02:31.883 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.883 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:31.884 DEBUG:boto:Port: 8001 2022-09-12T19:02:31.884 DEBUG:boto:Params: {} 2022-09-12T19:02:31.885 DEBUG:boto:Token: None 2022-09-12T19:02:31.885 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-17/ 2022-09-12T19:02:31.886 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pH+Vhysh/GhcyFn0xIifYFi++IA= 2022-09-12T19:02:31.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pH+Vhysh/GhcyFn0xIifYFi++IA=', 'Content-Length': '0'} 2022-09-12T19:02:31.889 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001344eaf65628642b-00631f8247-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.890 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:02:31.891 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:02:31.891 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-12T19:02:31.892 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-12T19:02:31.892 DEBUG:boto:Method: GET 2022-09-12T19:02:31.893 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-12T19:02:31.893 DEBUG:boto:Data: 2022-09-12T19:02:31.894 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.894 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:31.895 DEBUG:boto:Port: 8001 2022-09-12T19:02:31.895 DEBUG:boto:Params: {} 2022-09-12T19:02:31.896 DEBUG:boto:Token: None 2022-09-12T19:02:31.896 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-17/?versions 2022-09-12T19:02:31.897 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LFoT1r59b10pfZIYuMBJug+mKt4= 2022-09-12T19:02:31.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LFoT1r59b10pfZIYuMBJug+mKt4=', 'Content-Length': '0'} 2022-09-12T19:02:31.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004262c6b4bc3d185d-00631f8247-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.901 DEBUG:boto:b'jdpiak-17obj-test-zone3999falseobj-test-zone3V6Ue4vhVY0WQ9FArGixLjY-hmND5K20true2022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:31.902 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:31.902 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:02:31.903 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?acl 2022-09-12T19:02:31.903 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?acl 2022-09-12T19:02:31.904 DEBUG:boto:Method: PUT 2022-09-12T19:02:31.904 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?acl 2022-09-12T19:02:31.905 DEBUG:boto:Data: 2022-09-12T19:02:31.905 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-12T19:02:31.906 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:31.906 DEBUG:boto:Port: 8001 2022-09-12T19:02:31.907 DEBUG:boto:Params: {} 2022-09-12T19:02:31.907 DEBUG:boto:Token: None 2022-09-12T19:02:31.908 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:02:31 GMT x-amz-acl:public-read /jdpiak-17/obj-test-zone3?acl 2022-09-12T19:02:31.909 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GV+REhqvxP1SLoXHbvNlgPBIwAg= 2022-09-12T19:02:31.909 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GV+REhqvxP1SLoXHbvNlgPBIwAg=', 'Content-Length': '0'} 2022-09-12T19:02:31.917 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000fd005f5852eab55-00631f8247-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.917 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:31.918 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:31.918 DEBUG:boto:Method: HEAD 2022-09-12T19:02:31.919 DEBUG:boto:Path: /jdpiak-18/ 2022-09-12T19:02:31.919 DEBUG:boto:Data: 2022-09-12T19:02:31.920 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.920 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:31.921 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.921 DEBUG:boto:Params: {} 2022-09-12T19:02:31.922 DEBUG:boto:Token: None 2022-09-12T19:02:31.923 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/ 2022-09-12T19:02:31.923 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qqyQ+V+Xy2yzntSUJvhWm3ghQ2s= 2022-09-12T19:02:31.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qqyQ+V+Xy2yzntSUJvhWm3ghQ2s=', 'Content-Length': '0'} 2022-09-12T19:02:31.926 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c16a0a7dac0897a8-00631f8247-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.927 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:31.927 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:31.928 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-12T19:02:31.928 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-12T19:02:31.929 DEBUG:boto:Method: GET 2022-09-12T19:02:31.929 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-12T19:02:31.930 DEBUG:boto:Data: 2022-09-12T19:02:31.930 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.931 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:31.931 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.932 DEBUG:boto:Params: {} 2022-09-12T19:02:31.932 DEBUG:boto:Token: None 2022-09-12T19:02:31.933 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/?versions 2022-09-12T19:02:31.933 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3pt1u4l9AUy7DMOeN7NKUbnKWSc= 2022-09-12T19:02:31.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3pt1u4l9AUy7DMOeN7NKUbnKWSc=', 'Content-Length': '0'} 2022-09-12T19:02:31.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046dcedb809832e78-00631f8247-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.938 DEBUG:boto:b'jdpiak-18obj-test-zone1999falseobj-test-zone1ToUtj2XoS.tz3goIbvjFwiigMyzGOpLtrue2022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:31.939 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:31.939 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:02:31.940 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?acl 2022-09-12T19:02:31.940 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?acl 2022-09-12T19:02:31.940 DEBUG:boto:Method: PUT 2022-09-12T19:02:31.941 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?acl 2022-09-12T19:02:31.941 DEBUG:boto:Data: 2022-09-12T19:02:31.942 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-12T19:02:31.942 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:02:31.943 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.943 DEBUG:boto:Params: {} 2022-09-12T19:02:31.944 DEBUG:boto:Token: None 2022-09-12T19:02:31.944 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:02:31 GMT x-amz-acl:public-read /jdpiak-18/obj-test-zone1?acl 2022-09-12T19:02:31.945 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sEDhTQHKn0TvcEPXu+6Ed2fZdDk= 2022-09-12T19:02:31.945 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sEDhTQHKn0TvcEPXu+6Ed2fZdDk=', 'Content-Length': '0'} 2022-09-12T19:02:31.953 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bfca1a7a5bfb28d3-00631f8247-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.954 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:31.954 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:31.955 DEBUG:boto:Method: HEAD 2022-09-12T19:02:31.955 DEBUG:boto:Path: /jdpiak-18/ 2022-09-12T19:02:31.956 DEBUG:boto:Data: 2022-09-12T19:02:31.956 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.957 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.957 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.957 DEBUG:boto:Params: {} 2022-09-12T19:02:31.958 DEBUG:boto:Token: None 2022-09-12T19:02:31.959 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/ 2022-09-12T19:02:31.959 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qqyQ+V+Xy2yzntSUJvhWm3ghQ2s= 2022-09-12T19:02:31.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qqyQ+V+Xy2yzntSUJvhWm3ghQ2s=', 'Content-Length': '0'} 2022-09-12T19:02:31.962 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f00827948ea036c5-00631f8247-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:31.963 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:31.964 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:31.964 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-12T19:02:31.964 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-12T19:02:31.965 DEBUG:boto:Method: GET 2022-09-12T19:02:31.965 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-12T19:02:31.966 DEBUG:boto:Data: 2022-09-12T19:02:31.967 DEBUG:boto:Headers: {} 2022-09-12T19:02:31.967 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:31.967 DEBUG:boto:Port: 8000 2022-09-12T19:02:31.968 DEBUG:boto:Params: {} 2022-09-12T19:02:31.969 DEBUG:boto:Token: None 2022-09-12T19:02:31.969 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:31 GMT /jdpiak-18/?versions 2022-09-12T19:02:31.970 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3pt1u4l9AUy7DMOeN7NKUbnKWSc= 2022-09-12T19:02:31.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3pt1u4l9AUy7DMOeN7NKUbnKWSc=', 'Content-Length': '0'} 2022-09-12T19:02:31.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f48a6d0b85e93bf-00631f8247-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:32.014 DEBUG:boto:b'jdpiak-18obj-test-zone2999falseobj-test-zone2DPksivOWVYhJDrEwQqHYV7wK1Ul1JkWtrue2022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:32.015 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:32.016 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:02:32.016 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?acl 2022-09-12T19:02:32.016 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?acl 2022-09-12T19:02:32.017 DEBUG:boto:Method: PUT 2022-09-12T19:02:32.017 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?acl 2022-09-12T19:02:32.018 DEBUG:boto:Data: 2022-09-12T19:02:32.019 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-12T19:02:32.019 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:02:32.020 DEBUG:boto:Port: 8000 2022-09-12T19:02:32.020 DEBUG:boto:Params: {} 2022-09-12T19:02:32.021 DEBUG:boto:Token: None 2022-09-12T19:02:32.021 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:02:32 GMT x-amz-acl:public-read /jdpiak-18/obj-test-zone2?acl 2022-09-12T19:02:32.022 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iswUfhfUnb1a16WkbPf0wPUYDJE= 2022-09-12T19:02:32.022 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iswUfhfUnb1a16WkbPf0wPUYDJE=', 'Content-Length': '0'} 2022-09-12T19:02:32.032 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cec40f1b986ad506-00631f8248-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:32.033 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:32.033 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:32.034 DEBUG:boto:Method: HEAD 2022-09-12T19:02:32.034 DEBUG:boto:Path: /jdpiak-18/ 2022-09-12T19:02:32.035 DEBUG:boto:Data: 2022-09-12T19:02:32.035 DEBUG:boto:Headers: {} 2022-09-12T19:02:32.036 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:32.036 DEBUG:boto:Port: 8001 2022-09-12T19:02:32.037 DEBUG:boto:Params: {} 2022-09-12T19:02:32.037 DEBUG:boto:Token: None 2022-09-12T19:02:32.038 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:02:32 GMT /jdpiak-18/ 2022-09-12T19:02:32.038 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OXnLeshVyOyBbkS565/OtJZj4ns= 2022-09-12T19:02:32.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OXnLeshVyOyBbkS565/OtJZj4ns=', 'Content-Length': '0'} 2022-09-12T19:02:32.042 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000584da00bbf57c8a4-00631f8248-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:32.042 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:02:32.043 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:02:32.043 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-12T19:02:32.044 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-12T19:02:32.044 DEBUG:boto:Method: GET 2022-09-12T19:02:32.045 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-12T19:02:32.045 DEBUG:boto:Data: 2022-09-12T19:02:32.046 DEBUG:boto:Headers: {} 2022-09-12T19:02:32.046 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:32.047 DEBUG:boto:Port: 8001 2022-09-12T19:02:32.047 DEBUG:boto:Params: {} 2022-09-12T19:02:32.048 DEBUG:boto:Token: None 2022-09-12T19:02:32.049 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:02:32 GMT /jdpiak-18/?versions 2022-09-12T19:02:32.049 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X4M7hM4tkb2RMXyJyC2B1bi2U1I= 2022-09-12T19:02:32.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X4M7hM4tkb2RMXyJyC2B1bi2U1I=', 'Content-Length': '0'} 2022-09-12T19:02:32.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ba24fda4bc9abb8-00631f8248-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:32.053 DEBUG:boto:b'jdpiak-18obj-test-zone3999falseobj-test-zone3iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Ztrue2022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:02:32.054 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:32.055 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:02:32.055 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?acl 2022-09-12T19:02:32.056 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?acl 2022-09-12T19:02:32.056 DEBUG:boto:Method: PUT 2022-09-12T19:02:32.057 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?acl 2022-09-12T19:02:32.057 DEBUG:boto:Data: 2022-09-12T19:02:32.058 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-12T19:02:32.058 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:02:32.059 DEBUG:boto:Port: 8001 2022-09-12T19:02:32.059 DEBUG:boto:Params: {} 2022-09-12T19:02:32.060 DEBUG:boto:Token: None 2022-09-12T19:02:32.060 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:02:32 GMT x-amz-acl:public-read /jdpiak-18/obj-test-zone3?acl 2022-09-12T19:02:32.061 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X+ywLFrgD1abQ7cpCvdW1M/DA9c= 2022-09-12T19:02:32.062 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:02:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X+ywLFrgD1abQ7cpCvdW1M/DA9c=', 'Content-Length': '0'} 2022-09-12T19:02:32.069 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ccf112cf9a53012f-00631f8248-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:02:32.070 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-16 2022-09-12T19:02:32.070 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:02:32.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:02:32.071 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:02:32.134 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:02:32.134 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:02:32.191 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:02:32.188+0000 7f8d8f92a600 1 waiting for incremental sync to catch up: 2022-09-12T19:02:32.191 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000010.20.5, , 00000000003.9.9, , 00000000008.19.9, , , , , ] 2022-09-12T19:02:32.191 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000011.23.5, , 00000000003.9.9, , 00000000008.19.9, , , , , ] 2022-09-12T19:02:37.192 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:02:37.189+0000 7f8d8f92a600 1 waiting for incremental sync to catch up: 2022-09-12T19:02:37.193 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000010.20.5, , 00000000003.9.9, , 00000000008.19.9, , , , , ] 2022-09-12T19:02:37.193 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000011.23.5, , 00000000003.9.9, , 00000000008.19.9, , , , , ] 2022-09-12T19:02:42.194 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:02:42.190+0000 7f8d8f92a600 1 waiting for incremental sync to catch up: 2022-09-12T19:02:42.194 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000010.20.5, , 00000000003.9.9, , 00000000009.22.5, , , , , ] 2022-09-12T19:02:42.195 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000011.23.5, , 00000000003.9.9, , 00000000008.19.9, , , , , ] 2022-09-12T19:02:47.195 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:02:47.192+0000 7f8d8f92a600 1 bucket sync caught up with source: 2022-09-12T19:02:47.196 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000011.23.5, , 00000000003.9.9, , 00000000009.22.5, , , , , ] 2022-09-12T19:02:47.196 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000011.23.5, , 00000000003.9.9, , 00000000008.19.9, , , , , ] 2022-09-12T19:02:47.196 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:02:47.192+0000 7f8d8f92a600 0 bucket checkpoint complete 2022-09-12T19:02:47.214 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-16 2022-09-12T19:02:47.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:02:47.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:02:47.217 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:02:47.282 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:02:47.283 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:02:47.320 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:02:47.318+0000 7f1d65175600 1 bucket sync caught up with source: 2022-09-12T19:02:47.320 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000011.23.5, , 00000000003.9.9, , 00000000009.22.5, , , , , ] 2022-09-12T19:02:47.320 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000011.23.5, , 00000000003.9.9, , 00000000009.22.5, , , , , ] 2022-09-12T19:02:47.321 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:02:47.318+0000 7f1d65175600 0 bucket checkpoint complete 2022-09-12T19:02:47.333 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-16 2022-09-12T19:02:47.334 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:02:47.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:02:47.335 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:02:47.397 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:02:47.398 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:02:47.442 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:02:47.439+0000 7fc089701600 1 bucket sync caught up with source: 2022-09-12T19:02:47.443 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000011.23.5, , 00000000003.9.9, , 00000000009.22.5, , , , , ] 2022-09-12T19:02:47.443 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000011.23.5, , 00000000003.9.9, , 00000000009.22.5, , , , , ] 2022-09-12T19:02:47.443 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:02:47.439+0000 7fc089701600 0 bucket checkpoint complete 2022-09-12T19:02:47.457 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-16 2022-09-12T19:02:47.458 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:02:47.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:02:47.459 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:02:47.519 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:02:47.519 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:02:47.557 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:02:47.554+0000 7ff25c7c6600 1 waiting for incremental sync to catch up: 2022-09-12T19:02:47.557 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.10.9, , 00000000010.34.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:02:47.558 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000004.13.5, , 00000000011.37.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:02:52.558 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:02:52.555+0000 7ff25c7c6600 1 waiting for incremental sync to catch up: 2022-09-12T19:02:52.559 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.10.9, , 00000000010.34.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:02:52.559 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000004.13.5, , 00000000011.37.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:02:57.560 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:02:57.556+0000 7ff25c7c6600 1 waiting for incremental sync to catch up: 2022-09-12T19:02:57.560 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.10.9, , 00000000010.34.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:02:57.561 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000004.13.5, , 00000000011.37.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:03:02.561 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:02.558+0000 7ff25c7c6600 1 waiting for incremental sync to catch up: 2022-09-12T19:03:02.561 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000004.13.5, , 00000000010.34.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:03:02.562 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000004.13.5, , 00000000011.37.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:03:07.563 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:07.559+0000 7ff25c7c6600 1 waiting for incremental sync to catch up: 2022-09-12T19:03:07.563 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000004.13.5, , 00000000010.34.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:03:07.564 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000004.13.5, , 00000000011.37.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:03:12.564 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:12.561+0000 7ff25c7c6600 1 waiting for incremental sync to catch up: 2022-09-12T19:03:12.565 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000004.13.5, , 00000000010.34.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:03:12.566 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000004.13.5, , 00000000011.37.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:03:17.566 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:17.562+0000 7ff25c7c6600 1 waiting for incremental sync to catch up: 2022-09-12T19:03:17.566 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000004.13.5, , 00000000010.34.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:03:17.566 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000004.13.5, , 00000000011.37.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:03:22.567 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:22.564+0000 7ff25c7c6600 1 waiting for incremental sync to catch up: 2022-09-12T19:03:22.568 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000004.13.5, , 00000000010.34.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:03:22.568 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000004.13.5, , 00000000011.37.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:03:27.569 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:27.565+0000 7ff25c7c6600 1 waiting for incremental sync to catch up: 2022-09-12T19:03:27.569 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000004.13.5, , 00000000010.34.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:03:27.570 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000004.13.5, , 00000000011.37.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:03:32.570 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:32.568+0000 7ff25c7c6600 1 waiting for incremental sync to catch up: 2022-09-12T19:03:32.570 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000004.13.5, , 00000000010.34.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:03:32.571 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000004.13.5, , 00000000011.37.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:03:37.571 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:37.569+0000 7ff25c7c6600 1 waiting for incremental sync to catch up: 2022-09-12T19:03:37.572 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000004.13.5, , 00000000010.34.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:03:37.572 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000004.13.5, , 00000000011.37.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:03:42.573 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:42.569+0000 7ff25c7c6600 1 bucket sync caught up with source: 2022-09-12T19:03:42.574 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000004.13.5, , 00000000011.37.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:03:42.574 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000004.13.5, , 00000000011.37.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:03:42.575 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:42.569+0000 7ff25c7c6600 0 bucket checkpoint complete 2022-09-12T19:03:42.591 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-16 2022-09-12T19:03:42.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:42.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:42.595 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:03:42.661 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:03:42.661 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:03:42.698 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:42.695+0000 7f1455417600 1 bucket sync caught up with source: 2022-09-12T19:03:42.699 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000004.13.5, , 00000000011.37.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:03:42.699 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000004.13.5, , 00000000011.37.5, , 00000000008.29.5, , , , , ] 2022-09-12T19:03:42.699 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:42.695+0000 7f1455417600 0 bucket checkpoint complete 2022-09-12T19:03:42.711 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-16 2022-09-12T19:03:42.714 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-16 2022-09-12T19:03:42.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:42.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:42.715 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:03:42.773 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:03:42.774 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:03:42.812 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:42.809+0000 7f0c8d69a600 1 bucket sync caught up with source: 2022-09-12T19:03:42.812 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000004.18.5, , 00000000004.25.5, , 00000000015.41.5, , , , , ] 2022-09-12T19:03:42.813 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000004.18.5, , 00000000004.25.5, , 00000000015.41.5, , , , , ] 2022-09-12T19:03:42.813 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:42.809+0000 7f0c8d69a600 0 bucket checkpoint complete 2022-09-12T19:03:42.830 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-16 2022-09-12T19:03:42.831 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:42.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:42.832 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:03:42.896 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:03:42.896 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:03:42.954 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:03:42.952+0000 7f07fc94a600 1 bucket sync caught up with source: 2022-09-12T19:03:42.955 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000004.18.5, , 00000000004.25.5, , 00000000015.41.5, , , , , ] 2022-09-12T19:03:42.955 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000004.18.5, , 00000000004.25.5, , 00000000015.41.5, , , , , ] 2022-09-12T19:03:42.955 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:03:42.952+0000 7f07fc94a600 0 bucket checkpoint complete 2022-09-12T19:03:42.978 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-16 2022-09-12T19:03:42.978 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-16 zones={test-zone2, test-zone1} 2022-09-12T19:03:42.979 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:42.980 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:42.980 DEBUG:boto:Method: HEAD 2022-09-12T19:03:42.981 DEBUG:boto:Path: /jdpiak-16/ 2022-09-12T19:03:42.981 DEBUG:boto:Data: 2022-09-12T19:03:42.982 DEBUG:boto:Headers: {} 2022-09-12T19:03:42.982 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:42.983 DEBUG:boto:Port: 8000 2022-09-12T19:03:42.983 DEBUG:boto:Params: {} 2022-09-12T19:03:42.984 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:03:42.985 DEBUG:boto:Token: None 2022-09-12T19:03:42.986 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:42 GMT /jdpiak-16/ 2022-09-12T19:03:42.986 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iHkZqZAQ5WB/dEWtRE1Q7ZsbbsY= 2022-09-12T19:03:42.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iHkZqZAQ5WB/dEWtRE1Q7ZsbbsY=', 'Content-Length': '0'} 2022-09-12T19:03:42.991 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000230bac3c9a83663b-00631f828e-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:03:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:42.991 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:42.992 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:42.992 DEBUG:boto:Method: HEAD 2022-09-12T19:03:42.993 DEBUG:boto:Path: /jdpiak-16/ 2022-09-12T19:03:42.993 DEBUG:boto:Data: 2022-09-12T19:03:42.994 DEBUG:boto:Headers: {} 2022-09-12T19:03:42.994 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:42.995 DEBUG:boto:Port: 8000 2022-09-12T19:03:42.995 DEBUG:boto:Params: {} 2022-09-12T19:03:42.996 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:03:42.997 DEBUG:boto:Token: None 2022-09-12T19:03:42.997 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:42 GMT /jdpiak-16/ 2022-09-12T19:03:42.998 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iHkZqZAQ5WB/dEWtRE1Q7ZsbbsY= 2022-09-12T19:03:42.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iHkZqZAQ5WB/dEWtRE1Q7ZsbbsY=', 'Content-Length': '0'} 2022-09-12T19:03:43.002 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003753e0f96a5f70ab-00631f828e-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:03:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.002 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:03:43.003 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:43.004 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:43.005 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:43.005 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:43.006 DEBUG:boto:Method: GET 2022-09-12T19:03:43.006 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:43.007 DEBUG:boto:Data: 2022-09-12T19:03:43.007 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.008 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.008 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.009 DEBUG:boto:Params: {} 2022-09-12T19:03:43.009 DEBUG:boto:Token: None 2022-09-12T19:03:43.010 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/?versions 2022-09-12T19:03:43.011 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zfpGNHgPM52OFBHBAivipBcPODE= 2022-09-12T19:03:43.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zfpGNHgPM52OFBHBAivipBcPODE=', 'Content-Length': '0'} 2022-09-12T19:03:43.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000642bf0e770fedc3-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.015 DEBUG:boto:b'jdpiak-16999falsedummydM1h1LlChQwJOE3oHBYl7Qu-tamZZbJtrue2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1-zshRkRrxKU0EJVPpVBUsfYl8JR7HFRtrue2022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone23vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5true2022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eEtrue2022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:43.016 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:03:43.017 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:03:43.017 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:03:43.018 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:03:43.018 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:03:43.019 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:43.020 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:43.021 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:43.021 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:43.022 DEBUG:boto:Method: GET 2022-09-12T19:03:43.023 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:43.023 DEBUG:boto:Data: 2022-09-12T19:03:43.024 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.024 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.025 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.026 DEBUG:boto:Params: {} 2022-09-12T19:03:43.027 DEBUG:boto:Token: None 2022-09-12T19:03:43.028 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/?versions 2022-09-12T19:03:43.029 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zfpGNHgPM52OFBHBAivipBcPODE= 2022-09-12T19:03:43.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zfpGNHgPM52OFBHBAivipBcPODE=', 'Content-Length': '0'} 2022-09-12T19:03:43.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc0810777ac4b71a-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.034 DEBUG:boto:b'jdpiak-16999falsedummydM1h1LlChQwJOE3oHBYl7Qu-tamZZbJtrue2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1-zshRkRrxKU0EJVPpVBUsfYl8JR7HFRtrue2022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone23vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5true2022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eEtrue2022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:43.036 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:03:43.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:03:43.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:03:43.038 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:03:43.039 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:43.040 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:43.041 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:43.041 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:43.042 DEBUG:boto:Method: GET 2022-09-12T19:03:43.042 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:43.043 DEBUG:boto:Data: 2022-09-12T19:03:43.043 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.044 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.044 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.045 DEBUG:boto:Params: {} 2022-09-12T19:03:43.046 DEBUG:boto:Token: None 2022-09-12T19:03:43.047 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/?versions 2022-09-12T19:03:43.048 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zfpGNHgPM52OFBHBAivipBcPODE= 2022-09-12T19:03:43.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zfpGNHgPM52OFBHBAivipBcPODE=', 'Content-Length': '0'} 2022-09-12T19:03:43.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002889bae832756bf0-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.052 DEBUG:boto:b'jdpiak-16999falsedummydM1h1LlChQwJOE3oHBYl7Qu-tamZZbJtrue2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1-zshRkRrxKU0EJVPpVBUsfYl8JR7HFRtrue2022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone23vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5true2022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eEtrue2022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:43.053 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:43.054 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:43.055 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:43.056 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:43.057 DEBUG:boto:Method: GET 2022-09-12T19:03:43.057 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:43.058 DEBUG:boto:Data: 2022-09-12T19:03:43.058 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.059 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.060 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.060 DEBUG:boto:Params: {} 2022-09-12T19:03:43.061 DEBUG:boto:Token: None 2022-09-12T19:03:43.061 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/?versions 2022-09-12T19:03:43.062 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zfpGNHgPM52OFBHBAivipBcPODE= 2022-09-12T19:03:43.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zfpGNHgPM52OFBHBAivipBcPODE=', 'Content-Length': '0'} 2022-09-12T19:03:43.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccb207cf45789149-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.066 DEBUG:boto:b'jdpiak-16999falsedummydM1h1LlChQwJOE3oHBYl7Qu-tamZZbJtrue2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1-zshRkRrxKU0EJVPpVBUsfYl8JR7HFRtrue2022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone23vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5true2022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eEtrue2022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:43.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:43.068 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:43.068 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:43.069 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.069 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.070 DEBUG:boto:Method: GET 2022-09-12T19:03:43.071 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.071 DEBUG:boto:Data: 2022-09-12T19:03:43.072 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.072 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.073 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.074 DEBUG:boto:Params: {} 2022-09-12T19:03:43.074 DEBUG:boto:Token: None 2022-09-12T19:03:43.075 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.076 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kUeE9KBxqAac5lhJRfQkZDusf2Q= 2022-09-12T19:03:43.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kUeE9KBxqAac5lhJRfQkZDusf2Q=', 'Content-Length': '0'} 2022-09-12T19:03:43.078 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000633189b705cf775-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.080 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:43.081 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:43.081 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.082 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.083 DEBUG:boto:Method: GET 2022-09-12T19:03:43.084 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.084 DEBUG:boto:Data: 2022-09-12T19:03:43.085 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.086 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.086 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.087 DEBUG:boto:Params: {} 2022-09-12T19:03:43.087 DEBUG:boto:Token: None 2022-09-12T19:03:43.088 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.088 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kUeE9KBxqAac5lhJRfQkZDusf2Q= 2022-09-12T19:03:43.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kUeE9KBxqAac5lhJRfQkZDusf2Q=', 'Content-Length': '0'} 2022-09-12T19:03:43.091 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c41ec71970caf387-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.092 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:43.092 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:43.093 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.093 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.094 DEBUG:boto:Method: HEAD 2022-09-12T19:03:43.095 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.095 DEBUG:boto:Data: 2022-09-12T19:03:43.096 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.097 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.097 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.098 DEBUG:boto:Params: {} 2022-09-12T19:03:43.099 DEBUG:boto:Token: None 2022-09-12T19:03:43.099 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.100 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SgiFB36oha+d3sG6e9P64Yfq8zA= 2022-09-12T19:03:43.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SgiFB36oha+d3sG6e9P64Yfq8zA=', 'Content-Length': '0'} 2022-09-12T19:03:43.102 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000314491a4d45a3e87-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.103 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:43.104 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:43.104 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.105 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.105 DEBUG:boto:Method: HEAD 2022-09-12T19:03:43.106 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.106 DEBUG:boto:Data: 2022-09-12T19:03:43.107 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.107 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.108 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.108 DEBUG:boto:Params: {} 2022-09-12T19:03:43.109 DEBUG:boto:Token: None 2022-09-12T19:03:43.109 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.110 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SgiFB36oha+d3sG6e9P64Yfq8zA= 2022-09-12T19:03:43.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SgiFB36oha+d3sG6e9P64Yfq8zA=', 'Content-Length': '0'} 2022-09-12T19:03:43.113 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f4aa7372725cc3b2-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:43.114 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:43.115 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:43.116 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.117 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.117 DEBUG:boto:Method: GET 2022-09-12T19:03:43.118 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.119 DEBUG:boto:Data: 2022-09-12T19:03:43.119 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.120 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.120 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.121 DEBUG:boto:Params: {} 2022-09-12T19:03:43.121 DEBUG:boto:Token: None 2022-09-12T19:03:43.122 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.123 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kUeE9KBxqAac5lhJRfQkZDusf2Q= 2022-09-12T19:03:43.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kUeE9KBxqAac5lhJRfQkZDusf2Q=', 'Content-Length': '0'} 2022-09-12T19:03:43.127 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000046d9395aa32532ed-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.128 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:43.129 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:43.129 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.130 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.130 DEBUG:boto:Method: GET 2022-09-12T19:03:43.131 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.131 DEBUG:boto:Data: 2022-09-12T19:03:43.132 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.132 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.133 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.134 DEBUG:boto:Params: {} 2022-09-12T19:03:43.134 DEBUG:boto:Token: None 2022-09-12T19:03:43.135 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.135 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kUeE9KBxqAac5lhJRfQkZDusf2Q= 2022-09-12T19:03:43.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kUeE9KBxqAac5lhJRfQkZDusf2Q=', 'Content-Length': '0'} 2022-09-12T19:03:43.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000068e2d2a8635ed209-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.139 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:43.139 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:43.140 DEBUG:boto:path=/jdpiak-16/?prefix=dummy 2022-09-12T19:03:43.140 DEBUG:boto:auth_path=/jdpiak-16/?prefix=dummy 2022-09-12T19:03:43.141 DEBUG:boto:Method: GET 2022-09-12T19:03:43.141 DEBUG:boto:Path: /jdpiak-16/?prefix=dummy 2022-09-12T19:03:43.142 DEBUG:boto:Data: 2022-09-12T19:03:43.142 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.143 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.143 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.144 DEBUG:boto:Params: {} 2022-09-12T19:03:43.144 DEBUG:boto:Token: None 2022-09-12T19:03:43.145 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/ 2022-09-12T19:03:43.146 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak= 2022-09-12T19:03:43.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak=', 'Content-Length': '0'} 2022-09-12T19:03:43.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfeeb8e843a3d27b-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.190 DEBUG:boto:b'jdpiak-16dummy1000falsedummy2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:43.191 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:43.192 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:43.193 DEBUG:boto:path=/jdpiak-16/?prefix=dummy 2022-09-12T19:03:43.194 DEBUG:boto:auth_path=/jdpiak-16/?prefix=dummy 2022-09-12T19:03:43.195 DEBUG:boto:Method: GET 2022-09-12T19:03:43.196 DEBUG:boto:Path: /jdpiak-16/?prefix=dummy 2022-09-12T19:03:43.197 DEBUG:boto:Data: 2022-09-12T19:03:43.198 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.199 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.200 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.201 DEBUG:boto:Params: {} 2022-09-12T19:03:43.202 DEBUG:boto:Token: None 2022-09-12T19:03:43.203 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/ 2022-09-12T19:03:43.204 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak= 2022-09-12T19:03:43.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak=', 'Content-Length': '0'} 2022-09-12T19:03:43.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026678bfb1690923e-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.208 DEBUG:boto:b'jdpiak-16dummy1000falsedummy2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:43.209 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:43.210 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:43.210 DEBUG:boto:Method: HEAD 2022-09-12T19:03:43.211 DEBUG:boto:Path: /jdpiak-16/dummy 2022-09-12T19:03:43.211 DEBUG:boto:Data: 2022-09-12T19:03:43.212 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.212 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.213 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.213 DEBUG:boto:Params: {} 2022-09-12T19:03:43.214 DEBUG:boto:Token: None 2022-09-12T19:03:43.214 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/dummy 2022-09-12T19:03:43.215 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PZ34SvyfZT8UCqge5pHdZRgxSjc= 2022-09-12T19:03:43.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PZ34SvyfZT8UCqge5pHdZRgxSjc=', 'Content-Length': '0'} 2022-09-12T19:03:43.217 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e0c0bbfb17eb803a-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.218 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:43.219 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:43.219 DEBUG:boto:Method: HEAD 2022-09-12T19:03:43.220 DEBUG:boto:Path: /jdpiak-16/dummy 2022-09-12T19:03:43.220 DEBUG:boto:Data: 2022-09-12T19:03:43.221 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.221 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.222 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.222 DEBUG:boto:Params: {} 2022-09-12T19:03:43.223 DEBUG:boto:Token: None 2022-09-12T19:03:43.223 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/dummy 2022-09-12T19:03:43.224 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PZ34SvyfZT8UCqge5pHdZRgxSjc= 2022-09-12T19:03:43.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PZ34SvyfZT8UCqge5pHdZRgxSjc=', 'Content-Length': '0'} 2022-09-12T19:03:43.226 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000fe716c05971671b-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:43.228 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:43.228 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:43.229 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.229 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.230 DEBUG:boto:Method: GET 2022-09-12T19:03:43.230 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.231 DEBUG:boto:Data: 2022-09-12T19:03:43.231 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.232 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.233 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.233 DEBUG:boto:Params: {} 2022-09-12T19:03:43.234 DEBUG:boto:Token: None 2022-09-12T19:03:43.235 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.235 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kUeE9KBxqAac5lhJRfQkZDusf2Q= 2022-09-12T19:03:43.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kUeE9KBxqAac5lhJRfQkZDusf2Q=', 'Content-Length': '0'} 2022-09-12T19:03:43.238 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002ac1c2b402a5132f-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.239 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:43.240 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:43.240 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.241 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.241 DEBUG:boto:Method: GET 2022-09-12T19:03:43.242 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.242 DEBUG:boto:Data: 2022-09-12T19:03:43.243 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.244 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.244 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.245 DEBUG:boto:Params: {} 2022-09-12T19:03:43.246 DEBUG:boto:Token: None 2022-09-12T19:03:43.247 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:43.248 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kUeE9KBxqAac5lhJRfQkZDusf2Q= 2022-09-12T19:03:43.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kUeE9KBxqAac5lhJRfQkZDusf2Q=', 'Content-Length': '0'} 2022-09-12T19:03:43.251 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a6efe14981a2742b-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.252 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:43.253 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:43.254 DEBUG:boto:path=/jdpiak-16/?prefix=dummy 2022-09-12T19:03:43.254 DEBUG:boto:auth_path=/jdpiak-16/?prefix=dummy 2022-09-12T19:03:43.255 DEBUG:boto:Method: GET 2022-09-12T19:03:43.256 DEBUG:boto:Path: /jdpiak-16/?prefix=dummy 2022-09-12T19:03:43.256 DEBUG:boto:Data: 2022-09-12T19:03:43.257 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.257 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.258 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.258 DEBUG:boto:Params: {} 2022-09-12T19:03:43.259 DEBUG:boto:Token: None 2022-09-12T19:03:43.260 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/ 2022-09-12T19:03:43.260 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak= 2022-09-12T19:03:43.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak=', 'Content-Length': '0'} 2022-09-12T19:03:43.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bcffff6da1cb6c2-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.265 DEBUG:boto:b'jdpiak-16dummy1000falsedummy2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:43.266 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:43.267 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:43.267 DEBUG:boto:path=/jdpiak-16/?prefix=dummy 2022-09-12T19:03:43.268 DEBUG:boto:auth_path=/jdpiak-16/?prefix=dummy 2022-09-12T19:03:43.268 DEBUG:boto:Method: GET 2022-09-12T19:03:43.269 DEBUG:boto:Path: /jdpiak-16/?prefix=dummy 2022-09-12T19:03:43.269 DEBUG:boto:Data: 2022-09-12T19:03:43.270 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.271 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.271 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.272 DEBUG:boto:Params: {} 2022-09-12T19:03:43.272 DEBUG:boto:Token: None 2022-09-12T19:03:43.273 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/ 2022-09-12T19:03:43.273 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak= 2022-09-12T19:03:43.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak=', 'Content-Length': '0'} 2022-09-12T19:03:43.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064b5db02aba545c4-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.277 DEBUG:boto:b'jdpiak-16dummy1000falsedummy2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:43.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:43.278 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:43.279 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:43.280 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.281 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.281 DEBUG:boto:Method: GET 2022-09-12T19:03:43.282 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.283 DEBUG:boto:Data: 2022-09-12T19:03:43.283 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.284 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.284 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.285 DEBUG:boto:Params: {} 2022-09-12T19:03:43.285 DEBUG:boto:Token: None 2022-09-12T19:03:43.286 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.286 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FNbA+76McZihAEHnXvCqYeVgp18= 2022-09-12T19:03:43.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FNbA+76McZihAEHnXvCqYeVgp18=', 'Content-Length': '0'} 2022-09-12T19:03:43.289 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8e66e223cfafcdb-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.290 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:43.290 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:43.291 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.291 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.292 DEBUG:boto:Method: GET 2022-09-12T19:03:43.292 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.293 DEBUG:boto:Data: 2022-09-12T19:03:43.293 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.294 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.294 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.295 DEBUG:boto:Params: {} 2022-09-12T19:03:43.295 DEBUG:boto:Token: None 2022-09-12T19:03:43.296 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.296 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FNbA+76McZihAEHnXvCqYeVgp18= 2022-09-12T19:03:43.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FNbA+76McZihAEHnXvCqYeVgp18=', 'Content-Length': '0'} 2022-09-12T19:03:43.299 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041d8ed470a2135a9-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.300 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:43.300 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:43.301 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.301 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.302 DEBUG:boto:Method: HEAD 2022-09-12T19:03:43.302 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.303 DEBUG:boto:Data: 2022-09-12T19:03:43.303 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.304 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.304 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.305 DEBUG:boto:Params: {} 2022-09-12T19:03:43.305 DEBUG:boto:Token: None 2022-09-12T19:03:43.306 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.306 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dFcCieG016shJgAOOCgI3ihQQyY= 2022-09-12T19:03:43.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dFcCieG016shJgAOOCgI3ihQQyY=', 'Content-Length': '0'} 2022-09-12T19:03:43.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000134ef48afbf626a2-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.310 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:43.310 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:43.311 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.311 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.312 DEBUG:boto:Method: HEAD 2022-09-12T19:03:43.312 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.313 DEBUG:boto:Data: 2022-09-12T19:03:43.313 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.314 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.314 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.315 DEBUG:boto:Params: {} 2022-09-12T19:03:43.315 DEBUG:boto:Token: None 2022-09-12T19:03:43.316 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.316 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dFcCieG016shJgAOOCgI3ihQQyY= 2022-09-12T19:03:43.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dFcCieG016shJgAOOCgI3ihQQyY=', 'Content-Length': '0'} 2022-09-12T19:03:43.318 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8ab28c9d75ca789-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:43.320 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:43.320 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:43.321 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.321 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.322 DEBUG:boto:Method: GET 2022-09-12T19:03:43.323 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.323 DEBUG:boto:Data: 2022-09-12T19:03:43.324 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.324 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.325 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.325 DEBUG:boto:Params: {} 2022-09-12T19:03:43.326 DEBUG:boto:Token: None 2022-09-12T19:03:43.327 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.327 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FNbA+76McZihAEHnXvCqYeVgp18= 2022-09-12T19:03:43.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FNbA+76McZihAEHnXvCqYeVgp18=', 'Content-Length': '0'} 2022-09-12T19:03:43.330 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072c12b1b04e704c1-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.331 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:43.331 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:43.332 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.332 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.333 DEBUG:boto:Method: GET 2022-09-12T19:03:43.333 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.334 DEBUG:boto:Data: 2022-09-12T19:03:43.334 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.335 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.336 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.336 DEBUG:boto:Params: {} 2022-09-12T19:03:43.337 DEBUG:boto:Token: None 2022-09-12T19:03:43.337 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.338 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FNbA+76McZihAEHnXvCqYeVgp18= 2022-09-12T19:03:43.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FNbA+76McZihAEHnXvCqYeVgp18=', 'Content-Length': '0'} 2022-09-12T19:03:43.341 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000832c514a986af120-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.382 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:43.383 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:43.384 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:43.384 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:43.385 DEBUG:boto:Method: GET 2022-09-12T19:03:43.385 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:43.386 DEBUG:boto:Data: 2022-09-12T19:03:43.387 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.387 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.388 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.388 DEBUG:boto:Params: {} 2022-09-12T19:03:43.389 DEBUG:boto:Token: None 2022-09-12T19:03:43.390 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/ 2022-09-12T19:03:43.390 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak= 2022-09-12T19:03:43.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak=', 'Content-Length': '0'} 2022-09-12T19:03:43.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cc6afe6bd73d601-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.394 DEBUG:boto:b'jdpiak-16obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:43.395 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:43.396 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:43.397 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:43.397 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:43.398 DEBUG:boto:Method: GET 2022-09-12T19:03:43.398 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:43.399 DEBUG:boto:Data: 2022-09-12T19:03:43.399 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.400 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.401 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.401 DEBUG:boto:Params: {} 2022-09-12T19:03:43.402 DEBUG:boto:Token: None 2022-09-12T19:03:43.402 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/ 2022-09-12T19:03:43.403 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak= 2022-09-12T19:03:43.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak=', 'Content-Length': '0'} 2022-09-12T19:03:43.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078bb7badd67d8e61-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.407 DEBUG:boto:b'jdpiak-16obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:43.408 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:43.408 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:43.409 DEBUG:boto:Method: HEAD 2022-09-12T19:03:43.409 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1 2022-09-12T19:03:43.410 DEBUG:boto:Data: 2022-09-12T19:03:43.410 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.411 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.411 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.412 DEBUG:boto:Params: {} 2022-09-12T19:03:43.412 DEBUG:boto:Token: None 2022-09-12T19:03:43.413 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone1 2022-09-12T19:03:43.414 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tORX0vFnMs+MFVN+zmrsjeOETaw= 2022-09-12T19:03:43.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tORX0vFnMs+MFVN+zmrsjeOETaw=', 'Content-Length': '0'} 2022-09-12T19:03:43.416 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060a4999eaa322eb0-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.417 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:43.418 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:43.418 DEBUG:boto:Method: HEAD 2022-09-12T19:03:43.419 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1 2022-09-12T19:03:43.419 DEBUG:boto:Data: 2022-09-12T19:03:43.420 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.420 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.421 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.421 DEBUG:boto:Params: {} 2022-09-12T19:03:43.422 DEBUG:boto:Token: None 2022-09-12T19:03:43.422 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone1 2022-09-12T19:03:43.423 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tORX0vFnMs+MFVN+zmrsjeOETaw= 2022-09-12T19:03:43.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tORX0vFnMs+MFVN+zmrsjeOETaw=', 'Content-Length': '0'} 2022-09-12T19:03:43.426 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a37ec4472131ee8-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:43.427 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:43.428 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:43.428 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.429 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.429 DEBUG:boto:Method: GET 2022-09-12T19:03:43.430 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.430 DEBUG:boto:Data: 2022-09-12T19:03:43.431 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.432 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.432 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.433 DEBUG:boto:Params: {} 2022-09-12T19:03:43.433 DEBUG:boto:Token: None 2022-09-12T19:03:43.434 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.435 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FNbA+76McZihAEHnXvCqYeVgp18= 2022-09-12T19:03:43.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FNbA+76McZihAEHnXvCqYeVgp18=', 'Content-Length': '0'} 2022-09-12T19:03:43.437 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000321ef06bd1f979ed-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.438 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:43.439 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:43.439 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.440 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.440 DEBUG:boto:Method: GET 2022-09-12T19:03:43.441 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.441 DEBUG:boto:Data: 2022-09-12T19:03:43.442 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.443 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.443 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.444 DEBUG:boto:Params: {} 2022-09-12T19:03:43.444 DEBUG:boto:Token: None 2022-09-12T19:03:43.445 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:43.446 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FNbA+76McZihAEHnXvCqYeVgp18= 2022-09-12T19:03:43.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FNbA+76McZihAEHnXvCqYeVgp18=', 'Content-Length': '0'} 2022-09-12T19:03:43.449 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e77397400dd3380b-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.450 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:43.450 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:43.451 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:43.451 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:43.452 DEBUG:boto:Method: GET 2022-09-12T19:03:43.452 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:43.453 DEBUG:boto:Data: 2022-09-12T19:03:43.453 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.454 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.455 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.455 DEBUG:boto:Params: {} 2022-09-12T19:03:43.456 DEBUG:boto:Token: None 2022-09-12T19:03:43.457 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/ 2022-09-12T19:03:43.457 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak= 2022-09-12T19:03:43.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak=', 'Content-Length': '0'} 2022-09-12T19:03:43.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081b95230d4a2e41f-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.461 DEBUG:boto:b'jdpiak-16obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:43.462 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:43.462 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:43.463 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:43.464 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:43.464 DEBUG:boto:Method: GET 2022-09-12T19:03:43.465 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:43.465 DEBUG:boto:Data: 2022-09-12T19:03:43.466 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.467 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.467 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.468 DEBUG:boto:Params: {} 2022-09-12T19:03:43.469 DEBUG:boto:Token: None 2022-09-12T19:03:43.469 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/ 2022-09-12T19:03:43.470 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak= 2022-09-12T19:03:43.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak=', 'Content-Length': '0'} 2022-09-12T19:03:43.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4f8218988e6792f-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.474 DEBUG:boto:b'jdpiak-16obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:43.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:43.475 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:43.476 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:43.476 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.477 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.477 DEBUG:boto:Method: GET 2022-09-12T19:03:43.478 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.478 DEBUG:boto:Data: 2022-09-12T19:03:43.479 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.480 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.480 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.481 DEBUG:boto:Params: {} 2022-09-12T19:03:43.481 DEBUG:boto:Token: None 2022-09-12T19:03:43.482 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.483 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s6u0lAztRK6mWiPKchOvVYipaPg= 2022-09-12T19:03:43.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s6u0lAztRK6mWiPKchOvVYipaPg=', 'Content-Length': '0'} 2022-09-12T19:03:43.486 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000370e053f96bdbe9f-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.487 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:43.488 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:43.488 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.489 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.490 DEBUG:boto:Method: GET 2022-09-12T19:03:43.490 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.491 DEBUG:boto:Data: 2022-09-12T19:03:43.492 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.493 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.494 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.495 DEBUG:boto:Params: {} 2022-09-12T19:03:43.496 DEBUG:boto:Token: None 2022-09-12T19:03:43.497 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.498 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s6u0lAztRK6mWiPKchOvVYipaPg= 2022-09-12T19:03:43.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s6u0lAztRK6mWiPKchOvVYipaPg=', 'Content-Length': '0'} 2022-09-12T19:03:43.501 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd208ab8d26793a9-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.502 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:43.503 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:43.503 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.504 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.504 DEBUG:boto:Method: HEAD 2022-09-12T19:03:43.505 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.506 DEBUG:boto:Data: 2022-09-12T19:03:43.506 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.507 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.507 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.508 DEBUG:boto:Params: {} 2022-09-12T19:03:43.508 DEBUG:boto:Token: None 2022-09-12T19:03:43.509 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.510 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uccDitmf9h6lIicNXJbBy+5lTc0= 2022-09-12T19:03:43.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uccDitmf9h6lIicNXJbBy+5lTc0=', 'Content-Length': '0'} 2022-09-12T19:03:43.512 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1610901edab63b6-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.513 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:43.514 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:43.514 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.515 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.515 DEBUG:boto:Method: HEAD 2022-09-12T19:03:43.516 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.516 DEBUG:boto:Data: 2022-09-12T19:03:43.517 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.517 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.518 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.518 DEBUG:boto:Params: {} 2022-09-12T19:03:43.519 DEBUG:boto:Token: None 2022-09-12T19:03:43.519 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.520 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uccDitmf9h6lIicNXJbBy+5lTc0= 2022-09-12T19:03:43.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uccDitmf9h6lIicNXJbBy+5lTc0=', 'Content-Length': '0'} 2022-09-12T19:03:43.522 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d36a500003ae8958-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:43.524 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:43.524 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:43.525 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.525 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.526 DEBUG:boto:Method: GET 2022-09-12T19:03:43.526 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.527 DEBUG:boto:Data: 2022-09-12T19:03:43.528 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.528 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.529 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.530 DEBUG:boto:Params: {} 2022-09-12T19:03:43.530 DEBUG:boto:Token: None 2022-09-12T19:03:43.531 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.532 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s6u0lAztRK6mWiPKchOvVYipaPg= 2022-09-12T19:03:43.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s6u0lAztRK6mWiPKchOvVYipaPg=', 'Content-Length': '0'} 2022-09-12T19:03:43.534 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5f99701d26f7b05-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.578 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:43.579 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:43.580 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.580 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.582 DEBUG:boto:Method: GET 2022-09-12T19:03:43.583 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.584 DEBUG:boto:Data: 2022-09-12T19:03:43.585 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.586 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.587 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.588 DEBUG:boto:Params: {} 2022-09-12T19:03:43.590 DEBUG:boto:Token: None 2022-09-12T19:03:43.591 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.592 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s6u0lAztRK6mWiPKchOvVYipaPg= 2022-09-12T19:03:43.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s6u0lAztRK6mWiPKchOvVYipaPg=', 'Content-Length': '0'} 2022-09-12T19:03:43.596 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa94502933e6e302-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.597 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:43.598 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:43.599 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:43.600 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:43.601 DEBUG:boto:Method: GET 2022-09-12T19:03:43.602 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:43.603 DEBUG:boto:Data: 2022-09-12T19:03:43.604 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.606 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.607 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.608 DEBUG:boto:Params: {} 2022-09-12T19:03:43.609 DEBUG:boto:Token: None 2022-09-12T19:03:43.610 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/ 2022-09-12T19:03:43.611 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak= 2022-09-12T19:03:43.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak=', 'Content-Length': '0'} 2022-09-12T19:03:43.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b60bdc563130bc6d-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.616 DEBUG:boto:b'jdpiak-16obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:43.617 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:43.618 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:43.619 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:43.619 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:43.620 DEBUG:boto:Method: GET 2022-09-12T19:03:43.621 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:43.621 DEBUG:boto:Data: 2022-09-12T19:03:43.622 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.622 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.623 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.623 DEBUG:boto:Params: {} 2022-09-12T19:03:43.624 DEBUG:boto:Token: None 2022-09-12T19:03:43.625 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/ 2022-09-12T19:03:43.625 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak= 2022-09-12T19:03:43.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak=', 'Content-Length': '0'} 2022-09-12T19:03:43.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bde171694058b03-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.630 DEBUG:boto:b'jdpiak-16obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:43.631 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:43.631 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:43.632 DEBUG:boto:Method: HEAD 2022-09-12T19:03:43.633 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2 2022-09-12T19:03:43.634 DEBUG:boto:Data: 2022-09-12T19:03:43.635 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.636 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.637 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.638 DEBUG:boto:Params: {} 2022-09-12T19:03:43.640 DEBUG:boto:Token: None 2022-09-12T19:03:43.641 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone2 2022-09-12T19:03:43.642 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2zTw/6gmZQnIcIk8o8UjpLVQ4Xg= 2022-09-12T19:03:43.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2zTw/6gmZQnIcIk8o8UjpLVQ4Xg=', 'Content-Length': '0'} 2022-09-12T19:03:43.646 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d58a35d7f38483f6-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.647 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:43.648 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:43.648 DEBUG:boto:Method: HEAD 2022-09-12T19:03:43.649 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2 2022-09-12T19:03:43.649 DEBUG:boto:Data: 2022-09-12T19:03:43.650 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.651 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.651 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.652 DEBUG:boto:Params: {} 2022-09-12T19:03:43.652 DEBUG:boto:Token: None 2022-09-12T19:03:43.653 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone2 2022-09-12T19:03:43.654 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2zTw/6gmZQnIcIk8o8UjpLVQ4Xg= 2022-09-12T19:03:43.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2zTw/6gmZQnIcIk8o8UjpLVQ4Xg=', 'Content-Length': '0'} 2022-09-12T19:03:43.657 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dedb9b7a4c0e15fe-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:43.659 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:43.660 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:43.661 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.662 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.663 DEBUG:boto:Method: GET 2022-09-12T19:03:43.663 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.664 DEBUG:boto:Data: 2022-09-12T19:03:43.666 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.666 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.667 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.668 DEBUG:boto:Params: {} 2022-09-12T19:03:43.669 DEBUG:boto:Token: None 2022-09-12T19:03:43.670 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.671 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s6u0lAztRK6mWiPKchOvVYipaPg= 2022-09-12T19:03:43.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s6u0lAztRK6mWiPKchOvVYipaPg=', 'Content-Length': '0'} 2022-09-12T19:03:43.674 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5a1ac043bc5d4e4-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.675 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:43.676 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:43.677 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.677 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.678 DEBUG:boto:Method: GET 2022-09-12T19:03:43.679 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.680 DEBUG:boto:Data: 2022-09-12T19:03:43.680 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.681 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.681 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.682 DEBUG:boto:Params: {} 2022-09-12T19:03:43.683 DEBUG:boto:Token: None 2022-09-12T19:03:43.683 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:43.684 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s6u0lAztRK6mWiPKchOvVYipaPg= 2022-09-12T19:03:43.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s6u0lAztRK6mWiPKchOvVYipaPg=', 'Content-Length': '0'} 2022-09-12T19:03:43.687 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000789abb408a8de618-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.688 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:43.689 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:43.689 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:43.690 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:43.691 DEBUG:boto:Method: GET 2022-09-12T19:03:43.692 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:43.692 DEBUG:boto:Data: 2022-09-12T19:03:43.693 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.693 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.694 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.695 DEBUG:boto:Params: {} 2022-09-12T19:03:43.695 DEBUG:boto:Token: None 2022-09-12T19:03:43.696 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/ 2022-09-12T19:03:43.697 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak= 2022-09-12T19:03:43.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak=', 'Content-Length': '0'} 2022-09-12T19:03:43.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d29152e1bb6f03ce-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.742 DEBUG:boto:b'jdpiak-16obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:43.743 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:43.744 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:43.745 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:43.746 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:43.746 DEBUG:boto:Method: GET 2022-09-12T19:03:43.747 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:43.748 DEBUG:boto:Data: 2022-09-12T19:03:43.748 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.749 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.750 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.750 DEBUG:boto:Params: {} 2022-09-12T19:03:43.751 DEBUG:boto:Token: None 2022-09-12T19:03:43.752 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/ 2022-09-12T19:03:43.753 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak= 2022-09-12T19:03:43.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak=', 'Content-Length': '0'} 2022-09-12T19:03:43.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f3ab881f3642cd9-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.757 DEBUG:boto:b'jdpiak-16obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:43.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:43.758 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:43.759 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:43.760 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.761 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.762 DEBUG:boto:Method: GET 2022-09-12T19:03:43.762 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.763 DEBUG:boto:Data: 2022-09-12T19:03:43.764 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.765 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.765 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.766 DEBUG:boto:Params: {} 2022-09-12T19:03:43.767 DEBUG:boto:Token: None 2022-09-12T19:03:43.768 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.768 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:roYurDg+ZioYhaOzuNJoTOmUZkk= 2022-09-12T19:03:43.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:roYurDg+ZioYhaOzuNJoTOmUZkk=', 'Content-Length': '0'} 2022-09-12T19:03:43.771 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac9ac26516389e6e-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.772 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:43.773 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:43.773 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.774 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.775 DEBUG:boto:Method: GET 2022-09-12T19:03:43.776 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.776 DEBUG:boto:Data: 2022-09-12T19:03:43.777 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.778 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.778 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.779 DEBUG:boto:Params: {} 2022-09-12T19:03:43.780 DEBUG:boto:Token: None 2022-09-12T19:03:43.780 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.781 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:roYurDg+ZioYhaOzuNJoTOmUZkk= 2022-09-12T19:03:43.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:roYurDg+ZioYhaOzuNJoTOmUZkk=', 'Content-Length': '0'} 2022-09-12T19:03:43.784 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d9daa5d9dc15eef7-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.785 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:43.786 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:43.787 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.787 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.788 DEBUG:boto:Method: HEAD 2022-09-12T19:03:43.788 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.789 DEBUG:boto:Data: 2022-09-12T19:03:43.790 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.790 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.791 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.792 DEBUG:boto:Params: {} 2022-09-12T19:03:43.792 DEBUG:boto:Token: None 2022-09-12T19:03:43.793 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.794 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AKaAqlvwZtg2uWeWf25+ns6jcvo= 2022-09-12T19:03:43.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AKaAqlvwZtg2uWeWf25+ns6jcvo=', 'Content-Length': '0'} 2022-09-12T19:03:43.796 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005012d0e2c655e359-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.797 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:43.798 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:43.799 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.799 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.800 DEBUG:boto:Method: HEAD 2022-09-12T19:03:43.801 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.801 DEBUG:boto:Data: 2022-09-12T19:03:43.802 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.803 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.803 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.804 DEBUG:boto:Params: {} 2022-09-12T19:03:43.805 DEBUG:boto:Token: None 2022-09-12T19:03:43.805 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.806 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AKaAqlvwZtg2uWeWf25+ns6jcvo= 2022-09-12T19:03:43.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AKaAqlvwZtg2uWeWf25+ns6jcvo=', 'Content-Length': '0'} 2022-09-12T19:03:43.809 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048f64fb2a1fcb288-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:43.810 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:43.811 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:43.812 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.812 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.813 DEBUG:boto:Method: GET 2022-09-12T19:03:43.814 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.814 DEBUG:boto:Data: 2022-09-12T19:03:43.815 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.816 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.816 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.817 DEBUG:boto:Params: {} 2022-09-12T19:03:43.818 DEBUG:boto:Token: None 2022-09-12T19:03:43.819 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.819 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:roYurDg+ZioYhaOzuNJoTOmUZkk= 2022-09-12T19:03:43.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:roYurDg+ZioYhaOzuNJoTOmUZkk=', 'Content-Length': '0'} 2022-09-12T19:03:43.822 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0b59cc183ba1d79-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.823 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:43.824 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:43.824 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.825 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.826 DEBUG:boto:Method: GET 2022-09-12T19:03:43.826 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.827 DEBUG:boto:Data: 2022-09-12T19:03:43.828 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.828 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.829 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.830 DEBUG:boto:Params: {} 2022-09-12T19:03:43.831 DEBUG:boto:Token: None 2022-09-12T19:03:43.831 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.832 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:roYurDg+ZioYhaOzuNJoTOmUZkk= 2022-09-12T19:03:43.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:roYurDg+ZioYhaOzuNJoTOmUZkk=', 'Content-Length': '0'} 2022-09-12T19:03:43.835 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053d5341e554e08e7-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.836 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:43.836 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:43.837 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:43.837 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:43.838 DEBUG:boto:Method: GET 2022-09-12T19:03:43.839 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:43.839 DEBUG:boto:Data: 2022-09-12T19:03:43.840 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.840 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.841 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.841 DEBUG:boto:Params: {} 2022-09-12T19:03:43.842 DEBUG:boto:Token: None 2022-09-12T19:03:43.842 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/ 2022-09-12T19:03:43.843 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak= 2022-09-12T19:03:43.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak=', 'Content-Length': '0'} 2022-09-12T19:03:43.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036937d2fbfb27b27-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.847 DEBUG:boto:b'jdpiak-16obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:43.847 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:43.848 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:43.849 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:43.849 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:43.850 DEBUG:boto:Method: GET 2022-09-12T19:03:43.850 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:43.851 DEBUG:boto:Data: 2022-09-12T19:03:43.851 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.852 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.852 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.853 DEBUG:boto:Params: {} 2022-09-12T19:03:43.853 DEBUG:boto:Token: None 2022-09-12T19:03:43.854 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/ 2022-09-12T19:03:43.855 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak= 2022-09-12T19:03:43.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak=', 'Content-Length': '0'} 2022-09-12T19:03:43.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6de7a4314e35f50-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.858 DEBUG:boto:b'jdpiak-16obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:43.859 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:43.860 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:43.860 DEBUG:boto:Method: HEAD 2022-09-12T19:03:43.861 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3 2022-09-12T19:03:43.861 DEBUG:boto:Data: 2022-09-12T19:03:43.862 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.862 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.863 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.863 DEBUG:boto:Params: {} 2022-09-12T19:03:43.864 DEBUG:boto:Token: None 2022-09-12T19:03:43.864 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone3 2022-09-12T19:03:43.865 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B7DVSkwKfvVktOi23AA4HkBTybY= 2022-09-12T19:03:43.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B7DVSkwKfvVktOi23AA4HkBTybY=', 'Content-Length': '0'} 2022-09-12T19:03:43.868 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae39ed9d0158c648-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.869 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:43.869 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:43.870 DEBUG:boto:Method: HEAD 2022-09-12T19:03:43.871 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3 2022-09-12T19:03:43.871 DEBUG:boto:Data: 2022-09-12T19:03:43.872 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.872 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.873 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.874 DEBUG:boto:Params: {} 2022-09-12T19:03:43.874 DEBUG:boto:Token: None 2022-09-12T19:03:43.875 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone3 2022-09-12T19:03:43.875 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B7DVSkwKfvVktOi23AA4HkBTybY= 2022-09-12T19:03:43.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B7DVSkwKfvVktOi23AA4HkBTybY=', 'Content-Length': '0'} 2022-09-12T19:03:43.878 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020a742268733776b-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:43.879 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:43.880 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:43.880 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.881 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.881 DEBUG:boto:Method: GET 2022-09-12T19:03:43.882 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.883 DEBUG:boto:Data: 2022-09-12T19:03:43.883 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.884 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.884 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.885 DEBUG:boto:Params: {} 2022-09-12T19:03:43.885 DEBUG:boto:Token: None 2022-09-12T19:03:43.886 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.887 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:roYurDg+ZioYhaOzuNJoTOmUZkk= 2022-09-12T19:03:43.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:roYurDg+ZioYhaOzuNJoTOmUZkk=', 'Content-Length': '0'} 2022-09-12T19:03:43.889 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000e8bfdddd76111b-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.930 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:43.931 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:43.931 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.932 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.932 DEBUG:boto:Method: GET 2022-09-12T19:03:43.933 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.933 DEBUG:boto:Data: 2022-09-12T19:03:43.934 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.934 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.935 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.935 DEBUG:boto:Params: {} 2022-09-12T19:03:43.936 DEBUG:boto:Token: None 2022-09-12T19:03:43.937 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:43.937 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:roYurDg+ZioYhaOzuNJoTOmUZkk= 2022-09-12T19:03:43.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:roYurDg+ZioYhaOzuNJoTOmUZkk=', 'Content-Length': '0'} 2022-09-12T19:03:43.940 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000121ba93ee39a7d6f-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.941 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:43.941 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:43.942 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:43.942 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:43.943 DEBUG:boto:Method: GET 2022-09-12T19:03:43.944 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:43.944 DEBUG:boto:Data: 2022-09-12T19:03:43.945 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.945 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:43.946 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.946 DEBUG:boto:Params: {} 2022-09-12T19:03:43.947 DEBUG:boto:Token: None 2022-09-12T19:03:43.947 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/ 2022-09-12T19:03:43.948 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak= 2022-09-12T19:03:43.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak=', 'Content-Length': '0'} 2022-09-12T19:03:43.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000274bb1ba97d47b03-00631f828f-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.952 DEBUG:boto:b'jdpiak-16obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:43.952 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:43.953 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:43.954 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:43.954 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:43.955 DEBUG:boto:Method: GET 2022-09-12T19:03:43.955 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:43.956 DEBUG:boto:Data: 2022-09-12T19:03:43.956 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.957 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.957 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.958 DEBUG:boto:Params: {} 2022-09-12T19:03:43.958 DEBUG:boto:Token: None 2022-09-12T19:03:43.959 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/ 2022-09-12T19:03:43.960 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak= 2022-09-12T19:03:43.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ittuz3qBluxIqpN9xlwHbC7RAak=', 'Content-Length': '0'} 2022-09-12T19:03:43.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f2c8b752e3f8d38-00631f828f-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.965 DEBUG:boto:b'jdpiak-16obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:43.965 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-16 zones={test-zone2, test-zone1} 2022-09-12T19:03:43.966 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-16 zones={test-zone3, test-zone1} 2022-09-12T19:03:43.967 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:43.967 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:43.968 DEBUG:boto:Method: HEAD 2022-09-12T19:03:43.969 DEBUG:boto:Path: /jdpiak-16/ 2022-09-12T19:03:43.969 DEBUG:boto:Data: 2022-09-12T19:03:43.970 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.971 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:43.971 DEBUG:boto:Port: 8001 2022-09-12T19:03:43.972 DEBUG:boto:Params: {} 2022-09-12T19:03:43.972 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T19:03:43.973 DEBUG:boto:Token: None 2022-09-12T19:03:43.973 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/ 2022-09-12T19:03:43.974 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kHEHqF2aPGUbUtDk2lFR3IooMg8= 2022-09-12T19:03:43.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kHEHqF2aPGUbUtDk2lFR3IooMg8=', 'Content-Length': '0'} 2022-09-12T19:03:43.978 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000300972f4388ecb19-00631f828f-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.978 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:43.979 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:43.979 DEBUG:boto:Method: HEAD 2022-09-12T19:03:43.980 DEBUG:boto:Path: /jdpiak-16/ 2022-09-12T19:03:43.980 DEBUG:boto:Data: 2022-09-12T19:03:43.981 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.981 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:43.982 DEBUG:boto:Port: 8000 2022-09-12T19:03:43.982 DEBUG:boto:Params: {} 2022-09-12T19:03:43.983 DEBUG:boto:Token: None 2022-09-12T19:03:43.984 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/ 2022-09-12T19:03:43.984 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kHEHqF2aPGUbUtDk2lFR3IooMg8= 2022-09-12T19:03:43.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kHEHqF2aPGUbUtDk2lFR3IooMg8=', 'Content-Length': '0'} 2022-09-12T19:03:43.987 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d90a135de4d62249-00631f828f-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.988 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:03:43.988 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:43.989 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:43.989 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:43.990 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:43.990 DEBUG:boto:Method: GET 2022-09-12T19:03:43.991 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:43.991 DEBUG:boto:Data: 2022-09-12T19:03:43.992 DEBUG:boto:Headers: {} 2022-09-12T19:03:43.992 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:43.993 DEBUG:boto:Port: 8001 2022-09-12T19:03:43.993 DEBUG:boto:Params: {} 2022-09-12T19:03:43.994 DEBUG:boto:Token: None 2022-09-12T19:03:43.994 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:43 GMT /jdpiak-16/?versions 2022-09-12T19:03:43.995 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zfpGNHgPM52OFBHBAivipBcPODE= 2022-09-12T19:03:43.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zfpGNHgPM52OFBHBAivipBcPODE=', 'Content-Length': '0'} 2022-09-12T19:03:43.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012d0fe72629e798c-00631f828f-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:43.999 DEBUG:boto:b'jdpiak-16999falsedummydM1h1LlChQwJOE3oHBYl7Qu-tamZZbJtrue2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1-zshRkRrxKU0EJVPpVBUsfYl8JR7HFRtrue2022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone23vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5true2022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eEtrue2022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:44.000 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:03:44.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:03:44.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:03:44.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:03:44.002 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:03:44.002 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:44.003 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:44.003 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:44.004 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:44.004 DEBUG:boto:Method: GET 2022-09-12T19:03:44.005 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:44.005 DEBUG:boto:Data: 2022-09-12T19:03:44.006 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.006 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.007 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.007 DEBUG:boto:Params: {} 2022-09-12T19:03:44.008 DEBUG:boto:Token: None 2022-09-12T19:03:44.009 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/?versions 2022-09-12T19:03:44.009 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kAijDXKnO5UDBivHePBzRREVyr0= 2022-09-12T19:03:44.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kAijDXKnO5UDBivHePBzRREVyr0=', 'Content-Length': '0'} 2022-09-12T19:03:44.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6c0706dad2c46fa-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.013 DEBUG:boto:b'jdpiak-16999falsedummydM1h1LlChQwJOE3oHBYl7Qu-tamZZbJtrue2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1-zshRkRrxKU0EJVPpVBUsfYl8JR7HFRtrue2022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone23vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5true2022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eEtrue2022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:44.014 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:03:44.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:03:44.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:03:44.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:03:44.016 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:44.016 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:44.017 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:44.017 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:44.018 DEBUG:boto:Method: GET 2022-09-12T19:03:44.018 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:44.019 DEBUG:boto:Data: 2022-09-12T19:03:44.019 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.020 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.020 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.021 DEBUG:boto:Params: {} 2022-09-12T19:03:44.021 DEBUG:boto:Token: None 2022-09-12T19:03:44.022 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/?versions 2022-09-12T19:03:44.023 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kAijDXKnO5UDBivHePBzRREVyr0= 2022-09-12T19:03:44.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kAijDXKnO5UDBivHePBzRREVyr0=', 'Content-Length': '0'} 2022-09-12T19:03:44.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005df2a7fad41b6684-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.027 DEBUG:boto:b'jdpiak-16999falsedummydM1h1LlChQwJOE3oHBYl7Qu-tamZZbJtrue2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1-zshRkRrxKU0EJVPpVBUsfYl8JR7HFRtrue2022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone23vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5true2022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eEtrue2022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:44.027 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:44.028 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:44.029 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:44.029 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:44.029 DEBUG:boto:Method: GET 2022-09-12T19:03:44.030 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:44.031 DEBUG:boto:Data: 2022-09-12T19:03:44.031 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.032 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.032 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.033 DEBUG:boto:Params: {} 2022-09-12T19:03:44.033 DEBUG:boto:Token: None 2022-09-12T19:03:44.034 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/?versions 2022-09-12T19:03:44.034 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kAijDXKnO5UDBivHePBzRREVyr0= 2022-09-12T19:03:44.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kAijDXKnO5UDBivHePBzRREVyr0=', 'Content-Length': '0'} 2022-09-12T19:03:44.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d1bd6bac8ee32f9-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.038 DEBUG:boto:b'jdpiak-16999falsedummydM1h1LlChQwJOE3oHBYl7Qu-tamZZbJtrue2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1-zshRkRrxKU0EJVPpVBUsfYl8JR7HFRtrue2022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone23vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5true2022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eEtrue2022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:44.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:44.039 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:44.040 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:44.041 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.041 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.042 DEBUG:boto:Method: GET 2022-09-12T19:03:44.042 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.043 DEBUG:boto:Data: 2022-09-12T19:03:44.043 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.044 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.044 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.045 DEBUG:boto:Params: {} 2022-09-12T19:03:44.045 DEBUG:boto:Token: None 2022-09-12T19:03:44.046 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.047 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BsYKIvr9uSdLkHFpLVwZhvy9W6U= 2022-09-12T19:03:44.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BsYKIvr9uSdLkHFpLVwZhvy9W6U=', 'Content-Length': '0'} 2022-09-12T19:03:44.049 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000034c068713510feb7-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.050 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:44.050 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:44.051 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.051 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.052 DEBUG:boto:Method: GET 2022-09-12T19:03:44.053 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.053 DEBUG:boto:Data: 2022-09-12T19:03:44.054 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.054 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.055 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.055 DEBUG:boto:Params: {} 2022-09-12T19:03:44.056 DEBUG:boto:Token: None 2022-09-12T19:03:44.057 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.057 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BsYKIvr9uSdLkHFpLVwZhvy9W6U= 2022-09-12T19:03:44.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BsYKIvr9uSdLkHFpLVwZhvy9W6U=', 'Content-Length': '0'} 2022-09-12T19:03:44.060 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b25e3a7a472c5b0f-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.061 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:44.061 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:44.062 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.062 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.063 DEBUG:boto:Method: HEAD 2022-09-12T19:03:44.063 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.064 DEBUG:boto:Data: 2022-09-12T19:03:44.064 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.065 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.065 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.066 DEBUG:boto:Params: {} 2022-09-12T19:03:44.066 DEBUG:boto:Token: None 2022-09-12T19:03:44.067 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.067 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UiJBl5xa8p+L6YPprMO0fyxgtvQ= 2022-09-12T19:03:44.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UiJBl5xa8p+L6YPprMO0fyxgtvQ=', 'Content-Length': '0'} 2022-09-12T19:03:44.069 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000302436547245a69a-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.070 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:44.071 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:44.071 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.072 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.072 DEBUG:boto:Method: HEAD 2022-09-12T19:03:44.073 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.073 DEBUG:boto:Data: 2022-09-12T19:03:44.074 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.074 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.075 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.075 DEBUG:boto:Params: {} 2022-09-12T19:03:44.076 DEBUG:boto:Token: None 2022-09-12T19:03:44.076 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.077 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UiJBl5xa8p+L6YPprMO0fyxgtvQ= 2022-09-12T19:03:44.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UiJBl5xa8p+L6YPprMO0fyxgtvQ=', 'Content-Length': '0'} 2022-09-12T19:03:44.080 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000772e23ff0fc3a733-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:44.081 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:44.081 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:44.082 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.082 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.083 DEBUG:boto:Method: GET 2022-09-12T19:03:44.083 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.084 DEBUG:boto:Data: 2022-09-12T19:03:44.085 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.085 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.086 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.086 DEBUG:boto:Params: {} 2022-09-12T19:03:44.087 DEBUG:boto:Token: None 2022-09-12T19:03:44.087 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.088 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BsYKIvr9uSdLkHFpLVwZhvy9W6U= 2022-09-12T19:03:44.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BsYKIvr9uSdLkHFpLVwZhvy9W6U=', 'Content-Length': '0'} 2022-09-12T19:03:44.091 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000022655e405088a513-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.091 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:44.092 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:44.092 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.093 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.093 DEBUG:boto:Method: GET 2022-09-12T19:03:44.094 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.094 DEBUG:boto:Data: 2022-09-12T19:03:44.095 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.095 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.096 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.096 DEBUG:boto:Params: {} 2022-09-12T19:03:44.097 DEBUG:boto:Token: None 2022-09-12T19:03:44.098 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.098 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BsYKIvr9uSdLkHFpLVwZhvy9W6U= 2022-09-12T19:03:44.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BsYKIvr9uSdLkHFpLVwZhvy9W6U=', 'Content-Length': '0'} 2022-09-12T19:03:44.101 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ee16b9887386414f-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.101 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:44.102 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:44.103 DEBUG:boto:path=/jdpiak-16/?prefix=dummy 2022-09-12T19:03:44.103 DEBUG:boto:auth_path=/jdpiak-16/?prefix=dummy 2022-09-12T19:03:44.104 DEBUG:boto:Method: GET 2022-09-12T19:03:44.104 DEBUG:boto:Path: /jdpiak-16/?prefix=dummy 2022-09-12T19:03:44.105 DEBUG:boto:Data: 2022-09-12T19:03:44.105 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.106 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.106 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.107 DEBUG:boto:Params: {} 2022-09-12T19:03:44.107 DEBUG:boto:Token: None 2022-09-12T19:03:44.108 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/ 2022-09-12T19:03:44.108 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM= 2022-09-12T19:03:44.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM=', 'Content-Length': '0'} 2022-09-12T19:03:44.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0c2d271aa039b65-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.154 DEBUG:boto:b'jdpiak-16dummy1000falsedummy2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:44.155 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:44.156 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:44.156 DEBUG:boto:path=/jdpiak-16/?prefix=dummy 2022-09-12T19:03:44.157 DEBUG:boto:auth_path=/jdpiak-16/?prefix=dummy 2022-09-12T19:03:44.157 DEBUG:boto:Method: GET 2022-09-12T19:03:44.158 DEBUG:boto:Path: /jdpiak-16/?prefix=dummy 2022-09-12T19:03:44.158 DEBUG:boto:Data: 2022-09-12T19:03:44.159 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.160 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.160 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.161 DEBUG:boto:Params: {} 2022-09-12T19:03:44.161 DEBUG:boto:Token: None 2022-09-12T19:03:44.162 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/ 2022-09-12T19:03:44.164 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM= 2022-09-12T19:03:44.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM=', 'Content-Length': '0'} 2022-09-12T19:03:44.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b474e7467c26b42b-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.169 DEBUG:boto:b'jdpiak-16dummy1000falsedummy2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:44.170 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:44.170 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:44.171 DEBUG:boto:Method: HEAD 2022-09-12T19:03:44.171 DEBUG:boto:Path: /jdpiak-16/dummy 2022-09-12T19:03:44.172 DEBUG:boto:Data: 2022-09-12T19:03:44.173 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.173 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.174 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.174 DEBUG:boto:Params: {} 2022-09-12T19:03:44.175 DEBUG:boto:Token: None 2022-09-12T19:03:44.175 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/dummy 2022-09-12T19:03:44.176 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b7RGGkRg03zifdzCheXUpN2xfDc= 2022-09-12T19:03:44.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b7RGGkRg03zifdzCheXUpN2xfDc=', 'Content-Length': '0'} 2022-09-12T19:03:44.179 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f5e9d870bf5e320b-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.180 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:44.180 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:44.181 DEBUG:boto:Method: HEAD 2022-09-12T19:03:44.181 DEBUG:boto:Path: /jdpiak-16/dummy 2022-09-12T19:03:44.182 DEBUG:boto:Data: 2022-09-12T19:03:44.182 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.183 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.183 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.184 DEBUG:boto:Params: {} 2022-09-12T19:03:44.185 DEBUG:boto:Token: None 2022-09-12T19:03:44.187 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/dummy 2022-09-12T19:03:44.187 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b7RGGkRg03zifdzCheXUpN2xfDc= 2022-09-12T19:03:44.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b7RGGkRg03zifdzCheXUpN2xfDc=', 'Content-Length': '0'} 2022-09-12T19:03:44.190 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000be2562c9185e7afa-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:44.191 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:44.192 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:44.193 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.193 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.194 DEBUG:boto:Method: GET 2022-09-12T19:03:44.194 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.195 DEBUG:boto:Data: 2022-09-12T19:03:44.196 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.196 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.197 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.198 DEBUG:boto:Params: {} 2022-09-12T19:03:44.198 DEBUG:boto:Token: None 2022-09-12T19:03:44.199 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.199 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BsYKIvr9uSdLkHFpLVwZhvy9W6U= 2022-09-12T19:03:44.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BsYKIvr9uSdLkHFpLVwZhvy9W6U=', 'Content-Length': '0'} 2022-09-12T19:03:44.202 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004762ba9528057de3-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.203 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:44.203 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:44.204 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.205 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.206 DEBUG:boto:Method: GET 2022-09-12T19:03:44.206 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.207 DEBUG:boto:Data: 2022-09-12T19:03:44.208 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.209 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.210 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.210 DEBUG:boto:Params: {} 2022-09-12T19:03:44.211 DEBUG:boto:Token: None 2022-09-12T19:03:44.212 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:44.212 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BsYKIvr9uSdLkHFpLVwZhvy9W6U= 2022-09-12T19:03:44.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BsYKIvr9uSdLkHFpLVwZhvy9W6U=', 'Content-Length': '0'} 2022-09-12T19:03:44.215 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000908aefb160d37d9d-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.215 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:44.216 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:44.217 DEBUG:boto:path=/jdpiak-16/?prefix=dummy 2022-09-12T19:03:44.217 DEBUG:boto:auth_path=/jdpiak-16/?prefix=dummy 2022-09-12T19:03:44.218 DEBUG:boto:Method: GET 2022-09-12T19:03:44.218 DEBUG:boto:Path: /jdpiak-16/?prefix=dummy 2022-09-12T19:03:44.219 DEBUG:boto:Data: 2022-09-12T19:03:44.220 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.220 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.221 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.221 DEBUG:boto:Params: {} 2022-09-12T19:03:44.222 DEBUG:boto:Token: None 2022-09-12T19:03:44.222 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/ 2022-09-12T19:03:44.223 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM= 2022-09-12T19:03:44.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM=', 'Content-Length': '0'} 2022-09-12T19:03:44.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c037e13a288753b-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.270 DEBUG:boto:b'jdpiak-16dummy1000falsedummy2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:44.271 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:44.272 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:44.272 DEBUG:boto:path=/jdpiak-16/?prefix=dummy 2022-09-12T19:03:44.273 DEBUG:boto:auth_path=/jdpiak-16/?prefix=dummy 2022-09-12T19:03:44.273 DEBUG:boto:Method: GET 2022-09-12T19:03:44.274 DEBUG:boto:Path: /jdpiak-16/?prefix=dummy 2022-09-12T19:03:44.274 DEBUG:boto:Data: 2022-09-12T19:03:44.275 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.276 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.276 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.277 DEBUG:boto:Params: {} 2022-09-12T19:03:44.277 DEBUG:boto:Token: None 2022-09-12T19:03:44.278 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/ 2022-09-12T19:03:44.279 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM= 2022-09-12T19:03:44.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM=', 'Content-Length': '0'} 2022-09-12T19:03:44.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b8a51f5bdecf102-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.283 DEBUG:boto:b'jdpiak-16dummy1000falsedummy2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:44.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:44.284 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:44.285 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:44.286 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.286 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.287 DEBUG:boto:Method: GET 2022-09-12T19:03:44.287 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.288 DEBUG:boto:Data: 2022-09-12T19:03:44.288 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.289 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.290 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.290 DEBUG:boto:Params: {} 2022-09-12T19:03:44.291 DEBUG:boto:Token: None 2022-09-12T19:03:44.292 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.292 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/NUgHBO0qvt+Tptbps6hKzbo5rA= 2022-09-12T19:03:44.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/NUgHBO0qvt+Tptbps6hKzbo5rA=', 'Content-Length': '0'} 2022-09-12T19:03:44.295 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000272ca694ac26cfe7-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.296 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:44.296 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:44.297 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.297 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.298 DEBUG:boto:Method: GET 2022-09-12T19:03:44.298 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.299 DEBUG:boto:Data: 2022-09-12T19:03:44.299 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.300 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.300 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.301 DEBUG:boto:Params: {} 2022-09-12T19:03:44.302 DEBUG:boto:Token: None 2022-09-12T19:03:44.302 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.303 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/NUgHBO0qvt+Tptbps6hKzbo5rA= 2022-09-12T19:03:44.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/NUgHBO0qvt+Tptbps6hKzbo5rA=', 'Content-Length': '0'} 2022-09-12T19:03:44.305 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa3902f060fea6d0-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.306 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:44.307 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:44.307 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.308 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.308 DEBUG:boto:Method: HEAD 2022-09-12T19:03:44.309 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.309 DEBUG:boto:Data: 2022-09-12T19:03:44.310 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.311 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.311 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.312 DEBUG:boto:Params: {} 2022-09-12T19:03:44.312 DEBUG:boto:Token: None 2022-09-12T19:03:44.313 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.313 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Mfuubs+Z/S9HafWoYOFWuMg/xgg= 2022-09-12T19:03:44.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Mfuubs+Z/S9HafWoYOFWuMg/xgg=', 'Content-Length': '0'} 2022-09-12T19:03:44.316 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b2d9e773348592a-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.317 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:44.317 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:44.318 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.318 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.319 DEBUG:boto:Method: HEAD 2022-09-12T19:03:44.319 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.320 DEBUG:boto:Data: 2022-09-12T19:03:44.320 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.321 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.321 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.322 DEBUG:boto:Params: {} 2022-09-12T19:03:44.322 DEBUG:boto:Token: None 2022-09-12T19:03:44.323 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.323 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Mfuubs+Z/S9HafWoYOFWuMg/xgg= 2022-09-12T19:03:44.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Mfuubs+Z/S9HafWoYOFWuMg/xgg=', 'Content-Length': '0'} 2022-09-12T19:03:44.326 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024b17cbd6e23b036-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:44.327 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:44.328 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:44.328 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.329 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.329 DEBUG:boto:Method: GET 2022-09-12T19:03:44.330 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.330 DEBUG:boto:Data: 2022-09-12T19:03:44.331 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.331 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.332 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.333 DEBUG:boto:Params: {} 2022-09-12T19:03:44.333 DEBUG:boto:Token: None 2022-09-12T19:03:44.334 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.334 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/NUgHBO0qvt+Tptbps6hKzbo5rA= 2022-09-12T19:03:44.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/NUgHBO0qvt+Tptbps6hKzbo5rA=', 'Content-Length': '0'} 2022-09-12T19:03:44.337 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a226052253c78fe-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.338 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:44.338 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:44.339 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.339 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.340 DEBUG:boto:Method: GET 2022-09-12T19:03:44.340 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.341 DEBUG:boto:Data: 2022-09-12T19:03:44.342 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.343 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.344 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.344 DEBUG:boto:Params: {} 2022-09-12T19:03:44.345 DEBUG:boto:Token: None 2022-09-12T19:03:44.346 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.346 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/NUgHBO0qvt+Tptbps6hKzbo5rA= 2022-09-12T19:03:44.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/NUgHBO0qvt+Tptbps6hKzbo5rA=', 'Content-Length': '0'} 2022-09-12T19:03:44.349 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf64d1160a261794-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.350 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:44.350 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:44.351 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:44.351 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:44.352 DEBUG:boto:Method: GET 2022-09-12T19:03:44.352 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:44.353 DEBUG:boto:Data: 2022-09-12T19:03:44.354 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.354 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.355 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.355 DEBUG:boto:Params: {} 2022-09-12T19:03:44.356 DEBUG:boto:Token: None 2022-09-12T19:03:44.357 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/ 2022-09-12T19:03:44.357 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM= 2022-09-12T19:03:44.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM=', 'Content-Length': '0'} 2022-09-12T19:03:44.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000875491d1e512f30f-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.361 DEBUG:boto:b'jdpiak-16obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:44.362 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:44.363 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:44.363 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:44.364 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:44.365 DEBUG:boto:Method: GET 2022-09-12T19:03:44.366 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:44.367 DEBUG:boto:Data: 2022-09-12T19:03:44.367 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.368 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.369 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.369 DEBUG:boto:Params: {} 2022-09-12T19:03:44.370 DEBUG:boto:Token: None 2022-09-12T19:03:44.371 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/ 2022-09-12T19:03:44.371 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM= 2022-09-12T19:03:44.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM=', 'Content-Length': '0'} 2022-09-12T19:03:44.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000286633be7a061d4d-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.375 DEBUG:boto:b'jdpiak-16obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:44.376 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:44.377 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:44.377 DEBUG:boto:Method: HEAD 2022-09-12T19:03:44.378 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1 2022-09-12T19:03:44.378 DEBUG:boto:Data: 2022-09-12T19:03:44.379 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.379 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.380 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.380 DEBUG:boto:Params: {} 2022-09-12T19:03:44.381 DEBUG:boto:Token: None 2022-09-12T19:03:44.381 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone1 2022-09-12T19:03:44.382 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ygIHEyV7GuVQJCI5Sa2058G9B8Y= 2022-09-12T19:03:44.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ygIHEyV7GuVQJCI5Sa2058G9B8Y=', 'Content-Length': '0'} 2022-09-12T19:03:44.385 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032d8dfa72d23d2b9-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.386 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:44.386 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:44.387 DEBUG:boto:Method: HEAD 2022-09-12T19:03:44.398 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1 2022-09-12T19:03:44.399 DEBUG:boto:Data: 2022-09-12T19:03:44.400 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.400 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.401 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.401 DEBUG:boto:Params: {} 2022-09-12T19:03:44.402 DEBUG:boto:Token: None 2022-09-12T19:03:44.402 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone1 2022-09-12T19:03:44.403 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ygIHEyV7GuVQJCI5Sa2058G9B8Y= 2022-09-12T19:03:44.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ygIHEyV7GuVQJCI5Sa2058G9B8Y=', 'Content-Length': '0'} 2022-09-12T19:03:44.406 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073db98845c87a2e2-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:44.407 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:44.408 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:44.408 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.409 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.409 DEBUG:boto:Method: GET 2022-09-12T19:03:44.410 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.410 DEBUG:boto:Data: 2022-09-12T19:03:44.411 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.412 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.412 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.413 DEBUG:boto:Params: {} 2022-09-12T19:03:44.413 DEBUG:boto:Token: None 2022-09-12T19:03:44.414 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.415 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/NUgHBO0qvt+Tptbps6hKzbo5rA= 2022-09-12T19:03:44.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/NUgHBO0qvt+Tptbps6hKzbo5rA=', 'Content-Length': '0'} 2022-09-12T19:03:44.417 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cfd78165db85ff2a-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.419 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:44.420 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:44.421 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.422 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.423 DEBUG:boto:Method: GET 2022-09-12T19:03:44.424 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.425 DEBUG:boto:Data: 2022-09-12T19:03:44.425 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.426 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.427 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.428 DEBUG:boto:Params: {} 2022-09-12T19:03:44.429 DEBUG:boto:Token: None 2022-09-12T19:03:44.430 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:44.430 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/NUgHBO0qvt+Tptbps6hKzbo5rA= 2022-09-12T19:03:44.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/NUgHBO0qvt+Tptbps6hKzbo5rA=', 'Content-Length': '0'} 2022-09-12T19:03:44.433 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000167d3b33157e8e75-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.478 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:44.479 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:44.480 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:44.480 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:44.481 DEBUG:boto:Method: GET 2022-09-12T19:03:44.481 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:44.482 DEBUG:boto:Data: 2022-09-12T19:03:44.482 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.483 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.483 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.484 DEBUG:boto:Params: {} 2022-09-12T19:03:44.485 DEBUG:boto:Token: None 2022-09-12T19:03:44.485 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/ 2022-09-12T19:03:44.486 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM= 2022-09-12T19:03:44.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM=', 'Content-Length': '0'} 2022-09-12T19:03:44.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068445bcf1ebf2a07-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.490 DEBUG:boto:b'jdpiak-16obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:44.491 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:44.491 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:44.492 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:44.492 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:44.493 DEBUG:boto:Method: GET 2022-09-12T19:03:44.494 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:44.494 DEBUG:boto:Data: 2022-09-12T19:03:44.495 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.495 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.496 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.496 DEBUG:boto:Params: {} 2022-09-12T19:03:44.497 DEBUG:boto:Token: None 2022-09-12T19:03:44.498 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/ 2022-09-12T19:03:44.498 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM= 2022-09-12T19:03:44.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM=', 'Content-Length': '0'} 2022-09-12T19:03:44.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ea36b493b24634b-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.503 DEBUG:boto:b'jdpiak-16obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:44.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:44.504 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:44.505 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:44.505 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.506 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.506 DEBUG:boto:Method: GET 2022-09-12T19:03:44.507 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.508 DEBUG:boto:Data: 2022-09-12T19:03:44.508 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.509 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.509 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.510 DEBUG:boto:Params: {} 2022-09-12T19:03:44.510 DEBUG:boto:Token: None 2022-09-12T19:03:44.511 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.512 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8Kxnye/9qVMwWfzrf6o6NXw7UqU= 2022-09-12T19:03:44.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8Kxnye/9qVMwWfzrf6o6NXw7UqU=', 'Content-Length': '0'} 2022-09-12T19:03:44.515 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b4c19b95d29781f5-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.515 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:44.516 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:44.517 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.517 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.518 DEBUG:boto:Method: GET 2022-09-12T19:03:44.518 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.519 DEBUG:boto:Data: 2022-09-12T19:03:44.519 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.520 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.520 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.521 DEBUG:boto:Params: {} 2022-09-12T19:03:44.522 DEBUG:boto:Token: None 2022-09-12T19:03:44.522 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.523 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8Kxnye/9qVMwWfzrf6o6NXw7UqU= 2022-09-12T19:03:44.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8Kxnye/9qVMwWfzrf6o6NXw7UqU=', 'Content-Length': '0'} 2022-09-12T19:03:44.526 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b313f9b2fca69ff-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.527 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:44.527 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:44.528 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.528 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.529 DEBUG:boto:Method: HEAD 2022-09-12T19:03:44.529 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.529 DEBUG:boto:Data: 2022-09-12T19:03:44.530 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.530 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.531 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.531 DEBUG:boto:Params: {} 2022-09-12T19:03:44.532 DEBUG:boto:Token: None 2022-09-12T19:03:44.533 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.534 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JJqUGvCbMxaewYgV67FHBbH22Ow= 2022-09-12T19:03:44.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JJqUGvCbMxaewYgV67FHBbH22Ow=', 'Content-Length': '0'} 2022-09-12T19:03:44.537 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dace6af2e068b42b-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.539 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:44.540 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:44.541 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.542 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.543 DEBUG:boto:Method: HEAD 2022-09-12T19:03:44.544 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.545 DEBUG:boto:Data: 2022-09-12T19:03:44.546 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.547 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.547 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.548 DEBUG:boto:Params: {} 2022-09-12T19:03:44.549 DEBUG:boto:Token: None 2022-09-12T19:03:44.550 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.551 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JJqUGvCbMxaewYgV67FHBbH22Ow= 2022-09-12T19:03:44.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JJqUGvCbMxaewYgV67FHBbH22Ow=', 'Content-Length': '0'} 2022-09-12T19:03:44.554 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000089cd793bfa7159e-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:44.556 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:44.557 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:44.558 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.559 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.560 DEBUG:boto:Method: GET 2022-09-12T19:03:44.561 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.561 DEBUG:boto:Data: 2022-09-12T19:03:44.562 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.563 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.564 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.565 DEBUG:boto:Params: {} 2022-09-12T19:03:44.566 DEBUG:boto:Token: None 2022-09-12T19:03:44.567 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.568 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8Kxnye/9qVMwWfzrf6o6NXw7UqU= 2022-09-12T19:03:44.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8Kxnye/9qVMwWfzrf6o6NXw7UqU=', 'Content-Length': '0'} 2022-09-12T19:03:44.571 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042594fdce1b035e3-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.573 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:44.573 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:44.574 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.575 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.576 DEBUG:boto:Method: GET 2022-09-12T19:03:44.577 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.578 DEBUG:boto:Data: 2022-09-12T19:03:44.579 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.580 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.580 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.581 DEBUG:boto:Params: {} 2022-09-12T19:03:44.582 DEBUG:boto:Token: None 2022-09-12T19:03:44.583 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.584 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8Kxnye/9qVMwWfzrf6o6NXw7UqU= 2022-09-12T19:03:44.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8Kxnye/9qVMwWfzrf6o6NXw7UqU=', 'Content-Length': '0'} 2022-09-12T19:03:44.588 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd759dc8b3c0eac0-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.630 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:44.631 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:44.631 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:44.632 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:44.633 DEBUG:boto:Method: GET 2022-09-12T19:03:44.633 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:44.634 DEBUG:boto:Data: 2022-09-12T19:03:44.634 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.635 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.635 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.636 DEBUG:boto:Params: {} 2022-09-12T19:03:44.636 DEBUG:boto:Token: None 2022-09-12T19:03:44.637 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/ 2022-09-12T19:03:44.637 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM= 2022-09-12T19:03:44.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM=', 'Content-Length': '0'} 2022-09-12T19:03:44.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000195e77042c361079-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.642 DEBUG:boto:b'jdpiak-16obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:44.643 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:44.643 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:44.644 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:44.644 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:44.645 DEBUG:boto:Method: GET 2022-09-12T19:03:44.645 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:44.646 DEBUG:boto:Data: 2022-09-12T19:03:44.647 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.647 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.648 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.649 DEBUG:boto:Params: {} 2022-09-12T19:03:44.649 DEBUG:boto:Token: None 2022-09-12T19:03:44.650 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/ 2022-09-12T19:03:44.651 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM= 2022-09-12T19:03:44.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM=', 'Content-Length': '0'} 2022-09-12T19:03:44.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bef8585348ff13b6-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.655 DEBUG:boto:b'jdpiak-16obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:44.655 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:44.656 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:44.656 DEBUG:boto:Method: HEAD 2022-09-12T19:03:44.657 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2 2022-09-12T19:03:44.657 DEBUG:boto:Data: 2022-09-12T19:03:44.658 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.658 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.659 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.659 DEBUG:boto:Params: {} 2022-09-12T19:03:44.660 DEBUG:boto:Token: None 2022-09-12T19:03:44.661 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone2 2022-09-12T19:03:44.662 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+REbcJTRLOxSPtlbdbIDHAwDAVo= 2022-09-12T19:03:44.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+REbcJTRLOxSPtlbdbIDHAwDAVo=', 'Content-Length': '0'} 2022-09-12T19:03:44.665 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2c8e25399e57eec-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.666 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:44.667 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:44.667 DEBUG:boto:Method: HEAD 2022-09-12T19:03:44.668 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2 2022-09-12T19:03:44.668 DEBUG:boto:Data: 2022-09-12T19:03:44.669 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.669 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.670 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.671 DEBUG:boto:Params: {} 2022-09-12T19:03:44.671 DEBUG:boto:Token: None 2022-09-12T19:03:44.672 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone2 2022-09-12T19:03:44.672 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+REbcJTRLOxSPtlbdbIDHAwDAVo= 2022-09-12T19:03:44.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+REbcJTRLOxSPtlbdbIDHAwDAVo=', 'Content-Length': '0'} 2022-09-12T19:03:44.676 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001bddf5e922fff03b-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:44.677 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:44.678 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:44.678 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.679 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.679 DEBUG:boto:Method: GET 2022-09-12T19:03:44.680 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.680 DEBUG:boto:Data: 2022-09-12T19:03:44.681 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.682 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.682 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.683 DEBUG:boto:Params: {} 2022-09-12T19:03:44.684 DEBUG:boto:Token: None 2022-09-12T19:03:44.684 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.685 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8Kxnye/9qVMwWfzrf6o6NXw7UqU= 2022-09-12T19:03:44.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8Kxnye/9qVMwWfzrf6o6NXw7UqU=', 'Content-Length': '0'} 2022-09-12T19:03:44.688 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df5945263e342637-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.688 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:44.689 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:44.690 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.690 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.691 DEBUG:boto:Method: GET 2022-09-12T19:03:44.691 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.692 DEBUG:boto:Data: 2022-09-12T19:03:44.692 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.693 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.694 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.694 DEBUG:boto:Params: {} 2022-09-12T19:03:44.695 DEBUG:boto:Token: None 2022-09-12T19:03:44.695 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:44.696 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8Kxnye/9qVMwWfzrf6o6NXw7UqU= 2022-09-12T19:03:44.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8Kxnye/9qVMwWfzrf6o6NXw7UqU=', 'Content-Length': '0'} 2022-09-12T19:03:44.699 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5109ca456f9428f-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.699 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:44.700 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:44.701 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:44.702 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:44.703 DEBUG:boto:Method: GET 2022-09-12T19:03:44.704 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:44.705 DEBUG:boto:Data: 2022-09-12T19:03:44.706 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.707 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.708 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.709 DEBUG:boto:Params: {} 2022-09-12T19:03:44.710 DEBUG:boto:Token: None 2022-09-12T19:03:44.711 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/ 2022-09-12T19:03:44.712 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM= 2022-09-12T19:03:44.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM=', 'Content-Length': '0'} 2022-09-12T19:03:44.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c3bd586a791e7c3-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.718 DEBUG:boto:b'jdpiak-16obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:44.720 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:44.721 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:44.722 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:44.723 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:44.724 DEBUG:boto:Method: GET 2022-09-12T19:03:44.725 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:44.726 DEBUG:boto:Data: 2022-09-12T19:03:44.727 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.728 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.729 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.730 DEBUG:boto:Params: {} 2022-09-12T19:03:44.731 DEBUG:boto:Token: None 2022-09-12T19:03:44.732 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/ 2022-09-12T19:03:44.733 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM= 2022-09-12T19:03:44.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM=', 'Content-Length': '0'} 2022-09-12T19:03:44.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009224b3038638eaf5-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.778 DEBUG:boto:b'jdpiak-16obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:44.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:44.780 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:44.780 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:44.781 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.781 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.782 DEBUG:boto:Method: GET 2022-09-12T19:03:44.782 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.783 DEBUG:boto:Data: 2022-09-12T19:03:44.783 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.784 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.785 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.785 DEBUG:boto:Params: {} 2022-09-12T19:03:44.786 DEBUG:boto:Token: None 2022-09-12T19:03:44.786 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.787 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qpXLLf5vYbc2Q9eZ3W+x3Cu+UrY= 2022-09-12T19:03:44.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qpXLLf5vYbc2Q9eZ3W+x3Cu+UrY=', 'Content-Length': '0'} 2022-09-12T19:03:44.790 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038b5b550604f583c-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.791 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:44.791 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:44.792 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.792 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.793 DEBUG:boto:Method: GET 2022-09-12T19:03:44.793 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.794 DEBUG:boto:Data: 2022-09-12T19:03:44.794 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.795 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.796 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.796 DEBUG:boto:Params: {} 2022-09-12T19:03:44.797 DEBUG:boto:Token: None 2022-09-12T19:03:44.797 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.798 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qpXLLf5vYbc2Q9eZ3W+x3Cu+UrY= 2022-09-12T19:03:44.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qpXLLf5vYbc2Q9eZ3W+x3Cu+UrY=', 'Content-Length': '0'} 2022-09-12T19:03:44.801 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be91596ff6cafb09-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.802 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:44.802 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:44.803 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.803 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.804 DEBUG:boto:Method: HEAD 2022-09-12T19:03:44.804 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.805 DEBUG:boto:Data: 2022-09-12T19:03:44.805 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.806 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.806 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.807 DEBUG:boto:Params: {} 2022-09-12T19:03:44.807 DEBUG:boto:Token: None 2022-09-12T19:03:44.808 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.808 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X4OjvG9IdOeOj0d43rWpbjClfuE= 2022-09-12T19:03:44.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X4OjvG9IdOeOj0d43rWpbjClfuE=', 'Content-Length': '0'} 2022-09-12T19:03:44.811 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077c42178fbd45b11-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.812 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:44.813 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:44.813 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.814 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.814 DEBUG:boto:Method: HEAD 2022-09-12T19:03:44.815 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.815 DEBUG:boto:Data: 2022-09-12T19:03:44.816 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.816 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.817 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.817 DEBUG:boto:Params: {} 2022-09-12T19:03:44.818 DEBUG:boto:Token: None 2022-09-12T19:03:44.818 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.819 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X4OjvG9IdOeOj0d43rWpbjClfuE= 2022-09-12T19:03:44.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X4OjvG9IdOeOj0d43rWpbjClfuE=', 'Content-Length': '0'} 2022-09-12T19:03:44.821 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022d043dda2ee9d88-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:44.823 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:44.823 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:44.824 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.824 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.825 DEBUG:boto:Method: GET 2022-09-12T19:03:44.826 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.826 DEBUG:boto:Data: 2022-09-12T19:03:44.827 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.827 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.828 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.828 DEBUG:boto:Params: {} 2022-09-12T19:03:44.829 DEBUG:boto:Token: None 2022-09-12T19:03:44.829 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.830 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qpXLLf5vYbc2Q9eZ3W+x3Cu+UrY= 2022-09-12T19:03:44.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qpXLLf5vYbc2Q9eZ3W+x3Cu+UrY=', 'Content-Length': '0'} 2022-09-12T19:03:44.833 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054cef395ea7988f2-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.874 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:44.875 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:44.876 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.876 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.877 DEBUG:boto:Method: GET 2022-09-12T19:03:44.877 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.879 DEBUG:boto:Data: 2022-09-12T19:03:44.879 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.880 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.880 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.881 DEBUG:boto:Params: {} 2022-09-12T19:03:44.882 DEBUG:boto:Token: None 2022-09-12T19:03:44.883 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.883 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qpXLLf5vYbc2Q9eZ3W+x3Cu+UrY= 2022-09-12T19:03:44.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qpXLLf5vYbc2Q9eZ3W+x3Cu+UrY=', 'Content-Length': '0'} 2022-09-12T19:03:44.887 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ebb5c614271070b-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.888 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:44.888 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:44.889 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:44.889 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:44.890 DEBUG:boto:Method: GET 2022-09-12T19:03:44.890 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:44.891 DEBUG:boto:Data: 2022-09-12T19:03:44.891 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.892 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.892 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.893 DEBUG:boto:Params: {} 2022-09-12T19:03:44.894 DEBUG:boto:Token: None 2022-09-12T19:03:44.895 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/ 2022-09-12T19:03:44.895 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM= 2022-09-12T19:03:44.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM=', 'Content-Length': '0'} 2022-09-12T19:03:44.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ae7a61b50f673b2-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.901 DEBUG:boto:b'jdpiak-16obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:44.902 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:44.902 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:44.903 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:44.903 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:44.904 DEBUG:boto:Method: GET 2022-09-12T19:03:44.904 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:44.905 DEBUG:boto:Data: 2022-09-12T19:03:44.905 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.906 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.906 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.907 DEBUG:boto:Params: {} 2022-09-12T19:03:44.908 DEBUG:boto:Token: None 2022-09-12T19:03:44.908 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/ 2022-09-12T19:03:44.909 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM= 2022-09-12T19:03:44.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7p2kp8o+Rn0t4f+jvksIWKBK7iM=', 'Content-Length': '0'} 2022-09-12T19:03:44.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3d3bd58d51c0c5f-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.913 DEBUG:boto:b'jdpiak-16obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:44.914 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:44.914 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:44.915 DEBUG:boto:Method: HEAD 2022-09-12T19:03:44.915 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3 2022-09-12T19:03:44.916 DEBUG:boto:Data: 2022-09-12T19:03:44.917 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.917 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.918 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.918 DEBUG:boto:Params: {} 2022-09-12T19:03:44.919 DEBUG:boto:Token: None 2022-09-12T19:03:44.919 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone3 2022-09-12T19:03:44.920 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lPWiUjVthZ6sCRnh/2qNyQcMENo= 2022-09-12T19:03:44.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lPWiUjVthZ6sCRnh/2qNyQcMENo=', 'Content-Length': '0'} 2022-09-12T19:03:44.923 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007df42699d855bff4-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.924 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:44.924 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:44.925 DEBUG:boto:Method: HEAD 2022-09-12T19:03:44.925 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3 2022-09-12T19:03:44.926 DEBUG:boto:Data: 2022-09-12T19:03:44.926 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.927 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.927 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.928 DEBUG:boto:Params: {} 2022-09-12T19:03:44.929 DEBUG:boto:Token: None 2022-09-12T19:03:44.929 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone3 2022-09-12T19:03:44.930 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lPWiUjVthZ6sCRnh/2qNyQcMENo= 2022-09-12T19:03:44.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lPWiUjVthZ6sCRnh/2qNyQcMENo=', 'Content-Length': '0'} 2022-09-12T19:03:44.933 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f060b62c79313710-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:44.934 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:44.935 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:44.935 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.936 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.936 DEBUG:boto:Method: GET 2022-09-12T19:03:44.937 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.937 DEBUG:boto:Data: 2022-09-12T19:03:44.938 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.939 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:44.939 DEBUG:boto:Port: 8001 2022-09-12T19:03:44.940 DEBUG:boto:Params: {} 2022-09-12T19:03:44.940 DEBUG:boto:Token: None 2022-09-12T19:03:44.941 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.941 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qpXLLf5vYbc2Q9eZ3W+x3Cu+UrY= 2022-09-12T19:03:44.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qpXLLf5vYbc2Q9eZ3W+x3Cu+UrY=', 'Content-Length': '0'} 2022-09-12T19:03:44.944 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c35a0a478b4eb39-00631f8290-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.986 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:44.987 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:44.987 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.988 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.988 DEBUG:boto:Method: GET 2022-09-12T19:03:44.989 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.989 DEBUG:boto:Data: 2022-09-12T19:03:44.990 DEBUG:boto:Headers: {} 2022-09-12T19:03:44.991 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:44.991 DEBUG:boto:Port: 8000 2022-09-12T19:03:44.992 DEBUG:boto:Params: {} 2022-09-12T19:03:44.992 DEBUG:boto:Token: None 2022-09-12T19:03:44.993 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:44 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:44.993 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qpXLLf5vYbc2Q9eZ3W+x3Cu+UrY= 2022-09-12T19:03:44.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qpXLLf5vYbc2Q9eZ3W+x3Cu+UrY=', 'Content-Length': '0'} 2022-09-12T19:03:44.997 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002100eb33829ffc89-00631f8290-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:44.997 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:44.998 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:44.998 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:44.999 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:45.000 DEBUG:boto:Method: GET 2022-09-12T19:03:45.000 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:45.001 DEBUG:boto:Data: 2022-09-12T19:03:45.001 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.002 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.002 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.003 DEBUG:boto:Params: {} 2022-09-12T19:03:45.003 DEBUG:boto:Token: None 2022-09-12T19:03:45.004 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/ 2022-09-12T19:03:45.005 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0= 2022-09-12T19:03:45.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0=', 'Content-Length': '0'} 2022-09-12T19:03:45.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000365dae5f2750012b-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.009 DEBUG:boto:b'jdpiak-16obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:45.010 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:45.010 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:45.011 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:45.011 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:45.012 DEBUG:boto:Method: GET 2022-09-12T19:03:45.012 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:45.013 DEBUG:boto:Data: 2022-09-12T19:03:45.013 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.014 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:45.014 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.015 DEBUG:boto:Params: {} 2022-09-12T19:03:45.015 DEBUG:boto:Token: None 2022-09-12T19:03:45.016 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/ 2022-09-12T19:03:45.016 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0= 2022-09-12T19:03:45.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0=', 'Content-Length': '0'} 2022-09-12T19:03:45.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bfb83172acb04c6-00631f8291-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.020 DEBUG:boto:b'jdpiak-16obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:45.021 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-16 zones={test-zone3, test-zone1} 2022-09-12T19:03:45.022 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-16 zones={test-zone3, test-zone2} 2022-09-12T19:03:45.022 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:45.023 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:45.023 DEBUG:boto:Method: HEAD 2022-09-12T19:03:45.024 DEBUG:boto:Path: /jdpiak-16/ 2022-09-12T19:03:45.024 DEBUG:boto:Data: 2022-09-12T19:03:45.025 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.025 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.026 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.026 DEBUG:boto:Params: {} 2022-09-12T19:03:45.027 DEBUG:boto:Token: None 2022-09-12T19:03:45.028 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/ 2022-09-12T19:03:45.028 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qopuzK8YgMSxZsz4RffwOCDekas= 2022-09-12T19:03:45.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qopuzK8YgMSxZsz4RffwOCDekas=', 'Content-Length': '0'} 2022-09-12T19:03:45.031 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010eb2294b6aa4f6d-00631f8291-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.032 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:45.032 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:45.033 DEBUG:boto:Method: HEAD 2022-09-12T19:03:45.033 DEBUG:boto:Path: /jdpiak-16/ 2022-09-12T19:03:45.034 DEBUG:boto:Data: 2022-09-12T19:03:45.034 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.035 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.035 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.036 DEBUG:boto:Params: {} 2022-09-12T19:03:45.036 DEBUG:boto:Token: None 2022-09-12T19:03:45.037 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/ 2022-09-12T19:03:45.038 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qopuzK8YgMSxZsz4RffwOCDekas= 2022-09-12T19:03:45.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qopuzK8YgMSxZsz4RffwOCDekas=', 'Content-Length': '0'} 2022-09-12T19:03:45.041 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002c51742a7e00536-00631f8291-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.042 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:03:45.042 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:45.043 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:45.043 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:45.044 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:45.044 DEBUG:boto:Method: GET 2022-09-12T19:03:45.045 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:45.045 DEBUG:boto:Data: 2022-09-12T19:03:45.046 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.047 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.047 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.048 DEBUG:boto:Params: {} 2022-09-12T19:03:45.048 DEBUG:boto:Token: None 2022-09-12T19:03:45.049 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/?versions 2022-09-12T19:03:45.050 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gxosgik07CWlc3Z2P9rSZL8U/w4= 2022-09-12T19:03:45.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gxosgik07CWlc3Z2P9rSZL8U/w4=', 'Content-Length': '0'} 2022-09-12T19:03:45.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046e1f5fd9e332314-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.054 DEBUG:boto:b'jdpiak-16999falsedummydM1h1LlChQwJOE3oHBYl7Qu-tamZZbJtrue2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1-zshRkRrxKU0EJVPpVBUsfYl8JR7HFRtrue2022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone23vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5true2022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eEtrue2022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:45.055 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:03:45.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:03:45.056 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:03:45.056 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:03:45.057 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:03:45.057 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:45.058 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:45.058 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:45.059 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:45.060 DEBUG:boto:Method: GET 2022-09-12T19:03:45.061 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:45.062 DEBUG:boto:Data: 2022-09-12T19:03:45.062 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.063 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.063 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.064 DEBUG:boto:Params: {} 2022-09-12T19:03:45.064 DEBUG:boto:Token: None 2022-09-12T19:03:45.065 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/?versions 2022-09-12T19:03:45.065 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gxosgik07CWlc3Z2P9rSZL8U/w4= 2022-09-12T19:03:45.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gxosgik07CWlc3Z2P9rSZL8U/w4=', 'Content-Length': '0'} 2022-09-12T19:03:45.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7b673a2f7ec9fbf-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.070 DEBUG:boto:b'jdpiak-16999falsedummydM1h1LlChQwJOE3oHBYl7Qu-tamZZbJtrue2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1-zshRkRrxKU0EJVPpVBUsfYl8JR7HFRtrue2022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone23vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5true2022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eEtrue2022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:45.070 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:03:45.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:03:45.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:03:45.072 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:03:45.072 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:45.073 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:45.073 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:45.074 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:45.074 DEBUG:boto:Method: GET 2022-09-12T19:03:45.075 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:45.075 DEBUG:boto:Data: 2022-09-12T19:03:45.076 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.076 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.077 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.077 DEBUG:boto:Params: {} 2022-09-12T19:03:45.078 DEBUG:boto:Token: None 2022-09-12T19:03:45.079 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/?versions 2022-09-12T19:03:45.079 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gxosgik07CWlc3Z2P9rSZL8U/w4= 2022-09-12T19:03:45.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gxosgik07CWlc3Z2P9rSZL8U/w4=', 'Content-Length': '0'} 2022-09-12T19:03:45.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047468447b01422df-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.084 DEBUG:boto:b'jdpiak-16999falsedummydM1h1LlChQwJOE3oHBYl7Qu-tamZZbJtrue2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1-zshRkRrxKU0EJVPpVBUsfYl8JR7HFRtrue2022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone23vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5true2022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eEtrue2022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:45.085 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:45.086 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:45.087 DEBUG:boto:path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:45.087 DEBUG:boto:auth_path=/jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:45.088 DEBUG:boto:Method: GET 2022-09-12T19:03:45.089 DEBUG:boto:Path: /jdpiak-16/?versions&max-keys=999 2022-09-12T19:03:45.089 DEBUG:boto:Data: 2022-09-12T19:03:45.090 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.091 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.092 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.092 DEBUG:boto:Params: {} 2022-09-12T19:03:45.093 DEBUG:boto:Token: None 2022-09-12T19:03:45.094 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/?versions 2022-09-12T19:03:45.095 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gxosgik07CWlc3Z2P9rSZL8U/w4= 2022-09-12T19:03:45.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gxosgik07CWlc3Z2P9rSZL8U/w4=', 'Content-Length': '0'} 2022-09-12T19:03:45.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000297b79202cad1118-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.101 DEBUG:boto:b'jdpiak-16999falsedummydM1h1LlChQwJOE3oHBYl7Qu-tamZZbJtrue2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1-zshRkRrxKU0EJVPpVBUsfYl8JR7HFRtrue2022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone23vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5true2022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eEtrue2022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:45.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:45.103 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:45.104 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:45.105 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.106 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.107 DEBUG:boto:Method: GET 2022-09-12T19:03:45.108 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.109 DEBUG:boto:Data: 2022-09-12T19:03:45.110 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.110 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.111 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.112 DEBUG:boto:Params: {} 2022-09-12T19:03:45.113 DEBUG:boto:Token: None 2022-09-12T19:03:45.114 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.115 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1HI6794+MToB62PuKX+wqBIlrok= 2022-09-12T19:03:45.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1HI6794+MToB62PuKX+wqBIlrok=', 'Content-Length': '0'} 2022-09-12T19:03:45.119 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001eefa78413ff9b2f-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.120 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:45.120 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:45.121 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.121 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.122 DEBUG:boto:Method: GET 2022-09-12T19:03:45.123 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.123 DEBUG:boto:Data: 2022-09-12T19:03:45.124 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.125 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.126 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.127 DEBUG:boto:Params: {} 2022-09-12T19:03:45.128 DEBUG:boto:Token: None 2022-09-12T19:03:45.129 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.130 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1HI6794+MToB62PuKX+wqBIlrok= 2022-09-12T19:03:45.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1HI6794+MToB62PuKX+wqBIlrok=', 'Content-Length': '0'} 2022-09-12T19:03:45.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d09cec8a1a6a645c-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.135 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:45.135 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:45.136 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.136 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.137 DEBUG:boto:Method: HEAD 2022-09-12T19:03:45.137 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.138 DEBUG:boto:Data: 2022-09-12T19:03:45.138 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.139 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.139 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.140 DEBUG:boto:Params: {} 2022-09-12T19:03:45.141 DEBUG:boto:Token: None 2022-09-12T19:03:45.141 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.142 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nWaGIZwhX55/RhKhk9JkIboSoFE= 2022-09-12T19:03:45.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nWaGIZwhX55/RhKhk9JkIboSoFE=', 'Content-Length': '0'} 2022-09-12T19:03:45.144 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005ecdacaca6d1d8d9-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.145 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:45.145 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:45.146 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.146 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.147 DEBUG:boto:Method: HEAD 2022-09-12T19:03:45.148 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.148 DEBUG:boto:Data: 2022-09-12T19:03:45.149 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.149 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.150 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.150 DEBUG:boto:Params: {} 2022-09-12T19:03:45.151 DEBUG:boto:Token: None 2022-09-12T19:03:45.151 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.152 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nWaGIZwhX55/RhKhk9JkIboSoFE= 2022-09-12T19:03:45.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nWaGIZwhX55/RhKhk9JkIboSoFE=', 'Content-Length': '0'} 2022-09-12T19:03:45.154 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000059355066118ed195-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:45.156 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:45.156 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:45.157 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.157 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.158 DEBUG:boto:Method: GET 2022-09-12T19:03:45.158 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.159 DEBUG:boto:Data: 2022-09-12T19:03:45.159 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.160 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.161 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.161 DEBUG:boto:Params: {} 2022-09-12T19:03:45.162 DEBUG:boto:Token: None 2022-09-12T19:03:45.162 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.163 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1HI6794+MToB62PuKX+wqBIlrok= 2022-09-12T19:03:45.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1HI6794+MToB62PuKX+wqBIlrok=', 'Content-Length': '0'} 2022-09-12T19:03:45.165 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002e2620edd9e0016e-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.166 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:45.167 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:45.168 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.168 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.169 DEBUG:boto:Method: GET 2022-09-12T19:03:45.169 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.170 DEBUG:boto:Data: 2022-09-12T19:03:45.170 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.171 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.171 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.172 DEBUG:boto:Params: {} 2022-09-12T19:03:45.173 DEBUG:boto:Token: None 2022-09-12T19:03:45.173 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.174 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1HI6794+MToB62PuKX+wqBIlrok= 2022-09-12T19:03:45.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1HI6794+MToB62PuKX+wqBIlrok=', 'Content-Length': '0'} 2022-09-12T19:03:45.177 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000547f0c356ffa453b-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.178 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:45.179 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:45.179 DEBUG:boto:path=/jdpiak-16/?prefix=dummy 2022-09-12T19:03:45.180 DEBUG:boto:auth_path=/jdpiak-16/?prefix=dummy 2022-09-12T19:03:45.180 DEBUG:boto:Method: GET 2022-09-12T19:03:45.181 DEBUG:boto:Path: /jdpiak-16/?prefix=dummy 2022-09-12T19:03:45.181 DEBUG:boto:Data: 2022-09-12T19:03:45.182 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.182 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.183 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.183 DEBUG:boto:Params: {} 2022-09-12T19:03:45.184 DEBUG:boto:Token: None 2022-09-12T19:03:45.184 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/ 2022-09-12T19:03:45.185 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0= 2022-09-12T19:03:45.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0=', 'Content-Length': '0'} 2022-09-12T19:03:45.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009661c4ed000a994d-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.230 DEBUG:boto:b'jdpiak-16dummy1000falsedummy2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:45.231 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:45.232 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:45.232 DEBUG:boto:path=/jdpiak-16/?prefix=dummy 2022-09-12T19:03:45.233 DEBUG:boto:auth_path=/jdpiak-16/?prefix=dummy 2022-09-12T19:03:45.233 DEBUG:boto:Method: GET 2022-09-12T19:03:45.234 DEBUG:boto:Path: /jdpiak-16/?prefix=dummy 2022-09-12T19:03:45.234 DEBUG:boto:Data: 2022-09-12T19:03:45.235 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.235 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.236 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.236 DEBUG:boto:Params: {} 2022-09-12T19:03:45.237 DEBUG:boto:Token: None 2022-09-12T19:03:45.238 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/ 2022-09-12T19:03:45.238 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0= 2022-09-12T19:03:45.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0=', 'Content-Length': '0'} 2022-09-12T19:03:45.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ea65ff2ff338384-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.242 DEBUG:boto:b'jdpiak-16dummy1000falsedummy2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:45.243 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:45.244 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:45.244 DEBUG:boto:Method: HEAD 2022-09-12T19:03:45.245 DEBUG:boto:Path: /jdpiak-16/dummy 2022-09-12T19:03:45.245 DEBUG:boto:Data: 2022-09-12T19:03:45.246 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.246 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.247 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.247 DEBUG:boto:Params: {} 2022-09-12T19:03:45.248 DEBUG:boto:Token: None 2022-09-12T19:03:45.248 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/dummy 2022-09-12T19:03:45.249 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ldezra4qffdw6v+DQCm3/iBY3dY= 2022-09-12T19:03:45.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ldezra4qffdw6v+DQCm3/iBY3dY=', 'Content-Length': '0'} 2022-09-12T19:03:45.252 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ab8683af6a479c5c-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.253 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:45.253 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:45.254 DEBUG:boto:Method: HEAD 2022-09-12T19:03:45.255 DEBUG:boto:Path: /jdpiak-16/dummy 2022-09-12T19:03:45.255 DEBUG:boto:Data: 2022-09-12T19:03:45.256 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.256 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.257 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.257 DEBUG:boto:Params: {} 2022-09-12T19:03:45.258 DEBUG:boto:Token: None 2022-09-12T19:03:45.258 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/dummy 2022-09-12T19:03:45.259 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ldezra4qffdw6v+DQCm3/iBY3dY= 2022-09-12T19:03:45.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ldezra4qffdw6v+DQCm3/iBY3dY=', 'Content-Length': '0'} 2022-09-12T19:03:45.262 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004b4e2a79c455c775-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:45.263 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:45.264 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:45.265 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.265 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.266 DEBUG:boto:Method: GET 2022-09-12T19:03:45.267 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.268 DEBUG:boto:Data: 2022-09-12T19:03:45.269 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.269 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.270 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.270 DEBUG:boto:Params: {} 2022-09-12T19:03:45.271 DEBUG:boto:Token: None 2022-09-12T19:03:45.272 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.272 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1HI6794+MToB62PuKX+wqBIlrok= 2022-09-12T19:03:45.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1HI6794+MToB62PuKX+wqBIlrok=', 'Content-Length': '0'} 2022-09-12T19:03:45.276 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003d64e696612a6d46-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.277 DEBUG:boto:path=/jdpiak-16/dummy 2022-09-12T19:03:45.278 DEBUG:boto:auth_path=/jdpiak-16/dummy 2022-09-12T19:03:45.279 DEBUG:boto:path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.280 DEBUG:boto:auth_path=/jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.281 DEBUG:boto:Method: GET 2022-09-12T19:03:45.282 DEBUG:boto:Path: /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.283 DEBUG:boto:Data: 2022-09-12T19:03:45.283 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.284 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.284 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.285 DEBUG:boto:Params: {} 2022-09-12T19:03:45.285 DEBUG:boto:Token: None 2022-09-12T19:03:45.286 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/dummy?versionId=dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ 2022-09-12T19:03:45.287 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1HI6794+MToB62PuKX+wqBIlrok= 2022-09-12T19:03:45.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1HI6794+MToB62PuKX+wqBIlrok=', 'Content-Length': '0'} 2022-09-12T19:03:45.290 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:57:52 GMT'), ('x-amz-version-id', 'dM1h1LlChQwJOE3oHBYl7Qu-tamZZbJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002b4e619720992a5d-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.291 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:45.291 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:45.292 DEBUG:boto:path=/jdpiak-16/?prefix=dummy 2022-09-12T19:03:45.292 DEBUG:boto:auth_path=/jdpiak-16/?prefix=dummy 2022-09-12T19:03:45.293 DEBUG:boto:Method: GET 2022-09-12T19:03:45.293 DEBUG:boto:Path: /jdpiak-16/?prefix=dummy 2022-09-12T19:03:45.294 DEBUG:boto:Data: 2022-09-12T19:03:45.295 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.295 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.296 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.296 DEBUG:boto:Params: {} 2022-09-12T19:03:45.297 DEBUG:boto:Token: None 2022-09-12T19:03:45.297 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/ 2022-09-12T19:03:45.298 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0= 2022-09-12T19:03:45.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0=', 'Content-Length': '0'} 2022-09-12T19:03:45.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001ea3ea86bc6582c-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.302 DEBUG:boto:b'jdpiak-16dummy1000falsedummy2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:45.303 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:45.303 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:45.304 DEBUG:boto:path=/jdpiak-16/?prefix=dummy 2022-09-12T19:03:45.305 DEBUG:boto:auth_path=/jdpiak-16/?prefix=dummy 2022-09-12T19:03:45.305 DEBUG:boto:Method: GET 2022-09-12T19:03:45.306 DEBUG:boto:Path: /jdpiak-16/?prefix=dummy 2022-09-12T19:03:45.306 DEBUG:boto:Data: 2022-09-12T19:03:45.307 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.307 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.308 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.308 DEBUG:boto:Params: {} 2022-09-12T19:03:45.309 DEBUG:boto:Token: None 2022-09-12T19:03:45.310 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/ 2022-09-12T19:03:45.311 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0= 2022-09-12T19:03:45.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0=', 'Content-Length': '0'} 2022-09-12T19:03:45.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066e03f097d15c703-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.314 DEBUG:boto:b'jdpiak-16dummy1000falsedummy2022-09-12T18:57:52.044Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:45.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:45.316 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:45.316 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:45.317 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.317 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.318 DEBUG:boto:Method: GET 2022-09-12T19:03:45.319 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.319 DEBUG:boto:Data: 2022-09-12T19:03:45.320 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.320 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.321 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.321 DEBUG:boto:Params: {} 2022-09-12T19:03:45.322 DEBUG:boto:Token: None 2022-09-12T19:03:45.323 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.323 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sEnBXXsInv2ohHYKoqojVThECLI= 2022-09-12T19:03:45.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sEnBXXsInv2ohHYKoqojVThECLI=', 'Content-Length': '0'} 2022-09-12T19:03:45.326 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000789c2f4ac95ad0a3-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.327 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:45.327 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:45.328 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.328 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.329 DEBUG:boto:Method: GET 2022-09-12T19:03:45.329 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.330 DEBUG:boto:Data: 2022-09-12T19:03:45.331 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.331 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.332 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.332 DEBUG:boto:Params: {} 2022-09-12T19:03:45.333 DEBUG:boto:Token: None 2022-09-12T19:03:45.333 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.334 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sEnBXXsInv2ohHYKoqojVThECLI= 2022-09-12T19:03:45.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sEnBXXsInv2ohHYKoqojVThECLI=', 'Content-Length': '0'} 2022-09-12T19:03:45.337 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c82cf4ab5e35983e-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.378 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:45.379 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:45.379 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.380 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.381 DEBUG:boto:Method: HEAD 2022-09-12T19:03:45.381 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.382 DEBUG:boto:Data: 2022-09-12T19:03:45.382 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.383 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.383 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.383 DEBUG:boto:Params: {} 2022-09-12T19:03:45.384 DEBUG:boto:Token: None 2022-09-12T19:03:45.385 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.385 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WgezkdGBlZM/o3VomyZUxWsEBV4= 2022-09-12T19:03:45.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WgezkdGBlZM/o3VomyZUxWsEBV4=', 'Content-Length': '0'} 2022-09-12T19:03:45.388 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a9566dc22d4fb58-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.389 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:45.389 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:45.390 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.390 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.391 DEBUG:boto:Method: HEAD 2022-09-12T19:03:45.391 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.392 DEBUG:boto:Data: 2022-09-12T19:03:45.392 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.393 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.393 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.394 DEBUG:boto:Params: {} 2022-09-12T19:03:45.394 DEBUG:boto:Token: None 2022-09-12T19:03:45.395 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.395 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WgezkdGBlZM/o3VomyZUxWsEBV4= 2022-09-12T19:03:45.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WgezkdGBlZM/o3VomyZUxWsEBV4=', 'Content-Length': '0'} 2022-09-12T19:03:45.398 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000086d211147778c34-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:45.399 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:45.400 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:45.400 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.401 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.401 DEBUG:boto:Method: GET 2022-09-12T19:03:45.402 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.402 DEBUG:boto:Data: 2022-09-12T19:03:45.403 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.403 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.404 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.405 DEBUG:boto:Params: {} 2022-09-12T19:03:45.405 DEBUG:boto:Token: None 2022-09-12T19:03:45.406 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.406 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sEnBXXsInv2ohHYKoqojVThECLI= 2022-09-12T19:03:45.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sEnBXXsInv2ohHYKoqojVThECLI=', 'Content-Length': '0'} 2022-09-12T19:03:45.409 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087d2cc432177ab3a-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.410 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:45.410 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:45.411 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.411 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.412 DEBUG:boto:Method: GET 2022-09-12T19:03:45.412 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.413 DEBUG:boto:Data: 2022-09-12T19:03:45.413 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.414 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.414 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.415 DEBUG:boto:Params: {} 2022-09-12T19:03:45.416 DEBUG:boto:Token: None 2022-09-12T19:03:45.416 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.417 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sEnBXXsInv2ohHYKoqojVThECLI= 2022-09-12T19:03:45.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sEnBXXsInv2ohHYKoqojVThECLI=', 'Content-Length': '0'} 2022-09-12T19:03:45.419 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000336434b821f21d32-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.420 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:45.421 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:45.421 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:45.422 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:45.422 DEBUG:boto:Method: GET 2022-09-12T19:03:45.423 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:45.423 DEBUG:boto:Data: 2022-09-12T19:03:45.424 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.424 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.425 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.425 DEBUG:boto:Params: {} 2022-09-12T19:03:45.426 DEBUG:boto:Token: None 2022-09-12T19:03:45.427 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/ 2022-09-12T19:03:45.427 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0= 2022-09-12T19:03:45.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0=', 'Content-Length': '0'} 2022-09-12T19:03:45.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e414e8a4211ae77e-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.431 DEBUG:boto:b'jdpiak-16obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:45.432 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:45.432 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:45.433 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:45.433 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:45.434 DEBUG:boto:Method: GET 2022-09-12T19:03:45.434 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:45.435 DEBUG:boto:Data: 2022-09-12T19:03:45.435 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.436 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.436 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.437 DEBUG:boto:Params: {} 2022-09-12T19:03:45.438 DEBUG:boto:Token: None 2022-09-12T19:03:45.438 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/ 2022-09-12T19:03:45.439 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0= 2022-09-12T19:03:45.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0=', 'Content-Length': '0'} 2022-09-12T19:03:45.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e08fae6f6fcef18f-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.443 DEBUG:boto:b'jdpiak-16obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:45.444 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:45.444 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:45.445 DEBUG:boto:Method: HEAD 2022-09-12T19:03:45.445 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1 2022-09-12T19:03:45.446 DEBUG:boto:Data: 2022-09-12T19:03:45.446 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.447 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.447 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.448 DEBUG:boto:Params: {} 2022-09-12T19:03:45.448 DEBUG:boto:Token: None 2022-09-12T19:03:45.449 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone1 2022-09-12T19:03:45.449 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ysoq+20vWCDkvwwf/mDT5Gq+3S8= 2022-09-12T19:03:45.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ysoq+20vWCDkvwwf/mDT5Gq+3S8=', 'Content-Length': '0'} 2022-09-12T19:03:45.452 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d53352cae44dcbe-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.453 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:45.453 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:45.454 DEBUG:boto:Method: HEAD 2022-09-12T19:03:45.454 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1 2022-09-12T19:03:45.455 DEBUG:boto:Data: 2022-09-12T19:03:45.455 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.456 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.456 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.457 DEBUG:boto:Params: {} 2022-09-12T19:03:45.457 DEBUG:boto:Token: None 2022-09-12T19:03:45.458 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone1 2022-09-12T19:03:45.458 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ysoq+20vWCDkvwwf/mDT5Gq+3S8= 2022-09-12T19:03:45.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ysoq+20vWCDkvwwf/mDT5Gq+3S8=', 'Content-Length': '0'} 2022-09-12T19:03:45.462 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b701df06dbadd5ee-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:45.463 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:45.464 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:45.464 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.465 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.465 DEBUG:boto:Method: GET 2022-09-12T19:03:45.466 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.466 DEBUG:boto:Data: 2022-09-12T19:03:45.467 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.467 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.468 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.469 DEBUG:boto:Params: {} 2022-09-12T19:03:45.469 DEBUG:boto:Token: None 2022-09-12T19:03:45.470 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.470 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sEnBXXsInv2ohHYKoqojVThECLI= 2022-09-12T19:03:45.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sEnBXXsInv2ohHYKoqojVThECLI=', 'Content-Length': '0'} 2022-09-12T19:03:45.473 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000365d72dc88cac44c-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.474 DEBUG:boto:path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:45.475 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1 2022-09-12T19:03:45.475 DEBUG:boto:path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.476 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.476 DEBUG:boto:Method: GET 2022-09-12T19:03:45.477 DEBUG:boto:Path: /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.477 DEBUG:boto:Data: 2022-09-12T19:03:45.478 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.478 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.479 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.479 DEBUG:boto:Params: {} 2022-09-12T19:03:45.480 DEBUG:boto:Token: None 2022-09-12T19:03:45.480 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone1?versionId=-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR 2022-09-12T19:03:45.481 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sEnBXXsInv2ohHYKoqojVThECLI= 2022-09-12T19:03:45.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sEnBXXsInv2ohHYKoqojVThECLI=', 'Content-Length': '0'} 2022-09-12T19:03:45.484 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '-zshRkRrxKU0EJVPpVBUsfYl8JR7HFR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfe84e26d9275e4f-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.526 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:45.527 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:45.528 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:45.529 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:45.530 DEBUG:boto:Method: GET 2022-09-12T19:03:45.531 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:45.532 DEBUG:boto:Data: 2022-09-12T19:03:45.533 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.534 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.535 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.536 DEBUG:boto:Params: {} 2022-09-12T19:03:45.537 DEBUG:boto:Token: None 2022-09-12T19:03:45.538 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/ 2022-09-12T19:03:45.539 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0= 2022-09-12T19:03:45.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0=', 'Content-Length': '0'} 2022-09-12T19:03:45.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c9e241a92325af0-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.545 DEBUG:boto:b'jdpiak-16obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:45.546 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:45.546 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:45.547 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:45.547 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:45.548 DEBUG:boto:Method: GET 2022-09-12T19:03:45.548 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone1 2022-09-12T19:03:45.549 DEBUG:boto:Data: 2022-09-12T19:03:45.549 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.550 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.550 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.551 DEBUG:boto:Params: {} 2022-09-12T19:03:45.551 DEBUG:boto:Token: None 2022-09-12T19:03:45.552 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/ 2022-09-12T19:03:45.553 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0= 2022-09-12T19:03:45.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0=', 'Content-Length': '0'} 2022-09-12T19:03:45.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ba03d337ba3695f-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.557 DEBUG:boto:b'jdpiak-16obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.029Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:45.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:45.558 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:45.559 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:45.560 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.560 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.561 DEBUG:boto:Method: GET 2022-09-12T19:03:45.561 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.562 DEBUG:boto:Data: 2022-09-12T19:03:45.562 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.563 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.563 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.564 DEBUG:boto:Params: {} 2022-09-12T19:03:45.564 DEBUG:boto:Token: None 2022-09-12T19:03:45.565 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.566 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GcmAJx8jaTykamwVEK94SvEXYGg= 2022-09-12T19:03:45.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GcmAJx8jaTykamwVEK94SvEXYGg=', 'Content-Length': '0'} 2022-09-12T19:03:45.569 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1225fbbaaf057c3-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.614 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:45.615 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:45.615 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.616 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.616 DEBUG:boto:Method: GET 2022-09-12T19:03:45.617 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.617 DEBUG:boto:Data: 2022-09-12T19:03:45.618 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.619 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.619 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.620 DEBUG:boto:Params: {} 2022-09-12T19:03:45.620 DEBUG:boto:Token: None 2022-09-12T19:03:45.621 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.621 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GcmAJx8jaTykamwVEK94SvEXYGg= 2022-09-12T19:03:45.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GcmAJx8jaTykamwVEK94SvEXYGg=', 'Content-Length': '0'} 2022-09-12T19:03:45.625 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000447ebdefa7d5c295-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.626 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:45.626 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:45.627 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.627 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.628 DEBUG:boto:Method: HEAD 2022-09-12T19:03:45.628 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.629 DEBUG:boto:Data: 2022-09-12T19:03:45.629 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.630 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.631 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.631 DEBUG:boto:Params: {} 2022-09-12T19:03:45.632 DEBUG:boto:Token: None 2022-09-12T19:03:45.633 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.634 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bNVdUVLWZggwH2dxWh+eV3B/I9Q= 2022-09-12T19:03:45.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bNVdUVLWZggwH2dxWh+eV3B/I9Q=', 'Content-Length': '0'} 2022-09-12T19:03:45.636 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000067e330f6c9bbfd8e-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.637 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:45.638 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:45.638 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.639 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.639 DEBUG:boto:Method: HEAD 2022-09-12T19:03:45.640 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.640 DEBUG:boto:Data: 2022-09-12T19:03:45.641 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.641 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.642 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.643 DEBUG:boto:Params: {} 2022-09-12T19:03:45.644 DEBUG:boto:Token: None 2022-09-12T19:03:45.645 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.646 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bNVdUVLWZggwH2dxWh+eV3B/I9Q= 2022-09-12T19:03:45.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bNVdUVLWZggwH2dxWh+eV3B/I9Q=', 'Content-Length': '0'} 2022-09-12T19:03:45.649 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001867364e5ea038df-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:45.650 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:45.651 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:45.651 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.652 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.652 DEBUG:boto:Method: GET 2022-09-12T19:03:45.653 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.653 DEBUG:boto:Data: 2022-09-12T19:03:45.654 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.654 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.655 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.655 DEBUG:boto:Params: {} 2022-09-12T19:03:45.656 DEBUG:boto:Token: None 2022-09-12T19:03:45.657 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.657 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GcmAJx8jaTykamwVEK94SvEXYGg= 2022-09-12T19:03:45.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GcmAJx8jaTykamwVEK94SvEXYGg=', 'Content-Length': '0'} 2022-09-12T19:03:45.660 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3cf04eec961343f-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.661 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:45.662 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:45.663 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.663 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.664 DEBUG:boto:Method: GET 2022-09-12T19:03:45.664 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.665 DEBUG:boto:Data: 2022-09-12T19:03:45.665 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.666 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.666 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.667 DEBUG:boto:Params: {} 2022-09-12T19:03:45.668 DEBUG:boto:Token: None 2022-09-12T19:03:45.668 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.669 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GcmAJx8jaTykamwVEK94SvEXYGg= 2022-09-12T19:03:45.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GcmAJx8jaTykamwVEK94SvEXYGg=', 'Content-Length': '0'} 2022-09-12T19:03:45.672 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000013b3495f705f69f6-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.673 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:45.673 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:45.674 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:45.675 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:45.675 DEBUG:boto:Method: GET 2022-09-12T19:03:45.676 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:45.676 DEBUG:boto:Data: 2022-09-12T19:03:45.677 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.677 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.678 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.678 DEBUG:boto:Params: {} 2022-09-12T19:03:45.679 DEBUG:boto:Token: None 2022-09-12T19:03:45.680 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/ 2022-09-12T19:03:45.680 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0= 2022-09-12T19:03:45.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0=', 'Content-Length': '0'} 2022-09-12T19:03:45.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af21d4586f736be9-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.684 DEBUG:boto:b'jdpiak-16obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:45.685 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:45.686 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:45.686 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:45.687 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:45.687 DEBUG:boto:Method: GET 2022-09-12T19:03:45.688 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:45.688 DEBUG:boto:Data: 2022-09-12T19:03:45.689 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.689 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.690 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.691 DEBUG:boto:Params: {} 2022-09-12T19:03:45.691 DEBUG:boto:Token: None 2022-09-12T19:03:45.692 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/ 2022-09-12T19:03:45.693 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0= 2022-09-12T19:03:45.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0=', 'Content-Length': '0'} 2022-09-12T19:03:45.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be7867b9abe4fa54-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.697 DEBUG:boto:b'jdpiak-16obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:45.697 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:45.698 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:45.699 DEBUG:boto:Method: HEAD 2022-09-12T19:03:45.699 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2 2022-09-12T19:03:45.700 DEBUG:boto:Data: 2022-09-12T19:03:45.700 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.701 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.701 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.702 DEBUG:boto:Params: {} 2022-09-12T19:03:45.702 DEBUG:boto:Token: None 2022-09-12T19:03:45.703 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone2 2022-09-12T19:03:45.703 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XUZSg6NDqLmjBTPVSML6XsagW0s= 2022-09-12T19:03:45.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XUZSg6NDqLmjBTPVSML6XsagW0s=', 'Content-Length': '0'} 2022-09-12T19:03:45.706 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec09c7a03bfe587f-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.707 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:45.708 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:45.708 DEBUG:boto:Method: HEAD 2022-09-12T19:03:45.709 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2 2022-09-12T19:03:45.709 DEBUG:boto:Data: 2022-09-12T19:03:45.710 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.710 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.711 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.711 DEBUG:boto:Params: {} 2022-09-12T19:03:45.712 DEBUG:boto:Token: None 2022-09-12T19:03:45.712 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone2 2022-09-12T19:03:45.713 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XUZSg6NDqLmjBTPVSML6XsagW0s= 2022-09-12T19:03:45.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XUZSg6NDqLmjBTPVSML6XsagW0s=', 'Content-Length': '0'} 2022-09-12T19:03:45.716 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2af18d32054e0a5-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:45.717 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:45.718 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:45.718 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.719 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.720 DEBUG:boto:Method: GET 2022-09-12T19:03:45.721 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.722 DEBUG:boto:Data: 2022-09-12T19:03:45.723 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.724 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.725 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.726 DEBUG:boto:Params: {} 2022-09-12T19:03:45.727 DEBUG:boto:Token: None 2022-09-12T19:03:45.728 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.730 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GcmAJx8jaTykamwVEK94SvEXYGg= 2022-09-12T19:03:45.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GcmAJx8jaTykamwVEK94SvEXYGg=', 'Content-Length': '0'} 2022-09-12T19:03:45.734 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097fd59f71447cdd7-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.774 DEBUG:boto:path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:45.775 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2 2022-09-12T19:03:45.775 DEBUG:boto:path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.776 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.776 DEBUG:boto:Method: GET 2022-09-12T19:03:45.777 DEBUG:boto:Path: /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.777 DEBUG:boto:Data: 2022-09-12T19:03:45.778 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.779 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.779 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.780 DEBUG:boto:Params: {} 2022-09-12T19:03:45.781 DEBUG:boto:Token: None 2022-09-12T19:03:45.782 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone2?versionId=3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5 2022-09-12T19:03:45.784 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GcmAJx8jaTykamwVEK94SvEXYGg= 2022-09-12T19:03:45.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GcmAJx8jaTykamwVEK94SvEXYGg=', 'Content-Length': '0'} 2022-09-12T19:03:45.787 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '3vAUDk61iJIo2fBYQQ4xbiFpg6qSMh5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057df1d80129867ba-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.789 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:45.789 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:45.790 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:45.791 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:45.791 DEBUG:boto:Method: GET 2022-09-12T19:03:45.792 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:45.792 DEBUG:boto:Data: 2022-09-12T19:03:45.793 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.793 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.794 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.794 DEBUG:boto:Params: {} 2022-09-12T19:03:45.795 DEBUG:boto:Token: None 2022-09-12T19:03:45.796 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/ 2022-09-12T19:03:45.796 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0= 2022-09-12T19:03:45.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0=', 'Content-Length': '0'} 2022-09-12T19:03:45.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001762bb0b48414d76-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.801 DEBUG:boto:b'jdpiak-16obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:45.802 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:45.803 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:45.804 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:45.805 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:45.805 DEBUG:boto:Method: GET 2022-09-12T19:03:45.806 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone2 2022-09-12T19:03:45.806 DEBUG:boto:Data: 2022-09-12T19:03:45.807 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.807 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.808 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.808 DEBUG:boto:Params: {} 2022-09-12T19:03:45.809 DEBUG:boto:Token: None 2022-09-12T19:03:45.810 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/ 2022-09-12T19:03:45.810 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0= 2022-09-12T19:03:45.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0=', 'Content-Length': '0'} 2022-09-12T19:03:45.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e753daa347225b88-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.814 DEBUG:boto:b'jdpiak-16obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.189Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:45.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:45.816 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:45.816 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:45.817 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.817 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.818 DEBUG:boto:Method: GET 2022-09-12T19:03:45.819 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.819 DEBUG:boto:Data: 2022-09-12T19:03:45.820 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.820 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.821 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.821 DEBUG:boto:Params: {} 2022-09-12T19:03:45.822 DEBUG:boto:Token: None 2022-09-12T19:03:45.822 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.823 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h8aLUPJcq6nPcGbtdMAiBpWZwxI= 2022-09-12T19:03:45.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h8aLUPJcq6nPcGbtdMAiBpWZwxI=', 'Content-Length': '0'} 2022-09-12T19:03:45.826 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006561b49b4dab3609-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.827 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:45.828 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:45.829 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.830 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.831 DEBUG:boto:Method: GET 2022-09-12T19:03:45.832 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.833 DEBUG:boto:Data: 2022-09-12T19:03:45.834 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.835 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.835 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.836 DEBUG:boto:Params: {} 2022-09-12T19:03:45.837 DEBUG:boto:Token: None 2022-09-12T19:03:45.837 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.838 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h8aLUPJcq6nPcGbtdMAiBpWZwxI= 2022-09-12T19:03:45.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h8aLUPJcq6nPcGbtdMAiBpWZwxI=', 'Content-Length': '0'} 2022-09-12T19:03:45.841 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000847e3e16664587cd-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.842 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:45.842 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:45.843 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.843 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.844 DEBUG:boto:Method: HEAD 2022-09-12T19:03:45.844 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.845 DEBUG:boto:Data: 2022-09-12T19:03:45.845 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.846 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.846 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.847 DEBUG:boto:Params: {} 2022-09-12T19:03:45.848 DEBUG:boto:Token: None 2022-09-12T19:03:45.848 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.849 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:794K5wVFlLsH4BdhQenyLMt0qvs= 2022-09-12T19:03:45.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:794K5wVFlLsH4BdhQenyLMt0qvs=', 'Content-Length': '0'} 2022-09-12T19:03:45.852 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f58dcbfa8a61f3eb-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.853 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:45.854 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:45.854 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.855 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.856 DEBUG:boto:Method: HEAD 2022-09-12T19:03:45.856 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.857 DEBUG:boto:Data: 2022-09-12T19:03:45.857 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.858 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.859 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.860 DEBUG:boto:Params: {} 2022-09-12T19:03:45.860 DEBUG:boto:Token: None 2022-09-12T19:03:45.861 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.861 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:794K5wVFlLsH4BdhQenyLMt0qvs= 2022-09-12T19:03:45.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:794K5wVFlLsH4BdhQenyLMt0qvs=', 'Content-Length': '0'} 2022-09-12T19:03:45.864 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035b3708858751b56-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:45.865 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:45.866 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:45.866 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.867 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.868 DEBUG:boto:Method: GET 2022-09-12T19:03:45.868 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.869 DEBUG:boto:Data: 2022-09-12T19:03:45.869 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.870 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.870 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.871 DEBUG:boto:Params: {} 2022-09-12T19:03:45.872 DEBUG:boto:Token: None 2022-09-12T19:03:45.872 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.873 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h8aLUPJcq6nPcGbtdMAiBpWZwxI= 2022-09-12T19:03:45.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h8aLUPJcq6nPcGbtdMAiBpWZwxI=', 'Content-Length': '0'} 2022-09-12T19:03:45.876 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f903b720e1e3fa1-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.876 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:45.877 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:45.878 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.878 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.879 DEBUG:boto:Method: GET 2022-09-12T19:03:45.879 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.880 DEBUG:boto:Data: 2022-09-12T19:03:45.880 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.881 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.882 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.882 DEBUG:boto:Params: {} 2022-09-12T19:03:45.883 DEBUG:boto:Token: None 2022-09-12T19:03:45.883 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.884 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h8aLUPJcq6nPcGbtdMAiBpWZwxI= 2022-09-12T19:03:45.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h8aLUPJcq6nPcGbtdMAiBpWZwxI=', 'Content-Length': '0'} 2022-09-12T19:03:45.887 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c16e151c3ea2c4fd-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.888 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:45.888 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:45.889 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:45.889 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:45.890 DEBUG:boto:Method: GET 2022-09-12T19:03:45.890 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:45.891 DEBUG:boto:Data: 2022-09-12T19:03:45.891 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.892 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.892 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.893 DEBUG:boto:Params: {} 2022-09-12T19:03:45.894 DEBUG:boto:Token: None 2022-09-12T19:03:45.894 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/ 2022-09-12T19:03:45.895 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0= 2022-09-12T19:03:45.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0=', 'Content-Length': '0'} 2022-09-12T19:03:45.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059757bf27a644df4-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.899 DEBUG:boto:b'jdpiak-16obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:45.900 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:45.900 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:45.901 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:45.901 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:45.902 DEBUG:boto:Method: GET 2022-09-12T19:03:45.903 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:45.903 DEBUG:boto:Data: 2022-09-12T19:03:45.904 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.904 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.905 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.905 DEBUG:boto:Params: {} 2022-09-12T19:03:45.906 DEBUG:boto:Token: None 2022-09-12T19:03:45.906 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/ 2022-09-12T19:03:45.907 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0= 2022-09-12T19:03:45.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A1cqJzuEYYAyYaRGcoQjEY/0bW0=', 'Content-Length': '0'} 2022-09-12T19:03:45.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000451338ba4d96e136-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.911 DEBUG:boto:b'jdpiak-16obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:45.912 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:45.912 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:45.913 DEBUG:boto:Method: HEAD 2022-09-12T19:03:45.914 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3 2022-09-12T19:03:45.914 DEBUG:boto:Data: 2022-09-12T19:03:45.915 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.915 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.916 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.916 DEBUG:boto:Params: {} 2022-09-12T19:03:45.917 DEBUG:boto:Token: None 2022-09-12T19:03:45.918 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone3 2022-09-12T19:03:45.918 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DLI5LJl2zDV5ccoHgAqjCD/Yup8= 2022-09-12T19:03:45.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DLI5LJl2zDV5ccoHgAqjCD/Yup8=', 'Content-Length': '0'} 2022-09-12T19:03:45.922 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b58940cfb5f3985-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.923 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:45.923 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:45.924 DEBUG:boto:Method: HEAD 2022-09-12T19:03:45.924 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3 2022-09-12T19:03:45.925 DEBUG:boto:Data: 2022-09-12T19:03:45.925 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.926 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.927 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.927 DEBUG:boto:Params: {} 2022-09-12T19:03:45.928 DEBUG:boto:Token: None 2022-09-12T19:03:45.928 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone3 2022-09-12T19:03:45.929 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DLI5LJl2zDV5ccoHgAqjCD/Yup8= 2022-09-12T19:03:45.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DLI5LJl2zDV5ccoHgAqjCD/Yup8=', 'Content-Length': '0'} 2022-09-12T19:03:45.932 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e1d5a22399e2333-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:45.933 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:45.934 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:45.935 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.935 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.936 DEBUG:boto:Method: GET 2022-09-12T19:03:45.936 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.937 DEBUG:boto:Data: 2022-09-12T19:03:45.937 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.938 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:45.939 DEBUG:boto:Port: 8001 2022-09-12T19:03:45.939 DEBUG:boto:Params: {} 2022-09-12T19:03:45.940 DEBUG:boto:Token: None 2022-09-12T19:03:45.941 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.941 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h8aLUPJcq6nPcGbtdMAiBpWZwxI= 2022-09-12T19:03:45.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h8aLUPJcq6nPcGbtdMAiBpWZwxI=', 'Content-Length': '0'} 2022-09-12T19:03:45.944 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c324eb2f4d08613d-00631f8291-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:45.986 DEBUG:boto:path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:45.987 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3 2022-09-12T19:03:45.988 DEBUG:boto:path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.989 DEBUG:boto:auth_path=/jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.989 DEBUG:boto:Method: GET 2022-09-12T19:03:45.990 DEBUG:boto:Path: /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.991 DEBUG:boto:Data: 2022-09-12T19:03:45.992 DEBUG:boto:Headers: {} 2022-09-12T19:03:45.992 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:45.993 DEBUG:boto:Port: 8000 2022-09-12T19:03:45.994 DEBUG:boto:Params: {} 2022-09-12T19:03:45.995 DEBUG:boto:Token: None 2022-09-12T19:03:45.996 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:45 GMT /jdpiak-16/obj-test-zone3?versionId=RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE 2022-09-12T19:03:45.997 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h8aLUPJcq6nPcGbtdMAiBpWZwxI= 2022-09-12T19:03:45.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h8aLUPJcq6nPcGbtdMAiBpWZwxI=', 'Content-Length': '0'} 2022-09-12T19:03:46.000 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'RiTU4X-B7B6HWzymkpb1Y7UEtQAz0eE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000564115e93d083111-00631f8291-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:46.001 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:46.001 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:46.002 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:46.003 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:46.003 DEBUG:boto:Method: GET 2022-09-12T19:03:46.004 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:46.005 DEBUG:boto:Data: 2022-09-12T19:03:46.005 DEBUG:boto:Headers: {} 2022-09-12T19:03:46.006 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:46.006 DEBUG:boto:Port: 8001 2022-09-12T19:03:46.007 DEBUG:boto:Params: {} 2022-09-12T19:03:46.008 DEBUG:boto:Token: None 2022-09-12T19:03:46.009 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:46 GMT /jdpiak-16/ 2022-09-12T19:03:46.010 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WqFQWPbRiBP/hp0OVwWQOh7O7ng= 2022-09-12T19:03:46.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WqFQWPbRiBP/hp0OVwWQOh7O7ng=', 'Content-Length': '0'} 2022-09-12T19:03:46.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8884cfc36ce9306-00631f8292-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:46.016 DEBUG:boto:b'jdpiak-16obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:46.017 DEBUG:boto:path=/jdpiak-16/ 2022-09-12T19:03:46.018 DEBUG:boto:auth_path=/jdpiak-16/ 2022-09-12T19:03:46.019 DEBUG:boto:path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:46.020 DEBUG:boto:auth_path=/jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:46.020 DEBUG:boto:Method: GET 2022-09-12T19:03:46.021 DEBUG:boto:Path: /jdpiak-16/?prefix=obj-test-zone3 2022-09-12T19:03:46.022 DEBUG:boto:Data: 2022-09-12T19:03:46.023 DEBUG:boto:Headers: {} 2022-09-12T19:03:46.024 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:46.024 DEBUG:boto:Port: 8000 2022-09-12T19:03:46.025 DEBUG:boto:Params: {} 2022-09-12T19:03:46.026 DEBUG:boto:Token: None 2022-09-12T19:03:46.027 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:46 GMT /jdpiak-16/ 2022-09-12T19:03:46.027 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WqFQWPbRiBP/hp0OVwWQOh7O7ng= 2022-09-12T19:03:46.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WqFQWPbRiBP/hp0OVwWQOh7O7ng=', 'Content-Length': '0'} 2022-09-12T19:03:46.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007880c937bc339609-00631f8292-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:46.032 DEBUG:boto:b'jdpiak-16obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:46.033 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-16 zones={test-zone3, test-zone2} 2022-09-12T19:03:46.033 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-17 2022-09-12T19:03:46.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:46.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:46.035 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:03:46.100 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:03:46.100 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:03:46.151 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:03:46.149+0000 7f46152b8600 1 bucket sync caught up with source: 2022-09-12T19:03:46.151 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000011.34.5, , 00000000004.13.5, , 00000000008.23.5, , , , , ] 2022-09-12T19:03:46.152 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000011.34.5, , 00000000004.13.5, , 00000000008.23.5, , , , , ] 2022-09-12T19:03:46.152 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:03:46.149+0000 7f46152b8600 0 bucket checkpoint complete 2022-09-12T19:03:46.167 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-17 2022-09-12T19:03:46.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:46.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:46.169 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:03:46.235 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:03:46.236 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:03:46.287 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:46.284+0000 7fd0b37f1600 1 bucket sync caught up with source: 2022-09-12T19:03:46.288 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000011.34.5, , 00000000004.13.5, , 00000000008.23.5, , , , , ] 2022-09-12T19:03:46.288 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000011.34.5, , 00000000004.13.5, , 00000000008.23.5, , , , , ] 2022-09-12T19:03:46.288 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:46.284+0000 7fd0b37f1600 0 bucket checkpoint complete 2022-09-12T19:03:46.310 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-17 2022-09-12T19:03:46.311 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:46.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:46.313 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:03:46.371 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:03:46.372 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:03:46.417 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:03:46.415+0000 7fdabb7c1600 1 bucket sync caught up with source: 2022-09-12T19:03:46.418 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000011.34.5, , 00000000004.13.5, , 00000000008.23.5, , , , , ] 2022-09-12T19:03:46.419 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000011.34.5, , 00000000004.13.5, , 00000000008.23.5, , , , , ] 2022-09-12T19:03:46.419 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:03:46.415+0000 7fdabb7c1600 0 bucket checkpoint complete 2022-09-12T19:03:46.434 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-17 2022-09-12T19:03:46.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:46.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:46.436 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:03:46.496 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:03:46.496 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:03:46.533 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:46.530+0000 7f0bd40c1600 1 bucket sync caught up with source: 2022-09-12T19:03:46.533 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000004.31.5, , 00000000011.42.5, , 00000000010.37.5, , , , , ] 2022-09-12T19:03:46.534 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000004.31.5, , 00000000011.42.5, , 00000000010.37.5, , , , , ] 2022-09-12T19:03:46.534 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:46.530+0000 7f0bd40c1600 0 bucket checkpoint complete 2022-09-12T19:03:46.545 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-17 2022-09-12T19:03:46.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:46.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:46.548 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:03:46.606 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:03:46.607 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:03:46.641 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:46.637+0000 7efc4002f600 1 bucket sync caught up with source: 2022-09-12T19:03:46.641 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000004.31.5, , 00000000011.42.5, , 00000000010.37.5, , , , , ] 2022-09-12T19:03:46.641 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000004.31.5, , 00000000011.42.5, , 00000000010.37.5, , , , , ] 2022-09-12T19:03:46.642 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:46.638+0000 7efc4002f600 0 bucket checkpoint complete 2022-09-12T19:03:46.655 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-17 2022-09-12T19:03:46.656 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-17 2022-09-12T19:03:46.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:46.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:46.661 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:03:46.720 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:03:46.720 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:03:46.764 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:46.761+0000 7fd61d516600 1 bucket sync caught up with source: 2022-09-12T19:03:46.765 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000004.27.5, , 00000000004.30.5, , 00000000015.44.5, , , , , ] 2022-09-12T19:03:46.765 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000004.27.5, , 00000000004.30.5, , 00000000015.44.5, , , , , ] 2022-09-12T19:03:46.765 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:46.761+0000 7fd61d516600 0 bucket checkpoint complete 2022-09-12T19:03:46.780 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-17 2022-09-12T19:03:46.781 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:46.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:46.783 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:03:46.840 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:03:46.840 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:03:46.890 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:03:46.887+0000 7f5f5ec76600 1 bucket sync caught up with source: 2022-09-12T19:03:46.890 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000004.27.5, , 00000000004.30.5, , 00000000015.44.5, , , , , ] 2022-09-12T19:03:46.890 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000004.27.5, , 00000000004.30.5, , 00000000015.44.5, , , , , ] 2022-09-12T19:03:46.891 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:03:46.887+0000 7f5f5ec76600 0 bucket checkpoint complete 2022-09-12T19:03:46.904 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-17 2022-09-12T19:03:46.905 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-17 zones={test-zone2, test-zone1} 2022-09-12T19:03:46.905 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:46.906 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:46.907 DEBUG:boto:Method: HEAD 2022-09-12T19:03:46.907 DEBUG:boto:Path: /jdpiak-17/ 2022-09-12T19:03:46.908 DEBUG:boto:Data: 2022-09-12T19:03:46.909 DEBUG:boto:Headers: {} 2022-09-12T19:03:46.909 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:46.910 DEBUG:boto:Port: 8000 2022-09-12T19:03:46.910 DEBUG:boto:Params: {} 2022-09-12T19:03:46.911 DEBUG:boto:Token: None 2022-09-12T19:03:46.912 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:46 GMT /jdpiak-17/ 2022-09-12T19:03:46.913 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oR8BWKMSaHb2FiNH/aPdrIJ1X4o= 2022-09-12T19:03:46.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oR8BWKMSaHb2FiNH/aPdrIJ1X4o=', 'Content-Length': '0'} 2022-09-12T19:03:46.917 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000befc960b75ce12ea-00631f8292-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:03:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:46.919 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:46.920 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:46.921 DEBUG:boto:Method: HEAD 2022-09-12T19:03:46.922 DEBUG:boto:Path: /jdpiak-17/ 2022-09-12T19:03:46.923 DEBUG:boto:Data: 2022-09-12T19:03:46.924 DEBUG:boto:Headers: {} 2022-09-12T19:03:46.925 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:46.926 DEBUG:boto:Port: 8000 2022-09-12T19:03:46.927 DEBUG:boto:Params: {} 2022-09-12T19:03:46.928 DEBUG:boto:Token: None 2022-09-12T19:03:46.929 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:46 GMT /jdpiak-17/ 2022-09-12T19:03:46.931 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oR8BWKMSaHb2FiNH/aPdrIJ1X4o= 2022-09-12T19:03:46.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oR8BWKMSaHb2FiNH/aPdrIJ1X4o=', 'Content-Length': '0'} 2022-09-12T19:03:46.935 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7431ced31365c43-00631f8292-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:03:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:46.936 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:03:46.937 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:46.938 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:46.939 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:46.940 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:46.942 DEBUG:boto:Method: GET 2022-09-12T19:03:46.943 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:46.944 DEBUG:boto:Data: 2022-09-12T19:03:46.945 DEBUG:boto:Headers: {} 2022-09-12T19:03:46.946 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:46.947 DEBUG:boto:Port: 8000 2022-09-12T19:03:46.948 DEBUG:boto:Params: {} 2022-09-12T19:03:46.949 DEBUG:boto:Token: None 2022-09-12T19:03:46.950 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:46 GMT /jdpiak-17/?versions 2022-09-12T19:03:46.952 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KopWxUKJDZUsWW1PR5PudFyZEDg= 2022-09-12T19:03:46.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KopWxUKJDZUsWW1PR5PudFyZEDg=', 'Content-Length': '0'} 2022-09-12T19:03:46.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057850cf9621accfd-00631f8292-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:46.956 DEBUG:boto:b'jdpiak-17999falsedummybSzfRw-Zsh7lYv-ClgdhoG098KzYg.Strue2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpktrue2022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone20f1rsOoGVVMRuyy11YfWbutRvo943lCtrue2022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3V6Ue4vhVY0WQ9FArGixLjY-hmND5K20true2022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:46.957 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:03:46.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:03:46.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:03:46.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:03:46.959 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:03:46.959 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:46.960 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:46.961 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:46.961 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:46.962 DEBUG:boto:Method: GET 2022-09-12T19:03:46.963 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:46.964 DEBUG:boto:Data: 2022-09-12T19:03:46.965 DEBUG:boto:Headers: {} 2022-09-12T19:03:46.965 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:46.966 DEBUG:boto:Port: 8000 2022-09-12T19:03:46.967 DEBUG:boto:Params: {} 2022-09-12T19:03:46.968 DEBUG:boto:Token: None 2022-09-12T19:03:46.969 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:46 GMT /jdpiak-17/?versions 2022-09-12T19:03:46.970 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KopWxUKJDZUsWW1PR5PudFyZEDg= 2022-09-12T19:03:46.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KopWxUKJDZUsWW1PR5PudFyZEDg=', 'Content-Length': '0'} 2022-09-12T19:03:46.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039b54ff376537560-00631f8292-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:46.974 DEBUG:boto:b'jdpiak-17999falsedummybSzfRw-Zsh7lYv-ClgdhoG098KzYg.Strue2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpktrue2022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone20f1rsOoGVVMRuyy11YfWbutRvo943lCtrue2022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3V6Ue4vhVY0WQ9FArGixLjY-hmND5K20true2022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:46.975 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:03:46.975 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:03:46.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:03:46.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:03:46.976 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:46.977 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:46.977 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:46.978 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:46.979 DEBUG:boto:Method: GET 2022-09-12T19:03:46.979 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:46.979 DEBUG:boto:Data: 2022-09-12T19:03:46.980 DEBUG:boto:Headers: {} 2022-09-12T19:03:46.980 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:46.981 DEBUG:boto:Port: 8000 2022-09-12T19:03:46.981 DEBUG:boto:Params: {} 2022-09-12T19:03:46.982 DEBUG:boto:Token: None 2022-09-12T19:03:46.983 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:46 GMT /jdpiak-17/?versions 2022-09-12T19:03:46.983 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KopWxUKJDZUsWW1PR5PudFyZEDg= 2022-09-12T19:03:46.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KopWxUKJDZUsWW1PR5PudFyZEDg=', 'Content-Length': '0'} 2022-09-12T19:03:46.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd7f647df74255fc-00631f8292-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:46.987 DEBUG:boto:b'jdpiak-17999falsedummybSzfRw-Zsh7lYv-ClgdhoG098KzYg.Strue2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpktrue2022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone20f1rsOoGVVMRuyy11YfWbutRvo943lCtrue2022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3V6Ue4vhVY0WQ9FArGixLjY-hmND5K20true2022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:46.988 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:46.988 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:46.989 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:46.990 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:46.990 DEBUG:boto:Method: GET 2022-09-12T19:03:46.991 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:46.991 DEBUG:boto:Data: 2022-09-12T19:03:46.992 DEBUG:boto:Headers: {} 2022-09-12T19:03:46.992 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:46.993 DEBUG:boto:Port: 8000 2022-09-12T19:03:46.993 DEBUG:boto:Params: {} 2022-09-12T19:03:46.994 DEBUG:boto:Token: None 2022-09-12T19:03:46.994 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:46 GMT /jdpiak-17/?versions 2022-09-12T19:03:46.995 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KopWxUKJDZUsWW1PR5PudFyZEDg= 2022-09-12T19:03:46.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KopWxUKJDZUsWW1PR5PudFyZEDg=', 'Content-Length': '0'} 2022-09-12T19:03:46.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007793ce64b3591b18-00631f8292-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:46.999 DEBUG:boto:b'jdpiak-17999falsedummybSzfRw-Zsh7lYv-ClgdhoG098KzYg.Strue2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpktrue2022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone20f1rsOoGVVMRuyy11YfWbutRvo943lCtrue2022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3V6Ue4vhVY0WQ9FArGixLjY-hmND5K20true2022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:47.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:47.001 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:47.001 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:47.002 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.003 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.003 DEBUG:boto:Method: GET 2022-09-12T19:03:47.004 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.004 DEBUG:boto:Data: 2022-09-12T19:03:47.005 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.006 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.007 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.008 DEBUG:boto:Params: {} 2022-09-12T19:03:47.009 DEBUG:boto:Token: None 2022-09-12T19:03:47.010 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.012 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Oi1i9b9urPnY29lCzsqtHNoZ5rQ= 2022-09-12T19:03:47.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Oi1i9b9urPnY29lCzsqtHNoZ5rQ=', 'Content-Length': '0'} 2022-09-12T19:03:47.015 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e69a931c41f998c4-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.017 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:47.017 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:47.018 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.018 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.019 DEBUG:boto:Method: GET 2022-09-12T19:03:47.019 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.020 DEBUG:boto:Data: 2022-09-12T19:03:47.020 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.021 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.022 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.022 DEBUG:boto:Params: {} 2022-09-12T19:03:47.023 DEBUG:boto:Token: None 2022-09-12T19:03:47.023 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.024 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Oi1i9b9urPnY29lCzsqtHNoZ5rQ= 2022-09-12T19:03:47.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Oi1i9b9urPnY29lCzsqtHNoZ5rQ=', 'Content-Length': '0'} 2022-09-12T19:03:47.026 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c31cb4d896da68b4-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.027 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:47.028 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:47.028 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.029 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.029 DEBUG:boto:Method: HEAD 2022-09-12T19:03:47.030 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.030 DEBUG:boto:Data: 2022-09-12T19:03:47.031 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.031 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.032 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.032 DEBUG:boto:Params: {} 2022-09-12T19:03:47.033 DEBUG:boto:Token: None 2022-09-12T19:03:47.034 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.034 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EHNPdM1p5drT/bx+M3mBu+LTBHo= 2022-09-12T19:03:47.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EHNPdM1p5drT/bx+M3mBu+LTBHo=', 'Content-Length': '0'} 2022-09-12T19:03:47.037 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c221ee5dfacc58ca-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.038 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:47.038 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:47.039 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.039 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.040 DEBUG:boto:Method: HEAD 2022-09-12T19:03:47.040 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.041 DEBUG:boto:Data: 2022-09-12T19:03:47.042 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.042 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.043 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.043 DEBUG:boto:Params: {} 2022-09-12T19:03:47.044 DEBUG:boto:Token: None 2022-09-12T19:03:47.045 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.045 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EHNPdM1p5drT/bx+M3mBu+LTBHo= 2022-09-12T19:03:47.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EHNPdM1p5drT/bx+M3mBu+LTBHo=', 'Content-Length': '0'} 2022-09-12T19:03:47.048 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002442e6013709a7c7-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:47.049 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:47.050 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:47.050 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.051 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.051 DEBUG:boto:Method: GET 2022-09-12T19:03:47.052 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.053 DEBUG:boto:Data: 2022-09-12T19:03:47.054 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.055 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.056 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.057 DEBUG:boto:Params: {} 2022-09-12T19:03:47.058 DEBUG:boto:Token: None 2022-09-12T19:03:47.059 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.060 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Oi1i9b9urPnY29lCzsqtHNoZ5rQ= 2022-09-12T19:03:47.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Oi1i9b9urPnY29lCzsqtHNoZ5rQ=', 'Content-Length': '0'} 2022-09-12T19:03:47.063 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b94541047e4ce407-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.064 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:47.065 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:47.066 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.067 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.068 DEBUG:boto:Method: GET 2022-09-12T19:03:47.069 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.070 DEBUG:boto:Data: 2022-09-12T19:03:47.071 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.072 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.072 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.073 DEBUG:boto:Params: {} 2022-09-12T19:03:47.074 DEBUG:boto:Token: None 2022-09-12T19:03:47.075 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.076 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Oi1i9b9urPnY29lCzsqtHNoZ5rQ= 2022-09-12T19:03:47.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Oi1i9b9urPnY29lCzsqtHNoZ5rQ=', 'Content-Length': '0'} 2022-09-12T19:03:47.080 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000212a5e63c48cd893-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.081 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:47.081 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:47.082 DEBUG:boto:path=/jdpiak-17/?prefix=dummy 2022-09-12T19:03:47.082 DEBUG:boto:auth_path=/jdpiak-17/?prefix=dummy 2022-09-12T19:03:47.083 DEBUG:boto:Method: GET 2022-09-12T19:03:47.083 DEBUG:boto:Path: /jdpiak-17/?prefix=dummy 2022-09-12T19:03:47.084 DEBUG:boto:Data: 2022-09-12T19:03:47.084 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.085 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.086 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.087 DEBUG:boto:Params: {} 2022-09-12T19:03:47.088 DEBUG:boto:Token: None 2022-09-12T19:03:47.089 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/ 2022-09-12T19:03:47.090 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g= 2022-09-12T19:03:47.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g=', 'Content-Length': '0'} 2022-09-12T19:03:47.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000264cec7c6f42045-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.094 DEBUG:boto:b'jdpiak-17dummy1000falsedummy2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:47.095 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:47.096 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:47.096 DEBUG:boto:path=/jdpiak-17/?prefix=dummy 2022-09-12T19:03:47.097 DEBUG:boto:auth_path=/jdpiak-17/?prefix=dummy 2022-09-12T19:03:47.098 DEBUG:boto:Method: GET 2022-09-12T19:03:47.098 DEBUG:boto:Path: /jdpiak-17/?prefix=dummy 2022-09-12T19:03:47.099 DEBUG:boto:Data: 2022-09-12T19:03:47.099 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.100 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.101 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.101 DEBUG:boto:Params: {} 2022-09-12T19:03:47.102 DEBUG:boto:Token: None 2022-09-12T19:03:47.103 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/ 2022-09-12T19:03:47.104 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g= 2022-09-12T19:03:47.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g=', 'Content-Length': '0'} 2022-09-12T19:03:47.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004599d229550308d6-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.150 DEBUG:boto:b'jdpiak-17dummy1000falsedummy2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:47.151 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:47.152 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:47.152 DEBUG:boto:Method: HEAD 2022-09-12T19:03:47.153 DEBUG:boto:Path: /jdpiak-17/dummy 2022-09-12T19:03:47.153 DEBUG:boto:Data: 2022-09-12T19:03:47.154 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.154 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.155 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.155 DEBUG:boto:Params: {} 2022-09-12T19:03:47.156 DEBUG:boto:Token: None 2022-09-12T19:03:47.156 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/dummy 2022-09-12T19:03:47.157 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fUQKgdc93dKCbssk5FRuFa+h1ek= 2022-09-12T19:03:47.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fUQKgdc93dKCbssk5FRuFa+h1ek=', 'Content-Length': '0'} 2022-09-12T19:03:47.159 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000b317eea1040bac5-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.160 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:47.161 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:47.161 DEBUG:boto:Method: HEAD 2022-09-12T19:03:47.162 DEBUG:boto:Path: /jdpiak-17/dummy 2022-09-12T19:03:47.162 DEBUG:boto:Data: 2022-09-12T19:03:47.163 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.163 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.164 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.164 DEBUG:boto:Params: {} 2022-09-12T19:03:47.165 DEBUG:boto:Token: None 2022-09-12T19:03:47.165 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/dummy 2022-09-12T19:03:47.166 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fUQKgdc93dKCbssk5FRuFa+h1ek= 2022-09-12T19:03:47.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fUQKgdc93dKCbssk5FRuFa+h1ek=', 'Content-Length': '0'} 2022-09-12T19:03:47.168 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008d68f3a09503ea7c-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:47.170 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:47.170 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:47.171 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.171 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.172 DEBUG:boto:Method: GET 2022-09-12T19:03:47.172 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.173 DEBUG:boto:Data: 2022-09-12T19:03:47.173 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.174 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.174 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.175 DEBUG:boto:Params: {} 2022-09-12T19:03:47.175 DEBUG:boto:Token: None 2022-09-12T19:03:47.176 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.177 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Oi1i9b9urPnY29lCzsqtHNoZ5rQ= 2022-09-12T19:03:47.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Oi1i9b9urPnY29lCzsqtHNoZ5rQ=', 'Content-Length': '0'} 2022-09-12T19:03:47.179 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006cf12cd267a86456-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.180 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:47.181 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:47.181 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.182 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.182 DEBUG:boto:Method: GET 2022-09-12T19:03:47.183 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.183 DEBUG:boto:Data: 2022-09-12T19:03:47.184 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.184 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.185 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.185 DEBUG:boto:Params: {} 2022-09-12T19:03:47.186 DEBUG:boto:Token: None 2022-09-12T19:03:47.187 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.187 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Oi1i9b9urPnY29lCzsqtHNoZ5rQ= 2022-09-12T19:03:47.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Oi1i9b9urPnY29lCzsqtHNoZ5rQ=', 'Content-Length': '0'} 2022-09-12T19:03:47.190 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000509a009d1ad5dda7-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.190 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:47.191 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:47.192 DEBUG:boto:path=/jdpiak-17/?prefix=dummy 2022-09-12T19:03:47.192 DEBUG:boto:auth_path=/jdpiak-17/?prefix=dummy 2022-09-12T19:03:47.193 DEBUG:boto:Method: GET 2022-09-12T19:03:47.193 DEBUG:boto:Path: /jdpiak-17/?prefix=dummy 2022-09-12T19:03:47.194 DEBUG:boto:Data: 2022-09-12T19:03:47.194 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.195 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.195 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.196 DEBUG:boto:Params: {} 2022-09-12T19:03:47.196 DEBUG:boto:Token: None 2022-09-12T19:03:47.197 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/ 2022-09-12T19:03:47.197 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g= 2022-09-12T19:03:47.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g=', 'Content-Length': '0'} 2022-09-12T19:03:47.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b480f1cba903cda9-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.201 DEBUG:boto:b'jdpiak-17dummy1000falsedummy2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:47.202 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:47.202 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:47.203 DEBUG:boto:path=/jdpiak-17/?prefix=dummy 2022-09-12T19:03:47.203 DEBUG:boto:auth_path=/jdpiak-17/?prefix=dummy 2022-09-12T19:03:47.204 DEBUG:boto:Method: GET 2022-09-12T19:03:47.204 DEBUG:boto:Path: /jdpiak-17/?prefix=dummy 2022-09-12T19:03:47.205 DEBUG:boto:Data: 2022-09-12T19:03:47.205 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.206 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.206 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.207 DEBUG:boto:Params: {} 2022-09-12T19:03:47.207 DEBUG:boto:Token: None 2022-09-12T19:03:47.208 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/ 2022-09-12T19:03:47.208 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g= 2022-09-12T19:03:47.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g=', 'Content-Length': '0'} 2022-09-12T19:03:47.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3a9293b7f02232c-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.212 DEBUG:boto:b'jdpiak-17dummy1000falsedummy2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:47.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:47.214 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:47.214 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:47.215 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.215 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.216 DEBUG:boto:Method: GET 2022-09-12T19:03:47.216 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.217 DEBUG:boto:Data: 2022-09-12T19:03:47.217 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.218 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.218 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.219 DEBUG:boto:Params: {} 2022-09-12T19:03:47.219 DEBUG:boto:Token: None 2022-09-12T19:03:47.220 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.221 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yAzyz20+m3bKrAI2FpJ0sMF3Jdc= 2022-09-12T19:03:47.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yAzyz20+m3bKrAI2FpJ0sMF3Jdc=', 'Content-Length': '0'} 2022-09-12T19:03:47.224 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000abdbe7f1b5445dc7-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.225 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:47.225 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:47.226 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.227 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.227 DEBUG:boto:Method: GET 2022-09-12T19:03:47.228 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.228 DEBUG:boto:Data: 2022-09-12T19:03:47.229 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.229 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.230 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.230 DEBUG:boto:Params: {} 2022-09-12T19:03:47.231 DEBUG:boto:Token: None 2022-09-12T19:03:47.231 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.232 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yAzyz20+m3bKrAI2FpJ0sMF3Jdc= 2022-09-12T19:03:47.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yAzyz20+m3bKrAI2FpJ0sMF3Jdc=', 'Content-Length': '0'} 2022-09-12T19:03:47.235 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae5762841caec62c-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.235 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:47.236 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:47.236 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.237 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.237 DEBUG:boto:Method: HEAD 2022-09-12T19:03:47.238 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.238 DEBUG:boto:Data: 2022-09-12T19:03:47.239 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.239 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.240 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.240 DEBUG:boto:Params: {} 2022-09-12T19:03:47.241 DEBUG:boto:Token: None 2022-09-12T19:03:47.242 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.242 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DDQwDcnPzHEiKJR5iWirsEsVpC8= 2022-09-12T19:03:47.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DDQwDcnPzHEiKJR5iWirsEsVpC8=', 'Content-Length': '0'} 2022-09-12T19:03:47.245 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aac8981c83d54a1b-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.246 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:47.246 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:47.247 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.247 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.248 DEBUG:boto:Method: HEAD 2022-09-12T19:03:47.248 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.249 DEBUG:boto:Data: 2022-09-12T19:03:47.249 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.250 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.250 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.251 DEBUG:boto:Params: {} 2022-09-12T19:03:47.251 DEBUG:boto:Token: None 2022-09-12T19:03:47.252 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.252 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DDQwDcnPzHEiKJR5iWirsEsVpC8= 2022-09-12T19:03:47.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DDQwDcnPzHEiKJR5iWirsEsVpC8=', 'Content-Length': '0'} 2022-09-12T19:03:47.255 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8162d21ef842b6e-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:47.256 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:47.257 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:47.257 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.258 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.259 DEBUG:boto:Method: GET 2022-09-12T19:03:47.259 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.260 DEBUG:boto:Data: 2022-09-12T19:03:47.260 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.261 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.261 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.262 DEBUG:boto:Params: {} 2022-09-12T19:03:47.263 DEBUG:boto:Token: None 2022-09-12T19:03:47.263 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.264 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yAzyz20+m3bKrAI2FpJ0sMF3Jdc= 2022-09-12T19:03:47.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yAzyz20+m3bKrAI2FpJ0sMF3Jdc=', 'Content-Length': '0'} 2022-09-12T19:03:47.266 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c6b3d3f7757813e-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.267 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:47.268 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:47.268 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.269 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.269 DEBUG:boto:Method: GET 2022-09-12T19:03:47.270 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.270 DEBUG:boto:Data: 2022-09-12T19:03:47.271 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.272 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.272 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.273 DEBUG:boto:Params: {} 2022-09-12T19:03:47.273 DEBUG:boto:Token: None 2022-09-12T19:03:47.274 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.275 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yAzyz20+m3bKrAI2FpJ0sMF3Jdc= 2022-09-12T19:03:47.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yAzyz20+m3bKrAI2FpJ0sMF3Jdc=', 'Content-Length': '0'} 2022-09-12T19:03:47.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007fd479ffaddb0f5d-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.278 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:47.279 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:47.279 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:47.280 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:47.280 DEBUG:boto:Method: GET 2022-09-12T19:03:47.281 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:47.281 DEBUG:boto:Data: 2022-09-12T19:03:47.282 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.283 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.283 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.284 DEBUG:boto:Params: {} 2022-09-12T19:03:47.284 DEBUG:boto:Token: None 2022-09-12T19:03:47.285 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/ 2022-09-12T19:03:47.286 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g= 2022-09-12T19:03:47.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g=', 'Content-Length': '0'} 2022-09-12T19:03:47.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d002931a72087b03-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.330 DEBUG:boto:b'jdpiak-17obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:47.331 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:47.332 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:47.332 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:47.333 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:47.333 DEBUG:boto:Method: GET 2022-09-12T19:03:47.334 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:47.334 DEBUG:boto:Data: 2022-09-12T19:03:47.335 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.335 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.336 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.336 DEBUG:boto:Params: {} 2022-09-12T19:03:47.337 DEBUG:boto:Token: None 2022-09-12T19:03:47.338 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/ 2022-09-12T19:03:47.338 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g= 2022-09-12T19:03:47.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g=', 'Content-Length': '0'} 2022-09-12T19:03:47.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001def00a174693389-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.342 DEBUG:boto:b'jdpiak-17obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:47.343 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:47.344 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:47.344 DEBUG:boto:Method: HEAD 2022-09-12T19:03:47.345 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1 2022-09-12T19:03:47.345 DEBUG:boto:Data: 2022-09-12T19:03:47.346 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.346 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.347 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.347 DEBUG:boto:Params: {} 2022-09-12T19:03:47.348 DEBUG:boto:Token: None 2022-09-12T19:03:47.349 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone1 2022-09-12T19:03:47.349 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uBswbPOveGHKsyfuT051bM7EMPc= 2022-09-12T19:03:47.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uBswbPOveGHKsyfuT051bM7EMPc=', 'Content-Length': '0'} 2022-09-12T19:03:47.352 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004a9576380ab73b1-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.353 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:47.353 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:47.354 DEBUG:boto:Method: HEAD 2022-09-12T19:03:47.354 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1 2022-09-12T19:03:47.355 DEBUG:boto:Data: 2022-09-12T19:03:47.355 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.356 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.356 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.357 DEBUG:boto:Params: {} 2022-09-12T19:03:47.357 DEBUG:boto:Token: None 2022-09-12T19:03:47.358 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone1 2022-09-12T19:03:47.359 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uBswbPOveGHKsyfuT051bM7EMPc= 2022-09-12T19:03:47.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uBswbPOveGHKsyfuT051bM7EMPc=', 'Content-Length': '0'} 2022-09-12T19:03:47.362 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000bd582af2d97a15f-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:47.363 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:47.363 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:47.364 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.365 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.365 DEBUG:boto:Method: GET 2022-09-12T19:03:47.366 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.366 DEBUG:boto:Data: 2022-09-12T19:03:47.367 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.367 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.368 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.368 DEBUG:boto:Params: {} 2022-09-12T19:03:47.369 DEBUG:boto:Token: None 2022-09-12T19:03:47.370 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.371 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yAzyz20+m3bKrAI2FpJ0sMF3Jdc= 2022-09-12T19:03:47.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yAzyz20+m3bKrAI2FpJ0sMF3Jdc=', 'Content-Length': '0'} 2022-09-12T19:03:47.373 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009bc1951a9544561d-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.374 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:47.375 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:47.375 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.376 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.376 DEBUG:boto:Method: GET 2022-09-12T19:03:47.377 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.377 DEBUG:boto:Data: 2022-09-12T19:03:47.378 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.379 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.379 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.380 DEBUG:boto:Params: {} 2022-09-12T19:03:47.380 DEBUG:boto:Token: None 2022-09-12T19:03:47.381 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:47.382 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yAzyz20+m3bKrAI2FpJ0sMF3Jdc= 2022-09-12T19:03:47.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yAzyz20+m3bKrAI2FpJ0sMF3Jdc=', 'Content-Length': '0'} 2022-09-12T19:03:47.384 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007fd85bec2b96633e-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.385 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:47.386 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:47.386 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:47.387 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:47.387 DEBUG:boto:Method: GET 2022-09-12T19:03:47.388 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:47.388 DEBUG:boto:Data: 2022-09-12T19:03:47.389 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.389 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.390 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.390 DEBUG:boto:Params: {} 2022-09-12T19:03:47.391 DEBUG:boto:Token: None 2022-09-12T19:03:47.392 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/ 2022-09-12T19:03:47.392 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g= 2022-09-12T19:03:47.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g=', 'Content-Length': '0'} 2022-09-12T19:03:47.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d16e2828e867fe29-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.397 DEBUG:boto:b'jdpiak-17obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:47.398 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:47.398 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:47.399 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:47.399 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:47.400 DEBUG:boto:Method: GET 2022-09-12T19:03:47.400 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:47.401 DEBUG:boto:Data: 2022-09-12T19:03:47.401 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.402 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.403 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.403 DEBUG:boto:Params: {} 2022-09-12T19:03:47.404 DEBUG:boto:Token: None 2022-09-12T19:03:47.404 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/ 2022-09-12T19:03:47.405 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g= 2022-09-12T19:03:47.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g=', 'Content-Length': '0'} 2022-09-12T19:03:47.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001050910e7184ac13-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.409 DEBUG:boto:b'jdpiak-17obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:47.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:47.410 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:47.411 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:47.412 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.412 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.413 DEBUG:boto:Method: GET 2022-09-12T19:03:47.413 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.414 DEBUG:boto:Data: 2022-09-12T19:03:47.415 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.415 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.416 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.416 DEBUG:boto:Params: {} 2022-09-12T19:03:47.417 DEBUG:boto:Token: None 2022-09-12T19:03:47.417 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.418 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VhfGqs3RFWA5UXNDciCgJKorjRA= 2022-09-12T19:03:47.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VhfGqs3RFWA5UXNDciCgJKorjRA=', 'Content-Length': '0'} 2022-09-12T19:03:47.421 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008389593dce6096bd-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.422 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:47.423 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:47.423 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.424 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.424 DEBUG:boto:Method: GET 2022-09-12T19:03:47.425 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.425 DEBUG:boto:Data: 2022-09-12T19:03:47.426 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.426 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.427 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.427 DEBUG:boto:Params: {} 2022-09-12T19:03:47.428 DEBUG:boto:Token: None 2022-09-12T19:03:47.429 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.429 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VhfGqs3RFWA5UXNDciCgJKorjRA= 2022-09-12T19:03:47.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VhfGqs3RFWA5UXNDciCgJKorjRA=', 'Content-Length': '0'} 2022-09-12T19:03:47.432 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae6543273aa51e58-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.433 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:47.434 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:47.434 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.435 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.436 DEBUG:boto:Method: HEAD 2022-09-12T19:03:47.436 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.437 DEBUG:boto:Data: 2022-09-12T19:03:47.437 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.438 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.439 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.439 DEBUG:boto:Params: {} 2022-09-12T19:03:47.440 DEBUG:boto:Token: None 2022-09-12T19:03:47.440 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.441 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ti29DNOTkDFdX/aLtJyztK+IVFQ= 2022-09-12T19:03:47.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ti29DNOTkDFdX/aLtJyztK+IVFQ=', 'Content-Length': '0'} 2022-09-12T19:03:47.444 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000006be650d60c06d9-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.444 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:47.445 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:47.446 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.446 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.447 DEBUG:boto:Method: HEAD 2022-09-12T19:03:47.447 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.448 DEBUG:boto:Data: 2022-09-12T19:03:47.448 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.449 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.449 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.450 DEBUG:boto:Params: {} 2022-09-12T19:03:47.450 DEBUG:boto:Token: None 2022-09-12T19:03:47.451 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.451 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ti29DNOTkDFdX/aLtJyztK+IVFQ= 2022-09-12T19:03:47.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ti29DNOTkDFdX/aLtJyztK+IVFQ=', 'Content-Length': '0'} 2022-09-12T19:03:47.454 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000efc54c5d451fb39b-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:47.455 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:47.456 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:47.456 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.457 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.457 DEBUG:boto:Method: GET 2022-09-12T19:03:47.458 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.458 DEBUG:boto:Data: 2022-09-12T19:03:47.459 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.459 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.460 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.460 DEBUG:boto:Params: {} 2022-09-12T19:03:47.461 DEBUG:boto:Token: None 2022-09-12T19:03:47.462 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.462 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VhfGqs3RFWA5UXNDciCgJKorjRA= 2022-09-12T19:03:47.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VhfGqs3RFWA5UXNDciCgJKorjRA=', 'Content-Length': '0'} 2022-09-12T19:03:47.465 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000476badf0bf23a8df-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.466 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:47.466 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:47.467 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.467 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.468 DEBUG:boto:Method: GET 2022-09-12T19:03:47.468 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.469 DEBUG:boto:Data: 2022-09-12T19:03:47.469 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.470 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.471 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.471 DEBUG:boto:Params: {} 2022-09-12T19:03:47.472 DEBUG:boto:Token: None 2022-09-12T19:03:47.472 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.473 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VhfGqs3RFWA5UXNDciCgJKorjRA= 2022-09-12T19:03:47.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VhfGqs3RFWA5UXNDciCgJKorjRA=', 'Content-Length': '0'} 2022-09-12T19:03:47.475 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000021d4bd6ca46decdc-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.476 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:47.477 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:47.477 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:47.478 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:47.478 DEBUG:boto:Method: GET 2022-09-12T19:03:47.479 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:47.480 DEBUG:boto:Data: 2022-09-12T19:03:47.480 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.481 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.481 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.482 DEBUG:boto:Params: {} 2022-09-12T19:03:47.482 DEBUG:boto:Token: None 2022-09-12T19:03:47.483 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/ 2022-09-12T19:03:47.483 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g= 2022-09-12T19:03:47.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g=', 'Content-Length': '0'} 2022-09-12T19:03:47.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000850c5780465a1f03-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.487 DEBUG:boto:b'jdpiak-17obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:47.488 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:47.489 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:47.489 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:47.490 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:47.490 DEBUG:boto:Method: GET 2022-09-12T19:03:47.491 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:47.491 DEBUG:boto:Data: 2022-09-12T19:03:47.492 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.492 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.493 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.494 DEBUG:boto:Params: {} 2022-09-12T19:03:47.494 DEBUG:boto:Token: None 2022-09-12T19:03:47.495 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/ 2022-09-12T19:03:47.495 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g= 2022-09-12T19:03:47.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g=', 'Content-Length': '0'} 2022-09-12T19:03:47.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bea657863b05964-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.499 DEBUG:boto:b'jdpiak-17obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:47.500 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:47.500 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:47.501 DEBUG:boto:Method: HEAD 2022-09-12T19:03:47.501 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2 2022-09-12T19:03:47.502 DEBUG:boto:Data: 2022-09-12T19:03:47.503 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.503 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.503 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.504 DEBUG:boto:Params: {} 2022-09-12T19:03:47.505 DEBUG:boto:Token: None 2022-09-12T19:03:47.505 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone2 2022-09-12T19:03:47.506 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+5XF8Tj+hhAwtBkSXc+2OGNy/Hw= 2022-09-12T19:03:47.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+5XF8Tj+hhAwtBkSXc+2OGNy/Hw=', 'Content-Length': '0'} 2022-09-12T19:03:47.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd771dc638adfe69-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.510 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:47.511 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:47.511 DEBUG:boto:Method: HEAD 2022-09-12T19:03:47.512 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2 2022-09-12T19:03:47.512 DEBUG:boto:Data: 2022-09-12T19:03:47.513 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.513 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.514 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.514 DEBUG:boto:Params: {} 2022-09-12T19:03:47.515 DEBUG:boto:Token: None 2022-09-12T19:03:47.515 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone2 2022-09-12T19:03:47.516 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+5XF8Tj+hhAwtBkSXc+2OGNy/Hw= 2022-09-12T19:03:47.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+5XF8Tj+hhAwtBkSXc+2OGNy/Hw=', 'Content-Length': '0'} 2022-09-12T19:03:47.519 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6fa9bc7e22bccd0-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:47.520 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:47.521 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:47.521 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.522 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.522 DEBUG:boto:Method: GET 2022-09-12T19:03:47.523 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.523 DEBUG:boto:Data: 2022-09-12T19:03:47.524 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.525 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.525 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.526 DEBUG:boto:Params: {} 2022-09-12T19:03:47.526 DEBUG:boto:Token: None 2022-09-12T19:03:47.527 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.528 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VhfGqs3RFWA5UXNDciCgJKorjRA= 2022-09-12T19:03:47.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VhfGqs3RFWA5UXNDciCgJKorjRA=', 'Content-Length': '0'} 2022-09-12T19:03:47.530 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014b6c74a04d0d049-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.531 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:47.531 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:47.532 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.532 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.533 DEBUG:boto:Method: GET 2022-09-12T19:03:47.533 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.534 DEBUG:boto:Data: 2022-09-12T19:03:47.535 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.535 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.536 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.536 DEBUG:boto:Params: {} 2022-09-12T19:03:47.537 DEBUG:boto:Token: None 2022-09-12T19:03:47.537 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:47.538 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VhfGqs3RFWA5UXNDciCgJKorjRA= 2022-09-12T19:03:47.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VhfGqs3RFWA5UXNDciCgJKorjRA=', 'Content-Length': '0'} 2022-09-12T19:03:47.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bffc92803dfcecc3-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.582 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:47.583 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:47.584 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:47.584 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:47.585 DEBUG:boto:Method: GET 2022-09-12T19:03:47.585 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:47.586 DEBUG:boto:Data: 2022-09-12T19:03:47.586 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.587 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.588 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.588 DEBUG:boto:Params: {} 2022-09-12T19:03:47.589 DEBUG:boto:Token: None 2022-09-12T19:03:47.590 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/ 2022-09-12T19:03:47.590 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g= 2022-09-12T19:03:47.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g=', 'Content-Length': '0'} 2022-09-12T19:03:47.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002af4eb183f19ec4b-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.594 DEBUG:boto:b'jdpiak-17obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:47.595 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:47.596 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:47.597 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:47.597 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:47.598 DEBUG:boto:Method: GET 2022-09-12T19:03:47.598 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:47.599 DEBUG:boto:Data: 2022-09-12T19:03:47.599 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.600 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.600 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.601 DEBUG:boto:Params: {} 2022-09-12T19:03:47.602 DEBUG:boto:Token: None 2022-09-12T19:03:47.603 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/ 2022-09-12T19:03:47.603 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g= 2022-09-12T19:03:47.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g=', 'Content-Length': '0'} 2022-09-12T19:03:47.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e47113be65cdab60-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.608 DEBUG:boto:b'jdpiak-17obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:47.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:47.609 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:47.610 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:47.610 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.611 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.611 DEBUG:boto:Method: GET 2022-09-12T19:03:47.612 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.612 DEBUG:boto:Data: 2022-09-12T19:03:47.613 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.613 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.614 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.615 DEBUG:boto:Params: {} 2022-09-12T19:03:47.615 DEBUG:boto:Token: None 2022-09-12T19:03:47.616 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.616 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FGMAfvKHyJeKBPp6v/9kcB/ypxo= 2022-09-12T19:03:47.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FGMAfvKHyJeKBPp6v/9kcB/ypxo=', 'Content-Length': '0'} 2022-09-12T19:03:47.619 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a048315d46238d7a-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.620 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:47.620 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:47.621 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.621 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.622 DEBUG:boto:Method: GET 2022-09-12T19:03:47.622 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.623 DEBUG:boto:Data: 2022-09-12T19:03:47.623 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.624 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.624 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.625 DEBUG:boto:Params: {} 2022-09-12T19:03:47.626 DEBUG:boto:Token: None 2022-09-12T19:03:47.626 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.627 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FGMAfvKHyJeKBPp6v/9kcB/ypxo= 2022-09-12T19:03:47.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FGMAfvKHyJeKBPp6v/9kcB/ypxo=', 'Content-Length': '0'} 2022-09-12T19:03:47.629 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004fe854d27c9194e8-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.630 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:47.631 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:47.631 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.632 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.632 DEBUG:boto:Method: HEAD 2022-09-12T19:03:47.633 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.633 DEBUG:boto:Data: 2022-09-12T19:03:47.633 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.634 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.634 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.635 DEBUG:boto:Params: {} 2022-09-12T19:03:47.636 DEBUG:boto:Token: None 2022-09-12T19:03:47.636 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.637 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qlCLXsQTq/FuQdM5DhfnztE/kNo= 2022-09-12T19:03:47.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qlCLXsQTq/FuQdM5DhfnztE/kNo=', 'Content-Length': '0'} 2022-09-12T19:03:47.639 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0993a0374fd51d8-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.640 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:47.641 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:47.641 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.642 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.642 DEBUG:boto:Method: HEAD 2022-09-12T19:03:47.643 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.644 DEBUG:boto:Data: 2022-09-12T19:03:47.644 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.645 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.645 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.646 DEBUG:boto:Params: {} 2022-09-12T19:03:47.647 DEBUG:boto:Token: None 2022-09-12T19:03:47.647 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.648 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qlCLXsQTq/FuQdM5DhfnztE/kNo= 2022-09-12T19:03:47.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qlCLXsQTq/FuQdM5DhfnztE/kNo=', 'Content-Length': '0'} 2022-09-12T19:03:47.650 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a3782fe7c14759b-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:47.652 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:47.652 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:47.653 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.653 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.654 DEBUG:boto:Method: GET 2022-09-12T19:03:47.655 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.655 DEBUG:boto:Data: 2022-09-12T19:03:47.656 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.656 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.657 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.657 DEBUG:boto:Params: {} 2022-09-12T19:03:47.658 DEBUG:boto:Token: None 2022-09-12T19:03:47.659 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.659 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FGMAfvKHyJeKBPp6v/9kcB/ypxo= 2022-09-12T19:03:47.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FGMAfvKHyJeKBPp6v/9kcB/ypxo=', 'Content-Length': '0'} 2022-09-12T19:03:47.662 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015f6758c7765384a-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.662 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:47.663 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:47.663 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.664 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.665 DEBUG:boto:Method: GET 2022-09-12T19:03:47.665 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.666 DEBUG:boto:Data: 2022-09-12T19:03:47.666 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.667 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.667 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.668 DEBUG:boto:Params: {} 2022-09-12T19:03:47.669 DEBUG:boto:Token: None 2022-09-12T19:03:47.669 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.670 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FGMAfvKHyJeKBPp6v/9kcB/ypxo= 2022-09-12T19:03:47.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FGMAfvKHyJeKBPp6v/9kcB/ypxo=', 'Content-Length': '0'} 2022-09-12T19:03:47.672 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051d2b103777d5009-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.714 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:47.715 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:47.716 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:47.717 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:47.718 DEBUG:boto:Method: GET 2022-09-12T19:03:47.719 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:47.719 DEBUG:boto:Data: 2022-09-12T19:03:47.720 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.720 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.721 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.721 DEBUG:boto:Params: {} 2022-09-12T19:03:47.722 DEBUG:boto:Token: None 2022-09-12T19:03:47.723 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/ 2022-09-12T19:03:47.724 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g= 2022-09-12T19:03:47.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g=', 'Content-Length': '0'} 2022-09-12T19:03:47.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007babd133abb9dc21-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.728 DEBUG:boto:b'jdpiak-17obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:47.729 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:47.729 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:47.730 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:47.730 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:47.731 DEBUG:boto:Method: GET 2022-09-12T19:03:47.731 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:47.732 DEBUG:boto:Data: 2022-09-12T19:03:47.732 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.733 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.733 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.734 DEBUG:boto:Params: {} 2022-09-12T19:03:47.734 DEBUG:boto:Token: None 2022-09-12T19:03:47.735 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/ 2022-09-12T19:03:47.736 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g= 2022-09-12T19:03:47.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g=', 'Content-Length': '0'} 2022-09-12T19:03:47.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c37beb1aeb2940c-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.740 DEBUG:boto:b'jdpiak-17obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:47.740 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:47.741 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:47.741 DEBUG:boto:Method: HEAD 2022-09-12T19:03:47.742 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3 2022-09-12T19:03:47.742 DEBUG:boto:Data: 2022-09-12T19:03:47.743 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.743 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.744 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.744 DEBUG:boto:Params: {} 2022-09-12T19:03:47.745 DEBUG:boto:Token: None 2022-09-12T19:03:47.746 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone3 2022-09-12T19:03:47.746 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JSNJTqCK+1817Jm29WSsxlIrBLI= 2022-09-12T19:03:47.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JSNJTqCK+1817Jm29WSsxlIrBLI=', 'Content-Length': '0'} 2022-09-12T19:03:47.749 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063a66b3df6fd6861-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.750 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:47.750 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:47.751 DEBUG:boto:Method: HEAD 2022-09-12T19:03:47.751 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3 2022-09-12T19:03:47.752 DEBUG:boto:Data: 2022-09-12T19:03:47.753 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.753 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.754 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.754 DEBUG:boto:Params: {} 2022-09-12T19:03:47.755 DEBUG:boto:Token: None 2022-09-12T19:03:47.755 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone3 2022-09-12T19:03:47.756 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JSNJTqCK+1817Jm29WSsxlIrBLI= 2022-09-12T19:03:47.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JSNJTqCK+1817Jm29WSsxlIrBLI=', 'Content-Length': '0'} 2022-09-12T19:03:47.759 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa57f827416c6c30-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:47.760 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:47.761 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:47.761 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.762 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.762 DEBUG:boto:Method: GET 2022-09-12T19:03:47.763 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.763 DEBUG:boto:Data: 2022-09-12T19:03:47.764 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.764 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.765 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.765 DEBUG:boto:Params: {} 2022-09-12T19:03:47.766 DEBUG:boto:Token: None 2022-09-12T19:03:47.767 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.767 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FGMAfvKHyJeKBPp6v/9kcB/ypxo= 2022-09-12T19:03:47.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FGMAfvKHyJeKBPp6v/9kcB/ypxo=', 'Content-Length': '0'} 2022-09-12T19:03:47.770 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c432b8102adcbe5c-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.810 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:47.811 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:47.811 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.812 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.812 DEBUG:boto:Method: GET 2022-09-12T19:03:47.813 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.813 DEBUG:boto:Data: 2022-09-12T19:03:47.814 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.815 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.815 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.816 DEBUG:boto:Params: {} 2022-09-12T19:03:47.816 DEBUG:boto:Token: None 2022-09-12T19:03:47.817 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:47.817 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FGMAfvKHyJeKBPp6v/9kcB/ypxo= 2022-09-12T19:03:47.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FGMAfvKHyJeKBPp6v/9kcB/ypxo=', 'Content-Length': '0'} 2022-09-12T19:03:47.820 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7269e70944016a9-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.821 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:47.822 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:47.822 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:47.823 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:47.823 DEBUG:boto:Method: GET 2022-09-12T19:03:47.824 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:47.824 DEBUG:boto:Data: 2022-09-12T19:03:47.825 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.825 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:47.826 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.826 DEBUG:boto:Params: {} 2022-09-12T19:03:47.827 DEBUG:boto:Token: None 2022-09-12T19:03:47.828 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/ 2022-09-12T19:03:47.828 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g= 2022-09-12T19:03:47.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g=', 'Content-Length': '0'} 2022-09-12T19:03:47.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008296e7bf78384d59-00631f8293-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.832 DEBUG:boto:b'jdpiak-17obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:47.833 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:47.833 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:47.834 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:47.835 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:47.835 DEBUG:boto:Method: GET 2022-09-12T19:03:47.836 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:47.836 DEBUG:boto:Data: 2022-09-12T19:03:47.837 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.837 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.838 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.839 DEBUG:boto:Params: {} 2022-09-12T19:03:47.840 DEBUG:boto:Token: None 2022-09-12T19:03:47.840 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/ 2022-09-12T19:03:47.841 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g= 2022-09-12T19:03:47.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g=', 'Content-Length': '0'} 2022-09-12T19:03:47.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000798e2783e151bd15-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.845 DEBUG:boto:b'jdpiak-17obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:47.846 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-17 zones={test-zone2, test-zone1} 2022-09-12T19:03:47.846 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-17 zones={test-zone3, test-zone1} 2022-09-12T19:03:47.847 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:47.847 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:47.848 DEBUG:boto:Method: HEAD 2022-09-12T19:03:47.848 DEBUG:boto:Path: /jdpiak-17/ 2022-09-12T19:03:47.849 DEBUG:boto:Data: 2022-09-12T19:03:47.849 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.850 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:47.850 DEBUG:boto:Port: 8001 2022-09-12T19:03:47.851 DEBUG:boto:Params: {} 2022-09-12T19:03:47.851 DEBUG:boto:Token: None 2022-09-12T19:03:47.852 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/ 2022-09-12T19:03:47.853 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:o3+8tNBj8Crxj3A0qM+k8hO++Wg= 2022-09-12T19:03:47.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:o3+8tNBj8Crxj3A0qM+k8hO++Wg=', 'Content-Length': '0'} 2022-09-12T19:03:47.856 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000057545a7d68b1c0f3-00631f8293-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.856 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:47.857 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:47.857 DEBUG:boto:Method: HEAD 2022-09-12T19:03:47.858 DEBUG:boto:Path: /jdpiak-17/ 2022-09-12T19:03:47.858 DEBUG:boto:Data: 2022-09-12T19:03:47.859 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.859 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.860 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.861 DEBUG:boto:Params: {} 2022-09-12T19:03:47.861 DEBUG:boto:Token: None 2022-09-12T19:03:47.862 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/ 2022-09-12T19:03:47.862 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:o3+8tNBj8Crxj3A0qM+k8hO++Wg= 2022-09-12T19:03:47.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:o3+8tNBj8Crxj3A0qM+k8hO++Wg=', 'Content-Length': '0'} 2022-09-12T19:03:47.865 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c34631c35dc48427-00631f8293-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.866 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:03:47.867 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:47.867 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:47.868 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:47.868 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:47.869 DEBUG:boto:Method: GET 2022-09-12T19:03:47.869 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:47.870 DEBUG:boto:Data: 2022-09-12T19:03:47.870 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.871 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:47.871 DEBUG:boto:Port: 8001 2022-09-12T19:03:47.872 DEBUG:boto:Params: {} 2022-09-12T19:03:47.872 DEBUG:boto:Token: None 2022-09-12T19:03:47.873 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/?versions 2022-09-12T19:03:47.873 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:spL9REzxlhxJItATELkvzLNTHEE= 2022-09-12T19:03:47.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:spL9REzxlhxJItATELkvzLNTHEE=', 'Content-Length': '0'} 2022-09-12T19:03:47.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008951afc57e9137fc-00631f8293-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.877 DEBUG:boto:b'jdpiak-17999falsedummybSzfRw-Zsh7lYv-ClgdhoG098KzYg.Strue2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpktrue2022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone20f1rsOoGVVMRuyy11YfWbutRvo943lCtrue2022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3V6Ue4vhVY0WQ9FArGixLjY-hmND5K20true2022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:47.878 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:03:47.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:03:47.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:03:47.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:03:47.880 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:03:47.881 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:47.881 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:47.882 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:47.882 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:47.883 DEBUG:boto:Method: GET 2022-09-12T19:03:47.883 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:47.884 DEBUG:boto:Data: 2022-09-12T19:03:47.884 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.885 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.885 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.886 DEBUG:boto:Params: {} 2022-09-12T19:03:47.886 DEBUG:boto:Token: None 2022-09-12T19:03:47.887 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/?versions 2022-09-12T19:03:47.888 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:spL9REzxlhxJItATELkvzLNTHEE= 2022-09-12T19:03:47.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:spL9REzxlhxJItATELkvzLNTHEE=', 'Content-Length': '0'} 2022-09-12T19:03:47.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039c9e3d383f1d435-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.891 DEBUG:boto:b'jdpiak-17999falsedummybSzfRw-Zsh7lYv-ClgdhoG098KzYg.Strue2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpktrue2022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone20f1rsOoGVVMRuyy11YfWbutRvo943lCtrue2022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3V6Ue4vhVY0WQ9FArGixLjY-hmND5K20true2022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:47.892 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:03:47.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:03:47.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:03:47.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:03:47.894 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:47.895 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:47.895 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:47.896 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:47.896 DEBUG:boto:Method: GET 2022-09-12T19:03:47.897 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:47.897 DEBUG:boto:Data: 2022-09-12T19:03:47.898 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.898 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:47.899 DEBUG:boto:Port: 8001 2022-09-12T19:03:47.899 DEBUG:boto:Params: {} 2022-09-12T19:03:47.900 DEBUG:boto:Token: None 2022-09-12T19:03:47.901 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/?versions 2022-09-12T19:03:47.901 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:spL9REzxlhxJItATELkvzLNTHEE= 2022-09-12T19:03:47.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:spL9REzxlhxJItATELkvzLNTHEE=', 'Content-Length': '0'} 2022-09-12T19:03:47.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef2857284646b2a8-00631f8293-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.905 DEBUG:boto:b'jdpiak-17999falsedummybSzfRw-Zsh7lYv-ClgdhoG098KzYg.Strue2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpktrue2022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone20f1rsOoGVVMRuyy11YfWbutRvo943lCtrue2022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3V6Ue4vhVY0WQ9FArGixLjY-hmND5K20true2022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:47.906 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:47.907 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:47.907 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:47.908 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:47.908 DEBUG:boto:Method: GET 2022-09-12T19:03:47.909 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:47.909 DEBUG:boto:Data: 2022-09-12T19:03:47.910 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.910 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.911 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.911 DEBUG:boto:Params: {} 2022-09-12T19:03:47.912 DEBUG:boto:Token: None 2022-09-12T19:03:47.912 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/?versions 2022-09-12T19:03:47.913 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:spL9REzxlhxJItATELkvzLNTHEE= 2022-09-12T19:03:47.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:spL9REzxlhxJItATELkvzLNTHEE=', 'Content-Length': '0'} 2022-09-12T19:03:47.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075fef06490b6abd3-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.917 DEBUG:boto:b'jdpiak-17999falsedummybSzfRw-Zsh7lYv-ClgdhoG098KzYg.Strue2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpktrue2022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone20f1rsOoGVVMRuyy11YfWbutRvo943lCtrue2022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3V6Ue4vhVY0WQ9FArGixLjY-hmND5K20true2022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:47.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:47.918 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:47.919 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:47.919 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.920 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.921 DEBUG:boto:Method: GET 2022-09-12T19:03:47.921 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.922 DEBUG:boto:Data: 2022-09-12T19:03:47.922 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.923 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:47.923 DEBUG:boto:Port: 8001 2022-09-12T19:03:47.924 DEBUG:boto:Params: {} 2022-09-12T19:03:47.925 DEBUG:boto:Token: None 2022-09-12T19:03:47.925 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.926 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Oi1i9b9urPnY29lCzsqtHNoZ5rQ= 2022-09-12T19:03:47.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Oi1i9b9urPnY29lCzsqtHNoZ5rQ=', 'Content-Length': '0'} 2022-09-12T19:03:47.928 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000707cccdfe0eb9460-00631f8293-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.929 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:47.930 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:47.931 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.932 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.932 DEBUG:boto:Method: GET 2022-09-12T19:03:47.933 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.934 DEBUG:boto:Data: 2022-09-12T19:03:47.935 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.936 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.936 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.937 DEBUG:boto:Params: {} 2022-09-12T19:03:47.938 DEBUG:boto:Token: None 2022-09-12T19:03:47.939 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.940 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Oi1i9b9urPnY29lCzsqtHNoZ5rQ= 2022-09-12T19:03:47.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Oi1i9b9urPnY29lCzsqtHNoZ5rQ=', 'Content-Length': '0'} 2022-09-12T19:03:47.943 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000db423dcac6c82f12-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.944 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:47.945 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:47.945 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.946 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.946 DEBUG:boto:Method: HEAD 2022-09-12T19:03:47.947 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.947 DEBUG:boto:Data: 2022-09-12T19:03:47.948 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.948 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:47.949 DEBUG:boto:Port: 8001 2022-09-12T19:03:47.949 DEBUG:boto:Params: {} 2022-09-12T19:03:47.950 DEBUG:boto:Token: None 2022-09-12T19:03:47.950 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.951 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EHNPdM1p5drT/bx+M3mBu+LTBHo= 2022-09-12T19:03:47.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EHNPdM1p5drT/bx+M3mBu+LTBHo=', 'Content-Length': '0'} 2022-09-12T19:03:47.953 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005d8c675a637a9636-00631f8293-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.954 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:47.955 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:47.955 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.956 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.956 DEBUG:boto:Method: HEAD 2022-09-12T19:03:47.957 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.957 DEBUG:boto:Data: 2022-09-12T19:03:47.958 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.958 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.959 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.959 DEBUG:boto:Params: {} 2022-09-12T19:03:47.960 DEBUG:boto:Token: None 2022-09-12T19:03:47.960 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.961 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EHNPdM1p5drT/bx+M3mBu+LTBHo= 2022-09-12T19:03:47.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EHNPdM1p5drT/bx+M3mBu+LTBHo=', 'Content-Length': '0'} 2022-09-12T19:03:47.963 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aea534a585b4a8ab-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:47.965 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:47.965 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:47.966 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.966 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.967 DEBUG:boto:Method: GET 2022-09-12T19:03:47.967 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.968 DEBUG:boto:Data: 2022-09-12T19:03:47.969 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.969 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:47.970 DEBUG:boto:Port: 8001 2022-09-12T19:03:47.970 DEBUG:boto:Params: {} 2022-09-12T19:03:47.971 DEBUG:boto:Token: None 2022-09-12T19:03:47.971 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.972 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Oi1i9b9urPnY29lCzsqtHNoZ5rQ= 2022-09-12T19:03:47.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Oi1i9b9urPnY29lCzsqtHNoZ5rQ=', 'Content-Length': '0'} 2022-09-12T19:03:47.974 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ca563d021e29089c-00631f8293-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.975 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:47.976 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:47.976 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.977 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.977 DEBUG:boto:Method: GET 2022-09-12T19:03:47.978 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.978 DEBUG:boto:Data: 2022-09-12T19:03:47.979 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.979 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:47.980 DEBUG:boto:Port: 8000 2022-09-12T19:03:47.980 DEBUG:boto:Params: {} 2022-09-12T19:03:47.981 DEBUG:boto:Token: None 2022-09-12T19:03:47.981 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:47.982 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Oi1i9b9urPnY29lCzsqtHNoZ5rQ= 2022-09-12T19:03:47.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Oi1i9b9urPnY29lCzsqtHNoZ5rQ=', 'Content-Length': '0'} 2022-09-12T19:03:47.984 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000021179509a215967c-00631f8293-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.985 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:47.986 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:47.986 DEBUG:boto:path=/jdpiak-17/?prefix=dummy 2022-09-12T19:03:47.987 DEBUG:boto:auth_path=/jdpiak-17/?prefix=dummy 2022-09-12T19:03:47.987 DEBUG:boto:Method: GET 2022-09-12T19:03:47.988 DEBUG:boto:Path: /jdpiak-17/?prefix=dummy 2022-09-12T19:03:47.988 DEBUG:boto:Data: 2022-09-12T19:03:47.989 DEBUG:boto:Headers: {} 2022-09-12T19:03:47.989 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:47.990 DEBUG:boto:Port: 8001 2022-09-12T19:03:47.990 DEBUG:boto:Params: {} 2022-09-12T19:03:47.991 DEBUG:boto:Token: None 2022-09-12T19:03:47.992 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:47 GMT /jdpiak-17/ 2022-09-12T19:03:47.992 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g= 2022-09-12T19:03:47.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S+3zNUKXmzLPEocB8mpkDsRLu4g=', 'Content-Length': '0'} 2022-09-12T19:03:47.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e241b18704f2836-00631f8293-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:47.996 DEBUG:boto:b'jdpiak-17dummy1000falsedummy2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:47.998 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:47.998 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:47.999 DEBUG:boto:path=/jdpiak-17/?prefix=dummy 2022-09-12T19:03:47.999 DEBUG:boto:auth_path=/jdpiak-17/?prefix=dummy 2022-09-12T19:03:48.000 DEBUG:boto:Method: GET 2022-09-12T19:03:48.000 DEBUG:boto:Path: /jdpiak-17/?prefix=dummy 2022-09-12T19:03:48.001 DEBUG:boto:Data: 2022-09-12T19:03:48.001 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.002 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:48.003 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.003 DEBUG:boto:Params: {} 2022-09-12T19:03:48.004 DEBUG:boto:Token: None 2022-09-12T19:03:48.004 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/ 2022-09-12T19:03:48.005 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4= 2022-09-12T19:03:48.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4=', 'Content-Length': '0'} 2022-09-12T19:03:48.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095fad6a3365a1b25-00631f8294-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.050 DEBUG:boto:b'jdpiak-17dummy1000falsedummy2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:48.051 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:48.052 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:48.052 DEBUG:boto:Method: HEAD 2022-09-12T19:03:48.053 DEBUG:boto:Path: /jdpiak-17/dummy 2022-09-12T19:03:48.053 DEBUG:boto:Data: 2022-09-12T19:03:48.054 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.055 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.055 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.055 DEBUG:boto:Params: {} 2022-09-12T19:03:48.056 DEBUG:boto:Token: None 2022-09-12T19:03:48.057 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/dummy 2022-09-12T19:03:48.057 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AUX1C2mvvn9WYSZK6M4BazVOTmA= 2022-09-12T19:03:48.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AUX1C2mvvn9WYSZK6M4BazVOTmA=', 'Content-Length': '0'} 2022-09-12T19:03:48.061 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000788bbc3d59b9a62b-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.061 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:48.062 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:48.063 DEBUG:boto:Method: HEAD 2022-09-12T19:03:48.063 DEBUG:boto:Path: /jdpiak-17/dummy 2022-09-12T19:03:48.064 DEBUG:boto:Data: 2022-09-12T19:03:48.064 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.065 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:48.065 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.066 DEBUG:boto:Params: {} 2022-09-12T19:03:48.067 DEBUG:boto:Token: None 2022-09-12T19:03:48.068 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/dummy 2022-09-12T19:03:48.068 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AUX1C2mvvn9WYSZK6M4BazVOTmA= 2022-09-12T19:03:48.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AUX1C2mvvn9WYSZK6M4BazVOTmA=', 'Content-Length': '0'} 2022-09-12T19:03:48.071 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000630fb5042c2d0d02-00631f8294-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:48.072 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:48.073 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:48.074 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.075 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.076 DEBUG:boto:Method: GET 2022-09-12T19:03:48.077 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.077 DEBUG:boto:Data: 2022-09-12T19:03:48.078 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.079 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.079 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.080 DEBUG:boto:Params: {} 2022-09-12T19:03:48.080 DEBUG:boto:Token: None 2022-09-12T19:03:48.081 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.082 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NAT5KTeVzVPX5v18OCPNdBJ7EJw= 2022-09-12T19:03:48.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NAT5KTeVzVPX5v18OCPNdBJ7EJw=', 'Content-Length': '0'} 2022-09-12T19:03:48.084 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000934078478e31bdc0-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.085 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:48.086 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:48.086 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.087 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.087 DEBUG:boto:Method: GET 2022-09-12T19:03:48.088 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.089 DEBUG:boto:Data: 2022-09-12T19:03:48.089 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.090 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:48.091 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.091 DEBUG:boto:Params: {} 2022-09-12T19:03:48.092 DEBUG:boto:Token: None 2022-09-12T19:03:48.092 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.093 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NAT5KTeVzVPX5v18OCPNdBJ7EJw= 2022-09-12T19:03:48.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NAT5KTeVzVPX5v18OCPNdBJ7EJw=', 'Content-Length': '0'} 2022-09-12T19:03:48.097 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e7b2fe8d12037b8e-00631f8294-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.098 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:48.099 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:48.099 DEBUG:boto:path=/jdpiak-17/?prefix=dummy 2022-09-12T19:03:48.100 DEBUG:boto:auth_path=/jdpiak-17/?prefix=dummy 2022-09-12T19:03:48.100 DEBUG:boto:Method: GET 2022-09-12T19:03:48.101 DEBUG:boto:Path: /jdpiak-17/?prefix=dummy 2022-09-12T19:03:48.101 DEBUG:boto:Data: 2022-09-12T19:03:48.102 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.103 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.104 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.104 DEBUG:boto:Params: {} 2022-09-12T19:03:48.105 DEBUG:boto:Token: None 2022-09-12T19:03:48.106 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/ 2022-09-12T19:03:48.106 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4= 2022-09-12T19:03:48.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4=', 'Content-Length': '0'} 2022-09-12T19:03:48.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021fd10f8d2d6b77d-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.110 DEBUG:boto:b'jdpiak-17dummy1000falsedummy2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:48.112 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:48.113 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:48.114 DEBUG:boto:path=/jdpiak-17/?prefix=dummy 2022-09-12T19:03:48.115 DEBUG:boto:auth_path=/jdpiak-17/?prefix=dummy 2022-09-12T19:03:48.116 DEBUG:boto:Method: GET 2022-09-12T19:03:48.117 DEBUG:boto:Path: /jdpiak-17/?prefix=dummy 2022-09-12T19:03:48.118 DEBUG:boto:Data: 2022-09-12T19:03:48.119 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.120 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:48.121 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.122 DEBUG:boto:Params: {} 2022-09-12T19:03:48.123 DEBUG:boto:Token: None 2022-09-12T19:03:48.124 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/ 2022-09-12T19:03:48.125 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4= 2022-09-12T19:03:48.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4=', 'Content-Length': '0'} 2022-09-12T19:03:48.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d29877926545f05a-00631f8294-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.131 DEBUG:boto:b'jdpiak-17dummy1000falsedummy2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:48.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:48.133 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:48.134 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:48.135 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.136 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.137 DEBUG:boto:Method: GET 2022-09-12T19:03:48.137 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.138 DEBUG:boto:Data: 2022-09-12T19:03:48.138 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.139 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.139 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.140 DEBUG:boto:Params: {} 2022-09-12T19:03:48.140 DEBUG:boto:Token: None 2022-09-12T19:03:48.141 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.142 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:25kldeCGG1OYU+WGj4LikCWF/2c= 2022-09-12T19:03:48.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:25kldeCGG1OYU+WGj4LikCWF/2c=', 'Content-Length': '0'} 2022-09-12T19:03:48.145 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082606e9e087d24f2-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.146 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:48.146 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:48.147 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.147 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.148 DEBUG:boto:Method: GET 2022-09-12T19:03:48.148 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.149 DEBUG:boto:Data: 2022-09-12T19:03:48.149 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.150 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:48.151 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.152 DEBUG:boto:Params: {} 2022-09-12T19:03:48.153 DEBUG:boto:Token: None 2022-09-12T19:03:48.153 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.154 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:25kldeCGG1OYU+WGj4LikCWF/2c= 2022-09-12T19:03:48.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:25kldeCGG1OYU+WGj4LikCWF/2c=', 'Content-Length': '0'} 2022-09-12T19:03:48.157 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009bb6f0869ff9074a-00631f8294-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.158 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:48.159 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:48.159 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.160 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.160 DEBUG:boto:Method: HEAD 2022-09-12T19:03:48.161 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.162 DEBUG:boto:Data: 2022-09-12T19:03:48.163 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.164 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.164 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.165 DEBUG:boto:Params: {} 2022-09-12T19:03:48.165 DEBUG:boto:Token: None 2022-09-12T19:03:48.166 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.166 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bO3QNxFSnhX+Xx3UU4hGzYTr6EY= 2022-09-12T19:03:48.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bO3QNxFSnhX+Xx3UU4hGzYTr6EY=', 'Content-Length': '0'} 2022-09-12T19:03:48.169 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d61ca8928ea5f0a-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.170 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:48.170 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:48.171 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.171 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.172 DEBUG:boto:Method: HEAD 2022-09-12T19:03:48.172 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.173 DEBUG:boto:Data: 2022-09-12T19:03:48.174 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.174 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:48.175 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.175 DEBUG:boto:Params: {} 2022-09-12T19:03:48.176 DEBUG:boto:Token: None 2022-09-12T19:03:48.177 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.177 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bO3QNxFSnhX+Xx3UU4hGzYTr6EY= 2022-09-12T19:03:48.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bO3QNxFSnhX+Xx3UU4hGzYTr6EY=', 'Content-Length': '0'} 2022-09-12T19:03:48.180 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093335e7256304721-00631f8294-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:48.181 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:48.181 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:48.182 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.183 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.183 DEBUG:boto:Method: GET 2022-09-12T19:03:48.184 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.184 DEBUG:boto:Data: 2022-09-12T19:03:48.185 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.186 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.186 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.187 DEBUG:boto:Params: {} 2022-09-12T19:03:48.187 DEBUG:boto:Token: None 2022-09-12T19:03:48.188 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.189 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:25kldeCGG1OYU+WGj4LikCWF/2c= 2022-09-12T19:03:48.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:25kldeCGG1OYU+WGj4LikCWF/2c=', 'Content-Length': '0'} 2022-09-12T19:03:48.192 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007211031e2d84fdea-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.193 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:48.194 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:48.194 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.195 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.196 DEBUG:boto:Method: GET 2022-09-12T19:03:48.196 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.197 DEBUG:boto:Data: 2022-09-12T19:03:48.197 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.198 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:48.199 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.199 DEBUG:boto:Params: {} 2022-09-12T19:03:48.200 DEBUG:boto:Token: None 2022-09-12T19:03:48.200 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.201 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:25kldeCGG1OYU+WGj4LikCWF/2c= 2022-09-12T19:03:48.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:25kldeCGG1OYU+WGj4LikCWF/2c=', 'Content-Length': '0'} 2022-09-12T19:03:48.204 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b50402fbe0b297b5-00631f8294-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.205 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:48.205 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:48.206 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:48.207 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:48.207 DEBUG:boto:Method: GET 2022-09-12T19:03:48.208 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:48.209 DEBUG:boto:Data: 2022-09-12T19:03:48.209 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.210 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.210 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.211 DEBUG:boto:Params: {} 2022-09-12T19:03:48.212 DEBUG:boto:Token: None 2022-09-12T19:03:48.212 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/ 2022-09-12T19:03:48.213 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4= 2022-09-12T19:03:48.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4=', 'Content-Length': '0'} 2022-09-12T19:03:48.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7ad97684cabbbd6-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.217 DEBUG:boto:b'jdpiak-17obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:48.218 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:48.219 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:48.220 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:48.220 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:48.221 DEBUG:boto:Method: GET 2022-09-12T19:03:48.221 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:48.222 DEBUG:boto:Data: 2022-09-12T19:03:48.222 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.223 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:48.223 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.224 DEBUG:boto:Params: {} 2022-09-12T19:03:48.225 DEBUG:boto:Token: None 2022-09-12T19:03:48.225 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/ 2022-09-12T19:03:48.226 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4= 2022-09-12T19:03:48.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4=', 'Content-Length': '0'} 2022-09-12T19:03:48.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a14e714f44aae07b-00631f8294-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.231 DEBUG:boto:b'jdpiak-17obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:48.233 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:48.234 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:48.235 DEBUG:boto:Method: HEAD 2022-09-12T19:03:48.236 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1 2022-09-12T19:03:48.236 DEBUG:boto:Data: 2022-09-12T19:03:48.237 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.238 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.239 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.240 DEBUG:boto:Params: {} 2022-09-12T19:03:48.241 DEBUG:boto:Token: None 2022-09-12T19:03:48.242 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone1 2022-09-12T19:03:48.243 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:at5fM0uLunXU+D632cSAlZUNH4w= 2022-09-12T19:03:48.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:at5fM0uLunXU+D632cSAlZUNH4w=', 'Content-Length': '0'} 2022-09-12T19:03:48.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a6c30990024b3fa-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.246 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:48.247 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:48.247 DEBUG:boto:Method: HEAD 2022-09-12T19:03:48.248 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1 2022-09-12T19:03:48.248 DEBUG:boto:Data: 2022-09-12T19:03:48.249 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.249 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:48.250 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.251 DEBUG:boto:Params: {} 2022-09-12T19:03:48.251 DEBUG:boto:Token: None 2022-09-12T19:03:48.252 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone1 2022-09-12T19:03:48.252 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:at5fM0uLunXU+D632cSAlZUNH4w= 2022-09-12T19:03:48.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:at5fM0uLunXU+D632cSAlZUNH4w=', 'Content-Length': '0'} 2022-09-12T19:03:48.255 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000769dc085031fb90-00631f8294-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:48.256 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:48.257 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:48.258 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.258 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.259 DEBUG:boto:Method: GET 2022-09-12T19:03:48.259 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.260 DEBUG:boto:Data: 2022-09-12T19:03:48.261 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.261 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.262 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.262 DEBUG:boto:Params: {} 2022-09-12T19:03:48.263 DEBUG:boto:Token: None 2022-09-12T19:03:48.264 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.264 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:25kldeCGG1OYU+WGj4LikCWF/2c= 2022-09-12T19:03:48.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:25kldeCGG1OYU+WGj4LikCWF/2c=', 'Content-Length': '0'} 2022-09-12T19:03:48.267 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d431cc6952027d24-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.268 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:48.268 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:48.269 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.269 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.270 DEBUG:boto:Method: GET 2022-09-12T19:03:48.270 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.271 DEBUG:boto:Data: 2022-09-12T19:03:48.271 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.272 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:48.272 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.273 DEBUG:boto:Params: {} 2022-09-12T19:03:48.274 DEBUG:boto:Token: None 2022-09-12T19:03:48.274 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:48.275 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:25kldeCGG1OYU+WGj4LikCWF/2c= 2022-09-12T19:03:48.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:25kldeCGG1OYU+WGj4LikCWF/2c=', 'Content-Length': '0'} 2022-09-12T19:03:48.278 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7798309d7311a57-00631f8294-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.318 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:48.319 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:48.319 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:48.320 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:48.321 DEBUG:boto:Method: GET 2022-09-12T19:03:48.321 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:48.322 DEBUG:boto:Data: 2022-09-12T19:03:48.322 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.323 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.323 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.324 DEBUG:boto:Params: {} 2022-09-12T19:03:48.325 DEBUG:boto:Token: None 2022-09-12T19:03:48.325 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/ 2022-09-12T19:03:48.326 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4= 2022-09-12T19:03:48.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4=', 'Content-Length': '0'} 2022-09-12T19:03:48.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010672881878c4c68-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.330 DEBUG:boto:b'jdpiak-17obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:48.331 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:48.331 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:48.332 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:48.332 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:48.333 DEBUG:boto:Method: GET 2022-09-12T19:03:48.334 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:48.334 DEBUG:boto:Data: 2022-09-12T19:03:48.335 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.335 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:48.336 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.336 DEBUG:boto:Params: {} 2022-09-12T19:03:48.337 DEBUG:boto:Token: None 2022-09-12T19:03:48.337 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/ 2022-09-12T19:03:48.338 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4= 2022-09-12T19:03:48.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4=', 'Content-Length': '0'} 2022-09-12T19:03:48.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a7a74ad311c9a97-00631f8294-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.343 DEBUG:boto:b'jdpiak-17obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:48.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:48.344 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:48.344 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:48.345 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.346 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.346 DEBUG:boto:Method: GET 2022-09-12T19:03:48.347 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.347 DEBUG:boto:Data: 2022-09-12T19:03:48.348 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.348 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.349 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.349 DEBUG:boto:Params: {} 2022-09-12T19:03:48.350 DEBUG:boto:Token: None 2022-09-12T19:03:48.351 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.351 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NBIbRz+Bt0dvcBIx7+cba+SVN1c= 2022-09-12T19:03:48.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NBIbRz+Bt0dvcBIx7+cba+SVN1c=', 'Content-Length': '0'} 2022-09-12T19:03:48.354 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ccddef4cc9b4805-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.355 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:48.355 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:48.356 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.356 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.357 DEBUG:boto:Method: GET 2022-09-12T19:03:48.357 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.358 DEBUG:boto:Data: 2022-09-12T19:03:48.358 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.359 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:48.359 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.360 DEBUG:boto:Params: {} 2022-09-12T19:03:48.360 DEBUG:boto:Token: None 2022-09-12T19:03:48.361 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.362 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NBIbRz+Bt0dvcBIx7+cba+SVN1c= 2022-09-12T19:03:48.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NBIbRz+Bt0dvcBIx7+cba+SVN1c=', 'Content-Length': '0'} 2022-09-12T19:03:48.364 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b54c62f9febd516b-00631f8294-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.365 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:48.366 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:48.366 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.367 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.367 DEBUG:boto:Method: HEAD 2022-09-12T19:03:48.368 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.368 DEBUG:boto:Data: 2022-09-12T19:03:48.369 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.369 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.370 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.370 DEBUG:boto:Params: {} 2022-09-12T19:03:48.371 DEBUG:boto:Token: None 2022-09-12T19:03:48.371 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.372 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DnbSMNqguqyUXjaoy0+MTn43xJc= 2022-09-12T19:03:48.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DnbSMNqguqyUXjaoy0+MTn43xJc=', 'Content-Length': '0'} 2022-09-12T19:03:48.375 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5f6b9b4ba5bc319-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.375 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:48.376 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:48.376 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.377 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.377 DEBUG:boto:Method: HEAD 2022-09-12T19:03:48.378 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.378 DEBUG:boto:Data: 2022-09-12T19:03:48.379 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.379 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:48.380 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.380 DEBUG:boto:Params: {} 2022-09-12T19:03:48.381 DEBUG:boto:Token: None 2022-09-12T19:03:48.381 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.382 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DnbSMNqguqyUXjaoy0+MTn43xJc= 2022-09-12T19:03:48.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DnbSMNqguqyUXjaoy0+MTn43xJc=', 'Content-Length': '0'} 2022-09-12T19:03:48.385 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003134c065b7b4439b-00631f8294-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:48.386 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:48.387 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:48.387 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.388 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.388 DEBUG:boto:Method: GET 2022-09-12T19:03:48.389 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.389 DEBUG:boto:Data: 2022-09-12T19:03:48.390 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.390 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.391 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.391 DEBUG:boto:Params: {} 2022-09-12T19:03:48.392 DEBUG:boto:Token: None 2022-09-12T19:03:48.393 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.393 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NBIbRz+Bt0dvcBIx7+cba+SVN1c= 2022-09-12T19:03:48.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NBIbRz+Bt0dvcBIx7+cba+SVN1c=', 'Content-Length': '0'} 2022-09-12T19:03:48.396 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c9a23da04e8402f-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.438 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:48.439 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:48.439 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.440 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.440 DEBUG:boto:Method: GET 2022-09-12T19:03:48.441 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.441 DEBUG:boto:Data: 2022-09-12T19:03:48.442 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.443 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:48.443 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.444 DEBUG:boto:Params: {} 2022-09-12T19:03:48.444 DEBUG:boto:Token: None 2022-09-12T19:03:48.445 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.446 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NBIbRz+Bt0dvcBIx7+cba+SVN1c= 2022-09-12T19:03:48.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NBIbRz+Bt0dvcBIx7+cba+SVN1c=', 'Content-Length': '0'} 2022-09-12T19:03:48.448 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d4c7251769b1c89-00631f8294-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.449 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:48.450 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:48.450 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:48.451 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:48.451 DEBUG:boto:Method: GET 2022-09-12T19:03:48.452 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:48.452 DEBUG:boto:Data: 2022-09-12T19:03:48.453 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.453 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.454 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.454 DEBUG:boto:Params: {} 2022-09-12T19:03:48.455 DEBUG:boto:Token: None 2022-09-12T19:03:48.455 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/ 2022-09-12T19:03:48.456 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4= 2022-09-12T19:03:48.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4=', 'Content-Length': '0'} 2022-09-12T19:03:48.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036e4b8b6e8a3c580-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.460 DEBUG:boto:b'jdpiak-17obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:48.461 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:48.462 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:48.462 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:48.463 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:48.463 DEBUG:boto:Method: GET 2022-09-12T19:03:48.464 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:48.464 DEBUG:boto:Data: 2022-09-12T19:03:48.465 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.465 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:48.466 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.466 DEBUG:boto:Params: {} 2022-09-12T19:03:48.467 DEBUG:boto:Token: None 2022-09-12T19:03:48.467 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/ 2022-09-12T19:03:48.468 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4= 2022-09-12T19:03:48.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4=', 'Content-Length': '0'} 2022-09-12T19:03:48.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdb2871230bd5936-00631f8294-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.472 DEBUG:boto:b'jdpiak-17obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:48.473 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:48.473 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:48.474 DEBUG:boto:Method: HEAD 2022-09-12T19:03:48.485 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2 2022-09-12T19:03:48.486 DEBUG:boto:Data: 2022-09-12T19:03:48.487 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.487 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.488 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.488 DEBUG:boto:Params: {} 2022-09-12T19:03:48.489 DEBUG:boto:Token: None 2022-09-12T19:03:48.489 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone2 2022-09-12T19:03:48.490 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9yRZabbEQsi+mkvH+WnAXf+sE7I= 2022-09-12T19:03:48.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9yRZabbEQsi+mkvH+WnAXf+sE7I=', 'Content-Length': '0'} 2022-09-12T19:03:48.494 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008440a9b1d1893501-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.495 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:48.495 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:48.496 DEBUG:boto:Method: HEAD 2022-09-12T19:03:48.496 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2 2022-09-12T19:03:48.497 DEBUG:boto:Data: 2022-09-12T19:03:48.497 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.498 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:48.498 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.499 DEBUG:boto:Params: {} 2022-09-12T19:03:48.499 DEBUG:boto:Token: None 2022-09-12T19:03:48.500 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone2 2022-09-12T19:03:48.500 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9yRZabbEQsi+mkvH+WnAXf+sE7I= 2022-09-12T19:03:48.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9yRZabbEQsi+mkvH+WnAXf+sE7I=', 'Content-Length': '0'} 2022-09-12T19:03:48.504 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac3bde65ec1f3957-00631f8294-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:48.505 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:48.505 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:48.506 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.506 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.507 DEBUG:boto:Method: GET 2022-09-12T19:03:48.508 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.509 DEBUG:boto:Data: 2022-09-12T19:03:48.510 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.511 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.512 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.513 DEBUG:boto:Params: {} 2022-09-12T19:03:48.514 DEBUG:boto:Token: None 2022-09-12T19:03:48.515 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.515 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NBIbRz+Bt0dvcBIx7+cba+SVN1c= 2022-09-12T19:03:48.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NBIbRz+Bt0dvcBIx7+cba+SVN1c=', 'Content-Length': '0'} 2022-09-12T19:03:48.518 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009aa64827cb020c9-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.519 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:48.519 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:48.520 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.520 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.521 DEBUG:boto:Method: GET 2022-09-12T19:03:48.521 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.522 DEBUG:boto:Data: 2022-09-12T19:03:48.523 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.523 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:48.524 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.524 DEBUG:boto:Params: {} 2022-09-12T19:03:48.525 DEBUG:boto:Token: None 2022-09-12T19:03:48.525 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:48.526 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NBIbRz+Bt0dvcBIx7+cba+SVN1c= 2022-09-12T19:03:48.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NBIbRz+Bt0dvcBIx7+cba+SVN1c=', 'Content-Length': '0'} 2022-09-12T19:03:48.529 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007daf603d12f3137-00631f8294-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.570 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:48.571 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:48.572 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:48.573 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:48.573 DEBUG:boto:Method: GET 2022-09-12T19:03:48.574 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:48.574 DEBUG:boto:Data: 2022-09-12T19:03:48.575 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.575 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.576 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.577 DEBUG:boto:Params: {} 2022-09-12T19:03:48.577 DEBUG:boto:Token: None 2022-09-12T19:03:48.578 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/ 2022-09-12T19:03:48.578 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4= 2022-09-12T19:03:48.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4=', 'Content-Length': '0'} 2022-09-12T19:03:48.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000627f24b188891d15-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.582 DEBUG:boto:b'jdpiak-17obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:48.583 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:48.584 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:48.584 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:48.585 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:48.585 DEBUG:boto:Method: GET 2022-09-12T19:03:48.586 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:48.586 DEBUG:boto:Data: 2022-09-12T19:03:48.587 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.587 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:48.588 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.589 DEBUG:boto:Params: {} 2022-09-12T19:03:48.589 DEBUG:boto:Token: None 2022-09-12T19:03:48.590 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/ 2022-09-12T19:03:48.591 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4= 2022-09-12T19:03:48.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4=', 'Content-Length': '0'} 2022-09-12T19:03:48.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000508b22b2fafaf9f4-00631f8294-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.596 DEBUG:boto:b'jdpiak-17obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:48.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:48.597 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:48.598 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:48.598 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.599 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.599 DEBUG:boto:Method: GET 2022-09-12T19:03:48.600 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.600 DEBUG:boto:Data: 2022-09-12T19:03:48.601 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.601 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.602 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.602 DEBUG:boto:Params: {} 2022-09-12T19:03:48.603 DEBUG:boto:Token: None 2022-09-12T19:03:48.603 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.604 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u1paHDEQRRxUBPKMmKtRJOYwBMQ= 2022-09-12T19:03:48.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u1paHDEQRRxUBPKMmKtRJOYwBMQ=', 'Content-Length': '0'} 2022-09-12T19:03:48.607 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079020b4b0660690b-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.650 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:48.651 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:48.652 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.653 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.654 DEBUG:boto:Method: GET 2022-09-12T19:03:48.655 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.656 DEBUG:boto:Data: 2022-09-12T19:03:48.657 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.658 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:48.659 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.660 DEBUG:boto:Params: {} 2022-09-12T19:03:48.661 DEBUG:boto:Token: None 2022-09-12T19:03:48.663 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.664 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u1paHDEQRRxUBPKMmKtRJOYwBMQ= 2022-09-12T19:03:48.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u1paHDEQRRxUBPKMmKtRJOYwBMQ=', 'Content-Length': '0'} 2022-09-12T19:03:48.668 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e5ea684369206bfb-00631f8294-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.670 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:48.671 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:48.672 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.673 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.673 DEBUG:boto:Method: HEAD 2022-09-12T19:03:48.674 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.678 DEBUG:boto:Data: 2022-09-12T19:03:48.679 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.679 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.680 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.680 DEBUG:boto:Params: {} 2022-09-12T19:03:48.681 DEBUG:boto:Token: None 2022-09-12T19:03:48.682 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.682 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FLUinXSql7nNQkWQZzD1lcGn0Vc= 2022-09-12T19:03:48.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FLUinXSql7nNQkWQZzD1lcGn0Vc=', 'Content-Length': '0'} 2022-09-12T19:03:48.685 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ccd463b6e46eda7d-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.686 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:48.687 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:48.687 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.688 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.688 DEBUG:boto:Method: HEAD 2022-09-12T19:03:48.689 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.689 DEBUG:boto:Data: 2022-09-12T19:03:48.690 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.691 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:48.691 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.692 DEBUG:boto:Params: {} 2022-09-12T19:03:48.692 DEBUG:boto:Token: None 2022-09-12T19:03:48.693 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.694 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FLUinXSql7nNQkWQZzD1lcGn0Vc= 2022-09-12T19:03:48.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FLUinXSql7nNQkWQZzD1lcGn0Vc=', 'Content-Length': '0'} 2022-09-12T19:03:48.697 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000db00cdb0b7817fe-00631f8294-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:48.698 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:48.699 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:48.699 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.700 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.700 DEBUG:boto:Method: GET 2022-09-12T19:03:48.701 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.702 DEBUG:boto:Data: 2022-09-12T19:03:48.702 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.703 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.703 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.704 DEBUG:boto:Params: {} 2022-09-12T19:03:48.704 DEBUG:boto:Token: None 2022-09-12T19:03:48.705 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.706 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u1paHDEQRRxUBPKMmKtRJOYwBMQ= 2022-09-12T19:03:48.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u1paHDEQRRxUBPKMmKtRJOYwBMQ=', 'Content-Length': '0'} 2022-09-12T19:03:48.709 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001bb41200a744155b-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.710 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:48.710 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:48.711 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.711 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.712 DEBUG:boto:Method: GET 2022-09-12T19:03:48.712 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.713 DEBUG:boto:Data: 2022-09-12T19:03:48.713 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.714 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:48.714 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.715 DEBUG:boto:Params: {} 2022-09-12T19:03:48.716 DEBUG:boto:Token: None 2022-09-12T19:03:48.716 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.717 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u1paHDEQRRxUBPKMmKtRJOYwBMQ= 2022-09-12T19:03:48.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u1paHDEQRRxUBPKMmKtRJOYwBMQ=', 'Content-Length': '0'} 2022-09-12T19:03:48.720 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a94d8cd99f9db87-00631f8294-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.721 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:48.722 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:48.722 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:48.723 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:48.724 DEBUG:boto:Method: GET 2022-09-12T19:03:48.724 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:48.725 DEBUG:boto:Data: 2022-09-12T19:03:48.725 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.726 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.727 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.727 DEBUG:boto:Params: {} 2022-09-12T19:03:48.728 DEBUG:boto:Token: None 2022-09-12T19:03:48.728 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/ 2022-09-12T19:03:48.729 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4= 2022-09-12T19:03:48.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4=', 'Content-Length': '0'} 2022-09-12T19:03:48.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063ea7f9de1182ac1-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.733 DEBUG:boto:b'jdpiak-17obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:48.734 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:48.734 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:48.735 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:48.735 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:48.736 DEBUG:boto:Method: GET 2022-09-12T19:03:48.737 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:48.737 DEBUG:boto:Data: 2022-09-12T19:03:48.738 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.738 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:48.739 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.739 DEBUG:boto:Params: {} 2022-09-12T19:03:48.741 DEBUG:boto:Token: None 2022-09-12T19:03:48.742 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/ 2022-09-12T19:03:48.743 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4= 2022-09-12T19:03:48.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4=', 'Content-Length': '0'} 2022-09-12T19:03:48.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ece4f91d25ad9e82-00631f8294-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.749 DEBUG:boto:b'jdpiak-17obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:48.749 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:48.750 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:48.751 DEBUG:boto:Method: HEAD 2022-09-12T19:03:48.751 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3 2022-09-12T19:03:48.752 DEBUG:boto:Data: 2022-09-12T19:03:48.752 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.753 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.753 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.754 DEBUG:boto:Params: {} 2022-09-12T19:03:48.754 DEBUG:boto:Token: None 2022-09-12T19:03:48.755 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone3 2022-09-12T19:03:48.756 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1vO7p0idwmZXe0g2W4+TbORm2E8= 2022-09-12T19:03:48.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1vO7p0idwmZXe0g2W4+TbORm2E8=', 'Content-Length': '0'} 2022-09-12T19:03:48.759 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef8c920a2d611cbd-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.760 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:48.760 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:48.761 DEBUG:boto:Method: HEAD 2022-09-12T19:03:48.761 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3 2022-09-12T19:03:48.762 DEBUG:boto:Data: 2022-09-12T19:03:48.762 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.763 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:48.764 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.764 DEBUG:boto:Params: {} 2022-09-12T19:03:48.765 DEBUG:boto:Token: None 2022-09-12T19:03:48.765 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone3 2022-09-12T19:03:48.766 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1vO7p0idwmZXe0g2W4+TbORm2E8= 2022-09-12T19:03:48.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1vO7p0idwmZXe0g2W4+TbORm2E8=', 'Content-Length': '0'} 2022-09-12T19:03:48.770 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5be2db2ba5e444b-00631f8294-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:48.771 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:48.772 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:48.772 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.773 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.773 DEBUG:boto:Method: GET 2022-09-12T19:03:48.774 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.775 DEBUG:boto:Data: 2022-09-12T19:03:48.775 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.776 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.776 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.777 DEBUG:boto:Params: {} 2022-09-12T19:03:48.777 DEBUG:boto:Token: None 2022-09-12T19:03:48.778 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.779 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u1paHDEQRRxUBPKMmKtRJOYwBMQ= 2022-09-12T19:03:48.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u1paHDEQRRxUBPKMmKtRJOYwBMQ=', 'Content-Length': '0'} 2022-09-12T19:03:48.781 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b1b8595cb5277f1-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.782 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:48.783 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:48.783 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.784 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.784 DEBUG:boto:Method: GET 2022-09-12T19:03:48.785 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.785 DEBUG:boto:Data: 2022-09-12T19:03:48.786 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.787 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:48.787 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.788 DEBUG:boto:Params: {} 2022-09-12T19:03:48.789 DEBUG:boto:Token: None 2022-09-12T19:03:48.790 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:48.790 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u1paHDEQRRxUBPKMmKtRJOYwBMQ= 2022-09-12T19:03:48.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u1paHDEQRRxUBPKMmKtRJOYwBMQ=', 'Content-Length': '0'} 2022-09-12T19:03:48.793 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004768570788b34e82-00631f8294-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.794 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:48.795 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:48.795 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:48.796 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:48.796 DEBUG:boto:Method: GET 2022-09-12T19:03:48.797 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:48.798 DEBUG:boto:Data: 2022-09-12T19:03:48.799 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.800 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.801 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.801 DEBUG:boto:Params: {} 2022-09-12T19:03:48.802 DEBUG:boto:Token: None 2022-09-12T19:03:48.802 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/ 2022-09-12T19:03:48.803 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4= 2022-09-12T19:03:48.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4=', 'Content-Length': '0'} 2022-09-12T19:03:48.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017ecedb6a821b617-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.807 DEBUG:boto:b'jdpiak-17obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:48.808 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:48.809 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:48.809 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:48.810 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:48.810 DEBUG:boto:Method: GET 2022-09-12T19:03:48.811 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:48.811 DEBUG:boto:Data: 2022-09-12T19:03:48.812 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.812 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:48.813 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.813 DEBUG:boto:Params: {} 2022-09-12T19:03:48.814 DEBUG:boto:Token: None 2022-09-12T19:03:48.815 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/ 2022-09-12T19:03:48.815 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4= 2022-09-12T19:03:48.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4=', 'Content-Length': '0'} 2022-09-12T19:03:48.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ced4609e76f14dd-00631f8294-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.820 DEBUG:boto:b'jdpiak-17obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:48.820 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-17 zones={test-zone3, test-zone1} 2022-09-12T19:03:48.821 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-17 zones={test-zone3, test-zone2} 2022-09-12T19:03:48.821 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:48.822 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:48.822 DEBUG:boto:Method: HEAD 2022-09-12T19:03:48.823 DEBUG:boto:Path: /jdpiak-17/ 2022-09-12T19:03:48.824 DEBUG:boto:Data: 2022-09-12T19:03:48.824 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.825 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.825 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.826 DEBUG:boto:Params: {} 2022-09-12T19:03:48.826 DEBUG:boto:Token: None 2022-09-12T19:03:48.827 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/ 2022-09-12T19:03:48.827 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:16AhNDwQZnNQF9F8KALdHB1OK4A= 2022-09-12T19:03:48.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:16AhNDwQZnNQF9F8KALdHB1OK4A=', 'Content-Length': '0'} 2022-09-12T19:03:48.831 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b20a7c8e54c7b081-00631f8294-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.831 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:48.832 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:48.832 DEBUG:boto:Method: HEAD 2022-09-12T19:03:48.833 DEBUG:boto:Path: /jdpiak-17/ 2022-09-12T19:03:48.833 DEBUG:boto:Data: 2022-09-12T19:03:48.834 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.835 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:48.835 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.836 DEBUG:boto:Params: {} 2022-09-12T19:03:48.836 DEBUG:boto:Token: None 2022-09-12T19:03:48.837 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/ 2022-09-12T19:03:48.837 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:16AhNDwQZnNQF9F8KALdHB1OK4A= 2022-09-12T19:03:48.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:16AhNDwQZnNQF9F8KALdHB1OK4A=', 'Content-Length': '0'} 2022-09-12T19:03:48.841 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e85a25254b146c29-00631f8294-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.841 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:03:48.842 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:48.843 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:48.843 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:48.844 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:48.844 DEBUG:boto:Method: GET 2022-09-12T19:03:48.845 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:48.845 DEBUG:boto:Data: 2022-09-12T19:03:48.846 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.847 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.847 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.848 DEBUG:boto:Params: {} 2022-09-12T19:03:48.848 DEBUG:boto:Token: None 2022-09-12T19:03:48.849 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/?versions 2022-09-12T19:03:48.850 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:45i7KnDTsxiw/j47Faj4IJFO/eQ= 2022-09-12T19:03:48.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:45i7KnDTsxiw/j47Faj4IJFO/eQ=', 'Content-Length': '0'} 2022-09-12T19:03:48.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba21c0479328f747-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.854 DEBUG:boto:b'jdpiak-17999falsedummybSzfRw-Zsh7lYv-ClgdhoG098KzYg.Strue2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpktrue2022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone20f1rsOoGVVMRuyy11YfWbutRvo943lCtrue2022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3V6Ue4vhVY0WQ9FArGixLjY-hmND5K20true2022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:48.855 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:03:48.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:03:48.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:03:48.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:03:48.857 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:03:48.857 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:48.858 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:48.858 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:48.859 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:48.859 DEBUG:boto:Method: GET 2022-09-12T19:03:48.860 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:48.860 DEBUG:boto:Data: 2022-09-12T19:03:48.861 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.861 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:48.862 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.862 DEBUG:boto:Params: {} 2022-09-12T19:03:48.863 DEBUG:boto:Token: None 2022-09-12T19:03:48.864 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/?versions 2022-09-12T19:03:48.864 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:45i7KnDTsxiw/j47Faj4IJFO/eQ= 2022-09-12T19:03:48.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:45i7KnDTsxiw/j47Faj4IJFO/eQ=', 'Content-Length': '0'} 2022-09-12T19:03:48.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd23819917987631-00631f8294-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.869 DEBUG:boto:b'jdpiak-17999falsedummybSzfRw-Zsh7lYv-ClgdhoG098KzYg.Strue2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpktrue2022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone20f1rsOoGVVMRuyy11YfWbutRvo943lCtrue2022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3V6Ue4vhVY0WQ9FArGixLjY-hmND5K20true2022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:48.869 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:03:48.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:03:48.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:03:48.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:03:48.871 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:48.872 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:48.872 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:48.873 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:48.873 DEBUG:boto:Method: GET 2022-09-12T19:03:48.874 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:48.875 DEBUG:boto:Data: 2022-09-12T19:03:48.875 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.876 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.876 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.877 DEBUG:boto:Params: {} 2022-09-12T19:03:48.877 DEBUG:boto:Token: None 2022-09-12T19:03:48.878 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/?versions 2022-09-12T19:03:48.878 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:45i7KnDTsxiw/j47Faj4IJFO/eQ= 2022-09-12T19:03:48.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:45i7KnDTsxiw/j47Faj4IJFO/eQ=', 'Content-Length': '0'} 2022-09-12T19:03:48.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab12aeb302ced568-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.882 DEBUG:boto:b'jdpiak-17999falsedummybSzfRw-Zsh7lYv-ClgdhoG098KzYg.Strue2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpktrue2022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone20f1rsOoGVVMRuyy11YfWbutRvo943lCtrue2022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3V6Ue4vhVY0WQ9FArGixLjY-hmND5K20true2022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:48.883 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:48.884 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:48.884 DEBUG:boto:path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:48.885 DEBUG:boto:auth_path=/jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:48.885 DEBUG:boto:Method: GET 2022-09-12T19:03:48.886 DEBUG:boto:Path: /jdpiak-17/?versions&max-keys=999 2022-09-12T19:03:48.886 DEBUG:boto:Data: 2022-09-12T19:03:48.887 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.887 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:48.888 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.888 DEBUG:boto:Params: {} 2022-09-12T19:03:48.889 DEBUG:boto:Token: None 2022-09-12T19:03:48.890 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/?versions 2022-09-12T19:03:48.890 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:45i7KnDTsxiw/j47Faj4IJFO/eQ= 2022-09-12T19:03:48.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:45i7KnDTsxiw/j47Faj4IJFO/eQ=', 'Content-Length': '0'} 2022-09-12T19:03:48.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4d05814a14f5e34-00631f8294-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.894 DEBUG:boto:b'jdpiak-17999falsedummybSzfRw-Zsh7lYv-ClgdhoG098KzYg.Strue2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpktrue2022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone20f1rsOoGVVMRuyy11YfWbutRvo943lCtrue2022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3V6Ue4vhVY0WQ9FArGixLjY-hmND5K20true2022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:48.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:48.896 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:48.896 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:48.897 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.897 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.898 DEBUG:boto:Method: GET 2022-09-12T19:03:48.898 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.899 DEBUG:boto:Data: 2022-09-12T19:03:48.899 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.900 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.900 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.901 DEBUG:boto:Params: {} 2022-09-12T19:03:48.901 DEBUG:boto:Token: None 2022-09-12T19:03:48.902 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.903 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NAT5KTeVzVPX5v18OCPNdBJ7EJw= 2022-09-12T19:03:48.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NAT5KTeVzVPX5v18OCPNdBJ7EJw=', 'Content-Length': '0'} 2022-09-12T19:03:48.905 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b33eb75d67f89e93-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.906 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:48.907 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:48.907 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.908 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.908 DEBUG:boto:Method: GET 2022-09-12T19:03:48.909 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.910 DEBUG:boto:Data: 2022-09-12T19:03:48.910 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.911 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:48.911 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.912 DEBUG:boto:Params: {} 2022-09-12T19:03:48.913 DEBUG:boto:Token: None 2022-09-12T19:03:48.913 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.914 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NAT5KTeVzVPX5v18OCPNdBJ7EJw= 2022-09-12T19:03:48.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NAT5KTeVzVPX5v18OCPNdBJ7EJw=', 'Content-Length': '0'} 2022-09-12T19:03:48.917 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ff4ea0f6091c82c5-00631f8294-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.918 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:48.918 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:48.919 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.919 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.920 DEBUG:boto:Method: HEAD 2022-09-12T19:03:48.920 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.921 DEBUG:boto:Data: 2022-09-12T19:03:48.921 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.922 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.922 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.923 DEBUG:boto:Params: {} 2022-09-12T19:03:48.923 DEBUG:boto:Token: None 2022-09-12T19:03:48.924 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.925 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DcQYGgNP8gHc2D/HkEMfdNxLviM= 2022-09-12T19:03:48.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DcQYGgNP8gHc2D/HkEMfdNxLviM=', 'Content-Length': '0'} 2022-09-12T19:03:48.927 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000694640a06162841f-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.928 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:48.929 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:48.929 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.930 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.930 DEBUG:boto:Method: HEAD 2022-09-12T19:03:48.931 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.931 DEBUG:boto:Data: 2022-09-12T19:03:48.932 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.932 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:48.933 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.933 DEBUG:boto:Params: {} 2022-09-12T19:03:48.934 DEBUG:boto:Token: None 2022-09-12T19:03:48.935 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.935 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DcQYGgNP8gHc2D/HkEMfdNxLviM= 2022-09-12T19:03:48.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DcQYGgNP8gHc2D/HkEMfdNxLviM=', 'Content-Length': '0'} 2022-09-12T19:03:48.938 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000633d85532b03b107-00631f8294-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:48.940 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:48.940 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:48.941 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.942 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.942 DEBUG:boto:Method: GET 2022-09-12T19:03:48.943 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.944 DEBUG:boto:Data: 2022-09-12T19:03:48.944 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.945 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.946 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.946 DEBUG:boto:Params: {} 2022-09-12T19:03:48.947 DEBUG:boto:Token: None 2022-09-12T19:03:48.948 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.949 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NAT5KTeVzVPX5v18OCPNdBJ7EJw= 2022-09-12T19:03:48.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NAT5KTeVzVPX5v18OCPNdBJ7EJw=', 'Content-Length': '0'} 2022-09-12T19:03:48.952 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002c357b9f25b8c99a-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.953 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:48.953 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:48.954 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.955 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.955 DEBUG:boto:Method: GET 2022-09-12T19:03:48.956 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.957 DEBUG:boto:Data: 2022-09-12T19:03:48.957 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.958 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:48.959 DEBUG:boto:Port: 8000 2022-09-12T19:03:48.959 DEBUG:boto:Params: {} 2022-09-12T19:03:48.960 DEBUG:boto:Token: None 2022-09-12T19:03:48.961 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:48.961 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NAT5KTeVzVPX5v18OCPNdBJ7EJw= 2022-09-12T19:03:48.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NAT5KTeVzVPX5v18OCPNdBJ7EJw=', 'Content-Length': '0'} 2022-09-12T19:03:48.964 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d441b7a5fbfc7231-00631f8294-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:48.965 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:48.966 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:48.967 DEBUG:boto:path=/jdpiak-17/?prefix=dummy 2022-09-12T19:03:48.967 DEBUG:boto:auth_path=/jdpiak-17/?prefix=dummy 2022-09-12T19:03:48.968 DEBUG:boto:Method: GET 2022-09-12T19:03:48.969 DEBUG:boto:Path: /jdpiak-17/?prefix=dummy 2022-09-12T19:03:48.969 DEBUG:boto:Data: 2022-09-12T19:03:48.970 DEBUG:boto:Headers: {} 2022-09-12T19:03:48.971 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:48.971 DEBUG:boto:Port: 8001 2022-09-12T19:03:48.972 DEBUG:boto:Params: {} 2022-09-12T19:03:48.973 DEBUG:boto:Token: None 2022-09-12T19:03:48.974 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:48 GMT /jdpiak-17/ 2022-09-12T19:03:48.975 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4= 2022-09-12T19:03:48.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QsOg8lVeumADph1eBugBKP2DfW4=', 'Content-Length': '0'} 2022-09-12T19:03:48.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfb7ddda01afc281-00631f8294-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.024 DEBUG:boto:b'jdpiak-17dummy1000falsedummy2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:49.026 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:49.026 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:49.027 DEBUG:boto:path=/jdpiak-17/?prefix=dummy 2022-09-12T19:03:49.027 DEBUG:boto:auth_path=/jdpiak-17/?prefix=dummy 2022-09-12T19:03:49.028 DEBUG:boto:Method: GET 2022-09-12T19:03:49.029 DEBUG:boto:Path: /jdpiak-17/?prefix=dummy 2022-09-12T19:03:49.030 DEBUG:boto:Data: 2022-09-12T19:03:49.031 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.031 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:49.032 DEBUG:boto:Port: 8000 2022-09-12T19:03:49.033 DEBUG:boto:Params: {} 2022-09-12T19:03:49.034 DEBUG:boto:Token: None 2022-09-12T19:03:49.035 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/ 2022-09-12T19:03:49.036 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ= 2022-09-12T19:03:49.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ=', 'Content-Length': '0'} 2022-09-12T19:03:49.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000416262f7bf168bb4-00631f8295-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.041 DEBUG:boto:b'jdpiak-17dummy1000falsedummy2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:49.042 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:49.043 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:49.044 DEBUG:boto:Method: HEAD 2022-09-12T19:03:49.045 DEBUG:boto:Path: /jdpiak-17/dummy 2022-09-12T19:03:49.045 DEBUG:boto:Data: 2022-09-12T19:03:49.046 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.047 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:49.047 DEBUG:boto:Port: 8001 2022-09-12T19:03:49.048 DEBUG:boto:Params: {} 2022-09-12T19:03:49.048 DEBUG:boto:Token: None 2022-09-12T19:03:49.049 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/dummy 2022-09-12T19:03:49.050 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nzb1OS2MecNxgaKIaRpEHR1r2Ws= 2022-09-12T19:03:49.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nzb1OS2MecNxgaKIaRpEHR1r2Ws=', 'Content-Length': '0'} 2022-09-12T19:03:49.053 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c3470d6bce6b3c5d-00631f8295-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.054 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:49.054 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:49.055 DEBUG:boto:Method: HEAD 2022-09-12T19:03:49.056 DEBUG:boto:Path: /jdpiak-17/dummy 2022-09-12T19:03:49.056 DEBUG:boto:Data: 2022-09-12T19:03:49.057 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.057 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:49.058 DEBUG:boto:Port: 8000 2022-09-12T19:03:49.058 DEBUG:boto:Params: {} 2022-09-12T19:03:49.059 DEBUG:boto:Token: None 2022-09-12T19:03:49.060 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/dummy 2022-09-12T19:03:49.060 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nzb1OS2MecNxgaKIaRpEHR1r2Ws= 2022-09-12T19:03:49.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nzb1OS2MecNxgaKIaRpEHR1r2Ws=', 'Content-Length': '0'} 2022-09-12T19:03:49.063 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000009c9df54a1a3179d-00631f8295-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:49.065 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:49.066 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:49.066 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:49.067 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:49.068 DEBUG:boto:Method: GET 2022-09-12T19:03:49.068 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:49.069 DEBUG:boto:Data: 2022-09-12T19:03:49.070 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.070 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:49.071 DEBUG:boto:Port: 8001 2022-09-12T19:03:49.071 DEBUG:boto:Params: {} 2022-09-12T19:03:49.072 DEBUG:boto:Token: None 2022-09-12T19:03:49.073 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:49.073 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1CK7Gl//fNzaqVX6vzCdx35Aszg= 2022-09-12T19:03:49.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1CK7Gl//fNzaqVX6vzCdx35Aszg=', 'Content-Length': '0'} 2022-09-12T19:03:49.076 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fbac14d94e6d27a8-00631f8295-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.077 DEBUG:boto:path=/jdpiak-17/dummy 2022-09-12T19:03:49.078 DEBUG:boto:auth_path=/jdpiak-17/dummy 2022-09-12T19:03:49.078 DEBUG:boto:path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:49.079 DEBUG:boto:auth_path=/jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:49.080 DEBUG:boto:Method: GET 2022-09-12T19:03:49.081 DEBUG:boto:Path: /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:49.081 DEBUG:boto:Data: 2022-09-12T19:03:49.082 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.082 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:49.083 DEBUG:boto:Port: 8000 2022-09-12T19:03:49.083 DEBUG:boto:Params: {} 2022-09-12T19:03:49.084 DEBUG:boto:Token: None 2022-09-12T19:03:49.085 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/dummy?versionId=bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S 2022-09-12T19:03:49.086 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1CK7Gl//fNzaqVX6vzCdx35Aszg= 2022-09-12T19:03:49.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1CK7Gl//fNzaqVX6vzCdx35Aszg=', 'Content-Length': '0'} 2022-09-12T19:03:49.088 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:07 GMT'), ('x-amz-version-id', 'bSzfRw-Zsh7lYv-ClgdhoG098KzYg.S'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000934462e990647698-00631f8295-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.089 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:49.090 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:49.091 DEBUG:boto:path=/jdpiak-17/?prefix=dummy 2022-09-12T19:03:49.092 DEBUG:boto:auth_path=/jdpiak-17/?prefix=dummy 2022-09-12T19:03:49.093 DEBUG:boto:Method: GET 2022-09-12T19:03:49.093 DEBUG:boto:Path: /jdpiak-17/?prefix=dummy 2022-09-12T19:03:49.094 DEBUG:boto:Data: 2022-09-12T19:03:49.094 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.095 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:49.096 DEBUG:boto:Port: 8001 2022-09-12T19:03:49.096 DEBUG:boto:Params: {} 2022-09-12T19:03:49.097 DEBUG:boto:Token: None 2022-09-12T19:03:49.098 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/ 2022-09-12T19:03:49.098 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ= 2022-09-12T19:03:49.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ=', 'Content-Length': '0'} 2022-09-12T19:03:49.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071b82332193e7f57-00631f8295-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.103 DEBUG:boto:b'jdpiak-17dummy1000falsedummy2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:49.104 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:49.105 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:49.105 DEBUG:boto:path=/jdpiak-17/?prefix=dummy 2022-09-12T19:03:49.106 DEBUG:boto:auth_path=/jdpiak-17/?prefix=dummy 2022-09-12T19:03:49.107 DEBUG:boto:Method: GET 2022-09-12T19:03:49.108 DEBUG:boto:Path: /jdpiak-17/?prefix=dummy 2022-09-12T19:03:49.108 DEBUG:boto:Data: 2022-09-12T19:03:49.109 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.110 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:49.110 DEBUG:boto:Port: 8000 2022-09-12T19:03:49.111 DEBUG:boto:Params: {} 2022-09-12T19:03:49.112 DEBUG:boto:Token: None 2022-09-12T19:03:49.112 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/ 2022-09-12T19:03:49.113 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ= 2022-09-12T19:03:49.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ=', 'Content-Length': '0'} 2022-09-12T19:03:49.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d03fd0835a28f69c-00631f8295-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.117 DEBUG:boto:b'jdpiak-17dummy1000falsedummy2022-09-12T18:58:07.423Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:49.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:49.119 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:49.120 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:49.120 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.121 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.122 DEBUG:boto:Method: GET 2022-09-12T19:03:49.123 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.123 DEBUG:boto:Data: 2022-09-12T19:03:49.124 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.124 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:49.125 DEBUG:boto:Port: 8001 2022-09-12T19:03:49.125 DEBUG:boto:Params: {} 2022-09-12T19:03:49.126 DEBUG:boto:Token: None 2022-09-12T19:03:49.126 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.127 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wCF/rSrsuvTvobREIWTc/EcUfBE= 2022-09-12T19:03:49.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wCF/rSrsuvTvobREIWTc/EcUfBE=', 'Content-Length': '0'} 2022-09-12T19:03:49.130 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d0a557d8ebe4afd-00631f8295-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.131 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:49.131 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:49.132 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.132 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.133 DEBUG:boto:Method: GET 2022-09-12T19:03:49.134 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.134 DEBUG:boto:Data: 2022-09-12T19:03:49.135 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.135 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:49.136 DEBUG:boto:Port: 8000 2022-09-12T19:03:49.136 DEBUG:boto:Params: {} 2022-09-12T19:03:49.137 DEBUG:boto:Token: None 2022-09-12T19:03:49.137 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.138 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wCF/rSrsuvTvobREIWTc/EcUfBE= 2022-09-12T19:03:49.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wCF/rSrsuvTvobREIWTc/EcUfBE=', 'Content-Length': '0'} 2022-09-12T19:03:49.141 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b230cc2bfa984caf-00631f8295-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.141 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:49.142 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:49.143 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.144 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.144 DEBUG:boto:Method: HEAD 2022-09-12T19:03:49.145 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.145 DEBUG:boto:Data: 2022-09-12T19:03:49.146 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.146 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:49.147 DEBUG:boto:Port: 8001 2022-09-12T19:03:49.147 DEBUG:boto:Params: {} 2022-09-12T19:03:49.148 DEBUG:boto:Token: None 2022-09-12T19:03:49.148 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.149 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JCHqvStKo7vXOAiC5qDoXuQV7AA= 2022-09-12T19:03:49.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JCHqvStKo7vXOAiC5qDoXuQV7AA=', 'Content-Length': '0'} 2022-09-12T19:03:49.152 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ede9ccb5695d6a3-00631f8295-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.152 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:49.153 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:49.154 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.155 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.155 DEBUG:boto:Method: HEAD 2022-09-12T19:03:49.156 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.156 DEBUG:boto:Data: 2022-09-12T19:03:49.157 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.157 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:49.158 DEBUG:boto:Port: 8000 2022-09-12T19:03:49.158 DEBUG:boto:Params: {} 2022-09-12T19:03:49.159 DEBUG:boto:Token: None 2022-09-12T19:03:49.159 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.160 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JCHqvStKo7vXOAiC5qDoXuQV7AA= 2022-09-12T19:03:49.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JCHqvStKo7vXOAiC5qDoXuQV7AA=', 'Content-Length': '0'} 2022-09-12T19:03:49.162 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045ade0ff1e66d4c7-00631f8295-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:49.164 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:49.164 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:49.165 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.165 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.166 DEBUG:boto:Method: GET 2022-09-12T19:03:49.166 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.167 DEBUG:boto:Data: 2022-09-12T19:03:49.167 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.168 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:49.168 DEBUG:boto:Port: 8001 2022-09-12T19:03:49.169 DEBUG:boto:Params: {} 2022-09-12T19:03:49.169 DEBUG:boto:Token: None 2022-09-12T19:03:49.170 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.171 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wCF/rSrsuvTvobREIWTc/EcUfBE= 2022-09-12T19:03:49.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wCF/rSrsuvTvobREIWTc/EcUfBE=', 'Content-Length': '0'} 2022-09-12T19:03:49.173 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000092f00a47cea83313-00631f8295-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.218 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:49.219 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:49.219 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.220 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.220 DEBUG:boto:Method: GET 2022-09-12T19:03:49.221 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.221 DEBUG:boto:Data: 2022-09-12T19:03:49.222 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.223 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:49.223 DEBUG:boto:Port: 8000 2022-09-12T19:03:49.224 DEBUG:boto:Params: {} 2022-09-12T19:03:49.224 DEBUG:boto:Token: None 2022-09-12T19:03:49.225 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.226 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wCF/rSrsuvTvobREIWTc/EcUfBE= 2022-09-12T19:03:49.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wCF/rSrsuvTvobREIWTc/EcUfBE=', 'Content-Length': '0'} 2022-09-12T19:03:49.229 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c7ebfa698b63a27-00631f8295-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.229 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:49.230 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:49.230 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:49.231 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:49.232 DEBUG:boto:Method: GET 2022-09-12T19:03:49.232 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:49.233 DEBUG:boto:Data: 2022-09-12T19:03:49.233 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.234 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:49.234 DEBUG:boto:Port: 8001 2022-09-12T19:03:49.235 DEBUG:boto:Params: {} 2022-09-12T19:03:49.236 DEBUG:boto:Token: None 2022-09-12T19:03:49.236 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/ 2022-09-12T19:03:49.237 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ= 2022-09-12T19:03:49.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ=', 'Content-Length': '0'} 2022-09-12T19:03:49.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df41d63b39abd531-00631f8295-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.240 DEBUG:boto:b'jdpiak-17obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:49.241 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:49.242 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:49.242 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:49.243 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:49.244 DEBUG:boto:Method: GET 2022-09-12T19:03:49.244 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:49.245 DEBUG:boto:Data: 2022-09-12T19:03:49.246 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.246 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:49.247 DEBUG:boto:Port: 8000 2022-09-12T19:03:49.247 DEBUG:boto:Params: {} 2022-09-12T19:03:49.248 DEBUG:boto:Token: None 2022-09-12T19:03:49.248 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/ 2022-09-12T19:03:49.249 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ= 2022-09-12T19:03:49.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ=', 'Content-Length': '0'} 2022-09-12T19:03:49.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecefd0c4b52aab78-00631f8295-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.253 DEBUG:boto:b'jdpiak-17obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:49.253 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:49.254 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:49.254 DEBUG:boto:Method: HEAD 2022-09-12T19:03:49.255 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1 2022-09-12T19:03:49.255 DEBUG:boto:Data: 2022-09-12T19:03:49.256 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.256 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:49.257 DEBUG:boto:Port: 8001 2022-09-12T19:03:49.257 DEBUG:boto:Params: {} 2022-09-12T19:03:49.258 DEBUG:boto:Token: None 2022-09-12T19:03:49.258 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone1 2022-09-12T19:03:49.259 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A84YDaYisvGqNJyBifxdw/n0vJQ= 2022-09-12T19:03:49.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A84YDaYisvGqNJyBifxdw/n0vJQ=', 'Content-Length': '0'} 2022-09-12T19:03:49.262 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008562e30028c0f25c-00631f8295-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.263 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:49.263 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:49.264 DEBUG:boto:Method: HEAD 2022-09-12T19:03:49.264 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1 2022-09-12T19:03:49.265 DEBUG:boto:Data: 2022-09-12T19:03:49.265 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.266 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:49.266 DEBUG:boto:Port: 8000 2022-09-12T19:03:49.267 DEBUG:boto:Params: {} 2022-09-12T19:03:49.267 DEBUG:boto:Token: None 2022-09-12T19:03:49.268 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone1 2022-09-12T19:03:49.268 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A84YDaYisvGqNJyBifxdw/n0vJQ= 2022-09-12T19:03:49.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A84YDaYisvGqNJyBifxdw/n0vJQ=', 'Content-Length': '0'} 2022-09-12T19:03:49.271 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ecb21c70900d8ef1-00631f8295-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:49.272 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:49.273 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:49.273 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.274 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.275 DEBUG:boto:Method: GET 2022-09-12T19:03:49.275 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.276 DEBUG:boto:Data: 2022-09-12T19:03:49.276 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.277 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:49.277 DEBUG:boto:Port: 8001 2022-09-12T19:03:49.278 DEBUG:boto:Params: {} 2022-09-12T19:03:49.278 DEBUG:boto:Token: None 2022-09-12T19:03:49.279 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.279 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wCF/rSrsuvTvobREIWTc/EcUfBE= 2022-09-12T19:03:49.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wCF/rSrsuvTvobREIWTc/EcUfBE=', 'Content-Length': '0'} 2022-09-12T19:03:49.282 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000914ae71a53ebbdbc-00631f8295-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.283 DEBUG:boto:path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:49.283 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1 2022-09-12T19:03:49.284 DEBUG:boto:path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.284 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.285 DEBUG:boto:Method: GET 2022-09-12T19:03:49.285 DEBUG:boto:Path: /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.286 DEBUG:boto:Data: 2022-09-12T19:03:49.286 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.287 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:49.287 DEBUG:boto:Port: 8000 2022-09-12T19:03:49.288 DEBUG:boto:Params: {} 2022-09-12T19:03:49.289 DEBUG:boto:Token: None 2022-09-12T19:03:49.289 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone1?versionId=Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk 2022-09-12T19:03:49.290 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wCF/rSrsuvTvobREIWTc/EcUfBE= 2022-09-12T19:03:49.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wCF/rSrsuvTvobREIWTc/EcUfBE=', 'Content-Length': '0'} 2022-09-12T19:03:49.292 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'Hi3Qnb.TdtDOTSwq0Kn02-Bu71kHRpk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087a5d6af92ce33e9-00631f8295-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.334 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:49.335 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:49.335 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:49.336 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:49.336 DEBUG:boto:Method: GET 2022-09-12T19:03:49.337 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:49.337 DEBUG:boto:Data: 2022-09-12T19:03:49.338 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.338 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:49.339 DEBUG:boto:Port: 8001 2022-09-12T19:03:49.340 DEBUG:boto:Params: {} 2022-09-12T19:03:49.341 DEBUG:boto:Token: None 2022-09-12T19:03:49.342 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/ 2022-09-12T19:03:49.343 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ= 2022-09-12T19:03:49.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ=', 'Content-Length': '0'} 2022-09-12T19:03:49.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000395f6f86eba9102b-00631f8295-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.347 DEBUG:boto:b'jdpiak-17obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:49.348 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:49.349 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:49.349 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:49.350 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:49.351 DEBUG:boto:Method: GET 2022-09-12T19:03:49.351 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone1 2022-09-12T19:03:49.352 DEBUG:boto:Data: 2022-09-12T19:03:49.352 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.353 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:49.353 DEBUG:boto:Port: 8000 2022-09-12T19:03:49.354 DEBUG:boto:Params: {} 2022-09-12T19:03:49.354 DEBUG:boto:Token: None 2022-09-12T19:03:49.355 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/ 2022-09-12T19:03:49.355 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ= 2022-09-12T19:03:49.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ=', 'Content-Length': '0'} 2022-09-12T19:03:49.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e858423cd36bdb72-00631f8295-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.359 DEBUG:boto:b'jdpiak-17obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.488Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:49.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:49.361 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:49.361 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:49.362 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.362 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.363 DEBUG:boto:Method: GET 2022-09-12T19:03:49.363 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.364 DEBUG:boto:Data: 2022-09-12T19:03:49.365 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.365 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:49.366 DEBUG:boto:Port: 8001 2022-09-12T19:03:49.366 DEBUG:boto:Params: {} 2022-09-12T19:03:49.367 DEBUG:boto:Token: None 2022-09-12T19:03:49.367 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.368 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4Q+wvG3lUrGUA6QldHhhHymCaHk= 2022-09-12T19:03:49.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4Q+wvG3lUrGUA6QldHhhHymCaHk=', 'Content-Length': '0'} 2022-09-12T19:03:49.371 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e121143600eb32ab-00631f8295-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.414 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:49.415 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:49.416 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.417 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.418 DEBUG:boto:Method: GET 2022-09-12T19:03:49.419 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.420 DEBUG:boto:Data: 2022-09-12T19:03:49.421 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.422 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:49.423 DEBUG:boto:Port: 8000 2022-09-12T19:03:49.423 DEBUG:boto:Params: {} 2022-09-12T19:03:49.424 DEBUG:boto:Token: None 2022-09-12T19:03:49.425 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.425 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4Q+wvG3lUrGUA6QldHhhHymCaHk= 2022-09-12T19:03:49.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4Q+wvG3lUrGUA6QldHhhHymCaHk=', 'Content-Length': '0'} 2022-09-12T19:03:49.428 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5a56820661c2687-00631f8295-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.430 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:49.430 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:49.431 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.431 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.432 DEBUG:boto:Method: HEAD 2022-09-12T19:03:49.433 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.433 DEBUG:boto:Data: 2022-09-12T19:03:49.434 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.435 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:49.435 DEBUG:boto:Port: 8001 2022-09-12T19:03:49.436 DEBUG:boto:Params: {} 2022-09-12T19:03:49.436 DEBUG:boto:Token: None 2022-09-12T19:03:49.437 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.437 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ffzq83u3RCEiPTA2HNA94C0CVRk= 2022-09-12T19:03:49.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ffzq83u3RCEiPTA2HNA94C0CVRk=', 'Content-Length': '0'} 2022-09-12T19:03:49.440 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1a4f1b1215e2328-00631f8295-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.441 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:49.441 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:49.442 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.443 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.443 DEBUG:boto:Method: HEAD 2022-09-12T19:03:49.444 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.445 DEBUG:boto:Data: 2022-09-12T19:03:49.445 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.446 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:49.446 DEBUG:boto:Port: 8000 2022-09-12T19:03:49.447 DEBUG:boto:Params: {} 2022-09-12T19:03:49.447 DEBUG:boto:Token: None 2022-09-12T19:03:49.448 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.448 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ffzq83u3RCEiPTA2HNA94C0CVRk= 2022-09-12T19:03:49.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ffzq83u3RCEiPTA2HNA94C0CVRk=', 'Content-Length': '0'} 2022-09-12T19:03:49.451 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000019beb834096602a-00631f8295-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:49.453 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:49.453 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:49.454 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.454 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.455 DEBUG:boto:Method: GET 2022-09-12T19:03:49.455 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.456 DEBUG:boto:Data: 2022-09-12T19:03:49.456 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.457 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:49.458 DEBUG:boto:Port: 8001 2022-09-12T19:03:49.458 DEBUG:boto:Params: {} 2022-09-12T19:03:49.459 DEBUG:boto:Token: None 2022-09-12T19:03:49.460 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.460 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4Q+wvG3lUrGUA6QldHhhHymCaHk= 2022-09-12T19:03:49.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4Q+wvG3lUrGUA6QldHhhHymCaHk=', 'Content-Length': '0'} 2022-09-12T19:03:49.463 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009822bf1e41da41e9-00631f8295-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.464 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:49.465 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:49.466 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.466 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.467 DEBUG:boto:Method: GET 2022-09-12T19:03:49.467 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.468 DEBUG:boto:Data: 2022-09-12T19:03:49.469 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.469 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:49.470 DEBUG:boto:Port: 8000 2022-09-12T19:03:49.470 DEBUG:boto:Params: {} 2022-09-12T19:03:49.471 DEBUG:boto:Token: None 2022-09-12T19:03:49.472 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.473 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4Q+wvG3lUrGUA6QldHhhHymCaHk= 2022-09-12T19:03:49.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4Q+wvG3lUrGUA6QldHhhHymCaHk=', 'Content-Length': '0'} 2022-09-12T19:03:49.476 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7b83cb00252f839-00631f8295-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.478 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:49.478 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:49.479 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:49.480 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:49.481 DEBUG:boto:Method: GET 2022-09-12T19:03:49.482 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:49.483 DEBUG:boto:Data: 2022-09-12T19:03:49.483 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.484 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:49.484 DEBUG:boto:Port: 8001 2022-09-12T19:03:49.485 DEBUG:boto:Params: {} 2022-09-12T19:03:49.486 DEBUG:boto:Token: None 2022-09-12T19:03:49.487 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/ 2022-09-12T19:03:49.487 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ= 2022-09-12T19:03:49.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ=', 'Content-Length': '0'} 2022-09-12T19:03:49.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abde28fddd3490e7-00631f8295-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.492 DEBUG:boto:b'jdpiak-17obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:49.493 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:49.493 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:49.494 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:49.494 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:49.495 DEBUG:boto:Method: GET 2022-09-12T19:03:49.495 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:49.496 DEBUG:boto:Data: 2022-09-12T19:03:49.497 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.497 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:49.498 DEBUG:boto:Port: 8000 2022-09-12T19:03:49.499 DEBUG:boto:Params: {} 2022-09-12T19:03:49.499 DEBUG:boto:Token: None 2022-09-12T19:03:49.500 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/ 2022-09-12T19:03:49.500 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ= 2022-09-12T19:03:49.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ=', 'Content-Length': '0'} 2022-09-12T19:03:49.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090f2cef22618082c-00631f8295-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.505 DEBUG:boto:b'jdpiak-17obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:49.506 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:49.506 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:49.507 DEBUG:boto:Method: HEAD 2022-09-12T19:03:49.507 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2 2022-09-12T19:03:49.508 DEBUG:boto:Data: 2022-09-12T19:03:49.508 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.509 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:49.509 DEBUG:boto:Port: 8001 2022-09-12T19:03:49.510 DEBUG:boto:Params: {} 2022-09-12T19:03:49.510 DEBUG:boto:Token: None 2022-09-12T19:03:49.511 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone2 2022-09-12T19:03:49.512 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L3R8K6FV7zdTtuw9w8zxic7V5Zk= 2022-09-12T19:03:49.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L3R8K6FV7zdTtuw9w8zxic7V5Zk=', 'Content-Length': '0'} 2022-09-12T19:03:49.515 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da5c3dfb52650dab-00631f8295-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.516 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:49.516 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:49.517 DEBUG:boto:Method: HEAD 2022-09-12T19:03:49.517 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2 2022-09-12T19:03:49.518 DEBUG:boto:Data: 2022-09-12T19:03:49.519 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.520 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:49.521 DEBUG:boto:Port: 8000 2022-09-12T19:03:49.521 DEBUG:boto:Params: {} 2022-09-12T19:03:49.522 DEBUG:boto:Token: None 2022-09-12T19:03:49.523 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone2 2022-09-12T19:03:49.524 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L3R8K6FV7zdTtuw9w8zxic7V5Zk= 2022-09-12T19:03:49.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L3R8K6FV7zdTtuw9w8zxic7V5Zk=', 'Content-Length': '0'} 2022-09-12T19:03:49.528 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000004ad27a5f558736-00631f8295-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:49.529 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:49.529 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:49.530 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.531 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.531 DEBUG:boto:Method: GET 2022-09-12T19:03:49.532 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.532 DEBUG:boto:Data: 2022-09-12T19:03:49.533 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.533 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:49.534 DEBUG:boto:Port: 8001 2022-09-12T19:03:49.535 DEBUG:boto:Params: {} 2022-09-12T19:03:49.535 DEBUG:boto:Token: None 2022-09-12T19:03:49.536 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.537 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4Q+wvG3lUrGUA6QldHhhHymCaHk= 2022-09-12T19:03:49.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4Q+wvG3lUrGUA6QldHhhHymCaHk=', 'Content-Length': '0'} 2022-09-12T19:03:49.539 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f3cd21187160c06-00631f8295-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.582 DEBUG:boto:path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:49.583 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2 2022-09-12T19:03:49.584 DEBUG:boto:path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.585 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.586 DEBUG:boto:Method: GET 2022-09-12T19:03:49.586 DEBUG:boto:Path: /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.587 DEBUG:boto:Data: 2022-09-12T19:03:49.587 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.588 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:49.589 DEBUG:boto:Port: 8000 2022-09-12T19:03:49.589 DEBUG:boto:Params: {} 2022-09-12T19:03:49.590 DEBUG:boto:Token: None 2022-09-12T19:03:49.591 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone2?versionId=0f1rsOoGVVMRuyy11YfWbutRvo943lC 2022-09-12T19:03:49.591 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4Q+wvG3lUrGUA6QldHhhHymCaHk= 2022-09-12T19:03:49.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4Q+wvG3lUrGUA6QldHhhHymCaHk=', 'Content-Length': '0'} 2022-09-12T19:03:49.595 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', '0f1rsOoGVVMRuyy11YfWbutRvo943lC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c77222b90ed781a-00631f8295-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.596 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:49.596 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:49.597 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:49.597 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:49.598 DEBUG:boto:Method: GET 2022-09-12T19:03:49.599 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:49.600 DEBUG:boto:Data: 2022-09-12T19:03:49.601 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.602 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:49.603 DEBUG:boto:Port: 8001 2022-09-12T19:03:49.604 DEBUG:boto:Params: {} 2022-09-12T19:03:49.605 DEBUG:boto:Token: None 2022-09-12T19:03:49.606 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/ 2022-09-12T19:03:49.607 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ= 2022-09-12T19:03:49.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ=', 'Content-Length': '0'} 2022-09-12T19:03:49.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e6e342a908e2ad4-00631f8295-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.611 DEBUG:boto:b'jdpiak-17obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:49.612 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:49.612 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:49.613 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:49.613 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:49.614 DEBUG:boto:Method: GET 2022-09-12T19:03:49.614 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone2 2022-09-12T19:03:49.615 DEBUG:boto:Data: 2022-09-12T19:03:49.615 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.616 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:49.616 DEBUG:boto:Port: 8000 2022-09-12T19:03:49.617 DEBUG:boto:Params: {} 2022-09-12T19:03:49.618 DEBUG:boto:Token: None 2022-09-12T19:03:49.618 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/ 2022-09-12T19:03:49.619 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ= 2022-09-12T19:03:49.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ=', 'Content-Length': '0'} 2022-09-12T19:03:49.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084d321c86f53b13e-00631f8295-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.623 DEBUG:boto:b'jdpiak-17obj-test-zone21000falseobj-test-zone22022-09-12T18:59:23.638Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:49.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:49.624 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:49.625 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:49.626 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.626 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.627 DEBUG:boto:Method: GET 2022-09-12T19:03:49.627 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.628 DEBUG:boto:Data: 2022-09-12T19:03:49.629 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.629 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:49.630 DEBUG:boto:Port: 8001 2022-09-12T19:03:49.630 DEBUG:boto:Params: {} 2022-09-12T19:03:49.631 DEBUG:boto:Token: None 2022-09-12T19:03:49.631 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.632 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Sxp7dO1Alhf5MvTgRklGjFBZVa4= 2022-09-12T19:03:49.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Sxp7dO1Alhf5MvTgRklGjFBZVa4=', 'Content-Length': '0'} 2022-09-12T19:03:49.635 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069b3650fda941b4a-00631f8295-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.636 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:49.637 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:49.637 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.638 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.638 DEBUG:boto:Method: GET 2022-09-12T19:03:49.639 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.639 DEBUG:boto:Data: 2022-09-12T19:03:49.640 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.640 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:49.641 DEBUG:boto:Port: 8000 2022-09-12T19:03:49.642 DEBUG:boto:Params: {} 2022-09-12T19:03:49.643 DEBUG:boto:Token: None 2022-09-12T19:03:49.644 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.645 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Sxp7dO1Alhf5MvTgRklGjFBZVa4= 2022-09-12T19:03:49.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Sxp7dO1Alhf5MvTgRklGjFBZVa4=', 'Content-Length': '0'} 2022-09-12T19:03:49.649 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000c349c527336460-00631f8295-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.690 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:49.691 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:49.692 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.692 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.693 DEBUG:boto:Method: HEAD 2022-09-12T19:03:49.694 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.694 DEBUG:boto:Data: 2022-09-12T19:03:49.695 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.696 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:49.696 DEBUG:boto:Port: 8001 2022-09-12T19:03:49.697 DEBUG:boto:Params: {} 2022-09-12T19:03:49.698 DEBUG:boto:Token: None 2022-09-12T19:03:49.698 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.699 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G8FO6W45TQQ8jaMp+0Z7JEx9Uvs= 2022-09-12T19:03:49.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G8FO6W45TQQ8jaMp+0Z7JEx9Uvs=', 'Content-Length': '0'} 2022-09-12T19:03:49.702 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db9f80a11af144f3-00631f8295-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.704 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:49.705 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:49.706 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.707 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.707 DEBUG:boto:Method: HEAD 2022-09-12T19:03:49.708 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.709 DEBUG:boto:Data: 2022-09-12T19:03:49.710 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.711 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:49.711 DEBUG:boto:Port: 8000 2022-09-12T19:03:49.712 DEBUG:boto:Params: {} 2022-09-12T19:03:49.712 DEBUG:boto:Token: None 2022-09-12T19:03:49.713 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.713 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G8FO6W45TQQ8jaMp+0Z7JEx9Uvs= 2022-09-12T19:03:49.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G8FO6W45TQQ8jaMp+0Z7JEx9Uvs=', 'Content-Length': '0'} 2022-09-12T19:03:49.716 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d93b1c180a8bc9a-00631f8295-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:49.717 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:49.718 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:49.719 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.719 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.720 DEBUG:boto:Method: GET 2022-09-12T19:03:49.720 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.721 DEBUG:boto:Data: 2022-09-12T19:03:49.721 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.722 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:49.722 DEBUG:boto:Port: 8001 2022-09-12T19:03:49.723 DEBUG:boto:Params: {} 2022-09-12T19:03:49.723 DEBUG:boto:Token: None 2022-09-12T19:03:49.724 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.725 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Sxp7dO1Alhf5MvTgRklGjFBZVa4= 2022-09-12T19:03:49.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Sxp7dO1Alhf5MvTgRklGjFBZVa4=', 'Content-Length': '0'} 2022-09-12T19:03:49.728 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076c16d6f4eeb72c9-00631f8295-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.728 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:49.729 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:49.729 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.730 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.731 DEBUG:boto:Method: GET 2022-09-12T19:03:49.731 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.732 DEBUG:boto:Data: 2022-09-12T19:03:49.732 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.733 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:49.734 DEBUG:boto:Port: 8000 2022-09-12T19:03:49.734 DEBUG:boto:Params: {} 2022-09-12T19:03:49.735 DEBUG:boto:Token: None 2022-09-12T19:03:49.735 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.736 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Sxp7dO1Alhf5MvTgRklGjFBZVa4= 2022-09-12T19:03:49.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Sxp7dO1Alhf5MvTgRklGjFBZVa4=', 'Content-Length': '0'} 2022-09-12T19:03:49.739 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000361810e5f39bd29-00631f8295-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.740 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:49.740 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:49.741 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:49.741 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:49.742 DEBUG:boto:Method: GET 2022-09-12T19:03:49.743 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:49.743 DEBUG:boto:Data: 2022-09-12T19:03:49.744 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.744 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:49.745 DEBUG:boto:Port: 8001 2022-09-12T19:03:49.745 DEBUG:boto:Params: {} 2022-09-12T19:03:49.746 DEBUG:boto:Token: None 2022-09-12T19:03:49.747 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/ 2022-09-12T19:03:49.748 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ= 2022-09-12T19:03:49.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ=', 'Content-Length': '0'} 2022-09-12T19:03:49.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077049bb343039eb6-00631f8295-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.752 DEBUG:boto:b'jdpiak-17obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:49.752 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:49.753 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:49.754 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:49.754 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:49.755 DEBUG:boto:Method: GET 2022-09-12T19:03:49.755 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:49.756 DEBUG:boto:Data: 2022-09-12T19:03:49.757 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.757 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:49.758 DEBUG:boto:Port: 8000 2022-09-12T19:03:49.759 DEBUG:boto:Params: {} 2022-09-12T19:03:49.759 DEBUG:boto:Token: None 2022-09-12T19:03:49.760 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/ 2022-09-12T19:03:49.760 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ= 2022-09-12T19:03:49.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ=', 'Content-Length': '0'} 2022-09-12T19:03:49.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000622867260c77efe9-00631f8295-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.765 DEBUG:boto:b'jdpiak-17obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:49.767 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:49.767 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:49.768 DEBUG:boto:Method: HEAD 2022-09-12T19:03:49.769 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3 2022-09-12T19:03:49.770 DEBUG:boto:Data: 2022-09-12T19:03:49.771 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.772 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:49.773 DEBUG:boto:Port: 8001 2022-09-12T19:03:49.773 DEBUG:boto:Params: {} 2022-09-12T19:03:49.774 DEBUG:boto:Token: None 2022-09-12T19:03:49.774 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone3 2022-09-12T19:03:49.775 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OaiefuIpkiRHRCp33o0t9jgOdgs= 2022-09-12T19:03:49.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OaiefuIpkiRHRCp33o0t9jgOdgs=', 'Content-Length': '0'} 2022-09-12T19:03:49.778 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9046ab71de53874-00631f8295-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.778 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:49.779 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:49.779 DEBUG:boto:Method: HEAD 2022-09-12T19:03:49.780 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3 2022-09-12T19:03:49.780 DEBUG:boto:Data: 2022-09-12T19:03:49.781 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.781 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:49.782 DEBUG:boto:Port: 8000 2022-09-12T19:03:49.782 DEBUG:boto:Params: {} 2022-09-12T19:03:49.783 DEBUG:boto:Token: None 2022-09-12T19:03:49.783 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone3 2022-09-12T19:03:49.784 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OaiefuIpkiRHRCp33o0t9jgOdgs= 2022-09-12T19:03:49.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OaiefuIpkiRHRCp33o0t9jgOdgs=', 'Content-Length': '0'} 2022-09-12T19:03:49.787 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000083f6b853e5a65589-00631f8295-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:49.788 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:49.789 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:49.789 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.790 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.790 DEBUG:boto:Method: GET 2022-09-12T19:03:49.791 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.791 DEBUG:boto:Data: 2022-09-12T19:03:49.792 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.792 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:49.793 DEBUG:boto:Port: 8001 2022-09-12T19:03:49.793 DEBUG:boto:Params: {} 2022-09-12T19:03:49.794 DEBUG:boto:Token: None 2022-09-12T19:03:49.795 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.795 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Sxp7dO1Alhf5MvTgRklGjFBZVa4= 2022-09-12T19:03:49.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Sxp7dO1Alhf5MvTgRklGjFBZVa4=', 'Content-Length': '0'} 2022-09-12T19:03:49.798 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005be073ea6a33e61f-00631f8295-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.842 DEBUG:boto:path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:49.843 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3 2022-09-12T19:03:49.843 DEBUG:boto:path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.844 DEBUG:boto:auth_path=/jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.844 DEBUG:boto:Method: GET 2022-09-12T19:03:49.845 DEBUG:boto:Path: /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.846 DEBUG:boto:Data: 2022-09-12T19:03:49.846 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.847 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:49.847 DEBUG:boto:Port: 8000 2022-09-12T19:03:49.848 DEBUG:boto:Params: {} 2022-09-12T19:03:49.849 DEBUG:boto:Token: None 2022-09-12T19:03:49.850 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/obj-test-zone3?versionId=V6Ue4vhVY0WQ9FArGixLjY-hmND5K20 2022-09-12T19:03:49.850 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Sxp7dO1Alhf5MvTgRklGjFBZVa4= 2022-09-12T19:03:49.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Sxp7dO1Alhf5MvTgRklGjFBZVa4=', 'Content-Length': '0'} 2022-09-12T19:03:49.853 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'V6Ue4vhVY0WQ9FArGixLjY-hmND5K20'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009af0de4c0b639070-00631f8295-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.854 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:49.855 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:49.855 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:49.856 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:49.857 DEBUG:boto:Method: GET 2022-09-12T19:03:49.857 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:49.858 DEBUG:boto:Data: 2022-09-12T19:03:49.858 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.859 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:49.859 DEBUG:boto:Port: 8001 2022-09-12T19:03:49.860 DEBUG:boto:Params: {} 2022-09-12T19:03:49.861 DEBUG:boto:Token: None 2022-09-12T19:03:49.861 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/ 2022-09-12T19:03:49.862 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ= 2022-09-12T19:03:49.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ=', 'Content-Length': '0'} 2022-09-12T19:03:49.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045e8f18ee8f34ea4-00631f8295-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.866 DEBUG:boto:b'jdpiak-17obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:49.867 DEBUG:boto:path=/jdpiak-17/ 2022-09-12T19:03:49.867 DEBUG:boto:auth_path=/jdpiak-17/ 2022-09-12T19:03:49.868 DEBUG:boto:path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:49.868 DEBUG:boto:auth_path=/jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:49.869 DEBUG:boto:Method: GET 2022-09-12T19:03:49.869 DEBUG:boto:Path: /jdpiak-17/?prefix=obj-test-zone3 2022-09-12T19:03:49.870 DEBUG:boto:Data: 2022-09-12T19:03:49.871 DEBUG:boto:Headers: {} 2022-09-12T19:03:49.871 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:49.872 DEBUG:boto:Port: 8000 2022-09-12T19:03:49.873 DEBUG:boto:Params: {} 2022-09-12T19:03:49.873 DEBUG:boto:Token: None 2022-09-12T19:03:49.874 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:49 GMT /jdpiak-17/ 2022-09-12T19:03:49.875 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ= 2022-09-12T19:03:49.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h+WSyerwxdEALTxrD7mR9B2scdQ=', 'Content-Length': '0'} 2022-09-12T19:03:49.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b663bd37c2ab2276-00631f8295-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:49.879 DEBUG:boto:b'jdpiak-17obj-test-zone31000falseobj-test-zone32022-09-12T18:59:23.801Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:49.880 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-17 zones={test-zone3, test-zone2} 2022-09-12T19:03:49.880 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-18 2022-09-12T19:03:49.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:49.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:49.883 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:03:49.946 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:03:49.947 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:03:49.992 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:03:49.989+0000 7f814bd31600 1 bucket sync caught up with source: 2022-09-12T19:03:49.993 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000011.34.5, , 00000000004.12.5, , 00000000008.21.5, , , , , ] 2022-09-12T19:03:49.993 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000011.34.5, , 00000000004.12.5, , 00000000008.21.5, , , , , ] 2022-09-12T19:03:49.993 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:03:49.989+0000 7f814bd31600 0 bucket checkpoint complete 2022-09-12T19:03:50.014 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-18 2022-09-12T19:03:50.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:50.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:50.016 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:03:50.082 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:03:50.083 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:03:50.123 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:50.120+0000 7efcd0d29600 1 bucket sync caught up with source: 2022-09-12T19:03:50.123 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000011.34.5, , 00000000004.12.5, , 00000000008.21.5, , , , , ] 2022-09-12T19:03:50.124 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000011.34.5, , 00000000004.12.5, , 00000000008.21.5, , , , , ] 2022-09-12T19:03:50.124 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:50.120+0000 7efcd0d29600 0 bucket checkpoint complete 2022-09-12T19:03:50.136 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-18 2022-09-12T19:03:50.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:50.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:50.138 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:03:50.195 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:03:50.195 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:03:50.244 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:03:50.240+0000 7ff5105b3600 1 bucket sync caught up with source: 2022-09-12T19:03:50.244 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000011.34.5, , 00000000004.12.5, , 00000000008.21.5, , , , , ] 2022-09-12T19:03:50.244 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000011.34.5, , 00000000004.12.5, , 00000000008.21.5, , , , , ] 2022-09-12T19:03:50.245 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:03:50.240+0000 7ff5105b3600 0 bucket checkpoint complete 2022-09-12T19:03:50.262 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-18 2022-09-12T19:03:50.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:50.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:50.263 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:03:50.328 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:03:50.328 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:03:50.373 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:50.370+0000 7f622cfd5600 1 bucket sync caught up with source: 2022-09-12T19:03:50.374 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000004.34.5, , 00000000011.37.5, , 00000000008.31.5, , , , , ] 2022-09-12T19:03:50.374 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000004.34.5, , 00000000011.37.5, , 00000000008.31.5, , , , , ] 2022-09-12T19:03:50.374 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:50.370+0000 7f622cfd5600 0 bucket checkpoint complete 2022-09-12T19:03:50.389 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-18 2022-09-12T19:03:50.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:50.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:50.391 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:03:50.448 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:03:50.448 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:03:50.500 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:50.491+0000 7f725d0d4600 1 bucket sync caught up with source: 2022-09-12T19:03:50.500 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000004.34.5, , 00000000011.37.5, , 00000000008.31.5, , , , , ] 2022-09-12T19:03:50.501 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000004.34.5, , 00000000011.37.5, , 00000000008.31.5, , , , , ] 2022-09-12T19:03:50.501 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:50.491+0000 7f725d0d4600 0 bucket checkpoint complete 2022-09-12T19:03:50.508 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-18 2022-09-12T19:03:50.509 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-18 2022-09-12T19:03:50.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:50.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:50.511 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:03:50.568 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:03:50.568 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:03:50.602 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:50.599+0000 7effea590600 1 bucket sync caught up with source: 2022-09-12T19:03:50.603 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000004.42.5, , 00000000004.22.5, , 00000000017.56.5, , , , , ] 2022-09-12T19:03:50.603 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000004.42.5, , 00000000004.22.5, , 00000000017.56.5, , , , , ] 2022-09-12T19:03:50.603 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:50.599+0000 7effea590600 0 bucket checkpoint complete 2022-09-12T19:03:50.615 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-18 2022-09-12T19:03:50.615 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:50.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:50.617 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:03:50.673 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:03:50.673 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:03:50.714 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:03:50.711+0000 7f43c0133600 1 bucket sync caught up with source: 2022-09-12T19:03:50.715 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000004.42.5, , 00000000004.22.5, , 00000000017.56.5, , , , , ] 2022-09-12T19:03:50.715 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000004.42.5, , 00000000004.22.5, , 00000000017.56.5, , , , , ] 2022-09-12T19:03:50.715 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:03:50.711+0000 7f43c0133600 0 bucket checkpoint complete 2022-09-12T19:03:50.728 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-18 2022-09-12T19:03:50.729 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-18 zones={test-zone2, test-zone1} 2022-09-12T19:03:50.730 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:50.730 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:50.731 DEBUG:boto:Method: HEAD 2022-09-12T19:03:50.731 DEBUG:boto:Path: /jdpiak-18/ 2022-09-12T19:03:50.732 DEBUG:boto:Data: 2022-09-12T19:03:50.732 DEBUG:boto:Headers: {} 2022-09-12T19:03:50.733 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:50.733 DEBUG:boto:Port: 8000 2022-09-12T19:03:50.734 DEBUG:boto:Params: {} 2022-09-12T19:03:50.735 DEBUG:boto:Token: None 2022-09-12T19:03:50.735 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:50 GMT /jdpiak-18/ 2022-09-12T19:03:50.736 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wGtDc73UOxZKa0+/PEkdoIJkxNY= 2022-09-12T19:03:50.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wGtDc73UOxZKa0+/PEkdoIJkxNY=', 'Content-Length': '0'} 2022-09-12T19:03:50.739 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009947cd270cbeab0a-00631f8296-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:03:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:50.739 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:50.740 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:50.740 DEBUG:boto:Method: HEAD 2022-09-12T19:03:50.741 DEBUG:boto:Path: /jdpiak-18/ 2022-09-12T19:03:50.741 DEBUG:boto:Data: 2022-09-12T19:03:50.742 DEBUG:boto:Headers: {} 2022-09-12T19:03:50.743 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:50.743 DEBUG:boto:Port: 8000 2022-09-12T19:03:50.744 DEBUG:boto:Params: {} 2022-09-12T19:03:50.744 DEBUG:boto:Token: None 2022-09-12T19:03:50.745 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:50 GMT /jdpiak-18/ 2022-09-12T19:03:50.745 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wGtDc73UOxZKa0+/PEkdoIJkxNY= 2022-09-12T19:03:50.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wGtDc73UOxZKa0+/PEkdoIJkxNY=', 'Content-Length': '0'} 2022-09-12T19:03:50.748 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2e1505ee14551ab-00631f8296-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:03:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:50.749 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:03:50.750 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:50.750 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:50.751 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:50.751 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:50.752 DEBUG:boto:Method: GET 2022-09-12T19:03:50.753 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:50.753 DEBUG:boto:Data: 2022-09-12T19:03:50.753 DEBUG:boto:Headers: {} 2022-09-12T19:03:50.754 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:50.755 DEBUG:boto:Port: 8000 2022-09-12T19:03:50.755 DEBUG:boto:Params: {} 2022-09-12T19:03:50.756 DEBUG:boto:Token: None 2022-09-12T19:03:50.756 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:50 GMT /jdpiak-18/?versions 2022-09-12T19:03:50.757 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AmlyHWW9soYplfOp6iMEH6h4WKc= 2022-09-12T19:03:50.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AmlyHWW9soYplfOp6iMEH6h4WKc=', 'Content-Length': '0'} 2022-09-12T19:03:50.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007369828cda4fa262-00631f8296-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:50.761 DEBUG:boto:b'jdpiak-18999falsedummyqUufSwmbKETbT7yNOH8sL0A4tddUbNOtrue2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ToUtj2XoS.tz3goIbvjFwiigMyzGOpLtrue2022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2DPksivOWVYhJDrEwQqHYV7wK1Ul1JkWtrue2022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Ztrue2022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:50.761 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:03:50.762 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:03:50.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:03:50.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:03:50.763 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:03:50.764 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:50.764 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:50.765 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:50.765 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:50.766 DEBUG:boto:Method: GET 2022-09-12T19:03:50.767 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:50.767 DEBUG:boto:Data: 2022-09-12T19:03:50.768 DEBUG:boto:Headers: {} 2022-09-12T19:03:50.768 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:50.768 DEBUG:boto:Port: 8000 2022-09-12T19:03:50.769 DEBUG:boto:Params: {} 2022-09-12T19:03:50.770 DEBUG:boto:Token: None 2022-09-12T19:03:50.770 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:50 GMT /jdpiak-18/?versions 2022-09-12T19:03:50.771 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AmlyHWW9soYplfOp6iMEH6h4WKc= 2022-09-12T19:03:50.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AmlyHWW9soYplfOp6iMEH6h4WKc=', 'Content-Length': '0'} 2022-09-12T19:03:50.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000460dcd395002cbed-00631f8296-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:50.774 DEBUG:boto:b'jdpiak-18999falsedummyqUufSwmbKETbT7yNOH8sL0A4tddUbNOtrue2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ToUtj2XoS.tz3goIbvjFwiigMyzGOpLtrue2022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2DPksivOWVYhJDrEwQqHYV7wK1Ul1JkWtrue2022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Ztrue2022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:50.775 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:03:50.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:03:50.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:03:50.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:03:50.777 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:50.778 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:50.778 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:50.779 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:50.779 DEBUG:boto:Method: GET 2022-09-12T19:03:50.780 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:50.780 DEBUG:boto:Data: 2022-09-12T19:03:50.781 DEBUG:boto:Headers: {} 2022-09-12T19:03:50.781 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:50.782 DEBUG:boto:Port: 8000 2022-09-12T19:03:50.783 DEBUG:boto:Params: {} 2022-09-12T19:03:50.783 DEBUG:boto:Token: None 2022-09-12T19:03:50.784 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:50 GMT /jdpiak-18/?versions 2022-09-12T19:03:50.784 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AmlyHWW9soYplfOp6iMEH6h4WKc= 2022-09-12T19:03:50.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AmlyHWW9soYplfOp6iMEH6h4WKc=', 'Content-Length': '0'} 2022-09-12T19:03:50.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098aec2cb103497d5-00631f8296-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:50.788 DEBUG:boto:b'jdpiak-18999falsedummyqUufSwmbKETbT7yNOH8sL0A4tddUbNOtrue2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ToUtj2XoS.tz3goIbvjFwiigMyzGOpLtrue2022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2DPksivOWVYhJDrEwQqHYV7wK1Ul1JkWtrue2022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Ztrue2022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:50.789 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:50.789 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:50.790 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:50.790 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:50.791 DEBUG:boto:Method: GET 2022-09-12T19:03:50.791 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:50.792 DEBUG:boto:Data: 2022-09-12T19:03:50.792 DEBUG:boto:Headers: {} 2022-09-12T19:03:50.793 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:50.793 DEBUG:boto:Port: 8000 2022-09-12T19:03:50.794 DEBUG:boto:Params: {} 2022-09-12T19:03:50.794 DEBUG:boto:Token: None 2022-09-12T19:03:50.795 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:50 GMT /jdpiak-18/?versions 2022-09-12T19:03:50.796 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AmlyHWW9soYplfOp6iMEH6h4WKc= 2022-09-12T19:03:50.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AmlyHWW9soYplfOp6iMEH6h4WKc=', 'Content-Length': '0'} 2022-09-12T19:03:50.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da23e45d6c3f4bc5-00631f8296-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:50.799 DEBUG:boto:b'jdpiak-18999falsedummyqUufSwmbKETbT7yNOH8sL0A4tddUbNOtrue2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ToUtj2XoS.tz3goIbvjFwiigMyzGOpLtrue2022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2DPksivOWVYhJDrEwQqHYV7wK1Ul1JkWtrue2022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Ztrue2022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:50.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:50.801 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:50.802 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:50.802 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.803 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.803 DEBUG:boto:Method: GET 2022-09-12T19:03:50.804 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.804 DEBUG:boto:Data: 2022-09-12T19:03:50.805 DEBUG:boto:Headers: {} 2022-09-12T19:03:50.805 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:50.806 DEBUG:boto:Port: 8000 2022-09-12T19:03:50.806 DEBUG:boto:Params: {} 2022-09-12T19:03:50.807 DEBUG:boto:Token: None 2022-09-12T19:03:50.808 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:50 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.808 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SXkdCBEGWTWWKUl10L+Yw5NOWyc= 2022-09-12T19:03:50.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SXkdCBEGWTWWKUl10L+Yw5NOWyc=', 'Content-Length': '0'} 2022-09-12T19:03:50.811 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000945682d6a5f7f933-00631f8296-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:50.812 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:50.812 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:50.813 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.813 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.814 DEBUG:boto:Method: GET 2022-09-12T19:03:50.814 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.815 DEBUG:boto:Data: 2022-09-12T19:03:50.815 DEBUG:boto:Headers: {} 2022-09-12T19:03:50.816 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:50.816 DEBUG:boto:Port: 8000 2022-09-12T19:03:50.817 DEBUG:boto:Params: {} 2022-09-12T19:03:50.818 DEBUG:boto:Token: None 2022-09-12T19:03:50.818 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:50 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.819 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SXkdCBEGWTWWKUl10L+Yw5NOWyc= 2022-09-12T19:03:50.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SXkdCBEGWTWWKUl10L+Yw5NOWyc=', 'Content-Length': '0'} 2022-09-12T19:03:50.821 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000088aafedf01dc2130-00631f8296-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:50.822 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:50.823 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:50.823 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.824 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.824 DEBUG:boto:Method: HEAD 2022-09-12T19:03:50.825 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.825 DEBUG:boto:Data: 2022-09-12T19:03:50.826 DEBUG:boto:Headers: {} 2022-09-12T19:03:50.827 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:50.827 DEBUG:boto:Port: 8000 2022-09-12T19:03:50.828 DEBUG:boto:Params: {} 2022-09-12T19:03:50.828 DEBUG:boto:Token: None 2022-09-12T19:03:50.829 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:50 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.830 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:g9qNPQIwgMaAUnFGvG/4S7Ayci8= 2022-09-12T19:03:50.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:g9qNPQIwgMaAUnFGvG/4S7Ayci8=', 'Content-Length': '0'} 2022-09-12T19:03:50.832 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002ce70524326999c2-00631f8296-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:50.833 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:50.833 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:50.834 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.834 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.835 DEBUG:boto:Method: HEAD 2022-09-12T19:03:50.835 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.836 DEBUG:boto:Data: 2022-09-12T19:03:50.836 DEBUG:boto:Headers: {} 2022-09-12T19:03:50.837 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:50.837 DEBUG:boto:Port: 8000 2022-09-12T19:03:50.838 DEBUG:boto:Params: {} 2022-09-12T19:03:50.838 DEBUG:boto:Token: None 2022-09-12T19:03:50.839 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:50 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.840 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:g9qNPQIwgMaAUnFGvG/4S7Ayci8= 2022-09-12T19:03:50.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:g9qNPQIwgMaAUnFGvG/4S7Ayci8=', 'Content-Length': '0'} 2022-09-12T19:03:50.842 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000045b3675af3c0bf65-00631f8296-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:50.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:50.843 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:50.844 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:50.844 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.845 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.846 DEBUG:boto:Method: GET 2022-09-12T19:03:50.846 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.847 DEBUG:boto:Data: 2022-09-12T19:03:50.847 DEBUG:boto:Headers: {} 2022-09-12T19:03:50.848 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:50.848 DEBUG:boto:Port: 8000 2022-09-12T19:03:50.849 DEBUG:boto:Params: {} 2022-09-12T19:03:50.850 DEBUG:boto:Token: None 2022-09-12T19:03:50.850 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:50 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.851 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SXkdCBEGWTWWKUl10L+Yw5NOWyc= 2022-09-12T19:03:50.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SXkdCBEGWTWWKUl10L+Yw5NOWyc=', 'Content-Length': '0'} 2022-09-12T19:03:50.854 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000006c366b7720be021-00631f8296-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:50.855 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:50.855 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:50.856 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.856 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.857 DEBUG:boto:Method: GET 2022-09-12T19:03:50.858 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.859 DEBUG:boto:Data: 2022-09-12T19:03:50.860 DEBUG:boto:Headers: {} 2022-09-12T19:03:50.861 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:50.862 DEBUG:boto:Port: 8000 2022-09-12T19:03:50.863 DEBUG:boto:Params: {} 2022-09-12T19:03:50.863 DEBUG:boto:Token: None 2022-09-12T19:03:50.864 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:50 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.866 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SXkdCBEGWTWWKUl10L+Yw5NOWyc= 2022-09-12T19:03:50.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SXkdCBEGWTWWKUl10L+Yw5NOWyc=', 'Content-Length': '0'} 2022-09-12T19:03:50.869 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b0f36c7d36e415d3-00631f8296-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:50.869 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:50.870 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:50.870 DEBUG:boto:path=/jdpiak-18/?prefix=dummy 2022-09-12T19:03:50.871 DEBUG:boto:auth_path=/jdpiak-18/?prefix=dummy 2022-09-12T19:03:50.871 DEBUG:boto:Method: GET 2022-09-12T19:03:50.872 DEBUG:boto:Path: /jdpiak-18/?prefix=dummy 2022-09-12T19:03:50.873 DEBUG:boto:Data: 2022-09-12T19:03:50.873 DEBUG:boto:Headers: {} 2022-09-12T19:03:50.874 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:50.874 DEBUG:boto:Port: 8000 2022-09-12T19:03:50.875 DEBUG:boto:Params: {} 2022-09-12T19:03:50.875 DEBUG:boto:Token: None 2022-09-12T19:03:50.876 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:50 GMT /jdpiak-18/ 2022-09-12T19:03:50.877 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uBdR1b/J8z4MPGVqEisFCueQYtE= 2022-09-12T19:03:50.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uBdR1b/J8z4MPGVqEisFCueQYtE=', 'Content-Length': '0'} 2022-09-12T19:03:50.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c14862f9e9959f5d-00631f8296-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:50.922 DEBUG:boto:b'jdpiak-18dummy1000falsedummy2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:50.923 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:50.924 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:50.925 DEBUG:boto:path=/jdpiak-18/?prefix=dummy 2022-09-12T19:03:50.926 DEBUG:boto:auth_path=/jdpiak-18/?prefix=dummy 2022-09-12T19:03:50.927 DEBUG:boto:Method: GET 2022-09-12T19:03:50.928 DEBUG:boto:Path: /jdpiak-18/?prefix=dummy 2022-09-12T19:03:50.928 DEBUG:boto:Data: 2022-09-12T19:03:50.929 DEBUG:boto:Headers: {} 2022-09-12T19:03:50.929 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:50.930 DEBUG:boto:Port: 8000 2022-09-12T19:03:50.931 DEBUG:boto:Params: {} 2022-09-12T19:03:50.931 DEBUG:boto:Token: None 2022-09-12T19:03:50.932 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:50 GMT /jdpiak-18/ 2022-09-12T19:03:50.933 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uBdR1b/J8z4MPGVqEisFCueQYtE= 2022-09-12T19:03:50.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uBdR1b/J8z4MPGVqEisFCueQYtE=', 'Content-Length': '0'} 2022-09-12T19:03:50.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0d76b2e3a213e0f-00631f8296-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:50.936 DEBUG:boto:b'jdpiak-18dummy1000falsedummy2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:50.937 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:50.938 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:50.938 DEBUG:boto:Method: HEAD 2022-09-12T19:03:50.939 DEBUG:boto:Path: /jdpiak-18/dummy 2022-09-12T19:03:50.939 DEBUG:boto:Data: 2022-09-12T19:03:50.940 DEBUG:boto:Headers: {} 2022-09-12T19:03:50.940 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:50.941 DEBUG:boto:Port: 8000 2022-09-12T19:03:50.941 DEBUG:boto:Params: {} 2022-09-12T19:03:50.942 DEBUG:boto:Token: None 2022-09-12T19:03:50.942 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:50 GMT /jdpiak-18/dummy 2022-09-12T19:03:50.943 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wxuBek8bQ+CPCAm1x9DKq3YTntA= 2022-09-12T19:03:50.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wxuBek8bQ+CPCAm1x9DKq3YTntA=', 'Content-Length': '0'} 2022-09-12T19:03:50.945 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000018eb636ba7eb5c6e-00631f8296-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:50.946 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:50.947 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:50.947 DEBUG:boto:Method: HEAD 2022-09-12T19:03:50.948 DEBUG:boto:Path: /jdpiak-18/dummy 2022-09-12T19:03:50.948 DEBUG:boto:Data: 2022-09-12T19:03:50.948 DEBUG:boto:Headers: {} 2022-09-12T19:03:50.949 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:50.949 DEBUG:boto:Port: 8000 2022-09-12T19:03:50.950 DEBUG:boto:Params: {} 2022-09-12T19:03:50.951 DEBUG:boto:Token: None 2022-09-12T19:03:50.951 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:50 GMT /jdpiak-18/dummy 2022-09-12T19:03:50.952 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wxuBek8bQ+CPCAm1x9DKq3YTntA= 2022-09-12T19:03:50.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wxuBek8bQ+CPCAm1x9DKq3YTntA=', 'Content-Length': '0'} 2022-09-12T19:03:50.954 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000099daa4db5b92634e-00631f8296-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:50.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:50.956 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:50.956 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:50.957 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.957 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.958 DEBUG:boto:Method: GET 2022-09-12T19:03:50.958 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.959 DEBUG:boto:Data: 2022-09-12T19:03:50.959 DEBUG:boto:Headers: {} 2022-09-12T19:03:50.960 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:50.960 DEBUG:boto:Port: 8000 2022-09-12T19:03:50.961 DEBUG:boto:Params: {} 2022-09-12T19:03:50.961 DEBUG:boto:Token: None 2022-09-12T19:03:50.962 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:50 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.963 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SXkdCBEGWTWWKUl10L+Yw5NOWyc= 2022-09-12T19:03:50.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SXkdCBEGWTWWKUl10L+Yw5NOWyc=', 'Content-Length': '0'} 2022-09-12T19:03:50.965 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000057427e0abfb97048-00631f8296-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:50.967 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:50.967 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:50.968 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.969 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.969 DEBUG:boto:Method: GET 2022-09-12T19:03:50.970 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.970 DEBUG:boto:Data: 2022-09-12T19:03:50.971 DEBUG:boto:Headers: {} 2022-09-12T19:03:50.971 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:50.972 DEBUG:boto:Port: 8000 2022-09-12T19:03:50.973 DEBUG:boto:Params: {} 2022-09-12T19:03:50.973 DEBUG:boto:Token: None 2022-09-12T19:03:50.974 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:50 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:50.975 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SXkdCBEGWTWWKUl10L+Yw5NOWyc= 2022-09-12T19:03:50.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SXkdCBEGWTWWKUl10L+Yw5NOWyc=', 'Content-Length': '0'} 2022-09-12T19:03:50.978 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bba1e729961a1dcb-00631f8296-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:50.978 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:50.979 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:50.980 DEBUG:boto:path=/jdpiak-18/?prefix=dummy 2022-09-12T19:03:50.980 DEBUG:boto:auth_path=/jdpiak-18/?prefix=dummy 2022-09-12T19:03:50.981 DEBUG:boto:Method: GET 2022-09-12T19:03:50.981 DEBUG:boto:Path: /jdpiak-18/?prefix=dummy 2022-09-12T19:03:50.982 DEBUG:boto:Data: 2022-09-12T19:03:50.983 DEBUG:boto:Headers: {} 2022-09-12T19:03:50.983 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:50.984 DEBUG:boto:Port: 8000 2022-09-12T19:03:50.984 DEBUG:boto:Params: {} 2022-09-12T19:03:50.985 DEBUG:boto:Token: None 2022-09-12T19:03:50.985 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:50 GMT /jdpiak-18/ 2022-09-12T19:03:50.986 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uBdR1b/J8z4MPGVqEisFCueQYtE= 2022-09-12T19:03:50.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uBdR1b/J8z4MPGVqEisFCueQYtE=', 'Content-Length': '0'} 2022-09-12T19:03:50.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007aea596e73832bc3-00631f8296-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:50.990 DEBUG:boto:b'jdpiak-18dummy1000falsedummy2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:50.991 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:50.991 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:50.992 DEBUG:boto:path=/jdpiak-18/?prefix=dummy 2022-09-12T19:03:50.992 DEBUG:boto:auth_path=/jdpiak-18/?prefix=dummy 2022-09-12T19:03:50.993 DEBUG:boto:Method: GET 2022-09-12T19:03:50.993 DEBUG:boto:Path: /jdpiak-18/?prefix=dummy 2022-09-12T19:03:50.994 DEBUG:boto:Data: 2022-09-12T19:03:50.994 DEBUG:boto:Headers: {} 2022-09-12T19:03:50.995 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:50.995 DEBUG:boto:Port: 8000 2022-09-12T19:03:50.996 DEBUG:boto:Params: {} 2022-09-12T19:03:50.997 DEBUG:boto:Token: None 2022-09-12T19:03:50.997 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:50 GMT /jdpiak-18/ 2022-09-12T19:03:50.998 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uBdR1b/J8z4MPGVqEisFCueQYtE= 2022-09-12T19:03:50.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uBdR1b/J8z4MPGVqEisFCueQYtE=', 'Content-Length': '0'} 2022-09-12T19:03:51.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bd8a2a667da95d7-00631f8296-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.002 DEBUG:boto:b'jdpiak-18dummy1000falsedummy2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:51.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:51.003 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.004 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.005 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.005 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.006 DEBUG:boto:Method: GET 2022-09-12T19:03:51.006 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.007 DEBUG:boto:Data: 2022-09-12T19:03:51.008 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.008 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:51.009 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.009 DEBUG:boto:Params: {} 2022-09-12T19:03:51.010 DEBUG:boto:Token: None 2022-09-12T19:03:51.010 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.011 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KO6bkEELlKbAcI8UYdlMYtAO8gs= 2022-09-12T19:03:51.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KO6bkEELlKbAcI8UYdlMYtAO8gs=', 'Content-Length': '0'} 2022-09-12T19:03:51.014 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3d22be3a739739c-00631f8297-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.015 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.015 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.016 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.016 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.017 DEBUG:boto:Method: GET 2022-09-12T19:03:51.017 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.018 DEBUG:boto:Data: 2022-09-12T19:03:51.018 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.019 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.019 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.020 DEBUG:boto:Params: {} 2022-09-12T19:03:51.021 DEBUG:boto:Token: None 2022-09-12T19:03:51.021 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.022 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KO6bkEELlKbAcI8UYdlMYtAO8gs= 2022-09-12T19:03:51.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KO6bkEELlKbAcI8UYdlMYtAO8gs=', 'Content-Length': '0'} 2022-09-12T19:03:51.025 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b928427c1fabb89a-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.025 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.026 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.026 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.027 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.028 DEBUG:boto:Method: HEAD 2022-09-12T19:03:51.028 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.029 DEBUG:boto:Data: 2022-09-12T19:03:51.029 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.030 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:51.030 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.031 DEBUG:boto:Params: {} 2022-09-12T19:03:51.031 DEBUG:boto:Token: None 2022-09-12T19:03:51.032 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.032 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FbTHdZgJ9Ymc1PXgbTic3UfouQo= 2022-09-12T19:03:51.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FbTHdZgJ9Ymc1PXgbTic3UfouQo=', 'Content-Length': '0'} 2022-09-12T19:03:51.035 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a6c0d918e2f15f6-00631f8297-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.036 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.036 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.037 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.037 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.038 DEBUG:boto:Method: HEAD 2022-09-12T19:03:51.038 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.039 DEBUG:boto:Data: 2022-09-12T19:03:51.039 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.040 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.040 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.041 DEBUG:boto:Params: {} 2022-09-12T19:03:51.042 DEBUG:boto:Token: None 2022-09-12T19:03:51.042 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.043 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FbTHdZgJ9Ymc1PXgbTic3UfouQo= 2022-09-12T19:03:51.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FbTHdZgJ9Ymc1PXgbTic3UfouQo=', 'Content-Length': '0'} 2022-09-12T19:03:51.045 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d9c6bca625ff2b80-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:51.047 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.047 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.048 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.048 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.049 DEBUG:boto:Method: GET 2022-09-12T19:03:51.050 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.050 DEBUG:boto:Data: 2022-09-12T19:03:51.051 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.051 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:51.052 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.053 DEBUG:boto:Params: {} 2022-09-12T19:03:51.053 DEBUG:boto:Token: None 2022-09-12T19:03:51.054 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.054 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KO6bkEELlKbAcI8UYdlMYtAO8gs= 2022-09-12T19:03:51.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KO6bkEELlKbAcI8UYdlMYtAO8gs=', 'Content-Length': '0'} 2022-09-12T19:03:51.057 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e878e99d128b98a-00631f8297-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.058 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.058 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.059 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.059 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.060 DEBUG:boto:Method: GET 2022-09-12T19:03:51.060 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.061 DEBUG:boto:Data: 2022-09-12T19:03:51.062 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.062 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.063 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.063 DEBUG:boto:Params: {} 2022-09-12T19:03:51.064 DEBUG:boto:Token: None 2022-09-12T19:03:51.064 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.065 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KO6bkEELlKbAcI8UYdlMYtAO8gs= 2022-09-12T19:03:51.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KO6bkEELlKbAcI8UYdlMYtAO8gs=', 'Content-Length': '0'} 2022-09-12T19:03:51.068 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000607b98fba3049668-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.069 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:51.069 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:51.070 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:51.070 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:51.071 DEBUG:boto:Method: GET 2022-09-12T19:03:51.071 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:51.072 DEBUG:boto:Data: 2022-09-12T19:03:51.072 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.073 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:51.074 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.074 DEBUG:boto:Params: {} 2022-09-12T19:03:51.075 DEBUG:boto:Token: None 2022-09-12T19:03:51.076 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/ 2022-09-12T19:03:51.076 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo= 2022-09-12T19:03:51.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo=', 'Content-Length': '0'} 2022-09-12T19:03:51.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba47bf878fc27fee-00631f8297-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.080 DEBUG:boto:b'jdpiak-18obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:51.081 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:51.082 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:51.082 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:51.083 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:51.083 DEBUG:boto:Method: GET 2022-09-12T19:03:51.084 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:51.084 DEBUG:boto:Data: 2022-09-12T19:03:51.085 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.085 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.086 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.086 DEBUG:boto:Params: {} 2022-09-12T19:03:51.087 DEBUG:boto:Token: None 2022-09-12T19:03:51.088 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/ 2022-09-12T19:03:51.089 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo= 2022-09-12T19:03:51.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo=', 'Content-Length': '0'} 2022-09-12T19:03:51.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000944c647d7583a86f-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.092 DEBUG:boto:b'jdpiak-18obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:51.093 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.094 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.094 DEBUG:boto:Method: HEAD 2022-09-12T19:03:51.095 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.095 DEBUG:boto:Data: 2022-09-12T19:03:51.096 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.097 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:51.097 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.098 DEBUG:boto:Params: {} 2022-09-12T19:03:51.098 DEBUG:boto:Token: None 2022-09-12T19:03:51.099 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.099 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2rB+H6ZEFSIedEGRK83rVYFKd9c= 2022-09-12T19:03:51.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2rB+H6ZEFSIedEGRK83rVYFKd9c=', 'Content-Length': '0'} 2022-09-12T19:03:51.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dcd8043de0a066f0-00631f8297-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.103 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.104 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.104 DEBUG:boto:Method: HEAD 2022-09-12T19:03:51.105 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.106 DEBUG:boto:Data: 2022-09-12T19:03:51.106 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.107 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.107 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.108 DEBUG:boto:Params: {} 2022-09-12T19:03:51.108 DEBUG:boto:Token: None 2022-09-12T19:03:51.109 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.109 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2rB+H6ZEFSIedEGRK83rVYFKd9c= 2022-09-12T19:03:51.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2rB+H6ZEFSIedEGRK83rVYFKd9c=', 'Content-Length': '0'} 2022-09-12T19:03:51.112 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017b87b22c256db19-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:51.114 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.114 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.115 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.115 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.116 DEBUG:boto:Method: GET 2022-09-12T19:03:51.116 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.117 DEBUG:boto:Data: 2022-09-12T19:03:51.117 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.118 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:51.119 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.119 DEBUG:boto:Params: {} 2022-09-12T19:03:51.120 DEBUG:boto:Token: None 2022-09-12T19:03:51.121 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.121 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KO6bkEELlKbAcI8UYdlMYtAO8gs= 2022-09-12T19:03:51.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KO6bkEELlKbAcI8UYdlMYtAO8gs=', 'Content-Length': '0'} 2022-09-12T19:03:51.124 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000975a3732ea557488-00631f8297-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.166 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.167 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.167 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.168 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.168 DEBUG:boto:Method: GET 2022-09-12T19:03:51.169 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.170 DEBUG:boto:Data: 2022-09-12T19:03:51.170 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.171 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.171 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.172 DEBUG:boto:Params: {} 2022-09-12T19:03:51.173 DEBUG:boto:Token: None 2022-09-12T19:03:51.174 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.174 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KO6bkEELlKbAcI8UYdlMYtAO8gs= 2022-09-12T19:03:51.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KO6bkEELlKbAcI8UYdlMYtAO8gs=', 'Content-Length': '0'} 2022-09-12T19:03:51.177 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7a0399511cbb434-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.178 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:51.178 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:51.179 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:51.180 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:51.180 DEBUG:boto:Method: GET 2022-09-12T19:03:51.181 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:51.182 DEBUG:boto:Data: 2022-09-12T19:03:51.182 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.183 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:51.183 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.184 DEBUG:boto:Params: {} 2022-09-12T19:03:51.184 DEBUG:boto:Token: None 2022-09-12T19:03:51.185 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/ 2022-09-12T19:03:51.185 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo= 2022-09-12T19:03:51.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo=', 'Content-Length': '0'} 2022-09-12T19:03:51.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003654958a7cf9d48f-00631f8297-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.190 DEBUG:boto:b'jdpiak-18obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:51.191 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:51.191 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:51.192 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:51.192 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:51.193 DEBUG:boto:Method: GET 2022-09-12T19:03:51.193 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:51.194 DEBUG:boto:Data: 2022-09-12T19:03:51.194 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.195 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.195 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.196 DEBUG:boto:Params: {} 2022-09-12T19:03:51.196 DEBUG:boto:Token: None 2022-09-12T19:03:51.197 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/ 2022-09-12T19:03:51.198 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo= 2022-09-12T19:03:51.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo=', 'Content-Length': '0'} 2022-09-12T19:03:51.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088ffdc4d87b3aef8-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.202 DEBUG:boto:b'jdpiak-18obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:51.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:51.203 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:51.203 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:51.204 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.204 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.205 DEBUG:boto:Method: GET 2022-09-12T19:03:51.206 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.206 DEBUG:boto:Data: 2022-09-12T19:03:51.207 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.207 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:51.208 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.208 DEBUG:boto:Params: {} 2022-09-12T19:03:51.209 DEBUG:boto:Token: None 2022-09-12T19:03:51.209 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.210 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c1Lf5vQUNNqZ7HhOHYfzouNwMwM= 2022-09-12T19:03:51.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c1Lf5vQUNNqZ7HhOHYfzouNwMwM=', 'Content-Length': '0'} 2022-09-12T19:03:51.212 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a67b1f0ea07e1da5-00631f8297-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.213 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:51.214 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:51.214 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.215 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.215 DEBUG:boto:Method: GET 2022-09-12T19:03:51.216 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.216 DEBUG:boto:Data: 2022-09-12T19:03:51.217 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.218 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.218 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.219 DEBUG:boto:Params: {} 2022-09-12T19:03:51.219 DEBUG:boto:Token: None 2022-09-12T19:03:51.220 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.220 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c1Lf5vQUNNqZ7HhOHYfzouNwMwM= 2022-09-12T19:03:51.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c1Lf5vQUNNqZ7HhOHYfzouNwMwM=', 'Content-Length': '0'} 2022-09-12T19:03:51.223 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4ded747309645d3-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.224 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:51.224 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:51.225 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.225 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.226 DEBUG:boto:Method: HEAD 2022-09-12T19:03:51.226 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.227 DEBUG:boto:Data: 2022-09-12T19:03:51.227 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.228 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:51.228 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.229 DEBUG:boto:Params: {} 2022-09-12T19:03:51.229 DEBUG:boto:Token: None 2022-09-12T19:03:51.230 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.230 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gg7XEAK7o6I+5h70lmXNYAxGh3A= 2022-09-12T19:03:51.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gg7XEAK7o6I+5h70lmXNYAxGh3A=', 'Content-Length': '0'} 2022-09-12T19:03:51.233 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000089ff8dbe29698c03-00631f8297-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.233 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:51.234 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:51.234 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.235 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.235 DEBUG:boto:Method: HEAD 2022-09-12T19:03:51.236 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.236 DEBUG:boto:Data: 2022-09-12T19:03:51.237 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.237 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.238 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.238 DEBUG:boto:Params: {} 2022-09-12T19:03:51.239 DEBUG:boto:Token: None 2022-09-12T19:03:51.240 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.240 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gg7XEAK7o6I+5h70lmXNYAxGh3A= 2022-09-12T19:03:51.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gg7XEAK7o6I+5h70lmXNYAxGh3A=', 'Content-Length': '0'} 2022-09-12T19:03:51.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5debcb2ecf4dc12-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:51.244 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:51.244 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:51.245 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.245 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.246 DEBUG:boto:Method: GET 2022-09-12T19:03:51.246 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.247 DEBUG:boto:Data: 2022-09-12T19:03:51.247 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.248 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:51.248 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.249 DEBUG:boto:Params: {} 2022-09-12T19:03:51.250 DEBUG:boto:Token: None 2022-09-12T19:03:51.250 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.251 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c1Lf5vQUNNqZ7HhOHYfzouNwMwM= 2022-09-12T19:03:51.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c1Lf5vQUNNqZ7HhOHYfzouNwMwM=', 'Content-Length': '0'} 2022-09-12T19:03:51.253 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052db2fc79cd041c0-00631f8297-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.254 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:51.255 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:51.255 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.256 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.256 DEBUG:boto:Method: GET 2022-09-12T19:03:51.257 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.257 DEBUG:boto:Data: 2022-09-12T19:03:51.258 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.258 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.259 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.259 DEBUG:boto:Params: {} 2022-09-12T19:03:51.260 DEBUG:boto:Token: None 2022-09-12T19:03:51.261 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.261 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c1Lf5vQUNNqZ7HhOHYfzouNwMwM= 2022-09-12T19:03:51.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c1Lf5vQUNNqZ7HhOHYfzouNwMwM=', 'Content-Length': '0'} 2022-09-12T19:03:51.264 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c6289d6a1064282-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.265 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:51.265 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:51.266 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:51.266 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:51.267 DEBUG:boto:Method: GET 2022-09-12T19:03:51.267 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:51.268 DEBUG:boto:Data: 2022-09-12T19:03:51.268 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.269 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:51.269 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.270 DEBUG:boto:Params: {} 2022-09-12T19:03:51.271 DEBUG:boto:Token: None 2022-09-12T19:03:51.271 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/ 2022-09-12T19:03:51.272 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo= 2022-09-12T19:03:51.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo=', 'Content-Length': '0'} 2022-09-12T19:03:51.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be46c8f3394d8140-00631f8297-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.276 DEBUG:boto:b'jdpiak-18obj-test-zone21000falseobj-test-zone22022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:51.277 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:51.277 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:51.278 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:51.278 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:51.279 DEBUG:boto:Method: GET 2022-09-12T19:03:51.279 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:51.280 DEBUG:boto:Data: 2022-09-12T19:03:51.280 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.281 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.281 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.282 DEBUG:boto:Params: {} 2022-09-12T19:03:51.283 DEBUG:boto:Token: None 2022-09-12T19:03:51.283 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/ 2022-09-12T19:03:51.284 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo= 2022-09-12T19:03:51.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo=', 'Content-Length': '0'} 2022-09-12T19:03:51.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008059199cb1decbee-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.287 DEBUG:boto:b'jdpiak-18obj-test-zone21000falseobj-test-zone22022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:51.288 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:51.289 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:51.289 DEBUG:boto:Method: HEAD 2022-09-12T19:03:51.290 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2 2022-09-12T19:03:51.290 DEBUG:boto:Data: 2022-09-12T19:03:51.291 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.291 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:51.292 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.292 DEBUG:boto:Params: {} 2022-09-12T19:03:51.293 DEBUG:boto:Token: None 2022-09-12T19:03:51.293 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone2 2022-09-12T19:03:51.294 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fkq5gDARIb01EvhACwv6l5VefFg= 2022-09-12T19:03:51.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fkq5gDARIb01EvhACwv6l5VefFg=', 'Content-Length': '0'} 2022-09-12T19:03:51.297 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af6b0a3c7b30ac1c-00631f8297-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.298 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:51.298 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:51.299 DEBUG:boto:Method: HEAD 2022-09-12T19:03:51.299 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2 2022-09-12T19:03:51.300 DEBUG:boto:Data: 2022-09-12T19:03:51.300 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.301 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.301 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.302 DEBUG:boto:Params: {} 2022-09-12T19:03:51.302 DEBUG:boto:Token: None 2022-09-12T19:03:51.303 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone2 2022-09-12T19:03:51.304 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fkq5gDARIb01EvhACwv6l5VefFg= 2022-09-12T19:03:51.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fkq5gDARIb01EvhACwv6l5VefFg=', 'Content-Length': '0'} 2022-09-12T19:03:51.307 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008bd9204e884ddd2e-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:51.308 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:51.309 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:51.309 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.310 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.310 DEBUG:boto:Method: GET 2022-09-12T19:03:51.311 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.311 DEBUG:boto:Data: 2022-09-12T19:03:51.312 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.313 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:51.313 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.314 DEBUG:boto:Params: {} 2022-09-12T19:03:51.314 DEBUG:boto:Token: None 2022-09-12T19:03:51.315 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.315 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c1Lf5vQUNNqZ7HhOHYfzouNwMwM= 2022-09-12T19:03:51.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c1Lf5vQUNNqZ7HhOHYfzouNwMwM=', 'Content-Length': '0'} 2022-09-12T19:03:51.318 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008dcdaf693e3b97e-00631f8297-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.319 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:51.319 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:51.320 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.320 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.321 DEBUG:boto:Method: GET 2022-09-12T19:03:51.321 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.322 DEBUG:boto:Data: 2022-09-12T19:03:51.322 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.323 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.324 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.324 DEBUG:boto:Params: {} 2022-09-12T19:03:51.325 DEBUG:boto:Token: None 2022-09-12T19:03:51.325 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:51.326 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c1Lf5vQUNNqZ7HhOHYfzouNwMwM= 2022-09-12T19:03:51.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c1Lf5vQUNNqZ7HhOHYfzouNwMwM=', 'Content-Length': '0'} 2022-09-12T19:03:51.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a9e23e99e188330-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.370 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:51.371 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:51.371 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:51.372 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:51.372 DEBUG:boto:Method: GET 2022-09-12T19:03:51.373 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:51.373 DEBUG:boto:Data: 2022-09-12T19:03:51.374 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.375 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:51.375 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.376 DEBUG:boto:Params: {} 2022-09-12T19:03:51.376 DEBUG:boto:Token: None 2022-09-12T19:03:51.377 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/ 2022-09-12T19:03:51.378 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo= 2022-09-12T19:03:51.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo=', 'Content-Length': '0'} 2022-09-12T19:03:51.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000278968f6be6f684-00631f8297-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.382 DEBUG:boto:b'jdpiak-18obj-test-zone21000falseobj-test-zone22022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:51.382 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:51.383 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:51.384 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:51.384 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:51.385 DEBUG:boto:Method: GET 2022-09-12T19:03:51.386 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:51.386 DEBUG:boto:Data: 2022-09-12T19:03:51.387 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.387 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.388 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.388 DEBUG:boto:Params: {} 2022-09-12T19:03:51.389 DEBUG:boto:Token: None 2022-09-12T19:03:51.390 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/ 2022-09-12T19:03:51.390 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo= 2022-09-12T19:03:51.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo=', 'Content-Length': '0'} 2022-09-12T19:03:51.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079a3e72588aac65a-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.394 DEBUG:boto:b'jdpiak-18obj-test-zone21000falseobj-test-zone22022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:51.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:51.396 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:51.396 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:51.397 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.397 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.398 DEBUG:boto:Method: GET 2022-09-12T19:03:51.399 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.399 DEBUG:boto:Data: 2022-09-12T19:03:51.400 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.400 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:51.401 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.401 DEBUG:boto:Params: {} 2022-09-12T19:03:51.402 DEBUG:boto:Token: None 2022-09-12T19:03:51.402 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.403 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3xycnzqTXCyR3gDZoc4H9Zk2TVQ= 2022-09-12T19:03:51.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3xycnzqTXCyR3gDZoc4H9Zk2TVQ=', 'Content-Length': '0'} 2022-09-12T19:03:51.406 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca66776670bb573a-00631f8297-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.407 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:51.407 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:51.408 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.408 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.409 DEBUG:boto:Method: GET 2022-09-12T19:03:51.409 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.410 DEBUG:boto:Data: 2022-09-12T19:03:51.410 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.411 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.411 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.412 DEBUG:boto:Params: {} 2022-09-12T19:03:51.413 DEBUG:boto:Token: None 2022-09-12T19:03:51.413 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.414 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3xycnzqTXCyR3gDZoc4H9Zk2TVQ= 2022-09-12T19:03:51.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3xycnzqTXCyR3gDZoc4H9Zk2TVQ=', 'Content-Length': '0'} 2022-09-12T19:03:51.417 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eba0da6d6c48effb-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.417 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:51.418 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:51.418 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.419 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.419 DEBUG:boto:Method: HEAD 2022-09-12T19:03:51.420 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.420 DEBUG:boto:Data: 2022-09-12T19:03:51.421 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.422 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:51.422 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.423 DEBUG:boto:Params: {} 2022-09-12T19:03:51.423 DEBUG:boto:Token: None 2022-09-12T19:03:51.424 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.424 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mmGgdkUJVo6yu4o3Z7KsLRqMVvU= 2022-09-12T19:03:51.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mmGgdkUJVo6yu4o3Z7KsLRqMVvU=', 'Content-Length': '0'} 2022-09-12T19:03:51.427 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c0f7dcde9dd22c1-00631f8297-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.428 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:51.428 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:51.429 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.429 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.430 DEBUG:boto:Method: HEAD 2022-09-12T19:03:51.430 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.431 DEBUG:boto:Data: 2022-09-12T19:03:51.431 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.432 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.432 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.433 DEBUG:boto:Params: {} 2022-09-12T19:03:51.433 DEBUG:boto:Token: None 2022-09-12T19:03:51.434 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.435 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mmGgdkUJVo6yu4o3Z7KsLRqMVvU= 2022-09-12T19:03:51.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mmGgdkUJVo6yu4o3Z7KsLRqMVvU=', 'Content-Length': '0'} 2022-09-12T19:03:51.437 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b07455aa09798353-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:51.439 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:51.439 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:51.440 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.440 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.441 DEBUG:boto:Method: GET 2022-09-12T19:03:51.441 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.442 DEBUG:boto:Data: 2022-09-12T19:03:51.442 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.443 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:51.443 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.444 DEBUG:boto:Params: {} 2022-09-12T19:03:51.445 DEBUG:boto:Token: None 2022-09-12T19:03:51.445 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.446 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3xycnzqTXCyR3gDZoc4H9Zk2TVQ= 2022-09-12T19:03:51.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3xycnzqTXCyR3gDZoc4H9Zk2TVQ=', 'Content-Length': '0'} 2022-09-12T19:03:51.449 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b13e7911b405b68d-00631f8297-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.449 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:51.450 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:51.451 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.451 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.452 DEBUG:boto:Method: GET 2022-09-12T19:03:51.452 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.453 DEBUG:boto:Data: 2022-09-12T19:03:51.453 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.455 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.455 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.456 DEBUG:boto:Params: {} 2022-09-12T19:03:51.456 DEBUG:boto:Token: None 2022-09-12T19:03:51.457 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.458 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3xycnzqTXCyR3gDZoc4H9Zk2TVQ= 2022-09-12T19:03:51.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3xycnzqTXCyR3gDZoc4H9Zk2TVQ=', 'Content-Length': '0'} 2022-09-12T19:03:51.460 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c29a1ce71d6d652-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.461 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:51.462 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:51.463 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:51.463 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:51.464 DEBUG:boto:Method: GET 2022-09-12T19:03:51.464 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:51.465 DEBUG:boto:Data: 2022-09-12T19:03:51.465 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.466 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:51.466 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.467 DEBUG:boto:Params: {} 2022-09-12T19:03:51.467 DEBUG:boto:Token: None 2022-09-12T19:03:51.468 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/ 2022-09-12T19:03:51.469 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo= 2022-09-12T19:03:51.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo=', 'Content-Length': '0'} 2022-09-12T19:03:51.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0b547ef9839c04c-00631f8297-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.472 DEBUG:boto:b'jdpiak-18obj-test-zone31000falseobj-test-zone32022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:51.473 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:51.474 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:51.474 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:51.475 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:51.475 DEBUG:boto:Method: GET 2022-09-12T19:03:51.476 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:51.476 DEBUG:boto:Data: 2022-09-12T19:03:51.477 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.477 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.478 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.478 DEBUG:boto:Params: {} 2022-09-12T19:03:51.479 DEBUG:boto:Token: None 2022-09-12T19:03:51.480 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/ 2022-09-12T19:03:51.480 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo= 2022-09-12T19:03:51.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo=', 'Content-Length': '0'} 2022-09-12T19:03:51.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c860b2a867633c3b-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.484 DEBUG:boto:b'jdpiak-18obj-test-zone31000falseobj-test-zone32022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:51.485 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:51.486 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:51.486 DEBUG:boto:Method: HEAD 2022-09-12T19:03:51.487 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3 2022-09-12T19:03:51.487 DEBUG:boto:Data: 2022-09-12T19:03:51.488 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.488 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:51.489 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.489 DEBUG:boto:Params: {} 2022-09-12T19:03:51.490 DEBUG:boto:Token: None 2022-09-12T19:03:51.490 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone3 2022-09-12T19:03:51.491 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AGPukExqfoqNNPIZTyTTWRPj230= 2022-09-12T19:03:51.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AGPukExqfoqNNPIZTyTTWRPj230=', 'Content-Length': '0'} 2022-09-12T19:03:51.494 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004bb01aa67e8c3bdc-00631f8297-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.495 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:51.495 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:51.496 DEBUG:boto:Method: HEAD 2022-09-12T19:03:51.496 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3 2022-09-12T19:03:51.497 DEBUG:boto:Data: 2022-09-12T19:03:51.497 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.498 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.498 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.499 DEBUG:boto:Params: {} 2022-09-12T19:03:51.499 DEBUG:boto:Token: None 2022-09-12T19:03:51.500 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone3 2022-09-12T19:03:51.501 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AGPukExqfoqNNPIZTyTTWRPj230= 2022-09-12T19:03:51.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AGPukExqfoqNNPIZTyTTWRPj230=', 'Content-Length': '0'} 2022-09-12T19:03:51.503 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6c2d6b58834c036-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:51.505 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:51.505 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:51.506 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.506 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.507 DEBUG:boto:Method: GET 2022-09-12T19:03:51.508 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.508 DEBUG:boto:Data: 2022-09-12T19:03:51.509 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.509 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:51.510 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.510 DEBUG:boto:Params: {} 2022-09-12T19:03:51.511 DEBUG:boto:Token: None 2022-09-12T19:03:51.512 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.512 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3xycnzqTXCyR3gDZoc4H9Zk2TVQ= 2022-09-12T19:03:51.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3xycnzqTXCyR3gDZoc4H9Zk2TVQ=', 'Content-Length': '0'} 2022-09-12T19:03:51.515 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b1124bd18c401e1-00631f8297-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.516 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:51.516 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:51.517 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.517 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.518 DEBUG:boto:Method: GET 2022-09-12T19:03:51.519 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.519 DEBUG:boto:Data: 2022-09-12T19:03:51.520 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.520 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.521 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.521 DEBUG:boto:Params: {} 2022-09-12T19:03:51.522 DEBUG:boto:Token: None 2022-09-12T19:03:51.522 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:51.523 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3xycnzqTXCyR3gDZoc4H9Zk2TVQ= 2022-09-12T19:03:51.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3xycnzqTXCyR3gDZoc4H9Zk2TVQ=', 'Content-Length': '0'} 2022-09-12T19:03:51.525 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e11962640eb355ef-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.566 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:51.567 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:51.567 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:51.568 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:51.568 DEBUG:boto:Method: GET 2022-09-12T19:03:51.569 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:51.569 DEBUG:boto:Data: 2022-09-12T19:03:51.570 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.571 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:51.571 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.572 DEBUG:boto:Params: {} 2022-09-12T19:03:51.572 DEBUG:boto:Token: None 2022-09-12T19:03:51.573 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/ 2022-09-12T19:03:51.573 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo= 2022-09-12T19:03:51.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo=', 'Content-Length': '0'} 2022-09-12T19:03:51.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000487709d5dc726923-00631f8297-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.577 DEBUG:boto:b'jdpiak-18obj-test-zone31000falseobj-test-zone32022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:51.578 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:51.579 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:51.579 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:51.580 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:51.580 DEBUG:boto:Method: GET 2022-09-12T19:03:51.581 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:51.582 DEBUG:boto:Data: 2022-09-12T19:03:51.582 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.583 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.583 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.584 DEBUG:boto:Params: {} 2022-09-12T19:03:51.585 DEBUG:boto:Token: None 2022-09-12T19:03:51.585 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/ 2022-09-12T19:03:51.586 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo= 2022-09-12T19:03:51.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo=', 'Content-Length': '0'} 2022-09-12T19:03:51.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff978cb26ed1db67-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.591 DEBUG:boto:b'jdpiak-18obj-test-zone31000falseobj-test-zone32022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:51.592 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-18 zones={test-zone2, test-zone1} 2022-09-12T19:03:51.593 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-18 zones={test-zone3, test-zone1} 2022-09-12T19:03:51.594 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:51.594 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:51.595 DEBUG:boto:Method: HEAD 2022-09-12T19:03:51.596 DEBUG:boto:Path: /jdpiak-18/ 2022-09-12T19:03:51.597 DEBUG:boto:Data: 2022-09-12T19:03:51.598 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.598 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:51.599 DEBUG:boto:Port: 8001 2022-09-12T19:03:51.600 DEBUG:boto:Params: {} 2022-09-12T19:03:51.600 DEBUG:boto:Token: None 2022-09-12T19:03:51.601 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/ 2022-09-12T19:03:51.602 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:raN4T32nbe7EN8lbikWqNgN1mVw= 2022-09-12T19:03:51.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:raN4T32nbe7EN8lbikWqNgN1mVw=', 'Content-Length': '0'} 2022-09-12T19:03:51.605 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035beeb6bbba90ba5-00631f8297-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.606 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:51.606 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:51.607 DEBUG:boto:Method: HEAD 2022-09-12T19:03:51.608 DEBUG:boto:Path: /jdpiak-18/ 2022-09-12T19:03:51.608 DEBUG:boto:Data: 2022-09-12T19:03:51.609 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.609 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.610 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.611 DEBUG:boto:Params: {} 2022-09-12T19:03:51.612 DEBUG:boto:Token: None 2022-09-12T19:03:51.613 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/ 2022-09-12T19:03:51.614 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:raN4T32nbe7EN8lbikWqNgN1mVw= 2022-09-12T19:03:51.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:raN4T32nbe7EN8lbikWqNgN1mVw=', 'Content-Length': '0'} 2022-09-12T19:03:51.618 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eeb55ca4ced8fdaf-00631f8297-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.619 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:03:51.620 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:51.620 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:51.621 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:51.621 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:51.622 DEBUG:boto:Method: GET 2022-09-12T19:03:51.623 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:51.623 DEBUG:boto:Data: 2022-09-12T19:03:51.624 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.624 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:51.625 DEBUG:boto:Port: 8001 2022-09-12T19:03:51.625 DEBUG:boto:Params: {} 2022-09-12T19:03:51.626 DEBUG:boto:Token: None 2022-09-12T19:03:51.627 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/?versions 2022-09-12T19:03:51.627 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5P/u274W+4kf5H617PcNQH5GHjY= 2022-09-12T19:03:51.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5P/u274W+4kf5H617PcNQH5GHjY=', 'Content-Length': '0'} 2022-09-12T19:03:51.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba97dbe5a56f0f12-00631f8297-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.632 DEBUG:boto:b'jdpiak-18999falsedummyqUufSwmbKETbT7yNOH8sL0A4tddUbNOtrue2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ToUtj2XoS.tz3goIbvjFwiigMyzGOpLtrue2022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2DPksivOWVYhJDrEwQqHYV7wK1Ul1JkWtrue2022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Ztrue2022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:51.633 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:03:51.633 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:03:51.634 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:03:51.634 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:03:51.635 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:03:51.635 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:51.636 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:51.636 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:51.637 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:51.637 DEBUG:boto:Method: GET 2022-09-12T19:03:51.638 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:51.638 DEBUG:boto:Data: 2022-09-12T19:03:51.639 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.640 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.640 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.641 DEBUG:boto:Params: {} 2022-09-12T19:03:51.642 DEBUG:boto:Token: None 2022-09-12T19:03:51.642 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/?versions 2022-09-12T19:03:51.643 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5P/u274W+4kf5H617PcNQH5GHjY= 2022-09-12T19:03:51.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5P/u274W+4kf5H617PcNQH5GHjY=', 'Content-Length': '0'} 2022-09-12T19:03:51.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1399b85b53f0810-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.647 DEBUG:boto:b'jdpiak-18999falsedummyqUufSwmbKETbT7yNOH8sL0A4tddUbNOtrue2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ToUtj2XoS.tz3goIbvjFwiigMyzGOpLtrue2022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2DPksivOWVYhJDrEwQqHYV7wK1Ul1JkWtrue2022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Ztrue2022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:51.648 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:03:51.649 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:03:51.649 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:03:51.650 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:03:51.650 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:51.651 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:51.652 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:51.653 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:51.654 DEBUG:boto:Method: GET 2022-09-12T19:03:51.655 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:51.655 DEBUG:boto:Data: 2022-09-12T19:03:51.656 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.656 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:51.657 DEBUG:boto:Port: 8001 2022-09-12T19:03:51.657 DEBUG:boto:Params: {} 2022-09-12T19:03:51.658 DEBUG:boto:Token: None 2022-09-12T19:03:51.659 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/?versions 2022-09-12T19:03:51.659 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5P/u274W+4kf5H617PcNQH5GHjY= 2022-09-12T19:03:51.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5P/u274W+4kf5H617PcNQH5GHjY=', 'Content-Length': '0'} 2022-09-12T19:03:51.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e16563b009559a35-00631f8297-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.664 DEBUG:boto:b'jdpiak-18999falsedummyqUufSwmbKETbT7yNOH8sL0A4tddUbNOtrue2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ToUtj2XoS.tz3goIbvjFwiigMyzGOpLtrue2022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2DPksivOWVYhJDrEwQqHYV7wK1Ul1JkWtrue2022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Ztrue2022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:51.665 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:51.665 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:51.666 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:51.666 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:51.667 DEBUG:boto:Method: GET 2022-09-12T19:03:51.667 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:51.668 DEBUG:boto:Data: 2022-09-12T19:03:51.669 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.669 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.670 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.670 DEBUG:boto:Params: {} 2022-09-12T19:03:51.671 DEBUG:boto:Token: None 2022-09-12T19:03:51.672 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/?versions 2022-09-12T19:03:51.672 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5P/u274W+4kf5H617PcNQH5GHjY= 2022-09-12T19:03:51.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5P/u274W+4kf5H617PcNQH5GHjY=', 'Content-Length': '0'} 2022-09-12T19:03:51.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eea93426929c8d91-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.676 DEBUG:boto:b'jdpiak-18999falsedummyqUufSwmbKETbT7yNOH8sL0A4tddUbNOtrue2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ToUtj2XoS.tz3goIbvjFwiigMyzGOpLtrue2022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2DPksivOWVYhJDrEwQqHYV7wK1Ul1JkWtrue2022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Ztrue2022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:51.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:51.678 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:51.678 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:51.679 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.679 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.680 DEBUG:boto:Method: GET 2022-09-12T19:03:51.680 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.681 DEBUG:boto:Data: 2022-09-12T19:03:51.682 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.682 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:51.683 DEBUG:boto:Port: 8001 2022-09-12T19:03:51.683 DEBUG:boto:Params: {} 2022-09-12T19:03:51.684 DEBUG:boto:Token: None 2022-09-12T19:03:51.684 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.685 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hsuZF5d4gTh5M+kY9/KIIdbzvBM= 2022-09-12T19:03:51.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hsuZF5d4gTh5M+kY9/KIIdbzvBM=', 'Content-Length': '0'} 2022-09-12T19:03:51.687 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000357f32a10dadbfb5-00631f8297-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.688 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:51.689 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:51.689 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.690 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.690 DEBUG:boto:Method: GET 2022-09-12T19:03:51.691 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.691 DEBUG:boto:Data: 2022-09-12T19:03:51.692 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.692 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.693 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.693 DEBUG:boto:Params: {} 2022-09-12T19:03:51.694 DEBUG:boto:Token: None 2022-09-12T19:03:51.694 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.695 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hsuZF5d4gTh5M+kY9/KIIdbzvBM= 2022-09-12T19:03:51.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hsuZF5d4gTh5M+kY9/KIIdbzvBM=', 'Content-Length': '0'} 2022-09-12T19:03:51.698 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002c928d8ae0cbbc32-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.698 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:51.699 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:51.699 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.700 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.700 DEBUG:boto:Method: HEAD 2022-09-12T19:03:51.701 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.701 DEBUG:boto:Data: 2022-09-12T19:03:51.702 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.702 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:51.703 DEBUG:boto:Port: 8001 2022-09-12T19:03:51.703 DEBUG:boto:Params: {} 2022-09-12T19:03:51.704 DEBUG:boto:Token: None 2022-09-12T19:03:51.704 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.705 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EK9FTFySIVUc8WUvCJH52D2pro8= 2022-09-12T19:03:51.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EK9FTFySIVUc8WUvCJH52D2pro8=', 'Content-Length': '0'} 2022-09-12T19:03:51.707 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000083999f58a6f4ab9c-00631f8297-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.708 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:51.709 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:51.709 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.710 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.710 DEBUG:boto:Method: HEAD 2022-09-12T19:03:51.711 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.711 DEBUG:boto:Data: 2022-09-12T19:03:51.712 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.712 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.713 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.713 DEBUG:boto:Params: {} 2022-09-12T19:03:51.714 DEBUG:boto:Token: None 2022-09-12T19:03:51.714 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.715 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EK9FTFySIVUc8WUvCJH52D2pro8= 2022-09-12T19:03:51.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EK9FTFySIVUc8WUvCJH52D2pro8=', 'Content-Length': '0'} 2022-09-12T19:03:51.717 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000466d6a05d8e2c4ec-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:51.718 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:51.719 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:51.719 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.720 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.720 DEBUG:boto:Method: GET 2022-09-12T19:03:51.721 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.721 DEBUG:boto:Data: 2022-09-12T19:03:51.722 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.723 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:51.723 DEBUG:boto:Port: 8001 2022-09-12T19:03:51.724 DEBUG:boto:Params: {} 2022-09-12T19:03:51.724 DEBUG:boto:Token: None 2022-09-12T19:03:51.725 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.726 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hsuZF5d4gTh5M+kY9/KIIdbzvBM= 2022-09-12T19:03:51.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hsuZF5d4gTh5M+kY9/KIIdbzvBM=', 'Content-Length': '0'} 2022-09-12T19:03:51.728 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000021eac21eef325e66-00631f8297-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.729 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:51.730 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:51.730 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.731 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.731 DEBUG:boto:Method: GET 2022-09-12T19:03:51.732 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.732 DEBUG:boto:Data: 2022-09-12T19:03:51.733 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.733 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.734 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.734 DEBUG:boto:Params: {} 2022-09-12T19:03:51.735 DEBUG:boto:Token: None 2022-09-12T19:03:51.736 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.736 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hsuZF5d4gTh5M+kY9/KIIdbzvBM= 2022-09-12T19:03:51.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hsuZF5d4gTh5M+kY9/KIIdbzvBM=', 'Content-Length': '0'} 2022-09-12T19:03:51.739 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000046fda5bed123b3fa-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.740 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:51.740 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:51.741 DEBUG:boto:path=/jdpiak-18/?prefix=dummy 2022-09-12T19:03:51.741 DEBUG:boto:auth_path=/jdpiak-18/?prefix=dummy 2022-09-12T19:03:51.742 DEBUG:boto:Method: GET 2022-09-12T19:03:51.742 DEBUG:boto:Path: /jdpiak-18/?prefix=dummy 2022-09-12T19:03:51.743 DEBUG:boto:Data: 2022-09-12T19:03:51.743 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.744 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:51.744 DEBUG:boto:Port: 8001 2022-09-12T19:03:51.745 DEBUG:boto:Params: {} 2022-09-12T19:03:51.745 DEBUG:boto:Token: None 2022-09-12T19:03:51.746 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/ 2022-09-12T19:03:51.747 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo= 2022-09-12T19:03:51.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo=', 'Content-Length': '0'} 2022-09-12T19:03:51.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000675d0aa87d5b0079-00631f8297-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.790 DEBUG:boto:b'jdpiak-18dummy1000falsedummy2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:51.791 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:51.792 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:51.792 DEBUG:boto:path=/jdpiak-18/?prefix=dummy 2022-09-12T19:03:51.793 DEBUG:boto:auth_path=/jdpiak-18/?prefix=dummy 2022-09-12T19:03:51.794 DEBUG:boto:Method: GET 2022-09-12T19:03:51.794 DEBUG:boto:Path: /jdpiak-18/?prefix=dummy 2022-09-12T19:03:51.795 DEBUG:boto:Data: 2022-09-12T19:03:51.795 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.796 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.796 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.797 DEBUG:boto:Params: {} 2022-09-12T19:03:51.797 DEBUG:boto:Token: None 2022-09-12T19:03:51.798 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/ 2022-09-12T19:03:51.799 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo= 2022-09-12T19:03:51.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo=', 'Content-Length': '0'} 2022-09-12T19:03:51.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008abd57bafea266e8-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.803 DEBUG:boto:b'jdpiak-18dummy1000falsedummy2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:51.804 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:51.805 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:51.805 DEBUG:boto:Method: HEAD 2022-09-12T19:03:51.806 DEBUG:boto:Path: /jdpiak-18/dummy 2022-09-12T19:03:51.807 DEBUG:boto:Data: 2022-09-12T19:03:51.808 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.808 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:51.809 DEBUG:boto:Port: 8001 2022-09-12T19:03:51.810 DEBUG:boto:Params: {} 2022-09-12T19:03:51.811 DEBUG:boto:Token: None 2022-09-12T19:03:51.811 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/dummy 2022-09-12T19:03:51.812 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1FAvv8NWCDGwmG5GvuHVulBjZWE= 2022-09-12T19:03:51.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1FAvv8NWCDGwmG5GvuHVulBjZWE=', 'Content-Length': '0'} 2022-09-12T19:03:51.816 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ede9e10ef3c98182-00631f8297-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.817 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:51.817 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:51.818 DEBUG:boto:Method: HEAD 2022-09-12T19:03:51.818 DEBUG:boto:Path: /jdpiak-18/dummy 2022-09-12T19:03:51.819 DEBUG:boto:Data: 2022-09-12T19:03:51.819 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.820 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.820 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.821 DEBUG:boto:Params: {} 2022-09-12T19:03:51.821 DEBUG:boto:Token: None 2022-09-12T19:03:51.822 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/dummy 2022-09-12T19:03:51.823 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1FAvv8NWCDGwmG5GvuHVulBjZWE= 2022-09-12T19:03:51.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1FAvv8NWCDGwmG5GvuHVulBjZWE=', 'Content-Length': '0'} 2022-09-12T19:03:51.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d56e896d424a2d91-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:51.827 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:51.828 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:51.829 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.829 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.830 DEBUG:boto:Method: GET 2022-09-12T19:03:51.831 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.832 DEBUG:boto:Data: 2022-09-12T19:03:51.833 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.833 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:51.834 DEBUG:boto:Port: 8001 2022-09-12T19:03:51.835 DEBUG:boto:Params: {} 2022-09-12T19:03:51.836 DEBUG:boto:Token: None 2022-09-12T19:03:51.837 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.838 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hsuZF5d4gTh5M+kY9/KIIdbzvBM= 2022-09-12T19:03:51.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hsuZF5d4gTh5M+kY9/KIIdbzvBM=', 'Content-Length': '0'} 2022-09-12T19:03:51.841 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000df3eb6ed9fc02047-00631f8297-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.842 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:51.843 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:51.843 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.844 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.845 DEBUG:boto:Method: GET 2022-09-12T19:03:51.845 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.846 DEBUG:boto:Data: 2022-09-12T19:03:51.847 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.848 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.848 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.849 DEBUG:boto:Params: {} 2022-09-12T19:03:51.850 DEBUG:boto:Token: None 2022-09-12T19:03:51.851 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:51.852 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hsuZF5d4gTh5M+kY9/KIIdbzvBM= 2022-09-12T19:03:51.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hsuZF5d4gTh5M+kY9/KIIdbzvBM=', 'Content-Length': '0'} 2022-09-12T19:03:51.855 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007628f5d49da48e5e-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.855 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:51.856 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:51.856 DEBUG:boto:path=/jdpiak-18/?prefix=dummy 2022-09-12T19:03:51.857 DEBUG:boto:auth_path=/jdpiak-18/?prefix=dummy 2022-09-12T19:03:51.857 DEBUG:boto:Method: GET 2022-09-12T19:03:51.858 DEBUG:boto:Path: /jdpiak-18/?prefix=dummy 2022-09-12T19:03:51.859 DEBUG:boto:Data: 2022-09-12T19:03:51.859 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.860 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:51.860 DEBUG:boto:Port: 8001 2022-09-12T19:03:51.861 DEBUG:boto:Params: {} 2022-09-12T19:03:51.861 DEBUG:boto:Token: None 2022-09-12T19:03:51.862 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/ 2022-09-12T19:03:51.863 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo= 2022-09-12T19:03:51.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo=', 'Content-Length': '0'} 2022-09-12T19:03:51.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8735694b6a8c4fe-00631f8297-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.867 DEBUG:boto:b'jdpiak-18dummy1000falsedummy2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:51.868 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:51.869 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:51.869 DEBUG:boto:path=/jdpiak-18/?prefix=dummy 2022-09-12T19:03:51.870 DEBUG:boto:auth_path=/jdpiak-18/?prefix=dummy 2022-09-12T19:03:51.870 DEBUG:boto:Method: GET 2022-09-12T19:03:51.871 DEBUG:boto:Path: /jdpiak-18/?prefix=dummy 2022-09-12T19:03:51.871 DEBUG:boto:Data: 2022-09-12T19:03:51.872 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.872 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.873 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.873 DEBUG:boto:Params: {} 2022-09-12T19:03:51.874 DEBUG:boto:Token: None 2022-09-12T19:03:51.875 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/ 2022-09-12T19:03:51.875 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo= 2022-09-12T19:03:51.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo=', 'Content-Length': '0'} 2022-09-12T19:03:51.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046dd4906b856bea9-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.926 DEBUG:boto:b'jdpiak-18dummy1000falsedummy2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:51.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:51.928 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.928 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.929 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.929 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.930 DEBUG:boto:Method: GET 2022-09-12T19:03:51.930 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.931 DEBUG:boto:Data: 2022-09-12T19:03:51.931 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.932 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:51.932 DEBUG:boto:Port: 8001 2022-09-12T19:03:51.933 DEBUG:boto:Params: {} 2022-09-12T19:03:51.933 DEBUG:boto:Token: None 2022-09-12T19:03:51.934 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.935 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KO6bkEELlKbAcI8UYdlMYtAO8gs= 2022-09-12T19:03:51.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KO6bkEELlKbAcI8UYdlMYtAO8gs=', 'Content-Length': '0'} 2022-09-12T19:03:51.937 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d66f2195da52e3cd-00631f8297-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.938 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.939 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.939 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.940 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.940 DEBUG:boto:Method: GET 2022-09-12T19:03:51.941 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.941 DEBUG:boto:Data: 2022-09-12T19:03:51.942 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.942 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.943 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.943 DEBUG:boto:Params: {} 2022-09-12T19:03:51.944 DEBUG:boto:Token: None 2022-09-12T19:03:51.945 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.945 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KO6bkEELlKbAcI8UYdlMYtAO8gs= 2022-09-12T19:03:51.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KO6bkEELlKbAcI8UYdlMYtAO8gs=', 'Content-Length': '0'} 2022-09-12T19:03:51.948 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a503d39aa87c400-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.949 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.949 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.950 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.951 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.951 DEBUG:boto:Method: HEAD 2022-09-12T19:03:51.952 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.952 DEBUG:boto:Data: 2022-09-12T19:03:51.953 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.953 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:51.954 DEBUG:boto:Port: 8001 2022-09-12T19:03:51.954 DEBUG:boto:Params: {} 2022-09-12T19:03:51.955 DEBUG:boto:Token: None 2022-09-12T19:03:51.955 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.956 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FbTHdZgJ9Ymc1PXgbTic3UfouQo= 2022-09-12T19:03:51.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FbTHdZgJ9Ymc1PXgbTic3UfouQo=', 'Content-Length': '0'} 2022-09-12T19:03:51.958 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b51f410959537038-00631f8297-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.959 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.960 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.960 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.961 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.961 DEBUG:boto:Method: HEAD 2022-09-12T19:03:51.962 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.962 DEBUG:boto:Data: 2022-09-12T19:03:51.963 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.963 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.964 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.964 DEBUG:boto:Params: {} 2022-09-12T19:03:51.965 DEBUG:boto:Token: None 2022-09-12T19:03:51.965 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.966 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FbTHdZgJ9Ymc1PXgbTic3UfouQo= 2022-09-12T19:03:51.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FbTHdZgJ9Ymc1PXgbTic3UfouQo=', 'Content-Length': '0'} 2022-09-12T19:03:51.968 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a953a2683e4ae4ef-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:51.970 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.970 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.971 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.971 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.972 DEBUG:boto:Method: GET 2022-09-12T19:03:51.972 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.973 DEBUG:boto:Data: 2022-09-12T19:03:51.974 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.974 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:51.975 DEBUG:boto:Port: 8001 2022-09-12T19:03:51.975 DEBUG:boto:Params: {} 2022-09-12T19:03:51.976 DEBUG:boto:Token: None 2022-09-12T19:03:51.976 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.977 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KO6bkEELlKbAcI8UYdlMYtAO8gs= 2022-09-12T19:03:51.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KO6bkEELlKbAcI8UYdlMYtAO8gs=', 'Content-Length': '0'} 2022-09-12T19:03:51.980 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000238345b4f09e2b65-00631f8297-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.980 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.981 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:51.982 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.982 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.983 DEBUG:boto:Method: GET 2022-09-12T19:03:51.983 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.984 DEBUG:boto:Data: 2022-09-12T19:03:51.984 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.985 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:51.985 DEBUG:boto:Port: 8000 2022-09-12T19:03:51.986 DEBUG:boto:Params: {} 2022-09-12T19:03:51.986 DEBUG:boto:Token: None 2022-09-12T19:03:51.987 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:51.988 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KO6bkEELlKbAcI8UYdlMYtAO8gs= 2022-09-12T19:03:51.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KO6bkEELlKbAcI8UYdlMYtAO8gs=', 'Content-Length': '0'} 2022-09-12T19:03:51.991 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000293ff044a46f2ee8-00631f8297-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:51.991 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:51.992 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:51.993 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:51.993 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:51.994 DEBUG:boto:Method: GET 2022-09-12T19:03:51.994 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:51.995 DEBUG:boto:Data: 2022-09-12T19:03:51.995 DEBUG:boto:Headers: {} 2022-09-12T19:03:51.996 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:51.996 DEBUG:boto:Port: 8001 2022-09-12T19:03:51.997 DEBUG:boto:Params: {} 2022-09-12T19:03:51.997 DEBUG:boto:Token: None 2022-09-12T19:03:51.998 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:51 GMT /jdpiak-18/ 2022-09-12T19:03:51.999 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo= 2022-09-12T19:03:51.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VH3y4uc3UYx6+2TsHFaCIviEILo=', 'Content-Length': '0'} 2022-09-12T19:03:52.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072e59730b2bd439d-00631f8297-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.003 DEBUG:boto:b'jdpiak-18obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:52.003 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:52.004 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:52.005 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:52.005 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:52.006 DEBUG:boto:Method: GET 2022-09-12T19:03:52.006 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:52.007 DEBUG:boto:Data: 2022-09-12T19:03:52.007 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.008 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:52.008 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.009 DEBUG:boto:Params: {} 2022-09-12T19:03:52.009 DEBUG:boto:Token: None 2022-09-12T19:03:52.010 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/ 2022-09-12T19:03:52.010 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU= 2022-09-12T19:03:52.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU=', 'Content-Length': '0'} 2022-09-12T19:03:52.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077bc565afca34a5d-00631f8298-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.015 DEBUG:boto:b'jdpiak-18obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:52.015 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.016 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.016 DEBUG:boto:Method: HEAD 2022-09-12T19:03:52.017 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.017 DEBUG:boto:Data: 2022-09-12T19:03:52.018 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.018 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.019 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.019 DEBUG:boto:Params: {} 2022-09-12T19:03:52.020 DEBUG:boto:Token: None 2022-09-12T19:03:52.021 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.021 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RIOgzOGHYY3LBpiUezhkgDGji0E= 2022-09-12T19:03:52.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RIOgzOGHYY3LBpiUezhkgDGji0E=', 'Content-Length': '0'} 2022-09-12T19:03:52.024 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031924c1531397336-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.025 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.025 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.026 DEBUG:boto:Method: HEAD 2022-09-12T19:03:52.026 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.027 DEBUG:boto:Data: 2022-09-12T19:03:52.027 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.028 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:52.029 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.029 DEBUG:boto:Params: {} 2022-09-12T19:03:52.030 DEBUG:boto:Token: None 2022-09-12T19:03:52.030 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.031 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RIOgzOGHYY3LBpiUezhkgDGji0E= 2022-09-12T19:03:52.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RIOgzOGHYY3LBpiUezhkgDGji0E=', 'Content-Length': '0'} 2022-09-12T19:03:52.034 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017a994a09e1371d0-00631f8298-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:52.035 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.036 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.036 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.037 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.037 DEBUG:boto:Method: GET 2022-09-12T19:03:52.038 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.038 DEBUG:boto:Data: 2022-09-12T19:03:52.039 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.039 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.040 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.040 DEBUG:boto:Params: {} 2022-09-12T19:03:52.041 DEBUG:boto:Token: None 2022-09-12T19:03:52.041 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.042 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tRskDkgeYMzChkJrsuEYKOHdNWk= 2022-09-12T19:03:52.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tRskDkgeYMzChkJrsuEYKOHdNWk=', 'Content-Length': '0'} 2022-09-12T19:03:52.045 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009f4d907abd5de87-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.086 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.087 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.087 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.088 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.089 DEBUG:boto:Method: GET 2022-09-12T19:03:52.089 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.090 DEBUG:boto:Data: 2022-09-12T19:03:52.091 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.091 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:52.092 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.092 DEBUG:boto:Params: {} 2022-09-12T19:03:52.093 DEBUG:boto:Token: None 2022-09-12T19:03:52.093 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.094 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tRskDkgeYMzChkJrsuEYKOHdNWk= 2022-09-12T19:03:52.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tRskDkgeYMzChkJrsuEYKOHdNWk=', 'Content-Length': '0'} 2022-09-12T19:03:52.098 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003021d072b7718d43-00631f8298-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.099 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:52.099 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:52.100 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:52.100 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:52.101 DEBUG:boto:Method: GET 2022-09-12T19:03:52.101 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:52.102 DEBUG:boto:Data: 2022-09-12T19:03:52.102 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.103 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.103 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.104 DEBUG:boto:Params: {} 2022-09-12T19:03:52.104 DEBUG:boto:Token: None 2022-09-12T19:03:52.105 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/ 2022-09-12T19:03:52.105 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU= 2022-09-12T19:03:52.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU=', 'Content-Length': '0'} 2022-09-12T19:03:52.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062a9811842104661-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.110 DEBUG:boto:b'jdpiak-18obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:52.111 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:52.111 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:52.112 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:52.113 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:52.113 DEBUG:boto:Method: GET 2022-09-12T19:03:52.114 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:52.114 DEBUG:boto:Data: 2022-09-12T19:03:52.115 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.115 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:52.116 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.117 DEBUG:boto:Params: {} 2022-09-12T19:03:52.118 DEBUG:boto:Token: None 2022-09-12T19:03:52.119 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/ 2022-09-12T19:03:52.120 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU= 2022-09-12T19:03:52.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU=', 'Content-Length': '0'} 2022-09-12T19:03:52.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a7e37af7efba686-00631f8298-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.125 DEBUG:boto:b'jdpiak-18obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:52.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:52.126 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:52.127 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:52.128 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.129 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.130 DEBUG:boto:Method: GET 2022-09-12T19:03:52.130 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.131 DEBUG:boto:Data: 2022-09-12T19:03:52.131 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.132 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.133 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.133 DEBUG:boto:Params: {} 2022-09-12T19:03:52.134 DEBUG:boto:Token: None 2022-09-12T19:03:52.135 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.135 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FE/AQDVEgTQNU6o5MUDanYuhJ8k= 2022-09-12T19:03:52.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FE/AQDVEgTQNU6o5MUDanYuhJ8k=', 'Content-Length': '0'} 2022-09-12T19:03:52.138 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8d6ace5a88b2bce-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.139 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:52.140 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:52.141 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.141 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.142 DEBUG:boto:Method: GET 2022-09-12T19:03:52.142 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.143 DEBUG:boto:Data: 2022-09-12T19:03:52.143 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.144 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:52.144 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.145 DEBUG:boto:Params: {} 2022-09-12T19:03:52.146 DEBUG:boto:Token: None 2022-09-12T19:03:52.146 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.147 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FE/AQDVEgTQNU6o5MUDanYuhJ8k= 2022-09-12T19:03:52.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FE/AQDVEgTQNU6o5MUDanYuhJ8k=', 'Content-Length': '0'} 2022-09-12T19:03:52.150 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1f0c4dd4459564d-00631f8298-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.151 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:52.152 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:52.153 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.153 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.153 DEBUG:boto:Method: HEAD 2022-09-12T19:03:52.154 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.154 DEBUG:boto:Data: 2022-09-12T19:03:52.155 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.155 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.156 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.157 DEBUG:boto:Params: {} 2022-09-12T19:03:52.158 DEBUG:boto:Token: None 2022-09-12T19:03:52.158 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.159 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k4F95WQ1DCVtLGm+Tak6eA7qeXE= 2022-09-12T19:03:52.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k4F95WQ1DCVtLGm+Tak6eA7qeXE=', 'Content-Length': '0'} 2022-09-12T19:03:52.161 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058641867c56da753-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.162 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:52.163 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:52.163 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.164 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.165 DEBUG:boto:Method: HEAD 2022-09-12T19:03:52.165 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.166 DEBUG:boto:Data: 2022-09-12T19:03:52.166 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.167 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:52.168 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.169 DEBUG:boto:Params: {} 2022-09-12T19:03:52.170 DEBUG:boto:Token: None 2022-09-12T19:03:52.170 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.171 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k4F95WQ1DCVtLGm+Tak6eA7qeXE= 2022-09-12T19:03:52.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k4F95WQ1DCVtLGm+Tak6eA7qeXE=', 'Content-Length': '0'} 2022-09-12T19:03:52.173 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000971c2125931a2be1-00631f8298-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:52.175 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:52.175 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:52.176 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.176 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.177 DEBUG:boto:Method: GET 2022-09-12T19:03:52.178 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.178 DEBUG:boto:Data: 2022-09-12T19:03:52.179 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.179 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.180 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.181 DEBUG:boto:Params: {} 2022-09-12T19:03:52.181 DEBUG:boto:Token: None 2022-09-12T19:03:52.182 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.183 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FE/AQDVEgTQNU6o5MUDanYuhJ8k= 2022-09-12T19:03:52.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FE/AQDVEgTQNU6o5MUDanYuhJ8k=', 'Content-Length': '0'} 2022-09-12T19:03:52.185 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b198a97174d8d2aa-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.186 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:52.187 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:52.187 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.188 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.188 DEBUG:boto:Method: GET 2022-09-12T19:03:52.189 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.189 DEBUG:boto:Data: 2022-09-12T19:03:52.190 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.190 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:52.191 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.191 DEBUG:boto:Params: {} 2022-09-12T19:03:52.192 DEBUG:boto:Token: None 2022-09-12T19:03:52.193 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.193 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FE/AQDVEgTQNU6o5MUDanYuhJ8k= 2022-09-12T19:03:52.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FE/AQDVEgTQNU6o5MUDanYuhJ8k=', 'Content-Length': '0'} 2022-09-12T19:03:52.196 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f910ca9016860f3c-00631f8298-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.197 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:52.197 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:52.198 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:52.198 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:52.199 DEBUG:boto:Method: GET 2022-09-12T19:03:52.200 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:52.200 DEBUG:boto:Data: 2022-09-12T19:03:52.201 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.202 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.202 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.203 DEBUG:boto:Params: {} 2022-09-12T19:03:52.204 DEBUG:boto:Token: None 2022-09-12T19:03:52.204 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/ 2022-09-12T19:03:52.205 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU= 2022-09-12T19:03:52.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU=', 'Content-Length': '0'} 2022-09-12T19:03:52.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056307887b7b1ce51-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.209 DEBUG:boto:b'jdpiak-18obj-test-zone21000falseobj-test-zone22022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:52.210 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:52.211 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:52.211 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:52.212 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:52.212 DEBUG:boto:Method: GET 2022-09-12T19:03:52.213 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:52.213 DEBUG:boto:Data: 2022-09-12T19:03:52.214 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.215 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:52.215 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.216 DEBUG:boto:Params: {} 2022-09-12T19:03:52.216 DEBUG:boto:Token: None 2022-09-12T19:03:52.217 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/ 2022-09-12T19:03:52.218 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU= 2022-09-12T19:03:52.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU=', 'Content-Length': '0'} 2022-09-12T19:03:52.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000784afa11a3a542d6-00631f8298-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.222 DEBUG:boto:b'jdpiak-18obj-test-zone21000falseobj-test-zone22022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:52.223 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:52.223 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:52.224 DEBUG:boto:Method: HEAD 2022-09-12T19:03:52.224 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2 2022-09-12T19:03:52.225 DEBUG:boto:Data: 2022-09-12T19:03:52.225 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.226 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.226 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.227 DEBUG:boto:Params: {} 2022-09-12T19:03:52.227 DEBUG:boto:Token: None 2022-09-12T19:03:52.228 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone2 2022-09-12T19:03:52.229 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7dttTeTf7U+kmQiUOW7ZhPxowyc= 2022-09-12T19:03:52.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7dttTeTf7U+kmQiUOW7ZhPxowyc=', 'Content-Length': '0'} 2022-09-12T19:03:52.232 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008448c4dcf9c8521d-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.232 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:52.233 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:52.234 DEBUG:boto:Method: HEAD 2022-09-12T19:03:52.234 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2 2022-09-12T19:03:52.235 DEBUG:boto:Data: 2022-09-12T19:03:52.235 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.236 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:52.236 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.237 DEBUG:boto:Params: {} 2022-09-12T19:03:52.237 DEBUG:boto:Token: None 2022-09-12T19:03:52.238 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone2 2022-09-12T19:03:52.238 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7dttTeTf7U+kmQiUOW7ZhPxowyc= 2022-09-12T19:03:52.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7dttTeTf7U+kmQiUOW7ZhPxowyc=', 'Content-Length': '0'} 2022-09-12T19:03:52.241 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000467bb04a2af82248-00631f8298-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:52.243 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:52.243 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:52.244 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.244 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.245 DEBUG:boto:Method: GET 2022-09-12T19:03:52.245 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.246 DEBUG:boto:Data: 2022-09-12T19:03:52.247 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.247 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.248 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.248 DEBUG:boto:Params: {} 2022-09-12T19:03:52.249 DEBUG:boto:Token: None 2022-09-12T19:03:52.250 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.250 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FE/AQDVEgTQNU6o5MUDanYuhJ8k= 2022-09-12T19:03:52.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FE/AQDVEgTQNU6o5MUDanYuhJ8k=', 'Content-Length': '0'} 2022-09-12T19:03:52.253 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a1a85c47e1e8450-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.255 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:52.255 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:52.256 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.256 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.257 DEBUG:boto:Method: GET 2022-09-12T19:03:52.257 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.258 DEBUG:boto:Data: 2022-09-12T19:03:52.258 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.259 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:52.260 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.260 DEBUG:boto:Params: {} 2022-09-12T19:03:52.261 DEBUG:boto:Token: None 2022-09-12T19:03:52.262 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:52.262 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FE/AQDVEgTQNU6o5MUDanYuhJ8k= 2022-09-12T19:03:52.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FE/AQDVEgTQNU6o5MUDanYuhJ8k=', 'Content-Length': '0'} 2022-09-12T19:03:52.265 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002fc37fe558f61d21-00631f8298-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.266 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:52.266 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:52.267 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:52.267 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:52.268 DEBUG:boto:Method: GET 2022-09-12T19:03:52.268 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:52.269 DEBUG:boto:Data: 2022-09-12T19:03:52.269 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.270 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.271 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.271 DEBUG:boto:Params: {} 2022-09-12T19:03:52.272 DEBUG:boto:Token: None 2022-09-12T19:03:52.272 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/ 2022-09-12T19:03:52.273 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU= 2022-09-12T19:03:52.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU=', 'Content-Length': '0'} 2022-09-12T19:03:52.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9c2a1b1eb0432e5-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.278 DEBUG:boto:b'jdpiak-18obj-test-zone21000falseobj-test-zone22022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:52.279 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:52.280 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:52.281 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:52.282 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:52.283 DEBUG:boto:Method: GET 2022-09-12T19:03:52.284 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:52.285 DEBUG:boto:Data: 2022-09-12T19:03:52.286 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.287 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:52.288 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.289 DEBUG:boto:Params: {} 2022-09-12T19:03:52.290 DEBUG:boto:Token: None 2022-09-12T19:03:52.291 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/ 2022-09-12T19:03:52.292 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU= 2022-09-12T19:03:52.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU=', 'Content-Length': '0'} 2022-09-12T19:03:52.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023388a389d138541-00631f8298-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.298 DEBUG:boto:b'jdpiak-18obj-test-zone21000falseobj-test-zone22022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:52.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:52.300 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:52.301 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:52.302 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.303 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.304 DEBUG:boto:Method: GET 2022-09-12T19:03:52.305 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.306 DEBUG:boto:Data: 2022-09-12T19:03:52.307 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.307 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.308 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.309 DEBUG:boto:Params: {} 2022-09-12T19:03:52.310 DEBUG:boto:Token: None 2022-09-12T19:03:52.311 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.312 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YmRvz/ihly9yfawDuUNmeY7gWZw= 2022-09-12T19:03:52.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YmRvz/ihly9yfawDuUNmeY7gWZw=', 'Content-Length': '0'} 2022-09-12T19:03:52.315 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b53ded2a944614e1-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.316 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:52.317 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:52.317 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.318 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.318 DEBUG:boto:Method: GET 2022-09-12T19:03:52.319 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.319 DEBUG:boto:Data: 2022-09-12T19:03:52.320 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.321 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:52.321 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.322 DEBUG:boto:Params: {} 2022-09-12T19:03:52.322 DEBUG:boto:Token: None 2022-09-12T19:03:52.323 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.324 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YmRvz/ihly9yfawDuUNmeY7gWZw= 2022-09-12T19:03:52.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YmRvz/ihly9yfawDuUNmeY7gWZw=', 'Content-Length': '0'} 2022-09-12T19:03:52.327 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006918eaf8a172c960-00631f8298-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.327 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:52.328 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:52.329 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.329 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.330 DEBUG:boto:Method: HEAD 2022-09-12T19:03:52.330 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.331 DEBUG:boto:Data: 2022-09-12T19:03:52.331 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.332 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.332 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.333 DEBUG:boto:Params: {} 2022-09-12T19:03:52.333 DEBUG:boto:Token: None 2022-09-12T19:03:52.334 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.335 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:C8eh03RdTsoH6Y5JLjiOitFsNtc= 2022-09-12T19:03:52.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:C8eh03RdTsoH6Y5JLjiOitFsNtc=', 'Content-Length': '0'} 2022-09-12T19:03:52.338 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000007afa45119de916-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.339 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:52.339 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:52.340 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.340 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.341 DEBUG:boto:Method: HEAD 2022-09-12T19:03:52.341 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.342 DEBUG:boto:Data: 2022-09-12T19:03:52.342 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.343 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:52.343 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.344 DEBUG:boto:Params: {} 2022-09-12T19:03:52.345 DEBUG:boto:Token: None 2022-09-12T19:03:52.345 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.346 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:C8eh03RdTsoH6Y5JLjiOitFsNtc= 2022-09-12T19:03:52.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:C8eh03RdTsoH6Y5JLjiOitFsNtc=', 'Content-Length': '0'} 2022-09-12T19:03:52.348 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009743247495128ddf-00631f8298-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:52.350 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:52.351 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:52.352 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.352 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.353 DEBUG:boto:Method: GET 2022-09-12T19:03:52.353 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.354 DEBUG:boto:Data: 2022-09-12T19:03:52.354 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.355 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.355 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.356 DEBUG:boto:Params: {} 2022-09-12T19:03:52.356 DEBUG:boto:Token: None 2022-09-12T19:03:52.357 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.358 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YmRvz/ihly9yfawDuUNmeY7gWZw= 2022-09-12T19:03:52.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YmRvz/ihly9yfawDuUNmeY7gWZw=', 'Content-Length': '0'} 2022-09-12T19:03:52.360 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b80477477a79fcf-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.361 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:52.362 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:52.362 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.363 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.363 DEBUG:boto:Method: GET 2022-09-12T19:03:52.364 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.364 DEBUG:boto:Data: 2022-09-12T19:03:52.365 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.366 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:52.366 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.367 DEBUG:boto:Params: {} 2022-09-12T19:03:52.367 DEBUG:boto:Token: None 2022-09-12T19:03:52.368 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.368 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YmRvz/ihly9yfawDuUNmeY7gWZw= 2022-09-12T19:03:52.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YmRvz/ihly9yfawDuUNmeY7gWZw=', 'Content-Length': '0'} 2022-09-12T19:03:52.371 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054a0e2a1ca7bc63b-00631f8298-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.372 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:52.372 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:52.373 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:52.373 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:52.374 DEBUG:boto:Method: GET 2022-09-12T19:03:52.374 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:52.375 DEBUG:boto:Data: 2022-09-12T19:03:52.375 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.376 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.376 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.377 DEBUG:boto:Params: {} 2022-09-12T19:03:52.378 DEBUG:boto:Token: None 2022-09-12T19:03:52.378 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/ 2022-09-12T19:03:52.379 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU= 2022-09-12T19:03:52.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU=', 'Content-Length': '0'} 2022-09-12T19:03:52.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e29378ec758572c2-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.383 DEBUG:boto:b'jdpiak-18obj-test-zone31000falseobj-test-zone32022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:52.384 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:52.384 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:52.385 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:52.385 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:52.386 DEBUG:boto:Method: GET 2022-09-12T19:03:52.386 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:52.387 DEBUG:boto:Data: 2022-09-12T19:03:52.388 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.388 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:52.389 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.389 DEBUG:boto:Params: {} 2022-09-12T19:03:52.390 DEBUG:boto:Token: None 2022-09-12T19:03:52.390 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/ 2022-09-12T19:03:52.391 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU= 2022-09-12T19:03:52.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU=', 'Content-Length': '0'} 2022-09-12T19:03:52.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bad2b004d0fc3d44-00631f8298-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.395 DEBUG:boto:b'jdpiak-18obj-test-zone31000falseobj-test-zone32022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:52.395 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:52.396 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:52.396 DEBUG:boto:Method: HEAD 2022-09-12T19:03:52.397 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3 2022-09-12T19:03:52.397 DEBUG:boto:Data: 2022-09-12T19:03:52.398 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.398 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.399 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.399 DEBUG:boto:Params: {} 2022-09-12T19:03:52.400 DEBUG:boto:Token: None 2022-09-12T19:03:52.401 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone3 2022-09-12T19:03:52.401 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gFx8Zx3Kpvehhv++t0rxPH99hBs= 2022-09-12T19:03:52.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gFx8Zx3Kpvehhv++t0rxPH99hBs=', 'Content-Length': '0'} 2022-09-12T19:03:52.404 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c4faf0fd5a02617-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.405 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:52.405 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:52.406 DEBUG:boto:Method: HEAD 2022-09-12T19:03:52.406 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3 2022-09-12T19:03:52.407 DEBUG:boto:Data: 2022-09-12T19:03:52.407 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.408 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:52.409 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.409 DEBUG:boto:Params: {} 2022-09-12T19:03:52.410 DEBUG:boto:Token: None 2022-09-12T19:03:52.410 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone3 2022-09-12T19:03:52.411 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gFx8Zx3Kpvehhv++t0rxPH99hBs= 2022-09-12T19:03:52.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gFx8Zx3Kpvehhv++t0rxPH99hBs=', 'Content-Length': '0'} 2022-09-12T19:03:52.414 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094b935a6c4005b5e-00631f8298-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:52.415 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:52.416 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:52.416 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.417 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.417 DEBUG:boto:Method: GET 2022-09-12T19:03:52.418 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.419 DEBUG:boto:Data: 2022-09-12T19:03:52.419 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.420 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.420 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.421 DEBUG:boto:Params: {} 2022-09-12T19:03:52.422 DEBUG:boto:Token: None 2022-09-12T19:03:52.423 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.424 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YmRvz/ihly9yfawDuUNmeY7gWZw= 2022-09-12T19:03:52.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YmRvz/ihly9yfawDuUNmeY7gWZw=', 'Content-Length': '0'} 2022-09-12T19:03:52.427 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ea4533a1218767f-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.470 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:52.471 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:52.472 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.472 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.473 DEBUG:boto:Method: GET 2022-09-12T19:03:52.473 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.474 DEBUG:boto:Data: 2022-09-12T19:03:52.474 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.475 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:52.476 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.476 DEBUG:boto:Params: {} 2022-09-12T19:03:52.477 DEBUG:boto:Token: None 2022-09-12T19:03:52.477 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:52.478 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YmRvz/ihly9yfawDuUNmeY7gWZw= 2022-09-12T19:03:52.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YmRvz/ihly9yfawDuUNmeY7gWZw=', 'Content-Length': '0'} 2022-09-12T19:03:52.481 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038a31013d05d9f2c-00631f8298-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.482 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:52.482 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:52.483 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:52.483 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:52.484 DEBUG:boto:Method: GET 2022-09-12T19:03:52.484 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:52.485 DEBUG:boto:Data: 2022-09-12T19:03:52.485 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.486 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.486 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.487 DEBUG:boto:Params: {} 2022-09-12T19:03:52.488 DEBUG:boto:Token: None 2022-09-12T19:03:52.488 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/ 2022-09-12T19:03:52.489 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU= 2022-09-12T19:03:52.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU=', 'Content-Length': '0'} 2022-09-12T19:03:52.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f9050b8d90758af-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.493 DEBUG:boto:b'jdpiak-18obj-test-zone31000falseobj-test-zone32022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:52.494 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:52.494 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:52.495 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:52.495 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:52.496 DEBUG:boto:Method: GET 2022-09-12T19:03:52.496 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:52.497 DEBUG:boto:Data: 2022-09-12T19:03:52.497 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.498 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:52.499 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.500 DEBUG:boto:Params: {} 2022-09-12T19:03:52.500 DEBUG:boto:Token: None 2022-09-12T19:03:52.501 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/ 2022-09-12T19:03:52.502 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU= 2022-09-12T19:03:52.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU=', 'Content-Length': '0'} 2022-09-12T19:03:52.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d325e0b76af4449d-00631f8298-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.506 DEBUG:boto:b'jdpiak-18obj-test-zone31000falseobj-test-zone32022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:52.506 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-18 zones={test-zone3, test-zone1} 2022-09-12T19:03:52.513 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-18 zones={test-zone3, test-zone2} 2022-09-12T19:03:52.514 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:52.514 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:52.515 DEBUG:boto:Method: HEAD 2022-09-12T19:03:52.515 DEBUG:boto:Path: /jdpiak-18/ 2022-09-12T19:03:52.516 DEBUG:boto:Data: 2022-09-12T19:03:52.517 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.517 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.518 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.518 DEBUG:boto:Params: {} 2022-09-12T19:03:52.519 DEBUG:boto:Token: None 2022-09-12T19:03:52.520 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/ 2022-09-12T19:03:52.520 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SoXbT7rihzNp4iccFdkgSgCWm+0= 2022-09-12T19:03:52.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SoXbT7rihzNp4iccFdkgSgCWm+0=', 'Content-Length': '0'} 2022-09-12T19:03:52.523 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a000bb099fee9e8e-00631f8298-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.524 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:52.525 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:52.525 DEBUG:boto:Method: HEAD 2022-09-12T19:03:52.526 DEBUG:boto:Path: /jdpiak-18/ 2022-09-12T19:03:52.526 DEBUG:boto:Data: 2022-09-12T19:03:52.527 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.527 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:52.528 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.528 DEBUG:boto:Params: {} 2022-09-12T19:03:52.529 DEBUG:boto:Token: None 2022-09-12T19:03:52.529 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/ 2022-09-12T19:03:52.530 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SoXbT7rihzNp4iccFdkgSgCWm+0= 2022-09-12T19:03:52.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SoXbT7rihzNp4iccFdkgSgCWm+0=', 'Content-Length': '0'} 2022-09-12T19:03:52.534 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000284749069efa1bfd-00631f8298-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.535 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:03:52.536 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:52.537 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:52.538 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:52.539 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:52.539 DEBUG:boto:Method: GET 2022-09-12T19:03:52.540 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:52.540 DEBUG:boto:Data: 2022-09-12T19:03:52.541 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.541 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.542 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.542 DEBUG:boto:Params: {} 2022-09-12T19:03:52.543 DEBUG:boto:Token: None 2022-09-12T19:03:52.543 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/?versions 2022-09-12T19:03:52.544 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RhmGFIPN96aO0iwHQG2UZWDvjOw= 2022-09-12T19:03:52.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RhmGFIPN96aO0iwHQG2UZWDvjOw=', 'Content-Length': '0'} 2022-09-12T19:03:52.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002740d39f8508762a-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.548 DEBUG:boto:b'jdpiak-18999falsedummyqUufSwmbKETbT7yNOH8sL0A4tddUbNOtrue2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ToUtj2XoS.tz3goIbvjFwiigMyzGOpLtrue2022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2DPksivOWVYhJDrEwQqHYV7wK1Ul1JkWtrue2022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Ztrue2022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:52.549 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:03:52.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:03:52.550 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:03:52.550 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:03:52.551 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:03:52.551 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:52.552 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:52.552 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:52.553 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:52.553 DEBUG:boto:Method: GET 2022-09-12T19:03:52.554 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:52.554 DEBUG:boto:Data: 2022-09-12T19:03:52.555 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.555 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:52.556 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.556 DEBUG:boto:Params: {} 2022-09-12T19:03:52.557 DEBUG:boto:Token: None 2022-09-12T19:03:52.557 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/?versions 2022-09-12T19:03:52.558 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RhmGFIPN96aO0iwHQG2UZWDvjOw= 2022-09-12T19:03:52.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RhmGFIPN96aO0iwHQG2UZWDvjOw=', 'Content-Length': '0'} 2022-09-12T19:03:52.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055e8d3f279c3ba50-00631f8298-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.562 DEBUG:boto:b'jdpiak-18999falsedummyqUufSwmbKETbT7yNOH8sL0A4tddUbNOtrue2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ToUtj2XoS.tz3goIbvjFwiigMyzGOpLtrue2022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2DPksivOWVYhJDrEwQqHYV7wK1Ul1JkWtrue2022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Ztrue2022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:52.563 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:03:52.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-12T19:03:52.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-12T19:03:52.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-12T19:03:52.565 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:52.565 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:52.566 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:52.567 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:52.567 DEBUG:boto:Method: GET 2022-09-12T19:03:52.568 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:52.568 DEBUG:boto:Data: 2022-09-12T19:03:52.569 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.569 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.569 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.570 DEBUG:boto:Params: {} 2022-09-12T19:03:52.571 DEBUG:boto:Token: None 2022-09-12T19:03:52.571 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/?versions 2022-09-12T19:03:52.572 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RhmGFIPN96aO0iwHQG2UZWDvjOw= 2022-09-12T19:03:52.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RhmGFIPN96aO0iwHQG2UZWDvjOw=', 'Content-Length': '0'} 2022-09-12T19:03:52.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f98de2c2be11277b-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.577 DEBUG:boto:b'jdpiak-18999falsedummyqUufSwmbKETbT7yNOH8sL0A4tddUbNOtrue2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ToUtj2XoS.tz3goIbvjFwiigMyzGOpLtrue2022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2DPksivOWVYhJDrEwQqHYV7wK1Ul1JkWtrue2022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Ztrue2022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:52.577 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:52.578 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:52.579 DEBUG:boto:path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:52.579 DEBUG:boto:auth_path=/jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:52.580 DEBUG:boto:Method: GET 2022-09-12T19:03:52.580 DEBUG:boto:Path: /jdpiak-18/?versions&max-keys=999 2022-09-12T19:03:52.581 DEBUG:boto:Data: 2022-09-12T19:03:52.582 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.582 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:52.583 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.584 DEBUG:boto:Params: {} 2022-09-12T19:03:52.585 DEBUG:boto:Token: None 2022-09-12T19:03:52.586 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/?versions 2022-09-12T19:03:52.586 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RhmGFIPN96aO0iwHQG2UZWDvjOw= 2022-09-12T19:03:52.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RhmGFIPN96aO0iwHQG2UZWDvjOw=', 'Content-Length': '0'} 2022-09-12T19:03:52.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e156805c66b3e7dd-00631f8298-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.590 DEBUG:boto:b'jdpiak-18999falsedummyqUufSwmbKETbT7yNOH8sL0A4tddUbNOtrue2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1ToUtj2XoS.tz3goIbvjFwiigMyzGOpLtrue2022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2DPksivOWVYhJDrEwQqHYV7wK1Ul1JkWtrue2022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Ztrue2022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:52.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:52.592 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:52.592 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:52.593 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.593 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.594 DEBUG:boto:Method: GET 2022-09-12T19:03:52.595 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.596 DEBUG:boto:Data: 2022-09-12T19:03:52.597 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.597 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.598 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.599 DEBUG:boto:Params: {} 2022-09-12T19:03:52.600 DEBUG:boto:Token: None 2022-09-12T19:03:52.600 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.601 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xq7/8WsK42d/jePCabsAeITADSM= 2022-09-12T19:03:52.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xq7/8WsK42d/jePCabsAeITADSM=', 'Content-Length': '0'} 2022-09-12T19:03:52.603 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000090f4a78198721029-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.604 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:52.604 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:52.605 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.606 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.607 DEBUG:boto:Method: GET 2022-09-12T19:03:52.607 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.608 DEBUG:boto:Data: 2022-09-12T19:03:52.609 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.609 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:52.610 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.610 DEBUG:boto:Params: {} 2022-09-12T19:03:52.611 DEBUG:boto:Token: None 2022-09-12T19:03:52.611 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.612 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xq7/8WsK42d/jePCabsAeITADSM= 2022-09-12T19:03:52.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xq7/8WsK42d/jePCabsAeITADSM=', 'Content-Length': '0'} 2022-09-12T19:03:52.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ed74baa75f45709f-00631f8298-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.615 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:52.616 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:52.616 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.617 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.617 DEBUG:boto:Method: HEAD 2022-09-12T19:03:52.618 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.618 DEBUG:boto:Data: 2022-09-12T19:03:52.619 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.619 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.620 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.621 DEBUG:boto:Params: {} 2022-09-12T19:03:52.621 DEBUG:boto:Token: None 2022-09-12T19:03:52.622 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.622 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4BtdqSCpVp/LBt1rlULRFIEIuIM= 2022-09-12T19:03:52.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4BtdqSCpVp/LBt1rlULRFIEIuIM=', 'Content-Length': '0'} 2022-09-12T19:03:52.625 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000155ec1732a621bcf-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.626 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:52.626 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:52.627 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.627 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.628 DEBUG:boto:Method: HEAD 2022-09-12T19:03:52.628 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.629 DEBUG:boto:Data: 2022-09-12T19:03:52.629 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.630 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:52.630 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.631 DEBUG:boto:Params: {} 2022-09-12T19:03:52.631 DEBUG:boto:Token: None 2022-09-12T19:03:52.632 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.632 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4BtdqSCpVp/LBt1rlULRFIEIuIM= 2022-09-12T19:03:52.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4BtdqSCpVp/LBt1rlULRFIEIuIM=', 'Content-Length': '0'} 2022-09-12T19:03:52.635 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005fdef07d4b8325b6-00631f8298-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:52.636 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:52.636 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:52.637 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.637 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.638 DEBUG:boto:Method: GET 2022-09-12T19:03:52.639 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.639 DEBUG:boto:Data: 2022-09-12T19:03:52.640 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.640 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.641 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.641 DEBUG:boto:Params: {} 2022-09-12T19:03:52.642 DEBUG:boto:Token: None 2022-09-12T19:03:52.642 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.643 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xq7/8WsK42d/jePCabsAeITADSM= 2022-09-12T19:03:52.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xq7/8WsK42d/jePCabsAeITADSM=', 'Content-Length': '0'} 2022-09-12T19:03:52.645 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000080f257ba48f3d992-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.646 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:52.647 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:52.647 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.648 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.648 DEBUG:boto:Method: GET 2022-09-12T19:03:52.649 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.649 DEBUG:boto:Data: 2022-09-12T19:03:52.650 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.650 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:52.651 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.651 DEBUG:boto:Params: {} 2022-09-12T19:03:52.652 DEBUG:boto:Token: None 2022-09-12T19:03:52.653 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.653 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xq7/8WsK42d/jePCabsAeITADSM= 2022-09-12T19:03:52.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xq7/8WsK42d/jePCabsAeITADSM=', 'Content-Length': '0'} 2022-09-12T19:03:52.656 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ca6fcee4bb341baf-00631f8298-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.657 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:52.658 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:52.659 DEBUG:boto:path=/jdpiak-18/?prefix=dummy 2022-09-12T19:03:52.660 DEBUG:boto:auth_path=/jdpiak-18/?prefix=dummy 2022-09-12T19:03:52.660 DEBUG:boto:Method: GET 2022-09-12T19:03:52.661 DEBUG:boto:Path: /jdpiak-18/?prefix=dummy 2022-09-12T19:03:52.662 DEBUG:boto:Data: 2022-09-12T19:03:52.663 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.664 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.664 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.665 DEBUG:boto:Params: {} 2022-09-12T19:03:52.666 DEBUG:boto:Token: None 2022-09-12T19:03:52.666 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/ 2022-09-12T19:03:52.667 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU= 2022-09-12T19:03:52.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU=', 'Content-Length': '0'} 2022-09-12T19:03:52.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e0989f3d05a7f76-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.714 DEBUG:boto:b'jdpiak-18dummy1000falsedummy2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:52.715 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:52.716 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:52.716 DEBUG:boto:path=/jdpiak-18/?prefix=dummy 2022-09-12T19:03:52.717 DEBUG:boto:auth_path=/jdpiak-18/?prefix=dummy 2022-09-12T19:03:52.718 DEBUG:boto:Method: GET 2022-09-12T19:03:52.718 DEBUG:boto:Path: /jdpiak-18/?prefix=dummy 2022-09-12T19:03:52.719 DEBUG:boto:Data: 2022-09-12T19:03:52.719 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.720 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:52.720 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.721 DEBUG:boto:Params: {} 2022-09-12T19:03:52.721 DEBUG:boto:Token: None 2022-09-12T19:03:52.722 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/ 2022-09-12T19:03:52.723 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU= 2022-09-12T19:03:52.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU=', 'Content-Length': '0'} 2022-09-12T19:03:52.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5438f6497eb3b61-00631f8298-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.727 DEBUG:boto:b'jdpiak-18dummy1000falsedummy2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:52.728 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:52.729 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:52.729 DEBUG:boto:Method: HEAD 2022-09-12T19:03:52.730 DEBUG:boto:Path: /jdpiak-18/dummy 2022-09-12T19:03:52.730 DEBUG:boto:Data: 2022-09-12T19:03:52.731 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.732 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.732 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.733 DEBUG:boto:Params: {} 2022-09-12T19:03:52.733 DEBUG:boto:Token: None 2022-09-12T19:03:52.734 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/dummy 2022-09-12T19:03:52.735 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Vd5u8twmMAlZ65H6F1IiTSXUbrc= 2022-09-12T19:03:52.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Vd5u8twmMAlZ65H6F1IiTSXUbrc=', 'Content-Length': '0'} 2022-09-12T19:03:52.739 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006b4e4725130a3601-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.740 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:52.740 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:52.741 DEBUG:boto:Method: HEAD 2022-09-12T19:03:52.741 DEBUG:boto:Path: /jdpiak-18/dummy 2022-09-12T19:03:52.742 DEBUG:boto:Data: 2022-09-12T19:03:52.742 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.743 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:52.743 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.744 DEBUG:boto:Params: {} 2022-09-12T19:03:52.745 DEBUG:boto:Token: None 2022-09-12T19:03:52.745 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/dummy 2022-09-12T19:03:52.747 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Vd5u8twmMAlZ65H6F1IiTSXUbrc= 2022-09-12T19:03:52.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Vd5u8twmMAlZ65H6F1IiTSXUbrc=', 'Content-Length': '0'} 2022-09-12T19:03:52.750 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003050d1d5d2bddeb7-00631f8298-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:03:52.752 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:52.752 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:52.753 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.754 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.755 DEBUG:boto:Method: GET 2022-09-12T19:03:52.756 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.756 DEBUG:boto:Data: 2022-09-12T19:03:52.757 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.758 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.758 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.759 DEBUG:boto:Params: {} 2022-09-12T19:03:52.760 DEBUG:boto:Token: None 2022-09-12T19:03:52.760 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.761 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xq7/8WsK42d/jePCabsAeITADSM= 2022-09-12T19:03:52.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xq7/8WsK42d/jePCabsAeITADSM=', 'Content-Length': '0'} 2022-09-12T19:03:52.764 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e22ba482b5d00f59-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.765 DEBUG:boto:path=/jdpiak-18/dummy 2022-09-12T19:03:52.765 DEBUG:boto:auth_path=/jdpiak-18/dummy 2022-09-12T19:03:52.766 DEBUG:boto:path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.766 DEBUG:boto:auth_path=/jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.767 DEBUG:boto:Method: GET 2022-09-12T19:03:52.768 DEBUG:boto:Path: /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.768 DEBUG:boto:Data: 2022-09-12T19:03:52.769 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.769 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:52.770 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.770 DEBUG:boto:Params: {} 2022-09-12T19:03:52.771 DEBUG:boto:Token: None 2022-09-12T19:03:52.772 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/dummy?versionId=qUufSwmbKETbT7yNOH8sL0A4tddUbNO 2022-09-12T19:03:52.772 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xq7/8WsK42d/jePCabsAeITADSM= 2022-09-12T19:03:52.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xq7/8WsK42d/jePCabsAeITADSM=', 'Content-Length': '0'} 2022-09-12T19:03:52.775 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 18:58:22 GMT'), ('x-amz-version-id', 'qUufSwmbKETbT7yNOH8sL0A4tddUbNO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007aa5e3fad79f1d9a-00631f8298-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.776 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:52.776 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:52.777 DEBUG:boto:path=/jdpiak-18/?prefix=dummy 2022-09-12T19:03:52.778 DEBUG:boto:auth_path=/jdpiak-18/?prefix=dummy 2022-09-12T19:03:52.778 DEBUG:boto:Method: GET 2022-09-12T19:03:52.779 DEBUG:boto:Path: /jdpiak-18/?prefix=dummy 2022-09-12T19:03:52.779 DEBUG:boto:Data: 2022-09-12T19:03:52.780 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.780 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.781 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.782 DEBUG:boto:Params: {} 2022-09-12T19:03:52.782 DEBUG:boto:Token: None 2022-09-12T19:03:52.783 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/ 2022-09-12T19:03:52.783 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU= 2022-09-12T19:03:52.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU=', 'Content-Length': '0'} 2022-09-12T19:03:52.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049cd15f5c7864bc2-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.788 DEBUG:boto:b'jdpiak-18dummy1000falsedummy2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:52.789 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:52.789 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:52.790 DEBUG:boto:path=/jdpiak-18/?prefix=dummy 2022-09-12T19:03:52.790 DEBUG:boto:auth_path=/jdpiak-18/?prefix=dummy 2022-09-12T19:03:52.791 DEBUG:boto:Method: GET 2022-09-12T19:03:52.792 DEBUG:boto:Path: /jdpiak-18/?prefix=dummy 2022-09-12T19:03:52.792 DEBUG:boto:Data: 2022-09-12T19:03:52.793 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.793 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:52.794 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.794 DEBUG:boto:Params: {} 2022-09-12T19:03:52.795 DEBUG:boto:Token: None 2022-09-12T19:03:52.796 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/ 2022-09-12T19:03:52.796 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU= 2022-09-12T19:03:52.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU=', 'Content-Length': '0'} 2022-09-12T19:03:52.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9590e23e5fdf1d6-00631f8298-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.842 DEBUG:boto:b'jdpiak-18dummy1000falsedummy2022-09-12T18:58:22.737Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:52.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:52.844 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.844 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.845 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.845 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.846 DEBUG:boto:Method: GET 2022-09-12T19:03:52.847 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.847 DEBUG:boto:Data: 2022-09-12T19:03:52.848 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.848 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.849 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.849 DEBUG:boto:Params: {} 2022-09-12T19:03:52.850 DEBUG:boto:Token: None 2022-09-12T19:03:52.851 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.851 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tRskDkgeYMzChkJrsuEYKOHdNWk= 2022-09-12T19:03:52.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tRskDkgeYMzChkJrsuEYKOHdNWk=', 'Content-Length': '0'} 2022-09-12T19:03:52.854 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e76c92c174953db-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.855 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.855 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.856 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.857 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.857 DEBUG:boto:Method: GET 2022-09-12T19:03:52.858 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.858 DEBUG:boto:Data: 2022-09-12T19:03:52.859 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.859 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:52.860 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.860 DEBUG:boto:Params: {} 2022-09-12T19:03:52.861 DEBUG:boto:Token: None 2022-09-12T19:03:52.862 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.862 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tRskDkgeYMzChkJrsuEYKOHdNWk= 2022-09-12T19:03:52.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tRskDkgeYMzChkJrsuEYKOHdNWk=', 'Content-Length': '0'} 2022-09-12T19:03:52.865 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009aff9fef9cbe75e8-00631f8298-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.866 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.866 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.867 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.867 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.868 DEBUG:boto:Method: HEAD 2022-09-12T19:03:52.868 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.869 DEBUG:boto:Data: 2022-09-12T19:03:52.870 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.870 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.871 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.871 DEBUG:boto:Params: {} 2022-09-12T19:03:52.872 DEBUG:boto:Token: None 2022-09-12T19:03:52.872 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.873 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5AYdToEYYnNRVR+rWxTi8RGByNk= 2022-09-12T19:03:52.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5AYdToEYYnNRVR+rWxTi8RGByNk=', 'Content-Length': '0'} 2022-09-12T19:03:52.876 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072a8eaef150df3f0-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.876 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.877 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.878 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.878 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.879 DEBUG:boto:Method: HEAD 2022-09-12T19:03:52.879 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.880 DEBUG:boto:Data: 2022-09-12T19:03:52.880 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.881 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:52.881 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.882 DEBUG:boto:Params: {} 2022-09-12T19:03:52.882 DEBUG:boto:Token: None 2022-09-12T19:03:52.883 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.883 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5AYdToEYYnNRVR+rWxTi8RGByNk= 2022-09-12T19:03:52.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5AYdToEYYnNRVR+rWxTi8RGByNk=', 'Content-Length': '0'} 2022-09-12T19:03:52.886 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000950f798501aa93e2-00631f8298-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:52.887 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.888 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.888 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.889 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.889 DEBUG:boto:Method: GET 2022-09-12T19:03:52.890 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.890 DEBUG:boto:Data: 2022-09-12T19:03:52.891 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.891 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.892 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.893 DEBUG:boto:Params: {} 2022-09-12T19:03:52.893 DEBUG:boto:Token: None 2022-09-12T19:03:52.894 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.895 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tRskDkgeYMzChkJrsuEYKOHdNWk= 2022-09-12T19:03:52.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tRskDkgeYMzChkJrsuEYKOHdNWk=', 'Content-Length': '0'} 2022-09-12T19:03:52.897 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000041c8784c9983faa-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.938 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.939 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.939 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.940 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.940 DEBUG:boto:Method: GET 2022-09-12T19:03:52.941 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.942 DEBUG:boto:Data: 2022-09-12T19:03:52.942 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.943 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:52.943 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.944 DEBUG:boto:Params: {} 2022-09-12T19:03:52.944 DEBUG:boto:Token: None 2022-09-12T19:03:52.945 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.946 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tRskDkgeYMzChkJrsuEYKOHdNWk= 2022-09-12T19:03:52.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tRskDkgeYMzChkJrsuEYKOHdNWk=', 'Content-Length': '0'} 2022-09-12T19:03:52.949 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed2335511079a1ca-00631f8298-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.949 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:52.950 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:52.950 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:52.951 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:52.952 DEBUG:boto:Method: GET 2022-09-12T19:03:52.952 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:52.953 DEBUG:boto:Data: 2022-09-12T19:03:52.953 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.954 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.954 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.955 DEBUG:boto:Params: {} 2022-09-12T19:03:52.955 DEBUG:boto:Token: None 2022-09-12T19:03:52.956 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/ 2022-09-12T19:03:52.956 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU= 2022-09-12T19:03:52.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU=', 'Content-Length': '0'} 2022-09-12T19:03:52.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009efbb4bec2426ad4-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.960 DEBUG:boto:b'jdpiak-18obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:52.961 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:52.962 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:52.962 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:52.963 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:52.963 DEBUG:boto:Method: GET 2022-09-12T19:03:52.964 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:52.964 DEBUG:boto:Data: 2022-09-12T19:03:52.965 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.965 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:52.966 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.967 DEBUG:boto:Params: {} 2022-09-12T19:03:52.967 DEBUG:boto:Token: None 2022-09-12T19:03:52.968 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/ 2022-09-12T19:03:52.968 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU= 2022-09-12T19:03:52.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CDAT1oArhsc9LN3+B0ercY3pRvU=', 'Content-Length': '0'} 2022-09-12T19:03:52.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6302b3fb011bd1a-00631f8298-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.973 DEBUG:boto:b'jdpiak-18obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:52.973 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.974 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.975 DEBUG:boto:Method: HEAD 2022-09-12T19:03:52.975 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.976 DEBUG:boto:Data: 2022-09-12T19:03:52.976 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.977 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.977 DEBUG:boto:Port: 8001 2022-09-12T19:03:52.978 DEBUG:boto:Params: {} 2022-09-12T19:03:52.978 DEBUG:boto:Token: None 2022-09-12T19:03:52.979 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.980 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RIOgzOGHYY3LBpiUezhkgDGji0E= 2022-09-12T19:03:52.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RIOgzOGHYY3LBpiUezhkgDGji0E=', 'Content-Length': '0'} 2022-09-12T19:03:52.983 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000212d9e4def32103a-00631f8298-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.984 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.984 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.985 DEBUG:boto:Method: HEAD 2022-09-12T19:03:52.985 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.986 DEBUG:boto:Data: 2022-09-12T19:03:52.986 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.987 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:52.988 DEBUG:boto:Port: 8000 2022-09-12T19:03:52.988 DEBUG:boto:Params: {} 2022-09-12T19:03:52.989 DEBUG:boto:Token: None 2022-09-12T19:03:52.989 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:52 GMT /jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.990 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RIOgzOGHYY3LBpiUezhkgDGji0E= 2022-09-12T19:03:52.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RIOgzOGHYY3LBpiUezhkgDGji0E=', 'Content-Length': '0'} 2022-09-12T19:03:52.993 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c556941c58e4653-00631f8298-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:52.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-12T19:03:52.994 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.995 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:52.995 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.996 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.996 DEBUG:boto:Method: GET 2022-09-12T19:03:52.997 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:52.998 DEBUG:boto:Data: 2022-09-12T19:03:52.998 DEBUG:boto:Headers: {} 2022-09-12T19:03:52.999 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:52.999 DEBUG:boto:Port: 8001 2022-09-12T19:03:53.000 DEBUG:boto:Params: {} 2022-09-12T19:03:53.000 DEBUG:boto:Token: None 2022-09-12T19:03:53.001 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:53.002 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DSshDyVyDBOgC7D3bow+dt6wwp4= 2022-09-12T19:03:53.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DSshDyVyDBOgC7D3bow+dt6wwp4=', 'Content-Length': '0'} 2022-09-12T19:03:53.005 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045a541a1b25beb37-00631f8299-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.006 DEBUG:boto:path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:53.006 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1 2022-09-12T19:03:53.007 DEBUG:boto:path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:53.007 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:53.008 DEBUG:boto:Method: GET 2022-09-12T19:03:53.008 DEBUG:boto:Path: /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:53.009 DEBUG:boto:Data: 2022-09-12T19:03:53.010 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.010 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:53.011 DEBUG:boto:Port: 8000 2022-09-12T19:03:53.011 DEBUG:boto:Params: {} 2022-09-12T19:03:53.012 DEBUG:boto:Token: None 2022-09-12T19:03:53.012 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/obj-test-zone1?versionId=ToUtj2XoS.tz3goIbvjFwiigMyzGOpL 2022-09-12T19:03:53.013 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DSshDyVyDBOgC7D3bow+dt6wwp4= 2022-09-12T19:03:53.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DSshDyVyDBOgC7D3bow+dt6wwp4=', 'Content-Length': '0'} 2022-09-12T19:03:53.016 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:31 GMT'), ('x-amz-version-id', 'ToUtj2XoS.tz3goIbvjFwiigMyzGOpL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9f85ba0f8b13f32-00631f8299-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.017 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:53.017 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:53.018 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:53.019 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:53.020 DEBUG:boto:Method: GET 2022-09-12T19:03:53.021 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:53.022 DEBUG:boto:Data: 2022-09-12T19:03:53.023 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.024 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:53.025 DEBUG:boto:Port: 8001 2022-09-12T19:03:53.026 DEBUG:boto:Params: {} 2022-09-12T19:03:53.027 DEBUG:boto:Token: None 2022-09-12T19:03:53.028 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/ 2022-09-12T19:03:53.029 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f5LArEF+c8TzcyrcDwy6LC9pK4U= 2022-09-12T19:03:53.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f5LArEF+c8TzcyrcDwy6LC9pK4U=', 'Content-Length': '0'} 2022-09-12T19:03:53.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdd1fbe5529c10bc-00631f8299-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.034 DEBUG:boto:b'jdpiak-18obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:53.035 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:53.035 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:53.036 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:53.036 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:53.037 DEBUG:boto:Method: GET 2022-09-12T19:03:53.038 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone1 2022-09-12T19:03:53.039 DEBUG:boto:Data: 2022-09-12T19:03:53.040 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.042 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:53.043 DEBUG:boto:Port: 8000 2022-09-12T19:03:53.044 DEBUG:boto:Params: {} 2022-09-12T19:03:53.045 DEBUG:boto:Token: None 2022-09-12T19:03:53.045 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/ 2022-09-12T19:03:53.046 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f5LArEF+c8TzcyrcDwy6LC9pK4U= 2022-09-12T19:03:53.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f5LArEF+c8TzcyrcDwy6LC9pK4U=', 'Content-Length': '0'} 2022-09-12T19:03:53.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008a1fd69e380c925-00631f8299-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.051 DEBUG:boto:b'jdpiak-18obj-test-zone11000falseobj-test-zone12022-09-12T18:59:23.966Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:53.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:53.053 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:53.054 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:53.055 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.056 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.057 DEBUG:boto:Method: GET 2022-09-12T19:03:53.058 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.059 DEBUG:boto:Data: 2022-09-12T19:03:53.060 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.060 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:53.061 DEBUG:boto:Port: 8001 2022-09-12T19:03:53.062 DEBUG:boto:Params: {} 2022-09-12T19:03:53.062 DEBUG:boto:Token: None 2022-09-12T19:03:53.063 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.064 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pkf0vUNc2ZMlojvjPgcxDZcSPAc= 2022-09-12T19:03:53.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pkf0vUNc2ZMlojvjPgcxDZcSPAc=', 'Content-Length': '0'} 2022-09-12T19:03:53.066 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019ab0e29f3b1f21e-00631f8299-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.068 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:53.069 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:53.070 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.071 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.072 DEBUG:boto:Method: GET 2022-09-12T19:03:53.072 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.074 DEBUG:boto:Data: 2022-09-12T19:03:53.075 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.075 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:53.076 DEBUG:boto:Port: 8000 2022-09-12T19:03:53.077 DEBUG:boto:Params: {} 2022-09-12T19:03:53.078 DEBUG:boto:Token: None 2022-09-12T19:03:53.079 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.080 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pkf0vUNc2ZMlojvjPgcxDZcSPAc= 2022-09-12T19:03:53.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pkf0vUNc2ZMlojvjPgcxDZcSPAc=', 'Content-Length': '0'} 2022-09-12T19:03:53.084 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0801b677311592f-00631f8299-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.085 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:53.086 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:53.087 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.087 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.088 DEBUG:boto:Method: HEAD 2022-09-12T19:03:53.088 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.089 DEBUG:boto:Data: 2022-09-12T19:03:53.089 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.090 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:53.091 DEBUG:boto:Port: 8001 2022-09-12T19:03:53.091 DEBUG:boto:Params: {} 2022-09-12T19:03:53.092 DEBUG:boto:Token: None 2022-09-12T19:03:53.092 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.093 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Byx2JUOHyNJjAmJIc9zL2+RJ36Y= 2022-09-12T19:03:53.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Byx2JUOHyNJjAmJIc9zL2+RJ36Y=', 'Content-Length': '0'} 2022-09-12T19:03:53.096 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001bd9e47e8ce92648-00631f8299-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.097 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:53.097 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:53.098 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.098 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.099 DEBUG:boto:Method: HEAD 2022-09-12T19:03:53.099 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.100 DEBUG:boto:Data: 2022-09-12T19:03:53.101 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.101 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:53.102 DEBUG:boto:Port: 8000 2022-09-12T19:03:53.102 DEBUG:boto:Params: {} 2022-09-12T19:03:53.103 DEBUG:boto:Token: None 2022-09-12T19:03:53.103 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.104 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Byx2JUOHyNJjAmJIc9zL2+RJ36Y= 2022-09-12T19:03:53.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Byx2JUOHyNJjAmJIc9zL2+RJ36Y=', 'Content-Length': '0'} 2022-09-12T19:03:53.107 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d7049573ffd0fc92-00631f8299-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:53.109 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:53.110 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:53.111 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.112 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.113 DEBUG:boto:Method: GET 2022-09-12T19:03:53.114 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.115 DEBUG:boto:Data: 2022-09-12T19:03:53.116 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.117 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:53.117 DEBUG:boto:Port: 8001 2022-09-12T19:03:53.118 DEBUG:boto:Params: {} 2022-09-12T19:03:53.118 DEBUG:boto:Token: None 2022-09-12T19:03:53.119 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.120 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pkf0vUNc2ZMlojvjPgcxDZcSPAc= 2022-09-12T19:03:53.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pkf0vUNc2ZMlojvjPgcxDZcSPAc=', 'Content-Length': '0'} 2022-09-12T19:03:53.123 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033964996d11a42dc-00631f8299-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.166 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:53.167 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:53.167 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.168 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.168 DEBUG:boto:Method: GET 2022-09-12T19:03:53.169 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.169 DEBUG:boto:Data: 2022-09-12T19:03:53.170 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.170 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:53.171 DEBUG:boto:Port: 8000 2022-09-12T19:03:53.172 DEBUG:boto:Params: {} 2022-09-12T19:03:53.172 DEBUG:boto:Token: None 2022-09-12T19:03:53.173 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.173 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pkf0vUNc2ZMlojvjPgcxDZcSPAc= 2022-09-12T19:03:53.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pkf0vUNc2ZMlojvjPgcxDZcSPAc=', 'Content-Length': '0'} 2022-09-12T19:03:53.176 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000701be918b05aa5d1-00631f8299-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.177 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:53.178 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:53.178 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:53.179 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:53.179 DEBUG:boto:Method: GET 2022-09-12T19:03:53.180 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:53.180 DEBUG:boto:Data: 2022-09-12T19:03:53.181 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.181 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:53.182 DEBUG:boto:Port: 8001 2022-09-12T19:03:53.183 DEBUG:boto:Params: {} 2022-09-12T19:03:53.183 DEBUG:boto:Token: None 2022-09-12T19:03:53.184 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/ 2022-09-12T19:03:53.184 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f5LArEF+c8TzcyrcDwy6LC9pK4U= 2022-09-12T19:03:53.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f5LArEF+c8TzcyrcDwy6LC9pK4U=', 'Content-Length': '0'} 2022-09-12T19:03:53.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045f0373323b05c13-00631f8299-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.189 DEBUG:boto:b'jdpiak-18obj-test-zone21000falseobj-test-zone22022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:53.189 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:53.190 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:53.191 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:53.191 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:53.192 DEBUG:boto:Method: GET 2022-09-12T19:03:53.192 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:53.193 DEBUG:boto:Data: 2022-09-12T19:03:53.193 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.194 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:53.194 DEBUG:boto:Port: 8000 2022-09-12T19:03:53.195 DEBUG:boto:Params: {} 2022-09-12T19:03:53.195 DEBUG:boto:Token: None 2022-09-12T19:03:53.196 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/ 2022-09-12T19:03:53.196 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f5LArEF+c8TzcyrcDwy6LC9pK4U= 2022-09-12T19:03:53.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f5LArEF+c8TzcyrcDwy6LC9pK4U=', 'Content-Length': '0'} 2022-09-12T19:03:53.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdd3ddcee9c26e73-00631f8299-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.200 DEBUG:boto:b'jdpiak-18obj-test-zone21000falseobj-test-zone22022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:53.201 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:53.201 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:53.202 DEBUG:boto:Method: HEAD 2022-09-12T19:03:53.202 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2 2022-09-12T19:03:53.203 DEBUG:boto:Data: 2022-09-12T19:03:53.203 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.204 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:53.204 DEBUG:boto:Port: 8001 2022-09-12T19:03:53.205 DEBUG:boto:Params: {} 2022-09-12T19:03:53.205 DEBUG:boto:Token: None 2022-09-12T19:03:53.206 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/obj-test-zone2 2022-09-12T19:03:53.207 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dy0jy99uwQv4BOox0K6VctMBiiA= 2022-09-12T19:03:53.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dy0jy99uwQv4BOox0K6VctMBiiA=', 'Content-Length': '0'} 2022-09-12T19:03:53.210 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a126bcba67c86e40-00631f8299-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.211 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:53.211 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:53.212 DEBUG:boto:Method: HEAD 2022-09-12T19:03:53.212 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2 2022-09-12T19:03:53.213 DEBUG:boto:Data: 2022-09-12T19:03:53.213 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.214 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:53.214 DEBUG:boto:Port: 8000 2022-09-12T19:03:53.215 DEBUG:boto:Params: {} 2022-09-12T19:03:53.215 DEBUG:boto:Token: None 2022-09-12T19:03:53.216 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/obj-test-zone2 2022-09-12T19:03:53.216 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dy0jy99uwQv4BOox0K6VctMBiiA= 2022-09-12T19:03:53.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dy0jy99uwQv4BOox0K6VctMBiiA=', 'Content-Length': '0'} 2022-09-12T19:03:53.219 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fdec780467c8d565-00631f8299-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-12T19:03:53.221 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:53.221 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:53.222 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.222 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.223 DEBUG:boto:Method: GET 2022-09-12T19:03:53.223 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.224 DEBUG:boto:Data: 2022-09-12T19:03:53.224 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.225 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:53.225 DEBUG:boto:Port: 8001 2022-09-12T19:03:53.226 DEBUG:boto:Params: {} 2022-09-12T19:03:53.226 DEBUG:boto:Token: None 2022-09-12T19:03:53.227 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.228 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pkf0vUNc2ZMlojvjPgcxDZcSPAc= 2022-09-12T19:03:53.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pkf0vUNc2ZMlojvjPgcxDZcSPAc=', 'Content-Length': '0'} 2022-09-12T19:03:53.230 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9b1f2f93cc8b449-00631f8299-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.231 DEBUG:boto:path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:53.232 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2 2022-09-12T19:03:53.232 DEBUG:boto:path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.233 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.233 DEBUG:boto:Method: GET 2022-09-12T19:03:53.234 DEBUG:boto:Path: /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.234 DEBUG:boto:Data: 2022-09-12T19:03:53.235 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.235 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:53.236 DEBUG:boto:Port: 8000 2022-09-12T19:03:53.236 DEBUG:boto:Params: {} 2022-09-12T19:03:53.237 DEBUG:boto:Token: None 2022-09-12T19:03:53.238 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/obj-test-zone2?versionId=DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW 2022-09-12T19:03:53.238 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pkf0vUNc2ZMlojvjPgcxDZcSPAc= 2022-09-12T19:03:53.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pkf0vUNc2ZMlojvjPgcxDZcSPAc=', 'Content-Length': '0'} 2022-09-12T19:03:53.241 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'DPksivOWVYhJDrEwQqHYV7wK1Ul1JkW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cfb373b2de177047-00631f8299-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.242 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:53.242 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:53.243 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:53.243 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:53.244 DEBUG:boto:Method: GET 2022-09-12T19:03:53.245 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:53.245 DEBUG:boto:Data: 2022-09-12T19:03:53.246 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.246 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:53.247 DEBUG:boto:Port: 8001 2022-09-12T19:03:53.247 DEBUG:boto:Params: {} 2022-09-12T19:03:53.248 DEBUG:boto:Token: None 2022-09-12T19:03:53.248 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/ 2022-09-12T19:03:53.249 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f5LArEF+c8TzcyrcDwy6LC9pK4U= 2022-09-12T19:03:53.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f5LArEF+c8TzcyrcDwy6LC9pK4U=', 'Content-Length': '0'} 2022-09-12T19:03:53.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d060c54bc0fd176-00631f8299-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.294 DEBUG:boto:b'jdpiak-18obj-test-zone21000falseobj-test-zone22022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:53.295 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:53.296 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:53.296 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:53.297 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:53.297 DEBUG:boto:Method: GET 2022-09-12T19:03:53.298 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone2 2022-09-12T19:03:53.298 DEBUG:boto:Data: 2022-09-12T19:03:53.299 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.299 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:53.300 DEBUG:boto:Port: 8000 2022-09-12T19:03:53.300 DEBUG:boto:Params: {} 2022-09-12T19:03:53.301 DEBUG:boto:Token: None 2022-09-12T19:03:53.302 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/ 2022-09-12T19:03:53.302 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f5LArEF+c8TzcyrcDwy6LC9pK4U= 2022-09-12T19:03:53.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f5LArEF+c8TzcyrcDwy6LC9pK4U=', 'Content-Length': '0'} 2022-09-12T19:03:53.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080fc2bf9d367b277-00631f8299-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.306 DEBUG:boto:b'jdpiak-18obj-test-zone21000falseobj-test-zone22022-09-12T18:59:24.127Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:53.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:53.308 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:53.308 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:53.309 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.309 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.310 DEBUG:boto:Method: GET 2022-09-12T19:03:53.311 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.311 DEBUG:boto:Data: 2022-09-12T19:03:53.312 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.312 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:53.313 DEBUG:boto:Port: 8001 2022-09-12T19:03:53.313 DEBUG:boto:Params: {} 2022-09-12T19:03:53.314 DEBUG:boto:Token: None 2022-09-12T19:03:53.314 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.315 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RWnO+OOFTC7ELNRTLiz8atWjiHQ= 2022-09-12T19:03:53.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RWnO+OOFTC7ELNRTLiz8atWjiHQ=', 'Content-Length': '0'} 2022-09-12T19:03:53.318 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f733c39e72147704-00631f8299-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.319 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:53.319 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:53.320 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.320 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.321 DEBUG:boto:Method: GET 2022-09-12T19:03:53.321 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.322 DEBUG:boto:Data: 2022-09-12T19:03:53.323 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.323 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:53.324 DEBUG:boto:Port: 8000 2022-09-12T19:03:53.324 DEBUG:boto:Params: {} 2022-09-12T19:03:53.325 DEBUG:boto:Token: None 2022-09-12T19:03:53.325 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.326 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RWnO+OOFTC7ELNRTLiz8atWjiHQ= 2022-09-12T19:03:53.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RWnO+OOFTC7ELNRTLiz8atWjiHQ=', 'Content-Length': '0'} 2022-09-12T19:03:53.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000426c98b8458b97ea-00631f8299-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.330 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:53.330 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:53.331 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.331 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.332 DEBUG:boto:Method: HEAD 2022-09-12T19:03:53.332 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.333 DEBUG:boto:Data: 2022-09-12T19:03:53.333 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.334 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:53.334 DEBUG:boto:Port: 8001 2022-09-12T19:03:53.335 DEBUG:boto:Params: {} 2022-09-12T19:03:53.335 DEBUG:boto:Token: None 2022-09-12T19:03:53.336 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.337 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vEVlRPQdmhfmgCi3Jcq8WfvzRjM= 2022-09-12T19:03:53.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vEVlRPQdmhfmgCi3Jcq8WfvzRjM=', 'Content-Length': '0'} 2022-09-12T19:03:53.339 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005fadf046aa3d3243-00631f8299-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.340 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:53.341 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:53.341 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.342 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.342 DEBUG:boto:Method: HEAD 2022-09-12T19:03:53.343 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.343 DEBUG:boto:Data: 2022-09-12T19:03:53.344 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.344 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:53.345 DEBUG:boto:Port: 8000 2022-09-12T19:03:53.346 DEBUG:boto:Params: {} 2022-09-12T19:03:53.346 DEBUG:boto:Token: None 2022-09-12T19:03:53.347 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.348 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vEVlRPQdmhfmgCi3Jcq8WfvzRjM= 2022-09-12T19:03:53.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vEVlRPQdmhfmgCi3Jcq8WfvzRjM=', 'Content-Length': '0'} 2022-09-12T19:03:53.350 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de5577d5810a58c2-00631f8299-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:53.352 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:53.352 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:53.353 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.353 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.354 DEBUG:boto:Method: GET 2022-09-12T19:03:53.355 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.355 DEBUG:boto:Data: 2022-09-12T19:03:53.356 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.356 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:53.357 DEBUG:boto:Port: 8001 2022-09-12T19:03:53.357 DEBUG:boto:Params: {} 2022-09-12T19:03:53.358 DEBUG:boto:Token: None 2022-09-12T19:03:53.359 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.359 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RWnO+OOFTC7ELNRTLiz8atWjiHQ= 2022-09-12T19:03:53.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RWnO+OOFTC7ELNRTLiz8atWjiHQ=', 'Content-Length': '0'} 2022-09-12T19:03:53.362 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001464dc5c01eb1d29-00631f8299-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.363 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:53.363 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:53.364 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.364 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.365 DEBUG:boto:Method: GET 2022-09-12T19:03:53.365 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.366 DEBUG:boto:Data: 2022-09-12T19:03:53.366 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.367 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:53.368 DEBUG:boto:Port: 8000 2022-09-12T19:03:53.368 DEBUG:boto:Params: {} 2022-09-12T19:03:53.369 DEBUG:boto:Token: None 2022-09-12T19:03:53.369 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.370 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RWnO+OOFTC7ELNRTLiz8atWjiHQ= 2022-09-12T19:03:53.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RWnO+OOFTC7ELNRTLiz8atWjiHQ=', 'Content-Length': '0'} 2022-09-12T19:03:53.373 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005cf920145b6f2963-00631f8299-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.374 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:53.375 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:53.375 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:53.376 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:53.376 DEBUG:boto:Method: GET 2022-09-12T19:03:53.377 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:53.377 DEBUG:boto:Data: 2022-09-12T19:03:53.378 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.378 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:53.379 DEBUG:boto:Port: 8001 2022-09-12T19:03:53.379 DEBUG:boto:Params: {} 2022-09-12T19:03:53.380 DEBUG:boto:Token: None 2022-09-12T19:03:53.381 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/ 2022-09-12T19:03:53.381 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f5LArEF+c8TzcyrcDwy6LC9pK4U= 2022-09-12T19:03:53.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f5LArEF+c8TzcyrcDwy6LC9pK4U=', 'Content-Length': '0'} 2022-09-12T19:03:53.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000868a472c26666c43-00631f8299-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.430 DEBUG:boto:b'jdpiak-18obj-test-zone31000falseobj-test-zone32022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:53.431 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:53.432 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:53.432 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:53.433 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:53.433 DEBUG:boto:Method: GET 2022-09-12T19:03:53.434 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:53.434 DEBUG:boto:Data: 2022-09-12T19:03:53.435 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.435 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:53.436 DEBUG:boto:Port: 8000 2022-09-12T19:03:53.436 DEBUG:boto:Params: {} 2022-09-12T19:03:53.437 DEBUG:boto:Token: None 2022-09-12T19:03:53.438 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/ 2022-09-12T19:03:53.438 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f5LArEF+c8TzcyrcDwy6LC9pK4U= 2022-09-12T19:03:53.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f5LArEF+c8TzcyrcDwy6LC9pK4U=', 'Content-Length': '0'} 2022-09-12T19:03:53.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059d18dea32336803-00631f8299-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.442 DEBUG:boto:b'jdpiak-18obj-test-zone31000falseobj-test-zone32022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:53.443 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:53.444 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:53.444 DEBUG:boto:Method: HEAD 2022-09-12T19:03:53.445 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3 2022-09-12T19:03:53.445 DEBUG:boto:Data: 2022-09-12T19:03:53.446 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.446 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:53.447 DEBUG:boto:Port: 8001 2022-09-12T19:03:53.447 DEBUG:boto:Params: {} 2022-09-12T19:03:53.448 DEBUG:boto:Token: None 2022-09-12T19:03:53.448 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/obj-test-zone3 2022-09-12T19:03:53.449 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m9cw6A8RF2dSzPyxhnI1dVhfOm0= 2022-09-12T19:03:53.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m9cw6A8RF2dSzPyxhnI1dVhfOm0=', 'Content-Length': '0'} 2022-09-12T19:03:53.452 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f69bae03f310283f-00631f8299-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.453 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:53.454 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:53.454 DEBUG:boto:Method: HEAD 2022-09-12T19:03:53.455 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3 2022-09-12T19:03:53.455 DEBUG:boto:Data: 2022-09-12T19:03:53.456 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.456 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:53.457 DEBUG:boto:Port: 8000 2022-09-12T19:03:53.457 DEBUG:boto:Params: {} 2022-09-12T19:03:53.458 DEBUG:boto:Token: None 2022-09-12T19:03:53.458 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/obj-test-zone3 2022-09-12T19:03:53.459 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m9cw6A8RF2dSzPyxhnI1dVhfOm0= 2022-09-12T19:03:53.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m9cw6A8RF2dSzPyxhnI1dVhfOm0=', 'Content-Length': '0'} 2022-09-12T19:03:53.462 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007aef092f5acdbc92-00631f8299-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-12T19:03:53.463 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:53.464 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:53.465 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.465 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.466 DEBUG:boto:Method: GET 2022-09-12T19:03:53.466 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.467 DEBUG:boto:Data: 2022-09-12T19:03:53.467 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.468 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:53.468 DEBUG:boto:Port: 8001 2022-09-12T19:03:53.469 DEBUG:boto:Params: {} 2022-09-12T19:03:53.469 DEBUG:boto:Token: None 2022-09-12T19:03:53.470 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.471 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RWnO+OOFTC7ELNRTLiz8atWjiHQ= 2022-09-12T19:03:53.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RWnO+OOFTC7ELNRTLiz8atWjiHQ=', 'Content-Length': '0'} 2022-09-12T19:03:53.474 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b32c0ead564627a-00631f8299-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.475 DEBUG:boto:path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:53.475 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3 2022-09-12T19:03:53.476 DEBUG:boto:path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.476 DEBUG:boto:auth_path=/jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.477 DEBUG:boto:Method: GET 2022-09-12T19:03:53.478 DEBUG:boto:Path: /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.478 DEBUG:boto:Data: 2022-09-12T19:03:53.479 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.479 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:53.480 DEBUG:boto:Port: 8000 2022-09-12T19:03:53.480 DEBUG:boto:Params: {} 2022-09-12T19:03:53.481 DEBUG:boto:Token: None 2022-09-12T19:03:53.481 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/obj-test-zone3?versionId=iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z 2022-09-12T19:03:53.482 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RWnO+OOFTC7ELNRTLiz8atWjiHQ= 2022-09-12T19:03:53.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RWnO+OOFTC7ELNRTLiz8atWjiHQ=', 'Content-Length': '0'} 2022-09-12T19:03:53.485 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:02:32 GMT'), ('x-amz-version-id', 'iRI0k3lKvGCJhQB-tl1a3AyYnW8xE2Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000edfd92108d0e7085-00631f8299-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.530 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:53.531 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:53.532 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:53.533 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:53.533 DEBUG:boto:Method: GET 2022-09-12T19:03:53.534 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:53.534 DEBUG:boto:Data: 2022-09-12T19:03:53.535 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.535 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:03:53.536 DEBUG:boto:Port: 8001 2022-09-12T19:03:53.536 DEBUG:boto:Params: {} 2022-09-12T19:03:53.537 DEBUG:boto:Token: None 2022-09-12T19:03:53.538 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/ 2022-09-12T19:03:53.538 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f5LArEF+c8TzcyrcDwy6LC9pK4U= 2022-09-12T19:03:53.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f5LArEF+c8TzcyrcDwy6LC9pK4U=', 'Content-Length': '0'} 2022-09-12T19:03:53.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bb9ad588762d05b-00631f8299-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.543 DEBUG:boto:b'jdpiak-18obj-test-zone31000falseobj-test-zone32022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:53.543 DEBUG:boto:path=/jdpiak-18/ 2022-09-12T19:03:53.544 DEBUG:boto:auth_path=/jdpiak-18/ 2022-09-12T19:03:53.545 DEBUG:boto:path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:53.545 DEBUG:boto:auth_path=/jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:53.546 DEBUG:boto:Method: GET 2022-09-12T19:03:53.546 DEBUG:boto:Path: /jdpiak-18/?prefix=obj-test-zone3 2022-09-12T19:03:53.547 DEBUG:boto:Data: 2022-09-12T19:03:53.547 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.548 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:03:53.548 DEBUG:boto:Port: 8000 2022-09-12T19:03:53.549 DEBUG:boto:Params: {} 2022-09-12T19:03:53.549 DEBUG:boto:Token: None 2022-09-12T19:03:53.550 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-18/ 2022-09-12T19:03:53.551 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f5LArEF+c8TzcyrcDwy6LC9pK4U= 2022-09-12T19:03:53.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f5LArEF+c8TzcyrcDwy6LC9pK4U=', 'Content-Length': '0'} 2022-09-12T19:03:53.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dd9da377ba3ca66-00631f8299-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.555 DEBUG:boto:b'jdpiak-18obj-test-zone31000falseobj-test-zone32022-09-12T18:59:24.316Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:03:53.556 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-18 zones={test-zone3, test-zone2} 2022-09-12T19:03:53.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2022-09-12T19:03:53.561 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:03:53.561 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:03:53.562 DEBUG:boto:Method: PUT 2022-09-12T19:03:53.562 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:03:53.563 DEBUG:boto:Data: 2022-09-12T19:03:53.563 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.564 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:53.564 DEBUG:boto:Port: 8000 2022-09-12T19:03:53.565 DEBUG:boto:Params: {} 2022-09-12T19:03:53.565 DEBUG:boto:Token: None 2022-09-12T19:03:53.566 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-19/ 2022-09-12T19:03:53.566 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:C3IFwWC9MD92GzZh1mXd6Wp1JqI= 2022-09-12T19:03:53.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:C3IFwWC9MD92GzZh1mXd6Wp1JqI=', 'Content-Length': '0'} 2022-09-12T19:03:53.584 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008e75507db61bda71-00631f8299-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.585 DEBUG:rgw_multi.tests:created bucket=jdpiak-19 2022-09-12T19:03:53.585 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:03:53.586 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:03:53.587 DEBUG:boto:path=/jdpiak-19/?versioning 2022-09-12T19:03:53.587 DEBUG:boto:auth_path=/jdpiak-19/?versioning 2022-09-12T19:03:53.587 DEBUG:boto:Method: PUT 2022-09-12T19:03:53.588 DEBUG:boto:Path: /jdpiak-19/?versioning 2022-09-12T19:03:53.588 DEBUG:boto:Data: Enabled Disabled 2022-09-12T19:03:53.589 DEBUG:boto:Headers: {} 2022-09-12T19:03:53.589 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:53.590 DEBUG:boto:Port: 8000 2022-09-12T19:03:53.590 DEBUG:boto:Params: {} 2022-09-12T19:03:53.591 DEBUG:boto:Token: None 2022-09-12T19:03:53.592 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:03:53 GMT /jdpiak-19/?versioning 2022-09-12T19:03:53.592 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E5Fm+ttHZOt4sbYQXCGgASWzJDg= 2022-09-12T19:03:53.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E5Fm+ttHZOt4sbYQXCGgASWzJDg=', 'Content-Length': '228'} 2022-09-12T19:03:53.600 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b40134e9a66f8dd2-00631f8299-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:03:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:53.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:03:53.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:03:53.602 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:03:53.665 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:03:53.665 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:03:53.726 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=c0f7a512-c940-4d1f-8adf-3484630f876b 2022-09-12T19:03:53.743 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:03:53.743 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:03:53.744 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:03:53.744 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:47:27.976965Z" 2022-09-12T19:03:53.744 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:03:53.744 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:03:53.745 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:03:53.745 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:03:53.746 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:03:53.746 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:03:53.746 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663009433.597955_184.1", 2022-09-12T19:03:53.746 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:03:53.597955Z" 2022-09-12T19:03:53.747 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:03:53.747 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:03:53.747 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:03:53.748 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:49:40.904048Z" 2022-09-12T19:03:53.748 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:03:53.748 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:03:53.750 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009433.597955_184.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:03:53.750 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T19:03:53.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:03:53.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:03:53.753 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:03:53.807 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:03:53.807 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:03:53.848 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:03:53.845+0000 7ff17f8e5600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:03:53.849 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:03:53.846+0000 7ff17f8e5600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:03:53.864 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:03:53.865 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:03:53.865 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:03:53.866 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:03:53.866 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:03:53.867 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:03:53.867 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:03:53.867 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:03:53.867 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:03:53.868 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:03:53.868 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:03:53.868 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:03:53.869 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:03:53.869 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:03:53.869 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:03:53.869 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:03:53.870 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:03:53.870 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:03:53.870 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:03:53.871 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:03:53.871 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:03:53.871 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:03:53.871 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:03:53.872 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:03:53.872 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:03:53.873 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:03:53.873 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:03:53.873 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:03:53.873 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:03:53.874 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:03:53.874 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:03:53.874 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:03:53.875 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:03:53.875 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:03:53.875 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:03:53.876 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:03:53.876 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:03:53.876 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663009433.572798_179.1", 2022-09-12T19:03:53.876 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:03:53.877 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:03:53.877 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:03:53.877 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:03:53.572798Z", 2022-09-12T19:03:53.877 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:03:53.878 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:03:53.878 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:03:53.878 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:03:53.878 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:03:53.879 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:03:53.879 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:03:53.879 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:03:53.879 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:03:53.880 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:03:53.880 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:03:53.880 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:03:53.881 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:03:53.881 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:03:53.881 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:03:53.881 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:03:53.882 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:03:53.882 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:03:53.882 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:03:53.882 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:03:53.883 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:03:53.883 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:03:53Z" 2022-09-12T19:03:53.883 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:03:53.884 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009433.572798_179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:03:53.572798Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:03:53Z" } 2022-09-12T19:03:53.885 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009433.572798_179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:03:53.572798Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:03:53.886 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009433.597955_184.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:03:53.886 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009433.572798_179.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:03:53.887 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663009433.597955_184.1 target=1_1663009433.572798_179.1 2022-09-12T19:03:58.887 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:03:58.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:03:58.889 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:03:58.947 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:03:58.947 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:03:58.990 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:03:58.987+0000 7f826ad30600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:03:58.991 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:03:58.988+0000 7f826ad30600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:03:59.004 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:03:59.005 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:03:59.006 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:03:59.006 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:03:59.007 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:03:59.007 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:03:59.008 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:03:59.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:03:59.009 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:03:59.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:03:59.010 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:03:59.010 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:03:59.011 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:03:59.011 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:03:59.012 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:03:59.012 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:03:59.013 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:03:59.013 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:03:59.014 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:03:59.014 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:03:59.015 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:03:59.015 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:03:59.016 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:03:59.016 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:03:59.017 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:03:59.017 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:03:59.018 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:03:59.018 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:03:59.019 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:03:59.019 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:03:59.020 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:03:59.020 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:03:59.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:03:59.021 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:03:59.022 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:03:59.022 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:03:59.023 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:03:59.023 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663009433.597955_184.1", 2022-09-12T19:03:59.024 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:03:59.025 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:03:59.025 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:03:59.025 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:03:53.597955Z", 2022-09-12T19:03:59.026 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:03:59.027 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:03:59.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:03:59.028 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:03:59.028 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:03:59.029 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:03:59.029 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:03:59.030 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:03:59.030 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:03:59.031 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:03:59.031 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:03:59.032 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:03:59.032 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:03:59.033 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:03:59.033 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:03:59.034 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:03:59.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:03:59.035 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:03:59.035 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:03:59.035 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:03:59.036 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:03:59.036 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:03:58Z" 2022-09-12T19:03:59.037 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:03:59.039 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009433.597955_184.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:03:53.597955Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:03:58Z" } 2022-09-12T19:03:59.040 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009433.597955_184.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:03:53.597955Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:03:59.041 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009433.597955_184.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:03:59.042 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009433.597955_184.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:03:59.043 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T19:03:59.044 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T19:03:59.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:03:59.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:03:59.047 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:03:59.106 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:03:59.106 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:03:59.142 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:59.138+0000 7fc3688fc600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:03:59.142 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:03:59.139+0000 7fc3688fc600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:03:59.155 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:03:59.155 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:03:59.155 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:03:59.156 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:03:59.156 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:03:59.156 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:03:59.156 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:03:59.157 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:03:59.157 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:03:59.157 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:03:59.158 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:03:59.158 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:03:59.158 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:03:59.159 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:03:59.159 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:03:59.159 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:03:59.160 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:03:59.160 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:03:59.160 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:03:59.160 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:03:59.161 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:03:59.161 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:03:59.161 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:03:59.161 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:03:59.162 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:03:59.162 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:03:59.162 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:03:59.162 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:03:59.163 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:03:59.163 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:03:59.163 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:03:59.163 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:03:59.164 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:03:59.164 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:03:59.164 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:03:59.164 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:03:59.165 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:03:59.165 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663009433.597955_184.1", 2022-09-12T19:03:59.165 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:03:59.165 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:03:59.166 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:03:59.166 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:03:53.597955Z", 2022-09-12T19:03:59.166 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:03:59.166 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:03:59.167 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:03:59.167 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:03:59.167 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:03:59.167 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:03:59.168 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:03:59.168 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:03:59.168 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:03:59.169 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:03:59.169 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:03:59.169 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:03:59.169 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:03:59.170 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:03:59.170 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:03:59.170 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:03:59.170 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:03:59.171 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:03:59.171 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:03:59.171 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:03:59.171 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:03:59.172 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:03:59Z" 2022-09-12T19:03:59.172 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:03:59.173 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009433.597955_184.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:03:53.597955Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:03:59Z" } 2022-09-12T19:03:59.174 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009433.597955_184.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:03:53.597955Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:03:59.174 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009433.597955_184.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:03:59.175 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009433.597955_184.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:03:59.175 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T19:03:59.176 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T19:03:59.176 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:03:59.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:03:59.178 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:03:59.228 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:03:59.229 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:03:59.268 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:03:59.265+0000 7fa8ee919600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:03:59.269 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:03:59.266+0000 7fa8ee919600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:03:59.282 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:03:59.283 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:03:59.283 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:03:59.283 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:03:59.284 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:03:59.284 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:03:59.284 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:03:59.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:03:59.285 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:03:59.285 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:03:59.285 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:03:59.286 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:03:59.286 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:03:59.287 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:03:59.287 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:03:59.288 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:03:59.288 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:03:59.288 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:03:59.289 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:03:59.289 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:03:59.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:03:59.289 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:03:59.290 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:03:59.290 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:03:59.291 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:03:59.291 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:03:59.292 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:03:59.292 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:03:59.293 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:03:59.293 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:03:59.294 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:03:59.294 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:03:59.295 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:03:59.295 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:03:59.296 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:03:59.297 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:03:59.297 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:03:59.298 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663009433.597955_184.1", 2022-09-12T19:03:59.298 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:03:59.299 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:03:59.299 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:03:59.300 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:03:53.597955Z", 2022-09-12T19:03:59.300 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:03:59.301 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:03:59.301 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:03:59.302 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:03:59.302 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:03:59.303 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:03:59.303 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:03:59.304 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:03:59.304 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:03:59.305 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:03:59.306 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:03:59.306 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:03:59.307 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:03:59.307 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:03:59.308 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:03:59.308 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:03:59.309 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:03:59.309 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:03:59.310 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:03:59.310 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:03:59.311 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:03:59.311 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:03:59Z" 2022-09-12T19:03:59.312 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:03:59.314 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009433.597955_184.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:03:53.597955Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:03:59Z" } 2022-09-12T19:03:59.315 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009433.597955_184.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:03:53.597955Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:03:59.316 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009433.597955_184.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:03:59.317 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009433.597955_184.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:03:59.317 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T19:03:59.318 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:03:59.319 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:03:59.320 DEBUG:boto:Method: HEAD 2022-09-12T19:03:59.321 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:03:59.322 DEBUG:boto:Data: 2022-09-12T19:03:59.323 DEBUG:boto:Headers: {} 2022-09-12T19:03:59.324 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:59.325 DEBUG:boto:Port: 8000 2022-09-12T19:03:59.326 DEBUG:boto:Params: {} 2022-09-12T19:03:59.327 DEBUG:boto:Token: None 2022-09-12T19:03:59.328 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:03:59 GMT /jdpiak-19/ 2022-09-12T19:03:59.329 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/OvkicviauDpFkWuZy3HQ5chgBM= 2022-09-12T19:03:59.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:03:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/OvkicviauDpFkWuZy3HQ5chgBM=', 'Content-Length': '0'} 2022-09-12T19:03:59.333 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d25026f74f1f5242-00631f829f-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:59.334 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:03:59.335 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:03:59.335 DEBUG:boto:Method: PUT 2022-09-12T19:03:59.336 DEBUG:boto:Path: /jdpiak-19/dummy 2022-09-12T19:03:59.336 DEBUG:boto:Data: 2022-09-12T19:03:59.337 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-12T19:03:59.337 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:03:59.338 DEBUG:boto:Port: 8000 2022-09-12T19:03:59.339 DEBUG:boto:Params: {} 2022-09-12T19:03:59.339 DEBUG:boto:Token: None 2022-09-12T19:03:59.340 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 12 Sep 2022 19:03:59 GMT /jdpiak-19/dummy 2022-09-12T19:03:59.340 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IpTjq64LDiFIvczIWW4v6n14HX0= 2022-09-12T19:03:59.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:03:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IpTjq64LDiFIvczIWW4v6n14HX0='} 2022-09-12T19:03:59.352 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-amz-request-id', 'tx000002081bb27c845f48e-00631f829f-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:03:59.353 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-19 2022-09-12T19:03:59.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:59.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:03:59.356 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:03:59.424 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:03:59.425 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:03:59.461 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:03:59.458+0000 7f1df828f600 1 waiting to reach incremental sync.. 2022-09-12T19:04:04.463 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:04:04.460+0000 7f1df828f600 1 bucket sync caught up with source: 2022-09-12T19:04:04.464 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000002.11.10, , , , , ] 2022-09-12T19:04:04.464 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000002.11.10, , , , , ] 2022-09-12T19:04:04.465 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:04:04.460+0000 7f1df828f600 0 bucket checkpoint complete 2022-09-12T19:04:04.479 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-19 2022-09-12T19:04:04.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:04:04.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:04:04.481 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:04:04.550 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:04:04.551 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:04:04.609 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:04:04.601+0000 7f9e62c64600 1 bucket sync caught up with source: 2022-09-12T19:04:04.609 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000002.11.10, , , , , ] 2022-09-12T19:04:04.610 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000002.11.10, , , , , ] 2022-09-12T19:04:04.611 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:04:04.601+0000 7f9e62c64600 0 bucket checkpoint complete 2022-09-12T19:04:04.625 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-19 2022-09-12T19:04:04.626 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:04:04.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:04:04.639 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:04:04.703 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:04:04.703 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:04:04.738 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:04:04.735+0000 7fbd0ccce600 1 waiting to reach incremental sync.. 2022-09-12T19:04:09.740 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:04:09.737+0000 7fbd0ccce600 1 bucket sync caught up with source: 2022-09-12T19:04:09.741 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000002.11.10, , , , , ] 2022-09-12T19:04:09.742 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000002.11.10, , , , , ] 2022-09-12T19:04:09.742 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:04:09.737+0000 7fbd0ccce600 0 bucket checkpoint complete 2022-09-12T19:04:09.765 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-19 2022-09-12T19:04:09.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:04:09.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:04:09.767 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:04:09.843 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:04:09.844 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:04:09.881 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:04:09.878+0000 7f38348f2600 1 waiting to reach incremental sync.. 2022-09-12T19:04:14.882 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:04:14.879+0000 7f38348f2600 1 waiting to reach incremental sync.. 2022-09-12T19:04:19.882 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:04:19.880+0000 7f38348f2600 1 waiting to reach incremental sync.. 2022-09-12T19:04:24.885 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:04:24.882+0000 7f38348f2600 1 bucket sync caught up with source: 2022-09-12T19:04:24.885 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000004.30.9, , , , , ] 2022-09-12T19:04:24.885 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000002.26.10, , , , , ] 2022-09-12T19:04:24.886 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:04:24.882+0000 7f38348f2600 0 bucket checkpoint complete 2022-09-12T19:04:24.905 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-19 2022-09-12T19:04:24.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:04:24.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:04:24.907 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:04:24.973 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:04:24.974 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:04:25.025 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:04:25.023+0000 7f3bb3dbe600 1 waiting for incremental sync to catch up: 2022-09-12T19:04:25.025 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000002.26.10, , , , , ] 2022-09-12T19:04:25.025 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.30.9, , , , , ] 2022-09-12T19:04:30.026 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:04:30.024+0000 7f3bb3dbe600 1 waiting for incremental sync to catch up: 2022-09-12T19:04:30.027 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000002.26.10, , , , , ] 2022-09-12T19:04:30.027 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.30.9, , , , , ] 2022-09-12T19:04:35.028 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:04:35.025+0000 7f3bb3dbe600 1 waiting for incremental sync to catch up: 2022-09-12T19:04:35.029 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000002.26.10, , , , , ] 2022-09-12T19:04:35.029 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.30.9, , , , , ] 2022-09-12T19:04:40.029 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:04:40.026+0000 7f3bb3dbe600 1 waiting for incremental sync to catch up: 2022-09-12T19:04:40.030 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000002.26.10, , , , , ] 2022-09-12T19:04:40.030 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.30.9, , , , , ] 2022-09-12T19:04:45.032 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:04:45.028+0000 7f3bb3dbe600 1 bucket sync caught up with source: 2022-09-12T19:04:45.033 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000004.30.9, , , , , ] 2022-09-12T19:04:45.034 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.30.9, , , , , ] 2022-09-12T19:04:45.034 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:04:45.028+0000 7f3bb3dbe600 0 bucket checkpoint complete 2022-09-12T19:04:45.053 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-19 2022-09-12T19:04:45.054 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-19 2022-09-12T19:04:45.055 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:04:45.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:04:45.057 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:04:45.122 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:04:45.122 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:04:45.180 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:04:45.177+0000 7f912fba1600 1 waiting for incremental sync to catch up: 2022-09-12T19:04:45.181 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000003.39.9, , , , , ] 2022-09-12T19:04:45.181 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.41.9, , , , , ] 2022-09-12T19:04:50.181 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:04:50.178+0000 7f912fba1600 1 waiting for incremental sync to catch up: 2022-09-12T19:04:50.182 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000003.39.9, , , , , ] 2022-09-12T19:04:50.182 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.41.9, , , , , ] 2022-09-12T19:04:55.183 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:04:55.180+0000 7f912fba1600 1 waiting for incremental sync to catch up: 2022-09-12T19:04:55.184 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000003.39.9, , , , , ] 2022-09-12T19:04:55.184 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.41.9, , , , , ] 2022-09-12T19:05:00.184 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:05:00.181+0000 7f912fba1600 1 waiting for incremental sync to catch up: 2022-09-12T19:05:00.185 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000003.39.9, , , , , ] 2022-09-12T19:05:00.185 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.41.9, , , , , ] 2022-09-12T19:05:05.186 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:05:05.183+0000 7f912fba1600 1 bucket sync caught up with source: 2022-09-12T19:05:05.187 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000004.41.9, , , , , ] 2022-09-12T19:05:05.187 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.41.9, , , , , ] 2022-09-12T19:05:05.187 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:05:05.183+0000 7f912fba1600 0 bucket checkpoint complete 2022-09-12T19:05:05.205 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-19 2022-09-12T19:05:05.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:05:05.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:05:05.209 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:05:05.274 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:05:05.274 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:05:05.319 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:05:05.316+0000 7fc897f7d600 1 bucket sync caught up with source: 2022-09-12T19:05:05.320 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000004.41.9, , , , , ] 2022-09-12T19:05:05.320 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000004.41.9, , , , , ] 2022-09-12T19:05:05.320 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:05:05.316+0000 7fc897f7d600 0 bucket checkpoint complete 2022-09-12T19:05:05.343 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-19 2022-09-12T19:05:05.343 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-19 zones={test-zone2, test-zone1} 2022-09-12T19:05:05.344 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:05.344 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:05.346 DEBUG:boto:Method: HEAD 2022-09-12T19:05:05.346 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:05.347 DEBUG:boto:Data: 2022-09-12T19:05:05.348 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.349 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:05.350 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.351 DEBUG:boto:Params: {} 2022-09-12T19:05:05.352 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:05:05.353 DEBUG:boto:Token: None 2022-09-12T19:05:05.353 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/ 2022-09-12T19:05:05.354 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oXbJ2KSvSrXP2QqdFKquJbrZuBY= 2022-09-12T19:05:05.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oXbJ2KSvSrXP2QqdFKquJbrZuBY=', 'Content-Length': '0'} 2022-09-12T19:05:05.360 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000773f498d5af06df6-00631f82e1-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.360 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:05.361 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:05.362 DEBUG:boto:Method: HEAD 2022-09-12T19:05:05.362 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:05.363 DEBUG:boto:Data: 2022-09-12T19:05:05.364 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.364 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:05.365 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.366 DEBUG:boto:Params: {} 2022-09-12T19:05:05.367 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:05:05.367 DEBUG:boto:Token: None 2022-09-12T19:05:05.368 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/ 2022-09-12T19:05:05.368 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oXbJ2KSvSrXP2QqdFKquJbrZuBY= 2022-09-12T19:05:05.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oXbJ2KSvSrXP2QqdFKquJbrZuBY=', 'Content-Length': '0'} 2022-09-12T19:05:05.373 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007432117f3572209-00631f82e1-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.374 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:05:05.375 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:05.376 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:05.376 DEBUG:boto:path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.377 DEBUG:boto:auth_path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.377 DEBUG:boto:Method: GET 2022-09-12T19:05:05.378 DEBUG:boto:Path: /jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.378 DEBUG:boto:Data: 2022-09-12T19:05:05.379 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.379 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:05.380 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.380 DEBUG:boto:Params: {} 2022-09-12T19:05:05.381 DEBUG:boto:Token: None 2022-09-12T19:05:05.381 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/?versions 2022-09-12T19:05:05.382 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L0wx09ybeEjwZvA6kGNiA0+cBk4= 2022-09-12T19:05:05.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L0wx09ybeEjwZvA6kGNiA0+cBk4=', 'Content-Length': '0'} 2022-09-12T19:05:05.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004c442214e924713-00631f82e1-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.386 DEBUG:boto:b'jdpiak-19999falsedummydAjTD8IRanrz7e2F.KgapZJi58DocMetrue2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:05:05.387 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:05:05.388 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:05:05.388 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:05.389 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:05.389 DEBUG:boto:path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.390 DEBUG:boto:auth_path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.390 DEBUG:boto:Method: GET 2022-09-12T19:05:05.391 DEBUG:boto:Path: /jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.391 DEBUG:boto:Data: 2022-09-12T19:05:05.392 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.392 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:05.393 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.393 DEBUG:boto:Params: {} 2022-09-12T19:05:05.394 DEBUG:boto:Token: None 2022-09-12T19:05:05.395 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/?versions 2022-09-12T19:05:05.396 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L0wx09ybeEjwZvA6kGNiA0+cBk4= 2022-09-12T19:05:05.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L0wx09ybeEjwZvA6kGNiA0+cBk4=', 'Content-Length': '0'} 2022-09-12T19:05:05.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d7b023a77b2f44c-00631f82e1-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.400 DEBUG:boto:b'jdpiak-19999falsedummydAjTD8IRanrz7e2F.KgapZJi58DocMetrue2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:05:05.401 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:05:05.401 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:05.402 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:05.403 DEBUG:boto:path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.403 DEBUG:boto:auth_path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.404 DEBUG:boto:Method: GET 2022-09-12T19:05:05.404 DEBUG:boto:Path: /jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.405 DEBUG:boto:Data: 2022-09-12T19:05:05.405 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.406 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:05.407 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.407 DEBUG:boto:Params: {} 2022-09-12T19:05:05.408 DEBUG:boto:Token: None 2022-09-12T19:05:05.409 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/?versions 2022-09-12T19:05:05.409 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L0wx09ybeEjwZvA6kGNiA0+cBk4= 2022-09-12T19:05:05.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L0wx09ybeEjwZvA6kGNiA0+cBk4=', 'Content-Length': '0'} 2022-09-12T19:05:05.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013c73d0eabd667a0-00631f82e1-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.413 DEBUG:boto:b'jdpiak-19999falsedummydAjTD8IRanrz7e2F.KgapZJi58DocMetrue2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:05:05.414 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:05.415 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:05.415 DEBUG:boto:path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.416 DEBUG:boto:auth_path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.417 DEBUG:boto:Method: GET 2022-09-12T19:05:05.418 DEBUG:boto:Path: /jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.419 DEBUG:boto:Data: 2022-09-12T19:05:05.419 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.420 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:05.421 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.422 DEBUG:boto:Params: {} 2022-09-12T19:05:05.423 DEBUG:boto:Token: None 2022-09-12T19:05:05.423 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/?versions 2022-09-12T19:05:05.424 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L0wx09ybeEjwZvA6kGNiA0+cBk4= 2022-09-12T19:05:05.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L0wx09ybeEjwZvA6kGNiA0+cBk4=', 'Content-Length': '0'} 2022-09-12T19:05:05.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000497738d8be8e8a08-00631f82e1-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.428 DEBUG:boto:b'jdpiak-19999falsedummydAjTD8IRanrz7e2F.KgapZJi58DocMetrue2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:05:05.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:05:05.430 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:05.431 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:05.431 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.432 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.432 DEBUG:boto:Method: GET 2022-09-12T19:05:05.433 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.433 DEBUG:boto:Data: 2022-09-12T19:05:05.434 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.434 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:05.435 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.435 DEBUG:boto:Params: {} 2022-09-12T19:05:05.436 DEBUG:boto:Token: None 2022-09-12T19:05:05.436 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.437 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YgJDgvZemzXcLi40YkYuT80tHPA= 2022-09-12T19:05:05.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YgJDgvZemzXcLi40YkYuT80tHPA=', 'Content-Length': '0'} 2022-09-12T19:05:05.440 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000053ecb2d1e1ee8362-00631f82e1-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.441 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:05.442 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:05.442 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.443 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.443 DEBUG:boto:Method: GET 2022-09-12T19:05:05.444 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.444 DEBUG:boto:Data: 2022-09-12T19:05:05.445 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.445 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:05.446 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.446 DEBUG:boto:Params: {} 2022-09-12T19:05:05.447 DEBUG:boto:Token: None 2022-09-12T19:05:05.447 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.448 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YgJDgvZemzXcLi40YkYuT80tHPA= 2022-09-12T19:05:05.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YgJDgvZemzXcLi40YkYuT80tHPA=', 'Content-Length': '0'} 2022-09-12T19:05:05.451 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006d51a48be0ceecd8-00631f82e1-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.452 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:05.452 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:05.453 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.453 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.454 DEBUG:boto:Method: HEAD 2022-09-12T19:05:05.454 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.455 DEBUG:boto:Data: 2022-09-12T19:05:05.455 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.456 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:05.456 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.457 DEBUG:boto:Params: {} 2022-09-12T19:05:05.457 DEBUG:boto:Token: None 2022-09-12T19:05:05.458 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.459 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6pa3YA7vHaij9RojutENZBWG2q4= 2022-09-12T19:05:05.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6pa3YA7vHaij9RojutENZBWG2q4=', 'Content-Length': '0'} 2022-09-12T19:05:05.461 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004779bc9f74b77484-00631f82e1-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.462 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:05.462 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:05.463 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.463 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.464 DEBUG:boto:Method: HEAD 2022-09-12T19:05:05.464 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.465 DEBUG:boto:Data: 2022-09-12T19:05:05.466 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.467 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:05.467 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.468 DEBUG:boto:Params: {} 2022-09-12T19:05:05.469 DEBUG:boto:Token: None 2022-09-12T19:05:05.469 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.470 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6pa3YA7vHaij9RojutENZBWG2q4= 2022-09-12T19:05:05.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6pa3YA7vHaij9RojutENZBWG2q4=', 'Content-Length': '0'} 2022-09-12T19:05:05.473 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000071c7f880b7f95398-00631f82e1-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:05:05.475 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:05.475 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:05.476 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.476 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.477 DEBUG:boto:Method: GET 2022-09-12T19:05:05.477 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.478 DEBUG:boto:Data: 2022-09-12T19:05:05.479 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.479 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:05.480 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.480 DEBUG:boto:Params: {} 2022-09-12T19:05:05.481 DEBUG:boto:Token: None 2022-09-12T19:05:05.481 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.482 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YgJDgvZemzXcLi40YkYuT80tHPA= 2022-09-12T19:05:05.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YgJDgvZemzXcLi40YkYuT80tHPA=', 'Content-Length': '0'} 2022-09-12T19:05:05.484 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000296282e4df75b2f1-00631f82e1-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.485 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:05.486 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:05.486 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.487 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.487 DEBUG:boto:Method: GET 2022-09-12T19:05:05.488 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.489 DEBUG:boto:Data: 2022-09-12T19:05:05.489 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.490 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:05.490 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.491 DEBUG:boto:Params: {} 2022-09-12T19:05:05.491 DEBUG:boto:Token: None 2022-09-12T19:05:05.492 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.493 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YgJDgvZemzXcLi40YkYuT80tHPA= 2022-09-12T19:05:05.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YgJDgvZemzXcLi40YkYuT80tHPA=', 'Content-Length': '0'} 2022-09-12T19:05:05.496 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000078770ce53e04eea6-00631f82e1-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.496 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:05.497 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:05.498 DEBUG:boto:path=/jdpiak-19/?prefix=dummy 2022-09-12T19:05:05.498 DEBUG:boto:auth_path=/jdpiak-19/?prefix=dummy 2022-09-12T19:05:05.499 DEBUG:boto:Method: GET 2022-09-12T19:05:05.499 DEBUG:boto:Path: /jdpiak-19/?prefix=dummy 2022-09-12T19:05:05.500 DEBUG:boto:Data: 2022-09-12T19:05:05.500 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.501 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:05.501 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.502 DEBUG:boto:Params: {} 2022-09-12T19:05:05.502 DEBUG:boto:Token: None 2022-09-12T19:05:05.503 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/ 2022-09-12T19:05:05.504 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:siY1oxC1SfOSIGeQzr82EKo+aJg= 2022-09-12T19:05:05.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:siY1oxC1SfOSIGeQzr82EKo+aJg=', 'Content-Length': '0'} 2022-09-12T19:05:05.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042e3c3f6e31f13f5-00631f82e1-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.507 DEBUG:boto:b'jdpiak-19dummy1000falsedummy2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:05:05.508 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:05.509 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:05.509 DEBUG:boto:path=/jdpiak-19/?prefix=dummy 2022-09-12T19:05:05.510 DEBUG:boto:auth_path=/jdpiak-19/?prefix=dummy 2022-09-12T19:05:05.510 DEBUG:boto:Method: GET 2022-09-12T19:05:05.511 DEBUG:boto:Path: /jdpiak-19/?prefix=dummy 2022-09-12T19:05:05.512 DEBUG:boto:Data: 2022-09-12T19:05:05.512 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.513 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:05.513 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.514 DEBUG:boto:Params: {} 2022-09-12T19:05:05.514 DEBUG:boto:Token: None 2022-09-12T19:05:05.515 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/ 2022-09-12T19:05:05.516 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:siY1oxC1SfOSIGeQzr82EKo+aJg= 2022-09-12T19:05:05.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:siY1oxC1SfOSIGeQzr82EKo+aJg=', 'Content-Length': '0'} 2022-09-12T19:05:05.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffbac9898123b857-00631f82e1-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.562 DEBUG:boto:b'jdpiak-19dummy1000falsedummy2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:05:05.563 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:05.563 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:05.564 DEBUG:boto:Method: HEAD 2022-09-12T19:05:05.564 DEBUG:boto:Path: /jdpiak-19/dummy 2022-09-12T19:05:05.565 DEBUG:boto:Data: 2022-09-12T19:05:05.565 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.566 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:05.566 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.567 DEBUG:boto:Params: {} 2022-09-12T19:05:05.567 DEBUG:boto:Token: None 2022-09-12T19:05:05.568 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/dummy 2022-09-12T19:05:05.568 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lb7J1L36CIde+ZktPmTT08gu138= 2022-09-12T19:05:05.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lb7J1L36CIde+ZktPmTT08gu138=', 'Content-Length': '0'} 2022-09-12T19:05:05.571 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000db7a0410e15d5187-00631f82e1-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.572 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:05.572 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:05.573 DEBUG:boto:Method: HEAD 2022-09-12T19:05:05.573 DEBUG:boto:Path: /jdpiak-19/dummy 2022-09-12T19:05:05.574 DEBUG:boto:Data: 2022-09-12T19:05:05.575 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.575 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:05.576 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.576 DEBUG:boto:Params: {} 2022-09-12T19:05:05.577 DEBUG:boto:Token: None 2022-09-12T19:05:05.577 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/dummy 2022-09-12T19:05:05.578 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lb7J1L36CIde+ZktPmTT08gu138= 2022-09-12T19:05:05.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lb7J1L36CIde+ZktPmTT08gu138=', 'Content-Length': '0'} 2022-09-12T19:05:05.581 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008d9b96790cfbc6c5-00631f82e1-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:05:05.582 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:05.583 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:05.583 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.584 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.584 DEBUG:boto:Method: GET 2022-09-12T19:05:05.585 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.586 DEBUG:boto:Data: 2022-09-12T19:05:05.586 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.587 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:05.587 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.588 DEBUG:boto:Params: {} 2022-09-12T19:05:05.588 DEBUG:boto:Token: None 2022-09-12T19:05:05.589 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.590 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YgJDgvZemzXcLi40YkYuT80tHPA= 2022-09-12T19:05:05.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YgJDgvZemzXcLi40YkYuT80tHPA=', 'Content-Length': '0'} 2022-09-12T19:05:05.592 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fa3c92e8d4b3852a-00631f82e1-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.593 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:05.594 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:05.594 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.595 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.595 DEBUG:boto:Method: GET 2022-09-12T19:05:05.596 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.596 DEBUG:boto:Data: 2022-09-12T19:05:05.597 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.597 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:05.598 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.599 DEBUG:boto:Params: {} 2022-09-12T19:05:05.599 DEBUG:boto:Token: None 2022-09-12T19:05:05.600 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.600 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YgJDgvZemzXcLi40YkYuT80tHPA= 2022-09-12T19:05:05.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YgJDgvZemzXcLi40YkYuT80tHPA=', 'Content-Length': '0'} 2022-09-12T19:05:05.603 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cd417aa0d6b13891-00631f82e1-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.604 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:05.604 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:05.605 DEBUG:boto:path=/jdpiak-19/?prefix=dummy 2022-09-12T19:05:05.605 DEBUG:boto:auth_path=/jdpiak-19/?prefix=dummy 2022-09-12T19:05:05.606 DEBUG:boto:Method: GET 2022-09-12T19:05:05.606 DEBUG:boto:Path: /jdpiak-19/?prefix=dummy 2022-09-12T19:05:05.607 DEBUG:boto:Data: 2022-09-12T19:05:05.607 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.608 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:05.609 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.609 DEBUG:boto:Params: {} 2022-09-12T19:05:05.610 DEBUG:boto:Token: None 2022-09-12T19:05:05.610 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/ 2022-09-12T19:05:05.611 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:siY1oxC1SfOSIGeQzr82EKo+aJg= 2022-09-12T19:05:05.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:siY1oxC1SfOSIGeQzr82EKo+aJg=', 'Content-Length': '0'} 2022-09-12T19:05:05.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092dd198297f98968-00631f82e1-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.621 DEBUG:boto:b'jdpiak-19dummy1000falsedummy2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:05:05.622 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:05.623 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:05.624 DEBUG:boto:path=/jdpiak-19/?prefix=dummy 2022-09-12T19:05:05.625 DEBUG:boto:auth_path=/jdpiak-19/?prefix=dummy 2022-09-12T19:05:05.626 DEBUG:boto:Method: GET 2022-09-12T19:05:05.627 DEBUG:boto:Path: /jdpiak-19/?prefix=dummy 2022-09-12T19:05:05.628 DEBUG:boto:Data: 2022-09-12T19:05:05.629 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.630 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:05.631 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.631 DEBUG:boto:Params: {} 2022-09-12T19:05:05.632 DEBUG:boto:Token: None 2022-09-12T19:05:05.632 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/ 2022-09-12T19:05:05.633 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:siY1oxC1SfOSIGeQzr82EKo+aJg= 2022-09-12T19:05:05.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:siY1oxC1SfOSIGeQzr82EKo+aJg=', 'Content-Length': '0'} 2022-09-12T19:05:05.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004468a8ff54bca379-00631f82e1-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.637 DEBUG:boto:b'jdpiak-19dummy1000falsedummy2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:05:05.638 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-19 zones={test-zone2, test-zone1} 2022-09-12T19:05:05.639 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-19 zones={test-zone3, test-zone1} 2022-09-12T19:05:05.639 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:05.640 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:05.640 DEBUG:boto:Method: HEAD 2022-09-12T19:05:05.641 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:05.641 DEBUG:boto:Data: 2022-09-12T19:05:05.641 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.642 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:05.642 DEBUG:boto:Port: 8001 2022-09-12T19:05:05.643 DEBUG:boto:Params: {} 2022-09-12T19:05:05.644 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T19:05:05.644 DEBUG:boto:Token: None 2022-09-12T19:05:05.645 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/ 2022-09-12T19:05:05.645 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oXbJ2KSvSrXP2QqdFKquJbrZuBY= 2022-09-12T19:05:05.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oXbJ2KSvSrXP2QqdFKquJbrZuBY=', 'Content-Length': '0'} 2022-09-12T19:05:05.651 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3a50f6a049d0bcf-00631f82e1-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.652 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:05.653 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:05.653 DEBUG:boto:Method: HEAD 2022-09-12T19:05:05.654 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:05.655 DEBUG:boto:Data: 2022-09-12T19:05:05.656 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.656 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:05.657 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.658 DEBUG:boto:Params: {} 2022-09-12T19:05:05.659 DEBUG:boto:Token: None 2022-09-12T19:05:05.660 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/ 2022-09-12T19:05:05.661 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oXbJ2KSvSrXP2QqdFKquJbrZuBY= 2022-09-12T19:05:05.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oXbJ2KSvSrXP2QqdFKquJbrZuBY=', 'Content-Length': '0'} 2022-09-12T19:05:05.665 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee7e7c162a12e720-00631f82e1-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.667 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:05:05.667 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:05.668 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:05.669 DEBUG:boto:path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.670 DEBUG:boto:auth_path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.671 DEBUG:boto:Method: GET 2022-09-12T19:05:05.672 DEBUG:boto:Path: /jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.673 DEBUG:boto:Data: 2022-09-12T19:05:05.674 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.674 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:05.675 DEBUG:boto:Port: 8001 2022-09-12T19:05:05.675 DEBUG:boto:Params: {} 2022-09-12T19:05:05.676 DEBUG:boto:Token: None 2022-09-12T19:05:05.677 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/?versions 2022-09-12T19:05:05.678 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L0wx09ybeEjwZvA6kGNiA0+cBk4= 2022-09-12T19:05:05.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L0wx09ybeEjwZvA6kGNiA0+cBk4=', 'Content-Length': '0'} 2022-09-12T19:05:05.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c66ba1cc6eebc6a1-00631f82e1-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.682 DEBUG:boto:b'jdpiak-19999falsedummydAjTD8IRanrz7e2F.KgapZJi58DocMetrue2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:05:05.684 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:05:05.684 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:05:05.685 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:05.686 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:05.687 DEBUG:boto:path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.688 DEBUG:boto:auth_path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.689 DEBUG:boto:Method: GET 2022-09-12T19:05:05.689 DEBUG:boto:Path: /jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.690 DEBUG:boto:Data: 2022-09-12T19:05:05.691 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.692 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:05.692 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.693 DEBUG:boto:Params: {} 2022-09-12T19:05:05.694 DEBUG:boto:Token: None 2022-09-12T19:05:05.694 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/?versions 2022-09-12T19:05:05.695 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L0wx09ybeEjwZvA6kGNiA0+cBk4= 2022-09-12T19:05:05.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L0wx09ybeEjwZvA6kGNiA0+cBk4=', 'Content-Length': '0'} 2022-09-12T19:05:05.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e8173d90fb42a50-00631f82e1-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.699 DEBUG:boto:b'jdpiak-19999falsedummydAjTD8IRanrz7e2F.KgapZJi58DocMetrue2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:05:05.700 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:05:05.700 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:05.701 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:05.701 DEBUG:boto:path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.702 DEBUG:boto:auth_path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.702 DEBUG:boto:Method: GET 2022-09-12T19:05:05.703 DEBUG:boto:Path: /jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.704 DEBUG:boto:Data: 2022-09-12T19:05:05.704 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.705 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:05.705 DEBUG:boto:Port: 8001 2022-09-12T19:05:05.706 DEBUG:boto:Params: {} 2022-09-12T19:05:05.706 DEBUG:boto:Token: None 2022-09-12T19:05:05.707 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/?versions 2022-09-12T19:05:05.707 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L0wx09ybeEjwZvA6kGNiA0+cBk4= 2022-09-12T19:05:05.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L0wx09ybeEjwZvA6kGNiA0+cBk4=', 'Content-Length': '0'} 2022-09-12T19:05:05.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000997050f253e5e687-00631f82e1-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.712 DEBUG:boto:b'jdpiak-19999falsedummydAjTD8IRanrz7e2F.KgapZJi58DocMetrue2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:05:05.712 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:05.713 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:05.714 DEBUG:boto:path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.714 DEBUG:boto:auth_path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.715 DEBUG:boto:Method: GET 2022-09-12T19:05:05.715 DEBUG:boto:Path: /jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.716 DEBUG:boto:Data: 2022-09-12T19:05:05.716 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.717 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:05.717 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.718 DEBUG:boto:Params: {} 2022-09-12T19:05:05.718 DEBUG:boto:Token: None 2022-09-12T19:05:05.719 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/?versions 2022-09-12T19:05:05.719 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L0wx09ybeEjwZvA6kGNiA0+cBk4= 2022-09-12T19:05:05.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L0wx09ybeEjwZvA6kGNiA0+cBk4=', 'Content-Length': '0'} 2022-09-12T19:05:05.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064e084040c75e4df-00631f82e1-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.724 DEBUG:boto:b'jdpiak-19999falsedummydAjTD8IRanrz7e2F.KgapZJi58DocMetrue2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:05:05.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:05:05.726 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:05.727 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:05.728 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.729 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.730 DEBUG:boto:Method: GET 2022-09-12T19:05:05.731 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.732 DEBUG:boto:Data: 2022-09-12T19:05:05.733 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.734 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:05.735 DEBUG:boto:Port: 8001 2022-09-12T19:05:05.735 DEBUG:boto:Params: {} 2022-09-12T19:05:05.737 DEBUG:boto:Token: None 2022-09-12T19:05:05.738 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.739 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YgJDgvZemzXcLi40YkYuT80tHPA= 2022-09-12T19:05:05.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YgJDgvZemzXcLi40YkYuT80tHPA=', 'Content-Length': '0'} 2022-09-12T19:05:05.742 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007afafd36fc221d49-00631f82e1-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.743 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:05.743 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:05.744 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.744 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.745 DEBUG:boto:Method: GET 2022-09-12T19:05:05.745 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.746 DEBUG:boto:Data: 2022-09-12T19:05:05.746 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.747 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:05.748 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.748 DEBUG:boto:Params: {} 2022-09-12T19:05:05.749 DEBUG:boto:Token: None 2022-09-12T19:05:05.749 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.750 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YgJDgvZemzXcLi40YkYuT80tHPA= 2022-09-12T19:05:05.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YgJDgvZemzXcLi40YkYuT80tHPA=', 'Content-Length': '0'} 2022-09-12T19:05:05.753 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000085921860deb6c1f1-00631f82e1-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.753 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:05.754 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:05.755 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.755 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.756 DEBUG:boto:Method: HEAD 2022-09-12T19:05:05.756 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.757 DEBUG:boto:Data: 2022-09-12T19:05:05.757 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.758 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:05.758 DEBUG:boto:Port: 8001 2022-09-12T19:05:05.759 DEBUG:boto:Params: {} 2022-09-12T19:05:05.760 DEBUG:boto:Token: None 2022-09-12T19:05:05.761 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.762 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6pa3YA7vHaij9RojutENZBWG2q4= 2022-09-12T19:05:05.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6pa3YA7vHaij9RojutENZBWG2q4=', 'Content-Length': '0'} 2022-09-12T19:05:05.768 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006602a50ed73bd291-00631f82e1-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.769 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:05.770 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:05.771 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.772 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.772 DEBUG:boto:Method: HEAD 2022-09-12T19:05:05.773 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.774 DEBUG:boto:Data: 2022-09-12T19:05:05.775 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.776 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:05.777 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.778 DEBUG:boto:Params: {} 2022-09-12T19:05:05.779 DEBUG:boto:Token: None 2022-09-12T19:05:05.780 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.780 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6pa3YA7vHaij9RojutENZBWG2q4= 2022-09-12T19:05:05.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6pa3YA7vHaij9RojutENZBWG2q4=', 'Content-Length': '0'} 2022-09-12T19:05:05.785 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009d2e29a78eab6f55-00631f82e1-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:05:05.786 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:05.787 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:05.788 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.788 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.789 DEBUG:boto:Method: GET 2022-09-12T19:05:05.790 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.790 DEBUG:boto:Data: 2022-09-12T19:05:05.791 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.791 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:05.792 DEBUG:boto:Port: 8001 2022-09-12T19:05:05.792 DEBUG:boto:Params: {} 2022-09-12T19:05:05.793 DEBUG:boto:Token: None 2022-09-12T19:05:05.794 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.795 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YgJDgvZemzXcLi40YkYuT80tHPA= 2022-09-12T19:05:05.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YgJDgvZemzXcLi40YkYuT80tHPA=', 'Content-Length': '0'} 2022-09-12T19:05:05.798 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e55cceb54a9e1cb8-00631f82e1-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.798 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:05.799 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:05.800 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.800 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.801 DEBUG:boto:Method: GET 2022-09-12T19:05:05.801 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.802 DEBUG:boto:Data: 2022-09-12T19:05:05.802 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.803 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:05.803 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.804 DEBUG:boto:Params: {} 2022-09-12T19:05:05.805 DEBUG:boto:Token: None 2022-09-12T19:05:05.805 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.806 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YgJDgvZemzXcLi40YkYuT80tHPA= 2022-09-12T19:05:05.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YgJDgvZemzXcLi40YkYuT80tHPA=', 'Content-Length': '0'} 2022-09-12T19:05:05.809 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a70ca509c21e49c8-00631f82e1-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.810 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:05.810 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:05.811 DEBUG:boto:path=/jdpiak-19/?prefix=dummy 2022-09-12T19:05:05.811 DEBUG:boto:auth_path=/jdpiak-19/?prefix=dummy 2022-09-12T19:05:05.812 DEBUG:boto:Method: GET 2022-09-12T19:05:05.813 DEBUG:boto:Path: /jdpiak-19/?prefix=dummy 2022-09-12T19:05:05.813 DEBUG:boto:Data: 2022-09-12T19:05:05.814 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.814 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:05.815 DEBUG:boto:Port: 8001 2022-09-12T19:05:05.815 DEBUG:boto:Params: {} 2022-09-12T19:05:05.816 DEBUG:boto:Token: None 2022-09-12T19:05:05.817 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/ 2022-09-12T19:05:05.817 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:siY1oxC1SfOSIGeQzr82EKo+aJg= 2022-09-12T19:05:05.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:siY1oxC1SfOSIGeQzr82EKo+aJg=', 'Content-Length': '0'} 2022-09-12T19:05:05.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048e6251cd785a11b-00631f82e1-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.862 DEBUG:boto:b'jdpiak-19dummy1000falsedummy2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:05:05.863 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:05.864 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:05.864 DEBUG:boto:path=/jdpiak-19/?prefix=dummy 2022-09-12T19:05:05.865 DEBUG:boto:auth_path=/jdpiak-19/?prefix=dummy 2022-09-12T19:05:05.865 DEBUG:boto:Method: GET 2022-09-12T19:05:05.866 DEBUG:boto:Path: /jdpiak-19/?prefix=dummy 2022-09-12T19:05:05.866 DEBUG:boto:Data: 2022-09-12T19:05:05.867 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.867 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:05.868 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.868 DEBUG:boto:Params: {} 2022-09-12T19:05:05.869 DEBUG:boto:Token: None 2022-09-12T19:05:05.870 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/ 2022-09-12T19:05:05.870 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:siY1oxC1SfOSIGeQzr82EKo+aJg= 2022-09-12T19:05:05.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:siY1oxC1SfOSIGeQzr82EKo+aJg=', 'Content-Length': '0'} 2022-09-12T19:05:05.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a6212152f9b4bf6-00631f82e1-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.874 DEBUG:boto:b'jdpiak-19dummy1000falsedummy2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:05:05.875 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:05.875 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:05.876 DEBUG:boto:Method: HEAD 2022-09-12T19:05:05.876 DEBUG:boto:Path: /jdpiak-19/dummy 2022-09-12T19:05:05.877 DEBUG:boto:Data: 2022-09-12T19:05:05.878 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.878 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:05.879 DEBUG:boto:Port: 8001 2022-09-12T19:05:05.879 DEBUG:boto:Params: {} 2022-09-12T19:05:05.880 DEBUG:boto:Token: None 2022-09-12T19:05:05.880 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/dummy 2022-09-12T19:05:05.881 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lb7J1L36CIde+ZktPmTT08gu138= 2022-09-12T19:05:05.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lb7J1L36CIde+ZktPmTT08gu138=', 'Content-Length': '0'} 2022-09-12T19:05:05.884 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000233afbb8dddcd443-00631f82e1-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.885 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:05.885 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:05.886 DEBUG:boto:Method: HEAD 2022-09-12T19:05:05.887 DEBUG:boto:Path: /jdpiak-19/dummy 2022-09-12T19:05:05.888 DEBUG:boto:Data: 2022-09-12T19:05:05.888 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.889 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:05.890 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.891 DEBUG:boto:Params: {} 2022-09-12T19:05:05.891 DEBUG:boto:Token: None 2022-09-12T19:05:05.892 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/dummy 2022-09-12T19:05:05.892 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lb7J1L36CIde+ZktPmTT08gu138= 2022-09-12T19:05:05.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lb7J1L36CIde+ZktPmTT08gu138=', 'Content-Length': '0'} 2022-09-12T19:05:05.895 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009fce7b845ae6f143-00631f82e1-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:05:05.897 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:05.897 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:05.898 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.898 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.899 DEBUG:boto:Method: GET 2022-09-12T19:05:05.899 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.900 DEBUG:boto:Data: 2022-09-12T19:05:05.900 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.901 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:05.901 DEBUG:boto:Port: 8001 2022-09-12T19:05:05.902 DEBUG:boto:Params: {} 2022-09-12T19:05:05.903 DEBUG:boto:Token: None 2022-09-12T19:05:05.903 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.904 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YgJDgvZemzXcLi40YkYuT80tHPA= 2022-09-12T19:05:05.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YgJDgvZemzXcLi40YkYuT80tHPA=', 'Content-Length': '0'} 2022-09-12T19:05:05.906 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001e344c1d7e1179a8-00631f82e1-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.907 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:05.908 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:05.908 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.909 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.909 DEBUG:boto:Method: GET 2022-09-12T19:05:05.910 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.910 DEBUG:boto:Data: 2022-09-12T19:05:05.911 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.912 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:05.912 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.913 DEBUG:boto:Params: {} 2022-09-12T19:05:05.913 DEBUG:boto:Token: None 2022-09-12T19:05:05.914 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:05.915 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YgJDgvZemzXcLi40YkYuT80tHPA= 2022-09-12T19:05:05.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YgJDgvZemzXcLi40YkYuT80tHPA=', 'Content-Length': '0'} 2022-09-12T19:05:05.917 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e2e04dd95cb04693-00631f82e1-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.918 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:05.919 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:05.919 DEBUG:boto:path=/jdpiak-19/?prefix=dummy 2022-09-12T19:05:05.920 DEBUG:boto:auth_path=/jdpiak-19/?prefix=dummy 2022-09-12T19:05:05.921 DEBUG:boto:Method: GET 2022-09-12T19:05:05.921 DEBUG:boto:Path: /jdpiak-19/?prefix=dummy 2022-09-12T19:05:05.922 DEBUG:boto:Data: 2022-09-12T19:05:05.923 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.923 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:05.924 DEBUG:boto:Port: 8001 2022-09-12T19:05:05.924 DEBUG:boto:Params: {} 2022-09-12T19:05:05.925 DEBUG:boto:Token: None 2022-09-12T19:05:05.926 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/ 2022-09-12T19:05:05.926 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:siY1oxC1SfOSIGeQzr82EKo+aJg= 2022-09-12T19:05:05.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:siY1oxC1SfOSIGeQzr82EKo+aJg=', 'Content-Length': '0'} 2022-09-12T19:05:05.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b450cb1177476db4-00631f82e1-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.930 DEBUG:boto:b'jdpiak-19dummy1000falsedummy2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:05:05.931 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:05.932 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:05.933 DEBUG:boto:path=/jdpiak-19/?prefix=dummy 2022-09-12T19:05:05.933 DEBUG:boto:auth_path=/jdpiak-19/?prefix=dummy 2022-09-12T19:05:05.934 DEBUG:boto:Method: GET 2022-09-12T19:05:05.934 DEBUG:boto:Path: /jdpiak-19/?prefix=dummy 2022-09-12T19:05:05.935 DEBUG:boto:Data: 2022-09-12T19:05:05.935 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.936 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:05.936 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.937 DEBUG:boto:Params: {} 2022-09-12T19:05:05.938 DEBUG:boto:Token: None 2022-09-12T19:05:05.938 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/ 2022-09-12T19:05:05.939 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:siY1oxC1SfOSIGeQzr82EKo+aJg= 2022-09-12T19:05:05.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:siY1oxC1SfOSIGeQzr82EKo+aJg=', 'Content-Length': '0'} 2022-09-12T19:05:05.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ec2c70a302960be-00631f82e1-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.943 DEBUG:boto:b'jdpiak-19dummy1000falsedummy2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:05:05.944 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-19 zones={test-zone3, test-zone1} 2022-09-12T19:05:05.944 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-19 zones={test-zone3, test-zone2} 2022-09-12T19:05:05.945 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:05.945 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:05.946 DEBUG:boto:Method: HEAD 2022-09-12T19:05:05.946 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:05.947 DEBUG:boto:Data: 2022-09-12T19:05:05.947 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.948 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:05.948 DEBUG:boto:Port: 8001 2022-09-12T19:05:05.949 DEBUG:boto:Params: {} 2022-09-12T19:05:05.949 DEBUG:boto:Token: None 2022-09-12T19:05:05.950 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/ 2022-09-12T19:05:05.951 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oXbJ2KSvSrXP2QqdFKquJbrZuBY= 2022-09-12T19:05:05.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oXbJ2KSvSrXP2QqdFKquJbrZuBY=', 'Content-Length': '0'} 2022-09-12T19:05:05.954 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000625e3bd927df04de-00631f82e1-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.955 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:05.955 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:05.956 DEBUG:boto:Method: HEAD 2022-09-12T19:05:05.956 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:05.957 DEBUG:boto:Data: 2022-09-12T19:05:05.957 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.958 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:05.958 DEBUG:boto:Port: 8000 2022-09-12T19:05:05.959 DEBUG:boto:Params: {} 2022-09-12T19:05:05.959 DEBUG:boto:Token: None 2022-09-12T19:05:05.960 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/ 2022-09-12T19:05:05.961 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oXbJ2KSvSrXP2QqdFKquJbrZuBY= 2022-09-12T19:05:05.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oXbJ2KSvSrXP2QqdFKquJbrZuBY=', 'Content-Length': '0'} 2022-09-12T19:05:05.964 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000edb477a00f653a35-00631f82e1-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:05.964 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:05:05.965 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:05.965 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:05.966 DEBUG:boto:path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.966 DEBUG:boto:auth_path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.967 DEBUG:boto:Method: GET 2022-09-12T19:05:05.967 DEBUG:boto:Path: /jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:05.968 DEBUG:boto:Data: 2022-09-12T19:05:05.968 DEBUG:boto:Headers: {} 2022-09-12T19:05:05.969 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:05.969 DEBUG:boto:Port: 8001 2022-09-12T19:05:05.970 DEBUG:boto:Params: {} 2022-09-12T19:05:05.971 DEBUG:boto:Token: None 2022-09-12T19:05:05.971 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:05 GMT /jdpiak-19/?versions 2022-09-12T19:05:05.972 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L0wx09ybeEjwZvA6kGNiA0+cBk4= 2022-09-12T19:05:05.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L0wx09ybeEjwZvA6kGNiA0+cBk4=', 'Content-Length': '0'} 2022-09-12T19:05:05.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011d00cb21ca3155f-00631f82e1-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:05:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.018 DEBUG:boto:b'jdpiak-19999falsedummydAjTD8IRanrz7e2F.KgapZJi58DocMetrue2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:05:06.019 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:05:06.020 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:05:06.020 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:06.021 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:06.021 DEBUG:boto:path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:06.022 DEBUG:boto:auth_path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:06.022 DEBUG:boto:Method: GET 2022-09-12T19:05:06.023 DEBUG:boto:Path: /jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:06.023 DEBUG:boto:Data: 2022-09-12T19:05:06.024 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.024 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:06.025 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.025 DEBUG:boto:Params: {} 2022-09-12T19:05:06.026 DEBUG:boto:Token: None 2022-09-12T19:05:06.026 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/?versions 2022-09-12T19:05:06.027 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:v+PJ11MAoXZxC/rQv35D+Kgu6gg= 2022-09-12T19:05:06.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:v+PJ11MAoXZxC/rQv35D+Kgu6gg=', 'Content-Length': '0'} 2022-09-12T19:05:06.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f44084331e501c6a-00631f82e2-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.031 DEBUG:boto:b'jdpiak-19999falsedummydAjTD8IRanrz7e2F.KgapZJi58DocMetrue2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:05:06.032 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:05:06.032 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:06.033 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:06.033 DEBUG:boto:path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:06.034 DEBUG:boto:auth_path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:06.034 DEBUG:boto:Method: GET 2022-09-12T19:05:06.035 DEBUG:boto:Path: /jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:06.035 DEBUG:boto:Data: 2022-09-12T19:05:06.036 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.036 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:06.037 DEBUG:boto:Port: 8001 2022-09-12T19:05:06.038 DEBUG:boto:Params: {} 2022-09-12T19:05:06.038 DEBUG:boto:Token: None 2022-09-12T19:05:06.039 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/?versions 2022-09-12T19:05:06.040 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:v+PJ11MAoXZxC/rQv35D+Kgu6gg= 2022-09-12T19:05:06.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:v+PJ11MAoXZxC/rQv35D+Kgu6gg=', 'Content-Length': '0'} 2022-09-12T19:05:06.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003578f4885a19431a-00631f82e2-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.043 DEBUG:boto:b'jdpiak-19999falsedummydAjTD8IRanrz7e2F.KgapZJi58DocMetrue2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:05:06.044 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:06.045 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:06.046 DEBUG:boto:path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:06.046 DEBUG:boto:auth_path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:06.047 DEBUG:boto:Method: GET 2022-09-12T19:05:06.047 DEBUG:boto:Path: /jdpiak-19/?versions&max-keys=999 2022-09-12T19:05:06.048 DEBUG:boto:Data: 2022-09-12T19:05:06.048 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.049 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:06.049 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.050 DEBUG:boto:Params: {} 2022-09-12T19:05:06.050 DEBUG:boto:Token: None 2022-09-12T19:05:06.051 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/?versions 2022-09-12T19:05:06.051 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:v+PJ11MAoXZxC/rQv35D+Kgu6gg= 2022-09-12T19:05:06.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:v+PJ11MAoXZxC/rQv35D+Kgu6gg=', 'Content-Length': '0'} 2022-09-12T19:05:06.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b515590444d7401-00631f82e2-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.102 DEBUG:boto:b'jdpiak-19999falsedummydAjTD8IRanrz7e2F.KgapZJi58DocMetrue2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:05:06.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:05:06.105 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:06.106 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:06.107 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.108 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.109 DEBUG:boto:Method: GET 2022-09-12T19:05:06.110 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.111 DEBUG:boto:Data: 2022-09-12T19:05:06.112 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.113 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:06.114 DEBUG:boto:Port: 8001 2022-09-12T19:05:06.115 DEBUG:boto:Params: {} 2022-09-12T19:05:06.116 DEBUG:boto:Token: None 2022-09-12T19:05:06.117 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.119 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HupePdTnAN+BNz1KUJO4YbyNlos= 2022-09-12T19:05:06.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HupePdTnAN+BNz1KUJO4YbyNlos=', 'Content-Length': '0'} 2022-09-12T19:05:06.123 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000143db6a2ee4c8a87-00631f82e2-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.124 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:06.125 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:06.126 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.127 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.128 DEBUG:boto:Method: GET 2022-09-12T19:05:06.129 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.131 DEBUG:boto:Data: 2022-09-12T19:05:06.132 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.133 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:06.134 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.135 DEBUG:boto:Params: {} 2022-09-12T19:05:06.136 DEBUG:boto:Token: None 2022-09-12T19:05:06.138 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.139 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HupePdTnAN+BNz1KUJO4YbyNlos= 2022-09-12T19:05:06.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HupePdTnAN+BNz1KUJO4YbyNlos=', 'Content-Length': '0'} 2022-09-12T19:05:06.143 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b2556367be8fb82d-00631f82e2-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.144 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:06.145 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:06.146 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.147 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.148 DEBUG:boto:Method: HEAD 2022-09-12T19:05:06.149 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.150 DEBUG:boto:Data: 2022-09-12T19:05:06.151 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.152 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:06.153 DEBUG:boto:Port: 8001 2022-09-12T19:05:06.154 DEBUG:boto:Params: {} 2022-09-12T19:05:06.155 DEBUG:boto:Token: None 2022-09-12T19:05:06.156 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.158 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UBgR5FwlFiGF4d9QaCwvVY8/W3Q= 2022-09-12T19:05:06.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UBgR5FwlFiGF4d9QaCwvVY8/W3Q=', 'Content-Length': '0'} 2022-09-12T19:05:06.161 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cab0d728953b4b9a-00631f82e2-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.163 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:06.164 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:06.164 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.165 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.166 DEBUG:boto:Method: HEAD 2022-09-12T19:05:06.167 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.168 DEBUG:boto:Data: 2022-09-12T19:05:06.169 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.170 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:06.171 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.172 DEBUG:boto:Params: {} 2022-09-12T19:05:06.173 DEBUG:boto:Token: None 2022-09-12T19:05:06.174 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.175 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UBgR5FwlFiGF4d9QaCwvVY8/W3Q= 2022-09-12T19:05:06.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UBgR5FwlFiGF4d9QaCwvVY8/W3Q=', 'Content-Length': '0'} 2022-09-12T19:05:06.178 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006815d94ac483a20a-00631f82e2-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:05:06.181 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:06.181 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:06.182 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.184 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.185 DEBUG:boto:Method: GET 2022-09-12T19:05:06.186 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.187 DEBUG:boto:Data: 2022-09-12T19:05:06.188 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.189 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:06.190 DEBUG:boto:Port: 8001 2022-09-12T19:05:06.191 DEBUG:boto:Params: {} 2022-09-12T19:05:06.193 DEBUG:boto:Token: None 2022-09-12T19:05:06.194 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.195 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HupePdTnAN+BNz1KUJO4YbyNlos= 2022-09-12T19:05:06.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HupePdTnAN+BNz1KUJO4YbyNlos=', 'Content-Length': '0'} 2022-09-12T19:05:06.199 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ba230abbbca21a76-00631f82e2-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.200 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:06.201 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:06.201 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.202 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.203 DEBUG:boto:Method: GET 2022-09-12T19:05:06.203 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.204 DEBUG:boto:Data: 2022-09-12T19:05:06.204 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.205 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:06.205 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.206 DEBUG:boto:Params: {} 2022-09-12T19:05:06.206 DEBUG:boto:Token: None 2022-09-12T19:05:06.207 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.208 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HupePdTnAN+BNz1KUJO4YbyNlos= 2022-09-12T19:05:06.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HupePdTnAN+BNz1KUJO4YbyNlos=', 'Content-Length': '0'} 2022-09-12T19:05:06.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000053ac94687fd5bc21-00631f82e2-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.211 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:06.212 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:06.212 DEBUG:boto:path=/jdpiak-19/?prefix=dummy 2022-09-12T19:05:06.213 DEBUG:boto:auth_path=/jdpiak-19/?prefix=dummy 2022-09-12T19:05:06.213 DEBUG:boto:Method: GET 2022-09-12T19:05:06.214 DEBUG:boto:Path: /jdpiak-19/?prefix=dummy 2022-09-12T19:05:06.214 DEBUG:boto:Data: 2022-09-12T19:05:06.215 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.215 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:06.216 DEBUG:boto:Port: 8001 2022-09-12T19:05:06.216 DEBUG:boto:Params: {} 2022-09-12T19:05:06.217 DEBUG:boto:Token: None 2022-09-12T19:05:06.218 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/ 2022-09-12T19:05:06.218 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KMQOUygQSXn4IBx6jM+8Bfetod0= 2022-09-12T19:05:06.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KMQOUygQSXn4IBx6jM+8Bfetod0=', 'Content-Length': '0'} 2022-09-12T19:05:06.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a221c3b824ef827-00631f82e2-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.263 DEBUG:boto:b'jdpiak-19dummy1000falsedummy2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:05:06.264 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:06.265 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:06.265 DEBUG:boto:path=/jdpiak-19/?prefix=dummy 2022-09-12T19:05:06.266 DEBUG:boto:auth_path=/jdpiak-19/?prefix=dummy 2022-09-12T19:05:06.266 DEBUG:boto:Method: GET 2022-09-12T19:05:06.267 DEBUG:boto:Path: /jdpiak-19/?prefix=dummy 2022-09-12T19:05:06.267 DEBUG:boto:Data: 2022-09-12T19:05:06.268 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.268 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:06.269 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.270 DEBUG:boto:Params: {} 2022-09-12T19:05:06.270 DEBUG:boto:Token: None 2022-09-12T19:05:06.271 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/ 2022-09-12T19:05:06.272 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KMQOUygQSXn4IBx6jM+8Bfetod0= 2022-09-12T19:05:06.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KMQOUygQSXn4IBx6jM+8Bfetod0=', 'Content-Length': '0'} 2022-09-12T19:05:06.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfedcb42c9c37023-00631f82e2-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.275 DEBUG:boto:b'jdpiak-19dummy1000falsedummy2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:05:06.276 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:06.277 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:06.277 DEBUG:boto:Method: HEAD 2022-09-12T19:05:06.278 DEBUG:boto:Path: /jdpiak-19/dummy 2022-09-12T19:05:06.278 DEBUG:boto:Data: 2022-09-12T19:05:06.279 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.279 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:06.280 DEBUG:boto:Port: 8001 2022-09-12T19:05:06.280 DEBUG:boto:Params: {} 2022-09-12T19:05:06.281 DEBUG:boto:Token: None 2022-09-12T19:05:06.282 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/dummy 2022-09-12T19:05:06.282 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Wbn5HWr0Ge1g7D0E6QObGkK9vMc= 2022-09-12T19:05:06.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Wbn5HWr0Ge1g7D0E6QObGkK9vMc=', 'Content-Length': '0'} 2022-09-12T19:05:06.285 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000583bb23c0d779a71-00631f82e2-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.286 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:06.287 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:06.287 DEBUG:boto:Method: HEAD 2022-09-12T19:05:06.288 DEBUG:boto:Path: /jdpiak-19/dummy 2022-09-12T19:05:06.288 DEBUG:boto:Data: 2022-09-12T19:05:06.289 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.289 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:06.289 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.290 DEBUG:boto:Params: {} 2022-09-12T19:05:06.291 DEBUG:boto:Token: None 2022-09-12T19:05:06.291 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/dummy 2022-09-12T19:05:06.292 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Wbn5HWr0Ge1g7D0E6QObGkK9vMc= 2022-09-12T19:05:06.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Wbn5HWr0Ge1g7D0E6QObGkK9vMc=', 'Content-Length': '0'} 2022-09-12T19:05:06.295 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000007d9e8c44719de5b-00631f82e2-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:05:06.296 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:06.296 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:06.297 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.297 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.298 DEBUG:boto:Method: GET 2022-09-12T19:05:06.299 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.299 DEBUG:boto:Data: 2022-09-12T19:05:06.300 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.300 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:06.301 DEBUG:boto:Port: 8001 2022-09-12T19:05:06.301 DEBUG:boto:Params: {} 2022-09-12T19:05:06.302 DEBUG:boto:Token: None 2022-09-12T19:05:06.303 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.303 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HupePdTnAN+BNz1KUJO4YbyNlos= 2022-09-12T19:05:06.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HupePdTnAN+BNz1KUJO4YbyNlos=', 'Content-Length': '0'} 2022-09-12T19:05:06.307 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003afb3f2c9df509ce-00631f82e2-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.309 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:05:06.310 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:05:06.311 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.312 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.313 DEBUG:boto:Method: GET 2022-09-12T19:05:06.314 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.315 DEBUG:boto:Data: 2022-09-12T19:05:06.316 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.317 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:06.318 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.319 DEBUG:boto:Params: {} 2022-09-12T19:05:06.320 DEBUG:boto:Token: None 2022-09-12T19:05:06.320 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:05:06.321 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HupePdTnAN+BNz1KUJO4YbyNlos= 2022-09-12T19:05:06.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HupePdTnAN+BNz1KUJO4YbyNlos=', 'Content-Length': '0'} 2022-09-12T19:05:06.324 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005a69f076d8adcabc-00631f82e2-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.325 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:06.325 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:06.327 DEBUG:boto:path=/jdpiak-19/?prefix=dummy 2022-09-12T19:05:06.327 DEBUG:boto:auth_path=/jdpiak-19/?prefix=dummy 2022-09-12T19:05:06.328 DEBUG:boto:Method: GET 2022-09-12T19:05:06.328 DEBUG:boto:Path: /jdpiak-19/?prefix=dummy 2022-09-12T19:05:06.331 DEBUG:boto:Data: 2022-09-12T19:05:06.332 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.333 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:06.333 DEBUG:boto:Port: 8001 2022-09-12T19:05:06.334 DEBUG:boto:Params: {} 2022-09-12T19:05:06.334 DEBUG:boto:Token: None 2022-09-12T19:05:06.335 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/ 2022-09-12T19:05:06.335 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KMQOUygQSXn4IBx6jM+8Bfetod0= 2022-09-12T19:05:06.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KMQOUygQSXn4IBx6jM+8Bfetod0=', 'Content-Length': '0'} 2022-09-12T19:05:06.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0069efa4546e5a8-00631f82e2-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.340 DEBUG:boto:b'jdpiak-19dummy1000falsedummy2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:05:06.341 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:06.342 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:06.342 DEBUG:boto:path=/jdpiak-19/?prefix=dummy 2022-09-12T19:05:06.343 DEBUG:boto:auth_path=/jdpiak-19/?prefix=dummy 2022-09-12T19:05:06.343 DEBUG:boto:Method: GET 2022-09-12T19:05:06.344 DEBUG:boto:Path: /jdpiak-19/?prefix=dummy 2022-09-12T19:05:06.345 DEBUG:boto:Data: 2022-09-12T19:05:06.345 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.346 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:06.347 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.347 DEBUG:boto:Params: {} 2022-09-12T19:05:06.348 DEBUG:boto:Token: None 2022-09-12T19:05:06.348 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/ 2022-09-12T19:05:06.349 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KMQOUygQSXn4IBx6jM+8Bfetod0= 2022-09-12T19:05:06.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KMQOUygQSXn4IBx6jM+8Bfetod0=', 'Content-Length': '0'} 2022-09-12T19:05:06.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9dad42ff3d955d3-00631f82e2-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.353 DEBUG:boto:b'jdpiak-19dummy1000falsedummy2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:05:06.354 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-19 zones={test-zone3, test-zone2} 2022-09-12T19:05:06.355 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:06.355 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:06.357 DEBUG:boto:Method: HEAD 2022-09-12T19:05:06.358 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:06.359 DEBUG:boto:Data: 2022-09-12T19:05:06.360 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.360 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:06.361 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.362 DEBUG:boto:Params: {} 2022-09-12T19:05:06.363 DEBUG:boto:Token: None 2022-09-12T19:05:06.364 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/ 2022-09-12T19:05:06.365 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg= 2022-09-12T19:05:06.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg=', 'Content-Length': '0'} 2022-09-12T19:05:06.369 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009afefb18c834cac1-00631f82e2-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.371 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:06.372 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:06.373 DEBUG:boto:Method: PUT 2022-09-12T19:05:06.373 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:06.374 DEBUG:boto:Data: 2022-09-12T19:05:06.375 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:06.376 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:06.377 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.378 DEBUG:boto:Params: {} 2022-09-12T19:05:06.379 DEBUG:boto:Token: None 2022-09-12T19:05:06.380 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/obj 2022-09-12T19:05:06.381 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU= 2022-09-12T19:05:06.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU='} 2022-09-12T19:05:06.396 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'JcC9N13lklBVOx234Q8MXQ8kAM8ByWg'), ('x-amz-request-id', 'tx00000fe4b072ce65ff0fb-00631f82e2-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.397 DEBUG:rgw_multi.tests:zone=test-zone1 version=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:05:06.397 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:06.398 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:06.398 DEBUG:boto:Method: HEAD 2022-09-12T19:05:06.399 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:06.400 DEBUG:boto:Data: 2022-09-12T19:05:06.400 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.401 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:06.401 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.402 DEBUG:boto:Params: {} 2022-09-12T19:05:06.402 DEBUG:boto:Token: None 2022-09-12T19:05:06.403 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/ 2022-09-12T19:05:06.404 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg= 2022-09-12T19:05:06.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg=', 'Content-Length': '0'} 2022-09-12T19:05:06.407 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5e2d9c3090472b6-00631f82e2-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.408 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:06.408 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:06.409 DEBUG:boto:Method: PUT 2022-09-12T19:05:06.409 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:06.410 DEBUG:boto:Data: 2022-09-12T19:05:06.410 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:06.411 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:06.411 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.412 DEBUG:boto:Params: {} 2022-09-12T19:05:06.412 DEBUG:boto:Token: None 2022-09-12T19:05:06.413 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/obj 2022-09-12T19:05:06.414 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU= 2022-09-12T19:05:06.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU='} 2022-09-12T19:05:06.458 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY'), ('x-amz-request-id', 'tx0000017cabfa5a0ad7064-00631f82e2-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.459 DEBUG:rgw_multi.tests:zone=test-zone2 version=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:05:06.460 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:06.460 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:06.461 DEBUG:boto:Method: HEAD 2022-09-12T19:05:06.461 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:06.462 DEBUG:boto:Data: 2022-09-12T19:05:06.463 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.464 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:06.465 DEBUG:boto:Port: 8001 2022-09-12T19:05:06.466 DEBUG:boto:Params: {} 2022-09-12T19:05:06.467 DEBUG:boto:Token: None 2022-09-12T19:05:06.468 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/ 2022-09-12T19:05:06.469 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg= 2022-09-12T19:05:06.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg=', 'Content-Length': '0'} 2022-09-12T19:05:06.475 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c8720e304e5622e-00631f82e2-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.476 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:06.477 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:06.478 DEBUG:boto:Method: PUT 2022-09-12T19:05:06.479 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:06.480 DEBUG:boto:Data: 2022-09-12T19:05:06.481 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:06.482 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:06.483 DEBUG:boto:Port: 8001 2022-09-12T19:05:06.483 DEBUG:boto:Params: {} 2022-09-12T19:05:06.485 DEBUG:boto:Token: None 2022-09-12T19:05:06.486 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/obj 2022-09-12T19:05:06.487 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU= 2022-09-12T19:05:06.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU='} 2022-09-12T19:05:06.538 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY'), ('x-amz-request-id', 'tx000007a967b7e70b3340d-00631f82e2-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.539 DEBUG:rgw_multi.tests:zone=test-zone3 version=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:05:06.539 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:06.540 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:06.541 DEBUG:boto:Method: HEAD 2022-09-12T19:05:06.542 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:06.543 DEBUG:boto:Data: 2022-09-12T19:05:06.544 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.544 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:06.545 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.546 DEBUG:boto:Params: {} 2022-09-12T19:05:06.547 DEBUG:boto:Token: None 2022-09-12T19:05:06.548 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/ 2022-09-12T19:05:06.549 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg= 2022-09-12T19:05:06.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg=', 'Content-Length': '0'} 2022-09-12T19:05:06.553 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059c98ac31645480f-00631f82e2-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.555 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:06.555 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:06.556 DEBUG:boto:Method: PUT 2022-09-12T19:05:06.557 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:06.558 DEBUG:boto:Data: 2022-09-12T19:05:06.559 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:06.560 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:06.561 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.561 DEBUG:boto:Params: {} 2022-09-12T19:05:06.563 DEBUG:boto:Token: None 2022-09-12T19:05:06.563 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/obj 2022-09-12T19:05:06.564 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU= 2022-09-12T19:05:06.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU='} 2022-09-12T19:05:06.579 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9'), ('x-amz-request-id', 'tx00000d0314d66e8df7ad0-00631f82e2-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.579 DEBUG:rgw_multi.tests:zone=test-zone1 version=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:05:06.580 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:06.580 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:06.581 DEBUG:boto:Method: HEAD 2022-09-12T19:05:06.581 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:06.582 DEBUG:boto:Data: 2022-09-12T19:05:06.582 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.583 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:06.583 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.584 DEBUG:boto:Params: {} 2022-09-12T19:05:06.584 DEBUG:boto:Token: None 2022-09-12T19:05:06.585 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/ 2022-09-12T19:05:06.585 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg= 2022-09-12T19:05:06.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg=', 'Content-Length': '0'} 2022-09-12T19:05:06.589 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000741f308386d90fdd-00631f82e2-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.590 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:06.591 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:06.592 DEBUG:boto:Method: PUT 2022-09-12T19:05:06.592 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:06.593 DEBUG:boto:Data: 2022-09-12T19:05:06.593 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:06.594 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:06.595 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.595 DEBUG:boto:Params: {} 2022-09-12T19:05:06.596 DEBUG:boto:Token: None 2022-09-12T19:05:06.596 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/obj 2022-09-12T19:05:06.597 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU= 2022-09-12T19:05:06.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU='} 2022-09-12T19:05:06.615 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd'), ('x-amz-request-id', 'tx00000e031071ce4d35bd5-00631f82e2-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.616 DEBUG:rgw_multi.tests:zone=test-zone2 version=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:05:06.617 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:06.617 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:06.618 DEBUG:boto:Method: HEAD 2022-09-12T19:05:06.618 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:06.619 DEBUG:boto:Data: 2022-09-12T19:05:06.619 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.620 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:06.621 DEBUG:boto:Port: 8001 2022-09-12T19:05:06.621 DEBUG:boto:Params: {} 2022-09-12T19:05:06.622 DEBUG:boto:Token: None 2022-09-12T19:05:06.622 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/ 2022-09-12T19:05:06.623 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg= 2022-09-12T19:05:06.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg=', 'Content-Length': '0'} 2022-09-12T19:05:06.626 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005f73943bbc45fc7-00631f82e2-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.627 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:06.627 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:06.628 DEBUG:boto:Method: PUT 2022-09-12T19:05:06.628 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:06.629 DEBUG:boto:Data: 2022-09-12T19:05:06.629 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:06.630 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:06.630 DEBUG:boto:Port: 8001 2022-09-12T19:05:06.631 DEBUG:boto:Params: {} 2022-09-12T19:05:06.631 DEBUG:boto:Token: None 2022-09-12T19:05:06.632 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/obj 2022-09-12T19:05:06.633 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU= 2022-09-12T19:05:06.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU='} 2022-09-12T19:05:06.647 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'j0eE53eOEuIue0LhYTVuZStUTmcOj7o'), ('x-amz-request-id', 'tx0000021ccd50a963a9a80-00631f82e2-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.648 DEBUG:rgw_multi.tests:zone=test-zone3 version=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:05:06.649 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:06.649 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:06.650 DEBUG:boto:Method: HEAD 2022-09-12T19:05:06.650 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:06.651 DEBUG:boto:Data: 2022-09-12T19:05:06.651 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.652 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:06.653 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.653 DEBUG:boto:Params: {} 2022-09-12T19:05:06.654 DEBUG:boto:Token: None 2022-09-12T19:05:06.654 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/ 2022-09-12T19:05:06.655 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg= 2022-09-12T19:05:06.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg=', 'Content-Length': '0'} 2022-09-12T19:05:06.658 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067a5b0ffbf799ae9-00631f82e2-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.659 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:06.659 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:06.660 DEBUG:boto:Method: PUT 2022-09-12T19:05:06.660 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:06.661 DEBUG:boto:Data: 2022-09-12T19:05:06.661 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:06.662 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:06.663 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.663 DEBUG:boto:Params: {} 2022-09-12T19:05:06.664 DEBUG:boto:Token: None 2022-09-12T19:05:06.664 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/obj 2022-09-12T19:05:06.665 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU= 2022-09-12T19:05:06.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU='} 2022-09-12T19:05:06.677 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'p5OIjHSzwvUDkIBYzRPdKBGezjIkrn-'), ('x-amz-request-id', 'tx00000f7f64905b492fe37-00631f82e2-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.678 DEBUG:rgw_multi.tests:zone=test-zone1 version=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:05:06.678 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:06.679 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:06.680 DEBUG:boto:Method: HEAD 2022-09-12T19:05:06.680 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:06.681 DEBUG:boto:Data: 2022-09-12T19:05:06.681 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.682 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:06.682 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.683 DEBUG:boto:Params: {} 2022-09-12T19:05:06.683 DEBUG:boto:Token: None 2022-09-12T19:05:06.684 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/ 2022-09-12T19:05:06.685 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg= 2022-09-12T19:05:06.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg=', 'Content-Length': '0'} 2022-09-12T19:05:06.689 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b57556147efa26be-00631f82e2-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.690 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:06.690 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:06.691 DEBUG:boto:Method: PUT 2022-09-12T19:05:06.691 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:06.692 DEBUG:boto:Data: 2022-09-12T19:05:06.692 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:06.693 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:06.693 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.694 DEBUG:boto:Params: {} 2022-09-12T19:05:06.695 DEBUG:boto:Token: None 2022-09-12T19:05:06.695 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/obj 2022-09-12T19:05:06.696 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU= 2022-09-12T19:05:06.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU='} 2022-09-12T19:05:06.714 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'lnpiHoSs91QVujLSWqHhvEGzZrF3-sA'), ('x-amz-request-id', 'tx0000095a2950bf3469e7f-00631f82e2-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.715 DEBUG:rgw_multi.tests:zone=test-zone2 version=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:05:06.716 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:06.716 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:06.717 DEBUG:boto:Method: HEAD 2022-09-12T19:05:06.717 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:06.718 DEBUG:boto:Data: 2022-09-12T19:05:06.718 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.719 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:06.719 DEBUG:boto:Port: 8001 2022-09-12T19:05:06.720 DEBUG:boto:Params: {} 2022-09-12T19:05:06.721 DEBUG:boto:Token: None 2022-09-12T19:05:06.722 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/ 2022-09-12T19:05:06.722 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg= 2022-09-12T19:05:06.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg=', 'Content-Length': '0'} 2022-09-12T19:05:06.725 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c5d5add587df80c-00631f82e2-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.726 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:06.727 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:06.727 DEBUG:boto:Method: PUT 2022-09-12T19:05:06.728 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:06.728 DEBUG:boto:Data: 2022-09-12T19:05:06.729 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:06.729 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:06.730 DEBUG:boto:Port: 8001 2022-09-12T19:05:06.730 DEBUG:boto:Params: {} 2022-09-12T19:05:06.731 DEBUG:boto:Token: None 2022-09-12T19:05:06.731 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/obj 2022-09-12T19:05:06.732 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU= 2022-09-12T19:05:06.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU='} 2022-09-12T19:05:06.748 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE'), ('x-amz-request-id', 'tx000000143de24984b26ff-00631f82e2-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.750 DEBUG:rgw_multi.tests:zone=test-zone3 version=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:05:06.750 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:06.751 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:06.751 DEBUG:boto:Method: HEAD 2022-09-12T19:05:06.752 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:06.752 DEBUG:boto:Data: 2022-09-12T19:05:06.753 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.753 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:06.754 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.754 DEBUG:boto:Params: {} 2022-09-12T19:05:06.755 DEBUG:boto:Token: None 2022-09-12T19:05:06.756 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/ 2022-09-12T19:05:06.757 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg= 2022-09-12T19:05:06.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg=', 'Content-Length': '0'} 2022-09-12T19:05:06.760 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c36f9fd19d0fc003-00631f82e2-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.762 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:06.763 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:06.764 DEBUG:boto:Method: PUT 2022-09-12T19:05:06.764 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:06.768 DEBUG:boto:Data: 2022-09-12T19:05:06.768 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:06.769 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:06.769 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.770 DEBUG:boto:Params: {} 2022-09-12T19:05:06.770 DEBUG:boto:Token: None 2022-09-12T19:05:06.771 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/obj 2022-09-12T19:05:06.771 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU= 2022-09-12T19:05:06.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU='} 2022-09-12T19:05:06.784 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2'), ('x-amz-request-id', 'tx0000016ac0d1c9871b8a4-00631f82e2-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.785 DEBUG:rgw_multi.tests:zone=test-zone1 version=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:05:06.785 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:06.786 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:06.787 DEBUG:boto:Method: HEAD 2022-09-12T19:05:06.787 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:06.788 DEBUG:boto:Data: 2022-09-12T19:05:06.788 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.789 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:06.789 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.790 DEBUG:boto:Params: {} 2022-09-12T19:05:06.791 DEBUG:boto:Token: None 2022-09-12T19:05:06.791 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/ 2022-09-12T19:05:06.792 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg= 2022-09-12T19:05:06.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg=', 'Content-Length': '0'} 2022-09-12T19:05:06.795 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f1290213b2098c5-00631f82e2-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.796 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:06.796 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:06.797 DEBUG:boto:Method: PUT 2022-09-12T19:05:06.797 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:06.798 DEBUG:boto:Data: 2022-09-12T19:05:06.799 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:06.799 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:06.800 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.800 DEBUG:boto:Params: {} 2022-09-12T19:05:06.801 DEBUG:boto:Token: None 2022-09-12T19:05:06.801 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/obj 2022-09-12T19:05:06.802 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU= 2022-09-12T19:05:06.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU='} 2022-09-12T19:05:06.813 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'BAbppx4RylEKnBrYesXt1QPJgsAhIBb'), ('x-amz-request-id', 'tx000008944a47ece06a068-00631f82e2-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.814 DEBUG:rgw_multi.tests:zone=test-zone2 version=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:05:06.815 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:06.815 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:06.816 DEBUG:boto:Method: HEAD 2022-09-12T19:05:06.816 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:06.817 DEBUG:boto:Data: 2022-09-12T19:05:06.817 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.818 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:06.818 DEBUG:boto:Port: 8001 2022-09-12T19:05:06.819 DEBUG:boto:Params: {} 2022-09-12T19:05:06.820 DEBUG:boto:Token: None 2022-09-12T19:05:06.820 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/ 2022-09-12T19:05:06.821 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg= 2022-09-12T19:05:06.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg=', 'Content-Length': '0'} 2022-09-12T19:05:06.824 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007451c3e476e24d1b-00631f82e2-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.826 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:06.826 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:06.827 DEBUG:boto:Method: PUT 2022-09-12T19:05:06.828 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:06.828 DEBUG:boto:Data: 2022-09-12T19:05:06.829 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:06.829 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:06.830 DEBUG:boto:Port: 8001 2022-09-12T19:05:06.830 DEBUG:boto:Params: {} 2022-09-12T19:05:06.831 DEBUG:boto:Token: None 2022-09-12T19:05:06.832 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/obj 2022-09-12T19:05:06.833 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU= 2022-09-12T19:05:06.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU='} 2022-09-12T19:05:06.845 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'e.6ruI53roqy4mmyPgzQ1kqMdajruFA'), ('x-amz-request-id', 'tx000004a851de0a8c5d76d-00631f82e2-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.846 DEBUG:rgw_multi.tests:zone=test-zone3 version=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:05:06.847 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:06.847 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:06.848 DEBUG:boto:Method: HEAD 2022-09-12T19:05:06.848 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:06.849 DEBUG:boto:Data: 2022-09-12T19:05:06.850 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.850 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:06.851 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.851 DEBUG:boto:Params: {} 2022-09-12T19:05:06.852 DEBUG:boto:Token: None 2022-09-12T19:05:06.853 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/ 2022-09-12T19:05:06.853 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg= 2022-09-12T19:05:06.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg=', 'Content-Length': '0'} 2022-09-12T19:05:06.858 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005dd218dd2175ef5c-00631f82e2-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.859 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:06.859 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:06.860 DEBUG:boto:Method: PUT 2022-09-12T19:05:06.860 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:06.861 DEBUG:boto:Data: 2022-09-12T19:05:06.861 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:06.862 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:06.862 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.863 DEBUG:boto:Params: {} 2022-09-12T19:05:06.863 DEBUG:boto:Token: None 2022-09-12T19:05:06.864 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/obj 2022-09-12T19:05:06.865 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU= 2022-09-12T19:05:06.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU='} 2022-09-12T19:05:06.877 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3'), ('x-amz-request-id', 'tx00000d80d687d505bdae1-00631f82e2-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.878 DEBUG:rgw_multi.tests:zone=test-zone1 version=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:05:06.879 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:06.880 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:06.881 DEBUG:boto:Method: HEAD 2022-09-12T19:05:06.882 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:06.883 DEBUG:boto:Data: 2022-09-12T19:05:06.884 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.885 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:06.886 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.887 DEBUG:boto:Params: {} 2022-09-12T19:05:06.888 DEBUG:boto:Token: None 2022-09-12T19:05:06.889 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/ 2022-09-12T19:05:06.890 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg= 2022-09-12T19:05:06.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg=', 'Content-Length': '0'} 2022-09-12T19:05:06.894 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000208b0ee490d1e03a-00631f82e2-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.895 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:06.897 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:06.897 DEBUG:boto:Method: PUT 2022-09-12T19:05:06.899 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:06.899 DEBUG:boto:Data: 2022-09-12T19:05:06.900 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:06.901 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:06.902 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.903 DEBUG:boto:Params: {} 2022-09-12T19:05:06.904 DEBUG:boto:Token: None 2022-09-12T19:05:06.905 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/obj 2022-09-12T19:05:06.906 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU= 2022-09-12T19:05:06.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU='} 2022-09-12T19:05:06.920 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H'), ('x-amz-request-id', 'tx00000a760d5216f10790f-00631f82e2-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.922 DEBUG:rgw_multi.tests:zone=test-zone2 version=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:05:06.923 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:06.924 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:06.925 DEBUG:boto:Method: HEAD 2022-09-12T19:05:06.925 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:06.926 DEBUG:boto:Data: 2022-09-12T19:05:06.927 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.928 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:06.930 DEBUG:boto:Port: 8001 2022-09-12T19:05:06.931 DEBUG:boto:Params: {} 2022-09-12T19:05:06.932 DEBUG:boto:Token: None 2022-09-12T19:05:06.933 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/ 2022-09-12T19:05:06.934 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg= 2022-09-12T19:05:06.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg=', 'Content-Length': '0'} 2022-09-12T19:05:06.938 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc350eadcddcbaa9-00631f82e2-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.939 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:06.940 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:06.941 DEBUG:boto:Method: PUT 2022-09-12T19:05:06.942 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:06.943 DEBUG:boto:Data: 2022-09-12T19:05:06.944 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:06.945 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:06.946 DEBUG:boto:Port: 8001 2022-09-12T19:05:06.947 DEBUG:boto:Params: {} 2022-09-12T19:05:06.948 DEBUG:boto:Token: None 2022-09-12T19:05:06.949 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/obj 2022-09-12T19:05:06.950 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU= 2022-09-12T19:05:06.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU='} 2022-09-12T19:05:06.964 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'QHXOu7SGV87vItdBeqaL.hYUlX--Ty8'), ('x-amz-request-id', 'tx00000e34dcd67049cd010-00631f82e2-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.965 DEBUG:rgw_multi.tests:zone=test-zone3 version=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:05:06.966 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:06.967 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:06.968 DEBUG:boto:Method: HEAD 2022-09-12T19:05:06.969 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:06.970 DEBUG:boto:Data: 2022-09-12T19:05:06.971 DEBUG:boto:Headers: {} 2022-09-12T19:05:06.972 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:06.973 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.974 DEBUG:boto:Params: {} 2022-09-12T19:05:06.975 DEBUG:boto:Token: None 2022-09-12T19:05:06.976 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/ 2022-09-12T19:05:06.977 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg= 2022-09-12T19:05:06.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qXbXgAnwcbz7PJALRsRdhGeRCwg=', 'Content-Length': '0'} 2022-09-12T19:05:06.981 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be018e77b85d66c6-00631f82e2-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:06.983 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:06.984 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:06.984 DEBUG:boto:Method: PUT 2022-09-12T19:05:06.985 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:06.986 DEBUG:boto:Data: 2022-09-12T19:05:06.987 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:06.988 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:06.989 DEBUG:boto:Port: 8000 2022-09-12T19:05:06.990 DEBUG:boto:Params: {} 2022-09-12T19:05:06.991 DEBUG:boto:Token: None 2022-09-12T19:05:06.992 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:06 GMT /jdpiak-19/obj 2022-09-12T19:05:06.993 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU= 2022-09-12T19:05:06.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:otejUvVbC1MLrTVpabJjFJwSbLU='} 2022-09-12T19:05:07.006 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0'), ('x-amz-request-id', 'tx0000068f23b80ca79a888-00631f82e2-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.007 DEBUG:rgw_multi.tests:zone=test-zone1 version=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:05:07.008 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:07.009 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:07.010 DEBUG:boto:Method: HEAD 2022-09-12T19:05:07.011 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:07.012 DEBUG:boto:Data: 2022-09-12T19:05:07.013 DEBUG:boto:Headers: {} 2022-09-12T19:05:07.014 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:07.015 DEBUG:boto:Port: 8000 2022-09-12T19:05:07.016 DEBUG:boto:Params: {} 2022-09-12T19:05:07.017 DEBUG:boto:Token: None 2022-09-12T19:05:07.018 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/ 2022-09-12T19:05:07.019 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc= 2022-09-12T19:05:07.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc=', 'Content-Length': '0'} 2022-09-12T19:05:07.024 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c891874cf250e17a-00631f82e3-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.025 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:07.026 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:07.027 DEBUG:boto:Method: PUT 2022-09-12T19:05:07.028 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:07.029 DEBUG:boto:Data: 2022-09-12T19:05:07.030 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:07.031 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:07.032 DEBUG:boto:Port: 8000 2022-09-12T19:05:07.033 DEBUG:boto:Params: {} 2022-09-12T19:05:07.034 DEBUG:boto:Token: None 2022-09-12T19:05:07.035 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/obj 2022-09-12T19:05:07.036 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8= 2022-09-12T19:05:07.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8='} 2022-09-12T19:05:07.049 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'tQHk21B862H4VHQxrZWEog8n-qgO99Q'), ('x-amz-request-id', 'tx00000d29e8cc083de99ac-00631f82e3-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.050 DEBUG:rgw_multi.tests:zone=test-zone2 version=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:05:07.051 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:07.052 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:07.053 DEBUG:boto:Method: HEAD 2022-09-12T19:05:07.053 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:07.054 DEBUG:boto:Data: 2022-09-12T19:05:07.056 DEBUG:boto:Headers: {} 2022-09-12T19:05:07.056 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:07.057 DEBUG:boto:Port: 8001 2022-09-12T19:05:07.058 DEBUG:boto:Params: {} 2022-09-12T19:05:07.058 DEBUG:boto:Token: None 2022-09-12T19:05:07.059 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/ 2022-09-12T19:05:07.060 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc= 2022-09-12T19:05:07.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc=', 'Content-Length': '0'} 2022-09-12T19:05:07.064 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa5b241361d12411-00631f82e3-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.066 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:07.067 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:07.067 DEBUG:boto:Method: PUT 2022-09-12T19:05:07.068 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:07.069 DEBUG:boto:Data: 2022-09-12T19:05:07.069 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:07.070 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:07.070 DEBUG:boto:Port: 8001 2022-09-12T19:05:07.071 DEBUG:boto:Params: {} 2022-09-12T19:05:07.072 DEBUG:boto:Token: None 2022-09-12T19:05:07.072 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/obj 2022-09-12T19:05:07.073 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8= 2022-09-12T19:05:07.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8='} 2022-09-12T19:05:07.094 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'K.LUooPdPnXZMzfYIxw-xagQpErk4CX'), ('x-amz-request-id', 'tx00000dfe71c50fc4f1891-00631f82e3-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.096 DEBUG:rgw_multi.tests:zone=test-zone3 version=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:05:07.096 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:07.097 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:07.098 DEBUG:boto:Method: HEAD 2022-09-12T19:05:07.099 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:07.100 DEBUG:boto:Data: 2022-09-12T19:05:07.100 DEBUG:boto:Headers: {} 2022-09-12T19:05:07.101 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:07.102 DEBUG:boto:Port: 8000 2022-09-12T19:05:07.103 DEBUG:boto:Params: {} 2022-09-12T19:05:07.104 DEBUG:boto:Token: None 2022-09-12T19:05:07.105 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/ 2022-09-12T19:05:07.105 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc= 2022-09-12T19:05:07.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc=', 'Content-Length': '0'} 2022-09-12T19:05:07.108 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d38ad6d4b214820c-00631f82e3-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.109 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:07.110 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:07.111 DEBUG:boto:Method: PUT 2022-09-12T19:05:07.111 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:07.112 DEBUG:boto:Data: 2022-09-12T19:05:07.112 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:07.113 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:07.114 DEBUG:boto:Port: 8000 2022-09-12T19:05:07.115 DEBUG:boto:Params: {} 2022-09-12T19:05:07.116 DEBUG:boto:Token: None 2022-09-12T19:05:07.117 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/obj 2022-09-12T19:05:07.118 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8= 2022-09-12T19:05:07.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8='} 2022-09-12T19:05:07.137 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Katr07t9mQV0b0gx44aB8d9VyeAhMS1'), ('x-amz-request-id', 'tx000007e5e58a54ea41a34-00631f82e3-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.138 DEBUG:rgw_multi.tests:zone=test-zone1 version=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:05:07.139 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:07.139 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:07.140 DEBUG:boto:Method: HEAD 2022-09-12T19:05:07.141 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:07.142 DEBUG:boto:Data: 2022-09-12T19:05:07.142 DEBUG:boto:Headers: {} 2022-09-12T19:05:07.143 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:07.144 DEBUG:boto:Port: 8000 2022-09-12T19:05:07.145 DEBUG:boto:Params: {} 2022-09-12T19:05:07.146 DEBUG:boto:Token: None 2022-09-12T19:05:07.147 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/ 2022-09-12T19:05:07.148 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc= 2022-09-12T19:05:07.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc=', 'Content-Length': '0'} 2022-09-12T19:05:07.152 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054479e0a029429ef-00631f82e3-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.153 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:07.154 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:07.155 DEBUG:boto:Method: PUT 2022-09-12T19:05:07.156 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:07.157 DEBUG:boto:Data: 2022-09-12T19:05:07.158 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:07.160 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:07.160 DEBUG:boto:Port: 8000 2022-09-12T19:05:07.161 DEBUG:boto:Params: {} 2022-09-12T19:05:07.162 DEBUG:boto:Token: None 2022-09-12T19:05:07.163 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/obj 2022-09-12T19:05:07.163 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8= 2022-09-12T19:05:07.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8='} 2022-09-12T19:05:07.175 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'UBsYcCA94H55K4ucww5vXNAxuAMKNSG'), ('x-amz-request-id', 'tx0000005084355aed87e06-00631f82e3-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.176 DEBUG:rgw_multi.tests:zone=test-zone2 version=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:05:07.176 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:07.177 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:07.177 DEBUG:boto:Method: HEAD 2022-09-12T19:05:07.178 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:07.178 DEBUG:boto:Data: 2022-09-12T19:05:07.179 DEBUG:boto:Headers: {} 2022-09-12T19:05:07.179 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:07.180 DEBUG:boto:Port: 8001 2022-09-12T19:05:07.180 DEBUG:boto:Params: {} 2022-09-12T19:05:07.181 DEBUG:boto:Token: None 2022-09-12T19:05:07.182 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/ 2022-09-12T19:05:07.182 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc= 2022-09-12T19:05:07.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc=', 'Content-Length': '0'} 2022-09-12T19:05:07.185 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9cb7942d3e228ba-00631f82e3-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.186 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:07.187 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:07.188 DEBUG:boto:Method: PUT 2022-09-12T19:05:07.188 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:07.189 DEBUG:boto:Data: 2022-09-12T19:05:07.190 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:07.192 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:07.193 DEBUG:boto:Port: 8001 2022-09-12T19:05:07.194 DEBUG:boto:Params: {} 2022-09-12T19:05:07.195 DEBUG:boto:Token: None 2022-09-12T19:05:07.197 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/obj 2022-09-12T19:05:07.198 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8= 2022-09-12T19:05:07.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8='} 2022-09-12T19:05:07.215 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW'), ('x-amz-request-id', 'tx00000873c65697f5ca341-00631f82e3-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.217 DEBUG:rgw_multi.tests:zone=test-zone3 version=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:05:07.218 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:07.219 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:07.219 DEBUG:boto:Method: HEAD 2022-09-12T19:05:07.220 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:07.221 DEBUG:boto:Data: 2022-09-12T19:05:07.221 DEBUG:boto:Headers: {} 2022-09-12T19:05:07.222 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:07.223 DEBUG:boto:Port: 8000 2022-09-12T19:05:07.223 DEBUG:boto:Params: {} 2022-09-12T19:05:07.224 DEBUG:boto:Token: None 2022-09-12T19:05:07.225 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/ 2022-09-12T19:05:07.226 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc= 2022-09-12T19:05:07.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc=', 'Content-Length': '0'} 2022-09-12T19:05:07.229 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000840bf702cade0135-00631f82e3-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.230 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:07.231 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:07.231 DEBUG:boto:Method: PUT 2022-09-12T19:05:07.232 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:07.232 DEBUG:boto:Data: 2022-09-12T19:05:07.233 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:07.233 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:07.234 DEBUG:boto:Port: 8000 2022-09-12T19:05:07.234 DEBUG:boto:Params: {} 2022-09-12T19:05:07.235 DEBUG:boto:Token: None 2022-09-12T19:05:07.236 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/obj 2022-09-12T19:05:07.236 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8= 2022-09-12T19:05:07.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8='} 2022-09-12T19:05:07.250 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'T.yM--FII-PB1GReM85mkNRGYz6Dkbz'), ('x-amz-request-id', 'tx00000acc1954eff5f543a-00631f82e3-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.251 DEBUG:rgw_multi.tests:zone=test-zone1 version=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:05:07.252 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:07.255 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:07.255 DEBUG:boto:Method: HEAD 2022-09-12T19:05:07.256 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:07.256 DEBUG:boto:Data: 2022-09-12T19:05:07.257 DEBUG:boto:Headers: {} 2022-09-12T19:05:07.257 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:07.258 DEBUG:boto:Port: 8000 2022-09-12T19:05:07.258 DEBUG:boto:Params: {} 2022-09-12T19:05:07.259 DEBUG:boto:Token: None 2022-09-12T19:05:07.259 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/ 2022-09-12T19:05:07.260 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc= 2022-09-12T19:05:07.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc=', 'Content-Length': '0'} 2022-09-12T19:05:07.263 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000428bd8524c1fc88e-00631f82e3-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.264 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:07.265 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:07.265 DEBUG:boto:Method: PUT 2022-09-12T19:05:07.266 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:07.266 DEBUG:boto:Data: 2022-09-12T19:05:07.267 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:07.267 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:07.268 DEBUG:boto:Port: 8000 2022-09-12T19:05:07.268 DEBUG:boto:Params: {} 2022-09-12T19:05:07.270 DEBUG:boto:Token: None 2022-09-12T19:05:07.270 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/obj 2022-09-12T19:05:07.272 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8= 2022-09-12T19:05:07.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8='} 2022-09-12T19:05:07.320 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ztQRphsNbUdApzTcOcagUJ56uTwo7J2'), ('x-amz-request-id', 'tx000007e54d2206c0d28b8-00631f82e3-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.323 DEBUG:rgw_multi.tests:zone=test-zone2 version=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:05:07.324 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:07.326 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:07.326 DEBUG:boto:Method: HEAD 2022-09-12T19:05:07.327 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:07.327 DEBUG:boto:Data: 2022-09-12T19:05:07.328 DEBUG:boto:Headers: {} 2022-09-12T19:05:07.328 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:07.329 DEBUG:boto:Port: 8001 2022-09-12T19:05:07.329 DEBUG:boto:Params: {} 2022-09-12T19:05:07.330 DEBUG:boto:Token: None 2022-09-12T19:05:07.331 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/ 2022-09-12T19:05:07.331 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc= 2022-09-12T19:05:07.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc=', 'Content-Length': '0'} 2022-09-12T19:05:07.334 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b80de2e6cf2295c1-00631f82e3-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.335 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:07.336 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:07.336 DEBUG:boto:Method: PUT 2022-09-12T19:05:07.337 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:07.337 DEBUG:boto:Data: 2022-09-12T19:05:07.338 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:07.338 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:07.339 DEBUG:boto:Port: 8001 2022-09-12T19:05:07.339 DEBUG:boto:Params: {} 2022-09-12T19:05:07.340 DEBUG:boto:Token: None 2022-09-12T19:05:07.341 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/obj 2022-09-12T19:05:07.342 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8= 2022-09-12T19:05:07.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8='} 2022-09-12T19:05:07.356 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Xb5UZAvutXF6GhBCMdUQgjOLi998.hK'), ('x-amz-request-id', 'tx00000e485db6c8dfdaf67-00631f82e3-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.357 DEBUG:rgw_multi.tests:zone=test-zone3 version=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:05:07.358 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:07.359 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:07.360 DEBUG:boto:Method: HEAD 2022-09-12T19:05:07.360 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:07.361 DEBUG:boto:Data: 2022-09-12T19:05:07.362 DEBUG:boto:Headers: {} 2022-09-12T19:05:07.363 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:07.364 DEBUG:boto:Port: 8000 2022-09-12T19:05:07.364 DEBUG:boto:Params: {} 2022-09-12T19:05:07.365 DEBUG:boto:Token: None 2022-09-12T19:05:07.366 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/ 2022-09-12T19:05:07.367 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc= 2022-09-12T19:05:07.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc=', 'Content-Length': '0'} 2022-09-12T19:05:07.371 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023de1dbe876dbca7-00631f82e3-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.372 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:07.373 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:07.374 DEBUG:boto:Method: PUT 2022-09-12T19:05:07.375 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:07.376 DEBUG:boto:Data: 2022-09-12T19:05:07.376 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:07.377 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:07.378 DEBUG:boto:Port: 8000 2022-09-12T19:05:07.379 DEBUG:boto:Params: {} 2022-09-12T19:05:07.380 DEBUG:boto:Token: None 2022-09-12T19:05:07.381 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/obj 2022-09-12T19:05:07.382 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8= 2022-09-12T19:05:07.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8='} 2022-09-12T19:05:07.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'pwncGyI2j8AeHH4hjzwsep2DWsp-l-A'), ('x-amz-request-id', 'tx0000005a9701bd758c1e7-00631f82e3-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.399 DEBUG:rgw_multi.tests:zone=test-zone1 version=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:05:07.399 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:07.400 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:07.401 DEBUG:boto:Method: HEAD 2022-09-12T19:05:07.401 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:07.402 DEBUG:boto:Data: 2022-09-12T19:05:07.402 DEBUG:boto:Headers: {} 2022-09-12T19:05:07.403 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:07.403 DEBUG:boto:Port: 8000 2022-09-12T19:05:07.404 DEBUG:boto:Params: {} 2022-09-12T19:05:07.404 DEBUG:boto:Token: None 2022-09-12T19:05:07.405 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/ 2022-09-12T19:05:07.406 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc= 2022-09-12T19:05:07.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc=', 'Content-Length': '0'} 2022-09-12T19:05:07.409 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000fabb44964e71f4-00631f82e3-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.410 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:07.410 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:07.411 DEBUG:boto:Method: PUT 2022-09-12T19:05:07.411 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:07.412 DEBUG:boto:Data: 2022-09-12T19:05:07.412 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:07.413 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:07.413 DEBUG:boto:Port: 8000 2022-09-12T19:05:07.414 DEBUG:boto:Params: {} 2022-09-12T19:05:07.417 DEBUG:boto:Token: None 2022-09-12T19:05:07.417 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/obj 2022-09-12T19:05:07.418 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8= 2022-09-12T19:05:07.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8='} 2022-09-12T19:05:07.430 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ'), ('x-amz-request-id', 'tx000005b7157b56ea69f5b-00631f82e3-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.430 DEBUG:rgw_multi.tests:zone=test-zone2 version=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:05:07.431 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:07.431 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:07.432 DEBUG:boto:Method: HEAD 2022-09-12T19:05:07.433 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:07.433 DEBUG:boto:Data: 2022-09-12T19:05:07.434 DEBUG:boto:Headers: {} 2022-09-12T19:05:07.434 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:07.435 DEBUG:boto:Port: 8001 2022-09-12T19:05:07.435 DEBUG:boto:Params: {} 2022-09-12T19:05:07.436 DEBUG:boto:Token: None 2022-09-12T19:05:07.436 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/ 2022-09-12T19:05:07.437 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc= 2022-09-12T19:05:07.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc=', 'Content-Length': '0'} 2022-09-12T19:05:07.442 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f97ca666e73a93e-00631f82e3-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.444 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:07.445 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:07.446 DEBUG:boto:Method: PUT 2022-09-12T19:05:07.447 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:07.448 DEBUG:boto:Data: 2022-09-12T19:05:07.449 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:07.450 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:07.451 DEBUG:boto:Port: 8001 2022-09-12T19:05:07.452 DEBUG:boto:Params: {} 2022-09-12T19:05:07.453 DEBUG:boto:Token: None 2022-09-12T19:05:07.454 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/obj 2022-09-12T19:05:07.455 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8= 2022-09-12T19:05:07.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8='} 2022-09-12T19:05:07.467 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh'), ('x-amz-request-id', 'tx00000ca18647666790d8a-00631f82e3-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.468 DEBUG:rgw_multi.tests:zone=test-zone3 version=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:05:07.469 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:07.469 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:07.470 DEBUG:boto:Method: HEAD 2022-09-12T19:05:07.471 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:07.471 DEBUG:boto:Data: 2022-09-12T19:05:07.472 DEBUG:boto:Headers: {} 2022-09-12T19:05:07.472 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:07.473 DEBUG:boto:Port: 8000 2022-09-12T19:05:07.473 DEBUG:boto:Params: {} 2022-09-12T19:05:07.474 DEBUG:boto:Token: None 2022-09-12T19:05:07.474 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/ 2022-09-12T19:05:07.475 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc= 2022-09-12T19:05:07.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc=', 'Content-Length': '0'} 2022-09-12T19:05:07.478 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d814d0df1d03906-00631f82e3-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.479 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:07.479 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:07.480 DEBUG:boto:Method: PUT 2022-09-12T19:05:07.480 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:07.481 DEBUG:boto:Data: 2022-09-12T19:05:07.481 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:07.482 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:05:07.482 DEBUG:boto:Port: 8000 2022-09-12T19:05:07.483 DEBUG:boto:Params: {} 2022-09-12T19:05:07.483 DEBUG:boto:Token: None 2022-09-12T19:05:07.484 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/obj 2022-09-12T19:05:07.485 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8= 2022-09-12T19:05:07.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8='} 2022-09-12T19:05:07.530 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul'), ('x-amz-request-id', 'tx00000c4b808e809c4a555-00631f82e3-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.531 DEBUG:rgw_multi.tests:zone=test-zone1 version=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:05:07.532 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:07.532 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:07.533 DEBUG:boto:Method: HEAD 2022-09-12T19:05:07.533 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:07.534 DEBUG:boto:Data: 2022-09-12T19:05:07.534 DEBUG:boto:Headers: {} 2022-09-12T19:05:07.535 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:07.535 DEBUG:boto:Port: 8000 2022-09-12T19:05:07.536 DEBUG:boto:Params: {} 2022-09-12T19:05:07.537 DEBUG:boto:Token: None 2022-09-12T19:05:07.537 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/ 2022-09-12T19:05:07.538 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc= 2022-09-12T19:05:07.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc=', 'Content-Length': '0'} 2022-09-12T19:05:07.541 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b33fceb5f73ba7a9-00631f82e3-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.542 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:07.542 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:07.543 DEBUG:boto:Method: PUT 2022-09-12T19:05:07.543 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:07.544 DEBUG:boto:Data: 2022-09-12T19:05:07.544 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:07.545 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:05:07.545 DEBUG:boto:Port: 8000 2022-09-12T19:05:07.546 DEBUG:boto:Params: {} 2022-09-12T19:05:07.546 DEBUG:boto:Token: None 2022-09-12T19:05:07.547 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/obj 2022-09-12T19:05:07.548 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8= 2022-09-12T19:05:07.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8='} 2022-09-12T19:05:07.559 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-amz-request-id', 'tx0000033ffd38371302725-00631f82e3-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.560 DEBUG:rgw_multi.tests:zone=test-zone2 version=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:05:07.561 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:05:07.561 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:05:07.562 DEBUG:boto:Method: HEAD 2022-09-12T19:05:07.563 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:05:07.564 DEBUG:boto:Data: 2022-09-12T19:05:07.564 DEBUG:boto:Headers: {} 2022-09-12T19:05:07.565 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:07.567 DEBUG:boto:Port: 8001 2022-09-12T19:05:07.567 DEBUG:boto:Params: {} 2022-09-12T19:05:07.568 DEBUG:boto:Token: None 2022-09-12T19:05:07.569 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/ 2022-09-12T19:05:07.569 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc= 2022-09-12T19:05:07.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2HRBXzRclCizTYXe2xm2sfI8Qgc=', 'Content-Length': '0'} 2022-09-12T19:05:07.573 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065f92748f92166f1-00631f82e3-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.573 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:05:07.574 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:05:07.575 DEBUG:boto:Method: PUT 2022-09-12T19:05:07.575 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:05:07.576 DEBUG:boto:Data: 2022-09-12T19:05:07.576 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-12T19:05:07.577 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:05:07.577 DEBUG:boto:Port: 8001 2022-09-12T19:05:07.578 DEBUG:boto:Params: {} 2022-09-12T19:05:07.578 DEBUG:boto:Token: None 2022-09-12T19:05:07.579 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Mon, 12 Sep 2022 19:05:07 GMT /jdpiak-19/obj 2022-09-12T19:05:07.580 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8= 2022-09-12T19:05:07.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:05:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wk4HxY1NOe3H3O3llNHCDLQakA8='} 2022-09-12T19:05:07.591 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'TM67hyaYWRVq578zUsjsXjOzkUYTIvA'), ('x-amz-request-id', 'tx000000d0c66269ddb89a9-00631f82e3-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:05:07.592 DEBUG:rgw_multi.tests:zone=test-zone3 version=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:05:07.593 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-19 2022-09-12T19:05:07.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:05:07.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:05:07.595 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:05:07.658 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:05:07.659 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:05:07.713 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:05:07.710+0000 7f056a12a600 1 waiting for incremental sync to catch up: 2022-09-12T19:05:07.713 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000005.17.9, , , , , ] 2022-09-12T19:05:07.714 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000005.17.9, , , 00000000020.75.11, , ] 2022-09-12T19:05:12.714 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:05:12.712+0000 7f056a12a600 1 waiting for incremental sync to catch up: 2022-09-12T19:05:12.715 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000005.17.9, , , , , ] 2022-09-12T19:05:12.715 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000005.17.9, , , 00000000020.75.11, , ] 2022-09-12T19:05:17.716 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:05:17.713+0000 7f056a12a600 1 waiting for incremental sync to catch up: 2022-09-12T19:05:17.716 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000005.17.9, , , , , ] 2022-09-12T19:05:17.717 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000005.17.9, , , 00000000020.75.11, , ] 2022-09-12T19:05:22.716 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:05:22.713+0000 7f056a12a600 1 bucket sync caught up with source: 2022-09-12T19:05:22.717 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000005.17.9, , , 00000000024.87.5, , ] 2022-09-12T19:05:22.717 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000005.17.9, , , 00000000020.75.11, , ] 2022-09-12T19:05:22.717 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:05:22.713+0000 7f056a12a600 0 bucket checkpoint complete 2022-09-12T19:05:22.730 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-19 2022-09-12T19:05:22.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:05:22.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:05:22.732 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:05:22.794 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:05:22.795 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:05:22.850 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:05:22.847+0000 7f92fb273600 1 bucket sync caught up with source: 2022-09-12T19:05:22.850 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000005.17.9, , , 00000000024.87.5, , ] 2022-09-12T19:05:22.851 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000005.17.9, , , 00000000024.87.5, , ] 2022-09-12T19:05:22.851 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:05:22.847+0000 7f92fb273600 0 bucket checkpoint complete 2022-09-12T19:05:22.868 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-19 2022-09-12T19:05:22.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:05:22.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:05:22.870 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:05:22.927 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:05:22.929 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:05:22.971 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:05:22.968+0000 7ff6a3dbc600 1 waiting for incremental sync to catch up: 2022-09-12T19:05:22.972 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000005.17.9, , , 00000000014.63.11, , ] 2022-09-12T19:05:22.972 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000005.17.9, , , 00000000024.87.5, , ] 2022-09-12T19:05:27.973 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:05:27.969+0000 7ff6a3dbc600 1 bucket sync caught up with source: 2022-09-12T19:05:27.973 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000005.17.9, , , 00000000024.87.5, , ] 2022-09-12T19:05:27.973 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000005.17.9, , , 00000000024.87.5, , ] 2022-09-12T19:05:27.974 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:05:27.969+0000 7ff6a3dbc600 0 bucket checkpoint complete 2022-09-12T19:05:27.994 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-19 2022-09-12T19:05:27.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:05:27.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:05:27.996 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:05:28.062 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:05:28.063 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:05:28.116 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:05:28.113+0000 7fa123169600 1 waiting for incremental sync to catch up: 2022-09-12T19:05:28.117 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000004.30.9, , , 00000000001.43.5, , ] 2022-09-12T19:05:28.117 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.30.9, , , 00000000042.142.8, , ] 2022-09-12T19:05:33.118 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:05:33.115+0000 7fa123169600 1 waiting for incremental sync to catch up: 2022-09-12T19:05:33.119 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000004.30.9, , , 00000000001.43.5, , ] 2022-09-12T19:05:33.119 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.30.9, , , 00000000042.142.8, , ] 2022-09-12T19:05:38.120 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:05:38.116+0000 7fa123169600 1 waiting for incremental sync to catch up: 2022-09-12T19:05:38.120 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000004.30.9, , , 00000000001.43.5, , ] 2022-09-12T19:05:38.120 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.30.9, , , 00000000042.142.8, , ] 2022-09-12T19:05:43.121 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:05:43.118+0000 7fa123169600 1 waiting for incremental sync to catch up: 2022-09-12T19:05:43.122 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000004.30.9, , , 00000000001.43.5, , ] 2022-09-12T19:05:43.122 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.30.9, , , 00000000042.142.8, , ] 2022-09-12T19:05:48.123 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:05:48.119+0000 7fa123169600 1 waiting for incremental sync to catch up: 2022-09-12T19:05:48.124 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000004.30.9, , , 00000000001.43.5, , ] 2022-09-12T19:05:48.124 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.30.9, , , 00000000042.142.8, , ] 2022-09-12T19:05:53.124 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:05:53.121+0000 7fa123169600 1 waiting for incremental sync to catch up: 2022-09-12T19:05:53.125 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000004.30.9, , , 00000000001.43.5, , ] 2022-09-12T19:05:53.125 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.30.9, , , 00000000042.142.8, , ] 2022-09-12T19:05:58.125 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:05:58.122+0000 7fa123169600 1 waiting for incremental sync to catch up: 2022-09-12T19:05:58.126 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000004.30.9, , , 00000000001.43.5, , ] 2022-09-12T19:05:58.127 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.30.9, , , 00000000042.142.8, , ] 2022-09-12T19:06:03.127 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:06:03.125+0000 7fa123169600 1 waiting for incremental sync to catch up: 2022-09-12T19:06:03.127 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000004.30.9, , , 00000000015.71.5, , ] 2022-09-12T19:06:03.128 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.30.9, , , 00000000042.142.8, , ] 2022-09-12T19:06:08.129 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:06:08.126+0000 7fa123169600 1 waiting for incremental sync to catch up: 2022-09-12T19:06:08.129 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000004.30.9, , , 00000000015.71.5, , ] 2022-09-12T19:06:08.130 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.30.9, , , 00000000042.142.8, , ] 2022-09-12T19:06:13.130 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:06:13.127+0000 7fa123169600 1 waiting for incremental sync to catch up: 2022-09-12T19:06:13.130 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000004.30.9, , , 00000000015.71.5, , ] 2022-09-12T19:06:13.131 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.30.9, , , 00000000042.142.8, , ] 2022-09-12T19:06:18.131 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:06:18.128+0000 7fa123169600 1 waiting for incremental sync to catch up: 2022-09-12T19:06:18.132 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000004.30.9, , , 00000000015.71.5, , ] 2022-09-12T19:06:18.132 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.30.9, , , 00000000042.142.8, , ] 2022-09-12T19:06:23.133 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:06:23.130+0000 7fa123169600 1 bucket sync caught up with source: 2022-09-12T19:06:23.133 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000004.30.9, , , 00000000044.146.9, , ] 2022-09-12T19:06:23.134 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.30.9, , , 00000000042.142.8, , ] 2022-09-12T19:06:23.134 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:06:23.130+0000 7fa123169600 0 bucket checkpoint complete 2022-09-12T19:06:23.150 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-19 2022-09-12T19:06:23.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:06:23.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:06:23.155 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:06:23.220 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:06:23.221 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:06:23.259 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:06:23.256+0000 7f1ebbf60600 1 bucket sync caught up with source: 2022-09-12T19:06:23.259 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000004.30.9, , , 00000000044.146.9, , ] 2022-09-12T19:06:23.260 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.30.9, , , 00000000044.146.9, , ] 2022-09-12T19:06:23.260 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:06:23.256+0000 7f1ebbf60600 0 bucket checkpoint complete 2022-09-12T19:06:23.279 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-19 2022-09-12T19:06:23.280 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-19 2022-09-12T19:06:23.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:06:23.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:06:23.282 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:06:23.340 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:06:23.341 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:06:23.396 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:06:23.393+0000 7f4bbacf1600 1 waiting for incremental sync to catch up: 2022-09-12T19:06:23.396 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000004.41.9, , , 00000000043.134.9, , ] 2022-09-12T19:06:23.397 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.41.9, , , 00000000044.136.9, , ] 2022-09-12T19:06:28.397 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:06:28.394+0000 7f4bbacf1600 1 waiting for incremental sync to catch up: 2022-09-12T19:06:28.398 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000004.41.9, , , 00000000043.134.9, , ] 2022-09-12T19:06:28.398 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.41.9, , , 00000000044.136.9, , ] 2022-09-12T19:06:33.399 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:06:33.396+0000 7f4bbacf1600 1 waiting for incremental sync to catch up: 2022-09-12T19:06:33.400 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000004.41.9, , , 00000000043.134.9, , ] 2022-09-12T19:06:33.401 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.41.9, , , 00000000044.136.9, , ] 2022-09-12T19:06:38.401 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:06:38.397+0000 7f4bbacf1600 1 waiting for incremental sync to catch up: 2022-09-12T19:06:38.401 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000004.41.9, , , 00000000043.134.9, , ] 2022-09-12T19:06:38.402 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.41.9, , , 00000000044.136.9, , ] 2022-09-12T19:06:43.402 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:06:43.399+0000 7f4bbacf1600 1 bucket sync caught up with source: 2022-09-12T19:06:43.403 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000004.41.9, , , 00000000044.136.9, , ] 2022-09-12T19:06:43.403 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000004.41.9, , , 00000000044.136.9, , ] 2022-09-12T19:06:43.403 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:06:43.399+0000 7f4bbacf1600 0 bucket checkpoint complete 2022-09-12T19:06:43.419 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-19 2022-09-12T19:06:43.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:06:43.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:06:43.422 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:06:43.487 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:06:43.487 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:06:43.526 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:06:43.524+0000 7efef863c600 1 bucket sync caught up with source: 2022-09-12T19:06:43.527 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000004.41.9, , , 00000000044.136.9, , ] 2022-09-12T19:06:43.527 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000004.41.9, , , 00000000044.136.9, , ] 2022-09-12T19:06:43.527 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:06:43.524+0000 7efef863c600 0 bucket checkpoint complete 2022-09-12T19:06:43.539 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-19 2022-09-12T19:06:43.540 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-19 zones={test-zone2, test-zone1} 2022-09-12T19:06:43.541 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:43.541 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:43.542 DEBUG:boto:Method: HEAD 2022-09-12T19:06:43.543 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:06:43.544 DEBUG:boto:Data: 2022-09-12T19:06:43.544 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.545 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:43.545 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.546 DEBUG:boto:Params: {} 2022-09-12T19:06:43.547 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:06:43.547 DEBUG:boto:Token: None 2022-09-12T19:06:43.548 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/ 2022-09-12T19:06:43.549 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1CLNnv9rxTMHQz4+1EvJIniWKN4= 2022-09-12T19:06:43.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1CLNnv9rxTMHQz4+1EvJIniWKN4=', 'Content-Length': '0'} 2022-09-12T19:06:43.554 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000231ea01952c7b98b-00631f8343-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.555 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:43.555 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:43.556 DEBUG:boto:Method: HEAD 2022-09-12T19:06:43.556 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:06:43.557 DEBUG:boto:Data: 2022-09-12T19:06:43.558 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.558 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:43.559 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.559 DEBUG:boto:Params: {} 2022-09-12T19:06:43.560 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:06:43.561 DEBUG:boto:Token: None 2022-09-12T19:06:43.561 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/ 2022-09-12T19:06:43.562 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1CLNnv9rxTMHQz4+1EvJIniWKN4= 2022-09-12T19:06:43.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1CLNnv9rxTMHQz4+1EvJIniWKN4=', 'Content-Length': '0'} 2022-09-12T19:06:43.566 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000057e5c7cd7dab4aa8-00631f8343-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.567 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:06:43.568 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:43.568 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:43.569 DEBUG:boto:path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:43.569 DEBUG:boto:auth_path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:43.570 DEBUG:boto:Method: GET 2022-09-12T19:06:43.571 DEBUG:boto:Path: /jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:43.572 DEBUG:boto:Data: 2022-09-12T19:06:43.573 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.574 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:43.575 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.576 DEBUG:boto:Params: {} 2022-09-12T19:06:43.577 DEBUG:boto:Token: None 2022-09-12T19:06:43.578 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/?versions 2022-09-12T19:06:43.579 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ftxaGXMgAow/3mK/RjPr7nOL3BY= 2022-09-12T19:06:43.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ftxaGXMgAow/3mK/RjPr7nOL3BY=', 'Content-Length': '0'} 2022-09-12T19:06:43.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afce40e4dcd6b2b7-00631f8343-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.584 DEBUG:boto:b'jdpiak-19999falsedummydAjTD8IRanrz7e2F.KgapZJi58DocMetrue2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZqtrue2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ulfalse2022-09-12T19:05:07.485Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTM67hyaYWRVq578zUsjsXjOzkUYTIvAfalse2022-09-12T19:05:07.581Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rhfalse2022-09-12T19:05:07.457Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpwncGyI2j8AeHH4hjzwsep2DWsp-l-Afalse2022-09-12T19:05:07.383Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsGTjDAFqZF1WL9AP98wlNVRWMwOQwYQfalse2022-09-12T19:05:07.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT.yM--FII-PB1GReM85mkNRGYz6Dkbzfalse2022-09-12T19:05:07.238Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXb5UZAvutXF6GhBCMdUQgjOLi998.hKfalse2022-09-12T19:05:07.343Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjztQRphsNbUdApzTcOcagUJ56uTwo7J2false2022-09-12T19:05:07.274Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKatr07t9mQV0b0gx44aB8d9VyeAhMS1false2022-09-12T19:05:07.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUBsYcCA94H55K4ucww5vXNAxuAMKNSGfalse2022-09-12T19:05:07.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkxSkXmfXKy0sdqagFrYSQNi7tMOKBoWfalse2022-09-12T19:05:07.199Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0false2022-09-12T19:05:06.995Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK.LUooPdPnXZMzfYIxw-xagQpErk4CXfalse2022-09-12T19:05:07.075Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtQHk21B862H4VHQxrZWEog8n-qgO99Qfalse2022-09-12T19:05:07.038Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQHXOu7SGV87vItdBeqaL.hYUlX--Ty8false2022-09-12T19:05:06.952Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT-rka2v4NcjXyfB8GzT0u7KHTz6OcF3false2022-09-12T19:05:06.866Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgdUGnE7SHabmaVwE2tf-YcFxl2tBu7Hfalse2022-09-12T19:05:06.908Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBAbppx4RylEKnBrYesXt1QPJgsAhIBbfalse2022-09-12T19:05:06.803Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNDM-EGOTc7UZbg9YQZwcM2uEk6P03H2false2022-09-12T19:05:06.773Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje.6ruI53roqy4mmyPgzQ1kqMdajruFAfalse2022-09-12T19:05:06.834Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLIaoO70PR8fRbTLQ-5Q.roWjA0lswqEfalse2022-09-12T19:05:06.733Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlnpiHoSs91QVujLSWqHhvEGzZrF3-sAfalse2022-09-12T19:05:06.698Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp5OIjHSzwvUDkIBYzRPdKBGezjIkrn-false2022-09-12T19:05:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjV.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9false2022-09-12T19:05:06.565Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiJi00oqpo4jP5T2qIUe4H2lLKJSYXZdfalse2022-09-12T19:05:06.598Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjj0eE53eOEuIue0LhYTVuZStUTmcOj7ofalse2022-09-12T19:05:06.634Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjACPdCUDfm32QLq6XbWYcqHiCLcEU2JYfalse2022-09-12T19:05:06.489Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJcC9N13lklBVOx234Q8MXQ8kAM8ByWgfalse2022-09-12T19:05:06.383Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjipYOl-Gqt65Y3nx.nVL2kL2DrysHNSYfalse2022-09-12T19:05:06.415Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:43.587 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:06:43.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.593 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.593 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.594 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.594 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.596 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.596 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.603 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:06:43.603 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:43.604 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:43.604 DEBUG:boto:path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:43.605 DEBUG:boto:auth_path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:43.605 DEBUG:boto:Method: GET 2022-09-12T19:06:43.606 DEBUG:boto:Path: /jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:43.607 DEBUG:boto:Data: 2022-09-12T19:06:43.607 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.608 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:43.608 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.609 DEBUG:boto:Params: {} 2022-09-12T19:06:43.609 DEBUG:boto:Token: None 2022-09-12T19:06:43.610 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/?versions 2022-09-12T19:06:43.611 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ftxaGXMgAow/3mK/RjPr7nOL3BY= 2022-09-12T19:06:43.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ftxaGXMgAow/3mK/RjPr7nOL3BY=', 'Content-Length': '0'} 2022-09-12T19:06:43.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e18078626cc0abc4-00631f8343-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.615 DEBUG:boto:b'jdpiak-19999falsedummydAjTD8IRanrz7e2F.KgapZJi58DocMetrue2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZqtrue2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ulfalse2022-09-12T19:05:07.485Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTM67hyaYWRVq578zUsjsXjOzkUYTIvAfalse2022-09-12T19:05:07.581Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rhfalse2022-09-12T19:05:07.457Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpwncGyI2j8AeHH4hjzwsep2DWsp-l-Afalse2022-09-12T19:05:07.383Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsGTjDAFqZF1WL9AP98wlNVRWMwOQwYQfalse2022-09-12T19:05:07.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT.yM--FII-PB1GReM85mkNRGYz6Dkbzfalse2022-09-12T19:05:07.238Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXb5UZAvutXF6GhBCMdUQgjOLi998.hKfalse2022-09-12T19:05:07.343Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjztQRphsNbUdApzTcOcagUJ56uTwo7J2false2022-09-12T19:05:07.274Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKatr07t9mQV0b0gx44aB8d9VyeAhMS1false2022-09-12T19:05:07.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUBsYcCA94H55K4ucww5vXNAxuAMKNSGfalse2022-09-12T19:05:07.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkxSkXmfXKy0sdqagFrYSQNi7tMOKBoWfalse2022-09-12T19:05:07.199Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0false2022-09-12T19:05:06.995Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK.LUooPdPnXZMzfYIxw-xagQpErk4CXfalse2022-09-12T19:05:07.075Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtQHk21B862H4VHQxrZWEog8n-qgO99Qfalse2022-09-12T19:05:07.038Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQHXOu7SGV87vItdBeqaL.hYUlX--Ty8false2022-09-12T19:05:06.952Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT-rka2v4NcjXyfB8GzT0u7KHTz6OcF3false2022-09-12T19:05:06.866Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgdUGnE7SHabmaVwE2tf-YcFxl2tBu7Hfalse2022-09-12T19:05:06.908Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBAbppx4RylEKnBrYesXt1QPJgsAhIBbfalse2022-09-12T19:05:06.803Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNDM-EGOTc7UZbg9YQZwcM2uEk6P03H2false2022-09-12T19:05:06.773Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje.6ruI53roqy4mmyPgzQ1kqMdajruFAfalse2022-09-12T19:05:06.834Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLIaoO70PR8fRbTLQ-5Q.roWjA0lswqEfalse2022-09-12T19:05:06.733Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlnpiHoSs91QVujLSWqHhvEGzZrF3-sAfalse2022-09-12T19:05:06.698Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp5OIjHSzwvUDkIBYzRPdKBGezjIkrn-false2022-09-12T19:05:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjV.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9false2022-09-12T19:05:06.565Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiJi00oqpo4jP5T2qIUe4H2lLKJSYXZdfalse2022-09-12T19:05:06.598Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjj0eE53eOEuIue0LhYTVuZStUTmcOj7ofalse2022-09-12T19:05:06.634Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjACPdCUDfm32QLq6XbWYcqHiCLcEU2JYfalse2022-09-12T19:05:06.489Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJcC9N13lklBVOx234Q8MXQ8kAM8ByWgfalse2022-09-12T19:05:06.383Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjipYOl-Gqt65Y3nx.nVL2kL2DrysHNSYfalse2022-09-12T19:05:06.415Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:43.618 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:06:43.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.620 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.620 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.620 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.621 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.621 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.624 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.624 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.625 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.625 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.628 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.628 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.628 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.631 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.631 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:43.632 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:43.633 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:43.633 DEBUG:boto:path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:43.634 DEBUG:boto:auth_path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:43.634 DEBUG:boto:Method: GET 2022-09-12T19:06:43.635 DEBUG:boto:Path: /jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:43.635 DEBUG:boto:Data: 2022-09-12T19:06:43.636 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.636 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:43.637 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.637 DEBUG:boto:Params: {} 2022-09-12T19:06:43.638 DEBUG:boto:Token: None 2022-09-12T19:06:43.639 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/?versions 2022-09-12T19:06:43.639 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ftxaGXMgAow/3mK/RjPr7nOL3BY= 2022-09-12T19:06:43.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ftxaGXMgAow/3mK/RjPr7nOL3BY=', 'Content-Length': '0'} 2022-09-12T19:06:43.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e7c3b348235ada7-00631f8343-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.644 DEBUG:boto:b'jdpiak-19999falsedummydAjTD8IRanrz7e2F.KgapZJi58DocMetrue2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZqtrue2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ulfalse2022-09-12T19:05:07.485Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTM67hyaYWRVq578zUsjsXjOzkUYTIvAfalse2022-09-12T19:05:07.581Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rhfalse2022-09-12T19:05:07.457Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpwncGyI2j8AeHH4hjzwsep2DWsp-l-Afalse2022-09-12T19:05:07.383Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsGTjDAFqZF1WL9AP98wlNVRWMwOQwYQfalse2022-09-12T19:05:07.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT.yM--FII-PB1GReM85mkNRGYz6Dkbzfalse2022-09-12T19:05:07.238Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXb5UZAvutXF6GhBCMdUQgjOLi998.hKfalse2022-09-12T19:05:07.343Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjztQRphsNbUdApzTcOcagUJ56uTwo7J2false2022-09-12T19:05:07.274Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKatr07t9mQV0b0gx44aB8d9VyeAhMS1false2022-09-12T19:05:07.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUBsYcCA94H55K4ucww5vXNAxuAMKNSGfalse2022-09-12T19:05:07.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkxSkXmfXKy0sdqagFrYSQNi7tMOKBoWfalse2022-09-12T19:05:07.199Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0false2022-09-12T19:05:06.995Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK.LUooPdPnXZMzfYIxw-xagQpErk4CXfalse2022-09-12T19:05:07.075Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtQHk21B862H4VHQxrZWEog8n-qgO99Qfalse2022-09-12T19:05:07.038Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQHXOu7SGV87vItdBeqaL.hYUlX--Ty8false2022-09-12T19:05:06.952Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT-rka2v4NcjXyfB8GzT0u7KHTz6OcF3false2022-09-12T19:05:06.866Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgdUGnE7SHabmaVwE2tf-YcFxl2tBu7Hfalse2022-09-12T19:05:06.908Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBAbppx4RylEKnBrYesXt1QPJgsAhIBbfalse2022-09-12T19:05:06.803Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNDM-EGOTc7UZbg9YQZwcM2uEk6P03H2false2022-09-12T19:05:06.773Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje.6ruI53roqy4mmyPgzQ1kqMdajruFAfalse2022-09-12T19:05:06.834Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLIaoO70PR8fRbTLQ-5Q.roWjA0lswqEfalse2022-09-12T19:05:06.733Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlnpiHoSs91QVujLSWqHhvEGzZrF3-sAfalse2022-09-12T19:05:06.698Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp5OIjHSzwvUDkIBYzRPdKBGezjIkrn-false2022-09-12T19:05:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjV.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9false2022-09-12T19:05:06.565Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiJi00oqpo4jP5T2qIUe4H2lLKJSYXZdfalse2022-09-12T19:05:06.598Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjj0eE53eOEuIue0LhYTVuZStUTmcOj7ofalse2022-09-12T19:05:06.634Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjACPdCUDfm32QLq6XbWYcqHiCLcEU2JYfalse2022-09-12T19:05:06.489Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJcC9N13lklBVOx234Q8MXQ8kAM8ByWgfalse2022-09-12T19:05:06.383Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjipYOl-Gqt65Y3nx.nVL2kL2DrysHNSYfalse2022-09-12T19:05:06.415Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:43.646 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:43.647 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:43.647 DEBUG:boto:path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:43.648 DEBUG:boto:auth_path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:43.648 DEBUG:boto:Method: GET 2022-09-12T19:06:43.649 DEBUG:boto:Path: /jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:43.649 DEBUG:boto:Data: 2022-09-12T19:06:43.650 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.650 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:43.651 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.651 DEBUG:boto:Params: {} 2022-09-12T19:06:43.652 DEBUG:boto:Token: None 2022-09-12T19:06:43.652 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/?versions 2022-09-12T19:06:43.653 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ftxaGXMgAow/3mK/RjPr7nOL3BY= 2022-09-12T19:06:43.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ftxaGXMgAow/3mK/RjPr7nOL3BY=', 'Content-Length': '0'} 2022-09-12T19:06:43.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074b1eeefc8b5a151-00631f8343-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.658 DEBUG:boto:b'jdpiak-19999falsedummydAjTD8IRanrz7e2F.KgapZJi58DocMetrue2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZqtrue2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ulfalse2022-09-12T19:05:07.485Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTM67hyaYWRVq578zUsjsXjOzkUYTIvAfalse2022-09-12T19:05:07.581Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rhfalse2022-09-12T19:05:07.457Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpwncGyI2j8AeHH4hjzwsep2DWsp-l-Afalse2022-09-12T19:05:07.383Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsGTjDAFqZF1WL9AP98wlNVRWMwOQwYQfalse2022-09-12T19:05:07.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT.yM--FII-PB1GReM85mkNRGYz6Dkbzfalse2022-09-12T19:05:07.238Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXb5UZAvutXF6GhBCMdUQgjOLi998.hKfalse2022-09-12T19:05:07.343Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjztQRphsNbUdApzTcOcagUJ56uTwo7J2false2022-09-12T19:05:07.274Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKatr07t9mQV0b0gx44aB8d9VyeAhMS1false2022-09-12T19:05:07.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUBsYcCA94H55K4ucww5vXNAxuAMKNSGfalse2022-09-12T19:05:07.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkxSkXmfXKy0sdqagFrYSQNi7tMOKBoWfalse2022-09-12T19:05:07.199Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0false2022-09-12T19:05:06.995Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK.LUooPdPnXZMzfYIxw-xagQpErk4CXfalse2022-09-12T19:05:07.075Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtQHk21B862H4VHQxrZWEog8n-qgO99Qfalse2022-09-12T19:05:07.038Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQHXOu7SGV87vItdBeqaL.hYUlX--Ty8false2022-09-12T19:05:06.952Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT-rka2v4NcjXyfB8GzT0u7KHTz6OcF3false2022-09-12T19:05:06.866Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgdUGnE7SHabmaVwE2tf-YcFxl2tBu7Hfalse2022-09-12T19:05:06.908Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBAbppx4RylEKnBrYesXt1QPJgsAhIBbfalse2022-09-12T19:05:06.803Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNDM-EGOTc7UZbg9YQZwcM2uEk6P03H2false2022-09-12T19:05:06.773Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje.6ruI53roqy4mmyPgzQ1kqMdajruFAfalse2022-09-12T19:05:06.834Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLIaoO70PR8fRbTLQ-5Q.roWjA0lswqEfalse2022-09-12T19:05:06.733Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlnpiHoSs91QVujLSWqHhvEGzZrF3-sAfalse2022-09-12T19:05:06.698Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp5OIjHSzwvUDkIBYzRPdKBGezjIkrn-false2022-09-12T19:05:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjV.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9false2022-09-12T19:05:06.565Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiJi00oqpo4jP5T2qIUe4H2lLKJSYXZdfalse2022-09-12T19:05:06.598Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjj0eE53eOEuIue0LhYTVuZStUTmcOj7ofalse2022-09-12T19:05:06.634Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjACPdCUDfm32QLq6XbWYcqHiCLcEU2JYfalse2022-09-12T19:05:06.489Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJcC9N13lklBVOx234Q8MXQ8kAM8ByWgfalse2022-09-12T19:05:06.383Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjipYOl-Gqt65Y3nx.nVL2kL2DrysHNSYfalse2022-09-12T19:05:06.415Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:43.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:06:43.661 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:43.662 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:43.662 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.663 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.663 DEBUG:boto:Method: GET 2022-09-12T19:06:43.664 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.665 DEBUG:boto:Data: 2022-09-12T19:06:43.665 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.666 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:43.666 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.667 DEBUG:boto:Params: {} 2022-09-12T19:06:43.667 DEBUG:boto:Token: None 2022-09-12T19:06:43.668 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.669 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4UCbxyHvX+SLIEKx9b1HEkh7q2Q= 2022-09-12T19:06:43.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4UCbxyHvX+SLIEKx9b1HEkh7q2Q=', 'Content-Length': '0'} 2022-09-12T19:06:43.679 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fedda875194aaa07-00631f8343-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.680 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:43.681 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:43.682 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.683 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.684 DEBUG:boto:Method: GET 2022-09-12T19:06:43.685 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.686 DEBUG:boto:Data: 2022-09-12T19:06:43.687 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.688 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:43.689 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.690 DEBUG:boto:Params: {} 2022-09-12T19:06:43.691 DEBUG:boto:Token: None 2022-09-12T19:06:43.692 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.693 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4UCbxyHvX+SLIEKx9b1HEkh7q2Q= 2022-09-12T19:06:43.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4UCbxyHvX+SLIEKx9b1HEkh7q2Q=', 'Content-Length': '0'} 2022-09-12T19:06:43.697 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000052c80fd7cdf177f2-00631f8343-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.699 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:43.700 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:43.701 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.702 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.703 DEBUG:boto:Method: HEAD 2022-09-12T19:06:43.704 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.705 DEBUG:boto:Data: 2022-09-12T19:06:43.706 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.707 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:43.708 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.710 DEBUG:boto:Params: {} 2022-09-12T19:06:43.711 DEBUG:boto:Token: None 2022-09-12T19:06:43.712 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.712 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+7w4kUqI8/s0OV4uHtv4R/UBFVA= 2022-09-12T19:06:43.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+7w4kUqI8/s0OV4uHtv4R/UBFVA=', 'Content-Length': '0'} 2022-09-12T19:06:43.715 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001b8b9e4c685d4584-00631f8343-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.716 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:43.716 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:43.718 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.719 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.720 DEBUG:boto:Method: HEAD 2022-09-12T19:06:43.720 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.721 DEBUG:boto:Data: 2022-09-12T19:06:43.721 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.722 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:43.722 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.723 DEBUG:boto:Params: {} 2022-09-12T19:06:43.724 DEBUG:boto:Token: None 2022-09-12T19:06:43.724 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.725 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+7w4kUqI8/s0OV4uHtv4R/UBFVA= 2022-09-12T19:06:43.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+7w4kUqI8/s0OV4uHtv4R/UBFVA=', 'Content-Length': '0'} 2022-09-12T19:06:43.728 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006375f6541d27de8b-00631f8343-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:06:43.729 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:43.730 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:43.730 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.731 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.732 DEBUG:boto:Method: GET 2022-09-12T19:06:43.732 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.733 DEBUG:boto:Data: 2022-09-12T19:06:43.733 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.734 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:43.735 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.735 DEBUG:boto:Params: {} 2022-09-12T19:06:43.736 DEBUG:boto:Token: None 2022-09-12T19:06:43.736 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.737 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4UCbxyHvX+SLIEKx9b1HEkh7q2Q= 2022-09-12T19:06:43.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4UCbxyHvX+SLIEKx9b1HEkh7q2Q=', 'Content-Length': '0'} 2022-09-12T19:06:43.741 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000057fe40e88ab71d8f-00631f8343-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.742 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:43.743 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:43.744 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.744 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.745 DEBUG:boto:Method: GET 2022-09-12T19:06:43.746 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.747 DEBUG:boto:Data: 2022-09-12T19:06:43.748 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.749 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:43.749 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.750 DEBUG:boto:Params: {} 2022-09-12T19:06:43.751 DEBUG:boto:Token: None 2022-09-12T19:06:43.753 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.754 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4UCbxyHvX+SLIEKx9b1HEkh7q2Q= 2022-09-12T19:06:43.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4UCbxyHvX+SLIEKx9b1HEkh7q2Q=', 'Content-Length': '0'} 2022-09-12T19:06:43.758 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ba7d0b9f651ac715-00631f8343-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.759 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:43.760 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:43.761 DEBUG:boto:path=/jdpiak-19/?prefix=dummy 2022-09-12T19:06:43.761 DEBUG:boto:auth_path=/jdpiak-19/?prefix=dummy 2022-09-12T19:06:43.762 DEBUG:boto:Method: GET 2022-09-12T19:06:43.762 DEBUG:boto:Path: /jdpiak-19/?prefix=dummy 2022-09-12T19:06:43.763 DEBUG:boto:Data: 2022-09-12T19:06:43.764 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.764 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:43.765 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.765 DEBUG:boto:Params: {} 2022-09-12T19:06:43.766 DEBUG:boto:Token: None 2022-09-12T19:06:43.767 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/ 2022-09-12T19:06:43.767 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W0IZ3MdZPz7Ih8015EMWM+Nu5o0= 2022-09-12T19:06:43.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W0IZ3MdZPz7Ih8015EMWM+Nu5o0=', 'Content-Length': '0'} 2022-09-12T19:06:43.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008af792c7dbaf1fe6-00631f8343-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.773 DEBUG:boto:b'jdpiak-19dummy1000falsedummy2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:43.774 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:43.775 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:43.776 DEBUG:boto:path=/jdpiak-19/?prefix=dummy 2022-09-12T19:06:43.777 DEBUG:boto:auth_path=/jdpiak-19/?prefix=dummy 2022-09-12T19:06:43.777 DEBUG:boto:Method: GET 2022-09-12T19:06:43.779 DEBUG:boto:Path: /jdpiak-19/?prefix=dummy 2022-09-12T19:06:43.780 DEBUG:boto:Data: 2022-09-12T19:06:43.781 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.782 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:43.784 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.785 DEBUG:boto:Params: {} 2022-09-12T19:06:43.785 DEBUG:boto:Token: None 2022-09-12T19:06:43.786 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/ 2022-09-12T19:06:43.787 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W0IZ3MdZPz7Ih8015EMWM+Nu5o0= 2022-09-12T19:06:43.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W0IZ3MdZPz7Ih8015EMWM+Nu5o0=', 'Content-Length': '0'} 2022-09-12T19:06:43.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efc78b53ebfe0a5e-00631f8343-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.791 DEBUG:boto:b'jdpiak-19dummy1000falsedummy2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:43.792 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:43.793 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:43.793 DEBUG:boto:Method: HEAD 2022-09-12T19:06:43.794 DEBUG:boto:Path: /jdpiak-19/dummy 2022-09-12T19:06:43.795 DEBUG:boto:Data: 2022-09-12T19:06:43.795 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.796 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:43.796 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.797 DEBUG:boto:Params: {} 2022-09-12T19:06:43.798 DEBUG:boto:Token: None 2022-09-12T19:06:43.798 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/dummy 2022-09-12T19:06:43.799 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rA3BqPxGXTXGoN3OEWX03xJgXmw= 2022-09-12T19:06:43.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rA3BqPxGXTXGoN3OEWX03xJgXmw=', 'Content-Length': '0'} 2022-09-12T19:06:43.802 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007432085f30c10989-00631f8343-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.803 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:43.803 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:43.804 DEBUG:boto:Method: HEAD 2022-09-12T19:06:43.804 DEBUG:boto:Path: /jdpiak-19/dummy 2022-09-12T19:06:43.805 DEBUG:boto:Data: 2022-09-12T19:06:43.806 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.807 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:43.808 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.809 DEBUG:boto:Params: {} 2022-09-12T19:06:43.810 DEBUG:boto:Token: None 2022-09-12T19:06:43.811 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/dummy 2022-09-12T19:06:43.811 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rA3BqPxGXTXGoN3OEWX03xJgXmw= 2022-09-12T19:06:43.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rA3BqPxGXTXGoN3OEWX03xJgXmw=', 'Content-Length': '0'} 2022-09-12T19:06:43.815 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fa9329d23cf4a182-00631f8343-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:06:43.816 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:43.817 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:43.817 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.818 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.819 DEBUG:boto:Method: GET 2022-09-12T19:06:43.820 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.820 DEBUG:boto:Data: 2022-09-12T19:06:43.821 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.821 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:43.823 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.824 DEBUG:boto:Params: {} 2022-09-12T19:06:43.824 DEBUG:boto:Token: None 2022-09-12T19:06:43.825 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.826 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4UCbxyHvX+SLIEKx9b1HEkh7q2Q= 2022-09-12T19:06:43.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4UCbxyHvX+SLIEKx9b1HEkh7q2Q=', 'Content-Length': '0'} 2022-09-12T19:06:43.828 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000007827d0c71b5d477-00631f8343-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.829 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:43.830 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:43.830 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.831 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.831 DEBUG:boto:Method: GET 2022-09-12T19:06:43.832 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.832 DEBUG:boto:Data: 2022-09-12T19:06:43.833 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.833 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:43.834 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.834 DEBUG:boto:Params: {} 2022-09-12T19:06:43.835 DEBUG:boto:Token: None 2022-09-12T19:06:43.836 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:43.836 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4UCbxyHvX+SLIEKx9b1HEkh7q2Q= 2022-09-12T19:06:43.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4UCbxyHvX+SLIEKx9b1HEkh7q2Q=', 'Content-Length': '0'} 2022-09-12T19:06:43.839 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003621a7f2deaca010-00631f8343-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.840 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:43.840 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:43.841 DEBUG:boto:path=/jdpiak-19/?prefix=dummy 2022-09-12T19:06:43.842 DEBUG:boto:auth_path=/jdpiak-19/?prefix=dummy 2022-09-12T19:06:43.842 DEBUG:boto:Method: GET 2022-09-12T19:06:43.843 DEBUG:boto:Path: /jdpiak-19/?prefix=dummy 2022-09-12T19:06:43.843 DEBUG:boto:Data: 2022-09-12T19:06:43.844 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.844 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:43.845 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.845 DEBUG:boto:Params: {} 2022-09-12T19:06:43.846 DEBUG:boto:Token: None 2022-09-12T19:06:43.846 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/ 2022-09-12T19:06:43.847 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W0IZ3MdZPz7Ih8015EMWM+Nu5o0= 2022-09-12T19:06:43.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W0IZ3MdZPz7Ih8015EMWM+Nu5o0=', 'Content-Length': '0'} 2022-09-12T19:06:43.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092e88f5478940e28-00631f8343-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.894 DEBUG:boto:b'jdpiak-19dummy1000falsedummy2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:43.895 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:43.896 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:43.896 DEBUG:boto:path=/jdpiak-19/?prefix=dummy 2022-09-12T19:06:43.897 DEBUG:boto:auth_path=/jdpiak-19/?prefix=dummy 2022-09-12T19:06:43.898 DEBUG:boto:Method: GET 2022-09-12T19:06:43.898 DEBUG:boto:Path: /jdpiak-19/?prefix=dummy 2022-09-12T19:06:43.899 DEBUG:boto:Data: 2022-09-12T19:06:43.899 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.900 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:43.900 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.901 DEBUG:boto:Params: {} 2022-09-12T19:06:43.902 DEBUG:boto:Token: None 2022-09-12T19:06:43.907 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/ 2022-09-12T19:06:43.908 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W0IZ3MdZPz7Ih8015EMWM+Nu5o0= 2022-09-12T19:06:43.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W0IZ3MdZPz7Ih8015EMWM+Nu5o0=', 'Content-Length': '0'} 2022-09-12T19:06:43.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbf64fca200b2226-00631f8343-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.912 DEBUG:boto:b'jdpiak-19dummy1000falsedummy2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:43.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:43.913 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:43.914 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:43.915 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:43.915 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:43.916 DEBUG:boto:Method: GET 2022-09-12T19:06:43.917 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:43.917 DEBUG:boto:Data: 2022-09-12T19:06:43.918 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.918 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:43.919 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.919 DEBUG:boto:Params: {} 2022-09-12T19:06:43.920 DEBUG:boto:Token: None 2022-09-12T19:06:43.921 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:43.921 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XODOTiVp/zVNqEVHWSYRcOcH5tg= 2022-09-12T19:06:43.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XODOTiVp/zVNqEVHWSYRcOcH5tg=', 'Content-Length': '0'} 2022-09-12T19:06:43.924 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095c615c528687f1d-00631f8343-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.925 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:43.926 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:43.926 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:43.927 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:43.927 DEBUG:boto:Method: GET 2022-09-12T19:06:43.928 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:43.928 DEBUG:boto:Data: 2022-09-12T19:06:43.929 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.930 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:43.930 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.931 DEBUG:boto:Params: {} 2022-09-12T19:06:43.931 DEBUG:boto:Token: None 2022-09-12T19:06:43.932 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:43.932 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XODOTiVp/zVNqEVHWSYRcOcH5tg= 2022-09-12T19:06:43.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XODOTiVp/zVNqEVHWSYRcOcH5tg=', 'Content-Length': '0'} 2022-09-12T19:06:43.935 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c194f6db2f6aeae-00631f8343-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.936 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:43.937 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:43.937 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:43.938 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:43.938 DEBUG:boto:Method: HEAD 2022-09-12T19:06:43.939 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:43.940 DEBUG:boto:Data: 2022-09-12T19:06:43.940 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.941 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:43.941 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.942 DEBUG:boto:Params: {} 2022-09-12T19:06:43.942 DEBUG:boto:Token: None 2022-09-12T19:06:43.943 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:43.943 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/tbjJSsB4OtCPV2YI6gmMbN0ujE= 2022-09-12T19:06:43.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/tbjJSsB4OtCPV2YI6gmMbN0ujE=', 'Content-Length': '0'} 2022-09-12T19:06:43.946 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1b4aa60d16ba682-00631f8343-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.947 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:43.947 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:43.948 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:43.949 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:43.949 DEBUG:boto:Method: HEAD 2022-09-12T19:06:43.950 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:43.950 DEBUG:boto:Data: 2022-09-12T19:06:43.951 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.951 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:43.952 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.953 DEBUG:boto:Params: {} 2022-09-12T19:06:43.953 DEBUG:boto:Token: None 2022-09-12T19:06:43.954 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:43.955 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/tbjJSsB4OtCPV2YI6gmMbN0ujE= 2022-09-12T19:06:43.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/tbjJSsB4OtCPV2YI6gmMbN0ujE=', 'Content-Length': '0'} 2022-09-12T19:06:43.957 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bdcf557776adaaed-00631f8343-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:43.959 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:43.960 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:43.960 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:43.961 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:43.961 DEBUG:boto:Method: GET 2022-09-12T19:06:43.962 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:43.962 DEBUG:boto:Data: 2022-09-12T19:06:43.963 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.964 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:43.964 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.965 DEBUG:boto:Params: {} 2022-09-12T19:06:43.965 DEBUG:boto:Token: None 2022-09-12T19:06:43.966 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:43.967 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XODOTiVp/zVNqEVHWSYRcOcH5tg= 2022-09-12T19:06:43.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XODOTiVp/zVNqEVHWSYRcOcH5tg=', 'Content-Length': '0'} 2022-09-12T19:06:43.969 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000453abe9492826208-00631f8343-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.970 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:43.971 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:43.971 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:43.972 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:43.972 DEBUG:boto:Method: GET 2022-09-12T19:06:43.973 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:43.973 DEBUG:boto:Data: 2022-09-12T19:06:43.974 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.975 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:43.975 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.976 DEBUG:boto:Params: {} 2022-09-12T19:06:43.976 DEBUG:boto:Token: None 2022-09-12T19:06:43.977 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:43.978 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XODOTiVp/zVNqEVHWSYRcOcH5tg= 2022-09-12T19:06:43.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XODOTiVp/zVNqEVHWSYRcOcH5tg=', 'Content-Length': '0'} 2022-09-12T19:06:43.981 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d799842be12b970-00631f8343-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.982 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:43.983 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:43.983 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:43.984 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:43.984 DEBUG:boto:Method: GET 2022-09-12T19:06:43.985 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:43.985 DEBUG:boto:Data: 2022-09-12T19:06:43.986 DEBUG:boto:Headers: {} 2022-09-12T19:06:43.987 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:43.987 DEBUG:boto:Port: 8000 2022-09-12T19:06:43.988 DEBUG:boto:Params: {} 2022-09-12T19:06:43.988 DEBUG:boto:Token: None 2022-09-12T19:06:43.989 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:43 GMT /jdpiak-19/ 2022-09-12T19:06:43.990 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W0IZ3MdZPz7Ih8015EMWM+Nu5o0= 2022-09-12T19:06:43.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W0IZ3MdZPz7Ih8015EMWM+Nu5o0=', 'Content-Length': '0'} 2022-09-12T19:06:43.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000496e6058b539472-00631f8343-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:43.993 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:43.994 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:44.005 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:44.006 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.007 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.007 DEBUG:boto:Method: GET 2022-09-12T19:06:44.008 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:44.008 DEBUG:boto:Data: 2022-09-12T19:06:44.009 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.009 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.010 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.011 DEBUG:boto:Params: {} 2022-09-12T19:06:44.011 DEBUG:boto:Token: None 2022-09-12T19:06:44.012 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/ 2022-09-12T19:06:44.013 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs= 2022-09-12T19:06:44.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs=', 'Content-Length': '0'} 2022-09-12T19:06:44.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7964b783a13e7f5-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.017 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:44.018 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.019 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.019 DEBUG:boto:Method: HEAD 2022-09-12T19:06:44.020 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:44.021 DEBUG:boto:Data: 2022-09-12T19:06:44.021 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.022 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.022 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.023 DEBUG:boto:Params: {} 2022-09-12T19:06:44.023 DEBUG:boto:Token: None 2022-09-12T19:06:44.024 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj 2022-09-12T19:06:44.024 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:neb+1Waat1etesg0gHGUltolAyk= 2022-09-12T19:06:44.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:neb+1Waat1etesg0gHGUltolAyk=', 'Content-Length': '0'} 2022-09-12T19:06:44.028 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000245ce91d0e25e1fa-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.029 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.030 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.030 DEBUG:boto:Method: HEAD 2022-09-12T19:06:44.031 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:44.031 DEBUG:boto:Data: 2022-09-12T19:06:44.032 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.032 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.033 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.033 DEBUG:boto:Params: {} 2022-09-12T19:06:44.034 DEBUG:boto:Token: None 2022-09-12T19:06:44.035 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj 2022-09-12T19:06:44.035 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:neb+1Waat1etesg0gHGUltolAyk= 2022-09-12T19:06:44.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:neb+1Waat1etesg0gHGUltolAyk=', 'Content-Length': '0'} 2022-09-12T19:06:44.039 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f934c5720a54a3e-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:44.040 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.041 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.042 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.042 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.043 DEBUG:boto:Method: GET 2022-09-12T19:06:44.043 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.044 DEBUG:boto:Data: 2022-09-12T19:06:44.044 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.045 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.045 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.046 DEBUG:boto:Params: {} 2022-09-12T19:06:44.046 DEBUG:boto:Token: None 2022-09-12T19:06:44.047 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.048 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e6dSi2GkU1Qc9ZWUdks0grQQPec= 2022-09-12T19:06:44.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e6dSi2GkU1Qc9ZWUdks0grQQPec=', 'Content-Length': '0'} 2022-09-12T19:06:44.050 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007164488b981e1760-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.094 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.095 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.096 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.096 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.097 DEBUG:boto:Method: GET 2022-09-12T19:06:44.097 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.098 DEBUG:boto:Data: 2022-09-12T19:06:44.099 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.099 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.100 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.100 DEBUG:boto:Params: {} 2022-09-12T19:06:44.101 DEBUG:boto:Token: None 2022-09-12T19:06:44.101 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.102 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e6dSi2GkU1Qc9ZWUdks0grQQPec= 2022-09-12T19:06:44.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e6dSi2GkU1Qc9ZWUdks0grQQPec=', 'Content-Length': '0'} 2022-09-12T19:06:44.105 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009516733d95782cbe-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.106 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:44.107 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:44.107 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.108 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.108 DEBUG:boto:Method: GET 2022-09-12T19:06:44.109 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:44.110 DEBUG:boto:Data: 2022-09-12T19:06:44.110 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.111 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.112 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.112 DEBUG:boto:Params: {} 2022-09-12T19:06:44.113 DEBUG:boto:Token: None 2022-09-12T19:06:44.113 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/ 2022-09-12T19:06:44.114 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs= 2022-09-12T19:06:44.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs=', 'Content-Length': '0'} 2022-09-12T19:06:44.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b46eb6394f42df63-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.118 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:44.119 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:44.120 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:44.120 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.121 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.121 DEBUG:boto:Method: GET 2022-09-12T19:06:44.122 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:44.123 DEBUG:boto:Data: 2022-09-12T19:06:44.123 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.124 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.125 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.127 DEBUG:boto:Params: {} 2022-09-12T19:06:44.127 DEBUG:boto:Token: None 2022-09-12T19:06:44.128 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/ 2022-09-12T19:06:44.129 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs= 2022-09-12T19:06:44.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs=', 'Content-Length': '0'} 2022-09-12T19:06:44.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005699b4e3b0005555-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.133 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:44.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:44.135 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.135 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.136 DEBUG:boto:path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:44.137 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:44.137 DEBUG:boto:Method: GET 2022-09-12T19:06:44.138 DEBUG:boto:Path: /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:44.138 DEBUG:boto:Data: 2022-09-12T19:06:44.139 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.139 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.140 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.140 DEBUG:boto:Params: {} 2022-09-12T19:06:44.141 DEBUG:boto:Token: None 2022-09-12T19:06:44.142 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:44.142 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UAEI5+zl5ACjp7wrf4GmX2NfyrA= 2022-09-12T19:06:44.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UAEI5+zl5ACjp7wrf4GmX2NfyrA=', 'Content-Length': '0'} 2022-09-12T19:06:44.145 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000915e0e60268da866-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.146 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.146 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.147 DEBUG:boto:path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:44.147 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:44.148 DEBUG:boto:Method: GET 2022-09-12T19:06:44.148 DEBUG:boto:Path: /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:44.149 DEBUG:boto:Data: 2022-09-12T19:06:44.150 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.150 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.151 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.151 DEBUG:boto:Params: {} 2022-09-12T19:06:44.152 DEBUG:boto:Token: None 2022-09-12T19:06:44.152 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:44.153 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UAEI5+zl5ACjp7wrf4GmX2NfyrA= 2022-09-12T19:06:44.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UAEI5+zl5ACjp7wrf4GmX2NfyrA=', 'Content-Length': '0'} 2022-09-12T19:06:44.155 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000382a301d52d1bdef-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.156 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.157 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.157 DEBUG:boto:path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:44.158 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:44.158 DEBUG:boto:Method: HEAD 2022-09-12T19:06:44.159 DEBUG:boto:Path: /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:44.159 DEBUG:boto:Data: 2022-09-12T19:06:44.160 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.160 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.161 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.161 DEBUG:boto:Params: {} 2022-09-12T19:06:44.162 DEBUG:boto:Token: None 2022-09-12T19:06:44.162 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:44.163 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cr9PeudccTDGaeo+TZacRqehQYU= 2022-09-12T19:06:44.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cr9PeudccTDGaeo+TZacRqehQYU=', 'Content-Length': '0'} 2022-09-12T19:06:44.165 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f865b2c557e48e84-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.166 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.167 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.167 DEBUG:boto:path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:44.168 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:44.168 DEBUG:boto:Method: HEAD 2022-09-12T19:06:44.169 DEBUG:boto:Path: /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:44.169 DEBUG:boto:Data: 2022-09-12T19:06:44.170 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.170 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.171 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.171 DEBUG:boto:Params: {} 2022-09-12T19:06:44.172 DEBUG:boto:Token: None 2022-09-12T19:06:44.172 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:44.173 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cr9PeudccTDGaeo+TZacRqehQYU= 2022-09-12T19:06:44.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cr9PeudccTDGaeo+TZacRqehQYU=', 'Content-Length': '0'} 2022-09-12T19:06:44.176 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2e5926bd04cadf9-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:44.177 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.178 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.178 DEBUG:boto:path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:44.179 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:44.179 DEBUG:boto:Method: GET 2022-09-12T19:06:44.180 DEBUG:boto:Path: /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:44.180 DEBUG:boto:Data: 2022-09-12T19:06:44.181 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.181 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.182 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.182 DEBUG:boto:Params: {} 2022-09-12T19:06:44.183 DEBUG:boto:Token: None 2022-09-12T19:06:44.184 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:44.184 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UAEI5+zl5ACjp7wrf4GmX2NfyrA= 2022-09-12T19:06:44.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UAEI5+zl5ACjp7wrf4GmX2NfyrA=', 'Content-Length': '0'} 2022-09-12T19:06:44.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000623c798157a9cf4b-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.188 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.188 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.189 DEBUG:boto:path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:44.189 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:44.190 DEBUG:boto:Method: GET 2022-09-12T19:06:44.190 DEBUG:boto:Path: /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:44.191 DEBUG:boto:Data: 2022-09-12T19:06:44.191 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.192 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.193 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.193 DEBUG:boto:Params: {} 2022-09-12T19:06:44.194 DEBUG:boto:Token: None 2022-09-12T19:06:44.194 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:44.195 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UAEI5+zl5ACjp7wrf4GmX2NfyrA= 2022-09-12T19:06:44.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UAEI5+zl5ACjp7wrf4GmX2NfyrA=', 'Content-Length': '0'} 2022-09-12T19:06:44.198 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5c6e880147c20c4-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.199 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:44.199 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:44.200 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.200 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.201 DEBUG:boto:Method: GET 2022-09-12T19:06:44.202 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:44.202 DEBUG:boto:Data: 2022-09-12T19:06:44.203 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.203 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.204 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.204 DEBUG:boto:Params: {} 2022-09-12T19:06:44.205 DEBUG:boto:Token: None 2022-09-12T19:06:44.206 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/ 2022-09-12T19:06:44.206 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs= 2022-09-12T19:06:44.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs=', 'Content-Length': '0'} 2022-09-12T19:06:44.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a82a0a839f8b0e31-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.210 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:44.211 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:44.212 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:44.212 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.213 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.213 DEBUG:boto:Method: GET 2022-09-12T19:06:44.214 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:44.214 DEBUG:boto:Data: 2022-09-12T19:06:44.215 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.216 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.216 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.217 DEBUG:boto:Params: {} 2022-09-12T19:06:44.217 DEBUG:boto:Token: None 2022-09-12T19:06:44.218 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/ 2022-09-12T19:06:44.219 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs= 2022-09-12T19:06:44.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs=', 'Content-Length': '0'} 2022-09-12T19:06:44.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009882f24b97d9144d-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.224 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:44.224 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.225 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.226 DEBUG:boto:Method: HEAD 2022-09-12T19:06:44.226 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:44.227 DEBUG:boto:Data: 2022-09-12T19:06:44.227 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.228 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.228 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.229 DEBUG:boto:Params: {} 2022-09-12T19:06:44.229 DEBUG:boto:Token: None 2022-09-12T19:06:44.230 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj 2022-09-12T19:06:44.231 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:neb+1Waat1etesg0gHGUltolAyk= 2022-09-12T19:06:44.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:neb+1Waat1etesg0gHGUltolAyk=', 'Content-Length': '0'} 2022-09-12T19:06:44.235 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000005cbc65df369b8c4-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.236 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.237 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.237 DEBUG:boto:Method: HEAD 2022-09-12T19:06:44.238 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:44.238 DEBUG:boto:Data: 2022-09-12T19:06:44.239 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.239 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.240 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.240 DEBUG:boto:Params: {} 2022-09-12T19:06:44.241 DEBUG:boto:Token: None 2022-09-12T19:06:44.241 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj 2022-09-12T19:06:44.242 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:neb+1Waat1etesg0gHGUltolAyk= 2022-09-12T19:06:44.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:neb+1Waat1etesg0gHGUltolAyk=', 'Content-Length': '0'} 2022-09-12T19:06:44.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb549f21f3e839c3-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:44.248 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.248 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.249 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.249 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.250 DEBUG:boto:Method: GET 2022-09-12T19:06:44.251 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.251 DEBUG:boto:Data: 2022-09-12T19:06:44.252 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.252 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.253 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.253 DEBUG:boto:Params: {} 2022-09-12T19:06:44.254 DEBUG:boto:Token: None 2022-09-12T19:06:44.255 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.255 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e6dSi2GkU1Qc9ZWUdks0grQQPec= 2022-09-12T19:06:44.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e6dSi2GkU1Qc9ZWUdks0grQQPec=', 'Content-Length': '0'} 2022-09-12T19:06:44.258 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3a13b262de2f48c-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.302 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.303 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.303 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.304 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.304 DEBUG:boto:Method: GET 2022-09-12T19:06:44.305 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.306 DEBUG:boto:Data: 2022-09-12T19:06:44.306 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.307 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.307 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.308 DEBUG:boto:Params: {} 2022-09-12T19:06:44.309 DEBUG:boto:Token: None 2022-09-12T19:06:44.310 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.311 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e6dSi2GkU1Qc9ZWUdks0grQQPec= 2022-09-12T19:06:44.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e6dSi2GkU1Qc9ZWUdks0grQQPec=', 'Content-Length': '0'} 2022-09-12T19:06:44.316 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a0d8370322c5575-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.317 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:44.318 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:44.319 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.320 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.321 DEBUG:boto:Method: GET 2022-09-12T19:06:44.322 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:44.323 DEBUG:boto:Data: 2022-09-12T19:06:44.324 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.325 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.326 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.326 DEBUG:boto:Params: {} 2022-09-12T19:06:44.327 DEBUG:boto:Token: None 2022-09-12T19:06:44.327 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/ 2022-09-12T19:06:44.328 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs= 2022-09-12T19:06:44.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs=', 'Content-Length': '0'} 2022-09-12T19:06:44.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064fcb50601b01590-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.332 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:44.333 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:44.333 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:44.334 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.335 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.335 DEBUG:boto:Method: GET 2022-09-12T19:06:44.336 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:44.336 DEBUG:boto:Data: 2022-09-12T19:06:44.337 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.337 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.338 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.338 DEBUG:boto:Params: {} 2022-09-12T19:06:44.339 DEBUG:boto:Token: None 2022-09-12T19:06:44.340 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/ 2022-09-12T19:06:44.340 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs= 2022-09-12T19:06:44.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs=', 'Content-Length': '0'} 2022-09-12T19:06:44.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d23b9725439e432-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.344 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:44.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:44.346 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.346 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.347 DEBUG:boto:path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:44.347 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:44.348 DEBUG:boto:Method: GET 2022-09-12T19:06:44.348 DEBUG:boto:Path: /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:44.349 DEBUG:boto:Data: 2022-09-12T19:06:44.350 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.350 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.351 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.352 DEBUG:boto:Params: {} 2022-09-12T19:06:44.353 DEBUG:boto:Token: None 2022-09-12T19:06:44.354 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:44.355 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jTV8yh34uMt0n5gPk1Jtg+W6EHQ= 2022-09-12T19:06:44.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jTV8yh34uMt0n5gPk1Jtg+W6EHQ=', 'Content-Length': '0'} 2022-09-12T19:06:44.357 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'TM67hyaYWRVq578zUsjsXjOzkUYTIvA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0bff989743292cc-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.359 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.359 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.360 DEBUG:boto:path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:44.361 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:44.362 DEBUG:boto:Method: GET 2022-09-12T19:06:44.363 DEBUG:boto:Path: /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:44.364 DEBUG:boto:Data: 2022-09-12T19:06:44.365 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.366 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.367 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.367 DEBUG:boto:Params: {} 2022-09-12T19:06:44.368 DEBUG:boto:Token: None 2022-09-12T19:06:44.369 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:44.370 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jTV8yh34uMt0n5gPk1Jtg+W6EHQ= 2022-09-12T19:06:44.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jTV8yh34uMt0n5gPk1Jtg+W6EHQ=', 'Content-Length': '0'} 2022-09-12T19:06:44.374 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'TM67hyaYWRVq578zUsjsXjOzkUYTIvA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee765d20ffe29546-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.375 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.376 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.377 DEBUG:boto:path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:44.378 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:44.379 DEBUG:boto:Method: HEAD 2022-09-12T19:06:44.380 DEBUG:boto:Path: /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:44.380 DEBUG:boto:Data: 2022-09-12T19:06:44.381 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.382 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.383 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.384 DEBUG:boto:Params: {} 2022-09-12T19:06:44.385 DEBUG:boto:Token: None 2022-09-12T19:06:44.386 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:44.387 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S84ZZJbqSMTIQb5XOE7EcfS87mY= 2022-09-12T19:06:44.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S84ZZJbqSMTIQb5XOE7EcfS87mY=', 'Content-Length': '0'} 2022-09-12T19:06:44.390 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'TM67hyaYWRVq578zUsjsXjOzkUYTIvA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041f263b19915f00c-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.392 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.393 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.393 DEBUG:boto:path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:44.394 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:44.394 DEBUG:boto:Method: HEAD 2022-09-12T19:06:44.395 DEBUG:boto:Path: /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:44.395 DEBUG:boto:Data: 2022-09-12T19:06:44.396 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.396 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.397 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.397 DEBUG:boto:Params: {} 2022-09-12T19:06:44.398 DEBUG:boto:Token: None 2022-09-12T19:06:44.399 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:44.400 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S84ZZJbqSMTIQb5XOE7EcfS87mY= 2022-09-12T19:06:44.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S84ZZJbqSMTIQb5XOE7EcfS87mY=', 'Content-Length': '0'} 2022-09-12T19:06:44.402 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'TM67hyaYWRVq578zUsjsXjOzkUYTIvA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077ee13b4fee91494-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:44.403 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.404 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.404 DEBUG:boto:path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:44.405 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:44.405 DEBUG:boto:Method: GET 2022-09-12T19:06:44.406 DEBUG:boto:Path: /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:44.407 DEBUG:boto:Data: 2022-09-12T19:06:44.407 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.408 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.409 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.409 DEBUG:boto:Params: {} 2022-09-12T19:06:44.410 DEBUG:boto:Token: None 2022-09-12T19:06:44.410 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:44.411 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jTV8yh34uMt0n5gPk1Jtg+W6EHQ= 2022-09-12T19:06:44.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jTV8yh34uMt0n5gPk1Jtg+W6EHQ=', 'Content-Length': '0'} 2022-09-12T19:06:44.413 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'TM67hyaYWRVq578zUsjsXjOzkUYTIvA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049514cf0cbe0a0d1-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.454 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.455 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.455 DEBUG:boto:path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:44.456 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:44.456 DEBUG:boto:Method: GET 2022-09-12T19:06:44.457 DEBUG:boto:Path: /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:44.457 DEBUG:boto:Data: 2022-09-12T19:06:44.458 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.459 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.459 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.460 DEBUG:boto:Params: {} 2022-09-12T19:06:44.460 DEBUG:boto:Token: None 2022-09-12T19:06:44.461 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:44.461 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jTV8yh34uMt0n5gPk1Jtg+W6EHQ= 2022-09-12T19:06:44.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jTV8yh34uMt0n5gPk1Jtg+W6EHQ=', 'Content-Length': '0'} 2022-09-12T19:06:44.464 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'TM67hyaYWRVq578zUsjsXjOzkUYTIvA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000389dc75c9aa0b39b-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.465 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:44.466 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:44.466 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.467 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.467 DEBUG:boto:Method: GET 2022-09-12T19:06:44.468 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:44.468 DEBUG:boto:Data: 2022-09-12T19:06:44.469 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.469 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.470 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.470 DEBUG:boto:Params: {} 2022-09-12T19:06:44.471 DEBUG:boto:Token: None 2022-09-12T19:06:44.471 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/ 2022-09-12T19:06:44.472 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs= 2022-09-12T19:06:44.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs=', 'Content-Length': '0'} 2022-09-12T19:06:44.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047f5330745fea56d-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.476 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:44.477 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:44.478 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:44.478 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.479 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.479 DEBUG:boto:Method: GET 2022-09-12T19:06:44.480 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:44.480 DEBUG:boto:Data: 2022-09-12T19:06:44.481 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.481 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.482 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.482 DEBUG:boto:Params: {} 2022-09-12T19:06:44.483 DEBUG:boto:Token: None 2022-09-12T19:06:44.483 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/ 2022-09-12T19:06:44.484 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs= 2022-09-12T19:06:44.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs=', 'Content-Length': '0'} 2022-09-12T19:06:44.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fe10f3641d79b31-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.488 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:44.489 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.490 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.490 DEBUG:boto:Method: HEAD 2022-09-12T19:06:44.491 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:44.491 DEBUG:boto:Data: 2022-09-12T19:06:44.492 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.492 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.493 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.493 DEBUG:boto:Params: {} 2022-09-12T19:06:44.494 DEBUG:boto:Token: None 2022-09-12T19:06:44.494 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj 2022-09-12T19:06:44.495 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:neb+1Waat1etesg0gHGUltolAyk= 2022-09-12T19:06:44.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:neb+1Waat1etesg0gHGUltolAyk=', 'Content-Length': '0'} 2022-09-12T19:06:44.499 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6a0f373871d31fe-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.499 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.500 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.500 DEBUG:boto:Method: HEAD 2022-09-12T19:06:44.501 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:44.501 DEBUG:boto:Data: 2022-09-12T19:06:44.502 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.502 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.503 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.503 DEBUG:boto:Params: {} 2022-09-12T19:06:44.504 DEBUG:boto:Token: None 2022-09-12T19:06:44.504 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj 2022-09-12T19:06:44.505 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:neb+1Waat1etesg0gHGUltolAyk= 2022-09-12T19:06:44.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:neb+1Waat1etesg0gHGUltolAyk=', 'Content-Length': '0'} 2022-09-12T19:06:44.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000533554a0d7ba357c-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:44.510 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.511 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.511 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.512 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.513 DEBUG:boto:Method: GET 2022-09-12T19:06:44.513 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.514 DEBUG:boto:Data: 2022-09-12T19:06:44.514 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.515 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.515 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.516 DEBUG:boto:Params: {} 2022-09-12T19:06:44.516 DEBUG:boto:Token: None 2022-09-12T19:06:44.517 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.517 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e6dSi2GkU1Qc9ZWUdks0grQQPec= 2022-09-12T19:06:44.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e6dSi2GkU1Qc9ZWUdks0grQQPec=', 'Content-Length': '0'} 2022-09-12T19:06:44.520 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa477b83ece21303-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.521 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.521 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.522 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.522 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.523 DEBUG:boto:Method: GET 2022-09-12T19:06:44.523 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.524 DEBUG:boto:Data: 2022-09-12T19:06:44.525 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.525 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.526 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.526 DEBUG:boto:Params: {} 2022-09-12T19:06:44.527 DEBUG:boto:Token: None 2022-09-12T19:06:44.527 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.528 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e6dSi2GkU1Qc9ZWUdks0grQQPec= 2022-09-12T19:06:44.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e6dSi2GkU1Qc9ZWUdks0grQQPec=', 'Content-Length': '0'} 2022-09-12T19:06:44.531 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae680a8027754915-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.578 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:44.579 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:44.580 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.580 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.581 DEBUG:boto:Method: GET 2022-09-12T19:06:44.581 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:44.582 DEBUG:boto:Data: 2022-09-12T19:06:44.582 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.583 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.583 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.584 DEBUG:boto:Params: {} 2022-09-12T19:06:44.585 DEBUG:boto:Token: None 2022-09-12T19:06:44.585 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/ 2022-09-12T19:06:44.586 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs= 2022-09-12T19:06:44.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs=', 'Content-Length': '0'} 2022-09-12T19:06:44.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000497788556752bcd-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.590 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:44.591 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:44.592 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:44.592 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.593 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.593 DEBUG:boto:Method: GET 2022-09-12T19:06:44.594 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:44.594 DEBUG:boto:Data: 2022-09-12T19:06:44.595 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.595 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.596 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.596 DEBUG:boto:Params: {} 2022-09-12T19:06:44.597 DEBUG:boto:Token: None 2022-09-12T19:06:44.598 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/ 2022-09-12T19:06:44.598 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs= 2022-09-12T19:06:44.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs=', 'Content-Length': '0'} 2022-09-12T19:06:44.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077fb9461eba8cda4-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.603 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:44.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:44.604 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.605 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.605 DEBUG:boto:path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:44.606 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:44.606 DEBUG:boto:Method: GET 2022-09-12T19:06:44.607 DEBUG:boto:Path: /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:44.607 DEBUG:boto:Data: 2022-09-12T19:06:44.608 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.608 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.609 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.610 DEBUG:boto:Params: {} 2022-09-12T19:06:44.610 DEBUG:boto:Token: None 2022-09-12T19:06:44.611 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:44.612 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RZENYDIzMY+qtedNIuFwe/NgUNE= 2022-09-12T19:06:44.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RZENYDIzMY+qtedNIuFwe/NgUNE=', 'Content-Length': '0'} 2022-09-12T19:06:44.614 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', '-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001459fcd5ee64ee62-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.615 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.615 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.616 DEBUG:boto:path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:44.617 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:44.617 DEBUG:boto:Method: GET 2022-09-12T19:06:44.618 DEBUG:boto:Path: /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:44.618 DEBUG:boto:Data: 2022-09-12T19:06:44.619 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.619 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.620 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.620 DEBUG:boto:Params: {} 2022-09-12T19:06:44.621 DEBUG:boto:Token: None 2022-09-12T19:06:44.622 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:44.622 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RZENYDIzMY+qtedNIuFwe/NgUNE= 2022-09-12T19:06:44.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RZENYDIzMY+qtedNIuFwe/NgUNE=', 'Content-Length': '0'} 2022-09-12T19:06:44.625 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', '-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002db97be32205b80-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.626 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.626 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.627 DEBUG:boto:path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:44.627 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:44.628 DEBUG:boto:Method: HEAD 2022-09-12T19:06:44.628 DEBUG:boto:Path: /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:44.629 DEBUG:boto:Data: 2022-09-12T19:06:44.629 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.630 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.630 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.631 DEBUG:boto:Params: {} 2022-09-12T19:06:44.631 DEBUG:boto:Token: None 2022-09-12T19:06:44.632 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:44.633 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ov/LokAnXASDxygTECjx1do/yOA= 2022-09-12T19:06:44.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ov/LokAnXASDxygTECjx1do/yOA=', 'Content-Length': '0'} 2022-09-12T19:06:44.635 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', '-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056b34295cae560e3-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.636 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.637 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.637 DEBUG:boto:path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:44.638 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:44.638 DEBUG:boto:Method: HEAD 2022-09-12T19:06:44.639 DEBUG:boto:Path: /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:44.639 DEBUG:boto:Data: 2022-09-12T19:06:44.640 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.640 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.641 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.641 DEBUG:boto:Params: {} 2022-09-12T19:06:44.642 DEBUG:boto:Token: None 2022-09-12T19:06:44.642 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:44.643 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ov/LokAnXASDxygTECjx1do/yOA= 2022-09-12T19:06:44.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ov/LokAnXASDxygTECjx1do/yOA=', 'Content-Length': '0'} 2022-09-12T19:06:44.646 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', '-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000384ee348251c8983-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:44.647 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.648 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.648 DEBUG:boto:path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:44.649 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:44.649 DEBUG:boto:Method: GET 2022-09-12T19:06:44.650 DEBUG:boto:Path: /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:44.651 DEBUG:boto:Data: 2022-09-12T19:06:44.651 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.652 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.652 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.653 DEBUG:boto:Params: {} 2022-09-12T19:06:44.654 DEBUG:boto:Token: None 2022-09-12T19:06:44.654 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:44.655 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RZENYDIzMY+qtedNIuFwe/NgUNE= 2022-09-12T19:06:44.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RZENYDIzMY+qtedNIuFwe/NgUNE=', 'Content-Length': '0'} 2022-09-12T19:06:44.658 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', '-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006979d51b55461f3c-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.698 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.699 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.699 DEBUG:boto:path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:44.700 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:44.701 DEBUG:boto:Method: GET 2022-09-12T19:06:44.701 DEBUG:boto:Path: /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:44.702 DEBUG:boto:Data: 2022-09-12T19:06:44.702 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.703 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.703 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.704 DEBUG:boto:Params: {} 2022-09-12T19:06:44.704 DEBUG:boto:Token: None 2022-09-12T19:06:44.705 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:44.706 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RZENYDIzMY+qtedNIuFwe/NgUNE= 2022-09-12T19:06:44.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RZENYDIzMY+qtedNIuFwe/NgUNE=', 'Content-Length': '0'} 2022-09-12T19:06:44.712 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', '-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e062ce6b6c342439-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.714 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:44.715 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:44.716 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.717 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.717 DEBUG:boto:Method: GET 2022-09-12T19:06:44.718 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:44.720 DEBUG:boto:Data: 2022-09-12T19:06:44.720 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.721 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.721 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.722 DEBUG:boto:Params: {} 2022-09-12T19:06:44.722 DEBUG:boto:Token: None 2022-09-12T19:06:44.723 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/ 2022-09-12T19:06:44.724 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs= 2022-09-12T19:06:44.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs=', 'Content-Length': '0'} 2022-09-12T19:06:44.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe8d36d0d7e2a2a7-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.728 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:44.729 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:44.729 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:44.730 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.730 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.731 DEBUG:boto:Method: GET 2022-09-12T19:06:44.731 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:44.732 DEBUG:boto:Data: 2022-09-12T19:06:44.732 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.733 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.733 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.734 DEBUG:boto:Params: {} 2022-09-12T19:06:44.735 DEBUG:boto:Token: None 2022-09-12T19:06:44.735 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/ 2022-09-12T19:06:44.736 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs= 2022-09-12T19:06:44.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs=', 'Content-Length': '0'} 2022-09-12T19:06:44.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa6e7034c75a08ae-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.740 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:44.741 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.741 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.742 DEBUG:boto:Method: HEAD 2022-09-12T19:06:44.743 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:44.743 DEBUG:boto:Data: 2022-09-12T19:06:44.744 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.744 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.745 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.745 DEBUG:boto:Params: {} 2022-09-12T19:06:44.746 DEBUG:boto:Token: None 2022-09-12T19:06:44.746 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj 2022-09-12T19:06:44.747 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:neb+1Waat1etesg0gHGUltolAyk= 2022-09-12T19:06:44.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:neb+1Waat1etesg0gHGUltolAyk=', 'Content-Length': '0'} 2022-09-12T19:06:44.751 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3aa3317e75b86f3-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.752 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.752 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.753 DEBUG:boto:Method: HEAD 2022-09-12T19:06:44.753 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:44.754 DEBUG:boto:Data: 2022-09-12T19:06:44.754 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.755 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.756 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.756 DEBUG:boto:Params: {} 2022-09-12T19:06:44.757 DEBUG:boto:Token: None 2022-09-12T19:06:44.757 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj 2022-09-12T19:06:44.758 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:neb+1Waat1etesg0gHGUltolAyk= 2022-09-12T19:06:44.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:neb+1Waat1etesg0gHGUltolAyk=', 'Content-Length': '0'} 2022-09-12T19:06:44.763 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ab57fecf8ef9b54-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:44.764 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.765 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.765 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.766 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.767 DEBUG:boto:Method: GET 2022-09-12T19:06:44.767 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.768 DEBUG:boto:Data: 2022-09-12T19:06:44.768 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.769 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.769 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.770 DEBUG:boto:Params: {} 2022-09-12T19:06:44.772 DEBUG:boto:Token: None 2022-09-12T19:06:44.772 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.773 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e6dSi2GkU1Qc9ZWUdks0grQQPec= 2022-09-12T19:06:44.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e6dSi2GkU1Qc9ZWUdks0grQQPec=', 'Content-Length': '0'} 2022-09-12T19:06:44.776 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bbccb53409e8cada-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.818 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.819 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.820 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.821 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.822 DEBUG:boto:Method: GET 2022-09-12T19:06:44.823 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.824 DEBUG:boto:Data: 2022-09-12T19:06:44.824 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.825 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.826 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.826 DEBUG:boto:Params: {} 2022-09-12T19:06:44.827 DEBUG:boto:Token: None 2022-09-12T19:06:44.827 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.828 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e6dSi2GkU1Qc9ZWUdks0grQQPec= 2022-09-12T19:06:44.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e6dSi2GkU1Qc9ZWUdks0grQQPec=', 'Content-Length': '0'} 2022-09-12T19:06:44.831 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000745d55f265f7af33-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.832 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:44.833 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:44.834 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.835 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.835 DEBUG:boto:Method: GET 2022-09-12T19:06:44.836 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:44.836 DEBUG:boto:Data: 2022-09-12T19:06:44.837 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.838 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.838 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.839 DEBUG:boto:Params: {} 2022-09-12T19:06:44.839 DEBUG:boto:Token: None 2022-09-12T19:06:44.840 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/ 2022-09-12T19:06:44.841 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs= 2022-09-12T19:06:44.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs=', 'Content-Length': '0'} 2022-09-12T19:06:44.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2ba67388cb78af1-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.845 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:44.846 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:44.846 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:44.847 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.847 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.848 DEBUG:boto:Method: GET 2022-09-12T19:06:44.849 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:44.849 DEBUG:boto:Data: 2022-09-12T19:06:44.850 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.850 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.851 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.851 DEBUG:boto:Params: {} 2022-09-12T19:06:44.852 DEBUG:boto:Token: None 2022-09-12T19:06:44.853 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/ 2022-09-12T19:06:44.853 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs= 2022-09-12T19:06:44.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs=', 'Content-Length': '0'} 2022-09-12T19:06:44.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f523bde82687b2e-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.858 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:44.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:44.860 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.861 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.862 DEBUG:boto:path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:44.862 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:44.863 DEBUG:boto:Method: GET 2022-09-12T19:06:44.864 DEBUG:boto:Path: /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:44.864 DEBUG:boto:Data: 2022-09-12T19:06:44.865 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.865 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.866 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.866 DEBUG:boto:Params: {} 2022-09-12T19:06:44.867 DEBUG:boto:Token: None 2022-09-12T19:06:44.868 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:44.868 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jvFE9oUsZtgKmOKXH/TcMPNNZ0Y= 2022-09-12T19:06:44.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jvFE9oUsZtgKmOKXH/TcMPNNZ0Y=', 'Content-Length': '0'} 2022-09-12T19:06:44.871 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'pwncGyI2j8AeHH4hjzwsep2DWsp-l-A'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3a7b6fe4121e689-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.872 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.873 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.873 DEBUG:boto:path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:44.874 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:44.874 DEBUG:boto:Method: GET 2022-09-12T19:06:44.875 DEBUG:boto:Path: /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:44.876 DEBUG:boto:Data: 2022-09-12T19:06:44.876 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.877 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.877 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.878 DEBUG:boto:Params: {} 2022-09-12T19:06:44.878 DEBUG:boto:Token: None 2022-09-12T19:06:44.879 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:44.880 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jvFE9oUsZtgKmOKXH/TcMPNNZ0Y= 2022-09-12T19:06:44.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jvFE9oUsZtgKmOKXH/TcMPNNZ0Y=', 'Content-Length': '0'} 2022-09-12T19:06:44.883 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'pwncGyI2j8AeHH4hjzwsep2DWsp-l-A'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a16302bf51093af0-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.884 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.884 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.885 DEBUG:boto:path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:44.885 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:44.886 DEBUG:boto:Method: HEAD 2022-09-12T19:06:44.886 DEBUG:boto:Path: /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:44.887 DEBUG:boto:Data: 2022-09-12T19:06:44.887 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.888 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.889 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.889 DEBUG:boto:Params: {} 2022-09-12T19:06:44.890 DEBUG:boto:Token: None 2022-09-12T19:06:44.891 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:44.892 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ej+jH89XkBabpR+uk4Trjux1gn4= 2022-09-12T19:06:44.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ej+jH89XkBabpR+uk4Trjux1gn4=', 'Content-Length': '0'} 2022-09-12T19:06:44.895 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'pwncGyI2j8AeHH4hjzwsep2DWsp-l-A'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a90cd845f06510d3-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.896 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.896 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.897 DEBUG:boto:path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:44.897 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:44.898 DEBUG:boto:Method: HEAD 2022-09-12T19:06:44.898 DEBUG:boto:Path: /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:44.899 DEBUG:boto:Data: 2022-09-12T19:06:44.899 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.900 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.900 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.901 DEBUG:boto:Params: {} 2022-09-12T19:06:44.901 DEBUG:boto:Token: None 2022-09-12T19:06:44.902 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:44.903 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ej+jH89XkBabpR+uk4Trjux1gn4= 2022-09-12T19:06:44.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ej+jH89XkBabpR+uk4Trjux1gn4=', 'Content-Length': '0'} 2022-09-12T19:06:44.905 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'pwncGyI2j8AeHH4hjzwsep2DWsp-l-A'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d9d2a2213fdd721a-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:44.907 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.907 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.908 DEBUG:boto:path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:44.908 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:44.909 DEBUG:boto:Method: GET 2022-09-12T19:06:44.910 DEBUG:boto:Path: /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:44.910 DEBUG:boto:Data: 2022-09-12T19:06:44.911 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.911 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.912 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.912 DEBUG:boto:Params: {} 2022-09-12T19:06:44.913 DEBUG:boto:Token: None 2022-09-12T19:06:44.914 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:44.914 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jvFE9oUsZtgKmOKXH/TcMPNNZ0Y= 2022-09-12T19:06:44.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jvFE9oUsZtgKmOKXH/TcMPNNZ0Y=', 'Content-Length': '0'} 2022-09-12T19:06:44.917 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'pwncGyI2j8AeHH4hjzwsep2DWsp-l-A'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001284ed525fde25fe-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.918 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.918 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.919 DEBUG:boto:path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:44.920 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:44.920 DEBUG:boto:Method: GET 2022-09-12T19:06:44.921 DEBUG:boto:Path: /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:44.921 DEBUG:boto:Data: 2022-09-12T19:06:44.922 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.922 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.923 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.923 DEBUG:boto:Params: {} 2022-09-12T19:06:44.924 DEBUG:boto:Token: None 2022-09-12T19:06:44.924 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:44.925 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jvFE9oUsZtgKmOKXH/TcMPNNZ0Y= 2022-09-12T19:06:44.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jvFE9oUsZtgKmOKXH/TcMPNNZ0Y=', 'Content-Length': '0'} 2022-09-12T19:06:44.928 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'pwncGyI2j8AeHH4hjzwsep2DWsp-l-A'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002bb19fd88624ae32-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.929 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:44.929 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:44.930 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.931 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.931 DEBUG:boto:Method: GET 2022-09-12T19:06:44.932 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:44.932 DEBUG:boto:Data: 2022-09-12T19:06:44.933 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.934 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.934 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.935 DEBUG:boto:Params: {} 2022-09-12T19:06:44.935 DEBUG:boto:Token: None 2022-09-12T19:06:44.936 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/ 2022-09-12T19:06:44.937 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs= 2022-09-12T19:06:44.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs=', 'Content-Length': '0'} 2022-09-12T19:06:44.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5997d2e697076f0-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.941 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:44.942 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:44.942 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:44.943 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.943 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:44.944 DEBUG:boto:Method: GET 2022-09-12T19:06:44.944 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:44.945 DEBUG:boto:Data: 2022-09-12T19:06:44.945 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.946 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.947 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.947 DEBUG:boto:Params: {} 2022-09-12T19:06:44.948 DEBUG:boto:Token: None 2022-09-12T19:06:44.948 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/ 2022-09-12T19:06:44.949 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs= 2022-09-12T19:06:44.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XrrFjv3sNRr08Kc6UYmPeXGgEHs=', 'Content-Length': '0'} 2022-09-12T19:06:44.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f0a30318f9b93e2-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.954 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:44.955 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.955 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.956 DEBUG:boto:Method: HEAD 2022-09-12T19:06:44.956 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:44.957 DEBUG:boto:Data: 2022-09-12T19:06:44.958 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.958 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.959 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.959 DEBUG:boto:Params: {} 2022-09-12T19:06:44.960 DEBUG:boto:Token: None 2022-09-12T19:06:44.960 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj 2022-09-12T19:06:44.961 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:neb+1Waat1etesg0gHGUltolAyk= 2022-09-12T19:06:44.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:neb+1Waat1etesg0gHGUltolAyk=', 'Content-Length': '0'} 2022-09-12T19:06:44.966 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054937f61ecd355ad-00631f8344-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.967 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.967 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.968 DEBUG:boto:Method: HEAD 2022-09-12T19:06:44.968 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:44.969 DEBUG:boto:Data: 2022-09-12T19:06:44.969 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.970 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:44.970 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.971 DEBUG:boto:Params: {} 2022-09-12T19:06:44.971 DEBUG:boto:Token: None 2022-09-12T19:06:44.972 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj 2022-09-12T19:06:44.974 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:neb+1Waat1etesg0gHGUltolAyk= 2022-09-12T19:06:44.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:neb+1Waat1etesg0gHGUltolAyk=', 'Content-Length': '0'} 2022-09-12T19:06:44.984 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbac2bab5a21365c-00631f8344-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:44.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:44.987 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:44.988 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:44.989 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.990 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.991 DEBUG:boto:Method: GET 2022-09-12T19:06:44.992 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:44.993 DEBUG:boto:Data: 2022-09-12T19:06:44.994 DEBUG:boto:Headers: {} 2022-09-12T19:06:44.995 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:44.996 DEBUG:boto:Port: 8000 2022-09-12T19:06:44.997 DEBUG:boto:Params: {} 2022-09-12T19:06:44.998 DEBUG:boto:Token: None 2022-09-12T19:06:44.999 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:44 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.000 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e6dSi2GkU1Qc9ZWUdks0grQQPec= 2022-09-12T19:06:45.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e6dSi2GkU1Qc9ZWUdks0grQQPec=', 'Content-Length': '0'} 2022-09-12T19:06:45.004 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b61f6f113aeed78f-00631f8345-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.046 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.048 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.048 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.049 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.049 DEBUG:boto:Method: GET 2022-09-12T19:06:45.050 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.050 DEBUG:boto:Data: 2022-09-12T19:06:45.051 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.052 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:45.052 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.053 DEBUG:boto:Params: {} 2022-09-12T19:06:45.053 DEBUG:boto:Token: None 2022-09-12T19:06:45.054 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.055 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VxZkLQtwwxbn+V8PGPdnMgviBfY= 2022-09-12T19:06:45.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VxZkLQtwwxbn+V8PGPdnMgviBfY=', 'Content-Length': '0'} 2022-09-12T19:06:45.059 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019e9379254da6413-00631f8345-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.060 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:45.061 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:45.062 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.062 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.063 DEBUG:boto:Method: GET 2022-09-12T19:06:45.064 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:45.065 DEBUG:boto:Data: 2022-09-12T19:06:45.065 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.066 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:45.067 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.068 DEBUG:boto:Params: {} 2022-09-12T19:06:45.069 DEBUG:boto:Token: None 2022-09-12T19:06:45.070 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/ 2022-09-12T19:06:45.071 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo= 2022-09-12T19:06:45.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo=', 'Content-Length': '0'} 2022-09-12T19:06:45.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe806ae1c7d46284-00631f8345-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.076 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:45.078 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:45.079 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:45.079 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.080 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.081 DEBUG:boto:Method: GET 2022-09-12T19:06:45.082 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:45.083 DEBUG:boto:Data: 2022-09-12T19:06:45.084 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.084 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:45.085 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.086 DEBUG:boto:Params: {} 2022-09-12T19:06:45.087 DEBUG:boto:Token: None 2022-09-12T19:06:45.088 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/ 2022-09-12T19:06:45.089 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo= 2022-09-12T19:06:45.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo=', 'Content-Length': '0'} 2022-09-12T19:06:45.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4dd0af664841576-00631f8345-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.134 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:45.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:45.136 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.136 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.137 DEBUG:boto:path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:45.137 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:45.138 DEBUG:boto:Method: GET 2022-09-12T19:06:45.138 DEBUG:boto:Path: /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:45.139 DEBUG:boto:Data: 2022-09-12T19:06:45.139 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.140 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:45.141 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.141 DEBUG:boto:Params: {} 2022-09-12T19:06:45.142 DEBUG:boto:Token: None 2022-09-12T19:06:45.143 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:45.144 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HRTPRgnbcJJbIbAAzG+EGruuKpg= 2022-09-12T19:06:45.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HRTPRgnbcJJbIbAAzG+EGruuKpg=', 'Content-Length': '0'} 2022-09-12T19:06:45.147 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a89a64e5be4554f8-00631f8345-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.148 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.149 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.150 DEBUG:boto:path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:45.151 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:45.151 DEBUG:boto:Method: GET 2022-09-12T19:06:45.152 DEBUG:boto:Path: /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:45.153 DEBUG:boto:Data: 2022-09-12T19:06:45.154 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.154 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:45.155 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.156 DEBUG:boto:Params: {} 2022-09-12T19:06:45.157 DEBUG:boto:Token: None 2022-09-12T19:06:45.158 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:45.159 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HRTPRgnbcJJbIbAAzG+EGruuKpg= 2022-09-12T19:06:45.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HRTPRgnbcJJbIbAAzG+EGruuKpg=', 'Content-Length': '0'} 2022-09-12T19:06:45.162 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca35e9fe8a563ef8-00631f8345-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.163 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.164 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.165 DEBUG:boto:path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:45.166 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:45.167 DEBUG:boto:Method: HEAD 2022-09-12T19:06:45.167 DEBUG:boto:Path: /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:45.168 DEBUG:boto:Data: 2022-09-12T19:06:45.169 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.170 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:45.171 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.171 DEBUG:boto:Params: {} 2022-09-12T19:06:45.172 DEBUG:boto:Token: None 2022-09-12T19:06:45.173 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:45.174 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vzrT3mWdaXANEydqBygMxVcsp6A= 2022-09-12T19:06:45.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vzrT3mWdaXANEydqBygMxVcsp6A=', 'Content-Length': '0'} 2022-09-12T19:06:45.177 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003afc36019c8b6261-00631f8345-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.178 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.179 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.180 DEBUG:boto:path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:45.180 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:45.181 DEBUG:boto:Method: HEAD 2022-09-12T19:06:45.182 DEBUG:boto:Path: /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:45.182 DEBUG:boto:Data: 2022-09-12T19:06:45.183 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.184 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:45.185 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.185 DEBUG:boto:Params: {} 2022-09-12T19:06:45.186 DEBUG:boto:Token: None 2022-09-12T19:06:45.187 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:45.188 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vzrT3mWdaXANEydqBygMxVcsp6A= 2022-09-12T19:06:45.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vzrT3mWdaXANEydqBygMxVcsp6A=', 'Content-Length': '0'} 2022-09-12T19:06:45.191 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000043c6b900481dc61d-00631f8345-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:45.193 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.194 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.195 DEBUG:boto:path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:45.195 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:45.196 DEBUG:boto:Method: GET 2022-09-12T19:06:45.197 DEBUG:boto:Path: /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:45.198 DEBUG:boto:Data: 2022-09-12T19:06:45.199 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.199 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:45.200 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.201 DEBUG:boto:Params: {} 2022-09-12T19:06:45.202 DEBUG:boto:Token: None 2022-09-12T19:06:45.203 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:45.204 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HRTPRgnbcJJbIbAAzG+EGruuKpg= 2022-09-12T19:06:45.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HRTPRgnbcJJbIbAAzG+EGruuKpg=', 'Content-Length': '0'} 2022-09-12T19:06:45.207 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d0eb984a730b8bd2-00631f8345-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.208 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.209 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.209 DEBUG:boto:path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:45.210 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:45.211 DEBUG:boto:Method: GET 2022-09-12T19:06:45.212 DEBUG:boto:Path: /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:45.213 DEBUG:boto:Data: 2022-09-12T19:06:45.214 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.214 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:45.215 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.215 DEBUG:boto:Params: {} 2022-09-12T19:06:45.216 DEBUG:boto:Token: None 2022-09-12T19:06:45.216 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:45.217 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HRTPRgnbcJJbIbAAzG+EGruuKpg= 2022-09-12T19:06:45.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HRTPRgnbcJJbIbAAzG+EGruuKpg=', 'Content-Length': '0'} 2022-09-12T19:06:45.220 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008fb2f95e1f598e1c-00631f8345-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.221 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:45.221 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:45.222 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.223 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.224 DEBUG:boto:Method: GET 2022-09-12T19:06:45.225 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:45.226 DEBUG:boto:Data: 2022-09-12T19:06:45.226 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.227 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:45.228 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.229 DEBUG:boto:Params: {} 2022-09-12T19:06:45.230 DEBUG:boto:Token: None 2022-09-12T19:06:45.231 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/ 2022-09-12T19:06:45.231 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo= 2022-09-12T19:06:45.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo=', 'Content-Length': '0'} 2022-09-12T19:06:45.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049ac1e525f464d83-00631f8345-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.282 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:45.283 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:45.284 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:45.285 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.285 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.286 DEBUG:boto:Method: GET 2022-09-12T19:06:45.286 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:45.287 DEBUG:boto:Data: 2022-09-12T19:06:45.287 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.288 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:45.289 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.290 DEBUG:boto:Params: {} 2022-09-12T19:06:45.290 DEBUG:boto:Token: None 2022-09-12T19:06:45.291 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/ 2022-09-12T19:06:45.291 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo= 2022-09-12T19:06:45.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo=', 'Content-Length': '0'} 2022-09-12T19:06:45.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc1299c222126bf3-00631f8345-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.296 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:45.297 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.298 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.298 DEBUG:boto:Method: HEAD 2022-09-12T19:06:45.299 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:45.299 DEBUG:boto:Data: 2022-09-12T19:06:45.300 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.300 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:45.301 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.301 DEBUG:boto:Params: {} 2022-09-12T19:06:45.302 DEBUG:boto:Token: None 2022-09-12T19:06:45.302 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj 2022-09-12T19:06:45.303 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V5KHsU4YJr6+P2ufm4RDU51vFZo= 2022-09-12T19:06:45.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V5KHsU4YJr6+P2ufm4RDU51vFZo=', 'Content-Length': '0'} 2022-09-12T19:06:45.307 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000157c6a25e04878ac-00631f8345-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.308 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.308 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.309 DEBUG:boto:Method: HEAD 2022-09-12T19:06:45.309 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:45.310 DEBUG:boto:Data: 2022-09-12T19:06:45.311 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.311 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:45.311 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.312 DEBUG:boto:Params: {} 2022-09-12T19:06:45.313 DEBUG:boto:Token: None 2022-09-12T19:06:45.313 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj 2022-09-12T19:06:45.314 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V5KHsU4YJr6+P2ufm4RDU51vFZo= 2022-09-12T19:06:45.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V5KHsU4YJr6+P2ufm4RDU51vFZo=', 'Content-Length': '0'} 2022-09-12T19:06:45.318 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb5c816e9efc5686-00631f8345-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:45.319 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.320 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.321 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.322 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.323 DEBUG:boto:Method: GET 2022-09-12T19:06:45.324 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.324 DEBUG:boto:Data: 2022-09-12T19:06:45.325 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.326 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:45.326 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.327 DEBUG:boto:Params: {} 2022-09-12T19:06:45.328 DEBUG:boto:Token: None 2022-09-12T19:06:45.328 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.329 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VxZkLQtwwxbn+V8PGPdnMgviBfY= 2022-09-12T19:06:45.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VxZkLQtwwxbn+V8PGPdnMgviBfY=', 'Content-Length': '0'} 2022-09-12T19:06:45.332 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025748322f2c97ad5-00631f8345-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.333 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.333 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.334 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.334 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.335 DEBUG:boto:Method: GET 2022-09-12T19:06:45.335 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.336 DEBUG:boto:Data: 2022-09-12T19:06:45.337 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.338 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:45.338 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.339 DEBUG:boto:Params: {} 2022-09-12T19:06:45.339 DEBUG:boto:Token: None 2022-09-12T19:06:45.340 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.341 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VxZkLQtwwxbn+V8PGPdnMgviBfY= 2022-09-12T19:06:45.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VxZkLQtwwxbn+V8PGPdnMgviBfY=', 'Content-Length': '0'} 2022-09-12T19:06:45.343 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c69a9ae785f8999f-00631f8345-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.386 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:45.387 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:45.387 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.388 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.388 DEBUG:boto:Method: GET 2022-09-12T19:06:45.389 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:45.390 DEBUG:boto:Data: 2022-09-12T19:06:45.391 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.391 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:45.392 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.392 DEBUG:boto:Params: {} 2022-09-12T19:06:45.393 DEBUG:boto:Token: None 2022-09-12T19:06:45.393 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/ 2022-09-12T19:06:45.394 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo= 2022-09-12T19:06:45.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo=', 'Content-Length': '0'} 2022-09-12T19:06:45.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091ad4efecb87b3ae-00631f8345-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.398 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:45.399 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:45.400 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:45.400 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.401 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.401 DEBUG:boto:Method: GET 2022-09-12T19:06:45.402 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:45.402 DEBUG:boto:Data: 2022-09-12T19:06:45.403 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.403 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:45.404 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.404 DEBUG:boto:Params: {} 2022-09-12T19:06:45.405 DEBUG:boto:Token: None 2022-09-12T19:06:45.405 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/ 2022-09-12T19:06:45.406 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo= 2022-09-12T19:06:45.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo=', 'Content-Length': '0'} 2022-09-12T19:06:45.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbd960942945085d-00631f8345-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.411 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:45.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:45.412 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.412 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.413 DEBUG:boto:path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:45.414 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:45.414 DEBUG:boto:Method: GET 2022-09-12T19:06:45.415 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:45.415 DEBUG:boto:Data: 2022-09-12T19:06:45.416 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.416 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:45.417 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.417 DEBUG:boto:Params: {} 2022-09-12T19:06:45.418 DEBUG:boto:Token: None 2022-09-12T19:06:45.419 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:45.419 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WgVOjDBMsDGcl6W6QsslDdapoRM= 2022-09-12T19:06:45.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WgVOjDBMsDGcl6W6QsslDdapoRM=', 'Content-Length': '0'} 2022-09-12T19:06:45.422 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'T.yM--FII-PB1GReM85mkNRGYz6Dkbz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e30ceb933e61555-00631f8345-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.423 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.423 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.424 DEBUG:boto:path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:45.424 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:45.425 DEBUG:boto:Method: GET 2022-09-12T19:06:45.425 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:45.426 DEBUG:boto:Data: 2022-09-12T19:06:45.426 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.427 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:45.428 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.428 DEBUG:boto:Params: {} 2022-09-12T19:06:45.429 DEBUG:boto:Token: None 2022-09-12T19:06:45.429 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:45.430 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WgVOjDBMsDGcl6W6QsslDdapoRM= 2022-09-12T19:06:45.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WgVOjDBMsDGcl6W6QsslDdapoRM=', 'Content-Length': '0'} 2022-09-12T19:06:45.433 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'T.yM--FII-PB1GReM85mkNRGYz6Dkbz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5606db946086b5b-00631f8345-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.434 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.434 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.435 DEBUG:boto:path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:45.435 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:45.436 DEBUG:boto:Method: HEAD 2022-09-12T19:06:45.436 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:45.437 DEBUG:boto:Data: 2022-09-12T19:06:45.437 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.438 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:45.438 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.439 DEBUG:boto:Params: {} 2022-09-12T19:06:45.439 DEBUG:boto:Token: None 2022-09-12T19:06:45.440 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:45.441 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YFnFyEBPilvdw5r4IdOHXGEw5PU= 2022-09-12T19:06:45.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YFnFyEBPilvdw5r4IdOHXGEw5PU=', 'Content-Length': '0'} 2022-09-12T19:06:45.444 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'T.yM--FII-PB1GReM85mkNRGYz6Dkbz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c44109c402cf7873-00631f8345-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.444 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.445 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.445 DEBUG:boto:path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:45.446 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:45.446 DEBUG:boto:Method: HEAD 2022-09-12T19:06:45.447 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:45.447 DEBUG:boto:Data: 2022-09-12T19:06:45.448 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.449 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:45.449 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.450 DEBUG:boto:Params: {} 2022-09-12T19:06:45.450 DEBUG:boto:Token: None 2022-09-12T19:06:45.451 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:45.452 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YFnFyEBPilvdw5r4IdOHXGEw5PU= 2022-09-12T19:06:45.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YFnFyEBPilvdw5r4IdOHXGEw5PU=', 'Content-Length': '0'} 2022-09-12T19:06:45.454 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'T.yM--FII-PB1GReM85mkNRGYz6Dkbz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004b45c1c27453a91-00631f8345-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:45.456 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.456 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.457 DEBUG:boto:path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:45.457 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:45.458 DEBUG:boto:Method: GET 2022-09-12T19:06:45.458 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:45.459 DEBUG:boto:Data: 2022-09-12T19:06:45.459 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.460 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:45.460 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.461 DEBUG:boto:Params: {} 2022-09-12T19:06:45.462 DEBUG:boto:Token: None 2022-09-12T19:06:45.462 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:45.463 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WgVOjDBMsDGcl6W6QsslDdapoRM= 2022-09-12T19:06:45.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WgVOjDBMsDGcl6W6QsslDdapoRM=', 'Content-Length': '0'} 2022-09-12T19:06:45.466 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'T.yM--FII-PB1GReM85mkNRGYz6Dkbz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e53e3da51d95f7a-00631f8345-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.506 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.507 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.509 DEBUG:boto:path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:45.509 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:45.510 DEBUG:boto:Method: GET 2022-09-12T19:06:45.511 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:45.512 DEBUG:boto:Data: 2022-09-12T19:06:45.513 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.514 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:45.515 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.516 DEBUG:boto:Params: {} 2022-09-12T19:06:45.517 DEBUG:boto:Token: None 2022-09-12T19:06:45.519 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:45.520 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WgVOjDBMsDGcl6W6QsslDdapoRM= 2022-09-12T19:06:45.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WgVOjDBMsDGcl6W6QsslDdapoRM=', 'Content-Length': '0'} 2022-09-12T19:06:45.524 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'T.yM--FII-PB1GReM85mkNRGYz6Dkbz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017fac571c186c333-00631f8345-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.526 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:45.527 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:45.528 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.529 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.530 DEBUG:boto:Method: GET 2022-09-12T19:06:45.530 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:45.531 DEBUG:boto:Data: 2022-09-12T19:06:45.531 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.532 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:45.533 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.533 DEBUG:boto:Params: {} 2022-09-12T19:06:45.534 DEBUG:boto:Token: None 2022-09-12T19:06:45.534 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/ 2022-09-12T19:06:45.535 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo= 2022-09-12T19:06:45.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo=', 'Content-Length': '0'} 2022-09-12T19:06:45.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acc7cc3eaf47c3fb-00631f8345-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.541 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:45.542 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:45.543 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:45.544 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.544 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.545 DEBUG:boto:Method: GET 2022-09-12T19:06:45.546 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:45.547 DEBUG:boto:Data: 2022-09-12T19:06:45.547 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.549 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:45.549 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.550 DEBUG:boto:Params: {} 2022-09-12T19:06:45.551 DEBUG:boto:Token: None 2022-09-12T19:06:45.551 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/ 2022-09-12T19:06:45.552 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo= 2022-09-12T19:06:45.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo=', 'Content-Length': '0'} 2022-09-12T19:06:45.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086e516c238e02ffc-00631f8345-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.557 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:45.558 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.559 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.560 DEBUG:boto:Method: HEAD 2022-09-12T19:06:45.560 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:45.561 DEBUG:boto:Data: 2022-09-12T19:06:45.562 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.563 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:45.563 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.564 DEBUG:boto:Params: {} 2022-09-12T19:06:45.564 DEBUG:boto:Token: None 2022-09-12T19:06:45.565 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj 2022-09-12T19:06:45.566 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V5KHsU4YJr6+P2ufm4RDU51vFZo= 2022-09-12T19:06:45.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V5KHsU4YJr6+P2ufm4RDU51vFZo=', 'Content-Length': '0'} 2022-09-12T19:06:45.572 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006411076323b57a80-00631f8345-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.574 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.574 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.575 DEBUG:boto:Method: HEAD 2022-09-12T19:06:45.575 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:45.576 DEBUG:boto:Data: 2022-09-12T19:06:45.576 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.577 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:45.578 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.578 DEBUG:boto:Params: {} 2022-09-12T19:06:45.579 DEBUG:boto:Token: None 2022-09-12T19:06:45.580 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj 2022-09-12T19:06:45.580 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V5KHsU4YJr6+P2ufm4RDU51vFZo= 2022-09-12T19:06:45.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V5KHsU4YJr6+P2ufm4RDU51vFZo=', 'Content-Length': '0'} 2022-09-12T19:06:45.585 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007795af3fa9f42b68-00631f8345-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:45.586 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.587 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.587 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.588 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.588 DEBUG:boto:Method: GET 2022-09-12T19:06:45.589 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.589 DEBUG:boto:Data: 2022-09-12T19:06:45.590 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.591 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:45.592 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.592 DEBUG:boto:Params: {} 2022-09-12T19:06:45.593 DEBUG:boto:Token: None 2022-09-12T19:06:45.593 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.594 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VxZkLQtwwxbn+V8PGPdnMgviBfY= 2022-09-12T19:06:45.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VxZkLQtwwxbn+V8PGPdnMgviBfY=', 'Content-Length': '0'} 2022-09-12T19:06:45.597 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048b46f75d932a9a2-00631f8345-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.642 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.643 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.644 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.644 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.645 DEBUG:boto:Method: GET 2022-09-12T19:06:45.645 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.646 DEBUG:boto:Data: 2022-09-12T19:06:45.646 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.647 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:45.647 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.648 DEBUG:boto:Params: {} 2022-09-12T19:06:45.649 DEBUG:boto:Token: None 2022-09-12T19:06:45.649 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.650 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VxZkLQtwwxbn+V8PGPdnMgviBfY= 2022-09-12T19:06:45.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VxZkLQtwwxbn+V8PGPdnMgviBfY=', 'Content-Length': '0'} 2022-09-12T19:06:45.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000864d4726ddc199ed-00631f8345-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.654 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:45.655 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:45.655 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.656 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.656 DEBUG:boto:Method: GET 2022-09-12T19:06:45.657 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:45.657 DEBUG:boto:Data: 2022-09-12T19:06:45.658 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.659 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:45.659 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.660 DEBUG:boto:Params: {} 2022-09-12T19:06:45.660 DEBUG:boto:Token: None 2022-09-12T19:06:45.661 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/ 2022-09-12T19:06:45.662 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo= 2022-09-12T19:06:45.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo=', 'Content-Length': '0'} 2022-09-12T19:06:45.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f4dbe6d34ed310a-00631f8345-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.667 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:45.668 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:45.669 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:45.670 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.670 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.671 DEBUG:boto:Method: GET 2022-09-12T19:06:45.672 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:45.672 DEBUG:boto:Data: 2022-09-12T19:06:45.673 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.673 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:45.674 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.674 DEBUG:boto:Params: {} 2022-09-12T19:06:45.675 DEBUG:boto:Token: None 2022-09-12T19:06:45.676 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/ 2022-09-12T19:06:45.677 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo= 2022-09-12T19:06:45.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo=', 'Content-Length': '0'} 2022-09-12T19:06:45.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2e620fbf96a4821-00631f8345-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.681 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:45.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:45.683 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.684 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.684 DEBUG:boto:path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:45.685 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:45.685 DEBUG:boto:Method: GET 2022-09-12T19:06:45.686 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:45.686 DEBUG:boto:Data: 2022-09-12T19:06:45.687 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.688 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:45.688 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.689 DEBUG:boto:Params: {} 2022-09-12T19:06:45.689 DEBUG:boto:Token: None 2022-09-12T19:06:45.690 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:45.691 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kOlvd5AorfoiOK61ndxBRoQGx+4= 2022-09-12T19:06:45.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kOlvd5AorfoiOK61ndxBRoQGx+4=', 'Content-Length': '0'} 2022-09-12T19:06:45.694 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Xb5UZAvutXF6GhBCMdUQgjOLi998.hK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db188d25e556b035-00631f8345-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.695 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.696 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.696 DEBUG:boto:path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:45.697 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:45.697 DEBUG:boto:Method: GET 2022-09-12T19:06:45.698 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:45.699 DEBUG:boto:Data: 2022-09-12T19:06:45.699 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.700 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:45.700 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.701 DEBUG:boto:Params: {} 2022-09-12T19:06:45.701 DEBUG:boto:Token: None 2022-09-12T19:06:45.702 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:45.703 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kOlvd5AorfoiOK61ndxBRoQGx+4= 2022-09-12T19:06:45.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kOlvd5AorfoiOK61ndxBRoQGx+4=', 'Content-Length': '0'} 2022-09-12T19:06:45.706 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Xb5UZAvutXF6GhBCMdUQgjOLi998.hK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003cb7ae2a8b38d95c-00631f8345-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.707 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.707 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.708 DEBUG:boto:path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:45.708 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:45.709 DEBUG:boto:Method: HEAD 2022-09-12T19:06:45.709 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:45.710 DEBUG:boto:Data: 2022-09-12T19:06:45.710 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.711 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:45.712 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.712 DEBUG:boto:Params: {} 2022-09-12T19:06:45.713 DEBUG:boto:Token: None 2022-09-12T19:06:45.713 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:45.714 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:40428O0je5rR0ijbIRzxL2ip3jA= 2022-09-12T19:06:45.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:40428O0je5rR0ijbIRzxL2ip3jA=', 'Content-Length': '0'} 2022-09-12T19:06:45.717 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Xb5UZAvutXF6GhBCMdUQgjOLi998.hK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d335dfd49f1fda0f-00631f8345-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.718 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.718 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.719 DEBUG:boto:path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:45.719 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:45.720 DEBUG:boto:Method: HEAD 2022-09-12T19:06:45.720 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:45.721 DEBUG:boto:Data: 2022-09-12T19:06:45.721 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.722 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:45.723 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.723 DEBUG:boto:Params: {} 2022-09-12T19:06:45.724 DEBUG:boto:Token: None 2022-09-12T19:06:45.725 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:45.725 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:40428O0je5rR0ijbIRzxL2ip3jA= 2022-09-12T19:06:45.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:40428O0je5rR0ijbIRzxL2ip3jA=', 'Content-Length': '0'} 2022-09-12T19:06:45.728 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Xb5UZAvutXF6GhBCMdUQgjOLi998.hK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e99523ef568371d0-00631f8345-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:45.729 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.730 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.730 DEBUG:boto:path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:45.731 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:45.732 DEBUG:boto:Method: GET 2022-09-12T19:06:45.732 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:45.733 DEBUG:boto:Data: 2022-09-12T19:06:45.734 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.734 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:45.735 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.735 DEBUG:boto:Params: {} 2022-09-12T19:06:45.736 DEBUG:boto:Token: None 2022-09-12T19:06:45.737 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:45.737 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kOlvd5AorfoiOK61ndxBRoQGx+4= 2022-09-12T19:06:45.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kOlvd5AorfoiOK61ndxBRoQGx+4=', 'Content-Length': '0'} 2022-09-12T19:06:45.740 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Xb5UZAvutXF6GhBCMdUQgjOLi998.hK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a277ad2fb43a1211-00631f8345-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.782 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.783 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.784 DEBUG:boto:path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:45.785 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:45.786 DEBUG:boto:Method: GET 2022-09-12T19:06:45.787 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:45.788 DEBUG:boto:Data: 2022-09-12T19:06:45.789 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.790 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:45.791 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.792 DEBUG:boto:Params: {} 2022-09-12T19:06:45.793 DEBUG:boto:Token: None 2022-09-12T19:06:45.794 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:45.795 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kOlvd5AorfoiOK61ndxBRoQGx+4= 2022-09-12T19:06:45.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kOlvd5AorfoiOK61ndxBRoQGx+4=', 'Content-Length': '0'} 2022-09-12T19:06:45.799 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Xb5UZAvutXF6GhBCMdUQgjOLi998.hK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9779b52d1123104-00631f8345-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.801 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:45.802 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:45.803 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.804 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.805 DEBUG:boto:Method: GET 2022-09-12T19:06:45.805 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:45.806 DEBUG:boto:Data: 2022-09-12T19:06:45.807 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.808 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:45.809 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.810 DEBUG:boto:Params: {} 2022-09-12T19:06:45.812 DEBUG:boto:Token: None 2022-09-12T19:06:45.813 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/ 2022-09-12T19:06:45.814 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo= 2022-09-12T19:06:45.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo=', 'Content-Length': '0'} 2022-09-12T19:06:45.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a2597174b2f0fc3-00631f8345-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.818 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:45.819 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:45.820 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:45.820 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.821 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.821 DEBUG:boto:Method: GET 2022-09-12T19:06:45.822 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:45.823 DEBUG:boto:Data: 2022-09-12T19:06:45.823 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.824 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:45.824 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.825 DEBUG:boto:Params: {} 2022-09-12T19:06:45.825 DEBUG:boto:Token: None 2022-09-12T19:06:45.826 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/ 2022-09-12T19:06:45.826 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo= 2022-09-12T19:06:45.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo=', 'Content-Length': '0'} 2022-09-12T19:06:45.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d6f86a6f424b0b4-00631f8345-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.831 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:45.832 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.832 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.833 DEBUG:boto:Method: HEAD 2022-09-12T19:06:45.833 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:45.834 DEBUG:boto:Data: 2022-09-12T19:06:45.834 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.835 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:45.835 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.835 DEBUG:boto:Params: {} 2022-09-12T19:06:45.836 DEBUG:boto:Token: None 2022-09-12T19:06:45.837 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj 2022-09-12T19:06:45.837 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V5KHsU4YJr6+P2ufm4RDU51vFZo= 2022-09-12T19:06:45.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V5KHsU4YJr6+P2ufm4RDU51vFZo=', 'Content-Length': '0'} 2022-09-12T19:06:45.841 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000443e0c567a412119-00631f8345-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.842 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.843 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.843 DEBUG:boto:Method: HEAD 2022-09-12T19:06:45.844 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:45.844 DEBUG:boto:Data: 2022-09-12T19:06:45.845 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.846 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:45.846 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.847 DEBUG:boto:Params: {} 2022-09-12T19:06:45.848 DEBUG:boto:Token: None 2022-09-12T19:06:45.848 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj 2022-09-12T19:06:45.849 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V5KHsU4YJr6+P2ufm4RDU51vFZo= 2022-09-12T19:06:45.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V5KHsU4YJr6+P2ufm4RDU51vFZo=', 'Content-Length': '0'} 2022-09-12T19:06:45.858 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069d39c59e02ef4cf-00631f8345-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:45.860 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.861 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.861 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.862 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.862 DEBUG:boto:Method: GET 2022-09-12T19:06:45.863 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.864 DEBUG:boto:Data: 2022-09-12T19:06:45.864 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.865 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:45.865 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.866 DEBUG:boto:Params: {} 2022-09-12T19:06:45.866 DEBUG:boto:Token: None 2022-09-12T19:06:45.867 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.867 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VxZkLQtwwxbn+V8PGPdnMgviBfY= 2022-09-12T19:06:45.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VxZkLQtwwxbn+V8PGPdnMgviBfY=', 'Content-Length': '0'} 2022-09-12T19:06:45.870 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a7c7d451c188c79-00631f8345-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.871 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.872 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.872 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.873 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.873 DEBUG:boto:Method: GET 2022-09-12T19:06:45.874 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.874 DEBUG:boto:Data: 2022-09-12T19:06:45.875 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.876 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:45.876 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.877 DEBUG:boto:Params: {} 2022-09-12T19:06:45.877 DEBUG:boto:Token: None 2022-09-12T19:06:45.878 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:45.879 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VxZkLQtwwxbn+V8PGPdnMgviBfY= 2022-09-12T19:06:45.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VxZkLQtwwxbn+V8PGPdnMgviBfY=', 'Content-Length': '0'} 2022-09-12T19:06:45.882 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d7488f165c3eca4-00631f8345-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.922 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:45.923 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:45.923 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.924 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.925 DEBUG:boto:Method: GET 2022-09-12T19:06:45.925 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:45.926 DEBUG:boto:Data: 2022-09-12T19:06:45.927 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.928 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:45.929 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.929 DEBUG:boto:Params: {} 2022-09-12T19:06:45.930 DEBUG:boto:Token: None 2022-09-12T19:06:45.931 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/ 2022-09-12T19:06:45.931 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo= 2022-09-12T19:06:45.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo=', 'Content-Length': '0'} 2022-09-12T19:06:45.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035ac0f6b0fa0f047-00631f8345-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.936 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:45.937 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:45.937 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:45.938 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.939 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:45.939 DEBUG:boto:Method: GET 2022-09-12T19:06:45.940 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:45.941 DEBUG:boto:Data: 2022-09-12T19:06:45.941 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.942 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:45.943 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.944 DEBUG:boto:Params: {} 2022-09-12T19:06:45.944 DEBUG:boto:Token: None 2022-09-12T19:06:45.945 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/ 2022-09-12T19:06:45.946 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo= 2022-09-12T19:06:45.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3GIKGnTHeVO+Kbr2ta6hB6tkpHo=', 'Content-Length': '0'} 2022-09-12T19:06:45.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025f948348a0266ec-00631f8345-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.951 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:45.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:45.953 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.954 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.954 DEBUG:boto:path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:45.955 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:45.956 DEBUG:boto:Method: GET 2022-09-12T19:06:45.956 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:45.957 DEBUG:boto:Data: 2022-09-12T19:06:45.957 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.958 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:45.959 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.960 DEBUG:boto:Params: {} 2022-09-12T19:06:45.962 DEBUG:boto:Token: None 2022-09-12T19:06:45.962 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:45.963 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gH+4psIGrE9VOrQl8UrCHs7FgFo= 2022-09-12T19:06:45.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gH+4psIGrE9VOrQl8UrCHs7FgFo=', 'Content-Length': '0'} 2022-09-12T19:06:45.966 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'ztQRphsNbUdApzTcOcagUJ56uTwo7J2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1f6a63b28600861-00631f8345-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.967 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.968 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.968 DEBUG:boto:path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:45.969 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:45.971 DEBUG:boto:Method: GET 2022-09-12T19:06:45.971 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:45.972 DEBUG:boto:Data: 2022-09-12T19:06:45.972 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.973 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:45.973 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.974 DEBUG:boto:Params: {} 2022-09-12T19:06:45.975 DEBUG:boto:Token: None 2022-09-12T19:06:45.976 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:45.977 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gH+4psIGrE9VOrQl8UrCHs7FgFo= 2022-09-12T19:06:45.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gH+4psIGrE9VOrQl8UrCHs7FgFo=', 'Content-Length': '0'} 2022-09-12T19:06:45.980 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'ztQRphsNbUdApzTcOcagUJ56uTwo7J2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055710ba30ba3245a-00631f8345-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.981 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.981 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.982 DEBUG:boto:path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:45.983 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:45.983 DEBUG:boto:Method: HEAD 2022-09-12T19:06:45.984 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:45.984 DEBUG:boto:Data: 2022-09-12T19:06:45.985 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.985 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:45.986 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.987 DEBUG:boto:Params: {} 2022-09-12T19:06:45.987 DEBUG:boto:Token: None 2022-09-12T19:06:45.988 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:45.988 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cgVmZPPkLX3fDxq3Snv/tQZBuQE= 2022-09-12T19:06:45.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cgVmZPPkLX3fDxq3Snv/tQZBuQE=', 'Content-Length': '0'} 2022-09-12T19:06:45.991 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'ztQRphsNbUdApzTcOcagUJ56uTwo7J2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eeb89964f7decd32-00631f8345-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:45.992 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:45.993 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:45.993 DEBUG:boto:path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:45.994 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:45.994 DEBUG:boto:Method: HEAD 2022-09-12T19:06:45.995 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:45.995 DEBUG:boto:Data: 2022-09-12T19:06:45.996 DEBUG:boto:Headers: {} 2022-09-12T19:06:45.996 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:45.997 DEBUG:boto:Port: 8000 2022-09-12T19:06:45.997 DEBUG:boto:Params: {} 2022-09-12T19:06:45.998 DEBUG:boto:Token: None 2022-09-12T19:06:45.998 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:45 GMT /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:45.999 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cgVmZPPkLX3fDxq3Snv/tQZBuQE= 2022-09-12T19:06:46.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cgVmZPPkLX3fDxq3Snv/tQZBuQE=', 'Content-Length': '0'} 2022-09-12T19:06:46.002 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'ztQRphsNbUdApzTcOcagUJ56uTwo7J2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069ac6073c130b085-00631f8345-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:46.004 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.004 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.005 DEBUG:boto:path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:46.006 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:46.007 DEBUG:boto:Method: GET 2022-09-12T19:06:46.007 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:46.008 DEBUG:boto:Data: 2022-09-12T19:06:46.009 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.009 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:46.010 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.011 DEBUG:boto:Params: {} 2022-09-12T19:06:46.024 DEBUG:boto:Token: None 2022-09-12T19:06:46.025 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:46.026 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Pf1VQUsqrG45tQNOU/PazHh0DgE= 2022-09-12T19:06:46.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Pf1VQUsqrG45tQNOU/PazHh0DgE=', 'Content-Length': '0'} 2022-09-12T19:06:46.029 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'ztQRphsNbUdApzTcOcagUJ56uTwo7J2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0a34deb8017c121-00631f8346-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.070 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.071 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.072 DEBUG:boto:path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:46.072 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:46.073 DEBUG:boto:Method: GET 2022-09-12T19:06:46.074 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:46.074 DEBUG:boto:Data: 2022-09-12T19:06:46.075 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.076 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:46.076 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.077 DEBUG:boto:Params: {} 2022-09-12T19:06:46.077 DEBUG:boto:Token: None 2022-09-12T19:06:46.078 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:46.079 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Pf1VQUsqrG45tQNOU/PazHh0DgE= 2022-09-12T19:06:46.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Pf1VQUsqrG45tQNOU/PazHh0DgE=', 'Content-Length': '0'} 2022-09-12T19:06:46.081 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'ztQRphsNbUdApzTcOcagUJ56uTwo7J2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026d06e08608e916b-00631f8346-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.082 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:46.083 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:46.083 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.084 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.085 DEBUG:boto:Method: GET 2022-09-12T19:06:46.085 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:46.086 DEBUG:boto:Data: 2022-09-12T19:06:46.086 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.087 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:46.087 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.088 DEBUG:boto:Params: {} 2022-09-12T19:06:46.088 DEBUG:boto:Token: None 2022-09-12T19:06:46.089 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/ 2022-09-12T19:06:46.090 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k= 2022-09-12T19:06:46.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k=', 'Content-Length': '0'} 2022-09-12T19:06:46.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075a71940843b0f11-00631f8346-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.094 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:46.095 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:46.095 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:46.096 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.097 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.097 DEBUG:boto:Method: GET 2022-09-12T19:06:46.098 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:46.098 DEBUG:boto:Data: 2022-09-12T19:06:46.099 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.100 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:46.100 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.101 DEBUG:boto:Params: {} 2022-09-12T19:06:46.102 DEBUG:boto:Token: None 2022-09-12T19:06:46.102 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/ 2022-09-12T19:06:46.103 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k= 2022-09-12T19:06:46.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k=', 'Content-Length': '0'} 2022-09-12T19:06:46.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d7c0d91e0cffedf-00631f8346-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.107 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:46.108 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.109 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.109 DEBUG:boto:Method: HEAD 2022-09-12T19:06:46.110 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:46.111 DEBUG:boto:Data: 2022-09-12T19:06:46.111 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.112 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:46.112 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.112 DEBUG:boto:Params: {} 2022-09-12T19:06:46.113 DEBUG:boto:Token: None 2022-09-12T19:06:46.114 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj 2022-09-12T19:06:46.114 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0dlGaR2E3sYEJsKOZ1p/Bmc1xDM= 2022-09-12T19:06:46.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0dlGaR2E3sYEJsKOZ1p/Bmc1xDM=', 'Content-Length': '0'} 2022-09-12T19:06:46.118 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a4220a4edd23b47-00631f8346-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.119 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.120 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.120 DEBUG:boto:Method: HEAD 2022-09-12T19:06:46.121 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:46.121 DEBUG:boto:Data: 2022-09-12T19:06:46.122 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.122 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:46.123 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.124 DEBUG:boto:Params: {} 2022-09-12T19:06:46.125 DEBUG:boto:Token: None 2022-09-12T19:06:46.125 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj 2022-09-12T19:06:46.126 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0dlGaR2E3sYEJsKOZ1p/Bmc1xDM= 2022-09-12T19:06:46.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0dlGaR2E3sYEJsKOZ1p/Bmc1xDM=', 'Content-Length': '0'} 2022-09-12T19:06:46.130 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0d7ba068290e28e-00631f8346-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:46.131 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.132 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.133 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:46.134 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:46.134 DEBUG:boto:Method: GET 2022-09-12T19:06:46.135 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:46.135 DEBUG:boto:Data: 2022-09-12T19:06:46.136 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.136 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:46.137 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.138 DEBUG:boto:Params: {} 2022-09-12T19:06:46.138 DEBUG:boto:Token: None 2022-09-12T19:06:46.139 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:46.140 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q4QeLdotuN+beITpwNRlRdH67qM= 2022-09-12T19:06:46.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q4QeLdotuN+beITpwNRlRdH67qM=', 'Content-Length': '0'} 2022-09-12T19:06:46.143 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f73ae26c98e6c897-00631f8346-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.144 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.145 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.145 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:46.146 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:46.147 DEBUG:boto:Method: GET 2022-09-12T19:06:46.147 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:46.148 DEBUG:boto:Data: 2022-09-12T19:06:46.149 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.150 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:46.150 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.151 DEBUG:boto:Params: {} 2022-09-12T19:06:46.152 DEBUG:boto:Token: None 2022-09-12T19:06:46.152 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:46.153 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q4QeLdotuN+beITpwNRlRdH67qM= 2022-09-12T19:06:46.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q4QeLdotuN+beITpwNRlRdH67qM=', 'Content-Length': '0'} 2022-09-12T19:06:46.156 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1d52374f8c3e26b-00631f8346-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.158 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:46.158 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:46.159 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.160 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.160 DEBUG:boto:Method: GET 2022-09-12T19:06:46.161 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:46.161 DEBUG:boto:Data: 2022-09-12T19:06:46.162 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.162 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:46.163 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.164 DEBUG:boto:Params: {} 2022-09-12T19:06:46.164 DEBUG:boto:Token: None 2022-09-12T19:06:46.165 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/ 2022-09-12T19:06:46.165 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k= 2022-09-12T19:06:46.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k=', 'Content-Length': '0'} 2022-09-12T19:06:46.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a66b1f4d6bfe6ccb-00631f8346-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.170 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:46.171 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:46.172 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:46.172 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.173 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.174 DEBUG:boto:Method: GET 2022-09-12T19:06:46.174 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:46.175 DEBUG:boto:Data: 2022-09-12T19:06:46.175 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.176 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:46.177 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.177 DEBUG:boto:Params: {} 2022-09-12T19:06:46.178 DEBUG:boto:Token: None 2022-09-12T19:06:46.178 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/ 2022-09-12T19:06:46.179 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k= 2022-09-12T19:06:46.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k=', 'Content-Length': '0'} 2022-09-12T19:06:46.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000306a6842f21d2153-00631f8346-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.184 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:46.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:46.185 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.186 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.187 DEBUG:boto:path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:46.187 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:46.188 DEBUG:boto:Method: GET 2022-09-12T19:06:46.188 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:46.189 DEBUG:boto:Data: 2022-09-12T19:06:46.189 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.190 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:46.190 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.191 DEBUG:boto:Params: {} 2022-09-12T19:06:46.192 DEBUG:boto:Token: None 2022-09-12T19:06:46.192 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:46.193 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l938w3NniEwhPVGLDikfW9IbOjg= 2022-09-12T19:06:46.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l938w3NniEwhPVGLDikfW9IbOjg=', 'Content-Length': '0'} 2022-09-12T19:06:46.196 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Katr07t9mQV0b0gx44aB8d9VyeAhMS1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066ef8f7857ab9283-00631f8346-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.196 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.197 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.198 DEBUG:boto:path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:46.198 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:46.199 DEBUG:boto:Method: GET 2022-09-12T19:06:46.200 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:46.200 DEBUG:boto:Data: 2022-09-12T19:06:46.201 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.202 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:46.202 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.203 DEBUG:boto:Params: {} 2022-09-12T19:06:46.203 DEBUG:boto:Token: None 2022-09-12T19:06:46.204 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:46.205 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l938w3NniEwhPVGLDikfW9IbOjg= 2022-09-12T19:06:46.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l938w3NniEwhPVGLDikfW9IbOjg=', 'Content-Length': '0'} 2022-09-12T19:06:46.208 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Katr07t9mQV0b0gx44aB8d9VyeAhMS1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071a966428a4d538d-00631f8346-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.250 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.251 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.252 DEBUG:boto:path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:46.252 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:46.253 DEBUG:boto:Method: HEAD 2022-09-12T19:06:46.254 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:46.254 DEBUG:boto:Data: 2022-09-12T19:06:46.255 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.256 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:46.256 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.257 DEBUG:boto:Params: {} 2022-09-12T19:06:46.257 DEBUG:boto:Token: None 2022-09-12T19:06:46.258 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:46.259 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y7Lr0QyjWNHnYGG5UtEvzC43yMU= 2022-09-12T19:06:46.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y7Lr0QyjWNHnYGG5UtEvzC43yMU=', 'Content-Length': '0'} 2022-09-12T19:06:46.262 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Katr07t9mQV0b0gx44aB8d9VyeAhMS1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eed34fca2aed357f-00631f8346-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.262 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.263 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.264 DEBUG:boto:path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:46.264 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:46.265 DEBUG:boto:Method: HEAD 2022-09-12T19:06:46.266 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:46.266 DEBUG:boto:Data: 2022-09-12T19:06:46.267 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.268 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:46.269 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.269 DEBUG:boto:Params: {} 2022-09-12T19:06:46.270 DEBUG:boto:Token: None 2022-09-12T19:06:46.270 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:46.271 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y7Lr0QyjWNHnYGG5UtEvzC43yMU= 2022-09-12T19:06:46.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y7Lr0QyjWNHnYGG5UtEvzC43yMU=', 'Content-Length': '0'} 2022-09-12T19:06:46.274 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Katr07t9mQV0b0gx44aB8d9VyeAhMS1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008152b8a63668a22d-00631f8346-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:46.276 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.276 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.277 DEBUG:boto:path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:46.279 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:46.280 DEBUG:boto:Method: GET 2022-09-12T19:06:46.280 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:46.281 DEBUG:boto:Data: 2022-09-12T19:06:46.282 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.283 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:46.283 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.284 DEBUG:boto:Params: {} 2022-09-12T19:06:46.285 DEBUG:boto:Token: None 2022-09-12T19:06:46.286 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:46.287 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l938w3NniEwhPVGLDikfW9IbOjg= 2022-09-12T19:06:46.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l938w3NniEwhPVGLDikfW9IbOjg=', 'Content-Length': '0'} 2022-09-12T19:06:46.291 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Katr07t9mQV0b0gx44aB8d9VyeAhMS1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ef0769fde3781d1-00631f8346-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.292 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.293 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.294 DEBUG:boto:path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:46.295 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:46.296 DEBUG:boto:Method: GET 2022-09-12T19:06:46.296 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:46.297 DEBUG:boto:Data: 2022-09-12T19:06:46.298 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.299 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:46.300 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.300 DEBUG:boto:Params: {} 2022-09-12T19:06:46.301 DEBUG:boto:Token: None 2022-09-12T19:06:46.302 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:46.303 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l938w3NniEwhPVGLDikfW9IbOjg= 2022-09-12T19:06:46.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l938w3NniEwhPVGLDikfW9IbOjg=', 'Content-Length': '0'} 2022-09-12T19:06:46.306 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Katr07t9mQV0b0gx44aB8d9VyeAhMS1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000538aa2f9977a24e8-00631f8346-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.307 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:46.308 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:46.308 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.309 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.310 DEBUG:boto:Method: GET 2022-09-12T19:06:46.311 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:46.312 DEBUG:boto:Data: 2022-09-12T19:06:46.313 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.314 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:46.315 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.316 DEBUG:boto:Params: {} 2022-09-12T19:06:46.317 DEBUG:boto:Token: None 2022-09-12T19:06:46.319 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/ 2022-09-12T19:06:46.320 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k= 2022-09-12T19:06:46.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k=', 'Content-Length': '0'} 2022-09-12T19:06:46.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bab3edbf0aae42e8-00631f8346-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.366 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:46.368 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:46.369 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:46.370 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.370 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.371 DEBUG:boto:Method: GET 2022-09-12T19:06:46.372 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:46.373 DEBUG:boto:Data: 2022-09-12T19:06:46.374 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.375 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:46.376 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.376 DEBUG:boto:Params: {} 2022-09-12T19:06:46.377 DEBUG:boto:Token: None 2022-09-12T19:06:46.378 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/ 2022-09-12T19:06:46.379 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k= 2022-09-12T19:06:46.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k=', 'Content-Length': '0'} 2022-09-12T19:06:46.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049e329e69a1c6283-00631f8346-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.385 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:46.385 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.386 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.387 DEBUG:boto:Method: HEAD 2022-09-12T19:06:46.387 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:46.388 DEBUG:boto:Data: 2022-09-12T19:06:46.388 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.389 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:46.389 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.390 DEBUG:boto:Params: {} 2022-09-12T19:06:46.390 DEBUG:boto:Token: None 2022-09-12T19:06:46.391 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj 2022-09-12T19:06:46.392 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0dlGaR2E3sYEJsKOZ1p/Bmc1xDM= 2022-09-12T19:06:46.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0dlGaR2E3sYEJsKOZ1p/Bmc1xDM=', 'Content-Length': '0'} 2022-09-12T19:06:46.396 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3fc65fbaf27f301-00631f8346-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.396 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.397 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.398 DEBUG:boto:Method: HEAD 2022-09-12T19:06:46.398 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:46.399 DEBUG:boto:Data: 2022-09-12T19:06:46.399 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.400 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:46.400 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.401 DEBUG:boto:Params: {} 2022-09-12T19:06:46.402 DEBUG:boto:Token: None 2022-09-12T19:06:46.403 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj 2022-09-12T19:06:46.403 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0dlGaR2E3sYEJsKOZ1p/Bmc1xDM= 2022-09-12T19:06:46.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0dlGaR2E3sYEJsKOZ1p/Bmc1xDM=', 'Content-Length': '0'} 2022-09-12T19:06:46.408 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef763a0db7e57829-00631f8346-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:46.409 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.410 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.411 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:46.411 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:46.412 DEBUG:boto:Method: GET 2022-09-12T19:06:46.412 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:46.413 DEBUG:boto:Data: 2022-09-12T19:06:46.414 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.414 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:46.415 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.415 DEBUG:boto:Params: {} 2022-09-12T19:06:46.416 DEBUG:boto:Token: None 2022-09-12T19:06:46.417 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:46.417 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q4QeLdotuN+beITpwNRlRdH67qM= 2022-09-12T19:06:46.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q4QeLdotuN+beITpwNRlRdH67qM=', 'Content-Length': '0'} 2022-09-12T19:06:46.420 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c718279969425b94-00631f8346-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.422 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.423 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.424 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:46.425 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:46.426 DEBUG:boto:Method: GET 2022-09-12T19:06:46.426 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:46.427 DEBUG:boto:Data: 2022-09-12T19:06:46.428 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.428 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:46.429 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.429 DEBUG:boto:Params: {} 2022-09-12T19:06:46.430 DEBUG:boto:Token: None 2022-09-12T19:06:46.431 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:46.431 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q4QeLdotuN+beITpwNRlRdH67qM= 2022-09-12T19:06:46.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q4QeLdotuN+beITpwNRlRdH67qM=', 'Content-Length': '0'} 2022-09-12T19:06:46.434 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d145c2b930ef143-00631f8346-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.478 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:46.479 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:46.480 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.481 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.481 DEBUG:boto:Method: GET 2022-09-12T19:06:46.482 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:46.482 DEBUG:boto:Data: 2022-09-12T19:06:46.483 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.484 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:46.485 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.485 DEBUG:boto:Params: {} 2022-09-12T19:06:46.486 DEBUG:boto:Token: None 2022-09-12T19:06:46.486 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/ 2022-09-12T19:06:46.487 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k= 2022-09-12T19:06:46.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k=', 'Content-Length': '0'} 2022-09-12T19:06:46.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082e3a29151ff82b6-00631f8346-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.492 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:46.494 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:46.495 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:46.496 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.497 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.498 DEBUG:boto:Method: GET 2022-09-12T19:06:46.499 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:46.500 DEBUG:boto:Data: 2022-09-12T19:06:46.501 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.502 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:46.502 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.503 DEBUG:boto:Params: {} 2022-09-12T19:06:46.504 DEBUG:boto:Token: None 2022-09-12T19:06:46.504 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/ 2022-09-12T19:06:46.505 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k= 2022-09-12T19:06:46.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k=', 'Content-Length': '0'} 2022-09-12T19:06:46.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3bff581d0219535-00631f8346-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.510 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:46.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:46.511 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.512 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.513 DEBUG:boto:path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:46.513 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:46.514 DEBUG:boto:Method: GET 2022-09-12T19:06:46.515 DEBUG:boto:Path: /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:46.516 DEBUG:boto:Data: 2022-09-12T19:06:46.517 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.518 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:46.518 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.519 DEBUG:boto:Params: {} 2022-09-12T19:06:46.520 DEBUG:boto:Token: None 2022-09-12T19:06:46.520 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:46.521 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IF7wh8kww9o3lskT0EUz8l0jukc= 2022-09-12T19:06:46.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IF7wh8kww9o3lskT0EUz8l0jukc=', 'Content-Length': '0'} 2022-09-12T19:06:46.524 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'UBsYcCA94H55K4ucww5vXNAxuAMKNSG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e7ecb4892ae3391-00631f8346-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.570 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.571 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.571 DEBUG:boto:path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:46.572 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:46.572 DEBUG:boto:Method: GET 2022-09-12T19:06:46.573 DEBUG:boto:Path: /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:46.574 DEBUG:boto:Data: 2022-09-12T19:06:46.574 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.575 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:46.575 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.576 DEBUG:boto:Params: {} 2022-09-12T19:06:46.576 DEBUG:boto:Token: None 2022-09-12T19:06:46.577 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:46.578 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IF7wh8kww9o3lskT0EUz8l0jukc= 2022-09-12T19:06:46.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IF7wh8kww9o3lskT0EUz8l0jukc=', 'Content-Length': '0'} 2022-09-12T19:06:46.581 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'UBsYcCA94H55K4ucww5vXNAxuAMKNSG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000040e3fd596c8d0fe1-00631f8346-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.582 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.582 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.583 DEBUG:boto:path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:46.584 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:46.584 DEBUG:boto:Method: HEAD 2022-09-12T19:06:46.585 DEBUG:boto:Path: /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:46.586 DEBUG:boto:Data: 2022-09-12T19:06:46.587 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.587 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:46.588 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.589 DEBUG:boto:Params: {} 2022-09-12T19:06:46.589 DEBUG:boto:Token: None 2022-09-12T19:06:46.590 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:46.591 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LCJEMgOp02UKQMx0nQCIlGPdl28= 2022-09-12T19:06:46.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LCJEMgOp02UKQMx0nQCIlGPdl28=', 'Content-Length': '0'} 2022-09-12T19:06:46.595 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'UBsYcCA94H55K4ucww5vXNAxuAMKNSG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c0fc8f630679fc8-00631f8346-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.596 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.596 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.597 DEBUG:boto:path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:46.598 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:46.598 DEBUG:boto:Method: HEAD 2022-09-12T19:06:46.599 DEBUG:boto:Path: /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:46.599 DEBUG:boto:Data: 2022-09-12T19:06:46.600 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.601 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:46.601 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.602 DEBUG:boto:Params: {} 2022-09-12T19:06:46.602 DEBUG:boto:Token: None 2022-09-12T19:06:46.603 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:46.604 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LCJEMgOp02UKQMx0nQCIlGPdl28= 2022-09-12T19:06:46.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LCJEMgOp02UKQMx0nQCIlGPdl28=', 'Content-Length': '0'} 2022-09-12T19:06:46.607 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'UBsYcCA94H55K4ucww5vXNAxuAMKNSG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2337628785a38e5-00631f8346-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:46.610 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.611 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.612 DEBUG:boto:path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:46.613 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:46.614 DEBUG:boto:Method: GET 2022-09-12T19:06:46.615 DEBUG:boto:Path: /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:46.615 DEBUG:boto:Data: 2022-09-12T19:06:46.616 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.617 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:46.618 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.619 DEBUG:boto:Params: {} 2022-09-12T19:06:46.620 DEBUG:boto:Token: None 2022-09-12T19:06:46.621 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:46.622 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IF7wh8kww9o3lskT0EUz8l0jukc= 2022-09-12T19:06:46.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IF7wh8kww9o3lskT0EUz8l0jukc=', 'Content-Length': '0'} 2022-09-12T19:06:46.624 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'UBsYcCA94H55K4ucww5vXNAxuAMKNSG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093073d5fb482401b-00631f8346-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.625 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.626 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.627 DEBUG:boto:path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:46.627 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:46.628 DEBUG:boto:Method: GET 2022-09-12T19:06:46.628 DEBUG:boto:Path: /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:46.629 DEBUG:boto:Data: 2022-09-12T19:06:46.629 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.630 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:46.630 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.631 DEBUG:boto:Params: {} 2022-09-12T19:06:46.631 DEBUG:boto:Token: None 2022-09-12T19:06:46.632 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:46.633 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IF7wh8kww9o3lskT0EUz8l0jukc= 2022-09-12T19:06:46.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IF7wh8kww9o3lskT0EUz8l0jukc=', 'Content-Length': '0'} 2022-09-12T19:06:46.636 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'UBsYcCA94H55K4ucww5vXNAxuAMKNSG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009aaf4771683c2584-00631f8346-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.638 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:46.639 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:46.639 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.640 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.640 DEBUG:boto:Method: GET 2022-09-12T19:06:46.641 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:46.641 DEBUG:boto:Data: 2022-09-12T19:06:46.642 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.643 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:46.643 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.644 DEBUG:boto:Params: {} 2022-09-12T19:06:46.644 DEBUG:boto:Token: None 2022-09-12T19:06:46.645 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/ 2022-09-12T19:06:46.646 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k= 2022-09-12T19:06:46.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k=', 'Content-Length': '0'} 2022-09-12T19:06:46.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d125600f14f8d85a-00631f8346-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.650 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:46.651 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:46.651 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:46.652 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.652 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.653 DEBUG:boto:Method: GET 2022-09-12T19:06:46.654 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:46.654 DEBUG:boto:Data: 2022-09-12T19:06:46.655 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.655 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:46.656 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.656 DEBUG:boto:Params: {} 2022-09-12T19:06:46.657 DEBUG:boto:Token: None 2022-09-12T19:06:46.657 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/ 2022-09-12T19:06:46.658 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k= 2022-09-12T19:06:46.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k=', 'Content-Length': '0'} 2022-09-12T19:06:46.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075a2368602cc15a3-00631f8346-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.702 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:46.703 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.704 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.704 DEBUG:boto:Method: HEAD 2022-09-12T19:06:46.705 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:46.705 DEBUG:boto:Data: 2022-09-12T19:06:46.706 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.706 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:46.707 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.707 DEBUG:boto:Params: {} 2022-09-12T19:06:46.708 DEBUG:boto:Token: None 2022-09-12T19:06:46.708 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj 2022-09-12T19:06:46.709 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0dlGaR2E3sYEJsKOZ1p/Bmc1xDM= 2022-09-12T19:06:46.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0dlGaR2E3sYEJsKOZ1p/Bmc1xDM=', 'Content-Length': '0'} 2022-09-12T19:06:46.714 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000529b3644cda45a46-00631f8346-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.715 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.716 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.716 DEBUG:boto:Method: HEAD 2022-09-12T19:06:46.717 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:46.717 DEBUG:boto:Data: 2022-09-12T19:06:46.718 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.719 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:46.719 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.720 DEBUG:boto:Params: {} 2022-09-12T19:06:46.721 DEBUG:boto:Token: None 2022-09-12T19:06:46.721 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj 2022-09-12T19:06:46.722 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0dlGaR2E3sYEJsKOZ1p/Bmc1xDM= 2022-09-12T19:06:46.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0dlGaR2E3sYEJsKOZ1p/Bmc1xDM=', 'Content-Length': '0'} 2022-09-12T19:06:46.726 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e4b7fa2e424b56f-00631f8346-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:46.728 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.728 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.729 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:46.730 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:46.730 DEBUG:boto:Method: GET 2022-09-12T19:06:46.731 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:46.731 DEBUG:boto:Data: 2022-09-12T19:06:46.732 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.732 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:46.733 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.734 DEBUG:boto:Params: {} 2022-09-12T19:06:46.734 DEBUG:boto:Token: None 2022-09-12T19:06:46.735 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:46.735 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q4QeLdotuN+beITpwNRlRdH67qM= 2022-09-12T19:06:46.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q4QeLdotuN+beITpwNRlRdH67qM=', 'Content-Length': '0'} 2022-09-12T19:06:46.738 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e82c1d73affbced-00631f8346-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.782 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.783 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.783 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:46.784 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:46.785 DEBUG:boto:Method: GET 2022-09-12T19:06:46.785 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:46.786 DEBUG:boto:Data: 2022-09-12T19:06:46.787 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.787 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:46.788 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.789 DEBUG:boto:Params: {} 2022-09-12T19:06:46.789 DEBUG:boto:Token: None 2022-09-12T19:06:46.790 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:46.790 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q4QeLdotuN+beITpwNRlRdH67qM= 2022-09-12T19:06:46.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q4QeLdotuN+beITpwNRlRdH67qM=', 'Content-Length': '0'} 2022-09-12T19:06:46.793 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a1beca3962ad0a9-00631f8346-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.795 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:46.795 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:46.796 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.797 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.797 DEBUG:boto:Method: GET 2022-09-12T19:06:46.798 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:46.798 DEBUG:boto:Data: 2022-09-12T19:06:46.799 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.800 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:46.800 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.801 DEBUG:boto:Params: {} 2022-09-12T19:06:46.801 DEBUG:boto:Token: None 2022-09-12T19:06:46.802 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/ 2022-09-12T19:06:46.803 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k= 2022-09-12T19:06:46.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k=', 'Content-Length': '0'} 2022-09-12T19:06:46.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082a8e4beb1e1ec21-00631f8346-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.807 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:46.808 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:46.808 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:46.809 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.810 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.810 DEBUG:boto:Method: GET 2022-09-12T19:06:46.811 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:46.811 DEBUG:boto:Data: 2022-09-12T19:06:46.812 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.812 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:46.813 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.814 DEBUG:boto:Params: {} 2022-09-12T19:06:46.814 DEBUG:boto:Token: None 2022-09-12T19:06:46.815 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/ 2022-09-12T19:06:46.816 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k= 2022-09-12T19:06:46.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k=', 'Content-Length': '0'} 2022-09-12T19:06:46.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cd1e9a6bf406bac-00631f8346-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.820 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:46.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:46.821 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.822 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.822 DEBUG:boto:path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:46.823 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:46.824 DEBUG:boto:Method: GET 2022-09-12T19:06:46.824 DEBUG:boto:Path: /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:46.825 DEBUG:boto:Data: 2022-09-12T19:06:46.825 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.826 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:46.827 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.828 DEBUG:boto:Params: {} 2022-09-12T19:06:46.828 DEBUG:boto:Token: None 2022-09-12T19:06:46.829 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:46.830 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eRejNeS8LyyZVu9R6ofwlj7he28= 2022-09-12T19:06:46.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eRejNeS8LyyZVu9R6ofwlj7he28=', 'Content-Length': '0'} 2022-09-12T19:06:46.833 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c28c372b7cc68d1e-00631f8346-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.834 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.834 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.835 DEBUG:boto:path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:46.836 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:46.836 DEBUG:boto:Method: GET 2022-09-12T19:06:46.837 DEBUG:boto:Path: /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:46.837 DEBUG:boto:Data: 2022-09-12T19:06:46.838 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.838 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:46.839 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.839 DEBUG:boto:Params: {} 2022-09-12T19:06:46.840 DEBUG:boto:Token: None 2022-09-12T19:06:46.841 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:46.841 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eRejNeS8LyyZVu9R6ofwlj7he28= 2022-09-12T19:06:46.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eRejNeS8LyyZVu9R6ofwlj7he28=', 'Content-Length': '0'} 2022-09-12T19:06:46.844 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad6122fe52ff1819-00631f8346-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.845 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.846 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.846 DEBUG:boto:path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:46.847 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:46.848 DEBUG:boto:Method: HEAD 2022-09-12T19:06:46.848 DEBUG:boto:Path: /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:46.849 DEBUG:boto:Data: 2022-09-12T19:06:46.850 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.850 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:46.851 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.851 DEBUG:boto:Params: {} 2022-09-12T19:06:46.852 DEBUG:boto:Token: None 2022-09-12T19:06:46.853 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:46.854 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5BX5Hoh8wHKOU/Lt0pAmKOYzycE= 2022-09-12T19:06:46.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5BX5Hoh8wHKOU/Lt0pAmKOYzycE=', 'Content-Length': '0'} 2022-09-12T19:06:46.857 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034574ae6cc9094c2-00631f8346-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.859 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.860 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.861 DEBUG:boto:path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:46.862 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:46.863 DEBUG:boto:Method: HEAD 2022-09-12T19:06:46.864 DEBUG:boto:Path: /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:46.864 DEBUG:boto:Data: 2022-09-12T19:06:46.865 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.866 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:46.867 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.868 DEBUG:boto:Params: {} 2022-09-12T19:06:46.870 DEBUG:boto:Token: None 2022-09-12T19:06:46.871 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:46.872 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5BX5Hoh8wHKOU/Lt0pAmKOYzycE= 2022-09-12T19:06:46.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5BX5Hoh8wHKOU/Lt0pAmKOYzycE=', 'Content-Length': '0'} 2022-09-12T19:06:46.875 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b53782aec9ee1aa8-00631f8346-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:46.877 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.878 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.879 DEBUG:boto:path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:46.880 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:46.881 DEBUG:boto:Method: GET 2022-09-12T19:06:46.882 DEBUG:boto:Path: /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:46.883 DEBUG:boto:Data: 2022-09-12T19:06:46.884 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.885 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:46.886 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.887 DEBUG:boto:Params: {} 2022-09-12T19:06:46.888 DEBUG:boto:Token: None 2022-09-12T19:06:46.889 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:46.891 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eRejNeS8LyyZVu9R6ofwlj7he28= 2022-09-12T19:06:46.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eRejNeS8LyyZVu9R6ofwlj7he28=', 'Content-Length': '0'} 2022-09-12T19:06:46.894 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000814305662925119d-00631f8346-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.896 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.897 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.898 DEBUG:boto:path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:46.899 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:46.900 DEBUG:boto:Method: GET 2022-09-12T19:06:46.901 DEBUG:boto:Path: /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:46.902 DEBUG:boto:Data: 2022-09-12T19:06:46.903 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.904 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:46.905 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.906 DEBUG:boto:Params: {} 2022-09-12T19:06:46.907 DEBUG:boto:Token: None 2022-09-12T19:06:46.909 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:46.910 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eRejNeS8LyyZVu9R6ofwlj7he28= 2022-09-12T19:06:46.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eRejNeS8LyyZVu9R6ofwlj7he28=', 'Content-Length': '0'} 2022-09-12T19:06:46.914 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf4020761928d71e-00631f8346-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.915 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:46.916 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:46.917 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.918 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.919 DEBUG:boto:Method: GET 2022-09-12T19:06:46.920 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:46.921 DEBUG:boto:Data: 2022-09-12T19:06:46.923 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.923 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:46.924 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.925 DEBUG:boto:Params: {} 2022-09-12T19:06:46.926 DEBUG:boto:Token: None 2022-09-12T19:06:46.928 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/ 2022-09-12T19:06:46.929 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k= 2022-09-12T19:06:46.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k=', 'Content-Length': '0'} 2022-09-12T19:06:46.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be782ba98d176db4-00631f8346-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.974 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:46.976 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:46.976 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:46.977 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.977 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:46.978 DEBUG:boto:Method: GET 2022-09-12T19:06:46.978 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:46.979 DEBUG:boto:Data: 2022-09-12T19:06:46.979 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.980 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:46.981 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.981 DEBUG:boto:Params: {} 2022-09-12T19:06:46.982 DEBUG:boto:Token: None 2022-09-12T19:06:46.983 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/ 2022-09-12T19:06:46.983 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k= 2022-09-12T19:06:46.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Et1/BgBR7L/B9Cu2Fk38RIBUz0k=', 'Content-Length': '0'} 2022-09-12T19:06:46.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c303e0755b2306b-00631f8346-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:46.993 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:46.994 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:46.995 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:46.995 DEBUG:boto:Method: HEAD 2022-09-12T19:06:46.996 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:46.996 DEBUG:boto:Data: 2022-09-12T19:06:46.997 DEBUG:boto:Headers: {} 2022-09-12T19:06:46.997 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:46.998 DEBUG:boto:Port: 8000 2022-09-12T19:06:46.998 DEBUG:boto:Params: {} 2022-09-12T19:06:46.999 DEBUG:boto:Token: None 2022-09-12T19:06:47.000 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:46 GMT /jdpiak-19/obj 2022-09-12T19:06:47.000 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0dlGaR2E3sYEJsKOZ1p/Bmc1xDM= 2022-09-12T19:06:47.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0dlGaR2E3sYEJsKOZ1p/Bmc1xDM=', 'Content-Length': '0'} 2022-09-12T19:06:47.005 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003457b523030bfe1a-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.006 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.006 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.007 DEBUG:boto:Method: HEAD 2022-09-12T19:06:47.008 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:47.008 DEBUG:boto:Data: 2022-09-12T19:06:47.009 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.009 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.009 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.010 DEBUG:boto:Params: {} 2022-09-12T19:06:47.011 DEBUG:boto:Token: None 2022-09-12T19:06:47.011 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj 2022-09-12T19:06:47.012 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yGDO47MNel40xy2FVu+i46FaOVA= 2022-09-12T19:06:47.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yGDO47MNel40xy2FVu+i46FaOVA=', 'Content-Length': '0'} 2022-09-12T19:06:47.016 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091ca88f9c8a5e75a-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:47.017 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.018 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.019 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.019 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.020 DEBUG:boto:Method: GET 2022-09-12T19:06:47.020 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.021 DEBUG:boto:Data: 2022-09-12T19:06:47.022 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.022 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.023 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.023 DEBUG:boto:Params: {} 2022-09-12T19:06:47.024 DEBUG:boto:Token: None 2022-09-12T19:06:47.025 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.027 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eSB4MgxWXpbnFC/jznLI/4xTiSw= 2022-09-12T19:06:47.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eSB4MgxWXpbnFC/jznLI/4xTiSw=', 'Content-Length': '0'} 2022-09-12T19:06:47.031 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001cca9948ff010948-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.031 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.032 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.033 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.033 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.034 DEBUG:boto:Method: GET 2022-09-12T19:06:47.034 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.035 DEBUG:boto:Data: 2022-09-12T19:06:47.035 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.036 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.036 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.037 DEBUG:boto:Params: {} 2022-09-12T19:06:47.037 DEBUG:boto:Token: None 2022-09-12T19:06:47.038 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.039 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eSB4MgxWXpbnFC/jznLI/4xTiSw= 2022-09-12T19:06:47.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eSB4MgxWXpbnFC/jznLI/4xTiSw=', 'Content-Length': '0'} 2022-09-12T19:06:47.041 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f0c11b9a690b5db-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.082 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:47.083 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:47.084 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.085 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.085 DEBUG:boto:Method: GET 2022-09-12T19:06:47.086 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:47.087 DEBUG:boto:Data: 2022-09-12T19:06:47.087 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.088 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.088 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.089 DEBUG:boto:Params: {} 2022-09-12T19:06:47.089 DEBUG:boto:Token: None 2022-09-12T19:06:47.090 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/ 2022-09-12T19:06:47.091 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y= 2022-09-12T19:06:47.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y=', 'Content-Length': '0'} 2022-09-12T19:06:47.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5d235e16059637f-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.095 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:47.096 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:47.096 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:47.097 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.098 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.098 DEBUG:boto:Method: GET 2022-09-12T19:06:47.099 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:47.099 DEBUG:boto:Data: 2022-09-12T19:06:47.100 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.100 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.101 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.101 DEBUG:boto:Params: {} 2022-09-12T19:06:47.102 DEBUG:boto:Token: None 2022-09-12T19:06:47.103 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/ 2022-09-12T19:06:47.103 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y= 2022-09-12T19:06:47.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y=', 'Content-Length': '0'} 2022-09-12T19:06:47.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be44a9df22c94976-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.108 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:47.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:47.110 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.111 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.111 DEBUG:boto:path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:47.112 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:47.113 DEBUG:boto:Method: GET 2022-09-12T19:06:47.113 DEBUG:boto:Path: /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:47.114 DEBUG:boto:Data: 2022-09-12T19:06:47.115 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.116 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.116 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.117 DEBUG:boto:Params: {} 2022-09-12T19:06:47.118 DEBUG:boto:Token: None 2022-09-12T19:06:47.118 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:47.119 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Xa+0z1yHXQAwmMSOJn+vyPLr0no= 2022-09-12T19:06:47.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Xa+0z1yHXQAwmMSOJn+vyPLr0no=', 'Content-Length': '0'} 2022-09-12T19:06:47.124 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', '.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000710aa5e69ffaa452-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.125 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.125 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.126 DEBUG:boto:path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:47.127 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:47.127 DEBUG:boto:Method: GET 2022-09-12T19:06:47.128 DEBUG:boto:Path: /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:47.129 DEBUG:boto:Data: 2022-09-12T19:06:47.129 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.130 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.130 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.131 DEBUG:boto:Params: {} 2022-09-12T19:06:47.132 DEBUG:boto:Token: None 2022-09-12T19:06:47.132 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:47.133 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Xa+0z1yHXQAwmMSOJn+vyPLr0no= 2022-09-12T19:06:47.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Xa+0z1yHXQAwmMSOJn+vyPLr0no=', 'Content-Length': '0'} 2022-09-12T19:06:47.136 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', '.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c149fe6fbc99f17b-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.138 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.139 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.139 DEBUG:boto:path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:47.140 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:47.141 DEBUG:boto:Method: HEAD 2022-09-12T19:06:47.142 DEBUG:boto:Path: /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:47.143 DEBUG:boto:Data: 2022-09-12T19:06:47.143 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.144 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.145 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.146 DEBUG:boto:Params: {} 2022-09-12T19:06:47.147 DEBUG:boto:Token: None 2022-09-12T19:06:47.148 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:47.149 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:i7aFLmcEdOuqupqU3/81N7kHesg= 2022-09-12T19:06:47.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:i7aFLmcEdOuqupqU3/81N7kHesg=', 'Content-Length': '0'} 2022-09-12T19:06:47.152 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', '.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc7f0dc2c0c83607-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.153 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.154 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.155 DEBUG:boto:path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:47.156 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:47.157 DEBUG:boto:Method: HEAD 2022-09-12T19:06:47.157 DEBUG:boto:Path: /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:47.158 DEBUG:boto:Data: 2022-09-12T19:06:47.159 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.160 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.160 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.161 DEBUG:boto:Params: {} 2022-09-12T19:06:47.162 DEBUG:boto:Token: None 2022-09-12T19:06:47.162 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:47.163 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:i7aFLmcEdOuqupqU3/81N7kHesg= 2022-09-12T19:06:47.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:i7aFLmcEdOuqupqU3/81N7kHesg=', 'Content-Length': '0'} 2022-09-12T19:06:47.166 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', '.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b9d4bb29935efb9-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:47.167 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.167 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.168 DEBUG:boto:path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:47.169 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:47.169 DEBUG:boto:Method: GET 2022-09-12T19:06:47.170 DEBUG:boto:Path: /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:47.171 DEBUG:boto:Data: 2022-09-12T19:06:47.171 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.172 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.173 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.173 DEBUG:boto:Params: {} 2022-09-12T19:06:47.174 DEBUG:boto:Token: None 2022-09-12T19:06:47.174 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:47.175 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Xa+0z1yHXQAwmMSOJn+vyPLr0no= 2022-09-12T19:06:47.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Xa+0z1yHXQAwmMSOJn+vyPLr0no=', 'Content-Length': '0'} 2022-09-12T19:06:47.178 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', '.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000758a3a2fb5bd22af-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.218 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.219 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.219 DEBUG:boto:path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:47.220 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:47.220 DEBUG:boto:Method: GET 2022-09-12T19:06:47.221 DEBUG:boto:Path: /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:47.221 DEBUG:boto:Data: 2022-09-12T19:06:47.222 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.223 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.223 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.224 DEBUG:boto:Params: {} 2022-09-12T19:06:47.225 DEBUG:boto:Token: None 2022-09-12T19:06:47.225 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:47.226 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Xa+0z1yHXQAwmMSOJn+vyPLr0no= 2022-09-12T19:06:47.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Xa+0z1yHXQAwmMSOJn+vyPLr0no=', 'Content-Length': '0'} 2022-09-12T19:06:47.229 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', '.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2daa207741f1f41-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.229 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:47.230 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:47.231 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.231 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.232 DEBUG:boto:Method: GET 2022-09-12T19:06:47.232 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:47.233 DEBUG:boto:Data: 2022-09-12T19:06:47.233 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.234 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.235 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.235 DEBUG:boto:Params: {} 2022-09-12T19:06:47.236 DEBUG:boto:Token: None 2022-09-12T19:06:47.236 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/ 2022-09-12T19:06:47.237 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y= 2022-09-12T19:06:47.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y=', 'Content-Length': '0'} 2022-09-12T19:06:47.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5da2cd9d6079ecd-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.241 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:47.242 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:47.243 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:47.243 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.244 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.245 DEBUG:boto:Method: GET 2022-09-12T19:06:47.245 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:47.246 DEBUG:boto:Data: 2022-09-12T19:06:47.246 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.247 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.247 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.248 DEBUG:boto:Params: {} 2022-09-12T19:06:47.249 DEBUG:boto:Token: None 2022-09-12T19:06:47.249 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/ 2022-09-12T19:06:47.250 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y= 2022-09-12T19:06:47.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y=', 'Content-Length': '0'} 2022-09-12T19:06:47.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000630ddfa73b76c91e-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.254 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:47.255 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.256 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.256 DEBUG:boto:Method: HEAD 2022-09-12T19:06:47.257 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:47.258 DEBUG:boto:Data: 2022-09-12T19:06:47.258 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.259 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.259 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.260 DEBUG:boto:Params: {} 2022-09-12T19:06:47.260 DEBUG:boto:Token: None 2022-09-12T19:06:47.261 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj 2022-09-12T19:06:47.262 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yGDO47MNel40xy2FVu+i46FaOVA= 2022-09-12T19:06:47.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yGDO47MNel40xy2FVu+i46FaOVA=', 'Content-Length': '0'} 2022-09-12T19:06:47.266 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3c0e5e16b684160-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.267 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.267 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.268 DEBUG:boto:Method: HEAD 2022-09-12T19:06:47.268 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:47.269 DEBUG:boto:Data: 2022-09-12T19:06:47.270 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.270 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.271 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.271 DEBUG:boto:Params: {} 2022-09-12T19:06:47.272 DEBUG:boto:Token: None 2022-09-12T19:06:47.272 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj 2022-09-12T19:06:47.273 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yGDO47MNel40xy2FVu+i46FaOVA= 2022-09-12T19:06:47.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yGDO47MNel40xy2FVu+i46FaOVA=', 'Content-Length': '0'} 2022-09-12T19:06:47.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cfdd3b8c4b6f81d1-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:47.278 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.279 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.280 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.280 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.281 DEBUG:boto:Method: GET 2022-09-12T19:06:47.281 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.282 DEBUG:boto:Data: 2022-09-12T19:06:47.283 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.284 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.285 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.286 DEBUG:boto:Params: {} 2022-09-12T19:06:47.287 DEBUG:boto:Token: None 2022-09-12T19:06:47.288 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.289 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eSB4MgxWXpbnFC/jznLI/4xTiSw= 2022-09-12T19:06:47.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eSB4MgxWXpbnFC/jznLI/4xTiSw=', 'Content-Length': '0'} 2022-09-12T19:06:47.293 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a3add95b4693282-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.294 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.295 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.295 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.296 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.296 DEBUG:boto:Method: GET 2022-09-12T19:06:47.297 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.297 DEBUG:boto:Data: 2022-09-12T19:06:47.298 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.299 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.299 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.300 DEBUG:boto:Params: {} 2022-09-12T19:06:47.301 DEBUG:boto:Token: None 2022-09-12T19:06:47.301 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.302 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eSB4MgxWXpbnFC/jznLI/4xTiSw= 2022-09-12T19:06:47.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eSB4MgxWXpbnFC/jznLI/4xTiSw=', 'Content-Length': '0'} 2022-09-12T19:06:47.304 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ced4763281e9a86-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.305 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:47.306 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:47.307 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.307 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.308 DEBUG:boto:Method: GET 2022-09-12T19:06:47.309 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:47.310 DEBUG:boto:Data: 2022-09-12T19:06:47.311 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.312 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.313 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.314 DEBUG:boto:Params: {} 2022-09-12T19:06:47.315 DEBUG:boto:Token: None 2022-09-12T19:06:47.316 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/ 2022-09-12T19:06:47.317 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y= 2022-09-12T19:06:47.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y=', 'Content-Length': '0'} 2022-09-12T19:06:47.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4fafa57beaa2c44-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.323 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:47.323 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:47.324 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:47.325 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.325 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.326 DEBUG:boto:Method: GET 2022-09-12T19:06:47.326 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:47.327 DEBUG:boto:Data: 2022-09-12T19:06:47.327 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.328 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.328 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.329 DEBUG:boto:Params: {} 2022-09-12T19:06:47.330 DEBUG:boto:Token: None 2022-09-12T19:06:47.330 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/ 2022-09-12T19:06:47.331 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y= 2022-09-12T19:06:47.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y=', 'Content-Length': '0'} 2022-09-12T19:06:47.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e96473b8de44e003-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.336 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:47.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:47.337 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.338 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.339 DEBUG:boto:path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:47.339 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:47.340 DEBUG:boto:Method: GET 2022-09-12T19:06:47.340 DEBUG:boto:Path: /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:47.341 DEBUG:boto:Data: 2022-09-12T19:06:47.341 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.342 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.342 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.343 DEBUG:boto:Params: {} 2022-09-12T19:06:47.345 DEBUG:boto:Token: None 2022-09-12T19:06:47.346 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:47.347 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mTPabM3AQAyjU1L3UhxK9vbCCXw= 2022-09-12T19:06:47.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mTPabM3AQAyjU1L3UhxK9vbCCXw=', 'Content-Length': '0'} 2022-09-12T19:06:47.351 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'K.LUooPdPnXZMzfYIxw-xagQpErk4CX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a892540a1b5022fd-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.394 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.395 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.396 DEBUG:boto:path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:47.397 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:47.398 DEBUG:boto:Method: GET 2022-09-12T19:06:47.399 DEBUG:boto:Path: /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:47.399 DEBUG:boto:Data: 2022-09-12T19:06:47.400 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.400 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.401 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.402 DEBUG:boto:Params: {} 2022-09-12T19:06:47.402 DEBUG:boto:Token: None 2022-09-12T19:06:47.403 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:47.404 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mTPabM3AQAyjU1L3UhxK9vbCCXw= 2022-09-12T19:06:47.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mTPabM3AQAyjU1L3UhxK9vbCCXw=', 'Content-Length': '0'} 2022-09-12T19:06:47.406 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'K.LUooPdPnXZMzfYIxw-xagQpErk4CX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000642a0799760b6ddb-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.407 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.408 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.408 DEBUG:boto:path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:47.409 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:47.409 DEBUG:boto:Method: HEAD 2022-09-12T19:06:47.410 DEBUG:boto:Path: /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:47.411 DEBUG:boto:Data: 2022-09-12T19:06:47.411 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.412 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.412 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.413 DEBUG:boto:Params: {} 2022-09-12T19:06:47.413 DEBUG:boto:Token: None 2022-09-12T19:06:47.414 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:47.414 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:njotCAIMO5U7M0lLqwKMFOZUGBA= 2022-09-12T19:06:47.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:njotCAIMO5U7M0lLqwKMFOZUGBA=', 'Content-Length': '0'} 2022-09-12T19:06:47.417 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'K.LUooPdPnXZMzfYIxw-xagQpErk4CX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bee1d5cd3db2c6e3-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.418 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.419 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.419 DEBUG:boto:path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:47.420 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:47.420 DEBUG:boto:Method: HEAD 2022-09-12T19:06:47.421 DEBUG:boto:Path: /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:47.421 DEBUG:boto:Data: 2022-09-12T19:06:47.422 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.422 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.423 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.423 DEBUG:boto:Params: {} 2022-09-12T19:06:47.424 DEBUG:boto:Token: None 2022-09-12T19:06:47.424 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:47.425 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:njotCAIMO5U7M0lLqwKMFOZUGBA= 2022-09-12T19:06:47.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:njotCAIMO5U7M0lLqwKMFOZUGBA=', 'Content-Length': '0'} 2022-09-12T19:06:47.428 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'K.LUooPdPnXZMzfYIxw-xagQpErk4CX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004304ab0bc2e19f29-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:47.429 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.430 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.430 DEBUG:boto:path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:47.431 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:47.432 DEBUG:boto:Method: GET 2022-09-12T19:06:47.432 DEBUG:boto:Path: /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:47.433 DEBUG:boto:Data: 2022-09-12T19:06:47.433 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.434 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.434 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.435 DEBUG:boto:Params: {} 2022-09-12T19:06:47.435 DEBUG:boto:Token: None 2022-09-12T19:06:47.436 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:47.437 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mTPabM3AQAyjU1L3UhxK9vbCCXw= 2022-09-12T19:06:47.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mTPabM3AQAyjU1L3UhxK9vbCCXw=', 'Content-Length': '0'} 2022-09-12T19:06:47.439 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'K.LUooPdPnXZMzfYIxw-xagQpErk4CX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b15af51ecfe0e52-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.440 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.441 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.441 DEBUG:boto:path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:47.442 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:47.442 DEBUG:boto:Method: GET 2022-09-12T19:06:47.443 DEBUG:boto:Path: /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:47.443 DEBUG:boto:Data: 2022-09-12T19:06:47.444 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.444 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.445 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.445 DEBUG:boto:Params: {} 2022-09-12T19:06:47.446 DEBUG:boto:Token: None 2022-09-12T19:06:47.447 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:47.447 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mTPabM3AQAyjU1L3UhxK9vbCCXw= 2022-09-12T19:06:47.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mTPabM3AQAyjU1L3UhxK9vbCCXw=', 'Content-Length': '0'} 2022-09-12T19:06:47.450 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'K.LUooPdPnXZMzfYIxw-xagQpErk4CX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd87097762e4b93b-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.494 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:47.495 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:47.496 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.496 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.497 DEBUG:boto:Method: GET 2022-09-12T19:06:47.497 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:47.498 DEBUG:boto:Data: 2022-09-12T19:06:47.499 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.499 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.500 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.500 DEBUG:boto:Params: {} 2022-09-12T19:06:47.501 DEBUG:boto:Token: None 2022-09-12T19:06:47.502 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/ 2022-09-12T19:06:47.502 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y= 2022-09-12T19:06:47.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y=', 'Content-Length': '0'} 2022-09-12T19:06:47.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cca689a2e0959486-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.507 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:47.507 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:47.508 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:47.509 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.509 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.510 DEBUG:boto:Method: GET 2022-09-12T19:06:47.510 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:47.511 DEBUG:boto:Data: 2022-09-12T19:06:47.511 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.512 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.512 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.513 DEBUG:boto:Params: {} 2022-09-12T19:06:47.513 DEBUG:boto:Token: None 2022-09-12T19:06:47.514 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/ 2022-09-12T19:06:47.515 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y= 2022-09-12T19:06:47.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y=', 'Content-Length': '0'} 2022-09-12T19:06:47.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d33f7314885917a-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.519 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:47.520 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.521 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.521 DEBUG:boto:Method: HEAD 2022-09-12T19:06:47.522 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:47.522 DEBUG:boto:Data: 2022-09-12T19:06:47.523 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.523 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.524 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.524 DEBUG:boto:Params: {} 2022-09-12T19:06:47.525 DEBUG:boto:Token: None 2022-09-12T19:06:47.526 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj 2022-09-12T19:06:47.526 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yGDO47MNel40xy2FVu+i46FaOVA= 2022-09-12T19:06:47.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yGDO47MNel40xy2FVu+i46FaOVA=', 'Content-Length': '0'} 2022-09-12T19:06:47.530 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008363c79009f8b1f0-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.531 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.532 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.532 DEBUG:boto:Method: HEAD 2022-09-12T19:06:47.533 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:47.533 DEBUG:boto:Data: 2022-09-12T19:06:47.534 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.534 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.535 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.535 DEBUG:boto:Params: {} 2022-09-12T19:06:47.536 DEBUG:boto:Token: None 2022-09-12T19:06:47.537 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj 2022-09-12T19:06:47.537 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yGDO47MNel40xy2FVu+i46FaOVA= 2022-09-12T19:06:47.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yGDO47MNel40xy2FVu+i46FaOVA=', 'Content-Length': '0'} 2022-09-12T19:06:47.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005752a12f865f4665-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:47.543 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.543 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.544 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.544 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.545 DEBUG:boto:Method: GET 2022-09-12T19:06:47.546 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.546 DEBUG:boto:Data: 2022-09-12T19:06:47.547 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.547 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.548 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.548 DEBUG:boto:Params: {} 2022-09-12T19:06:47.549 DEBUG:boto:Token: None 2022-09-12T19:06:47.549 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.550 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eSB4MgxWXpbnFC/jznLI/4xTiSw= 2022-09-12T19:06:47.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eSB4MgxWXpbnFC/jznLI/4xTiSw=', 'Content-Length': '0'} 2022-09-12T19:06:47.553 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062dd7ca1eace58d5-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.598 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.599 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.599 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.600 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.600 DEBUG:boto:Method: GET 2022-09-12T19:06:47.601 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.601 DEBUG:boto:Data: 2022-09-12T19:06:47.602 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.603 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.604 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.604 DEBUG:boto:Params: {} 2022-09-12T19:06:47.605 DEBUG:boto:Token: None 2022-09-12T19:06:47.606 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.607 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eSB4MgxWXpbnFC/jznLI/4xTiSw= 2022-09-12T19:06:47.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eSB4MgxWXpbnFC/jznLI/4xTiSw=', 'Content-Length': '0'} 2022-09-12T19:06:47.609 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4650b9a3a27de54-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.610 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:47.611 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:47.611 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.612 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.613 DEBUG:boto:Method: GET 2022-09-12T19:06:47.613 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:47.614 DEBUG:boto:Data: 2022-09-12T19:06:47.614 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.615 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.615 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.616 DEBUG:boto:Params: {} 2022-09-12T19:06:47.617 DEBUG:boto:Token: None 2022-09-12T19:06:47.618 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/ 2022-09-12T19:06:47.619 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y= 2022-09-12T19:06:47.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y=', 'Content-Length': '0'} 2022-09-12T19:06:47.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d6cd57532b23ae4-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.623 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:47.625 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:47.625 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:47.626 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.627 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.627 DEBUG:boto:Method: GET 2022-09-12T19:06:47.628 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:47.629 DEBUG:boto:Data: 2022-09-12T19:06:47.629 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.630 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.631 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.631 DEBUG:boto:Params: {} 2022-09-12T19:06:47.632 DEBUG:boto:Token: None 2022-09-12T19:06:47.633 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/ 2022-09-12T19:06:47.633 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y= 2022-09-12T19:06:47.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y=', 'Content-Length': '0'} 2022-09-12T19:06:47.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffe5703860c33273-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.638 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:47.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:47.640 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.641 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.642 DEBUG:boto:path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:47.643 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:47.644 DEBUG:boto:Method: GET 2022-09-12T19:06:47.645 DEBUG:boto:Path: /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:47.646 DEBUG:boto:Data: 2022-09-12T19:06:47.647 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.648 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.648 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.649 DEBUG:boto:Params: {} 2022-09-12T19:06:47.650 DEBUG:boto:Token: None 2022-09-12T19:06:47.651 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:47.652 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HhtAksHFBNOuplt0cqyK6D2PzEc= 2022-09-12T19:06:47.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HhtAksHFBNOuplt0cqyK6D2PzEc=', 'Content-Length': '0'} 2022-09-12T19:06:47.655 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'tQHk21B862H4VHQxrZWEog8n-qgO99Q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024e53e4179202c93-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.656 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.656 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.657 DEBUG:boto:path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:47.657 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:47.658 DEBUG:boto:Method: GET 2022-09-12T19:06:47.658 DEBUG:boto:Path: /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:47.659 DEBUG:boto:Data: 2022-09-12T19:06:47.659 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.660 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.661 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.661 DEBUG:boto:Params: {} 2022-09-12T19:06:47.662 DEBUG:boto:Token: None 2022-09-12T19:06:47.663 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:47.663 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HhtAksHFBNOuplt0cqyK6D2PzEc= 2022-09-12T19:06:47.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HhtAksHFBNOuplt0cqyK6D2PzEc=', 'Content-Length': '0'} 2022-09-12T19:06:47.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'tQHk21B862H4VHQxrZWEog8n-qgO99Q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b84f6e37ec5f4e6e-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.667 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.667 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.668 DEBUG:boto:path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:47.668 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:47.669 DEBUG:boto:Method: HEAD 2022-09-12T19:06:47.669 DEBUG:boto:Path: /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:47.670 DEBUG:boto:Data: 2022-09-12T19:06:47.671 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.671 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.672 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.672 DEBUG:boto:Params: {} 2022-09-12T19:06:47.673 DEBUG:boto:Token: None 2022-09-12T19:06:47.674 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:47.674 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LeO1QMFMe3F4s/r3JNp4Swb5lg0= 2022-09-12T19:06:47.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LeO1QMFMe3F4s/r3JNp4Swb5lg0=', 'Content-Length': '0'} 2022-09-12T19:06:47.677 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'tQHk21B862H4VHQxrZWEog8n-qgO99Q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d57f0ef1bb6a3e4-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.678 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.678 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.679 DEBUG:boto:path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:47.679 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:47.680 DEBUG:boto:Method: HEAD 2022-09-12T19:06:47.680 DEBUG:boto:Path: /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:47.681 DEBUG:boto:Data: 2022-09-12T19:06:47.681 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.682 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.682 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.683 DEBUG:boto:Params: {} 2022-09-12T19:06:47.683 DEBUG:boto:Token: None 2022-09-12T19:06:47.684 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:47.685 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LeO1QMFMe3F4s/r3JNp4Swb5lg0= 2022-09-12T19:06:47.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LeO1QMFMe3F4s/r3JNp4Swb5lg0=', 'Content-Length': '0'} 2022-09-12T19:06:47.687 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'tQHk21B862H4VHQxrZWEog8n-qgO99Q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002181994a80e814b3-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:47.688 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.689 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.689 DEBUG:boto:path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:47.690 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:47.691 DEBUG:boto:Method: GET 2022-09-12T19:06:47.691 DEBUG:boto:Path: /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:47.692 DEBUG:boto:Data: 2022-09-12T19:06:47.692 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.693 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.693 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.694 DEBUG:boto:Params: {} 2022-09-12T19:06:47.694 DEBUG:boto:Token: None 2022-09-12T19:06:47.695 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:47.696 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HhtAksHFBNOuplt0cqyK6D2PzEc= 2022-09-12T19:06:47.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HhtAksHFBNOuplt0cqyK6D2PzEc=', 'Content-Length': '0'} 2022-09-12T19:06:47.699 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'tQHk21B862H4VHQxrZWEog8n-qgO99Q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dbb7f9b1a7f7b551-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.742 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.743 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.743 DEBUG:boto:path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:47.744 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:47.745 DEBUG:boto:Method: GET 2022-09-12T19:06:47.745 DEBUG:boto:Path: /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:47.746 DEBUG:boto:Data: 2022-09-12T19:06:47.746 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.747 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.747 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.748 DEBUG:boto:Params: {} 2022-09-12T19:06:47.749 DEBUG:boto:Token: None 2022-09-12T19:06:47.749 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:47.750 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HhtAksHFBNOuplt0cqyK6D2PzEc= 2022-09-12T19:06:47.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HhtAksHFBNOuplt0cqyK6D2PzEc=', 'Content-Length': '0'} 2022-09-12T19:06:47.753 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'tQHk21B862H4VHQxrZWEog8n-qgO99Q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3f934572bd4ffa7-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.753 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:47.754 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:47.755 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.755 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.756 DEBUG:boto:Method: GET 2022-09-12T19:06:47.757 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:47.758 DEBUG:boto:Data: 2022-09-12T19:06:47.759 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.760 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.761 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.762 DEBUG:boto:Params: {} 2022-09-12T19:06:47.763 DEBUG:boto:Token: None 2022-09-12T19:06:47.764 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/ 2022-09-12T19:06:47.765 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y= 2022-09-12T19:06:47.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y=', 'Content-Length': '0'} 2022-09-12T19:06:47.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cec854eec9623eae-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.770 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:47.771 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:47.772 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:47.772 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.773 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.773 DEBUG:boto:Method: GET 2022-09-12T19:06:47.774 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:47.774 DEBUG:boto:Data: 2022-09-12T19:06:47.775 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.776 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.776 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.777 DEBUG:boto:Params: {} 2022-09-12T19:06:47.778 DEBUG:boto:Token: None 2022-09-12T19:06:47.778 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/ 2022-09-12T19:06:47.779 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y= 2022-09-12T19:06:47.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y=', 'Content-Length': '0'} 2022-09-12T19:06:47.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dd7a624da5b458b-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.783 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:47.784 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.785 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.785 DEBUG:boto:Method: HEAD 2022-09-12T19:06:47.786 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:47.787 DEBUG:boto:Data: 2022-09-12T19:06:47.787 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.788 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.789 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.789 DEBUG:boto:Params: {} 2022-09-12T19:06:47.790 DEBUG:boto:Token: None 2022-09-12T19:06:47.790 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj 2022-09-12T19:06:47.791 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yGDO47MNel40xy2FVu+i46FaOVA= 2022-09-12T19:06:47.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yGDO47MNel40xy2FVu+i46FaOVA=', 'Content-Length': '0'} 2022-09-12T19:06:47.795 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e2471b25cb89c42-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.796 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.796 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.797 DEBUG:boto:Method: HEAD 2022-09-12T19:06:47.797 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:47.798 DEBUG:boto:Data: 2022-09-12T19:06:47.799 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.799 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.800 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.800 DEBUG:boto:Params: {} 2022-09-12T19:06:47.801 DEBUG:boto:Token: None 2022-09-12T19:06:47.801 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj 2022-09-12T19:06:47.802 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yGDO47MNel40xy2FVu+i46FaOVA= 2022-09-12T19:06:47.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yGDO47MNel40xy2FVu+i46FaOVA=', 'Content-Length': '0'} 2022-09-12T19:06:47.806 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085fe4625d967cc52-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:47.807 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.808 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.809 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.809 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.810 DEBUG:boto:Method: GET 2022-09-12T19:06:47.811 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.811 DEBUG:boto:Data: 2022-09-12T19:06:47.812 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.812 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.813 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.813 DEBUG:boto:Params: {} 2022-09-12T19:06:47.814 DEBUG:boto:Token: None 2022-09-12T19:06:47.814 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.815 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eSB4MgxWXpbnFC/jznLI/4xTiSw= 2022-09-12T19:06:47.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eSB4MgxWXpbnFC/jznLI/4xTiSw=', 'Content-Length': '0'} 2022-09-12T19:06:47.818 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000365558ca611c0510-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.819 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.820 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.820 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.821 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.821 DEBUG:boto:Method: GET 2022-09-12T19:06:47.822 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.823 DEBUG:boto:Data: 2022-09-12T19:06:47.824 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.824 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.825 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.825 DEBUG:boto:Params: {} 2022-09-12T19:06:47.826 DEBUG:boto:Token: None 2022-09-12T19:06:47.826 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:47.827 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eSB4MgxWXpbnFC/jznLI/4xTiSw= 2022-09-12T19:06:47.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eSB4MgxWXpbnFC/jznLI/4xTiSw=', 'Content-Length': '0'} 2022-09-12T19:06:47.830 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2657337db4c6ee6-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.831 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:47.831 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:47.832 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.832 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.833 DEBUG:boto:Method: GET 2022-09-12T19:06:47.834 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:47.834 DEBUG:boto:Data: 2022-09-12T19:06:47.835 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.835 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.836 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.837 DEBUG:boto:Params: {} 2022-09-12T19:06:47.837 DEBUG:boto:Token: None 2022-09-12T19:06:47.838 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/ 2022-09-12T19:06:47.839 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y= 2022-09-12T19:06:47.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y=', 'Content-Length': '0'} 2022-09-12T19:06:47.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000848e793d393b9624-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.886 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:47.888 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:47.889 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:47.890 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.890 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.892 DEBUG:boto:Method: GET 2022-09-12T19:06:47.893 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:47.894 DEBUG:boto:Data: 2022-09-12T19:06:47.895 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.896 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.897 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.898 DEBUG:boto:Params: {} 2022-09-12T19:06:47.899 DEBUG:boto:Token: None 2022-09-12T19:06:47.900 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/ 2022-09-12T19:06:47.901 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y= 2022-09-12T19:06:47.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y=', 'Content-Length': '0'} 2022-09-12T19:06:47.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ce3df59f52c0a7c-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.910 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:47.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:47.913 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.913 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.914 DEBUG:boto:path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:47.915 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:47.916 DEBUG:boto:Method: GET 2022-09-12T19:06:47.917 DEBUG:boto:Path: /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:47.918 DEBUG:boto:Data: 2022-09-12T19:06:47.919 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.920 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.921 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.922 DEBUG:boto:Params: {} 2022-09-12T19:06:47.923 DEBUG:boto:Token: None 2022-09-12T19:06:47.924 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:47.925 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hYyKIDpQ3Hy/KQvKHmrh9QNVhfg= 2022-09-12T19:06:47.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hYyKIDpQ3Hy/KQvKHmrh9QNVhfg=', 'Content-Length': '0'} 2022-09-12T19:06:47.929 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'QHXOu7SGV87vItdBeqaL.hYUlX--Ty8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005be4a42590180831-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.929 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.930 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.930 DEBUG:boto:path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:47.931 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:47.932 DEBUG:boto:Method: GET 2022-09-12T19:06:47.932 DEBUG:boto:Path: /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:47.933 DEBUG:boto:Data: 2022-09-12T19:06:47.933 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.934 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.934 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.935 DEBUG:boto:Params: {} 2022-09-12T19:06:47.935 DEBUG:boto:Token: None 2022-09-12T19:06:47.936 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:47.936 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hYyKIDpQ3Hy/KQvKHmrh9QNVhfg= 2022-09-12T19:06:47.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hYyKIDpQ3Hy/KQvKHmrh9QNVhfg=', 'Content-Length': '0'} 2022-09-12T19:06:47.939 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'QHXOu7SGV87vItdBeqaL.hYUlX--Ty8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ebea20af008e68d-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.941 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.941 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.942 DEBUG:boto:path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:47.943 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:47.943 DEBUG:boto:Method: HEAD 2022-09-12T19:06:47.944 DEBUG:boto:Path: /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:47.944 DEBUG:boto:Data: 2022-09-12T19:06:47.945 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.945 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.946 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.946 DEBUG:boto:Params: {} 2022-09-12T19:06:47.947 DEBUG:boto:Token: None 2022-09-12T19:06:47.947 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:47.948 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Mx6t66dS63HD3cDKSRC6J2s3r+k= 2022-09-12T19:06:47.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Mx6t66dS63HD3cDKSRC6J2s3r+k=', 'Content-Length': '0'} 2022-09-12T19:06:47.952 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'QHXOu7SGV87vItdBeqaL.hYUlX--Ty8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c74a4c42a0836fa1-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.952 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.953 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.953 DEBUG:boto:path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:47.954 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:47.954 DEBUG:boto:Method: HEAD 2022-09-12T19:06:47.955 DEBUG:boto:Path: /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:47.955 DEBUG:boto:Data: 2022-09-12T19:06:47.956 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.956 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.957 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.957 DEBUG:boto:Params: {} 2022-09-12T19:06:47.958 DEBUG:boto:Token: None 2022-09-12T19:06:47.959 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:47.959 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Mx6t66dS63HD3cDKSRC6J2s3r+k= 2022-09-12T19:06:47.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Mx6t66dS63HD3cDKSRC6J2s3r+k=', 'Content-Length': '0'} 2022-09-12T19:06:47.962 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'QHXOu7SGV87vItdBeqaL.hYUlX--Ty8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fdfd315dfb7735d5-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:47.963 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.964 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.964 DEBUG:boto:path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:47.965 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:47.966 DEBUG:boto:Method: GET 2022-09-12T19:06:47.966 DEBUG:boto:Path: /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:47.967 DEBUG:boto:Data: 2022-09-12T19:06:47.967 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.968 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.968 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.969 DEBUG:boto:Params: {} 2022-09-12T19:06:47.969 DEBUG:boto:Token: None 2022-09-12T19:06:47.970 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:47.970 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hYyKIDpQ3Hy/KQvKHmrh9QNVhfg= 2022-09-12T19:06:47.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hYyKIDpQ3Hy/KQvKHmrh9QNVhfg=', 'Content-Length': '0'} 2022-09-12T19:06:47.973 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'QHXOu7SGV87vItdBeqaL.hYUlX--Ty8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b308a680adb6a779-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.974 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:47.975 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:47.975 DEBUG:boto:path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:47.976 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:47.976 DEBUG:boto:Method: GET 2022-09-12T19:06:47.977 DEBUG:boto:Path: /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:47.977 DEBUG:boto:Data: 2022-09-12T19:06:47.978 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.978 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:47.979 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.979 DEBUG:boto:Params: {} 2022-09-12T19:06:47.980 DEBUG:boto:Token: None 2022-09-12T19:06:47.981 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:47.981 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hYyKIDpQ3Hy/KQvKHmrh9QNVhfg= 2022-09-12T19:06:47.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hYyKIDpQ3Hy/KQvKHmrh9QNVhfg=', 'Content-Length': '0'} 2022-09-12T19:06:47.984 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'QHXOu7SGV87vItdBeqaL.hYUlX--Ty8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af04442ec622834c-00631f8347-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.985 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:47.986 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:47.986 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.987 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:47.987 DEBUG:boto:Method: GET 2022-09-12T19:06:47.988 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:47.989 DEBUG:boto:Data: 2022-09-12T19:06:47.989 DEBUG:boto:Headers: {} 2022-09-12T19:06:47.990 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:47.990 DEBUG:boto:Port: 8000 2022-09-12T19:06:47.991 DEBUG:boto:Params: {} 2022-09-12T19:06:47.992 DEBUG:boto:Token: None 2022-09-12T19:06:47.993 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:47 GMT /jdpiak-19/ 2022-09-12T19:06:47.993 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y= 2022-09-12T19:06:47.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0A9Z3EC0sKsSR5y0EpKarcYFn+Y=', 'Content-Length': '0'} 2022-09-12T19:06:47.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2ccb93c51fafa1d-00631f8347-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:47.998 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:47.999 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:47.999 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:48.000 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.000 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.001 DEBUG:boto:Method: GET 2022-09-12T19:06:48.002 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:48.002 DEBUG:boto:Data: 2022-09-12T19:06:48.003 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.003 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.004 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.004 DEBUG:boto:Params: {} 2022-09-12T19:06:48.005 DEBUG:boto:Token: None 2022-09-12T19:06:48.006 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/ 2022-09-12T19:06:48.007 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI= 2022-09-12T19:06:48.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI=', 'Content-Length': '0'} 2022-09-12T19:06:48.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043aa620f3f6f5156-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.011 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:48.012 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.012 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.013 DEBUG:boto:Method: HEAD 2022-09-12T19:06:48.013 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:48.014 DEBUG:boto:Data: 2022-09-12T19:06:48.015 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.015 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.016 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.016 DEBUG:boto:Params: {} 2022-09-12T19:06:48.017 DEBUG:boto:Token: None 2022-09-12T19:06:48.017 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj 2022-09-12T19:06:48.018 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dzd4UqxQnNFEdXMd5MkVZ6Zir3g= 2022-09-12T19:06:48.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dzd4UqxQnNFEdXMd5MkVZ6Zir3g=', 'Content-Length': '0'} 2022-09-12T19:06:48.022 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000043cfcdf07074a386-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.023 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.023 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.024 DEBUG:boto:Method: HEAD 2022-09-12T19:06:48.025 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:48.025 DEBUG:boto:Data: 2022-09-12T19:06:48.026 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.026 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.027 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.027 DEBUG:boto:Params: {} 2022-09-12T19:06:48.028 DEBUG:boto:Token: None 2022-09-12T19:06:48.039 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj 2022-09-12T19:06:48.039 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dzd4UqxQnNFEdXMd5MkVZ6Zir3g= 2022-09-12T19:06:48.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dzd4UqxQnNFEdXMd5MkVZ6Zir3g=', 'Content-Length': '0'} 2022-09-12T19:06:48.044 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b82ab2fe95f2c329-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:48.045 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.046 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.047 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.047 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.048 DEBUG:boto:Method: GET 2022-09-12T19:06:48.049 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.050 DEBUG:boto:Data: 2022-09-12T19:06:48.051 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.052 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.053 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.055 DEBUG:boto:Params: {} 2022-09-12T19:06:48.056 DEBUG:boto:Token: None 2022-09-12T19:06:48.057 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.058 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Vc/yEKLSLT1WQbl6XMHi7EONcV8= 2022-09-12T19:06:48.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Vc/yEKLSLT1WQbl6XMHi7EONcV8=', 'Content-Length': '0'} 2022-09-12T19:06:48.062 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a0eb1da8d439ba1-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.063 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.063 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.064 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.065 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.065 DEBUG:boto:Method: GET 2022-09-12T19:06:48.066 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.066 DEBUG:boto:Data: 2022-09-12T19:06:48.067 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.067 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.068 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.068 DEBUG:boto:Params: {} 2022-09-12T19:06:48.069 DEBUG:boto:Token: None 2022-09-12T19:06:48.070 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.070 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Vc/yEKLSLT1WQbl6XMHi7EONcV8= 2022-09-12T19:06:48.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Vc/yEKLSLT1WQbl6XMHi7EONcV8=', 'Content-Length': '0'} 2022-09-12T19:06:48.074 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa0acd9e2e5a0547-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.118 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:48.119 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:48.119 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.120 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.121 DEBUG:boto:Method: GET 2022-09-12T19:06:48.121 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:48.122 DEBUG:boto:Data: 2022-09-12T19:06:48.122 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.123 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.124 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.124 DEBUG:boto:Params: {} 2022-09-12T19:06:48.125 DEBUG:boto:Token: None 2022-09-12T19:06:48.125 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/ 2022-09-12T19:06:48.126 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI= 2022-09-12T19:06:48.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI=', 'Content-Length': '0'} 2022-09-12T19:06:48.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092d38a0130c6c96f-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.130 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:48.131 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:48.132 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:48.132 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.133 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.133 DEBUG:boto:Method: GET 2022-09-12T19:06:48.134 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:48.134 DEBUG:boto:Data: 2022-09-12T19:06:48.135 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.135 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.136 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.136 DEBUG:boto:Params: {} 2022-09-12T19:06:48.137 DEBUG:boto:Token: None 2022-09-12T19:06:48.138 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/ 2022-09-12T19:06:48.138 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI= 2022-09-12T19:06:48.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI=', 'Content-Length': '0'} 2022-09-12T19:06:48.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b023d97ac2a9906b-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.143 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:48.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:48.144 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.145 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.145 DEBUG:boto:path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:48.146 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:48.146 DEBUG:boto:Method: GET 2022-09-12T19:06:48.147 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:48.147 DEBUG:boto:Data: 2022-09-12T19:06:48.148 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.149 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.149 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.150 DEBUG:boto:Params: {} 2022-09-12T19:06:48.150 DEBUG:boto:Token: None 2022-09-12T19:06:48.151 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:48.151 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f7t88m+T50nPxQf8iZd45R/C2oA= 2022-09-12T19:06:48.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f7t88m+T50nPxQf8iZd45R/C2oA=', 'Content-Length': '0'} 2022-09-12T19:06:48.154 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049ff5a5e8f0fcd50-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.155 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.156 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.156 DEBUG:boto:path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:48.157 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:48.157 DEBUG:boto:Method: GET 2022-09-12T19:06:48.158 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:48.158 DEBUG:boto:Data: 2022-09-12T19:06:48.159 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.159 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.160 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.160 DEBUG:boto:Params: {} 2022-09-12T19:06:48.161 DEBUG:boto:Token: None 2022-09-12T19:06:48.162 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:48.162 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f7t88m+T50nPxQf8iZd45R/C2oA= 2022-09-12T19:06:48.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f7t88m+T50nPxQf8iZd45R/C2oA=', 'Content-Length': '0'} 2022-09-12T19:06:48.165 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096750c2f63143a1a-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.166 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.167 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.167 DEBUG:boto:path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:48.168 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:48.168 DEBUG:boto:Method: HEAD 2022-09-12T19:06:48.169 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:48.169 DEBUG:boto:Data: 2022-09-12T19:06:48.170 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.170 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.171 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.171 DEBUG:boto:Params: {} 2022-09-12T19:06:48.172 DEBUG:boto:Token: None 2022-09-12T19:06:48.172 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:48.173 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6wrx1bYCJznlAei+KhPkfCz0Sco= 2022-09-12T19:06:48.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6wrx1bYCJznlAei+KhPkfCz0Sco=', 'Content-Length': '0'} 2022-09-12T19:06:48.176 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015fb5ef77c42723d-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.176 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.177 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.177 DEBUG:boto:path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:48.178 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:48.178 DEBUG:boto:Method: HEAD 2022-09-12T19:06:48.179 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:48.179 DEBUG:boto:Data: 2022-09-12T19:06:48.180 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.180 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.181 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.181 DEBUG:boto:Params: {} 2022-09-12T19:06:48.182 DEBUG:boto:Token: None 2022-09-12T19:06:48.183 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:48.183 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6wrx1bYCJznlAei+KhPkfCz0Sco= 2022-09-12T19:06:48.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6wrx1bYCJznlAei+KhPkfCz0Sco=', 'Content-Length': '0'} 2022-09-12T19:06:48.186 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005cfd0733dbd51702-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:48.187 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.188 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.188 DEBUG:boto:path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:48.189 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:48.189 DEBUG:boto:Method: GET 2022-09-12T19:06:48.190 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:48.190 DEBUG:boto:Data: 2022-09-12T19:06:48.191 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.191 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.192 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.192 DEBUG:boto:Params: {} 2022-09-12T19:06:48.193 DEBUG:boto:Token: None 2022-09-12T19:06:48.194 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:48.194 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f7t88m+T50nPxQf8iZd45R/C2oA= 2022-09-12T19:06:48.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f7t88m+T50nPxQf8iZd45R/C2oA=', 'Content-Length': '0'} 2022-09-12T19:06:48.197 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059e7307d4aa59f97-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.198 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.198 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.199 DEBUG:boto:path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:48.199 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:48.200 DEBUG:boto:Method: GET 2022-09-12T19:06:48.200 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:48.201 DEBUG:boto:Data: 2022-09-12T19:06:48.202 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.202 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.203 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.203 DEBUG:boto:Params: {} 2022-09-12T19:06:48.204 DEBUG:boto:Token: None 2022-09-12T19:06:48.204 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:48.205 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f7t88m+T50nPxQf8iZd45R/C2oA= 2022-09-12T19:06:48.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f7t88m+T50nPxQf8iZd45R/C2oA=', 'Content-Length': '0'} 2022-09-12T19:06:48.208 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a3fc537585c1dc5-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.209 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:48.209 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:48.210 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.210 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.211 DEBUG:boto:Method: GET 2022-09-12T19:06:48.211 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:48.212 DEBUG:boto:Data: 2022-09-12T19:06:48.212 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.213 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.214 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.214 DEBUG:boto:Params: {} 2022-09-12T19:06:48.215 DEBUG:boto:Token: None 2022-09-12T19:06:48.215 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/ 2022-09-12T19:06:48.216 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI= 2022-09-12T19:06:48.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI=', 'Content-Length': '0'} 2022-09-12T19:06:48.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b94135636b2e2479-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.220 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:48.221 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:48.221 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:48.222 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.223 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.223 DEBUG:boto:Method: GET 2022-09-12T19:06:48.224 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:48.224 DEBUG:boto:Data: 2022-09-12T19:06:48.225 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.226 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.226 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.227 DEBUG:boto:Params: {} 2022-09-12T19:06:48.228 DEBUG:boto:Token: None 2022-09-12T19:06:48.229 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/ 2022-09-12T19:06:48.230 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI= 2022-09-12T19:06:48.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI=', 'Content-Length': '0'} 2022-09-12T19:06:48.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bb6c2ab08722481-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.235 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:48.236 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.237 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.237 DEBUG:boto:Method: HEAD 2022-09-12T19:06:48.238 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:48.238 DEBUG:boto:Data: 2022-09-12T19:06:48.239 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.239 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.240 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.240 DEBUG:boto:Params: {} 2022-09-12T19:06:48.241 DEBUG:boto:Token: None 2022-09-12T19:06:48.242 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj 2022-09-12T19:06:48.242 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dzd4UqxQnNFEdXMd5MkVZ6Zir3g= 2022-09-12T19:06:48.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dzd4UqxQnNFEdXMd5MkVZ6Zir3g=', 'Content-Length': '0'} 2022-09-12T19:06:48.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0ce8f63cf19680c-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.247 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.248 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.248 DEBUG:boto:Method: HEAD 2022-09-12T19:06:48.249 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:48.249 DEBUG:boto:Data: 2022-09-12T19:06:48.250 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.250 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.251 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.252 DEBUG:boto:Params: {} 2022-09-12T19:06:48.252 DEBUG:boto:Token: None 2022-09-12T19:06:48.253 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj 2022-09-12T19:06:48.254 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dzd4UqxQnNFEdXMd5MkVZ6Zir3g= 2022-09-12T19:06:48.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dzd4UqxQnNFEdXMd5MkVZ6Zir3g=', 'Content-Length': '0'} 2022-09-12T19:06:48.258 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4225253c20fe04e-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:48.259 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.260 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.261 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.261 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.262 DEBUG:boto:Method: GET 2022-09-12T19:06:48.262 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.263 DEBUG:boto:Data: 2022-09-12T19:06:48.263 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.264 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.264 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.265 DEBUG:boto:Params: {} 2022-09-12T19:06:48.266 DEBUG:boto:Token: None 2022-09-12T19:06:48.266 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.267 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Vc/yEKLSLT1WQbl6XMHi7EONcV8= 2022-09-12T19:06:48.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Vc/yEKLSLT1WQbl6XMHi7EONcV8=', 'Content-Length': '0'} 2022-09-12T19:06:48.270 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b5e26aeb70329b8-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.314 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.315 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.315 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.316 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.317 DEBUG:boto:Method: GET 2022-09-12T19:06:48.317 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.318 DEBUG:boto:Data: 2022-09-12T19:06:48.318 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.319 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.319 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.320 DEBUG:boto:Params: {} 2022-09-12T19:06:48.321 DEBUG:boto:Token: None 2022-09-12T19:06:48.321 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.322 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Vc/yEKLSLT1WQbl6XMHi7EONcV8= 2022-09-12T19:06:48.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Vc/yEKLSLT1WQbl6XMHi7EONcV8=', 'Content-Length': '0'} 2022-09-12T19:06:48.325 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017e09f4c58728bc1-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.326 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:48.326 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:48.327 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.327 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.328 DEBUG:boto:Method: GET 2022-09-12T19:06:48.328 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:48.329 DEBUG:boto:Data: 2022-09-12T19:06:48.329 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.330 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.330 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.331 DEBUG:boto:Params: {} 2022-09-12T19:06:48.332 DEBUG:boto:Token: None 2022-09-12T19:06:48.332 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/ 2022-09-12T19:06:48.333 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI= 2022-09-12T19:06:48.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI=', 'Content-Length': '0'} 2022-09-12T19:06:48.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f9bf6bad25961e7-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.337 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:48.338 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:48.338 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:48.339 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.339 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.340 DEBUG:boto:Method: GET 2022-09-12T19:06:48.340 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:48.341 DEBUG:boto:Data: 2022-09-12T19:06:48.341 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.342 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.343 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.343 DEBUG:boto:Params: {} 2022-09-12T19:06:48.344 DEBUG:boto:Token: None 2022-09-12T19:06:48.344 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/ 2022-09-12T19:06:48.345 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI= 2022-09-12T19:06:48.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI=', 'Content-Length': '0'} 2022-09-12T19:06:48.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c180e80ae60f53a-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.349 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:48.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:48.351 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.351 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.352 DEBUG:boto:path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:48.352 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:48.353 DEBUG:boto:Method: GET 2022-09-12T19:06:48.353 DEBUG:boto:Path: /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:48.354 DEBUG:boto:Data: 2022-09-12T19:06:48.355 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.355 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.356 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.356 DEBUG:boto:Params: {} 2022-09-12T19:06:48.357 DEBUG:boto:Token: None 2022-09-12T19:06:48.358 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:48.359 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VWS0RH5Gln1ggah6GUr6Ta16N2M= 2022-09-12T19:06:48.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VWS0RH5Gln1ggah6GUr6Ta16N2M=', 'Content-Length': '0'} 2022-09-12T19:06:48.362 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c28f664a9ecf0917-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.362 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.363 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.364 DEBUG:boto:path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:48.364 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:48.365 DEBUG:boto:Method: GET 2022-09-12T19:06:48.366 DEBUG:boto:Path: /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:48.366 DEBUG:boto:Data: 2022-09-12T19:06:48.367 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.367 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.368 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.368 DEBUG:boto:Params: {} 2022-09-12T19:06:48.369 DEBUG:boto:Token: None 2022-09-12T19:06:48.370 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:48.370 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VWS0RH5Gln1ggah6GUr6Ta16N2M= 2022-09-12T19:06:48.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VWS0RH5Gln1ggah6GUr6Ta16N2M=', 'Content-Length': '0'} 2022-09-12T19:06:48.374 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd1ee9ac3c8644aa-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.375 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.375 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.376 DEBUG:boto:path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:48.376 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:48.377 DEBUG:boto:Method: HEAD 2022-09-12T19:06:48.377 DEBUG:boto:Path: /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:48.378 DEBUG:boto:Data: 2022-09-12T19:06:48.378 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.379 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.379 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.380 DEBUG:boto:Params: {} 2022-09-12T19:06:48.380 DEBUG:boto:Token: None 2022-09-12T19:06:48.381 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:48.382 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TnYy1T+lVxSmbVLVHn7XfWxhDRo= 2022-09-12T19:06:48.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TnYy1T+lVxSmbVLVHn7XfWxhDRo=', 'Content-Length': '0'} 2022-09-12T19:06:48.384 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003934a8577b5e2dc4-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.385 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.386 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.386 DEBUG:boto:path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:48.387 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:48.387 DEBUG:boto:Method: HEAD 2022-09-12T19:06:48.388 DEBUG:boto:Path: /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:48.388 DEBUG:boto:Data: 2022-09-12T19:06:48.389 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.389 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.390 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.390 DEBUG:boto:Params: {} 2022-09-12T19:06:48.391 DEBUG:boto:Token: None 2022-09-12T19:06:48.391 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:48.392 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TnYy1T+lVxSmbVLVHn7XfWxhDRo= 2022-09-12T19:06:48.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TnYy1T+lVxSmbVLVHn7XfWxhDRo=', 'Content-Length': '0'} 2022-09-12T19:06:48.395 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cac399ec3451a2b0-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:48.396 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.397 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.397 DEBUG:boto:path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:48.398 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:48.398 DEBUG:boto:Method: GET 2022-09-12T19:06:48.399 DEBUG:boto:Path: /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:48.399 DEBUG:boto:Data: 2022-09-12T19:06:48.400 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.400 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.401 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.401 DEBUG:boto:Params: {} 2022-09-12T19:06:48.402 DEBUG:boto:Token: None 2022-09-12T19:06:48.402 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:48.403 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VWS0RH5Gln1ggah6GUr6Ta16N2M= 2022-09-12T19:06:48.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VWS0RH5Gln1ggah6GUr6Ta16N2M=', 'Content-Length': '0'} 2022-09-12T19:06:48.406 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000298d28a03e984c2a-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.407 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.408 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.408 DEBUG:boto:path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:48.409 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:48.409 DEBUG:boto:Method: GET 2022-09-12T19:06:48.410 DEBUG:boto:Path: /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:48.410 DEBUG:boto:Data: 2022-09-12T19:06:48.411 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.411 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.412 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.412 DEBUG:boto:Params: {} 2022-09-12T19:06:48.413 DEBUG:boto:Token: None 2022-09-12T19:06:48.414 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:48.414 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VWS0RH5Gln1ggah6GUr6Ta16N2M= 2022-09-12T19:06:48.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VWS0RH5Gln1ggah6GUr6Ta16N2M=', 'Content-Length': '0'} 2022-09-12T19:06:48.417 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b62cb36608bbd589-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.418 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:48.419 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:48.420 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.420 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.421 DEBUG:boto:Method: GET 2022-09-12T19:06:48.422 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:48.422 DEBUG:boto:Data: 2022-09-12T19:06:48.423 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.423 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.424 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.425 DEBUG:boto:Params: {} 2022-09-12T19:06:48.425 DEBUG:boto:Token: None 2022-09-12T19:06:48.426 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/ 2022-09-12T19:06:48.427 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI= 2022-09-12T19:06:48.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI=', 'Content-Length': '0'} 2022-09-12T19:06:48.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092357b5621950a5f-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.431 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:48.432 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:48.433 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:48.433 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.434 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.434 DEBUG:boto:Method: GET 2022-09-12T19:06:48.435 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:48.436 DEBUG:boto:Data: 2022-09-12T19:06:48.436 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.437 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.438 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.438 DEBUG:boto:Params: {} 2022-09-12T19:06:48.439 DEBUG:boto:Token: None 2022-09-12T19:06:48.440 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/ 2022-09-12T19:06:48.441 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI= 2022-09-12T19:06:48.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI=', 'Content-Length': '0'} 2022-09-12T19:06:48.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd97212843aa70ee-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.452 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:48.453 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.454 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.455 DEBUG:boto:Method: HEAD 2022-09-12T19:06:48.455 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:48.456 DEBUG:boto:Data: 2022-09-12T19:06:48.457 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.458 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.459 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.460 DEBUG:boto:Params: {} 2022-09-12T19:06:48.461 DEBUG:boto:Token: None 2022-09-12T19:06:48.462 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj 2022-09-12T19:06:48.463 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dzd4UqxQnNFEdXMd5MkVZ6Zir3g= 2022-09-12T19:06:48.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dzd4UqxQnNFEdXMd5MkVZ6Zir3g=', 'Content-Length': '0'} 2022-09-12T19:06:48.468 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a48b8e94fa502c0a-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.469 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.470 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.470 DEBUG:boto:Method: HEAD 2022-09-12T19:06:48.471 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:48.471 DEBUG:boto:Data: 2022-09-12T19:06:48.472 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.472 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.473 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.473 DEBUG:boto:Params: {} 2022-09-12T19:06:48.474 DEBUG:boto:Token: None 2022-09-12T19:06:48.474 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj 2022-09-12T19:06:48.475 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dzd4UqxQnNFEdXMd5MkVZ6Zir3g= 2022-09-12T19:06:48.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dzd4UqxQnNFEdXMd5MkVZ6Zir3g=', 'Content-Length': '0'} 2022-09-12T19:06:48.479 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d96c697c5354a2ff-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:48.481 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.481 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.482 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.483 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.483 DEBUG:boto:Method: GET 2022-09-12T19:06:48.484 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.484 DEBUG:boto:Data: 2022-09-12T19:06:48.485 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.485 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.486 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.487 DEBUG:boto:Params: {} 2022-09-12T19:06:48.488 DEBUG:boto:Token: None 2022-09-12T19:06:48.489 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.490 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Vc/yEKLSLT1WQbl6XMHi7EONcV8= 2022-09-12T19:06:48.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Vc/yEKLSLT1WQbl6XMHi7EONcV8=', 'Content-Length': '0'} 2022-09-12T19:06:48.494 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac707bcc67e02cae-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.538 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.539 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.539 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.540 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.540 DEBUG:boto:Method: GET 2022-09-12T19:06:48.541 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.541 DEBUG:boto:Data: 2022-09-12T19:06:48.542 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.543 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.543 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.544 DEBUG:boto:Params: {} 2022-09-12T19:06:48.545 DEBUG:boto:Token: None 2022-09-12T19:06:48.545 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.546 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Vc/yEKLSLT1WQbl6XMHi7EONcV8= 2022-09-12T19:06:48.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Vc/yEKLSLT1WQbl6XMHi7EONcV8=', 'Content-Length': '0'} 2022-09-12T19:06:48.550 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c14baee41a6290aa-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.551 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:48.551 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:48.552 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.553 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.553 DEBUG:boto:Method: GET 2022-09-12T19:06:48.554 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:48.554 DEBUG:boto:Data: 2022-09-12T19:06:48.555 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.555 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.556 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.556 DEBUG:boto:Params: {} 2022-09-12T19:06:48.557 DEBUG:boto:Token: None 2022-09-12T19:06:48.558 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/ 2022-09-12T19:06:48.558 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI= 2022-09-12T19:06:48.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI=', 'Content-Length': '0'} 2022-09-12T19:06:48.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca614652471da9c7-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.562 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:48.563 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:48.564 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:48.564 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.565 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.565 DEBUG:boto:Method: GET 2022-09-12T19:06:48.566 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:48.567 DEBUG:boto:Data: 2022-09-12T19:06:48.567 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.568 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.568 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.569 DEBUG:boto:Params: {} 2022-09-12T19:06:48.569 DEBUG:boto:Token: None 2022-09-12T19:06:48.570 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/ 2022-09-12T19:06:48.571 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI= 2022-09-12T19:06:48.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI=', 'Content-Length': '0'} 2022-09-12T19:06:48.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007981f36e9f23eff7-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.618 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:48.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:48.620 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.620 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.621 DEBUG:boto:path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:48.622 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:48.622 DEBUG:boto:Method: GET 2022-09-12T19:06:48.623 DEBUG:boto:Path: /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:48.624 DEBUG:boto:Data: 2022-09-12T19:06:48.624 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.625 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.625 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.626 DEBUG:boto:Params: {} 2022-09-12T19:06:48.626 DEBUG:boto:Token: None 2022-09-12T19:06:48.627 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:48.628 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:euL5ujalZX8hP1svAyY+kF+ea3M= 2022-09-12T19:06:48.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:euL5ujalZX8hP1svAyY+kF+ea3M=', 'Content-Length': '0'} 2022-09-12T19:06:48.630 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'BAbppx4RylEKnBrYesXt1QPJgsAhIBb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff22707c5f881153-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.631 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.632 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.632 DEBUG:boto:path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:48.633 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:48.633 DEBUG:boto:Method: GET 2022-09-12T19:06:48.634 DEBUG:boto:Path: /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:48.635 DEBUG:boto:Data: 2022-09-12T19:06:48.635 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.636 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.636 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.637 DEBUG:boto:Params: {} 2022-09-12T19:06:48.637 DEBUG:boto:Token: None 2022-09-12T19:06:48.638 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:48.638 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:euL5ujalZX8hP1svAyY+kF+ea3M= 2022-09-12T19:06:48.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:euL5ujalZX8hP1svAyY+kF+ea3M=', 'Content-Length': '0'} 2022-09-12T19:06:48.642 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'BAbppx4RylEKnBrYesXt1QPJgsAhIBb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004098e22b67eeaa48-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.643 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.643 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.644 DEBUG:boto:path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:48.644 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:48.645 DEBUG:boto:Method: HEAD 2022-09-12T19:06:48.645 DEBUG:boto:Path: /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:48.646 DEBUG:boto:Data: 2022-09-12T19:06:48.646 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.647 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.647 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.648 DEBUG:boto:Params: {} 2022-09-12T19:06:48.648 DEBUG:boto:Token: None 2022-09-12T19:06:48.649 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:48.650 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YCJrXLsqzZC6RUNnIBakQ64+iy4= 2022-09-12T19:06:48.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YCJrXLsqzZC6RUNnIBakQ64+iy4=', 'Content-Length': '0'} 2022-09-12T19:06:48.652 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'BAbppx4RylEKnBrYesXt1QPJgsAhIBb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000175bbad4b11f5a2e-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.653 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.654 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.654 DEBUG:boto:path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:48.655 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:48.655 DEBUG:boto:Method: HEAD 2022-09-12T19:06:48.656 DEBUG:boto:Path: /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:48.657 DEBUG:boto:Data: 2022-09-12T19:06:48.657 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.657 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.658 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.659 DEBUG:boto:Params: {} 2022-09-12T19:06:48.659 DEBUG:boto:Token: None 2022-09-12T19:06:48.660 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:48.660 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YCJrXLsqzZC6RUNnIBakQ64+iy4= 2022-09-12T19:06:48.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YCJrXLsqzZC6RUNnIBakQ64+iy4=', 'Content-Length': '0'} 2022-09-12T19:06:48.663 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'BAbppx4RylEKnBrYesXt1QPJgsAhIBb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008bad49110c7d2cba-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:48.664 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.665 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.665 DEBUG:boto:path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:48.666 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:48.667 DEBUG:boto:Method: GET 2022-09-12T19:06:48.667 DEBUG:boto:Path: /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:48.668 DEBUG:boto:Data: 2022-09-12T19:06:48.668 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.669 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.669 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.670 DEBUG:boto:Params: {} 2022-09-12T19:06:48.670 DEBUG:boto:Token: None 2022-09-12T19:06:48.671 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:48.671 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:euL5ujalZX8hP1svAyY+kF+ea3M= 2022-09-12T19:06:48.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:euL5ujalZX8hP1svAyY+kF+ea3M=', 'Content-Length': '0'} 2022-09-12T19:06:48.679 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'BAbppx4RylEKnBrYesXt1QPJgsAhIBb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010f9bebc34cba967-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.680 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.681 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.682 DEBUG:boto:path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:48.683 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:48.684 DEBUG:boto:Method: GET 2022-09-12T19:06:48.685 DEBUG:boto:Path: /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:48.686 DEBUG:boto:Data: 2022-09-12T19:06:48.686 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.687 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.688 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.689 DEBUG:boto:Params: {} 2022-09-12T19:06:48.690 DEBUG:boto:Token: None 2022-09-12T19:06:48.691 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:48.693 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:euL5ujalZX8hP1svAyY+kF+ea3M= 2022-09-12T19:06:48.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:euL5ujalZX8hP1svAyY+kF+ea3M=', 'Content-Length': '0'} 2022-09-12T19:06:48.696 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'BAbppx4RylEKnBrYesXt1QPJgsAhIBb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006fbc011c00eb60f2-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.738 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:48.739 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:48.740 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.740 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.741 DEBUG:boto:Method: GET 2022-09-12T19:06:48.741 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:48.742 DEBUG:boto:Data: 2022-09-12T19:06:48.743 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.744 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.744 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.745 DEBUG:boto:Params: {} 2022-09-12T19:06:48.745 DEBUG:boto:Token: None 2022-09-12T19:06:48.746 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/ 2022-09-12T19:06:48.746 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI= 2022-09-12T19:06:48.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI=', 'Content-Length': '0'} 2022-09-12T19:06:48.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a931570527edc0e-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.751 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:48.752 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:48.753 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:48.754 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.754 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.755 DEBUG:boto:Method: GET 2022-09-12T19:06:48.755 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:48.756 DEBUG:boto:Data: 2022-09-12T19:06:48.756 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.757 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.758 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.758 DEBUG:boto:Params: {} 2022-09-12T19:06:48.759 DEBUG:boto:Token: None 2022-09-12T19:06:48.759 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/ 2022-09-12T19:06:48.760 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI= 2022-09-12T19:06:48.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI=', 'Content-Length': '0'} 2022-09-12T19:06:48.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cc53f52806c46fa-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.765 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:48.766 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.766 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.767 DEBUG:boto:Method: HEAD 2022-09-12T19:06:48.768 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:48.769 DEBUG:boto:Data: 2022-09-12T19:06:48.769 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.770 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.771 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.772 DEBUG:boto:Params: {} 2022-09-12T19:06:48.773 DEBUG:boto:Token: None 2022-09-12T19:06:48.774 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj 2022-09-12T19:06:48.776 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dzd4UqxQnNFEdXMd5MkVZ6Zir3g= 2022-09-12T19:06:48.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dzd4UqxQnNFEdXMd5MkVZ6Zir3g=', 'Content-Length': '0'} 2022-09-12T19:06:48.781 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070559f6f4957e033-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.781 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.782 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.783 DEBUG:boto:Method: HEAD 2022-09-12T19:06:48.783 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:48.784 DEBUG:boto:Data: 2022-09-12T19:06:48.784 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.785 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.785 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.786 DEBUG:boto:Params: {} 2022-09-12T19:06:48.786 DEBUG:boto:Token: None 2022-09-12T19:06:48.787 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj 2022-09-12T19:06:48.787 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dzd4UqxQnNFEdXMd5MkVZ6Zir3g= 2022-09-12T19:06:48.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dzd4UqxQnNFEdXMd5MkVZ6Zir3g=', 'Content-Length': '0'} 2022-09-12T19:06:48.792 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000617520b81b9b4965-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:48.793 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.794 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.794 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.795 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.795 DEBUG:boto:Method: GET 2022-09-12T19:06:48.796 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.796 DEBUG:boto:Data: 2022-09-12T19:06:48.797 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.797 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.798 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.799 DEBUG:boto:Params: {} 2022-09-12T19:06:48.799 DEBUG:boto:Token: None 2022-09-12T19:06:48.800 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.801 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Vc/yEKLSLT1WQbl6XMHi7EONcV8= 2022-09-12T19:06:48.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Vc/yEKLSLT1WQbl6XMHi7EONcV8=', 'Content-Length': '0'} 2022-09-12T19:06:48.804 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d58c9ee8b001c40-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.846 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.847 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.847 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.848 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.849 DEBUG:boto:Method: GET 2022-09-12T19:06:48.850 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.851 DEBUG:boto:Data: 2022-09-12T19:06:48.852 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.853 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.854 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.855 DEBUG:boto:Params: {} 2022-09-12T19:06:48.856 DEBUG:boto:Token: None 2022-09-12T19:06:48.857 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:48.858 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Vc/yEKLSLT1WQbl6XMHi7EONcV8= 2022-09-12T19:06:48.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Vc/yEKLSLT1WQbl6XMHi7EONcV8=', 'Content-Length': '0'} 2022-09-12T19:06:48.862 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a0d76fecbc9f39e-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.863 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:48.864 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:48.864 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.865 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.866 DEBUG:boto:Method: GET 2022-09-12T19:06:48.866 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:48.867 DEBUG:boto:Data: 2022-09-12T19:06:48.867 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.868 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.868 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.869 DEBUG:boto:Params: {} 2022-09-12T19:06:48.869 DEBUG:boto:Token: None 2022-09-12T19:06:48.870 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/ 2022-09-12T19:06:48.871 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI= 2022-09-12T19:06:48.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI=', 'Content-Length': '0'} 2022-09-12T19:06:48.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d21d2fcfabea687d-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.875 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:48.876 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:48.877 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:48.877 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.878 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:48.878 DEBUG:boto:Method: GET 2022-09-12T19:06:48.879 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:48.879 DEBUG:boto:Data: 2022-09-12T19:06:48.880 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.880 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.881 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.881 DEBUG:boto:Params: {} 2022-09-12T19:06:48.882 DEBUG:boto:Token: None 2022-09-12T19:06:48.884 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/ 2022-09-12T19:06:48.884 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI= 2022-09-12T19:06:48.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/kvsKpCOy+glHssSwp9U6DH3QPI=', 'Content-Length': '0'} 2022-09-12T19:06:48.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000060b840523e3004b-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.930 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:48.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:48.932 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.932 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.933 DEBUG:boto:path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:48.933 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:48.934 DEBUG:boto:Method: GET 2022-09-12T19:06:48.934 DEBUG:boto:Path: /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:48.935 DEBUG:boto:Data: 2022-09-12T19:06:48.936 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.936 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.937 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.937 DEBUG:boto:Params: {} 2022-09-12T19:06:48.938 DEBUG:boto:Token: None 2022-09-12T19:06:48.938 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:48.939 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OuN6kA7/0y0yEFwHAOKQlTiUxPo= 2022-09-12T19:06:48.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OuN6kA7/0y0yEFwHAOKQlTiUxPo=', 'Content-Length': '0'} 2022-09-12T19:06:48.942 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005669049237b3a914-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.943 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.944 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.945 DEBUG:boto:path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:48.945 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:48.946 DEBUG:boto:Method: GET 2022-09-12T19:06:48.946 DEBUG:boto:Path: /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:48.947 DEBUG:boto:Data: 2022-09-12T19:06:48.947 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.948 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.949 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.949 DEBUG:boto:Params: {} 2022-09-12T19:06:48.950 DEBUG:boto:Token: None 2022-09-12T19:06:48.951 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:48.952 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OuN6kA7/0y0yEFwHAOKQlTiUxPo= 2022-09-12T19:06:48.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OuN6kA7/0y0yEFwHAOKQlTiUxPo=', 'Content-Length': '0'} 2022-09-12T19:06:48.955 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a598a53efad6225-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.956 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.957 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.957 DEBUG:boto:path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:48.958 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:48.958 DEBUG:boto:Method: HEAD 2022-09-12T19:06:48.959 DEBUG:boto:Path: /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:48.959 DEBUG:boto:Data: 2022-09-12T19:06:48.960 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.960 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.961 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.961 DEBUG:boto:Params: {} 2022-09-12T19:06:48.962 DEBUG:boto:Token: None 2022-09-12T19:06:48.962 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:48.963 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rd6anIJEwh+4f0jEL3ydb4O4fj8= 2022-09-12T19:06:48.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rd6anIJEwh+4f0jEL3ydb4O4fj8=', 'Content-Length': '0'} 2022-09-12T19:06:48.966 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000642f18ca0f4e3765-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.967 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.967 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.968 DEBUG:boto:path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:48.968 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:48.969 DEBUG:boto:Method: HEAD 2022-09-12T19:06:48.969 DEBUG:boto:Path: /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:48.970 DEBUG:boto:Data: 2022-09-12T19:06:48.971 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.971 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:48.972 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.972 DEBUG:boto:Params: {} 2022-09-12T19:06:48.973 DEBUG:boto:Token: None 2022-09-12T19:06:48.974 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:48.974 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rd6anIJEwh+4f0jEL3ydb4O4fj8= 2022-09-12T19:06:48.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rd6anIJEwh+4f0jEL3ydb4O4fj8=', 'Content-Length': '0'} 2022-09-12T19:06:48.977 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007005c5135585ad49-00631f8348-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:48.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:48.978 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:48.979 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:48.980 DEBUG:boto:path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:48.980 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:48.981 DEBUG:boto:Method: GET 2022-09-12T19:06:48.981 DEBUG:boto:Path: /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:48.982 DEBUG:boto:Data: 2022-09-12T19:06:48.982 DEBUG:boto:Headers: {} 2022-09-12T19:06:48.983 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:48.983 DEBUG:boto:Port: 8000 2022-09-12T19:06:48.984 DEBUG:boto:Params: {} 2022-09-12T19:06:48.984 DEBUG:boto:Token: None 2022-09-12T19:06:48.985 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:48 GMT /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:48.986 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OuN6kA7/0y0yEFwHAOKQlTiUxPo= 2022-09-12T19:06:48.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OuN6kA7/0y0yEFwHAOKQlTiUxPo=', 'Content-Length': '0'} 2022-09-12T19:06:48.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006eb284e0d99b45d4-00631f8348-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.030 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.031 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.031 DEBUG:boto:path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:49.032 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:49.033 DEBUG:boto:Method: GET 2022-09-12T19:06:49.033 DEBUG:boto:Path: /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:49.034 DEBUG:boto:Data: 2022-09-12T19:06:49.034 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.035 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.035 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.036 DEBUG:boto:Params: {} 2022-09-12T19:06:49.037 DEBUG:boto:Token: None 2022-09-12T19:06:49.037 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:49.038 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qM0SuJ54wkqcc9uqUqo7IaTcJFM= 2022-09-12T19:06:49.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qM0SuJ54wkqcc9uqUqo7IaTcJFM=', 'Content-Length': '0'} 2022-09-12T19:06:49.041 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a28f0a5eafaa0ebf-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.042 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:49.042 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:49.043 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.043 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.044 DEBUG:boto:Method: GET 2022-09-12T19:06:49.045 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:49.045 DEBUG:boto:Data: 2022-09-12T19:06:49.046 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.046 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.047 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.048 DEBUG:boto:Params: {} 2022-09-12T19:06:49.048 DEBUG:boto:Token: None 2022-09-12T19:06:49.049 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/ 2022-09-12T19:06:49.050 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q= 2022-09-12T19:06:49.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q=', 'Content-Length': '0'} 2022-09-12T19:06:49.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040adbcb699cb3931-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.054 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:49.055 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:49.056 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:49.056 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.057 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.058 DEBUG:boto:Method: GET 2022-09-12T19:06:49.059 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:49.059 DEBUG:boto:Data: 2022-09-12T19:06:49.060 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.061 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.061 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.062 DEBUG:boto:Params: {} 2022-09-12T19:06:49.063 DEBUG:boto:Token: None 2022-09-12T19:06:49.064 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/ 2022-09-12T19:06:49.065 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q= 2022-09-12T19:06:49.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q=', 'Content-Length': '0'} 2022-09-12T19:06:49.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7e2ebb5c63a1503-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.070 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:49.071 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.071 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.072 DEBUG:boto:Method: HEAD 2022-09-12T19:06:49.072 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:49.073 DEBUG:boto:Data: 2022-09-12T19:06:49.074 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.074 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.075 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.076 DEBUG:boto:Params: {} 2022-09-12T19:06:49.077 DEBUG:boto:Token: None 2022-09-12T19:06:49.079 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj 2022-09-12T19:06:49.080 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m+oU3ZetMFMXG9cpQiFhLij5WxI= 2022-09-12T19:06:49.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m+oU3ZetMFMXG9cpQiFhLij5WxI=', 'Content-Length': '0'} 2022-09-12T19:06:49.085 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d83bbe16938de44a-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.086 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.087 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.087 DEBUG:boto:Method: HEAD 2022-09-12T19:06:49.088 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:49.088 DEBUG:boto:Data: 2022-09-12T19:06:49.089 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.090 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.090 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.091 DEBUG:boto:Params: {} 2022-09-12T19:06:49.092 DEBUG:boto:Token: None 2022-09-12T19:06:49.092 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj 2022-09-12T19:06:49.093 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m+oU3ZetMFMXG9cpQiFhLij5WxI= 2022-09-12T19:06:49.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m+oU3ZetMFMXG9cpQiFhLij5WxI=', 'Content-Length': '0'} 2022-09-12T19:06:49.097 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c508ed4106003262-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:49.098 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.099 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.099 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.100 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.100 DEBUG:boto:Method: GET 2022-09-12T19:06:49.101 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.101 DEBUG:boto:Data: 2022-09-12T19:06:49.102 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.102 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.103 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.104 DEBUG:boto:Params: {} 2022-09-12T19:06:49.104 DEBUG:boto:Token: None 2022-09-12T19:06:49.105 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.105 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yvjX/QRIcpCBP5BFCEnqaAGHZvk= 2022-09-12T19:06:49.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yvjX/QRIcpCBP5BFCEnqaAGHZvk=', 'Content-Length': '0'} 2022-09-12T19:06:49.108 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bdde632c2f0b404f-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.109 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.110 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.110 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.111 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.111 DEBUG:boto:Method: GET 2022-09-12T19:06:49.112 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.113 DEBUG:boto:Data: 2022-09-12T19:06:49.113 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.114 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.115 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.115 DEBUG:boto:Params: {} 2022-09-12T19:06:49.116 DEBUG:boto:Token: None 2022-09-12T19:06:49.116 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.117 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yvjX/QRIcpCBP5BFCEnqaAGHZvk= 2022-09-12T19:06:49.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yvjX/QRIcpCBP5BFCEnqaAGHZvk=', 'Content-Length': '0'} 2022-09-12T19:06:49.120 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f7870607a53a863-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.122 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:49.123 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:49.124 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.124 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.126 DEBUG:boto:Method: GET 2022-09-12T19:06:49.127 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:49.127 DEBUG:boto:Data: 2022-09-12T19:06:49.128 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.129 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.130 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.131 DEBUG:boto:Params: {} 2022-09-12T19:06:49.132 DEBUG:boto:Token: None 2022-09-12T19:06:49.132 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/ 2022-09-12T19:06:49.133 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q= 2022-09-12T19:06:49.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q=', 'Content-Length': '0'} 2022-09-12T19:06:49.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000059646a0039491a1-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.137 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:49.138 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:49.139 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:49.139 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.140 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.141 DEBUG:boto:Method: GET 2022-09-12T19:06:49.141 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:49.142 DEBUG:boto:Data: 2022-09-12T19:06:49.143 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.144 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.145 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.147 DEBUG:boto:Params: {} 2022-09-12T19:06:49.148 DEBUG:boto:Token: None 2022-09-12T19:06:49.149 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/ 2022-09-12T19:06:49.150 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q= 2022-09-12T19:06:49.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q=', 'Content-Length': '0'} 2022-09-12T19:06:49.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009de0da1f329f9917-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.156 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:49.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:49.158 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.159 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.160 DEBUG:boto:path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:49.161 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:49.162 DEBUG:boto:Method: GET 2022-09-12T19:06:49.163 DEBUG:boto:Path: /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:49.164 DEBUG:boto:Data: 2022-09-12T19:06:49.165 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.166 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.167 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.168 DEBUG:boto:Params: {} 2022-09-12T19:06:49.169 DEBUG:boto:Token: None 2022-09-12T19:06:49.170 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:49.172 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wocwHRc90kbcGHq/6hpl4X9VTwY= 2022-09-12T19:06:49.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wocwHRc90kbcGHq/6hpl4X9VTwY=', 'Content-Length': '0'} 2022-09-12T19:06:49.175 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'e.6ruI53roqy4mmyPgzQ1kqMdajruFA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b70d1d821fe6d687-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.177 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.178 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.179 DEBUG:boto:path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:49.180 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:49.181 DEBUG:boto:Method: GET 2022-09-12T19:06:49.182 DEBUG:boto:Path: /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:49.183 DEBUG:boto:Data: 2022-09-12T19:06:49.183 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.184 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.184 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.185 DEBUG:boto:Params: {} 2022-09-12T19:06:49.185 DEBUG:boto:Token: None 2022-09-12T19:06:49.186 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:49.187 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wocwHRc90kbcGHq/6hpl4X9VTwY= 2022-09-12T19:06:49.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wocwHRc90kbcGHq/6hpl4X9VTwY=', 'Content-Length': '0'} 2022-09-12T19:06:49.189 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'e.6ruI53roqy4mmyPgzQ1kqMdajruFA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e08ad2dbf210a2a-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.234 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.235 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.235 DEBUG:boto:path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:49.236 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:49.236 DEBUG:boto:Method: HEAD 2022-09-12T19:06:49.237 DEBUG:boto:Path: /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:49.237 DEBUG:boto:Data: 2022-09-12T19:06:49.238 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.238 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.239 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.239 DEBUG:boto:Params: {} 2022-09-12T19:06:49.240 DEBUG:boto:Token: None 2022-09-12T19:06:49.241 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:49.241 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rHvWdwhVQiKYiu4lm+eF2ZZnhjM= 2022-09-12T19:06:49.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rHvWdwhVQiKYiu4lm+eF2ZZnhjM=', 'Content-Length': '0'} 2022-09-12T19:06:49.245 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'e.6ruI53roqy4mmyPgzQ1kqMdajruFA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075f4e824eb97caee-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.247 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.248 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.248 DEBUG:boto:path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:49.249 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:49.250 DEBUG:boto:Method: HEAD 2022-09-12T19:06:49.251 DEBUG:boto:Path: /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:49.252 DEBUG:boto:Data: 2022-09-12T19:06:49.253 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.254 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.255 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.256 DEBUG:boto:Params: {} 2022-09-12T19:06:49.257 DEBUG:boto:Token: None 2022-09-12T19:06:49.258 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:49.259 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rHvWdwhVQiKYiu4lm+eF2ZZnhjM= 2022-09-12T19:06:49.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rHvWdwhVQiKYiu4lm+eF2ZZnhjM=', 'Content-Length': '0'} 2022-09-12T19:06:49.263 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'e.6ruI53roqy4mmyPgzQ1kqMdajruFA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dab29e279e93a061-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:49.264 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.265 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.265 DEBUG:boto:path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:49.266 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:49.266 DEBUG:boto:Method: GET 2022-09-12T19:06:49.267 DEBUG:boto:Path: /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:49.268 DEBUG:boto:Data: 2022-09-12T19:06:49.268 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.269 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.270 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.270 DEBUG:boto:Params: {} 2022-09-12T19:06:49.271 DEBUG:boto:Token: None 2022-09-12T19:06:49.272 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:49.272 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wocwHRc90kbcGHq/6hpl4X9VTwY= 2022-09-12T19:06:49.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wocwHRc90kbcGHq/6hpl4X9VTwY=', 'Content-Length': '0'} 2022-09-12T19:06:49.275 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'e.6ruI53roqy4mmyPgzQ1kqMdajruFA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075bfd90f35896e01-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.276 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.276 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.277 DEBUG:boto:path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:49.277 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:49.278 DEBUG:boto:Method: GET 2022-09-12T19:06:49.278 DEBUG:boto:Path: /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:49.279 DEBUG:boto:Data: 2022-09-12T19:06:49.279 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.280 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.280 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.281 DEBUG:boto:Params: {} 2022-09-12T19:06:49.281 DEBUG:boto:Token: None 2022-09-12T19:06:49.282 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:49.283 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wocwHRc90kbcGHq/6hpl4X9VTwY= 2022-09-12T19:06:49.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wocwHRc90kbcGHq/6hpl4X9VTwY=', 'Content-Length': '0'} 2022-09-12T19:06:49.285 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'e.6ruI53roqy4mmyPgzQ1kqMdajruFA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000335e67082803984c-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.286 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:49.287 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:49.287 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.288 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.288 DEBUG:boto:Method: GET 2022-09-12T19:06:49.289 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:49.289 DEBUG:boto:Data: 2022-09-12T19:06:49.290 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.290 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.291 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.291 DEBUG:boto:Params: {} 2022-09-12T19:06:49.292 DEBUG:boto:Token: None 2022-09-12T19:06:49.293 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/ 2022-09-12T19:06:49.293 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q= 2022-09-12T19:06:49.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q=', 'Content-Length': '0'} 2022-09-12T19:06:49.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f4d9f3284e7e1f9-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.297 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:49.298 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:49.299 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:49.299 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.300 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.300 DEBUG:boto:Method: GET 2022-09-12T19:06:49.301 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:49.301 DEBUG:boto:Data: 2022-09-12T19:06:49.302 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.302 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.303 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.304 DEBUG:boto:Params: {} 2022-09-12T19:06:49.304 DEBUG:boto:Token: None 2022-09-12T19:06:49.305 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/ 2022-09-12T19:06:49.305 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q= 2022-09-12T19:06:49.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q=', 'Content-Length': '0'} 2022-09-12T19:06:49.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6600d38de8bcc54-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.310 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:49.310 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.311 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.312 DEBUG:boto:Method: HEAD 2022-09-12T19:06:49.312 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:49.313 DEBUG:boto:Data: 2022-09-12T19:06:49.313 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.314 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.314 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.315 DEBUG:boto:Params: {} 2022-09-12T19:06:49.315 DEBUG:boto:Token: None 2022-09-12T19:06:49.316 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj 2022-09-12T19:06:49.317 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m+oU3ZetMFMXG9cpQiFhLij5WxI= 2022-09-12T19:06:49.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m+oU3ZetMFMXG9cpQiFhLij5WxI=', 'Content-Length': '0'} 2022-09-12T19:06:49.321 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9ef20b07cdd869c-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.322 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.322 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.323 DEBUG:boto:Method: HEAD 2022-09-12T19:06:49.323 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:49.324 DEBUG:boto:Data: 2022-09-12T19:06:49.324 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.325 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.325 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.326 DEBUG:boto:Params: {} 2022-09-12T19:06:49.326 DEBUG:boto:Token: None 2022-09-12T19:06:49.327 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj 2022-09-12T19:06:49.327 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m+oU3ZetMFMXG9cpQiFhLij5WxI= 2022-09-12T19:06:49.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m+oU3ZetMFMXG9cpQiFhLij5WxI=', 'Content-Length': '0'} 2022-09-12T19:06:49.332 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000021ecf9ea9263e361-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:49.333 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.334 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.334 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.335 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.335 DEBUG:boto:Method: GET 2022-09-12T19:06:49.336 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.336 DEBUG:boto:Data: 2022-09-12T19:06:49.337 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.337 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.338 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.339 DEBUG:boto:Params: {} 2022-09-12T19:06:49.339 DEBUG:boto:Token: None 2022-09-12T19:06:49.340 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.341 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yvjX/QRIcpCBP5BFCEnqaAGHZvk= 2022-09-12T19:06:49.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yvjX/QRIcpCBP5BFCEnqaAGHZvk=', 'Content-Length': '0'} 2022-09-12T19:06:49.345 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003749f37f52fb5d5d-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.346 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.347 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.348 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.349 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.350 DEBUG:boto:Method: GET 2022-09-12T19:06:49.351 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.352 DEBUG:boto:Data: 2022-09-12T19:06:49.353 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.353 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.354 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.354 DEBUG:boto:Params: {} 2022-09-12T19:06:49.355 DEBUG:boto:Token: None 2022-09-12T19:06:49.356 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.356 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yvjX/QRIcpCBP5BFCEnqaAGHZvk= 2022-09-12T19:06:49.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yvjX/QRIcpCBP5BFCEnqaAGHZvk=', 'Content-Length': '0'} 2022-09-12T19:06:49.359 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000707149e679920bb5-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.406 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:49.407 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:49.408 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.409 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.410 DEBUG:boto:Method: GET 2022-09-12T19:06:49.411 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:49.412 DEBUG:boto:Data: 2022-09-12T19:06:49.413 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.414 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.414 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.415 DEBUG:boto:Params: {} 2022-09-12T19:06:49.416 DEBUG:boto:Token: None 2022-09-12T19:06:49.416 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/ 2022-09-12T19:06:49.417 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q= 2022-09-12T19:06:49.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q=', 'Content-Length': '0'} 2022-09-12T19:06:49.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000371f8d85948af4ed-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.422 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:49.423 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:49.423 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:49.424 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.424 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.425 DEBUG:boto:Method: GET 2022-09-12T19:06:49.425 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:49.426 DEBUG:boto:Data: 2022-09-12T19:06:49.427 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.427 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.428 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.428 DEBUG:boto:Params: {} 2022-09-12T19:06:49.429 DEBUG:boto:Token: None 2022-09-12T19:06:49.430 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/ 2022-09-12T19:06:49.431 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q= 2022-09-12T19:06:49.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q=', 'Content-Length': '0'} 2022-09-12T19:06:49.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a74aa2d58cfdd08-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.435 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:49.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:49.436 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.437 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.437 DEBUG:boto:path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:49.438 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:49.439 DEBUG:boto:Method: GET 2022-09-12T19:06:49.439 DEBUG:boto:Path: /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:49.440 DEBUG:boto:Data: 2022-09-12T19:06:49.441 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.441 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.442 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.443 DEBUG:boto:Params: {} 2022-09-12T19:06:49.443 DEBUG:boto:Token: None 2022-09-12T19:06:49.444 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:49.445 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jXabdca1T8mkJVkwValCVQxwxNw= 2022-09-12T19:06:49.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jXabdca1T8mkJVkwValCVQxwxNw=', 'Content-Length': '0'} 2022-09-12T19:06:49.448 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c64e796d7afa29c-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.450 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.451 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.451 DEBUG:boto:path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:49.452 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:49.453 DEBUG:boto:Method: GET 2022-09-12T19:06:49.454 DEBUG:boto:Path: /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:49.455 DEBUG:boto:Data: 2022-09-12T19:06:49.456 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.457 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.458 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.458 DEBUG:boto:Params: {} 2022-09-12T19:06:49.459 DEBUG:boto:Token: None 2022-09-12T19:06:49.460 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:49.460 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jXabdca1T8mkJVkwValCVQxwxNw= 2022-09-12T19:06:49.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jXabdca1T8mkJVkwValCVQxwxNw=', 'Content-Length': '0'} 2022-09-12T19:06:49.463 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b48a2280df9b532-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.464 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.465 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.465 DEBUG:boto:path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:49.466 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:49.467 DEBUG:boto:Method: HEAD 2022-09-12T19:06:49.467 DEBUG:boto:Path: /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:49.468 DEBUG:boto:Data: 2022-09-12T19:06:49.468 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.469 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.469 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.470 DEBUG:boto:Params: {} 2022-09-12T19:06:49.470 DEBUG:boto:Token: None 2022-09-12T19:06:49.471 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:49.472 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bIm1JWme4pKUmjXpF90lJYmxXzY= 2022-09-12T19:06:49.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bIm1JWme4pKUmjXpF90lJYmxXzY=', 'Content-Length': '0'} 2022-09-12T19:06:49.474 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010b5f4bcf0d5c86c-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.475 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.476 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.476 DEBUG:boto:path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:49.477 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:49.477 DEBUG:boto:Method: HEAD 2022-09-12T19:06:49.478 DEBUG:boto:Path: /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:49.478 DEBUG:boto:Data: 2022-09-12T19:06:49.479 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.479 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.480 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.480 DEBUG:boto:Params: {} 2022-09-12T19:06:49.481 DEBUG:boto:Token: None 2022-09-12T19:06:49.482 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:49.482 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bIm1JWme4pKUmjXpF90lJYmxXzY= 2022-09-12T19:06:49.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bIm1JWme4pKUmjXpF90lJYmxXzY=', 'Content-Length': '0'} 2022-09-12T19:06:49.485 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002fb244ce82be4369-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:49.487 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.487 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.488 DEBUG:boto:path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:49.488 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:49.489 DEBUG:boto:Method: GET 2022-09-12T19:06:49.489 DEBUG:boto:Path: /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:49.490 DEBUG:boto:Data: 2022-09-12T19:06:49.490 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.491 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.491 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.492 DEBUG:boto:Params: {} 2022-09-12T19:06:49.493 DEBUG:boto:Token: None 2022-09-12T19:06:49.493 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:49.494 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jXabdca1T8mkJVkwValCVQxwxNw= 2022-09-12T19:06:49.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jXabdca1T8mkJVkwValCVQxwxNw=', 'Content-Length': '0'} 2022-09-12T19:06:49.497 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000730f1a1bfcbb9e5b-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.497 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.498 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.499 DEBUG:boto:path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:49.499 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:49.500 DEBUG:boto:Method: GET 2022-09-12T19:06:49.500 DEBUG:boto:Path: /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:49.501 DEBUG:boto:Data: 2022-09-12T19:06:49.501 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.502 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.502 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.503 DEBUG:boto:Params: {} 2022-09-12T19:06:49.503 DEBUG:boto:Token: None 2022-09-12T19:06:49.504 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:49.505 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jXabdca1T8mkJVkwValCVQxwxNw= 2022-09-12T19:06:49.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jXabdca1T8mkJVkwValCVQxwxNw=', 'Content-Length': '0'} 2022-09-12T19:06:49.508 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007adec2ca8213b44f-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.508 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:49.509 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:49.509 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.510 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.511 DEBUG:boto:Method: GET 2022-09-12T19:06:49.511 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:49.512 DEBUG:boto:Data: 2022-09-12T19:06:49.512 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.513 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.513 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.514 DEBUG:boto:Params: {} 2022-09-12T19:06:49.514 DEBUG:boto:Token: None 2022-09-12T19:06:49.515 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/ 2022-09-12T19:06:49.516 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q= 2022-09-12T19:06:49.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q=', 'Content-Length': '0'} 2022-09-12T19:06:49.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e44b863423852356-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.520 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:49.521 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:49.521 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:49.522 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.522 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.523 DEBUG:boto:Method: GET 2022-09-12T19:06:49.523 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:49.524 DEBUG:boto:Data: 2022-09-12T19:06:49.524 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.525 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.526 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.527 DEBUG:boto:Params: {} 2022-09-12T19:06:49.528 DEBUG:boto:Token: None 2022-09-12T19:06:49.529 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/ 2022-09-12T19:06:49.530 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q= 2022-09-12T19:06:49.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q=', 'Content-Length': '0'} 2022-09-12T19:06:49.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b018d32b9b45781-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.534 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:49.535 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.536 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.536 DEBUG:boto:Method: HEAD 2022-09-12T19:06:49.537 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:49.537 DEBUG:boto:Data: 2022-09-12T19:06:49.538 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.538 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.539 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.539 DEBUG:boto:Params: {} 2022-09-12T19:06:49.540 DEBUG:boto:Token: None 2022-09-12T19:06:49.540 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj 2022-09-12T19:06:49.541 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m+oU3ZetMFMXG9cpQiFhLij5WxI= 2022-09-12T19:06:49.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m+oU3ZetMFMXG9cpQiFhLij5WxI=', 'Content-Length': '0'} 2022-09-12T19:06:49.545 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000926b3111ad1a8549-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.546 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.546 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.547 DEBUG:boto:Method: HEAD 2022-09-12T19:06:49.547 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:49.548 DEBUG:boto:Data: 2022-09-12T19:06:49.548 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.549 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.549 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.550 DEBUG:boto:Params: {} 2022-09-12T19:06:49.550 DEBUG:boto:Token: None 2022-09-12T19:06:49.551 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj 2022-09-12T19:06:49.551 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m+oU3ZetMFMXG9cpQiFhLij5WxI= 2022-09-12T19:06:49.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m+oU3ZetMFMXG9cpQiFhLij5WxI=', 'Content-Length': '0'} 2022-09-12T19:06:49.556 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000defa839b25c612a8-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:49.557 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.557 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.558 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.558 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.559 DEBUG:boto:Method: GET 2022-09-12T19:06:49.560 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.560 DEBUG:boto:Data: 2022-09-12T19:06:49.561 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.561 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.562 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.562 DEBUG:boto:Params: {} 2022-09-12T19:06:49.563 DEBUG:boto:Token: None 2022-09-12T19:06:49.563 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.564 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yvjX/QRIcpCBP5BFCEnqaAGHZvk= 2022-09-12T19:06:49.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yvjX/QRIcpCBP5BFCEnqaAGHZvk=', 'Content-Length': '0'} 2022-09-12T19:06:49.567 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000438cdf559d029b19-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.610 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.611 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.612 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.612 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.613 DEBUG:boto:Method: GET 2022-09-12T19:06:49.614 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.614 DEBUG:boto:Data: 2022-09-12T19:06:49.615 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.615 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.616 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.616 DEBUG:boto:Params: {} 2022-09-12T19:06:49.617 DEBUG:boto:Token: None 2022-09-12T19:06:49.618 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.618 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yvjX/QRIcpCBP5BFCEnqaAGHZvk= 2022-09-12T19:06:49.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yvjX/QRIcpCBP5BFCEnqaAGHZvk=', 'Content-Length': '0'} 2022-09-12T19:06:49.621 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e426a744bac82375-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.622 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:49.623 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:49.623 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.624 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.624 DEBUG:boto:Method: GET 2022-09-12T19:06:49.625 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:49.625 DEBUG:boto:Data: 2022-09-12T19:06:49.626 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.627 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.627 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.628 DEBUG:boto:Params: {} 2022-09-12T19:06:49.628 DEBUG:boto:Token: None 2022-09-12T19:06:49.629 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/ 2022-09-12T19:06:49.629 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q= 2022-09-12T19:06:49.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q=', 'Content-Length': '0'} 2022-09-12T19:06:49.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001780af4e24e831f3-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.635 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:49.637 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:49.638 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:49.639 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.640 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.641 DEBUG:boto:Method: GET 2022-09-12T19:06:49.642 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:49.643 DEBUG:boto:Data: 2022-09-12T19:06:49.644 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.645 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.645 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.646 DEBUG:boto:Params: {} 2022-09-12T19:06:49.647 DEBUG:boto:Token: None 2022-09-12T19:06:49.647 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/ 2022-09-12T19:06:49.648 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q= 2022-09-12T19:06:49.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q=', 'Content-Length': '0'} 2022-09-12T19:06:49.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000704fba73c376fcfa-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.653 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:49.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:49.655 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.655 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.656 DEBUG:boto:path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:49.656 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:49.657 DEBUG:boto:Method: GET 2022-09-12T19:06:49.657 DEBUG:boto:Path: /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:49.658 DEBUG:boto:Data: 2022-09-12T19:06:49.658 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.659 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.659 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.660 DEBUG:boto:Params: {} 2022-09-12T19:06:49.660 DEBUG:boto:Token: None 2022-09-12T19:06:49.661 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:49.662 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0E91AvmoThSuHnqGBLM6AVIJfvA= 2022-09-12T19:06:49.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0E91AvmoThSuHnqGBLM6AVIJfvA=', 'Content-Length': '0'} 2022-09-12T19:06:49.665 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'lnpiHoSs91QVujLSWqHhvEGzZrF3-sA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a99c0d9e6216c67b-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.666 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.666 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.667 DEBUG:boto:path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:49.667 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:49.668 DEBUG:boto:Method: GET 2022-09-12T19:06:49.668 DEBUG:boto:Path: /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:49.669 DEBUG:boto:Data: 2022-09-12T19:06:49.670 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.670 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.671 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.671 DEBUG:boto:Params: {} 2022-09-12T19:06:49.672 DEBUG:boto:Token: None 2022-09-12T19:06:49.672 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:49.673 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0E91AvmoThSuHnqGBLM6AVIJfvA= 2022-09-12T19:06:49.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0E91AvmoThSuHnqGBLM6AVIJfvA=', 'Content-Length': '0'} 2022-09-12T19:06:49.676 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'lnpiHoSs91QVujLSWqHhvEGzZrF3-sA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000832bd8c55a9b886-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.677 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.677 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.678 DEBUG:boto:path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:49.678 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:49.679 DEBUG:boto:Method: HEAD 2022-09-12T19:06:49.679 DEBUG:boto:Path: /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:49.680 DEBUG:boto:Data: 2022-09-12T19:06:49.680 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.681 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.681 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.682 DEBUG:boto:Params: {} 2022-09-12T19:06:49.682 DEBUG:boto:Token: None 2022-09-12T19:06:49.683 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:49.684 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YMtyinv7QEuJpiNTPneslnFRsB0= 2022-09-12T19:06:49.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YMtyinv7QEuJpiNTPneslnFRsB0=', 'Content-Length': '0'} 2022-09-12T19:06:49.695 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'lnpiHoSs91QVujLSWqHhvEGzZrF3-sA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016ac5ce5c15ce564-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.696 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.697 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.697 DEBUG:boto:path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:49.698 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:49.698 DEBUG:boto:Method: HEAD 2022-09-12T19:06:49.699 DEBUG:boto:Path: /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:49.699 DEBUG:boto:Data: 2022-09-12T19:06:49.700 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.700 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.701 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.702 DEBUG:boto:Params: {} 2022-09-12T19:06:49.703 DEBUG:boto:Token: None 2022-09-12T19:06:49.703 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:49.704 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YMtyinv7QEuJpiNTPneslnFRsB0= 2022-09-12T19:06:49.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YMtyinv7QEuJpiNTPneslnFRsB0=', 'Content-Length': '0'} 2022-09-12T19:06:49.707 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'lnpiHoSs91QVujLSWqHhvEGzZrF3-sA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de21bd4bf9a2104e-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:49.708 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.708 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.709 DEBUG:boto:path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:49.710 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:49.710 DEBUG:boto:Method: GET 2022-09-12T19:06:49.711 DEBUG:boto:Path: /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:49.711 DEBUG:boto:Data: 2022-09-12T19:06:49.712 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.712 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.713 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.713 DEBUG:boto:Params: {} 2022-09-12T19:06:49.714 DEBUG:boto:Token: None 2022-09-12T19:06:49.715 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:49.715 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0E91AvmoThSuHnqGBLM6AVIJfvA= 2022-09-12T19:06:49.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0E91AvmoThSuHnqGBLM6AVIJfvA=', 'Content-Length': '0'} 2022-09-12T19:06:49.719 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'lnpiHoSs91QVujLSWqHhvEGzZrF3-sA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000012813735f89aec3d-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.762 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.763 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.764 DEBUG:boto:path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:49.764 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:49.765 DEBUG:boto:Method: GET 2022-09-12T19:06:49.765 DEBUG:boto:Path: /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:49.766 DEBUG:boto:Data: 2022-09-12T19:06:49.766 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.767 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.767 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.768 DEBUG:boto:Params: {} 2022-09-12T19:06:49.769 DEBUG:boto:Token: None 2022-09-12T19:06:49.769 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:49.770 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0E91AvmoThSuHnqGBLM6AVIJfvA= 2022-09-12T19:06:49.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0E91AvmoThSuHnqGBLM6AVIJfvA=', 'Content-Length': '0'} 2022-09-12T19:06:49.773 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'lnpiHoSs91QVujLSWqHhvEGzZrF3-sA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000018e5323c7af7601-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.774 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:49.775 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:49.775 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.776 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.776 DEBUG:boto:Method: GET 2022-09-12T19:06:49.777 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:49.777 DEBUG:boto:Data: 2022-09-12T19:06:49.778 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.779 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.780 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.780 DEBUG:boto:Params: {} 2022-09-12T19:06:49.781 DEBUG:boto:Token: None 2022-09-12T19:06:49.781 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/ 2022-09-12T19:06:49.782 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q= 2022-09-12T19:06:49.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q=', 'Content-Length': '0'} 2022-09-12T19:06:49.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d94d1f21474dc20-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.787 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:49.788 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:49.788 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:49.789 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.789 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.790 DEBUG:boto:Method: GET 2022-09-12T19:06:49.790 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:49.791 DEBUG:boto:Data: 2022-09-12T19:06:49.791 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.792 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.793 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.793 DEBUG:boto:Params: {} 2022-09-12T19:06:49.794 DEBUG:boto:Token: None 2022-09-12T19:06:49.795 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/ 2022-09-12T19:06:49.795 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q= 2022-09-12T19:06:49.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q=', 'Content-Length': '0'} 2022-09-12T19:06:49.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000597bb20da9e0046e-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.800 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:49.800 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.801 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.802 DEBUG:boto:Method: HEAD 2022-09-12T19:06:49.802 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:49.803 DEBUG:boto:Data: 2022-09-12T19:06:49.803 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.804 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.804 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.805 DEBUG:boto:Params: {} 2022-09-12T19:06:49.806 DEBUG:boto:Token: None 2022-09-12T19:06:49.806 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj 2022-09-12T19:06:49.807 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m+oU3ZetMFMXG9cpQiFhLij5WxI= 2022-09-12T19:06:49.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m+oU3ZetMFMXG9cpQiFhLij5WxI=', 'Content-Length': '0'} 2022-09-12T19:06:49.812 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3314f0ac7b6e34c-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.813 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.814 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.814 DEBUG:boto:Method: HEAD 2022-09-12T19:06:49.815 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:49.816 DEBUG:boto:Data: 2022-09-12T19:06:49.816 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.817 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.817 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.818 DEBUG:boto:Params: {} 2022-09-12T19:06:49.819 DEBUG:boto:Token: None 2022-09-12T19:06:49.820 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj 2022-09-12T19:06:49.821 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m+oU3ZetMFMXG9cpQiFhLij5WxI= 2022-09-12T19:06:49.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m+oU3ZetMFMXG9cpQiFhLij5WxI=', 'Content-Length': '0'} 2022-09-12T19:06:49.826 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d3b947dd035b5a5-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:49.828 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.829 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.830 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.831 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.831 DEBUG:boto:Method: GET 2022-09-12T19:06:49.832 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.832 DEBUG:boto:Data: 2022-09-12T19:06:49.833 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.834 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.834 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.835 DEBUG:boto:Params: {} 2022-09-12T19:06:49.836 DEBUG:boto:Token: None 2022-09-12T19:06:49.836 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.837 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yvjX/QRIcpCBP5BFCEnqaAGHZvk= 2022-09-12T19:06:49.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yvjX/QRIcpCBP5BFCEnqaAGHZvk=', 'Content-Length': '0'} 2022-09-12T19:06:49.840 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc05337402d9102c-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.841 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.842 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.842 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.843 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.843 DEBUG:boto:Method: GET 2022-09-12T19:06:49.844 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.844 DEBUG:boto:Data: 2022-09-12T19:06:49.845 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.845 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.846 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.847 DEBUG:boto:Params: {} 2022-09-12T19:06:49.847 DEBUG:boto:Token: None 2022-09-12T19:06:49.848 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:49.849 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yvjX/QRIcpCBP5BFCEnqaAGHZvk= 2022-09-12T19:06:49.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yvjX/QRIcpCBP5BFCEnqaAGHZvk=', 'Content-Length': '0'} 2022-09-12T19:06:49.853 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b369ada5706aa78f-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.894 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:49.895 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:49.896 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.897 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.897 DEBUG:boto:Method: GET 2022-09-12T19:06:49.898 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:49.899 DEBUG:boto:Data: 2022-09-12T19:06:49.899 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.900 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.900 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.901 DEBUG:boto:Params: {} 2022-09-12T19:06:49.901 DEBUG:boto:Token: None 2022-09-12T19:06:49.902 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/ 2022-09-12T19:06:49.903 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q= 2022-09-12T19:06:49.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q=', 'Content-Length': '0'} 2022-09-12T19:06:49.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009396eb4c640a8c8b-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.907 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:49.908 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:49.909 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:49.909 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.910 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:49.912 DEBUG:boto:Method: GET 2022-09-12T19:06:49.913 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:49.914 DEBUG:boto:Data: 2022-09-12T19:06:49.915 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.915 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.916 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.917 DEBUG:boto:Params: {} 2022-09-12T19:06:49.918 DEBUG:boto:Token: None 2022-09-12T19:06:49.918 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/ 2022-09-12T19:06:49.919 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q= 2022-09-12T19:06:49.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6v5sQV1gEFzCz/1yE6C/3KVAM6Q=', 'Content-Length': '0'} 2022-09-12T19:06:49.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095dd9c9239309f1d-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.924 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:49.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:49.927 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.928 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.929 DEBUG:boto:path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:49.929 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:49.930 DEBUG:boto:Method: GET 2022-09-12T19:06:49.930 DEBUG:boto:Path: /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:49.931 DEBUG:boto:Data: 2022-09-12T19:06:49.932 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.932 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.933 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.933 DEBUG:boto:Params: {} 2022-09-12T19:06:49.934 DEBUG:boto:Token: None 2022-09-12T19:06:49.935 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:49.935 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0vLJoZwk1S4NbrQho5QF9pNA9gg= 2022-09-12T19:06:49.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0vLJoZwk1S4NbrQho5QF9pNA9gg=', 'Content-Length': '0'} 2022-09-12T19:06:49.938 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'p5OIjHSzwvUDkIBYzRPdKBGezjIkrn-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080339547f44f0c35-00631f8349-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.982 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.983 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.983 DEBUG:boto:path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:49.984 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:49.984 DEBUG:boto:Method: GET 2022-09-12T19:06:49.985 DEBUG:boto:Path: /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:49.985 DEBUG:boto:Data: 2022-09-12T19:06:49.986 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.987 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:49.987 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.988 DEBUG:boto:Params: {} 2022-09-12T19:06:49.988 DEBUG:boto:Token: None 2022-09-12T19:06:49.989 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:49 GMT /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:49.990 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0vLJoZwk1S4NbrQho5QF9pNA9gg= 2022-09-12T19:06:49.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0vLJoZwk1S4NbrQho5QF9pNA9gg=', 'Content-Length': '0'} 2022-09-12T19:06:49.992 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'p5OIjHSzwvUDkIBYzRPdKBGezjIkrn-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a88f44a8098e8aa6-00631f8349-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:49.993 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:49.994 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:49.994 DEBUG:boto:path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:49.995 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:49.995 DEBUG:boto:Method: HEAD 2022-09-12T19:06:49.996 DEBUG:boto:Path: /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:49.996 DEBUG:boto:Data: 2022-09-12T19:06:49.997 DEBUG:boto:Headers: {} 2022-09-12T19:06:49.997 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:49.998 DEBUG:boto:Port: 8000 2022-09-12T19:06:49.998 DEBUG:boto:Params: {} 2022-09-12T19:06:49.999 DEBUG:boto:Token: None 2022-09-12T19:06:50.000 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:50.000 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RnZFMfX3+ITrOZ28bwtNKpDfnHA= 2022-09-12T19:06:50.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RnZFMfX3+ITrOZ28bwtNKpDfnHA=', 'Content-Length': '0'} 2022-09-12T19:06:50.003 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'p5OIjHSzwvUDkIBYzRPdKBGezjIkrn-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e239838f52f5ee84-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.004 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.005 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.005 DEBUG:boto:path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:50.006 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:50.006 DEBUG:boto:Method: HEAD 2022-09-12T19:06:50.007 DEBUG:boto:Path: /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:50.007 DEBUG:boto:Data: 2022-09-12T19:06:50.008 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.008 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.009 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.009 DEBUG:boto:Params: {} 2022-09-12T19:06:50.010 DEBUG:boto:Token: None 2022-09-12T19:06:50.010 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:50.011 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RnZFMfX3+ITrOZ28bwtNKpDfnHA= 2022-09-12T19:06:50.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RnZFMfX3+ITrOZ28bwtNKpDfnHA=', 'Content-Length': '0'} 2022-09-12T19:06:50.014 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'p5OIjHSzwvUDkIBYzRPdKBGezjIkrn-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3536cd126cfc041-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:50.015 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.016 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.016 DEBUG:boto:path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:50.017 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:50.017 DEBUG:boto:Method: GET 2022-09-12T19:06:50.018 DEBUG:boto:Path: /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:50.019 DEBUG:boto:Data: 2022-09-12T19:06:50.019 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.020 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.020 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.021 DEBUG:boto:Params: {} 2022-09-12T19:06:50.021 DEBUG:boto:Token: None 2022-09-12T19:06:50.022 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:50.023 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JmeYsFKCZob1AEz+4UveuCKhI74= 2022-09-12T19:06:50.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JmeYsFKCZob1AEz+4UveuCKhI74=', 'Content-Length': '0'} 2022-09-12T19:06:50.025 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'p5OIjHSzwvUDkIBYzRPdKBGezjIkrn-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007637da98c1033cbb-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.026 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.027 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.027 DEBUG:boto:path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:50.028 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:50.028 DEBUG:boto:Method: GET 2022-09-12T19:06:50.029 DEBUG:boto:Path: /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:50.030 DEBUG:boto:Data: 2022-09-12T19:06:50.030 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.031 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.032 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.032 DEBUG:boto:Params: {} 2022-09-12T19:06:50.033 DEBUG:boto:Token: None 2022-09-12T19:06:50.034 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:50.034 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JmeYsFKCZob1AEz+4UveuCKhI74= 2022-09-12T19:06:50.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JmeYsFKCZob1AEz+4UveuCKhI74=', 'Content-Length': '0'} 2022-09-12T19:06:50.037 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'p5OIjHSzwvUDkIBYzRPdKBGezjIkrn-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8a00f9d4644e698-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.038 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:50.039 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:50.039 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.040 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.041 DEBUG:boto:Method: GET 2022-09-12T19:06:50.041 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:50.042 DEBUG:boto:Data: 2022-09-12T19:06:50.042 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.051 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.051 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.052 DEBUG:boto:Params: {} 2022-09-12T19:06:50.052 DEBUG:boto:Token: None 2022-09-12T19:06:50.053 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/ 2022-09-12T19:06:50.054 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg= 2022-09-12T19:06:50.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg=', 'Content-Length': '0'} 2022-09-12T19:06:50.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ecd9d355ee39c46-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.058 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:50.059 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:50.059 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:50.060 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.060 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.061 DEBUG:boto:Method: GET 2022-09-12T19:06:50.062 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:50.062 DEBUG:boto:Data: 2022-09-12T19:06:50.063 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.064 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.064 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.065 DEBUG:boto:Params: {} 2022-09-12T19:06:50.065 DEBUG:boto:Token: None 2022-09-12T19:06:50.066 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/ 2022-09-12T19:06:50.067 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg= 2022-09-12T19:06:50.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg=', 'Content-Length': '0'} 2022-09-12T19:06:50.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b682563373f2436-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.071 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:50.072 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.073 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.073 DEBUG:boto:Method: HEAD 2022-09-12T19:06:50.074 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:50.074 DEBUG:boto:Data: 2022-09-12T19:06:50.075 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.075 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.076 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.076 DEBUG:boto:Params: {} 2022-09-12T19:06:50.077 DEBUG:boto:Token: None 2022-09-12T19:06:50.077 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj 2022-09-12T19:06:50.078 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XQ6JaNqez0JaMgw6jEyRz1hH/hU= 2022-09-12T19:06:50.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XQ6JaNqez0JaMgw6jEyRz1hH/hU=', 'Content-Length': '0'} 2022-09-12T19:06:50.082 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc50b3c1db539e24-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.083 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.083 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.084 DEBUG:boto:Method: HEAD 2022-09-12T19:06:50.084 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:50.085 DEBUG:boto:Data: 2022-09-12T19:06:50.085 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.086 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.086 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.087 DEBUG:boto:Params: {} 2022-09-12T19:06:50.087 DEBUG:boto:Token: None 2022-09-12T19:06:50.088 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj 2022-09-12T19:06:50.089 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XQ6JaNqez0JaMgw6jEyRz1hH/hU= 2022-09-12T19:06:50.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XQ6JaNqez0JaMgw6jEyRz1hH/hU=', 'Content-Length': '0'} 2022-09-12T19:06:50.093 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003bbde25b6fb90186-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:50.095 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.095 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.096 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.096 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.097 DEBUG:boto:Method: GET 2022-09-12T19:06:50.098 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.099 DEBUG:boto:Data: 2022-09-12T19:06:50.099 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.100 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.100 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.101 DEBUG:boto:Params: {} 2022-09-12T19:06:50.102 DEBUG:boto:Token: None 2022-09-12T19:06:50.102 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.103 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KErGjxGs5OOydwcqyZhnZjQqhKU= 2022-09-12T19:06:50.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KErGjxGs5OOydwcqyZhnZjQqhKU=', 'Content-Length': '0'} 2022-09-12T19:06:50.106 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000021947430ac6262b-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.107 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.108 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.108 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.109 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.109 DEBUG:boto:Method: GET 2022-09-12T19:06:50.110 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.110 DEBUG:boto:Data: 2022-09-12T19:06:50.111 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.111 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.112 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.112 DEBUG:boto:Params: {} 2022-09-12T19:06:50.113 DEBUG:boto:Token: None 2022-09-12T19:06:50.114 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.114 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KErGjxGs5OOydwcqyZhnZjQqhKU= 2022-09-12T19:06:50.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KErGjxGs5OOydwcqyZhnZjQqhKU=', 'Content-Length': '0'} 2022-09-12T19:06:50.117 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6d54173da25be33-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.158 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:50.159 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:50.159 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.160 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.160 DEBUG:boto:Method: GET 2022-09-12T19:06:50.161 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:50.162 DEBUG:boto:Data: 2022-09-12T19:06:50.162 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.163 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.163 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.164 DEBUG:boto:Params: {} 2022-09-12T19:06:50.164 DEBUG:boto:Token: None 2022-09-12T19:06:50.165 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/ 2022-09-12T19:06:50.165 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg= 2022-09-12T19:06:50.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg=', 'Content-Length': '0'} 2022-09-12T19:06:50.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e267fcfea889176f-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.170 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:50.170 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:50.171 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:50.171 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.172 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.172 DEBUG:boto:Method: GET 2022-09-12T19:06:50.173 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:50.174 DEBUG:boto:Data: 2022-09-12T19:06:50.174 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.175 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.175 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.177 DEBUG:boto:Params: {} 2022-09-12T19:06:50.178 DEBUG:boto:Token: None 2022-09-12T19:06:50.179 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/ 2022-09-12T19:06:50.181 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg= 2022-09-12T19:06:50.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg=', 'Content-Length': '0'} 2022-09-12T19:06:50.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083a43e7c9976180f-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.187 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:50.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:50.189 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.190 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.191 DEBUG:boto:path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:50.192 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:50.193 DEBUG:boto:Method: GET 2022-09-12T19:06:50.194 DEBUG:boto:Path: /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:50.195 DEBUG:boto:Data: 2022-09-12T19:06:50.196 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.197 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.198 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.199 DEBUG:boto:Params: {} 2022-09-12T19:06:50.201 DEBUG:boto:Token: None 2022-09-12T19:06:50.202 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:50.203 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x1OFTwlkU9KEatnnY4i7jiImWSM= 2022-09-12T19:06:50.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x1OFTwlkU9KEatnnY4i7jiImWSM=', 'Content-Length': '0'} 2022-09-12T19:06:50.207 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b2e29237a67797a-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.208 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.210 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.211 DEBUG:boto:path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:50.212 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:50.213 DEBUG:boto:Method: GET 2022-09-12T19:06:50.214 DEBUG:boto:Path: /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:50.215 DEBUG:boto:Data: 2022-09-12T19:06:50.216 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.217 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.218 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.219 DEBUG:boto:Params: {} 2022-09-12T19:06:50.221 DEBUG:boto:Token: None 2022-09-12T19:06:50.222 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:50.223 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x1OFTwlkU9KEatnnY4i7jiImWSM= 2022-09-12T19:06:50.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x1OFTwlkU9KEatnnY4i7jiImWSM=', 'Content-Length': '0'} 2022-09-12T19:06:50.227 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7324906afea7a20-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.229 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.230 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.231 DEBUG:boto:path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:50.232 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:50.233 DEBUG:boto:Method: HEAD 2022-09-12T19:06:50.234 DEBUG:boto:Path: /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:50.235 DEBUG:boto:Data: 2022-09-12T19:06:50.236 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.237 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.238 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.239 DEBUG:boto:Params: {} 2022-09-12T19:06:50.240 DEBUG:boto:Token: None 2022-09-12T19:06:50.241 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:50.242 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wdIjCGVjrRb7cqRB4U6c6PMoFS0= 2022-09-12T19:06:50.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wdIjCGVjrRb7cqRB4U6c6PMoFS0=', 'Content-Length': '0'} 2022-09-12T19:06:50.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069d4090a97ecb154-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.247 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.249 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.250 DEBUG:boto:path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:50.251 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:50.252 DEBUG:boto:Method: HEAD 2022-09-12T19:06:50.253 DEBUG:boto:Path: /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:50.254 DEBUG:boto:Data: 2022-09-12T19:06:50.255 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.256 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.257 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.258 DEBUG:boto:Params: {} 2022-09-12T19:06:50.259 DEBUG:boto:Token: None 2022-09-12T19:06:50.260 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:50.261 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wdIjCGVjrRb7cqRB4U6c6PMoFS0= 2022-09-12T19:06:50.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wdIjCGVjrRb7cqRB4U6c6PMoFS0=', 'Content-Length': '0'} 2022-09-12T19:06:50.265 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6c37f46dea6e82e-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:50.268 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.270 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.271 DEBUG:boto:path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:50.272 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:50.273 DEBUG:boto:Method: GET 2022-09-12T19:06:50.274 DEBUG:boto:Path: /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:50.275 DEBUG:boto:Data: 2022-09-12T19:06:50.275 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.276 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.276 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.277 DEBUG:boto:Params: {} 2022-09-12T19:06:50.278 DEBUG:boto:Token: None 2022-09-12T19:06:50.278 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:50.279 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x1OFTwlkU9KEatnnY4i7jiImWSM= 2022-09-12T19:06:50.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x1OFTwlkU9KEatnnY4i7jiImWSM=', 'Content-Length': '0'} 2022-09-12T19:06:50.282 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001593024876adb15-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.283 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.283 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.284 DEBUG:boto:path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:50.284 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:50.285 DEBUG:boto:Method: GET 2022-09-12T19:06:50.286 DEBUG:boto:Path: /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:50.286 DEBUG:boto:Data: 2022-09-12T19:06:50.287 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.287 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.288 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.288 DEBUG:boto:Params: {} 2022-09-12T19:06:50.289 DEBUG:boto:Token: None 2022-09-12T19:06:50.290 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:50.290 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x1OFTwlkU9KEatnnY4i7jiImWSM= 2022-09-12T19:06:50.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x1OFTwlkU9KEatnnY4i7jiImWSM=', 'Content-Length': '0'} 2022-09-12T19:06:50.293 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0e3965797bafe6c-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.294 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:50.294 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:50.295 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.295 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.296 DEBUG:boto:Method: GET 2022-09-12T19:06:50.297 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:50.297 DEBUG:boto:Data: 2022-09-12T19:06:50.298 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.298 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.299 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.299 DEBUG:boto:Params: {} 2022-09-12T19:06:50.300 DEBUG:boto:Token: None 2022-09-12T19:06:50.300 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/ 2022-09-12T19:06:50.301 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg= 2022-09-12T19:06:50.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg=', 'Content-Length': '0'} 2022-09-12T19:06:50.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abff6aaa6b6cd266-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.346 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:50.347 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:50.348 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:50.348 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.349 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.349 DEBUG:boto:Method: GET 2022-09-12T19:06:50.350 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:50.350 DEBUG:boto:Data: 2022-09-12T19:06:50.351 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.351 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.352 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.352 DEBUG:boto:Params: {} 2022-09-12T19:06:50.353 DEBUG:boto:Token: None 2022-09-12T19:06:50.354 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/ 2022-09-12T19:06:50.355 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg= 2022-09-12T19:06:50.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg=', 'Content-Length': '0'} 2022-09-12T19:06:50.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087d237cd55545f1d-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.359 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:50.360 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.361 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.361 DEBUG:boto:Method: HEAD 2022-09-12T19:06:50.362 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:50.362 DEBUG:boto:Data: 2022-09-12T19:06:50.363 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.363 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.364 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.364 DEBUG:boto:Params: {} 2022-09-12T19:06:50.365 DEBUG:boto:Token: None 2022-09-12T19:06:50.365 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj 2022-09-12T19:06:50.366 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XQ6JaNqez0JaMgw6jEyRz1hH/hU= 2022-09-12T19:06:50.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XQ6JaNqez0JaMgw6jEyRz1hH/hU=', 'Content-Length': '0'} 2022-09-12T19:06:50.370 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd33fc5fcb2fe99b-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.371 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.372 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.372 DEBUG:boto:Method: HEAD 2022-09-12T19:06:50.373 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:50.374 DEBUG:boto:Data: 2022-09-12T19:06:50.374 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.375 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.375 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.376 DEBUG:boto:Params: {} 2022-09-12T19:06:50.376 DEBUG:boto:Token: None 2022-09-12T19:06:50.377 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj 2022-09-12T19:06:50.377 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XQ6JaNqez0JaMgw6jEyRz1hH/hU= 2022-09-12T19:06:50.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XQ6JaNqez0JaMgw6jEyRz1hH/hU=', 'Content-Length': '0'} 2022-09-12T19:06:50.381 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057a26d6672f0739d-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:50.383 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.383 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.384 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.385 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.385 DEBUG:boto:Method: GET 2022-09-12T19:06:50.386 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.386 DEBUG:boto:Data: 2022-09-12T19:06:50.387 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.388 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.388 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.389 DEBUG:boto:Params: {} 2022-09-12T19:06:50.389 DEBUG:boto:Token: None 2022-09-12T19:06:50.390 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.390 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KErGjxGs5OOydwcqyZhnZjQqhKU= 2022-09-12T19:06:50.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KErGjxGs5OOydwcqyZhnZjQqhKU=', 'Content-Length': '0'} 2022-09-12T19:06:50.394 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007cc7c2834a494cbf-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.394 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.395 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.396 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.396 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.397 DEBUG:boto:Method: GET 2022-09-12T19:06:50.397 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.398 DEBUG:boto:Data: 2022-09-12T19:06:50.398 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.399 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.399 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.400 DEBUG:boto:Params: {} 2022-09-12T19:06:50.401 DEBUG:boto:Token: None 2022-09-12T19:06:50.401 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.402 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KErGjxGs5OOydwcqyZhnZjQqhKU= 2022-09-12T19:06:50.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KErGjxGs5OOydwcqyZhnZjQqhKU=', 'Content-Length': '0'} 2022-09-12T19:06:50.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000626d16c64631946f-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.450 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:50.452 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:50.452 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.453 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.454 DEBUG:boto:Method: GET 2022-09-12T19:06:50.455 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:50.456 DEBUG:boto:Data: 2022-09-12T19:06:50.457 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.458 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.459 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.460 DEBUG:boto:Params: {} 2022-09-12T19:06:50.461 DEBUG:boto:Token: None 2022-09-12T19:06:50.462 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/ 2022-09-12T19:06:50.463 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg= 2022-09-12T19:06:50.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg=', 'Content-Length': '0'} 2022-09-12T19:06:50.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2be0cb04b706efe-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.468 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:50.468 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:50.469 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:50.470 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.470 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.471 DEBUG:boto:Method: GET 2022-09-12T19:06:50.471 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:50.472 DEBUG:boto:Data: 2022-09-12T19:06:50.473 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.473 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.474 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.474 DEBUG:boto:Params: {} 2022-09-12T19:06:50.475 DEBUG:boto:Token: None 2022-09-12T19:06:50.475 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/ 2022-09-12T19:06:50.476 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg= 2022-09-12T19:06:50.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg=', 'Content-Length': '0'} 2022-09-12T19:06:50.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087da49da78e67f8d-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.480 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:50.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:50.482 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.483 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.484 DEBUG:boto:path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:50.485 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:50.486 DEBUG:boto:Method: GET 2022-09-12T19:06:50.486 DEBUG:boto:Path: /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:50.487 DEBUG:boto:Data: 2022-09-12T19:06:50.488 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.489 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.490 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.491 DEBUG:boto:Params: {} 2022-09-12T19:06:50.492 DEBUG:boto:Token: None 2022-09-12T19:06:50.493 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:50.494 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J/G2pydZWLS4mNqqSdci6eQ0kWE= 2022-09-12T19:06:50.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J/G2pydZWLS4mNqqSdci6eQ0kWE=', 'Content-Length': '0'} 2022-09-12T19:06:50.497 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000157ea65805bb7522-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.538 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.539 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.540 DEBUG:boto:path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:50.540 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:50.541 DEBUG:boto:Method: GET 2022-09-12T19:06:50.541 DEBUG:boto:Path: /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:50.542 DEBUG:boto:Data: 2022-09-12T19:06:50.543 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.543 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.544 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.544 DEBUG:boto:Params: {} 2022-09-12T19:06:50.545 DEBUG:boto:Token: None 2022-09-12T19:06:50.545 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:50.546 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J/G2pydZWLS4mNqqSdci6eQ0kWE= 2022-09-12T19:06:50.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J/G2pydZWLS4mNqqSdci6eQ0kWE=', 'Content-Length': '0'} 2022-09-12T19:06:50.549 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000219eecce8b1ec4f9-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.551 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.552 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.552 DEBUG:boto:path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:50.553 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:50.554 DEBUG:boto:Method: HEAD 2022-09-12T19:06:50.554 DEBUG:boto:Path: /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:50.555 DEBUG:boto:Data: 2022-09-12T19:06:50.555 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.556 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.556 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.557 DEBUG:boto:Params: {} 2022-09-12T19:06:50.557 DEBUG:boto:Token: None 2022-09-12T19:06:50.558 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:50.559 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wnEWOijuLOD750ELwkNdSLfllUw= 2022-09-12T19:06:50.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wnEWOijuLOD750ELwkNdSLfllUw=', 'Content-Length': '0'} 2022-09-12T19:06:50.561 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff4dd9cb22f7ef56-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.562 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.563 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.563 DEBUG:boto:path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:50.564 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:50.564 DEBUG:boto:Method: HEAD 2022-09-12T19:06:50.565 DEBUG:boto:Path: /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:50.565 DEBUG:boto:Data: 2022-09-12T19:06:50.566 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.567 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.567 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.568 DEBUG:boto:Params: {} 2022-09-12T19:06:50.568 DEBUG:boto:Token: None 2022-09-12T19:06:50.569 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:50.569 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wnEWOijuLOD750ELwkNdSLfllUw= 2022-09-12T19:06:50.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wnEWOijuLOD750ELwkNdSLfllUw=', 'Content-Length': '0'} 2022-09-12T19:06:50.572 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e85cdd223004124e-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:50.573 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.574 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.574 DEBUG:boto:path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:50.575 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:50.576 DEBUG:boto:Method: GET 2022-09-12T19:06:50.576 DEBUG:boto:Path: /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:50.577 DEBUG:boto:Data: 2022-09-12T19:06:50.577 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.578 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.578 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.579 DEBUG:boto:Params: {} 2022-09-12T19:06:50.579 DEBUG:boto:Token: None 2022-09-12T19:06:50.580 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:50.581 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J/G2pydZWLS4mNqqSdci6eQ0kWE= 2022-09-12T19:06:50.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J/G2pydZWLS4mNqqSdci6eQ0kWE=', 'Content-Length': '0'} 2022-09-12T19:06:50.584 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e53fcce1ea4cde4-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.584 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.585 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.585 DEBUG:boto:path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:50.586 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:50.587 DEBUG:boto:Method: GET 2022-09-12T19:06:50.587 DEBUG:boto:Path: /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:50.588 DEBUG:boto:Data: 2022-09-12T19:06:50.588 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.589 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.589 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.590 DEBUG:boto:Params: {} 2022-09-12T19:06:50.591 DEBUG:boto:Token: None 2022-09-12T19:06:50.591 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:50.592 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J/G2pydZWLS4mNqqSdci6eQ0kWE= 2022-09-12T19:06:50.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J/G2pydZWLS4mNqqSdci6eQ0kWE=', 'Content-Length': '0'} 2022-09-12T19:06:50.595 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba0760738c7fc7e4-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.596 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:50.596 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:50.597 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.597 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.598 DEBUG:boto:Method: GET 2022-09-12T19:06:50.599 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:50.599 DEBUG:boto:Data: 2022-09-12T19:06:50.600 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.600 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.601 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.601 DEBUG:boto:Params: {} 2022-09-12T19:06:50.602 DEBUG:boto:Token: None 2022-09-12T19:06:50.603 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/ 2022-09-12T19:06:50.603 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg= 2022-09-12T19:06:50.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg=', 'Content-Length': '0'} 2022-09-12T19:06:50.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b609fb79ba94e72e-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.607 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:50.608 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:50.609 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:50.609 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.610 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.610 DEBUG:boto:Method: GET 2022-09-12T19:06:50.611 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:50.611 DEBUG:boto:Data: 2022-09-12T19:06:50.612 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.612 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.613 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.613 DEBUG:boto:Params: {} 2022-09-12T19:06:50.614 DEBUG:boto:Token: None 2022-09-12T19:06:50.615 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/ 2022-09-12T19:06:50.615 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg= 2022-09-12T19:06:50.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg=', 'Content-Length': '0'} 2022-09-12T19:06:50.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000827c05232e7cb680-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.619 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:50.620 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.621 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.621 DEBUG:boto:Method: HEAD 2022-09-12T19:06:50.622 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:50.622 DEBUG:boto:Data: 2022-09-12T19:06:50.623 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.623 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.624 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.624 DEBUG:boto:Params: {} 2022-09-12T19:06:50.625 DEBUG:boto:Token: None 2022-09-12T19:06:50.625 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj 2022-09-12T19:06:50.626 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XQ6JaNqez0JaMgw6jEyRz1hH/hU= 2022-09-12T19:06:50.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XQ6JaNqez0JaMgw6jEyRz1hH/hU=', 'Content-Length': '0'} 2022-09-12T19:06:50.630 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000219b66e1bc91c773-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.631 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.631 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.632 DEBUG:boto:Method: HEAD 2022-09-12T19:06:50.632 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:50.633 DEBUG:boto:Data: 2022-09-12T19:06:50.633 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.634 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.634 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.635 DEBUG:boto:Params: {} 2022-09-12T19:06:50.635 DEBUG:boto:Token: None 2022-09-12T19:06:50.636 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj 2022-09-12T19:06:50.637 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XQ6JaNqez0JaMgw6jEyRz1hH/hU= 2022-09-12T19:06:50.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XQ6JaNqez0JaMgw6jEyRz1hH/hU=', 'Content-Length': '0'} 2022-09-12T19:06:50.641 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000e1288478d184b4-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:50.642 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.643 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.643 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.644 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.644 DEBUG:boto:Method: GET 2022-09-12T19:06:50.645 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.645 DEBUG:boto:Data: 2022-09-12T19:06:50.646 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.646 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.647 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.647 DEBUG:boto:Params: {} 2022-09-12T19:06:50.648 DEBUG:boto:Token: None 2022-09-12T19:06:50.649 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.649 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KErGjxGs5OOydwcqyZhnZjQqhKU= 2022-09-12T19:06:50.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KErGjxGs5OOydwcqyZhnZjQqhKU=', 'Content-Length': '0'} 2022-09-12T19:06:50.652 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000577c9c568e06d4d6-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.694 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.695 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.695 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.696 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.697 DEBUG:boto:Method: GET 2022-09-12T19:06:50.697 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.698 DEBUG:boto:Data: 2022-09-12T19:06:50.698 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.699 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.700 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.700 DEBUG:boto:Params: {} 2022-09-12T19:06:50.701 DEBUG:boto:Token: None 2022-09-12T19:06:50.702 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.703 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KErGjxGs5OOydwcqyZhnZjQqhKU= 2022-09-12T19:06:50.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KErGjxGs5OOydwcqyZhnZjQqhKU=', 'Content-Length': '0'} 2022-09-12T19:06:50.705 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb4b496eca5a2674-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.706 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:50.707 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:50.707 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.708 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.709 DEBUG:boto:Method: GET 2022-09-12T19:06:50.709 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:50.710 DEBUG:boto:Data: 2022-09-12T19:06:50.710 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.711 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.711 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.712 DEBUG:boto:Params: {} 2022-09-12T19:06:50.712 DEBUG:boto:Token: None 2022-09-12T19:06:50.713 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/ 2022-09-12T19:06:50.713 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg= 2022-09-12T19:06:50.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg=', 'Content-Length': '0'} 2022-09-12T19:06:50.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000887860cb0c563334-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.718 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:50.719 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:50.719 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:50.720 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.721 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.721 DEBUG:boto:Method: GET 2022-09-12T19:06:50.722 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:50.722 DEBUG:boto:Data: 2022-09-12T19:06:50.723 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.723 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.724 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.724 DEBUG:boto:Params: {} 2022-09-12T19:06:50.725 DEBUG:boto:Token: None 2022-09-12T19:06:50.726 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/ 2022-09-12T19:06:50.726 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg= 2022-09-12T19:06:50.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg=', 'Content-Length': '0'} 2022-09-12T19:06:50.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f21667a5718770cb-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.731 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:50.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:50.732 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.733 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.733 DEBUG:boto:path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:50.734 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:50.734 DEBUG:boto:Method: GET 2022-09-12T19:06:50.735 DEBUG:boto:Path: /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:50.735 DEBUG:boto:Data: 2022-09-12T19:06:50.736 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.737 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.737 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.738 DEBUG:boto:Params: {} 2022-09-12T19:06:50.739 DEBUG:boto:Token: None 2022-09-12T19:06:50.739 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:50.740 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8NssmMoLITTwYwRp1GOIWF0VpPQ= 2022-09-12T19:06:50.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8NssmMoLITTwYwRp1GOIWF0VpPQ=', 'Content-Length': '0'} 2022-09-12T19:06:50.743 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'j0eE53eOEuIue0LhYTVuZStUTmcOj7o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c2b84c809887102-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.744 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.744 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.745 DEBUG:boto:path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:50.745 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:50.746 DEBUG:boto:Method: GET 2022-09-12T19:06:50.747 DEBUG:boto:Path: /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:50.747 DEBUG:boto:Data: 2022-09-12T19:06:50.748 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.748 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.749 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.749 DEBUG:boto:Params: {} 2022-09-12T19:06:50.750 DEBUG:boto:Token: None 2022-09-12T19:06:50.751 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:50.752 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8NssmMoLITTwYwRp1GOIWF0VpPQ= 2022-09-12T19:06:50.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8NssmMoLITTwYwRp1GOIWF0VpPQ=', 'Content-Length': '0'} 2022-09-12T19:06:50.755 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'j0eE53eOEuIue0LhYTVuZStUTmcOj7o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad59fac727591186-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.757 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.758 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.759 DEBUG:boto:path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:50.760 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:50.761 DEBUG:boto:Method: HEAD 2022-09-12T19:06:50.762 DEBUG:boto:Path: /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:50.763 DEBUG:boto:Data: 2022-09-12T19:06:50.764 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.764 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.765 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.765 DEBUG:boto:Params: {} 2022-09-12T19:06:50.766 DEBUG:boto:Token: None 2022-09-12T19:06:50.767 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:50.767 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5M/NYsQ2emoWh22249FcfYWmG5U= 2022-09-12T19:06:50.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5M/NYsQ2emoWh22249FcfYWmG5U=', 'Content-Length': '0'} 2022-09-12T19:06:50.770 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'j0eE53eOEuIue0LhYTVuZStUTmcOj7o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000196a3121afd6f58c-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.771 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.771 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.772 DEBUG:boto:path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:50.772 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:50.773 DEBUG:boto:Method: HEAD 2022-09-12T19:06:50.773 DEBUG:boto:Path: /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:50.774 DEBUG:boto:Data: 2022-09-12T19:06:50.774 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.775 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.776 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.776 DEBUG:boto:Params: {} 2022-09-12T19:06:50.777 DEBUG:boto:Token: None 2022-09-12T19:06:50.777 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:50.778 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5M/NYsQ2emoWh22249FcfYWmG5U= 2022-09-12T19:06:50.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5M/NYsQ2emoWh22249FcfYWmG5U=', 'Content-Length': '0'} 2022-09-12T19:06:50.781 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'j0eE53eOEuIue0LhYTVuZStUTmcOj7o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6d46566494c2ffa-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:50.782 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.783 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.784 DEBUG:boto:path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:50.784 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:50.785 DEBUG:boto:Method: GET 2022-09-12T19:06:50.786 DEBUG:boto:Path: /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:50.787 DEBUG:boto:Data: 2022-09-12T19:06:50.788 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.789 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.790 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.791 DEBUG:boto:Params: {} 2022-09-12T19:06:50.792 DEBUG:boto:Token: None 2022-09-12T19:06:50.794 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:50.795 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8NssmMoLITTwYwRp1GOIWF0VpPQ= 2022-09-12T19:06:50.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8NssmMoLITTwYwRp1GOIWF0VpPQ=', 'Content-Length': '0'} 2022-09-12T19:06:50.799 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'j0eE53eOEuIue0LhYTVuZStUTmcOj7o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d643eaa6b0d153c-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.842 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.843 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.844 DEBUG:boto:path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:50.844 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:50.845 DEBUG:boto:Method: GET 2022-09-12T19:06:50.845 DEBUG:boto:Path: /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:50.846 DEBUG:boto:Data: 2022-09-12T19:06:50.847 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.847 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.848 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.848 DEBUG:boto:Params: {} 2022-09-12T19:06:50.849 DEBUG:boto:Token: None 2022-09-12T19:06:50.849 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:50.850 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8NssmMoLITTwYwRp1GOIWF0VpPQ= 2022-09-12T19:06:50.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8NssmMoLITTwYwRp1GOIWF0VpPQ=', 'Content-Length': '0'} 2022-09-12T19:06:50.853 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'j0eE53eOEuIue0LhYTVuZStUTmcOj7o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e65c55ced69d379d-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.854 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:50.854 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:50.855 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.857 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.857 DEBUG:boto:Method: GET 2022-09-12T19:06:50.858 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:50.858 DEBUG:boto:Data: 2022-09-12T19:06:50.859 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.860 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.860 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.861 DEBUG:boto:Params: {} 2022-09-12T19:06:50.862 DEBUG:boto:Token: None 2022-09-12T19:06:50.863 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/ 2022-09-12T19:06:50.863 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg= 2022-09-12T19:06:50.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg=', 'Content-Length': '0'} 2022-09-12T19:06:50.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000749d9607d955362c-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.868 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:50.868 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:50.869 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:50.870 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.870 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.871 DEBUG:boto:Method: GET 2022-09-12T19:06:50.871 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:50.872 DEBUG:boto:Data: 2022-09-12T19:06:50.872 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.873 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.873 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.874 DEBUG:boto:Params: {} 2022-09-12T19:06:50.874 DEBUG:boto:Token: None 2022-09-12T19:06:50.875 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/ 2022-09-12T19:06:50.876 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg= 2022-09-12T19:06:50.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg=', 'Content-Length': '0'} 2022-09-12T19:06:50.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee64af5feedc986a-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.882 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:50.884 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.885 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.886 DEBUG:boto:Method: HEAD 2022-09-12T19:06:50.887 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:50.887 DEBUG:boto:Data: 2022-09-12T19:06:50.888 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.889 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.890 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.890 DEBUG:boto:Params: {} 2022-09-12T19:06:50.891 DEBUG:boto:Token: None 2022-09-12T19:06:50.892 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj 2022-09-12T19:06:50.893 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XQ6JaNqez0JaMgw6jEyRz1hH/hU= 2022-09-12T19:06:50.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XQ6JaNqez0JaMgw6jEyRz1hH/hU=', 'Content-Length': '0'} 2022-09-12T19:06:50.897 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e353ad0587e0f2e-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.897 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.898 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.899 DEBUG:boto:Method: HEAD 2022-09-12T19:06:50.899 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:50.900 DEBUG:boto:Data: 2022-09-12T19:06:50.900 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.901 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.901 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.902 DEBUG:boto:Params: {} 2022-09-12T19:06:50.902 DEBUG:boto:Token: None 2022-09-12T19:06:50.903 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj 2022-09-12T19:06:50.903 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XQ6JaNqez0JaMgw6jEyRz1hH/hU= 2022-09-12T19:06:50.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XQ6JaNqez0JaMgw6jEyRz1hH/hU=', 'Content-Length': '0'} 2022-09-12T19:06:50.907 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fdc677466a6b6452-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:50.909 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.910 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.911 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.911 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.912 DEBUG:boto:Method: GET 2022-09-12T19:06:50.913 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.914 DEBUG:boto:Data: 2022-09-12T19:06:50.914 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.915 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.915 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.916 DEBUG:boto:Params: {} 2022-09-12T19:06:50.917 DEBUG:boto:Token: None 2022-09-12T19:06:50.917 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.918 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KErGjxGs5OOydwcqyZhnZjQqhKU= 2022-09-12T19:06:50.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KErGjxGs5OOydwcqyZhnZjQqhKU=', 'Content-Length': '0'} 2022-09-12T19:06:50.921 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007919cba9b4aea19a-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.922 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:50.922 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:50.923 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.923 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.924 DEBUG:boto:Method: GET 2022-09-12T19:06:50.925 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.925 DEBUG:boto:Data: 2022-09-12T19:06:50.926 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.926 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.927 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.927 DEBUG:boto:Params: {} 2022-09-12T19:06:50.928 DEBUG:boto:Token: None 2022-09-12T19:06:50.928 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:50.929 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KErGjxGs5OOydwcqyZhnZjQqhKU= 2022-09-12T19:06:50.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KErGjxGs5OOydwcqyZhnZjQqhKU=', 'Content-Length': '0'} 2022-09-12T19:06:50.932 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c7086284d2de007-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.974 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:50.975 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:50.975 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.976 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.976 DEBUG:boto:Method: GET 2022-09-12T19:06:50.977 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:50.978 DEBUG:boto:Data: 2022-09-12T19:06:50.978 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.979 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:50.979 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.980 DEBUG:boto:Params: {} 2022-09-12T19:06:50.981 DEBUG:boto:Token: None 2022-09-12T19:06:50.981 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/ 2022-09-12T19:06:50.982 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg= 2022-09-12T19:06:50.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg=', 'Content-Length': '0'} 2022-09-12T19:06:50.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c151a55ea9503e8-00631f834a-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:50.986 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:50.988 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:50.988 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:50.989 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.989 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:50.990 DEBUG:boto:Method: GET 2022-09-12T19:06:50.991 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:50.992 DEBUG:boto:Data: 2022-09-12T19:06:50.992 DEBUG:boto:Headers: {} 2022-09-12T19:06:50.993 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:50.993 DEBUG:boto:Port: 8000 2022-09-12T19:06:50.994 DEBUG:boto:Params: {} 2022-09-12T19:06:50.994 DEBUG:boto:Token: None 2022-09-12T19:06:50.995 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:50 GMT /jdpiak-19/ 2022-09-12T19:06:50.996 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg= 2022-09-12T19:06:50.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:50 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rcVw5dJmJLvbU6Vmy5lJt0AaCKg=', 'Content-Length': '0'} 2022-09-12T19:06:51.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eafdabeebc52819d-00631f834a-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.001 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:51.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:51.002 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.003 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.004 DEBUG:boto:path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:51.004 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:51.005 DEBUG:boto:Method: GET 2022-09-12T19:06:51.005 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:51.006 DEBUG:boto:Data: 2022-09-12T19:06:51.007 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.008 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:51.009 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.010 DEBUG:boto:Params: {} 2022-09-12T19:06:51.010 DEBUG:boto:Token: None 2022-09-12T19:06:51.011 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:51.012 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PabJthTxTdmLmtZ9K77VEBSz0jw= 2022-09-12T19:06:51.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PabJthTxTdmLmtZ9K77VEBSz0jw=', 'Content-Length': '0'} 2022-09-12T19:06:51.015 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000738d74d7c9b00d35-00631f834b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.016 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.016 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.017 DEBUG:boto:path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:51.017 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:51.018 DEBUG:boto:Method: GET 2022-09-12T19:06:51.019 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:51.019 DEBUG:boto:Data: 2022-09-12T19:06:51.020 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.020 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.021 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.022 DEBUG:boto:Params: {} 2022-09-12T19:06:51.022 DEBUG:boto:Token: None 2022-09-12T19:06:51.023 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:51.024 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PabJthTxTdmLmtZ9K77VEBSz0jw= 2022-09-12T19:06:51.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PabJthTxTdmLmtZ9K77VEBSz0jw=', 'Content-Length': '0'} 2022-09-12T19:06:51.027 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7d2d9cda86a7b5b-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.028 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.029 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.029 DEBUG:boto:path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:51.030 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:51.030 DEBUG:boto:Method: HEAD 2022-09-12T19:06:51.031 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:51.031 DEBUG:boto:Data: 2022-09-12T19:06:51.032 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.032 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:51.033 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.033 DEBUG:boto:Params: {} 2022-09-12T19:06:51.034 DEBUG:boto:Token: None 2022-09-12T19:06:51.035 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:51.035 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uFRKpVHAg3Q3BmG6yi196N4cxsQ= 2022-09-12T19:06:51.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uFRKpVHAg3Q3BmG6yi196N4cxsQ=', 'Content-Length': '0'} 2022-09-12T19:06:51.038 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af22dd78f76bb0cf-00631f834b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.039 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.039 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.040 DEBUG:boto:path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:51.040 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:51.041 DEBUG:boto:Method: HEAD 2022-09-12T19:06:51.042 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:51.042 DEBUG:boto:Data: 2022-09-12T19:06:51.043 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.044 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.044 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.045 DEBUG:boto:Params: {} 2022-09-12T19:06:51.045 DEBUG:boto:Token: None 2022-09-12T19:06:51.046 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:51.047 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uFRKpVHAg3Q3BmG6yi196N4cxsQ= 2022-09-12T19:06:51.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uFRKpVHAg3Q3BmG6yi196N4cxsQ=', 'Content-Length': '0'} 2022-09-12T19:06:51.049 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042284bee3e820740-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:51.051 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.051 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.052 DEBUG:boto:path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:51.052 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:51.053 DEBUG:boto:Method: GET 2022-09-12T19:06:51.053 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:51.054 DEBUG:boto:Data: 2022-09-12T19:06:51.055 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.055 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:51.056 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.056 DEBUG:boto:Params: {} 2022-09-12T19:06:51.057 DEBUG:boto:Token: None 2022-09-12T19:06:51.057 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:51.058 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PabJthTxTdmLmtZ9K77VEBSz0jw= 2022-09-12T19:06:51.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PabJthTxTdmLmtZ9K77VEBSz0jw=', 'Content-Length': '0'} 2022-09-12T19:06:51.061 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000675cc22fd830227e-00631f834b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.102 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.103 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.104 DEBUG:boto:path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:51.105 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:51.105 DEBUG:boto:Method: GET 2022-09-12T19:06:51.106 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:51.106 DEBUG:boto:Data: 2022-09-12T19:06:51.107 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.108 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.108 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.109 DEBUG:boto:Params: {} 2022-09-12T19:06:51.109 DEBUG:boto:Token: None 2022-09-12T19:06:51.110 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:51.111 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PabJthTxTdmLmtZ9K77VEBSz0jw= 2022-09-12T19:06:51.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PabJthTxTdmLmtZ9K77VEBSz0jw=', 'Content-Length': '0'} 2022-09-12T19:06:51.115 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000093c5bc28e6758d8-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.117 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:51.117 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:51.118 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:51.119 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:51.119 DEBUG:boto:Method: GET 2022-09-12T19:06:51.120 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:51.121 DEBUG:boto:Data: 2022-09-12T19:06:51.121 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.122 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:51.122 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.123 DEBUG:boto:Params: {} 2022-09-12T19:06:51.124 DEBUG:boto:Token: None 2022-09-12T19:06:51.124 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/ 2022-09-12T19:06:51.125 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0= 2022-09-12T19:06:51.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0=', 'Content-Length': '0'} 2022-09-12T19:06:51.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e5169058c46dee5-00631f834b-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.129 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:51.130 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:51.131 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:51.131 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:51.132 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:51.132 DEBUG:boto:Method: GET 2022-09-12T19:06:51.133 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:51.133 DEBUG:boto:Data: 2022-09-12T19:06:51.134 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.135 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.135 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.136 DEBUG:boto:Params: {} 2022-09-12T19:06:51.137 DEBUG:boto:Token: None 2022-09-12T19:06:51.138 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/ 2022-09-12T19:06:51.138 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0= 2022-09-12T19:06:51.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0=', 'Content-Length': '0'} 2022-09-12T19:06:51.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b89f8d470390599-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.144 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:51.146 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.147 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.148 DEBUG:boto:Method: HEAD 2022-09-12T19:06:51.149 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:51.150 DEBUG:boto:Data: 2022-09-12T19:06:51.150 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.152 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:51.153 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.154 DEBUG:boto:Params: {} 2022-09-12T19:06:51.155 DEBUG:boto:Token: None 2022-09-12T19:06:51.156 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj 2022-09-12T19:06:51.157 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FgALKL3prsLBGofnjrT6ORBquz0= 2022-09-12T19:06:51.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FgALKL3prsLBGofnjrT6ORBquz0=', 'Content-Length': '0'} 2022-09-12T19:06:51.162 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052ba9106e225dc7a-00631f834b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.164 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.165 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.166 DEBUG:boto:Method: HEAD 2022-09-12T19:06:51.167 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:51.168 DEBUG:boto:Data: 2022-09-12T19:06:51.169 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.170 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.171 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.172 DEBUG:boto:Params: {} 2022-09-12T19:06:51.173 DEBUG:boto:Token: None 2022-09-12T19:06:51.174 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj 2022-09-12T19:06:51.175 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FgALKL3prsLBGofnjrT6ORBquz0= 2022-09-12T19:06:51.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FgALKL3prsLBGofnjrT6ORBquz0=', 'Content-Length': '0'} 2022-09-12T19:06:51.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000724d48bd16917b1e-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:51.183 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.183 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.184 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:51.184 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:51.185 DEBUG:boto:Method: GET 2022-09-12T19:06:51.186 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:51.186 DEBUG:boto:Data: 2022-09-12T19:06:51.187 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.187 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:51.188 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.188 DEBUG:boto:Params: {} 2022-09-12T19:06:51.189 DEBUG:boto:Token: None 2022-09-12T19:06:51.189 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:51.190 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A2rzOZiGTdArdCLG+ahlkht4jKU= 2022-09-12T19:06:51.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A2rzOZiGTdArdCLG+ahlkht4jKU=', 'Content-Length': '0'} 2022-09-12T19:06:51.193 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a788d5d772fdcfd4-00631f834b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.194 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.194 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.195 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:51.195 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:51.196 DEBUG:boto:Method: GET 2022-09-12T19:06:51.197 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:51.197 DEBUG:boto:Data: 2022-09-12T19:06:51.198 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.198 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.199 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.199 DEBUG:boto:Params: {} 2022-09-12T19:06:51.200 DEBUG:boto:Token: None 2022-09-12T19:06:51.201 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:51.201 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A2rzOZiGTdArdCLG+ahlkht4jKU= 2022-09-12T19:06:51.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A2rzOZiGTdArdCLG+ahlkht4jKU=', 'Content-Length': '0'} 2022-09-12T19:06:51.204 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058b662c37943f89a-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.205 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:51.206 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:51.206 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:51.207 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:51.207 DEBUG:boto:Method: GET 2022-09-12T19:06:51.208 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:51.208 DEBUG:boto:Data: 2022-09-12T19:06:51.209 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.209 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:51.210 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.211 DEBUG:boto:Params: {} 2022-09-12T19:06:51.211 DEBUG:boto:Token: None 2022-09-12T19:06:51.212 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/ 2022-09-12T19:06:51.212 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0= 2022-09-12T19:06:51.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0=', 'Content-Length': '0'} 2022-09-12T19:06:51.216 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f39f10d0d19a606-00631f834b-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.217 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:51.218 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:51.219 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:51.219 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:51.220 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:51.220 DEBUG:boto:Method: GET 2022-09-12T19:06:51.221 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:51.221 DEBUG:boto:Data: 2022-09-12T19:06:51.222 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.222 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.223 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.223 DEBUG:boto:Params: {} 2022-09-12T19:06:51.224 DEBUG:boto:Token: None 2022-09-12T19:06:51.225 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/ 2022-09-12T19:06:51.225 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0= 2022-09-12T19:06:51.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0=', 'Content-Length': '0'} 2022-09-12T19:06:51.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042c1480c33be4289-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.230 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:51.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:51.231 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.232 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.232 DEBUG:boto:path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:51.233 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:51.234 DEBUG:boto:Method: GET 2022-09-12T19:06:51.234 DEBUG:boto:Path: /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:51.235 DEBUG:boto:Data: 2022-09-12T19:06:51.235 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.236 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:51.237 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.237 DEBUG:boto:Params: {} 2022-09-12T19:06:51.238 DEBUG:boto:Token: None 2022-09-12T19:06:51.238 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:51.239 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MBz8wCng1+PNOyGLEZ/TxIgWopE= 2022-09-12T19:06:51.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MBz8wCng1+PNOyGLEZ/TxIgWopE=', 'Content-Length': '0'} 2022-09-12T19:06:51.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'JcC9N13lklBVOx234Q8MXQ8kAM8ByWg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000465e87d9b6d8e025-00631f834b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.243 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.244 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.244 DEBUG:boto:path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:51.245 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:51.245 DEBUG:boto:Method: GET 2022-09-12T19:06:51.246 DEBUG:boto:Path: /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:51.246 DEBUG:boto:Data: 2022-09-12T19:06:51.247 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.247 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.248 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.248 DEBUG:boto:Params: {} 2022-09-12T19:06:51.249 DEBUG:boto:Token: None 2022-09-12T19:06:51.249 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:51.250 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MBz8wCng1+PNOyGLEZ/TxIgWopE= 2022-09-12T19:06:51.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MBz8wCng1+PNOyGLEZ/TxIgWopE=', 'Content-Length': '0'} 2022-09-12T19:06:51.254 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'JcC9N13lklBVOx234Q8MXQ8kAM8ByWg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000857593df3eaa2fe5-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.254 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.255 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.255 DEBUG:boto:path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:51.256 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:51.257 DEBUG:boto:Method: HEAD 2022-09-12T19:06:51.257 DEBUG:boto:Path: /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:51.258 DEBUG:boto:Data: 2022-09-12T19:06:51.258 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.259 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:51.259 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.260 DEBUG:boto:Params: {} 2022-09-12T19:06:51.260 DEBUG:boto:Token: None 2022-09-12T19:06:51.261 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:51.262 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fOZk/fWaNSnesPWK+dD4OgeDU8o= 2022-09-12T19:06:51.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fOZk/fWaNSnesPWK+dD4OgeDU8o=', 'Content-Length': '0'} 2022-09-12T19:06:51.264 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'JcC9N13lklBVOx234Q8MXQ8kAM8ByWg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000087394831668412f-00631f834b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.265 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.265 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.266 DEBUG:boto:path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:51.266 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:51.267 DEBUG:boto:Method: HEAD 2022-09-12T19:06:51.267 DEBUG:boto:Path: /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:51.268 DEBUG:boto:Data: 2022-09-12T19:06:51.269 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.269 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.270 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.270 DEBUG:boto:Params: {} 2022-09-12T19:06:51.271 DEBUG:boto:Token: None 2022-09-12T19:06:51.271 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:51.272 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fOZk/fWaNSnesPWK+dD4OgeDU8o= 2022-09-12T19:06:51.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fOZk/fWaNSnesPWK+dD4OgeDU8o=', 'Content-Length': '0'} 2022-09-12T19:06:51.275 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'JcC9N13lklBVOx234Q8MXQ8kAM8ByWg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000455d6d645bfd18a6-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:51.276 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.277 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.277 DEBUG:boto:path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:51.278 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:51.279 DEBUG:boto:Method: GET 2022-09-12T19:06:51.279 DEBUG:boto:Path: /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:51.280 DEBUG:boto:Data: 2022-09-12T19:06:51.280 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.281 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:51.281 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.282 DEBUG:boto:Params: {} 2022-09-12T19:06:51.283 DEBUG:boto:Token: None 2022-09-12T19:06:51.283 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:51.284 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MBz8wCng1+PNOyGLEZ/TxIgWopE= 2022-09-12T19:06:51.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MBz8wCng1+PNOyGLEZ/TxIgWopE=', 'Content-Length': '0'} 2022-09-12T19:06:51.287 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'JcC9N13lklBVOx234Q8MXQ8kAM8ByWg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086d9aebfdf5ad7b7-00631f834b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.334 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.335 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.335 DEBUG:boto:path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:51.336 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:51.337 DEBUG:boto:Method: GET 2022-09-12T19:06:51.337 DEBUG:boto:Path: /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:51.338 DEBUG:boto:Data: 2022-09-12T19:06:51.338 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.339 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.339 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.340 DEBUG:boto:Params: {} 2022-09-12T19:06:51.340 DEBUG:boto:Token: None 2022-09-12T19:06:51.341 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:51.342 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MBz8wCng1+PNOyGLEZ/TxIgWopE= 2022-09-12T19:06:51.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MBz8wCng1+PNOyGLEZ/TxIgWopE=', 'Content-Length': '0'} 2022-09-12T19:06:51.345 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'JcC9N13lklBVOx234Q8MXQ8kAM8ByWg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eec5413975690e2c-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.346 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:51.346 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:51.347 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:51.347 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:51.348 DEBUG:boto:Method: GET 2022-09-12T19:06:51.349 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:51.349 DEBUG:boto:Data: 2022-09-12T19:06:51.350 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.351 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:51.351 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.352 DEBUG:boto:Params: {} 2022-09-12T19:06:51.352 DEBUG:boto:Token: None 2022-09-12T19:06:51.353 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/ 2022-09-12T19:06:51.354 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0= 2022-09-12T19:06:51.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0=', 'Content-Length': '0'} 2022-09-12T19:06:51.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070e195a510ca6cb9-00631f834b-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.358 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:51.359 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:51.360 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:51.360 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:51.361 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:51.361 DEBUG:boto:Method: GET 2022-09-12T19:06:51.362 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:51.362 DEBUG:boto:Data: 2022-09-12T19:06:51.363 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.364 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.364 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.365 DEBUG:boto:Params: {} 2022-09-12T19:06:51.366 DEBUG:boto:Token: None 2022-09-12T19:06:51.366 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/ 2022-09-12T19:06:51.367 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0= 2022-09-12T19:06:51.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0=', 'Content-Length': '0'} 2022-09-12T19:06:51.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000699592700f2d58eb-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.372 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:51.373 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.374 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.374 DEBUG:boto:Method: HEAD 2022-09-12T19:06:51.375 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:51.376 DEBUG:boto:Data: 2022-09-12T19:06:51.376 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.377 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:51.377 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.378 DEBUG:boto:Params: {} 2022-09-12T19:06:51.378 DEBUG:boto:Token: None 2022-09-12T19:06:51.379 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj 2022-09-12T19:06:51.380 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FgALKL3prsLBGofnjrT6ORBquz0= 2022-09-12T19:06:51.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FgALKL3prsLBGofnjrT6ORBquz0=', 'Content-Length': '0'} 2022-09-12T19:06:51.384 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a22a4c26854dcd5-00631f834b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.386 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.387 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.388 DEBUG:boto:Method: HEAD 2022-09-12T19:06:51.389 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:51.390 DEBUG:boto:Data: 2022-09-12T19:06:51.391 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.392 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.393 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.394 DEBUG:boto:Params: {} 2022-09-12T19:06:51.395 DEBUG:boto:Token: None 2022-09-12T19:06:51.396 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj 2022-09-12T19:06:51.397 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FgALKL3prsLBGofnjrT6ORBquz0= 2022-09-12T19:06:51.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FgALKL3prsLBGofnjrT6ORBquz0=', 'Content-Length': '0'} 2022-09-12T19:06:51.403 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000946f21ed467ebaf3-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:51.405 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.406 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.406 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:51.407 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:51.407 DEBUG:boto:Method: GET 2022-09-12T19:06:51.408 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:51.409 DEBUG:boto:Data: 2022-09-12T19:06:51.410 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.411 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:51.412 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.413 DEBUG:boto:Params: {} 2022-09-12T19:06:51.414 DEBUG:boto:Token: None 2022-09-12T19:06:51.415 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:51.416 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A2rzOZiGTdArdCLG+ahlkht4jKU= 2022-09-12T19:06:51.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A2rzOZiGTdArdCLG+ahlkht4jKU=', 'Content-Length': '0'} 2022-09-12T19:06:51.419 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ad9f3730a94f7e7-00631f834b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.420 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.420 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.421 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:51.421 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:51.422 DEBUG:boto:Method: GET 2022-09-12T19:06:51.423 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:51.424 DEBUG:boto:Data: 2022-09-12T19:06:51.425 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.426 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.427 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.428 DEBUG:boto:Params: {} 2022-09-12T19:06:51.429 DEBUG:boto:Token: None 2022-09-12T19:06:51.430 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:51.431 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A2rzOZiGTdArdCLG+ahlkht4jKU= 2022-09-12T19:06:51.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A2rzOZiGTdArdCLG+ahlkht4jKU=', 'Content-Length': '0'} 2022-09-12T19:06:51.435 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000971fab62e3dd9a2c-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.478 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:51.479 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:51.479 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:51.480 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:51.480 DEBUG:boto:Method: GET 2022-09-12T19:06:51.481 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:51.481 DEBUG:boto:Data: 2022-09-12T19:06:51.482 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.483 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:51.483 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.484 DEBUG:boto:Params: {} 2022-09-12T19:06:51.484 DEBUG:boto:Token: None 2022-09-12T19:06:51.485 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/ 2022-09-12T19:06:51.486 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0= 2022-09-12T19:06:51.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0=', 'Content-Length': '0'} 2022-09-12T19:06:51.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e55e4083886b1e9-00631f834b-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.490 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:51.490 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:51.491 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:51.491 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:51.492 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:51.493 DEBUG:boto:Method: GET 2022-09-12T19:06:51.493 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:51.494 DEBUG:boto:Data: 2022-09-12T19:06:51.494 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.495 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.495 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.496 DEBUG:boto:Params: {} 2022-09-12T19:06:51.496 DEBUG:boto:Token: None 2022-09-12T19:06:51.497 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/ 2022-09-12T19:06:51.498 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0= 2022-09-12T19:06:51.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0=', 'Content-Length': '0'} 2022-09-12T19:06:51.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b02245b0739c2b7-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.502 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:51.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:51.503 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.504 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.504 DEBUG:boto:path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:51.505 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:51.505 DEBUG:boto:Method: GET 2022-09-12T19:06:51.506 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:51.506 DEBUG:boto:Data: 2022-09-12T19:06:51.507 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.507 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:51.508 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.508 DEBUG:boto:Params: {} 2022-09-12T19:06:51.509 DEBUG:boto:Token: None 2022-09-12T19:06:51.510 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:51.510 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UvdDq46HFqdkig5SlUcY5YqWzXI= 2022-09-12T19:06:51.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UvdDq46HFqdkig5SlUcY5YqWzXI=', 'Content-Length': '0'} 2022-09-12T19:06:51.513 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000018c23fb40ebfe113-00631f834b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.514 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.514 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.515 DEBUG:boto:path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:51.516 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:51.516 DEBUG:boto:Method: GET 2022-09-12T19:06:51.517 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:51.517 DEBUG:boto:Data: 2022-09-12T19:06:51.518 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.518 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.519 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.519 DEBUG:boto:Params: {} 2022-09-12T19:06:51.520 DEBUG:boto:Token: None 2022-09-12T19:06:51.521 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:51.521 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UvdDq46HFqdkig5SlUcY5YqWzXI= 2022-09-12T19:06:51.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UvdDq46HFqdkig5SlUcY5YqWzXI=', 'Content-Length': '0'} 2022-09-12T19:06:51.525 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af5defd942e66271-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.526 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.526 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.527 DEBUG:boto:path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:51.527 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:51.528 DEBUG:boto:Method: HEAD 2022-09-12T19:06:51.528 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:51.529 DEBUG:boto:Data: 2022-09-12T19:06:51.529 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.530 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:51.530 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.531 DEBUG:boto:Params: {} 2022-09-12T19:06:51.532 DEBUG:boto:Token: None 2022-09-12T19:06:51.532 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:51.533 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8tMd2xot7hWxa08NuA+VSiY7z6U= 2022-09-12T19:06:51.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8tMd2xot7hWxa08NuA+VSiY7z6U=', 'Content-Length': '0'} 2022-09-12T19:06:51.536 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb9a09126d2edaa6-00631f834b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.536 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.537 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.537 DEBUG:boto:path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:51.538 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:51.538 DEBUG:boto:Method: HEAD 2022-09-12T19:06:51.539 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:51.539 DEBUG:boto:Data: 2022-09-12T19:06:51.540 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.541 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.541 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.542 DEBUG:boto:Params: {} 2022-09-12T19:06:51.542 DEBUG:boto:Token: None 2022-09-12T19:06:51.543 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:51.543 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8tMd2xot7hWxa08NuA+VSiY7z6U= 2022-09-12T19:06:51.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8tMd2xot7hWxa08NuA+VSiY7z6U=', 'Content-Length': '0'} 2022-09-12T19:06:51.546 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce5fec9b3dd119fa-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:51.547 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.548 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.548 DEBUG:boto:path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:51.549 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:51.549 DEBUG:boto:Method: GET 2022-09-12T19:06:51.550 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:51.551 DEBUG:boto:Data: 2022-09-12T19:06:51.551 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.552 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:51.552 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.553 DEBUG:boto:Params: {} 2022-09-12T19:06:51.553 DEBUG:boto:Token: None 2022-09-12T19:06:51.554 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:51.555 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UvdDq46HFqdkig5SlUcY5YqWzXI= 2022-09-12T19:06:51.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UvdDq46HFqdkig5SlUcY5YqWzXI=', 'Content-Length': '0'} 2022-09-12T19:06:51.557 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ea0a9e87165748c-00631f834b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.558 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.559 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.559 DEBUG:boto:path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:51.560 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:51.560 DEBUG:boto:Method: GET 2022-09-12T19:06:51.561 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:51.562 DEBUG:boto:Data: 2022-09-12T19:06:51.562 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.563 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.564 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.564 DEBUG:boto:Params: {} 2022-09-12T19:06:51.565 DEBUG:boto:Token: None 2022-09-12T19:06:51.566 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:51.567 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UvdDq46HFqdkig5SlUcY5YqWzXI= 2022-09-12T19:06:51.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UvdDq46HFqdkig5SlUcY5YqWzXI=', 'Content-Length': '0'} 2022-09-12T19:06:51.569 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c289dcc4c7d5b655-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.570 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:51.571 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:51.571 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:51.572 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:51.572 DEBUG:boto:Method: GET 2022-09-12T19:06:51.573 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:51.573 DEBUG:boto:Data: 2022-09-12T19:06:51.574 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.574 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:51.575 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.576 DEBUG:boto:Params: {} 2022-09-12T19:06:51.576 DEBUG:boto:Token: None 2022-09-12T19:06:51.577 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/ 2022-09-12T19:06:51.577 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0= 2022-09-12T19:06:51.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0=', 'Content-Length': '0'} 2022-09-12T19:06:51.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d11a4904e51df83-00631f834b-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.582 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:51.583 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:51.583 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:51.584 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:51.584 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:51.585 DEBUG:boto:Method: GET 2022-09-12T19:06:51.586 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:51.586 DEBUG:boto:Data: 2022-09-12T19:06:51.587 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.587 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.588 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.588 DEBUG:boto:Params: {} 2022-09-12T19:06:51.589 DEBUG:boto:Token: None 2022-09-12T19:06:51.590 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/ 2022-09-12T19:06:51.590 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0= 2022-09-12T19:06:51.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0=', 'Content-Length': '0'} 2022-09-12T19:06:51.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5340319fea1fa92-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.595 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:51.596 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.596 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.597 DEBUG:boto:Method: HEAD 2022-09-12T19:06:51.597 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:51.598 DEBUG:boto:Data: 2022-09-12T19:06:51.598 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.599 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:51.599 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.600 DEBUG:boto:Params: {} 2022-09-12T19:06:51.600 DEBUG:boto:Token: None 2022-09-12T19:06:51.601 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj 2022-09-12T19:06:51.602 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FgALKL3prsLBGofnjrT6ORBquz0= 2022-09-12T19:06:51.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FgALKL3prsLBGofnjrT6ORBquz0=', 'Content-Length': '0'} 2022-09-12T19:06:51.606 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b617c9374eda5883-00631f834b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.606 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.607 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.607 DEBUG:boto:Method: HEAD 2022-09-12T19:06:51.608 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:51.608 DEBUG:boto:Data: 2022-09-12T19:06:51.609 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.609 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.610 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.610 DEBUG:boto:Params: {} 2022-09-12T19:06:51.611 DEBUG:boto:Token: None 2022-09-12T19:06:51.611 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj 2022-09-12T19:06:51.612 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FgALKL3prsLBGofnjrT6ORBquz0= 2022-09-12T19:06:51.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FgALKL3prsLBGofnjrT6ORBquz0=', 'Content-Length': '0'} 2022-09-12T19:06:51.616 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc9902755080abb3-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:51.617 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.618 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.618 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:51.619 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:51.619 DEBUG:boto:Method: GET 2022-09-12T19:06:51.620 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:51.620 DEBUG:boto:Data: 2022-09-12T19:06:51.621 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.621 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:51.622 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.623 DEBUG:boto:Params: {} 2022-09-12T19:06:51.623 DEBUG:boto:Token: None 2022-09-12T19:06:51.624 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:51.624 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A2rzOZiGTdArdCLG+ahlkht4jKU= 2022-09-12T19:06:51.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A2rzOZiGTdArdCLG+ahlkht4jKU=', 'Content-Length': '0'} 2022-09-12T19:06:51.627 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cadc6e20f016191c-00631f834b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.628 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:51.628 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:51.629 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:51.629 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:51.630 DEBUG:boto:Method: GET 2022-09-12T19:06:51.630 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:51.631 DEBUG:boto:Data: 2022-09-12T19:06:51.631 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.632 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.632 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.633 DEBUG:boto:Params: {} 2022-09-12T19:06:51.633 DEBUG:boto:Token: None 2022-09-12T19:06:51.634 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:51.635 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A2rzOZiGTdArdCLG+ahlkht4jKU= 2022-09-12T19:06:51.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A2rzOZiGTdArdCLG+ahlkht4jKU=', 'Content-Length': '0'} 2022-09-12T19:06:51.637 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a99e2e54f98184f-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.682 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:51.683 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:51.683 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:51.684 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:51.684 DEBUG:boto:Method: GET 2022-09-12T19:06:51.685 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:51.685 DEBUG:boto:Data: 2022-09-12T19:06:51.686 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.687 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:51.687 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.688 DEBUG:boto:Params: {} 2022-09-12T19:06:51.688 DEBUG:boto:Token: None 2022-09-12T19:06:51.689 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/ 2022-09-12T19:06:51.689 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0= 2022-09-12T19:06:51.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0=', 'Content-Length': '0'} 2022-09-12T19:06:51.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f07598f92f496bc2-00631f834b-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.694 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:51.694 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:51.695 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:51.696 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:51.696 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:51.697 DEBUG:boto:Method: GET 2022-09-12T19:06:51.697 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:51.698 DEBUG:boto:Data: 2022-09-12T19:06:51.698 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.699 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.699 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.700 DEBUG:boto:Params: {} 2022-09-12T19:06:51.700 DEBUG:boto:Token: None 2022-09-12T19:06:51.701 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/ 2022-09-12T19:06:51.702 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0= 2022-09-12T19:06:51.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0=', 'Content-Length': '0'} 2022-09-12T19:06:51.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000220ece676be7050-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.707 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:51.708 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-19 zones={test-zone2, test-zone1} 2022-09-12T19:06:51.708 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-19 zones={test-zone3, test-zone1} 2022-09-12T19:06:51.709 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:51.709 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:51.710 DEBUG:boto:Method: HEAD 2022-09-12T19:06:51.711 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:06:51.711 DEBUG:boto:Data: 2022-09-12T19:06:51.712 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.712 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:51.713 DEBUG:boto:Port: 8001 2022-09-12T19:06:51.714 DEBUG:boto:Params: {} 2022-09-12T19:06:51.714 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T19:06:51.715 DEBUG:boto:Token: None 2022-09-12T19:06:51.716 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/ 2022-09-12T19:06:51.716 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sLEpp/YVPviKNxtu4vYT4nnJx9U= 2022-09-12T19:06:51.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sLEpp/YVPviKNxtu4vYT4nnJx9U=', 'Content-Length': '0'} 2022-09-12T19:06:51.722 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004caff2c1b41685b0-00631f834b-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.723 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:51.724 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:51.725 DEBUG:boto:Method: HEAD 2022-09-12T19:06:51.726 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:06:51.726 DEBUG:boto:Data: 2022-09-12T19:06:51.727 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.728 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.729 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.730 DEBUG:boto:Params: {} 2022-09-12T19:06:51.730 DEBUG:boto:Token: None 2022-09-12T19:06:51.731 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/ 2022-09-12T19:06:51.732 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sLEpp/YVPviKNxtu4vYT4nnJx9U= 2022-09-12T19:06:51.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sLEpp/YVPviKNxtu4vYT4nnJx9U=', 'Content-Length': '0'} 2022-09-12T19:06:51.735 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002727fd24c81acdc-00631f834b-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.736 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:06:51.736 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:51.737 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:51.737 DEBUG:boto:path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:51.738 DEBUG:boto:auth_path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:51.738 DEBUG:boto:Method: GET 2022-09-12T19:06:51.739 DEBUG:boto:Path: /jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:51.740 DEBUG:boto:Data: 2022-09-12T19:06:51.741 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.741 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:51.742 DEBUG:boto:Port: 8001 2022-09-12T19:06:51.742 DEBUG:boto:Params: {} 2022-09-12T19:06:51.743 DEBUG:boto:Token: None 2022-09-12T19:06:51.744 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/?versions 2022-09-12T19:06:51.744 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HCfjd6cV7GwZaWIrO45JeuNSfJc= 2022-09-12T19:06:51.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HCfjd6cV7GwZaWIrO45JeuNSfJc=', 'Content-Length': '0'} 2022-09-12T19:06:51.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000244ca09db489b0-00631f834b-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.749 DEBUG:boto:b'jdpiak-19999falsedummydAjTD8IRanrz7e2F.KgapZJi58DocMetrue2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZqtrue2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ulfalse2022-09-12T19:05:07.485Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTM67hyaYWRVq578zUsjsXjOzkUYTIvAfalse2022-09-12T19:05:07.581Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rhfalse2022-09-12T19:05:07.457Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpwncGyI2j8AeHH4hjzwsep2DWsp-l-Afalse2022-09-12T19:05:07.383Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsGTjDAFqZF1WL9AP98wlNVRWMwOQwYQfalse2022-09-12T19:05:07.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT.yM--FII-PB1GReM85mkNRGYz6Dkbzfalse2022-09-12T19:05:07.238Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXb5UZAvutXF6GhBCMdUQgjOLi998.hKfalse2022-09-12T19:05:07.343Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjztQRphsNbUdApzTcOcagUJ56uTwo7J2false2022-09-12T19:05:07.274Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKatr07t9mQV0b0gx44aB8d9VyeAhMS1false2022-09-12T19:05:07.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUBsYcCA94H55K4ucww5vXNAxuAMKNSGfalse2022-09-12T19:05:07.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkxSkXmfXKy0sdqagFrYSQNi7tMOKBoWfalse2022-09-12T19:05:07.199Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0false2022-09-12T19:05:06.995Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK.LUooPdPnXZMzfYIxw-xagQpErk4CXfalse2022-09-12T19:05:07.075Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtQHk21B862H4VHQxrZWEog8n-qgO99Qfalse2022-09-12T19:05:07.038Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQHXOu7SGV87vItdBeqaL.hYUlX--Ty8false2022-09-12T19:05:06.952Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT-rka2v4NcjXyfB8GzT0u7KHTz6OcF3false2022-09-12T19:05:06.866Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgdUGnE7SHabmaVwE2tf-YcFxl2tBu7Hfalse2022-09-12T19:05:06.908Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBAbppx4RylEKnBrYesXt1QPJgsAhIBbfalse2022-09-12T19:05:06.803Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNDM-EGOTc7UZbg9YQZwcM2uEk6P03H2false2022-09-12T19:05:06.773Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje.6ruI53roqy4mmyPgzQ1kqMdajruFAfalse2022-09-12T19:05:06.834Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLIaoO70PR8fRbTLQ-5Q.roWjA0lswqEfalse2022-09-12T19:05:06.733Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlnpiHoSs91QVujLSWqHhvEGzZrF3-sAfalse2022-09-12T19:05:06.698Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp5OIjHSzwvUDkIBYzRPdKBGezjIkrn-false2022-09-12T19:05:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjV.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9false2022-09-12T19:05:06.565Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiJi00oqpo4jP5T2qIUe4H2lLKJSYXZdfalse2022-09-12T19:05:06.598Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjj0eE53eOEuIue0LhYTVuZStUTmcOj7ofalse2022-09-12T19:05:06.634Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjACPdCUDfm32QLq6XbWYcqHiCLcEU2JYfalse2022-09-12T19:05:06.489Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJcC9N13lklBVOx234Q8MXQ8kAM8ByWgfalse2022-09-12T19:05:06.383Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjipYOl-Gqt65Y3nx.nVL2kL2DrysHNSYfalse2022-09-12T19:05:06.415Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:51.751 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:06:51.752 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.752 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.753 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.753 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.754 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.754 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.755 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.755 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.756 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.756 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.757 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.757 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.762 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.762 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.766 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:06:51.767 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:51.767 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:51.768 DEBUG:boto:path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:51.768 DEBUG:boto:auth_path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:51.769 DEBUG:boto:Method: GET 2022-09-12T19:06:51.770 DEBUG:boto:Path: /jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:51.770 DEBUG:boto:Data: 2022-09-12T19:06:51.771 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.771 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.772 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.772 DEBUG:boto:Params: {} 2022-09-12T19:06:51.773 DEBUG:boto:Token: None 2022-09-12T19:06:51.773 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/?versions 2022-09-12T19:06:51.774 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HCfjd6cV7GwZaWIrO45JeuNSfJc= 2022-09-12T19:06:51.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HCfjd6cV7GwZaWIrO45JeuNSfJc=', 'Content-Length': '0'} 2022-09-12T19:06:51.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079885d9ee95531dd-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.779 DEBUG:boto:b'jdpiak-19999falsedummydAjTD8IRanrz7e2F.KgapZJi58DocMetrue2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZqtrue2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ulfalse2022-09-12T19:05:07.485Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTM67hyaYWRVq578zUsjsXjOzkUYTIvAfalse2022-09-12T19:05:07.581Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rhfalse2022-09-12T19:05:07.457Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpwncGyI2j8AeHH4hjzwsep2DWsp-l-Afalse2022-09-12T19:05:07.383Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsGTjDAFqZF1WL9AP98wlNVRWMwOQwYQfalse2022-09-12T19:05:07.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT.yM--FII-PB1GReM85mkNRGYz6Dkbzfalse2022-09-12T19:05:07.238Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXb5UZAvutXF6GhBCMdUQgjOLi998.hKfalse2022-09-12T19:05:07.343Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjztQRphsNbUdApzTcOcagUJ56uTwo7J2false2022-09-12T19:05:07.274Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKatr07t9mQV0b0gx44aB8d9VyeAhMS1false2022-09-12T19:05:07.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUBsYcCA94H55K4ucww5vXNAxuAMKNSGfalse2022-09-12T19:05:07.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkxSkXmfXKy0sdqagFrYSQNi7tMOKBoWfalse2022-09-12T19:05:07.199Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0false2022-09-12T19:05:06.995Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK.LUooPdPnXZMzfYIxw-xagQpErk4CXfalse2022-09-12T19:05:07.075Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtQHk21B862H4VHQxrZWEog8n-qgO99Qfalse2022-09-12T19:05:07.038Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQHXOu7SGV87vItdBeqaL.hYUlX--Ty8false2022-09-12T19:05:06.952Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT-rka2v4NcjXyfB8GzT0u7KHTz6OcF3false2022-09-12T19:05:06.866Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgdUGnE7SHabmaVwE2tf-YcFxl2tBu7Hfalse2022-09-12T19:05:06.908Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBAbppx4RylEKnBrYesXt1QPJgsAhIBbfalse2022-09-12T19:05:06.803Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNDM-EGOTc7UZbg9YQZwcM2uEk6P03H2false2022-09-12T19:05:06.773Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje.6ruI53roqy4mmyPgzQ1kqMdajruFAfalse2022-09-12T19:05:06.834Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLIaoO70PR8fRbTLQ-5Q.roWjA0lswqEfalse2022-09-12T19:05:06.733Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlnpiHoSs91QVujLSWqHhvEGzZrF3-sAfalse2022-09-12T19:05:06.698Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp5OIjHSzwvUDkIBYzRPdKBGezjIkrn-false2022-09-12T19:05:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjV.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9false2022-09-12T19:05:06.565Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiJi00oqpo4jP5T2qIUe4H2lLKJSYXZdfalse2022-09-12T19:05:06.598Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjj0eE53eOEuIue0LhYTVuZStUTmcOj7ofalse2022-09-12T19:05:06.634Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjACPdCUDfm32QLq6XbWYcqHiCLcEU2JYfalse2022-09-12T19:05:06.489Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJcC9N13lklBVOx234Q8MXQ8kAM8ByWgfalse2022-09-12T19:05:06.383Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjipYOl-Gqt65Y3nx.nVL2kL2DrysHNSYfalse2022-09-12T19:05:06.415Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:51.781 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:06:51.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.788 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.788 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.789 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.789 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.792 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.792 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.793 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.793 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.794 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.794 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.795 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.796 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.796 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:51.797 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:51.797 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:51.798 DEBUG:boto:path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:51.798 DEBUG:boto:auth_path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:51.799 DEBUG:boto:Method: GET 2022-09-12T19:06:51.799 DEBUG:boto:Path: /jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:51.800 DEBUG:boto:Data: 2022-09-12T19:06:51.800 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.801 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:51.802 DEBUG:boto:Port: 8001 2022-09-12T19:06:51.802 DEBUG:boto:Params: {} 2022-09-12T19:06:51.803 DEBUG:boto:Token: None 2022-09-12T19:06:51.804 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/?versions 2022-09-12T19:06:51.804 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HCfjd6cV7GwZaWIrO45JeuNSfJc= 2022-09-12T19:06:51.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HCfjd6cV7GwZaWIrO45JeuNSfJc=', 'Content-Length': '0'} 2022-09-12T19:06:51.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b8915171b6c83c8-00631f834b-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.810 DEBUG:boto:b'jdpiak-19999falsedummydAjTD8IRanrz7e2F.KgapZJi58DocMetrue2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZqtrue2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ulfalse2022-09-12T19:05:07.485Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTM67hyaYWRVq578zUsjsXjOzkUYTIvAfalse2022-09-12T19:05:07.581Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rhfalse2022-09-12T19:05:07.457Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpwncGyI2j8AeHH4hjzwsep2DWsp-l-Afalse2022-09-12T19:05:07.383Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsGTjDAFqZF1WL9AP98wlNVRWMwOQwYQfalse2022-09-12T19:05:07.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT.yM--FII-PB1GReM85mkNRGYz6Dkbzfalse2022-09-12T19:05:07.238Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXb5UZAvutXF6GhBCMdUQgjOLi998.hKfalse2022-09-12T19:05:07.343Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjztQRphsNbUdApzTcOcagUJ56uTwo7J2false2022-09-12T19:05:07.274Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKatr07t9mQV0b0gx44aB8d9VyeAhMS1false2022-09-12T19:05:07.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUBsYcCA94H55K4ucww5vXNAxuAMKNSGfalse2022-09-12T19:05:07.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkxSkXmfXKy0sdqagFrYSQNi7tMOKBoWfalse2022-09-12T19:05:07.199Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0false2022-09-12T19:05:06.995Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK.LUooPdPnXZMzfYIxw-xagQpErk4CXfalse2022-09-12T19:05:07.075Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtQHk21B862H4VHQxrZWEog8n-qgO99Qfalse2022-09-12T19:05:07.038Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQHXOu7SGV87vItdBeqaL.hYUlX--Ty8false2022-09-12T19:05:06.952Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT-rka2v4NcjXyfB8GzT0u7KHTz6OcF3false2022-09-12T19:05:06.866Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgdUGnE7SHabmaVwE2tf-YcFxl2tBu7Hfalse2022-09-12T19:05:06.908Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBAbppx4RylEKnBrYesXt1QPJgsAhIBbfalse2022-09-12T19:05:06.803Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNDM-EGOTc7UZbg9YQZwcM2uEk6P03H2false2022-09-12T19:05:06.773Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje.6ruI53roqy4mmyPgzQ1kqMdajruFAfalse2022-09-12T19:05:06.834Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLIaoO70PR8fRbTLQ-5Q.roWjA0lswqEfalse2022-09-12T19:05:06.733Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlnpiHoSs91QVujLSWqHhvEGzZrF3-sAfalse2022-09-12T19:05:06.698Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp5OIjHSzwvUDkIBYzRPdKBGezjIkrn-false2022-09-12T19:05:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjV.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9false2022-09-12T19:05:06.565Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiJi00oqpo4jP5T2qIUe4H2lLKJSYXZdfalse2022-09-12T19:05:06.598Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjj0eE53eOEuIue0LhYTVuZStUTmcOj7ofalse2022-09-12T19:05:06.634Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjACPdCUDfm32QLq6XbWYcqHiCLcEU2JYfalse2022-09-12T19:05:06.489Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJcC9N13lklBVOx234Q8MXQ8kAM8ByWgfalse2022-09-12T19:05:06.383Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjipYOl-Gqt65Y3nx.nVL2kL2DrysHNSYfalse2022-09-12T19:05:06.415Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:51.812 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:51.812 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:51.813 DEBUG:boto:path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:51.813 DEBUG:boto:auth_path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:51.814 DEBUG:boto:Method: GET 2022-09-12T19:06:51.814 DEBUG:boto:Path: /jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:51.815 DEBUG:boto:Data: 2022-09-12T19:06:51.815 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.816 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.817 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.817 DEBUG:boto:Params: {} 2022-09-12T19:06:51.818 DEBUG:boto:Token: None 2022-09-12T19:06:51.819 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/?versions 2022-09-12T19:06:51.820 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HCfjd6cV7GwZaWIrO45JeuNSfJc= 2022-09-12T19:06:51.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HCfjd6cV7GwZaWIrO45JeuNSfJc=', 'Content-Length': '0'} 2022-09-12T19:06:51.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae35254e0d76726d-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.825 DEBUG:boto:b'jdpiak-19999falsedummydAjTD8IRanrz7e2F.KgapZJi58DocMetrue2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZqtrue2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ulfalse2022-09-12T19:05:07.485Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTM67hyaYWRVq578zUsjsXjOzkUYTIvAfalse2022-09-12T19:05:07.581Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rhfalse2022-09-12T19:05:07.457Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpwncGyI2j8AeHH4hjzwsep2DWsp-l-Afalse2022-09-12T19:05:07.383Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsGTjDAFqZF1WL9AP98wlNVRWMwOQwYQfalse2022-09-12T19:05:07.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT.yM--FII-PB1GReM85mkNRGYz6Dkbzfalse2022-09-12T19:05:07.238Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXb5UZAvutXF6GhBCMdUQgjOLi998.hKfalse2022-09-12T19:05:07.343Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjztQRphsNbUdApzTcOcagUJ56uTwo7J2false2022-09-12T19:05:07.274Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKatr07t9mQV0b0gx44aB8d9VyeAhMS1false2022-09-12T19:05:07.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUBsYcCA94H55K4ucww5vXNAxuAMKNSGfalse2022-09-12T19:05:07.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkxSkXmfXKy0sdqagFrYSQNi7tMOKBoWfalse2022-09-12T19:05:07.199Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0false2022-09-12T19:05:06.995Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK.LUooPdPnXZMzfYIxw-xagQpErk4CXfalse2022-09-12T19:05:07.075Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtQHk21B862H4VHQxrZWEog8n-qgO99Qfalse2022-09-12T19:05:07.038Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQHXOu7SGV87vItdBeqaL.hYUlX--Ty8false2022-09-12T19:05:06.952Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT-rka2v4NcjXyfB8GzT0u7KHTz6OcF3false2022-09-12T19:05:06.866Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgdUGnE7SHabmaVwE2tf-YcFxl2tBu7Hfalse2022-09-12T19:05:06.908Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBAbppx4RylEKnBrYesXt1QPJgsAhIBbfalse2022-09-12T19:05:06.803Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNDM-EGOTc7UZbg9YQZwcM2uEk6P03H2false2022-09-12T19:05:06.773Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje.6ruI53roqy4mmyPgzQ1kqMdajruFAfalse2022-09-12T19:05:06.834Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLIaoO70PR8fRbTLQ-5Q.roWjA0lswqEfalse2022-09-12T19:05:06.733Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlnpiHoSs91QVujLSWqHhvEGzZrF3-sAfalse2022-09-12T19:05:06.698Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp5OIjHSzwvUDkIBYzRPdKBGezjIkrn-false2022-09-12T19:05:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjV.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9false2022-09-12T19:05:06.565Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiJi00oqpo4jP5T2qIUe4H2lLKJSYXZdfalse2022-09-12T19:05:06.598Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjj0eE53eOEuIue0LhYTVuZStUTmcOj7ofalse2022-09-12T19:05:06.634Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjACPdCUDfm32QLq6XbWYcqHiCLcEU2JYfalse2022-09-12T19:05:06.489Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJcC9N13lklBVOx234Q8MXQ8kAM8ByWgfalse2022-09-12T19:05:06.383Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjipYOl-Gqt65Y3nx.nVL2kL2DrysHNSYfalse2022-09-12T19:05:06.415Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:51.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:06:51.828 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:51.829 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:51.829 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.830 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.830 DEBUG:boto:Method: GET 2022-09-12T19:06:51.831 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.832 DEBUG:boto:Data: 2022-09-12T19:06:51.832 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.833 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:51.834 DEBUG:boto:Port: 8001 2022-09-12T19:06:51.835 DEBUG:boto:Params: {} 2022-09-12T19:06:51.835 DEBUG:boto:Token: None 2022-09-12T19:06:51.836 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.836 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W8k7PcmIQwWA/XUguh2n9T63qW0= 2022-09-12T19:06:51.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W8k7PcmIQwWA/XUguh2n9T63qW0=', 'Content-Length': '0'} 2022-09-12T19:06:51.839 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000092c0e369a0231b11-00631f834b-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.840 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:51.841 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:51.841 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.842 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.842 DEBUG:boto:Method: GET 2022-09-12T19:06:51.843 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.843 DEBUG:boto:Data: 2022-09-12T19:06:51.844 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.844 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.845 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.845 DEBUG:boto:Params: {} 2022-09-12T19:06:51.846 DEBUG:boto:Token: None 2022-09-12T19:06:51.847 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.848 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W8k7PcmIQwWA/XUguh2n9T63qW0= 2022-09-12T19:06:51.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W8k7PcmIQwWA/XUguh2n9T63qW0=', 'Content-Length': '0'} 2022-09-12T19:06:51.851 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000015f50552ba8faa0-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.852 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:51.852 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:51.853 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.853 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.854 DEBUG:boto:Method: HEAD 2022-09-12T19:06:51.855 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.855 DEBUG:boto:Data: 2022-09-12T19:06:51.856 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.856 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:51.857 DEBUG:boto:Port: 8001 2022-09-12T19:06:51.857 DEBUG:boto:Params: {} 2022-09-12T19:06:51.858 DEBUG:boto:Token: None 2022-09-12T19:06:51.858 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.859 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Qjpy5JP19lgK/6NUlEy9ZCS+Ypw= 2022-09-12T19:06:51.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Qjpy5JP19lgK/6NUlEy9ZCS+Ypw=', 'Content-Length': '0'} 2022-09-12T19:06:51.862 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c066f477ccdf26e2-00631f834b-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.863 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:51.863 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:51.864 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.864 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.865 DEBUG:boto:Method: HEAD 2022-09-12T19:06:51.865 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.866 DEBUG:boto:Data: 2022-09-12T19:06:51.866 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.867 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.867 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.868 DEBUG:boto:Params: {} 2022-09-12T19:06:51.868 DEBUG:boto:Token: None 2022-09-12T19:06:51.869 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.869 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Qjpy5JP19lgK/6NUlEy9ZCS+Ypw= 2022-09-12T19:06:51.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Qjpy5JP19lgK/6NUlEy9ZCS+Ypw=', 'Content-Length': '0'} 2022-09-12T19:06:51.872 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bc63141727eea3b8-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:06:51.873 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:51.874 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:51.875 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.875 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.876 DEBUG:boto:Method: GET 2022-09-12T19:06:51.876 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.877 DEBUG:boto:Data: 2022-09-12T19:06:51.877 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.878 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:51.878 DEBUG:boto:Port: 8001 2022-09-12T19:06:51.879 DEBUG:boto:Params: {} 2022-09-12T19:06:51.880 DEBUG:boto:Token: None 2022-09-12T19:06:51.880 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.881 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W8k7PcmIQwWA/XUguh2n9T63qW0= 2022-09-12T19:06:51.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W8k7PcmIQwWA/XUguh2n9T63qW0=', 'Content-Length': '0'} 2022-09-12T19:06:51.884 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000391bb119ada4cde-00631f834b-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.884 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:51.885 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:51.886 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.886 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.887 DEBUG:boto:Method: GET 2022-09-12T19:06:51.887 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.888 DEBUG:boto:Data: 2022-09-12T19:06:51.888 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.889 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.889 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.890 DEBUG:boto:Params: {} 2022-09-12T19:06:51.890 DEBUG:boto:Token: None 2022-09-12T19:06:51.891 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.892 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W8k7PcmIQwWA/XUguh2n9T63qW0= 2022-09-12T19:06:51.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W8k7PcmIQwWA/XUguh2n9T63qW0=', 'Content-Length': '0'} 2022-09-12T19:06:51.894 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f3fce65b70ff355a-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.895 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:51.896 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:51.896 DEBUG:boto:path=/jdpiak-19/?prefix=dummy 2022-09-12T19:06:51.897 DEBUG:boto:auth_path=/jdpiak-19/?prefix=dummy 2022-09-12T19:06:51.897 DEBUG:boto:Method: GET 2022-09-12T19:06:51.898 DEBUG:boto:Path: /jdpiak-19/?prefix=dummy 2022-09-12T19:06:51.898 DEBUG:boto:Data: 2022-09-12T19:06:51.899 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.900 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:51.900 DEBUG:boto:Port: 8001 2022-09-12T19:06:51.901 DEBUG:boto:Params: {} 2022-09-12T19:06:51.901 DEBUG:boto:Token: None 2022-09-12T19:06:51.902 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/ 2022-09-12T19:06:51.902 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0= 2022-09-12T19:06:51.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0=', 'Content-Length': '0'} 2022-09-12T19:06:51.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003736afddf1ce2c1a-00631f834b-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.906 DEBUG:boto:b'jdpiak-19dummy1000falsedummy2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:51.907 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:51.908 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:51.908 DEBUG:boto:path=/jdpiak-19/?prefix=dummy 2022-09-12T19:06:51.909 DEBUG:boto:auth_path=/jdpiak-19/?prefix=dummy 2022-09-12T19:06:51.909 DEBUG:boto:Method: GET 2022-09-12T19:06:51.910 DEBUG:boto:Path: /jdpiak-19/?prefix=dummy 2022-09-12T19:06:51.911 DEBUG:boto:Data: 2022-09-12T19:06:51.911 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.912 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.912 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.913 DEBUG:boto:Params: {} 2022-09-12T19:06:51.913 DEBUG:boto:Token: None 2022-09-12T19:06:51.914 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/ 2022-09-12T19:06:51.915 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0= 2022-09-12T19:06:51.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hcgyYhF4O+/SVyQZzoEGVQiTft0=', 'Content-Length': '0'} 2022-09-12T19:06:51.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003928a1835065b870-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.964 DEBUG:boto:b'jdpiak-19dummy1000falsedummy2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:51.965 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:51.966 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:51.966 DEBUG:boto:Method: HEAD 2022-09-12T19:06:51.967 DEBUG:boto:Path: /jdpiak-19/dummy 2022-09-12T19:06:51.967 DEBUG:boto:Data: 2022-09-12T19:06:51.968 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.968 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:51.969 DEBUG:boto:Port: 8001 2022-09-12T19:06:51.969 DEBUG:boto:Params: {} 2022-09-12T19:06:51.970 DEBUG:boto:Token: None 2022-09-12T19:06:51.970 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/dummy 2022-09-12T19:06:51.971 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UQHO3SH41bwJxlq6gSE6wmmxXjk= 2022-09-12T19:06:51.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UQHO3SH41bwJxlq6gSE6wmmxXjk=', 'Content-Length': '0'} 2022-09-12T19:06:51.974 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008d242869cd2448f5-00631f834b-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.975 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:51.976 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:51.976 DEBUG:boto:Method: HEAD 2022-09-12T19:06:51.977 DEBUG:boto:Path: /jdpiak-19/dummy 2022-09-12T19:06:51.977 DEBUG:boto:Data: 2022-09-12T19:06:51.978 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.979 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:51.979 DEBUG:boto:Port: 8000 2022-09-12T19:06:51.980 DEBUG:boto:Params: {} 2022-09-12T19:06:51.980 DEBUG:boto:Token: None 2022-09-12T19:06:51.981 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/dummy 2022-09-12T19:06:51.982 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UQHO3SH41bwJxlq6gSE6wmmxXjk= 2022-09-12T19:06:51.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UQHO3SH41bwJxlq6gSE6wmmxXjk=', 'Content-Length': '0'} 2022-09-12T19:06:51.985 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d8584756b609928d-00631f834b-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:06:51.987 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:51.988 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:51.988 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.989 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.989 DEBUG:boto:Method: GET 2022-09-12T19:06:51.990 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.991 DEBUG:boto:Data: 2022-09-12T19:06:51.991 DEBUG:boto:Headers: {} 2022-09-12T19:06:51.992 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:51.992 DEBUG:boto:Port: 8001 2022-09-12T19:06:51.993 DEBUG:boto:Params: {} 2022-09-12T19:06:51.993 DEBUG:boto:Token: None 2022-09-12T19:06:51.994 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:51 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:51.995 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W8k7PcmIQwWA/XUguh2n9T63qW0= 2022-09-12T19:06:51.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W8k7PcmIQwWA/XUguh2n9T63qW0=', 'Content-Length': '0'} 2022-09-12T19:06:51.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a92cd77cd93dba34-00631f834b-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:51.999 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:51.999 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:52.000 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:52.000 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:52.001 DEBUG:boto:Method: GET 2022-09-12T19:06:52.001 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:52.002 DEBUG:boto:Data: 2022-09-12T19:06:52.003 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.003 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.004 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.005 DEBUG:boto:Params: {} 2022-09-12T19:06:52.005 DEBUG:boto:Token: None 2022-09-12T19:06:52.006 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:52.006 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OjkXWgk7Y20ZeIE1wJSSJ0lbeCc= 2022-09-12T19:06:52.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OjkXWgk7Y20ZeIE1wJSSJ0lbeCc=', 'Content-Length': '0'} 2022-09-12T19:06:52.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b8884f9018207a18-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.011 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:52.012 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:52.012 DEBUG:boto:path=/jdpiak-19/?prefix=dummy 2022-09-12T19:06:52.013 DEBUG:boto:auth_path=/jdpiak-19/?prefix=dummy 2022-09-12T19:06:52.013 DEBUG:boto:Method: GET 2022-09-12T19:06:52.014 DEBUG:boto:Path: /jdpiak-19/?prefix=dummy 2022-09-12T19:06:52.014 DEBUG:boto:Data: 2022-09-12T19:06:52.015 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.016 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:52.016 DEBUG:boto:Port: 8001 2022-09-12T19:06:52.017 DEBUG:boto:Params: {} 2022-09-12T19:06:52.018 DEBUG:boto:Token: None 2022-09-12T19:06:52.018 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/ 2022-09-12T19:06:52.019 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q= 2022-09-12T19:06:52.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q=', 'Content-Length': '0'} 2022-09-12T19:06:52.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031e3172da081b21e-00631f834c-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.023 DEBUG:boto:b'jdpiak-19dummy1000falsedummy2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:52.025 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:52.026 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:52.026 DEBUG:boto:path=/jdpiak-19/?prefix=dummy 2022-09-12T19:06:52.027 DEBUG:boto:auth_path=/jdpiak-19/?prefix=dummy 2022-09-12T19:06:52.027 DEBUG:boto:Method: GET 2022-09-12T19:06:52.028 DEBUG:boto:Path: /jdpiak-19/?prefix=dummy 2022-09-12T19:06:52.028 DEBUG:boto:Data: 2022-09-12T19:06:52.029 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.029 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.030 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.031 DEBUG:boto:Params: {} 2022-09-12T19:06:52.031 DEBUG:boto:Token: None 2022-09-12T19:06:52.032 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/ 2022-09-12T19:06:52.033 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q= 2022-09-12T19:06:52.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q=', 'Content-Length': '0'} 2022-09-12T19:06:52.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eedb91a6e704566a-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.037 DEBUG:boto:b'jdpiak-19dummy1000falsedummy2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:52.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:52.039 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.039 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.040 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.041 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.042 DEBUG:boto:Method: GET 2022-09-12T19:06:52.042 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.043 DEBUG:boto:Data: 2022-09-12T19:06:52.043 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.044 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:52.044 DEBUG:boto:Port: 8001 2022-09-12T19:06:52.045 DEBUG:boto:Params: {} 2022-09-12T19:06:52.046 DEBUG:boto:Token: None 2022-09-12T19:06:52.046 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.047 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G9S2/1J1GGwfrHXupbgdOQnFt7M= 2022-09-12T19:06:52.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G9S2/1J1GGwfrHXupbgdOQnFt7M=', 'Content-Length': '0'} 2022-09-12T19:06:52.051 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002c81a3d33451122-00631f834c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.052 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.052 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.053 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.054 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.055 DEBUG:boto:Method: GET 2022-09-12T19:06:52.056 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.057 DEBUG:boto:Data: 2022-09-12T19:06:52.058 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.067 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.068 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.069 DEBUG:boto:Params: {} 2022-09-12T19:06:52.069 DEBUG:boto:Token: None 2022-09-12T19:06:52.070 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.071 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G9S2/1J1GGwfrHXupbgdOQnFt7M= 2022-09-12T19:06:52.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G9S2/1J1GGwfrHXupbgdOQnFt7M=', 'Content-Length': '0'} 2022-09-12T19:06:52.074 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006026e1dfb0d767c-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.075 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.076 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.076 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.077 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.077 DEBUG:boto:Method: HEAD 2022-09-12T19:06:52.078 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.079 DEBUG:boto:Data: 2022-09-12T19:06:52.080 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.081 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:52.082 DEBUG:boto:Port: 8001 2022-09-12T19:06:52.083 DEBUG:boto:Params: {} 2022-09-12T19:06:52.084 DEBUG:boto:Token: None 2022-09-12T19:06:52.085 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.085 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0QtgZQy8Hy+VJ7+2r22+VSRWov8= 2022-09-12T19:06:52.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0QtgZQy8Hy+VJ7+2r22+VSRWov8=', 'Content-Length': '0'} 2022-09-12T19:06:52.088 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000814e9afd9a0ad5f4-00631f834c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.089 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.089 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.090 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.090 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.091 DEBUG:boto:Method: HEAD 2022-09-12T19:06:52.091 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.092 DEBUG:boto:Data: 2022-09-12T19:06:52.092 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.093 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.094 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.094 DEBUG:boto:Params: {} 2022-09-12T19:06:52.095 DEBUG:boto:Token: None 2022-09-12T19:06:52.096 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.096 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0QtgZQy8Hy+VJ7+2r22+VSRWov8= 2022-09-12T19:06:52.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0QtgZQy8Hy+VJ7+2r22+VSRWov8=', 'Content-Length': '0'} 2022-09-12T19:06:52.099 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008aa77c96492d1f1f-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:52.101 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.101 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.102 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.103 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.104 DEBUG:boto:Method: GET 2022-09-12T19:06:52.104 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.105 DEBUG:boto:Data: 2022-09-12T19:06:52.106 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.107 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:52.108 DEBUG:boto:Port: 8001 2022-09-12T19:06:52.109 DEBUG:boto:Params: {} 2022-09-12T19:06:52.111 DEBUG:boto:Token: None 2022-09-12T19:06:52.112 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.114 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G9S2/1J1GGwfrHXupbgdOQnFt7M= 2022-09-12T19:06:52.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G9S2/1J1GGwfrHXupbgdOQnFt7M=', 'Content-Length': '0'} 2022-09-12T19:06:52.117 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ce6861f514542e6-00631f834c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.158 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.159 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.160 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.161 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.162 DEBUG:boto:Method: GET 2022-09-12T19:06:52.162 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.163 DEBUG:boto:Data: 2022-09-12T19:06:52.163 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.164 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.164 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.165 DEBUG:boto:Params: {} 2022-09-12T19:06:52.166 DEBUG:boto:Token: None 2022-09-12T19:06:52.166 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.167 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G9S2/1J1GGwfrHXupbgdOQnFt7M= 2022-09-12T19:06:52.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G9S2/1J1GGwfrHXupbgdOQnFt7M=', 'Content-Length': '0'} 2022-09-12T19:06:52.170 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f3a73f9603e5ef6b-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.171 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:52.172 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:52.172 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.173 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.173 DEBUG:boto:Method: GET 2022-09-12T19:06:52.174 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:52.175 DEBUG:boto:Data: 2022-09-12T19:06:52.175 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.176 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:52.176 DEBUG:boto:Port: 8001 2022-09-12T19:06:52.177 DEBUG:boto:Params: {} 2022-09-12T19:06:52.177 DEBUG:boto:Token: None 2022-09-12T19:06:52.178 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/ 2022-09-12T19:06:52.178 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q= 2022-09-12T19:06:52.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q=', 'Content-Length': '0'} 2022-09-12T19:06:52.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9d123d99bd06b57-00631f834c-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.183 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:52.183 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:52.184 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:52.185 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.185 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.186 DEBUG:boto:Method: GET 2022-09-12T19:06:52.186 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:52.187 DEBUG:boto:Data: 2022-09-12T19:06:52.187 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.188 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.188 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.189 DEBUG:boto:Params: {} 2022-09-12T19:06:52.190 DEBUG:boto:Token: None 2022-09-12T19:06:52.190 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/ 2022-09-12T19:06:52.191 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q= 2022-09-12T19:06:52.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q=', 'Content-Length': '0'} 2022-09-12T19:06:52.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000589ff3fdf818e2ec-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.242 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:52.244 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.245 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.246 DEBUG:boto:Method: HEAD 2022-09-12T19:06:52.247 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:52.247 DEBUG:boto:Data: 2022-09-12T19:06:52.248 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.249 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:52.250 DEBUG:boto:Port: 8001 2022-09-12T19:06:52.251 DEBUG:boto:Params: {} 2022-09-12T19:06:52.252 DEBUG:boto:Token: None 2022-09-12T19:06:52.254 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj 2022-09-12T19:06:52.255 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ghjtoDqlATXYCcFxaA/fsEbNyGM= 2022-09-12T19:06:52.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ghjtoDqlATXYCcFxaA/fsEbNyGM=', 'Content-Length': '0'} 2022-09-12T19:06:52.260 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000088697620e5f16001-00631f834c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.261 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.262 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.263 DEBUG:boto:Method: HEAD 2022-09-12T19:06:52.264 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:52.265 DEBUG:boto:Data: 2022-09-12T19:06:52.266 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.267 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.268 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.269 DEBUG:boto:Params: {} 2022-09-12T19:06:52.271 DEBUG:boto:Token: None 2022-09-12T19:06:52.272 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj 2022-09-12T19:06:52.273 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ghjtoDqlATXYCcFxaA/fsEbNyGM= 2022-09-12T19:06:52.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ghjtoDqlATXYCcFxaA/fsEbNyGM=', 'Content-Length': '0'} 2022-09-12T19:06:52.278 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064f5b5826ae9ad60-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:52.280 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.281 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.282 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.283 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.284 DEBUG:boto:Method: GET 2022-09-12T19:06:52.285 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.286 DEBUG:boto:Data: 2022-09-12T19:06:52.287 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.289 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:52.290 DEBUG:boto:Port: 8001 2022-09-12T19:06:52.291 DEBUG:boto:Params: {} 2022-09-12T19:06:52.292 DEBUG:boto:Token: None 2022-09-12T19:06:52.293 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.294 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G9S2/1J1GGwfrHXupbgdOQnFt7M= 2022-09-12T19:06:52.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G9S2/1J1GGwfrHXupbgdOQnFt7M=', 'Content-Length': '0'} 2022-09-12T19:06:52.298 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b489a810a0cdcca4-00631f834c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.299 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.300 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.300 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.301 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.302 DEBUG:boto:Method: GET 2022-09-12T19:06:52.302 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.303 DEBUG:boto:Data: 2022-09-12T19:06:52.303 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.304 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.304 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.305 DEBUG:boto:Params: {} 2022-09-12T19:06:52.306 DEBUG:boto:Token: None 2022-09-12T19:06:52.306 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.307 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G9S2/1J1GGwfrHXupbgdOQnFt7M= 2022-09-12T19:06:52.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G9S2/1J1GGwfrHXupbgdOQnFt7M=', 'Content-Length': '0'} 2022-09-12T19:06:52.310 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000370e096618dea540-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.311 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:52.311 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:52.312 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.312 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.313 DEBUG:boto:Method: GET 2022-09-12T19:06:52.314 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:52.314 DEBUG:boto:Data: 2022-09-12T19:06:52.315 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.316 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:52.316 DEBUG:boto:Port: 8001 2022-09-12T19:06:52.317 DEBUG:boto:Params: {} 2022-09-12T19:06:52.318 DEBUG:boto:Token: None 2022-09-12T19:06:52.319 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/ 2022-09-12T19:06:52.321 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q= 2022-09-12T19:06:52.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q=', 'Content-Length': '0'} 2022-09-12T19:06:52.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd1913d6be6a346d-00631f834c-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.366 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:52.367 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:52.368 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:52.368 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.369 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.369 DEBUG:boto:Method: GET 2022-09-12T19:06:52.370 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:52.370 DEBUG:boto:Data: 2022-09-12T19:06:52.371 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.372 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.372 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.373 DEBUG:boto:Params: {} 2022-09-12T19:06:52.374 DEBUG:boto:Token: None 2022-09-12T19:06:52.374 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/ 2022-09-12T19:06:52.375 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q= 2022-09-12T19:06:52.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q=', 'Content-Length': '0'} 2022-09-12T19:06:52.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd780748306f03fe-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.379 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:52.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:52.381 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.381 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.382 DEBUG:boto:path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:52.382 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:52.383 DEBUG:boto:Method: GET 2022-09-12T19:06:52.383 DEBUG:boto:Path: /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:52.384 DEBUG:boto:Data: 2022-09-12T19:06:52.385 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.385 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:52.386 DEBUG:boto:Port: 8001 2022-09-12T19:06:52.386 DEBUG:boto:Params: {} 2022-09-12T19:06:52.387 DEBUG:boto:Token: None 2022-09-12T19:06:52.387 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:52.388 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eJL+epHZ4+Jyqhex2fU0b/qayyc= 2022-09-12T19:06:52.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eJL+epHZ4+Jyqhex2fU0b/qayyc=', 'Content-Length': '0'} 2022-09-12T19:06:52.391 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ed27e447f75de27-00631f834c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.392 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.393 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.393 DEBUG:boto:path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:52.394 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:52.395 DEBUG:boto:Method: GET 2022-09-12T19:06:52.396 DEBUG:boto:Path: /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:52.397 DEBUG:boto:Data: 2022-09-12T19:06:52.398 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.399 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.399 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.400 DEBUG:boto:Params: {} 2022-09-12T19:06:52.400 DEBUG:boto:Token: None 2022-09-12T19:06:52.401 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:52.402 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eJL+epHZ4+Jyqhex2fU0b/qayyc= 2022-09-12T19:06:52.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eJL+epHZ4+Jyqhex2fU0b/qayyc=', 'Content-Length': '0'} 2022-09-12T19:06:52.404 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3578c2164070f74-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.446 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.447 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.447 DEBUG:boto:path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:52.448 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:52.448 DEBUG:boto:Method: HEAD 2022-09-12T19:06:52.449 DEBUG:boto:Path: /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:52.449 DEBUG:boto:Data: 2022-09-12T19:06:52.450 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.451 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:52.451 DEBUG:boto:Port: 8001 2022-09-12T19:06:52.452 DEBUG:boto:Params: {} 2022-09-12T19:06:52.452 DEBUG:boto:Token: None 2022-09-12T19:06:52.453 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:52.453 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bX1grRgIiBFaXY2PEHp0n9uuZTc= 2022-09-12T19:06:52.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bX1grRgIiBFaXY2PEHp0n9uuZTc=', 'Content-Length': '0'} 2022-09-12T19:06:52.456 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f638583e81754c42-00631f834c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.457 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.458 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.459 DEBUG:boto:path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:52.459 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:52.460 DEBUG:boto:Method: HEAD 2022-09-12T19:06:52.460 DEBUG:boto:Path: /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:52.461 DEBUG:boto:Data: 2022-09-12T19:06:52.461 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.462 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.462 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.463 DEBUG:boto:Params: {} 2022-09-12T19:06:52.464 DEBUG:boto:Token: None 2022-09-12T19:06:52.464 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:52.465 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bX1grRgIiBFaXY2PEHp0n9uuZTc= 2022-09-12T19:06:52.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bX1grRgIiBFaXY2PEHp0n9uuZTc=', 'Content-Length': '0'} 2022-09-12T19:06:52.467 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008089daaab2c0c42c-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:52.469 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.469 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.470 DEBUG:boto:path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:52.471 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:52.472 DEBUG:boto:Method: GET 2022-09-12T19:06:52.473 DEBUG:boto:Path: /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:52.474 DEBUG:boto:Data: 2022-09-12T19:06:52.476 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.477 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:52.478 DEBUG:boto:Port: 8001 2022-09-12T19:06:52.479 DEBUG:boto:Params: {} 2022-09-12T19:06:52.480 DEBUG:boto:Token: None 2022-09-12T19:06:52.481 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:52.481 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eJL+epHZ4+Jyqhex2fU0b/qayyc= 2022-09-12T19:06:52.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eJL+epHZ4+Jyqhex2fU0b/qayyc=', 'Content-Length': '0'} 2022-09-12T19:06:52.484 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef5531e9472eb1a0-00631f834c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.485 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.486 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.486 DEBUG:boto:path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:52.487 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:52.487 DEBUG:boto:Method: GET 2022-09-12T19:06:52.488 DEBUG:boto:Path: /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:52.488 DEBUG:boto:Data: 2022-09-12T19:06:52.489 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.490 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.490 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.491 DEBUG:boto:Params: {} 2022-09-12T19:06:52.492 DEBUG:boto:Token: None 2022-09-12T19:06:52.493 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:06:52.493 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eJL+epHZ4+Jyqhex2fU0b/qayyc= 2022-09-12T19:06:52.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eJL+epHZ4+Jyqhex2fU0b/qayyc=', 'Content-Length': '0'} 2022-09-12T19:06:52.496 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e91f882885f4d10-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.497 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:52.498 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:52.498 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.499 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.500 DEBUG:boto:Method: GET 2022-09-12T19:06:52.501 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:52.501 DEBUG:boto:Data: 2022-09-12T19:06:52.502 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.502 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:52.503 DEBUG:boto:Port: 8001 2022-09-12T19:06:52.503 DEBUG:boto:Params: {} 2022-09-12T19:06:52.504 DEBUG:boto:Token: None 2022-09-12T19:06:52.505 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/ 2022-09-12T19:06:52.505 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q= 2022-09-12T19:06:52.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q=', 'Content-Length': '0'} 2022-09-12T19:06:52.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000127ae33f39cf8376-00631f834c-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.510 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:52.511 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:52.511 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:52.512 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.512 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.513 DEBUG:boto:Method: GET 2022-09-12T19:06:52.514 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:52.514 DEBUG:boto:Data: 2022-09-12T19:06:52.515 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.516 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.516 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.517 DEBUG:boto:Params: {} 2022-09-12T19:06:52.518 DEBUG:boto:Token: None 2022-09-12T19:06:52.518 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/ 2022-09-12T19:06:52.519 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q= 2022-09-12T19:06:52.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q=', 'Content-Length': '0'} 2022-09-12T19:06:52.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa4efea6c728e429-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.523 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:52.524 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.525 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.525 DEBUG:boto:Method: HEAD 2022-09-12T19:06:52.526 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:52.527 DEBUG:boto:Data: 2022-09-12T19:06:52.528 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.528 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:52.529 DEBUG:boto:Port: 8001 2022-09-12T19:06:52.530 DEBUG:boto:Params: {} 2022-09-12T19:06:52.531 DEBUG:boto:Token: None 2022-09-12T19:06:52.531 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj 2022-09-12T19:06:52.532 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ghjtoDqlATXYCcFxaA/fsEbNyGM= 2022-09-12T19:06:52.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ghjtoDqlATXYCcFxaA/fsEbNyGM=', 'Content-Length': '0'} 2022-09-12T19:06:52.536 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035180f16a8ad99c2-00631f834c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.538 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.539 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.540 DEBUG:boto:Method: HEAD 2022-09-12T19:06:52.540 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:52.541 DEBUG:boto:Data: 2022-09-12T19:06:52.541 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.542 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.542 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.543 DEBUG:boto:Params: {} 2022-09-12T19:06:52.544 DEBUG:boto:Token: None 2022-09-12T19:06:52.545 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj 2022-09-12T19:06:52.547 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ghjtoDqlATXYCcFxaA/fsEbNyGM= 2022-09-12T19:06:52.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ghjtoDqlATXYCcFxaA/fsEbNyGM=', 'Content-Length': '0'} 2022-09-12T19:06:52.552 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c21bd3b359632255-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:52.554 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.554 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.555 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.556 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.557 DEBUG:boto:Method: GET 2022-09-12T19:06:52.558 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.559 DEBUG:boto:Data: 2022-09-12T19:06:52.559 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.560 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:52.561 DEBUG:boto:Port: 8001 2022-09-12T19:06:52.561 DEBUG:boto:Params: {} 2022-09-12T19:06:52.562 DEBUG:boto:Token: None 2022-09-12T19:06:52.563 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.565 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G9S2/1J1GGwfrHXupbgdOQnFt7M= 2022-09-12T19:06:52.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G9S2/1J1GGwfrHXupbgdOQnFt7M=', 'Content-Length': '0'} 2022-09-12T19:06:52.570 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a67920121a755ab-00631f834c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.571 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.572 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.573 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.574 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.575 DEBUG:boto:Method: GET 2022-09-12T19:06:52.576 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.576 DEBUG:boto:Data: 2022-09-12T19:06:52.577 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.579 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.580 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.581 DEBUG:boto:Params: {} 2022-09-12T19:06:52.582 DEBUG:boto:Token: None 2022-09-12T19:06:52.583 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.585 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G9S2/1J1GGwfrHXupbgdOQnFt7M= 2022-09-12T19:06:52.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G9S2/1J1GGwfrHXupbgdOQnFt7M=', 'Content-Length': '0'} 2022-09-12T19:06:52.588 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ad54f786953ff81-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.630 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:52.631 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:52.631 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.632 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.632 DEBUG:boto:Method: GET 2022-09-12T19:06:52.633 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:52.633 DEBUG:boto:Data: 2022-09-12T19:06:52.634 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.634 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:52.635 DEBUG:boto:Port: 8001 2022-09-12T19:06:52.636 DEBUG:boto:Params: {} 2022-09-12T19:06:52.636 DEBUG:boto:Token: None 2022-09-12T19:06:52.637 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/ 2022-09-12T19:06:52.637 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q= 2022-09-12T19:06:52.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q=', 'Content-Length': '0'} 2022-09-12T19:06:52.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fcea057adc92cc7-00631f834c-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.642 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:52.642 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:52.643 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:52.644 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.644 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.645 DEBUG:boto:Method: GET 2022-09-12T19:06:52.645 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:52.646 DEBUG:boto:Data: 2022-09-12T19:06:52.646 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.647 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.647 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.648 DEBUG:boto:Params: {} 2022-09-12T19:06:52.648 DEBUG:boto:Token: None 2022-09-12T19:06:52.649 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/ 2022-09-12T19:06:52.649 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q= 2022-09-12T19:06:52.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q=', 'Content-Length': '0'} 2022-09-12T19:06:52.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a4db4beddf458d7-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.654 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:52.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:52.655 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.656 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.656 DEBUG:boto:path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:52.657 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:52.657 DEBUG:boto:Method: GET 2022-09-12T19:06:52.658 DEBUG:boto:Path: /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:52.658 DEBUG:boto:Data: 2022-09-12T19:06:52.659 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.659 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:52.660 DEBUG:boto:Port: 8001 2022-09-12T19:06:52.660 DEBUG:boto:Params: {} 2022-09-12T19:06:52.661 DEBUG:boto:Token: None 2022-09-12T19:06:52.661 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:52.662 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GZp7FgjOKlNTNwLKlLeDY1b6ZFU= 2022-09-12T19:06:52.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GZp7FgjOKlNTNwLKlLeDY1b6ZFU=', 'Content-Length': '0'} 2022-09-12T19:06:52.665 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'TM67hyaYWRVq578zUsjsXjOzkUYTIvA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000043679e76820bbd7d-00631f834c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.666 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.666 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.667 DEBUG:boto:path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:52.667 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:52.668 DEBUG:boto:Method: GET 2022-09-12T19:06:52.668 DEBUG:boto:Path: /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:52.669 DEBUG:boto:Data: 2022-09-12T19:06:52.670 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.670 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.671 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.671 DEBUG:boto:Params: {} 2022-09-12T19:06:52.672 DEBUG:boto:Token: None 2022-09-12T19:06:52.673 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:52.673 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GZp7FgjOKlNTNwLKlLeDY1b6ZFU= 2022-09-12T19:06:52.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GZp7FgjOKlNTNwLKlLeDY1b6ZFU=', 'Content-Length': '0'} 2022-09-12T19:06:52.676 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'TM67hyaYWRVq578zUsjsXjOzkUYTIvA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000820b42ecda82aecb-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.677 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.677 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.678 DEBUG:boto:path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:52.678 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:52.679 DEBUG:boto:Method: HEAD 2022-09-12T19:06:52.679 DEBUG:boto:Path: /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:52.680 DEBUG:boto:Data: 2022-09-12T19:06:52.680 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.681 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:52.681 DEBUG:boto:Port: 8001 2022-09-12T19:06:52.682 DEBUG:boto:Params: {} 2022-09-12T19:06:52.682 DEBUG:boto:Token: None 2022-09-12T19:06:52.683 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:52.684 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:74CUaWvX53VHTSecyNEbPdTSszA= 2022-09-12T19:06:52.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:74CUaWvX53VHTSecyNEbPdTSszA=', 'Content-Length': '0'} 2022-09-12T19:06:52.686 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'TM67hyaYWRVq578zUsjsXjOzkUYTIvA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033992f95cda23b95-00631f834c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.687 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.687 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.688 DEBUG:boto:path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:52.688 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:52.689 DEBUG:boto:Method: HEAD 2022-09-12T19:06:52.689 DEBUG:boto:Path: /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:52.690 DEBUG:boto:Data: 2022-09-12T19:06:52.690 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.691 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.691 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.692 DEBUG:boto:Params: {} 2022-09-12T19:06:52.692 DEBUG:boto:Token: None 2022-09-12T19:06:52.693 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:52.693 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:74CUaWvX53VHTSecyNEbPdTSszA= 2022-09-12T19:06:52.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:74CUaWvX53VHTSecyNEbPdTSszA=', 'Content-Length': '0'} 2022-09-12T19:06:52.696 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'TM67hyaYWRVq578zUsjsXjOzkUYTIvA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d740ae04d1c249ba-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:52.697 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.698 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.698 DEBUG:boto:path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:52.699 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:52.699 DEBUG:boto:Method: GET 2022-09-12T19:06:52.700 DEBUG:boto:Path: /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:52.701 DEBUG:boto:Data: 2022-09-12T19:06:52.701 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.702 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:52.702 DEBUG:boto:Port: 8001 2022-09-12T19:06:52.703 DEBUG:boto:Params: {} 2022-09-12T19:06:52.703 DEBUG:boto:Token: None 2022-09-12T19:06:52.704 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:52.704 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GZp7FgjOKlNTNwLKlLeDY1b6ZFU= 2022-09-12T19:06:52.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GZp7FgjOKlNTNwLKlLeDY1b6ZFU=', 'Content-Length': '0'} 2022-09-12T19:06:52.707 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'TM67hyaYWRVq578zUsjsXjOzkUYTIvA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b30dfe40eb2986a4-00631f834c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.754 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.755 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.755 DEBUG:boto:path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:52.756 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:52.756 DEBUG:boto:Method: GET 2022-09-12T19:06:52.757 DEBUG:boto:Path: /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:52.757 DEBUG:boto:Data: 2022-09-12T19:06:52.758 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.759 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.760 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.760 DEBUG:boto:Params: {} 2022-09-12T19:06:52.761 DEBUG:boto:Token: None 2022-09-12T19:06:52.762 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:06:52.763 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GZp7FgjOKlNTNwLKlLeDY1b6ZFU= 2022-09-12T19:06:52.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GZp7FgjOKlNTNwLKlLeDY1b6ZFU=', 'Content-Length': '0'} 2022-09-12T19:06:52.766 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'TM67hyaYWRVq578zUsjsXjOzkUYTIvA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000beb12160c34b19d4-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.767 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:52.767 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:52.768 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.768 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.769 DEBUG:boto:Method: GET 2022-09-12T19:06:52.769 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:52.770 DEBUG:boto:Data: 2022-09-12T19:06:52.770 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.771 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:52.772 DEBUG:boto:Port: 8001 2022-09-12T19:06:52.773 DEBUG:boto:Params: {} 2022-09-12T19:06:52.774 DEBUG:boto:Token: None 2022-09-12T19:06:52.774 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/ 2022-09-12T19:06:52.775 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q= 2022-09-12T19:06:52.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q=', 'Content-Length': '0'} 2022-09-12T19:06:52.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060b1610737ee4c72-00631f834c-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.780 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:52.781 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:52.781 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:52.782 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.782 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.783 DEBUG:boto:Method: GET 2022-09-12T19:06:52.783 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:52.784 DEBUG:boto:Data: 2022-09-12T19:06:52.785 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.785 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.786 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.786 DEBUG:boto:Params: {} 2022-09-12T19:06:52.787 DEBUG:boto:Token: None 2022-09-12T19:06:52.787 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/ 2022-09-12T19:06:52.788 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q= 2022-09-12T19:06:52.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q=', 'Content-Length': '0'} 2022-09-12T19:06:52.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032c2b4fb4ee5ba90-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.793 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:52.794 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.795 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.795 DEBUG:boto:Method: HEAD 2022-09-12T19:06:52.796 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:52.797 DEBUG:boto:Data: 2022-09-12T19:06:52.797 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.798 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:52.798 DEBUG:boto:Port: 8001 2022-09-12T19:06:52.799 DEBUG:boto:Params: {} 2022-09-12T19:06:52.800 DEBUG:boto:Token: None 2022-09-12T19:06:52.800 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj 2022-09-12T19:06:52.801 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ghjtoDqlATXYCcFxaA/fsEbNyGM= 2022-09-12T19:06:52.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ghjtoDqlATXYCcFxaA/fsEbNyGM=', 'Content-Length': '0'} 2022-09-12T19:06:52.806 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090d384ec1a2b9050-00631f834c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.807 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.808 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.808 DEBUG:boto:Method: HEAD 2022-09-12T19:06:52.809 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:52.809 DEBUG:boto:Data: 2022-09-12T19:06:52.810 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.810 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.811 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.811 DEBUG:boto:Params: {} 2022-09-12T19:06:52.812 DEBUG:boto:Token: None 2022-09-12T19:06:52.813 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj 2022-09-12T19:06:52.813 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ghjtoDqlATXYCcFxaA/fsEbNyGM= 2022-09-12T19:06:52.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ghjtoDqlATXYCcFxaA/fsEbNyGM=', 'Content-Length': '0'} 2022-09-12T19:06:52.817 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000426c2a740cfeec42-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:52.819 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.820 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.820 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.821 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.821 DEBUG:boto:Method: GET 2022-09-12T19:06:52.822 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.822 DEBUG:boto:Data: 2022-09-12T19:06:52.823 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.823 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:52.824 DEBUG:boto:Port: 8001 2022-09-12T19:06:52.825 DEBUG:boto:Params: {} 2022-09-12T19:06:52.825 DEBUG:boto:Token: None 2022-09-12T19:06:52.826 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.826 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G9S2/1J1GGwfrHXupbgdOQnFt7M= 2022-09-12T19:06:52.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G9S2/1J1GGwfrHXupbgdOQnFt7M=', 'Content-Length': '0'} 2022-09-12T19:06:52.830 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af5e97c01901de92-00631f834c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.870 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.871 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.872 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.873 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.874 DEBUG:boto:Method: GET 2022-09-12T19:06:52.875 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.876 DEBUG:boto:Data: 2022-09-12T19:06:52.877 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.878 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.879 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.880 DEBUG:boto:Params: {} 2022-09-12T19:06:52.881 DEBUG:boto:Token: None 2022-09-12T19:06:52.882 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:52.883 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G9S2/1J1GGwfrHXupbgdOQnFt7M= 2022-09-12T19:06:52.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G9S2/1J1GGwfrHXupbgdOQnFt7M=', 'Content-Length': '0'} 2022-09-12T19:06:52.885 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2406f145f7b11fd-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.886 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:52.887 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:52.887 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.888 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.888 DEBUG:boto:Method: GET 2022-09-12T19:06:52.889 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:52.890 DEBUG:boto:Data: 2022-09-12T19:06:52.890 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.891 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:52.891 DEBUG:boto:Port: 8001 2022-09-12T19:06:52.892 DEBUG:boto:Params: {} 2022-09-12T19:06:52.892 DEBUG:boto:Token: None 2022-09-12T19:06:52.893 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/ 2022-09-12T19:06:52.894 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q= 2022-09-12T19:06:52.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q=', 'Content-Length': '0'} 2022-09-12T19:06:52.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007086bf617ae3f259-00631f834c-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.898 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:52.899 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:52.900 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:52.901 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.902 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.903 DEBUG:boto:Method: GET 2022-09-12T19:06:52.904 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:52.905 DEBUG:boto:Data: 2022-09-12T19:06:52.906 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.906 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.907 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.908 DEBUG:boto:Params: {} 2022-09-12T19:06:52.908 DEBUG:boto:Token: None 2022-09-12T19:06:52.909 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/ 2022-09-12T19:06:52.909 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q= 2022-09-12T19:06:52.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WaStB5W6eqL0MTXdSq7yPtfzq8Q=', 'Content-Length': '0'} 2022-09-12T19:06:52.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000773a4d02cc448e68-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.915 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:52.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:52.917 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.918 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.919 DEBUG:boto:path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:52.919 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:52.920 DEBUG:boto:Method: GET 2022-09-12T19:06:52.921 DEBUG:boto:Path: /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:52.921 DEBUG:boto:Data: 2022-09-12T19:06:52.922 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.922 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:52.923 DEBUG:boto:Port: 8001 2022-09-12T19:06:52.923 DEBUG:boto:Params: {} 2022-09-12T19:06:52.924 DEBUG:boto:Token: None 2022-09-12T19:06:52.925 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:52.925 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A05kldRQqaFkwlf+BQJQ4pz1juE= 2022-09-12T19:06:52.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A05kldRQqaFkwlf+BQJQ4pz1juE=', 'Content-Length': '0'} 2022-09-12T19:06:52.928 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', '-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000886b27d9445fe732-00631f834c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.930 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.931 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.932 DEBUG:boto:path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:52.933 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:52.934 DEBUG:boto:Method: GET 2022-09-12T19:06:52.935 DEBUG:boto:Path: /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:52.936 DEBUG:boto:Data: 2022-09-12T19:06:52.937 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.938 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.938 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.939 DEBUG:boto:Params: {} 2022-09-12T19:06:52.940 DEBUG:boto:Token: None 2022-09-12T19:06:52.940 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:52.941 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A05kldRQqaFkwlf+BQJQ4pz1juE= 2022-09-12T19:06:52.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A05kldRQqaFkwlf+BQJQ4pz1juE=', 'Content-Length': '0'} 2022-09-12T19:06:52.944 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', '-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006eca2281761a050-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.945 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.945 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.946 DEBUG:boto:path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:52.946 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:52.947 DEBUG:boto:Method: HEAD 2022-09-12T19:06:52.947 DEBUG:boto:Path: /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:52.948 DEBUG:boto:Data: 2022-09-12T19:06:52.948 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.949 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:52.949 DEBUG:boto:Port: 8001 2022-09-12T19:06:52.950 DEBUG:boto:Params: {} 2022-09-12T19:06:52.950 DEBUG:boto:Token: None 2022-09-12T19:06:52.951 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:52.951 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hdb9fdr8ytr37Lj3w2kQ+mYVnS0= 2022-09-12T19:06:52.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hdb9fdr8ytr37Lj3w2kQ+mYVnS0=', 'Content-Length': '0'} 2022-09-12T19:06:52.954 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', '-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0f16d3910b17658-00631f834c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.955 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.955 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.956 DEBUG:boto:path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:52.957 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:52.957 DEBUG:boto:Method: HEAD 2022-09-12T19:06:52.958 DEBUG:boto:Path: /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:52.958 DEBUG:boto:Data: 2022-09-12T19:06:52.959 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.959 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.960 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.961 DEBUG:boto:Params: {} 2022-09-12T19:06:52.961 DEBUG:boto:Token: None 2022-09-12T19:06:52.962 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:52.963 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hdb9fdr8ytr37Lj3w2kQ+mYVnS0= 2022-09-12T19:06:52.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hdb9fdr8ytr37Lj3w2kQ+mYVnS0=', 'Content-Length': '0'} 2022-09-12T19:06:52.966 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', '-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4bf4ad6e29b11dc-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:52.967 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.968 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.968 DEBUG:boto:path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:52.969 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:52.970 DEBUG:boto:Method: GET 2022-09-12T19:06:52.970 DEBUG:boto:Path: /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:52.971 DEBUG:boto:Data: 2022-09-12T19:06:52.971 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.972 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:52.972 DEBUG:boto:Port: 8001 2022-09-12T19:06:52.973 DEBUG:boto:Params: {} 2022-09-12T19:06:52.974 DEBUG:boto:Token: None 2022-09-12T19:06:52.975 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:52.976 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A05kldRQqaFkwlf+BQJQ4pz1juE= 2022-09-12T19:06:52.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A05kldRQqaFkwlf+BQJQ4pz1juE=', 'Content-Length': '0'} 2022-09-12T19:06:52.978 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', '-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001788620e979a1d7a-00631f834c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.979 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:52.980 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:52.981 DEBUG:boto:path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:52.981 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:52.982 DEBUG:boto:Method: GET 2022-09-12T19:06:52.983 DEBUG:boto:Path: /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:52.984 DEBUG:boto:Data: 2022-09-12T19:06:52.985 DEBUG:boto:Headers: {} 2022-09-12T19:06:52.986 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:52.987 DEBUG:boto:Port: 8000 2022-09-12T19:06:52.988 DEBUG:boto:Params: {} 2022-09-12T19:06:52.989 DEBUG:boto:Token: None 2022-09-12T19:06:52.989 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:52 GMT /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:06:52.990 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A05kldRQqaFkwlf+BQJQ4pz1juE= 2022-09-12T19:06:52.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A05kldRQqaFkwlf+BQJQ4pz1juE=', 'Content-Length': '0'} 2022-09-12T19:06:52.994 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', '-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0ed0415ed9d3307-00631f834c-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:52.995 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:52.996 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:52.997 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.997 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:52.998 DEBUG:boto:Method: GET 2022-09-12T19:06:52.999 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:52.999 DEBUG:boto:Data: 2022-09-12T19:06:53.000 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.000 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.001 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.002 DEBUG:boto:Params: {} 2022-09-12T19:06:53.002 DEBUG:boto:Token: None 2022-09-12T19:06:53.003 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/ 2022-09-12T19:06:53.003 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8= 2022-09-12T19:06:53.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8=', 'Content-Length': '0'} 2022-09-12T19:06:53.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000aabcd637fc395db-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.054 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:53.055 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:53.056 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:53.056 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.057 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.057 DEBUG:boto:Method: GET 2022-09-12T19:06:53.058 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:53.059 DEBUG:boto:Data: 2022-09-12T19:06:53.059 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.060 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.060 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.061 DEBUG:boto:Params: {} 2022-09-12T19:06:53.062 DEBUG:boto:Token: None 2022-09-12T19:06:53.062 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/ 2022-09-12T19:06:53.063 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8= 2022-09-12T19:06:53.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8=', 'Content-Length': '0'} 2022-09-12T19:06:53.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a39a5e2fc8d43d3b-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.067 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:53.068 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.069 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.070 DEBUG:boto:Method: HEAD 2022-09-12T19:06:53.070 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:53.071 DEBUG:boto:Data: 2022-09-12T19:06:53.072 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.073 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.073 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.074 DEBUG:boto:Params: {} 2022-09-12T19:06:53.075 DEBUG:boto:Token: None 2022-09-12T19:06:53.076 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj 2022-09-12T19:06:53.076 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9hUIlNVqxQQwnBvp1XB6gqWEUYo= 2022-09-12T19:06:53.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9hUIlNVqxQQwnBvp1XB6gqWEUYo=', 'Content-Length': '0'} 2022-09-12T19:06:53.081 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077fb96326c970569-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.083 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.084 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.084 DEBUG:boto:Method: HEAD 2022-09-12T19:06:53.085 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:53.085 DEBUG:boto:Data: 2022-09-12T19:06:53.086 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.086 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.087 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.087 DEBUG:boto:Params: {} 2022-09-12T19:06:53.088 DEBUG:boto:Token: None 2022-09-12T19:06:53.089 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj 2022-09-12T19:06:53.089 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9hUIlNVqxQQwnBvp1XB6gqWEUYo= 2022-09-12T19:06:53.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9hUIlNVqxQQwnBvp1XB6gqWEUYo=', 'Content-Length': '0'} 2022-09-12T19:06:53.093 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b489e09c3dcdc91d-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:53.095 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.095 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.096 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.096 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.097 DEBUG:boto:Method: GET 2022-09-12T19:06:53.098 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.098 DEBUG:boto:Data: 2022-09-12T19:06:53.099 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.100 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.100 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.102 DEBUG:boto:Params: {} 2022-09-12T19:06:53.103 DEBUG:boto:Token: None 2022-09-12T19:06:53.104 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.105 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JYEx2EaWUyI8tGvpT64xNVdxD/g= 2022-09-12T19:06:53.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JYEx2EaWUyI8tGvpT64xNVdxD/g=', 'Content-Length': '0'} 2022-09-12T19:06:53.109 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0e53e6b7138d6ce-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.109 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.110 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.111 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.112 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.113 DEBUG:boto:Method: GET 2022-09-12T19:06:53.114 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.115 DEBUG:boto:Data: 2022-09-12T19:06:53.115 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.116 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.117 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.118 DEBUG:boto:Params: {} 2022-09-12T19:06:53.119 DEBUG:boto:Token: None 2022-09-12T19:06:53.120 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.121 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JYEx2EaWUyI8tGvpT64xNVdxD/g= 2022-09-12T19:06:53.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JYEx2EaWUyI8tGvpT64xNVdxD/g=', 'Content-Length': '0'} 2022-09-12T19:06:53.124 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002bd0dd880bbd518d-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.166 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:53.167 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:53.168 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.168 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.169 DEBUG:boto:Method: GET 2022-09-12T19:06:53.169 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:53.170 DEBUG:boto:Data: 2022-09-12T19:06:53.171 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.171 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.172 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.172 DEBUG:boto:Params: {} 2022-09-12T19:06:53.173 DEBUG:boto:Token: None 2022-09-12T19:06:53.174 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/ 2022-09-12T19:06:53.175 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8= 2022-09-12T19:06:53.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8=', 'Content-Length': '0'} 2022-09-12T19:06:53.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012e32ddf3624f621-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.181 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:53.182 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:53.183 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:53.184 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.185 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.186 DEBUG:boto:Method: GET 2022-09-12T19:06:53.187 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:53.187 DEBUG:boto:Data: 2022-09-12T19:06:53.188 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.188 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.189 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.189 DEBUG:boto:Params: {} 2022-09-12T19:06:53.190 DEBUG:boto:Token: None 2022-09-12T19:06:53.191 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/ 2022-09-12T19:06:53.191 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8= 2022-09-12T19:06:53.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8=', 'Content-Length': '0'} 2022-09-12T19:06:53.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009589bae5b4761d6a-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.196 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:53.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:53.197 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.198 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.198 DEBUG:boto:path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:53.199 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:53.199 DEBUG:boto:Method: GET 2022-09-12T19:06:53.200 DEBUG:boto:Path: /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:53.200 DEBUG:boto:Data: 2022-09-12T19:06:53.201 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.202 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.202 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.203 DEBUG:boto:Params: {} 2022-09-12T19:06:53.203 DEBUG:boto:Token: None 2022-09-12T19:06:53.204 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:53.204 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EP4kcfe+edRNS2oo1qrIAPHcYYw= 2022-09-12T19:06:53.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EP4kcfe+edRNS2oo1qrIAPHcYYw=', 'Content-Length': '0'} 2022-09-12T19:06:53.208 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'pwncGyI2j8AeHH4hjzwsep2DWsp-l-A'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005cb692eeb9f82bc8-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.209 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.209 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.210 DEBUG:boto:path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:53.211 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:53.212 DEBUG:boto:Method: GET 2022-09-12T19:06:53.213 DEBUG:boto:Path: /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:53.214 DEBUG:boto:Data: 2022-09-12T19:06:53.215 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.216 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.217 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.218 DEBUG:boto:Params: {} 2022-09-12T19:06:53.219 DEBUG:boto:Token: None 2022-09-12T19:06:53.220 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:53.221 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EP4kcfe+edRNS2oo1qrIAPHcYYw= 2022-09-12T19:06:53.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EP4kcfe+edRNS2oo1qrIAPHcYYw=', 'Content-Length': '0'} 2022-09-12T19:06:53.226 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'pwncGyI2j8AeHH4hjzwsep2DWsp-l-A'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009bfd39caabc8013a-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.227 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.228 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.229 DEBUG:boto:path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:53.230 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:53.230 DEBUG:boto:Method: HEAD 2022-09-12T19:06:53.231 DEBUG:boto:Path: /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:53.232 DEBUG:boto:Data: 2022-09-12T19:06:53.232 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.233 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.234 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.235 DEBUG:boto:Params: {} 2022-09-12T19:06:53.236 DEBUG:boto:Token: None 2022-09-12T19:06:53.237 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:53.238 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qMNpzAniT9Rr2HOdfK8cE53aL3w= 2022-09-12T19:06:53.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qMNpzAniT9Rr2HOdfK8cE53aL3w=', 'Content-Length': '0'} 2022-09-12T19:06:53.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'pwncGyI2j8AeHH4hjzwsep2DWsp-l-A'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004199b0a99bd51761-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.243 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.244 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.245 DEBUG:boto:path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:53.246 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:53.246 DEBUG:boto:Method: HEAD 2022-09-12T19:06:53.247 DEBUG:boto:Path: /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:53.248 DEBUG:boto:Data: 2022-09-12T19:06:53.248 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.249 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.249 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.250 DEBUG:boto:Params: {} 2022-09-12T19:06:53.250 DEBUG:boto:Token: None 2022-09-12T19:06:53.251 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:53.251 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qMNpzAniT9Rr2HOdfK8cE53aL3w= 2022-09-12T19:06:53.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qMNpzAniT9Rr2HOdfK8cE53aL3w=', 'Content-Length': '0'} 2022-09-12T19:06:53.254 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'pwncGyI2j8AeHH4hjzwsep2DWsp-l-A'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031bf8b9365cb8976-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:53.256 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.256 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.257 DEBUG:boto:path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:53.257 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:53.258 DEBUG:boto:Method: GET 2022-09-12T19:06:53.259 DEBUG:boto:Path: /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:53.259 DEBUG:boto:Data: 2022-09-12T19:06:53.260 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.260 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.261 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.261 DEBUG:boto:Params: {} 2022-09-12T19:06:53.262 DEBUG:boto:Token: None 2022-09-12T19:06:53.262 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:53.263 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EP4kcfe+edRNS2oo1qrIAPHcYYw= 2022-09-12T19:06:53.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EP4kcfe+edRNS2oo1qrIAPHcYYw=', 'Content-Length': '0'} 2022-09-12T19:06:53.266 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'pwncGyI2j8AeHH4hjzwsep2DWsp-l-A'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003971ed33e750eb40-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.267 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.267 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.268 DEBUG:boto:path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:53.268 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:53.269 DEBUG:boto:Method: GET 2022-09-12T19:06:53.269 DEBUG:boto:Path: /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:53.270 DEBUG:boto:Data: 2022-09-12T19:06:53.270 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.271 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.272 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.272 DEBUG:boto:Params: {} 2022-09-12T19:06:53.273 DEBUG:boto:Token: None 2022-09-12T19:06:53.273 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:06:53.274 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EP4kcfe+edRNS2oo1qrIAPHcYYw= 2022-09-12T19:06:53.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EP4kcfe+edRNS2oo1qrIAPHcYYw=', 'Content-Length': '0'} 2022-09-12T19:06:53.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'pwncGyI2j8AeHH4hjzwsep2DWsp-l-A'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005edb671ab13006b3-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.318 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:53.319 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:53.320 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.321 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.322 DEBUG:boto:Method: GET 2022-09-12T19:06:53.323 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:53.323 DEBUG:boto:Data: 2022-09-12T19:06:53.324 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.325 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.326 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.327 DEBUG:boto:Params: {} 2022-09-12T19:06:53.328 DEBUG:boto:Token: None 2022-09-12T19:06:53.329 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/ 2022-09-12T19:06:53.330 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8= 2022-09-12T19:06:53.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8=', 'Content-Length': '0'} 2022-09-12T19:06:53.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c69fb845a57a30d0-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.336 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:53.338 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:53.339 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:53.340 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.341 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.342 DEBUG:boto:Method: GET 2022-09-12T19:06:53.343 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:53.344 DEBUG:boto:Data: 2022-09-12T19:06:53.345 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.346 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.347 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.348 DEBUG:boto:Params: {} 2022-09-12T19:06:53.349 DEBUG:boto:Token: None 2022-09-12T19:06:53.350 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/ 2022-09-12T19:06:53.351 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8= 2022-09-12T19:06:53.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8=', 'Content-Length': '0'} 2022-09-12T19:06:53.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bc4bff09519e7ed-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.357 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:53.359 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.360 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.361 DEBUG:boto:Method: HEAD 2022-09-12T19:06:53.362 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:53.363 DEBUG:boto:Data: 2022-09-12T19:06:53.364 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.364 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.365 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.366 DEBUG:boto:Params: {} 2022-09-12T19:06:53.367 DEBUG:boto:Token: None 2022-09-12T19:06:53.369 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj 2022-09-12T19:06:53.370 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9hUIlNVqxQQwnBvp1XB6gqWEUYo= 2022-09-12T19:06:53.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9hUIlNVqxQQwnBvp1XB6gqWEUYo=', 'Content-Length': '0'} 2022-09-12T19:06:53.375 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b34b1ce6c38354df-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.376 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.377 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.378 DEBUG:boto:Method: HEAD 2022-09-12T19:06:53.380 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:53.381 DEBUG:boto:Data: 2022-09-12T19:06:53.382 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.383 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.384 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.385 DEBUG:boto:Params: {} 2022-09-12T19:06:53.386 DEBUG:boto:Token: None 2022-09-12T19:06:53.387 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj 2022-09-12T19:06:53.388 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9hUIlNVqxQQwnBvp1XB6gqWEUYo= 2022-09-12T19:06:53.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9hUIlNVqxQQwnBvp1XB6gqWEUYo=', 'Content-Length': '0'} 2022-09-12T19:06:53.394 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae4b99ba91879f61-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:53.396 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.397 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.398 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.399 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.400 DEBUG:boto:Method: GET 2022-09-12T19:06:53.401 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.401 DEBUG:boto:Data: 2022-09-12T19:06:53.402 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.402 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.403 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.404 DEBUG:boto:Params: {} 2022-09-12T19:06:53.404 DEBUG:boto:Token: None 2022-09-12T19:06:53.405 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.405 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JYEx2EaWUyI8tGvpT64xNVdxD/g= 2022-09-12T19:06:53.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JYEx2EaWUyI8tGvpT64xNVdxD/g=', 'Content-Length': '0'} 2022-09-12T19:06:53.408 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003bbd939fc7584cd3-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.450 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.451 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.451 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.452 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.452 DEBUG:boto:Method: GET 2022-09-12T19:06:53.453 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.453 DEBUG:boto:Data: 2022-09-12T19:06:53.454 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.454 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.455 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.456 DEBUG:boto:Params: {} 2022-09-12T19:06:53.456 DEBUG:boto:Token: None 2022-09-12T19:06:53.457 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.457 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JYEx2EaWUyI8tGvpT64xNVdxD/g= 2022-09-12T19:06:53.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JYEx2EaWUyI8tGvpT64xNVdxD/g=', 'Content-Length': '0'} 2022-09-12T19:06:53.460 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba847f1b375a9cf5-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.461 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:53.462 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:53.462 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.463 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.463 DEBUG:boto:Method: GET 2022-09-12T19:06:53.464 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:53.465 DEBUG:boto:Data: 2022-09-12T19:06:53.465 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.466 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.466 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.467 DEBUG:boto:Params: {} 2022-09-12T19:06:53.467 DEBUG:boto:Token: None 2022-09-12T19:06:53.468 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/ 2022-09-12T19:06:53.469 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8= 2022-09-12T19:06:53.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8=', 'Content-Length': '0'} 2022-09-12T19:06:53.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da9631138a8a9ad6-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.473 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:53.474 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:53.474 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:53.475 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.475 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.476 DEBUG:boto:Method: GET 2022-09-12T19:06:53.477 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:53.477 DEBUG:boto:Data: 2022-09-12T19:06:53.478 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.478 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.479 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.479 DEBUG:boto:Params: {} 2022-09-12T19:06:53.480 DEBUG:boto:Token: None 2022-09-12T19:06:53.480 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/ 2022-09-12T19:06:53.481 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8= 2022-09-12T19:06:53.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8=', 'Content-Length': '0'} 2022-09-12T19:06:53.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000642b42b597a5d7f4-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.485 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:53.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:53.487 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.487 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.488 DEBUG:boto:path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:53.488 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:53.489 DEBUG:boto:Method: GET 2022-09-12T19:06:53.489 DEBUG:boto:Path: /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:53.490 DEBUG:boto:Data: 2022-09-12T19:06:53.490 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.491 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.492 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.492 DEBUG:boto:Params: {} 2022-09-12T19:06:53.493 DEBUG:boto:Token: None 2022-09-12T19:06:53.493 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:53.494 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4khYd9GexRU6x2T8WRTmy1/tyGY= 2022-09-12T19:06:53.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4khYd9GexRU6x2T8WRTmy1/tyGY=', 'Content-Length': '0'} 2022-09-12T19:06:53.497 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d78cd0e25d9bb68f-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.498 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.498 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.499 DEBUG:boto:path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:53.499 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:53.500 DEBUG:boto:Method: GET 2022-09-12T19:06:53.500 DEBUG:boto:Path: /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:53.501 DEBUG:boto:Data: 2022-09-12T19:06:53.501 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.502 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.502 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.503 DEBUG:boto:Params: {} 2022-09-12T19:06:53.503 DEBUG:boto:Token: None 2022-09-12T19:06:53.504 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:53.505 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4khYd9GexRU6x2T8WRTmy1/tyGY= 2022-09-12T19:06:53.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4khYd9GexRU6x2T8WRTmy1/tyGY=', 'Content-Length': '0'} 2022-09-12T19:06:53.508 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045c946042b039755-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.508 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.509 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.509 DEBUG:boto:path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:53.510 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:53.510 DEBUG:boto:Method: HEAD 2022-09-12T19:06:53.511 DEBUG:boto:Path: /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:53.511 DEBUG:boto:Data: 2022-09-12T19:06:53.512 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.512 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.513 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.513 DEBUG:boto:Params: {} 2022-09-12T19:06:53.514 DEBUG:boto:Token: None 2022-09-12T19:06:53.515 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:53.515 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xZp63lUioCVyQ+qDyCYYLw2O4v8= 2022-09-12T19:06:53.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xZp63lUioCVyQ+qDyCYYLw2O4v8=', 'Content-Length': '0'} 2022-09-12T19:06:53.518 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009114f58c29b7fed3-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.519 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.519 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.520 DEBUG:boto:path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:53.520 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:53.521 DEBUG:boto:Method: HEAD 2022-09-12T19:06:53.521 DEBUG:boto:Path: /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:53.522 DEBUG:boto:Data: 2022-09-12T19:06:53.522 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.523 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.523 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.524 DEBUG:boto:Params: {} 2022-09-12T19:06:53.524 DEBUG:boto:Token: None 2022-09-12T19:06:53.525 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:53.526 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xZp63lUioCVyQ+qDyCYYLw2O4v8= 2022-09-12T19:06:53.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xZp63lUioCVyQ+qDyCYYLw2O4v8=', 'Content-Length': '0'} 2022-09-12T19:06:53.528 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000acb9b83576f55e53-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:53.530 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.531 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.531 DEBUG:boto:path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:53.532 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:53.532 DEBUG:boto:Method: GET 2022-09-12T19:06:53.533 DEBUG:boto:Path: /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:53.533 DEBUG:boto:Data: 2022-09-12T19:06:53.534 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.535 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.536 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.537 DEBUG:boto:Params: {} 2022-09-12T19:06:53.537 DEBUG:boto:Token: None 2022-09-12T19:06:53.538 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:53.538 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4khYd9GexRU6x2T8WRTmy1/tyGY= 2022-09-12T19:06:53.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4khYd9GexRU6x2T8WRTmy1/tyGY=', 'Content-Length': '0'} 2022-09-12T19:06:53.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002369bd3d58679240-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.542 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.542 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.543 DEBUG:boto:path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:53.544 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:53.544 DEBUG:boto:Method: GET 2022-09-12T19:06:53.545 DEBUG:boto:Path: /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:53.545 DEBUG:boto:Data: 2022-09-12T19:06:53.546 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.546 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.547 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.547 DEBUG:boto:Params: {} 2022-09-12T19:06:53.548 DEBUG:boto:Token: None 2022-09-12T19:06:53.549 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:06:53.549 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4khYd9GexRU6x2T8WRTmy1/tyGY= 2022-09-12T19:06:53.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4khYd9GexRU6x2T8WRTmy1/tyGY=', 'Content-Length': '0'} 2022-09-12T19:06:53.552 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f774ecfa5d147fdf-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.553 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:53.553 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:53.554 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.554 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.555 DEBUG:boto:Method: GET 2022-09-12T19:06:53.555 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:53.556 DEBUG:boto:Data: 2022-09-12T19:06:53.556 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.557 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.557 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.558 DEBUG:boto:Params: {} 2022-09-12T19:06:53.559 DEBUG:boto:Token: None 2022-09-12T19:06:53.559 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/ 2022-09-12T19:06:53.560 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8= 2022-09-12T19:06:53.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8=', 'Content-Length': '0'} 2022-09-12T19:06:53.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039bf59c79c420a58-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.565 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:53.566 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:53.567 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:53.567 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.568 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.568 DEBUG:boto:Method: GET 2022-09-12T19:06:53.569 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:53.569 DEBUG:boto:Data: 2022-09-12T19:06:53.570 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.571 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.571 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.572 DEBUG:boto:Params: {} 2022-09-12T19:06:53.572 DEBUG:boto:Token: None 2022-09-12T19:06:53.573 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/ 2022-09-12T19:06:53.574 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8= 2022-09-12T19:06:53.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8=', 'Content-Length': '0'} 2022-09-12T19:06:53.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002951ea30dc8886a-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.578 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:53.579 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.580 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.580 DEBUG:boto:Method: HEAD 2022-09-12T19:06:53.581 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:53.581 DEBUG:boto:Data: 2022-09-12T19:06:53.582 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.582 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.583 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.583 DEBUG:boto:Params: {} 2022-09-12T19:06:53.584 DEBUG:boto:Token: None 2022-09-12T19:06:53.584 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj 2022-09-12T19:06:53.585 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9hUIlNVqxQQwnBvp1XB6gqWEUYo= 2022-09-12T19:06:53.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9hUIlNVqxQQwnBvp1XB6gqWEUYo=', 'Content-Length': '0'} 2022-09-12T19:06:53.589 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e5ddae54e2517a3-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.590 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.590 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.591 DEBUG:boto:Method: HEAD 2022-09-12T19:06:53.592 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:53.592 DEBUG:boto:Data: 2022-09-12T19:06:53.593 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.593 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.594 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.594 DEBUG:boto:Params: {} 2022-09-12T19:06:53.595 DEBUG:boto:Token: None 2022-09-12T19:06:53.596 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj 2022-09-12T19:06:53.596 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9hUIlNVqxQQwnBvp1XB6gqWEUYo= 2022-09-12T19:06:53.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9hUIlNVqxQQwnBvp1XB6gqWEUYo=', 'Content-Length': '0'} 2022-09-12T19:06:53.601 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000226ae8d1280ae603-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:53.603 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.604 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.605 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.605 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.606 DEBUG:boto:Method: GET 2022-09-12T19:06:53.606 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.607 DEBUG:boto:Data: 2022-09-12T19:06:53.607 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.608 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.609 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.609 DEBUG:boto:Params: {} 2022-09-12T19:06:53.610 DEBUG:boto:Token: None 2022-09-12T19:06:53.611 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.611 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JYEx2EaWUyI8tGvpT64xNVdxD/g= 2022-09-12T19:06:53.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JYEx2EaWUyI8tGvpT64xNVdxD/g=', 'Content-Length': '0'} 2022-09-12T19:06:53.614 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000502aeeb824928203-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.615 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.615 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.616 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.617 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.617 DEBUG:boto:Method: GET 2022-09-12T19:06:53.618 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.618 DEBUG:boto:Data: 2022-09-12T19:06:53.619 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.619 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.620 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.621 DEBUG:boto:Params: {} 2022-09-12T19:06:53.621 DEBUG:boto:Token: None 2022-09-12T19:06:53.622 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.622 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JYEx2EaWUyI8tGvpT64xNVdxD/g= 2022-09-12T19:06:53.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JYEx2EaWUyI8tGvpT64xNVdxD/g=', 'Content-Length': '0'} 2022-09-12T19:06:53.625 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001517ac54b607a6db-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.666 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:53.667 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:53.668 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.668 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.669 DEBUG:boto:Method: GET 2022-09-12T19:06:53.669 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:53.670 DEBUG:boto:Data: 2022-09-12T19:06:53.670 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.671 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.671 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.672 DEBUG:boto:Params: {} 2022-09-12T19:06:53.672 DEBUG:boto:Token: None 2022-09-12T19:06:53.673 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/ 2022-09-12T19:06:53.674 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8= 2022-09-12T19:06:53.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8=', 'Content-Length': '0'} 2022-09-12T19:06:53.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000140b0f3e73ffb9c6-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.678 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:53.679 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:53.681 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:53.682 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.682 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.683 DEBUG:boto:Method: GET 2022-09-12T19:06:53.684 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:53.684 DEBUG:boto:Data: 2022-09-12T19:06:53.685 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.686 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.686 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.687 DEBUG:boto:Params: {} 2022-09-12T19:06:53.688 DEBUG:boto:Token: None 2022-09-12T19:06:53.688 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/ 2022-09-12T19:06:53.689 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8= 2022-09-12T19:06:53.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8=', 'Content-Length': '0'} 2022-09-12T19:06:53.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003db3e0dce211e01-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.693 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:53.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:53.695 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.695 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.696 DEBUG:boto:path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:53.697 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:53.697 DEBUG:boto:Method: GET 2022-09-12T19:06:53.698 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:53.698 DEBUG:boto:Data: 2022-09-12T19:06:53.699 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.699 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.700 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.700 DEBUG:boto:Params: {} 2022-09-12T19:06:53.701 DEBUG:boto:Token: None 2022-09-12T19:06:53.702 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:53.703 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P2urftDE41/UC3szRZOw/73lJXE= 2022-09-12T19:06:53.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P2urftDE41/UC3szRZOw/73lJXE=', 'Content-Length': '0'} 2022-09-12T19:06:53.707 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'T.yM--FII-PB1GReM85mkNRGYz6Dkbz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6129634c43fa455-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.709 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.710 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.710 DEBUG:boto:path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:53.711 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:53.712 DEBUG:boto:Method: GET 2022-09-12T19:06:53.713 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:53.714 DEBUG:boto:Data: 2022-09-12T19:06:53.715 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.716 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.717 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.718 DEBUG:boto:Params: {} 2022-09-12T19:06:53.719 DEBUG:boto:Token: None 2022-09-12T19:06:53.720 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:53.721 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P2urftDE41/UC3szRZOw/73lJXE= 2022-09-12T19:06:53.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P2urftDE41/UC3szRZOw/73lJXE=', 'Content-Length': '0'} 2022-09-12T19:06:53.724 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'T.yM--FII-PB1GReM85mkNRGYz6Dkbz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4135d73966e777f-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.726 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.727 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.727 DEBUG:boto:path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:53.728 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:53.728 DEBUG:boto:Method: HEAD 2022-09-12T19:06:53.729 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:53.729 DEBUG:boto:Data: 2022-09-12T19:06:53.730 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.730 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.731 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.731 DEBUG:boto:Params: {} 2022-09-12T19:06:53.732 DEBUG:boto:Token: None 2022-09-12T19:06:53.733 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:53.733 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lUZtC3GXiUVypE3TRWIY0NX0GBw= 2022-09-12T19:06:53.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lUZtC3GXiUVypE3TRWIY0NX0GBw=', 'Content-Length': '0'} 2022-09-12T19:06:53.736 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'T.yM--FII-PB1GReM85mkNRGYz6Dkbz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000acc815b554f4e562-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.737 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.737 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.738 DEBUG:boto:path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:53.738 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:53.739 DEBUG:boto:Method: HEAD 2022-09-12T19:06:53.739 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:53.740 DEBUG:boto:Data: 2022-09-12T19:06:53.740 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.741 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.741 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.742 DEBUG:boto:Params: {} 2022-09-12T19:06:53.743 DEBUG:boto:Token: None 2022-09-12T19:06:53.743 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:53.744 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lUZtC3GXiUVypE3TRWIY0NX0GBw= 2022-09-12T19:06:53.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lUZtC3GXiUVypE3TRWIY0NX0GBw=', 'Content-Length': '0'} 2022-09-12T19:06:53.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'T.yM--FII-PB1GReM85mkNRGYz6Dkbz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5616317a3de621e-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:53.748 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.748 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.749 DEBUG:boto:path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:53.749 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:53.750 DEBUG:boto:Method: GET 2022-09-12T19:06:53.751 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:53.751 DEBUG:boto:Data: 2022-09-12T19:06:53.752 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.752 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.753 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.753 DEBUG:boto:Params: {} 2022-09-12T19:06:53.754 DEBUG:boto:Token: None 2022-09-12T19:06:53.755 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:53.755 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P2urftDE41/UC3szRZOw/73lJXE= 2022-09-12T19:06:53.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P2urftDE41/UC3szRZOw/73lJXE=', 'Content-Length': '0'} 2022-09-12T19:06:53.758 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'T.yM--FII-PB1GReM85mkNRGYz6Dkbz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000022f59785c788893-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.759 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.760 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.760 DEBUG:boto:path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:53.761 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:53.761 DEBUG:boto:Method: GET 2022-09-12T19:06:53.762 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:53.762 DEBUG:boto:Data: 2022-09-12T19:06:53.763 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.763 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.764 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.765 DEBUG:boto:Params: {} 2022-09-12T19:06:53.766 DEBUG:boto:Token: None 2022-09-12T19:06:53.766 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:06:53.767 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P2urftDE41/UC3szRZOw/73lJXE= 2022-09-12T19:06:53.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P2urftDE41/UC3szRZOw/73lJXE=', 'Content-Length': '0'} 2022-09-12T19:06:53.770 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'T.yM--FII-PB1GReM85mkNRGYz6Dkbz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000585b47f8c0389a34-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.771 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:53.771 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:53.772 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.773 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.773 DEBUG:boto:Method: GET 2022-09-12T19:06:53.774 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:53.774 DEBUG:boto:Data: 2022-09-12T19:06:53.775 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.775 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.776 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.777 DEBUG:boto:Params: {} 2022-09-12T19:06:53.778 DEBUG:boto:Token: None 2022-09-12T19:06:53.778 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/ 2022-09-12T19:06:53.779 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8= 2022-09-12T19:06:53.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8=', 'Content-Length': '0'} 2022-09-12T19:06:53.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b30c58b71302c6c-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.784 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:53.785 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:53.785 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:53.786 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.787 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.788 DEBUG:boto:Method: GET 2022-09-12T19:06:53.788 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:53.789 DEBUG:boto:Data: 2022-09-12T19:06:53.789 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.790 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.790 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.791 DEBUG:boto:Params: {} 2022-09-12T19:06:53.791 DEBUG:boto:Token: None 2022-09-12T19:06:53.792 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/ 2022-09-12T19:06:53.793 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8= 2022-09-12T19:06:53.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8=', 'Content-Length': '0'} 2022-09-12T19:06:53.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e663c801283411be-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.838 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:53.840 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.841 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.842 DEBUG:boto:Method: HEAD 2022-09-12T19:06:53.843 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:53.844 DEBUG:boto:Data: 2022-09-12T19:06:53.845 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.846 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.847 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.848 DEBUG:boto:Params: {} 2022-09-12T19:06:53.849 DEBUG:boto:Token: None 2022-09-12T19:06:53.850 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj 2022-09-12T19:06:53.851 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9hUIlNVqxQQwnBvp1XB6gqWEUYo= 2022-09-12T19:06:53.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9hUIlNVqxQQwnBvp1XB6gqWEUYo=', 'Content-Length': '0'} 2022-09-12T19:06:53.857 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029718b1c150237e6-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.858 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.859 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.860 DEBUG:boto:Method: HEAD 2022-09-12T19:06:53.861 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:53.862 DEBUG:boto:Data: 2022-09-12T19:06:53.863 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.864 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.865 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.866 DEBUG:boto:Params: {} 2022-09-12T19:06:53.867 DEBUG:boto:Token: None 2022-09-12T19:06:53.868 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj 2022-09-12T19:06:53.869 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9hUIlNVqxQQwnBvp1XB6gqWEUYo= 2022-09-12T19:06:53.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9hUIlNVqxQQwnBvp1XB6gqWEUYo=', 'Content-Length': '0'} 2022-09-12T19:06:53.874 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b8287d99f40e56b-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:53.877 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.878 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.879 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.880 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.881 DEBUG:boto:Method: GET 2022-09-12T19:06:53.882 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.883 DEBUG:boto:Data: 2022-09-12T19:06:53.884 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.885 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.886 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.887 DEBUG:boto:Params: {} 2022-09-12T19:06:53.888 DEBUG:boto:Token: None 2022-09-12T19:06:53.889 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.890 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JYEx2EaWUyI8tGvpT64xNVdxD/g= 2022-09-12T19:06:53.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JYEx2EaWUyI8tGvpT64xNVdxD/g=', 'Content-Length': '0'} 2022-09-12T19:06:53.894 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3e6e76d292e723a-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.938 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.939 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.940 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.941 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.942 DEBUG:boto:Method: GET 2022-09-12T19:06:53.943 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.944 DEBUG:boto:Data: 2022-09-12T19:06:53.945 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.946 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.947 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.949 DEBUG:boto:Params: {} 2022-09-12T19:06:53.950 DEBUG:boto:Token: None 2022-09-12T19:06:53.950 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:53.951 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JYEx2EaWUyI8tGvpT64xNVdxD/g= 2022-09-12T19:06:53.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JYEx2EaWUyI8tGvpT64xNVdxD/g=', 'Content-Length': '0'} 2022-09-12T19:06:53.954 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c8e368106fffe4d1-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.955 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:53.956 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:53.956 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.957 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.957 DEBUG:boto:Method: GET 2022-09-12T19:06:53.958 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:53.958 DEBUG:boto:Data: 2022-09-12T19:06:53.959 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.960 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.960 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.961 DEBUG:boto:Params: {} 2022-09-12T19:06:53.961 DEBUG:boto:Token: None 2022-09-12T19:06:53.962 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/ 2022-09-12T19:06:53.962 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8= 2022-09-12T19:06:53.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8=', 'Content-Length': '0'} 2022-09-12T19:06:53.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b88940a693c4ae1a-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.967 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:53.968 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:53.968 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:53.969 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.969 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:53.970 DEBUG:boto:Method: GET 2022-09-12T19:06:53.970 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:53.971 DEBUG:boto:Data: 2022-09-12T19:06:53.971 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.972 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.972 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.973 DEBUG:boto:Params: {} 2022-09-12T19:06:53.973 DEBUG:boto:Token: None 2022-09-12T19:06:53.974 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/ 2022-09-12T19:06:53.975 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8= 2022-09-12T19:06:53.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kaXjhM1ZO2L3lop4yOBnwXg0rd8=', 'Content-Length': '0'} 2022-09-12T19:06:53.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043019cf1b1c83da2-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.979 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:53.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:53.980 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.981 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.981 DEBUG:boto:path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:53.982 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:53.982 DEBUG:boto:Method: GET 2022-09-12T19:06:53.983 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:53.983 DEBUG:boto:Data: 2022-09-12T19:06:53.984 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.984 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:53.985 DEBUG:boto:Port: 8001 2022-09-12T19:06:53.985 DEBUG:boto:Params: {} 2022-09-12T19:06:53.986 DEBUG:boto:Token: None 2022-09-12T19:06:53.987 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:53.987 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QfHqeBJxmsfWejmFEP3LRGQCz2w= 2022-09-12T19:06:53.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QfHqeBJxmsfWejmFEP3LRGQCz2w=', 'Content-Length': '0'} 2022-09-12T19:06:53.990 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Xb5UZAvutXF6GhBCMdUQgjOLi998.hK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a09f3c23e9d6a7e-00631f834d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:53.991 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:53.991 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:53.992 DEBUG:boto:path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:53.992 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:53.993 DEBUG:boto:Method: GET 2022-09-12T19:06:53.993 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:53.994 DEBUG:boto:Data: 2022-09-12T19:06:53.995 DEBUG:boto:Headers: {} 2022-09-12T19:06:53.995 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:53.996 DEBUG:boto:Port: 8000 2022-09-12T19:06:53.996 DEBUG:boto:Params: {} 2022-09-12T19:06:53.997 DEBUG:boto:Token: None 2022-09-12T19:06:53.997 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:53 GMT /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:53.998 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QfHqeBJxmsfWejmFEP3LRGQCz2w= 2022-09-12T19:06:53.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QfHqeBJxmsfWejmFEP3LRGQCz2w=', 'Content-Length': '0'} 2022-09-12T19:06:54.001 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Xb5UZAvutXF6GhBCMdUQgjOLi998.hK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000481d5d07df190b3b-00631f834d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.042 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.043 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.043 DEBUG:boto:path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:54.044 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:54.044 DEBUG:boto:Method: HEAD 2022-09-12T19:06:54.045 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:54.045 DEBUG:boto:Data: 2022-09-12T19:06:54.046 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.046 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.047 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.047 DEBUG:boto:Params: {} 2022-09-12T19:06:54.048 DEBUG:boto:Token: None 2022-09-12T19:06:54.048 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:54.049 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:r6z/n+QCf4T220wVOugFQ3xy85w= 2022-09-12T19:06:54.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:r6z/n+QCf4T220wVOugFQ3xy85w=', 'Content-Length': '0'} 2022-09-12T19:06:54.052 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Xb5UZAvutXF6GhBCMdUQgjOLi998.hK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b80bbc2f34cc5e1-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.053 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.053 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.054 DEBUG:boto:path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:54.054 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:54.055 DEBUG:boto:Method: HEAD 2022-09-12T19:06:54.055 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:54.056 DEBUG:boto:Data: 2022-09-12T19:06:54.056 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.057 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.057 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.059 DEBUG:boto:Params: {} 2022-09-12T19:06:54.059 DEBUG:boto:Token: None 2022-09-12T19:06:54.060 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:54.060 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:r6z/n+QCf4T220wVOugFQ3xy85w= 2022-09-12T19:06:54.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:r6z/n+QCf4T220wVOugFQ3xy85w=', 'Content-Length': '0'} 2022-09-12T19:06:54.063 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Xb5UZAvutXF6GhBCMdUQgjOLi998.hK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e182b5031282e4b-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:54.065 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.065 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.066 DEBUG:boto:path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:54.067 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:54.068 DEBUG:boto:Method: GET 2022-09-12T19:06:54.069 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:54.070 DEBUG:boto:Data: 2022-09-12T19:06:54.070 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.071 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.071 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.072 DEBUG:boto:Params: {} 2022-09-12T19:06:54.072 DEBUG:boto:Token: None 2022-09-12T19:06:54.073 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:54.074 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E4nM1jpGSBu32BoB2+SmbahaBzk= 2022-09-12T19:06:54.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E4nM1jpGSBu32BoB2+SmbahaBzk=', 'Content-Length': '0'} 2022-09-12T19:06:54.077 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Xb5UZAvutXF6GhBCMdUQgjOLi998.hK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c48f3cfa1db5fcb-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.078 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.087 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.088 DEBUG:boto:path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:54.089 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:54.090 DEBUG:boto:Method: GET 2022-09-12T19:06:54.091 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:54.092 DEBUG:boto:Data: 2022-09-12T19:06:54.093 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.094 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.095 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.096 DEBUG:boto:Params: {} 2022-09-12T19:06:54.097 DEBUG:boto:Token: None 2022-09-12T19:06:54.098 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:06:54.099 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E4nM1jpGSBu32BoB2+SmbahaBzk= 2022-09-12T19:06:54.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E4nM1jpGSBu32BoB2+SmbahaBzk=', 'Content-Length': '0'} 2022-09-12T19:06:54.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Xb5UZAvutXF6GhBCMdUQgjOLi998.hK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000691c979303d5bd8d-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.103 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:54.103 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:54.104 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.104 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.105 DEBUG:boto:Method: GET 2022-09-12T19:06:54.106 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:54.107 DEBUG:boto:Data: 2022-09-12T19:06:54.108 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.109 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.110 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.111 DEBUG:boto:Params: {} 2022-09-12T19:06:54.112 DEBUG:boto:Token: None 2022-09-12T19:06:54.113 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/ 2022-09-12T19:06:54.114 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo= 2022-09-12T19:06:54.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo=', 'Content-Length': '0'} 2022-09-12T19:06:54.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b6427f60b68c9a1-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.119 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:54.121 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:54.122 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:54.123 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.124 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.125 DEBUG:boto:Method: GET 2022-09-12T19:06:54.126 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:54.127 DEBUG:boto:Data: 2022-09-12T19:06:54.128 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.128 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.129 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.130 DEBUG:boto:Params: {} 2022-09-12T19:06:54.131 DEBUG:boto:Token: None 2022-09-12T19:06:54.131 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/ 2022-09-12T19:06:54.132 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo= 2022-09-12T19:06:54.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo=', 'Content-Length': '0'} 2022-09-12T19:06:54.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016631f13318e4ec0-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.136 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:54.137 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.137 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.138 DEBUG:boto:Method: HEAD 2022-09-12T19:06:54.138 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:54.139 DEBUG:boto:Data: 2022-09-12T19:06:54.139 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.140 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.140 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.141 DEBUG:boto:Params: {} 2022-09-12T19:06:54.142 DEBUG:boto:Token: None 2022-09-12T19:06:54.142 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj 2022-09-12T19:06:54.143 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LRoEHf7DrF1jVmt+ZC7Xc/G/050= 2022-09-12T19:06:54.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LRoEHf7DrF1jVmt+ZC7Xc/G/050=', 'Content-Length': '0'} 2022-09-12T19:06:54.149 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b75db427f44954d6-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.150 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.150 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.151 DEBUG:boto:Method: HEAD 2022-09-12T19:06:54.151 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:54.152 DEBUG:boto:Data: 2022-09-12T19:06:54.152 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.153 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.153 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.154 DEBUG:boto:Params: {} 2022-09-12T19:06:54.155 DEBUG:boto:Token: None 2022-09-12T19:06:54.155 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj 2022-09-12T19:06:54.156 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LRoEHf7DrF1jVmt+ZC7Xc/G/050= 2022-09-12T19:06:54.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LRoEHf7DrF1jVmt+ZC7Xc/G/050=', 'Content-Length': '0'} 2022-09-12T19:06:54.161 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000613946c2e9c5a7b0-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:54.163 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.163 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.164 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.164 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.165 DEBUG:boto:Method: GET 2022-09-12T19:06:54.166 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.166 DEBUG:boto:Data: 2022-09-12T19:06:54.167 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.167 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.168 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.169 DEBUG:boto:Params: {} 2022-09-12T19:06:54.169 DEBUG:boto:Token: None 2022-09-12T19:06:54.170 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.171 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f7KicsRggX2GVofXMr3QxGKS8N4= 2022-09-12T19:06:54.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f7KicsRggX2GVofXMr3QxGKS8N4=', 'Content-Length': '0'} 2022-09-12T19:06:54.174 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001cedb8cc82486fef-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.218 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.219 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.219 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.220 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.220 DEBUG:boto:Method: GET 2022-09-12T19:06:54.221 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.221 DEBUG:boto:Data: 2022-09-12T19:06:54.222 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.223 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.223 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.224 DEBUG:boto:Params: {} 2022-09-12T19:06:54.224 DEBUG:boto:Token: None 2022-09-12T19:06:54.225 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.225 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f7KicsRggX2GVofXMr3QxGKS8N4= 2022-09-12T19:06:54.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f7KicsRggX2GVofXMr3QxGKS8N4=', 'Content-Length': '0'} 2022-09-12T19:06:54.229 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5715953f37c9595-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.229 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:54.230 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:54.231 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.231 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.232 DEBUG:boto:Method: GET 2022-09-12T19:06:54.232 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:54.233 DEBUG:boto:Data: 2022-09-12T19:06:54.233 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.234 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.234 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.235 DEBUG:boto:Params: {} 2022-09-12T19:06:54.235 DEBUG:boto:Token: None 2022-09-12T19:06:54.236 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/ 2022-09-12T19:06:54.236 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo= 2022-09-12T19:06:54.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo=', 'Content-Length': '0'} 2022-09-12T19:06:54.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e09ebfc0aa06a7ae-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.241 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:54.241 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:54.242 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:54.243 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.243 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.244 DEBUG:boto:Method: GET 2022-09-12T19:06:54.244 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:54.245 DEBUG:boto:Data: 2022-09-12T19:06:54.245 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.246 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.246 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.247 DEBUG:boto:Params: {} 2022-09-12T19:06:54.247 DEBUG:boto:Token: None 2022-09-12T19:06:54.248 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/ 2022-09-12T19:06:54.249 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo= 2022-09-12T19:06:54.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo=', 'Content-Length': '0'} 2022-09-12T19:06:54.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b1919776d4a42f5-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.253 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:54.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:54.254 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.255 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.255 DEBUG:boto:path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:54.256 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:54.256 DEBUG:boto:Method: GET 2022-09-12T19:06:54.257 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:54.257 DEBUG:boto:Data: 2022-09-12T19:06:54.258 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.259 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.259 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.260 DEBUG:boto:Params: {} 2022-09-12T19:06:54.260 DEBUG:boto:Token: None 2022-09-12T19:06:54.261 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:54.261 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TuK74mT8BPn3ROZXUqFNkR0R0N8= 2022-09-12T19:06:54.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TuK74mT8BPn3ROZXUqFNkR0R0N8=', 'Content-Length': '0'} 2022-09-12T19:06:54.264 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'ztQRphsNbUdApzTcOcagUJ56uTwo7J2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3743c023a521a05-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.265 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.265 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.266 DEBUG:boto:path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:54.267 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:54.267 DEBUG:boto:Method: GET 2022-09-12T19:06:54.268 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:54.268 DEBUG:boto:Data: 2022-09-12T19:06:54.269 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.269 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.270 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.270 DEBUG:boto:Params: {} 2022-09-12T19:06:54.271 DEBUG:boto:Token: None 2022-09-12T19:06:54.272 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:54.272 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TuK74mT8BPn3ROZXUqFNkR0R0N8= 2022-09-12T19:06:54.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TuK74mT8BPn3ROZXUqFNkR0R0N8=', 'Content-Length': '0'} 2022-09-12T19:06:54.275 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'ztQRphsNbUdApzTcOcagUJ56uTwo7J2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a9e129eb5392c53-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.276 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.276 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.277 DEBUG:boto:path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:54.277 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:54.278 DEBUG:boto:Method: HEAD 2022-09-12T19:06:54.278 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:54.279 DEBUG:boto:Data: 2022-09-12T19:06:54.279 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.280 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.280 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.281 DEBUG:boto:Params: {} 2022-09-12T19:06:54.281 DEBUG:boto:Token: None 2022-09-12T19:06:54.282 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:54.282 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p4o2oHKUTJx/KW9jNT8Z7HSHurU= 2022-09-12T19:06:54.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p4o2oHKUTJx/KW9jNT8Z7HSHurU=', 'Content-Length': '0'} 2022-09-12T19:06:54.285 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'ztQRphsNbUdApzTcOcagUJ56uTwo7J2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005dd845d54c41d712-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.286 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.286 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.287 DEBUG:boto:path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:54.287 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:54.288 DEBUG:boto:Method: HEAD 2022-09-12T19:06:54.288 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:54.289 DEBUG:boto:Data: 2022-09-12T19:06:54.289 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.290 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.290 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.291 DEBUG:boto:Params: {} 2022-09-12T19:06:54.291 DEBUG:boto:Token: None 2022-09-12T19:06:54.292 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:54.293 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p4o2oHKUTJx/KW9jNT8Z7HSHurU= 2022-09-12T19:06:54.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p4o2oHKUTJx/KW9jNT8Z7HSHurU=', 'Content-Length': '0'} 2022-09-12T19:06:54.295 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'ztQRphsNbUdApzTcOcagUJ56uTwo7J2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fdbcf2f908f77d61-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:54.297 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.297 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.298 DEBUG:boto:path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:54.298 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:54.299 DEBUG:boto:Method: GET 2022-09-12T19:06:54.299 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:54.300 DEBUG:boto:Data: 2022-09-12T19:06:54.300 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.301 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.301 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.302 DEBUG:boto:Params: {} 2022-09-12T19:06:54.302 DEBUG:boto:Token: None 2022-09-12T19:06:54.303 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:54.304 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TuK74mT8BPn3ROZXUqFNkR0R0N8= 2022-09-12T19:06:54.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TuK74mT8BPn3ROZXUqFNkR0R0N8=', 'Content-Length': '0'} 2022-09-12T19:06:54.307 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'ztQRphsNbUdApzTcOcagUJ56uTwo7J2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000214dbd49e855f933-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.307 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.308 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.309 DEBUG:boto:path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:54.309 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:54.310 DEBUG:boto:Method: GET 2022-09-12T19:06:54.310 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:54.311 DEBUG:boto:Data: 2022-09-12T19:06:54.311 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.312 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.312 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.313 DEBUG:boto:Params: {} 2022-09-12T19:06:54.313 DEBUG:boto:Token: None 2022-09-12T19:06:54.314 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:06:54.315 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TuK74mT8BPn3ROZXUqFNkR0R0N8= 2022-09-12T19:06:54.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TuK74mT8BPn3ROZXUqFNkR0R0N8=', 'Content-Length': '0'} 2022-09-12T19:06:54.317 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'ztQRphsNbUdApzTcOcagUJ56uTwo7J2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003725e279d3d8a121-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.318 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:54.319 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:54.319 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.320 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.320 DEBUG:boto:Method: GET 2022-09-12T19:06:54.321 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:54.321 DEBUG:boto:Data: 2022-09-12T19:06:54.322 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.322 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.323 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.324 DEBUG:boto:Params: {} 2022-09-12T19:06:54.324 DEBUG:boto:Token: None 2022-09-12T19:06:54.325 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/ 2022-09-12T19:06:54.325 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo= 2022-09-12T19:06:54.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo=', 'Content-Length': '0'} 2022-09-12T19:06:54.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da22cb8fdc3cf7dd-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.330 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:54.331 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:54.331 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:54.332 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.332 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.333 DEBUG:boto:Method: GET 2022-09-12T19:06:54.334 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:54.334 DEBUG:boto:Data: 2022-09-12T19:06:54.335 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.335 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.336 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.336 DEBUG:boto:Params: {} 2022-09-12T19:06:54.337 DEBUG:boto:Token: None 2022-09-12T19:06:54.337 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/ 2022-09-12T19:06:54.338 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo= 2022-09-12T19:06:54.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo=', 'Content-Length': '0'} 2022-09-12T19:06:54.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf961b1131409944-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.342 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:54.343 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.343 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.344 DEBUG:boto:Method: HEAD 2022-09-12T19:06:54.344 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:54.345 DEBUG:boto:Data: 2022-09-12T19:06:54.345 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.346 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.346 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.347 DEBUG:boto:Params: {} 2022-09-12T19:06:54.347 DEBUG:boto:Token: None 2022-09-12T19:06:54.348 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj 2022-09-12T19:06:54.349 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LRoEHf7DrF1jVmt+ZC7Xc/G/050= 2022-09-12T19:06:54.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LRoEHf7DrF1jVmt+ZC7Xc/G/050=', 'Content-Length': '0'} 2022-09-12T19:06:54.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ee8e390bbc56cae-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.354 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.354 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.355 DEBUG:boto:Method: HEAD 2022-09-12T19:06:54.355 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:54.356 DEBUG:boto:Data: 2022-09-12T19:06:54.356 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.357 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.357 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.358 DEBUG:boto:Params: {} 2022-09-12T19:06:54.359 DEBUG:boto:Token: None 2022-09-12T19:06:54.359 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj 2022-09-12T19:06:54.360 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LRoEHf7DrF1jVmt+ZC7Xc/G/050= 2022-09-12T19:06:54.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LRoEHf7DrF1jVmt+ZC7Xc/G/050=', 'Content-Length': '0'} 2022-09-12T19:06:54.364 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6c1b62f143c12f2-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:54.365 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.366 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.366 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.367 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.368 DEBUG:boto:Method: GET 2022-09-12T19:06:54.368 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.369 DEBUG:boto:Data: 2022-09-12T19:06:54.369 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.370 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.370 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.371 DEBUG:boto:Params: {} 2022-09-12T19:06:54.371 DEBUG:boto:Token: None 2022-09-12T19:06:54.372 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.373 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f7KicsRggX2GVofXMr3QxGKS8N4= 2022-09-12T19:06:54.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f7KicsRggX2GVofXMr3QxGKS8N4=', 'Content-Length': '0'} 2022-09-12T19:06:54.375 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a93f5b52582acf1-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.376 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.377 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.377 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.378 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.378 DEBUG:boto:Method: GET 2022-09-12T19:06:54.379 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.380 DEBUG:boto:Data: 2022-09-12T19:06:54.380 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.381 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.381 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.382 DEBUG:boto:Params: {} 2022-09-12T19:06:54.382 DEBUG:boto:Token: None 2022-09-12T19:06:54.383 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.384 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f7KicsRggX2GVofXMr3QxGKS8N4= 2022-09-12T19:06:54.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f7KicsRggX2GVofXMr3QxGKS8N4=', 'Content-Length': '0'} 2022-09-12T19:06:54.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000942dfa14dc456c9b-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.387 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:54.388 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:54.388 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.389 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.389 DEBUG:boto:Method: GET 2022-09-12T19:06:54.390 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:54.390 DEBUG:boto:Data: 2022-09-12T19:06:54.391 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.391 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.392 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.393 DEBUG:boto:Params: {} 2022-09-12T19:06:54.393 DEBUG:boto:Token: None 2022-09-12T19:06:54.394 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/ 2022-09-12T19:06:54.394 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo= 2022-09-12T19:06:54.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo=', 'Content-Length': '0'} 2022-09-12T19:06:54.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087e0572a1376070e-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.399 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:54.399 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:54.400 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:54.401 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.401 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.402 DEBUG:boto:Method: GET 2022-09-12T19:06:54.402 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:54.403 DEBUG:boto:Data: 2022-09-12T19:06:54.403 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.404 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.404 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.405 DEBUG:boto:Params: {} 2022-09-12T19:06:54.405 DEBUG:boto:Token: None 2022-09-12T19:06:54.406 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/ 2022-09-12T19:06:54.407 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo= 2022-09-12T19:06:54.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo=', 'Content-Length': '0'} 2022-09-12T19:06:54.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd179fe13fa76ad5-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.454 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:54.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:54.456 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.456 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.457 DEBUG:boto:path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:54.457 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:54.458 DEBUG:boto:Method: GET 2022-09-12T19:06:54.458 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:54.459 DEBUG:boto:Data: 2022-09-12T19:06:54.459 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.460 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.460 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.461 DEBUG:boto:Params: {} 2022-09-12T19:06:54.461 DEBUG:boto:Token: None 2022-09-12T19:06:54.462 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:54.463 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GI0fK2+qpoOVoOQfTbZ/eUKLhX0= 2022-09-12T19:06:54.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GI0fK2+qpoOVoOQfTbZ/eUKLhX0=', 'Content-Length': '0'} 2022-09-12T19:06:54.466 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Katr07t9mQV0b0gx44aB8d9VyeAhMS1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b4431b067245e32-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.466 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.467 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.467 DEBUG:boto:path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:54.468 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:54.469 DEBUG:boto:Method: GET 2022-09-12T19:06:54.469 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:54.470 DEBUG:boto:Data: 2022-09-12T19:06:54.470 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.471 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.471 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.472 DEBUG:boto:Params: {} 2022-09-12T19:06:54.472 DEBUG:boto:Token: None 2022-09-12T19:06:54.473 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:54.473 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GI0fK2+qpoOVoOQfTbZ/eUKLhX0= 2022-09-12T19:06:54.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GI0fK2+qpoOVoOQfTbZ/eUKLhX0=', 'Content-Length': '0'} 2022-09-12T19:06:54.476 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Katr07t9mQV0b0gx44aB8d9VyeAhMS1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022fd1ab221d374a3-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.477 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.478 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.478 DEBUG:boto:path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:54.479 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:54.479 DEBUG:boto:Method: HEAD 2022-09-12T19:06:54.480 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:54.480 DEBUG:boto:Data: 2022-09-12T19:06:54.481 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.481 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.482 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.482 DEBUG:boto:Params: {} 2022-09-12T19:06:54.483 DEBUG:boto:Token: None 2022-09-12T19:06:54.483 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:54.484 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CQ+fH696mZndaIBOPkX+C42wpq4= 2022-09-12T19:06:54.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CQ+fH696mZndaIBOPkX+C42wpq4=', 'Content-Length': '0'} 2022-09-12T19:06:54.486 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Katr07t9mQV0b0gx44aB8d9VyeAhMS1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a67ed7b5090eb14a-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.487 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.488 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.488 DEBUG:boto:path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:54.488 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:54.489 DEBUG:boto:Method: HEAD 2022-09-12T19:06:54.489 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:54.490 DEBUG:boto:Data: 2022-09-12T19:06:54.490 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.491 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.491 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.492 DEBUG:boto:Params: {} 2022-09-12T19:06:54.492 DEBUG:boto:Token: None 2022-09-12T19:06:54.493 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:54.494 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CQ+fH696mZndaIBOPkX+C42wpq4= 2022-09-12T19:06:54.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CQ+fH696mZndaIBOPkX+C42wpq4=', 'Content-Length': '0'} 2022-09-12T19:06:54.496 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Katr07t9mQV0b0gx44aB8d9VyeAhMS1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5194e58cc6bfe45-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:54.498 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.498 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.499 DEBUG:boto:path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:54.499 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:54.500 DEBUG:boto:Method: GET 2022-09-12T19:06:54.500 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:54.501 DEBUG:boto:Data: 2022-09-12T19:06:54.501 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.502 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.502 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.503 DEBUG:boto:Params: {} 2022-09-12T19:06:54.503 DEBUG:boto:Token: None 2022-09-12T19:06:54.504 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:54.505 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GI0fK2+qpoOVoOQfTbZ/eUKLhX0= 2022-09-12T19:06:54.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GI0fK2+qpoOVoOQfTbZ/eUKLhX0=', 'Content-Length': '0'} 2022-09-12T19:06:54.507 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Katr07t9mQV0b0gx44aB8d9VyeAhMS1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff7ff14d537a411d-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.550 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.551 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.551 DEBUG:boto:path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:54.552 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:54.552 DEBUG:boto:Method: GET 2022-09-12T19:06:54.553 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:54.553 DEBUG:boto:Data: 2022-09-12T19:06:54.554 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.554 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.555 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.556 DEBUG:boto:Params: {} 2022-09-12T19:06:54.556 DEBUG:boto:Token: None 2022-09-12T19:06:54.557 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:06:54.557 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GI0fK2+qpoOVoOQfTbZ/eUKLhX0= 2022-09-12T19:06:54.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GI0fK2+qpoOVoOQfTbZ/eUKLhX0=', 'Content-Length': '0'} 2022-09-12T19:06:54.560 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Katr07t9mQV0b0gx44aB8d9VyeAhMS1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000407705668a2c4bd9-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.561 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:54.562 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:54.562 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.563 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.563 DEBUG:boto:Method: GET 2022-09-12T19:06:54.564 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:54.564 DEBUG:boto:Data: 2022-09-12T19:06:54.565 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.565 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.566 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.566 DEBUG:boto:Params: {} 2022-09-12T19:06:54.567 DEBUG:boto:Token: None 2022-09-12T19:06:54.568 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/ 2022-09-12T19:06:54.568 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo= 2022-09-12T19:06:54.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo=', 'Content-Length': '0'} 2022-09-12T19:06:54.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038cf0b6e5eb39702-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.572 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:54.573 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:54.574 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:54.574 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.575 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.575 DEBUG:boto:Method: GET 2022-09-12T19:06:54.576 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:54.576 DEBUG:boto:Data: 2022-09-12T19:06:54.577 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.577 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.578 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.579 DEBUG:boto:Params: {} 2022-09-12T19:06:54.579 DEBUG:boto:Token: None 2022-09-12T19:06:54.580 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/ 2022-09-12T19:06:54.580 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo= 2022-09-12T19:06:54.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo=', 'Content-Length': '0'} 2022-09-12T19:06:54.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebef96ddcd78b8ab-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.585 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:54.586 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.586 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.587 DEBUG:boto:Method: HEAD 2022-09-12T19:06:54.587 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:54.588 DEBUG:boto:Data: 2022-09-12T19:06:54.588 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.589 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.589 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.590 DEBUG:boto:Params: {} 2022-09-12T19:06:54.590 DEBUG:boto:Token: None 2022-09-12T19:06:54.591 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj 2022-09-12T19:06:54.592 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LRoEHf7DrF1jVmt+ZC7Xc/G/050= 2022-09-12T19:06:54.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LRoEHf7DrF1jVmt+ZC7Xc/G/050=', 'Content-Length': '0'} 2022-09-12T19:06:54.596 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce03d9898571a20d-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.597 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.597 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.598 DEBUG:boto:Method: HEAD 2022-09-12T19:06:54.598 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:54.599 DEBUG:boto:Data: 2022-09-12T19:06:54.599 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.600 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.600 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.601 DEBUG:boto:Params: {} 2022-09-12T19:06:54.601 DEBUG:boto:Token: None 2022-09-12T19:06:54.602 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj 2022-09-12T19:06:54.603 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LRoEHf7DrF1jVmt+ZC7Xc/G/050= 2022-09-12T19:06:54.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LRoEHf7DrF1jVmt+ZC7Xc/G/050=', 'Content-Length': '0'} 2022-09-12T19:06:54.606 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c9001bb7283d1a5-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:54.608 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.608 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.609 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.609 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.610 DEBUG:boto:Method: GET 2022-09-12T19:06:54.610 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.611 DEBUG:boto:Data: 2022-09-12T19:06:54.611 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.612 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.612 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.613 DEBUG:boto:Params: {} 2022-09-12T19:06:54.613 DEBUG:boto:Token: None 2022-09-12T19:06:54.614 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.615 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f7KicsRggX2GVofXMr3QxGKS8N4= 2022-09-12T19:06:54.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f7KicsRggX2GVofXMr3QxGKS8N4=', 'Content-Length': '0'} 2022-09-12T19:06:54.617 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e508a82f229467a6-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.658 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.659 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.659 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.660 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.660 DEBUG:boto:Method: GET 2022-09-12T19:06:54.661 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.661 DEBUG:boto:Data: 2022-09-12T19:06:54.662 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.662 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.663 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.664 DEBUG:boto:Params: {} 2022-09-12T19:06:54.664 DEBUG:boto:Token: None 2022-09-12T19:06:54.665 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.665 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f7KicsRggX2GVofXMr3QxGKS8N4= 2022-09-12T19:06:54.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f7KicsRggX2GVofXMr3QxGKS8N4=', 'Content-Length': '0'} 2022-09-12T19:06:54.668 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078aa575a8bf253ac-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.669 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:54.670 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:54.670 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.671 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.671 DEBUG:boto:Method: GET 2022-09-12T19:06:54.672 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:54.672 DEBUG:boto:Data: 2022-09-12T19:06:54.673 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.673 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.674 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.674 DEBUG:boto:Params: {} 2022-09-12T19:06:54.675 DEBUG:boto:Token: None 2022-09-12T19:06:54.676 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/ 2022-09-12T19:06:54.676 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo= 2022-09-12T19:06:54.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo=', 'Content-Length': '0'} 2022-09-12T19:06:54.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7786291233fbbf1-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.681 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:54.681 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:54.682 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:54.683 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.683 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.684 DEBUG:boto:Method: GET 2022-09-12T19:06:54.684 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:54.685 DEBUG:boto:Data: 2022-09-12T19:06:54.685 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.686 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.686 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.687 DEBUG:boto:Params: {} 2022-09-12T19:06:54.687 DEBUG:boto:Token: None 2022-09-12T19:06:54.688 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/ 2022-09-12T19:06:54.689 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo= 2022-09-12T19:06:54.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo=', 'Content-Length': '0'} 2022-09-12T19:06:54.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a93cb2effb1f4428-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.693 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:54.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:54.709 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.709 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.710 DEBUG:boto:path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:54.710 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:54.711 DEBUG:boto:Method: GET 2022-09-12T19:06:54.711 DEBUG:boto:Path: /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:54.712 DEBUG:boto:Data: 2022-09-12T19:06:54.713 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.713 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.714 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.714 DEBUG:boto:Params: {} 2022-09-12T19:06:54.718 DEBUG:boto:Token: None 2022-09-12T19:06:54.719 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:54.719 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:++AwOAdDyifsUYLwo2pagwEsR+4= 2022-09-12T19:06:54.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:++AwOAdDyifsUYLwo2pagwEsR+4=', 'Content-Length': '0'} 2022-09-12T19:06:54.722 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'UBsYcCA94H55K4ucww5vXNAxuAMKNSG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070cd4af3a8b400d5-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.723 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.724 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.724 DEBUG:boto:path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:54.725 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:54.726 DEBUG:boto:Method: GET 2022-09-12T19:06:54.726 DEBUG:boto:Path: /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:54.727 DEBUG:boto:Data: 2022-09-12T19:06:54.727 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.728 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.728 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.729 DEBUG:boto:Params: {} 2022-09-12T19:06:54.729 DEBUG:boto:Token: None 2022-09-12T19:06:54.730 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:54.731 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:++AwOAdDyifsUYLwo2pagwEsR+4= 2022-09-12T19:06:54.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:++AwOAdDyifsUYLwo2pagwEsR+4=', 'Content-Length': '0'} 2022-09-12T19:06:54.733 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'UBsYcCA94H55K4ucww5vXNAxuAMKNSG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000826d2b2590a4faaa-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.734 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.735 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.735 DEBUG:boto:path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:54.736 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:54.736 DEBUG:boto:Method: HEAD 2022-09-12T19:06:54.737 DEBUG:boto:Path: /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:54.737 DEBUG:boto:Data: 2022-09-12T19:06:54.738 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.738 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.739 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.739 DEBUG:boto:Params: {} 2022-09-12T19:06:54.740 DEBUG:boto:Token: None 2022-09-12T19:06:54.741 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:54.741 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:swRPOd6ipMfD0aw6XPo18fYVjLo= 2022-09-12T19:06:54.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:swRPOd6ipMfD0aw6XPo18fYVjLo=', 'Content-Length': '0'} 2022-09-12T19:06:54.744 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'UBsYcCA94H55K4ucww5vXNAxuAMKNSG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081aedf1e78bb293b-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.745 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.745 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.746 DEBUG:boto:path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:54.746 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:54.747 DEBUG:boto:Method: HEAD 2022-09-12T19:06:54.747 DEBUG:boto:Path: /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:54.748 DEBUG:boto:Data: 2022-09-12T19:06:54.748 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.749 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.749 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.750 DEBUG:boto:Params: {} 2022-09-12T19:06:54.750 DEBUG:boto:Token: None 2022-09-12T19:06:54.751 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:54.752 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:swRPOd6ipMfD0aw6XPo18fYVjLo= 2022-09-12T19:06:54.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:swRPOd6ipMfD0aw6XPo18fYVjLo=', 'Content-Length': '0'} 2022-09-12T19:06:54.754 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'UBsYcCA94H55K4ucww5vXNAxuAMKNSG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca29f149e69bfd3f-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:54.756 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.756 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.757 DEBUG:boto:path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:54.757 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:54.758 DEBUG:boto:Method: GET 2022-09-12T19:06:54.758 DEBUG:boto:Path: /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:54.759 DEBUG:boto:Data: 2022-09-12T19:06:54.760 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.760 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.761 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.761 DEBUG:boto:Params: {} 2022-09-12T19:06:54.762 DEBUG:boto:Token: None 2022-09-12T19:06:54.762 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:54.763 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:++AwOAdDyifsUYLwo2pagwEsR+4= 2022-09-12T19:06:54.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:++AwOAdDyifsUYLwo2pagwEsR+4=', 'Content-Length': '0'} 2022-09-12T19:06:54.766 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'UBsYcCA94H55K4ucww5vXNAxuAMKNSG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e96fb2d7a2e3adc-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.767 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.767 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.768 DEBUG:boto:path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:54.768 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:54.769 DEBUG:boto:Method: GET 2022-09-12T19:06:54.769 DEBUG:boto:Path: /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:54.770 DEBUG:boto:Data: 2022-09-12T19:06:54.770 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.771 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.771 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.772 DEBUG:boto:Params: {} 2022-09-12T19:06:54.772 DEBUG:boto:Token: None 2022-09-12T19:06:54.773 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:06:54.774 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:++AwOAdDyifsUYLwo2pagwEsR+4= 2022-09-12T19:06:54.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:++AwOAdDyifsUYLwo2pagwEsR+4=', 'Content-Length': '0'} 2022-09-12T19:06:54.776 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'UBsYcCA94H55K4ucww5vXNAxuAMKNSG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004130997eb6996387-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.822 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:54.823 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:54.823 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.824 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.824 DEBUG:boto:Method: GET 2022-09-12T19:06:54.825 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:54.825 DEBUG:boto:Data: 2022-09-12T19:06:54.826 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.826 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.827 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.828 DEBUG:boto:Params: {} 2022-09-12T19:06:54.828 DEBUG:boto:Token: None 2022-09-12T19:06:54.829 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/ 2022-09-12T19:06:54.829 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo= 2022-09-12T19:06:54.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo=', 'Content-Length': '0'} 2022-09-12T19:06:54.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3be4759d3867393-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.834 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:54.835 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:54.836 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:54.837 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.837 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.838 DEBUG:boto:Method: GET 2022-09-12T19:06:54.839 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:54.840 DEBUG:boto:Data: 2022-09-12T19:06:54.841 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.842 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.843 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.844 DEBUG:boto:Params: {} 2022-09-12T19:06:54.845 DEBUG:boto:Token: None 2022-09-12T19:06:54.846 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/ 2022-09-12T19:06:54.847 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo= 2022-09-12T19:06:54.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo=', 'Content-Length': '0'} 2022-09-12T19:06:54.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000354f70ba8fa6bf63-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.853 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:54.855 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.856 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.857 DEBUG:boto:Method: HEAD 2022-09-12T19:06:54.858 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:54.859 DEBUG:boto:Data: 2022-09-12T19:06:54.860 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.860 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.861 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.862 DEBUG:boto:Params: {} 2022-09-12T19:06:54.863 DEBUG:boto:Token: None 2022-09-12T19:06:54.864 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj 2022-09-12T19:06:54.865 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LRoEHf7DrF1jVmt+ZC7Xc/G/050= 2022-09-12T19:06:54.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LRoEHf7DrF1jVmt+ZC7Xc/G/050=', 'Content-Length': '0'} 2022-09-12T19:06:54.871 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025bdc58e84f3b0b3-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.872 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.873 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.873 DEBUG:boto:Method: HEAD 2022-09-12T19:06:54.874 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:54.874 DEBUG:boto:Data: 2022-09-12T19:06:54.875 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.875 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.876 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.877 DEBUG:boto:Params: {} 2022-09-12T19:06:54.877 DEBUG:boto:Token: None 2022-09-12T19:06:54.878 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj 2022-09-12T19:06:54.878 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LRoEHf7DrF1jVmt+ZC7Xc/G/050= 2022-09-12T19:06:54.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LRoEHf7DrF1jVmt+ZC7Xc/G/050=', 'Content-Length': '0'} 2022-09-12T19:06:54.883 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000832c0a34f3f66469-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:54.885 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.885 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.886 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.886 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.887 DEBUG:boto:Method: GET 2022-09-12T19:06:54.887 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.888 DEBUG:boto:Data: 2022-09-12T19:06:54.888 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.889 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.889 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.890 DEBUG:boto:Params: {} 2022-09-12T19:06:54.891 DEBUG:boto:Token: None 2022-09-12T19:06:54.891 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.892 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f7KicsRggX2GVofXMr3QxGKS8N4= 2022-09-12T19:06:54.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f7KicsRggX2GVofXMr3QxGKS8N4=', 'Content-Length': '0'} 2022-09-12T19:06:54.895 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8eb08e1b98008ca-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.938 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.939 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.939 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.940 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.940 DEBUG:boto:Method: GET 2022-09-12T19:06:54.941 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.941 DEBUG:boto:Data: 2022-09-12T19:06:54.942 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.942 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.943 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.944 DEBUG:boto:Params: {} 2022-09-12T19:06:54.944 DEBUG:boto:Token: None 2022-09-12T19:06:54.945 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:54.945 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f7KicsRggX2GVofXMr3QxGKS8N4= 2022-09-12T19:06:54.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f7KicsRggX2GVofXMr3QxGKS8N4=', 'Content-Length': '0'} 2022-09-12T19:06:54.948 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d66f0187c9e275c1-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.949 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:54.949 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:54.950 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.951 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.951 DEBUG:boto:Method: GET 2022-09-12T19:06:54.952 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:54.953 DEBUG:boto:Data: 2022-09-12T19:06:54.953 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.954 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.954 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.955 DEBUG:boto:Params: {} 2022-09-12T19:06:54.955 DEBUG:boto:Token: None 2022-09-12T19:06:54.956 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/ 2022-09-12T19:06:54.956 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo= 2022-09-12T19:06:54.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo=', 'Content-Length': '0'} 2022-09-12T19:06:54.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc7ff8b71946ec26-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.960 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:54.961 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:54.962 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:54.962 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.963 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:54.963 DEBUG:boto:Method: GET 2022-09-12T19:06:54.964 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:54.964 DEBUG:boto:Data: 2022-09-12T19:06:54.965 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.965 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.966 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.966 DEBUG:boto:Params: {} 2022-09-12T19:06:54.967 DEBUG:boto:Token: None 2022-09-12T19:06:54.968 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/ 2022-09-12T19:06:54.968 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo= 2022-09-12T19:06:54.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6Ijm4KuriPeBDZaWPIQzWqDF9Oo=', 'Content-Length': '0'} 2022-09-12T19:06:54.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000780f28c0763e0c89-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.972 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:54.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:54.974 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.974 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.975 DEBUG:boto:path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:54.975 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:54.976 DEBUG:boto:Method: GET 2022-09-12T19:06:54.976 DEBUG:boto:Path: /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:54.977 DEBUG:boto:Data: 2022-09-12T19:06:54.977 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.978 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:54.978 DEBUG:boto:Port: 8001 2022-09-12T19:06:54.979 DEBUG:boto:Params: {} 2022-09-12T19:06:54.979 DEBUG:boto:Token: None 2022-09-12T19:06:54.980 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:54.981 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ABC9PesarSuKswdlOF21/Ul0bIg= 2022-09-12T19:06:54.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ABC9PesarSuKswdlOF21/Ul0bIg=', 'Content-Length': '0'} 2022-09-12T19:06:54.984 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6793f894a938933-00631f834e-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.985 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.985 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.986 DEBUG:boto:path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:54.986 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:54.987 DEBUG:boto:Method: GET 2022-09-12T19:06:54.987 DEBUG:boto:Path: /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:54.988 DEBUG:boto:Data: 2022-09-12T19:06:54.988 DEBUG:boto:Headers: {} 2022-09-12T19:06:54.989 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:54.989 DEBUG:boto:Port: 8000 2022-09-12T19:06:54.990 DEBUG:boto:Params: {} 2022-09-12T19:06:54.990 DEBUG:boto:Token: None 2022-09-12T19:06:54.991 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:54 GMT /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:54.992 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ABC9PesarSuKswdlOF21/Ul0bIg= 2022-09-12T19:06:54.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ABC9PesarSuKswdlOF21/Ul0bIg=', 'Content-Length': '0'} 2022-09-12T19:06:54.995 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005fb7ca14b78a91d9-00631f834e-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:54.996 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:54.996 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:54.997 DEBUG:boto:path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:54.997 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:54.998 DEBUG:boto:Method: HEAD 2022-09-12T19:06:54.998 DEBUG:boto:Path: /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:54.999 DEBUG:boto:Data: 2022-09-12T19:06:54.999 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.000 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.000 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.001 DEBUG:boto:Params: {} 2022-09-12T19:06:55.002 DEBUG:boto:Token: None 2022-09-12T19:06:55.002 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:55.003 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ORT/cjhcDfXoMUy4rPgdMsEeMTk= 2022-09-12T19:06:55.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ORT/cjhcDfXoMUy4rPgdMsEeMTk=', 'Content-Length': '0'} 2022-09-12T19:06:55.006 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7b4e051091503a0-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.007 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.007 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.008 DEBUG:boto:path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:55.008 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:55.009 DEBUG:boto:Method: HEAD 2022-09-12T19:06:55.009 DEBUG:boto:Path: /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:55.010 DEBUG:boto:Data: 2022-09-12T19:06:55.010 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.011 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.012 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.012 DEBUG:boto:Params: {} 2022-09-12T19:06:55.013 DEBUG:boto:Token: None 2022-09-12T19:06:55.014 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:55.014 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ORT/cjhcDfXoMUy4rPgdMsEeMTk= 2022-09-12T19:06:55.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ORT/cjhcDfXoMUy4rPgdMsEeMTk=', 'Content-Length': '0'} 2022-09-12T19:06:55.017 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055e0c92dd9ee0ffb-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:55.018 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.019 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.019 DEBUG:boto:path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:55.020 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:55.020 DEBUG:boto:Method: GET 2022-09-12T19:06:55.021 DEBUG:boto:Path: /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:55.021 DEBUG:boto:Data: 2022-09-12T19:06:55.022 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.023 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.023 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.024 DEBUG:boto:Params: {} 2022-09-12T19:06:55.024 DEBUG:boto:Token: None 2022-09-12T19:06:55.025 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:55.026 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WMfPr4yiBSEP98QnE2ctoHIDah4= 2022-09-12T19:06:55.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WMfPr4yiBSEP98QnE2ctoHIDah4=', 'Content-Length': '0'} 2022-09-12T19:06:55.028 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000821e64cfaa61a37d-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.029 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.030 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.030 DEBUG:boto:path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:55.031 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:55.032 DEBUG:boto:Method: GET 2022-09-12T19:06:55.033 DEBUG:boto:Path: /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:55.034 DEBUG:boto:Data: 2022-09-12T19:06:55.035 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.036 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.036 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.037 DEBUG:boto:Params: {} 2022-09-12T19:06:55.039 DEBUG:boto:Token: None 2022-09-12T19:06:55.040 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:06:55.041 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WMfPr4yiBSEP98QnE2ctoHIDah4= 2022-09-12T19:06:55.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WMfPr4yiBSEP98QnE2ctoHIDah4=', 'Content-Length': '0'} 2022-09-12T19:06:55.044 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e5779f4d0b1c30c4-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.090 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:55.091 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:55.091 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.092 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.092 DEBUG:boto:Method: GET 2022-09-12T19:06:55.093 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:55.094 DEBUG:boto:Data: 2022-09-12T19:06:55.094 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.095 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.095 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.096 DEBUG:boto:Params: {} 2022-09-12T19:06:55.096 DEBUG:boto:Token: None 2022-09-12T19:06:55.097 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/ 2022-09-12T19:06:55.098 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0= 2022-09-12T19:06:55.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0=', 'Content-Length': '0'} 2022-09-12T19:06:55.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017e5ded6097a4605-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.102 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:55.103 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:55.103 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:55.104 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.104 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.105 DEBUG:boto:Method: GET 2022-09-12T19:06:55.105 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:55.106 DEBUG:boto:Data: 2022-09-12T19:06:55.106 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.107 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.107 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.108 DEBUG:boto:Params: {} 2022-09-12T19:06:55.109 DEBUG:boto:Token: None 2022-09-12T19:06:55.109 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/ 2022-09-12T19:06:55.110 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0= 2022-09-12T19:06:55.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0=', 'Content-Length': '0'} 2022-09-12T19:06:55.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007245ae613061c87e-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.114 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:55.115 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.116 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.116 DEBUG:boto:Method: HEAD 2022-09-12T19:06:55.117 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:55.118 DEBUG:boto:Data: 2022-09-12T19:06:55.118 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.119 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.119 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.120 DEBUG:boto:Params: {} 2022-09-12T19:06:55.120 DEBUG:boto:Token: None 2022-09-12T19:06:55.121 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj 2022-09-12T19:06:55.122 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HSZf881YoCGY71g/xUI74CG6fzs= 2022-09-12T19:06:55.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HSZf881YoCGY71g/xUI74CG6fzs=', 'Content-Length': '0'} 2022-09-12T19:06:55.126 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dbf378efab47377a-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.127 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.127 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.128 DEBUG:boto:Method: HEAD 2022-09-12T19:06:55.128 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:55.129 DEBUG:boto:Data: 2022-09-12T19:06:55.129 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.130 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.130 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.131 DEBUG:boto:Params: {} 2022-09-12T19:06:55.132 DEBUG:boto:Token: None 2022-09-12T19:06:55.132 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj 2022-09-12T19:06:55.133 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HSZf881YoCGY71g/xUI74CG6fzs= 2022-09-12T19:06:55.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HSZf881YoCGY71g/xUI74CG6fzs=', 'Content-Length': '0'} 2022-09-12T19:06:55.137 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5e27fc9dadfc416-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:55.139 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.139 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.140 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.140 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.141 DEBUG:boto:Method: GET 2022-09-12T19:06:55.142 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.142 DEBUG:boto:Data: 2022-09-12T19:06:55.143 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.143 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.144 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.144 DEBUG:boto:Params: {} 2022-09-12T19:06:55.145 DEBUG:boto:Token: None 2022-09-12T19:06:55.146 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.147 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GfslJb7/2ga4627svAL/BaeCF+Q= 2022-09-12T19:06:55.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GfslJb7/2ga4627svAL/BaeCF+Q=', 'Content-Length': '0'} 2022-09-12T19:06:55.151 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072797f03c31f80d2-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.152 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.153 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.153 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.154 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.154 DEBUG:boto:Method: GET 2022-09-12T19:06:55.155 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.155 DEBUG:boto:Data: 2022-09-12T19:06:55.156 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.156 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.157 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.158 DEBUG:boto:Params: {} 2022-09-12T19:06:55.158 DEBUG:boto:Token: None 2022-09-12T19:06:55.159 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.159 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GfslJb7/2ga4627svAL/BaeCF+Q= 2022-09-12T19:06:55.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GfslJb7/2ga4627svAL/BaeCF+Q=', 'Content-Length': '0'} 2022-09-12T19:06:55.162 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000023fd566b0a746424-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.163 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:55.164 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:55.164 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.165 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.165 DEBUG:boto:Method: GET 2022-09-12T19:06:55.166 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:55.166 DEBUG:boto:Data: 2022-09-12T19:06:55.167 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.167 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.168 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.168 DEBUG:boto:Params: {} 2022-09-12T19:06:55.169 DEBUG:boto:Token: None 2022-09-12T19:06:55.169 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/ 2022-09-12T19:06:55.170 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0= 2022-09-12T19:06:55.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0=', 'Content-Length': '0'} 2022-09-12T19:06:55.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042cee715300fd631-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.218 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:55.219 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:55.220 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:55.220 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.221 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.221 DEBUG:boto:Method: GET 2022-09-12T19:06:55.222 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:55.222 DEBUG:boto:Data: 2022-09-12T19:06:55.223 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.223 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.224 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.225 DEBUG:boto:Params: {} 2022-09-12T19:06:55.225 DEBUG:boto:Token: None 2022-09-12T19:06:55.226 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/ 2022-09-12T19:06:55.227 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0= 2022-09-12T19:06:55.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0=', 'Content-Length': '0'} 2022-09-12T19:06:55.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003edbc8452304385d-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.232 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:55.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:55.233 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.234 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.234 DEBUG:boto:path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:55.235 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:55.235 DEBUG:boto:Method: GET 2022-09-12T19:06:55.236 DEBUG:boto:Path: /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:55.236 DEBUG:boto:Data: 2022-09-12T19:06:55.237 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.237 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.238 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.239 DEBUG:boto:Params: {} 2022-09-12T19:06:55.239 DEBUG:boto:Token: None 2022-09-12T19:06:55.240 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:55.241 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c+MazPZvNTKm9q+VcWcgSy0jU8Q= 2022-09-12T19:06:55.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c+MazPZvNTKm9q+VcWcgSy0jU8Q=', 'Content-Length': '0'} 2022-09-12T19:06:55.243 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', '.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b161747024d6323-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.244 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.245 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.246 DEBUG:boto:path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:55.246 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:55.247 DEBUG:boto:Method: GET 2022-09-12T19:06:55.247 DEBUG:boto:Path: /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:55.248 DEBUG:boto:Data: 2022-09-12T19:06:55.248 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.249 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.250 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.250 DEBUG:boto:Params: {} 2022-09-12T19:06:55.251 DEBUG:boto:Token: None 2022-09-12T19:06:55.251 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:55.252 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c+MazPZvNTKm9q+VcWcgSy0jU8Q= 2022-09-12T19:06:55.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c+MazPZvNTKm9q+VcWcgSy0jU8Q=', 'Content-Length': '0'} 2022-09-12T19:06:55.255 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', '.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b00f2906a67c545-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.256 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.256 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.257 DEBUG:boto:path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:55.257 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:55.258 DEBUG:boto:Method: HEAD 2022-09-12T19:06:55.258 DEBUG:boto:Path: /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:55.259 DEBUG:boto:Data: 2022-09-12T19:06:55.259 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.260 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.260 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.261 DEBUG:boto:Params: {} 2022-09-12T19:06:55.261 DEBUG:boto:Token: None 2022-09-12T19:06:55.262 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:55.263 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:00EJQz4IUQ4Fhs0s+BLBrt/lL3s= 2022-09-12T19:06:55.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:00EJQz4IUQ4Fhs0s+BLBrt/lL3s=', 'Content-Length': '0'} 2022-09-12T19:06:55.266 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', '.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9afd8a308b2c35c-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.267 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.267 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.268 DEBUG:boto:path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:55.268 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:55.269 DEBUG:boto:Method: HEAD 2022-09-12T19:06:55.269 DEBUG:boto:Path: /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:55.270 DEBUG:boto:Data: 2022-09-12T19:06:55.271 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.271 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.271 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.272 DEBUG:boto:Params: {} 2022-09-12T19:06:55.273 DEBUG:boto:Token: None 2022-09-12T19:06:55.273 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:55.274 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:00EJQz4IUQ4Fhs0s+BLBrt/lL3s= 2022-09-12T19:06:55.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:00EJQz4IUQ4Fhs0s+BLBrt/lL3s=', 'Content-Length': '0'} 2022-09-12T19:06:55.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', '.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000294352416090cb3-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:55.278 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.279 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.279 DEBUG:boto:path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:55.280 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:55.281 DEBUG:boto:Method: GET 2022-09-12T19:06:55.282 DEBUG:boto:Path: /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:55.283 DEBUG:boto:Data: 2022-09-12T19:06:55.283 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.284 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.285 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.286 DEBUG:boto:Params: {} 2022-09-12T19:06:55.286 DEBUG:boto:Token: None 2022-09-12T19:06:55.287 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:55.287 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c+MazPZvNTKm9q+VcWcgSy0jU8Q= 2022-09-12T19:06:55.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c+MazPZvNTKm9q+VcWcgSy0jU8Q=', 'Content-Length': '0'} 2022-09-12T19:06:55.290 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', '.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000200b55168fd1c02b-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.291 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.292 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.293 DEBUG:boto:path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:55.293 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:55.294 DEBUG:boto:Method: GET 2022-09-12T19:06:55.294 DEBUG:boto:Path: /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:55.295 DEBUG:boto:Data: 2022-09-12T19:06:55.296 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.296 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.297 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.297 DEBUG:boto:Params: {} 2022-09-12T19:06:55.298 DEBUG:boto:Token: None 2022-09-12T19:06:55.298 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:06:55.299 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c+MazPZvNTKm9q+VcWcgSy0jU8Q= 2022-09-12T19:06:55.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c+MazPZvNTKm9q+VcWcgSy0jU8Q=', 'Content-Length': '0'} 2022-09-12T19:06:55.302 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', '.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5c0907a339b8c98-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.346 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:55.347 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:55.347 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.348 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.348 DEBUG:boto:Method: GET 2022-09-12T19:06:55.349 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:55.350 DEBUG:boto:Data: 2022-09-12T19:06:55.351 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.351 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.352 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.352 DEBUG:boto:Params: {} 2022-09-12T19:06:55.353 DEBUG:boto:Token: None 2022-09-12T19:06:55.354 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/ 2022-09-12T19:06:55.354 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0= 2022-09-12T19:06:55.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0=', 'Content-Length': '0'} 2022-09-12T19:06:55.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7cd51d9d23f52cd-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.359 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:55.360 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:55.361 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:55.362 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.363 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.363 DEBUG:boto:Method: GET 2022-09-12T19:06:55.364 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:55.365 DEBUG:boto:Data: 2022-09-12T19:06:55.366 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.366 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.367 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.367 DEBUG:boto:Params: {} 2022-09-12T19:06:55.368 DEBUG:boto:Token: None 2022-09-12T19:06:55.368 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/ 2022-09-12T19:06:55.369 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0= 2022-09-12T19:06:55.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0=', 'Content-Length': '0'} 2022-09-12T19:06:55.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbe09235d9bf4739-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.373 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:55.374 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.375 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.375 DEBUG:boto:Method: HEAD 2022-09-12T19:06:55.376 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:55.376 DEBUG:boto:Data: 2022-09-12T19:06:55.377 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.377 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.378 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.378 DEBUG:boto:Params: {} 2022-09-12T19:06:55.379 DEBUG:boto:Token: None 2022-09-12T19:06:55.380 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj 2022-09-12T19:06:55.380 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HSZf881YoCGY71g/xUI74CG6fzs= 2022-09-12T19:06:55.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HSZf881YoCGY71g/xUI74CG6fzs=', 'Content-Length': '0'} 2022-09-12T19:06:55.385 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa527e5a9284a88f-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.385 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.386 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.386 DEBUG:boto:Method: HEAD 2022-09-12T19:06:55.387 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:55.387 DEBUG:boto:Data: 2022-09-12T19:06:55.388 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.388 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.389 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.390 DEBUG:boto:Params: {} 2022-09-12T19:06:55.390 DEBUG:boto:Token: None 2022-09-12T19:06:55.391 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj 2022-09-12T19:06:55.391 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HSZf881YoCGY71g/xUI74CG6fzs= 2022-09-12T19:06:55.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HSZf881YoCGY71g/xUI74CG6fzs=', 'Content-Length': '0'} 2022-09-12T19:06:55.396 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000408791e306d89d67-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:55.397 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.398 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.398 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.399 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.399 DEBUG:boto:Method: GET 2022-09-12T19:06:55.400 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.400 DEBUG:boto:Data: 2022-09-12T19:06:55.401 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.401 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.402 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.402 DEBUG:boto:Params: {} 2022-09-12T19:06:55.403 DEBUG:boto:Token: None 2022-09-12T19:06:55.404 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.404 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GfslJb7/2ga4627svAL/BaeCF+Q= 2022-09-12T19:06:55.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GfslJb7/2ga4627svAL/BaeCF+Q=', 'Content-Length': '0'} 2022-09-12T19:06:55.407 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038c9904c0f1b5783-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.454 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.455 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.456 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.457 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.457 DEBUG:boto:Method: GET 2022-09-12T19:06:55.458 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.459 DEBUG:boto:Data: 2022-09-12T19:06:55.459 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.460 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.460 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.461 DEBUG:boto:Params: {} 2022-09-12T19:06:55.462 DEBUG:boto:Token: None 2022-09-12T19:06:55.463 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.463 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GfslJb7/2ga4627svAL/BaeCF+Q= 2022-09-12T19:06:55.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GfslJb7/2ga4627svAL/BaeCF+Q=', 'Content-Length': '0'} 2022-09-12T19:06:55.466 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010f66a4d7fa0b5f3-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.467 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:55.468 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:55.468 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.469 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.469 DEBUG:boto:Method: GET 2022-09-12T19:06:55.470 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:55.471 DEBUG:boto:Data: 2022-09-12T19:06:55.472 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.472 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.473 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.473 DEBUG:boto:Params: {} 2022-09-12T19:06:55.474 DEBUG:boto:Token: None 2022-09-12T19:06:55.475 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/ 2022-09-12T19:06:55.476 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0= 2022-09-12T19:06:55.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0=', 'Content-Length': '0'} 2022-09-12T19:06:55.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3315e058f65c9ad-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.481 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:55.482 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:55.482 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:55.483 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.483 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.484 DEBUG:boto:Method: GET 2022-09-12T19:06:55.484 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:55.485 DEBUG:boto:Data: 2022-09-12T19:06:55.486 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.487 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.488 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.489 DEBUG:boto:Params: {} 2022-09-12T19:06:55.490 DEBUG:boto:Token: None 2022-09-12T19:06:55.491 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/ 2022-09-12T19:06:55.492 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0= 2022-09-12T19:06:55.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0=', 'Content-Length': '0'} 2022-09-12T19:06:55.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c92a1c8632c82f90-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.497 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:55.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:55.499 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.499 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.500 DEBUG:boto:path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:55.500 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:55.501 DEBUG:boto:Method: GET 2022-09-12T19:06:55.501 DEBUG:boto:Path: /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:55.502 DEBUG:boto:Data: 2022-09-12T19:06:55.502 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.503 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.503 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.504 DEBUG:boto:Params: {} 2022-09-12T19:06:55.505 DEBUG:boto:Token: None 2022-09-12T19:06:55.505 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:55.506 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+w42mPV7d2I6nKKX0IFHjlDb2vI= 2022-09-12T19:06:55.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+w42mPV7d2I6nKKX0IFHjlDb2vI=', 'Content-Length': '0'} 2022-09-12T19:06:55.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'K.LUooPdPnXZMzfYIxw-xagQpErk4CX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d192bf9f2c7d286a-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.510 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.510 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.511 DEBUG:boto:path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:55.511 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:55.512 DEBUG:boto:Method: GET 2022-09-12T19:06:55.512 DEBUG:boto:Path: /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:55.513 DEBUG:boto:Data: 2022-09-12T19:06:55.513 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.514 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.515 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.515 DEBUG:boto:Params: {} 2022-09-12T19:06:55.516 DEBUG:boto:Token: None 2022-09-12T19:06:55.516 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:55.517 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+w42mPV7d2I6nKKX0IFHjlDb2vI= 2022-09-12T19:06:55.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+w42mPV7d2I6nKKX0IFHjlDb2vI=', 'Content-Length': '0'} 2022-09-12T19:06:55.520 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'K.LUooPdPnXZMzfYIxw-xagQpErk4CX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b029f2b4635b2d7-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.521 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.521 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.522 DEBUG:boto:path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:55.522 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:55.523 DEBUG:boto:Method: HEAD 2022-09-12T19:06:55.523 DEBUG:boto:Path: /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:55.524 DEBUG:boto:Data: 2022-09-12T19:06:55.524 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.525 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.525 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.526 DEBUG:boto:Params: {} 2022-09-12T19:06:55.526 DEBUG:boto:Token: None 2022-09-12T19:06:55.527 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:55.527 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/F+ayLqALSNE6G+WBzpAWGv9Ztc= 2022-09-12T19:06:55.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/F+ayLqALSNE6G+WBzpAWGv9Ztc=', 'Content-Length': '0'} 2022-09-12T19:06:55.530 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'K.LUooPdPnXZMzfYIxw-xagQpErk4CX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e777121256ae2ec2-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.531 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.532 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.532 DEBUG:boto:path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:55.533 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:55.534 DEBUG:boto:Method: HEAD 2022-09-12T19:06:55.534 DEBUG:boto:Path: /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:55.535 DEBUG:boto:Data: 2022-09-12T19:06:55.535 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.536 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.536 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.537 DEBUG:boto:Params: {} 2022-09-12T19:06:55.537 DEBUG:boto:Token: None 2022-09-12T19:06:55.538 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:55.539 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/F+ayLqALSNE6G+WBzpAWGv9Ztc= 2022-09-12T19:06:55.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/F+ayLqALSNE6G+WBzpAWGv9Ztc=', 'Content-Length': '0'} 2022-09-12T19:06:55.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'K.LUooPdPnXZMzfYIxw-xagQpErk4CX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a783c11ad7e91a2c-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:55.542 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.543 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.543 DEBUG:boto:path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:55.544 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:55.545 DEBUG:boto:Method: GET 2022-09-12T19:06:55.545 DEBUG:boto:Path: /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:55.546 DEBUG:boto:Data: 2022-09-12T19:06:55.546 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.547 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.547 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.548 DEBUG:boto:Params: {} 2022-09-12T19:06:55.549 DEBUG:boto:Token: None 2022-09-12T19:06:55.549 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:55.550 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+w42mPV7d2I6nKKX0IFHjlDb2vI= 2022-09-12T19:06:55.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+w42mPV7d2I6nKKX0IFHjlDb2vI=', 'Content-Length': '0'} 2022-09-12T19:06:55.553 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'K.LUooPdPnXZMzfYIxw-xagQpErk4CX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d11802f3ba976d3-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.554 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.555 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.555 DEBUG:boto:path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:55.556 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:55.556 DEBUG:boto:Method: GET 2022-09-12T19:06:55.557 DEBUG:boto:Path: /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:55.558 DEBUG:boto:Data: 2022-09-12T19:06:55.558 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.559 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.559 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.560 DEBUG:boto:Params: {} 2022-09-12T19:06:55.560 DEBUG:boto:Token: None 2022-09-12T19:06:55.561 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:06:55.562 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+w42mPV7d2I6nKKX0IFHjlDb2vI= 2022-09-12T19:06:55.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+w42mPV7d2I6nKKX0IFHjlDb2vI=', 'Content-Length': '0'} 2022-09-12T19:06:55.564 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'K.LUooPdPnXZMzfYIxw-xagQpErk4CX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000283c8b121cab15c5-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.565 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:55.566 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:55.566 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.567 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.567 DEBUG:boto:Method: GET 2022-09-12T19:06:55.568 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:55.568 DEBUG:boto:Data: 2022-09-12T19:06:55.569 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.570 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.570 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.571 DEBUG:boto:Params: {} 2022-09-12T19:06:55.571 DEBUG:boto:Token: None 2022-09-12T19:06:55.572 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/ 2022-09-12T19:06:55.572 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0= 2022-09-12T19:06:55.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0=', 'Content-Length': '0'} 2022-09-12T19:06:55.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b31c76d3570b9a3-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.577 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:55.578 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:55.579 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:55.579 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.580 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.580 DEBUG:boto:Method: GET 2022-09-12T19:06:55.581 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:55.582 DEBUG:boto:Data: 2022-09-12T19:06:55.582 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.583 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.583 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.584 DEBUG:boto:Params: {} 2022-09-12T19:06:55.584 DEBUG:boto:Token: None 2022-09-12T19:06:55.585 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/ 2022-09-12T19:06:55.586 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0= 2022-09-12T19:06:55.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0=', 'Content-Length': '0'} 2022-09-12T19:06:55.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b3c057ed62a53c4-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.590 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:55.591 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.592 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.592 DEBUG:boto:Method: HEAD 2022-09-12T19:06:55.593 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:55.593 DEBUG:boto:Data: 2022-09-12T19:06:55.594 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.595 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.596 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.597 DEBUG:boto:Params: {} 2022-09-12T19:06:55.598 DEBUG:boto:Token: None 2022-09-12T19:06:55.599 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj 2022-09-12T19:06:55.600 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HSZf881YoCGY71g/xUI74CG6fzs= 2022-09-12T19:06:55.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HSZf881YoCGY71g/xUI74CG6fzs=', 'Content-Length': '0'} 2022-09-12T19:06:55.604 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007bf06f695c84f85e-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.605 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.605 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.606 DEBUG:boto:Method: HEAD 2022-09-12T19:06:55.607 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:55.607 DEBUG:boto:Data: 2022-09-12T19:06:55.608 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.608 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.609 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.609 DEBUG:boto:Params: {} 2022-09-12T19:06:55.610 DEBUG:boto:Token: None 2022-09-12T19:06:55.610 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj 2022-09-12T19:06:55.611 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HSZf881YoCGY71g/xUI74CG6fzs= 2022-09-12T19:06:55.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HSZf881YoCGY71g/xUI74CG6fzs=', 'Content-Length': '0'} 2022-09-12T19:06:55.615 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000555ff7e1c997ab7f-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:55.616 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.617 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.617 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.618 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.618 DEBUG:boto:Method: GET 2022-09-12T19:06:55.619 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.620 DEBUG:boto:Data: 2022-09-12T19:06:55.620 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.621 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.621 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.622 DEBUG:boto:Params: {} 2022-09-12T19:06:55.623 DEBUG:boto:Token: None 2022-09-12T19:06:55.623 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.624 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GfslJb7/2ga4627svAL/BaeCF+Q= 2022-09-12T19:06:55.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GfslJb7/2ga4627svAL/BaeCF+Q=', 'Content-Length': '0'} 2022-09-12T19:06:55.627 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007274596a9f6515ed-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.670 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.671 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.671 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.672 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.672 DEBUG:boto:Method: GET 2022-09-12T19:06:55.673 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.673 DEBUG:boto:Data: 2022-09-12T19:06:55.674 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.675 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.675 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.676 DEBUG:boto:Params: {} 2022-09-12T19:06:55.676 DEBUG:boto:Token: None 2022-09-12T19:06:55.677 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.677 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GfslJb7/2ga4627svAL/BaeCF+Q= 2022-09-12T19:06:55.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GfslJb7/2ga4627svAL/BaeCF+Q=', 'Content-Length': '0'} 2022-09-12T19:06:55.680 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd78123176080a77-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.681 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:55.682 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:55.682 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.683 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.683 DEBUG:boto:Method: GET 2022-09-12T19:06:55.684 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:55.684 DEBUG:boto:Data: 2022-09-12T19:06:55.685 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.685 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.686 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.686 DEBUG:boto:Params: {} 2022-09-12T19:06:55.687 DEBUG:boto:Token: None 2022-09-12T19:06:55.688 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/ 2022-09-12T19:06:55.688 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0= 2022-09-12T19:06:55.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0=', 'Content-Length': '0'} 2022-09-12T19:06:55.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069333a759fbaba9d-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.693 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:55.693 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:55.694 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:55.695 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.695 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.696 DEBUG:boto:Method: GET 2022-09-12T19:06:55.696 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:55.697 DEBUG:boto:Data: 2022-09-12T19:06:55.697 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.698 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.698 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.699 DEBUG:boto:Params: {} 2022-09-12T19:06:55.699 DEBUG:boto:Token: None 2022-09-12T19:06:55.700 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/ 2022-09-12T19:06:55.701 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0= 2022-09-12T19:06:55.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0=', 'Content-Length': '0'} 2022-09-12T19:06:55.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000565e568c3d90494d-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.705 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:55.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:55.706 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.707 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.707 DEBUG:boto:path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:55.708 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:55.708 DEBUG:boto:Method: GET 2022-09-12T19:06:55.709 DEBUG:boto:Path: /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:55.709 DEBUG:boto:Data: 2022-09-12T19:06:55.710 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.710 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.711 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.712 DEBUG:boto:Params: {} 2022-09-12T19:06:55.712 DEBUG:boto:Token: None 2022-09-12T19:06:55.713 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:55.713 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SWd2i7/agTLDJuc/xgx1r+ZucwM= 2022-09-12T19:06:55.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SWd2i7/agTLDJuc/xgx1r+ZucwM=', 'Content-Length': '0'} 2022-09-12T19:06:55.716 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'tQHk21B862H4VHQxrZWEog8n-qgO99Q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093510d798f557445-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.717 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.718 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.718 DEBUG:boto:path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:55.719 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:55.719 DEBUG:boto:Method: GET 2022-09-12T19:06:55.720 DEBUG:boto:Path: /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:55.720 DEBUG:boto:Data: 2022-09-12T19:06:55.721 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.721 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.722 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.722 DEBUG:boto:Params: {} 2022-09-12T19:06:55.723 DEBUG:boto:Token: None 2022-09-12T19:06:55.723 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:55.724 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SWd2i7/agTLDJuc/xgx1r+ZucwM= 2022-09-12T19:06:55.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SWd2i7/agTLDJuc/xgx1r+ZucwM=', 'Content-Length': '0'} 2022-09-12T19:06:55.727 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'tQHk21B862H4VHQxrZWEog8n-qgO99Q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000968b002525435fa7-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.727 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.728 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.728 DEBUG:boto:path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:55.729 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:55.729 DEBUG:boto:Method: HEAD 2022-09-12T19:06:55.730 DEBUG:boto:Path: /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:55.731 DEBUG:boto:Data: 2022-09-12T19:06:55.731 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.731 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.732 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.732 DEBUG:boto:Params: {} 2022-09-12T19:06:55.733 DEBUG:boto:Token: None 2022-09-12T19:06:55.734 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:55.734 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dom1+s53eCwT2PcXaSjqN5SrlCw= 2022-09-12T19:06:55.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dom1+s53eCwT2PcXaSjqN5SrlCw=', 'Content-Length': '0'} 2022-09-12T19:06:55.737 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'tQHk21B862H4VHQxrZWEog8n-qgO99Q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057aa74f8912259d9-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.737 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.738 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.738 DEBUG:boto:path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:55.739 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:55.739 DEBUG:boto:Method: HEAD 2022-09-12T19:06:55.740 DEBUG:boto:Path: /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:55.740 DEBUG:boto:Data: 2022-09-12T19:06:55.741 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.741 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.742 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.742 DEBUG:boto:Params: {} 2022-09-12T19:06:55.743 DEBUG:boto:Token: None 2022-09-12T19:06:55.743 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:55.744 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dom1+s53eCwT2PcXaSjqN5SrlCw= 2022-09-12T19:06:55.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dom1+s53eCwT2PcXaSjqN5SrlCw=', 'Content-Length': '0'} 2022-09-12T19:06:55.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'tQHk21B862H4VHQxrZWEog8n-qgO99Q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000023f0ef9eae44492-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:55.748 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.748 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.749 DEBUG:boto:path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:55.749 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:55.750 DEBUG:boto:Method: GET 2022-09-12T19:06:55.750 DEBUG:boto:Path: /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:55.751 DEBUG:boto:Data: 2022-09-12T19:06:55.752 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.752 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.753 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.753 DEBUG:boto:Params: {} 2022-09-12T19:06:55.754 DEBUG:boto:Token: None 2022-09-12T19:06:55.754 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:55.755 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SWd2i7/agTLDJuc/xgx1r+ZucwM= 2022-09-12T19:06:55.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SWd2i7/agTLDJuc/xgx1r+ZucwM=', 'Content-Length': '0'} 2022-09-12T19:06:55.758 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'tQHk21B862H4VHQxrZWEog8n-qgO99Q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a34784674b6faf71-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.758 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.759 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.760 DEBUG:boto:path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:55.760 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:55.761 DEBUG:boto:Method: GET 2022-09-12T19:06:55.761 DEBUG:boto:Path: /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:55.762 DEBUG:boto:Data: 2022-09-12T19:06:55.762 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.763 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.763 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.764 DEBUG:boto:Params: {} 2022-09-12T19:06:55.764 DEBUG:boto:Token: None 2022-09-12T19:06:55.765 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:06:55.765 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SWd2i7/agTLDJuc/xgx1r+ZucwM= 2022-09-12T19:06:55.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SWd2i7/agTLDJuc/xgx1r+ZucwM=', 'Content-Length': '0'} 2022-09-12T19:06:55.768 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'tQHk21B862H4VHQxrZWEog8n-qgO99Q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d0ca4e0de072b7c-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.810 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:55.811 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:55.811 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.812 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.812 DEBUG:boto:Method: GET 2022-09-12T19:06:55.813 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:55.813 DEBUG:boto:Data: 2022-09-12T19:06:55.814 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.814 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.815 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.816 DEBUG:boto:Params: {} 2022-09-12T19:06:55.816 DEBUG:boto:Token: None 2022-09-12T19:06:55.817 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/ 2022-09-12T19:06:55.817 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0= 2022-09-12T19:06:55.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0=', 'Content-Length': '0'} 2022-09-12T19:06:55.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b09d4d3d8c018ea0-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.822 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:55.824 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:55.825 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:55.826 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.827 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.828 DEBUG:boto:Method: GET 2022-09-12T19:06:55.829 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:55.829 DEBUG:boto:Data: 2022-09-12T19:06:55.830 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.831 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.832 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.833 DEBUG:boto:Params: {} 2022-09-12T19:06:55.834 DEBUG:boto:Token: None 2022-09-12T19:06:55.835 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/ 2022-09-12T19:06:55.836 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0= 2022-09-12T19:06:55.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0=', 'Content-Length': '0'} 2022-09-12T19:06:55.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc76a83bb26623d9-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.842 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:55.843 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.844 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.845 DEBUG:boto:Method: HEAD 2022-09-12T19:06:55.846 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:55.846 DEBUG:boto:Data: 2022-09-12T19:06:55.847 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.847 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.848 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.849 DEBUG:boto:Params: {} 2022-09-12T19:06:55.850 DEBUG:boto:Token: None 2022-09-12T19:06:55.851 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj 2022-09-12T19:06:55.852 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HSZf881YoCGY71g/xUI74CG6fzs= 2022-09-12T19:06:55.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HSZf881YoCGY71g/xUI74CG6fzs=', 'Content-Length': '0'} 2022-09-12T19:06:55.857 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002bc9882448ec769d-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.858 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.859 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.860 DEBUG:boto:Method: HEAD 2022-09-12T19:06:55.860 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:55.861 DEBUG:boto:Data: 2022-09-12T19:06:55.862 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.863 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.864 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.865 DEBUG:boto:Params: {} 2022-09-12T19:06:55.865 DEBUG:boto:Token: None 2022-09-12T19:06:55.866 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj 2022-09-12T19:06:55.867 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HSZf881YoCGY71g/xUI74CG6fzs= 2022-09-12T19:06:55.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HSZf881YoCGY71g/xUI74CG6fzs=', 'Content-Length': '0'} 2022-09-12T19:06:55.872 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046ed7c6f0754146f-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:55.874 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.875 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.876 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.877 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.878 DEBUG:boto:Method: GET 2022-09-12T19:06:55.879 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.880 DEBUG:boto:Data: 2022-09-12T19:06:55.880 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.881 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.882 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.883 DEBUG:boto:Params: {} 2022-09-12T19:06:55.884 DEBUG:boto:Token: None 2022-09-12T19:06:55.885 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.886 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GfslJb7/2ga4627svAL/BaeCF+Q= 2022-09-12T19:06:55.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GfslJb7/2ga4627svAL/BaeCF+Q=', 'Content-Length': '0'} 2022-09-12T19:06:55.890 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080c87fccd35ce640-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.890 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.891 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.892 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.892 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.893 DEBUG:boto:Method: GET 2022-09-12T19:06:55.893 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.894 DEBUG:boto:Data: 2022-09-12T19:06:55.894 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.895 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.896 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.896 DEBUG:boto:Params: {} 2022-09-12T19:06:55.897 DEBUG:boto:Token: None 2022-09-12T19:06:55.897 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:55.898 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GfslJb7/2ga4627svAL/BaeCF+Q= 2022-09-12T19:06:55.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GfslJb7/2ga4627svAL/BaeCF+Q=', 'Content-Length': '0'} 2022-09-12T19:06:55.901 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f83583b721e2ca47-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.901 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:55.902 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:55.903 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.903 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.904 DEBUG:boto:Method: GET 2022-09-12T19:06:55.904 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:55.905 DEBUG:boto:Data: 2022-09-12T19:06:55.906 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.906 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.907 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.907 DEBUG:boto:Params: {} 2022-09-12T19:06:55.908 DEBUG:boto:Token: None 2022-09-12T19:06:55.908 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/ 2022-09-12T19:06:55.909 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0= 2022-09-12T19:06:55.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0=', 'Content-Length': '0'} 2022-09-12T19:06:55.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f72b5aa021ebe1ea-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.914 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:55.915 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:55.915 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:55.916 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.916 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:55.917 DEBUG:boto:Method: GET 2022-09-12T19:06:55.917 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:55.918 DEBUG:boto:Data: 2022-09-12T19:06:55.918 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.919 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.920 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.920 DEBUG:boto:Params: {} 2022-09-12T19:06:55.921 DEBUG:boto:Token: None 2022-09-12T19:06:55.922 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/ 2022-09-12T19:06:55.922 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0= 2022-09-12T19:06:55.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gDaVPlP++DtU5eSlCt7XLHHSxi0=', 'Content-Length': '0'} 2022-09-12T19:06:55.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c884fc300393ce0-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.926 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:55.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:55.928 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.929 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.929 DEBUG:boto:path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:55.930 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:55.930 DEBUG:boto:Method: GET 2022-09-12T19:06:55.931 DEBUG:boto:Path: /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:55.931 DEBUG:boto:Data: 2022-09-12T19:06:55.932 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.932 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.933 DEBUG:boto:Port: 8001 2022-09-12T19:06:55.933 DEBUG:boto:Params: {} 2022-09-12T19:06:55.934 DEBUG:boto:Token: None 2022-09-12T19:06:55.935 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:55.935 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QiZhqZOc+PeFnDybOsRA0Pu7+yg= 2022-09-12T19:06:55.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QiZhqZOc+PeFnDybOsRA0Pu7+yg=', 'Content-Length': '0'} 2022-09-12T19:06:55.938 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'QHXOu7SGV87vItdBeqaL.hYUlX--Ty8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b0bb84b70b297c6-00631f834f-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.939 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.940 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.940 DEBUG:boto:path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:55.941 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:55.941 DEBUG:boto:Method: GET 2022-09-12T19:06:55.942 DEBUG:boto:Path: /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:55.943 DEBUG:boto:Data: 2022-09-12T19:06:55.943 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.944 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:55.945 DEBUG:boto:Port: 8000 2022-09-12T19:06:55.945 DEBUG:boto:Params: {} 2022-09-12T19:06:55.946 DEBUG:boto:Token: None 2022-09-12T19:06:55.946 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:55 GMT /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:55.947 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QiZhqZOc+PeFnDybOsRA0Pu7+yg= 2022-09-12T19:06:55.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QiZhqZOc+PeFnDybOsRA0Pu7+yg=', 'Content-Length': '0'} 2022-09-12T19:06:55.950 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'QHXOu7SGV87vItdBeqaL.hYUlX--Ty8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042c9ccaf92144455-00631f834f-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:55.994 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:55.995 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:55.995 DEBUG:boto:path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:55.996 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:55.996 DEBUG:boto:Method: HEAD 2022-09-12T19:06:55.997 DEBUG:boto:Path: /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:55.997 DEBUG:boto:Data: 2022-09-12T19:06:55.998 DEBUG:boto:Headers: {} 2022-09-12T19:06:55.999 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:55.999 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.000 DEBUG:boto:Params: {} 2022-09-12T19:06:56.001 DEBUG:boto:Token: None 2022-09-12T19:06:56.001 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:56.002 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oHt98GO5tzuJLAWggCdGKKLWaXo= 2022-09-12T19:06:56.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oHt98GO5tzuJLAWggCdGKKLWaXo=', 'Content-Length': '0'} 2022-09-12T19:06:56.005 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'QHXOu7SGV87vItdBeqaL.hYUlX--Ty8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1cf56aedf4d0d1a-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.006 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.007 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.008 DEBUG:boto:path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:56.009 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:56.010 DEBUG:boto:Method: HEAD 2022-09-12T19:06:56.011 DEBUG:boto:Path: /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:56.012 DEBUG:boto:Data: 2022-09-12T19:06:56.013 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.013 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.014 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.014 DEBUG:boto:Params: {} 2022-09-12T19:06:56.015 DEBUG:boto:Token: None 2022-09-12T19:06:56.016 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:56.017 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oHt98GO5tzuJLAWggCdGKKLWaXo= 2022-09-12T19:06:56.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oHt98GO5tzuJLAWggCdGKKLWaXo=', 'Content-Length': '0'} 2022-09-12T19:06:56.021 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'QHXOu7SGV87vItdBeqaL.hYUlX--Ty8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d85955504085b8f-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:56.023 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.023 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.024 DEBUG:boto:path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:56.024 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:56.025 DEBUG:boto:Method: GET 2022-09-12T19:06:56.026 DEBUG:boto:Path: /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:56.026 DEBUG:boto:Data: 2022-09-12T19:06:56.027 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.027 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.028 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.028 DEBUG:boto:Params: {} 2022-09-12T19:06:56.029 DEBUG:boto:Token: None 2022-09-12T19:06:56.029 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:56.030 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wsfaitRMgc2hrtNfgFjrJWO3BEA= 2022-09-12T19:06:56.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wsfaitRMgc2hrtNfgFjrJWO3BEA=', 'Content-Length': '0'} 2022-09-12T19:06:56.033 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'QHXOu7SGV87vItdBeqaL.hYUlX--Ty8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d8e861a19d3ca22-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.033 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.034 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.035 DEBUG:boto:path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:56.035 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:56.036 DEBUG:boto:Method: GET 2022-09-12T19:06:56.036 DEBUG:boto:Path: /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:56.037 DEBUG:boto:Data: 2022-09-12T19:06:56.037 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.038 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.038 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.039 DEBUG:boto:Params: {} 2022-09-12T19:06:56.040 DEBUG:boto:Token: None 2022-09-12T19:06:56.040 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:06:56.041 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wsfaitRMgc2hrtNfgFjrJWO3BEA= 2022-09-12T19:06:56.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wsfaitRMgc2hrtNfgFjrJWO3BEA=', 'Content-Length': '0'} 2022-09-12T19:06:56.044 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'QHXOu7SGV87vItdBeqaL.hYUlX--Ty8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000667c12291d29ec37-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.044 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:56.045 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:56.046 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.046 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.047 DEBUG:boto:Method: GET 2022-09-12T19:06:56.047 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:56.048 DEBUG:boto:Data: 2022-09-12T19:06:56.049 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.049 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.050 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.050 DEBUG:boto:Params: {} 2022-09-12T19:06:56.051 DEBUG:boto:Token: None 2022-09-12T19:06:56.051 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/ 2022-09-12T19:06:56.052 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0= 2022-09-12T19:06:56.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0=', 'Content-Length': '0'} 2022-09-12T19:06:56.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044711c8daa4ff633-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.056 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:56.057 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:56.058 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:56.058 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.059 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.060 DEBUG:boto:Method: GET 2022-09-12T19:06:56.060 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:56.061 DEBUG:boto:Data: 2022-09-12T19:06:56.061 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.062 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.062 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.063 DEBUG:boto:Params: {} 2022-09-12T19:06:56.064 DEBUG:boto:Token: None 2022-09-12T19:06:56.064 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/ 2022-09-12T19:06:56.065 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0= 2022-09-12T19:06:56.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0=', 'Content-Length': '0'} 2022-09-12T19:06:56.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1ccc9d2f07cb7ae-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.069 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:56.070 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.071 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.071 DEBUG:boto:Method: HEAD 2022-09-12T19:06:56.072 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:56.072 DEBUG:boto:Data: 2022-09-12T19:06:56.073 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.073 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.074 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.074 DEBUG:boto:Params: {} 2022-09-12T19:06:56.075 DEBUG:boto:Token: None 2022-09-12T19:06:56.075 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj 2022-09-12T19:06:56.076 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G0EsLocFA3wL8nym5dgrQtYnvUA= 2022-09-12T19:06:56.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G0EsLocFA3wL8nym5dgrQtYnvUA=', 'Content-Length': '0'} 2022-09-12T19:06:56.080 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b7a0aaa44bc608f-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.081 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.081 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.082 DEBUG:boto:Method: HEAD 2022-09-12T19:06:56.082 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:56.083 DEBUG:boto:Data: 2022-09-12T19:06:56.083 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.084 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.084 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.085 DEBUG:boto:Params: {} 2022-09-12T19:06:56.085 DEBUG:boto:Token: None 2022-09-12T19:06:56.086 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj 2022-09-12T19:06:56.087 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G0EsLocFA3wL8nym5dgrQtYnvUA= 2022-09-12T19:06:56.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G0EsLocFA3wL8nym5dgrQtYnvUA=', 'Content-Length': '0'} 2022-09-12T19:06:56.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090e6c2357817e423-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:56.104 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.105 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.105 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.106 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.106 DEBUG:boto:Method: GET 2022-09-12T19:06:56.107 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.107 DEBUG:boto:Data: 2022-09-12T19:06:56.108 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.108 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.109 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.109 DEBUG:boto:Params: {} 2022-09-12T19:06:56.110 DEBUG:boto:Token: None 2022-09-12T19:06:56.111 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.111 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NLPDHfOQZES/zo4l4ir+0s3MJh0= 2022-09-12T19:06:56.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NLPDHfOQZES/zo4l4ir+0s3MJh0=', 'Content-Length': '0'} 2022-09-12T19:06:56.114 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000074c51ca39ea3b1c7-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.158 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.159 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.159 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.160 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.160 DEBUG:boto:Method: GET 2022-09-12T19:06:56.161 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.162 DEBUG:boto:Data: 2022-09-12T19:06:56.162 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.163 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.163 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.164 DEBUG:boto:Params: {} 2022-09-12T19:06:56.164 DEBUG:boto:Token: None 2022-09-12T19:06:56.165 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.166 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NLPDHfOQZES/zo4l4ir+0s3MJh0= 2022-09-12T19:06:56.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NLPDHfOQZES/zo4l4ir+0s3MJh0=', 'Content-Length': '0'} 2022-09-12T19:06:56.171 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031db1c0908af6c09-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.172 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:56.173 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:56.174 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.175 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.176 DEBUG:boto:Method: GET 2022-09-12T19:06:56.176 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:56.177 DEBUG:boto:Data: 2022-09-12T19:06:56.178 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.179 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.179 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.180 DEBUG:boto:Params: {} 2022-09-12T19:06:56.181 DEBUG:boto:Token: None 2022-09-12T19:06:56.182 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/ 2022-09-12T19:06:56.183 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0= 2022-09-12T19:06:56.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0=', 'Content-Length': '0'} 2022-09-12T19:06:56.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000844b00189ebce3d2-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.189 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:56.190 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:56.190 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:56.191 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.192 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.192 DEBUG:boto:Method: GET 2022-09-12T19:06:56.193 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:56.193 DEBUG:boto:Data: 2022-09-12T19:06:56.194 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.194 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.195 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.195 DEBUG:boto:Params: {} 2022-09-12T19:06:56.196 DEBUG:boto:Token: None 2022-09-12T19:06:56.197 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/ 2022-09-12T19:06:56.197 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0= 2022-09-12T19:06:56.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0=', 'Content-Length': '0'} 2022-09-12T19:06:56.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ea7376aadb2897e-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.202 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:56.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:56.204 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.204 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.205 DEBUG:boto:path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:56.205 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:56.206 DEBUG:boto:Method: GET 2022-09-12T19:06:56.206 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:56.207 DEBUG:boto:Data: 2022-09-12T19:06:56.207 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.208 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.208 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.209 DEBUG:boto:Params: {} 2022-09-12T19:06:56.210 DEBUG:boto:Token: None 2022-09-12T19:06:56.211 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:56.212 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l6mirf17kFEQteRqRVAriI/Xk+U= 2022-09-12T19:06:56.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l6mirf17kFEQteRqRVAriI/Xk+U=', 'Content-Length': '0'} 2022-09-12T19:06:56.215 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000006c8331a68082e4-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.217 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.217 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.218 DEBUG:boto:path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:56.219 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:56.220 DEBUG:boto:Method: GET 2022-09-12T19:06:56.221 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:56.222 DEBUG:boto:Data: 2022-09-12T19:06:56.223 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.223 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.224 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.225 DEBUG:boto:Params: {} 2022-09-12T19:06:56.225 DEBUG:boto:Token: None 2022-09-12T19:06:56.226 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:56.227 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l6mirf17kFEQteRqRVAriI/Xk+U= 2022-09-12T19:06:56.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l6mirf17kFEQteRqRVAriI/Xk+U=', 'Content-Length': '0'} 2022-09-12T19:06:56.229 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a05debcc6aedf0a-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.230 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.231 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.231 DEBUG:boto:path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:56.232 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:56.232 DEBUG:boto:Method: HEAD 2022-09-12T19:06:56.233 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:56.233 DEBUG:boto:Data: 2022-09-12T19:06:56.234 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.234 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.235 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.235 DEBUG:boto:Params: {} 2022-09-12T19:06:56.236 DEBUG:boto:Token: None 2022-09-12T19:06:56.236 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:56.237 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GNa466yQGNCy5SMM18V2mTi/Fxo= 2022-09-12T19:06:56.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GNa466yQGNCy5SMM18V2mTi/Fxo=', 'Content-Length': '0'} 2022-09-12T19:06:56.240 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059358fdb321a4a0d-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.241 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.241 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.242 DEBUG:boto:path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:56.242 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:56.243 DEBUG:boto:Method: HEAD 2022-09-12T19:06:56.243 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:56.244 DEBUG:boto:Data: 2022-09-12T19:06:56.244 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.245 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.245 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.246 DEBUG:boto:Params: {} 2022-09-12T19:06:56.247 DEBUG:boto:Token: None 2022-09-12T19:06:56.248 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:56.249 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GNa466yQGNCy5SMM18V2mTi/Fxo= 2022-09-12T19:06:56.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GNa466yQGNCy5SMM18V2mTi/Fxo=', 'Content-Length': '0'} 2022-09-12T19:06:56.253 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076f1376cd1188ec1-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:56.254 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.255 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.256 DEBUG:boto:path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:56.257 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:56.257 DEBUG:boto:Method: GET 2022-09-12T19:06:56.258 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:56.258 DEBUG:boto:Data: 2022-09-12T19:06:56.259 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.259 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.260 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.260 DEBUG:boto:Params: {} 2022-09-12T19:06:56.261 DEBUG:boto:Token: None 2022-09-12T19:06:56.262 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:56.262 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l6mirf17kFEQteRqRVAriI/Xk+U= 2022-09-12T19:06:56.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l6mirf17kFEQteRqRVAriI/Xk+U=', 'Content-Length': '0'} 2022-09-12T19:06:56.265 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001051f3e8a780cd18-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.266 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.266 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.267 DEBUG:boto:path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:56.267 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:56.268 DEBUG:boto:Method: GET 2022-09-12T19:06:56.269 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:56.269 DEBUG:boto:Data: 2022-09-12T19:06:56.270 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.270 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.271 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.271 DEBUG:boto:Params: {} 2022-09-12T19:06:56.272 DEBUG:boto:Token: None 2022-09-12T19:06:56.272 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:06:56.273 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l6mirf17kFEQteRqRVAriI/Xk+U= 2022-09-12T19:06:56.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l6mirf17kFEQteRqRVAriI/Xk+U=', 'Content-Length': '0'} 2022-09-12T19:06:56.276 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d8654b6a62f365e-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.277 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:56.278 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:56.279 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.279 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.280 DEBUG:boto:Method: GET 2022-09-12T19:06:56.280 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:56.281 DEBUG:boto:Data: 2022-09-12T19:06:56.281 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.282 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.282 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.283 DEBUG:boto:Params: {} 2022-09-12T19:06:56.283 DEBUG:boto:Token: None 2022-09-12T19:06:56.284 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/ 2022-09-12T19:06:56.284 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0= 2022-09-12T19:06:56.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0=', 'Content-Length': '0'} 2022-09-12T19:06:56.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fae74b0fe248b61f-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.289 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:56.289 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:56.290 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:56.290 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.291 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.291 DEBUG:boto:Method: GET 2022-09-12T19:06:56.292 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:56.292 DEBUG:boto:Data: 2022-09-12T19:06:56.293 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.293 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.294 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.295 DEBUG:boto:Params: {} 2022-09-12T19:06:56.295 DEBUG:boto:Token: None 2022-09-12T19:06:56.296 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/ 2022-09-12T19:06:56.296 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0= 2022-09-12T19:06:56.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0=', 'Content-Length': '0'} 2022-09-12T19:06:56.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000036554e2d5bab61a-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.301 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:56.301 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.302 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.302 DEBUG:boto:Method: HEAD 2022-09-12T19:06:56.303 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:56.303 DEBUG:boto:Data: 2022-09-12T19:06:56.304 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.304 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.305 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.305 DEBUG:boto:Params: {} 2022-09-12T19:06:56.306 DEBUG:boto:Token: None 2022-09-12T19:06:56.306 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj 2022-09-12T19:06:56.307 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G0EsLocFA3wL8nym5dgrQtYnvUA= 2022-09-12T19:06:56.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G0EsLocFA3wL8nym5dgrQtYnvUA=', 'Content-Length': '0'} 2022-09-12T19:06:56.311 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086a02d7b61956c2f-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.312 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.313 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.313 DEBUG:boto:Method: HEAD 2022-09-12T19:06:56.314 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:56.314 DEBUG:boto:Data: 2022-09-12T19:06:56.315 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.315 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.316 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.316 DEBUG:boto:Params: {} 2022-09-12T19:06:56.317 DEBUG:boto:Token: None 2022-09-12T19:06:56.318 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj 2022-09-12T19:06:56.318 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G0EsLocFA3wL8nym5dgrQtYnvUA= 2022-09-12T19:06:56.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G0EsLocFA3wL8nym5dgrQtYnvUA=', 'Content-Length': '0'} 2022-09-12T19:06:56.323 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ae8056301d5a20d-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:56.324 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.324 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.325 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.325 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.326 DEBUG:boto:Method: GET 2022-09-12T19:06:56.327 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.327 DEBUG:boto:Data: 2022-09-12T19:06:56.328 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.328 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.329 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.329 DEBUG:boto:Params: {} 2022-09-12T19:06:56.330 DEBUG:boto:Token: None 2022-09-12T19:06:56.331 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.331 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NLPDHfOQZES/zo4l4ir+0s3MJh0= 2022-09-12T19:06:56.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NLPDHfOQZES/zo4l4ir+0s3MJh0=', 'Content-Length': '0'} 2022-09-12T19:06:56.334 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d42c73d11420d789-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.378 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.379 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.379 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.380 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.380 DEBUG:boto:Method: GET 2022-09-12T19:06:56.381 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.381 DEBUG:boto:Data: 2022-09-12T19:06:56.382 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.382 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.383 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.384 DEBUG:boto:Params: {} 2022-09-12T19:06:56.384 DEBUG:boto:Token: None 2022-09-12T19:06:56.385 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.386 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NLPDHfOQZES/zo4l4ir+0s3MJh0= 2022-09-12T19:06:56.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NLPDHfOQZES/zo4l4ir+0s3MJh0=', 'Content-Length': '0'} 2022-09-12T19:06:56.390 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024796cd3adca5858-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.391 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:56.392 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:56.393 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.394 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.395 DEBUG:boto:Method: GET 2022-09-12T19:06:56.395 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:56.396 DEBUG:boto:Data: 2022-09-12T19:06:56.396 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.397 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.397 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.398 DEBUG:boto:Params: {} 2022-09-12T19:06:56.399 DEBUG:boto:Token: None 2022-09-12T19:06:56.399 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/ 2022-09-12T19:06:56.400 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0= 2022-09-12T19:06:56.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0=', 'Content-Length': '0'} 2022-09-12T19:06:56.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ceb56be696c4688-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.405 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:56.406 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:56.407 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:56.407 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.408 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.408 DEBUG:boto:Method: GET 2022-09-12T19:06:56.409 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:56.409 DEBUG:boto:Data: 2022-09-12T19:06:56.410 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.410 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.411 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.412 DEBUG:boto:Params: {} 2022-09-12T19:06:56.412 DEBUG:boto:Token: None 2022-09-12T19:06:56.413 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/ 2022-09-12T19:06:56.414 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0= 2022-09-12T19:06:56.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0=', 'Content-Length': '0'} 2022-09-12T19:06:56.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fb309a547b0dc81-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.419 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:56.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:56.421 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.421 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.422 DEBUG:boto:path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:56.422 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:56.423 DEBUG:boto:Method: GET 2022-09-12T19:06:56.423 DEBUG:boto:Path: /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:56.424 DEBUG:boto:Data: 2022-09-12T19:06:56.424 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.425 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.426 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.426 DEBUG:boto:Params: {} 2022-09-12T19:06:56.427 DEBUG:boto:Token: None 2022-09-12T19:06:56.427 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:56.428 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DS7x+bMHp0nOkIXblcJgVn0a1yI= 2022-09-12T19:06:56.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DS7x+bMHp0nOkIXblcJgVn0a1yI=', 'Content-Length': '0'} 2022-09-12T19:06:56.431 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fad65bca32f2fb77-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.431 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.432 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.432 DEBUG:boto:path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:56.433 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:56.434 DEBUG:boto:Method: GET 2022-09-12T19:06:56.434 DEBUG:boto:Path: /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:56.435 DEBUG:boto:Data: 2022-09-12T19:06:56.435 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.436 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.436 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.437 DEBUG:boto:Params: {} 2022-09-12T19:06:56.437 DEBUG:boto:Token: None 2022-09-12T19:06:56.438 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:56.439 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DS7x+bMHp0nOkIXblcJgVn0a1yI= 2022-09-12T19:06:56.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DS7x+bMHp0nOkIXblcJgVn0a1yI=', 'Content-Length': '0'} 2022-09-12T19:06:56.441 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006351b3be849a1eaf-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.482 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.483 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.484 DEBUG:boto:path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:56.485 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:56.486 DEBUG:boto:Method: HEAD 2022-09-12T19:06:56.486 DEBUG:boto:Path: /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:56.487 DEBUG:boto:Data: 2022-09-12T19:06:56.487 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.488 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.488 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.489 DEBUG:boto:Params: {} 2022-09-12T19:06:56.489 DEBUG:boto:Token: None 2022-09-12T19:06:56.490 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:56.491 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G6rWgrJqt1yXWBrx3+wY2Xrj0Iw= 2022-09-12T19:06:56.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G6rWgrJqt1yXWBrx3+wY2Xrj0Iw=', 'Content-Length': '0'} 2022-09-12T19:06:56.494 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097b9cb46eebc7eb2-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.494 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.495 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.496 DEBUG:boto:path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:56.496 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:56.497 DEBUG:boto:Method: HEAD 2022-09-12T19:06:56.497 DEBUG:boto:Path: /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:56.498 DEBUG:boto:Data: 2022-09-12T19:06:56.498 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.499 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.499 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.500 DEBUG:boto:Params: {} 2022-09-12T19:06:56.500 DEBUG:boto:Token: None 2022-09-12T19:06:56.501 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:56.502 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G6rWgrJqt1yXWBrx3+wY2Xrj0Iw= 2022-09-12T19:06:56.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G6rWgrJqt1yXWBrx3+wY2Xrj0Iw=', 'Content-Length': '0'} 2022-09-12T19:06:56.504 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ee4e0b8ed29fe97-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:56.506 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.506 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.507 DEBUG:boto:path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:56.508 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:56.508 DEBUG:boto:Method: GET 2022-09-12T19:06:56.509 DEBUG:boto:Path: /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:56.509 DEBUG:boto:Data: 2022-09-12T19:06:56.510 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.511 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.511 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.512 DEBUG:boto:Params: {} 2022-09-12T19:06:56.512 DEBUG:boto:Token: None 2022-09-12T19:06:56.513 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:56.514 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DS7x+bMHp0nOkIXblcJgVn0a1yI= 2022-09-12T19:06:56.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DS7x+bMHp0nOkIXblcJgVn0a1yI=', 'Content-Length': '0'} 2022-09-12T19:06:56.516 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c66dec7b1bcca894-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.517 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.518 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.518 DEBUG:boto:path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:56.519 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:56.519 DEBUG:boto:Method: GET 2022-09-12T19:06:56.520 DEBUG:boto:Path: /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:56.520 DEBUG:boto:Data: 2022-09-12T19:06:56.521 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.522 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.522 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.523 DEBUG:boto:Params: {} 2022-09-12T19:06:56.523 DEBUG:boto:Token: None 2022-09-12T19:06:56.524 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:06:56.525 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DS7x+bMHp0nOkIXblcJgVn0a1yI= 2022-09-12T19:06:56.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DS7x+bMHp0nOkIXblcJgVn0a1yI=', 'Content-Length': '0'} 2022-09-12T19:06:56.527 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e5be4f7ecbcd121-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.528 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:56.529 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:56.529 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.530 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.530 DEBUG:boto:Method: GET 2022-09-12T19:06:56.531 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:56.532 DEBUG:boto:Data: 2022-09-12T19:06:56.533 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.534 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.535 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.536 DEBUG:boto:Params: {} 2022-09-12T19:06:56.537 DEBUG:boto:Token: None 2022-09-12T19:06:56.538 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/ 2022-09-12T19:06:56.539 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0= 2022-09-12T19:06:56.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0=', 'Content-Length': '0'} 2022-09-12T19:06:56.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008454a0f90ed84641-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.544 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:56.545 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:56.546 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:56.546 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.547 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.547 DEBUG:boto:Method: GET 2022-09-12T19:06:56.548 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:56.548 DEBUG:boto:Data: 2022-09-12T19:06:56.549 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.550 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.550 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.551 DEBUG:boto:Params: {} 2022-09-12T19:06:56.551 DEBUG:boto:Token: None 2022-09-12T19:06:56.552 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/ 2022-09-12T19:06:56.553 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0= 2022-09-12T19:06:56.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0=', 'Content-Length': '0'} 2022-09-12T19:06:56.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002efca336dc8d0664-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.557 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:56.558 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.559 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.559 DEBUG:boto:Method: HEAD 2022-09-12T19:06:56.560 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:56.560 DEBUG:boto:Data: 2022-09-12T19:06:56.561 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.561 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.562 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.562 DEBUG:boto:Params: {} 2022-09-12T19:06:56.563 DEBUG:boto:Token: None 2022-09-12T19:06:56.563 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj 2022-09-12T19:06:56.564 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G0EsLocFA3wL8nym5dgrQtYnvUA= 2022-09-12T19:06:56.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G0EsLocFA3wL8nym5dgrQtYnvUA=', 'Content-Length': '0'} 2022-09-12T19:06:56.569 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3324823e775b958-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.571 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.571 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.572 DEBUG:boto:Method: HEAD 2022-09-12T19:06:56.573 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:56.574 DEBUG:boto:Data: 2022-09-12T19:06:56.575 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.575 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.576 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.576 DEBUG:boto:Params: {} 2022-09-12T19:06:56.577 DEBUG:boto:Token: None 2022-09-12T19:06:56.577 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj 2022-09-12T19:06:56.578 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G0EsLocFA3wL8nym5dgrQtYnvUA= 2022-09-12T19:06:56.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G0EsLocFA3wL8nym5dgrQtYnvUA=', 'Content-Length': '0'} 2022-09-12T19:06:56.582 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009696dbd83a1dbc17-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:56.583 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.584 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.584 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.585 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.586 DEBUG:boto:Method: GET 2022-09-12T19:06:56.587 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.588 DEBUG:boto:Data: 2022-09-12T19:06:56.589 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.590 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.591 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.592 DEBUG:boto:Params: {} 2022-09-12T19:06:56.593 DEBUG:boto:Token: None 2022-09-12T19:06:56.594 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.595 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NLPDHfOQZES/zo4l4ir+0s3MJh0= 2022-09-12T19:06:56.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NLPDHfOQZES/zo4l4ir+0s3MJh0=', 'Content-Length': '0'} 2022-09-12T19:06:56.599 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034d0d69da2315b7f-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.642 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.643 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.643 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.644 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.644 DEBUG:boto:Method: GET 2022-09-12T19:06:56.645 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.645 DEBUG:boto:Data: 2022-09-12T19:06:56.646 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.646 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.647 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.648 DEBUG:boto:Params: {} 2022-09-12T19:06:56.648 DEBUG:boto:Token: None 2022-09-12T19:06:56.649 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.649 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NLPDHfOQZES/zo4l4ir+0s3MJh0= 2022-09-12T19:06:56.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NLPDHfOQZES/zo4l4ir+0s3MJh0=', 'Content-Length': '0'} 2022-09-12T19:06:56.652 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d99883cbf1a0fa6-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.653 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:56.654 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:56.657 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.658 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.659 DEBUG:boto:Method: GET 2022-09-12T19:06:56.659 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:56.660 DEBUG:boto:Data: 2022-09-12T19:06:56.660 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.661 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.662 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.663 DEBUG:boto:Params: {} 2022-09-12T19:06:56.663 DEBUG:boto:Token: None 2022-09-12T19:06:56.664 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/ 2022-09-12T19:06:56.665 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0= 2022-09-12T19:06:56.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0=', 'Content-Length': '0'} 2022-09-12T19:06:56.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecf0920a18491a10-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.670 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:56.671 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:56.671 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:56.672 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.672 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.673 DEBUG:boto:Method: GET 2022-09-12T19:06:56.673 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:56.674 DEBUG:boto:Data: 2022-09-12T19:06:56.674 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.675 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.675 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.676 DEBUG:boto:Params: {} 2022-09-12T19:06:56.676 DEBUG:boto:Token: None 2022-09-12T19:06:56.677 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/ 2022-09-12T19:06:56.678 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0= 2022-09-12T19:06:56.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0=', 'Content-Length': '0'} 2022-09-12T19:06:56.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009a28e5677d3b409-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.682 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:56.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:56.683 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.684 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.685 DEBUG:boto:path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:56.685 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:56.686 DEBUG:boto:Method: GET 2022-09-12T19:06:56.686 DEBUG:boto:Path: /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:56.687 DEBUG:boto:Data: 2022-09-12T19:06:56.687 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.688 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.688 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.689 DEBUG:boto:Params: {} 2022-09-12T19:06:56.689 DEBUG:boto:Token: None 2022-09-12T19:06:56.690 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:56.691 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NV7n2/5ruM9ks1OG8fklTzvXKyk= 2022-09-12T19:06:56.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NV7n2/5ruM9ks1OG8fklTzvXKyk=', 'Content-Length': '0'} 2022-09-12T19:06:56.693 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'BAbppx4RylEKnBrYesXt1QPJgsAhIBb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f852e685b3f10596-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.694 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.695 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.696 DEBUG:boto:path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:56.696 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:56.697 DEBUG:boto:Method: GET 2022-09-12T19:06:56.697 DEBUG:boto:Path: /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:56.698 DEBUG:boto:Data: 2022-09-12T19:06:56.698 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.699 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.700 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.700 DEBUG:boto:Params: {} 2022-09-12T19:06:56.701 DEBUG:boto:Token: None 2022-09-12T19:06:56.702 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:56.703 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NV7n2/5ruM9ks1OG8fklTzvXKyk= 2022-09-12T19:06:56.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NV7n2/5ruM9ks1OG8fklTzvXKyk=', 'Content-Length': '0'} 2022-09-12T19:06:56.705 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'BAbppx4RylEKnBrYesXt1QPJgsAhIBb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076b2bf7047a3aa04-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.706 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.707 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.708 DEBUG:boto:path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:56.708 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:56.709 DEBUG:boto:Method: HEAD 2022-09-12T19:06:56.709 DEBUG:boto:Path: /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:56.710 DEBUG:boto:Data: 2022-09-12T19:06:56.710 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.711 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.711 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.712 DEBUG:boto:Params: {} 2022-09-12T19:06:56.712 DEBUG:boto:Token: None 2022-09-12T19:06:56.713 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:56.714 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AdRzvQVbeNl93v4YCQd0GfH4dWk= 2022-09-12T19:06:56.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AdRzvQVbeNl93v4YCQd0GfH4dWk=', 'Content-Length': '0'} 2022-09-12T19:06:56.716 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'BAbppx4RylEKnBrYesXt1QPJgsAhIBb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e5fe520f9f9d7ff-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.717 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.718 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.718 DEBUG:boto:path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:56.719 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:56.719 DEBUG:boto:Method: HEAD 2022-09-12T19:06:56.720 DEBUG:boto:Path: /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:56.720 DEBUG:boto:Data: 2022-09-12T19:06:56.721 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.721 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.722 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.722 DEBUG:boto:Params: {} 2022-09-12T19:06:56.723 DEBUG:boto:Token: None 2022-09-12T19:06:56.723 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:56.724 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AdRzvQVbeNl93v4YCQd0GfH4dWk= 2022-09-12T19:06:56.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AdRzvQVbeNl93v4YCQd0GfH4dWk=', 'Content-Length': '0'} 2022-09-12T19:06:56.727 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'BAbppx4RylEKnBrYesXt1QPJgsAhIBb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000623df2d1992b1212-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:56.728 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.729 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.730 DEBUG:boto:path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:56.730 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:56.731 DEBUG:boto:Method: GET 2022-09-12T19:06:56.731 DEBUG:boto:Path: /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:56.732 DEBUG:boto:Data: 2022-09-12T19:06:56.733 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.733 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.734 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.734 DEBUG:boto:Params: {} 2022-09-12T19:06:56.735 DEBUG:boto:Token: None 2022-09-12T19:06:56.736 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:56.736 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NV7n2/5ruM9ks1OG8fklTzvXKyk= 2022-09-12T19:06:56.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NV7n2/5ruM9ks1OG8fklTzvXKyk=', 'Content-Length': '0'} 2022-09-12T19:06:56.739 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'BAbppx4RylEKnBrYesXt1QPJgsAhIBb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0b50e799ac2143f-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.740 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.740 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.741 DEBUG:boto:path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:56.741 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:56.742 DEBUG:boto:Method: GET 2022-09-12T19:06:56.742 DEBUG:boto:Path: /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:56.743 DEBUG:boto:Data: 2022-09-12T19:06:56.743 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.744 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.744 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.745 DEBUG:boto:Params: {} 2022-09-12T19:06:56.746 DEBUG:boto:Token: None 2022-09-12T19:06:56.746 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:06:56.747 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NV7n2/5ruM9ks1OG8fklTzvXKyk= 2022-09-12T19:06:56.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NV7n2/5ruM9ks1OG8fklTzvXKyk=', 'Content-Length': '0'} 2022-09-12T19:06:56.749 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'BAbppx4RylEKnBrYesXt1QPJgsAhIBb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e2c778dba0a02b6-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.750 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:56.751 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:56.751 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.752 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.752 DEBUG:boto:Method: GET 2022-09-12T19:06:56.753 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:56.753 DEBUG:boto:Data: 2022-09-12T19:06:56.754 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.754 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.755 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.756 DEBUG:boto:Params: {} 2022-09-12T19:06:56.756 DEBUG:boto:Token: None 2022-09-12T19:06:56.757 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/ 2022-09-12T19:06:56.757 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0= 2022-09-12T19:06:56.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0=', 'Content-Length': '0'} 2022-09-12T19:06:56.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e23584a2c67c0ba-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.762 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:56.763 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:56.763 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:56.764 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.764 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.765 DEBUG:boto:Method: GET 2022-09-12T19:06:56.765 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:56.766 DEBUG:boto:Data: 2022-09-12T19:06:56.767 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.767 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.768 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.768 DEBUG:boto:Params: {} 2022-09-12T19:06:56.769 DEBUG:boto:Token: None 2022-09-12T19:06:56.769 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/ 2022-09-12T19:06:56.770 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0= 2022-09-12T19:06:56.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0=', 'Content-Length': '0'} 2022-09-12T19:06:56.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a38b8980adb53596-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.775 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:56.775 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.776 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.776 DEBUG:boto:Method: HEAD 2022-09-12T19:06:56.777 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:56.778 DEBUG:boto:Data: 2022-09-12T19:06:56.778 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.779 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.779 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.780 DEBUG:boto:Params: {} 2022-09-12T19:06:56.780 DEBUG:boto:Token: None 2022-09-12T19:06:56.781 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj 2022-09-12T19:06:56.781 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G0EsLocFA3wL8nym5dgrQtYnvUA= 2022-09-12T19:06:56.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G0EsLocFA3wL8nym5dgrQtYnvUA=', 'Content-Length': '0'} 2022-09-12T19:06:56.786 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f3d815b60f9883b-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.787 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.788 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.788 DEBUG:boto:Method: HEAD 2022-09-12T19:06:56.789 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:56.789 DEBUG:boto:Data: 2022-09-12T19:06:56.790 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.791 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.792 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.793 DEBUG:boto:Params: {} 2022-09-12T19:06:56.793 DEBUG:boto:Token: None 2022-09-12T19:06:56.794 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj 2022-09-12T19:06:56.795 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G0EsLocFA3wL8nym5dgrQtYnvUA= 2022-09-12T19:06:56.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G0EsLocFA3wL8nym5dgrQtYnvUA=', 'Content-Length': '0'} 2022-09-12T19:06:56.799 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec8db6aa6573d190-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:56.800 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.801 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.801 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.802 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.802 DEBUG:boto:Method: GET 2022-09-12T19:06:56.803 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.804 DEBUG:boto:Data: 2022-09-12T19:06:56.804 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.805 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.805 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.806 DEBUG:boto:Params: {} 2022-09-12T19:06:56.807 DEBUG:boto:Token: None 2022-09-12T19:06:56.807 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.808 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NLPDHfOQZES/zo4l4ir+0s3MJh0= 2022-09-12T19:06:56.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NLPDHfOQZES/zo4l4ir+0s3MJh0=', 'Content-Length': '0'} 2022-09-12T19:06:56.811 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1381cfe711266a1-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.812 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.812 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.813 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.813 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.814 DEBUG:boto:Method: GET 2022-09-12T19:06:56.814 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.815 DEBUG:boto:Data: 2022-09-12T19:06:56.816 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.816 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.817 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.817 DEBUG:boto:Params: {} 2022-09-12T19:06:56.818 DEBUG:boto:Token: None 2022-09-12T19:06:56.818 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:56.819 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NLPDHfOQZES/zo4l4ir+0s3MJh0= 2022-09-12T19:06:56.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NLPDHfOQZES/zo4l4ir+0s3MJh0=', 'Content-Length': '0'} 2022-09-12T19:06:56.822 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027d3a653d14b7ac8-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.866 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:56.867 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:56.868 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.869 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.870 DEBUG:boto:Method: GET 2022-09-12T19:06:56.871 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:56.872 DEBUG:boto:Data: 2022-09-12T19:06:56.873 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.874 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.875 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.876 DEBUG:boto:Params: {} 2022-09-12T19:06:56.877 DEBUG:boto:Token: None 2022-09-12T19:06:56.877 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/ 2022-09-12T19:06:56.878 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0= 2022-09-12T19:06:56.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0=', 'Content-Length': '0'} 2022-09-12T19:06:56.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9d6df01ac3d4864-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.884 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:56.886 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:56.887 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:56.888 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.889 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:56.890 DEBUG:boto:Method: GET 2022-09-12T19:06:56.891 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:56.892 DEBUG:boto:Data: 2022-09-12T19:06:56.894 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.895 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.896 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.896 DEBUG:boto:Params: {} 2022-09-12T19:06:56.897 DEBUG:boto:Token: None 2022-09-12T19:06:56.898 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/ 2022-09-12T19:06:56.898 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0= 2022-09-12T19:06:56.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FDunu/kvPYlg9aID0UswGsKScg0=', 'Content-Length': '0'} 2022-09-12T19:06:56.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cce845fa170d9f4-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.903 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:56.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:56.904 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.905 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.905 DEBUG:boto:path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:56.906 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:56.907 DEBUG:boto:Method: GET 2022-09-12T19:06:56.907 DEBUG:boto:Path: /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:56.908 DEBUG:boto:Data: 2022-09-12T19:06:56.908 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.909 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.909 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.910 DEBUG:boto:Params: {} 2022-09-12T19:06:56.911 DEBUG:boto:Token: None 2022-09-12T19:06:56.911 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:56.912 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Hf1haSr8Li3o4TcnSYYnGM+rmvQ= 2022-09-12T19:06:56.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Hf1haSr8Li3o4TcnSYYnGM+rmvQ=', 'Content-Length': '0'} 2022-09-12T19:06:56.915 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000571c6b14b904965-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.958 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.959 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.959 DEBUG:boto:path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:56.960 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:56.960 DEBUG:boto:Method: GET 2022-09-12T19:06:56.961 DEBUG:boto:Path: /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:56.961 DEBUG:boto:Data: 2022-09-12T19:06:56.962 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.963 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.963 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.964 DEBUG:boto:Params: {} 2022-09-12T19:06:56.964 DEBUG:boto:Token: None 2022-09-12T19:06:56.965 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:56.966 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Hf1haSr8Li3o4TcnSYYnGM+rmvQ= 2022-09-12T19:06:56.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Hf1haSr8Li3o4TcnSYYnGM+rmvQ=', 'Content-Length': '0'} 2022-09-12T19:06:56.969 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb90d22a336c01e7-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.970 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.970 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.971 DEBUG:boto:path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:56.971 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:56.972 DEBUG:boto:Method: HEAD 2022-09-12T19:06:56.972 DEBUG:boto:Path: /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:56.973 DEBUG:boto:Data: 2022-09-12T19:06:56.973 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.974 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.975 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.975 DEBUG:boto:Params: {} 2022-09-12T19:06:56.976 DEBUG:boto:Token: None 2022-09-12T19:06:56.976 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:56.977 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tVkRb8R8lgAYOugQJfnLuyrev+E= 2022-09-12T19:06:56.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tVkRb8R8lgAYOugQJfnLuyrev+E=', 'Content-Length': '0'} 2022-09-12T19:06:56.980 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a01c43e3beebbbae-00631f8350-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.981 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.981 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.982 DEBUG:boto:path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:56.982 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:56.983 DEBUG:boto:Method: HEAD 2022-09-12T19:06:56.983 DEBUG:boto:Path: /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:56.984 DEBUG:boto:Data: 2022-09-12T19:06:56.984 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.985 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:56.986 DEBUG:boto:Port: 8000 2022-09-12T19:06:56.986 DEBUG:boto:Params: {} 2022-09-12T19:06:56.987 DEBUG:boto:Token: None 2022-09-12T19:06:56.987 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:56.988 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tVkRb8R8lgAYOugQJfnLuyrev+E= 2022-09-12T19:06:56.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tVkRb8R8lgAYOugQJfnLuyrev+E=', 'Content-Length': '0'} 2022-09-12T19:06:56.991 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d7993e9fbd641fe-00631f8350-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:56.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:56.992 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:56.993 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:56.993 DEBUG:boto:path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:56.994 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:56.995 DEBUG:boto:Method: GET 2022-09-12T19:06:56.995 DEBUG:boto:Path: /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:56.996 DEBUG:boto:Data: 2022-09-12T19:06:56.996 DEBUG:boto:Headers: {} 2022-09-12T19:06:56.997 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:56.997 DEBUG:boto:Port: 8001 2022-09-12T19:06:56.998 DEBUG:boto:Params: {} 2022-09-12T19:06:56.999 DEBUG:boto:Token: None 2022-09-12T19:06:56.999 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:56 GMT /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:57.000 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Hf1haSr8Li3o4TcnSYYnGM+rmvQ= 2022-09-12T19:06:57.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Hf1haSr8Li3o4TcnSYYnGM+rmvQ=', 'Content-Length': '0'} 2022-09-12T19:06:57.003 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a74d56acefb7ca8-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.004 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.004 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.005 DEBUG:boto:path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:57.005 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:57.006 DEBUG:boto:Method: GET 2022-09-12T19:06:57.006 DEBUG:boto:Path: /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:57.007 DEBUG:boto:Data: 2022-09-12T19:06:57.008 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.009 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.009 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.010 DEBUG:boto:Params: {} 2022-09-12T19:06:57.011 DEBUG:boto:Token: None 2022-09-12T19:06:57.012 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:06:57.012 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zy87187F5fyt7LjLVXnsGpDtV5k= 2022-09-12T19:06:57.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zy87187F5fyt7LjLVXnsGpDtV5k=', 'Content-Length': '0'} 2022-09-12T19:06:57.015 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004aae11eaa97d807b-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.016 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:57.016 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:57.017 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.017 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.018 DEBUG:boto:Method: GET 2022-09-12T19:06:57.019 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:57.020 DEBUG:boto:Data: 2022-09-12T19:06:57.021 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.022 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.023 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.024 DEBUG:boto:Params: {} 2022-09-12T19:06:57.025 DEBUG:boto:Token: None 2022-09-12T19:06:57.025 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/ 2022-09-12T19:06:57.026 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM= 2022-09-12T19:06:57.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM=', 'Content-Length': '0'} 2022-09-12T19:06:57.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b09ab21c35de926e-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.030 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:57.031 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:57.032 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:57.032 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.033 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.033 DEBUG:boto:Method: GET 2022-09-12T19:06:57.034 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:57.034 DEBUG:boto:Data: 2022-09-12T19:06:57.035 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.035 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.036 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.036 DEBUG:boto:Params: {} 2022-09-12T19:06:57.037 DEBUG:boto:Token: None 2022-09-12T19:06:57.038 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/ 2022-09-12T19:06:57.038 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM= 2022-09-12T19:06:57.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM=', 'Content-Length': '0'} 2022-09-12T19:06:57.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024ac1097263ab98f-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.082 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:57.084 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.085 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.085 DEBUG:boto:Method: HEAD 2022-09-12T19:06:57.086 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:57.087 DEBUG:boto:Data: 2022-09-12T19:06:57.088 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.088 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.089 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.089 DEBUG:boto:Params: {} 2022-09-12T19:06:57.090 DEBUG:boto:Token: None 2022-09-12T19:06:57.091 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj 2022-09-12T19:06:57.092 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1sMtbQoDERmFKhk7PwGtbxek1S0= 2022-09-12T19:06:57.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1sMtbQoDERmFKhk7PwGtbxek1S0=', 'Content-Length': '0'} 2022-09-12T19:06:57.097 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e5b8be6df0f6fa0-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.098 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.099 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.099 DEBUG:boto:Method: HEAD 2022-09-12T19:06:57.100 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:57.100 DEBUG:boto:Data: 2022-09-12T19:06:57.101 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.102 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.102 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.103 DEBUG:boto:Params: {} 2022-09-12T19:06:57.103 DEBUG:boto:Token: None 2022-09-12T19:06:57.104 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj 2022-09-12T19:06:57.104 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1sMtbQoDERmFKhk7PwGtbxek1S0= 2022-09-12T19:06:57.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1sMtbQoDERmFKhk7PwGtbxek1S0=', 'Content-Length': '0'} 2022-09-12T19:06:57.109 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004c724d3c969262f-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:57.111 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.112 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.113 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.114 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.115 DEBUG:boto:Method: GET 2022-09-12T19:06:57.116 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.117 DEBUG:boto:Data: 2022-09-12T19:06:57.118 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.119 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.120 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.121 DEBUG:boto:Params: {} 2022-09-12T19:06:57.122 DEBUG:boto:Token: None 2022-09-12T19:06:57.123 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.124 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OEatJPdbrF2sgqd6SE8bUl/BWa4= 2022-09-12T19:06:57.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OEatJPdbrF2sgqd6SE8bUl/BWa4=', 'Content-Length': '0'} 2022-09-12T19:06:57.128 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e923e8484441be16-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.130 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.131 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.132 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.133 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.133 DEBUG:boto:Method: GET 2022-09-12T19:06:57.134 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.135 DEBUG:boto:Data: 2022-09-12T19:06:57.136 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.137 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.138 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.139 DEBUG:boto:Params: {} 2022-09-12T19:06:57.140 DEBUG:boto:Token: None 2022-09-12T19:06:57.142 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.143 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OEatJPdbrF2sgqd6SE8bUl/BWa4= 2022-09-12T19:06:57.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OEatJPdbrF2sgqd6SE8bUl/BWa4=', 'Content-Length': '0'} 2022-09-12T19:06:57.147 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009661a54befd211c3-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.149 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:57.150 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:57.151 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.152 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.153 DEBUG:boto:Method: GET 2022-09-12T19:06:57.154 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:57.155 DEBUG:boto:Data: 2022-09-12T19:06:57.155 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.156 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.157 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.158 DEBUG:boto:Params: {} 2022-09-12T19:06:57.159 DEBUG:boto:Token: None 2022-09-12T19:06:57.160 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/ 2022-09-12T19:06:57.161 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM= 2022-09-12T19:06:57.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM=', 'Content-Length': '0'} 2022-09-12T19:06:57.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005abd6c8ba94e6824-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.167 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:57.169 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:57.170 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:57.171 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.171 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.172 DEBUG:boto:Method: GET 2022-09-12T19:06:57.173 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:57.174 DEBUG:boto:Data: 2022-09-12T19:06:57.175 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.175 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.176 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.177 DEBUG:boto:Params: {} 2022-09-12T19:06:57.177 DEBUG:boto:Token: None 2022-09-12T19:06:57.178 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/ 2022-09-12T19:06:57.178 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM= 2022-09-12T19:06:57.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM=', 'Content-Length': '0'} 2022-09-12T19:06:57.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d53e8754a356023-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.226 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:57.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:57.228 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.228 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.229 DEBUG:boto:path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:57.229 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:57.230 DEBUG:boto:Method: GET 2022-09-12T19:06:57.230 DEBUG:boto:Path: /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:57.231 DEBUG:boto:Data: 2022-09-12T19:06:57.231 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.232 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.233 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.233 DEBUG:boto:Params: {} 2022-09-12T19:06:57.234 DEBUG:boto:Token: None 2022-09-12T19:06:57.235 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:57.235 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Xa4eDI1p3k5nn6YxcQkBTQC4DiY= 2022-09-12T19:06:57.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Xa4eDI1p3k5nn6YxcQkBTQC4DiY=', 'Content-Length': '0'} 2022-09-12T19:06:57.238 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'e.6ruI53roqy4mmyPgzQ1kqMdajruFA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4a8a67245f01f38-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.239 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.240 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.240 DEBUG:boto:path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:57.241 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:57.241 DEBUG:boto:Method: GET 2022-09-12T19:06:57.242 DEBUG:boto:Path: /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:57.243 DEBUG:boto:Data: 2022-09-12T19:06:57.243 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.244 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.244 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.245 DEBUG:boto:Params: {} 2022-09-12T19:06:57.245 DEBUG:boto:Token: None 2022-09-12T19:06:57.246 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:57.246 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Xa4eDI1p3k5nn6YxcQkBTQC4DiY= 2022-09-12T19:06:57.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Xa4eDI1p3k5nn6YxcQkBTQC4DiY=', 'Content-Length': '0'} 2022-09-12T19:06:57.249 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'e.6ruI53roqy4mmyPgzQ1kqMdajruFA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1e5dc190c152de4-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.250 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.250 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.251 DEBUG:boto:path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:57.251 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:57.252 DEBUG:boto:Method: HEAD 2022-09-12T19:06:57.252 DEBUG:boto:Path: /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:57.253 DEBUG:boto:Data: 2022-09-12T19:06:57.253 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.254 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.254 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.255 DEBUG:boto:Params: {} 2022-09-12T19:06:57.255 DEBUG:boto:Token: None 2022-09-12T19:06:57.256 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:57.257 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/d+QZT3gG5kRG6WsJQyDoTZdZDI= 2022-09-12T19:06:57.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/d+QZT3gG5kRG6WsJQyDoTZdZDI=', 'Content-Length': '0'} 2022-09-12T19:06:57.259 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'e.6ruI53roqy4mmyPgzQ1kqMdajruFA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051a02c400ef2e4d2-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.260 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.260 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.261 DEBUG:boto:path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:57.261 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:57.262 DEBUG:boto:Method: HEAD 2022-09-12T19:06:57.262 DEBUG:boto:Path: /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:57.263 DEBUG:boto:Data: 2022-09-12T19:06:57.264 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.265 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.266 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.267 DEBUG:boto:Params: {} 2022-09-12T19:06:57.267 DEBUG:boto:Token: None 2022-09-12T19:06:57.268 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:57.269 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/d+QZT3gG5kRG6WsJQyDoTZdZDI= 2022-09-12T19:06:57.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/d+QZT3gG5kRG6WsJQyDoTZdZDI=', 'Content-Length': '0'} 2022-09-12T19:06:57.272 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'e.6ruI53roqy4mmyPgzQ1kqMdajruFA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d68380e1ccfa53da-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:57.273 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.274 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.274 DEBUG:boto:path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:57.275 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:57.276 DEBUG:boto:Method: GET 2022-09-12T19:06:57.276 DEBUG:boto:Path: /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:57.277 DEBUG:boto:Data: 2022-09-12T19:06:57.277 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.278 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.278 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.279 DEBUG:boto:Params: {} 2022-09-12T19:06:57.279 DEBUG:boto:Token: None 2022-09-12T19:06:57.280 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:57.281 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Xa4eDI1p3k5nn6YxcQkBTQC4DiY= 2022-09-12T19:06:57.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Xa4eDI1p3k5nn6YxcQkBTQC4DiY=', 'Content-Length': '0'} 2022-09-12T19:06:57.285 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'e.6ruI53roqy4mmyPgzQ1kqMdajruFA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f33e814b4c54dbb-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.326 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.327 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.328 DEBUG:boto:path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:57.328 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:57.329 DEBUG:boto:Method: GET 2022-09-12T19:06:57.329 DEBUG:boto:Path: /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:57.330 DEBUG:boto:Data: 2022-09-12T19:06:57.330 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.331 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.331 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.332 DEBUG:boto:Params: {} 2022-09-12T19:06:57.333 DEBUG:boto:Token: None 2022-09-12T19:06:57.333 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:06:57.334 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Xa4eDI1p3k5nn6YxcQkBTQC4DiY= 2022-09-12T19:06:57.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Xa4eDI1p3k5nn6YxcQkBTQC4DiY=', 'Content-Length': '0'} 2022-09-12T19:06:57.337 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'e.6ruI53roqy4mmyPgzQ1kqMdajruFA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000878dde01c79f4784-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.338 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:57.339 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:57.339 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.340 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.340 DEBUG:boto:Method: GET 2022-09-12T19:06:57.341 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:57.341 DEBUG:boto:Data: 2022-09-12T19:06:57.342 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.343 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.343 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.344 DEBUG:boto:Params: {} 2022-09-12T19:06:57.344 DEBUG:boto:Token: None 2022-09-12T19:06:57.345 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/ 2022-09-12T19:06:57.346 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM= 2022-09-12T19:06:57.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM=', 'Content-Length': '0'} 2022-09-12T19:06:57.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2f1ca4cf46dac97-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.351 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:57.352 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:57.353 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:57.354 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.355 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.355 DEBUG:boto:Method: GET 2022-09-12T19:06:57.356 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:57.357 DEBUG:boto:Data: 2022-09-12T19:06:57.357 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.358 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.358 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.359 DEBUG:boto:Params: {} 2022-09-12T19:06:57.360 DEBUG:boto:Token: None 2022-09-12T19:06:57.361 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/ 2022-09-12T19:06:57.361 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM= 2022-09-12T19:06:57.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM=', 'Content-Length': '0'} 2022-09-12T19:06:57.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3d4d3254393acee-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.366 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:57.367 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.368 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.368 DEBUG:boto:Method: HEAD 2022-09-12T19:06:57.369 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:57.369 DEBUG:boto:Data: 2022-09-12T19:06:57.370 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.371 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.372 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.373 DEBUG:boto:Params: {} 2022-09-12T19:06:57.374 DEBUG:boto:Token: None 2022-09-12T19:06:57.375 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj 2022-09-12T19:06:57.376 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1sMtbQoDERmFKhk7PwGtbxek1S0= 2022-09-12T19:06:57.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1sMtbQoDERmFKhk7PwGtbxek1S0=', 'Content-Length': '0'} 2022-09-12T19:06:57.380 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b8f30f5fd73d017-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.381 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.381 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.382 DEBUG:boto:Method: HEAD 2022-09-12T19:06:57.382 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:57.383 DEBUG:boto:Data: 2022-09-12T19:06:57.384 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.384 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.385 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.385 DEBUG:boto:Params: {} 2022-09-12T19:06:57.386 DEBUG:boto:Token: None 2022-09-12T19:06:57.386 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj 2022-09-12T19:06:57.387 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1sMtbQoDERmFKhk7PwGtbxek1S0= 2022-09-12T19:06:57.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1sMtbQoDERmFKhk7PwGtbxek1S0=', 'Content-Length': '0'} 2022-09-12T19:06:57.391 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003565f0c399ac5d14-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:57.393 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.393 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.394 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.394 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.395 DEBUG:boto:Method: GET 2022-09-12T19:06:57.396 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.396 DEBUG:boto:Data: 2022-09-12T19:06:57.397 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.397 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.398 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.398 DEBUG:boto:Params: {} 2022-09-12T19:06:57.399 DEBUG:boto:Token: None 2022-09-12T19:06:57.400 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.401 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OEatJPdbrF2sgqd6SE8bUl/BWa4= 2022-09-12T19:06:57.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OEatJPdbrF2sgqd6SE8bUl/BWa4=', 'Content-Length': '0'} 2022-09-12T19:06:57.403 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1ba638060ce9a33-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.404 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.405 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.405 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.406 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.406 DEBUG:boto:Method: GET 2022-09-12T19:06:57.407 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.407 DEBUG:boto:Data: 2022-09-12T19:06:57.408 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.408 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.409 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.409 DEBUG:boto:Params: {} 2022-09-12T19:06:57.410 DEBUG:boto:Token: None 2022-09-12T19:06:57.411 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.411 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OEatJPdbrF2sgqd6SE8bUl/BWa4= 2022-09-12T19:06:57.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OEatJPdbrF2sgqd6SE8bUl/BWa4=', 'Content-Length': '0'} 2022-09-12T19:06:57.414 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004ed8c3afb6d00c2-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.415 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:57.415 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:57.416 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.416 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.417 DEBUG:boto:Method: GET 2022-09-12T19:06:57.417 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:57.418 DEBUG:boto:Data: 2022-09-12T19:06:57.418 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.419 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.419 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.420 DEBUG:boto:Params: {} 2022-09-12T19:06:57.420 DEBUG:boto:Token: None 2022-09-12T19:06:57.421 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/ 2022-09-12T19:06:57.422 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM= 2022-09-12T19:06:57.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM=', 'Content-Length': '0'} 2022-09-12T19:06:57.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e15319652807bf74-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.426 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:57.427 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:57.427 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:57.428 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.428 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.429 DEBUG:boto:Method: GET 2022-09-12T19:06:57.429 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:57.430 DEBUG:boto:Data: 2022-09-12T19:06:57.430 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.431 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.431 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.432 DEBUG:boto:Params: {} 2022-09-12T19:06:57.433 DEBUG:boto:Token: None 2022-09-12T19:06:57.433 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/ 2022-09-12T19:06:57.434 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM= 2022-09-12T19:06:57.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM=', 'Content-Length': '0'} 2022-09-12T19:06:57.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038bc1add281fda05-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.478 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:57.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:57.480 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.481 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.482 DEBUG:boto:path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:57.482 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:57.483 DEBUG:boto:Method: GET 2022-09-12T19:06:57.484 DEBUG:boto:Path: /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:57.485 DEBUG:boto:Data: 2022-09-12T19:06:57.485 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.486 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.487 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.487 DEBUG:boto:Params: {} 2022-09-12T19:06:57.488 DEBUG:boto:Token: None 2022-09-12T19:06:57.489 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:57.490 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2NkpW3TCR9cxbyAgLxhNlouF/NQ= 2022-09-12T19:06:57.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2NkpW3TCR9cxbyAgLxhNlouF/NQ=', 'Content-Length': '0'} 2022-09-12T19:06:57.495 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d35e53305dd946d1-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.496 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.496 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.497 DEBUG:boto:path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:57.498 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:57.498 DEBUG:boto:Method: GET 2022-09-12T19:06:57.499 DEBUG:boto:Path: /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:57.499 DEBUG:boto:Data: 2022-09-12T19:06:57.500 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.500 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.501 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.501 DEBUG:boto:Params: {} 2022-09-12T19:06:57.502 DEBUG:boto:Token: None 2022-09-12T19:06:57.502 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:57.503 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2NkpW3TCR9cxbyAgLxhNlouF/NQ= 2022-09-12T19:06:57.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2NkpW3TCR9cxbyAgLxhNlouF/NQ=', 'Content-Length': '0'} 2022-09-12T19:06:57.506 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a31d7b3f3c9c223-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.507 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.507 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.508 DEBUG:boto:path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:57.508 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:57.509 DEBUG:boto:Method: HEAD 2022-09-12T19:06:57.509 DEBUG:boto:Path: /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:57.510 DEBUG:boto:Data: 2022-09-12T19:06:57.510 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.511 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.511 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.512 DEBUG:boto:Params: {} 2022-09-12T19:06:57.512 DEBUG:boto:Token: None 2022-09-12T19:06:57.513 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:57.514 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vGMzXITQ0zRWU0ennEDVdvgUBG8= 2022-09-12T19:06:57.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vGMzXITQ0zRWU0ennEDVdvgUBG8=', 'Content-Length': '0'} 2022-09-12T19:06:57.517 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006bca326a20408772-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.518 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.518 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.519 DEBUG:boto:path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:57.519 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:57.520 DEBUG:boto:Method: HEAD 2022-09-12T19:06:57.520 DEBUG:boto:Path: /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:57.521 DEBUG:boto:Data: 2022-09-12T19:06:57.522 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.523 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.524 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.525 DEBUG:boto:Params: {} 2022-09-12T19:06:57.525 DEBUG:boto:Token: None 2022-09-12T19:06:57.526 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:57.527 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vGMzXITQ0zRWU0ennEDVdvgUBG8= 2022-09-12T19:06:57.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vGMzXITQ0zRWU0ennEDVdvgUBG8=', 'Content-Length': '0'} 2022-09-12T19:06:57.529 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f332386fff696bf7-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:57.531 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.532 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.532 DEBUG:boto:path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:57.533 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:57.533 DEBUG:boto:Method: GET 2022-09-12T19:06:57.534 DEBUG:boto:Path: /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:57.534 DEBUG:boto:Data: 2022-09-12T19:06:57.535 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.535 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.536 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.536 DEBUG:boto:Params: {} 2022-09-12T19:06:57.537 DEBUG:boto:Token: None 2022-09-12T19:06:57.538 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:57.539 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2NkpW3TCR9cxbyAgLxhNlouF/NQ= 2022-09-12T19:06:57.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2NkpW3TCR9cxbyAgLxhNlouF/NQ=', 'Content-Length': '0'} 2022-09-12T19:06:57.542 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e148e5286c596ec7-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.582 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.583 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.584 DEBUG:boto:path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:57.585 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:57.585 DEBUG:boto:Method: GET 2022-09-12T19:06:57.586 DEBUG:boto:Path: /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:57.587 DEBUG:boto:Data: 2022-09-12T19:06:57.587 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.588 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.588 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.589 DEBUG:boto:Params: {} 2022-09-12T19:06:57.589 DEBUG:boto:Token: None 2022-09-12T19:06:57.590 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:06:57.591 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2NkpW3TCR9cxbyAgLxhNlouF/NQ= 2022-09-12T19:06:57.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2NkpW3TCR9cxbyAgLxhNlouF/NQ=', 'Content-Length': '0'} 2022-09-12T19:06:57.594 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002bca0c2e8ca40127-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.595 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:57.595 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:57.596 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.597 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.597 DEBUG:boto:Method: GET 2022-09-12T19:06:57.598 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:57.598 DEBUG:boto:Data: 2022-09-12T19:06:57.599 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.599 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.600 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.601 DEBUG:boto:Params: {} 2022-09-12T19:06:57.601 DEBUG:boto:Token: None 2022-09-12T19:06:57.602 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/ 2022-09-12T19:06:57.602 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM= 2022-09-12T19:06:57.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM=', 'Content-Length': '0'} 2022-09-12T19:06:57.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdedf985cd7d1674-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.607 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:57.608 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:57.608 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:57.609 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.609 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.610 DEBUG:boto:Method: GET 2022-09-12T19:06:57.611 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:57.611 DEBUG:boto:Data: 2022-09-12T19:06:57.612 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.612 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.613 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.614 DEBUG:boto:Params: {} 2022-09-12T19:06:57.614 DEBUG:boto:Token: None 2022-09-12T19:06:57.615 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/ 2022-09-12T19:06:57.616 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM= 2022-09-12T19:06:57.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM=', 'Content-Length': '0'} 2022-09-12T19:06:57.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028d180ff019273e0-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.662 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:57.663 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.664 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.664 DEBUG:boto:Method: HEAD 2022-09-12T19:06:57.665 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:57.665 DEBUG:boto:Data: 2022-09-12T19:06:57.666 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.666 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.667 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.667 DEBUG:boto:Params: {} 2022-09-12T19:06:57.668 DEBUG:boto:Token: None 2022-09-12T19:06:57.669 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj 2022-09-12T19:06:57.669 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1sMtbQoDERmFKhk7PwGtbxek1S0= 2022-09-12T19:06:57.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1sMtbQoDERmFKhk7PwGtbxek1S0=', 'Content-Length': '0'} 2022-09-12T19:06:57.673 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d7350542570e632-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.674 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.675 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.675 DEBUG:boto:Method: HEAD 2022-09-12T19:06:57.676 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:57.676 DEBUG:boto:Data: 2022-09-12T19:06:57.677 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.677 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.678 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.679 DEBUG:boto:Params: {} 2022-09-12T19:06:57.679 DEBUG:boto:Token: None 2022-09-12T19:06:57.680 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj 2022-09-12T19:06:57.681 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1sMtbQoDERmFKhk7PwGtbxek1S0= 2022-09-12T19:06:57.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1sMtbQoDERmFKhk7PwGtbxek1S0=', 'Content-Length': '0'} 2022-09-12T19:06:57.685 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062c5c7edc27c00b5-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:57.686 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.686 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.687 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.688 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.688 DEBUG:boto:Method: GET 2022-09-12T19:06:57.689 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.689 DEBUG:boto:Data: 2022-09-12T19:06:57.690 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.690 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.691 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.692 DEBUG:boto:Params: {} 2022-09-12T19:06:57.692 DEBUG:boto:Token: None 2022-09-12T19:06:57.693 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.693 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OEatJPdbrF2sgqd6SE8bUl/BWa4= 2022-09-12T19:06:57.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OEatJPdbrF2sgqd6SE8bUl/BWa4=', 'Content-Length': '0'} 2022-09-12T19:06:57.696 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007571ed1c3f45e073-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.697 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.698 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.698 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.699 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.699 DEBUG:boto:Method: GET 2022-09-12T19:06:57.700 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.701 DEBUG:boto:Data: 2022-09-12T19:06:57.701 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.702 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.702 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.703 DEBUG:boto:Params: {} 2022-09-12T19:06:57.704 DEBUG:boto:Token: None 2022-09-12T19:06:57.704 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.705 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OEatJPdbrF2sgqd6SE8bUl/BWa4= 2022-09-12T19:06:57.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OEatJPdbrF2sgqd6SE8bUl/BWa4=', 'Content-Length': '0'} 2022-09-12T19:06:57.708 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002116fb2153271fc6-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.709 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:57.710 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:57.710 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.711 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.712 DEBUG:boto:Method: GET 2022-09-12T19:06:57.713 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:57.713 DEBUG:boto:Data: 2022-09-12T19:06:57.714 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.714 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.715 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.715 DEBUG:boto:Params: {} 2022-09-12T19:06:57.716 DEBUG:boto:Token: None 2022-09-12T19:06:57.716 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/ 2022-09-12T19:06:57.717 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM= 2022-09-12T19:06:57.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM=', 'Content-Length': '0'} 2022-09-12T19:06:57.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000975e5c300a759cae-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.721 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:57.722 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:57.723 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:57.724 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.724 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.725 DEBUG:boto:Method: GET 2022-09-12T19:06:57.726 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:57.726 DEBUG:boto:Data: 2022-09-12T19:06:57.727 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.727 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.728 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.729 DEBUG:boto:Params: {} 2022-09-12T19:06:57.729 DEBUG:boto:Token: None 2022-09-12T19:06:57.730 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/ 2022-09-12T19:06:57.730 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM= 2022-09-12T19:06:57.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM=', 'Content-Length': '0'} 2022-09-12T19:06:57.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000235ec4f93831028d-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.735 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:57.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:57.737 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.737 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.738 DEBUG:boto:path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:57.738 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:57.739 DEBUG:boto:Method: GET 2022-09-12T19:06:57.739 DEBUG:boto:Path: /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:57.740 DEBUG:boto:Data: 2022-09-12T19:06:57.740 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.741 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.741 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.742 DEBUG:boto:Params: {} 2022-09-12T19:06:57.743 DEBUG:boto:Token: None 2022-09-12T19:06:57.743 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:57.744 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HweLPaGZ4h9sxxBgSm/GDZgAE9s= 2022-09-12T19:06:57.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HweLPaGZ4h9sxxBgSm/GDZgAE9s=', 'Content-Length': '0'} 2022-09-12T19:06:57.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'lnpiHoSs91QVujLSWqHhvEGzZrF3-sA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093a9c64e0f8f9f8d-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.747 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.748 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.749 DEBUG:boto:path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:57.749 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:57.750 DEBUG:boto:Method: GET 2022-09-12T19:06:57.750 DEBUG:boto:Path: /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:57.751 DEBUG:boto:Data: 2022-09-12T19:06:57.751 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.752 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.752 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.753 DEBUG:boto:Params: {} 2022-09-12T19:06:57.753 DEBUG:boto:Token: None 2022-09-12T19:06:57.754 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:57.755 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HweLPaGZ4h9sxxBgSm/GDZgAE9s= 2022-09-12T19:06:57.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HweLPaGZ4h9sxxBgSm/GDZgAE9s=', 'Content-Length': '0'} 2022-09-12T19:06:57.758 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'lnpiHoSs91QVujLSWqHhvEGzZrF3-sA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ef7c01fbaee4aaa-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.759 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.760 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.761 DEBUG:boto:path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:57.762 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:57.763 DEBUG:boto:Method: HEAD 2022-09-12T19:06:57.763 DEBUG:boto:Path: /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:57.764 DEBUG:boto:Data: 2022-09-12T19:06:57.764 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.765 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.765 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.766 DEBUG:boto:Params: {} 2022-09-12T19:06:57.766 DEBUG:boto:Token: None 2022-09-12T19:06:57.767 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:57.767 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7ciYG4dvVKar9oxlSJb5zqegukQ= 2022-09-12T19:06:57.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7ciYG4dvVKar9oxlSJb5zqegukQ=', 'Content-Length': '0'} 2022-09-12T19:06:57.770 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'lnpiHoSs91QVujLSWqHhvEGzZrF3-sA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c44ef5cf269f54af-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.771 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.771 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.772 DEBUG:boto:path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:57.772 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:57.773 DEBUG:boto:Method: HEAD 2022-09-12T19:06:57.773 DEBUG:boto:Path: /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:57.774 DEBUG:boto:Data: 2022-09-12T19:06:57.774 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.775 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.775 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.776 DEBUG:boto:Params: {} 2022-09-12T19:06:57.776 DEBUG:boto:Token: None 2022-09-12T19:06:57.777 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:57.777 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7ciYG4dvVKar9oxlSJb5zqegukQ= 2022-09-12T19:06:57.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7ciYG4dvVKar9oxlSJb5zqegukQ=', 'Content-Length': '0'} 2022-09-12T19:06:57.780 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'lnpiHoSs91QVujLSWqHhvEGzZrF3-sA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026b23c61e46c03ab-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:57.781 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.782 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.782 DEBUG:boto:path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:57.783 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:57.783 DEBUG:boto:Method: GET 2022-09-12T19:06:57.784 DEBUG:boto:Path: /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:57.785 DEBUG:boto:Data: 2022-09-12T19:06:57.785 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.786 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.786 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.787 DEBUG:boto:Params: {} 2022-09-12T19:06:57.787 DEBUG:boto:Token: None 2022-09-12T19:06:57.788 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:57.788 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HweLPaGZ4h9sxxBgSm/GDZgAE9s= 2022-09-12T19:06:57.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HweLPaGZ4h9sxxBgSm/GDZgAE9s=', 'Content-Length': '0'} 2022-09-12T19:06:57.791 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'lnpiHoSs91QVujLSWqHhvEGzZrF3-sA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f99a793232afb243-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.838 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.839 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.840 DEBUG:boto:path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:57.841 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:57.841 DEBUG:boto:Method: GET 2022-09-12T19:06:57.843 DEBUG:boto:Path: /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:57.843 DEBUG:boto:Data: 2022-09-12T19:06:57.844 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.845 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.846 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.847 DEBUG:boto:Params: {} 2022-09-12T19:06:57.848 DEBUG:boto:Token: None 2022-09-12T19:06:57.850 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:06:57.851 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HweLPaGZ4h9sxxBgSm/GDZgAE9s= 2022-09-12T19:06:57.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HweLPaGZ4h9sxxBgSm/GDZgAE9s=', 'Content-Length': '0'} 2022-09-12T19:06:57.855 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'lnpiHoSs91QVujLSWqHhvEGzZrF3-sA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030c2aa54356eed97-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.856 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:57.857 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:57.858 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.859 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.860 DEBUG:boto:Method: GET 2022-09-12T19:06:57.860 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:57.861 DEBUG:boto:Data: 2022-09-12T19:06:57.862 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.862 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.863 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.863 DEBUG:boto:Params: {} 2022-09-12T19:06:57.864 DEBUG:boto:Token: None 2022-09-12T19:06:57.865 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/ 2022-09-12T19:06:57.865 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM= 2022-09-12T19:06:57.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM=', 'Content-Length': '0'} 2022-09-12T19:06:57.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5bb7ed37a1bab3c-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.870 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:57.871 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:57.871 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:57.872 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.872 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.873 DEBUG:boto:Method: GET 2022-09-12T19:06:57.874 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:57.874 DEBUG:boto:Data: 2022-09-12T19:06:57.875 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.876 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.876 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.877 DEBUG:boto:Params: {} 2022-09-12T19:06:57.877 DEBUG:boto:Token: None 2022-09-12T19:06:57.878 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/ 2022-09-12T19:06:57.879 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM= 2022-09-12T19:06:57.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM=', 'Content-Length': '0'} 2022-09-12T19:06:57.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abc7ba355c07f15b-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.885 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:57.887 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.888 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.889 DEBUG:boto:Method: HEAD 2022-09-12T19:06:57.889 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:57.890 DEBUG:boto:Data: 2022-09-12T19:06:57.891 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.891 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.892 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.892 DEBUG:boto:Params: {} 2022-09-12T19:06:57.893 DEBUG:boto:Token: None 2022-09-12T19:06:57.893 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj 2022-09-12T19:06:57.894 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1sMtbQoDERmFKhk7PwGtbxek1S0= 2022-09-12T19:06:57.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1sMtbQoDERmFKhk7PwGtbxek1S0=', 'Content-Length': '0'} 2022-09-12T19:06:57.899 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014f08f6a26ed09c4-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.900 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.900 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.901 DEBUG:boto:Method: HEAD 2022-09-12T19:06:57.901 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:57.902 DEBUG:boto:Data: 2022-09-12T19:06:57.902 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.903 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.903 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.904 DEBUG:boto:Params: {} 2022-09-12T19:06:57.904 DEBUG:boto:Token: None 2022-09-12T19:06:57.905 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj 2022-09-12T19:06:57.905 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1sMtbQoDERmFKhk7PwGtbxek1S0= 2022-09-12T19:06:57.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1sMtbQoDERmFKhk7PwGtbxek1S0=', 'Content-Length': '0'} 2022-09-12T19:06:57.910 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009235d2724e41ce1d-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:57.911 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.912 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.912 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.913 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.913 DEBUG:boto:Method: GET 2022-09-12T19:06:57.914 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.915 DEBUG:boto:Data: 2022-09-12T19:06:57.915 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.916 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.917 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.918 DEBUG:boto:Params: {} 2022-09-12T19:06:57.919 DEBUG:boto:Token: None 2022-09-12T19:06:57.920 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.921 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OEatJPdbrF2sgqd6SE8bUl/BWa4= 2022-09-12T19:06:57.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OEatJPdbrF2sgqd6SE8bUl/BWa4=', 'Content-Length': '0'} 2022-09-12T19:06:57.924 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4e9449f898e2af3-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.925 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.926 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.926 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.927 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.927 DEBUG:boto:Method: GET 2022-09-12T19:06:57.928 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.929 DEBUG:boto:Data: 2022-09-12T19:06:57.929 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.930 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.930 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.931 DEBUG:boto:Params: {} 2022-09-12T19:06:57.931 DEBUG:boto:Token: None 2022-09-12T19:06:57.932 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:57.933 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OEatJPdbrF2sgqd6SE8bUl/BWa4= 2022-09-12T19:06:57.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OEatJPdbrF2sgqd6SE8bUl/BWa4=', 'Content-Length': '0'} 2022-09-12T19:06:57.936 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000004651368fe29f2f-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.936 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:57.937 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:57.938 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.938 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.939 DEBUG:boto:Method: GET 2022-09-12T19:06:57.939 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:57.940 DEBUG:boto:Data: 2022-09-12T19:06:57.941 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.941 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.942 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.942 DEBUG:boto:Params: {} 2022-09-12T19:06:57.943 DEBUG:boto:Token: None 2022-09-12T19:06:57.944 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/ 2022-09-12T19:06:57.944 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM= 2022-09-12T19:06:57.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM=', 'Content-Length': '0'} 2022-09-12T19:06:57.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060fd410a217e969e-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.948 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:57.949 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:57.950 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:57.950 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.951 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:57.951 DEBUG:boto:Method: GET 2022-09-12T19:06:57.952 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:57.953 DEBUG:boto:Data: 2022-09-12T19:06:57.954 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.955 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:57.955 DEBUG:boto:Port: 8000 2022-09-12T19:06:57.956 DEBUG:boto:Params: {} 2022-09-12T19:06:57.957 DEBUG:boto:Token: None 2022-09-12T19:06:57.957 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/ 2022-09-12T19:06:57.958 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM= 2022-09-12T19:06:57.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e53iOfuyNJNOUMlfGiLO2s30QJM=', 'Content-Length': '0'} 2022-09-12T19:06:57.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000818579353d9d78b7-00631f8351-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:57.963 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:57.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:57.964 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:57.964 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:57.965 DEBUG:boto:path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:57.966 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:57.967 DEBUG:boto:Method: GET 2022-09-12T19:06:57.967 DEBUG:boto:Path: /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:57.968 DEBUG:boto:Data: 2022-09-12T19:06:57.968 DEBUG:boto:Headers: {} 2022-09-12T19:06:57.969 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:57.969 DEBUG:boto:Port: 8001 2022-09-12T19:06:57.970 DEBUG:boto:Params: {} 2022-09-12T19:06:57.971 DEBUG:boto:Token: None 2022-09-12T19:06:57.971 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:57 GMT /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:57.972 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8FFdsL5OXDH+J0dPIMefTqWLnQ0= 2022-09-12T19:06:57.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8FFdsL5OXDH+J0dPIMefTqWLnQ0=', 'Content-Length': '0'} 2022-09-12T19:06:57.975 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'p5OIjHSzwvUDkIBYzRPdKBGezjIkrn-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eee5d0c6a847b40f-00631f8351-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.018 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.019 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.019 DEBUG:boto:path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:58.020 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:58.021 DEBUG:boto:Method: GET 2022-09-12T19:06:58.022 DEBUG:boto:Path: /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:58.023 DEBUG:boto:Data: 2022-09-12T19:06:58.025 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.026 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.027 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.028 DEBUG:boto:Params: {} 2022-09-12T19:06:58.029 DEBUG:boto:Token: None 2022-09-12T19:06:58.030 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:58.031 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7SdTYaGGCGUSGHMijekaiW24tv0= 2022-09-12T19:06:58.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7SdTYaGGCGUSGHMijekaiW24tv0=', 'Content-Length': '0'} 2022-09-12T19:06:58.035 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'p5OIjHSzwvUDkIBYzRPdKBGezjIkrn-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000027ad44d92254b18-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.036 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.037 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.037 DEBUG:boto:path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:58.038 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:58.039 DEBUG:boto:Method: HEAD 2022-09-12T19:06:58.039 DEBUG:boto:Path: /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:58.040 DEBUG:boto:Data: 2022-09-12T19:06:58.040 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.041 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.041 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.042 DEBUG:boto:Params: {} 2022-09-12T19:06:58.043 DEBUG:boto:Token: None 2022-09-12T19:06:58.043 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:58.044 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NGk6NzBjAfOWHbkThH5H9BBZvU0= 2022-09-12T19:06:58.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NGk6NzBjAfOWHbkThH5H9BBZvU0=', 'Content-Length': '0'} 2022-09-12T19:06:58.046 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'p5OIjHSzwvUDkIBYzRPdKBGezjIkrn-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad3c2e9edf4395d8-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.047 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.048 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.048 DEBUG:boto:path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:58.049 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:58.049 DEBUG:boto:Method: HEAD 2022-09-12T19:06:58.050 DEBUG:boto:Path: /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:58.050 DEBUG:boto:Data: 2022-09-12T19:06:58.051 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.051 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.052 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.052 DEBUG:boto:Params: {} 2022-09-12T19:06:58.053 DEBUG:boto:Token: None 2022-09-12T19:06:58.054 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:58.055 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NGk6NzBjAfOWHbkThH5H9BBZvU0= 2022-09-12T19:06:58.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NGk6NzBjAfOWHbkThH5H9BBZvU0=', 'Content-Length': '0'} 2022-09-12T19:06:58.057 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'p5OIjHSzwvUDkIBYzRPdKBGezjIkrn-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069e3c9e32c293979-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:58.059 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.059 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.060 DEBUG:boto:path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:58.060 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:58.061 DEBUG:boto:Method: GET 2022-09-12T19:06:58.061 DEBUG:boto:Path: /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:58.062 DEBUG:boto:Data: 2022-09-12T19:06:58.063 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.063 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.064 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.064 DEBUG:boto:Params: {} 2022-09-12T19:06:58.065 DEBUG:boto:Token: None 2022-09-12T19:06:58.066 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:58.066 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7SdTYaGGCGUSGHMijekaiW24tv0= 2022-09-12T19:06:58.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7SdTYaGGCGUSGHMijekaiW24tv0=', 'Content-Length': '0'} 2022-09-12T19:06:58.069 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'p5OIjHSzwvUDkIBYzRPdKBGezjIkrn-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006dcbf9a51db7518f-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.070 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.070 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.071 DEBUG:boto:path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:58.071 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:58.072 DEBUG:boto:Method: GET 2022-09-12T19:06:58.073 DEBUG:boto:Path: /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:58.073 DEBUG:boto:Data: 2022-09-12T19:06:58.074 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.074 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.075 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.075 DEBUG:boto:Params: {} 2022-09-12T19:06:58.076 DEBUG:boto:Token: None 2022-09-12T19:06:58.077 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:06:58.077 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7SdTYaGGCGUSGHMijekaiW24tv0= 2022-09-12T19:06:58.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7SdTYaGGCGUSGHMijekaiW24tv0=', 'Content-Length': '0'} 2022-09-12T19:06:58.080 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'p5OIjHSzwvUDkIBYzRPdKBGezjIkrn-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e4c31097f1ecc71-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.081 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:58.081 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:58.082 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.083 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.083 DEBUG:boto:Method: GET 2022-09-12T19:06:58.084 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:58.084 DEBUG:boto:Data: 2022-09-12T19:06:58.085 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.085 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.086 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.086 DEBUG:boto:Params: {} 2022-09-12T19:06:58.087 DEBUG:boto:Token: None 2022-09-12T19:06:58.088 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/ 2022-09-12T19:06:58.088 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg= 2022-09-12T19:06:58.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg=', 'Content-Length': '0'} 2022-09-12T19:06:58.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000177e5f25f5384cbb-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.092 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:58.093 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:58.094 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:58.094 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.095 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.095 DEBUG:boto:Method: GET 2022-09-12T19:06:58.096 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:58.096 DEBUG:boto:Data: 2022-09-12T19:06:58.097 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.097 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.098 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.099 DEBUG:boto:Params: {} 2022-09-12T19:06:58.099 DEBUG:boto:Token: None 2022-09-12T19:06:58.100 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/ 2022-09-12T19:06:58.100 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg= 2022-09-12T19:06:58.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg=', 'Content-Length': '0'} 2022-09-12T19:06:58.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a0e55937df0ec2a-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.105 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:58.106 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.106 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.116 DEBUG:boto:Method: HEAD 2022-09-12T19:06:58.117 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:58.118 DEBUG:boto:Data: 2022-09-12T19:06:58.118 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.119 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.119 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.120 DEBUG:boto:Params: {} 2022-09-12T19:06:58.121 DEBUG:boto:Token: None 2022-09-12T19:06:58.121 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj 2022-09-12T19:06:58.122 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Y2Fe3XT78Ao/jgDmCP3K8mxANc= 2022-09-12T19:06:58.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Y2Fe3XT78Ao/jgDmCP3K8mxANc=', 'Content-Length': '0'} 2022-09-12T19:06:58.126 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000942174c7f072cc2-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.127 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.127 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.128 DEBUG:boto:Method: HEAD 2022-09-12T19:06:58.128 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:58.129 DEBUG:boto:Data: 2022-09-12T19:06:58.129 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.130 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.130 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.131 DEBUG:boto:Params: {} 2022-09-12T19:06:58.132 DEBUG:boto:Token: None 2022-09-12T19:06:58.132 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj 2022-09-12T19:06:58.133 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Y2Fe3XT78Ao/jgDmCP3K8mxANc= 2022-09-12T19:06:58.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Y2Fe3XT78Ao/jgDmCP3K8mxANc=', 'Content-Length': '0'} 2022-09-12T19:06:58.137 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab647710f3fadae8-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:58.138 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.139 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.139 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.140 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.141 DEBUG:boto:Method: GET 2022-09-12T19:06:58.141 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.142 DEBUG:boto:Data: 2022-09-12T19:06:58.142 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.143 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.144 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.145 DEBUG:boto:Params: {} 2022-09-12T19:06:58.145 DEBUG:boto:Token: None 2022-09-12T19:06:58.146 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.147 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B7co6InOJZmjD+FMGQlHUp/72B0= 2022-09-12T19:06:58.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B7co6InOJZmjD+FMGQlHUp/72B0=', 'Content-Length': '0'} 2022-09-12T19:06:58.150 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004fa0942d303f3447-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.194 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.195 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.196 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.196 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.197 DEBUG:boto:Method: GET 2022-09-12T19:06:58.197 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.198 DEBUG:boto:Data: 2022-09-12T19:06:58.199 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.199 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.200 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.201 DEBUG:boto:Params: {} 2022-09-12T19:06:58.201 DEBUG:boto:Token: None 2022-09-12T19:06:58.202 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.203 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B7co6InOJZmjD+FMGQlHUp/72B0= 2022-09-12T19:06:58.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B7co6InOJZmjD+FMGQlHUp/72B0=', 'Content-Length': '0'} 2022-09-12T19:06:58.205 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009482e567693caa4b-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.206 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:58.207 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:58.208 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.208 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.209 DEBUG:boto:Method: GET 2022-09-12T19:06:58.209 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:58.210 DEBUG:boto:Data: 2022-09-12T19:06:58.210 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.211 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.212 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.213 DEBUG:boto:Params: {} 2022-09-12T19:06:58.214 DEBUG:boto:Token: None 2022-09-12T19:06:58.215 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/ 2022-09-12T19:06:58.216 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg= 2022-09-12T19:06:58.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg=', 'Content-Length': '0'} 2022-09-12T19:06:58.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002594c745fefb1614-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.223 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:58.224 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:58.224 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:58.225 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.225 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.226 DEBUG:boto:Method: GET 2022-09-12T19:06:58.227 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:58.227 DEBUG:boto:Data: 2022-09-12T19:06:58.228 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.228 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.229 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.229 DEBUG:boto:Params: {} 2022-09-12T19:06:58.230 DEBUG:boto:Token: None 2022-09-12T19:06:58.231 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/ 2022-09-12T19:06:58.231 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg= 2022-09-12T19:06:58.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg=', 'Content-Length': '0'} 2022-09-12T19:06:58.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e85c5c8a6e32330-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.236 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:58.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:58.237 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.238 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.239 DEBUG:boto:path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:58.240 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:58.241 DEBUG:boto:Method: GET 2022-09-12T19:06:58.242 DEBUG:boto:Path: /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:58.243 DEBUG:boto:Data: 2022-09-12T19:06:58.244 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.244 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.245 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.246 DEBUG:boto:Params: {} 2022-09-12T19:06:58.246 DEBUG:boto:Token: None 2022-09-12T19:06:58.247 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:58.248 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VpE2K5QVyWHSacLSRxwRy5+yfEI= 2022-09-12T19:06:58.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VpE2K5QVyWHSacLSRxwRy5+yfEI=', 'Content-Length': '0'} 2022-09-12T19:06:58.251 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000bcb0498cc09ab53-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.252 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.252 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.253 DEBUG:boto:path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:58.253 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:58.254 DEBUG:boto:Method: GET 2022-09-12T19:06:58.254 DEBUG:boto:Path: /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:58.255 DEBUG:boto:Data: 2022-09-12T19:06:58.256 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.257 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.258 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.259 DEBUG:boto:Params: {} 2022-09-12T19:06:58.260 DEBUG:boto:Token: None 2022-09-12T19:06:58.261 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:58.262 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VpE2K5QVyWHSacLSRxwRy5+yfEI= 2022-09-12T19:06:58.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VpE2K5QVyWHSacLSRxwRy5+yfEI=', 'Content-Length': '0'} 2022-09-12T19:06:58.265 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d725b5807ccaa5d1-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.266 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.267 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.267 DEBUG:boto:path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:58.268 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:58.268 DEBUG:boto:Method: HEAD 2022-09-12T19:06:58.269 DEBUG:boto:Path: /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:58.269 DEBUG:boto:Data: 2022-09-12T19:06:58.270 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.271 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.271 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.272 DEBUG:boto:Params: {} 2022-09-12T19:06:58.272 DEBUG:boto:Token: None 2022-09-12T19:06:58.273 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:58.273 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fcNZnbdPUhHJC94OArm/ThHVwGQ= 2022-09-12T19:06:58.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fcNZnbdPUhHJC94OArm/ThHVwGQ=', 'Content-Length': '0'} 2022-09-12T19:06:58.276 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c5b867ea8484cdc-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.277 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.277 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.278 DEBUG:boto:path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:58.278 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:58.279 DEBUG:boto:Method: HEAD 2022-09-12T19:06:58.279 DEBUG:boto:Path: /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:58.280 DEBUG:boto:Data: 2022-09-12T19:06:58.281 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.281 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.282 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.283 DEBUG:boto:Params: {} 2022-09-12T19:06:58.283 DEBUG:boto:Token: None 2022-09-12T19:06:58.284 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:58.284 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fcNZnbdPUhHJC94OArm/ThHVwGQ= 2022-09-12T19:06:58.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fcNZnbdPUhHJC94OArm/ThHVwGQ=', 'Content-Length': '0'} 2022-09-12T19:06:58.287 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9e619bafbb0eced-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:58.290 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.291 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.292 DEBUG:boto:path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:58.292 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:58.293 DEBUG:boto:Method: GET 2022-09-12T19:06:58.294 DEBUG:boto:Path: /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:58.294 DEBUG:boto:Data: 2022-09-12T19:06:58.295 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.295 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.296 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.296 DEBUG:boto:Params: {} 2022-09-12T19:06:58.297 DEBUG:boto:Token: None 2022-09-12T19:06:58.298 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:58.299 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VpE2K5QVyWHSacLSRxwRy5+yfEI= 2022-09-12T19:06:58.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VpE2K5QVyWHSacLSRxwRy5+yfEI=', 'Content-Length': '0'} 2022-09-12T19:06:58.303 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8510d2bc12e1cf8-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.304 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.305 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.306 DEBUG:boto:path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:58.307 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:58.308 DEBUG:boto:Method: GET 2022-09-12T19:06:58.309 DEBUG:boto:Path: /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:58.310 DEBUG:boto:Data: 2022-09-12T19:06:58.311 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.312 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.312 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.313 DEBUG:boto:Params: {} 2022-09-12T19:06:58.313 DEBUG:boto:Token: None 2022-09-12T19:06:58.314 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:06:58.315 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VpE2K5QVyWHSacLSRxwRy5+yfEI= 2022-09-12T19:06:58.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VpE2K5QVyWHSacLSRxwRy5+yfEI=', 'Content-Length': '0'} 2022-09-12T19:06:58.318 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b597e1533637a418-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.319 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:58.319 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:58.320 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.320 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.321 DEBUG:boto:Method: GET 2022-09-12T19:06:58.321 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:58.322 DEBUG:boto:Data: 2022-09-12T19:06:58.323 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.323 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.324 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.324 DEBUG:boto:Params: {} 2022-09-12T19:06:58.325 DEBUG:boto:Token: None 2022-09-12T19:06:58.326 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/ 2022-09-12T19:06:58.326 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg= 2022-09-12T19:06:58.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg=', 'Content-Length': '0'} 2022-09-12T19:06:58.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000193c8e47c8505459-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.370 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:58.372 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:58.373 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:58.373 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.374 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.375 DEBUG:boto:Method: GET 2022-09-12T19:06:58.376 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:58.377 DEBUG:boto:Data: 2022-09-12T19:06:58.378 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.379 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.380 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.381 DEBUG:boto:Params: {} 2022-09-12T19:06:58.382 DEBUG:boto:Token: None 2022-09-12T19:06:58.383 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/ 2022-09-12T19:06:58.384 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg= 2022-09-12T19:06:58.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg=', 'Content-Length': '0'} 2022-09-12T19:06:58.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f136698311bc526c-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.390 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:58.391 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.391 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.392 DEBUG:boto:Method: HEAD 2022-09-12T19:06:58.392 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:58.393 DEBUG:boto:Data: 2022-09-12T19:06:58.393 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.394 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.394 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.395 DEBUG:boto:Params: {} 2022-09-12T19:06:58.395 DEBUG:boto:Token: None 2022-09-12T19:06:58.396 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj 2022-09-12T19:06:58.397 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Y2Fe3XT78Ao/jgDmCP3K8mxANc= 2022-09-12T19:06:58.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Y2Fe3XT78Ao/jgDmCP3K8mxANc=', 'Content-Length': '0'} 2022-09-12T19:06:58.401 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007611fe6dd6679726-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.402 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.403 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.403 DEBUG:boto:Method: HEAD 2022-09-12T19:06:58.404 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:58.404 DEBUG:boto:Data: 2022-09-12T19:06:58.405 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.405 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.406 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.407 DEBUG:boto:Params: {} 2022-09-12T19:06:58.407 DEBUG:boto:Token: None 2022-09-12T19:06:58.408 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj 2022-09-12T19:06:58.408 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Y2Fe3XT78Ao/jgDmCP3K8mxANc= 2022-09-12T19:06:58.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Y2Fe3XT78Ao/jgDmCP3K8mxANc=', 'Content-Length': '0'} 2022-09-12T19:06:58.413 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b4d16fa7f0b00539-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:58.415 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.415 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.416 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.417 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.417 DEBUG:boto:Method: GET 2022-09-12T19:06:58.418 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.419 DEBUG:boto:Data: 2022-09-12T19:06:58.421 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.424 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.424 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.425 DEBUG:boto:Params: {} 2022-09-12T19:06:58.426 DEBUG:boto:Token: None 2022-09-12T19:06:58.428 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.433 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B7co6InOJZmjD+FMGQlHUp/72B0= 2022-09-12T19:06:58.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B7co6InOJZmjD+FMGQlHUp/72B0=', 'Content-Length': '0'} 2022-09-12T19:06:58.438 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ae7dc1ecca98926-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.439 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.440 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.440 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.441 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.441 DEBUG:boto:Method: GET 2022-09-12T19:06:58.442 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.443 DEBUG:boto:Data: 2022-09-12T19:06:58.443 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.444 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.445 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.445 DEBUG:boto:Params: {} 2022-09-12T19:06:58.446 DEBUG:boto:Token: None 2022-09-12T19:06:58.447 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.447 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B7co6InOJZmjD+FMGQlHUp/72B0= 2022-09-12T19:06:58.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B7co6InOJZmjD+FMGQlHUp/72B0=', 'Content-Length': '0'} 2022-09-12T19:06:58.450 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037b23900ccb0b9b7-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.452 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:58.453 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:58.454 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.455 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.456 DEBUG:boto:Method: GET 2022-09-12T19:06:58.457 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:58.458 DEBUG:boto:Data: 2022-09-12T19:06:58.459 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.460 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.461 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.462 DEBUG:boto:Params: {} 2022-09-12T19:06:58.463 DEBUG:boto:Token: None 2022-09-12T19:06:58.463 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/ 2022-09-12T19:06:58.464 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg= 2022-09-12T19:06:58.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg=', 'Content-Length': '0'} 2022-09-12T19:06:58.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d81d37e06c0d768f-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.469 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:58.471 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:58.472 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:58.472 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.473 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.473 DEBUG:boto:Method: GET 2022-09-12T19:06:58.474 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:58.475 DEBUG:boto:Data: 2022-09-12T19:06:58.475 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.476 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.476 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.477 DEBUG:boto:Params: {} 2022-09-12T19:06:58.478 DEBUG:boto:Token: None 2022-09-12T19:06:58.479 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/ 2022-09-12T19:06:58.479 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg= 2022-09-12T19:06:58.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg=', 'Content-Length': '0'} 2022-09-12T19:06:58.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d5b6e7985f6ddda-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.526 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:58.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:58.528 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.529 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.530 DEBUG:boto:path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:58.530 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:58.531 DEBUG:boto:Method: GET 2022-09-12T19:06:58.531 DEBUG:boto:Path: /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:58.532 DEBUG:boto:Data: 2022-09-12T19:06:58.532 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.533 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.533 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.534 DEBUG:boto:Params: {} 2022-09-12T19:06:58.535 DEBUG:boto:Token: None 2022-09-12T19:06:58.535 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:58.536 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IdjBagOEihV69z4Tyj74tEaP2kg= 2022-09-12T19:06:58.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IdjBagOEihV69z4Tyj74tEaP2kg=', 'Content-Length': '0'} 2022-09-12T19:06:58.540 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b57f15adfed715a7-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.541 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.542 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.543 DEBUG:boto:path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:58.544 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:58.545 DEBUG:boto:Method: GET 2022-09-12T19:06:58.546 DEBUG:boto:Path: /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:58.546 DEBUG:boto:Data: 2022-09-12T19:06:58.547 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.548 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.548 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.549 DEBUG:boto:Params: {} 2022-09-12T19:06:58.549 DEBUG:boto:Token: None 2022-09-12T19:06:58.550 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:58.551 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IdjBagOEihV69z4Tyj74tEaP2kg= 2022-09-12T19:06:58.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IdjBagOEihV69z4Tyj74tEaP2kg=', 'Content-Length': '0'} 2022-09-12T19:06:58.554 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ef381ad4dc10f1d-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.555 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.555 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.556 DEBUG:boto:path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:58.556 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:58.557 DEBUG:boto:Method: HEAD 2022-09-12T19:06:58.558 DEBUG:boto:Path: /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:58.558 DEBUG:boto:Data: 2022-09-12T19:06:58.559 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.559 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.560 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.560 DEBUG:boto:Params: {} 2022-09-12T19:06:58.561 DEBUG:boto:Token: None 2022-09-12T19:06:58.562 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:58.562 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:i2gh7kfnHv3lLbk7PaxFmevr1ag= 2022-09-12T19:06:58.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:i2gh7kfnHv3lLbk7PaxFmevr1ag=', 'Content-Length': '0'} 2022-09-12T19:06:58.565 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a27a5a2d1385cbdb-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.566 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.567 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.568 DEBUG:boto:path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:58.569 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:58.569 DEBUG:boto:Method: HEAD 2022-09-12T19:06:58.570 DEBUG:boto:Path: /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:58.571 DEBUG:boto:Data: 2022-09-12T19:06:58.571 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.572 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.572 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.573 DEBUG:boto:Params: {} 2022-09-12T19:06:58.574 DEBUG:boto:Token: None 2022-09-12T19:06:58.574 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:58.576 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:i2gh7kfnHv3lLbk7PaxFmevr1ag= 2022-09-12T19:06:58.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:i2gh7kfnHv3lLbk7PaxFmevr1ag=', 'Content-Length': '0'} 2022-09-12T19:06:58.579 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b63480a88906324a-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:58.580 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.580 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.581 DEBUG:boto:path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:58.581 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:58.582 DEBUG:boto:Method: GET 2022-09-12T19:06:58.583 DEBUG:boto:Path: /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:58.584 DEBUG:boto:Data: 2022-09-12T19:06:58.584 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.585 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.585 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.586 DEBUG:boto:Params: {} 2022-09-12T19:06:58.587 DEBUG:boto:Token: None 2022-09-12T19:06:58.587 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:58.588 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IdjBagOEihV69z4Tyj74tEaP2kg= 2022-09-12T19:06:58.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IdjBagOEihV69z4Tyj74tEaP2kg=', 'Content-Length': '0'} 2022-09-12T19:06:58.590 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b4895994f88dea71-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.591 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.592 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.593 DEBUG:boto:path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:58.594 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:58.594 DEBUG:boto:Method: GET 2022-09-12T19:06:58.595 DEBUG:boto:Path: /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:58.596 DEBUG:boto:Data: 2022-09-12T19:06:58.596 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.597 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.597 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.598 DEBUG:boto:Params: {} 2022-09-12T19:06:58.598 DEBUG:boto:Token: None 2022-09-12T19:06:58.599 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:06:58.600 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IdjBagOEihV69z4Tyj74tEaP2kg= 2022-09-12T19:06:58.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IdjBagOEihV69z4Tyj74tEaP2kg=', 'Content-Length': '0'} 2022-09-12T19:06:58.604 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000162e90c8bd8d0d81-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.646 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:58.647 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:58.647 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.648 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.649 DEBUG:boto:Method: GET 2022-09-12T19:06:58.649 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:58.650 DEBUG:boto:Data: 2022-09-12T19:06:58.651 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.651 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.652 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.652 DEBUG:boto:Params: {} 2022-09-12T19:06:58.653 DEBUG:boto:Token: None 2022-09-12T19:06:58.653 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/ 2022-09-12T19:06:58.654 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg= 2022-09-12T19:06:58.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg=', 'Content-Length': '0'} 2022-09-12T19:06:58.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051573a8aa62d2590-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.659 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:58.660 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:58.661 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:58.661 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.662 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.662 DEBUG:boto:Method: GET 2022-09-12T19:06:58.663 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:58.663 DEBUG:boto:Data: 2022-09-12T19:06:58.664 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.665 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.665 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.666 DEBUG:boto:Params: {} 2022-09-12T19:06:58.666 DEBUG:boto:Token: None 2022-09-12T19:06:58.667 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/ 2022-09-12T19:06:58.668 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg= 2022-09-12T19:06:58.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg=', 'Content-Length': '0'} 2022-09-12T19:06:58.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d660e3aa0c853986-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.672 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:58.673 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.673 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.674 DEBUG:boto:Method: HEAD 2022-09-12T19:06:58.675 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:58.675 DEBUG:boto:Data: 2022-09-12T19:06:58.676 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.676 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.677 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.677 DEBUG:boto:Params: {} 2022-09-12T19:06:58.679 DEBUG:boto:Token: None 2022-09-12T19:06:58.679 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj 2022-09-12T19:06:58.680 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Y2Fe3XT78Ao/jgDmCP3K8mxANc= 2022-09-12T19:06:58.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Y2Fe3XT78Ao/jgDmCP3K8mxANc=', 'Content-Length': '0'} 2022-09-12T19:06:58.684 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000048283de1734fe72-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.685 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.686 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.687 DEBUG:boto:Method: HEAD 2022-09-12T19:06:58.687 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:58.688 DEBUG:boto:Data: 2022-09-12T19:06:58.688 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.689 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.689 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.690 DEBUG:boto:Params: {} 2022-09-12T19:06:58.690 DEBUG:boto:Token: None 2022-09-12T19:06:58.691 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj 2022-09-12T19:06:58.692 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Y2Fe3XT78Ao/jgDmCP3K8mxANc= 2022-09-12T19:06:58.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Y2Fe3XT78Ao/jgDmCP3K8mxANc=', 'Content-Length': '0'} 2022-09-12T19:06:58.696 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000782d8cdb0c408bd1-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:58.698 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.698 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.699 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.699 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.700 DEBUG:boto:Method: GET 2022-09-12T19:06:58.701 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.701 DEBUG:boto:Data: 2022-09-12T19:06:58.702 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.702 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.703 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.703 DEBUG:boto:Params: {} 2022-09-12T19:06:58.704 DEBUG:boto:Token: None 2022-09-12T19:06:58.704 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.705 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B7co6InOJZmjD+FMGQlHUp/72B0= 2022-09-12T19:06:58.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B7co6InOJZmjD+FMGQlHUp/72B0=', 'Content-Length': '0'} 2022-09-12T19:06:58.708 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0529051b7bb34b3-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.750 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.751 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.751 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.752 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.752 DEBUG:boto:Method: GET 2022-09-12T19:06:58.753 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.754 DEBUG:boto:Data: 2022-09-12T19:06:58.754 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.755 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.755 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.756 DEBUG:boto:Params: {} 2022-09-12T19:06:58.757 DEBUG:boto:Token: None 2022-09-12T19:06:58.757 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.758 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B7co6InOJZmjD+FMGQlHUp/72B0= 2022-09-12T19:06:58.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B7co6InOJZmjD+FMGQlHUp/72B0=', 'Content-Length': '0'} 2022-09-12T19:06:58.761 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd1c36a55a21efa1-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.762 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:58.762 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:58.763 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.764 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.764 DEBUG:boto:Method: GET 2022-09-12T19:06:58.765 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:58.766 DEBUG:boto:Data: 2022-09-12T19:06:58.766 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.767 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.767 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.768 DEBUG:boto:Params: {} 2022-09-12T19:06:58.768 DEBUG:boto:Token: None 2022-09-12T19:06:58.769 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/ 2022-09-12T19:06:58.770 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg= 2022-09-12T19:06:58.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg=', 'Content-Length': '0'} 2022-09-12T19:06:58.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064fa46e2c2a014ca-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.774 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:58.775 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:58.776 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:58.776 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.777 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.777 DEBUG:boto:Method: GET 2022-09-12T19:06:58.778 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:58.778 DEBUG:boto:Data: 2022-09-12T19:06:58.779 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.780 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.780 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.781 DEBUG:boto:Params: {} 2022-09-12T19:06:58.781 DEBUG:boto:Token: None 2022-09-12T19:06:58.782 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/ 2022-09-12T19:06:58.783 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg= 2022-09-12T19:06:58.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg=', 'Content-Length': '0'} 2022-09-12T19:06:58.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007455041721acb337-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.788 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:58.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:58.789 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.790 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.791 DEBUG:boto:path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:58.791 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:58.792 DEBUG:boto:Method: GET 2022-09-12T19:06:58.792 DEBUG:boto:Path: /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:58.793 DEBUG:boto:Data: 2022-09-12T19:06:58.793 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.794 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.794 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.795 DEBUG:boto:Params: {} 2022-09-12T19:06:58.796 DEBUG:boto:Token: None 2022-09-12T19:06:58.796 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:58.797 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dRQhnZdgjWe0kfK1vBSmK21dViE= 2022-09-12T19:06:58.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dRQhnZdgjWe0kfK1vBSmK21dViE=', 'Content-Length': '0'} 2022-09-12T19:06:58.800 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'j0eE53eOEuIue0LhYTVuZStUTmcOj7o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3d06ce461fbd14e-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.801 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.801 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.802 DEBUG:boto:path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:58.802 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:58.803 DEBUG:boto:Method: GET 2022-09-12T19:06:58.803 DEBUG:boto:Path: /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:58.804 DEBUG:boto:Data: 2022-09-12T19:06:58.805 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.805 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.806 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.807 DEBUG:boto:Params: {} 2022-09-12T19:06:58.807 DEBUG:boto:Token: None 2022-09-12T19:06:58.808 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:58.809 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dRQhnZdgjWe0kfK1vBSmK21dViE= 2022-09-12T19:06:58.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dRQhnZdgjWe0kfK1vBSmK21dViE=', 'Content-Length': '0'} 2022-09-12T19:06:58.811 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'j0eE53eOEuIue0LhYTVuZStUTmcOj7o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003171da118d80b392-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.854 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.855 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.855 DEBUG:boto:path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:58.856 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:58.856 DEBUG:boto:Method: HEAD 2022-09-12T19:06:58.857 DEBUG:boto:Path: /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:58.857 DEBUG:boto:Data: 2022-09-12T19:06:58.858 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.858 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.859 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.859 DEBUG:boto:Params: {} 2022-09-12T19:06:58.860 DEBUG:boto:Token: None 2022-09-12T19:06:58.860 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:58.861 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:INVQuvAPVTRqAVCqlS15nr8nD0I= 2022-09-12T19:06:58.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:INVQuvAPVTRqAVCqlS15nr8nD0I=', 'Content-Length': '0'} 2022-09-12T19:06:58.864 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'j0eE53eOEuIue0LhYTVuZStUTmcOj7o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0383678d422bdfc-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.865 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.865 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.866 DEBUG:boto:path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:58.866 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:58.867 DEBUG:boto:Method: HEAD 2022-09-12T19:06:58.867 DEBUG:boto:Path: /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:58.868 DEBUG:boto:Data: 2022-09-12T19:06:58.868 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.869 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.869 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.870 DEBUG:boto:Params: {} 2022-09-12T19:06:58.870 DEBUG:boto:Token: None 2022-09-12T19:06:58.871 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:58.871 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:INVQuvAPVTRqAVCqlS15nr8nD0I= 2022-09-12T19:06:58.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:INVQuvAPVTRqAVCqlS15nr8nD0I=', 'Content-Length': '0'} 2022-09-12T19:06:58.874 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'j0eE53eOEuIue0LhYTVuZStUTmcOj7o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000aa9634cc2c72632-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:58.875 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.876 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.876 DEBUG:boto:path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:58.877 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:58.878 DEBUG:boto:Method: GET 2022-09-12T19:06:58.878 DEBUG:boto:Path: /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:58.879 DEBUG:boto:Data: 2022-09-12T19:06:58.879 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.880 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.880 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.881 DEBUG:boto:Params: {} 2022-09-12T19:06:58.881 DEBUG:boto:Token: None 2022-09-12T19:06:58.882 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:58.882 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dRQhnZdgjWe0kfK1vBSmK21dViE= 2022-09-12T19:06:58.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dRQhnZdgjWe0kfK1vBSmK21dViE=', 'Content-Length': '0'} 2022-09-12T19:06:58.885 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'j0eE53eOEuIue0LhYTVuZStUTmcOj7o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001cbed59c767933df-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.886 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.887 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.887 DEBUG:boto:path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:58.888 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:58.888 DEBUG:boto:Method: GET 2022-09-12T19:06:58.889 DEBUG:boto:Path: /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:58.889 DEBUG:boto:Data: 2022-09-12T19:06:58.890 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.890 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.891 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.891 DEBUG:boto:Params: {} 2022-09-12T19:06:58.892 DEBUG:boto:Token: None 2022-09-12T19:06:58.892 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:06:58.893 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dRQhnZdgjWe0kfK1vBSmK21dViE= 2022-09-12T19:06:58.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dRQhnZdgjWe0kfK1vBSmK21dViE=', 'Content-Length': '0'} 2022-09-12T19:06:58.896 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'j0eE53eOEuIue0LhYTVuZStUTmcOj7o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000207745486aee8368-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.897 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:58.897 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:58.898 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.898 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.899 DEBUG:boto:Method: GET 2022-09-12T19:06:58.899 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:58.900 DEBUG:boto:Data: 2022-09-12T19:06:58.901 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.901 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.902 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.902 DEBUG:boto:Params: {} 2022-09-12T19:06:58.903 DEBUG:boto:Token: None 2022-09-12T19:06:58.903 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/ 2022-09-12T19:06:58.904 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg= 2022-09-12T19:06:58.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg=', 'Content-Length': '0'} 2022-09-12T19:06:58.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6d711c0c8435b77-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.908 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:58.909 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:58.910 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:58.910 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.911 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:58.911 DEBUG:boto:Method: GET 2022-09-12T19:06:58.912 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:58.912 DEBUG:boto:Data: 2022-09-12T19:06:58.913 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.914 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.914 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.915 DEBUG:boto:Params: {} 2022-09-12T19:06:58.915 DEBUG:boto:Token: None 2022-09-12T19:06:58.916 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/ 2022-09-12T19:06:58.916 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg= 2022-09-12T19:06:58.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6NDKCySWqIU+vUeMfXZHgRVXDXg=', 'Content-Length': '0'} 2022-09-12T19:06:58.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8ce10325dde5771-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.921 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:58.921 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.922 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.922 DEBUG:boto:Method: HEAD 2022-09-12T19:06:58.923 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:58.923 DEBUG:boto:Data: 2022-09-12T19:06:58.924 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.924 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.925 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.925 DEBUG:boto:Params: {} 2022-09-12T19:06:58.926 DEBUG:boto:Token: None 2022-09-12T19:06:58.926 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj 2022-09-12T19:06:58.927 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Y2Fe3XT78Ao/jgDmCP3K8mxANc= 2022-09-12T19:06:58.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Y2Fe3XT78Ao/jgDmCP3K8mxANc=', 'Content-Length': '0'} 2022-09-12T19:06:58.931 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dce7009990186d98-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.931 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.932 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.932 DEBUG:boto:Method: HEAD 2022-09-12T19:06:58.933 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:58.933 DEBUG:boto:Data: 2022-09-12T19:06:58.934 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.934 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:58.935 DEBUG:boto:Port: 8000 2022-09-12T19:06:58.935 DEBUG:boto:Params: {} 2022-09-12T19:06:58.936 DEBUG:boto:Token: None 2022-09-12T19:06:58.936 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj 2022-09-12T19:06:58.937 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Y2Fe3XT78Ao/jgDmCP3K8mxANc= 2022-09-12T19:06:58.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Y2Fe3XT78Ao/jgDmCP3K8mxANc=', 'Content-Length': '0'} 2022-09-12T19:06:58.941 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000647bae3255e59fc3-00631f8352-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:58.942 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.943 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.943 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.944 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.944 DEBUG:boto:Method: GET 2022-09-12T19:06:58.945 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.945 DEBUG:boto:Data: 2022-09-12T19:06:58.946 DEBUG:boto:Headers: {} 2022-09-12T19:06:58.946 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:58.947 DEBUG:boto:Port: 8001 2022-09-12T19:06:58.947 DEBUG:boto:Params: {} 2022-09-12T19:06:58.948 DEBUG:boto:Token: None 2022-09-12T19:06:58.949 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:58 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.950 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B7co6InOJZmjD+FMGQlHUp/72B0= 2022-09-12T19:06:58.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B7co6InOJZmjD+FMGQlHUp/72B0=', 'Content-Length': '0'} 2022-09-12T19:06:58.953 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000133e53fc74fed1e0-00631f8352-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:58.994 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:58.995 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:58.996 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.997 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.998 DEBUG:boto:Method: GET 2022-09-12T19:06:58.999 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:58.999 DEBUG:boto:Data: 2022-09-12T19:06:59.000 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.000 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:59.001 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.001 DEBUG:boto:Params: {} 2022-09-12T19:06:59.002 DEBUG:boto:Token: None 2022-09-12T19:06:59.003 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:59.003 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6sHDJ5ygzXk2JJJOw9Wch8ih4qk= 2022-09-12T19:06:59.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6sHDJ5ygzXk2JJJOw9Wch8ih4qk=', 'Content-Length': '0'} 2022-09-12T19:06:59.006 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000507d37de57ca5af9-00631f8353-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.007 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:59.008 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:59.008 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.009 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.009 DEBUG:boto:Method: GET 2022-09-12T19:06:59.010 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:59.010 DEBUG:boto:Data: 2022-09-12T19:06:59.011 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.011 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.012 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.013 DEBUG:boto:Params: {} 2022-09-12T19:06:59.013 DEBUG:boto:Token: None 2022-09-12T19:06:59.014 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/ 2022-09-12T19:06:59.014 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us= 2022-09-12T19:06:59.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us=', 'Content-Length': '0'} 2022-09-12T19:06:59.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c398dafe4242a08a-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.019 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:59.020 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:59.021 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:59.021 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.022 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.022 DEBUG:boto:Method: GET 2022-09-12T19:06:59.023 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:59.023 DEBUG:boto:Data: 2022-09-12T19:06:59.024 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.025 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:59.025 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.026 DEBUG:boto:Params: {} 2022-09-12T19:06:59.026 DEBUG:boto:Token: None 2022-09-12T19:06:59.027 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/ 2022-09-12T19:06:59.027 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us= 2022-09-12T19:06:59.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us=', 'Content-Length': '0'} 2022-09-12T19:06:59.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000780e8650a2f009ce-00631f8353-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.074 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:59.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:59.076 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.076 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.077 DEBUG:boto:path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:59.077 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:59.078 DEBUG:boto:Method: GET 2022-09-12T19:06:59.078 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:59.079 DEBUG:boto:Data: 2022-09-12T19:06:59.079 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.080 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.080 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.081 DEBUG:boto:Params: {} 2022-09-12T19:06:59.081 DEBUG:boto:Token: None 2022-09-12T19:06:59.082 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:59.083 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XuGAyIg19WR114SgAVC54pgckuE= 2022-09-12T19:06:59.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XuGAyIg19WR114SgAVC54pgckuE=', 'Content-Length': '0'} 2022-09-12T19:06:59.086 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be8b2efc4d499b20-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.087 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.087 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.088 DEBUG:boto:path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:59.088 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:59.089 DEBUG:boto:Method: GET 2022-09-12T19:06:59.089 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:59.090 DEBUG:boto:Data: 2022-09-12T19:06:59.091 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.091 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:59.092 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.092 DEBUG:boto:Params: {} 2022-09-12T19:06:59.093 DEBUG:boto:Token: None 2022-09-12T19:06:59.094 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:59.094 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XuGAyIg19WR114SgAVC54pgckuE= 2022-09-12T19:06:59.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XuGAyIg19WR114SgAVC54pgckuE=', 'Content-Length': '0'} 2022-09-12T19:06:59.097 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a5fb63af9eb2a58-00631f8353-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.098 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.099 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.099 DEBUG:boto:path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:59.100 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:59.100 DEBUG:boto:Method: HEAD 2022-09-12T19:06:59.101 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:59.102 DEBUG:boto:Data: 2022-09-12T19:06:59.103 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.104 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.104 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.105 DEBUG:boto:Params: {} 2022-09-12T19:06:59.106 DEBUG:boto:Token: None 2022-09-12T19:06:59.107 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:59.108 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:77YYNkQzR6/8GvVQKMG4Hvx5udY= 2022-09-12T19:06:59.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:77YYNkQzR6/8GvVQKMG4Hvx5udY=', 'Content-Length': '0'} 2022-09-12T19:06:59.111 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008070431112c76533-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.112 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.112 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.113 DEBUG:boto:path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:59.113 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:59.114 DEBUG:boto:Method: HEAD 2022-09-12T19:06:59.114 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:59.115 DEBUG:boto:Data: 2022-09-12T19:06:59.116 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.116 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:59.117 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.118 DEBUG:boto:Params: {} 2022-09-12T19:06:59.119 DEBUG:boto:Token: None 2022-09-12T19:06:59.120 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:59.121 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:77YYNkQzR6/8GvVQKMG4Hvx5udY= 2022-09-12T19:06:59.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:77YYNkQzR6/8GvVQKMG4Hvx5udY=', 'Content-Length': '0'} 2022-09-12T19:06:59.124 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004db61083a11566d1-00631f8353-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:59.125 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.125 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.126 DEBUG:boto:path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:59.127 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:59.127 DEBUG:boto:Method: GET 2022-09-12T19:06:59.128 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:59.128 DEBUG:boto:Data: 2022-09-12T19:06:59.129 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.129 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.130 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.130 DEBUG:boto:Params: {} 2022-09-12T19:06:59.131 DEBUG:boto:Token: None 2022-09-12T19:06:59.131 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:59.132 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XuGAyIg19WR114SgAVC54pgckuE= 2022-09-12T19:06:59.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XuGAyIg19WR114SgAVC54pgckuE=', 'Content-Length': '0'} 2022-09-12T19:06:59.135 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024af0e092ec95286-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.178 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.179 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.179 DEBUG:boto:path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:59.180 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:59.180 DEBUG:boto:Method: GET 2022-09-12T19:06:59.181 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:59.181 DEBUG:boto:Data: 2022-09-12T19:06:59.182 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.183 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:59.183 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.184 DEBUG:boto:Params: {} 2022-09-12T19:06:59.184 DEBUG:boto:Token: None 2022-09-12T19:06:59.185 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:06:59.185 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XuGAyIg19WR114SgAVC54pgckuE= 2022-09-12T19:06:59.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XuGAyIg19WR114SgAVC54pgckuE=', 'Content-Length': '0'} 2022-09-12T19:06:59.188 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029a7d92f6ea8c441-00631f8353-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.189 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:59.190 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:59.190 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.191 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.191 DEBUG:boto:Method: GET 2022-09-12T19:06:59.192 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:59.193 DEBUG:boto:Data: 2022-09-12T19:06:59.193 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.194 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.194 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.195 DEBUG:boto:Params: {} 2022-09-12T19:06:59.196 DEBUG:boto:Token: None 2022-09-12T19:06:59.196 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/ 2022-09-12T19:06:59.197 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us= 2022-09-12T19:06:59.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us=', 'Content-Length': '0'} 2022-09-12T19:06:59.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e8280ae99952db8-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.202 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:59.202 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:59.203 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:59.204 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.204 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.205 DEBUG:boto:Method: GET 2022-09-12T19:06:59.205 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:59.206 DEBUG:boto:Data: 2022-09-12T19:06:59.206 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.207 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:59.207 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.208 DEBUG:boto:Params: {} 2022-09-12T19:06:59.209 DEBUG:boto:Token: None 2022-09-12T19:06:59.209 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/ 2022-09-12T19:06:59.210 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us= 2022-09-12T19:06:59.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us=', 'Content-Length': '0'} 2022-09-12T19:06:59.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ee0f8720c2bd084-00631f8353-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.214 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:59.215 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.216 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.216 DEBUG:boto:Method: HEAD 2022-09-12T19:06:59.217 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:59.218 DEBUG:boto:Data: 2022-09-12T19:06:59.218 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.219 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.219 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.220 DEBUG:boto:Params: {} 2022-09-12T19:06:59.220 DEBUG:boto:Token: None 2022-09-12T19:06:59.221 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj 2022-09-12T19:06:59.221 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WMpQAVPw2A1m4wO+0LgeIb+sj7U= 2022-09-12T19:06:59.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WMpQAVPw2A1m4wO+0LgeIb+sj7U=', 'Content-Length': '0'} 2022-09-12T19:06:59.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000795d7ad06adff680-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.226 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.227 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.227 DEBUG:boto:Method: HEAD 2022-09-12T19:06:59.228 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:59.229 DEBUG:boto:Data: 2022-09-12T19:06:59.229 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.230 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:59.230 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.231 DEBUG:boto:Params: {} 2022-09-12T19:06:59.231 DEBUG:boto:Token: None 2022-09-12T19:06:59.232 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj 2022-09-12T19:06:59.232 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WMpQAVPw2A1m4wO+0LgeIb+sj7U= 2022-09-12T19:06:59.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WMpQAVPw2A1m4wO+0LgeIb+sj7U=', 'Content-Length': '0'} 2022-09-12T19:06:59.237 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002dfecdceaef8dbaa-00631f8353-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:59.238 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.239 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.239 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:59.240 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:59.240 DEBUG:boto:Method: GET 2022-09-12T19:06:59.241 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:59.241 DEBUG:boto:Data: 2022-09-12T19:06:59.242 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.243 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.243 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.244 DEBUG:boto:Params: {} 2022-09-12T19:06:59.244 DEBUG:boto:Token: None 2022-09-12T19:06:59.245 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:59.246 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6sHDJ5ygzXk2JJJOw9Wch8ih4qk= 2022-09-12T19:06:59.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6sHDJ5ygzXk2JJJOw9Wch8ih4qk=', 'Content-Length': '0'} 2022-09-12T19:06:59.248 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfaa397f82a3f0ca-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.249 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.250 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.250 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:59.251 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:59.251 DEBUG:boto:Method: GET 2022-09-12T19:06:59.252 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:59.253 DEBUG:boto:Data: 2022-09-12T19:06:59.253 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.254 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:59.254 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.255 DEBUG:boto:Params: {} 2022-09-12T19:06:59.255 DEBUG:boto:Token: None 2022-09-12T19:06:59.256 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:59.257 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6sHDJ5ygzXk2JJJOw9Wch8ih4qk= 2022-09-12T19:06:59.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6sHDJ5ygzXk2JJJOw9Wch8ih4qk=', 'Content-Length': '0'} 2022-09-12T19:06:59.259 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d8a26c7fbc956d3-00631f8353-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.260 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:59.261 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:59.261 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.262 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.262 DEBUG:boto:Method: GET 2022-09-12T19:06:59.263 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:59.263 DEBUG:boto:Data: 2022-09-12T19:06:59.264 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.264 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.265 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.265 DEBUG:boto:Params: {} 2022-09-12T19:06:59.266 DEBUG:boto:Token: None 2022-09-12T19:06:59.267 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/ 2022-09-12T19:06:59.267 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us= 2022-09-12T19:06:59.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us=', 'Content-Length': '0'} 2022-09-12T19:06:59.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5d1070585069fc6-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.272 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:59.272 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:59.273 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:59.273 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.274 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.275 DEBUG:boto:Method: GET 2022-09-12T19:06:59.275 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:59.276 DEBUG:boto:Data: 2022-09-12T19:06:59.276 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.277 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:59.277 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.278 DEBUG:boto:Params: {} 2022-09-12T19:06:59.278 DEBUG:boto:Token: None 2022-09-12T19:06:59.279 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/ 2022-09-12T19:06:59.279 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us= 2022-09-12T19:06:59.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us=', 'Content-Length': '0'} 2022-09-12T19:06:59.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005239a798456ed685-00631f8353-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.284 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:59.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:59.285 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.286 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.286 DEBUG:boto:path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:59.287 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:59.287 DEBUG:boto:Method: GET 2022-09-12T19:06:59.288 DEBUG:boto:Path: /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:59.288 DEBUG:boto:Data: 2022-09-12T19:06:59.289 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.289 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.290 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.291 DEBUG:boto:Params: {} 2022-09-12T19:06:59.291 DEBUG:boto:Token: None 2022-09-12T19:06:59.292 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:59.292 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YQbG5ecKhY6UkFy1/OZ5IInlnnw= 2022-09-12T19:06:59.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YQbG5ecKhY6UkFy1/OZ5IInlnnw=', 'Content-Length': '0'} 2022-09-12T19:06:59.295 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'JcC9N13lklBVOx234Q8MXQ8kAM8ByWg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a014e735785da5b5-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.296 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.297 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.297 DEBUG:boto:path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:59.298 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:59.298 DEBUG:boto:Method: GET 2022-09-12T19:06:59.299 DEBUG:boto:Path: /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:59.300 DEBUG:boto:Data: 2022-09-12T19:06:59.300 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.301 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:59.301 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.302 DEBUG:boto:Params: {} 2022-09-12T19:06:59.302 DEBUG:boto:Token: None 2022-09-12T19:06:59.303 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:59.303 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YQbG5ecKhY6UkFy1/OZ5IInlnnw= 2022-09-12T19:06:59.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YQbG5ecKhY6UkFy1/OZ5IInlnnw=', 'Content-Length': '0'} 2022-09-12T19:06:59.307 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'JcC9N13lklBVOx234Q8MXQ8kAM8ByWg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000213051e051ddf016-00631f8353-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.307 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.308 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.308 DEBUG:boto:path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:59.309 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:59.309 DEBUG:boto:Method: HEAD 2022-09-12T19:06:59.310 DEBUG:boto:Path: /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:59.310 DEBUG:boto:Data: 2022-09-12T19:06:59.311 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.311 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.312 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.312 DEBUG:boto:Params: {} 2022-09-12T19:06:59.313 DEBUG:boto:Token: None 2022-09-12T19:06:59.313 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:59.314 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cTIM3fWMocvAB75mfg3gxTe0n/Q= 2022-09-12T19:06:59.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cTIM3fWMocvAB75mfg3gxTe0n/Q=', 'Content-Length': '0'} 2022-09-12T19:06:59.317 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'JcC9N13lklBVOx234Q8MXQ8kAM8ByWg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000005741707fb58078e-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.318 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.318 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.319 DEBUG:boto:path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:59.320 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:59.321 DEBUG:boto:Method: HEAD 2022-09-12T19:06:59.322 DEBUG:boto:Path: /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:59.323 DEBUG:boto:Data: 2022-09-12T19:06:59.324 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.325 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:59.326 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.327 DEBUG:boto:Params: {} 2022-09-12T19:06:59.327 DEBUG:boto:Token: None 2022-09-12T19:06:59.328 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:59.329 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cTIM3fWMocvAB75mfg3gxTe0n/Q= 2022-09-12T19:06:59.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cTIM3fWMocvAB75mfg3gxTe0n/Q=', 'Content-Length': '0'} 2022-09-12T19:06:59.331 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'JcC9N13lklBVOx234Q8MXQ8kAM8ByWg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c57f1d6022c9f8d1-00631f8353-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:59.333 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.333 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.334 DEBUG:boto:path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:59.334 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:59.335 DEBUG:boto:Method: GET 2022-09-12T19:06:59.336 DEBUG:boto:Path: /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:59.336 DEBUG:boto:Data: 2022-09-12T19:06:59.337 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.337 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.338 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.338 DEBUG:boto:Params: {} 2022-09-12T19:06:59.339 DEBUG:boto:Token: None 2022-09-12T19:06:59.340 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:59.340 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YQbG5ecKhY6UkFy1/OZ5IInlnnw= 2022-09-12T19:06:59.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YQbG5ecKhY6UkFy1/OZ5IInlnnw=', 'Content-Length': '0'} 2022-09-12T19:06:59.343 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'JcC9N13lklBVOx234Q8MXQ8kAM8ByWg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026e0736197ece9db-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.344 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.344 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.345 DEBUG:boto:path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:59.345 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:59.346 DEBUG:boto:Method: GET 2022-09-12T19:06:59.346 DEBUG:boto:Path: /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:59.347 DEBUG:boto:Data: 2022-09-12T19:06:59.347 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.348 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:59.348 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.349 DEBUG:boto:Params: {} 2022-09-12T19:06:59.350 DEBUG:boto:Token: None 2022-09-12T19:06:59.350 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:06:59.351 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YQbG5ecKhY6UkFy1/OZ5IInlnnw= 2022-09-12T19:06:59.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YQbG5ecKhY6UkFy1/OZ5IInlnnw=', 'Content-Length': '0'} 2022-09-12T19:06:59.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'JcC9N13lklBVOx234Q8MXQ8kAM8ByWg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc6ac7bb1b157d28-00631f8353-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.394 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:59.395 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:59.395 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.396 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.396 DEBUG:boto:Method: GET 2022-09-12T19:06:59.397 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:59.398 DEBUG:boto:Data: 2022-09-12T19:06:59.398 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.399 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.399 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.400 DEBUG:boto:Params: {} 2022-09-12T19:06:59.401 DEBUG:boto:Token: None 2022-09-12T19:06:59.401 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/ 2022-09-12T19:06:59.402 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us= 2022-09-12T19:06:59.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us=', 'Content-Length': '0'} 2022-09-12T19:06:59.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9896697f13b4208-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.406 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:59.407 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:59.407 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:59.408 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.408 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.409 DEBUG:boto:Method: GET 2022-09-12T19:06:59.410 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:59.410 DEBUG:boto:Data: 2022-09-12T19:06:59.411 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.411 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:59.412 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.412 DEBUG:boto:Params: {} 2022-09-12T19:06:59.413 DEBUG:boto:Token: None 2022-09-12T19:06:59.413 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/ 2022-09-12T19:06:59.414 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us= 2022-09-12T19:06:59.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us=', 'Content-Length': '0'} 2022-09-12T19:06:59.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc469cbc2ea0e45d-00631f8353-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.419 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:59.420 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.420 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.421 DEBUG:boto:Method: HEAD 2022-09-12T19:06:59.421 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:59.422 DEBUG:boto:Data: 2022-09-12T19:06:59.422 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.423 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.423 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.424 DEBUG:boto:Params: {} 2022-09-12T19:06:59.424 DEBUG:boto:Token: None 2022-09-12T19:06:59.425 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj 2022-09-12T19:06:59.425 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WMpQAVPw2A1m4wO+0LgeIb+sj7U= 2022-09-12T19:06:59.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WMpQAVPw2A1m4wO+0LgeIb+sj7U=', 'Content-Length': '0'} 2022-09-12T19:06:59.430 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000938fd06d3f3cab0e-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.430 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.431 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.431 DEBUG:boto:Method: HEAD 2022-09-12T19:06:59.432 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:59.432 DEBUG:boto:Data: 2022-09-12T19:06:59.433 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.433 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:59.434 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.434 DEBUG:boto:Params: {} 2022-09-12T19:06:59.435 DEBUG:boto:Token: None 2022-09-12T19:06:59.436 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj 2022-09-12T19:06:59.436 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WMpQAVPw2A1m4wO+0LgeIb+sj7U= 2022-09-12T19:06:59.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WMpQAVPw2A1m4wO+0LgeIb+sj7U=', 'Content-Length': '0'} 2022-09-12T19:06:59.440 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b970e0555d6f649e-00631f8353-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:59.442 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.442 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.443 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:59.443 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:59.444 DEBUG:boto:Method: GET 2022-09-12T19:06:59.444 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:59.445 DEBUG:boto:Data: 2022-09-12T19:06:59.445 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.446 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.446 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.447 DEBUG:boto:Params: {} 2022-09-12T19:06:59.448 DEBUG:boto:Token: None 2022-09-12T19:06:59.448 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:59.449 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6sHDJ5ygzXk2JJJOw9Wch8ih4qk= 2022-09-12T19:06:59.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6sHDJ5ygzXk2JJJOw9Wch8ih4qk=', 'Content-Length': '0'} 2022-09-12T19:06:59.451 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e11a102c67d2e9bf-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.452 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.453 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.453 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:59.454 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:59.454 DEBUG:boto:Method: GET 2022-09-12T19:06:59.455 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:59.456 DEBUG:boto:Data: 2022-09-12T19:06:59.456 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.457 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:59.457 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.458 DEBUG:boto:Params: {} 2022-09-12T19:06:59.458 DEBUG:boto:Token: None 2022-09-12T19:06:59.459 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:59.460 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6sHDJ5ygzXk2JJJOw9Wch8ih4qk= 2022-09-12T19:06:59.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6sHDJ5ygzXk2JJJOw9Wch8ih4qk=', 'Content-Length': '0'} 2022-09-12T19:06:59.462 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064c960bc2eb7d9f7-00631f8353-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.506 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:59.507 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:59.507 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.508 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.509 DEBUG:boto:Method: GET 2022-09-12T19:06:59.509 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:59.510 DEBUG:boto:Data: 2022-09-12T19:06:59.511 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.511 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.512 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.512 DEBUG:boto:Params: {} 2022-09-12T19:06:59.513 DEBUG:boto:Token: None 2022-09-12T19:06:59.514 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/ 2022-09-12T19:06:59.514 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us= 2022-09-12T19:06:59.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us=', 'Content-Length': '0'} 2022-09-12T19:06:59.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1c93355b9704a5a-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.519 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:59.520 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:59.520 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:59.521 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.522 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.522 DEBUG:boto:Method: GET 2022-09-12T19:06:59.523 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:59.523 DEBUG:boto:Data: 2022-09-12T19:06:59.524 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.525 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:59.525 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.526 DEBUG:boto:Params: {} 2022-09-12T19:06:59.527 DEBUG:boto:Token: None 2022-09-12T19:06:59.527 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/ 2022-09-12T19:06:59.528 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us= 2022-09-12T19:06:59.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us=', 'Content-Length': '0'} 2022-09-12T19:06:59.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3ee0d0d738107dd-00631f8353-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.532 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:59.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:59.534 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.534 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.535 DEBUG:boto:path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:59.536 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:59.536 DEBUG:boto:Method: GET 2022-09-12T19:06:59.537 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:59.537 DEBUG:boto:Data: 2022-09-12T19:06:59.538 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.539 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.539 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.540 DEBUG:boto:Params: {} 2022-09-12T19:06:59.540 DEBUG:boto:Token: None 2022-09-12T19:06:59.541 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:59.541 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:a5dAPLbNVGy40uP78Ir99+B5ylo= 2022-09-12T19:06:59.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:a5dAPLbNVGy40uP78Ir99+B5ylo=', 'Content-Length': '0'} 2022-09-12T19:06:59.544 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c10467450c217f6b-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.586 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.587 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.588 DEBUG:boto:path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:59.588 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:59.589 DEBUG:boto:Method: GET 2022-09-12T19:06:59.590 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:59.590 DEBUG:boto:Data: 2022-09-12T19:06:59.591 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.592 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:59.592 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.593 DEBUG:boto:Params: {} 2022-09-12T19:06:59.593 DEBUG:boto:Token: None 2022-09-12T19:06:59.594 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:59.595 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:a5dAPLbNVGy40uP78Ir99+B5ylo= 2022-09-12T19:06:59.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:a5dAPLbNVGy40uP78Ir99+B5ylo=', 'Content-Length': '0'} 2022-09-12T19:06:59.597 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a19795d4477f7e38-00631f8353-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.598 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.599 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.599 DEBUG:boto:path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:59.600 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:59.600 DEBUG:boto:Method: HEAD 2022-09-12T19:06:59.601 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:59.601 DEBUG:boto:Data: 2022-09-12T19:06:59.602 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.603 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.603 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.604 DEBUG:boto:Params: {} 2022-09-12T19:06:59.604 DEBUG:boto:Token: None 2022-09-12T19:06:59.605 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:59.606 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cdLKOYapfhVrBi9fhlTL1fZfC0A= 2022-09-12T19:06:59.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cdLKOYapfhVrBi9fhlTL1fZfC0A=', 'Content-Length': '0'} 2022-09-12T19:06:59.608 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e85511ba30ad0e36-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.609 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.610 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.610 DEBUG:boto:path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:59.611 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:59.611 DEBUG:boto:Method: HEAD 2022-09-12T19:06:59.612 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:59.612 DEBUG:boto:Data: 2022-09-12T19:06:59.613 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.613 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:59.614 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.614 DEBUG:boto:Params: {} 2022-09-12T19:06:59.615 DEBUG:boto:Token: None 2022-09-12T19:06:59.615 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:59.616 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cdLKOYapfhVrBi9fhlTL1fZfC0A= 2022-09-12T19:06:59.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cdLKOYapfhVrBi9fhlTL1fZfC0A=', 'Content-Length': '0'} 2022-09-12T19:06:59.619 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ece51f36b937ae61-00631f8353-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:59.620 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.620 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.621 DEBUG:boto:path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:59.621 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:59.622 DEBUG:boto:Method: GET 2022-09-12T19:06:59.623 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:59.623 DEBUG:boto:Data: 2022-09-12T19:06:59.624 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.624 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.625 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.625 DEBUG:boto:Params: {} 2022-09-12T19:06:59.626 DEBUG:boto:Token: None 2022-09-12T19:06:59.626 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:59.627 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:a5dAPLbNVGy40uP78Ir99+B5ylo= 2022-09-12T19:06:59.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:a5dAPLbNVGy40uP78Ir99+B5ylo=', 'Content-Length': '0'} 2022-09-12T19:06:59.630 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7d6f5180390da49-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.631 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.631 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.632 DEBUG:boto:path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:59.632 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:59.633 DEBUG:boto:Method: GET 2022-09-12T19:06:59.633 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:59.634 DEBUG:boto:Data: 2022-09-12T19:06:59.634 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.635 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:59.635 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.636 DEBUG:boto:Params: {} 2022-09-12T19:06:59.637 DEBUG:boto:Token: None 2022-09-12T19:06:59.637 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:06:59.638 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:a5dAPLbNVGy40uP78Ir99+B5ylo= 2022-09-12T19:06:59.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:a5dAPLbNVGy40uP78Ir99+B5ylo=', 'Content-Length': '0'} 2022-09-12T19:06:59.640 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056ee5b0d197f5885-00631f8353-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.641 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:59.642 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:59.643 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.643 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.644 DEBUG:boto:Method: GET 2022-09-12T19:06:59.644 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:59.645 DEBUG:boto:Data: 2022-09-12T19:06:59.645 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.646 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.646 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.647 DEBUG:boto:Params: {} 2022-09-12T19:06:59.647 DEBUG:boto:Token: None 2022-09-12T19:06:59.648 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/ 2022-09-12T19:06:59.649 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us= 2022-09-12T19:06:59.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us=', 'Content-Length': '0'} 2022-09-12T19:06:59.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000106b156b8aec934-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.653 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:59.654 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:59.654 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:59.655 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.656 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.656 DEBUG:boto:Method: GET 2022-09-12T19:06:59.657 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:59.657 DEBUG:boto:Data: 2022-09-12T19:06:59.658 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.658 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:59.659 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.659 DEBUG:boto:Params: {} 2022-09-12T19:06:59.660 DEBUG:boto:Token: None 2022-09-12T19:06:59.661 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/ 2022-09-12T19:06:59.661 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us= 2022-09-12T19:06:59.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us=', 'Content-Length': '0'} 2022-09-12T19:06:59.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000604109df1adba26f-00631f8353-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.666 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:59.666 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.667 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.667 DEBUG:boto:Method: HEAD 2022-09-12T19:06:59.668 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:59.668 DEBUG:boto:Data: 2022-09-12T19:06:59.669 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.669 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.670 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.670 DEBUG:boto:Params: {} 2022-09-12T19:06:59.671 DEBUG:boto:Token: None 2022-09-12T19:06:59.672 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj 2022-09-12T19:06:59.672 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WMpQAVPw2A1m4wO+0LgeIb+sj7U= 2022-09-12T19:06:59.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WMpQAVPw2A1m4wO+0LgeIb+sj7U=', 'Content-Length': '0'} 2022-09-12T19:06:59.676 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d06540340d229147-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.677 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.678 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.678 DEBUG:boto:Method: HEAD 2022-09-12T19:06:59.679 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:06:59.679 DEBUG:boto:Data: 2022-09-12T19:06:59.680 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.680 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:59.681 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.681 DEBUG:boto:Params: {} 2022-09-12T19:06:59.682 DEBUG:boto:Token: None 2022-09-12T19:06:59.682 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj 2022-09-12T19:06:59.683 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WMpQAVPw2A1m4wO+0LgeIb+sj7U= 2022-09-12T19:06:59.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WMpQAVPw2A1m4wO+0LgeIb+sj7U=', 'Content-Length': '0'} 2022-09-12T19:06:59.687 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e6c7b8c2e95b181-00631f8353-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:06:59.690 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.691 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.692 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:59.693 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:59.694 DEBUG:boto:Method: GET 2022-09-12T19:06:59.695 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:59.696 DEBUG:boto:Data: 2022-09-12T19:06:59.697 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.698 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.699 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.700 DEBUG:boto:Params: {} 2022-09-12T19:06:59.701 DEBUG:boto:Token: None 2022-09-12T19:06:59.702 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:59.703 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6sHDJ5ygzXk2JJJOw9Wch8ih4qk= 2022-09-12T19:06:59.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6sHDJ5ygzXk2JJJOw9Wch8ih4qk=', 'Content-Length': '0'} 2022-09-12T19:06:59.707 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051794e8aa042edb3-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.750 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:06:59.752 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:06:59.753 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:59.754 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:59.755 DEBUG:boto:Method: GET 2022-09-12T19:06:59.757 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:59.757 DEBUG:boto:Data: 2022-09-12T19:06:59.758 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.758 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:59.759 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.760 DEBUG:boto:Params: {} 2022-09-12T19:06:59.760 DEBUG:boto:Token: None 2022-09-12T19:06:59.761 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:06:59.761 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6sHDJ5ygzXk2JJJOw9Wch8ih4qk= 2022-09-12T19:06:59.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6sHDJ5ygzXk2JJJOw9Wch8ih4qk=', 'Content-Length': '0'} 2022-09-12T19:06:59.766 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cdf4fdd866ba8dd4-00631f8353-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.766 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:59.767 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:59.768 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.768 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.769 DEBUG:boto:Method: GET 2022-09-12T19:06:59.769 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:59.770 DEBUG:boto:Data: 2022-09-12T19:06:59.770 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.771 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.771 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.772 DEBUG:boto:Params: {} 2022-09-12T19:06:59.774 DEBUG:boto:Token: None 2022-09-12T19:06:59.774 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/ 2022-09-12T19:06:59.775 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us= 2022-09-12T19:06:59.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us=', 'Content-Length': '0'} 2022-09-12T19:06:59.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be7483b5ad170528-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.779 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:59.780 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:59.780 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:59.781 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.781 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:06:59.782 DEBUG:boto:Method: GET 2022-09-12T19:06:59.783 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:06:59.783 DEBUG:boto:Data: 2022-09-12T19:06:59.784 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.784 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:06:59.785 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.785 DEBUG:boto:Params: {} 2022-09-12T19:06:59.786 DEBUG:boto:Token: None 2022-09-12T19:06:59.787 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/ 2022-09-12T19:06:59.787 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us= 2022-09-12T19:06:59.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us=', 'Content-Length': '0'} 2022-09-12T19:06:59.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000719fe184289f78f2-00631f8353-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.791 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:59.793 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-19 zones={test-zone3, test-zone1} 2022-09-12T19:06:59.793 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-19 zones={test-zone3, test-zone2} 2022-09-12T19:06:59.794 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:59.794 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:59.795 DEBUG:boto:Method: HEAD 2022-09-12T19:06:59.796 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:06:59.796 DEBUG:boto:Data: 2022-09-12T19:06:59.797 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.797 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.798 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.798 DEBUG:boto:Params: {} 2022-09-12T19:06:59.799 DEBUG:boto:Token: None 2022-09-12T19:06:59.800 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/ 2022-09-12T19:06:59.800 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aaRDfOSPOeFT6SKAlQTdVg73xGs= 2022-09-12T19:06:59.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aaRDfOSPOeFT6SKAlQTdVg73xGs=', 'Content-Length': '0'} 2022-09-12T19:06:59.804 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ccefe3b5fefa78d-00631f8353-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.804 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:59.805 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:59.806 DEBUG:boto:Method: HEAD 2022-09-12T19:06:59.807 DEBUG:boto:Path: /jdpiak-19/ 2022-09-12T19:06:59.808 DEBUG:boto:Data: 2022-09-12T19:06:59.808 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.809 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:59.810 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.811 DEBUG:boto:Params: {} 2022-09-12T19:06:59.812 DEBUG:boto:Token: None 2022-09-12T19:06:59.813 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/ 2022-09-12T19:06:59.813 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aaRDfOSPOeFT6SKAlQTdVg73xGs= 2022-09-12T19:06:59.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aaRDfOSPOeFT6SKAlQTdVg73xGs=', 'Content-Length': '0'} 2022-09-12T19:06:59.817 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005181a035e4754bcf-00631f8353-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.817 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:06:59.818 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:59.818 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:59.819 DEBUG:boto:path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:59.819 DEBUG:boto:auth_path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:59.820 DEBUG:boto:Method: GET 2022-09-12T19:06:59.820 DEBUG:boto:Path: /jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:59.821 DEBUG:boto:Data: 2022-09-12T19:06:59.821 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.822 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.823 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.823 DEBUG:boto:Params: {} 2022-09-12T19:06:59.824 DEBUG:boto:Token: None 2022-09-12T19:06:59.824 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/?versions 2022-09-12T19:06:59.825 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:26HiQOBxfSDx5sbTY0Weo6ySb68= 2022-09-12T19:06:59.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:26HiQOBxfSDx5sbTY0Weo6ySb68=', 'Content-Length': '0'} 2022-09-12T19:06:59.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000411399a6574ba554-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.830 DEBUG:boto:b'jdpiak-19999falsedummydAjTD8IRanrz7e2F.KgapZJi58DocMetrue2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZqtrue2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ulfalse2022-09-12T19:05:07.485Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTM67hyaYWRVq578zUsjsXjOzkUYTIvAfalse2022-09-12T19:05:07.581Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rhfalse2022-09-12T19:05:07.457Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpwncGyI2j8AeHH4hjzwsep2DWsp-l-Afalse2022-09-12T19:05:07.383Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsGTjDAFqZF1WL9AP98wlNVRWMwOQwYQfalse2022-09-12T19:05:07.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT.yM--FII-PB1GReM85mkNRGYz6Dkbzfalse2022-09-12T19:05:07.238Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXb5UZAvutXF6GhBCMdUQgjOLi998.hKfalse2022-09-12T19:05:07.343Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjztQRphsNbUdApzTcOcagUJ56uTwo7J2false2022-09-12T19:05:07.274Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKatr07t9mQV0b0gx44aB8d9VyeAhMS1false2022-09-12T19:05:07.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUBsYcCA94H55K4ucww5vXNAxuAMKNSGfalse2022-09-12T19:05:07.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkxSkXmfXKy0sdqagFrYSQNi7tMOKBoWfalse2022-09-12T19:05:07.199Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0false2022-09-12T19:05:06.995Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK.LUooPdPnXZMzfYIxw-xagQpErk4CXfalse2022-09-12T19:05:07.075Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtQHk21B862H4VHQxrZWEog8n-qgO99Qfalse2022-09-12T19:05:07.038Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQHXOu7SGV87vItdBeqaL.hYUlX--Ty8false2022-09-12T19:05:06.952Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT-rka2v4NcjXyfB8GzT0u7KHTz6OcF3false2022-09-12T19:05:06.866Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgdUGnE7SHabmaVwE2tf-YcFxl2tBu7Hfalse2022-09-12T19:05:06.908Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBAbppx4RylEKnBrYesXt1QPJgsAhIBbfalse2022-09-12T19:05:06.803Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNDM-EGOTc7UZbg9YQZwcM2uEk6P03H2false2022-09-12T19:05:06.773Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje.6ruI53roqy4mmyPgzQ1kqMdajruFAfalse2022-09-12T19:05:06.834Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLIaoO70PR8fRbTLQ-5Q.roWjA0lswqEfalse2022-09-12T19:05:06.733Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlnpiHoSs91QVujLSWqHhvEGzZrF3-sAfalse2022-09-12T19:05:06.698Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp5OIjHSzwvUDkIBYzRPdKBGezjIkrn-false2022-09-12T19:05:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjV.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9false2022-09-12T19:05:06.565Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiJi00oqpo4jP5T2qIUe4H2lLKJSYXZdfalse2022-09-12T19:05:06.598Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjj0eE53eOEuIue0LhYTVuZStUTmcOj7ofalse2022-09-12T19:05:06.634Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjACPdCUDfm32QLq6XbWYcqHiCLcEU2JYfalse2022-09-12T19:05:06.489Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJcC9N13lklBVOx234Q8MXQ8kAM8ByWgfalse2022-09-12T19:05:06.383Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjipYOl-Gqt65Y3nx.nVL2kL2DrysHNSYfalse2022-09-12T19:05:06.415Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:59.832 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:06:59.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.849 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:06:59.849 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:59.850 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:59.850 DEBUG:boto:path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:59.851 DEBUG:boto:auth_path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:59.851 DEBUG:boto:Method: GET 2022-09-12T19:06:59.852 DEBUG:boto:Path: /jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:59.852 DEBUG:boto:Data: 2022-09-12T19:06:59.853 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.853 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:59.854 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.854 DEBUG:boto:Params: {} 2022-09-12T19:06:59.855 DEBUG:boto:Token: None 2022-09-12T19:06:59.855 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/?versions 2022-09-12T19:06:59.856 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:26HiQOBxfSDx5sbTY0Weo6ySb68= 2022-09-12T19:06:59.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:26HiQOBxfSDx5sbTY0Weo6ySb68=', 'Content-Length': '0'} 2022-09-12T19:06:59.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db9ac6ad2e6ba866-00631f8353-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.861 DEBUG:boto:b'jdpiak-19999falsedummydAjTD8IRanrz7e2F.KgapZJi58DocMetrue2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZqtrue2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ulfalse2022-09-12T19:05:07.485Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTM67hyaYWRVq578zUsjsXjOzkUYTIvAfalse2022-09-12T19:05:07.581Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rhfalse2022-09-12T19:05:07.457Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpwncGyI2j8AeHH4hjzwsep2DWsp-l-Afalse2022-09-12T19:05:07.383Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsGTjDAFqZF1WL9AP98wlNVRWMwOQwYQfalse2022-09-12T19:05:07.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT.yM--FII-PB1GReM85mkNRGYz6Dkbzfalse2022-09-12T19:05:07.238Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXb5UZAvutXF6GhBCMdUQgjOLi998.hKfalse2022-09-12T19:05:07.343Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjztQRphsNbUdApzTcOcagUJ56uTwo7J2false2022-09-12T19:05:07.274Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKatr07t9mQV0b0gx44aB8d9VyeAhMS1false2022-09-12T19:05:07.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUBsYcCA94H55K4ucww5vXNAxuAMKNSGfalse2022-09-12T19:05:07.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkxSkXmfXKy0sdqagFrYSQNi7tMOKBoWfalse2022-09-12T19:05:07.199Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0false2022-09-12T19:05:06.995Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK.LUooPdPnXZMzfYIxw-xagQpErk4CXfalse2022-09-12T19:05:07.075Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtQHk21B862H4VHQxrZWEog8n-qgO99Qfalse2022-09-12T19:05:07.038Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQHXOu7SGV87vItdBeqaL.hYUlX--Ty8false2022-09-12T19:05:06.952Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT-rka2v4NcjXyfB8GzT0u7KHTz6OcF3false2022-09-12T19:05:06.866Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgdUGnE7SHabmaVwE2tf-YcFxl2tBu7Hfalse2022-09-12T19:05:06.908Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBAbppx4RylEKnBrYesXt1QPJgsAhIBbfalse2022-09-12T19:05:06.803Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNDM-EGOTc7UZbg9YQZwcM2uEk6P03H2false2022-09-12T19:05:06.773Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje.6ruI53roqy4mmyPgzQ1kqMdajruFAfalse2022-09-12T19:05:06.834Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLIaoO70PR8fRbTLQ-5Q.roWjA0lswqEfalse2022-09-12T19:05:06.733Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlnpiHoSs91QVujLSWqHhvEGzZrF3-sAfalse2022-09-12T19:05:06.698Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp5OIjHSzwvUDkIBYzRPdKBGezjIkrn-false2022-09-12T19:05:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjV.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9false2022-09-12T19:05:06.565Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiJi00oqpo4jP5T2qIUe4H2lLKJSYXZdfalse2022-09-12T19:05:06.598Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjj0eE53eOEuIue0LhYTVuZStUTmcOj7ofalse2022-09-12T19:05:06.634Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjACPdCUDfm32QLq6XbWYcqHiCLcEU2JYfalse2022-09-12T19:05:06.489Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJcC9N13lklBVOx234Q8MXQ8kAM8ByWgfalse2022-09-12T19:05:06.383Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjipYOl-Gqt65Y3nx.nVL2kL2DrysHNSYfalse2022-09-12T19:05:06.415Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:59.863 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-12T19:06:59.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:06:59.878 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:59.878 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:59.879 DEBUG:boto:path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:59.879 DEBUG:boto:auth_path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:59.880 DEBUG:boto:Method: GET 2022-09-12T19:06:59.880 DEBUG:boto:Path: /jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:59.881 DEBUG:boto:Data: 2022-09-12T19:06:59.881 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.882 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.882 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.883 DEBUG:boto:Params: {} 2022-09-12T19:06:59.884 DEBUG:boto:Token: None 2022-09-12T19:06:59.884 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/?versions 2022-09-12T19:06:59.885 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:26HiQOBxfSDx5sbTY0Weo6ySb68= 2022-09-12T19:06:59.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:26HiQOBxfSDx5sbTY0Weo6ySb68=', 'Content-Length': '0'} 2022-09-12T19:06:59.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065cebe0b390440d9-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.890 DEBUG:boto:b'jdpiak-19999falsedummydAjTD8IRanrz7e2F.KgapZJi58DocMetrue2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZqtrue2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ulfalse2022-09-12T19:05:07.485Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTM67hyaYWRVq578zUsjsXjOzkUYTIvAfalse2022-09-12T19:05:07.581Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rhfalse2022-09-12T19:05:07.457Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpwncGyI2j8AeHH4hjzwsep2DWsp-l-Afalse2022-09-12T19:05:07.383Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsGTjDAFqZF1WL9AP98wlNVRWMwOQwYQfalse2022-09-12T19:05:07.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT.yM--FII-PB1GReM85mkNRGYz6Dkbzfalse2022-09-12T19:05:07.238Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXb5UZAvutXF6GhBCMdUQgjOLi998.hKfalse2022-09-12T19:05:07.343Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjztQRphsNbUdApzTcOcagUJ56uTwo7J2false2022-09-12T19:05:07.274Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKatr07t9mQV0b0gx44aB8d9VyeAhMS1false2022-09-12T19:05:07.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUBsYcCA94H55K4ucww5vXNAxuAMKNSGfalse2022-09-12T19:05:07.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkxSkXmfXKy0sdqagFrYSQNi7tMOKBoWfalse2022-09-12T19:05:07.199Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0false2022-09-12T19:05:06.995Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK.LUooPdPnXZMzfYIxw-xagQpErk4CXfalse2022-09-12T19:05:07.075Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtQHk21B862H4VHQxrZWEog8n-qgO99Qfalse2022-09-12T19:05:07.038Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQHXOu7SGV87vItdBeqaL.hYUlX--Ty8false2022-09-12T19:05:06.952Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT-rka2v4NcjXyfB8GzT0u7KHTz6OcF3false2022-09-12T19:05:06.866Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgdUGnE7SHabmaVwE2tf-YcFxl2tBu7Hfalse2022-09-12T19:05:06.908Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBAbppx4RylEKnBrYesXt1QPJgsAhIBbfalse2022-09-12T19:05:06.803Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNDM-EGOTc7UZbg9YQZwcM2uEk6P03H2false2022-09-12T19:05:06.773Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje.6ruI53roqy4mmyPgzQ1kqMdajruFAfalse2022-09-12T19:05:06.834Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLIaoO70PR8fRbTLQ-5Q.roWjA0lswqEfalse2022-09-12T19:05:06.733Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlnpiHoSs91QVujLSWqHhvEGzZrF3-sAfalse2022-09-12T19:05:06.698Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp5OIjHSzwvUDkIBYzRPdKBGezjIkrn-false2022-09-12T19:05:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjV.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9false2022-09-12T19:05:06.565Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiJi00oqpo4jP5T2qIUe4H2lLKJSYXZdfalse2022-09-12T19:05:06.598Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjj0eE53eOEuIue0LhYTVuZStUTmcOj7ofalse2022-09-12T19:05:06.634Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjACPdCUDfm32QLq6XbWYcqHiCLcEU2JYfalse2022-09-12T19:05:06.489Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJcC9N13lklBVOx234Q8MXQ8kAM8ByWgfalse2022-09-12T19:05:06.383Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjipYOl-Gqt65Y3nx.nVL2kL2DrysHNSYfalse2022-09-12T19:05:06.415Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:59.892 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:59.893 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:59.893 DEBUG:boto:path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:59.894 DEBUG:boto:auth_path=/jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:59.894 DEBUG:boto:Method: GET 2022-09-12T19:06:59.895 DEBUG:boto:Path: /jdpiak-19/?versions&max-keys=999 2022-09-12T19:06:59.895 DEBUG:boto:Data: 2022-09-12T19:06:59.896 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.896 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:59.897 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.898 DEBUG:boto:Params: {} 2022-09-12T19:06:59.898 DEBUG:boto:Token: None 2022-09-12T19:06:59.899 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/?versions 2022-09-12T19:06:59.899 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:26HiQOBxfSDx5sbTY0Weo6ySb68= 2022-09-12T19:06:59.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:26HiQOBxfSDx5sbTY0Weo6ySb68=', 'Content-Length': '0'} 2022-09-12T19:06:59.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4c40f8d74a5a431-00631f8353-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.904 DEBUG:boto:b'jdpiak-19999falsedummydAjTD8IRanrz7e2F.KgapZJi58DocMetrue2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZqtrue2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ulfalse2022-09-12T19:05:07.485Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTM67hyaYWRVq578zUsjsXjOzkUYTIvAfalse2022-09-12T19:05:07.581Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rhfalse2022-09-12T19:05:07.457Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpwncGyI2j8AeHH4hjzwsep2DWsp-l-Afalse2022-09-12T19:05:07.383Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsGTjDAFqZF1WL9AP98wlNVRWMwOQwYQfalse2022-09-12T19:05:07.419Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT.yM--FII-PB1GReM85mkNRGYz6Dkbzfalse2022-09-12T19:05:07.238Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXb5UZAvutXF6GhBCMdUQgjOLi998.hKfalse2022-09-12T19:05:07.343Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjztQRphsNbUdApzTcOcagUJ56uTwo7J2false2022-09-12T19:05:07.274Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKatr07t9mQV0b0gx44aB8d9VyeAhMS1false2022-09-12T19:05:07.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUBsYcCA94H55K4ucww5vXNAxuAMKNSGfalse2022-09-12T19:05:07.165Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjkxSkXmfXKy0sdqagFrYSQNi7tMOKBoWfalse2022-09-12T19:05:07.199Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0false2022-09-12T19:05:06.995Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK.LUooPdPnXZMzfYIxw-xagQpErk4CXfalse2022-09-12T19:05:07.075Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtQHk21B862H4VHQxrZWEog8n-qgO99Qfalse2022-09-12T19:05:07.038Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQHXOu7SGV87vItdBeqaL.hYUlX--Ty8false2022-09-12T19:05:06.952Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjT-rka2v4NcjXyfB8GzT0u7KHTz6OcF3false2022-09-12T19:05:06.866Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgdUGnE7SHabmaVwE2tf-YcFxl2tBu7Hfalse2022-09-12T19:05:06.908Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBAbppx4RylEKnBrYesXt1QPJgsAhIBbfalse2022-09-12T19:05:06.803Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNDM-EGOTc7UZbg9YQZwcM2uEk6P03H2false2022-09-12T19:05:06.773Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje.6ruI53roqy4mmyPgzQ1kqMdajruFAfalse2022-09-12T19:05:06.834Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLIaoO70PR8fRbTLQ-5Q.roWjA0lswqEfalse2022-09-12T19:05:06.733Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlnpiHoSs91QVujLSWqHhvEGzZrF3-sAfalse2022-09-12T19:05:06.698Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp5OIjHSzwvUDkIBYzRPdKBGezjIkrn-false2022-09-12T19:05:06.666Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjV.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9false2022-09-12T19:05:06.565Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiJi00oqpo4jP5T2qIUe4H2lLKJSYXZdfalse2022-09-12T19:05:06.598Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjj0eE53eOEuIue0LhYTVuZStUTmcOj7ofalse2022-09-12T19:05:06.634Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjACPdCUDfm32QLq6XbWYcqHiCLcEU2JYfalse2022-09-12T19:05:06.489Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJcC9N13lklBVOx234Q8MXQ8kAM8ByWgfalse2022-09-12T19:05:06.383Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjipYOl-Gqt65Y3nx.nVL2kL2DrysHNSYfalse2022-09-12T19:05:06.415Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:59.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:06:59.907 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:59.907 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:59.908 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:59.908 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:59.909 DEBUG:boto:Method: GET 2022-09-12T19:06:59.909 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:59.910 DEBUG:boto:Data: 2022-09-12T19:06:59.911 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.911 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.912 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.912 DEBUG:boto:Params: {} 2022-09-12T19:06:59.913 DEBUG:boto:Token: None 2022-09-12T19:06:59.913 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:59.914 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qt0rGrfJsvVITenJ013RJPUFp7g= 2022-09-12T19:06:59.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qt0rGrfJsvVITenJ013RJPUFp7g=', 'Content-Length': '0'} 2022-09-12T19:06:59.917 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a5f9f11de0a48276-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.917 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:59.918 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:59.918 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:59.919 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:59.919 DEBUG:boto:Method: GET 2022-09-12T19:06:59.920 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:59.921 DEBUG:boto:Data: 2022-09-12T19:06:59.921 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.922 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:59.922 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.923 DEBUG:boto:Params: {} 2022-09-12T19:06:59.923 DEBUG:boto:Token: None 2022-09-12T19:06:59.924 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:59.925 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qt0rGrfJsvVITenJ013RJPUFp7g= 2022-09-12T19:06:59.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qt0rGrfJsvVITenJ013RJPUFp7g=', 'Content-Length': '0'} 2022-09-12T19:06:59.927 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b65a4e446c2a7589-00631f8353-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.928 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:59.928 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:59.929 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:59.929 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:59.930 DEBUG:boto:Method: HEAD 2022-09-12T19:06:59.930 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:59.931 DEBUG:boto:Data: 2022-09-12T19:06:59.931 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.932 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.932 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.933 DEBUG:boto:Params: {} 2022-09-12T19:06:59.934 DEBUG:boto:Token: None 2022-09-12T19:06:59.934 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:59.935 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QzXyewmCWzzs+zaD8Y/JvVtnsLI= 2022-09-12T19:06:59.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QzXyewmCWzzs+zaD8Y/JvVtnsLI=', 'Content-Length': '0'} 2022-09-12T19:06:59.937 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fb78b812f71ac292-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.938 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:59.938 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:59.939 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:59.939 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:59.940 DEBUG:boto:Method: HEAD 2022-09-12T19:06:59.940 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:59.941 DEBUG:boto:Data: 2022-09-12T19:06:59.941 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.942 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:59.943 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.943 DEBUG:boto:Params: {} 2022-09-12T19:06:59.944 DEBUG:boto:Token: None 2022-09-12T19:06:59.944 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:59.945 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QzXyewmCWzzs+zaD8Y/JvVtnsLI= 2022-09-12T19:06:59.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QzXyewmCWzzs+zaD8Y/JvVtnsLI=', 'Content-Length': '0'} 2022-09-12T19:06:59.947 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bb8d1b51ebc7a945-00631f8353-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:06:59.949 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:59.949 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:59.950 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:59.950 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:59.951 DEBUG:boto:Method: GET 2022-09-12T19:06:59.951 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:59.952 DEBUG:boto:Data: 2022-09-12T19:06:59.952 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.953 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.954 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.954 DEBUG:boto:Params: {} 2022-09-12T19:06:59.955 DEBUG:boto:Token: None 2022-09-12T19:06:59.955 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:59.956 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qt0rGrfJsvVITenJ013RJPUFp7g= 2022-09-12T19:06:59.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qt0rGrfJsvVITenJ013RJPUFp7g=', 'Content-Length': '0'} 2022-09-12T19:06:59.959 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001875e150645fad82-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.959 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:06:59.960 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:06:59.960 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:59.961 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:59.961 DEBUG:boto:Method: GET 2022-09-12T19:06:59.962 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:59.963 DEBUG:boto:Data: 2022-09-12T19:06:59.963 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.964 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:59.964 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.965 DEBUG:boto:Params: {} 2022-09-12T19:06:59.965 DEBUG:boto:Token: None 2022-09-12T19:06:59.966 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:06:59.967 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qt0rGrfJsvVITenJ013RJPUFp7g= 2022-09-12T19:06:59.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qt0rGrfJsvVITenJ013RJPUFp7g=', 'Content-Length': '0'} 2022-09-12T19:06:59.970 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007555999ab85a7867-00631f8353-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.971 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:59.972 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:59.972 DEBUG:boto:path=/jdpiak-19/?prefix=dummy 2022-09-12T19:06:59.973 DEBUG:boto:auth_path=/jdpiak-19/?prefix=dummy 2022-09-12T19:06:59.973 DEBUG:boto:Method: GET 2022-09-12T19:06:59.974 DEBUG:boto:Path: /jdpiak-19/?prefix=dummy 2022-09-12T19:06:59.974 DEBUG:boto:Data: 2022-09-12T19:06:59.975 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.976 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:06:59.976 DEBUG:boto:Port: 8001 2022-09-12T19:06:59.977 DEBUG:boto:Params: {} 2022-09-12T19:06:59.977 DEBUG:boto:Token: None 2022-09-12T19:06:59.978 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/ 2022-09-12T19:06:59.979 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us= 2022-09-12T19:06:59.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us=', 'Content-Length': '0'} 2022-09-12T19:06:59.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d99d84ba8287e84-00631f8353-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:06:59.983 DEBUG:boto:b'jdpiak-19dummy1000falsedummy2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:06:59.984 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:06:59.984 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:06:59.985 DEBUG:boto:path=/jdpiak-19/?prefix=dummy 2022-09-12T19:06:59.986 DEBUG:boto:auth_path=/jdpiak-19/?prefix=dummy 2022-09-12T19:06:59.986 DEBUG:boto:Method: GET 2022-09-12T19:06:59.987 DEBUG:boto:Path: /jdpiak-19/?prefix=dummy 2022-09-12T19:06:59.987 DEBUG:boto:Data: 2022-09-12T19:06:59.988 DEBUG:boto:Headers: {} 2022-09-12T19:06:59.989 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:06:59.989 DEBUG:boto:Port: 8000 2022-09-12T19:06:59.990 DEBUG:boto:Params: {} 2022-09-12T19:06:59.990 DEBUG:boto:Token: None 2022-09-12T19:06:59.991 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:06:59 GMT /jdpiak-19/ 2022-09-12T19:06:59.992 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us= 2022-09-12T19:06:59.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:06:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wJitXLUB4GE/gkvuyS/QXhMm4Us=', 'Content-Length': '0'} 2022-09-12T19:06:59.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049d7459500d6ae68-00631f8353-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:06:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.042 DEBUG:boto:b'jdpiak-19dummy1000falsedummy2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:00.043 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:07:00.043 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:07:00.044 DEBUG:boto:Method: HEAD 2022-09-12T19:07:00.045 DEBUG:boto:Path: /jdpiak-19/dummy 2022-09-12T19:07:00.045 DEBUG:boto:Data: 2022-09-12T19:07:00.046 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.046 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.047 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.047 DEBUG:boto:Params: {} 2022-09-12T19:07:00.048 DEBUG:boto:Token: None 2022-09-12T19:07:00.048 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/dummy 2022-09-12T19:07:00.049 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kI03MhrMjZetegNzhEqBnLWwhmw= 2022-09-12T19:07:00.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kI03MhrMjZetegNzhEqBnLWwhmw=', 'Content-Length': '0'} 2022-09-12T19:07:00.052 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000782cdcf4320abfb0-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.053 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:07:00.053 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:07:00.054 DEBUG:boto:Method: HEAD 2022-09-12T19:07:00.054 DEBUG:boto:Path: /jdpiak-19/dummy 2022-09-12T19:07:00.055 DEBUG:boto:Data: 2022-09-12T19:07:00.055 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.056 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.056 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.057 DEBUG:boto:Params: {} 2022-09-12T19:07:00.057 DEBUG:boto:Token: None 2022-09-12T19:07:00.058 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/dummy 2022-09-12T19:07:00.059 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kI03MhrMjZetegNzhEqBnLWwhmw= 2022-09-12T19:07:00.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kI03MhrMjZetegNzhEqBnLWwhmw=', 'Content-Length': '0'} 2022-09-12T19:07:00.062 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000009aa1b4757218315-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-12T19:07:00.063 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:07:00.064 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:07:00.064 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:07:00.065 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:07:00.065 DEBUG:boto:Method: GET 2022-09-12T19:07:00.066 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:07:00.066 DEBUG:boto:Data: 2022-09-12T19:07:00.067 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.067 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.068 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.068 DEBUG:boto:Params: {} 2022-09-12T19:07:00.069 DEBUG:boto:Token: None 2022-09-12T19:07:00.070 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:07:00.070 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ou7HL2AJ/oSfi+8ILabMOOSmq1s= 2022-09-12T19:07:00.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ou7HL2AJ/oSfi+8ILabMOOSmq1s=', 'Content-Length': '0'} 2022-09-12T19:07:00.073 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000059004644de0b842-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.074 DEBUG:boto:path=/jdpiak-19/dummy 2022-09-12T19:07:00.074 DEBUG:boto:auth_path=/jdpiak-19/dummy 2022-09-12T19:07:00.075 DEBUG:boto:path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:07:00.075 DEBUG:boto:auth_path=/jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:07:00.076 DEBUG:boto:Method: GET 2022-09-12T19:07:00.076 DEBUG:boto:Path: /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:07:00.077 DEBUG:boto:Data: 2022-09-12T19:07:00.078 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.078 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.079 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.079 DEBUG:boto:Params: {} 2022-09-12T19:07:00.080 DEBUG:boto:Token: None 2022-09-12T19:07:00.080 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/dummy?versionId=dAjTD8IRanrz7e2F.KgapZJi58DocMe 2022-09-12T19:07:00.081 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ou7HL2AJ/oSfi+8ILabMOOSmq1s= 2022-09-12T19:07:00.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ou7HL2AJ/oSfi+8ILabMOOSmq1s=', 'Content-Length': '0'} 2022-09-12T19:07:00.084 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:03:59 GMT'), ('x-amz-version-id', 'dAjTD8IRanrz7e2F.KgapZJi58DocMe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f2e0583f02058a79-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.085 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:00.085 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:00.086 DEBUG:boto:path=/jdpiak-19/?prefix=dummy 2022-09-12T19:07:00.086 DEBUG:boto:auth_path=/jdpiak-19/?prefix=dummy 2022-09-12T19:07:00.087 DEBUG:boto:Method: GET 2022-09-12T19:07:00.087 DEBUG:boto:Path: /jdpiak-19/?prefix=dummy 2022-09-12T19:07:00.088 DEBUG:boto:Data: 2022-09-12T19:07:00.088 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.089 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.089 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.090 DEBUG:boto:Params: {} 2022-09-12T19:07:00.090 DEBUG:boto:Token: None 2022-09-12T19:07:00.091 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/ 2022-09-12T19:07:00.092 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM= 2022-09-12T19:07:00.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM=', 'Content-Length': '0'} 2022-09-12T19:07:00.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003c021fa66ed05c2-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.096 DEBUG:boto:b'jdpiak-19dummy1000falsedummy2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:00.097 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:00.097 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:00.098 DEBUG:boto:path=/jdpiak-19/?prefix=dummy 2022-09-12T19:07:00.098 DEBUG:boto:auth_path=/jdpiak-19/?prefix=dummy 2022-09-12T19:07:00.099 DEBUG:boto:Method: GET 2022-09-12T19:07:00.099 DEBUG:boto:Path: /jdpiak-19/?prefix=dummy 2022-09-12T19:07:00.100 DEBUG:boto:Data: 2022-09-12T19:07:00.100 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.101 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.101 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.102 DEBUG:boto:Params: {} 2022-09-12T19:07:00.102 DEBUG:boto:Token: None 2022-09-12T19:07:00.103 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/ 2022-09-12T19:07:00.104 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM= 2022-09-12T19:07:00.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM=', 'Content-Length': '0'} 2022-09-12T19:07:00.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000551140596aebc439-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.108 DEBUG:boto:b'jdpiak-19dummy1000falsedummy2022-09-12T19:03:59.342Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:00.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:00.109 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.110 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.110 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.111 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.111 DEBUG:boto:Method: GET 2022-09-12T19:07:00.112 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.112 DEBUG:boto:Data: 2022-09-12T19:07:00.113 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.114 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.114 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.115 DEBUG:boto:Params: {} 2022-09-12T19:07:00.115 DEBUG:boto:Token: None 2022-09-12T19:07:00.116 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.116 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xnrBMH3xFSKE/P9hjIntOdHfhak= 2022-09-12T19:07:00.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xnrBMH3xFSKE/P9hjIntOdHfhak=', 'Content-Length': '0'} 2022-09-12T19:07:00.119 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002bb489358466ba17-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.120 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.121 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.121 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.122 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.122 DEBUG:boto:Method: GET 2022-09-12T19:07:00.123 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.131 DEBUG:boto:Data: 2022-09-12T19:07:00.132 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.133 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.133 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.134 DEBUG:boto:Params: {} 2022-09-12T19:07:00.135 DEBUG:boto:Token: None 2022-09-12T19:07:00.135 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.136 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xnrBMH3xFSKE/P9hjIntOdHfhak= 2022-09-12T19:07:00.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xnrBMH3xFSKE/P9hjIntOdHfhak=', 'Content-Length': '0'} 2022-09-12T19:07:00.139 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5b3f5bc9cc7458e-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.140 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.140 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.141 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.141 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.142 DEBUG:boto:Method: HEAD 2022-09-12T19:07:00.142 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.143 DEBUG:boto:Data: 2022-09-12T19:07:00.143 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.144 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.144 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.145 DEBUG:boto:Params: {} 2022-09-12T19:07:00.145 DEBUG:boto:Token: None 2022-09-12T19:07:00.146 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.147 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mWSb7vQ16y4I0Mxvoxz8pfcTX4s= 2022-09-12T19:07:00.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mWSb7vQ16y4I0Mxvoxz8pfcTX4s=', 'Content-Length': '0'} 2022-09-12T19:07:00.150 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004cbb760f1b9b22a3-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.151 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.151 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.152 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.152 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.153 DEBUG:boto:Method: HEAD 2022-09-12T19:07:00.153 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.154 DEBUG:boto:Data: 2022-09-12T19:07:00.155 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.155 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.155 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.156 DEBUG:boto:Params: {} 2022-09-12T19:07:00.157 DEBUG:boto:Token: None 2022-09-12T19:07:00.157 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.158 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mWSb7vQ16y4I0Mxvoxz8pfcTX4s= 2022-09-12T19:07:00.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mWSb7vQ16y4I0Mxvoxz8pfcTX4s=', 'Content-Length': '0'} 2022-09-12T19:07:00.161 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a61bd6c1c2f3d001-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:00.162 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.163 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.163 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.164 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.164 DEBUG:boto:Method: GET 2022-09-12T19:07:00.165 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.165 DEBUG:boto:Data: 2022-09-12T19:07:00.166 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.166 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.167 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.167 DEBUG:boto:Params: {} 2022-09-12T19:07:00.168 DEBUG:boto:Token: None 2022-09-12T19:07:00.169 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.169 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xnrBMH3xFSKE/P9hjIntOdHfhak= 2022-09-12T19:07:00.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xnrBMH3xFSKE/P9hjIntOdHfhak=', 'Content-Length': '0'} 2022-09-12T19:07:00.172 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be0ec188fd4fdcbd-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.173 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.174 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.175 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.176 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.176 DEBUG:boto:Method: GET 2022-09-12T19:07:00.177 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.178 DEBUG:boto:Data: 2022-09-12T19:07:00.179 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.179 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.180 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.180 DEBUG:boto:Params: {} 2022-09-12T19:07:00.181 DEBUG:boto:Token: None 2022-09-12T19:07:00.182 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.183 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xnrBMH3xFSKE/P9hjIntOdHfhak= 2022-09-12T19:07:00.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xnrBMH3xFSKE/P9hjIntOdHfhak=', 'Content-Length': '0'} 2022-09-12T19:07:00.186 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e82ee7d91e5346f-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.230 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:00.231 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:00.231 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:00.233 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:00.233 DEBUG:boto:Method: GET 2022-09-12T19:07:00.234 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:00.234 DEBUG:boto:Data: 2022-09-12T19:07:00.235 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.235 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.236 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.237 DEBUG:boto:Params: {} 2022-09-12T19:07:00.237 DEBUG:boto:Token: None 2022-09-12T19:07:00.238 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/ 2022-09-12T19:07:00.238 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM= 2022-09-12T19:07:00.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM=', 'Content-Length': '0'} 2022-09-12T19:07:00.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aed4196492508511-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.243 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:00.244 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:00.244 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:00.245 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:00.245 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:00.246 DEBUG:boto:Method: GET 2022-09-12T19:07:00.247 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:00.248 DEBUG:boto:Data: 2022-09-12T19:07:00.248 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.249 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.249 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.250 DEBUG:boto:Params: {} 2022-09-12T19:07:00.251 DEBUG:boto:Token: None 2022-09-12T19:07:00.251 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/ 2022-09-12T19:07:00.252 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM= 2022-09-12T19:07:00.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM=', 'Content-Length': '0'} 2022-09-12T19:07:00.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000056c7ed09641a0fb-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.256 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:00.256 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.257 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.257 DEBUG:boto:Method: HEAD 2022-09-12T19:07:00.258 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:00.259 DEBUG:boto:Data: 2022-09-12T19:07:00.259 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.260 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.260 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.261 DEBUG:boto:Params: {} 2022-09-12T19:07:00.262 DEBUG:boto:Token: None 2022-09-12T19:07:00.264 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj 2022-09-12T19:07:00.265 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/SRbOnRq3u+w1jQHvGboCsCe5a4= 2022-09-12T19:07:00.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/SRbOnRq3u+w1jQHvGboCsCe5a4=', 'Content-Length': '0'} 2022-09-12T19:07:00.270 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b41a578b8a6c76c0-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.271 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.272 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.272 DEBUG:boto:Method: HEAD 2022-09-12T19:07:00.273 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:00.274 DEBUG:boto:Data: 2022-09-12T19:07:00.275 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.276 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.276 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.277 DEBUG:boto:Params: {} 2022-09-12T19:07:00.278 DEBUG:boto:Token: None 2022-09-12T19:07:00.279 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj 2022-09-12T19:07:00.280 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/SRbOnRq3u+w1jQHvGboCsCe5a4= 2022-09-12T19:07:00.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/SRbOnRq3u+w1jQHvGboCsCe5a4=', 'Content-Length': '0'} 2022-09-12T19:07:00.284 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008535f20a1c756912-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:00.285 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.285 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.286 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.287 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.287 DEBUG:boto:Method: GET 2022-09-12T19:07:00.288 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.288 DEBUG:boto:Data: 2022-09-12T19:07:00.289 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.289 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.290 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.291 DEBUG:boto:Params: {} 2022-09-12T19:07:00.291 DEBUG:boto:Token: None 2022-09-12T19:07:00.292 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.293 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xnrBMH3xFSKE/P9hjIntOdHfhak= 2022-09-12T19:07:00.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xnrBMH3xFSKE/P9hjIntOdHfhak=', 'Content-Length': '0'} 2022-09-12T19:07:00.297 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000cf26650486cd20d-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.342 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.343 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.345 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.346 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.347 DEBUG:boto:Method: GET 2022-09-12T19:07:00.347 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.348 DEBUG:boto:Data: 2022-09-12T19:07:00.348 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.349 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.349 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.350 DEBUG:boto:Params: {} 2022-09-12T19:07:00.351 DEBUG:boto:Token: None 2022-09-12T19:07:00.351 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.352 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xnrBMH3xFSKE/P9hjIntOdHfhak= 2022-09-12T19:07:00.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xnrBMH3xFSKE/P9hjIntOdHfhak=', 'Content-Length': '0'} 2022-09-12T19:07:00.355 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f62214f69698015b-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.356 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:00.357 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:00.357 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:00.358 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:00.359 DEBUG:boto:Method: GET 2022-09-12T19:07:00.359 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:00.360 DEBUG:boto:Data: 2022-09-12T19:07:00.360 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.361 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.361 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.362 DEBUG:boto:Params: {} 2022-09-12T19:07:00.362 DEBUG:boto:Token: None 2022-09-12T19:07:00.363 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/ 2022-09-12T19:07:00.364 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM= 2022-09-12T19:07:00.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM=', 'Content-Length': '0'} 2022-09-12T19:07:00.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082c9e6321f2a7226-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.368 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:00.369 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:00.369 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:00.370 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:00.370 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:00.371 DEBUG:boto:Method: GET 2022-09-12T19:07:00.371 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:00.372 DEBUG:boto:Data: 2022-09-12T19:07:00.373 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.373 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.374 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.374 DEBUG:boto:Params: {} 2022-09-12T19:07:00.375 DEBUG:boto:Token: None 2022-09-12T19:07:00.376 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/ 2022-09-12T19:07:00.376 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM= 2022-09-12T19:07:00.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM=', 'Content-Length': '0'} 2022-09-12T19:07:00.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092a0677577516a68-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.426 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:00.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:00.429 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.429 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.430 DEBUG:boto:path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:07:00.431 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:07:00.432 DEBUG:boto:Method: GET 2022-09-12T19:07:00.433 DEBUG:boto:Path: /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:07:00.433 DEBUG:boto:Data: 2022-09-12T19:07:00.434 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.435 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.435 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.436 DEBUG:boto:Params: {} 2022-09-12T19:07:00.436 DEBUG:boto:Token: None 2022-09-12T19:07:00.437 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:07:00.438 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rj8t7NkM9wMYbn2HpJfXtdWpZXY= 2022-09-12T19:07:00.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rj8t7NkM9wMYbn2HpJfXtdWpZXY=', 'Content-Length': '0'} 2022-09-12T19:07:00.440 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030889af76eca271f-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.441 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.442 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.442 DEBUG:boto:path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:07:00.443 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:07:00.444 DEBUG:boto:Method: GET 2022-09-12T19:07:00.444 DEBUG:boto:Path: /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:07:00.445 DEBUG:boto:Data: 2022-09-12T19:07:00.445 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.446 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.446 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.447 DEBUG:boto:Params: {} 2022-09-12T19:07:00.447 DEBUG:boto:Token: None 2022-09-12T19:07:00.448 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:07:00.449 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rj8t7NkM9wMYbn2HpJfXtdWpZXY= 2022-09-12T19:07:00.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rj8t7NkM9wMYbn2HpJfXtdWpZXY=', 'Content-Length': '0'} 2022-09-12T19:07:00.451 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087ae86bd876841fe-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.452 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.453 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.453 DEBUG:boto:path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:07:00.454 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:07:00.454 DEBUG:boto:Method: HEAD 2022-09-12T19:07:00.455 DEBUG:boto:Path: /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:07:00.456 DEBUG:boto:Data: 2022-09-12T19:07:00.456 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.457 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.457 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.458 DEBUG:boto:Params: {} 2022-09-12T19:07:00.458 DEBUG:boto:Token: None 2022-09-12T19:07:00.459 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:07:00.459 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3AzIgDZ0Ecma9034jcpen647ems= 2022-09-12T19:07:00.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3AzIgDZ0Ecma9034jcpen647ems=', 'Content-Length': '0'} 2022-09-12T19:07:00.462 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041c2b6582599077c-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.463 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.463 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.464 DEBUG:boto:path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:07:00.464 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:07:00.465 DEBUG:boto:Method: HEAD 2022-09-12T19:07:00.465 DEBUG:boto:Path: /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:07:00.466 DEBUG:boto:Data: 2022-09-12T19:07:00.466 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.467 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.467 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.468 DEBUG:boto:Params: {} 2022-09-12T19:07:00.468 DEBUG:boto:Token: None 2022-09-12T19:07:00.469 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:07:00.470 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3AzIgDZ0Ecma9034jcpen647ems= 2022-09-12T19:07:00.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3AzIgDZ0Ecma9034jcpen647ems=', 'Content-Length': '0'} 2022-09-12T19:07:00.472 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000970a7c8e06c71184-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:00.473 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.474 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.475 DEBUG:boto:path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:07:00.475 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:07:00.476 DEBUG:boto:Method: GET 2022-09-12T19:07:00.476 DEBUG:boto:Path: /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:07:00.477 DEBUG:boto:Data: 2022-09-12T19:07:00.477 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.478 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.479 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.479 DEBUG:boto:Params: {} 2022-09-12T19:07:00.480 DEBUG:boto:Token: None 2022-09-12T19:07:00.480 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:07:00.481 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rj8t7NkM9wMYbn2HpJfXtdWpZXY= 2022-09-12T19:07:00.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rj8t7NkM9wMYbn2HpJfXtdWpZXY=', 'Content-Length': '0'} 2022-09-12T19:07:00.484 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b62ee0baa68f2d78-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.485 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.485 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.486 DEBUG:boto:path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:07:00.486 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:07:00.487 DEBUG:boto:Method: GET 2022-09-12T19:07:00.488 DEBUG:boto:Path: /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:07:00.488 DEBUG:boto:Data: 2022-09-12T19:07:00.489 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.489 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.490 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.491 DEBUG:boto:Params: {} 2022-09-12T19:07:00.491 DEBUG:boto:Token: None 2022-09-12T19:07:00.492 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul 2022-09-12T19:07:00.492 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rj8t7NkM9wMYbn2HpJfXtdWpZXY= 2022-09-12T19:07:00.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rj8t7NkM9wMYbn2HpJfXtdWpZXY=', 'Content-Length': '0'} 2022-09-12T19:07:00.495 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'RzUl8X5tNQPueFqOSnUVQ7Pv1K6T9ul'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006780090c6fa638d-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.496 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:00.497 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:00.497 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:00.498 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:00.499 DEBUG:boto:Method: GET 2022-09-12T19:07:00.499 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:00.500 DEBUG:boto:Data: 2022-09-12T19:07:00.500 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.501 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.502 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.502 DEBUG:boto:Params: {} 2022-09-12T19:07:00.503 DEBUG:boto:Token: None 2022-09-12T19:07:00.504 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/ 2022-09-12T19:07:00.504 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM= 2022-09-12T19:07:00.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM=', 'Content-Length': '0'} 2022-09-12T19:07:00.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066d726f753235a20-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.510 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:00.511 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:00.512 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:00.513 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:00.513 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:00.514 DEBUG:boto:Method: GET 2022-09-12T19:07:00.514 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:00.515 DEBUG:boto:Data: 2022-09-12T19:07:00.515 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.516 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.517 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.517 DEBUG:boto:Params: {} 2022-09-12T19:07:00.518 DEBUG:boto:Token: None 2022-09-12T19:07:00.519 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/ 2022-09-12T19:07:00.520 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM= 2022-09-12T19:07:00.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM=', 'Content-Length': '0'} 2022-09-12T19:07:00.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea5e701b92ae3cfc-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.524 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:00.525 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.526 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.526 DEBUG:boto:Method: HEAD 2022-09-12T19:07:00.527 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:00.527 DEBUG:boto:Data: 2022-09-12T19:07:00.528 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.528 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.529 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.529 DEBUG:boto:Params: {} 2022-09-12T19:07:00.530 DEBUG:boto:Token: None 2022-09-12T19:07:00.531 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj 2022-09-12T19:07:00.531 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/SRbOnRq3u+w1jQHvGboCsCe5a4= 2022-09-12T19:07:00.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/SRbOnRq3u+w1jQHvGboCsCe5a4=', 'Content-Length': '0'} 2022-09-12T19:07:00.536 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071030fdc3c022767-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.536 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.537 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.538 DEBUG:boto:Method: HEAD 2022-09-12T19:07:00.539 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:00.539 DEBUG:boto:Data: 2022-09-12T19:07:00.540 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.540 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.541 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.542 DEBUG:boto:Params: {} 2022-09-12T19:07:00.543 DEBUG:boto:Token: None 2022-09-12T19:07:00.544 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj 2022-09-12T19:07:00.545 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/SRbOnRq3u+w1jQHvGboCsCe5a4= 2022-09-12T19:07:00.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/SRbOnRq3u+w1jQHvGboCsCe5a4=', 'Content-Length': '0'} 2022-09-12T19:07:00.549 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056d38ba7599e7493-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:00.551 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.552 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.552 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.553 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.553 DEBUG:boto:Method: GET 2022-09-12T19:07:00.554 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.556 DEBUG:boto:Data: 2022-09-12T19:07:00.556 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.557 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.558 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.558 DEBUG:boto:Params: {} 2022-09-12T19:07:00.559 DEBUG:boto:Token: None 2022-09-12T19:07:00.560 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.561 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xnrBMH3xFSKE/P9hjIntOdHfhak= 2022-09-12T19:07:00.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xnrBMH3xFSKE/P9hjIntOdHfhak=', 'Content-Length': '0'} 2022-09-12T19:07:00.564 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007eecf291f0a09ea3-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.607 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.607 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.608 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.608 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.609 DEBUG:boto:Method: GET 2022-09-12T19:07:00.610 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.610 DEBUG:boto:Data: 2022-09-12T19:07:00.611 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.611 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.612 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.612 DEBUG:boto:Params: {} 2022-09-12T19:07:00.613 DEBUG:boto:Token: None 2022-09-12T19:07:00.613 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.614 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xnrBMH3xFSKE/P9hjIntOdHfhak= 2022-09-12T19:07:00.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xnrBMH3xFSKE/P9hjIntOdHfhak=', 'Content-Length': '0'} 2022-09-12T19:07:00.617 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002802979ce455d6b8-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.618 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:00.618 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:00.619 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:00.619 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:00.620 DEBUG:boto:Method: GET 2022-09-12T19:07:00.621 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:00.621 DEBUG:boto:Data: 2022-09-12T19:07:00.623 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.624 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.625 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.626 DEBUG:boto:Params: {} 2022-09-12T19:07:00.627 DEBUG:boto:Token: None 2022-09-12T19:07:00.628 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/ 2022-09-12T19:07:00.629 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM= 2022-09-12T19:07:00.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM=', 'Content-Length': '0'} 2022-09-12T19:07:00.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007be741bca77e7ef8-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.634 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:00.636 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:00.637 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:00.638 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:00.639 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:00.640 DEBUG:boto:Method: GET 2022-09-12T19:07:00.641 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:00.642 DEBUG:boto:Data: 2022-09-12T19:07:00.643 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.643 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.644 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.645 DEBUG:boto:Params: {} 2022-09-12T19:07:00.646 DEBUG:boto:Token: None 2022-09-12T19:07:00.647 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/ 2022-09-12T19:07:00.648 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM= 2022-09-12T19:07:00.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM=', 'Content-Length': '0'} 2022-09-12T19:07:00.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff0a28b76bc98348-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.653 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:00.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:00.655 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.655 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.656 DEBUG:boto:path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:07:00.656 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:07:00.657 DEBUG:boto:Method: GET 2022-09-12T19:07:00.657 DEBUG:boto:Path: /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:07:00.658 DEBUG:boto:Data: 2022-09-12T19:07:00.658 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.659 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.659 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.660 DEBUG:boto:Params: {} 2022-09-12T19:07:00.661 DEBUG:boto:Token: None 2022-09-12T19:07:00.661 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:07:00.662 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:91Ppqt0Wx1uIke0HxcOGQwTkNEg= 2022-09-12T19:07:00.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:91Ppqt0Wx1uIke0HxcOGQwTkNEg=', 'Content-Length': '0'} 2022-09-12T19:07:00.665 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'TM67hyaYWRVq578zUsjsXjOzkUYTIvA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000accd97ca281cfcd4-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.666 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.666 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.667 DEBUG:boto:path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:07:00.667 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:07:00.668 DEBUG:boto:Method: GET 2022-09-12T19:07:00.669 DEBUG:boto:Path: /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:07:00.669 DEBUG:boto:Data: 2022-09-12T19:07:00.670 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.670 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.671 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.671 DEBUG:boto:Params: {} 2022-09-12T19:07:00.672 DEBUG:boto:Token: None 2022-09-12T19:07:00.672 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:07:00.673 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:91Ppqt0Wx1uIke0HxcOGQwTkNEg= 2022-09-12T19:07:00.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:91Ppqt0Wx1uIke0HxcOGQwTkNEg=', 'Content-Length': '0'} 2022-09-12T19:07:00.676 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'TM67hyaYWRVq578zUsjsXjOzkUYTIvA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5d7d378019d56fe-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.677 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.677 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.678 DEBUG:boto:path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:07:00.679 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:07:00.679 DEBUG:boto:Method: HEAD 2022-09-12T19:07:00.680 DEBUG:boto:Path: /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:07:00.680 DEBUG:boto:Data: 2022-09-12T19:07:00.681 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.681 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.682 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.682 DEBUG:boto:Params: {} 2022-09-12T19:07:00.683 DEBUG:boto:Token: None 2022-09-12T19:07:00.683 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:07:00.684 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:acyPOiOwpAVn1mKlUGgrBnsH70k= 2022-09-12T19:07:00.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:acyPOiOwpAVn1mKlUGgrBnsH70k=', 'Content-Length': '0'} 2022-09-12T19:07:00.687 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'TM67hyaYWRVq578zUsjsXjOzkUYTIvA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002df205c7a0eb7497-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.687 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.688 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.688 DEBUG:boto:path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:07:00.689 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:07:00.689 DEBUG:boto:Method: HEAD 2022-09-12T19:07:00.690 DEBUG:boto:Path: /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:07:00.691 DEBUG:boto:Data: 2022-09-12T19:07:00.691 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.692 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.692 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.693 DEBUG:boto:Params: {} 2022-09-12T19:07:00.693 DEBUG:boto:Token: None 2022-09-12T19:07:00.694 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:07:00.694 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:acyPOiOwpAVn1mKlUGgrBnsH70k= 2022-09-12T19:07:00.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:acyPOiOwpAVn1mKlUGgrBnsH70k=', 'Content-Length': '0'} 2022-09-12T19:07:00.697 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'TM67hyaYWRVq578zUsjsXjOzkUYTIvA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4c9ed63f6db8120-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:00.698 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.699 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.699 DEBUG:boto:path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:07:00.700 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:07:00.701 DEBUG:boto:Method: GET 2022-09-12T19:07:00.703 DEBUG:boto:Path: /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:07:00.703 DEBUG:boto:Data: 2022-09-12T19:07:00.704 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.705 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.706 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.707 DEBUG:boto:Params: {} 2022-09-12T19:07:00.707 DEBUG:boto:Token: None 2022-09-12T19:07:00.708 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:07:00.709 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:91Ppqt0Wx1uIke0HxcOGQwTkNEg= 2022-09-12T19:07:00.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:91Ppqt0Wx1uIke0HxcOGQwTkNEg=', 'Content-Length': '0'} 2022-09-12T19:07:00.711 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'TM67hyaYWRVq578zUsjsXjOzkUYTIvA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016548b82b7d4b56c-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.712 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.713 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.714 DEBUG:boto:path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:07:00.714 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:07:00.715 DEBUG:boto:Method: GET 2022-09-12T19:07:00.715 DEBUG:boto:Path: /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:07:00.716 DEBUG:boto:Data: 2022-09-12T19:07:00.717 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.717 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.718 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.718 DEBUG:boto:Params: {} 2022-09-12T19:07:00.719 DEBUG:boto:Token: None 2022-09-12T19:07:00.719 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=TM67hyaYWRVq578zUsjsXjOzkUYTIvA 2022-09-12T19:07:00.720 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:91Ppqt0Wx1uIke0HxcOGQwTkNEg= 2022-09-12T19:07:00.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:91Ppqt0Wx1uIke0HxcOGQwTkNEg=', 'Content-Length': '0'} 2022-09-12T19:07:00.723 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'TM67hyaYWRVq578zUsjsXjOzkUYTIvA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ac4a1ff7bbd4185-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.724 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:00.724 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:00.725 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:00.725 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:00.726 DEBUG:boto:Method: GET 2022-09-12T19:07:00.726 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:00.727 DEBUG:boto:Data: 2022-09-12T19:07:00.728 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.729 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.730 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.730 DEBUG:boto:Params: {} 2022-09-12T19:07:00.731 DEBUG:boto:Token: None 2022-09-12T19:07:00.731 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/ 2022-09-12T19:07:00.732 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM= 2022-09-12T19:07:00.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM=', 'Content-Length': '0'} 2022-09-12T19:07:00.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e12125df09f9c885-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.736 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:00.737 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:00.738 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:00.739 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:00.739 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:00.740 DEBUG:boto:Method: GET 2022-09-12T19:07:00.740 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:00.741 DEBUG:boto:Data: 2022-09-12T19:07:00.741 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.742 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.742 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.743 DEBUG:boto:Params: {} 2022-09-12T19:07:00.743 DEBUG:boto:Token: None 2022-09-12T19:07:00.744 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/ 2022-09-12T19:07:00.745 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM= 2022-09-12T19:07:00.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM=', 'Content-Length': '0'} 2022-09-12T19:07:00.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000be0e40983b31514-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.790 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:00.791 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.792 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.792 DEBUG:boto:Method: HEAD 2022-09-12T19:07:00.793 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:00.793 DEBUG:boto:Data: 2022-09-12T19:07:00.794 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.794 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.795 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.795 DEBUG:boto:Params: {} 2022-09-12T19:07:00.796 DEBUG:boto:Token: None 2022-09-12T19:07:00.796 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj 2022-09-12T19:07:00.797 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/SRbOnRq3u+w1jQHvGboCsCe5a4= 2022-09-12T19:07:00.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/SRbOnRq3u+w1jQHvGboCsCe5a4=', 'Content-Length': '0'} 2022-09-12T19:07:00.801 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7649a22ddf6ee5c-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.802 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.803 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.803 DEBUG:boto:Method: HEAD 2022-09-12T19:07:00.804 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:00.804 DEBUG:boto:Data: 2022-09-12T19:07:00.805 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.805 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.806 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.806 DEBUG:boto:Params: {} 2022-09-12T19:07:00.807 DEBUG:boto:Token: None 2022-09-12T19:07:00.807 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj 2022-09-12T19:07:00.808 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/SRbOnRq3u+w1jQHvGboCsCe5a4= 2022-09-12T19:07:00.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/SRbOnRq3u+w1jQHvGboCsCe5a4=', 'Content-Length': '0'} 2022-09-12T19:07:00.812 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047994d4824dd7ed9-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:00.813 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.814 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.814 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.815 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.815 DEBUG:boto:Method: GET 2022-09-12T19:07:00.816 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.816 DEBUG:boto:Data: 2022-09-12T19:07:00.817 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.817 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.818 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.818 DEBUG:boto:Params: {} 2022-09-12T19:07:00.819 DEBUG:boto:Token: None 2022-09-12T19:07:00.820 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.820 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xnrBMH3xFSKE/P9hjIntOdHfhak= 2022-09-12T19:07:00.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xnrBMH3xFSKE/P9hjIntOdHfhak=', 'Content-Length': '0'} 2022-09-12T19:07:00.823 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007bdc1c00eff66294-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.824 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.824 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.825 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.826 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.826 DEBUG:boto:Method: GET 2022-09-12T19:07:00.827 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.827 DEBUG:boto:Data: 2022-09-12T19:07:00.828 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.828 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.829 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.829 DEBUG:boto:Params: {} 2022-09-12T19:07:00.830 DEBUG:boto:Token: None 2022-09-12T19:07:00.831 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:00.831 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xnrBMH3xFSKE/P9hjIntOdHfhak= 2022-09-12T19:07:00.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xnrBMH3xFSKE/P9hjIntOdHfhak=', 'Content-Length': '0'} 2022-09-12T19:07:00.834 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009990d49d80ac7716-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.835 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:00.835 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:00.836 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:00.836 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:00.837 DEBUG:boto:Method: GET 2022-09-12T19:07:00.838 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:00.838 DEBUG:boto:Data: 2022-09-12T19:07:00.839 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.839 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.840 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.840 DEBUG:boto:Params: {} 2022-09-12T19:07:00.841 DEBUG:boto:Token: None 2022-09-12T19:07:00.841 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/ 2022-09-12T19:07:00.842 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM= 2022-09-12T19:07:00.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM=', 'Content-Length': '0'} 2022-09-12T19:07:00.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a575e07ba5f4cd1b-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.886 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:00.887 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:00.888 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:00.888 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:00.889 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:00.889 DEBUG:boto:Method: GET 2022-09-12T19:07:00.890 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:00.890 DEBUG:boto:Data: 2022-09-12T19:07:00.891 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.891 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.892 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.892 DEBUG:boto:Params: {} 2022-09-12T19:07:00.893 DEBUG:boto:Token: None 2022-09-12T19:07:00.894 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/ 2022-09-12T19:07:00.894 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM= 2022-09-12T19:07:00.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b6brTjLPOMAGvwx0M6NiXmv6mAM=', 'Content-Length': '0'} 2022-09-12T19:07:00.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd42987309465daf-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.899 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:00.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:00.900 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.901 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.901 DEBUG:boto:path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:07:00.902 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:07:00.903 DEBUG:boto:Method: GET 2022-09-12T19:07:00.904 DEBUG:boto:Path: /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:07:00.904 DEBUG:boto:Data: 2022-09-12T19:07:00.905 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.905 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.906 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.906 DEBUG:boto:Params: {} 2022-09-12T19:07:00.907 DEBUG:boto:Token: None 2022-09-12T19:07:00.908 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:07:00.908 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bI7N1CEOQ4sDvXvycBvADuUElkc= 2022-09-12T19:07:00.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bI7N1CEOQ4sDvXvycBvADuUElkc=', 'Content-Length': '0'} 2022-09-12T19:07:00.911 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', '-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000060406094c40a16b-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.912 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.912 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.913 DEBUG:boto:path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:07:00.913 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:07:00.914 DEBUG:boto:Method: GET 2022-09-12T19:07:00.914 DEBUG:boto:Path: /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:07:00.915 DEBUG:boto:Data: 2022-09-12T19:07:00.915 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.916 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.916 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.917 DEBUG:boto:Params: {} 2022-09-12T19:07:00.917 DEBUG:boto:Token: None 2022-09-12T19:07:00.918 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:07:00.919 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bI7N1CEOQ4sDvXvycBvADuUElkc= 2022-09-12T19:07:00.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bI7N1CEOQ4sDvXvycBvADuUElkc=', 'Content-Length': '0'} 2022-09-12T19:07:00.922 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', '-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ab16e9de2e5204a-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.962 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.963 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.963 DEBUG:boto:path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:07:00.964 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:07:00.964 DEBUG:boto:Method: HEAD 2022-09-12T19:07:00.965 DEBUG:boto:Path: /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:07:00.965 DEBUG:boto:Data: 2022-09-12T19:07:00.966 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.966 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.967 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.967 DEBUG:boto:Params: {} 2022-09-12T19:07:00.968 DEBUG:boto:Token: None 2022-09-12T19:07:00.969 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:07:00.969 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t6Wn+UE9wjXb0qzXYM4KTO0cb9c= 2022-09-12T19:07:00.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t6Wn+UE9wjXb0qzXYM4KTO0cb9c=', 'Content-Length': '0'} 2022-09-12T19:07:00.972 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', '-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cfff5c584c1560ec-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.973 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.973 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.974 DEBUG:boto:path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:07:00.974 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:07:00.975 DEBUG:boto:Method: HEAD 2022-09-12T19:07:00.975 DEBUG:boto:Path: /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:07:00.976 DEBUG:boto:Data: 2022-09-12T19:07:00.976 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.977 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:00.977 DEBUG:boto:Port: 8000 2022-09-12T19:07:00.978 DEBUG:boto:Params: {} 2022-09-12T19:07:00.978 DEBUG:boto:Token: None 2022-09-12T19:07:00.979 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:07:00.979 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t6Wn+UE9wjXb0qzXYM4KTO0cb9c= 2022-09-12T19:07:00.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t6Wn+UE9wjXb0qzXYM4KTO0cb9c=', 'Content-Length': '0'} 2022-09-12T19:07:00.982 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', '-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e222cb732c3341a-00631f8354-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:00.984 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.984 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.985 DEBUG:boto:path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:07:00.985 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:07:00.986 DEBUG:boto:Method: GET 2022-09-12T19:07:00.987 DEBUG:boto:Path: /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:07:00.987 DEBUG:boto:Data: 2022-09-12T19:07:00.988 DEBUG:boto:Headers: {} 2022-09-12T19:07:00.988 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:00.989 DEBUG:boto:Port: 8001 2022-09-12T19:07:00.989 DEBUG:boto:Params: {} 2022-09-12T19:07:00.990 DEBUG:boto:Token: None 2022-09-12T19:07:00.991 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:00 GMT /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:07:00.991 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bI7N1CEOQ4sDvXvycBvADuUElkc= 2022-09-12T19:07:00.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bI7N1CEOQ4sDvXvycBvADuUElkc=', 'Content-Length': '0'} 2022-09-12T19:07:00.994 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', '-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054515eca03082179-00631f8354-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:00.995 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:00.996 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:00.996 DEBUG:boto:path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:07:00.997 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:07:00.997 DEBUG:boto:Method: GET 2022-09-12T19:07:00.998 DEBUG:boto:Path: /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:07:00.998 DEBUG:boto:Data: 2022-09-12T19:07:00.999 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.000 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.000 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.001 DEBUG:boto:Params: {} 2022-09-12T19:07:01.001 DEBUG:boto:Token: None 2022-09-12T19:07:01.002 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh 2022-09-12T19:07:01.002 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HJ+75jpOJ1tJlE5ZU1ZqESaC5q0= 2022-09-12T19:07:01.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HJ+75jpOJ1tJlE5ZU1ZqESaC5q0=', 'Content-Length': '0'} 2022-09-12T19:07:01.005 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', '-QoQyBF5hY1.NuPeIZeaAQlt6O1k3rh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bac2b8dea0e65a38-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.006 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:01.007 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:01.007 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.008 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.008 DEBUG:boto:Method: GET 2022-09-12T19:07:01.009 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:01.010 DEBUG:boto:Data: 2022-09-12T19:07:01.010 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.011 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.011 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.012 DEBUG:boto:Params: {} 2022-09-12T19:07:01.013 DEBUG:boto:Token: None 2022-09-12T19:07:01.013 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/ 2022-09-12T19:07:01.014 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk= 2022-09-12T19:07:01.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk=', 'Content-Length': '0'} 2022-09-12T19:07:01.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085295924a693c62e-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.018 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:01.019 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:01.020 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:01.020 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.021 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.021 DEBUG:boto:Method: GET 2022-09-12T19:07:01.022 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:01.022 DEBUG:boto:Data: 2022-09-12T19:07:01.023 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.023 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.024 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.024 DEBUG:boto:Params: {} 2022-09-12T19:07:01.025 DEBUG:boto:Token: None 2022-09-12T19:07:01.026 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/ 2022-09-12T19:07:01.026 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk= 2022-09-12T19:07:01.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk=', 'Content-Length': '0'} 2022-09-12T19:07:01.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d3acaa8e8b35f12-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.030 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:01.031 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.032 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.032 DEBUG:boto:Method: HEAD 2022-09-12T19:07:01.033 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:01.033 DEBUG:boto:Data: 2022-09-12T19:07:01.034 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.034 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.035 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.035 DEBUG:boto:Params: {} 2022-09-12T19:07:01.036 DEBUG:boto:Token: None 2022-09-12T19:07:01.037 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj 2022-09-12T19:07:01.038 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1uP79igP8gMuPMLMKZFxL0z16YU= 2022-09-12T19:07:01.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1uP79igP8gMuPMLMKZFxL0z16YU=', 'Content-Length': '0'} 2022-09-12T19:07:01.042 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000853a36f9ccf7e4d1-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.043 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.043 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.044 DEBUG:boto:Method: HEAD 2022-09-12T19:07:01.045 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:01.045 DEBUG:boto:Data: 2022-09-12T19:07:01.046 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.046 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.047 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.047 DEBUG:boto:Params: {} 2022-09-12T19:07:01.048 DEBUG:boto:Token: None 2022-09-12T19:07:01.048 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj 2022-09-12T19:07:01.049 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1uP79igP8gMuPMLMKZFxL0z16YU= 2022-09-12T19:07:01.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1uP79igP8gMuPMLMKZFxL0z16YU=', 'Content-Length': '0'} 2022-09-12T19:07:01.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bba54c1953bbd989-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:01.054 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.055 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.055 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.056 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.056 DEBUG:boto:Method: GET 2022-09-12T19:07:01.057 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.058 DEBUG:boto:Data: 2022-09-12T19:07:01.058 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.059 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.059 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.060 DEBUG:boto:Params: {} 2022-09-12T19:07:01.061 DEBUG:boto:Token: None 2022-09-12T19:07:01.061 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.062 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vJemCwWDn/S/aHgZQMwDeB2U5vA= 2022-09-12T19:07:01.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vJemCwWDn/S/aHgZQMwDeB2U5vA=', 'Content-Length': '0'} 2022-09-12T19:07:01.065 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f685b4a9ae236f5-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.106 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.107 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.108 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.109 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.109 DEBUG:boto:Method: GET 2022-09-12T19:07:01.110 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.110 DEBUG:boto:Data: 2022-09-12T19:07:01.111 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.112 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.112 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.113 DEBUG:boto:Params: {} 2022-09-12T19:07:01.113 DEBUG:boto:Token: None 2022-09-12T19:07:01.114 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.115 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vJemCwWDn/S/aHgZQMwDeB2U5vA= 2022-09-12T19:07:01.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vJemCwWDn/S/aHgZQMwDeB2U5vA=', 'Content-Length': '0'} 2022-09-12T19:07:01.117 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df6612dccf2c0085-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.118 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:01.119 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:01.120 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.120 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.121 DEBUG:boto:Method: GET 2022-09-12T19:07:01.121 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:01.122 DEBUG:boto:Data: 2022-09-12T19:07:01.122 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.123 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.123 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.124 DEBUG:boto:Params: {} 2022-09-12T19:07:01.124 DEBUG:boto:Token: None 2022-09-12T19:07:01.125 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/ 2022-09-12T19:07:01.125 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk= 2022-09-12T19:07:01.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk=', 'Content-Length': '0'} 2022-09-12T19:07:01.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d30cb7d483c184f-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.130 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:01.131 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:01.131 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:01.132 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.133 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.133 DEBUG:boto:Method: GET 2022-09-12T19:07:01.134 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:01.134 DEBUG:boto:Data: 2022-09-12T19:07:01.135 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.135 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.136 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.136 DEBUG:boto:Params: {} 2022-09-12T19:07:01.137 DEBUG:boto:Token: None 2022-09-12T19:07:01.137 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/ 2022-09-12T19:07:01.138 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk= 2022-09-12T19:07:01.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk=', 'Content-Length': '0'} 2022-09-12T19:07:01.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000679836bf560542b6-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.142 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:01.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:01.144 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.144 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.145 DEBUG:boto:path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:07:01.145 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:07:01.146 DEBUG:boto:Method: GET 2022-09-12T19:07:01.146 DEBUG:boto:Path: /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:07:01.147 DEBUG:boto:Data: 2022-09-12T19:07:01.147 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.148 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.149 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.149 DEBUG:boto:Params: {} 2022-09-12T19:07:01.150 DEBUG:boto:Token: None 2022-09-12T19:07:01.150 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:07:01.151 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fwn8T5aurEVs4/m+UIkUr52gAl0= 2022-09-12T19:07:01.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fwn8T5aurEVs4/m+UIkUr52gAl0=', 'Content-Length': '0'} 2022-09-12T19:07:01.154 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'pwncGyI2j8AeHH4hjzwsep2DWsp-l-A'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd58ae0f9cb856a1-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.155 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.155 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.156 DEBUG:boto:path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:07:01.157 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:07:01.157 DEBUG:boto:Method: GET 2022-09-12T19:07:01.158 DEBUG:boto:Path: /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:07:01.158 DEBUG:boto:Data: 2022-09-12T19:07:01.159 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.159 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.160 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.161 DEBUG:boto:Params: {} 2022-09-12T19:07:01.161 DEBUG:boto:Token: None 2022-09-12T19:07:01.162 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:07:01.163 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fwn8T5aurEVs4/m+UIkUr52gAl0= 2022-09-12T19:07:01.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fwn8T5aurEVs4/m+UIkUr52gAl0=', 'Content-Length': '0'} 2022-09-12T19:07:01.165 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'pwncGyI2j8AeHH4hjzwsep2DWsp-l-A'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001fcc102b1d435157-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.166 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.167 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.167 DEBUG:boto:path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:07:01.168 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:07:01.168 DEBUG:boto:Method: HEAD 2022-09-12T19:07:01.169 DEBUG:boto:Path: /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:07:01.169 DEBUG:boto:Data: 2022-09-12T19:07:01.170 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.170 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.171 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.172 DEBUG:boto:Params: {} 2022-09-12T19:07:01.172 DEBUG:boto:Token: None 2022-09-12T19:07:01.173 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:07:01.173 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9gKN0iG2VnGw0TdgnZc3+9gLjEc= 2022-09-12T19:07:01.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9gKN0iG2VnGw0TdgnZc3+9gLjEc=', 'Content-Length': '0'} 2022-09-12T19:07:01.176 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'pwncGyI2j8AeHH4hjzwsep2DWsp-l-A'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e75190619a8a4ce-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.177 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.177 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.178 DEBUG:boto:path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:07:01.178 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:07:01.179 DEBUG:boto:Method: HEAD 2022-09-12T19:07:01.179 DEBUG:boto:Path: /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:07:01.180 DEBUG:boto:Data: 2022-09-12T19:07:01.180 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.181 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.182 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.182 DEBUG:boto:Params: {} 2022-09-12T19:07:01.183 DEBUG:boto:Token: None 2022-09-12T19:07:01.183 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:07:01.184 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9gKN0iG2VnGw0TdgnZc3+9gLjEc= 2022-09-12T19:07:01.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9gKN0iG2VnGw0TdgnZc3+9gLjEc=', 'Content-Length': '0'} 2022-09-12T19:07:01.186 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'pwncGyI2j8AeHH4hjzwsep2DWsp-l-A'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004898098448d29d36-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:01.188 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.188 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.189 DEBUG:boto:path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:07:01.189 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:07:01.190 DEBUG:boto:Method: GET 2022-09-12T19:07:01.190 DEBUG:boto:Path: /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:07:01.191 DEBUG:boto:Data: 2022-09-12T19:07:01.191 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.192 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.193 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.193 DEBUG:boto:Params: {} 2022-09-12T19:07:01.194 DEBUG:boto:Token: None 2022-09-12T19:07:01.195 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:07:01.195 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fwn8T5aurEVs4/m+UIkUr52gAl0= 2022-09-12T19:07:01.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fwn8T5aurEVs4/m+UIkUr52gAl0=', 'Content-Length': '0'} 2022-09-12T19:07:01.199 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'pwncGyI2j8AeHH4hjzwsep2DWsp-l-A'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0bd050d303bced2-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.199 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.200 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.200 DEBUG:boto:path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:07:01.201 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:07:01.202 DEBUG:boto:Method: GET 2022-09-12T19:07:01.202 DEBUG:boto:Path: /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:07:01.203 DEBUG:boto:Data: 2022-09-12T19:07:01.204 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.204 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.205 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.206 DEBUG:boto:Params: {} 2022-09-12T19:07:01.206 DEBUG:boto:Token: None 2022-09-12T19:07:01.207 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=pwncGyI2j8AeHH4hjzwsep2DWsp-l-A 2022-09-12T19:07:01.208 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fwn8T5aurEVs4/m+UIkUr52gAl0= 2022-09-12T19:07:01.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fwn8T5aurEVs4/m+UIkUr52gAl0=', 'Content-Length': '0'} 2022-09-12T19:07:01.210 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'pwncGyI2j8AeHH4hjzwsep2DWsp-l-A'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db6634ceb894a25f-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.254 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:01.255 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:01.255 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.256 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.257 DEBUG:boto:Method: GET 2022-09-12T19:07:01.257 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:01.258 DEBUG:boto:Data: 2022-09-12T19:07:01.259 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.259 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.260 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.260 DEBUG:boto:Params: {} 2022-09-12T19:07:01.261 DEBUG:boto:Token: None 2022-09-12T19:07:01.262 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/ 2022-09-12T19:07:01.262 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk= 2022-09-12T19:07:01.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk=', 'Content-Length': '0'} 2022-09-12T19:07:01.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd1ea9c6dbc0b7b3-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.269 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:01.270 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:01.271 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:01.272 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.273 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.274 DEBUG:boto:Method: GET 2022-09-12T19:07:01.275 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:01.276 DEBUG:boto:Data: 2022-09-12T19:07:01.277 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.278 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.279 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.280 DEBUG:boto:Params: {} 2022-09-12T19:07:01.282 DEBUG:boto:Token: None 2022-09-12T19:07:01.283 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/ 2022-09-12T19:07:01.285 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk= 2022-09-12T19:07:01.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk=', 'Content-Length': '0'} 2022-09-12T19:07:01.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2a35a7c3a89a882-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.292 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:01.294 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.295 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.297 DEBUG:boto:Method: HEAD 2022-09-12T19:07:01.299 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:01.300 DEBUG:boto:Data: 2022-09-12T19:07:01.301 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.302 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.303 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.305 DEBUG:boto:Params: {} 2022-09-12T19:07:01.306 DEBUG:boto:Token: None 2022-09-12T19:07:01.308 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj 2022-09-12T19:07:01.309 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1uP79igP8gMuPMLMKZFxL0z16YU= 2022-09-12T19:07:01.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1uP79igP8gMuPMLMKZFxL0z16YU=', 'Content-Length': '0'} 2022-09-12T19:07:01.315 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e37e59506a3d7c9b-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.316 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.317 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.318 DEBUG:boto:Method: HEAD 2022-09-12T19:07:01.319 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:01.319 DEBUG:boto:Data: 2022-09-12T19:07:01.320 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.321 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.321 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.322 DEBUG:boto:Params: {} 2022-09-12T19:07:01.323 DEBUG:boto:Token: None 2022-09-12T19:07:01.323 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj 2022-09-12T19:07:01.324 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1uP79igP8gMuPMLMKZFxL0z16YU= 2022-09-12T19:07:01.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1uP79igP8gMuPMLMKZFxL0z16YU=', 'Content-Length': '0'} 2022-09-12T19:07:01.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053f37aaedc2a604b-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:01.331 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.331 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.332 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.333 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.334 DEBUG:boto:Method: GET 2022-09-12T19:07:01.335 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.336 DEBUG:boto:Data: 2022-09-12T19:07:01.337 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.337 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.338 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.339 DEBUG:boto:Params: {} 2022-09-12T19:07:01.339 DEBUG:boto:Token: None 2022-09-12T19:07:01.340 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.341 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vJemCwWDn/S/aHgZQMwDeB2U5vA= 2022-09-12T19:07:01.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vJemCwWDn/S/aHgZQMwDeB2U5vA=', 'Content-Length': '0'} 2022-09-12T19:07:01.344 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e119d582c9bd3840-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.386 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.387 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.388 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.389 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.389 DEBUG:boto:Method: GET 2022-09-12T19:07:01.390 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.391 DEBUG:boto:Data: 2022-09-12T19:07:01.392 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.393 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.394 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.394 DEBUG:boto:Params: {} 2022-09-12T19:07:01.395 DEBUG:boto:Token: None 2022-09-12T19:07:01.396 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.397 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vJemCwWDn/S/aHgZQMwDeB2U5vA= 2022-09-12T19:07:01.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vJemCwWDn/S/aHgZQMwDeB2U5vA=', 'Content-Length': '0'} 2022-09-12T19:07:01.401 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9eecd8612365a5a-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.402 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:01.402 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:01.403 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.404 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.404 DEBUG:boto:Method: GET 2022-09-12T19:07:01.405 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:01.406 DEBUG:boto:Data: 2022-09-12T19:07:01.407 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.408 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.408 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.409 DEBUG:boto:Params: {} 2022-09-12T19:07:01.410 DEBUG:boto:Token: None 2022-09-12T19:07:01.411 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/ 2022-09-12T19:07:01.411 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk= 2022-09-12T19:07:01.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk=', 'Content-Length': '0'} 2022-09-12T19:07:01.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4b1dd0f9f8c91b2-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.416 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:01.417 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:01.418 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:01.418 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.420 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.420 DEBUG:boto:Method: GET 2022-09-12T19:07:01.421 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:01.421 DEBUG:boto:Data: 2022-09-12T19:07:01.422 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.423 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.423 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.424 DEBUG:boto:Params: {} 2022-09-12T19:07:01.424 DEBUG:boto:Token: None 2022-09-12T19:07:01.425 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/ 2022-09-12T19:07:01.426 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk= 2022-09-12T19:07:01.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk=', 'Content-Length': '0'} 2022-09-12T19:07:01.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff94e9fc14cbb0c9-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.431 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:01.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:01.432 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.433 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.434 DEBUG:boto:path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:07:01.434 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:07:01.435 DEBUG:boto:Method: GET 2022-09-12T19:07:01.436 DEBUG:boto:Path: /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:07:01.436 DEBUG:boto:Data: 2022-09-12T19:07:01.437 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.437 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.438 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.438 DEBUG:boto:Params: {} 2022-09-12T19:07:01.439 DEBUG:boto:Token: None 2022-09-12T19:07:01.440 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:07:01.440 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dTb2vA6oxGShBvpX28nEN8IzRqw= 2022-09-12T19:07:01.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dTb2vA6oxGShBvpX28nEN8IzRqw=', 'Content-Length': '0'} 2022-09-12T19:07:01.443 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c819504ccb763ce-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.448 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.448 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.449 DEBUG:boto:path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:07:01.449 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:07:01.450 DEBUG:boto:Method: GET 2022-09-12T19:07:01.450 DEBUG:boto:Path: /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:07:01.451 DEBUG:boto:Data: 2022-09-12T19:07:01.452 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.452 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.453 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.453 DEBUG:boto:Params: {} 2022-09-12T19:07:01.454 DEBUG:boto:Token: None 2022-09-12T19:07:01.454 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:07:01.455 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dTb2vA6oxGShBvpX28nEN8IzRqw= 2022-09-12T19:07:01.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dTb2vA6oxGShBvpX28nEN8IzRqw=', 'Content-Length': '0'} 2022-09-12T19:07:01.458 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6bc3056ff263cf5-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.506 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.508 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.509 DEBUG:boto:path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:07:01.510 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:07:01.511 DEBUG:boto:Method: HEAD 2022-09-12T19:07:01.512 DEBUG:boto:Path: /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:07:01.513 DEBUG:boto:Data: 2022-09-12T19:07:01.514 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.514 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.515 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.516 DEBUG:boto:Params: {} 2022-09-12T19:07:01.516 DEBUG:boto:Token: None 2022-09-12T19:07:01.517 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:07:01.518 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bdKL7IKxidXluaZbKcxyzX5HHsw= 2022-09-12T19:07:01.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bdKL7IKxidXluaZbKcxyzX5HHsw=', 'Content-Length': '0'} 2022-09-12T19:07:01.521 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f062f86fa1c0a770-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.522 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.523 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.523 DEBUG:boto:path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:07:01.524 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:07:01.524 DEBUG:boto:Method: HEAD 2022-09-12T19:07:01.525 DEBUG:boto:Path: /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:07:01.525 DEBUG:boto:Data: 2022-09-12T19:07:01.526 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.527 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.527 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.528 DEBUG:boto:Params: {} 2022-09-12T19:07:01.529 DEBUG:boto:Token: None 2022-09-12T19:07:01.529 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:07:01.530 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bdKL7IKxidXluaZbKcxyzX5HHsw= 2022-09-12T19:07:01.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bdKL7IKxidXluaZbKcxyzX5HHsw=', 'Content-Length': '0'} 2022-09-12T19:07:01.533 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029db0400ac5c17df-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:01.535 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.536 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.536 DEBUG:boto:path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:07:01.537 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:07:01.537 DEBUG:boto:Method: GET 2022-09-12T19:07:01.538 DEBUG:boto:Path: /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:07:01.539 DEBUG:boto:Data: 2022-09-12T19:07:01.539 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.540 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.540 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.541 DEBUG:boto:Params: {} 2022-09-12T19:07:01.542 DEBUG:boto:Token: None 2022-09-12T19:07:01.542 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:07:01.544 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dTb2vA6oxGShBvpX28nEN8IzRqw= 2022-09-12T19:07:01.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dTb2vA6oxGShBvpX28nEN8IzRqw=', 'Content-Length': '0'} 2022-09-12T19:07:01.547 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e98094d84b5a985-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.549 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.550 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.550 DEBUG:boto:path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:07:01.551 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:07:01.552 DEBUG:boto:Method: GET 2022-09-12T19:07:01.553 DEBUG:boto:Path: /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:07:01.554 DEBUG:boto:Data: 2022-09-12T19:07:01.555 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.556 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.557 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.558 DEBUG:boto:Params: {} 2022-09-12T19:07:01.559 DEBUG:boto:Token: None 2022-09-12T19:07:01.560 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ 2022-09-12T19:07:01.561 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dTb2vA6oxGShBvpX28nEN8IzRqw= 2022-09-12T19:07:01.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dTb2vA6oxGShBvpX28nEN8IzRqw=', 'Content-Length': '0'} 2022-09-12T19:07:01.565 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'sGTjDAFqZF1WL9AP98wlNVRWMwOQwYQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003197bdd27c004c83-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.566 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:01.567 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:01.568 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.569 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.570 DEBUG:boto:Method: GET 2022-09-12T19:07:01.571 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:01.572 DEBUG:boto:Data: 2022-09-12T19:07:01.572 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.573 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.574 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.575 DEBUG:boto:Params: {} 2022-09-12T19:07:01.576 DEBUG:boto:Token: None 2022-09-12T19:07:01.577 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/ 2022-09-12T19:07:01.578 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk= 2022-09-12T19:07:01.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk=', 'Content-Length': '0'} 2022-09-12T19:07:01.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bda617d7d0393c28-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.583 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:01.584 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:01.585 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:01.586 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.587 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.587 DEBUG:boto:Method: GET 2022-09-12T19:07:01.588 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:01.588 DEBUG:boto:Data: 2022-09-12T19:07:01.589 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.589 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.590 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.591 DEBUG:boto:Params: {} 2022-09-12T19:07:01.591 DEBUG:boto:Token: None 2022-09-12T19:07:01.592 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/ 2022-09-12T19:07:01.593 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk= 2022-09-12T19:07:01.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk=', 'Content-Length': '0'} 2022-09-12T19:07:01.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f5e229f9ceef89c-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.597 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:01.598 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.599 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.600 DEBUG:boto:Method: HEAD 2022-09-12T19:07:01.600 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:01.601 DEBUG:boto:Data: 2022-09-12T19:07:01.602 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.603 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.604 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.605 DEBUG:boto:Params: {} 2022-09-12T19:07:01.606 DEBUG:boto:Token: None 2022-09-12T19:07:01.607 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj 2022-09-12T19:07:01.608 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1uP79igP8gMuPMLMKZFxL0z16YU= 2022-09-12T19:07:01.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1uP79igP8gMuPMLMKZFxL0z16YU=', 'Content-Length': '0'} 2022-09-12T19:07:01.613 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e096a3b58154eb78-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.614 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.615 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.616 DEBUG:boto:Method: HEAD 2022-09-12T19:07:01.617 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:01.618 DEBUG:boto:Data: 2022-09-12T19:07:01.619 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.620 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.620 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.621 DEBUG:boto:Params: {} 2022-09-12T19:07:01.622 DEBUG:boto:Token: None 2022-09-12T19:07:01.623 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj 2022-09-12T19:07:01.624 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1uP79igP8gMuPMLMKZFxL0z16YU= 2022-09-12T19:07:01.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1uP79igP8gMuPMLMKZFxL0z16YU=', 'Content-Length': '0'} 2022-09-12T19:07:01.629 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000960fc483f220bf9f-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:01.631 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.632 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.633 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.634 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.635 DEBUG:boto:Method: GET 2022-09-12T19:07:01.636 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.637 DEBUG:boto:Data: 2022-09-12T19:07:01.638 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.639 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.639 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.640 DEBUG:boto:Params: {} 2022-09-12T19:07:01.641 DEBUG:boto:Token: None 2022-09-12T19:07:01.642 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.643 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vJemCwWDn/S/aHgZQMwDeB2U5vA= 2022-09-12T19:07:01.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vJemCwWDn/S/aHgZQMwDeB2U5vA=', 'Content-Length': '0'} 2022-09-12T19:07:01.647 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000f633dd43b55784-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.690 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.691 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.692 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.693 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.694 DEBUG:boto:Method: GET 2022-09-12T19:07:01.695 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.696 DEBUG:boto:Data: 2022-09-12T19:07:01.697 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.698 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.698 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.699 DEBUG:boto:Params: {} 2022-09-12T19:07:01.699 DEBUG:boto:Token: None 2022-09-12T19:07:01.700 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.701 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vJemCwWDn/S/aHgZQMwDeB2U5vA= 2022-09-12T19:07:01.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vJemCwWDn/S/aHgZQMwDeB2U5vA=', 'Content-Length': '0'} 2022-09-12T19:07:01.704 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f50784b8ad1ad5f3-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.705 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:01.706 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:01.706 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.707 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.707 DEBUG:boto:Method: GET 2022-09-12T19:07:01.708 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:01.709 DEBUG:boto:Data: 2022-09-12T19:07:01.709 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.710 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.710 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.711 DEBUG:boto:Params: {} 2022-09-12T19:07:01.712 DEBUG:boto:Token: None 2022-09-12T19:07:01.712 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/ 2022-09-12T19:07:01.713 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk= 2022-09-12T19:07:01.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk=', 'Content-Length': '0'} 2022-09-12T19:07:01.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a8628ff31c4b057-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.717 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:01.718 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:01.718 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:01.719 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.720 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.721 DEBUG:boto:Method: GET 2022-09-12T19:07:01.721 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:01.722 DEBUG:boto:Data: 2022-09-12T19:07:01.722 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.723 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.723 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.724 DEBUG:boto:Params: {} 2022-09-12T19:07:01.725 DEBUG:boto:Token: None 2022-09-12T19:07:01.725 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/ 2022-09-12T19:07:01.726 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk= 2022-09-12T19:07:01.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk=', 'Content-Length': '0'} 2022-09-12T19:07:01.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c05026c71b066046-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.730 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:01.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:01.731 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.732 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.733 DEBUG:boto:path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:07:01.733 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:07:01.734 DEBUG:boto:Method: GET 2022-09-12T19:07:01.734 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:07:01.735 DEBUG:boto:Data: 2022-09-12T19:07:01.735 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.736 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.736 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.737 DEBUG:boto:Params: {} 2022-09-12T19:07:01.738 DEBUG:boto:Token: None 2022-09-12T19:07:01.738 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:07:01.739 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BE8lTniL+DXB/K4saBwn3PTD2Aw= 2022-09-12T19:07:01.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BE8lTniL+DXB/K4saBwn3PTD2Aw=', 'Content-Length': '0'} 2022-09-12T19:07:01.742 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'T.yM--FII-PB1GReM85mkNRGYz6Dkbz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe79ee0974118268-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.743 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.744 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.745 DEBUG:boto:path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:07:01.746 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:07:01.747 DEBUG:boto:Method: GET 2022-09-12T19:07:01.748 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:07:01.749 DEBUG:boto:Data: 2022-09-12T19:07:01.750 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.751 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.752 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.752 DEBUG:boto:Params: {} 2022-09-12T19:07:01.753 DEBUG:boto:Token: None 2022-09-12T19:07:01.754 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:07:01.755 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BE8lTniL+DXB/K4saBwn3PTD2Aw= 2022-09-12T19:07:01.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BE8lTniL+DXB/K4saBwn3PTD2Aw=', 'Content-Length': '0'} 2022-09-12T19:07:01.759 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'T.yM--FII-PB1GReM85mkNRGYz6Dkbz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005dea9a4632c412cf-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.760 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.761 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.762 DEBUG:boto:path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:07:01.763 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:07:01.764 DEBUG:boto:Method: HEAD 2022-09-12T19:07:01.765 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:07:01.765 DEBUG:boto:Data: 2022-09-12T19:07:01.766 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.767 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.768 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.768 DEBUG:boto:Params: {} 2022-09-12T19:07:01.769 DEBUG:boto:Token: None 2022-09-12T19:07:01.769 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:07:01.770 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qzaVbLEc4kDjKmlbGcpvS5rLWDA= 2022-09-12T19:07:01.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qzaVbLEc4kDjKmlbGcpvS5rLWDA=', 'Content-Length': '0'} 2022-09-12T19:07:01.773 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'T.yM--FII-PB1GReM85mkNRGYz6Dkbz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000327d6e09c791a3fa-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.774 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.774 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.775 DEBUG:boto:path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:07:01.775 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:07:01.776 DEBUG:boto:Method: HEAD 2022-09-12T19:07:01.776 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:07:01.777 DEBUG:boto:Data: 2022-09-12T19:07:01.777 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.778 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.778 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.779 DEBUG:boto:Params: {} 2022-09-12T19:07:01.780 DEBUG:boto:Token: None 2022-09-12T19:07:01.780 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:07:01.781 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qzaVbLEc4kDjKmlbGcpvS5rLWDA= 2022-09-12T19:07:01.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qzaVbLEc4kDjKmlbGcpvS5rLWDA=', 'Content-Length': '0'} 2022-09-12T19:07:01.783 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'T.yM--FII-PB1GReM85mkNRGYz6Dkbz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053124ad9df988cd2-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:01.785 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.785 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.786 DEBUG:boto:path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:07:01.786 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:07:01.787 DEBUG:boto:Method: GET 2022-09-12T19:07:01.788 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:07:01.788 DEBUG:boto:Data: 2022-09-12T19:07:01.789 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.789 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.790 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.790 DEBUG:boto:Params: {} 2022-09-12T19:07:01.791 DEBUG:boto:Token: None 2022-09-12T19:07:01.792 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:07:01.793 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BE8lTniL+DXB/K4saBwn3PTD2Aw= 2022-09-12T19:07:01.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BE8lTniL+DXB/K4saBwn3PTD2Aw=', 'Content-Length': '0'} 2022-09-12T19:07:01.795 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'T.yM--FII-PB1GReM85mkNRGYz6Dkbz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1db3076132b27bb-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.796 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.797 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.798 DEBUG:boto:path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:07:01.798 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:07:01.799 DEBUG:boto:Method: GET 2022-09-12T19:07:01.799 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:07:01.800 DEBUG:boto:Data: 2022-09-12T19:07:01.801 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.801 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.802 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.802 DEBUG:boto:Params: {} 2022-09-12T19:07:01.803 DEBUG:boto:Token: None 2022-09-12T19:07:01.804 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=T.yM--FII-PB1GReM85mkNRGYz6Dkbz 2022-09-12T19:07:01.805 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BE8lTniL+DXB/K4saBwn3PTD2Aw= 2022-09-12T19:07:01.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BE8lTniL+DXB/K4saBwn3PTD2Aw=', 'Content-Length': '0'} 2022-09-12T19:07:01.807 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'T.yM--FII-PB1GReM85mkNRGYz6Dkbz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a85f790986adae6-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.808 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:01.809 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:01.810 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.810 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.811 DEBUG:boto:Method: GET 2022-09-12T19:07:01.811 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:01.812 DEBUG:boto:Data: 2022-09-12T19:07:01.813 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.814 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.815 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.816 DEBUG:boto:Params: {} 2022-09-12T19:07:01.817 DEBUG:boto:Token: None 2022-09-12T19:07:01.818 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/ 2022-09-12T19:07:01.819 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk= 2022-09-12T19:07:01.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk=', 'Content-Length': '0'} 2022-09-12T19:07:01.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b1111304ebc8fff-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.824 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:01.825 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:01.826 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:01.826 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.827 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.828 DEBUG:boto:Method: GET 2022-09-12T19:07:01.828 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:01.829 DEBUG:boto:Data: 2022-09-12T19:07:01.829 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.830 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.830 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.831 DEBUG:boto:Params: {} 2022-09-12T19:07:01.831 DEBUG:boto:Token: None 2022-09-12T19:07:01.832 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/ 2022-09-12T19:07:01.833 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk= 2022-09-12T19:07:01.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk=', 'Content-Length': '0'} 2022-09-12T19:07:01.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7589036d169a4bf-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.837 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:01.838 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.838 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.839 DEBUG:boto:Method: HEAD 2022-09-12T19:07:01.839 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:01.840 DEBUG:boto:Data: 2022-09-12T19:07:01.840 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.841 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.841 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.842 DEBUG:boto:Params: {} 2022-09-12T19:07:01.842 DEBUG:boto:Token: None 2022-09-12T19:07:01.843 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj 2022-09-12T19:07:01.843 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1uP79igP8gMuPMLMKZFxL0z16YU= 2022-09-12T19:07:01.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1uP79igP8gMuPMLMKZFxL0z16YU=', 'Content-Length': '0'} 2022-09-12T19:07:01.847 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006eb23dfeb540e0da-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.848 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.848 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.849 DEBUG:boto:Method: HEAD 2022-09-12T19:07:01.849 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:01.850 DEBUG:boto:Data: 2022-09-12T19:07:01.850 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.851 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.851 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.852 DEBUG:boto:Params: {} 2022-09-12T19:07:01.852 DEBUG:boto:Token: None 2022-09-12T19:07:01.853 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj 2022-09-12T19:07:01.854 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1uP79igP8gMuPMLMKZFxL0z16YU= 2022-09-12T19:07:01.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1uP79igP8gMuPMLMKZFxL0z16YU=', 'Content-Length': '0'} 2022-09-12T19:07:01.858 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007003db235a423de7-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:01.859 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.860 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.860 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.861 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.861 DEBUG:boto:Method: GET 2022-09-12T19:07:01.862 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.862 DEBUG:boto:Data: 2022-09-12T19:07:01.863 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.863 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.864 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.864 DEBUG:boto:Params: {} 2022-09-12T19:07:01.865 DEBUG:boto:Token: None 2022-09-12T19:07:01.866 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.866 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vJemCwWDn/S/aHgZQMwDeB2U5vA= 2022-09-12T19:07:01.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vJemCwWDn/S/aHgZQMwDeB2U5vA=', 'Content-Length': '0'} 2022-09-12T19:07:01.869 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000040cc4eb4c75666bc-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.911 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.912 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.913 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.914 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.915 DEBUG:boto:Method: GET 2022-09-12T19:07:01.915 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.916 DEBUG:boto:Data: 2022-09-12T19:07:01.917 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.918 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.918 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.919 DEBUG:boto:Params: {} 2022-09-12T19:07:01.919 DEBUG:boto:Token: None 2022-09-12T19:07:01.920 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:01.921 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vJemCwWDn/S/aHgZQMwDeB2U5vA= 2022-09-12T19:07:01.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vJemCwWDn/S/aHgZQMwDeB2U5vA=', 'Content-Length': '0'} 2022-09-12T19:07:01.924 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035567d7b2d373235-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.924 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:01.925 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:01.926 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.926 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.927 DEBUG:boto:Method: GET 2022-09-12T19:07:01.927 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:01.928 DEBUG:boto:Data: 2022-09-12T19:07:01.928 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.929 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.929 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.930 DEBUG:boto:Params: {} 2022-09-12T19:07:01.931 DEBUG:boto:Token: None 2022-09-12T19:07:01.931 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/ 2022-09-12T19:07:01.932 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk= 2022-09-12T19:07:01.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk=', 'Content-Length': '0'} 2022-09-12T19:07:01.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e9e8a2fda8bd91d-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.936 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:01.937 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:01.937 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:01.938 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.938 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:01.939 DEBUG:boto:Method: GET 2022-09-12T19:07:01.940 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:01.940 DEBUG:boto:Data: 2022-09-12T19:07:01.941 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.941 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.942 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.943 DEBUG:boto:Params: {} 2022-09-12T19:07:01.943 DEBUG:boto:Token: None 2022-09-12T19:07:01.944 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/ 2022-09-12T19:07:01.944 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk= 2022-09-12T19:07:01.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/pGtZ2Y0n/huCoCNumcaxrij9rk=', 'Content-Length': '0'} 2022-09-12T19:07:01.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062c971e52384c8c4-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.948 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:01.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:01.950 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.950 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.951 DEBUG:boto:path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:07:01.951 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:07:01.952 DEBUG:boto:Method: GET 2022-09-12T19:07:01.952 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:07:01.953 DEBUG:boto:Data: 2022-09-12T19:07:01.953 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.954 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.955 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.955 DEBUG:boto:Params: {} 2022-09-12T19:07:01.956 DEBUG:boto:Token: None 2022-09-12T19:07:01.957 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:07:01.957 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3YZRI6JFzLvdTa78uBTXpw7X8Us= 2022-09-12T19:07:01.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3YZRI6JFzLvdTa78uBTXpw7X8Us=', 'Content-Length': '0'} 2022-09-12T19:07:01.960 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Xb5UZAvutXF6GhBCMdUQgjOLi998.hK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002514bb51d3d98363-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.961 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.962 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.962 DEBUG:boto:path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:07:01.963 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:07:01.963 DEBUG:boto:Method: GET 2022-09-12T19:07:01.964 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:07:01.964 DEBUG:boto:Data: 2022-09-12T19:07:01.965 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.965 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.966 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.966 DEBUG:boto:Params: {} 2022-09-12T19:07:01.967 DEBUG:boto:Token: None 2022-09-12T19:07:01.968 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:07:01.968 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3YZRI6JFzLvdTa78uBTXpw7X8Us= 2022-09-12T19:07:01.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3YZRI6JFzLvdTa78uBTXpw7X8Us=', 'Content-Length': '0'} 2022-09-12T19:07:01.971 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Xb5UZAvutXF6GhBCMdUQgjOLi998.hK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007472b1b5da19404c-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.972 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.972 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.973 DEBUG:boto:path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:07:01.973 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:07:01.974 DEBUG:boto:Method: HEAD 2022-09-12T19:07:01.974 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:07:01.975 DEBUG:boto:Data: 2022-09-12T19:07:01.975 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.976 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:01.976 DEBUG:boto:Port: 8001 2022-09-12T19:07:01.977 DEBUG:boto:Params: {} 2022-09-12T19:07:01.977 DEBUG:boto:Token: None 2022-09-12T19:07:01.978 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:07:01.978 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:84c0NMeIaz4T03ONQ3CHvKofxTc= 2022-09-12T19:07:01.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:84c0NMeIaz4T03ONQ3CHvKofxTc=', 'Content-Length': '0'} 2022-09-12T19:07:01.981 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Xb5UZAvutXF6GhBCMdUQgjOLi998.hK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a50ce3234e03136-00631f8355-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.982 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.982 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.983 DEBUG:boto:path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:07:01.983 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:07:01.984 DEBUG:boto:Method: HEAD 2022-09-12T19:07:01.985 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:07:01.986 DEBUG:boto:Data: 2022-09-12T19:07:01.987 DEBUG:boto:Headers: {} 2022-09-12T19:07:01.987 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:01.988 DEBUG:boto:Port: 8000 2022-09-12T19:07:01.989 DEBUG:boto:Params: {} 2022-09-12T19:07:01.990 DEBUG:boto:Token: None 2022-09-12T19:07:01.991 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:01 GMT /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:07:01.992 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:84c0NMeIaz4T03ONQ3CHvKofxTc= 2022-09-12T19:07:01.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:84c0NMeIaz4T03ONQ3CHvKofxTc=', 'Content-Length': '0'} 2022-09-12T19:07:01.995 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Xb5UZAvutXF6GhBCMdUQgjOLi998.hK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e95a90c4cc6a0e9-00631f8355-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:01.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:01.996 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:01.997 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:01.997 DEBUG:boto:path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:07:01.998 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:07:01.999 DEBUG:boto:Method: GET 2022-09-12T19:07:01.999 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:07:02.000 DEBUG:boto:Data: 2022-09-12T19:07:02.000 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.001 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.001 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.002 DEBUG:boto:Params: {} 2022-09-12T19:07:02.002 DEBUG:boto:Token: None 2022-09-12T19:07:02.003 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:07:02.004 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6owblKzx6Jd0yZHQKQUkbRV1HT8= 2022-09-12T19:07:02.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6owblKzx6Jd0yZHQKQUkbRV1HT8=', 'Content-Length': '0'} 2022-09-12T19:07:02.006 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Xb5UZAvutXF6GhBCMdUQgjOLi998.hK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095726dc24140a68f-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.007 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.008 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.008 DEBUG:boto:path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:07:02.009 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:07:02.009 DEBUG:boto:Method: GET 2022-09-12T19:07:02.010 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:07:02.011 DEBUG:boto:Data: 2022-09-12T19:07:02.011 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.012 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.012 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.013 DEBUG:boto:Params: {} 2022-09-12T19:07:02.013 DEBUG:boto:Token: None 2022-09-12T19:07:02.014 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=Xb5UZAvutXF6GhBCMdUQgjOLi998.hK 2022-09-12T19:07:02.015 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6owblKzx6Jd0yZHQKQUkbRV1HT8= 2022-09-12T19:07:02.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6owblKzx6Jd0yZHQKQUkbRV1HT8=', 'Content-Length': '0'} 2022-09-12T19:07:02.017 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Xb5UZAvutXF6GhBCMdUQgjOLi998.hK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099d06a049c819dc3-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.058 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:02.059 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:02.059 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.060 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.061 DEBUG:boto:Method: GET 2022-09-12T19:07:02.061 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:02.062 DEBUG:boto:Data: 2022-09-12T19:07:02.062 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.063 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.063 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.064 DEBUG:boto:Params: {} 2022-09-12T19:07:02.064 DEBUG:boto:Token: None 2022-09-12T19:07:02.065 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/ 2022-09-12T19:07:02.066 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA= 2022-09-12T19:07:02.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA=', 'Content-Length': '0'} 2022-09-12T19:07:02.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a85d4c2068ef5cc8-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.070 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:02.071 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:02.072 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:02.072 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.073 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.073 DEBUG:boto:Method: GET 2022-09-12T19:07:02.074 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:02.074 DEBUG:boto:Data: 2022-09-12T19:07:02.075 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.075 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.076 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.077 DEBUG:boto:Params: {} 2022-09-12T19:07:02.078 DEBUG:boto:Token: None 2022-09-12T19:07:02.078 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/ 2022-09-12T19:07:02.079 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA= 2022-09-12T19:07:02.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA=', 'Content-Length': '0'} 2022-09-12T19:07:02.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068a42f9fc3d6162e-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.083 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:02.084 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.085 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.086 DEBUG:boto:Method: HEAD 2022-09-12T19:07:02.087 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:02.088 DEBUG:boto:Data: 2022-09-12T19:07:02.089 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.091 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.091 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.092 DEBUG:boto:Params: {} 2022-09-12T19:07:02.093 DEBUG:boto:Token: None 2022-09-12T19:07:02.093 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj 2022-09-12T19:07:02.094 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WeZyJospFU7xYVCz/MpW6ZDU3M4= 2022-09-12T19:07:02.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WeZyJospFU7xYVCz/MpW6ZDU3M4=', 'Content-Length': '0'} 2022-09-12T19:07:02.098 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a9ee7c28288e945-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.099 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.100 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.101 DEBUG:boto:Method: HEAD 2022-09-12T19:07:02.101 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:02.102 DEBUG:boto:Data: 2022-09-12T19:07:02.102 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.103 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.103 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.104 DEBUG:boto:Params: {} 2022-09-12T19:07:02.104 DEBUG:boto:Token: None 2022-09-12T19:07:02.105 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj 2022-09-12T19:07:02.105 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WeZyJospFU7xYVCz/MpW6ZDU3M4= 2022-09-12T19:07:02.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WeZyJospFU7xYVCz/MpW6ZDU3M4=', 'Content-Length': '0'} 2022-09-12T19:07:02.109 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8b01a81607baebf-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:02.111 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.111 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.112 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.113 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.114 DEBUG:boto:Method: GET 2022-09-12T19:07:02.115 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.116 DEBUG:boto:Data: 2022-09-12T19:07:02.117 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.118 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.119 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.120 DEBUG:boto:Params: {} 2022-09-12T19:07:02.121 DEBUG:boto:Token: None 2022-09-12T19:07:02.122 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.124 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HaivMvI9CZ6izJ4q+Pel5pjwrTo= 2022-09-12T19:07:02.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HaivMvI9CZ6izJ4q+Pel5pjwrTo=', 'Content-Length': '0'} 2022-09-12T19:07:02.127 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0822a1a012f31f0-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.174 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.176 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.176 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.177 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.178 DEBUG:boto:Method: GET 2022-09-12T19:07:02.178 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.179 DEBUG:boto:Data: 2022-09-12T19:07:02.180 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.180 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.181 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.182 DEBUG:boto:Params: {} 2022-09-12T19:07:02.183 DEBUG:boto:Token: None 2022-09-12T19:07:02.185 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.186 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HaivMvI9CZ6izJ4q+Pel5pjwrTo= 2022-09-12T19:07:02.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HaivMvI9CZ6izJ4q+Pel5pjwrTo=', 'Content-Length': '0'} 2022-09-12T19:07:02.189 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5f25c457d48f3ff-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.190 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:02.190 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:02.191 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.191 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.192 DEBUG:boto:Method: GET 2022-09-12T19:07:02.192 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:02.193 DEBUG:boto:Data: 2022-09-12T19:07:02.193 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.194 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.195 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.195 DEBUG:boto:Params: {} 2022-09-12T19:07:02.196 DEBUG:boto:Token: None 2022-09-12T19:07:02.196 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/ 2022-09-12T19:07:02.197 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA= 2022-09-12T19:07:02.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA=', 'Content-Length': '0'} 2022-09-12T19:07:02.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f2d6acea0c8dc68-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.201 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:02.202 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:02.203 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:02.203 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.204 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.205 DEBUG:boto:Method: GET 2022-09-12T19:07:02.205 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:02.206 DEBUG:boto:Data: 2022-09-12T19:07:02.206 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.207 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.207 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.208 DEBUG:boto:Params: {} 2022-09-12T19:07:02.209 DEBUG:boto:Token: None 2022-09-12T19:07:02.209 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/ 2022-09-12T19:07:02.210 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA= 2022-09-12T19:07:02.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA=', 'Content-Length': '0'} 2022-09-12T19:07:02.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006de81c290b82d7d4-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.214 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:02.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:02.216 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.216 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.217 DEBUG:boto:path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:07:02.217 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:07:02.218 DEBUG:boto:Method: GET 2022-09-12T19:07:02.218 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:07:02.219 DEBUG:boto:Data: 2022-09-12T19:07:02.219 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.220 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.220 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.221 DEBUG:boto:Params: {} 2022-09-12T19:07:02.221 DEBUG:boto:Token: None 2022-09-12T19:07:02.222 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:07:02.223 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MjTf8Tn4cOUHV9/QEqhsnONm3gA= 2022-09-12T19:07:02.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MjTf8Tn4cOUHV9/QEqhsnONm3gA=', 'Content-Length': '0'} 2022-09-12T19:07:02.226 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'ztQRphsNbUdApzTcOcagUJ56uTwo7J2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b769bad41f336e2-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.227 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.227 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.228 DEBUG:boto:path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:07:02.228 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:07:02.229 DEBUG:boto:Method: GET 2022-09-12T19:07:02.229 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:07:02.230 DEBUG:boto:Data: 2022-09-12T19:07:02.230 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.231 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.231 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.232 DEBUG:boto:Params: {} 2022-09-12T19:07:02.232 DEBUG:boto:Token: None 2022-09-12T19:07:02.233 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:07:02.234 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MjTf8Tn4cOUHV9/QEqhsnONm3gA= 2022-09-12T19:07:02.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MjTf8Tn4cOUHV9/QEqhsnONm3gA=', 'Content-Length': '0'} 2022-09-12T19:07:02.236 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'ztQRphsNbUdApzTcOcagUJ56uTwo7J2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007fb3b9e96218053a-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.237 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.238 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.238 DEBUG:boto:path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:07:02.239 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:07:02.240 DEBUG:boto:Method: HEAD 2022-09-12T19:07:02.240 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:07:02.241 DEBUG:boto:Data: 2022-09-12T19:07:02.241 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.242 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.242 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.243 DEBUG:boto:Params: {} 2022-09-12T19:07:02.243 DEBUG:boto:Token: None 2022-09-12T19:07:02.244 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:07:02.244 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0D4S1EZX0MoHX9rdCL4rJUwLJzU= 2022-09-12T19:07:02.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0D4S1EZX0MoHX9rdCL4rJUwLJzU=', 'Content-Length': '0'} 2022-09-12T19:07:02.247 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'ztQRphsNbUdApzTcOcagUJ56uTwo7J2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d282ed3d594274b2-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.248 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.248 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.249 DEBUG:boto:path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:07:02.249 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:07:02.250 DEBUG:boto:Method: HEAD 2022-09-12T19:07:02.251 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:07:02.251 DEBUG:boto:Data: 2022-09-12T19:07:02.252 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.252 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.253 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.253 DEBUG:boto:Params: {} 2022-09-12T19:07:02.254 DEBUG:boto:Token: None 2022-09-12T19:07:02.254 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:07:02.255 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0D4S1EZX0MoHX9rdCL4rJUwLJzU= 2022-09-12T19:07:02.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0D4S1EZX0MoHX9rdCL4rJUwLJzU=', 'Content-Length': '0'} 2022-09-12T19:07:02.257 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'ztQRphsNbUdApzTcOcagUJ56uTwo7J2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c14ae9b0b5403546-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:02.259 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.259 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.260 DEBUG:boto:path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:07:02.260 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:07:02.261 DEBUG:boto:Method: GET 2022-09-12T19:07:02.261 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:07:02.262 DEBUG:boto:Data: 2022-09-12T19:07:02.262 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.263 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.264 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.264 DEBUG:boto:Params: {} 2022-09-12T19:07:02.265 DEBUG:boto:Token: None 2022-09-12T19:07:02.265 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:07:02.266 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MjTf8Tn4cOUHV9/QEqhsnONm3gA= 2022-09-12T19:07:02.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MjTf8Tn4cOUHV9/QEqhsnONm3gA=', 'Content-Length': '0'} 2022-09-12T19:07:02.269 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'ztQRphsNbUdApzTcOcagUJ56uTwo7J2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099d9987851ee25af-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.269 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.270 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.271 DEBUG:boto:path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:07:02.271 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:07:02.272 DEBUG:boto:Method: GET 2022-09-12T19:07:02.272 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:07:02.273 DEBUG:boto:Data: 2022-09-12T19:07:02.273 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.274 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.274 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.275 DEBUG:boto:Params: {} 2022-09-12T19:07:02.275 DEBUG:boto:Token: None 2022-09-12T19:07:02.276 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=ztQRphsNbUdApzTcOcagUJ56uTwo7J2 2022-09-12T19:07:02.277 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MjTf8Tn4cOUHV9/QEqhsnONm3gA= 2022-09-12T19:07:02.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MjTf8Tn4cOUHV9/QEqhsnONm3gA=', 'Content-Length': '0'} 2022-09-12T19:07:02.280 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'ztQRphsNbUdApzTcOcagUJ56uTwo7J2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6dff386d65df754-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.280 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:02.281 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:02.282 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.282 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.283 DEBUG:boto:Method: GET 2022-09-12T19:07:02.283 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:02.284 DEBUG:boto:Data: 2022-09-12T19:07:02.284 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.285 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.285 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.286 DEBUG:boto:Params: {} 2022-09-12T19:07:02.286 DEBUG:boto:Token: None 2022-09-12T19:07:02.287 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/ 2022-09-12T19:07:02.288 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA= 2022-09-12T19:07:02.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA=', 'Content-Length': '0'} 2022-09-12T19:07:02.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad45a40239112d7f-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.292 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:02.293 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:02.294 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:02.294 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.295 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.296 DEBUG:boto:Method: GET 2022-09-12T19:07:02.296 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:02.297 DEBUG:boto:Data: 2022-09-12T19:07:02.297 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.298 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.298 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.299 DEBUG:boto:Params: {} 2022-09-12T19:07:02.300 DEBUG:boto:Token: None 2022-09-12T19:07:02.300 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/ 2022-09-12T19:07:02.301 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA= 2022-09-12T19:07:02.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA=', 'Content-Length': '0'} 2022-09-12T19:07:02.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7a5424806b2b9f6-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.305 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:02.306 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.307 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.307 DEBUG:boto:Method: HEAD 2022-09-12T19:07:02.308 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:02.308 DEBUG:boto:Data: 2022-09-12T19:07:02.309 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.309 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.310 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.310 DEBUG:boto:Params: {} 2022-09-12T19:07:02.311 DEBUG:boto:Token: None 2022-09-12T19:07:02.312 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj 2022-09-12T19:07:02.313 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WeZyJospFU7xYVCz/MpW6ZDU3M4= 2022-09-12T19:07:02.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WeZyJospFU7xYVCz/MpW6ZDU3M4=', 'Content-Length': '0'} 2022-09-12T19:07:02.318 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1ea602ebcfb52ce-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.319 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.320 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.320 DEBUG:boto:Method: HEAD 2022-09-12T19:07:02.321 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:02.322 DEBUG:boto:Data: 2022-09-12T19:07:02.322 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.323 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.323 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.324 DEBUG:boto:Params: {} 2022-09-12T19:07:02.325 DEBUG:boto:Token: None 2022-09-12T19:07:02.325 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj 2022-09-12T19:07:02.326 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WeZyJospFU7xYVCz/MpW6ZDU3M4= 2022-09-12T19:07:02.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WeZyJospFU7xYVCz/MpW6ZDU3M4=', 'Content-Length': '0'} 2022-09-12T19:07:02.330 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000074c4daa87ccd2e39-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:02.331 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.332 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.332 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.333 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.334 DEBUG:boto:Method: GET 2022-09-12T19:07:02.334 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.335 DEBUG:boto:Data: 2022-09-12T19:07:02.335 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.336 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.336 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.337 DEBUG:boto:Params: {} 2022-09-12T19:07:02.337 DEBUG:boto:Token: None 2022-09-12T19:07:02.338 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.339 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HaivMvI9CZ6izJ4q+Pel5pjwrTo= 2022-09-12T19:07:02.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HaivMvI9CZ6izJ4q+Pel5pjwrTo=', 'Content-Length': '0'} 2022-09-12T19:07:02.341 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d485541128926a64-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.342 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.343 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.343 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.344 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.344 DEBUG:boto:Method: GET 2022-09-12T19:07:02.345 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.346 DEBUG:boto:Data: 2022-09-12T19:07:02.346 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.347 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.347 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.348 DEBUG:boto:Params: {} 2022-09-12T19:07:02.348 DEBUG:boto:Token: None 2022-09-12T19:07:02.349 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.349 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HaivMvI9CZ6izJ4q+Pel5pjwrTo= 2022-09-12T19:07:02.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HaivMvI9CZ6izJ4q+Pel5pjwrTo=', 'Content-Length': '0'} 2022-09-12T19:07:02.352 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000423a349fedc21f21-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.394 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:02.395 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:02.395 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.396 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.397 DEBUG:boto:Method: GET 2022-09-12T19:07:02.397 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:02.398 DEBUG:boto:Data: 2022-09-12T19:07:02.399 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.400 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.401 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.402 DEBUG:boto:Params: {} 2022-09-12T19:07:02.402 DEBUG:boto:Token: None 2022-09-12T19:07:02.403 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/ 2022-09-12T19:07:02.404 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA= 2022-09-12T19:07:02.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA=', 'Content-Length': '0'} 2022-09-12T19:07:02.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce122f46df766363-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.408 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:02.409 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:02.410 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:02.410 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.411 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.411 DEBUG:boto:Method: GET 2022-09-12T19:07:02.412 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:02.413 DEBUG:boto:Data: 2022-09-12T19:07:02.413 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.414 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.414 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.415 DEBUG:boto:Params: {} 2022-09-12T19:07:02.415 DEBUG:boto:Token: None 2022-09-12T19:07:02.416 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/ 2022-09-12T19:07:02.416 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA= 2022-09-12T19:07:02.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA=', 'Content-Length': '0'} 2022-09-12T19:07:02.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094fcd7f7739c7868-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.421 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:02.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:02.422 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.422 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.423 DEBUG:boto:path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:07:02.424 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:07:02.424 DEBUG:boto:Method: GET 2022-09-12T19:07:02.425 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:07:02.425 DEBUG:boto:Data: 2022-09-12T19:07:02.426 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.426 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.427 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.427 DEBUG:boto:Params: {} 2022-09-12T19:07:02.428 DEBUG:boto:Token: None 2022-09-12T19:07:02.429 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:07:02.429 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8qlCMpnME/UkqKiIY/DUXuQwx0g= 2022-09-12T19:07:02.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8qlCMpnME/UkqKiIY/DUXuQwx0g=', 'Content-Length': '0'} 2022-09-12T19:07:02.432 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Katr07t9mQV0b0gx44aB8d9VyeAhMS1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c41e24319d3f2476-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.474 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.475 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.475 DEBUG:boto:path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:07:02.476 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:07:02.476 DEBUG:boto:Method: GET 2022-09-12T19:07:02.477 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:07:02.478 DEBUG:boto:Data: 2022-09-12T19:07:02.478 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.479 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.479 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.480 DEBUG:boto:Params: {} 2022-09-12T19:07:02.480 DEBUG:boto:Token: None 2022-09-12T19:07:02.481 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:07:02.482 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8qlCMpnME/UkqKiIY/DUXuQwx0g= 2022-09-12T19:07:02.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8qlCMpnME/UkqKiIY/DUXuQwx0g=', 'Content-Length': '0'} 2022-09-12T19:07:02.485 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Katr07t9mQV0b0gx44aB8d9VyeAhMS1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8cb1bd939f8d20d-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.486 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.486 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.487 DEBUG:boto:path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:07:02.487 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:07:02.488 DEBUG:boto:Method: HEAD 2022-09-12T19:07:02.489 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:07:02.489 DEBUG:boto:Data: 2022-09-12T19:07:02.490 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.490 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.491 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.491 DEBUG:boto:Params: {} 2022-09-12T19:07:02.492 DEBUG:boto:Token: None 2022-09-12T19:07:02.492 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:07:02.493 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KJPQzFPhL9Z9jyJtDiKgkPGFfY4= 2022-09-12T19:07:02.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KJPQzFPhL9Z9jyJtDiKgkPGFfY4=', 'Content-Length': '0'} 2022-09-12T19:07:02.496 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Katr07t9mQV0b0gx44aB8d9VyeAhMS1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055849a7c13637a28-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.496 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.497 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.497 DEBUG:boto:path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:07:02.498 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:07:02.499 DEBUG:boto:Method: HEAD 2022-09-12T19:07:02.499 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:07:02.500 DEBUG:boto:Data: 2022-09-12T19:07:02.500 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.501 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.501 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.502 DEBUG:boto:Params: {} 2022-09-12T19:07:02.503 DEBUG:boto:Token: None 2022-09-12T19:07:02.503 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:07:02.504 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KJPQzFPhL9Z9jyJtDiKgkPGFfY4= 2022-09-12T19:07:02.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KJPQzFPhL9Z9jyJtDiKgkPGFfY4=', 'Content-Length': '0'} 2022-09-12T19:07:02.507 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Katr07t9mQV0b0gx44aB8d9VyeAhMS1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd83a99847db313a-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:02.508 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.509 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.510 DEBUG:boto:path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:07:02.511 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:07:02.512 DEBUG:boto:Method: GET 2022-09-12T19:07:02.513 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:07:02.514 DEBUG:boto:Data: 2022-09-12T19:07:02.515 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.516 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.517 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.518 DEBUG:boto:Params: {} 2022-09-12T19:07:02.519 DEBUG:boto:Token: None 2022-09-12T19:07:02.520 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:07:02.521 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8qlCMpnME/UkqKiIY/DUXuQwx0g= 2022-09-12T19:07:02.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8qlCMpnME/UkqKiIY/DUXuQwx0g=', 'Content-Length': '0'} 2022-09-12T19:07:02.525 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Katr07t9mQV0b0gx44aB8d9VyeAhMS1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003bb8ba064a97d9ef-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.527 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.528 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.528 DEBUG:boto:path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:07:02.529 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:07:02.530 DEBUG:boto:Method: GET 2022-09-12T19:07:02.531 DEBUG:boto:Path: /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:07:02.532 DEBUG:boto:Data: 2022-09-12T19:07:02.533 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.534 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.535 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.536 DEBUG:boto:Params: {} 2022-09-12T19:07:02.538 DEBUG:boto:Token: None 2022-09-12T19:07:02.539 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=Katr07t9mQV0b0gx44aB8d9VyeAhMS1 2022-09-12T19:07:02.539 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8qlCMpnME/UkqKiIY/DUXuQwx0g= 2022-09-12T19:07:02.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8qlCMpnME/UkqKiIY/DUXuQwx0g=', 'Content-Length': '0'} 2022-09-12T19:07:02.542 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'Katr07t9mQV0b0gx44aB8d9VyeAhMS1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c31e2a85567db212-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.543 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:02.543 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:02.544 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.545 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.545 DEBUG:boto:Method: GET 2022-09-12T19:07:02.546 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:02.546 DEBUG:boto:Data: 2022-09-12T19:07:02.547 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.547 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.548 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.548 DEBUG:boto:Params: {} 2022-09-12T19:07:02.549 DEBUG:boto:Token: None 2022-09-12T19:07:02.549 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/ 2022-09-12T19:07:02.550 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA= 2022-09-12T19:07:02.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA=', 'Content-Length': '0'} 2022-09-12T19:07:02.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003101d05118205ffa-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.555 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:02.557 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:02.558 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:02.559 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.560 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.560 DEBUG:boto:Method: GET 2022-09-12T19:07:02.561 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:02.562 DEBUG:boto:Data: 2022-09-12T19:07:02.563 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.563 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.564 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.564 DEBUG:boto:Params: {} 2022-09-12T19:07:02.565 DEBUG:boto:Token: None 2022-09-12T19:07:02.566 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/ 2022-09-12T19:07:02.566 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA= 2022-09-12T19:07:02.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA=', 'Content-Length': '0'} 2022-09-12T19:07:02.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d6c703bfec7852c-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.614 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:02.615 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.616 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.616 DEBUG:boto:Method: HEAD 2022-09-12T19:07:02.617 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:02.617 DEBUG:boto:Data: 2022-09-12T19:07:02.618 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.618 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.619 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.619 DEBUG:boto:Params: {} 2022-09-12T19:07:02.620 DEBUG:boto:Token: None 2022-09-12T19:07:02.621 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj 2022-09-12T19:07:02.621 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WeZyJospFU7xYVCz/MpW6ZDU3M4= 2022-09-12T19:07:02.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WeZyJospFU7xYVCz/MpW6ZDU3M4=', 'Content-Length': '0'} 2022-09-12T19:07:02.625 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090e129f799922831-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.626 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.627 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.627 DEBUG:boto:Method: HEAD 2022-09-12T19:07:02.628 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:02.628 DEBUG:boto:Data: 2022-09-12T19:07:02.629 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.629 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.630 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.630 DEBUG:boto:Params: {} 2022-09-12T19:07:02.631 DEBUG:boto:Token: None 2022-09-12T19:07:02.632 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj 2022-09-12T19:07:02.632 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WeZyJospFU7xYVCz/MpW6ZDU3M4= 2022-09-12T19:07:02.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WeZyJospFU7xYVCz/MpW6ZDU3M4=', 'Content-Length': '0'} 2022-09-12T19:07:02.636 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002518eb28798a5c8e-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:02.638 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.638 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.639 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.640 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.640 DEBUG:boto:Method: GET 2022-09-12T19:07:02.641 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.641 DEBUG:boto:Data: 2022-09-12T19:07:02.642 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.642 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.643 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.644 DEBUG:boto:Params: {} 2022-09-12T19:07:02.644 DEBUG:boto:Token: None 2022-09-12T19:07:02.645 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.646 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HaivMvI9CZ6izJ4q+Pel5pjwrTo= 2022-09-12T19:07:02.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HaivMvI9CZ6izJ4q+Pel5pjwrTo=', 'Content-Length': '0'} 2022-09-12T19:07:02.648 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048298ab31dfd015b-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.649 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.650 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.650 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.651 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.651 DEBUG:boto:Method: GET 2022-09-12T19:07:02.652 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.653 DEBUG:boto:Data: 2022-09-12T19:07:02.653 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.654 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.655 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.655 DEBUG:boto:Params: {} 2022-09-12T19:07:02.656 DEBUG:boto:Token: None 2022-09-12T19:07:02.657 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.658 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HaivMvI9CZ6izJ4q+Pel5pjwrTo= 2022-09-12T19:07:02.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HaivMvI9CZ6izJ4q+Pel5pjwrTo=', 'Content-Length': '0'} 2022-09-12T19:07:02.661 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b587cdda779b180d-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.661 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:02.662 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:02.662 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.663 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.664 DEBUG:boto:Method: GET 2022-09-12T19:07:02.664 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:02.665 DEBUG:boto:Data: 2022-09-12T19:07:02.665 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.666 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.667 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.667 DEBUG:boto:Params: {} 2022-09-12T19:07:02.668 DEBUG:boto:Token: None 2022-09-12T19:07:02.668 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/ 2022-09-12T19:07:02.669 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA= 2022-09-12T19:07:02.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA=', 'Content-Length': '0'} 2022-09-12T19:07:02.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008beacb3417dcb417-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.673 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:02.674 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:02.675 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:02.675 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.676 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.676 DEBUG:boto:Method: GET 2022-09-12T19:07:02.677 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:02.677 DEBUG:boto:Data: 2022-09-12T19:07:02.678 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.679 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.680 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.681 DEBUG:boto:Params: {} 2022-09-12T19:07:02.682 DEBUG:boto:Token: None 2022-09-12T19:07:02.683 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/ 2022-09-12T19:07:02.684 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA= 2022-09-12T19:07:02.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA=', 'Content-Length': '0'} 2022-09-12T19:07:02.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036916253ab58238d-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.689 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:02.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:02.691 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.691 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.692 DEBUG:boto:path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:07:02.692 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:07:02.693 DEBUG:boto:Method: GET 2022-09-12T19:07:02.694 DEBUG:boto:Path: /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:07:02.694 DEBUG:boto:Data: 2022-09-12T19:07:02.695 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.695 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.696 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.696 DEBUG:boto:Params: {} 2022-09-12T19:07:02.697 DEBUG:boto:Token: None 2022-09-12T19:07:02.698 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:07:02.698 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MeknfwYQSqzDihUUKqHGmP3G2Xg= 2022-09-12T19:07:02.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MeknfwYQSqzDihUUKqHGmP3G2Xg=', 'Content-Length': '0'} 2022-09-12T19:07:02.701 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'UBsYcCA94H55K4ucww5vXNAxuAMKNSG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f720b54f2c52e11-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.702 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.702 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.703 DEBUG:boto:path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:07:02.704 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:07:02.705 DEBUG:boto:Method: GET 2022-09-12T19:07:02.705 DEBUG:boto:Path: /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:07:02.706 DEBUG:boto:Data: 2022-09-12T19:07:02.707 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.708 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.709 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.710 DEBUG:boto:Params: {} 2022-09-12T19:07:02.711 DEBUG:boto:Token: None 2022-09-12T19:07:02.712 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:07:02.713 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MeknfwYQSqzDihUUKqHGmP3G2Xg= 2022-09-12T19:07:02.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MeknfwYQSqzDihUUKqHGmP3G2Xg=', 'Content-Length': '0'} 2022-09-12T19:07:02.716 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'UBsYcCA94H55K4ucww5vXNAxuAMKNSG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007fb282c6c2876dc0-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.718 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.718 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.719 DEBUG:boto:path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:07:02.719 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:07:02.720 DEBUG:boto:Method: HEAD 2022-09-12T19:07:02.720 DEBUG:boto:Path: /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:07:02.721 DEBUG:boto:Data: 2022-09-12T19:07:02.721 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.722 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.722 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.723 DEBUG:boto:Params: {} 2022-09-12T19:07:02.723 DEBUG:boto:Token: None 2022-09-12T19:07:02.724 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:07:02.724 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BBcMw9mcGMc8uuLukigAwERhJqU= 2022-09-12T19:07:02.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BBcMw9mcGMc8uuLukigAwERhJqU=', 'Content-Length': '0'} 2022-09-12T19:07:02.727 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'UBsYcCA94H55K4ucww5vXNAxuAMKNSG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000039df8c219f54a67d-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.728 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.728 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.729 DEBUG:boto:path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:07:02.729 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:07:02.730 DEBUG:boto:Method: HEAD 2022-09-12T19:07:02.730 DEBUG:boto:Path: /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:07:02.731 DEBUG:boto:Data: 2022-09-12T19:07:02.732 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.732 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.733 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.733 DEBUG:boto:Params: {} 2022-09-12T19:07:02.734 DEBUG:boto:Token: None 2022-09-12T19:07:02.735 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:07:02.735 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BBcMw9mcGMc8uuLukigAwERhJqU= 2022-09-12T19:07:02.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BBcMw9mcGMc8uuLukigAwERhJqU=', 'Content-Length': '0'} 2022-09-12T19:07:02.738 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'UBsYcCA94H55K4ucww5vXNAxuAMKNSG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057930fc991980d2b-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:02.740 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.740 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.741 DEBUG:boto:path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:07:02.741 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:07:02.742 DEBUG:boto:Method: GET 2022-09-12T19:07:02.742 DEBUG:boto:Path: /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:07:02.743 DEBUG:boto:Data: 2022-09-12T19:07:02.743 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.744 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.744 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.745 DEBUG:boto:Params: {} 2022-09-12T19:07:02.746 DEBUG:boto:Token: None 2022-09-12T19:07:02.746 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:07:02.747 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MeknfwYQSqzDihUUKqHGmP3G2Xg= 2022-09-12T19:07:02.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MeknfwYQSqzDihUUKqHGmP3G2Xg=', 'Content-Length': '0'} 2022-09-12T19:07:02.749 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'UBsYcCA94H55K4ucww5vXNAxuAMKNSG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0fd69b0d658e2e6-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.750 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.751 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.751 DEBUG:boto:path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:07:02.752 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:07:02.752 DEBUG:boto:Method: GET 2022-09-12T19:07:02.753 DEBUG:boto:Path: /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:07:02.753 DEBUG:boto:Data: 2022-09-12T19:07:02.754 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.755 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.755 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.756 DEBUG:boto:Params: {} 2022-09-12T19:07:02.756 DEBUG:boto:Token: None 2022-09-12T19:07:02.757 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=UBsYcCA94H55K4ucww5vXNAxuAMKNSG 2022-09-12T19:07:02.757 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MeknfwYQSqzDihUUKqHGmP3G2Xg= 2022-09-12T19:07:02.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MeknfwYQSqzDihUUKqHGmP3G2Xg=', 'Content-Length': '0'} 2022-09-12T19:07:02.760 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'UBsYcCA94H55K4ucww5vXNAxuAMKNSG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016d562744b106c3f-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.802 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:02.803 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:02.803 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.804 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.804 DEBUG:boto:Method: GET 2022-09-12T19:07:02.805 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:02.806 DEBUG:boto:Data: 2022-09-12T19:07:02.806 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.807 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.807 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.808 DEBUG:boto:Params: {} 2022-09-12T19:07:02.808 DEBUG:boto:Token: None 2022-09-12T19:07:02.809 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/ 2022-09-12T19:07:02.810 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA= 2022-09-12T19:07:02.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA=', 'Content-Length': '0'} 2022-09-12T19:07:02.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7baa4827e26150d-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.814 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:02.815 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:02.816 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:02.817 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.817 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.818 DEBUG:boto:Method: GET 2022-09-12T19:07:02.818 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:02.819 DEBUG:boto:Data: 2022-09-12T19:07:02.820 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.820 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.821 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.822 DEBUG:boto:Params: {} 2022-09-12T19:07:02.823 DEBUG:boto:Token: None 2022-09-12T19:07:02.823 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/ 2022-09-12T19:07:02.825 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA= 2022-09-12T19:07:02.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA=', 'Content-Length': '0'} 2022-09-12T19:07:02.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb3eb26db571b1a5-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.829 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:02.830 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.831 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.831 DEBUG:boto:Method: HEAD 2022-09-12T19:07:02.832 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:02.832 DEBUG:boto:Data: 2022-09-12T19:07:02.833 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.833 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.834 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.834 DEBUG:boto:Params: {} 2022-09-12T19:07:02.835 DEBUG:boto:Token: None 2022-09-12T19:07:02.835 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj 2022-09-12T19:07:02.836 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WeZyJospFU7xYVCz/MpW6ZDU3M4= 2022-09-12T19:07:02.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WeZyJospFU7xYVCz/MpW6ZDU3M4=', 'Content-Length': '0'} 2022-09-12T19:07:02.840 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d73281f38d1c981c-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.841 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.842 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.842 DEBUG:boto:Method: HEAD 2022-09-12T19:07:02.843 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:02.843 DEBUG:boto:Data: 2022-09-12T19:07:02.844 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.844 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.845 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.845 DEBUG:boto:Params: {} 2022-09-12T19:07:02.846 DEBUG:boto:Token: None 2022-09-12T19:07:02.847 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj 2022-09-12T19:07:02.847 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WeZyJospFU7xYVCz/MpW6ZDU3M4= 2022-09-12T19:07:02.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WeZyJospFU7xYVCz/MpW6ZDU3M4=', 'Content-Length': '0'} 2022-09-12T19:07:02.851 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000feeb9a944038840b-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:02.853 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.853 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.854 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.854 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.855 DEBUG:boto:Method: GET 2022-09-12T19:07:02.855 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.856 DEBUG:boto:Data: 2022-09-12T19:07:02.856 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.857 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.857 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.858 DEBUG:boto:Params: {} 2022-09-12T19:07:02.859 DEBUG:boto:Token: None 2022-09-12T19:07:02.859 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.860 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HaivMvI9CZ6izJ4q+Pel5pjwrTo= 2022-09-12T19:07:02.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HaivMvI9CZ6izJ4q+Pel5pjwrTo=', 'Content-Length': '0'} 2022-09-12T19:07:02.863 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000924317939c77acb4-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.906 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.907 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.907 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.908 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.908 DEBUG:boto:Method: GET 2022-09-12T19:07:02.909 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.910 DEBUG:boto:Data: 2022-09-12T19:07:02.910 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.911 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.911 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.912 DEBUG:boto:Params: {} 2022-09-12T19:07:02.912 DEBUG:boto:Token: None 2022-09-12T19:07:02.913 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:02.914 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HaivMvI9CZ6izJ4q+Pel5pjwrTo= 2022-09-12T19:07:02.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HaivMvI9CZ6izJ4q+Pel5pjwrTo=', 'Content-Length': '0'} 2022-09-12T19:07:02.917 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b886c86fcf8a0ad7-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.918 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:02.918 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:02.919 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.919 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.920 DEBUG:boto:Method: GET 2022-09-12T19:07:02.920 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:02.921 DEBUG:boto:Data: 2022-09-12T19:07:02.921 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.922 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.923 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.923 DEBUG:boto:Params: {} 2022-09-12T19:07:02.924 DEBUG:boto:Token: None 2022-09-12T19:07:02.924 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/ 2022-09-12T19:07:02.925 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA= 2022-09-12T19:07:02.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA=', 'Content-Length': '0'} 2022-09-12T19:07:02.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7f72aa3c8475b5c-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.929 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:02.930 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:02.931 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:02.931 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.932 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:02.932 DEBUG:boto:Method: GET 2022-09-12T19:07:02.933 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:02.933 DEBUG:boto:Data: 2022-09-12T19:07:02.934 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.934 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.935 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.935 DEBUG:boto:Params: {} 2022-09-12T19:07:02.936 DEBUG:boto:Token: None 2022-09-12T19:07:02.936 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/ 2022-09-12T19:07:02.937 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA= 2022-09-12T19:07:02.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IpbiegDoPu9FkfbYlQqSiBN8fgA=', 'Content-Length': '0'} 2022-09-12T19:07:02.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da186d4a8b00273b-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.941 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:02.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:02.942 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.943 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.943 DEBUG:boto:path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:07:02.944 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:07:02.944 DEBUG:boto:Method: GET 2022-09-12T19:07:02.945 DEBUG:boto:Path: /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:07:02.945 DEBUG:boto:Data: 2022-09-12T19:07:02.946 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.946 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.947 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.947 DEBUG:boto:Params: {} 2022-09-12T19:07:02.948 DEBUG:boto:Token: None 2022-09-12T19:07:02.948 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:07:02.949 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BESud2UWw0q/K+aKK6F6K8SjJiU= 2022-09-12T19:07:02.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BESud2UWw0q/K+aKK6F6K8SjJiU=', 'Content-Length': '0'} 2022-09-12T19:07:02.952 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ecce50b606fcb20-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.952 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.953 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.954 DEBUG:boto:path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:07:02.954 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:07:02.955 DEBUG:boto:Method: GET 2022-09-12T19:07:02.955 DEBUG:boto:Path: /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:07:02.956 DEBUG:boto:Data: 2022-09-12T19:07:02.956 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.957 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.957 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.958 DEBUG:boto:Params: {} 2022-09-12T19:07:02.959 DEBUG:boto:Token: None 2022-09-12T19:07:02.959 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:07:02.960 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BESud2UWw0q/K+aKK6F6K8SjJiU= 2022-09-12T19:07:02.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BESud2UWw0q/K+aKK6F6K8SjJiU=', 'Content-Length': '0'} 2022-09-12T19:07:02.963 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d53cc30dbb77f7d-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.964 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.964 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.965 DEBUG:boto:path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:07:02.965 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:07:02.966 DEBUG:boto:Method: HEAD 2022-09-12T19:07:02.966 DEBUG:boto:Path: /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:07:02.967 DEBUG:boto:Data: 2022-09-12T19:07:02.967 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.968 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.968 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.969 DEBUG:boto:Params: {} 2022-09-12T19:07:02.969 DEBUG:boto:Token: None 2022-09-12T19:07:02.970 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:07:02.970 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/es97rRaJsUY2nh5dXE+gLMK4hE= 2022-09-12T19:07:02.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/es97rRaJsUY2nh5dXE+gLMK4hE=', 'Content-Length': '0'} 2022-09-12T19:07:02.973 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9ffacb4b94ba95c-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.974 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.974 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.975 DEBUG:boto:path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:07:02.975 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:07:02.976 DEBUG:boto:Method: HEAD 2022-09-12T19:07:02.976 DEBUG:boto:Path: /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:07:02.977 DEBUG:boto:Data: 2022-09-12T19:07:02.977 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.978 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:02.978 DEBUG:boto:Port: 8000 2022-09-12T19:07:02.979 DEBUG:boto:Params: {} 2022-09-12T19:07:02.979 DEBUG:boto:Token: None 2022-09-12T19:07:02.980 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:07:02.980 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/es97rRaJsUY2nh5dXE+gLMK4hE= 2022-09-12T19:07:02.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/es97rRaJsUY2nh5dXE+gLMK4hE=', 'Content-Length': '0'} 2022-09-12T19:07:02.985 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e519cb446c86ce5-00631f8356-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:02.986 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.987 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.987 DEBUG:boto:path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:07:02.988 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:07:02.988 DEBUG:boto:Method: GET 2022-09-12T19:07:02.989 DEBUG:boto:Path: /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:07:02.990 DEBUG:boto:Data: 2022-09-12T19:07:02.990 DEBUG:boto:Headers: {} 2022-09-12T19:07:02.991 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:02.991 DEBUG:boto:Port: 8001 2022-09-12T19:07:02.992 DEBUG:boto:Params: {} 2022-09-12T19:07:02.992 DEBUG:boto:Token: None 2022-09-12T19:07:02.993 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:02 GMT /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:07:02.993 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BESud2UWw0q/K+aKK6F6K8SjJiU= 2022-09-12T19:07:02.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BESud2UWw0q/K+aKK6F6K8SjJiU=', 'Content-Length': '0'} 2022-09-12T19:07:02.996 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002310e65648c13e1f-00631f8356-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:02.997 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:02.998 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:02.998 DEBUG:boto:path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:07:02.999 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:07:02.999 DEBUG:boto:Method: GET 2022-09-12T19:07:03.000 DEBUG:boto:Path: /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:07:03.000 DEBUG:boto:Data: 2022-09-12T19:07:03.001 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.001 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.002 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.003 DEBUG:boto:Params: {} 2022-09-12T19:07:03.003 DEBUG:boto:Token: None 2022-09-12T19:07:03.004 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW 2022-09-12T19:07:03.004 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IZgSn9dqyhxKUV/yE8p6uVL08cw= 2022-09-12T19:07:03.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IZgSn9dqyhxKUV/yE8p6uVL08cw=', 'Content-Length': '0'} 2022-09-12T19:07:03.007 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'kxSkXmfXKy0sdqagFrYSQNi7tMOKBoW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa8087438e616017-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.050 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:03.051 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:03.051 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.052 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.052 DEBUG:boto:Method: GET 2022-09-12T19:07:03.053 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:03.053 DEBUG:boto:Data: 2022-09-12T19:07:03.054 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.055 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.055 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.056 DEBUG:boto:Params: {} 2022-09-12T19:07:03.056 DEBUG:boto:Token: None 2022-09-12T19:07:03.057 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/ 2022-09-12T19:07:03.058 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY= 2022-09-12T19:07:03.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY=', 'Content-Length': '0'} 2022-09-12T19:07:03.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d222e04b96dee05b-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.062 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:03.063 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:03.063 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:03.064 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.064 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.065 DEBUG:boto:Method: GET 2022-09-12T19:07:03.066 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:03.067 DEBUG:boto:Data: 2022-09-12T19:07:03.067 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.068 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.069 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.069 DEBUG:boto:Params: {} 2022-09-12T19:07:03.070 DEBUG:boto:Token: None 2022-09-12T19:07:03.071 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/ 2022-09-12T19:07:03.072 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY= 2022-09-12T19:07:03.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY=', 'Content-Length': '0'} 2022-09-12T19:07:03.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc86d0047b6e43f6-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.077 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:03.078 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.079 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.080 DEBUG:boto:Method: HEAD 2022-09-12T19:07:03.080 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:03.081 DEBUG:boto:Data: 2022-09-12T19:07:03.081 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.082 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.082 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.083 DEBUG:boto:Params: {} 2022-09-12T19:07:03.083 DEBUG:boto:Token: None 2022-09-12T19:07:03.084 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj 2022-09-12T19:07:03.084 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fzHvj7kseHGogy+PwK5FBfL8He8= 2022-09-12T19:07:03.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fzHvj7kseHGogy+PwK5FBfL8He8=', 'Content-Length': '0'} 2022-09-12T19:07:03.088 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4509596fbdadc81-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.089 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.090 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.090 DEBUG:boto:Method: HEAD 2022-09-12T19:07:03.091 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:03.091 DEBUG:boto:Data: 2022-09-12T19:07:03.092 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.092 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.093 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.093 DEBUG:boto:Params: {} 2022-09-12T19:07:03.094 DEBUG:boto:Token: None 2022-09-12T19:07:03.095 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj 2022-09-12T19:07:03.095 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fzHvj7kseHGogy+PwK5FBfL8He8= 2022-09-12T19:07:03.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fzHvj7kseHGogy+PwK5FBfL8He8=', 'Content-Length': '0'} 2022-09-12T19:07:03.100 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005113088d2ce1f983-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:03.101 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.101 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.102 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.103 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.103 DEBUG:boto:Method: GET 2022-09-12T19:07:03.104 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.104 DEBUG:boto:Data: 2022-09-12T19:07:03.105 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.105 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.106 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.107 DEBUG:boto:Params: {} 2022-09-12T19:07:03.107 DEBUG:boto:Token: None 2022-09-12T19:07:03.108 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.108 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1kwxQeng7WSb/gC/creMgfsnrWk= 2022-09-12T19:07:03.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1kwxQeng7WSb/gC/creMgfsnrWk=', 'Content-Length': '0'} 2022-09-12T19:07:03.111 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000419a590bf22fe9b5-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.112 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.113 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.113 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.114 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.114 DEBUG:boto:Method: GET 2022-09-12T19:07:03.115 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.115 DEBUG:boto:Data: 2022-09-12T19:07:03.116 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.116 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.117 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.117 DEBUG:boto:Params: {} 2022-09-12T19:07:03.118 DEBUG:boto:Token: None 2022-09-12T19:07:03.119 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.119 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1kwxQeng7WSb/gC/creMgfsnrWk= 2022-09-12T19:07:03.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1kwxQeng7WSb/gC/creMgfsnrWk=', 'Content-Length': '0'} 2022-09-12T19:07:03.122 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d0f7daa5bc8b8cd2-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.123 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:03.123 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:03.124 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.124 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.125 DEBUG:boto:Method: GET 2022-09-12T19:07:03.125 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:03.126 DEBUG:boto:Data: 2022-09-12T19:07:03.126 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.127 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.127 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.128 DEBUG:boto:Params: {} 2022-09-12T19:07:03.129 DEBUG:boto:Token: None 2022-09-12T19:07:03.129 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/ 2022-09-12T19:07:03.130 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY= 2022-09-12T19:07:03.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY=', 'Content-Length': '0'} 2022-09-12T19:07:03.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fb7975c9d87c0a9-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.134 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:03.135 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:03.135 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:03.136 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.136 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.137 DEBUG:boto:Method: GET 2022-09-12T19:07:03.137 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:03.138 DEBUG:boto:Data: 2022-09-12T19:07:03.138 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.139 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.139 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.140 DEBUG:boto:Params: {} 2022-09-12T19:07:03.141 DEBUG:boto:Token: None 2022-09-12T19:07:03.141 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/ 2022-09-12T19:07:03.142 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY= 2022-09-12T19:07:03.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY=', 'Content-Length': '0'} 2022-09-12T19:07:03.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000941674484c0119e6-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.186 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:03.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:03.189 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.190 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.191 DEBUG:boto:path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:07:03.192 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:07:03.193 DEBUG:boto:Method: GET 2022-09-12T19:07:03.194 DEBUG:boto:Path: /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:07:03.195 DEBUG:boto:Data: 2022-09-12T19:07:03.196 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.196 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.197 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.198 DEBUG:boto:Params: {} 2022-09-12T19:07:03.199 DEBUG:boto:Token: None 2022-09-12T19:07:03.200 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:07:03.200 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MCrw7CA8Wj2mL5A4GNnSVM0uJBs= 2022-09-12T19:07:03.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MCrw7CA8Wj2mL5A4GNnSVM0uJBs=', 'Content-Length': '0'} 2022-09-12T19:07:03.203 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', '.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000efb9eaba6edf1171-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.204 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.205 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.205 DEBUG:boto:path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:07:03.206 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:07:03.206 DEBUG:boto:Method: GET 2022-09-12T19:07:03.207 DEBUG:boto:Path: /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:07:03.207 DEBUG:boto:Data: 2022-09-12T19:07:03.208 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.208 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.209 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.209 DEBUG:boto:Params: {} 2022-09-12T19:07:03.210 DEBUG:boto:Token: None 2022-09-12T19:07:03.211 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:07:03.211 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MCrw7CA8Wj2mL5A4GNnSVM0uJBs= 2022-09-12T19:07:03.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MCrw7CA8Wj2mL5A4GNnSVM0uJBs=', 'Content-Length': '0'} 2022-09-12T19:07:03.214 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', '.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000115d8cb9e40e2f3a-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.215 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.216 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.216 DEBUG:boto:path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:07:03.217 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:07:03.218 DEBUG:boto:Method: HEAD 2022-09-12T19:07:03.218 DEBUG:boto:Path: /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:07:03.219 DEBUG:boto:Data: 2022-09-12T19:07:03.220 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.220 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.221 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.221 DEBUG:boto:Params: {} 2022-09-12T19:07:03.222 DEBUG:boto:Token: None 2022-09-12T19:07:03.223 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:07:03.224 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f8hLC+tBUfjnmPeOIQHZ7d9+sWY= 2022-09-12T19:07:03.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f8hLC+tBUfjnmPeOIQHZ7d9+sWY=', 'Content-Length': '0'} 2022-09-12T19:07:03.228 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', '.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2801f1208f1e513-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.229 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.230 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.231 DEBUG:boto:path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:07:03.231 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:07:03.232 DEBUG:boto:Method: HEAD 2022-09-12T19:07:03.232 DEBUG:boto:Path: /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:07:03.233 DEBUG:boto:Data: 2022-09-12T19:07:03.233 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.234 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.234 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.235 DEBUG:boto:Params: {} 2022-09-12T19:07:03.236 DEBUG:boto:Token: None 2022-09-12T19:07:03.237 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:07:03.238 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f8hLC+tBUfjnmPeOIQHZ7d9+sWY= 2022-09-12T19:07:03.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f8hLC+tBUfjnmPeOIQHZ7d9+sWY=', 'Content-Length': '0'} 2022-09-12T19:07:03.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', '.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009954a2bd21c06c19-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:03.244 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.245 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.245 DEBUG:boto:path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:07:03.246 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:07:03.247 DEBUG:boto:Method: GET 2022-09-12T19:07:03.248 DEBUG:boto:Path: /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:07:03.249 DEBUG:boto:Data: 2022-09-12T19:07:03.250 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.250 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.251 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.252 DEBUG:boto:Params: {} 2022-09-12T19:07:03.252 DEBUG:boto:Token: None 2022-09-12T19:07:03.253 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:07:03.255 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MCrw7CA8Wj2mL5A4GNnSVM0uJBs= 2022-09-12T19:07:03.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MCrw7CA8Wj2mL5A4GNnSVM0uJBs=', 'Content-Length': '0'} 2022-09-12T19:07:03.259 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', '.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d982edacdf9f916-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.306 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.307 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.307 DEBUG:boto:path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:07:03.308 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:07:03.309 DEBUG:boto:Method: GET 2022-09-12T19:07:03.309 DEBUG:boto:Path: /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:07:03.310 DEBUG:boto:Data: 2022-09-12T19:07:03.311 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.312 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.313 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.314 DEBUG:boto:Params: {} 2022-09-12T19:07:03.315 DEBUG:boto:Token: None 2022-09-12T19:07:03.316 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0 2022-09-12T19:07:03.317 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MCrw7CA8Wj2mL5A4GNnSVM0uJBs= 2022-09-12T19:07:03.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MCrw7CA8Wj2mL5A4GNnSVM0uJBs=', 'Content-Length': '0'} 2022-09-12T19:07:03.321 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', '.oHjYZ5KXG.R75pyoS8SIRBAqD3iIp0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002818cb9247573aa0-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.322 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:03.324 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:03.325 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.325 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.326 DEBUG:boto:Method: GET 2022-09-12T19:07:03.327 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:03.328 DEBUG:boto:Data: 2022-09-12T19:07:03.329 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.330 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.331 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.331 DEBUG:boto:Params: {} 2022-09-12T19:07:03.332 DEBUG:boto:Token: None 2022-09-12T19:07:03.333 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/ 2022-09-12T19:07:03.333 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY= 2022-09-12T19:07:03.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY=', 'Content-Length': '0'} 2022-09-12T19:07:03.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017877fe3a39cfdb1-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.337 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:03.338 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:03.339 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:03.339 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.340 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.340 DEBUG:boto:Method: GET 2022-09-12T19:07:03.341 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:03.341 DEBUG:boto:Data: 2022-09-12T19:07:03.342 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.342 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.343 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.343 DEBUG:boto:Params: {} 2022-09-12T19:07:03.344 DEBUG:boto:Token: None 2022-09-12T19:07:03.345 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/ 2022-09-12T19:07:03.345 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY= 2022-09-12T19:07:03.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY=', 'Content-Length': '0'} 2022-09-12T19:07:03.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004417aef3cd607953-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.349 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:03.350 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.351 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.351 DEBUG:boto:Method: HEAD 2022-09-12T19:07:03.352 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:03.352 DEBUG:boto:Data: 2022-09-12T19:07:03.353 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.353 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.354 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.354 DEBUG:boto:Params: {} 2022-09-12T19:07:03.355 DEBUG:boto:Token: None 2022-09-12T19:07:03.356 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj 2022-09-12T19:07:03.356 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fzHvj7kseHGogy+PwK5FBfL8He8= 2022-09-12T19:07:03.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fzHvj7kseHGogy+PwK5FBfL8He8=', 'Content-Length': '0'} 2022-09-12T19:07:03.361 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003fb5d80c7820e60-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.361 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.362 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.363 DEBUG:boto:Method: HEAD 2022-09-12T19:07:03.363 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:03.364 DEBUG:boto:Data: 2022-09-12T19:07:03.364 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.365 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.366 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.366 DEBUG:boto:Params: {} 2022-09-12T19:07:03.367 DEBUG:boto:Token: None 2022-09-12T19:07:03.367 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj 2022-09-12T19:07:03.368 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fzHvj7kseHGogy+PwK5FBfL8He8= 2022-09-12T19:07:03.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fzHvj7kseHGogy+PwK5FBfL8He8=', 'Content-Length': '0'} 2022-09-12T19:07:03.372 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8f35d815f360d04-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:03.373 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.374 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.375 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.375 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.376 DEBUG:boto:Method: GET 2022-09-12T19:07:03.376 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.377 DEBUG:boto:Data: 2022-09-12T19:07:03.377 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.378 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.379 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.379 DEBUG:boto:Params: {} 2022-09-12T19:07:03.380 DEBUG:boto:Token: None 2022-09-12T19:07:03.381 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.382 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1kwxQeng7WSb/gC/creMgfsnrWk= 2022-09-12T19:07:03.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1kwxQeng7WSb/gC/creMgfsnrWk=', 'Content-Length': '0'} 2022-09-12T19:07:03.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003549ca8c29fc745-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.387 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.387 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.388 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.388 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.389 DEBUG:boto:Method: GET 2022-09-12T19:07:03.390 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.390 DEBUG:boto:Data: 2022-09-12T19:07:03.391 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.391 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.392 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.392 DEBUG:boto:Params: {} 2022-09-12T19:07:03.393 DEBUG:boto:Token: None 2022-09-12T19:07:03.394 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.395 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1kwxQeng7WSb/gC/creMgfsnrWk= 2022-09-12T19:07:03.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1kwxQeng7WSb/gC/creMgfsnrWk=', 'Content-Length': '0'} 2022-09-12T19:07:03.398 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea7aad980e4eb896-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.442 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:03.443 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:03.443 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.444 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.445 DEBUG:boto:Method: GET 2022-09-12T19:07:03.445 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:03.446 DEBUG:boto:Data: 2022-09-12T19:07:03.446 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.447 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.447 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.448 DEBUG:boto:Params: {} 2022-09-12T19:07:03.449 DEBUG:boto:Token: None 2022-09-12T19:07:03.449 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/ 2022-09-12T19:07:03.450 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY= 2022-09-12T19:07:03.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY=', 'Content-Length': '0'} 2022-09-12T19:07:03.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b14094e948e7b43-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.455 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:03.455 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:03.456 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:03.457 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.457 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.458 DEBUG:boto:Method: GET 2022-09-12T19:07:03.459 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:03.459 DEBUG:boto:Data: 2022-09-12T19:07:03.460 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.461 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.461 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.462 DEBUG:boto:Params: {} 2022-09-12T19:07:03.463 DEBUG:boto:Token: None 2022-09-12T19:07:03.463 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/ 2022-09-12T19:07:03.464 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY= 2022-09-12T19:07:03.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY=', 'Content-Length': '0'} 2022-09-12T19:07:03.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000000514cc9b1cea48-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.469 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:03.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:03.472 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.473 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.474 DEBUG:boto:path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:07:03.475 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:07:03.476 DEBUG:boto:Method: GET 2022-09-12T19:07:03.477 DEBUG:boto:Path: /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:07:03.478 DEBUG:boto:Data: 2022-09-12T19:07:03.479 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.480 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.481 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.482 DEBUG:boto:Params: {} 2022-09-12T19:07:03.483 DEBUG:boto:Token: None 2022-09-12T19:07:03.484 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:07:03.485 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3GBTvt3aJvwc6xVElNTJwj4kLF8= 2022-09-12T19:07:03.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3GBTvt3aJvwc6xVElNTJwj4kLF8=', 'Content-Length': '0'} 2022-09-12T19:07:03.489 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'K.LUooPdPnXZMzfYIxw-xagQpErk4CX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001616e464997dcadb-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.530 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.531 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.531 DEBUG:boto:path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:07:03.532 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:07:03.532 DEBUG:boto:Method: GET 2022-09-12T19:07:03.533 DEBUG:boto:Path: /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:07:03.534 DEBUG:boto:Data: 2022-09-12T19:07:03.534 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.535 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.535 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.536 DEBUG:boto:Params: {} 2022-09-12T19:07:03.537 DEBUG:boto:Token: None 2022-09-12T19:07:03.537 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:07:03.538 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3GBTvt3aJvwc6xVElNTJwj4kLF8= 2022-09-12T19:07:03.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3GBTvt3aJvwc6xVElNTJwj4kLF8=', 'Content-Length': '0'} 2022-09-12T19:07:03.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'K.LUooPdPnXZMzfYIxw-xagQpErk4CX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065b878ebcd352727-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.542 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.542 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.543 DEBUG:boto:path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:07:03.543 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:07:03.544 DEBUG:boto:Method: HEAD 2022-09-12T19:07:03.544 DEBUG:boto:Path: /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:07:03.545 DEBUG:boto:Data: 2022-09-12T19:07:03.545 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.546 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.547 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.547 DEBUG:boto:Params: {} 2022-09-12T19:07:03.548 DEBUG:boto:Token: None 2022-09-12T19:07:03.548 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:07:03.549 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IOoT8zYdT4DN2BPY4Ruvl2vGk8c= 2022-09-12T19:07:03.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IOoT8zYdT4DN2BPY4Ruvl2vGk8c=', 'Content-Length': '0'} 2022-09-12T19:07:03.552 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'K.LUooPdPnXZMzfYIxw-xagQpErk4CX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e02f82171d5117bd-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.553 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.554 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.554 DEBUG:boto:path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:07:03.555 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:07:03.556 DEBUG:boto:Method: HEAD 2022-09-12T19:07:03.556 DEBUG:boto:Path: /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:07:03.557 DEBUG:boto:Data: 2022-09-12T19:07:03.557 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.558 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.559 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.559 DEBUG:boto:Params: {} 2022-09-12T19:07:03.560 DEBUG:boto:Token: None 2022-09-12T19:07:03.561 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:07:03.561 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IOoT8zYdT4DN2BPY4Ruvl2vGk8c= 2022-09-12T19:07:03.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IOoT8zYdT4DN2BPY4Ruvl2vGk8c=', 'Content-Length': '0'} 2022-09-12T19:07:03.564 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'K.LUooPdPnXZMzfYIxw-xagQpErk4CX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015c6b67ffad498fc-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:03.565 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.566 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.567 DEBUG:boto:path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:07:03.567 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:07:03.568 DEBUG:boto:Method: GET 2022-09-12T19:07:03.568 DEBUG:boto:Path: /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:07:03.569 DEBUG:boto:Data: 2022-09-12T19:07:03.570 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.570 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.571 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.572 DEBUG:boto:Params: {} 2022-09-12T19:07:03.572 DEBUG:boto:Token: None 2022-09-12T19:07:03.573 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:07:03.574 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3GBTvt3aJvwc6xVElNTJwj4kLF8= 2022-09-12T19:07:03.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3GBTvt3aJvwc6xVElNTJwj4kLF8=', 'Content-Length': '0'} 2022-09-12T19:07:03.576 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'K.LUooPdPnXZMzfYIxw-xagQpErk4CX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000964d8bdda0ff2c3d-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.577 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.578 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.578 DEBUG:boto:path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:07:03.579 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:07:03.579 DEBUG:boto:Method: GET 2022-09-12T19:07:03.580 DEBUG:boto:Path: /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:07:03.580 DEBUG:boto:Data: 2022-09-12T19:07:03.581 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.582 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.582 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.583 DEBUG:boto:Params: {} 2022-09-12T19:07:03.583 DEBUG:boto:Token: None 2022-09-12T19:07:03.584 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=K.LUooPdPnXZMzfYIxw-xagQpErk4CX 2022-09-12T19:07:03.585 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3GBTvt3aJvwc6xVElNTJwj4kLF8= 2022-09-12T19:07:03.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3GBTvt3aJvwc6xVElNTJwj4kLF8=', 'Content-Length': '0'} 2022-09-12T19:07:03.588 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'K.LUooPdPnXZMzfYIxw-xagQpErk4CX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000309fe7dde1e05558-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.589 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:03.590 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:03.591 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.592 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.592 DEBUG:boto:Method: GET 2022-09-12T19:07:03.593 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:03.594 DEBUG:boto:Data: 2022-09-12T19:07:03.595 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.596 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.597 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.598 DEBUG:boto:Params: {} 2022-09-12T19:07:03.599 DEBUG:boto:Token: None 2022-09-12T19:07:03.600 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/ 2022-09-12T19:07:03.601 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY= 2022-09-12T19:07:03.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY=', 'Content-Length': '0'} 2022-09-12T19:07:03.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006effcb977b211a64-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.605 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:03.606 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:03.607 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:03.607 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.608 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.609 DEBUG:boto:Method: GET 2022-09-12T19:07:03.610 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:03.611 DEBUG:boto:Data: 2022-09-12T19:07:03.612 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.613 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.614 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.615 DEBUG:boto:Params: {} 2022-09-12T19:07:03.616 DEBUG:boto:Token: None 2022-09-12T19:07:03.617 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/ 2022-09-12T19:07:03.618 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY= 2022-09-12T19:07:03.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY=', 'Content-Length': '0'} 2022-09-12T19:07:03.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1e535c536c92433-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.622 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:03.623 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.624 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.624 DEBUG:boto:Method: HEAD 2022-09-12T19:07:03.625 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:03.626 DEBUG:boto:Data: 2022-09-12T19:07:03.626 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.627 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.627 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.628 DEBUG:boto:Params: {} 2022-09-12T19:07:03.628 DEBUG:boto:Token: None 2022-09-12T19:07:03.629 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj 2022-09-12T19:07:03.629 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fzHvj7kseHGogy+PwK5FBfL8He8= 2022-09-12T19:07:03.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fzHvj7kseHGogy+PwK5FBfL8He8=', 'Content-Length': '0'} 2022-09-12T19:07:03.634 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f38ab0fb5eaba725-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.634 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.635 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.636 DEBUG:boto:Method: HEAD 2022-09-12T19:07:03.636 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:03.637 DEBUG:boto:Data: 2022-09-12T19:07:03.637 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.638 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.638 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.639 DEBUG:boto:Params: {} 2022-09-12T19:07:03.639 DEBUG:boto:Token: None 2022-09-12T19:07:03.640 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj 2022-09-12T19:07:03.641 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fzHvj7kseHGogy+PwK5FBfL8He8= 2022-09-12T19:07:03.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fzHvj7kseHGogy+PwK5FBfL8He8=', 'Content-Length': '0'} 2022-09-12T19:07:03.645 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df2edc6afe59ed58-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:03.646 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.647 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.647 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.648 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.648 DEBUG:boto:Method: GET 2022-09-12T19:07:03.649 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.649 DEBUG:boto:Data: 2022-09-12T19:07:03.650 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.650 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.651 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.652 DEBUG:boto:Params: {} 2022-09-12T19:07:03.652 DEBUG:boto:Token: None 2022-09-12T19:07:03.653 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.654 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1kwxQeng7WSb/gC/creMgfsnrWk= 2022-09-12T19:07:03.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1kwxQeng7WSb/gC/creMgfsnrWk=', 'Content-Length': '0'} 2022-09-12T19:07:03.657 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000519241e26bff4d3d-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.658 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.658 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.659 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.659 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.660 DEBUG:boto:Method: GET 2022-09-12T19:07:03.661 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.661 DEBUG:boto:Data: 2022-09-12T19:07:03.662 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.662 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.663 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.663 DEBUG:boto:Params: {} 2022-09-12T19:07:03.664 DEBUG:boto:Token: None 2022-09-12T19:07:03.664 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.665 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1kwxQeng7WSb/gC/creMgfsnrWk= 2022-09-12T19:07:03.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1kwxQeng7WSb/gC/creMgfsnrWk=', 'Content-Length': '0'} 2022-09-12T19:07:03.668 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b41383ba0a2077c-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.669 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:03.669 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:03.670 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.670 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.671 DEBUG:boto:Method: GET 2022-09-12T19:07:03.672 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:03.672 DEBUG:boto:Data: 2022-09-12T19:07:03.673 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.673 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.674 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.674 DEBUG:boto:Params: {} 2022-09-12T19:07:03.675 DEBUG:boto:Token: None 2022-09-12T19:07:03.675 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/ 2022-09-12T19:07:03.676 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY= 2022-09-12T19:07:03.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY=', 'Content-Length': '0'} 2022-09-12T19:07:03.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fcff79b9dd4d690-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.680 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:03.681 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:03.682 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:03.683 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.683 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.684 DEBUG:boto:Method: GET 2022-09-12T19:07:03.684 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:03.685 DEBUG:boto:Data: 2022-09-12T19:07:03.686 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.686 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.687 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.687 DEBUG:boto:Params: {} 2022-09-12T19:07:03.688 DEBUG:boto:Token: None 2022-09-12T19:07:03.688 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/ 2022-09-12T19:07:03.689 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY= 2022-09-12T19:07:03.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY=', 'Content-Length': '0'} 2022-09-12T19:07:03.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e309f89bbb1d2a77-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.693 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:03.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:03.694 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.695 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.696 DEBUG:boto:path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:07:03.696 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:07:03.697 DEBUG:boto:Method: GET 2022-09-12T19:07:03.697 DEBUG:boto:Path: /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:07:03.698 DEBUG:boto:Data: 2022-09-12T19:07:03.698 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.699 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.699 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.700 DEBUG:boto:Params: {} 2022-09-12T19:07:03.700 DEBUG:boto:Token: None 2022-09-12T19:07:03.701 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:07:03.702 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4RL2j9w2uFyqFsI5UEroJEY0zWs= 2022-09-12T19:07:03.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4RL2j9w2uFyqFsI5UEroJEY0zWs=', 'Content-Length': '0'} 2022-09-12T19:07:03.705 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'tQHk21B862H4VHQxrZWEog8n-qgO99Q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000906c791e4157fe00-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.706 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.706 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.707 DEBUG:boto:path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:07:03.707 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:07:03.708 DEBUG:boto:Method: GET 2022-09-12T19:07:03.708 DEBUG:boto:Path: /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:07:03.709 DEBUG:boto:Data: 2022-09-12T19:07:03.709 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.710 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.710 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.711 DEBUG:boto:Params: {} 2022-09-12T19:07:03.712 DEBUG:boto:Token: None 2022-09-12T19:07:03.713 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:07:03.714 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4RL2j9w2uFyqFsI5UEroJEY0zWs= 2022-09-12T19:07:03.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4RL2j9w2uFyqFsI5UEroJEY0zWs=', 'Content-Length': '0'} 2022-09-12T19:07:03.717 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'tQHk21B862H4VHQxrZWEog8n-qgO99Q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b31bb208721e94dd-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.719 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.720 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.721 DEBUG:boto:path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:07:03.722 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:07:03.722 DEBUG:boto:Method: HEAD 2022-09-12T19:07:03.723 DEBUG:boto:Path: /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:07:03.724 DEBUG:boto:Data: 2022-09-12T19:07:03.725 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.726 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.727 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.728 DEBUG:boto:Params: {} 2022-09-12T19:07:03.729 DEBUG:boto:Token: None 2022-09-12T19:07:03.730 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:07:03.731 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EKEJrzil5oDvNXLO6JYX6mtI45Y= 2022-09-12T19:07:03.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EKEJrzil5oDvNXLO6JYX6mtI45Y=', 'Content-Length': '0'} 2022-09-12T19:07:03.735 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'tQHk21B862H4VHQxrZWEog8n-qgO99Q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7cca14a57b38f59-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.737 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.737 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.738 DEBUG:boto:path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:07:03.739 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:07:03.740 DEBUG:boto:Method: HEAD 2022-09-12T19:07:03.741 DEBUG:boto:Path: /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:07:03.742 DEBUG:boto:Data: 2022-09-12T19:07:03.743 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.744 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.745 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.746 DEBUG:boto:Params: {} 2022-09-12T19:07:03.747 DEBUG:boto:Token: None 2022-09-12T19:07:03.748 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:07:03.749 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EKEJrzil5oDvNXLO6JYX6mtI45Y= 2022-09-12T19:07:03.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EKEJrzil5oDvNXLO6JYX6mtI45Y=', 'Content-Length': '0'} 2022-09-12T19:07:03.753 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'tQHk21B862H4VHQxrZWEog8n-qgO99Q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099bbacb679a3b97c-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:03.755 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.756 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.757 DEBUG:boto:path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:07:03.758 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:07:03.759 DEBUG:boto:Method: GET 2022-09-12T19:07:03.760 DEBUG:boto:Path: /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:07:03.761 DEBUG:boto:Data: 2022-09-12T19:07:03.762 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.763 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.765 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.765 DEBUG:boto:Params: {} 2022-09-12T19:07:03.767 DEBUG:boto:Token: None 2022-09-12T19:07:03.768 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:07:03.768 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4RL2j9w2uFyqFsI5UEroJEY0zWs= 2022-09-12T19:07:03.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4RL2j9w2uFyqFsI5UEroJEY0zWs=', 'Content-Length': '0'} 2022-09-12T19:07:03.771 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'tQHk21B862H4VHQxrZWEog8n-qgO99Q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d92b321fe127d66-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.772 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.773 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.773 DEBUG:boto:path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:07:03.774 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:07:03.775 DEBUG:boto:Method: GET 2022-09-12T19:07:03.775 DEBUG:boto:Path: /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:07:03.776 DEBUG:boto:Data: 2022-09-12T19:07:03.776 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.777 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.777 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.778 DEBUG:boto:Params: {} 2022-09-12T19:07:03.778 DEBUG:boto:Token: None 2022-09-12T19:07:03.779 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=tQHk21B862H4VHQxrZWEog8n-qgO99Q 2022-09-12T19:07:03.780 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4RL2j9w2uFyqFsI5UEroJEY0zWs= 2022-09-12T19:07:03.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4RL2j9w2uFyqFsI5UEroJEY0zWs=', 'Content-Length': '0'} 2022-09-12T19:07:03.782 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'tQHk21B862H4VHQxrZWEog8n-qgO99Q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be3afac52b625de8-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.783 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:03.784 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:03.784 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.785 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.786 DEBUG:boto:Method: GET 2022-09-12T19:07:03.786 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:03.787 DEBUG:boto:Data: 2022-09-12T19:07:03.787 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.788 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.788 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.789 DEBUG:boto:Params: {} 2022-09-12T19:07:03.789 DEBUG:boto:Token: None 2022-09-12T19:07:03.790 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/ 2022-09-12T19:07:03.791 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY= 2022-09-12T19:07:03.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY=', 'Content-Length': '0'} 2022-09-12T19:07:03.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af5475cd66b4f6ef-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.795 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:03.797 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:03.798 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:03.799 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.800 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.801 DEBUG:boto:Method: GET 2022-09-12T19:07:03.802 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:03.803 DEBUG:boto:Data: 2022-09-12T19:07:03.803 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.804 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.805 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.805 DEBUG:boto:Params: {} 2022-09-12T19:07:03.806 DEBUG:boto:Token: None 2022-09-12T19:07:03.806 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/ 2022-09-12T19:07:03.807 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY= 2022-09-12T19:07:03.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY=', 'Content-Length': '0'} 2022-09-12T19:07:03.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016126addce02d676-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.858 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:03.859 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.860 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.860 DEBUG:boto:Method: HEAD 2022-09-12T19:07:03.861 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:03.861 DEBUG:boto:Data: 2022-09-12T19:07:03.862 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.862 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.863 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.863 DEBUG:boto:Params: {} 2022-09-12T19:07:03.864 DEBUG:boto:Token: None 2022-09-12T19:07:03.865 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj 2022-09-12T19:07:03.866 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fzHvj7kseHGogy+PwK5FBfL8He8= 2022-09-12T19:07:03.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fzHvj7kseHGogy+PwK5FBfL8He8=', 'Content-Length': '0'} 2022-09-12T19:07:03.870 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3183ebc8d1f4c78-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.871 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.871 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.872 DEBUG:boto:Method: HEAD 2022-09-12T19:07:03.873 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:03.873 DEBUG:boto:Data: 2022-09-12T19:07:03.874 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.874 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.875 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.876 DEBUG:boto:Params: {} 2022-09-12T19:07:03.876 DEBUG:boto:Token: None 2022-09-12T19:07:03.877 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj 2022-09-12T19:07:03.878 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fzHvj7kseHGogy+PwK5FBfL8He8= 2022-09-12T19:07:03.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fzHvj7kseHGogy+PwK5FBfL8He8=', 'Content-Length': '0'} 2022-09-12T19:07:03.882 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c76325c4deffd54e-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:03.883 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.884 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.884 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.885 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.885 DEBUG:boto:Method: GET 2022-09-12T19:07:03.886 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.887 DEBUG:boto:Data: 2022-09-12T19:07:03.887 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.888 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.888 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.889 DEBUG:boto:Params: {} 2022-09-12T19:07:03.890 DEBUG:boto:Token: None 2022-09-12T19:07:03.890 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.891 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1kwxQeng7WSb/gC/creMgfsnrWk= 2022-09-12T19:07:03.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1kwxQeng7WSb/gC/creMgfsnrWk=', 'Content-Length': '0'} 2022-09-12T19:07:03.894 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb4c50b831608c24-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.895 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.895 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.896 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.897 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.898 DEBUG:boto:Method: GET 2022-09-12T19:07:03.899 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.900 DEBUG:boto:Data: 2022-09-12T19:07:03.901 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.902 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.902 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.903 DEBUG:boto:Params: {} 2022-09-12T19:07:03.904 DEBUG:boto:Token: None 2022-09-12T19:07:03.905 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:03.905 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1kwxQeng7WSb/gC/creMgfsnrWk= 2022-09-12T19:07:03.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1kwxQeng7WSb/gC/creMgfsnrWk=', 'Content-Length': '0'} 2022-09-12T19:07:03.908 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006cc66a24f9ad5583-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.909 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:03.910 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:03.910 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.911 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.911 DEBUG:boto:Method: GET 2022-09-12T19:07:03.912 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:03.913 DEBUG:boto:Data: 2022-09-12T19:07:03.913 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.914 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.915 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.915 DEBUG:boto:Params: {} 2022-09-12T19:07:03.916 DEBUG:boto:Token: None 2022-09-12T19:07:03.916 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/ 2022-09-12T19:07:03.917 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY= 2022-09-12T19:07:03.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY=', 'Content-Length': '0'} 2022-09-12T19:07:03.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7f002821cac9d26-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.921 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:03.922 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:03.923 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:03.923 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.924 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:03.925 DEBUG:boto:Method: GET 2022-09-12T19:07:03.925 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:03.926 DEBUG:boto:Data: 2022-09-12T19:07:03.926 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.927 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.928 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.929 DEBUG:boto:Params: {} 2022-09-12T19:07:03.930 DEBUG:boto:Token: None 2022-09-12T19:07:03.931 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/ 2022-09-12T19:07:03.932 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY= 2022-09-12T19:07:03.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2cg8Xp+a/g4JUi0IWxOlBSdf5DY=', 'Content-Length': '0'} 2022-09-12T19:07:03.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b90d7100deb8f84b-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.938 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:03.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:03.939 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.940 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.941 DEBUG:boto:path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:07:03.941 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:07:03.942 DEBUG:boto:Method: GET 2022-09-12T19:07:03.942 DEBUG:boto:Path: /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:07:03.943 DEBUG:boto:Data: 2022-09-12T19:07:03.944 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.945 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.946 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.947 DEBUG:boto:Params: {} 2022-09-12T19:07:03.947 DEBUG:boto:Token: None 2022-09-12T19:07:03.948 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:07:03.949 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eS9UQ+bNdvXw9FNaAZIFwm82l4E= 2022-09-12T19:07:03.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eS9UQ+bNdvXw9FNaAZIFwm82l4E=', 'Content-Length': '0'} 2022-09-12T19:07:03.951 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'QHXOu7SGV87vItdBeqaL.hYUlX--Ty8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082e98a6b595915c0-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.952 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.953 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.953 DEBUG:boto:path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:07:03.954 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:07:03.954 DEBUG:boto:Method: GET 2022-09-12T19:07:03.955 DEBUG:boto:Path: /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:07:03.956 DEBUG:boto:Data: 2022-09-12T19:07:03.956 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.957 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.957 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.958 DEBUG:boto:Params: {} 2022-09-12T19:07:03.959 DEBUG:boto:Token: None 2022-09-12T19:07:03.959 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:07:03.960 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eS9UQ+bNdvXw9FNaAZIFwm82l4E= 2022-09-12T19:07:03.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eS9UQ+bNdvXw9FNaAZIFwm82l4E=', 'Content-Length': '0'} 2022-09-12T19:07:03.963 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'QHXOu7SGV87vItdBeqaL.hYUlX--Ty8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cdd3cc9860ee9132-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.964 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.965 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.965 DEBUG:boto:path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:07:03.966 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:07:03.967 DEBUG:boto:Method: HEAD 2022-09-12T19:07:03.968 DEBUG:boto:Path: /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:07:03.969 DEBUG:boto:Data: 2022-09-12T19:07:03.969 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.970 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.970 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.971 DEBUG:boto:Params: {} 2022-09-12T19:07:03.972 DEBUG:boto:Token: None 2022-09-12T19:07:03.972 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:07:03.973 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:g7X7wY6yc4l0IBYSamZB0aXwHDo= 2022-09-12T19:07:03.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:g7X7wY6yc4l0IBYSamZB0aXwHDo=', 'Content-Length': '0'} 2022-09-12T19:07:03.976 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'QHXOu7SGV87vItdBeqaL.hYUlX--Ty8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1c21fa23850d4f1-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.977 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.977 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.978 DEBUG:boto:path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:07:03.978 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:07:03.979 DEBUG:boto:Method: HEAD 2022-09-12T19:07:03.979 DEBUG:boto:Path: /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:07:03.980 DEBUG:boto:Data: 2022-09-12T19:07:03.980 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.981 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:03.981 DEBUG:boto:Port: 8000 2022-09-12T19:07:03.982 DEBUG:boto:Params: {} 2022-09-12T19:07:03.982 DEBUG:boto:Token: None 2022-09-12T19:07:03.983 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:07:03.984 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:g7X7wY6yc4l0IBYSamZB0aXwHDo= 2022-09-12T19:07:03.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:g7X7wY6yc4l0IBYSamZB0aXwHDo=', 'Content-Length': '0'} 2022-09-12T19:07:03.986 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'QHXOu7SGV87vItdBeqaL.hYUlX--Ty8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca1cafb89a3994a2-00631f8357-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:03.987 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.988 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:03.989 DEBUG:boto:path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:07:03.989 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:07:03.990 DEBUG:boto:Method: GET 2022-09-12T19:07:03.990 DEBUG:boto:Path: /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:07:03.991 DEBUG:boto:Data: 2022-09-12T19:07:03.991 DEBUG:boto:Headers: {} 2022-09-12T19:07:03.992 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:03.992 DEBUG:boto:Port: 8001 2022-09-12T19:07:03.993 DEBUG:boto:Params: {} 2022-09-12T19:07:03.993 DEBUG:boto:Token: None 2022-09-12T19:07:03.994 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:03 GMT /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:07:03.995 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eS9UQ+bNdvXw9FNaAZIFwm82l4E= 2022-09-12T19:07:03.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eS9UQ+bNdvXw9FNaAZIFwm82l4E=', 'Content-Length': '0'} 2022-09-12T19:07:03.997 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'QHXOu7SGV87vItdBeqaL.hYUlX--Ty8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005aaf8cce8bcdc2ce-00631f8357-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:03.998 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:03.999 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.000 DEBUG:boto:path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:07:04.000 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:07:04.001 DEBUG:boto:Method: GET 2022-09-12T19:07:04.001 DEBUG:boto:Path: /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:07:04.002 DEBUG:boto:Data: 2022-09-12T19:07:04.002 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.003 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:04.003 DEBUG:boto:Port: 8000 2022-09-12T19:07:04.004 DEBUG:boto:Params: {} 2022-09-12T19:07:04.005 DEBUG:boto:Token: None 2022-09-12T19:07:04.005 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=QHXOu7SGV87vItdBeqaL.hYUlX--Ty8 2022-09-12T19:07:04.006 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KqIkgLW7nuo+4MrwDjD+IRR2+cM= 2022-09-12T19:07:04.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KqIkgLW7nuo+4MrwDjD+IRR2+cM=', 'Content-Length': '0'} 2022-09-12T19:07:04.009 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'QHXOu7SGV87vItdBeqaL.hYUlX--Ty8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ba886a17b9d5644-00631f8358-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.050 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:04.052 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:04.053 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.054 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.055 DEBUG:boto:Method: GET 2022-09-12T19:07:04.056 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:04.057 DEBUG:boto:Data: 2022-09-12T19:07:04.058 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.059 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:04.060 DEBUG:boto:Port: 8001 2022-09-12T19:07:04.061 DEBUG:boto:Params: {} 2022-09-12T19:07:04.062 DEBUG:boto:Token: None 2022-09-12T19:07:04.063 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/ 2022-09-12T19:07:04.064 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs= 2022-09-12T19:07:04.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs=', 'Content-Length': '0'} 2022-09-12T19:07:04.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f13e13f280402fa-00631f8358-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.071 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:04.073 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:04.074 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:04.075 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.076 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.077 DEBUG:boto:Method: GET 2022-09-12T19:07:04.078 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:04.079 DEBUG:boto:Data: 2022-09-12T19:07:04.080 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.081 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:04.082 DEBUG:boto:Port: 8000 2022-09-12T19:07:04.083 DEBUG:boto:Params: {} 2022-09-12T19:07:04.083 DEBUG:boto:Token: None 2022-09-12T19:07:04.084 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/ 2022-09-12T19:07:04.085 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs= 2022-09-12T19:07:04.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs=', 'Content-Length': '0'} 2022-09-12T19:07:04.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da14b7d92414e18e-00631f8358-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.089 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:04.090 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.091 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.091 DEBUG:boto:Method: HEAD 2022-09-12T19:07:04.092 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:04.092 DEBUG:boto:Data: 2022-09-12T19:07:04.093 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.093 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:04.094 DEBUG:boto:Port: 8001 2022-09-12T19:07:04.095 DEBUG:boto:Params: {} 2022-09-12T19:07:04.095 DEBUG:boto:Token: None 2022-09-12T19:07:04.096 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj 2022-09-12T19:07:04.096 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qNKPFCByVwlzR46AvTHaLiUNsS0= 2022-09-12T19:07:04.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qNKPFCByVwlzR46AvTHaLiUNsS0=', 'Content-Length': '0'} 2022-09-12T19:07:04.101 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000447554ced4517c01-00631f8358-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.102 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.103 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.104 DEBUG:boto:Method: HEAD 2022-09-12T19:07:04.104 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:04.105 DEBUG:boto:Data: 2022-09-12T19:07:04.105 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.106 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:04.106 DEBUG:boto:Port: 8000 2022-09-12T19:07:04.107 DEBUG:boto:Params: {} 2022-09-12T19:07:04.107 DEBUG:boto:Token: None 2022-09-12T19:07:04.108 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj 2022-09-12T19:07:04.108 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qNKPFCByVwlzR46AvTHaLiUNsS0= 2022-09-12T19:07:04.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qNKPFCByVwlzR46AvTHaLiUNsS0=', 'Content-Length': '0'} 2022-09-12T19:07:04.113 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091531e778aa055a6-00631f8358-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:04.114 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.115 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.115 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.116 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.116 DEBUG:boto:Method: GET 2022-09-12T19:07:04.117 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.117 DEBUG:boto:Data: 2022-09-12T19:07:04.118 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.118 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:04.119 DEBUG:boto:Port: 8001 2022-09-12T19:07:04.119 DEBUG:boto:Params: {} 2022-09-12T19:07:04.120 DEBUG:boto:Token: None 2022-09-12T19:07:04.121 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.121 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mLmGlAHthfxBGCR43Dobd6gJAT4= 2022-09-12T19:07:04.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mLmGlAHthfxBGCR43Dobd6gJAT4=', 'Content-Length': '0'} 2022-09-12T19:07:04.124 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3de8833be097541-00631f8358-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.125 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.126 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.127 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.128 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.129 DEBUG:boto:Method: GET 2022-09-12T19:07:04.130 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.131 DEBUG:boto:Data: 2022-09-12T19:07:04.132 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.133 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:04.134 DEBUG:boto:Port: 8000 2022-09-12T19:07:04.135 DEBUG:boto:Params: {} 2022-09-12T19:07:04.136 DEBUG:boto:Token: None 2022-09-12T19:07:04.137 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.138 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mLmGlAHthfxBGCR43Dobd6gJAT4= 2022-09-12T19:07:04.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mLmGlAHthfxBGCR43Dobd6gJAT4=', 'Content-Length': '0'} 2022-09-12T19:07:04.141 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009568a587bcf6d61c-00631f8358-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.142 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:04.142 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:04.143 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.143 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.144 DEBUG:boto:Method: GET 2022-09-12T19:07:04.144 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:04.145 DEBUG:boto:Data: 2022-09-12T19:07:04.145 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.146 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:04.146 DEBUG:boto:Port: 8001 2022-09-12T19:07:04.147 DEBUG:boto:Params: {} 2022-09-12T19:07:04.148 DEBUG:boto:Token: None 2022-09-12T19:07:04.148 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/ 2022-09-12T19:07:04.149 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs= 2022-09-12T19:07:04.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs=', 'Content-Length': '0'} 2022-09-12T19:07:04.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fc6aec5dae046aa-00631f8358-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.153 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:04.165 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:04.166 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:04.167 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.167 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.168 DEBUG:boto:Method: GET 2022-09-12T19:07:04.168 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:04.169 DEBUG:boto:Data: 2022-09-12T19:07:04.169 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.170 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:04.170 DEBUG:boto:Port: 8000 2022-09-12T19:07:04.171 DEBUG:boto:Params: {} 2022-09-12T19:07:04.171 DEBUG:boto:Token: None 2022-09-12T19:07:04.172 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/ 2022-09-12T19:07:04.173 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs= 2022-09-12T19:07:04.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs=', 'Content-Length': '0'} 2022-09-12T19:07:04.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e52113c8a8b30e9e-00631f8358-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.178 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:04.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:04.179 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.180 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.180 DEBUG:boto:path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:07:04.181 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:07:04.181 DEBUG:boto:Method: GET 2022-09-12T19:07:04.182 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:07:04.183 DEBUG:boto:Data: 2022-09-12T19:07:04.183 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.184 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:04.184 DEBUG:boto:Port: 8001 2022-09-12T19:07:04.185 DEBUG:boto:Params: {} 2022-09-12T19:07:04.185 DEBUG:boto:Token: None 2022-09-12T19:07:04.186 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:07:04.187 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OYcW1uZygDpuqu0iC2Fdb2hs41A= 2022-09-12T19:07:04.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OYcW1uZygDpuqu0iC2Fdb2hs41A=', 'Content-Length': '0'} 2022-09-12T19:07:04.190 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000050b5a91aca0d0c30-00631f8358-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.191 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.192 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.193 DEBUG:boto:path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:07:04.194 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:07:04.195 DEBUG:boto:Method: GET 2022-09-12T19:07:04.196 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:07:04.197 DEBUG:boto:Data: 2022-09-12T19:07:04.198 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.199 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:04.200 DEBUG:boto:Port: 8000 2022-09-12T19:07:04.201 DEBUG:boto:Params: {} 2022-09-12T19:07:04.202 DEBUG:boto:Token: None 2022-09-12T19:07:04.203 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:07:04.203 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OYcW1uZygDpuqu0iC2Fdb2hs41A= 2022-09-12T19:07:04.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OYcW1uZygDpuqu0iC2Fdb2hs41A=', 'Content-Length': '0'} 2022-09-12T19:07:04.206 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000859c242285a26e10-00631f8358-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.207 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.208 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.208 DEBUG:boto:path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:07:04.209 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:07:04.210 DEBUG:boto:Method: HEAD 2022-09-12T19:07:04.210 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:07:04.211 DEBUG:boto:Data: 2022-09-12T19:07:04.212 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.212 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:04.213 DEBUG:boto:Port: 8001 2022-09-12T19:07:04.213 DEBUG:boto:Params: {} 2022-09-12T19:07:04.214 DEBUG:boto:Token: None 2022-09-12T19:07:04.214 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:07:04.215 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nDqZq3lLlAAZ1d9pdZA/APr8leo= 2022-09-12T19:07:04.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nDqZq3lLlAAZ1d9pdZA/APr8leo=', 'Content-Length': '0'} 2022-09-12T19:07:04.218 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6540ebf37db0a11-00631f8358-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.219 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.219 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.220 DEBUG:boto:path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:07:04.220 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:07:04.221 DEBUG:boto:Method: HEAD 2022-09-12T19:07:04.221 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:07:04.222 DEBUG:boto:Data: 2022-09-12T19:07:04.222 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.223 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:04.223 DEBUG:boto:Port: 8000 2022-09-12T19:07:04.224 DEBUG:boto:Params: {} 2022-09-12T19:07:04.224 DEBUG:boto:Token: None 2022-09-12T19:07:04.225 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:07:04.226 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nDqZq3lLlAAZ1d9pdZA/APr8leo= 2022-09-12T19:07:04.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nDqZq3lLlAAZ1d9pdZA/APr8leo=', 'Content-Length': '0'} 2022-09-12T19:07:04.229 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6bcb28321e80f9e-00631f8358-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:04.231 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.232 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.233 DEBUG:boto:path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:07:04.233 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:07:04.234 DEBUG:boto:Method: GET 2022-09-12T19:07:04.235 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:07:04.236 DEBUG:boto:Data: 2022-09-12T19:07:04.236 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.237 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:04.238 DEBUG:boto:Port: 8001 2022-09-12T19:07:04.238 DEBUG:boto:Params: {} 2022-09-12T19:07:04.239 DEBUG:boto:Token: None 2022-09-12T19:07:04.240 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:07:04.240 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OYcW1uZygDpuqu0iC2Fdb2hs41A= 2022-09-12T19:07:04.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OYcW1uZygDpuqu0iC2Fdb2hs41A=', 'Content-Length': '0'} 2022-09-12T19:07:04.243 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8347031f494be2d-00631f8358-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.290 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.292 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.292 DEBUG:boto:path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:07:04.293 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:07:04.293 DEBUG:boto:Method: GET 2022-09-12T19:07:04.294 DEBUG:boto:Path: /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:07:04.295 DEBUG:boto:Data: 2022-09-12T19:07:04.295 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.296 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:04.296 DEBUG:boto:Port: 8000 2022-09-12T19:07:04.297 DEBUG:boto:Params: {} 2022-09-12T19:07:04.297 DEBUG:boto:Token: None 2022-09-12T19:07:04.298 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3 2022-09-12T19:07:04.299 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OYcW1uZygDpuqu0iC2Fdb2hs41A= 2022-09-12T19:07:04.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OYcW1uZygDpuqu0iC2Fdb2hs41A=', 'Content-Length': '0'} 2022-09-12T19:07:04.302 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'T-rka2v4NcjXyfB8GzT0u7KHTz6OcF3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbfcafa124ee3dcb-00631f8358-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.303 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:04.303 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:04.304 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.304 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.305 DEBUG:boto:Method: GET 2022-09-12T19:07:04.306 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:04.306 DEBUG:boto:Data: 2022-09-12T19:07:04.307 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.308 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:04.308 DEBUG:boto:Port: 8001 2022-09-12T19:07:04.309 DEBUG:boto:Params: {} 2022-09-12T19:07:04.310 DEBUG:boto:Token: None 2022-09-12T19:07:04.310 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/ 2022-09-12T19:07:04.311 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs= 2022-09-12T19:07:04.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs=', 'Content-Length': '0'} 2022-09-12T19:07:04.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d8d6ddfaa1c252f-00631f8358-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.316 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:04.317 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:04.318 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:04.319 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.320 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.320 DEBUG:boto:Method: GET 2022-09-12T19:07:04.321 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:04.321 DEBUG:boto:Data: 2022-09-12T19:07:04.322 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.322 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:04.323 DEBUG:boto:Port: 8000 2022-09-12T19:07:04.324 DEBUG:boto:Params: {} 2022-09-12T19:07:04.324 DEBUG:boto:Token: None 2022-09-12T19:07:04.325 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/ 2022-09-12T19:07:04.326 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs= 2022-09-12T19:07:04.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs=', 'Content-Length': '0'} 2022-09-12T19:07:04.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080a5f8d99e538adf-00631f8358-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.370 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:04.371 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.372 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.372 DEBUG:boto:Method: HEAD 2022-09-12T19:07:04.373 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:04.374 DEBUG:boto:Data: 2022-09-12T19:07:04.374 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.375 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:04.375 DEBUG:boto:Port: 8001 2022-09-12T19:07:04.376 DEBUG:boto:Params: {} 2022-09-12T19:07:04.376 DEBUG:boto:Token: None 2022-09-12T19:07:04.377 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj 2022-09-12T19:07:04.378 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qNKPFCByVwlzR46AvTHaLiUNsS0= 2022-09-12T19:07:04.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qNKPFCByVwlzR46AvTHaLiUNsS0=', 'Content-Length': '0'} 2022-09-12T19:07:04.383 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080dcf9955399b3ca-00631f8358-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.384 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.384 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.385 DEBUG:boto:Method: HEAD 2022-09-12T19:07:04.385 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:04.386 DEBUG:boto:Data: 2022-09-12T19:07:04.386 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.387 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:04.387 DEBUG:boto:Port: 8000 2022-09-12T19:07:04.388 DEBUG:boto:Params: {} 2022-09-12T19:07:04.389 DEBUG:boto:Token: None 2022-09-12T19:07:04.389 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj 2022-09-12T19:07:04.390 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qNKPFCByVwlzR46AvTHaLiUNsS0= 2022-09-12T19:07:04.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qNKPFCByVwlzR46AvTHaLiUNsS0=', 'Content-Length': '0'} 2022-09-12T19:07:04.394 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3aa2bb74392f325-00631f8358-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:04.395 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.396 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.396 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.397 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.397 DEBUG:boto:Method: GET 2022-09-12T19:07:04.398 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.399 DEBUG:boto:Data: 2022-09-12T19:07:04.399 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.400 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:04.400 DEBUG:boto:Port: 8001 2022-09-12T19:07:04.401 DEBUG:boto:Params: {} 2022-09-12T19:07:04.401 DEBUG:boto:Token: None 2022-09-12T19:07:04.402 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.402 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mLmGlAHthfxBGCR43Dobd6gJAT4= 2022-09-12T19:07:04.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mLmGlAHthfxBGCR43Dobd6gJAT4=', 'Content-Length': '0'} 2022-09-12T19:07:04.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d910e2d6e2f9064a-00631f8358-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.406 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.407 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.407 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.408 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.408 DEBUG:boto:Method: GET 2022-09-12T19:07:04.409 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.409 DEBUG:boto:Data: 2022-09-12T19:07:04.410 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.410 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:04.411 DEBUG:boto:Port: 8000 2022-09-12T19:07:04.411 DEBUG:boto:Params: {} 2022-09-12T19:07:04.412 DEBUG:boto:Token: None 2022-09-12T19:07:04.413 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.413 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mLmGlAHthfxBGCR43Dobd6gJAT4= 2022-09-12T19:07:04.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mLmGlAHthfxBGCR43Dobd6gJAT4=', 'Content-Length': '0'} 2022-09-12T19:07:04.416 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b21e361af284467-00631f8358-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.417 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:04.417 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:04.418 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.418 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.419 DEBUG:boto:Method: GET 2022-09-12T19:07:04.419 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:04.420 DEBUG:boto:Data: 2022-09-12T19:07:04.420 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.421 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:04.421 DEBUG:boto:Port: 8001 2022-09-12T19:07:04.422 DEBUG:boto:Params: {} 2022-09-12T19:07:04.422 DEBUG:boto:Token: None 2022-09-12T19:07:04.423 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/ 2022-09-12T19:07:04.424 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs= 2022-09-12T19:07:04.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs=', 'Content-Length': '0'} 2022-09-12T19:07:04.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca85353f34c4016e-00631f8358-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.428 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:04.429 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:04.429 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:04.430 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.430 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.431 DEBUG:boto:Method: GET 2022-09-12T19:07:04.431 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:04.432 DEBUG:boto:Data: 2022-09-12T19:07:04.433 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.433 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:04.434 DEBUG:boto:Port: 8000 2022-09-12T19:07:04.434 DEBUG:boto:Params: {} 2022-09-12T19:07:04.435 DEBUG:boto:Token: None 2022-09-12T19:07:04.435 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/ 2022-09-12T19:07:04.436 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs= 2022-09-12T19:07:04.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs=', 'Content-Length': '0'} 2022-09-12T19:07:04.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9131390c28aac89-00631f8358-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.441 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:04.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:04.442 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.443 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.443 DEBUG:boto:path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:07:04.444 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:07:04.444 DEBUG:boto:Method: GET 2022-09-12T19:07:04.445 DEBUG:boto:Path: /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:07:04.445 DEBUG:boto:Data: 2022-09-12T19:07:04.446 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.446 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:04.447 DEBUG:boto:Port: 8001 2022-09-12T19:07:04.447 DEBUG:boto:Params: {} 2022-09-12T19:07:04.448 DEBUG:boto:Token: None 2022-09-12T19:07:04.448 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:07:04.449 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UHefSLbTAmThINuJbrqC2iYfCtU= 2022-09-12T19:07:04.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UHefSLbTAmThINuJbrqC2iYfCtU=', 'Content-Length': '0'} 2022-09-12T19:07:04.452 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b2c65b622e1c54e4-00631f8358-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.494 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.496 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.497 DEBUG:boto:path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:07:04.498 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:07:04.498 DEBUG:boto:Method: GET 2022-09-12T19:07:04.499 DEBUG:boto:Path: /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:07:04.500 DEBUG:boto:Data: 2022-09-12T19:07:04.501 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.502 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:04.502 DEBUG:boto:Port: 8000 2022-09-12T19:07:04.504 DEBUG:boto:Params: {} 2022-09-12T19:07:04.505 DEBUG:boto:Token: None 2022-09-12T19:07:04.505 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:07:04.506 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UHefSLbTAmThINuJbrqC2iYfCtU= 2022-09-12T19:07:04.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UHefSLbTAmThINuJbrqC2iYfCtU=', 'Content-Length': '0'} 2022-09-12T19:07:04.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000487d304103e286bd-00631f8358-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.510 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.511 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.511 DEBUG:boto:path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:07:04.512 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:07:04.512 DEBUG:boto:Method: HEAD 2022-09-12T19:07:04.513 DEBUG:boto:Path: /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:07:04.513 DEBUG:boto:Data: 2022-09-12T19:07:04.514 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.514 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:04.515 DEBUG:boto:Port: 8001 2022-09-12T19:07:04.515 DEBUG:boto:Params: {} 2022-09-12T19:07:04.516 DEBUG:boto:Token: None 2022-09-12T19:07:04.516 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:07:04.517 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:O1Vgo4xLa/RmMQQ+/aQHFSVB7MU= 2022-09-12T19:07:04.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:O1Vgo4xLa/RmMQQ+/aQHFSVB7MU=', 'Content-Length': '0'} 2022-09-12T19:07:04.520 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007709eba0fc7341e2-00631f8358-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.520 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.521 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.522 DEBUG:boto:path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:07:04.522 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:07:04.523 DEBUG:boto:Method: HEAD 2022-09-12T19:07:04.523 DEBUG:boto:Path: /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:07:04.524 DEBUG:boto:Data: 2022-09-12T19:07:04.524 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.525 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:04.525 DEBUG:boto:Port: 8000 2022-09-12T19:07:04.526 DEBUG:boto:Params: {} 2022-09-12T19:07:04.526 DEBUG:boto:Token: None 2022-09-12T19:07:04.527 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:07:04.527 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:O1Vgo4xLa/RmMQQ+/aQHFSVB7MU= 2022-09-12T19:07:04.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:O1Vgo4xLa/RmMQQ+/aQHFSVB7MU=', 'Content-Length': '0'} 2022-09-12T19:07:04.530 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048a0ac5fb6a80332-00631f8358-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:04.532 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.534 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.534 DEBUG:boto:path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:07:04.535 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:07:04.536 DEBUG:boto:Method: GET 2022-09-12T19:07:04.537 DEBUG:boto:Path: /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:07:04.537 DEBUG:boto:Data: 2022-09-12T19:07:04.538 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.538 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:04.539 DEBUG:boto:Port: 8001 2022-09-12T19:07:04.540 DEBUG:boto:Params: {} 2022-09-12T19:07:04.540 DEBUG:boto:Token: None 2022-09-12T19:07:04.541 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:07:04.542 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UHefSLbTAmThINuJbrqC2iYfCtU= 2022-09-12T19:07:04.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UHefSLbTAmThINuJbrqC2iYfCtU=', 'Content-Length': '0'} 2022-09-12T19:07:04.545 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c877ce10c1176062-00631f8358-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.547 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.547 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.548 DEBUG:boto:path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:07:04.548 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:07:04.549 DEBUG:boto:Method: GET 2022-09-12T19:07:04.549 DEBUG:boto:Path: /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:07:04.550 DEBUG:boto:Data: 2022-09-12T19:07:04.550 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.551 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:04.552 DEBUG:boto:Port: 8000 2022-09-12T19:07:04.552 DEBUG:boto:Params: {} 2022-09-12T19:07:04.553 DEBUG:boto:Token: None 2022-09-12T19:07:04.553 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H 2022-09-12T19:07:04.554 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UHefSLbTAmThINuJbrqC2iYfCtU= 2022-09-12T19:07:04.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UHefSLbTAmThINuJbrqC2iYfCtU=', 'Content-Length': '0'} 2022-09-12T19:07:04.557 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'gdUGnE7SHabmaVwE2tf-YcFxl2tBu7H'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a64a058d723fbb29-00631f8358-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.602 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:04.603 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:04.603 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.604 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.604 DEBUG:boto:Method: GET 2022-09-12T19:07:04.605 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:04.605 DEBUG:boto:Data: 2022-09-12T19:07:04.606 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.606 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:04.607 DEBUG:boto:Port: 8001 2022-09-12T19:07:04.608 DEBUG:boto:Params: {} 2022-09-12T19:07:04.608 DEBUG:boto:Token: None 2022-09-12T19:07:04.609 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/ 2022-09-12T19:07:04.610 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs= 2022-09-12T19:07:04.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs=', 'Content-Length': '0'} 2022-09-12T19:07:04.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005ce73b2d2e210b3-00631f8358-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.615 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:04.616 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:04.617 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:04.617 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.618 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.618 DEBUG:boto:Method: GET 2022-09-12T19:07:04.619 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:04.620 DEBUG:boto:Data: 2022-09-12T19:07:04.620 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.621 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:04.622 DEBUG:boto:Port: 8000 2022-09-12T19:07:04.623 DEBUG:boto:Params: {} 2022-09-12T19:07:04.624 DEBUG:boto:Token: None 2022-09-12T19:07:04.625 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/ 2022-09-12T19:07:04.626 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs= 2022-09-12T19:07:04.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs=', 'Content-Length': '0'} 2022-09-12T19:07:04.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf47ba861f31e1d6-00631f8358-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.631 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:04.632 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.633 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.634 DEBUG:boto:Method: HEAD 2022-09-12T19:07:04.634 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:04.635 DEBUG:boto:Data: 2022-09-12T19:07:04.635 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.636 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:04.636 DEBUG:boto:Port: 8001 2022-09-12T19:07:04.637 DEBUG:boto:Params: {} 2022-09-12T19:07:04.638 DEBUG:boto:Token: None 2022-09-12T19:07:04.638 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj 2022-09-12T19:07:04.639 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qNKPFCByVwlzR46AvTHaLiUNsS0= 2022-09-12T19:07:04.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qNKPFCByVwlzR46AvTHaLiUNsS0=', 'Content-Length': '0'} 2022-09-12T19:07:04.643 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000220d2e9496a61aa6-00631f8358-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.643 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.644 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.644 DEBUG:boto:Method: HEAD 2022-09-12T19:07:04.645 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:04.645 DEBUG:boto:Data: 2022-09-12T19:07:04.646 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.646 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:04.647 DEBUG:boto:Port: 8000 2022-09-12T19:07:04.647 DEBUG:boto:Params: {} 2022-09-12T19:07:04.648 DEBUG:boto:Token: None 2022-09-12T19:07:04.649 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj 2022-09-12T19:07:04.649 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qNKPFCByVwlzR46AvTHaLiUNsS0= 2022-09-12T19:07:04.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qNKPFCByVwlzR46AvTHaLiUNsS0=', 'Content-Length': '0'} 2022-09-12T19:07:04.654 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000913f914383b7e88a-00631f8358-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:04.655 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.655 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.656 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.656 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.657 DEBUG:boto:Method: GET 2022-09-12T19:07:04.658 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.658 DEBUG:boto:Data: 2022-09-12T19:07:04.659 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.660 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:04.660 DEBUG:boto:Port: 8001 2022-09-12T19:07:04.661 DEBUG:boto:Params: {} 2022-09-12T19:07:04.661 DEBUG:boto:Token: None 2022-09-12T19:07:04.662 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.663 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mLmGlAHthfxBGCR43Dobd6gJAT4= 2022-09-12T19:07:04.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mLmGlAHthfxBGCR43Dobd6gJAT4=', 'Content-Length': '0'} 2022-09-12T19:07:04.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080cc20992fb206e1-00631f8358-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.706 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.707 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.707 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.708 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.709 DEBUG:boto:Method: GET 2022-09-12T19:07:04.709 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.710 DEBUG:boto:Data: 2022-09-12T19:07:04.710 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.711 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:04.711 DEBUG:boto:Port: 8000 2022-09-12T19:07:04.712 DEBUG:boto:Params: {} 2022-09-12T19:07:04.712 DEBUG:boto:Token: None 2022-09-12T19:07:04.713 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.714 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mLmGlAHthfxBGCR43Dobd6gJAT4= 2022-09-12T19:07:04.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mLmGlAHthfxBGCR43Dobd6gJAT4=', 'Content-Length': '0'} 2022-09-12T19:07:04.717 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027604afaa8348798-00631f8358-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.717 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:04.718 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:04.719 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.719 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.720 DEBUG:boto:Method: GET 2022-09-12T19:07:04.721 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:04.721 DEBUG:boto:Data: 2022-09-12T19:07:04.733 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.734 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:04.735 DEBUG:boto:Port: 8001 2022-09-12T19:07:04.736 DEBUG:boto:Params: {} 2022-09-12T19:07:04.737 DEBUG:boto:Token: None 2022-09-12T19:07:04.737 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/ 2022-09-12T19:07:04.738 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs= 2022-09-12T19:07:04.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs=', 'Content-Length': '0'} 2022-09-12T19:07:04.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009185fc26a0c4f770-00631f8358-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.743 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:04.744 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:04.744 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:04.745 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.745 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.746 DEBUG:boto:Method: GET 2022-09-12T19:07:04.747 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:04.747 DEBUG:boto:Data: 2022-09-12T19:07:04.748 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.748 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:04.749 DEBUG:boto:Port: 8000 2022-09-12T19:07:04.749 DEBUG:boto:Params: {} 2022-09-12T19:07:04.750 DEBUG:boto:Token: None 2022-09-12T19:07:04.750 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/ 2022-09-12T19:07:04.751 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs= 2022-09-12T19:07:04.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs=', 'Content-Length': '0'} 2022-09-12T19:07:04.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c96fcf9df1933cc8-00631f8358-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.755 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:04.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:04.757 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.757 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.758 DEBUG:boto:path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:07:04.759 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:07:04.759 DEBUG:boto:Method: GET 2022-09-12T19:07:04.760 DEBUG:boto:Path: /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:07:04.760 DEBUG:boto:Data: 2022-09-12T19:07:04.761 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.761 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:04.762 DEBUG:boto:Port: 8001 2022-09-12T19:07:04.762 DEBUG:boto:Params: {} 2022-09-12T19:07:04.763 DEBUG:boto:Token: None 2022-09-12T19:07:04.764 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:07:04.764 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:76yQIi8C7fIth4mmwQVVziXkcpc= 2022-09-12T19:07:04.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:76yQIi8C7fIth4mmwQVVziXkcpc=', 'Content-Length': '0'} 2022-09-12T19:07:04.767 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'BAbppx4RylEKnBrYesXt1QPJgsAhIBb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079374147d0627b64-00631f8358-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.768 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.768 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.769 DEBUG:boto:path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:07:04.769 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:07:04.770 DEBUG:boto:Method: GET 2022-09-12T19:07:04.771 DEBUG:boto:Path: /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:07:04.771 DEBUG:boto:Data: 2022-09-12T19:07:04.772 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.772 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:04.773 DEBUG:boto:Port: 8000 2022-09-12T19:07:04.774 DEBUG:boto:Params: {} 2022-09-12T19:07:04.775 DEBUG:boto:Token: None 2022-09-12T19:07:04.776 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:07:04.777 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:76yQIi8C7fIth4mmwQVVziXkcpc= 2022-09-12T19:07:04.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:76yQIi8C7fIth4mmwQVVziXkcpc=', 'Content-Length': '0'} 2022-09-12T19:07:04.780 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'BAbppx4RylEKnBrYesXt1QPJgsAhIBb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060998bf4ff027f26-00631f8358-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.826 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.827 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.827 DEBUG:boto:path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:07:04.828 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:07:04.828 DEBUG:boto:Method: HEAD 2022-09-12T19:07:04.829 DEBUG:boto:Path: /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:07:04.829 DEBUG:boto:Data: 2022-09-12T19:07:04.830 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.830 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:04.831 DEBUG:boto:Port: 8001 2022-09-12T19:07:04.831 DEBUG:boto:Params: {} 2022-09-12T19:07:04.832 DEBUG:boto:Token: None 2022-09-12T19:07:04.833 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:07:04.834 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QDJ89A2PouPAl7uPOWzcH/th3F0= 2022-09-12T19:07:04.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QDJ89A2PouPAl7uPOWzcH/th3F0=', 'Content-Length': '0'} 2022-09-12T19:07:04.837 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'BAbppx4RylEKnBrYesXt1QPJgsAhIBb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000458f152753306037-00631f8358-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.838 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.838 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.839 DEBUG:boto:path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:07:04.839 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:07:04.840 DEBUG:boto:Method: HEAD 2022-09-12T19:07:04.841 DEBUG:boto:Path: /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:07:04.841 DEBUG:boto:Data: 2022-09-12T19:07:04.842 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.842 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:04.843 DEBUG:boto:Port: 8000 2022-09-12T19:07:04.843 DEBUG:boto:Params: {} 2022-09-12T19:07:04.844 DEBUG:boto:Token: None 2022-09-12T19:07:04.845 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:07:04.845 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QDJ89A2PouPAl7uPOWzcH/th3F0= 2022-09-12T19:07:04.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QDJ89A2PouPAl7uPOWzcH/th3F0=', 'Content-Length': '0'} 2022-09-12T19:07:04.848 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'BAbppx4RylEKnBrYesXt1QPJgsAhIBb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000216728e8f1c17243-00631f8358-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:04.850 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.850 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.851 DEBUG:boto:path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:07:04.851 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:07:04.852 DEBUG:boto:Method: GET 2022-09-12T19:07:04.852 DEBUG:boto:Path: /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:07:04.853 DEBUG:boto:Data: 2022-09-12T19:07:04.854 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.854 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:04.855 DEBUG:boto:Port: 8001 2022-09-12T19:07:04.855 DEBUG:boto:Params: {} 2022-09-12T19:07:04.856 DEBUG:boto:Token: None 2022-09-12T19:07:04.857 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:07:04.857 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:76yQIi8C7fIth4mmwQVVziXkcpc= 2022-09-12T19:07:04.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:76yQIi8C7fIth4mmwQVVziXkcpc=', 'Content-Length': '0'} 2022-09-12T19:07:04.860 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'BAbppx4RylEKnBrYesXt1QPJgsAhIBb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d3ade0892a42fd4-00631f8358-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.861 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.862 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.862 DEBUG:boto:path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:07:04.863 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:07:04.863 DEBUG:boto:Method: GET 2022-09-12T19:07:04.864 DEBUG:boto:Path: /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:07:04.865 DEBUG:boto:Data: 2022-09-12T19:07:04.865 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.866 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:04.866 DEBUG:boto:Port: 8000 2022-09-12T19:07:04.867 DEBUG:boto:Params: {} 2022-09-12T19:07:04.867 DEBUG:boto:Token: None 2022-09-12T19:07:04.868 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=BAbppx4RylEKnBrYesXt1QPJgsAhIBb 2022-09-12T19:07:04.869 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:76yQIi8C7fIth4mmwQVVziXkcpc= 2022-09-12T19:07:04.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:76yQIi8C7fIth4mmwQVVziXkcpc=', 'Content-Length': '0'} 2022-09-12T19:07:04.871 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'BAbppx4RylEKnBrYesXt1QPJgsAhIBb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076b4fc896d8d1e40-00631f8358-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.872 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:04.873 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:04.873 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.874 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.874 DEBUG:boto:Method: GET 2022-09-12T19:07:04.875 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:04.876 DEBUG:boto:Data: 2022-09-12T19:07:04.876 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.877 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:04.877 DEBUG:boto:Port: 8001 2022-09-12T19:07:04.878 DEBUG:boto:Params: {} 2022-09-12T19:07:04.878 DEBUG:boto:Token: None 2022-09-12T19:07:04.879 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/ 2022-09-12T19:07:04.880 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs= 2022-09-12T19:07:04.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs=', 'Content-Length': '0'} 2022-09-12T19:07:04.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000721f13b7641456d8-00631f8358-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.884 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:04.885 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:04.885 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:04.886 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.886 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:04.887 DEBUG:boto:Method: GET 2022-09-12T19:07:04.887 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:04.888 DEBUG:boto:Data: 2022-09-12T19:07:04.888 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.889 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:04.889 DEBUG:boto:Port: 8000 2022-09-12T19:07:04.890 DEBUG:boto:Params: {} 2022-09-12T19:07:04.891 DEBUG:boto:Token: None 2022-09-12T19:07:04.891 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/ 2022-09-12T19:07:04.892 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs= 2022-09-12T19:07:04.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vrJ05Wvmfp5COPXEMYfPWVtybxs=', 'Content-Length': '0'} 2022-09-12T19:07:04.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e5df2521a29069e-00631f8358-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.896 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:04.898 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.899 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.900 DEBUG:boto:Method: HEAD 2022-09-12T19:07:04.901 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:04.902 DEBUG:boto:Data: 2022-09-12T19:07:04.903 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.904 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:04.905 DEBUG:boto:Port: 8001 2022-09-12T19:07:04.906 DEBUG:boto:Params: {} 2022-09-12T19:07:04.907 DEBUG:boto:Token: None 2022-09-12T19:07:04.907 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj 2022-09-12T19:07:04.908 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qNKPFCByVwlzR46AvTHaLiUNsS0= 2022-09-12T19:07:04.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qNKPFCByVwlzR46AvTHaLiUNsS0=', 'Content-Length': '0'} 2022-09-12T19:07:04.912 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d0003303ba2cef5a-00631f8358-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.913 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.914 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.914 DEBUG:boto:Method: HEAD 2022-09-12T19:07:04.915 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:04.916 DEBUG:boto:Data: 2022-09-12T19:07:04.917 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.917 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:04.918 DEBUG:boto:Port: 8000 2022-09-12T19:07:04.918 DEBUG:boto:Params: {} 2022-09-12T19:07:04.919 DEBUG:boto:Token: None 2022-09-12T19:07:04.920 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj 2022-09-12T19:07:04.920 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qNKPFCByVwlzR46AvTHaLiUNsS0= 2022-09-12T19:07:04.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qNKPFCByVwlzR46AvTHaLiUNsS0=', 'Content-Length': '0'} 2022-09-12T19:07:04.924 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000813d1e75a2433e92-00631f8358-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:04.926 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.926 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.927 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.928 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.929 DEBUG:boto:Method: GET 2022-09-12T19:07:04.929 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.930 DEBUG:boto:Data: 2022-09-12T19:07:04.930 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.931 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:04.932 DEBUG:boto:Port: 8001 2022-09-12T19:07:04.932 DEBUG:boto:Params: {} 2022-09-12T19:07:04.933 DEBUG:boto:Token: None 2022-09-12T19:07:04.934 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.935 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mLmGlAHthfxBGCR43Dobd6gJAT4= 2022-09-12T19:07:04.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mLmGlAHthfxBGCR43Dobd6gJAT4=', 'Content-Length': '0'} 2022-09-12T19:07:04.938 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000672fb5053d6d9487-00631f8358-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:04.982 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:04.984 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:04.985 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.986 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.987 DEBUG:boto:Method: GET 2022-09-12T19:07:04.988 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.989 DEBUG:boto:Data: 2022-09-12T19:07:04.990 DEBUG:boto:Headers: {} 2022-09-12T19:07:04.991 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:04.992 DEBUG:boto:Port: 8000 2022-09-12T19:07:04.993 DEBUG:boto:Params: {} 2022-09-12T19:07:04.994 DEBUG:boto:Token: None 2022-09-12T19:07:04.996 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:04 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:04.997 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mLmGlAHthfxBGCR43Dobd6gJAT4= 2022-09-12T19:07:04.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mLmGlAHthfxBGCR43Dobd6gJAT4=', 'Content-Length': '0'} 2022-09-12T19:07:05.001 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfabc3c6ccd83e51-00631f8358-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.003 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:05.003 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:05.004 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.005 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.007 DEBUG:boto:Method: GET 2022-09-12T19:07:05.008 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:05.009 DEBUG:boto:Data: 2022-09-12T19:07:05.010 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.011 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.012 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.013 DEBUG:boto:Params: {} 2022-09-12T19:07:05.014 DEBUG:boto:Token: None 2022-09-12T19:07:05.015 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/ 2022-09-12T19:07:05.016 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs= 2022-09-12T19:07:05.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs=', 'Content-Length': '0'} 2022-09-12T19:07:05.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a054fed1d71ba9f-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.022 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:05.024 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:05.025 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:05.026 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.027 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.027 DEBUG:boto:Method: GET 2022-09-12T19:07:05.028 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:05.029 DEBUG:boto:Data: 2022-09-12T19:07:05.030 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.031 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.032 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.033 DEBUG:boto:Params: {} 2022-09-12T19:07:05.033 DEBUG:boto:Token: None 2022-09-12T19:07:05.034 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/ 2022-09-12T19:07:05.035 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs= 2022-09-12T19:07:05.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs=', 'Content-Length': '0'} 2022-09-12T19:07:05.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000448c895bbaead3db-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.039 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:05.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:05.040 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.041 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.041 DEBUG:boto:path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:07:05.042 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:07:05.042 DEBUG:boto:Method: GET 2022-09-12T19:07:05.043 DEBUG:boto:Path: /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:07:05.043 DEBUG:boto:Data: 2022-09-12T19:07:05.044 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.044 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.045 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.046 DEBUG:boto:Params: {} 2022-09-12T19:07:05.047 DEBUG:boto:Token: None 2022-09-12T19:07:05.047 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:07:05.048 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/e6CKZs/anlW10hI68xgkvNtrOk= 2022-09-12T19:07:05.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/e6CKZs/anlW10hI68xgkvNtrOk=', 'Content-Length': '0'} 2022-09-12T19:07:05.051 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001bc666f90e0b8d1d-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.052 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.052 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.053 DEBUG:boto:path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:07:05.053 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:07:05.054 DEBUG:boto:Method: GET 2022-09-12T19:07:05.054 DEBUG:boto:Path: /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:07:05.055 DEBUG:boto:Data: 2022-09-12T19:07:05.055 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.056 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.056 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.057 DEBUG:boto:Params: {} 2022-09-12T19:07:05.058 DEBUG:boto:Token: None 2022-09-12T19:07:05.058 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:07:05.059 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/e6CKZs/anlW10hI68xgkvNtrOk= 2022-09-12T19:07:05.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/e6CKZs/anlW10hI68xgkvNtrOk=', 'Content-Length': '0'} 2022-09-12T19:07:05.061 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4401dc229c64a47-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.106 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.107 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.108 DEBUG:boto:path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:07:05.109 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:07:05.109 DEBUG:boto:Method: HEAD 2022-09-12T19:07:05.110 DEBUG:boto:Path: /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:07:05.111 DEBUG:boto:Data: 2022-09-12T19:07:05.112 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.113 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.114 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.114 DEBUG:boto:Params: {} 2022-09-12T19:07:05.115 DEBUG:boto:Token: None 2022-09-12T19:07:05.116 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:07:05.117 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WLiuxvpM5A8vXyUb3skHN5/WBOI= 2022-09-12T19:07:05.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WLiuxvpM5A8vXyUb3skHN5/WBOI=', 'Content-Length': '0'} 2022-09-12T19:07:05.121 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002302363e40660f20-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.122 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.123 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.123 DEBUG:boto:path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:07:05.124 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:07:05.125 DEBUG:boto:Method: HEAD 2022-09-12T19:07:05.126 DEBUG:boto:Path: /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:07:05.127 DEBUG:boto:Data: 2022-09-12T19:07:05.128 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.129 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.129 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.130 DEBUG:boto:Params: {} 2022-09-12T19:07:05.131 DEBUG:boto:Token: None 2022-09-12T19:07:05.132 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:07:05.133 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WLiuxvpM5A8vXyUb3skHN5/WBOI= 2022-09-12T19:07:05.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WLiuxvpM5A8vXyUb3skHN5/WBOI=', 'Content-Length': '0'} 2022-09-12T19:07:05.136 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f24e5f3d2567894f-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:05.138 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.139 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.140 DEBUG:boto:path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:07:05.141 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:07:05.142 DEBUG:boto:Method: GET 2022-09-12T19:07:05.143 DEBUG:boto:Path: /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:07:05.144 DEBUG:boto:Data: 2022-09-12T19:07:05.144 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.145 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.146 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.147 DEBUG:boto:Params: {} 2022-09-12T19:07:05.148 DEBUG:boto:Token: None 2022-09-12T19:07:05.149 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:07:05.150 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/e6CKZs/anlW10hI68xgkvNtrOk= 2022-09-12T19:07:05.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/e6CKZs/anlW10hI68xgkvNtrOk=', 'Content-Length': '0'} 2022-09-12T19:07:05.153 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004988eab1d1adfdd4-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.154 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.155 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.156 DEBUG:boto:path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:07:05.157 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:07:05.158 DEBUG:boto:Method: GET 2022-09-12T19:07:05.159 DEBUG:boto:Path: /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:07:05.160 DEBUG:boto:Data: 2022-09-12T19:07:05.161 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.161 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.162 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.163 DEBUG:boto:Params: {} 2022-09-12T19:07:05.164 DEBUG:boto:Token: None 2022-09-12T19:07:05.165 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2 2022-09-12T19:07:05.165 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/e6CKZs/anlW10hI68xgkvNtrOk= 2022-09-12T19:07:05.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/e6CKZs/anlW10hI68xgkvNtrOk=', 'Content-Length': '0'} 2022-09-12T19:07:05.168 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'NDM-EGOTc7UZbg9YQZwcM2uEk6P03H2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4d7887371234229-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.169 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:05.170 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:05.170 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.171 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.172 DEBUG:boto:Method: GET 2022-09-12T19:07:05.172 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:05.173 DEBUG:boto:Data: 2022-09-12T19:07:05.174 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.174 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.175 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.176 DEBUG:boto:Params: {} 2022-09-12T19:07:05.176 DEBUG:boto:Token: None 2022-09-12T19:07:05.177 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/ 2022-09-12T19:07:05.177 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs= 2022-09-12T19:07:05.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs=', 'Content-Length': '0'} 2022-09-12T19:07:05.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd1e2b321ad729de-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.182 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:05.183 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:05.183 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:05.184 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.184 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.185 DEBUG:boto:Method: GET 2022-09-12T19:07:05.185 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:05.186 DEBUG:boto:Data: 2022-09-12T19:07:05.186 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.187 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.187 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.188 DEBUG:boto:Params: {} 2022-09-12T19:07:05.189 DEBUG:boto:Token: None 2022-09-12T19:07:05.189 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/ 2022-09-12T19:07:05.190 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs= 2022-09-12T19:07:05.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs=', 'Content-Length': '0'} 2022-09-12T19:07:05.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062e87b038e280735-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.194 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:05.195 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.195 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.196 DEBUG:boto:Method: HEAD 2022-09-12T19:07:05.196 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:05.197 DEBUG:boto:Data: 2022-09-12T19:07:05.197 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.198 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.198 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.199 DEBUG:boto:Params: {} 2022-09-12T19:07:05.199 DEBUG:boto:Token: None 2022-09-12T19:07:05.200 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj 2022-09-12T19:07:05.200 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z24c/C19g/dBP2LgbgsuEUvugx0= 2022-09-12T19:07:05.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z24c/C19g/dBP2LgbgsuEUvugx0=', 'Content-Length': '0'} 2022-09-12T19:07:05.204 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000397632d7982fb707-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.205 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.206 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.206 DEBUG:boto:Method: HEAD 2022-09-12T19:07:05.207 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:05.207 DEBUG:boto:Data: 2022-09-12T19:07:05.208 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.208 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.209 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.209 DEBUG:boto:Params: {} 2022-09-12T19:07:05.210 DEBUG:boto:Token: None 2022-09-12T19:07:05.211 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj 2022-09-12T19:07:05.212 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z24c/C19g/dBP2LgbgsuEUvugx0= 2022-09-12T19:07:05.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z24c/C19g/dBP2LgbgsuEUvugx0=', 'Content-Length': '0'} 2022-09-12T19:07:05.216 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d71c0e2b89857a8-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:05.217 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.218 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.219 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:05.219 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:05.220 DEBUG:boto:Method: GET 2022-09-12T19:07:05.220 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:05.221 DEBUG:boto:Data: 2022-09-12T19:07:05.221 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.222 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.222 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.223 DEBUG:boto:Params: {} 2022-09-12T19:07:05.223 DEBUG:boto:Token: None 2022-09-12T19:07:05.224 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:05.225 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:r09ZJ+QTgMsAOIq3ZtDPqOzjQug= 2022-09-12T19:07:05.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:r09ZJ+QTgMsAOIq3ZtDPqOzjQug=', 'Content-Length': '0'} 2022-09-12T19:07:05.228 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd89b97763bb265c-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.228 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.229 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.229 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:05.230 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:05.231 DEBUG:boto:Method: GET 2022-09-12T19:07:05.231 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:05.232 DEBUG:boto:Data: 2022-09-12T19:07:05.232 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.233 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.233 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.234 DEBUG:boto:Params: {} 2022-09-12T19:07:05.234 DEBUG:boto:Token: None 2022-09-12T19:07:05.235 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:05.236 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:r09ZJ+QTgMsAOIq3ZtDPqOzjQug= 2022-09-12T19:07:05.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:r09ZJ+QTgMsAOIq3ZtDPqOzjQug=', 'Content-Length': '0'} 2022-09-12T19:07:05.238 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000058aea242ff861bd-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.239 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:05.240 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:05.241 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.242 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.243 DEBUG:boto:Method: GET 2022-09-12T19:07:05.244 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:05.245 DEBUG:boto:Data: 2022-09-12T19:07:05.246 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.247 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.248 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.248 DEBUG:boto:Params: {} 2022-09-12T19:07:05.249 DEBUG:boto:Token: None 2022-09-12T19:07:05.250 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/ 2022-09-12T19:07:05.251 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs= 2022-09-12T19:07:05.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs=', 'Content-Length': '0'} 2022-09-12T19:07:05.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019e85e07654960e4-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.255 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:05.256 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:05.257 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:05.257 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.258 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.258 DEBUG:boto:Method: GET 2022-09-12T19:07:05.259 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:05.260 DEBUG:boto:Data: 2022-09-12T19:07:05.260 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.261 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.261 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.262 DEBUG:boto:Params: {} 2022-09-12T19:07:05.263 DEBUG:boto:Token: None 2022-09-12T19:07:05.263 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/ 2022-09-12T19:07:05.264 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs= 2022-09-12T19:07:05.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs=', 'Content-Length': '0'} 2022-09-12T19:07:05.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4b2a0c903ec0b3a-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.310 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:05.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:05.312 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.313 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.314 DEBUG:boto:path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:07:05.315 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:07:05.316 DEBUG:boto:Method: GET 2022-09-12T19:07:05.317 DEBUG:boto:Path: /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:07:05.318 DEBUG:boto:Data: 2022-09-12T19:07:05.319 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.320 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.321 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.322 DEBUG:boto:Params: {} 2022-09-12T19:07:05.323 DEBUG:boto:Token: None 2022-09-12T19:07:05.324 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:07:05.326 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nf/OBzQKBcvEuOcuV/kM7rTzSQM= 2022-09-12T19:07:05.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nf/OBzQKBcvEuOcuV/kM7rTzSQM=', 'Content-Length': '0'} 2022-09-12T19:07:05.330 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'e.6ruI53roqy4mmyPgzQ1kqMdajruFA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006203afe4b8147254-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.331 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.332 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.333 DEBUG:boto:path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:07:05.334 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:07:05.335 DEBUG:boto:Method: GET 2022-09-12T19:07:05.336 DEBUG:boto:Path: /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:07:05.337 DEBUG:boto:Data: 2022-09-12T19:07:05.338 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.339 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.340 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.341 DEBUG:boto:Params: {} 2022-09-12T19:07:05.342 DEBUG:boto:Token: None 2022-09-12T19:07:05.344 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:07:05.345 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nf/OBzQKBcvEuOcuV/kM7rTzSQM= 2022-09-12T19:07:05.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nf/OBzQKBcvEuOcuV/kM7rTzSQM=', 'Content-Length': '0'} 2022-09-12T19:07:05.349 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'e.6ruI53roqy4mmyPgzQ1kqMdajruFA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000496b8db8c29db2bc-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.350 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.351 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.352 DEBUG:boto:path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:07:05.353 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:07:05.354 DEBUG:boto:Method: HEAD 2022-09-12T19:07:05.355 DEBUG:boto:Path: /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:07:05.356 DEBUG:boto:Data: 2022-09-12T19:07:05.357 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.358 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.359 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.360 DEBUG:boto:Params: {} 2022-09-12T19:07:05.361 DEBUG:boto:Token: None 2022-09-12T19:07:05.362 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:07:05.363 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aVHkTEuBRUFf5Y7LxTVq+OXGOz0= 2022-09-12T19:07:05.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aVHkTEuBRUFf5Y7LxTVq+OXGOz0=', 'Content-Length': '0'} 2022-09-12T19:07:05.367 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'e.6ruI53roqy4mmyPgzQ1kqMdajruFA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006bf0537bae320b28-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.368 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.369 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.370 DEBUG:boto:path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:07:05.371 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:07:05.372 DEBUG:boto:Method: HEAD 2022-09-12T19:07:05.373 DEBUG:boto:Path: /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:07:05.374 DEBUG:boto:Data: 2022-09-12T19:07:05.375 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.376 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.377 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.378 DEBUG:boto:Params: {} 2022-09-12T19:07:05.379 DEBUG:boto:Token: None 2022-09-12T19:07:05.379 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:07:05.380 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aVHkTEuBRUFf5Y7LxTVq+OXGOz0= 2022-09-12T19:07:05.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aVHkTEuBRUFf5Y7LxTVq+OXGOz0=', 'Content-Length': '0'} 2022-09-12T19:07:05.384 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'e.6ruI53roqy4mmyPgzQ1kqMdajruFA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000902b461351609b8f-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:05.385 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.386 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.386 DEBUG:boto:path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:07:05.387 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:07:05.387 DEBUG:boto:Method: GET 2022-09-12T19:07:05.388 DEBUG:boto:Path: /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:07:05.389 DEBUG:boto:Data: 2022-09-12T19:07:05.389 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.390 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.390 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.391 DEBUG:boto:Params: {} 2022-09-12T19:07:05.391 DEBUG:boto:Token: None 2022-09-12T19:07:05.392 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:07:05.393 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nf/OBzQKBcvEuOcuV/kM7rTzSQM= 2022-09-12T19:07:05.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nf/OBzQKBcvEuOcuV/kM7rTzSQM=', 'Content-Length': '0'} 2022-09-12T19:07:05.395 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'e.6ruI53roqy4mmyPgzQ1kqMdajruFA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000083f1009176f6f170-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.442 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.443 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.444 DEBUG:boto:path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:07:05.445 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:07:05.446 DEBUG:boto:Method: GET 2022-09-12T19:07:05.447 DEBUG:boto:Path: /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:07:05.448 DEBUG:boto:Data: 2022-09-12T19:07:05.449 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.450 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.451 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.452 DEBUG:boto:Params: {} 2022-09-12T19:07:05.453 DEBUG:boto:Token: None 2022-09-12T19:07:05.454 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=e.6ruI53roqy4mmyPgzQ1kqMdajruFA 2022-09-12T19:07:05.455 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nf/OBzQKBcvEuOcuV/kM7rTzSQM= 2022-09-12T19:07:05.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nf/OBzQKBcvEuOcuV/kM7rTzSQM=', 'Content-Length': '0'} 2022-09-12T19:07:05.459 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'e.6ruI53roqy4mmyPgzQ1kqMdajruFA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c34968d203df1b81-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.460 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:05.460 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:05.461 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.462 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.463 DEBUG:boto:Method: GET 2022-09-12T19:07:05.464 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:05.465 DEBUG:boto:Data: 2022-09-12T19:07:05.466 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.467 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.468 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.469 DEBUG:boto:Params: {} 2022-09-12T19:07:05.470 DEBUG:boto:Token: None 2022-09-12T19:07:05.471 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/ 2022-09-12T19:07:05.472 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs= 2022-09-12T19:07:05.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs=', 'Content-Length': '0'} 2022-09-12T19:07:05.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3a9add143013889-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.476 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:05.477 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:05.478 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:05.478 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.479 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.479 DEBUG:boto:Method: GET 2022-09-12T19:07:05.480 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:05.480 DEBUG:boto:Data: 2022-09-12T19:07:05.481 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.482 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.482 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.483 DEBUG:boto:Params: {} 2022-09-12T19:07:05.483 DEBUG:boto:Token: None 2022-09-12T19:07:05.484 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/ 2022-09-12T19:07:05.485 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs= 2022-09-12T19:07:05.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs=', 'Content-Length': '0'} 2022-09-12T19:07:05.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049e44bbdaec12faa-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.489 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:05.490 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.491 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.492 DEBUG:boto:Method: HEAD 2022-09-12T19:07:05.493 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:05.494 DEBUG:boto:Data: 2022-09-12T19:07:05.495 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.496 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.497 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.498 DEBUG:boto:Params: {} 2022-09-12T19:07:05.499 DEBUG:boto:Token: None 2022-09-12T19:07:05.499 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj 2022-09-12T19:07:05.500 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z24c/C19g/dBP2LgbgsuEUvugx0= 2022-09-12T19:07:05.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z24c/C19g/dBP2LgbgsuEUvugx0=', 'Content-Length': '0'} 2022-09-12T19:07:05.505 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000349f1b3120e8a86f-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.506 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.506 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.507 DEBUG:boto:Method: HEAD 2022-09-12T19:07:05.508 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:05.509 DEBUG:boto:Data: 2022-09-12T19:07:05.510 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.511 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.512 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.513 DEBUG:boto:Params: {} 2022-09-12T19:07:05.514 DEBUG:boto:Token: None 2022-09-12T19:07:05.515 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj 2022-09-12T19:07:05.517 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z24c/C19g/dBP2LgbgsuEUvugx0= 2022-09-12T19:07:05.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z24c/C19g/dBP2LgbgsuEUvugx0=', 'Content-Length': '0'} 2022-09-12T19:07:05.522 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d83f1700cb059797-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:05.524 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.525 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.526 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:05.527 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:05.529 DEBUG:boto:Method: GET 2022-09-12T19:07:05.530 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:05.531 DEBUG:boto:Data: 2022-09-12T19:07:05.532 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.533 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.534 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.535 DEBUG:boto:Params: {} 2022-09-12T19:07:05.536 DEBUG:boto:Token: None 2022-09-12T19:07:05.537 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:05.538 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:r09ZJ+QTgMsAOIq3ZtDPqOzjQug= 2022-09-12T19:07:05.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:r09ZJ+QTgMsAOIq3ZtDPqOzjQug=', 'Content-Length': '0'} 2022-09-12T19:07:05.542 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002563d30063d4a7a5-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.544 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.545 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.546 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:05.547 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:05.548 DEBUG:boto:Method: GET 2022-09-12T19:07:05.549 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:05.550 DEBUG:boto:Data: 2022-09-12T19:07:05.551 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.552 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.553 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.554 DEBUG:boto:Params: {} 2022-09-12T19:07:05.556 DEBUG:boto:Token: None 2022-09-12T19:07:05.557 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:05.558 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:r09ZJ+QTgMsAOIq3ZtDPqOzjQug= 2022-09-12T19:07:05.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:r09ZJ+QTgMsAOIq3ZtDPqOzjQug=', 'Content-Length': '0'} 2022-09-12T19:07:05.561 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078e8c4878fdbe9ee-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.563 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:05.564 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:05.565 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.565 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.566 DEBUG:boto:Method: GET 2022-09-12T19:07:05.567 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:05.567 DEBUG:boto:Data: 2022-09-12T19:07:05.568 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.568 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.569 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.569 DEBUG:boto:Params: {} 2022-09-12T19:07:05.570 DEBUG:boto:Token: None 2022-09-12T19:07:05.571 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/ 2022-09-12T19:07:05.571 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs= 2022-09-12T19:07:05.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs=', 'Content-Length': '0'} 2022-09-12T19:07:05.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f5e87df50dbfd1d-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.618 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:05.619 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:05.620 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:05.620 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.621 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.621 DEBUG:boto:Method: GET 2022-09-12T19:07:05.622 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:05.623 DEBUG:boto:Data: 2022-09-12T19:07:05.623 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.624 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.624 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.625 DEBUG:boto:Params: {} 2022-09-12T19:07:05.625 DEBUG:boto:Token: None 2022-09-12T19:07:05.626 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/ 2022-09-12T19:07:05.627 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs= 2022-09-12T19:07:05.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs=', 'Content-Length': '0'} 2022-09-12T19:07:05.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f836c58246eaaff6-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.631 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:05.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:05.632 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.633 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.633 DEBUG:boto:path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:07:05.634 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:07:05.634 DEBUG:boto:Method: GET 2022-09-12T19:07:05.635 DEBUG:boto:Path: /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:07:05.635 DEBUG:boto:Data: 2022-09-12T19:07:05.636 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.637 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.637 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.638 DEBUG:boto:Params: {} 2022-09-12T19:07:05.638 DEBUG:boto:Token: None 2022-09-12T19:07:05.639 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:07:05.640 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6v4M5Ff0oIOvrLllfU7WXYpdh3c= 2022-09-12T19:07:05.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6v4M5Ff0oIOvrLllfU7WXYpdh3c=', 'Content-Length': '0'} 2022-09-12T19:07:05.643 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047c46a1dc115153c-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.644 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.644 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.645 DEBUG:boto:path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:07:05.645 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:07:05.646 DEBUG:boto:Method: GET 2022-09-12T19:07:05.646 DEBUG:boto:Path: /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:07:05.647 DEBUG:boto:Data: 2022-09-12T19:07:05.647 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.648 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.649 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.649 DEBUG:boto:Params: {} 2022-09-12T19:07:05.650 DEBUG:boto:Token: None 2022-09-12T19:07:05.650 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:07:05.651 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6v4M5Ff0oIOvrLllfU7WXYpdh3c= 2022-09-12T19:07:05.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6v4M5Ff0oIOvrLllfU7WXYpdh3c=', 'Content-Length': '0'} 2022-09-12T19:07:05.654 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042df19e7fec08040-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.698 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.699 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.700 DEBUG:boto:path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:07:05.700 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:07:05.701 DEBUG:boto:Method: HEAD 2022-09-12T19:07:05.701 DEBUG:boto:Path: /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:07:05.702 DEBUG:boto:Data: 2022-09-12T19:07:05.702 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.703 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.703 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.704 DEBUG:boto:Params: {} 2022-09-12T19:07:05.704 DEBUG:boto:Token: None 2022-09-12T19:07:05.705 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:07:05.706 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:63xJsY5RQryDGmxFHVRQa/hHOo4= 2022-09-12T19:07:05.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:63xJsY5RQryDGmxFHVRQa/hHOo4=', 'Content-Length': '0'} 2022-09-12T19:07:05.708 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000477c22b02fa366cb-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.709 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.710 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.710 DEBUG:boto:path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:07:05.711 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:07:05.712 DEBUG:boto:Method: HEAD 2022-09-12T19:07:05.712 DEBUG:boto:Path: /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:07:05.713 DEBUG:boto:Data: 2022-09-12T19:07:05.713 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.714 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.714 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.715 DEBUG:boto:Params: {} 2022-09-12T19:07:05.716 DEBUG:boto:Token: None 2022-09-12T19:07:05.716 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:07:05.717 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:63xJsY5RQryDGmxFHVRQa/hHOo4= 2022-09-12T19:07:05.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:63xJsY5RQryDGmxFHVRQa/hHOo4=', 'Content-Length': '0'} 2022-09-12T19:07:05.720 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a0e31694b64a7f7-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:05.721 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.722 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.722 DEBUG:boto:path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:07:05.723 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:07:05.723 DEBUG:boto:Method: GET 2022-09-12T19:07:05.724 DEBUG:boto:Path: /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:07:05.724 DEBUG:boto:Data: 2022-09-12T19:07:05.725 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.725 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.726 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.726 DEBUG:boto:Params: {} 2022-09-12T19:07:05.727 DEBUG:boto:Token: None 2022-09-12T19:07:05.728 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:07:05.728 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6v4M5Ff0oIOvrLllfU7WXYpdh3c= 2022-09-12T19:07:05.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6v4M5Ff0oIOvrLllfU7WXYpdh3c=', 'Content-Length': '0'} 2022-09-12T19:07:05.731 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000583b8e4ffb0b7439-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.732 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.733 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.733 DEBUG:boto:path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:07:05.734 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:07:05.734 DEBUG:boto:Method: GET 2022-09-12T19:07:05.735 DEBUG:boto:Path: /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:07:05.735 DEBUG:boto:Data: 2022-09-12T19:07:05.736 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.736 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.737 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.737 DEBUG:boto:Params: {} 2022-09-12T19:07:05.738 DEBUG:boto:Token: None 2022-09-12T19:07:05.739 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE 2022-09-12T19:07:05.739 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6v4M5Ff0oIOvrLllfU7WXYpdh3c= 2022-09-12T19:07:05.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6v4M5Ff0oIOvrLllfU7WXYpdh3c=', 'Content-Length': '0'} 2022-09-12T19:07:05.742 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'LIaoO70PR8fRbTLQ-5Q.roWjA0lswqE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000973045a4e799f751-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.743 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:05.744 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:05.744 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.745 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.745 DEBUG:boto:Method: GET 2022-09-12T19:07:05.746 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:05.747 DEBUG:boto:Data: 2022-09-12T19:07:05.747 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.748 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.748 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.749 DEBUG:boto:Params: {} 2022-09-12T19:07:05.749 DEBUG:boto:Token: None 2022-09-12T19:07:05.750 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/ 2022-09-12T19:07:05.751 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs= 2022-09-12T19:07:05.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs=', 'Content-Length': '0'} 2022-09-12T19:07:05.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000908e3f20136dff11-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.755 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:05.756 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:05.756 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:05.757 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.757 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.758 DEBUG:boto:Method: GET 2022-09-12T19:07:05.759 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:05.760 DEBUG:boto:Data: 2022-09-12T19:07:05.760 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.761 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.761 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.762 DEBUG:boto:Params: {} 2022-09-12T19:07:05.762 DEBUG:boto:Token: None 2022-09-12T19:07:05.763 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/ 2022-09-12T19:07:05.764 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs= 2022-09-12T19:07:05.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs=', 'Content-Length': '0'} 2022-09-12T19:07:05.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c87533dceb6e72fd-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.768 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:05.768 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.769 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.770 DEBUG:boto:Method: HEAD 2022-09-12T19:07:05.770 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:05.771 DEBUG:boto:Data: 2022-09-12T19:07:05.771 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.772 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.772 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.773 DEBUG:boto:Params: {} 2022-09-12T19:07:05.773 DEBUG:boto:Token: None 2022-09-12T19:07:05.774 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj 2022-09-12T19:07:05.775 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z24c/C19g/dBP2LgbgsuEUvugx0= 2022-09-12T19:07:05.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z24c/C19g/dBP2LgbgsuEUvugx0=', 'Content-Length': '0'} 2022-09-12T19:07:05.779 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d45589d3a6447bec-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.779 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.780 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.781 DEBUG:boto:Method: HEAD 2022-09-12T19:07:05.781 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:05.782 DEBUG:boto:Data: 2022-09-12T19:07:05.782 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.783 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.783 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.784 DEBUG:boto:Params: {} 2022-09-12T19:07:05.784 DEBUG:boto:Token: None 2022-09-12T19:07:05.785 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj 2022-09-12T19:07:05.785 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z24c/C19g/dBP2LgbgsuEUvugx0= 2022-09-12T19:07:05.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z24c/C19g/dBP2LgbgsuEUvugx0=', 'Content-Length': '0'} 2022-09-12T19:07:05.789 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062b7b35d89e45f07-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:05.791 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.791 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.792 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:05.792 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:05.793 DEBUG:boto:Method: GET 2022-09-12T19:07:05.793 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:05.794 DEBUG:boto:Data: 2022-09-12T19:07:05.795 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.795 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.796 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.796 DEBUG:boto:Params: {} 2022-09-12T19:07:05.797 DEBUG:boto:Token: None 2022-09-12T19:07:05.798 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:05.798 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:r09ZJ+QTgMsAOIq3ZtDPqOzjQug= 2022-09-12T19:07:05.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:r09ZJ+QTgMsAOIq3ZtDPqOzjQug=', 'Content-Length': '0'} 2022-09-12T19:07:05.801 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015690f5ff33ce95c-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.842 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.843 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.844 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:05.845 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:05.846 DEBUG:boto:Method: GET 2022-09-12T19:07:05.847 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:05.848 DEBUG:boto:Data: 2022-09-12T19:07:05.849 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.850 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.851 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.852 DEBUG:boto:Params: {} 2022-09-12T19:07:05.852 DEBUG:boto:Token: None 2022-09-12T19:07:05.853 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:05.854 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:r09ZJ+QTgMsAOIq3ZtDPqOzjQug= 2022-09-12T19:07:05.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:r09ZJ+QTgMsAOIq3ZtDPqOzjQug=', 'Content-Length': '0'} 2022-09-12T19:07:05.857 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f78eec29b22d9235-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.858 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:05.859 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:05.859 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.860 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.861 DEBUG:boto:Method: GET 2022-09-12T19:07:05.862 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:05.863 DEBUG:boto:Data: 2022-09-12T19:07:05.864 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.864 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.865 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.866 DEBUG:boto:Params: {} 2022-09-12T19:07:05.867 DEBUG:boto:Token: None 2022-09-12T19:07:05.869 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/ 2022-09-12T19:07:05.870 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs= 2022-09-12T19:07:05.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs=', 'Content-Length': '0'} 2022-09-12T19:07:05.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007510b70a648108db-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.875 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:05.876 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:05.876 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:05.877 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.877 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.878 DEBUG:boto:Method: GET 2022-09-12T19:07:05.880 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:05.881 DEBUG:boto:Data: 2022-09-12T19:07:05.882 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.883 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.884 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.885 DEBUG:boto:Params: {} 2022-09-12T19:07:05.885 DEBUG:boto:Token: None 2022-09-12T19:07:05.886 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/ 2022-09-12T19:07:05.888 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs= 2022-09-12T19:07:05.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs=', 'Content-Length': '0'} 2022-09-12T19:07:05.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005837d991947bb9a4-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.892 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:05.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:05.893 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.894 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.894 DEBUG:boto:path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:07:05.895 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:07:05.895 DEBUG:boto:Method: GET 2022-09-12T19:07:05.896 DEBUG:boto:Path: /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:07:05.896 DEBUG:boto:Data: 2022-09-12T19:07:05.897 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.898 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.898 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.899 DEBUG:boto:Params: {} 2022-09-12T19:07:05.900 DEBUG:boto:Token: None 2022-09-12T19:07:05.900 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:07:05.901 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1dnxyPYGbAXX9diNgq6WBl83k1w= 2022-09-12T19:07:05.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1dnxyPYGbAXX9diNgq6WBl83k1w=', 'Content-Length': '0'} 2022-09-12T19:07:05.904 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'lnpiHoSs91QVujLSWqHhvEGzZrF3-sA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000031eb0829b321212-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.905 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.906 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.907 DEBUG:boto:path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:07:05.907 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:07:05.908 DEBUG:boto:Method: GET 2022-09-12T19:07:05.908 DEBUG:boto:Path: /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:07:05.909 DEBUG:boto:Data: 2022-09-12T19:07:05.910 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.910 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.911 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.911 DEBUG:boto:Params: {} 2022-09-12T19:07:05.912 DEBUG:boto:Token: None 2022-09-12T19:07:05.913 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:07:05.913 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1dnxyPYGbAXX9diNgq6WBl83k1w= 2022-09-12T19:07:05.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1dnxyPYGbAXX9diNgq6WBl83k1w=', 'Content-Length': '0'} 2022-09-12T19:07:05.916 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'lnpiHoSs91QVujLSWqHhvEGzZrF3-sA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f50670e91fa870c-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.917 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.918 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.918 DEBUG:boto:path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:07:05.919 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:07:05.919 DEBUG:boto:Method: HEAD 2022-09-12T19:07:05.920 DEBUG:boto:Path: /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:07:05.920 DEBUG:boto:Data: 2022-09-12T19:07:05.921 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.922 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.922 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.923 DEBUG:boto:Params: {} 2022-09-12T19:07:05.924 DEBUG:boto:Token: None 2022-09-12T19:07:05.924 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:07:05.925 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DA/lrOwzDrUla71ENN0TZ41ePLA= 2022-09-12T19:07:05.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DA/lrOwzDrUla71ENN0TZ41ePLA=', 'Content-Length': '0'} 2022-09-12T19:07:05.928 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'lnpiHoSs91QVujLSWqHhvEGzZrF3-sA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbeb4c623a1e5083-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.928 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.929 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.929 DEBUG:boto:path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:07:05.930 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:07:05.931 DEBUG:boto:Method: HEAD 2022-09-12T19:07:05.931 DEBUG:boto:Path: /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:07:05.932 DEBUG:boto:Data: 2022-09-12T19:07:05.932 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.933 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.933 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.934 DEBUG:boto:Params: {} 2022-09-12T19:07:05.935 DEBUG:boto:Token: None 2022-09-12T19:07:05.935 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:07:05.936 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DA/lrOwzDrUla71ENN0TZ41ePLA= 2022-09-12T19:07:05.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DA/lrOwzDrUla71ENN0TZ41ePLA=', 'Content-Length': '0'} 2022-09-12T19:07:05.939 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'lnpiHoSs91QVujLSWqHhvEGzZrF3-sA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000839ff33265b13501-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:05.941 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.942 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.943 DEBUG:boto:path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:07:05.944 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:07:05.945 DEBUG:boto:Method: GET 2022-09-12T19:07:05.947 DEBUG:boto:Path: /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:07:05.948 DEBUG:boto:Data: 2022-09-12T19:07:05.949 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.950 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.951 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.952 DEBUG:boto:Params: {} 2022-09-12T19:07:05.953 DEBUG:boto:Token: None 2022-09-12T19:07:05.954 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:07:05.954 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1dnxyPYGbAXX9diNgq6WBl83k1w= 2022-09-12T19:07:05.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1dnxyPYGbAXX9diNgq6WBl83k1w=', 'Content-Length': '0'} 2022-09-12T19:07:05.957 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'lnpiHoSs91QVujLSWqHhvEGzZrF3-sA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2dbce0c2dfd7eb6-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.958 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:05.959 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:05.959 DEBUG:boto:path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:07:05.960 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:07:05.960 DEBUG:boto:Method: GET 2022-09-12T19:07:05.961 DEBUG:boto:Path: /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:07:05.961 DEBUG:boto:Data: 2022-09-12T19:07:05.962 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.963 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.963 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.964 DEBUG:boto:Params: {} 2022-09-12T19:07:05.965 DEBUG:boto:Token: None 2022-09-12T19:07:05.966 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/obj?versionId=lnpiHoSs91QVujLSWqHhvEGzZrF3-sA 2022-09-12T19:07:05.968 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1dnxyPYGbAXX9diNgq6WBl83k1w= 2022-09-12T19:07:05.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1dnxyPYGbAXX9diNgq6WBl83k1w=', 'Content-Length': '0'} 2022-09-12T19:07:05.971 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'lnpiHoSs91QVujLSWqHhvEGzZrF3-sA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000067c36956fd69004-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.972 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:05.973 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:05.974 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.974 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.975 DEBUG:boto:Method: GET 2022-09-12T19:07:05.975 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:05.976 DEBUG:boto:Data: 2022-09-12T19:07:05.976 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.977 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:05.978 DEBUG:boto:Port: 8001 2022-09-12T19:07:05.978 DEBUG:boto:Params: {} 2022-09-12T19:07:05.979 DEBUG:boto:Token: None 2022-09-12T19:07:05.980 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/ 2022-09-12T19:07:05.980 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs= 2022-09-12T19:07:05.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs=', 'Content-Length': '0'} 2022-09-12T19:07:05.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a88de168d35e5bc9-00631f8359-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:05.985 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:05.986 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:05.987 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:05.987 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.988 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:05.989 DEBUG:boto:Method: GET 2022-09-12T19:07:05.989 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:05.990 DEBUG:boto:Data: 2022-09-12T19:07:05.991 DEBUG:boto:Headers: {} 2022-09-12T19:07:05.991 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:05.992 DEBUG:boto:Port: 8000 2022-09-12T19:07:05.992 DEBUG:boto:Params: {} 2022-09-12T19:07:05.993 DEBUG:boto:Token: None 2022-09-12T19:07:05.994 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:05 GMT /jdpiak-19/ 2022-09-12T19:07:05.994 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs= 2022-09-12T19:07:05.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uCeMBKuocK5qou1x8yukFgG6yFs=', 'Content-Length': '0'} 2022-09-12T19:07:05.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fab5a2ed6b08de8-00631f8359-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.042 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:06.043 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.044 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.045 DEBUG:boto:Method: HEAD 2022-09-12T19:07:06.045 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:06.046 DEBUG:boto:Data: 2022-09-12T19:07:06.047 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.048 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:06.049 DEBUG:boto:Port: 8001 2022-09-12T19:07:06.050 DEBUG:boto:Params: {} 2022-09-12T19:07:06.051 DEBUG:boto:Token: None 2022-09-12T19:07:06.052 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj 2022-09-12T19:07:06.052 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ts8+S2ZH+pG9pDjQXe4nrVd5KOU= 2022-09-12T19:07:06.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ts8+S2ZH+pG9pDjQXe4nrVd5KOU=', 'Content-Length': '0'} 2022-09-12T19:07:06.057 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c113890dcecadcef-00631f835a-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.058 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.058 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.059 DEBUG:boto:Method: HEAD 2022-09-12T19:07:06.059 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:06.060 DEBUG:boto:Data: 2022-09-12T19:07:06.060 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.061 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:06.061 DEBUG:boto:Port: 8000 2022-09-12T19:07:06.062 DEBUG:boto:Params: {} 2022-09-12T19:07:06.063 DEBUG:boto:Token: None 2022-09-12T19:07:06.063 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj 2022-09-12T19:07:06.064 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ts8+S2ZH+pG9pDjQXe4nrVd5KOU= 2022-09-12T19:07:06.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ts8+S2ZH+pG9pDjQXe4nrVd5KOU=', 'Content-Length': '0'} 2022-09-12T19:07:06.068 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006ece528b9be3ff5-00631f835a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:06.070 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.070 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.071 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:06.071 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:06.072 DEBUG:boto:Method: GET 2022-09-12T19:07:06.073 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:06.073 DEBUG:boto:Data: 2022-09-12T19:07:06.074 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.074 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:06.075 DEBUG:boto:Port: 8001 2022-09-12T19:07:06.076 DEBUG:boto:Params: {} 2022-09-12T19:07:06.077 DEBUG:boto:Token: None 2022-09-12T19:07:06.077 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:06.078 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HSZKt/4lzWNVTh04unKX5gKFmgc= 2022-09-12T19:07:06.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HSZKt/4lzWNVTh04unKX5gKFmgc=', 'Content-Length': '0'} 2022-09-12T19:07:06.082 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4cad0c30b77ee98-00631f835a-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.082 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.083 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.083 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:06.084 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:06.085 DEBUG:boto:Method: GET 2022-09-12T19:07:06.085 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:06.086 DEBUG:boto:Data: 2022-09-12T19:07:06.086 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.087 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:06.088 DEBUG:boto:Port: 8000 2022-09-12T19:07:06.089 DEBUG:boto:Params: {} 2022-09-12T19:07:06.090 DEBUG:boto:Token: None 2022-09-12T19:07:06.091 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:06.092 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HSZKt/4lzWNVTh04unKX5gKFmgc= 2022-09-12T19:07:06.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HSZKt/4lzWNVTh04unKX5gKFmgc=', 'Content-Length': '0'} 2022-09-12T19:07:06.094 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064c497bf3a7c49d0-00631f835a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.095 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:06.096 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:06.096 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:06.097 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:06.098 DEBUG:boto:Method: GET 2022-09-12T19:07:06.098 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:06.099 DEBUG:boto:Data: 2022-09-12T19:07:06.100 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.101 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:06.102 DEBUG:boto:Port: 8001 2022-09-12T19:07:06.103 DEBUG:boto:Params: {} 2022-09-12T19:07:06.104 DEBUG:boto:Token: None 2022-09-12T19:07:06.106 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/ 2022-09-12T19:07:06.107 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0FR+DbCBV8STjgUD2ywMnjTfF2E= 2022-09-12T19:07:06.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0FR+DbCBV8STjgUD2ywMnjTfF2E=', 'Content-Length': '0'} 2022-09-12T19:07:06.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb44cdf7d74c3ef7-00631f835a-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.113 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:06.114 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:06.115 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:06.116 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:06.117 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:06.118 DEBUG:boto:Method: GET 2022-09-12T19:07:06.119 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:06.120 DEBUG:boto:Data: 2022-09-12T19:07:06.121 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.122 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:06.123 DEBUG:boto:Port: 8000 2022-09-12T19:07:06.123 DEBUG:boto:Params: {} 2022-09-12T19:07:06.124 DEBUG:boto:Token: None 2022-09-12T19:07:06.125 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/ 2022-09-12T19:07:06.125 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0FR+DbCBV8STjgUD2ywMnjTfF2E= 2022-09-12T19:07:06.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0FR+DbCBV8STjgUD2ywMnjTfF2E=', 'Content-Length': '0'} 2022-09-12T19:07:06.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdc2a91796cf4ad6-00631f835a-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.182 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:06.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:06.184 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.184 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.185 DEBUG:boto:path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:07:06.185 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:07:06.186 DEBUG:boto:Method: GET 2022-09-12T19:07:06.187 DEBUG:boto:Path: /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:07:06.187 DEBUG:boto:Data: 2022-09-12T19:07:06.188 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.188 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:06.189 DEBUG:boto:Port: 8001 2022-09-12T19:07:06.190 DEBUG:boto:Params: {} 2022-09-12T19:07:06.190 DEBUG:boto:Token: None 2022-09-12T19:07:06.191 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:07:06.192 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8m+nuCaNK91s7e9IQBpkbM9QOm0= 2022-09-12T19:07:06.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8m+nuCaNK91s7e9IQBpkbM9QOm0=', 'Content-Length': '0'} 2022-09-12T19:07:06.195 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'p5OIjHSzwvUDkIBYzRPdKBGezjIkrn-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b517a3e071b40756-00631f835a-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.196 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.196 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.197 DEBUG:boto:path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:07:06.197 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:07:06.198 DEBUG:boto:Method: GET 2022-09-12T19:07:06.198 DEBUG:boto:Path: /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:07:06.199 DEBUG:boto:Data: 2022-09-12T19:07:06.200 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.200 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:06.201 DEBUG:boto:Port: 8000 2022-09-12T19:07:06.201 DEBUG:boto:Params: {} 2022-09-12T19:07:06.202 DEBUG:boto:Token: None 2022-09-12T19:07:06.203 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:07:06.203 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8m+nuCaNK91s7e9IQBpkbM9QOm0= 2022-09-12T19:07:06.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8m+nuCaNK91s7e9IQBpkbM9QOm0=', 'Content-Length': '0'} 2022-09-12T19:07:06.206 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'p5OIjHSzwvUDkIBYzRPdKBGezjIkrn-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082fd06aa578f3d0a-00631f835a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.207 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.208 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.208 DEBUG:boto:path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:07:06.209 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:07:06.209 DEBUG:boto:Method: HEAD 2022-09-12T19:07:06.210 DEBUG:boto:Path: /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:07:06.210 DEBUG:boto:Data: 2022-09-12T19:07:06.211 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.211 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:06.212 DEBUG:boto:Port: 8001 2022-09-12T19:07:06.212 DEBUG:boto:Params: {} 2022-09-12T19:07:06.213 DEBUG:boto:Token: None 2022-09-12T19:07:06.214 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:07:06.214 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JjAoYReWq5Yc9U85CkrK7Yvmicw= 2022-09-12T19:07:06.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JjAoYReWq5Yc9U85CkrK7Yvmicw=', 'Content-Length': '0'} 2022-09-12T19:07:06.217 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'p5OIjHSzwvUDkIBYzRPdKBGezjIkrn-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034c0b95e5e91aa39-00631f835a-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.218 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.218 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.219 DEBUG:boto:path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:07:06.219 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:07:06.220 DEBUG:boto:Method: HEAD 2022-09-12T19:07:06.220 DEBUG:boto:Path: /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:07:06.221 DEBUG:boto:Data: 2022-09-12T19:07:06.221 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.222 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:06.222 DEBUG:boto:Port: 8000 2022-09-12T19:07:06.223 DEBUG:boto:Params: {} 2022-09-12T19:07:06.224 DEBUG:boto:Token: None 2022-09-12T19:07:06.224 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:07:06.225 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JjAoYReWq5Yc9U85CkrK7Yvmicw= 2022-09-12T19:07:06.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JjAoYReWq5Yc9U85CkrK7Yvmicw=', 'Content-Length': '0'} 2022-09-12T19:07:06.227 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'p5OIjHSzwvUDkIBYzRPdKBGezjIkrn-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004339f7b618850829-00631f835a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:06.229 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.229 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.230 DEBUG:boto:path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:07:06.230 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:07:06.231 DEBUG:boto:Method: GET 2022-09-12T19:07:06.231 DEBUG:boto:Path: /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:07:06.232 DEBUG:boto:Data: 2022-09-12T19:07:06.232 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.233 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:06.234 DEBUG:boto:Port: 8001 2022-09-12T19:07:06.234 DEBUG:boto:Params: {} 2022-09-12T19:07:06.235 DEBUG:boto:Token: None 2022-09-12T19:07:06.236 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:07:06.236 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8m+nuCaNK91s7e9IQBpkbM9QOm0= 2022-09-12T19:07:06.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8m+nuCaNK91s7e9IQBpkbM9QOm0=', 'Content-Length': '0'} 2022-09-12T19:07:06.239 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'p5OIjHSzwvUDkIBYzRPdKBGezjIkrn-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094bc87df78090936-00631f835a-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.282 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.283 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.283 DEBUG:boto:path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:07:06.284 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:07:06.284 DEBUG:boto:Method: GET 2022-09-12T19:07:06.285 DEBUG:boto:Path: /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:07:06.286 DEBUG:boto:Data: 2022-09-12T19:07:06.286 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.287 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:06.287 DEBUG:boto:Port: 8000 2022-09-12T19:07:06.288 DEBUG:boto:Params: {} 2022-09-12T19:07:06.288 DEBUG:boto:Token: None 2022-09-12T19:07:06.289 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj?versionId=p5OIjHSzwvUDkIBYzRPdKBGezjIkrn- 2022-09-12T19:07:06.290 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8m+nuCaNK91s7e9IQBpkbM9QOm0= 2022-09-12T19:07:06.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8m+nuCaNK91s7e9IQBpkbM9QOm0=', 'Content-Length': '0'} 2022-09-12T19:07:06.292 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'p5OIjHSzwvUDkIBYzRPdKBGezjIkrn-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001aa0fb50932333bb-00631f835a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.293 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:06.294 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:06.294 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:06.295 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:06.295 DEBUG:boto:Method: GET 2022-09-12T19:07:06.296 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:06.296 DEBUG:boto:Data: 2022-09-12T19:07:06.297 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.297 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:06.298 DEBUG:boto:Port: 8001 2022-09-12T19:07:06.299 DEBUG:boto:Params: {} 2022-09-12T19:07:06.299 DEBUG:boto:Token: None 2022-09-12T19:07:06.300 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/ 2022-09-12T19:07:06.300 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0FR+DbCBV8STjgUD2ywMnjTfF2E= 2022-09-12T19:07:06.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0FR+DbCBV8STjgUD2ywMnjTfF2E=', 'Content-Length': '0'} 2022-09-12T19:07:06.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa9b07f5c5e7024d-00631f835a-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.305 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:06.306 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:06.306 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:06.307 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:06.307 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:06.308 DEBUG:boto:Method: GET 2022-09-12T19:07:06.308 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:06.309 DEBUG:boto:Data: 2022-09-12T19:07:06.309 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.310 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:06.310 DEBUG:boto:Port: 8000 2022-09-12T19:07:06.311 DEBUG:boto:Params: {} 2022-09-12T19:07:06.312 DEBUG:boto:Token: None 2022-09-12T19:07:06.312 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/ 2022-09-12T19:07:06.313 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0FR+DbCBV8STjgUD2ywMnjTfF2E= 2022-09-12T19:07:06.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0FR+DbCBV8STjgUD2ywMnjTfF2E=', 'Content-Length': '0'} 2022-09-12T19:07:06.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072dda1931dc0ab8e-00631f835a-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.317 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:06.318 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.319 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.319 DEBUG:boto:Method: HEAD 2022-09-12T19:07:06.320 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:06.320 DEBUG:boto:Data: 2022-09-12T19:07:06.321 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.321 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:06.322 DEBUG:boto:Port: 8001 2022-09-12T19:07:06.322 DEBUG:boto:Params: {} 2022-09-12T19:07:06.323 DEBUG:boto:Token: None 2022-09-12T19:07:06.323 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj 2022-09-12T19:07:06.324 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ts8+S2ZH+pG9pDjQXe4nrVd5KOU= 2022-09-12T19:07:06.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ts8+S2ZH+pG9pDjQXe4nrVd5KOU=', 'Content-Length': '0'} 2022-09-12T19:07:06.328 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e53f8eaaca643a11-00631f835a-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.329 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.329 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.330 DEBUG:boto:Method: HEAD 2022-09-12T19:07:06.330 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:06.331 DEBUG:boto:Data: 2022-09-12T19:07:06.331 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.332 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:06.332 DEBUG:boto:Port: 8000 2022-09-12T19:07:06.333 DEBUG:boto:Params: {} 2022-09-12T19:07:06.334 DEBUG:boto:Token: None 2022-09-12T19:07:06.334 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj 2022-09-12T19:07:06.335 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ts8+S2ZH+pG9pDjQXe4nrVd5KOU= 2022-09-12T19:07:06.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ts8+S2ZH+pG9pDjQXe4nrVd5KOU=', 'Content-Length': '0'} 2022-09-12T19:07:06.339 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003269b44229cc9815-00631f835a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:06.340 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.341 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.341 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:06.342 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:06.343 DEBUG:boto:Method: GET 2022-09-12T19:07:06.343 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:06.344 DEBUG:boto:Data: 2022-09-12T19:07:06.344 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.345 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:06.346 DEBUG:boto:Port: 8001 2022-09-12T19:07:06.347 DEBUG:boto:Params: {} 2022-09-12T19:07:06.348 DEBUG:boto:Token: None 2022-09-12T19:07:06.349 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:06.349 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HSZKt/4lzWNVTh04unKX5gKFmgc= 2022-09-12T19:07:06.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HSZKt/4lzWNVTh04unKX5gKFmgc=', 'Content-Length': '0'} 2022-09-12T19:07:06.352 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000af37fb5c4bd17d8-00631f835a-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.353 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.354 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.354 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:06.355 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:06.355 DEBUG:boto:Method: GET 2022-09-12T19:07:06.356 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:06.356 DEBUG:boto:Data: 2022-09-12T19:07:06.357 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.357 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:06.358 DEBUG:boto:Port: 8000 2022-09-12T19:07:06.359 DEBUG:boto:Params: {} 2022-09-12T19:07:06.360 DEBUG:boto:Token: None 2022-09-12T19:07:06.361 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:06.362 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HSZKt/4lzWNVTh04unKX5gKFmgc= 2022-09-12T19:07:06.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HSZKt/4lzWNVTh04unKX5gKFmgc=', 'Content-Length': '0'} 2022-09-12T19:07:06.366 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e221c7fc271d7cee-00631f835a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.406 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:06.408 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:06.408 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:06.409 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:06.409 DEBUG:boto:Method: GET 2022-09-12T19:07:06.410 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:06.410 DEBUG:boto:Data: 2022-09-12T19:07:06.411 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.412 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:06.412 DEBUG:boto:Port: 8001 2022-09-12T19:07:06.413 DEBUG:boto:Params: {} 2022-09-12T19:07:06.414 DEBUG:boto:Token: None 2022-09-12T19:07:06.414 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/ 2022-09-12T19:07:06.415 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0FR+DbCBV8STjgUD2ywMnjTfF2E= 2022-09-12T19:07:06.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0FR+DbCBV8STjgUD2ywMnjTfF2E=', 'Content-Length': '0'} 2022-09-12T19:07:06.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000883c3682b14ee081-00631f835a-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.419 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:06.420 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:06.421 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:06.422 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:06.422 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:06.423 DEBUG:boto:Method: GET 2022-09-12T19:07:06.424 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:06.424 DEBUG:boto:Data: 2022-09-12T19:07:06.425 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.425 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:06.426 DEBUG:boto:Port: 8000 2022-09-12T19:07:06.426 DEBUG:boto:Params: {} 2022-09-12T19:07:06.427 DEBUG:boto:Token: None 2022-09-12T19:07:06.427 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/ 2022-09-12T19:07:06.428 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0FR+DbCBV8STjgUD2ywMnjTfF2E= 2022-09-12T19:07:06.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0FR+DbCBV8STjgUD2ywMnjTfF2E=', 'Content-Length': '0'} 2022-09-12T19:07:06.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f094099f4bbb2f83-00631f835a-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.434 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:06.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:06.436 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.436 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.437 DEBUG:boto:path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:07:06.437 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:07:06.438 DEBUG:boto:Method: GET 2022-09-12T19:07:06.438 DEBUG:boto:Path: /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:07:06.439 DEBUG:boto:Data: 2022-09-12T19:07:06.439 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.440 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:06.441 DEBUG:boto:Port: 8001 2022-09-12T19:07:06.441 DEBUG:boto:Params: {} 2022-09-12T19:07:06.442 DEBUG:boto:Token: None 2022-09-12T19:07:06.442 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:07:06.443 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RrjLUV9GhEZF+EXeASoVg+5/wIo= 2022-09-12T19:07:06.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RrjLUV9GhEZF+EXeASoVg+5/wIo=', 'Content-Length': '0'} 2022-09-12T19:07:06.446 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea6a87b0c63e4d1a-00631f835a-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.501 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.501 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.502 DEBUG:boto:path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:07:06.503 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:07:06.503 DEBUG:boto:Method: GET 2022-09-12T19:07:06.504 DEBUG:boto:Path: /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:07:06.504 DEBUG:boto:Data: 2022-09-12T19:07:06.505 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.506 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:06.506 DEBUG:boto:Port: 8000 2022-09-12T19:07:06.507 DEBUG:boto:Params: {} 2022-09-12T19:07:06.507 DEBUG:boto:Token: None 2022-09-12T19:07:06.508 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:07:06.509 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RrjLUV9GhEZF+EXeASoVg+5/wIo= 2022-09-12T19:07:06.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RrjLUV9GhEZF+EXeASoVg+5/wIo=', 'Content-Length': '0'} 2022-09-12T19:07:06.512 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eafd8e9c2d48d044-00631f835a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.513 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.514 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.514 DEBUG:boto:path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:07:06.515 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:07:06.516 DEBUG:boto:Method: HEAD 2022-09-12T19:07:06.516 DEBUG:boto:Path: /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:07:06.517 DEBUG:boto:Data: 2022-09-12T19:07:06.517 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.518 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:06.518 DEBUG:boto:Port: 8001 2022-09-12T19:07:06.519 DEBUG:boto:Params: {} 2022-09-12T19:07:06.519 DEBUG:boto:Token: None 2022-09-12T19:07:06.520 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:07:06.521 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WUWRIjVuSF7UiFMIeXNRqGSsAJQ= 2022-09-12T19:07:06.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WUWRIjVuSF7UiFMIeXNRqGSsAJQ=', 'Content-Length': '0'} 2022-09-12T19:07:06.532 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031f16c06fa11858c-00631f835a-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.533 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.533 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.534 DEBUG:boto:path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:07:06.535 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:07:06.535 DEBUG:boto:Method: HEAD 2022-09-12T19:07:06.536 DEBUG:boto:Path: /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:07:06.536 DEBUG:boto:Data: 2022-09-12T19:07:06.537 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.537 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:06.538 DEBUG:boto:Port: 8000 2022-09-12T19:07:06.538 DEBUG:boto:Params: {} 2022-09-12T19:07:06.539 DEBUG:boto:Token: None 2022-09-12T19:07:06.540 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:07:06.540 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WUWRIjVuSF7UiFMIeXNRqGSsAJQ= 2022-09-12T19:07:06.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WUWRIjVuSF7UiFMIeXNRqGSsAJQ=', 'Content-Length': '0'} 2022-09-12T19:07:06.543 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb8d70069738c498-00631f835a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:06.544 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.545 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.545 DEBUG:boto:path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:07:06.546 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:07:06.547 DEBUG:boto:Method: GET 2022-09-12T19:07:06.547 DEBUG:boto:Path: /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:07:06.548 DEBUG:boto:Data: 2022-09-12T19:07:06.548 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.549 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:06.549 DEBUG:boto:Port: 8001 2022-09-12T19:07:06.550 DEBUG:boto:Params: {} 2022-09-12T19:07:06.551 DEBUG:boto:Token: None 2022-09-12T19:07:06.551 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:07:06.552 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RrjLUV9GhEZF+EXeASoVg+5/wIo= 2022-09-12T19:07:06.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RrjLUV9GhEZF+EXeASoVg+5/wIo=', 'Content-Length': '0'} 2022-09-12T19:07:06.555 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000308a47eacac6c004-00631f835a-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.556 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.556 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.557 DEBUG:boto:path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:07:06.558 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:07:06.558 DEBUG:boto:Method: GET 2022-09-12T19:07:06.559 DEBUG:boto:Path: /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:07:06.561 DEBUG:boto:Data: 2022-09-12T19:07:06.562 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.562 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:06.563 DEBUG:boto:Port: 8000 2022-09-12T19:07:06.563 DEBUG:boto:Params: {} 2022-09-12T19:07:06.564 DEBUG:boto:Token: None 2022-09-12T19:07:06.565 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj?versionId=V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9 2022-09-12T19:07:06.565 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RrjLUV9GhEZF+EXeASoVg+5/wIo= 2022-09-12T19:07:06.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RrjLUV9GhEZF+EXeASoVg+5/wIo=', 'Content-Length': '0'} 2022-09-12T19:07:06.568 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'V.HzbjOt7Vp5hVPJz0Vw26sSZAjUm.9'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f15dc8efc2f2af17-00631f835a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.572 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:06.572 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:06.573 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:06.573 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:06.574 DEBUG:boto:Method: GET 2022-09-12T19:07:06.575 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:06.575 DEBUG:boto:Data: 2022-09-12T19:07:06.576 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.576 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:06.577 DEBUG:boto:Port: 8001 2022-09-12T19:07:06.577 DEBUG:boto:Params: {} 2022-09-12T19:07:06.578 DEBUG:boto:Token: None 2022-09-12T19:07:06.579 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/ 2022-09-12T19:07:06.579 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0FR+DbCBV8STjgUD2ywMnjTfF2E= 2022-09-12T19:07:06.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0FR+DbCBV8STjgUD2ywMnjTfF2E=', 'Content-Length': '0'} 2022-09-12T19:07:06.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1177bd3952abea8-00631f835a-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.633 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:06.634 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:06.635 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:06.636 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:06.636 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:06.637 DEBUG:boto:Method: GET 2022-09-12T19:07:06.637 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:06.638 DEBUG:boto:Data: 2022-09-12T19:07:06.638 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.639 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:06.640 DEBUG:boto:Port: 8000 2022-09-12T19:07:06.640 DEBUG:boto:Params: {} 2022-09-12T19:07:06.641 DEBUG:boto:Token: None 2022-09-12T19:07:06.641 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/ 2022-09-12T19:07:06.642 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0FR+DbCBV8STjgUD2ywMnjTfF2E= 2022-09-12T19:07:06.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0FR+DbCBV8STjgUD2ywMnjTfF2E=', 'Content-Length': '0'} 2022-09-12T19:07:06.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8b29398c529c676-00631f835a-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.646 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:06.647 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.648 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.648 DEBUG:boto:Method: HEAD 2022-09-12T19:07:06.649 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:06.650 DEBUG:boto:Data: 2022-09-12T19:07:06.650 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.651 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:06.651 DEBUG:boto:Port: 8001 2022-09-12T19:07:06.652 DEBUG:boto:Params: {} 2022-09-12T19:07:06.653 DEBUG:boto:Token: None 2022-09-12T19:07:06.653 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj 2022-09-12T19:07:06.654 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ts8+S2ZH+pG9pDjQXe4nrVd5KOU= 2022-09-12T19:07:06.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ts8+S2ZH+pG9pDjQXe4nrVd5KOU=', 'Content-Length': '0'} 2022-09-12T19:07:06.658 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de1bfb7c85cadbcd-00631f835a-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.659 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.660 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.660 DEBUG:boto:Method: HEAD 2022-09-12T19:07:06.661 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:06.662 DEBUG:boto:Data: 2022-09-12T19:07:06.662 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.663 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:06.663 DEBUG:boto:Port: 8000 2022-09-12T19:07:06.664 DEBUG:boto:Params: {} 2022-09-12T19:07:06.665 DEBUG:boto:Token: None 2022-09-12T19:07:06.665 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj 2022-09-12T19:07:06.666 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ts8+S2ZH+pG9pDjQXe4nrVd5KOU= 2022-09-12T19:07:06.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ts8+S2ZH+pG9pDjQXe4nrVd5KOU=', 'Content-Length': '0'} 2022-09-12T19:07:06.670 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008007025b983dcfd0-00631f835a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:06.672 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.672 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.673 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:06.673 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:06.674 DEBUG:boto:Method: GET 2022-09-12T19:07:06.674 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:06.675 DEBUG:boto:Data: 2022-09-12T19:07:06.676 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.676 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:06.677 DEBUG:boto:Port: 8001 2022-09-12T19:07:06.677 DEBUG:boto:Params: {} 2022-09-12T19:07:06.678 DEBUG:boto:Token: None 2022-09-12T19:07:06.679 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:06.679 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HSZKt/4lzWNVTh04unKX5gKFmgc= 2022-09-12T19:07:06.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HSZKt/4lzWNVTh04unKX5gKFmgc=', 'Content-Length': '0'} 2022-09-12T19:07:06.682 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000706ee083bcf1ec14-00631f835a-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.683 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.684 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.684 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:06.685 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:06.686 DEBUG:boto:Method: GET 2022-09-12T19:07:06.686 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:06.687 DEBUG:boto:Data: 2022-09-12T19:07:06.688 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.688 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:06.689 DEBUG:boto:Port: 8000 2022-09-12T19:07:06.689 DEBUG:boto:Params: {} 2022-09-12T19:07:06.690 DEBUG:boto:Token: None 2022-09-12T19:07:06.690 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:06.691 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HSZKt/4lzWNVTh04unKX5gKFmgc= 2022-09-12T19:07:06.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HSZKt/4lzWNVTh04unKX5gKFmgc=', 'Content-Length': '0'} 2022-09-12T19:07:06.694 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa922f5c1c8080be-00631f835a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.738 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:06.739 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:06.739 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:06.740 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:06.740 DEBUG:boto:Method: GET 2022-09-12T19:07:06.741 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:06.741 DEBUG:boto:Data: 2022-09-12T19:07:06.742 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.743 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:06.743 DEBUG:boto:Port: 8001 2022-09-12T19:07:06.744 DEBUG:boto:Params: {} 2022-09-12T19:07:06.744 DEBUG:boto:Token: None 2022-09-12T19:07:06.745 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/ 2022-09-12T19:07:06.746 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0FR+DbCBV8STjgUD2ywMnjTfF2E= 2022-09-12T19:07:06.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0FR+DbCBV8STjgUD2ywMnjTfF2E=', 'Content-Length': '0'} 2022-09-12T19:07:06.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8bfe12857b23e06-00631f835a-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.750 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:06.751 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:06.752 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:06.752 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:06.753 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:06.753 DEBUG:boto:Method: GET 2022-09-12T19:07:06.754 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:06.755 DEBUG:boto:Data: 2022-09-12T19:07:06.755 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.756 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:06.756 DEBUG:boto:Port: 8000 2022-09-12T19:07:06.757 DEBUG:boto:Params: {} 2022-09-12T19:07:06.757 DEBUG:boto:Token: None 2022-09-12T19:07:06.758 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/ 2022-09-12T19:07:06.759 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0FR+DbCBV8STjgUD2ywMnjTfF2E= 2022-09-12T19:07:06.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0FR+DbCBV8STjgUD2ywMnjTfF2E=', 'Content-Length': '0'} 2022-09-12T19:07:06.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c09e390ca507054-00631f835a-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.763 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:06.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:06.764 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.765 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.765 DEBUG:boto:path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:07:06.766 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:07:06.766 DEBUG:boto:Method: GET 2022-09-12T19:07:06.767 DEBUG:boto:Path: /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:07:06.768 DEBUG:boto:Data: 2022-09-12T19:07:06.768 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.769 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:06.769 DEBUG:boto:Port: 8001 2022-09-12T19:07:06.770 DEBUG:boto:Params: {} 2022-09-12T19:07:06.771 DEBUG:boto:Token: None 2022-09-12T19:07:06.771 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:07:06.772 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eFeCuTJ75xCb4wRmykxW3bwJcf4= 2022-09-12T19:07:06.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eFeCuTJ75xCb4wRmykxW3bwJcf4=', 'Content-Length': '0'} 2022-09-12T19:07:06.775 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aaf1cf39a6f30fa7-00631f835a-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.827 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.827 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.828 DEBUG:boto:path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:07:06.829 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:07:06.829 DEBUG:boto:Method: GET 2022-09-12T19:07:06.830 DEBUG:boto:Path: /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:07:06.830 DEBUG:boto:Data: 2022-09-12T19:07:06.831 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.832 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:06.832 DEBUG:boto:Port: 8000 2022-09-12T19:07:06.833 DEBUG:boto:Params: {} 2022-09-12T19:07:06.834 DEBUG:boto:Token: None 2022-09-12T19:07:06.834 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:07:06.835 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eFeCuTJ75xCb4wRmykxW3bwJcf4= 2022-09-12T19:07:06.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eFeCuTJ75xCb4wRmykxW3bwJcf4=', 'Content-Length': '0'} 2022-09-12T19:07:06.838 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000812ebf23c354bacf-00631f835a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.842 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.843 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.843 DEBUG:boto:path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:07:06.844 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:07:06.844 DEBUG:boto:Method: HEAD 2022-09-12T19:07:06.845 DEBUG:boto:Path: /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:07:06.845 DEBUG:boto:Data: 2022-09-12T19:07:06.846 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.846 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:06.847 DEBUG:boto:Port: 8001 2022-09-12T19:07:06.848 DEBUG:boto:Params: {} 2022-09-12T19:07:06.848 DEBUG:boto:Token: None 2022-09-12T19:07:06.849 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:07:06.850 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2kK7t2c3m8uDVWdL76kaCTeiB2g= 2022-09-12T19:07:06.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2kK7t2c3m8uDVWdL76kaCTeiB2g=', 'Content-Length': '0'} 2022-09-12T19:07:06.865 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003382a6fbbf37f0fa-00631f835a-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.866 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.866 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.867 DEBUG:boto:path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:07:06.867 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:07:06.868 DEBUG:boto:Method: HEAD 2022-09-12T19:07:06.869 DEBUG:boto:Path: /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:07:06.869 DEBUG:boto:Data: 2022-09-12T19:07:06.870 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.871 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:06.872 DEBUG:boto:Port: 8000 2022-09-12T19:07:06.872 DEBUG:boto:Params: {} 2022-09-12T19:07:06.873 DEBUG:boto:Token: None 2022-09-12T19:07:06.874 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:07:06.874 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2kK7t2c3m8uDVWdL76kaCTeiB2g= 2022-09-12T19:07:06.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2kK7t2c3m8uDVWdL76kaCTeiB2g=', 'Content-Length': '0'} 2022-09-12T19:07:06.877 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000065a36c4801aaa77-00631f835a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:06.879 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.879 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.880 DEBUG:boto:path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:07:06.880 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:07:06.881 DEBUG:boto:Method: GET 2022-09-12T19:07:06.881 DEBUG:boto:Path: /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:07:06.882 DEBUG:boto:Data: 2022-09-12T19:07:06.882 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.883 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:06.883 DEBUG:boto:Port: 8001 2022-09-12T19:07:06.884 DEBUG:boto:Params: {} 2022-09-12T19:07:06.885 DEBUG:boto:Token: None 2022-09-12T19:07:06.885 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:07:06.886 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eFeCuTJ75xCb4wRmykxW3bwJcf4= 2022-09-12T19:07:06.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eFeCuTJ75xCb4wRmykxW3bwJcf4=', 'Content-Length': '0'} 2022-09-12T19:07:06.889 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000118ea76a92e837f1-00631f835a-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.892 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.892 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.893 DEBUG:boto:path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:07:06.893 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:07:06.894 DEBUG:boto:Method: GET 2022-09-12T19:07:06.895 DEBUG:boto:Path: /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:07:06.895 DEBUG:boto:Data: 2022-09-12T19:07:06.896 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.896 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:06.897 DEBUG:boto:Port: 8000 2022-09-12T19:07:06.897 DEBUG:boto:Params: {} 2022-09-12T19:07:06.898 DEBUG:boto:Token: None 2022-09-12T19:07:06.899 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj?versionId=iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd 2022-09-12T19:07:06.899 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eFeCuTJ75xCb4wRmykxW3bwJcf4= 2022-09-12T19:07:06.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eFeCuTJ75xCb4wRmykxW3bwJcf4=', 'Content-Length': '0'} 2022-09-12T19:07:06.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'iJi00oqpo4jP5T2qIUe4H2lLKJSYXZd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076cde414b33c6f2d-00631f835a-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.906 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:06.907 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:06.908 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:06.909 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:06.910 DEBUG:boto:Method: GET 2022-09-12T19:07:06.912 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:06.913 DEBUG:boto:Data: 2022-09-12T19:07:06.914 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.915 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:06.916 DEBUG:boto:Port: 8001 2022-09-12T19:07:06.916 DEBUG:boto:Params: {} 2022-09-12T19:07:06.917 DEBUG:boto:Token: None 2022-09-12T19:07:06.918 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/ 2022-09-12T19:07:06.918 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0FR+DbCBV8STjgUD2ywMnjTfF2E= 2022-09-12T19:07:06.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0FR+DbCBV8STjgUD2ywMnjTfF2E=', 'Content-Length': '0'} 2022-09-12T19:07:06.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ff505bc9ec5cf40-00631f835a-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.967 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:06.968 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:06.969 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:06.970 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:06.970 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:06.971 DEBUG:boto:Method: GET 2022-09-12T19:07:06.971 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:06.972 DEBUG:boto:Data: 2022-09-12T19:07:06.972 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.973 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:06.974 DEBUG:boto:Port: 8000 2022-09-12T19:07:06.974 DEBUG:boto:Params: {} 2022-09-12T19:07:06.975 DEBUG:boto:Token: None 2022-09-12T19:07:06.976 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/ 2022-09-12T19:07:06.977 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0FR+DbCBV8STjgUD2ywMnjTfF2E= 2022-09-12T19:07:06.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0FR+DbCBV8STjgUD2ywMnjTfF2E=', 'Content-Length': '0'} 2022-09-12T19:07:06.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d7e09f8a58ac3e2-00631f835a-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.982 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:06.983 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.983 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.984 DEBUG:boto:Method: HEAD 2022-09-12T19:07:06.984 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:06.985 DEBUG:boto:Data: 2022-09-12T19:07:06.986 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.986 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:06.987 DEBUG:boto:Port: 8001 2022-09-12T19:07:06.987 DEBUG:boto:Params: {} 2022-09-12T19:07:06.988 DEBUG:boto:Token: None 2022-09-12T19:07:06.989 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:06 GMT /jdpiak-19/obj 2022-09-12T19:07:06.989 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ts8+S2ZH+pG9pDjQXe4nrVd5KOU= 2022-09-12T19:07:06.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ts8+S2ZH+pG9pDjQXe4nrVd5KOU=', 'Content-Length': '0'} 2022-09-12T19:07:06.994 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000066da759f855b642-00631f835a-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:06.995 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:06.995 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:06.996 DEBUG:boto:Method: HEAD 2022-09-12T19:07:06.997 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:06.997 DEBUG:boto:Data: 2022-09-12T19:07:06.998 DEBUG:boto:Headers: {} 2022-09-12T19:07:06.998 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:06.999 DEBUG:boto:Port: 8000 2022-09-12T19:07:07.000 DEBUG:boto:Params: {} 2022-09-12T19:07:07.000 DEBUG:boto:Token: None 2022-09-12T19:07:07.001 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj 2022-09-12T19:07:07.002 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RGisZBGOrVxLqBndkdLbo4Tu2MU= 2022-09-12T19:07:07.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RGisZBGOrVxLqBndkdLbo4Tu2MU=', 'Content-Length': '0'} 2022-09-12T19:07:07.007 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000694a2cd16461dd4d-00631f835b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:07.010 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.011 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.011 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:07.012 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:07.013 DEBUG:boto:Method: GET 2022-09-12T19:07:07.014 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:07.014 DEBUG:boto:Data: 2022-09-12T19:07:07.015 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.015 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:07.016 DEBUG:boto:Port: 8001 2022-09-12T19:07:07.017 DEBUG:boto:Params: {} 2022-09-12T19:07:07.017 DEBUG:boto:Token: None 2022-09-12T19:07:07.018 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:07.019 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cEsWsSkcOUySUYbmTRdJhnBF0qU= 2022-09-12T19:07:07.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cEsWsSkcOUySUYbmTRdJhnBF0qU=', 'Content-Length': '0'} 2022-09-12T19:07:07.022 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000413056741437a2fe-00631f835b-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.023 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.024 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.025 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:07.025 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:07.026 DEBUG:boto:Method: GET 2022-09-12T19:07:07.027 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:07.027 DEBUG:boto:Data: 2022-09-12T19:07:07.028 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.029 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:07.030 DEBUG:boto:Port: 8000 2022-09-12T19:07:07.031 DEBUG:boto:Params: {} 2022-09-12T19:07:07.031 DEBUG:boto:Token: None 2022-09-12T19:07:07.032 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:07.034 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cEsWsSkcOUySUYbmTRdJhnBF0qU= 2022-09-12T19:07:07.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cEsWsSkcOUySUYbmTRdJhnBF0qU=', 'Content-Length': '0'} 2022-09-12T19:07:07.038 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d0d86c191e00298-00631f835b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.079 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:07.080 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:07.081 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:07.082 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:07.082 DEBUG:boto:Method: GET 2022-09-12T19:07:07.083 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:07.084 DEBUG:boto:Data: 2022-09-12T19:07:07.084 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.085 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:07.086 DEBUG:boto:Port: 8001 2022-09-12T19:07:07.086 DEBUG:boto:Params: {} 2022-09-12T19:07:07.087 DEBUG:boto:Token: None 2022-09-12T19:07:07.087 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/ 2022-09-12T19:07:07.088 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ectbrol5GfNgKed38OsF7+Su7m8= 2022-09-12T19:07:07.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ectbrol5GfNgKed38OsF7+Su7m8=', 'Content-Length': '0'} 2022-09-12T19:07:07.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007348e87ba20c24b1-00631f835b-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.093 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:07.094 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:07.094 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:07.095 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:07.095 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:07.096 DEBUG:boto:Method: GET 2022-09-12T19:07:07.096 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:07.097 DEBUG:boto:Data: 2022-09-12T19:07:07.097 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.098 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:07.099 DEBUG:boto:Port: 8000 2022-09-12T19:07:07.099 DEBUG:boto:Params: {} 2022-09-12T19:07:07.100 DEBUG:boto:Token: None 2022-09-12T19:07:07.101 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/ 2022-09-12T19:07:07.101 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ectbrol5GfNgKed38OsF7+Su7m8= 2022-09-12T19:07:07.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ectbrol5GfNgKed38OsF7+Su7m8=', 'Content-Length': '0'} 2022-09-12T19:07:07.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002122521723ee24ee-00631f835b-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.106 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:07.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:07.107 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.108 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.108 DEBUG:boto:path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:07:07.109 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:07:07.110 DEBUG:boto:Method: GET 2022-09-12T19:07:07.111 DEBUG:boto:Path: /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:07:07.111 DEBUG:boto:Data: 2022-09-12T19:07:07.112 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.112 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:07.113 DEBUG:boto:Port: 8001 2022-09-12T19:07:07.113 DEBUG:boto:Params: {} 2022-09-12T19:07:07.114 DEBUG:boto:Token: None 2022-09-12T19:07:07.114 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:07:07.115 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/Rb9NGl524v3Mj+X5JDvycr9Dg8= 2022-09-12T19:07:07.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/Rb9NGl524v3Mj+X5JDvycr9Dg8=', 'Content-Length': '0'} 2022-09-12T19:07:07.118 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'j0eE53eOEuIue0LhYTVuZStUTmcOj7o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003aad1b110ae44565-00631f835b-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.165 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.166 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.167 DEBUG:boto:path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:07:07.167 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:07:07.168 DEBUG:boto:Method: GET 2022-09-12T19:07:07.169 DEBUG:boto:Path: /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:07:07.170 DEBUG:boto:Data: 2022-09-12T19:07:07.171 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.172 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:07.172 DEBUG:boto:Port: 8000 2022-09-12T19:07:07.173 DEBUG:boto:Params: {} 2022-09-12T19:07:07.174 DEBUG:boto:Token: None 2022-09-12T19:07:07.175 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:07:07.176 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/Rb9NGl524v3Mj+X5JDvycr9Dg8= 2022-09-12T19:07:07.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/Rb9NGl524v3Mj+X5JDvycr9Dg8=', 'Content-Length': '0'} 2022-09-12T19:07:07.179 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'j0eE53eOEuIue0LhYTVuZStUTmcOj7o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b692391c19809e8d-00631f835b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.182 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.183 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.184 DEBUG:boto:path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:07:07.184 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:07:07.185 DEBUG:boto:Method: HEAD 2022-09-12T19:07:07.186 DEBUG:boto:Path: /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:07:07.187 DEBUG:boto:Data: 2022-09-12T19:07:07.187 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.188 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:07.189 DEBUG:boto:Port: 8001 2022-09-12T19:07:07.189 DEBUG:boto:Params: {} 2022-09-12T19:07:07.190 DEBUG:boto:Token: None 2022-09-12T19:07:07.191 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:07:07.192 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WsHHTAJJE2kV1VW4YmasQ3dpfyk= 2022-09-12T19:07:07.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WsHHTAJJE2kV1VW4YmasQ3dpfyk=', 'Content-Length': '0'} 2022-09-12T19:07:07.196 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'j0eE53eOEuIue0LhYTVuZStUTmcOj7o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000bf2a2e59d7ed4be-00631f835b-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.197 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.197 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.198 DEBUG:boto:path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:07:07.199 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:07:07.200 DEBUG:boto:Method: HEAD 2022-09-12T19:07:07.200 DEBUG:boto:Path: /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:07:07.201 DEBUG:boto:Data: 2022-09-12T19:07:07.202 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.203 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:07.204 DEBUG:boto:Port: 8000 2022-09-12T19:07:07.205 DEBUG:boto:Params: {} 2022-09-12T19:07:07.206 DEBUG:boto:Token: None 2022-09-12T19:07:07.207 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:07:07.207 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WsHHTAJJE2kV1VW4YmasQ3dpfyk= 2022-09-12T19:07:07.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WsHHTAJJE2kV1VW4YmasQ3dpfyk=', 'Content-Length': '0'} 2022-09-12T19:07:07.210 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'j0eE53eOEuIue0LhYTVuZStUTmcOj7o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f897d1f98c70441-00631f835b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:07.212 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.213 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.213 DEBUG:boto:path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:07:07.214 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:07:07.215 DEBUG:boto:Method: GET 2022-09-12T19:07:07.215 DEBUG:boto:Path: /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:07:07.216 DEBUG:boto:Data: 2022-09-12T19:07:07.217 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.218 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:07.219 DEBUG:boto:Port: 8001 2022-09-12T19:07:07.220 DEBUG:boto:Params: {} 2022-09-12T19:07:07.221 DEBUG:boto:Token: None 2022-09-12T19:07:07.222 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:07:07.223 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/Rb9NGl524v3Mj+X5JDvycr9Dg8= 2022-09-12T19:07:07.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/Rb9NGl524v3Mj+X5JDvycr9Dg8=', 'Content-Length': '0'} 2022-09-12T19:07:07.226 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'j0eE53eOEuIue0LhYTVuZStUTmcOj7o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008483a330aa9150fc-00631f835b-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.228 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.228 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.229 DEBUG:boto:path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:07:07.230 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:07:07.231 DEBUG:boto:Method: GET 2022-09-12T19:07:07.232 DEBUG:boto:Path: /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:07:07.233 DEBUG:boto:Data: 2022-09-12T19:07:07.233 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.234 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:07.235 DEBUG:boto:Port: 8000 2022-09-12T19:07:07.236 DEBUG:boto:Params: {} 2022-09-12T19:07:07.237 DEBUG:boto:Token: None 2022-09-12T19:07:07.237 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj?versionId=j0eE53eOEuIue0LhYTVuZStUTmcOj7o 2022-09-12T19:07:07.238 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/Rb9NGl524v3Mj+X5JDvycr9Dg8= 2022-09-12T19:07:07.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/Rb9NGl524v3Mj+X5JDvycr9Dg8=', 'Content-Length': '0'} 2022-09-12T19:07:07.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'j0eE53eOEuIue0LhYTVuZStUTmcOj7o'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a8338b28ff38101-00631f835b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.294 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:07.295 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:07.296 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:07.297 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:07.298 DEBUG:boto:Method: GET 2022-09-12T19:07:07.299 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:07.300 DEBUG:boto:Data: 2022-09-12T19:07:07.301 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.302 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:07.303 DEBUG:boto:Port: 8001 2022-09-12T19:07:07.304 DEBUG:boto:Params: {} 2022-09-12T19:07:07.305 DEBUG:boto:Token: None 2022-09-12T19:07:07.307 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/ 2022-09-12T19:07:07.308 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ectbrol5GfNgKed38OsF7+Su7m8= 2022-09-12T19:07:07.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ectbrol5GfNgKed38OsF7+Su7m8=', 'Content-Length': '0'} 2022-09-12T19:07:07.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094a4dfc1562b1672-00631f835b-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.313 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:07.314 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:07.315 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:07.316 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:07.317 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:07.318 DEBUG:boto:Method: GET 2022-09-12T19:07:07.318 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:07.319 DEBUG:boto:Data: 2022-09-12T19:07:07.320 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.321 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:07.322 DEBUG:boto:Port: 8000 2022-09-12T19:07:07.323 DEBUG:boto:Params: {} 2022-09-12T19:07:07.324 DEBUG:boto:Token: None 2022-09-12T19:07:07.325 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/ 2022-09-12T19:07:07.326 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ectbrol5GfNgKed38OsF7+Su7m8= 2022-09-12T19:07:07.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ectbrol5GfNgKed38OsF7+Su7m8=', 'Content-Length': '0'} 2022-09-12T19:07:07.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6b14564a5c3d8d8-00631f835b-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.332 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:07.333 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.334 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.335 DEBUG:boto:Method: HEAD 2022-09-12T19:07:07.336 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:07.337 DEBUG:boto:Data: 2022-09-12T19:07:07.338 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.339 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:07.340 DEBUG:boto:Port: 8001 2022-09-12T19:07:07.341 DEBUG:boto:Params: {} 2022-09-12T19:07:07.341 DEBUG:boto:Token: None 2022-09-12T19:07:07.342 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj 2022-09-12T19:07:07.343 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RGisZBGOrVxLqBndkdLbo4Tu2MU= 2022-09-12T19:07:07.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RGisZBGOrVxLqBndkdLbo4Tu2MU=', 'Content-Length': '0'} 2022-09-12T19:07:07.349 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000911f94731c1a22a4-00631f835b-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.350 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.351 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.352 DEBUG:boto:Method: HEAD 2022-09-12T19:07:07.352 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:07.353 DEBUG:boto:Data: 2022-09-12T19:07:07.354 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.355 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:07.356 DEBUG:boto:Port: 8000 2022-09-12T19:07:07.357 DEBUG:boto:Params: {} 2022-09-12T19:07:07.358 DEBUG:boto:Token: None 2022-09-12T19:07:07.358 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj 2022-09-12T19:07:07.359 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RGisZBGOrVxLqBndkdLbo4Tu2MU= 2022-09-12T19:07:07.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RGisZBGOrVxLqBndkdLbo4Tu2MU=', 'Content-Length': '0'} 2022-09-12T19:07:07.367 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ea8d048ff635a43-00631f835b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:07.369 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.370 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.371 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:07.371 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:07.372 DEBUG:boto:Method: GET 2022-09-12T19:07:07.372 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:07.373 DEBUG:boto:Data: 2022-09-12T19:07:07.374 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.374 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:07.375 DEBUG:boto:Port: 8001 2022-09-12T19:07:07.375 DEBUG:boto:Params: {} 2022-09-12T19:07:07.376 DEBUG:boto:Token: None 2022-09-12T19:07:07.377 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:07.377 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cEsWsSkcOUySUYbmTRdJhnBF0qU= 2022-09-12T19:07:07.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cEsWsSkcOUySUYbmTRdJhnBF0qU=', 'Content-Length': '0'} 2022-09-12T19:07:07.380 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d12165928aba86f-00631f835b-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.381 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.382 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.383 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:07.383 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:07.384 DEBUG:boto:Method: GET 2022-09-12T19:07:07.384 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:07.385 DEBUG:boto:Data: 2022-09-12T19:07:07.385 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.386 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:07.387 DEBUG:boto:Port: 8000 2022-09-12T19:07:07.387 DEBUG:boto:Params: {} 2022-09-12T19:07:07.388 DEBUG:boto:Token: None 2022-09-12T19:07:07.388 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:07.389 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cEsWsSkcOUySUYbmTRdJhnBF0qU= 2022-09-12T19:07:07.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cEsWsSkcOUySUYbmTRdJhnBF0qU=', 'Content-Length': '0'} 2022-09-12T19:07:07.392 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c7c1433da3523f5-00631f835b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.439 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:07.441 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:07.442 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:07.443 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:07.444 DEBUG:boto:Method: GET 2022-09-12T19:07:07.445 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:07.446 DEBUG:boto:Data: 2022-09-12T19:07:07.447 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.448 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:07.449 DEBUG:boto:Port: 8001 2022-09-12T19:07:07.451 DEBUG:boto:Params: {} 2022-09-12T19:07:07.452 DEBUG:boto:Token: None 2022-09-12T19:07:07.453 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/ 2022-09-12T19:07:07.454 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ectbrol5GfNgKed38OsF7+Su7m8= 2022-09-12T19:07:07.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ectbrol5GfNgKed38OsF7+Su7m8=', 'Content-Length': '0'} 2022-09-12T19:07:07.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e9b440d4cb4f572-00631f835b-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.460 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:07.462 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:07.463 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:07.464 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:07.465 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:07.466 DEBUG:boto:Method: GET 2022-09-12T19:07:07.467 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:07.468 DEBUG:boto:Data: 2022-09-12T19:07:07.469 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.470 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:07.471 DEBUG:boto:Port: 8000 2022-09-12T19:07:07.473 DEBUG:boto:Params: {} 2022-09-12T19:07:07.474 DEBUG:boto:Token: None 2022-09-12T19:07:07.476 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/ 2022-09-12T19:07:07.478 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ectbrol5GfNgKed38OsF7+Su7m8= 2022-09-12T19:07:07.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ectbrol5GfNgKed38OsF7+Su7m8=', 'Content-Length': '0'} 2022-09-12T19:07:07.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9f380427b087a5c-00631f835b-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.485 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:07.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:07.487 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.487 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.488 DEBUG:boto:path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:07:07.488 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:07:07.489 DEBUG:boto:Method: GET 2022-09-12T19:07:07.489 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:07:07.490 DEBUG:boto:Data: 2022-09-12T19:07:07.491 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.492 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:07.493 DEBUG:boto:Port: 8001 2022-09-12T19:07:07.494 DEBUG:boto:Params: {} 2022-09-12T19:07:07.495 DEBUG:boto:Token: None 2022-09-12T19:07:07.497 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:07:07.498 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZCthoOgtE8aC6tZdO8lH12ovjss= 2022-09-12T19:07:07.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZCthoOgtE8aC6tZdO8lH12ovjss=', 'Content-Length': '0'} 2022-09-12T19:07:07.502 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000fbbfd6dbf069bf0-00631f835b-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.504 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.505 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.506 DEBUG:boto:path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:07:07.507 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:07:07.508 DEBUG:boto:Method: GET 2022-09-12T19:07:07.509 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:07:07.510 DEBUG:boto:Data: 2022-09-12T19:07:07.511 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.512 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:07.513 DEBUG:boto:Port: 8000 2022-09-12T19:07:07.514 DEBUG:boto:Params: {} 2022-09-12T19:07:07.515 DEBUG:boto:Token: None 2022-09-12T19:07:07.515 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:07:07.516 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZCthoOgtE8aC6tZdO8lH12ovjss= 2022-09-12T19:07:07.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZCthoOgtE8aC6tZdO8lH12ovjss=', 'Content-Length': '0'} 2022-09-12T19:07:07.520 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a31a3d5592fb2fb4-00631f835b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.522 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.523 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.523 DEBUG:boto:path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:07:07.524 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:07:07.524 DEBUG:boto:Method: HEAD 2022-09-12T19:07:07.525 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:07:07.525 DEBUG:boto:Data: 2022-09-12T19:07:07.526 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.527 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:07.527 DEBUG:boto:Port: 8001 2022-09-12T19:07:07.528 DEBUG:boto:Params: {} 2022-09-12T19:07:07.528 DEBUG:boto:Token: None 2022-09-12T19:07:07.529 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:07:07.530 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cHCYd47/2XrysMc9ewZUQFRJAcU= 2022-09-12T19:07:07.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cHCYd47/2XrysMc9ewZUQFRJAcU=', 'Content-Length': '0'} 2022-09-12T19:07:07.534 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009109f2df842af36b-00631f835b-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.535 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.535 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.536 DEBUG:boto:path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:07:07.537 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:07:07.537 DEBUG:boto:Method: HEAD 2022-09-12T19:07:07.538 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:07:07.538 DEBUG:boto:Data: 2022-09-12T19:07:07.539 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.539 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:07.540 DEBUG:boto:Port: 8000 2022-09-12T19:07:07.541 DEBUG:boto:Params: {} 2022-09-12T19:07:07.542 DEBUG:boto:Token: None 2022-09-12T19:07:07.543 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:07:07.543 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cHCYd47/2XrysMc9ewZUQFRJAcU= 2022-09-12T19:07:07.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cHCYd47/2XrysMc9ewZUQFRJAcU=', 'Content-Length': '0'} 2022-09-12T19:07:07.547 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a78782a0c5069cec-00631f835b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:07.548 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.549 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.550 DEBUG:boto:path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:07:07.551 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:07:07.551 DEBUG:boto:Method: GET 2022-09-12T19:07:07.552 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:07:07.552 DEBUG:boto:Data: 2022-09-12T19:07:07.553 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.554 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:07.554 DEBUG:boto:Port: 8001 2022-09-12T19:07:07.555 DEBUG:boto:Params: {} 2022-09-12T19:07:07.555 DEBUG:boto:Token: None 2022-09-12T19:07:07.556 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:07:07.557 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZCthoOgtE8aC6tZdO8lH12ovjss= 2022-09-12T19:07:07.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZCthoOgtE8aC6tZdO8lH12ovjss=', 'Content-Length': '0'} 2022-09-12T19:07:07.561 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005bea2c91e680fd14-00631f835b-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.604 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.605 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.606 DEBUG:boto:path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:07:07.606 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:07:07.607 DEBUG:boto:Method: GET 2022-09-12T19:07:07.608 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:07:07.608 DEBUG:boto:Data: 2022-09-12T19:07:07.609 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.609 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:07.610 DEBUG:boto:Port: 8000 2022-09-12T19:07:07.610 DEBUG:boto:Params: {} 2022-09-12T19:07:07.611 DEBUG:boto:Token: None 2022-09-12T19:07:07.612 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj?versionId=ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY 2022-09-12T19:07:07.612 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZCthoOgtE8aC6tZdO8lH12ovjss= 2022-09-12T19:07:07.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZCthoOgtE8aC6tZdO8lH12ovjss=', 'Content-Length': '0'} 2022-09-12T19:07:07.634 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ACPdCUDfm32QLq6XbWYcqHiCLcEU2JY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e213ee098c95e89-00631f835b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.636 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:07.637 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:07.637 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:07.638 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:07.638 DEBUG:boto:Method: GET 2022-09-12T19:07:07.639 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:07.639 DEBUG:boto:Data: 2022-09-12T19:07:07.640 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.640 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:07.641 DEBUG:boto:Port: 8001 2022-09-12T19:07:07.642 DEBUG:boto:Params: {} 2022-09-12T19:07:07.642 DEBUG:boto:Token: None 2022-09-12T19:07:07.643 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/ 2022-09-12T19:07:07.644 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ectbrol5GfNgKed38OsF7+Su7m8= 2022-09-12T19:07:07.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ectbrol5GfNgKed38OsF7+Su7m8=', 'Content-Length': '0'} 2022-09-12T19:07:07.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050fef55de14360d9-00631f835b-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.650 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:07.651 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:07.652 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:07.652 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:07.653 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:07.654 DEBUG:boto:Method: GET 2022-09-12T19:07:07.655 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:07.656 DEBUG:boto:Data: 2022-09-12T19:07:07.656 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.657 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:07.657 DEBUG:boto:Port: 8000 2022-09-12T19:07:07.658 DEBUG:boto:Params: {} 2022-09-12T19:07:07.658 DEBUG:boto:Token: None 2022-09-12T19:07:07.659 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/ 2022-09-12T19:07:07.660 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ectbrol5GfNgKed38OsF7+Su7m8= 2022-09-12T19:07:07.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ectbrol5GfNgKed38OsF7+Su7m8=', 'Content-Length': '0'} 2022-09-12T19:07:07.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055c5f5ef1813c2a9-00631f835b-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.708 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:07.710 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.711 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.711 DEBUG:boto:Method: HEAD 2022-09-12T19:07:07.712 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:07.713 DEBUG:boto:Data: 2022-09-12T19:07:07.714 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.715 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:07.715 DEBUG:boto:Port: 8001 2022-09-12T19:07:07.716 DEBUG:boto:Params: {} 2022-09-12T19:07:07.717 DEBUG:boto:Token: None 2022-09-12T19:07:07.718 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj 2022-09-12T19:07:07.719 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RGisZBGOrVxLqBndkdLbo4Tu2MU= 2022-09-12T19:07:07.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RGisZBGOrVxLqBndkdLbo4Tu2MU=', 'Content-Length': '0'} 2022-09-12T19:07:07.726 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009bc20831d04a5fc3-00631f835b-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.728 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.728 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.729 DEBUG:boto:Method: HEAD 2022-09-12T19:07:07.730 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:07.731 DEBUG:boto:Data: 2022-09-12T19:07:07.732 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.733 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:07.734 DEBUG:boto:Port: 8000 2022-09-12T19:07:07.735 DEBUG:boto:Params: {} 2022-09-12T19:07:07.736 DEBUG:boto:Token: None 2022-09-12T19:07:07.737 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj 2022-09-12T19:07:07.738 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RGisZBGOrVxLqBndkdLbo4Tu2MU= 2022-09-12T19:07:07.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RGisZBGOrVxLqBndkdLbo4Tu2MU=', 'Content-Length': '0'} 2022-09-12T19:07:07.743 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005541bd1d48c82113-00631f835b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:07.744 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.745 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.746 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:07.747 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:07.748 DEBUG:boto:Method: GET 2022-09-12T19:07:07.749 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:07.749 DEBUG:boto:Data: 2022-09-12T19:07:07.750 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.752 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:07.753 DEBUG:boto:Port: 8001 2022-09-12T19:07:07.754 DEBUG:boto:Params: {} 2022-09-12T19:07:07.755 DEBUG:boto:Token: None 2022-09-12T19:07:07.755 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:07.756 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cEsWsSkcOUySUYbmTRdJhnBF0qU= 2022-09-12T19:07:07.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cEsWsSkcOUySUYbmTRdJhnBF0qU=', 'Content-Length': '0'} 2022-09-12T19:07:07.759 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014b72293b3a211ae-00631f835b-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.760 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.761 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.762 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:07.763 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:07.763 DEBUG:boto:Method: GET 2022-09-12T19:07:07.764 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:07.764 DEBUG:boto:Data: 2022-09-12T19:07:07.765 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.767 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:07.768 DEBUG:boto:Port: 8000 2022-09-12T19:07:07.769 DEBUG:boto:Params: {} 2022-09-12T19:07:07.770 DEBUG:boto:Token: None 2022-09-12T19:07:07.771 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:07.773 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cEsWsSkcOUySUYbmTRdJhnBF0qU= 2022-09-12T19:07:07.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cEsWsSkcOUySUYbmTRdJhnBF0qU=', 'Content-Length': '0'} 2022-09-12T19:07:07.777 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057e12d3aa13da4cb-00631f835b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.779 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:07.781 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:07.782 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:07.783 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:07.784 DEBUG:boto:Method: GET 2022-09-12T19:07:07.785 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:07.786 DEBUG:boto:Data: 2022-09-12T19:07:07.788 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.790 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:07.791 DEBUG:boto:Port: 8001 2022-09-12T19:07:07.792 DEBUG:boto:Params: {} 2022-09-12T19:07:07.793 DEBUG:boto:Token: None 2022-09-12T19:07:07.795 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/ 2022-09-12T19:07:07.796 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ectbrol5GfNgKed38OsF7+Su7m8= 2022-09-12T19:07:07.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ectbrol5GfNgKed38OsF7+Su7m8=', 'Content-Length': '0'} 2022-09-12T19:07:07.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000081f8b37fd5c302b-00631f835b-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.803 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:07.805 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:07.805 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:07.806 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:07.807 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:07.807 DEBUG:boto:Method: GET 2022-09-12T19:07:07.808 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:07.809 DEBUG:boto:Data: 2022-09-12T19:07:07.810 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.811 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:07.811 DEBUG:boto:Port: 8000 2022-09-12T19:07:07.812 DEBUG:boto:Params: {} 2022-09-12T19:07:07.813 DEBUG:boto:Token: None 2022-09-12T19:07:07.814 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/ 2022-09-12T19:07:07.815 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ectbrol5GfNgKed38OsF7+Su7m8= 2022-09-12T19:07:07.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ectbrol5GfNgKed38OsF7+Su7m8=', 'Content-Length': '0'} 2022-09-12T19:07:07.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b1a3e848395a650-00631f835b-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.820 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:07.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:07.822 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.823 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.824 DEBUG:boto:path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:07:07.825 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:07:07.826 DEBUG:boto:Method: GET 2022-09-12T19:07:07.827 DEBUG:boto:Path: /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:07:07.828 DEBUG:boto:Data: 2022-09-12T19:07:07.829 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.831 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:07.832 DEBUG:boto:Port: 8001 2022-09-12T19:07:07.833 DEBUG:boto:Params: {} 2022-09-12T19:07:07.834 DEBUG:boto:Token: None 2022-09-12T19:07:07.835 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:07:07.836 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aG4D5OIvsEvfTXguDADz8bSpa2s= 2022-09-12T19:07:07.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aG4D5OIvsEvfTXguDADz8bSpa2s=', 'Content-Length': '0'} 2022-09-12T19:07:07.841 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'JcC9N13lklBVOx234Q8MXQ8kAM8ByWg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a28567fdbec97bab-00631f835b-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.843 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.843 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.844 DEBUG:boto:path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:07:07.845 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:07:07.846 DEBUG:boto:Method: GET 2022-09-12T19:07:07.847 DEBUG:boto:Path: /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:07:07.848 DEBUG:boto:Data: 2022-09-12T19:07:07.849 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.851 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:07.852 DEBUG:boto:Port: 8000 2022-09-12T19:07:07.853 DEBUG:boto:Params: {} 2022-09-12T19:07:07.854 DEBUG:boto:Token: None 2022-09-12T19:07:07.856 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:07:07.857 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aG4D5OIvsEvfTXguDADz8bSpa2s= 2022-09-12T19:07:07.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aG4D5OIvsEvfTXguDADz8bSpa2s=', 'Content-Length': '0'} 2022-09-12T19:07:07.861 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'JcC9N13lklBVOx234Q8MXQ8kAM8ByWg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000947b49f2027c99cc-00631f835b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.862 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.863 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.864 DEBUG:boto:path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:07:07.864 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:07:07.865 DEBUG:boto:Method: HEAD 2022-09-12T19:07:07.866 DEBUG:boto:Path: /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:07:07.866 DEBUG:boto:Data: 2022-09-12T19:07:07.867 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.867 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:07.868 DEBUG:boto:Port: 8001 2022-09-12T19:07:07.869 DEBUG:boto:Params: {} 2022-09-12T19:07:07.870 DEBUG:boto:Token: None 2022-09-12T19:07:07.871 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:07:07.871 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:muowIXm/J4QsPe8JzLNmqdQbo8A= 2022-09-12T19:07:07.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:muowIXm/J4QsPe8JzLNmqdQbo8A=', 'Content-Length': '0'} 2022-09-12T19:07:07.874 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'JcC9N13lklBVOx234Q8MXQ8kAM8ByWg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000141a02389ea7e419-00631f835b-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.875 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.875 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.876 DEBUG:boto:path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:07:07.876 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:07:07.877 DEBUG:boto:Method: HEAD 2022-09-12T19:07:07.877 DEBUG:boto:Path: /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:07:07.878 DEBUG:boto:Data: 2022-09-12T19:07:07.878 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.879 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:07.879 DEBUG:boto:Port: 8000 2022-09-12T19:07:07.880 DEBUG:boto:Params: {} 2022-09-12T19:07:07.880 DEBUG:boto:Token: None 2022-09-12T19:07:07.881 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:07:07.882 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:muowIXm/J4QsPe8JzLNmqdQbo8A= 2022-09-12T19:07:07.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:muowIXm/J4QsPe8JzLNmqdQbo8A=', 'Content-Length': '0'} 2022-09-12T19:07:07.885 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'JcC9N13lklBVOx234Q8MXQ8kAM8ByWg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ff21836fe607d7f-00631f835b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:07.886 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.887 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.888 DEBUG:boto:path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:07:07.889 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:07:07.890 DEBUG:boto:Method: GET 2022-09-12T19:07:07.891 DEBUG:boto:Path: /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:07:07.892 DEBUG:boto:Data: 2022-09-12T19:07:07.892 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.893 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:07.894 DEBUG:boto:Port: 8001 2022-09-12T19:07:07.895 DEBUG:boto:Params: {} 2022-09-12T19:07:07.897 DEBUG:boto:Token: None 2022-09-12T19:07:07.898 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:07:07.900 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aG4D5OIvsEvfTXguDADz8bSpa2s= 2022-09-12T19:07:07.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aG4D5OIvsEvfTXguDADz8bSpa2s=', 'Content-Length': '0'} 2022-09-12T19:07:07.904 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'JcC9N13lklBVOx234Q8MXQ8kAM8ByWg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086eccc4c2d0047e8-00631f835b-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.946 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:07.947 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:07.948 DEBUG:boto:path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:07:07.949 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:07:07.949 DEBUG:boto:Method: GET 2022-09-12T19:07:07.950 DEBUG:boto:Path: /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:07:07.950 DEBUG:boto:Data: 2022-09-12T19:07:07.951 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.952 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:07.952 DEBUG:boto:Port: 8000 2022-09-12T19:07:07.953 DEBUG:boto:Params: {} 2022-09-12T19:07:07.954 DEBUG:boto:Token: None 2022-09-12T19:07:07.954 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/obj?versionId=JcC9N13lklBVOx234Q8MXQ8kAM8ByWg 2022-09-12T19:07:07.955 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aG4D5OIvsEvfTXguDADz8bSpa2s= 2022-09-12T19:07:07.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aG4D5OIvsEvfTXguDADz8bSpa2s=', 'Content-Length': '0'} 2022-09-12T19:07:07.959 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'JcC9N13lklBVOx234Q8MXQ8kAM8ByWg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9394882faad5dc2-00631f835b-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.960 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:07.960 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:07.961 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:07.962 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:07.962 DEBUG:boto:Method: GET 2022-09-12T19:07:07.963 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:07.963 DEBUG:boto:Data: 2022-09-12T19:07:07.964 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.964 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:07.965 DEBUG:boto:Port: 8001 2022-09-12T19:07:07.966 DEBUG:boto:Params: {} 2022-09-12T19:07:07.967 DEBUG:boto:Token: None 2022-09-12T19:07:07.968 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/ 2022-09-12T19:07:07.968 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ectbrol5GfNgKed38OsF7+Su7m8= 2022-09-12T19:07:07.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ectbrol5GfNgKed38OsF7+Su7m8=', 'Content-Length': '0'} 2022-09-12T19:07:07.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005052f31f2a647cf2-00631f835b-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:07.973 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:07.975 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:07.976 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:07.976 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:07.977 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:07.977 DEBUG:boto:Method: GET 2022-09-12T19:07:07.978 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:07.979 DEBUG:boto:Data: 2022-09-12T19:07:07.980 DEBUG:boto:Headers: {} 2022-09-12T19:07:07.981 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:07.982 DEBUG:boto:Port: 8000 2022-09-12T19:07:07.983 DEBUG:boto:Params: {} 2022-09-12T19:07:07.984 DEBUG:boto:Token: None 2022-09-12T19:07:07.984 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:07 GMT /jdpiak-19/ 2022-09-12T19:07:07.985 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ectbrol5GfNgKed38OsF7+Su7m8= 2022-09-12T19:07:07.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ectbrol5GfNgKed38OsF7+Su7m8=', 'Content-Length': '0'} 2022-09-12T19:07:07.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001df86524c58f8c16-00631f835b-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:08.030 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:08.031 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:08.032 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:08.032 DEBUG:boto:Method: HEAD 2022-09-12T19:07:08.033 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:08.033 DEBUG:boto:Data: 2022-09-12T19:07:08.034 DEBUG:boto:Headers: {} 2022-09-12T19:07:08.034 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:08.035 DEBUG:boto:Port: 8001 2022-09-12T19:07:08.035 DEBUG:boto:Params: {} 2022-09-12T19:07:08.036 DEBUG:boto:Token: None 2022-09-12T19:07:08.037 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:08 GMT /jdpiak-19/obj 2022-09-12T19:07:08.037 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ByLJsxXURNVqAHy2Eo8QUwzRPpo= 2022-09-12T19:07:08.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ByLJsxXURNVqAHy2Eo8QUwzRPpo=', 'Content-Length': '0'} 2022-09-12T19:07:08.042 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027e83b265f663736-00631f835c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:08.042 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:08.043 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:08.043 DEBUG:boto:Method: HEAD 2022-09-12T19:07:08.044 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:08.045 DEBUG:boto:Data: 2022-09-12T19:07:08.045 DEBUG:boto:Headers: {} 2022-09-12T19:07:08.046 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:08.046 DEBUG:boto:Port: 8000 2022-09-12T19:07:08.047 DEBUG:boto:Params: {} 2022-09-12T19:07:08.047 DEBUG:boto:Token: None 2022-09-12T19:07:08.048 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:08 GMT /jdpiak-19/obj 2022-09-12T19:07:08.048 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ByLJsxXURNVqAHy2Eo8QUwzRPpo= 2022-09-12T19:07:08.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ByLJsxXURNVqAHy2Eo8QUwzRPpo=', 'Content-Length': '0'} 2022-09-12T19:07:08.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000628fe5211fd9cbe1-00631f835c-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:08.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:08.054 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:08.055 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:08.055 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:08.056 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:08.056 DEBUG:boto:Method: GET 2022-09-12T19:07:08.057 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:08.057 DEBUG:boto:Data: 2022-09-12T19:07:08.058 DEBUG:boto:Headers: {} 2022-09-12T19:07:08.059 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:08.059 DEBUG:boto:Port: 8001 2022-09-12T19:07:08.060 DEBUG:boto:Params: {} 2022-09-12T19:07:08.060 DEBUG:boto:Token: None 2022-09-12T19:07:08.061 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:08 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:08.061 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UQuguaEIZnyosQmM2JVj5XC3Cbk= 2022-09-12T19:07:08.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UQuguaEIZnyosQmM2JVj5XC3Cbk=', 'Content-Length': '0'} 2022-09-12T19:07:08.065 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090458135c5385b23-00631f835c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:08.065 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:08.066 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:08.067 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:08.067 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:08.068 DEBUG:boto:Method: GET 2022-09-12T19:07:08.068 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:08.069 DEBUG:boto:Data: 2022-09-12T19:07:08.069 DEBUG:boto:Headers: {} 2022-09-12T19:07:08.070 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:08.071 DEBUG:boto:Port: 8000 2022-09-12T19:07:08.071 DEBUG:boto:Params: {} 2022-09-12T19:07:08.072 DEBUG:boto:Token: None 2022-09-12T19:07:08.072 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:08 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:08.073 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UQuguaEIZnyosQmM2JVj5XC3Cbk= 2022-09-12T19:07:08.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UQuguaEIZnyosQmM2JVj5XC3Cbk=', 'Content-Length': '0'} 2022-09-12T19:07:08.076 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020179067d5b558a1-00631f835c-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:08.077 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:08.077 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:08.078 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:08.079 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:08.079 DEBUG:boto:Method: GET 2022-09-12T19:07:08.080 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:08.080 DEBUG:boto:Data: 2022-09-12T19:07:08.081 DEBUG:boto:Headers: {} 2022-09-12T19:07:08.081 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:08.082 DEBUG:boto:Port: 8001 2022-09-12T19:07:08.082 DEBUG:boto:Params: {} 2022-09-12T19:07:08.083 DEBUG:boto:Token: None 2022-09-12T19:07:08.084 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:08 GMT /jdpiak-19/ 2022-09-12T19:07:08.084 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EYYwIq4SPJTXlXPUacYuR+PWgyU= 2022-09-12T19:07:08.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EYYwIq4SPJTXlXPUacYuR+PWgyU=', 'Content-Length': '0'} 2022-09-12T19:07:08.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000744f3356262b9867-00631f835c-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:08.088 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:08.089 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:08.090 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:08.090 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:08.091 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:08.092 DEBUG:boto:Method: GET 2022-09-12T19:07:08.092 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:08.093 DEBUG:boto:Data: 2022-09-12T19:07:08.093 DEBUG:boto:Headers: {} 2022-09-12T19:07:08.094 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:08.094 DEBUG:boto:Port: 8000 2022-09-12T19:07:08.095 DEBUG:boto:Params: {} 2022-09-12T19:07:08.096 DEBUG:boto:Token: None 2022-09-12T19:07:08.097 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:08 GMT /jdpiak-19/ 2022-09-12T19:07:08.098 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EYYwIq4SPJTXlXPUacYuR+PWgyU= 2022-09-12T19:07:08.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EYYwIq4SPJTXlXPUacYuR+PWgyU=', 'Content-Length': '0'} 2022-09-12T19:07:08.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003aaa45ab16dfa58c-00631f835c-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:08.102 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:08.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:08.103 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:08.104 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:08.104 DEBUG:boto:path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:07:08.105 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:07:08.105 DEBUG:boto:Method: GET 2022-09-12T19:07:08.106 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:07:08.107 DEBUG:boto:Data: 2022-09-12T19:07:08.107 DEBUG:boto:Headers: {} 2022-09-12T19:07:08.108 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:08.108 DEBUG:boto:Port: 8001 2022-09-12T19:07:08.109 DEBUG:boto:Params: {} 2022-09-12T19:07:08.109 DEBUG:boto:Token: None 2022-09-12T19:07:08.110 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:08 GMT /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:07:08.111 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e1lFU6UpTtzL8iziRUQqYaHxsfE= 2022-09-12T19:07:08.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e1lFU6UpTtzL8iziRUQqYaHxsfE=', 'Content-Length': '0'} 2022-09-12T19:07:08.114 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000408714d646b6d0a4-00631f835c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:08.115 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:08.115 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:08.116 DEBUG:boto:path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:07:08.117 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:07:08.117 DEBUG:boto:Method: GET 2022-09-12T19:07:08.118 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:07:08.119 DEBUG:boto:Data: 2022-09-12T19:07:08.120 DEBUG:boto:Headers: {} 2022-09-12T19:07:08.120 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:08.121 DEBUG:boto:Port: 8000 2022-09-12T19:07:08.121 DEBUG:boto:Params: {} 2022-09-12T19:07:08.122 DEBUG:boto:Token: None 2022-09-12T19:07:08.123 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:08 GMT /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:07:08.123 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e1lFU6UpTtzL8iziRUQqYaHxsfE= 2022-09-12T19:07:08.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e1lFU6UpTtzL8iziRUQqYaHxsfE=', 'Content-Length': '0'} 2022-09-12T19:07:08.126 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0304efd866645df-00631f835c-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:08.127 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:08.128 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:08.128 DEBUG:boto:path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:07:08.129 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:07:08.129 DEBUG:boto:Method: HEAD 2022-09-12T19:07:08.130 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:07:08.130 DEBUG:boto:Data: 2022-09-12T19:07:08.131 DEBUG:boto:Headers: {} 2022-09-12T19:07:08.131 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:08.132 DEBUG:boto:Port: 8001 2022-09-12T19:07:08.133 DEBUG:boto:Params: {} 2022-09-12T19:07:08.134 DEBUG:boto:Token: None 2022-09-12T19:07:08.135 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:08 GMT /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:07:08.136 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FhX3pqcpqDS4HB7/PfPIregl/bI= 2022-09-12T19:07:08.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FhX3pqcpqDS4HB7/PfPIregl/bI=', 'Content-Length': '0'} 2022-09-12T19:07:08.140 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000701584af691fe471-00631f835c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:08.141 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:08.142 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:08.143 DEBUG:boto:path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:07:08.143 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:07:08.144 DEBUG:boto:Method: HEAD 2022-09-12T19:07:08.145 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:07:08.145 DEBUG:boto:Data: 2022-09-12T19:07:08.146 DEBUG:boto:Headers: {} 2022-09-12T19:07:08.146 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:08.147 DEBUG:boto:Port: 8000 2022-09-12T19:07:08.148 DEBUG:boto:Params: {} 2022-09-12T19:07:08.148 DEBUG:boto:Token: None 2022-09-12T19:07:08.149 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:08 GMT /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:07:08.150 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FhX3pqcpqDS4HB7/PfPIregl/bI= 2022-09-12T19:07:08.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FhX3pqcpqDS4HB7/PfPIregl/bI=', 'Content-Length': '0'} 2022-09-12T19:07:08.153 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c2afb828320c95a-00631f835c-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:08.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:08.154 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:08.155 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:08.155 DEBUG:boto:path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:07:08.156 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:07:08.156 DEBUG:boto:Method: GET 2022-09-12T19:07:08.158 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:07:08.159 DEBUG:boto:Data: 2022-09-12T19:07:08.160 DEBUG:boto:Headers: {} 2022-09-12T19:07:08.161 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:08.162 DEBUG:boto:Port: 8001 2022-09-12T19:07:08.163 DEBUG:boto:Params: {} 2022-09-12T19:07:08.164 DEBUG:boto:Token: None 2022-09-12T19:07:08.165 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:08 GMT /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:07:08.166 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e1lFU6UpTtzL8iziRUQqYaHxsfE= 2022-09-12T19:07:08.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e1lFU6UpTtzL8iziRUQqYaHxsfE=', 'Content-Length': '0'} 2022-09-12T19:07:08.170 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002bf8adf2f6e651a4-00631f835c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:08.210 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:08.211 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:08.212 DEBUG:boto:path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:07:08.213 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:07:08.214 DEBUG:boto:Method: GET 2022-09-12T19:07:08.215 DEBUG:boto:Path: /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:07:08.217 DEBUG:boto:Data: 2022-09-12T19:07:08.218 DEBUG:boto:Headers: {} 2022-09-12T19:07:08.219 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:08.221 DEBUG:boto:Port: 8000 2022-09-12T19:07:08.222 DEBUG:boto:Params: {} 2022-09-12T19:07:08.223 DEBUG:boto:Token: None 2022-09-12T19:07:08.225 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:08 GMT /jdpiak-19/obj?versionId=ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY 2022-09-12T19:07:08.226 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e1lFU6UpTtzL8iziRUQqYaHxsfE= 2022-09-12T19:07:08.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e1lFU6UpTtzL8iziRUQqYaHxsfE=', 'Content-Length': '0'} 2022-09-12T19:07:08.230 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:06 GMT'), ('x-amz-version-id', 'ipYOl-Gqt65Y3nx.nVL2kL2DrysHNSY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ebf9f812a036f108-00631f835c-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:08.232 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:08.232 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:08.233 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:08.234 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:08.235 DEBUG:boto:Method: GET 2022-09-12T19:07:08.236 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:08.237 DEBUG:boto:Data: 2022-09-12T19:07:08.238 DEBUG:boto:Headers: {} 2022-09-12T19:07:08.240 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:08.241 DEBUG:boto:Port: 8001 2022-09-12T19:07:08.242 DEBUG:boto:Params: {} 2022-09-12T19:07:08.243 DEBUG:boto:Token: None 2022-09-12T19:07:08.245 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:08 GMT /jdpiak-19/ 2022-09-12T19:07:08.246 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EYYwIq4SPJTXlXPUacYuR+PWgyU= 2022-09-12T19:07:08.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EYYwIq4SPJTXlXPUacYuR+PWgyU=', 'Content-Length': '0'} 2022-09-12T19:07:08.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007c4d424777449ab-00631f835c-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:08.252 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:08.253 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:08.254 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:08.254 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:08.255 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:08.255 DEBUG:boto:Method: GET 2022-09-12T19:07:08.256 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:08.257 DEBUG:boto:Data: 2022-09-12T19:07:08.257 DEBUG:boto:Headers: {} 2022-09-12T19:07:08.258 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:08.259 DEBUG:boto:Port: 8000 2022-09-12T19:07:08.260 DEBUG:boto:Params: {} 2022-09-12T19:07:08.261 DEBUG:boto:Token: None 2022-09-12T19:07:08.262 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:08 GMT /jdpiak-19/ 2022-09-12T19:07:08.263 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EYYwIq4SPJTXlXPUacYuR+PWgyU= 2022-09-12T19:07:08.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EYYwIq4SPJTXlXPUacYuR+PWgyU=', 'Content-Length': '0'} 2022-09-12T19:07:08.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032f0bf9ceba571ee-00631f835c-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:08.310 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:08.311 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:08.312 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:08.312 DEBUG:boto:Method: HEAD 2022-09-12T19:07:08.313 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:08.313 DEBUG:boto:Data: 2022-09-12T19:07:08.314 DEBUG:boto:Headers: {} 2022-09-12T19:07:08.314 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:08.315 DEBUG:boto:Port: 8001 2022-09-12T19:07:08.315 DEBUG:boto:Params: {} 2022-09-12T19:07:08.316 DEBUG:boto:Token: None 2022-09-12T19:07:08.316 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:08 GMT /jdpiak-19/obj 2022-09-12T19:07:08.317 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ByLJsxXURNVqAHy2Eo8QUwzRPpo= 2022-09-12T19:07:08.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ByLJsxXURNVqAHy2Eo8QUwzRPpo=', 'Content-Length': '0'} 2022-09-12T19:07:08.322 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2cac63bb2112867-00631f835c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:08.322 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:08.323 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:08.324 DEBUG:boto:Method: HEAD 2022-09-12T19:07:08.324 DEBUG:boto:Path: /jdpiak-19/obj 2022-09-12T19:07:08.325 DEBUG:boto:Data: 2022-09-12T19:07:08.326 DEBUG:boto:Headers: {} 2022-09-12T19:07:08.326 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:08.327 DEBUG:boto:Port: 8000 2022-09-12T19:07:08.327 DEBUG:boto:Params: {} 2022-09-12T19:07:08.328 DEBUG:boto:Token: None 2022-09-12T19:07:08.329 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:08 GMT /jdpiak-19/obj 2022-09-12T19:07:08.329 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ByLJsxXURNVqAHy2Eo8QUwzRPpo= 2022-09-12T19:07:08.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ByLJsxXURNVqAHy2Eo8QUwzRPpo=', 'Content-Length': '0'} 2022-09-12T19:07:08.333 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e17846cfb90d17e-00631f835c-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:08.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:08.335 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:08.336 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:08.336 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:08.337 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:08.338 DEBUG:boto:Method: GET 2022-09-12T19:07:08.339 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:08.339 DEBUG:boto:Data: 2022-09-12T19:07:08.340 DEBUG:boto:Headers: {} 2022-09-12T19:07:08.340 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:08.341 DEBUG:boto:Port: 8001 2022-09-12T19:07:08.342 DEBUG:boto:Params: {} 2022-09-12T19:07:08.342 DEBUG:boto:Token: None 2022-09-12T19:07:08.343 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:08 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:08.344 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UQuguaEIZnyosQmM2JVj5XC3Cbk= 2022-09-12T19:07:08.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UQuguaEIZnyosQmM2JVj5XC3Cbk=', 'Content-Length': '0'} 2022-09-12T19:07:08.347 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c45c3e53f6925a2f-00631f835c-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:08.347 DEBUG:boto:path=/jdpiak-19/obj 2022-09-12T19:07:08.348 DEBUG:boto:auth_path=/jdpiak-19/obj 2022-09-12T19:07:08.349 DEBUG:boto:path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:08.349 DEBUG:boto:auth_path=/jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:08.350 DEBUG:boto:Method: GET 2022-09-12T19:07:08.350 DEBUG:boto:Path: /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:08.351 DEBUG:boto:Data: 2022-09-12T19:07:08.352 DEBUG:boto:Headers: {} 2022-09-12T19:07:08.352 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:08.353 DEBUG:boto:Port: 8000 2022-09-12T19:07:08.353 DEBUG:boto:Params: {} 2022-09-12T19:07:08.354 DEBUG:boto:Token: None 2022-09-12T19:07:08.355 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:08 GMT /jdpiak-19/obj?versionId=AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq 2022-09-12T19:07:08.355 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UQuguaEIZnyosQmM2JVj5XC3Cbk= 2022-09-12T19:07:08.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UQuguaEIZnyosQmM2JVj5XC3Cbk=', 'Content-Length': '0'} 2022-09-12T19:07:08.358 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:05:07 GMT'), ('x-amz-version-id', 'AVsSpje6MdNGaPZ3oG9lpWJ.mI2tvZq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036d1a4441fa9032e-00631f835c-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:08.359 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:08.360 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:08.361 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:08.362 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:08.363 DEBUG:boto:Method: GET 2022-09-12T19:07:08.364 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:08.364 DEBUG:boto:Data: 2022-09-12T19:07:08.365 DEBUG:boto:Headers: {} 2022-09-12T19:07:08.365 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:08.366 DEBUG:boto:Port: 8001 2022-09-12T19:07:08.366 DEBUG:boto:Params: {} 2022-09-12T19:07:08.367 DEBUG:boto:Token: None 2022-09-12T19:07:08.368 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:08 GMT /jdpiak-19/ 2022-09-12T19:07:08.368 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EYYwIq4SPJTXlXPUacYuR+PWgyU= 2022-09-12T19:07:08.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EYYwIq4SPJTXlXPUacYuR+PWgyU=', 'Content-Length': '0'} 2022-09-12T19:07:08.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ed2d3da4b5c7b1d-00631f835c-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:08.373 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:08.375 DEBUG:boto:path=/jdpiak-19/ 2022-09-12T19:07:08.376 DEBUG:boto:auth_path=/jdpiak-19/ 2022-09-12T19:07:08.377 DEBUG:boto:path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:08.378 DEBUG:boto:auth_path=/jdpiak-19/?prefix=obj 2022-09-12T19:07:08.379 DEBUG:boto:Method: GET 2022-09-12T19:07:08.379 DEBUG:boto:Path: /jdpiak-19/?prefix=obj 2022-09-12T19:07:08.380 DEBUG:boto:Data: 2022-09-12T19:07:08.381 DEBUG:boto:Headers: {} 2022-09-12T19:07:08.381 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:08.382 DEBUG:boto:Port: 8000 2022-09-12T19:07:08.382 DEBUG:boto:Params: {} 2022-09-12T19:07:08.383 DEBUG:boto:Token: None 2022-09-12T19:07:08.384 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:08 GMT /jdpiak-19/ 2022-09-12T19:07:08.384 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EYYwIq4SPJTXlXPUacYuR+PWgyU= 2022-09-12T19:07:08.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EYYwIq4SPJTXlXPUacYuR+PWgyU=', 'Content-Length': '0'} 2022-09-12T19:07:08.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f80ff539df332d7-00631f835c-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:08.388 DEBUG:boto:b'jdpiak-19obj1000falseobj2022-09-12T19:05:07.549Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:08.389 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-19 zones={test-zone3, test-zone2} 2022-09-12T19:07:08.390 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2022-09-12T19:07:08.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:08.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:08.392 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:07:08.450 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:08.450 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:08.518 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:07:08.518 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:08.519 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-12T19:07:08.519 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:03:01.269036Z" 2022-09-12T19:07:08.520 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:08.520 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:08.521 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000001499", 2022-09-12T19:07:08.521 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:06:24.886577Z" 2022-09-12T19:07:08.522 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:08.522 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:08.523 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-12T19:07:08.523 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:04:56.879502Z" 2022-09-12T19:07:08.524 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:08.524 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:08.525 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:07:08.525 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:07:08.526 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:08.526 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:07:08.529 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006434', 3: ''} 2022-09-12T19:07:08.529 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2022-09-12T19:07:08.531 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:08.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:08.533 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:07:08.591 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:08.592 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:08.641 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:07:08.642 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:07:08.642 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:07:08.643 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:07:08.643 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:07:08.643 INFO:teuthology.orchestra.run.smithi138.stdout: "instance_id": 4124212413593488935 2022-09-12T19:07:08.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:08.644 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:07:08.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:08.644 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:07:08.645 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:08.645 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:07:08.645 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-12T19:07:08.645 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:08.646 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:07:08.646 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:08.646 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:03:01.269036Z" 2022-09-12T19:07:08.646 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:08.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:08.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:08.647 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:07:08.648 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:08.648 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:07:08.648 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000001499", 2022-09-12T19:07:08.648 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:08.649 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:07:08.649 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:08.649 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:06:24.886577Z" 2022-09-12T19:07:08.650 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:08.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:08.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:08.651 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:07:08.651 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:08.652 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:07:08.652 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-12T19:07:08.653 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:08.653 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:07:08.653 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:08.653 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:04:56.879502Z" 2022-09-12T19:07:08.654 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:08.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:08.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:08.654 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:07:08.655 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:08.655 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:07:08.655 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:07:08.655 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:08.656 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:07:08.656 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:08.656 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000" 2022-09-12T19:07:08.656 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:08.657 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:08.657 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:07:08.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:08.657 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:07:08.658 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 0, 2022-09-12T19:07:08.658 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 0 2022-09-12T19:07:08.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:08.659 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:07:08Z" 2022-09-12T19:07:08.659 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:07:08.660 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 4124212413593488935 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:03:01.269036Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:06:24.886577Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:04:56.879502Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:07:08Z" } 2022-09-12T19:07:08.661 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:03:01.269036Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:06:24.886577Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:04:56.879502Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2022-09-12T19:07:08.661 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006434', 3: ''} 2022-09-12T19:07:08.662 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006434', 3: ''} 2022-09-12T19:07:08.662 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2022-09-12T19:07:08.663 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone3 2022-09-12T19:07:08.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:08.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:08.665 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:07:08.716 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:08.717 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:08.764 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:07:08.765 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:08.765 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-12T19:07:08.765 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:03:01.269036Z" 2022-09-12T19:07:08.766 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:08.766 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:08.766 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000001499", 2022-09-12T19:07:08.767 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:06:24.886577Z" 2022-09-12T19:07:08.767 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:08.767 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:08.767 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-12T19:07:08.768 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:04:56.879502Z" 2022-09-12T19:07:08.768 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:08.768 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:08.768 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:07:08.768 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:07:08.769 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:08.769 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:07:08.770 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006434', 3: ''} 2022-09-12T19:07:08.771 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2022-09-12T19:07:08.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:08.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:08.772 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:07:08.825 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:08.825 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:08.869 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:07:08.870 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:07:08.870 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:07:08.871 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:07:08.871 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:07:08.871 INFO:teuthology.orchestra.run.smithi035.stdout: "instance_id": 16681684026229800507 2022-09-12T19:07:08.871 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:08.872 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:07:08.872 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:08.872 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:07:08.872 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:08.873 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:07:08.873 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-12T19:07:08.873 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:08.874 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:07:08.874 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:08.874 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:03:01.269036Z" 2022-09-12T19:07:08.875 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:08.875 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:08.875 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:08.876 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:07:08.876 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:08.876 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:07:08.877 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000001499", 2022-09-12T19:07:08.877 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:08.877 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:07:08.877 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:08.878 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:06:24.886577Z" 2022-09-12T19:07:08.878 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:08.878 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:08.879 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:08.879 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:07:08.879 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:08.880 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:07:08.880 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-12T19:07:08.880 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:08.880 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:07:08.881 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:08.881 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:04:56.879502Z" 2022-09-12T19:07:08.881 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:08.881 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:08.882 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:08.882 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:07:08.882 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:08.883 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:07:08.883 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:07:08.883 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:08.883 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:07:08.884 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:08.884 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000" 2022-09-12T19:07:08.884 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:08.884 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:08.885 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:07:08.885 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:08.885 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:07:08.886 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 0, 2022-09-12T19:07:08.886 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 0 2022-09-12T19:07:08.886 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:08.887 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:07:08Z" 2022-09-12T19:07:08.887 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:07:08.888 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16681684026229800507 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:03:01.269036Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:06:24.886577Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:04:56.879502Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:07:08Z" } 2022-09-12T19:07:08.889 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:03:01.269036Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:06:24.886577Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:04:56.879502Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2022-09-12T19:07:08.889 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006434', 3: ''} 2022-09-12T19:07:08.890 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006434', 3: ''} 2022-09-12T19:07:08.890 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2022-09-12T19:07:08.891 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone4 2022-09-12T19:07:08.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:08.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:08.892 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:07:08.942 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:08.942 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:09.002 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:07:09.003 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:09.003 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-12T19:07:09.003 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:03:01.269036Z" 2022-09-12T19:07:09.004 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:09.004 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:09.005 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000001499", 2022-09-12T19:07:09.005 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:06:24.886577Z" 2022-09-12T19:07:09.005 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:09.006 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:09.006 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-12T19:07:09.006 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:04:56.879502Z" 2022-09-12T19:07:09.006 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:09.007 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:09.007 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:07:09.007 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:07:09.007 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:09.008 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:07:09.009 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006434', 3: ''} 2022-09-12T19:07:09.010 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2022-09-12T19:07:09.011 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:09.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:09.012 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:07:09.072 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:09.072 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:09.131 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:07:09.132 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:07:09.132 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:07:09.132 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:07:09.132 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:07:09.133 INFO:teuthology.orchestra.run.smithi138.stdout: "instance_id": 14449053848353038014 2022-09-12T19:07:09.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:09.133 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:07:09.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:09.134 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:07:09.134 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:09.134 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:07:09.135 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-12T19:07:09.135 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:09.135 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:07:09.136 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:09.136 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:03:01.269036Z" 2022-09-12T19:07:09.136 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:09.137 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:09.137 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:09.137 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:07:09.138 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:09.138 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:07:09.138 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000001499", 2022-09-12T19:07:09.138 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:09.139 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:07:09.139 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:09.139 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:06:24.886577Z" 2022-09-12T19:07:09.140 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:09.140 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:09.140 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:09.140 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:07:09.141 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:09.141 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:07:09.141 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-12T19:07:09.142 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:09.142 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:07:09.143 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:09.143 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:04:56.879502Z" 2022-09-12T19:07:09.143 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:09.143 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:09.144 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:09.144 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:07:09.144 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:09.144 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:07:09.145 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:07:09.145 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:09.145 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:07:09.146 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:09.146 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000" 2022-09-12T19:07:09.146 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:09.146 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:09.147 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:07:09.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:09.148 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:07:09.148 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 0, 2022-09-12T19:07:09.148 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 0 2022-09-12T19:07:09.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:09.149 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:07:09Z" 2022-09-12T19:07:09.149 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:07:09.150 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14449053848353038014 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:03:01.269036Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:06:24.886577Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:04:56.879502Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:07:09Z" } 2022-09-12T19:07:09.151 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:03:01.269036Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:06:24.886577Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:04:56.879502Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2022-09-12T19:07:09.152 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006434', 3: ''} 2022-09-12T19:07:09.153 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006434', 3: ''} 2022-09-12T19:07:09.153 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2022-09-12T19:07:09.154 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2022-09-12T19:07:09.154 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:09.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:09.156 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:07:09.205 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:09.206 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:09.268 INFO:teuthology.orchestra.run.smithi138.stdout:[ 2022-09-12T19:07:09.269 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:09.269 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-12T19:07:09.269 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:02:32.028847Z" 2022-09-12T19:07:09.270 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:09.270 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:09.270 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000001358", 2022-09-12T19:07:09.271 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:06:02.695476Z" 2022-09-12T19:07:09.271 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:09.271 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:09.271 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-12T19:07:09.272 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:04:34.689861Z" 2022-09-12T19:07:09.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:09.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:09.272 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:07:09.273 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "0.000000" 2022-09-12T19:07:09.273 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:09.273 INFO:teuthology.orchestra.run.smithi138.stdout:] 2022-09-12T19:07:09.274 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001358', 2: '00000000000000000000:00000000000000006434', 3: ''} 2022-09-12T19:07:09.275 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2022-09-12T19:07:09.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:09.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:09.277 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:07:09.330 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:09.331 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:09.393 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:07:09.394 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:07:09.394 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:07:09.395 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:07:09.395 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:07:09.396 INFO:teuthology.orchestra.run.smithi035.stdout: "instance_id": 2819386190429732693 2022-09-12T19:07:09.396 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:09.397 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:07:09.397 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:09.398 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:07:09.399 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:09.399 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:07:09.400 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-12T19:07:09.400 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:09.401 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:07:09.401 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:09.402 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:02:32.028847Z" 2022-09-12T19:07:09.403 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:09.403 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:09.404 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:09.404 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:07:09.405 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:09.405 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:07:09.405 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000001358", 2022-09-12T19:07:09.405 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:09.406 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:07:09.406 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:09.406 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:06:02.695476Z" 2022-09-12T19:07:09.406 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:09.407 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:09.407 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:09.407 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:07:09.408 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:09.408 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:07:09.408 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-12T19:07:09.408 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:09.409 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:07:09.409 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:09.409 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:04:34.689861Z" 2022-09-12T19:07:09.409 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:09.410 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:09.410 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:09.410 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:07:09.410 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:09.411 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:07:09.411 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:07:09.411 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:09.412 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:07:09.412 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:09.412 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000" 2022-09-12T19:07:09.413 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:09.413 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:09.413 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:07:09.413 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:09.414 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:07:09.414 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 0, 2022-09-12T19:07:09.414 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 0 2022-09-12T19:07:09.414 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:09.415 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:07:09Z" 2022-09-12T19:07:09.415 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:07:09.416 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 2819386190429732693 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002486", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:02:32.028847Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001358", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:06:02.695476Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:04:34.689861Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:07:09Z" } 2022-09-12T19:07:09.417 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:02:32.028847Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001358', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:06:02.695476Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:04:34.689861Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2022-09-12T19:07:09.418 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001358', 2: '00000000000000000000:00000000000000006434', 3: ''} 2022-09-12T19:07:09.418 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001358', 2: '00000000000000000000:00000000000000006434', 3: ''} 2022-09-12T19:07:09.419 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2022-09-12T19:07:09.419 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone3 2022-09-12T19:07:09.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:09.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:09.421 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:07:09.470 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:09.471 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:09.521 INFO:teuthology.orchestra.run.smithi138.stdout:[ 2022-09-12T19:07:09.521 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:09.522 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-12T19:07:09.522 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:02:32.028847Z" 2022-09-12T19:07:09.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:09.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:09.523 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000001358", 2022-09-12T19:07:09.523 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:06:02.695476Z" 2022-09-12T19:07:09.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:09.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:09.524 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-12T19:07:09.524 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:04:34.689861Z" 2022-09-12T19:07:09.525 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:09.525 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:09.525 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:07:09.525 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "0.000000" 2022-09-12T19:07:09.526 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:09.526 INFO:teuthology.orchestra.run.smithi138.stdout:] 2022-09-12T19:07:09.528 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001358', 2: '00000000000000000000:00000000000000006434', 3: ''} 2022-09-12T19:07:09.528 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2022-09-12T19:07:09.529 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:09.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:09.530 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:07:09.583 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:09.583 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:09.634 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:07:09.635 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:07:09.635 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:07:09.635 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:07:09.635 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:07:09.636 INFO:teuthology.orchestra.run.smithi035.stdout: "instance_id": 16602476873656756125 2022-09-12T19:07:09.636 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:09.636 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:07:09.637 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:09.637 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:07:09.637 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:09.637 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:07:09.638 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-12T19:07:09.638 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:09.638 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:07:09.638 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:09.639 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:02:32.028847Z" 2022-09-12T19:07:09.639 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:09.639 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:09.640 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:09.640 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:07:09.640 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:09.640 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:07:09.641 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000001358", 2022-09-12T19:07:09.641 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:09.641 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:07:09.641 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:09.642 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:06:02.695476Z" 2022-09-12T19:07:09.642 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:09.642 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:09.643 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:09.643 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:07:09.643 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:09.643 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:07:09.644 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-12T19:07:09.644 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:09.644 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:07:09.644 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:09.645 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:04:34.689861Z" 2022-09-12T19:07:09.645 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:09.645 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:09.645 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:09.646 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:07:09.646 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:09.646 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:07:09.646 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:07:09.647 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:09.647 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:07:09.647 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:09.647 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000" 2022-09-12T19:07:09.648 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:09.648 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:09.648 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:07:09.648 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:09.649 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:07:09.649 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 0, 2022-09-12T19:07:09.649 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 0 2022-09-12T19:07:09.649 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:09.650 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:07:09Z" 2022-09-12T19:07:09.650 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:07:09.652 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16602476873656756125 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002486", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:02:32.028847Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001358", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:06:02.695476Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:04:34.689861Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:07:09Z" } 2022-09-12T19:07:09.652 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:02:32.028847Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001358', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:06:02.695476Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:04:34.689861Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2022-09-12T19:07:09.653 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001358', 2: '00000000000000000000:00000000000000006434', 3: ''} 2022-09-12T19:07:09.653 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001358', 2: '00000000000000000000:00000000000000006434', 3: ''} 2022-09-12T19:07:09.654 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2022-09-12T19:07:09.654 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone4 2022-09-12T19:07:09.655 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone1 2022-09-12T19:07:09.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:09.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:09.657 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:07:09.707 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:09.707 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:09.753 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:07:09.753 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:09.753 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002768", 2022-09-12T19:07:09.754 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:02:41.932652Z" 2022-09-12T19:07:09.754 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:09.754 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:09.754 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000001499", 2022-09-12T19:07:09.755 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:06:24.730144Z" 2022-09-12T19:07:09.755 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:09.755 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:09.756 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-12T19:07:09.756 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:04:56.722572Z" 2022-09-12T19:07:09.756 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:09.756 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:09.757 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:07:09.757 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:07:09.757 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:09.758 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:07:09.761 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000002768', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006152', 3: ''} 2022-09-12T19:07:09.762 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2022-09-12T19:07:09.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:09.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:09.763 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:07:09.816 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:09.817 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:09.891 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:07:09.891 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:07:09.891 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:07:09.892 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:07:09.892 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:07:09.892 INFO:teuthology.orchestra.run.smithi035.stdout: "instance_id": 9390097840922254989 2022-09-12T19:07:09.892 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:09.893 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:07:09.893 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:09.893 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:07:09.893 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:09.894 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:07:09.894 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002768", 2022-09-12T19:07:09.894 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:09.894 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:07:09.895 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:09.895 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:02:41.932652Z" 2022-09-12T19:07:09.895 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:09.895 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:09.896 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:09.896 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:07:09.896 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:09.897 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:07:09.897 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000001499", 2022-09-12T19:07:09.897 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:09.897 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:07:09.898 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:09.898 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:06:24.730144Z" 2022-09-12T19:07:09.898 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:09.899 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:09.899 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:09.899 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:07:09.899 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:09.900 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:07:09.900 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-12T19:07:09.900 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:09.900 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:07:09.901 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:09.901 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:04:56.722572Z" 2022-09-12T19:07:09.901 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:09.902 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:09.902 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:09.902 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:07:09.902 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:09.903 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:07:09.903 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:07:09.903 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:09.903 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:07:09.904 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:09.904 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000" 2022-09-12T19:07:09.904 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:09.904 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:09.905 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:07:09.905 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:09.905 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:07:09.905 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 0, 2022-09-12T19:07:09.906 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 0 2022-09-12T19:07:09.906 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:09.906 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:07:09Z" 2022-09-12T19:07:09.907 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:07:09.908 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 9390097840922254989 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002768", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:02:41.932652Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:06:24.730144Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006152", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:04:56.722572Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:07:09Z" } 2022-09-12T19:07:09.908 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002768', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:02:41.932652Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:06:24.730144Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006152', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:04:56.722572Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2022-09-12T19:07:09.909 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002768', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006152', 3: ''} 2022-09-12T19:07:09.909 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002768', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006152', 3: ''} 2022-09-12T19:07:09.910 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2022-09-12T19:07:09.910 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone2 2022-09-12T19:07:09.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:09.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:09.912 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:07:09.962 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:09.962 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:10.028 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:07:10.029 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:10.029 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002768", 2022-09-12T19:07:10.030 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:02:41.932652Z" 2022-09-12T19:07:10.030 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:10.030 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:10.031 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000001499", 2022-09-12T19:07:10.031 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:06:24.730144Z" 2022-09-12T19:07:10.031 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:10.031 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:10.032 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-12T19:07:10.032 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:04:56.722572Z" 2022-09-12T19:07:10.032 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:10.032 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:10.033 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:07:10.033 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:07:10.033 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:10.034 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:07:10.035 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000002768', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006152', 3: ''} 2022-09-12T19:07:10.036 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2022-09-12T19:07:10.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:10.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:10.037 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:07:10.089 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:10.089 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:10.135 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:07:10.135 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:07:10.136 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:07:10.136 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:07:10.137 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:07:10.137 INFO:teuthology.orchestra.run.smithi138.stdout: "instance_id": 15445976120917781840 2022-09-12T19:07:10.138 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:10.138 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:07:10.138 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:10.139 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:07:10.139 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:10.140 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:07:10.140 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000002768", 2022-09-12T19:07:10.141 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:10.141 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:07:10.142 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:10.142 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:02:41.932652Z" 2022-09-12T19:07:10.143 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:10.143 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:10.144 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:10.144 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:07:10.144 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:10.145 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:07:10.145 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000001499", 2022-09-12T19:07:10.146 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:10.146 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:07:10.147 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:10.147 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:06:24.730144Z" 2022-09-12T19:07:10.148 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:10.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:10.149 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:10.149 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:07:10.150 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:10.150 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:07:10.151 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-12T19:07:10.151 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:10.152 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:07:10.153 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:10.153 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:04:56.722572Z" 2022-09-12T19:07:10.154 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:10.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:10.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:10.155 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:07:10.156 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:10.156 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:07:10.157 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:07:10.157 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:10.158 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:07:10.159 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:10.160 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000" 2022-09-12T19:07:10.160 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:10.160 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:10.160 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:07:10.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:10.161 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:07:10.161 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 0, 2022-09-12T19:07:10.161 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 0 2022-09-12T19:07:10.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:10.162 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:07:10Z" 2022-09-12T19:07:10.162 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:07:10.164 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 15445976120917781840 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002768", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:02:41.932652Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:06:24.730144Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006152", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:04:56.722572Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:07:10Z" } 2022-09-12T19:07:10.165 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002768', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:02:41.932652Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:06:24.730144Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006152', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:04:56.722572Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2022-09-12T19:07:10.165 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002768', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006152', 3: ''} 2022-09-12T19:07:10.166 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002768', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006152', 3: ''} 2022-09-12T19:07:10.167 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2022-09-12T19:07:10.167 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone4 2022-09-12T19:07:10.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2022-09-12T19:07:10.171 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:10.172 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:10.172 DEBUG:boto:Method: PUT 2022-09-12T19:07:10.173 DEBUG:boto:Path: /jdpiak-20/ 2022-09-12T19:07:10.173 DEBUG:boto:Data: 2022-09-12T19:07:10.174 DEBUG:boto:Headers: {} 2022-09-12T19:07:10.174 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:10.176 DEBUG:boto:Port: 8000 2022-09-12T19:07:10.176 DEBUG:boto:Params: {} 2022-09-12T19:07:10.177 DEBUG:boto:Token: None 2022-09-12T19:07:10.177 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:07:10 GMT /jdpiak-20/ 2022-09-12T19:07:10.178 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dpg3EdJKpub0WbCCMjgh9Q2otso= 2022-09-12T19:07:10.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dpg3EdJKpub0WbCCMjgh9Q2otso=', 'Content-Length': '0'} 2022-09-12T19:07:10.212 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006602aab75a22aec6-00631f835e-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:07:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:10.213 DEBUG:rgw_multi.tests:created bucket=jdpiak-20 2022-09-12T19:07:10.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:10.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:10.215 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:07:10.265 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:10.265 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:10.297 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=c0f7a512-c940-4d1f-8adf-3484630f876b 2022-09-12T19:07:10.309 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:07:10.309 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:10.310 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:07:10.310 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:47:27.976965Z" 2022-09-12T19:07:10.310 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:10.311 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:10.311 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:07:10.311 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:07:10.311 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:10.312 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:10.312 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663009630.200316_196.1", 2022-09-12T19:07:10.312 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:07:10.200316Z" 2022-09-12T19:07:10.313 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:10.313 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:10.313 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:07:10.313 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:49:40.904048Z" 2022-09-12T19:07:10.314 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:10.314 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:07:10.315 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009630.200316_196.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:07:10.316 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T19:07:10.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:10.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:10.317 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:07:10.369 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:10.370 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:10.424 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:07:10.421+0000 7fd79aa80600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:07:10.425 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:07:10.423+0000 7fd79aa80600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:07:10.445 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:07:10.446 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:07:10.446 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:07:10.446 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:07:10.447 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:07:10.447 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:07:10.447 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:07:10.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:10.448 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:07:10.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:10.448 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:07:10.448 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:10.449 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:07:10.449 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:07:10.449 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:10.450 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:07:10.450 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:10.450 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:07:10.450 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:07:10.451 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:10.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:10.451 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:10.451 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:07:10.452 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:10.452 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:07:10.452 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:07:10.453 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:10.453 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:07:10.453 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:10.453 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:07:10.454 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:07:10.454 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:10.454 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:10.455 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:10.455 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:07:10.455 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:10.456 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:07:10.456 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663009630.200316_196.1", 2022-09-12T19:07:10.456 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:10.456 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:07:10.457 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:10.457 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:07:10.200316Z", 2022-09-12T19:07:10.457 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:07:10.458 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:10.458 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:10.458 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:10.459 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:07:10.459 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:10.459 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:07:10.459 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:07:10.460 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:10.460 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:07:10.460 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:10.461 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:07:10.461 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:07:10.461 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:10.461 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:10.462 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:07:10.462 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:10.462 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:07:10.462 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:07:10.463 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:07:10.463 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:10.463 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:07:10Z" 2022-09-12T19:07:10.463 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:07:10.465 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009630.200316_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:07:10.200316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:07:10Z" } 2022-09-12T19:07:10.465 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009630.200316_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:07:10.200316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:07:10.466 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009630.200316_196.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:07:10.467 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009630.200316_196.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:07:10.467 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T19:07:10.468 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T19:07:10.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:10.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:10.469 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:07:10.520 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:10.521 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:10.558 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:07:10.555+0000 7fe01c67b600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:07:10.559 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:07:10.556+0000 7fe01c67b600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:07:10.578 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:07:10.579 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:07:10.579 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:07:10.579 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:07:10.579 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:07:10.580 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:07:10.580 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:07:10.580 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:10.581 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:07:10.581 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:10.581 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:07:10.582 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:10.582 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:07:10.582 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:07:10.582 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:10.583 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:07:10.583 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:10.583 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:07:10.583 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:07:10.584 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:10.584 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:10.584 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:10.584 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:07:10.585 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:10.585 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:07:10.585 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:07:10.585 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:10.586 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:07:10.586 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:10.586 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:07:10.586 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:07:10.587 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:10.587 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:10.587 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:10.587 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:07:10.588 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:10.588 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:07:10.588 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663009630.186894_193.1", 2022-09-12T19:07:10.588 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:10.589 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:07:10.589 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:10.590 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:07:10.186894Z", 2022-09-12T19:07:10.590 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:07:10.590 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:10.590 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:10.591 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:10.591 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:07:10.591 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:10.591 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:07:10.592 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:07:10.592 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:10.592 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:07:10.592 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:10.593 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:07:10.593 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:07:10.593 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:10.593 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:10.594 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:07:10.594 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:10.594 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:07:10.594 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:07:10.595 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:07:10.595 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:10.595 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:07:10Z" 2022-09-12T19:07:10.595 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:07:10.597 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009630.186894_193.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:07:10.186894Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:07:10Z" } 2022-09-12T19:07:10.598 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009630.186894_193.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:07:10.186894Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:07:10.598 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009630.200316_196.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:07:10.599 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009630.186894_193.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:07:10.599 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1663009630.200316_196.1 target=1_1663009630.186894_193.1 2022-09-12T19:07:15.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:15.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:15.603 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:07:15.661 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:15.661 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:15.709 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:07:15.706+0000 7f05ffc94600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:07:15.710 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:07:15.708+0000 7f05ffc94600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:07:15.729 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:07:15.729 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:07:15.730 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:07:15.730 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:07:15.731 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:07:15.731 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:07:15.732 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:07:15.732 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:15.733 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:07:15.733 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:15.734 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:07:15.734 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:15.735 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:07:15.735 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:07:15.736 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:15.736 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:07:15.737 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:15.737 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:07:15.737 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:07:15.738 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:15.738 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:15.739 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:15.739 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:07:15.740 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:15.740 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:07:15.741 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:07:15.741 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:15.741 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:07:15.742 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:15.742 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:07:15.743 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:07:15.743 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:15.744 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:15.744 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:15.745 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:07:15.745 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:15.745 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:07:15.746 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663009630.200316_196.1", 2022-09-12T19:07:15.746 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:15.747 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:07:15.747 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:15.747 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:07:10.200316Z", 2022-09-12T19:07:15.748 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:07:15.748 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:15.749 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:15.749 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:15.749 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:07:15.750 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:15.750 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:07:15.751 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:07:15.751 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:15.752 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:07:15.752 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:15.752 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:07:15.753 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:07:15.753 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:15.754 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:15.754 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:07:15.755 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:15.755 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:07:15.755 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:07:15.756 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:07:15.756 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:15.757 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:07:15Z" 2022-09-12T19:07:15.757 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:07:15.759 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009630.200316_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:07:10.200316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:07:15Z" } 2022-09-12T19:07:15.760 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009630.200316_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:07:10.200316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:07:15.761 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009630.200316_196.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:07:15.762 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009630.200316_196.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:07:15.763 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T19:07:15.764 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T19:07:15.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:15.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:15.766 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:07:15.824 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:15.824 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:15.872 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:07:15.869+0000 7f3def510600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:07:15.873 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:07:15.870+0000 7f3def510600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:07:15.889 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:07:15.889 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:07:15.889 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:07:15.890 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:07:15.890 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:07:15.890 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:07:15.891 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:07:15.891 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:15.891 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:07:15.891 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:15.892 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:07:15.892 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:15.892 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:07:15.893 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:07:15.893 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:15.893 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:07:15.894 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:15.894 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:07:15.894 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:07:15.895 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:15.895 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:15.895 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:15.895 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:07:15.896 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:15.896 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:07:15.896 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:07:15.897 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:15.897 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:07:15.897 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:15.897 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:07:15.898 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:07:15.898 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:15.898 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:15.899 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:15.899 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:07:15.899 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:15.899 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:07:15.900 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663009630.200316_196.1", 2022-09-12T19:07:15.900 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:15.900 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:07:15.901 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:15.901 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:07:10.200316Z", 2022-09-12T19:07:15.901 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:07:15.901 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:15.902 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:15.902 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:15.902 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:07:15.903 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:15.903 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:07:15.903 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:07:15.903 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:15.904 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:07:15.904 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:15.904 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:07:15.904 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:07:15.905 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:15.905 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:15.905 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:07:15.905 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:15.906 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:07:15.906 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:07:15.906 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:07:15.907 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:15.907 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:07:15Z" 2022-09-12T19:07:15.907 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:07:15.909 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009630.200316_196.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:07:10.200316Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:07:15Z" } 2022-09-12T19:07:15.909 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009630.200316_196.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:07:10.200316Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:07:15.910 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009630.200316_196.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:07:15.910 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009630.200316_196.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:07:15.911 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T19:07:15.912 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:15.912 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:15.913 DEBUG:boto:Method: HEAD 2022-09-12T19:07:15.913 DEBUG:boto:Path: /jdpiak-20/ 2022-09-12T19:07:15.914 DEBUG:boto:Data: 2022-09-12T19:07:15.914 DEBUG:boto:Headers: {} 2022-09-12T19:07:15.915 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:15.915 DEBUG:boto:Port: 8000 2022-09-12T19:07:15.916 DEBUG:boto:Params: {} 2022-09-12T19:07:15.916 DEBUG:boto:Token: None 2022-09-12T19:07:15.917 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:15 GMT /jdpiak-20/ 2022-09-12T19:07:15.918 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0qWuteKhv1b980g2axsIK7o8qqo= 2022-09-12T19:07:15.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0qWuteKhv1b980g2axsIK7o8qqo=', 'Content-Length': '0'} 2022-09-12T19:07:15.921 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc47faffe242f007-00631f8363-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:15.922 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:15.923 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:15.923 DEBUG:boto:Method: PUT 2022-09-12T19:07:15.924 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:07:15.924 DEBUG:boto:Data: 2022-09-12T19:07:15.925 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-12T19:07:15.925 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:15.927 DEBUG:boto:Port: 8000 2022-09-12T19:07:15.927 DEBUG:boto:Params: {} 2022-09-12T19:07:15.928 DEBUG:boto:Token: None 2022-09-12T19:07:15.929 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 12 Sep 2022 19:07:15 GMT /jdpiak-20/obj 2022-09-12T19:07:15.929 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KaKaTpo+vU2eiwpWFnePqaq1DDs= 2022-09-12T19:07:15.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:07:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KaKaTpo+vU2eiwpWFnePqaq1DDs='} 2022-09-12T19:07:15.937 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f67b8a14777bdb3-00631f8363-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:15.938 DEBUG:rgw_multi.tests:created initial version id=None 2022-09-12T19:07:15.939 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-20 2022-09-12T19:07:15.940 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:07:15.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:07:15.941 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:07:15.998 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:15.999 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:16.032 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:07:16.029+0000 7f8c5a5cc600 1 waiting to reach incremental sync.. 2022-09-12T19:07:21.033 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:07:21.030+0000 7f8c5a5cc600 1 waiting to reach incremental sync.. 2022-09-12T19:07:26.035 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:07:26.031+0000 7f8c5a5cc600 1 bucket sync caught up with source: 2022-09-12T19:07:26.035 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000001.26.5, , ] 2022-09-12T19:07:26.036 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000001.26.5, , ] 2022-09-12T19:07:26.036 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:07:26.031+0000 7f8c5a5cc600 0 bucket checkpoint complete 2022-09-12T19:07:26.052 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-20 2022-09-12T19:07:26.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:07:26.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:07:26.054 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:07:26.120 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:26.121 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:26.178 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:07:26.175+0000 7ffa6928e600 1 bucket sync caught up with source: 2022-09-12T19:07:26.178 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000001.26.5, , ] 2022-09-12T19:07:26.179 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000001.26.5, , ] 2022-09-12T19:07:26.179 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:07:26.175+0000 7ffa6928e600 0 bucket checkpoint complete 2022-09-12T19:07:26.194 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-20 2022-09-12T19:07:26.195 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:07:26.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:07:26.197 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:07:26.261 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:26.261 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:26.300 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:07:26.297+0000 7fcc65600600 1 waiting to reach incremental sync.. 2022-09-12T19:07:31.302 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:07:31.300+0000 7fcc65600600 1 bucket sync caught up with source: 2022-09-12T19:07:31.303 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000001.26.5, , ] 2022-09-12T19:07:31.303 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000001.26.5, , ] 2022-09-12T19:07:31.303 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:07:31.300+0000 7fcc65600600 0 bucket checkpoint complete 2022-09-12T19:07:31.322 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-20 2022-09-12T19:07:31.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:07:31.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:07:31.326 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:07:31.392 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:31.393 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:31.441 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:07:31.438+0000 7fa9ad170600 1 waiting to reach incremental sync.. 2022-09-12T19:07:36.442 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:07:36.439+0000 7fa9ad170600 1 waiting to reach incremental sync.. 2022-09-12T19:07:41.442 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:07:41.440+0000 7fa9ad170600 1 waiting to reach incremental sync.. 2022-09-12T19:07:46.444 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:07:46.442+0000 7fa9ad170600 1 bucket sync caught up with source: 2022-09-12T19:07:46.445 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000001.40.5, , ] 2022-09-12T19:07:46.445 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000001.40.5, , ] 2022-09-12T19:07:46.445 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:07:46.442+0000 7fa9ad170600 0 bucket checkpoint complete 2022-09-12T19:07:46.464 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-20 2022-09-12T19:07:46.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:07:46.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:07:46.466 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:07:46.530 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:46.531 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:46.575 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:07:46.573+0000 7fc2506e3600 1 bucket sync caught up with source: 2022-09-12T19:07:46.576 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000001.40.5, , ] 2022-09-12T19:07:46.576 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000001.40.5, , ] 2022-09-12T19:07:46.577 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:07:46.573+0000 7fc2506e3600 0 bucket checkpoint complete 2022-09-12T19:07:46.596 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-20 2022-09-12T19:07:46.597 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-20 2022-09-12T19:07:46.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:07:46.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:07:46.598 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:07:46.657 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:46.658 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:46.709 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:07:46.707+0000 7fd2794c7600 1 bucket sync caught up with source: 2022-09-12T19:07:46.709 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000001.48.5, , ] 2022-09-12T19:07:46.710 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000001.48.5, , ] 2022-09-12T19:07:46.710 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:07:46.707+0000 7fd2794c7600 0 bucket checkpoint complete 2022-09-12T19:07:46.730 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-20 2022-09-12T19:07:46.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:07:46.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:07:46.731 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:07:46.795 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:46.795 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:46.833 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:07:46.830+0000 7f5eea3a4600 1 bucket sync caught up with source: 2022-09-12T19:07:46.834 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000001.48.5, , ] 2022-09-12T19:07:46.834 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000001.48.5, , ] 2022-09-12T19:07:46.834 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:07:46.830+0000 7f5eea3a4600 0 bucket checkpoint complete 2022-09-12T19:07:46.850 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-20 2022-09-12T19:07:46.851 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-20 zones={test-zone2, test-zone1} 2022-09-12T19:07:46.852 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:46.852 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:46.853 DEBUG:boto:Method: HEAD 2022-09-12T19:07:46.853 DEBUG:boto:Path: /jdpiak-20/ 2022-09-12T19:07:46.854 DEBUG:boto:Data: 2022-09-12T19:07:46.855 DEBUG:boto:Headers: {} 2022-09-12T19:07:46.855 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:46.856 DEBUG:boto:Port: 8000 2022-09-12T19:07:46.856 DEBUG:boto:Params: {} 2022-09-12T19:07:46.857 DEBUG:boto:Token: None 2022-09-12T19:07:46.858 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:46 GMT /jdpiak-20/ 2022-09-12T19:07:46.858 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l/IjDNwyLNPt6XqXfsIZZRBRQbk= 2022-09-12T19:07:46.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l/IjDNwyLNPt6XqXfsIZZRBRQbk=', 'Content-Length': '0'} 2022-09-12T19:07:46.861 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031ba9ea75f224939-00631f8382-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:07:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:46.862 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:46.863 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:46.863 DEBUG:boto:Method: HEAD 2022-09-12T19:07:46.864 DEBUG:boto:Path: /jdpiak-20/ 2022-09-12T19:07:46.864 DEBUG:boto:Data: 2022-09-12T19:07:46.865 DEBUG:boto:Headers: {} 2022-09-12T19:07:46.865 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:46.866 DEBUG:boto:Port: 8000 2022-09-12T19:07:46.866 DEBUG:boto:Params: {} 2022-09-12T19:07:46.867 DEBUG:boto:Token: None 2022-09-12T19:07:46.867 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:46 GMT /jdpiak-20/ 2022-09-12T19:07:46.868 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l/IjDNwyLNPt6XqXfsIZZRBRQbk= 2022-09-12T19:07:46.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l/IjDNwyLNPt6XqXfsIZZRBRQbk=', 'Content-Length': '0'} 2022-09-12T19:07:46.871 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013fb6eb9f05785e9-00631f8382-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:07:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:46.872 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:07:46.872 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:46.873 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:46.874 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:46.874 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:46.875 DEBUG:boto:Method: GET 2022-09-12T19:07:46.875 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:46.876 DEBUG:boto:Data: 2022-09-12T19:07:46.876 DEBUG:boto:Headers: {} 2022-09-12T19:07:46.877 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:46.877 DEBUG:boto:Port: 8000 2022-09-12T19:07:46.878 DEBUG:boto:Params: {} 2022-09-12T19:07:46.878 DEBUG:boto:Token: None 2022-09-12T19:07:46.879 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:46 GMT /jdpiak-20/?versions 2022-09-12T19:07:46.880 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qa2iZbMekWg46j3gFwraoncbwbw= 2022-09-12T19:07:46.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qa2iZbMekWg46j3gFwraoncbwbw=', 'Content-Length': '0'} 2022-09-12T19:07:46.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2bd4ae1480c7256-00631f8382-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:46.883 DEBUG:boto:b'jdpiak-20999falseobjnulltrue2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:46.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:07:46.885 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:07:46.885 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:46.886 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:46.886 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:46.887 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:46.887 DEBUG:boto:Method: GET 2022-09-12T19:07:46.888 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:46.888 DEBUG:boto:Data: 2022-09-12T19:07:46.889 DEBUG:boto:Headers: {} 2022-09-12T19:07:46.889 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:46.890 DEBUG:boto:Port: 8000 2022-09-12T19:07:46.890 DEBUG:boto:Params: {} 2022-09-12T19:07:46.891 DEBUG:boto:Token: None 2022-09-12T19:07:46.892 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:46 GMT /jdpiak-20/?versions 2022-09-12T19:07:46.892 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qa2iZbMekWg46j3gFwraoncbwbw= 2022-09-12T19:07:46.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qa2iZbMekWg46j3gFwraoncbwbw=', 'Content-Length': '0'} 2022-09-12T19:07:46.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006675e1e470aceae4-00631f8382-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:46.938 DEBUG:boto:b'jdpiak-20999falseobjnulltrue2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:46.939 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:07:46.940 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:46.940 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:46.941 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:46.941 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:46.942 DEBUG:boto:Method: GET 2022-09-12T19:07:46.942 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:46.943 DEBUG:boto:Data: 2022-09-12T19:07:46.943 DEBUG:boto:Headers: {} 2022-09-12T19:07:46.944 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:46.945 DEBUG:boto:Port: 8000 2022-09-12T19:07:46.945 DEBUG:boto:Params: {} 2022-09-12T19:07:46.946 DEBUG:boto:Token: None 2022-09-12T19:07:46.946 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:46 GMT /jdpiak-20/?versions 2022-09-12T19:07:46.947 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qa2iZbMekWg46j3gFwraoncbwbw= 2022-09-12T19:07:46.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qa2iZbMekWg46j3gFwraoncbwbw=', 'Content-Length': '0'} 2022-09-12T19:07:46.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f86a8969532085f1-00631f8382-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:46.951 DEBUG:boto:b'jdpiak-20999falseobjnulltrue2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:46.952 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:46.952 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:46.953 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:46.953 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:46.954 DEBUG:boto:Method: GET 2022-09-12T19:07:46.954 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:46.955 DEBUG:boto:Data: 2022-09-12T19:07:46.955 DEBUG:boto:Headers: {} 2022-09-12T19:07:46.956 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:46.956 DEBUG:boto:Port: 8000 2022-09-12T19:07:46.957 DEBUG:boto:Params: {} 2022-09-12T19:07:46.957 DEBUG:boto:Token: None 2022-09-12T19:07:46.958 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:46 GMT /jdpiak-20/?versions 2022-09-12T19:07:46.958 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qa2iZbMekWg46j3gFwraoncbwbw= 2022-09-12T19:07:46.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qa2iZbMekWg46j3gFwraoncbwbw=', 'Content-Length': '0'} 2022-09-12T19:07:46.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0d967389ead268b-00631f8382-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:46.962 DEBUG:boto:b'jdpiak-20999falseobjnulltrue2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:46.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:46.964 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:46.964 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:46.965 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:46.965 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:46.966 DEBUG:boto:Method: GET 2022-09-12T19:07:46.966 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:07:46.967 DEBUG:boto:Data: 2022-09-12T19:07:46.968 DEBUG:boto:Headers: {} 2022-09-12T19:07:46.968 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:46.969 DEBUG:boto:Port: 8000 2022-09-12T19:07:46.969 DEBUG:boto:Params: {} 2022-09-12T19:07:46.970 DEBUG:boto:Token: None 2022-09-12T19:07:46.970 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:46 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:07:46.971 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ud2drkKjnRazTB176fzU6pe+kZ8= 2022-09-12T19:07:46.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ud2drkKjnRazTB176fzU6pe+kZ8=', 'Content-Length': '0'} 2022-09-12T19:07:46.974 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000071f0d544acc1706a-00631f8382-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:46.974 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:46.975 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:46.976 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:46.976 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:46.977 DEBUG:boto:Method: GET 2022-09-12T19:07:46.977 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:07:46.978 DEBUG:boto:Data: 2022-09-12T19:07:46.978 DEBUG:boto:Headers: {} 2022-09-12T19:07:46.979 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:46.979 DEBUG:boto:Port: 8000 2022-09-12T19:07:46.980 DEBUG:boto:Params: {} 2022-09-12T19:07:46.981 DEBUG:boto:Token: None 2022-09-12T19:07:46.981 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:46 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:07:46.982 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ud2drkKjnRazTB176fzU6pe+kZ8= 2022-09-12T19:07:46.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ud2drkKjnRazTB176fzU6pe+kZ8=', 'Content-Length': '0'} 2022-09-12T19:07:46.985 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009506b1b100df5491-00631f8382-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:46.986 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:46.986 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:46.987 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:46.987 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:46.988 DEBUG:boto:Method: HEAD 2022-09-12T19:07:46.988 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:07:46.989 DEBUG:boto:Data: 2022-09-12T19:07:46.989 DEBUG:boto:Headers: {} 2022-09-12T19:07:46.990 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:46.990 DEBUG:boto:Port: 8000 2022-09-12T19:07:46.991 DEBUG:boto:Params: {} 2022-09-12T19:07:46.991 DEBUG:boto:Token: None 2022-09-12T19:07:46.992 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:46 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:07:46.992 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SFSl6m3qfyRKoafQD7dU2Mxm1sM= 2022-09-12T19:07:46.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SFSl6m3qfyRKoafQD7dU2Mxm1sM=', 'Content-Length': '0'} 2022-09-12T19:07:46.995 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000087efeba410f16062-00631f8382-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:46.996 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:46.996 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:46.997 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:46.997 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:46.998 DEBUG:boto:Method: HEAD 2022-09-12T19:07:46.998 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:07:46.999 DEBUG:boto:Data: 2022-09-12T19:07:47.000 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.000 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:47.001 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.001 DEBUG:boto:Params: {} 2022-09-12T19:07:47.002 DEBUG:boto:Token: None 2022-09-12T19:07:47.002 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.003 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EymisvfQch4bLJOL+AcXQdPPnhc= 2022-09-12T19:07:47.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EymisvfQch4bLJOL+AcXQdPPnhc=', 'Content-Length': '0'} 2022-09-12T19:07:47.005 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009ee341723830f11e-00631f8383-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:47.007 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.007 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.008 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.008 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.009 DEBUG:boto:Method: GET 2022-09-12T19:07:47.009 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.010 DEBUG:boto:Data: 2022-09-12T19:07:47.010 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.011 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:47.011 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.012 DEBUG:boto:Params: {} 2022-09-12T19:07:47.013 DEBUG:boto:Token: None 2022-09-12T19:07:47.013 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.014 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TlsgCO2lR0g1jKrPX1JIbNTNPKs= 2022-09-12T19:07:47.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TlsgCO2lR0g1jKrPX1JIbNTNPKs=', 'Content-Length': '0'} 2022-09-12T19:07:47.017 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001c376a37035f377d-00631f8383-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.017 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.018 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.018 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.019 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.019 DEBUG:boto:Method: GET 2022-09-12T19:07:47.020 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.020 DEBUG:boto:Data: 2022-09-12T19:07:47.021 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.021 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:47.022 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.023 DEBUG:boto:Params: {} 2022-09-12T19:07:47.023 DEBUG:boto:Token: None 2022-09-12T19:07:47.024 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.025 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TlsgCO2lR0g1jKrPX1JIbNTNPKs= 2022-09-12T19:07:47.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TlsgCO2lR0g1jKrPX1JIbNTNPKs=', 'Content-Length': '0'} 2022-09-12T19:07:47.028 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f93f338b63a91a9f-00631f8383-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.029 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:47.029 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:47.030 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:07:47.030 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:07:47.031 DEBUG:boto:Method: GET 2022-09-12T19:07:47.032 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:07:47.032 DEBUG:boto:Data: 2022-09-12T19:07:47.033 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.033 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:47.034 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.034 DEBUG:boto:Params: {} 2022-09-12T19:07:47.035 DEBUG:boto:Token: None 2022-09-12T19:07:47.035 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/ 2022-09-12T19:07:47.036 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uR9yfrjpqtBsOVMTtsw6EmyNVCM= 2022-09-12T19:07:47.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uR9yfrjpqtBsOVMTtsw6EmyNVCM=', 'Content-Length': '0'} 2022-09-12T19:07:47.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fee15107f99c5be3-00631f8383-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.082 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:47.083 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:47.084 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:47.085 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:07:47.085 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:07:47.086 DEBUG:boto:Method: GET 2022-09-12T19:07:47.086 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:07:47.087 DEBUG:boto:Data: 2022-09-12T19:07:47.087 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.088 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:47.089 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.089 DEBUG:boto:Params: {} 2022-09-12T19:07:47.090 DEBUG:boto:Token: None 2022-09-12T19:07:47.090 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/ 2022-09-12T19:07:47.091 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uR9yfrjpqtBsOVMTtsw6EmyNVCM= 2022-09-12T19:07:47.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uR9yfrjpqtBsOVMTtsw6EmyNVCM=', 'Content-Length': '0'} 2022-09-12T19:07:47.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040ce612b5762607f-00631f8383-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.095 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:47.096 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.097 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.097 DEBUG:boto:Method: HEAD 2022-09-12T19:07:47.098 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:07:47.098 DEBUG:boto:Data: 2022-09-12T19:07:47.099 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.099 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:47.100 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.100 DEBUG:boto:Params: {} 2022-09-12T19:07:47.101 DEBUG:boto:Token: None 2022-09-12T19:07:47.101 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj 2022-09-12T19:07:47.102 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AYf/iXvAmsm33DYjkWvxExinQfk= 2022-09-12T19:07:47.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AYf/iXvAmsm33DYjkWvxExinQfk=', 'Content-Length': '0'} 2022-09-12T19:07:47.104 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000007ea8f3f511aeefa-00631f8383-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.105 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.106 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.106 DEBUG:boto:Method: HEAD 2022-09-12T19:07:47.107 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:07:47.107 DEBUG:boto:Data: 2022-09-12T19:07:47.108 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.108 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:47.109 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.109 DEBUG:boto:Params: {} 2022-09-12T19:07:47.110 DEBUG:boto:Token: None 2022-09-12T19:07:47.111 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj 2022-09-12T19:07:47.111 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AYf/iXvAmsm33DYjkWvxExinQfk= 2022-09-12T19:07:47.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AYf/iXvAmsm33DYjkWvxExinQfk=', 'Content-Length': '0'} 2022-09-12T19:07:47.114 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007faf1f4c05a4fcd4-00631f8383-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:47.115 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.116 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.116 DEBUG:boto:Method: GET 2022-09-12T19:07:47.117 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:07:47.117 DEBUG:boto:Data: 2022-09-12T19:07:47.118 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.119 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:47.119 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.120 DEBUG:boto:Params: {} 2022-09-12T19:07:47.120 DEBUG:boto:Token: None 2022-09-12T19:07:47.121 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj 2022-09-12T19:07:47.122 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lmuPPn1s8xAxs9d668XPIML3W7Y= 2022-09-12T19:07:47.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lmuPPn1s8xAxs9d668XPIML3W7Y=', 'Content-Length': '0'} 2022-09-12T19:07:47.124 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000141744ecb6ff89c4-00631f8383-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.125 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.126 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.126 DEBUG:boto:Method: GET 2022-09-12T19:07:47.127 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:07:47.127 DEBUG:boto:Data: 2022-09-12T19:07:47.128 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.129 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:47.129 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.130 DEBUG:boto:Params: {} 2022-09-12T19:07:47.130 DEBUG:boto:Token: None 2022-09-12T19:07:47.131 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj 2022-09-12T19:07:47.132 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lmuPPn1s8xAxs9d668XPIML3W7Y= 2022-09-12T19:07:47.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lmuPPn1s8xAxs9d668XPIML3W7Y=', 'Content-Length': '0'} 2022-09-12T19:07:47.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000778d5de865ece194-00631f8383-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.135 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:47.136 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:47.136 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:07:47.137 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:07:47.138 DEBUG:boto:Method: GET 2022-09-12T19:07:47.138 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:07:47.139 DEBUG:boto:Data: 2022-09-12T19:07:47.139 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.140 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:47.140 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.141 DEBUG:boto:Params: {} 2022-09-12T19:07:47.141 DEBUG:boto:Token: None 2022-09-12T19:07:47.142 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/ 2022-09-12T19:07:47.143 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uR9yfrjpqtBsOVMTtsw6EmyNVCM= 2022-09-12T19:07:47.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uR9yfrjpqtBsOVMTtsw6EmyNVCM=', 'Content-Length': '0'} 2022-09-12T19:07:47.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffc35de7678a73e6-00631f8383-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.146 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:47.147 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:47.148 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:47.148 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:07:47.149 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:07:47.149 DEBUG:boto:Method: GET 2022-09-12T19:07:47.150 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:07:47.150 DEBUG:boto:Data: 2022-09-12T19:07:47.151 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.152 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:47.152 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.153 DEBUG:boto:Params: {} 2022-09-12T19:07:47.154 DEBUG:boto:Token: None 2022-09-12T19:07:47.154 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/ 2022-09-12T19:07:47.155 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uR9yfrjpqtBsOVMTtsw6EmyNVCM= 2022-09-12T19:07:47.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uR9yfrjpqtBsOVMTtsw6EmyNVCM=', 'Content-Length': '0'} 2022-09-12T19:07:47.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010a5378fdb311259-00631f8383-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.198 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:47.199 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-20 zones={test-zone2, test-zone1} 2022-09-12T19:07:47.200 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-20 zones={test-zone3, test-zone1} 2022-09-12T19:07:47.200 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:47.201 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:47.202 DEBUG:boto:Method: HEAD 2022-09-12T19:07:47.202 DEBUG:boto:Path: /jdpiak-20/ 2022-09-12T19:07:47.203 DEBUG:boto:Data: 2022-09-12T19:07:47.204 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.204 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:47.205 DEBUG:boto:Port: 8001 2022-09-12T19:07:47.206 DEBUG:boto:Params: {} 2022-09-12T19:07:47.207 DEBUG:boto:Token: None 2022-09-12T19:07:47.207 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/ 2022-09-12T19:07:47.208 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:16p/6aMB8nsKu0LHl5JJG/8niUY= 2022-09-12T19:07:47.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:16p/6aMB8nsKu0LHl5JJG/8niUY=', 'Content-Length': '0'} 2022-09-12T19:07:47.212 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5ca4bc89144491d-00631f8383-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.212 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:47.213 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:47.213 DEBUG:boto:Method: HEAD 2022-09-12T19:07:47.214 DEBUG:boto:Path: /jdpiak-20/ 2022-09-12T19:07:47.215 DEBUG:boto:Data: 2022-09-12T19:07:47.215 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.216 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:47.216 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.217 DEBUG:boto:Params: {} 2022-09-12T19:07:47.218 DEBUG:boto:Token: None 2022-09-12T19:07:47.219 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/ 2022-09-12T19:07:47.219 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:16p/6aMB8nsKu0LHl5JJG/8niUY= 2022-09-12T19:07:47.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:16p/6aMB8nsKu0LHl5JJG/8niUY=', 'Content-Length': '0'} 2022-09-12T19:07:47.222 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000def51f27e1cfeebf-00631f8383-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.223 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:07:47.224 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:47.226 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:47.227 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:47.228 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:47.229 DEBUG:boto:Method: GET 2022-09-12T19:07:47.230 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:47.231 DEBUG:boto:Data: 2022-09-12T19:07:47.232 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.233 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:47.234 DEBUG:boto:Port: 8001 2022-09-12T19:07:47.235 DEBUG:boto:Params: {} 2022-09-12T19:07:47.236 DEBUG:boto:Token: None 2022-09-12T19:07:47.236 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/?versions 2022-09-12T19:07:47.237 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YUmnSUtqWjpwX9lwS59KZ6rxjdA= 2022-09-12T19:07:47.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YUmnSUtqWjpwX9lwS59KZ6rxjdA=', 'Content-Length': '0'} 2022-09-12T19:07:47.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c31f3041dcb8d010-00631f8383-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.241 DEBUG:boto:b'jdpiak-20999falseobjnulltrue2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:47.242 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:07:47.242 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:07:47.243 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:47.244 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:47.244 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:47.245 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:47.245 DEBUG:boto:Method: GET 2022-09-12T19:07:47.246 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:47.246 DEBUG:boto:Data: 2022-09-12T19:07:47.247 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.247 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:47.248 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.248 DEBUG:boto:Params: {} 2022-09-12T19:07:47.249 DEBUG:boto:Token: None 2022-09-12T19:07:47.250 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/?versions 2022-09-12T19:07:47.251 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YUmnSUtqWjpwX9lwS59KZ6rxjdA= 2022-09-12T19:07:47.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YUmnSUtqWjpwX9lwS59KZ6rxjdA=', 'Content-Length': '0'} 2022-09-12T19:07:47.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d49193cff0c3461-00631f8383-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.254 DEBUG:boto:b'jdpiak-20999falseobjnulltrue2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:47.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:07:47.256 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:47.256 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:47.257 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:47.258 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:47.258 DEBUG:boto:Method: GET 2022-09-12T19:07:47.259 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:47.259 DEBUG:boto:Data: 2022-09-12T19:07:47.260 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.261 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:47.261 DEBUG:boto:Port: 8001 2022-09-12T19:07:47.262 DEBUG:boto:Params: {} 2022-09-12T19:07:47.262 DEBUG:boto:Token: None 2022-09-12T19:07:47.263 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/?versions 2022-09-12T19:07:47.264 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YUmnSUtqWjpwX9lwS59KZ6rxjdA= 2022-09-12T19:07:47.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YUmnSUtqWjpwX9lwS59KZ6rxjdA=', 'Content-Length': '0'} 2022-09-12T19:07:47.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007823291e383641ca-00631f8383-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.267 DEBUG:boto:b'jdpiak-20999falseobjnulltrue2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:47.268 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:47.269 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:47.269 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:47.270 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:47.270 DEBUG:boto:Method: GET 2022-09-12T19:07:47.271 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:47.271 DEBUG:boto:Data: 2022-09-12T19:07:47.272 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.272 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:47.273 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.273 DEBUG:boto:Params: {} 2022-09-12T19:07:47.274 DEBUG:boto:Token: None 2022-09-12T19:07:47.275 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/?versions 2022-09-12T19:07:47.276 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YUmnSUtqWjpwX9lwS59KZ6rxjdA= 2022-09-12T19:07:47.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YUmnSUtqWjpwX9lwS59KZ6rxjdA=', 'Content-Length': '0'} 2022-09-12T19:07:47.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9fc1ce445ad1a1b-00631f8383-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.322 DEBUG:boto:b'jdpiak-20999falseobjnulltrue2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:47.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:47.324 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.325 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.326 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.327 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.328 DEBUG:boto:Method: GET 2022-09-12T19:07:47.329 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.329 DEBUG:boto:Data: 2022-09-12T19:07:47.330 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.330 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:47.331 DEBUG:boto:Port: 8001 2022-09-12T19:07:47.332 DEBUG:boto:Params: {} 2022-09-12T19:07:47.332 DEBUG:boto:Token: None 2022-09-12T19:07:47.333 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.334 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TlsgCO2lR0g1jKrPX1JIbNTNPKs= 2022-09-12T19:07:47.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TlsgCO2lR0g1jKrPX1JIbNTNPKs=', 'Content-Length': '0'} 2022-09-12T19:07:47.337 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d9cdf044577df133-00631f8383-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.338 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.339 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.339 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.340 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.341 DEBUG:boto:Method: GET 2022-09-12T19:07:47.341 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.342 DEBUG:boto:Data: 2022-09-12T19:07:47.343 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.344 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:47.344 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.345 DEBUG:boto:Params: {} 2022-09-12T19:07:47.346 DEBUG:boto:Token: None 2022-09-12T19:07:47.346 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.347 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TlsgCO2lR0g1jKrPX1JIbNTNPKs= 2022-09-12T19:07:47.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TlsgCO2lR0g1jKrPX1JIbNTNPKs=', 'Content-Length': '0'} 2022-09-12T19:07:47.351 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009fea1dd406f2cc80-00631f8383-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.351 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.352 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.353 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.353 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.354 DEBUG:boto:Method: HEAD 2022-09-12T19:07:47.354 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.355 DEBUG:boto:Data: 2022-09-12T19:07:47.355 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.356 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:47.356 DEBUG:boto:Port: 8001 2022-09-12T19:07:47.357 DEBUG:boto:Params: {} 2022-09-12T19:07:47.357 DEBUG:boto:Token: None 2022-09-12T19:07:47.358 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.359 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EymisvfQch4bLJOL+AcXQdPPnhc= 2022-09-12T19:07:47.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EymisvfQch4bLJOL+AcXQdPPnhc=', 'Content-Length': '0'} 2022-09-12T19:07:47.361 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aafe10716fdd91e4-00631f8383-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.363 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.363 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.364 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.364 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.365 DEBUG:boto:Method: HEAD 2022-09-12T19:07:47.365 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.366 DEBUG:boto:Data: 2022-09-12T19:07:47.367 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.367 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:47.368 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.369 DEBUG:boto:Params: {} 2022-09-12T19:07:47.371 DEBUG:boto:Token: None 2022-09-12T19:07:47.372 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.373 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EymisvfQch4bLJOL+AcXQdPPnhc= 2022-09-12T19:07:47.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EymisvfQch4bLJOL+AcXQdPPnhc=', 'Content-Length': '0'} 2022-09-12T19:07:47.376 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000acf273dc8f6211d3-00631f8383-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:47.379 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.380 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.381 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.382 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.383 DEBUG:boto:Method: GET 2022-09-12T19:07:47.384 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.385 DEBUG:boto:Data: 2022-09-12T19:07:47.386 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.387 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:47.388 DEBUG:boto:Port: 8001 2022-09-12T19:07:47.389 DEBUG:boto:Params: {} 2022-09-12T19:07:47.391 DEBUG:boto:Token: None 2022-09-12T19:07:47.392 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.393 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TlsgCO2lR0g1jKrPX1JIbNTNPKs= 2022-09-12T19:07:47.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TlsgCO2lR0g1jKrPX1JIbNTNPKs=', 'Content-Length': '0'} 2022-09-12T19:07:47.397 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009438cbf568cbf90b-00631f8383-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.399 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.399 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.400 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.401 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.402 DEBUG:boto:Method: GET 2022-09-12T19:07:47.404 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.405 DEBUG:boto:Data: 2022-09-12T19:07:47.406 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.407 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:47.407 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.408 DEBUG:boto:Params: {} 2022-09-12T19:07:47.410 DEBUG:boto:Token: None 2022-09-12T19:07:47.411 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.412 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TlsgCO2lR0g1jKrPX1JIbNTNPKs= 2022-09-12T19:07:47.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TlsgCO2lR0g1jKrPX1JIbNTNPKs=', 'Content-Length': '0'} 2022-09-12T19:07:47.416 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b78d458f59715372-00631f8383-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.417 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:47.418 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:47.419 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:07:47.420 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:07:47.421 DEBUG:boto:Method: GET 2022-09-12T19:07:47.422 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:07:47.423 DEBUG:boto:Data: 2022-09-12T19:07:47.424 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.425 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:47.426 DEBUG:boto:Port: 8001 2022-09-12T19:07:47.427 DEBUG:boto:Params: {} 2022-09-12T19:07:47.429 DEBUG:boto:Token: None 2022-09-12T19:07:47.430 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/ 2022-09-12T19:07:47.431 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uR9yfrjpqtBsOVMTtsw6EmyNVCM= 2022-09-12T19:07:47.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uR9yfrjpqtBsOVMTtsw6EmyNVCM=', 'Content-Length': '0'} 2022-09-12T19:07:47.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f5d1e5b57744cd4-00631f8383-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.437 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:47.439 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:47.440 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:47.441 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:07:47.442 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:07:47.443 DEBUG:boto:Method: GET 2022-09-12T19:07:47.445 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:07:47.446 DEBUG:boto:Data: 2022-09-12T19:07:47.447 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.448 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:47.449 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.450 DEBUG:boto:Params: {} 2022-09-12T19:07:47.451 DEBUG:boto:Token: None 2022-09-12T19:07:47.452 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/ 2022-09-12T19:07:47.454 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uR9yfrjpqtBsOVMTtsw6EmyNVCM= 2022-09-12T19:07:47.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uR9yfrjpqtBsOVMTtsw6EmyNVCM=', 'Content-Length': '0'} 2022-09-12T19:07:47.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097dbd7ef650e610a-00631f8383-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.459 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:47.461 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.462 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.463 DEBUG:boto:Method: HEAD 2022-09-12T19:07:47.464 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:07:47.465 DEBUG:boto:Data: 2022-09-12T19:07:47.466 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.467 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:47.468 DEBUG:boto:Port: 8001 2022-09-12T19:07:47.469 DEBUG:boto:Params: {} 2022-09-12T19:07:47.470 DEBUG:boto:Token: None 2022-09-12T19:07:47.471 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj 2022-09-12T19:07:47.472 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AYf/iXvAmsm33DYjkWvxExinQfk= 2022-09-12T19:07:47.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AYf/iXvAmsm33DYjkWvxExinQfk=', 'Content-Length': '0'} 2022-09-12T19:07:47.476 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000020635154af546471-00631f8383-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.477 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.478 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.479 DEBUG:boto:Method: HEAD 2022-09-12T19:07:47.480 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:07:47.481 DEBUG:boto:Data: 2022-09-12T19:07:47.482 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.483 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:47.484 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.485 DEBUG:boto:Params: {} 2022-09-12T19:07:47.486 DEBUG:boto:Token: None 2022-09-12T19:07:47.487 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj 2022-09-12T19:07:47.488 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AYf/iXvAmsm33DYjkWvxExinQfk= 2022-09-12T19:07:47.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AYf/iXvAmsm33DYjkWvxExinQfk=', 'Content-Length': '0'} 2022-09-12T19:07:47.491 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d06c095c08082ede-00631f8383-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:47.494 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.495 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.496 DEBUG:boto:Method: GET 2022-09-12T19:07:47.497 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:07:47.498 DEBUG:boto:Data: 2022-09-12T19:07:47.499 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.500 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:47.501 DEBUG:boto:Port: 8001 2022-09-12T19:07:47.502 DEBUG:boto:Params: {} 2022-09-12T19:07:47.503 DEBUG:boto:Token: None 2022-09-12T19:07:47.505 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj 2022-09-12T19:07:47.506 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lmuPPn1s8xAxs9d668XPIML3W7Y= 2022-09-12T19:07:47.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lmuPPn1s8xAxs9d668XPIML3W7Y=', 'Content-Length': '0'} 2022-09-12T19:07:47.511 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008f53a6e9bc9a1a8d-00631f8383-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.512 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.513 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.514 DEBUG:boto:Method: GET 2022-09-12T19:07:47.516 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:07:47.517 DEBUG:boto:Data: 2022-09-12T19:07:47.517 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.518 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:47.519 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.520 DEBUG:boto:Params: {} 2022-09-12T19:07:47.520 DEBUG:boto:Token: None 2022-09-12T19:07:47.521 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj 2022-09-12T19:07:47.521 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lmuPPn1s8xAxs9d668XPIML3W7Y= 2022-09-12T19:07:47.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lmuPPn1s8xAxs9d668XPIML3W7Y=', 'Content-Length': '0'} 2022-09-12T19:07:47.524 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000085c831c49d584511-00631f8383-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.525 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:47.525 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:47.526 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:07:47.526 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:07:47.527 DEBUG:boto:Method: GET 2022-09-12T19:07:47.527 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:07:47.528 DEBUG:boto:Data: 2022-09-12T19:07:47.529 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.529 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:47.530 DEBUG:boto:Port: 8001 2022-09-12T19:07:47.530 DEBUG:boto:Params: {} 2022-09-12T19:07:47.531 DEBUG:boto:Token: None 2022-09-12T19:07:47.531 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/ 2022-09-12T19:07:47.532 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uR9yfrjpqtBsOVMTtsw6EmyNVCM= 2022-09-12T19:07:47.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uR9yfrjpqtBsOVMTtsw6EmyNVCM=', 'Content-Length': '0'} 2022-09-12T19:07:47.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068e11826806e6273-00631f8383-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.582 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:47.584 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:47.585 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:47.586 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:07:47.586 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:07:47.587 DEBUG:boto:Method: GET 2022-09-12T19:07:47.588 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:07:47.589 DEBUG:boto:Data: 2022-09-12T19:07:47.590 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.591 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:47.592 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.593 DEBUG:boto:Params: {} 2022-09-12T19:07:47.594 DEBUG:boto:Token: None 2022-09-12T19:07:47.595 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/ 2022-09-12T19:07:47.596 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uR9yfrjpqtBsOVMTtsw6EmyNVCM= 2022-09-12T19:07:47.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uR9yfrjpqtBsOVMTtsw6EmyNVCM=', 'Content-Length': '0'} 2022-09-12T19:07:47.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083eb87652860250d-00631f8383-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.603 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:47.604 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-20 zones={test-zone3, test-zone1} 2022-09-12T19:07:47.605 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-20 zones={test-zone3, test-zone2} 2022-09-12T19:07:47.606 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:47.606 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:47.607 DEBUG:boto:Method: HEAD 2022-09-12T19:07:47.608 DEBUG:boto:Path: /jdpiak-20/ 2022-09-12T19:07:47.609 DEBUG:boto:Data: 2022-09-12T19:07:47.610 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.611 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:47.612 DEBUG:boto:Port: 8001 2022-09-12T19:07:47.612 DEBUG:boto:Params: {} 2022-09-12T19:07:47.613 DEBUG:boto:Token: None 2022-09-12T19:07:47.614 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/ 2022-09-12T19:07:47.616 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:16p/6aMB8nsKu0LHl5JJG/8niUY= 2022-09-12T19:07:47.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:16p/6aMB8nsKu0LHl5JJG/8niUY=', 'Content-Length': '0'} 2022-09-12T19:07:47.620 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010eea2a0823376e9-00631f8383-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.667 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:47.668 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:47.669 DEBUG:boto:Method: HEAD 2022-09-12T19:07:47.670 DEBUG:boto:Path: /jdpiak-20/ 2022-09-12T19:07:47.670 DEBUG:boto:Data: 2022-09-12T19:07:47.671 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.671 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:47.672 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.672 DEBUG:boto:Params: {} 2022-09-12T19:07:47.673 DEBUG:boto:Token: None 2022-09-12T19:07:47.674 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/ 2022-09-12T19:07:47.674 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:16p/6aMB8nsKu0LHl5JJG/8niUY= 2022-09-12T19:07:47.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:16p/6aMB8nsKu0LHl5JJG/8niUY=', 'Content-Length': '0'} 2022-09-12T19:07:47.679 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b192a9ab613ff40-00631f8383-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.681 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:07:47.682 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:47.683 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:47.683 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:47.684 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:47.685 DEBUG:boto:Method: GET 2022-09-12T19:07:47.686 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:47.687 DEBUG:boto:Data: 2022-09-12T19:07:47.688 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.689 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:47.690 DEBUG:boto:Port: 8001 2022-09-12T19:07:47.690 DEBUG:boto:Params: {} 2022-09-12T19:07:47.692 DEBUG:boto:Token: None 2022-09-12T19:07:47.692 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/?versions 2022-09-12T19:07:47.693 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YUmnSUtqWjpwX9lwS59KZ6rxjdA= 2022-09-12T19:07:47.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YUmnSUtqWjpwX9lwS59KZ6rxjdA=', 'Content-Length': '0'} 2022-09-12T19:07:47.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000682e726061f62480-00631f8383-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.699 DEBUG:boto:b'jdpiak-20999falseobjnulltrue2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:47.700 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:07:47.701 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:07:47.702 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:47.702 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:47.703 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:47.704 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:47.704 DEBUG:boto:Method: GET 2022-09-12T19:07:47.705 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:47.706 DEBUG:boto:Data: 2022-09-12T19:07:47.706 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.707 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:47.707 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.708 DEBUG:boto:Params: {} 2022-09-12T19:07:47.709 DEBUG:boto:Token: None 2022-09-12T19:07:47.710 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/?versions 2022-09-12T19:07:47.710 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YUmnSUtqWjpwX9lwS59KZ6rxjdA= 2022-09-12T19:07:47.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YUmnSUtqWjpwX9lwS59KZ6rxjdA=', 'Content-Length': '0'} 2022-09-12T19:07:47.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e87b81606708409-00631f8383-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.716 DEBUG:boto:b'jdpiak-20999falseobjnulltrue2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:47.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:07:47.718 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:47.718 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:47.719 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:47.719 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:47.720 DEBUG:boto:Method: GET 2022-09-12T19:07:47.721 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:47.721 DEBUG:boto:Data: 2022-09-12T19:07:47.722 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.723 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:47.723 DEBUG:boto:Port: 8001 2022-09-12T19:07:47.724 DEBUG:boto:Params: {} 2022-09-12T19:07:47.724 DEBUG:boto:Token: None 2022-09-12T19:07:47.725 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/?versions 2022-09-12T19:07:47.726 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YUmnSUtqWjpwX9lwS59KZ6rxjdA= 2022-09-12T19:07:47.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YUmnSUtqWjpwX9lwS59KZ6rxjdA=', 'Content-Length': '0'} 2022-09-12T19:07:47.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cbbc86665d4c040-00631f8383-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.730 DEBUG:boto:b'jdpiak-20999falseobjnulltrue2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:47.731 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:47.732 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:47.732 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:47.733 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:47.734 DEBUG:boto:Method: GET 2022-09-12T19:07:47.734 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:07:47.735 DEBUG:boto:Data: 2022-09-12T19:07:47.736 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.736 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:47.737 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.737 DEBUG:boto:Params: {} 2022-09-12T19:07:47.738 DEBUG:boto:Token: None 2022-09-12T19:07:47.739 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/?versions 2022-09-12T19:07:47.740 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YUmnSUtqWjpwX9lwS59KZ6rxjdA= 2022-09-12T19:07:47.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YUmnSUtqWjpwX9lwS59KZ6rxjdA=', 'Content-Length': '0'} 2022-09-12T19:07:47.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d769bc92e01328f0-00631f8383-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.748 DEBUG:boto:b'jdpiak-20999falseobjnulltrue2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:47.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:47.752 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.753 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.754 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.754 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.756 DEBUG:boto:Method: GET 2022-09-12T19:07:47.757 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.758 DEBUG:boto:Data: 2022-09-12T19:07:47.759 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.759 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:47.760 DEBUG:boto:Port: 8001 2022-09-12T19:07:47.760 DEBUG:boto:Params: {} 2022-09-12T19:07:47.761 DEBUG:boto:Token: None 2022-09-12T19:07:47.762 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.762 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TlsgCO2lR0g1jKrPX1JIbNTNPKs= 2022-09-12T19:07:47.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TlsgCO2lR0g1jKrPX1JIbNTNPKs=', 'Content-Length': '0'} 2022-09-12T19:07:47.766 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000103a7f2b1ffc15b9-00631f8383-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.767 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.768 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.769 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.770 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.771 DEBUG:boto:Method: GET 2022-09-12T19:07:47.772 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.773 DEBUG:boto:Data: 2022-09-12T19:07:47.774 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.775 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:47.776 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.777 DEBUG:boto:Params: {} 2022-09-12T19:07:47.779 DEBUG:boto:Token: None 2022-09-12T19:07:47.780 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.781 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TlsgCO2lR0g1jKrPX1JIbNTNPKs= 2022-09-12T19:07:47.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TlsgCO2lR0g1jKrPX1JIbNTNPKs=', 'Content-Length': '0'} 2022-09-12T19:07:47.786 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007e5067a5a849b1cb-00631f8383-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.788 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.789 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.790 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.791 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.792 DEBUG:boto:Method: HEAD 2022-09-12T19:07:47.793 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.794 DEBUG:boto:Data: 2022-09-12T19:07:47.795 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.796 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:47.797 DEBUG:boto:Port: 8001 2022-09-12T19:07:47.798 DEBUG:boto:Params: {} 2022-09-12T19:07:47.800 DEBUG:boto:Token: None 2022-09-12T19:07:47.801 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.802 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EymisvfQch4bLJOL+AcXQdPPnhc= 2022-09-12T19:07:47.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EymisvfQch4bLJOL+AcXQdPPnhc=', 'Content-Length': '0'} 2022-09-12T19:07:47.806 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000750f789d41333658-00631f8383-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.807 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.808 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.810 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.811 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.812 DEBUG:boto:Method: HEAD 2022-09-12T19:07:47.813 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.814 DEBUG:boto:Data: 2022-09-12T19:07:47.814 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.815 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:47.816 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.816 DEBUG:boto:Params: {} 2022-09-12T19:07:47.817 DEBUG:boto:Token: None 2022-09-12T19:07:47.818 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.819 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EymisvfQch4bLJOL+AcXQdPPnhc= 2022-09-12T19:07:47.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EymisvfQch4bLJOL+AcXQdPPnhc=', 'Content-Length': '0'} 2022-09-12T19:07:47.822 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003c87c15bf730b9fa-00631f8383-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:47.824 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.825 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.826 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.827 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.827 DEBUG:boto:Method: GET 2022-09-12T19:07:47.828 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.829 DEBUG:boto:Data: 2022-09-12T19:07:47.830 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.831 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:47.831 DEBUG:boto:Port: 8001 2022-09-12T19:07:47.832 DEBUG:boto:Params: {} 2022-09-12T19:07:47.833 DEBUG:boto:Token: None 2022-09-12T19:07:47.834 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.835 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TlsgCO2lR0g1jKrPX1JIbNTNPKs= 2022-09-12T19:07:47.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TlsgCO2lR0g1jKrPX1JIbNTNPKs=', 'Content-Length': '0'} 2022-09-12T19:07:47.839 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000855ed22092f1b33d-00631f8383-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.840 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.841 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.842 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.843 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.843 DEBUG:boto:Method: GET 2022-09-12T19:07:47.844 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.844 DEBUG:boto:Data: 2022-09-12T19:07:47.845 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.846 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:47.847 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.848 DEBUG:boto:Params: {} 2022-09-12T19:07:47.850 DEBUG:boto:Token: None 2022-09-12T19:07:47.851 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:07:47.852 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TlsgCO2lR0g1jKrPX1JIbNTNPKs= 2022-09-12T19:07:47.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TlsgCO2lR0g1jKrPX1JIbNTNPKs=', 'Content-Length': '0'} 2022-09-12T19:07:47.857 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000001a6d74cda66a4f4-00631f8383-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.858 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:47.859 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:47.859 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:07:47.860 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:07:47.860 DEBUG:boto:Method: GET 2022-09-12T19:07:47.861 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:07:47.862 DEBUG:boto:Data: 2022-09-12T19:07:47.863 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.863 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:47.864 DEBUG:boto:Port: 8001 2022-09-12T19:07:47.864 DEBUG:boto:Params: {} 2022-09-12T19:07:47.865 DEBUG:boto:Token: None 2022-09-12T19:07:47.865 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/ 2022-09-12T19:07:47.866 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uR9yfrjpqtBsOVMTtsw6EmyNVCM= 2022-09-12T19:07:47.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uR9yfrjpqtBsOVMTtsw6EmyNVCM=', 'Content-Length': '0'} 2022-09-12T19:07:47.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c7c910f1a3e52db-00631f8383-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.925 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:47.930 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:47.932 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:47.933 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:07:47.934 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:07:47.934 DEBUG:boto:Method: GET 2022-09-12T19:07:47.935 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:07:47.937 DEBUG:boto:Data: 2022-09-12T19:07:47.938 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.938 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:47.939 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.940 DEBUG:boto:Params: {} 2022-09-12T19:07:47.940 DEBUG:boto:Token: None 2022-09-12T19:07:47.941 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/ 2022-09-12T19:07:47.942 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uR9yfrjpqtBsOVMTtsw6EmyNVCM= 2022-09-12T19:07:47.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uR9yfrjpqtBsOVMTtsw6EmyNVCM=', 'Content-Length': '0'} 2022-09-12T19:07:47.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d62c8e1535997ee4-00631f8383-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.956 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:47.957 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.957 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.958 DEBUG:boto:Method: HEAD 2022-09-12T19:07:47.958 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:07:47.959 DEBUG:boto:Data: 2022-09-12T19:07:47.959 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.960 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:47.960 DEBUG:boto:Port: 8001 2022-09-12T19:07:47.961 DEBUG:boto:Params: {} 2022-09-12T19:07:47.961 DEBUG:boto:Token: None 2022-09-12T19:07:47.962 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj 2022-09-12T19:07:47.963 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AYf/iXvAmsm33DYjkWvxExinQfk= 2022-09-12T19:07:47.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AYf/iXvAmsm33DYjkWvxExinQfk=', 'Content-Length': '0'} 2022-09-12T19:07:47.966 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004a829251da9f94ba-00631f8383-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.966 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.967 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.967 DEBUG:boto:Method: HEAD 2022-09-12T19:07:47.968 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:07:47.968 DEBUG:boto:Data: 2022-09-12T19:07:47.969 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.969 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:47.970 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.970 DEBUG:boto:Params: {} 2022-09-12T19:07:47.971 DEBUG:boto:Token: None 2022-09-12T19:07:47.972 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj 2022-09-12T19:07:47.972 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AYf/iXvAmsm33DYjkWvxExinQfk= 2022-09-12T19:07:47.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AYf/iXvAmsm33DYjkWvxExinQfk=', 'Content-Length': '0'} 2022-09-12T19:07:47.975 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000523511536b36178e-00631f8383-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:07:47.977 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.977 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.978 DEBUG:boto:Method: GET 2022-09-12T19:07:47.978 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:07:47.979 DEBUG:boto:Data: 2022-09-12T19:07:47.979 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.980 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:47.980 DEBUG:boto:Port: 8001 2022-09-12T19:07:47.981 DEBUG:boto:Params: {} 2022-09-12T19:07:47.982 DEBUG:boto:Token: None 2022-09-12T19:07:47.982 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj 2022-09-12T19:07:47.983 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lmuPPn1s8xAxs9d668XPIML3W7Y= 2022-09-12T19:07:47.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lmuPPn1s8xAxs9d668XPIML3W7Y=', 'Content-Length': '0'} 2022-09-12T19:07:47.987 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d85152f6a4f9cec6-00631f8383-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.988 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:47.988 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:47.989 DEBUG:boto:Method: GET 2022-09-12T19:07:47.990 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:07:47.990 DEBUG:boto:Data: 2022-09-12T19:07:47.991 DEBUG:boto:Headers: {} 2022-09-12T19:07:47.991 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:47.992 DEBUG:boto:Port: 8000 2022-09-12T19:07:47.992 DEBUG:boto:Params: {} 2022-09-12T19:07:47.993 DEBUG:boto:Token: None 2022-09-12T19:07:47.994 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:47 GMT /jdpiak-20/obj 2022-09-12T19:07:47.994 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lmuPPn1s8xAxs9d668XPIML3W7Y= 2022-09-12T19:07:47.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lmuPPn1s8xAxs9d668XPIML3W7Y=', 'Content-Length': '0'} 2022-09-12T19:07:47.997 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fae1e3a51ccd51d1-00631f8383-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:07:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:47.998 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:47.999 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:48.000 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:07:48.000 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:07:48.001 DEBUG:boto:Method: GET 2022-09-12T19:07:48.001 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:07:48.002 DEBUG:boto:Data: 2022-09-12T19:07:48.003 DEBUG:boto:Headers: {} 2022-09-12T19:07:48.003 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:07:48.004 DEBUG:boto:Port: 8001 2022-09-12T19:07:48.004 DEBUG:boto:Params: {} 2022-09-12T19:07:48.005 DEBUG:boto:Token: None 2022-09-12T19:07:48.005 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:48 GMT /jdpiak-20/ 2022-09-12T19:07:48.006 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zQJEz+dHFLeenYSpHG2NFQgBW24= 2022-09-12T19:07:48.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zQJEz+dHFLeenYSpHG2NFQgBW24=', 'Content-Length': '0'} 2022-09-12T19:07:48.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aedee02119206d19-00631f8384-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:48.010 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:48.011 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:48.012 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:48.012 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:07:48.013 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:07:48.014 DEBUG:boto:Method: GET 2022-09-12T19:07:48.014 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:07:48.015 DEBUG:boto:Data: 2022-09-12T19:07:48.016 DEBUG:boto:Headers: {} 2022-09-12T19:07:48.017 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:07:48.017 DEBUG:boto:Port: 8000 2022-09-12T19:07:48.018 DEBUG:boto:Params: {} 2022-09-12T19:07:48.019 DEBUG:boto:Token: None 2022-09-12T19:07:48.020 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:07:48 GMT /jdpiak-20/ 2022-09-12T19:07:48.021 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zQJEz+dHFLeenYSpHG2NFQgBW24= 2022-09-12T19:07:48.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zQJEz+dHFLeenYSpHG2NFQgBW24=', 'Content-Length': '0'} 2022-09-12T19:07:48.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db94220dc64273c4-00631f8384-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:48.025 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:07:48.026 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-20 zones={test-zone3, test-zone2} 2022-09-12T19:07:48.027 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:48.028 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:48.028 DEBUG:boto:path=/jdpiak-20/?versioning 2022-09-12T19:07:48.029 DEBUG:boto:auth_path=/jdpiak-20/?versioning 2022-09-12T19:07:48.029 DEBUG:boto:Method: PUT 2022-09-12T19:07:48.030 DEBUG:boto:Path: /jdpiak-20/?versioning 2022-09-12T19:07:48.031 DEBUG:boto:Data: Enabled Disabled 2022-09-12T19:07:48.031 DEBUG:boto:Headers: {} 2022-09-12T19:07:48.032 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:48.033 DEBUG:boto:Port: 8000 2022-09-12T19:07:48.034 DEBUG:boto:Params: {} 2022-09-12T19:07:48.035 DEBUG:boto:Token: None 2022-09-12T19:07:48.036 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:07:48 GMT /jdpiak-20/?versioning 2022-09-12T19:07:48.036 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YTh7sAnJXffaoIuGYpTP2Uxk7r4= 2022-09-12T19:07:48.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YTh7sAnJXffaoIuGYpTP2Uxk7r4=', 'Content-Length': '228'} 2022-09-12T19:07:48.044 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000511cc4e42b12efb3-00631f8384-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:48.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:48.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:48.047 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:07:48.102 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:48.102 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:48.134 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=c0f7a512-c940-4d1f-8adf-3484630f876b 2022-09-12T19:07:48.147 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:07:48.148 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:48.148 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:07:48.148 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:47:27.976965Z" 2022-09-12T19:07:48.149 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:48.149 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:48.150 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:07:48.150 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:07:48.150 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:48.151 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:48.151 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663009668.041232_205.1", 2022-09-12T19:07:48.152 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:07:48.041232Z" 2022-09-12T19:07:48.152 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:48.152 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:48.153 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:07:48.153 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:49:40.904048Z" 2022-09-12T19:07:48.153 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:48.154 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:07:48.155 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009668.041232_205.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:07:48.156 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T19:07:48.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:48.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:48.158 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:07:48.215 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:48.216 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:48.262 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:07:48.259+0000 7f75a4b54600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:07:48.263 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:07:48.260+0000 7f75a4b54600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:07:48.281 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:07:48.281 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:07:48.281 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:07:48.282 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:07:48.282 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:07:48.282 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:07:48.283 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:07:48.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:48.283 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:07:48.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:48.284 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:07:48.284 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:48.284 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:07:48.285 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:07:48.285 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:48.285 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:07:48.285 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:48.286 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:07:48.286 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:07:48.286 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:48.286 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:48.287 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:48.287 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:07:48.287 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:48.287 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:07:48.288 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:07:48.288 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:48.288 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:07:48.288 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:48.289 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:07:48.289 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:07:48.289 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:48.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:48.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:48.290 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:07:48.290 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:48.290 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:07:48.291 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663009668.041232_205.1", 2022-09-12T19:07:48.291 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:48.291 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:07:48.291 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:48.292 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:07:48.041232Z", 2022-09-12T19:07:48.292 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:07:48.292 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:48.292 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:48.293 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:48.293 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:07:48.293 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:48.293 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:07:48.294 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:07:48.294 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:48.294 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:07:48.294 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:48.295 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:07:48.295 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:07:48.295 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:48.295 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:48.296 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:07:48.296 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:48.296 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:07:48.296 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:07:48.297 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:07:48.297 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:48.297 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:07:48Z" 2022-09-12T19:07:48.297 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:07:48.299 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009668.041232_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:07:48.041232Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:07:48Z" } 2022-09-12T19:07:48.299 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009668.041232_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:07:48.041232Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:07:48.300 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009668.041232_205.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:07:48.300 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009668.041232_205.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:07:48.301 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T19:07:48.301 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T19:07:48.302 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:48.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:48.303 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:07:48.356 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:48.357 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:48.389 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:07:48.387+0000 7f729dcdc600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:07:48.390 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:07:48.388+0000 7f729dcdc600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:07:48.404 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:07:48.404 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:07:48.405 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:07:48.405 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:07:48.405 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:07:48.405 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:07:48.406 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:07:48.406 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:48.406 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:07:48.407 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:48.407 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:07:48.407 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:48.407 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:07:48.408 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:07:48.408 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:48.408 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:07:48.408 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:48.409 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:07:48.409 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:07:48.409 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:48.410 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:48.410 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:48.410 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:07:48.410 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:48.411 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:07:48.411 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:07:48.411 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:48.412 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:07:48.412 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:48.412 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:07:48.413 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:07:48.413 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:48.413 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:48.414 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:48.414 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:07:48.414 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:48.414 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:07:48.415 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663009668.041232_205.1", 2022-09-12T19:07:48.415 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:48.415 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:07:48.415 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:48.416 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:07:48.041232Z", 2022-09-12T19:07:48.416 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:07:48.416 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:48.416 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:48.417 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:07:48.417 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:07:48.417 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:07:48.418 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:07:48.418 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:07:48.418 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:07:48.418 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:07:48.419 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:07:48.419 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:07:48.419 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:07:48.419 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:48.420 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:07:48.420 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:07:48.420 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:48.420 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:07:48.421 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:07:48.421 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:07:48.421 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:07:48.422 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:07:48Z" 2022-09-12T19:07:48.422 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:07:48.423 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009668.041232_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:07:48.041232Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:07:48Z" } 2022-09-12T19:07:48.424 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009668.041232_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:07:48.041232Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:07:48.424 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009668.041232_205.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:07:48.425 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009668.041232_205.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:07:48.425 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T19:07:48.426 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T19:07:48.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:48.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:07:48.428 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:07:48.478 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:48.478 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:48.510 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:07:48.507+0000 7f9cfbfe1600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:07:48.511 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:07:48.508+0000 7f9cfbfe1600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:07:48.523 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:07:48.524 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:07:48.524 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:07:48.524 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:07:48.524 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:07:48.525 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:07:48.525 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:07:48.525 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:48.525 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:07:48.526 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:48.526 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:07:48.526 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:48.527 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:07:48.527 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:07:48.527 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:48.527 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:07:48.528 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:48.528 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:07:48.528 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:07:48.529 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:48.529 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:48.529 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:48.529 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:07:48.530 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:48.530 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:07:48.530 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:07:48.531 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:48.531 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:07:48.531 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:48.531 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:07:48.532 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:07:48.532 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:48.532 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:48.532 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:48.533 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:07:48.533 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:48.533 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:07:48.533 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663009668.041232_205.1", 2022-09-12T19:07:48.534 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:48.534 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:07:48.534 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:48.535 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:07:48.041232Z", 2022-09-12T19:07:48.535 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:07:48.535 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:48.535 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:48.536 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:07:48.536 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:07:48.536 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:07:48.536 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:07:48.537 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:07:48.537 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:07:48.537 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:07:48.537 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:07:48.538 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:07:48.548 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:07:48.548 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:48.549 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:07:48.549 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:07:48.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:48.549 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:07:48.550 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:07:48.550 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:07:48.550 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:07:48.551 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:07:48Z" 2022-09-12T19:07:48.551 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:07:48.552 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009668.041232_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:07:48.041232Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:07:48Z" } 2022-09-12T19:07:48.553 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009668.041232_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:07:48.041232Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:07:48.554 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009668.041232_205.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:07:48.554 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009668.041232_205.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:07:48.555 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T19:07:48.555 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:07:48.556 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:07:48.556 DEBUG:boto:Method: HEAD 2022-09-12T19:07:48.557 DEBUG:boto:Path: /jdpiak-20/ 2022-09-12T19:07:48.557 DEBUG:boto:Data: 2022-09-12T19:07:48.558 DEBUG:boto:Headers: {} 2022-09-12T19:07:48.558 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:48.559 DEBUG:boto:Port: 8000 2022-09-12T19:07:48.559 DEBUG:boto:Params: {} 2022-09-12T19:07:48.560 DEBUG:boto:Token: None 2022-09-12T19:07:48.561 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:07:48 GMT /jdpiak-20/ 2022-09-12T19:07:48.561 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ULE4yQjl3cOlp4PHxRGhfIK8wcg= 2022-09-12T19:07:48.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:07:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ULE4yQjl3cOlp4PHxRGhfIK8wcg=', 'Content-Length': '0'} 2022-09-12T19:07:48.564 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed6da727be3f4d97-00631f8384-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:48.565 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:07:48.566 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:07:48.566 DEBUG:boto:Method: PUT 2022-09-12T19:07:48.567 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:07:48.567 DEBUG:boto:Data: 2022-09-12T19:07:48.568 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-12T19:07:48.568 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:07:48.569 DEBUG:boto:Port: 8000 2022-09-12T19:07:48.569 DEBUG:boto:Params: {} 2022-09-12T19:07:48.570 DEBUG:boto:Token: None 2022-09-12T19:07:48.571 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 12 Sep 2022 19:07:48 GMT /jdpiak-20/obj 2022-09-12T19:07:48.571 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PlKHadS7N6N2bM/pFHAQHfrHNlE= 2022-09-12T19:07:48.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:07:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PlKHadS7N6N2bM/pFHAQHfrHNlE='} 2022-09-12T19:07:48.592 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-amz-request-id', 'tx0000007937dee2a137a37-00631f8384-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:07:48.593 DEBUG:rgw_multi.tests:created new version id=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:07:48.593 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-20 2022-09-12T19:07:48.594 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:07:48.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:07:48.597 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:07:48.653 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:07:48.654 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:07:48.687 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:07:48.684+0000 7f7c1f783600 1 waiting for incremental sync to catch up: 2022-09-12T19:07:48.687 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000001.26.5, , ] 2022-09-12T19:07:48.688 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000003.29.12, , ] 2022-09-12T19:07:53.688 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:07:53.685+0000 7f7c1f783600 1 waiting for incremental sync to catch up: 2022-09-12T19:07:53.689 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000001.26.5, , ] 2022-09-12T19:07:53.689 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000003.29.12, , ] 2022-09-12T19:07:58.690 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:07:58.687+0000 7f7c1f783600 1 waiting for incremental sync to catch up: 2022-09-12T19:07:58.690 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000001.26.5, , ] 2022-09-12T19:07:58.691 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000003.29.12, , ] 2022-09-12T19:08:03.691 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:08:03.688+0000 7f7c1f783600 1 bucket sync caught up with source: 2022-09-12T19:08:03.692 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000003.29.12, , ] 2022-09-12T19:08:03.692 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000003.29.12, , ] 2022-09-12T19:08:03.692 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:08:03.688+0000 7f7c1f783600 0 bucket checkpoint complete 2022-09-12T19:08:03.709 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-20 2022-09-12T19:08:03.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:08:03.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:08:03.711 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:08:03.777 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:08:03.778 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:08:03.824 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:08:03.821+0000 7f5e0045f600 1 bucket sync caught up with source: 2022-09-12T19:08:03.824 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000003.29.12, , ] 2022-09-12T19:08:03.824 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000003.29.12, , ] 2022-09-12T19:08:03.825 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:08:03.821+0000 7f5e0045f600 0 bucket checkpoint complete 2022-09-12T19:08:03.837 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-20 2022-09-12T19:08:03.838 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:08:03.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:08:03.839 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:08:03.903 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:08:03.904 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:08:03.950 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:08:03.947+0000 7fe73c663600 1 waiting for incremental sync to catch up: 2022-09-12T19:08:03.951 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000001.26.5, , ] 2022-09-12T19:08:03.951 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000003.29.12, , ] 2022-09-12T19:08:08.951 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:08:08.948+0000 7fe73c663600 1 bucket sync caught up with source: 2022-09-12T19:08:08.952 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000003.29.12, , ] 2022-09-12T19:08:08.952 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000003.29.12, , ] 2022-09-12T19:08:08.952 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:08:08.948+0000 7fe73c663600 0 bucket checkpoint complete 2022-09-12T19:08:08.967 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-20 2022-09-12T19:08:08.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:08:08.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:08:08.968 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:08:09.035 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:08:09.035 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:08:09.087 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:08:09.084+0000 7f23c6851600 1 waiting for incremental sync to catch up: 2022-09-12T19:08:09.088 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000001.40.5, , ] 2022-09-12T19:08:09.088 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000003.44.12, , ] 2022-09-12T19:08:14.089 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:08:14.086+0000 7f23c6851600 1 waiting for incremental sync to catch up: 2022-09-12T19:08:14.089 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000001.40.5, , ] 2022-09-12T19:08:14.090 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000003.44.12, , ] 2022-09-12T19:08:19.091 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:08:19.087+0000 7f23c6851600 1 waiting for incremental sync to catch up: 2022-09-12T19:08:19.091 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000001.40.5, , ] 2022-09-12T19:08:19.092 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000003.44.12, , ] 2022-09-12T19:08:24.092 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:08:24.089+0000 7f23c6851600 1 bucket sync caught up with source: 2022-09-12T19:08:24.092 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000004.46.9, , ] 2022-09-12T19:08:24.093 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000003.44.12, , ] 2022-09-12T19:08:24.093 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:08:24.089+0000 7f23c6851600 0 bucket checkpoint complete 2022-09-12T19:08:24.114 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-20 2022-09-12T19:08:24.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:08:24.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:08:24.117 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:08:24.181 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:08:24.182 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:08:24.230 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:08:24.227+0000 7fbdd905d600 1 bucket sync caught up with source: 2022-09-12T19:08:24.231 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000004.46.9, , ] 2022-09-12T19:08:24.231 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000004.46.9, , ] 2022-09-12T19:08:24.232 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:08:24.227+0000 7fbdd905d600 0 bucket checkpoint complete 2022-09-12T19:08:24.248 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-20 2022-09-12T19:08:24.249 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-20 2022-09-12T19:08:24.249 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:08:24.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:08:24.250 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:08:24.308 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:08:24.309 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:08:24.344 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:08:24.341+0000 7fa01142d600 1 waiting for incremental sync to catch up: 2022-09-12T19:08:24.344 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000003.52.12, , ] 2022-09-12T19:08:24.345 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000004.54.9, , ] 2022-09-12T19:08:29.346 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:08:29.343+0000 7fa01142d600 1 waiting for incremental sync to catch up: 2022-09-12T19:08:29.346 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000003.52.12, , ] 2022-09-12T19:08:29.347 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000004.54.9, , ] 2022-09-12T19:08:34.347 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:08:34.345+0000 7fa01142d600 1 waiting for incremental sync to catch up: 2022-09-12T19:08:34.348 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000003.52.12, , ] 2022-09-12T19:08:34.348 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000004.54.9, , ] 2022-09-12T19:08:39.349 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:08:39.347+0000 7fa01142d600 1 waiting for incremental sync to catch up: 2022-09-12T19:08:39.350 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000003.52.12, , ] 2022-09-12T19:08:39.350 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000004.54.9, , ] 2022-09-12T19:08:44.350 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:08:44.347+0000 7fa01142d600 1 bucket sync caught up with source: 2022-09-12T19:08:44.351 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000004.54.9, , ] 2022-09-12T19:08:44.351 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000004.54.9, , ] 2022-09-12T19:08:44.352 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:08:44.347+0000 7fa01142d600 0 bucket checkpoint complete 2022-09-12T19:08:44.371 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-20 2022-09-12T19:08:44.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:08:44.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:08:44.373 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:08:44.438 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:08:44.438 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:08:44.496 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:08:44.493+0000 7fd31cb4f600 1 bucket sync caught up with source: 2022-09-12T19:08:44.496 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000004.54.9, , ] 2022-09-12T19:08:44.497 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000004.54.9, , ] 2022-09-12T19:08:44.497 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:08:44.493+0000 7fd31cb4f600 0 bucket checkpoint complete 2022-09-12T19:08:44.517 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-20 2022-09-12T19:08:44.517 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-20 zones={test-zone2, test-zone1} 2022-09-12T19:08:44.518 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:44.519 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:44.519 DEBUG:boto:Method: HEAD 2022-09-12T19:08:44.520 DEBUG:boto:Path: /jdpiak-20/ 2022-09-12T19:08:44.520 DEBUG:boto:Data: 2022-09-12T19:08:44.521 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.521 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:44.522 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.522 DEBUG:boto:Params: {} 2022-09-12T19:08:44.523 DEBUG:boto:Token: None 2022-09-12T19:08:44.524 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/ 2022-09-12T19:08:44.524 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4BiHQJ9P4QQcf7wrGfReMVcq4T8= 2022-09-12T19:08:44.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4BiHQJ9P4QQcf7wrGfReMVcq4T8=', 'Content-Length': '0'} 2022-09-12T19:08:44.528 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eac93abdd73df7e3-00631f83bc-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.528 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:44.529 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:44.529 DEBUG:boto:Method: HEAD 2022-09-12T19:08:44.530 DEBUG:boto:Path: /jdpiak-20/ 2022-09-12T19:08:44.530 DEBUG:boto:Data: 2022-09-12T19:08:44.531 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.531 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:44.532 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.532 DEBUG:boto:Params: {} 2022-09-12T19:08:44.533 DEBUG:boto:Token: None 2022-09-12T19:08:44.533 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/ 2022-09-12T19:08:44.534 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4BiHQJ9P4QQcf7wrGfReMVcq4T8= 2022-09-12T19:08:44.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4BiHQJ9P4QQcf7wrGfReMVcq4T8=', 'Content-Length': '0'} 2022-09-12T19:08:44.537 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab92915598bf8064-00631f83bc-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.538 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:08:44.538 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:44.539 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:44.540 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:44.540 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:44.541 DEBUG:boto:Method: GET 2022-09-12T19:08:44.541 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:44.542 DEBUG:boto:Data: 2022-09-12T19:08:44.542 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.543 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:44.543 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.544 DEBUG:boto:Params: {} 2022-09-12T19:08:44.544 DEBUG:boto:Token: None 2022-09-12T19:08:44.545 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/?versions 2022-09-12T19:08:44.545 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6XD3Hw+iPzxJ9Xott+fL0mwjEr4= 2022-09-12T19:08:44.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6XD3Hw+iPzxJ9Xott+fL0mwjEr4=', 'Content-Length': '0'} 2022-09-12T19:08:44.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d53780b908febbc-00631f83bc-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.549 DEBUG:boto:b'jdpiak-20999falseobjDt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.true2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:44.550 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:08:44.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:08:44.551 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:08:44.552 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:44.552 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:44.553 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:44.553 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:44.554 DEBUG:boto:Method: GET 2022-09-12T19:08:44.554 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:44.555 DEBUG:boto:Data: 2022-09-12T19:08:44.555 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.556 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:44.556 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.557 DEBUG:boto:Params: {} 2022-09-12T19:08:44.557 DEBUG:boto:Token: None 2022-09-12T19:08:44.558 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/?versions 2022-09-12T19:08:44.559 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6XD3Hw+iPzxJ9Xott+fL0mwjEr4= 2022-09-12T19:08:44.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6XD3Hw+iPzxJ9Xott+fL0mwjEr4=', 'Content-Length': '0'} 2022-09-12T19:08:44.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6b75d1825a841c9-00631f83bc-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.563 DEBUG:boto:b'jdpiak-20999falseobjDt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.true2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:44.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:08:44.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:08:44.565 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:44.565 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:44.566 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:44.567 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:44.568 DEBUG:boto:Method: GET 2022-09-12T19:08:44.568 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:44.569 DEBUG:boto:Data: 2022-09-12T19:08:44.569 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.570 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:44.570 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.571 DEBUG:boto:Params: {} 2022-09-12T19:08:44.571 DEBUG:boto:Token: None 2022-09-12T19:08:44.572 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/?versions 2022-09-12T19:08:44.572 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6XD3Hw+iPzxJ9Xott+fL0mwjEr4= 2022-09-12T19:08:44.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6XD3Hw+iPzxJ9Xott+fL0mwjEr4=', 'Content-Length': '0'} 2022-09-12T19:08:44.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000639a162a23e3948c-00631f83bc-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.576 DEBUG:boto:b'jdpiak-20999falseobjDt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.true2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:44.577 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:44.578 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:44.578 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:44.579 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:44.579 DEBUG:boto:Method: GET 2022-09-12T19:08:44.580 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:44.580 DEBUG:boto:Data: 2022-09-12T19:08:44.581 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.581 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:44.582 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.582 DEBUG:boto:Params: {} 2022-09-12T19:08:44.583 DEBUG:boto:Token: None 2022-09-12T19:08:44.584 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/?versions 2022-09-12T19:08:44.584 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6XD3Hw+iPzxJ9Xott+fL0mwjEr4= 2022-09-12T19:08:44.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6XD3Hw+iPzxJ9Xott+fL0mwjEr4=', 'Content-Length': '0'} 2022-09-12T19:08:44.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004be4d06286c7f86a-00631f83bc-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.588 DEBUG:boto:b'jdpiak-20999falseobjDt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.true2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:44.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:08:44.590 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:44.591 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:44.592 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.593 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.594 DEBUG:boto:Method: GET 2022-09-12T19:08:44.595 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.596 DEBUG:boto:Data: 2022-09-12T19:08:44.597 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.597 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:44.598 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.599 DEBUG:boto:Params: {} 2022-09-12T19:08:44.600 DEBUG:boto:Token: None 2022-09-12T19:08:44.602 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.602 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tK349zOJDHFcEhd8EnG7jpsc2dg= 2022-09-12T19:08:44.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tK349zOJDHFcEhd8EnG7jpsc2dg=', 'Content-Length': '0'} 2022-09-12T19:08:44.605 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000921b2d69a41a7e67-00631f83bc-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.606 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:44.606 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:44.607 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.607 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.608 DEBUG:boto:Method: GET 2022-09-12T19:08:44.608 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.609 DEBUG:boto:Data: 2022-09-12T19:08:44.610 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.610 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:44.611 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.611 DEBUG:boto:Params: {} 2022-09-12T19:08:44.612 DEBUG:boto:Token: None 2022-09-12T19:08:44.612 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.613 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tK349zOJDHFcEhd8EnG7jpsc2dg= 2022-09-12T19:08:44.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tK349zOJDHFcEhd8EnG7jpsc2dg=', 'Content-Length': '0'} 2022-09-12T19:08:44.616 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000849cef52a3d6b8ea-00631f83bc-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.617 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:44.617 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:44.618 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.618 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.619 DEBUG:boto:Method: HEAD 2022-09-12T19:08:44.619 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.620 DEBUG:boto:Data: 2022-09-12T19:08:44.620 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.621 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:44.621 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.622 DEBUG:boto:Params: {} 2022-09-12T19:08:44.622 DEBUG:boto:Token: None 2022-09-12T19:08:44.623 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.624 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Hu3/UBsDZeGcnov/v+6ZjvdNKlU= 2022-09-12T19:08:44.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Hu3/UBsDZeGcnov/v+6ZjvdNKlU=', 'Content-Length': '0'} 2022-09-12T19:08:44.626 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000212c4951c0528b61-00631f83bc-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.627 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:44.627 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:44.628 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.628 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.629 DEBUG:boto:Method: HEAD 2022-09-12T19:08:44.629 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.630 DEBUG:boto:Data: 2022-09-12T19:08:44.630 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.631 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:44.631 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.632 DEBUG:boto:Params: {} 2022-09-12T19:08:44.633 DEBUG:boto:Token: None 2022-09-12T19:08:44.633 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.634 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Hu3/UBsDZeGcnov/v+6ZjvdNKlU= 2022-09-12T19:08:44.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Hu3/UBsDZeGcnov/v+6ZjvdNKlU=', 'Content-Length': '0'} 2022-09-12T19:08:44.637 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f4f1efe71bb65b28-00631f83bc-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:08:44.638 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:44.639 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:44.639 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.640 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.641 DEBUG:boto:Method: GET 2022-09-12T19:08:44.641 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.642 DEBUG:boto:Data: 2022-09-12T19:08:44.642 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.643 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:44.643 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.644 DEBUG:boto:Params: {} 2022-09-12T19:08:44.645 DEBUG:boto:Token: None 2022-09-12T19:08:44.647 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.648 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tK349zOJDHFcEhd8EnG7jpsc2dg= 2022-09-12T19:08:44.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tK349zOJDHFcEhd8EnG7jpsc2dg=', 'Content-Length': '0'} 2022-09-12T19:08:44.651 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ca4a2df56ef94973-00631f83bc-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.652 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:44.652 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:44.653 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.653 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.654 DEBUG:boto:Method: GET 2022-09-12T19:08:44.654 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.655 DEBUG:boto:Data: 2022-09-12T19:08:44.655 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.656 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:44.656 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.657 DEBUG:boto:Params: {} 2022-09-12T19:08:44.658 DEBUG:boto:Token: None 2022-09-12T19:08:44.658 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.659 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tK349zOJDHFcEhd8EnG7jpsc2dg= 2022-09-12T19:08:44.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tK349zOJDHFcEhd8EnG7jpsc2dg=', 'Content-Length': '0'} 2022-09-12T19:08:44.662 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b5362c699a659a0f-00631f83bc-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.663 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:44.663 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:44.664 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:44.664 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:44.665 DEBUG:boto:Method: GET 2022-09-12T19:08:44.665 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:08:44.666 DEBUG:boto:Data: 2022-09-12T19:08:44.666 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.667 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:44.667 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.668 DEBUG:boto:Params: {} 2022-09-12T19:08:44.669 DEBUG:boto:Token: None 2022-09-12T19:08:44.669 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/ 2022-09-12T19:08:44.670 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zvbyfIrnOfr73wi76heG/7DpCJ4= 2022-09-12T19:08:44.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zvbyfIrnOfr73wi76heG/7DpCJ4=', 'Content-Length': '0'} 2022-09-12T19:08:44.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8ad74f2a4fff875-00631f83bc-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.714 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:44.715 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:44.716 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:44.716 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:44.717 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:44.718 DEBUG:boto:Method: GET 2022-09-12T19:08:44.719 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:08:44.720 DEBUG:boto:Data: 2022-09-12T19:08:44.721 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.722 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:44.723 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.724 DEBUG:boto:Params: {} 2022-09-12T19:08:44.725 DEBUG:boto:Token: None 2022-09-12T19:08:44.726 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/ 2022-09-12T19:08:44.727 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zvbyfIrnOfr73wi76heG/7DpCJ4= 2022-09-12T19:08:44.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zvbyfIrnOfr73wi76heG/7DpCJ4=', 'Content-Length': '0'} 2022-09-12T19:08:44.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004570b50c3f3ac74a-00631f83bc-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.732 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:44.733 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:44.733 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:44.734 DEBUG:boto:Method: HEAD 2022-09-12T19:08:44.734 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:08:44.735 DEBUG:boto:Data: 2022-09-12T19:08:44.735 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.736 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:44.736 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.737 DEBUG:boto:Params: {} 2022-09-12T19:08:44.738 DEBUG:boto:Token: None 2022-09-12T19:08:44.738 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/obj 2022-09-12T19:08:44.739 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XXhM3mB6OF08I5GpyTG8DdIPu20= 2022-09-12T19:08:44.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XXhM3mB6OF08I5GpyTG8DdIPu20=', 'Content-Length': '0'} 2022-09-12T19:08:44.742 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000071101a9c6c4782f4-00631f83bc-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.744 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:44.744 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:44.745 DEBUG:boto:Method: HEAD 2022-09-12T19:08:44.746 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:08:44.747 DEBUG:boto:Data: 2022-09-12T19:08:44.748 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.749 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:44.750 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.750 DEBUG:boto:Params: {} 2022-09-12T19:08:44.751 DEBUG:boto:Token: None 2022-09-12T19:08:44.752 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/obj 2022-09-12T19:08:44.753 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XXhM3mB6OF08I5GpyTG8DdIPu20= 2022-09-12T19:08:44.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XXhM3mB6OF08I5GpyTG8DdIPu20=', 'Content-Length': '0'} 2022-09-12T19:08:44.756 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007b6035c4e55520bd-00631f83bc-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:08:44.757 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:44.758 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:44.758 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.759 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.759 DEBUG:boto:Method: GET 2022-09-12T19:08:44.760 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.760 DEBUG:boto:Data: 2022-09-12T19:08:44.761 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.761 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:44.762 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.763 DEBUG:boto:Params: {} 2022-09-12T19:08:44.763 DEBUG:boto:Token: None 2022-09-12T19:08:44.764 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.765 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tK349zOJDHFcEhd8EnG7jpsc2dg= 2022-09-12T19:08:44.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tK349zOJDHFcEhd8EnG7jpsc2dg=', 'Content-Length': '0'} 2022-09-12T19:08:44.767 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000094f14a0606a2dc0d-00631f83bc-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.768 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:44.769 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:44.769 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.770 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.770 DEBUG:boto:Method: GET 2022-09-12T19:08:44.771 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.771 DEBUG:boto:Data: 2022-09-12T19:08:44.772 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.772 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:44.773 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.774 DEBUG:boto:Params: {} 2022-09-12T19:08:44.774 DEBUG:boto:Token: None 2022-09-12T19:08:44.775 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:44.776 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tK349zOJDHFcEhd8EnG7jpsc2dg= 2022-09-12T19:08:44.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tK349zOJDHFcEhd8EnG7jpsc2dg=', 'Content-Length': '0'} 2022-09-12T19:08:44.778 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000002887cca134fb4a3-00631f83bc-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.779 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:44.780 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:44.780 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:44.781 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:44.781 DEBUG:boto:Method: GET 2022-09-12T19:08:44.782 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:08:44.782 DEBUG:boto:Data: 2022-09-12T19:08:44.783 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.783 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:44.784 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.785 DEBUG:boto:Params: {} 2022-09-12T19:08:44.785 DEBUG:boto:Token: None 2022-09-12T19:08:44.786 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/ 2022-09-12T19:08:44.787 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zvbyfIrnOfr73wi76heG/7DpCJ4= 2022-09-12T19:08:44.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zvbyfIrnOfr73wi76heG/7DpCJ4=', 'Content-Length': '0'} 2022-09-12T19:08:44.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f8ad62ca9bb5203-00631f83bc-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.790 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:44.791 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:44.792 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:44.793 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:44.793 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:44.794 DEBUG:boto:Method: GET 2022-09-12T19:08:44.794 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:08:44.795 DEBUG:boto:Data: 2022-09-12T19:08:44.796 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.796 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:44.797 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.797 DEBUG:boto:Params: {} 2022-09-12T19:08:44.798 DEBUG:boto:Token: None 2022-09-12T19:08:44.798 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/ 2022-09-12T19:08:44.799 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zvbyfIrnOfr73wi76heG/7DpCJ4= 2022-09-12T19:08:44.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zvbyfIrnOfr73wi76heG/7DpCJ4=', 'Content-Length': '0'} 2022-09-12T19:08:44.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f254ad7f6146300f-00631f83bc-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.846 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:44.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:08:44.848 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:44.848 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:44.849 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:44.849 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:44.850 DEBUG:boto:Method: GET 2022-09-12T19:08:44.850 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:08:44.851 DEBUG:boto:Data: 2022-09-12T19:08:44.851 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.852 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:44.853 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.853 DEBUG:boto:Params: {} 2022-09-12T19:08:44.854 DEBUG:boto:Token: None 2022-09-12T19:08:44.854 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:08:44.855 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wnvbpUwOnZVrwJFxTGFuas7aasY= 2022-09-12T19:08:44.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wnvbpUwOnZVrwJFxTGFuas7aasY=', 'Content-Length': '0'} 2022-09-12T19:08:44.858 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003e6e12075f0bd000-00631f83bc-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.859 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:44.859 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:44.860 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:44.860 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:44.861 DEBUG:boto:Method: GET 2022-09-12T19:08:44.862 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:08:44.862 DEBUG:boto:Data: 2022-09-12T19:08:44.863 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.863 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:44.864 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.864 DEBUG:boto:Params: {} 2022-09-12T19:08:44.865 DEBUG:boto:Token: None 2022-09-12T19:08:44.866 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:08:44.866 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wnvbpUwOnZVrwJFxTGFuas7aasY= 2022-09-12T19:08:44.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wnvbpUwOnZVrwJFxTGFuas7aasY=', 'Content-Length': '0'} 2022-09-12T19:08:44.869 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f3088ca5a07523c3-00631f83bc-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.870 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:44.871 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:44.871 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:44.872 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:44.872 DEBUG:boto:Method: HEAD 2022-09-12T19:08:44.873 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:08:44.873 DEBUG:boto:Data: 2022-09-12T19:08:44.874 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.874 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:44.875 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.875 DEBUG:boto:Params: {} 2022-09-12T19:08:44.876 DEBUG:boto:Token: None 2022-09-12T19:08:44.876 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:08:44.877 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LsZzTs/4rDl57exHTnlx96TGu34= 2022-09-12T19:08:44.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LsZzTs/4rDl57exHTnlx96TGu34=', 'Content-Length': '0'} 2022-09-12T19:08:44.879 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b7174f65da60ee7a-00631f83bc-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.880 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:44.881 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:44.881 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:44.882 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:44.882 DEBUG:boto:Method: HEAD 2022-09-12T19:08:44.883 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:08:44.883 DEBUG:boto:Data: 2022-09-12T19:08:44.884 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.884 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:44.885 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.885 DEBUG:boto:Params: {} 2022-09-12T19:08:44.886 DEBUG:boto:Token: None 2022-09-12T19:08:44.886 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:08:44.887 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LsZzTs/4rDl57exHTnlx96TGu34= 2022-09-12T19:08:44.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LsZzTs/4rDl57exHTnlx96TGu34=', 'Content-Length': '0'} 2022-09-12T19:08:44.890 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008a236fb6906b077d-00631f83bc-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:08:44.891 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:44.892 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:44.892 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:44.893 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:44.893 DEBUG:boto:Method: GET 2022-09-12T19:08:44.894 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:08:44.894 DEBUG:boto:Data: 2022-09-12T19:08:44.895 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.895 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:44.896 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.896 DEBUG:boto:Params: {} 2022-09-12T19:08:44.897 DEBUG:boto:Token: None 2022-09-12T19:08:44.906 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:08:44.906 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wnvbpUwOnZVrwJFxTGFuas7aasY= 2022-09-12T19:08:44.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wnvbpUwOnZVrwJFxTGFuas7aasY=', 'Content-Length': '0'} 2022-09-12T19:08:44.909 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000080b22315db747443-00631f83bc-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.910 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:44.910 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:44.911 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:44.912 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:44.912 DEBUG:boto:Method: GET 2022-09-12T19:08:44.913 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:08:44.913 DEBUG:boto:Data: 2022-09-12T19:08:44.914 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.914 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:44.915 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.916 DEBUG:boto:Params: {} 2022-09-12T19:08:44.916 DEBUG:boto:Token: None 2022-09-12T19:08:44.917 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:08:44.918 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wnvbpUwOnZVrwJFxTGFuas7aasY= 2022-09-12T19:08:44.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wnvbpUwOnZVrwJFxTGFuas7aasY=', 'Content-Length': '0'} 2022-09-12T19:08:44.920 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009887a9c9831b98be-00631f83bc-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.921 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:44.922 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:44.922 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:44.923 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:44.923 DEBUG:boto:Method: GET 2022-09-12T19:08:44.924 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:08:44.924 DEBUG:boto:Data: 2022-09-12T19:08:44.925 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.925 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:44.926 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.927 DEBUG:boto:Params: {} 2022-09-12T19:08:44.927 DEBUG:boto:Token: None 2022-09-12T19:08:44.928 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/ 2022-09-12T19:08:44.928 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zvbyfIrnOfr73wi76heG/7DpCJ4= 2022-09-12T19:08:44.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zvbyfIrnOfr73wi76heG/7DpCJ4=', 'Content-Length': '0'} 2022-09-12T19:08:44.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002916b7f200b2af65-00631f83bc-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.974 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:44.975 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:44.976 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:44.976 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:44.977 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:44.978 DEBUG:boto:Method: GET 2022-09-12T19:08:44.978 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:08:44.979 DEBUG:boto:Data: 2022-09-12T19:08:44.979 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.980 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:44.980 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.981 DEBUG:boto:Params: {} 2022-09-12T19:08:44.981 DEBUG:boto:Token: None 2022-09-12T19:08:44.982 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/ 2022-09-12T19:08:44.983 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zvbyfIrnOfr73wi76heG/7DpCJ4= 2022-09-12T19:08:44.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zvbyfIrnOfr73wi76heG/7DpCJ4=', 'Content-Length': '0'} 2022-09-12T19:08:44.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000466e2e3b3d82dce4-00631f83bc-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.988 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:44.989 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:44.990 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:44.990 DEBUG:boto:Method: HEAD 2022-09-12T19:08:44.991 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:08:44.991 DEBUG:boto:Data: 2022-09-12T19:08:44.992 DEBUG:boto:Headers: {} 2022-09-12T19:08:44.992 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:44.993 DEBUG:boto:Port: 8000 2022-09-12T19:08:44.993 DEBUG:boto:Params: {} 2022-09-12T19:08:44.994 DEBUG:boto:Token: None 2022-09-12T19:08:44.995 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:44 GMT /jdpiak-20/obj 2022-09-12T19:08:44.995 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XXhM3mB6OF08I5GpyTG8DdIPu20= 2022-09-12T19:08:44.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XXhM3mB6OF08I5GpyTG8DdIPu20=', 'Content-Length': '0'} 2022-09-12T19:08:44.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000220481f4a3ce545d-00631f83bc-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:44.999 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:44.999 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.000 DEBUG:boto:Method: HEAD 2022-09-12T19:08:45.000 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:08:45.001 DEBUG:boto:Data: 2022-09-12T19:08:45.001 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.002 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:45.002 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.003 DEBUG:boto:Params: {} 2022-09-12T19:08:45.003 DEBUG:boto:Token: None 2022-09-12T19:08:45.004 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj 2022-09-12T19:08:45.005 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vRimNojtI2PNmw+RGFUK4lMtVhQ= 2022-09-12T19:08:45.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vRimNojtI2PNmw+RGFUK4lMtVhQ=', 'Content-Length': '0'} 2022-09-12T19:08:45.008 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001f36daf717e853b6-00631f83bd-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:08:45.010 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.010 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.011 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.011 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.012 DEBUG:boto:Method: GET 2022-09-12T19:08:45.013 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.013 DEBUG:boto:Data: 2022-09-12T19:08:45.014 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.014 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:45.015 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.015 DEBUG:boto:Params: {} 2022-09-12T19:08:45.016 DEBUG:boto:Token: None 2022-09-12T19:08:45.017 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.017 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI= 2022-09-12T19:08:45.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI=', 'Content-Length': '0'} 2022-09-12T19:08:45.020 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008dfd33d31cb8512d-00631f83bd-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.021 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.022 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.022 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.023 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.023 DEBUG:boto:Method: GET 2022-09-12T19:08:45.024 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.024 DEBUG:boto:Data: 2022-09-12T19:08:45.025 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.026 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:45.026 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.027 DEBUG:boto:Params: {} 2022-09-12T19:08:45.027 DEBUG:boto:Token: None 2022-09-12T19:08:45.028 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.029 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI= 2022-09-12T19:08:45.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI=', 'Content-Length': '0'} 2022-09-12T19:08:45.032 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000041a614cb76a72d9c-00631f83bd-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.033 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.033 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.034 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.034 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.035 DEBUG:boto:Method: GET 2022-09-12T19:08:45.035 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:08:45.036 DEBUG:boto:Data: 2022-09-12T19:08:45.037 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.037 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:45.038 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.038 DEBUG:boto:Params: {} 2022-09-12T19:08:45.039 DEBUG:boto:Token: None 2022-09-12T19:08:45.040 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/ 2022-09-12T19:08:45.040 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60= 2022-09-12T19:08:45.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60=', 'Content-Length': '0'} 2022-09-12T19:08:45.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000faa80bc5729d4ad5-00631f83bd-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.044 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:45.045 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.045 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.046 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.046 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.047 DEBUG:boto:Method: GET 2022-09-12T19:08:45.047 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:08:45.048 DEBUG:boto:Data: 2022-09-12T19:08:45.049 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.049 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:45.050 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.050 DEBUG:boto:Params: {} 2022-09-12T19:08:45.051 DEBUG:boto:Token: None 2022-09-12T19:08:45.051 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/ 2022-09-12T19:08:45.052 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60= 2022-09-12T19:08:45.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60=', 'Content-Length': '0'} 2022-09-12T19:08:45.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c343e938321485e-00631f83bd-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.056 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:45.057 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-20 zones={test-zone2, test-zone1} 2022-09-12T19:08:45.057 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-20 zones={test-zone3, test-zone1} 2022-09-12T19:08:45.058 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.059 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.059 DEBUG:boto:Method: HEAD 2022-09-12T19:08:45.060 DEBUG:boto:Path: /jdpiak-20/ 2022-09-12T19:08:45.060 DEBUG:boto:Data: 2022-09-12T19:08:45.061 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.061 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.062 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.062 DEBUG:boto:Params: {} 2022-09-12T19:08:45.063 DEBUG:boto:Token: None 2022-09-12T19:08:45.063 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/ 2022-09-12T19:08:45.064 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LLBZ2f7Xtc+BF1zIqDL769IPw1A= 2022-09-12T19:08:45.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LLBZ2f7Xtc+BF1zIqDL769IPw1A=', 'Content-Length': '0'} 2022-09-12T19:08:45.067 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000991b9c6e630a2c08-00631f83bd-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.068 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.068 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.069 DEBUG:boto:Method: HEAD 2022-09-12T19:08:45.069 DEBUG:boto:Path: /jdpiak-20/ 2022-09-12T19:08:45.070 DEBUG:boto:Data: 2022-09-12T19:08:45.070 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.071 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:45.072 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.072 DEBUG:boto:Params: {} 2022-09-12T19:08:45.073 DEBUG:boto:Token: None 2022-09-12T19:08:45.074 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/ 2022-09-12T19:08:45.074 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LLBZ2f7Xtc+BF1zIqDL769IPw1A= 2022-09-12T19:08:45.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LLBZ2f7Xtc+BF1zIqDL769IPw1A=', 'Content-Length': '0'} 2022-09-12T19:08:45.077 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008bc132a1c2c3a213-00631f83bd-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.078 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:08:45.079 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.079 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.080 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:45.080 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:45.081 DEBUG:boto:Method: GET 2022-09-12T19:08:45.081 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:45.082 DEBUG:boto:Data: 2022-09-12T19:08:45.082 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.083 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.083 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.084 DEBUG:boto:Params: {} 2022-09-12T19:08:45.085 DEBUG:boto:Token: None 2022-09-12T19:08:45.085 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/?versions 2022-09-12T19:08:45.086 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2omzHkKOOSaH6oCh94Kpp27cmrA= 2022-09-12T19:08:45.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2omzHkKOOSaH6oCh94Kpp27cmrA=', 'Content-Length': '0'} 2022-09-12T19:08:45.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e238c2f0f02d2532-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.090 DEBUG:boto:b'jdpiak-20999falseobjDt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.true2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:45.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:08:45.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:08:45.092 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:08:45.092 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.093 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.093 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:45.094 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:45.094 DEBUG:boto:Method: GET 2022-09-12T19:08:45.095 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:45.095 DEBUG:boto:Data: 2022-09-12T19:08:45.096 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.096 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:45.097 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.097 DEBUG:boto:Params: {} 2022-09-12T19:08:45.098 DEBUG:boto:Token: None 2022-09-12T19:08:45.099 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/?versions 2022-09-12T19:08:45.099 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2omzHkKOOSaH6oCh94Kpp27cmrA= 2022-09-12T19:08:45.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2omzHkKOOSaH6oCh94Kpp27cmrA=', 'Content-Length': '0'} 2022-09-12T19:08:45.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000126e7e69dea11d32-00631f83bd-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.146 DEBUG:boto:b'jdpiak-20999falseobjDt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.true2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:45.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:08:45.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:08:45.148 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.149 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.149 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:45.150 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:45.150 DEBUG:boto:Method: GET 2022-09-12T19:08:45.151 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:45.151 DEBUG:boto:Data: 2022-09-12T19:08:45.152 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.152 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.153 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.153 DEBUG:boto:Params: {} 2022-09-12T19:08:45.154 DEBUG:boto:Token: None 2022-09-12T19:08:45.155 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/?versions 2022-09-12T19:08:45.155 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2omzHkKOOSaH6oCh94Kpp27cmrA= 2022-09-12T19:08:45.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2omzHkKOOSaH6oCh94Kpp27cmrA=', 'Content-Length': '0'} 2022-09-12T19:08:45.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000525f6e1a24ee7c9a-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.159 DEBUG:boto:b'jdpiak-20999falseobjDt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.true2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:45.160 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.161 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.161 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:45.162 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:45.162 DEBUG:boto:Method: GET 2022-09-12T19:08:45.163 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:45.163 DEBUG:boto:Data: 2022-09-12T19:08:45.164 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.164 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:45.165 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.165 DEBUG:boto:Params: {} 2022-09-12T19:08:45.166 DEBUG:boto:Token: None 2022-09-12T19:08:45.167 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/?versions 2022-09-12T19:08:45.167 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2omzHkKOOSaH6oCh94Kpp27cmrA= 2022-09-12T19:08:45.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2omzHkKOOSaH6oCh94Kpp27cmrA=', 'Content-Length': '0'} 2022-09-12T19:08:45.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bf4a625a5bf9270-00631f83bd-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.171 DEBUG:boto:b'jdpiak-20999falseobjDt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.true2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:45.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:08:45.173 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.173 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.174 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.175 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.175 DEBUG:boto:Method: GET 2022-09-12T19:08:45.176 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.176 DEBUG:boto:Data: 2022-09-12T19:08:45.177 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.177 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.178 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.178 DEBUG:boto:Params: {} 2022-09-12T19:08:45.179 DEBUG:boto:Token: None 2022-09-12T19:08:45.180 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.180 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI= 2022-09-12T19:08:45.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI=', 'Content-Length': '0'} 2022-09-12T19:08:45.183 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000082a7fa7439ba1317-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.184 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.184 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.185 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.186 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.186 DEBUG:boto:Method: GET 2022-09-12T19:08:45.187 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.187 DEBUG:boto:Data: 2022-09-12T19:08:45.188 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.188 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:45.189 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.189 DEBUG:boto:Params: {} 2022-09-12T19:08:45.190 DEBUG:boto:Token: None 2022-09-12T19:08:45.191 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.191 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI= 2022-09-12T19:08:45.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI=', 'Content-Length': '0'} 2022-09-12T19:08:45.194 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002891663f215a781e-00631f83bd-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.195 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.196 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.196 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.197 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.197 DEBUG:boto:Method: HEAD 2022-09-12T19:08:45.198 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.198 DEBUG:boto:Data: 2022-09-12T19:08:45.199 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.199 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.200 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.200 DEBUG:boto:Params: {} 2022-09-12T19:08:45.201 DEBUG:boto:Token: None 2022-09-12T19:08:45.202 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.202 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5PH9pDYLnmiS/Lrv52CwGhSpnoI= 2022-09-12T19:08:45.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5PH9pDYLnmiS/Lrv52CwGhSpnoI=', 'Content-Length': '0'} 2022-09-12T19:08:45.205 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d1fd90bef460fc18-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.206 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.206 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.207 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.207 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.208 DEBUG:boto:Method: HEAD 2022-09-12T19:08:45.208 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.209 DEBUG:boto:Data: 2022-09-12T19:08:45.209 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.210 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:45.210 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.211 DEBUG:boto:Params: {} 2022-09-12T19:08:45.211 DEBUG:boto:Token: None 2022-09-12T19:08:45.212 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.213 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5PH9pDYLnmiS/Lrv52CwGhSpnoI= 2022-09-12T19:08:45.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5PH9pDYLnmiS/Lrv52CwGhSpnoI=', 'Content-Length': '0'} 2022-09-12T19:08:45.215 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009ea7c2b2b5224f06-00631f83bd-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:08:45.217 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.217 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.218 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.218 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.219 DEBUG:boto:Method: GET 2022-09-12T19:08:45.219 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.220 DEBUG:boto:Data: 2022-09-12T19:08:45.220 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.221 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.222 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.222 DEBUG:boto:Params: {} 2022-09-12T19:08:45.223 DEBUG:boto:Token: None 2022-09-12T19:08:45.223 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.224 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI= 2022-09-12T19:08:45.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI=', 'Content-Length': '0'} 2022-09-12T19:08:45.227 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001355ac2ceddf217d-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.228 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.229 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.230 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.230 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.231 DEBUG:boto:Method: GET 2022-09-12T19:08:45.231 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.232 DEBUG:boto:Data: 2022-09-12T19:08:45.233 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.233 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:45.234 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.235 DEBUG:boto:Params: {} 2022-09-12T19:08:45.236 DEBUG:boto:Token: None 2022-09-12T19:08:45.237 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.237 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI= 2022-09-12T19:08:45.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI=', 'Content-Length': '0'} 2022-09-12T19:08:45.240 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b57546ae77148142-00631f83bd-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.241 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.242 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.243 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.244 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.244 DEBUG:boto:Method: GET 2022-09-12T19:08:45.245 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:08:45.246 DEBUG:boto:Data: 2022-09-12T19:08:45.247 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.247 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.248 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.249 DEBUG:boto:Params: {} 2022-09-12T19:08:45.250 DEBUG:boto:Token: None 2022-09-12T19:08:45.251 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/ 2022-09-12T19:08:45.252 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60= 2022-09-12T19:08:45.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60=', 'Content-Length': '0'} 2022-09-12T19:08:45.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d47d69e61974b8ee-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.299 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:45.300 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.301 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.301 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.302 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.302 DEBUG:boto:Method: GET 2022-09-12T19:08:45.303 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:08:45.303 DEBUG:boto:Data: 2022-09-12T19:08:45.304 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.304 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:45.305 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.306 DEBUG:boto:Params: {} 2022-09-12T19:08:45.306 DEBUG:boto:Token: None 2022-09-12T19:08:45.307 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/ 2022-09-12T19:08:45.308 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60= 2022-09-12T19:08:45.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60=', 'Content-Length': '0'} 2022-09-12T19:08:45.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f83c9eeac6e1e79-00631f83bd-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.312 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:45.312 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.313 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.314 DEBUG:boto:Method: HEAD 2022-09-12T19:08:45.314 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:08:45.315 DEBUG:boto:Data: 2022-09-12T19:08:45.315 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.316 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.316 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.317 DEBUG:boto:Params: {} 2022-09-12T19:08:45.317 DEBUG:boto:Token: None 2022-09-12T19:08:45.318 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj 2022-09-12T19:08:45.318 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vRimNojtI2PNmw+RGFUK4lMtVhQ= 2022-09-12T19:08:45.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vRimNojtI2PNmw+RGFUK4lMtVhQ=', 'Content-Length': '0'} 2022-09-12T19:08:45.322 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004a6a6eb7796d3a78-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.323 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.323 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.324 DEBUG:boto:Method: HEAD 2022-09-12T19:08:45.324 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:08:45.325 DEBUG:boto:Data: 2022-09-12T19:08:45.325 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.326 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:45.326 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.327 DEBUG:boto:Params: {} 2022-09-12T19:08:45.327 DEBUG:boto:Token: None 2022-09-12T19:08:45.328 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj 2022-09-12T19:08:45.329 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vRimNojtI2PNmw+RGFUK4lMtVhQ= 2022-09-12T19:08:45.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vRimNojtI2PNmw+RGFUK4lMtVhQ=', 'Content-Length': '0'} 2022-09-12T19:08:45.332 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005010cdd13e2eca22-00631f83bd-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:08:45.333 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.334 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.334 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.335 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.335 DEBUG:boto:Method: GET 2022-09-12T19:08:45.336 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.336 DEBUG:boto:Data: 2022-09-12T19:08:45.337 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.338 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.338 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.339 DEBUG:boto:Params: {} 2022-09-12T19:08:45.339 DEBUG:boto:Token: None 2022-09-12T19:08:45.340 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.340 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI= 2022-09-12T19:08:45.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI=', 'Content-Length': '0'} 2022-09-12T19:08:45.343 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b2cc33eaad9ac5df-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.344 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.345 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.345 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.346 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.346 DEBUG:boto:Method: GET 2022-09-12T19:08:45.347 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.347 DEBUG:boto:Data: 2022-09-12T19:08:45.348 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.348 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:45.349 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.349 DEBUG:boto:Params: {} 2022-09-12T19:08:45.350 DEBUG:boto:Token: None 2022-09-12T19:08:45.351 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.351 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI= 2022-09-12T19:08:45.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI=', 'Content-Length': '0'} 2022-09-12T19:08:45.354 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a8453af49130e1f7-00631f83bd-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.355 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.356 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.356 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.357 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.357 DEBUG:boto:Method: GET 2022-09-12T19:08:45.358 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:08:45.358 DEBUG:boto:Data: 2022-09-12T19:08:45.359 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.359 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.360 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.360 DEBUG:boto:Params: {} 2022-09-12T19:08:45.361 DEBUG:boto:Token: None 2022-09-12T19:08:45.362 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/ 2022-09-12T19:08:45.362 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60= 2022-09-12T19:08:45.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60=', 'Content-Length': '0'} 2022-09-12T19:08:45.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec4e8a2dff6d3fd2-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.366 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:45.367 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.368 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.368 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.369 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.369 DEBUG:boto:Method: GET 2022-09-12T19:08:45.370 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:08:45.370 DEBUG:boto:Data: 2022-09-12T19:08:45.371 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.371 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:45.372 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.372 DEBUG:boto:Params: {} 2022-09-12T19:08:45.373 DEBUG:boto:Token: None 2022-09-12T19:08:45.374 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/ 2022-09-12T19:08:45.374 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60= 2022-09-12T19:08:45.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60=', 'Content-Length': '0'} 2022-09-12T19:08:45.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000479eb017e7fc7eff-00631f83bd-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.378 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:45.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:08:45.380 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.380 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.381 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.381 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.382 DEBUG:boto:Method: GET 2022-09-12T19:08:45.382 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.383 DEBUG:boto:Data: 2022-09-12T19:08:45.383 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.384 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.384 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.385 DEBUG:boto:Params: {} 2022-09-12T19:08:45.385 DEBUG:boto:Token: None 2022-09-12T19:08:45.386 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.387 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7GNT81RlR1kWvIfxvuJJ0utPwWU= 2022-09-12T19:08:45.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7GNT81RlR1kWvIfxvuJJ0utPwWU=', 'Content-Length': '0'} 2022-09-12T19:08:45.390 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b41c091968c573ce-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.391 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.392 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.393 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.394 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.394 DEBUG:boto:Method: GET 2022-09-12T19:08:45.395 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.395 DEBUG:boto:Data: 2022-09-12T19:08:45.396 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.397 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:45.397 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.399 DEBUG:boto:Params: {} 2022-09-12T19:08:45.399 DEBUG:boto:Token: None 2022-09-12T19:08:45.400 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.401 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7GNT81RlR1kWvIfxvuJJ0utPwWU= 2022-09-12T19:08:45.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7GNT81RlR1kWvIfxvuJJ0utPwWU=', 'Content-Length': '0'} 2022-09-12T19:08:45.404 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000034aaabe70f4d9f17-00631f83bd-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.404 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.405 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.405 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.406 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.407 DEBUG:boto:Method: HEAD 2022-09-12T19:08:45.407 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.408 DEBUG:boto:Data: 2022-09-12T19:08:45.408 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.408 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.409 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.409 DEBUG:boto:Params: {} 2022-09-12T19:08:45.410 DEBUG:boto:Token: None 2022-09-12T19:08:45.411 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.411 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x51FvIqkx0YrgsA9a99SenYozO8= 2022-09-12T19:08:45.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x51FvIqkx0YrgsA9a99SenYozO8=', 'Content-Length': '0'} 2022-09-12T19:08:45.414 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000549cca4ab0d83811-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.415 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.415 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.416 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.416 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.417 DEBUG:boto:Method: HEAD 2022-09-12T19:08:45.417 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.418 DEBUG:boto:Data: 2022-09-12T19:08:45.418 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.419 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:45.419 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.420 DEBUG:boto:Params: {} 2022-09-12T19:08:45.420 DEBUG:boto:Token: None 2022-09-12T19:08:45.421 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.421 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x51FvIqkx0YrgsA9a99SenYozO8= 2022-09-12T19:08:45.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x51FvIqkx0YrgsA9a99SenYozO8=', 'Content-Length': '0'} 2022-09-12T19:08:45.424 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d8d69b0dff646af4-00631f83bd-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:08:45.425 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.426 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.427 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.427 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.428 DEBUG:boto:Method: GET 2022-09-12T19:08:45.428 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.429 DEBUG:boto:Data: 2022-09-12T19:08:45.429 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.430 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.430 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.431 DEBUG:boto:Params: {} 2022-09-12T19:08:45.431 DEBUG:boto:Token: None 2022-09-12T19:08:45.432 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.433 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7GNT81RlR1kWvIfxvuJJ0utPwWU= 2022-09-12T19:08:45.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7GNT81RlR1kWvIfxvuJJ0utPwWU=', 'Content-Length': '0'} 2022-09-12T19:08:45.435 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009b2698f5997083cc-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.436 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.437 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.437 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.438 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.438 DEBUG:boto:Method: GET 2022-09-12T19:08:45.439 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.439 DEBUG:boto:Data: 2022-09-12T19:08:45.440 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.440 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:45.441 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.441 DEBUG:boto:Params: {} 2022-09-12T19:08:45.442 DEBUG:boto:Token: None 2022-09-12T19:08:45.442 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.443 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7GNT81RlR1kWvIfxvuJJ0utPwWU= 2022-09-12T19:08:45.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7GNT81RlR1kWvIfxvuJJ0utPwWU=', 'Content-Length': '0'} 2022-09-12T19:08:45.446 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000ac4fcacea3dddb4-00631f83bd-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.446 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.447 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.448 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.448 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.449 DEBUG:boto:Method: GET 2022-09-12T19:08:45.449 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:08:45.450 DEBUG:boto:Data: 2022-09-12T19:08:45.450 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.451 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.451 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.452 DEBUG:boto:Params: {} 2022-09-12T19:08:45.452 DEBUG:boto:Token: None 2022-09-12T19:08:45.453 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/ 2022-09-12T19:08:45.453 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60= 2022-09-12T19:08:45.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60=', 'Content-Length': '0'} 2022-09-12T19:08:45.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2f417e782cd13c1-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.457 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:45.458 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.459 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.459 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.460 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.460 DEBUG:boto:Method: GET 2022-09-12T19:08:45.461 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:08:45.461 DEBUG:boto:Data: 2022-09-12T19:08:45.462 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.462 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:45.463 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.463 DEBUG:boto:Params: {} 2022-09-12T19:08:45.464 DEBUG:boto:Token: None 2022-09-12T19:08:45.465 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/ 2022-09-12T19:08:45.465 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60= 2022-09-12T19:08:45.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60=', 'Content-Length': '0'} 2022-09-12T19:08:45.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a52c5c4f72363c8-00631f83bd-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.469 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:45.470 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.470 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.471 DEBUG:boto:Method: HEAD 2022-09-12T19:08:45.472 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:08:45.472 DEBUG:boto:Data: 2022-09-12T19:08:45.472 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.473 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.473 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.474 DEBUG:boto:Params: {} 2022-09-12T19:08:45.475 DEBUG:boto:Token: None 2022-09-12T19:08:45.475 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj 2022-09-12T19:08:45.476 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vRimNojtI2PNmw+RGFUK4lMtVhQ= 2022-09-12T19:08:45.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vRimNojtI2PNmw+RGFUK4lMtVhQ=', 'Content-Length': '0'} 2022-09-12T19:08:45.479 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004ee1f25c6c56bbbf-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.480 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.480 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.481 DEBUG:boto:Method: HEAD 2022-09-12T19:08:45.481 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:08:45.482 DEBUG:boto:Data: 2022-09-12T19:08:45.482 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.483 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:45.483 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.484 DEBUG:boto:Params: {} 2022-09-12T19:08:45.484 DEBUG:boto:Token: None 2022-09-12T19:08:45.485 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj 2022-09-12T19:08:45.485 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vRimNojtI2PNmw+RGFUK4lMtVhQ= 2022-09-12T19:08:45.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vRimNojtI2PNmw+RGFUK4lMtVhQ=', 'Content-Length': '0'} 2022-09-12T19:08:45.488 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005d83a7b366e0bae7-00631f83bd-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:08:45.489 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.490 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.491 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.491 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.492 DEBUG:boto:Method: GET 2022-09-12T19:08:45.492 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.493 DEBUG:boto:Data: 2022-09-12T19:08:45.493 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.494 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.494 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.495 DEBUG:boto:Params: {} 2022-09-12T19:08:45.495 DEBUG:boto:Token: None 2022-09-12T19:08:45.496 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.497 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI= 2022-09-12T19:08:45.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI=', 'Content-Length': '0'} 2022-09-12T19:08:45.499 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000df8d5886f32c21d1-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.500 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.500 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.501 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.501 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.502 DEBUG:boto:Method: GET 2022-09-12T19:08:45.503 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.503 DEBUG:boto:Data: 2022-09-12T19:08:45.504 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.504 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:45.505 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.505 DEBUG:boto:Params: {} 2022-09-12T19:08:45.506 DEBUG:boto:Token: None 2022-09-12T19:08:45.506 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.507 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI= 2022-09-12T19:08:45.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI=', 'Content-Length': '0'} 2022-09-12T19:08:45.510 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d36b64d291a645b2-00631f83bd-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.511 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.511 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.512 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.512 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.513 DEBUG:boto:Method: GET 2022-09-12T19:08:45.513 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:08:45.514 DEBUG:boto:Data: 2022-09-12T19:08:45.514 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.515 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.515 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.516 DEBUG:boto:Params: {} 2022-09-12T19:08:45.516 DEBUG:boto:Token: None 2022-09-12T19:08:45.517 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/ 2022-09-12T19:08:45.518 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60= 2022-09-12T19:08:45.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60=', 'Content-Length': '0'} 2022-09-12T19:08:45.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd0eb56ccd1a9b55-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.562 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:45.563 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.564 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.564 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.565 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.565 DEBUG:boto:Method: GET 2022-09-12T19:08:45.566 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:08:45.566 DEBUG:boto:Data: 2022-09-12T19:08:45.567 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.567 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:45.568 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.568 DEBUG:boto:Params: {} 2022-09-12T19:08:45.569 DEBUG:boto:Token: None 2022-09-12T19:08:45.569 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/ 2022-09-12T19:08:45.570 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60= 2022-09-12T19:08:45.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60=', 'Content-Length': '0'} 2022-09-12T19:08:45.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093616747f2041587-00631f83bd-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.574 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:45.575 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-20 zones={test-zone3, test-zone1} 2022-09-12T19:08:45.575 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-20 zones={test-zone3, test-zone2} 2022-09-12T19:08:45.576 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.576 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.577 DEBUG:boto:Method: HEAD 2022-09-12T19:08:45.577 DEBUG:boto:Path: /jdpiak-20/ 2022-09-12T19:08:45.578 DEBUG:boto:Data: 2022-09-12T19:08:45.578 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.579 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.579 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.580 DEBUG:boto:Params: {} 2022-09-12T19:08:45.580 DEBUG:boto:Token: None 2022-09-12T19:08:45.581 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/ 2022-09-12T19:08:45.582 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LLBZ2f7Xtc+BF1zIqDL769IPw1A= 2022-09-12T19:08:45.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LLBZ2f7Xtc+BF1zIqDL769IPw1A=', 'Content-Length': '0'} 2022-09-12T19:08:45.585 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0642d6240314818-00631f83bd-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.585 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.586 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.586 DEBUG:boto:Method: HEAD 2022-09-12T19:08:45.587 DEBUG:boto:Path: /jdpiak-20/ 2022-09-12T19:08:45.587 DEBUG:boto:Data: 2022-09-12T19:08:45.588 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.588 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:45.589 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.589 DEBUG:boto:Params: {} 2022-09-12T19:08:45.590 DEBUG:boto:Token: None 2022-09-12T19:08:45.590 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/ 2022-09-12T19:08:45.591 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LLBZ2f7Xtc+BF1zIqDL769IPw1A= 2022-09-12T19:08:45.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LLBZ2f7Xtc+BF1zIqDL769IPw1A=', 'Content-Length': '0'} 2022-09-12T19:08:45.594 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065468644dd198d6c-00631f83bd-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.595 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:08:45.595 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.596 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.596 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:45.597 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:45.597 DEBUG:boto:Method: GET 2022-09-12T19:08:45.598 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:45.598 DEBUG:boto:Data: 2022-09-12T19:08:45.599 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.599 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.600 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.600 DEBUG:boto:Params: {} 2022-09-12T19:08:45.601 DEBUG:boto:Token: None 2022-09-12T19:08:45.601 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/?versions 2022-09-12T19:08:45.602 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2omzHkKOOSaH6oCh94Kpp27cmrA= 2022-09-12T19:08:45.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2omzHkKOOSaH6oCh94Kpp27cmrA=', 'Content-Length': '0'} 2022-09-12T19:08:45.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036f2e7f3aee9a431-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.606 DEBUG:boto:b'jdpiak-20999falseobjDt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.true2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:45.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:08:45.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:08:45.608 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:08:45.608 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.609 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.609 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:45.610 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:45.610 DEBUG:boto:Method: GET 2022-09-12T19:08:45.611 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:45.611 DEBUG:boto:Data: 2022-09-12T19:08:45.612 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.612 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:45.613 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.613 DEBUG:boto:Params: {} 2022-09-12T19:08:45.614 DEBUG:boto:Token: None 2022-09-12T19:08:45.614 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/?versions 2022-09-12T19:08:45.615 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2omzHkKOOSaH6oCh94Kpp27cmrA= 2022-09-12T19:08:45.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2omzHkKOOSaH6oCh94Kpp27cmrA=', 'Content-Length': '0'} 2022-09-12T19:08:45.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd31e8b2fa84501e-00631f83bd-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.619 DEBUG:boto:b'jdpiak-20999falseobjDt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.true2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:45.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:08:45.620 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:08:45.620 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.621 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.621 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:45.622 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:45.622 DEBUG:boto:Method: GET 2022-09-12T19:08:45.623 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:45.623 DEBUG:boto:Data: 2022-09-12T19:08:45.624 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.624 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.625 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.625 DEBUG:boto:Params: {} 2022-09-12T19:08:45.626 DEBUG:boto:Token: None 2022-09-12T19:08:45.627 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/?versions 2022-09-12T19:08:45.627 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2omzHkKOOSaH6oCh94Kpp27cmrA= 2022-09-12T19:08:45.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2omzHkKOOSaH6oCh94Kpp27cmrA=', 'Content-Length': '0'} 2022-09-12T19:08:45.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021552043321e41da-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.631 DEBUG:boto:b'jdpiak-20999falseobjDt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.true2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:45.632 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.633 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.633 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:45.634 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:45.634 DEBUG:boto:Method: GET 2022-09-12T19:08:45.635 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:08:45.635 DEBUG:boto:Data: 2022-09-12T19:08:45.636 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.636 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:45.637 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.637 DEBUG:boto:Params: {} 2022-09-12T19:08:45.638 DEBUG:boto:Token: None 2022-09-12T19:08:45.638 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/?versions 2022-09-12T19:08:45.639 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2omzHkKOOSaH6oCh94Kpp27cmrA= 2022-09-12T19:08:45.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2omzHkKOOSaH6oCh94Kpp27cmrA=', 'Content-Length': '0'} 2022-09-12T19:08:45.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007eb1b4e20eeffb2c-00631f83bd-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.682 DEBUG:boto:b'jdpiak-20999falseobjDt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.true2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-12T19:07:15.931Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:45.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:08:45.684 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.684 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.685 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.685 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.686 DEBUG:boto:Method: GET 2022-09-12T19:08:45.686 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.687 DEBUG:boto:Data: 2022-09-12T19:08:45.687 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.688 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.689 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.689 DEBUG:boto:Params: {} 2022-09-12T19:08:45.690 DEBUG:boto:Token: None 2022-09-12T19:08:45.690 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.691 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI= 2022-09-12T19:08:45.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI=', 'Content-Length': '0'} 2022-09-12T19:08:45.694 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b62d067a335756f7-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.695 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.695 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.696 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.696 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.697 DEBUG:boto:Method: GET 2022-09-12T19:08:45.697 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.698 DEBUG:boto:Data: 2022-09-12T19:08:45.698 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.699 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:45.700 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.700 DEBUG:boto:Params: {} 2022-09-12T19:08:45.701 DEBUG:boto:Token: None 2022-09-12T19:08:45.701 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.702 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI= 2022-09-12T19:08:45.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI=', 'Content-Length': '0'} 2022-09-12T19:08:45.704 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000037a19be3477d369f-00631f83bd-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.705 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.706 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.706 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.707 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.707 DEBUG:boto:Method: HEAD 2022-09-12T19:08:45.708 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.708 DEBUG:boto:Data: 2022-09-12T19:08:45.709 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.709 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.710 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.710 DEBUG:boto:Params: {} 2022-09-12T19:08:45.711 DEBUG:boto:Token: None 2022-09-12T19:08:45.711 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.712 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5PH9pDYLnmiS/Lrv52CwGhSpnoI= 2022-09-12T19:08:45.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5PH9pDYLnmiS/Lrv52CwGhSpnoI=', 'Content-Length': '0'} 2022-09-12T19:08:45.714 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008c5306528c3e66d3-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.715 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.716 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.716 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.717 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.717 DEBUG:boto:Method: HEAD 2022-09-12T19:08:45.717 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.718 DEBUG:boto:Data: 2022-09-12T19:08:45.718 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.719 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:45.719 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.720 DEBUG:boto:Params: {} 2022-09-12T19:08:45.721 DEBUG:boto:Token: None 2022-09-12T19:08:45.721 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.722 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5PH9pDYLnmiS/Lrv52CwGhSpnoI= 2022-09-12T19:08:45.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5PH9pDYLnmiS/Lrv52CwGhSpnoI=', 'Content-Length': '0'} 2022-09-12T19:08:45.725 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000074249d024eef6999-00631f83bd-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:08:45.726 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.727 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.727 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.728 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.729 DEBUG:boto:Method: GET 2022-09-12T19:08:45.729 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.730 DEBUG:boto:Data: 2022-09-12T19:08:45.730 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.731 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.731 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.732 DEBUG:boto:Params: {} 2022-09-12T19:08:45.732 DEBUG:boto:Token: None 2022-09-12T19:08:45.733 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.734 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI= 2022-09-12T19:08:45.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI=', 'Content-Length': '0'} 2022-09-12T19:08:45.736 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000140fae85d2d75af6-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.737 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.737 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.738 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.738 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.739 DEBUG:boto:Method: GET 2022-09-12T19:08:45.740 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.740 DEBUG:boto:Data: 2022-09-12T19:08:45.741 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.741 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:45.742 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.742 DEBUG:boto:Params: {} 2022-09-12T19:08:45.743 DEBUG:boto:Token: None 2022-09-12T19:08:45.743 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.744 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI= 2022-09-12T19:08:45.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI=', 'Content-Length': '0'} 2022-09-12T19:08:45.747 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006a4deb2b19deb0b6-00631f83bd-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.747 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.748 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.748 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.749 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.749 DEBUG:boto:Method: GET 2022-09-12T19:08:45.750 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:08:45.750 DEBUG:boto:Data: 2022-09-12T19:08:45.751 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.751 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.752 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.753 DEBUG:boto:Params: {} 2022-09-12T19:08:45.753 DEBUG:boto:Token: None 2022-09-12T19:08:45.754 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/ 2022-09-12T19:08:45.754 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60= 2022-09-12T19:08:45.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60=', 'Content-Length': '0'} 2022-09-12T19:08:45.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b42234dc92f2e332-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.759 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:45.759 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.760 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.760 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.761 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.761 DEBUG:boto:Method: GET 2022-09-12T19:08:45.762 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:08:45.762 DEBUG:boto:Data: 2022-09-12T19:08:45.763 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.763 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:45.764 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.764 DEBUG:boto:Params: {} 2022-09-12T19:08:45.765 DEBUG:boto:Token: None 2022-09-12T19:08:45.766 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/ 2022-09-12T19:08:45.766 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60= 2022-09-12T19:08:45.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60=', 'Content-Length': '0'} 2022-09-12T19:08:45.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009398487f9720964a-00631f83bd-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.770 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:45.771 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.772 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.772 DEBUG:boto:Method: HEAD 2022-09-12T19:08:45.773 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:08:45.773 DEBUG:boto:Data: 2022-09-12T19:08:45.774 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.774 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.775 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.775 DEBUG:boto:Params: {} 2022-09-12T19:08:45.776 DEBUG:boto:Token: None 2022-09-12T19:08:45.777 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj 2022-09-12T19:08:45.778 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vRimNojtI2PNmw+RGFUK4lMtVhQ= 2022-09-12T19:08:45.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vRimNojtI2PNmw+RGFUK4lMtVhQ=', 'Content-Length': '0'} 2022-09-12T19:08:45.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004a9eba604b4252e2-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.782 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.783 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.784 DEBUG:boto:Method: HEAD 2022-09-12T19:08:45.784 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:08:45.785 DEBUG:boto:Data: 2022-09-12T19:08:45.785 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.786 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:45.786 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.787 DEBUG:boto:Params: {} 2022-09-12T19:08:45.787 DEBUG:boto:Token: None 2022-09-12T19:08:45.788 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj 2022-09-12T19:08:45.788 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vRimNojtI2PNmw+RGFUK4lMtVhQ= 2022-09-12T19:08:45.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vRimNojtI2PNmw+RGFUK4lMtVhQ=', 'Content-Length': '0'} 2022-09-12T19:08:45.791 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000513b3dcccbc11e2d-00631f83bd-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:08:45.792 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.793 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.793 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.794 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.795 DEBUG:boto:Method: GET 2022-09-12T19:08:45.795 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.796 DEBUG:boto:Data: 2022-09-12T19:08:45.796 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.797 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.797 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.798 DEBUG:boto:Params: {} 2022-09-12T19:08:45.799 DEBUG:boto:Token: None 2022-09-12T19:08:45.799 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.800 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI= 2022-09-12T19:08:45.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI=', 'Content-Length': '0'} 2022-09-12T19:08:45.803 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f13cb45e4bdc4866-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.804 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.804 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.805 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.805 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.806 DEBUG:boto:Method: GET 2022-09-12T19:08:45.806 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.807 DEBUG:boto:Data: 2022-09-12T19:08:45.808 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.808 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:45.809 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.810 DEBUG:boto:Params: {} 2022-09-12T19:08:45.810 DEBUG:boto:Token: None 2022-09-12T19:08:45.811 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.812 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI= 2022-09-12T19:08:45.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0JkxTAZB6FvmTSGW+Qj7gLpuyAI=', 'Content-Length': '0'} 2022-09-12T19:08:45.814 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000033007fb2ff7e3a5a-00631f83bd-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.815 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.816 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.816 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.817 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.817 DEBUG:boto:Method: GET 2022-09-12T19:08:45.818 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:08:45.819 DEBUG:boto:Data: 2022-09-12T19:08:45.819 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.820 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.821 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.821 DEBUG:boto:Params: {} 2022-09-12T19:08:45.822 DEBUG:boto:Token: None 2022-09-12T19:08:45.822 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/ 2022-09-12T19:08:45.823 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60= 2022-09-12T19:08:45.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60=', 'Content-Length': '0'} 2022-09-12T19:08:45.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ffa46c93092a9bb-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.870 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:45.871 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.872 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.872 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.873 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.873 DEBUG:boto:Method: GET 2022-09-12T19:08:45.874 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:08:45.874 DEBUG:boto:Data: 2022-09-12T19:08:45.875 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.875 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:45.876 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.876 DEBUG:boto:Params: {} 2022-09-12T19:08:45.877 DEBUG:boto:Token: None 2022-09-12T19:08:45.878 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/ 2022-09-12T19:08:45.878 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60= 2022-09-12T19:08:45.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60=', 'Content-Length': '0'} 2022-09-12T19:08:45.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac410f39159b4f5a-00631f83bd-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.882 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:45.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:08:45.884 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.884 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.885 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.885 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.886 DEBUG:boto:Method: GET 2022-09-12T19:08:45.886 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.887 DEBUG:boto:Data: 2022-09-12T19:08:45.887 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.888 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.888 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.889 DEBUG:boto:Params: {} 2022-09-12T19:08:45.890 DEBUG:boto:Token: None 2022-09-12T19:08:45.890 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.891 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7GNT81RlR1kWvIfxvuJJ0utPwWU= 2022-09-12T19:08:45.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7GNT81RlR1kWvIfxvuJJ0utPwWU=', 'Content-Length': '0'} 2022-09-12T19:08:45.894 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008cf10a6d96056843-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.895 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.895 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.896 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.896 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.897 DEBUG:boto:Method: GET 2022-09-12T19:08:45.897 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.898 DEBUG:boto:Data: 2022-09-12T19:08:45.898 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.899 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:45.899 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.900 DEBUG:boto:Params: {} 2022-09-12T19:08:45.901 DEBUG:boto:Token: None 2022-09-12T19:08:45.901 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.902 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7GNT81RlR1kWvIfxvuJJ0utPwWU= 2022-09-12T19:08:45.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7GNT81RlR1kWvIfxvuJJ0utPwWU=', 'Content-Length': '0'} 2022-09-12T19:08:45.905 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000130c2152368f2e89-00631f83bd-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.905 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.906 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.906 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.907 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.907 DEBUG:boto:Method: HEAD 2022-09-12T19:08:45.908 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.908 DEBUG:boto:Data: 2022-09-12T19:08:45.909 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.909 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.910 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.910 DEBUG:boto:Params: {} 2022-09-12T19:08:45.911 DEBUG:boto:Token: None 2022-09-12T19:08:45.911 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.912 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x51FvIqkx0YrgsA9a99SenYozO8= 2022-09-12T19:08:45.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x51FvIqkx0YrgsA9a99SenYozO8=', 'Content-Length': '0'} 2022-09-12T19:08:45.915 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000029ab096b114327f7-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.916 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.916 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.917 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.917 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.918 DEBUG:boto:Method: HEAD 2022-09-12T19:08:45.918 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.919 DEBUG:boto:Data: 2022-09-12T19:08:45.919 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.920 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:45.920 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.921 DEBUG:boto:Params: {} 2022-09-12T19:08:45.921 DEBUG:boto:Token: None 2022-09-12T19:08:45.922 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.922 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x51FvIqkx0YrgsA9a99SenYozO8= 2022-09-12T19:08:45.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x51FvIqkx0YrgsA9a99SenYozO8=', 'Content-Length': '0'} 2022-09-12T19:08:45.925 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bc4fb560b8dddf63-00631f83bd-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:08:45.927 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.927 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.928 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.928 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.929 DEBUG:boto:Method: GET 2022-09-12T19:08:45.929 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.930 DEBUG:boto:Data: 2022-09-12T19:08:45.930 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.931 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.931 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.932 DEBUG:boto:Params: {} 2022-09-12T19:08:45.933 DEBUG:boto:Token: None 2022-09-12T19:08:45.933 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.934 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7GNT81RlR1kWvIfxvuJJ0utPwWU= 2022-09-12T19:08:45.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7GNT81RlR1kWvIfxvuJJ0utPwWU=', 'Content-Length': '0'} 2022-09-12T19:08:45.937 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cd5e9c18473cca14-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.937 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.938 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.939 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.939 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.940 DEBUG:boto:Method: GET 2022-09-12T19:08:45.940 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.941 DEBUG:boto:Data: 2022-09-12T19:08:45.941 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.942 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:45.942 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.943 DEBUG:boto:Params: {} 2022-09-12T19:08:45.943 DEBUG:boto:Token: None 2022-09-12T19:08:45.944 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:08:45.945 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7GNT81RlR1kWvIfxvuJJ0utPwWU= 2022-09-12T19:08:45.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7GNT81RlR1kWvIfxvuJJ0utPwWU=', 'Content-Length': '0'} 2022-09-12T19:08:45.947 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f32b438eee177fe6-00631f83bd-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.948 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.949 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.949 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.950 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.950 DEBUG:boto:Method: GET 2022-09-12T19:08:45.951 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:08:45.951 DEBUG:boto:Data: 2022-09-12T19:08:45.952 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.952 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.953 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.953 DEBUG:boto:Params: {} 2022-09-12T19:08:45.954 DEBUG:boto:Token: None 2022-09-12T19:08:45.955 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/ 2022-09-12T19:08:45.955 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60= 2022-09-12T19:08:45.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60=', 'Content-Length': '0'} 2022-09-12T19:08:45.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee9b77ac8731a43f-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.959 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:45.960 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:45.961 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:45.961 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.962 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:45.962 DEBUG:boto:Method: GET 2022-09-12T19:08:45.963 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:08:45.963 DEBUG:boto:Data: 2022-09-12T19:08:45.964 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.964 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:45.965 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.965 DEBUG:boto:Params: {} 2022-09-12T19:08:45.966 DEBUG:boto:Token: None 2022-09-12T19:08:45.967 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/ 2022-09-12T19:08:45.967 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60= 2022-09-12T19:08:45.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QI9eqIiI8gGnDkJHpOn2A8Yqh60=', 'Content-Length': '0'} 2022-09-12T19:08:45.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007542f70ba9c80c58-00631f83bd-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.972 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:45.972 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.973 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.974 DEBUG:boto:Method: HEAD 2022-09-12T19:08:45.974 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:08:45.975 DEBUG:boto:Data: 2022-09-12T19:08:45.975 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.976 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.977 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.977 DEBUG:boto:Params: {} 2022-09-12T19:08:45.978 DEBUG:boto:Token: None 2022-09-12T19:08:45.979 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj 2022-09-12T19:08:45.979 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vRimNojtI2PNmw+RGFUK4lMtVhQ= 2022-09-12T19:08:45.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vRimNojtI2PNmw+RGFUK4lMtVhQ=', 'Content-Length': '0'} 2022-09-12T19:08:45.982 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000071f8a8a5302eebd7-00631f83bd-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.983 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.984 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.984 DEBUG:boto:Method: HEAD 2022-09-12T19:08:45.985 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:08:45.985 DEBUG:boto:Data: 2022-09-12T19:08:45.986 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.987 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:45.987 DEBUG:boto:Port: 8000 2022-09-12T19:08:45.988 DEBUG:boto:Params: {} 2022-09-12T19:08:45.988 DEBUG:boto:Token: None 2022-09-12T19:08:45.989 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:45 GMT /jdpiak-20/obj 2022-09-12T19:08:45.989 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vRimNojtI2PNmw+RGFUK4lMtVhQ= 2022-09-12T19:08:45.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vRimNojtI2PNmw+RGFUK4lMtVhQ=', 'Content-Length': '0'} 2022-09-12T19:08:45.992 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000483f27d233a34967-00631f83bd-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:45.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:08:45.994 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:45.994 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:45.995 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.995 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.996 DEBUG:boto:Method: GET 2022-09-12T19:08:45.997 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:45.997 DEBUG:boto:Data: 2022-09-12T19:08:45.998 DEBUG:boto:Headers: {} 2022-09-12T19:08:45.998 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:45.999 DEBUG:boto:Port: 8001 2022-09-12T19:08:45.999 DEBUG:boto:Params: {} 2022-09-12T19:08:46.000 DEBUG:boto:Token: None 2022-09-12T19:08:46.001 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:46 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:46.001 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G/4jVr/0Fc1Ut8HyJ3GWET1cbXM= 2022-09-12T19:08:46.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G/4jVr/0Fc1Ut8HyJ3GWET1cbXM=', 'Content-Length': '0'} 2022-09-12T19:08:46.004 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f9341161adf5fe15-00631f83be-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:46.005 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:46.005 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:46.006 DEBUG:boto:path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:46.006 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:46.007 DEBUG:boto:Method: GET 2022-09-12T19:08:46.007 DEBUG:boto:Path: /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:46.008 DEBUG:boto:Data: 2022-09-12T19:08:46.009 DEBUG:boto:Headers: {} 2022-09-12T19:08:46.009 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:46.010 DEBUG:boto:Port: 8000 2022-09-12T19:08:46.010 DEBUG:boto:Params: {} 2022-09-12T19:08:46.011 DEBUG:boto:Token: None 2022-09-12T19:08:46.012 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:46 GMT /jdpiak-20/obj?versionId=Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2. 2022-09-12T19:08:46.012 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G/4jVr/0Fc1Ut8HyJ3GWET1cbXM= 2022-09-12T19:08:46.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G/4jVr/0Fc1Ut8HyJ3GWET1cbXM=', 'Content-Length': '0'} 2022-09-12T19:08:46.015 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:48 GMT'), ('x-amz-version-id', 'Dt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000066e1fb2de0dc956c-00631f83be-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:08:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:46.016 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:46.017 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:46.017 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:46.018 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:46.018 DEBUG:boto:Method: GET 2022-09-12T19:08:46.019 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:08:46.019 DEBUG:boto:Data: 2022-09-12T19:08:46.020 DEBUG:boto:Headers: {} 2022-09-12T19:08:46.021 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:08:46.021 DEBUG:boto:Port: 8001 2022-09-12T19:08:46.022 DEBUG:boto:Params: {} 2022-09-12T19:08:46.022 DEBUG:boto:Token: None 2022-09-12T19:08:46.023 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:46 GMT /jdpiak-20/ 2022-09-12T19:08:46.023 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V7/jUpRasRSflprZHDiFQCbtcf0= 2022-09-12T19:08:46.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V7/jUpRasRSflprZHDiFQCbtcf0=', 'Content-Length': '0'} 2022-09-12T19:08:46.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e3f5ab749bcd247-00631f83be-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:46.070 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:46.071 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:46.072 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:46.072 DEBUG:boto:path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:46.073 DEBUG:boto:auth_path=/jdpiak-20/?prefix=obj 2022-09-12T19:08:46.073 DEBUG:boto:Method: GET 2022-09-12T19:08:46.074 DEBUG:boto:Path: /jdpiak-20/?prefix=obj 2022-09-12T19:08:46.074 DEBUG:boto:Data: 2022-09-12T19:08:46.075 DEBUG:boto:Headers: {} 2022-09-12T19:08:46.075 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:08:46.076 DEBUG:boto:Port: 8000 2022-09-12T19:08:46.076 DEBUG:boto:Params: {} 2022-09-12T19:08:46.077 DEBUG:boto:Token: None 2022-09-12T19:08:46.078 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:08:46 GMT /jdpiak-20/ 2022-09-12T19:08:46.078 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V7/jUpRasRSflprZHDiFQCbtcf0= 2022-09-12T19:08:46.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V7/jUpRasRSflprZHDiFQCbtcf0=', 'Content-Length': '0'} 2022-09-12T19:08:46.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000175c72678f446207-00631f83be-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:08:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:46.083 DEBUG:boto:b'jdpiak-20obj1000falseobj2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:08:46.083 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-20 zones={test-zone3, test-zone2} 2022-09-12T19:08:46.084 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:46.085 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:46.085 DEBUG:boto:path=/jdpiak-20/?versioning 2022-09-12T19:08:46.086 DEBUG:boto:auth_path=/jdpiak-20/?versioning 2022-09-12T19:08:46.086 DEBUG:boto:Method: PUT 2022-09-12T19:08:46.087 DEBUG:boto:Path: /jdpiak-20/?versioning 2022-09-12T19:08:46.087 DEBUG:boto:Data: Suspended Disabled 2022-09-12T19:08:46.087 DEBUG:boto:Headers: {} 2022-09-12T19:08:46.088 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:46.088 DEBUG:boto:Port: 8000 2022-09-12T19:08:46.089 DEBUG:boto:Params: {} 2022-09-12T19:08:46.089 DEBUG:boto:Token: None 2022-09-12T19:08:46.090 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:08:46 GMT /jdpiak-20/?versioning 2022-09-12T19:08:46.091 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jWGBAHMIViOdRzHow9LOv3FPM4s= 2022-09-12T19:08:46.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jWGBAHMIViOdRzHow9LOv3FPM4s=', 'Content-Length': '230'} 2022-09-12T19:08:46.099 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001b19283eb4de3ca8-00631f83be-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:08:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:46.100 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:08:46.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:08:46.102 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:08:46.161 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:08:46.161 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:08:46.206 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=c0f7a512-c940-4d1f-8adf-3484630f876b 2022-09-12T19:08:46.219 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:08:46.219 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:08:46.220 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:08:46.220 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:47:27.976965Z" 2022-09-12T19:08:46.220 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:08:46.220 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:08:46.221 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:08:46.221 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:08:46.221 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:08:46.222 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:08:46.222 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663009726.095686_208.1", 2022-09-12T19:08:46.222 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:08:46.095686Z" 2022-09-12T19:08:46.222 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:08:46.223 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:08:46.223 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:08:46.223 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:49:40.904048Z" 2022-09-12T19:08:46.224 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:08:46.224 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:08:46.225 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009726.095686_208.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:08:46.226 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T19:08:46.226 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:08:46.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:08:46.228 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:08:46.287 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:08:46.287 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:08:46.334 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:08:46.331+0000 7f4c928be600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:08:46.335 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:08:46.333+0000 7f4c928be600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:08:46.348 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:08:46.348 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:08:46.348 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:08:46.349 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:08:46.349 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:08:46.349 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:08:46.349 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:08:46.350 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:08:46.350 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:08:46.350 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:08:46.351 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:08:46.351 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:08:46.351 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:08:46.351 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:08:46.352 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:08:46.352 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:08:46.352 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:08:46.352 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:08:46.353 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:08:46.353 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:08:46.353 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:08:46.354 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:08:46.354 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:08:46.354 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:08:46.354 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:08:46.355 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:08:46.355 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:08:46.355 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:08:46.355 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:08:46.356 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:08:46.356 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:08:46.356 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:08:46.357 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:08:46.357 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:08:46.357 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:08:46.357 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:08:46.358 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:08:46.358 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663009726.095686_208.1", 2022-09-12T19:08:46.358 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:08:46.359 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:08:46.359 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:08:46.359 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:08:46.095686Z", 2022-09-12T19:08:46.360 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:08:46.360 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:08:46.360 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:08:46.360 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:08:46.361 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:08:46.361 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:08:46.361 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:08:46.362 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:08:46.362 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:08:46.362 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:08:46.362 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:08:46.363 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:08:46.363 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:08:46.363 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:08:46.363 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:08:46.364 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:08:46.364 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:08:46.364 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:08:46.365 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:08:46.365 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:08:46.365 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:08:46.365 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:08:46Z" 2022-09-12T19:08:46.366 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:08:46.367 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009726.095686_208.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:08:46.095686Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:08:46Z" } 2022-09-12T19:08:46.367 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009726.095686_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:08:46.095686Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:08:46.368 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009726.095686_208.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:08:46.369 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009726.095686_208.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:08:46.369 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T19:08:46.369 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T19:08:46.370 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:08:46.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:08:46.371 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:08:46.424 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:08:46.424 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:08:46.454 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:08:46.450+0000 7f2cef0af600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:08:46.455 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:08:46.452+0000 7f2cef0af600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:08:46.472 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:08:46.472 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:08:46.473 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:08:46.473 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:08:46.473 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:08:46.473 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:08:46.474 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:08:46.474 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:08:46.474 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:08:46.475 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:08:46.475 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:08:46.475 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:08:46.475 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:08:46.476 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:08:46.476 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:08:46.476 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:08:46.476 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:08:46.477 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:08:46.477 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:08:46.477 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:08:46.477 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:08:46.478 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:08:46.478 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:08:46.478 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:08:46.478 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:08:46.479 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:08:46.479 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:08:46.479 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:08:46.480 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:08:46.480 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:08:46.480 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:08:46.481 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:08:46.481 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:08:46.481 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:08:46.482 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:08:46.482 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:08:46.482 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:08:46.482 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663009726.091934_207.1", 2022-09-12T19:08:46.483 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:08:46.483 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:08:46.483 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:08:46.483 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:08:46.091934Z", 2022-09-12T19:08:46.484 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:08:46.484 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:08:46.484 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:08:46.484 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:08:46.485 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:08:46.485 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:08:46.485 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:08:46.485 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:08:46.486 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:08:46.486 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:08:46.486 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:08:46.487 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:08:46.487 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:08:46.487 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:08:46.487 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:08:46.488 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:08:46.488 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:08:46.488 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:08:46.488 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:08:46.489 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:08:46.489 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:08:46.489 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:08:46Z" 2022-09-12T19:08:46.489 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:08:46.491 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009726.091934_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:08:46.091934Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:08:46Z" } 2022-09-12T19:08:46.491 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009726.091934_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:08:46.091934Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:08:46.492 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009726.095686_208.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:08:46.492 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009726.091934_207.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:08:46.493 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1663009726.095686_208.1 target=1_1663009726.091934_207.1 2022-09-12T19:08:51.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:08:51.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:08:51.495 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:08:51.555 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:08:51.556 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:08:51.587 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:08:51.584+0000 7f7b24119600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:08:51.588 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:08:51.585+0000 7f7b24119600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:08:51.607 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:08:51.608 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:08:51.608 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:08:51.609 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:08:51.609 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:08:51.609 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:08:51.609 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:08:51.610 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:08:51.610 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:08:51.610 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:08:51.610 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:08:51.611 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:08:51.611 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:08:51.611 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:08:51.611 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:08:51.612 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:08:51.612 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:08:51.612 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:08:51.612 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:08:51.613 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:08:51.613 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:08:51.613 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:08:51.613 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:08:51.614 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:08:51.614 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:08:51.614 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:08:51.614 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:08:51.615 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:08:51.615 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:08:51.615 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:08:51.615 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:08:51.616 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:08:51.616 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:08:51.616 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:08:51.616 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:08:51.617 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:08:51.617 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:08:51.617 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663009726.095686_208.1", 2022-09-12T19:08:51.618 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:08:51.618 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:08:51.619 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:08:51.619 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:08:46.095686Z", 2022-09-12T19:08:51.619 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:08:51.619 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:08:51.620 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:08:51.620 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:08:51.620 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:08:51.620 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:08:51.621 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:08:51.621 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:08:51.621 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:08:51.622 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:08:51.622 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:08:51.622 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:08:51.622 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:08:51.623 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:08:51.623 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:08:51.623 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:08:51.623 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:08:51.624 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:08:51.624 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:08:51.624 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:08:51.624 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:08:51.625 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:08:51Z" 2022-09-12T19:08:51.625 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:08:51.626 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009726.095686_208.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:08:46.095686Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:08:51Z" } 2022-09-12T19:08:51.627 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009726.095686_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:08:46.095686Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:08:51.627 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009726.095686_208.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:08:51.628 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009726.095686_208.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:08:51.628 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T19:08:51.629 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T19:08:51.629 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:08:51.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:08:51.631 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:08:51.689 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:08:51.689 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:08:51.728 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:08:51.725+0000 7facf1e1d600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:08:51.729 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:08:51.726+0000 7facf1e1d600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:08:51.748 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:08:51.749 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:08:51.749 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:08:51.750 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:08:51.750 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:08:51.750 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:08:51.751 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:08:51.751 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:08:51.751 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:08:51.751 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:08:51.752 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:08:51.752 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:08:51.752 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:08:51.752 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:08:51.753 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:08:51.753 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:08:51.753 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:08:51.753 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:08:51.754 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:08:51.754 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:08:51.754 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:08:51.755 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:08:51.755 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:08:51.755 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:08:51.755 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:08:51.756 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:08:51.756 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:08:51.756 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:08:51.756 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:08:51.757 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:08:51.757 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:08:51.757 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:08:51.757 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:08:51.758 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:08:51.758 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:08:51.758 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:08:51.759 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:08:51.759 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663009726.095686_208.1", 2022-09-12T19:08:51.759 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:08:51.759 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:08:51.760 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:08:51.760 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:08:46.095686Z", 2022-09-12T19:08:51.760 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:08:51.760 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:08:51.761 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:08:51.761 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:08:51.761 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:08:51.761 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:08:51.762 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:08:51.762 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:08:51.762 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:08:51.763 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:08:51.763 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:08:51.763 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:08:51.763 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:08:51.764 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:08:51.764 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:08:51.764 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:08:51.764 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:08:51.765 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:08:51.765 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:08:51.765 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:08:51.765 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:08:51.766 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:08:51Z" 2022-09-12T19:08:51.766 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:08:51.767 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009726.095686_208.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:08:46.095686Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:08:51Z" } 2022-09-12T19:08:51.768 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009726.095686_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:08:46.095686Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:08:51.768 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009726.095686_208.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:08:51.769 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009726.095686_208.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:08:51.769 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T19:08:51.770 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:08:51.770 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:08:51.771 DEBUG:boto:Method: HEAD 2022-09-12T19:08:51.771 DEBUG:boto:Path: /jdpiak-20/ 2022-09-12T19:08:51.772 DEBUG:boto:Data: 2022-09-12T19:08:51.772 DEBUG:boto:Headers: {} 2022-09-12T19:08:51.773 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:51.773 DEBUG:boto:Port: 8000 2022-09-12T19:08:51.774 DEBUG:boto:Params: {} 2022-09-12T19:08:51.775 DEBUG:boto:Token: None 2022-09-12T19:08:51.775 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:08:51 GMT /jdpiak-20/ 2022-09-12T19:08:51.776 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hyIo61CazZO62cHzRfqTqejrqNE= 2022-09-12T19:08:51.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:08:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hyIo61CazZO62cHzRfqTqejrqNE=', 'Content-Length': '0'} 2022-09-12T19:08:51.779 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7601828ce271233-00631f83c3-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:08:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:51.780 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:08:51.780 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:08:51.781 DEBUG:boto:Method: PUT 2022-09-12T19:08:51.781 DEBUG:boto:Path: /jdpiak-20/obj 2022-09-12T19:08:51.782 DEBUG:boto:Data: 2022-09-12T19:08:51.782 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-12T19:08:51.783 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:08:51.783 DEBUG:boto:Port: 8000 2022-09-12T19:08:51.784 DEBUG:boto:Params: {} 2022-09-12T19:08:51.784 DEBUG:boto:Token: None 2022-09-12T19:08:51.785 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 12 Sep 2022 19:08:51 GMT /jdpiak-20/obj 2022-09-12T19:08:51.786 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7BVipJDQ2+vvnS7E2L+VB9Gyx+I= 2022-09-12T19:08:51.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:08:51 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7BVipJDQ2+vvnS7E2L+VB9Gyx+I='} 2022-09-12T19:08:51.830 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001353df7c819ba4f6-00631f83c3-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:08:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:08:51.831 DEBUG:rgw_multi.tests:created null version id=None 2022-09-12T19:08:51.832 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-20 2022-09-12T19:08:51.833 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:08:51.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:08:51.835 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:08:51.892 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:08:51.893 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:08:51.947 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:08:51.944+0000 7f82dcd10600 1 waiting for incremental sync to catch up: 2022-09-12T19:08:51.947 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000003.29.12, , ] 2022-09-12T19:08:51.948 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000005.33.17, , ] 2022-09-12T19:08:56.948 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:08:56.945+0000 7f82dcd10600 1 waiting for incremental sync to catch up: 2022-09-12T19:08:56.949 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000003.29.12, , ] 2022-09-12T19:08:56.950 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000005.33.17, , ] 2022-09-12T19:09:01.950 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:09:01.948+0000 7f82dcd10600 1 waiting for incremental sync to catch up: 2022-09-12T19:09:01.950 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000003.29.12, , ] 2022-09-12T19:09:01.950 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000005.33.17, , ] 2022-09-12T19:09:06.951 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:09:06.949+0000 7f82dcd10600 1 waiting for incremental sync to catch up: 2022-09-12T19:09:06.952 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000003.29.12, , ] 2022-09-12T19:09:06.952 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000005.33.17, , ] 2022-09-12T19:09:11.953 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:09:11.950+0000 7f82dcd10600 1 waiting for incremental sync to catch up: 2022-09-12T19:09:11.953 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000003.29.12, , ] 2022-09-12T19:09:11.954 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000005.33.17, , ] 2022-09-12T19:09:16.955 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:09:16.951+0000 7f82dcd10600 1 waiting for incremental sync to catch up: 2022-09-12T19:09:16.955 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000003.29.12, , ] 2022-09-12T19:09:16.955 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000005.33.17, , ] 2022-09-12T19:09:21.956 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:09:21.953+0000 7f82dcd10600 1 waiting for incremental sync to catch up: 2022-09-12T19:09:21.956 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000003.29.12, , ] 2022-09-12T19:09:21.957 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000005.33.17, , ] 2022-09-12T19:09:26.957 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:09:26.954+0000 7f82dcd10600 1 waiting for incremental sync to catch up: 2022-09-12T19:09:26.958 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000003.29.12, , ] 2022-09-12T19:09:26.958 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000005.33.17, , ] 2022-09-12T19:09:31.959 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:09:31.956+0000 7f82dcd10600 1 waiting for incremental sync to catch up: 2022-09-12T19:09:31.959 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000003.29.12, , ] 2022-09-12T19:09:31.960 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000005.33.17, , ] 2022-09-12T19:09:36.960 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:09:36.957+0000 7f82dcd10600 1 waiting for incremental sync to catch up: 2022-09-12T19:09:36.961 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000003.29.12, , ] 2022-09-12T19:09:36.961 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000005.33.17, , ] 2022-09-12T19:09:41.962 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:09:41.959+0000 7f82dcd10600 1 waiting for incremental sync to catch up: 2022-09-12T19:09:41.963 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000003.29.12, , ] 2022-09-12T19:09:41.963 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000005.33.17, , ] 2022-09-12T19:09:46.963 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:09:46.961+0000 7f82dcd10600 1 bucket sync caught up with source: 2022-09-12T19:09:46.964 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000005.33.17, , ] 2022-09-12T19:09:46.964 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000005.33.17, , ] 2022-09-12T19:09:46.965 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:09:46.961+0000 7f82dcd10600 0 bucket checkpoint complete 2022-09-12T19:09:46.984 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-20 2022-09-12T19:09:46.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:09:46.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:09:46.986 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:09:47.053 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:09:47.054 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:09:47.091 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:09:47.088+0000 7f9e98fb2600 1 bucket sync caught up with source: 2022-09-12T19:09:47.092 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000005.33.17, , ] 2022-09-12T19:09:47.092 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000005.33.17, , ] 2022-09-12T19:09:47.092 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:09:47.088+0000 7f9e98fb2600 0 bucket checkpoint complete 2022-09-12T19:09:47.105 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-20 2022-09-12T19:09:47.105 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:09:47.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:09:47.107 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:09:47.169 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:09:47.170 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:09:47.210 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:09:47.208+0000 7f69dda72600 1 bucket sync caught up with source: 2022-09-12T19:09:47.211 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000005.33.17, , ] 2022-09-12T19:09:47.212 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000005.33.17, , ] 2022-09-12T19:09:47.212 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:09:47.208+0000 7f69dda72600 0 bucket checkpoint complete 2022-09-12T19:09:47.222 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-20 2022-09-12T19:09:47.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:09:47.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:09:47.224 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:09:47.283 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:09:47.284 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:09:47.330 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:09:47.327+0000 7f1a5ef1c600 1 waiting for incremental sync to catch up: 2022-09-12T19:09:47.330 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000005.48.17, , ] 2022-09-12T19:09:47.330 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000007.53.14, , ] 2022-09-12T19:09:52.331 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:09:52.328+0000 7f1a5ef1c600 1 waiting for incremental sync to catch up: 2022-09-12T19:09:52.332 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000005.48.17, , ] 2022-09-12T19:09:52.332 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000007.53.14, , ] 2022-09-12T19:09:57.333 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:09:57.330+0000 7f1a5ef1c600 1 waiting for incremental sync to catch up: 2022-09-12T19:09:57.334 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000005.48.17, , ] 2022-09-12T19:09:57.334 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000007.53.14, , ] 2022-09-12T19:10:02.335 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:10:02.331+0000 7f1a5ef1c600 1 waiting for incremental sync to catch up: 2022-09-12T19:10:02.335 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000005.48.17, , ] 2022-09-12T19:10:02.335 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000007.53.14, , ] 2022-09-12T19:10:07.336 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:10:07.333+0000 7f1a5ef1c600 1 waiting for incremental sync to catch up: 2022-09-12T19:10:07.336 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000005.48.17, , ] 2022-09-12T19:10:07.337 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000007.53.14, , ] 2022-09-12T19:10:12.337 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:10:12.335+0000 7f1a5ef1c600 1 waiting for incremental sync to catch up: 2022-09-12T19:10:12.338 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000005.48.17, , ] 2022-09-12T19:10:12.338 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000007.53.14, , ] 2022-09-12T19:10:17.339 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:10:17.336+0000 7f1a5ef1c600 1 waiting for incremental sync to catch up: 2022-09-12T19:10:17.340 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000005.48.17, , ] 2022-09-12T19:10:17.340 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000007.53.14, , ] 2022-09-12T19:10:22.340 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:10:22.337+0000 7f1a5ef1c600 1 bucket sync caught up with source: 2022-09-12T19:10:22.340 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000007.53.14, , ] 2022-09-12T19:10:22.341 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000007.53.14, , ] 2022-09-12T19:10:22.341 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:10:22.337+0000 7f1a5ef1c600 0 bucket checkpoint complete 2022-09-12T19:10:22.365 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-20 2022-09-12T19:10:22.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:10:22.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:10:22.368 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:10:22.432 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:10:22.433 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:10:22.481 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:10:22.478+0000 7f8e7f422600 1 bucket sync caught up with source: 2022-09-12T19:10:22.482 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000007.53.14, , ] 2022-09-12T19:10:22.482 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000007.53.14, , ] 2022-09-12T19:10:22.482 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:10:22.478+0000 7f8e7f422600 0 bucket checkpoint complete 2022-09-12T19:10:22.497 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-20 2022-09-12T19:10:22.498 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-20 2022-09-12T19:10:22.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:10:22.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:10:22.501 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:10:22.559 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:10:22.559 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:10:22.593 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:10:22.590+0000 7fdbd2a27600 1 waiting for incremental sync to catch up: 2022-09-12T19:10:22.593 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000006.60.17, , ] 2022-09-12T19:10:22.593 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000007.62.14, , ] 2022-09-12T19:10:27.594 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:10:27.591+0000 7fdbd2a27600 1 waiting for incremental sync to catch up: 2022-09-12T19:10:27.595 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000006.60.17, , ] 2022-09-12T19:10:27.595 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000007.62.14, , ] 2022-09-12T19:10:32.596 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:10:32.593+0000 7fdbd2a27600 1 waiting for incremental sync to catch up: 2022-09-12T19:10:32.596 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000006.60.17, , ] 2022-09-12T19:10:32.597 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000007.62.14, , ] 2022-09-12T19:10:37.598 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:10:37.594+0000 7fdbd2a27600 1 waiting for incremental sync to catch up: 2022-09-12T19:10:37.598 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000006.60.17, , ] 2022-09-12T19:10:37.599 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000007.62.14, , ] 2022-09-12T19:10:42.599 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:10:42.596+0000 7fdbd2a27600 1 bucket sync caught up with source: 2022-09-12T19:10:42.599 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000007.62.14, , ] 2022-09-12T19:10:42.600 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000007.62.14, , ] 2022-09-12T19:10:42.600 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:10:42.596+0000 7fdbd2a27600 0 bucket checkpoint complete 2022-09-12T19:10:42.619 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-20 2022-09-12T19:10:42.620 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:10:42.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:10:42.622 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:10:42.687 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:10:42.687 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:10:42.732 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:10:42.729+0000 7f9e2f51e600 1 bucket sync caught up with source: 2022-09-12T19:10:42.732 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000007.62.14, , ] 2022-09-12T19:10:42.732 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000007.62.14, , ] 2022-09-12T19:10:42.733 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:10:42.729+0000 7f9e2f51e600 0 bucket checkpoint complete 2022-09-12T19:10:42.746 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-20 2022-09-12T19:10:42.747 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-20 zones={test-zone2, test-zone1} 2022-09-12T19:10:42.747 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:10:42.748 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:10:42.749 DEBUG:boto:Method: HEAD 2022-09-12T19:10:42.749 DEBUG:boto:Path: /jdpiak-20/ 2022-09-12T19:10:42.750 DEBUG:boto:Data: 2022-09-12T19:10:42.750 DEBUG:boto:Headers: {} 2022-09-12T19:10:42.751 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:10:42.751 DEBUG:boto:Port: 8000 2022-09-12T19:10:42.752 DEBUG:boto:Params: {} 2022-09-12T19:10:42.753 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:10:42.754 DEBUG:boto:Token: None 2022-09-12T19:10:42.755 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:10:42 GMT /jdpiak-20/ 2022-09-12T19:10:42.755 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W97Mr0KQcArLwchQGQZqgOFZnjg= 2022-09-12T19:10:42.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:10:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W97Mr0KQcArLwchQGQZqgOFZnjg=', 'Content-Length': '0'} 2022-09-12T19:10:42.760 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae5ed1486c835881-00631f8432-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:10:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:42.761 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:10:42.762 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:10:42.762 DEBUG:boto:Method: HEAD 2022-09-12T19:10:42.763 DEBUG:boto:Path: /jdpiak-20/ 2022-09-12T19:10:42.763 DEBUG:boto:Data: 2022-09-12T19:10:42.764 DEBUG:boto:Headers: {} 2022-09-12T19:10:42.764 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:10:42.765 DEBUG:boto:Port: 8000 2022-09-12T19:10:42.766 DEBUG:boto:Params: {} 2022-09-12T19:10:42.766 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:10:42.767 DEBUG:boto:Token: None 2022-09-12T19:10:42.767 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:10:42 GMT /jdpiak-20/ 2022-09-12T19:10:42.768 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W97Mr0KQcArLwchQGQZqgOFZnjg= 2022-09-12T19:10:42.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:10:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W97Mr0KQcArLwchQGQZqgOFZnjg=', 'Content-Length': '0'} 2022-09-12T19:10:42.772 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b327e65bb46c6f8d-00631f8432-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:10:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:42.773 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:10:42.774 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:10:42.775 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:10:42.776 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:10:42.777 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:10:42.778 DEBUG:boto:Method: GET 2022-09-12T19:10:42.779 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:10:42.780 DEBUG:boto:Data: 2022-09-12T19:10:42.781 DEBUG:boto:Headers: {} 2022-09-12T19:10:42.782 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:10:42.783 DEBUG:boto:Port: 8000 2022-09-12T19:10:42.784 DEBUG:boto:Params: {} 2022-09-12T19:10:42.785 DEBUG:boto:Token: None 2022-09-12T19:10:42.786 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:10:42 GMT /jdpiak-20/?versions 2022-09-12T19:10:42.787 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nTl34tC3l8NqM6VYg0BFIA1a7U8= 2022-09-12T19:10:42.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:10:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nTl34tC3l8NqM6VYg0BFIA1a7U8=', 'Content-Length': '0'} 2022-09-12T19:10:42.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edff46246cb2ba4f-00631f8432-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:10:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:42.792 DEBUG:boto:b'jdpiak-20999falseobjnulltrue2022-09-12T19:08:51.787Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.false2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:10:42.794 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:10:42.795 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:10:42.796 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:10:42.797 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:10:42.798 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:10:42.799 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:10:42.800 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:10:42.801 DEBUG:boto:Method: GET 2022-09-12T19:10:42.802 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:10:42.803 DEBUG:boto:Data: 2022-09-12T19:10:42.803 DEBUG:boto:Headers: {} 2022-09-12T19:10:42.804 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:10:42.805 DEBUG:boto:Port: 8000 2022-09-12T19:10:42.806 DEBUG:boto:Params: {} 2022-09-12T19:10:42.807 DEBUG:boto:Token: None 2022-09-12T19:10:42.808 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:10:42 GMT /jdpiak-20/?versions 2022-09-12T19:10:42.809 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nTl34tC3l8NqM6VYg0BFIA1a7U8= 2022-09-12T19:10:42.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:10:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nTl34tC3l8NqM6VYg0BFIA1a7U8=', 'Content-Length': '0'} 2022-09-12T19:10:42.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e46da35a22526ba-00631f8432-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:10:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:42.815 DEBUG:boto:b'jdpiak-20999falseobjnulltrue2022-09-12T19:08:51.787Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.false2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:10:42.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:10:42.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:10:42.818 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:10:42.819 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:10:42.820 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:10:42.821 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:10:42.822 DEBUG:boto:Method: GET 2022-09-12T19:10:42.823 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:10:42.823 DEBUG:boto:Data: 2022-09-12T19:10:42.824 DEBUG:boto:Headers: {} 2022-09-12T19:10:42.825 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:10:42.826 DEBUG:boto:Port: 8000 2022-09-12T19:10:42.827 DEBUG:boto:Params: {} 2022-09-12T19:10:42.829 DEBUG:boto:Token: None 2022-09-12T19:10:42.830 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:10:42 GMT /jdpiak-20/?versions 2022-09-12T19:10:42.831 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nTl34tC3l8NqM6VYg0BFIA1a7U8= 2022-09-12T19:10:42.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:10:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nTl34tC3l8NqM6VYg0BFIA1a7U8=', 'Content-Length': '0'} 2022-09-12T19:10:42.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056af4fab44732996-00631f8432-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:10:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:42.836 DEBUG:boto:b'jdpiak-20999falseobjnulltrue2022-09-12T19:08:51.787Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.false2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:10:42.838 DEBUG:boto:path=/jdpiak-20/ 2022-09-12T19:10:42.839 DEBUG:boto:auth_path=/jdpiak-20/ 2022-09-12T19:10:42.840 DEBUG:boto:path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:10:42.841 DEBUG:boto:auth_path=/jdpiak-20/?versions&max-keys=999 2022-09-12T19:10:42.842 DEBUG:boto:Method: GET 2022-09-12T19:10:42.843 DEBUG:boto:Path: /jdpiak-20/?versions&max-keys=999 2022-09-12T19:10:42.843 DEBUG:boto:Data: 2022-09-12T19:10:42.844 DEBUG:boto:Headers: {} 2022-09-12T19:10:42.845 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:10:42.846 DEBUG:boto:Port: 8000 2022-09-12T19:10:42.847 DEBUG:boto:Params: {} 2022-09-12T19:10:42.848 DEBUG:boto:Token: None 2022-09-12T19:10:42.849 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:10:42 GMT /jdpiak-20/?versions 2022-09-12T19:10:42.850 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nTl34tC3l8NqM6VYg0BFIA1a7U8= 2022-09-12T19:10:42.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:10:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nTl34tC3l8NqM6VYg0BFIA1a7U8=', 'Content-Length': '0'} 2022-09-12T19:10:42.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000addc1887e22f6ed6-00631f8432-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:10:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:42.855 DEBUG:boto:b'jdpiak-20999falseobjnulltrue2022-09-12T19:08:51.787Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDt-Ae1brnJ3RAmI9vVo8.e2rC0nzK2.false2022-09-12T19:07:48.572Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:10:42.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:10:42.858 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:10:42.859 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:10:42.859 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:10:42.860 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:10:42.861 DEBUG:boto:Method: GET 2022-09-12T19:10:42.862 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:10:42.863 DEBUG:boto:Data: 2022-09-12T19:10:42.864 DEBUG:boto:Headers: {} 2022-09-12T19:10:42.865 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:10:42.866 DEBUG:boto:Port: 8000 2022-09-12T19:10:42.867 DEBUG:boto:Params: {} 2022-09-12T19:10:42.868 DEBUG:boto:Token: None 2022-09-12T19:10:42.868 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:10:42 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:10:42.869 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yRNKhYwNsKBVtkc1/t8yf2kHeBE= 2022-09-12T19:10:42.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:10:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yRNKhYwNsKBVtkc1/t8yf2kHeBE=', 'Content-Length': '0'} 2022-09-12T19:10:42.872 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a5f72b5374a0f810-00631f8432-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:10:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:42.872 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:10:42.873 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:10:42.873 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:10:42.874 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:10:42.875 DEBUG:boto:Method: GET 2022-09-12T19:10:42.875 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:10:42.876 DEBUG:boto:Data: 2022-09-12T19:10:42.876 DEBUG:boto:Headers: {} 2022-09-12T19:10:42.877 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:10:42.877 DEBUG:boto:Port: 8000 2022-09-12T19:10:42.878 DEBUG:boto:Params: {} 2022-09-12T19:10:42.878 DEBUG:boto:Token: None 2022-09-12T19:10:42.879 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:10:42 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:10:42.880 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yRNKhYwNsKBVtkc1/t8yf2kHeBE= 2022-09-12T19:10:42.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:10:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yRNKhYwNsKBVtkc1/t8yf2kHeBE=', 'Content-Length': '0'} 2022-09-12T19:10:42.882 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:08:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d72a615c6215e76e-00631f8432-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:10:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:42.883 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:10:42.884 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:10:42.884 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:10:42.885 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:10:42.885 DEBUG:boto:Method: HEAD 2022-09-12T19:10:42.886 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:10:42.886 DEBUG:boto:Data: 2022-09-12T19:10:42.887 DEBUG:boto:Headers: {} 2022-09-12T19:10:42.887 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:10:42.888 DEBUG:boto:Port: 8000 2022-09-12T19:10:42.888 DEBUG:boto:Params: {} 2022-09-12T19:10:42.889 DEBUG:boto:Token: None 2022-09-12T19:10:42.889 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:10:42 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:10:42.890 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Xmu6c+mKbI9bfN34E9sviWk6MXU= 2022-09-12T19:10:42.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:10:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Xmu6c+mKbI9bfN34E9sviWk6MXU=', 'Content-Length': '0'} 2022-09-12T19:10:42.892 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:07:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000965bf8b506dce917-00631f8432-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:10:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:42.893 DEBUG:boto:path=/jdpiak-20/obj 2022-09-12T19:10:42.893 DEBUG:boto:auth_path=/jdpiak-20/obj 2022-09-12T19:10:42.894 DEBUG:boto:path=/jdpiak-20/obj?versionId=null 2022-09-12T19:10:42.894 DEBUG:boto:auth_path=/jdpiak-20/obj?versionId=null 2022-09-12T19:10:42.895 DEBUG:boto:Method: HEAD 2022-09-12T19:10:42.895 DEBUG:boto:Path: /jdpiak-20/obj?versionId=null 2022-09-12T19:10:42.896 DEBUG:boto:Data: 2022-09-12T19:10:42.896 DEBUG:boto:Headers: {} 2022-09-12T19:10:42.897 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:10:42.897 DEBUG:boto:Port: 8000 2022-09-12T19:10:42.898 DEBUG:boto:Params: {} 2022-09-12T19:10:42.898 DEBUG:boto:Token: None 2022-09-12T19:10:42.899 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:10:42 GMT /jdpiak-20/obj?versionId=null 2022-09-12T19:10:42.899 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Xmu6c+mKbI9bfN34E9sviWk6MXU= 2022-09-12T19:10:42.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:10:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Xmu6c+mKbI9bfN34E9sviWk6MXU=', 'Content-Length': '0'} 2022-09-12T19:10:42.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:08:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b0659d1c7cc7c1dd-00631f8432-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:10:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:42.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:10:42.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... FAIL 2022-09-12T19:10:42.909 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-21 2022-09-12T19:10:42.909 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:10:42.910 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:10:42.911 DEBUG:boto:Method: PUT 2022-09-12T19:10:42.911 DEBUG:boto:Path: /jdpiak-21/ 2022-09-12T19:10:42.912 DEBUG:boto:Data: 2022-09-12T19:10:42.912 DEBUG:boto:Headers: {} 2022-09-12T19:10:42.913 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:10:42.913 DEBUG:boto:Port: 8000 2022-09-12T19:10:42.914 DEBUG:boto:Params: {} 2022-09-12T19:10:42.914 DEBUG:boto:Token: None 2022-09-12T19:10:42.915 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:10:42 GMT /jdpiak-21/ 2022-09-12T19:10:42.915 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vN5g0/6IhCjrqthRNI4X+BclDWE= 2022-09-12T19:10:42.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:10:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vN5g0/6IhCjrqthRNI4X+BclDWE=', 'Content-Length': '0'} 2022-09-12T19:10:42.942 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000021581c6d3a04a401-00631f8432-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:10:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:42.943 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jdpiak-22 2022-09-12T19:10:42.944 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:10:42.944 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:10:42.945 DEBUG:boto:Method: PUT 2022-09-12T19:10:42.945 DEBUG:boto:Path: /jdpiak-22/ 2022-09-12T19:10:42.946 DEBUG:boto:Data: 2022-09-12T19:10:42.946 DEBUG:boto:Headers: {} 2022-09-12T19:10:42.947 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:10:42.947 DEBUG:boto:Port: 8000 2022-09-12T19:10:42.948 DEBUG:boto:Params: {} 2022-09-12T19:10:42.948 DEBUG:boto:Token: None 2022-09-12T19:10:42.949 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:10:42 GMT /jdpiak-22/ 2022-09-12T19:10:42.949 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BFtWDGr6gDU0pNa38W1PVOY89iA= 2022-09-12T19:10:42.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:10:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BFtWDGr6gDU0pNa38W1PVOY89iA=', 'Content-Length': '0'} 2022-09-12T19:10:42.985 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c08ec82e39b8bd4c-00631f8432-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:10:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:42.986 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jdpiak-23 2022-09-12T19:10:42.986 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:10:42.987 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:10:42.987 DEBUG:boto:Method: PUT 2022-09-12T19:10:42.988 DEBUG:boto:Path: /jdpiak-23/ 2022-09-12T19:10:42.989 DEBUG:boto:Data: 2022-09-12T19:10:42.989 DEBUG:boto:Headers: {} 2022-09-12T19:10:42.990 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:10:42.991 DEBUG:boto:Port: 8001 2022-09-12T19:10:42.991 DEBUG:boto:Params: {} 2022-09-12T19:10:42.992 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T19:10:42.992 DEBUG:boto:Token: None 2022-09-12T19:10:42.993 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:10:42 GMT /jdpiak-23/ 2022-09-12T19:10:42.993 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3D0yFQJLMkq9m4kgtPn0c1VD+Ns= 2022-09-12T19:10:42.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:10:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3D0yFQJLMkq9m4kgtPn0c1VD+Ns=', 'Content-Length': '0'} 2022-09-12T19:10:43.031 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c254767e6510123d-00631f8432-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:10:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:43.032 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:10:43.033 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:10:43.033 DEBUG:boto:path=/jdpiak-21/?versioning 2022-09-12T19:10:43.034 DEBUG:boto:auth_path=/jdpiak-21/?versioning 2022-09-12T19:10:43.034 DEBUG:boto:Method: PUT 2022-09-12T19:10:43.035 DEBUG:boto:Path: /jdpiak-21/?versioning 2022-09-12T19:10:43.035 DEBUG:boto:Data: Enabled Disabled 2022-09-12T19:10:43.036 DEBUG:boto:Headers: {} 2022-09-12T19:10:43.037 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:10:43.038 DEBUG:boto:Port: 8000 2022-09-12T19:10:43.038 DEBUG:boto:Params: {} 2022-09-12T19:10:43.039 DEBUG:boto:Token: None 2022-09-12T19:10:43.039 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:10:43 GMT /jdpiak-21/?versioning 2022-09-12T19:10:43.040 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pobHXHxmctiK8I4HmUzc+FEB/zk= 2022-09-12T19:10:43.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:10:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pobHXHxmctiK8I4HmUzc+FEB/zk=', 'Content-Length': '228'} 2022-09-12T19:10:43.049 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005f03a5fcd4ffda30-00631f8433-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:10:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:43.050 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:10:43.050 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:10:43.051 DEBUG:boto:path=/jdpiak-22/?versioning 2022-09-12T19:10:43.051 DEBUG:boto:auth_path=/jdpiak-22/?versioning 2022-09-12T19:10:43.051 DEBUG:boto:Method: PUT 2022-09-12T19:10:43.052 DEBUG:boto:Path: /jdpiak-22/?versioning 2022-09-12T19:10:43.052 DEBUG:boto:Data: Enabled Disabled 2022-09-12T19:10:43.053 DEBUG:boto:Headers: {} 2022-09-12T19:10:43.053 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:10:43.054 DEBUG:boto:Port: 8000 2022-09-12T19:10:43.054 DEBUG:boto:Params: {} 2022-09-12T19:10:43.055 DEBUG:boto:Token: None 2022-09-12T19:10:43.055 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:10:43 GMT /jdpiak-22/?versioning 2022-09-12T19:10:43.056 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iQLZXf3us0PeofSD+qwsaaHeiwA= 2022-09-12T19:10:43.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:10:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iQLZXf3us0PeofSD+qwsaaHeiwA=', 'Content-Length': '228'} 2022-09-12T19:10:43.068 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002621f29664f38bc4-00631f8433-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:10:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:43.069 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:10:43.070 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:10:43.070 DEBUG:boto:path=/jdpiak-23/?versioning 2022-09-12T19:10:43.071 DEBUG:boto:auth_path=/jdpiak-23/?versioning 2022-09-12T19:10:43.071 DEBUG:boto:Method: PUT 2022-09-12T19:10:43.072 DEBUG:boto:Path: /jdpiak-23/?versioning 2022-09-12T19:10:43.072 DEBUG:boto:Data: Enabled Disabled 2022-09-12T19:10:43.072 DEBUG:boto:Headers: {} 2022-09-12T19:10:43.073 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:10:43.073 DEBUG:boto:Port: 8001 2022-09-12T19:10:43.074 DEBUG:boto:Params: {} 2022-09-12T19:10:43.074 DEBUG:boto:Token: None 2022-09-12T19:10:43.075 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:10:43 GMT /jdpiak-23/?versioning 2022-09-12T19:10:43.076 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hhGZxsHjmxqBTKL37qQHquqfyIQ= 2022-09-12T19:10:43.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:10:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hhGZxsHjmxqBTKL37qQHquqfyIQ=', 'Content-Length': '228'} 2022-09-12T19:10:43.084 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000cd3552d59ad3675-00631f8433-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:10:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:43.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:10:43.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:10:43.086 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:10:43.140 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:10:43.141 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:10:43.176 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=c0f7a512-c940-4d1f-8adf-3484630f876b 2022-09-12T19:10:43.192 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:10:43.192 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:10:43.193 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:10:43.193 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:47:27.976965Z" 2022-09-12T19:10:43.193 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:10:43.194 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:10:43.194 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:10:43.195 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:10:43.195 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:10:43.196 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:10:43.196 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663009843.080040_228.1", 2022-09-12T19:10:43.196 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:10:43.080040Z" 2022-09-12T19:10:43.197 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:10:43.197 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:10:43.198 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:10:43.198 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:49:40.904048Z" 2022-09-12T19:10:43.198 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:10:43.199 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:10:43.201 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009843.080040_228.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:10:43.202 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T19:10:43.203 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:10:43.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:10:43.205 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:10:43.264 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:10:43.265 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:10:43.318 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:10:43.315+0000 7f92ead55600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:10:43.319 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:10:43.316+0000 7f92ead55600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:10:43.331 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:10:43.332 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:10:43.332 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:10:43.333 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:10:43.333 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:10:43.334 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:10:43.334 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:10:43.335 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:10:43.335 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:10:43.336 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:10:43.336 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:10:43.337 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:10:43.337 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:10:43.338 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:10:43.338 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:10:43.339 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:10:43.339 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:10:43.340 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:10:43.340 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:10:43.341 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:10:43.341 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:10:43.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:10:43.342 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:10:43.342 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:10:43.343 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:10:43.343 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:10:43.344 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:10:43.344 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:10:43.345 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:10:43.345 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:10:43.346 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:10:43.346 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:10:43.347 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:10:43.347 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:10:43.347 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:10:43.348 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:10:43.348 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:10:43.349 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663009843.006494_219.1", 2022-09-12T19:10:43.349 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:10:43.350 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:10:43.350 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:10:43.351 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:43.006494Z", 2022-09-12T19:10:43.351 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:10:43.351 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:10:43.352 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:10:43.352 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:10:43.353 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:10:43.353 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:10:43.353 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:10:43.354 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:10:43.354 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:10:43.355 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:10:43.355 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:10:43.356 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:10:43.356 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:10:43.357 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:10:43.357 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:10:43.357 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:10:43.358 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:10:43.358 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:10:43.359 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:10:43.359 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:10:43.360 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:10:43.360 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:10:43Z" 2022-09-12T19:10:43.361 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:10:43.363 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009843.006494_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:10:43.006494Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:10:43Z" } 2022-09-12T19:10:43.364 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009843.006494_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:10:43.006494Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:10:43.364 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009843.080040_228.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:10:43.365 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009843.006494_219.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:10:43.366 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663009843.080040_228.1 target=1_1663009843.006494_219.1 2022-09-12T19:10:48.368 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:10:48.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:10:48.369 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:10:48.427 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:10:48.428 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:10:48.462 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:10:48.459+0000 7f2db9edb600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:10:48.463 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:10:48.460+0000 7f2db9edb600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:10:48.476 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:10:48.476 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:10:48.476 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:10:48.477 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:10:48.477 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:10:48.477 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:10:48.478 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:10:48.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:10:48.478 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:10:48.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:10:48.479 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:10:48.479 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:10:48.479 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:10:48.479 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:10:48.480 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:10:48.480 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:10:48.480 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:10:48.480 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:10:48.481 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:10:48.481 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:10:48.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:10:48.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:10:48.482 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:10:48.482 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:10:48.483 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:10:48.483 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:10:48.483 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:10:48.483 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:10:48.484 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:10:48.484 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:10:48.484 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:10:48.484 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:10:48.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:10:48.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:10:48.485 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:10:48.485 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:10:48.486 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:10:48.486 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663009842.930693_212.1", 2022-09-12T19:10:48.487 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:10:48.487 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:10:48.487 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:10:48.488 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:42.930693Z", 2022-09-12T19:10:48.488 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:10:48.488 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:10:48.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:10:48.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:10:48.489 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:10:48.489 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:10:48.490 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:10:48.490 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:10:48.490 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:10:48.491 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:10:48.491 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:10:48.491 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:10:48.492 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:10:48.492 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:10:48.492 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:10:48.492 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:10:48.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:10:48.493 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:10:48.493 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:10:48.494 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:10:48.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:10:48.494 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:10:48Z" 2022-09-12T19:10:48.495 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:10:48.496 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009842.930693_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:10:42.930693Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:10:48Z" } 2022-09-12T19:10:48.498 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009842.930693_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:10:42.930693Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:10:48.499 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009843.080040_228.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:10:48.500 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009842.930693_212.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:10:48.500 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663009843.080040_228.1 target=1_1663009842.930693_212.1 2022-09-12T19:10:53.502 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:10:53.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:10:53.503 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:10:53.560 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:10:53.561 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:10:53.604 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:10:53.601+0000 7fd21b4fd600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:10:53.605 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:10:53.602+0000 7fd21b4fd600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:10:53.625 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:10:53.625 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:10:53.625 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:10:53.626 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:10:53.626 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:10:53.626 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:10:53.627 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:10:53.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:10:53.627 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:10:53.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:10:53.628 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:10:53.628 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:10:53.628 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:10:53.629 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:10:53.629 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:10:53.629 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:10:53.630 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:10:53.630 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:10:53.630 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:10:53.630 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:10:53.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:10:53.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:10:53.631 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:10:53.631 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:10:53.632 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:10:53.632 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:10:53.632 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:10:53.632 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:10:53.633 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:10:53.633 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:10:53.633 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:10:53.634 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:10:53.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:10:53.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:10:53.634 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:10:53.635 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:10:53.635 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:10:53.635 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663009843.080040_228.1", 2022-09-12T19:10:53.635 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:10:53.636 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:10:53.636 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:10:53.636 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:43.080040Z", 2022-09-12T19:10:53.636 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:10:53.637 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:10:53.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:10:53.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:10:53.638 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:10:53.638 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:10:53.638 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:10:53.638 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:10:53.639 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:10:53.639 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:10:53.640 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:10:53.640 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:10:53.641 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:10:53.641 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:10:53.642 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:10:53.642 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:10:53.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:10:53.643 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:10:53.643 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:10:53.643 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:10:53.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:10:53.644 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:10:53Z" 2022-09-12T19:10:53.644 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:10:53.645 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009843.080040_228.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:10:43.080040Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:10:53Z" } 2022-09-12T19:10:53.646 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009843.080040_228.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:10:43.080040Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:10:53.647 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009843.080040_228.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:10:53.647 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009843.080040_228.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:10:53.648 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T19:10:53.648 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T19:10:53.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:10:53.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:10:53.650 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:10:53.709 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:10:53.709 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:10:53.752 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:10:53.749+0000 7f10ddd03600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:10:53.754 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:10:53.751+0000 7f10ddd03600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:10:53.766 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:10:53.767 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:10:53.767 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:10:53.767 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:10:53.768 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:10:53.768 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:10:53.768 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:10:53.768 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:10:53.769 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:10:53.769 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:10:53.769 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:10:53.769 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:10:53.770 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:10:53.770 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:10:53.770 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:10:53.771 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:10:53.771 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:10:53.771 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:10:53.771 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:10:53.772 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:10:53.772 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:10:53.772 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:10:53.772 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:10:53.773 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:10:53.773 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:10:53.773 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:10:53.773 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:10:53.774 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:10:53.774 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:10:53.774 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:10:53.774 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:10:53.775 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:10:53.775 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:10:53.775 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:10:53.775 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:10:53.776 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:10:53.776 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:10:53.776 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663009843.080040_228.1", 2022-09-12T19:10:53.776 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:10:53.777 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:10:53.777 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:10:53.777 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:43.080040Z", 2022-09-12T19:10:53.777 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:10:53.778 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:10:53.778 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:10:53.778 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:10:53.778 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:10:53.779 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:10:53.779 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:10:53.779 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:10:53.779 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:10:53.780 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:10:53.780 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:10:53.780 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:10:53.780 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:10:53.781 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:10:53.781 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:10:53.781 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:10:53.781 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:10:53.782 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:10:53.782 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:10:53.782 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:10:53.782 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:10:53.783 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:10:53Z" 2022-09-12T19:10:53.783 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:10:53.784 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009843.080040_228.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:10:43.080040Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:10:53Z" } 2022-09-12T19:10:53.785 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009843.080040_228.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:10:43.080040Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:10:53.785 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009843.080040_228.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:10:53.786 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009843.080040_228.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:10:53.786 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T19:10:53.787 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T19:10:53.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:10:53.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:10:53.788 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:10:53.841 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:10:53.842 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:10:53.870 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:10:53.868+0000 7f98c5204600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:10:53.871 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:10:53.869+0000 7f98c5204600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:10:53.883 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:10:53.883 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:10:53.883 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:10:53.884 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:10:53.884 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:10:53.884 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:10:53.885 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:10:53.885 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:10:53.885 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:10:53.886 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:10:53.886 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:10:53.886 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:10:53.886 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:10:53.887 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:10:53.887 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:10:53.887 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:10:53.887 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:10:53.888 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:10:53.888 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:10:53.888 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:10:53.888 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:10:53.889 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:10:53.889 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:10:53.889 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:10:53.889 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:10:53.890 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:10:53.890 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:10:53.891 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:10:53.891 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:10:53.891 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:10:53.892 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:10:53.892 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:10:53.892 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:10:53.892 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:10:53.893 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:10:53.893 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:10:53.894 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:10:53.894 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663009843.080040_228.1", 2022-09-12T19:10:53.895 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:10:53.895 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:10:53.896 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:10:53.896 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:43.080040Z", 2022-09-12T19:10:53.896 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:10:53.896 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:10:53.897 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:10:53.897 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:10:53.897 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:10:53.897 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:10:53.898 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:10:53.898 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:10:53.898 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:10:53.898 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:10:53.899 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:10:53.899 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:10:53.899 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:10:53.899 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:10:53.900 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:10:53.900 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:10:53.900 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:10:53.900 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:10:53.901 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:10:53.901 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:10:53.901 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:10:53.901 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:10:53Z" 2022-09-12T19:10:53.901 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:10:53.903 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009843.080040_228.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:10:43.080040Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:10:53Z" } 2022-09-12T19:10:53.904 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009843.080040_228.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:10:43.080040Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:10:53.904 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009843.080040_228.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:10:53.904 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009843.080040_228.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:10:53.905 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T19:10:53.905 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:10:53.906 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:10:53.907 DEBUG:boto:Method: HEAD 2022-09-12T19:10:53.907 DEBUG:boto:Path: /jdpiak-21/ 2022-09-12T19:10:53.908 DEBUG:boto:Data: 2022-09-12T19:10:53.908 DEBUG:boto:Headers: {} 2022-09-12T19:10:53.909 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:10:53.909 DEBUG:boto:Port: 8000 2022-09-12T19:10:53.910 DEBUG:boto:Params: {} 2022-09-12T19:10:53.910 DEBUG:boto:Token: None 2022-09-12T19:10:53.911 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:10:53 GMT /jdpiak-21/ 2022-09-12T19:10:53.911 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lr69UeAGhnMLbTfimuhVHust4Y8= 2022-09-12T19:10:53.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:10:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lr69UeAGhnMLbTfimuhVHust4Y8=', 'Content-Length': '0'} 2022-09-12T19:10:53.914 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054f6306f09f42ad4-00631f843d-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:53.915 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:10:53.916 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:10:53.916 DEBUG:boto:Method: PUT 2022-09-12T19:10:53.917 DEBUG:boto:Path: /jdpiak-21/obj 2022-09-12T19:10:53.917 DEBUG:boto:Data: 2022-09-12T19:10:53.918 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-12T19:10:53.919 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:10:53.919 DEBUG:boto:Port: 8000 2022-09-12T19:10:53.919 DEBUG:boto:Params: {} 2022-09-12T19:10:53.920 DEBUG:boto:Token: None 2022-09-12T19:10:53.921 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 12 Sep 2022 19:10:53 GMT /jdpiak-21/obj 2022-09-12T19:10:53.921 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TmqeET+LYIjDkEtAiS3YMmqpfkU= 2022-09-12T19:10:53.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:10:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TmqeET+LYIjDkEtAiS3YMmqpfkU='} 2022-09-12T19:10:53.933 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf'), ('x-amz-request-id', 'tx00000af443fd09664e4bc-00631f843d-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:53.934 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:10:53.934 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:10:53.935 DEBUG:boto:Method: HEAD 2022-09-12T19:10:53.935 DEBUG:boto:Path: /jdpiak-21/ 2022-09-12T19:10:53.935 DEBUG:boto:Data: 2022-09-12T19:10:53.936 DEBUG:boto:Headers: {} 2022-09-12T19:10:53.936 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:10:53.937 DEBUG:boto:Port: 8000 2022-09-12T19:10:53.938 DEBUG:boto:Params: {} 2022-09-12T19:10:53.938 DEBUG:boto:Token: None 2022-09-12T19:10:53.939 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:10:53 GMT /jdpiak-21/ 2022-09-12T19:10:53.939 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lr69UeAGhnMLbTfimuhVHust4Y8= 2022-09-12T19:10:53.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:10:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lr69UeAGhnMLbTfimuhVHust4Y8=', 'Content-Length': '0'} 2022-09-12T19:10:53.942 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1ac1df80f19e8d5-00631f843d-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:53.943 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:10:53.944 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:10:53.944 DEBUG:boto:Method: DELETE 2022-09-12T19:10:53.945 DEBUG:boto:Path: /jdpiak-21/obj 2022-09-12T19:10:53.945 DEBUG:boto:Data: 2022-09-12T19:10:53.946 DEBUG:boto:Headers: {} 2022-09-12T19:10:53.946 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:10:53.947 DEBUG:boto:Port: 8000 2022-09-12T19:10:53.947 DEBUG:boto:Params: {} 2022-09-12T19:10:53.948 DEBUG:boto:Token: None 2022-09-12T19:10:53.948 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 19:10:53 GMT /jdpiak-21/obj 2022-09-12T19:10:53.949 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cC+cfsQ3jaA1W2Eaqjq4wRvw8NM= 2022-09-12T19:10:53.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:10:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cC+cfsQ3jaA1W2Eaqjq4wRvw8NM=', 'Content-Length': '0'} 2022-09-12T19:10:53.957 DEBUG:boto:Response headers: [('x-amz-version-id', '6-xRDJ4ZvT4AR9bc0UAvLPOlra45TdY'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000031c4d7021cece2ec-00631f843d-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:53.957 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:10:53.958 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:10:53.958 DEBUG:boto:Method: HEAD 2022-09-12T19:10:53.959 DEBUG:boto:Path: /jdpiak-22/ 2022-09-12T19:10:53.959 DEBUG:boto:Data: 2022-09-12T19:10:53.960 DEBUG:boto:Headers: {} 2022-09-12T19:10:53.960 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:10:53.961 DEBUG:boto:Port: 8000 2022-09-12T19:10:53.962 DEBUG:boto:Params: {} 2022-09-12T19:10:53.963 DEBUG:boto:Token: None 2022-09-12T19:10:53.963 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:10:53 GMT /jdpiak-22/ 2022-09-12T19:10:53.964 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W6ebNYm8eMfvmao0GUriUKquV9Y= 2022-09-12T19:10:53.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:10:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W6ebNYm8eMfvmao0GUriUKquV9Y=', 'Content-Length': '0'} 2022-09-12T19:10:53.967 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c3200cbfcdd6002-00631f843d-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:53.968 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:10:53.968 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:10:53.969 DEBUG:boto:Method: PUT 2022-09-12T19:10:53.969 DEBUG:boto:Path: /jdpiak-22/obj 2022-09-12T19:10:53.970 DEBUG:boto:Data: 2022-09-12T19:10:53.970 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-12T19:10:53.971 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:10:53.971 DEBUG:boto:Port: 8000 2022-09-12T19:10:53.972 DEBUG:boto:Params: {} 2022-09-12T19:10:53.972 DEBUG:boto:Token: None 2022-09-12T19:10:53.973 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 12 Sep 2022 19:10:53 GMT /jdpiak-22/obj 2022-09-12T19:10:53.973 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Bzt69lmc8PpUgx3mlDeX4MtQABs= 2022-09-12T19:10:53.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:10:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Bzt69lmc8PpUgx3mlDeX4MtQABs='} 2022-09-12T19:10:54.018 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'fU7DRGj75cTxlyr6yy4TPugTgwe2QhY'), ('x-amz-request-id', 'tx0000086d1d4165830e630-00631f843d-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:54.019 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:10:54.019 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:10:54.020 DEBUG:boto:Method: HEAD 2022-09-12T19:10:54.021 DEBUG:boto:Path: /jdpiak-22/ 2022-09-12T19:10:54.021 DEBUG:boto:Data: 2022-09-12T19:10:54.022 DEBUG:boto:Headers: {} 2022-09-12T19:10:54.023 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:10:54.023 DEBUG:boto:Port: 8000 2022-09-12T19:10:54.024 DEBUG:boto:Params: {} 2022-09-12T19:10:54.025 DEBUG:boto:Token: None 2022-09-12T19:10:54.025 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:10:54 GMT /jdpiak-22/ 2022-09-12T19:10:54.026 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3VLZjM6rMEgGF3ZbkB5RkiNC2Yw= 2022-09-12T19:10:54.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:10:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3VLZjM6rMEgGF3ZbkB5RkiNC2Yw=', 'Content-Length': '0'} 2022-09-12T19:10:54.029 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090499d4de0c8c217-00631f843e-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:10:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:54.029 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:10:54.030 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:10:54.030 DEBUG:boto:Method: DELETE 2022-09-12T19:10:54.031 DEBUG:boto:Path: /jdpiak-22/obj 2022-09-12T19:10:54.032 DEBUG:boto:Data: 2022-09-12T19:10:54.032 DEBUG:boto:Headers: {} 2022-09-12T19:10:54.033 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:10:54.033 DEBUG:boto:Port: 8000 2022-09-12T19:10:54.033 DEBUG:boto:Params: {} 2022-09-12T19:10:54.034 DEBUG:boto:Token: None 2022-09-12T19:10:54.035 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 19:10:54 GMT /jdpiak-22/obj 2022-09-12T19:10:54.035 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6MfO+WPa+Gb+UZcfCSyRQJa7jDw= 2022-09-12T19:10:54.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:10:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6MfO+WPa+Gb+UZcfCSyRQJa7jDw=', 'Content-Length': '0'} 2022-09-12T19:10:54.042 DEBUG:boto:Response headers: [('x-amz-version-id', 'C-GaJtcvvXQHXDwz1TiDh8pTDLyLIrX'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000ba63cf0773cd7b0b-00631f843e-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 19:10:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:54.043 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:10:54.043 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:10:54.044 DEBUG:boto:Method: HEAD 2022-09-12T19:10:54.044 DEBUG:boto:Path: /jdpiak-23/ 2022-09-12T19:10:54.045 DEBUG:boto:Data: 2022-09-12T19:10:54.045 DEBUG:boto:Headers: {} 2022-09-12T19:10:54.046 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:10:54.046 DEBUG:boto:Port: 8001 2022-09-12T19:10:54.047 DEBUG:boto:Params: {} 2022-09-12T19:10:54.047 DEBUG:boto:Token: None 2022-09-12T19:10:54.048 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:10:54 GMT /jdpiak-23/ 2022-09-12T19:10:54.049 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/840g1fUykh69BJ9feVI55N40rY= 2022-09-12T19:10:54.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:10:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/840g1fUykh69BJ9feVI55N40rY=', 'Content-Length': '0'} 2022-09-12T19:10:54.052 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b77fb857e6431ad9-00631f843e-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:10:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:54.052 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:10:54.053 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:10:54.053 DEBUG:boto:Method: PUT 2022-09-12T19:10:54.054 DEBUG:boto:Path: /jdpiak-23/obj 2022-09-12T19:10:54.055 DEBUG:boto:Data: 2022-09-12T19:10:54.055 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-12T19:10:54.056 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:10:54.056 DEBUG:boto:Port: 8001 2022-09-12T19:10:54.057 DEBUG:boto:Params: {} 2022-09-12T19:10:54.057 DEBUG:boto:Token: None 2022-09-12T19:10:54.058 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 12 Sep 2022 19:10:54 GMT /jdpiak-23/obj 2022-09-12T19:10:54.058 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LP55oOGFu4dxLcXjPBbLNPlZz+c= 2022-09-12T19:10:54.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:10:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LP55oOGFu4dxLcXjPBbLNPlZz+c='} 2022-09-12T19:10:54.072 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0'), ('x-amz-request-id', 'tx000009c977036c2350110-00631f843e-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 19:10:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:54.072 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:10:54.073 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:10:54.073 DEBUG:boto:Method: HEAD 2022-09-12T19:10:54.074 DEBUG:boto:Path: /jdpiak-23/ 2022-09-12T19:10:54.075 DEBUG:boto:Data: 2022-09-12T19:10:54.075 DEBUG:boto:Headers: {} 2022-09-12T19:10:54.075 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:10:54.076 DEBUG:boto:Port: 8001 2022-09-12T19:10:54.077 DEBUG:boto:Params: {} 2022-09-12T19:10:54.078 DEBUG:boto:Token: None 2022-09-12T19:10:54.079 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:10:54 GMT /jdpiak-23/ 2022-09-12T19:10:54.079 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/840g1fUykh69BJ9feVI55N40rY= 2022-09-12T19:10:54.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:10:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/840g1fUykh69BJ9feVI55N40rY=', 'Content-Length': '0'} 2022-09-12T19:10:54.082 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4bbf1a3eafe9293-00631f843e-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:10:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:54.083 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:10:54.083 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:10:54.084 DEBUG:boto:Method: DELETE 2022-09-12T19:10:54.084 DEBUG:boto:Path: /jdpiak-23/obj 2022-09-12T19:10:54.085 DEBUG:boto:Data: 2022-09-12T19:10:54.085 DEBUG:boto:Headers: {} 2022-09-12T19:10:54.086 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:10:54.086 DEBUG:boto:Port: 8001 2022-09-12T19:10:54.087 DEBUG:boto:Params: {} 2022-09-12T19:10:54.087 DEBUG:boto:Token: None 2022-09-12T19:10:54.088 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 19:10:54 GMT /jdpiak-23/obj 2022-09-12T19:10:54.088 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3lE5nHxeVo6iIy/h2SW66miRGv8= 2022-09-12T19:10:54.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:10:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3lE5nHxeVo6iIy/h2SW66miRGv8=', 'Content-Length': '0'} 2022-09-12T19:10:54.100 DEBUG:boto:Response headers: [('x-amz-version-id', 'rKzI1doaCWXfUPsgqDJaJz0BenA63mM'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000009e7b98f78f038dcb-00631f843e-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 19:10:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:10:54.102 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-21 2022-09-12T19:10:54.102 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:10:54.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:10:54.104 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:10:54.163 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:10:54.163 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:10:54.206 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:10:54.203+0000 7f2e82b3c600 1 waiting to reach incremental sync.. 2022-09-12T19:10:59.207 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:10:59.204+0000 7f2e82b3c600 1 waiting to reach incremental sync.. 2022-09-12T19:11:04.210 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:04.206+0000 7f2e82b3c600 1 bucket sync caught up with source: 2022-09-12T19:11:04.211 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000003.11.11, , ] 2022-09-12T19:11:04.211 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000003.11.11, , ] 2022-09-12T19:11:04.211 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:04.206+0000 7f2e82b3c600 0 bucket checkpoint complete 2022-09-12T19:11:04.229 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-21 2022-09-12T19:11:04.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:04.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:04.232 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:11:04.297 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:04.298 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:04.345 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:04.343+0000 7fbf7662a600 1 bucket sync caught up with source: 2022-09-12T19:11:04.345 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000003.11.11, , ] 2022-09-12T19:11:04.346 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000003.11.11, , ] 2022-09-12T19:11:04.346 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:04.343+0000 7fbf7662a600 0 bucket checkpoint complete 2022-09-12T19:11:04.360 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-21 2022-09-12T19:11:04.361 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:04.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:04.362 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:11:04.426 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:04.427 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:04.467 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:04.464+0000 7f2c55264600 1 waiting to reach incremental sync.. 2022-09-12T19:11:09.469 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:09.466+0000 7f2c55264600 1 bucket sync caught up with source: 2022-09-12T19:11:09.470 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000003.11.11, , ] 2022-09-12T19:11:09.471 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000003.11.11, , ] 2022-09-12T19:11:09.471 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:09.466+0000 7f2c55264600 0 bucket checkpoint complete 2022-09-12T19:11:09.487 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-21 2022-09-12T19:11:09.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:09.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:09.489 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:11:09.556 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:09.557 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:09.595 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:09.593+0000 7f060a44e600 1 waiting to reach incremental sync.. 2022-09-12T19:11:14.595 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:14.592+0000 7f060a44e600 1 waiting to reach incremental sync.. 2022-09-12T19:11:19.596 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:19.593+0000 7f060a44e600 1 waiting to reach incremental sync.. 2022-09-12T19:11:24.598 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:24.595+0000 7f060a44e600 1 bucket sync caught up with source: 2022-09-12T19:11:24.598 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000002.44.5, , ] 2022-09-12T19:11:24.599 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000002.44.5, , ] 2022-09-12T19:11:24.599 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:24.595+0000 7f060a44e600 0 bucket checkpoint complete 2022-09-12T19:11:24.616 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-21 2022-09-12T19:11:24.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:24.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:24.618 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:11:24.683 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:24.683 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:24.731 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:24.728+0000 7f48404ed600 1 bucket sync caught up with source: 2022-09-12T19:11:24.731 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000002.44.5, , ] 2022-09-12T19:11:24.731 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000002.44.5, , ] 2022-09-12T19:11:24.731 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:24.728+0000 7f48404ed600 0 bucket checkpoint complete 2022-09-12T19:11:24.745 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-21 2022-09-12T19:11:24.746 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-21 2022-09-12T19:11:24.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:24.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:24.748 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:11:24.806 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:24.806 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:24.851 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:24.848+0000 7fb7b2dbb600 1 bucket sync caught up with source: 2022-09-12T19:11:24.851 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000002.38.5, , ] 2022-09-12T19:11:24.852 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000002.38.5, , ] 2022-09-12T19:11:24.852 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:24.848+0000 7fb7b2dbb600 0 bucket checkpoint complete 2022-09-12T19:11:24.866 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-21 2022-09-12T19:11:24.867 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:24.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:24.868 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:11:24.933 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:24.934 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:24.993 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:24.990+0000 7fd389c30600 1 bucket sync caught up with source: 2022-09-12T19:11:24.994 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000002.38.5, , ] 2022-09-12T19:11:24.994 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000002.38.5, , ] 2022-09-12T19:11:24.994 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:24.990+0000 7fd389c30600 0 bucket checkpoint complete 2022-09-12T19:11:25.016 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-21 2022-09-12T19:11:25.017 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-21 zones={test-zone2, test-zone1} 2022-09-12T19:11:25.018 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:11:25.018 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:11:25.019 DEBUG:boto:Method: HEAD 2022-09-12T19:11:25.019 DEBUG:boto:Path: /jdpiak-21/ 2022-09-12T19:11:25.020 DEBUG:boto:Data: 2022-09-12T19:11:25.020 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.021 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:25.021 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.022 DEBUG:boto:Params: {} 2022-09-12T19:11:25.023 DEBUG:boto:Token: None 2022-09-12T19:11:25.023 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/ 2022-09-12T19:11:25.024 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nf2oAs0gTxOo8tif5sJUUF7vdWY= 2022-09-12T19:11:25.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nf2oAs0gTxOo8tif5sJUUF7vdWY=', 'Content-Length': '0'} 2022-09-12T19:11:25.027 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc018926a1a46ee7-00631f845d-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.028 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:11:25.028 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:11:25.029 DEBUG:boto:Method: HEAD 2022-09-12T19:11:25.029 DEBUG:boto:Path: /jdpiak-21/ 2022-09-12T19:11:25.030 DEBUG:boto:Data: 2022-09-12T19:11:25.030 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.031 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:25.031 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.032 DEBUG:boto:Params: {} 2022-09-12T19:11:25.032 DEBUG:boto:Token: None 2022-09-12T19:11:25.033 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/ 2022-09-12T19:11:25.034 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nf2oAs0gTxOo8tif5sJUUF7vdWY= 2022-09-12T19:11:25.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nf2oAs0gTxOo8tif5sJUUF7vdWY=', 'Content-Length': '0'} 2022-09-12T19:11:25.036 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1ea6fa464e93d54-00631f845d-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.037 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:11:25.038 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:11:25.038 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:11:25.039 DEBUG:boto:path=/jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.039 DEBUG:boto:auth_path=/jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.040 DEBUG:boto:Method: GET 2022-09-12T19:11:25.040 DEBUG:boto:Path: /jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.041 DEBUG:boto:Data: 2022-09-12T19:11:25.041 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.042 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:25.042 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.043 DEBUG:boto:Params: {} 2022-09-12T19:11:25.043 DEBUG:boto:Token: None 2022-09-12T19:11:25.044 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/?versions 2022-09-12T19:11:25.045 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Rpo55xkwEk/RL8hDefAvsTGPaJw= 2022-09-12T19:11:25.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Rpo55xkwEk/RL8hDefAvsTGPaJw=', 'Content-Length': '0'} 2022-09-12T19:11:25.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dc9c0095998cca3-00631f845d-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.048 DEBUG:boto:b'jdpiak-21999falseobj6-xRDJ4ZvT4AR9bc0UAvLPOlra45TdYtrue2022-09-12T19:10:53.950Zrgw-multisite-test-userMultisite Test UserNormalobjOrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIffalse2022-09-12T19:10:53.922Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:25.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:25.050 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:25.050 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:11:25.051 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:11:25.052 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:11:25.052 DEBUG:boto:path=/jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.053 DEBUG:boto:auth_path=/jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.053 DEBUG:boto:Method: GET 2022-09-12T19:11:25.054 DEBUG:boto:Path: /jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.054 DEBUG:boto:Data: 2022-09-12T19:11:25.055 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.055 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:25.056 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.056 DEBUG:boto:Params: {} 2022-09-12T19:11:25.057 DEBUG:boto:Token: None 2022-09-12T19:11:25.057 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/?versions 2022-09-12T19:11:25.058 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Rpo55xkwEk/RL8hDefAvsTGPaJw= 2022-09-12T19:11:25.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Rpo55xkwEk/RL8hDefAvsTGPaJw=', 'Content-Length': '0'} 2022-09-12T19:11:25.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050d8ac453ecab61d-00631f845d-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.062 DEBUG:boto:b'jdpiak-21999falseobj6-xRDJ4ZvT4AR9bc0UAvLPOlra45TdYtrue2022-09-12T19:10:53.950Zrgw-multisite-test-userMultisite Test UserNormalobjOrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIffalse2022-09-12T19:10:53.922Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:25.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:25.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:25.064 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:11:25.064 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:11:25.065 DEBUG:boto:path=/jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.065 DEBUG:boto:auth_path=/jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.066 DEBUG:boto:Method: GET 2022-09-12T19:11:25.066 DEBUG:boto:Path: /jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.067 DEBUG:boto:Data: 2022-09-12T19:11:25.067 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.068 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:25.068 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.069 DEBUG:boto:Params: {} 2022-09-12T19:11:25.070 DEBUG:boto:Token: None 2022-09-12T19:11:25.070 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/?versions 2022-09-12T19:11:25.071 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Rpo55xkwEk/RL8hDefAvsTGPaJw= 2022-09-12T19:11:25.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Rpo55xkwEk/RL8hDefAvsTGPaJw=', 'Content-Length': '0'} 2022-09-12T19:11:25.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070db66d5650cb638-00631f845d-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.075 DEBUG:boto:b'jdpiak-21999falseobj6-xRDJ4ZvT4AR9bc0UAvLPOlra45TdYtrue2022-09-12T19:10:53.950Zrgw-multisite-test-userMultisite Test UserNormalobjOrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIffalse2022-09-12T19:10:53.922Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:25.075 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:11:25.076 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:11:25.076 DEBUG:boto:path=/jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.077 DEBUG:boto:auth_path=/jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.077 DEBUG:boto:Method: GET 2022-09-12T19:11:25.078 DEBUG:boto:Path: /jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.079 DEBUG:boto:Data: 2022-09-12T19:11:25.079 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.080 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:25.080 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.081 DEBUG:boto:Params: {} 2022-09-12T19:11:25.081 DEBUG:boto:Token: None 2022-09-12T19:11:25.082 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/?versions 2022-09-12T19:11:25.082 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Rpo55xkwEk/RL8hDefAvsTGPaJw= 2022-09-12T19:11:25.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Rpo55xkwEk/RL8hDefAvsTGPaJw=', 'Content-Length': '0'} 2022-09-12T19:11:25.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f816c8335f2c59b-00631f845d-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.086 DEBUG:boto:b'jdpiak-21999falseobj6-xRDJ4ZvT4AR9bc0UAvLPOlra45TdYtrue2022-09-12T19:10:53.950Zrgw-multisite-test-userMultisite Test UserNormalobjOrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIffalse2022-09-12T19:10:53.922Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:25.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:25.087 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.088 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.088 DEBUG:boto:Method: HEAD 2022-09-12T19:11:25.089 DEBUG:boto:Path: /jdpiak-21/obj 2022-09-12T19:11:25.089 DEBUG:boto:Data: 2022-09-12T19:11:25.090 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.091 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:25.091 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.092 DEBUG:boto:Params: {} 2022-09-12T19:11:25.092 DEBUG:boto:Token: None 2022-09-12T19:11:25.093 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj 2022-09-12T19:11:25.093 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4tK4PlFsSJcs3Dl3IbrikaMIm/I= 2022-09-12T19:11:25.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4tK4PlFsSJcs3Dl3IbrikaMIm/I=', 'Content-Length': '0'} 2022-09-12T19:11:25.096 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000031a198b289da314d-00631f845d-10e4-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.097 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.097 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.098 DEBUG:boto:Method: HEAD 2022-09-12T19:11:25.099 DEBUG:boto:Path: /jdpiak-21/obj 2022-09-12T19:11:25.099 DEBUG:boto:Data: 2022-09-12T19:11:25.100 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.100 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:25.101 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.101 DEBUG:boto:Params: {} 2022-09-12T19:11:25.102 DEBUG:boto:Token: None 2022-09-12T19:11:25.102 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj 2022-09-12T19:11:25.103 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4tK4PlFsSJcs3Dl3IbrikaMIm/I= 2022-09-12T19:11:25.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4tK4PlFsSJcs3Dl3IbrikaMIm/I=', 'Content-Length': '0'} 2022-09-12T19:11:25.106 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000c33d8ec26fdbe2d4-00631f845d-10f6-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:25.107 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.107 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.108 DEBUG:boto:path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.108 DEBUG:boto:auth_path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.109 DEBUG:boto:Method: GET 2022-09-12T19:11:25.110 DEBUG:boto:Path: /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.110 DEBUG:boto:Data: 2022-09-12T19:11:25.111 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.111 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:25.112 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.112 DEBUG:boto:Params: {} 2022-09-12T19:11:25.113 DEBUG:boto:Token: None 2022-09-12T19:11:25.113 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.114 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UWPLbYE5aoR/fEzoWhKIaNy1A1Q= 2022-09-12T19:11:25.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UWPLbYE5aoR/fEzoWhKIaNy1A1Q=', 'Content-Length': '0'} 2022-09-12T19:11:25.117 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a2d1663e37dbfc1e-00631f845d-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.117 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.118 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.118 DEBUG:boto:path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.119 DEBUG:boto:auth_path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.120 DEBUG:boto:Method: GET 2022-09-12T19:11:25.120 DEBUG:boto:Path: /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.121 DEBUG:boto:Data: 2022-09-12T19:11:25.121 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.122 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:25.122 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.123 DEBUG:boto:Params: {} 2022-09-12T19:11:25.123 DEBUG:boto:Token: None 2022-09-12T19:11:25.124 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.125 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UWPLbYE5aoR/fEzoWhKIaNy1A1Q= 2022-09-12T19:11:25.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UWPLbYE5aoR/fEzoWhKIaNy1A1Q=', 'Content-Length': '0'} 2022-09-12T19:11:25.127 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000f51e32754e93148-00631f845d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.128 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.128 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.129 DEBUG:boto:path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.129 DEBUG:boto:auth_path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.130 DEBUG:boto:Method: HEAD 2022-09-12T19:11:25.130 DEBUG:boto:Path: /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.131 DEBUG:boto:Data: 2022-09-12T19:11:25.131 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.132 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:25.132 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.133 DEBUG:boto:Params: {} 2022-09-12T19:11:25.133 DEBUG:boto:Token: None 2022-09-12T19:11:25.134 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.135 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kJ9mzKbaBX3Lwnriltw4bl7zLlk= 2022-09-12T19:11:25.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kJ9mzKbaBX3Lwnriltw4bl7zLlk=', 'Content-Length': '0'} 2022-09-12T19:11:25.137 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000dc06460ecfaf6cb-00631f845d-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.137 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.138 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.139 DEBUG:boto:path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.139 DEBUG:boto:auth_path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.140 DEBUG:boto:Method: HEAD 2022-09-12T19:11:25.140 DEBUG:boto:Path: /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.141 DEBUG:boto:Data: 2022-09-12T19:11:25.141 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.142 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:25.142 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.143 DEBUG:boto:Params: {} 2022-09-12T19:11:25.143 DEBUG:boto:Token: None 2022-09-12T19:11:25.144 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.144 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kJ9mzKbaBX3Lwnriltw4bl7zLlk= 2022-09-12T19:11:25.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kJ9mzKbaBX3Lwnriltw4bl7zLlk=', 'Content-Length': '0'} 2022-09-12T19:11:25.147 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003c1d1bc3459fd1f8-00631f845d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:25.148 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.148 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.149 DEBUG:boto:path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.149 DEBUG:boto:auth_path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.150 DEBUG:boto:Method: GET 2022-09-12T19:11:25.151 DEBUG:boto:Path: /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.151 DEBUG:boto:Data: 2022-09-12T19:11:25.152 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.152 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:25.153 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.153 DEBUG:boto:Params: {} 2022-09-12T19:11:25.154 DEBUG:boto:Token: None 2022-09-12T19:11:25.162 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.163 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UWPLbYE5aoR/fEzoWhKIaNy1A1Q= 2022-09-12T19:11:25.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UWPLbYE5aoR/fEzoWhKIaNy1A1Q=', 'Content-Length': '0'} 2022-09-12T19:11:25.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f7cd80f94a6ac4f0-00631f845d-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.167 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.167 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.168 DEBUG:boto:path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.168 DEBUG:boto:auth_path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.169 DEBUG:boto:Method: GET 2022-09-12T19:11:25.170 DEBUG:boto:Path: /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.170 DEBUG:boto:Data: 2022-09-12T19:11:25.171 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.171 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:25.172 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.172 DEBUG:boto:Params: {} 2022-09-12T19:11:25.173 DEBUG:boto:Token: None 2022-09-12T19:11:25.174 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.174 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UWPLbYE5aoR/fEzoWhKIaNy1A1Q= 2022-09-12T19:11:25.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UWPLbYE5aoR/fEzoWhKIaNy1A1Q=', 'Content-Length': '0'} 2022-09-12T19:11:25.177 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a56d345e335ff40b-00631f845d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.178 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:11:25.179 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:11:25.179 DEBUG:boto:path=/jdpiak-21/?prefix=obj 2022-09-12T19:11:25.180 DEBUG:boto:auth_path=/jdpiak-21/?prefix=obj 2022-09-12T19:11:25.180 DEBUG:boto:Method: GET 2022-09-12T19:11:25.181 DEBUG:boto:Path: /jdpiak-21/?prefix=obj 2022-09-12T19:11:25.182 DEBUG:boto:Data: 2022-09-12T19:11:25.182 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.183 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:25.183 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.184 DEBUG:boto:Params: {} 2022-09-12T19:11:25.184 DEBUG:boto:Token: None 2022-09-12T19:11:25.185 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/ 2022-09-12T19:11:25.185 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uMETpC26kRgtXWYFVDqmyf+29mU= 2022-09-12T19:11:25.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uMETpC26kRgtXWYFVDqmyf+29mU=', 'Content-Length': '0'} 2022-09-12T19:11:25.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000153b61838610d1f0-00631f845d-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.230 DEBUG:boto:b'jdpiak-21obj1000false' 2022-09-12T19:11:25.231 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:11:25.232 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:11:25.232 DEBUG:boto:path=/jdpiak-21/?prefix=obj 2022-09-12T19:11:25.233 DEBUG:boto:auth_path=/jdpiak-21/?prefix=obj 2022-09-12T19:11:25.233 DEBUG:boto:Method: GET 2022-09-12T19:11:25.234 DEBUG:boto:Path: /jdpiak-21/?prefix=obj 2022-09-12T19:11:25.234 DEBUG:boto:Data: 2022-09-12T19:11:25.235 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.235 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:25.236 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.237 DEBUG:boto:Params: {} 2022-09-12T19:11:25.237 DEBUG:boto:Token: None 2022-09-12T19:11:25.238 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/ 2022-09-12T19:11:25.238 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uMETpC26kRgtXWYFVDqmyf+29mU= 2022-09-12T19:11:25.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uMETpC26kRgtXWYFVDqmyf+29mU=', 'Content-Length': '0'} 2022-09-12T19:11:25.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3227b9eed3cdde5-00631f845d-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.242 DEBUG:boto:b'jdpiak-21obj1000false' 2022-09-12T19:11:25.243 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.243 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.244 DEBUG:boto:Method: HEAD 2022-09-12T19:11:25.244 DEBUG:boto:Path: /jdpiak-21/obj 2022-09-12T19:11:25.245 DEBUG:boto:Data: 2022-09-12T19:11:25.246 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.246 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:25.247 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.247 DEBUG:boto:Params: {} 2022-09-12T19:11:25.248 DEBUG:boto:Token: None 2022-09-12T19:11:25.248 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj 2022-09-12T19:11:25.249 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4tK4PlFsSJcs3Dl3IbrikaMIm/I= 2022-09-12T19:11:25.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4tK4PlFsSJcs3Dl3IbrikaMIm/I=', 'Content-Length': '0'} 2022-09-12T19:11:25.252 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000000a15d0ff144132b4-00631f845d-10e4-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.252 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.253 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.253 DEBUG:boto:Method: HEAD 2022-09-12T19:11:25.254 DEBUG:boto:Path: /jdpiak-21/obj 2022-09-12T19:11:25.254 DEBUG:boto:Data: 2022-09-12T19:11:25.255 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.255 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:25.256 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.256 DEBUG:boto:Params: {} 2022-09-12T19:11:25.257 DEBUG:boto:Token: None 2022-09-12T19:11:25.257 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj 2022-09-12T19:11:25.258 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4tK4PlFsSJcs3Dl3IbrikaMIm/I= 2022-09-12T19:11:25.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4tK4PlFsSJcs3Dl3IbrikaMIm/I=', 'Content-Length': '0'} 2022-09-12T19:11:25.261 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000d17907a77bea5445-00631f845d-10f6-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.261 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-21 zones={test-zone2, test-zone1} 2022-09-12T19:11:25.262 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-21 zones={test-zone3, test-zone1} 2022-09-12T19:11:25.262 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:11:25.263 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:11:25.264 DEBUG:boto:Method: HEAD 2022-09-12T19:11:25.265 DEBUG:boto:Path: /jdpiak-21/ 2022-09-12T19:11:25.266 DEBUG:boto:Data: 2022-09-12T19:11:25.266 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.267 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:25.267 DEBUG:boto:Port: 8001 2022-09-12T19:11:25.268 DEBUG:boto:Params: {} 2022-09-12T19:11:25.268 DEBUG:boto:Token: None 2022-09-12T19:11:25.269 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/ 2022-09-12T19:11:25.269 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nf2oAs0gTxOo8tif5sJUUF7vdWY= 2022-09-12T19:11:25.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nf2oAs0gTxOo8tif5sJUUF7vdWY=', 'Content-Length': '0'} 2022-09-12T19:11:25.272 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000485afe677a7dd13b-00631f845d-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.273 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:11:25.274 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:11:25.274 DEBUG:boto:Method: HEAD 2022-09-12T19:11:25.275 DEBUG:boto:Path: /jdpiak-21/ 2022-09-12T19:11:25.275 DEBUG:boto:Data: 2022-09-12T19:11:25.276 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.276 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:25.277 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.277 DEBUG:boto:Params: {} 2022-09-12T19:11:25.278 DEBUG:boto:Token: None 2022-09-12T19:11:25.278 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/ 2022-09-12T19:11:25.279 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nf2oAs0gTxOo8tif5sJUUF7vdWY= 2022-09-12T19:11:25.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nf2oAs0gTxOo8tif5sJUUF7vdWY=', 'Content-Length': '0'} 2022-09-12T19:11:25.282 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d7ad2d93f685a63-00631f845d-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.283 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:11:25.283 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:11:25.284 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:11:25.284 DEBUG:boto:path=/jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.285 DEBUG:boto:auth_path=/jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.285 DEBUG:boto:Method: GET 2022-09-12T19:11:25.286 DEBUG:boto:Path: /jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.286 DEBUG:boto:Data: 2022-09-12T19:11:25.287 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.287 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:25.288 DEBUG:boto:Port: 8001 2022-09-12T19:11:25.289 DEBUG:boto:Params: {} 2022-09-12T19:11:25.289 DEBUG:boto:Token: None 2022-09-12T19:11:25.290 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/?versions 2022-09-12T19:11:25.290 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Rpo55xkwEk/RL8hDefAvsTGPaJw= 2022-09-12T19:11:25.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Rpo55xkwEk/RL8hDefAvsTGPaJw=', 'Content-Length': '0'} 2022-09-12T19:11:25.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e109fa70280d6d57-00631f845d-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.294 DEBUG:boto:b'jdpiak-21999falseobj6-xRDJ4ZvT4AR9bc0UAvLPOlra45TdYtrue2022-09-12T19:10:53.950Zrgw-multisite-test-userMultisite Test UserNormalobjOrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIffalse2022-09-12T19:10:53.922Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:25.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:25.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:25.296 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:11:25.297 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:11:25.297 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:11:25.298 DEBUG:boto:path=/jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.298 DEBUG:boto:auth_path=/jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.299 DEBUG:boto:Method: GET 2022-09-12T19:11:25.300 DEBUG:boto:Path: /jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.300 DEBUG:boto:Data: 2022-09-12T19:11:25.301 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.301 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:25.302 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.302 DEBUG:boto:Params: {} 2022-09-12T19:11:25.303 DEBUG:boto:Token: None 2022-09-12T19:11:25.303 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/?versions 2022-09-12T19:11:25.304 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Rpo55xkwEk/RL8hDefAvsTGPaJw= 2022-09-12T19:11:25.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Rpo55xkwEk/RL8hDefAvsTGPaJw=', 'Content-Length': '0'} 2022-09-12T19:11:25.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050ea983c6fb85095-00631f845d-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.308 DEBUG:boto:b'jdpiak-21999falseobj6-xRDJ4ZvT4AR9bc0UAvLPOlra45TdYtrue2022-09-12T19:10:53.950Zrgw-multisite-test-userMultisite Test UserNormalobjOrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIffalse2022-09-12T19:10:53.922Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:25.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:25.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:25.310 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:11:25.310 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:11:25.311 DEBUG:boto:path=/jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.311 DEBUG:boto:auth_path=/jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.312 DEBUG:boto:Method: GET 2022-09-12T19:11:25.312 DEBUG:boto:Path: /jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.313 DEBUG:boto:Data: 2022-09-12T19:11:25.313 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.314 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:25.314 DEBUG:boto:Port: 8001 2022-09-12T19:11:25.315 DEBUG:boto:Params: {} 2022-09-12T19:11:25.315 DEBUG:boto:Token: None 2022-09-12T19:11:25.316 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/?versions 2022-09-12T19:11:25.316 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Rpo55xkwEk/RL8hDefAvsTGPaJw= 2022-09-12T19:11:25.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Rpo55xkwEk/RL8hDefAvsTGPaJw=', 'Content-Length': '0'} 2022-09-12T19:11:25.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ad25118804ba9cd-00631f845d-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.362 DEBUG:boto:b'jdpiak-21999falseobj6-xRDJ4ZvT4AR9bc0UAvLPOlra45TdYtrue2022-09-12T19:10:53.950Zrgw-multisite-test-userMultisite Test UserNormalobjOrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIffalse2022-09-12T19:10:53.922Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:25.363 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:11:25.364 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:11:25.365 DEBUG:boto:path=/jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.365 DEBUG:boto:auth_path=/jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.366 DEBUG:boto:Method: GET 2022-09-12T19:11:25.367 DEBUG:boto:Path: /jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.368 DEBUG:boto:Data: 2022-09-12T19:11:25.368 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.369 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:25.370 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.371 DEBUG:boto:Params: {} 2022-09-12T19:11:25.372 DEBUG:boto:Token: None 2022-09-12T19:11:25.372 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/?versions 2022-09-12T19:11:25.373 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Rpo55xkwEk/RL8hDefAvsTGPaJw= 2022-09-12T19:11:25.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Rpo55xkwEk/RL8hDefAvsTGPaJw=', 'Content-Length': '0'} 2022-09-12T19:11:25.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ae80e56266bdcc2-00631f845d-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.377 DEBUG:boto:b'jdpiak-21999falseobj6-xRDJ4ZvT4AR9bc0UAvLPOlra45TdYtrue2022-09-12T19:10:53.950Zrgw-multisite-test-userMultisite Test UserNormalobjOrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIffalse2022-09-12T19:10:53.922Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:25.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:25.378 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.379 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.379 DEBUG:boto:Method: HEAD 2022-09-12T19:11:25.380 DEBUG:boto:Path: /jdpiak-21/obj 2022-09-12T19:11:25.380 DEBUG:boto:Data: 2022-09-12T19:11:25.381 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.381 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:25.382 DEBUG:boto:Port: 8001 2022-09-12T19:11:25.383 DEBUG:boto:Params: {} 2022-09-12T19:11:25.383 DEBUG:boto:Token: None 2022-09-12T19:11:25.384 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj 2022-09-12T19:11:25.385 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4tK4PlFsSJcs3Dl3IbrikaMIm/I= 2022-09-12T19:11:25.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4tK4PlFsSJcs3Dl3IbrikaMIm/I=', 'Content-Length': '0'} 2022-09-12T19:11:25.388 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000c13a22821a79d644-00631f845d-10f2-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.389 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.390 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.391 DEBUG:boto:Method: HEAD 2022-09-12T19:11:25.392 DEBUG:boto:Path: /jdpiak-21/obj 2022-09-12T19:11:25.393 DEBUG:boto:Data: 2022-09-12T19:11:25.394 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.395 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:25.396 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.397 DEBUG:boto:Params: {} 2022-09-12T19:11:25.398 DEBUG:boto:Token: None 2022-09-12T19:11:25.399 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj 2022-09-12T19:11:25.400 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4tK4PlFsSJcs3Dl3IbrikaMIm/I= 2022-09-12T19:11:25.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4tK4PlFsSJcs3Dl3IbrikaMIm/I=', 'Content-Length': '0'} 2022-09-12T19:11:25.404 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000dc80c3eca17e5c2d-00631f845d-10f6-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:25.406 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.407 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.408 DEBUG:boto:path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.409 DEBUG:boto:auth_path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.410 DEBUG:boto:Method: GET 2022-09-12T19:11:25.410 DEBUG:boto:Path: /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.411 DEBUG:boto:Data: 2022-09-12T19:11:25.411 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.412 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:25.413 DEBUG:boto:Port: 8001 2022-09-12T19:11:25.413 DEBUG:boto:Params: {} 2022-09-12T19:11:25.414 DEBUG:boto:Token: None 2022-09-12T19:11:25.414 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.415 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UWPLbYE5aoR/fEzoWhKIaNy1A1Q= 2022-09-12T19:11:25.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UWPLbYE5aoR/fEzoWhKIaNy1A1Q=', 'Content-Length': '0'} 2022-09-12T19:11:25.418 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f6b4b749ca0ba895-00631f845d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.418 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.419 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.420 DEBUG:boto:path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.420 DEBUG:boto:auth_path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.421 DEBUG:boto:Method: GET 2022-09-12T19:11:25.421 DEBUG:boto:Path: /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.422 DEBUG:boto:Data: 2022-09-12T19:11:25.422 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.423 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:25.423 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.424 DEBUG:boto:Params: {} 2022-09-12T19:11:25.424 DEBUG:boto:Token: None 2022-09-12T19:11:25.425 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.426 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UWPLbYE5aoR/fEzoWhKIaNy1A1Q= 2022-09-12T19:11:25.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UWPLbYE5aoR/fEzoWhKIaNy1A1Q=', 'Content-Length': '0'} 2022-09-12T19:11:25.428 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000092bbfbc49c2eb65a-00631f845d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.430 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.431 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.432 DEBUG:boto:path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.433 DEBUG:boto:auth_path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.434 DEBUG:boto:Method: HEAD 2022-09-12T19:11:25.434 DEBUG:boto:Path: /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.435 DEBUG:boto:Data: 2022-09-12T19:11:25.436 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.437 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:25.438 DEBUG:boto:Port: 8001 2022-09-12T19:11:25.439 DEBUG:boto:Params: {} 2022-09-12T19:11:25.440 DEBUG:boto:Token: None 2022-09-12T19:11:25.441 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.442 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kJ9mzKbaBX3Lwnriltw4bl7zLlk= 2022-09-12T19:11:25.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kJ9mzKbaBX3Lwnriltw4bl7zLlk=', 'Content-Length': '0'} 2022-09-12T19:11:25.444 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f2be948b311ddc4e-00631f845d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.445 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.445 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.446 DEBUG:boto:path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.446 DEBUG:boto:auth_path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.447 DEBUG:boto:Method: HEAD 2022-09-12T19:11:25.447 DEBUG:boto:Path: /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.448 DEBUG:boto:Data: 2022-09-12T19:11:25.448 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.449 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:25.449 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.450 DEBUG:boto:Params: {} 2022-09-12T19:11:25.450 DEBUG:boto:Token: None 2022-09-12T19:11:25.451 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.452 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kJ9mzKbaBX3Lwnriltw4bl7zLlk= 2022-09-12T19:11:25.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kJ9mzKbaBX3Lwnriltw4bl7zLlk=', 'Content-Length': '0'} 2022-09-12T19:11:25.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000270724e3eff2013c-00631f845d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:25.455 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.456 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.456 DEBUG:boto:path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.457 DEBUG:boto:auth_path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.457 DEBUG:boto:Method: GET 2022-09-12T19:11:25.458 DEBUG:boto:Path: /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.459 DEBUG:boto:Data: 2022-09-12T19:11:25.459 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.460 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:25.460 DEBUG:boto:Port: 8001 2022-09-12T19:11:25.461 DEBUG:boto:Params: {} 2022-09-12T19:11:25.461 DEBUG:boto:Token: None 2022-09-12T19:11:25.462 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.463 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UWPLbYE5aoR/fEzoWhKIaNy1A1Q= 2022-09-12T19:11:25.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UWPLbYE5aoR/fEzoWhKIaNy1A1Q=', 'Content-Length': '0'} 2022-09-12T19:11:25.465 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009cb3a10fb75a2ae7-00631f845d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.466 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.466 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.467 DEBUG:boto:path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.467 DEBUG:boto:auth_path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.468 DEBUG:boto:Method: GET 2022-09-12T19:11:25.468 DEBUG:boto:Path: /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.469 DEBUG:boto:Data: 2022-09-12T19:11:25.470 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.470 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:25.471 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.471 DEBUG:boto:Params: {} 2022-09-12T19:11:25.472 DEBUG:boto:Token: None 2022-09-12T19:11:25.472 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.473 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UWPLbYE5aoR/fEzoWhKIaNy1A1Q= 2022-09-12T19:11:25.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UWPLbYE5aoR/fEzoWhKIaNy1A1Q=', 'Content-Length': '0'} 2022-09-12T19:11:25.476 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f7ecd4a3db4933d9-00631f845d-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.477 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:11:25.477 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:11:25.478 DEBUG:boto:path=/jdpiak-21/?prefix=obj 2022-09-12T19:11:25.478 DEBUG:boto:auth_path=/jdpiak-21/?prefix=obj 2022-09-12T19:11:25.479 DEBUG:boto:Method: GET 2022-09-12T19:11:25.479 DEBUG:boto:Path: /jdpiak-21/?prefix=obj 2022-09-12T19:11:25.480 DEBUG:boto:Data: 2022-09-12T19:11:25.480 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.481 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:25.481 DEBUG:boto:Port: 8001 2022-09-12T19:11:25.482 DEBUG:boto:Params: {} 2022-09-12T19:11:25.483 DEBUG:boto:Token: None 2022-09-12T19:11:25.483 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/ 2022-09-12T19:11:25.484 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uMETpC26kRgtXWYFVDqmyf+29mU= 2022-09-12T19:11:25.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uMETpC26kRgtXWYFVDqmyf+29mU=', 'Content-Length': '0'} 2022-09-12T19:11:25.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dc86c65ab67e7af-00631f845d-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.488 DEBUG:boto:b'jdpiak-21obj1000false' 2022-09-12T19:11:25.488 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:11:25.489 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:11:25.490 DEBUG:boto:path=/jdpiak-21/?prefix=obj 2022-09-12T19:11:25.490 DEBUG:boto:auth_path=/jdpiak-21/?prefix=obj 2022-09-12T19:11:25.491 DEBUG:boto:Method: GET 2022-09-12T19:11:25.491 DEBUG:boto:Path: /jdpiak-21/?prefix=obj 2022-09-12T19:11:25.492 DEBUG:boto:Data: 2022-09-12T19:11:25.492 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.493 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:25.493 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.494 DEBUG:boto:Params: {} 2022-09-12T19:11:25.494 DEBUG:boto:Token: None 2022-09-12T19:11:25.495 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/ 2022-09-12T19:11:25.496 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uMETpC26kRgtXWYFVDqmyf+29mU= 2022-09-12T19:11:25.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uMETpC26kRgtXWYFVDqmyf+29mU=', 'Content-Length': '0'} 2022-09-12T19:11:25.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d3b0f254bd34308-00631f845d-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.542 DEBUG:boto:b'jdpiak-21obj1000false' 2022-09-12T19:11:25.543 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.544 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.544 DEBUG:boto:Method: HEAD 2022-09-12T19:11:25.545 DEBUG:boto:Path: /jdpiak-21/obj 2022-09-12T19:11:25.545 DEBUG:boto:Data: 2022-09-12T19:11:25.546 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.546 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:25.547 DEBUG:boto:Port: 8001 2022-09-12T19:11:25.547 DEBUG:boto:Params: {} 2022-09-12T19:11:25.548 DEBUG:boto:Token: None 2022-09-12T19:11:25.548 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj 2022-09-12T19:11:25.549 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4tK4PlFsSJcs3Dl3IbrikaMIm/I= 2022-09-12T19:11:25.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4tK4PlFsSJcs3Dl3IbrikaMIm/I=', 'Content-Length': '0'} 2022-09-12T19:11:25.552 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000056cdcc4f670b1d1d-00631f845d-10f2-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.553 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.554 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.555 DEBUG:boto:Method: HEAD 2022-09-12T19:11:25.555 DEBUG:boto:Path: /jdpiak-21/obj 2022-09-12T19:11:25.556 DEBUG:boto:Data: 2022-09-12T19:11:25.557 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.558 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:25.559 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.559 DEBUG:boto:Params: {} 2022-09-12T19:11:25.560 DEBUG:boto:Token: None 2022-09-12T19:11:25.561 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj 2022-09-12T19:11:25.562 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4tK4PlFsSJcs3Dl3IbrikaMIm/I= 2022-09-12T19:11:25.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4tK4PlFsSJcs3Dl3IbrikaMIm/I=', 'Content-Length': '0'} 2022-09-12T19:11:25.566 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000067d778c3c716a5c4-00631f845d-10f6-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.567 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-21 zones={test-zone3, test-zone1} 2022-09-12T19:11:25.568 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-21 zones={test-zone3, test-zone2} 2022-09-12T19:11:25.568 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:11:25.569 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:11:25.569 DEBUG:boto:Method: HEAD 2022-09-12T19:11:25.570 DEBUG:boto:Path: /jdpiak-21/ 2022-09-12T19:11:25.570 DEBUG:boto:Data: 2022-09-12T19:11:25.571 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.571 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:25.572 DEBUG:boto:Port: 8001 2022-09-12T19:11:25.572 DEBUG:boto:Params: {} 2022-09-12T19:11:25.573 DEBUG:boto:Token: None 2022-09-12T19:11:25.574 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/ 2022-09-12T19:11:25.574 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nf2oAs0gTxOo8tif5sJUUF7vdWY= 2022-09-12T19:11:25.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nf2oAs0gTxOo8tif5sJUUF7vdWY=', 'Content-Length': '0'} 2022-09-12T19:11:25.577 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f13f61612b2b579-00631f845d-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.578 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:11:25.579 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:11:25.580 DEBUG:boto:Method: HEAD 2022-09-12T19:11:25.580 DEBUG:boto:Path: /jdpiak-21/ 2022-09-12T19:11:25.581 DEBUG:boto:Data: 2022-09-12T19:11:25.581 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.582 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:25.582 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.583 DEBUG:boto:Params: {} 2022-09-12T19:11:25.584 DEBUG:boto:Token: None 2022-09-12T19:11:25.584 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/ 2022-09-12T19:11:25.585 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nf2oAs0gTxOo8tif5sJUUF7vdWY= 2022-09-12T19:11:25.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nf2oAs0gTxOo8tif5sJUUF7vdWY=', 'Content-Length': '0'} 2022-09-12T19:11:25.588 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6b0b10439ac47fa-00631f845d-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.588 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:11:25.589 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:11:25.589 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:11:25.590 DEBUG:boto:path=/jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.590 DEBUG:boto:auth_path=/jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.591 DEBUG:boto:Method: GET 2022-09-12T19:11:25.591 DEBUG:boto:Path: /jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.592 DEBUG:boto:Data: 2022-09-12T19:11:25.592 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.593 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:25.593 DEBUG:boto:Port: 8001 2022-09-12T19:11:25.594 DEBUG:boto:Params: {} 2022-09-12T19:11:25.594 DEBUG:boto:Token: None 2022-09-12T19:11:25.595 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/?versions 2022-09-12T19:11:25.596 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Rpo55xkwEk/RL8hDefAvsTGPaJw= 2022-09-12T19:11:25.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Rpo55xkwEk/RL8hDefAvsTGPaJw=', 'Content-Length': '0'} 2022-09-12T19:11:25.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009eb29e2dc695b19f-00631f845d-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.599 DEBUG:boto:b'jdpiak-21999falseobj6-xRDJ4ZvT4AR9bc0UAvLPOlra45TdYtrue2022-09-12T19:10:53.950Zrgw-multisite-test-userMultisite Test UserNormalobjOrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIffalse2022-09-12T19:10:53.922Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:25.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:25.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:25.601 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:11:25.602 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:11:25.602 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:11:25.603 DEBUG:boto:path=/jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.603 DEBUG:boto:auth_path=/jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.604 DEBUG:boto:Method: GET 2022-09-12T19:11:25.604 DEBUG:boto:Path: /jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.605 DEBUG:boto:Data: 2022-09-12T19:11:25.605 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.606 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:25.607 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.607 DEBUG:boto:Params: {} 2022-09-12T19:11:25.609 DEBUG:boto:Token: None 2022-09-12T19:11:25.610 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/?versions 2022-09-12T19:11:25.611 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Rpo55xkwEk/RL8hDefAvsTGPaJw= 2022-09-12T19:11:25.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Rpo55xkwEk/RL8hDefAvsTGPaJw=', 'Content-Length': '0'} 2022-09-12T19:11:25.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1ce25a4e2de1adc-00631f845d-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.616 DEBUG:boto:b'jdpiak-21999falseobj6-xRDJ4ZvT4AR9bc0UAvLPOlra45TdYtrue2022-09-12T19:10:53.950Zrgw-multisite-test-userMultisite Test UserNormalobjOrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIffalse2022-09-12T19:10:53.922Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:25.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:25.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:25.618 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:11:25.619 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:11:25.619 DEBUG:boto:path=/jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.620 DEBUG:boto:auth_path=/jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.620 DEBUG:boto:Method: GET 2022-09-12T19:11:25.621 DEBUG:boto:Path: /jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.622 DEBUG:boto:Data: 2022-09-12T19:11:25.622 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.623 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:25.623 DEBUG:boto:Port: 8001 2022-09-12T19:11:25.624 DEBUG:boto:Params: {} 2022-09-12T19:11:25.624 DEBUG:boto:Token: None 2022-09-12T19:11:25.625 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/?versions 2022-09-12T19:11:25.625 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Rpo55xkwEk/RL8hDefAvsTGPaJw= 2022-09-12T19:11:25.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Rpo55xkwEk/RL8hDefAvsTGPaJw=', 'Content-Length': '0'} 2022-09-12T19:11:25.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000589d2924097cb1e5-00631f845d-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.630 DEBUG:boto:b'jdpiak-21999falseobj6-xRDJ4ZvT4AR9bc0UAvLPOlra45TdYtrue2022-09-12T19:10:53.950Zrgw-multisite-test-userMultisite Test UserNormalobjOrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIffalse2022-09-12T19:10:53.922Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:25.631 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:11:25.631 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:11:25.632 DEBUG:boto:path=/jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.632 DEBUG:boto:auth_path=/jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.633 DEBUG:boto:Method: GET 2022-09-12T19:11:25.633 DEBUG:boto:Path: /jdpiak-21/?versions&max-keys=999 2022-09-12T19:11:25.634 DEBUG:boto:Data: 2022-09-12T19:11:25.635 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.635 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:25.636 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.636 DEBUG:boto:Params: {} 2022-09-12T19:11:25.637 DEBUG:boto:Token: None 2022-09-12T19:11:25.637 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/?versions 2022-09-12T19:11:25.638 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Rpo55xkwEk/RL8hDefAvsTGPaJw= 2022-09-12T19:11:25.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Rpo55xkwEk/RL8hDefAvsTGPaJw=', 'Content-Length': '0'} 2022-09-12T19:11:25.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4f25864114c23d2-00631f845d-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.641 DEBUG:boto:b'jdpiak-21999falseobj6-xRDJ4ZvT4AR9bc0UAvLPOlra45TdYtrue2022-09-12T19:10:53.950Zrgw-multisite-test-userMultisite Test UserNormalobjOrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIffalse2022-09-12T19:10:53.922Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:25.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:25.643 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.643 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.644 DEBUG:boto:Method: HEAD 2022-09-12T19:11:25.644 DEBUG:boto:Path: /jdpiak-21/obj 2022-09-12T19:11:25.645 DEBUG:boto:Data: 2022-09-12T19:11:25.645 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.646 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:25.646 DEBUG:boto:Port: 8001 2022-09-12T19:11:25.647 DEBUG:boto:Params: {} 2022-09-12T19:11:25.647 DEBUG:boto:Token: None 2022-09-12T19:11:25.648 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj 2022-09-12T19:11:25.649 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4tK4PlFsSJcs3Dl3IbrikaMIm/I= 2022-09-12T19:11:25.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4tK4PlFsSJcs3Dl3IbrikaMIm/I=', 'Content-Length': '0'} 2022-09-12T19:11:25.651 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000048d5ef7687622d0f-00631f845d-10f2-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.652 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.652 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.653 DEBUG:boto:Method: HEAD 2022-09-12T19:11:25.653 DEBUG:boto:Path: /jdpiak-21/obj 2022-09-12T19:11:25.654 DEBUG:boto:Data: 2022-09-12T19:11:25.654 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.655 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:25.655 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.656 DEBUG:boto:Params: {} 2022-09-12T19:11:25.656 DEBUG:boto:Token: None 2022-09-12T19:11:25.657 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj 2022-09-12T19:11:25.658 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4tK4PlFsSJcs3Dl3IbrikaMIm/I= 2022-09-12T19:11:25.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4tK4PlFsSJcs3Dl3IbrikaMIm/I=', 'Content-Length': '0'} 2022-09-12T19:11:25.660 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000633257da6f83a101-00631f845d-10e4-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:25.661 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.662 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.662 DEBUG:boto:path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.663 DEBUG:boto:auth_path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.663 DEBUG:boto:Method: GET 2022-09-12T19:11:25.664 DEBUG:boto:Path: /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.664 DEBUG:boto:Data: 2022-09-12T19:11:25.665 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.665 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:25.666 DEBUG:boto:Port: 8001 2022-09-12T19:11:25.667 DEBUG:boto:Params: {} 2022-09-12T19:11:25.667 DEBUG:boto:Token: None 2022-09-12T19:11:25.668 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.668 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UWPLbYE5aoR/fEzoWhKIaNy1A1Q= 2022-09-12T19:11:25.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UWPLbYE5aoR/fEzoWhKIaNy1A1Q=', 'Content-Length': '0'} 2022-09-12T19:11:25.671 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dea1a38d916e65e6-00631f845d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.672 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.672 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.673 DEBUG:boto:path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.673 DEBUG:boto:auth_path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.674 DEBUG:boto:Method: GET 2022-09-12T19:11:25.674 DEBUG:boto:Path: /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.675 DEBUG:boto:Data: 2022-09-12T19:11:25.675 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.676 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:25.677 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.677 DEBUG:boto:Params: {} 2022-09-12T19:11:25.678 DEBUG:boto:Token: None 2022-09-12T19:11:25.678 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.679 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UWPLbYE5aoR/fEzoWhKIaNy1A1Q= 2022-09-12T19:11:25.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UWPLbYE5aoR/fEzoWhKIaNy1A1Q=', 'Content-Length': '0'} 2022-09-12T19:11:25.681 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007ffa03306de88908-00631f845d-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.682 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.683 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.683 DEBUG:boto:path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.684 DEBUG:boto:auth_path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.684 DEBUG:boto:Method: HEAD 2022-09-12T19:11:25.685 DEBUG:boto:Path: /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.685 DEBUG:boto:Data: 2022-09-12T19:11:25.686 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.686 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:25.687 DEBUG:boto:Port: 8001 2022-09-12T19:11:25.687 DEBUG:boto:Params: {} 2022-09-12T19:11:25.688 DEBUG:boto:Token: None 2022-09-12T19:11:25.688 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.689 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kJ9mzKbaBX3Lwnriltw4bl7zLlk= 2022-09-12T19:11:25.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kJ9mzKbaBX3Lwnriltw4bl7zLlk=', 'Content-Length': '0'} 2022-09-12T19:11:25.691 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d0c9884c7b03f243-00631f845d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.692 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.693 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.693 DEBUG:boto:path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.694 DEBUG:boto:auth_path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.694 DEBUG:boto:Method: HEAD 2022-09-12T19:11:25.695 DEBUG:boto:Path: /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.695 DEBUG:boto:Data: 2022-09-12T19:11:25.696 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.696 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:25.697 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.697 DEBUG:boto:Params: {} 2022-09-12T19:11:25.698 DEBUG:boto:Token: None 2022-09-12T19:11:25.698 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.699 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kJ9mzKbaBX3Lwnriltw4bl7zLlk= 2022-09-12T19:11:25.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kJ9mzKbaBX3Lwnriltw4bl7zLlk=', 'Content-Length': '0'} 2022-09-12T19:11:25.701 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000030c587922d799b86-00631f845d-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:25.703 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.703 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.704 DEBUG:boto:path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.704 DEBUG:boto:auth_path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.705 DEBUG:boto:Method: GET 2022-09-12T19:11:25.706 DEBUG:boto:Path: /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.706 DEBUG:boto:Data: 2022-09-12T19:11:25.707 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.707 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:25.708 DEBUG:boto:Port: 8001 2022-09-12T19:11:25.708 DEBUG:boto:Params: {} 2022-09-12T19:11:25.709 DEBUG:boto:Token: None 2022-09-12T19:11:25.710 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.710 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UWPLbYE5aoR/fEzoWhKIaNy1A1Q= 2022-09-12T19:11:25.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UWPLbYE5aoR/fEzoWhKIaNy1A1Q=', 'Content-Length': '0'} 2022-09-12T19:11:25.713 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ffd4c3695383518c-00631f845d-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.713 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.714 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.715 DEBUG:boto:path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.715 DEBUG:boto:auth_path=/jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.716 DEBUG:boto:Method: GET 2022-09-12T19:11:25.716 DEBUG:boto:Path: /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.717 DEBUG:boto:Data: 2022-09-12T19:11:25.717 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.718 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:25.718 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.719 DEBUG:boto:Params: {} 2022-09-12T19:11:25.719 DEBUG:boto:Token: None 2022-09-12T19:11:25.720 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj?versionId=OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf 2022-09-12T19:11:25.721 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UWPLbYE5aoR/fEzoWhKIaNy1A1Q= 2022-09-12T19:11:25.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UWPLbYE5aoR/fEzoWhKIaNy1A1Q=', 'Content-Length': '0'} 2022-09-12T19:11:25.723 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'OrkEQBwEuRDcmC-rz9Ta-4bHmlW8wIf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000131a2bd6ea000664-00631f845d-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.724 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:11:25.724 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:11:25.725 DEBUG:boto:path=/jdpiak-21/?prefix=obj 2022-09-12T19:11:25.725 DEBUG:boto:auth_path=/jdpiak-21/?prefix=obj 2022-09-12T19:11:25.726 DEBUG:boto:Method: GET 2022-09-12T19:11:25.726 DEBUG:boto:Path: /jdpiak-21/?prefix=obj 2022-09-12T19:11:25.727 DEBUG:boto:Data: 2022-09-12T19:11:25.728 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.728 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:25.729 DEBUG:boto:Port: 8001 2022-09-12T19:11:25.729 DEBUG:boto:Params: {} 2022-09-12T19:11:25.730 DEBUG:boto:Token: None 2022-09-12T19:11:25.730 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/ 2022-09-12T19:11:25.731 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uMETpC26kRgtXWYFVDqmyf+29mU= 2022-09-12T19:11:25.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uMETpC26kRgtXWYFVDqmyf+29mU=', 'Content-Length': '0'} 2022-09-12T19:11:25.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbc1fbe649fd3608-00631f845d-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.735 DEBUG:boto:b'jdpiak-21obj1000false' 2022-09-12T19:11:25.736 DEBUG:boto:path=/jdpiak-21/ 2022-09-12T19:11:25.736 DEBUG:boto:auth_path=/jdpiak-21/ 2022-09-12T19:11:25.737 DEBUG:boto:path=/jdpiak-21/?prefix=obj 2022-09-12T19:11:25.737 DEBUG:boto:auth_path=/jdpiak-21/?prefix=obj 2022-09-12T19:11:25.738 DEBUG:boto:Method: GET 2022-09-12T19:11:25.738 DEBUG:boto:Path: /jdpiak-21/?prefix=obj 2022-09-12T19:11:25.739 DEBUG:boto:Data: 2022-09-12T19:11:25.739 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.740 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:25.740 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.741 DEBUG:boto:Params: {} 2022-09-12T19:11:25.741 DEBUG:boto:Token: None 2022-09-12T19:11:25.742 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/ 2022-09-12T19:11:25.743 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uMETpC26kRgtXWYFVDqmyf+29mU= 2022-09-12T19:11:25.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uMETpC26kRgtXWYFVDqmyf+29mU=', 'Content-Length': '0'} 2022-09-12T19:11:25.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df364dac4f752ecf-00631f845d-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.746 DEBUG:boto:b'jdpiak-21obj1000false' 2022-09-12T19:11:25.747 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.748 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.748 DEBUG:boto:Method: HEAD 2022-09-12T19:11:25.749 DEBUG:boto:Path: /jdpiak-21/obj 2022-09-12T19:11:25.749 DEBUG:boto:Data: 2022-09-12T19:11:25.750 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.750 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:25.751 DEBUG:boto:Port: 8001 2022-09-12T19:11:25.751 DEBUG:boto:Params: {} 2022-09-12T19:11:25.752 DEBUG:boto:Token: None 2022-09-12T19:11:25.752 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj 2022-09-12T19:11:25.753 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4tK4PlFsSJcs3Dl3IbrikaMIm/I= 2022-09-12T19:11:25.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4tK4PlFsSJcs3Dl3IbrikaMIm/I=', 'Content-Length': '0'} 2022-09-12T19:11:25.756 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000003af2bc5fc0905383-00631f845d-10f2-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.756 DEBUG:boto:path=/jdpiak-21/obj 2022-09-12T19:11:25.757 DEBUG:boto:auth_path=/jdpiak-21/obj 2022-09-12T19:11:25.757 DEBUG:boto:Method: HEAD 2022-09-12T19:11:25.758 DEBUG:boto:Path: /jdpiak-21/obj 2022-09-12T19:11:25.758 DEBUG:boto:Data: 2022-09-12T19:11:25.759 DEBUG:boto:Headers: {} 2022-09-12T19:11:25.759 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:25.760 DEBUG:boto:Port: 8000 2022-09-12T19:11:25.760 DEBUG:boto:Params: {} 2022-09-12T19:11:25.761 DEBUG:boto:Token: None 2022-09-12T19:11:25.761 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:25 GMT /jdpiak-21/obj 2022-09-12T19:11:25.762 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4tK4PlFsSJcs3Dl3IbrikaMIm/I= 2022-09-12T19:11:25.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4tK4PlFsSJcs3Dl3IbrikaMIm/I=', 'Content-Length': '0'} 2022-09-12T19:11:25.764 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000062aaa263d66d88b5-00631f845d-10e4-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:25.765 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-21 zones={test-zone3, test-zone2} 2022-09-12T19:11:25.766 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-22 2022-09-12T19:11:25.766 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:25.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:25.767 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:11:25.823 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:25.824 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:25.864 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:25.861+0000 7fe39eed2600 1 waiting to reach incremental sync.. 2022-09-12T19:11:30.865 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:30.862+0000 7fe39eed2600 1 waiting to reach incremental sync.. 2022-09-12T19:11:35.866 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:35.863+0000 7fe39eed2600 1 waiting to reach incremental sync.. 2022-09-12T19:11:40.866 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:40.863+0000 7fe39eed2600 1 waiting to reach incremental sync.. 2022-09-12T19:11:45.868 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:45.865+0000 7fe39eed2600 1 bucket sync caught up with source: 2022-09-12T19:11:45.869 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000002.14.5, , ] 2022-09-12T19:11:45.869 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000002.14.5, , ] 2022-09-12T19:11:45.869 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:45.865+0000 7fe39eed2600 0 bucket checkpoint complete 2022-09-12T19:11:45.885 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-22 2022-09-12T19:11:45.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:45.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:45.887 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:11:45.955 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:45.956 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:45.995 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:45.992+0000 7f2af55f7600 1 bucket sync caught up with source: 2022-09-12T19:11:45.995 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000002.14.5, , ] 2022-09-12T19:11:45.995 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000002.14.5, , ] 2022-09-12T19:11:45.996 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:45.992+0000 7f2af55f7600 0 bucket checkpoint complete 2022-09-12T19:11:46.008 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-22 2022-09-12T19:11:46.009 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:46.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:46.010 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:11:46.073 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:46.073 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:46.112 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:46.109+0000 7f9874d97600 1 bucket sync caught up with source: 2022-09-12T19:11:46.112 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000002.14.5, , ] 2022-09-12T19:11:46.112 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000002.14.5, , ] 2022-09-12T19:11:46.113 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:46.109+0000 7f9874d97600 0 bucket checkpoint complete 2022-09-12T19:11:46.124 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-22 2022-09-12T19:11:46.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:46.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:46.126 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:11:46.185 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:46.186 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:46.232 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:46.229+0000 7fc6c5e08600 1 bucket sync caught up with source: 2022-09-12T19:11:46.232 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000003.64.11, , ] 2022-09-12T19:11:46.233 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000003.64.11, , ] 2022-09-12T19:11:46.233 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:46.229+0000 7fc6c5e08600 0 bucket checkpoint complete 2022-09-12T19:11:46.250 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-22 2022-09-12T19:11:46.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:46.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:46.253 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:11:46.310 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:46.310 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:46.349 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:46.346+0000 7fde2585b600 1 bucket sync caught up with source: 2022-09-12T19:11:46.350 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000003.64.11, , ] 2022-09-12T19:11:46.350 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000003.64.11, , ] 2022-09-12T19:11:46.351 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:46.346+0000 7fde2585b600 0 bucket checkpoint complete 2022-09-12T19:11:46.364 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-22 2022-09-12T19:11:46.365 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-22 2022-09-12T19:11:46.365 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:46.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:46.367 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:11:46.428 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:46.428 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:46.474 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:46.468+0000 7f674db55600 1 bucket sync caught up with source: 2022-09-12T19:11:46.474 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000002.70.5, , ] 2022-09-12T19:11:46.474 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000002.70.5, , ] 2022-09-12T19:11:46.475 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:46.468+0000 7f674db55600 0 bucket checkpoint complete 2022-09-12T19:11:46.489 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-22 2022-09-12T19:11:46.490 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:46.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:46.491 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:11:46.550 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:46.551 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:46.597 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:46.594+0000 7f2bfa471600 1 bucket sync caught up with source: 2022-09-12T19:11:46.597 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000002.70.5, , ] 2022-09-12T19:11:46.597 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000002.70.5, , ] 2022-09-12T19:11:46.598 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:46.594+0000 7f2bfa471600 0 bucket checkpoint complete 2022-09-12T19:11:46.620 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-22 2022-09-12T19:11:46.621 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-22 zones={test-zone2, test-zone1} 2022-09-12T19:11:46.621 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:11:46.622 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:11:46.622 DEBUG:boto:Method: HEAD 2022-09-12T19:11:46.623 DEBUG:boto:Path: /jdpiak-22/ 2022-09-12T19:11:46.623 DEBUG:boto:Data: 2022-09-12T19:11:46.624 DEBUG:boto:Headers: {} 2022-09-12T19:11:46.624 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:46.625 DEBUG:boto:Port: 8000 2022-09-12T19:11:46.625 DEBUG:boto:Params: {} 2022-09-12T19:11:46.626 DEBUG:boto:Token: None 2022-09-12T19:11:46.627 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:46 GMT /jdpiak-22/ 2022-09-12T19:11:46.628 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Frddw2ufUki90ig83A56dQ+OWY8= 2022-09-12T19:11:46.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Frddw2ufUki90ig83A56dQ+OWY8=', 'Content-Length': '0'} 2022-09-12T19:11:46.631 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002bfcae960461e9c6-00631f8472-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:46.632 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:11:46.632 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:11:46.633 DEBUG:boto:Method: HEAD 2022-09-12T19:11:46.634 DEBUG:boto:Path: /jdpiak-22/ 2022-09-12T19:11:46.635 DEBUG:boto:Data: 2022-09-12T19:11:46.636 DEBUG:boto:Headers: {} 2022-09-12T19:11:46.637 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:46.638 DEBUG:boto:Port: 8000 2022-09-12T19:11:46.639 DEBUG:boto:Params: {} 2022-09-12T19:11:46.640 DEBUG:boto:Token: None 2022-09-12T19:11:46.641 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:46 GMT /jdpiak-22/ 2022-09-12T19:11:46.642 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Frddw2ufUki90ig83A56dQ+OWY8= 2022-09-12T19:11:46.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Frddw2ufUki90ig83A56dQ+OWY8=', 'Content-Length': '0'} 2022-09-12T19:11:46.646 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd29ad9501e341e2-00631f8472-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:46.647 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:11:46.647 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:11:46.648 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:11:46.649 DEBUG:boto:path=/jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:46.649 DEBUG:boto:auth_path=/jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:46.650 DEBUG:boto:Method: GET 2022-09-12T19:11:46.651 DEBUG:boto:Path: /jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:46.651 DEBUG:boto:Data: 2022-09-12T19:11:46.652 DEBUG:boto:Headers: {} 2022-09-12T19:11:46.652 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:46.653 DEBUG:boto:Port: 8000 2022-09-12T19:11:46.653 DEBUG:boto:Params: {} 2022-09-12T19:11:46.654 DEBUG:boto:Token: None 2022-09-12T19:11:46.655 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:46 GMT /jdpiak-22/?versions 2022-09-12T19:11:46.655 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l7S4u7ltcElf9UoTpXMiAT9fL+w= 2022-09-12T19:11:46.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l7S4u7ltcElf9UoTpXMiAT9fL+w=', 'Content-Length': '0'} 2022-09-12T19:11:46.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dade371c7a72f4ba-00631f8472-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:46.659 DEBUG:boto:b'jdpiak-22999falseobjC-GaJtcvvXQHXDwz1TiDh8pTDLyLIrXtrue2022-09-12T19:10:54.036Zrgw-multisite-test-userMultisite Test UserNormalobjfU7DRGj75cTxlyr6yy4TPugTgwe2QhYfalse2022-09-12T19:10:53.974Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:46.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:46.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:46.661 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:11:46.662 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:11:46.662 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:11:46.663 DEBUG:boto:path=/jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:46.664 DEBUG:boto:auth_path=/jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:46.665 DEBUG:boto:Method: GET 2022-09-12T19:11:46.666 DEBUG:boto:Path: /jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:46.667 DEBUG:boto:Data: 2022-09-12T19:11:46.668 DEBUG:boto:Headers: {} 2022-09-12T19:11:46.668 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:46.669 DEBUG:boto:Port: 8000 2022-09-12T19:11:46.670 DEBUG:boto:Params: {} 2022-09-12T19:11:46.671 DEBUG:boto:Token: None 2022-09-12T19:11:46.672 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:46 GMT /jdpiak-22/?versions 2022-09-12T19:11:46.672 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l7S4u7ltcElf9UoTpXMiAT9fL+w= 2022-09-12T19:11:46.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l7S4u7ltcElf9UoTpXMiAT9fL+w=', 'Content-Length': '0'} 2022-09-12T19:11:46.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de1ccaa5f008fa4d-00631f8472-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:46.677 DEBUG:boto:b'jdpiak-22999falseobjC-GaJtcvvXQHXDwz1TiDh8pTDLyLIrXtrue2022-09-12T19:10:54.036Zrgw-multisite-test-userMultisite Test UserNormalobjfU7DRGj75cTxlyr6yy4TPugTgwe2QhYfalse2022-09-12T19:10:53.974Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:46.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:46.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:46.679 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:11:46.679 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:11:46.680 DEBUG:boto:path=/jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:46.680 DEBUG:boto:auth_path=/jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:46.681 DEBUG:boto:Method: GET 2022-09-12T19:11:46.681 DEBUG:boto:Path: /jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:46.682 DEBUG:boto:Data: 2022-09-12T19:11:46.682 DEBUG:boto:Headers: {} 2022-09-12T19:11:46.683 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:46.683 DEBUG:boto:Port: 8000 2022-09-12T19:11:46.684 DEBUG:boto:Params: {} 2022-09-12T19:11:46.685 DEBUG:boto:Token: None 2022-09-12T19:11:46.685 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:46 GMT /jdpiak-22/?versions 2022-09-12T19:11:46.686 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l7S4u7ltcElf9UoTpXMiAT9fL+w= 2022-09-12T19:11:46.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l7S4u7ltcElf9UoTpXMiAT9fL+w=', 'Content-Length': '0'} 2022-09-12T19:11:46.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac5930aa75b20ec2-00631f8472-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:46.698 DEBUG:boto:b'jdpiak-22999falseobjC-GaJtcvvXQHXDwz1TiDh8pTDLyLIrXtrue2022-09-12T19:10:54.036Zrgw-multisite-test-userMultisite Test UserNormalobjfU7DRGj75cTxlyr6yy4TPugTgwe2QhYfalse2022-09-12T19:10:53.974Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:46.699 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:11:46.700 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:11:46.701 DEBUG:boto:path=/jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:46.701 DEBUG:boto:auth_path=/jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:46.702 DEBUG:boto:Method: GET 2022-09-12T19:11:46.703 DEBUG:boto:Path: /jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:46.703 DEBUG:boto:Data: 2022-09-12T19:11:46.704 DEBUG:boto:Headers: {} 2022-09-12T19:11:46.704 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:46.705 DEBUG:boto:Port: 8000 2022-09-12T19:11:46.705 DEBUG:boto:Params: {} 2022-09-12T19:11:46.706 DEBUG:boto:Token: None 2022-09-12T19:11:46.707 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:46 GMT /jdpiak-22/?versions 2022-09-12T19:11:46.707 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l7S4u7ltcElf9UoTpXMiAT9fL+w= 2022-09-12T19:11:46.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l7S4u7ltcElf9UoTpXMiAT9fL+w=', 'Content-Length': '0'} 2022-09-12T19:11:46.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033ace342e845b2ab-00631f8472-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:46.711 DEBUG:boto:b'jdpiak-22999falseobjC-GaJtcvvXQHXDwz1TiDh8pTDLyLIrXtrue2022-09-12T19:10:54.036Zrgw-multisite-test-userMultisite Test UserNormalobjfU7DRGj75cTxlyr6yy4TPugTgwe2QhYfalse2022-09-12T19:10:53.974Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:46.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:46.713 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:46.713 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:46.714 DEBUG:boto:Method: HEAD 2022-09-12T19:11:46.715 DEBUG:boto:Path: /jdpiak-22/obj 2022-09-12T19:11:46.715 DEBUG:boto:Data: 2022-09-12T19:11:46.716 DEBUG:boto:Headers: {} 2022-09-12T19:11:46.716 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:46.717 DEBUG:boto:Port: 8000 2022-09-12T19:11:46.717 DEBUG:boto:Params: {} 2022-09-12T19:11:46.718 DEBUG:boto:Token: None 2022-09-12T19:11:46.719 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:46 GMT /jdpiak-22/obj 2022-09-12T19:11:46.719 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:APb72KGVgCjIpGO2p+ywi4HN6GM= 2022-09-12T19:11:46.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:APb72KGVgCjIpGO2p+ywi4HN6GM=', 'Content-Length': '0'} 2022-09-12T19:11:46.723 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000f040d607a0477d3d-00631f8472-10e4-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:46.723 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:46.724 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:46.724 DEBUG:boto:Method: HEAD 2022-09-12T19:11:46.725 DEBUG:boto:Path: /jdpiak-22/obj 2022-09-12T19:11:46.725 DEBUG:boto:Data: 2022-09-12T19:11:46.726 DEBUG:boto:Headers: {} 2022-09-12T19:11:46.726 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:46.727 DEBUG:boto:Port: 8000 2022-09-12T19:11:46.727 DEBUG:boto:Params: {} 2022-09-12T19:11:46.728 DEBUG:boto:Token: None 2022-09-12T19:11:46.729 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:46 GMT /jdpiak-22/obj 2022-09-12T19:11:46.730 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:APb72KGVgCjIpGO2p+ywi4HN6GM= 2022-09-12T19:11:46.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:APb72KGVgCjIpGO2p+ywi4HN6GM=', 'Content-Length': '0'} 2022-09-12T19:11:46.734 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000047cb636dd4ad1663-00631f8472-10f6-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:46.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:46.736 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:46.737 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:46.737 DEBUG:boto:path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:46.738 DEBUG:boto:auth_path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:46.739 DEBUG:boto:Method: GET 2022-09-12T19:11:46.740 DEBUG:boto:Path: /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:46.741 DEBUG:boto:Data: 2022-09-12T19:11:46.742 DEBUG:boto:Headers: {} 2022-09-12T19:11:46.743 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:46.744 DEBUG:boto:Port: 8000 2022-09-12T19:11:46.745 DEBUG:boto:Params: {} 2022-09-12T19:11:46.746 DEBUG:boto:Token: None 2022-09-12T19:11:46.747 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:46 GMT /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:46.748 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vy5oRYb7cUCjDF390q4fhBHOtZI= 2022-09-12T19:11:46.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vy5oRYb7cUCjDF390q4fhBHOtZI=', 'Content-Length': '0'} 2022-09-12T19:11:46.751 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'fU7DRGj75cTxlyr6yy4TPugTgwe2QhY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c05787c1e3c89836-00631f8472-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:46.752 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:46.753 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:46.753 DEBUG:boto:path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:46.754 DEBUG:boto:auth_path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:46.754 DEBUG:boto:Method: GET 2022-09-12T19:11:46.755 DEBUG:boto:Path: /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:46.755 DEBUG:boto:Data: 2022-09-12T19:11:46.756 DEBUG:boto:Headers: {} 2022-09-12T19:11:46.757 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:46.757 DEBUG:boto:Port: 8000 2022-09-12T19:11:46.758 DEBUG:boto:Params: {} 2022-09-12T19:11:46.758 DEBUG:boto:Token: None 2022-09-12T19:11:46.759 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:46 GMT /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:46.760 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vy5oRYb7cUCjDF390q4fhBHOtZI= 2022-09-12T19:11:46.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vy5oRYb7cUCjDF390q4fhBHOtZI=', 'Content-Length': '0'} 2022-09-12T19:11:46.763 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'fU7DRGj75cTxlyr6yy4TPugTgwe2QhY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000825d5416f1042343-00631f8472-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:46.764 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:46.765 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:46.766 DEBUG:boto:path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:46.767 DEBUG:boto:auth_path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:46.768 DEBUG:boto:Method: HEAD 2022-09-12T19:11:46.769 DEBUG:boto:Path: /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:46.770 DEBUG:boto:Data: 2022-09-12T19:11:46.771 DEBUG:boto:Headers: {} 2022-09-12T19:11:46.772 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:46.773 DEBUG:boto:Port: 8000 2022-09-12T19:11:46.774 DEBUG:boto:Params: {} 2022-09-12T19:11:46.775 DEBUG:boto:Token: None 2022-09-12T19:11:46.776 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:46 GMT /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:46.777 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jxVyqO2JCkrsSrSGN+2TjZ08zwk= 2022-09-12T19:11:46.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jxVyqO2JCkrsSrSGN+2TjZ08zwk=', 'Content-Length': '0'} 2022-09-12T19:11:46.780 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'fU7DRGj75cTxlyr6yy4TPugTgwe2QhY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a76e6b4de6d6a77a-00631f8472-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:46.782 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:46.783 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:46.784 DEBUG:boto:path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:46.785 DEBUG:boto:auth_path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:46.785 DEBUG:boto:Method: HEAD 2022-09-12T19:11:46.786 DEBUG:boto:Path: /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:46.787 DEBUG:boto:Data: 2022-09-12T19:11:46.788 DEBUG:boto:Headers: {} 2022-09-12T19:11:46.789 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:46.790 DEBUG:boto:Port: 8000 2022-09-12T19:11:46.791 DEBUG:boto:Params: {} 2022-09-12T19:11:46.792 DEBUG:boto:Token: None 2022-09-12T19:11:46.793 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:46 GMT /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:46.794 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jxVyqO2JCkrsSrSGN+2TjZ08zwk= 2022-09-12T19:11:46.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jxVyqO2JCkrsSrSGN+2TjZ08zwk=', 'Content-Length': '0'} 2022-09-12T19:11:46.798 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'fU7DRGj75cTxlyr6yy4TPugTgwe2QhY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b1eb21393ffacacf-00631f8472-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:46.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:46.800 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:46.801 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:46.802 DEBUG:boto:path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:46.803 DEBUG:boto:auth_path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:46.804 DEBUG:boto:Method: GET 2022-09-12T19:11:46.805 DEBUG:boto:Path: /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:46.806 DEBUG:boto:Data: 2022-09-12T19:11:46.807 DEBUG:boto:Headers: {} 2022-09-12T19:11:46.808 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:46.809 DEBUG:boto:Port: 8000 2022-09-12T19:11:46.810 DEBUG:boto:Params: {} 2022-09-12T19:11:46.812 DEBUG:boto:Token: None 2022-09-12T19:11:46.813 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:46 GMT /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:46.814 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vy5oRYb7cUCjDF390q4fhBHOtZI= 2022-09-12T19:11:46.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vy5oRYb7cUCjDF390q4fhBHOtZI=', 'Content-Length': '0'} 2022-09-12T19:11:46.817 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'fU7DRGj75cTxlyr6yy4TPugTgwe2QhY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006e9d34c2eb72c312-00631f8472-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:46.818 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:46.819 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:46.819 DEBUG:boto:path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:46.820 DEBUG:boto:auth_path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:46.820 DEBUG:boto:Method: GET 2022-09-12T19:11:46.821 DEBUG:boto:Path: /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:46.821 DEBUG:boto:Data: 2022-09-12T19:11:46.822 DEBUG:boto:Headers: {} 2022-09-12T19:11:46.823 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:46.823 DEBUG:boto:Port: 8000 2022-09-12T19:11:46.824 DEBUG:boto:Params: {} 2022-09-12T19:11:46.824 DEBUG:boto:Token: None 2022-09-12T19:11:46.825 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:46 GMT /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:46.826 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vy5oRYb7cUCjDF390q4fhBHOtZI= 2022-09-12T19:11:46.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vy5oRYb7cUCjDF390q4fhBHOtZI=', 'Content-Length': '0'} 2022-09-12T19:11:46.828 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'fU7DRGj75cTxlyr6yy4TPugTgwe2QhY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005435056bf91b3230-00631f8472-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:46.829 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:11:46.829 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:11:46.830 DEBUG:boto:path=/jdpiak-22/?prefix=obj 2022-09-12T19:11:46.830 DEBUG:boto:auth_path=/jdpiak-22/?prefix=obj 2022-09-12T19:11:46.831 DEBUG:boto:Method: GET 2022-09-12T19:11:46.832 DEBUG:boto:Path: /jdpiak-22/?prefix=obj 2022-09-12T19:11:46.832 DEBUG:boto:Data: 2022-09-12T19:11:46.833 DEBUG:boto:Headers: {} 2022-09-12T19:11:46.833 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:46.834 DEBUG:boto:Port: 8000 2022-09-12T19:11:46.834 DEBUG:boto:Params: {} 2022-09-12T19:11:46.835 DEBUG:boto:Token: None 2022-09-12T19:11:46.835 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:46 GMT /jdpiak-22/ 2022-09-12T19:11:46.836 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V1RMTDu3wW5zPbsPeMJxhgvz8mM= 2022-09-12T19:11:46.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V1RMTDu3wW5zPbsPeMJxhgvz8mM=', 'Content-Length': '0'} 2022-09-12T19:11:46.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b71d2ee05d6f3769-00631f8472-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:46.886 DEBUG:boto:b'jdpiak-22obj1000false' 2022-09-12T19:11:46.887 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:11:46.888 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:11:46.888 DEBUG:boto:path=/jdpiak-22/?prefix=obj 2022-09-12T19:11:46.889 DEBUG:boto:auth_path=/jdpiak-22/?prefix=obj 2022-09-12T19:11:46.890 DEBUG:boto:Method: GET 2022-09-12T19:11:46.890 DEBUG:boto:Path: /jdpiak-22/?prefix=obj 2022-09-12T19:11:46.891 DEBUG:boto:Data: 2022-09-12T19:11:46.891 DEBUG:boto:Headers: {} 2022-09-12T19:11:46.892 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:46.892 DEBUG:boto:Port: 8000 2022-09-12T19:11:46.893 DEBUG:boto:Params: {} 2022-09-12T19:11:46.893 DEBUG:boto:Token: None 2022-09-12T19:11:46.894 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:46 GMT /jdpiak-22/ 2022-09-12T19:11:46.895 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V1RMTDu3wW5zPbsPeMJxhgvz8mM= 2022-09-12T19:11:46.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V1RMTDu3wW5zPbsPeMJxhgvz8mM=', 'Content-Length': '0'} 2022-09-12T19:11:46.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000224f6acb8e9ba7d0-00631f8472-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:46.899 DEBUG:boto:b'jdpiak-22obj1000false' 2022-09-12T19:11:46.899 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:46.900 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:46.900 DEBUG:boto:Method: HEAD 2022-09-12T19:11:46.901 DEBUG:boto:Path: /jdpiak-22/obj 2022-09-12T19:11:46.902 DEBUG:boto:Data: 2022-09-12T19:11:46.902 DEBUG:boto:Headers: {} 2022-09-12T19:11:46.903 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:46.903 DEBUG:boto:Port: 8000 2022-09-12T19:11:46.904 DEBUG:boto:Params: {} 2022-09-12T19:11:46.904 DEBUG:boto:Token: None 2022-09-12T19:11:46.905 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:46 GMT /jdpiak-22/obj 2022-09-12T19:11:46.905 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:APb72KGVgCjIpGO2p+ywi4HN6GM= 2022-09-12T19:11:46.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:APb72KGVgCjIpGO2p+ywi4HN6GM=', 'Content-Length': '0'} 2022-09-12T19:11:46.908 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000699101c6b7d4cb67-00631f8472-10e4-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:46.909 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:46.909 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:46.910 DEBUG:boto:Method: HEAD 2022-09-12T19:11:46.911 DEBUG:boto:Path: /jdpiak-22/obj 2022-09-12T19:11:46.911 DEBUG:boto:Data: 2022-09-12T19:11:46.911 DEBUG:boto:Headers: {} 2022-09-12T19:11:46.912 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:46.912 DEBUG:boto:Port: 8000 2022-09-12T19:11:46.913 DEBUG:boto:Params: {} 2022-09-12T19:11:46.913 DEBUG:boto:Token: None 2022-09-12T19:11:46.914 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:46 GMT /jdpiak-22/obj 2022-09-12T19:11:46.915 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:APb72KGVgCjIpGO2p+ywi4HN6GM= 2022-09-12T19:11:46.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:APb72KGVgCjIpGO2p+ywi4HN6GM=', 'Content-Length': '0'} 2022-09-12T19:11:46.917 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000ad9c836ac6803b8d-00631f8472-10f6-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:46.919 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-22 zones={test-zone2, test-zone1} 2022-09-12T19:11:46.920 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-22 zones={test-zone3, test-zone1} 2022-09-12T19:11:46.920 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:11:46.921 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:11:46.921 DEBUG:boto:Method: HEAD 2022-09-12T19:11:46.922 DEBUG:boto:Path: /jdpiak-22/ 2022-09-12T19:11:46.923 DEBUG:boto:Data: 2022-09-12T19:11:46.924 DEBUG:boto:Headers: {} 2022-09-12T19:11:46.924 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:46.925 DEBUG:boto:Port: 8001 2022-09-12T19:11:46.926 DEBUG:boto:Params: {} 2022-09-12T19:11:46.927 DEBUG:boto:Token: None 2022-09-12T19:11:46.928 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:46 GMT /jdpiak-22/ 2022-09-12T19:11:46.929 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Frddw2ufUki90ig83A56dQ+OWY8= 2022-09-12T19:11:46.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Frddw2ufUki90ig83A56dQ+OWY8=', 'Content-Length': '0'} 2022-09-12T19:11:46.933 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035834883a60efab2-00631f8472-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:46.934 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:11:46.935 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:11:46.936 DEBUG:boto:Method: HEAD 2022-09-12T19:11:46.936 DEBUG:boto:Path: /jdpiak-22/ 2022-09-12T19:11:46.937 DEBUG:boto:Data: 2022-09-12T19:11:46.938 DEBUG:boto:Headers: {} 2022-09-12T19:11:46.939 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:46.940 DEBUG:boto:Port: 8000 2022-09-12T19:11:46.941 DEBUG:boto:Params: {} 2022-09-12T19:11:46.942 DEBUG:boto:Token: None 2022-09-12T19:11:46.942 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:46 GMT /jdpiak-22/ 2022-09-12T19:11:46.943 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Frddw2ufUki90ig83A56dQ+OWY8= 2022-09-12T19:11:46.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Frddw2ufUki90ig83A56dQ+OWY8=', 'Content-Length': '0'} 2022-09-12T19:11:46.947 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008078ca41f26c4ac0-00631f8472-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:46.948 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:11:46.949 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:11:46.949 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:11:46.950 DEBUG:boto:path=/jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:46.950 DEBUG:boto:auth_path=/jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:46.951 DEBUG:boto:Method: GET 2022-09-12T19:11:46.951 DEBUG:boto:Path: /jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:46.952 DEBUG:boto:Data: 2022-09-12T19:11:46.952 DEBUG:boto:Headers: {} 2022-09-12T19:11:46.953 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:46.953 DEBUG:boto:Port: 8001 2022-09-12T19:11:46.954 DEBUG:boto:Params: {} 2022-09-12T19:11:46.955 DEBUG:boto:Token: None 2022-09-12T19:11:46.955 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:46 GMT /jdpiak-22/?versions 2022-09-12T19:11:46.956 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l7S4u7ltcElf9UoTpXMiAT9fL+w= 2022-09-12T19:11:46.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l7S4u7ltcElf9UoTpXMiAT9fL+w=', 'Content-Length': '0'} 2022-09-12T19:11:46.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bca795acca0b99b5-00631f8472-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:46.959 DEBUG:boto:b'jdpiak-22999falseobjC-GaJtcvvXQHXDwz1TiDh8pTDLyLIrXtrue2022-09-12T19:10:54.036Zrgw-multisite-test-userMultisite Test UserNormalobjfU7DRGj75cTxlyr6yy4TPugTgwe2QhYfalse2022-09-12T19:10:53.974Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:46.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:46.961 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:46.961 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:11:46.962 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:11:46.962 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:11:46.963 DEBUG:boto:path=/jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:46.963 DEBUG:boto:auth_path=/jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:46.964 DEBUG:boto:Method: GET 2022-09-12T19:11:46.964 DEBUG:boto:Path: /jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:46.965 DEBUG:boto:Data: 2022-09-12T19:11:46.965 DEBUG:boto:Headers: {} 2022-09-12T19:11:46.966 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:46.966 DEBUG:boto:Port: 8000 2022-09-12T19:11:46.967 DEBUG:boto:Params: {} 2022-09-12T19:11:46.967 DEBUG:boto:Token: None 2022-09-12T19:11:46.968 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:46 GMT /jdpiak-22/?versions 2022-09-12T19:11:46.969 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l7S4u7ltcElf9UoTpXMiAT9fL+w= 2022-09-12T19:11:46.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l7S4u7ltcElf9UoTpXMiAT9fL+w=', 'Content-Length': '0'} 2022-09-12T19:11:46.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f741bf799e005560-00631f8472-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:46.973 DEBUG:boto:b'jdpiak-22999falseobjC-GaJtcvvXQHXDwz1TiDh8pTDLyLIrXtrue2022-09-12T19:10:54.036Zrgw-multisite-test-userMultisite Test UserNormalobjfU7DRGj75cTxlyr6yy4TPugTgwe2QhYfalse2022-09-12T19:10:53.974Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:46.974 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:46.974 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:46.975 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:11:46.975 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:11:46.976 DEBUG:boto:path=/jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:46.976 DEBUG:boto:auth_path=/jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:46.977 DEBUG:boto:Method: GET 2022-09-12T19:11:46.977 DEBUG:boto:Path: /jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:46.978 DEBUG:boto:Data: 2022-09-12T19:11:46.978 DEBUG:boto:Headers: {} 2022-09-12T19:11:46.979 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:46.979 DEBUG:boto:Port: 8001 2022-09-12T19:11:46.980 DEBUG:boto:Params: {} 2022-09-12T19:11:46.981 DEBUG:boto:Token: None 2022-09-12T19:11:46.981 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:46 GMT /jdpiak-22/?versions 2022-09-12T19:11:46.982 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l7S4u7ltcElf9UoTpXMiAT9fL+w= 2022-09-12T19:11:46.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l7S4u7ltcElf9UoTpXMiAT9fL+w=', 'Content-Length': '0'} 2022-09-12T19:11:46.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008553a8b85989905-00631f8472-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:46.986 DEBUG:boto:b'jdpiak-22999falseobjC-GaJtcvvXQHXDwz1TiDh8pTDLyLIrXtrue2022-09-12T19:10:54.036Zrgw-multisite-test-userMultisite Test UserNormalobjfU7DRGj75cTxlyr6yy4TPugTgwe2QhYfalse2022-09-12T19:10:53.974Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:46.986 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:11:46.987 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:11:46.987 DEBUG:boto:path=/jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:46.988 DEBUG:boto:auth_path=/jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:46.988 DEBUG:boto:Method: GET 2022-09-12T19:11:46.989 DEBUG:boto:Path: /jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:46.989 DEBUG:boto:Data: 2022-09-12T19:11:46.990 DEBUG:boto:Headers: {} 2022-09-12T19:11:46.990 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:46.991 DEBUG:boto:Port: 8000 2022-09-12T19:11:46.991 DEBUG:boto:Params: {} 2022-09-12T19:11:46.992 DEBUG:boto:Token: None 2022-09-12T19:11:46.993 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:46 GMT /jdpiak-22/?versions 2022-09-12T19:11:46.993 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l7S4u7ltcElf9UoTpXMiAT9fL+w= 2022-09-12T19:11:46.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l7S4u7ltcElf9UoTpXMiAT9fL+w=', 'Content-Length': '0'} 2022-09-12T19:11:46.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017bddc51049f7c6a-00631f8472-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:46.997 DEBUG:boto:b'jdpiak-22999falseobjC-GaJtcvvXQHXDwz1TiDh8pTDLyLIrXtrue2022-09-12T19:10:54.036Zrgw-multisite-test-userMultisite Test UserNormalobjfU7DRGj75cTxlyr6yy4TPugTgwe2QhYfalse2022-09-12T19:10:53.974Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:46.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:46.999 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:47.000 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:47.001 DEBUG:boto:Method: HEAD 2022-09-12T19:11:47.001 DEBUG:boto:Path: /jdpiak-22/obj 2022-09-12T19:11:47.002 DEBUG:boto:Data: 2022-09-12T19:11:47.003 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.003 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:47.004 DEBUG:boto:Port: 8001 2022-09-12T19:11:47.004 DEBUG:boto:Params: {} 2022-09-12T19:11:47.005 DEBUG:boto:Token: None 2022-09-12T19:11:47.005 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/obj 2022-09-12T19:11:47.006 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zFlnHlELH2wu/bDMeUjcK6fy49Y= 2022-09-12T19:11:47.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zFlnHlELH2wu/bDMeUjcK6fy49Y=', 'Content-Length': '0'} 2022-09-12T19:11:47.009 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000da56578ecbc6ecc8-00631f8473-10f2-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.010 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:47.011 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:47.012 DEBUG:boto:Method: HEAD 2022-09-12T19:11:47.012 DEBUG:boto:Path: /jdpiak-22/obj 2022-09-12T19:11:47.013 DEBUG:boto:Data: 2022-09-12T19:11:47.014 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.015 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:47.016 DEBUG:boto:Port: 8000 2022-09-12T19:11:47.016 DEBUG:boto:Params: {} 2022-09-12T19:11:47.017 DEBUG:boto:Token: None 2022-09-12T19:11:47.017 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/obj 2022-09-12T19:11:47.018 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zFlnHlELH2wu/bDMeUjcK6fy49Y= 2022-09-12T19:11:47.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zFlnHlELH2wu/bDMeUjcK6fy49Y=', 'Content-Length': '0'} 2022-09-12T19:11:47.021 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000085450e275c366083-00631f8473-10f6-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:47.022 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:47.023 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:47.023 DEBUG:boto:path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.024 DEBUG:boto:auth_path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.024 DEBUG:boto:Method: GET 2022-09-12T19:11:47.025 DEBUG:boto:Path: /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.025 DEBUG:boto:Data: 2022-09-12T19:11:47.026 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.027 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:47.028 DEBUG:boto:Port: 8001 2022-09-12T19:11:47.029 DEBUG:boto:Params: {} 2022-09-12T19:11:47.030 DEBUG:boto:Token: None 2022-09-12T19:11:47.031 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.032 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9R7OZgPGIAcorQWveP1yJRJNBdM= 2022-09-12T19:11:47.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9R7OZgPGIAcorQWveP1yJRJNBdM=', 'Content-Length': '0'} 2022-09-12T19:11:47.036 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'fU7DRGj75cTxlyr6yy4TPugTgwe2QhY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ec23c7d0919b381f-00631f8473-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.037 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:47.038 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:47.039 DEBUG:boto:path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.040 DEBUG:boto:auth_path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.041 DEBUG:boto:Method: GET 2022-09-12T19:11:47.042 DEBUG:boto:Path: /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.043 DEBUG:boto:Data: 2022-09-12T19:11:47.044 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.045 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:47.045 DEBUG:boto:Port: 8000 2022-09-12T19:11:47.046 DEBUG:boto:Params: {} 2022-09-12T19:11:47.047 DEBUG:boto:Token: None 2022-09-12T19:11:47.049 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.049 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9R7OZgPGIAcorQWveP1yJRJNBdM= 2022-09-12T19:11:47.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9R7OZgPGIAcorQWveP1yJRJNBdM=', 'Content-Length': '0'} 2022-09-12T19:11:47.052 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'fU7DRGj75cTxlyr6yy4TPugTgwe2QhY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d20b46ddfc897e1c-00631f8473-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.052 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:47.053 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:47.053 DEBUG:boto:path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.054 DEBUG:boto:auth_path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.054 DEBUG:boto:Method: HEAD 2022-09-12T19:11:47.055 DEBUG:boto:Path: /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.055 DEBUG:boto:Data: 2022-09-12T19:11:47.056 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.056 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:47.057 DEBUG:boto:Port: 8001 2022-09-12T19:11:47.057 DEBUG:boto:Params: {} 2022-09-12T19:11:47.058 DEBUG:boto:Token: None 2022-09-12T19:11:47.058 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.059 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:olzindqOpL2uXLco+a5K0Or0mYM= 2022-09-12T19:11:47.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:olzindqOpL2uXLco+a5K0Or0mYM=', 'Content-Length': '0'} 2022-09-12T19:11:47.062 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'fU7DRGj75cTxlyr6yy4TPugTgwe2QhY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004d289756132e1699-00631f8473-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.062 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:47.063 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:47.064 DEBUG:boto:path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.064 DEBUG:boto:auth_path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.065 DEBUG:boto:Method: HEAD 2022-09-12T19:11:47.065 DEBUG:boto:Path: /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.066 DEBUG:boto:Data: 2022-09-12T19:11:47.067 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.067 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:47.068 DEBUG:boto:Port: 8000 2022-09-12T19:11:47.068 DEBUG:boto:Params: {} 2022-09-12T19:11:47.069 DEBUG:boto:Token: None 2022-09-12T19:11:47.069 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.070 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:olzindqOpL2uXLco+a5K0Or0mYM= 2022-09-12T19:11:47.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:olzindqOpL2uXLco+a5K0Or0mYM=', 'Content-Length': '0'} 2022-09-12T19:11:47.072 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'fU7DRGj75cTxlyr6yy4TPugTgwe2QhY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c77dbc6101c17427-00631f8473-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:47.074 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:47.074 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:47.075 DEBUG:boto:path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.075 DEBUG:boto:auth_path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.076 DEBUG:boto:Method: GET 2022-09-12T19:11:47.076 DEBUG:boto:Path: /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.077 DEBUG:boto:Data: 2022-09-12T19:11:47.078 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.078 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:47.079 DEBUG:boto:Port: 8001 2022-09-12T19:11:47.079 DEBUG:boto:Params: {} 2022-09-12T19:11:47.080 DEBUG:boto:Token: None 2022-09-12T19:11:47.080 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.081 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9R7OZgPGIAcorQWveP1yJRJNBdM= 2022-09-12T19:11:47.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9R7OZgPGIAcorQWveP1yJRJNBdM=', 'Content-Length': '0'} 2022-09-12T19:11:47.084 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'fU7DRGj75cTxlyr6yy4TPugTgwe2QhY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000024dee18335d3e1fb-00631f8473-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.084 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:47.085 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:47.086 DEBUG:boto:path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.086 DEBUG:boto:auth_path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.087 DEBUG:boto:Method: GET 2022-09-12T19:11:47.087 DEBUG:boto:Path: /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.088 DEBUG:boto:Data: 2022-09-12T19:11:47.088 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.089 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:47.089 DEBUG:boto:Port: 8000 2022-09-12T19:11:47.090 DEBUG:boto:Params: {} 2022-09-12T19:11:47.091 DEBUG:boto:Token: None 2022-09-12T19:11:47.091 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.092 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9R7OZgPGIAcorQWveP1yJRJNBdM= 2022-09-12T19:11:47.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9R7OZgPGIAcorQWveP1yJRJNBdM=', 'Content-Length': '0'} 2022-09-12T19:11:47.095 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'fU7DRGj75cTxlyr6yy4TPugTgwe2QhY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009cfd6d7c904b09b5-00631f8473-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.096 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:11:47.096 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:11:47.097 DEBUG:boto:path=/jdpiak-22/?prefix=obj 2022-09-12T19:11:47.097 DEBUG:boto:auth_path=/jdpiak-22/?prefix=obj 2022-09-12T19:11:47.098 DEBUG:boto:Method: GET 2022-09-12T19:11:47.098 DEBUG:boto:Path: /jdpiak-22/?prefix=obj 2022-09-12T19:11:47.099 DEBUG:boto:Data: 2022-09-12T19:11:47.099 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.100 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:47.100 DEBUG:boto:Port: 8001 2022-09-12T19:11:47.101 DEBUG:boto:Params: {} 2022-09-12T19:11:47.102 DEBUG:boto:Token: None 2022-09-12T19:11:47.102 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/ 2022-09-12T19:11:47.103 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cK2qCA+LoRv2VuR+oHadis/UnJM= 2022-09-12T19:11:47.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cK2qCA+LoRv2VuR+oHadis/UnJM=', 'Content-Length': '0'} 2022-09-12T19:11:47.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000664dfd477b2b922-00631f8473-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.107 DEBUG:boto:b'jdpiak-22obj1000false' 2022-09-12T19:11:47.107 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:11:47.108 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:11:47.108 DEBUG:boto:path=/jdpiak-22/?prefix=obj 2022-09-12T19:11:47.109 DEBUG:boto:auth_path=/jdpiak-22/?prefix=obj 2022-09-12T19:11:47.110 DEBUG:boto:Method: GET 2022-09-12T19:11:47.110 DEBUG:boto:Path: /jdpiak-22/?prefix=obj 2022-09-12T19:11:47.111 DEBUG:boto:Data: 2022-09-12T19:11:47.111 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.112 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:47.112 DEBUG:boto:Port: 8000 2022-09-12T19:11:47.113 DEBUG:boto:Params: {} 2022-09-12T19:11:47.113 DEBUG:boto:Token: None 2022-09-12T19:11:47.114 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/ 2022-09-12T19:11:47.114 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cK2qCA+LoRv2VuR+oHadis/UnJM= 2022-09-12T19:11:47.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cK2qCA+LoRv2VuR+oHadis/UnJM=', 'Content-Length': '0'} 2022-09-12T19:11:47.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038a9e2d950319cca-00631f8473-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.162 DEBUG:boto:b'jdpiak-22obj1000false' 2022-09-12T19:11:47.164 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:47.164 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:47.165 DEBUG:boto:Method: HEAD 2022-09-12T19:11:47.166 DEBUG:boto:Path: /jdpiak-22/obj 2022-09-12T19:11:47.167 DEBUG:boto:Data: 2022-09-12T19:11:47.168 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.169 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:47.170 DEBUG:boto:Port: 8001 2022-09-12T19:11:47.171 DEBUG:boto:Params: {} 2022-09-12T19:11:47.172 DEBUG:boto:Token: None 2022-09-12T19:11:47.173 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/obj 2022-09-12T19:11:47.174 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zFlnHlELH2wu/bDMeUjcK6fy49Y= 2022-09-12T19:11:47.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zFlnHlELH2wu/bDMeUjcK6fy49Y=', 'Content-Length': '0'} 2022-09-12T19:11:47.178 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000019b3c8226d61dd7c-00631f8473-10f2-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.179 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:47.180 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:47.181 DEBUG:boto:Method: HEAD 2022-09-12T19:11:47.181 DEBUG:boto:Path: /jdpiak-22/obj 2022-09-12T19:11:47.182 DEBUG:boto:Data: 2022-09-12T19:11:47.183 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.184 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:47.185 DEBUG:boto:Port: 8000 2022-09-12T19:11:47.186 DEBUG:boto:Params: {} 2022-09-12T19:11:47.187 DEBUG:boto:Token: None 2022-09-12T19:11:47.188 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/obj 2022-09-12T19:11:47.189 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zFlnHlELH2wu/bDMeUjcK6fy49Y= 2022-09-12T19:11:47.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zFlnHlELH2wu/bDMeUjcK6fy49Y=', 'Content-Length': '0'} 2022-09-12T19:11:47.192 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000c1e8b7b2a348c87c-00631f8473-10f6-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.193 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-22 zones={test-zone3, test-zone1} 2022-09-12T19:11:47.194 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-22 zones={test-zone3, test-zone2} 2022-09-12T19:11:47.195 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:11:47.196 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:11:47.197 DEBUG:boto:Method: HEAD 2022-09-12T19:11:47.198 DEBUG:boto:Path: /jdpiak-22/ 2022-09-12T19:11:47.199 DEBUG:boto:Data: 2022-09-12T19:11:47.199 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.200 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:47.201 DEBUG:boto:Port: 8001 2022-09-12T19:11:47.202 DEBUG:boto:Params: {} 2022-09-12T19:11:47.203 DEBUG:boto:Token: None 2022-09-12T19:11:47.204 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/ 2022-09-12T19:11:47.205 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8dr+Hq5ZQChOW3zOTBXNgYjJ9VM= 2022-09-12T19:11:47.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8dr+Hq5ZQChOW3zOTBXNgYjJ9VM=', 'Content-Length': '0'} 2022-09-12T19:11:47.209 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e749d70da47acbc-00631f8473-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.210 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:11:47.211 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:11:47.212 DEBUG:boto:Method: HEAD 2022-09-12T19:11:47.213 DEBUG:boto:Path: /jdpiak-22/ 2022-09-12T19:11:47.214 DEBUG:boto:Data: 2022-09-12T19:11:47.215 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.215 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:47.216 DEBUG:boto:Port: 8000 2022-09-12T19:11:47.216 DEBUG:boto:Params: {} 2022-09-12T19:11:47.217 DEBUG:boto:Token: None 2022-09-12T19:11:47.217 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/ 2022-09-12T19:11:47.218 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8dr+Hq5ZQChOW3zOTBXNgYjJ9VM= 2022-09-12T19:11:47.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8dr+Hq5ZQChOW3zOTBXNgYjJ9VM=', 'Content-Length': '0'} 2022-09-12T19:11:47.221 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035ce301890372b11-00631f8473-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.222 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:11:47.222 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:11:47.223 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:11:47.224 DEBUG:boto:path=/jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:47.224 DEBUG:boto:auth_path=/jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:47.225 DEBUG:boto:Method: GET 2022-09-12T19:11:47.225 DEBUG:boto:Path: /jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:47.226 DEBUG:boto:Data: 2022-09-12T19:11:47.226 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.227 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:47.227 DEBUG:boto:Port: 8001 2022-09-12T19:11:47.228 DEBUG:boto:Params: {} 2022-09-12T19:11:47.228 DEBUG:boto:Token: None 2022-09-12T19:11:47.229 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/?versions 2022-09-12T19:11:47.229 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A5/T9p683vWkzMjBw61fL5orQvc= 2022-09-12T19:11:47.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A5/T9p683vWkzMjBw61fL5orQvc=', 'Content-Length': '0'} 2022-09-12T19:11:47.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063c64ed5d1abb8cc-00631f8473-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.233 DEBUG:boto:b'jdpiak-22999falseobjC-GaJtcvvXQHXDwz1TiDh8pTDLyLIrXtrue2022-09-12T19:10:54.036Zrgw-multisite-test-userMultisite Test UserNormalobjfU7DRGj75cTxlyr6yy4TPugTgwe2QhYfalse2022-09-12T19:10:53.974Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:47.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:47.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:47.235 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:11:47.235 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:11:47.236 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:11:47.236 DEBUG:boto:path=/jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:47.238 DEBUG:boto:auth_path=/jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:47.238 DEBUG:boto:Method: GET 2022-09-12T19:11:47.239 DEBUG:boto:Path: /jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:47.239 DEBUG:boto:Data: 2022-09-12T19:11:47.240 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.240 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:47.241 DEBUG:boto:Port: 8000 2022-09-12T19:11:47.241 DEBUG:boto:Params: {} 2022-09-12T19:11:47.242 DEBUG:boto:Token: None 2022-09-12T19:11:47.242 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/?versions 2022-09-12T19:11:47.243 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A5/T9p683vWkzMjBw61fL5orQvc= 2022-09-12T19:11:47.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A5/T9p683vWkzMjBw61fL5orQvc=', 'Content-Length': '0'} 2022-09-12T19:11:47.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f876f847b3746dcf-00631f8473-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.247 DEBUG:boto:b'jdpiak-22999falseobjC-GaJtcvvXQHXDwz1TiDh8pTDLyLIrXtrue2022-09-12T19:10:54.036Zrgw-multisite-test-userMultisite Test UserNormalobjfU7DRGj75cTxlyr6yy4TPugTgwe2QhYfalse2022-09-12T19:10:53.974Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:47.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:47.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:47.248 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:11:47.249 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:11:47.249 DEBUG:boto:path=/jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:47.250 DEBUG:boto:auth_path=/jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:47.250 DEBUG:boto:Method: GET 2022-09-12T19:11:47.251 DEBUG:boto:Path: /jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:47.251 DEBUG:boto:Data: 2022-09-12T19:11:47.252 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.252 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:47.253 DEBUG:boto:Port: 8001 2022-09-12T19:11:47.253 DEBUG:boto:Params: {} 2022-09-12T19:11:47.254 DEBUG:boto:Token: None 2022-09-12T19:11:47.255 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/?versions 2022-09-12T19:11:47.255 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A5/T9p683vWkzMjBw61fL5orQvc= 2022-09-12T19:11:47.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A5/T9p683vWkzMjBw61fL5orQvc=', 'Content-Length': '0'} 2022-09-12T19:11:47.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c63a45211e3b3a5b-00631f8473-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.259 DEBUG:boto:b'jdpiak-22999falseobjC-GaJtcvvXQHXDwz1TiDh8pTDLyLIrXtrue2022-09-12T19:10:54.036Zrgw-multisite-test-userMultisite Test UserNormalobjfU7DRGj75cTxlyr6yy4TPugTgwe2QhYfalse2022-09-12T19:10:53.974Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:47.260 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:11:47.260 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:11:47.261 DEBUG:boto:path=/jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:47.261 DEBUG:boto:auth_path=/jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:47.262 DEBUG:boto:Method: GET 2022-09-12T19:11:47.262 DEBUG:boto:Path: /jdpiak-22/?versions&max-keys=999 2022-09-12T19:11:47.263 DEBUG:boto:Data: 2022-09-12T19:11:47.263 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.264 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:47.264 DEBUG:boto:Port: 8000 2022-09-12T19:11:47.265 DEBUG:boto:Params: {} 2022-09-12T19:11:47.265 DEBUG:boto:Token: None 2022-09-12T19:11:47.266 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/?versions 2022-09-12T19:11:47.267 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A5/T9p683vWkzMjBw61fL5orQvc= 2022-09-12T19:11:47.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A5/T9p683vWkzMjBw61fL5orQvc=', 'Content-Length': '0'} 2022-09-12T19:11:47.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cdf7e0af61bd882-00631f8473-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.270 DEBUG:boto:b'jdpiak-22999falseobjC-GaJtcvvXQHXDwz1TiDh8pTDLyLIrXtrue2022-09-12T19:10:54.036Zrgw-multisite-test-userMultisite Test UserNormalobjfU7DRGj75cTxlyr6yy4TPugTgwe2QhYfalse2022-09-12T19:10:53.974Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:47.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:47.272 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:47.272 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:47.273 DEBUG:boto:Method: HEAD 2022-09-12T19:11:47.273 DEBUG:boto:Path: /jdpiak-22/obj 2022-09-12T19:11:47.274 DEBUG:boto:Data: 2022-09-12T19:11:47.274 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.275 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:47.275 DEBUG:boto:Port: 8001 2022-09-12T19:11:47.276 DEBUG:boto:Params: {} 2022-09-12T19:11:47.276 DEBUG:boto:Token: None 2022-09-12T19:11:47.277 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/obj 2022-09-12T19:11:47.278 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zFlnHlELH2wu/bDMeUjcK6fy49Y= 2022-09-12T19:11:47.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zFlnHlELH2wu/bDMeUjcK6fy49Y=', 'Content-Length': '0'} 2022-09-12T19:11:47.281 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000e389ad30b96f9d0c-00631f8473-10f2-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.281 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:47.282 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:47.282 DEBUG:boto:Method: HEAD 2022-09-12T19:11:47.283 DEBUG:boto:Path: /jdpiak-22/obj 2022-09-12T19:11:47.283 DEBUG:boto:Data: 2022-09-12T19:11:47.284 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.284 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:47.285 DEBUG:boto:Port: 8000 2022-09-12T19:11:47.285 DEBUG:boto:Params: {} 2022-09-12T19:11:47.286 DEBUG:boto:Token: None 2022-09-12T19:11:47.286 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/obj 2022-09-12T19:11:47.287 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zFlnHlELH2wu/bDMeUjcK6fy49Y= 2022-09-12T19:11:47.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zFlnHlELH2wu/bDMeUjcK6fy49Y=', 'Content-Length': '0'} 2022-09-12T19:11:47.290 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000e84169b69c04b29f-00631f8473-10e4-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:47.291 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:47.292 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:47.292 DEBUG:boto:path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.293 DEBUG:boto:auth_path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.293 DEBUG:boto:Method: GET 2022-09-12T19:11:47.294 DEBUG:boto:Path: /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.294 DEBUG:boto:Data: 2022-09-12T19:11:47.295 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.295 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:47.296 DEBUG:boto:Port: 8001 2022-09-12T19:11:47.296 DEBUG:boto:Params: {} 2022-09-12T19:11:47.297 DEBUG:boto:Token: None 2022-09-12T19:11:47.298 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.298 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9R7OZgPGIAcorQWveP1yJRJNBdM= 2022-09-12T19:11:47.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9R7OZgPGIAcorQWveP1yJRJNBdM=', 'Content-Length': '0'} 2022-09-12T19:11:47.301 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'fU7DRGj75cTxlyr6yy4TPugTgwe2QhY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d70a10b34f781b73-00631f8473-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.302 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:47.302 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:47.303 DEBUG:boto:path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.303 DEBUG:boto:auth_path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.304 DEBUG:boto:Method: GET 2022-09-12T19:11:47.304 DEBUG:boto:Path: /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.305 DEBUG:boto:Data: 2022-09-12T19:11:47.305 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.306 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:47.306 DEBUG:boto:Port: 8000 2022-09-12T19:11:47.307 DEBUG:boto:Params: {} 2022-09-12T19:11:47.307 DEBUG:boto:Token: None 2022-09-12T19:11:47.308 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.309 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9R7OZgPGIAcorQWveP1yJRJNBdM= 2022-09-12T19:11:47.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9R7OZgPGIAcorQWveP1yJRJNBdM=', 'Content-Length': '0'} 2022-09-12T19:11:47.311 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'fU7DRGj75cTxlyr6yy4TPugTgwe2QhY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c8dd1779b2ef5424-00631f8473-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.312 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:47.313 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:47.313 DEBUG:boto:path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.314 DEBUG:boto:auth_path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.314 DEBUG:boto:Method: HEAD 2022-09-12T19:11:47.315 DEBUG:boto:Path: /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.315 DEBUG:boto:Data: 2022-09-12T19:11:47.316 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.316 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:47.317 DEBUG:boto:Port: 8001 2022-09-12T19:11:47.317 DEBUG:boto:Params: {} 2022-09-12T19:11:47.318 DEBUG:boto:Token: None 2022-09-12T19:11:47.318 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.319 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:olzindqOpL2uXLco+a5K0Or0mYM= 2022-09-12T19:11:47.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:olzindqOpL2uXLco+a5K0Or0mYM=', 'Content-Length': '0'} 2022-09-12T19:11:47.321 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'fU7DRGj75cTxlyr6yy4TPugTgwe2QhY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000706e76690687b0eb-00631f8473-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.322 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:47.323 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:47.323 DEBUG:boto:path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.324 DEBUG:boto:auth_path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.324 DEBUG:boto:Method: HEAD 2022-09-12T19:11:47.325 DEBUG:boto:Path: /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.325 DEBUG:boto:Data: 2022-09-12T19:11:47.326 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.327 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:47.327 DEBUG:boto:Port: 8000 2022-09-12T19:11:47.327 DEBUG:boto:Params: {} 2022-09-12T19:11:47.328 DEBUG:boto:Token: None 2022-09-12T19:11:47.329 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.329 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:olzindqOpL2uXLco+a5K0Or0mYM= 2022-09-12T19:11:47.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:olzindqOpL2uXLco+a5K0Or0mYM=', 'Content-Length': '0'} 2022-09-12T19:11:47.332 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'fU7DRGj75cTxlyr6yy4TPugTgwe2QhY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009eb826971cfd1ca4-00631f8473-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:47.333 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:47.334 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:47.335 DEBUG:boto:path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.335 DEBUG:boto:auth_path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.336 DEBUG:boto:Method: GET 2022-09-12T19:11:47.336 DEBUG:boto:Path: /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.337 DEBUG:boto:Data: 2022-09-12T19:11:47.337 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.338 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:47.338 DEBUG:boto:Port: 8001 2022-09-12T19:11:47.339 DEBUG:boto:Params: {} 2022-09-12T19:11:47.339 DEBUG:boto:Token: None 2022-09-12T19:11:47.340 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.341 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9R7OZgPGIAcorQWveP1yJRJNBdM= 2022-09-12T19:11:47.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9R7OZgPGIAcorQWveP1yJRJNBdM=', 'Content-Length': '0'} 2022-09-12T19:11:47.343 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'fU7DRGj75cTxlyr6yy4TPugTgwe2QhY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004d6be2c8963d7d77-00631f8473-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.344 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:47.345 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:47.345 DEBUG:boto:path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.346 DEBUG:boto:auth_path=/jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.346 DEBUG:boto:Method: GET 2022-09-12T19:11:47.347 DEBUG:boto:Path: /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.347 DEBUG:boto:Data: 2022-09-12T19:11:47.348 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.349 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:47.349 DEBUG:boto:Port: 8000 2022-09-12T19:11:47.350 DEBUG:boto:Params: {} 2022-09-12T19:11:47.350 DEBUG:boto:Token: None 2022-09-12T19:11:47.351 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/obj?versionId=fU7DRGj75cTxlyr6yy4TPugTgwe2QhY 2022-09-12T19:11:47.352 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9R7OZgPGIAcorQWveP1yJRJNBdM= 2022-09-12T19:11:47.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9R7OZgPGIAcorQWveP1yJRJNBdM=', 'Content-Length': '0'} 2022-09-12T19:11:47.354 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:53 GMT'), ('x-amz-version-id', 'fU7DRGj75cTxlyr6yy4TPugTgwe2QhY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d24e91cc7f5532d2-00631f8473-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.355 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:11:47.356 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:11:47.356 DEBUG:boto:path=/jdpiak-22/?prefix=obj 2022-09-12T19:11:47.357 DEBUG:boto:auth_path=/jdpiak-22/?prefix=obj 2022-09-12T19:11:47.357 DEBUG:boto:Method: GET 2022-09-12T19:11:47.358 DEBUG:boto:Path: /jdpiak-22/?prefix=obj 2022-09-12T19:11:47.359 DEBUG:boto:Data: 2022-09-12T19:11:47.359 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.360 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:47.361 DEBUG:boto:Port: 8001 2022-09-12T19:11:47.362 DEBUG:boto:Params: {} 2022-09-12T19:11:47.363 DEBUG:boto:Token: None 2022-09-12T19:11:47.363 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/ 2022-09-12T19:11:47.364 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cK2qCA+LoRv2VuR+oHadis/UnJM= 2022-09-12T19:11:47.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cK2qCA+LoRv2VuR+oHadis/UnJM=', 'Content-Length': '0'} 2022-09-12T19:11:47.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d35789bce2e174c-00631f8473-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.367 DEBUG:boto:b'jdpiak-22obj1000false' 2022-09-12T19:11:47.368 DEBUG:boto:path=/jdpiak-22/ 2022-09-12T19:11:47.369 DEBUG:boto:auth_path=/jdpiak-22/ 2022-09-12T19:11:47.370 DEBUG:boto:path=/jdpiak-22/?prefix=obj 2022-09-12T19:11:47.371 DEBUG:boto:auth_path=/jdpiak-22/?prefix=obj 2022-09-12T19:11:47.372 DEBUG:boto:Method: GET 2022-09-12T19:11:47.373 DEBUG:boto:Path: /jdpiak-22/?prefix=obj 2022-09-12T19:11:47.374 DEBUG:boto:Data: 2022-09-12T19:11:47.376 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.377 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:47.378 DEBUG:boto:Port: 8000 2022-09-12T19:11:47.378 DEBUG:boto:Params: {} 2022-09-12T19:11:47.379 DEBUG:boto:Token: None 2022-09-12T19:11:47.380 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/ 2022-09-12T19:11:47.380 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cK2qCA+LoRv2VuR+oHadis/UnJM= 2022-09-12T19:11:47.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cK2qCA+LoRv2VuR+oHadis/UnJM=', 'Content-Length': '0'} 2022-09-12T19:11:47.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba5edb6ae9b5fa49-00631f8473-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.426 DEBUG:boto:b'jdpiak-22obj1000false' 2022-09-12T19:11:47.427 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:47.428 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:47.429 DEBUG:boto:Method: HEAD 2022-09-12T19:11:47.429 DEBUG:boto:Path: /jdpiak-22/obj 2022-09-12T19:11:47.430 DEBUG:boto:Data: 2022-09-12T19:11:47.430 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.431 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:47.431 DEBUG:boto:Port: 8001 2022-09-12T19:11:47.432 DEBUG:boto:Params: {} 2022-09-12T19:11:47.432 DEBUG:boto:Token: None 2022-09-12T19:11:47.434 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/obj 2022-09-12T19:11:47.434 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zFlnHlELH2wu/bDMeUjcK6fy49Y= 2022-09-12T19:11:47.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zFlnHlELH2wu/bDMeUjcK6fy49Y=', 'Content-Length': '0'} 2022-09-12T19:11:47.437 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000003638b81b329c055d-00631f8473-10f2-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.438 DEBUG:boto:path=/jdpiak-22/obj 2022-09-12T19:11:47.438 DEBUG:boto:auth_path=/jdpiak-22/obj 2022-09-12T19:11:47.439 DEBUG:boto:Method: HEAD 2022-09-12T19:11:47.440 DEBUG:boto:Path: /jdpiak-22/obj 2022-09-12T19:11:47.440 DEBUG:boto:Data: 2022-09-12T19:11:47.441 DEBUG:boto:Headers: {} 2022-09-12T19:11:47.441 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:47.442 DEBUG:boto:Port: 8000 2022-09-12T19:11:47.442 DEBUG:boto:Params: {} 2022-09-12T19:11:47.443 DEBUG:boto:Token: None 2022-09-12T19:11:47.444 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:47 GMT /jdpiak-22/obj 2022-09-12T19:11:47.445 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zFlnHlELH2wu/bDMeUjcK6fy49Y= 2022-09-12T19:11:47.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:47 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zFlnHlELH2wu/bDMeUjcK6fy49Y=', 'Content-Length': '0'} 2022-09-12T19:11:47.448 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000f72ff644fee64ea1-00631f8473-10e4-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:47.448 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-22 zones={test-zone3, test-zone2} 2022-09-12T19:11:47.449 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-23 2022-09-12T19:11:47.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:47.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:47.451 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:11:47.516 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:47.516 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:47.574 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:47.571+0000 7f9edfb82600 1 bucket sync caught up with source: 2022-09-12T19:11:47.575 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000002.13.5, , ] 2022-09-12T19:11:47.575 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000002.13.5, , ] 2022-09-12T19:11:47.575 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:47.571+0000 7f9edfb82600 0 bucket checkpoint complete 2022-09-12T19:11:47.594 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-23 2022-09-12T19:11:47.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:47.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:47.596 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:11:47.659 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:47.659 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:47.696 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:47.693+0000 7f1dca696600 1 bucket sync caught up with source: 2022-09-12T19:11:47.696 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000002.13.5, , ] 2022-09-12T19:11:47.697 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000002.13.5, , ] 2022-09-12T19:11:47.697 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:47.693+0000 7f1dca696600 0 bucket checkpoint complete 2022-09-12T19:11:47.709 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-23 2022-09-12T19:11:47.710 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:47.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:47.711 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:11:47.766 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:47.767 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:47.801 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:47.798+0000 7fb098374600 1 bucket sync caught up with source: 2022-09-12T19:11:47.801 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000002.13.5, , ] 2022-09-12T19:11:47.802 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000002.13.5, , ] 2022-09-12T19:11:47.802 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:47.798+0000 7fb098374600 0 bucket checkpoint complete 2022-09-12T19:11:47.813 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-23 2022-09-12T19:11:47.813 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:47.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:47.814 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:11:47.873 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:47.873 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:47.910 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:47.907+0000 7f2787d5c600 1 bucket sync caught up with source: 2022-09-12T19:11:47.911 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000002.86.5, , ] 2022-09-12T19:11:47.911 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000002.86.5, , ] 2022-09-12T19:11:47.911 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:47.907+0000 7f2787d5c600 0 bucket checkpoint complete 2022-09-12T19:11:47.923 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-23 2022-09-12T19:11:47.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:47.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:47.925 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:11:47.982 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:47.982 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:48.028 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:48.025+0000 7fc90754a600 1 bucket sync caught up with source: 2022-09-12T19:11:48.029 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000002.86.5, , ] 2022-09-12T19:11:48.029 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000002.86.5, , ] 2022-09-12T19:11:48.029 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:48.025+0000 7fc90754a600 0 bucket checkpoint complete 2022-09-12T19:11:48.049 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-23 2022-09-12T19:11:48.050 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-23 2022-09-12T19:11:48.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:48.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:48.051 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:11:48.109 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:48.110 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:48.155 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:48.152+0000 7fda95fa7600 1 bucket sync caught up with source: 2022-09-12T19:11:48.156 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000003.81.11, , ] 2022-09-12T19:11:48.156 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000003.81.11, , ] 2022-09-12T19:11:48.156 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:48.152+0000 7fda95fa7600 0 bucket checkpoint complete 2022-09-12T19:11:48.172 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-23 2022-09-12T19:11:48.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:48.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:48.174 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:11:48.231 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:48.231 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:48.265 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:48.263+0000 7fde782d4600 1 bucket sync caught up with source: 2022-09-12T19:11:48.266 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000003.81.11, , ] 2022-09-12T19:11:48.266 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000003.81.11, , ] 2022-09-12T19:11:48.266 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:48.263+0000 7fde782d4600 0 bucket checkpoint complete 2022-09-12T19:11:48.280 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-23 2022-09-12T19:11:48.281 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-23 zones={test-zone2, test-zone1} 2022-09-12T19:11:48.282 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:11:48.282 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:11:48.283 DEBUG:boto:Method: HEAD 2022-09-12T19:11:48.283 DEBUG:boto:Path: /jdpiak-23/ 2022-09-12T19:11:48.284 DEBUG:boto:Data: 2022-09-12T19:11:48.284 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.285 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:48.285 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.286 DEBUG:boto:Params: {} 2022-09-12T19:11:48.286 DEBUG:boto:Token: None 2022-09-12T19:11:48.287 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/ 2022-09-12T19:11:48.287 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iaiFgnZ3IiJlfZzcLCgsFyhbkX4= 2022-09-12T19:11:48.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iaiFgnZ3IiJlfZzcLCgsFyhbkX4=', 'Content-Length': '0'} 2022-09-12T19:11:48.291 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a25f4c17b37eb908-00631f8474-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.291 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:11:48.292 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:11:48.292 DEBUG:boto:Method: HEAD 2022-09-12T19:11:48.293 DEBUG:boto:Path: /jdpiak-23/ 2022-09-12T19:11:48.293 DEBUG:boto:Data: 2022-09-12T19:11:48.294 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.294 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:48.295 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.295 DEBUG:boto:Params: {} 2022-09-12T19:11:48.296 DEBUG:boto:Token: None 2022-09-12T19:11:48.296 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/ 2022-09-12T19:11:48.297 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iaiFgnZ3IiJlfZzcLCgsFyhbkX4= 2022-09-12T19:11:48.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iaiFgnZ3IiJlfZzcLCgsFyhbkX4=', 'Content-Length': '0'} 2022-09-12T19:11:48.300 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001c8969ea90ee8cb-00631f8474-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.300 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:11:48.301 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:11:48.301 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:11:48.302 DEBUG:boto:path=/jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.303 DEBUG:boto:auth_path=/jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.303 DEBUG:boto:Method: GET 2022-09-12T19:11:48.304 DEBUG:boto:Path: /jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.304 DEBUG:boto:Data: 2022-09-12T19:11:48.305 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.305 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:48.306 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.306 DEBUG:boto:Params: {} 2022-09-12T19:11:48.307 DEBUG:boto:Token: None 2022-09-12T19:11:48.307 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/?versions 2022-09-12T19:11:48.308 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qqIx8v1E8w7GcIuqBacxrlKSyzY= 2022-09-12T19:11:48.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qqIx8v1E8w7GcIuqBacxrlKSyzY=', 'Content-Length': '0'} 2022-09-12T19:11:48.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008857294b8355e9b7-00631f8474-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.312 DEBUG:boto:b'jdpiak-23999falseobjrKzI1doaCWXfUPsgqDJaJz0BenA63mMtrue2022-09-12T19:10:54.090Zrgw-multisite-test-userMultisite Test UserNormalobjTBH3pBU5gf-t2nym3FdhrHv6BSTFHv0false2022-09-12T19:10:54.060Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:48.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:48.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:48.314 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:11:48.314 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:11:48.315 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:11:48.316 DEBUG:boto:path=/jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.316 DEBUG:boto:auth_path=/jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.317 DEBUG:boto:Method: GET 2022-09-12T19:11:48.317 DEBUG:boto:Path: /jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.318 DEBUG:boto:Data: 2022-09-12T19:11:48.318 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.319 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:48.319 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.320 DEBUG:boto:Params: {} 2022-09-12T19:11:48.320 DEBUG:boto:Token: None 2022-09-12T19:11:48.321 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/?versions 2022-09-12T19:11:48.322 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qqIx8v1E8w7GcIuqBacxrlKSyzY= 2022-09-12T19:11:48.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qqIx8v1E8w7GcIuqBacxrlKSyzY=', 'Content-Length': '0'} 2022-09-12T19:11:48.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f5b30a51a40d018-00631f8474-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.326 DEBUG:boto:b'jdpiak-23999falseobjrKzI1doaCWXfUPsgqDJaJz0BenA63mMtrue2022-09-12T19:10:54.090Zrgw-multisite-test-userMultisite Test UserNormalobjTBH3pBU5gf-t2nym3FdhrHv6BSTFHv0false2022-09-12T19:10:54.060Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:48.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:48.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:48.327 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:11:48.328 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:11:48.329 DEBUG:boto:path=/jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.329 DEBUG:boto:auth_path=/jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.330 DEBUG:boto:Method: GET 2022-09-12T19:11:48.330 DEBUG:boto:Path: /jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.331 DEBUG:boto:Data: 2022-09-12T19:11:48.331 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.332 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:48.332 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.333 DEBUG:boto:Params: {} 2022-09-12T19:11:48.333 DEBUG:boto:Token: None 2022-09-12T19:11:48.334 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/?versions 2022-09-12T19:11:48.335 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qqIx8v1E8w7GcIuqBacxrlKSyzY= 2022-09-12T19:11:48.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qqIx8v1E8w7GcIuqBacxrlKSyzY=', 'Content-Length': '0'} 2022-09-12T19:11:48.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f16706157fe88c77-00631f8474-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.378 DEBUG:boto:b'jdpiak-23999falseobjrKzI1doaCWXfUPsgqDJaJz0BenA63mMtrue2022-09-12T19:10:54.090Zrgw-multisite-test-userMultisite Test UserNormalobjTBH3pBU5gf-t2nym3FdhrHv6BSTFHv0false2022-09-12T19:10:54.060Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:48.379 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:11:48.380 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:11:48.380 DEBUG:boto:path=/jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.381 DEBUG:boto:auth_path=/jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.381 DEBUG:boto:Method: GET 2022-09-12T19:11:48.382 DEBUG:boto:Path: /jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.382 DEBUG:boto:Data: 2022-09-12T19:11:48.383 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.383 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:48.384 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.385 DEBUG:boto:Params: {} 2022-09-12T19:11:48.385 DEBUG:boto:Token: None 2022-09-12T19:11:48.386 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/?versions 2022-09-12T19:11:48.387 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qqIx8v1E8w7GcIuqBacxrlKSyzY= 2022-09-12T19:11:48.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qqIx8v1E8w7GcIuqBacxrlKSyzY=', 'Content-Length': '0'} 2022-09-12T19:11:48.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003695d60673d5ecaa-00631f8474-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.391 DEBUG:boto:b'jdpiak-23999falseobjrKzI1doaCWXfUPsgqDJaJz0BenA63mMtrue2022-09-12T19:10:54.090Zrgw-multisite-test-userMultisite Test UserNormalobjTBH3pBU5gf-t2nym3FdhrHv6BSTFHv0false2022-09-12T19:10:54.060Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:48.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:48.392 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:48.393 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:48.394 DEBUG:boto:Method: HEAD 2022-09-12T19:11:48.394 DEBUG:boto:Path: /jdpiak-23/obj 2022-09-12T19:11:48.395 DEBUG:boto:Data: 2022-09-12T19:11:48.395 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.396 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:48.396 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.397 DEBUG:boto:Params: {} 2022-09-12T19:11:48.398 DEBUG:boto:Token: None 2022-09-12T19:11:48.398 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/obj 2022-09-12T19:11:48.399 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:M+ABgLkzdkOTBK0NgrC/Rm/I5Mc= 2022-09-12T19:11:48.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:M+ABgLkzdkOTBK0NgrC/Rm/I5Mc=', 'Content-Length': '0'} 2022-09-12T19:11:48.402 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000fb047bc2daa462fb-00631f8474-10e4-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.403 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:48.403 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:48.404 DEBUG:boto:Method: HEAD 2022-09-12T19:11:48.404 DEBUG:boto:Path: /jdpiak-23/obj 2022-09-12T19:11:48.405 DEBUG:boto:Data: 2022-09-12T19:11:48.406 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.406 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:48.407 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.408 DEBUG:boto:Params: {} 2022-09-12T19:11:48.408 DEBUG:boto:Token: None 2022-09-12T19:11:48.409 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/obj 2022-09-12T19:11:48.409 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:M+ABgLkzdkOTBK0NgrC/Rm/I5Mc= 2022-09-12T19:11:48.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:M+ABgLkzdkOTBK0NgrC/Rm/I5Mc=', 'Content-Length': '0'} 2022-09-12T19:11:48.412 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000006168816c22715960-00631f8474-10f6-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:48.414 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:48.414 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:48.415 DEBUG:boto:path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.415 DEBUG:boto:auth_path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.416 DEBUG:boto:Method: GET 2022-09-12T19:11:48.416 DEBUG:boto:Path: /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.417 DEBUG:boto:Data: 2022-09-12T19:11:48.417 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.418 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:48.419 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.419 DEBUG:boto:Params: {} 2022-09-12T19:11:48.420 DEBUG:boto:Token: None 2022-09-12T19:11:48.421 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.421 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BZLXb7NZEEErz4FDniJHNeMa//s= 2022-09-12T19:11:48.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BZLXb7NZEEErz4FDniJHNeMa//s=', 'Content-Length': '0'} 2022-09-12T19:11:48.424 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:54 GMT'), ('x-amz-version-id', 'TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004c27d8b2e7699174-00631f8474-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.425 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:48.426 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:48.427 DEBUG:boto:path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.428 DEBUG:boto:auth_path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.428 DEBUG:boto:Method: GET 2022-09-12T19:11:48.429 DEBUG:boto:Path: /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.429 DEBUG:boto:Data: 2022-09-12T19:11:48.430 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.430 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:48.431 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.432 DEBUG:boto:Params: {} 2022-09-12T19:11:48.432 DEBUG:boto:Token: None 2022-09-12T19:11:48.433 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.434 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BZLXb7NZEEErz4FDniJHNeMa//s= 2022-09-12T19:11:48.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BZLXb7NZEEErz4FDniJHNeMa//s=', 'Content-Length': '0'} 2022-09-12T19:11:48.436 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:54 GMT'), ('x-amz-version-id', 'TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004e62551bf9fb4517-00631f8474-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.437 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:48.437 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:48.438 DEBUG:boto:path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.438 DEBUG:boto:auth_path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.439 DEBUG:boto:Method: HEAD 2022-09-12T19:11:48.440 DEBUG:boto:Path: /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.440 DEBUG:boto:Data: 2022-09-12T19:11:48.441 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.441 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:48.442 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.443 DEBUG:boto:Params: {} 2022-09-12T19:11:48.443 DEBUG:boto:Token: None 2022-09-12T19:11:48.444 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.444 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TGGzqg6Dcd8xr354Rf/lxWZFRLE= 2022-09-12T19:11:48.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TGGzqg6Dcd8xr354Rf/lxWZFRLE=', 'Content-Length': '0'} 2022-09-12T19:11:48.447 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:54 GMT'), ('x-amz-version-id', 'TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004b38df0b5894af94-00631f8474-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.447 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:48.448 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:48.448 DEBUG:boto:path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.449 DEBUG:boto:auth_path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.450 DEBUG:boto:Method: HEAD 2022-09-12T19:11:48.450 DEBUG:boto:Path: /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.451 DEBUG:boto:Data: 2022-09-12T19:11:48.452 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.453 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:48.453 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.454 DEBUG:boto:Params: {} 2022-09-12T19:11:48.455 DEBUG:boto:Token: None 2022-09-12T19:11:48.457 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.458 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TGGzqg6Dcd8xr354Rf/lxWZFRLE= 2022-09-12T19:11:48.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TGGzqg6Dcd8xr354Rf/lxWZFRLE=', 'Content-Length': '0'} 2022-09-12T19:11:48.460 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:54 GMT'), ('x-amz-version-id', 'TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000313d678d27bdcb8f-00631f8474-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:48.462 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:48.463 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:48.463 DEBUG:boto:path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.464 DEBUG:boto:auth_path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.465 DEBUG:boto:Method: GET 2022-09-12T19:11:48.466 DEBUG:boto:Path: /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.467 DEBUG:boto:Data: 2022-09-12T19:11:48.467 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.468 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:48.469 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.469 DEBUG:boto:Params: {} 2022-09-12T19:11:48.470 DEBUG:boto:Token: None 2022-09-12T19:11:48.471 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.472 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BZLXb7NZEEErz4FDniJHNeMa//s= 2022-09-12T19:11:48.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BZLXb7NZEEErz4FDniJHNeMa//s=', 'Content-Length': '0'} 2022-09-12T19:11:48.474 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:54 GMT'), ('x-amz-version-id', 'TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000006a0c4ccaf2dd10f-00631f8474-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.475 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:48.476 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:48.476 DEBUG:boto:path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.477 DEBUG:boto:auth_path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.477 DEBUG:boto:Method: GET 2022-09-12T19:11:48.478 DEBUG:boto:Path: /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.478 DEBUG:boto:Data: 2022-09-12T19:11:48.479 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.479 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:48.480 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.480 DEBUG:boto:Params: {} 2022-09-12T19:11:48.481 DEBUG:boto:Token: None 2022-09-12T19:11:48.482 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.483 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BZLXb7NZEEErz4FDniJHNeMa//s= 2022-09-12T19:11:48.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BZLXb7NZEEErz4FDniJHNeMa//s=', 'Content-Length': '0'} 2022-09-12T19:11:48.485 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:54 GMT'), ('x-amz-version-id', 'TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dab735f679962d96-00631f8474-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.486 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:11:48.487 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:11:48.487 DEBUG:boto:path=/jdpiak-23/?prefix=obj 2022-09-12T19:11:48.488 DEBUG:boto:auth_path=/jdpiak-23/?prefix=obj 2022-09-12T19:11:48.488 DEBUG:boto:Method: GET 2022-09-12T19:11:48.489 DEBUG:boto:Path: /jdpiak-23/?prefix=obj 2022-09-12T19:11:48.490 DEBUG:boto:Data: 2022-09-12T19:11:48.490 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.491 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:48.491 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.492 DEBUG:boto:Params: {} 2022-09-12T19:11:48.493 DEBUG:boto:Token: None 2022-09-12T19:11:48.493 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/ 2022-09-12T19:11:48.494 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vRC9zKNpGpmGdclTUDHLJvDBbSk= 2022-09-12T19:11:48.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vRC9zKNpGpmGdclTUDHLJvDBbSk=', 'Content-Length': '0'} 2022-09-12T19:11:48.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7f7b28f0dfc27f0-00631f8474-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.538 DEBUG:boto:b'jdpiak-23obj1000false' 2022-09-12T19:11:48.539 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:11:48.540 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:11:48.540 DEBUG:boto:path=/jdpiak-23/?prefix=obj 2022-09-12T19:11:48.541 DEBUG:boto:auth_path=/jdpiak-23/?prefix=obj 2022-09-12T19:11:48.542 DEBUG:boto:Method: GET 2022-09-12T19:11:48.542 DEBUG:boto:Path: /jdpiak-23/?prefix=obj 2022-09-12T19:11:48.543 DEBUG:boto:Data: 2022-09-12T19:11:48.543 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.544 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:48.544 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.545 DEBUG:boto:Params: {} 2022-09-12T19:11:48.545 DEBUG:boto:Token: None 2022-09-12T19:11:48.546 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/ 2022-09-12T19:11:48.547 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vRC9zKNpGpmGdclTUDHLJvDBbSk= 2022-09-12T19:11:48.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vRC9zKNpGpmGdclTUDHLJvDBbSk=', 'Content-Length': '0'} 2022-09-12T19:11:48.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003394e09d97fa3e3b-00631f8474-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.551 DEBUG:boto:b'jdpiak-23obj1000false' 2022-09-12T19:11:48.551 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:48.552 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:48.553 DEBUG:boto:Method: HEAD 2022-09-12T19:11:48.553 DEBUG:boto:Path: /jdpiak-23/obj 2022-09-12T19:11:48.554 DEBUG:boto:Data: 2022-09-12T19:11:48.554 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.555 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:48.555 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.556 DEBUG:boto:Params: {} 2022-09-12T19:11:48.556 DEBUG:boto:Token: None 2022-09-12T19:11:48.557 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/obj 2022-09-12T19:11:48.557 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:M+ABgLkzdkOTBK0NgrC/Rm/I5Mc= 2022-09-12T19:11:48.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:M+ABgLkzdkOTBK0NgrC/Rm/I5Mc=', 'Content-Length': '0'} 2022-09-12T19:11:48.560 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000855e1ed7b6ff8522-00631f8474-10e4-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.561 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:48.561 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:48.562 DEBUG:boto:Method: HEAD 2022-09-12T19:11:48.563 DEBUG:boto:Path: /jdpiak-23/obj 2022-09-12T19:11:48.563 DEBUG:boto:Data: 2022-09-12T19:11:48.564 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.564 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:48.565 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.565 DEBUG:boto:Params: {} 2022-09-12T19:11:48.566 DEBUG:boto:Token: None 2022-09-12T19:11:48.567 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/obj 2022-09-12T19:11:48.567 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:M+ABgLkzdkOTBK0NgrC/Rm/I5Mc= 2022-09-12T19:11:48.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:M+ABgLkzdkOTBK0NgrC/Rm/I5Mc=', 'Content-Length': '0'} 2022-09-12T19:11:48.570 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000008e87d1e5b3ad7a94-00631f8474-10f6-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.571 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-23 zones={test-zone2, test-zone1} 2022-09-12T19:11:48.571 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-23 zones={test-zone3, test-zone1} 2022-09-12T19:11:48.572 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:11:48.572 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:11:48.573 DEBUG:boto:Method: HEAD 2022-09-12T19:11:48.574 DEBUG:boto:Path: /jdpiak-23/ 2022-09-12T19:11:48.574 DEBUG:boto:Data: 2022-09-12T19:11:48.575 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.575 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:48.576 DEBUG:boto:Port: 8001 2022-09-12T19:11:48.576 DEBUG:boto:Params: {} 2022-09-12T19:11:48.577 DEBUG:boto:Token: None 2022-09-12T19:11:48.578 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/ 2022-09-12T19:11:48.578 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iaiFgnZ3IiJlfZzcLCgsFyhbkX4= 2022-09-12T19:11:48.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iaiFgnZ3IiJlfZzcLCgsFyhbkX4=', 'Content-Length': '0'} 2022-09-12T19:11:48.581 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a63d39ab625e5b6-00631f8474-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.582 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:11:48.582 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:11:48.583 DEBUG:boto:Method: HEAD 2022-09-12T19:11:48.583 DEBUG:boto:Path: /jdpiak-23/ 2022-09-12T19:11:48.584 DEBUG:boto:Data: 2022-09-12T19:11:48.584 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.585 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:48.585 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.586 DEBUG:boto:Params: {} 2022-09-12T19:11:48.586 DEBUG:boto:Token: None 2022-09-12T19:11:48.587 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/ 2022-09-12T19:11:48.587 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iaiFgnZ3IiJlfZzcLCgsFyhbkX4= 2022-09-12T19:11:48.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iaiFgnZ3IiJlfZzcLCgsFyhbkX4=', 'Content-Length': '0'} 2022-09-12T19:11:48.590 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e77a4e7e7fac0bd-00631f8474-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.591 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:11:48.592 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:11:48.592 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:11:48.593 DEBUG:boto:path=/jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.593 DEBUG:boto:auth_path=/jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.594 DEBUG:boto:Method: GET 2022-09-12T19:11:48.594 DEBUG:boto:Path: /jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.595 DEBUG:boto:Data: 2022-09-12T19:11:48.595 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.596 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:48.596 DEBUG:boto:Port: 8001 2022-09-12T19:11:48.597 DEBUG:boto:Params: {} 2022-09-12T19:11:48.597 DEBUG:boto:Token: None 2022-09-12T19:11:48.598 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/?versions 2022-09-12T19:11:48.599 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qqIx8v1E8w7GcIuqBacxrlKSyzY= 2022-09-12T19:11:48.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qqIx8v1E8w7GcIuqBacxrlKSyzY=', 'Content-Length': '0'} 2022-09-12T19:11:48.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1ea83fb6f569ce2-00631f8474-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.602 DEBUG:boto:b'jdpiak-23999falseobjrKzI1doaCWXfUPsgqDJaJz0BenA63mMtrue2022-09-12T19:10:54.090Zrgw-multisite-test-userMultisite Test UserNormalobjTBH3pBU5gf-t2nym3FdhrHv6BSTFHv0false2022-09-12T19:10:54.060Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:48.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:48.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:48.604 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:11:48.604 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:11:48.605 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:11:48.606 DEBUG:boto:path=/jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.606 DEBUG:boto:auth_path=/jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.607 DEBUG:boto:Method: GET 2022-09-12T19:11:48.607 DEBUG:boto:Path: /jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.608 DEBUG:boto:Data: 2022-09-12T19:11:48.608 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.609 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:48.609 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.610 DEBUG:boto:Params: {} 2022-09-12T19:11:48.610 DEBUG:boto:Token: None 2022-09-12T19:11:48.611 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/?versions 2022-09-12T19:11:48.611 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qqIx8v1E8w7GcIuqBacxrlKSyzY= 2022-09-12T19:11:48.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qqIx8v1E8w7GcIuqBacxrlKSyzY=', 'Content-Length': '0'} 2022-09-12T19:11:48.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9d0eaab7e7c5373-00631f8474-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.658 DEBUG:boto:b'jdpiak-23999falseobjrKzI1doaCWXfUPsgqDJaJz0BenA63mMtrue2022-09-12T19:10:54.090Zrgw-multisite-test-userMultisite Test UserNormalobjTBH3pBU5gf-t2nym3FdhrHv6BSTFHv0false2022-09-12T19:10:54.060Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:48.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:48.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:48.661 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:11:48.662 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:11:48.662 DEBUG:boto:path=/jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.663 DEBUG:boto:auth_path=/jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.664 DEBUG:boto:Method: GET 2022-09-12T19:11:48.664 DEBUG:boto:Path: /jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.665 DEBUG:boto:Data: 2022-09-12T19:11:48.665 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.666 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:48.667 DEBUG:boto:Port: 8001 2022-09-12T19:11:48.667 DEBUG:boto:Params: {} 2022-09-12T19:11:48.668 DEBUG:boto:Token: None 2022-09-12T19:11:48.668 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/?versions 2022-09-12T19:11:48.669 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qqIx8v1E8w7GcIuqBacxrlKSyzY= 2022-09-12T19:11:48.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qqIx8v1E8w7GcIuqBacxrlKSyzY=', 'Content-Length': '0'} 2022-09-12T19:11:48.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009644c58320f5868b-00631f8474-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.673 DEBUG:boto:b'jdpiak-23999falseobjrKzI1doaCWXfUPsgqDJaJz0BenA63mMtrue2022-09-12T19:10:54.090Zrgw-multisite-test-userMultisite Test UserNormalobjTBH3pBU5gf-t2nym3FdhrHv6BSTFHv0false2022-09-12T19:10:54.060Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:48.674 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:11:48.674 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:11:48.675 DEBUG:boto:path=/jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.675 DEBUG:boto:auth_path=/jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.676 DEBUG:boto:Method: GET 2022-09-12T19:11:48.677 DEBUG:boto:Path: /jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.677 DEBUG:boto:Data: 2022-09-12T19:11:48.678 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.678 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:48.679 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.679 DEBUG:boto:Params: {} 2022-09-12T19:11:48.680 DEBUG:boto:Token: None 2022-09-12T19:11:48.681 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/?versions 2022-09-12T19:11:48.681 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qqIx8v1E8w7GcIuqBacxrlKSyzY= 2022-09-12T19:11:48.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qqIx8v1E8w7GcIuqBacxrlKSyzY=', 'Content-Length': '0'} 2022-09-12T19:11:48.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be0230ee8baf422d-00631f8474-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.687 DEBUG:boto:b'jdpiak-23999falseobjrKzI1doaCWXfUPsgqDJaJz0BenA63mMtrue2022-09-12T19:10:54.090Zrgw-multisite-test-userMultisite Test UserNormalobjTBH3pBU5gf-t2nym3FdhrHv6BSTFHv0false2022-09-12T19:10:54.060Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:48.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:48.688 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:48.689 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:48.689 DEBUG:boto:Method: HEAD 2022-09-12T19:11:48.690 DEBUG:boto:Path: /jdpiak-23/obj 2022-09-12T19:11:48.690 DEBUG:boto:Data: 2022-09-12T19:11:48.691 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.692 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:48.692 DEBUG:boto:Port: 8001 2022-09-12T19:11:48.693 DEBUG:boto:Params: {} 2022-09-12T19:11:48.693 DEBUG:boto:Token: None 2022-09-12T19:11:48.694 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/obj 2022-09-12T19:11:48.695 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:M+ABgLkzdkOTBK0NgrC/Rm/I5Mc= 2022-09-12T19:11:48.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:M+ABgLkzdkOTBK0NgrC/Rm/I5Mc=', 'Content-Length': '0'} 2022-09-12T19:11:48.703 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000008f0bcabc62296e4-00631f8474-10f2-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.705 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:48.706 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:48.707 DEBUG:boto:Method: HEAD 2022-09-12T19:11:48.708 DEBUG:boto:Path: /jdpiak-23/obj 2022-09-12T19:11:48.709 DEBUG:boto:Data: 2022-09-12T19:11:48.709 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.710 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:48.711 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.712 DEBUG:boto:Params: {} 2022-09-12T19:11:48.713 DEBUG:boto:Token: None 2022-09-12T19:11:48.715 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/obj 2022-09-12T19:11:48.716 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:M+ABgLkzdkOTBK0NgrC/Rm/I5Mc= 2022-09-12T19:11:48.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:M+ABgLkzdkOTBK0NgrC/Rm/I5Mc=', 'Content-Length': '0'} 2022-09-12T19:11:48.720 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000020f0943aa1656d7e-00631f8474-10f6-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:48.722 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:48.723 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:48.724 DEBUG:boto:path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.725 DEBUG:boto:auth_path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.726 DEBUG:boto:Method: GET 2022-09-12T19:11:48.727 DEBUG:boto:Path: /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.728 DEBUG:boto:Data: 2022-09-12T19:11:48.729 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.729 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:48.730 DEBUG:boto:Port: 8001 2022-09-12T19:11:48.731 DEBUG:boto:Params: {} 2022-09-12T19:11:48.732 DEBUG:boto:Token: None 2022-09-12T19:11:48.733 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.734 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BZLXb7NZEEErz4FDniJHNeMa//s= 2022-09-12T19:11:48.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BZLXb7NZEEErz4FDniJHNeMa//s=', 'Content-Length': '0'} 2022-09-12T19:11:48.738 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:54 GMT'), ('x-amz-version-id', 'TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c576a4b1ca95cad7-00631f8474-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.739 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:48.739 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:48.740 DEBUG:boto:path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.740 DEBUG:boto:auth_path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.741 DEBUG:boto:Method: GET 2022-09-12T19:11:48.741 DEBUG:boto:Path: /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.742 DEBUG:boto:Data: 2022-09-12T19:11:48.742 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.743 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:48.744 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.745 DEBUG:boto:Params: {} 2022-09-12T19:11:48.745 DEBUG:boto:Token: None 2022-09-12T19:11:48.746 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.747 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BZLXb7NZEEErz4FDniJHNeMa//s= 2022-09-12T19:11:48.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BZLXb7NZEEErz4FDniJHNeMa//s=', 'Content-Length': '0'} 2022-09-12T19:11:48.750 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:54 GMT'), ('x-amz-version-id', 'TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b1bf78c35c27b686-00631f8474-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.750 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:48.751 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:48.752 DEBUG:boto:path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.752 DEBUG:boto:auth_path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.753 DEBUG:boto:Method: HEAD 2022-09-12T19:11:48.753 DEBUG:boto:Path: /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.754 DEBUG:boto:Data: 2022-09-12T19:11:48.755 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.755 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:48.756 DEBUG:boto:Port: 8001 2022-09-12T19:11:48.756 DEBUG:boto:Params: {} 2022-09-12T19:11:48.757 DEBUG:boto:Token: None 2022-09-12T19:11:48.758 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.758 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TGGzqg6Dcd8xr354Rf/lxWZFRLE= 2022-09-12T19:11:48.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TGGzqg6Dcd8xr354Rf/lxWZFRLE=', 'Content-Length': '0'} 2022-09-12T19:11:48.762 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:54 GMT'), ('x-amz-version-id', 'TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000e92951338a82116-00631f8474-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.763 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:48.763 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:48.764 DEBUG:boto:path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.765 DEBUG:boto:auth_path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.765 DEBUG:boto:Method: HEAD 2022-09-12T19:11:48.766 DEBUG:boto:Path: /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.766 DEBUG:boto:Data: 2022-09-12T19:11:48.767 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.767 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:48.768 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.769 DEBUG:boto:Params: {} 2022-09-12T19:11:48.769 DEBUG:boto:Token: None 2022-09-12T19:11:48.770 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.771 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TGGzqg6Dcd8xr354Rf/lxWZFRLE= 2022-09-12T19:11:48.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TGGzqg6Dcd8xr354Rf/lxWZFRLE=', 'Content-Length': '0'} 2022-09-12T19:11:48.773 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:54 GMT'), ('x-amz-version-id', 'TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d14004ec3f09d8fd-00631f8474-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:48.776 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:48.777 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:48.778 DEBUG:boto:path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.779 DEBUG:boto:auth_path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.780 DEBUG:boto:Method: GET 2022-09-12T19:11:48.781 DEBUG:boto:Path: /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.782 DEBUG:boto:Data: 2022-09-12T19:11:48.783 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.784 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:48.785 DEBUG:boto:Port: 8001 2022-09-12T19:11:48.786 DEBUG:boto:Params: {} 2022-09-12T19:11:48.787 DEBUG:boto:Token: None 2022-09-12T19:11:48.788 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.789 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BZLXb7NZEEErz4FDniJHNeMa//s= 2022-09-12T19:11:48.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BZLXb7NZEEErz4FDniJHNeMa//s=', 'Content-Length': '0'} 2022-09-12T19:11:48.792 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:54 GMT'), ('x-amz-version-id', 'TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004e7bfca8222257a9-00631f8474-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.794 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:48.795 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:48.796 DEBUG:boto:path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.797 DEBUG:boto:auth_path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.797 DEBUG:boto:Method: GET 2022-09-12T19:11:48.798 DEBUG:boto:Path: /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.799 DEBUG:boto:Data: 2022-09-12T19:11:48.800 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.801 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:48.802 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.803 DEBUG:boto:Params: {} 2022-09-12T19:11:48.804 DEBUG:boto:Token: None 2022-09-12T19:11:48.805 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:48.806 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BZLXb7NZEEErz4FDniJHNeMa//s= 2022-09-12T19:11:48.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BZLXb7NZEEErz4FDniJHNeMa//s=', 'Content-Length': '0'} 2022-09-12T19:11:48.809 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:54 GMT'), ('x-amz-version-id', 'TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d9623022950700f0-00631f8474-10f6-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.809 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:11:48.810 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:11:48.811 DEBUG:boto:path=/jdpiak-23/?prefix=obj 2022-09-12T19:11:48.812 DEBUG:boto:auth_path=/jdpiak-23/?prefix=obj 2022-09-12T19:11:48.813 DEBUG:boto:Method: GET 2022-09-12T19:11:48.814 DEBUG:boto:Path: /jdpiak-23/?prefix=obj 2022-09-12T19:11:48.815 DEBUG:boto:Data: 2022-09-12T19:11:48.816 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.817 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:48.817 DEBUG:boto:Port: 8001 2022-09-12T19:11:48.818 DEBUG:boto:Params: {} 2022-09-12T19:11:48.819 DEBUG:boto:Token: None 2022-09-12T19:11:48.819 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/ 2022-09-12T19:11:48.820 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vRC9zKNpGpmGdclTUDHLJvDBbSk= 2022-09-12T19:11:48.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vRC9zKNpGpmGdclTUDHLJvDBbSk=', 'Content-Length': '0'} 2022-09-12T19:11:48.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000680fbee8233f9a0b-00631f8474-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.866 DEBUG:boto:b'jdpiak-23obj1000false' 2022-09-12T19:11:48.867 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:11:48.868 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:11:48.868 DEBUG:boto:path=/jdpiak-23/?prefix=obj 2022-09-12T19:11:48.869 DEBUG:boto:auth_path=/jdpiak-23/?prefix=obj 2022-09-12T19:11:48.869 DEBUG:boto:Method: GET 2022-09-12T19:11:48.870 DEBUG:boto:Path: /jdpiak-23/?prefix=obj 2022-09-12T19:11:48.870 DEBUG:boto:Data: 2022-09-12T19:11:48.871 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.872 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:48.872 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.873 DEBUG:boto:Params: {} 2022-09-12T19:11:48.874 DEBUG:boto:Token: None 2022-09-12T19:11:48.875 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/ 2022-09-12T19:11:48.876 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vRC9zKNpGpmGdclTUDHLJvDBbSk= 2022-09-12T19:11:48.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vRC9zKNpGpmGdclTUDHLJvDBbSk=', 'Content-Length': '0'} 2022-09-12T19:11:48.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bfe10b4b9fde218-00631f8474-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.880 DEBUG:boto:b'jdpiak-23obj1000false' 2022-09-12T19:11:48.881 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:48.881 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:48.882 DEBUG:boto:Method: HEAD 2022-09-12T19:11:48.882 DEBUG:boto:Path: /jdpiak-23/obj 2022-09-12T19:11:48.883 DEBUG:boto:Data: 2022-09-12T19:11:48.884 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.884 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:48.885 DEBUG:boto:Port: 8001 2022-09-12T19:11:48.885 DEBUG:boto:Params: {} 2022-09-12T19:11:48.886 DEBUG:boto:Token: None 2022-09-12T19:11:48.886 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/obj 2022-09-12T19:11:48.887 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:M+ABgLkzdkOTBK0NgrC/Rm/I5Mc= 2022-09-12T19:11:48.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:M+ABgLkzdkOTBK0NgrC/Rm/I5Mc=', 'Content-Length': '0'} 2022-09-12T19:11:48.890 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000a9337aa140a49048-00631f8474-10f2-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.890 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:48.891 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:48.891 DEBUG:boto:Method: HEAD 2022-09-12T19:11:48.892 DEBUG:boto:Path: /jdpiak-23/obj 2022-09-12T19:11:48.892 DEBUG:boto:Data: 2022-09-12T19:11:48.893 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.893 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:48.894 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.894 DEBUG:boto:Params: {} 2022-09-12T19:11:48.895 DEBUG:boto:Token: None 2022-09-12T19:11:48.895 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/obj 2022-09-12T19:11:48.896 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:M+ABgLkzdkOTBK0NgrC/Rm/I5Mc= 2022-09-12T19:11:48.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:M+ABgLkzdkOTBK0NgrC/Rm/I5Mc=', 'Content-Length': '0'} 2022-09-12T19:11:48.899 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000008165a9fced61a8c1-00631f8474-10f6-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.899 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-23 zones={test-zone3, test-zone1} 2022-09-12T19:11:48.900 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-23 zones={test-zone3, test-zone2} 2022-09-12T19:11:48.900 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:11:48.901 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:11:48.902 DEBUG:boto:Method: HEAD 2022-09-12T19:11:48.902 DEBUG:boto:Path: /jdpiak-23/ 2022-09-12T19:11:48.903 DEBUG:boto:Data: 2022-09-12T19:11:48.903 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.904 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:48.904 DEBUG:boto:Port: 8001 2022-09-12T19:11:48.905 DEBUG:boto:Params: {} 2022-09-12T19:11:48.905 DEBUG:boto:Token: None 2022-09-12T19:11:48.906 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/ 2022-09-12T19:11:48.906 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iaiFgnZ3IiJlfZzcLCgsFyhbkX4= 2022-09-12T19:11:48.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iaiFgnZ3IiJlfZzcLCgsFyhbkX4=', 'Content-Length': '0'} 2022-09-12T19:11:48.909 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a594cecd02ee6c7-00631f8474-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.910 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:11:48.910 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:11:48.911 DEBUG:boto:Method: HEAD 2022-09-12T19:11:48.911 DEBUG:boto:Path: /jdpiak-23/ 2022-09-12T19:11:48.912 DEBUG:boto:Data: 2022-09-12T19:11:48.912 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.913 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:48.913 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.914 DEBUG:boto:Params: {} 2022-09-12T19:11:48.914 DEBUG:boto:Token: None 2022-09-12T19:11:48.915 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/ 2022-09-12T19:11:48.915 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iaiFgnZ3IiJlfZzcLCgsFyhbkX4= 2022-09-12T19:11:48.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iaiFgnZ3IiJlfZzcLCgsFyhbkX4=', 'Content-Length': '0'} 2022-09-12T19:11:48.918 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ded743071189abbc-00631f8474-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.919 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:11:48.920 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:11:48.920 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:11:48.921 DEBUG:boto:path=/jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.921 DEBUG:boto:auth_path=/jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.922 DEBUG:boto:Method: GET 2022-09-12T19:11:48.922 DEBUG:boto:Path: /jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.923 DEBUG:boto:Data: 2022-09-12T19:11:48.923 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.924 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:48.924 DEBUG:boto:Port: 8001 2022-09-12T19:11:48.925 DEBUG:boto:Params: {} 2022-09-12T19:11:48.925 DEBUG:boto:Token: None 2022-09-12T19:11:48.926 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/?versions 2022-09-12T19:11:48.926 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qqIx8v1E8w7GcIuqBacxrlKSyzY= 2022-09-12T19:11:48.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qqIx8v1E8w7GcIuqBacxrlKSyzY=', 'Content-Length': '0'} 2022-09-12T19:11:48.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004de18983ec91a2a3-00631f8474-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.930 DEBUG:boto:b'jdpiak-23999falseobjrKzI1doaCWXfUPsgqDJaJz0BenA63mMtrue2022-09-12T19:10:54.090Zrgw-multisite-test-userMultisite Test UserNormalobjTBH3pBU5gf-t2nym3FdhrHv6BSTFHv0false2022-09-12T19:10:54.060Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:48.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:48.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:48.932 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:11:48.933 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:11:48.933 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:11:48.934 DEBUG:boto:path=/jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.934 DEBUG:boto:auth_path=/jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.935 DEBUG:boto:Method: GET 2022-09-12T19:11:48.935 DEBUG:boto:Path: /jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.936 DEBUG:boto:Data: 2022-09-12T19:11:48.936 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.937 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:48.937 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.938 DEBUG:boto:Params: {} 2022-09-12T19:11:48.938 DEBUG:boto:Token: None 2022-09-12T19:11:48.939 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/?versions 2022-09-12T19:11:48.939 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qqIx8v1E8w7GcIuqBacxrlKSyzY= 2022-09-12T19:11:48.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qqIx8v1E8w7GcIuqBacxrlKSyzY=', 'Content-Length': '0'} 2022-09-12T19:11:48.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083b6818ff7107645-00631f8474-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.943 DEBUG:boto:b'jdpiak-23999falseobjrKzI1doaCWXfUPsgqDJaJz0BenA63mMtrue2022-09-12T19:10:54.090Zrgw-multisite-test-userMultisite Test UserNormalobjTBH3pBU5gf-t2nym3FdhrHv6BSTFHv0false2022-09-12T19:10:54.060Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:48.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:48.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:11:48.945 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:11:48.945 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:11:48.946 DEBUG:boto:path=/jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.947 DEBUG:boto:auth_path=/jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.947 DEBUG:boto:Method: GET 2022-09-12T19:11:48.948 DEBUG:boto:Path: /jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.948 DEBUG:boto:Data: 2022-09-12T19:11:48.949 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.949 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:48.950 DEBUG:boto:Port: 8001 2022-09-12T19:11:48.950 DEBUG:boto:Params: {} 2022-09-12T19:11:48.951 DEBUG:boto:Token: None 2022-09-12T19:11:48.952 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/?versions 2022-09-12T19:11:48.952 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qqIx8v1E8w7GcIuqBacxrlKSyzY= 2022-09-12T19:11:48.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qqIx8v1E8w7GcIuqBacxrlKSyzY=', 'Content-Length': '0'} 2022-09-12T19:11:48.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d246a995f1382f89-00631f8474-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:48.956 DEBUG:boto:b'jdpiak-23999falseobjrKzI1doaCWXfUPsgqDJaJz0BenA63mMtrue2022-09-12T19:10:54.090Zrgw-multisite-test-userMultisite Test UserNormalobjTBH3pBU5gf-t2nym3FdhrHv6BSTFHv0false2022-09-12T19:10:54.060Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:48.957 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:11:48.958 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:11:48.958 DEBUG:boto:path=/jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.959 DEBUG:boto:auth_path=/jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.959 DEBUG:boto:Method: GET 2022-09-12T19:11:48.960 DEBUG:boto:Path: /jdpiak-23/?versions&max-keys=999 2022-09-12T19:11:48.961 DEBUG:boto:Data: 2022-09-12T19:11:48.961 DEBUG:boto:Headers: {} 2022-09-12T19:11:48.962 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:48.962 DEBUG:boto:Port: 8000 2022-09-12T19:11:48.963 DEBUG:boto:Params: {} 2022-09-12T19:11:48.963 DEBUG:boto:Token: None 2022-09-12T19:11:48.964 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:48 GMT /jdpiak-23/?versions 2022-09-12T19:11:48.965 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qqIx8v1E8w7GcIuqBacxrlKSyzY= 2022-09-12T19:11:48.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:48 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qqIx8v1E8w7GcIuqBacxrlKSyzY=', 'Content-Length': '0'} 2022-09-12T19:11:48.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000beb2c3f899f7e1c6-00631f8474-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:49.010 DEBUG:boto:b'jdpiak-23999falseobjrKzI1doaCWXfUPsgqDJaJz0BenA63mMtrue2022-09-12T19:10:54.090Zrgw-multisite-test-userMultisite Test UserNormalobjTBH3pBU5gf-t2nym3FdhrHv6BSTFHv0false2022-09-12T19:10:54.060Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:11:49.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:49.013 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:49.014 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:49.015 DEBUG:boto:Method: HEAD 2022-09-12T19:11:49.016 DEBUG:boto:Path: /jdpiak-23/obj 2022-09-12T19:11:49.017 DEBUG:boto:Data: 2022-09-12T19:11:49.018 DEBUG:boto:Headers: {} 2022-09-12T19:11:49.019 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:49.020 DEBUG:boto:Port: 8001 2022-09-12T19:11:49.021 DEBUG:boto:Params: {} 2022-09-12T19:11:49.022 DEBUG:boto:Token: None 2022-09-12T19:11:49.023 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:49 GMT /jdpiak-23/obj 2022-09-12T19:11:49.024 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uComJATT1unenkyhFhvuIXrTbiI= 2022-09-12T19:11:49.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uComJATT1unenkyhFhvuIXrTbiI=', 'Content-Length': '0'} 2022-09-12T19:11:49.027 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000009fc17f1931fb3948-00631f8475-10f2-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:49.028 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:49.029 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:49.030 DEBUG:boto:Method: HEAD 2022-09-12T19:11:49.031 DEBUG:boto:Path: /jdpiak-23/obj 2022-09-12T19:11:49.031 DEBUG:boto:Data: 2022-09-12T19:11:49.032 DEBUG:boto:Headers: {} 2022-09-12T19:11:49.032 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:49.033 DEBUG:boto:Port: 8000 2022-09-12T19:11:49.033 DEBUG:boto:Params: {} 2022-09-12T19:11:49.034 DEBUG:boto:Token: None 2022-09-12T19:11:49.035 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:49 GMT /jdpiak-23/obj 2022-09-12T19:11:49.035 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uComJATT1unenkyhFhvuIXrTbiI= 2022-09-12T19:11:49.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uComJATT1unenkyhFhvuIXrTbiI=', 'Content-Length': '0'} 2022-09-12T19:11:49.038 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000008a363426305cde4b-00631f8475-10e4-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:49.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:49.039 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:49.040 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:49.040 DEBUG:boto:path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:49.041 DEBUG:boto:auth_path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:49.042 DEBUG:boto:Method: GET 2022-09-12T19:11:49.042 DEBUG:boto:Path: /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:49.043 DEBUG:boto:Data: 2022-09-12T19:11:49.043 DEBUG:boto:Headers: {} 2022-09-12T19:11:49.044 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:49.044 DEBUG:boto:Port: 8001 2022-09-12T19:11:49.045 DEBUG:boto:Params: {} 2022-09-12T19:11:49.045 DEBUG:boto:Token: None 2022-09-12T19:11:49.046 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:49 GMT /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:49.047 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FGthtKGN61gBOaaACGstqaNVjJo= 2022-09-12T19:11:49.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FGthtKGN61gBOaaACGstqaNVjJo=', 'Content-Length': '0'} 2022-09-12T19:11:49.049 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:54 GMT'), ('x-amz-version-id', 'TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006c2e7c648f1919c2-00631f8475-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:49.050 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:49.051 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:49.051 DEBUG:boto:path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:49.052 DEBUG:boto:auth_path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:49.052 DEBUG:boto:Method: GET 2022-09-12T19:11:49.053 DEBUG:boto:Path: /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:49.053 DEBUG:boto:Data: 2022-09-12T19:11:49.054 DEBUG:boto:Headers: {} 2022-09-12T19:11:49.055 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:49.056 DEBUG:boto:Port: 8000 2022-09-12T19:11:49.057 DEBUG:boto:Params: {} 2022-09-12T19:11:49.057 DEBUG:boto:Token: None 2022-09-12T19:11:49.058 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:49 GMT /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:49.059 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FGthtKGN61gBOaaACGstqaNVjJo= 2022-09-12T19:11:49.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FGthtKGN61gBOaaACGstqaNVjJo=', 'Content-Length': '0'} 2022-09-12T19:11:49.062 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:54 GMT'), ('x-amz-version-id', 'TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aeb279b653e252a3-00631f8475-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:49.064 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:49.064 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:49.065 DEBUG:boto:path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:49.066 DEBUG:boto:auth_path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:49.067 DEBUG:boto:Method: HEAD 2022-09-12T19:11:49.068 DEBUG:boto:Path: /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:49.069 DEBUG:boto:Data: 2022-09-12T19:11:49.069 DEBUG:boto:Headers: {} 2022-09-12T19:11:49.070 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:49.071 DEBUG:boto:Port: 8001 2022-09-12T19:11:49.071 DEBUG:boto:Params: {} 2022-09-12T19:11:49.072 DEBUG:boto:Token: None 2022-09-12T19:11:49.073 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:49 GMT /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:49.074 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UqISw8lzKe0xIku6lfgEXpD0yE8= 2022-09-12T19:11:49.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UqISw8lzKe0xIku6lfgEXpD0yE8=', 'Content-Length': '0'} 2022-09-12T19:11:49.077 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:54 GMT'), ('x-amz-version-id', 'TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a2ef9d3936ba51cb-00631f8475-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:49.077 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:49.078 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:49.079 DEBUG:boto:path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:49.079 DEBUG:boto:auth_path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:49.080 DEBUG:boto:Method: HEAD 2022-09-12T19:11:49.080 DEBUG:boto:Path: /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:49.081 DEBUG:boto:Data: 2022-09-12T19:11:49.081 DEBUG:boto:Headers: {} 2022-09-12T19:11:49.082 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:49.082 DEBUG:boto:Port: 8000 2022-09-12T19:11:49.083 DEBUG:boto:Params: {} 2022-09-12T19:11:49.083 DEBUG:boto:Token: None 2022-09-12T19:11:49.084 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:49 GMT /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:49.085 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UqISw8lzKe0xIku6lfgEXpD0yE8= 2022-09-12T19:11:49.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UqISw8lzKe0xIku6lfgEXpD0yE8=', 'Content-Length': '0'} 2022-09-12T19:11:49.087 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:54 GMT'), ('x-amz-version-id', 'TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001b1abca90445614d-00631f8475-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:49.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:11:49.089 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:49.089 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:49.090 DEBUG:boto:path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:49.091 DEBUG:boto:auth_path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:49.091 DEBUG:boto:Method: GET 2022-09-12T19:11:49.092 DEBUG:boto:Path: /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:49.092 DEBUG:boto:Data: 2022-09-12T19:11:49.093 DEBUG:boto:Headers: {} 2022-09-12T19:11:49.093 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:49.094 DEBUG:boto:Port: 8001 2022-09-12T19:11:49.094 DEBUG:boto:Params: {} 2022-09-12T19:11:49.095 DEBUG:boto:Token: None 2022-09-12T19:11:49.096 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:49 GMT /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:49.096 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FGthtKGN61gBOaaACGstqaNVjJo= 2022-09-12T19:11:49.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FGthtKGN61gBOaaACGstqaNVjJo=', 'Content-Length': '0'} 2022-09-12T19:11:49.099 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:54 GMT'), ('x-amz-version-id', 'TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ba8be36d01d3510b-00631f8475-10f2-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:49.100 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:49.101 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:49.101 DEBUG:boto:path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:49.102 DEBUG:boto:auth_path=/jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:49.102 DEBUG:boto:Method: GET 2022-09-12T19:11:49.103 DEBUG:boto:Path: /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:49.103 DEBUG:boto:Data: 2022-09-12T19:11:49.104 DEBUG:boto:Headers: {} 2022-09-12T19:11:49.104 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:49.105 DEBUG:boto:Port: 8000 2022-09-12T19:11:49.106 DEBUG:boto:Params: {} 2022-09-12T19:11:49.106 DEBUG:boto:Token: None 2022-09-12T19:11:49.107 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:49 GMT /jdpiak-23/obj?versionId=TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0 2022-09-12T19:11:49.108 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FGthtKGN61gBOaaACGstqaNVjJo= 2022-09-12T19:11:49.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FGthtKGN61gBOaaACGstqaNVjJo=', 'Content-Length': '0'} 2022-09-12T19:11:49.111 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:10:54 GMT'), ('x-amz-version-id', 'TBH3pBU5gf-t2nym3FdhrHv6BSTFHv0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000005098473d387aab3-00631f8475-10e4-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:11:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:49.111 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:11:49.112 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:11:49.112 DEBUG:boto:path=/jdpiak-23/?prefix=obj 2022-09-12T19:11:49.113 DEBUG:boto:auth_path=/jdpiak-23/?prefix=obj 2022-09-12T19:11:49.113 DEBUG:boto:Method: GET 2022-09-12T19:11:49.114 DEBUG:boto:Path: /jdpiak-23/?prefix=obj 2022-09-12T19:11:49.115 DEBUG:boto:Data: 2022-09-12T19:11:49.115 DEBUG:boto:Headers: {} 2022-09-12T19:11:49.116 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:49.116 DEBUG:boto:Port: 8001 2022-09-12T19:11:49.117 DEBUG:boto:Params: {} 2022-09-12T19:11:49.117 DEBUG:boto:Token: None 2022-09-12T19:11:49.118 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:49 GMT /jdpiak-23/ 2022-09-12T19:11:49.119 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hWq+mmYMjMGPLe53vo4LwOOg5NA= 2022-09-12T19:11:49.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hWq+mmYMjMGPLe53vo4LwOOg5NA=', 'Content-Length': '0'} 2022-09-12T19:11:49.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007379c717b21651fb-00631f8475-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:49.123 DEBUG:boto:b'jdpiak-23obj1000false' 2022-09-12T19:11:49.124 DEBUG:boto:path=/jdpiak-23/ 2022-09-12T19:11:49.124 DEBUG:boto:auth_path=/jdpiak-23/ 2022-09-12T19:11:49.125 DEBUG:boto:path=/jdpiak-23/?prefix=obj 2022-09-12T19:11:49.125 DEBUG:boto:auth_path=/jdpiak-23/?prefix=obj 2022-09-12T19:11:49.126 DEBUG:boto:Method: GET 2022-09-12T19:11:49.127 DEBUG:boto:Path: /jdpiak-23/?prefix=obj 2022-09-12T19:11:49.127 DEBUG:boto:Data: 2022-09-12T19:11:49.128 DEBUG:boto:Headers: {} 2022-09-12T19:11:49.128 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:49.129 DEBUG:boto:Port: 8000 2022-09-12T19:11:49.129 DEBUG:boto:Params: {} 2022-09-12T19:11:49.130 DEBUG:boto:Token: None 2022-09-12T19:11:49.131 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:11:49 GMT /jdpiak-23/ 2022-09-12T19:11:49.131 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hWq+mmYMjMGPLe53vo4LwOOg5NA= 2022-09-12T19:11:49.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hWq+mmYMjMGPLe53vo4LwOOg5NA=', 'Content-Length': '0'} 2022-09-12T19:11:49.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3e2f2a2c1c6a3e3-00631f8475-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:49.178 DEBUG:boto:b'jdpiak-23obj1000false' 2022-09-12T19:11:49.179 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:49.180 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:49.180 DEBUG:boto:Method: HEAD 2022-09-12T19:11:49.181 DEBUG:boto:Path: /jdpiak-23/obj 2022-09-12T19:11:49.181 DEBUG:boto:Data: 2022-09-12T19:11:49.182 DEBUG:boto:Headers: {} 2022-09-12T19:11:49.182 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:49.183 DEBUG:boto:Port: 8001 2022-09-12T19:11:49.183 DEBUG:boto:Params: {} 2022-09-12T19:11:49.184 DEBUG:boto:Token: None 2022-09-12T19:11:49.185 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:49 GMT /jdpiak-23/obj 2022-09-12T19:11:49.185 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uComJATT1unenkyhFhvuIXrTbiI= 2022-09-12T19:11:49.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uComJATT1unenkyhFhvuIXrTbiI=', 'Content-Length': '0'} 2022-09-12T19:11:49.188 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000eefbbdc2e6e36aa3-00631f8475-10f2-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:49.189 DEBUG:boto:path=/jdpiak-23/obj 2022-09-12T19:11:49.189 DEBUG:boto:auth_path=/jdpiak-23/obj 2022-09-12T19:11:49.190 DEBUG:boto:Method: HEAD 2022-09-12T19:11:49.191 DEBUG:boto:Path: /jdpiak-23/obj 2022-09-12T19:11:49.191 DEBUG:boto:Data: 2022-09-12T19:11:49.192 DEBUG:boto:Headers: {} 2022-09-12T19:11:49.192 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:49.193 DEBUG:boto:Port: 8000 2022-09-12T19:11:49.193 DEBUG:boto:Params: {} 2022-09-12T19:11:49.194 DEBUG:boto:Token: None 2022-09-12T19:11:49.194 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:49 GMT /jdpiak-23/obj 2022-09-12T19:11:49.195 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uComJATT1unenkyhFhvuIXrTbiI= 2022-09-12T19:11:49.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uComJATT1unenkyhFhvuIXrTbiI=', 'Content-Length': '0'} 2022-09-12T19:11:49.198 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000d1e65104080bdef2-00631f8475-10e4-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:11:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:49.199 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-23 zones={test-zone3, test-zone2} 2022-09-12T19:11:49.200 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2022-09-12T19:11:49.201 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-24 2022-09-12T19:11:49.201 DEBUG:boto:path=/jdpiak-24/ 2022-09-12T19:11:49.202 DEBUG:boto:auth_path=/jdpiak-24/ 2022-09-12T19:11:49.202 DEBUG:boto:Method: PUT 2022-09-12T19:11:49.203 DEBUG:boto:Path: /jdpiak-24/ 2022-09-12T19:11:49.203 DEBUG:boto:Data: 2022-09-12T19:11:49.204 DEBUG:boto:Headers: {} 2022-09-12T19:11:49.204 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:49.205 DEBUG:boto:Port: 8000 2022-09-12T19:11:49.205 DEBUG:boto:Params: {} 2022-09-12T19:11:49.206 DEBUG:boto:Token: None 2022-09-12T19:11:49.207 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:11:49 GMT /jdpiak-24/ 2022-09-12T19:11:49.207 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NNzJ3HC7L2F5enopkVNvpnRRXJo= 2022-09-12T19:11:49.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NNzJ3HC7L2F5enopkVNvpnRRXJo=', 'Content-Length': '0'} 2022-09-12T19:11:49.225 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001508bfb30ea059f7-00631f8475-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:49.226 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jdpiak-25 2022-09-12T19:11:49.226 DEBUG:boto:path=/jdpiak-25/ 2022-09-12T19:11:49.227 DEBUG:boto:auth_path=/jdpiak-25/ 2022-09-12T19:11:49.227 DEBUG:boto:Method: PUT 2022-09-12T19:11:49.228 DEBUG:boto:Path: /jdpiak-25/ 2022-09-12T19:11:49.228 DEBUG:boto:Data: 2022-09-12T19:11:49.229 DEBUG:boto:Headers: {} 2022-09-12T19:11:49.229 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:49.230 DEBUG:boto:Port: 8000 2022-09-12T19:11:49.231 DEBUG:boto:Params: {} 2022-09-12T19:11:49.231 DEBUG:boto:Token: None 2022-09-12T19:11:49.232 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:11:49 GMT /jdpiak-25/ 2022-09-12T19:11:49.233 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BnZz4bzQy4MWe2dnk+oQIuL3ZQs= 2022-09-12T19:11:49.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BnZz4bzQy4MWe2dnk+oQIuL3ZQs=', 'Content-Length': '0'} 2022-09-12T19:11:49.265 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004051a7d9607fd84b-00631f8475-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:49.266 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jdpiak-26 2022-09-12T19:11:49.267 DEBUG:boto:path=/jdpiak-26/ 2022-09-12T19:11:49.267 DEBUG:boto:auth_path=/jdpiak-26/ 2022-09-12T19:11:49.268 DEBUG:boto:Method: PUT 2022-09-12T19:11:49.269 DEBUG:boto:Path: /jdpiak-26/ 2022-09-12T19:11:49.269 DEBUG:boto:Data: 2022-09-12T19:11:49.270 DEBUG:boto:Headers: {} 2022-09-12T19:11:49.270 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:49.271 DEBUG:boto:Port: 8001 2022-09-12T19:11:49.271 DEBUG:boto:Params: {} 2022-09-12T19:11:49.272 DEBUG:boto:Token: None 2022-09-12T19:11:49.272 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:11:49 GMT /jdpiak-26/ 2022-09-12T19:11:49.273 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hmgI5e8szWfEFYYA+WGA6GBijdQ= 2022-09-12T19:11:49.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hmgI5e8szWfEFYYA+WGA6GBijdQ=', 'Content-Length': '0'} 2022-09-12T19:11:49.301 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bafc66c7b590dacd-00631f8475-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:49.302 DEBUG:boto:path=/jdpiak-24/ 2022-09-12T19:11:49.302 DEBUG:boto:auth_path=/jdpiak-24/ 2022-09-12T19:11:49.303 DEBUG:boto:path=/jdpiak-24/?versioning 2022-09-12T19:11:49.303 DEBUG:boto:auth_path=/jdpiak-24/?versioning 2022-09-12T19:11:49.304 DEBUG:boto:Method: PUT 2022-09-12T19:11:49.304 DEBUG:boto:Path: /jdpiak-24/?versioning 2022-09-12T19:11:49.305 DEBUG:boto:Data: Enabled Disabled 2022-09-12T19:11:49.305 DEBUG:boto:Headers: {} 2022-09-12T19:11:49.306 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:49.306 DEBUG:boto:Port: 8000 2022-09-12T19:11:49.307 DEBUG:boto:Params: {} 2022-09-12T19:11:49.307 DEBUG:boto:Token: None 2022-09-12T19:11:49.308 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:11:49 GMT /jdpiak-24/?versioning 2022-09-12T19:11:49.308 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZmYEJVmQYl1WP7yCmhpJWPLfFWI= 2022-09-12T19:11:49.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZmYEJVmQYl1WP7yCmhpJWPLfFWI=', 'Content-Length': '228'} 2022-09-12T19:11:49.314 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000028df840ab12e37c1-00631f8475-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:49.315 DEBUG:boto:path=/jdpiak-24/ 2022-09-12T19:11:49.315 DEBUG:boto:auth_path=/jdpiak-24/ 2022-09-12T19:11:49.316 DEBUG:boto:path=/jdpiak-24/?versioning 2022-09-12T19:11:49.316 DEBUG:boto:auth_path=/jdpiak-24/?versioning 2022-09-12T19:11:49.317 DEBUG:boto:Method: PUT 2022-09-12T19:11:49.317 DEBUG:boto:Path: /jdpiak-24/?versioning 2022-09-12T19:11:49.318 DEBUG:boto:Data: Suspended Disabled 2022-09-12T19:11:49.318 DEBUG:boto:Headers: {} 2022-09-12T19:11:49.319 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:49.319 DEBUG:boto:Port: 8000 2022-09-12T19:11:49.320 DEBUG:boto:Params: {} 2022-09-12T19:11:49.320 DEBUG:boto:Token: None 2022-09-12T19:11:49.321 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:11:49 GMT /jdpiak-24/?versioning 2022-09-12T19:11:49.322 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZmYEJVmQYl1WP7yCmhpJWPLfFWI= 2022-09-12T19:11:49.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZmYEJVmQYl1WP7yCmhpJWPLfFWI=', 'Content-Length': '230'} 2022-09-12T19:11:49.340 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000858125ba2da56a3f-00631f8475-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:49.341 DEBUG:boto:path=/jdpiak-25/ 2022-09-12T19:11:49.342 DEBUG:boto:auth_path=/jdpiak-25/ 2022-09-12T19:11:49.342 DEBUG:boto:path=/jdpiak-25/?versioning 2022-09-12T19:11:49.343 DEBUG:boto:auth_path=/jdpiak-25/?versioning 2022-09-12T19:11:49.343 DEBUG:boto:Method: PUT 2022-09-12T19:11:49.344 DEBUG:boto:Path: /jdpiak-25/?versioning 2022-09-12T19:11:49.344 DEBUG:boto:Data: Enabled Disabled 2022-09-12T19:11:49.345 DEBUG:boto:Headers: {} 2022-09-12T19:11:49.345 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:49.346 DEBUG:boto:Port: 8000 2022-09-12T19:11:49.347 DEBUG:boto:Params: {} 2022-09-12T19:11:49.347 DEBUG:boto:Token: None 2022-09-12T19:11:49.348 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:11:49 GMT /jdpiak-25/?versioning 2022-09-12T19:11:49.349 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZDBGAMvfp3O4p/vEAJ/71arKVWU= 2022-09-12T19:11:49.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZDBGAMvfp3O4p/vEAJ/71arKVWU=', 'Content-Length': '228'} 2022-09-12T19:11:49.360 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000049711c619c396a9c-00631f8475-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:49.361 DEBUG:boto:path=/jdpiak-25/ 2022-09-12T19:11:49.362 DEBUG:boto:auth_path=/jdpiak-25/ 2022-09-12T19:11:49.362 DEBUG:boto:path=/jdpiak-25/?versioning 2022-09-12T19:11:49.363 DEBUG:boto:auth_path=/jdpiak-25/?versioning 2022-09-12T19:11:49.363 DEBUG:boto:Method: PUT 2022-09-12T19:11:49.364 DEBUG:boto:Path: /jdpiak-25/?versioning 2022-09-12T19:11:49.364 DEBUG:boto:Data: Suspended Disabled 2022-09-12T19:11:49.365 DEBUG:boto:Headers: {} 2022-09-12T19:11:49.365 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:49.365 DEBUG:boto:Port: 8000 2022-09-12T19:11:49.366 DEBUG:boto:Params: {} 2022-09-12T19:11:49.367 DEBUG:boto:Token: None 2022-09-12T19:11:49.367 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:11:49 GMT /jdpiak-25/?versioning 2022-09-12T19:11:49.368 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZDBGAMvfp3O4p/vEAJ/71arKVWU= 2022-09-12T19:11:49.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZDBGAMvfp3O4p/vEAJ/71arKVWU=', 'Content-Length': '230'} 2022-09-12T19:11:49.379 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000093c7ff808f90ffc5-00631f8475-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:49.380 DEBUG:boto:path=/jdpiak-26/ 2022-09-12T19:11:49.381 DEBUG:boto:auth_path=/jdpiak-26/ 2022-09-12T19:11:49.382 DEBUG:boto:path=/jdpiak-26/?versioning 2022-09-12T19:11:49.383 DEBUG:boto:auth_path=/jdpiak-26/?versioning 2022-09-12T19:11:49.384 DEBUG:boto:Method: PUT 2022-09-12T19:11:49.385 DEBUG:boto:Path: /jdpiak-26/?versioning 2022-09-12T19:11:49.386 DEBUG:boto:Data: Enabled Disabled 2022-09-12T19:11:49.387 DEBUG:boto:Headers: {} 2022-09-12T19:11:49.387 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:49.389 DEBUG:boto:Port: 8001 2022-09-12T19:11:49.389 DEBUG:boto:Params: {} 2022-09-12T19:11:49.391 DEBUG:boto:Token: None 2022-09-12T19:11:49.391 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:11:49 GMT /jdpiak-26/?versioning 2022-09-12T19:11:49.392 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fjhaUNA2iR8bFb9xtZWJitHyVS0= 2022-09-12T19:11:49.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fjhaUNA2iR8bFb9xtZWJitHyVS0=', 'Content-Length': '228'} 2022-09-12T19:11:49.405 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000025db04f504d977f2-00631f8475-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:49.406 DEBUG:boto:path=/jdpiak-26/ 2022-09-12T19:11:49.407 DEBUG:boto:auth_path=/jdpiak-26/ 2022-09-12T19:11:49.407 DEBUG:boto:path=/jdpiak-26/?versioning 2022-09-12T19:11:49.408 DEBUG:boto:auth_path=/jdpiak-26/?versioning 2022-09-12T19:11:49.408 DEBUG:boto:Method: PUT 2022-09-12T19:11:49.409 DEBUG:boto:Path: /jdpiak-26/?versioning 2022-09-12T19:11:49.409 DEBUG:boto:Data: Suspended Disabled 2022-09-12T19:11:49.410 DEBUG:boto:Headers: {} 2022-09-12T19:11:49.411 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:49.412 DEBUG:boto:Port: 8001 2022-09-12T19:11:49.413 DEBUG:boto:Params: {} 2022-09-12T19:11:49.414 DEBUG:boto:Token: None 2022-09-12T19:11:49.415 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:11:49 GMT /jdpiak-26/?versioning 2022-09-12T19:11:49.416 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fjhaUNA2iR8bFb9xtZWJitHyVS0= 2022-09-12T19:11:49.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:49 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fjhaUNA2iR8bFb9xtZWJitHyVS0=', 'Content-Length': '230'} 2022-09-12T19:11:49.427 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fafd64c337d2fe99-00631f8475-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:49.428 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:11:49.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:11:49.430 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:11:49.486 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:49.487 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:49.530 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=c0f7a512-c940-4d1f-8adf-3484630f876b 2022-09-12T19:11:49.542 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:11:49.542 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:11:49.543 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:11:49.543 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:47:27.976965Z" 2022-09-12T19:11:49.543 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:11:49.543 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:11:49.544 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:11:49.544 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:11:49.544 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:11:49.544 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:11:49.545 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663009909.421418_254.1", 2022-09-12T19:11:49.545 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:11:49.421418Z" 2022-09-12T19:11:49.545 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:11:49.545 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:11:49.546 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:11:49.546 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:49:40.904048Z" 2022-09-12T19:11:49.546 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:11:49.546 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:11:49.548 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009909.421418_254.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:11:49.548 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T19:11:49.549 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:11:49.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:11:49.550 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:11:49.599 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:49.600 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:49.655 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:49.652+0000 7f0a1596d600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:11:49.656 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:49.653+0000 7f0a1596d600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:11:49.674 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:11:49.675 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:11:49.676 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:11:49.676 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:11:49.676 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:11:49.677 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:11:49.677 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:11:49.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:11:49.678 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:11:49.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:11:49.679 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:11:49.680 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:11:49.680 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:11:49.681 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:11:49.681 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:11:49.681 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:11:49.681 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:11:49.682 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:11:49.682 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:11:49.682 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:11:49.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:11:49.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:11:49.683 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:11:49.683 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:11:49.683 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:11:49.684 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:11:49.684 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:11:49.684 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:11:49.684 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:11:49.685 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:11:49.685 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:11:49.685 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:11:49.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:11:49.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:11:49.686 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:11:49.686 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:11:49.687 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:11:49.687 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663009909.309215_243.1", 2022-09-12T19:11:49.687 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:11:49.687 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:11:49.688 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:11:49.688 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.309215Z", 2022-09-12T19:11:49.688 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:11:49.688 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:11:49.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:11:49.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:11:49.689 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:11:49.690 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:11:49.690 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:11:49.690 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:11:49.690 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:11:49.691 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:11:49.691 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:11:49.691 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:11:49.691 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:11:49.692 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:11:49.692 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:11:49.692 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:11:49.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:11:49.693 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:11:49.693 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:11:49.693 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:11:49.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:11:49.694 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:11:49Z" 2022-09-12T19:11:49.694 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:11:49.695 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009909.309215_243.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:11:49.309215Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:11:49Z" } 2022-09-12T19:11:49.696 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009909.309215_243.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:11:49.309215Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:11:49.696 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009909.421418_254.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:11:49.697 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009909.309215_243.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:11:49.697 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663009909.421418_254.1 target=1_1663009909.309215_243.1 2022-09-12T19:11:54.699 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:11:54.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:11:54.701 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:11:54.758 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:54.759 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:54.796 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:54.793+0000 7fcb32cc1600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:11:54.797 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:54.794+0000 7fcb32cc1600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:11:54.813 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:11:54.814 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:11:54.814 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:11:54.814 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:11:54.815 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:11:54.815 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:11:54.815 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:11:54.816 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:11:54.816 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:11:54.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:11:54.816 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:11:54.817 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:11:54.817 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:11:54.817 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:11:54.818 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:11:54.818 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:11:54.818 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:11:54.818 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:11:54.819 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:11:54.819 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:11:54.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:11:54.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:11:54.820 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:11:54.820 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:11:54.820 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:11:54.821 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:11:54.821 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:11:54.822 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:11:54.822 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:11:54.822 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:11:54.823 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:11:54.823 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:11:54.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:11:54.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:11:54.825 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:11:54.825 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:11:54.826 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:11:54.826 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663009909.421418_254.1", 2022-09-12T19:11:54.827 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:11:54.827 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:11:54.828 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:11:54.828 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.421418Z", 2022-09-12T19:11:54.829 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:11:54.829 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:11:54.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:11:54.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:11:54.830 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:11:54.831 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:11:54.831 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:11:54.832 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:11:54.832 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:11:54.833 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:11:54.833 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:11:54.833 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:11:54.833 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:11:54.834 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:11:54.834 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:11:54.834 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:11:54.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:11:54.835 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:11:54.835 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:11:54.835 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:11:54.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:11:54.836 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:11:54Z" 2022-09-12T19:11:54.836 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:11:54.837 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009909.421418_254.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:11:49.421418Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:11:54Z" } 2022-09-12T19:11:54.838 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009909.421418_254.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:11:49.421418Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:11:54.839 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009909.421418_254.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:11:54.839 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009909.421418_254.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:11:54.840 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T19:11:54.840 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T19:11:54.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:11:54.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:11:54.842 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:11:54.902 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:54.902 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:54.944 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:54.942+0000 7f4105999600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:11:54.945 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:11:54.943+0000 7f4105999600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:11:54.963 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:11:54.963 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:11:54.964 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:11:54.964 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:11:54.964 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:11:54.965 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:11:54.965 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:11:54.965 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:11:54.965 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:11:54.966 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:11:54.966 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:11:54.966 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:11:54.966 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:11:54.967 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:11:54.967 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:11:54.967 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:11:54.967 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:11:54.968 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:11:54.968 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:11:54.968 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:11:54.968 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:11:54.969 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:11:54.969 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:11:54.969 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:11:54.969 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:11:54.970 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:11:54.970 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:11:54.970 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:11:54.970 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:11:54.971 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:11:54.971 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:11:54.971 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:11:54.971 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:11:54.972 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:11:54.972 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:11:54.972 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:11:54.972 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:11:54.973 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663009909.421418_254.1", 2022-09-12T19:11:54.973 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:11:54.973 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:11:54.973 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:11:54.974 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.421418Z", 2022-09-12T19:11:54.974 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:11:54.974 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:11:54.975 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:11:54.975 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:11:54.975 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:11:54.975 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:11:54.976 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:11:54.976 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:11:54.976 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:11:54.976 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:11:54.977 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:11:54.977 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:11:54.977 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:11:54.977 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:11:54.978 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:11:54.978 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:11:54.978 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:11:54.978 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:11:54.979 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:11:54.979 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:11:54.979 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:11:54.979 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:11:54Z" 2022-09-12T19:11:54.980 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:11:54.981 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009909.421418_254.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:11:49.421418Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:11:54Z" } 2022-09-12T19:11:54.982 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009909.421418_254.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:11:49.421418Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:11:54.982 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009909.421418_254.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:11:54.983 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009909.421418_254.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:11:54.983 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T19:11:54.984 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T19:11:54.984 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:11:54.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:11:54.985 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:11:55.036 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:55.037 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:55.081 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:55.078+0000 7fbb0d23e600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:11:55.082 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:55.079+0000 7fbb0d23e600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:11:55.101 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:11:55.101 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:11:55.102 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:11:55.102 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:11:55.102 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:11:55.103 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:11:55.103 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:11:55.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:11:55.104 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:11:55.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:11:55.104 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:11:55.104 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:11:55.105 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:11:55.105 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:11:55.105 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:11:55.105 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:11:55.106 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:11:55.106 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:11:55.106 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:11:55.106 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:11:55.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:11:55.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:11:55.107 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:11:55.107 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:11:55.108 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:11:55.108 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:11:55.108 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:11:55.108 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:11:55.109 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:11:55.109 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:11:55.109 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:11:55.109 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:11:55.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:11:55.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:11:55.110 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:11:55.111 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:11:55.111 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:11:55.111 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663009909.421418_254.1", 2022-09-12T19:11:55.111 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:11:55.112 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:11:55.112 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:11:55.112 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.421418Z", 2022-09-12T19:11:55.112 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:11:55.113 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:11:55.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:11:55.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:11:55.113 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:11:55.114 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:11:55.114 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:11:55.114 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:11:55.114 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:11:55.115 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:11:55.115 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:11:55.115 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:11:55.115 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:11:55.115 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:11:55.116 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:11:55.116 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:11:55.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:11:55.116 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:11:55.117 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:11:55.117 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:11:55.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:11:55.117 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:11:55Z" 2022-09-12T19:11:55.118 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:11:55.119 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663009909.421418_254.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:11:49.421418Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:11:55Z" } 2022-09-12T19:11:55.120 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663009909.421418_254.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:11:49.421418Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:11:55.120 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009909.421418_254.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:11:55.121 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663009909.421418_254.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:11:55.121 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T19:11:55.122 DEBUG:boto:path=/jdpiak-24/ 2022-09-12T19:11:55.122 DEBUG:boto:auth_path=/jdpiak-24/ 2022-09-12T19:11:55.123 DEBUG:boto:Method: HEAD 2022-09-12T19:11:55.123 DEBUG:boto:Path: /jdpiak-24/ 2022-09-12T19:11:55.124 DEBUG:boto:Data: 2022-09-12T19:11:55.124 DEBUG:boto:Headers: {} 2022-09-12T19:11:55.125 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:55.125 DEBUG:boto:Port: 8000 2022-09-12T19:11:55.126 DEBUG:boto:Params: {} 2022-09-12T19:11:55.126 DEBUG:boto:Token: None 2022-09-12T19:11:55.127 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:55 GMT /jdpiak-24/ 2022-09-12T19:11:55.127 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4YhMOGfFUmU4mXmGOVlgeOEh6Wg= 2022-09-12T19:11:55.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4YhMOGfFUmU4mXmGOVlgeOEh6Wg=', 'Content-Length': '0'} 2022-09-12T19:11:55.130 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf73bf2a324b256b-00631f847b-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:55.131 DEBUG:boto:path=/jdpiak-24/obj 2022-09-12T19:11:55.132 DEBUG:boto:auth_path=/jdpiak-24/obj 2022-09-12T19:11:55.132 DEBUG:boto:Method: PUT 2022-09-12T19:11:55.133 DEBUG:boto:Path: /jdpiak-24/obj 2022-09-12T19:11:55.133 DEBUG:boto:Data: 2022-09-12T19:11:55.134 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-12T19:11:55.134 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:55.135 DEBUG:boto:Port: 8000 2022-09-12T19:11:55.135 DEBUG:boto:Params: {} 2022-09-12T19:11:55.136 DEBUG:boto:Token: None 2022-09-12T19:11:55.136 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 12 Sep 2022 19:11:55 GMT /jdpiak-24/obj 2022-09-12T19:11:55.137 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PjWfNfmZ2QqrditwXKfCd4e21KQ= 2022-09-12T19:11:55.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:11:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PjWfNfmZ2QqrditwXKfCd4e21KQ='} 2022-09-12T19:11:55.144 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000007af049dc7f4378e-00631f847b-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:11:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:55.144 DEBUG:boto:path=/jdpiak-24/ 2022-09-12T19:11:55.145 DEBUG:boto:auth_path=/jdpiak-24/ 2022-09-12T19:11:55.146 DEBUG:boto:Method: HEAD 2022-09-12T19:11:55.146 DEBUG:boto:Path: /jdpiak-24/ 2022-09-12T19:11:55.147 DEBUG:boto:Data: 2022-09-12T19:11:55.147 DEBUG:boto:Headers: {} 2022-09-12T19:11:55.147 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:55.148 DEBUG:boto:Port: 8000 2022-09-12T19:11:55.149 DEBUG:boto:Params: {} 2022-09-12T19:11:55.150 DEBUG:boto:Token: None 2022-09-12T19:11:55.150 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:55 GMT /jdpiak-24/ 2022-09-12T19:11:55.151 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4YhMOGfFUmU4mXmGOVlgeOEh6Wg= 2022-09-12T19:11:55.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4YhMOGfFUmU4mXmGOVlgeOEh6Wg=', 'Content-Length': '0'} 2022-09-12T19:11:55.154 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8cf7e6419aa7fa9-00631f847b-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:55.154 DEBUG:boto:path=/jdpiak-24/obj 2022-09-12T19:11:55.155 DEBUG:boto:auth_path=/jdpiak-24/obj 2022-09-12T19:11:55.155 DEBUG:boto:Method: DELETE 2022-09-12T19:11:55.156 DEBUG:boto:Path: /jdpiak-24/obj 2022-09-12T19:11:55.156 DEBUG:boto:Data: 2022-09-12T19:11:55.157 DEBUG:boto:Headers: {} 2022-09-12T19:11:55.157 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:11:55.158 DEBUG:boto:Port: 8000 2022-09-12T19:11:55.158 DEBUG:boto:Params: {} 2022-09-12T19:11:55.159 DEBUG:boto:Token: None 2022-09-12T19:11:55.159 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 19:11:55 GMT /jdpiak-24/obj 2022-09-12T19:11:55.160 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dCpuQwVypI1RXr8yYH4/3H5wn9g= 2022-09-12T19:11:55.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dCpuQwVypI1RXr8yYH4/3H5wn9g=', 'Content-Length': '0'} 2022-09-12T19:11:55.170 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000d2fb8b35b4f64883-00631f847b-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:11:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:55.170 DEBUG:boto:path=/jdpiak-25/ 2022-09-12T19:11:55.171 DEBUG:boto:auth_path=/jdpiak-25/ 2022-09-12T19:11:55.172 DEBUG:boto:Method: HEAD 2022-09-12T19:11:55.172 DEBUG:boto:Path: /jdpiak-25/ 2022-09-12T19:11:55.172 DEBUG:boto:Data: 2022-09-12T19:11:55.173 DEBUG:boto:Headers: {} 2022-09-12T19:11:55.173 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:55.174 DEBUG:boto:Port: 8000 2022-09-12T19:11:55.175 DEBUG:boto:Params: {} 2022-09-12T19:11:55.175 DEBUG:boto:Token: None 2022-09-12T19:11:55.176 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:55 GMT /jdpiak-25/ 2022-09-12T19:11:55.176 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m/Za4QcEki/cLoN5ZK0/fPvoLrE= 2022-09-12T19:11:55.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m/Za4QcEki/cLoN5ZK0/fPvoLrE=', 'Content-Length': '0'} 2022-09-12T19:11:55.179 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008dc52404dc6d13c-00631f847b-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:55.180 DEBUG:boto:path=/jdpiak-25/obj 2022-09-12T19:11:55.181 DEBUG:boto:auth_path=/jdpiak-25/obj 2022-09-12T19:11:55.181 DEBUG:boto:Method: PUT 2022-09-12T19:11:55.182 DEBUG:boto:Path: /jdpiak-25/obj 2022-09-12T19:11:55.182 DEBUG:boto:Data: 2022-09-12T19:11:55.183 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-12T19:11:55.183 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:55.184 DEBUG:boto:Port: 8000 2022-09-12T19:11:55.185 DEBUG:boto:Params: {} 2022-09-12T19:11:55.185 DEBUG:boto:Token: None 2022-09-12T19:11:55.186 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 12 Sep 2022 19:11:55 GMT /jdpiak-25/obj 2022-09-12T19:11:55.187 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MopXhutnEV8fWSZAjYyc31Ylczw= 2022-09-12T19:11:55.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:11:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MopXhutnEV8fWSZAjYyc31Ylczw='} 2022-09-12T19:11:55.198 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000034a9f0a3af4434b9-00631f847b-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 19:11:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:55.199 DEBUG:boto:path=/jdpiak-25/ 2022-09-12T19:11:55.199 DEBUG:boto:auth_path=/jdpiak-25/ 2022-09-12T19:11:55.200 DEBUG:boto:Method: HEAD 2022-09-12T19:11:55.200 DEBUG:boto:Path: /jdpiak-25/ 2022-09-12T19:11:55.202 DEBUG:boto:Data: 2022-09-12T19:11:55.202 DEBUG:boto:Headers: {} 2022-09-12T19:11:55.206 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:55.207 DEBUG:boto:Port: 8000 2022-09-12T19:11:55.208 DEBUG:boto:Params: {} 2022-09-12T19:11:55.208 DEBUG:boto:Token: None 2022-09-12T19:11:55.209 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:55 GMT /jdpiak-25/ 2022-09-12T19:11:55.209 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m/Za4QcEki/cLoN5ZK0/fPvoLrE= 2022-09-12T19:11:55.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m/Za4QcEki/cLoN5ZK0/fPvoLrE=', 'Content-Length': '0'} 2022-09-12T19:11:55.212 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b2150dd0403266b-00631f847b-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:55.213 DEBUG:boto:path=/jdpiak-25/obj 2022-09-12T19:11:55.214 DEBUG:boto:auth_path=/jdpiak-25/obj 2022-09-12T19:11:55.214 DEBUG:boto:Method: DELETE 2022-09-12T19:11:55.215 DEBUG:boto:Path: /jdpiak-25/obj 2022-09-12T19:11:55.215 DEBUG:boto:Data: 2022-09-12T19:11:55.216 DEBUG:boto:Headers: {} 2022-09-12T19:11:55.216 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:11:55.217 DEBUG:boto:Port: 8000 2022-09-12T19:11:55.218 DEBUG:boto:Params: {} 2022-09-12T19:11:55.218 DEBUG:boto:Token: None 2022-09-12T19:11:55.219 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 19:11:55 GMT /jdpiak-25/obj 2022-09-12T19:11:55.220 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iPpcwtqf3KbPwBfDKusc6r1a4Fc= 2022-09-12T19:11:55.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iPpcwtqf3KbPwBfDKusc6r1a4Fc=', 'Content-Length': '0'} 2022-09-12T19:11:55.238 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000027eb86410fc8be08-00631f847b-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 19:11:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:55.240 DEBUG:boto:path=/jdpiak-26/ 2022-09-12T19:11:55.241 DEBUG:boto:auth_path=/jdpiak-26/ 2022-09-12T19:11:55.241 DEBUG:boto:Method: HEAD 2022-09-12T19:11:55.242 DEBUG:boto:Path: /jdpiak-26/ 2022-09-12T19:11:55.242 DEBUG:boto:Data: 2022-09-12T19:11:55.243 DEBUG:boto:Headers: {} 2022-09-12T19:11:55.243 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:55.244 DEBUG:boto:Port: 8001 2022-09-12T19:11:55.244 DEBUG:boto:Params: {} 2022-09-12T19:11:55.245 DEBUG:boto:Token: None 2022-09-12T19:11:55.246 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:55 GMT /jdpiak-26/ 2022-09-12T19:11:55.247 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x/Pa9o85ayqUSHstrYd/rOY922A= 2022-09-12T19:11:55.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x/Pa9o85ayqUSHstrYd/rOY922A=', 'Content-Length': '0'} 2022-09-12T19:11:55.250 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f12cfc89bc44ca0-00631f847b-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:55.251 DEBUG:boto:path=/jdpiak-26/obj 2022-09-12T19:11:55.252 DEBUG:boto:auth_path=/jdpiak-26/obj 2022-09-12T19:11:55.252 DEBUG:boto:Method: PUT 2022-09-12T19:11:55.253 DEBUG:boto:Path: /jdpiak-26/obj 2022-09-12T19:11:55.254 DEBUG:boto:Data: 2022-09-12T19:11:55.254 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-12T19:11:55.255 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:55.255 DEBUG:boto:Port: 8001 2022-09-12T19:11:55.256 DEBUG:boto:Params: {} 2022-09-12T19:11:55.256 DEBUG:boto:Token: None 2022-09-12T19:11:55.257 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 12 Sep 2022 19:11:55 GMT /jdpiak-26/obj 2022-09-12T19:11:55.257 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TWwM5Li5SXwwcrr/bfme5iYt864= 2022-09-12T19:11:55.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:11:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TWwM5Li5SXwwcrr/bfme5iYt864='} 2022-09-12T19:11:55.264 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d550ff2e1c01b897-00631f847b-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 19:11:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:55.265 DEBUG:boto:path=/jdpiak-26/ 2022-09-12T19:11:55.265 DEBUG:boto:auth_path=/jdpiak-26/ 2022-09-12T19:11:55.266 DEBUG:boto:Method: HEAD 2022-09-12T19:11:55.266 DEBUG:boto:Path: /jdpiak-26/ 2022-09-12T19:11:55.267 DEBUG:boto:Data: 2022-09-12T19:11:55.268 DEBUG:boto:Headers: {} 2022-09-12T19:11:55.268 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:55.269 DEBUG:boto:Port: 8001 2022-09-12T19:11:55.269 DEBUG:boto:Params: {} 2022-09-12T19:11:55.270 DEBUG:boto:Token: None 2022-09-12T19:11:55.271 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:11:55 GMT /jdpiak-26/ 2022-09-12T19:11:55.272 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x/Pa9o85ayqUSHstrYd/rOY922A= 2022-09-12T19:11:55.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x/Pa9o85ayqUSHstrYd/rOY922A=', 'Content-Length': '0'} 2022-09-12T19:11:55.276 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004afe5c6956d20684-00631f847b-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:11:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:55.277 DEBUG:boto:path=/jdpiak-26/obj 2022-09-12T19:11:55.277 DEBUG:boto:auth_path=/jdpiak-26/obj 2022-09-12T19:11:55.278 DEBUG:boto:Method: DELETE 2022-09-12T19:11:55.278 DEBUG:boto:Path: /jdpiak-26/obj 2022-09-12T19:11:55.279 DEBUG:boto:Data: 2022-09-12T19:11:55.279 DEBUG:boto:Headers: {} 2022-09-12T19:11:55.280 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:11:55.280 DEBUG:boto:Port: 8001 2022-09-12T19:11:55.281 DEBUG:boto:Params: {} 2022-09-12T19:11:55.281 DEBUG:boto:Token: None 2022-09-12T19:11:55.282 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 19:11:55 GMT /jdpiak-26/obj 2022-09-12T19:11:55.283 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5/4l+sfQJ9+S36F3JOaYVs2/txI= 2022-09-12T19:11:55.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:11:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5/4l+sfQJ9+S36F3JOaYVs2/txI=', 'Content-Length': '0'} 2022-09-12T19:11:55.294 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000009251e447337eb326-00631f847b-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 19:11:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:11:55.294 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-24 2022-09-12T19:11:55.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:55.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:11:55.296 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:11:55.352 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:11:55.353 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:11:55.398 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:11:55.395+0000 7febcdc6c600 1 waiting to reach incremental sync.. 2022-09-12T19:12:00.399 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:12:00.397+0000 7febcdc6c600 1 waiting to reach incremental sync.. 2022-09-12T19:12:05.399 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:12:05.398+0000 7febcdc6c600 1 waiting to reach incremental sync.. 2022-09-12T19:12:10.401 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:12:10.397+0000 7febcdc6c600 1 waiting to reach incremental sync.. 2022-09-12T19:12:15.401 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:12:15.398+0000 7febcdc6c600 1 waiting to reach incremental sync.. 2022-09-12T19:12:20.402 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:12:20.399+0000 7febcdc6c600 1 waiting to reach incremental sync.. 2022-09-12T19:12:25.402 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:12:25.399+0000 7febcdc6c600 1 waiting to reach incremental sync.. 2022-09-12T19:12:30.403 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:12:30.400+0000 7febcdc6c600 1 waiting to reach incremental sync.. 2022-09-12T19:12:35.404 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:12:35.401+0000 7febcdc6c600 1 waiting to reach incremental sync.. 2022-09-12T19:12:40.404 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:12:40.401+0000 7febcdc6c600 1 waiting to reach incremental sync.. 2022-09-12T19:12:45.405 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:12:45.402+0000 7febcdc6c600 1 waiting to reach incremental sync.. 2022-09-12T19:12:50.405 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:12:50.403+0000 7febcdc6c600 1 waiting to reach incremental sync.. 2022-09-12T19:12:55.406 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:12:55.403+0000 7febcdc6c600 1 waiting to reach incremental sync.. 2022-09-12T19:13:00.407 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:00.404+0000 7febcdc6c600 1 waiting to reach incremental sync.. 2022-09-12T19:13:05.409 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:05.406+0000 7febcdc6c600 1 bucket sync caught up with source: 2022-09-12T19:13:05.409 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000003.13.6, , ] 2022-09-12T19:13:05.409 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000003.13.6, , ] 2022-09-12T19:13:05.410 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:05.406+0000 7febcdc6c600 0 bucket checkpoint complete 2022-09-12T19:13:05.428 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-24 2022-09-12T19:13:05.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:05.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:05.432 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:13:05.500 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:13:05.500 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:13:05.549 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:05.546+0000 7effbaff3600 1 bucket sync caught up with source: 2022-09-12T19:13:05.550 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000003.13.6, , ] 2022-09-12T19:13:05.550 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000003.13.6, , ] 2022-09-12T19:13:05.551 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:05.546+0000 7effbaff3600 0 bucket checkpoint complete 2022-09-12T19:13:05.563 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-24 2022-09-12T19:13:05.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:05.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:05.566 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:13:05.631 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:13:05.631 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:13:05.681 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:05.678+0000 7fc00f390600 1 bucket sync caught up with source: 2022-09-12T19:13:05.681 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000003.13.6, , ] 2022-09-12T19:13:05.682 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000003.13.6, , ] 2022-09-12T19:13:05.682 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:05.678+0000 7fc00f390600 0 bucket checkpoint complete 2022-09-12T19:13:05.700 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-24 2022-09-12T19:13:05.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:05.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:05.702 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:13:05.762 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:13:05.763 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:13:05.801 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:05.798+0000 7f1078dab600 1 bucket sync caught up with source: 2022-09-12T19:13:05.802 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000001.49.12, , ] 2022-09-12T19:13:05.802 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000001.49.12, , ] 2022-09-12T19:13:05.802 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:05.798+0000 7f1078dab600 0 bucket checkpoint complete 2022-09-12T19:13:05.818 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-24 2022-09-12T19:13:05.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:05.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:05.820 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:13:05.878 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:13:05.878 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:13:05.915 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:05.912+0000 7f4089635600 1 bucket sync caught up with source: 2022-09-12T19:13:05.915 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000001.49.12, , ] 2022-09-12T19:13:05.916 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000001.49.12, , ] 2022-09-12T19:13:05.916 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:05.912+0000 7f4089635600 0 bucket checkpoint complete 2022-09-12T19:13:05.929 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-24 2022-09-12T19:13:05.930 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-24 2022-09-12T19:13:05.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:05.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:05.933 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:13:05.991 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:13:05.992 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:13:06.028 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:06.025+0000 7fed5ef9a600 1 bucket sync caught up with source: 2022-09-12T19:13:06.029 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000001.55.12, , ] 2022-09-12T19:13:06.029 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000001.55.12, , ] 2022-09-12T19:13:06.029 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:06.025+0000 7fed5ef9a600 0 bucket checkpoint complete 2022-09-12T19:13:06.041 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-24 2022-09-12T19:13:06.041 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:06.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:06.043 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:13:06.101 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:13:06.102 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:13:06.150 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:06.147+0000 7f0e70345600 1 bucket sync caught up with source: 2022-09-12T19:13:06.151 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000001.55.12, , ] 2022-09-12T19:13:06.151 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000001.55.12, , ] 2022-09-12T19:13:06.151 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:06.147+0000 7f0e70345600 0 bucket checkpoint complete 2022-09-12T19:13:06.168 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-24 2022-09-12T19:13:06.168 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-24 zones={test-zone2, test-zone1} 2022-09-12T19:13:06.169 DEBUG:boto:path=/jdpiak-24/ 2022-09-12T19:13:06.170 DEBUG:boto:auth_path=/jdpiak-24/ 2022-09-12T19:13:06.171 DEBUG:boto:Method: HEAD 2022-09-12T19:13:06.172 DEBUG:boto:Path: /jdpiak-24/ 2022-09-12T19:13:06.172 DEBUG:boto:Data: 2022-09-12T19:13:06.173 DEBUG:boto:Headers: {} 2022-09-12T19:13:06.173 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:06.174 DEBUG:boto:Port: 8000 2022-09-12T19:13:06.175 DEBUG:boto:Params: {} 2022-09-12T19:13:06.176 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:13:06.177 DEBUG:boto:Token: None 2022-09-12T19:13:06.178 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:06 GMT /jdpiak-24/ 2022-09-12T19:13:06.180 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:973Wvn0AAuzUtkMpv9K3yWXS/00= 2022-09-12T19:13:06.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:973Wvn0AAuzUtkMpv9K3yWXS/00=', 'Content-Length': '0'} 2022-09-12T19:13:06.185 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac3da15fd367839d-00631f84c2-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:06.187 DEBUG:boto:path=/jdpiak-24/ 2022-09-12T19:13:06.188 DEBUG:boto:auth_path=/jdpiak-24/ 2022-09-12T19:13:06.188 DEBUG:boto:Method: HEAD 2022-09-12T19:13:06.189 DEBUG:boto:Path: /jdpiak-24/ 2022-09-12T19:13:06.190 DEBUG:boto:Data: 2022-09-12T19:13:06.191 DEBUG:boto:Headers: {} 2022-09-12T19:13:06.191 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:06.192 DEBUG:boto:Port: 8000 2022-09-12T19:13:06.192 DEBUG:boto:Params: {} 2022-09-12T19:13:06.193 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:13:06.193 DEBUG:boto:Token: None 2022-09-12T19:13:06.194 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:06 GMT /jdpiak-24/ 2022-09-12T19:13:06.195 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:973Wvn0AAuzUtkMpv9K3yWXS/00= 2022-09-12T19:13:06.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:973Wvn0AAuzUtkMpv9K3yWXS/00=', 'Content-Length': '0'} 2022-09-12T19:13:06.198 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a010490a1a17ec43-00631f84c2-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:06.199 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:13:06.199 DEBUG:boto:path=/jdpiak-24/ 2022-09-12T19:13:06.200 DEBUG:boto:auth_path=/jdpiak-24/ 2022-09-12T19:13:06.201 DEBUG:boto:path=/jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.201 DEBUG:boto:auth_path=/jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.202 DEBUG:boto:Method: GET 2022-09-12T19:13:06.202 DEBUG:boto:Path: /jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.203 DEBUG:boto:Data: 2022-09-12T19:13:06.204 DEBUG:boto:Headers: {} 2022-09-12T19:13:06.204 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:06.205 DEBUG:boto:Port: 8000 2022-09-12T19:13:06.205 DEBUG:boto:Params: {} 2022-09-12T19:13:06.206 DEBUG:boto:Token: None 2022-09-12T19:13:06.206 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:06 GMT /jdpiak-24/?versions 2022-09-12T19:13:06.207 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A01VPJ0MfaoxOHf1QEAMzWP435E= 2022-09-12T19:13:06.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A01VPJ0MfaoxOHf1QEAMzWP435E=', 'Content-Length': '0'} 2022-09-12T19:13:06.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004949e4fdff8b7d31-00631f84c2-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:06.211 DEBUG:boto:b'jdpiak-24999falseobjnulltrue2022-09-12T19:11:55.161Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:06.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:13:06.212 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:13:06.213 DEBUG:boto:path=/jdpiak-24/ 2022-09-12T19:13:06.213 DEBUG:boto:auth_path=/jdpiak-24/ 2022-09-12T19:13:06.214 DEBUG:boto:path=/jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.214 DEBUG:boto:auth_path=/jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.215 DEBUG:boto:Method: GET 2022-09-12T19:13:06.215 DEBUG:boto:Path: /jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.216 DEBUG:boto:Data: 2022-09-12T19:13:06.216 DEBUG:boto:Headers: {} 2022-09-12T19:13:06.217 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:06.217 DEBUG:boto:Port: 8000 2022-09-12T19:13:06.218 DEBUG:boto:Params: {} 2022-09-12T19:13:06.218 DEBUG:boto:Token: None 2022-09-12T19:13:06.219 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:06 GMT /jdpiak-24/?versions 2022-09-12T19:13:06.219 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A01VPJ0MfaoxOHf1QEAMzWP435E= 2022-09-12T19:13:06.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A01VPJ0MfaoxOHf1QEAMzWP435E=', 'Content-Length': '0'} 2022-09-12T19:13:06.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038caec8f41b68d88-00631f84c2-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:06.223 DEBUG:boto:b'jdpiak-24999falseobjnulltrue2022-09-12T19:11:55.161Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:06.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:13:06.224 DEBUG:boto:path=/jdpiak-24/ 2022-09-12T19:13:06.225 DEBUG:boto:auth_path=/jdpiak-24/ 2022-09-12T19:13:06.225 DEBUG:boto:path=/jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.226 DEBUG:boto:auth_path=/jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.226 DEBUG:boto:Method: GET 2022-09-12T19:13:06.227 DEBUG:boto:Path: /jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.227 DEBUG:boto:Data: 2022-09-12T19:13:06.228 DEBUG:boto:Headers: {} 2022-09-12T19:13:06.228 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:06.229 DEBUG:boto:Port: 8000 2022-09-12T19:13:06.229 DEBUG:boto:Params: {} 2022-09-12T19:13:06.230 DEBUG:boto:Token: None 2022-09-12T19:13:06.230 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:06 GMT /jdpiak-24/?versions 2022-09-12T19:13:06.231 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A01VPJ0MfaoxOHf1QEAMzWP435E= 2022-09-12T19:13:06.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A01VPJ0MfaoxOHf1QEAMzWP435E=', 'Content-Length': '0'} 2022-09-12T19:13:06.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a85cd0ce29bac15-00631f84c2-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:06.235 DEBUG:boto:b'jdpiak-24999falseobjnulltrue2022-09-12T19:11:55.161Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:06.235 DEBUG:boto:path=/jdpiak-24/ 2022-09-12T19:13:06.236 DEBUG:boto:auth_path=/jdpiak-24/ 2022-09-12T19:13:06.236 DEBUG:boto:path=/jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.237 DEBUG:boto:auth_path=/jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.237 DEBUG:boto:Method: GET 2022-09-12T19:13:06.238 DEBUG:boto:Path: /jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.238 DEBUG:boto:Data: 2022-09-12T19:13:06.239 DEBUG:boto:Headers: {} 2022-09-12T19:13:06.239 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:06.240 DEBUG:boto:Port: 8000 2022-09-12T19:13:06.240 DEBUG:boto:Params: {} 2022-09-12T19:13:06.241 DEBUG:boto:Token: None 2022-09-12T19:13:06.241 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:06 GMT /jdpiak-24/?versions 2022-09-12T19:13:06.242 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A01VPJ0MfaoxOHf1QEAMzWP435E= 2022-09-12T19:13:06.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A01VPJ0MfaoxOHf1QEAMzWP435E=', 'Content-Length': '0'} 2022-09-12T19:13:06.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fb42711ba119150-00631f84c2-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:06.245 DEBUG:boto:b'jdpiak-24999falseobjnulltrue2022-09-12T19:11:55.161Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:06.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:13:06.247 DEBUG:boto:path=/jdpiak-24/obj 2022-09-12T19:13:06.247 DEBUG:boto:auth_path=/jdpiak-24/obj 2022-09-12T19:13:06.248 DEBUG:boto:Method: HEAD 2022-09-12T19:13:06.248 DEBUG:boto:Path: /jdpiak-24/obj 2022-09-12T19:13:06.249 DEBUG:boto:Data: 2022-09-12T19:13:06.249 DEBUG:boto:Headers: {} 2022-09-12T19:13:06.250 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:06.250 DEBUG:boto:Port: 8000 2022-09-12T19:13:06.251 DEBUG:boto:Params: {} 2022-09-12T19:13:06.251 DEBUG:boto:Token: None 2022-09-12T19:13:06.252 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:06 GMT /jdpiak-24/obj 2022-09-12T19:13:06.252 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uClQX2sOQWaBg90NJNwBk2xVYA4= 2022-09-12T19:13:06.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uClQX2sOQWaBg90NJNwBk2xVYA4=', 'Content-Length': '0'} 2022-09-12T19:13:06.255 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000005af7107eb8dd259d-00631f84c2-10e4-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:06.255 DEBUG:boto:path=/jdpiak-24/obj 2022-09-12T19:13:06.256 DEBUG:boto:auth_path=/jdpiak-24/obj 2022-09-12T19:13:06.257 DEBUG:boto:Method: HEAD 2022-09-12T19:13:06.257 DEBUG:boto:Path: /jdpiak-24/obj 2022-09-12T19:13:06.257 DEBUG:boto:Data: 2022-09-12T19:13:06.258 DEBUG:boto:Headers: {} 2022-09-12T19:13:06.258 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:06.259 DEBUG:boto:Port: 8000 2022-09-12T19:13:06.259 DEBUG:boto:Params: {} 2022-09-12T19:13:06.260 DEBUG:boto:Token: None 2022-09-12T19:13:06.261 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:06 GMT /jdpiak-24/obj 2022-09-12T19:13:06.261 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uClQX2sOQWaBg90NJNwBk2xVYA4= 2022-09-12T19:13:06.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uClQX2sOQWaBg90NJNwBk2xVYA4=', 'Content-Length': '0'} 2022-09-12T19:13:06.263 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000005649a89b22936ce8-00631f84c2-10f6-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:06.264 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-24 zones={test-zone2, test-zone1} 2022-09-12T19:13:06.265 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-24 zones={test-zone3, test-zone1} 2022-09-12T19:13:06.265 DEBUG:boto:path=/jdpiak-24/ 2022-09-12T19:13:06.265 DEBUG:boto:auth_path=/jdpiak-24/ 2022-09-12T19:13:06.266 DEBUG:boto:Method: HEAD 2022-09-12T19:13:06.266 DEBUG:boto:Path: /jdpiak-24/ 2022-09-12T19:13:06.267 DEBUG:boto:Data: 2022-09-12T19:13:06.267 DEBUG:boto:Headers: {} 2022-09-12T19:13:06.268 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:06.268 DEBUG:boto:Port: 8001 2022-09-12T19:13:06.269 DEBUG:boto:Params: {} 2022-09-12T19:13:06.270 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T19:13:06.270 DEBUG:boto:Token: None 2022-09-12T19:13:06.271 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:06 GMT /jdpiak-24/ 2022-09-12T19:13:06.272 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:973Wvn0AAuzUtkMpv9K3yWXS/00= 2022-09-12T19:13:06.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:973Wvn0AAuzUtkMpv9K3yWXS/00=', 'Content-Length': '0'} 2022-09-12T19:13:06.276 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cac9e120ec925709-00631f84c2-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:06.276 DEBUG:boto:path=/jdpiak-24/ 2022-09-12T19:13:06.277 DEBUG:boto:auth_path=/jdpiak-24/ 2022-09-12T19:13:06.278 DEBUG:boto:Method: HEAD 2022-09-12T19:13:06.278 DEBUG:boto:Path: /jdpiak-24/ 2022-09-12T19:13:06.279 DEBUG:boto:Data: 2022-09-12T19:13:06.279 DEBUG:boto:Headers: {} 2022-09-12T19:13:06.280 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:06.280 DEBUG:boto:Port: 8000 2022-09-12T19:13:06.281 DEBUG:boto:Params: {} 2022-09-12T19:13:06.281 DEBUG:boto:Token: None 2022-09-12T19:13:06.282 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:06 GMT /jdpiak-24/ 2022-09-12T19:13:06.282 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:973Wvn0AAuzUtkMpv9K3yWXS/00= 2022-09-12T19:13:06.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:973Wvn0AAuzUtkMpv9K3yWXS/00=', 'Content-Length': '0'} 2022-09-12T19:13:06.285 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1e8adbf3e2e9fa8-00631f84c2-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:06.286 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:13:06.287 DEBUG:boto:path=/jdpiak-24/ 2022-09-12T19:13:06.287 DEBUG:boto:auth_path=/jdpiak-24/ 2022-09-12T19:13:06.288 DEBUG:boto:path=/jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.288 DEBUG:boto:auth_path=/jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.289 DEBUG:boto:Method: GET 2022-09-12T19:13:06.289 DEBUG:boto:Path: /jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.290 DEBUG:boto:Data: 2022-09-12T19:13:06.290 DEBUG:boto:Headers: {} 2022-09-12T19:13:06.291 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:06.291 DEBUG:boto:Port: 8001 2022-09-12T19:13:06.292 DEBUG:boto:Params: {} 2022-09-12T19:13:06.292 DEBUG:boto:Token: None 2022-09-12T19:13:06.293 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:06 GMT /jdpiak-24/?versions 2022-09-12T19:13:06.293 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A01VPJ0MfaoxOHf1QEAMzWP435E= 2022-09-12T19:13:06.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A01VPJ0MfaoxOHf1QEAMzWP435E=', 'Content-Length': '0'} 2022-09-12T19:13:06.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004408dc30b33dc3e6-00631f84c2-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:06.297 DEBUG:boto:b'jdpiak-24999falseobjnulltrue2022-09-12T19:11:55.161Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:06.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:13:06.298 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:13:06.299 DEBUG:boto:path=/jdpiak-24/ 2022-09-12T19:13:06.299 DEBUG:boto:auth_path=/jdpiak-24/ 2022-09-12T19:13:06.300 DEBUG:boto:path=/jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.300 DEBUG:boto:auth_path=/jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.301 DEBUG:boto:Method: GET 2022-09-12T19:13:06.301 DEBUG:boto:Path: /jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.302 DEBUG:boto:Data: 2022-09-12T19:13:06.302 DEBUG:boto:Headers: {} 2022-09-12T19:13:06.303 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:06.303 DEBUG:boto:Port: 8000 2022-09-12T19:13:06.304 DEBUG:boto:Params: {} 2022-09-12T19:13:06.304 DEBUG:boto:Token: None 2022-09-12T19:13:06.305 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:06 GMT /jdpiak-24/?versions 2022-09-12T19:13:06.306 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A01VPJ0MfaoxOHf1QEAMzWP435E= 2022-09-12T19:13:06.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A01VPJ0MfaoxOHf1QEAMzWP435E=', 'Content-Length': '0'} 2022-09-12T19:13:06.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0dc1d6562ae8481-00631f84c2-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:06.314 DEBUG:boto:b'jdpiak-24999falseobjnulltrue2022-09-12T19:11:55.161Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:06.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:13:06.316 DEBUG:boto:path=/jdpiak-24/ 2022-09-12T19:13:06.317 DEBUG:boto:auth_path=/jdpiak-24/ 2022-09-12T19:13:06.317 DEBUG:boto:path=/jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.318 DEBUG:boto:auth_path=/jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.318 DEBUG:boto:Method: GET 2022-09-12T19:13:06.319 DEBUG:boto:Path: /jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.319 DEBUG:boto:Data: 2022-09-12T19:13:06.320 DEBUG:boto:Headers: {} 2022-09-12T19:13:06.320 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:06.321 DEBUG:boto:Port: 8001 2022-09-12T19:13:06.322 DEBUG:boto:Params: {} 2022-09-12T19:13:06.322 DEBUG:boto:Token: None 2022-09-12T19:13:06.323 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:06 GMT /jdpiak-24/?versions 2022-09-12T19:13:06.324 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A01VPJ0MfaoxOHf1QEAMzWP435E= 2022-09-12T19:13:06.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A01VPJ0MfaoxOHf1QEAMzWP435E=', 'Content-Length': '0'} 2022-09-12T19:13:06.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002eefd96a4d995a9-00631f84c2-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:06.329 DEBUG:boto:b'jdpiak-24999falseobjnulltrue2022-09-12T19:11:55.161Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:06.330 DEBUG:boto:path=/jdpiak-24/ 2022-09-12T19:13:06.330 DEBUG:boto:auth_path=/jdpiak-24/ 2022-09-12T19:13:06.331 DEBUG:boto:path=/jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.331 DEBUG:boto:auth_path=/jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.332 DEBUG:boto:Method: GET 2022-09-12T19:13:06.332 DEBUG:boto:Path: /jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.333 DEBUG:boto:Data: 2022-09-12T19:13:06.333 DEBUG:boto:Headers: {} 2022-09-12T19:13:06.334 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:06.335 DEBUG:boto:Port: 8000 2022-09-12T19:13:06.336 DEBUG:boto:Params: {} 2022-09-12T19:13:06.336 DEBUG:boto:Token: None 2022-09-12T19:13:06.337 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:06 GMT /jdpiak-24/?versions 2022-09-12T19:13:06.338 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A01VPJ0MfaoxOHf1QEAMzWP435E= 2022-09-12T19:13:06.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A01VPJ0MfaoxOHf1QEAMzWP435E=', 'Content-Length': '0'} 2022-09-12T19:13:06.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000162dad44c8debd58-00631f84c2-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:06.382 DEBUG:boto:b'jdpiak-24999falseobjnulltrue2022-09-12T19:11:55.161Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:06.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:13:06.384 DEBUG:boto:path=/jdpiak-24/obj 2022-09-12T19:13:06.384 DEBUG:boto:auth_path=/jdpiak-24/obj 2022-09-12T19:13:06.385 DEBUG:boto:Method: HEAD 2022-09-12T19:13:06.385 DEBUG:boto:Path: /jdpiak-24/obj 2022-09-12T19:13:06.386 DEBUG:boto:Data: 2022-09-12T19:13:06.386 DEBUG:boto:Headers: {} 2022-09-12T19:13:06.387 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:06.387 DEBUG:boto:Port: 8001 2022-09-12T19:13:06.388 DEBUG:boto:Params: {} 2022-09-12T19:13:06.388 DEBUG:boto:Token: None 2022-09-12T19:13:06.389 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:06 GMT /jdpiak-24/obj 2022-09-12T19:13:06.389 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uClQX2sOQWaBg90NJNwBk2xVYA4= 2022-09-12T19:13:06.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uClQX2sOQWaBg90NJNwBk2xVYA4=', 'Content-Length': '0'} 2022-09-12T19:13:06.392 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000cc794a532c1b5737-00631f84c2-10f2-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:06.393 DEBUG:boto:path=/jdpiak-24/obj 2022-09-12T19:13:06.393 DEBUG:boto:auth_path=/jdpiak-24/obj 2022-09-12T19:13:06.394 DEBUG:boto:Method: HEAD 2022-09-12T19:13:06.394 DEBUG:boto:Path: /jdpiak-24/obj 2022-09-12T19:13:06.395 DEBUG:boto:Data: 2022-09-12T19:13:06.395 DEBUG:boto:Headers: {} 2022-09-12T19:13:06.396 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:06.396 DEBUG:boto:Port: 8000 2022-09-12T19:13:06.397 DEBUG:boto:Params: {} 2022-09-12T19:13:06.397 DEBUG:boto:Token: None 2022-09-12T19:13:06.398 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:06 GMT /jdpiak-24/obj 2022-09-12T19:13:06.398 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uClQX2sOQWaBg90NJNwBk2xVYA4= 2022-09-12T19:13:06.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uClQX2sOQWaBg90NJNwBk2xVYA4=', 'Content-Length': '0'} 2022-09-12T19:13:06.401 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000de6fb41cf5785c08-00631f84c2-10f6-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:06.401 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-24 zones={test-zone3, test-zone1} 2022-09-12T19:13:06.402 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-24 zones={test-zone3, test-zone2} 2022-09-12T19:13:06.402 DEBUG:boto:path=/jdpiak-24/ 2022-09-12T19:13:06.403 DEBUG:boto:auth_path=/jdpiak-24/ 2022-09-12T19:13:06.403 DEBUG:boto:Method: HEAD 2022-09-12T19:13:06.404 DEBUG:boto:Path: /jdpiak-24/ 2022-09-12T19:13:06.404 DEBUG:boto:Data: 2022-09-12T19:13:06.405 DEBUG:boto:Headers: {} 2022-09-12T19:13:06.405 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:06.406 DEBUG:boto:Port: 8001 2022-09-12T19:13:06.406 DEBUG:boto:Params: {} 2022-09-12T19:13:06.407 DEBUG:boto:Token: None 2022-09-12T19:13:06.407 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:06 GMT /jdpiak-24/ 2022-09-12T19:13:06.408 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:973Wvn0AAuzUtkMpv9K3yWXS/00= 2022-09-12T19:13:06.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:973Wvn0AAuzUtkMpv9K3yWXS/00=', 'Content-Length': '0'} 2022-09-12T19:13:06.411 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1a69c3adc1ec197-00631f84c2-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:06.412 DEBUG:boto:path=/jdpiak-24/ 2022-09-12T19:13:06.412 DEBUG:boto:auth_path=/jdpiak-24/ 2022-09-12T19:13:06.413 DEBUG:boto:Method: HEAD 2022-09-12T19:13:06.413 DEBUG:boto:Path: /jdpiak-24/ 2022-09-12T19:13:06.414 DEBUG:boto:Data: 2022-09-12T19:13:06.414 DEBUG:boto:Headers: {} 2022-09-12T19:13:06.415 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:06.415 DEBUG:boto:Port: 8000 2022-09-12T19:13:06.416 DEBUG:boto:Params: {} 2022-09-12T19:13:06.416 DEBUG:boto:Token: None 2022-09-12T19:13:06.417 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:06 GMT /jdpiak-24/ 2022-09-12T19:13:06.417 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:973Wvn0AAuzUtkMpv9K3yWXS/00= 2022-09-12T19:13:06.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:973Wvn0AAuzUtkMpv9K3yWXS/00=', 'Content-Length': '0'} 2022-09-12T19:13:06.420 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004848d18da8494c24-00631f84c2-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:06.421 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:13:06.421 DEBUG:boto:path=/jdpiak-24/ 2022-09-12T19:13:06.422 DEBUG:boto:auth_path=/jdpiak-24/ 2022-09-12T19:13:06.422 DEBUG:boto:path=/jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.423 DEBUG:boto:auth_path=/jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.423 DEBUG:boto:Method: GET 2022-09-12T19:13:06.424 DEBUG:boto:Path: /jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.424 DEBUG:boto:Data: 2022-09-12T19:13:06.425 DEBUG:boto:Headers: {} 2022-09-12T19:13:06.425 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:06.426 DEBUG:boto:Port: 8001 2022-09-12T19:13:06.427 DEBUG:boto:Params: {} 2022-09-12T19:13:06.427 DEBUG:boto:Token: None 2022-09-12T19:13:06.428 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:06 GMT /jdpiak-24/?versions 2022-09-12T19:13:06.429 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A01VPJ0MfaoxOHf1QEAMzWP435E= 2022-09-12T19:13:06.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A01VPJ0MfaoxOHf1QEAMzWP435E=', 'Content-Length': '0'} 2022-09-12T19:13:06.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002712c4b1e6172d5-00631f84c2-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:06.433 DEBUG:boto:b'jdpiak-24999falseobjnulltrue2022-09-12T19:11:55.161Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:06.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:13:06.435 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:13:06.436 DEBUG:boto:path=/jdpiak-24/ 2022-09-12T19:13:06.436 DEBUG:boto:auth_path=/jdpiak-24/ 2022-09-12T19:13:06.437 DEBUG:boto:path=/jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.437 DEBUG:boto:auth_path=/jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.438 DEBUG:boto:Method: GET 2022-09-12T19:13:06.438 DEBUG:boto:Path: /jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.439 DEBUG:boto:Data: 2022-09-12T19:13:06.439 DEBUG:boto:Headers: {} 2022-09-12T19:13:06.440 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:06.440 DEBUG:boto:Port: 8000 2022-09-12T19:13:06.441 DEBUG:boto:Params: {} 2022-09-12T19:13:06.441 DEBUG:boto:Token: None 2022-09-12T19:13:06.442 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:06 GMT /jdpiak-24/?versions 2022-09-12T19:13:06.443 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A01VPJ0MfaoxOHf1QEAMzWP435E= 2022-09-12T19:13:06.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A01VPJ0MfaoxOHf1QEAMzWP435E=', 'Content-Length': '0'} 2022-09-12T19:13:06.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c21b2cbc03d2602-00631f84c2-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:06.446 DEBUG:boto:b'jdpiak-24999falseobjnulltrue2022-09-12T19:11:55.161Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:06.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:13:06.447 DEBUG:boto:path=/jdpiak-24/ 2022-09-12T19:13:06.448 DEBUG:boto:auth_path=/jdpiak-24/ 2022-09-12T19:13:06.448 DEBUG:boto:path=/jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.449 DEBUG:boto:auth_path=/jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.449 DEBUG:boto:Method: GET 2022-09-12T19:13:06.450 DEBUG:boto:Path: /jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.450 DEBUG:boto:Data: 2022-09-12T19:13:06.451 DEBUG:boto:Headers: {} 2022-09-12T19:13:06.451 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:06.452 DEBUG:boto:Port: 8001 2022-09-12T19:13:06.452 DEBUG:boto:Params: {} 2022-09-12T19:13:06.453 DEBUG:boto:Token: None 2022-09-12T19:13:06.454 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:06 GMT /jdpiak-24/?versions 2022-09-12T19:13:06.455 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A01VPJ0MfaoxOHf1QEAMzWP435E= 2022-09-12T19:13:06.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A01VPJ0MfaoxOHf1QEAMzWP435E=', 'Content-Length': '0'} 2022-09-12T19:13:06.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bb597e8af3ae4e5-00631f84c2-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:06.459 DEBUG:boto:b'jdpiak-24999falseobjnulltrue2022-09-12T19:11:55.161Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:06.460 DEBUG:boto:path=/jdpiak-24/ 2022-09-12T19:13:06.461 DEBUG:boto:auth_path=/jdpiak-24/ 2022-09-12T19:13:06.461 DEBUG:boto:path=/jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.462 DEBUG:boto:auth_path=/jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.463 DEBUG:boto:Method: GET 2022-09-12T19:13:06.463 DEBUG:boto:Path: /jdpiak-24/?versions&max-keys=999 2022-09-12T19:13:06.464 DEBUG:boto:Data: 2022-09-12T19:13:06.464 DEBUG:boto:Headers: {} 2022-09-12T19:13:06.465 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:06.465 DEBUG:boto:Port: 8000 2022-09-12T19:13:06.466 DEBUG:boto:Params: {} 2022-09-12T19:13:06.467 DEBUG:boto:Token: None 2022-09-12T19:13:06.468 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:06 GMT /jdpiak-24/?versions 2022-09-12T19:13:06.468 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A01VPJ0MfaoxOHf1QEAMzWP435E= 2022-09-12T19:13:06.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A01VPJ0MfaoxOHf1QEAMzWP435E=', 'Content-Length': '0'} 2022-09-12T19:13:06.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024c09a92aec64ed7-00631f84c2-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:06.472 DEBUG:boto:b'jdpiak-24999falseobjnulltrue2022-09-12T19:11:55.161Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:06.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:13:06.473 DEBUG:boto:path=/jdpiak-24/obj 2022-09-12T19:13:06.474 DEBUG:boto:auth_path=/jdpiak-24/obj 2022-09-12T19:13:06.474 DEBUG:boto:Method: HEAD 2022-09-12T19:13:06.475 DEBUG:boto:Path: /jdpiak-24/obj 2022-09-12T19:13:06.475 DEBUG:boto:Data: 2022-09-12T19:13:06.476 DEBUG:boto:Headers: {} 2022-09-12T19:13:06.476 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:06.477 DEBUG:boto:Port: 8001 2022-09-12T19:13:06.477 DEBUG:boto:Params: {} 2022-09-12T19:13:06.478 DEBUG:boto:Token: None 2022-09-12T19:13:06.478 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:06 GMT /jdpiak-24/obj 2022-09-12T19:13:06.479 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uClQX2sOQWaBg90NJNwBk2xVYA4= 2022-09-12T19:13:06.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uClQX2sOQWaBg90NJNwBk2xVYA4=', 'Content-Length': '0'} 2022-09-12T19:13:06.481 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000eb0fa30af2f7856d-00631f84c2-10f2-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:06.482 DEBUG:boto:path=/jdpiak-24/obj 2022-09-12T19:13:06.483 DEBUG:boto:auth_path=/jdpiak-24/obj 2022-09-12T19:13:06.483 DEBUG:boto:Method: HEAD 2022-09-12T19:13:06.484 DEBUG:boto:Path: /jdpiak-24/obj 2022-09-12T19:13:06.484 DEBUG:boto:Data: 2022-09-12T19:13:06.485 DEBUG:boto:Headers: {} 2022-09-12T19:13:06.485 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:06.486 DEBUG:boto:Port: 8000 2022-09-12T19:13:06.486 DEBUG:boto:Params: {} 2022-09-12T19:13:06.487 DEBUG:boto:Token: None 2022-09-12T19:13:06.488 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:06 GMT /jdpiak-24/obj 2022-09-12T19:13:06.488 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uClQX2sOQWaBg90NJNwBk2xVYA4= 2022-09-12T19:13:06.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uClQX2sOQWaBg90NJNwBk2xVYA4=', 'Content-Length': '0'} 2022-09-12T19:13:06.491 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000005ac079c939f6b46e-00631f84c2-10e4-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:06.492 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-24 zones={test-zone3, test-zone2} 2022-09-12T19:13:06.492 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-25 2022-09-12T19:13:06.493 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:06.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:06.494 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:13:06.550 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:13:06.551 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:13:06.584 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:06.582+0000 7f074e0a7600 1 bucket sync caught up with source: 2022-09-12T19:13:06.585 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000001.21.12, , ] 2022-09-12T19:13:06.585 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000001.21.12, , ] 2022-09-12T19:13:06.585 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:06.582+0000 7f074e0a7600 0 bucket checkpoint complete 2022-09-12T19:13:06.598 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-25 2022-09-12T19:13:06.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:06.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:06.599 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:13:06.658 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:13:06.659 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:13:06.707 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:06.704+0000 7f988d8aa600 1 bucket sync caught up with source: 2022-09-12T19:13:06.708 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000001.21.12, , ] 2022-09-12T19:13:06.708 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000001.21.12, , ] 2022-09-12T19:13:06.708 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:06.704+0000 7f988d8aa600 0 bucket checkpoint complete 2022-09-12T19:13:06.723 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-25 2022-09-12T19:13:06.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:06.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:06.727 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:13:06.783 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:13:06.783 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:13:06.830 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:06.827+0000 7fe6be8b1600 1 bucket sync caught up with source: 2022-09-12T19:13:06.831 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000001.21.12, , ] 2022-09-12T19:13:06.831 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000001.21.12, , ] 2022-09-12T19:13:06.831 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:06.827+0000 7fe6be8b1600 0 bucket checkpoint complete 2022-09-12T19:13:06.850 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-25 2022-09-12T19:13:06.850 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:06.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:06.851 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:13:06.910 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:13:06.911 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:13:06.957 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:06.954+0000 7fb4ddfa5600 1 bucket sync caught up with source: 2022-09-12T19:13:06.958 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000003.126.6, , ] 2022-09-12T19:13:06.958 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000003.126.6, , ] 2022-09-12T19:13:06.958 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:06.954+0000 7fb4ddfa5600 0 bucket checkpoint complete 2022-09-12T19:13:06.970 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-25 2022-09-12T19:13:06.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:06.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:06.972 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:13:07.029 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:13:07.029 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:13:07.065 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:07.062+0000 7fda2fda7600 1 bucket sync caught up with source: 2022-09-12T19:13:07.066 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000003.126.6, , ] 2022-09-12T19:13:07.066 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000003.126.6, , ] 2022-09-12T19:13:07.067 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:07.062+0000 7fda2fda7600 0 bucket checkpoint complete 2022-09-12T19:13:07.083 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-25 2022-09-12T19:13:07.083 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-25 2022-09-12T19:13:07.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:07.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:07.085 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:13:07.142 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:13:07.143 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:13:07.194 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:07.191+0000 7ff654f4e600 1 bucket sync caught up with source: 2022-09-12T19:13:07.195 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000001.90.12, , ] 2022-09-12T19:13:07.195 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000001.90.12, , ] 2022-09-12T19:13:07.196 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:07.191+0000 7ff654f4e600 0 bucket checkpoint complete 2022-09-12T19:13:07.209 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-25 2022-09-12T19:13:07.210 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:07.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:07.211 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:13:07.270 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:13:07.270 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:13:07.318 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:07.315+0000 7f885d558600 1 waiting to reach incremental sync.. 2022-09-12T19:13:12.319 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:12.316+0000 7f885d558600 1 waiting to reach incremental sync.. 2022-09-12T19:13:17.320 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:17.317+0000 7f885d558600 1 waiting to reach incremental sync.. 2022-09-12T19:13:22.320 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:22.318+0000 7f885d558600 1 waiting to reach incremental sync.. 2022-09-12T19:13:27.321 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:27.318+0000 7f885d558600 1 waiting to reach incremental sync.. 2022-09-12T19:13:32.322 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:32.320+0000 7f885d558600 1 waiting to reach incremental sync.. 2022-09-12T19:13:37.322 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:37.320+0000 7f885d558600 1 waiting to reach incremental sync.. 2022-09-12T19:13:42.324 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:42.321+0000 7f885d558600 1 bucket sync caught up with source: 2022-09-12T19:13:42.325 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000001.90.12, , ] 2022-09-12T19:13:42.325 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000001.90.12, , ] 2022-09-12T19:13:42.325 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:42.321+0000 7f885d558600 0 bucket checkpoint complete 2022-09-12T19:13:42.341 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-25 2022-09-12T19:13:42.342 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-25 zones={test-zone2, test-zone1} 2022-09-12T19:13:42.343 DEBUG:boto:path=/jdpiak-25/ 2022-09-12T19:13:42.343 DEBUG:boto:auth_path=/jdpiak-25/ 2022-09-12T19:13:42.344 DEBUG:boto:Method: HEAD 2022-09-12T19:13:42.345 DEBUG:boto:Path: /jdpiak-25/ 2022-09-12T19:13:42.346 DEBUG:boto:Data: 2022-09-12T19:13:42.346 DEBUG:boto:Headers: {} 2022-09-12T19:13:42.347 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:42.347 DEBUG:boto:Port: 8000 2022-09-12T19:13:42.348 DEBUG:boto:Params: {} 2022-09-12T19:13:42.349 DEBUG:boto:Token: None 2022-09-12T19:13:42.349 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:42 GMT /jdpiak-25/ 2022-09-12T19:13:42.350 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fi1qSq8zinsxAEJCFed5BjG5sYk= 2022-09-12T19:13:42.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fi1qSq8zinsxAEJCFed5BjG5sYk=', 'Content-Length': '0'} 2022-09-12T19:13:42.354 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d211c2abcfe4016-00631f84e6-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:42.355 DEBUG:boto:path=/jdpiak-25/ 2022-09-12T19:13:42.356 DEBUG:boto:auth_path=/jdpiak-25/ 2022-09-12T19:13:42.357 DEBUG:boto:Method: HEAD 2022-09-12T19:13:42.358 DEBUG:boto:Path: /jdpiak-25/ 2022-09-12T19:13:42.359 DEBUG:boto:Data: 2022-09-12T19:13:42.360 DEBUG:boto:Headers: {} 2022-09-12T19:13:42.360 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:42.361 DEBUG:boto:Port: 8000 2022-09-12T19:13:42.361 DEBUG:boto:Params: {} 2022-09-12T19:13:42.362 DEBUG:boto:Token: None 2022-09-12T19:13:42.362 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:42 GMT /jdpiak-25/ 2022-09-12T19:13:42.363 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fi1qSq8zinsxAEJCFed5BjG5sYk= 2022-09-12T19:13:42.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fi1qSq8zinsxAEJCFed5BjG5sYk=', 'Content-Length': '0'} 2022-09-12T19:13:42.367 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f384a97fc7194aec-00631f84e6-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:42.367 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:13:42.368 DEBUG:boto:path=/jdpiak-25/ 2022-09-12T19:13:42.368 DEBUG:boto:auth_path=/jdpiak-25/ 2022-09-12T19:13:42.369 DEBUG:boto:path=/jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.369 DEBUG:boto:auth_path=/jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.370 DEBUG:boto:Method: GET 2022-09-12T19:13:42.370 DEBUG:boto:Path: /jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.371 DEBUG:boto:Data: 2022-09-12T19:13:42.371 DEBUG:boto:Headers: {} 2022-09-12T19:13:42.372 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:42.372 DEBUG:boto:Port: 8000 2022-09-12T19:13:42.373 DEBUG:boto:Params: {} 2022-09-12T19:13:42.374 DEBUG:boto:Token: None 2022-09-12T19:13:42.374 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:42 GMT /jdpiak-25/?versions 2022-09-12T19:13:42.375 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6KmGPDqrOnzHsX1SNNsaXsGRrvw= 2022-09-12T19:13:42.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6KmGPDqrOnzHsX1SNNsaXsGRrvw=', 'Content-Length': '0'} 2022-09-12T19:13:42.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076fc83b21c63e9be-00631f84e6-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:42.378 DEBUG:boto:b'jdpiak-25999falseobjnulltrue2022-09-12T19:11:55.222Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:42.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:13:42.380 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:13:42.381 DEBUG:boto:path=/jdpiak-25/ 2022-09-12T19:13:42.381 DEBUG:boto:auth_path=/jdpiak-25/ 2022-09-12T19:13:42.382 DEBUG:boto:path=/jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.382 DEBUG:boto:auth_path=/jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.383 DEBUG:boto:Method: GET 2022-09-12T19:13:42.383 DEBUG:boto:Path: /jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.384 DEBUG:boto:Data: 2022-09-12T19:13:42.384 DEBUG:boto:Headers: {} 2022-09-12T19:13:42.385 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:42.385 DEBUG:boto:Port: 8000 2022-09-12T19:13:42.386 DEBUG:boto:Params: {} 2022-09-12T19:13:42.386 DEBUG:boto:Token: None 2022-09-12T19:13:42.387 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:42 GMT /jdpiak-25/?versions 2022-09-12T19:13:42.388 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6KmGPDqrOnzHsX1SNNsaXsGRrvw= 2022-09-12T19:13:42.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6KmGPDqrOnzHsX1SNNsaXsGRrvw=', 'Content-Length': '0'} 2022-09-12T19:13:42.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004a53dc3665ee9c7-00631f84e6-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:42.391 DEBUG:boto:b'jdpiak-25999falseobjnulltrue2022-09-12T19:11:55.222Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:42.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:13:42.393 DEBUG:boto:path=/jdpiak-25/ 2022-09-12T19:13:42.393 DEBUG:boto:auth_path=/jdpiak-25/ 2022-09-12T19:13:42.394 DEBUG:boto:path=/jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.394 DEBUG:boto:auth_path=/jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.395 DEBUG:boto:Method: GET 2022-09-12T19:13:42.395 DEBUG:boto:Path: /jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.396 DEBUG:boto:Data: 2022-09-12T19:13:42.397 DEBUG:boto:Headers: {} 2022-09-12T19:13:42.397 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:42.398 DEBUG:boto:Port: 8000 2022-09-12T19:13:42.398 DEBUG:boto:Params: {} 2022-09-12T19:13:42.399 DEBUG:boto:Token: None 2022-09-12T19:13:42.399 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:42 GMT /jdpiak-25/?versions 2022-09-12T19:13:42.400 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6KmGPDqrOnzHsX1SNNsaXsGRrvw= 2022-09-12T19:13:42.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6KmGPDqrOnzHsX1SNNsaXsGRrvw=', 'Content-Length': '0'} 2022-09-12T19:13:42.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031b7e5a7f9ae10db-00631f84e6-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:42.404 DEBUG:boto:b'jdpiak-25999falseobjnulltrue2022-09-12T19:11:55.222Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:42.405 DEBUG:boto:path=/jdpiak-25/ 2022-09-12T19:13:42.405 DEBUG:boto:auth_path=/jdpiak-25/ 2022-09-12T19:13:42.406 DEBUG:boto:path=/jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.406 DEBUG:boto:auth_path=/jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.407 DEBUG:boto:Method: GET 2022-09-12T19:13:42.407 DEBUG:boto:Path: /jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.408 DEBUG:boto:Data: 2022-09-12T19:13:42.408 DEBUG:boto:Headers: {} 2022-09-12T19:13:42.409 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:42.409 DEBUG:boto:Port: 8000 2022-09-12T19:13:42.410 DEBUG:boto:Params: {} 2022-09-12T19:13:42.410 DEBUG:boto:Token: None 2022-09-12T19:13:42.411 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:42 GMT /jdpiak-25/?versions 2022-09-12T19:13:42.411 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6KmGPDqrOnzHsX1SNNsaXsGRrvw= 2022-09-12T19:13:42.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6KmGPDqrOnzHsX1SNNsaXsGRrvw=', 'Content-Length': '0'} 2022-09-12T19:13:42.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b5da6ceb5703daf-00631f84e6-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:42.458 DEBUG:boto:b'jdpiak-25999falseobjnulltrue2022-09-12T19:11:55.222Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:42.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:13:42.460 DEBUG:boto:path=/jdpiak-25/obj 2022-09-12T19:13:42.460 DEBUG:boto:auth_path=/jdpiak-25/obj 2022-09-12T19:13:42.461 DEBUG:boto:Method: HEAD 2022-09-12T19:13:42.462 DEBUG:boto:Path: /jdpiak-25/obj 2022-09-12T19:13:42.463 DEBUG:boto:Data: 2022-09-12T19:13:42.463 DEBUG:boto:Headers: {} 2022-09-12T19:13:42.464 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:42.464 DEBUG:boto:Port: 8000 2022-09-12T19:13:42.465 DEBUG:boto:Params: {} 2022-09-12T19:13:42.465 DEBUG:boto:Token: None 2022-09-12T19:13:42.466 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:42 GMT /jdpiak-25/obj 2022-09-12T19:13:42.467 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XNPeJ0zB1u8qhDyOXNSbvoXtEv4= 2022-09-12T19:13:42.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XNPeJ0zB1u8qhDyOXNSbvoXtEv4=', 'Content-Length': '0'} 2022-09-12T19:13:42.469 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000005040d8d061b2b6af-00631f84e6-10e4-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:42.470 DEBUG:boto:path=/jdpiak-25/obj 2022-09-12T19:13:42.471 DEBUG:boto:auth_path=/jdpiak-25/obj 2022-09-12T19:13:42.471 DEBUG:boto:Method: HEAD 2022-09-12T19:13:42.472 DEBUG:boto:Path: /jdpiak-25/obj 2022-09-12T19:13:42.472 DEBUG:boto:Data: 2022-09-12T19:13:42.473 DEBUG:boto:Headers: {} 2022-09-12T19:13:42.473 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:42.474 DEBUG:boto:Port: 8000 2022-09-12T19:13:42.475 DEBUG:boto:Params: {} 2022-09-12T19:13:42.475 DEBUG:boto:Token: None 2022-09-12T19:13:42.476 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:42 GMT /jdpiak-25/obj 2022-09-12T19:13:42.476 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XNPeJ0zB1u8qhDyOXNSbvoXtEv4= 2022-09-12T19:13:42.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XNPeJ0zB1u8qhDyOXNSbvoXtEv4=', 'Content-Length': '0'} 2022-09-12T19:13:42.479 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000000a3b193cb2ff8d21-00631f84e6-10f6-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:42.479 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-25 zones={test-zone2, test-zone1} 2022-09-12T19:13:42.480 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-25 zones={test-zone3, test-zone1} 2022-09-12T19:13:42.480 DEBUG:boto:path=/jdpiak-25/ 2022-09-12T19:13:42.481 DEBUG:boto:auth_path=/jdpiak-25/ 2022-09-12T19:13:42.481 DEBUG:boto:Method: HEAD 2022-09-12T19:13:42.482 DEBUG:boto:Path: /jdpiak-25/ 2022-09-12T19:13:42.483 DEBUG:boto:Data: 2022-09-12T19:13:42.483 DEBUG:boto:Headers: {} 2022-09-12T19:13:42.484 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:42.484 DEBUG:boto:Port: 8001 2022-09-12T19:13:42.485 DEBUG:boto:Params: {} 2022-09-12T19:13:42.485 DEBUG:boto:Token: None 2022-09-12T19:13:42.486 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:42 GMT /jdpiak-25/ 2022-09-12T19:13:42.487 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fi1qSq8zinsxAEJCFed5BjG5sYk= 2022-09-12T19:13:42.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fi1qSq8zinsxAEJCFed5BjG5sYk=', 'Content-Length': '0'} 2022-09-12T19:13:42.490 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eec0dc88b49d5ffd-00631f84e6-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:42.490 DEBUG:boto:path=/jdpiak-25/ 2022-09-12T19:13:42.491 DEBUG:boto:auth_path=/jdpiak-25/ 2022-09-12T19:13:42.491 DEBUG:boto:Method: HEAD 2022-09-12T19:13:42.492 DEBUG:boto:Path: /jdpiak-25/ 2022-09-12T19:13:42.492 DEBUG:boto:Data: 2022-09-12T19:13:42.493 DEBUG:boto:Headers: {} 2022-09-12T19:13:42.494 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:42.495 DEBUG:boto:Port: 8000 2022-09-12T19:13:42.495 DEBUG:boto:Params: {} 2022-09-12T19:13:42.496 DEBUG:boto:Token: None 2022-09-12T19:13:42.496 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:42 GMT /jdpiak-25/ 2022-09-12T19:13:42.497 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fi1qSq8zinsxAEJCFed5BjG5sYk= 2022-09-12T19:13:42.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fi1qSq8zinsxAEJCFed5BjG5sYk=', 'Content-Length': '0'} 2022-09-12T19:13:42.500 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa22466e57f6ba26-00631f84e6-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:42.501 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:13:42.502 DEBUG:boto:path=/jdpiak-25/ 2022-09-12T19:13:42.502 DEBUG:boto:auth_path=/jdpiak-25/ 2022-09-12T19:13:42.503 DEBUG:boto:path=/jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.503 DEBUG:boto:auth_path=/jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.504 DEBUG:boto:Method: GET 2022-09-12T19:13:42.504 DEBUG:boto:Path: /jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.505 DEBUG:boto:Data: 2022-09-12T19:13:42.505 DEBUG:boto:Headers: {} 2022-09-12T19:13:42.506 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:42.506 DEBUG:boto:Port: 8001 2022-09-12T19:13:42.507 DEBUG:boto:Params: {} 2022-09-12T19:13:42.508 DEBUG:boto:Token: None 2022-09-12T19:13:42.508 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:42 GMT /jdpiak-25/?versions 2022-09-12T19:13:42.509 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6KmGPDqrOnzHsX1SNNsaXsGRrvw= 2022-09-12T19:13:42.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6KmGPDqrOnzHsX1SNNsaXsGRrvw=', 'Content-Length': '0'} 2022-09-12T19:13:42.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061e841ac26bfac63-00631f84e6-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:42.512 DEBUG:boto:b'jdpiak-25999falseobjnulltrue2022-09-12T19:11:55.222Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:42.513 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:13:42.514 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:13:42.514 DEBUG:boto:path=/jdpiak-25/ 2022-09-12T19:13:42.515 DEBUG:boto:auth_path=/jdpiak-25/ 2022-09-12T19:13:42.515 DEBUG:boto:path=/jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.516 DEBUG:boto:auth_path=/jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.516 DEBUG:boto:Method: GET 2022-09-12T19:13:42.517 DEBUG:boto:Path: /jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.517 DEBUG:boto:Data: 2022-09-12T19:13:42.518 DEBUG:boto:Headers: {} 2022-09-12T19:13:42.519 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:42.519 DEBUG:boto:Port: 8000 2022-09-12T19:13:42.520 DEBUG:boto:Params: {} 2022-09-12T19:13:42.520 DEBUG:boto:Token: None 2022-09-12T19:13:42.521 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:42 GMT /jdpiak-25/?versions 2022-09-12T19:13:42.521 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6KmGPDqrOnzHsX1SNNsaXsGRrvw= 2022-09-12T19:13:42.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6KmGPDqrOnzHsX1SNNsaXsGRrvw=', 'Content-Length': '0'} 2022-09-12T19:13:42.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021181d10e7d87d82-00631f84e6-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:42.527 DEBUG:boto:b'jdpiak-25999falseobjnulltrue2022-09-12T19:11:55.222Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:42.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:13:42.529 DEBUG:boto:path=/jdpiak-25/ 2022-09-12T19:13:42.530 DEBUG:boto:auth_path=/jdpiak-25/ 2022-09-12T19:13:42.531 DEBUG:boto:path=/jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.531 DEBUG:boto:auth_path=/jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.532 DEBUG:boto:Method: GET 2022-09-12T19:13:42.533 DEBUG:boto:Path: /jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.534 DEBUG:boto:Data: 2022-09-12T19:13:42.535 DEBUG:boto:Headers: {} 2022-09-12T19:13:42.536 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:42.537 DEBUG:boto:Port: 8001 2022-09-12T19:13:42.538 DEBUG:boto:Params: {} 2022-09-12T19:13:42.539 DEBUG:boto:Token: None 2022-09-12T19:13:42.540 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:42 GMT /jdpiak-25/?versions 2022-09-12T19:13:42.541 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6KmGPDqrOnzHsX1SNNsaXsGRrvw= 2022-09-12T19:13:42.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6KmGPDqrOnzHsX1SNNsaXsGRrvw=', 'Content-Length': '0'} 2022-09-12T19:13:42.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041e104fd309a324d-00631f84e6-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:42.546 DEBUG:boto:b'jdpiak-25999falseobjnulltrue2022-09-12T19:11:55.222Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:42.547 DEBUG:boto:path=/jdpiak-25/ 2022-09-12T19:13:42.548 DEBUG:boto:auth_path=/jdpiak-25/ 2022-09-12T19:13:42.548 DEBUG:boto:path=/jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.549 DEBUG:boto:auth_path=/jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.549 DEBUG:boto:Method: GET 2022-09-12T19:13:42.550 DEBUG:boto:Path: /jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.550 DEBUG:boto:Data: 2022-09-12T19:13:42.551 DEBUG:boto:Headers: {} 2022-09-12T19:13:42.552 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:42.553 DEBUG:boto:Port: 8000 2022-09-12T19:13:42.554 DEBUG:boto:Params: {} 2022-09-12T19:13:42.554 DEBUG:boto:Token: None 2022-09-12T19:13:42.555 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:42 GMT /jdpiak-25/?versions 2022-09-12T19:13:42.555 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6KmGPDqrOnzHsX1SNNsaXsGRrvw= 2022-09-12T19:13:42.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6KmGPDqrOnzHsX1SNNsaXsGRrvw=', 'Content-Length': '0'} 2022-09-12T19:13:42.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cde087c36863acad-00631f84e6-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:42.559 DEBUG:boto:b'jdpiak-25999falseobjnulltrue2022-09-12T19:11:55.222Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:42.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:13:42.561 DEBUG:boto:path=/jdpiak-25/obj 2022-09-12T19:13:42.562 DEBUG:boto:auth_path=/jdpiak-25/obj 2022-09-12T19:13:42.562 DEBUG:boto:Method: HEAD 2022-09-12T19:13:42.563 DEBUG:boto:Path: /jdpiak-25/obj 2022-09-12T19:13:42.564 DEBUG:boto:Data: 2022-09-12T19:13:42.565 DEBUG:boto:Headers: {} 2022-09-12T19:13:42.566 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:42.567 DEBUG:boto:Port: 8001 2022-09-12T19:13:42.568 DEBUG:boto:Params: {} 2022-09-12T19:13:42.569 DEBUG:boto:Token: None 2022-09-12T19:13:42.570 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:42 GMT /jdpiak-25/obj 2022-09-12T19:13:42.571 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XNPeJ0zB1u8qhDyOXNSbvoXtEv4= 2022-09-12T19:13:42.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XNPeJ0zB1u8qhDyOXNSbvoXtEv4=', 'Content-Length': '0'} 2022-09-12T19:13:42.574 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000008ed639ea69921b57-00631f84e6-10f2-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:42.575 DEBUG:boto:path=/jdpiak-25/obj 2022-09-12T19:13:42.576 DEBUG:boto:auth_path=/jdpiak-25/obj 2022-09-12T19:13:42.576 DEBUG:boto:Method: HEAD 2022-09-12T19:13:42.577 DEBUG:boto:Path: /jdpiak-25/obj 2022-09-12T19:13:42.577 DEBUG:boto:Data: 2022-09-12T19:13:42.578 DEBUG:boto:Headers: {} 2022-09-12T19:13:42.579 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:42.579 DEBUG:boto:Port: 8000 2022-09-12T19:13:42.580 DEBUG:boto:Params: {} 2022-09-12T19:13:42.580 DEBUG:boto:Token: None 2022-09-12T19:13:42.581 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:42 GMT /jdpiak-25/obj 2022-09-12T19:13:42.581 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XNPeJ0zB1u8qhDyOXNSbvoXtEv4= 2022-09-12T19:13:42.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XNPeJ0zB1u8qhDyOXNSbvoXtEv4=', 'Content-Length': '0'} 2022-09-12T19:13:42.584 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000085d426c2bf2b1089-00631f84e6-10f6-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:42.585 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-25 zones={test-zone3, test-zone1} 2022-09-12T19:13:42.586 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-25 zones={test-zone3, test-zone2} 2022-09-12T19:13:42.587 DEBUG:boto:path=/jdpiak-25/ 2022-09-12T19:13:42.587 DEBUG:boto:auth_path=/jdpiak-25/ 2022-09-12T19:13:42.588 DEBUG:boto:Method: HEAD 2022-09-12T19:13:42.588 DEBUG:boto:Path: /jdpiak-25/ 2022-09-12T19:13:42.589 DEBUG:boto:Data: 2022-09-12T19:13:42.589 DEBUG:boto:Headers: {} 2022-09-12T19:13:42.590 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:42.590 DEBUG:boto:Port: 8001 2022-09-12T19:13:42.591 DEBUG:boto:Params: {} 2022-09-12T19:13:42.592 DEBUG:boto:Token: None 2022-09-12T19:13:42.592 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:42 GMT /jdpiak-25/ 2022-09-12T19:13:42.593 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fi1qSq8zinsxAEJCFed5BjG5sYk= 2022-09-12T19:13:42.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fi1qSq8zinsxAEJCFed5BjG5sYk=', 'Content-Length': '0'} 2022-09-12T19:13:42.596 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066b374fc84beb813-00631f84e6-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:42.597 DEBUG:boto:path=/jdpiak-25/ 2022-09-12T19:13:42.597 DEBUG:boto:auth_path=/jdpiak-25/ 2022-09-12T19:13:42.598 DEBUG:boto:Method: HEAD 2022-09-12T19:13:42.598 DEBUG:boto:Path: /jdpiak-25/ 2022-09-12T19:13:42.599 DEBUG:boto:Data: 2022-09-12T19:13:42.599 DEBUG:boto:Headers: {} 2022-09-12T19:13:42.600 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:42.600 DEBUG:boto:Port: 8000 2022-09-12T19:13:42.601 DEBUG:boto:Params: {} 2022-09-12T19:13:42.601 DEBUG:boto:Token: None 2022-09-12T19:13:42.602 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:42 GMT /jdpiak-25/ 2022-09-12T19:13:42.603 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fi1qSq8zinsxAEJCFed5BjG5sYk= 2022-09-12T19:13:42.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fi1qSq8zinsxAEJCFed5BjG5sYk=', 'Content-Length': '0'} 2022-09-12T19:13:42.606 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000661262dfe9eae5b1-00631f84e6-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:42.606 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:13:42.607 DEBUG:boto:path=/jdpiak-25/ 2022-09-12T19:13:42.608 DEBUG:boto:auth_path=/jdpiak-25/ 2022-09-12T19:13:42.608 DEBUG:boto:path=/jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.609 DEBUG:boto:auth_path=/jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.609 DEBUG:boto:Method: GET 2022-09-12T19:13:42.610 DEBUG:boto:Path: /jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.610 DEBUG:boto:Data: 2022-09-12T19:13:42.611 DEBUG:boto:Headers: {} 2022-09-12T19:13:42.611 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:42.612 DEBUG:boto:Port: 8001 2022-09-12T19:13:42.612 DEBUG:boto:Params: {} 2022-09-12T19:13:42.613 DEBUG:boto:Token: None 2022-09-12T19:13:42.614 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:42 GMT /jdpiak-25/?versions 2022-09-12T19:13:42.614 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6KmGPDqrOnzHsX1SNNsaXsGRrvw= 2022-09-12T19:13:42.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6KmGPDqrOnzHsX1SNNsaXsGRrvw=', 'Content-Length': '0'} 2022-09-12T19:13:42.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046f6c4c9ca094e21-00631f84e6-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:42.662 DEBUG:boto:b'jdpiak-25999falseobjnulltrue2022-09-12T19:11:55.222Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:42.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:13:42.664 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:13:42.664 DEBUG:boto:path=/jdpiak-25/ 2022-09-12T19:13:42.665 DEBUG:boto:auth_path=/jdpiak-25/ 2022-09-12T19:13:42.665 DEBUG:boto:path=/jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.666 DEBUG:boto:auth_path=/jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.667 DEBUG:boto:Method: GET 2022-09-12T19:13:42.667 DEBUG:boto:Path: /jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.668 DEBUG:boto:Data: 2022-09-12T19:13:42.668 DEBUG:boto:Headers: {} 2022-09-12T19:13:42.669 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:42.669 DEBUG:boto:Port: 8000 2022-09-12T19:13:42.670 DEBUG:boto:Params: {} 2022-09-12T19:13:42.671 DEBUG:boto:Token: None 2022-09-12T19:13:42.671 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:42 GMT /jdpiak-25/?versions 2022-09-12T19:13:42.672 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6KmGPDqrOnzHsX1SNNsaXsGRrvw= 2022-09-12T19:13:42.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6KmGPDqrOnzHsX1SNNsaXsGRrvw=', 'Content-Length': '0'} 2022-09-12T19:13:42.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d44d0e47cc986311-00631f84e6-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:42.677 DEBUG:boto:b'jdpiak-25999falseobjnulltrue2022-09-12T19:11:55.222Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:42.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:13:42.678 DEBUG:boto:path=/jdpiak-25/ 2022-09-12T19:13:42.679 DEBUG:boto:auth_path=/jdpiak-25/ 2022-09-12T19:13:42.679 DEBUG:boto:path=/jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.680 DEBUG:boto:auth_path=/jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.680 DEBUG:boto:Method: GET 2022-09-12T19:13:42.681 DEBUG:boto:Path: /jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.681 DEBUG:boto:Data: 2022-09-12T19:13:42.682 DEBUG:boto:Headers: {} 2022-09-12T19:13:42.683 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:42.683 DEBUG:boto:Port: 8001 2022-09-12T19:13:42.684 DEBUG:boto:Params: {} 2022-09-12T19:13:42.684 DEBUG:boto:Token: None 2022-09-12T19:13:42.685 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:42 GMT /jdpiak-25/?versions 2022-09-12T19:13:42.686 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6KmGPDqrOnzHsX1SNNsaXsGRrvw= 2022-09-12T19:13:42.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6KmGPDqrOnzHsX1SNNsaXsGRrvw=', 'Content-Length': '0'} 2022-09-12T19:13:42.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8c144e9ce247457-00631f84e6-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:42.691 DEBUG:boto:b'jdpiak-25999falseobjnulltrue2022-09-12T19:11:55.222Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:42.692 DEBUG:boto:path=/jdpiak-25/ 2022-09-12T19:13:42.693 DEBUG:boto:auth_path=/jdpiak-25/ 2022-09-12T19:13:42.694 DEBUG:boto:path=/jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.695 DEBUG:boto:auth_path=/jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.696 DEBUG:boto:Method: GET 2022-09-12T19:13:42.697 DEBUG:boto:Path: /jdpiak-25/?versions&max-keys=999 2022-09-12T19:13:42.697 DEBUG:boto:Data: 2022-09-12T19:13:42.698 DEBUG:boto:Headers: {} 2022-09-12T19:13:42.699 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:42.700 DEBUG:boto:Port: 8000 2022-09-12T19:13:42.701 DEBUG:boto:Params: {} 2022-09-12T19:13:42.702 DEBUG:boto:Token: None 2022-09-12T19:13:42.703 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:42 GMT /jdpiak-25/?versions 2022-09-12T19:13:42.704 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6KmGPDqrOnzHsX1SNNsaXsGRrvw= 2022-09-12T19:13:42.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6KmGPDqrOnzHsX1SNNsaXsGRrvw=', 'Content-Length': '0'} 2022-09-12T19:13:42.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fd6de6ca7ef7b69-00631f84e6-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:42.709 DEBUG:boto:b'jdpiak-25999falseobjnulltrue2022-09-12T19:11:55.222Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:42.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:13:42.711 DEBUG:boto:path=/jdpiak-25/obj 2022-09-12T19:13:42.712 DEBUG:boto:auth_path=/jdpiak-25/obj 2022-09-12T19:13:42.712 DEBUG:boto:Method: HEAD 2022-09-12T19:13:42.713 DEBUG:boto:Path: /jdpiak-25/obj 2022-09-12T19:13:42.713 DEBUG:boto:Data: 2022-09-12T19:13:42.714 DEBUG:boto:Headers: {} 2022-09-12T19:13:42.714 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:42.715 DEBUG:boto:Port: 8001 2022-09-12T19:13:42.715 DEBUG:boto:Params: {} 2022-09-12T19:13:42.716 DEBUG:boto:Token: None 2022-09-12T19:13:42.717 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:42 GMT /jdpiak-25/obj 2022-09-12T19:13:42.717 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XNPeJ0zB1u8qhDyOXNSbvoXtEv4= 2022-09-12T19:13:42.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XNPeJ0zB1u8qhDyOXNSbvoXtEv4=', 'Content-Length': '0'} 2022-09-12T19:13:42.720 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000057367bdd172344ef-00631f84e6-10f2-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:42.721 DEBUG:boto:path=/jdpiak-25/obj 2022-09-12T19:13:42.721 DEBUG:boto:auth_path=/jdpiak-25/obj 2022-09-12T19:13:42.722 DEBUG:boto:Method: HEAD 2022-09-12T19:13:42.722 DEBUG:boto:Path: /jdpiak-25/obj 2022-09-12T19:13:42.723 DEBUG:boto:Data: 2022-09-12T19:13:42.724 DEBUG:boto:Headers: {} 2022-09-12T19:13:42.724 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:42.725 DEBUG:boto:Port: 8000 2022-09-12T19:13:42.725 DEBUG:boto:Params: {} 2022-09-12T19:13:42.726 DEBUG:boto:Token: None 2022-09-12T19:13:42.726 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:42 GMT /jdpiak-25/obj 2022-09-12T19:13:42.727 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XNPeJ0zB1u8qhDyOXNSbvoXtEv4= 2022-09-12T19:13:42.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XNPeJ0zB1u8qhDyOXNSbvoXtEv4=', 'Content-Length': '0'} 2022-09-12T19:13:42.730 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000004691d1e60490ad77-00631f84e6-10e4-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:42.730 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-25 zones={test-zone3, test-zone2} 2022-09-12T19:13:42.731 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-26 2022-09-12T19:13:42.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:42.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:42.733 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:13:42.797 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:13:42.798 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:13:42.836 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:42.833+0000 7f04cf8cb600 1 bucket sync caught up with source: 2022-09-12T19:13:42.836 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000001.5.12, , ] 2022-09-12T19:13:42.837 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000001.5.12, , ] 2022-09-12T19:13:42.837 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:42.833+0000 7f04cf8cb600 0 bucket checkpoint complete 2022-09-12T19:13:42.853 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-26 2022-09-12T19:13:42.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:42.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:42.855 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:13:42.921 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:13:42.921 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:13:42.971 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:42.968+0000 7f65cd7e1600 1 bucket sync caught up with source: 2022-09-12T19:13:42.971 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000001.5.12, , ] 2022-09-12T19:13:42.972 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000001.5.12, , ] 2022-09-12T19:13:42.972 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:42.968+0000 7f65cd7e1600 0 bucket checkpoint complete 2022-09-12T19:13:42.985 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-26 2022-09-12T19:13:42.986 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:42.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:42.988 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:13:43.045 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:13:43.046 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:13:43.103 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:43.100+0000 7f1221c9e600 1 bucket sync caught up with source: 2022-09-12T19:13:43.103 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000001.5.12, , ] 2022-09-12T19:13:43.103 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000001.5.12, , ] 2022-09-12T19:13:43.104 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:43.100+0000 7f1221c9e600 0 bucket checkpoint complete 2022-09-12T19:13:43.119 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-26 2022-09-12T19:13:43.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:43.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:43.121 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:13:43.181 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:13:43.182 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:13:43.216 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:43.213+0000 7fcd24fcd600 1 bucket sync caught up with source: 2022-09-12T19:13:43.217 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000001.11.12, , ] 2022-09-12T19:13:43.217 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000001.11.12, , ] 2022-09-12T19:13:43.217 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:43.213+0000 7fcd24fcd600 0 bucket checkpoint complete 2022-09-12T19:13:43.229 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-26 2022-09-12T19:13:43.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:43.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:43.231 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:13:43.289 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:13:43.289 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:13:43.328 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:43.325+0000 7ff56e999600 1 bucket sync caught up with source: 2022-09-12T19:13:43.329 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000001.11.12, , ] 2022-09-12T19:13:43.329 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000001.11.12, , ] 2022-09-12T19:13:43.330 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:43.325+0000 7ff56e999600 0 bucket checkpoint complete 2022-09-12T19:13:43.345 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-26 2022-09-12T19:13:43.346 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-26 2022-09-12T19:13:43.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:43.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:43.347 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:13:43.404 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:13:43.405 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:13:43.452 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:43.450+0000 7efd54007600 1 bucket sync caught up with source: 2022-09-12T19:13:43.453 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000003.26.6, , ] 2022-09-12T19:13:43.453 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000003.26.6, , ] 2022-09-12T19:13:43.453 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:43.450+0000 7efd54007600 0 bucket checkpoint complete 2022-09-12T19:13:43.468 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-26 2022-09-12T19:13:43.469 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:43.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:13:43.470 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:13:43.526 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:13:43.527 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:13:43.586 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:43.584+0000 7fd8d4a64600 1 bucket sync caught up with source: 2022-09-12T19:13:43.587 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000003.26.6, , ] 2022-09-12T19:13:43.587 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000003.26.6, , ] 2022-09-12T19:13:43.587 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:43.584+0000 7fd8d4a64600 0 bucket checkpoint complete 2022-09-12T19:13:43.610 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-26 2022-09-12T19:13:43.611 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-26 zones={test-zone2, test-zone1} 2022-09-12T19:13:43.611 DEBUG:boto:path=/jdpiak-26/ 2022-09-12T19:13:43.612 DEBUG:boto:auth_path=/jdpiak-26/ 2022-09-12T19:13:43.612 DEBUG:boto:Method: HEAD 2022-09-12T19:13:43.613 DEBUG:boto:Path: /jdpiak-26/ 2022-09-12T19:13:43.613 DEBUG:boto:Data: 2022-09-12T19:13:43.614 DEBUG:boto:Headers: {} 2022-09-12T19:13:43.615 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:43.616 DEBUG:boto:Port: 8000 2022-09-12T19:13:43.616 DEBUG:boto:Params: {} 2022-09-12T19:13:43.617 DEBUG:boto:Token: None 2022-09-12T19:13:43.617 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:43 GMT /jdpiak-26/ 2022-09-12T19:13:43.618 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:86+G1HvzAxAd2hff3Nvtpvs+IEE= 2022-09-12T19:13:43.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:86+G1HvzAxAd2hff3Nvtpvs+IEE=', 'Content-Length': '0'} 2022-09-12T19:13:43.622 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000045026ec2b162d91a-00631f84e7-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:43.623 DEBUG:boto:path=/jdpiak-26/ 2022-09-12T19:13:43.624 DEBUG:boto:auth_path=/jdpiak-26/ 2022-09-12T19:13:43.625 DEBUG:boto:Method: HEAD 2022-09-12T19:13:43.626 DEBUG:boto:Path: /jdpiak-26/ 2022-09-12T19:13:43.627 DEBUG:boto:Data: 2022-09-12T19:13:43.628 DEBUG:boto:Headers: {} 2022-09-12T19:13:43.628 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:43.629 DEBUG:boto:Port: 8000 2022-09-12T19:13:43.629 DEBUG:boto:Params: {} 2022-09-12T19:13:43.630 DEBUG:boto:Token: None 2022-09-12T19:13:43.631 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:43 GMT /jdpiak-26/ 2022-09-12T19:13:43.631 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:86+G1HvzAxAd2hff3Nvtpvs+IEE= 2022-09-12T19:13:43.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:86+G1HvzAxAd2hff3Nvtpvs+IEE=', 'Content-Length': '0'} 2022-09-12T19:13:43.634 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029853dccaf24307f-00631f84e7-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:43.635 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:13:43.635 DEBUG:boto:path=/jdpiak-26/ 2022-09-12T19:13:43.636 DEBUG:boto:auth_path=/jdpiak-26/ 2022-09-12T19:13:43.636 DEBUG:boto:path=/jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.637 DEBUG:boto:auth_path=/jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.638 DEBUG:boto:Method: GET 2022-09-12T19:13:43.638 DEBUG:boto:Path: /jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.639 DEBUG:boto:Data: 2022-09-12T19:13:43.639 DEBUG:boto:Headers: {} 2022-09-12T19:13:43.640 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:43.640 DEBUG:boto:Port: 8000 2022-09-12T19:13:43.641 DEBUG:boto:Params: {} 2022-09-12T19:13:43.641 DEBUG:boto:Token: None 2022-09-12T19:13:43.642 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:43 GMT /jdpiak-26/?versions 2022-09-12T19:13:43.643 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pMD8Cb8AVD8MOgaD1cjF2Tumg8k= 2022-09-12T19:13:43.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pMD8Cb8AVD8MOgaD1cjF2Tumg8k=', 'Content-Length': '0'} 2022-09-12T19:13:43.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cdd97cf9cbd5f53-00631f84e7-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:43.646 DEBUG:boto:b'jdpiak-26999falseobjnulltrue2022-09-12T19:11:55.284Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:43.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:13:43.648 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:13:43.648 DEBUG:boto:path=/jdpiak-26/ 2022-09-12T19:13:43.649 DEBUG:boto:auth_path=/jdpiak-26/ 2022-09-12T19:13:43.649 DEBUG:boto:path=/jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.650 DEBUG:boto:auth_path=/jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.651 DEBUG:boto:Method: GET 2022-09-12T19:13:43.651 DEBUG:boto:Path: /jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.652 DEBUG:boto:Data: 2022-09-12T19:13:43.652 DEBUG:boto:Headers: {} 2022-09-12T19:13:43.653 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:43.653 DEBUG:boto:Port: 8000 2022-09-12T19:13:43.654 DEBUG:boto:Params: {} 2022-09-12T19:13:43.654 DEBUG:boto:Token: None 2022-09-12T19:13:43.655 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:43 GMT /jdpiak-26/?versions 2022-09-12T19:13:43.656 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pMD8Cb8AVD8MOgaD1cjF2Tumg8k= 2022-09-12T19:13:43.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pMD8Cb8AVD8MOgaD1cjF2Tumg8k=', 'Content-Length': '0'} 2022-09-12T19:13:43.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d74682df4f5da069-00631f84e7-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:43.659 DEBUG:boto:b'jdpiak-26999falseobjnulltrue2022-09-12T19:11:55.284Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:43.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:13:43.661 DEBUG:boto:path=/jdpiak-26/ 2022-09-12T19:13:43.661 DEBUG:boto:auth_path=/jdpiak-26/ 2022-09-12T19:13:43.662 DEBUG:boto:path=/jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.662 DEBUG:boto:auth_path=/jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.663 DEBUG:boto:Method: GET 2022-09-12T19:13:43.664 DEBUG:boto:Path: /jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.664 DEBUG:boto:Data: 2022-09-12T19:13:43.665 DEBUG:boto:Headers: {} 2022-09-12T19:13:43.665 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:43.666 DEBUG:boto:Port: 8000 2022-09-12T19:13:43.666 DEBUG:boto:Params: {} 2022-09-12T19:13:43.667 DEBUG:boto:Token: None 2022-09-12T19:13:43.668 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:43 GMT /jdpiak-26/?versions 2022-09-12T19:13:43.668 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pMD8Cb8AVD8MOgaD1cjF2Tumg8k= 2022-09-12T19:13:43.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pMD8Cb8AVD8MOgaD1cjF2Tumg8k=', 'Content-Length': '0'} 2022-09-12T19:13:43.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1f2289e1acbff82-00631f84e7-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:43.672 DEBUG:boto:b'jdpiak-26999falseobjnulltrue2022-09-12T19:11:55.284Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:43.673 DEBUG:boto:path=/jdpiak-26/ 2022-09-12T19:13:43.673 DEBUG:boto:auth_path=/jdpiak-26/ 2022-09-12T19:13:43.674 DEBUG:boto:path=/jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.680 DEBUG:boto:auth_path=/jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.681 DEBUG:boto:Method: GET 2022-09-12T19:13:43.682 DEBUG:boto:Path: /jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.683 DEBUG:boto:Data: 2022-09-12T19:13:43.684 DEBUG:boto:Headers: {} 2022-09-12T19:13:43.685 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:43.685 DEBUG:boto:Port: 8000 2022-09-12T19:13:43.686 DEBUG:boto:Params: {} 2022-09-12T19:13:43.687 DEBUG:boto:Token: None 2022-09-12T19:13:43.687 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:43 GMT /jdpiak-26/?versions 2022-09-12T19:13:43.688 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pMD8Cb8AVD8MOgaD1cjF2Tumg8k= 2022-09-12T19:13:43.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pMD8Cb8AVD8MOgaD1cjF2Tumg8k=', 'Content-Length': '0'} 2022-09-12T19:13:43.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8c38b60f8dcbe57-00631f84e7-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:43.693 DEBUG:boto:b'jdpiak-26999falseobjnulltrue2022-09-12T19:11:55.284Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:43.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:13:43.694 DEBUG:boto:path=/jdpiak-26/obj 2022-09-12T19:13:43.694 DEBUG:boto:auth_path=/jdpiak-26/obj 2022-09-12T19:13:43.695 DEBUG:boto:Method: HEAD 2022-09-12T19:13:43.696 DEBUG:boto:Path: /jdpiak-26/obj 2022-09-12T19:13:43.696 DEBUG:boto:Data: 2022-09-12T19:13:43.697 DEBUG:boto:Headers: {} 2022-09-12T19:13:43.697 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:43.698 DEBUG:boto:Port: 8000 2022-09-12T19:13:43.698 DEBUG:boto:Params: {} 2022-09-12T19:13:43.699 DEBUG:boto:Token: None 2022-09-12T19:13:43.699 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:43 GMT /jdpiak-26/obj 2022-09-12T19:13:43.700 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WrcYzCz3CpEfowcJqJMFUD7PHMA= 2022-09-12T19:13:43.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WrcYzCz3CpEfowcJqJMFUD7PHMA=', 'Content-Length': '0'} 2022-09-12T19:13:43.702 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000f27e901b5c578552-00631f84e7-10e4-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:43.703 DEBUG:boto:path=/jdpiak-26/obj 2022-09-12T19:13:43.704 DEBUG:boto:auth_path=/jdpiak-26/obj 2022-09-12T19:13:43.705 DEBUG:boto:Method: HEAD 2022-09-12T19:13:43.706 DEBUG:boto:Path: /jdpiak-26/obj 2022-09-12T19:13:43.707 DEBUG:boto:Data: 2022-09-12T19:13:43.708 DEBUG:boto:Headers: {} 2022-09-12T19:13:43.709 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:43.710 DEBUG:boto:Port: 8000 2022-09-12T19:13:43.711 DEBUG:boto:Params: {} 2022-09-12T19:13:43.712 DEBUG:boto:Token: None 2022-09-12T19:13:43.713 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:43 GMT /jdpiak-26/obj 2022-09-12T19:13:43.714 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WrcYzCz3CpEfowcJqJMFUD7PHMA= 2022-09-12T19:13:43.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WrcYzCz3CpEfowcJqJMFUD7PHMA=', 'Content-Length': '0'} 2022-09-12T19:13:43.717 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000818e00f11d89e918-00631f84e7-10f6-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:43.718 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-26 zones={test-zone2, test-zone1} 2022-09-12T19:13:43.719 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-26 zones={test-zone3, test-zone1} 2022-09-12T19:13:43.719 DEBUG:boto:path=/jdpiak-26/ 2022-09-12T19:13:43.720 DEBUG:boto:auth_path=/jdpiak-26/ 2022-09-12T19:13:43.721 DEBUG:boto:Method: HEAD 2022-09-12T19:13:43.721 DEBUG:boto:Path: /jdpiak-26/ 2022-09-12T19:13:43.722 DEBUG:boto:Data: 2022-09-12T19:13:43.722 DEBUG:boto:Headers: {} 2022-09-12T19:13:43.723 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:43.723 DEBUG:boto:Port: 8001 2022-09-12T19:13:43.724 DEBUG:boto:Params: {} 2022-09-12T19:13:43.724 DEBUG:boto:Token: None 2022-09-12T19:13:43.725 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:43 GMT /jdpiak-26/ 2022-09-12T19:13:43.725 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:86+G1HvzAxAd2hff3Nvtpvs+IEE= 2022-09-12T19:13:43.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:86+G1HvzAxAd2hff3Nvtpvs+IEE=', 'Content-Length': '0'} 2022-09-12T19:13:43.728 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd8ea11ac7af3906-00631f84e7-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:43.729 DEBUG:boto:path=/jdpiak-26/ 2022-09-12T19:13:43.730 DEBUG:boto:auth_path=/jdpiak-26/ 2022-09-12T19:13:43.730 DEBUG:boto:Method: HEAD 2022-09-12T19:13:43.731 DEBUG:boto:Path: /jdpiak-26/ 2022-09-12T19:13:43.731 DEBUG:boto:Data: 2022-09-12T19:13:43.732 DEBUG:boto:Headers: {} 2022-09-12T19:13:43.732 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:43.733 DEBUG:boto:Port: 8000 2022-09-12T19:13:43.733 DEBUG:boto:Params: {} 2022-09-12T19:13:43.734 DEBUG:boto:Token: None 2022-09-12T19:13:43.735 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:43 GMT /jdpiak-26/ 2022-09-12T19:13:43.735 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:86+G1HvzAxAd2hff3Nvtpvs+IEE= 2022-09-12T19:13:43.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:86+G1HvzAxAd2hff3Nvtpvs+IEE=', 'Content-Length': '0'} 2022-09-12T19:13:43.738 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6bcd834753c254e-00631f84e7-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:43.739 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:13:43.740 DEBUG:boto:path=/jdpiak-26/ 2022-09-12T19:13:43.741 DEBUG:boto:auth_path=/jdpiak-26/ 2022-09-12T19:13:43.741 DEBUG:boto:path=/jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.742 DEBUG:boto:auth_path=/jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.742 DEBUG:boto:Method: GET 2022-09-12T19:13:43.743 DEBUG:boto:Path: /jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.743 DEBUG:boto:Data: 2022-09-12T19:13:43.744 DEBUG:boto:Headers: {} 2022-09-12T19:13:43.744 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:43.745 DEBUG:boto:Port: 8001 2022-09-12T19:13:43.745 DEBUG:boto:Params: {} 2022-09-12T19:13:43.746 DEBUG:boto:Token: None 2022-09-12T19:13:43.747 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:43 GMT /jdpiak-26/?versions 2022-09-12T19:13:43.747 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pMD8Cb8AVD8MOgaD1cjF2Tumg8k= 2022-09-12T19:13:43.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pMD8Cb8AVD8MOgaD1cjF2Tumg8k=', 'Content-Length': '0'} 2022-09-12T19:13:43.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037773c148c36979d-00631f84e7-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:43.751 DEBUG:boto:b'jdpiak-26999falseobjnulltrue2022-09-12T19:11:55.284Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:43.752 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:13:43.753 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:13:43.753 DEBUG:boto:path=/jdpiak-26/ 2022-09-12T19:13:43.754 DEBUG:boto:auth_path=/jdpiak-26/ 2022-09-12T19:13:43.754 DEBUG:boto:path=/jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.755 DEBUG:boto:auth_path=/jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.755 DEBUG:boto:Method: GET 2022-09-12T19:13:43.756 DEBUG:boto:Path: /jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.756 DEBUG:boto:Data: 2022-09-12T19:13:43.757 DEBUG:boto:Headers: {} 2022-09-12T19:13:43.757 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:43.758 DEBUG:boto:Port: 8000 2022-09-12T19:13:43.758 DEBUG:boto:Params: {} 2022-09-12T19:13:43.759 DEBUG:boto:Token: None 2022-09-12T19:13:43.760 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:43 GMT /jdpiak-26/?versions 2022-09-12T19:13:43.760 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pMD8Cb8AVD8MOgaD1cjF2Tumg8k= 2022-09-12T19:13:43.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pMD8Cb8AVD8MOgaD1cjF2Tumg8k=', 'Content-Length': '0'} 2022-09-12T19:13:43.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000229e608a0d2f2cb2-00631f84e7-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:43.810 DEBUG:boto:b'jdpiak-26999falseobjnulltrue2022-09-12T19:11:55.284Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:43.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:13:43.811 DEBUG:boto:path=/jdpiak-26/ 2022-09-12T19:13:43.812 DEBUG:boto:auth_path=/jdpiak-26/ 2022-09-12T19:13:43.813 DEBUG:boto:path=/jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.813 DEBUG:boto:auth_path=/jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.814 DEBUG:boto:Method: GET 2022-09-12T19:13:43.814 DEBUG:boto:Path: /jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.815 DEBUG:boto:Data: 2022-09-12T19:13:43.815 DEBUG:boto:Headers: {} 2022-09-12T19:13:43.816 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:43.816 DEBUG:boto:Port: 8001 2022-09-12T19:13:43.817 DEBUG:boto:Params: {} 2022-09-12T19:13:43.817 DEBUG:boto:Token: None 2022-09-12T19:13:43.818 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:43 GMT /jdpiak-26/?versions 2022-09-12T19:13:43.819 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pMD8Cb8AVD8MOgaD1cjF2Tumg8k= 2022-09-12T19:13:43.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pMD8Cb8AVD8MOgaD1cjF2Tumg8k=', 'Content-Length': '0'} 2022-09-12T19:13:43.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0c94790f9d159f9-00631f84e7-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:43.822 DEBUG:boto:b'jdpiak-26999falseobjnulltrue2022-09-12T19:11:55.284Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:43.823 DEBUG:boto:path=/jdpiak-26/ 2022-09-12T19:13:43.824 DEBUG:boto:auth_path=/jdpiak-26/ 2022-09-12T19:13:43.824 DEBUG:boto:path=/jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.825 DEBUG:boto:auth_path=/jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.825 DEBUG:boto:Method: GET 2022-09-12T19:13:43.826 DEBUG:boto:Path: /jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.827 DEBUG:boto:Data: 2022-09-12T19:13:43.827 DEBUG:boto:Headers: {} 2022-09-12T19:13:43.828 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:43.828 DEBUG:boto:Port: 8000 2022-09-12T19:13:43.829 DEBUG:boto:Params: {} 2022-09-12T19:13:43.829 DEBUG:boto:Token: None 2022-09-12T19:13:43.830 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:43 GMT /jdpiak-26/?versions 2022-09-12T19:13:43.830 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pMD8Cb8AVD8MOgaD1cjF2Tumg8k= 2022-09-12T19:13:43.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pMD8Cb8AVD8MOgaD1cjF2Tumg8k=', 'Content-Length': '0'} 2022-09-12T19:13:43.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3a4d31025a9ee43-00631f84e7-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:43.834 DEBUG:boto:b'jdpiak-26999falseobjnulltrue2022-09-12T19:11:55.284Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:43.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:13:43.836 DEBUG:boto:path=/jdpiak-26/obj 2022-09-12T19:13:43.836 DEBUG:boto:auth_path=/jdpiak-26/obj 2022-09-12T19:13:43.837 DEBUG:boto:Method: HEAD 2022-09-12T19:13:43.837 DEBUG:boto:Path: /jdpiak-26/obj 2022-09-12T19:13:43.838 DEBUG:boto:Data: 2022-09-12T19:13:43.838 DEBUG:boto:Headers: {} 2022-09-12T19:13:43.839 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:43.839 DEBUG:boto:Port: 8001 2022-09-12T19:13:43.840 DEBUG:boto:Params: {} 2022-09-12T19:13:43.840 DEBUG:boto:Token: None 2022-09-12T19:13:43.841 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:43 GMT /jdpiak-26/obj 2022-09-12T19:13:43.842 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WrcYzCz3CpEfowcJqJMFUD7PHMA= 2022-09-12T19:13:43.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WrcYzCz3CpEfowcJqJMFUD7PHMA=', 'Content-Length': '0'} 2022-09-12T19:13:43.844 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000006671127af925f9e0-00631f84e7-10f2-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:43.845 DEBUG:boto:path=/jdpiak-26/obj 2022-09-12T19:13:43.845 DEBUG:boto:auth_path=/jdpiak-26/obj 2022-09-12T19:13:43.846 DEBUG:boto:Method: HEAD 2022-09-12T19:13:43.847 DEBUG:boto:Path: /jdpiak-26/obj 2022-09-12T19:13:43.847 DEBUG:boto:Data: 2022-09-12T19:13:43.848 DEBUG:boto:Headers: {} 2022-09-12T19:13:43.849 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:43.850 DEBUG:boto:Port: 8000 2022-09-12T19:13:43.851 DEBUG:boto:Params: {} 2022-09-12T19:13:43.852 DEBUG:boto:Token: None 2022-09-12T19:13:43.853 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:43 GMT /jdpiak-26/obj 2022-09-12T19:13:43.853 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WrcYzCz3CpEfowcJqJMFUD7PHMA= 2022-09-12T19:13:43.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WrcYzCz3CpEfowcJqJMFUD7PHMA=', 'Content-Length': '0'} 2022-09-12T19:13:43.856 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000facd50210f5ddfee-00631f84e7-10f6-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:43.856 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-26 zones={test-zone3, test-zone1} 2022-09-12T19:13:43.857 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-26 zones={test-zone3, test-zone2} 2022-09-12T19:13:43.858 DEBUG:boto:path=/jdpiak-26/ 2022-09-12T19:13:43.858 DEBUG:boto:auth_path=/jdpiak-26/ 2022-09-12T19:13:43.859 DEBUG:boto:Method: HEAD 2022-09-12T19:13:43.859 DEBUG:boto:Path: /jdpiak-26/ 2022-09-12T19:13:43.860 DEBUG:boto:Data: 2022-09-12T19:13:43.861 DEBUG:boto:Headers: {} 2022-09-12T19:13:43.862 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:43.863 DEBUG:boto:Port: 8001 2022-09-12T19:13:43.864 DEBUG:boto:Params: {} 2022-09-12T19:13:43.865 DEBUG:boto:Token: None 2022-09-12T19:13:43.866 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:43 GMT /jdpiak-26/ 2022-09-12T19:13:43.867 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:86+G1HvzAxAd2hff3Nvtpvs+IEE= 2022-09-12T19:13:43.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:86+G1HvzAxAd2hff3Nvtpvs+IEE=', 'Content-Length': '0'} 2022-09-12T19:13:43.870 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004babb2fabc39401f-00631f84e7-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:43.871 DEBUG:boto:path=/jdpiak-26/ 2022-09-12T19:13:43.872 DEBUG:boto:auth_path=/jdpiak-26/ 2022-09-12T19:13:43.872 DEBUG:boto:Method: HEAD 2022-09-12T19:13:43.873 DEBUG:boto:Path: /jdpiak-26/ 2022-09-12T19:13:43.873 DEBUG:boto:Data: 2022-09-12T19:13:43.874 DEBUG:boto:Headers: {} 2022-09-12T19:13:43.875 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:43.875 DEBUG:boto:Port: 8000 2022-09-12T19:13:43.876 DEBUG:boto:Params: {} 2022-09-12T19:13:43.876 DEBUG:boto:Token: None 2022-09-12T19:13:43.877 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:43 GMT /jdpiak-26/ 2022-09-12T19:13:43.878 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:86+G1HvzAxAd2hff3Nvtpvs+IEE= 2022-09-12T19:13:43.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:86+G1HvzAxAd2hff3Nvtpvs+IEE=', 'Content-Length': '0'} 2022-09-12T19:13:43.881 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000609835f2cf9c5aeb-00631f84e7-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:43.883 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:13:43.883 DEBUG:boto:path=/jdpiak-26/ 2022-09-12T19:13:43.884 DEBUG:boto:auth_path=/jdpiak-26/ 2022-09-12T19:13:43.884 DEBUG:boto:path=/jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.885 DEBUG:boto:auth_path=/jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.885 DEBUG:boto:Method: GET 2022-09-12T19:13:43.886 DEBUG:boto:Path: /jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.887 DEBUG:boto:Data: 2022-09-12T19:13:43.887 DEBUG:boto:Headers: {} 2022-09-12T19:13:43.888 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:43.888 DEBUG:boto:Port: 8001 2022-09-12T19:13:43.889 DEBUG:boto:Params: {} 2022-09-12T19:13:43.889 DEBUG:boto:Token: None 2022-09-12T19:13:43.890 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:43 GMT /jdpiak-26/?versions 2022-09-12T19:13:43.891 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pMD8Cb8AVD8MOgaD1cjF2Tumg8k= 2022-09-12T19:13:43.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pMD8Cb8AVD8MOgaD1cjF2Tumg8k=', 'Content-Length': '0'} 2022-09-12T19:13:43.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6e607bef478b930-00631f84e7-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:43.895 DEBUG:boto:b'jdpiak-26999falseobjnulltrue2022-09-12T19:11:55.284Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:43.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:13:43.896 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:13:43.897 DEBUG:boto:path=/jdpiak-26/ 2022-09-12T19:13:43.897 DEBUG:boto:auth_path=/jdpiak-26/ 2022-09-12T19:13:43.898 DEBUG:boto:path=/jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.898 DEBUG:boto:auth_path=/jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.899 DEBUG:boto:Method: GET 2022-09-12T19:13:43.900 DEBUG:boto:Path: /jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.900 DEBUG:boto:Data: 2022-09-12T19:13:43.901 DEBUG:boto:Headers: {} 2022-09-12T19:13:43.902 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:43.903 DEBUG:boto:Port: 8000 2022-09-12T19:13:43.903 DEBUG:boto:Params: {} 2022-09-12T19:13:43.904 DEBUG:boto:Token: None 2022-09-12T19:13:43.904 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:43 GMT /jdpiak-26/?versions 2022-09-12T19:13:43.905 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pMD8Cb8AVD8MOgaD1cjF2Tumg8k= 2022-09-12T19:13:43.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pMD8Cb8AVD8MOgaD1cjF2Tumg8k=', 'Content-Length': '0'} 2022-09-12T19:13:43.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f409d2d391d305d-00631f84e7-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:43.909 DEBUG:boto:b'jdpiak-26999falseobjnulltrue2022-09-12T19:11:55.284Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:43.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-12T19:13:43.911 DEBUG:boto:path=/jdpiak-26/ 2022-09-12T19:13:43.911 DEBUG:boto:auth_path=/jdpiak-26/ 2022-09-12T19:13:43.912 DEBUG:boto:path=/jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.912 DEBUG:boto:auth_path=/jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.913 DEBUG:boto:Method: GET 2022-09-12T19:13:43.913 DEBUG:boto:Path: /jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.914 DEBUG:boto:Data: 2022-09-12T19:13:43.914 DEBUG:boto:Headers: {} 2022-09-12T19:13:43.915 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:43.916 DEBUG:boto:Port: 8001 2022-09-12T19:13:43.917 DEBUG:boto:Params: {} 2022-09-12T19:13:43.917 DEBUG:boto:Token: None 2022-09-12T19:13:43.918 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:43 GMT /jdpiak-26/?versions 2022-09-12T19:13:43.919 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pMD8Cb8AVD8MOgaD1cjF2Tumg8k= 2022-09-12T19:13:43.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pMD8Cb8AVD8MOgaD1cjF2Tumg8k=', 'Content-Length': '0'} 2022-09-12T19:13:43.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de80cd223a376831-00631f84e7-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:43.966 DEBUG:boto:b'jdpiak-26999falseobjnulltrue2022-09-12T19:11:55.284Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:43.970 DEBUG:boto:path=/jdpiak-26/ 2022-09-12T19:13:43.970 DEBUG:boto:auth_path=/jdpiak-26/ 2022-09-12T19:13:43.971 DEBUG:boto:path=/jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.972 DEBUG:boto:auth_path=/jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.972 DEBUG:boto:Method: GET 2022-09-12T19:13:43.973 DEBUG:boto:Path: /jdpiak-26/?versions&max-keys=999 2022-09-12T19:13:43.974 DEBUG:boto:Data: 2022-09-12T19:13:43.974 DEBUG:boto:Headers: {} 2022-09-12T19:13:43.975 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:43.975 DEBUG:boto:Port: 8000 2022-09-12T19:13:43.976 DEBUG:boto:Params: {} 2022-09-12T19:13:43.976 DEBUG:boto:Token: None 2022-09-12T19:13:43.977 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:43 GMT /jdpiak-26/?versions 2022-09-12T19:13:43.977 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pMD8Cb8AVD8MOgaD1cjF2Tumg8k= 2022-09-12T19:13:43.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pMD8Cb8AVD8MOgaD1cjF2Tumg8k=', 'Content-Length': '0'} 2022-09-12T19:13:43.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075d00dc0052efa96-00631f84e7-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:43.982 DEBUG:boto:b'jdpiak-26999falseobjnulltrue2022-09-12T19:11:55.284Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:13:43.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-12T19:13:43.985 DEBUG:boto:path=/jdpiak-26/obj 2022-09-12T19:13:43.986 DEBUG:boto:auth_path=/jdpiak-26/obj 2022-09-12T19:13:43.987 DEBUG:boto:Method: HEAD 2022-09-12T19:13:43.988 DEBUG:boto:Path: /jdpiak-26/obj 2022-09-12T19:13:43.988 DEBUG:boto:Data: 2022-09-12T19:13:43.989 DEBUG:boto:Headers: {} 2022-09-12T19:13:43.990 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:43.990 DEBUG:boto:Port: 8001 2022-09-12T19:13:43.991 DEBUG:boto:Params: {} 2022-09-12T19:13:43.992 DEBUG:boto:Token: None 2022-09-12T19:13:43.993 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:43 GMT /jdpiak-26/obj 2022-09-12T19:13:43.994 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WrcYzCz3CpEfowcJqJMFUD7PHMA= 2022-09-12T19:13:43.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WrcYzCz3CpEfowcJqJMFUD7PHMA=', 'Content-Length': '0'} 2022-09-12T19:13:43.996 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000c7010fb5115b85ef-00631f84e7-10f2-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:43.997 DEBUG:boto:path=/jdpiak-26/obj 2022-09-12T19:13:43.998 DEBUG:boto:auth_path=/jdpiak-26/obj 2022-09-12T19:13:43.998 DEBUG:boto:Method: HEAD 2022-09-12T19:13:43.999 DEBUG:boto:Path: /jdpiak-26/obj 2022-09-12T19:13:43.999 DEBUG:boto:Data: 2022-09-12T19:13:44.000 DEBUG:boto:Headers: {} 2022-09-12T19:13:44.000 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:44.001 DEBUG:boto:Port: 8000 2022-09-12T19:13:44.002 DEBUG:boto:Params: {} 2022-09-12T19:13:44.002 DEBUG:boto:Token: None 2022-09-12T19:13:44.003 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:13:44 GMT /jdpiak-26/obj 2022-09-12T19:13:44.003 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TrQ/DZ2prawDBn0D0+brrL1mXKY= 2022-09-12T19:13:44.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TrQ/DZ2prawDBn0D0+brrL1mXKY=', 'Content-Length': '0'} 2022-09-12T19:13:44.006 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000d9dd654b44d85a49-00631f84e8-10e4-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.006 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-26 zones={test-zone3, test-zone2} 2022-09-12T19:13:44.007 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2022-09-12T19:13:44.009 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-27 2022-09-12T19:13:44.009 DEBUG:boto:path=/jdpiak-27/ 2022-09-12T19:13:44.010 DEBUG:boto:auth_path=/jdpiak-27/ 2022-09-12T19:13:44.010 DEBUG:boto:Method: PUT 2022-09-12T19:13:44.011 DEBUG:boto:Path: /jdpiak-27/ 2022-09-12T19:13:44.011 DEBUG:boto:Data: 2022-09-12T19:13:44.012 DEBUG:boto:Headers: {} 2022-09-12T19:13:44.012 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:44.013 DEBUG:boto:Port: 8000 2022-09-12T19:13:44.013 DEBUG:boto:Params: {} 2022-09-12T19:13:44.014 DEBUG:boto:Token: None 2022-09-12T19:13:44.015 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:13:44 GMT /jdpiak-27/ 2022-09-12T19:13:44.015 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:35cA+pBDTyfQ7UhSBNrVZw8XWiU= 2022-09-12T19:13:44.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:35cA+pBDTyfQ7UhSBNrVZw8XWiU=', 'Content-Length': '0'} 2022-09-12T19:13:44.039 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a3dff37c7857ccc5-00631f84e8-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.040 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jdpiak-28 2022-09-12T19:13:44.040 DEBUG:boto:path=/jdpiak-28/ 2022-09-12T19:13:44.041 DEBUG:boto:auth_path=/jdpiak-28/ 2022-09-12T19:13:44.042 DEBUG:boto:Method: PUT 2022-09-12T19:13:44.042 DEBUG:boto:Path: /jdpiak-28/ 2022-09-12T19:13:44.043 DEBUG:boto:Data: 2022-09-12T19:13:44.044 DEBUG:boto:Headers: {} 2022-09-12T19:13:44.045 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:44.046 DEBUG:boto:Port: 8000 2022-09-12T19:13:44.047 DEBUG:boto:Params: {} 2022-09-12T19:13:44.048 DEBUG:boto:Token: None 2022-09-12T19:13:44.049 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:13:44 GMT /jdpiak-28/ 2022-09-12T19:13:44.050 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jbEFTT8OJeL2fmoYjAPdPQn/W6M= 2022-09-12T19:13:44.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jbEFTT8OJeL2fmoYjAPdPQn/W6M=', 'Content-Length': '0'} 2022-09-12T19:13:44.091 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000009ac12b559509489-00631f84e8-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.092 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jdpiak-29 2022-09-12T19:13:44.093 DEBUG:boto:path=/jdpiak-29/ 2022-09-12T19:13:44.094 DEBUG:boto:auth_path=/jdpiak-29/ 2022-09-12T19:13:44.094 DEBUG:boto:Method: PUT 2022-09-12T19:13:44.095 DEBUG:boto:Path: /jdpiak-29/ 2022-09-12T19:13:44.096 DEBUG:boto:Data: 2022-09-12T19:13:44.097 DEBUG:boto:Headers: {} 2022-09-12T19:13:44.098 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:44.099 DEBUG:boto:Port: 8001 2022-09-12T19:13:44.100 DEBUG:boto:Params: {} 2022-09-12T19:13:44.100 DEBUG:boto:Token: None 2022-09-12T19:13:44.101 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:13:44 GMT /jdpiak-29/ 2022-09-12T19:13:44.102 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:i7uqR8g1EV/HkLoTDqlXA0yvzL0= 2022-09-12T19:13:44.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:i7uqR8g1EV/HkLoTDqlXA0yvzL0=', 'Content-Length': '0'} 2022-09-12T19:13:44.140 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e70441ddf4f36345-00631f84e8-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.142 DEBUG:boto:path=/jdpiak-27/ 2022-09-12T19:13:44.143 DEBUG:boto:auth_path=/jdpiak-27/ 2022-09-12T19:13:44.143 DEBUG:boto:path=/jdpiak-27/?versioning 2022-09-12T19:13:44.144 DEBUG:boto:auth_path=/jdpiak-27/?versioning 2022-09-12T19:13:44.145 DEBUG:boto:Method: PUT 2022-09-12T19:13:44.146 DEBUG:boto:Path: /jdpiak-27/?versioning 2022-09-12T19:13:44.147 DEBUG:boto:Data: Enabled Disabled 2022-09-12T19:13:44.148 DEBUG:boto:Headers: {} 2022-09-12T19:13:44.149 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:44.150 DEBUG:boto:Port: 8000 2022-09-12T19:13:44.150 DEBUG:boto:Params: {} 2022-09-12T19:13:44.151 DEBUG:boto:Token: None 2022-09-12T19:13:44.152 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:13:44 GMT /jdpiak-27/?versioning 2022-09-12T19:13:44.153 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:36FvKYziHqAmYGqImIO4H/Ke4Ws= 2022-09-12T19:13:44.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:36FvKYziHqAmYGqImIO4H/Ke4Ws=', 'Content-Length': '228'} 2022-09-12T19:13:44.158 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007e8cbd09196c4b4c-00631f84e8-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.159 DEBUG:boto:path=/jdpiak-27/ 2022-09-12T19:13:44.160 DEBUG:boto:auth_path=/jdpiak-27/ 2022-09-12T19:13:44.160 DEBUG:boto:path=/jdpiak-27/?versioning 2022-09-12T19:13:44.161 DEBUG:boto:auth_path=/jdpiak-27/?versioning 2022-09-12T19:13:44.161 DEBUG:boto:Method: GET 2022-09-12T19:13:44.162 DEBUG:boto:Path: /jdpiak-27/?versioning 2022-09-12T19:13:44.163 DEBUG:boto:Data: 2022-09-12T19:13:44.164 DEBUG:boto:Headers: {} 2022-09-12T19:13:44.164 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:44.165 DEBUG:boto:Port: 8000 2022-09-12T19:13:44.166 DEBUG:boto:Params: {} 2022-09-12T19:13:44.167 DEBUG:boto:Token: None 2022-09-12T19:13:44.168 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:44 GMT /jdpiak-27/?versioning 2022-09-12T19:13:44.169 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W/FVbmPMyK3y26X1xYpuTnttrOY= 2022-09-12T19:13:44.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W/FVbmPMyK3y26X1xYpuTnttrOY=', 'Content-Length': '0'} 2022-09-12T19:13:44.172 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008c7033e482ff4925-00631f84e8-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.173 DEBUG:boto:EnabledDisabled 2022-09-12T19:13:44.175 DEBUG:boto:path=/jdpiak-28/ 2022-09-12T19:13:44.175 DEBUG:boto:auth_path=/jdpiak-28/ 2022-09-12T19:13:44.176 DEBUG:boto:path=/jdpiak-28/?versioning 2022-09-12T19:13:44.176 DEBUG:boto:auth_path=/jdpiak-28/?versioning 2022-09-12T19:13:44.177 DEBUG:boto:Method: PUT 2022-09-12T19:13:44.177 DEBUG:boto:Path: /jdpiak-28/?versioning 2022-09-12T19:13:44.178 DEBUG:boto:Data: Enabled Disabled 2022-09-12T19:13:44.178 DEBUG:boto:Headers: {} 2022-09-12T19:13:44.179 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:44.179 DEBUG:boto:Port: 8000 2022-09-12T19:13:44.180 DEBUG:boto:Params: {} 2022-09-12T19:13:44.180 DEBUG:boto:Token: None 2022-09-12T19:13:44.181 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:13:44 GMT /jdpiak-28/?versioning 2022-09-12T19:13:44.181 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BPW4wtbTWeNa25NS5LbpcK9E5hQ= 2022-09-12T19:13:44.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BPW4wtbTWeNa25NS5LbpcK9E5hQ=', 'Content-Length': '228'} 2022-09-12T19:13:44.198 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e4a6c80ff099c14c-00631f84e8-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.199 DEBUG:boto:path=/jdpiak-28/ 2022-09-12T19:13:44.199 DEBUG:boto:auth_path=/jdpiak-28/ 2022-09-12T19:13:44.200 DEBUG:boto:path=/jdpiak-28/?versioning 2022-09-12T19:13:44.200 DEBUG:boto:auth_path=/jdpiak-28/?versioning 2022-09-12T19:13:44.201 DEBUG:boto:Method: GET 2022-09-12T19:13:44.201 DEBUG:boto:Path: /jdpiak-28/?versioning 2022-09-12T19:13:44.202 DEBUG:boto:Data: 2022-09-12T19:13:44.202 DEBUG:boto:Headers: {} 2022-09-12T19:13:44.203 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:44.203 DEBUG:boto:Port: 8000 2022-09-12T19:13:44.204 DEBUG:boto:Params: {} 2022-09-12T19:13:44.204 DEBUG:boto:Token: None 2022-09-12T19:13:44.205 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:44 GMT /jdpiak-28/?versioning 2022-09-12T19:13:44.205 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0ye58bigG+2AoXxw988ddsW2hRA= 2022-09-12T19:13:44.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0ye58bigG+2AoXxw988ddsW2hRA=', 'Content-Length': '0'} 2022-09-12T19:13:44.207 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cf9a9b84a049273c-00631f84e8-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.250 DEBUG:boto:EnabledDisabled 2022-09-12T19:13:44.251 DEBUG:boto:path=/jdpiak-29/ 2022-09-12T19:13:44.251 DEBUG:boto:auth_path=/jdpiak-29/ 2022-09-12T19:13:44.252 DEBUG:boto:path=/jdpiak-29/?versioning 2022-09-12T19:13:44.252 DEBUG:boto:auth_path=/jdpiak-29/?versioning 2022-09-12T19:13:44.252 DEBUG:boto:Method: PUT 2022-09-12T19:13:44.253 DEBUG:boto:Path: /jdpiak-29/?versioning 2022-09-12T19:13:44.253 DEBUG:boto:Data: Enabled Disabled 2022-09-12T19:13:44.254 DEBUG:boto:Headers: {} 2022-09-12T19:13:44.254 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:44.255 DEBUG:boto:Port: 8001 2022-09-12T19:13:44.255 DEBUG:boto:Params: {} 2022-09-12T19:13:44.256 DEBUG:boto:Token: None 2022-09-12T19:13:44.256 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:13:44 GMT /jdpiak-29/?versioning 2022-09-12T19:13:44.257 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PiTPliUWyp9m9f6cxteEKUNcUyk= 2022-09-12T19:13:44.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PiTPliUWyp9m9f6cxteEKUNcUyk=', 'Content-Length': '228'} 2022-09-12T19:13:44.265 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fa47801cef3db23d-00631f84e8-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.266 DEBUG:boto:path=/jdpiak-29/ 2022-09-12T19:13:44.266 DEBUG:boto:auth_path=/jdpiak-29/ 2022-09-12T19:13:44.267 DEBUG:boto:path=/jdpiak-29/?versioning 2022-09-12T19:13:44.267 DEBUG:boto:auth_path=/jdpiak-29/?versioning 2022-09-12T19:13:44.268 DEBUG:boto:Method: GET 2022-09-12T19:13:44.268 DEBUG:boto:Path: /jdpiak-29/?versioning 2022-09-12T19:13:44.269 DEBUG:boto:Data: 2022-09-12T19:13:44.270 DEBUG:boto:Headers: {} 2022-09-12T19:13:44.270 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:44.271 DEBUG:boto:Port: 8001 2022-09-12T19:13:44.271 DEBUG:boto:Params: {} 2022-09-12T19:13:44.272 DEBUG:boto:Token: None 2022-09-12T19:13:44.272 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:44 GMT /jdpiak-29/?versioning 2022-09-12T19:13:44.273 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bH5529gjGuhmxvP22XKqLHrtko0= 2022-09-12T19:13:44.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bH5529gjGuhmxvP22XKqLHrtko0=', 'Content-Length': '0'} 2022-09-12T19:13:44.275 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000035d1e47876966ed5-00631f84e8-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.276 DEBUG:boto:EnabledDisabled 2022-09-12T19:13:44.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2022-09-12T19:13:44.279 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-30 2022-09-12T19:13:44.280 DEBUG:boto:path=/jdpiak-30/ 2022-09-12T19:13:44.281 DEBUG:boto:auth_path=/jdpiak-30/ 2022-09-12T19:13:44.281 DEBUG:boto:Method: PUT 2022-09-12T19:13:44.282 DEBUG:boto:Path: /jdpiak-30/ 2022-09-12T19:13:44.283 DEBUG:boto:Data: 2022-09-12T19:13:44.284 DEBUG:boto:Headers: {} 2022-09-12T19:13:44.285 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:44.286 DEBUG:boto:Port: 8000 2022-09-12T19:13:44.286 DEBUG:boto:Params: {} 2022-09-12T19:13:44.287 DEBUG:boto:Token: None 2022-09-12T19:13:44.288 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:13:44 GMT /jdpiak-30/ 2022-09-12T19:13:44.288 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PtjcTyjhyEw++3Ubn5GoU0MEHHc= 2022-09-12T19:13:44.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PtjcTyjhyEw++3Ubn5GoU0MEHHc=', 'Content-Length': '0'} 2022-09-12T19:13:44.304 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007a641f329ce9e387-00631f84e8-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.305 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jdpiak-31 2022-09-12T19:13:44.306 DEBUG:boto:path=/jdpiak-31/ 2022-09-12T19:13:44.306 DEBUG:boto:auth_path=/jdpiak-31/ 2022-09-12T19:13:44.307 DEBUG:boto:Method: PUT 2022-09-12T19:13:44.307 DEBUG:boto:Path: /jdpiak-31/ 2022-09-12T19:13:44.308 DEBUG:boto:Data: 2022-09-12T19:13:44.308 DEBUG:boto:Headers: {} 2022-09-12T19:13:44.309 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:44.309 DEBUG:boto:Port: 8000 2022-09-12T19:13:44.310 DEBUG:boto:Params: {} 2022-09-12T19:13:44.310 DEBUG:boto:Token: None 2022-09-12T19:13:44.311 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:13:44 GMT /jdpiak-31/ 2022-09-12T19:13:44.312 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DWy91FwUriie1gGP3pgNw7p3yb4= 2022-09-12T19:13:44.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DWy91FwUriie1gGP3pgNw7p3yb4=', 'Content-Length': '0'} 2022-09-12T19:13:44.363 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ad06d4ea82f049f4-00631f84e8-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.364 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jdpiak-32 2022-09-12T19:13:44.365 DEBUG:boto:path=/jdpiak-32/ 2022-09-12T19:13:44.365 DEBUG:boto:auth_path=/jdpiak-32/ 2022-09-12T19:13:44.366 DEBUG:boto:Method: PUT 2022-09-12T19:13:44.367 DEBUG:boto:Path: /jdpiak-32/ 2022-09-12T19:13:44.367 DEBUG:boto:Data: 2022-09-12T19:13:44.368 DEBUG:boto:Headers: {} 2022-09-12T19:13:44.368 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:44.369 DEBUG:boto:Port: 8001 2022-09-12T19:13:44.369 DEBUG:boto:Params: {} 2022-09-12T19:13:44.370 DEBUG:boto:Token: None 2022-09-12T19:13:44.371 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:13:44 GMT /jdpiak-32/ 2022-09-12T19:13:44.371 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PCd2MnfE3G25OvzipPzEYV/jIFo= 2022-09-12T19:13:44.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PCd2MnfE3G25OvzipPzEYV/jIFo=', 'Content-Length': '0'} 2022-09-12T19:13:44.407 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000781edcd183eccc83-00631f84e8-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.408 DEBUG:boto:path=/jdpiak-30/ 2022-09-12T19:13:44.409 DEBUG:boto:auth_path=/jdpiak-30/ 2022-09-12T19:13:44.410 DEBUG:boto:path=/jdpiak-30/?acl 2022-09-12T19:13:44.410 DEBUG:boto:auth_path=/jdpiak-30/?acl 2022-09-12T19:13:44.411 DEBUG:boto:Method: GET 2022-09-12T19:13:44.411 DEBUG:boto:Path: /jdpiak-30/?acl 2022-09-12T19:13:44.412 DEBUG:boto:Data: 2022-09-12T19:13:44.412 DEBUG:boto:Headers: {} 2022-09-12T19:13:44.413 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:44.413 DEBUG:boto:Port: 8000 2022-09-12T19:13:44.414 DEBUG:boto:Params: {} 2022-09-12T19:13:44.414 DEBUG:boto:Token: None 2022-09-12T19:13:44.415 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:44 GMT /jdpiak-30/?acl 2022-09-12T19:13:44.415 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eTvtOEjouysVV3blXsvtmrsHZSw= 2022-09-12T19:13:44.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eTvtOEjouysVV3blXsvtmrsHZSw=', 'Content-Length': '0'} 2022-09-12T19:13:44.417 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001952c0f02aa732a5-00631f84e8-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.418 DEBUG:boto:path=/jdpiak-30/ 2022-09-12T19:13:44.419 DEBUG:boto:auth_path=/jdpiak-30/ 2022-09-12T19:13:44.419 DEBUG:boto:path=/jdpiak-30/?acl 2022-09-12T19:13:44.420 DEBUG:boto:auth_path=/jdpiak-30/?acl 2022-09-12T19:13:44.420 DEBUG:boto:Method: PUT 2022-09-12T19:13:44.421 DEBUG:boto:Path: /jdpiak-30/?acl 2022-09-12T19:13:44.421 DEBUG:boto:Data: 2022-09-12T19:13:44.422 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-12T19:13:44.422 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:44.423 DEBUG:boto:Port: 8000 2022-09-12T19:13:44.424 DEBUG:boto:Params: {} 2022-09-12T19:13:44.424 DEBUG:boto:Token: None 2022-09-12T19:13:44.425 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:13:44 GMT x-amz-acl:public-read /jdpiak-30/?acl 2022-09-12T19:13:44.425 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MQgwHzLnfco60g8YacH3Tmid8jY= 2022-09-12T19:13:44.426 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MQgwHzLnfco60g8YacH3Tmid8jY=', 'Content-Length': '0'} 2022-09-12T19:13:44.434 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000066995b85a22b7692-00631f84e8-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.435 DEBUG:boto:path=/jdpiak-30/ 2022-09-12T19:13:44.435 DEBUG:boto:auth_path=/jdpiak-30/ 2022-09-12T19:13:44.436 DEBUG:boto:path=/jdpiak-30/?acl 2022-09-12T19:13:44.436 DEBUG:boto:auth_path=/jdpiak-30/?acl 2022-09-12T19:13:44.437 DEBUG:boto:Method: GET 2022-09-12T19:13:44.437 DEBUG:boto:Path: /jdpiak-30/?acl 2022-09-12T19:13:44.438 DEBUG:boto:Data: 2022-09-12T19:13:44.438 DEBUG:boto:Headers: {} 2022-09-12T19:13:44.439 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:44.439 DEBUG:boto:Port: 8000 2022-09-12T19:13:44.440 DEBUG:boto:Params: {} 2022-09-12T19:13:44.440 DEBUG:boto:Token: None 2022-09-12T19:13:44.441 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:44 GMT /jdpiak-30/?acl 2022-09-12T19:13:44.441 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eTvtOEjouysVV3blXsvtmrsHZSw= 2022-09-12T19:13:44.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eTvtOEjouysVV3blXsvtmrsHZSw=', 'Content-Length': '0'} 2022-09-12T19:13:44.443 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000032a75c8248b83c7a-00631f84e8-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.486 DEBUG:boto:path=/jdpiak-31/ 2022-09-12T19:13:44.487 DEBUG:boto:auth_path=/jdpiak-31/ 2022-09-12T19:13:44.488 DEBUG:boto:path=/jdpiak-31/?acl 2022-09-12T19:13:44.489 DEBUG:boto:auth_path=/jdpiak-31/?acl 2022-09-12T19:13:44.490 DEBUG:boto:Method: GET 2022-09-12T19:13:44.490 DEBUG:boto:Path: /jdpiak-31/?acl 2022-09-12T19:13:44.491 DEBUG:boto:Data: 2022-09-12T19:13:44.491 DEBUG:boto:Headers: {} 2022-09-12T19:13:44.492 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:44.492 DEBUG:boto:Port: 8000 2022-09-12T19:13:44.493 DEBUG:boto:Params: {} 2022-09-12T19:13:44.493 DEBUG:boto:Token: None 2022-09-12T19:13:44.494 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:44 GMT /jdpiak-31/?acl 2022-09-12T19:13:44.494 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6etIBKjeSO7CHNjFkUnUk+45gPg= 2022-09-12T19:13:44.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6etIBKjeSO7CHNjFkUnUk+45gPg=', 'Content-Length': '0'} 2022-09-12T19:13:44.496 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005e300a824f45386e-00631f84e8-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.497 DEBUG:boto:path=/jdpiak-31/ 2022-09-12T19:13:44.498 DEBUG:boto:auth_path=/jdpiak-31/ 2022-09-12T19:13:44.498 DEBUG:boto:path=/jdpiak-31/?acl 2022-09-12T19:13:44.499 DEBUG:boto:auth_path=/jdpiak-31/?acl 2022-09-12T19:13:44.500 DEBUG:boto:Method: PUT 2022-09-12T19:13:44.500 DEBUG:boto:Path: /jdpiak-31/?acl 2022-09-12T19:13:44.501 DEBUG:boto:Data: 2022-09-12T19:13:44.501 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-12T19:13:44.502 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:44.502 DEBUG:boto:Port: 8000 2022-09-12T19:13:44.503 DEBUG:boto:Params: {} 2022-09-12T19:13:44.503 DEBUG:boto:Token: None 2022-09-12T19:13:44.504 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:13:44 GMT x-amz-acl:public-read /jdpiak-31/?acl 2022-09-12T19:13:44.504 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u12Mj1lsl4iMoHdk6XvSobeBpMo= 2022-09-12T19:13:44.505 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u12Mj1lsl4iMoHdk6XvSobeBpMo=', 'Content-Length': '0'} 2022-09-12T19:13:44.516 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006265752cdc4b5188-00631f84e8-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.517 DEBUG:boto:path=/jdpiak-31/ 2022-09-12T19:13:44.518 DEBUG:boto:auth_path=/jdpiak-31/ 2022-09-12T19:13:44.519 DEBUG:boto:path=/jdpiak-31/?acl 2022-09-12T19:13:44.519 DEBUG:boto:auth_path=/jdpiak-31/?acl 2022-09-12T19:13:44.520 DEBUG:boto:Method: GET 2022-09-12T19:13:44.520 DEBUG:boto:Path: /jdpiak-31/?acl 2022-09-12T19:13:44.521 DEBUG:boto:Data: 2022-09-12T19:13:44.521 DEBUG:boto:Headers: {} 2022-09-12T19:13:44.521 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:44.522 DEBUG:boto:Port: 8000 2022-09-12T19:13:44.522 DEBUG:boto:Params: {} 2022-09-12T19:13:44.523 DEBUG:boto:Token: None 2022-09-12T19:13:44.523 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:44 GMT /jdpiak-31/?acl 2022-09-12T19:13:44.524 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6etIBKjeSO7CHNjFkUnUk+45gPg= 2022-09-12T19:13:44.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6etIBKjeSO7CHNjFkUnUk+45gPg=', 'Content-Length': '0'} 2022-09-12T19:13:44.526 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c45efa025d66a47d-00631f84e8-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.527 DEBUG:boto:path=/jdpiak-32/ 2022-09-12T19:13:44.527 DEBUG:boto:auth_path=/jdpiak-32/ 2022-09-12T19:13:44.528 DEBUG:boto:path=/jdpiak-32/?acl 2022-09-12T19:13:44.528 DEBUG:boto:auth_path=/jdpiak-32/?acl 2022-09-12T19:13:44.529 DEBUG:boto:Method: GET 2022-09-12T19:13:44.529 DEBUG:boto:Path: /jdpiak-32/?acl 2022-09-12T19:13:44.530 DEBUG:boto:Data: 2022-09-12T19:13:44.530 DEBUG:boto:Headers: {} 2022-09-12T19:13:44.531 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:44.532 DEBUG:boto:Port: 8001 2022-09-12T19:13:44.532 DEBUG:boto:Params: {} 2022-09-12T19:13:44.533 DEBUG:boto:Token: None 2022-09-12T19:13:44.534 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:44 GMT /jdpiak-32/?acl 2022-09-12T19:13:44.535 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2p/uKyze5NTf2odZ8gbfioXz/T4= 2022-09-12T19:13:44.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2p/uKyze5NTf2odZ8gbfioXz/T4=', 'Content-Length': '0'} 2022-09-12T19:13:44.537 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000775f3ebc3ea45aad-00631f84e8-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.538 DEBUG:boto:path=/jdpiak-32/ 2022-09-12T19:13:44.538 DEBUG:boto:auth_path=/jdpiak-32/ 2022-09-12T19:13:44.539 DEBUG:boto:path=/jdpiak-32/?acl 2022-09-12T19:13:44.539 DEBUG:boto:auth_path=/jdpiak-32/?acl 2022-09-12T19:13:44.539 DEBUG:boto:Method: PUT 2022-09-12T19:13:44.540 DEBUG:boto:Path: /jdpiak-32/?acl 2022-09-12T19:13:44.540 DEBUG:boto:Data: 2022-09-12T19:13:44.541 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-12T19:13:44.541 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:44.542 DEBUG:boto:Port: 8001 2022-09-12T19:13:44.542 DEBUG:boto:Params: {} 2022-09-12T19:13:44.543 DEBUG:boto:Token: None 2022-09-12T19:13:44.543 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:13:44 GMT x-amz-acl:public-read /jdpiak-32/?acl 2022-09-12T19:13:44.544 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8CyN/k0++H5/kTKMw9NTwlEYz0A= 2022-09-12T19:13:44.545 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8CyN/k0++H5/kTKMw9NTwlEYz0A=', 'Content-Length': '0'} 2022-09-12T19:13:44.562 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e5183fe5a1b8a5f7-00631f84e8-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.564 DEBUG:boto:path=/jdpiak-32/ 2022-09-12T19:13:44.564 DEBUG:boto:auth_path=/jdpiak-32/ 2022-09-12T19:13:44.565 DEBUG:boto:path=/jdpiak-32/?acl 2022-09-12T19:13:44.565 DEBUG:boto:auth_path=/jdpiak-32/?acl 2022-09-12T19:13:44.565 DEBUG:boto:Method: GET 2022-09-12T19:13:44.566 DEBUG:boto:Path: /jdpiak-32/?acl 2022-09-12T19:13:44.566 DEBUG:boto:Data: 2022-09-12T19:13:44.567 DEBUG:boto:Headers: {} 2022-09-12T19:13:44.567 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:44.568 DEBUG:boto:Port: 8001 2022-09-12T19:13:44.568 DEBUG:boto:Params: {} 2022-09-12T19:13:44.569 DEBUG:boto:Token: None 2022-09-12T19:13:44.569 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:44 GMT /jdpiak-32/?acl 2022-09-12T19:13:44.570 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2p/uKyze5NTf2odZ8gbfioXz/T4= 2022-09-12T19:13:44.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2p/uKyze5NTf2odZ8gbfioXz/T4=', 'Content-Length': '0'} 2022-09-12T19:13:44.572 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000679463fa600f86c0-00631f84e8-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2022-09-12T19:13:44.615 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-33 2022-09-12T19:13:44.616 DEBUG:boto:path=/jdpiak-33/ 2022-09-12T19:13:44.616 DEBUG:boto:auth_path=/jdpiak-33/ 2022-09-12T19:13:44.617 DEBUG:boto:Method: PUT 2022-09-12T19:13:44.618 DEBUG:boto:Path: /jdpiak-33/ 2022-09-12T19:13:44.618 DEBUG:boto:Data: 2022-09-12T19:13:44.619 DEBUG:boto:Headers: {} 2022-09-12T19:13:44.619 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:44.620 DEBUG:boto:Port: 8000 2022-09-12T19:13:44.620 DEBUG:boto:Params: {} 2022-09-12T19:13:44.621 DEBUG:boto:Token: None 2022-09-12T19:13:44.622 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:13:44 GMT /jdpiak-33/ 2022-09-12T19:13:44.622 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YjrdddDPf4hobo97U+xfhDhJ04I= 2022-09-12T19:13:44.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YjrdddDPf4hobo97U+xfhDhJ04I=', 'Content-Length': '0'} 2022-09-12T19:13:44.638 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003743892408b3cc63-00631f84e8-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.639 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jdpiak-34 2022-09-12T19:13:44.640 DEBUG:boto:path=/jdpiak-34/ 2022-09-12T19:13:44.640 DEBUG:boto:auth_path=/jdpiak-34/ 2022-09-12T19:13:44.641 DEBUG:boto:Method: PUT 2022-09-12T19:13:44.641 DEBUG:boto:Path: /jdpiak-34/ 2022-09-12T19:13:44.642 DEBUG:boto:Data: 2022-09-12T19:13:44.642 DEBUG:boto:Headers: {} 2022-09-12T19:13:44.643 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:44.643 DEBUG:boto:Port: 8000 2022-09-12T19:13:44.644 DEBUG:boto:Params: {} 2022-09-12T19:13:44.644 DEBUG:boto:Token: None 2022-09-12T19:13:44.645 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:13:44 GMT /jdpiak-34/ 2022-09-12T19:13:44.645 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CzYqnPqgsavk4DDq8BAw7cCiN00= 2022-09-12T19:13:44.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CzYqnPqgsavk4DDq8BAw7cCiN00=', 'Content-Length': '0'} 2022-09-12T19:13:44.671 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000476a374b7f00ee2c-00631f84e8-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.672 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jdpiak-35 2022-09-12T19:13:44.672 DEBUG:boto:path=/jdpiak-35/ 2022-09-12T19:13:44.673 DEBUG:boto:auth_path=/jdpiak-35/ 2022-09-12T19:13:44.673 DEBUG:boto:Method: PUT 2022-09-12T19:13:44.674 DEBUG:boto:Path: /jdpiak-35/ 2022-09-12T19:13:44.674 DEBUG:boto:Data: 2022-09-12T19:13:44.675 DEBUG:boto:Headers: {} 2022-09-12T19:13:44.675 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:44.676 DEBUG:boto:Port: 8001 2022-09-12T19:13:44.676 DEBUG:boto:Params: {} 2022-09-12T19:13:44.677 DEBUG:boto:Token: None 2022-09-12T19:13:44.678 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:13:44 GMT /jdpiak-35/ 2022-09-12T19:13:44.679 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MFn/QeD6lFo8w+1ro7O5kczRhjM= 2022-09-12T19:13:44.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MFn/QeD6lFo8w+1ro7O5kczRhjM=', 'Content-Length': '0'} 2022-09-12T19:13:44.704 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005e2854bb5b05d7e0-00631f84e8-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.705 DEBUG:boto:path=/jdpiak-33/ 2022-09-12T19:13:44.706 DEBUG:boto:auth_path=/jdpiak-33/ 2022-09-12T19:13:44.707 DEBUG:boto:path=/jdpiak-33/?cors 2022-09-12T19:13:44.707 DEBUG:boto:auth_path=/jdpiak-33/?cors 2022-09-12T19:13:44.708 DEBUG:boto:Method: PUT 2022-09-12T19:13:44.708 DEBUG:boto:Path: /jdpiak-33/?cors 2022-09-12T19:13:44.709 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2022-09-12T19:13:44.709 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2022-09-12T19:13:44.709 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:44.710 DEBUG:boto:Port: 8000 2022-09-12T19:13:44.710 DEBUG:boto:Params: {} 2022-09-12T19:13:44.711 DEBUG:boto:Token: None 2022-09-12T19:13:44.712 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Mon, 12 Sep 2022 19:13:44 GMT /jdpiak-33/?cors 2022-09-12T19:13:44.712 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nv57VZiri/fr2mpDvYWUwBztbBA= 2022-09-12T19:13:44.713 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nv57VZiri/fr2mpDvYWUwBztbBA=', 'Content-Length': '218'} 2022-09-12T19:13:44.718 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ddb56bdaeca5b235-00631f84e8-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.719 DEBUG:boto:path=/jdpiak-33/ 2022-09-12T19:13:44.719 DEBUG:boto:auth_path=/jdpiak-33/ 2022-09-12T19:13:44.720 DEBUG:boto:path=/jdpiak-33/?cors 2022-09-12T19:13:44.720 DEBUG:boto:auth_path=/jdpiak-33/?cors 2022-09-12T19:13:44.721 DEBUG:boto:Method: GET 2022-09-12T19:13:44.721 DEBUG:boto:Path: /jdpiak-33/?cors 2022-09-12T19:13:44.722 DEBUG:boto:Data: 2022-09-12T19:13:44.722 DEBUG:boto:Headers: {} 2022-09-12T19:13:44.723 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:44.723 DEBUG:boto:Port: 8000 2022-09-12T19:13:44.723 DEBUG:boto:Params: {} 2022-09-12T19:13:44.724 DEBUG:boto:Token: None 2022-09-12T19:13:44.725 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:44 GMT /jdpiak-33/?cors 2022-09-12T19:13:44.725 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R0KNiE9LLJKaZ/KdrXotzu3iUA8= 2022-09-12T19:13:44.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R0KNiE9LLJKaZ/KdrXotzu3iUA8=', 'Content-Length': '0'} 2022-09-12T19:13:44.727 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000004a4263957027de2-00631f84e8-10f6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Mon, 12 Sep 2022 19:13:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:44.728 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2022-09-12T19:13:44.729 DEBUG:boto:path=/jdpiak-34/ 2022-09-12T19:13:44.729 DEBUG:boto:auth_path=/jdpiak-34/ 2022-09-12T19:13:44.730 DEBUG:boto:path=/jdpiak-34/?cors 2022-09-12T19:13:44.730 DEBUG:boto:auth_path=/jdpiak-34/?cors 2022-09-12T19:13:44.731 DEBUG:boto:Method: PUT 2022-09-12T19:13:44.731 DEBUG:boto:Path: /jdpiak-34/?cors 2022-09-12T19:13:44.732 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2022-09-12T19:13:44.732 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2022-09-12T19:13:44.733 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:44.733 DEBUG:boto:Port: 8000 2022-09-12T19:13:44.734 DEBUG:boto:Params: {} 2022-09-12T19:13:44.734 DEBUG:boto:Token: None 2022-09-12T19:13:44.735 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Mon, 12 Sep 2022 19:13:44 GMT /jdpiak-34/?cors 2022-09-12T19:13:44.735 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HLdaAoFmETvHd+QDFIBaM3/8/UU= 2022-09-12T19:13:44.736 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:44 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HLdaAoFmETvHd+QDFIBaM3/8/UU=', 'Content-Length': '218'} 2022-09-12T19:13:45.746 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006df707233210b227-00631f84e8-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:45.747 DEBUG:boto:path=/jdpiak-34/ 2022-09-12T19:13:45.748 DEBUG:boto:auth_path=/jdpiak-34/ 2022-09-12T19:13:45.748 DEBUG:boto:path=/jdpiak-34/?cors 2022-09-12T19:13:45.749 DEBUG:boto:auth_path=/jdpiak-34/?cors 2022-09-12T19:13:45.749 DEBUG:boto:Method: GET 2022-09-12T19:13:45.750 DEBUG:boto:Path: /jdpiak-34/?cors 2022-09-12T19:13:45.750 DEBUG:boto:Data: 2022-09-12T19:13:45.751 DEBUG:boto:Headers: {} 2022-09-12T19:13:45.752 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:45.753 DEBUG:boto:Port: 8000 2022-09-12T19:13:45.754 DEBUG:boto:Params: {} 2022-09-12T19:13:45.755 DEBUG:boto:Token: None 2022-09-12T19:13:45.756 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:45 GMT /jdpiak-34/?cors 2022-09-12T19:13:45.757 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oQ/3OltJSHN3joPwdoXRwXRtB7A= 2022-09-12T19:13:45.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oQ/3OltJSHN3joPwdoXRwXRtB7A=', 'Content-Length': '0'} 2022-09-12T19:13:45.760 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006c5f9b954041ef79-00631f84e9-10e4-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Mon, 12 Sep 2022 19:13:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:45.802 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2022-09-12T19:13:45.803 DEBUG:boto:path=/jdpiak-35/ 2022-09-12T19:13:45.804 DEBUG:boto:auth_path=/jdpiak-35/ 2022-09-12T19:13:45.804 DEBUG:boto:path=/jdpiak-35/?cors 2022-09-12T19:13:45.805 DEBUG:boto:auth_path=/jdpiak-35/?cors 2022-09-12T19:13:45.805 DEBUG:boto:Method: PUT 2022-09-12T19:13:45.806 DEBUG:boto:Path: /jdpiak-35/?cors 2022-09-12T19:13:45.806 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2022-09-12T19:13:45.807 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2022-09-12T19:13:45.807 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:45.808 DEBUG:boto:Port: 8001 2022-09-12T19:13:45.808 DEBUG:boto:Params: {} 2022-09-12T19:13:45.809 DEBUG:boto:Token: None 2022-09-12T19:13:45.810 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Mon, 12 Sep 2022 19:13:45 GMT /jdpiak-35/?cors 2022-09-12T19:13:45.810 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BkY4sbiETClS1I005bqwTRvhhdY= 2022-09-12T19:13:45.811 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:45 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BkY4sbiETClS1I005bqwTRvhhdY=', 'Content-Length': '218'} 2022-09-12T19:13:46.824 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d78d47c79a9543c1-00631f84e9-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:46.825 DEBUG:boto:path=/jdpiak-35/ 2022-09-12T19:13:46.825 DEBUG:boto:auth_path=/jdpiak-35/ 2022-09-12T19:13:46.826 DEBUG:boto:path=/jdpiak-35/?cors 2022-09-12T19:13:46.826 DEBUG:boto:auth_path=/jdpiak-35/?cors 2022-09-12T19:13:46.827 DEBUG:boto:Method: GET 2022-09-12T19:13:46.827 DEBUG:boto:Path: /jdpiak-35/?cors 2022-09-12T19:13:46.828 DEBUG:boto:Data: 2022-09-12T19:13:46.828 DEBUG:boto:Headers: {} 2022-09-12T19:13:46.829 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:46.829 DEBUG:boto:Port: 8001 2022-09-12T19:13:46.830 DEBUG:boto:Params: {} 2022-09-12T19:13:46.830 DEBUG:boto:Token: None 2022-09-12T19:13:46.831 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:13:46 GMT /jdpiak-35/?cors 2022-09-12T19:13:46.831 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MFST8FA7S2afcwc9k8WUGuUzSt0= 2022-09-12T19:13:46.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MFST8FA7S2afcwc9k8WUGuUzSt0=', 'Content-Length': '0'} 2022-09-12T19:13:46.833 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005f031e110f4b0d9d-00631f84ea-10f2-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Mon, 12 Sep 2022 19:13:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:46.834 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2022-09-12T19:13:46.835 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2022-09-12T19:13:46.836 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-36 2022-09-12T19:13:46.836 DEBUG:boto:path=/jdpiak-36/ 2022-09-12T19:13:46.837 DEBUG:boto:auth_path=/jdpiak-36/ 2022-09-12T19:13:46.837 DEBUG:boto:Method: PUT 2022-09-12T19:13:46.838 DEBUG:boto:Path: /jdpiak-36/ 2022-09-12T19:13:46.838 DEBUG:boto:Data: 2022-09-12T19:13:46.839 DEBUG:boto:Headers: {} 2022-09-12T19:13:46.839 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:13:46.840 DEBUG:boto:Port: 8000 2022-09-12T19:13:46.840 DEBUG:boto:Params: {} 2022-09-12T19:13:46.841 DEBUG:boto:Token: None 2022-09-12T19:13:46.841 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:13:46 GMT /jdpiak-36/ 2022-09-12T19:13:46.842 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OSN4CRe0zmU9uyPDLRqsvO5+VPo= 2022-09-12T19:13:46.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OSN4CRe0zmU9uyPDLRqsvO5+VPo=', 'Content-Length': '0'} 2022-09-12T19:13:46.883 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000db44755b69db3059-00631f84ea-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:46.884 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jdpiak-37 2022-09-12T19:13:46.885 DEBUG:boto:path=/jdpiak-37/ 2022-09-12T19:13:46.885 DEBUG:boto:auth_path=/jdpiak-37/ 2022-09-12T19:13:46.886 DEBUG:boto:Method: PUT 2022-09-12T19:13:46.886 DEBUG:boto:Path: /jdpiak-37/ 2022-09-12T19:13:46.887 DEBUG:boto:Data: 2022-09-12T19:13:46.887 DEBUG:boto:Headers: {} 2022-09-12T19:13:46.888 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:13:46.888 DEBUG:boto:Port: 8000 2022-09-12T19:13:46.889 DEBUG:boto:Params: {} 2022-09-12T19:13:46.889 DEBUG:boto:Token: None 2022-09-12T19:13:46.890 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:13:46 GMT /jdpiak-37/ 2022-09-12T19:13:46.890 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Joyh2/yFxcj9LqVcQ0zMHe9NfOo= 2022-09-12T19:13:46.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Joyh2/yFxcj9LqVcQ0zMHe9NfOo=', 'Content-Length': '0'} 2022-09-12T19:13:46.932 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000645ac19773513f4-00631f84ea-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:46.933 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jdpiak-38 2022-09-12T19:13:46.934 DEBUG:boto:path=/jdpiak-38/ 2022-09-12T19:13:46.934 DEBUG:boto:auth_path=/jdpiak-38/ 2022-09-12T19:13:46.935 DEBUG:boto:Method: PUT 2022-09-12T19:13:46.935 DEBUG:boto:Path: /jdpiak-38/ 2022-09-12T19:13:46.936 DEBUG:boto:Data: 2022-09-12T19:13:46.936 DEBUG:boto:Headers: {} 2022-09-12T19:13:46.937 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:13:46.937 DEBUG:boto:Port: 8001 2022-09-12T19:13:46.938 DEBUG:boto:Params: {} 2022-09-12T19:13:46.938 DEBUG:boto:Token: None 2022-09-12T19:13:46.939 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:13:46 GMT /jdpiak-38/ 2022-09-12T19:13:46.939 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DoT4CVdzbz4ClfJWn966beVWeE4= 2022-09-12T19:13:46.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:13:46 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DoT4CVdzbz4ClfJWn966beVWeE4=', 'Content-Length': '0'} 2022-09-12T19:13:46.964 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000aee0a707826bdf3c-00631f84ea-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:13:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:13:46.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:13:46.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:13:46.966 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:13:47.018 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:13:47.018 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:13:47.053 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=c0f7a512-c940-4d1f-8adf-3484630f876b 2022-09-12T19:13:47.069 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:13:47.070 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:13:47.070 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:13:47.070 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:47:27.976965Z" 2022-09-12T19:13:47.071 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:13:47.071 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:13:47.071 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:13:47.071 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:13:47.072 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:13:47.072 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:13:47.072 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663010026.953073_323.1", 2022-09-12T19:13:47.072 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:13:46.953073Z" 2022-09-12T19:13:47.073 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:13:47.073 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:13:47.073 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:13:47.074 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:49:40.904048Z" 2022-09-12T19:13:47.074 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:13:47.074 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:13:47.075 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010026.953073_323.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:13:47.076 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T19:13:47.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:13:47.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:13:47.078 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:13:47.145 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:13:47.146 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:13:47.205 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:47.202+0000 7fa11cdd4600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:13:47.206 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:47.203+0000 7fa11cdd4600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:13:47.224 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:13:47.224 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:13:47.224 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:13:47.225 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:13:47.225 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:13:47.226 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:13:47.226 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:13:47.227 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:13:47.227 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:13:47.227 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:13:47.227 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:13:47.228 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:13:47.228 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:13:47.228 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:13:47.228 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:13:47.229 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:13:47.229 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:13:47.229 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:13:47.230 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:13:47.230 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:13:47.230 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:13:47.230 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:13:47.231 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:13:47.231 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:13:47.231 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:13:47.231 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:13:47.232 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:13:47.232 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:13:47.233 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:13:47.233 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:13:47.233 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:13:47.233 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:13:47.234 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:13:47.234 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:13:47.234 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:13:47.235 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:13:47.235 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:13:47.235 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010024.183174_272.1", 2022-09-12T19:13:47.235 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:13:47.236 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:13:47.236 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:13:47.236 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.183174Z", 2022-09-12T19:13:47.237 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:13:47.237 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:13:47.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:13:47.238 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:13:47.239 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:13:47.239 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:13:47.239 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:13:47.240 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:13:47.240 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:13:47.240 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:13:47.241 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:13:47.241 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:13:47.242 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:13:47.242 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:13:47.242 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:13:47.243 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:13:47.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:13:47.243 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:13:47.243 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:13:47.244 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:13:47.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:13:47.244 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:13:47Z" 2022-09-12T19:13:47.244 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:13:47.246 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010024.183174_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:13:44.183174Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:13:47Z" } 2022-09-12T19:13:47.247 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010024.183174_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:13:44.183174Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:13:47.248 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010026.953073_323.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:13:47.248 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010024.183174_272.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:13:47.249 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010026.953073_323.1 target=1_1663010024.183174_272.1 2022-09-12T19:13:52.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:13:52.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:13:52.253 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:13:52.307 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:13:52.307 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:13:52.341 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:52.338+0000 7f3758715600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:13:52.342 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:13:52.339+0000 7f3758715600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:13:52.355 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:13:52.355 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:13:52.356 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:13:52.356 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:13:52.356 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:13:52.356 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:13:52.357 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:13:52.357 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:13:52.357 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:13:52.357 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:13:52.358 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:13:52.358 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:13:52.358 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:13:52.358 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:13:52.359 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:13:52.359 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:13:52.359 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:13:52.359 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:13:52.360 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:13:52.360 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:13:52.360 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:13:52.360 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:13:52.361 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:13:52.361 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:13:52.361 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:13:52.361 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:13:52.362 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:13:52.362 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:13:52.362 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:13:52.362 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:13:52.363 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:13:52.363 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:13:52.363 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:13:52.363 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:13:52.364 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:13:52.364 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:13:52.364 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:13:52.364 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010026.953073_323.1", 2022-09-12T19:13:52.365 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:13:52.365 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:13:52.365 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:13:52.365 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.953073Z", 2022-09-12T19:13:52.366 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:13:52.366 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:13:52.366 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:13:52.367 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:13:52.367 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:13:52.368 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:13:52.368 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:13:52.369 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:13:52.369 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:13:52.370 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:13:52.370 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:13:52.371 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:13:52.371 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:13:52.372 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:13:52.372 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:13:52.373 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:13:52.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:13:52.374 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:13:52.374 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:13:52.375 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:13:52.375 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:13:52.376 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:13:52Z" 2022-09-12T19:13:52.376 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:13:52.378 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010026.953073_323.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:13:46.953073Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:13:52Z" } 2022-09-12T19:13:52.379 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010026.953073_323.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:13:46.953073Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:13:52.380 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010026.953073_323.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:13:52.380 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010026.953073_323.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:13:52.381 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T19:13:52.381 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T19:13:52.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:13:52.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:13:52.383 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:13:52.443 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:13:52.443 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:13:52.499 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:52.496+0000 7f9867249600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:13:52.500 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:52.497+0000 7f9867249600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:13:52.515 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:13:52.516 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:13:52.516 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:13:52.516 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:13:52.517 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:13:52.517 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:13:52.517 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:13:52.518 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:13:52.518 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:13:52.518 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:13:52.518 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:13:52.519 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:13:52.519 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:13:52.519 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:13:52.519 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:13:52.520 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:13:52.520 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:13:52.520 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:13:52.520 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:13:52.521 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:13:52.521 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:13:52.521 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:13:52.521 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:13:52.522 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:13:52.522 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:13:52.522 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:13:52.522 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:13:52.523 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:13:52.523 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:13:52.523 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:13:52.523 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:13:52.524 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:13:52.524 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:13:52.524 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:13:52.524 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:13:52.525 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:13:52.525 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:13:52.525 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663010024.295442_276.1", 2022-09-12T19:13:52.525 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:13:52.527 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:13:52.527 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:13:52.527 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.295442Z", 2022-09-12T19:13:52.528 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:13:52.528 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:13:52.528 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:13:52.529 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:13:52.529 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:13:52.529 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:13:52.529 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:13:52.530 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:13:52.530 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:13:52.531 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:13:52.531 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:13:52.532 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:13:52.532 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:13:52.533 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:13:52.533 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:13:52.534 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:13:52.534 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:13:52.534 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:13:52.535 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:13:52.535 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:13:52.536 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:13:52.536 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:13:52Z" 2022-09-12T19:13:52.537 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:13:52.538 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010024.295442_276.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:13:44.295442Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:13:52Z" } 2022-09-12T19:13:52.539 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010024.295442_276.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:13:44.295442Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:13:52.539 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010026.953073_323.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:13:52.540 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010024.295442_276.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:13:52.540 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1663010026.953073_323.1 target=1_1663010024.295442_276.1 2022-09-12T19:13:57.542 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:13:57.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:13:57.544 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:13:57.601 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:13:57.602 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:13:57.633 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:57.630+0000 7efbff436600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:13:57.634 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:13:57.631+0000 7efbff436600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:13:57.671 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:13:57.671 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:13:57.671 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:13:57.672 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:13:57.672 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:13:57.672 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:13:57.672 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:13:57.673 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:13:57.673 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:13:57.673 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:13:57.673 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:13:57.674 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:13:57.674 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:13:57.674 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:13:57.674 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:13:57.675 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:13:57.675 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:13:57.675 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:13:57.676 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:13:57.676 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:13:57.676 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:13:57.677 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:13:57.677 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:13:57.678 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:13:57.678 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:13:57.679 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:13:57.679 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:13:57.680 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:13:57.680 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:13:57.681 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:13:57.681 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:13:57.682 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:13:57.682 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:13:57.682 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:13:57.683 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:13:57.683 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:13:57.684 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:13:57.685 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663010024.183174_272.1", 2022-09-12T19:13:57.686 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:13:57.687 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:13:57.687 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:13:57.688 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.183174Z", 2022-09-12T19:13:57.688 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:13:57.689 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:13:57.689 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:13:57.689 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:13:57.690 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:13:57.690 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:13:57.691 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:13:57.691 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:13:57.692 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:13:57.692 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:13:57.692 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:13:57.692 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:13:57.693 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:13:57.693 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:13:57.693 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:13:57.693 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:13:57.694 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:13:57.694 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:13:57.694 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:13:57.695 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:13:57.695 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:13:57.696 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:13:57Z" 2022-09-12T19:13:57.696 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:13:57.697 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010024.183174_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:13:44.183174Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:13:57Z" } 2022-09-12T19:13:57.698 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010024.183174_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:13:44.183174Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:13:57.698 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010026.953073_323.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:13:57.699 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010024.183174_272.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:13:57.699 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1663010026.953073_323.1 target=1_1663010024.183174_272.1 2022-09-12T19:14:02.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:14:02.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:14:02.702 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:14:02.762 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:14:02.762 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:14:02.804 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:14:02.801+0000 7fa201c93600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:14:02.804 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:14:02.802+0000 7fa201c93600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:14:02.819 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:14:02.819 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:14:02.820 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:14:02.820 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:14:02.820 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:14:02.820 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:14:02.821 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:14:02.821 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:02.821 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:14:02.822 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:14:02.822 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:14:02.822 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:14:02.822 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:14:02.823 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:14:02.823 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:14:02.823 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:14:02.823 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:14:02.823 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:14:02.824 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:14:02.824 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:14:02.824 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:02.824 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:14:02.825 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:14:02.825 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:14:02.825 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:14:02.826 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:14:02.826 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:14:02.826 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:14:02.826 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:14:02.827 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:14:02.827 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:14:02.827 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:14:02.827 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:02.828 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:14:02.828 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:14:02.828 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:14:02.829 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:14:02.829 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663010024.183174_272.1", 2022-09-12T19:14:02.829 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:14:02.830 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:14:02.830 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:14:02.830 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.183174Z", 2022-09-12T19:14:02.830 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:14:02.831 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:14:02.831 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:02.831 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:14:02.831 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:14:02.832 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:14:02.832 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:14:02.832 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:14:02.832 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:14:02.833 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:14:02.833 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:14:02.833 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:14:02.833 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:14:02.834 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:14:02.834 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:14:02.834 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:14:02.834 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:02.835 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:14:02.835 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:14:02.836 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:14:02.836 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:02.836 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:14:02Z" 2022-09-12T19:14:02.836 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:14:02.838 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010024.183174_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:13:44.183174Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:14:02Z" } 2022-09-12T19:14:02.838 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010024.183174_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:13:44.183174Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:14:02.839 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010026.953073_323.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:14:02.839 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010024.183174_272.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:14:02.840 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1663010026.953073_323.1 target=1_1663010024.183174_272.1 2022-09-12T19:14:07.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:14:07.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:14:07.843 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:14:07.905 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:14:07.906 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:14:07.936 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:14:07.933+0000 7fa823bf4600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:14:07.937 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:14:07.934+0000 7fa823bf4600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:14:07.950 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:14:07.950 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:14:07.951 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:14:07.951 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:14:07.951 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:14:07.951 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:14:07.952 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:14:07.952 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:07.952 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:14:07.953 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:14:07.953 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:14:07.953 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:14:07.953 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:14:07.954 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:14:07.954 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:14:07.954 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:14:07.954 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:14:07.955 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:14:07.955 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:14:07.955 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:14:07.955 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:07.956 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:14:07.956 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:14:07.958 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:14:07.958 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:14:07.958 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:14:07.959 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:14:07.959 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:14:07.959 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:14:07.959 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:14:07.960 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:14:07.960 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:14:07.960 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:07.960 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:14:07.961 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:14:07.961 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:14:07.961 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:14:07.962 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663010024.183174_272.1", 2022-09-12T19:14:07.962 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:14:07.962 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:14:07.962 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:14:07.963 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.183174Z", 2022-09-12T19:14:07.963 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:14:07.963 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:14:07.963 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:07.964 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:14:07.964 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:14:07.964 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:14:07.964 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:14:07.965 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:14:07.965 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:14:07.965 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:14:07.965 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:14:07.966 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:14:07.966 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:14:07.966 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:14:07.967 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:14:07.967 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:14:07.967 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:07.967 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:14:07.968 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:14:07.968 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:14:07.968 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:07.968 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:14:07Z" 2022-09-12T19:14:07.969 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:14:07.970 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010024.183174_272.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:13:44.183174Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:14:07Z" } 2022-09-12T19:14:07.971 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010024.183174_272.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:13:44.183174Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:14:07.972 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010026.953073_323.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:14:07.973 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010024.183174_272.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:14:07.974 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1663010026.953073_323.1 target=1_1663010024.183174_272.1 2022-09-12T19:14:12.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:14:12.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:14:12.978 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:14:13.037 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:14:13.037 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:14:13.070 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:14:13.067+0000 7f45ea299600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:14:13.071 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:14:13.068+0000 7f45ea299600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:14:13.084 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:14:13.084 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:14:13.084 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:14:13.085 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:14:13.085 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:14:13.085 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:14:13.085 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:14:13.086 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:13.086 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:14:13.086 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:14:13.086 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:14:13.087 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:14:13.087 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:14:13.087 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:14:13.087 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:14:13.088 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:14:13.088 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:14:13.089 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:14:13.089 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:14:13.089 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:14:13.089 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:13.090 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:14:13.090 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:14:13.090 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:14:13.090 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:14:13.091 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:14:13.091 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:14:13.091 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:14:13.092 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:14:13.092 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:14:13.093 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:14:13.093 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:14:13.094 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:13.094 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:14:13.095 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:14:13.095 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:14:13.095 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:14:13.096 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663010026.953073_323.1", 2022-09-12T19:14:13.096 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:14:13.097 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:14:13.097 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:14:13.097 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.953073Z", 2022-09-12T19:14:13.098 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:14:13.098 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:14:13.099 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:13.099 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:14:13.099 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:14:13.100 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:14:13.100 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:14:13.101 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:14:13.101 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:14:13.101 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:14:13.102 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:14:13.102 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:14:13.103 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:14:13.103 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:14:13.103 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:14:13.104 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:14:13.104 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:13.105 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:14:13.105 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:14:13.105 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:14:13.106 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:13.106 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:14:13Z" 2022-09-12T19:14:13.107 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:14:13.108 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010026.953073_323.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:13:46.953073Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:14:13Z" } 2022-09-12T19:14:13.109 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010026.953073_323.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:13:46.953073Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:14:13.110 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010026.953073_323.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:14:13.111 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010026.953073_323.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:14:13.112 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T19:14:13.113 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T19:14:13.114 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:14:13.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:14:13.115 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:14:13.178 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:14:13.179 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:14:13.223 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:14:13.220+0000 7f4d0d331600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:14:13.224 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:14:13.221+0000 7f4d0d331600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:14:13.240 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:14:13.241 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:14:13.241 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:14:13.242 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:14:13.242 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:14:13.242 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:14:13.242 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:14:13.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:13.243 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:14:13.243 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:13.243 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:14:13.244 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:14:13.244 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:14:13.244 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:14:13.244 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:14:13.245 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:14:13.245 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:14:13.246 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:14:13.246 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:14:13.246 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:13.246 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:13.247 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:13.247 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:14:13.247 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:14:13.248 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:14:13.248 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:14:13.248 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:14:13.249 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:14:13.249 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:14:13.249 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:14:13.249 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:14:13.249 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:13.250 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:13.250 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:13.250 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:14:13.250 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:14:13.251 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:14:13.251 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010024.121725_268.1", 2022-09-12T19:14:13.252 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:14:13.252 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:14:13.252 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:14:13.252 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.121725Z", 2022-09-12T19:14:13.253 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:14:13.253 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:13.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:13.253 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:13.253 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:14:13.254 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:14:13.254 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:14:13.254 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:14:13.254 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:14:13.255 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:14:13.255 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:14:13.255 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:14:13.255 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:14:13.256 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:13.256 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:13.256 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:14:13.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:13.257 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:14:13.257 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:14:13.257 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:14:13.257 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:13.258 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:14:13Z" 2022-09-12T19:14:13.258 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:14:13.259 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010024.121725_268.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:13:44.121725Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:14:13Z" } 2022-09-12T19:14:13.260 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010024.121725_268.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:13:44.121725Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:14:13.260 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010026.953073_323.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:14:13.261 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010024.121725_268.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:14:13.261 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1663010026.953073_323.1 target=1_1663010024.121725_268.1 2022-09-12T19:14:18.263 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:14:18.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:14:18.265 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:14:18.321 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:14:18.321 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:14:18.356 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:14:18.354+0000 7fe4d856f600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:14:18.357 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:14:18.355+0000 7fe4d856f600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:14:18.379 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:14:18.379 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:14:18.379 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:14:18.380 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:14:18.380 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:14:18.380 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:14:18.380 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:14:18.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:18.381 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:14:18.381 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:18.382 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:14:18.382 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:14:18.382 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:14:18.383 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:14:18.383 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:14:18.383 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:14:18.384 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:14:18.384 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:14:18.384 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:14:18.385 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:18.385 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:18.385 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:18.385 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:14:18.386 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:14:18.386 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:14:18.386 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:14:18.387 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:14:18.387 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:14:18.387 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:14:18.388 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:14:18.388 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:14:18.388 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:18.388 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:18.389 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:18.389 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:14:18.389 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:14:18.389 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:14:18.390 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010026.953073_323.1", 2022-09-12T19:14:18.390 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:14:18.390 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:14:18.391 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:14:18.391 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.953073Z", 2022-09-12T19:14:18.392 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:14:18.392 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:18.392 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:18.392 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:18.393 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:14:18.393 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:14:18.393 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:14:18.393 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:14:18.394 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:14:18.394 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:14:18.394 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:14:18.394 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:14:18.395 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:14:18.395 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:18.396 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:18.396 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:14:18.396 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:18.396 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:14:18.397 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:14:18.397 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:14:18.397 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:18.398 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:14:18Z" 2022-09-12T19:14:18.398 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:14:18.400 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010026.953073_323.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:13:46.953073Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:14:18Z" } 2022-09-12T19:14:18.400 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010026.953073_323.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:13:46.953073Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:14:18.401 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010026.953073_323.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:14:18.402 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010026.953073_323.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:14:18.402 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T19:14:18.403 DEBUG:boto:path=/jdpiak-36/ 2022-09-12T19:14:18.404 DEBUG:boto:auth_path=/jdpiak-36/ 2022-09-12T19:14:18.404 DEBUG:boto:Method: HEAD 2022-09-12T19:14:18.405 DEBUG:boto:Path: /jdpiak-36/ 2022-09-12T19:14:18.405 DEBUG:boto:Data: 2022-09-12T19:14:18.406 DEBUG:boto:Headers: {} 2022-09-12T19:14:18.406 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:14:18.407 DEBUG:boto:Port: 8000 2022-09-12T19:14:18.407 DEBUG:boto:Params: {} 2022-09-12T19:14:18.408 DEBUG:boto:Token: None 2022-09-12T19:14:18.409 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:14:18 GMT /jdpiak-36/ 2022-09-12T19:14:18.409 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IE6g0RBoAKUrFRSQRj4LUXW8b8o= 2022-09-12T19:14:18.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IE6g0RBoAKUrFRSQRj4LUXW8b8o=', 'Content-Length': '0'} 2022-09-12T19:14:18.413 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000024ef31b91399c77-00631f850a-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:14:18.414 DEBUG:boto:path=/jdpiak-36/foo 2022-09-12T19:14:18.415 DEBUG:boto:auth_path=/jdpiak-36/foo 2022-09-12T19:14:18.415 DEBUG:boto:Method: PUT 2022-09-12T19:14:18.416 DEBUG:boto:Path: /jdpiak-36/foo 2022-09-12T19:14:18.416 DEBUG:boto:Data: 2022-09-12T19:14:18.417 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:14:18.417 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:14:18.418 DEBUG:boto:Port: 8000 2022-09-12T19:14:18.418 DEBUG:boto:Params: {} 2022-09-12T19:14:18.419 DEBUG:boto:Token: None 2022-09-12T19:14:18.420 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Mon, 12 Sep 2022 19:14:18 GMT /jdpiak-36/foo 2022-09-12T19:14:18.420 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bv6Cpvw1s7mrWhm+YoIWJLJmkK4= 2022-09-12T19:14:18.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bv6Cpvw1s7mrWhm+YoIWJLJmkK4='} 2022-09-12T19:14:18.436 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000adbf2bc80cacbf38-00631f850a-10f6-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:14:18.437 DEBUG:boto:path=/jdpiak-36/ 2022-09-12T19:14:18.437 DEBUG:boto:auth_path=/jdpiak-36/ 2022-09-12T19:14:18.438 DEBUG:boto:Method: DELETE 2022-09-12T19:14:18.439 DEBUG:boto:Path: /jdpiak-36/ 2022-09-12T19:14:18.440 DEBUG:boto:Data: 2022-09-12T19:14:18.440 DEBUG:boto:Headers: {} 2022-09-12T19:14:18.441 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:14:18.442 DEBUG:boto:Port: 8000 2022-09-12T19:14:18.443 DEBUG:boto:Params: {} 2022-09-12T19:14:18.443 DEBUG:boto:Token: None 2022-09-12T19:14:18.444 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 19:14:18 GMT /jdpiak-36/ 2022-09-12T19:14:18.445 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fema/TO9JHhLcxm5nt4xG+agbhU= 2022-09-12T19:14:18.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fema/TO9JHhLcxm5nt4xG+agbhU=', 'Content-Length': '0'} 2022-09-12T19:14:18.455 DEBUG:boto:Response headers: [('Content-Length', '234'), ('x-amz-request-id', 'tx0000034f6e5b0a6e87024-00631f850a-10f6-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:14:18.499 DEBUG:boto:path=/jdpiak-37/ 2022-09-12T19:14:18.500 DEBUG:boto:auth_path=/jdpiak-37/ 2022-09-12T19:14:18.501 DEBUG:boto:Method: HEAD 2022-09-12T19:14:18.502 DEBUG:boto:Path: /jdpiak-37/ 2022-09-12T19:14:18.502 DEBUG:boto:Data: 2022-09-12T19:14:18.503 DEBUG:boto:Headers: {} 2022-09-12T19:14:18.503 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:14:18.504 DEBUG:boto:Port: 8000 2022-09-12T19:14:18.504 DEBUG:boto:Params: {} 2022-09-12T19:14:18.505 DEBUG:boto:Token: None 2022-09-12T19:14:18.506 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:14:18 GMT /jdpiak-37/ 2022-09-12T19:14:18.507 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FcUjQj4nwYfB8MD0oEkWTSbM2E0= 2022-09-12T19:14:18.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FcUjQj4nwYfB8MD0oEkWTSbM2E0=', 'Content-Length': '0'} 2022-09-12T19:14:18.511 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000311a61e6b1ba377-00631f850a-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:14:18.512 DEBUG:boto:path=/jdpiak-37/foo 2022-09-12T19:14:18.513 DEBUG:boto:auth_path=/jdpiak-37/foo 2022-09-12T19:14:18.513 DEBUG:boto:Method: PUT 2022-09-12T19:14:18.514 DEBUG:boto:Path: /jdpiak-37/foo 2022-09-12T19:14:18.514 DEBUG:boto:Data: 2022-09-12T19:14:18.515 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:14:18.515 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:14:18.516 DEBUG:boto:Port: 8000 2022-09-12T19:14:18.516 DEBUG:boto:Params: {} 2022-09-12T19:14:18.517 DEBUG:boto:Token: None 2022-09-12T19:14:18.518 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Mon, 12 Sep 2022 19:14:18 GMT /jdpiak-37/foo 2022-09-12T19:14:18.519 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s+1d+MZme+f5oUCEEH/e2PmHMjI= 2022-09-12T19:14:18.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s+1d+MZme+f5oUCEEH/e2PmHMjI='} 2022-09-12T19:14:18.538 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000006f20249d9208e84-00631f850a-10e4-test-zone2'), ('Date', 'Mon, 12 Sep 2022 19:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:14:18.539 DEBUG:boto:path=/jdpiak-37/ 2022-09-12T19:14:18.539 DEBUG:boto:auth_path=/jdpiak-37/ 2022-09-12T19:14:18.540 DEBUG:boto:Method: DELETE 2022-09-12T19:14:18.540 DEBUG:boto:Path: /jdpiak-37/ 2022-09-12T19:14:18.541 DEBUG:boto:Data: 2022-09-12T19:14:18.541 DEBUG:boto:Headers: {} 2022-09-12T19:14:18.542 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:14:18.542 DEBUG:boto:Port: 8000 2022-09-12T19:14:18.543 DEBUG:boto:Params: {} 2022-09-12T19:14:18.543 DEBUG:boto:Token: None 2022-09-12T19:14:18.544 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 19:14:18 GMT /jdpiak-37/ 2022-09-12T19:14:18.544 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KrF94MorOOD3RSTTx0CH09GFgXo= 2022-09-12T19:14:18.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KrF94MorOOD3RSTTx0CH09GFgXo=', 'Content-Length': '0'} 2022-09-12T19:14:18.551 DEBUG:boto:Response headers: [('Content-Length', '234'), ('x-amz-request-id', 'tx0000040571902aafacad0-00631f850a-10e4-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:14:18.552 DEBUG:boto:path=/jdpiak-38/ 2022-09-12T19:14:18.553 DEBUG:boto:auth_path=/jdpiak-38/ 2022-09-12T19:14:18.554 DEBUG:boto:Method: HEAD 2022-09-12T19:14:18.554 DEBUG:boto:Path: /jdpiak-38/ 2022-09-12T19:14:18.555 DEBUG:boto:Data: 2022-09-12T19:14:18.555 DEBUG:boto:Headers: {} 2022-09-12T19:14:18.556 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:14:18.556 DEBUG:boto:Port: 8001 2022-09-12T19:14:18.557 DEBUG:boto:Params: {} 2022-09-12T19:14:18.557 DEBUG:boto:Token: None 2022-09-12T19:14:18.558 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:14:18 GMT /jdpiak-38/ 2022-09-12T19:14:18.559 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZUI+c1eye+Egoa5br295U0Nvpzw= 2022-09-12T19:14:18.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZUI+c1eye+Egoa5br295U0Nvpzw=', 'Content-Length': '0'} 2022-09-12T19:14:18.562 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9c5279296413a2e-00631f850a-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:14:18.563 DEBUG:boto:path=/jdpiak-38/foo 2022-09-12T19:14:18.563 DEBUG:boto:auth_path=/jdpiak-38/foo 2022-09-12T19:14:18.564 DEBUG:boto:Method: PUT 2022-09-12T19:14:18.564 DEBUG:boto:Path: /jdpiak-38/foo 2022-09-12T19:14:18.565 DEBUG:boto:Data: 2022-09-12T19:14:18.565 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:14:18.566 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:14:18.566 DEBUG:boto:Port: 8001 2022-09-12T19:14:18.567 DEBUG:boto:Params: {} 2022-09-12T19:14:18.568 DEBUG:boto:Token: None 2022-09-12T19:14:18.569 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Mon, 12 Sep 2022 19:14:18 GMT /jdpiak-38/foo 2022-09-12T19:14:18.569 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/AY4XSZeXw7mZltMPqqp8ELd1Gc= 2022-09-12T19:14:18.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/AY4XSZeXw7mZltMPqqp8ELd1Gc='} 2022-09-12T19:14:18.618 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3b3ad13c384bdd5-00631f850a-10f2-test-zone3'), ('Date', 'Mon, 12 Sep 2022 19:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:14:18.620 DEBUG:boto:path=/jdpiak-38/ 2022-09-12T19:14:18.621 DEBUG:boto:auth_path=/jdpiak-38/ 2022-09-12T19:14:18.622 DEBUG:boto:Method: DELETE 2022-09-12T19:14:18.622 DEBUG:boto:Path: /jdpiak-38/ 2022-09-12T19:14:18.623 DEBUG:boto:Data: 2022-09-12T19:14:18.624 DEBUG:boto:Headers: {} 2022-09-12T19:14:18.625 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:14:18.626 DEBUG:boto:Port: 8001 2022-09-12T19:14:18.627 DEBUG:boto:Params: {} 2022-09-12T19:14:18.628 DEBUG:boto:Token: None 2022-09-12T19:14:18.629 DEBUG:boto:StringToSign: DELETE Mon, 12 Sep 2022 19:14:18 GMT /jdpiak-38/ 2022-09-12T19:14:18.630 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P6yLigdA4ZPbZr7jgctL4DcK46s= 2022-09-12T19:14:18.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P6yLigdA4ZPbZr7jgctL4DcK46s=', 'Content-Length': '0'} 2022-09-12T19:14:18.640 DEBUG:boto:Response headers: [('Content-Length', '234'), ('x-amz-request-id', 'tx0000098d894aa82b0a11b-00631f850a-10f2-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:14:18.641 DEBUG:boto:path=/jdpiak-36/ 2022-09-12T19:14:18.642 DEBUG:boto:auth_path=/jdpiak-36/ 2022-09-12T19:14:18.643 DEBUG:boto:Method: HEAD 2022-09-12T19:14:18.643 DEBUG:boto:Path: /jdpiak-36/ 2022-09-12T19:14:18.644 DEBUG:boto:Data: 2022-09-12T19:14:18.645 DEBUG:boto:Headers: {} 2022-09-12T19:14:18.646 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:14:18.647 DEBUG:boto:Port: 8000 2022-09-12T19:14:18.648 DEBUG:boto:Params: {} 2022-09-12T19:14:18.649 DEBUG:boto:Token: None 2022-09-12T19:14:18.650 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:14:18 GMT /jdpiak-36/ 2022-09-12T19:14:18.651 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IE6g0RBoAKUrFRSQRj4LUXW8b8o= 2022-09-12T19:14:18.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IE6g0RBoAKUrFRSQRj4LUXW8b8o=', 'Content-Length': '0'} 2022-09-12T19:14:18.655 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000542234e5d2f6596-00631f850a-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:14:18.656 DEBUG:boto:path=/jdpiak-37/ 2022-09-12T19:14:18.657 DEBUG:boto:auth_path=/jdpiak-37/ 2022-09-12T19:14:18.658 DEBUG:boto:Method: HEAD 2022-09-12T19:14:18.659 DEBUG:boto:Path: /jdpiak-37/ 2022-09-12T19:14:18.660 DEBUG:boto:Data: 2022-09-12T19:14:18.661 DEBUG:boto:Headers: {} 2022-09-12T19:14:18.662 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:14:18.663 DEBUG:boto:Port: 8000 2022-09-12T19:14:18.664 DEBUG:boto:Params: {} 2022-09-12T19:14:18.665 DEBUG:boto:Token: None 2022-09-12T19:14:18.666 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:14:18 GMT /jdpiak-37/ 2022-09-12T19:14:18.667 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FcUjQj4nwYfB8MD0oEkWTSbM2E0= 2022-09-12T19:14:18.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FcUjQj4nwYfB8MD0oEkWTSbM2E0=', 'Content-Length': '0'} 2022-09-12T19:14:18.671 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000070a9e6b6822dd00f-00631f850a-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:14:18.672 DEBUG:boto:path=/jdpiak-38/ 2022-09-12T19:14:18.672 DEBUG:boto:auth_path=/jdpiak-38/ 2022-09-12T19:14:18.673 DEBUG:boto:Method: HEAD 2022-09-12T19:14:18.673 DEBUG:boto:Path: /jdpiak-38/ 2022-09-12T19:14:18.674 DEBUG:boto:Data: 2022-09-12T19:14:18.675 DEBUG:boto:Headers: {} 2022-09-12T19:14:18.675 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:14:18.676 DEBUG:boto:Port: 8000 2022-09-12T19:14:18.676 DEBUG:boto:Params: {} 2022-09-12T19:14:18.677 DEBUG:boto:Token: None 2022-09-12T19:14:18.677 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:14:18 GMT /jdpiak-38/ 2022-09-12T19:14:18.678 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZUI+c1eye+Egoa5br295U0Nvpzw= 2022-09-12T19:14:18.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZUI+c1eye+Egoa5br295U0Nvpzw=', 'Content-Length': '0'} 2022-09-12T19:14:18.682 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c903f1629fac8ca6-00631f850a-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:14:18.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... ok 2022-09-12T19:14:18.686 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-39 2022-09-12T19:14:18.687 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:14:18.688 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:14:18.689 DEBUG:boto:Method: PUT 2022-09-12T19:14:18.689 DEBUG:boto:Path: /jdpiak-39/ 2022-09-12T19:14:18.691 DEBUG:boto:Data: 2022-09-12T19:14:18.691 DEBUG:boto:Headers: {} 2022-09-12T19:14:18.692 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:14:18.693 DEBUG:boto:Port: 8000 2022-09-12T19:14:18.694 DEBUG:boto:Params: {} 2022-09-12T19:14:18.695 DEBUG:boto:Token: None 2022-09-12T19:14:18.696 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:14:18 GMT /jdpiak-39/ 2022-09-12T19:14:18.697 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Mtn/hLS0wgIOZ1+4S3Wo+a78cG0= 2022-09-12T19:14:18.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Mtn/hLS0wgIOZ1+4S3Wo+a78cG0=', 'Content-Length': '0'} 2022-09-12T19:14:18.725 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000af486bc2ba166eb4-00631f850a-10f6-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:14:18.726 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jdpiak-40 2022-09-12T19:14:18.727 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:14:18.727 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:14:18.728 DEBUG:boto:Method: PUT 2022-09-12T19:14:18.728 DEBUG:boto:Path: /jdpiak-40/ 2022-09-12T19:14:18.729 DEBUG:boto:Data: 2022-09-12T19:14:18.729 DEBUG:boto:Headers: {} 2022-09-12T19:14:18.730 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:14:18.730 DEBUG:boto:Port: 8000 2022-09-12T19:14:18.731 DEBUG:boto:Params: {} 2022-09-12T19:14:18.731 DEBUG:boto:Token: None 2022-09-12T19:14:18.732 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:14:18 GMT /jdpiak-40/ 2022-09-12T19:14:18.732 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cZrXTBhqAmpDy92iB6r1nQIWxIM= 2022-09-12T19:14:18.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cZrXTBhqAmpDy92iB6r1nQIWxIM=', 'Content-Length': '0'} 2022-09-12T19:14:18.777 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001f9fc01e637d8667-00631f850a-10e4-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:14:18.778 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jdpiak-41 2022-09-12T19:14:18.779 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:14:18.779 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:14:18.780 DEBUG:boto:Method: PUT 2022-09-12T19:14:18.780 DEBUG:boto:Path: /jdpiak-41/ 2022-09-12T19:14:18.781 DEBUG:boto:Data: 2022-09-12T19:14:18.781 DEBUG:boto:Headers: {} 2022-09-12T19:14:18.782 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:14:18.782 DEBUG:boto:Port: 8001 2022-09-12T19:14:18.783 DEBUG:boto:Params: {} 2022-09-12T19:14:18.783 DEBUG:boto:Token: None 2022-09-12T19:14:18.784 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:14:18 GMT /jdpiak-41/ 2022-09-12T19:14:18.785 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ty6ku74ce3Im7psZB5eTwkQjexE= 2022-09-12T19:14:18.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ty6ku74ce3Im7psZB5eTwkQjexE=', 'Content-Length': '0'} 2022-09-12T19:14:18.817 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002f9c382fb3e9405c-00631f850a-10f2-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:14:18.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:14:18.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:14:18.819 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:14:18.870 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:14:18.871 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:14:18.902 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=c0f7a512-c940-4d1f-8adf-3484630f876b 2022-09-12T19:14:18.914 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:14:18.914 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:14:18.914 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:14:18.915 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:47:27.976965Z" 2022-09-12T19:14:18.915 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:18.915 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:14:18.915 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:14:18.916 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:14:18.916 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:18.916 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:14:18.917 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663010058.804299_335.1", 2022-09-12T19:14:18.917 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:14:18.804299Z" 2022-09-12T19:14:18.917 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:18.917 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:14:18.918 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:14:18.918 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T18:49:40.904048Z" 2022-09-12T19:14:18.918 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:14:18.919 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:14:18.920 DEBUG:rgw_multi.tests:master meta markers={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010058.804299_335.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:14:18.920 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T19:14:18.921 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:14:18.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:14:18.922 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:14:18.980 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:14:18.981 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:14:19.033 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:14:19.031+0000 7f961e06d600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:14:19.034 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:14:19.032+0000 7f961e06d600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:14:19.046 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:14:19.046 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:14:19.047 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:14:19.047 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:14:19.047 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:14:19.048 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:14:19.048 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:14:19.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:19.049 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:14:19.049 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:19.049 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:14:19.050 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:14:19.050 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:14:19.050 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:14:19.051 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:14:19.051 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:14:19.051 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:14:19.052 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:14:19.052 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:14:19.052 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:19.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:19.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:19.053 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:14:19.054 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:14:19.054 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:14:19.054 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:14:19.054 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:14:19.055 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:14:19.055 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:14:19.055 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:14:19.055 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:14:19.056 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:19.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:19.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:19.056 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:14:19.057 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:14:19.057 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:14:19.057 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010058.804299_335.1", 2022-09-12T19:14:19.058 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:14:19.058 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:14:19.058 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:14:19.058 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.804299Z", 2022-09-12T19:14:19.059 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:14:19.059 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:19.059 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:19.059 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:19.060 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:14:19.060 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:14:19.060 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:14:19.060 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:14:19.061 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:14:19.061 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:14:19.061 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:14:19.061 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:14:19.062 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:14:19.062 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:19.062 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:19.063 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:14:19.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:19.064 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:14:19.065 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:14:19.065 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:14:19.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:19.066 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:14:19Z" 2022-09-12T19:14:19.066 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:14:19.067 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010058.804299_335.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:14:18.804299Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:14:19Z" } 2022-09-12T19:14:19.068 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010058.804299_335.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:14:18.804299Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:14:19.068 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010058.804299_335.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:14:19.069 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010058.804299_335.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:14:19.069 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T19:14:19.070 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T19:14:19.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:14:19.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:14:19.072 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:14:19.123 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:14:19.124 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:14:19.158 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:14:19.154+0000 7f9cd7193600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:14:19.158 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:14:19.155+0000 7f9cd7193600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:14:19.170 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:14:19.170 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:14:19.171 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:14:19.171 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:14:19.171 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:14:19.172 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:14:19.172 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:14:19.172 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:19.172 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:14:19.173 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:14:19.173 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:14:19.173 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:14:19.173 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:14:19.174 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:14:19.174 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:14:19.174 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:14:19.175 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:14:19.175 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:14:19.175 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:14:19.175 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:14:19.176 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:19.176 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:14:19.176 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:14:19.176 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:14:19.177 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:14:19.177 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:14:19.177 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:14:19.178 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:14:19.178 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:14:19.178 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:14:19.178 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:14:19.179 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:14:19.179 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:19.179 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:14:19.179 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:14:19.180 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:14:19.180 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:14:19.180 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663010058.804299_335.1", 2022-09-12T19:14:19.181 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:14:19.182 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:14:19.182 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:14:19.182 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:14:18.804299Z", 2022-09-12T19:14:19.182 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:14:19.183 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:14:19.183 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:19.183 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:14:19.183 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:14:19.184 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:14:19.184 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:14:19.184 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:14:19.184 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:14:19.185 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:14:19.185 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:14:19.185 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:14:19.185 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:14:19.186 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:14:19.186 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:14:19.186 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:14:19.186 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:19.187 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:14:19.187 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:14:19.187 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:14:19.187 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:14:19.188 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:14:19Z" 2022-09-12T19:14:19.188 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:14:19.189 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010058.804299_335.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:14:18.804299Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:14:19Z" } 2022-09-12T19:14:19.190 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010058.804299_335.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:14:18.804299Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:14:19.190 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010058.804299_335.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:14:19.191 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010058.804299_335.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:14:19.191 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T19:14:19.192 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T19:14:19.192 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:14:19.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:14:19.193 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:14:19.245 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:14:19.245 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:14:19.273 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:14:19.271+0000 7fec7c884600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:14:19.274 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:14:19.272+0000 7fec7c884600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:14:19.288 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:14:19.288 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:14:19.288 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:14:19.288 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:14:19.289 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:14:19.289 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:14:19.289 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:14:19.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:19.290 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:14:19.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:19.290 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:14:19.291 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:14:19.291 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:14:19.291 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:14:19.292 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:14:19.292 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:14:19.292 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:14:19.292 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:14:19.293 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:14:19.293 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:19.293 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:19.293 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:19.294 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:14:19.294 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:14:19.294 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:14:19.294 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:14:19.295 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:14:19.295 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:14:19.295 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:14:19.295 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:14:19.296 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:14:19.296 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:19.297 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:19.297 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:19.297 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:14:19.297 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:14:19.298 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:14:19.298 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010058.804299_335.1", 2022-09-12T19:14:19.298 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:14:19.298 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:14:19.299 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:14:19.299 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.804299Z", 2022-09-12T19:14:19.299 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:14:19.300 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:19.300 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:19.300 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:19.300 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:14:19.301 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:14:19.301 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:14:19.301 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:14:19.301 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:14:19.302 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:14:19.302 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:14:19.302 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:14:19.302 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:14:19.303 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:19.303 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:19.303 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:14:19.303 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:19.304 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:14:19.304 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:14:19.304 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:14:19.304 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:19.305 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:14:19Z" 2022-09-12T19:14:19.305 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:14:19.308 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "c0f7a512-c940-4d1f-8adf-3484630f876b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010058.804299_335.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:14:18.804299Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:14:19Z" } 2022-09-12T19:14:19.309 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010058.804299_335.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:14:18.804299Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:14:19.309 DEBUG:rgw_multi.tests:log_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010058.804299_335.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:14:19.310 DEBUG:rgw_multi.tests:sync_status={0: '1_1663008447.976965_53.1', 1: '', 2: '1_1663010058.804299_335.1', 3: '1_1663008580.904048_86.1'} 2022-09-12T19:14:19.310 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T19:14:19.311 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2022-09-12T19:14:19.312 INFO:teuthology.orchestra.run:waiting for 300 2022-09-12T19:14:19.313 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:14:19.310+0000 7faaa5ef5700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 110528) UID: 0 2022-09-12T19:14:19.313 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:14:19.310+0000 7faaaa160600 -1 shutting down 2022-09-12T19:14:20.924 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:14:20.922+0000 7fa8c0897700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-16) 2022-09-12T19:14:21.046 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:14:21.044+0000 7fa8c0897700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-16) 2022-09-12T19:14:21.109 INFO:tasks.rgw.c1.client.0.smithi035.stdout:2022-09-12T19:14:21.107+0000 7f29e274c700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-5) 2022-09-12T19:14:21.125 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:14:21.123+0000 7fa8c0897700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-5) 2022-09-12T19:14:21.229 INFO:tasks.rgw.c1.client.0.smithi035.stdout:2022-09-12T19:14:21.227+0000 7f29e274c700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-5) 2022-09-12T19:14:21.313 INFO:tasks.rgw.c1.client.0.smithi035.stdout:2022-09-12T19:14:21.311+0000 7f29e274c700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-5) 2022-09-12T19:14:24.339 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2022-09-12T19:14:25.414 INFO:tasks.rgw.c1.client.1:Stopped 2022-09-12T19:14:25.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:14:25.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:14:25.416 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T19:14:25.472 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:14:25.472 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:14:25.495 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:14:25.492+0000 7f0c41457600 0 NOTICE: overriding master zone: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47 2022-09-12T19:14:25.495 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:14:25.492+0000 7f0c41457600 0 failed reading obj info from .rgw.root:zone_info.823fb3a5-4d8a-49dc-ae64-e05804bbea94: (2) No such file or directory 2022-09-12T19:14:25.496 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:14:25.492+0000 7f0c41457600 0 WARNING: could not read zone params for zone id=823fb3a5-4d8a-49dc-ae64-e05804bbea94 name=test-zone3 2022-09-12T19:14:25.497 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:14:25.493+0000 7f0c41457600 0 failed reading obj info from .rgw.root:zone_info.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47: (2) No such file or directory 2022-09-12T19:14:25.497 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:14:25.493+0000 7f0c41457600 0 WARNING: could not read zone params for zone id=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47 name=test-zone1 2022-09-12T19:14:25.501 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:14:25.501 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:14:25.501 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone2", 2022-09-12T19:14:25.502 INFO:teuthology.orchestra.run.smithi138.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-09-12T19:14:25.502 INFO:teuthology.orchestra.run.smithi138.stdout: "control_pool": "test-zone2.rgw.control", 2022-09-12T19:14:25.502 INFO:teuthology.orchestra.run.smithi138.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-09-12T19:14:25.503 INFO:teuthology.orchestra.run.smithi138.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-09-12T19:14:25.503 INFO:teuthology.orchestra.run.smithi138.stdout: "log_pool": "test-zone2.rgw.log", 2022-09-12T19:14:25.503 INFO:teuthology.orchestra.run.smithi138.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-09-12T19:14:25.503 INFO:teuthology.orchestra.run.smithi138.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-09-12T19:14:25.504 INFO:teuthology.orchestra.run.smithi138.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-09-12T19:14:25.504 INFO:teuthology.orchestra.run.smithi138.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-09-12T19:14:25.504 INFO:teuthology.orchestra.run.smithi138.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-09-12T19:14:25.504 INFO:teuthology.orchestra.run.smithi138.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-09-12T19:14:25.505 INFO:teuthology.orchestra.run.smithi138.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-09-12T19:14:25.505 INFO:teuthology.orchestra.run.smithi138.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-09-12T19:14:25.505 INFO:teuthology.orchestra.run.smithi138.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-09-12T19:14:25.505 INFO:teuthology.orchestra.run.smithi138.stdout: "system_key": { 2022-09-12T19:14:25.506 INFO:teuthology.orchestra.run.smithi138.stdout: "access_key": "RDIEFLSOLOEHUH1F", 2022-09-12T19:14:25.506 INFO:teuthology.orchestra.run.smithi138.stdout: "secret_key": "LDCKRd1RecZNgErNcVDnUigDhC7QggCQ" 2022-09-12T19:14:25.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:25.506 INFO:teuthology.orchestra.run.smithi138.stdout: "placement_pools": [ 2022-09-12T19:14:25.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:25.507 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "default-placement", 2022-09-12T19:14:25.507 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:14:25.508 INFO:teuthology.orchestra.run.smithi138.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-09-12T19:14:25.508 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_classes": { 2022-09-12T19:14:25.508 INFO:teuthology.orchestra.run.smithi138.stdout: "STANDARD": { 2022-09-12T19:14:25.509 INFO:teuthology.orchestra.run.smithi138.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2022-09-12T19:14:25.509 INFO:teuthology.orchestra.run.smithi138.stdout: "compression_type": "random" 2022-09-12T19:14:25.509 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:25.509 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:25.510 INFO:teuthology.orchestra.run.smithi138.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-09-12T19:14:25.510 INFO:teuthology.orchestra.run.smithi138.stdout: "index_type": 0 2022-09-12T19:14:25.510 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:25.512 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:25.512 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:14:25.513 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:14:25.513 INFO:teuthology.orchestra.run.smithi138.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-09-12T19:14:25.513 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:14:25.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:14:25.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:14:25.516 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T19:14:25.567 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:14:25.568 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:14:25.616 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:14:25.613+0000 7f8f48d00600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:14:25.617 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:14:25.614+0000 7f8f48d00600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:14:25.629 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:14:25.630 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e", 2022-09-12T19:14:25.630 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2022-09-12T19:14:25.631 INFO:teuthology.orchestra.run.smithi138.stdout: "predecessor_uuid": "c0f7a512-c940-4d1f-8adf-3484630f876b", 2022-09-12T19:14:25.631 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": [ 2022-09-12T19:14:25.631 INFO:teuthology.orchestra.run.smithi138.stdout: "1_1663008447.976965_53.1", 2022-09-12T19:14:25.631 INFO:teuthology.orchestra.run.smithi138.stdout: "", 2022-09-12T19:14:25.632 INFO:teuthology.orchestra.run.smithi138.stdout: "1_1663010058.804299_335.1", 2022-09-12T19:14:25.632 INFO:teuthology.orchestra.run.smithi138.stdout: "1_1663008580.904048_86.1" 2022-09-12T19:14:25.632 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:14:25.633 INFO:teuthology.orchestra.run.smithi138.stdout: "period_map": { 2022-09-12T19:14:25.633 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e", 2022-09-12T19:14:25.633 INFO:teuthology.orchestra.run.smithi138.stdout: "zonegroups": [ 2022-09-12T19:14:25.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:25.634 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "test-zonegroup", 2022-09-12T19:14:25.634 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zonegroup", 2022-09-12T19:14:25.634 INFO:teuthology.orchestra.run.smithi138.stdout: "api_name": "", 2022-09-12T19:14:25.635 INFO:teuthology.orchestra.run.smithi138.stdout: "is_master": true, 2022-09-12T19:14:25.635 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:14:25.635 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T19:14:25.636 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:14:25.636 INFO:teuthology.orchestra.run.smithi138.stdout: "hostnames": [], 2022-09-12T19:14:25.636 INFO:teuthology.orchestra.run.smithi138.stdout: "hostnames_s3website": [], 2022-09-12T19:14:25.636 INFO:teuthology.orchestra.run.smithi138.stdout: "master_zone": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:14:25.637 INFO:teuthology.orchestra.run.smithi138.stdout: "zones": [ 2022-09-12T19:14:25.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:25.637 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T19:14:25.638 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone3", 2022-09-12T19:14:25.638 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:14:25.638 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi035.front.sepia.ceph.com:8001" 2022-09-12T19:14:25.639 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:14:25.639 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T19:14:25.639 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T19:14:25.639 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:14:25.640 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T19:14:25.640 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "", 2022-09-12T19:14:25.640 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": true, 2022-09-12T19:14:25.640 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [], 2022-09-12T19:14:25.641 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T19:14:25.641 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T19:14:25.641 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:14:25.642 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:14:25.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:25.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:25.642 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "eb41eca3-59e3-4448-adac-b7c5067fb406", 2022-09-12T19:14:25.643 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone4", 2022-09-12T19:14:25.643 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:14:25.643 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi138.front.sepia.ceph.com:8001" 2022-09-12T19:14:25.643 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:14:25.644 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T19:14:25.644 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T19:14:25.645 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:14:25.645 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T19:14:25.645 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "pubsub", 2022-09-12T19:14:25.645 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": false, 2022-09-12T19:14:25.646 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [ 2022-09-12T19:14:25.646 INFO:teuthology.orchestra.run.smithi138.stdout: "test-zone1" 2022-09-12T19:14:25.646 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:14:25.647 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T19:14:25.647 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T19:14:25.647 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:14:25.648 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:14:25.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:25.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:25.649 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:14:25.650 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone1", 2022-09-12T19:14:25.650 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:14:25.651 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T19:14:25.651 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:14:25.651 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T19:14:25.652 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T19:14:25.652 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:14:25.652 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T19:14:25.652 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "", 2022-09-12T19:14:25.653 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": true, 2022-09-12T19:14:25.653 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [], 2022-09-12T19:14:25.654 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T19:14:25.654 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T19:14:25.654 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:14:25.655 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:14:25.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:25.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:25.655 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:14:25.656 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone2", 2022-09-12T19:14:25.656 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:14:25.656 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi138.front.sepia.ceph.com:8000" 2022-09-12T19:14:25.656 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:14:25.657 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T19:14:25.657 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T19:14:25.657 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:14:25.657 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T19:14:25.658 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "", 2022-09-12T19:14:25.658 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": true, 2022-09-12T19:14:25.658 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [], 2022-09-12T19:14:25.659 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T19:14:25.659 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T19:14:25.659 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:14:25.659 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:14:25.660 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:25.660 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:14:25.660 INFO:teuthology.orchestra.run.smithi138.stdout: "placement_targets": [ 2022-09-12T19:14:25.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:25.662 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "default-placement", 2022-09-12T19:14:25.663 INFO:teuthology.orchestra.run.smithi138.stdout: "tags": [], 2022-09-12T19:14:25.663 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_classes": [ 2022-09-12T19:14:25.663 INFO:teuthology.orchestra.run.smithi138.stdout: "STANDARD" 2022-09-12T19:14:25.664 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:14:25.664 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:25.664 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:14:25.665 INFO:teuthology.orchestra.run.smithi138.stdout: "default_placement": "default-placement", 2022-09-12T19:14:25.665 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:14:25.665 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_policy": { 2022-09-12T19:14:25.665 INFO:teuthology.orchestra.run.smithi138.stdout: "groups": [] 2022-09-12T19:14:25.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:25.666 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled_features": [ 2022-09-12T19:14:25.667 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:14:25.667 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:14:25.668 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:25.668 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:14:25.669 INFO:teuthology.orchestra.run.smithi138.stdout: "short_zone_ids": [ 2022-09-12T19:14:25.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:25.670 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T19:14:25.670 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 227168046 2022-09-12T19:14:25.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:25.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:25.672 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "eb41eca3-59e3-4448-adac-b7c5067fb406", 2022-09-12T19:14:25.672 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 2080702172 2022-09-12T19:14:25.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:25.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:25.673 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:14:25.673 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 422195086 2022-09-12T19:14:25.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:25.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:14:25.674 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:14:25.674 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 2881234289 2022-09-12T19:14:25.675 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:25.675 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:14:25.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:25.676 INFO:teuthology.orchestra.run.smithi138.stdout: "master_zonegroup": "test-zonegroup", 2022-09-12T19:14:25.676 INFO:teuthology.orchestra.run.smithi138.stdout: "master_zone": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:14:25.676 INFO:teuthology.orchestra.run.smithi138.stdout: "period_config": { 2022-09-12T19:14:25.676 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_quota": { 2022-09-12T19:14:25.677 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false, 2022-09-12T19:14:25.677 INFO:teuthology.orchestra.run.smithi138.stdout: "check_on_raw": false, 2022-09-12T19:14:25.677 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size": -1, 2022-09-12T19:14:25.677 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size_kb": 0, 2022-09-12T19:14:25.678 INFO:teuthology.orchestra.run.smithi138.stdout: "max_objects": -1 2022-09-12T19:14:25.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:25.678 INFO:teuthology.orchestra.run.smithi138.stdout: "user_quota": { 2022-09-12T19:14:25.678 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false, 2022-09-12T19:14:25.679 INFO:teuthology.orchestra.run.smithi138.stdout: "check_on_raw": false, 2022-09-12T19:14:25.679 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size": -1, 2022-09-12T19:14:25.679 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size_kb": 0, 2022-09-12T19:14:25.679 INFO:teuthology.orchestra.run.smithi138.stdout: "max_objects": -1 2022-09-12T19:14:25.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:25.680 INFO:teuthology.orchestra.run.smithi138.stdout: "user_ratelimit": { 2022-09-12T19:14:25.680 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_ops": 0, 2022-09-12T19:14:25.680 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_ops": 0, 2022-09-12T19:14:25.681 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_bytes": 0, 2022-09-12T19:14:25.681 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_bytes": 0, 2022-09-12T19:14:25.681 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false 2022-09-12T19:14:25.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:25.682 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_ratelimit": { 2022-09-12T19:14:25.682 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_ops": 0, 2022-09-12T19:14:25.683 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_ops": 0, 2022-09-12T19:14:25.683 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_bytes": 0, 2022-09-12T19:14:25.683 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_bytes": 0, 2022-09-12T19:14:25.683 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false 2022-09-12T19:14:25.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:25.684 INFO:teuthology.orchestra.run.smithi138.stdout: "anonymous_ratelimit": { 2022-09-12T19:14:25.684 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_ops": 0, 2022-09-12T19:14:25.684 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_ops": 0, 2022-09-12T19:14:25.685 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_bytes": 0, 2022-09-12T19:14:25.685 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_bytes": 0, 2022-09-12T19:14:25.685 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false 2022-09-12T19:14:25.685 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:14:25.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:14:25.686 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:14:25.686 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_name": "test-realm", 2022-09-12T19:14:25.686 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 3 2022-09-12T19:14:25.687 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:14:25.687 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2022-09-12T19:14:29.944 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2022-09-12T19:15:25.689 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-42 2022-09-12T19:15:25.691 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:15:25.692 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:15:25.693 DEBUG:boto:Method: PUT 2022-09-12T19:15:25.693 DEBUG:boto:Path: /jdpiak-42/ 2022-09-12T19:15:25.694 DEBUG:boto:Data: 2022-09-12T19:15:25.694 DEBUG:boto:Headers: {} 2022-09-12T19:15:25.695 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:15:25.695 DEBUG:boto:Port: 8000 2022-09-12T19:15:25.696 DEBUG:boto:Params: {} 2022-09-12T19:15:25.697 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:15:25.698 DEBUG:boto:Token: None 2022-09-12T19:15:25.698 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:15:25 GMT /jdpiak-42/ 2022-09-12T19:15:25.699 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:47aguzKTsvQ4PRWAKNoJKi9kM/I= 2022-09-12T19:15:25.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:15:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:47aguzKTsvQ4PRWAKNoJKi9kM/I=', 'Content-Length': '0'} 2022-09-12T19:15:25.740 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003b4ae3ccf638805f-00631f854d-1416-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:15:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:15:25.741 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jdpiak-43 2022-09-12T19:15:25.741 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:15:25.742 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:15:25.742 DEBUG:boto:Method: PUT 2022-09-12T19:15:25.743 DEBUG:boto:Path: /jdpiak-43/ 2022-09-12T19:15:25.743 DEBUG:boto:Data: 2022-09-12T19:15:25.744 DEBUG:boto:Headers: {} 2022-09-12T19:15:25.744 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:15:25.745 DEBUG:boto:Port: 8000 2022-09-12T19:15:25.745 DEBUG:boto:Params: {} 2022-09-12T19:15:25.746 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:15:25.747 DEBUG:boto:Token: None 2022-09-12T19:15:25.747 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:15:25 GMT /jdpiak-43/ 2022-09-12T19:15:25.748 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QU64i35TEuHBG/0xxz4KneisLbg= 2022-09-12T19:15:25.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:15:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QU64i35TEuHBG/0xxz4KneisLbg=', 'Content-Length': '0'} 2022-09-12T19:15:25.775 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000737d3737f5755ce4-00631f854d-142a-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:15:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:15:25.777 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:15:25.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:15:25.780 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:15:25.835 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:15:25.835 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:15:25.872 INFO:teuthology.orchestra.run.smithi138.stderr:No --period given, using current period=2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e 2022-09-12T19:15:25.888 INFO:teuthology.orchestra.run.smithi138.stdout:[ 2022-09-12T19:15:25.888 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:15:25.889 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:15:25.889 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "0.000000" 2022-09-12T19:15:25.891 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:15:25.891 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:15:25.891 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:15:25.892 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "0.000000" 2022-09-12T19:15:25.892 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:15:25.893 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:15:25.893 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010125.771044_393.1", 2022-09-12T19:15:25.894 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:15:25.771044Z" 2022-09-12T19:15:25.894 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:15:25.895 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:15:25.895 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:15:25.896 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "0.000000" 2022-09-12T19:15:25.896 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:15:25.896 INFO:teuthology.orchestra.run.smithi138.stdout:] 2022-09-12T19:15:25.899 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1663010125.771044_393.1', 3: ''} 2022-09-12T19:15:25.900 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2022-09-12T19:15:25.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:15:25.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:15:25.902 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:15:25.963 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:15:25.964 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:15:26.002 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:15:25.999+0000 7f9be1caf600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:15:26.004 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:15:26.002+0000 7f9be1caf600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:15:26.022 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:15:26.023 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:15:26.023 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:15:26.024 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:15:26.024 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:15:26.025 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e", 2022-09-12T19:15:26.025 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 3 2022-09-12T19:15:26.026 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:26.026 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:15:26.027 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:15:26.027 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:15:26.027 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:15:26.028 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 0, 2022-09-12T19:15:26.028 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "bucket.instance:jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29", 2022-09-12T19:15:26.029 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:15:26.029 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 81, 2022-09-12T19:15:26.029 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 21, 2022-09-12T19:15:26.029 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:15:26.030 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 0 2022-09-12T19:15:26.030 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:15:26.030 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:26.031 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:15:26.031 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:15:26.031 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:15:26.031 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:15:26.032 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:15:26.032 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:15:26.032 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:15:26.033 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:15:26.033 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:15:26.033 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 3 2022-09-12T19:15:26.034 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:15:26.034 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:26.034 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:15:26.034 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:15:26.035 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:15:26.035 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:15:26.036 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663010125.715126_386.1", 2022-09-12T19:15:26.036 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:15:26.036 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:15:26.036 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:15:26.037 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:15:25.715126Z", 2022-09-12T19:15:26.037 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 3 2022-09-12T19:15:26.037 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:15:26.037 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:26.038 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:15:26.038 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:15:26.038 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:15:26.039 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:15:26.039 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:15:26.039 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:15:26.039 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:15:26.040 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:15:26.040 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:15:26.040 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 3 2022-09-12T19:15:26.041 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:15:26.041 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:15:26.041 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:15:26.041 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:26.042 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:15:26.042 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 81, 2022-09-12T19:15:26.042 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 21 2022-09-12T19:15:26.042 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:26.043 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:15:26Z" 2022-09-12T19:15:26.043 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:15:26.045 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e", "realm_epoch": 3 }, "markers": [ { "key": 0, "val": { "state": 0, "marker": "bucket.instance:jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29", "next_step_marker": "", "total_entries": 81, "pos": 21, "timestamp": "0.000000", "realm_epoch": 0 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 3 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010125.715126_386.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:15:25.715126Z", "realm_epoch": 3 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 3 } } ] }, "full_sync": { "total": 81, "complete": 21 }, "current_time": "2022-09-12T19:15:26Z" } 2022-09-12T19:15:26.045 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 0, 'marker': 'bucket.instance:jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29', 'next_step_marker': '', 'total_entries': 81, 'pos': 21, 'timestamp': '0.000000', 'realm_epoch': 0}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 3}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010125.715126_386.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:15:25.715126Z', 'realm_epoch': 3}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 3}}] 2022-09-12T19:15:26.046 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010125.771044_393.1', 3: ''} 2022-09-12T19:15:26.047 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010125.715126_386.1', 3: ''} 2022-09-12T19:15:26.047 WARNING:rgw_multi.tests:zone test-zone1 behind master: shard=2 master=1_1663010125.771044_393.1 target=1_1663010125.715126_386.1 2022-09-12T19:15:31.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:15:31.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:15:31.050 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:15:31.109 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:15:31.110 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:15:31.147 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:15:31.144+0000 7f870cfe2600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:15:31.148 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:15:31.145+0000 7f870cfe2600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:15:31.164 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:15:31.164 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:15:31.165 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:15:31.165 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:15:31.165 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:15:31.166 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e", 2022-09-12T19:15:31.166 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 3 2022-09-12T19:15:31.166 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:31.166 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:15:31.167 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:15:31.167 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:15:31.167 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:15:31.167 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 0, 2022-09-12T19:15:31.168 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "bucket.instance:jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29", 2022-09-12T19:15:31.168 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:15:31.168 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 81, 2022-09-12T19:15:31.169 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 21, 2022-09-12T19:15:31.169 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:15:31.169 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 0 2022-09-12T19:15:31.169 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:15:31.170 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:31.170 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:15:31.170 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:15:31.170 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:15:31.171 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:15:31.171 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:15:31.171 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:15:31.172 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:15:31.172 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:15:31.172 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:15:31.172 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 3 2022-09-12T19:15:31.173 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:15:31.173 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:31.174 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:15:31.174 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:15:31.175 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:15:31.175 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:15:31.175 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663010125.771044_393.1", 2022-09-12T19:15:31.175 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:15:31.176 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:15:31.176 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:15:31.177 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:15:25.771044Z", 2022-09-12T19:15:31.177 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 3 2022-09-12T19:15:31.178 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:15:31.178 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:31.179 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:15:31.179 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:15:31.180 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:15:31.180 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:15:31.181 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:15:31.181 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:15:31.182 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:15:31.182 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:15:31.183 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:15:31.183 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 3 2022-09-12T19:15:31.184 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:15:31.184 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:15:31.185 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:15:31.185 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:31.186 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:15:31.186 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 81, 2022-09-12T19:15:31.187 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 21 2022-09-12T19:15:31.187 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:31.188 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:15:31Z" 2022-09-12T19:15:31.188 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:15:31.189 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e", "realm_epoch": 3 }, "markers": [ { "key": 0, "val": { "state": 0, "marker": "bucket.instance:jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29", "next_step_marker": "", "total_entries": 81, "pos": 21, "timestamp": "0.000000", "realm_epoch": 0 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 3 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010125.771044_393.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:15:25.771044Z", "realm_epoch": 3 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 3 } } ] }, "full_sync": { "total": 81, "complete": 21 }, "current_time": "2022-09-12T19:15:31Z" } 2022-09-12T19:15:31.190 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 0, 'marker': 'bucket.instance:jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29', 'next_step_marker': '', 'total_entries': 81, 'pos': 21, 'timestamp': '0.000000', 'realm_epoch': 0}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 3}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010125.771044_393.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:15:25.771044Z', 'realm_epoch': 3}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 3}}] 2022-09-12T19:15:31.191 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010125.771044_393.1', 3: ''} 2022-09-12T19:15:31.192 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010125.771044_393.1', 3: ''} 2022-09-12T19:15:31.193 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2022-09-12T19:15:31.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:15:31.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:15:31.196 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:15:31.248 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:15:31.249 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:15:31.270 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:15:31.267+0000 7ff7c7617600 0 NOTICE: overriding master zone: fdf3b910-2ea3-4164-a3f8-b3e146af1c44 2022-09-12T19:15:31.271 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:15:31.267+0000 7ff7c7617600 0 failed reading obj info from .rgw.root:zone_info.eb41eca3-59e3-4448-adac-b7c5067fb406: (2) No such file or directory 2022-09-12T19:15:31.271 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:15:31.267+0000 7ff7c7617600 0 WARNING: could not read zone params for zone id=eb41eca3-59e3-4448-adac-b7c5067fb406 name=test-zone4 2022-09-12T19:15:31.272 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:15:31.269+0000 7ff7c7617600 0 failed reading obj info from .rgw.root:zone_info.fdf3b910-2ea3-4164-a3f8-b3e146af1c44: (2) No such file or directory 2022-09-12T19:15:31.273 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:15:31.269+0000 7ff7c7617600 0 WARNING: could not read zone params for zone id=fdf3b910-2ea3-4164-a3f8-b3e146af1c44 name=test-zone2 2022-09-12T19:15:31.278 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:15:31.278 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:15:31.279 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone1", 2022-09-12T19:15:31.279 INFO:teuthology.orchestra.run.smithi035.stdout: "domain_root": "test-zone1.rgw.meta:root", 2022-09-12T19:15:31.279 INFO:teuthology.orchestra.run.smithi035.stdout: "control_pool": "test-zone1.rgw.control", 2022-09-12T19:15:31.280 INFO:teuthology.orchestra.run.smithi035.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2022-09-12T19:15:31.280 INFO:teuthology.orchestra.run.smithi035.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2022-09-12T19:15:31.280 INFO:teuthology.orchestra.run.smithi035.stdout: "log_pool": "test-zone1.rgw.log", 2022-09-12T19:15:31.281 INFO:teuthology.orchestra.run.smithi035.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2022-09-12T19:15:31.281 INFO:teuthology.orchestra.run.smithi035.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2022-09-12T19:15:31.282 INFO:teuthology.orchestra.run.smithi035.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2022-09-12T19:15:31.282 INFO:teuthology.orchestra.run.smithi035.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2022-09-12T19:15:31.283 INFO:teuthology.orchestra.run.smithi035.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2022-09-12T19:15:31.283 INFO:teuthology.orchestra.run.smithi035.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2022-09-12T19:15:31.284 INFO:teuthology.orchestra.run.smithi035.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2022-09-12T19:15:31.284 INFO:teuthology.orchestra.run.smithi035.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2022-09-12T19:15:31.293 INFO:teuthology.orchestra.run.smithi035.stdout: "otp_pool": "test-zone1.rgw.otp", 2022-09-12T19:15:31.294 INFO:teuthology.orchestra.run.smithi035.stdout: "system_key": { 2022-09-12T19:15:31.294 INFO:teuthology.orchestra.run.smithi035.stdout: "access_key": "RDIEFLSOLOEHUH1F", 2022-09-12T19:15:31.295 INFO:teuthology.orchestra.run.smithi035.stdout: "secret_key": "LDCKRd1RecZNgErNcVDnUigDhC7QggCQ" 2022-09-12T19:15:31.295 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:31.296 INFO:teuthology.orchestra.run.smithi035.stdout: "placement_pools": [ 2022-09-12T19:15:31.296 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:15:31.297 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "default-placement", 2022-09-12T19:15:31.297 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:15:31.298 INFO:teuthology.orchestra.run.smithi035.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2022-09-12T19:15:31.298 INFO:teuthology.orchestra.run.smithi035.stdout: "storage_classes": { 2022-09-12T19:15:31.299 INFO:teuthology.orchestra.run.smithi035.stdout: "STANDARD": { 2022-09-12T19:15:31.299 INFO:teuthology.orchestra.run.smithi035.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2022-09-12T19:15:31.300 INFO:teuthology.orchestra.run.smithi035.stdout: "compression_type": "random" 2022-09-12T19:15:31.300 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:15:31.301 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:31.301 INFO:teuthology.orchestra.run.smithi035.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2022-09-12T19:15:31.302 INFO:teuthology.orchestra.run.smithi035.stdout: "index_type": 0 2022-09-12T19:15:31.302 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:15:31.303 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:15:31.303 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:15:31.304 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:15:31.304 INFO:teuthology.orchestra.run.smithi035.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2022-09-12T19:15:31.305 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:15:31.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:15:31.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:15:31.308 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:15:31.358 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:15:31.359 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:15:31.400 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:15:31.396+0000 7fa70815f600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:15:31.401 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:15:31.397+0000 7fa70815f600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:15:31.411 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:15:31.412 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:15:31.413 INFO:teuthology.orchestra.run.smithi035.stdout: "epoch": 1, 2022-09-12T19:15:31.413 INFO:teuthology.orchestra.run.smithi035.stdout: "predecessor_uuid": "2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e", 2022-09-12T19:15:31.414 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": [ 2022-09-12T19:15:31.414 INFO:teuthology.orchestra.run.smithi035.stdout: "", 2022-09-12T19:15:31.414 INFO:teuthology.orchestra.run.smithi035.stdout: "", 2022-09-12T19:15:31.415 INFO:teuthology.orchestra.run.smithi035.stdout: "1_1663010125.771044_393.1", 2022-09-12T19:15:31.415 INFO:teuthology.orchestra.run.smithi035.stdout: "" 2022-09-12T19:15:31.415 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:15:31.416 INFO:teuthology.orchestra.run.smithi035.stdout: "period_map": { 2022-09-12T19:15:31.416 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:15:31.417 INFO:teuthology.orchestra.run.smithi035.stdout: "zonegroups": [ 2022-09-12T19:15:31.417 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:15:31.418 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "test-zonegroup", 2022-09-12T19:15:31.418 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zonegroup", 2022-09-12T19:15:31.419 INFO:teuthology.orchestra.run.smithi035.stdout: "api_name": "", 2022-09-12T19:15:31.419 INFO:teuthology.orchestra.run.smithi035.stdout: "is_master": true, 2022-09-12T19:15:31.420 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T19:15:31.420 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T19:15:31.421 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:15:31.421 INFO:teuthology.orchestra.run.smithi035.stdout: "hostnames": [], 2022-09-12T19:15:31.421 INFO:teuthology.orchestra.run.smithi035.stdout: "hostnames_s3website": [], 2022-09-12T19:15:31.422 INFO:teuthology.orchestra.run.smithi035.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:15:31.422 INFO:teuthology.orchestra.run.smithi035.stdout: "zones": [ 2022-09-12T19:15:31.423 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:15:31.423 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T19:15:31.423 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone3", 2022-09-12T19:15:31.424 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T19:15:31.424 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi035.front.sepia.ceph.com:8001" 2022-09-12T19:15:31.425 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:15:31.425 INFO:teuthology.orchestra.run.smithi035.stdout: "log_meta": false, 2022-09-12T19:15:31.426 INFO:teuthology.orchestra.run.smithi035.stdout: "log_data": true, 2022-09-12T19:15:31.426 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:15:31.426 INFO:teuthology.orchestra.run.smithi035.stdout: "read_only": false, 2022-09-12T19:15:31.426 INFO:teuthology.orchestra.run.smithi035.stdout: "tier_type": "", 2022-09-12T19:15:31.427 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from_all": true, 2022-09-12T19:15:31.427 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from": [], 2022-09-12T19:15:31.427 INFO:teuthology.orchestra.run.smithi035.stdout: "redirect_zone": "", 2022-09-12T19:15:31.427 INFO:teuthology.orchestra.run.smithi035.stdout: "supported_features": [ 2022-09-12T19:15:31.428 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:15:31.428 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:15:31.428 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:31.428 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:15:31.429 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "eb41eca3-59e3-4448-adac-b7c5067fb406", 2022-09-12T19:15:31.429 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone4", 2022-09-12T19:15:31.429 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T19:15:31.430 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi138.front.sepia.ceph.com:8001" 2022-09-12T19:15:31.430 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:15:31.431 INFO:teuthology.orchestra.run.smithi035.stdout: "log_meta": false, 2022-09-12T19:15:31.431 INFO:teuthology.orchestra.run.smithi035.stdout: "log_data": true, 2022-09-12T19:15:31.432 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:15:31.432 INFO:teuthology.orchestra.run.smithi035.stdout: "read_only": false, 2022-09-12T19:15:31.432 INFO:teuthology.orchestra.run.smithi035.stdout: "tier_type": "pubsub", 2022-09-12T19:15:31.432 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from_all": false, 2022-09-12T19:15:31.433 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from": [ 2022-09-12T19:15:31.433 INFO:teuthology.orchestra.run.smithi035.stdout: "test-zone1" 2022-09-12T19:15:31.433 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:15:31.433 INFO:teuthology.orchestra.run.smithi035.stdout: "redirect_zone": "", 2022-09-12T19:15:31.434 INFO:teuthology.orchestra.run.smithi035.stdout: "supported_features": [ 2022-09-12T19:15:31.434 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:15:31.435 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:15:31.435 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:31.436 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:15:31.436 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:15:31.437 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone1", 2022-09-12T19:15:31.437 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T19:15:31.438 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T19:15:31.438 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:15:31.439 INFO:teuthology.orchestra.run.smithi035.stdout: "log_meta": false, 2022-09-12T19:15:31.439 INFO:teuthology.orchestra.run.smithi035.stdout: "log_data": true, 2022-09-12T19:15:31.440 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:15:31.440 INFO:teuthology.orchestra.run.smithi035.stdout: "read_only": false, 2022-09-12T19:15:31.441 INFO:teuthology.orchestra.run.smithi035.stdout: "tier_type": "", 2022-09-12T19:15:31.441 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from_all": true, 2022-09-12T19:15:31.442 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from": [], 2022-09-12T19:15:31.442 INFO:teuthology.orchestra.run.smithi035.stdout: "redirect_zone": "", 2022-09-12T19:15:31.443 INFO:teuthology.orchestra.run.smithi035.stdout: "supported_features": [ 2022-09-12T19:15:31.443 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:15:31.444 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:15:31.444 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:31.445 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:15:31.445 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:15:31.446 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone2", 2022-09-12T19:15:31.446 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T19:15:31.447 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi138.front.sepia.ceph.com:8000" 2022-09-12T19:15:31.447 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:15:31.448 INFO:teuthology.orchestra.run.smithi035.stdout: "log_meta": false, 2022-09-12T19:15:31.448 INFO:teuthology.orchestra.run.smithi035.stdout: "log_data": true, 2022-09-12T19:15:31.449 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:15:31.449 INFO:teuthology.orchestra.run.smithi035.stdout: "read_only": false, 2022-09-12T19:15:31.450 INFO:teuthology.orchestra.run.smithi035.stdout: "tier_type": "", 2022-09-12T19:15:31.450 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from_all": true, 2022-09-12T19:15:31.451 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from": [], 2022-09-12T19:15:31.451 INFO:teuthology.orchestra.run.smithi035.stdout: "redirect_zone": "", 2022-09-12T19:15:31.451 INFO:teuthology.orchestra.run.smithi035.stdout: "supported_features": [ 2022-09-12T19:15:31.452 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:15:31.452 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:15:31.453 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:15:31.453 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:15:31.454 INFO:teuthology.orchestra.run.smithi035.stdout: "placement_targets": [ 2022-09-12T19:15:31.454 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:15:31.455 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "default-placement", 2022-09-12T19:15:31.457 INFO:teuthology.orchestra.run.smithi035.stdout: "tags": [], 2022-09-12T19:15:31.458 INFO:teuthology.orchestra.run.smithi035.stdout: "storage_classes": [ 2022-09-12T19:15:31.458 INFO:teuthology.orchestra.run.smithi035.stdout: "STANDARD" 2022-09-12T19:15:31.459 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:15:31.459 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:15:31.460 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:15:31.460 INFO:teuthology.orchestra.run.smithi035.stdout: "default_placement": "default-placement", 2022-09-12T19:15:31.461 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:15:31.461 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_policy": { 2022-09-12T19:15:31.462 INFO:teuthology.orchestra.run.smithi035.stdout: "groups": [] 2022-09-12T19:15:31.462 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:31.463 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled_features": [ 2022-09-12T19:15:31.463 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:15:31.464 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:15:31.464 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:15:31.465 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:15:31.465 INFO:teuthology.orchestra.run.smithi035.stdout: "short_zone_ids": [ 2022-09-12T19:15:31.466 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:15:31.466 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T19:15:31.467 INFO:teuthology.orchestra.run.smithi035.stdout: "val": 227168046 2022-09-12T19:15:31.467 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:31.468 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:15:31.468 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "eb41eca3-59e3-4448-adac-b7c5067fb406", 2022-09-12T19:15:31.468 INFO:teuthology.orchestra.run.smithi035.stdout: "val": 2080702172 2022-09-12T19:15:31.469 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:31.469 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:15:31.470 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:15:31.470 INFO:teuthology.orchestra.run.smithi035.stdout: "val": 422195086 2022-09-12T19:15:31.470 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:31.471 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:15:31.471 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:15:31.471 INFO:teuthology.orchestra.run.smithi035.stdout: "val": 2881234289 2022-09-12T19:15:31.472 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:15:31.473 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:15:31.473 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:31.474 INFO:teuthology.orchestra.run.smithi035.stdout: "master_zonegroup": "test-zonegroup", 2022-09-12T19:15:31.474 INFO:teuthology.orchestra.run.smithi035.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:15:31.475 INFO:teuthology.orchestra.run.smithi035.stdout: "period_config": { 2022-09-12T19:15:31.475 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_quota": { 2022-09-12T19:15:31.476 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false, 2022-09-12T19:15:31.476 INFO:teuthology.orchestra.run.smithi035.stdout: "check_on_raw": false, 2022-09-12T19:15:31.476 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size": -1, 2022-09-12T19:15:31.476 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size_kb": 0, 2022-09-12T19:15:31.477 INFO:teuthology.orchestra.run.smithi035.stdout: "max_objects": -1 2022-09-12T19:15:31.477 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:31.477 INFO:teuthology.orchestra.run.smithi035.stdout: "user_quota": { 2022-09-12T19:15:31.477 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false, 2022-09-12T19:15:31.478 INFO:teuthology.orchestra.run.smithi035.stdout: "check_on_raw": false, 2022-09-12T19:15:31.478 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size": -1, 2022-09-12T19:15:31.478 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size_kb": 0, 2022-09-12T19:15:31.479 INFO:teuthology.orchestra.run.smithi035.stdout: "max_objects": -1 2022-09-12T19:15:31.479 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:31.479 INFO:teuthology.orchestra.run.smithi035.stdout: "user_ratelimit": { 2022-09-12T19:15:31.479 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_ops": 0, 2022-09-12T19:15:31.480 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_ops": 0, 2022-09-12T19:15:31.480 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_bytes": 0, 2022-09-12T19:15:31.480 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_bytes": 0, 2022-09-12T19:15:31.481 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false 2022-09-12T19:15:31.481 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:31.481 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_ratelimit": { 2022-09-12T19:15:31.481 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_ops": 0, 2022-09-12T19:15:31.482 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_ops": 0, 2022-09-12T19:15:31.482 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_bytes": 0, 2022-09-12T19:15:31.482 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_bytes": 0, 2022-09-12T19:15:31.483 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false 2022-09-12T19:15:31.483 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:31.483 INFO:teuthology.orchestra.run.smithi035.stdout: "anonymous_ratelimit": { 2022-09-12T19:15:31.483 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_ops": 0, 2022-09-12T19:15:31.484 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_ops": 0, 2022-09-12T19:15:31.484 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_bytes": 0, 2022-09-12T19:15:31.484 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_bytes": 0, 2022-09-12T19:15:31.484 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false 2022-09-12T19:15:31.485 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:15:31.485 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:15:31.485 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:15:31.485 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_name": "test-realm", 2022-09-12T19:15:31.485 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:15:31.486 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:15:31.486 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2022-09-12T19:16:31.488 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-44 2022-09-12T19:16:31.489 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:31.490 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:31.491 DEBUG:boto:Method: PUT 2022-09-12T19:16:31.491 DEBUG:boto:Path: /jdpiak-44/ 2022-09-12T19:16:31.492 DEBUG:boto:Data: 2022-09-12T19:16:31.494 DEBUG:boto:Headers: {} 2022-09-12T19:16:31.494 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:31.495 DEBUG:boto:Port: 8000 2022-09-12T19:16:31.495 DEBUG:boto:Params: {} 2022-09-12T19:16:31.496 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:16:31.497 DEBUG:boto:Token: None 2022-09-12T19:16:31.498 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:16:31 GMT /jdpiak-44/ 2022-09-12T19:16:31.498 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JVX43SP8th2MJg2XKHWCb0jcSEA= 2022-09-12T19:16:31.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JVX43SP8th2MJg2XKHWCb0jcSEA=', 'Content-Length': '0'} 2022-09-12T19:16:31.522 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000492cf697a9301ff2-00631f858f-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:31.523 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jdpiak-45 2022-09-12T19:16:31.524 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:31.524 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:31.525 DEBUG:boto:Method: PUT 2022-09-12T19:16:31.525 DEBUG:boto:Path: /jdpiak-45/ 2022-09-12T19:16:31.526 DEBUG:boto:Data: 2022-09-12T19:16:31.526 DEBUG:boto:Headers: {} 2022-09-12T19:16:31.527 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:31.527 DEBUG:boto:Port: 8000 2022-09-12T19:16:31.527 DEBUG:boto:Params: {} 2022-09-12T19:16:31.528 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:16:31.529 DEBUG:boto:Token: None 2022-09-12T19:16:31.529 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:16:31 GMT /jdpiak-45/ 2022-09-12T19:16:31.530 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rSCO9q0cj1tjRBqy/Qj0/CXY0jk= 2022-09-12T19:16:31.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rSCO9q0cj1tjRBqy/Qj0/CXY0jk=', 'Content-Length': '0'} 2022-09-12T19:16:31.569 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000addbfc5653b9f407-00631f858f-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:31.570 INFO:tasks.rgw.c1.client.1:Restarting daemon 2022-09-12T19:16:31.571 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2022-09-12T19:16:31.575 INFO:tasks.rgw.c1.client.1:Started 2022-09-12T19:16:31.577 DEBUG:teuthology.orchestra.run.smithi035:> curl http://smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:31.602 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-12T19:16:31.603 INFO:teuthology.orchestra.run.smithi035.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-12T19:16:31.604 INFO:teuthology.orchestra.run.smithi035.stderr: Dload Upload Total Spent Left Speed 2022-09-12T19:16:31.604 INFO:teuthology.orchestra.run.smithi035.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi035.front.sepia.ceph.com port 8001: Connection refused 2022-09-12T19:16:32.605 DEBUG:teuthology.orchestra.run.smithi035:> curl http://smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:32.632 INFO:teuthology.orchestra.run.smithi035.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-12T19:16:32.632 INFO:teuthology.orchestra.run.smithi035.stderr: Dload Upload Total Spent Left Speed 2022-09-12T19:16:32.633 INFO:teuthology.orchestra.run.smithi035.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-12T19:16:32.634 INFO:teuthology.orchestra.run.smithi035.stdout:anonymous 2022-09-12T19:16:32.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:32.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:32.636 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:32.729 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:32.729 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:32.765 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=38ca1f57-1358-496c-a99b-df49ee365a4f 2022-09-12T19:16:32.781 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:16:32.781 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:32.782 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:16:32.782 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:16:32.783 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:32.783 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:32.784 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:16:32.784 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:16:32.785 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:32.785 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:32.786 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663010191.550373_215.1", 2022-09-12T19:16:32.786 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:16:31.550373Z" 2022-09-12T19:16:32.786 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:32.787 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:32.787 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:16:32.787 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:16:32.791 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:32.792 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:16:32.794 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1663010191.550373_215.1', 3: ''} 2022-09-12T19:16:32.794 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T19:16:32.795 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:32.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:32.797 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:32.856 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:32.857 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:32.918 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:16:32.916+0000 7f664d039600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:16:32.918 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:16:32.917+0000 7f664d039600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:16:32.947 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:16:32.948 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:16:32.948 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:16:32.948 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:16:32.949 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:16:32.949 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:16:32.949 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:16:32.949 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:32.950 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:16:32.950 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:32.950 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:16:32.951 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:16:32.951 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:16:32.951 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:16:32.951 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:16:32.952 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:16:32.952 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:16:32.952 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:16:32.952 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:16:32.953 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:32.953 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:32.953 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:32.953 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:16:32.954 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:16:32.954 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:16:32.954 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:16:32.955 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:16:32.955 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:16:32.955 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:16:32.955 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:16:32.956 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:16:32.956 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:32.956 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:32.956 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:32.957 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:16:32.957 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:16:32.957 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:16:32.958 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010191.510731_208.1", 2022-09-12T19:16:32.958 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:16:32.958 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:16:32.958 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:16:32.959 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:31.510731Z", 2022-09-12T19:16:32.959 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:16:32.959 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:32.959 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:32.960 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:32.960 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:16:32.960 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:16:32.960 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:16:32.961 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:16:32.961 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:16:32.961 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:16:32.962 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:16:32.962 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:16:32.962 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:16:32.962 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:32.963 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:32.963 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:16:32.963 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:32.963 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:16:32.964 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:16:32.964 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:16:32.965 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:32.965 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:16:32Z" 2022-09-12T19:16:32.966 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:16:32.968 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010191.510731_208.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:31.510731Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:16:32Z" } 2022-09-12T19:16:32.969 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010191.510731_208.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:31.510731Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:16:32.971 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010191.550373_215.1', 3: ''} 2022-09-12T19:16:32.972 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010191.510731_208.1', 3: ''} 2022-09-12T19:16:32.972 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010191.550373_215.1 target=1_1663010191.510731_208.1 2022-09-12T19:16:37.974 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:37.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:37.976 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:38.034 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:38.035 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:38.065 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:16:38.063+0000 7feb78b72600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:16:38.066 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:16:38.065+0000 7feb78b72600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:16:38.080 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:16:38.080 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:16:38.081 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:16:38.081 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:16:38.081 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:16:38.082 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:16:38.082 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:16:38.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:38.083 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:16:38.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:38.084 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:16:38.085 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:16:38.085 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:16:38.086 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:16:38.086 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:16:38.086 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:16:38.086 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:16:38.087 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:16:38.087 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:16:38.087 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:38.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:38.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:38.088 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:16:38.088 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:16:38.088 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:16:38.089 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:16:38.089 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:16:38.089 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:16:38.090 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:16:38.090 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:16:38.090 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:16:38.090 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:38.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:38.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:38.091 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:16:38.092 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:16:38.092 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:16:38.092 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010191.550373_215.1", 2022-09-12T19:16:38.093 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:16:38.093 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:16:38.093 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:16:38.093 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:31.550373Z", 2022-09-12T19:16:38.094 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:16:38.094 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:38.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:38.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:38.096 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:16:38.097 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:16:38.097 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:16:38.097 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:16:38.098 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:16:38.098 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:16:38.098 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:16:38.099 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:16:38.099 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:16:38.099 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:38.099 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:38.100 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:16:38.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:38.100 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:16:38.100 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:16:38.101 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:16:38.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:38.101 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:16:38Z" 2022-09-12T19:16:38.101 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:16:38.103 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010191.550373_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:31.550373Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:16:38Z" } 2022-09-12T19:16:38.103 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010191.550373_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:31.550373Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:16:38.104 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010191.550373_215.1', 3: ''} 2022-09-12T19:16:38.105 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010191.550373_215.1', 3: ''} 2022-09-12T19:16:38.105 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T19:16:38.106 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T19:16:38.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:38.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:38.107 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:38.169 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:38.171 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:38.218 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:16:38.215+0000 7fa29f643600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:16:38.220 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:16:38.217+0000 7fa29f643600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:16:38.238 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:16:38.239 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:16:38.239 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:16:38.239 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:16:38.240 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:16:38.240 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:16:38.240 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:16:38.241 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:38.241 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:16:38.242 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:38.242 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:16:38.242 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:16:38.243 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:16:38.243 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:16:38.243 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:16:38.244 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:16:38.244 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:16:38.244 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:16:38.244 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:16:38.245 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:38.245 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:38.245 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:38.246 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:16:38.246 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:16:38.247 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:16:38.247 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:16:38.247 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:16:38.248 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:16:38.248 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:16:38.249 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:16:38.249 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:16:38.249 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:38.250 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:38.250 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:38.251 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:16:38.251 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:16:38.252 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:16:38.252 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663010191.550373_215.1", 2022-09-12T19:16:38.252 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:16:38.253 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:16:38.253 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:16:38.254 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:31.550373Z", 2022-09-12T19:16:38.254 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:16:38.254 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:38.255 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:38.255 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:38.256 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:16:38.256 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:16:38.256 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:16:38.257 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:16:38.258 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:16:38.259 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:16:38.259 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:16:38.259 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:16:38.260 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:16:38.260 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:38.261 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:38.261 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:16:38.261 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:38.262 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:16:38.262 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:16:38.263 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:16:38.263 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:38.264 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:16:38Z" 2022-09-12T19:16:38.264 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:16:38.266 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010191.550373_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:31.550373Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:16:38Z" } 2022-09-12T19:16:38.267 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010191.550373_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:31.550373Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:16:38.268 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010191.550373_215.1', 3: ''} 2022-09-12T19:16:38.268 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010191.550373_215.1', 3: ''} 2022-09-12T19:16:38.269 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T19:16:38.270 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T19:16:38.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:38.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:38.273 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:38.325 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:38.326 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:38.364 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:16:38.362+0000 7f225468e600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:16:38.365 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:16:38.363+0000 7f225468e600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:16:38.385 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:16:38.386 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:16:38.386 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:16:38.386 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:16:38.387 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:16:38.387 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:16:38.387 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:16:38.388 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:38.388 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:16:38.388 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:38.388 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:16:38.389 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:16:38.389 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:16:38.389 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:16:38.389 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:16:38.390 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:16:38.390 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:16:38.390 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:16:38.390 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:16:38.391 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:38.391 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:38.391 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:38.391 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:16:38.392 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:16:38.392 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:16:38.392 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:16:38.392 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:16:38.393 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:16:38.393 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:16:38.394 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:16:38.394 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:16:38.394 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:38.394 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:38.395 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:38.395 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:16:38.395 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:16:38.396 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:16:38.396 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010191.550373_215.1", 2022-09-12T19:16:38.397 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:16:38.397 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:16:38.397 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:16:38.398 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:31.550373Z", 2022-09-12T19:16:38.398 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:16:38.399 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:38.399 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:38.400 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:38.400 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:16:38.400 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:16:38.401 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:16:38.401 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:16:38.401 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:16:38.401 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:16:38.402 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:16:38.402 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:16:38.402 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:16:38.403 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:38.403 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:38.403 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:16:38.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:38.404 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:16:38.405 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:16:38.405 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:16:38.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:38.406 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:16:38Z" 2022-09-12T19:16:38.406 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:16:38.408 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010191.550373_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:31.550373Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:16:38Z" } 2022-09-12T19:16:38.408 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010191.550373_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:31.550373Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:16:38.409 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010191.550373_215.1', 3: ''} 2022-09-12T19:16:38.409 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010191.550373_215.1', 3: ''} 2022-09-12T19:16:38.410 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T19:16:38.410 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-39 zones={test-zone2, test-zone1} 2022-09-12T19:16:38.411 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.412 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.412 DEBUG:boto:Method: HEAD 2022-09-12T19:16:38.413 DEBUG:boto:Path: /jdpiak-39/ 2022-09-12T19:16:38.413 DEBUG:boto:Data: 2022-09-12T19:16:38.414 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.414 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:38.415 DEBUG:boto:Port: 8000 2022-09-12T19:16:38.415 DEBUG:boto:Params: {} 2022-09-12T19:16:38.416 DEBUG:boto:Token: None 2022-09-12T19:16:38.417 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/ 2022-09-12T19:16:38.417 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ASYsRotZm8Jxmtv/Jao3heASk9U= 2022-09-12T19:16:38.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ASYsRotZm8Jxmtv/Jao3heASk9U=', 'Content-Length': '0'} 2022-09-12T19:16:38.421 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2080dbd955fbbac-00631f8596-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.421 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.422 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.423 DEBUG:boto:Method: HEAD 2022-09-12T19:16:38.423 DEBUG:boto:Path: /jdpiak-39/ 2022-09-12T19:16:38.424 DEBUG:boto:Data: 2022-09-12T19:16:38.424 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.425 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:38.425 DEBUG:boto:Port: 8000 2022-09-12T19:16:38.426 DEBUG:boto:Params: {} 2022-09-12T19:16:38.426 DEBUG:boto:Token: None 2022-09-12T19:16:38.427 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/ 2022-09-12T19:16:38.428 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ASYsRotZm8Jxmtv/Jao3heASk9U= 2022-09-12T19:16:38.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ASYsRotZm8Jxmtv/Jao3heASk9U=', 'Content-Length': '0'} 2022-09-12T19:16:38.431 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007fc11db64008c58a-00631f8596-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.432 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:38.433 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.433 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.434 DEBUG:boto:path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.434 DEBUG:boto:auth_path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.435 DEBUG:boto:Method: GET 2022-09-12T19:16:38.436 DEBUG:boto:Path: /jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.436 DEBUG:boto:Data: 2022-09-12T19:16:38.437 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.437 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:38.438 DEBUG:boto:Port: 8000 2022-09-12T19:16:38.438 DEBUG:boto:Params: {} 2022-09-12T19:16:38.439 DEBUG:boto:Token: None 2022-09-12T19:16:38.440 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/?versions 2022-09-12T19:16:38.440 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk= 2022-09-12T19:16:38.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk=', 'Content-Length': '0'} 2022-09-12T19:16:38.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002aaaf7c3d2355812-00631f8596-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.444 DEBUG:boto:b'jdpiak-39999false' 2022-09-12T19:16:38.445 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:38.446 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.447 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.447 DEBUG:boto:path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.447 DEBUG:boto:auth_path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.448 DEBUG:boto:Method: GET 2022-09-12T19:16:38.449 DEBUG:boto:Path: /jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.449 DEBUG:boto:Data: 2022-09-12T19:16:38.450 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.450 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:38.451 DEBUG:boto:Port: 8000 2022-09-12T19:16:38.451 DEBUG:boto:Params: {} 2022-09-12T19:16:38.452 DEBUG:boto:Token: None 2022-09-12T19:16:38.452 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/?versions 2022-09-12T19:16:38.453 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk= 2022-09-12T19:16:38.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk=', 'Content-Length': '0'} 2022-09-12T19:16:38.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe9aa720bc3e09d5-00631f8596-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.457 DEBUG:boto:b'jdpiak-39999false' 2022-09-12T19:16:38.458 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.459 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.459 DEBUG:boto:path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.460 DEBUG:boto:auth_path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.460 DEBUG:boto:Method: GET 2022-09-12T19:16:38.461 DEBUG:boto:Path: /jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.461 DEBUG:boto:Data: 2022-09-12T19:16:38.462 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.462 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:38.463 DEBUG:boto:Port: 8000 2022-09-12T19:16:38.463 DEBUG:boto:Params: {} 2022-09-12T19:16:38.464 DEBUG:boto:Token: None 2022-09-12T19:16:38.464 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/?versions 2022-09-12T19:16:38.465 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk= 2022-09-12T19:16:38.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk=', 'Content-Length': '0'} 2022-09-12T19:16:38.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c6f90de2f63d5c3-00631f8596-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.469 DEBUG:boto:b'jdpiak-39999false' 2022-09-12T19:16:38.470 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.471 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.471 DEBUG:boto:path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.472 DEBUG:boto:auth_path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.472 DEBUG:boto:Method: GET 2022-09-12T19:16:38.473 DEBUG:boto:Path: /jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.473 DEBUG:boto:Data: 2022-09-12T19:16:38.474 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.474 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:38.475 DEBUG:boto:Port: 8000 2022-09-12T19:16:38.475 DEBUG:boto:Params: {} 2022-09-12T19:16:38.476 DEBUG:boto:Token: None 2022-09-12T19:16:38.477 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/?versions 2022-09-12T19:16:38.477 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk= 2022-09-12T19:16:38.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk=', 'Content-Length': '0'} 2022-09-12T19:16:38.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000224d8c8099510520-00631f8596-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.482 DEBUG:boto:b'jdpiak-39999false' 2022-09-12T19:16:38.484 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-39 zones={test-zone2, test-zone1} 2022-09-12T19:16:38.485 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-39 zones={test-zone3, test-zone1} 2022-09-12T19:16:38.486 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.487 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.489 DEBUG:boto:Method: HEAD 2022-09-12T19:16:38.489 DEBUG:boto:Path: /jdpiak-39/ 2022-09-12T19:16:38.490 DEBUG:boto:Data: 2022-09-12T19:16:38.490 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.491 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:38.492 DEBUG:boto:Port: 8001 2022-09-12T19:16:38.492 DEBUG:boto:Params: {} 2022-09-12T19:16:38.493 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T19:16:38.493 DEBUG:boto:Token: None 2022-09-12T19:16:38.494 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/ 2022-09-12T19:16:38.495 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ASYsRotZm8Jxmtv/Jao3heASk9U= 2022-09-12T19:16:38.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ASYsRotZm8Jxmtv/Jao3heASk9U=', 'Content-Length': '0'} 2022-09-12T19:16:38.500 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000974b16cf28632d36-00631f8596-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.501 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.502 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.503 DEBUG:boto:Method: HEAD 2022-09-12T19:16:38.504 DEBUG:boto:Path: /jdpiak-39/ 2022-09-12T19:16:38.505 DEBUG:boto:Data: 2022-09-12T19:16:38.506 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.507 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:38.508 DEBUG:boto:Port: 8000 2022-09-12T19:16:38.509 DEBUG:boto:Params: {} 2022-09-12T19:16:38.510 DEBUG:boto:Token: None 2022-09-12T19:16:38.511 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/ 2022-09-12T19:16:38.513 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ASYsRotZm8Jxmtv/Jao3heASk9U= 2022-09-12T19:16:38.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ASYsRotZm8Jxmtv/Jao3heASk9U=', 'Content-Length': '0'} 2022-09-12T19:16:38.517 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002752a026a5d6aa42-00631f8596-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.518 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:38.518 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.519 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.520 DEBUG:boto:path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.521 DEBUG:boto:auth_path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.522 DEBUG:boto:Method: GET 2022-09-12T19:16:38.523 DEBUG:boto:Path: /jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.523 DEBUG:boto:Data: 2022-09-12T19:16:38.524 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.524 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:38.525 DEBUG:boto:Port: 8001 2022-09-12T19:16:38.525 DEBUG:boto:Params: {} 2022-09-12T19:16:38.526 DEBUG:boto:Token: None 2022-09-12T19:16:38.527 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/?versions 2022-09-12T19:16:38.528 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk= 2022-09-12T19:16:38.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk=', 'Content-Length': '0'} 2022-09-12T19:16:38.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe9be29dde0a32f8-00631f8596-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.532 DEBUG:boto:b'jdpiak-39999false' 2022-09-12T19:16:38.533 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:38.533 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.534 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.534 DEBUG:boto:path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.535 DEBUG:boto:auth_path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.535 DEBUG:boto:Method: GET 2022-09-12T19:16:38.536 DEBUG:boto:Path: /jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.536 DEBUG:boto:Data: 2022-09-12T19:16:38.537 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.537 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:38.538 DEBUG:boto:Port: 8000 2022-09-12T19:16:38.539 DEBUG:boto:Params: {} 2022-09-12T19:16:38.539 DEBUG:boto:Token: None 2022-09-12T19:16:38.540 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/?versions 2022-09-12T19:16:38.540 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk= 2022-09-12T19:16:38.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk=', 'Content-Length': '0'} 2022-09-12T19:16:38.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005d9425dddc34f2c-00631f8596-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.544 DEBUG:boto:b'jdpiak-39999false' 2022-09-12T19:16:38.545 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.545 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.546 DEBUG:boto:path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.547 DEBUG:boto:auth_path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.547 DEBUG:boto:Method: GET 2022-09-12T19:16:38.548 DEBUG:boto:Path: /jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.549 DEBUG:boto:Data: 2022-09-12T19:16:38.550 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.551 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:38.552 DEBUG:boto:Port: 8001 2022-09-12T19:16:38.552 DEBUG:boto:Params: {} 2022-09-12T19:16:38.553 DEBUG:boto:Token: None 2022-09-12T19:16:38.553 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/?versions 2022-09-12T19:16:38.554 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk= 2022-09-12T19:16:38.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk=', 'Content-Length': '0'} 2022-09-12T19:16:38.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001df89339dd319d28-00631f8596-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.558 DEBUG:boto:b'jdpiak-39999false' 2022-09-12T19:16:38.558 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.559 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.560 DEBUG:boto:path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.560 DEBUG:boto:auth_path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.561 DEBUG:boto:Method: GET 2022-09-12T19:16:38.561 DEBUG:boto:Path: /jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.562 DEBUG:boto:Data: 2022-09-12T19:16:38.562 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.563 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:38.563 DEBUG:boto:Port: 8000 2022-09-12T19:16:38.564 DEBUG:boto:Params: {} 2022-09-12T19:16:38.564 DEBUG:boto:Token: None 2022-09-12T19:16:38.565 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/?versions 2022-09-12T19:16:38.566 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk= 2022-09-12T19:16:38.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk=', 'Content-Length': '0'} 2022-09-12T19:16:38.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ab1200bda9acb17-00631f8596-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.614 DEBUG:boto:b'jdpiak-39999false' 2022-09-12T19:16:38.615 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-39 zones={test-zone3, test-zone1} 2022-09-12T19:16:38.616 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-39 zones={test-zone1, test-zone2} 2022-09-12T19:16:38.616 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.617 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.617 DEBUG:boto:Method: HEAD 2022-09-12T19:16:38.618 DEBUG:boto:Path: /jdpiak-39/ 2022-09-12T19:16:38.618 DEBUG:boto:Data: 2022-09-12T19:16:38.619 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.619 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:38.620 DEBUG:boto:Port: 8000 2022-09-12T19:16:38.620 DEBUG:boto:Params: {} 2022-09-12T19:16:38.621 DEBUG:boto:Token: None 2022-09-12T19:16:38.622 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/ 2022-09-12T19:16:38.622 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ASYsRotZm8Jxmtv/Jao3heASk9U= 2022-09-12T19:16:38.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ASYsRotZm8Jxmtv/Jao3heASk9U=', 'Content-Length': '0'} 2022-09-12T19:16:38.625 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c43170e9831579ba-00631f8596-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.626 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.627 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.627 DEBUG:boto:Method: HEAD 2022-09-12T19:16:38.628 DEBUG:boto:Path: /jdpiak-39/ 2022-09-12T19:16:38.628 DEBUG:boto:Data: 2022-09-12T19:16:38.629 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.629 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:38.630 DEBUG:boto:Port: 8000 2022-09-12T19:16:38.630 DEBUG:boto:Params: {} 2022-09-12T19:16:38.631 DEBUG:boto:Token: None 2022-09-12T19:16:38.631 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/ 2022-09-12T19:16:38.632 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ASYsRotZm8Jxmtv/Jao3heASk9U= 2022-09-12T19:16:38.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ASYsRotZm8Jxmtv/Jao3heASk9U=', 'Content-Length': '0'} 2022-09-12T19:16:38.635 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6377a0a4f8012c1-00631f8596-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.636 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:38.636 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.637 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.637 DEBUG:boto:path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.638 DEBUG:boto:auth_path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.638 DEBUG:boto:Method: GET 2022-09-12T19:16:38.639 DEBUG:boto:Path: /jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.639 DEBUG:boto:Data: 2022-09-12T19:16:38.640 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.640 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:38.641 DEBUG:boto:Port: 8000 2022-09-12T19:16:38.641 DEBUG:boto:Params: {} 2022-09-12T19:16:38.642 DEBUG:boto:Token: None 2022-09-12T19:16:38.643 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/?versions 2022-09-12T19:16:38.643 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk= 2022-09-12T19:16:38.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk=', 'Content-Length': '0'} 2022-09-12T19:16:38.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ed4dd6636d38258-00631f8596-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.647 DEBUG:boto:b'jdpiak-39999false' 2022-09-12T19:16:38.648 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:38.649 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.649 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.650 DEBUG:boto:path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.651 DEBUG:boto:auth_path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.652 DEBUG:boto:Method: GET 2022-09-12T19:16:38.653 DEBUG:boto:Path: /jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.653 DEBUG:boto:Data: 2022-09-12T19:16:38.654 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.655 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:38.656 DEBUG:boto:Port: 8000 2022-09-12T19:16:38.656 DEBUG:boto:Params: {} 2022-09-12T19:16:38.657 DEBUG:boto:Token: None 2022-09-12T19:16:38.658 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/?versions 2022-09-12T19:16:38.658 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk= 2022-09-12T19:16:38.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk=', 'Content-Length': '0'} 2022-09-12T19:16:38.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000453a6b14e7afbf07-00631f8596-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.662 DEBUG:boto:b'jdpiak-39999false' 2022-09-12T19:16:38.663 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.664 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.664 DEBUG:boto:path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.665 DEBUG:boto:auth_path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.665 DEBUG:boto:Method: GET 2022-09-12T19:16:38.666 DEBUG:boto:Path: /jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.667 DEBUG:boto:Data: 2022-09-12T19:16:38.667 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.668 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:38.668 DEBUG:boto:Port: 8000 2022-09-12T19:16:38.669 DEBUG:boto:Params: {} 2022-09-12T19:16:38.669 DEBUG:boto:Token: None 2022-09-12T19:16:38.670 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/?versions 2022-09-12T19:16:38.671 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk= 2022-09-12T19:16:38.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk=', 'Content-Length': '0'} 2022-09-12T19:16:38.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5d820c6a9431cd5-00631f8596-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.675 DEBUG:boto:b'jdpiak-39999false' 2022-09-12T19:16:38.676 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.677 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.677 DEBUG:boto:path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.678 DEBUG:boto:auth_path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.679 DEBUG:boto:Method: GET 2022-09-12T19:16:38.679 DEBUG:boto:Path: /jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.680 DEBUG:boto:Data: 2022-09-12T19:16:38.681 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.681 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:38.682 DEBUG:boto:Port: 8000 2022-09-12T19:16:38.682 DEBUG:boto:Params: {} 2022-09-12T19:16:38.683 DEBUG:boto:Token: None 2022-09-12T19:16:38.684 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/?versions 2022-09-12T19:16:38.684 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk= 2022-09-12T19:16:38.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk=', 'Content-Length': '0'} 2022-09-12T19:16:38.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077f0ce9665547e46-00631f8596-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.688 DEBUG:boto:b'jdpiak-39999false' 2022-09-12T19:16:38.689 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-39 zones={test-zone1, test-zone2} 2022-09-12T19:16:38.690 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-39 zones={test-zone3, test-zone2} 2022-09-12T19:16:38.691 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.692 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.693 DEBUG:boto:Method: HEAD 2022-09-12T19:16:38.693 DEBUG:boto:Path: /jdpiak-39/ 2022-09-12T19:16:38.694 DEBUG:boto:Data: 2022-09-12T19:16:38.695 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.696 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:38.697 DEBUG:boto:Port: 8001 2022-09-12T19:16:38.698 DEBUG:boto:Params: {} 2022-09-12T19:16:38.699 DEBUG:boto:Token: None 2022-09-12T19:16:38.700 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/ 2022-09-12T19:16:38.701 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ASYsRotZm8Jxmtv/Jao3heASk9U= 2022-09-12T19:16:38.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ASYsRotZm8Jxmtv/Jao3heASk9U=', 'Content-Length': '0'} 2022-09-12T19:16:38.705 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084e25a4d3da5d51a-00631f8596-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.705 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.706 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.706 DEBUG:boto:Method: HEAD 2022-09-12T19:16:38.707 DEBUG:boto:Path: /jdpiak-39/ 2022-09-12T19:16:38.708 DEBUG:boto:Data: 2022-09-12T19:16:38.708 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.709 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:38.709 DEBUG:boto:Port: 8000 2022-09-12T19:16:38.710 DEBUG:boto:Params: {} 2022-09-12T19:16:38.711 DEBUG:boto:Token: None 2022-09-12T19:16:38.711 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/ 2022-09-12T19:16:38.712 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ASYsRotZm8Jxmtv/Jao3heASk9U= 2022-09-12T19:16:38.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ASYsRotZm8Jxmtv/Jao3heASk9U=', 'Content-Length': '0'} 2022-09-12T19:16:38.715 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ab4d880e3124dfc-00631f8596-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.716 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:38.716 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.717 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.717 DEBUG:boto:path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.718 DEBUG:boto:auth_path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.718 DEBUG:boto:Method: GET 2022-09-12T19:16:38.719 DEBUG:boto:Path: /jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.719 DEBUG:boto:Data: 2022-09-12T19:16:38.720 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.720 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:38.721 DEBUG:boto:Port: 8001 2022-09-12T19:16:38.721 DEBUG:boto:Params: {} 2022-09-12T19:16:38.722 DEBUG:boto:Token: None 2022-09-12T19:16:38.723 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/?versions 2022-09-12T19:16:38.723 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk= 2022-09-12T19:16:38.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk=', 'Content-Length': '0'} 2022-09-12T19:16:38.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc68726ce8c1a8f4-00631f8596-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.731 DEBUG:boto:b'jdpiak-39999false' 2022-09-12T19:16:38.731 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:38.732 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.733 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.733 DEBUG:boto:path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.734 DEBUG:boto:auth_path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.734 DEBUG:boto:Method: GET 2022-09-12T19:16:38.735 DEBUG:boto:Path: /jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.735 DEBUG:boto:Data: 2022-09-12T19:16:38.736 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.736 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:38.737 DEBUG:boto:Port: 8000 2022-09-12T19:16:38.737 DEBUG:boto:Params: {} 2022-09-12T19:16:38.738 DEBUG:boto:Token: None 2022-09-12T19:16:38.738 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/?versions 2022-09-12T19:16:38.739 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk= 2022-09-12T19:16:38.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk=', 'Content-Length': '0'} 2022-09-12T19:16:38.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061f118cd2f63bef6-00631f8596-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.790 DEBUG:boto:b'jdpiak-39999false' 2022-09-12T19:16:38.791 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.792 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.793 DEBUG:boto:path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.794 DEBUG:boto:auth_path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.795 DEBUG:boto:Method: GET 2022-09-12T19:16:38.796 DEBUG:boto:Path: /jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.797 DEBUG:boto:Data: 2022-09-12T19:16:38.798 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.799 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:38.800 DEBUG:boto:Port: 8001 2022-09-12T19:16:38.800 DEBUG:boto:Params: {} 2022-09-12T19:16:38.801 DEBUG:boto:Token: None 2022-09-12T19:16:38.802 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/?versions 2022-09-12T19:16:38.803 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk= 2022-09-12T19:16:38.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk=', 'Content-Length': '0'} 2022-09-12T19:16:38.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000972725c0b9a51360-00631f8596-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.809 DEBUG:boto:b'jdpiak-39999false' 2022-09-12T19:16:38.810 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.811 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.812 DEBUG:boto:path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.812 DEBUG:boto:auth_path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.813 DEBUG:boto:Method: GET 2022-09-12T19:16:38.814 DEBUG:boto:Path: /jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.815 DEBUG:boto:Data: 2022-09-12T19:16:38.816 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.816 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:38.817 DEBUG:boto:Port: 8000 2022-09-12T19:16:38.817 DEBUG:boto:Params: {} 2022-09-12T19:16:38.818 DEBUG:boto:Token: None 2022-09-12T19:16:38.818 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/?versions 2022-09-12T19:16:38.819 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk= 2022-09-12T19:16:38.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk=', 'Content-Length': '0'} 2022-09-12T19:16:38.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000894a829cd0622f1a-00631f8596-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.823 DEBUG:boto:b'jdpiak-39999false' 2022-09-12T19:16:38.824 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-39 zones={test-zone3, test-zone2} 2022-09-12T19:16:38.824 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-39 zones={test-zone1, test-zone3} 2022-09-12T19:16:38.825 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.826 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.826 DEBUG:boto:Method: HEAD 2022-09-12T19:16:38.827 DEBUG:boto:Path: /jdpiak-39/ 2022-09-12T19:16:38.827 DEBUG:boto:Data: 2022-09-12T19:16:38.828 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.828 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:38.829 DEBUG:boto:Port: 8000 2022-09-12T19:16:38.829 DEBUG:boto:Params: {} 2022-09-12T19:16:38.830 DEBUG:boto:Token: None 2022-09-12T19:16:38.830 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/ 2022-09-12T19:16:38.831 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ASYsRotZm8Jxmtv/Jao3heASk9U= 2022-09-12T19:16:38.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ASYsRotZm8Jxmtv/Jao3heASk9U=', 'Content-Length': '0'} 2022-09-12T19:16:38.834 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c45d1c49e86723a6-00631f8596-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.835 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.835 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.836 DEBUG:boto:Method: HEAD 2022-09-12T19:16:38.836 DEBUG:boto:Path: /jdpiak-39/ 2022-09-12T19:16:38.837 DEBUG:boto:Data: 2022-09-12T19:16:38.837 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.838 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:38.838 DEBUG:boto:Port: 8001 2022-09-12T19:16:38.839 DEBUG:boto:Params: {} 2022-09-12T19:16:38.839 DEBUG:boto:Token: None 2022-09-12T19:16:38.840 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/ 2022-09-12T19:16:38.841 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ASYsRotZm8Jxmtv/Jao3heASk9U= 2022-09-12T19:16:38.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ASYsRotZm8Jxmtv/Jao3heASk9U=', 'Content-Length': '0'} 2022-09-12T19:16:38.844 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be466b230930f320-00631f8596-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.844 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:38.845 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.846 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.846 DEBUG:boto:path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.847 DEBUG:boto:auth_path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.847 DEBUG:boto:Method: GET 2022-09-12T19:16:38.848 DEBUG:boto:Path: /jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.848 DEBUG:boto:Data: 2022-09-12T19:16:38.849 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.849 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:38.850 DEBUG:boto:Port: 8000 2022-09-12T19:16:38.850 DEBUG:boto:Params: {} 2022-09-12T19:16:38.851 DEBUG:boto:Token: None 2022-09-12T19:16:38.852 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/?versions 2022-09-12T19:16:38.853 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk= 2022-09-12T19:16:38.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk=', 'Content-Length': '0'} 2022-09-12T19:16:38.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013bd56cc1825d0dd-00631f8596-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.898 DEBUG:boto:b'jdpiak-39999false' 2022-09-12T19:16:38.899 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:38.900 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.901 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.901 DEBUG:boto:path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.902 DEBUG:boto:auth_path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.903 DEBUG:boto:Method: GET 2022-09-12T19:16:38.904 DEBUG:boto:Path: /jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.905 DEBUG:boto:Data: 2022-09-12T19:16:38.906 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.906 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:38.907 DEBUG:boto:Port: 8001 2022-09-12T19:16:38.907 DEBUG:boto:Params: {} 2022-09-12T19:16:38.908 DEBUG:boto:Token: None 2022-09-12T19:16:38.908 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/?versions 2022-09-12T19:16:38.909 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk= 2022-09-12T19:16:38.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk=', 'Content-Length': '0'} 2022-09-12T19:16:38.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b5e24f676232d40-00631f8596-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.914 DEBUG:boto:b'jdpiak-39999false' 2022-09-12T19:16:38.915 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.916 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.916 DEBUG:boto:path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.917 DEBUG:boto:auth_path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.917 DEBUG:boto:Method: GET 2022-09-12T19:16:38.918 DEBUG:boto:Path: /jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.918 DEBUG:boto:Data: 2022-09-12T19:16:38.919 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.920 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:38.920 DEBUG:boto:Port: 8000 2022-09-12T19:16:38.921 DEBUG:boto:Params: {} 2022-09-12T19:16:38.922 DEBUG:boto:Token: None 2022-09-12T19:16:38.923 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/?versions 2022-09-12T19:16:38.923 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk= 2022-09-12T19:16:38.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk=', 'Content-Length': '0'} 2022-09-12T19:16:38.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe9193e594c749be-00631f8596-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.927 DEBUG:boto:b'jdpiak-39999false' 2022-09-12T19:16:38.928 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.929 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.929 DEBUG:boto:path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.930 DEBUG:boto:auth_path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.931 DEBUG:boto:Method: GET 2022-09-12T19:16:38.931 DEBUG:boto:Path: /jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.932 DEBUG:boto:Data: 2022-09-12T19:16:38.932 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.933 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:38.934 DEBUG:boto:Port: 8001 2022-09-12T19:16:38.934 DEBUG:boto:Params: {} 2022-09-12T19:16:38.935 DEBUG:boto:Token: None 2022-09-12T19:16:38.935 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/?versions 2022-09-12T19:16:38.936 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk= 2022-09-12T19:16:38.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk=', 'Content-Length': '0'} 2022-09-12T19:16:38.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008488ee8de1a9b0e3-00631f8596-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.940 DEBUG:boto:b'jdpiak-39999false' 2022-09-12T19:16:38.941 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-39 zones={test-zone1, test-zone3} 2022-09-12T19:16:38.942 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-39 zones={test-zone2, test-zone3} 2022-09-12T19:16:38.942 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.943 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.943 DEBUG:boto:Method: HEAD 2022-09-12T19:16:38.944 DEBUG:boto:Path: /jdpiak-39/ 2022-09-12T19:16:38.944 DEBUG:boto:Data: 2022-09-12T19:16:38.945 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.945 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:38.946 DEBUG:boto:Port: 8000 2022-09-12T19:16:38.946 DEBUG:boto:Params: {} 2022-09-12T19:16:38.947 DEBUG:boto:Token: None 2022-09-12T19:16:38.947 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/ 2022-09-12T19:16:38.948 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ASYsRotZm8Jxmtv/Jao3heASk9U= 2022-09-12T19:16:38.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ASYsRotZm8Jxmtv/Jao3heASk9U=', 'Content-Length': '0'} 2022-09-12T19:16:38.952 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005505d0e10d92e1d8-00631f8596-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.952 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.953 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.953 DEBUG:boto:Method: HEAD 2022-09-12T19:16:38.954 DEBUG:boto:Path: /jdpiak-39/ 2022-09-12T19:16:38.954 DEBUG:boto:Data: 2022-09-12T19:16:38.955 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.955 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:38.956 DEBUG:boto:Port: 8001 2022-09-12T19:16:38.956 DEBUG:boto:Params: {} 2022-09-12T19:16:38.957 DEBUG:boto:Token: None 2022-09-12T19:16:38.957 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/ 2022-09-12T19:16:38.958 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ASYsRotZm8Jxmtv/Jao3heASk9U= 2022-09-12T19:16:38.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ASYsRotZm8Jxmtv/Jao3heASk9U=', 'Content-Length': '0'} 2022-09-12T19:16:38.961 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000296ff629c555124b-00631f8596-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.962 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:38.962 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.963 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.963 DEBUG:boto:path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.964 DEBUG:boto:auth_path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.964 DEBUG:boto:Method: GET 2022-09-12T19:16:38.965 DEBUG:boto:Path: /jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.965 DEBUG:boto:Data: 2022-09-12T19:16:38.966 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.966 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:38.967 DEBUG:boto:Port: 8000 2022-09-12T19:16:38.967 DEBUG:boto:Params: {} 2022-09-12T19:16:38.968 DEBUG:boto:Token: None 2022-09-12T19:16:38.968 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/?versions 2022-09-12T19:16:38.969 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk= 2022-09-12T19:16:38.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk=', 'Content-Length': '0'} 2022-09-12T19:16:38.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005a8f0bb9605d027-00631f8596-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.973 DEBUG:boto:b'jdpiak-39999false' 2022-09-12T19:16:38.974 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:38.974 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.975 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.975 DEBUG:boto:path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.976 DEBUG:boto:auth_path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.976 DEBUG:boto:Method: GET 2022-09-12T19:16:38.977 DEBUG:boto:Path: /jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.977 DEBUG:boto:Data: 2022-09-12T19:16:38.978 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.978 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:38.979 DEBUG:boto:Port: 8001 2022-09-12T19:16:38.979 DEBUG:boto:Params: {} 2022-09-12T19:16:38.980 DEBUG:boto:Token: None 2022-09-12T19:16:38.980 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/?versions 2022-09-12T19:16:38.981 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk= 2022-09-12T19:16:38.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk=', 'Content-Length': '0'} 2022-09-12T19:16:38.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c975db5a0b53d526-00631f8596-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.985 DEBUG:boto:b'jdpiak-39999false' 2022-09-12T19:16:38.986 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.986 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.987 DEBUG:boto:path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.987 DEBUG:boto:auth_path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.988 DEBUG:boto:Method: GET 2022-09-12T19:16:38.988 DEBUG:boto:Path: /jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.989 DEBUG:boto:Data: 2022-09-12T19:16:38.989 DEBUG:boto:Headers: {} 2022-09-12T19:16:38.990 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:38.990 DEBUG:boto:Port: 8000 2022-09-12T19:16:38.991 DEBUG:boto:Params: {} 2022-09-12T19:16:38.991 DEBUG:boto:Token: None 2022-09-12T19:16:38.992 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:38 GMT /jdpiak-39/?versions 2022-09-12T19:16:38.992 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk= 2022-09-12T19:16:38.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:exVso2FAIxgOvGnQYVd79Mn/pDk=', 'Content-Length': '0'} 2022-09-12T19:16:38.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095c26bca475e2d31-00631f8596-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:38.996 DEBUG:boto:b'jdpiak-39999false' 2022-09-12T19:16:38.997 DEBUG:boto:path=/jdpiak-39/ 2022-09-12T19:16:38.997 DEBUG:boto:auth_path=/jdpiak-39/ 2022-09-12T19:16:38.998 DEBUG:boto:path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.998 DEBUG:boto:auth_path=/jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:38.999 DEBUG:boto:Method: GET 2022-09-12T19:16:38.999 DEBUG:boto:Path: /jdpiak-39/?versions&max-keys=999 2022-09-12T19:16:39.000 DEBUG:boto:Data: 2022-09-12T19:16:39.000 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.001 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:39.001 DEBUG:boto:Port: 8001 2022-09-12T19:16:39.002 DEBUG:boto:Params: {} 2022-09-12T19:16:39.002 DEBUG:boto:Token: None 2022-09-12T19:16:39.003 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-39/?versions 2022-09-12T19:16:39.004 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XqvBQrquDzpvs8OXP6GAC7VS9kw= 2022-09-12T19:16:39.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XqvBQrquDzpvs8OXP6GAC7VS9kw=', 'Content-Length': '0'} 2022-09-12T19:16:39.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d9bdbcc3088432b-00631f8597-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.008 DEBUG:boto:b'jdpiak-39999false' 2022-09-12T19:16:39.008 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-39 zones={test-zone2, test-zone3} 2022-09-12T19:16:39.009 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-40 zones={test-zone2, test-zone1} 2022-09-12T19:16:39.010 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.010 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.011 DEBUG:boto:Method: HEAD 2022-09-12T19:16:39.011 DEBUG:boto:Path: /jdpiak-40/ 2022-09-12T19:16:39.012 DEBUG:boto:Data: 2022-09-12T19:16:39.013 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.013 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:39.014 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.015 DEBUG:boto:Params: {} 2022-09-12T19:16:39.016 DEBUG:boto:Token: None 2022-09-12T19:16:39.017 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/ 2022-09-12T19:16:39.018 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AwMK/lk1Rj+iaU4QQbmXX9yRK4w= 2022-09-12T19:16:39.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AwMK/lk1Rj+iaU4QQbmXX9yRK4w=', 'Content-Length': '0'} 2022-09-12T19:16:39.022 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006027928d8b707046-00631f8597-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.023 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.024 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.024 DEBUG:boto:Method: HEAD 2022-09-12T19:16:39.025 DEBUG:boto:Path: /jdpiak-40/ 2022-09-12T19:16:39.025 DEBUG:boto:Data: 2022-09-12T19:16:39.026 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.026 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:39.027 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.027 DEBUG:boto:Params: {} 2022-09-12T19:16:39.028 DEBUG:boto:Token: None 2022-09-12T19:16:39.028 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/ 2022-09-12T19:16:39.029 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AwMK/lk1Rj+iaU4QQbmXX9yRK4w= 2022-09-12T19:16:39.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AwMK/lk1Rj+iaU4QQbmXX9yRK4w=', 'Content-Length': '0'} 2022-09-12T19:16:39.033 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d46013580daf09c4-00631f8597-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.034 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:39.035 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.036 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.037 DEBUG:boto:path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.037 DEBUG:boto:auth_path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.038 DEBUG:boto:Method: GET 2022-09-12T19:16:39.039 DEBUG:boto:Path: /jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.040 DEBUG:boto:Data: 2022-09-12T19:16:39.041 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.042 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:39.043 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.044 DEBUG:boto:Params: {} 2022-09-12T19:16:39.044 DEBUG:boto:Token: None 2022-09-12T19:16:39.045 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/?versions 2022-09-12T19:16:39.045 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c= 2022-09-12T19:16:39.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c=', 'Content-Length': '0'} 2022-09-12T19:16:39.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005be9a4ab20d0093-00631f8597-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.049 DEBUG:boto:b'jdpiak-40999false' 2022-09-12T19:16:39.050 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:39.051 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.051 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.052 DEBUG:boto:path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.052 DEBUG:boto:auth_path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.053 DEBUG:boto:Method: GET 2022-09-12T19:16:39.053 DEBUG:boto:Path: /jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.054 DEBUG:boto:Data: 2022-09-12T19:16:39.054 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.055 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:39.055 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.056 DEBUG:boto:Params: {} 2022-09-12T19:16:39.056 DEBUG:boto:Token: None 2022-09-12T19:16:39.057 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/?versions 2022-09-12T19:16:39.058 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c= 2022-09-12T19:16:39.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c=', 'Content-Length': '0'} 2022-09-12T19:16:39.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095d634dfecfc4dfc-00631f8597-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.062 DEBUG:boto:b'jdpiak-40999false' 2022-09-12T19:16:39.063 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.063 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.064 DEBUG:boto:path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.064 DEBUG:boto:auth_path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.065 DEBUG:boto:Method: GET 2022-09-12T19:16:39.065 DEBUG:boto:Path: /jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.066 DEBUG:boto:Data: 2022-09-12T19:16:39.066 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.067 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:39.067 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.068 DEBUG:boto:Params: {} 2022-09-12T19:16:39.068 DEBUG:boto:Token: None 2022-09-12T19:16:39.069 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/?versions 2022-09-12T19:16:39.069 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c= 2022-09-12T19:16:39.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c=', 'Content-Length': '0'} 2022-09-12T19:16:39.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd0642a557edca03-00631f8597-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.073 DEBUG:boto:b'jdpiak-40999false' 2022-09-12T19:16:39.074 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.075 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.075 DEBUG:boto:path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.076 DEBUG:boto:auth_path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.076 DEBUG:boto:Method: GET 2022-09-12T19:16:39.077 DEBUG:boto:Path: /jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.078 DEBUG:boto:Data: 2022-09-12T19:16:39.079 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.079 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:39.080 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.080 DEBUG:boto:Params: {} 2022-09-12T19:16:39.081 DEBUG:boto:Token: None 2022-09-12T19:16:39.081 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/?versions 2022-09-12T19:16:39.082 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c= 2022-09-12T19:16:39.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c=', 'Content-Length': '0'} 2022-09-12T19:16:39.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091fddb9a9ac3eb73-00631f8597-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.086 DEBUG:boto:b'jdpiak-40999false' 2022-09-12T19:16:39.086 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-40 zones={test-zone2, test-zone1} 2022-09-12T19:16:39.087 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-40 zones={test-zone3, test-zone1} 2022-09-12T19:16:39.088 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.088 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.089 DEBUG:boto:Method: HEAD 2022-09-12T19:16:39.090 DEBUG:boto:Path: /jdpiak-40/ 2022-09-12T19:16:39.090 DEBUG:boto:Data: 2022-09-12T19:16:39.091 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.091 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:39.092 DEBUG:boto:Port: 8001 2022-09-12T19:16:39.092 DEBUG:boto:Params: {} 2022-09-12T19:16:39.093 DEBUG:boto:Token: None 2022-09-12T19:16:39.093 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/ 2022-09-12T19:16:39.094 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AwMK/lk1Rj+iaU4QQbmXX9yRK4w= 2022-09-12T19:16:39.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AwMK/lk1Rj+iaU4QQbmXX9yRK4w=', 'Content-Length': '0'} 2022-09-12T19:16:39.098 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b305ac0d26b12793-00631f8597-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.099 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.099 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.100 DEBUG:boto:Method: HEAD 2022-09-12T19:16:39.101 DEBUG:boto:Path: /jdpiak-40/ 2022-09-12T19:16:39.101 DEBUG:boto:Data: 2022-09-12T19:16:39.102 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.102 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:39.103 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.103 DEBUG:boto:Params: {} 2022-09-12T19:16:39.104 DEBUG:boto:Token: None 2022-09-12T19:16:39.104 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/ 2022-09-12T19:16:39.105 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AwMK/lk1Rj+iaU4QQbmXX9yRK4w= 2022-09-12T19:16:39.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AwMK/lk1Rj+iaU4QQbmXX9yRK4w=', 'Content-Length': '0'} 2022-09-12T19:16:39.108 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ecbc0eb9a9e8f315-00631f8597-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.109 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:39.109 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.110 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.110 DEBUG:boto:path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.111 DEBUG:boto:auth_path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.112 DEBUG:boto:Method: GET 2022-09-12T19:16:39.113 DEBUG:boto:Path: /jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.114 DEBUG:boto:Data: 2022-09-12T19:16:39.114 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.115 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:39.116 DEBUG:boto:Port: 8001 2022-09-12T19:16:39.117 DEBUG:boto:Params: {} 2022-09-12T19:16:39.117 DEBUG:boto:Token: None 2022-09-12T19:16:39.118 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/?versions 2022-09-12T19:16:39.119 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c= 2022-09-12T19:16:39.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c=', 'Content-Length': '0'} 2022-09-12T19:16:39.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c60c4fe38c99903-00631f8597-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.124 DEBUG:boto:b'jdpiak-40999false' 2022-09-12T19:16:39.124 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:39.125 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.126 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.126 DEBUG:boto:path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.127 DEBUG:boto:auth_path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.127 DEBUG:boto:Method: GET 2022-09-12T19:16:39.128 DEBUG:boto:Path: /jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.128 DEBUG:boto:Data: 2022-09-12T19:16:39.129 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.129 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:39.130 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.130 DEBUG:boto:Params: {} 2022-09-12T19:16:39.131 DEBUG:boto:Token: None 2022-09-12T19:16:39.131 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/?versions 2022-09-12T19:16:39.132 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c= 2022-09-12T19:16:39.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c=', 'Content-Length': '0'} 2022-09-12T19:16:39.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e224bd5fab8a5634-00631f8597-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.138 DEBUG:boto:b'jdpiak-40999false' 2022-09-12T19:16:39.139 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.139 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.140 DEBUG:boto:path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.141 DEBUG:boto:auth_path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.142 DEBUG:boto:Method: GET 2022-09-12T19:16:39.142 DEBUG:boto:Path: /jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.143 DEBUG:boto:Data: 2022-09-12T19:16:39.143 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.144 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:39.144 DEBUG:boto:Port: 8001 2022-09-12T19:16:39.145 DEBUG:boto:Params: {} 2022-09-12T19:16:39.146 DEBUG:boto:Token: None 2022-09-12T19:16:39.146 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/?versions 2022-09-12T19:16:39.147 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c= 2022-09-12T19:16:39.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c=', 'Content-Length': '0'} 2022-09-12T19:16:39.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af9e7c925d91bb2d-00631f8597-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.152 DEBUG:boto:b'jdpiak-40999false' 2022-09-12T19:16:39.153 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.154 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.154 DEBUG:boto:path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.155 DEBUG:boto:auth_path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.156 DEBUG:boto:Method: GET 2022-09-12T19:16:39.156 DEBUG:boto:Path: /jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.157 DEBUG:boto:Data: 2022-09-12T19:16:39.158 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.159 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:39.160 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.161 DEBUG:boto:Params: {} 2022-09-12T19:16:39.162 DEBUG:boto:Token: None 2022-09-12T19:16:39.163 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/?versions 2022-09-12T19:16:39.163 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c= 2022-09-12T19:16:39.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c=', 'Content-Length': '0'} 2022-09-12T19:16:39.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000346e7601ae51dba8-00631f8597-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.167 DEBUG:boto:b'jdpiak-40999false' 2022-09-12T19:16:39.169 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-40 zones={test-zone3, test-zone1} 2022-09-12T19:16:39.169 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-40 zones={test-zone1, test-zone2} 2022-09-12T19:16:39.170 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.171 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.171 DEBUG:boto:Method: HEAD 2022-09-12T19:16:39.172 DEBUG:boto:Path: /jdpiak-40/ 2022-09-12T19:16:39.172 DEBUG:boto:Data: 2022-09-12T19:16:39.173 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.173 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:39.174 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.174 DEBUG:boto:Params: {} 2022-09-12T19:16:39.175 DEBUG:boto:Token: None 2022-09-12T19:16:39.175 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/ 2022-09-12T19:16:39.176 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AwMK/lk1Rj+iaU4QQbmXX9yRK4w= 2022-09-12T19:16:39.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AwMK/lk1Rj+iaU4QQbmXX9yRK4w=', 'Content-Length': '0'} 2022-09-12T19:16:39.179 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065d000205e3219ec-00631f8597-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.180 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.180 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.181 DEBUG:boto:Method: HEAD 2022-09-12T19:16:39.181 DEBUG:boto:Path: /jdpiak-40/ 2022-09-12T19:16:39.182 DEBUG:boto:Data: 2022-09-12T19:16:39.182 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.183 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:39.183 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.184 DEBUG:boto:Params: {} 2022-09-12T19:16:39.184 DEBUG:boto:Token: None 2022-09-12T19:16:39.185 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/ 2022-09-12T19:16:39.186 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AwMK/lk1Rj+iaU4QQbmXX9yRK4w= 2022-09-12T19:16:39.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AwMK/lk1Rj+iaU4QQbmXX9yRK4w=', 'Content-Length': '0'} 2022-09-12T19:16:39.189 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5b664e23c5b867e-00631f8597-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.189 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:39.190 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.190 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.191 DEBUG:boto:path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.191 DEBUG:boto:auth_path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.192 DEBUG:boto:Method: GET 2022-09-12T19:16:39.192 DEBUG:boto:Path: /jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.193 DEBUG:boto:Data: 2022-09-12T19:16:39.193 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.194 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:39.194 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.205 DEBUG:boto:Params: {} 2022-09-12T19:16:39.206 DEBUG:boto:Token: None 2022-09-12T19:16:39.206 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/?versions 2022-09-12T19:16:39.207 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c= 2022-09-12T19:16:39.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c=', 'Content-Length': '0'} 2022-09-12T19:16:39.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071ed255448a8c507-00631f8597-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.258 DEBUG:boto:b'jdpiak-40999false' 2022-09-12T19:16:39.259 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:39.260 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.260 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.261 DEBUG:boto:path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.261 DEBUG:boto:auth_path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.262 DEBUG:boto:Method: GET 2022-09-12T19:16:39.262 DEBUG:boto:Path: /jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.263 DEBUG:boto:Data: 2022-09-12T19:16:39.263 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.264 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:39.264 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.265 DEBUG:boto:Params: {} 2022-09-12T19:16:39.266 DEBUG:boto:Token: None 2022-09-12T19:16:39.266 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/?versions 2022-09-12T19:16:39.267 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c= 2022-09-12T19:16:39.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c=', 'Content-Length': '0'} 2022-09-12T19:16:39.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d70c3b5debedb03e-00631f8597-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.271 DEBUG:boto:b'jdpiak-40999false' 2022-09-12T19:16:39.272 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.272 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.273 DEBUG:boto:path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.273 DEBUG:boto:auth_path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.274 DEBUG:boto:Method: GET 2022-09-12T19:16:39.274 DEBUG:boto:Path: /jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.275 DEBUG:boto:Data: 2022-09-12T19:16:39.275 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.276 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:39.276 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.277 DEBUG:boto:Params: {} 2022-09-12T19:16:39.277 DEBUG:boto:Token: None 2022-09-12T19:16:39.278 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/?versions 2022-09-12T19:16:39.279 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c= 2022-09-12T19:16:39.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c=', 'Content-Length': '0'} 2022-09-12T19:16:39.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000283ebc07bc5add3e-00631f8597-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.287 DEBUG:boto:b'jdpiak-40999false' 2022-09-12T19:16:39.288 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.289 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.290 DEBUG:boto:path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.291 DEBUG:boto:auth_path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.291 DEBUG:boto:Method: GET 2022-09-12T19:16:39.292 DEBUG:boto:Path: /jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.292 DEBUG:boto:Data: 2022-09-12T19:16:39.293 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.293 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:39.294 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.294 DEBUG:boto:Params: {} 2022-09-12T19:16:39.295 DEBUG:boto:Token: None 2022-09-12T19:16:39.296 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/?versions 2022-09-12T19:16:39.296 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c= 2022-09-12T19:16:39.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c=', 'Content-Length': '0'} 2022-09-12T19:16:39.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b1effc5b3d9a54c-00631f8597-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.342 DEBUG:boto:b'jdpiak-40999false' 2022-09-12T19:16:39.343 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-40 zones={test-zone1, test-zone2} 2022-09-12T19:16:39.344 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-40 zones={test-zone3, test-zone2} 2022-09-12T19:16:39.344 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.345 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.345 DEBUG:boto:Method: HEAD 2022-09-12T19:16:39.346 DEBUG:boto:Path: /jdpiak-40/ 2022-09-12T19:16:39.346 DEBUG:boto:Data: 2022-09-12T19:16:39.347 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.348 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:39.349 DEBUG:boto:Port: 8001 2022-09-12T19:16:39.349 DEBUG:boto:Params: {} 2022-09-12T19:16:39.350 DEBUG:boto:Token: None 2022-09-12T19:16:39.350 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/ 2022-09-12T19:16:39.351 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AwMK/lk1Rj+iaU4QQbmXX9yRK4w= 2022-09-12T19:16:39.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AwMK/lk1Rj+iaU4QQbmXX9yRK4w=', 'Content-Length': '0'} 2022-09-12T19:16:39.355 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3aa4606f1104671-00631f8597-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.356 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.357 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.358 DEBUG:boto:Method: HEAD 2022-09-12T19:16:39.359 DEBUG:boto:Path: /jdpiak-40/ 2022-09-12T19:16:39.360 DEBUG:boto:Data: 2022-09-12T19:16:39.361 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.361 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:39.362 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.363 DEBUG:boto:Params: {} 2022-09-12T19:16:39.363 DEBUG:boto:Token: None 2022-09-12T19:16:39.364 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/ 2022-09-12T19:16:39.364 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AwMK/lk1Rj+iaU4QQbmXX9yRK4w= 2022-09-12T19:16:39.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AwMK/lk1Rj+iaU4QQbmXX9yRK4w=', 'Content-Length': '0'} 2022-09-12T19:16:39.368 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf3ac09d578f175e-00631f8597-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.369 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:39.369 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.370 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.370 DEBUG:boto:path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.371 DEBUG:boto:auth_path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.371 DEBUG:boto:Method: GET 2022-09-12T19:16:39.372 DEBUG:boto:Path: /jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.372 DEBUG:boto:Data: 2022-09-12T19:16:39.373 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.373 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:39.374 DEBUG:boto:Port: 8001 2022-09-12T19:16:39.374 DEBUG:boto:Params: {} 2022-09-12T19:16:39.375 DEBUG:boto:Token: None 2022-09-12T19:16:39.375 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/?versions 2022-09-12T19:16:39.376 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c= 2022-09-12T19:16:39.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c=', 'Content-Length': '0'} 2022-09-12T19:16:39.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff4557b645f1963b-00631f8597-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.426 DEBUG:boto:b'jdpiak-40999false' 2022-09-12T19:16:39.427 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:39.428 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.429 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.429 DEBUG:boto:path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.430 DEBUG:boto:auth_path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.430 DEBUG:boto:Method: GET 2022-09-12T19:16:39.431 DEBUG:boto:Path: /jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.432 DEBUG:boto:Data: 2022-09-12T19:16:39.432 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.433 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:39.433 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.434 DEBUG:boto:Params: {} 2022-09-12T19:16:39.435 DEBUG:boto:Token: None 2022-09-12T19:16:39.435 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/?versions 2022-09-12T19:16:39.436 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c= 2022-09-12T19:16:39.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c=', 'Content-Length': '0'} 2022-09-12T19:16:39.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075c645df81135227-00631f8597-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.444 DEBUG:boto:b'jdpiak-40999false' 2022-09-12T19:16:39.445 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.446 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.446 DEBUG:boto:path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.447 DEBUG:boto:auth_path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.448 DEBUG:boto:Method: GET 2022-09-12T19:16:39.449 DEBUG:boto:Path: /jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.450 DEBUG:boto:Data: 2022-09-12T19:16:39.451 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.452 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:39.453 DEBUG:boto:Port: 8001 2022-09-12T19:16:39.454 DEBUG:boto:Params: {} 2022-09-12T19:16:39.455 DEBUG:boto:Token: None 2022-09-12T19:16:39.456 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/?versions 2022-09-12T19:16:39.457 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c= 2022-09-12T19:16:39.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c=', 'Content-Length': '0'} 2022-09-12T19:16:39.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4b67f579fed5c58-00631f8597-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.462 DEBUG:boto:b'jdpiak-40999false' 2022-09-12T19:16:39.463 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.464 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.465 DEBUG:boto:path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.466 DEBUG:boto:auth_path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.467 DEBUG:boto:Method: GET 2022-09-12T19:16:39.468 DEBUG:boto:Path: /jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.469 DEBUG:boto:Data: 2022-09-12T19:16:39.470 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.471 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:39.471 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.472 DEBUG:boto:Params: {} 2022-09-12T19:16:39.473 DEBUG:boto:Token: None 2022-09-12T19:16:39.474 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/?versions 2022-09-12T19:16:39.475 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c= 2022-09-12T19:16:39.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c=', 'Content-Length': '0'} 2022-09-12T19:16:39.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089a22b8dc52c581e-00631f8597-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.480 DEBUG:boto:b'jdpiak-40999false' 2022-09-12T19:16:39.481 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-40 zones={test-zone3, test-zone2} 2022-09-12T19:16:39.481 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-40 zones={test-zone1, test-zone3} 2022-09-12T19:16:39.482 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.483 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.484 DEBUG:boto:Method: HEAD 2022-09-12T19:16:39.484 DEBUG:boto:Path: /jdpiak-40/ 2022-09-12T19:16:39.485 DEBUG:boto:Data: 2022-09-12T19:16:39.485 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.486 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:39.487 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.488 DEBUG:boto:Params: {} 2022-09-12T19:16:39.489 DEBUG:boto:Token: None 2022-09-12T19:16:39.489 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/ 2022-09-12T19:16:39.491 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AwMK/lk1Rj+iaU4QQbmXX9yRK4w= 2022-09-12T19:16:39.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AwMK/lk1Rj+iaU4QQbmXX9yRK4w=', 'Content-Length': '0'} 2022-09-12T19:16:39.495 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bacf69cb231a8422-00631f8597-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.496 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.496 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.497 DEBUG:boto:Method: HEAD 2022-09-12T19:16:39.498 DEBUG:boto:Path: /jdpiak-40/ 2022-09-12T19:16:39.498 DEBUG:boto:Data: 2022-09-12T19:16:39.499 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.500 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:39.501 DEBUG:boto:Port: 8001 2022-09-12T19:16:39.502 DEBUG:boto:Params: {} 2022-09-12T19:16:39.503 DEBUG:boto:Token: None 2022-09-12T19:16:39.504 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/ 2022-09-12T19:16:39.505 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AwMK/lk1Rj+iaU4QQbmXX9yRK4w= 2022-09-12T19:16:39.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AwMK/lk1Rj+iaU4QQbmXX9yRK4w=', 'Content-Length': '0'} 2022-09-12T19:16:39.508 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000049d6472ffe5f7635-00631f8597-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.509 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:39.509 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.510 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.511 DEBUG:boto:path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.511 DEBUG:boto:auth_path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.512 DEBUG:boto:Method: GET 2022-09-12T19:16:39.512 DEBUG:boto:Path: /jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.513 DEBUG:boto:Data: 2022-09-12T19:16:39.513 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.514 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:39.515 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.515 DEBUG:boto:Params: {} 2022-09-12T19:16:39.516 DEBUG:boto:Token: None 2022-09-12T19:16:39.516 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/?versions 2022-09-12T19:16:39.517 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c= 2022-09-12T19:16:39.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c=', 'Content-Length': '0'} 2022-09-12T19:16:39.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000693005fafaf2dfb1-00631f8597-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.522 DEBUG:boto:b'jdpiak-40999false' 2022-09-12T19:16:39.523 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:39.524 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.525 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.526 DEBUG:boto:path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.527 DEBUG:boto:auth_path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.528 DEBUG:boto:Method: GET 2022-09-12T19:16:39.529 DEBUG:boto:Path: /jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.530 DEBUG:boto:Data: 2022-09-12T19:16:39.530 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.531 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:39.532 DEBUG:boto:Port: 8001 2022-09-12T19:16:39.533 DEBUG:boto:Params: {} 2022-09-12T19:16:39.534 DEBUG:boto:Token: None 2022-09-12T19:16:39.536 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/?versions 2022-09-12T19:16:39.537 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c= 2022-09-12T19:16:39.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c=', 'Content-Length': '0'} 2022-09-12T19:16:39.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a5296603ed3ea94-00631f8597-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.543 DEBUG:boto:b'jdpiak-40999false' 2022-09-12T19:16:39.544 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.545 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.546 DEBUG:boto:path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.547 DEBUG:boto:auth_path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.548 DEBUG:boto:Method: GET 2022-09-12T19:16:39.549 DEBUG:boto:Path: /jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.550 DEBUG:boto:Data: 2022-09-12T19:16:39.551 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.551 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:39.552 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.552 DEBUG:boto:Params: {} 2022-09-12T19:16:39.553 DEBUG:boto:Token: None 2022-09-12T19:16:39.553 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/?versions 2022-09-12T19:16:39.554 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c= 2022-09-12T19:16:39.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c=', 'Content-Length': '0'} 2022-09-12T19:16:39.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c764fa894f67e39d-00631f8597-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.558 DEBUG:boto:b'jdpiak-40999false' 2022-09-12T19:16:39.559 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.559 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.560 DEBUG:boto:path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.560 DEBUG:boto:auth_path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.561 DEBUG:boto:Method: GET 2022-09-12T19:16:39.562 DEBUG:boto:Path: /jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.563 DEBUG:boto:Data: 2022-09-12T19:16:39.564 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.564 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:39.565 DEBUG:boto:Port: 8001 2022-09-12T19:16:39.566 DEBUG:boto:Params: {} 2022-09-12T19:16:39.567 DEBUG:boto:Token: None 2022-09-12T19:16:39.567 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/?versions 2022-09-12T19:16:39.568 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c= 2022-09-12T19:16:39.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c=', 'Content-Length': '0'} 2022-09-12T19:16:39.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062840d858d18e662-00631f8597-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.572 DEBUG:boto:b'jdpiak-40999false' 2022-09-12T19:16:39.572 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-40 zones={test-zone1, test-zone3} 2022-09-12T19:16:39.573 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-40 zones={test-zone2, test-zone3} 2022-09-12T19:16:39.573 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.574 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.574 DEBUG:boto:Method: HEAD 2022-09-12T19:16:39.575 DEBUG:boto:Path: /jdpiak-40/ 2022-09-12T19:16:39.575 DEBUG:boto:Data: 2022-09-12T19:16:39.576 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.576 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:39.577 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.577 DEBUG:boto:Params: {} 2022-09-12T19:16:39.578 DEBUG:boto:Token: None 2022-09-12T19:16:39.579 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/ 2022-09-12T19:16:39.579 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AwMK/lk1Rj+iaU4QQbmXX9yRK4w= 2022-09-12T19:16:39.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AwMK/lk1Rj+iaU4QQbmXX9yRK4w=', 'Content-Length': '0'} 2022-09-12T19:16:39.583 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012afd2a3df9d2439-00631f8597-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.583 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.584 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.584 DEBUG:boto:Method: HEAD 2022-09-12T19:16:39.585 DEBUG:boto:Path: /jdpiak-40/ 2022-09-12T19:16:39.585 DEBUG:boto:Data: 2022-09-12T19:16:39.586 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.586 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:39.587 DEBUG:boto:Port: 8001 2022-09-12T19:16:39.587 DEBUG:boto:Params: {} 2022-09-12T19:16:39.588 DEBUG:boto:Token: None 2022-09-12T19:16:39.588 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/ 2022-09-12T19:16:39.589 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AwMK/lk1Rj+iaU4QQbmXX9yRK4w= 2022-09-12T19:16:39.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AwMK/lk1Rj+iaU4QQbmXX9yRK4w=', 'Content-Length': '0'} 2022-09-12T19:16:39.592 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007100a23e7d369db8-00631f8597-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.593 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:39.594 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.594 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.595 DEBUG:boto:path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.595 DEBUG:boto:auth_path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.596 DEBUG:boto:Method: GET 2022-09-12T19:16:39.596 DEBUG:boto:Path: /jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.597 DEBUG:boto:Data: 2022-09-12T19:16:39.597 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.598 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:39.598 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.599 DEBUG:boto:Params: {} 2022-09-12T19:16:39.600 DEBUG:boto:Token: None 2022-09-12T19:16:39.600 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/?versions 2022-09-12T19:16:39.601 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c= 2022-09-12T19:16:39.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c=', 'Content-Length': '0'} 2022-09-12T19:16:39.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006258b9b354adf3dc-00631f8597-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.647 DEBUG:boto:b'jdpiak-40999false' 2022-09-12T19:16:39.648 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:39.649 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.649 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.650 DEBUG:boto:path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.650 DEBUG:boto:auth_path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.651 DEBUG:boto:Method: GET 2022-09-12T19:16:39.651 DEBUG:boto:Path: /jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.652 DEBUG:boto:Data: 2022-09-12T19:16:39.652 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.653 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:39.653 DEBUG:boto:Port: 8001 2022-09-12T19:16:39.654 DEBUG:boto:Params: {} 2022-09-12T19:16:39.654 DEBUG:boto:Token: None 2022-09-12T19:16:39.655 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/?versions 2022-09-12T19:16:39.656 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c= 2022-09-12T19:16:39.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c=', 'Content-Length': '0'} 2022-09-12T19:16:39.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dbb34cb84221787-00631f8597-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.660 DEBUG:boto:b'jdpiak-40999false' 2022-09-12T19:16:39.660 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.661 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.662 DEBUG:boto:path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.662 DEBUG:boto:auth_path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.663 DEBUG:boto:Method: GET 2022-09-12T19:16:39.663 DEBUG:boto:Path: /jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.664 DEBUG:boto:Data: 2022-09-12T19:16:39.664 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.665 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:39.665 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.666 DEBUG:boto:Params: {} 2022-09-12T19:16:39.666 DEBUG:boto:Token: None 2022-09-12T19:16:39.667 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/?versions 2022-09-12T19:16:39.668 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c= 2022-09-12T19:16:39.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c=', 'Content-Length': '0'} 2022-09-12T19:16:39.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007550695f0f2bd309-00631f8597-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.671 DEBUG:boto:b'jdpiak-40999false' 2022-09-12T19:16:39.672 DEBUG:boto:path=/jdpiak-40/ 2022-09-12T19:16:39.673 DEBUG:boto:auth_path=/jdpiak-40/ 2022-09-12T19:16:39.674 DEBUG:boto:path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.676 DEBUG:boto:auth_path=/jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.676 DEBUG:boto:Method: GET 2022-09-12T19:16:39.677 DEBUG:boto:Path: /jdpiak-40/?versions&max-keys=999 2022-09-12T19:16:39.677 DEBUG:boto:Data: 2022-09-12T19:16:39.678 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.678 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:39.679 DEBUG:boto:Port: 8001 2022-09-12T19:16:39.679 DEBUG:boto:Params: {} 2022-09-12T19:16:39.680 DEBUG:boto:Token: None 2022-09-12T19:16:39.681 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-40/?versions 2022-09-12T19:16:39.681 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c= 2022-09-12T19:16:39.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZNFQTih5sW3ABJqqdweHkbOX2c=', 'Content-Length': '0'} 2022-09-12T19:16:39.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036c4697456bdd4a6-00631f8597-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.726 DEBUG:boto:b'jdpiak-40999false' 2022-09-12T19:16:39.727 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-40 zones={test-zone2, test-zone3} 2022-09-12T19:16:39.728 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-41 zones={test-zone2, test-zone1} 2022-09-12T19:16:39.728 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:39.729 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:39.729 DEBUG:boto:Method: HEAD 2022-09-12T19:16:39.730 DEBUG:boto:Path: /jdpiak-41/ 2022-09-12T19:16:39.730 DEBUG:boto:Data: 2022-09-12T19:16:39.731 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.731 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:39.732 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.732 DEBUG:boto:Params: {} 2022-09-12T19:16:39.733 DEBUG:boto:Token: None 2022-09-12T19:16:39.734 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-41/ 2022-09-12T19:16:39.734 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8K4CTrCmSW3lsRHZ67b/k/7MyOc= 2022-09-12T19:16:39.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8K4CTrCmSW3lsRHZ67b/k/7MyOc=', 'Content-Length': '0'} 2022-09-12T19:16:39.741 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098f16b66bfc12932-00631f8597-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.742 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:39.743 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:39.744 DEBUG:boto:Method: HEAD 2022-09-12T19:16:39.744 DEBUG:boto:Path: /jdpiak-41/ 2022-09-12T19:16:39.745 DEBUG:boto:Data: 2022-09-12T19:16:39.745 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.746 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:39.746 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.747 DEBUG:boto:Params: {} 2022-09-12T19:16:39.747 DEBUG:boto:Token: None 2022-09-12T19:16:39.748 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-41/ 2022-09-12T19:16:39.749 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8K4CTrCmSW3lsRHZ67b/k/7MyOc= 2022-09-12T19:16:39.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8K4CTrCmSW3lsRHZ67b/k/7MyOc=', 'Content-Length': '0'} 2022-09-12T19:16:39.753 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1e4304730ae13c6-00631f8597-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.753 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:39.754 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:39.754 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:39.755 DEBUG:boto:path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.755 DEBUG:boto:auth_path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.756 DEBUG:boto:Method: GET 2022-09-12T19:16:39.756 DEBUG:boto:Path: /jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.757 DEBUG:boto:Data: 2022-09-12T19:16:39.758 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.758 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:39.759 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.759 DEBUG:boto:Params: {} 2022-09-12T19:16:39.760 DEBUG:boto:Token: None 2022-09-12T19:16:39.760 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-41/?versions 2022-09-12T19:16:39.761 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Uk91gdxrUnY7olAuGytpFaXwqWk= 2022-09-12T19:16:39.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Uk91gdxrUnY7olAuGytpFaXwqWk=', 'Content-Length': '0'} 2022-09-12T19:16:39.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000474b0747806b00e4-00631f8597-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.766 DEBUG:boto:b'jdpiak-41999false' 2022-09-12T19:16:39.767 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:39.767 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:39.768 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:39.769 DEBUG:boto:path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.769 DEBUG:boto:auth_path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.770 DEBUG:boto:Method: GET 2022-09-12T19:16:39.770 DEBUG:boto:Path: /jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.771 DEBUG:boto:Data: 2022-09-12T19:16:39.772 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.772 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:39.773 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.773 DEBUG:boto:Params: {} 2022-09-12T19:16:39.774 DEBUG:boto:Token: None 2022-09-12T19:16:39.775 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-41/?versions 2022-09-12T19:16:39.775 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Uk91gdxrUnY7olAuGytpFaXwqWk= 2022-09-12T19:16:39.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Uk91gdxrUnY7olAuGytpFaXwqWk=', 'Content-Length': '0'} 2022-09-12T19:16:39.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002648d810579f4f30-00631f8597-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.779 DEBUG:boto:b'jdpiak-41999false' 2022-09-12T19:16:39.780 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:39.781 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:39.781 DEBUG:boto:path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.782 DEBUG:boto:auth_path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.782 DEBUG:boto:Method: GET 2022-09-12T19:16:39.783 DEBUG:boto:Path: /jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.784 DEBUG:boto:Data: 2022-09-12T19:16:39.784 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.785 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:39.785 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.786 DEBUG:boto:Params: {} 2022-09-12T19:16:39.786 DEBUG:boto:Token: None 2022-09-12T19:16:39.787 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-41/?versions 2022-09-12T19:16:39.788 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Uk91gdxrUnY7olAuGytpFaXwqWk= 2022-09-12T19:16:39.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Uk91gdxrUnY7olAuGytpFaXwqWk=', 'Content-Length': '0'} 2022-09-12T19:16:39.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffb7e31c0463142d-00631f8597-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.793 DEBUG:boto:b'jdpiak-41999false' 2022-09-12T19:16:39.794 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:39.794 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:39.795 DEBUG:boto:path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.796 DEBUG:boto:auth_path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.796 DEBUG:boto:Method: GET 2022-09-12T19:16:39.797 DEBUG:boto:Path: /jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.797 DEBUG:boto:Data: 2022-09-12T19:16:39.798 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.799 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:39.800 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.800 DEBUG:boto:Params: {} 2022-09-12T19:16:39.801 DEBUG:boto:Token: None 2022-09-12T19:16:39.801 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-41/?versions 2022-09-12T19:16:39.802 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Uk91gdxrUnY7olAuGytpFaXwqWk= 2022-09-12T19:16:39.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Uk91gdxrUnY7olAuGytpFaXwqWk=', 'Content-Length': '0'} 2022-09-12T19:16:39.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fddcbca8d65cda62-00631f8597-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.854 DEBUG:boto:b'jdpiak-41999false' 2022-09-12T19:16:39.855 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-41 zones={test-zone2, test-zone1} 2022-09-12T19:16:39.855 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-41 zones={test-zone3, test-zone1} 2022-09-12T19:16:39.856 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:39.856 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:39.857 DEBUG:boto:Method: HEAD 2022-09-12T19:16:39.857 DEBUG:boto:Path: /jdpiak-41/ 2022-09-12T19:16:39.858 DEBUG:boto:Data: 2022-09-12T19:16:39.858 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.859 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:39.859 DEBUG:boto:Port: 8001 2022-09-12T19:16:39.860 DEBUG:boto:Params: {} 2022-09-12T19:16:39.860 DEBUG:boto:Token: None 2022-09-12T19:16:39.861 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-41/ 2022-09-12T19:16:39.862 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8K4CTrCmSW3lsRHZ67b/k/7MyOc= 2022-09-12T19:16:39.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8K4CTrCmSW3lsRHZ67b/k/7MyOc=', 'Content-Length': '0'} 2022-09-12T19:16:39.866 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008665c9e5efbf63e2-00631f8597-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.867 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:39.867 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:39.868 DEBUG:boto:Method: HEAD 2022-09-12T19:16:39.869 DEBUG:boto:Path: /jdpiak-41/ 2022-09-12T19:16:39.869 DEBUG:boto:Data: 2022-09-12T19:16:39.870 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.870 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:39.871 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.871 DEBUG:boto:Params: {} 2022-09-12T19:16:39.872 DEBUG:boto:Token: None 2022-09-12T19:16:39.873 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-41/ 2022-09-12T19:16:39.873 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8K4CTrCmSW3lsRHZ67b/k/7MyOc= 2022-09-12T19:16:39.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8K4CTrCmSW3lsRHZ67b/k/7MyOc=', 'Content-Length': '0'} 2022-09-12T19:16:39.876 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e016d7fc77241ad5-00631f8597-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.877 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:39.877 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:39.878 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:39.879 DEBUG:boto:path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.879 DEBUG:boto:auth_path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.880 DEBUG:boto:Method: GET 2022-09-12T19:16:39.880 DEBUG:boto:Path: /jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.881 DEBUG:boto:Data: 2022-09-12T19:16:39.881 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.882 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:39.882 DEBUG:boto:Port: 8001 2022-09-12T19:16:39.883 DEBUG:boto:Params: {} 2022-09-12T19:16:39.884 DEBUG:boto:Token: None 2022-09-12T19:16:39.885 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-41/?versions 2022-09-12T19:16:39.886 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Uk91gdxrUnY7olAuGytpFaXwqWk= 2022-09-12T19:16:39.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Uk91gdxrUnY7olAuGytpFaXwqWk=', 'Content-Length': '0'} 2022-09-12T19:16:39.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000990728cde5f59e0e-00631f8597-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.890 DEBUG:boto:b'jdpiak-41999false' 2022-09-12T19:16:39.891 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:39.892 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:39.892 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:39.893 DEBUG:boto:path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.894 DEBUG:boto:auth_path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.894 DEBUG:boto:Method: GET 2022-09-12T19:16:39.896 DEBUG:boto:Path: /jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.897 DEBUG:boto:Data: 2022-09-12T19:16:39.898 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.898 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:39.899 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.900 DEBUG:boto:Params: {} 2022-09-12T19:16:39.901 DEBUG:boto:Token: None 2022-09-12T19:16:39.902 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-41/?versions 2022-09-12T19:16:39.903 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Uk91gdxrUnY7olAuGytpFaXwqWk= 2022-09-12T19:16:39.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Uk91gdxrUnY7olAuGytpFaXwqWk=', 'Content-Length': '0'} 2022-09-12T19:16:39.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc8749ce39e2cfc3-00631f8597-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.909 DEBUG:boto:b'jdpiak-41999false' 2022-09-12T19:16:39.910 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:39.911 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:39.911 DEBUG:boto:path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.912 DEBUG:boto:auth_path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.913 DEBUG:boto:Method: GET 2022-09-12T19:16:39.914 DEBUG:boto:Path: /jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.914 DEBUG:boto:Data: 2022-09-12T19:16:39.915 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.915 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:39.916 DEBUG:boto:Port: 8001 2022-09-12T19:16:39.917 DEBUG:boto:Params: {} 2022-09-12T19:16:39.917 DEBUG:boto:Token: None 2022-09-12T19:16:39.918 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-41/?versions 2022-09-12T19:16:39.918 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Uk91gdxrUnY7olAuGytpFaXwqWk= 2022-09-12T19:16:39.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Uk91gdxrUnY7olAuGytpFaXwqWk=', 'Content-Length': '0'} 2022-09-12T19:16:39.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d221d9170fe2edad-00631f8597-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.923 DEBUG:boto:b'jdpiak-41999false' 2022-09-12T19:16:39.924 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:39.925 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:39.925 DEBUG:boto:path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.926 DEBUG:boto:auth_path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.926 DEBUG:boto:Method: GET 2022-09-12T19:16:39.927 DEBUG:boto:Path: /jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.927 DEBUG:boto:Data: 2022-09-12T19:16:39.928 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.928 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:39.928 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.929 DEBUG:boto:Params: {} 2022-09-12T19:16:39.930 DEBUG:boto:Token: None 2022-09-12T19:16:39.930 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-41/?versions 2022-09-12T19:16:39.931 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Uk91gdxrUnY7olAuGytpFaXwqWk= 2022-09-12T19:16:39.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Uk91gdxrUnY7olAuGytpFaXwqWk=', 'Content-Length': '0'} 2022-09-12T19:16:39.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e52a1b6a4a36e7d6-00631f8597-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.935 DEBUG:boto:b'jdpiak-41999false' 2022-09-12T19:16:39.936 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-41 zones={test-zone3, test-zone1} 2022-09-12T19:16:39.936 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-41 zones={test-zone1, test-zone2} 2022-09-12T19:16:39.937 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:39.937 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:39.938 DEBUG:boto:Method: HEAD 2022-09-12T19:16:39.938 DEBUG:boto:Path: /jdpiak-41/ 2022-09-12T19:16:39.939 DEBUG:boto:Data: 2022-09-12T19:16:39.939 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.940 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:39.941 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.942 DEBUG:boto:Params: {} 2022-09-12T19:16:39.943 DEBUG:boto:Token: None 2022-09-12T19:16:39.944 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-41/ 2022-09-12T19:16:39.945 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8K4CTrCmSW3lsRHZ67b/k/7MyOc= 2022-09-12T19:16:39.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8K4CTrCmSW3lsRHZ67b/k/7MyOc=', 'Content-Length': '0'} 2022-09-12T19:16:39.948 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000443af281e323aa83-00631f8597-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.949 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:39.949 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:39.950 DEBUG:boto:Method: HEAD 2022-09-12T19:16:39.951 DEBUG:boto:Path: /jdpiak-41/ 2022-09-12T19:16:39.951 DEBUG:boto:Data: 2022-09-12T19:16:39.952 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.952 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:39.953 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.953 DEBUG:boto:Params: {} 2022-09-12T19:16:39.954 DEBUG:boto:Token: None 2022-09-12T19:16:39.954 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-41/ 2022-09-12T19:16:39.955 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8K4CTrCmSW3lsRHZ67b/k/7MyOc= 2022-09-12T19:16:39.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8K4CTrCmSW3lsRHZ67b/k/7MyOc=', 'Content-Length': '0'} 2022-09-12T19:16:39.958 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000333754a41e4f1b3c-00631f8597-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:39.959 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:39.960 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:39.960 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:39.961 DEBUG:boto:path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.961 DEBUG:boto:auth_path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.962 DEBUG:boto:Method: GET 2022-09-12T19:16:39.962 DEBUG:boto:Path: /jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:39.963 DEBUG:boto:Data: 2022-09-12T19:16:39.963 DEBUG:boto:Headers: {} 2022-09-12T19:16:39.964 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:39.964 DEBUG:boto:Port: 8000 2022-09-12T19:16:39.965 DEBUG:boto:Params: {} 2022-09-12T19:16:39.965 DEBUG:boto:Token: None 2022-09-12T19:16:39.966 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:39 GMT /jdpiak-41/?versions 2022-09-12T19:16:39.966 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Uk91gdxrUnY7olAuGytpFaXwqWk= 2022-09-12T19:16:39.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Uk91gdxrUnY7olAuGytpFaXwqWk=', 'Content-Length': '0'} 2022-09-12T19:16:39.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebcfbc15774f22bb-00631f8597-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.010 DEBUG:boto:b'jdpiak-41999false' 2022-09-12T19:16:40.011 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:40.012 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:40.012 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:40.013 DEBUG:boto:path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.013 DEBUG:boto:auth_path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.014 DEBUG:boto:Method: GET 2022-09-12T19:16:40.014 DEBUG:boto:Path: /jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.015 DEBUG:boto:Data: 2022-09-12T19:16:40.015 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.016 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:40.017 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.017 DEBUG:boto:Params: {} 2022-09-12T19:16:40.018 DEBUG:boto:Token: None 2022-09-12T19:16:40.018 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-41/?versions 2022-09-12T19:16:40.019 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM= 2022-09-12T19:16:40.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM=', 'Content-Length': '0'} 2022-09-12T19:16:40.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000458fb82226c5edc6-00631f8598-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.023 DEBUG:boto:b'jdpiak-41999false' 2022-09-12T19:16:40.024 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:40.024 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:40.025 DEBUG:boto:path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.026 DEBUG:boto:auth_path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.027 DEBUG:boto:Method: GET 2022-09-12T19:16:40.028 DEBUG:boto:Path: /jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.029 DEBUG:boto:Data: 2022-09-12T19:16:40.030 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.030 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:40.031 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.032 DEBUG:boto:Params: {} 2022-09-12T19:16:40.033 DEBUG:boto:Token: None 2022-09-12T19:16:40.034 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-41/?versions 2022-09-12T19:16:40.035 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM= 2022-09-12T19:16:40.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM=', 'Content-Length': '0'} 2022-09-12T19:16:40.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a69b28380e710045-00631f8598-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.040 DEBUG:boto:b'jdpiak-41999false' 2022-09-12T19:16:40.042 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:40.043 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:40.044 DEBUG:boto:path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.045 DEBUG:boto:auth_path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.045 DEBUG:boto:Method: GET 2022-09-12T19:16:40.046 DEBUG:boto:Path: /jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.047 DEBUG:boto:Data: 2022-09-12T19:16:40.048 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.049 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:40.050 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.051 DEBUG:boto:Params: {} 2022-09-12T19:16:40.052 DEBUG:boto:Token: None 2022-09-12T19:16:40.053 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-41/?versions 2022-09-12T19:16:40.054 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM= 2022-09-12T19:16:40.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM=', 'Content-Length': '0'} 2022-09-12T19:16:40.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000126855e1361b4a91-00631f8598-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.102 DEBUG:boto:b'jdpiak-41999false' 2022-09-12T19:16:40.104 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-41 zones={test-zone1, test-zone2} 2022-09-12T19:16:40.104 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-41 zones={test-zone3, test-zone2} 2022-09-12T19:16:40.105 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:40.106 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:40.106 DEBUG:boto:Method: HEAD 2022-09-12T19:16:40.107 DEBUG:boto:Path: /jdpiak-41/ 2022-09-12T19:16:40.108 DEBUG:boto:Data: 2022-09-12T19:16:40.108 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.109 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:40.109 DEBUG:boto:Port: 8001 2022-09-12T19:16:40.110 DEBUG:boto:Params: {} 2022-09-12T19:16:40.111 DEBUG:boto:Token: None 2022-09-12T19:16:40.111 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-41/ 2022-09-12T19:16:40.112 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JpYhQt2vR808vI2CNej7yKdby8w= 2022-09-12T19:16:40.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JpYhQt2vR808vI2CNej7yKdby8w=', 'Content-Length': '0'} 2022-09-12T19:16:40.116 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087cb6ab058673db4-00631f8598-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.117 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:40.118 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:40.119 DEBUG:boto:Method: HEAD 2022-09-12T19:16:40.121 DEBUG:boto:Path: /jdpiak-41/ 2022-09-12T19:16:40.131 DEBUG:boto:Data: 2022-09-12T19:16:40.132 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.133 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:40.134 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.135 DEBUG:boto:Params: {} 2022-09-12T19:16:40.136 DEBUG:boto:Token: None 2022-09-12T19:16:40.137 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-41/ 2022-09-12T19:16:40.138 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JpYhQt2vR808vI2CNej7yKdby8w= 2022-09-12T19:16:40.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JpYhQt2vR808vI2CNej7yKdby8w=', 'Content-Length': '0'} 2022-09-12T19:16:40.142 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f663103bea04d8c-00631f8598-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.143 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:40.144 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:40.145 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:40.146 DEBUG:boto:path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.147 DEBUG:boto:auth_path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.147 DEBUG:boto:Method: GET 2022-09-12T19:16:40.148 DEBUG:boto:Path: /jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.149 DEBUG:boto:Data: 2022-09-12T19:16:40.150 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.150 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:40.151 DEBUG:boto:Port: 8001 2022-09-12T19:16:40.151 DEBUG:boto:Params: {} 2022-09-12T19:16:40.152 DEBUG:boto:Token: None 2022-09-12T19:16:40.153 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-41/?versions 2022-09-12T19:16:40.154 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM= 2022-09-12T19:16:40.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM=', 'Content-Length': '0'} 2022-09-12T19:16:40.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a65058ed30bdb34a-00631f8598-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.158 DEBUG:boto:b'jdpiak-41999false' 2022-09-12T19:16:40.159 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:40.159 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:40.160 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:40.160 DEBUG:boto:path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.161 DEBUG:boto:auth_path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.162 DEBUG:boto:Method: GET 2022-09-12T19:16:40.162 DEBUG:boto:Path: /jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.163 DEBUG:boto:Data: 2022-09-12T19:16:40.164 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.164 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:40.165 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.165 DEBUG:boto:Params: {} 2022-09-12T19:16:40.166 DEBUG:boto:Token: None 2022-09-12T19:16:40.166 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-41/?versions 2022-09-12T19:16:40.167 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM= 2022-09-12T19:16:40.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM=', 'Content-Length': '0'} 2022-09-12T19:16:40.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7b171cac68943d3-00631f8598-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.171 DEBUG:boto:b'jdpiak-41999false' 2022-09-12T19:16:40.172 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:40.172 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:40.173 DEBUG:boto:path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.174 DEBUG:boto:auth_path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.174 DEBUG:boto:Method: GET 2022-09-12T19:16:40.175 DEBUG:boto:Path: /jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.176 DEBUG:boto:Data: 2022-09-12T19:16:40.177 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.178 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:40.179 DEBUG:boto:Port: 8001 2022-09-12T19:16:40.180 DEBUG:boto:Params: {} 2022-09-12T19:16:40.181 DEBUG:boto:Token: None 2022-09-12T19:16:40.182 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-41/?versions 2022-09-12T19:16:40.183 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM= 2022-09-12T19:16:40.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM=', 'Content-Length': '0'} 2022-09-12T19:16:40.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004616676f39690192-00631f8598-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.192 DEBUG:boto:b'jdpiak-41999false' 2022-09-12T19:16:40.193 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:40.194 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:40.194 DEBUG:boto:path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.195 DEBUG:boto:auth_path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.196 DEBUG:boto:Method: GET 2022-09-12T19:16:40.196 DEBUG:boto:Path: /jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.197 DEBUG:boto:Data: 2022-09-12T19:16:40.197 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.198 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:40.198 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.199 DEBUG:boto:Params: {} 2022-09-12T19:16:40.199 DEBUG:boto:Token: None 2022-09-12T19:16:40.200 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-41/?versions 2022-09-12T19:16:40.201 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM= 2022-09-12T19:16:40.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM=', 'Content-Length': '0'} 2022-09-12T19:16:40.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a541e889ff0e8c97-00631f8598-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.253 DEBUG:boto:b'jdpiak-41999false' 2022-09-12T19:16:40.254 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-41 zones={test-zone3, test-zone2} 2022-09-12T19:16:40.255 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-41 zones={test-zone1, test-zone3} 2022-09-12T19:16:40.256 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:40.257 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:40.257 DEBUG:boto:Method: HEAD 2022-09-12T19:16:40.258 DEBUG:boto:Path: /jdpiak-41/ 2022-09-12T19:16:40.259 DEBUG:boto:Data: 2022-09-12T19:16:40.259 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.260 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:40.260 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.261 DEBUG:boto:Params: {} 2022-09-12T19:16:40.261 DEBUG:boto:Token: None 2022-09-12T19:16:40.262 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-41/ 2022-09-12T19:16:40.263 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JpYhQt2vR808vI2CNej7yKdby8w= 2022-09-12T19:16:40.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JpYhQt2vR808vI2CNej7yKdby8w=', 'Content-Length': '0'} 2022-09-12T19:16:40.267 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f78e8e7a02216d54-00631f8598-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.268 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:40.269 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:40.270 DEBUG:boto:Method: HEAD 2022-09-12T19:16:40.271 DEBUG:boto:Path: /jdpiak-41/ 2022-09-12T19:16:40.272 DEBUG:boto:Data: 2022-09-12T19:16:40.273 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.274 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:40.275 DEBUG:boto:Port: 8001 2022-09-12T19:16:40.276 DEBUG:boto:Params: {} 2022-09-12T19:16:40.285 DEBUG:boto:Token: None 2022-09-12T19:16:40.287 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-41/ 2022-09-12T19:16:40.288 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JpYhQt2vR808vI2CNej7yKdby8w= 2022-09-12T19:16:40.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JpYhQt2vR808vI2CNej7yKdby8w=', 'Content-Length': '0'} 2022-09-12T19:16:40.292 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e55c51228a3cf9d-00631f8598-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.294 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:40.294 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:40.295 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:40.296 DEBUG:boto:path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.296 DEBUG:boto:auth_path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.297 DEBUG:boto:Method: GET 2022-09-12T19:16:40.298 DEBUG:boto:Path: /jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.298 DEBUG:boto:Data: 2022-09-12T19:16:40.299 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.300 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:40.300 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.301 DEBUG:boto:Params: {} 2022-09-12T19:16:40.302 DEBUG:boto:Token: None 2022-09-12T19:16:40.302 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-41/?versions 2022-09-12T19:16:40.303 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM= 2022-09-12T19:16:40.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM=', 'Content-Length': '0'} 2022-09-12T19:16:40.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009898c196c73aa2b7-00631f8598-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.308 DEBUG:boto:b'jdpiak-41999false' 2022-09-12T19:16:40.309 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:40.309 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:40.310 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:40.311 DEBUG:boto:path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.311 DEBUG:boto:auth_path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.312 DEBUG:boto:Method: GET 2022-09-12T19:16:40.313 DEBUG:boto:Path: /jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.313 DEBUG:boto:Data: 2022-09-12T19:16:40.314 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.315 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:40.316 DEBUG:boto:Port: 8001 2022-09-12T19:16:40.316 DEBUG:boto:Params: {} 2022-09-12T19:16:40.318 DEBUG:boto:Token: None 2022-09-12T19:16:40.319 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-41/?versions 2022-09-12T19:16:40.320 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM= 2022-09-12T19:16:40.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM=', 'Content-Length': '0'} 2022-09-12T19:16:40.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccfec432a00df0db-00631f8598-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.326 DEBUG:boto:b'jdpiak-41999false' 2022-09-12T19:16:40.327 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:40.328 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:40.329 DEBUG:boto:path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.330 DEBUG:boto:auth_path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.330 DEBUG:boto:Method: GET 2022-09-12T19:16:40.331 DEBUG:boto:Path: /jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.331 DEBUG:boto:Data: 2022-09-12T19:16:40.332 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.333 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:40.334 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.335 DEBUG:boto:Params: {} 2022-09-12T19:16:40.336 DEBUG:boto:Token: None 2022-09-12T19:16:40.337 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-41/?versions 2022-09-12T19:16:40.338 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM= 2022-09-12T19:16:40.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM=', 'Content-Length': '0'} 2022-09-12T19:16:40.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e66ab9f7380ed7f-00631f8598-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.390 DEBUG:boto:b'jdpiak-41999false' 2022-09-12T19:16:40.392 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:40.393 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:40.394 DEBUG:boto:path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.394 DEBUG:boto:auth_path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.395 DEBUG:boto:Method: GET 2022-09-12T19:16:40.396 DEBUG:boto:Path: /jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.397 DEBUG:boto:Data: 2022-09-12T19:16:40.397 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.398 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:40.399 DEBUG:boto:Port: 8001 2022-09-12T19:16:40.399 DEBUG:boto:Params: {} 2022-09-12T19:16:40.400 DEBUG:boto:Token: None 2022-09-12T19:16:40.400 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-41/?versions 2022-09-12T19:16:40.401 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM= 2022-09-12T19:16:40.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM=', 'Content-Length': '0'} 2022-09-12T19:16:40.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b74a83b91cf487ac-00631f8598-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.406 DEBUG:boto:b'jdpiak-41999false' 2022-09-12T19:16:40.407 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-41 zones={test-zone1, test-zone3} 2022-09-12T19:16:40.408 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-41 zones={test-zone2, test-zone3} 2022-09-12T19:16:40.409 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:40.410 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:40.410 DEBUG:boto:Method: HEAD 2022-09-12T19:16:40.411 DEBUG:boto:Path: /jdpiak-41/ 2022-09-12T19:16:40.412 DEBUG:boto:Data: 2022-09-12T19:16:40.413 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.414 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:40.415 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.416 DEBUG:boto:Params: {} 2022-09-12T19:16:40.417 DEBUG:boto:Token: None 2022-09-12T19:16:40.418 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-41/ 2022-09-12T19:16:40.419 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JpYhQt2vR808vI2CNej7yKdby8w= 2022-09-12T19:16:40.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JpYhQt2vR808vI2CNej7yKdby8w=', 'Content-Length': '0'} 2022-09-12T19:16:40.423 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000575a727708b95b98-00631f8598-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.424 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:40.425 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:40.426 DEBUG:boto:Method: HEAD 2022-09-12T19:16:40.427 DEBUG:boto:Path: /jdpiak-41/ 2022-09-12T19:16:40.428 DEBUG:boto:Data: 2022-09-12T19:16:40.429 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.430 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:40.431 DEBUG:boto:Port: 8001 2022-09-12T19:16:40.432 DEBUG:boto:Params: {} 2022-09-12T19:16:40.433 DEBUG:boto:Token: None 2022-09-12T19:16:40.433 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-41/ 2022-09-12T19:16:40.434 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JpYhQt2vR808vI2CNej7yKdby8w= 2022-09-12T19:16:40.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JpYhQt2vR808vI2CNej7yKdby8w=', 'Content-Length': '0'} 2022-09-12T19:16:40.438 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b07dd340e77c1603-00631f8598-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.440 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:40.440 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:40.441 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:40.442 DEBUG:boto:path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.443 DEBUG:boto:auth_path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.444 DEBUG:boto:Method: GET 2022-09-12T19:16:40.445 DEBUG:boto:Path: /jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.446 DEBUG:boto:Data: 2022-09-12T19:16:40.446 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.447 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:40.448 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.449 DEBUG:boto:Params: {} 2022-09-12T19:16:40.450 DEBUG:boto:Token: None 2022-09-12T19:16:40.451 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-41/?versions 2022-09-12T19:16:40.452 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM= 2022-09-12T19:16:40.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM=', 'Content-Length': '0'} 2022-09-12T19:16:40.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006279402402e0a3a8-00631f8598-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.458 DEBUG:boto:b'jdpiak-41999false' 2022-09-12T19:16:40.459 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:40.460 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:40.461 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:40.461 DEBUG:boto:path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.462 DEBUG:boto:auth_path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.463 DEBUG:boto:Method: GET 2022-09-12T19:16:40.464 DEBUG:boto:Path: /jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.465 DEBUG:boto:Data: 2022-09-12T19:16:40.466 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.466 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:40.467 DEBUG:boto:Port: 8001 2022-09-12T19:16:40.468 DEBUG:boto:Params: {} 2022-09-12T19:16:40.469 DEBUG:boto:Token: None 2022-09-12T19:16:40.469 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-41/?versions 2022-09-12T19:16:40.470 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM= 2022-09-12T19:16:40.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM=', 'Content-Length': '0'} 2022-09-12T19:16:40.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e3cdd5de2e18506-00631f8598-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.522 DEBUG:boto:b'jdpiak-41999false' 2022-09-12T19:16:40.523 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:40.524 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:40.524 DEBUG:boto:path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.525 DEBUG:boto:auth_path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.525 DEBUG:boto:Method: GET 2022-09-12T19:16:40.526 DEBUG:boto:Path: /jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.527 DEBUG:boto:Data: 2022-09-12T19:16:40.528 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.529 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:40.530 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.531 DEBUG:boto:Params: {} 2022-09-12T19:16:40.532 DEBUG:boto:Token: None 2022-09-12T19:16:40.532 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-41/?versions 2022-09-12T19:16:40.533 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM= 2022-09-12T19:16:40.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM=', 'Content-Length': '0'} 2022-09-12T19:16:40.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007024f61cd4a1b6a8-00631f8598-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.539 DEBUG:boto:b'jdpiak-41999false' 2022-09-12T19:16:40.541 DEBUG:boto:path=/jdpiak-41/ 2022-09-12T19:16:40.542 DEBUG:boto:auth_path=/jdpiak-41/ 2022-09-12T19:16:40.542 DEBUG:boto:path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.543 DEBUG:boto:auth_path=/jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.544 DEBUG:boto:Method: GET 2022-09-12T19:16:40.545 DEBUG:boto:Path: /jdpiak-41/?versions&max-keys=999 2022-09-12T19:16:40.546 DEBUG:boto:Data: 2022-09-12T19:16:40.546 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.548 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:40.548 DEBUG:boto:Port: 8001 2022-09-12T19:16:40.549 DEBUG:boto:Params: {} 2022-09-12T19:16:40.549 DEBUG:boto:Token: None 2022-09-12T19:16:40.550 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-41/?versions 2022-09-12T19:16:40.551 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM= 2022-09-12T19:16:40.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J559Asv4GSQSEux+8flMEBq3DnM=', 'Content-Length': '0'} 2022-09-12T19:16:40.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062b2abcbf590ba73-00631f8598-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.556 DEBUG:boto:b'jdpiak-41999false' 2022-09-12T19:16:40.557 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-41 zones={test-zone2, test-zone3} 2022-09-12T19:16:40.558 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-42 zones={test-zone2, test-zone1} 2022-09-12T19:16:40.559 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:40.560 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:40.561 DEBUG:boto:Method: HEAD 2022-09-12T19:16:40.562 DEBUG:boto:Path: /jdpiak-42/ 2022-09-12T19:16:40.563 DEBUG:boto:Data: 2022-09-12T19:16:40.564 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.565 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:40.566 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.567 DEBUG:boto:Params: {} 2022-09-12T19:16:40.569 DEBUG:boto:Token: None 2022-09-12T19:16:40.570 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-42/ 2022-09-12T19:16:40.571 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P9umfhE1YLDFREppxpA114j8aoI= 2022-09-12T19:16:40.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P9umfhE1YLDFREppxpA114j8aoI=', 'Content-Length': '0'} 2022-09-12T19:16:40.575 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b71484f8ebfca3f5-00631f8598-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.577 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:40.578 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:40.579 DEBUG:boto:Method: HEAD 2022-09-12T19:16:40.580 DEBUG:boto:Path: /jdpiak-42/ 2022-09-12T19:16:40.580 DEBUG:boto:Data: 2022-09-12T19:16:40.581 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.581 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:40.582 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.582 DEBUG:boto:Params: {} 2022-09-12T19:16:40.583 DEBUG:boto:Token: None 2022-09-12T19:16:40.583 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-42/ 2022-09-12T19:16:40.584 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P9umfhE1YLDFREppxpA114j8aoI= 2022-09-12T19:16:40.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P9umfhE1YLDFREppxpA114j8aoI=', 'Content-Length': '0'} 2022-09-12T19:16:40.587 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003158415cecf631cb-00631f8598-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.588 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:40.589 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:40.589 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:40.590 DEBUG:boto:path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.590 DEBUG:boto:auth_path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.591 DEBUG:boto:Method: GET 2022-09-12T19:16:40.591 DEBUG:boto:Path: /jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.592 DEBUG:boto:Data: 2022-09-12T19:16:40.592 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.593 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:40.594 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.594 DEBUG:boto:Params: {} 2022-09-12T19:16:40.595 DEBUG:boto:Token: None 2022-09-12T19:16:40.595 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-42/?versions 2022-09-12T19:16:40.597 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U= 2022-09-12T19:16:40.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U=', 'Content-Length': '0'} 2022-09-12T19:16:40.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007852b2cd196859eb-00631f8598-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.642 DEBUG:boto:b'jdpiak-42999false' 2022-09-12T19:16:40.643 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:40.644 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:40.644 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:40.645 DEBUG:boto:path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.645 DEBUG:boto:auth_path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.646 DEBUG:boto:Method: GET 2022-09-12T19:16:40.646 DEBUG:boto:Path: /jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.648 DEBUG:boto:Data: 2022-09-12T19:16:40.649 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.649 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:40.650 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.650 DEBUG:boto:Params: {} 2022-09-12T19:16:40.651 DEBUG:boto:Token: None 2022-09-12T19:16:40.652 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-42/?versions 2022-09-12T19:16:40.652 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U= 2022-09-12T19:16:40.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U=', 'Content-Length': '0'} 2022-09-12T19:16:40.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff256cb06773945e-00631f8598-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.656 DEBUG:boto:b'jdpiak-42999false' 2022-09-12T19:16:40.658 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:40.658 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:40.659 DEBUG:boto:path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.660 DEBUG:boto:auth_path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.660 DEBUG:boto:Method: GET 2022-09-12T19:16:40.661 DEBUG:boto:Path: /jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.661 DEBUG:boto:Data: 2022-09-12T19:16:40.662 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.662 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:40.663 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.663 DEBUG:boto:Params: {} 2022-09-12T19:16:40.664 DEBUG:boto:Token: None 2022-09-12T19:16:40.665 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-42/?versions 2022-09-12T19:16:40.665 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U= 2022-09-12T19:16:40.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U=', 'Content-Length': '0'} 2022-09-12T19:16:40.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076de23bb0880c739-00631f8598-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.669 DEBUG:boto:b'jdpiak-42999false' 2022-09-12T19:16:40.670 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:40.670 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:40.671 DEBUG:boto:path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.671 DEBUG:boto:auth_path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.672 DEBUG:boto:Method: GET 2022-09-12T19:16:40.672 DEBUG:boto:Path: /jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.673 DEBUG:boto:Data: 2022-09-12T19:16:40.674 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.674 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:40.675 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.675 DEBUG:boto:Params: {} 2022-09-12T19:16:40.676 DEBUG:boto:Token: None 2022-09-12T19:16:40.677 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-42/?versions 2022-09-12T19:16:40.677 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U= 2022-09-12T19:16:40.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U=', 'Content-Length': '0'} 2022-09-12T19:16:40.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6677ad48dc6f6cd-00631f8598-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.682 DEBUG:boto:b'jdpiak-42999false' 2022-09-12T19:16:40.682 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-42 zones={test-zone2, test-zone1} 2022-09-12T19:16:40.683 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-42 zones={test-zone3, test-zone1} 2022-09-12T19:16:40.683 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:40.684 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:40.685 DEBUG:boto:Method: HEAD 2022-09-12T19:16:40.685 DEBUG:boto:Path: /jdpiak-42/ 2022-09-12T19:16:40.686 DEBUG:boto:Data: 2022-09-12T19:16:40.686 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.687 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:40.687 DEBUG:boto:Port: 8001 2022-09-12T19:16:40.688 DEBUG:boto:Params: {} 2022-09-12T19:16:40.688 DEBUG:boto:Token: None 2022-09-12T19:16:40.689 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-42/ 2022-09-12T19:16:40.690 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P9umfhE1YLDFREppxpA114j8aoI= 2022-09-12T19:16:40.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P9umfhE1YLDFREppxpA114j8aoI=', 'Content-Length': '0'} 2022-09-12T19:16:40.693 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4003303822c3eec-00631f8598-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.693 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:40.694 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:40.695 DEBUG:boto:Method: HEAD 2022-09-12T19:16:40.695 DEBUG:boto:Path: /jdpiak-42/ 2022-09-12T19:16:40.695 DEBUG:boto:Data: 2022-09-12T19:16:40.696 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.696 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:40.697 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.697 DEBUG:boto:Params: {} 2022-09-12T19:16:40.698 DEBUG:boto:Token: None 2022-09-12T19:16:40.699 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-42/ 2022-09-12T19:16:40.699 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P9umfhE1YLDFREppxpA114j8aoI= 2022-09-12T19:16:40.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P9umfhE1YLDFREppxpA114j8aoI=', 'Content-Length': '0'} 2022-09-12T19:16:40.702 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bab090add115a8fb-00631f8598-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.703 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:40.704 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:40.704 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:40.705 DEBUG:boto:path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.705 DEBUG:boto:auth_path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.706 DEBUG:boto:Method: GET 2022-09-12T19:16:40.706 DEBUG:boto:Path: /jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.707 DEBUG:boto:Data: 2022-09-12T19:16:40.707 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.708 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:40.708 DEBUG:boto:Port: 8001 2022-09-12T19:16:40.709 DEBUG:boto:Params: {} 2022-09-12T19:16:40.709 DEBUG:boto:Token: None 2022-09-12T19:16:40.710 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-42/?versions 2022-09-12T19:16:40.711 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U= 2022-09-12T19:16:40.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U=', 'Content-Length': '0'} 2022-09-12T19:16:40.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff1bd63ffcb2864f-00631f8598-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.715 DEBUG:boto:b'jdpiak-42999false' 2022-09-12T19:16:40.716 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:40.717 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:40.717 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:40.718 DEBUG:boto:path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.719 DEBUG:boto:auth_path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.719 DEBUG:boto:Method: GET 2022-09-12T19:16:40.720 DEBUG:boto:Path: /jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.721 DEBUG:boto:Data: 2022-09-12T19:16:40.722 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.723 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:40.723 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.724 DEBUG:boto:Params: {} 2022-09-12T19:16:40.725 DEBUG:boto:Token: None 2022-09-12T19:16:40.726 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-42/?versions 2022-09-12T19:16:40.727 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U= 2022-09-12T19:16:40.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U=', 'Content-Length': '0'} 2022-09-12T19:16:40.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000350eb73dde06eef-00631f8598-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.732 DEBUG:boto:b'jdpiak-42999false' 2022-09-12T19:16:40.733 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:40.734 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:40.736 DEBUG:boto:path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.739 DEBUG:boto:auth_path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.739 DEBUG:boto:Method: GET 2022-09-12T19:16:40.740 DEBUG:boto:Path: /jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.740 DEBUG:boto:Data: 2022-09-12T19:16:40.741 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.741 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:40.742 DEBUG:boto:Port: 8001 2022-09-12T19:16:40.743 DEBUG:boto:Params: {} 2022-09-12T19:16:40.743 DEBUG:boto:Token: None 2022-09-12T19:16:40.746 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-42/?versions 2022-09-12T19:16:40.746 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U= 2022-09-12T19:16:40.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U=', 'Content-Length': '0'} 2022-09-12T19:16:40.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb8e4ce1e87da315-00631f8598-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.751 DEBUG:boto:b'jdpiak-42999false' 2022-09-12T19:16:40.752 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:40.752 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:40.753 DEBUG:boto:path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.753 DEBUG:boto:auth_path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.754 DEBUG:boto:Method: GET 2022-09-12T19:16:40.754 DEBUG:boto:Path: /jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.755 DEBUG:boto:Data: 2022-09-12T19:16:40.755 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.756 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:40.756 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.757 DEBUG:boto:Params: {} 2022-09-12T19:16:40.757 DEBUG:boto:Token: None 2022-09-12T19:16:40.758 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-42/?versions 2022-09-12T19:16:40.759 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U= 2022-09-12T19:16:40.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U=', 'Content-Length': '0'} 2022-09-12T19:16:40.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab5117f084b3b9ef-00631f8598-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.763 DEBUG:boto:b'jdpiak-42999false' 2022-09-12T19:16:40.764 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-42 zones={test-zone3, test-zone1} 2022-09-12T19:16:40.764 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-42 zones={test-zone1, test-zone2} 2022-09-12T19:16:40.765 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:40.765 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:40.766 DEBUG:boto:Method: HEAD 2022-09-12T19:16:40.766 DEBUG:boto:Path: /jdpiak-42/ 2022-09-12T19:16:40.767 DEBUG:boto:Data: 2022-09-12T19:16:40.767 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.768 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:40.768 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.769 DEBUG:boto:Params: {} 2022-09-12T19:16:40.769 DEBUG:boto:Token: None 2022-09-12T19:16:40.770 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-42/ 2022-09-12T19:16:40.771 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P9umfhE1YLDFREppxpA114j8aoI= 2022-09-12T19:16:40.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P9umfhE1YLDFREppxpA114j8aoI=', 'Content-Length': '0'} 2022-09-12T19:16:40.774 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af2ad5780da0dd8d-00631f8598-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.775 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:40.775 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:40.776 DEBUG:boto:Method: HEAD 2022-09-12T19:16:40.776 DEBUG:boto:Path: /jdpiak-42/ 2022-09-12T19:16:40.777 DEBUG:boto:Data: 2022-09-12T19:16:40.777 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.778 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:40.778 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.779 DEBUG:boto:Params: {} 2022-09-12T19:16:40.779 DEBUG:boto:Token: None 2022-09-12T19:16:40.780 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-42/ 2022-09-12T19:16:40.781 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P9umfhE1YLDFREppxpA114j8aoI= 2022-09-12T19:16:40.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P9umfhE1YLDFREppxpA114j8aoI=', 'Content-Length': '0'} 2022-09-12T19:16:40.784 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a384896416a097d-00631f8598-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.784 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:40.785 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:40.786 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:40.786 DEBUG:boto:path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.787 DEBUG:boto:auth_path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.787 DEBUG:boto:Method: GET 2022-09-12T19:16:40.788 DEBUG:boto:Path: /jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.788 DEBUG:boto:Data: 2022-09-12T19:16:40.789 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.789 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:40.790 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.790 DEBUG:boto:Params: {} 2022-09-12T19:16:40.791 DEBUG:boto:Token: None 2022-09-12T19:16:40.792 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-42/?versions 2022-09-12T19:16:40.792 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U= 2022-09-12T19:16:40.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U=', 'Content-Length': '0'} 2022-09-12T19:16:40.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000832af0a5639050c7-00631f8598-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.842 DEBUG:boto:b'jdpiak-42999false' 2022-09-12T19:16:40.843 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:40.844 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:40.844 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:40.845 DEBUG:boto:path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.845 DEBUG:boto:auth_path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.846 DEBUG:boto:Method: GET 2022-09-12T19:16:40.847 DEBUG:boto:Path: /jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.847 DEBUG:boto:Data: 2022-09-12T19:16:40.848 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.848 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:40.849 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.849 DEBUG:boto:Params: {} 2022-09-12T19:16:40.850 DEBUG:boto:Token: None 2022-09-12T19:16:40.850 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-42/?versions 2022-09-12T19:16:40.851 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U= 2022-09-12T19:16:40.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U=', 'Content-Length': '0'} 2022-09-12T19:16:40.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039c632811f239c7e-00631f8598-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.859 DEBUG:boto:b'jdpiak-42999false' 2022-09-12T19:16:40.860 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:40.861 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:40.862 DEBUG:boto:path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.863 DEBUG:boto:auth_path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.863 DEBUG:boto:Method: GET 2022-09-12T19:16:40.864 DEBUG:boto:Path: /jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.865 DEBUG:boto:Data: 2022-09-12T19:16:40.866 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.867 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:40.868 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.868 DEBUG:boto:Params: {} 2022-09-12T19:16:40.869 DEBUG:boto:Token: None 2022-09-12T19:16:40.870 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-42/?versions 2022-09-12T19:16:40.871 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U= 2022-09-12T19:16:40.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U=', 'Content-Length': '0'} 2022-09-12T19:16:40.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f46fb04fcf85b3fd-00631f8598-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.875 DEBUG:boto:b'jdpiak-42999false' 2022-09-12T19:16:40.876 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:40.876 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:40.877 DEBUG:boto:path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.877 DEBUG:boto:auth_path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.878 DEBUG:boto:Method: GET 2022-09-12T19:16:40.878 DEBUG:boto:Path: /jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.879 DEBUG:boto:Data: 2022-09-12T19:16:40.880 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.880 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:40.881 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.881 DEBUG:boto:Params: {} 2022-09-12T19:16:40.882 DEBUG:boto:Token: None 2022-09-12T19:16:40.883 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-42/?versions 2022-09-12T19:16:40.883 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U= 2022-09-12T19:16:40.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U=', 'Content-Length': '0'} 2022-09-12T19:16:40.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060c5d9cc37a60479-00631f8598-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.888 DEBUG:boto:b'jdpiak-42999false' 2022-09-12T19:16:40.889 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-42 zones={test-zone1, test-zone2} 2022-09-12T19:16:40.890 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-42 zones={test-zone3, test-zone2} 2022-09-12T19:16:40.890 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:40.891 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:40.891 DEBUG:boto:Method: HEAD 2022-09-12T19:16:40.892 DEBUG:boto:Path: /jdpiak-42/ 2022-09-12T19:16:40.892 DEBUG:boto:Data: 2022-09-12T19:16:40.893 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.893 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:40.894 DEBUG:boto:Port: 8001 2022-09-12T19:16:40.894 DEBUG:boto:Params: {} 2022-09-12T19:16:40.895 DEBUG:boto:Token: None 2022-09-12T19:16:40.895 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-42/ 2022-09-12T19:16:40.896 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P9umfhE1YLDFREppxpA114j8aoI= 2022-09-12T19:16:40.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P9umfhE1YLDFREppxpA114j8aoI=', 'Content-Length': '0'} 2022-09-12T19:16:40.899 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c0f24d0f437c49bc-00631f8598-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.900 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:40.901 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:40.901 DEBUG:boto:Method: HEAD 2022-09-12T19:16:40.902 DEBUG:boto:Path: /jdpiak-42/ 2022-09-12T19:16:40.902 DEBUG:boto:Data: 2022-09-12T19:16:40.903 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.903 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:40.904 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.904 DEBUG:boto:Params: {} 2022-09-12T19:16:40.905 DEBUG:boto:Token: None 2022-09-12T19:16:40.906 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-42/ 2022-09-12T19:16:40.906 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P9umfhE1YLDFREppxpA114j8aoI= 2022-09-12T19:16:40.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P9umfhE1YLDFREppxpA114j8aoI=', 'Content-Length': '0'} 2022-09-12T19:16:40.910 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba4207ff2130da99-00631f8598-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.910 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:40.911 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:40.912 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:40.912 DEBUG:boto:path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.913 DEBUG:boto:auth_path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.913 DEBUG:boto:Method: GET 2022-09-12T19:16:40.914 DEBUG:boto:Path: /jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.914 DEBUG:boto:Data: 2022-09-12T19:16:40.915 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.915 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:40.916 DEBUG:boto:Port: 8001 2022-09-12T19:16:40.917 DEBUG:boto:Params: {} 2022-09-12T19:16:40.917 DEBUG:boto:Token: None 2022-09-12T19:16:40.918 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-42/?versions 2022-09-12T19:16:40.920 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U= 2022-09-12T19:16:40.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U=', 'Content-Length': '0'} 2022-09-12T19:16:40.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a29183cd7484b432-00631f8598-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.966 DEBUG:boto:b'jdpiak-42999false' 2022-09-12T19:16:40.967 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:40.968 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:40.968 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:40.969 DEBUG:boto:path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.969 DEBUG:boto:auth_path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.970 DEBUG:boto:Method: GET 2022-09-12T19:16:40.971 DEBUG:boto:Path: /jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.971 DEBUG:boto:Data: 2022-09-12T19:16:40.972 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.972 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:40.973 DEBUG:boto:Port: 8000 2022-09-12T19:16:40.973 DEBUG:boto:Params: {} 2022-09-12T19:16:40.974 DEBUG:boto:Token: None 2022-09-12T19:16:40.974 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-42/?versions 2022-09-12T19:16:40.975 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U= 2022-09-12T19:16:40.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U=', 'Content-Length': '0'} 2022-09-12T19:16:40.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b53be248881d498b-00631f8598-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.980 DEBUG:boto:b'jdpiak-42999false' 2022-09-12T19:16:40.980 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:40.981 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:40.981 DEBUG:boto:path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.982 DEBUG:boto:auth_path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.983 DEBUG:boto:Method: GET 2022-09-12T19:16:40.984 DEBUG:boto:Path: /jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.984 DEBUG:boto:Data: 2022-09-12T19:16:40.985 DEBUG:boto:Headers: {} 2022-09-12T19:16:40.987 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:40.988 DEBUG:boto:Port: 8001 2022-09-12T19:16:40.988 DEBUG:boto:Params: {} 2022-09-12T19:16:40.989 DEBUG:boto:Token: None 2022-09-12T19:16:40.990 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:40 GMT /jdpiak-42/?versions 2022-09-12T19:16:40.991 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U= 2022-09-12T19:16:40.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DtMqti0F+uZ7Au0lr7f1C6v1x0U=', 'Content-Length': '0'} 2022-09-12T19:16:40.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000331e19fe2e013526-00631f8598-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:40.996 DEBUG:boto:b'jdpiak-42999false' 2022-09-12T19:16:40.997 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:40.998 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:40.998 DEBUG:boto:path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.999 DEBUG:boto:auth_path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:40.999 DEBUG:boto:Method: GET 2022-09-12T19:16:41.000 DEBUG:boto:Path: /jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:41.000 DEBUG:boto:Data: 2022-09-12T19:16:41.001 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.001 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:41.002 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.002 DEBUG:boto:Params: {} 2022-09-12T19:16:41.003 DEBUG:boto:Token: None 2022-09-12T19:16:41.004 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-42/?versions 2022-09-12T19:16:41.004 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wt50JE8PFMxtqqBeaH/SdaUsjDo= 2022-09-12T19:16:41.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wt50JE8PFMxtqqBeaH/SdaUsjDo=', 'Content-Length': '0'} 2022-09-12T19:16:41.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008d76aae49dc2b03-00631f8599-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.008 DEBUG:boto:b'jdpiak-42999false' 2022-09-12T19:16:41.009 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-42 zones={test-zone3, test-zone2} 2022-09-12T19:16:41.009 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-42 zones={test-zone1, test-zone3} 2022-09-12T19:16:41.010 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:41.010 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:41.011 DEBUG:boto:Method: HEAD 2022-09-12T19:16:41.011 DEBUG:boto:Path: /jdpiak-42/ 2022-09-12T19:16:41.012 DEBUG:boto:Data: 2022-09-12T19:16:41.012 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.013 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:41.013 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.014 DEBUG:boto:Params: {} 2022-09-12T19:16:41.014 DEBUG:boto:Token: None 2022-09-12T19:16:41.015 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-42/ 2022-09-12T19:16:41.016 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OAOifXq7uReM2ddyD+EJ3JiuyZw= 2022-09-12T19:16:41.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OAOifXq7uReM2ddyD+EJ3JiuyZw=', 'Content-Length': '0'} 2022-09-12T19:16:41.019 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069143e6d3bcd8688-00631f8599-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.020 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:41.020 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:41.021 DEBUG:boto:Method: HEAD 2022-09-12T19:16:41.023 DEBUG:boto:Path: /jdpiak-42/ 2022-09-12T19:16:41.024 DEBUG:boto:Data: 2022-09-12T19:16:41.025 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.025 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:41.027 DEBUG:boto:Port: 8001 2022-09-12T19:16:41.028 DEBUG:boto:Params: {} 2022-09-12T19:16:41.029 DEBUG:boto:Token: None 2022-09-12T19:16:41.030 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-42/ 2022-09-12T19:16:41.031 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OAOifXq7uReM2ddyD+EJ3JiuyZw= 2022-09-12T19:16:41.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OAOifXq7uReM2ddyD+EJ3JiuyZw=', 'Content-Length': '0'} 2022-09-12T19:16:41.037 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a35ad70bb44fc092-00631f8599-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.038 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:41.039 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:41.040 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:41.041 DEBUG:boto:path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:41.043 DEBUG:boto:auth_path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:41.044 DEBUG:boto:Method: GET 2022-09-12T19:16:41.045 DEBUG:boto:Path: /jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:41.046 DEBUG:boto:Data: 2022-09-12T19:16:41.047 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.048 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:41.049 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.050 DEBUG:boto:Params: {} 2022-09-12T19:16:41.051 DEBUG:boto:Token: None 2022-09-12T19:16:41.052 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-42/?versions 2022-09-12T19:16:41.053 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wt50JE8PFMxtqqBeaH/SdaUsjDo= 2022-09-12T19:16:41.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wt50JE8PFMxtqqBeaH/SdaUsjDo=', 'Content-Length': '0'} 2022-09-12T19:16:41.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d89ec461dc6bfd4-00631f8599-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.060 DEBUG:boto:b'jdpiak-42999false' 2022-09-12T19:16:41.061 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:41.061 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:41.062 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:41.062 DEBUG:boto:path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:41.063 DEBUG:boto:auth_path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:41.063 DEBUG:boto:Method: GET 2022-09-12T19:16:41.064 DEBUG:boto:Path: /jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:41.064 DEBUG:boto:Data: 2022-09-12T19:16:41.065 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.065 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:41.066 DEBUG:boto:Port: 8001 2022-09-12T19:16:41.067 DEBUG:boto:Params: {} 2022-09-12T19:16:41.067 DEBUG:boto:Token: None 2022-09-12T19:16:41.068 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-42/?versions 2022-09-12T19:16:41.068 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wt50JE8PFMxtqqBeaH/SdaUsjDo= 2022-09-12T19:16:41.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wt50JE8PFMxtqqBeaH/SdaUsjDo=', 'Content-Length': '0'} 2022-09-12T19:16:41.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058f242fc2e4ff452-00631f8599-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.074 DEBUG:boto:b'jdpiak-42999false' 2022-09-12T19:16:41.075 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:41.076 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:41.077 DEBUG:boto:path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:41.078 DEBUG:boto:auth_path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:41.079 DEBUG:boto:Method: GET 2022-09-12T19:16:41.079 DEBUG:boto:Path: /jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:41.080 DEBUG:boto:Data: 2022-09-12T19:16:41.081 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.082 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:41.083 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.084 DEBUG:boto:Params: {} 2022-09-12T19:16:41.085 DEBUG:boto:Token: None 2022-09-12T19:16:41.086 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-42/?versions 2022-09-12T19:16:41.087 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wt50JE8PFMxtqqBeaH/SdaUsjDo= 2022-09-12T19:16:41.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wt50JE8PFMxtqqBeaH/SdaUsjDo=', 'Content-Length': '0'} 2022-09-12T19:16:41.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e96a8f71004f613-00631f8599-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.134 DEBUG:boto:b'jdpiak-42999false' 2022-09-12T19:16:41.136 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:41.137 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:41.138 DEBUG:boto:path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:41.139 DEBUG:boto:auth_path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:41.140 DEBUG:boto:Method: GET 2022-09-12T19:16:41.141 DEBUG:boto:Path: /jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:41.142 DEBUG:boto:Data: 2022-09-12T19:16:41.143 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.144 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:41.144 DEBUG:boto:Port: 8001 2022-09-12T19:16:41.145 DEBUG:boto:Params: {} 2022-09-12T19:16:41.146 DEBUG:boto:Token: None 2022-09-12T19:16:41.147 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-42/?versions 2022-09-12T19:16:41.148 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wt50JE8PFMxtqqBeaH/SdaUsjDo= 2022-09-12T19:16:41.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wt50JE8PFMxtqqBeaH/SdaUsjDo=', 'Content-Length': '0'} 2022-09-12T19:16:41.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ee5566b29851373-00631f8599-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.155 DEBUG:boto:b'jdpiak-42999false' 2022-09-12T19:16:41.156 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-42 zones={test-zone1, test-zone3} 2022-09-12T19:16:41.157 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-42 zones={test-zone2, test-zone3} 2022-09-12T19:16:41.158 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:41.159 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:41.160 DEBUG:boto:Method: HEAD 2022-09-12T19:16:41.161 DEBUG:boto:Path: /jdpiak-42/ 2022-09-12T19:16:41.161 DEBUG:boto:Data: 2022-09-12T19:16:41.162 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.162 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:41.163 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.163 DEBUG:boto:Params: {} 2022-09-12T19:16:41.164 DEBUG:boto:Token: None 2022-09-12T19:16:41.164 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-42/ 2022-09-12T19:16:41.165 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OAOifXq7uReM2ddyD+EJ3JiuyZw= 2022-09-12T19:16:41.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OAOifXq7uReM2ddyD+EJ3JiuyZw=', 'Content-Length': '0'} 2022-09-12T19:16:41.169 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066e7078d77a74932-00631f8599-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.169 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:41.170 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:41.170 DEBUG:boto:Method: HEAD 2022-09-12T19:16:41.171 DEBUG:boto:Path: /jdpiak-42/ 2022-09-12T19:16:41.171 DEBUG:boto:Data: 2022-09-12T19:16:41.172 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.172 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:41.173 DEBUG:boto:Port: 8001 2022-09-12T19:16:41.173 DEBUG:boto:Params: {} 2022-09-12T19:16:41.174 DEBUG:boto:Token: None 2022-09-12T19:16:41.175 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-42/ 2022-09-12T19:16:41.175 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OAOifXq7uReM2ddyD+EJ3JiuyZw= 2022-09-12T19:16:41.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OAOifXq7uReM2ddyD+EJ3JiuyZw=', 'Content-Length': '0'} 2022-09-12T19:16:41.178 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c38cf3ffa79d9f6d-00631f8599-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.179 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:41.180 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:41.180 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:41.181 DEBUG:boto:path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:41.181 DEBUG:boto:auth_path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:41.182 DEBUG:boto:Method: GET 2022-09-12T19:16:41.182 DEBUG:boto:Path: /jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:41.183 DEBUG:boto:Data: 2022-09-12T19:16:41.184 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.184 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:41.185 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.185 DEBUG:boto:Params: {} 2022-09-12T19:16:41.186 DEBUG:boto:Token: None 2022-09-12T19:16:41.186 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-42/?versions 2022-09-12T19:16:41.187 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wt50JE8PFMxtqqBeaH/SdaUsjDo= 2022-09-12T19:16:41.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wt50JE8PFMxtqqBeaH/SdaUsjDo=', 'Content-Length': '0'} 2022-09-12T19:16:41.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d91ebfd77f2b8d2-00631f8599-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.191 DEBUG:boto:b'jdpiak-42999false' 2022-09-12T19:16:41.192 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:41.192 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:41.193 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:41.193 DEBUG:boto:path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:41.194 DEBUG:boto:auth_path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:41.195 DEBUG:boto:Method: GET 2022-09-12T19:16:41.195 DEBUG:boto:Path: /jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:41.196 DEBUG:boto:Data: 2022-09-12T19:16:41.196 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.197 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:41.197 DEBUG:boto:Port: 8001 2022-09-12T19:16:41.198 DEBUG:boto:Params: {} 2022-09-12T19:16:41.198 DEBUG:boto:Token: None 2022-09-12T19:16:41.199 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-42/?versions 2022-09-12T19:16:41.200 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wt50JE8PFMxtqqBeaH/SdaUsjDo= 2022-09-12T19:16:41.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wt50JE8PFMxtqqBeaH/SdaUsjDo=', 'Content-Length': '0'} 2022-09-12T19:16:41.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e72b92ca9eb9cd4a-00631f8599-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.246 DEBUG:boto:b'jdpiak-42999false' 2022-09-12T19:16:41.247 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:41.248 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:41.248 DEBUG:boto:path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:41.249 DEBUG:boto:auth_path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:41.249 DEBUG:boto:Method: GET 2022-09-12T19:16:41.250 DEBUG:boto:Path: /jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:41.250 DEBUG:boto:Data: 2022-09-12T19:16:41.251 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.251 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:41.252 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.253 DEBUG:boto:Params: {} 2022-09-12T19:16:41.253 DEBUG:boto:Token: None 2022-09-12T19:16:41.254 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-42/?versions 2022-09-12T19:16:41.254 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wt50JE8PFMxtqqBeaH/SdaUsjDo= 2022-09-12T19:16:41.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wt50JE8PFMxtqqBeaH/SdaUsjDo=', 'Content-Length': '0'} 2022-09-12T19:16:41.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a095ffc72d5a97f-00631f8599-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.258 DEBUG:boto:b'jdpiak-42999false' 2022-09-12T19:16:41.259 DEBUG:boto:path=/jdpiak-42/ 2022-09-12T19:16:41.260 DEBUG:boto:auth_path=/jdpiak-42/ 2022-09-12T19:16:41.260 DEBUG:boto:path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:41.261 DEBUG:boto:auth_path=/jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:41.261 DEBUG:boto:Method: GET 2022-09-12T19:16:41.262 DEBUG:boto:Path: /jdpiak-42/?versions&max-keys=999 2022-09-12T19:16:41.262 DEBUG:boto:Data: 2022-09-12T19:16:41.263 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.263 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:41.264 DEBUG:boto:Port: 8001 2022-09-12T19:16:41.265 DEBUG:boto:Params: {} 2022-09-12T19:16:41.266 DEBUG:boto:Token: None 2022-09-12T19:16:41.266 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-42/?versions 2022-09-12T19:16:41.267 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wt50JE8PFMxtqqBeaH/SdaUsjDo= 2022-09-12T19:16:41.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wt50JE8PFMxtqqBeaH/SdaUsjDo=', 'Content-Length': '0'} 2022-09-12T19:16:41.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068fad2c9b25fbeab-00631f8599-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.271 DEBUG:boto:b'jdpiak-42999false' 2022-09-12T19:16:41.272 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-42 zones={test-zone2, test-zone3} 2022-09-12T19:16:41.273 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-43 zones={test-zone2, test-zone1} 2022-09-12T19:16:41.274 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.275 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.276 DEBUG:boto:Method: HEAD 2022-09-12T19:16:41.277 DEBUG:boto:Path: /jdpiak-43/ 2022-09-12T19:16:41.278 DEBUG:boto:Data: 2022-09-12T19:16:41.279 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.280 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:41.281 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.282 DEBUG:boto:Params: {} 2022-09-12T19:16:41.283 DEBUG:boto:Token: None 2022-09-12T19:16:41.284 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/ 2022-09-12T19:16:41.285 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0sO+Kle2deunPxiFkucUGPc/Tw= 2022-09-12T19:16:41.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0sO+Kle2deunPxiFkucUGPc/Tw=', 'Content-Length': '0'} 2022-09-12T19:16:41.289 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d37d0cd61fde04c-00631f8599-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.291 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.292 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.293 DEBUG:boto:Method: HEAD 2022-09-12T19:16:41.293 DEBUG:boto:Path: /jdpiak-43/ 2022-09-12T19:16:41.294 DEBUG:boto:Data: 2022-09-12T19:16:41.295 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.296 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:41.297 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.298 DEBUG:boto:Params: {} 2022-09-12T19:16:41.299 DEBUG:boto:Token: None 2022-09-12T19:16:41.300 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/ 2022-09-12T19:16:41.301 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0sO+Kle2deunPxiFkucUGPc/Tw= 2022-09-12T19:16:41.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0sO+Kle2deunPxiFkucUGPc/Tw=', 'Content-Length': '0'} 2022-09-12T19:16:41.306 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6aaf8448a7a3dfb-00631f8599-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.307 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:41.308 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.308 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.309 DEBUG:boto:path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.309 DEBUG:boto:auth_path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.310 DEBUG:boto:Method: GET 2022-09-12T19:16:41.310 DEBUG:boto:Path: /jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.311 DEBUG:boto:Data: 2022-09-12T19:16:41.311 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.312 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:41.312 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.313 DEBUG:boto:Params: {} 2022-09-12T19:16:41.314 DEBUG:boto:Token: None 2022-09-12T19:16:41.316 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/?versions 2022-09-12T19:16:41.317 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk= 2022-09-12T19:16:41.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk=', 'Content-Length': '0'} 2022-09-12T19:16:41.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b83828477cdb0551-00631f8599-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.322 DEBUG:boto:b'jdpiak-43999false' 2022-09-12T19:16:41.323 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:41.324 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.325 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.326 DEBUG:boto:path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.327 DEBUG:boto:auth_path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.328 DEBUG:boto:Method: GET 2022-09-12T19:16:41.329 DEBUG:boto:Path: /jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.330 DEBUG:boto:Data: 2022-09-12T19:16:41.331 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.332 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:41.333 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.333 DEBUG:boto:Params: {} 2022-09-12T19:16:41.335 DEBUG:boto:Token: None 2022-09-12T19:16:41.336 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/?versions 2022-09-12T19:16:41.337 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk= 2022-09-12T19:16:41.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk=', 'Content-Length': '0'} 2022-09-12T19:16:41.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000328546934da49841-00631f8599-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.342 DEBUG:boto:b'jdpiak-43999false' 2022-09-12T19:16:41.344 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.345 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.346 DEBUG:boto:path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.347 DEBUG:boto:auth_path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.348 DEBUG:boto:Method: GET 2022-09-12T19:16:41.349 DEBUG:boto:Path: /jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.349 DEBUG:boto:Data: 2022-09-12T19:16:41.350 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.351 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:41.352 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.353 DEBUG:boto:Params: {} 2022-09-12T19:16:41.354 DEBUG:boto:Token: None 2022-09-12T19:16:41.355 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/?versions 2022-09-12T19:16:41.356 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk= 2022-09-12T19:16:41.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk=', 'Content-Length': '0'} 2022-09-12T19:16:41.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063b10651e769368e-00631f8599-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.361 DEBUG:boto:b'jdpiak-43999false' 2022-09-12T19:16:41.362 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.363 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.364 DEBUG:boto:path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.365 DEBUG:boto:auth_path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.365 DEBUG:boto:Method: GET 2022-09-12T19:16:41.366 DEBUG:boto:Path: /jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.367 DEBUG:boto:Data: 2022-09-12T19:16:41.368 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.369 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:41.369 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.370 DEBUG:boto:Params: {} 2022-09-12T19:16:41.371 DEBUG:boto:Token: None 2022-09-12T19:16:41.373 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/?versions 2022-09-12T19:16:41.374 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk= 2022-09-12T19:16:41.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk=', 'Content-Length': '0'} 2022-09-12T19:16:41.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095450c33fb3687af-00631f8599-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.378 DEBUG:boto:b'jdpiak-43999false' 2022-09-12T19:16:41.379 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-43 zones={test-zone2, test-zone1} 2022-09-12T19:16:41.380 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-43 zones={test-zone3, test-zone1} 2022-09-12T19:16:41.380 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.381 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.382 DEBUG:boto:Method: HEAD 2022-09-12T19:16:41.383 DEBUG:boto:Path: /jdpiak-43/ 2022-09-12T19:16:41.384 DEBUG:boto:Data: 2022-09-12T19:16:41.385 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.385 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:41.386 DEBUG:boto:Port: 8001 2022-09-12T19:16:41.387 DEBUG:boto:Params: {} 2022-09-12T19:16:41.388 DEBUG:boto:Token: None 2022-09-12T19:16:41.389 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/ 2022-09-12T19:16:41.390 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0sO+Kle2deunPxiFkucUGPc/Tw= 2022-09-12T19:16:41.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0sO+Kle2deunPxiFkucUGPc/Tw=', 'Content-Length': '0'} 2022-09-12T19:16:41.394 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c0d0017c2cead5c-00631f8599-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.394 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.395 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.395 DEBUG:boto:Method: HEAD 2022-09-12T19:16:41.396 DEBUG:boto:Path: /jdpiak-43/ 2022-09-12T19:16:41.396 DEBUG:boto:Data: 2022-09-12T19:16:41.397 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.397 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:41.398 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.398 DEBUG:boto:Params: {} 2022-09-12T19:16:41.399 DEBUG:boto:Token: None 2022-09-12T19:16:41.400 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/ 2022-09-12T19:16:41.400 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0sO+Kle2deunPxiFkucUGPc/Tw= 2022-09-12T19:16:41.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0sO+Kle2deunPxiFkucUGPc/Tw=', 'Content-Length': '0'} 2022-09-12T19:16:41.404 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dada60998291e9d8-00631f8599-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.404 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:41.405 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.406 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.406 DEBUG:boto:path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.407 DEBUG:boto:auth_path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.408 DEBUG:boto:Method: GET 2022-09-12T19:16:41.408 DEBUG:boto:Path: /jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.409 DEBUG:boto:Data: 2022-09-12T19:16:41.409 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.410 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:41.410 DEBUG:boto:Port: 8001 2022-09-12T19:16:41.411 DEBUG:boto:Params: {} 2022-09-12T19:16:41.411 DEBUG:boto:Token: None 2022-09-12T19:16:41.412 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/?versions 2022-09-12T19:16:41.413 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk= 2022-09-12T19:16:41.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk=', 'Content-Length': '0'} 2022-09-12T19:16:41.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034346f366c11c219-00631f8599-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.417 DEBUG:boto:b'jdpiak-43999false' 2022-09-12T19:16:41.417 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:41.418 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.419 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.419 DEBUG:boto:path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.420 DEBUG:boto:auth_path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.420 DEBUG:boto:Method: GET 2022-09-12T19:16:41.421 DEBUG:boto:Path: /jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.422 DEBUG:boto:Data: 2022-09-12T19:16:41.422 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.423 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:41.423 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.424 DEBUG:boto:Params: {} 2022-09-12T19:16:41.425 DEBUG:boto:Token: None 2022-09-12T19:16:41.426 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/?versions 2022-09-12T19:16:41.427 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk= 2022-09-12T19:16:41.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk=', 'Content-Length': '0'} 2022-09-12T19:16:41.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cf4f5d6589c84ea-00631f8599-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.432 DEBUG:boto:b'jdpiak-43999false' 2022-09-12T19:16:41.433 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.434 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.435 DEBUG:boto:path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.436 DEBUG:boto:auth_path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.437 DEBUG:boto:Method: GET 2022-09-12T19:16:41.438 DEBUG:boto:Path: /jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.438 DEBUG:boto:Data: 2022-09-12T19:16:41.439 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.440 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:41.441 DEBUG:boto:Port: 8001 2022-09-12T19:16:41.441 DEBUG:boto:Params: {} 2022-09-12T19:16:41.442 DEBUG:boto:Token: None 2022-09-12T19:16:41.442 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/?versions 2022-09-12T19:16:41.443 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk= 2022-09-12T19:16:41.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk=', 'Content-Length': '0'} 2022-09-12T19:16:41.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009246ceb6844185d6-00631f8599-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.447 DEBUG:boto:b'jdpiak-43999false' 2022-09-12T19:16:41.448 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.449 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.449 DEBUG:boto:path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.449 DEBUG:boto:auth_path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.450 DEBUG:boto:Method: GET 2022-09-12T19:16:41.451 DEBUG:boto:Path: /jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.451 DEBUG:boto:Data: 2022-09-12T19:16:41.452 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.452 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:41.453 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.453 DEBUG:boto:Params: {} 2022-09-12T19:16:41.454 DEBUG:boto:Token: None 2022-09-12T19:16:41.454 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/?versions 2022-09-12T19:16:41.455 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk= 2022-09-12T19:16:41.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk=', 'Content-Length': '0'} 2022-09-12T19:16:41.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4285384b667f96e-00631f8599-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.459 DEBUG:boto:b'jdpiak-43999false' 2022-09-12T19:16:41.460 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-43 zones={test-zone3, test-zone1} 2022-09-12T19:16:41.461 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-43 zones={test-zone1, test-zone2} 2022-09-12T19:16:41.461 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.462 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.463 DEBUG:boto:Method: HEAD 2022-09-12T19:16:41.463 DEBUG:boto:Path: /jdpiak-43/ 2022-09-12T19:16:41.464 DEBUG:boto:Data: 2022-09-12T19:16:41.465 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.466 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:41.467 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.468 DEBUG:boto:Params: {} 2022-09-12T19:16:41.469 DEBUG:boto:Token: None 2022-09-12T19:16:41.470 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/ 2022-09-12T19:16:41.471 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0sO+Kle2deunPxiFkucUGPc/Tw= 2022-09-12T19:16:41.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0sO+Kle2deunPxiFkucUGPc/Tw=', 'Content-Length': '0'} 2022-09-12T19:16:41.475 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2a685da5aa54bb1-00631f8599-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.476 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.477 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.478 DEBUG:boto:Method: HEAD 2022-09-12T19:16:41.479 DEBUG:boto:Path: /jdpiak-43/ 2022-09-12T19:16:41.480 DEBUG:boto:Data: 2022-09-12T19:16:41.480 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.481 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:41.482 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.483 DEBUG:boto:Params: {} 2022-09-12T19:16:41.484 DEBUG:boto:Token: None 2022-09-12T19:16:41.485 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/ 2022-09-12T19:16:41.486 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0sO+Kle2deunPxiFkucUGPc/Tw= 2022-09-12T19:16:41.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0sO+Kle2deunPxiFkucUGPc/Tw=', 'Content-Length': '0'} 2022-09-12T19:16:41.490 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d42f4c9e89e4ab73-00631f8599-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.491 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:41.492 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.493 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.494 DEBUG:boto:path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.495 DEBUG:boto:auth_path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.496 DEBUG:boto:Method: GET 2022-09-12T19:16:41.497 DEBUG:boto:Path: /jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.497 DEBUG:boto:Data: 2022-09-12T19:16:41.498 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.499 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:41.500 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.501 DEBUG:boto:Params: {} 2022-09-12T19:16:41.502 DEBUG:boto:Token: None 2022-09-12T19:16:41.503 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/?versions 2022-09-12T19:16:41.504 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk= 2022-09-12T19:16:41.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk=', 'Content-Length': '0'} 2022-09-12T19:16:41.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000115e0b4f24f9054c-00631f8599-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.550 DEBUG:boto:b'jdpiak-43999false' 2022-09-12T19:16:41.552 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:41.553 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.553 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.554 DEBUG:boto:path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.555 DEBUG:boto:auth_path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.556 DEBUG:boto:Method: GET 2022-09-12T19:16:41.557 DEBUG:boto:Path: /jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.558 DEBUG:boto:Data: 2022-09-12T19:16:41.559 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.560 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:41.561 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.561 DEBUG:boto:Params: {} 2022-09-12T19:16:41.562 DEBUG:boto:Token: None 2022-09-12T19:16:41.563 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/?versions 2022-09-12T19:16:41.564 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk= 2022-09-12T19:16:41.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk=', 'Content-Length': '0'} 2022-09-12T19:16:41.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052f26bd907449367-00631f8599-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.569 DEBUG:boto:b'jdpiak-43999false' 2022-09-12T19:16:41.570 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.571 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.572 DEBUG:boto:path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.573 DEBUG:boto:auth_path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.574 DEBUG:boto:Method: GET 2022-09-12T19:16:41.575 DEBUG:boto:Path: /jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.576 DEBUG:boto:Data: 2022-09-12T19:16:41.577 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.578 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:41.579 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.579 DEBUG:boto:Params: {} 2022-09-12T19:16:41.580 DEBUG:boto:Token: None 2022-09-12T19:16:41.581 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/?versions 2022-09-12T19:16:41.582 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk= 2022-09-12T19:16:41.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk=', 'Content-Length': '0'} 2022-09-12T19:16:41.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a94b2a9a668a1b00-00631f8599-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.587 DEBUG:boto:b'jdpiak-43999false' 2022-09-12T19:16:41.589 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.589 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.590 DEBUG:boto:path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.591 DEBUG:boto:auth_path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.592 DEBUG:boto:Method: GET 2022-09-12T19:16:41.593 DEBUG:boto:Path: /jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.594 DEBUG:boto:Data: 2022-09-12T19:16:41.594 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.595 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:41.595 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.596 DEBUG:boto:Params: {} 2022-09-12T19:16:41.596 DEBUG:boto:Token: None 2022-09-12T19:16:41.597 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/?versions 2022-09-12T19:16:41.597 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk= 2022-09-12T19:16:41.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk=', 'Content-Length': '0'} 2022-09-12T19:16:41.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008972e03603f2ca23-00631f8599-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.601 DEBUG:boto:b'jdpiak-43999false' 2022-09-12T19:16:41.602 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-43 zones={test-zone1, test-zone2} 2022-09-12T19:16:41.603 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-43 zones={test-zone3, test-zone2} 2022-09-12T19:16:41.603 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.604 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.605 DEBUG:boto:Method: HEAD 2022-09-12T19:16:41.605 DEBUG:boto:Path: /jdpiak-43/ 2022-09-12T19:16:41.606 DEBUG:boto:Data: 2022-09-12T19:16:41.607 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.607 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:41.608 DEBUG:boto:Port: 8001 2022-09-12T19:16:41.608 DEBUG:boto:Params: {} 2022-09-12T19:16:41.609 DEBUG:boto:Token: None 2022-09-12T19:16:41.609 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/ 2022-09-12T19:16:41.610 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0sO+Kle2deunPxiFkucUGPc/Tw= 2022-09-12T19:16:41.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0sO+Kle2deunPxiFkucUGPc/Tw=', 'Content-Length': '0'} 2022-09-12T19:16:41.613 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bba9e86c310f828c-00631f8599-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.614 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.615 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.616 DEBUG:boto:Method: HEAD 2022-09-12T19:16:41.616 DEBUG:boto:Path: /jdpiak-43/ 2022-09-12T19:16:41.617 DEBUG:boto:Data: 2022-09-12T19:16:41.617 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.618 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:41.619 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.619 DEBUG:boto:Params: {} 2022-09-12T19:16:41.620 DEBUG:boto:Token: None 2022-09-12T19:16:41.620 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/ 2022-09-12T19:16:41.621 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0sO+Kle2deunPxiFkucUGPc/Tw= 2022-09-12T19:16:41.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0sO+Kle2deunPxiFkucUGPc/Tw=', 'Content-Length': '0'} 2022-09-12T19:16:41.625 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b65f6baf8bd5d95-00631f8599-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.625 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:41.626 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.627 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.627 DEBUG:boto:path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.628 DEBUG:boto:auth_path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.629 DEBUG:boto:Method: GET 2022-09-12T19:16:41.629 DEBUG:boto:Path: /jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.630 DEBUG:boto:Data: 2022-09-12T19:16:41.630 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.631 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:41.631 DEBUG:boto:Port: 8001 2022-09-12T19:16:41.632 DEBUG:boto:Params: {} 2022-09-12T19:16:41.633 DEBUG:boto:Token: None 2022-09-12T19:16:41.633 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/?versions 2022-09-12T19:16:41.634 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk= 2022-09-12T19:16:41.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk=', 'Content-Length': '0'} 2022-09-12T19:16:41.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc0c7d06df9a1ad0-00631f8599-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.639 DEBUG:boto:b'jdpiak-43999false' 2022-09-12T19:16:41.640 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:41.640 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.641 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.641 DEBUG:boto:path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.642 DEBUG:boto:auth_path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.642 DEBUG:boto:Method: GET 2022-09-12T19:16:41.643 DEBUG:boto:Path: /jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.643 DEBUG:boto:Data: 2022-09-12T19:16:41.644 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.644 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:41.645 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.645 DEBUG:boto:Params: {} 2022-09-12T19:16:41.646 DEBUG:boto:Token: None 2022-09-12T19:16:41.646 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/?versions 2022-09-12T19:16:41.647 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk= 2022-09-12T19:16:41.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk=', 'Content-Length': '0'} 2022-09-12T19:16:41.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cc5b50e5cce1709-00631f8599-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.651 DEBUG:boto:b'jdpiak-43999false' 2022-09-12T19:16:41.652 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.652 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.653 DEBUG:boto:path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.653 DEBUG:boto:auth_path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.654 DEBUG:boto:Method: GET 2022-09-12T19:16:41.655 DEBUG:boto:Path: /jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.655 DEBUG:boto:Data: 2022-09-12T19:16:41.656 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.656 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:41.657 DEBUG:boto:Port: 8001 2022-09-12T19:16:41.657 DEBUG:boto:Params: {} 2022-09-12T19:16:41.658 DEBUG:boto:Token: None 2022-09-12T19:16:41.659 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/?versions 2022-09-12T19:16:41.659 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk= 2022-09-12T19:16:41.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk=', 'Content-Length': '0'} 2022-09-12T19:16:41.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005159562a3aec01d3-00631f8599-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.663 DEBUG:boto:b'jdpiak-43999false' 2022-09-12T19:16:41.664 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.665 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.665 DEBUG:boto:path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.666 DEBUG:boto:auth_path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.667 DEBUG:boto:Method: GET 2022-09-12T19:16:41.667 DEBUG:boto:Path: /jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.668 DEBUG:boto:Data: 2022-09-12T19:16:41.668 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.669 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:41.669 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.670 DEBUG:boto:Params: {} 2022-09-12T19:16:41.671 DEBUG:boto:Token: None 2022-09-12T19:16:41.671 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/?versions 2022-09-12T19:16:41.672 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk= 2022-09-12T19:16:41.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk=', 'Content-Length': '0'} 2022-09-12T19:16:41.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008be94f10d4eab48-00631f8599-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.678 DEBUG:boto:b'jdpiak-43999false' 2022-09-12T19:16:41.679 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-43 zones={test-zone3, test-zone2} 2022-09-12T19:16:41.680 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-43 zones={test-zone1, test-zone3} 2022-09-12T19:16:41.681 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.682 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.682 DEBUG:boto:Method: HEAD 2022-09-12T19:16:41.683 DEBUG:boto:Path: /jdpiak-43/ 2022-09-12T19:16:41.684 DEBUG:boto:Data: 2022-09-12T19:16:41.685 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.688 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:41.688 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.689 DEBUG:boto:Params: {} 2022-09-12T19:16:41.690 DEBUG:boto:Token: None 2022-09-12T19:16:41.690 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/ 2022-09-12T19:16:41.691 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0sO+Kle2deunPxiFkucUGPc/Tw= 2022-09-12T19:16:41.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0sO+Kle2deunPxiFkucUGPc/Tw=', 'Content-Length': '0'} 2022-09-12T19:16:41.695 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec66f2df040e1615-00631f8599-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.696 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.697 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.697 DEBUG:boto:Method: HEAD 2022-09-12T19:16:41.698 DEBUG:boto:Path: /jdpiak-43/ 2022-09-12T19:16:41.698 DEBUG:boto:Data: 2022-09-12T19:16:41.699 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.699 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:41.700 DEBUG:boto:Port: 8001 2022-09-12T19:16:41.700 DEBUG:boto:Params: {} 2022-09-12T19:16:41.702 DEBUG:boto:Token: None 2022-09-12T19:16:41.703 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/ 2022-09-12T19:16:41.704 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0sO+Kle2deunPxiFkucUGPc/Tw= 2022-09-12T19:16:41.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0sO+Kle2deunPxiFkucUGPc/Tw=', 'Content-Length': '0'} 2022-09-12T19:16:41.713 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001bb1fc19157449ee-00631f8599-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.714 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:41.715 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.716 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.716 DEBUG:boto:path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.717 DEBUG:boto:auth_path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.717 DEBUG:boto:Method: GET 2022-09-12T19:16:41.718 DEBUG:boto:Path: /jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.719 DEBUG:boto:Data: 2022-09-12T19:16:41.719 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.720 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:41.721 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.722 DEBUG:boto:Params: {} 2022-09-12T19:16:41.723 DEBUG:boto:Token: None 2022-09-12T19:16:41.723 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/?versions 2022-09-12T19:16:41.724 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk= 2022-09-12T19:16:41.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk=', 'Content-Length': '0'} 2022-09-12T19:16:41.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000552096334849234d-00631f8599-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.735 DEBUG:boto:b'jdpiak-43999false' 2022-09-12T19:16:41.736 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:41.736 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.737 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.738 DEBUG:boto:path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.739 DEBUG:boto:auth_path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.740 DEBUG:boto:Method: GET 2022-09-12T19:16:41.740 DEBUG:boto:Path: /jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.741 DEBUG:boto:Data: 2022-09-12T19:16:41.742 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.742 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:41.743 DEBUG:boto:Port: 8001 2022-09-12T19:16:41.744 DEBUG:boto:Params: {} 2022-09-12T19:16:41.744 DEBUG:boto:Token: None 2022-09-12T19:16:41.745 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/?versions 2022-09-12T19:16:41.746 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk= 2022-09-12T19:16:41.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk=', 'Content-Length': '0'} 2022-09-12T19:16:41.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9e5eff6c6453c36-00631f8599-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.759 DEBUG:boto:b'jdpiak-43999false' 2022-09-12T19:16:41.760 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.761 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.761 DEBUG:boto:path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.762 DEBUG:boto:auth_path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.762 DEBUG:boto:Method: GET 2022-09-12T19:16:41.763 DEBUG:boto:Path: /jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.763 DEBUG:boto:Data: 2022-09-12T19:16:41.764 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.764 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:41.765 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.766 DEBUG:boto:Params: {} 2022-09-12T19:16:41.770 DEBUG:boto:Token: None 2022-09-12T19:16:41.771 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/?versions 2022-09-12T19:16:41.772 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk= 2022-09-12T19:16:41.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk=', 'Content-Length': '0'} 2022-09-12T19:16:41.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b14a1e9a2d24b8a-00631f8599-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.818 DEBUG:boto:b'jdpiak-43999false' 2022-09-12T19:16:41.819 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.820 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.820 DEBUG:boto:path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.821 DEBUG:boto:auth_path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.822 DEBUG:boto:Method: GET 2022-09-12T19:16:41.822 DEBUG:boto:Path: /jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.823 DEBUG:boto:Data: 2022-09-12T19:16:41.823 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.824 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:41.825 DEBUG:boto:Port: 8001 2022-09-12T19:16:41.825 DEBUG:boto:Params: {} 2022-09-12T19:16:41.827 DEBUG:boto:Token: None 2022-09-12T19:16:41.829 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/?versions 2022-09-12T19:16:41.830 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk= 2022-09-12T19:16:41.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk=', 'Content-Length': '0'} 2022-09-12T19:16:41.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003483821a7997e452-00631f8599-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.836 DEBUG:boto:b'jdpiak-43999false' 2022-09-12T19:16:41.837 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-43 zones={test-zone1, test-zone3} 2022-09-12T19:16:41.837 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-43 zones={test-zone2, test-zone3} 2022-09-12T19:16:41.838 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.839 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.839 DEBUG:boto:Method: HEAD 2022-09-12T19:16:41.840 DEBUG:boto:Path: /jdpiak-43/ 2022-09-12T19:16:41.840 DEBUG:boto:Data: 2022-09-12T19:16:41.841 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.841 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:41.842 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.842 DEBUG:boto:Params: {} 2022-09-12T19:16:41.843 DEBUG:boto:Token: None 2022-09-12T19:16:41.844 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/ 2022-09-12T19:16:41.844 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0sO+Kle2deunPxiFkucUGPc/Tw= 2022-09-12T19:16:41.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0sO+Kle2deunPxiFkucUGPc/Tw=', 'Content-Length': '0'} 2022-09-12T19:16:41.858 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000407deef815b1446f-00631f8599-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.860 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.861 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.862 DEBUG:boto:Method: HEAD 2022-09-12T19:16:41.863 DEBUG:boto:Path: /jdpiak-43/ 2022-09-12T19:16:41.864 DEBUG:boto:Data: 2022-09-12T19:16:41.864 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.865 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:41.866 DEBUG:boto:Port: 8001 2022-09-12T19:16:41.867 DEBUG:boto:Params: {} 2022-09-12T19:16:41.868 DEBUG:boto:Token: None 2022-09-12T19:16:41.868 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/ 2022-09-12T19:16:41.869 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0sO+Kle2deunPxiFkucUGPc/Tw= 2022-09-12T19:16:41.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0sO+Kle2deunPxiFkucUGPc/Tw=', 'Content-Length': '0'} 2022-09-12T19:16:41.875 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae348561bb72328f-00631f8599-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.876 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:41.876 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.877 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.877 DEBUG:boto:path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.878 DEBUG:boto:auth_path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.878 DEBUG:boto:Method: GET 2022-09-12T19:16:41.879 DEBUG:boto:Path: /jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.879 DEBUG:boto:Data: 2022-09-12T19:16:41.880 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.880 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:41.881 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.882 DEBUG:boto:Params: {} 2022-09-12T19:16:41.883 DEBUG:boto:Token: None 2022-09-12T19:16:41.884 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/?versions 2022-09-12T19:16:41.885 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk= 2022-09-12T19:16:41.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk=', 'Content-Length': '0'} 2022-09-12T19:16:41.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091d3d1490b553b39-00631f8599-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.934 DEBUG:boto:b'jdpiak-43999false' 2022-09-12T19:16:41.936 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:41.936 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.937 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.938 DEBUG:boto:path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.939 DEBUG:boto:auth_path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.939 DEBUG:boto:Method: GET 2022-09-12T19:16:41.940 DEBUG:boto:Path: /jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.940 DEBUG:boto:Data: 2022-09-12T19:16:41.941 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.941 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:41.942 DEBUG:boto:Port: 8001 2022-09-12T19:16:41.942 DEBUG:boto:Params: {} 2022-09-12T19:16:41.943 DEBUG:boto:Token: None 2022-09-12T19:16:41.944 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/?versions 2022-09-12T19:16:41.944 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk= 2022-09-12T19:16:41.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk=', 'Content-Length': '0'} 2022-09-12T19:16:41.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000392d1394180d22b9-00631f8599-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.957 DEBUG:boto:b'jdpiak-43999false' 2022-09-12T19:16:41.958 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.959 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.960 DEBUG:boto:path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.961 DEBUG:boto:auth_path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.962 DEBUG:boto:Method: GET 2022-09-12T19:16:41.963 DEBUG:boto:Path: /jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.964 DEBUG:boto:Data: 2022-09-12T19:16:41.965 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.965 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:41.966 DEBUG:boto:Port: 8000 2022-09-12T19:16:41.967 DEBUG:boto:Params: {} 2022-09-12T19:16:41.968 DEBUG:boto:Token: None 2022-09-12T19:16:41.969 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/?versions 2022-09-12T19:16:41.970 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk= 2022-09-12T19:16:41.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk=', 'Content-Length': '0'} 2022-09-12T19:16:41.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055b0c5e5ba5f44fa-00631f8599-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:41.980 DEBUG:boto:b'jdpiak-43999false' 2022-09-12T19:16:41.981 DEBUG:boto:path=/jdpiak-43/ 2022-09-12T19:16:41.987 DEBUG:boto:auth_path=/jdpiak-43/ 2022-09-12T19:16:41.987 DEBUG:boto:path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.988 DEBUG:boto:auth_path=/jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.989 DEBUG:boto:Method: GET 2022-09-12T19:16:41.990 DEBUG:boto:Path: /jdpiak-43/?versions&max-keys=999 2022-09-12T19:16:41.990 DEBUG:boto:Data: 2022-09-12T19:16:41.991 DEBUG:boto:Headers: {} 2022-09-12T19:16:41.992 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:41.992 DEBUG:boto:Port: 8001 2022-09-12T19:16:41.993 DEBUG:boto:Params: {} 2022-09-12T19:16:41.994 DEBUG:boto:Token: None 2022-09-12T19:16:41.995 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:41 GMT /jdpiak-43/?versions 2022-09-12T19:16:41.995 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk= 2022-09-12T19:16:41.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dl9JCLrPcOheL4MtF1xd+kd11Vk=', 'Content-Length': '0'} 2022-09-12T19:16:42.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb14c2ac13a20cfc-00631f8599-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.002 DEBUG:boto:b'jdpiak-43999false' 2022-09-12T19:16:42.004 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-43 zones={test-zone2, test-zone3} 2022-09-12T19:16:42.004 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-44 zones={test-zone2, test-zone1} 2022-09-12T19:16:42.005 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.006 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.007 DEBUG:boto:Method: HEAD 2022-09-12T19:16:42.010 DEBUG:boto:Path: /jdpiak-44/ 2022-09-12T19:16:42.011 DEBUG:boto:Data: 2022-09-12T19:16:42.012 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.012 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:42.013 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.014 DEBUG:boto:Params: {} 2022-09-12T19:16:42.015 DEBUG:boto:Token: None 2022-09-12T19:16:42.016 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/ 2022-09-12T19:16:42.017 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TYqjcE4wY/KjzxyZ8V01xgWyuXc= 2022-09-12T19:16:42.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TYqjcE4wY/KjzxyZ8V01xgWyuXc=', 'Content-Length': '0'} 2022-09-12T19:16:42.025 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ded04b24feb0308-00631f859a-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.027 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.027 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.028 DEBUG:boto:Method: HEAD 2022-09-12T19:16:42.029 DEBUG:boto:Path: /jdpiak-44/ 2022-09-12T19:16:42.029 DEBUG:boto:Data: 2022-09-12T19:16:42.030 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.031 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:42.032 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.032 DEBUG:boto:Params: {} 2022-09-12T19:16:42.033 DEBUG:boto:Token: None 2022-09-12T19:16:42.034 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/ 2022-09-12T19:16:42.034 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TYqjcE4wY/KjzxyZ8V01xgWyuXc= 2022-09-12T19:16:42.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TYqjcE4wY/KjzxyZ8V01xgWyuXc=', 'Content-Length': '0'} 2022-09-12T19:16:42.044 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000022c5ce8c45dfff6e-00631f859a-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.045 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:42.046 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.047 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.048 DEBUG:boto:path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.049 DEBUG:boto:auth_path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.050 DEBUG:boto:Method: GET 2022-09-12T19:16:42.051 DEBUG:boto:Path: /jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.052 DEBUG:boto:Data: 2022-09-12T19:16:42.053 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.054 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:42.054 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.055 DEBUG:boto:Params: {} 2022-09-12T19:16:42.055 DEBUG:boto:Token: None 2022-09-12T19:16:42.056 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/?versions 2022-09-12T19:16:42.057 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE= 2022-09-12T19:16:42.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE=', 'Content-Length': '0'} 2022-09-12T19:16:42.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d960ad7e4ebd5187-00631f859a-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.069 DEBUG:boto:b'jdpiak-44999false' 2022-09-12T19:16:42.070 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:42.071 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.072 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.072 DEBUG:boto:path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.074 DEBUG:boto:auth_path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.075 DEBUG:boto:Method: GET 2022-09-12T19:16:42.076 DEBUG:boto:Path: /jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.076 DEBUG:boto:Data: 2022-09-12T19:16:42.077 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.078 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:42.079 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.080 DEBUG:boto:Params: {} 2022-09-12T19:16:42.081 DEBUG:boto:Token: None 2022-09-12T19:16:42.082 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/?versions 2022-09-12T19:16:42.083 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE= 2022-09-12T19:16:42.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE=', 'Content-Length': '0'} 2022-09-12T19:16:42.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f056cb00e2fbf9f3-00631f859a-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.095 DEBUG:boto:b'jdpiak-44999false' 2022-09-12T19:16:42.096 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.097 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.098 DEBUG:boto:path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.099 DEBUG:boto:auth_path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.100 DEBUG:boto:Method: GET 2022-09-12T19:16:42.100 DEBUG:boto:Path: /jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.101 DEBUG:boto:Data: 2022-09-12T19:16:42.102 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.103 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:42.104 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.104 DEBUG:boto:Params: {} 2022-09-12T19:16:42.106 DEBUG:boto:Token: None 2022-09-12T19:16:42.107 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/?versions 2022-09-12T19:16:42.107 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE= 2022-09-12T19:16:42.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE=', 'Content-Length': '0'} 2022-09-12T19:16:42.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d63b9813ec0f9e04-00631f859a-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.113 DEBUG:boto:b'jdpiak-44999false' 2022-09-12T19:16:42.114 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.115 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.117 DEBUG:boto:path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.118 DEBUG:boto:auth_path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.119 DEBUG:boto:Method: GET 2022-09-12T19:16:42.120 DEBUG:boto:Path: /jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.121 DEBUG:boto:Data: 2022-09-12T19:16:42.122 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.123 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:42.124 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.125 DEBUG:boto:Params: {} 2022-09-12T19:16:42.126 DEBUG:boto:Token: None 2022-09-12T19:16:42.127 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/?versions 2022-09-12T19:16:42.128 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE= 2022-09-12T19:16:42.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE=', 'Content-Length': '0'} 2022-09-12T19:16:42.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049da057d69d62120-00631f859a-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.137 DEBUG:boto:b'jdpiak-44999false' 2022-09-12T19:16:42.138 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-44 zones={test-zone2, test-zone1} 2022-09-12T19:16:42.139 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-44 zones={test-zone3, test-zone1} 2022-09-12T19:16:42.139 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.140 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.141 DEBUG:boto:Method: HEAD 2022-09-12T19:16:42.142 DEBUG:boto:Path: /jdpiak-44/ 2022-09-12T19:16:42.143 DEBUG:boto:Data: 2022-09-12T19:16:42.144 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.145 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:42.146 DEBUG:boto:Port: 8001 2022-09-12T19:16:42.147 DEBUG:boto:Params: {} 2022-09-12T19:16:42.148 DEBUG:boto:Token: None 2022-09-12T19:16:42.149 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/ 2022-09-12T19:16:42.150 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TYqjcE4wY/KjzxyZ8V01xgWyuXc= 2022-09-12T19:16:42.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TYqjcE4wY/KjzxyZ8V01xgWyuXc=', 'Content-Length': '0'} 2022-09-12T19:16:42.155 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d7e0d56e7438ab5d-00631f859a-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.156 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.156 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.157 DEBUG:boto:Method: HEAD 2022-09-12T19:16:42.157 DEBUG:boto:Path: /jdpiak-44/ 2022-09-12T19:16:42.158 DEBUG:boto:Data: 2022-09-12T19:16:42.158 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.159 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:42.159 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.160 DEBUG:boto:Params: {} 2022-09-12T19:16:42.161 DEBUG:boto:Token: None 2022-09-12T19:16:42.162 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/ 2022-09-12T19:16:42.162 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TYqjcE4wY/KjzxyZ8V01xgWyuXc= 2022-09-12T19:16:42.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TYqjcE4wY/KjzxyZ8V01xgWyuXc=', 'Content-Length': '0'} 2022-09-12T19:16:42.169 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090102617e82baa40-00631f859a-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.170 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:42.170 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.172 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.172 DEBUG:boto:path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.173 DEBUG:boto:auth_path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.173 DEBUG:boto:Method: GET 2022-09-12T19:16:42.174 DEBUG:boto:Path: /jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.175 DEBUG:boto:Data: 2022-09-12T19:16:42.175 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.176 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:42.177 DEBUG:boto:Port: 8001 2022-09-12T19:16:42.177 DEBUG:boto:Params: {} 2022-09-12T19:16:42.178 DEBUG:boto:Token: None 2022-09-12T19:16:42.179 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/?versions 2022-09-12T19:16:42.179 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE= 2022-09-12T19:16:42.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE=', 'Content-Length': '0'} 2022-09-12T19:16:42.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053d2c12213c19667-00631f859a-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.186 DEBUG:boto:b'jdpiak-44999false' 2022-09-12T19:16:42.187 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:42.188 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.188 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.189 DEBUG:boto:path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.189 DEBUG:boto:auth_path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.190 DEBUG:boto:Method: GET 2022-09-12T19:16:42.191 DEBUG:boto:Path: /jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.191 DEBUG:boto:Data: 2022-09-12T19:16:42.192 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.193 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:42.193 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.194 DEBUG:boto:Params: {} 2022-09-12T19:16:42.194 DEBUG:boto:Token: None 2022-09-12T19:16:42.195 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/?versions 2022-09-12T19:16:42.195 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE= 2022-09-12T19:16:42.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE=', 'Content-Length': '0'} 2022-09-12T19:16:42.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bc492661dbe29db-00631f859a-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.244 DEBUG:boto:b'jdpiak-44999false' 2022-09-12T19:16:42.245 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.245 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.246 DEBUG:boto:path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.247 DEBUG:boto:auth_path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.247 DEBUG:boto:Method: GET 2022-09-12T19:16:42.248 DEBUG:boto:Path: /jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.249 DEBUG:boto:Data: 2022-09-12T19:16:42.249 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.250 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:42.251 DEBUG:boto:Port: 8001 2022-09-12T19:16:42.251 DEBUG:boto:Params: {} 2022-09-12T19:16:42.252 DEBUG:boto:Token: None 2022-09-12T19:16:42.254 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/?versions 2022-09-12T19:16:42.255 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE= 2022-09-12T19:16:42.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE=', 'Content-Length': '0'} 2022-09-12T19:16:42.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc042181f65a039f-00631f859a-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.263 DEBUG:boto:b'jdpiak-44999false' 2022-09-12T19:16:42.264 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.264 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.265 DEBUG:boto:path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.265 DEBUG:boto:auth_path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.266 DEBUG:boto:Method: GET 2022-09-12T19:16:42.266 DEBUG:boto:Path: /jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.267 DEBUG:boto:Data: 2022-09-12T19:16:42.267 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.268 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:42.269 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.270 DEBUG:boto:Params: {} 2022-09-12T19:16:42.270 DEBUG:boto:Token: None 2022-09-12T19:16:42.271 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/?versions 2022-09-12T19:16:42.271 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE= 2022-09-12T19:16:42.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE=', 'Content-Length': '0'} 2022-09-12T19:16:42.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ae0944350c9d3a4-00631f859a-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.278 DEBUG:boto:b'jdpiak-44999false' 2022-09-12T19:16:42.279 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-44 zones={test-zone3, test-zone1} 2022-09-12T19:16:42.280 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-44 zones={test-zone1, test-zone2} 2022-09-12T19:16:42.281 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.281 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.282 DEBUG:boto:Method: HEAD 2022-09-12T19:16:42.284 DEBUG:boto:Path: /jdpiak-44/ 2022-09-12T19:16:42.285 DEBUG:boto:Data: 2022-09-12T19:16:42.286 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.287 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:42.288 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.289 DEBUG:boto:Params: {} 2022-09-12T19:16:42.290 DEBUG:boto:Token: None 2022-09-12T19:16:42.291 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/ 2022-09-12T19:16:42.292 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TYqjcE4wY/KjzxyZ8V01xgWyuXc= 2022-09-12T19:16:42.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TYqjcE4wY/KjzxyZ8V01xgWyuXc=', 'Content-Length': '0'} 2022-09-12T19:16:42.304 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093e13c3c80f0c237-00631f859a-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.305 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.305 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.306 DEBUG:boto:Method: HEAD 2022-09-12T19:16:42.307 DEBUG:boto:Path: /jdpiak-44/ 2022-09-12T19:16:42.308 DEBUG:boto:Data: 2022-09-12T19:16:42.308 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.309 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:42.310 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.311 DEBUG:boto:Params: {} 2022-09-12T19:16:42.312 DEBUG:boto:Token: None 2022-09-12T19:16:42.312 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/ 2022-09-12T19:16:42.313 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TYqjcE4wY/KjzxyZ8V01xgWyuXc= 2022-09-12T19:16:42.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TYqjcE4wY/KjzxyZ8V01xgWyuXc=', 'Content-Length': '0'} 2022-09-12T19:16:42.317 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062ea11e95ed5160d-00631f859a-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.319 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:42.320 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.321 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.321 DEBUG:boto:path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.323 DEBUG:boto:auth_path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.324 DEBUG:boto:Method: GET 2022-09-12T19:16:42.325 DEBUG:boto:Path: /jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.326 DEBUG:boto:Data: 2022-09-12T19:16:42.327 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.327 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:42.328 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.329 DEBUG:boto:Params: {} 2022-09-12T19:16:42.330 DEBUG:boto:Token: None 2022-09-12T19:16:42.331 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/?versions 2022-09-12T19:16:42.331 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE= 2022-09-12T19:16:42.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE=', 'Content-Length': '0'} 2022-09-12T19:16:42.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031b15090197bafe4-00631f859a-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.379 DEBUG:boto:b'jdpiak-44999false' 2022-09-12T19:16:42.381 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:42.382 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.383 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.384 DEBUG:boto:path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.385 DEBUG:boto:auth_path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.385 DEBUG:boto:Method: GET 2022-09-12T19:16:42.387 DEBUG:boto:Path: /jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.388 DEBUG:boto:Data: 2022-09-12T19:16:42.388 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.389 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:42.389 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.390 DEBUG:boto:Params: {} 2022-09-12T19:16:42.391 DEBUG:boto:Token: None 2022-09-12T19:16:42.392 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/?versions 2022-09-12T19:16:42.393 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE= 2022-09-12T19:16:42.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE=', 'Content-Length': '0'} 2022-09-12T19:16:42.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c42f5e26f3066cc-00631f859a-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.398 DEBUG:boto:b'jdpiak-44999false' 2022-09-12T19:16:42.400 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.401 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.401 DEBUG:boto:path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.402 DEBUG:boto:auth_path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.403 DEBUG:boto:Method: GET 2022-09-12T19:16:42.404 DEBUG:boto:Path: /jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.405 DEBUG:boto:Data: 2022-09-12T19:16:42.405 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.406 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:42.407 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.408 DEBUG:boto:Params: {} 2022-09-12T19:16:42.409 DEBUG:boto:Token: None 2022-09-12T19:16:42.410 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/?versions 2022-09-12T19:16:42.411 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE= 2022-09-12T19:16:42.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE=', 'Content-Length': '0'} 2022-09-12T19:16:42.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5085cc39be6a87c-00631f859a-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.417 DEBUG:boto:b'jdpiak-44999false' 2022-09-12T19:16:42.419 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.419 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.420 DEBUG:boto:path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.421 DEBUG:boto:auth_path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.422 DEBUG:boto:Method: GET 2022-09-12T19:16:42.423 DEBUG:boto:Path: /jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.424 DEBUG:boto:Data: 2022-09-12T19:16:42.425 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.426 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:42.427 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.428 DEBUG:boto:Params: {} 2022-09-12T19:16:42.429 DEBUG:boto:Token: None 2022-09-12T19:16:42.430 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/?versions 2022-09-12T19:16:42.431 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE= 2022-09-12T19:16:42.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE=', 'Content-Length': '0'} 2022-09-12T19:16:42.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0b271ea03e301c9-00631f859a-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.444 DEBUG:boto:b'jdpiak-44999false' 2022-09-12T19:16:42.445 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-44 zones={test-zone1, test-zone2} 2022-09-12T19:16:42.446 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-44 zones={test-zone3, test-zone2} 2022-09-12T19:16:42.447 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.448 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.448 DEBUG:boto:Method: HEAD 2022-09-12T19:16:42.449 DEBUG:boto:Path: /jdpiak-44/ 2022-09-12T19:16:42.449 DEBUG:boto:Data: 2022-09-12T19:16:42.450 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.450 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:42.451 DEBUG:boto:Port: 8001 2022-09-12T19:16:42.451 DEBUG:boto:Params: {} 2022-09-12T19:16:42.452 DEBUG:boto:Token: None 2022-09-12T19:16:42.453 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/ 2022-09-12T19:16:42.453 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TYqjcE4wY/KjzxyZ8V01xgWyuXc= 2022-09-12T19:16:42.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TYqjcE4wY/KjzxyZ8V01xgWyuXc=', 'Content-Length': '0'} 2022-09-12T19:16:42.459 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000971bbdcb602665dd-00631f859a-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.460 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.460 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.461 DEBUG:boto:Method: HEAD 2022-09-12T19:16:42.461 DEBUG:boto:Path: /jdpiak-44/ 2022-09-12T19:16:42.462 DEBUG:boto:Data: 2022-09-12T19:16:42.463 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.463 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:42.464 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.464 DEBUG:boto:Params: {} 2022-09-12T19:16:42.465 DEBUG:boto:Token: None 2022-09-12T19:16:42.465 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/ 2022-09-12T19:16:42.466 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TYqjcE4wY/KjzxyZ8V01xgWyuXc= 2022-09-12T19:16:42.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TYqjcE4wY/KjzxyZ8V01xgWyuXc=', 'Content-Length': '0'} 2022-09-12T19:16:42.471 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b8741846ba07e28-00631f859a-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.473 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:42.474 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.474 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.475 DEBUG:boto:path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.476 DEBUG:boto:auth_path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.477 DEBUG:boto:Method: GET 2022-09-12T19:16:42.477 DEBUG:boto:Path: /jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.478 DEBUG:boto:Data: 2022-09-12T19:16:42.479 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.479 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:42.480 DEBUG:boto:Port: 8001 2022-09-12T19:16:42.480 DEBUG:boto:Params: {} 2022-09-12T19:16:42.481 DEBUG:boto:Token: None 2022-09-12T19:16:42.482 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/?versions 2022-09-12T19:16:42.482 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE= 2022-09-12T19:16:42.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE=', 'Content-Length': '0'} 2022-09-12T19:16:42.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000211784e899d009f2-00631f859a-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.486 DEBUG:boto:b'jdpiak-44999false' 2022-09-12T19:16:42.487 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:42.488 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.488 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.489 DEBUG:boto:path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.489 DEBUG:boto:auth_path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.490 DEBUG:boto:Method: GET 2022-09-12T19:16:42.490 DEBUG:boto:Path: /jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.491 DEBUG:boto:Data: 2022-09-12T19:16:42.491 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.492 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:42.492 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.493 DEBUG:boto:Params: {} 2022-09-12T19:16:42.494 DEBUG:boto:Token: None 2022-09-12T19:16:42.494 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/?versions 2022-09-12T19:16:42.495 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE= 2022-09-12T19:16:42.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE=', 'Content-Length': '0'} 2022-09-12T19:16:42.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a647793ca3c0bf9-00631f859a-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.542 DEBUG:boto:b'jdpiak-44999false' 2022-09-12T19:16:42.544 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.545 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.546 DEBUG:boto:path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.547 DEBUG:boto:auth_path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.548 DEBUG:boto:Method: GET 2022-09-12T19:16:42.549 DEBUG:boto:Path: /jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.550 DEBUG:boto:Data: 2022-09-12T19:16:42.551 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.551 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:42.552 DEBUG:boto:Port: 8001 2022-09-12T19:16:42.553 DEBUG:boto:Params: {} 2022-09-12T19:16:42.553 DEBUG:boto:Token: None 2022-09-12T19:16:42.554 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/?versions 2022-09-12T19:16:42.555 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE= 2022-09-12T19:16:42.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE=', 'Content-Length': '0'} 2022-09-12T19:16:42.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbe6ae519be5cbc1-00631f859a-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.559 DEBUG:boto:b'jdpiak-44999false' 2022-09-12T19:16:42.559 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.560 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.560 DEBUG:boto:path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.561 DEBUG:boto:auth_path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.562 DEBUG:boto:Method: GET 2022-09-12T19:16:42.562 DEBUG:boto:Path: /jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.563 DEBUG:boto:Data: 2022-09-12T19:16:42.563 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.564 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:42.564 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.565 DEBUG:boto:Params: {} 2022-09-12T19:16:42.565 DEBUG:boto:Token: None 2022-09-12T19:16:42.566 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/?versions 2022-09-12T19:16:42.567 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE= 2022-09-12T19:16:42.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE=', 'Content-Length': '0'} 2022-09-12T19:16:42.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d22dc2532bae0d11-00631f859a-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.571 DEBUG:boto:b'jdpiak-44999false' 2022-09-12T19:16:42.571 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-44 zones={test-zone3, test-zone2} 2022-09-12T19:16:42.572 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-44 zones={test-zone1, test-zone3} 2022-09-12T19:16:42.573 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.573 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.574 DEBUG:boto:Method: HEAD 2022-09-12T19:16:42.575 DEBUG:boto:Path: /jdpiak-44/ 2022-09-12T19:16:42.575 DEBUG:boto:Data: 2022-09-12T19:16:42.576 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.576 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:42.577 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.577 DEBUG:boto:Params: {} 2022-09-12T19:16:42.578 DEBUG:boto:Token: None 2022-09-12T19:16:42.579 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/ 2022-09-12T19:16:42.579 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TYqjcE4wY/KjzxyZ8V01xgWyuXc= 2022-09-12T19:16:42.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TYqjcE4wY/KjzxyZ8V01xgWyuXc=', 'Content-Length': '0'} 2022-09-12T19:16:42.583 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000775a8270d0eb4a55-00631f859a-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.583 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.584 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.585 DEBUG:boto:Method: HEAD 2022-09-12T19:16:42.585 DEBUG:boto:Path: /jdpiak-44/ 2022-09-12T19:16:42.586 DEBUG:boto:Data: 2022-09-12T19:16:42.586 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.587 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:42.588 DEBUG:boto:Port: 8001 2022-09-12T19:16:42.588 DEBUG:boto:Params: {} 2022-09-12T19:16:42.589 DEBUG:boto:Token: None 2022-09-12T19:16:42.590 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/ 2022-09-12T19:16:42.590 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TYqjcE4wY/KjzxyZ8V01xgWyuXc= 2022-09-12T19:16:42.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TYqjcE4wY/KjzxyZ8V01xgWyuXc=', 'Content-Length': '0'} 2022-09-12T19:16:42.594 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000048d09b974bec8fee-00631f859a-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.594 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:42.595 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.596 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.597 DEBUG:boto:path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.597 DEBUG:boto:auth_path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.598 DEBUG:boto:Method: GET 2022-09-12T19:16:42.599 DEBUG:boto:Path: /jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.599 DEBUG:boto:Data: 2022-09-12T19:16:42.600 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.601 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:42.601 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.602 DEBUG:boto:Params: {} 2022-09-12T19:16:42.603 DEBUG:boto:Token: None 2022-09-12T19:16:42.604 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/?versions 2022-09-12T19:16:42.604 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE= 2022-09-12T19:16:42.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE=', 'Content-Length': '0'} 2022-09-12T19:16:42.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ed1b2c617195015-00631f859a-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.609 DEBUG:boto:b'jdpiak-44999false' 2022-09-12T19:16:42.609 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:42.610 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.611 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.612 DEBUG:boto:path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.613 DEBUG:boto:auth_path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.613 DEBUG:boto:Method: GET 2022-09-12T19:16:42.614 DEBUG:boto:Path: /jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.614 DEBUG:boto:Data: 2022-09-12T19:16:42.615 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.616 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:42.616 DEBUG:boto:Port: 8001 2022-09-12T19:16:42.617 DEBUG:boto:Params: {} 2022-09-12T19:16:42.618 DEBUG:boto:Token: None 2022-09-12T19:16:42.619 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/?versions 2022-09-12T19:16:42.620 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE= 2022-09-12T19:16:42.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE=', 'Content-Length': '0'} 2022-09-12T19:16:42.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9319a32ea4176e4-00631f859a-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.670 DEBUG:boto:b'jdpiak-44999false' 2022-09-12T19:16:42.671 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.672 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.672 DEBUG:boto:path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.673 DEBUG:boto:auth_path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.673 DEBUG:boto:Method: GET 2022-09-12T19:16:42.674 DEBUG:boto:Path: /jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.674 DEBUG:boto:Data: 2022-09-12T19:16:42.675 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.675 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:42.676 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.676 DEBUG:boto:Params: {} 2022-09-12T19:16:42.677 DEBUG:boto:Token: None 2022-09-12T19:16:42.678 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/?versions 2022-09-12T19:16:42.678 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE= 2022-09-12T19:16:42.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE=', 'Content-Length': '0'} 2022-09-12T19:16:42.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a25838ea12a2e55a-00631f859a-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.683 DEBUG:boto:b'jdpiak-44999false' 2022-09-12T19:16:42.683 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.684 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.685 DEBUG:boto:path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.685 DEBUG:boto:auth_path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.686 DEBUG:boto:Method: GET 2022-09-12T19:16:42.686 DEBUG:boto:Path: /jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.687 DEBUG:boto:Data: 2022-09-12T19:16:42.688 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.688 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:42.689 DEBUG:boto:Port: 8001 2022-09-12T19:16:42.690 DEBUG:boto:Params: {} 2022-09-12T19:16:42.690 DEBUG:boto:Token: None 2022-09-12T19:16:42.691 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/?versions 2022-09-12T19:16:42.692 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE= 2022-09-12T19:16:42.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE=', 'Content-Length': '0'} 2022-09-12T19:16:42.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4900dc868b45bdb-00631f859a-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.696 DEBUG:boto:b'jdpiak-44999false' 2022-09-12T19:16:42.696 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-44 zones={test-zone1, test-zone3} 2022-09-12T19:16:42.697 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-44 zones={test-zone2, test-zone3} 2022-09-12T19:16:42.698 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.698 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.699 DEBUG:boto:Method: HEAD 2022-09-12T19:16:42.699 DEBUG:boto:Path: /jdpiak-44/ 2022-09-12T19:16:42.700 DEBUG:boto:Data: 2022-09-12T19:16:42.700 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.701 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:42.701 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.702 DEBUG:boto:Params: {} 2022-09-12T19:16:42.702 DEBUG:boto:Token: None 2022-09-12T19:16:42.703 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/ 2022-09-12T19:16:42.703 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TYqjcE4wY/KjzxyZ8V01xgWyuXc= 2022-09-12T19:16:42.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TYqjcE4wY/KjzxyZ8V01xgWyuXc=', 'Content-Length': '0'} 2022-09-12T19:16:42.707 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000032d67015ed585fcd-00631f859a-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.707 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.708 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.708 DEBUG:boto:Method: HEAD 2022-09-12T19:16:42.709 DEBUG:boto:Path: /jdpiak-44/ 2022-09-12T19:16:42.709 DEBUG:boto:Data: 2022-09-12T19:16:42.710 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.710 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:42.711 DEBUG:boto:Port: 8001 2022-09-12T19:16:42.711 DEBUG:boto:Params: {} 2022-09-12T19:16:42.712 DEBUG:boto:Token: None 2022-09-12T19:16:42.712 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/ 2022-09-12T19:16:42.713 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TYqjcE4wY/KjzxyZ8V01xgWyuXc= 2022-09-12T19:16:42.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TYqjcE4wY/KjzxyZ8V01xgWyuXc=', 'Content-Length': '0'} 2022-09-12T19:16:42.716 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a475dc0559aade53-00631f859a-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.717 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:42.717 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.718 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.718 DEBUG:boto:path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.719 DEBUG:boto:auth_path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.719 DEBUG:boto:Method: GET 2022-09-12T19:16:42.720 DEBUG:boto:Path: /jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.720 DEBUG:boto:Data: 2022-09-12T19:16:42.721 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.721 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:42.722 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.722 DEBUG:boto:Params: {} 2022-09-12T19:16:42.723 DEBUG:boto:Token: None 2022-09-12T19:16:42.723 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/?versions 2022-09-12T19:16:42.724 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE= 2022-09-12T19:16:42.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE=', 'Content-Length': '0'} 2022-09-12T19:16:42.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035b0098319c1833b-00631f859a-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.728 DEBUG:boto:b'jdpiak-44999false' 2022-09-12T19:16:42.728 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:42.729 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.729 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.730 DEBUG:boto:path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.730 DEBUG:boto:auth_path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.731 DEBUG:boto:Method: GET 2022-09-12T19:16:42.731 DEBUG:boto:Path: /jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.732 DEBUG:boto:Data: 2022-09-12T19:16:42.732 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.733 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:42.733 DEBUG:boto:Port: 8001 2022-09-12T19:16:42.734 DEBUG:boto:Params: {} 2022-09-12T19:16:42.735 DEBUG:boto:Token: None 2022-09-12T19:16:42.735 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/?versions 2022-09-12T19:16:42.736 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE= 2022-09-12T19:16:42.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE=', 'Content-Length': '0'} 2022-09-12T19:16:42.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042999cd0dba750ad-00631f859a-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.740 DEBUG:boto:b'jdpiak-44999false' 2022-09-12T19:16:42.740 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.741 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.741 DEBUG:boto:path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.742 DEBUG:boto:auth_path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.742 DEBUG:boto:Method: GET 2022-09-12T19:16:42.743 DEBUG:boto:Path: /jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.743 DEBUG:boto:Data: 2022-09-12T19:16:42.744 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.744 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:42.745 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.745 DEBUG:boto:Params: {} 2022-09-12T19:16:42.746 DEBUG:boto:Token: None 2022-09-12T19:16:42.747 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/?versions 2022-09-12T19:16:42.748 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE= 2022-09-12T19:16:42.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE=', 'Content-Length': '0'} 2022-09-12T19:16:42.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000213eb741b3e822d4-00631f859a-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.798 DEBUG:boto:b'jdpiak-44999false' 2022-09-12T19:16:42.800 DEBUG:boto:path=/jdpiak-44/ 2022-09-12T19:16:42.800 DEBUG:boto:auth_path=/jdpiak-44/ 2022-09-12T19:16:42.801 DEBUG:boto:path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.801 DEBUG:boto:auth_path=/jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.802 DEBUG:boto:Method: GET 2022-09-12T19:16:42.802 DEBUG:boto:Path: /jdpiak-44/?versions&max-keys=999 2022-09-12T19:16:42.803 DEBUG:boto:Data: 2022-09-12T19:16:42.803 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.804 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:42.804 DEBUG:boto:Port: 8001 2022-09-12T19:16:42.805 DEBUG:boto:Params: {} 2022-09-12T19:16:42.805 DEBUG:boto:Token: None 2022-09-12T19:16:42.806 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-44/?versions 2022-09-12T19:16:42.806 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE= 2022-09-12T19:16:42.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CwfgPFXOReKLwYkFq/3ZZXsWNjE=', 'Content-Length': '0'} 2022-09-12T19:16:42.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072adc353912fee82-00631f859a-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.810 DEBUG:boto:b'jdpiak-44999false' 2022-09-12T19:16:42.811 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-44 zones={test-zone2, test-zone3} 2022-09-12T19:16:42.812 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-45 zones={test-zone2, test-zone1} 2022-09-12T19:16:42.812 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:42.813 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:42.813 DEBUG:boto:Method: HEAD 2022-09-12T19:16:42.814 DEBUG:boto:Path: /jdpiak-45/ 2022-09-12T19:16:42.814 DEBUG:boto:Data: 2022-09-12T19:16:42.815 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.815 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:42.816 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.816 DEBUG:boto:Params: {} 2022-09-12T19:16:42.817 DEBUG:boto:Token: None 2022-09-12T19:16:42.817 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-45/ 2022-09-12T19:16:42.818 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1wvncN83tgM7ArkULVlK6fNgbdM= 2022-09-12T19:16:42.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1wvncN83tgM7ArkULVlK6fNgbdM=', 'Content-Length': '0'} 2022-09-12T19:16:42.821 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000789ba2fa816699c-00631f859a-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.822 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:42.822 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:42.823 DEBUG:boto:Method: HEAD 2022-09-12T19:16:42.823 DEBUG:boto:Path: /jdpiak-45/ 2022-09-12T19:16:42.824 DEBUG:boto:Data: 2022-09-12T19:16:42.824 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.825 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:42.825 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.826 DEBUG:boto:Params: {} 2022-09-12T19:16:42.827 DEBUG:boto:Token: None 2022-09-12T19:16:42.828 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-45/ 2022-09-12T19:16:42.828 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1wvncN83tgM7ArkULVlK6fNgbdM= 2022-09-12T19:16:42.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1wvncN83tgM7ArkULVlK6fNgbdM=', 'Content-Length': '0'} 2022-09-12T19:16:42.832 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003cb35eec556d376c-00631f859a-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.833 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:42.834 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:42.834 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:42.835 DEBUG:boto:path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:42.835 DEBUG:boto:auth_path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:42.836 DEBUG:boto:Method: GET 2022-09-12T19:16:42.836 DEBUG:boto:Path: /jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:42.837 DEBUG:boto:Data: 2022-09-12T19:16:42.837 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.838 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:42.839 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.839 DEBUG:boto:Params: {} 2022-09-12T19:16:42.840 DEBUG:boto:Token: None 2022-09-12T19:16:42.840 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-45/?versions 2022-09-12T19:16:42.841 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oEiIXVVIl8AfqeTy3l2T94+bFvU= 2022-09-12T19:16:42.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oEiIXVVIl8AfqeTy3l2T94+bFvU=', 'Content-Length': '0'} 2022-09-12T19:16:42.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e13c9fb8a1454e00-00631f859a-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.845 DEBUG:boto:b'jdpiak-45999false' 2022-09-12T19:16:42.845 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:42.846 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:42.846 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:42.847 DEBUG:boto:path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:42.847 DEBUG:boto:auth_path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:42.848 DEBUG:boto:Method: GET 2022-09-12T19:16:42.848 DEBUG:boto:Path: /jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:42.849 DEBUG:boto:Data: 2022-09-12T19:16:42.849 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.850 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:42.851 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.851 DEBUG:boto:Params: {} 2022-09-12T19:16:42.852 DEBUG:boto:Token: None 2022-09-12T19:16:42.852 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-45/?versions 2022-09-12T19:16:42.853 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oEiIXVVIl8AfqeTy3l2T94+bFvU= 2022-09-12T19:16:42.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oEiIXVVIl8AfqeTy3l2T94+bFvU=', 'Content-Length': '0'} 2022-09-12T19:16:42.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a412479fc7f20816-00631f859a-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.857 DEBUG:boto:b'jdpiak-45999false' 2022-09-12T19:16:42.858 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:42.858 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:42.859 DEBUG:boto:path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:42.859 DEBUG:boto:auth_path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:42.860 DEBUG:boto:Method: GET 2022-09-12T19:16:42.861 DEBUG:boto:Path: /jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:42.861 DEBUG:boto:Data: 2022-09-12T19:16:42.862 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.862 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:42.863 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.863 DEBUG:boto:Params: {} 2022-09-12T19:16:42.864 DEBUG:boto:Token: None 2022-09-12T19:16:42.864 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-45/?versions 2022-09-12T19:16:42.865 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oEiIXVVIl8AfqeTy3l2T94+bFvU= 2022-09-12T19:16:42.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oEiIXVVIl8AfqeTy3l2T94+bFvU=', 'Content-Length': '0'} 2022-09-12T19:16:42.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000221db07a0de4178e-00631f859a-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.869 DEBUG:boto:b'jdpiak-45999false' 2022-09-12T19:16:42.870 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:42.871 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:42.871 DEBUG:boto:path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:42.872 DEBUG:boto:auth_path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:42.872 DEBUG:boto:Method: GET 2022-09-12T19:16:42.873 DEBUG:boto:Path: /jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:42.873 DEBUG:boto:Data: 2022-09-12T19:16:42.874 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.875 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:42.876 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.876 DEBUG:boto:Params: {} 2022-09-12T19:16:42.877 DEBUG:boto:Token: None 2022-09-12T19:16:42.877 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-45/?versions 2022-09-12T19:16:42.878 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oEiIXVVIl8AfqeTy3l2T94+bFvU= 2022-09-12T19:16:42.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oEiIXVVIl8AfqeTy3l2T94+bFvU=', 'Content-Length': '0'} 2022-09-12T19:16:42.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e77cf1fdda10937-00631f859a-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.883 DEBUG:boto:b'jdpiak-45999false' 2022-09-12T19:16:42.884 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-45 zones={test-zone2, test-zone1} 2022-09-12T19:16:42.885 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-45 zones={test-zone3, test-zone1} 2022-09-12T19:16:42.885 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:42.886 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:42.886 DEBUG:boto:Method: HEAD 2022-09-12T19:16:42.887 DEBUG:boto:Path: /jdpiak-45/ 2022-09-12T19:16:42.888 DEBUG:boto:Data: 2022-09-12T19:16:42.888 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.889 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:42.889 DEBUG:boto:Port: 8001 2022-09-12T19:16:42.890 DEBUG:boto:Params: {} 2022-09-12T19:16:42.890 DEBUG:boto:Token: None 2022-09-12T19:16:42.891 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-45/ 2022-09-12T19:16:42.892 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1wvncN83tgM7ArkULVlK6fNgbdM= 2022-09-12T19:16:42.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1wvncN83tgM7ArkULVlK6fNgbdM=', 'Content-Length': '0'} 2022-09-12T19:16:42.895 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002379af0d05c0e871-00631f859a-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.896 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:42.896 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:42.897 DEBUG:boto:Method: HEAD 2022-09-12T19:16:42.897 DEBUG:boto:Path: /jdpiak-45/ 2022-09-12T19:16:42.898 DEBUG:boto:Data: 2022-09-12T19:16:42.898 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.899 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:42.899 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.900 DEBUG:boto:Params: {} 2022-09-12T19:16:42.900 DEBUG:boto:Token: None 2022-09-12T19:16:42.901 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-45/ 2022-09-12T19:16:42.902 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1wvncN83tgM7ArkULVlK6fNgbdM= 2022-09-12T19:16:42.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1wvncN83tgM7ArkULVlK6fNgbdM=', 'Content-Length': '0'} 2022-09-12T19:16:42.905 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023963810a81d4e11-00631f859a-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.905 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:42.906 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:42.907 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:42.907 DEBUG:boto:path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:42.908 DEBUG:boto:auth_path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:42.908 DEBUG:boto:Method: GET 2022-09-12T19:16:42.909 DEBUG:boto:Path: /jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:42.909 DEBUG:boto:Data: 2022-09-12T19:16:42.910 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.910 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:42.911 DEBUG:boto:Port: 8001 2022-09-12T19:16:42.912 DEBUG:boto:Params: {} 2022-09-12T19:16:42.913 DEBUG:boto:Token: None 2022-09-12T19:16:42.913 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-45/?versions 2022-09-12T19:16:42.914 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oEiIXVVIl8AfqeTy3l2T94+bFvU= 2022-09-12T19:16:42.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oEiIXVVIl8AfqeTy3l2T94+bFvU=', 'Content-Length': '0'} 2022-09-12T19:16:42.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005063eb025e258f47-00631f859a-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.958 DEBUG:boto:b'jdpiak-45999false' 2022-09-12T19:16:42.961 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:42.962 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:42.962 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:42.963 DEBUG:boto:path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:42.964 DEBUG:boto:auth_path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:42.965 DEBUG:boto:Method: GET 2022-09-12T19:16:42.966 DEBUG:boto:Path: /jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:42.967 DEBUG:boto:Data: 2022-09-12T19:16:42.968 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.969 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:42.970 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.971 DEBUG:boto:Params: {} 2022-09-12T19:16:42.972 DEBUG:boto:Token: None 2022-09-12T19:16:42.973 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-45/?versions 2022-09-12T19:16:42.973 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oEiIXVVIl8AfqeTy3l2T94+bFvU= 2022-09-12T19:16:42.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oEiIXVVIl8AfqeTy3l2T94+bFvU=', 'Content-Length': '0'} 2022-09-12T19:16:42.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038cffc7c5a536f0f-00631f859a-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.977 DEBUG:boto:b'jdpiak-45999false' 2022-09-12T19:16:42.978 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:42.979 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:42.979 DEBUG:boto:path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:42.980 DEBUG:boto:auth_path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:42.980 DEBUG:boto:Method: GET 2022-09-12T19:16:42.981 DEBUG:boto:Path: /jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:42.981 DEBUG:boto:Data: 2022-09-12T19:16:42.982 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.982 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:42.983 DEBUG:boto:Port: 8001 2022-09-12T19:16:42.983 DEBUG:boto:Params: {} 2022-09-12T19:16:42.984 DEBUG:boto:Token: None 2022-09-12T19:16:42.984 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-45/?versions 2022-09-12T19:16:42.985 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oEiIXVVIl8AfqeTy3l2T94+bFvU= 2022-09-12T19:16:42.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oEiIXVVIl8AfqeTy3l2T94+bFvU=', 'Content-Length': '0'} 2022-09-12T19:16:42.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd39fb055e392f60-00631f859a-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:42.989 DEBUG:boto:b'jdpiak-45999false' 2022-09-12T19:16:42.990 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:42.990 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:42.991 DEBUG:boto:path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:42.991 DEBUG:boto:auth_path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:42.992 DEBUG:boto:Method: GET 2022-09-12T19:16:42.992 DEBUG:boto:Path: /jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:42.993 DEBUG:boto:Data: 2022-09-12T19:16:42.993 DEBUG:boto:Headers: {} 2022-09-12T19:16:42.994 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:42.994 DEBUG:boto:Port: 8000 2022-09-12T19:16:42.995 DEBUG:boto:Params: {} 2022-09-12T19:16:42.995 DEBUG:boto:Token: None 2022-09-12T19:16:42.996 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:42 GMT /jdpiak-45/?versions 2022-09-12T19:16:42.997 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oEiIXVVIl8AfqeTy3l2T94+bFvU= 2022-09-12T19:16:42.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oEiIXVVIl8AfqeTy3l2T94+bFvU=', 'Content-Length': '0'} 2022-09-12T19:16:43.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f66d0b5f0874188-00631f859a-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:43.001 DEBUG:boto:b'jdpiak-45999false' 2022-09-12T19:16:43.002 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-45 zones={test-zone3, test-zone1} 2022-09-12T19:16:43.002 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-45 zones={test-zone1, test-zone2} 2022-09-12T19:16:43.003 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:43.003 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:43.004 DEBUG:boto:Method: HEAD 2022-09-12T19:16:43.004 DEBUG:boto:Path: /jdpiak-45/ 2022-09-12T19:16:43.005 DEBUG:boto:Data: 2022-09-12T19:16:43.005 DEBUG:boto:Headers: {} 2022-09-12T19:16:43.006 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:43.006 DEBUG:boto:Port: 8000 2022-09-12T19:16:43.007 DEBUG:boto:Params: {} 2022-09-12T19:16:43.007 DEBUG:boto:Token: None 2022-09-12T19:16:43.008 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:43 GMT /jdpiak-45/ 2022-09-12T19:16:43.008 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8JsmPFSnaFMbWL1w6B5okwjXZfY= 2022-09-12T19:16:43.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8JsmPFSnaFMbWL1w6B5okwjXZfY=', 'Content-Length': '0'} 2022-09-12T19:16:43.012 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f589fe0a16e65ae2-00631f859b-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:43.013 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:43.014 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:43.014 DEBUG:boto:Method: HEAD 2022-09-12T19:16:43.015 DEBUG:boto:Path: /jdpiak-45/ 2022-09-12T19:16:43.015 DEBUG:boto:Data: 2022-09-12T19:16:43.016 DEBUG:boto:Headers: {} 2022-09-12T19:16:43.016 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:43.017 DEBUG:boto:Port: 8000 2022-09-12T19:16:43.017 DEBUG:boto:Params: {} 2022-09-12T19:16:43.018 DEBUG:boto:Token: None 2022-09-12T19:16:43.019 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:43 GMT /jdpiak-45/ 2022-09-12T19:16:43.019 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8JsmPFSnaFMbWL1w6B5okwjXZfY= 2022-09-12T19:16:43.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8JsmPFSnaFMbWL1w6B5okwjXZfY=', 'Content-Length': '0'} 2022-09-12T19:16:43.024 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083dd50015debf616-00631f859b-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:43.025 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:43.025 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:43.026 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:43.026 DEBUG:boto:path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.027 DEBUG:boto:auth_path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.027 DEBUG:boto:Method: GET 2022-09-12T19:16:43.028 DEBUG:boto:Path: /jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.028 DEBUG:boto:Data: 2022-09-12T19:16:43.029 DEBUG:boto:Headers: {} 2022-09-12T19:16:43.030 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:43.030 DEBUG:boto:Port: 8000 2022-09-12T19:16:43.031 DEBUG:boto:Params: {} 2022-09-12T19:16:43.031 DEBUG:boto:Token: None 2022-09-12T19:16:43.032 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:43 GMT /jdpiak-45/?versions 2022-09-12T19:16:43.033 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk= 2022-09-12T19:16:43.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk=', 'Content-Length': '0'} 2022-09-12T19:16:43.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020f1a18d2b847ed8-00631f859b-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:43.037 DEBUG:boto:b'jdpiak-45999false' 2022-09-12T19:16:43.038 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:43.039 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:43.039 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:43.040 DEBUG:boto:path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.040 DEBUG:boto:auth_path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.041 DEBUG:boto:Method: GET 2022-09-12T19:16:43.041 DEBUG:boto:Path: /jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.042 DEBUG:boto:Data: 2022-09-12T19:16:43.042 DEBUG:boto:Headers: {} 2022-09-12T19:16:43.043 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:43.043 DEBUG:boto:Port: 8000 2022-09-12T19:16:43.044 DEBUG:boto:Params: {} 2022-09-12T19:16:43.044 DEBUG:boto:Token: None 2022-09-12T19:16:43.045 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:43 GMT /jdpiak-45/?versions 2022-09-12T19:16:43.046 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk= 2022-09-12T19:16:43.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk=', 'Content-Length': '0'} 2022-09-12T19:16:43.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd9eaeb5a348c45d-00631f859b-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:43.049 DEBUG:boto:b'jdpiak-45999false' 2022-09-12T19:16:43.050 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:43.051 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:43.051 DEBUG:boto:path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.052 DEBUG:boto:auth_path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.052 DEBUG:boto:Method: GET 2022-09-12T19:16:43.053 DEBUG:boto:Path: /jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.053 DEBUG:boto:Data: 2022-09-12T19:16:43.054 DEBUG:boto:Headers: {} 2022-09-12T19:16:43.054 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:43.055 DEBUG:boto:Port: 8000 2022-09-12T19:16:43.055 DEBUG:boto:Params: {} 2022-09-12T19:16:43.056 DEBUG:boto:Token: None 2022-09-12T19:16:43.057 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:43 GMT /jdpiak-45/?versions 2022-09-12T19:16:43.058 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk= 2022-09-12T19:16:43.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk=', 'Content-Length': '0'} 2022-09-12T19:16:43.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7ea7fd3a9363fb7-00631f859b-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:43.061 DEBUG:boto:b'jdpiak-45999false' 2022-09-12T19:16:43.062 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:43.063 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:43.063 DEBUG:boto:path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.064 DEBUG:boto:auth_path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.064 DEBUG:boto:Method: GET 2022-09-12T19:16:43.065 DEBUG:boto:Path: /jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.066 DEBUG:boto:Data: 2022-09-12T19:16:43.067 DEBUG:boto:Headers: {} 2022-09-12T19:16:43.068 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:43.069 DEBUG:boto:Port: 8000 2022-09-12T19:16:43.069 DEBUG:boto:Params: {} 2022-09-12T19:16:43.070 DEBUG:boto:Token: None 2022-09-12T19:16:43.071 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:43 GMT /jdpiak-45/?versions 2022-09-12T19:16:43.072 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk= 2022-09-12T19:16:43.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk=', 'Content-Length': '0'} 2022-09-12T19:16:43.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000160c9ca6d3cf158a-00631f859b-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:43.077 DEBUG:boto:b'jdpiak-45999false' 2022-09-12T19:16:43.078 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-45 zones={test-zone1, test-zone2} 2022-09-12T19:16:43.079 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-45 zones={test-zone3, test-zone2} 2022-09-12T19:16:43.080 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:43.080 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:43.081 DEBUG:boto:Method: HEAD 2022-09-12T19:16:43.081 DEBUG:boto:Path: /jdpiak-45/ 2022-09-12T19:16:43.082 DEBUG:boto:Data: 2022-09-12T19:16:43.083 DEBUG:boto:Headers: {} 2022-09-12T19:16:43.084 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:43.084 DEBUG:boto:Port: 8001 2022-09-12T19:16:43.085 DEBUG:boto:Params: {} 2022-09-12T19:16:43.086 DEBUG:boto:Token: None 2022-09-12T19:16:43.087 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:43 GMT /jdpiak-45/ 2022-09-12T19:16:43.087 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8JsmPFSnaFMbWL1w6B5okwjXZfY= 2022-09-12T19:16:43.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8JsmPFSnaFMbWL1w6B5okwjXZfY=', 'Content-Length': '0'} 2022-09-12T19:16:43.091 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000adf538e6c505f7e-00631f859b-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:43.091 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:43.092 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:43.093 DEBUG:boto:Method: HEAD 2022-09-12T19:16:43.094 DEBUG:boto:Path: /jdpiak-45/ 2022-09-12T19:16:43.094 DEBUG:boto:Data: 2022-09-12T19:16:43.095 DEBUG:boto:Headers: {} 2022-09-12T19:16:43.096 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:43.096 DEBUG:boto:Port: 8000 2022-09-12T19:16:43.097 DEBUG:boto:Params: {} 2022-09-12T19:16:43.097 DEBUG:boto:Token: None 2022-09-12T19:16:43.098 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:43 GMT /jdpiak-45/ 2022-09-12T19:16:43.098 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8JsmPFSnaFMbWL1w6B5okwjXZfY= 2022-09-12T19:16:43.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8JsmPFSnaFMbWL1w6B5okwjXZfY=', 'Content-Length': '0'} 2022-09-12T19:16:43.102 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000434bcd553dfd2531-00631f859b-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:43.102 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:43.103 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:43.104 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:43.104 DEBUG:boto:path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.105 DEBUG:boto:auth_path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.105 DEBUG:boto:Method: GET 2022-09-12T19:16:43.106 DEBUG:boto:Path: /jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.107 DEBUG:boto:Data: 2022-09-12T19:16:43.107 DEBUG:boto:Headers: {} 2022-09-12T19:16:43.108 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:43.108 DEBUG:boto:Port: 8001 2022-09-12T19:16:43.109 DEBUG:boto:Params: {} 2022-09-12T19:16:43.110 DEBUG:boto:Token: None 2022-09-12T19:16:43.110 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:43 GMT /jdpiak-45/?versions 2022-09-12T19:16:43.111 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk= 2022-09-12T19:16:43.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk=', 'Content-Length': '0'} 2022-09-12T19:16:43.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b2075a013938fc8-00631f859b-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:43.116 DEBUG:boto:b'jdpiak-45999false' 2022-09-12T19:16:43.116 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:43.117 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:43.117 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:43.118 DEBUG:boto:path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.119 DEBUG:boto:auth_path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.120 DEBUG:boto:Method: GET 2022-09-12T19:16:43.120 DEBUG:boto:Path: /jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.121 DEBUG:boto:Data: 2022-09-12T19:16:43.122 DEBUG:boto:Headers: {} 2022-09-12T19:16:43.122 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:43.123 DEBUG:boto:Port: 8000 2022-09-12T19:16:43.123 DEBUG:boto:Params: {} 2022-09-12T19:16:43.124 DEBUG:boto:Token: None 2022-09-12T19:16:43.124 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:43 GMT /jdpiak-45/?versions 2022-09-12T19:16:43.125 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk= 2022-09-12T19:16:43.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk=', 'Content-Length': '0'} 2022-09-12T19:16:43.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cb7de87e408579b-00631f859b-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:43.170 DEBUG:boto:b'jdpiak-45999false' 2022-09-12T19:16:43.171 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:43.171 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:43.172 DEBUG:boto:path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.173 DEBUG:boto:auth_path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.173 DEBUG:boto:Method: GET 2022-09-12T19:16:43.174 DEBUG:boto:Path: /jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.175 DEBUG:boto:Data: 2022-09-12T19:16:43.175 DEBUG:boto:Headers: {} 2022-09-12T19:16:43.176 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:43.176 DEBUG:boto:Port: 8001 2022-09-12T19:16:43.177 DEBUG:boto:Params: {} 2022-09-12T19:16:43.178 DEBUG:boto:Token: None 2022-09-12T19:16:43.178 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:43 GMT /jdpiak-45/?versions 2022-09-12T19:16:43.179 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk= 2022-09-12T19:16:43.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk=', 'Content-Length': '0'} 2022-09-12T19:16:43.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099a73aa67d5c90c7-00631f859b-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:43.183 DEBUG:boto:b'jdpiak-45999false' 2022-09-12T19:16:43.184 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:43.185 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:43.186 DEBUG:boto:path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.186 DEBUG:boto:auth_path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.187 DEBUG:boto:Method: GET 2022-09-12T19:16:43.187 DEBUG:boto:Path: /jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.188 DEBUG:boto:Data: 2022-09-12T19:16:43.188 DEBUG:boto:Headers: {} 2022-09-12T19:16:43.189 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:43.189 DEBUG:boto:Port: 8000 2022-09-12T19:16:43.190 DEBUG:boto:Params: {} 2022-09-12T19:16:43.191 DEBUG:boto:Token: None 2022-09-12T19:16:43.191 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:43 GMT /jdpiak-45/?versions 2022-09-12T19:16:43.192 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk= 2022-09-12T19:16:43.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk=', 'Content-Length': '0'} 2022-09-12T19:16:43.198 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b7c2ab5dbbfc80a-00631f859b-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:43.200 DEBUG:boto:b'jdpiak-45999false' 2022-09-12T19:16:43.202 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-45 zones={test-zone3, test-zone2} 2022-09-12T19:16:43.203 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-45 zones={test-zone1, test-zone3} 2022-09-12T19:16:43.204 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:43.206 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:43.209 DEBUG:boto:Method: HEAD 2022-09-12T19:16:43.210 DEBUG:boto:Path: /jdpiak-45/ 2022-09-12T19:16:43.211 DEBUG:boto:Data: 2022-09-12T19:16:43.212 DEBUG:boto:Headers: {} 2022-09-12T19:16:43.213 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:43.214 DEBUG:boto:Port: 8000 2022-09-12T19:16:43.215 DEBUG:boto:Params: {} 2022-09-12T19:16:43.216 DEBUG:boto:Token: None 2022-09-12T19:16:43.217 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:43 GMT /jdpiak-45/ 2022-09-12T19:16:43.218 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8JsmPFSnaFMbWL1w6B5okwjXZfY= 2022-09-12T19:16:43.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8JsmPFSnaFMbWL1w6B5okwjXZfY=', 'Content-Length': '0'} 2022-09-12T19:16:43.223 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd308fea49b20ca8-00631f859b-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:43.224 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:43.225 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:43.226 DEBUG:boto:Method: HEAD 2022-09-12T19:16:43.227 DEBUG:boto:Path: /jdpiak-45/ 2022-09-12T19:16:43.235 DEBUG:boto:Data: 2022-09-12T19:16:43.235 DEBUG:boto:Headers: {} 2022-09-12T19:16:43.237 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:43.237 DEBUG:boto:Port: 8001 2022-09-12T19:16:43.238 DEBUG:boto:Params: {} 2022-09-12T19:16:43.238 DEBUG:boto:Token: None 2022-09-12T19:16:43.239 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:43 GMT /jdpiak-45/ 2022-09-12T19:16:43.240 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8JsmPFSnaFMbWL1w6B5okwjXZfY= 2022-09-12T19:16:43.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8JsmPFSnaFMbWL1w6B5okwjXZfY=', 'Content-Length': '0'} 2022-09-12T19:16:43.243 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002087cd1be537011c-00631f859b-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:43.244 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:43.244 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:43.246 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:43.246 DEBUG:boto:path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.247 DEBUG:boto:auth_path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.248 DEBUG:boto:Method: GET 2022-09-12T19:16:43.248 DEBUG:boto:Path: /jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.249 DEBUG:boto:Data: 2022-09-12T19:16:43.251 DEBUG:boto:Headers: {} 2022-09-12T19:16:43.252 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:43.252 DEBUG:boto:Port: 8000 2022-09-12T19:16:43.253 DEBUG:boto:Params: {} 2022-09-12T19:16:43.254 DEBUG:boto:Token: None 2022-09-12T19:16:43.255 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:43 GMT /jdpiak-45/?versions 2022-09-12T19:16:43.256 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk= 2022-09-12T19:16:43.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk=', 'Content-Length': '0'} 2022-09-12T19:16:43.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000050fbb80037ef9d7-00631f859b-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:43.262 DEBUG:boto:b'jdpiak-45999false' 2022-09-12T19:16:43.263 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:43.264 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:43.265 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:43.266 DEBUG:boto:path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.266 DEBUG:boto:auth_path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.267 DEBUG:boto:Method: GET 2022-09-12T19:16:43.267 DEBUG:boto:Path: /jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.268 DEBUG:boto:Data: 2022-09-12T19:16:43.268 DEBUG:boto:Headers: {} 2022-09-12T19:16:43.269 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:43.269 DEBUG:boto:Port: 8001 2022-09-12T19:16:43.270 DEBUG:boto:Params: {} 2022-09-12T19:16:43.270 DEBUG:boto:Token: None 2022-09-12T19:16:43.271 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:43 GMT /jdpiak-45/?versions 2022-09-12T19:16:43.271 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk= 2022-09-12T19:16:43.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk=', 'Content-Length': '0'} 2022-09-12T19:16:43.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c3c74e8b94e7bf8-00631f859b-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:43.323 DEBUG:boto:b'jdpiak-45999false' 2022-09-12T19:16:43.325 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:43.325 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:43.326 DEBUG:boto:path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.327 DEBUG:boto:auth_path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.327 DEBUG:boto:Method: GET 2022-09-12T19:16:43.328 DEBUG:boto:Path: /jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.328 DEBUG:boto:Data: 2022-09-12T19:16:43.329 DEBUG:boto:Headers: {} 2022-09-12T19:16:43.329 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:43.330 DEBUG:boto:Port: 8000 2022-09-12T19:16:43.330 DEBUG:boto:Params: {} 2022-09-12T19:16:43.331 DEBUG:boto:Token: None 2022-09-12T19:16:43.332 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:43 GMT /jdpiak-45/?versions 2022-09-12T19:16:43.332 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk= 2022-09-12T19:16:43.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk=', 'Content-Length': '0'} 2022-09-12T19:16:43.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2bdc1faa520fc9f-00631f859b-1432-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:43.336 DEBUG:boto:b'jdpiak-45999false' 2022-09-12T19:16:43.337 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:43.337 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:43.338 DEBUG:boto:path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.338 DEBUG:boto:auth_path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.339 DEBUG:boto:Method: GET 2022-09-12T19:16:43.339 DEBUG:boto:Path: /jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.340 DEBUG:boto:Data: 2022-09-12T19:16:43.340 DEBUG:boto:Headers: {} 2022-09-12T19:16:43.341 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:43.342 DEBUG:boto:Port: 8001 2022-09-12T19:16:43.342 DEBUG:boto:Params: {} 2022-09-12T19:16:43.343 DEBUG:boto:Token: None 2022-09-12T19:16:43.344 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:43 GMT /jdpiak-45/?versions 2022-09-12T19:16:43.344 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk= 2022-09-12T19:16:43.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk=', 'Content-Length': '0'} 2022-09-12T19:16:43.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008a79d8846106bd6-00631f859b-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:43.349 DEBUG:boto:b'jdpiak-45999false' 2022-09-12T19:16:43.350 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-45 zones={test-zone1, test-zone3} 2022-09-12T19:16:43.351 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-45 zones={test-zone2, test-zone3} 2022-09-12T19:16:43.352 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:43.353 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:43.354 DEBUG:boto:Method: HEAD 2022-09-12T19:16:43.355 DEBUG:boto:Path: /jdpiak-45/ 2022-09-12T19:16:43.356 DEBUG:boto:Data: 2022-09-12T19:16:43.356 DEBUG:boto:Headers: {} 2022-09-12T19:16:43.357 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:43.358 DEBUG:boto:Port: 8000 2022-09-12T19:16:43.359 DEBUG:boto:Params: {} 2022-09-12T19:16:43.359 DEBUG:boto:Token: None 2022-09-12T19:16:43.360 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:43 GMT /jdpiak-45/ 2022-09-12T19:16:43.361 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8JsmPFSnaFMbWL1w6B5okwjXZfY= 2022-09-12T19:16:43.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8JsmPFSnaFMbWL1w6B5okwjXZfY=', 'Content-Length': '0'} 2022-09-12T19:16:43.364 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038974a89c6e0e73c-00631f859b-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:43.365 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:43.366 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:43.367 DEBUG:boto:Method: HEAD 2022-09-12T19:16:43.368 DEBUG:boto:Path: /jdpiak-45/ 2022-09-12T19:16:43.369 DEBUG:boto:Data: 2022-09-12T19:16:43.369 DEBUG:boto:Headers: {} 2022-09-12T19:16:43.370 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:43.371 DEBUG:boto:Port: 8001 2022-09-12T19:16:43.372 DEBUG:boto:Params: {} 2022-09-12T19:16:43.373 DEBUG:boto:Token: None 2022-09-12T19:16:43.374 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:43 GMT /jdpiak-45/ 2022-09-12T19:16:43.375 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8JsmPFSnaFMbWL1w6B5okwjXZfY= 2022-09-12T19:16:43.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8JsmPFSnaFMbWL1w6B5okwjXZfY=', 'Content-Length': '0'} 2022-09-12T19:16:43.379 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6cf6dd79d2e6335-00631f859b-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:43.380 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:16:43.381 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:43.382 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:43.383 DEBUG:boto:path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.383 DEBUG:boto:auth_path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.384 DEBUG:boto:Method: GET 2022-09-12T19:16:43.384 DEBUG:boto:Path: /jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.385 DEBUG:boto:Data: 2022-09-12T19:16:43.385 DEBUG:boto:Headers: {} 2022-09-12T19:16:43.386 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:43.386 DEBUG:boto:Port: 8000 2022-09-12T19:16:43.387 DEBUG:boto:Params: {} 2022-09-12T19:16:43.387 DEBUG:boto:Token: None 2022-09-12T19:16:43.388 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:43 GMT /jdpiak-45/?versions 2022-09-12T19:16:43.389 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk= 2022-09-12T19:16:43.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk=', 'Content-Length': '0'} 2022-09-12T19:16:43.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7cfb924edcd4405-00631f859b-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:43.394 DEBUG:boto:b'jdpiak-45999false' 2022-09-12T19:16:43.395 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:16:43.396 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:43.397 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:43.398 DEBUG:boto:path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.399 DEBUG:boto:auth_path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.400 DEBUG:boto:Method: GET 2022-09-12T19:16:43.401 DEBUG:boto:Path: /jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.401 DEBUG:boto:Data: 2022-09-12T19:16:43.402 DEBUG:boto:Headers: {} 2022-09-12T19:16:43.403 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:43.404 DEBUG:boto:Port: 8001 2022-09-12T19:16:43.404 DEBUG:boto:Params: {} 2022-09-12T19:16:43.405 DEBUG:boto:Token: None 2022-09-12T19:16:43.405 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:43 GMT /jdpiak-45/?versions 2022-09-12T19:16:43.406 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk= 2022-09-12T19:16:43.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk=', 'Content-Length': '0'} 2022-09-12T19:16:43.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbc3b8910e08ee69-00631f859b-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:43.452 DEBUG:boto:b'jdpiak-45999false' 2022-09-12T19:16:43.454 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:43.455 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:43.456 DEBUG:boto:path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.457 DEBUG:boto:auth_path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.458 DEBUG:boto:Method: GET 2022-09-12T19:16:43.459 DEBUG:boto:Path: /jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.460 DEBUG:boto:Data: 2022-09-12T19:16:43.460 DEBUG:boto:Headers: {} 2022-09-12T19:16:43.461 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:43.461 DEBUG:boto:Port: 8000 2022-09-12T19:16:43.462 DEBUG:boto:Params: {} 2022-09-12T19:16:43.463 DEBUG:boto:Token: None 2022-09-12T19:16:43.463 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:43 GMT /jdpiak-45/?versions 2022-09-12T19:16:43.464 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk= 2022-09-12T19:16:43.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk=', 'Content-Length': '0'} 2022-09-12T19:16:43.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a11f0c297635314e-00631f859b-1438-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:43.468 DEBUG:boto:b'jdpiak-45999false' 2022-09-12T19:16:43.468 DEBUG:boto:path=/jdpiak-45/ 2022-09-12T19:16:43.469 DEBUG:boto:auth_path=/jdpiak-45/ 2022-09-12T19:16:43.470 DEBUG:boto:path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.470 DEBUG:boto:auth_path=/jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.471 DEBUG:boto:Method: GET 2022-09-12T19:16:43.471 DEBUG:boto:Path: /jdpiak-45/?versions&max-keys=999 2022-09-12T19:16:43.472 DEBUG:boto:Data: 2022-09-12T19:16:43.473 DEBUG:boto:Headers: {} 2022-09-12T19:16:43.473 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:43.474 DEBUG:boto:Port: 8001 2022-09-12T19:16:43.474 DEBUG:boto:Params: {} 2022-09-12T19:16:43.475 DEBUG:boto:Token: None 2022-09-12T19:16:43.475 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:16:43 GMT /jdpiak-45/?versions 2022-09-12T19:16:43.476 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk= 2022-09-12T19:16:43.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:43 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xViDkh1/1ZC7oFC06+xWGd8gxFk=', 'Content-Length': '0'} 2022-09-12T19:16:43.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000399d5e28aeacf13f-00631f859b-1438-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:16:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:43.483 DEBUG:boto:b'jdpiak-45999false' 2022-09-12T19:16:43.484 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-45 zones={test-zone2, test-zone3} 2022-09-12T19:16:43.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'c0f7a512-c940-4d1f-8adf-3484630f876b', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:43.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'c0f7a512-c940-4d1f-8adf-3484630f876b', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:43.487 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period c0f7a512-c940-4d1f-8adf-3484630f876b --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:43.546 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:43.547 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:43.605 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:16:43.606 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.606 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008447.969948_52.1", 2022-09-12T19:16:43.606 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "user", 2022-09-12T19:16:43.607 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "rgw-multisite-test-user", 2022-09-12T19:16:43.607 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.969948859Z", 2022-09-12T19:16:43.608 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.608 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.608 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.609 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.609 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.609 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.610 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_X5YlnCJyC_iIiZ1qmhasQLd", 2022-09-12T19:16:43.611 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.611 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.611 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.612 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.612 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.612 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.612 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.613 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.613 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008447.976965_53.1", 2022-09-12T19:16:43.614 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "user", 2022-09-12T19:16:43.614 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "rgw-multisite-test-user", 2022-09-12T19:16:43.615 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965355Z", 2022-09-12T19:16:43.615 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.616 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.616 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.616 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.617 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.617 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.618 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_X5YlnCJyC_iIiZ1qmhasQLd", 2022-09-12T19:16:43.618 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.618 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.619 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.619 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.620 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.620 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.620 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.621 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.621 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008586.711103_57.1", 2022-09-12T19:16:43.622 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.622 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10", 2022-09-12T19:16:43.623 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:46.711103783Z", 2022-09-12T19:16:43.623 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.623 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.623 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.624 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.624 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.624 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.624 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_0WIw9i-PdtM0sq6xV05tENT", 2022-09-12T19:16:43.625 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.625 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.625 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.625 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.626 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.626 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.626 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.626 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.627 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008586.714487_58.1", 2022-09-12T19:16:43.627 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.627 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10", 2022-09-12T19:16:43.627 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:46.714487285Z", 2022-09-12T19:16:43.628 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.628 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.628 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.628 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.629 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.629 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.629 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_0WIw9i-PdtM0sq6xV05tENT", 2022-09-12T19:16:43.629 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.629 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.630 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.630 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.630 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.630 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.631 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.631 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.631 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008586.715813_59.1", 2022-09-12T19:16:43.631 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.632 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-10", 2022-09-12T19:16:43.632 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:46.715813657Z", 2022-09-12T19:16:43.632 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.632 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.633 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.633 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.633 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.633 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.634 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_nPF7oiiG3E2RTyZZOeDLhOv", 2022-09-12T19:16:43.634 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.634 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.634 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.635 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.635 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.635 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.635 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.636 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.636 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008586.718771_60.1", 2022-09-12T19:16:43.636 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.636 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-10", 2022-09-12T19:16:43.637 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:46.718771977Z", 2022-09-12T19:16:43.637 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.637 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.637 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.637 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.638 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.638 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.638 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_nPF7oiiG3E2RTyZZOeDLhOv", 2022-09-12T19:16:43.638 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.639 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.639 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.639 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.639 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.640 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.640 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.640 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.640 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008586.736648_61.1", 2022-09-12T19:16:43.641 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.641 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11", 2022-09-12T19:16:43.641 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:46.736648541Z", 2022-09-12T19:16:43.641 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.642 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.642 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.642 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.642 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.643 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.643 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_DXlcSXme1LlpozfB5A2QmS2", 2022-09-12T19:16:43.643 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.643 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.644 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.644 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.644 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.644 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.645 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.645 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.645 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008586.739749_62.1", 2022-09-12T19:16:43.645 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.646 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11", 2022-09-12T19:16:43.646 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:46.739749085Z", 2022-09-12T19:16:43.646 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.646 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.647 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.647 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.647 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.647 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.648 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_DXlcSXme1LlpozfB5A2QmS2", 2022-09-12T19:16:43.648 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.648 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.648 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.649 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.649 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.649 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.650 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.650 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.650 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008586.740926_63.1", 2022-09-12T19:16:43.650 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.651 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-11", 2022-09-12T19:16:43.651 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:46.740926183Z", 2022-09-12T19:16:43.651 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.651 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.652 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.652 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.652 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.653 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.653 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_QTYYWzCEPLTd7eDR1Si_D1k", 2022-09-12T19:16:43.653 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.653 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.654 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.654 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.654 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.654 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.655 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.655 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.655 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008586.744033_64.1", 2022-09-12T19:16:43.655 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.656 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-11", 2022-09-12T19:16:43.656 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:46.744033830Z", 2022-09-12T19:16:43.656 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.656 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.657 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.657 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.657 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.657 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.658 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_QTYYWzCEPLTd7eDR1Si_D1k", 2022-09-12T19:16:43.658 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.658 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.658 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.659 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.659 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.660 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.660 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.660 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.661 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008586.773460_65.1", 2022-09-12T19:16:43.661 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.661 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12", 2022-09-12T19:16:43.661 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:46.773460810Z", 2022-09-12T19:16:43.662 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.662 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.662 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.662 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.663 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.663 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.663 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_jMUPDDQJMus4ft3ku4UXAZe", 2022-09-12T19:16:43.663 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.664 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.664 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.664 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.664 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.665 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.665 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.665 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.665 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008586.776584_66.1", 2022-09-12T19:16:43.666 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.666 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12", 2022-09-12T19:16:43.666 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:46.776584217Z", 2022-09-12T19:16:43.666 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.667 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.667 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.667 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.667 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.668 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.668 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_jMUPDDQJMus4ft3ku4UXAZe", 2022-09-12T19:16:43.668 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.668 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.669 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.669 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.669 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.669 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.670 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.670 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.670 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008586.777646_67.1", 2022-09-12T19:16:43.670 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.671 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-12", 2022-09-12T19:16:43.671 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:46.777646572Z", 2022-09-12T19:16:43.671 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.671 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.672 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.672 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.672 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.672 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.672 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_dds05uMalANntddWdTKqimb", 2022-09-12T19:16:43.673 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.673 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.673 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.674 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.674 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.674 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.674 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.674 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.675 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008586.780479_68.1", 2022-09-12T19:16:43.675 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.675 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-12", 2022-09-12T19:16:43.676 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:46.780479129Z", 2022-09-12T19:16:43.676 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.676 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.676 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.677 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.677 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.677 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.677 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_dds05uMalANntddWdTKqimb", 2022-09-12T19:16:43.678 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.678 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.678 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.678 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.679 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.679 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.679 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.679 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.680 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008900.617421_102.1", 2022-09-12T19:16:43.680 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.680 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13", 2022-09-12T19:16:43.680 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:55:00.617421183Z", 2022-09-12T19:16:43.681 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.681 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.681 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.681 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.682 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.682 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.682 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_LDyblCFwL38h9zUN-Es9_0k", 2022-09-12T19:16:43.683 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.683 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.683 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.683 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.684 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.684 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.684 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.684 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.685 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008900.621771_103.1", 2022-09-12T19:16:43.685 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.685 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13", 2022-09-12T19:16:43.686 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:55:00.621771922Z", 2022-09-12T19:16:43.686 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.686 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.686 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.687 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.687 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.687 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.687 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_LDyblCFwL38h9zUN-Es9_0k", 2022-09-12T19:16:43.688 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.688 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.688 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.689 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.689 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.689 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.689 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.690 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.690 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008900.623473_104.1", 2022-09-12T19:16:43.690 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.690 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-13", 2022-09-12T19:16:43.691 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:55:00.623473623Z", 2022-09-12T19:16:43.691 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.691 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.691 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.692 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.692 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.692 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.692 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_HUc36aHifT_7iFz2OXdLjDc", 2022-09-12T19:16:43.693 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.693 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.693 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.693 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.694 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.694 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.694 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.694 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.695 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008900.629061_105.1", 2022-09-12T19:16:43.695 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.695 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-13", 2022-09-12T19:16:43.695 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:55:00.629061119Z", 2022-09-12T19:16:43.696 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.696 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.696 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.696 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.697 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.697 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.697 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_HUc36aHifT_7iFz2OXdLjDc", 2022-09-12T19:16:43.697 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.698 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.698 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.698 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.698 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.699 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.699 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.699 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.699 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008900.663664_106.1", 2022-09-12T19:16:43.700 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.700 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14", 2022-09-12T19:16:43.700 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:55:00.663664982Z", 2022-09-12T19:16:43.701 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.701 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.701 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.702 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.702 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.702 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.702 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_ULxp_YwqTEGv8QDweh9UlvR", 2022-09-12T19:16:43.703 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.703 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.703 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.703 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.704 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.704 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.704 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.704 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.705 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008900.667714_107.1", 2022-09-12T19:16:43.705 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.705 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14", 2022-09-12T19:16:43.705 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:55:00.667714167Z", 2022-09-12T19:16:43.706 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.706 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.706 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.706 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.707 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.707 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.707 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_ULxp_YwqTEGv8QDweh9UlvR", 2022-09-12T19:16:43.707 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.708 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.708 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.708 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.708 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.709 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.709 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.709 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.709 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008900.669743_108.1", 2022-09-12T19:16:43.710 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.710 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-14", 2022-09-12T19:16:43.710 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:55:00.669743466Z", 2022-09-12T19:16:43.711 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.711 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.711 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.711 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.712 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.712 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.712 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Zohm3vyJdVty1-BsovuZx7M", 2022-09-12T19:16:43.712 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.712 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.713 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.713 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.713 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.713 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.714 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.714 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.714 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008900.675413_109.1", 2022-09-12T19:16:43.714 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.715 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-14", 2022-09-12T19:16:43.715 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:55:00.675413478Z", 2022-09-12T19:16:43.715 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.715 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.716 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.716 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.716 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.716 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.717 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Zohm3vyJdVty1-BsovuZx7M", 2022-09-12T19:16:43.717 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.717 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.717 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.718 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.718 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.718 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.718 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.719 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.719 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008900.713502_110.1", 2022-09-12T19:16:43.719 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.719 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15", 2022-09-12T19:16:43.720 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:55:00.713502192Z", 2022-09-12T19:16:43.720 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.720 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.720 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.721 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.721 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.721 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.721 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_AKH72TwSsQVrOOdb7zgDFt1", 2022-09-12T19:16:43.722 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.722 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.722 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.722 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.723 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.723 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.723 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.724 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.724 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008900.716530_111.1", 2022-09-12T19:16:43.724 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.724 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15", 2022-09-12T19:16:43.725 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:55:00.716530511Z", 2022-09-12T19:16:43.725 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.725 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.725 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.726 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.726 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.726 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.726 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_AKH72TwSsQVrOOdb7zgDFt1", 2022-09-12T19:16:43.726 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.727 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.727 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.727 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.727 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.728 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.728 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.728 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.729 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008900.718433_112.1", 2022-09-12T19:16:43.729 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.729 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-15", 2022-09-12T19:16:43.729 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:55:00.718433663Z", 2022-09-12T19:16:43.730 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.730 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.730 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.731 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.731 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.731 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.731 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_i61YjmwdeL9p7lC_EIZUJTo", 2022-09-12T19:16:43.732 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.732 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.732 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.732 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.733 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.733 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.733 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.733 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.734 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008900.721341_113.1", 2022-09-12T19:16:43.734 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.734 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-15", 2022-09-12T19:16:43.734 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:55:00.721341013Z", 2022-09-12T19:16:43.735 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.735 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.735 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.735 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.736 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.736 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.736 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_i61YjmwdeL9p7lC_EIZUJTo", 2022-09-12T19:16:43.736 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.737 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.737 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.737 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.737 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.738 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.738 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.738 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.738 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.182484_132.1", 2022-09-12T19:16:43.739 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.739 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16", 2022-09-12T19:16:43.739 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.182484917Z", 2022-09-12T19:16:43.739 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.740 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.740 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.740 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.740 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.741 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.741 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_V99BgSq-hvAYBvkex60C8n2", 2022-09-12T19:16:43.741 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.741 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.742 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.742 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.742 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.742 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.743 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.743 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.743 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.186031_133.1", 2022-09-12T19:16:43.743 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.744 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16", 2022-09-12T19:16:43.744 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.186031455Z", 2022-09-12T19:16:43.744 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.744 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.745 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.745 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.745 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.745 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.746 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_V99BgSq-hvAYBvkex60C8n2", 2022-09-12T19:16:43.746 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.746 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.746 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.747 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.747 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.747 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.747 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.748 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.748 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.187394_134.1", 2022-09-12T19:16:43.748 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.748 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-16", 2022-09-12T19:16:43.748 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.187394124Z", 2022-09-12T19:16:43.749 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.749 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.749 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.749 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.750 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.750 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.750 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_ehigIfW992JSI46zHL04P5J", 2022-09-12T19:16:43.750 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.751 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.751 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.751 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.751 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.752 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.752 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.752 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.752 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.193049_135.1", 2022-09-12T19:16:43.753 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.753 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-16", 2022-09-12T19:16:43.753 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.193049111Z", 2022-09-12T19:16:43.753 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.754 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.754 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.754 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.754 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.755 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.755 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_ehigIfW992JSI46zHL04P5J", 2022-09-12T19:16:43.755 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.756 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.756 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.756 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.756 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.757 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.757 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.757 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.757 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.223026_136.1", 2022-09-12T19:16:43.758 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.758 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17", 2022-09-12T19:16:43.758 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.223026973Z", 2022-09-12T19:16:43.758 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.759 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.759 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.759 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.759 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.760 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.760 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_BCZEMlNNfcs_zua_I4p9TaW", 2022-09-12T19:16:43.760 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.761 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.761 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.761 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.761 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.762 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.762 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.762 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.762 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.226575_137.1", 2022-09-12T19:16:43.763 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.763 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17", 2022-09-12T19:16:43.763 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.226575054Z", 2022-09-12T19:16:43.764 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.764 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.764 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.764 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.765 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.765 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.765 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_BCZEMlNNfcs_zua_I4p9TaW", 2022-09-12T19:16:43.766 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.766 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.766 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.766 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.767 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.767 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.767 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.767 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.768 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.227798_138.1", 2022-09-12T19:16:43.768 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.768 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-17", 2022-09-12T19:16:43.768 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.227798137Z", 2022-09-12T19:16:43.769 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.769 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.769 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.769 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.770 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.770 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.770 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_i8wOFAvzgTHQCJOO8N6Bnam", 2022-09-12T19:16:43.770 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.771 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.771 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.771 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.771 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.772 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.772 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.772 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.772 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.232329_139.1", 2022-09-12T19:16:43.773 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.773 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-17", 2022-09-12T19:16:43.773 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.232329182Z", 2022-09-12T19:16:43.774 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.774 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.774 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.774 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.775 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.775 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.775 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_i8wOFAvzgTHQCJOO8N6Bnam", 2022-09-12T19:16:43.775 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.776 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.776 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.776 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.776 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.777 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.777 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.777 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.777 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.268820_140.1", 2022-09-12T19:16:43.777 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.778 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18", 2022-09-12T19:16:43.778 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.268820788Z", 2022-09-12T19:16:43.778 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.778 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.779 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.779 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.779 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.779 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.780 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_5GsV_zMTqFSGin4m-BDaD32", 2022-09-12T19:16:43.780 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.780 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.780 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.781 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.781 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.781 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.781 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.782 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.782 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.282707_141.1", 2022-09-12T19:16:43.782 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.782 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18", 2022-09-12T19:16:43.783 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.282707229Z", 2022-09-12T19:16:43.783 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.783 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.783 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.784 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.784 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.784 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.784 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_5GsV_zMTqFSGin4m-BDaD32", 2022-09-12T19:16:43.785 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.785 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.785 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.786 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.786 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.786 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.787 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.787 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.787 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.284810_142.1", 2022-09-12T19:16:43.787 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.787 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-18", 2022-09-12T19:16:43.788 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.284810891Z", 2022-09-12T19:16:43.788 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.788 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.788 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.789 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.789 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.789 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.789 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Ow-fYJ8C7sQ2JLuZ3C6E5Oy", 2022-09-12T19:16:43.790 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.790 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.790 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.791 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.791 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.791 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.791 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.791 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.792 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.287795_143.1", 2022-09-12T19:16:43.792 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.792 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-18", 2022-09-12T19:16:43.792 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.287795384Z", 2022-09-12T19:16:43.793 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.794 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.795 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.795 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.796 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.796 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.797 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Ow-fYJ8C7sQ2JLuZ3C6E5Oy", 2022-09-12T19:16:43.797 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.798 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.798 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.799 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.799 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.800 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.800 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.801 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.801 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.308137_144.1", 2022-09-12T19:16:43.801 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.802 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16", 2022-09-12T19:16:43.802 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.308137433Z", 2022-09-12T19:16:43.803 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.803 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.804 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_V99BgSq-hvAYBvkex60C8n2", 2022-09-12T19:16:43.804 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.805 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.805 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.806 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_V99BgSq-hvAYBvkex60C8n2", 2022-09-12T19:16:43.806 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:43.807 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.807 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.807 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.808 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.808 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.809 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.809 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.810 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.310988_145.1", 2022-09-12T19:16:43.810 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.811 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16", 2022-09-12T19:16:43.811 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.310988969Z", 2022-09-12T19:16:43.812 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.812 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.813 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_V99BgSq-hvAYBvkex60C8n2", 2022-09-12T19:16:43.813 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.814 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.814 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.815 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_V99BgSq-hvAYBvkex60C8n2", 2022-09-12T19:16:43.815 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:43.816 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.816 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.817 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.817 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.818 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.818 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.819 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.819 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.322795_146.1", 2022-09-12T19:16:43.820 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.820 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17", 2022-09-12T19:16:43.821 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.322795845Z", 2022-09-12T19:16:43.821 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.822 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.823 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_BCZEMlNNfcs_zua_I4p9TaW", 2022-09-12T19:16:43.823 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.824 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.824 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.825 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_BCZEMlNNfcs_zua_I4p9TaW", 2022-09-12T19:16:43.825 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:43.826 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.826 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.827 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.827 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.828 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.828 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.829 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.829 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.325825_147.1", 2022-09-12T19:16:43.830 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.830 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17", 2022-09-12T19:16:43.831 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.325825003Z", 2022-09-12T19:16:43.831 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.832 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.832 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_BCZEMlNNfcs_zua_I4p9TaW", 2022-09-12T19:16:43.833 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.833 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.833 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.834 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_BCZEMlNNfcs_zua_I4p9TaW", 2022-09-12T19:16:43.834 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:43.834 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.835 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.835 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.835 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.835 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.836 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.836 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.836 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.343478_148.1", 2022-09-12T19:16:43.837 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.837 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18", 2022-09-12T19:16:43.837 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.343478792Z", 2022-09-12T19:16:43.837 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.838 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.838 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_5GsV_zMTqFSGin4m-BDaD32", 2022-09-12T19:16:43.838 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.838 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.839 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.839 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_5GsV_zMTqFSGin4m-BDaD32", 2022-09-12T19:16:43.839 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:43.839 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.840 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.840 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.840 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.841 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.841 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.841 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.841 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.346375_149.1", 2022-09-12T19:16:43.842 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.842 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18", 2022-09-12T19:16:43.842 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.346375662Z", 2022-09-12T19:16:43.842 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.843 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.843 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_5GsV_zMTqFSGin4m-BDaD32", 2022-09-12T19:16:43.843 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.844 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.845 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.845 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_5GsV_zMTqFSGin4m-BDaD32", 2022-09-12T19:16:43.845 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:43.846 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.846 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.846 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.847 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.847 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.847 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.847 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.848 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009433.572798_179.1", 2022-09-12T19:16:43.848 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.848 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19", 2022-09-12T19:16:43.848 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:03:53.572798305Z", 2022-09-12T19:16:43.849 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.849 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.849 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.850 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.850 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.850 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.850 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_r98RnZbo3gbuHHD5iR9vaQX", 2022-09-12T19:16:43.851 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.851 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.851 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.852 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.852 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.852 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.852 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.853 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.853 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009433.576124_180.1", 2022-09-12T19:16:43.853 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.853 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19", 2022-09-12T19:16:43.854 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:03:53.576124288Z", 2022-09-12T19:16:43.854 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.854 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.854 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.855 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.855 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.855 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.855 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_r98RnZbo3gbuHHD5iR9vaQX", 2022-09-12T19:16:43.856 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.856 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.856 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.857 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.857 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.857 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.857 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.858 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.858 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009433.577370_181.1", 2022-09-12T19:16:43.858 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.858 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-19", 2022-09-12T19:16:43.859 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:03:53.577370053Z", 2022-09-12T19:16:43.859 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.859 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.860 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.860 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.860 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.860 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.861 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_FepK5riJM4xGtHwIBSHMDgN", 2022-09-12T19:16:43.861 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.861 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.862 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.862 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.862 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.863 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.863 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.863 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.863 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009433.580348_182.1", 2022-09-12T19:16:43.864 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.864 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-19", 2022-09-12T19:16:43.864 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:03:53.580348934Z", 2022-09-12T19:16:43.864 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.865 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.865 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.865 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.865 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.866 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.866 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_FepK5riJM4xGtHwIBSHMDgN", 2022-09-12T19:16:43.866 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.867 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.867 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.867 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.867 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.868 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.868 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.868 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.868 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009433.593338_183.1", 2022-09-12T19:16:43.869 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.869 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19", 2022-09-12T19:16:43.869 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:03:53.593338729Z", 2022-09-12T19:16:43.869 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.870 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.870 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_r98RnZbo3gbuHHD5iR9vaQX", 2022-09-12T19:16:43.870 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.871 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.871 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.871 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_r98RnZbo3gbuHHD5iR9vaQX", 2022-09-12T19:16:43.871 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:43.872 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.872 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.872 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.872 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.873 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.873 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.874 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.874 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009433.597955_184.1", 2022-09-12T19:16:43.874 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.875 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19", 2022-09-12T19:16:43.875 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:03:53.597955912Z", 2022-09-12T19:16:43.875 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.876 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.876 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_r98RnZbo3gbuHHD5iR9vaQX", 2022-09-12T19:16:43.876 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.876 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.877 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.877 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_r98RnZbo3gbuHHD5iR9vaQX", 2022-09-12T19:16:43.877 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:43.877 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.878 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.878 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.878 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.878 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.879 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.879 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.879 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009630.186894_193.1", 2022-09-12T19:16:43.880 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.880 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:16:43.880 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:07:10.186894533Z", 2022-09-12T19:16:43.880 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.881 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.881 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.881 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.881 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.882 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.882 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:43.882 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.883 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.883 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.883 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.883 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.884 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.884 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.884 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.884 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009630.192780_194.1", 2022-09-12T19:16:43.885 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.885 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:16:43.885 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:07:10.192780968Z", 2022-09-12T19:16:43.885 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.886 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.886 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.886 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.887 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.887 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.887 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:43.887 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.888 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.888 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.888 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.888 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.889 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.889 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.889 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.889 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009630.194692_195.1", 2022-09-12T19:16:43.890 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.890 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-20", 2022-09-12T19:16:43.890 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:07:10.194692069Z", 2022-09-12T19:16:43.891 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.891 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.891 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.891 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.892 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.892 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.892 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_vBgV2Ivb4f-6P44JJPXNdtC", 2022-09-12T19:16:43.892 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.893 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.893 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.893 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.894 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.894 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.894 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.894 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.895 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009630.200316_196.1", 2022-09-12T19:16:43.895 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.895 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-20", 2022-09-12T19:16:43.895 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:07:10.200316464Z", 2022-09-12T19:16:43.896 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.896 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.896 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.896 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.897 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.897 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.897 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_vBgV2Ivb4f-6P44JJPXNdtC", 2022-09-12T19:16:43.898 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.898 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.898 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.898 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.899 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.899 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.899 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.900 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.900 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009668.037998_204.1", 2022-09-12T19:16:43.900 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.900 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:16:43.901 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:07:48.037998619Z", 2022-09-12T19:16:43.901 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.901 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.901 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:43.902 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.902 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.902 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.902 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:43.903 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:43.903 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.903 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.904 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.904 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.904 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.904 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.905 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.905 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009668.041232_205.1", 2022-09-12T19:16:43.905 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.905 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:16:43.906 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:07:48.041232517Z", 2022-09-12T19:16:43.906 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.906 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.906 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:43.907 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.907 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.907 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.907 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:43.908 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:43.908 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.908 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.908 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.909 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.909 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.909 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.910 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.910 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009726.091934_207.1", 2022-09-12T19:16:43.910 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.910 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:16:43.911 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:08:46.091934751Z", 2022-09-12T19:16:43.911 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.911 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.911 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:43.912 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:43.912 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.912 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.912 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:43.913 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 3 2022-09-12T19:16:43.913 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.913 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.913 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.914 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.914 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.914 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.915 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.915 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009726.095686_208.1", 2022-09-12T19:16:43.915 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.915 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:16:43.916 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:08:46.095686282Z", 2022-09-12T19:16:43.916 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.916 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.916 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:43.917 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:43.917 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.917 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.917 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:43.918 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 3 2022-09-12T19:16:43.918 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.918 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.919 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.919 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.919 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.919 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.920 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.920 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009842.925161_211.1", 2022-09-12T19:16:43.920 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.920 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21", 2022-09-12T19:16:43.921 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:42.925161228Z", 2022-09-12T19:16:43.921 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.921 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.921 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.922 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.922 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.922 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.923 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_wA9vY_PYbZdc0eUi2hSxrjB", 2022-09-12T19:16:43.923 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.923 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.923 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.924 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.924 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.924 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.924 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.925 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.925 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009842.930693_212.1", 2022-09-12T19:16:43.925 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.925 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21", 2022-09-12T19:16:43.926 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:42.930693175Z", 2022-09-12T19:16:43.926 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.926 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.927 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.927 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.927 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.927 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.928 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_wA9vY_PYbZdc0eUi2hSxrjB", 2022-09-12T19:16:43.928 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.928 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.928 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.929 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.929 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.929 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.929 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.930 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.930 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009842.932348_213.1", 2022-09-12T19:16:43.930 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.930 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-21", 2022-09-12T19:16:43.931 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:42.932348571Z", 2022-09-12T19:16:43.931 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.931 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.932 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.932 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.932 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.932 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.933 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_YhO7K8SUXb88GVnz95qTMjU", 2022-09-12T19:16:43.933 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.933 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.933 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.935 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.936 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.936 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.937 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.937 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.937 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009842.938156_214.1", 2022-09-12T19:16:43.938 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.938 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-21", 2022-09-12T19:16:43.938 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:42.938156918Z", 2022-09-12T19:16:43.939 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.939 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.939 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.939 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.940 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.940 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.940 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_YhO7K8SUXb88GVnz95qTMjU", 2022-09-12T19:16:43.941 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.941 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.941 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.941 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.942 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.942 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.943 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.943 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.943 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009842.960242_215.1", 2022-09-12T19:16:43.944 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.944 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22", 2022-09-12T19:16:43.945 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:42.960242821Z", 2022-09-12T19:16:43.945 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.946 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.946 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.947 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.947 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.947 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.948 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_jCE0vI8jcGzuttLr9LzBCWJ", 2022-09-12T19:16:43.948 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.948 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.949 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.949 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.949 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.950 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.950 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.951 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.951 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009842.965349_216.1", 2022-09-12T19:16:43.952 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.952 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22", 2022-09-12T19:16:43.952 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:42.965349366Z", 2022-09-12T19:16:43.953 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.953 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.954 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.954 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.954 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.955 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.955 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_jCE0vI8jcGzuttLr9LzBCWJ", 2022-09-12T19:16:43.956 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.956 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.957 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.957 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.957 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.958 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.958 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.959 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.959 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009842.966491_217.1", 2022-09-12T19:16:43.960 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.960 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-22", 2022-09-12T19:16:43.961 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:42.966491565Z", 2022-09-12T19:16:43.961 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.962 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.962 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.963 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.963 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.964 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.964 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_90qhmHoutPPaDynubPFZH9h", 2022-09-12T19:16:43.965 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.965 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.966 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.966 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.967 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.967 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.968 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.968 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.969 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009842.970703_218.1", 2022-09-12T19:16:43.969 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.970 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-22", 2022-09-12T19:16:43.970 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:42.970703166Z", 2022-09-12T19:16:43.970 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.971 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.972 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.972 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.972 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.973 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.973 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_90qhmHoutPPaDynubPFZH9h", 2022-09-12T19:16:43.974 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.974 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.975 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.975 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.976 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.976 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.977 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.977 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.978 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009843.006494_219.1", 2022-09-12T19:16:43.978 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.979 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23", 2022-09-12T19:16:43.979 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:43.006494173Z", 2022-09-12T19:16:43.980 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.980 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.981 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.981 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.981 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.982 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.983 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_8EYxPua6E_71y78ZWvmZh7v", 2022-09-12T19:16:43.983 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.984 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.984 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.985 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:43.985 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.986 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.986 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.987 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.987 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009843.009740_220.1", 2022-09-12T19:16:43.988 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:43.988 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23", 2022-09-12T19:16:43.989 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:43.009740592Z", 2022-09-12T19:16:43.989 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.990 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.990 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.990 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.991 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.991 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:43.992 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_8EYxPua6E_71y78ZWvmZh7v", 2022-09-12T19:16:43.992 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:43.992 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.993 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:43.993 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:43.994 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.994 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:43.995 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:43.995 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:43.996 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009843.011011_221.1", 2022-09-12T19:16:43.996 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:43.997 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-23", 2022-09-12T19:16:43.997 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:43.011011338Z", 2022-09-12T19:16:43.998 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:43.998 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:43.998 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:43.999 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:43.999 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.000 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.001 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_EWfzEo4mEzPqDzTBu2wloku", 2022-09-12T19:16:44.001 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.001 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.002 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.002 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.003 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.003 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.004 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.004 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.005 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009843.015119_222.1", 2022-09-12T19:16:44.005 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.006 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-23", 2022-09-12T19:16:44.006 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:43.015119585Z", 2022-09-12T19:16:44.006 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.007 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.007 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.008 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.008 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.009 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.009 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_EWfzEo4mEzPqDzTBu2wloku", 2022-09-12T19:16:44.010 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.010 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.010 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.011 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.011 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.012 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.012 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.013 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.013 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009843.040988_223.1", 2022-09-12T19:16:44.014 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.014 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21", 2022-09-12T19:16:44.015 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:43.040988054Z", 2022-09-12T19:16:44.015 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.016 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.016 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_wA9vY_PYbZdc0eUi2hSxrjB", 2022-09-12T19:16:44.017 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.017 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.018 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.018 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_wA9vY_PYbZdc0eUi2hSxrjB", 2022-09-12T19:16:44.019 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.019 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.020 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.020 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.020 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.021 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.021 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.021 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.021 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009843.045887_224.1", 2022-09-12T19:16:44.022 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.022 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21", 2022-09-12T19:16:44.022 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:43.045887351Z", 2022-09-12T19:16:44.023 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.023 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.023 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_wA9vY_PYbZdc0eUi2hSxrjB", 2022-09-12T19:16:44.024 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.024 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.024 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.024 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_wA9vY_PYbZdc0eUi2hSxrjB", 2022-09-12T19:16:44.025 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.025 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.025 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.025 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.026 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.026 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.026 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.026 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.027 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009843.057828_225.1", 2022-09-12T19:16:44.027 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.027 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22", 2022-09-12T19:16:44.027 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:43.057828971Z", 2022-09-12T19:16:44.028 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.028 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.028 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_jCE0vI8jcGzuttLr9LzBCWJ", 2022-09-12T19:16:44.028 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.029 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.029 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.029 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_jCE0vI8jcGzuttLr9LzBCWJ", 2022-09-12T19:16:44.029 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.030 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.030 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.030 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.030 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.031 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.031 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.031 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.031 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009843.060815_226.1", 2022-09-12T19:16:44.032 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.032 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22", 2022-09-12T19:16:44.032 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:43.060815675Z", 2022-09-12T19:16:44.032 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.033 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.033 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_jCE0vI8jcGzuttLr9LzBCWJ", 2022-09-12T19:16:44.033 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.033 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.034 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.034 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_jCE0vI8jcGzuttLr9LzBCWJ", 2022-09-12T19:16:44.034 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.035 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.035 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.035 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.036 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.036 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.036 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.036 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.037 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009843.077332_227.1", 2022-09-12T19:16:44.037 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.037 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23", 2022-09-12T19:16:44.037 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:43.077332048Z", 2022-09-12T19:16:44.038 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.038 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.038 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_8EYxPua6E_71y78ZWvmZh7v", 2022-09-12T19:16:44.038 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.039 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.039 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.039 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_8EYxPua6E_71y78ZWvmZh7v", 2022-09-12T19:16:44.039 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.040 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.040 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.040 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.040 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.041 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.041 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.041 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.041 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009843.080040_228.1", 2022-09-12T19:16:44.042 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.042 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23", 2022-09-12T19:16:44.042 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:43.080040677Z", 2022-09-12T19:16:44.042 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.043 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.043 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_8EYxPua6E_71y78ZWvmZh7v", 2022-09-12T19:16:44.043 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.043 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.044 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.044 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_8EYxPua6E_71y78ZWvmZh7v", 2022-09-12T19:16:44.044 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.044 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.045 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.045 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.045 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.045 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.046 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.046 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.046 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.213940_231.1", 2022-09-12T19:16:44.046 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.047 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:16:44.047 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.213940834Z", 2022-09-12T19:16:44.047 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.047 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.048 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.048 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.048 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.048 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.049 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:44.049 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.049 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.049 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.050 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.050 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.050 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.050 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.051 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.051 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.217329_232.1", 2022-09-12T19:16:44.051 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.051 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:16:44.052 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.217329420Z", 2022-09-12T19:16:44.052 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.052 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.052 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.053 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.053 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.053 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.053 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:44.054 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.054 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.054 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.054 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.055 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.055 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.055 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.055 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.056 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.218459_233.1", 2022-09-12T19:16:44.056 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.056 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-24", 2022-09-12T19:16:44.057 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.218459288Z", 2022-09-12T19:16:44.057 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.057 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.058 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.058 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.058 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.058 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.059 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_zt4J3xCzAzkFDmMYSv2ZhpD", 2022-09-12T19:16:44.059 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.059 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.060 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.060 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.060 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.060 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.061 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.061 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.061 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.221613_234.1", 2022-09-12T19:16:44.061 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.062 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-24", 2022-09-12T19:16:44.062 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.221613900Z", 2022-09-12T19:16:44.062 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.062 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.063 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.063 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.063 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.064 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.064 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_zt4J3xCzAzkFDmMYSv2ZhpD", 2022-09-12T19:16:44.064 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.064 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.065 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.065 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.065 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.065 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.066 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.066 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.066 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.242266_235.1", 2022-09-12T19:16:44.066 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.067 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:16:44.067 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.242266250Z", 2022-09-12T19:16:44.067 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.067 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.068 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.068 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.068 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.068 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.069 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:44.069 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.069 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.069 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.070 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.070 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.070 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.070 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.071 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.071 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.245374_236.1", 2022-09-12T19:16:44.071 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.071 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:16:44.072 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.245374907Z", 2022-09-12T19:16:44.072 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.072 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.072 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.073 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.073 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.073 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.073 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:44.074 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.074 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.074 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.074 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.075 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.075 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.075 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.075 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.076 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.246464_237.1", 2022-09-12T19:16:44.076 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.076 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-25", 2022-09-12T19:16:44.076 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.246464476Z", 2022-09-12T19:16:44.077 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.077 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.077 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.077 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.078 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.078 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.078 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_FuWLdq4G8Pitukb9rq9CEoZ", 2022-09-12T19:16:44.078 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.079 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.079 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.079 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.079 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.080 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.080 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.080 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.080 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.249409_238.1", 2022-09-12T19:16:44.081 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.081 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-25", 2022-09-12T19:16:44.081 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.249409215Z", 2022-09-12T19:16:44.081 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.082 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.082 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.082 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.082 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.083 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.083 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_FuWLdq4G8Pitukb9rq9CEoZ", 2022-09-12T19:16:44.083 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.083 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.084 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.084 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.084 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.085 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.085 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.085 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.085 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.282380_239.1", 2022-09-12T19:16:44.086 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.086 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:16:44.086 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.282380399Z", 2022-09-12T19:16:44.086 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.087 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.087 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.087 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.087 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.088 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.088 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:44.088 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.088 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.089 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.089 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.089 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.089 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.090 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.090 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.090 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.285471_240.1", 2022-09-12T19:16:44.090 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.091 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:16:44.091 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.285471857Z", 2022-09-12T19:16:44.091 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.091 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.092 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.092 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.092 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.092 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.093 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:44.093 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.093 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.093 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.094 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.094 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.094 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.094 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.095 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.095 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.286588_241.1", 2022-09-12T19:16:44.095 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.095 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-26", 2022-09-12T19:16:44.096 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.286588982Z", 2022-09-12T19:16:44.096 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.096 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.096 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.097 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.097 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.097 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.097 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_-j_vUqBTsANTAcA80szuZUK", 2022-09-12T19:16:44.098 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.098 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.098 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.098 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.099 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.099 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.099 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.099 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.099 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.289386_242.1", 2022-09-12T19:16:44.100 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.100 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-26", 2022-09-12T19:16:44.100 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.289386844Z", 2022-09-12T19:16:44.100 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.101 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.101 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.101 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.101 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.102 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.102 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_-j_vUqBTsANTAcA80szuZUK", 2022-09-12T19:16:44.102 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.103 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.103 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.103 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.103 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.104 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.104 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.104 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.104 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.309215_243.1", 2022-09-12T19:16:44.105 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.105 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:16:44.105 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.309215519Z", 2022-09-12T19:16:44.105 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.106 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.106 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:44.106 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.106 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.107 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.107 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:44.107 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.108 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.108 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.108 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.108 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.109 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.109 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.109 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.109 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.312069_244.1", 2022-09-12T19:16:44.110 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.110 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:16:44.110 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.312069106Z", 2022-09-12T19:16:44.110 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.111 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.111 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:44.111 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.111 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.112 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.112 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:44.112 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.112 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.113 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.113 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.113 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.113 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.114 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.114 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.114 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.323214_245.1", 2022-09-12T19:16:44.114 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.115 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:16:44.115 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.323214538Z", 2022-09-12T19:16:44.115 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.115 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.116 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:44.116 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.116 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.116 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.116 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:44.117 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 3 2022-09-12T19:16:44.117 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.117 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.117 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.118 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.118 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.118 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.118 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.119 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.334825_246.1", 2022-09-12T19:16:44.119 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.119 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:16:44.119 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.334825968Z", 2022-09-12T19:16:44.120 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.120 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.120 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:44.121 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.121 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.121 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.121 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:44.122 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 3 2022-09-12T19:16:44.122 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.122 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.122 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.123 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.123 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.123 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.123 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.124 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.350812_247.1", 2022-09-12T19:16:44.124 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.124 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:16:44.124 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.350812474Z", 2022-09-12T19:16:44.125 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.125 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.125 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:44.125 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.126 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.126 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.126 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:44.126 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.127 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.127 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.127 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.127 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.128 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.128 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.128 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.128 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.355013_248.1", 2022-09-12T19:16:44.129 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.129 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:16:44.129 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.355013585Z", 2022-09-12T19:16:44.129 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.130 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.130 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:44.130 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.130 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.131 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.131 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:44.131 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.131 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.132 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.132 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.132 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.132 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.133 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.133 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.133 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.369892_249.1", 2022-09-12T19:16:44.133 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.134 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:16:44.134 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.369892532Z", 2022-09-12T19:16:44.134 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.135 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.135 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:44.135 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.135 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.136 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.136 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:44.136 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 3 2022-09-12T19:16:44.136 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.136 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.137 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.137 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.137 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.137 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.138 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.138 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.374217_250.1", 2022-09-12T19:16:44.138 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.138 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:16:44.139 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.374217196Z", 2022-09-12T19:16:44.139 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.139 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.139 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:44.140 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.140 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.140 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.140 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:44.141 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 3 2022-09-12T19:16:44.141 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.141 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.141 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.141 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.142 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.142 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.142 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.143 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.394432_251.1", 2022-09-12T19:16:44.143 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.143 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:16:44.143 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.394432026Z", 2022-09-12T19:16:44.143 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.144 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.144 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:44.144 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.144 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.145 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.145 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:44.145 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.145 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.146 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.146 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.146 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.146 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.147 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.147 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.147 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.398632_252.1", 2022-09-12T19:16:44.147 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.148 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:16:44.148 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.398632136Z", 2022-09-12T19:16:44.148 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.148 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.148 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:44.149 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.149 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.149 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.149 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:44.150 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.150 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.150 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.150 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.151 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.151 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.151 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.151 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.152 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.418847_253.1", 2022-09-12T19:16:44.152 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.152 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:16:44.152 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.418847506Z", 2022-09-12T19:16:44.153 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.153 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.153 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:44.153 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.154 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.154 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.154 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:44.154 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 3 2022-09-12T19:16:44.155 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.155 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.155 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.155 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.156 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.156 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.156 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.156 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.421418_254.1", 2022-09-12T19:16:44.157 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.157 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:16:44.157 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.421418488Z", 2022-09-12T19:16:44.157 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.157 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.158 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:44.158 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.158 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.158 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.159 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:44.159 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 3 2022-09-12T19:16:44.159 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.159 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.160 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.160 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.160 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.160 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.161 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.161 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.022864_258.1", 2022-09-12T19:16:44.161 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.161 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-27:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.27", 2022-09-12T19:16:44.162 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.022864695Z", 2022-09-12T19:16:44.162 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.162 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.163 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.163 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.163 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.163 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.164 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_4Ia0kBYSOXAPun0srTpHOgg", 2022-09-12T19:16:44.164 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.164 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.164 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.165 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.165 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.165 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.165 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.166 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.166 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.027319_259.1", 2022-09-12T19:16:44.166 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.166 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-27:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.27", 2022-09-12T19:16:44.167 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.027319061Z", 2022-09-12T19:16:44.167 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.167 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.167 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.168 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.168 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.168 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.168 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_4Ia0kBYSOXAPun0srTpHOgg", 2022-09-12T19:16:44.169 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.169 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.169 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.169 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.170 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.170 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.170 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.171 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.171 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.029529_260.1", 2022-09-12T19:16:44.171 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.171 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-27", 2022-09-12T19:16:44.171 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.029529115Z", 2022-09-12T19:16:44.172 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.172 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.172 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.172 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.173 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.173 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.173 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_hL6ubPjUnxDbFclj75F-9Yw", 2022-09-12T19:16:44.173 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.174 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.174 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.174 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.174 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.175 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.175 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.175 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.175 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.035074_261.1", 2022-09-12T19:16:44.176 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.176 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-27", 2022-09-12T19:16:44.176 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.035074135Z", 2022-09-12T19:16:44.176 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.177 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.177 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.177 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.177 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.178 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.178 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_hL6ubPjUnxDbFclj75F-9Yw", 2022-09-12T19:16:44.178 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.178 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.179 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.179 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.179 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.179 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.180 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.180 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.180 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.060437_262.1", 2022-09-12T19:16:44.180 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.181 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-28:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.28", 2022-09-12T19:16:44.181 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.060437057Z", 2022-09-12T19:16:44.181 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.181 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.182 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.182 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.182 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.182 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.183 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_F91L-y4ApRIftBlrzdVSKhB", 2022-09-12T19:16:44.183 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.183 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.183 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.184 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.184 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.184 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.184 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.185 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.185 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.067920_263.1", 2022-09-12T19:16:44.185 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.185 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-28:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.28", 2022-09-12T19:16:44.186 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.067920385Z", 2022-09-12T19:16:44.186 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.186 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.186 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.186 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.187 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.187 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.187 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_F91L-y4ApRIftBlrzdVSKhB", 2022-09-12T19:16:44.187 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.188 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.188 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.188 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.188 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.189 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.189 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.189 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.189 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.068850_264.1", 2022-09-12T19:16:44.190 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.190 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-28", 2022-09-12T19:16:44.190 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.068850124Z", 2022-09-12T19:16:44.190 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.191 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.191 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.191 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.191 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.191 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.192 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_4foA5qWzZ6DKNqXOUrqtpMi", 2022-09-12T19:16:44.192 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.192 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.192 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.193 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.193 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.193 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.193 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.194 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.194 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.074343_265.1", 2022-09-12T19:16:44.194 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.194 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-28", 2022-09-12T19:16:44.195 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.074343801Z", 2022-09-12T19:16:44.195 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.195 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.195 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.196 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.196 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.196 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.196 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_4foA5qWzZ6DKNqXOUrqtpMi", 2022-09-12T19:16:44.197 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.197 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.197 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.197 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.198 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.198 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.198 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.198 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.199 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.114860_266.1", 2022-09-12T19:16:44.199 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.199 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29", 2022-09-12T19:16:44.199 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.114860178Z", 2022-09-12T19:16:44.200 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.200 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.200 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.200 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.200 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.201 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.201 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_kHZQcZIbg231-V_zVbHphPU", 2022-09-12T19:16:44.201 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.201 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.202 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.202 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.202 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.202 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.203 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.203 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.203 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.119876_267.1", 2022-09-12T19:16:44.203 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.204 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29", 2022-09-12T19:16:44.204 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.119876089Z", 2022-09-12T19:16:44.204 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.204 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.205 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.205 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.205 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.205 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.206 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_kHZQcZIbg231-V_zVbHphPU", 2022-09-12T19:16:44.206 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.206 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.206 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.207 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.207 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.207 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.207 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.208 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.208 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.121725_268.1", 2022-09-12T19:16:44.208 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.208 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-29", 2022-09-12T19:16:44.209 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.121725517Z", 2022-09-12T19:16:44.209 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.209 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.209 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.210 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.210 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.210 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.210 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_2ti8fe4vkZVz-LOJO9SiYze", 2022-09-12T19:16:44.210 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.211 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.211 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.211 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.211 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.212 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.212 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.212 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.212 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.125711_269.1", 2022-09-12T19:16:44.213 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.213 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-29", 2022-09-12T19:16:44.213 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.125711914Z", 2022-09-12T19:16:44.213 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.214 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.214 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.214 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.214 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.215 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.215 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_2ti8fe4vkZVz-LOJO9SiYze", 2022-09-12T19:16:44.215 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.215 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.216 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.216 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.216 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.216 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.216 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.217 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.217 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.153919_270.1", 2022-09-12T19:16:44.217 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.217 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-27:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.27", 2022-09-12T19:16:44.218 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.153919920Z", 2022-09-12T19:16:44.218 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.218 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.218 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_4Ia0kBYSOXAPun0srTpHOgg", 2022-09-12T19:16:44.219 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.219 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.219 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.219 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_4Ia0kBYSOXAPun0srTpHOgg", 2022-09-12T19:16:44.220 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.220 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.220 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.220 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.220 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.221 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.221 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.221 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.221 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.156407_271.1", 2022-09-12T19:16:44.222 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.222 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-27:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.27", 2022-09-12T19:16:44.222 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.156407416Z", 2022-09-12T19:16:44.222 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.223 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.223 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_4Ia0kBYSOXAPun0srTpHOgg", 2022-09-12T19:16:44.223 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.223 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.224 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.224 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_4Ia0kBYSOXAPun0srTpHOgg", 2022-09-12T19:16:44.224 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.224 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.225 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.225 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.225 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.225 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.226 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.226 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.226 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.183174_272.1", 2022-09-12T19:16:44.226 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.227 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-28:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.28", 2022-09-12T19:16:44.227 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.183174956Z", 2022-09-12T19:16:44.227 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.227 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.228 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_F91L-y4ApRIftBlrzdVSKhB", 2022-09-12T19:16:44.228 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.228 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.228 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.229 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_F91L-y4ApRIftBlrzdVSKhB", 2022-09-12T19:16:44.229 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.229 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.229 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.229 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.230 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.230 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.230 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.230 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.231 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.186673_273.1", 2022-09-12T19:16:44.231 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.231 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-28:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.28", 2022-09-12T19:16:44.231 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.186673632Z", 2022-09-12T19:16:44.232 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.232 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.232 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_F91L-y4ApRIftBlrzdVSKhB", 2022-09-12T19:16:44.232 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.233 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.233 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.233 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_F91L-y4ApRIftBlrzdVSKhB", 2022-09-12T19:16:44.233 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.234 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.234 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.234 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.234 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.235 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.235 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.235 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.236 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.258924_274.1", 2022-09-12T19:16:44.236 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.236 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29", 2022-09-12T19:16:44.236 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.258924078Z", 2022-09-12T19:16:44.237 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.237 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.237 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_kHZQcZIbg231-V_zVbHphPU", 2022-09-12T19:16:44.237 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.238 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.238 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.238 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_kHZQcZIbg231-V_zVbHphPU", 2022-09-12T19:16:44.238 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.239 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.239 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.239 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.239 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.240 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.240 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.240 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.240 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.261587_275.1", 2022-09-12T19:16:44.241 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.241 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29", 2022-09-12T19:16:44.241 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.261587678Z", 2022-09-12T19:16:44.241 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.242 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.242 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_kHZQcZIbg231-V_zVbHphPU", 2022-09-12T19:16:44.242 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.242 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.243 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.243 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_kHZQcZIbg231-V_zVbHphPU", 2022-09-12T19:16:44.243 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.244 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.244 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.244 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.245 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.245 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.245 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.245 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.246 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.295442_276.1", 2022-09-12T19:16:44.246 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.246 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-30:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.30", 2022-09-12T19:16:44.246 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.295442828Z", 2022-09-12T19:16:44.247 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.247 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.247 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.247 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.248 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.248 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.248 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_xXGncRxFB5weAPxs1wnNxZq", 2022-09-12T19:16:44.248 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.249 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.249 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.249 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.249 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.250 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.250 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.250 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.250 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.297949_277.1", 2022-09-12T19:16:44.251 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.251 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-30:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.30", 2022-09-12T19:16:44.251 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.297949202Z", 2022-09-12T19:16:44.251 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.252 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.252 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.252 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.252 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.252 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.253 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_xXGncRxFB5weAPxs1wnNxZq", 2022-09-12T19:16:44.253 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.253 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.253 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.254 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.254 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.254 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.254 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.255 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.255 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.298820_278.1", 2022-09-12T19:16:44.255 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.255 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-30", 2022-09-12T19:16:44.256 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.298820021Z", 2022-09-12T19:16:44.256 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.256 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.256 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.257 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.257 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.257 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.257 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_o1lHEa9WwJ0A1-_nw8PZQAy", 2022-09-12T19:16:44.258 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.258 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.258 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.258 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.259 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.259 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.259 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.259 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.260 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.301106_279.1", 2022-09-12T19:16:44.260 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.260 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-30", 2022-09-12T19:16:44.260 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.301106622Z", 2022-09-12T19:16:44.260 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.261 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.261 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.261 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.261 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.262 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.262 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_o1lHEa9WwJ0A1-_nw8PZQAy", 2022-09-12T19:16:44.262 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.262 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.263 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.263 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.263 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.263 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.264 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.264 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.264 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.318892_280.1", 2022-09-12T19:16:44.264 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.265 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-31:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.31", 2022-09-12T19:16:44.265 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.318892577Z", 2022-09-12T19:16:44.265 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.265 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.266 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.266 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.266 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.266 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.267 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_aXhKq1Q-6is3aiib6Zv69Wn", 2022-09-12T19:16:44.267 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.267 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.267 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.268 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.268 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.268 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.268 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.269 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.269 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.321371_281.1", 2022-09-12T19:16:44.269 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.269 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-31:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.31", 2022-09-12T19:16:44.270 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.321371696Z", 2022-09-12T19:16:44.270 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.270 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.270 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.271 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.271 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.271 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.271 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_aXhKq1Q-6is3aiib6Zv69Wn", 2022-09-12T19:16:44.272 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.272 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.272 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.272 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.272 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.273 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.273 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.273 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.273 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.323664_282.1", 2022-09-12T19:16:44.274 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.274 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-31", 2022-09-12T19:16:44.274 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.323664287Z", 2022-09-12T19:16:44.274 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.275 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.275 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.275 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.275 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.276 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.276 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_ZXsXmNqWQzafZqU8AgaaFgE", 2022-09-12T19:16:44.276 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.276 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.277 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.277 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.277 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.277 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.278 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.278 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.278 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.332269_283.1", 2022-09-12T19:16:44.278 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.279 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-31", 2022-09-12T19:16:44.279 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.332269695Z", 2022-09-12T19:16:44.279 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.279 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.280 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.280 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.280 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.280 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.280 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_ZXsXmNqWQzafZqU8AgaaFgE", 2022-09-12T19:16:44.281 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.281 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.281 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.281 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.282 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.282 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.282 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.282 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.283 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.379861_284.1", 2022-09-12T19:16:44.283 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.283 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-32:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.32", 2022-09-12T19:16:44.283 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.379861593Z", 2022-09-12T19:16:44.284 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.284 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.284 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.284 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.285 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.285 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.285 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UeviL3qodKD5NX6Gmu8wcLZ", 2022-09-12T19:16:44.285 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.286 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.286 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.286 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.286 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.287 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.287 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.287 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.287 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.385256_285.1", 2022-09-12T19:16:44.288 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.288 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-32:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.32", 2022-09-12T19:16:44.288 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.385256336Z", 2022-09-12T19:16:44.288 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.289 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.289 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.289 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.289 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.290 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.290 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UeviL3qodKD5NX6Gmu8wcLZ", 2022-09-12T19:16:44.290 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.290 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.291 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.291 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.291 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.291 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.292 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.292 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.292 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.388164_286.1", 2022-09-12T19:16:44.292 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.293 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-32", 2022-09-12T19:16:44.293 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.388164607Z", 2022-09-12T19:16:44.293 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.293 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.293 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.294 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.294 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.294 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.294 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_sSvBc9sHjXg70XiY66Bq6ME", 2022-09-12T19:16:44.295 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.295 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.295 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.295 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.296 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.296 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.296 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.296 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.297 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.392572_287.1", 2022-09-12T19:16:44.297 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.297 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-32", 2022-09-12T19:16:44.297 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.392572009Z", 2022-09-12T19:16:44.298 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.298 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.298 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.298 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.299 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.299 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.299 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_sSvBc9sHjXg70XiY66Bq6ME", 2022-09-12T19:16:44.299 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.300 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.300 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.300 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.300 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.301 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.301 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.301 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.301 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.426848_288.1", 2022-09-12T19:16:44.302 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.302 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-30:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.30", 2022-09-12T19:16:44.302 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.426848512Z", 2022-09-12T19:16:44.302 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.303 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.303 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_xXGncRxFB5weAPxs1wnNxZq", 2022-09-12T19:16:44.303 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.303 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.304 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.304 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_xXGncRxFB5weAPxs1wnNxZq", 2022-09-12T19:16:44.304 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.304 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.305 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.305 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.305 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.306 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.306 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.306 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.306 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.431873_289.1", 2022-09-12T19:16:44.307 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.307 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-30:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.30", 2022-09-12T19:16:44.307 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.431873816Z", 2022-09-12T19:16:44.307 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.308 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.308 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_xXGncRxFB5weAPxs1wnNxZq", 2022-09-12T19:16:44.308 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.308 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.309 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.309 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_xXGncRxFB5weAPxs1wnNxZq", 2022-09-12T19:16:44.309 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.309 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.309 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.310 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.310 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.310 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.310 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.311 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.311 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.505940_290.1", 2022-09-12T19:16:44.311 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.311 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-31:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.31", 2022-09-12T19:16:44.312 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.505940792Z", 2022-09-12T19:16:44.312 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.312 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.312 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_aXhKq1Q-6is3aiib6Zv69Wn", 2022-09-12T19:16:44.313 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.313 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.313 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.313 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_aXhKq1Q-6is3aiib6Zv69Wn", 2022-09-12T19:16:44.314 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.314 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.314 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.314 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.315 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.315 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.315 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.315 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.316 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.509791_291.1", 2022-09-12T19:16:44.316 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.316 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-31:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.31", 2022-09-12T19:16:44.316 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.509791403Z", 2022-09-12T19:16:44.316 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.317 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.317 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_aXhKq1Q-6is3aiib6Zv69Wn", 2022-09-12T19:16:44.317 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.317 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.318 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.318 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_aXhKq1Q-6is3aiib6Zv69Wn", 2022-09-12T19:16:44.318 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.318 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.319 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.319 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.319 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.319 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.320 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.320 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.320 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.545979_292.1", 2022-09-12T19:16:44.320 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.321 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-32:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.32", 2022-09-12T19:16:44.321 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.545979472Z", 2022-09-12T19:16:44.321 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.322 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.322 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UeviL3qodKD5NX6Gmu8wcLZ", 2022-09-12T19:16:44.322 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.322 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.323 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.323 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UeviL3qodKD5NX6Gmu8wcLZ", 2022-09-12T19:16:44.323 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.323 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.324 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.324 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.324 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.324 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.325 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.325 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.325 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.555815_293.1", 2022-09-12T19:16:44.325 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.326 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-32:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.32", 2022-09-12T19:16:44.326 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.555815933Z", 2022-09-12T19:16:44.326 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.326 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.327 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UeviL3qodKD5NX6Gmu8wcLZ", 2022-09-12T19:16:44.327 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.327 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.327 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.328 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UeviL3qodKD5NX6Gmu8wcLZ", 2022-09-12T19:16:44.328 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.328 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.329 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.329 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.329 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.329 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.330 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.330 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.330 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.629617_294.1", 2022-09-12T19:16:44.330 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.331 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-33:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.33", 2022-09-12T19:16:44.331 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.629617979Z", 2022-09-12T19:16:44.331 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.332 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.332 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.332 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.332 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.333 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.333 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_9Dhxqv90pxoXFdD9Kil8AZS", 2022-09-12T19:16:44.333 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.333 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.334 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.334 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.334 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.334 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.335 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.335 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.335 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.632160_295.1", 2022-09-12T19:16:44.335 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.335 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-33:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.33", 2022-09-12T19:16:44.336 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.632160672Z", 2022-09-12T19:16:44.336 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.336 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.337 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.337 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.337 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.338 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.338 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_9Dhxqv90pxoXFdD9Kil8AZS", 2022-09-12T19:16:44.338 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.338 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.339 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.339 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.339 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.340 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.340 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.340 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.340 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.633025_296.1", 2022-09-12T19:16:44.341 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.341 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-33", 2022-09-12T19:16:44.341 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.633025749Z", 2022-09-12T19:16:44.341 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.342 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.342 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.342 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.342 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.343 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.343 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_duz73YpnwikZnC64B4KGaWm", 2022-09-12T19:16:44.343 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.343 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.344 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.344 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.344 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.344 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.345 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.345 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.345 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.635657_297.1", 2022-09-12T19:16:44.345 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.346 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-33", 2022-09-12T19:16:44.346 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.635657460Z", 2022-09-12T19:16:44.346 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.346 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.347 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.347 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.347 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.348 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.348 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_duz73YpnwikZnC64B4KGaWm", 2022-09-12T19:16:44.348 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.349 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.349 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.349 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.349 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.349 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.350 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.350 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.350 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.653881_298.1", 2022-09-12T19:16:44.350 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.351 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-34:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.34", 2022-09-12T19:16:44.351 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.653881272Z", 2022-09-12T19:16:44.351 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.351 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.352 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.352 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.352 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.352 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.353 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_T6BXjPTsg5YjV989eew_nUj", 2022-09-12T19:16:44.353 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.354 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.354 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.354 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.355 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.355 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.355 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.355 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.356 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.656302_299.1", 2022-09-12T19:16:44.356 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.356 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-34:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.34", 2022-09-12T19:16:44.357 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.656302045Z", 2022-09-12T19:16:44.357 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.357 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.357 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.358 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.358 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.358 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.359 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_T6BXjPTsg5YjV989eew_nUj", 2022-09-12T19:16:44.359 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.359 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.359 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.360 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.360 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.360 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.360 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.361 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.361 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.657143_300.1", 2022-09-12T19:16:44.361 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.361 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-34", 2022-09-12T19:16:44.362 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.657143981Z", 2022-09-12T19:16:44.362 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.362 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.363 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.363 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.363 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.363 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.364 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_zJMQJW7qDi486fSD79XXMMX", 2022-09-12T19:16:44.364 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.364 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.364 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.365 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.365 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.365 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.366 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.366 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.366 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.659579_301.1", 2022-09-12T19:16:44.366 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.367 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-34", 2022-09-12T19:16:44.367 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.659579967Z", 2022-09-12T19:16:44.367 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.367 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.368 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.368 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.368 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.368 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.369 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_zJMQJW7qDi486fSD79XXMMX", 2022-09-12T19:16:44.369 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.369 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.369 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.370 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.370 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.370 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.370 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.371 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.371 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.688010_302.1", 2022-09-12T19:16:44.371 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.371 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35", 2022-09-12T19:16:44.372 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.688010386Z", 2022-09-12T19:16:44.372 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.372 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.372 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.373 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.373 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.373 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.373 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_RT_qLkUYTVFuTZ0Mzyw3ra0", 2022-09-12T19:16:44.374 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.374 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.374 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.374 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.375 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.375 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.375 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.375 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.376 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.690504_303.1", 2022-09-12T19:16:44.376 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.376 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35", 2022-09-12T19:16:44.376 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.690504973Z", 2022-09-12T19:16:44.376 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.377 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.377 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.377 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.377 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.378 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.378 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_RT_qLkUYTVFuTZ0Mzyw3ra0", 2022-09-12T19:16:44.378 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.378 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.379 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.379 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.379 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.379 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.380 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.380 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.380 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.691394_304.1", 2022-09-12T19:16:44.380 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.381 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-35", 2022-09-12T19:16:44.381 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.691394342Z", 2022-09-12T19:16:44.381 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.381 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.382 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.382 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.382 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.382 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.383 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_E0ZY7pht4AH9rgoAN9wcr0i", 2022-09-12T19:16:44.383 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.383 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.383 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.384 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.384 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.384 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.385 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.385 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.385 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.693991_305.1", 2022-09-12T19:16:44.385 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.386 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-35", 2022-09-12T19:16:44.386 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.693991405Z", 2022-09-12T19:16:44.386 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.386 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.387 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.387 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.387 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.387 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.388 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_E0ZY7pht4AH9rgoAN9wcr0i", 2022-09-12T19:16:44.388 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.388 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.388 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.389 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.389 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.389 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.389 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.389 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.390 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.713639_306.1", 2022-09-12T19:16:44.390 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.390 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-33:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.33", 2022-09-12T19:16:44.391 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.713639843Z", 2022-09-12T19:16:44.391 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.391 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.391 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_9Dhxqv90pxoXFdD9Kil8AZS", 2022-09-12T19:16:44.392 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.392 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.392 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.392 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_9Dhxqv90pxoXFdD9Kil8AZS", 2022-09-12T19:16:44.393 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.393 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.393 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.393 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.394 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.394 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.394 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.395 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.395 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.716209_307.1", 2022-09-12T19:16:44.395 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.395 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-33:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.33", 2022-09-12T19:16:44.396 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.716209907Z", 2022-09-12T19:16:44.396 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.396 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.396 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_9Dhxqv90pxoXFdD9Kil8AZS", 2022-09-12T19:16:44.396 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.397 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.397 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.397 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_9Dhxqv90pxoXFdD9Kil8AZS", 2022-09-12T19:16:44.397 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.398 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.398 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.398 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.398 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.399 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.399 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.399 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.399 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010025.737428_308.1", 2022-09-12T19:16:44.400 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.400 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-34:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.34", 2022-09-12T19:16:44.400 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:45.737428711Z", 2022-09-12T19:16:44.400 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.401 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.401 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_T6BXjPTsg5YjV989eew_nUj", 2022-09-12T19:16:44.401 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.401 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.402 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.402 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_T6BXjPTsg5YjV989eew_nUj", 2022-09-12T19:16:44.402 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.402 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.403 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.403 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.403 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.403 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.404 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.404 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.404 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010025.741603_309.1", 2022-09-12T19:16:44.404 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.404 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-34:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.34", 2022-09-12T19:16:44.405 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:45.741603314Z", 2022-09-12T19:16:44.405 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.405 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.405 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_T6BXjPTsg5YjV989eew_nUj", 2022-09-12T19:16:44.406 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.406 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.406 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.406 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_T6BXjPTsg5YjV989eew_nUj", 2022-09-12T19:16:44.407 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.407 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.407 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.407 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.408 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.408 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.408 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.408 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.409 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.813360_310.1", 2022-09-12T19:16:44.409 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.409 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35", 2022-09-12T19:16:44.409 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.813360734Z", 2022-09-12T19:16:44.410 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.410 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.410 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_RT_qLkUYTVFuTZ0Mzyw3ra0", 2022-09-12T19:16:44.410 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.411 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.411 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.411 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_RT_qLkUYTVFuTZ0Mzyw3ra0", 2022-09-12T19:16:44.411 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.412 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.412 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.412 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.412 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.413 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.413 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.413 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.413 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.817978_311.1", 2022-09-12T19:16:44.414 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.414 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35", 2022-09-12T19:16:44.414 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.817978446Z", 2022-09-12T19:16:44.414 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.415 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.415 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_RT_qLkUYTVFuTZ0Mzyw3ra0", 2022-09-12T19:16:44.415 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.415 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.416 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.416 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_RT_qLkUYTVFuTZ0Mzyw3ra0", 2022-09-12T19:16:44.416 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.416 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.416 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.417 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.417 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.417 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.418 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.418 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.418 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.851414_312.1", 2022-09-12T19:16:44.418 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.419 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36", 2022-09-12T19:16:44.419 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.851414155Z", 2022-09-12T19:16:44.419 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.419 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.420 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.420 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.420 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.420 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.421 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_QWEAbq70_sTavFlIROKha_u", 2022-09-12T19:16:44.421 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.421 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.421 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.422 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.422 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.422 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.422 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.423 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.423 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.860552_313.1", 2022-09-12T19:16:44.423 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.423 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36", 2022-09-12T19:16:44.424 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.860552661Z", 2022-09-12T19:16:44.424 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.424 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.424 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.424 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.425 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.425 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.425 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_QWEAbq70_sTavFlIROKha_u", 2022-09-12T19:16:44.425 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.426 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.426 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.426 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.427 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.427 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.427 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.428 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.428 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.867783_314.1", 2022-09-12T19:16:44.428 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.429 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-36", 2022-09-12T19:16:44.429 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.867783999Z", 2022-09-12T19:16:44.429 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.429 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.430 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.430 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.430 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.430 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.431 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_-cyyq5stbLWU_c8ISOn28nE", 2022-09-12T19:16:44.431 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.431 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.431 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.432 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.432 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.432 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.432 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.433 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.433 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.879969_315.1", 2022-09-12T19:16:44.433 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.433 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-36", 2022-09-12T19:16:44.434 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.879969477Z", 2022-09-12T19:16:44.434 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.434 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.435 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.435 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.435 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.435 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.436 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_-cyyq5stbLWU_c8ISOn28nE", 2022-09-12T19:16:44.436 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.436 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.437 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.437 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.437 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.437 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.437 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.438 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.438 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.914600_316.1", 2022-09-12T19:16:44.438 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.439 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37", 2022-09-12T19:16:44.439 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.914600273Z", 2022-09-12T19:16:44.439 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.439 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.439 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.440 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.440 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.440 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.440 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_BzeIqN5hzm9Zg_oOIhlkjQJ", 2022-09-12T19:16:44.441 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.441 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.441 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.441 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.442 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.442 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.442 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.443 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.443 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.917584_317.1", 2022-09-12T19:16:44.443 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.444 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37", 2022-09-12T19:16:44.444 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.917584610Z", 2022-09-12T19:16:44.444 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.444 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.445 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.445 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.445 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.445 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.446 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_BzeIqN5hzm9Zg_oOIhlkjQJ", 2022-09-12T19:16:44.446 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.446 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.446 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.447 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.447 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.447 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.447 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.448 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.448 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.918659_318.1", 2022-09-12T19:16:44.448 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.448 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-37", 2022-09-12T19:16:44.448 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.918659379Z", 2022-09-12T19:16:44.449 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.449 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.449 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.449 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.450 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.450 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.450 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_WGVNJNvbu5mulQzf6ElKa4a", 2022-09-12T19:16:44.450 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.451 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.451 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.451 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.452 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.452 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.452 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.453 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.453 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.921237_319.1", 2022-09-12T19:16:44.453 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.453 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-37", 2022-09-12T19:16:44.454 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.921237396Z", 2022-09-12T19:16:44.454 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.454 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.454 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.455 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.455 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.455 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.455 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_WGVNJNvbu5mulQzf6ElKa4a", 2022-09-12T19:16:44.455 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.456 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.456 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.456 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.456 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.457 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.457 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.457 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.457 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.946958_320.1", 2022-09-12T19:16:44.458 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.458 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38", 2022-09-12T19:16:44.458 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.946958410Z", 2022-09-12T19:16:44.459 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.459 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.459 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.459 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.459 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.460 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.460 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_IWIU9MxoQv-Gr3d7VAe8c_0", 2022-09-12T19:16:44.461 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.461 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.461 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.461 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.462 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.462 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.462 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.462 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.463 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.949538_321.1", 2022-09-12T19:16:44.463 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.463 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38", 2022-09-12T19:16:44.463 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.949538394Z", 2022-09-12T19:16:44.463 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.464 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.464 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.464 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.464 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.465 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.465 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_IWIU9MxoQv-Gr3d7VAe8c_0", 2022-09-12T19:16:44.465 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.465 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.466 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.466 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.466 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.466 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.467 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.467 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.467 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.950529_322.1", 2022-09-12T19:16:44.467 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.468 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-38", 2022-09-12T19:16:44.468 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.950529882Z", 2022-09-12T19:16:44.468 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.469 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.469 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.469 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.469 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.470 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.470 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_QAVKfkG00yr1qxv971lPJ9X", 2022-09-12T19:16:44.470 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.471 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.471 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.471 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.471 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.472 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.472 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.472 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.472 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.953073_323.1", 2022-09-12T19:16:44.473 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.473 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-38", 2022-09-12T19:16:44.473 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.953073366Z", 2022-09-12T19:16:44.473 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.474 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.474 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.474 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.474 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.475 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.475 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_QAVKfkG00yr1qxv971lPJ9X", 2022-09-12T19:16:44.475 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.475 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.475 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.476 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.476 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.476 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.476 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.477 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.477 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010058.708837_324.1", 2022-09-12T19:16:44.477 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.477 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-39:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.39", 2022-09-12T19:16:44.478 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:14:18.708837670Z", 2022-09-12T19:16:44.478 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.478 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.478 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.479 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.479 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.479 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.479 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_8tqVsOF3WKb2me5I95G3qgR", 2022-09-12T19:16:44.480 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.480 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.480 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.480 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.481 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.481 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.481 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.481 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.482 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010058.713291_325.1", 2022-09-12T19:16:44.482 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.482 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-39:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.39", 2022-09-12T19:16:44.482 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:14:18.713291435Z", 2022-09-12T19:16:44.483 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.483 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.483 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.483 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.484 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.484 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.484 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_8tqVsOF3WKb2me5I95G3qgR", 2022-09-12T19:16:44.484 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.484 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.485 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.485 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.485 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.485 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.486 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.486 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.486 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010058.715054_326.1", 2022-09-12T19:16:44.486 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.487 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-39", 2022-09-12T19:16:44.487 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:14:18.715054633Z", 2022-09-12T19:16:44.487 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.487 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.488 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.488 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.488 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.488 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.489 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_xgZt1r-qtjp-K4E_Z1U_XYi", 2022-09-12T19:16:44.489 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.489 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.489 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.490 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.490 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.490 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.490 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.490 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.491 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010058.720459_327.1", 2022-09-12T19:16:44.491 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.491 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-39", 2022-09-12T19:16:44.491 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:14:18.720459880Z", 2022-09-12T19:16:44.492 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.492 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.492 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.492 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.493 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.493 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.493 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_xgZt1r-qtjp-K4E_Z1U_XYi", 2022-09-12T19:16:44.493 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.494 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.494 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.494 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.494 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.495 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.495 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.495 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.495 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010058.744939_328.1", 2022-09-12T19:16:44.496 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.496 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-40:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.40", 2022-09-12T19:16:44.496 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:14:18.744939676Z", 2022-09-12T19:16:44.496 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.497 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.497 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.497 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.497 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.498 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.498 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_XRz8ZhFtWNauZ5YqelhcHjG", 2022-09-12T19:16:44.498 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.498 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.499 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.499 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.499 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.499 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.499 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.500 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.500 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010058.753933_329.1", 2022-09-12T19:16:44.500 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.500 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-40:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.40", 2022-09-12T19:16:44.501 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:14:18.753933965Z", 2022-09-12T19:16:44.501 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.501 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.501 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.502 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.502 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.502 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.503 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_XRz8ZhFtWNauZ5YqelhcHjG", 2022-09-12T19:16:44.503 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.503 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.503 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.504 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.504 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.504 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.504 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.504 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.505 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010058.758572_330.1", 2022-09-12T19:16:44.505 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.505 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-40", 2022-09-12T19:16:44.505 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:14:18.758572726Z", 2022-09-12T19:16:44.506 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.506 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.506 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.507 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.507 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.507 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.507 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_4IJ7vbCVAqQL9xJtZhDAZ4r", 2022-09-12T19:16:44.508 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.508 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.508 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.508 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.508 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.509 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.509 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.509 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.509 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010058.761625_331.1", 2022-09-12T19:16:44.510 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.510 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-40", 2022-09-12T19:16:44.510 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:14:18.761625471Z", 2022-09-12T19:16:44.510 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.511 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.511 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.511 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.511 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.512 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.512 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_4IJ7vbCVAqQL9xJtZhDAZ4r", 2022-09-12T19:16:44.512 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.513 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.513 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.513 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.513 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.514 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.514 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.514 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.514 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010058.795251_332.1", 2022-09-12T19:16:44.515 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.515 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-41:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.41", 2022-09-12T19:16:44.515 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:14:18.795251326Z", 2022-09-12T19:16:44.515 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.515 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.516 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.516 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.516 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.516 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.517 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_WLs8_gJqx8-eJNrUvhbOESF", 2022-09-12T19:16:44.517 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.517 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.517 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.518 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.518 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.518 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.518 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.519 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.519 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010058.799161_333.1", 2022-09-12T19:16:44.519 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.520 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-41:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.41", 2022-09-12T19:16:44.520 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:14:18.799161992Z", 2022-09-12T19:16:44.520 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.520 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.521 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.521 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.521 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.521 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.522 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_WLs8_gJqx8-eJNrUvhbOESF", 2022-09-12T19:16:44.522 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.522 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.522 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.523 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.523 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.523 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.523 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.524 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.524 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010058.800547_334.1", 2022-09-12T19:16:44.524 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.524 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-41", 2022-09-12T19:16:44.525 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:14:18.800547915Z", 2022-09-12T19:16:44.525 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.525 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.525 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.526 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.526 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.526 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.526 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Rx20gRuO3XO_dd_rDNjdCZL", 2022-09-12T19:16:44.527 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.527 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.527 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.527 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.528 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.528 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.528 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.529 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.529 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010058.804299_335.1", 2022-09-12T19:16:44.529 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.529 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-41", 2022-09-12T19:16:44.530 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:14:18.804299896Z", 2022-09-12T19:16:44.530 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.530 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.530 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.531 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.531 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.531 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.533 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Rx20gRuO3XO_dd_rDNjdCZL", 2022-09-12T19:16:44.533 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.534 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.534 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.534 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.534 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.535 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.535 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.535 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.535 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008569.937424_38.1", 2022-09-12T19:16:44.536 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.536 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-1:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.1", 2022-09-12T19:16:44.536 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:29.937424694Z", 2022-09-12T19:16:44.536 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.537 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.537 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.537 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.538 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.538 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.538 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "__lpWFIiN4hfllNCufyv6oPF", 2022-09-12T19:16:44.538 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.539 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.539 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.539 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.539 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.540 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.540 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.540 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.540 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008569.948669_40.1", 2022-09-12T19:16:44.541 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.541 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-1:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.1", 2022-09-12T19:16:44.541 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:29.948669068Z", 2022-09-12T19:16:44.541 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.542 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.542 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.542 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.543 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.543 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.543 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "__lpWFIiN4hfllNCufyv6oPF", 2022-09-12T19:16:44.543 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.544 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.544 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.544 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.544 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.545 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.545 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.545 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.545 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008569.949794_41.1", 2022-09-12T19:16:44.545 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.546 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-1", 2022-09-12T19:16:44.546 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:29.949794341Z", 2022-09-12T19:16:44.547 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.547 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.547 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.547 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.548 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.548 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.548 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_724Zi_oQbv20Wa_KFzslGam", 2022-09-12T19:16:44.548 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.549 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.549 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.549 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.549 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.550 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.550 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.550 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.550 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008569.952619_42.1", 2022-09-12T19:16:44.551 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.551 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-1", 2022-09-12T19:16:44.551 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:29.952619357Z", 2022-09-12T19:16:44.551 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.552 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.552 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.552 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.552 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.553 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.553 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_724Zi_oQbv20Wa_KFzslGam", 2022-09-12T19:16:44.553 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.553 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.554 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.554 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.554 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.554 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.555 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.555 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.555 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008569.979576_43.1", 2022-09-12T19:16:44.556 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.556 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-2:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.2", 2022-09-12T19:16:44.556 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:29.979576716Z", 2022-09-12T19:16:44.556 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.557 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.557 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.557 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.557 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.558 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.558 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_yWZcOeCLp3AOZWE4KEf6DqY", 2022-09-12T19:16:44.558 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.558 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.559 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.559 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.559 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.559 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.560 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.560 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.560 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008569.982366_44.1", 2022-09-12T19:16:44.560 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.561 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-2:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.2", 2022-09-12T19:16:44.561 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:29.982366892Z", 2022-09-12T19:16:44.561 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.561 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.562 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.562 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.562 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.562 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.563 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_yWZcOeCLp3AOZWE4KEf6DqY", 2022-09-12T19:16:44.563 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.563 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.564 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.564 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.564 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.564 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.565 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.565 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.565 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008569.983351_45.1", 2022-09-12T19:16:44.565 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.566 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-2", 2022-09-12T19:16:44.566 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:29.983351998Z", 2022-09-12T19:16:44.566 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.566 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.567 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.567 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.567 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.567 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.568 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_JYImBEwDt-Z7UabuOPjifZb", 2022-09-12T19:16:44.568 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.568 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.568 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.569 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.569 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.569 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.569 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.570 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.570 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008569.985969_46.1", 2022-09-12T19:16:44.570 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.570 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-2", 2022-09-12T19:16:44.571 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:29.985969116Z", 2022-09-12T19:16:44.571 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.571 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.572 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.572 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.572 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.572 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.572 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_JYImBEwDt-Z7UabuOPjifZb", 2022-09-12T19:16:44.573 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.573 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.573 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.573 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.574 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.574 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.574 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.574 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.575 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008572.534688_47.1", 2022-09-12T19:16:44.575 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.575 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-3:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.3", 2022-09-12T19:16:44.576 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:32.534688656Z", 2022-09-12T19:16:44.576 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.576 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.577 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.577 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.577 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.577 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.578 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_2N-a-PDWYhR0LHw5JLpZR4c", 2022-09-12T19:16:44.578 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.578 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.578 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.579 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.579 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.579 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.579 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.580 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.580 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008572.537960_48.1", 2022-09-12T19:16:44.580 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.580 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-3:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.3", 2022-09-12T19:16:44.580 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:32.537960225Z", 2022-09-12T19:16:44.581 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.581 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.581 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.581 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.582 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.582 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.582 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_2N-a-PDWYhR0LHw5JLpZR4c", 2022-09-12T19:16:44.583 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.583 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.583 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.583 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.583 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.584 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.584 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.584 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.585 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008572.539055_49.1", 2022-09-12T19:16:44.585 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.585 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-3", 2022-09-12T19:16:44.585 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:32.539055121Z", 2022-09-12T19:16:44.586 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.586 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.586 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.586 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.587 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.587 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.587 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_84xzHNW5sbis6XdPCoJHPOk", 2022-09-12T19:16:44.587 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.588 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.588 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.588 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.588 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.589 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.589 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.589 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.589 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008572.542081_50.1", 2022-09-12T19:16:44.590 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.590 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-3", 2022-09-12T19:16:44.590 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:32.542081201Z", 2022-09-12T19:16:44.590 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.591 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.591 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.591 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.591 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.592 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.592 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_84xzHNW5sbis6XdPCoJHPOk", 2022-09-12T19:16:44.592 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.592 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.593 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.593 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.593 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.594 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.594 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.594 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.594 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008578.327919_51.1", 2022-09-12T19:16:44.595 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.595 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-4:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.4", 2022-09-12T19:16:44.595 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.327919700Z", 2022-09-12T19:16:44.595 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.596 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.596 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.596 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.596 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.597 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.597 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_k7Q_LlQwzVHeoxMKmCvxjHo", 2022-09-12T19:16:44.597 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.597 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.598 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.598 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.598 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.598 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.599 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.599 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.599 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008578.330590_52.1", 2022-09-12T19:16:44.599 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.600 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-4:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.4", 2022-09-12T19:16:44.600 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.330590808Z", 2022-09-12T19:16:44.600 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.600 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.601 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.601 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.601 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.602 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.602 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_k7Q_LlQwzVHeoxMKmCvxjHo", 2022-09-12T19:16:44.603 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.603 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.604 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.604 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.605 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.605 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.606 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.606 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.607 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008578.331599_53.1", 2022-09-12T19:16:44.607 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.607 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-4", 2022-09-12T19:16:44.608 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.331599255Z", 2022-09-12T19:16:44.608 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.609 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.609 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.609 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.610 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.610 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.610 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Uwy2BFzmIVF9oQbfNuX8-c2", 2022-09-12T19:16:44.611 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.611 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.612 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.612 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.612 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.613 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.613 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.614 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.614 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008578.334356_54.1", 2022-09-12T19:16:44.614 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.615 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-4", 2022-09-12T19:16:44.615 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.334356674Z", 2022-09-12T19:16:44.615 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.616 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.616 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.616 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.616 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.617 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.617 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Uwy2BFzmIVF9oQbfNuX8-c2", 2022-09-12T19:16:44.618 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.618 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.619 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.619 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.620 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.620 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.621 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.621 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.622 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008578.354410_55.1", 2022-09-12T19:16:44.622 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.622 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-5:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.5", 2022-09-12T19:16:44.623 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.354410340Z", 2022-09-12T19:16:44.623 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.623 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.623 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.624 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.624 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.624 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.624 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_DOCacF6mLBz24cosgaizYjM", 2022-09-12T19:16:44.625 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.625 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.625 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.626 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.626 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.626 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.627 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.627 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.627 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008578.357067_56.1", 2022-09-12T19:16:44.628 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.628 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-5:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.5", 2022-09-12T19:16:44.628 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.357067384Z", 2022-09-12T19:16:44.628 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.629 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.629 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.629 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.629 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.630 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.630 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_DOCacF6mLBz24cosgaizYjM", 2022-09-12T19:16:44.630 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.630 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.631 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.631 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.631 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.631 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.632 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.632 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.632 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008578.357978_57.1", 2022-09-12T19:16:44.632 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.633 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-5", 2022-09-12T19:16:44.633 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.357978793Z", 2022-09-12T19:16:44.633 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.633 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.634 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.634 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.634 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.635 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.635 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_PK2zvS8BU3_kRxhnsNsDeRa", 2022-09-12T19:16:44.636 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.636 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.637 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.637 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.637 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.638 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.638 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.639 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.639 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008578.360592_58.1", 2022-09-12T19:16:44.640 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.640 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-5", 2022-09-12T19:16:44.641 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.360592786Z", 2022-09-12T19:16:44.641 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.641 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.642 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.642 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.643 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.643 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.643 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_PK2zvS8BU3_kRxhnsNsDeRa", 2022-09-12T19:16:44.644 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.644 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.645 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.645 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.646 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.646 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.646 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.647 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.647 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008578.393761_59.1", 2022-09-12T19:16:44.648 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.648 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-6:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.6", 2022-09-12T19:16:44.649 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.393761186Z", 2022-09-12T19:16:44.649 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.649 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.650 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.650 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.650 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.650 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.651 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_zr7R-yTI4PYqVN0hq56_Jgr", 2022-09-12T19:16:44.651 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.652 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.652 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.652 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.652 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.653 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.653 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.653 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.653 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008578.396555_60.1", 2022-09-12T19:16:44.654 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.654 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-6:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.6", 2022-09-12T19:16:44.655 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.396555557Z", 2022-09-12T19:16:44.655 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.655 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.656 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.656 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.656 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.656 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.657 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_zr7R-yTI4PYqVN0hq56_Jgr", 2022-09-12T19:16:44.657 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.657 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.657 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.658 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.658 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.658 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.658 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.659 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.659 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008578.397479_61.1", 2022-09-12T19:16:44.659 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.660 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-6", 2022-09-12T19:16:44.660 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.397479637Z", 2022-09-12T19:16:44.660 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.661 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.661 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.662 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.662 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.662 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.663 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_9moMpukFqONyJQ22546lFFJ", 2022-09-12T19:16:44.663 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.664 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.664 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.665 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.665 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.666 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.666 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.667 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.667 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008578.400100_62.1", 2022-09-12T19:16:44.668 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.668 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-6", 2022-09-12T19:16:44.669 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.400100117Z", 2022-09-12T19:16:44.669 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.669 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.670 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.670 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.671 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.671 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.671 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_9moMpukFqONyJQ22546lFFJ", 2022-09-12T19:16:44.672 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.672 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.673 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.673 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.673 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.674 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.674 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.674 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.675 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.041323_63.1", 2022-09-12T19:16:44.675 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.675 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-7:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.7", 2022-09-12T19:16:44.675 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.041323973Z", 2022-09-12T19:16:44.676 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.676 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.676 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.676 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.677 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.677 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.677 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_14zIMAUeR0_uC-IGvsiZyFE", 2022-09-12T19:16:44.677 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.678 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.678 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.678 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.678 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.679 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.679 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.679 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.679 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.043790_64.1", 2022-09-12T19:16:44.680 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.680 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-7:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.7", 2022-09-12T19:16:44.680 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.043790091Z", 2022-09-12T19:16:44.681 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.681 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.682 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.682 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.683 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.683 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.683 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_14zIMAUeR0_uC-IGvsiZyFE", 2022-09-12T19:16:44.684 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.684 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.684 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.685 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.685 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.686 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.686 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.687 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.687 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.044741_65.1", 2022-09-12T19:16:44.688 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.688 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-7", 2022-09-12T19:16:44.689 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.044741403Z", 2022-09-12T19:16:44.689 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.690 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.690 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.691 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.691 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.691 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.692 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_40hjwX5ISF-woQ9EMT_esot", 2022-09-12T19:16:44.692 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.693 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.693 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.694 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.694 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.695 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.695 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.696 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.696 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.047046_66.1", 2022-09-12T19:16:44.697 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.697 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-7", 2022-09-12T19:16:44.698 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.047046794Z", 2022-09-12T19:16:44.698 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.699 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.699 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.700 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.700 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.701 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.701 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_40hjwX5ISF-woQ9EMT_esot", 2022-09-12T19:16:44.702 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.702 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.703 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.703 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.704 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.704 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.705 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.705 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.706 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.068972_67.1", 2022-09-12T19:16:44.706 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.707 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-8:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.8", 2022-09-12T19:16:44.707 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.068972049Z", 2022-09-12T19:16:44.708 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.708 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.709 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.709 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.710 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.710 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.711 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Cbx4jHMPdXce-GRW3F5bG0e", 2022-09-12T19:16:44.711 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.712 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.713 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.713 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.714 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.714 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.715 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.715 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.716 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.071464_68.1", 2022-09-12T19:16:44.716 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.717 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-8:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.8", 2022-09-12T19:16:44.717 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.071464065Z", 2022-09-12T19:16:44.718 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.719 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.719 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.720 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.720 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.721 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.721 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Cbx4jHMPdXce-GRW3F5bG0e", 2022-09-12T19:16:44.722 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.722 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.723 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.723 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.724 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.724 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.725 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.725 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.725 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.072340_69.1", 2022-09-12T19:16:44.726 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.727 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-8", 2022-09-12T19:16:44.727 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.072340420Z", 2022-09-12T19:16:44.728 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.729 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.731 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.731 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.732 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.732 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.733 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_mshfSar0kbJ0_j03QlyLl5l", 2022-09-12T19:16:44.733 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.734 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.735 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.735 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.735 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.736 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.737 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.737 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.738 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.074696_70.1", 2022-09-12T19:16:44.739 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.739 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-8", 2022-09-12T19:16:44.740 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.074696150Z", 2022-09-12T19:16:44.740 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.741 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.741 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.742 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.742 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.743 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.743 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_mshfSar0kbJ0_j03QlyLl5l", 2022-09-12T19:16:44.744 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.744 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.745 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.745 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.746 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.746 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.747 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.747 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.748 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.106134_71.1", 2022-09-12T19:16:44.748 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.748 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9", 2022-09-12T19:16:44.748 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.106134737Z", 2022-09-12T19:16:44.749 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.749 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.749 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.749 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.750 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.750 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.750 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_F8AkuhSt1lRvPdknmmo8vhx", 2022-09-12T19:16:44.750 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.751 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.751 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.751 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.751 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.752 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.752 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.752 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.752 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.110078_72.1", 2022-09-12T19:16:44.753 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.753 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9", 2022-09-12T19:16:44.753 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.110078653Z", 2022-09-12T19:16:44.753 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.754 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.754 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.754 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.755 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.755 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.755 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_F8AkuhSt1lRvPdknmmo8vhx", 2022-09-12T19:16:44.755 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.756 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.756 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.756 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.756 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.757 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.757 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.757 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.757 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.111034_73.1", 2022-09-12T19:16:44.758 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.758 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-9", 2022-09-12T19:16:44.758 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.111034634Z", 2022-09-12T19:16:44.758 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.759 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.759 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.759 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.759 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.760 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.760 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_dnyYyWYZArJtIne5MIt7vUD", 2022-09-12T19:16:44.760 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.760 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.761 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.761 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.761 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.761 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.762 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.762 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.762 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.116857_74.1", 2022-09-12T19:16:44.762 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.763 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-9", 2022-09-12T19:16:44.763 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.116857200Z", 2022-09-12T19:16:44.763 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.763 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.764 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:44.764 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:44.764 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.764 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.765 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_dnyYyWYZArJtIne5MIt7vUD", 2022-09-12T19:16:44.765 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.765 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.765 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.766 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.766 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.766 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.766 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.767 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.767 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.768561_75.1", 2022-09-12T19:16:44.767 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.767 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-7:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.7", 2022-09-12T19:16:44.768 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.768561107Z", 2022-09-12T19:16:44.768 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.768 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.768 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_14zIMAUeR0_uC-IGvsiZyFE", 2022-09-12T19:16:44.769 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.769 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.769 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.769 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_14zIMAUeR0_uC-IGvsiZyFE", 2022-09-12T19:16:44.770 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.770 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.770 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.770 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.771 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.771 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.771 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.771 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.772 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.771274_76.1", 2022-09-12T19:16:44.772 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.772 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-7:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.7", 2022-09-12T19:16:44.772 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.771274788Z", 2022-09-12T19:16:44.773 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.773 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.773 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_14zIMAUeR0_uC-IGvsiZyFE", 2022-09-12T19:16:44.773 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.774 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.774 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.774 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_14zIMAUeR0_uC-IGvsiZyFE", 2022-09-12T19:16:44.775 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.775 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.775 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.775 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.776 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.776 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.776 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.776 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.777 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.783884_77.1", 2022-09-12T19:16:44.777 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.777 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-7", 2022-09-12T19:16:44.777 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.783884039Z", 2022-09-12T19:16:44.778 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.778 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.778 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_40hjwX5ISF-woQ9EMT_esot", 2022-09-12T19:16:44.778 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.779 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.779 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.779 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_40hjwX5ISF-woQ9EMT_esot", 2022-09-12T19:16:44.779 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.780 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.780 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.780 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "remove" 2022-09-12T19:16:44.780 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.781 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.781 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.781 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.783 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.786242_78.1", 2022-09-12T19:16:44.783 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.783 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-7", 2022-09-12T19:16:44.783 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.786242763Z", 2022-09-12T19:16:44.784 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.784 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.784 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_40hjwX5ISF-woQ9EMT_esot", 2022-09-12T19:16:44.785 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.785 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.785 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.785 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_40hjwX5ISF-woQ9EMT_esot", 2022-09-12T19:16:44.786 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.786 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.786 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.787 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.787 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.787 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.787 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.788 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.788 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.817097_79.1", 2022-09-12T19:16:44.788 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.788 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-8:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.8", 2022-09-12T19:16:44.789 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.817097161Z", 2022-09-12T19:16:44.789 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.789 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.789 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Cbx4jHMPdXce-GRW3F5bG0e", 2022-09-12T19:16:44.790 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.790 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.791 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.791 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Cbx4jHMPdXce-GRW3F5bG0e", 2022-09-12T19:16:44.792 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.792 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.793 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.793 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.793 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.794 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.794 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.795 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.795 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.821207_80.1", 2022-09-12T19:16:44.796 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.796 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-8:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.8", 2022-09-12T19:16:44.797 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.821207606Z", 2022-09-12T19:16:44.797 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.798 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.798 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Cbx4jHMPdXce-GRW3F5bG0e", 2022-09-12T19:16:44.798 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.799 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.799 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.799 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Cbx4jHMPdXce-GRW3F5bG0e", 2022-09-12T19:16:44.799 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.800 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.800 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.801 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.801 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.801 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.802 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.802 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.803 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.833382_81.1", 2022-09-12T19:16:44.803 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.804 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-8", 2022-09-12T19:16:44.804 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.833382871Z", 2022-09-12T19:16:44.805 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.805 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.806 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_mshfSar0kbJ0_j03QlyLl5l", 2022-09-12T19:16:44.806 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.806 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.807 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.807 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_mshfSar0kbJ0_j03QlyLl5l", 2022-09-12T19:16:44.807 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.808 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.808 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.809 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "remove" 2022-09-12T19:16:44.809 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.809 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.810 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.810 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.811 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.835519_82.1", 2022-09-12T19:16:44.811 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.812 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-8", 2022-09-12T19:16:44.812 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.835519055Z", 2022-09-12T19:16:44.812 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.813 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.813 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_mshfSar0kbJ0_j03QlyLl5l", 2022-09-12T19:16:44.813 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.814 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.814 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.814 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_mshfSar0kbJ0_j03QlyLl5l", 2022-09-12T19:16:44.815 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.815 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.815 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.815 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.816 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.816 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.816 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.817 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.817 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.890492_83.1", 2022-09-12T19:16:44.817 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.818 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9", 2022-09-12T19:16:44.818 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.890492647Z", 2022-09-12T19:16:44.818 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.819 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.819 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_F8AkuhSt1lRvPdknmmo8vhx", 2022-09-12T19:16:44.820 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.820 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.821 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.821 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_F8AkuhSt1lRvPdknmmo8vhx", 2022-09-12T19:16:44.821 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.822 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.822 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.823 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:44.823 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.823 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.824 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.824 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.825 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.893091_84.1", 2022-09-12T19:16:44.825 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.826 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9", 2022-09-12T19:16:44.826 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.893091078Z", 2022-09-12T19:16:44.827 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.828 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.828 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_F8AkuhSt1lRvPdknmmo8vhx", 2022-09-12T19:16:44.829 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.829 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.830 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.830 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_F8AkuhSt1lRvPdknmmo8vhx", 2022-09-12T19:16:44.831 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.831 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.832 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.832 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.832 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.832 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.833 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.833 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.834 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.901637_85.1", 2022-09-12T19:16:44.834 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.834 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-9", 2022-09-12T19:16:44.835 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.901637143Z", 2022-09-12T19:16:44.835 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.836 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.836 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_dnyYyWYZArJtIne5MIt7vUD", 2022-09-12T19:16:44.837 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.838 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.838 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.839 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_dnyYyWYZArJtIne5MIt7vUD", 2022-09-12T19:16:44.839 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.839 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.840 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.840 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "remove" 2022-09-12T19:16:44.841 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.841 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.841 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.842 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:44.842 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.904048_86.1", 2022-09-12T19:16:44.843 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:44.843 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-9", 2022-09-12T19:16:44.844 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048979Z", 2022-09-12T19:16:44.844 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:44.844 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:44.845 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_dnyYyWYZArJtIne5MIt7vUD", 2022-09-12T19:16:44.845 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:44.846 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.846 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:44.846 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_dnyYyWYZArJtIne5MIt7vUD", 2022-09-12T19:16:44.847 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:44.847 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:44.847 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:44.847 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:44.848 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.848 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.848 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:44.848 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:16:44.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'c0f7a512-c940-4d1f-8adf-3484630f876b', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:44.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'c0f7a512-c940-4d1f-8adf-3484630f876b', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:44.854 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period c0f7a512-c940-4d1f-8adf-3484630f876b --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:44.911 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:44.912 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:44.971 INFO:teuthology.orchestra.run.smithi138.stdout:[ 2022-09-12T19:16:44.972 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:44.973 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008447.969948_52.1", 2022-09-12T19:16:44.973 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "user", 2022-09-12T19:16:44.974 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "rgw-multisite-test-user", 2022-09-12T19:16:44.975 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.969948859Z", 2022-09-12T19:16:44.975 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:44.976 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:44.976 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:44.976 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:44.977 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:44.978 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:44.978 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_X5YlnCJyC_iIiZ1qmhasQLd", 2022-09-12T19:16:44.979 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:44.979 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:44.980 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:44.980 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:44.981 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:44.981 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:44.982 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:44.982 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:44.983 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008447.976965_53.1", 2022-09-12T19:16:44.983 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "user", 2022-09-12T19:16:44.983 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "rgw-multisite-test-user", 2022-09-12T19:16:44.984 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965355Z", 2022-09-12T19:16:44.984 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:44.985 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:44.985 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:44.985 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:44.986 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:44.987 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:44.987 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_X5YlnCJyC_iIiZ1qmhasQLd", 2022-09-12T19:16:44.988 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:44.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:44.989 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:44.989 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:44.990 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:44.990 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:44.990 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:44.991 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:44.991 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008586.711103_57.1", 2022-09-12T19:16:44.991 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.991 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10", 2022-09-12T19:16:44.992 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.711103783Z", 2022-09-12T19:16:44.992 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:44.992 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:44.992 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:44.993 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:44.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:44.993 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:44.994 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_0WIw9i-PdtM0sq6xV05tENT", 2022-09-12T19:16:44.994 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:44.994 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:44.994 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:44.995 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:44.995 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:44.995 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:44.996 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:44.996 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:44.997 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008586.714487_58.1", 2022-09-12T19:16:44.997 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:44.997 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10", 2022-09-12T19:16:44.997 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.714487285Z", 2022-09-12T19:16:44.998 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:44.998 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:44.999 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:44.999 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:44.999 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:44.999 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.000 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_0WIw9i-PdtM0sq6xV05tENT", 2022-09-12T19:16:45.000 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.001 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.002 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.003 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.003 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.004 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.004 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.005 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008586.715813_59.1", 2022-09-12T19:16:45.005 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.006 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-10", 2022-09-12T19:16:45.006 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.715813657Z", 2022-09-12T19:16:45.007 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.007 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.008 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.008 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.009 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.010 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_nPF7oiiG3E2RTyZZOeDLhOv", 2022-09-12T19:16:45.011 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.011 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.012 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.012 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.012 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.014 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.014 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.015 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.015 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008586.718771_60.1", 2022-09-12T19:16:45.016 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.016 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-10", 2022-09-12T19:16:45.017 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.718771977Z", 2022-09-12T19:16:45.017 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.018 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.018 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.019 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.019 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.020 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.021 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_nPF7oiiG3E2RTyZZOeDLhOv", 2022-09-12T19:16:45.021 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.022 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.022 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.023 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.023 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.024 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.025 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.025 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008586.736648_61.1", 2022-09-12T19:16:45.026 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.026 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11", 2022-09-12T19:16:45.027 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.736648541Z", 2022-09-12T19:16:45.027 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.028 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.028 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.029 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.030 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.030 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.031 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_DXlcSXme1LlpozfB5A2QmS2", 2022-09-12T19:16:45.031 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.032 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.033 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.033 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.034 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.035 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.035 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008586.739749_62.1", 2022-09-12T19:16:45.036 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.037 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11", 2022-09-12T19:16:45.037 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.739749085Z", 2022-09-12T19:16:45.038 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.038 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.039 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.039 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.040 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.041 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_DXlcSXme1LlpozfB5A2QmS2", 2022-09-12T19:16:45.041 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.042 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.042 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.042 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.043 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.044 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008586.740926_63.1", 2022-09-12T19:16:45.045 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.045 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-11", 2022-09-12T19:16:45.045 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.740926183Z", 2022-09-12T19:16:45.046 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.046 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.047 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.047 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.048 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.048 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_QTYYWzCEPLTd7eDR1Si_D1k", 2022-09-12T19:16:45.049 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.050 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.050 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.051 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.051 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.052 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008586.744033_64.1", 2022-09-12T19:16:45.053 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.053 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-11", 2022-09-12T19:16:45.054 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.744033830Z", 2022-09-12T19:16:45.054 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.055 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.055 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.055 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.056 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.057 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_QTYYWzCEPLTd7eDR1Si_D1k", 2022-09-12T19:16:45.057 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.058 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.058 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.059 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.059 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.060 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.061 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008586.773460_65.1", 2022-09-12T19:16:45.061 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.062 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12", 2022-09-12T19:16:45.062 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.773460810Z", 2022-09-12T19:16:45.062 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.063 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.064 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.064 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.065 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.065 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_jMUPDDQJMus4ft3ku4UXAZe", 2022-09-12T19:16:45.065 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.066 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.066 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.066 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.067 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.067 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.067 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008586.776584_66.1", 2022-09-12T19:16:45.068 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.068 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12", 2022-09-12T19:16:45.068 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.776584217Z", 2022-09-12T19:16:45.068 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.069 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.069 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.069 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.070 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.070 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_jMUPDDQJMus4ft3ku4UXAZe", 2022-09-12T19:16:45.070 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.071 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.071 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.071 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.072 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.072 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008586.777646_67.1", 2022-09-12T19:16:45.073 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.073 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-12", 2022-09-12T19:16:45.073 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.777646572Z", 2022-09-12T19:16:45.073 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.074 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.074 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.074 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.075 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.075 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_dds05uMalANntddWdTKqimb", 2022-09-12T19:16:45.075 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.076 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.076 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.076 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.077 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.077 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008586.780479_68.1", 2022-09-12T19:16:45.078 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.078 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-12", 2022-09-12T19:16:45.078 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.780479129Z", 2022-09-12T19:16:45.078 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.079 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.079 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.079 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.080 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.080 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_dds05uMalANntddWdTKqimb", 2022-09-12T19:16:45.080 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.081 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.081 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.081 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.082 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.082 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008900.617421_102.1", 2022-09-12T19:16:45.083 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.083 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13", 2022-09-12T19:16:45.083 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.617421183Z", 2022-09-12T19:16:45.083 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.084 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.084 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.084 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.085 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.085 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_LDyblCFwL38h9zUN-Es9_0k", 2022-09-12T19:16:45.085 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.086 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.086 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.086 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.086 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.087 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008900.621771_103.1", 2022-09-12T19:16:45.087 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.088 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13", 2022-09-12T19:16:45.088 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.621771922Z", 2022-09-12T19:16:45.088 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.088 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.089 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.089 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.089 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.090 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_LDyblCFwL38h9zUN-Es9_0k", 2022-09-12T19:16:45.090 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.090 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.091 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.091 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.091 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.092 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008900.623473_104.1", 2022-09-12T19:16:45.092 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.092 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-13", 2022-09-12T19:16:45.093 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.623473623Z", 2022-09-12T19:16:45.093 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.093 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.094 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.094 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.094 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.095 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_HUc36aHifT_7iFz2OXdLjDc", 2022-09-12T19:16:45.095 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.095 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.096 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.096 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.096 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.097 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008900.629061_105.1", 2022-09-12T19:16:45.097 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.097 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-13", 2022-09-12T19:16:45.098 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.629061119Z", 2022-09-12T19:16:45.098 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.098 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.098 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.099 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.099 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.100 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_HUc36aHifT_7iFz2OXdLjDc", 2022-09-12T19:16:45.100 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.100 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.101 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.101 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.101 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.102 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.102 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008900.663664_106.1", 2022-09-12T19:16:45.102 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.103 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14", 2022-09-12T19:16:45.103 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.663664982Z", 2022-09-12T19:16:45.103 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.104 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.104 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.104 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.105 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.105 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_ULxp_YwqTEGv8QDweh9UlvR", 2022-09-12T19:16:45.105 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.106 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.106 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.106 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.106 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.107 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008900.667714_107.1", 2022-09-12T19:16:45.107 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.108 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14", 2022-09-12T19:16:45.108 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.667714167Z", 2022-09-12T19:16:45.108 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.108 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.109 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.109 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.110 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.110 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_ULxp_YwqTEGv8QDweh9UlvR", 2022-09-12T19:16:45.110 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.111 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.111 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.111 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.112 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.112 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008900.669743_108.1", 2022-09-12T19:16:45.113 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.113 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-14", 2022-09-12T19:16:45.113 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.669743466Z", 2022-09-12T19:16:45.113 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.113 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.114 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.114 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.115 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.115 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Zohm3vyJdVty1-BsovuZx7M", 2022-09-12T19:16:45.115 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.116 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.116 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.116 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.116 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.117 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008900.675413_109.1", 2022-09-12T19:16:45.117 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.118 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-14", 2022-09-12T19:16:45.118 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.675413478Z", 2022-09-12T19:16:45.118 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.119 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.119 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.119 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.120 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.120 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Zohm3vyJdVty1-BsovuZx7M", 2022-09-12T19:16:45.120 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.120 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.121 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.121 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.121 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.121 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.122 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008900.713502_110.1", 2022-09-12T19:16:45.122 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.123 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15", 2022-09-12T19:16:45.123 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.713502192Z", 2022-09-12T19:16:45.123 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.123 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.124 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.124 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.124 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.124 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.125 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_AKH72TwSsQVrOOdb7zgDFt1", 2022-09-12T19:16:45.125 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.125 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.125 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.126 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.126 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.126 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.127 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008900.716530_111.1", 2022-09-12T19:16:45.127 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.128 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15", 2022-09-12T19:16:45.128 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.716530511Z", 2022-09-12T19:16:45.128 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.128 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.129 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.129 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.129 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.129 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.129 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_AKH72TwSsQVrOOdb7zgDFt1", 2022-09-12T19:16:45.130 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.130 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.130 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.130 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.131 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.131 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.131 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.132 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008900.718433_112.1", 2022-09-12T19:16:45.132 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.132 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-15", 2022-09-12T19:16:45.132 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.718433663Z", 2022-09-12T19:16:45.133 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.133 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.133 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.133 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.134 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.134 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_i61YjmwdeL9p7lC_EIZUJTo", 2022-09-12T19:16:45.135 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.135 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.135 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.135 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.136 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.136 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.136 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.136 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.136 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008900.721341_113.1", 2022-09-12T19:16:45.137 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.137 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-15", 2022-09-12T19:16:45.137 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.721341013Z", 2022-09-12T19:16:45.137 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.138 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.138 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.138 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.138 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.139 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.139 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_i61YjmwdeL9p7lC_EIZUJTo", 2022-09-12T19:16:45.139 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.139 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.140 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.140 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.140 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.140 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.141 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.141 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.141 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.182484_132.1", 2022-09-12T19:16:45.141 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.142 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16", 2022-09-12T19:16:45.142 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.182484917Z", 2022-09-12T19:16:45.142 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.142 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.143 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.143 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.143 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.143 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.144 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_V99BgSq-hvAYBvkex60C8n2", 2022-09-12T19:16:45.144 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.144 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.144 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.145 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.145 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.145 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.145 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.146 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.146 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.186031_133.1", 2022-09-12T19:16:45.146 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.146 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16", 2022-09-12T19:16:45.147 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.186031455Z", 2022-09-12T19:16:45.147 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.147 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.147 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.148 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.148 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.148 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_V99BgSq-hvAYBvkex60C8n2", 2022-09-12T19:16:45.149 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.149 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.149 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.149 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.150 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.150 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.151 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.187394_134.1", 2022-09-12T19:16:45.151 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.151 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-16", 2022-09-12T19:16:45.151 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.187394124Z", 2022-09-12T19:16:45.152 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.152 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.152 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.152 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.153 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.153 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_ehigIfW992JSI46zHL04P5J", 2022-09-12T19:16:45.153 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.154 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.154 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.154 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.155 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.155 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.193049_135.1", 2022-09-12T19:16:45.156 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.156 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-16", 2022-09-12T19:16:45.156 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.193049111Z", 2022-09-12T19:16:45.156 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.157 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.157 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.157 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.158 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.158 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_ehigIfW992JSI46zHL04P5J", 2022-09-12T19:16:45.158 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.158 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.159 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.159 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.159 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.159 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.160 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.160 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.223026_136.1", 2022-09-12T19:16:45.160 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.161 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17", 2022-09-12T19:16:45.161 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.223026973Z", 2022-09-12T19:16:45.161 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.161 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.162 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.162 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.162 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.163 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_BCZEMlNNfcs_zua_I4p9TaW", 2022-09-12T19:16:45.163 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.163 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.164 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.164 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.164 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.165 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.226575_137.1", 2022-09-12T19:16:45.165 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.165 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17", 2022-09-12T19:16:45.166 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.226575054Z", 2022-09-12T19:16:45.166 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.166 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.166 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.167 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.167 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.167 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.168 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_BCZEMlNNfcs_zua_I4p9TaW", 2022-09-12T19:16:45.168 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.168 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.168 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.169 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.169 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.169 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.169 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.170 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.170 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.227798_138.1", 2022-09-12T19:16:45.170 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.170 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-17", 2022-09-12T19:16:45.171 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.227798137Z", 2022-09-12T19:16:45.171 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.171 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.172 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.172 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.172 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.172 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.173 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_i8wOFAvzgTHQCJOO8N6Bnam", 2022-09-12T19:16:45.173 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.173 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.173 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.174 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.174 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.174 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.174 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.175 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.175 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.232329_139.1", 2022-09-12T19:16:45.175 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.176 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-17", 2022-09-12T19:16:45.176 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.232329182Z", 2022-09-12T19:16:45.176 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.176 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.177 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.177 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.177 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.177 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.178 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_i8wOFAvzgTHQCJOO8N6Bnam", 2022-09-12T19:16:45.178 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.178 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.178 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.179 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.179 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.179 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.179 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.180 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.180 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.268820_140.1", 2022-09-12T19:16:45.180 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.180 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18", 2022-09-12T19:16:45.181 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.268820788Z", 2022-09-12T19:16:45.181 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.181 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.182 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.182 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.182 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.182 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.183 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_5GsV_zMTqFSGin4m-BDaD32", 2022-09-12T19:16:45.183 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.183 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.183 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.184 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.184 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.184 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.184 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.185 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.185 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.282707_141.1", 2022-09-12T19:16:45.185 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.186 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18", 2022-09-12T19:16:45.186 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.282707229Z", 2022-09-12T19:16:45.186 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.186 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.187 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.187 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.187 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.187 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.188 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_5GsV_zMTqFSGin4m-BDaD32", 2022-09-12T19:16:45.188 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.188 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.188 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.189 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.189 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.189 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.189 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.190 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.190 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.284810_142.1", 2022-09-12T19:16:45.190 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.190 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-18", 2022-09-12T19:16:45.191 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.284810891Z", 2022-09-12T19:16:45.191 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.191 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.191 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.192 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.192 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.192 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.192 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Ow-fYJ8C7sQ2JLuZ3C6E5Oy", 2022-09-12T19:16:45.193 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.193 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.193 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.193 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.194 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.194 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.194 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.194 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.195 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.287795_143.1", 2022-09-12T19:16:45.195 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.195 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-18", 2022-09-12T19:16:45.195 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.287795384Z", 2022-09-12T19:16:45.196 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.196 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.196 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.196 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.197 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.197 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.197 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Ow-fYJ8C7sQ2JLuZ3C6E5Oy", 2022-09-12T19:16:45.197 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.198 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.198 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.198 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.198 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.199 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.199 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.199 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.199 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.308137_144.1", 2022-09-12T19:16:45.200 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.200 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16", 2022-09-12T19:16:45.200 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.308137433Z", 2022-09-12T19:16:45.201 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.201 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.201 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_V99BgSq-hvAYBvkex60C8n2", 2022-09-12T19:16:45.202 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.202 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.202 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.203 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_V99BgSq-hvAYBvkex60C8n2", 2022-09-12T19:16:45.203 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.203 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.204 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.204 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.204 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.204 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.205 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.205 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.205 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.310988_145.1", 2022-09-12T19:16:45.205 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.206 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16", 2022-09-12T19:16:45.206 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.310988969Z", 2022-09-12T19:16:45.206 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.207 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.207 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_V99BgSq-hvAYBvkex60C8n2", 2022-09-12T19:16:45.207 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.208 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.208 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_V99BgSq-hvAYBvkex60C8n2", 2022-09-12T19:16:45.208 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.209 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.209 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.209 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.209 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.210 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.210 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.211 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.211 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.322795_146.1", 2022-09-12T19:16:45.211 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.211 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17", 2022-09-12T19:16:45.212 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.322795845Z", 2022-09-12T19:16:45.212 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.212 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.212 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_BCZEMlNNfcs_zua_I4p9TaW", 2022-09-12T19:16:45.213 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.213 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.213 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.213 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_BCZEMlNNfcs_zua_I4p9TaW", 2022-09-12T19:16:45.214 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.214 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.214 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.214 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.215 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.215 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.215 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.215 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.216 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.325825_147.1", 2022-09-12T19:16:45.216 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.216 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17", 2022-09-12T19:16:45.217 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.325825003Z", 2022-09-12T19:16:45.217 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.217 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.217 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_BCZEMlNNfcs_zua_I4p9TaW", 2022-09-12T19:16:45.218 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.218 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.218 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.218 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_BCZEMlNNfcs_zua_I4p9TaW", 2022-09-12T19:16:45.219 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.219 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.219 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.220 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.220 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.220 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.220 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.221 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.221 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.343478_148.1", 2022-09-12T19:16:45.221 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.221 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18", 2022-09-12T19:16:45.222 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.343478792Z", 2022-09-12T19:16:45.222 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.222 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.223 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_5GsV_zMTqFSGin4m-BDaD32", 2022-09-12T19:16:45.223 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.223 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.223 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.224 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_5GsV_zMTqFSGin4m-BDaD32", 2022-09-12T19:16:45.224 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.224 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.224 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.225 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.225 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.225 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.226 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.226 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.226 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.346375_149.1", 2022-09-12T19:16:45.226 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.227 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18", 2022-09-12T19:16:45.227 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.346375662Z", 2022-09-12T19:16:45.227 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.227 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.228 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_5GsV_zMTqFSGin4m-BDaD32", 2022-09-12T19:16:45.228 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.228 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.228 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.229 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_5GsV_zMTqFSGin4m-BDaD32", 2022-09-12T19:16:45.229 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.229 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.229 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.230 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.230 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.230 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.231 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.231 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.231 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009433.572798_179.1", 2022-09-12T19:16:45.231 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.232 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19", 2022-09-12T19:16:45.232 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:03:53.572798305Z", 2022-09-12T19:16:45.232 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.232 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.233 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.233 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.233 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.234 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.234 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_r98RnZbo3gbuHHD5iR9vaQX", 2022-09-12T19:16:45.234 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.235 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.235 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.235 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.235 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.236 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.236 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.236 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.236 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009433.576124_180.1", 2022-09-12T19:16:45.237 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.237 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19", 2022-09-12T19:16:45.237 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:03:53.576124288Z", 2022-09-12T19:16:45.237 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.238 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.238 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.238 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.239 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.239 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_r98RnZbo3gbuHHD5iR9vaQX", 2022-09-12T19:16:45.239 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.239 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.240 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.240 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.240 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.240 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.240 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.241 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.241 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009433.577370_181.1", 2022-09-12T19:16:45.241 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.241 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-19", 2022-09-12T19:16:45.242 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:03:53.577370053Z", 2022-09-12T19:16:45.242 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.242 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.243 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.243 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.243 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.243 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_FepK5riJM4xGtHwIBSHMDgN", 2022-09-12T19:16:45.244 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.244 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.244 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.245 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.245 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.245 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.245 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.246 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009433.580348_182.1", 2022-09-12T19:16:45.246 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.246 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-19", 2022-09-12T19:16:45.246 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:03:53.580348934Z", 2022-09-12T19:16:45.254 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.254 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.254 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.255 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.255 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.255 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.255 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_FepK5riJM4xGtHwIBSHMDgN", 2022-09-12T19:16:45.256 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.256 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.256 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.257 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.257 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.257 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.258 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.258 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009433.593338_183.1", 2022-09-12T19:16:45.258 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.258 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19", 2022-09-12T19:16:45.259 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:03:53.593338729Z", 2022-09-12T19:16:45.259 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.259 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.259 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_r98RnZbo3gbuHHD5iR9vaQX", 2022-09-12T19:16:45.260 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.260 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.260 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_r98RnZbo3gbuHHD5iR9vaQX", 2022-09-12T19:16:45.261 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.261 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.261 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.262 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.262 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.262 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.263 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.263 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009433.597955_184.1", 2022-09-12T19:16:45.263 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.263 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19", 2022-09-12T19:16:45.263 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:03:53.597955912Z", 2022-09-12T19:16:45.264 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.264 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.264 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_r98RnZbo3gbuHHD5iR9vaQX", 2022-09-12T19:16:45.264 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.265 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.265 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.265 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_r98RnZbo3gbuHHD5iR9vaQX", 2022-09-12T19:16:45.265 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.266 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.266 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.266 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.267 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.267 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.267 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.267 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.268 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009630.186894_193.1", 2022-09-12T19:16:45.268 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.268 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:16:45.268 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:07:10.186894533Z", 2022-09-12T19:16:45.269 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.269 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.269 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.269 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.270 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.270 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.270 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:45.270 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.271 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.271 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.271 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.271 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.272 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.272 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009630.192780_194.1", 2022-09-12T19:16:45.273 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.273 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:16:45.273 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:07:10.192780968Z", 2022-09-12T19:16:45.273 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.274 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.274 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.274 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.274 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.275 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.275 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:45.275 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.275 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.276 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.276 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.276 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.276 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.277 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.277 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009630.194692_195.1", 2022-09-12T19:16:45.277 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.278 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-20", 2022-09-12T19:16:45.278 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:07:10.194692069Z", 2022-09-12T19:16:45.278 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.278 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.279 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.279 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.279 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.279 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.280 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_vBgV2Ivb4f-6P44JJPXNdtC", 2022-09-12T19:16:45.280 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.280 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.280 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.281 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.281 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.281 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.281 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.282 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.282 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009630.200316_196.1", 2022-09-12T19:16:45.282 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.282 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-20", 2022-09-12T19:16:45.283 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:07:10.200316464Z", 2022-09-12T19:16:45.283 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.283 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.283 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.284 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.284 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.284 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.284 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_vBgV2Ivb4f-6P44JJPXNdtC", 2022-09-12T19:16:45.285 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.285 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.285 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.286 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.286 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.286 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.286 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.287 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009668.037998_204.1", 2022-09-12T19:16:45.287 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.287 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:16:45.287 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:07:48.037998619Z", 2022-09-12T19:16:45.287 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.288 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.288 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:45.288 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.288 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.289 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.289 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:45.289 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.290 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.290 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.290 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.290 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.291 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.291 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.291 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009668.041232_205.1", 2022-09-12T19:16:45.291 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.292 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:16:45.292 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:07:48.041232517Z", 2022-09-12T19:16:45.292 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.292 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.293 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:45.293 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.293 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.293 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.294 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:45.294 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.294 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.294 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.295 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.295 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.295 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.295 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.296 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.296 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009726.091934_207.1", 2022-09-12T19:16:45.296 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.296 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:16:45.297 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:08:46.091934751Z", 2022-09-12T19:16:45.297 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.297 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.297 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:45.298 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.298 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.298 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.298 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:45.299 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 3 2022-09-12T19:16:45.299 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.299 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.299 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.300 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.300 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.300 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.300 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.301 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009726.095686_208.1", 2022-09-12T19:16:45.301 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.301 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:16:45.301 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:08:46.095686282Z", 2022-09-12T19:16:45.302 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.302 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.302 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:45.302 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.303 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.303 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.303 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:45.304 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 3 2022-09-12T19:16:45.304 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.305 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.305 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.306 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.306 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.307 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.307 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.308 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009842.925161_211.1", 2022-09-12T19:16:45.308 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.309 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21", 2022-09-12T19:16:45.309 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:42.925161228Z", 2022-09-12T19:16:45.310 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.310 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.310 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.311 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.311 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.312 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.312 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_wA9vY_PYbZdc0eUi2hSxrjB", 2022-09-12T19:16:45.313 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.313 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.313 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.314 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.314 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.314 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.315 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.315 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.316 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009842.930693_212.1", 2022-09-12T19:16:45.316 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.316 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21", 2022-09-12T19:16:45.317 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:42.930693175Z", 2022-09-12T19:16:45.317 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.317 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.317 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.318 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.318 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.318 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.319 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_wA9vY_PYbZdc0eUi2hSxrjB", 2022-09-12T19:16:45.319 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.319 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.319 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.320 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.320 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.320 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.320 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.320 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.321 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009842.932348_213.1", 2022-09-12T19:16:45.321 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.321 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-21", 2022-09-12T19:16:45.321 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:42.932348571Z", 2022-09-12T19:16:45.322 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.322 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.322 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.323 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.323 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.323 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.323 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_YhO7K8SUXb88GVnz95qTMjU", 2022-09-12T19:16:45.324 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.324 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.324 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.324 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.324 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.325 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.325 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.326 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.326 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009842.938156_214.1", 2022-09-12T19:16:45.327 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.327 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-21", 2022-09-12T19:16:45.328 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:42.938156918Z", 2022-09-12T19:16:45.328 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.328 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.329 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.329 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.330 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.330 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.331 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_YhO7K8SUXb88GVnz95qTMjU", 2022-09-12T19:16:45.331 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.332 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.332 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.333 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.333 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.334 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.334 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.335 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.335 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009842.960242_215.1", 2022-09-12T19:16:45.335 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.336 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22", 2022-09-12T19:16:45.336 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:42.960242821Z", 2022-09-12T19:16:45.337 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.337 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.337 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.338 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.338 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.339 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.339 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_jCE0vI8jcGzuttLr9LzBCWJ", 2022-09-12T19:16:45.340 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.341 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.341 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.341 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.341 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.342 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.342 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.342 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009842.965349_216.1", 2022-09-12T19:16:45.342 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.343 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22", 2022-09-12T19:16:45.343 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:42.965349366Z", 2022-09-12T19:16:45.343 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.343 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.344 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.344 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.344 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.344 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.345 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_jCE0vI8jcGzuttLr9LzBCWJ", 2022-09-12T19:16:45.345 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.345 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.345 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.346 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.347 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.347 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.347 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.347 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.348 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009842.966491_217.1", 2022-09-12T19:16:45.348 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.349 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-22", 2022-09-12T19:16:45.349 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:42.966491565Z", 2022-09-12T19:16:45.349 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.349 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.350 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.350 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.350 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.350 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.351 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_90qhmHoutPPaDynubPFZH9h", 2022-09-12T19:16:45.351 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.352 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.352 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.353 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.353 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.354 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.354 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.355 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.355 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009842.970703_218.1", 2022-09-12T19:16:45.356 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.356 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-22", 2022-09-12T19:16:45.357 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:42.970703166Z", 2022-09-12T19:16:45.357 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.358 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.358 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.359 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.359 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.360 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.360 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_90qhmHoutPPaDynubPFZH9h", 2022-09-12T19:16:45.360 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.361 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.361 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.362 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.362 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.363 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.363 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.364 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.364 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009843.006494_219.1", 2022-09-12T19:16:45.365 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.365 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23", 2022-09-12T19:16:45.366 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:43.006494173Z", 2022-09-12T19:16:45.366 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.367 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.367 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.368 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.368 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.369 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.369 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_8EYxPua6E_71y78ZWvmZh7v", 2022-09-12T19:16:45.369 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.370 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.370 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.371 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.371 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.371 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.372 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.372 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.373 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009843.009740_220.1", 2022-09-12T19:16:45.373 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.373 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23", 2022-09-12T19:16:45.374 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:43.009740592Z", 2022-09-12T19:16:45.374 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.375 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.375 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.376 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.376 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.377 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.377 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_8EYxPua6E_71y78ZWvmZh7v", 2022-09-12T19:16:45.378 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.378 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.378 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.379 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.379 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.380 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.380 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009843.011011_221.1", 2022-09-12T19:16:45.381 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.381 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-23", 2022-09-12T19:16:45.381 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:43.011011338Z", 2022-09-12T19:16:45.382 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.382 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.382 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.382 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.383 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.383 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.383 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_EWfzEo4mEzPqDzTBu2wloku", 2022-09-12T19:16:45.383 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.384 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.384 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.384 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.384 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.385 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.385 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.385 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.385 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009843.015119_222.1", 2022-09-12T19:16:45.386 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.386 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-23", 2022-09-12T19:16:45.386 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:43.015119585Z", 2022-09-12T19:16:45.387 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.387 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.388 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.388 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.388 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.389 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.389 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_EWfzEo4mEzPqDzTBu2wloku", 2022-09-12T19:16:45.389 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.390 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.390 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.390 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.390 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.391 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.391 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.391 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.392 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009843.040988_223.1", 2022-09-12T19:16:45.392 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.392 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21", 2022-09-12T19:16:45.392 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:43.040988054Z", 2022-09-12T19:16:45.393 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.393 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.393 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_wA9vY_PYbZdc0eUi2hSxrjB", 2022-09-12T19:16:45.393 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.394 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.394 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.394 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_wA9vY_PYbZdc0eUi2hSxrjB", 2022-09-12T19:16:45.395 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.395 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.395 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.395 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.396 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.396 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.396 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.397 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.397 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009843.045887_224.1", 2022-09-12T19:16:45.397 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.398 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21", 2022-09-12T19:16:45.398 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:43.045887351Z", 2022-09-12T19:16:45.398 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.398 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.399 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_wA9vY_PYbZdc0eUi2hSxrjB", 2022-09-12T19:16:45.399 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.399 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.400 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.400 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_wA9vY_PYbZdc0eUi2hSxrjB", 2022-09-12T19:16:45.400 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.401 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.401 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.401 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.401 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.402 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.402 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009843.057828_225.1", 2022-09-12T19:16:45.403 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.403 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22", 2022-09-12T19:16:45.403 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:43.057828971Z", 2022-09-12T19:16:45.403 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.404 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.404 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_jCE0vI8jcGzuttLr9LzBCWJ", 2022-09-12T19:16:45.405 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.405 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.405 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.405 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_jCE0vI8jcGzuttLr9LzBCWJ", 2022-09-12T19:16:45.406 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.406 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.406 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.407 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.407 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.408 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009843.060815_226.1", 2022-09-12T19:16:45.408 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.408 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22", 2022-09-12T19:16:45.409 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:43.060815675Z", 2022-09-12T19:16:45.409 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.409 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.409 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_jCE0vI8jcGzuttLr9LzBCWJ", 2022-09-12T19:16:45.410 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.410 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.410 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_jCE0vI8jcGzuttLr9LzBCWJ", 2022-09-12T19:16:45.411 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.411 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.411 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.412 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.412 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.413 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.413 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009843.077332_227.1", 2022-09-12T19:16:45.413 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.414 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23", 2022-09-12T19:16:45.414 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:43.077332048Z", 2022-09-12T19:16:45.414 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.414 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.415 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_8EYxPua6E_71y78ZWvmZh7v", 2022-09-12T19:16:45.415 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.416 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.416 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_8EYxPua6E_71y78ZWvmZh7v", 2022-09-12T19:16:45.416 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.417 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.417 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.417 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.417 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.418 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009843.080040_228.1", 2022-09-12T19:16:45.418 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.419 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23", 2022-09-12T19:16:45.419 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:43.080040677Z", 2022-09-12T19:16:45.419 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.419 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.420 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_8EYxPua6E_71y78ZWvmZh7v", 2022-09-12T19:16:45.420 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.421 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.421 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.421 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_8EYxPua6E_71y78ZWvmZh7v", 2022-09-12T19:16:45.422 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.422 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.423 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.423 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.423 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.424 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.213940_231.1", 2022-09-12T19:16:45.424 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.424 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:16:45.425 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.213940834Z", 2022-09-12T19:16:45.425 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.425 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.425 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.426 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.426 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.426 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.426 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:45.427 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.427 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.427 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.427 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.428 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.428 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.428 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.429 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.217329_232.1", 2022-09-12T19:16:45.429 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.429 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:16:45.430 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.217329420Z", 2022-09-12T19:16:45.430 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.431 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.431 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.431 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.432 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.432 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:45.432 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.433 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.433 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.433 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.433 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.434 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.434 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.434 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.218459_233.1", 2022-09-12T19:16:45.435 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.435 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-24", 2022-09-12T19:16:45.435 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.218459288Z", 2022-09-12T19:16:45.435 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.436 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.436 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.436 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.437 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.437 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_zt4J3xCzAzkFDmMYSv2ZhpD", 2022-09-12T19:16:45.437 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.438 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.438 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.438 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.439 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.439 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.439 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.440 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.440 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.221613_234.1", 2022-09-12T19:16:45.440 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.440 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-24", 2022-09-12T19:16:45.441 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.221613900Z", 2022-09-12T19:16:45.441 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.441 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.441 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.442 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.442 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.442 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_zt4J3xCzAzkFDmMYSv2ZhpD", 2022-09-12T19:16:45.443 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.443 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.443 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.443 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.444 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.444 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.444 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.445 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.242266_235.1", 2022-09-12T19:16:45.445 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.445 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:16:45.446 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.242266250Z", 2022-09-12T19:16:45.446 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.446 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.446 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.447 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.447 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.447 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:45.448 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.448 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.448 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.449 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.449 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.449 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.450 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.450 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.245374_236.1", 2022-09-12T19:16:45.450 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.450 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:16:45.451 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.245374907Z", 2022-09-12T19:16:45.451 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.451 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.451 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.452 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.452 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.452 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:45.453 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.453 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.453 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.453 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.454 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.454 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.454 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.455 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.455 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.246464_237.1", 2022-09-12T19:16:45.455 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.456 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-25", 2022-09-12T19:16:45.456 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.246464476Z", 2022-09-12T19:16:45.456 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.456 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.457 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.457 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.457 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.457 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.458 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_FuWLdq4G8Pitukb9rq9CEoZ", 2022-09-12T19:16:45.458 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.458 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.459 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.459 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.459 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.459 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.460 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.460 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.460 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.249409_238.1", 2022-09-12T19:16:45.460 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.461 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-25", 2022-09-12T19:16:45.461 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.249409215Z", 2022-09-12T19:16:45.461 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.462 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.462 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.462 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.462 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.463 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.463 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_FuWLdq4G8Pitukb9rq9CEoZ", 2022-09-12T19:16:45.464 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.464 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.464 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.464 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.465 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.465 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.466 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.466 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.282380_239.1", 2022-09-12T19:16:45.466 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.467 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:16:45.467 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.282380399Z", 2022-09-12T19:16:45.467 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.468 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.468 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.468 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.469 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.469 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:45.469 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.470 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.470 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.470 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.470 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.471 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.471 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.285471_240.1", 2022-09-12T19:16:45.472 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.472 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:16:45.472 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.285471857Z", 2022-09-12T19:16:45.473 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.473 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.473 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.474 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.474 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.474 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:45.475 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.475 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.475 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.476 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.476 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.477 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.286588_241.1", 2022-09-12T19:16:45.477 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.477 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-26", 2022-09-12T19:16:45.477 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.286588982Z", 2022-09-12T19:16:45.478 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.478 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.478 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.478 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.479 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.479 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_-j_vUqBTsANTAcA80szuZUK", 2022-09-12T19:16:45.479 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.480 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.480 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.480 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.481 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.481 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.289386_242.1", 2022-09-12T19:16:45.482 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.482 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-26", 2022-09-12T19:16:45.482 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.289386844Z", 2022-09-12T19:16:45.482 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.483 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.483 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.483 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.484 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.484 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_-j_vUqBTsANTAcA80szuZUK", 2022-09-12T19:16:45.484 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.485 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.485 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.485 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.486 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.487 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.309215_243.1", 2022-09-12T19:16:45.487 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.487 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:16:45.487 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.309215519Z", 2022-09-12T19:16:45.488 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.488 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.488 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:45.488 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.489 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.489 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:45.490 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.490 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.490 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.491 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.491 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.492 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.312069_244.1", 2022-09-12T19:16:45.492 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.492 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:16:45.492 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.312069106Z", 2022-09-12T19:16:45.493 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.493 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.494 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:45.494 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.494 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.495 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:45.495 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.496 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.496 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.496 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.496 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.497 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.323214_245.1", 2022-09-12T19:16:45.497 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.498 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:16:45.498 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.323214538Z", 2022-09-12T19:16:45.498 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.498 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.499 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:45.499 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.499 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.500 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:45.500 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 3 2022-09-12T19:16:45.500 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.500 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.501 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.501 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.501 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.501 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.502 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.502 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.334825_246.1", 2022-09-12T19:16:45.502 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.502 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:16:45.503 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.334825968Z", 2022-09-12T19:16:45.503 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.503 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.503 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:45.504 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.504 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.504 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.504 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:45.505 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 3 2022-09-12T19:16:45.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.505 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.506 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.506 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.506 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.507 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.350812_247.1", 2022-09-12T19:16:45.507 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.507 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:16:45.508 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.350812474Z", 2022-09-12T19:16:45.508 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.508 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.508 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:45.509 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.509 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.509 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.509 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:45.510 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.510 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.511 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.511 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.511 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.512 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.512 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.355013_248.1", 2022-09-12T19:16:45.512 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.512 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:16:45.513 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.355013585Z", 2022-09-12T19:16:45.513 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.513 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.513 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:45.514 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.514 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.515 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:45.515 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.515 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.516 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.516 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.516 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.517 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.517 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.369892_249.1", 2022-09-12T19:16:45.517 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.517 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:16:45.518 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.369892532Z", 2022-09-12T19:16:45.518 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.518 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.519 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:45.519 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.519 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.520 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.520 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:45.520 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 3 2022-09-12T19:16:45.520 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.521 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.521 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.521 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.522 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.523 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.374217_250.1", 2022-09-12T19:16:45.524 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.524 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:16:45.525 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.374217196Z", 2022-09-12T19:16:45.525 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.526 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.526 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:45.527 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.527 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.528 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.528 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:45.529 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 3 2022-09-12T19:16:45.529 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.530 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.530 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.531 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.531 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.532 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.532 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.532 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.394432_251.1", 2022-09-12T19:16:45.533 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.533 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:16:45.534 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.394432026Z", 2022-09-12T19:16:45.534 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.535 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.535 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:45.536 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.536 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.537 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.537 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:45.538 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.538 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.538 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.539 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.539 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.539 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.539 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.540 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.540 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.398632_252.1", 2022-09-12T19:16:45.540 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.540 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:16:45.541 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.398632136Z", 2022-09-12T19:16:45.541 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.541 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.542 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:45.542 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.542 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.542 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.543 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:45.543 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.543 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.543 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.544 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.544 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.544 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.545 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.418847_253.1", 2022-09-12T19:16:45.545 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.546 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:16:45.546 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.418847506Z", 2022-09-12T19:16:45.546 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.547 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.547 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:45.547 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.548 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.548 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:45.549 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 3 2022-09-12T19:16:45.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.549 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.549 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.550 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.550 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.550 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.551 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.421418_254.1", 2022-09-12T19:16:45.551 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.551 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:16:45.552 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.421418488Z", 2022-09-12T19:16:45.552 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.552 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.553 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:45.553 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.554 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.554 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:45.554 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 3 2022-09-12T19:16:45.554 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.555 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.555 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.556 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.556 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.557 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.557 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.022864_258.1", 2022-09-12T19:16:45.557 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.558 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-27:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.27", 2022-09-12T19:16:45.558 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.022864695Z", 2022-09-12T19:16:45.558 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.559 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.559 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.559 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.560 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.560 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_4Ia0kBYSOXAPun0srTpHOgg", 2022-09-12T19:16:45.560 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.561 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.561 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.561 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.562 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.562 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.562 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.563 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.027319_259.1", 2022-09-12T19:16:45.563 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.563 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-27:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.27", 2022-09-12T19:16:45.563 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.027319061Z", 2022-09-12T19:16:45.564 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.564 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.564 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.565 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.565 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.565 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.566 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_4Ia0kBYSOXAPun0srTpHOgg", 2022-09-12T19:16:45.566 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.566 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.566 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.567 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.567 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.567 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.568 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.568 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.029529_260.1", 2022-09-12T19:16:45.568 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.568 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-27", 2022-09-12T19:16:45.569 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.029529115Z", 2022-09-12T19:16:45.569 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.569 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.570 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.570 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.570 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.570 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.571 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_hL6ubPjUnxDbFclj75F-9Yw", 2022-09-12T19:16:45.571 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.571 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.571 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.572 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.572 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.572 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.573 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.573 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.035074_261.1", 2022-09-12T19:16:45.574 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.574 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-27", 2022-09-12T19:16:45.574 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.035074135Z", 2022-09-12T19:16:45.574 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.575 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.575 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.575 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.576 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.576 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_hL6ubPjUnxDbFclj75F-9Yw", 2022-09-12T19:16:45.576 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.576 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.577 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.577 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.577 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.577 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.578 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.578 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.060437_262.1", 2022-09-12T19:16:45.578 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.579 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-28:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.28", 2022-09-12T19:16:45.579 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.060437057Z", 2022-09-12T19:16:45.579 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.579 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.580 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.580 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.581 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.581 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_F91L-y4ApRIftBlrzdVSKhB", 2022-09-12T19:16:45.581 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.582 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.582 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.583 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.583 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.584 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.067920_263.1", 2022-09-12T19:16:45.584 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.584 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-28:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.28", 2022-09-12T19:16:45.584 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.067920385Z", 2022-09-12T19:16:45.585 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.585 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.585 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.585 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.586 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.586 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_F91L-y4ApRIftBlrzdVSKhB", 2022-09-12T19:16:45.587 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.587 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.588 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.588 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.588 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.589 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.068850_264.1", 2022-09-12T19:16:45.589 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.590 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-28", 2022-09-12T19:16:45.590 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.068850124Z", 2022-09-12T19:16:45.590 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.590 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.591 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.591 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.591 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.592 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_4foA5qWzZ6DKNqXOUrqtpMi", 2022-09-12T19:16:45.592 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.592 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.593 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.593 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.593 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.594 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.074343_265.1", 2022-09-12T19:16:45.595 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.595 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-28", 2022-09-12T19:16:45.595 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.074343801Z", 2022-09-12T19:16:45.595 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.596 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.596 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.596 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.596 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.597 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.597 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_4foA5qWzZ6DKNqXOUrqtpMi", 2022-09-12T19:16:45.597 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.598 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.598 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.598 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.598 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.599 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.114860_266.1", 2022-09-12T19:16:45.600 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.600 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29", 2022-09-12T19:16:45.600 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.114860178Z", 2022-09-12T19:16:45.600 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.601 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.601 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.601 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.602 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.602 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_kHZQcZIbg231-V_zVbHphPU", 2022-09-12T19:16:45.602 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.603 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.603 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.604 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.604 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.605 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.119876_267.1", 2022-09-12T19:16:45.605 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.605 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29", 2022-09-12T19:16:45.605 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.119876089Z", 2022-09-12T19:16:45.606 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.606 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.606 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.607 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.607 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.607 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_kHZQcZIbg231-V_zVbHphPU", 2022-09-12T19:16:45.608 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.608 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.608 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.609 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.609 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.610 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.121725_268.1", 2022-09-12T19:16:45.610 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.610 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-29", 2022-09-12T19:16:45.610 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.121725517Z", 2022-09-12T19:16:45.611 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.611 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.611 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.611 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.612 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.612 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_2ti8fe4vkZVz-LOJO9SiYze", 2022-09-12T19:16:45.612 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.613 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.613 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.614 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.614 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.615 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.125711_269.1", 2022-09-12T19:16:45.615 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.615 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-29", 2022-09-12T19:16:45.616 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.125711914Z", 2022-09-12T19:16:45.616 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.617 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.617 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.617 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.617 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.618 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.618 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_2ti8fe4vkZVz-LOJO9SiYze", 2022-09-12T19:16:45.618 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.619 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.619 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.619 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.620 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.620 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.153919_270.1", 2022-09-12T19:16:45.621 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.621 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-27:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.27", 2022-09-12T19:16:45.621 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.153919920Z", 2022-09-12T19:16:45.621 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.622 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.622 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_4Ia0kBYSOXAPun0srTpHOgg", 2022-09-12T19:16:45.622 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.623 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.623 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_4Ia0kBYSOXAPun0srTpHOgg", 2022-09-12T19:16:45.623 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.624 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.624 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.624 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.625 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.625 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.625 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.625 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.156407_271.1", 2022-09-12T19:16:45.626 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.626 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-27:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.27", 2022-09-12T19:16:45.626 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.156407416Z", 2022-09-12T19:16:45.627 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.627 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.627 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_4Ia0kBYSOXAPun0srTpHOgg", 2022-09-12T19:16:45.627 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.628 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.628 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_4Ia0kBYSOXAPun0srTpHOgg", 2022-09-12T19:16:45.628 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.629 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.629 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.630 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.630 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.630 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.631 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.183174_272.1", 2022-09-12T19:16:45.631 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.631 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-28:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.28", 2022-09-12T19:16:45.631 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.183174956Z", 2022-09-12T19:16:45.632 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.632 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.632 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_F91L-y4ApRIftBlrzdVSKhB", 2022-09-12T19:16:45.633 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.633 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.633 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_F91L-y4ApRIftBlrzdVSKhB", 2022-09-12T19:16:45.634 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.634 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.634 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.635 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.635 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.646 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.186673_273.1", 2022-09-12T19:16:45.647 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.647 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-28:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.28", 2022-09-12T19:16:45.647 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.186673632Z", 2022-09-12T19:16:45.648 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.648 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.649 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_F91L-y4ApRIftBlrzdVSKhB", 2022-09-12T19:16:45.649 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.650 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.651 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_F91L-y4ApRIftBlrzdVSKhB", 2022-09-12T19:16:45.651 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.652 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.652 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.653 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.653 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.655 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.258924_274.1", 2022-09-12T19:16:45.656 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.656 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29", 2022-09-12T19:16:45.656 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.258924078Z", 2022-09-12T19:16:45.657 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.657 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.658 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_kHZQcZIbg231-V_zVbHphPU", 2022-09-12T19:16:45.658 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.659 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.660 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_kHZQcZIbg231-V_zVbHphPU", 2022-09-12T19:16:45.660 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.661 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.662 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.662 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.662 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.664 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.261587_275.1", 2022-09-12T19:16:45.664 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.665 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29", 2022-09-12T19:16:45.665 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.261587678Z", 2022-09-12T19:16:45.666 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.666 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.667 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_kHZQcZIbg231-V_zVbHphPU", 2022-09-12T19:16:45.667 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.668 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.668 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_kHZQcZIbg231-V_zVbHphPU", 2022-09-12T19:16:45.669 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.670 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.670 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.671 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.671 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.673 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.295442_276.1", 2022-09-12T19:16:45.673 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.674 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-30:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.30", 2022-09-12T19:16:45.674 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.295442828Z", 2022-09-12T19:16:45.675 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.675 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.676 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.676 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.677 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.677 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_xXGncRxFB5weAPxs1wnNxZq", 2022-09-12T19:16:45.677 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.678 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.678 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.679 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.679 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.680 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.297949_277.1", 2022-09-12T19:16:45.680 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.680 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-30:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.30", 2022-09-12T19:16:45.680 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.297949202Z", 2022-09-12T19:16:45.681 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.681 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.681 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.681 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.682 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.683 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_xXGncRxFB5weAPxs1wnNxZq", 2022-09-12T19:16:45.683 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.683 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.684 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.684 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.684 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.685 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.298820_278.1", 2022-09-12T19:16:45.685 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.686 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-30", 2022-09-12T19:16:45.686 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.298820021Z", 2022-09-12T19:16:45.686 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.687 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.687 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.688 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.688 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.689 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_o1lHEa9WwJ0A1-_nw8PZQAy", 2022-09-12T19:16:45.689 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.689 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.690 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.690 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.690 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.691 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.301106_279.1", 2022-09-12T19:16:45.692 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.692 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-30", 2022-09-12T19:16:45.692 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.301106622Z", 2022-09-12T19:16:45.693 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.693 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.693 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.694 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.695 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.695 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_o1lHEa9WwJ0A1-_nw8PZQAy", 2022-09-12T19:16:45.696 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.697 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.697 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.698 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.698 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.700 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.318892_280.1", 2022-09-12T19:16:45.700 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.701 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-31:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.31", 2022-09-12T19:16:45.701 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.318892577Z", 2022-09-12T19:16:45.702 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.702 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.703 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.704 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.705 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.705 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_aXhKq1Q-6is3aiib6Zv69Wn", 2022-09-12T19:16:45.706 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.707 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.707 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.708 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.708 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.710 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.321371_281.1", 2022-09-12T19:16:45.710 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.711 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-31:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.31", 2022-09-12T19:16:45.711 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.321371696Z", 2022-09-12T19:16:45.712 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.712 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.713 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.713 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.715 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.715 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_aXhKq1Q-6is3aiib6Zv69Wn", 2022-09-12T19:16:45.716 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.717 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.717 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.718 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.718 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.719 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.720 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.323664_282.1", 2022-09-12T19:16:45.720 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.721 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-31", 2022-09-12T19:16:45.721 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.323664287Z", 2022-09-12T19:16:45.722 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.723 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.723 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.724 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.725 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.725 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_ZXsXmNqWQzafZqU8AgaaFgE", 2022-09-12T19:16:45.726 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.726 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.727 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.727 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.728 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.729 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.730 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.332269_283.1", 2022-09-12T19:16:45.731 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.731 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-31", 2022-09-12T19:16:45.732 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.332269695Z", 2022-09-12T19:16:45.732 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.733 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.733 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.734 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.735 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.735 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_ZXsXmNqWQzafZqU8AgaaFgE", 2022-09-12T19:16:45.736 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.737 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.737 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.738 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.738 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.740 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.379861_284.1", 2022-09-12T19:16:45.740 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.740 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-32:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.32", 2022-09-12T19:16:45.741 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.379861593Z", 2022-09-12T19:16:45.741 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.742 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.742 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.743 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.744 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.745 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UeviL3qodKD5NX6Gmu8wcLZ", 2022-09-12T19:16:45.745 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.746 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.746 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.747 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.747 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.748 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.385256_285.1", 2022-09-12T19:16:45.749 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.749 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-32:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.32", 2022-09-12T19:16:45.750 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.385256336Z", 2022-09-12T19:16:45.750 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.751 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.751 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.752 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.752 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.752 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.753 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UeviL3qodKD5NX6Gmu8wcLZ", 2022-09-12T19:16:45.753 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.754 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.754 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.755 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.755 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.755 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.756 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.756 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.757 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.388164_286.1", 2022-09-12T19:16:45.757 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.757 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-32", 2022-09-12T19:16:45.758 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.388164607Z", 2022-09-12T19:16:45.758 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.758 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.759 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.759 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.760 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.760 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.760 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_sSvBc9sHjXg70XiY66Bq6ME", 2022-09-12T19:16:45.761 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.761 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.762 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.762 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.763 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.763 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.763 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.764 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.764 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.392572_287.1", 2022-09-12T19:16:45.765 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.765 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-32", 2022-09-12T19:16:45.766 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.392572009Z", 2022-09-12T19:16:45.766 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.766 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.766 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.767 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.767 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.767 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.768 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_sSvBc9sHjXg70XiY66Bq6ME", 2022-09-12T19:16:45.768 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.768 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.769 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.769 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.769 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.770 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.770 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.770 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.771 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.426848_288.1", 2022-09-12T19:16:45.771 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.771 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-30:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.30", 2022-09-12T19:16:45.771 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.426848512Z", 2022-09-12T19:16:45.772 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.772 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.773 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_xXGncRxFB5weAPxs1wnNxZq", 2022-09-12T19:16:45.773 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.774 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.774 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.774 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_xXGncRxFB5weAPxs1wnNxZq", 2022-09-12T19:16:45.774 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.775 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.775 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.776 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.776 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.777 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.777 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.778 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.778 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.431873_289.1", 2022-09-12T19:16:45.779 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.779 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-30:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.30", 2022-09-12T19:16:45.780 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.431873816Z", 2022-09-12T19:16:45.780 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.780 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.781 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_xXGncRxFB5weAPxs1wnNxZq", 2022-09-12T19:16:45.781 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.782 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.782 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.783 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_xXGncRxFB5weAPxs1wnNxZq", 2022-09-12T19:16:45.783 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.783 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.784 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.784 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.784 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.784 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.785 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.785 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.786 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.505940_290.1", 2022-09-12T19:16:45.786 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.787 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-31:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.31", 2022-09-12T19:16:45.787 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.505940792Z", 2022-09-12T19:16:45.788 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.788 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.788 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_aXhKq1Q-6is3aiib6Zv69Wn", 2022-09-12T19:16:45.789 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.789 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.790 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.791 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_aXhKq1Q-6is3aiib6Zv69Wn", 2022-09-12T19:16:45.791 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.792 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.792 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.793 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.793 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.794 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.794 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.795 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.795 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.509791_291.1", 2022-09-12T19:16:45.796 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.796 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-31:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.31", 2022-09-12T19:16:45.797 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.509791403Z", 2022-09-12T19:16:45.798 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.798 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.799 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_aXhKq1Q-6is3aiib6Zv69Wn", 2022-09-12T19:16:45.799 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.800 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.800 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.801 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_aXhKq1Q-6is3aiib6Zv69Wn", 2022-09-12T19:16:45.801 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.802 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.802 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.803 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.803 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.804 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.804 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.805 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.805 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.545979_292.1", 2022-09-12T19:16:45.806 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.806 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-32:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.32", 2022-09-12T19:16:45.806 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.545979472Z", 2022-09-12T19:16:45.807 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.807 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.808 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UeviL3qodKD5NX6Gmu8wcLZ", 2022-09-12T19:16:45.808 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.810 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.810 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UeviL3qodKD5NX6Gmu8wcLZ", 2022-09-12T19:16:45.810 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.811 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.811 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.812 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.812 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.813 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.814 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.555815_293.1", 2022-09-12T19:16:45.814 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.815 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-32:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.32", 2022-09-12T19:16:45.815 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.555815933Z", 2022-09-12T19:16:45.816 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.816 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.817 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UeviL3qodKD5NX6Gmu8wcLZ", 2022-09-12T19:16:45.817 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.818 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.818 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UeviL3qodKD5NX6Gmu8wcLZ", 2022-09-12T19:16:45.818 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.819 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.819 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.820 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.820 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.821 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.629617_294.1", 2022-09-12T19:16:45.821 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.821 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-33:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.33", 2022-09-12T19:16:45.822 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.629617979Z", 2022-09-12T19:16:45.822 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.822 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.823 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.823 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.824 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.824 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_9Dhxqv90pxoXFdD9Kil8AZS", 2022-09-12T19:16:45.825 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.826 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.827 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.827 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.827 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.829 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.632160_295.1", 2022-09-12T19:16:45.829 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.830 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-33:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.33", 2022-09-12T19:16:45.830 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.632160672Z", 2022-09-12T19:16:45.831 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.831 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.831 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.832 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.832 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.833 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.833 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_9Dhxqv90pxoXFdD9Kil8AZS", 2022-09-12T19:16:45.834 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.835 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.835 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.835 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.836 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.837 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.633025_296.1", 2022-09-12T19:16:45.838 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.838 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-33", 2022-09-12T19:16:45.839 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.633025749Z", 2022-09-12T19:16:45.839 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.840 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.840 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.840 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.841 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.841 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_duz73YpnwikZnC64B4KGaWm", 2022-09-12T19:16:45.841 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.842 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.842 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.842 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.843 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.843 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.844 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.844 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.635657_297.1", 2022-09-12T19:16:45.844 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.845 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-33", 2022-09-12T19:16:45.845 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.635657460Z", 2022-09-12T19:16:45.845 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.846 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.846 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.846 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.846 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.847 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.847 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_duz73YpnwikZnC64B4KGaWm", 2022-09-12T19:16:45.848 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.848 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.849 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.849 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.849 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.850 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.851 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.851 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.653881_298.1", 2022-09-12T19:16:45.851 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.852 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-34:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.34", 2022-09-12T19:16:45.852 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.653881272Z", 2022-09-12T19:16:45.852 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.852 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.853 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.853 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.853 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.854 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.854 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_T6BXjPTsg5YjV989eew_nUj", 2022-09-12T19:16:45.855 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.855 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.856 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.856 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.856 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.857 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.857 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.656302_299.1", 2022-09-12T19:16:45.857 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.858 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-34:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.34", 2022-09-12T19:16:45.858 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.656302045Z", 2022-09-12T19:16:45.859 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.859 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.859 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.860 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.860 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.861 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.861 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_T6BXjPTsg5YjV989eew_nUj", 2022-09-12T19:16:45.862 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.862 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.863 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.863 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.863 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.864 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.864 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.864 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.865 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.657143_300.1", 2022-09-12T19:16:45.865 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.865 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-34", 2022-09-12T19:16:45.866 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.657143981Z", 2022-09-12T19:16:45.866 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.867 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.867 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.867 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.867 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.868 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.868 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_zJMQJW7qDi486fSD79XXMMX", 2022-09-12T19:16:45.869 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.869 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.870 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.870 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.871 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.871 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.871 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.872 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.872 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.659579_301.1", 2022-09-12T19:16:45.872 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.872 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-34", 2022-09-12T19:16:45.873 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.659579967Z", 2022-09-12T19:16:45.873 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.873 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.874 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.874 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.874 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.874 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.875 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_zJMQJW7qDi486fSD79XXMMX", 2022-09-12T19:16:45.875 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.876 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.876 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.876 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.877 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.877 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.878 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.878 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.879 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.688010_302.1", 2022-09-12T19:16:45.879 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.879 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35", 2022-09-12T19:16:45.880 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.688010386Z", 2022-09-12T19:16:45.880 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.880 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.881 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.881 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.882 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.882 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.882 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_RT_qLkUYTVFuTZ0Mzyw3ra0", 2022-09-12T19:16:45.883 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.883 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.884 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.884 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.885 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.885 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.885 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.886 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.886 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.690504_303.1", 2022-09-12T19:16:45.886 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.887 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35", 2022-09-12T19:16:45.887 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.690504973Z", 2022-09-12T19:16:45.888 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.888 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.889 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.889 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.890 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.890 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.891 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_RT_qLkUYTVFuTZ0Mzyw3ra0", 2022-09-12T19:16:45.891 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.892 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.892 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.893 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.893 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.894 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.894 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.895 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.895 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.691394_304.1", 2022-09-12T19:16:45.896 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.896 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-35", 2022-09-12T19:16:45.897 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.691394342Z", 2022-09-12T19:16:45.897 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.898 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.898 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.899 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.899 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.900 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.900 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_E0ZY7pht4AH9rgoAN9wcr0i", 2022-09-12T19:16:45.901 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.901 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.902 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.902 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.903 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.903 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.904 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.904 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.905 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.693991_305.1", 2022-09-12T19:16:45.905 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.906 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-35", 2022-09-12T19:16:45.906 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.693991405Z", 2022-09-12T19:16:45.906 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.907 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.907 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.908 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.908 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.909 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.909 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_E0ZY7pht4AH9rgoAN9wcr0i", 2022-09-12T19:16:45.910 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.910 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.911 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.911 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.912 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.912 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.913 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.913 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.914 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.713639_306.1", 2022-09-12T19:16:45.914 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.915 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-33:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.33", 2022-09-12T19:16:45.915 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.713639843Z", 2022-09-12T19:16:45.916 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.916 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.917 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_9Dhxqv90pxoXFdD9Kil8AZS", 2022-09-12T19:16:45.917 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.918 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.918 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.919 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_9Dhxqv90pxoXFdD9Kil8AZS", 2022-09-12T19:16:45.919 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.920 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.920 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.921 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.921 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.922 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.922 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.923 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.923 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.716209_307.1", 2022-09-12T19:16:45.924 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.924 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-33:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.33", 2022-09-12T19:16:45.925 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.716209907Z", 2022-09-12T19:16:45.925 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.926 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.926 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_9Dhxqv90pxoXFdD9Kil8AZS", 2022-09-12T19:16:45.927 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.927 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.928 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.928 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_9Dhxqv90pxoXFdD9Kil8AZS", 2022-09-12T19:16:45.929 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.929 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.930 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.930 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.931 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.931 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.932 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.932 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.933 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010025.737428_308.1", 2022-09-12T19:16:45.933 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.934 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-34:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.34", 2022-09-12T19:16:45.934 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:45.737428711Z", 2022-09-12T19:16:45.935 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.935 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.936 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_T6BXjPTsg5YjV989eew_nUj", 2022-09-12T19:16:45.936 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.937 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.937 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.938 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_T6BXjPTsg5YjV989eew_nUj", 2022-09-12T19:16:45.938 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.939 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.939 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.940 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.940 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.941 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.941 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.942 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.942 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010025.741603_309.1", 2022-09-12T19:16:45.943 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.943 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-34:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.34", 2022-09-12T19:16:45.944 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:45.741603314Z", 2022-09-12T19:16:45.944 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.945 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.945 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_T6BXjPTsg5YjV989eew_nUj", 2022-09-12T19:16:45.946 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.946 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.947 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.947 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_T6BXjPTsg5YjV989eew_nUj", 2022-09-12T19:16:45.948 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.948 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.949 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.949 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.950 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.950 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.950 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.951 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.951 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.813360_310.1", 2022-09-12T19:16:45.952 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.952 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35", 2022-09-12T19:16:45.952 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.813360734Z", 2022-09-12T19:16:45.953 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.953 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.954 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_RT_qLkUYTVFuTZ0Mzyw3ra0", 2022-09-12T19:16:45.954 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.955 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.955 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.955 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_RT_qLkUYTVFuTZ0Mzyw3ra0", 2022-09-12T19:16:45.956 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.956 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.957 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.957 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.958 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.958 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.959 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.959 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.960 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.817978_311.1", 2022-09-12T19:16:45.960 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.961 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35", 2022-09-12T19:16:45.961 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.817978446Z", 2022-09-12T19:16:45.961 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.962 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.962 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_RT_qLkUYTVFuTZ0Mzyw3ra0", 2022-09-12T19:16:45.963 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.963 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.964 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.964 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_RT_qLkUYTVFuTZ0Mzyw3ra0", 2022-09-12T19:16:45.965 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:45.965 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.966 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.966 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.967 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.967 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.968 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.968 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.969 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.851414_312.1", 2022-09-12T19:16:45.969 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.970 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36", 2022-09-12T19:16:45.970 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.851414155Z", 2022-09-12T19:16:45.971 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.971 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.971 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.972 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.972 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.973 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.973 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_QWEAbq70_sTavFlIROKha_u", 2022-09-12T19:16:45.974 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.974 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.975 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.975 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.976 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.976 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.977 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.977 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.977 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.860552_313.1", 2022-09-12T19:16:45.978 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:45.979 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36", 2022-09-12T19:16:45.979 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.860552661Z", 2022-09-12T19:16:45.980 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.980 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.981 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.981 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.982 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.982 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.983 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_QWEAbq70_sTavFlIROKha_u", 2022-09-12T19:16:45.983 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.984 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.985 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:45.985 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.986 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.986 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.987 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.987 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.867783_314.1", 2022-09-12T19:16:45.988 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.988 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-36", 2022-09-12T19:16:45.989 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.867783999Z", 2022-09-12T19:16:45.989 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.990 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.990 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.991 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.992 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:45.992 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_-cyyq5stbLWU_c8ISOn28nE", 2022-09-12T19:16:45.993 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:45.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.993 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:45.994 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:45.994 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.995 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:45.995 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:45.996 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:45.996 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.879969_315.1", 2022-09-12T19:16:45.997 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:45.997 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-36", 2022-09-12T19:16:45.997 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.879969477Z", 2022-09-12T19:16:45.998 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:45.998 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:45.999 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:45.999 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:45.999 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.000 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.000 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_-cyyq5stbLWU_c8ISOn28nE", 2022-09-12T19:16:46.000 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.001 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.002 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.002 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.003 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.003 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.004 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.914600_316.1", 2022-09-12T19:16:46.004 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.005 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37", 2022-09-12T19:16:46.005 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.914600273Z", 2022-09-12T19:16:46.005 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.006 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.007 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.007 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.008 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.009 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_BzeIqN5hzm9Zg_oOIhlkjQJ", 2022-09-12T19:16:46.009 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.010 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.010 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.011 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.011 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.011 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.012 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.012 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.917584_317.1", 2022-09-12T19:16:46.013 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.013 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37", 2022-09-12T19:16:46.014 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.917584610Z", 2022-09-12T19:16:46.014 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.014 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.015 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.015 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.016 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.016 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.017 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_BzeIqN5hzm9Zg_oOIhlkjQJ", 2022-09-12T19:16:46.017 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.018 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.018 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.019 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.019 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.019 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.020 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.021 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.918659_318.1", 2022-09-12T19:16:46.021 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.021 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-37", 2022-09-12T19:16:46.022 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.918659379Z", 2022-09-12T19:16:46.022 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.023 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.023 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.024 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.024 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.024 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.024 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_WGVNJNvbu5mulQzf6ElKa4a", 2022-09-12T19:16:46.025 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.025 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.025 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.026 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.026 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.027 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.027 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.028 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.921237_319.1", 2022-09-12T19:16:46.028 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.028 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-37", 2022-09-12T19:16:46.029 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.921237396Z", 2022-09-12T19:16:46.029 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.029 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.030 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.030 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.030 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.031 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.031 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_WGVNJNvbu5mulQzf6ElKa4a", 2022-09-12T19:16:46.031 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.032 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.032 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.033 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.033 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.034 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.946958_320.1", 2022-09-12T19:16:46.034 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.034 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38", 2022-09-12T19:16:46.034 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.946958410Z", 2022-09-12T19:16:46.035 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.035 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.036 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.036 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.036 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.037 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.037 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_IWIU9MxoQv-Gr3d7VAe8c_0", 2022-09-12T19:16:46.037 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.037 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.038 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.038 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.038 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.038 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.039 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.949538_321.1", 2022-09-12T19:16:46.040 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.040 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38", 2022-09-12T19:16:46.040 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.949538394Z", 2022-09-12T19:16:46.040 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.041 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.041 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.041 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.042 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.042 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_IWIU9MxoQv-Gr3d7VAe8c_0", 2022-09-12T19:16:46.042 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.043 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.043 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.043 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.044 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.044 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.950529_322.1", 2022-09-12T19:16:46.045 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.045 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-38", 2022-09-12T19:16:46.045 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.950529882Z", 2022-09-12T19:16:46.045 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.046 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.046 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.046 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.047 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.047 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_QAVKfkG00yr1qxv971lPJ9X", 2022-09-12T19:16:46.048 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.048 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.048 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.049 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.049 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.049 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.050 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.953073_323.1", 2022-09-12T19:16:46.050 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.050 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-38", 2022-09-12T19:16:46.051 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.953073366Z", 2022-09-12T19:16:46.051 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.051 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.051 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.052 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.052 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.053 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_QAVKfkG00yr1qxv971lPJ9X", 2022-09-12T19:16:46.053 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.053 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.054 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.054 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.054 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.055 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010058.708837_324.1", 2022-09-12T19:16:46.056 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.056 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-39:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.39", 2022-09-12T19:16:46.056 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.708837670Z", 2022-09-12T19:16:46.057 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.057 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.057 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.057 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.058 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.058 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.058 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_8tqVsOF3WKb2me5I95G3qgR", 2022-09-12T19:16:46.059 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.059 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.059 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.059 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.060 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.060 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.060 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.061 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010058.713291_325.1", 2022-09-12T19:16:46.061 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.061 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-39:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.39", 2022-09-12T19:16:46.062 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.713291435Z", 2022-09-12T19:16:46.062 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.062 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.062 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.063 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.063 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.064 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_8tqVsOF3WKb2me5I95G3qgR", 2022-09-12T19:16:46.064 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.064 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.065 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.065 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.065 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.066 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010058.715054_326.1", 2022-09-12T19:16:46.066 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.067 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-39", 2022-09-12T19:16:46.067 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.715054633Z", 2022-09-12T19:16:46.067 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.068 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.068 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.068 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.069 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.069 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_xgZt1r-qtjp-K4E_Z1U_XYi", 2022-09-12T19:16:46.069 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.070 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.070 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.070 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.071 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.071 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010058.720459_327.1", 2022-09-12T19:16:46.072 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.072 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-39", 2022-09-12T19:16:46.072 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.720459880Z", 2022-09-12T19:16:46.072 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.073 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.073 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.073 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.074 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.074 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_xgZt1r-qtjp-K4E_Z1U_XYi", 2022-09-12T19:16:46.074 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.075 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.075 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.076 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.076 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.077 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010058.744939_328.1", 2022-09-12T19:16:46.077 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.077 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-40:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.40", 2022-09-12T19:16:46.078 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.744939676Z", 2022-09-12T19:16:46.078 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.078 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.078 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.079 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.079 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.079 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_XRz8ZhFtWNauZ5YqelhcHjG", 2022-09-12T19:16:46.080 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.080 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.081 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.081 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.081 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.082 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010058.753933_329.1", 2022-09-12T19:16:46.082 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.083 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-40:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.40", 2022-09-12T19:16:46.083 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.753933965Z", 2022-09-12T19:16:46.083 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.083 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.084 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.084 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.085 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.085 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_XRz8ZhFtWNauZ5YqelhcHjG", 2022-09-12T19:16:46.085 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.086 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.086 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.086 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.087 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.087 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010058.758572_330.1", 2022-09-12T19:16:46.088 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.088 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-40", 2022-09-12T19:16:46.088 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.758572726Z", 2022-09-12T19:16:46.088 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.089 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.089 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.089 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.090 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.090 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_4IJ7vbCVAqQL9xJtZhDAZ4r", 2022-09-12T19:16:46.091 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.091 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.091 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.092 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.092 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.093 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010058.761625_331.1", 2022-09-12T19:16:46.093 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.093 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-40", 2022-09-12T19:16:46.093 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.761625471Z", 2022-09-12T19:16:46.094 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.094 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.094 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.094 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.095 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.095 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_4IJ7vbCVAqQL9xJtZhDAZ4r", 2022-09-12T19:16:46.095 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.096 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.096 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.097 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.097 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.098 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010058.795251_332.1", 2022-09-12T19:16:46.098 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.098 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-41:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.41", 2022-09-12T19:16:46.098 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.795251326Z", 2022-09-12T19:16:46.099 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.099 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.099 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.099 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.100 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.100 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_WLs8_gJqx8-eJNrUvhbOESF", 2022-09-12T19:16:46.101 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.102 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.102 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.102 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.102 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.103 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010058.799161_333.1", 2022-09-12T19:16:46.104 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.104 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-41:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.41", 2022-09-12T19:16:46.104 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.799161992Z", 2022-09-12T19:16:46.104 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.105 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.105 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.105 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.106 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.106 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_WLs8_gJqx8-eJNrUvhbOESF", 2022-09-12T19:16:46.106 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.107 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.107 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.108 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.108 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.109 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010058.800547_334.1", 2022-09-12T19:16:46.109 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.109 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-41", 2022-09-12T19:16:46.109 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.800547915Z", 2022-09-12T19:16:46.110 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.110 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.110 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.111 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.111 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.111 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Rx20gRuO3XO_dd_rDNjdCZL", 2022-09-12T19:16:46.112 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.112 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.112 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.113 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.113 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.114 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010058.804299_335.1", 2022-09-12T19:16:46.114 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.114 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-41", 2022-09-12T19:16:46.115 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.804299896Z", 2022-09-12T19:16:46.115 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.115 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.115 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.116 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.116 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.116 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Rx20gRuO3XO_dd_rDNjdCZL", 2022-09-12T19:16:46.117 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.117 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.118 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.118 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.118 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.119 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008569.937424_38.1", 2022-09-12T19:16:46.119 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.119 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-1:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.1", 2022-09-12T19:16:46.120 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:29.937424694Z", 2022-09-12T19:16:46.120 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.120 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.121 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.121 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.122 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.122 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "__lpWFIiN4hfllNCufyv6oPF", 2022-09-12T19:16:46.122 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.123 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.123 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.123 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.124 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.124 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.124 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.124 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008569.948669_40.1", 2022-09-12T19:16:46.125 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.125 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-1:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.1", 2022-09-12T19:16:46.125 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:29.948669068Z", 2022-09-12T19:16:46.125 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.126 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.126 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.126 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.127 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.127 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "__lpWFIiN4hfllNCufyv6oPF", 2022-09-12T19:16:46.127 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.128 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.128 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.128 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.129 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.129 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.129 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.129 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008569.949794_41.1", 2022-09-12T19:16:46.130 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.130 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-1", 2022-09-12T19:16:46.130 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:29.949794341Z", 2022-09-12T19:16:46.130 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.131 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.131 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.131 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.132 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.132 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_724Zi_oQbv20Wa_KFzslGam", 2022-09-12T19:16:46.132 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.133 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.133 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.133 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.134 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.134 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008569.952619_42.1", 2022-09-12T19:16:46.135 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.135 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-1", 2022-09-12T19:16:46.135 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:29.952619357Z", 2022-09-12T19:16:46.136 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.136 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.136 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.136 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.137 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.137 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.137 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_724Zi_oQbv20Wa_KFzslGam", 2022-09-12T19:16:46.137 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.138 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.138 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.138 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.138 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.139 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.139 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.139 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.139 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008569.979576_43.1", 2022-09-12T19:16:46.140 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.140 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-2:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.2", 2022-09-12T19:16:46.140 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:29.979576716Z", 2022-09-12T19:16:46.141 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.141 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.141 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.142 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.142 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.142 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.143 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_yWZcOeCLp3AOZWE4KEf6DqY", 2022-09-12T19:16:46.143 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.143 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.144 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.144 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.144 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.144 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.145 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.145 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.145 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008569.982366_44.1", 2022-09-12T19:16:46.146 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.146 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-2:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.2", 2022-09-12T19:16:46.146 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:29.982366892Z", 2022-09-12T19:16:46.146 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.147 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.147 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.147 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.148 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.148 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_yWZcOeCLp3AOZWE4KEf6DqY", 2022-09-12T19:16:46.148 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.149 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.149 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.149 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.149 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.150 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.150 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008569.983351_45.1", 2022-09-12T19:16:46.151 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.151 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-2", 2022-09-12T19:16:46.151 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:29.983351998Z", 2022-09-12T19:16:46.152 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.152 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.152 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.152 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.153 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.154 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_JYImBEwDt-Z7UabuOPjifZb", 2022-09-12T19:16:46.154 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.155 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.156 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.156 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.157 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.158 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.158 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008569.985969_46.1", 2022-09-12T19:16:46.159 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.159 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-2", 2022-09-12T19:16:46.160 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:29.985969116Z", 2022-09-12T19:16:46.160 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.161 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.161 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.162 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.163 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.163 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_JYImBEwDt-Z7UabuOPjifZb", 2022-09-12T19:16:46.164 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.165 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.165 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.166 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.166 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.167 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.167 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.168 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008572.534688_47.1", 2022-09-12T19:16:46.168 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.169 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-3:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.3", 2022-09-12T19:16:46.169 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:32.534688656Z", 2022-09-12T19:16:46.170 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.170 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.171 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.171 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.172 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.172 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.173 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_2N-a-PDWYhR0LHw5JLpZR4c", 2022-09-12T19:16:46.173 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.174 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.175 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.175 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.176 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.176 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.177 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.177 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.178 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008572.537960_48.1", 2022-09-12T19:16:46.178 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.179 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-3:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.3", 2022-09-12T19:16:46.179 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:32.537960225Z", 2022-09-12T19:16:46.180 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.180 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.181 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.181 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.182 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.182 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.183 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_2N-a-PDWYhR0LHw5JLpZR4c", 2022-09-12T19:16:46.183 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.184 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.184 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.185 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.185 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.186 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.186 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.187 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.187 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008572.539055_49.1", 2022-09-12T19:16:46.188 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.188 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-3", 2022-09-12T19:16:46.189 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:32.539055121Z", 2022-09-12T19:16:46.189 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.190 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.190 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.191 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.191 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.192 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.192 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_84xzHNW5sbis6XdPCoJHPOk", 2022-09-12T19:16:46.192 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.193 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.194 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.194 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.194 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.195 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.195 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.196 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.196 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008572.542081_50.1", 2022-09-12T19:16:46.197 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.197 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-3", 2022-09-12T19:16:46.198 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:32.542081201Z", 2022-09-12T19:16:46.198 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.199 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.199 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.200 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.200 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.201 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.201 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_84xzHNW5sbis6XdPCoJHPOk", 2022-09-12T19:16:46.202 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.202 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.203 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.203 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.204 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.204 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.205 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.205 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.206 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008578.327919_51.1", 2022-09-12T19:16:46.206 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.207 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-4:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.4", 2022-09-12T19:16:46.207 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.327919700Z", 2022-09-12T19:16:46.208 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.208 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.209 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.209 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.210 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.210 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.211 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_k7Q_LlQwzVHeoxMKmCvxjHo", 2022-09-12T19:16:46.211 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.212 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.212 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.213 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.213 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.214 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.215 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.215 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.216 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008578.330590_52.1", 2022-09-12T19:16:46.216 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.217 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-4:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.4", 2022-09-12T19:16:46.217 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.330590808Z", 2022-09-12T19:16:46.218 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.218 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.219 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.219 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.220 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.220 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.220 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_k7Q_LlQwzVHeoxMKmCvxjHo", 2022-09-12T19:16:46.221 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.221 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.221 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.221 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.222 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.222 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.222 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.222 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.223 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008578.331599_53.1", 2022-09-12T19:16:46.223 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.223 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-4", 2022-09-12T19:16:46.224 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.331599255Z", 2022-09-12T19:16:46.224 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.224 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.224 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.225 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.225 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.225 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.225 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Uwy2BFzmIVF9oQbfNuX8-c2", 2022-09-12T19:16:46.226 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.226 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.226 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.226 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.227 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.227 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.227 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.227 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.228 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008578.334356_54.1", 2022-09-12T19:16:46.228 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.228 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-4", 2022-09-12T19:16:46.229 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.334356674Z", 2022-09-12T19:16:46.229 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.229 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.229 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.230 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.230 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.230 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.230 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Uwy2BFzmIVF9oQbfNuX8-c2", 2022-09-12T19:16:46.231 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.231 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.231 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.232 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.232 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.233 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.233 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.233 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.233 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008578.354410_55.1", 2022-09-12T19:16:46.234 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.234 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-5:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.5", 2022-09-12T19:16:46.234 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.354410340Z", 2022-09-12T19:16:46.235 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.235 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.235 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.235 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.236 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.236 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.237 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_DOCacF6mLBz24cosgaizYjM", 2022-09-12T19:16:46.237 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.237 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.237 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.238 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.238 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.238 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.239 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.239 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008578.357067_56.1", 2022-09-12T19:16:46.239 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.240 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-5:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.5", 2022-09-12T19:16:46.240 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.357067384Z", 2022-09-12T19:16:46.240 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.241 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.241 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.241 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.241 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.242 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.242 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_DOCacF6mLBz24cosgaizYjM", 2022-09-12T19:16:46.242 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.242 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.243 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.243 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.243 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.243 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.244 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.244 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008578.357978_57.1", 2022-09-12T19:16:46.245 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.245 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-5", 2022-09-12T19:16:46.245 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.357978793Z", 2022-09-12T19:16:46.246 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.246 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.246 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.247 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.247 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.247 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.247 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_PK2zvS8BU3_kRxhnsNsDeRa", 2022-09-12T19:16:46.248 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.248 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.248 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.249 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.249 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.249 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.249 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.250 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.250 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008578.360592_58.1", 2022-09-12T19:16:46.250 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.251 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-5", 2022-09-12T19:16:46.251 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.360592786Z", 2022-09-12T19:16:46.251 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.251 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.252 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.252 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.252 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.252 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.253 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_PK2zvS8BU3_kRxhnsNsDeRa", 2022-09-12T19:16:46.253 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.253 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.254 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.254 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.255 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.255 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.255 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.256 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008578.393761_59.1", 2022-09-12T19:16:46.256 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.256 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-6:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.6", 2022-09-12T19:16:46.256 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.393761186Z", 2022-09-12T19:16:46.257 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.257 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.257 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.257 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.258 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.258 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.258 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_zr7R-yTI4PYqVN0hq56_Jgr", 2022-09-12T19:16:46.258 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.259 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.259 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.259 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.260 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.260 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.260 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.261 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008578.396555_60.1", 2022-09-12T19:16:46.261 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.261 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-6:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.6", 2022-09-12T19:16:46.261 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.396555557Z", 2022-09-12T19:16:46.262 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.262 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.262 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.263 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.263 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.263 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.263 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_zr7R-yTI4PYqVN0hq56_Jgr", 2022-09-12T19:16:46.264 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.264 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.264 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.265 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.265 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.265 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.265 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.266 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008578.397479_61.1", 2022-09-12T19:16:46.266 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.266 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-6", 2022-09-12T19:16:46.267 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.397479637Z", 2022-09-12T19:16:46.267 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.267 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.267 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.268 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.268 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.268 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.268 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_9moMpukFqONyJQ22546lFFJ", 2022-09-12T19:16:46.269 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.269 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.269 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.270 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.270 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.270 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.270 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.271 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.271 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008578.400100_62.1", 2022-09-12T19:16:46.271 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.272 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-6", 2022-09-12T19:16:46.272 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.400100117Z", 2022-09-12T19:16:46.272 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.273 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.273 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.273 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.273 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.274 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.274 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_9moMpukFqONyJQ22546lFFJ", 2022-09-12T19:16:46.274 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.275 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.275 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.275 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.275 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.276 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.276 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.276 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.276 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.041323_63.1", 2022-09-12T19:16:46.277 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.277 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-7:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.7", 2022-09-12T19:16:46.277 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.041323973Z", 2022-09-12T19:16:46.277 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.278 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.278 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.278 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.279 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.279 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.279 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_14zIMAUeR0_uC-IGvsiZyFE", 2022-09-12T19:16:46.280 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.280 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.280 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.281 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.281 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.281 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.281 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.282 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.282 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.043790_64.1", 2022-09-12T19:16:46.282 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.283 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-7:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.7", 2022-09-12T19:16:46.283 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.043790091Z", 2022-09-12T19:16:46.283 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.284 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.284 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.284 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.284 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.285 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.285 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_14zIMAUeR0_uC-IGvsiZyFE", 2022-09-12T19:16:46.285 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.286 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.286 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.286 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.287 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.287 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.287 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.044741_65.1", 2022-09-12T19:16:46.288 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.288 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-7", 2022-09-12T19:16:46.288 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.044741403Z", 2022-09-12T19:16:46.289 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.289 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.289 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.289 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.290 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.290 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_40hjwX5ISF-woQ9EMT_esot", 2022-09-12T19:16:46.291 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.291 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.291 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.291 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.292 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.292 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.292 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.292 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.293 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.047046_66.1", 2022-09-12T19:16:46.293 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.293 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-7", 2022-09-12T19:16:46.293 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.047046794Z", 2022-09-12T19:16:46.294 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.294 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.294 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.294 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.295 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.295 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.295 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_40hjwX5ISF-woQ9EMT_esot", 2022-09-12T19:16:46.296 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.296 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.296 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.296 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.297 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.297 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.297 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.297 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.298 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.068972_67.1", 2022-09-12T19:16:46.298 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.298 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-8:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.8", 2022-09-12T19:16:46.299 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.068972049Z", 2022-09-12T19:16:46.299 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.299 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.299 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.300 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.300 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.300 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.300 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Cbx4jHMPdXce-GRW3F5bG0e", 2022-09-12T19:16:46.301 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.301 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.301 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.301 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.302 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.302 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.302 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.303 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.303 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.071464_68.1", 2022-09-12T19:16:46.303 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.303 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-8:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.8", 2022-09-12T19:16:46.304 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.071464065Z", 2022-09-12T19:16:46.304 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.304 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.305 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.305 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.305 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.305 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.307 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Cbx4jHMPdXce-GRW3F5bG0e", 2022-09-12T19:16:46.307 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.308 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.308 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.308 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.308 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.309 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.309 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.309 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.309 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.072340_69.1", 2022-09-12T19:16:46.310 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.310 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-8", 2022-09-12T19:16:46.310 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.072340420Z", 2022-09-12T19:16:46.311 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.311 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.311 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.311 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.312 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.312 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.312 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_mshfSar0kbJ0_j03QlyLl5l", 2022-09-12T19:16:46.313 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.313 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.313 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.313 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.314 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.314 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.314 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.314 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.315 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.074696_70.1", 2022-09-12T19:16:46.315 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.315 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-8", 2022-09-12T19:16:46.316 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.074696150Z", 2022-09-12T19:16:46.316 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.316 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.316 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.317 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.317 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.317 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.317 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_mshfSar0kbJ0_j03QlyLl5l", 2022-09-12T19:16:46.318 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.318 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.318 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.319 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.319 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.319 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.319 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.320 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.320 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.106134_71.1", 2022-09-12T19:16:46.320 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.320 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9", 2022-09-12T19:16:46.321 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.106134737Z", 2022-09-12T19:16:46.321 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.321 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.322 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.322 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.322 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.322 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.323 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_F8AkuhSt1lRvPdknmmo8vhx", 2022-09-12T19:16:46.323 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.323 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.323 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.324 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.324 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.324 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.325 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.325 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.325 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.110078_72.1", 2022-09-12T19:16:46.325 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.326 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9", 2022-09-12T19:16:46.326 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.110078653Z", 2022-09-12T19:16:46.326 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.327 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.327 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.327 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.327 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.328 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.328 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_F8AkuhSt1lRvPdknmmo8vhx", 2022-09-12T19:16:46.328 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.329 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.329 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.329 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.329 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.330 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.330 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.330 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.331 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.111034_73.1", 2022-09-12T19:16:46.331 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.331 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-9", 2022-09-12T19:16:46.331 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.111034634Z", 2022-09-12T19:16:46.332 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.332 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.332 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.332 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.333 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.333 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.333 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_dnyYyWYZArJtIne5MIt7vUD", 2022-09-12T19:16:46.334 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.334 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.334 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.334 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.335 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.335 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.335 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.335 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.336 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.116857_74.1", 2022-09-12T19:16:46.336 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.336 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-9", 2022-09-12T19:16:46.337 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.116857200Z", 2022-09-12T19:16:46.337 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.337 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.337 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:46.338 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:46.338 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.338 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.338 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_dnyYyWYZArJtIne5MIt7vUD", 2022-09-12T19:16:46.339 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.339 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.339 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.340 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.340 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.340 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.341 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.768561_75.1", 2022-09-12T19:16:46.341 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.342 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-7:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.7", 2022-09-12T19:16:46.342 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.768561107Z", 2022-09-12T19:16:46.342 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.342 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.343 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_14zIMAUeR0_uC-IGvsiZyFE", 2022-09-12T19:16:46.343 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.343 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.343 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.344 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_14zIMAUeR0_uC-IGvsiZyFE", 2022-09-12T19:16:46.344 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:46.344 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.344 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.345 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.345 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.345 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.346 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.346 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.346 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.771274_76.1", 2022-09-12T19:16:46.346 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.347 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-7:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.7", 2022-09-12T19:16:46.347 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.771274788Z", 2022-09-12T19:16:46.347 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.347 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.348 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_14zIMAUeR0_uC-IGvsiZyFE", 2022-09-12T19:16:46.348 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.348 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.349 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.349 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_14zIMAUeR0_uC-IGvsiZyFE", 2022-09-12T19:16:46.349 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:46.349 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.350 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.350 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.350 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.350 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.351 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.351 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.351 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.783884_77.1", 2022-09-12T19:16:46.352 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.352 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-7", 2022-09-12T19:16:46.352 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.783884039Z", 2022-09-12T19:16:46.352 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.353 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.353 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_40hjwX5ISF-woQ9EMT_esot", 2022-09-12T19:16:46.353 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.354 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.354 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.354 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_40hjwX5ISF-woQ9EMT_esot", 2022-09-12T19:16:46.354 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:46.355 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.355 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.355 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "remove" 2022-09-12T19:16:46.355 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.356 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.356 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.356 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.357 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.786242_78.1", 2022-09-12T19:16:46.357 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.357 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-7", 2022-09-12T19:16:46.357 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.786242763Z", 2022-09-12T19:16:46.358 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.358 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.358 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_40hjwX5ISF-woQ9EMT_esot", 2022-09-12T19:16:46.358 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.359 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.359 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.359 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_40hjwX5ISF-woQ9EMT_esot", 2022-09-12T19:16:46.360 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:46.360 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.360 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.360 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.361 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.361 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.361 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.361 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.362 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.817097_79.1", 2022-09-12T19:16:46.362 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.362 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-8:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.8", 2022-09-12T19:16:46.363 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.817097161Z", 2022-09-12T19:16:46.363 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.363 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.364 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Cbx4jHMPdXce-GRW3F5bG0e", 2022-09-12T19:16:46.364 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.364 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.364 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.365 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Cbx4jHMPdXce-GRW3F5bG0e", 2022-09-12T19:16:46.365 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:46.365 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.366 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.366 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.366 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.366 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.367 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.367 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.367 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.821207_80.1", 2022-09-12T19:16:46.368 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.368 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-8:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.8", 2022-09-12T19:16:46.368 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.821207606Z", 2022-09-12T19:16:46.368 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.369 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.369 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Cbx4jHMPdXce-GRW3F5bG0e", 2022-09-12T19:16:46.369 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.370 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.370 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Cbx4jHMPdXce-GRW3F5bG0e", 2022-09-12T19:16:46.370 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:46.371 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.371 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.371 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.371 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.372 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.372 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.372 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.372 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.833382_81.1", 2022-09-12T19:16:46.373 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.373 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-8", 2022-09-12T19:16:46.373 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.833382871Z", 2022-09-12T19:16:46.374 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.374 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.374 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_mshfSar0kbJ0_j03QlyLl5l", 2022-09-12T19:16:46.374 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.375 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.375 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.375 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_mshfSar0kbJ0_j03QlyLl5l", 2022-09-12T19:16:46.376 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:46.376 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.376 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.376 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "remove" 2022-09-12T19:16:46.377 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.377 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.378 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.835519_82.1", 2022-09-12T19:16:46.378 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.378 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-8", 2022-09-12T19:16:46.379 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.835519055Z", 2022-09-12T19:16:46.379 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.379 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.379 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_mshfSar0kbJ0_j03QlyLl5l", 2022-09-12T19:16:46.380 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.380 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.380 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_mshfSar0kbJ0_j03QlyLl5l", 2022-09-12T19:16:46.381 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:46.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.381 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.381 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.382 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.382 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.382 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.383 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.383 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.890492_83.1", 2022-09-12T19:16:46.383 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.383 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9", 2022-09-12T19:16:46.384 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.890492647Z", 2022-09-12T19:16:46.384 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.384 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.384 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_F8AkuhSt1lRvPdknmmo8vhx", 2022-09-12T19:16:46.385 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.385 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.385 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.386 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_F8AkuhSt1lRvPdknmmo8vhx", 2022-09-12T19:16:46.386 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:46.386 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.386 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.387 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:46.387 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.387 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.387 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.388 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.388 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.893091_84.1", 2022-09-12T19:16:46.388 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.389 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9", 2022-09-12T19:16:46.389 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.893091078Z", 2022-09-12T19:16:46.389 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.389 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.390 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_F8AkuhSt1lRvPdknmmo8vhx", 2022-09-12T19:16:46.390 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.390 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.390 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.391 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_F8AkuhSt1lRvPdknmmo8vhx", 2022-09-12T19:16:46.391 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:46.391 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.391 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.392 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.392 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.392 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.392 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.393 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.393 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.901637_85.1", 2022-09-12T19:16:46.393 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.394 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-9", 2022-09-12T19:16:46.394 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.901637143Z", 2022-09-12T19:16:46.394 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.394 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.395 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_dnyYyWYZArJtIne5MIt7vUD", 2022-09-12T19:16:46.395 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.395 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.395 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.396 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_dnyYyWYZArJtIne5MIt7vUD", 2022-09-12T19:16:46.396 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:46.396 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.396 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.397 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "remove" 2022-09-12T19:16:46.397 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.399 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.399 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.399 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:46.400 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.904048_86.1", 2022-09-12T19:16:46.400 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:46.400 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-9", 2022-09-12T19:16:46.400 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048979Z", 2022-09-12T19:16:46.401 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:46.401 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:46.401 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_dnyYyWYZArJtIne5MIt7vUD", 2022-09-12T19:16:46.402 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:46.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.402 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:46.402 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_dnyYyWYZArJtIne5MIt7vUD", 2022-09-12T19:16:46.403 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:46.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:46.403 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:46.403 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:46.404 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.404 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.404 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:46.405 INFO:teuthology.orchestra.run.smithi138.stdout:] 2022-09-12T19:16:46.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'c0f7a512-c940-4d1f-8adf-3484630f876b', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:46.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'c0f7a512-c940-4d1f-8adf-3484630f876b', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:46.408 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period c0f7a512-c940-4d1f-8adf-3484630f876b --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:46.465 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:46.466 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:46.516 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:16:46.517 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.517 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008447.969948_52.1", 2022-09-12T19:16:46.517 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "user", 2022-09-12T19:16:46.518 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "rgw-multisite-test-user", 2022-09-12T19:16:46.518 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.969948859Z", 2022-09-12T19:16:46.519 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.519 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.520 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.520 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.521 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.521 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.521 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_X5YlnCJyC_iIiZ1qmhasQLd", 2022-09-12T19:16:46.522 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.522 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.523 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.524 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.524 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.524 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.525 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.526 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.526 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008447.976965_53.1", 2022-09-12T19:16:46.526 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "user", 2022-09-12T19:16:46.527 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "rgw-multisite-test-user", 2022-09-12T19:16:46.527 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965355Z", 2022-09-12T19:16:46.528 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.528 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.528 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.529 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.529 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.529 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.530 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_X5YlnCJyC_iIiZ1qmhasQLd", 2022-09-12T19:16:46.530 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.530 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.530 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.531 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.531 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.531 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.531 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.532 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.532 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008586.711103_57.1", 2022-09-12T19:16:46.532 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.533 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10", 2022-09-12T19:16:46.533 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:46.711103783Z", 2022-09-12T19:16:46.533 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.533 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.534 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.534 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.534 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.534 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.535 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_0WIw9i-PdtM0sq6xV05tENT", 2022-09-12T19:16:46.535 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.535 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.536 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.536 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.536 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.537 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.537 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.537 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.537 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008586.714487_58.1", 2022-09-12T19:16:46.538 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.538 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10", 2022-09-12T19:16:46.538 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:46.714487285Z", 2022-09-12T19:16:46.539 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.539 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.539 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.539 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.540 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.540 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.540 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_0WIw9i-PdtM0sq6xV05tENT", 2022-09-12T19:16:46.540 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.541 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.541 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.541 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.541 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.542 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.542 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.542 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.543 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008586.715813_59.1", 2022-09-12T19:16:46.543 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.543 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-10", 2022-09-12T19:16:46.543 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:46.715813657Z", 2022-09-12T19:16:46.544 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.544 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.544 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.544 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.545 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.545 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.545 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_nPF7oiiG3E2RTyZZOeDLhOv", 2022-09-12T19:16:46.546 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.546 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.546 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.547 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.547 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.547 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.548 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.548 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.548 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008586.718771_60.1", 2022-09-12T19:16:46.549 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.550 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-10", 2022-09-12T19:16:46.550 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:46.718771977Z", 2022-09-12T19:16:46.550 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.551 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.551 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.551 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.552 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.552 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.552 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_nPF7oiiG3E2RTyZZOeDLhOv", 2022-09-12T19:16:46.552 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.553 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.553 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.553 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.553 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.554 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.554 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.554 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.554 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008586.736648_61.1", 2022-09-12T19:16:46.555 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.555 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11", 2022-09-12T19:16:46.555 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:46.736648541Z", 2022-09-12T19:16:46.556 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.556 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.556 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.556 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.557 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.557 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.557 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_DXlcSXme1LlpozfB5A2QmS2", 2022-09-12T19:16:46.557 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.558 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.558 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.558 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.559 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.559 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.559 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.559 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.560 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008586.739749_62.1", 2022-09-12T19:16:46.560 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.560 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11", 2022-09-12T19:16:46.560 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:46.739749085Z", 2022-09-12T19:16:46.561 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.561 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.561 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.561 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.562 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.562 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.562 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_DXlcSXme1LlpozfB5A2QmS2", 2022-09-12T19:16:46.563 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.563 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.563 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.563 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.564 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.564 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.564 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.564 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.565 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008586.740926_63.1", 2022-09-12T19:16:46.565 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.565 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-11", 2022-09-12T19:16:46.565 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:46.740926183Z", 2022-09-12T19:16:46.566 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.566 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.567 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.567 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.567 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.567 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.568 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_QTYYWzCEPLTd7eDR1Si_D1k", 2022-09-12T19:16:46.568 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.568 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.568 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.569 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.569 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.569 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.569 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.570 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.570 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008586.744033_64.1", 2022-09-12T19:16:46.570 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.571 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-11", 2022-09-12T19:16:46.571 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:46.744033830Z", 2022-09-12T19:16:46.571 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.571 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.572 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.572 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.572 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.572 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.573 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_QTYYWzCEPLTd7eDR1Si_D1k", 2022-09-12T19:16:46.573 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.573 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.573 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.574 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.574 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.575 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.575 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.575 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.575 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008586.773460_65.1", 2022-09-12T19:16:46.576 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.576 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12", 2022-09-12T19:16:46.576 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:46.773460810Z", 2022-09-12T19:16:46.576 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.577 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.577 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.577 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.577 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.577 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.578 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_jMUPDDQJMus4ft3ku4UXAZe", 2022-09-12T19:16:46.578 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.578 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.579 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.579 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.579 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.579 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.580 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.580 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.580 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008586.776584_66.1", 2022-09-12T19:16:46.580 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.580 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12", 2022-09-12T19:16:46.581 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:46.776584217Z", 2022-09-12T19:16:46.581 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.581 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.581 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.582 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.582 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.583 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.583 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_jMUPDDQJMus4ft3ku4UXAZe", 2022-09-12T19:16:46.583 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.583 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.584 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.584 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.584 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.584 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.585 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.585 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.585 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008586.777646_67.1", 2022-09-12T19:16:46.585 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.586 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-12", 2022-09-12T19:16:46.586 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:46.777646572Z", 2022-09-12T19:16:46.586 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.586 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.587 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.587 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.587 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.587 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.588 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_dds05uMalANntddWdTKqimb", 2022-09-12T19:16:46.588 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.588 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.588 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.589 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.589 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.589 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.589 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.590 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.590 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008586.780479_68.1", 2022-09-12T19:16:46.590 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.590 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-12", 2022-09-12T19:16:46.591 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:46.780479129Z", 2022-09-12T19:16:46.591 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.591 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.592 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.592 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.592 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.592 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.593 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_dds05uMalANntddWdTKqimb", 2022-09-12T19:16:46.593 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.593 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.593 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.594 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.594 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.594 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.594 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.595 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.595 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008900.617421_102.1", 2022-09-12T19:16:46.595 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.595 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13", 2022-09-12T19:16:46.595 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:55:00.617421183Z", 2022-09-12T19:16:46.596 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.596 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.596 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.597 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.597 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.597 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.597 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_LDyblCFwL38h9zUN-Es9_0k", 2022-09-12T19:16:46.597 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.598 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.598 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.598 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.599 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.599 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.599 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.600 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.600 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008900.621771_103.1", 2022-09-12T19:16:46.600 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.600 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13", 2022-09-12T19:16:46.601 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:55:00.621771922Z", 2022-09-12T19:16:46.601 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.601 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.601 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.602 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.602 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.602 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.602 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_LDyblCFwL38h9zUN-Es9_0k", 2022-09-12T19:16:46.603 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.603 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.603 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.603 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.604 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.604 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.604 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.604 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.605 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008900.623473_104.1", 2022-09-12T19:16:46.605 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.605 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-13", 2022-09-12T19:16:46.606 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:55:00.623473623Z", 2022-09-12T19:16:46.606 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.606 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.606 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.607 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.607 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.607 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.608 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_HUc36aHifT_7iFz2OXdLjDc", 2022-09-12T19:16:46.608 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.608 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.609 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.609 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.609 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.609 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.610 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.610 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.610 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008900.629061_105.1", 2022-09-12T19:16:46.610 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.611 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-13", 2022-09-12T19:16:46.611 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:55:00.629061119Z", 2022-09-12T19:16:46.611 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.612 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.612 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.612 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.612 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.613 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.613 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_HUc36aHifT_7iFz2OXdLjDc", 2022-09-12T19:16:46.613 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.613 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.614 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.614 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.614 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.614 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.615 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.615 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.615 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008900.663664_106.1", 2022-09-12T19:16:46.616 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.616 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14", 2022-09-12T19:16:46.616 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:55:00.663664982Z", 2022-09-12T19:16:46.617 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.617 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.617 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.618 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.618 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.618 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.618 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_ULxp_YwqTEGv8QDweh9UlvR", 2022-09-12T19:16:46.619 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.619 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.619 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.619 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.620 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.620 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.620 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.621 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.621 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008900.667714_107.1", 2022-09-12T19:16:46.621 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.621 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14", 2022-09-12T19:16:46.622 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:55:00.667714167Z", 2022-09-12T19:16:46.622 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.622 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.623 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.623 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.623 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.623 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.624 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_ULxp_YwqTEGv8QDweh9UlvR", 2022-09-12T19:16:46.625 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.625 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.625 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.625 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.626 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.626 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.626 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.626 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.627 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008900.669743_108.1", 2022-09-12T19:16:46.627 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.627 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-14", 2022-09-12T19:16:46.627 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:55:00.669743466Z", 2022-09-12T19:16:46.628 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.628 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.628 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.628 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.629 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.629 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.629 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Zohm3vyJdVty1-BsovuZx7M", 2022-09-12T19:16:46.629 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.630 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.630 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.630 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.631 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.631 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.631 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.631 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.632 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008900.675413_109.1", 2022-09-12T19:16:46.632 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.632 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-14", 2022-09-12T19:16:46.633 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:55:00.675413478Z", 2022-09-12T19:16:46.633 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.633 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.633 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.634 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.634 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.634 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.635 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Zohm3vyJdVty1-BsovuZx7M", 2022-09-12T19:16:46.635 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.635 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.635 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.636 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.636 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.636 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.636 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.637 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.637 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008900.713502_110.1", 2022-09-12T19:16:46.637 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.637 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15", 2022-09-12T19:16:46.638 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:55:00.713502192Z", 2022-09-12T19:16:46.638 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.638 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.638 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.639 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.639 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.640 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.641 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_AKH72TwSsQVrOOdb7zgDFt1", 2022-09-12T19:16:46.641 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.642 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.642 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.643 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.643 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.644 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.644 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.645 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.645 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008900.716530_111.1", 2022-09-12T19:16:46.645 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.646 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15", 2022-09-12T19:16:46.646 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:55:00.716530511Z", 2022-09-12T19:16:46.647 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.647 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.648 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.648 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.649 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.649 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.650 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_AKH72TwSsQVrOOdb7zgDFt1", 2022-09-12T19:16:46.650 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.651 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.651 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.652 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.652 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.652 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.652 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.653 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.653 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008900.718433_112.1", 2022-09-12T19:16:46.653 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.653 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-15", 2022-09-12T19:16:46.654 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:55:00.718433663Z", 2022-09-12T19:16:46.654 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.654 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.655 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.655 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.655 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.655 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.656 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_i61YjmwdeL9p7lC_EIZUJTo", 2022-09-12T19:16:46.656 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.656 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.657 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.657 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.658 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.658 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.658 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.658 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.660 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008900.721341_113.1", 2022-09-12T19:16:46.660 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.660 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-15", 2022-09-12T19:16:46.661 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:55:00.721341013Z", 2022-09-12T19:16:46.661 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.661 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.662 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.662 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.662 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.662 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.663 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_i61YjmwdeL9p7lC_EIZUJTo", 2022-09-12T19:16:46.663 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.664 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.664 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.664 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.665 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.665 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.666 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.666 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.667 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.182484_132.1", 2022-09-12T19:16:46.667 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.668 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16", 2022-09-12T19:16:46.668 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.182484917Z", 2022-09-12T19:16:46.669 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.669 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.670 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.670 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.671 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.671 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.671 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_V99BgSq-hvAYBvkex60C8n2", 2022-09-12T19:16:46.672 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.672 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.673 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.673 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.674 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.675 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.675 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.675 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.676 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.186031_133.1", 2022-09-12T19:16:46.677 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.677 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16", 2022-09-12T19:16:46.678 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.186031455Z", 2022-09-12T19:16:46.678 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.678 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.679 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.680 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.680 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.680 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.681 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_V99BgSq-hvAYBvkex60C8n2", 2022-09-12T19:16:46.681 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.681 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.682 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.682 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.683 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.683 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.683 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.683 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.684 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.187394_134.1", 2022-09-12T19:16:46.684 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.684 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-16", 2022-09-12T19:16:46.684 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.187394124Z", 2022-09-12T19:16:46.685 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.685 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.685 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.685 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.686 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.686 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.686 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_ehigIfW992JSI46zHL04P5J", 2022-09-12T19:16:46.686 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.687 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.687 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.687 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.687 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.688 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.688 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.688 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.688 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.193049_135.1", 2022-09-12T19:16:46.689 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.689 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-16", 2022-09-12T19:16:46.689 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.193049111Z", 2022-09-12T19:16:46.689 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.690 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.690 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.690 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.691 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.691 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.691 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_ehigIfW992JSI46zHL04P5J", 2022-09-12T19:16:46.691 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.692 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.692 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.692 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.692 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.693 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.693 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.693 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.693 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.223026_136.1", 2022-09-12T19:16:46.694 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.694 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17", 2022-09-12T19:16:46.694 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.223026973Z", 2022-09-12T19:16:46.694 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.695 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.695 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.695 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.695 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.696 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.696 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_BCZEMlNNfcs_zua_I4p9TaW", 2022-09-12T19:16:46.696 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.696 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.697 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.697 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.697 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.697 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.698 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.698 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.698 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.226575_137.1", 2022-09-12T19:16:46.699 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.699 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17", 2022-09-12T19:16:46.699 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.226575054Z", 2022-09-12T19:16:46.699 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.700 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.700 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.700 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.700 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.701 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.701 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_BCZEMlNNfcs_zua_I4p9TaW", 2022-09-12T19:16:46.701 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.702 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.702 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.702 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.703 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.703 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.703 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.703 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.704 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.227798_138.1", 2022-09-12T19:16:46.704 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.704 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-17", 2022-09-12T19:16:46.705 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.227798137Z", 2022-09-12T19:16:46.705 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.706 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.706 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.706 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.707 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.707 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.708 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_i8wOFAvzgTHQCJOO8N6Bnam", 2022-09-12T19:16:46.708 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.708 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.708 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.709 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.709 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.710 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.710 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.710 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.711 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.232329_139.1", 2022-09-12T19:16:46.711 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.712 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-17", 2022-09-12T19:16:46.712 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.232329182Z", 2022-09-12T19:16:46.713 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.713 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.714 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.714 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.715 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.715 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.716 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_i8wOFAvzgTHQCJOO8N6Bnam", 2022-09-12T19:16:46.716 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.717 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.717 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.718 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.718 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.719 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.719 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.719 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.720 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.268820_140.1", 2022-09-12T19:16:46.720 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.721 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18", 2022-09-12T19:16:46.721 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.268820788Z", 2022-09-12T19:16:46.722 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.722 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.723 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.723 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.724 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.724 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.724 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_5GsV_zMTqFSGin4m-BDaD32", 2022-09-12T19:16:46.725 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.725 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.726 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.726 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.726 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.727 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.727 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.727 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.728 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.282707_141.1", 2022-09-12T19:16:46.728 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.728 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18", 2022-09-12T19:16:46.729 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.282707229Z", 2022-09-12T19:16:46.729 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.729 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.730 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.730 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.730 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.731 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.731 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_5GsV_zMTqFSGin4m-BDaD32", 2022-09-12T19:16:46.732 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.732 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.732 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.732 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.733 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.733 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.733 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.734 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.734 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.284810_142.1", 2022-09-12T19:16:46.734 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.734 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-18", 2022-09-12T19:16:46.735 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.284810891Z", 2022-09-12T19:16:46.735 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.735 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.736 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.736 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.736 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.736 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.737 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Ow-fYJ8C7sQ2JLuZ3C6E5Oy", 2022-09-12T19:16:46.737 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.737 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.738 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.738 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.738 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.739 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.739 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.740 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.740 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.287795_143.1", 2022-09-12T19:16:46.740 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.741 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-18", 2022-09-12T19:16:46.741 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.287795384Z", 2022-09-12T19:16:46.741 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.741 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.742 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.742 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.742 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.742 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.743 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Ow-fYJ8C7sQ2JLuZ3C6E5Oy", 2022-09-12T19:16:46.743 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.743 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.743 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.744 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.744 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.744 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.744 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.745 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.745 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.308137_144.1", 2022-09-12T19:16:46.745 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.745 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16", 2022-09-12T19:16:46.746 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.308137433Z", 2022-09-12T19:16:46.746 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.746 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.746 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_V99BgSq-hvAYBvkex60C8n2", 2022-09-12T19:16:46.747 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.747 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.747 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.748 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_V99BgSq-hvAYBvkex60C8n2", 2022-09-12T19:16:46.748 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:46.749 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.749 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.749 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.750 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.750 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.750 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.751 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.751 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.310988_145.1", 2022-09-12T19:16:46.751 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.752 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16", 2022-09-12T19:16:46.752 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.310988969Z", 2022-09-12T19:16:46.752 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.753 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.753 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_V99BgSq-hvAYBvkex60C8n2", 2022-09-12T19:16:46.753 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.754 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.754 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.754 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_V99BgSq-hvAYBvkex60C8n2", 2022-09-12T19:16:46.755 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:46.755 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.755 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.755 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.756 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.756 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.756 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.757 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.757 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.322795_146.1", 2022-09-12T19:16:46.757 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.758 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17", 2022-09-12T19:16:46.758 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.322795845Z", 2022-09-12T19:16:46.758 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.758 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.759 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_BCZEMlNNfcs_zua_I4p9TaW", 2022-09-12T19:16:46.759 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.759 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.759 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.759 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_BCZEMlNNfcs_zua_I4p9TaW", 2022-09-12T19:16:46.760 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:46.760 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.760 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.760 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.761 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.761 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.761 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.761 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.762 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.325825_147.1", 2022-09-12T19:16:46.762 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.762 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17", 2022-09-12T19:16:46.762 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.325825003Z", 2022-09-12T19:16:46.763 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.763 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.763 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_BCZEMlNNfcs_zua_I4p9TaW", 2022-09-12T19:16:46.764 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.764 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.764 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.765 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_BCZEMlNNfcs_zua_I4p9TaW", 2022-09-12T19:16:46.765 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:46.765 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.765 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.766 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.766 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.766 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.767 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.767 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.767 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.343478_148.1", 2022-09-12T19:16:46.767 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.768 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18", 2022-09-12T19:16:46.768 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.343478792Z", 2022-09-12T19:16:46.768 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.769 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.769 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_5GsV_zMTqFSGin4m-BDaD32", 2022-09-12T19:16:46.769 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.770 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.770 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.770 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_5GsV_zMTqFSGin4m-BDaD32", 2022-09-12T19:16:46.771 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:46.771 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.771 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.772 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.772 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.772 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.772 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.773 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.773 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009066.346375_149.1", 2022-09-12T19:16:46.773 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.773 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18", 2022-09-12T19:16:46.774 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:57:46.346375662Z", 2022-09-12T19:16:46.774 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.774 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.775 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_5GsV_zMTqFSGin4m-BDaD32", 2022-09-12T19:16:46.775 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.775 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.775 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.776 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_5GsV_zMTqFSGin4m-BDaD32", 2022-09-12T19:16:46.776 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:46.776 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.777 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.777 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.777 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.778 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.778 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.778 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.779 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009433.572798_179.1", 2022-09-12T19:16:46.779 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.779 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19", 2022-09-12T19:16:46.779 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:03:53.572798305Z", 2022-09-12T19:16:46.780 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.780 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.780 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.781 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.781 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.781 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.782 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_r98RnZbo3gbuHHD5iR9vaQX", 2022-09-12T19:16:46.782 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.782 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.783 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.783 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.784 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.784 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.784 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.784 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.785 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009433.576124_180.1", 2022-09-12T19:16:46.785 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.785 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19", 2022-09-12T19:16:46.785 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:03:53.576124288Z", 2022-09-12T19:16:46.786 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.786 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.786 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.786 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.787 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.787 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.787 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_r98RnZbo3gbuHHD5iR9vaQX", 2022-09-12T19:16:46.787 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.788 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.788 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.788 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.788 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.789 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.789 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.790 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.790 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009433.577370_181.1", 2022-09-12T19:16:46.790 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.790 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-19", 2022-09-12T19:16:46.791 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:03:53.577370053Z", 2022-09-12T19:16:46.791 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.791 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.791 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.792 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.792 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.792 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.792 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_FepK5riJM4xGtHwIBSHMDgN", 2022-09-12T19:16:46.793 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.793 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.793 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.793 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.794 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.794 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.794 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.794 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.795 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009433.580348_182.1", 2022-09-12T19:16:46.795 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.795 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-19", 2022-09-12T19:16:46.795 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:03:53.580348934Z", 2022-09-12T19:16:46.796 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.796 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.796 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.796 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.797 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.797 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.798 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_FepK5riJM4xGtHwIBSHMDgN", 2022-09-12T19:16:46.798 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.798 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.798 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.799 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.799 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.799 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.800 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.800 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.800 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009433.593338_183.1", 2022-09-12T19:16:46.800 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.801 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19", 2022-09-12T19:16:46.801 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:03:53.593338729Z", 2022-09-12T19:16:46.801 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.801 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.802 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_r98RnZbo3gbuHHD5iR9vaQX", 2022-09-12T19:16:46.802 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.802 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.803 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.803 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_r98RnZbo3gbuHHD5iR9vaQX", 2022-09-12T19:16:46.803 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:46.804 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.804 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.804 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.805 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.805 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.805 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.805 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.806 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009433.597955_184.1", 2022-09-12T19:16:46.806 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.807 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19", 2022-09-12T19:16:46.807 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:03:53.597955912Z", 2022-09-12T19:16:46.807 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.807 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.808 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_r98RnZbo3gbuHHD5iR9vaQX", 2022-09-12T19:16:46.808 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.808 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.809 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.809 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_r98RnZbo3gbuHHD5iR9vaQX", 2022-09-12T19:16:46.809 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:46.809 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.810 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.810 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.810 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.810 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.811 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.811 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.811 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009630.186894_193.1", 2022-09-12T19:16:46.811 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.812 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:16:46.812 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:07:10.186894533Z", 2022-09-12T19:16:46.812 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.812 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.813 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.813 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.813 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.813 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.814 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:46.814 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.814 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.815 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.815 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.815 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.815 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.816 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.816 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.816 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009630.192780_194.1", 2022-09-12T19:16:46.816 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.817 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:16:46.817 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:07:10.192780968Z", 2022-09-12T19:16:46.817 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.817 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.818 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.818 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.818 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.818 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.819 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:46.819 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.819 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.819 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.819 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.820 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.820 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.820 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.820 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.821 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009630.194692_195.1", 2022-09-12T19:16:46.821 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.821 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-20", 2022-09-12T19:16:46.821 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:07:10.194692069Z", 2022-09-12T19:16:46.822 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.822 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.823 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.823 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.823 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.823 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.824 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_vBgV2Ivb4f-6P44JJPXNdtC", 2022-09-12T19:16:46.824 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.824 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.824 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.825 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.825 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.825 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.825 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.826 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.826 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009630.200316_196.1", 2022-09-12T19:16:46.826 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.826 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-20", 2022-09-12T19:16:46.827 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:07:10.200316464Z", 2022-09-12T19:16:46.827 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.827 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.828 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.828 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.828 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.828 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.829 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_vBgV2Ivb4f-6P44JJPXNdtC", 2022-09-12T19:16:46.829 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.829 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.829 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.830 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.830 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.830 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.830 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.831 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.831 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009668.037998_204.1", 2022-09-12T19:16:46.831 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.831 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:16:46.832 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:07:48.037998619Z", 2022-09-12T19:16:46.832 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.832 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.832 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:46.833 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.833 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.833 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.833 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:46.834 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:46.834 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.835 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.835 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.835 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.836 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.836 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.836 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.836 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009668.041232_205.1", 2022-09-12T19:16:46.837 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.837 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:16:46.837 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:07:48.041232517Z", 2022-09-12T19:16:46.837 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.838 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.838 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:46.838 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.839 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.839 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.840 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:46.841 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:46.841 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.842 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.842 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.843 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.843 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.844 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.844 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.845 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009726.091934_207.1", 2022-09-12T19:16:46.845 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.846 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:16:46.846 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:08:46.091934751Z", 2022-09-12T19:16:46.847 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.847 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.848 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:46.848 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:46.849 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.849 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.850 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:46.851 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 3 2022-09-12T19:16:46.851 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.852 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.852 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.853 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.854 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.854 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.855 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.855 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009726.095686_208.1", 2022-09-12T19:16:46.856 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.856 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:16:46.857 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:08:46.095686282Z", 2022-09-12T19:16:46.857 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.858 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.858 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:46.859 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:46.860 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.860 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.861 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:46.861 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 3 2022-09-12T19:16:46.861 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.862 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.862 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.863 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.863 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.864 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.864 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.864 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009842.925161_211.1", 2022-09-12T19:16:46.864 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.865 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21", 2022-09-12T19:16:46.865 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:42.925161228Z", 2022-09-12T19:16:46.865 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.866 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.866 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.867 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.867 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.868 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.868 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_wA9vY_PYbZdc0eUi2hSxrjB", 2022-09-12T19:16:46.869 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.870 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.870 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.871 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.871 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.872 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.872 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.873 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.873 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009842.930693_212.1", 2022-09-12T19:16:46.874 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.874 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21", 2022-09-12T19:16:46.875 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:42.930693175Z", 2022-09-12T19:16:46.875 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.876 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.876 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.877 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.877 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.878 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.878 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_wA9vY_PYbZdc0eUi2hSxrjB", 2022-09-12T19:16:46.879 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.879 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.880 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.880 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.881 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.881 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.882 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.882 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.883 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009842.932348_213.1", 2022-09-12T19:16:46.883 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.884 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-21", 2022-09-12T19:16:46.884 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:42.932348571Z", 2022-09-12T19:16:46.885 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.885 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.886 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.886 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.887 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.887 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.888 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_YhO7K8SUXb88GVnz95qTMjU", 2022-09-12T19:16:46.888 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.889 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.889 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.890 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.890 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.891 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.891 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.892 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.892 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009842.938156_214.1", 2022-09-12T19:16:46.893 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.893 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-21", 2022-09-12T19:16:46.894 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:42.938156918Z", 2022-09-12T19:16:46.894 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.895 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.895 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.896 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.896 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.897 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.897 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_YhO7K8SUXb88GVnz95qTMjU", 2022-09-12T19:16:46.898 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.898 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.899 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.899 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.900 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.900 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.901 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.901 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.901 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009842.960242_215.1", 2022-09-12T19:16:46.901 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.902 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22", 2022-09-12T19:16:46.902 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:42.960242821Z", 2022-09-12T19:16:46.903 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.903 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.903 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.903 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.904 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.904 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.904 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_jCE0vI8jcGzuttLr9LzBCWJ", 2022-09-12T19:16:46.905 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.905 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.905 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.905 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.906 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.906 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.906 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.907 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.907 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009842.965349_216.1", 2022-09-12T19:16:46.907 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.907 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22", 2022-09-12T19:16:46.908 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:42.965349366Z", 2022-09-12T19:16:46.908 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.908 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.909 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.909 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.909 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.910 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.910 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_jCE0vI8jcGzuttLr9LzBCWJ", 2022-09-12T19:16:46.910 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.910 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.911 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.911 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.911 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.912 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.912 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.912 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.913 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009842.966491_217.1", 2022-09-12T19:16:46.913 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.913 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-22", 2022-09-12T19:16:46.913 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:42.966491565Z", 2022-09-12T19:16:46.914 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.914 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.914 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.915 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.915 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.915 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.915 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_90qhmHoutPPaDynubPFZH9h", 2022-09-12T19:16:46.916 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.916 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.916 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.916 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.917 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.917 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.917 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.918 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.918 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009842.970703_218.1", 2022-09-12T19:16:46.918 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.918 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-22", 2022-09-12T19:16:46.919 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:42.970703166Z", 2022-09-12T19:16:46.919 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.919 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.919 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.920 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.920 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.920 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.921 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_90qhmHoutPPaDynubPFZH9h", 2022-09-12T19:16:46.921 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.921 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.922 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.922 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.922 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.922 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.923 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.923 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.923 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009843.006494_219.1", 2022-09-12T19:16:46.923 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.924 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23", 2022-09-12T19:16:46.924 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:43.006494173Z", 2022-09-12T19:16:46.924 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.925 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.925 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.925 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.925 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.926 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.926 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_8EYxPua6E_71y78ZWvmZh7v", 2022-09-12T19:16:46.926 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.926 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.927 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.927 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.927 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.927 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.928 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.928 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.928 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009843.009740_220.1", 2022-09-12T19:16:46.929 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.929 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23", 2022-09-12T19:16:46.929 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:43.009740592Z", 2022-09-12T19:16:46.930 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.930 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.930 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.930 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.931 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.931 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.931 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_8EYxPua6E_71y78ZWvmZh7v", 2022-09-12T19:16:46.931 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.932 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.932 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.932 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.932 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.933 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.933 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.933 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.934 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009843.011011_221.1", 2022-09-12T19:16:46.934 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.934 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-23", 2022-09-12T19:16:46.934 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:43.011011338Z", 2022-09-12T19:16:46.935 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.935 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.935 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.935 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.936 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.936 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.936 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_EWfzEo4mEzPqDzTBu2wloku", 2022-09-12T19:16:46.936 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.937 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.937 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.937 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.937 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.938 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.938 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.938 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.938 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009843.015119_222.1", 2022-09-12T19:16:46.939 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:46.939 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-23", 2022-09-12T19:16:46.939 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:43.015119585Z", 2022-09-12T19:16:46.939 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.940 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.940 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.940 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.940 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.941 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.941 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_EWfzEo4mEzPqDzTBu2wloku", 2022-09-12T19:16:46.941 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.941 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.942 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.942 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.942 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.942 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.943 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.943 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.943 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009843.040988_223.1", 2022-09-12T19:16:46.943 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.944 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21", 2022-09-12T19:16:46.944 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:43.040988054Z", 2022-09-12T19:16:46.944 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.944 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.945 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_wA9vY_PYbZdc0eUi2hSxrjB", 2022-09-12T19:16:46.945 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.945 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.945 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.946 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_wA9vY_PYbZdc0eUi2hSxrjB", 2022-09-12T19:16:46.946 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:46.946 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.946 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.947 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.947 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.947 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.948 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.948 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.948 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009843.045887_224.1", 2022-09-12T19:16:46.948 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.949 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21", 2022-09-12T19:16:46.949 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:43.045887351Z", 2022-09-12T19:16:46.949 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.950 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.950 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_wA9vY_PYbZdc0eUi2hSxrjB", 2022-09-12T19:16:46.950 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.950 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.951 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.951 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_wA9vY_PYbZdc0eUi2hSxrjB", 2022-09-12T19:16:46.951 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:46.952 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.952 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.952 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.952 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.953 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.953 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.953 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.954 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009843.057828_225.1", 2022-09-12T19:16:46.954 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.954 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22", 2022-09-12T19:16:46.955 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:43.057828971Z", 2022-09-12T19:16:46.955 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.955 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.955 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_jCE0vI8jcGzuttLr9LzBCWJ", 2022-09-12T19:16:46.956 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.956 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.957 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.957 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_jCE0vI8jcGzuttLr9LzBCWJ", 2022-09-12T19:16:46.957 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:46.957 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.958 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.958 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.958 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.959 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.959 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.959 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.959 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009843.060815_226.1", 2022-09-12T19:16:46.960 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.960 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22", 2022-09-12T19:16:46.960 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:43.060815675Z", 2022-09-12T19:16:46.961 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.961 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.961 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_jCE0vI8jcGzuttLr9LzBCWJ", 2022-09-12T19:16:46.961 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.962 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.962 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.962 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_jCE0vI8jcGzuttLr9LzBCWJ", 2022-09-12T19:16:46.962 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:46.963 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.963 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.963 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.963 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.964 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.964 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.964 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.965 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009843.077332_227.1", 2022-09-12T19:16:46.965 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.966 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23", 2022-09-12T19:16:46.966 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:43.077332048Z", 2022-09-12T19:16:46.967 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.968 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.969 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_8EYxPua6E_71y78ZWvmZh7v", 2022-09-12T19:16:46.969 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.970 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.971 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.972 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_8EYxPua6E_71y78ZWvmZh7v", 2022-09-12T19:16:46.973 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:46.973 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.974 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.975 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.976 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.976 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.977 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.978 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.978 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009843.080040_228.1", 2022-09-12T19:16:46.979 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.980 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23", 2022-09-12T19:16:46.981 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:10:43.080040677Z", 2022-09-12T19:16:46.981 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.982 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.983 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_8EYxPua6E_71y78ZWvmZh7v", 2022-09-12T19:16:46.984 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.984 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.985 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.985 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_8EYxPua6E_71y78ZWvmZh7v", 2022-09-12T19:16:46.986 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:46.986 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.987 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.987 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:46.988 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.988 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.989 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.989 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.989 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.213940_231.1", 2022-09-12T19:16:46.989 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.990 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:16:46.990 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.213940834Z", 2022-09-12T19:16:46.990 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.991 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.991 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.991 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:46.992 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.992 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:46.992 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:46.992 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:46.993 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.993 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:46.993 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:46.994 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.994 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:46.995 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:46.995 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:46.996 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.217329_232.1", 2022-09-12T19:16:46.996 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:46.997 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:16:46.997 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.217329420Z", 2022-09-12T19:16:46.998 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:46.998 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:46.999 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:46.999 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.000 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.000 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.001 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:47.001 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.002 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.002 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.003 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.003 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.004 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.004 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.005 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.005 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.218459_233.1", 2022-09-12T19:16:47.006 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.006 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-24", 2022-09-12T19:16:47.007 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.218459288Z", 2022-09-12T19:16:47.007 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.008 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.008 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.009 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.009 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.010 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.010 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_zt4J3xCzAzkFDmMYSv2ZhpD", 2022-09-12T19:16:47.011 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.011 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.012 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.012 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.013 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.013 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.014 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.014 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.015 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.221613_234.1", 2022-09-12T19:16:47.015 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.016 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-24", 2022-09-12T19:16:47.016 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.221613900Z", 2022-09-12T19:16:47.016 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.017 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.017 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.018 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.018 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.019 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.019 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_zt4J3xCzAzkFDmMYSv2ZhpD", 2022-09-12T19:16:47.020 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.020 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.021 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.021 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.022 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.022 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.023 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.023 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.024 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.242266_235.1", 2022-09-12T19:16:47.024 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.025 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:16:47.025 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.242266250Z", 2022-09-12T19:16:47.026 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.026 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.027 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.027 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.028 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.028 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.029 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:47.029 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.030 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.030 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.031 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.031 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.032 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.032 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.033 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.033 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.245374_236.1", 2022-09-12T19:16:47.034 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.035 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:16:47.035 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.245374907Z", 2022-09-12T19:16:47.036 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.036 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.037 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.037 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.038 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.038 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.039 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:47.039 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.039 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.040 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.040 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.041 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.041 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.041 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.042 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.042 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.246464_237.1", 2022-09-12T19:16:47.043 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.043 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-25", 2022-09-12T19:16:47.044 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.246464476Z", 2022-09-12T19:16:47.045 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.045 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.045 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.046 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.046 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.046 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.047 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_FuWLdq4G8Pitukb9rq9CEoZ", 2022-09-12T19:16:47.047 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.047 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.048 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.048 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.048 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.049 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.049 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.050 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.050 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.249409_238.1", 2022-09-12T19:16:47.051 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.051 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-25", 2022-09-12T19:16:47.052 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.249409215Z", 2022-09-12T19:16:47.052 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.053 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.053 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.054 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.054 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.054 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.054 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_FuWLdq4G8Pitukb9rq9CEoZ", 2022-09-12T19:16:47.055 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.055 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.055 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.056 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.056 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.057 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.057 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.058 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.058 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.282380_239.1", 2022-09-12T19:16:47.059 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.059 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:16:47.060 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.282380399Z", 2022-09-12T19:16:47.060 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.061 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.061 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.062 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.062 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.062 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.062 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:47.063 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.063 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.063 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.064 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.064 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.064 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.064 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.065 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.065 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.285471_240.1", 2022-09-12T19:16:47.065 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.065 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:16:47.066 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.285471857Z", 2022-09-12T19:16:47.066 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.066 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.066 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.067 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.067 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.067 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.068 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:47.068 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.068 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.069 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.069 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.069 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.069 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.070 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.070 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.070 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.286588_241.1", 2022-09-12T19:16:47.070 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.071 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-26", 2022-09-12T19:16:47.071 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.286588982Z", 2022-09-12T19:16:47.071 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.071 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.072 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.072 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.072 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.072 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.073 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_-j_vUqBTsANTAcA80szuZUK", 2022-09-12T19:16:47.073 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.073 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.073 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.074 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.074 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.074 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.075 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.075 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.075 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.289386_242.1", 2022-09-12T19:16:47.075 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.076 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-26", 2022-09-12T19:16:47.076 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.289386844Z", 2022-09-12T19:16:47.076 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.077 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.077 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.077 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.077 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.078 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.078 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_-j_vUqBTsANTAcA80szuZUK", 2022-09-12T19:16:47.078 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.078 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.079 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.079 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.080 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.080 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.080 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.080 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.081 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.309215_243.1", 2022-09-12T19:16:47.081 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.081 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:16:47.082 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.309215519Z", 2022-09-12T19:16:47.082 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.082 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.082 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:47.083 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.083 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.083 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.083 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:47.084 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.084 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.084 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.084 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.085 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.085 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.085 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.085 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.086 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.312069_244.1", 2022-09-12T19:16:47.086 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.086 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:16:47.086 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.312069106Z", 2022-09-12T19:16:47.087 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.087 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.087 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:47.087 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.088 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.088 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.088 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:47.089 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.089 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.089 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.089 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.090 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.090 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.090 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.090 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.091 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.323214_245.1", 2022-09-12T19:16:47.091 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.091 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:16:47.091 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.323214538Z", 2022-09-12T19:16:47.092 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.092 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.092 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:47.093 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.093 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.093 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.093 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:47.094 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 3 2022-09-12T19:16:47.094 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.094 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.094 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.095 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.095 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.095 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.095 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.096 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.334825_246.1", 2022-09-12T19:16:47.096 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.096 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:16:47.096 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.334825968Z", 2022-09-12T19:16:47.097 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.097 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.097 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:47.097 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.098 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.098 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.098 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:47.098 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 3 2022-09-12T19:16:47.099 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.099 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.099 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.100 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.100 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.100 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.101 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.101 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.350812_247.1", 2022-09-12T19:16:47.101 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.101 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:16:47.102 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.350812474Z", 2022-09-12T19:16:47.102 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.102 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.102 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:47.103 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.103 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.103 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.103 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:47.104 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.104 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.104 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.104 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.105 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.105 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.105 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.105 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.106 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.355013_248.1", 2022-09-12T19:16:47.106 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.106 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:16:47.107 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.355013585Z", 2022-09-12T19:16:47.107 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.107 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.107 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:47.108 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.108 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.108 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.108 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:47.109 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.109 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.109 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.109 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.110 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.110 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.111 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.111 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.112 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.369892_249.1", 2022-09-12T19:16:47.112 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.113 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:16:47.113 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.369892532Z", 2022-09-12T19:16:47.114 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.114 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.114 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:47.115 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.115 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.115 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.115 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:47.116 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 3 2022-09-12T19:16:47.116 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.116 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.116 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.117 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.117 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.117 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.118 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.118 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.374217_250.1", 2022-09-12T19:16:47.118 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.118 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:16:47.119 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.374217196Z", 2022-09-12T19:16:47.119 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.119 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.119 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:47.120 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.120 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.120 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.120 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:47.121 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 3 2022-09-12T19:16:47.121 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.121 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.122 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.122 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.122 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.122 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.123 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.123 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.394432_251.1", 2022-09-12T19:16:47.123 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.123 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:16:47.124 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.394432026Z", 2022-09-12T19:16:47.124 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.124 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.124 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:47.125 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.125 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.125 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.125 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:47.126 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.126 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.126 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.127 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.127 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.127 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.128 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.128 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.128 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.398632_252.1", 2022-09-12T19:16:47.129 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.129 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:16:47.129 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.398632136Z", 2022-09-12T19:16:47.129 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.130 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.130 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:47.130 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.131 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.131 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.131 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:47.131 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.132 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.132 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.132 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.132 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.133 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.133 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.133 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.133 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.418847_253.1", 2022-09-12T19:16:47.134 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.134 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:16:47.134 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.418847506Z", 2022-09-12T19:16:47.135 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.135 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.135 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:47.135 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.136 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.136 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.136 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:47.136 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 3 2022-09-12T19:16:47.137 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.137 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.137 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.137 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.138 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.138 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.138 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.139 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663009909.421418_254.1", 2022-09-12T19:16:47.139 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.139 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:16:47.139 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:11:49.421418488Z", 2022-09-12T19:16:47.140 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.140 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.140 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:47.140 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.141 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.141 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.141 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:47.142 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 3 2022-09-12T19:16:47.142 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.143 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.143 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.144 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.144 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.145 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.145 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.146 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.022864_258.1", 2022-09-12T19:16:47.146 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.147 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-27:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.27", 2022-09-12T19:16:47.147 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.022864695Z", 2022-09-12T19:16:47.147 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.148 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.148 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.148 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.149 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.149 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.149 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_4Ia0kBYSOXAPun0srTpHOgg", 2022-09-12T19:16:47.149 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.150 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.150 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.150 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.151 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.151 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.151 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.151 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.152 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.027319_259.1", 2022-09-12T19:16:47.152 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.152 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-27:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.27", 2022-09-12T19:16:47.153 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.027319061Z", 2022-09-12T19:16:47.153 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.153 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.153 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.154 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.154 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.154 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.155 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_4Ia0kBYSOXAPun0srTpHOgg", 2022-09-12T19:16:47.155 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.156 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.156 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.157 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.157 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.158 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.158 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.159 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.159 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.029529_260.1", 2022-09-12T19:16:47.160 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.160 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-27", 2022-09-12T19:16:47.161 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.029529115Z", 2022-09-12T19:16:47.161 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.162 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.162 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.163 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.163 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.164 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.164 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_hL6ubPjUnxDbFclj75F-9Yw", 2022-09-12T19:16:47.165 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.165 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.166 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.166 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.167 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.167 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.168 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.168 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.169 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.035074_261.1", 2022-09-12T19:16:47.169 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.169 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-27", 2022-09-12T19:16:47.170 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.035074135Z", 2022-09-12T19:16:47.170 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.171 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.171 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.172 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.172 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.173 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.173 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_hL6ubPjUnxDbFclj75F-9Yw", 2022-09-12T19:16:47.174 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.174 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.175 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.175 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.176 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.176 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.177 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.177 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.178 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.060437_262.1", 2022-09-12T19:16:47.178 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.179 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-28:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.28", 2022-09-12T19:16:47.179 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.060437057Z", 2022-09-12T19:16:47.180 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.180 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.180 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.181 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.181 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.182 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.182 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_F91L-y4ApRIftBlrzdVSKhB", 2022-09-12T19:16:47.183 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.183 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.183 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.184 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.184 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.185 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.185 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.186 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.186 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.067920_263.1", 2022-09-12T19:16:47.187 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.187 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-28:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.28", 2022-09-12T19:16:47.187 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.067920385Z", 2022-09-12T19:16:47.188 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.188 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.189 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.189 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.190 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.190 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.190 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_F91L-y4ApRIftBlrzdVSKhB", 2022-09-12T19:16:47.191 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.191 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.191 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.192 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.192 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.193 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.193 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.194 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.194 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.068850_264.1", 2022-09-12T19:16:47.195 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.195 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-28", 2022-09-12T19:16:47.196 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.068850124Z", 2022-09-12T19:16:47.196 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.197 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.197 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.198 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.198 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.198 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.199 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_4foA5qWzZ6DKNqXOUrqtpMi", 2022-09-12T19:16:47.199 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.200 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.200 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.200 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.201 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.201 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.202 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.202 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.203 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.074343_265.1", 2022-09-12T19:16:47.203 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.203 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-28", 2022-09-12T19:16:47.204 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.074343801Z", 2022-09-12T19:16:47.204 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.205 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.205 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.206 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.206 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.207 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.207 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_4foA5qWzZ6DKNqXOUrqtpMi", 2022-09-12T19:16:47.208 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.208 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.209 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.209 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.210 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.210 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.211 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.211 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.212 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.114860_266.1", 2022-09-12T19:16:47.213 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.213 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29", 2022-09-12T19:16:47.214 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.114860178Z", 2022-09-12T19:16:47.214 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.215 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.215 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.215 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.216 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.217 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.217 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_kHZQcZIbg231-V_zVbHphPU", 2022-09-12T19:16:47.217 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.218 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.218 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.219 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.219 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.220 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.220 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.221 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.221 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.119876_267.1", 2022-09-12T19:16:47.222 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.222 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29", 2022-09-12T19:16:47.223 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.119876089Z", 2022-09-12T19:16:47.223 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.224 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.224 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.225 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.225 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.226 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.226 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_kHZQcZIbg231-V_zVbHphPU", 2022-09-12T19:16:47.227 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.227 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.228 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.228 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.229 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.229 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.229 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.230 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.230 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.121725_268.1", 2022-09-12T19:16:47.231 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.231 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-29", 2022-09-12T19:16:47.232 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.121725517Z", 2022-09-12T19:16:47.232 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.233 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.233 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.234 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.234 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.234 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.235 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_2ti8fe4vkZVz-LOJO9SiYze", 2022-09-12T19:16:47.235 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.236 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.236 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.237 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.237 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.238 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.238 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.239 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.239 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.125711_269.1", 2022-09-12T19:16:47.239 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.239 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-29", 2022-09-12T19:16:47.240 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.125711914Z", 2022-09-12T19:16:47.240 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.240 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.240 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.241 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.241 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.241 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.241 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_2ti8fe4vkZVz-LOJO9SiYze", 2022-09-12T19:16:47.242 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.242 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.242 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.242 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.243 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.243 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.243 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.243 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.244 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.153919_270.1", 2022-09-12T19:16:47.244 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.244 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-27:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.27", 2022-09-12T19:16:47.245 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.153919920Z", 2022-09-12T19:16:47.245 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.245 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.245 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_4Ia0kBYSOXAPun0srTpHOgg", 2022-09-12T19:16:47.246 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.246 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.246 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.247 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_4Ia0kBYSOXAPun0srTpHOgg", 2022-09-12T19:16:47.247 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.247 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.247 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.248 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.248 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.248 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.248 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.249 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.249 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.156407_271.1", 2022-09-12T19:16:47.249 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.249 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-27:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.27", 2022-09-12T19:16:47.250 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.156407416Z", 2022-09-12T19:16:47.250 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.250 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.250 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_4Ia0kBYSOXAPun0srTpHOgg", 2022-09-12T19:16:47.251 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.251 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.251 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.251 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_4Ia0kBYSOXAPun0srTpHOgg", 2022-09-12T19:16:47.252 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.252 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.252 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.252 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.253 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.253 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.253 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.253 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.254 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.183174_272.1", 2022-09-12T19:16:47.254 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.254 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-28:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.28", 2022-09-12T19:16:47.254 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.183174956Z", 2022-09-12T19:16:47.255 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.255 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.255 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_F91L-y4ApRIftBlrzdVSKhB", 2022-09-12T19:16:47.256 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.256 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.256 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.256 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_F91L-y4ApRIftBlrzdVSKhB", 2022-09-12T19:16:47.256 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.257 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.257 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.257 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.258 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.258 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.266 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.266 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.266 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.186673_273.1", 2022-09-12T19:16:47.267 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.267 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-28:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.28", 2022-09-12T19:16:47.267 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.186673632Z", 2022-09-12T19:16:47.267 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.268 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.268 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_F91L-y4ApRIftBlrzdVSKhB", 2022-09-12T19:16:47.268 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.268 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.269 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.269 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_F91L-y4ApRIftBlrzdVSKhB", 2022-09-12T19:16:47.269 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.269 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.270 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.270 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.270 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.270 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.271 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.271 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.271 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.258924_274.1", 2022-09-12T19:16:47.272 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.272 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29", 2022-09-12T19:16:47.272 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.258924078Z", 2022-09-12T19:16:47.272 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.273 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.273 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_kHZQcZIbg231-V_zVbHphPU", 2022-09-12T19:16:47.273 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.273 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.274 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.274 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_kHZQcZIbg231-V_zVbHphPU", 2022-09-12T19:16:47.274 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.274 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.275 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.275 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.275 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.275 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.276 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.276 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.276 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.261587_275.1", 2022-09-12T19:16:47.276 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.277 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29", 2022-09-12T19:16:47.277 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.261587678Z", 2022-09-12T19:16:47.277 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.277 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.278 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_kHZQcZIbg231-V_zVbHphPU", 2022-09-12T19:16:47.278 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.278 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.278 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.279 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_kHZQcZIbg231-V_zVbHphPU", 2022-09-12T19:16:47.279 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.279 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.279 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.280 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.280 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.280 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.280 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.281 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.281 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.295442_276.1", 2022-09-12T19:16:47.281 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.281 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-30:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.30", 2022-09-12T19:16:47.282 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.295442828Z", 2022-09-12T19:16:47.282 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.282 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.282 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.283 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.283 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.283 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.283 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_xXGncRxFB5weAPxs1wnNxZq", 2022-09-12T19:16:47.284 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.284 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.284 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.284 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.285 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.285 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.285 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.285 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.286 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.297949_277.1", 2022-09-12T19:16:47.286 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.286 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-30:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.30", 2022-09-12T19:16:47.287 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.297949202Z", 2022-09-12T19:16:47.287 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.287 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.287 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.288 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.288 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.288 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.288 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_xXGncRxFB5weAPxs1wnNxZq", 2022-09-12T19:16:47.289 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.289 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.289 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.289 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.290 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.290 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.290 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.290 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.291 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.298820_278.1", 2022-09-12T19:16:47.291 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.291 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-30", 2022-09-12T19:16:47.291 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.298820021Z", 2022-09-12T19:16:47.292 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.292 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.292 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.292 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.293 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.293 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.293 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_o1lHEa9WwJ0A1-_nw8PZQAy", 2022-09-12T19:16:47.293 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.294 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.294 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.294 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.294 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.295 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.295 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.295 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.296 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.301106_279.1", 2022-09-12T19:16:47.296 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.296 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-30", 2022-09-12T19:16:47.296 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.301106622Z", 2022-09-12T19:16:47.297 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.297 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.297 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.297 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.298 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.298 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.298 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_o1lHEa9WwJ0A1-_nw8PZQAy", 2022-09-12T19:16:47.298 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.299 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.299 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.299 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.299 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.300 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.300 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.300 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.300 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.318892_280.1", 2022-09-12T19:16:47.301 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.301 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-31:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.31", 2022-09-12T19:16:47.301 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.318892577Z", 2022-09-12T19:16:47.301 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.302 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.302 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.302 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.302 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.303 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.303 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_aXhKq1Q-6is3aiib6Zv69Wn", 2022-09-12T19:16:47.303 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.303 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.304 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.304 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.304 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.305 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.305 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.305 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.305 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.321371_281.1", 2022-09-12T19:16:47.306 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.306 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-31:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.31", 2022-09-12T19:16:47.306 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.321371696Z", 2022-09-12T19:16:47.306 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.307 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.307 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.307 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.308 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.308 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.308 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_aXhKq1Q-6is3aiib6Zv69Wn", 2022-09-12T19:16:47.308 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.309 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.309 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.309 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.309 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.310 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.310 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.310 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.310 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.323664_282.1", 2022-09-12T19:16:47.311 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.311 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-31", 2022-09-12T19:16:47.311 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.323664287Z", 2022-09-12T19:16:47.311 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.312 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.312 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.312 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.312 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.313 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.313 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_ZXsXmNqWQzafZqU8AgaaFgE", 2022-09-12T19:16:47.313 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.313 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.314 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.314 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.314 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.315 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.315 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.315 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.315 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.332269_283.1", 2022-09-12T19:16:47.316 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.316 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-31", 2022-09-12T19:16:47.316 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.332269695Z", 2022-09-12T19:16:47.316 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.317 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.317 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.317 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.318 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.318 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.318 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_ZXsXmNqWQzafZqU8AgaaFgE", 2022-09-12T19:16:47.318 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.319 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.319 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.319 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.319 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.320 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.320 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.320 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.320 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.379861_284.1", 2022-09-12T19:16:47.321 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.321 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-32:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.32", 2022-09-12T19:16:47.321 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.379861593Z", 2022-09-12T19:16:47.321 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.322 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.322 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.322 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.322 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.323 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.323 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UeviL3qodKD5NX6Gmu8wcLZ", 2022-09-12T19:16:47.323 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.323 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.324 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.324 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.324 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.324 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.325 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.325 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.325 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.385256_285.1", 2022-09-12T19:16:47.325 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.326 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-32:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.32", 2022-09-12T19:16:47.326 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.385256336Z", 2022-09-12T19:16:47.326 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.326 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.327 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.327 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.327 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.327 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.328 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UeviL3qodKD5NX6Gmu8wcLZ", 2022-09-12T19:16:47.328 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.328 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.328 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.329 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.329 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.329 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.329 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.330 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.330 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.388164_286.1", 2022-09-12T19:16:47.330 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.331 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-32", 2022-09-12T19:16:47.331 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.388164607Z", 2022-09-12T19:16:47.331 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.331 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.332 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.332 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.332 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.332 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.333 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_sSvBc9sHjXg70XiY66Bq6ME", 2022-09-12T19:16:47.333 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.333 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.333 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.334 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.334 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.334 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.334 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.335 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.335 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.392572_287.1", 2022-09-12T19:16:47.335 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.335 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-32", 2022-09-12T19:16:47.336 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.392572009Z", 2022-09-12T19:16:47.336 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.336 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.336 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.337 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.337 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.337 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.337 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_sSvBc9sHjXg70XiY66Bq6ME", 2022-09-12T19:16:47.338 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.338 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.338 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.338 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.339 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.339 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.339 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.339 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.340 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.426848_288.1", 2022-09-12T19:16:47.340 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.340 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-30:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.30", 2022-09-12T19:16:47.340 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.426848512Z", 2022-09-12T19:16:47.341 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.341 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.341 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_xXGncRxFB5weAPxs1wnNxZq", 2022-09-12T19:16:47.341 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.342 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.342 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.342 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_xXGncRxFB5weAPxs1wnNxZq", 2022-09-12T19:16:47.343 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.343 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.343 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.343 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.344 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.344 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.344 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.344 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.345 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.431873_289.1", 2022-09-12T19:16:47.345 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.345 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-30:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.30", 2022-09-12T19:16:47.345 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.431873816Z", 2022-09-12T19:16:47.346 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.346 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.346 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_xXGncRxFB5weAPxs1wnNxZq", 2022-09-12T19:16:47.346 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.347 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.347 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.347 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_xXGncRxFB5weAPxs1wnNxZq", 2022-09-12T19:16:47.347 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.348 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.348 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.349 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.349 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.349 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.349 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.350 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.350 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.505940_290.1", 2022-09-12T19:16:47.350 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.350 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-31:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.31", 2022-09-12T19:16:47.351 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.505940792Z", 2022-09-12T19:16:47.351 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.351 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.352 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_aXhKq1Q-6is3aiib6Zv69Wn", 2022-09-12T19:16:47.352 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.352 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.352 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.353 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_aXhKq1Q-6is3aiib6Zv69Wn", 2022-09-12T19:16:47.353 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.353 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.353 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.354 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.354 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.354 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.354 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.355 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.355 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.509791_291.1", 2022-09-12T19:16:47.355 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.355 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-31:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.31", 2022-09-12T19:16:47.356 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.509791403Z", 2022-09-12T19:16:47.356 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.356 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.356 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_aXhKq1Q-6is3aiib6Zv69Wn", 2022-09-12T19:16:47.357 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.357 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.357 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.357 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_aXhKq1Q-6is3aiib6Zv69Wn", 2022-09-12T19:16:47.358 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.359 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.359 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.359 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.359 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.360 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.360 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.360 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.360 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.545979_292.1", 2022-09-12T19:16:47.361 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.361 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-32:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.32", 2022-09-12T19:16:47.361 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.545979472Z", 2022-09-12T19:16:47.361 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.362 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.362 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UeviL3qodKD5NX6Gmu8wcLZ", 2022-09-12T19:16:47.362 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.362 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.363 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.363 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UeviL3qodKD5NX6Gmu8wcLZ", 2022-09-12T19:16:47.363 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.363 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.364 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.364 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.364 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.364 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.365 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.365 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.365 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.555815_293.1", 2022-09-12T19:16:47.365 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.366 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-32:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.32", 2022-09-12T19:16:47.366 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.555815933Z", 2022-09-12T19:16:47.366 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.367 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.367 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UeviL3qodKD5NX6Gmu8wcLZ", 2022-09-12T19:16:47.367 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.367 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.368 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.368 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_UeviL3qodKD5NX6Gmu8wcLZ", 2022-09-12T19:16:47.368 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.368 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.369 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.369 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.369 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.369 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.370 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.370 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.370 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.629617_294.1", 2022-09-12T19:16:47.370 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.371 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-33:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.33", 2022-09-12T19:16:47.371 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.629617979Z", 2022-09-12T19:16:47.371 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.371 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.372 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.372 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.372 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.372 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.373 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_9Dhxqv90pxoXFdD9Kil8AZS", 2022-09-12T19:16:47.373 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.373 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.374 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.374 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.374 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.375 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.375 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.375 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.376 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.632160_295.1", 2022-09-12T19:16:47.376 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.377 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-33:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.33", 2022-09-12T19:16:47.377 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.632160672Z", 2022-09-12T19:16:47.377 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.378 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.378 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.378 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.379 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.379 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.380 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_9Dhxqv90pxoXFdD9Kil8AZS", 2022-09-12T19:16:47.380 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.380 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.381 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.381 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.382 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.382 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.382 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.383 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.383 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.633025_296.1", 2022-09-12T19:16:47.383 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.384 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-33", 2022-09-12T19:16:47.384 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.633025749Z", 2022-09-12T19:16:47.384 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.384 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.385 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.385 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.385 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.386 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.386 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_duz73YpnwikZnC64B4KGaWm", 2022-09-12T19:16:47.386 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.386 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.387 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.387 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.387 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.387 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.388 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.388 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.388 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.635657_297.1", 2022-09-12T19:16:47.388 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.389 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-33", 2022-09-12T19:16:47.389 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.635657460Z", 2022-09-12T19:16:47.389 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.389 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.390 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.390 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.390 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.390 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.391 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_duz73YpnwikZnC64B4KGaWm", 2022-09-12T19:16:47.391 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.391 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.391 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.392 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.392 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.392 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.392 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.393 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.393 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.653881_298.1", 2022-09-12T19:16:47.393 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.393 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-34:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.34", 2022-09-12T19:16:47.394 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.653881272Z", 2022-09-12T19:16:47.394 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.394 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.395 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.395 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.395 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.395 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.396 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_T6BXjPTsg5YjV989eew_nUj", 2022-09-12T19:16:47.396 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.396 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.396 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.397 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.397 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.397 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.397 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.398 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.398 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.656302_299.1", 2022-09-12T19:16:47.398 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.398 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-34:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.34", 2022-09-12T19:16:47.399 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.656302045Z", 2022-09-12T19:16:47.399 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.399 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.399 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.400 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.400 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.400 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.400 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_T6BXjPTsg5YjV989eew_nUj", 2022-09-12T19:16:47.401 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.401 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.401 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.401 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.402 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.402 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.402 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.402 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.403 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.657143_300.1", 2022-09-12T19:16:47.403 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.403 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-34", 2022-09-12T19:16:47.403 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.657143981Z", 2022-09-12T19:16:47.404 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.404 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.404 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.404 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.405 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.405 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.405 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_zJMQJW7qDi486fSD79XXMMX", 2022-09-12T19:16:47.405 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.406 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.406 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.406 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.406 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.407 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.407 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.407 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.407 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.659579_301.1", 2022-09-12T19:16:47.408 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.408 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-34", 2022-09-12T19:16:47.408 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.659579967Z", 2022-09-12T19:16:47.408 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.409 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.409 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.409 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.409 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.410 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.410 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_zJMQJW7qDi486fSD79XXMMX", 2022-09-12T19:16:47.410 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.410 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.411 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.411 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.411 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.411 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.412 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.412 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.412 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.688010_302.1", 2022-09-12T19:16:47.413 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.413 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35", 2022-09-12T19:16:47.413 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.688010386Z", 2022-09-12T19:16:47.414 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.414 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.414 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.414 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.415 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.415 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.415 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_RT_qLkUYTVFuTZ0Mzyw3ra0", 2022-09-12T19:16:47.416 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.416 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.416 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.416 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.417 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.417 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.417 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.417 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.418 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.690504_303.1", 2022-09-12T19:16:47.418 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.418 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35", 2022-09-12T19:16:47.418 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.690504973Z", 2022-09-12T19:16:47.419 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.419 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.419 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.419 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.420 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.420 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.420 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_RT_qLkUYTVFuTZ0Mzyw3ra0", 2022-09-12T19:16:47.421 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.421 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.421 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.421 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.422 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.422 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.422 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.422 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.423 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.691394_304.1", 2022-09-12T19:16:47.423 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.423 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-35", 2022-09-12T19:16:47.423 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.691394342Z", 2022-09-12T19:16:47.424 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.424 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.424 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.424 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.425 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.425 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.425 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_E0ZY7pht4AH9rgoAN9wcr0i", 2022-09-12T19:16:47.425 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.426 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.426 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.426 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.427 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.427 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.427 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.427 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.428 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.693991_305.1", 2022-09-12T19:16:47.428 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.428 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-35", 2022-09-12T19:16:47.428 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.693991405Z", 2022-09-12T19:16:47.429 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.429 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.429 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.429 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.430 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.430 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.430 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_E0ZY7pht4AH9rgoAN9wcr0i", 2022-09-12T19:16:47.431 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.431 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.431 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.431 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.432 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.432 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.432 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.432 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.433 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.713639_306.1", 2022-09-12T19:16:47.433 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.433 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-33:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.33", 2022-09-12T19:16:47.433 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.713639843Z", 2022-09-12T19:16:47.434 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.434 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.434 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_9Dhxqv90pxoXFdD9Kil8AZS", 2022-09-12T19:16:47.434 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.435 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.435 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.435 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_9Dhxqv90pxoXFdD9Kil8AZS", 2022-09-12T19:16:47.435 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.436 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.436 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.436 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.436 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.437 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.437 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.437 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.437 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010024.716209_307.1", 2022-09-12T19:16:47.438 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.438 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-33:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.33", 2022-09-12T19:16:47.438 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:44.716209907Z", 2022-09-12T19:16:47.438 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.439 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.439 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_9Dhxqv90pxoXFdD9Kil8AZS", 2022-09-12T19:16:47.439 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.440 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.440 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.440 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_9Dhxqv90pxoXFdD9Kil8AZS", 2022-09-12T19:16:47.440 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.441 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.441 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.441 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.441 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.442 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.442 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.442 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.442 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010025.737428_308.1", 2022-09-12T19:16:47.443 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.443 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-34:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.34", 2022-09-12T19:16:47.443 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:45.737428711Z", 2022-09-12T19:16:47.443 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.444 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.444 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_T6BXjPTsg5YjV989eew_nUj", 2022-09-12T19:16:47.444 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.444 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.445 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.445 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_T6BXjPTsg5YjV989eew_nUj", 2022-09-12T19:16:47.445 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.445 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.446 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.446 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.446 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.447 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.447 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.447 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.448 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010025.741603_309.1", 2022-09-12T19:16:47.448 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.448 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-34:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.34", 2022-09-12T19:16:47.448 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:45.741603314Z", 2022-09-12T19:16:47.449 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.449 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.449 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_T6BXjPTsg5YjV989eew_nUj", 2022-09-12T19:16:47.449 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.450 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.450 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.450 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_T6BXjPTsg5YjV989eew_nUj", 2022-09-12T19:16:47.451 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.451 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.451 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.451 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.452 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.452 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.452 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.452 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.453 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.813360_310.1", 2022-09-12T19:16:47.453 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.453 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35", 2022-09-12T19:16:47.453 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.813360734Z", 2022-09-12T19:16:47.454 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.454 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.454 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_RT_qLkUYTVFuTZ0Mzyw3ra0", 2022-09-12T19:16:47.455 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.455 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.455 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.455 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_RT_qLkUYTVFuTZ0Mzyw3ra0", 2022-09-12T19:16:47.456 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.456 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.456 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.456 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.457 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.457 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.457 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.457 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.458 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.817978_311.1", 2022-09-12T19:16:47.458 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.458 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35", 2022-09-12T19:16:47.458 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.817978446Z", 2022-09-12T19:16:47.459 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.459 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.459 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_RT_qLkUYTVFuTZ0Mzyw3ra0", 2022-09-12T19:16:47.459 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.460 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.460 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.460 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_RT_qLkUYTVFuTZ0Mzyw3ra0", 2022-09-12T19:16:47.460 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.461 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.461 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.461 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.462 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.462 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.462 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.462 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.463 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.851414_312.1", 2022-09-12T19:16:47.463 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.463 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36", 2022-09-12T19:16:47.463 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.851414155Z", 2022-09-12T19:16:47.464 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.464 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.464 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.464 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.465 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.465 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.465 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_QWEAbq70_sTavFlIROKha_u", 2022-09-12T19:16:47.465 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.466 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.466 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.466 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.466 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.467 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.467 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.467 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.467 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.860552_313.1", 2022-09-12T19:16:47.468 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.468 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36", 2022-09-12T19:16:47.468 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.860552661Z", 2022-09-12T19:16:47.468 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.469 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.469 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.470 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.470 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.470 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.470 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_QWEAbq70_sTavFlIROKha_u", 2022-09-12T19:16:47.471 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.471 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.471 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.471 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.472 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.472 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.472 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.472 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.473 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.867783_314.1", 2022-09-12T19:16:47.473 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.473 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-36", 2022-09-12T19:16:47.473 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.867783999Z", 2022-09-12T19:16:47.474 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.474 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.474 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.474 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.475 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.475 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.475 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_-cyyq5stbLWU_c8ISOn28nE", 2022-09-12T19:16:47.475 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.476 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.476 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.476 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.476 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.477 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.477 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.477 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.477 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.879969_315.1", 2022-09-12T19:16:47.478 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.478 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-36", 2022-09-12T19:16:47.478 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.879969477Z", 2022-09-12T19:16:47.479 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.479 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.479 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.479 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.480 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.480 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.480 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_-cyyq5stbLWU_c8ISOn28nE", 2022-09-12T19:16:47.481 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.481 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.481 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.482 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.482 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.482 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.482 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.483 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.483 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.914600_316.1", 2022-09-12T19:16:47.483 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.483 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37", 2022-09-12T19:16:47.484 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.914600273Z", 2022-09-12T19:16:47.484 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.484 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.484 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.485 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.485 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.485 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.485 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_BzeIqN5hzm9Zg_oOIhlkjQJ", 2022-09-12T19:16:47.486 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.486 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.487 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.487 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.487 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.488 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.488 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.489 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.489 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.917584_317.1", 2022-09-12T19:16:47.490 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.490 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37", 2022-09-12T19:16:47.491 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.917584610Z", 2022-09-12T19:16:47.491 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.492 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.492 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.493 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.493 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.493 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.494 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_BzeIqN5hzm9Zg_oOIhlkjQJ", 2022-09-12T19:16:47.494 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.495 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.495 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.495 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.496 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.496 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.497 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.497 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.498 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.918659_318.1", 2022-09-12T19:16:47.498 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.498 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-37", 2022-09-12T19:16:47.499 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.918659379Z", 2022-09-12T19:16:47.499 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.499 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.500 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.500 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.501 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.501 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.502 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_WGVNJNvbu5mulQzf6ElKa4a", 2022-09-12T19:16:47.502 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.503 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.503 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.504 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.504 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.504 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.505 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.505 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.506 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.921237_319.1", 2022-09-12T19:16:47.506 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.507 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-37", 2022-09-12T19:16:47.507 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.921237396Z", 2022-09-12T19:16:47.508 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.508 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.509 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.509 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.510 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.510 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.511 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_WGVNJNvbu5mulQzf6ElKa4a", 2022-09-12T19:16:47.511 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.511 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.512 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.512 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.513 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.513 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.513 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.514 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.514 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.946958_320.1", 2022-09-12T19:16:47.514 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.515 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38", 2022-09-12T19:16:47.515 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.946958410Z", 2022-09-12T19:16:47.515 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.516 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.516 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.516 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.516 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.517 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.517 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_IWIU9MxoQv-Gr3d7VAe8c_0", 2022-09-12T19:16:47.517 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.518 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.518 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.519 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.519 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.519 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.519 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.520 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.520 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.949538_321.1", 2022-09-12T19:16:47.520 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.521 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38", 2022-09-12T19:16:47.521 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.949538394Z", 2022-09-12T19:16:47.522 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.522 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.522 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.523 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.523 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.523 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.524 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_IWIU9MxoQv-Gr3d7VAe8c_0", 2022-09-12T19:16:47.524 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.525 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.525 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.525 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.526 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.526 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.527 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.527 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.528 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.950529_322.1", 2022-09-12T19:16:47.528 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.529 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-38", 2022-09-12T19:16:47.529 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.950529882Z", 2022-09-12T19:16:47.530 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.530 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.530 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.531 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.531 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.532 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.532 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_QAVKfkG00yr1qxv971lPJ9X", 2022-09-12T19:16:47.532 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.533 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.533 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.534 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.534 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.534 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.535 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.535 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.536 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010026.953073_323.1", 2022-09-12T19:16:47.536 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.536 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-38", 2022-09-12T19:16:47.537 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:13:46.953073366Z", 2022-09-12T19:16:47.537 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.538 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.538 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.538 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.539 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.539 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.540 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_QAVKfkG00yr1qxv971lPJ9X", 2022-09-12T19:16:47.540 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.540 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.541 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.541 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.542 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.542 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.543 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.543 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.543 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010058.708837_324.1", 2022-09-12T19:16:47.544 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.544 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-39:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.39", 2022-09-12T19:16:47.545 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:14:18.708837670Z", 2022-09-12T19:16:47.545 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.545 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.546 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.546 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.547 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.547 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.547 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_8tqVsOF3WKb2me5I95G3qgR", 2022-09-12T19:16:47.548 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.548 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.549 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.549 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.549 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.550 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.550 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.551 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.551 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010058.713291_325.1", 2022-09-12T19:16:47.551 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.552 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-39:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.39", 2022-09-12T19:16:47.552 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:14:18.713291435Z", 2022-09-12T19:16:47.553 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.553 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.553 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.554 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.554 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.555 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.555 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_8tqVsOF3WKb2me5I95G3qgR", 2022-09-12T19:16:47.555 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.556 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.556 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.557 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.557 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.557 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.558 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.558 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.559 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010058.715054_326.1", 2022-09-12T19:16:47.559 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.559 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-39", 2022-09-12T19:16:47.560 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:14:18.715054633Z", 2022-09-12T19:16:47.560 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.561 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.561 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.561 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.562 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.562 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.563 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_xgZt1r-qtjp-K4E_Z1U_XYi", 2022-09-12T19:16:47.563 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.563 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.564 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.564 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.564 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.565 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.565 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.566 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.566 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010058.720459_327.1", 2022-09-12T19:16:47.566 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.567 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-39", 2022-09-12T19:16:47.567 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:14:18.720459880Z", 2022-09-12T19:16:47.568 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.568 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.568 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.569 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.569 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.569 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.570 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_xgZt1r-qtjp-K4E_Z1U_XYi", 2022-09-12T19:16:47.570 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.570 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.570 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.571 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.571 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.571 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.571 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.572 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.572 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010058.744939_328.1", 2022-09-12T19:16:47.572 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.572 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-40:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.40", 2022-09-12T19:16:47.573 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:14:18.744939676Z", 2022-09-12T19:16:47.573 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.573 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.573 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.574 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.574 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.574 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.574 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_XRz8ZhFtWNauZ5YqelhcHjG", 2022-09-12T19:16:47.575 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.575 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.575 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.575 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.576 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.576 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.576 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.576 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.577 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010058.753933_329.1", 2022-09-12T19:16:47.577 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.577 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-40:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.40", 2022-09-12T19:16:47.577 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:14:18.753933965Z", 2022-09-12T19:16:47.578 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.578 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.578 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.578 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.579 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.579 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.579 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_XRz8ZhFtWNauZ5YqelhcHjG", 2022-09-12T19:16:47.579 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.580 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.580 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.580 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.580 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.581 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.581 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.581 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.582 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010058.758572_330.1", 2022-09-12T19:16:47.582 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.582 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-40", 2022-09-12T19:16:47.582 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:14:18.758572726Z", 2022-09-12T19:16:47.583 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.583 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.583 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.583 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.584 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.584 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.584 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_4IJ7vbCVAqQL9xJtZhDAZ4r", 2022-09-12T19:16:47.584 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.585 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.585 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.585 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.585 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.586 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.586 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.586 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.586 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010058.761625_331.1", 2022-09-12T19:16:47.587 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.587 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-40", 2022-09-12T19:16:47.587 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:14:18.761625471Z", 2022-09-12T19:16:47.587 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.588 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.588 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.588 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.588 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.589 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.589 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_4IJ7vbCVAqQL9xJtZhDAZ4r", 2022-09-12T19:16:47.589 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.589 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.590 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.590 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.590 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.590 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.591 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.591 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.591 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010058.795251_332.1", 2022-09-12T19:16:47.591 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.592 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-41:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.41", 2022-09-12T19:16:47.592 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:14:18.795251326Z", 2022-09-12T19:16:47.592 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.593 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.593 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.593 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.593 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.594 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.594 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_WLs8_gJqx8-eJNrUvhbOESF", 2022-09-12T19:16:47.594 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.594 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.595 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.595 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.595 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.596 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.596 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.596 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.596 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010058.799161_333.1", 2022-09-12T19:16:47.597 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.597 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-41:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.41", 2022-09-12T19:16:47.597 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:14:18.799161992Z", 2022-09-12T19:16:47.597 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.598 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.598 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.598 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.598 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.599 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.599 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_WLs8_gJqx8-eJNrUvhbOESF", 2022-09-12T19:16:47.599 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.600 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.600 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.600 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.601 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.601 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.601 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.601 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.602 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010058.800547_334.1", 2022-09-12T19:16:47.602 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.602 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-41", 2022-09-12T19:16:47.603 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:14:18.800547915Z", 2022-09-12T19:16:47.603 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.603 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.604 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.604 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.604 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.605 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.605 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Rx20gRuO3XO_dd_rDNjdCZL", 2022-09-12T19:16:47.605 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.605 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.606 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.606 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.606 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.607 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.607 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.607 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.607 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010058.804299_335.1", 2022-09-12T19:16:47.608 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.608 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-41", 2022-09-12T19:16:47.608 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:14:18.804299896Z", 2022-09-12T19:16:47.609 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.609 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.609 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.610 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.610 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.611 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.611 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Rx20gRuO3XO_dd_rDNjdCZL", 2022-09-12T19:16:47.612 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.612 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.613 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.613 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.614 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.614 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.615 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.615 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.616 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008569.937424_38.1", 2022-09-12T19:16:47.616 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.617 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-1:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.1", 2022-09-12T19:16:47.617 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:29.937424694Z", 2022-09-12T19:16:47.618 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.618 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.619 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.619 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.620 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.620 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.621 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "__lpWFIiN4hfllNCufyv6oPF", 2022-09-12T19:16:47.621 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.622 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.622 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.623 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.623 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.623 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.624 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.624 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.625 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008569.948669_40.1", 2022-09-12T19:16:47.625 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.626 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-1:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.1", 2022-09-12T19:16:47.626 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:29.948669068Z", 2022-09-12T19:16:47.626 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.627 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.627 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.627 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.627 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.628 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.628 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "__lpWFIiN4hfllNCufyv6oPF", 2022-09-12T19:16:47.628 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.629 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.629 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.629 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.629 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.630 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.630 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.631 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.631 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008569.949794_41.1", 2022-09-12T19:16:47.631 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.632 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-1", 2022-09-12T19:16:47.632 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:29.949794341Z", 2022-09-12T19:16:47.632 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.632 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.633 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.633 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.633 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.634 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.634 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_724Zi_oQbv20Wa_KFzslGam", 2022-09-12T19:16:47.634 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.635 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.635 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.635 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.635 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.636 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.636 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.636 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.636 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008569.952619_42.1", 2022-09-12T19:16:47.637 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.637 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-1", 2022-09-12T19:16:47.637 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:29.952619357Z", 2022-09-12T19:16:47.638 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.638 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.638 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.638 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.639 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.639 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.639 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_724Zi_oQbv20Wa_KFzslGam", 2022-09-12T19:16:47.640 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.640 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.640 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.641 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.641 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.641 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.641 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.642 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.642 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008569.979576_43.1", 2022-09-12T19:16:47.642 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.642 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-2:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.2", 2022-09-12T19:16:47.643 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:29.979576716Z", 2022-09-12T19:16:47.643 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.643 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.643 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.644 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.644 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.644 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.644 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_yWZcOeCLp3AOZWE4KEf6DqY", 2022-09-12T19:16:47.645 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.645 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.645 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.645 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.646 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.646 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.646 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.646 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.647 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008569.982366_44.1", 2022-09-12T19:16:47.647 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.647 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-2:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.2", 2022-09-12T19:16:47.647 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:29.982366892Z", 2022-09-12T19:16:47.648 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.648 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.648 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.648 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.649 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.649 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.649 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_yWZcOeCLp3AOZWE4KEf6DqY", 2022-09-12T19:16:47.650 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.650 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.650 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.650 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.651 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.651 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.651 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.651 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.652 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008569.983351_45.1", 2022-09-12T19:16:47.652 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.652 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-2", 2022-09-12T19:16:47.652 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:29.983351998Z", 2022-09-12T19:16:47.653 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.653 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.653 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.653 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.654 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.654 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.654 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_JYImBEwDt-Z7UabuOPjifZb", 2022-09-12T19:16:47.655 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.655 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.655 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.655 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.656 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.656 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.656 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.656 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.657 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008569.985969_46.1", 2022-09-12T19:16:47.657 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.657 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-2", 2022-09-12T19:16:47.657 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:29.985969116Z", 2022-09-12T19:16:47.658 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.658 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.658 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.658 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.659 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.659 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.659 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_JYImBEwDt-Z7UabuOPjifZb", 2022-09-12T19:16:47.659 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.660 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.660 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.660 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.660 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.661 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.661 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.661 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.662 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008572.534688_47.1", 2022-09-12T19:16:47.662 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.662 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-3:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.3", 2022-09-12T19:16:47.662 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:32.534688656Z", 2022-09-12T19:16:47.663 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.663 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.663 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.663 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.664 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.664 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.664 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_2N-a-PDWYhR0LHw5JLpZR4c", 2022-09-12T19:16:47.665 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.665 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.665 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.665 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.665 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.666 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.666 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.666 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.667 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008572.537960_48.1", 2022-09-12T19:16:47.667 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.667 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-3:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.3", 2022-09-12T19:16:47.667 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:32.537960225Z", 2022-09-12T19:16:47.668 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.668 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.668 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.668 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.669 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.669 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.669 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_2N-a-PDWYhR0LHw5JLpZR4c", 2022-09-12T19:16:47.669 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.670 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.670 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.670 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.670 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.671 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.671 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.671 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.672 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008572.539055_49.1", 2022-09-12T19:16:47.672 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.672 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-3", 2022-09-12T19:16:47.672 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:32.539055121Z", 2022-09-12T19:16:47.673 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.673 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.673 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.673 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.674 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.674 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.674 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_84xzHNW5sbis6XdPCoJHPOk", 2022-09-12T19:16:47.674 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.675 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.675 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.675 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.675 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.676 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.676 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.676 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.676 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008572.542081_50.1", 2022-09-12T19:16:47.677 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.677 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-3", 2022-09-12T19:16:47.677 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:32.542081201Z", 2022-09-12T19:16:47.677 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.678 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.678 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.678 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.679 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.679 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.679 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_84xzHNW5sbis6XdPCoJHPOk", 2022-09-12T19:16:47.679 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.679 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.680 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.680 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.680 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.681 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.681 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.681 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.681 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008578.327919_51.1", 2022-09-12T19:16:47.682 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.682 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-4:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.4", 2022-09-12T19:16:47.682 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.327919700Z", 2022-09-12T19:16:47.682 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.683 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.683 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.683 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.683 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.684 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.684 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_k7Q_LlQwzVHeoxMKmCvxjHo", 2022-09-12T19:16:47.684 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.684 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.685 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.685 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.685 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.685 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.686 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.686 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.686 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008578.330590_52.1", 2022-09-12T19:16:47.686 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.687 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-4:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.4", 2022-09-12T19:16:47.687 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.330590808Z", 2022-09-12T19:16:47.687 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.687 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.688 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.688 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.688 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.688 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.689 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_k7Q_LlQwzVHeoxMKmCvxjHo", 2022-09-12T19:16:47.689 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.689 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.689 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.690 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.690 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.690 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.691 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.691 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.691 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008578.331599_53.1", 2022-09-12T19:16:47.691 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.692 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-4", 2022-09-12T19:16:47.692 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.331599255Z", 2022-09-12T19:16:47.692 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.692 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.693 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.693 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.693 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.693 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.694 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Uwy2BFzmIVF9oQbfNuX8-c2", 2022-09-12T19:16:47.694 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.694 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.695 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.695 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.695 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.695 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.696 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.696 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.696 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008578.334356_54.1", 2022-09-12T19:16:47.696 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.697 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-4", 2022-09-12T19:16:47.697 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.334356674Z", 2022-09-12T19:16:47.697 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.697 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.698 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.698 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.698 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.698 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.699 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Uwy2BFzmIVF9oQbfNuX8-c2", 2022-09-12T19:16:47.699 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.699 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.699 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.700 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.700 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.700 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.700 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.701 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.701 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008578.354410_55.1", 2022-09-12T19:16:47.701 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.702 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-5:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.5", 2022-09-12T19:16:47.702 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.354410340Z", 2022-09-12T19:16:47.702 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.702 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.703 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.703 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.703 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.704 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.704 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_DOCacF6mLBz24cosgaizYjM", 2022-09-12T19:16:47.704 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.705 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.705 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.705 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.706 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.706 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.706 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.706 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.707 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008578.357067_56.1", 2022-09-12T19:16:47.707 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.707 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-5:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.5", 2022-09-12T19:16:47.707 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.357067384Z", 2022-09-12T19:16:47.708 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.708 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.708 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.709 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.709 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.709 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.709 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_DOCacF6mLBz24cosgaizYjM", 2022-09-12T19:16:47.710 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.710 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.710 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.710 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.711 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.711 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.711 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.711 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.712 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008578.357978_57.1", 2022-09-12T19:16:47.712 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.712 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-5", 2022-09-12T19:16:47.713 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.357978793Z", 2022-09-12T19:16:47.713 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.713 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.713 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.714 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.714 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.714 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.715 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_PK2zvS8BU3_kRxhnsNsDeRa", 2022-09-12T19:16:47.715 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.715 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.715 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.716 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.716 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.716 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.716 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.717 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.717 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008578.360592_58.1", 2022-09-12T19:16:47.717 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.718 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-5", 2022-09-12T19:16:47.718 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.360592786Z", 2022-09-12T19:16:47.718 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.719 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.719 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.719 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.719 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.720 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.720 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_PK2zvS8BU3_kRxhnsNsDeRa", 2022-09-12T19:16:47.720 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.720 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.721 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.721 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.721 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.722 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.722 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.722 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.722 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008578.393761_59.1", 2022-09-12T19:16:47.723 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.723 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-6:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.6", 2022-09-12T19:16:47.723 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.393761186Z", 2022-09-12T19:16:47.723 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.724 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.724 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.724 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.724 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.725 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.725 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_zr7R-yTI4PYqVN0hq56_Jgr", 2022-09-12T19:16:47.725 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.725 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.726 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.726 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.726 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.727 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.727 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.727 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.727 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008578.396555_60.1", 2022-09-12T19:16:47.728 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.728 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-6:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.6", 2022-09-12T19:16:47.728 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.396555557Z", 2022-09-12T19:16:47.728 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.729 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.729 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.729 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.729 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.730 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.730 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_zr7R-yTI4PYqVN0hq56_Jgr", 2022-09-12T19:16:47.730 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.730 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.731 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.731 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.731 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.731 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.732 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.732 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.732 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008578.397479_61.1", 2022-09-12T19:16:47.733 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.733 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-6", 2022-09-12T19:16:47.733 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.397479637Z", 2022-09-12T19:16:47.734 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.734 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.735 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.735 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.735 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.736 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.736 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_9moMpukFqONyJQ22546lFFJ", 2022-09-12T19:16:47.736 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.737 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.737 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.738 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.738 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.738 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.739 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.739 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.739 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008578.400100_62.1", 2022-09-12T19:16:47.740 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.740 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-6", 2022-09-12T19:16:47.741 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:38.400100117Z", 2022-09-12T19:16:47.741 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.741 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.742 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.742 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.742 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.743 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.743 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_9moMpukFqONyJQ22546lFFJ", 2022-09-12T19:16:47.743 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.744 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.744 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.744 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.744 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.745 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.745 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.745 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.745 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.041323_63.1", 2022-09-12T19:16:47.746 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.746 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-7:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.7", 2022-09-12T19:16:47.746 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.041323973Z", 2022-09-12T19:16:47.746 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.747 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.747 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.747 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.747 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.748 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.748 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_14zIMAUeR0_uC-IGvsiZyFE", 2022-09-12T19:16:47.748 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.748 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.749 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.749 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.749 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.749 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.750 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.750 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.750 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.043790_64.1", 2022-09-12T19:16:47.750 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.751 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-7:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.7", 2022-09-12T19:16:47.751 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.043790091Z", 2022-09-12T19:16:47.751 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.752 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.752 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.752 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.752 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.753 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.753 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_14zIMAUeR0_uC-IGvsiZyFE", 2022-09-12T19:16:47.753 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.753 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.754 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.754 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.754 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.754 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.755 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.755 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.755 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.044741_65.1", 2022-09-12T19:16:47.755 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.756 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-7", 2022-09-12T19:16:47.756 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.044741403Z", 2022-09-12T19:16:47.756 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.756 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.757 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.757 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.757 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.757 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.758 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_40hjwX5ISF-woQ9EMT_esot", 2022-09-12T19:16:47.758 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.758 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.758 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.759 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.759 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.759 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.759 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.760 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.760 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.047046_66.1", 2022-09-12T19:16:47.760 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.760 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-7", 2022-09-12T19:16:47.761 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.047046794Z", 2022-09-12T19:16:47.761 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.761 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.762 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.762 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.762 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.762 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.763 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_40hjwX5ISF-woQ9EMT_esot", 2022-09-12T19:16:47.763 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.763 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.763 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.764 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.764 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.764 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.765 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.765 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.765 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.068972_67.1", 2022-09-12T19:16:47.765 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.766 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-8:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.8", 2022-09-12T19:16:47.766 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.068972049Z", 2022-09-12T19:16:47.766 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.766 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.767 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.767 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.767 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.767 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.768 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Cbx4jHMPdXce-GRW3F5bG0e", 2022-09-12T19:16:47.768 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.768 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.768 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.769 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.769 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.769 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.769 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.770 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.770 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.071464_68.1", 2022-09-12T19:16:47.770 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.771 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-8:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.8", 2022-09-12T19:16:47.771 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.071464065Z", 2022-09-12T19:16:47.771 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.771 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.772 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.772 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.772 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.772 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.773 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Cbx4jHMPdXce-GRW3F5bG0e", 2022-09-12T19:16:47.773 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.773 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.773 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.774 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.774 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.774 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.774 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.775 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.775 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.072340_69.1", 2022-09-12T19:16:47.775 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.775 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-8", 2022-09-12T19:16:47.776 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.072340420Z", 2022-09-12T19:16:47.776 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.776 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.776 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.777 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.777 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.777 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.777 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_mshfSar0kbJ0_j03QlyLl5l", 2022-09-12T19:16:47.778 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.778 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.778 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.778 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.779 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.779 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.779 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.780 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.780 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.074696_70.1", 2022-09-12T19:16:47.780 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.780 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-8", 2022-09-12T19:16:47.781 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.074696150Z", 2022-09-12T19:16:47.781 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.781 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.781 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.782 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.782 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.782 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.782 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_mshfSar0kbJ0_j03QlyLl5l", 2022-09-12T19:16:47.783 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.783 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.783 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.783 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.784 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.784 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.784 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.784 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.785 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.106134_71.1", 2022-09-12T19:16:47.785 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.785 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9", 2022-09-12T19:16:47.785 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.106134737Z", 2022-09-12T19:16:47.786 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.786 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.786 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.786 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.787 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.787 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.787 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_F8AkuhSt1lRvPdknmmo8vhx", 2022-09-12T19:16:47.787 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.788 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.788 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.788 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.788 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.789 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.789 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.789 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.789 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.110078_72.1", 2022-09-12T19:16:47.790 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.790 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9", 2022-09-12T19:16:47.790 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.110078653Z", 2022-09-12T19:16:47.790 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.791 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.791 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.791 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.791 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.792 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.792 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_F8AkuhSt1lRvPdknmmo8vhx", 2022-09-12T19:16:47.792 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.792 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.793 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.793 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.793 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.793 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.794 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.794 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.794 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.111034_73.1", 2022-09-12T19:16:47.795 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.795 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-9", 2022-09-12T19:16:47.795 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.111034634Z", 2022-09-12T19:16:47.795 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.796 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.796 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.796 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.796 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.797 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.797 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_dnyYyWYZArJtIne5MIt7vUD", 2022-09-12T19:16:47.797 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.797 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.798 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.798 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.798 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.798 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.799 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.799 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.799 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.116857_74.1", 2022-09-12T19:16:47.799 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.800 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-9", 2022-09-12T19:16:47.800 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.116857200Z", 2022-09-12T19:16:47.800 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.800 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.801 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:47.801 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:47.801 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.801 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.802 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_dnyYyWYZArJtIne5MIt7vUD", 2022-09-12T19:16:47.802 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.802 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.802 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.803 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.803 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.803 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.803 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.804 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.804 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.768561_75.1", 2022-09-12T19:16:47.804 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.804 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-7:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.7", 2022-09-12T19:16:47.805 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.768561107Z", 2022-09-12T19:16:47.805 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.805 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.805 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_14zIMAUeR0_uC-IGvsiZyFE", 2022-09-12T19:16:47.806 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.806 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.806 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.806 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_14zIMAUeR0_uC-IGvsiZyFE", 2022-09-12T19:16:47.807 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.807 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.807 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.807 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.808 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.808 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.808 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.808 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.809 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.771274_76.1", 2022-09-12T19:16:47.809 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.809 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-7:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.7", 2022-09-12T19:16:47.809 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.771274788Z", 2022-09-12T19:16:47.810 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.810 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.810 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_14zIMAUeR0_uC-IGvsiZyFE", 2022-09-12T19:16:47.810 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.811 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.811 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.811 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_14zIMAUeR0_uC-IGvsiZyFE", 2022-09-12T19:16:47.811 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.812 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.812 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.812 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.813 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.813 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.813 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.813 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.814 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.783884_77.1", 2022-09-12T19:16:47.814 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.814 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-7", 2022-09-12T19:16:47.814 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.783884039Z", 2022-09-12T19:16:47.815 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.815 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.815 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_40hjwX5ISF-woQ9EMT_esot", 2022-09-12T19:16:47.815 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.816 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.816 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.816 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_40hjwX5ISF-woQ9EMT_esot", 2022-09-12T19:16:47.817 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.817 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.817 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.817 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "remove" 2022-09-12T19:16:47.818 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.818 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.818 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.818 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.819 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.786242_78.1", 2022-09-12T19:16:47.819 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.819 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-7", 2022-09-12T19:16:47.819 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.786242763Z", 2022-09-12T19:16:47.820 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.820 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.820 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_40hjwX5ISF-woQ9EMT_esot", 2022-09-12T19:16:47.821 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.821 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.821 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.821 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_40hjwX5ISF-woQ9EMT_esot", 2022-09-12T19:16:47.822 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.822 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.822 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.822 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.823 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.823 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.823 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.823 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.824 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.817097_79.1", 2022-09-12T19:16:47.824 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.824 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-8:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.8", 2022-09-12T19:16:47.824 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.817097161Z", 2022-09-12T19:16:47.825 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.825 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.825 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Cbx4jHMPdXce-GRW3F5bG0e", 2022-09-12T19:16:47.826 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.826 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.826 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.827 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Cbx4jHMPdXce-GRW3F5bG0e", 2022-09-12T19:16:47.827 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.827 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.828 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.828 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.829 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.829 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.830 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.830 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.830 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.821207_80.1", 2022-09-12T19:16:47.831 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.831 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-8:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.8", 2022-09-12T19:16:47.831 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.821207606Z", 2022-09-12T19:16:47.832 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.832 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.832 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Cbx4jHMPdXce-GRW3F5bG0e", 2022-09-12T19:16:47.833 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.833 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.833 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.833 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Cbx4jHMPdXce-GRW3F5bG0e", 2022-09-12T19:16:47.834 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.834 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.834 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.835 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.835 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.835 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.835 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.836 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.836 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.833382_81.1", 2022-09-12T19:16:47.836 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.836 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-8", 2022-09-12T19:16:47.836 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.833382871Z", 2022-09-12T19:16:47.837 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.837 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.837 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_mshfSar0kbJ0_j03QlyLl5l", 2022-09-12T19:16:47.838 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.838 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.838 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.838 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_mshfSar0kbJ0_j03QlyLl5l", 2022-09-12T19:16:47.839 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.839 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.839 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.839 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "remove" 2022-09-12T19:16:47.840 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.840 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.840 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.841 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.841 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.835519_82.1", 2022-09-12T19:16:47.841 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.841 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-8", 2022-09-12T19:16:47.842 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.835519055Z", 2022-09-12T19:16:47.842 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.842 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.843 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_mshfSar0kbJ0_j03QlyLl5l", 2022-09-12T19:16:47.843 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.843 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.844 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.844 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_mshfSar0kbJ0_j03QlyLl5l", 2022-09-12T19:16:47.844 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.844 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.845 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.845 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.845 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.845 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.846 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.846 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.846 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.890492_83.1", 2022-09-12T19:16:47.846 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.847 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9", 2022-09-12T19:16:47.847 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.890492647Z", 2022-09-12T19:16:47.847 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.847 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.848 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_F8AkuhSt1lRvPdknmmo8vhx", 2022-09-12T19:16:47.848 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.848 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.848 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.849 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_F8AkuhSt1lRvPdknmmo8vhx", 2022-09-12T19:16:47.849 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.849 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.849 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.850 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:47.850 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.850 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.851 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.851 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.851 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.893091_84.1", 2022-09-12T19:16:47.851 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.852 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9", 2022-09-12T19:16:47.852 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.893091078Z", 2022-09-12T19:16:47.852 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.852 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.853 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_F8AkuhSt1lRvPdknmmo8vhx", 2022-09-12T19:16:47.853 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.853 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.853 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.854 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_F8AkuhSt1lRvPdknmmo8vhx", 2022-09-12T19:16:47.854 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.854 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.854 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.855 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.855 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.855 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.855 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.856 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.856 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.901637_85.1", 2022-09-12T19:16:47.856 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.856 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-9", 2022-09-12T19:16:47.857 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.901637143Z", 2022-09-12T19:16:47.857 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.857 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.857 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_dnyYyWYZArJtIne5MIt7vUD", 2022-09-12T19:16:47.858 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.858 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.858 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.858 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_dnyYyWYZArJtIne5MIt7vUD", 2022-09-12T19:16:47.859 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.859 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.859 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.859 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "remove" 2022-09-12T19:16:47.860 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.861 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.861 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.862 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:47.862 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663008580.904048_86.1", 2022-09-12T19:16:47.862 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:47.863 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-9", 2022-09-12T19:16:47.863 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048979Z", 2022-09-12T19:16:47.863 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:47.863 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:47.864 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_dnyYyWYZArJtIne5MIt7vUD", 2022-09-12T19:16:47.864 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:47.864 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.864 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:47.865 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_dnyYyWYZArJtIne5MIt7vUD", 2022-09-12T19:16:47.865 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 2 2022-09-12T19:16:47.865 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:47.865 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:47.866 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:47.866 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.866 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.867 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:47.867 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:16:47.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'c0f7a512-c940-4d1f-8adf-3484630f876b', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:47.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'c0f7a512-c940-4d1f-8adf-3484630f876b', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:47.870 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period c0f7a512-c940-4d1f-8adf-3484630f876b --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:47.925 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:47.925 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:47.963 INFO:teuthology.orchestra.run.smithi138.stdout:[ 2022-09-12T19:16:47.964 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:47.965 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008447.969948_52.1", 2022-09-12T19:16:47.965 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "user", 2022-09-12T19:16:47.966 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "rgw-multisite-test-user", 2022-09-12T19:16:47.967 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.969948859Z", 2022-09-12T19:16:47.968 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:47.968 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:47.968 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:47.969 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:47.969 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:47.970 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:47.971 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_X5YlnCJyC_iIiZ1qmhasQLd", 2022-09-12T19:16:47.972 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:47.973 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:47.973 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:47.974 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:47.975 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:47.975 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:47.976 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:47.978 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:47.978 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008447.976965_53.1", 2022-09-12T19:16:47.978 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "user", 2022-09-12T19:16:47.979 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "rgw-multisite-test-user", 2022-09-12T19:16:47.979 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965355Z", 2022-09-12T19:16:47.979 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:47.980 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:47.980 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:47.980 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:47.980 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:47.981 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:47.981 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_X5YlnCJyC_iIiZ1qmhasQLd", 2022-09-12T19:16:47.981 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:47.981 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:47.982 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:47.982 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:47.982 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:47.982 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:47.983 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:47.983 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:47.983 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008586.711103_57.1", 2022-09-12T19:16:47.983 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.984 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10", 2022-09-12T19:16:47.984 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.711103783Z", 2022-09-12T19:16:47.984 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:47.984 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:47.985 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:47.985 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:47.985 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:47.985 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:47.986 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_0WIw9i-PdtM0sq6xV05tENT", 2022-09-12T19:16:47.986 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:47.986 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:47.987 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:47.987 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:47.987 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:47.987 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:47.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:47.988 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:47.988 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008586.714487_58.1", 2022-09-12T19:16:47.988 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:47.989 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10", 2022-09-12T19:16:47.989 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.714487285Z", 2022-09-12T19:16:47.989 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:47.990 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:47.990 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:47.990 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:47.990 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:47.991 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:47.991 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_0WIw9i-PdtM0sq6xV05tENT", 2022-09-12T19:16:47.991 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:47.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:47.992 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:47.992 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:47.992 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:47.992 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:47.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:47.993 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:47.993 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008586.715813_59.1", 2022-09-12T19:16:47.993 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:47.994 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-10", 2022-09-12T19:16:47.994 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.715813657Z", 2022-09-12T19:16:47.994 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:47.995 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:47.995 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:47.995 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:47.995 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:47.995 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:47.996 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_nPF7oiiG3E2RTyZZOeDLhOv", 2022-09-12T19:16:47.996 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:47.996 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:47.996 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:47.997 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:47.997 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:47.997 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:47.998 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:47.998 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:47.998 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008586.718771_60.1", 2022-09-12T19:16:47.998 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:47.999 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-10", 2022-09-12T19:16:47.999 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.718771977Z", 2022-09-12T19:16:47.999 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:47.999 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.000 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.000 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.000 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.000 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.001 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_nPF7oiiG3E2RTyZZOeDLhOv", 2022-09-12T19:16:48.001 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.001 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.002 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.002 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.002 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.003 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.003 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008586.736648_61.1", 2022-09-12T19:16:48.003 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.004 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11", 2022-09-12T19:16:48.004 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.736648541Z", 2022-09-12T19:16:48.004 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.004 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.005 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.005 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.005 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.005 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.006 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_DXlcSXme1LlpozfB5A2QmS2", 2022-09-12T19:16:48.006 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.006 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.007 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.007 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.007 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.007 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.008 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.008 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008586.739749_62.1", 2022-09-12T19:16:48.008 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.009 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11", 2022-09-12T19:16:48.009 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.739749085Z", 2022-09-12T19:16:48.009 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.010 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.010 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.010 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.010 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.011 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.011 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_DXlcSXme1LlpozfB5A2QmS2", 2022-09-12T19:16:48.011 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.011 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.012 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.012 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.012 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.012 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.013 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.013 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.013 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008586.740926_63.1", 2022-09-12T19:16:48.013 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.014 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-11", 2022-09-12T19:16:48.014 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.740926183Z", 2022-09-12T19:16:48.014 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.014 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.015 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.015 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.016 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.016 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.016 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_QTYYWzCEPLTd7eDR1Si_D1k", 2022-09-12T19:16:48.016 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.017 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.017 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.017 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.018 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.018 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.018 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.019 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008586.744033_64.1", 2022-09-12T19:16:48.019 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.019 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-11", 2022-09-12T19:16:48.019 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.744033830Z", 2022-09-12T19:16:48.020 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.020 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.021 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.021 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.021 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.022 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_QTYYWzCEPLTd7eDR1Si_D1k", 2022-09-12T19:16:48.022 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.022 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.022 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.023 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.023 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.023 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.023 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.024 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.024 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008586.773460_65.1", 2022-09-12T19:16:48.024 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.024 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12", 2022-09-12T19:16:48.025 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.773460810Z", 2022-09-12T19:16:48.025 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.025 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.025 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.026 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.026 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.026 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.026 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_jMUPDDQJMus4ft3ku4UXAZe", 2022-09-12T19:16:48.027 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.027 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.027 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.028 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.028 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.028 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.028 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.029 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008586.776584_66.1", 2022-09-12T19:16:48.029 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.029 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12", 2022-09-12T19:16:48.030 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.776584217Z", 2022-09-12T19:16:48.030 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.030 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.030 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.031 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.031 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.031 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.031 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_jMUPDDQJMus4ft3ku4UXAZe", 2022-09-12T19:16:48.032 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.032 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.032 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.033 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.033 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.034 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008586.777646_67.1", 2022-09-12T19:16:48.034 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.034 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-12", 2022-09-12T19:16:48.034 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.777646572Z", 2022-09-12T19:16:48.035 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.035 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.035 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.035 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.036 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.036 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.036 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_dds05uMalANntddWdTKqimb", 2022-09-12T19:16:48.037 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.037 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.037 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.037 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.038 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.038 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.039 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008586.780479_68.1", 2022-09-12T19:16:48.039 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.039 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-12", 2022-09-12T19:16:48.039 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:46.780479129Z", 2022-09-12T19:16:48.040 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.040 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.040 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.041 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.041 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.041 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_dds05uMalANntddWdTKqimb", 2022-09-12T19:16:48.042 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.042 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.042 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.043 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.043 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.044 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008900.617421_102.1", 2022-09-12T19:16:48.044 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.044 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13", 2022-09-12T19:16:48.044 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.617421183Z", 2022-09-12T19:16:48.045 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.045 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.045 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.045 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.046 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.046 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.046 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_LDyblCFwL38h9zUN-Es9_0k", 2022-09-12T19:16:48.046 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.047 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.047 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.047 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.048 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.048 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008900.621771_103.1", 2022-09-12T19:16:48.049 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.049 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13", 2022-09-12T19:16:48.049 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.621771922Z", 2022-09-12T19:16:48.050 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.050 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.050 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.050 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.051 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.051 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_LDyblCFwL38h9zUN-Es9_0k", 2022-09-12T19:16:48.051 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.052 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.052 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.052 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.053 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.053 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008900.623473_104.1", 2022-09-12T19:16:48.054 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.054 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-13", 2022-09-12T19:16:48.054 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.623473623Z", 2022-09-12T19:16:48.054 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.055 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.055 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.055 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.056 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.056 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_HUc36aHifT_7iFz2OXdLjDc", 2022-09-12T19:16:48.056 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.057 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.057 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.057 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.057 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.058 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.058 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.058 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008900.629061_105.1", 2022-09-12T19:16:48.059 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.059 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-13", 2022-09-12T19:16:48.059 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.629061119Z", 2022-09-12T19:16:48.059 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.060 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.060 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.060 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.060 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.061 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_HUc36aHifT_7iFz2OXdLjDc", 2022-09-12T19:16:48.061 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.061 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.061 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.062 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.062 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.062 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.063 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.063 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008900.663664_106.1", 2022-09-12T19:16:48.063 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.064 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14", 2022-09-12T19:16:48.064 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.663664982Z", 2022-09-12T19:16:48.064 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.064 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.065 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.065 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.065 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.066 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_ULxp_YwqTEGv8QDweh9UlvR", 2022-09-12T19:16:48.066 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.067 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.067 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.067 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.068 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.068 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008900.667714_107.1", 2022-09-12T19:16:48.069 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.069 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14", 2022-09-12T19:16:48.069 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.667714167Z", 2022-09-12T19:16:48.069 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.070 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.070 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.070 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.071 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.071 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_ULxp_YwqTEGv8QDweh9UlvR", 2022-09-12T19:16:48.071 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.072 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.072 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.072 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.073 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.073 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008900.669743_108.1", 2022-09-12T19:16:48.074 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.074 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-14", 2022-09-12T19:16:48.074 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.669743466Z", 2022-09-12T19:16:48.075 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.075 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.075 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.075 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.076 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.076 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Zohm3vyJdVty1-BsovuZx7M", 2022-09-12T19:16:48.076 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.077 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.077 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.078 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.079 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.080 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008900.675413_109.1", 2022-09-12T19:16:48.080 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.080 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-14", 2022-09-12T19:16:48.080 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.675413478Z", 2022-09-12T19:16:48.081 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.081 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.081 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.081 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.082 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.082 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Zohm3vyJdVty1-BsovuZx7M", 2022-09-12T19:16:48.082 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.083 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.083 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.084 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.084 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.085 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008900.713502_110.1", 2022-09-12T19:16:48.085 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.085 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15", 2022-09-12T19:16:48.085 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.713502192Z", 2022-09-12T19:16:48.086 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.086 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.086 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.086 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.087 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.087 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_AKH72TwSsQVrOOdb7zgDFt1", 2022-09-12T19:16:48.088 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.088 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.088 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.089 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.089 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.090 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008900.716530_111.1", 2022-09-12T19:16:48.090 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.091 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15", 2022-09-12T19:16:48.091 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.716530511Z", 2022-09-12T19:16:48.091 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.091 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.092 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.092 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.092 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.093 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_AKH72TwSsQVrOOdb7zgDFt1", 2022-09-12T19:16:48.093 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.093 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.094 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.094 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.094 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.095 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008900.718433_112.1", 2022-09-12T19:16:48.095 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.095 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-15", 2022-09-12T19:16:48.096 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.718433663Z", 2022-09-12T19:16:48.096 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.096 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.096 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.097 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.097 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.097 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_i61YjmwdeL9p7lC_EIZUJTo", 2022-09-12T19:16:48.098 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.098 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.099 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.099 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.099 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.100 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.100 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008900.721341_113.1", 2022-09-12T19:16:48.100 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.100 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-15", 2022-09-12T19:16:48.101 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:55:00.721341013Z", 2022-09-12T19:16:48.101 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.101 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.101 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.102 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.102 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.102 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.102 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_i61YjmwdeL9p7lC_EIZUJTo", 2022-09-12T19:16:48.103 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.103 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.103 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.104 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.104 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.105 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.182484_132.1", 2022-09-12T19:16:48.105 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.106 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16", 2022-09-12T19:16:48.106 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.182484917Z", 2022-09-12T19:16:48.106 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.106 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.107 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.107 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.107 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.108 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_V99BgSq-hvAYBvkex60C8n2", 2022-09-12T19:16:48.108 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.109 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.109 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.109 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.109 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.110 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.186031_133.1", 2022-09-12T19:16:48.110 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.111 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16", 2022-09-12T19:16:48.111 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.186031455Z", 2022-09-12T19:16:48.111 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.111 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.112 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.112 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.112 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.113 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_V99BgSq-hvAYBvkex60C8n2", 2022-09-12T19:16:48.113 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.113 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.114 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.114 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.114 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.115 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.187394_134.1", 2022-09-12T19:16:48.115 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.116 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-16", 2022-09-12T19:16:48.116 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.187394124Z", 2022-09-12T19:16:48.116 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.117 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.117 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.117 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.118 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.118 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_ehigIfW992JSI46zHL04P5J", 2022-09-12T19:16:48.118 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.119 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.119 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.119 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.119 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.120 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.120 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.193049_135.1", 2022-09-12T19:16:48.121 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.121 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-16", 2022-09-12T19:16:48.121 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.193049111Z", 2022-09-12T19:16:48.121 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.122 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.122 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.122 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.123 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.123 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_ehigIfW992JSI46zHL04P5J", 2022-09-12T19:16:48.123 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.124 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.124 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.124 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.124 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.125 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.125 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.125 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.125 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.223026_136.1", 2022-09-12T19:16:48.126 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.126 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17", 2022-09-12T19:16:48.126 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.223026973Z", 2022-09-12T19:16:48.126 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.127 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.127 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.127 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.128 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.128 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_BCZEMlNNfcs_zua_I4p9TaW", 2022-09-12T19:16:48.128 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.129 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.129 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.129 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.129 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.130 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.130 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.130 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.131 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.226575_137.1", 2022-09-12T19:16:48.131 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.131 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17", 2022-09-12T19:16:48.132 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.226575054Z", 2022-09-12T19:16:48.132 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.132 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.132 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.133 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.133 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.134 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_BCZEMlNNfcs_zua_I4p9TaW", 2022-09-12T19:16:48.134 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.134 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.135 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.135 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.135 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.135 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.136 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.136 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.227798_138.1", 2022-09-12T19:16:48.136 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.136 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-17", 2022-09-12T19:16:48.137 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.227798137Z", 2022-09-12T19:16:48.137 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.137 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.137 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.138 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.138 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.138 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.138 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_i8wOFAvzgTHQCJOO8N6Bnam", 2022-09-12T19:16:48.139 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.139 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.139 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.139 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.140 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.140 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.140 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.140 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.141 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.232329_139.1", 2022-09-12T19:16:48.141 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.141 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-17", 2022-09-12T19:16:48.141 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.232329182Z", 2022-09-12T19:16:48.142 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.142 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.142 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.142 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.143 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.143 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.143 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_i8wOFAvzgTHQCJOO8N6Bnam", 2022-09-12T19:16:48.143 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.144 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.144 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.144 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.144 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.145 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.145 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.145 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.145 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.268820_140.1", 2022-09-12T19:16:48.146 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.146 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18", 2022-09-12T19:16:48.146 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.268820788Z", 2022-09-12T19:16:48.146 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.147 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.147 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.147 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.148 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.148 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_5GsV_zMTqFSGin4m-BDaD32", 2022-09-12T19:16:48.148 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.149 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.149 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.149 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.149 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.150 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.282707_141.1", 2022-09-12T19:16:48.150 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.151 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18", 2022-09-12T19:16:48.151 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.282707229Z", 2022-09-12T19:16:48.151 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.151 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.152 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.152 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.152 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.152 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.153 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_5GsV_zMTqFSGin4m-BDaD32", 2022-09-12T19:16:48.153 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.153 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.154 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.154 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.154 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.155 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.284810_142.1", 2022-09-12T19:16:48.155 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.156 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-18", 2022-09-12T19:16:48.156 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.284810891Z", 2022-09-12T19:16:48.156 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.156 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.157 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.157 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.157 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.158 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Ow-fYJ8C7sQ2JLuZ3C6E5Oy", 2022-09-12T19:16:48.158 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.158 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.158 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.159 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.159 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.159 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.160 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.287795_143.1", 2022-09-12T19:16:48.160 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.161 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-18", 2022-09-12T19:16:48.161 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.287795384Z", 2022-09-12T19:16:48.161 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.161 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.162 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.162 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.162 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.163 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Ow-fYJ8C7sQ2JLuZ3C6E5Oy", 2022-09-12T19:16:48.163 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.163 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.163 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.164 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.164 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.165 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.308137_144.1", 2022-09-12T19:16:48.165 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.165 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16", 2022-09-12T19:16:48.166 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.308137433Z", 2022-09-12T19:16:48.166 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.166 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.166 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_V99BgSq-hvAYBvkex60C8n2", 2022-09-12T19:16:48.167 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.167 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.167 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.167 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_V99BgSq-hvAYBvkex60C8n2", 2022-09-12T19:16:48.168 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.168 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.168 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.168 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.169 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.169 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.169 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.169 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.170 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.310988_145.1", 2022-09-12T19:16:48.170 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.170 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16", 2022-09-12T19:16:48.170 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.310988969Z", 2022-09-12T19:16:48.171 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.171 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.171 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_V99BgSq-hvAYBvkex60C8n2", 2022-09-12T19:16:48.171 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.172 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.172 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.172 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_V99BgSq-hvAYBvkex60C8n2", 2022-09-12T19:16:48.172 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.173 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.173 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.174 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.174 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.174 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.174 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.175 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.175 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.322795_146.1", 2022-09-12T19:16:48.175 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.175 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17", 2022-09-12T19:16:48.176 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.322795845Z", 2022-09-12T19:16:48.176 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.176 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.176 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_BCZEMlNNfcs_zua_I4p9TaW", 2022-09-12T19:16:48.177 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.177 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.177 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.177 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_BCZEMlNNfcs_zua_I4p9TaW", 2022-09-12T19:16:48.178 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.178 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.178 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.178 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.179 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.179 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.179 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.179 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.180 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.325825_147.1", 2022-09-12T19:16:48.180 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.180 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17", 2022-09-12T19:16:48.180 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.325825003Z", 2022-09-12T19:16:48.181 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.181 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.181 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_BCZEMlNNfcs_zua_I4p9TaW", 2022-09-12T19:16:48.181 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.182 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.182 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.182 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_BCZEMlNNfcs_zua_I4p9TaW", 2022-09-12T19:16:48.182 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.183 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.183 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.183 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.183 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.184 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.184 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.184 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.184 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.343478_148.1", 2022-09-12T19:16:48.185 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.185 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18", 2022-09-12T19:16:48.185 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.343478792Z", 2022-09-12T19:16:48.185 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.186 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.186 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_5GsV_zMTqFSGin4m-BDaD32", 2022-09-12T19:16:48.186 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.187 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.187 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.188 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_5GsV_zMTqFSGin4m-BDaD32", 2022-09-12T19:16:48.188 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.188 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.189 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.190 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.190 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.191 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.191 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.192 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.192 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009066.346375_149.1", 2022-09-12T19:16:48.193 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.193 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18", 2022-09-12T19:16:48.193 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:57:46.346375662Z", 2022-09-12T19:16:48.194 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.194 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.195 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_5GsV_zMTqFSGin4m-BDaD32", 2022-09-12T19:16:48.195 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.196 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.196 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.197 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_5GsV_zMTqFSGin4m-BDaD32", 2022-09-12T19:16:48.197 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.197 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.198 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.198 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.199 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.199 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.200 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.200 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.201 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009433.572798_179.1", 2022-09-12T19:16:48.201 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.202 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19", 2022-09-12T19:16:48.202 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:03:53.572798305Z", 2022-09-12T19:16:48.203 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.203 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.204 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.204 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.204 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.204 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.205 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_r98RnZbo3gbuHHD5iR9vaQX", 2022-09-12T19:16:48.205 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.205 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.206 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.206 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.206 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.206 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.207 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.207 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009433.576124_180.1", 2022-09-12T19:16:48.207 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.208 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19", 2022-09-12T19:16:48.208 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:03:53.576124288Z", 2022-09-12T19:16:48.208 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.208 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.209 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.209 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.209 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.209 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.210 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_r98RnZbo3gbuHHD5iR9vaQX", 2022-09-12T19:16:48.210 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.210 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.211 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.211 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.211 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.211 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.212 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.212 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.212 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009433.577370_181.1", 2022-09-12T19:16:48.212 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.213 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-19", 2022-09-12T19:16:48.213 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:03:53.577370053Z", 2022-09-12T19:16:48.213 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.213 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.214 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.214 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.214 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.214 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.215 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_FepK5riJM4xGtHwIBSHMDgN", 2022-09-12T19:16:48.215 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.215 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.215 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.216 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.216 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.216 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.216 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.217 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.217 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009433.580348_182.1", 2022-09-12T19:16:48.217 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.217 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-19", 2022-09-12T19:16:48.218 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:03:53.580348934Z", 2022-09-12T19:16:48.218 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.218 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.218 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.219 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.219 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.219 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.219 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_FepK5riJM4xGtHwIBSHMDgN", 2022-09-12T19:16:48.220 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.220 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.220 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.220 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.221 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.221 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.221 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.221 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.222 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009433.593338_183.1", 2022-09-12T19:16:48.222 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.222 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19", 2022-09-12T19:16:48.222 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:03:53.593338729Z", 2022-09-12T19:16:48.223 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.223 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.223 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_r98RnZbo3gbuHHD5iR9vaQX", 2022-09-12T19:16:48.223 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.224 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.224 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.224 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_r98RnZbo3gbuHHD5iR9vaQX", 2022-09-12T19:16:48.224 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.225 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.225 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.225 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.225 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.226 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.226 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.226 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.226 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009433.597955_184.1", 2022-09-12T19:16:48.227 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.227 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19", 2022-09-12T19:16:48.227 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:03:53.597955912Z", 2022-09-12T19:16:48.227 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.228 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.228 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_r98RnZbo3gbuHHD5iR9vaQX", 2022-09-12T19:16:48.228 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.228 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.229 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.229 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_r98RnZbo3gbuHHD5iR9vaQX", 2022-09-12T19:16:48.229 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.230 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.230 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.230 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.230 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.231 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.231 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.231 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.231 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009630.186894_193.1", 2022-09-12T19:16:48.232 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.232 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:16:48.232 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:07:10.186894533Z", 2022-09-12T19:16:48.232 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.233 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.233 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.233 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.233 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.234 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.234 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:48.234 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.234 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.235 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.235 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.235 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.235 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.236 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.236 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.236 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009630.192780_194.1", 2022-09-12T19:16:48.236 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.237 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:16:48.237 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:07:10.192780968Z", 2022-09-12T19:16:48.237 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.237 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.238 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.238 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.238 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.239 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:48.239 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.239 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.239 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.240 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.240 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.240 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.240 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.241 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.241 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009630.194692_195.1", 2022-09-12T19:16:48.241 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.241 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-20", 2022-09-12T19:16:48.242 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:07:10.194692069Z", 2022-09-12T19:16:48.242 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.242 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.242 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.243 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.243 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.243 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_vBgV2Ivb4f-6P44JJPXNdtC", 2022-09-12T19:16:48.244 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.244 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.244 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.245 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.245 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.245 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.245 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.246 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009630.200316_196.1", 2022-09-12T19:16:48.246 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.246 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-20", 2022-09-12T19:16:48.246 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:07:10.200316464Z", 2022-09-12T19:16:48.247 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.247 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.247 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.247 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.248 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.248 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.248 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_vBgV2Ivb4f-6P44JJPXNdtC", 2022-09-12T19:16:48.248 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.249 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.249 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.249 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.249 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.250 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.250 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.250 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.250 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009668.037998_204.1", 2022-09-12T19:16:48.251 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.251 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:16:48.251 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:07:48.037998619Z", 2022-09-12T19:16:48.251 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.252 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.252 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:48.252 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.252 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.253 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.253 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:48.253 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.254 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.254 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.254 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.254 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.255 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.255 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.255 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009668.041232_205.1", 2022-09-12T19:16:48.255 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.256 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:16:48.256 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:07:48.041232517Z", 2022-09-12T19:16:48.256 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.256 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.257 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:48.257 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.257 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.258 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.258 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:48.258 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.258 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.259 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.259 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.259 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.259 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.260 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.260 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009726.091934_207.1", 2022-09-12T19:16:48.260 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.261 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:16:48.261 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:08:46.091934751Z", 2022-09-12T19:16:48.261 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.261 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.262 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:48.262 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.262 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.262 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.263 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:48.263 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 3 2022-09-12T19:16:48.263 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.264 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.264 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.264 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.264 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.265 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.265 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.265 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009726.095686_208.1", 2022-09-12T19:16:48.265 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.266 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:16:48.266 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:08:46.095686282Z", 2022-09-12T19:16:48.266 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.267 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.267 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:48.267 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.267 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.268 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.268 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_voO3vuhI39MNLGF0NQ-EjAl", 2022-09-12T19:16:48.268 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 3 2022-09-12T19:16:48.268 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.269 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.269 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.269 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.269 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.270 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.270 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.270 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009842.925161_211.1", 2022-09-12T19:16:48.270 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.271 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21", 2022-09-12T19:16:48.271 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:42.925161228Z", 2022-09-12T19:16:48.271 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.271 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.272 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.272 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.272 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.273 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_wA9vY_PYbZdc0eUi2hSxrjB", 2022-09-12T19:16:48.273 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.273 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.273 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.274 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.274 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.274 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.274 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.275 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.275 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009842.930693_212.1", 2022-09-12T19:16:48.275 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.275 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21", 2022-09-12T19:16:48.276 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:42.930693175Z", 2022-09-12T19:16:48.276 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.276 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.276 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.277 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.277 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.277 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_wA9vY_PYbZdc0eUi2hSxrjB", 2022-09-12T19:16:48.278 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.278 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.278 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.279 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.279 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.279 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.279 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.280 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009842.932348_213.1", 2022-09-12T19:16:48.280 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.280 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-21", 2022-09-12T19:16:48.280 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:42.932348571Z", 2022-09-12T19:16:48.281 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.281 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.281 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.281 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.282 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.282 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_YhO7K8SUXb88GVnz95qTMjU", 2022-09-12T19:16:48.282 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.283 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.283 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.283 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.284 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.284 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.284 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.285 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009842.938156_214.1", 2022-09-12T19:16:48.285 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.285 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-21", 2022-09-12T19:16:48.285 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:42.938156918Z", 2022-09-12T19:16:48.286 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.286 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.286 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.286 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.287 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.287 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_YhO7K8SUXb88GVnz95qTMjU", 2022-09-12T19:16:48.287 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.288 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.288 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.288 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.288 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.289 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.289 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.289 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009842.960242_215.1", 2022-09-12T19:16:48.290 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.290 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22", 2022-09-12T19:16:48.290 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:42.960242821Z", 2022-09-12T19:16:48.290 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.291 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.291 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.291 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.292 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.292 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.292 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_jCE0vI8jcGzuttLr9LzBCWJ", 2022-09-12T19:16:48.292 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.293 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.293 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.293 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.293 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.294 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.294 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.294 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.294 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009842.965349_216.1", 2022-09-12T19:16:48.295 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.295 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22", 2022-09-12T19:16:48.295 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:42.965349366Z", 2022-09-12T19:16:48.295 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.296 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.296 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.296 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.296 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.297 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.297 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_jCE0vI8jcGzuttLr9LzBCWJ", 2022-09-12T19:16:48.297 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.297 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.298 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.298 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.298 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.299 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.299 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.299 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.299 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009842.966491_217.1", 2022-09-12T19:16:48.300 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.300 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-22", 2022-09-12T19:16:48.300 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:42.966491565Z", 2022-09-12T19:16:48.300 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.301 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.301 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.301 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.302 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.302 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.302 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_90qhmHoutPPaDynubPFZH9h", 2022-09-12T19:16:48.302 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.303 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.303 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.303 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.303 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.304 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.304 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.304 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.304 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009842.970703_218.1", 2022-09-12T19:16:48.305 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.305 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-22", 2022-09-12T19:16:48.305 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:42.970703166Z", 2022-09-12T19:16:48.305 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.306 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.306 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.306 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.307 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.307 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.307 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_90qhmHoutPPaDynubPFZH9h", 2022-09-12T19:16:48.307 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.308 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.308 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.308 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.308 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.309 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.309 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.309 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.310 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009843.006494_219.1", 2022-09-12T19:16:48.310 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.310 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23", 2022-09-12T19:16:48.310 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:43.006494173Z", 2022-09-12T19:16:48.311 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.311 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.311 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.312 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.312 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.312 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.312 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_8EYxPua6E_71y78ZWvmZh7v", 2022-09-12T19:16:48.313 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.313 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.313 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.314 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.314 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.314 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.314 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.315 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.315 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009843.009740_220.1", 2022-09-12T19:16:48.315 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.316 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23", 2022-09-12T19:16:48.316 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:43.009740592Z", 2022-09-12T19:16:48.316 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.316 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.317 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.317 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.317 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.318 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.318 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_8EYxPua6E_71y78ZWvmZh7v", 2022-09-12T19:16:48.318 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.318 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.319 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.319 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.319 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.320 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.320 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.320 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.320 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009843.011011_221.1", 2022-09-12T19:16:48.321 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.321 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-23", 2022-09-12T19:16:48.321 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:43.011011338Z", 2022-09-12T19:16:48.322 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.322 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.322 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.322 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.323 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.323 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.323 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_EWfzEo4mEzPqDzTBu2wloku", 2022-09-12T19:16:48.323 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.324 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.324 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.324 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.325 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.325 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.325 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.325 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.326 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009843.015119_222.1", 2022-09-12T19:16:48.326 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.326 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-23", 2022-09-12T19:16:48.326 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:43.015119585Z", 2022-09-12T19:16:48.327 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.327 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.327 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.327 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.328 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.328 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.328 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_EWfzEo4mEzPqDzTBu2wloku", 2022-09-12T19:16:48.328 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.329 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.329 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.329 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.329 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.330 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.330 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.330 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.330 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009843.040988_223.1", 2022-09-12T19:16:48.331 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.331 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21", 2022-09-12T19:16:48.331 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:43.040988054Z", 2022-09-12T19:16:48.331 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.332 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.332 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_wA9vY_PYbZdc0eUi2hSxrjB", 2022-09-12T19:16:48.332 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.332 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.333 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.333 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_wA9vY_PYbZdc0eUi2hSxrjB", 2022-09-12T19:16:48.333 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.333 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.334 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.334 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.334 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.335 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.335 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.335 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.335 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009843.045887_224.1", 2022-09-12T19:16:48.336 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.336 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21", 2022-09-12T19:16:48.336 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:43.045887351Z", 2022-09-12T19:16:48.336 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.337 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.337 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_wA9vY_PYbZdc0eUi2hSxrjB", 2022-09-12T19:16:48.337 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.337 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.337 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.338 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_wA9vY_PYbZdc0eUi2hSxrjB", 2022-09-12T19:16:48.338 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.338 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.339 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.339 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.339 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.339 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.340 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.340 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009843.057828_225.1", 2022-09-12T19:16:48.340 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.341 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22", 2022-09-12T19:16:48.341 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:43.057828971Z", 2022-09-12T19:16:48.341 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.341 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.342 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_jCE0vI8jcGzuttLr9LzBCWJ", 2022-09-12T19:16:48.342 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.342 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.342 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.343 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_jCE0vI8jcGzuttLr9LzBCWJ", 2022-09-12T19:16:48.343 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.343 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.343 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.344 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.344 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.344 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.345 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.345 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.345 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009843.060815_226.1", 2022-09-12T19:16:48.345 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.346 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22", 2022-09-12T19:16:48.346 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:43.060815675Z", 2022-09-12T19:16:48.346 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.346 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.347 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_jCE0vI8jcGzuttLr9LzBCWJ", 2022-09-12T19:16:48.347 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.347 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.347 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.348 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_jCE0vI8jcGzuttLr9LzBCWJ", 2022-09-12T19:16:48.348 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.348 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.348 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.349 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.349 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.349 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.349 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.350 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.350 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009843.077332_227.1", 2022-09-12T19:16:48.350 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.350 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23", 2022-09-12T19:16:48.351 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:43.077332048Z", 2022-09-12T19:16:48.351 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.351 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.352 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_8EYxPua6E_71y78ZWvmZh7v", 2022-09-12T19:16:48.352 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.352 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.352 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.353 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_8EYxPua6E_71y78ZWvmZh7v", 2022-09-12T19:16:48.353 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.353 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.353 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.354 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.354 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.354 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.354 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.355 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.355 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009843.080040_228.1", 2022-09-12T19:16:48.355 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.355 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23", 2022-09-12T19:16:48.356 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:10:43.080040677Z", 2022-09-12T19:16:48.356 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.356 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.356 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_8EYxPua6E_71y78ZWvmZh7v", 2022-09-12T19:16:48.357 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.357 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.357 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.357 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_8EYxPua6E_71y78ZWvmZh7v", 2022-09-12T19:16:48.358 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.358 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.358 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.358 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.359 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.359 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.359 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.359 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.360 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.213940_231.1", 2022-09-12T19:16:48.360 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.360 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:16:48.360 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.213940834Z", 2022-09-12T19:16:48.361 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.361 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.361 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.361 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.362 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.362 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.362 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:48.363 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.363 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.363 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.363 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.364 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.364 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.364 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.364 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.365 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.217329_232.1", 2022-09-12T19:16:48.365 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.365 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:16:48.365 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.217329420Z", 2022-09-12T19:16:48.366 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.366 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.366 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.366 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.367 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.367 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.367 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:48.367 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.368 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.368 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.368 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.369 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.369 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.369 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.370 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.218459_233.1", 2022-09-12T19:16:48.370 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.370 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-24", 2022-09-12T19:16:48.370 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.218459288Z", 2022-09-12T19:16:48.371 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.371 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.371 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.371 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.372 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.372 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.372 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_zt4J3xCzAzkFDmMYSv2ZhpD", 2022-09-12T19:16:48.372 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.373 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.373 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.373 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.374 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.374 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.374 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.375 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.221613_234.1", 2022-09-12T19:16:48.375 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.375 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-24", 2022-09-12T19:16:48.375 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.221613900Z", 2022-09-12T19:16:48.376 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.376 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.376 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.377 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.377 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.377 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_zt4J3xCzAzkFDmMYSv2ZhpD", 2022-09-12T19:16:48.378 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.378 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.378 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.378 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.379 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.379 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.379 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.379 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.380 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.242266_235.1", 2022-09-12T19:16:48.380 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.380 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:16:48.380 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.242266250Z", 2022-09-12T19:16:48.381 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.381 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.381 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.381 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.382 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.382 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.382 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:48.382 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.383 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.383 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.383 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.383 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.384 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.384 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.384 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.384 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.245374_236.1", 2022-09-12T19:16:48.385 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.385 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:16:48.385 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.245374907Z", 2022-09-12T19:16:48.385 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.386 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.386 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.386 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.386 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.387 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.387 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:48.387 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.387 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.388 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.388 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.388 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.389 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.389 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.389 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.389 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.246464_237.1", 2022-09-12T19:16:48.390 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.390 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-25", 2022-09-12T19:16:48.390 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.246464476Z", 2022-09-12T19:16:48.391 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.391 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.391 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.391 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.392 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.392 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.392 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_FuWLdq4G8Pitukb9rq9CEoZ", 2022-09-12T19:16:48.392 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.393 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.393 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.393 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.393 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.394 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.394 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.394 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.394 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.249409_238.1", 2022-09-12T19:16:48.395 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.395 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-25", 2022-09-12T19:16:48.395 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.249409215Z", 2022-09-12T19:16:48.395 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.396 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.396 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.396 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.396 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.397 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.397 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_FuWLdq4G8Pitukb9rq9CEoZ", 2022-09-12T19:16:48.397 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.397 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.398 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.398 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.398 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.398 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.399 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.399 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.399 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.282380_239.1", 2022-09-12T19:16:48.399 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.400 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:16:48.400 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.282380399Z", 2022-09-12T19:16:48.400 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.400 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.401 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.401 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.401 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.401 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.402 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:48.402 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.402 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.403 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.403 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.403 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.404 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.404 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.285471_240.1", 2022-09-12T19:16:48.404 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.404 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:16:48.405 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.285471857Z", 2022-09-12T19:16:48.405 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.405 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.405 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.406 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.406 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.406 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:48.407 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.407 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.407 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.408 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.408 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.409 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.286588_241.1", 2022-09-12T19:16:48.409 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.409 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-26", 2022-09-12T19:16:48.410 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.286588982Z", 2022-09-12T19:16:48.410 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.410 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.410 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.411 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.411 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.411 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_-j_vUqBTsANTAcA80szuZUK", 2022-09-12T19:16:48.412 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.412 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.412 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.413 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.413 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.413 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.414 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.289386_242.1", 2022-09-12T19:16:48.414 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.414 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-26", 2022-09-12T19:16:48.414 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.289386844Z", 2022-09-12T19:16:48.415 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.415 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.415 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.416 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.416 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.416 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_-j_vUqBTsANTAcA80szuZUK", 2022-09-12T19:16:48.417 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.417 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.417 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.418 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.418 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.419 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.309215_243.1", 2022-09-12T19:16:48.419 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.419 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:16:48.419 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.309215519Z", 2022-09-12T19:16:48.420 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.420 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.420 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:48.420 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.421 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.421 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.421 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:48.421 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.422 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.422 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.422 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.423 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.423 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.312069_244.1", 2022-09-12T19:16:48.424 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.424 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:16:48.424 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.312069106Z", 2022-09-12T19:16:48.424 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.425 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.425 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:48.425 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.425 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.426 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.426 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:48.426 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.426 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.427 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.427 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.427 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.427 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.428 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.428 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.428 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.323214_245.1", 2022-09-12T19:16:48.428 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.429 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:16:48.429 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.323214538Z", 2022-09-12T19:16:48.429 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.429 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.430 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:48.430 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.430 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.431 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:48.431 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 3 2022-09-12T19:16:48.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.431 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.432 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.432 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.432 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.433 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.433 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.334825_246.1", 2022-09-12T19:16:48.433 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.433 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:16:48.434 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.334825968Z", 2022-09-12T19:16:48.434 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.434 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.435 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:48.435 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.435 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.435 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "__iZFNN8HRgthIwCDpCjGdGQ", 2022-09-12T19:16:48.436 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 3 2022-09-12T19:16:48.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.436 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.436 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.437 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.437 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.438 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.350812_247.1", 2022-09-12T19:16:48.438 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.438 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:16:48.438 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.350812474Z", 2022-09-12T19:16:48.439 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.439 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.439 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:48.439 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.440 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.440 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.440 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:48.440 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.441 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.441 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.441 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.442 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.442 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.443 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.355013_248.1", 2022-09-12T19:16:48.443 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.443 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:16:48.443 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.355013585Z", 2022-09-12T19:16:48.444 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.444 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.444 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:48.445 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.445 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.445 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:48.446 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.446 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.446 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.446 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.447 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.447 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.447 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.448 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.369892_249.1", 2022-09-12T19:16:48.448 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.448 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:16:48.448 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.369892532Z", 2022-09-12T19:16:48.449 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.449 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.449 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:48.449 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.450 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.450 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.450 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:48.450 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 3 2022-09-12T19:16:48.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.451 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.451 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.451 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.452 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.452 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.452 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.374217_250.1", 2022-09-12T19:16:48.453 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.453 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:16:48.453 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.374217196Z", 2022-09-12T19:16:48.453 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.454 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.454 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:48.454 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.454 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.455 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.455 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UzmpuY5OTtYCOa-odzyyoM6", 2022-09-12T19:16:48.455 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 3 2022-09-12T19:16:48.455 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.456 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.456 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.456 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.456 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.457 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.457 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.457 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.394432_251.1", 2022-09-12T19:16:48.457 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.458 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:16:48.458 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.394432026Z", 2022-09-12T19:16:48.458 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.458 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.459 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:48.459 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.459 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.459 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.460 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:48.460 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.460 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.460 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.461 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.461 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.461 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.461 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.462 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.462 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.398632_252.1", 2022-09-12T19:16:48.462 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.462 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:16:48.463 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.398632136Z", 2022-09-12T19:16:48.463 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.463 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.463 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:48.464 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.464 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.464 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.464 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:48.465 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.465 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.465 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.466 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.466 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.466 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.466 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.467 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.418847_253.1", 2022-09-12T19:16:48.467 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.467 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:16:48.468 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.418847506Z", 2022-09-12T19:16:48.468 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.468 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.468 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:48.468 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.469 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.469 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:48.469 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 3 2022-09-12T19:16:48.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.470 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.470 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.471 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.471 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.471 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.472 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663009909.421418_254.1", 2022-09-12T19:16:48.472 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.472 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:16:48.472 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:11:49.421418488Z", 2022-09-12T19:16:48.473 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.473 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.473 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:48.473 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.474 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.474 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_gf9PWTqBnpTUtathOu6zY6k", 2022-09-12T19:16:48.474 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 3 2022-09-12T19:16:48.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.475 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.475 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.475 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.475 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.476 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.022864_258.1", 2022-09-12T19:16:48.477 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.477 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-27:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.27", 2022-09-12T19:16:48.477 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.022864695Z", 2022-09-12T19:16:48.477 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.478 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.478 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.478 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.479 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.479 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_4Ia0kBYSOXAPun0srTpHOgg", 2022-09-12T19:16:48.479 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.480 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.480 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.480 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.480 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.481 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.027319_259.1", 2022-09-12T19:16:48.481 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.482 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-27:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.27", 2022-09-12T19:16:48.482 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.027319061Z", 2022-09-12T19:16:48.482 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.482 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.483 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.483 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.483 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.484 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_4Ia0kBYSOXAPun0srTpHOgg", 2022-09-12T19:16:48.484 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.484 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.485 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.485 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.485 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.486 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.029529_260.1", 2022-09-12T19:16:48.486 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.486 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-27", 2022-09-12T19:16:48.487 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.029529115Z", 2022-09-12T19:16:48.487 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.487 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.487 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.488 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.488 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.488 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_hL6ubPjUnxDbFclj75F-9Yw", 2022-09-12T19:16:48.489 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.489 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.489 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.490 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.490 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.491 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.035074_261.1", 2022-09-12T19:16:48.491 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.491 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-27", 2022-09-12T19:16:48.491 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.035074135Z", 2022-09-12T19:16:48.492 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.492 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.492 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.492 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.493 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.493 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_hL6ubPjUnxDbFclj75F-9Yw", 2022-09-12T19:16:48.493 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.494 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.494 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.495 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.495 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.495 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.060437_262.1", 2022-09-12T19:16:48.496 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.496 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-28:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.28", 2022-09-12T19:16:48.496 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.060437057Z", 2022-09-12T19:16:48.496 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.497 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.497 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.497 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.498 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.498 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_F91L-y4ApRIftBlrzdVSKhB", 2022-09-12T19:16:48.498 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.499 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.499 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.499 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.500 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.500 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.500 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.500 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.067920_263.1", 2022-09-12T19:16:48.501 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.501 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-28:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.28", 2022-09-12T19:16:48.501 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.067920385Z", 2022-09-12T19:16:48.501 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.502 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.502 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.502 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.502 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.503 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.503 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_F91L-y4ApRIftBlrzdVSKhB", 2022-09-12T19:16:48.503 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.504 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.504 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.504 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.504 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.505 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.505 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.068850_264.1", 2022-09-12T19:16:48.506 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.506 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-28", 2022-09-12T19:16:48.506 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.068850124Z", 2022-09-12T19:16:48.506 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.507 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.507 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.507 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.508 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.508 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_4foA5qWzZ6DKNqXOUrqtpMi", 2022-09-12T19:16:48.508 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.508 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.509 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.509 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.509 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.509 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.510 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.074343_265.1", 2022-09-12T19:16:48.511 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.511 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-28", 2022-09-12T19:16:48.511 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.074343801Z", 2022-09-12T19:16:48.511 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.512 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.512 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.512 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.512 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.513 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.513 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_4foA5qWzZ6DKNqXOUrqtpMi", 2022-09-12T19:16:48.513 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.513 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.514 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.514 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.514 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.514 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.515 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.114860_266.1", 2022-09-12T19:16:48.515 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.516 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29", 2022-09-12T19:16:48.516 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.114860178Z", 2022-09-12T19:16:48.516 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.516 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.517 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.517 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.517 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.518 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.518 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_kHZQcZIbg231-V_zVbHphPU", 2022-09-12T19:16:48.518 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.519 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.519 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.519 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.520 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.520 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.520 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.520 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.119876_267.1", 2022-09-12T19:16:48.521 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.521 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29", 2022-09-12T19:16:48.521 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.119876089Z", 2022-09-12T19:16:48.521 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.522 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.522 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.522 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.523 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.523 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_kHZQcZIbg231-V_zVbHphPU", 2022-09-12T19:16:48.523 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.524 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.524 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.524 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.525 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.525 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.525 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.525 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.121725_268.1", 2022-09-12T19:16:48.526 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.526 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-29", 2022-09-12T19:16:48.526 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.121725517Z", 2022-09-12T19:16:48.527 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.527 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.527 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.527 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.528 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.528 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.528 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_2ti8fe4vkZVz-LOJO9SiYze", 2022-09-12T19:16:48.528 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.529 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.529 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.529 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.529 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.530 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.530 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.530 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.530 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.125711_269.1", 2022-09-12T19:16:48.531 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.531 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-29", 2022-09-12T19:16:48.531 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.125711914Z", 2022-09-12T19:16:48.531 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.532 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.532 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.532 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.532 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.533 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.533 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_2ti8fe4vkZVz-LOJO9SiYze", 2022-09-12T19:16:48.533 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.533 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.534 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.534 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.534 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.535 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.535 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.535 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.535 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.153919_270.1", 2022-09-12T19:16:48.536 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.536 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-27:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.27", 2022-09-12T19:16:48.536 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.153919920Z", 2022-09-12T19:16:48.537 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.537 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.537 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_4Ia0kBYSOXAPun0srTpHOgg", 2022-09-12T19:16:48.537 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.538 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.538 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.538 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_4Ia0kBYSOXAPun0srTpHOgg", 2022-09-12T19:16:48.538 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.539 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.539 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.539 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.540 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.540 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.540 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.540 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.541 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.156407_271.1", 2022-09-12T19:16:48.541 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.541 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-27:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.27", 2022-09-12T19:16:48.542 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.156407416Z", 2022-09-12T19:16:48.542 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.542 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.542 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_4Ia0kBYSOXAPun0srTpHOgg", 2022-09-12T19:16:48.543 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.543 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.543 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.544 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_4Ia0kBYSOXAPun0srTpHOgg", 2022-09-12T19:16:48.544 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.544 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.545 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.545 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.545 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.546 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.183174_272.1", 2022-09-12T19:16:48.546 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.547 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-28:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.28", 2022-09-12T19:16:48.547 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.183174956Z", 2022-09-12T19:16:48.547 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.547 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.548 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_F91L-y4ApRIftBlrzdVSKhB", 2022-09-12T19:16:48.548 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.548 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.549 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_F91L-y4ApRIftBlrzdVSKhB", 2022-09-12T19:16:48.549 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.549 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.550 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.550 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.550 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.550 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.551 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.186673_273.1", 2022-09-12T19:16:48.551 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.552 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-28:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.28", 2022-09-12T19:16:48.552 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.186673632Z", 2022-09-12T19:16:48.552 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.552 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.553 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_F91L-y4ApRIftBlrzdVSKhB", 2022-09-12T19:16:48.553 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.554 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.554 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_F91L-y4ApRIftBlrzdVSKhB", 2022-09-12T19:16:48.554 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.555 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.555 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.555 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.556 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.556 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.557 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.557 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.258924_274.1", 2022-09-12T19:16:48.557 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.558 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29", 2022-09-12T19:16:48.558 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.258924078Z", 2022-09-12T19:16:48.559 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.559 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.559 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_kHZQcZIbg231-V_zVbHphPU", 2022-09-12T19:16:48.560 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.561 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.561 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_kHZQcZIbg231-V_zVbHphPU", 2022-09-12T19:16:48.562 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.562 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.562 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.563 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.563 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.563 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.563 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.564 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.564 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.261587_275.1", 2022-09-12T19:16:48.564 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.564 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29", 2022-09-12T19:16:48.565 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.261587678Z", 2022-09-12T19:16:48.565 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.565 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.565 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_kHZQcZIbg231-V_zVbHphPU", 2022-09-12T19:16:48.566 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.566 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.566 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.566 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_kHZQcZIbg231-V_zVbHphPU", 2022-09-12T19:16:48.567 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.567 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.567 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.568 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.568 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.568 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.568 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.569 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.295442_276.1", 2022-09-12T19:16:48.569 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.569 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-30:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.30", 2022-09-12T19:16:48.569 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.295442828Z", 2022-09-12T19:16:48.570 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.570 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.570 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.570 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.571 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.571 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.572 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_xXGncRxFB5weAPxs1wnNxZq", 2022-09-12T19:16:48.572 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.573 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.573 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.574 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.574 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.574 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.574 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.575 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.297949_277.1", 2022-09-12T19:16:48.575 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.575 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-30:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.30", 2022-09-12T19:16:48.575 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.297949202Z", 2022-09-12T19:16:48.576 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.576 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.576 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.576 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.577 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.577 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_xXGncRxFB5weAPxs1wnNxZq", 2022-09-12T19:16:48.578 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.578 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.578 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.579 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.579 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.580 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.298820_278.1", 2022-09-12T19:16:48.580 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.580 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-30", 2022-09-12T19:16:48.580 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.298820021Z", 2022-09-12T19:16:48.581 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.581 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.581 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.581 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.582 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.582 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_o1lHEa9WwJ0A1-_nw8PZQAy", 2022-09-12T19:16:48.582 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.583 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.583 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.583 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.584 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.584 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.584 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.301106_279.1", 2022-09-12T19:16:48.585 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.585 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-30", 2022-09-12T19:16:48.585 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.301106622Z", 2022-09-12T19:16:48.585 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.586 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.586 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.586 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.587 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.587 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_o1lHEa9WwJ0A1-_nw8PZQAy", 2022-09-12T19:16:48.587 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.588 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.588 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.588 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.589 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.589 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.318892_280.1", 2022-09-12T19:16:48.590 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.590 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-31:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.31", 2022-09-12T19:16:48.590 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.318892577Z", 2022-09-12T19:16:48.590 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.591 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.591 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.591 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.592 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.592 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_aXhKq1Q-6is3aiib6Zv69Wn", 2022-09-12T19:16:48.592 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.593 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.593 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.593 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.593 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.594 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.321371_281.1", 2022-09-12T19:16:48.594 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.595 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-31:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.31", 2022-09-12T19:16:48.595 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.321371696Z", 2022-09-12T19:16:48.595 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.596 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.596 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.596 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.596 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.597 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.597 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_aXhKq1Q-6is3aiib6Zv69Wn", 2022-09-12T19:16:48.597 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.598 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.598 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.598 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.599 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.599 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.323664_282.1", 2022-09-12T19:16:48.600 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.600 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-31", 2022-09-12T19:16:48.600 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.323664287Z", 2022-09-12T19:16:48.600 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.601 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.601 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.601 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.602 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.602 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_ZXsXmNqWQzafZqU8AgaaFgE", 2022-09-12T19:16:48.602 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.602 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.603 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.603 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.603 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.603 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.604 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.332269_283.1", 2022-09-12T19:16:48.604 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.605 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-31", 2022-09-12T19:16:48.605 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.332269695Z", 2022-09-12T19:16:48.605 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.605 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.606 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.606 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.606 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.607 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_ZXsXmNqWQzafZqU8AgaaFgE", 2022-09-12T19:16:48.607 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.607 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.608 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.608 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.608 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.609 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.379861_284.1", 2022-09-12T19:16:48.609 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.609 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-32:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.32", 2022-09-12T19:16:48.610 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.379861593Z", 2022-09-12T19:16:48.610 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.610 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.610 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.611 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.611 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.611 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UeviL3qodKD5NX6Gmu8wcLZ", 2022-09-12T19:16:48.612 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.612 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.612 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.613 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.613 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.614 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.614 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.385256_285.1", 2022-09-12T19:16:48.614 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.614 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-32:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.32", 2022-09-12T19:16:48.615 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.385256336Z", 2022-09-12T19:16:48.615 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.615 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.615 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.616 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.616 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.616 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.617 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UeviL3qodKD5NX6Gmu8wcLZ", 2022-09-12T19:16:48.617 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.617 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.617 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.618 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.618 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.618 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.618 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.619 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.388164_286.1", 2022-09-12T19:16:48.619 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.619 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-32", 2022-09-12T19:16:48.620 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.388164607Z", 2022-09-12T19:16:48.620 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.620 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.620 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.621 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.621 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.621 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_sSvBc9sHjXg70XiY66Bq6ME", 2022-09-12T19:16:48.622 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.622 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.622 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.622 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.623 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.623 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.624 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.392572_287.1", 2022-09-12T19:16:48.624 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.624 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-32", 2022-09-12T19:16:48.624 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.392572009Z", 2022-09-12T19:16:48.625 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.625 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.625 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.625 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.626 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.626 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_sSvBc9sHjXg70XiY66Bq6ME", 2022-09-12T19:16:48.627 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.627 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.627 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.628 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.628 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.629 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.426848_288.1", 2022-09-12T19:16:48.629 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.629 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-30:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.30", 2022-09-12T19:16:48.629 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.426848512Z", 2022-09-12T19:16:48.630 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.630 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.630 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_xXGncRxFB5weAPxs1wnNxZq", 2022-09-12T19:16:48.630 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.631 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.631 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_xXGncRxFB5weAPxs1wnNxZq", 2022-09-12T19:16:48.631 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.632 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.632 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.632 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.633 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.633 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.431873_289.1", 2022-09-12T19:16:48.634 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.634 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-30:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.30", 2022-09-12T19:16:48.634 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.431873816Z", 2022-09-12T19:16:48.634 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.635 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.635 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_xXGncRxFB5weAPxs1wnNxZq", 2022-09-12T19:16:48.635 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.636 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.636 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_xXGncRxFB5weAPxs1wnNxZq", 2022-09-12T19:16:48.636 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.637 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.637 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.637 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.637 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.638 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.505940_290.1", 2022-09-12T19:16:48.638 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.639 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-31:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.31", 2022-09-12T19:16:48.639 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.505940792Z", 2022-09-12T19:16:48.639 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.639 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.640 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_aXhKq1Q-6is3aiib6Zv69Wn", 2022-09-12T19:16:48.640 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.640 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.641 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_aXhKq1Q-6is3aiib6Zv69Wn", 2022-09-12T19:16:48.641 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.641 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.642 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.642 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.642 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.643 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.509791_291.1", 2022-09-12T19:16:48.643 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.643 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-31:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.31", 2022-09-12T19:16:48.644 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.509791403Z", 2022-09-12T19:16:48.644 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.644 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.644 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_aXhKq1Q-6is3aiib6Zv69Wn", 2022-09-12T19:16:48.645 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.645 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.645 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_aXhKq1Q-6is3aiib6Zv69Wn", 2022-09-12T19:16:48.646 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.646 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.647 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.647 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.647 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.648 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.545979_292.1", 2022-09-12T19:16:48.648 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.648 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-32:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.32", 2022-09-12T19:16:48.648 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.545979472Z", 2022-09-12T19:16:48.649 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.649 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.649 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UeviL3qodKD5NX6Gmu8wcLZ", 2022-09-12T19:16:48.649 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.650 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.650 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UeviL3qodKD5NX6Gmu8wcLZ", 2022-09-12T19:16:48.650 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.651 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.651 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.652 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.652 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.653 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.555815_293.1", 2022-09-12T19:16:48.653 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.653 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-32:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.32", 2022-09-12T19:16:48.653 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.555815933Z", 2022-09-12T19:16:48.654 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.654 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.654 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UeviL3qodKD5NX6Gmu8wcLZ", 2022-09-12T19:16:48.654 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.655 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.655 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_UeviL3qodKD5NX6Gmu8wcLZ", 2022-09-12T19:16:48.655 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.656 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.656 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.656 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.657 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.657 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.629617_294.1", 2022-09-12T19:16:48.658 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.658 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-33:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.33", 2022-09-12T19:16:48.658 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.629617979Z", 2022-09-12T19:16:48.658 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.659 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.659 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.659 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.660 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.660 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_9Dhxqv90pxoXFdD9Kil8AZS", 2022-09-12T19:16:48.660 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.661 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.661 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.661 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.661 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.662 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.632160_295.1", 2022-09-12T19:16:48.662 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.663 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-33:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.33", 2022-09-12T19:16:48.663 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.632160672Z", 2022-09-12T19:16:48.663 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.664 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.664 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.664 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.665 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.665 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_9Dhxqv90pxoXFdD9Kil8AZS", 2022-09-12T19:16:48.665 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.666 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.666 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.666 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.667 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.667 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.633025_296.1", 2022-09-12T19:16:48.668 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.668 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-33", 2022-09-12T19:16:48.668 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.633025749Z", 2022-09-12T19:16:48.668 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.669 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.669 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.669 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.670 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.670 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_duz73YpnwikZnC64B4KGaWm", 2022-09-12T19:16:48.670 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.671 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.671 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.671 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.672 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.672 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.635657_297.1", 2022-09-12T19:16:48.673 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.673 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-33", 2022-09-12T19:16:48.673 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.635657460Z", 2022-09-12T19:16:48.673 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.674 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.674 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.674 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.675 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.675 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_duz73YpnwikZnC64B4KGaWm", 2022-09-12T19:16:48.675 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.676 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.676 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.676 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.676 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.677 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.653881_298.1", 2022-09-12T19:16:48.677 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.678 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-34:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.34", 2022-09-12T19:16:48.678 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.653881272Z", 2022-09-12T19:16:48.678 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.679 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.679 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.679 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.680 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.680 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_T6BXjPTsg5YjV989eew_nUj", 2022-09-12T19:16:48.680 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.680 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.681 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.681 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.681 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.682 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.656302_299.1", 2022-09-12T19:16:48.682 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.683 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-34:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.34", 2022-09-12T19:16:48.683 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.656302045Z", 2022-09-12T19:16:48.683 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.683 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.684 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.684 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.685 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.685 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_T6BXjPTsg5YjV989eew_nUj", 2022-09-12T19:16:48.686 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.686 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.687 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.687 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.688 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.689 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.657143_300.1", 2022-09-12T19:16:48.689 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.690 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-34", 2022-09-12T19:16:48.690 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.657143981Z", 2022-09-12T19:16:48.691 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.691 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.691 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.692 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.693 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.693 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_zJMQJW7qDi486fSD79XXMMX", 2022-09-12T19:16:48.693 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.694 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.695 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.695 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.695 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.696 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.659579_301.1", 2022-09-12T19:16:48.697 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.697 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-34", 2022-09-12T19:16:48.697 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.659579967Z", 2022-09-12T19:16:48.697 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.698 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.698 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.698 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.699 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.699 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_zJMQJW7qDi486fSD79XXMMX", 2022-09-12T19:16:48.699 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.700 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.700 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.700 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.700 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.701 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.688010_302.1", 2022-09-12T19:16:48.701 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.702 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35", 2022-09-12T19:16:48.702 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.688010386Z", 2022-09-12T19:16:48.702 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.702 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.703 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.703 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.703 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.704 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_RT_qLkUYTVFuTZ0Mzyw3ra0", 2022-09-12T19:16:48.704 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.705 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.705 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.705 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.705 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.706 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.690504_303.1", 2022-09-12T19:16:48.706 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.707 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35", 2022-09-12T19:16:48.707 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.690504973Z", 2022-09-12T19:16:48.707 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.707 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.708 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.708 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.708 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.709 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_RT_qLkUYTVFuTZ0Mzyw3ra0", 2022-09-12T19:16:48.709 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.709 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.710 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.710 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.710 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.711 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.691394_304.1", 2022-09-12T19:16:48.711 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.711 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-35", 2022-09-12T19:16:48.712 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.691394342Z", 2022-09-12T19:16:48.712 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.712 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.712 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.713 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.713 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.713 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_E0ZY7pht4AH9rgoAN9wcr0i", 2022-09-12T19:16:48.714 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.714 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.715 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.715 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.715 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.716 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.693991_305.1", 2022-09-12T19:16:48.716 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.716 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-35", 2022-09-12T19:16:48.717 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.693991405Z", 2022-09-12T19:16:48.717 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.717 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.717 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.718 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.718 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.718 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.718 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_E0ZY7pht4AH9rgoAN9wcr0i", 2022-09-12T19:16:48.719 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.719 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.719 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.719 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.720 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.720 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.721 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.713639_306.1", 2022-09-12T19:16:48.721 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.721 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-33:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.33", 2022-09-12T19:16:48.721 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.713639843Z", 2022-09-12T19:16:48.722 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.722 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.722 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_9Dhxqv90pxoXFdD9Kil8AZS", 2022-09-12T19:16:48.722 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.723 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.723 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.723 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_9Dhxqv90pxoXFdD9Kil8AZS", 2022-09-12T19:16:48.724 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.724 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.724 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.725 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.725 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.726 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010024.716209_307.1", 2022-09-12T19:16:48.726 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.726 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-33:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.33", 2022-09-12T19:16:48.726 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:44.716209907Z", 2022-09-12T19:16:48.727 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.727 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.727 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_9Dhxqv90pxoXFdD9Kil8AZS", 2022-09-12T19:16:48.727 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.728 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.728 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_9Dhxqv90pxoXFdD9Kil8AZS", 2022-09-12T19:16:48.728 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.729 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.729 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.730 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.730 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.731 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010025.737428_308.1", 2022-09-12T19:16:48.731 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.731 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-34:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.34", 2022-09-12T19:16:48.731 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:45.737428711Z", 2022-09-12T19:16:48.732 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.732 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.732 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_T6BXjPTsg5YjV989eew_nUj", 2022-09-12T19:16:48.732 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.733 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.733 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_T6BXjPTsg5YjV989eew_nUj", 2022-09-12T19:16:48.733 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.734 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.734 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.734 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.735 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.735 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.735 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010025.741603_309.1", 2022-09-12T19:16:48.736 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.736 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-34:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.34", 2022-09-12T19:16:48.736 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:45.741603314Z", 2022-09-12T19:16:48.736 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.737 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.737 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_T6BXjPTsg5YjV989eew_nUj", 2022-09-12T19:16:48.737 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.738 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.738 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_T6BXjPTsg5YjV989eew_nUj", 2022-09-12T19:16:48.738 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.739 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.739 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.739 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.739 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.740 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.813360_310.1", 2022-09-12T19:16:48.740 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.741 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35", 2022-09-12T19:16:48.741 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.813360734Z", 2022-09-12T19:16:48.741 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.741 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.742 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_RT_qLkUYTVFuTZ0Mzyw3ra0", 2022-09-12T19:16:48.742 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.742 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.743 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_RT_qLkUYTVFuTZ0Mzyw3ra0", 2022-09-12T19:16:48.743 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.743 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.744 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.744 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.744 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.745 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.817978_311.1", 2022-09-12T19:16:48.745 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.745 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35", 2022-09-12T19:16:48.746 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.817978446Z", 2022-09-12T19:16:48.746 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.746 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.746 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_RT_qLkUYTVFuTZ0Mzyw3ra0", 2022-09-12T19:16:48.747 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.747 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.747 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_RT_qLkUYTVFuTZ0Mzyw3ra0", 2022-09-12T19:16:48.748 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:48.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.748 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.748 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.749 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.749 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.750 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.750 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.851414_312.1", 2022-09-12T19:16:48.750 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.750 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36", 2022-09-12T19:16:48.751 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.851414155Z", 2022-09-12T19:16:48.751 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.751 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.751 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.752 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.752 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.752 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.752 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_QWEAbq70_sTavFlIROKha_u", 2022-09-12T19:16:48.753 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.753 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.753 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.753 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.753 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.754 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.754 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.754 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.755 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.860552_313.1", 2022-09-12T19:16:48.755 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.755 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36", 2022-09-12T19:16:48.755 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.860552661Z", 2022-09-12T19:16:48.756 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.756 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.756 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.756 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.757 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.757 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.757 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_QWEAbq70_sTavFlIROKha_u", 2022-09-12T19:16:48.757 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.758 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.758 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.758 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.758 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.759 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.759 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.759 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.759 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.867783_314.1", 2022-09-12T19:16:48.760 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.760 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-36", 2022-09-12T19:16:48.760 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.867783999Z", 2022-09-12T19:16:48.760 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.761 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.761 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.761 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.761 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.762 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.762 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_-cyyq5stbLWU_c8ISOn28nE", 2022-09-12T19:16:48.762 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.762 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.763 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.763 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.763 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.764 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.764 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.764 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.764 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.879969_315.1", 2022-09-12T19:16:48.765 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.765 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-36", 2022-09-12T19:16:48.765 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.879969477Z", 2022-09-12T19:16:48.765 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.766 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.766 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.766 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.766 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.767 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.767 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_-cyyq5stbLWU_c8ISOn28nE", 2022-09-12T19:16:48.767 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.767 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.768 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.768 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.768 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.768 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.769 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.769 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.769 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.914600_316.1", 2022-09-12T19:16:48.769 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.770 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37", 2022-09-12T19:16:48.770 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.914600273Z", 2022-09-12T19:16:48.770 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.770 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.771 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.771 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.771 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.771 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.772 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_BzeIqN5hzm9Zg_oOIhlkjQJ", 2022-09-12T19:16:48.772 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.772 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.772 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.773 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.773 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.773 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.773 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.774 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.774 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.917584_317.1", 2022-09-12T19:16:48.774 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.774 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37", 2022-09-12T19:16:48.775 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.917584610Z", 2022-09-12T19:16:48.775 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.775 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.776 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.776 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.776 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.776 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.777 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_BzeIqN5hzm9Zg_oOIhlkjQJ", 2022-09-12T19:16:48.777 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.777 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.777 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.778 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.778 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.778 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.778 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.779 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.779 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.918659_318.1", 2022-09-12T19:16:48.779 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.780 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-37", 2022-09-12T19:16:48.780 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.918659379Z", 2022-09-12T19:16:48.780 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.780 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.781 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.781 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.781 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.781 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.782 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_WGVNJNvbu5mulQzf6ElKa4a", 2022-09-12T19:16:48.782 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.782 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.782 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.783 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.783 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.783 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.783 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.784 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.784 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.921237_319.1", 2022-09-12T19:16:48.784 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.784 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-37", 2022-09-12T19:16:48.785 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.921237396Z", 2022-09-12T19:16:48.785 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.785 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.785 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.786 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.786 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.786 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.786 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_WGVNJNvbu5mulQzf6ElKa4a", 2022-09-12T19:16:48.787 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.787 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.787 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.787 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.788 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.788 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.788 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.788 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.789 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.946958_320.1", 2022-09-12T19:16:48.789 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.789 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38", 2022-09-12T19:16:48.789 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.946958410Z", 2022-09-12T19:16:48.790 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.790 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.790 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.790 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.791 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.791 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.791 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_IWIU9MxoQv-Gr3d7VAe8c_0", 2022-09-12T19:16:48.791 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.792 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.792 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.792 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.792 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.793 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.793 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.793 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.793 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.949538_321.1", 2022-09-12T19:16:48.794 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.794 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38", 2022-09-12T19:16:48.794 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.949538394Z", 2022-09-12T19:16:48.794 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.795 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.795 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.795 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.795 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.796 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.796 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_IWIU9MxoQv-Gr3d7VAe8c_0", 2022-09-12T19:16:48.796 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.796 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.797 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.797 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.797 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.797 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.798 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.798 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.798 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.950529_322.1", 2022-09-12T19:16:48.798 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.799 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-38", 2022-09-12T19:16:48.799 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.950529882Z", 2022-09-12T19:16:48.799 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.799 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.800 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.800 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.800 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.800 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.800 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_QAVKfkG00yr1qxv971lPJ9X", 2022-09-12T19:16:48.801 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.801 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.801 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.801 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.802 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.802 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.802 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.803 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.803 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010026.953073_323.1", 2022-09-12T19:16:48.803 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.803 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-38", 2022-09-12T19:16:48.804 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:13:46.953073366Z", 2022-09-12T19:16:48.804 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.804 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.804 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.804 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.805 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.805 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.805 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_QAVKfkG00yr1qxv971lPJ9X", 2022-09-12T19:16:48.805 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.806 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.806 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.806 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.807 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.807 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.807 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.807 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.807 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010058.708837_324.1", 2022-09-12T19:16:48.808 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.808 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-39:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.39", 2022-09-12T19:16:48.808 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.708837670Z", 2022-09-12T19:16:48.809 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.809 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.809 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.809 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.810 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.810 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.810 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_8tqVsOF3WKb2me5I95G3qgR", 2022-09-12T19:16:48.810 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.811 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.811 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.811 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.811 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.811 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.812 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.812 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.812 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010058.713291_325.1", 2022-09-12T19:16:48.812 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.813 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-39:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.39", 2022-09-12T19:16:48.813 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.713291435Z", 2022-09-12T19:16:48.813 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.814 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.814 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.814 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.814 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.815 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.815 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_8tqVsOF3WKb2me5I95G3qgR", 2022-09-12T19:16:48.815 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.815 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.816 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.816 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.816 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.816 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.817 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010058.715054_326.1", 2022-09-12T19:16:48.817 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.818 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-39", 2022-09-12T19:16:48.818 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.715054633Z", 2022-09-12T19:16:48.818 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.818 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.819 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.819 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.819 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.820 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_xgZt1r-qtjp-K4E_Z1U_XYi", 2022-09-12T19:16:48.820 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.820 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.821 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.821 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.821 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.822 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010058.720459_327.1", 2022-09-12T19:16:48.822 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.822 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-39", 2022-09-12T19:16:48.823 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.720459880Z", 2022-09-12T19:16:48.823 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.823 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.823 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.824 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.824 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.824 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_xgZt1r-qtjp-K4E_Z1U_XYi", 2022-09-12T19:16:48.825 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.825 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.825 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.826 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.826 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.827 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010058.744939_328.1", 2022-09-12T19:16:48.827 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.827 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-40:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.40", 2022-09-12T19:16:48.827 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.744939676Z", 2022-09-12T19:16:48.828 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.828 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.828 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.828 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.829 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.829 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_XRz8ZhFtWNauZ5YqelhcHjG", 2022-09-12T19:16:48.829 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.830 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.830 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.830 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.831 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.831 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010058.753933_329.1", 2022-09-12T19:16:48.832 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.832 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-40:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.40", 2022-09-12T19:16:48.832 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.753933965Z", 2022-09-12T19:16:48.832 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.833 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.833 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.833 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.834 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.834 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_XRz8ZhFtWNauZ5YqelhcHjG", 2022-09-12T19:16:48.834 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.835 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.835 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.835 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.835 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.836 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010058.758572_330.1", 2022-09-12T19:16:48.836 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.837 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-40", 2022-09-12T19:16:48.837 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.758572726Z", 2022-09-12T19:16:48.837 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.837 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.838 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.838 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.838 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.839 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_4IJ7vbCVAqQL9xJtZhDAZ4r", 2022-09-12T19:16:48.839 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.839 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.840 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.840 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.840 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.841 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010058.761625_331.1", 2022-09-12T19:16:48.841 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.841 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-40", 2022-09-12T19:16:48.842 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.761625471Z", 2022-09-12T19:16:48.842 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.842 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.842 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.843 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.843 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.843 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_4IJ7vbCVAqQL9xJtZhDAZ4r", 2022-09-12T19:16:48.844 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.844 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.844 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.844 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.845 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.845 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.845 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.845 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.846 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010058.795251_332.1", 2022-09-12T19:16:48.846 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.846 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-41:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.41", 2022-09-12T19:16:48.846 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.795251326Z", 2022-09-12T19:16:48.847 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.847 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.847 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.847 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.848 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.848 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.848 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_WLs8_gJqx8-eJNrUvhbOESF", 2022-09-12T19:16:48.848 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.849 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.849 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.849 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.849 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.850 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.850 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.850 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010058.799161_333.1", 2022-09-12T19:16:48.851 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.851 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-41:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.41", 2022-09-12T19:16:48.851 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.799161992Z", 2022-09-12T19:16:48.851 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.852 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.852 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.852 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.852 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.853 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.853 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_WLs8_gJqx8-eJNrUvhbOESF", 2022-09-12T19:16:48.853 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.853 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.854 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.854 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.854 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.854 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.855 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010058.800547_334.1", 2022-09-12T19:16:48.855 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.856 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-41", 2022-09-12T19:16:48.856 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.800547915Z", 2022-09-12T19:16:48.856 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.856 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.857 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.857 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.857 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.857 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.858 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Rx20gRuO3XO_dd_rDNjdCZL", 2022-09-12T19:16:48.858 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.858 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.858 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.859 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.859 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.859 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.860 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010058.804299_335.1", 2022-09-12T19:16:48.860 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.860 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-41", 2022-09-12T19:16:48.861 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:14:18.804299896Z", 2022-09-12T19:16:48.861 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.861 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.861 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.862 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.862 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.862 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.862 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Rx20gRuO3XO_dd_rDNjdCZL", 2022-09-12T19:16:48.863 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.863 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.863 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.863 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.864 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.864 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.864 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.864 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.865 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008569.937424_38.1", 2022-09-12T19:16:48.865 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.865 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-1:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.1", 2022-09-12T19:16:48.865 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:29.937424694Z", 2022-09-12T19:16:48.866 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.866 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.866 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.866 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.867 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.867 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.867 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "__lpWFIiN4hfllNCufyv6oPF", 2022-09-12T19:16:48.867 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.868 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.868 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.868 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.868 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.869 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.869 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.869 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.869 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008569.948669_40.1", 2022-09-12T19:16:48.870 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.870 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-1:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.1", 2022-09-12T19:16:48.870 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:29.948669068Z", 2022-09-12T19:16:48.870 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.871 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.871 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.871 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.871 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.872 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.872 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "__lpWFIiN4hfllNCufyv6oPF", 2022-09-12T19:16:48.872 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.872 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.873 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.873 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.873 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.873 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.874 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.874 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.874 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008569.949794_41.1", 2022-09-12T19:16:48.874 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.875 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-1", 2022-09-12T19:16:48.875 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:29.949794341Z", 2022-09-12T19:16:48.875 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.875 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.876 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.876 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.876 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.876 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.877 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_724Zi_oQbv20Wa_KFzslGam", 2022-09-12T19:16:48.877 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.877 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.877 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.878 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.878 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.878 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.878 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.879 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.879 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008569.952619_42.1", 2022-09-12T19:16:48.879 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.879 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-1", 2022-09-12T19:16:48.880 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:29.952619357Z", 2022-09-12T19:16:48.880 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.880 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.880 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.881 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.881 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.881 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.881 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_724Zi_oQbv20Wa_KFzslGam", 2022-09-12T19:16:48.882 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.882 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.882 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.882 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.883 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.883 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.883 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.883 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.884 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008569.979576_43.1", 2022-09-12T19:16:48.884 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.884 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-2:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.2", 2022-09-12T19:16:48.884 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:29.979576716Z", 2022-09-12T19:16:48.885 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.885 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.885 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.885 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.886 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.886 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.887 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_yWZcOeCLp3AOZWE4KEf6DqY", 2022-09-12T19:16:48.887 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.887 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.888 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.888 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.889 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.889 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.889 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.890 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.890 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008569.982366_44.1", 2022-09-12T19:16:48.891 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.891 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-2:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.2", 2022-09-12T19:16:48.891 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:29.982366892Z", 2022-09-12T19:16:48.892 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.892 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.893 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.893 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.893 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.894 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.894 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_yWZcOeCLp3AOZWE4KEf6DqY", 2022-09-12T19:16:48.895 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.895 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.896 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.896 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.897 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.897 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.897 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.898 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.898 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008569.983351_45.1", 2022-09-12T19:16:48.898 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.898 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-2", 2022-09-12T19:16:48.899 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:29.983351998Z", 2022-09-12T19:16:48.899 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.899 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.899 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.900 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.900 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.900 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.900 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_JYImBEwDt-Z7UabuOPjifZb", 2022-09-12T19:16:48.901 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.901 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.901 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.901 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.902 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.902 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.902 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.902 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.903 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008569.985969_46.1", 2022-09-12T19:16:48.903 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.903 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-2", 2022-09-12T19:16:48.903 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:29.985969116Z", 2022-09-12T19:16:48.904 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.904 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.904 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.904 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.905 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.905 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.905 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_JYImBEwDt-Z7UabuOPjifZb", 2022-09-12T19:16:48.905 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.906 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.906 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.906 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.906 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.907 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.907 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.907 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.907 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008572.534688_47.1", 2022-09-12T19:16:48.908 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.908 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-3:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.3", 2022-09-12T19:16:48.908 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:32.534688656Z", 2022-09-12T19:16:48.908 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.909 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.909 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.909 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.909 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.910 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.910 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_2N-a-PDWYhR0LHw5JLpZR4c", 2022-09-12T19:16:48.910 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.910 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.911 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.911 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.911 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.911 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.912 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.912 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.912 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008572.537960_48.1", 2022-09-12T19:16:48.912 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.913 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-3:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.3", 2022-09-12T19:16:48.913 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:32.537960225Z", 2022-09-12T19:16:48.913 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.913 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.914 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.914 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.914 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.914 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.915 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_2N-a-PDWYhR0LHw5JLpZR4c", 2022-09-12T19:16:48.915 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.915 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.915 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.916 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.916 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.916 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.916 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.917 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.917 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008572.539055_49.1", 2022-09-12T19:16:48.917 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.917 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-3", 2022-09-12T19:16:48.918 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:32.539055121Z", 2022-09-12T19:16:48.918 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.918 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.918 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.919 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.919 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.919 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.919 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_84xzHNW5sbis6XdPCoJHPOk", 2022-09-12T19:16:48.920 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.920 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.920 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.920 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.921 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.921 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.921 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.921 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.922 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008572.542081_50.1", 2022-09-12T19:16:48.922 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.922 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-3", 2022-09-12T19:16:48.922 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:32.542081201Z", 2022-09-12T19:16:48.923 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.923 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.923 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.923 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.924 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.924 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.924 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_84xzHNW5sbis6XdPCoJHPOk", 2022-09-12T19:16:48.924 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.925 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.925 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.925 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.925 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.926 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.926 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.926 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.926 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008578.327919_51.1", 2022-09-12T19:16:48.927 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.927 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-4:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.4", 2022-09-12T19:16:48.927 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.327919700Z", 2022-09-12T19:16:48.927 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.928 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.928 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.928 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.928 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.929 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.929 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_k7Q_LlQwzVHeoxMKmCvxjHo", 2022-09-12T19:16:48.929 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.930 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.930 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.930 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.930 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.931 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.931 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.931 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.931 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008578.330590_52.1", 2022-09-12T19:16:48.932 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.932 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-4:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.4", 2022-09-12T19:16:48.932 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.330590808Z", 2022-09-12T19:16:48.932 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.933 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.933 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.933 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.933 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.934 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.934 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_k7Q_LlQwzVHeoxMKmCvxjHo", 2022-09-12T19:16:48.934 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.934 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.935 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.935 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.935 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.935 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.936 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.936 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.936 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008578.331599_53.1", 2022-09-12T19:16:48.936 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.937 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-4", 2022-09-12T19:16:48.937 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.331599255Z", 2022-09-12T19:16:48.937 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.937 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.938 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.938 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.938 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.938 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.939 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Uwy2BFzmIVF9oQbfNuX8-c2", 2022-09-12T19:16:48.939 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.939 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.939 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.940 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.940 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.940 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.940 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.941 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.941 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008578.334356_54.1", 2022-09-12T19:16:48.941 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.941 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-4", 2022-09-12T19:16:48.942 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.334356674Z", 2022-09-12T19:16:48.942 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.942 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.942 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.943 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.943 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.943 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.943 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Uwy2BFzmIVF9oQbfNuX8-c2", 2022-09-12T19:16:48.944 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.944 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.944 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.944 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.945 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.945 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.945 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.945 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.946 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008578.354410_55.1", 2022-09-12T19:16:48.946 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.946 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-5:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.5", 2022-09-12T19:16:48.946 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.354410340Z", 2022-09-12T19:16:48.947 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.947 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.947 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.947 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.948 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.948 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.948 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_DOCacF6mLBz24cosgaizYjM", 2022-09-12T19:16:48.948 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.949 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.949 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.949 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.949 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.950 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.950 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.950 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.950 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008578.357067_56.1", 2022-09-12T19:16:48.951 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.951 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-5:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.5", 2022-09-12T19:16:48.951 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.357067384Z", 2022-09-12T19:16:48.951 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.952 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.952 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.952 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.952 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.953 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.953 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_DOCacF6mLBz24cosgaizYjM", 2022-09-12T19:16:48.953 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.953 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.954 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.954 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.954 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.955 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.955 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.955 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.956 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008578.357978_57.1", 2022-09-12T19:16:48.956 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.956 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-5", 2022-09-12T19:16:48.957 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.357978793Z", 2022-09-12T19:16:48.957 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.957 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.957 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.958 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.958 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.958 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.958 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_PK2zvS8BU3_kRxhnsNsDeRa", 2022-09-12T19:16:48.959 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.959 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.959 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.959 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.960 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.960 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.960 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.960 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.961 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008578.360592_58.1", 2022-09-12T19:16:48.961 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.961 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-5", 2022-09-12T19:16:48.961 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.360592786Z", 2022-09-12T19:16:48.962 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.962 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.962 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.962 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.963 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.963 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.963 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_PK2zvS8BU3_kRxhnsNsDeRa", 2022-09-12T19:16:48.963 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.964 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.964 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.964 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.964 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.965 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.965 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.965 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.965 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008578.393761_59.1", 2022-09-12T19:16:48.966 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.966 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-6:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.6", 2022-09-12T19:16:48.966 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.393761186Z", 2022-09-12T19:16:48.966 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.967 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.967 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.967 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.967 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.968 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.968 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_zr7R-yTI4PYqVN0hq56_Jgr", 2022-09-12T19:16:48.968 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.968 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.969 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.969 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.969 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.969 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.970 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.970 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.970 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008578.396555_60.1", 2022-09-12T19:16:48.970 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.971 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-6:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.6", 2022-09-12T19:16:48.971 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.396555557Z", 2022-09-12T19:16:48.971 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.971 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.972 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.972 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.972 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.972 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.973 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_zr7R-yTI4PYqVN0hq56_Jgr", 2022-09-12T19:16:48.973 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.973 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.973 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.974 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.974 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.974 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.974 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.975 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.975 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008578.397479_61.1", 2022-09-12T19:16:48.975 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.975 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-6", 2022-09-12T19:16:48.976 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.397479637Z", 2022-09-12T19:16:48.976 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.976 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.976 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.977 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.977 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.977 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.977 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_9moMpukFqONyJQ22546lFFJ", 2022-09-12T19:16:48.978 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.978 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.978 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.978 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.979 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.979 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.979 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.979 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.980 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008578.400100_62.1", 2022-09-12T19:16:48.980 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.980 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-6", 2022-09-12T19:16:48.980 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:38.400100117Z", 2022-09-12T19:16:48.981 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.981 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.981 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.981 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.982 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.982 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.982 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_9moMpukFqONyJQ22546lFFJ", 2022-09-12T19:16:48.982 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.983 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.983 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.983 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.983 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.984 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.984 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.041323_63.1", 2022-09-12T19:16:48.985 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.985 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-7:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.7", 2022-09-12T19:16:48.985 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.041323973Z", 2022-09-12T19:16:48.985 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.986 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.986 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.986 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.986 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.987 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.987 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_14zIMAUeR0_uC-IGvsiZyFE", 2022-09-12T19:16:48.987 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.987 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.988 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.988 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.988 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.988 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.989 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.989 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.989 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.043790_64.1", 2022-09-12T19:16:48.989 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:48.990 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-7:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.7", 2022-09-12T19:16:48.990 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.043790091Z", 2022-09-12T19:16:48.990 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.990 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.991 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.991 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.991 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.992 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_14zIMAUeR0_uC-IGvsiZyFE", 2022-09-12T19:16:48.992 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.992 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.992 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.993 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:48.993 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.993 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.994 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.994 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.044741_65.1", 2022-09-12T19:16:48.994 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.994 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-7", 2022-09-12T19:16:48.995 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.044741403Z", 2022-09-12T19:16:48.995 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:48.995 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:48.995 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:48.996 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:48.996 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.996 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:48.996 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_40hjwX5ISF-woQ9EMT_esot", 2022-09-12T19:16:48.997 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:48.997 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.997 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:48.997 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:48.998 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.998 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:48.998 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:48.999 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:48.999 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.047046_66.1", 2022-09-12T19:16:48.999 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:48.999 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-7", 2022-09-12T19:16:49.000 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.047046794Z", 2022-09-12T19:16:49.000 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.000 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.000 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.001 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.001 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.002 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_40hjwX5ISF-woQ9EMT_esot", 2022-09-12T19:16:49.002 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.002 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.002 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.003 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:49.003 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.003 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.004 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.004 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.068972_67.1", 2022-09-12T19:16:49.004 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.004 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-8:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.8", 2022-09-12T19:16:49.005 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.068972049Z", 2022-09-12T19:16:49.005 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.005 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.005 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.006 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.006 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.006 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.006 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Cbx4jHMPdXce-GRW3F5bG0e", 2022-09-12T19:16:49.007 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.007 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.007 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:49.008 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.008 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.009 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.071464_68.1", 2022-09-12T19:16:49.009 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.009 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-8:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.8", 2022-09-12T19:16:49.010 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.071464065Z", 2022-09-12T19:16:49.010 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.010 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.010 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.011 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.011 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.011 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.011 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Cbx4jHMPdXce-GRW3F5bG0e", 2022-09-12T19:16:49.012 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.012 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.012 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.012 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:49.013 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.013 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.013 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.013 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.014 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.072340_69.1", 2022-09-12T19:16:49.014 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:49.014 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-8", 2022-09-12T19:16:49.014 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.072340420Z", 2022-09-12T19:16:49.015 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.015 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.015 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.015 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.016 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.016 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.016 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_mshfSar0kbJ0_j03QlyLl5l", 2022-09-12T19:16:49.016 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.017 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.017 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.017 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:49.017 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.018 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.018 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.018 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.074696_70.1", 2022-09-12T19:16:49.019 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:49.019 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-8", 2022-09-12T19:16:49.019 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.074696150Z", 2022-09-12T19:16:49.019 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.020 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.020 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.020 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.021 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.021 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_mshfSar0kbJ0_j03QlyLl5l", 2022-09-12T19:16:49.021 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.022 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.022 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:49.022 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.022 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.023 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.023 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.023 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.106134_71.1", 2022-09-12T19:16:49.023 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.024 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9", 2022-09-12T19:16:49.024 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.106134737Z", 2022-09-12T19:16:49.024 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.024 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.025 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.025 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.025 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.025 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.026 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_F8AkuhSt1lRvPdknmmo8vhx", 2022-09-12T19:16:49.026 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.026 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.027 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.027 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:49.027 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.027 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.028 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.028 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.028 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.110078_72.1", 2022-09-12T19:16:49.028 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.029 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9", 2022-09-12T19:16:49.029 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.110078653Z", 2022-09-12T19:16:49.029 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.029 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.030 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.030 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.030 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.030 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.031 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_F8AkuhSt1lRvPdknmmo8vhx", 2022-09-12T19:16:49.031 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.031 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.031 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.032 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:49.032 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.032 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.033 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.111034_73.1", 2022-09-12T19:16:49.033 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:49.033 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-9", 2022-09-12T19:16:49.034 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.111034634Z", 2022-09-12T19:16:49.034 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.034 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.034 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.035 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.035 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.035 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.035 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_dnyYyWYZArJtIne5MIt7vUD", 2022-09-12T19:16:49.036 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.036 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.036 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.036 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:49.037 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.037 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.037 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.037 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.038 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.116857_74.1", 2022-09-12T19:16:49.038 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:49.038 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-9", 2022-09-12T19:16:49.038 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.116857200Z", 2022-09-12T19:16:49.039 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.039 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.039 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.039 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.040 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.040 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_dnyYyWYZArJtIne5MIt7vUD", 2022-09-12T19:16:49.040 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.041 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.041 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:49.041 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.042 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.042 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.768561_75.1", 2022-09-12T19:16:49.043 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.043 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-7:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.7", 2022-09-12T19:16:49.043 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.768561107Z", 2022-09-12T19:16:49.043 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.044 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.044 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_14zIMAUeR0_uC-IGvsiZyFE", 2022-09-12T19:16:49.044 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.045 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.045 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_14zIMAUeR0_uC-IGvsiZyFE", 2022-09-12T19:16:49.045 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:49.045 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.046 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.046 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:49.046 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.046 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.047 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.771274_76.1", 2022-09-12T19:16:49.047 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.048 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-7:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.7", 2022-09-12T19:16:49.048 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.771274788Z", 2022-09-12T19:16:49.048 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.048 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.049 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_14zIMAUeR0_uC-IGvsiZyFE", 2022-09-12T19:16:49.049 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.049 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.050 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_14zIMAUeR0_uC-IGvsiZyFE", 2022-09-12T19:16:49.050 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:49.050 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.050 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.051 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:49.051 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.051 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.052 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.783884_77.1", 2022-09-12T19:16:49.052 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:49.052 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-7", 2022-09-12T19:16:49.053 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.783884039Z", 2022-09-12T19:16:49.053 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.053 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.053 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_40hjwX5ISF-woQ9EMT_esot", 2022-09-12T19:16:49.054 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.054 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.054 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.054 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_40hjwX5ISF-woQ9EMT_esot", 2022-09-12T19:16:49.055 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:49.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.055 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.055 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "remove" 2022-09-12T19:16:49.056 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.056 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.057 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.786242_78.1", 2022-09-12T19:16:49.057 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:49.057 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-7", 2022-09-12T19:16:49.057 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.786242763Z", 2022-09-12T19:16:49.058 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.058 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.058 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_40hjwX5ISF-woQ9EMT_esot", 2022-09-12T19:16:49.059 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.059 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.059 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.059 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_40hjwX5ISF-woQ9EMT_esot", 2022-09-12T19:16:49.060 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:49.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.060 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.060 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:49.061 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.061 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.061 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.062 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.817097_79.1", 2022-09-12T19:16:49.062 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.062 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-8:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.8", 2022-09-12T19:16:49.062 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.817097161Z", 2022-09-12T19:16:49.063 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.063 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.063 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Cbx4jHMPdXce-GRW3F5bG0e", 2022-09-12T19:16:49.063 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.064 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.064 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Cbx4jHMPdXce-GRW3F5bG0e", 2022-09-12T19:16:49.064 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:49.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.065 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.065 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:49.065 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.066 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.066 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.821207_80.1", 2022-09-12T19:16:49.067 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.067 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-8:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.8", 2022-09-12T19:16:49.067 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.821207606Z", 2022-09-12T19:16:49.067 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.068 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.068 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Cbx4jHMPdXce-GRW3F5bG0e", 2022-09-12T19:16:49.068 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.069 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.069 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Cbx4jHMPdXce-GRW3F5bG0e", 2022-09-12T19:16:49.069 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:49.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.070 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.070 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:49.070 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.070 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.071 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.833382_81.1", 2022-09-12T19:16:49.071 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:49.072 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-8", 2022-09-12T19:16:49.072 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.833382871Z", 2022-09-12T19:16:49.072 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.072 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.073 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_mshfSar0kbJ0_j03QlyLl5l", 2022-09-12T19:16:49.073 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.073 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.074 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_mshfSar0kbJ0_j03QlyLl5l", 2022-09-12T19:16:49.074 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:49.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.074 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.075 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "remove" 2022-09-12T19:16:49.075 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.075 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.076 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.835519_82.1", 2022-09-12T19:16:49.076 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:49.076 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-8", 2022-09-12T19:16:49.077 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.835519055Z", 2022-09-12T19:16:49.077 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.077 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.077 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_mshfSar0kbJ0_j03QlyLl5l", 2022-09-12T19:16:49.078 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.078 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.079 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_mshfSar0kbJ0_j03QlyLl5l", 2022-09-12T19:16:49.079 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:49.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.079 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.080 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:49.080 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.080 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.081 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.890492_83.1", 2022-09-12T19:16:49.081 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.081 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9", 2022-09-12T19:16:49.081 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.890492647Z", 2022-09-12T19:16:49.082 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.082 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.082 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_F8AkuhSt1lRvPdknmmo8vhx", 2022-09-12T19:16:49.082 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.083 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.083 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_F8AkuhSt1lRvPdknmmo8vhx", 2022-09-12T19:16:49.083 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:49.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.084 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.084 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:49.084 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.085 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.085 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.893091_84.1", 2022-09-12T19:16:49.086 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.086 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9", 2022-09-12T19:16:49.086 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.893091078Z", 2022-09-12T19:16:49.086 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.087 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.087 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_F8AkuhSt1lRvPdknmmo8vhx", 2022-09-12T19:16:49.087 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.088 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.088 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_F8AkuhSt1lRvPdknmmo8vhx", 2022-09-12T19:16:49.088 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:49.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.089 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.089 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:49.089 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.089 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.090 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.901637_85.1", 2022-09-12T19:16:49.090 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:49.091 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-9", 2022-09-12T19:16:49.091 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.901637143Z", 2022-09-12T19:16:49.091 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.091 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.092 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_dnyYyWYZArJtIne5MIt7vUD", 2022-09-12T19:16:49.092 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.093 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.093 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_dnyYyWYZArJtIne5MIt7vUD", 2022-09-12T19:16:49.093 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:49.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.094 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.094 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "remove" 2022-09-12T19:16:49.094 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.096 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.097 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663008580.904048_86.1", 2022-09-12T19:16:49.097 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:49.097 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-9", 2022-09-12T19:16:49.097 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048979Z", 2022-09-12T19:16:49.098 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.098 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.098 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_dnyYyWYZArJtIne5MIt7vUD", 2022-09-12T19:16:49.098 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.099 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.099 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_dnyYyWYZArJtIne5MIt7vUD", 2022-09-12T19:16:49.100 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 2 2022-09-12T19:16:49.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.100 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.100 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:49.101 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.101 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.101 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.101 INFO:teuthology.orchestra.run.smithi138.stdout:] 2022-09-12T19:16:49.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:49.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:49.104 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:49.162 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:49.162 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:49.205 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:16:49.206 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:49.206 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010125.715126_386.1", 2022-09-12T19:16:49.206 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.207 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-42:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.1", 2022-09-12T19:16:49.207 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:15:25.715126121Z", 2022-09-12T19:16:49.207 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:49.207 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:49.208 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:49.208 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:49.208 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.208 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:49.209 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_sEGI8pRxeteGQh5ygHjkUPt", 2022-09-12T19:16:49.209 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:49.209 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.210 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:49.210 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:49.210 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.210 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.211 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.211 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:49.211 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010125.720586_387.1", 2022-09-12T19:16:49.211 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.212 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-42:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.1", 2022-09-12T19:16:49.212 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:15:25.720586511Z", 2022-09-12T19:16:49.212 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:49.212 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:49.213 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:49.213 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:49.213 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.213 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:49.214 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_sEGI8pRxeteGQh5ygHjkUPt", 2022-09-12T19:16:49.214 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:49.215 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.215 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:49.215 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:49.215 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.216 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.216 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.216 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:49.217 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010125.721803_388.1", 2022-09-12T19:16:49.217 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:49.217 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-42", 2022-09-12T19:16:49.219 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:15:25.721803501Z", 2022-09-12T19:16:49.219 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:49.219 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:49.219 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:49.220 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:49.220 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.220 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:49.220 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Ag-99SdvgpyLskHB83qFHq4", 2022-09-12T19:16:49.221 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:49.221 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.221 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:49.221 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:49.222 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.222 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.222 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.223 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:49.223 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010125.725396_389.1", 2022-09-12T19:16:49.223 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:49.223 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-42", 2022-09-12T19:16:49.224 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:15:25.725396897Z", 2022-09-12T19:16:49.224 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:49.224 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:49.224 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:49.225 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:49.225 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.225 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:49.225 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Ag-99SdvgpyLskHB83qFHq4", 2022-09-12T19:16:49.226 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:49.226 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.226 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:49.227 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:49.227 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.227 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.227 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.228 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:49.228 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010125.758275_390.1", 2022-09-12T19:16:49.228 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.228 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-43:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.2", 2022-09-12T19:16:49.229 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:15:25.758275933Z", 2022-09-12T19:16:49.229 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:49.229 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:49.229 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:49.230 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:49.230 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.230 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:49.230 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_9F48KhqFqfH-pqNFpZHfdd_", 2022-09-12T19:16:49.231 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:49.231 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.231 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:49.232 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:49.232 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.232 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.232 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.233 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:49.233 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010125.763146_391.1", 2022-09-12T19:16:49.233 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.233 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-43:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.2", 2022-09-12T19:16:49.234 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:15:25.763146942Z", 2022-09-12T19:16:49.234 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:49.234 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:49.235 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:49.235 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:49.235 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.235 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:49.236 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_9F48KhqFqfH-pqNFpZHfdd_", 2022-09-12T19:16:49.236 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:49.236 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.236 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:49.237 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:49.237 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.237 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.237 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.238 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:49.238 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010125.765034_392.1", 2022-09-12T19:16:49.238 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:49.239 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-43", 2022-09-12T19:16:49.239 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:15:25.765034886Z", 2022-09-12T19:16:49.239 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:49.239 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:49.240 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:49.240 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:49.240 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.240 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:49.241 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_vCCML12c_6m5F1--Bo_1DVW", 2022-09-12T19:16:49.241 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:49.241 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.241 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:49.242 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:49.242 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.242 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.242 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.243 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:49.243 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010125.771044_393.1", 2022-09-12T19:16:49.243 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:49.243 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-43", 2022-09-12T19:16:49.244 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:15:25.771044284Z", 2022-09-12T19:16:49.244 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:49.244 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:49.244 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:49.245 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:49.245 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.245 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:49.245 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_vCCML12c_6m5F1--Bo_1DVW", 2022-09-12T19:16:49.246 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:49.246 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.246 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:49.248 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:49.248 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.248 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.248 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.249 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:16:49.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:49.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:49.251 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:49.306 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:49.306 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:49.351 INFO:teuthology.orchestra.run.smithi138.stdout:[ 2022-09-12T19:16:49.352 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.353 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010125.715126_386.1", 2022-09-12T19:16:49.353 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.354 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-42:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.1", 2022-09-12T19:16:49.354 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:15:25.715126121Z", 2022-09-12T19:16:49.355 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.355 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.355 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.355 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.356 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.356 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.356 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_sEGI8pRxeteGQh5ygHjkUPt", 2022-09-12T19:16:49.356 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.357 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.357 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.357 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:49.357 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.358 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.358 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.358 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.358 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010125.720586_387.1", 2022-09-12T19:16:49.359 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.359 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-42:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.1", 2022-09-12T19:16:49.359 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:15:25.720586511Z", 2022-09-12T19:16:49.359 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.361 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.361 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.361 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_sEGI8pRxeteGQh5ygHjkUPt", 2022-09-12T19:16:49.361 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.362 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.362 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.362 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010125.721803_388.1", 2022-09-12T19:16:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-42", 2022-09-12T19:16:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:15:25.721803501Z", 2022-09-12T19:16:49.365 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.365 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.365 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.365 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.366 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.366 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.366 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Ag-99SdvgpyLskHB83qFHq4", 2022-09-12T19:16:49.367 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.367 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.367 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.367 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:49.368 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.368 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.369 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.370 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010125.725396_389.1", 2022-09-12T19:16:49.371 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:49.371 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-42", 2022-09-12T19:16:49.371 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:15:25.725396897Z", 2022-09-12T19:16:49.372 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.372 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.373 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.374 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.374 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.374 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.375 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Ag-99SdvgpyLskHB83qFHq4", 2022-09-12T19:16:49.375 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.376 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.376 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.376 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:49.377 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.377 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.378 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010125.758275_390.1", 2022-09-12T19:16:49.378 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.378 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-43:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.2", 2022-09-12T19:16:49.379 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:15:25.758275933Z", 2022-09-12T19:16:49.379 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.379 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.379 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.379 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.380 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.380 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_9F48KhqFqfH-pqNFpZHfdd_", 2022-09-12T19:16:49.380 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.381 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.381 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:49.381 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.382 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.382 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.382 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.383 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010125.763146_391.1", 2022-09-12T19:16:49.383 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.383 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-43:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.2", 2022-09-12T19:16:49.383 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:15:25.763146942Z", 2022-09-12T19:16:49.384 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.384 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.384 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.384 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.385 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.385 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.385 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_9F48KhqFqfH-pqNFpZHfdd_", 2022-09-12T19:16:49.386 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.386 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.386 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.386 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:49.387 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.387 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.387 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.387 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.388 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010125.765034_392.1", 2022-09-12T19:16:49.388 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:49.388 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-43", 2022-09-12T19:16:49.388 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:15:25.765034886Z", 2022-09-12T19:16:49.389 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.389 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.389 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.390 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.390 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.390 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.391 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_vCCML12c_6m5F1--Bo_1DVW", 2022-09-12T19:16:49.391 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.391 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.391 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.392 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:49.392 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.392 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.392 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.393 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.393 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010125.771044_393.1", 2022-09-12T19:16:49.395 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:49.395 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-43", 2022-09-12T19:16:49.396 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:15:25.771044284Z", 2022-09-12T19:16:49.396 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.396 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.397 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.397 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.397 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.397 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.398 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_vCCML12c_6m5F1--Bo_1DVW", 2022-09-12T19:16:49.398 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.399 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.399 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.401 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:49.401 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.401 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.402 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.403 INFO:teuthology.orchestra.run.smithi138.stdout:] 2022-09-12T19:16:49.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:49.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:49.405 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:49.458 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:49.458 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:49.491 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:16:49.492 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:49.492 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010125.715126_386.1", 2022-09-12T19:16:49.493 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.493 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-42:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.1", 2022-09-12T19:16:49.493 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:15:25.715126121Z", 2022-09-12T19:16:49.494 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:49.494 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:49.495 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:49.495 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:49.495 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.496 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:49.496 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_sEGI8pRxeteGQh5ygHjkUPt", 2022-09-12T19:16:49.496 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:49.497 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.497 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:49.497 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:49.497 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.498 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.498 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.498 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:49.498 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010125.720586_387.1", 2022-09-12T19:16:49.499 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.499 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-42:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.1", 2022-09-12T19:16:49.500 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:15:25.720586511Z", 2022-09-12T19:16:49.500 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:49.500 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:49.500 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:49.501 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:49.501 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.501 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:49.502 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_sEGI8pRxeteGQh5ygHjkUPt", 2022-09-12T19:16:49.502 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:49.502 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.503 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:49.503 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:49.503 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.503 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.504 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.504 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:49.504 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010125.721803_388.1", 2022-09-12T19:16:49.504 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:49.505 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-42", 2022-09-12T19:16:49.506 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:15:25.721803501Z", 2022-09-12T19:16:49.506 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:49.506 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:49.506 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:49.507 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:49.507 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.507 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:49.507 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Ag-99SdvgpyLskHB83qFHq4", 2022-09-12T19:16:49.508 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:49.508 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.508 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:49.508 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:49.509 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.509 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.509 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.509 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:49.510 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010125.725396_389.1", 2022-09-12T19:16:49.510 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:49.510 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-42", 2022-09-12T19:16:49.511 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:15:25.725396897Z", 2022-09-12T19:16:49.511 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:49.511 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:49.511 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:49.512 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:49.512 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.512 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:49.512 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Ag-99SdvgpyLskHB83qFHq4", 2022-09-12T19:16:49.513 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:49.513 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.513 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:49.514 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:49.514 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.514 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.515 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.515 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:49.515 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010125.758275_390.1", 2022-09-12T19:16:49.515 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.516 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-43:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.2", 2022-09-12T19:16:49.516 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:15:25.758275933Z", 2022-09-12T19:16:49.516 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:49.517 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:49.517 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:49.517 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:49.517 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.518 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:49.518 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_9F48KhqFqfH-pqNFpZHfdd_", 2022-09-12T19:16:49.518 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:49.518 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.519 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:49.519 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:49.519 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.519 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.520 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.520 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:49.520 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010125.763146_391.1", 2022-09-12T19:16:49.520 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.521 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-43:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.2", 2022-09-12T19:16:49.521 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:15:25.763146942Z", 2022-09-12T19:16:49.522 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:49.522 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:49.522 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:49.523 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:49.523 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.523 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:49.524 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_9F48KhqFqfH-pqNFpZHfdd_", 2022-09-12T19:16:49.524 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:49.524 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.524 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:49.525 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:49.525 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.525 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.526 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.526 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:49.526 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010125.765034_392.1", 2022-09-12T19:16:49.527 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:49.527 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-43", 2022-09-12T19:16:49.527 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:15:25.765034886Z", 2022-09-12T19:16:49.527 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:49.528 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:49.528 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:49.528 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:49.528 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.529 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:49.529 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_vCCML12c_6m5F1--Bo_1DVW", 2022-09-12T19:16:49.529 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:49.530 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.530 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:49.530 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:49.531 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.531 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.531 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.532 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:49.532 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010125.771044_393.1", 2022-09-12T19:16:49.532 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:49.533 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-43", 2022-09-12T19:16:49.533 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:15:25.771044284Z", 2022-09-12T19:16:49.533 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:49.534 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:49.534 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:49.534 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:49.534 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.535 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:49.535 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_vCCML12c_6m5F1--Bo_1DVW", 2022-09-12T19:16:49.535 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:49.535 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.536 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:49.537 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:49.537 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.538 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.538 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.538 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:16:49.539 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:49.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:49.540 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:49.592 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:49.592 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:49.627 INFO:teuthology.orchestra.run.smithi138.stdout:[ 2022-09-12T19:16:49.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.629 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010125.715126_386.1", 2022-09-12T19:16:49.629 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.629 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-42:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.1", 2022-09-12T19:16:49.630 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:15:25.715126121Z", 2022-09-12T19:16:49.630 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.630 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.630 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.631 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.631 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.631 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_sEGI8pRxeteGQh5ygHjkUPt", 2022-09-12T19:16:49.632 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.632 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.632 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:49.633 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.633 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.634 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010125.720586_387.1", 2022-09-12T19:16:49.634 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.635 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-42:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.1", 2022-09-12T19:16:49.635 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:15:25.720586511Z", 2022-09-12T19:16:49.635 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.635 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.636 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.636 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.636 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.637 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_sEGI8pRxeteGQh5ygHjkUPt", 2022-09-12T19:16:49.637 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.637 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.637 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:49.638 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.638 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.639 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010125.721803_388.1", 2022-09-12T19:16:49.639 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:49.639 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-42", 2022-09-12T19:16:49.640 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:15:25.721803501Z", 2022-09-12T19:16:49.640 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.640 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.640 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.640 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.641 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.641 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Ag-99SdvgpyLskHB83qFHq4", 2022-09-12T19:16:49.641 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.642 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.642 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:49.642 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.643 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.643 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010125.725396_389.1", 2022-09-12T19:16:49.644 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:49.644 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-42", 2022-09-12T19:16:49.644 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:15:25.725396897Z", 2022-09-12T19:16:49.645 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.645 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.645 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.646 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.646 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.646 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Ag-99SdvgpyLskHB83qFHq4", 2022-09-12T19:16:49.647 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.647 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.647 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:49.648 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.648 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.649 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010125.758275_390.1", 2022-09-12T19:16:49.649 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.649 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-43:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.2", 2022-09-12T19:16:49.649 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:15:25.758275933Z", 2022-09-12T19:16:49.650 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.650 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.650 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.650 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.651 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.651 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_9F48KhqFqfH-pqNFpZHfdd_", 2022-09-12T19:16:49.651 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.652 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.652 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:49.652 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.653 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.653 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010125.763146_391.1", 2022-09-12T19:16:49.653 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.654 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-43:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.2", 2022-09-12T19:16:49.654 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:15:25.763146942Z", 2022-09-12T19:16:49.654 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.655 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.655 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.655 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.656 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.656 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_9F48KhqFqfH-pqNFpZHfdd_", 2022-09-12T19:16:49.657 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.657 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.657 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:49.658 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.658 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.659 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010125.765034_392.1", 2022-09-12T19:16:49.659 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:49.659 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-43", 2022-09-12T19:16:49.659 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:15:25.765034886Z", 2022-09-12T19:16:49.660 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.660 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.660 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.660 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.661 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.661 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_vCCML12c_6m5F1--Bo_1DVW", 2022-09-12T19:16:49.661 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.662 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.662 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:49.662 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.663 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.663 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010125.771044_393.1", 2022-09-12T19:16:49.664 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:49.664 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-43", 2022-09-12T19:16:49.664 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:15:25.771044284Z", 2022-09-12T19:16:49.665 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.665 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.665 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.665 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.666 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.667 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_vCCML12c_6m5F1--Bo_1DVW", 2022-09-12T19:16:49.667 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.667 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.669 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:49.669 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.669 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.670 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.670 INFO:teuthology.orchestra.run.smithi138.stdout:] 2022-09-12T19:16:49.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '38ca1f57-1358-496c-a99b-df49ee365a4f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:49.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '38ca1f57-1358-496c-a99b-df49ee365a4f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:49.672 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 38ca1f57-1358-496c-a99b-df49ee365a4f --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:49.722 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:49.723 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:49.768 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:16:49.769 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:49.769 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010191.510731_208.1", 2022-09-12T19:16:49.770 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.770 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-44:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.1", 2022-09-12T19:16:49.771 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:31.510731037Z", 2022-09-12T19:16:49.771 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:49.772 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:49.772 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:49.773 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:49.773 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.773 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:49.774 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Pg1zfl1dF8t12Djr4Pm1HoT", 2022-09-12T19:16:49.775 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:49.776 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.776 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:49.777 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:49.777 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.777 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.778 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.778 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:49.778 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010191.513914_209.1", 2022-09-12T19:16:49.778 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.779 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-44:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.1", 2022-09-12T19:16:49.779 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:31.513914414Z", 2022-09-12T19:16:49.779 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:49.779 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:49.780 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:49.780 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:49.780 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.780 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:49.781 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Pg1zfl1dF8t12Djr4Pm1HoT", 2022-09-12T19:16:49.781 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:49.781 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.782 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:49.782 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:49.783 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.783 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.785 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.786 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:49.786 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010191.515103_210.1", 2022-09-12T19:16:49.787 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:49.787 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-44", 2022-09-12T19:16:49.787 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:31.515103443Z", 2022-09-12T19:16:49.787 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:49.788 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:49.788 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:49.788 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:49.788 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.789 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:49.789 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_kF5YStHsDmDO-742oWj4mbn", 2022-09-12T19:16:49.789 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:49.789 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.790 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:49.790 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:49.790 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.791 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.791 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.792 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:49.792 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010191.518508_211.1", 2022-09-12T19:16:49.792 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:49.792 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-44", 2022-09-12T19:16:49.793 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:31.518508227Z", 2022-09-12T19:16:49.793 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:49.793 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:49.793 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:49.793 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:49.794 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.794 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:49.795 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_kF5YStHsDmDO-742oWj4mbn", 2022-09-12T19:16:49.795 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:49.795 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.796 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:49.796 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:49.796 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.796 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.797 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.797 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:49.797 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010191.543171_212.1", 2022-09-12T19:16:49.797 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.798 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-45:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.2", 2022-09-12T19:16:49.799 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:31.543171608Z", 2022-09-12T19:16:49.799 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:49.799 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:49.799 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:49.800 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:49.800 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.800 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:49.801 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_lKxLgSxmBA-yye6CNxpmNOp", 2022-09-12T19:16:49.801 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:49.801 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.802 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:49.802 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:49.802 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.803 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.803 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.803 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:49.803 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010191.546410_213.1", 2022-09-12T19:16:49.804 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.804 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-45:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.2", 2022-09-12T19:16:49.805 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:31.546410409Z", 2022-09-12T19:16:49.805 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:49.805 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:49.805 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:49.806 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:49.806 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.806 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:49.807 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_lKxLgSxmBA-yye6CNxpmNOp", 2022-09-12T19:16:49.807 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:49.807 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.808 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:49.808 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:49.808 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.809 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.809 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.809 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:49.809 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010191.547519_214.1", 2022-09-12T19:16:49.810 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:49.810 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-45", 2022-09-12T19:16:49.810 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:31.547519699Z", 2022-09-12T19:16:49.810 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:49.811 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:49.811 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:49.811 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:49.811 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.812 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:49.812 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Co45FPrasavphghvnqw56yF", 2022-09-12T19:16:49.812 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:49.812 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.813 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:49.813 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:49.813 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.814 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.814 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.814 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:49.814 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010191.550373_215.1", 2022-09-12T19:16:49.815 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:49.815 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-45", 2022-09-12T19:16:49.815 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:31.550373485Z", 2022-09-12T19:16:49.816 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:49.816 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:49.818 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:49.818 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:49.819 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.819 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:49.819 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Co45FPrasavphghvnqw56yF", 2022-09-12T19:16:49.820 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:49.820 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:49.820 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:49.822 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:49.823 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.823 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.823 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:49.824 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:16:49.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '38ca1f57-1358-496c-a99b-df49ee365a4f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:49.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '38ca1f57-1358-496c-a99b-df49ee365a4f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:49.826 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 38ca1f57-1358-496c-a99b-df49ee365a4f --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:49.876 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:49.876 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:49.911 INFO:teuthology.orchestra.run.smithi138.stdout:[ 2022-09-12T19:16:49.911 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.912 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010191.510731_208.1", 2022-09-12T19:16:49.912 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.913 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-44:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.1", 2022-09-12T19:16:49.913 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:31.510731037Z", 2022-09-12T19:16:49.914 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.914 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.914 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.915 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.915 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.916 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.916 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Pg1zfl1dF8t12Djr4Pm1HoT", 2022-09-12T19:16:49.916 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.916 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.917 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.917 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:49.917 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.917 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.918 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.918 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.918 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010191.513914_209.1", 2022-09-12T19:16:49.918 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.919 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-44:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.1", 2022-09-12T19:16:49.919 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:31.513914414Z", 2022-09-12T19:16:49.919 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.919 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.920 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.920 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.920 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.920 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.921 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Pg1zfl1dF8t12Djr4Pm1HoT", 2022-09-12T19:16:49.921 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.921 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.921 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.922 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:49.922 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.922 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.922 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.923 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.923 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010191.515103_210.1", 2022-09-12T19:16:49.923 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:49.923 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-44", 2022-09-12T19:16:49.924 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:31.515103443Z", 2022-09-12T19:16:49.924 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.924 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.925 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.925 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.925 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.926 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.926 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_kF5YStHsDmDO-742oWj4mbn", 2022-09-12T19:16:49.926 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.926 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.927 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.927 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:49.927 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.927 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.928 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.928 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.928 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010191.518508_211.1", 2022-09-12T19:16:49.929 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:49.929 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-44", 2022-09-12T19:16:49.929 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:31.518508227Z", 2022-09-12T19:16:49.930 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.930 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.930 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.930 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.931 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.931 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.931 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_kF5YStHsDmDO-742oWj4mbn", 2022-09-12T19:16:49.932 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.932 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.932 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.932 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:49.932 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.933 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.933 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.933 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.933 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010191.543171_212.1", 2022-09-12T19:16:49.934 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.934 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-45:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.2", 2022-09-12T19:16:49.934 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:31.543171608Z", 2022-09-12T19:16:49.935 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.935 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.935 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.935 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.936 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.936 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.936 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_lKxLgSxmBA-yye6CNxpmNOp", 2022-09-12T19:16:49.936 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.937 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.937 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.937 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:49.937 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.938 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.938 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.938 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.939 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010191.546410_213.1", 2022-09-12T19:16:49.939 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:49.939 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-45:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.2", 2022-09-12T19:16:49.939 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:31.546410409Z", 2022-09-12T19:16:49.940 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.940 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.940 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.940 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.941 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.941 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.941 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_lKxLgSxmBA-yye6CNxpmNOp", 2022-09-12T19:16:49.941 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.942 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.942 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.942 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:49.942 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.943 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.943 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.943 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.943 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010191.547519_214.1", 2022-09-12T19:16:49.944 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:49.944 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-45", 2022-09-12T19:16:49.944 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:31.547519699Z", 2022-09-12T19:16:49.944 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.945 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.945 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.945 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.945 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.946 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.946 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Co45FPrasavphghvnqw56yF", 2022-09-12T19:16:49.946 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.946 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.947 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.947 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:49.947 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.947 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.948 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.948 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:49.948 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010191.550373_215.1", 2022-09-12T19:16:49.948 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:49.949 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-45", 2022-09-12T19:16:49.949 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:31.550373485Z", 2022-09-12T19:16:49.949 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:49.949 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:49.950 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:49.950 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:49.950 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.950 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:49.951 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Co45FPrasavphghvnqw56yF", 2022-09-12T19:16:49.951 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:49.951 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:49.951 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:49.953 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:49.953 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.953 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.954 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:49.954 INFO:teuthology.orchestra.run.smithi138.stdout:] 2022-09-12T19:16:49.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '38ca1f57-1358-496c-a99b-df49ee365a4f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:49.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '38ca1f57-1358-496c-a99b-df49ee365a4f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:49.956 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 38ca1f57-1358-496c-a99b-df49ee365a4f --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:50.006 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:50.006 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:50.039 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:16:50.039 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:50.039 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010191.510731_208.1", 2022-09-12T19:16:50.040 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:50.040 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-44:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.1", 2022-09-12T19:16:50.040 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:31.510731037Z", 2022-09-12T19:16:50.040 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:50.041 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:50.041 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:50.041 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:50.041 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:50.042 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:50.042 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Pg1zfl1dF8t12Djr4Pm1HoT", 2022-09-12T19:16:50.042 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:50.043 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:50.043 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:50.043 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:50.043 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:50.044 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:50.044 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:50.044 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:50.044 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010191.513914_209.1", 2022-09-12T19:16:50.045 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:50.045 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-44:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.1", 2022-09-12T19:16:50.045 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:31.513914414Z", 2022-09-12T19:16:50.045 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:50.046 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:50.046 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:50.046 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:50.046 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:50.047 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:50.047 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Pg1zfl1dF8t12Djr4Pm1HoT", 2022-09-12T19:16:50.047 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:50.047 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:50.048 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:50.048 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:50.048 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:50.048 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:50.049 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:50.049 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:50.049 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010191.515103_210.1", 2022-09-12T19:16:50.049 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:50.050 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-44", 2022-09-12T19:16:50.050 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:31.515103443Z", 2022-09-12T19:16:50.050 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:50.050 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:50.051 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:50.051 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:50.051 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:50.051 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:50.052 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_kF5YStHsDmDO-742oWj4mbn", 2022-09-12T19:16:50.052 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:50.052 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:50.052 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:50.053 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:50.053 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:50.053 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:50.053 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:50.054 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:50.054 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010191.518508_211.1", 2022-09-12T19:16:50.054 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:50.054 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-44", 2022-09-12T19:16:50.055 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:31.518508227Z", 2022-09-12T19:16:50.055 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:50.055 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:50.055 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:50.056 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:50.056 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:50.056 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:50.056 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_kF5YStHsDmDO-742oWj4mbn", 2022-09-12T19:16:50.057 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:50.057 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:50.057 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:50.057 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:50.058 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:50.058 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:50.058 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:50.058 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:50.059 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010191.543171_212.1", 2022-09-12T19:16:50.059 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:50.059 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-45:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.2", 2022-09-12T19:16:50.059 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:31.543171608Z", 2022-09-12T19:16:50.060 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:50.060 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:50.060 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:50.060 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:50.061 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:50.061 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:50.061 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_lKxLgSxmBA-yye6CNxpmNOp", 2022-09-12T19:16:50.061 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:50.062 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:50.062 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:50.062 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:50.062 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:50.063 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:50.063 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:50.063 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:50.063 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010191.546410_213.1", 2022-09-12T19:16:50.064 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket.instance", 2022-09-12T19:16:50.064 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-45:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.2", 2022-09-12T19:16:50.064 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:31.546410409Z", 2022-09-12T19:16:50.064 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:50.065 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:50.065 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:50.065 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:50.065 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:50.066 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:50.066 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_lKxLgSxmBA-yye6CNxpmNOp", 2022-09-12T19:16:50.066 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:50.066 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:50.067 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:50.067 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:50.067 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:50.067 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:50.068 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:50.068 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:50.068 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010191.547519_214.1", 2022-09-12T19:16:50.068 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:50.068 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-45", 2022-09-12T19:16:50.069 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:31.547519699Z", 2022-09-12T19:16:50.069 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:50.069 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:50.069 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:50.070 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:50.070 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:50.070 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:50.070 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Co45FPrasavphghvnqw56yF", 2022-09-12T19:16:50.071 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:50.071 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:50.071 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:50.071 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "write" 2022-09-12T19:16:50.072 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:50.072 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:50.072 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:50.072 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:50.073 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "1_1663010191.550373_215.1", 2022-09-12T19:16:50.073 INFO:teuthology.orchestra.run.smithi035.stdout: "section": "bucket", 2022-09-12T19:16:50.073 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "jdpiak-45", 2022-09-12T19:16:50.073 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:31.550373485Z", 2022-09-12T19:16:50.074 INFO:teuthology.orchestra.run.smithi035.stdout: "data": { 2022-09-12T19:16:50.074 INFO:teuthology.orchestra.run.smithi035.stdout: "read_version": { 2022-09-12T19:16:50.074 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "", 2022-09-12T19:16:50.074 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 0 2022-09-12T19:16:50.075 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:50.075 INFO:teuthology.orchestra.run.smithi035.stdout: "write_version": { 2022-09-12T19:16:50.075 INFO:teuthology.orchestra.run.smithi035.stdout: "tag": "_Co45FPrasavphghvnqw56yF", 2022-09-12T19:16:50.075 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": 1 2022-09-12T19:16:50.076 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:50.076 INFO:teuthology.orchestra.run.smithi035.stdout: "status": { 2022-09-12T19:16:50.077 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "complete" 2022-09-12T19:16:50.077 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:50.078 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:50.078 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:50.078 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:16:50.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '38ca1f57-1358-496c-a99b-df49ee365a4f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:50.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '38ca1f57-1358-496c-a99b-df49ee365a4f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:50.080 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 38ca1f57-1358-496c-a99b-df49ee365a4f --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:50.128 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:50.129 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:50.167 INFO:teuthology.orchestra.run.smithi138.stdout:[ 2022-09-12T19:16:50.168 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:50.169 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010191.510731_208.1", 2022-09-12T19:16:50.169 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:50.170 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-44:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.1", 2022-09-12T19:16:50.170 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:31.510731037Z", 2022-09-12T19:16:50.171 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:50.171 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:50.172 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:50.172 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:50.173 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:50.173 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:50.174 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Pg1zfl1dF8t12Djr4Pm1HoT", 2022-09-12T19:16:50.174 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:50.175 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:50.175 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:50.176 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:50.176 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:50.177 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:50.177 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:50.178 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:50.178 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010191.513914_209.1", 2022-09-12T19:16:50.179 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:50.179 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-44:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.1", 2022-09-12T19:16:50.179 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:31.513914414Z", 2022-09-12T19:16:50.179 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:50.180 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:50.180 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:50.180 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:50.181 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:50.181 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:50.181 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Pg1zfl1dF8t12Djr4Pm1HoT", 2022-09-12T19:16:50.181 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:50.182 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:50.182 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:50.182 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:50.183 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:50.183 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:50.183 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:50.183 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:50.184 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010191.515103_210.1", 2022-09-12T19:16:50.184 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:50.184 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-44", 2022-09-12T19:16:50.185 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:31.515103443Z", 2022-09-12T19:16:50.185 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:50.185 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:50.186 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:50.186 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:50.186 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:50.187 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:50.187 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_kF5YStHsDmDO-742oWj4mbn", 2022-09-12T19:16:50.187 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:50.187 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:50.188 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:50.188 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:50.188 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:50.189 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:50.189 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:50.189 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:50.189 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010191.518508_211.1", 2022-09-12T19:16:50.190 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:50.190 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-44", 2022-09-12T19:16:50.190 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:31.518508227Z", 2022-09-12T19:16:50.191 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:50.191 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:50.191 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:50.191 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:50.192 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:50.192 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:50.192 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_kF5YStHsDmDO-742oWj4mbn", 2022-09-12T19:16:50.192 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:50.193 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:50.193 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:50.193 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:50.194 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:50.194 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:50.195 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:50.195 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:50.195 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010191.543171_212.1", 2022-09-12T19:16:50.195 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:50.196 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-45:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.2", 2022-09-12T19:16:50.196 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:31.543171608Z", 2022-09-12T19:16:50.196 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:50.196 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:50.197 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:50.197 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:50.197 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:50.198 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:50.198 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_lKxLgSxmBA-yye6CNxpmNOp", 2022-09-12T19:16:50.198 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:50.198 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:50.199 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:50.199 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:50.199 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:50.199 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:50.199 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:50.200 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:50.200 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010191.546410_213.1", 2022-09-12T19:16:50.200 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket.instance", 2022-09-12T19:16:50.200 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-45:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.2", 2022-09-12T19:16:50.201 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:31.546410409Z", 2022-09-12T19:16:50.201 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:50.201 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:50.201 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:50.202 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:50.202 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:50.202 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:50.202 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_lKxLgSxmBA-yye6CNxpmNOp", 2022-09-12T19:16:50.203 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:50.203 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:50.203 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:50.203 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:50.204 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:50.204 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:50.204 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:50.204 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:50.205 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010191.547519_214.1", 2022-09-12T19:16:50.205 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:50.205 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-45", 2022-09-12T19:16:50.205 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:31.547519699Z", 2022-09-12T19:16:50.206 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:50.206 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:50.206 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:50.206 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:50.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:50.207 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:50.207 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Co45FPrasavphghvnqw56yF", 2022-09-12T19:16:50.207 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:50.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:50.208 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:50.208 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "write" 2022-09-12T19:16:50.208 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:50.208 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:50.209 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:50.209 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:50.209 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "1_1663010191.550373_215.1", 2022-09-12T19:16:50.209 INFO:teuthology.orchestra.run.smithi138.stdout: "section": "bucket", 2022-09-12T19:16:50.210 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "jdpiak-45", 2022-09-12T19:16:50.210 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:31.550373485Z", 2022-09-12T19:16:50.210 INFO:teuthology.orchestra.run.smithi138.stdout: "data": { 2022-09-12T19:16:50.210 INFO:teuthology.orchestra.run.smithi138.stdout: "read_version": { 2022-09-12T19:16:50.211 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "", 2022-09-12T19:16:50.211 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 0 2022-09-12T19:16:50.211 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:50.211 INFO:teuthology.orchestra.run.smithi138.stdout: "write_version": { 2022-09-12T19:16:50.212 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Co45FPrasavphghvnqw56yF", 2022-09-12T19:16:50.212 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": 1 2022-09-12T19:16:50.212 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:50.212 INFO:teuthology.orchestra.run.smithi138.stdout: "status": { 2022-09-12T19:16:50.214 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "complete" 2022-09-12T19:16:50.214 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:50.214 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:50.214 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:50.215 INFO:teuthology.orchestra.run.smithi138.stdout:] 2022-09-12T19:16:50.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:16:50.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:16:50.216 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:16:50.263 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:50.264 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:50.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:16:50.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:16:50.329 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T19:16:50.374 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:50.375 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:50.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:16:50.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:16:50.550 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:16:50.597 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:50.598 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:50.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:16:50.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:16:50.666 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T19:16:50.723 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:50.723 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:50.882 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'c0f7a512-c940-4d1f-8adf-3484630f876b', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:50.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'c0f7a512-c940-4d1f-8adf-3484630f876b', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:50.884 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period c0f7a512-c940-4d1f-8adf-3484630f876b --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:50.935 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:50.935 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:51.002 INFO:teuthology.orchestra.run.smithi035.stdout:[] 2022-09-12T19:16:51.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'c0f7a512-c940-4d1f-8adf-3484630f876b', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:51.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'c0f7a512-c940-4d1f-8adf-3484630f876b', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:51.005 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period c0f7a512-c940-4d1f-8adf-3484630f876b --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:51.056 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:51.056 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:51.113 INFO:teuthology.orchestra.run.smithi138.stdout:[] 2022-09-12T19:16:51.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'c0f7a512-c940-4d1f-8adf-3484630f876b', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:51.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'c0f7a512-c940-4d1f-8adf-3484630f876b', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:51.116 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period c0f7a512-c940-4d1f-8adf-3484630f876b --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:51.166 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:51.167 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:51.230 INFO:teuthology.orchestra.run.smithi035.stdout:[] 2022-09-12T19:16:51.231 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'c0f7a512-c940-4d1f-8adf-3484630f876b', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:51.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'c0f7a512-c940-4d1f-8adf-3484630f876b', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:51.234 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period c0f7a512-c940-4d1f-8adf-3484630f876b --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:51.282 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:51.283 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:51.355 INFO:teuthology.orchestra.run.smithi138.stdout:[] 2022-09-12T19:16:51.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:51.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:51.358 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:51.408 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:51.408 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:51.454 INFO:teuthology.orchestra.run.smithi035.stdout:[] 2022-09-12T19:16:51.464 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:51.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:51.467 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:51.515 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:51.516 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:51.576 INFO:teuthology.orchestra.run.smithi138.stdout:[] 2022-09-12T19:16:51.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:51.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:51.579 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:51.631 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:51.631 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:51.699 INFO:teuthology.orchestra.run.smithi035.stdout:[] 2022-09-12T19:16:51.701 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:51.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:51.703 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:51.754 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:51.754 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:51.806 INFO:teuthology.orchestra.run.smithi138.stdout:[] 2022-09-12T19:16:51.808 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '38ca1f57-1358-496c-a99b-df49ee365a4f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:51.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '38ca1f57-1358-496c-a99b-df49ee365a4f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:51.811 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 38ca1f57-1358-496c-a99b-df49ee365a4f --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:51.866 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:51.866 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:51.921 INFO:teuthology.orchestra.run.smithi035.stdout:[] 2022-09-12T19:16:51.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '38ca1f57-1358-496c-a99b-df49ee365a4f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:51.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '38ca1f57-1358-496c-a99b-df49ee365a4f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:51.927 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 38ca1f57-1358-496c-a99b-df49ee365a4f --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:51.976 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:51.976 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:52.029 INFO:teuthology.orchestra.run.smithi138.stdout:[] 2022-09-12T19:16:52.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '38ca1f57-1358-496c-a99b-df49ee365a4f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:52.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '38ca1f57-1358-496c-a99b-df49ee365a4f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:52.034 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 38ca1f57-1358-496c-a99b-df49ee365a4f --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:52.092 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:52.093 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:52.135 INFO:teuthology.orchestra.run.smithi035.stdout:[] 2022-09-12T19:16:52.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '38ca1f57-1358-496c-a99b-df49ee365a4f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:52.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '38ca1f57-1358-496c-a99b-df49ee365a4f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:52.140 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 38ca1f57-1358-496c-a99b-df49ee365a4f --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:52.190 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:52.190 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:52.245 INFO:teuthology.orchestra.run.smithi138.stdout:[] 2022-09-12T19:16:52.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... ok 2022-09-12T19:16:52.256 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-46 2022-09-12T19:16:52.257 DEBUG:boto:path=/jdpiak-46/ 2022-09-12T19:16:52.258 DEBUG:boto:auth_path=/jdpiak-46/ 2022-09-12T19:16:52.259 DEBUG:boto:Method: PUT 2022-09-12T19:16:52.260 DEBUG:boto:Path: /jdpiak-46/ 2022-09-12T19:16:52.261 DEBUG:boto:Data: 2022-09-12T19:16:52.262 DEBUG:boto:Headers: {} 2022-09-12T19:16:52.262 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:52.263 DEBUG:boto:Port: 8000 2022-09-12T19:16:52.264 DEBUG:boto:Params: {} 2022-09-12T19:16:52.264 DEBUG:boto:Token: None 2022-09-12T19:16:52.265 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:16:52 GMT /jdpiak-46/ 2022-09-12T19:16:52.266 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oRVfNP9GtrbUKOii8WKVtpHDz9M= 2022-09-12T19:16:52.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oRVfNP9GtrbUKOii8WKVtpHDz9M=', 'Content-Length': '0'} 2022-09-12T19:16:52.293 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005a3a47d9679e7484-00631f85a4-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:52.294 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jdpiak-47 2022-09-12T19:16:52.295 DEBUG:boto:path=/jdpiak-47/ 2022-09-12T19:16:52.295 DEBUG:boto:auth_path=/jdpiak-47/ 2022-09-12T19:16:52.296 DEBUG:boto:Method: PUT 2022-09-12T19:16:52.296 DEBUG:boto:Path: /jdpiak-47/ 2022-09-12T19:16:52.297 DEBUG:boto:Data: 2022-09-12T19:16:52.298 DEBUG:boto:Headers: {} 2022-09-12T19:16:52.299 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:52.299 DEBUG:boto:Port: 8000 2022-09-12T19:16:52.300 DEBUG:boto:Params: {} 2022-09-12T19:16:52.301 DEBUG:boto:Token: None 2022-09-12T19:16:52.301 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:16:52 GMT /jdpiak-47/ 2022-09-12T19:16:52.302 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ez+faJPenwjuEueRD2wOqKacXGA= 2022-09-12T19:16:52.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ez+faJPenwjuEueRD2wOqKacXGA=', 'Content-Length': '0'} 2022-09-12T19:16:52.342 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009f3466ecd5d3765a-00631f85a4-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:52.343 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jdpiak-48 2022-09-12T19:16:52.343 DEBUG:boto:path=/jdpiak-48/ 2022-09-12T19:16:52.344 DEBUG:boto:auth_path=/jdpiak-48/ 2022-09-12T19:16:52.344 DEBUG:boto:Method: PUT 2022-09-12T19:16:52.345 DEBUG:boto:Path: /jdpiak-48/ 2022-09-12T19:16:52.346 DEBUG:boto:Data: 2022-09-12T19:16:52.346 DEBUG:boto:Headers: {} 2022-09-12T19:16:52.347 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:52.347 DEBUG:boto:Port: 8001 2022-09-12T19:16:52.348 DEBUG:boto:Params: {} 2022-09-12T19:16:52.348 DEBUG:boto:Token: None 2022-09-12T19:16:52.349 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:16:52 GMT /jdpiak-48/ 2022-09-12T19:16:52.356 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1IQaR3yr6V4DPyGFwdugjSKcFUk= 2022-09-12T19:16:52.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1IQaR3yr6V4DPyGFwdugjSKcFUk=', 'Content-Length': '0'} 2022-09-12T19:16:52.386 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000640c7c1f4f216cd3-00631f85a4-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:52.388 DEBUG:boto:path=/jdpiak-46/ 2022-09-12T19:16:52.389 DEBUG:boto:auth_path=/jdpiak-46/ 2022-09-12T19:16:52.390 DEBUG:boto:Method: HEAD 2022-09-12T19:16:52.390 DEBUG:boto:Path: /jdpiak-46/ 2022-09-12T19:16:52.391 DEBUG:boto:Data: 2022-09-12T19:16:52.392 DEBUG:boto:Headers: {} 2022-09-12T19:16:52.394 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:52.396 DEBUG:boto:Port: 8000 2022-09-12T19:16:52.396 DEBUG:boto:Params: {} 2022-09-12T19:16:52.397 DEBUG:boto:Token: None 2022-09-12T19:16:52.397 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:52 GMT /jdpiak-46/ 2022-09-12T19:16:52.398 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Vya1r0UHwyCDfrS/AYz/WOvILWM= 2022-09-12T19:16:52.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Vya1r0UHwyCDfrS/AYz/WOvILWM=', 'Content-Length': '0'} 2022-09-12T19:16:52.401 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053f297a7e7bc2b44-00631f85a4-1432-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:52.402 DEBUG:boto:path=/jdpiak-46/key 2022-09-12T19:16:52.403 DEBUG:boto:auth_path=/jdpiak-46/key 2022-09-12T19:16:52.404 DEBUG:boto:Method: PUT 2022-09-12T19:16:52.404 DEBUG:boto:Path: /jdpiak-46/key 2022-09-12T19:16:52.405 DEBUG:boto:Data: 2022-09-12T19:16:52.406 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2022-09-12T19:16:52.407 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:16:52.408 DEBUG:boto:Port: 8000 2022-09-12T19:16:52.409 DEBUG:boto:Params: {} 2022-09-12T19:16:52.410 DEBUG:boto:Token: None 2022-09-12T19:16:52.411 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Mon, 12 Sep 2022 19:16:52 GMT /jdpiak-46/key 2022-09-12T19:16:52.412 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jQJtR/f1A7kt4T09uU9sHMGn7bI= 2022-09-12T19:16:52.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:16:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jQJtR/f1A7kt4T09uU9sHMGn7bI='} 2022-09-12T19:16:52.425 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000176783b43c51c955-00631f85a4-1432-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:16:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:52.426 DEBUG:boto:path=/jdpiak-47/ 2022-09-12T19:16:52.427 DEBUG:boto:auth_path=/jdpiak-47/ 2022-09-12T19:16:52.427 DEBUG:boto:Method: HEAD 2022-09-12T19:16:52.428 DEBUG:boto:Path: /jdpiak-47/ 2022-09-12T19:16:52.429 DEBUG:boto:Data: 2022-09-12T19:16:52.429 DEBUG:boto:Headers: {} 2022-09-12T19:16:52.430 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:52.430 DEBUG:boto:Port: 8000 2022-09-12T19:16:52.431 DEBUG:boto:Params: {} 2022-09-12T19:16:52.431 DEBUG:boto:Token: None 2022-09-12T19:16:52.432 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:52 GMT /jdpiak-47/ 2022-09-12T19:16:52.432 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rWNVRUx4eFCjIWqu6I/nM7KhSd4= 2022-09-12T19:16:52.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rWNVRUx4eFCjIWqu6I/nM7KhSd4=', 'Content-Length': '0'} 2022-09-12T19:16:52.438 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007db6c6085e2d6126-00631f85a4-1438-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:52.439 DEBUG:boto:path=/jdpiak-47/key 2022-09-12T19:16:52.440 DEBUG:boto:auth_path=/jdpiak-47/key 2022-09-12T19:16:52.440 DEBUG:boto:Method: PUT 2022-09-12T19:16:52.441 DEBUG:boto:Path: /jdpiak-47/key 2022-09-12T19:16:52.441 DEBUG:boto:Data: 2022-09-12T19:16:52.442 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2022-09-12T19:16:52.442 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:16:52.443 DEBUG:boto:Port: 8000 2022-09-12T19:16:52.443 DEBUG:boto:Params: {} 2022-09-12T19:16:52.444 DEBUG:boto:Token: None 2022-09-12T19:16:52.445 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Mon, 12 Sep 2022 19:16:52 GMT /jdpiak-47/key 2022-09-12T19:16:52.445 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yX7jmscb4rPBNgHSFj7CZbQTg1M= 2022-09-12T19:16:52.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:16:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yX7jmscb4rPBNgHSFj7CZbQTg1M='} 2022-09-12T19:16:52.465 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003fa31d0f5d06198d-00631f85a4-1438-test-zone2'), ('Date', 'Mon, 12 Sep 2022 19:16:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:52.465 DEBUG:boto:path=/jdpiak-48/ 2022-09-12T19:16:52.466 DEBUG:boto:auth_path=/jdpiak-48/ 2022-09-12T19:16:52.467 DEBUG:boto:Method: HEAD 2022-09-12T19:16:52.467 DEBUG:boto:Path: /jdpiak-48/ 2022-09-12T19:16:52.468 DEBUG:boto:Data: 2022-09-12T19:16:52.468 DEBUG:boto:Headers: {} 2022-09-12T19:16:52.469 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:52.469 DEBUG:boto:Port: 8001 2022-09-12T19:16:52.470 DEBUG:boto:Params: {} 2022-09-12T19:16:52.470 DEBUG:boto:Token: None 2022-09-12T19:16:52.471 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:16:52 GMT /jdpiak-48/ 2022-09-12T19:16:52.472 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qya9sw7fstVj00uRwOr24KYqC8Q= 2022-09-12T19:16:52.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:16:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qya9sw7fstVj00uRwOr24KYqC8Q=', 'Content-Length': '0'} 2022-09-12T19:16:52.475 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3c9097a57f93283-00631f85a4-1438-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:16:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:52.476 DEBUG:boto:path=/jdpiak-48/key 2022-09-12T19:16:52.476 DEBUG:boto:auth_path=/jdpiak-48/key 2022-09-12T19:16:52.477 DEBUG:boto:Method: PUT 2022-09-12T19:16:52.477 DEBUG:boto:Path: /jdpiak-48/key 2022-09-12T19:16:52.478 DEBUG:boto:Data: 2022-09-12T19:16:52.478 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2022-09-12T19:16:52.479 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:16:52.479 DEBUG:boto:Port: 8001 2022-09-12T19:16:52.480 DEBUG:boto:Params: {} 2022-09-12T19:16:52.480 DEBUG:boto:Token: None 2022-09-12T19:16:52.481 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Mon, 12 Sep 2022 19:16:52 GMT /jdpiak-48/key 2022-09-12T19:16:52.482 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xJ9STTP4329bjBI92zlLs3Pc9Lo= 2022-09-12T19:16:52.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:16:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xJ9STTP4329bjBI92zlLs3Pc9Lo='} 2022-09-12T19:16:52.497 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b3d0df0fdc8a7d7-00631f85a4-1438-test-zone3'), ('Date', 'Mon, 12 Sep 2022 19:16:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:16:52.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:52.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:52.499 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:52.549 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:52.550 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:52.581 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=38ca1f57-1358-496c-a99b-df49ee365a4f 2022-09-12T19:16:52.595 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:16:52.596 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:52.596 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:16:52.597 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:16:52.597 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:52.598 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:52.598 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:16:52.599 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:16:52.599 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:52.600 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:52.600 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663010212.371326_235.1", 2022-09-12T19:16:52.601 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:16:52.371326Z" 2022-09-12T19:16:52.601 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:52.602 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:52.602 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:16:52.603 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:16:52.603 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:52.604 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:16:52.606 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1663010212.371326_235.1', 3: ''} 2022-09-12T19:16:52.607 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T19:16:52.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:52.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:52.610 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:52.661 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:52.662 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:52.711 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:16:52.709+0000 7f9d104b3600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:16:52.712 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:16:52.710+0000 7f9d104b3600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:16:52.735 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:16:52.736 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:16:52.736 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:16:52.736 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:16:52.737 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:16:52.737 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:16:52.737 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:16:52.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:52.738 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:16:52.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:52.738 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:16:52.739 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:16:52.740 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:16:52.740 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:16:52.741 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:16:52.741 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:16:52.741 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:16:52.741 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:16:52.742 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:16:52.742 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:52.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:52.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:52.743 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:16:52.743 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:16:52.743 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:16:52.744 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:16:52.744 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:16:52.744 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:16:52.744 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:16:52.745 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:16:52.745 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:16:52.746 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:52.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:52.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:52.746 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:16:52.747 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:16:52.747 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:16:52.747 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010212.371326_235.1", 2022-09-12T19:16:52.748 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:16:52.748 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:16:52.749 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:16:52.749 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:52.371326Z", 2022-09-12T19:16:52.749 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:16:52.750 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:52.750 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:52.751 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:52.751 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:16:52.752 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:16:52.752 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:16:52.753 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:16:52.753 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:16:52.754 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:16:52.754 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:16:52.754 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:16:52.755 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:16:52.755 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:52.756 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:52.757 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:16:52.757 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:52.758 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:16:52.758 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:16:52.759 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:16:52.759 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:52.760 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:16:52Z" 2022-09-12T19:16:52.760 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:16:52.762 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010212.371326_235.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:52.371326Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:16:52Z" } 2022-09-12T19:16:52.763 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010212.371326_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:52.371326Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:16:52.764 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010212.371326_235.1', 3: ''} 2022-09-12T19:16:52.765 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010212.371326_235.1', 3: ''} 2022-09-12T19:16:52.767 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T19:16:52.768 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T19:16:52.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:52.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:52.771 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:52.831 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:52.831 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:52.879 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:16:52.877+0000 7f113a17e600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:16:52.880 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:16:52.878+0000 7f113a17e600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:16:52.897 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:16:52.898 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:16:52.898 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:16:52.899 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:16:52.899 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:16:52.900 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:16:52.900 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:16:52.900 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:52.901 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:16:52.901 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:52.902 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:16:52.902 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:16:52.902 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:16:52.903 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:16:52.903 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:16:52.903 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:16:52.903 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:16:52.904 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:16:52.904 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:16:52.904 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:52.904 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:52.905 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:52.905 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:16:52.905 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:16:52.906 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:16:52.906 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:16:52.907 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:16:52.907 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:16:52.907 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:16:52.907 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:16:52.908 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:16:52.908 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:52.908 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:52.908 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:52.909 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:16:52.909 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:16:52.910 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:16:52.910 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663010212.371326_235.1", 2022-09-12T19:16:52.911 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:16:52.911 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:16:52.911 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:16:52.911 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:52.371326Z", 2022-09-12T19:16:52.912 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:16:52.913 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:52.913 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:52.914 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:16:52.914 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:16:52.915 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:16:52.915 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:16:52.916 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:16:52.916 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:16:52.917 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:16:52.917 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:16:52.918 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:16:52.918 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:16:52.918 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:52.919 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:16:52.919 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:16:52.920 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:52.920 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:16:52.921 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:16:52.921 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:16:52.923 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:16:52.923 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:16:52Z" 2022-09-12T19:16:52.924 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:16:52.926 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010212.371326_235.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:52.371326Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:16:52Z" } 2022-09-12T19:16:52.927 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010212.371326_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:52.371326Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:16:52.928 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010212.371326_235.1', 3: ''} 2022-09-12T19:16:52.929 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010212.371326_235.1', 3: ''} 2022-09-12T19:16:52.929 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T19:16:52.930 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T19:16:52.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:52.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:52.933 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:52.984 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:52.985 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:53.031 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:16:53.028+0000 7f41f0df3600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:16:53.032 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:16:53.029+0000 7f41f0df3600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:16:53.051 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:16:53.052 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:16:53.052 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:16:53.052 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:16:53.053 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:16:53.053 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:16:53.053 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:16:53.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:53.054 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:16:53.054 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:53.054 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:16:53.054 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:16:53.055 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:16:53.055 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:16:53.055 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:16:53.056 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:16:53.056 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:16:53.056 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:16:53.056 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:16:53.057 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:53.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:53.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:53.058 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:16:53.058 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:16:53.058 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:16:53.058 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:16:53.059 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:16:53.059 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:16:53.059 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:16:53.060 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:16:53.060 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:16:53.060 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:53.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:53.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:53.061 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:16:53.061 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:16:53.062 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:16:53.062 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010212.276165_224.1", 2022-09-12T19:16:53.062 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:16:53.062 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:16:53.063 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:16:53.063 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:52.276165Z", 2022-09-12T19:16:53.063 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:16:53.064 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:53.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:53.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:53.064 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:16:53.065 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:16:53.065 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:16:53.065 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:16:53.066 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:16:53.066 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:16:53.066 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:16:53.066 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:16:53.067 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:16:53.067 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:53.067 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:53.068 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:16:53.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:53.068 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:16:53.068 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:16:53.069 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:16:53.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:53.069 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:16:53Z" 2022-09-12T19:16:53.070 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:16:53.071 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010212.276165_224.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:52.276165Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:16:53Z" } 2022-09-12T19:16:53.072 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010212.276165_224.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:52.276165Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:16:53.072 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010212.371326_235.1', 3: ''} 2022-09-12T19:16:53.073 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010212.276165_224.1', 3: ''} 2022-09-12T19:16:53.073 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1663010212.371326_235.1 target=1_1663010212.276165_224.1 2022-09-12T19:16:58.075 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:58.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:16:58.078 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:16:58.137 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:16:58.138 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:16:58.190 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:16:58.187+0000 7f19816c1600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:16:58.194 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:16:58.189+0000 7f19816c1600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:16:58.220 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:16:58.221 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:16:58.221 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:16:58.221 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:16:58.221 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:16:58.222 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:16:58.222 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:16:58.231 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:58.231 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:16:58.231 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:58.231 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:16:58.232 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:16:58.232 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:16:58.232 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:16:58.233 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:16:58.233 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:16:58.233 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:16:58.233 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:16:58.234 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:16:58.234 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:58.234 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:58.235 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:58.235 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:16:58.235 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:16:58.235 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:16:58.236 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:16:58.236 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:16:58.236 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:16:58.236 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:16:58.237 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:16:58.237 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:16:58.237 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:58.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:58.238 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:58.238 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:16:58.238 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:16:58.239 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:16:58.239 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010212.369010_234.1", 2022-09-12T19:16:58.239 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:16:58.240 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:16:58.240 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:16:58.240 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:52.369010Z", 2022-09-12T19:16:58.240 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:16:58.241 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:58.241 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:58.242 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:16:58.242 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:16:58.242 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:16:58.242 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:16:58.243 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:16:58.243 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:16:58.243 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:16:58.243 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:16:58.244 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:16:58.244 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:16:58.244 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:58.245 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:16:58.245 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:16:58.245 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:58.245 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:16:58.246 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:16:58.246 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:16:58.246 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:16:58.246 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:16:58Z" 2022-09-12T19:16:58.247 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:16:58.247 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010212.369010_234.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:52.369010Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:16:58Z" } 2022-09-12T19:16:58.248 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010212.369010_234.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:52.369010Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:16:58.248 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010212.371326_235.1', 3: ''} 2022-09-12T19:16:58.249 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010212.369010_234.1', 3: ''} 2022-09-12T19:16:58.249 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1663010212.371326_235.1 target=1_1663010212.369010_234.1 2022-09-12T19:17:03.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:03.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:03.253 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:03.310 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:03.311 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:03.347 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:17:03.345+0000 7f19202f1600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:17:03.348 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:17:03.346+0000 7f19202f1600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:17:03.364 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:17:03.365 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:17:03.365 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:17:03.366 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:17:03.366 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:17:03.366 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:17:03.366 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:17:03.367 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:03.367 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:17:03.367 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:03.368 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:17:03.368 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:17:03.368 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:17:03.368 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:17:03.369 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:17:03.369 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:17:03.369 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:17:03.369 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:17:03.370 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:17:03.370 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:03.370 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:03.371 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:03.371 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:17:03.371 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:17:03.371 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:17:03.372 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:17:03.372 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:17:03.372 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:17:03.372 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:17:03.373 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:17:03.373 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:17:03.373 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:03.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:03.374 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:03.374 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:17:03.374 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:17:03.375 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:17:03.375 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010212.371326_235.1", 2022-09-12T19:17:03.375 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:17:03.375 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:17:03.376 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:17:03.376 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:52.371326Z", 2022-09-12T19:17:03.376 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:17:03.376 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:03.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:03.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:03.377 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:17:03.377 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:17:03.378 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:17:03.378 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:17:03.378 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:17:03.378 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:17:03.379 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:17:03.379 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:17:03.379 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:17:03.379 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:03.380 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:03.380 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:17:03.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:03.380 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:17:03.381 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:17:03.381 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:17:03.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:03.381 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:17:03Z" 2022-09-12T19:17:03.382 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:17:03.383 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010212.371326_235.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:52.371326Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:17:03Z" } 2022-09-12T19:17:03.384 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010212.371326_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:52.371326Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:17:03.385 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010212.371326_235.1', 3: ''} 2022-09-12T19:17:03.385 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010212.371326_235.1', 3: ''} 2022-09-12T19:17:03.386 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T19:17:03.386 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2022-09-12T19:17:03.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:03.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:03.388 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:03.445 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:03.446 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:03.527 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:17:03.527 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:03.528 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-12T19:17:03.528 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:03:01.269036Z" 2022-09-12T19:17:03.529 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:03.529 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:03.529 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000004600", 2022-09-12T19:17:03.529 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:16:52.837473Z" 2022-09-12T19:17:03.530 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:03.530 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:03.530 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-12T19:17:03.530 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:04:56.879502Z" 2022-09-12T19:17:03.531 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:03.531 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:03.531 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:17:03.531 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:16:51.797188Z" 2022-09-12T19:17:03.532 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:03.532 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:17:03.532 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004600', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:03.533 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2022-09-12T19:17:03.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:03.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:03.535 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:03.588 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:03.588 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:03.646 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:17:03.647 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:17:03.647 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:17:03.647 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:17:03.648 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:17:03.648 INFO:teuthology.orchestra.run.smithi138.stdout: "instance_id": 4124212413593488935 2022-09-12T19:17:03.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:03.648 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:17:03.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:03.649 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:17:03.649 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:17:03.650 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:17:03.650 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-12T19:17:03.650 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:17:03.650 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:17:03.651 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:17:03.651 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:03:01.269036Z" 2022-09-12T19:17:03.651 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:03.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:03.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:03.653 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:17:03.653 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:17:03.653 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:17:03.654 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000004460", 2022-09-12T19:17:03.654 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:17:03.654 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:17:03.654 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:17:03.655 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:52.419318Z" 2022-09-12T19:17:03.655 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:03.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:03.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:03.656 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:17:03.656 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:17:03.656 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:17:03.657 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-12T19:17:03.657 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:17:03.657 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:17:03.658 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:17:03.658 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:04:56.879502Z" 2022-09-12T19:17:03.658 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:03.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:03.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:03.659 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:17:03.659 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:17:03.660 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:17:03.660 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:17:03.660 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:17:03.660 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:17:03.661 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:17:03.661 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:51.797188Z" 2022-09-12T19:17:03.661 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:03.662 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:03.662 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:17:03.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:03.663 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:17:03.664 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 0, 2022-09-12T19:17:03.664 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 0 2022-09-12T19:17:03.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:03.665 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:17:03Z" 2022-09-12T19:17:03.665 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:17:03.667 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 4124212413593488935 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:03:01.269036Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004460", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:52.419318Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:04:56.879502Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:51.797188Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:17:03Z" } 2022-09-12T19:17:03.667 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:03:01.269036Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004460', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:52.419318Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:04:56.879502Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:51.797188Z'}}] 2022-09-12T19:17:03.668 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004600', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:03.668 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004460', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:03.669 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000004600 target=00000000000000000000:00000000000000004460 2022-09-12T19:17:08.670 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:08.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:08.673 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:08.732 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:08.733 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:08.811 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:17:08.811 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:17:08.812 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:17:08.812 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:17:08.812 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:17:08.812 INFO:teuthology.orchestra.run.smithi138.stdout: "instance_id": 4124212413593488935 2022-09-12T19:17:08.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:08.813 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:17:08.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:08.813 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:17:08.814 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:17:08.814 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:17:08.814 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-12T19:17:08.815 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:17:08.815 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:17:08.815 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:17:08.815 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:03:01.269036Z" 2022-09-12T19:17:08.816 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:08.816 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:08.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:08.816 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:17:08.817 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:17:08.817 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:17:08.817 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000004460", 2022-09-12T19:17:08.817 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:17:08.818 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:17:08.818 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:17:08.818 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:52.419318Z" 2022-09-12T19:17:08.818 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:08.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:08.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:08.819 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:17:08.819 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:17:08.820 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:17:08.820 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-12T19:17:08.820 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:17:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:17:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:17:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:04:56.879502Z" 2022-09-12T19:17:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:17:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:17:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:17:08.824 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:17:08.824 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:17:08.824 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:17:08.824 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:17:08.825 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:51.797188Z" 2022-09-12T19:17:08.825 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:08.825 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:08.825 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:17:08.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:08.826 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:17:08.826 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 0, 2022-09-12T19:17:08.826 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 0 2022-09-12T19:17:08.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:08.827 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:17:08Z" 2022-09-12T19:17:08.827 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:17:08.828 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 4124212413593488935 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:03:01.269036Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004460", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:52.419318Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:04:56.879502Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:51.797188Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:17:08Z" } 2022-09-12T19:17:08.829 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:03:01.269036Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004460', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:52.419318Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:04:56.879502Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:51.797188Z'}}] 2022-09-12T19:17:08.830 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004600', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:08.830 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004460', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:08.831 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000004600 target=00000000000000000000:00000000000000004460 2022-09-12T19:17:13.832 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:13.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:13.833 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:13.893 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:13.894 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:13.952 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:17:13.953 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:17:13.954 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:17:13.954 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:17:13.954 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:17:13.955 INFO:teuthology.orchestra.run.smithi138.stdout: "instance_id": 4124212413593488935 2022-09-12T19:17:13.955 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:13.956 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:17:13.956 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:13.957 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:17:13.957 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:17:13.957 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:17:13.958 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-12T19:17:13.958 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:17:13.958 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:17:13.958 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:17:13.959 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:03:01.269036Z" 2022-09-12T19:17:13.959 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:13.959 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:13.959 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:13.960 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:17:13.960 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:17:13.960 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:17:13.961 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000004740", 2022-09-12T19:17:13.961 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:17:13.961 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:17:13.961 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:17:13.962 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:17:11.711910Z" 2022-09-12T19:17:13.962 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:13.962 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:13.962 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:13.963 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:17:13.963 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:17:13.963 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:17:13.963 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-12T19:17:13.964 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:17:13.964 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:17:13.965 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:17:13.965 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:04:56.879502Z" 2022-09-12T19:17:13.965 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:13.966 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:13.966 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:13.966 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:17:13.967 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:17:13.967 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:17:13.967 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:17:13.968 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:17:13.968 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:17:13.971 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:17:13.971 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:51.797188Z" 2022-09-12T19:17:13.972 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:13.972 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:13.973 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:17:13.973 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:13.973 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:17:13.974 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 0, 2022-09-12T19:17:13.974 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 0 2022-09-12T19:17:13.975 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:13.975 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:17:13Z" 2022-09-12T19:17:13.976 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:17:13.978 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 4124212413593488935 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:03:01.269036Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004740", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:17:11.711910Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:04:56.879502Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:51.797188Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:17:13Z" } 2022-09-12T19:17:13.979 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:03:01.269036Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004740', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:17:11.711910Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:04:56.879502Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:51.797188Z'}}] 2022-09-12T19:17:13.980 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004600', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:13.981 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004740', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:13.982 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2022-09-12T19:17:13.983 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone3 2022-09-12T19:17:13.984 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:13.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:13.986 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:14.044 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:14.044 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:14.101 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:17:14.101 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:14.102 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-12T19:17:14.102 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:03:01.269036Z" 2022-09-12T19:17:14.102 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:14.102 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:14.103 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000004740", 2022-09-12T19:17:14.103 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:17:11.711910Z" 2022-09-12T19:17:14.104 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:14.104 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:14.104 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-12T19:17:14.104 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:04:56.879502Z" 2022-09-12T19:17:14.105 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:14.105 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:14.105 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:17:14.105 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:16:51.797188Z" 2022-09-12T19:17:14.106 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:14.106 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:17:14.107 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004740', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:14.108 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2022-09-12T19:17:14.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:14.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:14.110 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:14.164 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:14.164 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:14.221 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:17:14.222 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:17:14.222 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:17:14.222 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:17:14.223 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:17:14.223 INFO:teuthology.orchestra.run.smithi035.stdout: "instance_id": 16681684026229800507 2022-09-12T19:17:14.223 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:14.223 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:17:14.224 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:14.224 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:17:14.225 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:14.225 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:14.226 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-12T19:17:14.226 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:14.227 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:14.227 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:14.228 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:03:01.269036Z" 2022-09-12T19:17:14.228 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:14.229 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:14.229 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:14.230 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:17:14.230 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:14.231 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:14.231 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000004740", 2022-09-12T19:17:14.232 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:14.232 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:14.233 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:14.233 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:17:11.711910Z" 2022-09-12T19:17:14.234 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:14.234 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:14.235 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:14.235 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:17:14.236 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:14.236 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:14.237 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-12T19:17:14.237 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:14.238 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:14.238 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:14.239 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:04:56.879502Z" 2022-09-12T19:17:14.239 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:14.240 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:14.240 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:14.241 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:17:14.241 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:14.242 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:14.242 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:17:14.243 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:14.243 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:14.244 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:14.244 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:51.797188Z" 2022-09-12T19:17:14.245 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:14.245 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:14.246 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:17:14.246 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:14.247 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:17:14.247 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 0, 2022-09-12T19:17:14.248 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 0 2022-09-12T19:17:14.248 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:14.249 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:17:14Z" 2022-09-12T19:17:14.249 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:17:14.251 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16681684026229800507 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:03:01.269036Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004740", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:17:11.711910Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:04:56.879502Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:51.797188Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:17:14Z" } 2022-09-12T19:17:14.253 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:03:01.269036Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004740', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:17:11.711910Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:04:56.879502Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:51.797188Z'}}] 2022-09-12T19:17:14.254 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004740', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:14.255 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004740', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:14.255 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2022-09-12T19:17:14.256 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone4 2022-09-12T19:17:14.257 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:14.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:14.259 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:14.313 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:14.314 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:14.392 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:17:14.392 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:14.392 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-12T19:17:14.393 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:03:01.269036Z" 2022-09-12T19:17:14.393 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:14.394 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:14.394 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000004740", 2022-09-12T19:17:14.395 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:17:11.711910Z" 2022-09-12T19:17:14.395 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:14.395 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:14.396 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-12T19:17:14.397 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:04:56.879502Z" 2022-09-12T19:17:14.397 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:14.398 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:14.398 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:17:14.398 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:16:51.797188Z" 2022-09-12T19:17:14.398 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:14.399 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:17:14.400 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004740', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:14.401 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2022-09-12T19:17:14.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:14.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:14.404 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:14.458 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:14.459 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:14.502 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:17:14.502 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:17:14.503 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:17:14.503 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:17:14.504 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:17:14.504 INFO:teuthology.orchestra.run.smithi138.stdout: "instance_id": 14449053848353038014 2022-09-12T19:17:14.504 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:14.504 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:17:14.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:14.505 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:17:14.505 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:17:14.506 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:17:14.506 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-12T19:17:14.506 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:17:14.507 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:17:14.507 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:17:14.507 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:03:01.269036Z" 2022-09-12T19:17:14.508 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:14.508 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:14.509 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:14.509 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:17:14.510 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:17:14.510 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:17:14.511 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000004740", 2022-09-12T19:17:14.511 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:17:14.511 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:17:14.511 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:17:14.512 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:17:11.711910Z" 2022-09-12T19:17:14.512 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:14.512 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:14.512 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:14.513 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:17:14.513 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:17:14.513 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:17:14.515 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-12T19:17:14.515 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:17:14.516 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:17:14.516 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:17:14.517 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:04:56.879502Z" 2022-09-12T19:17:14.517 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:14.517 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:14.518 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:14.518 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:17:14.519 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:17:14.519 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:17:14.519 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:17:14.520 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:17:14.520 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:17:14.520 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:17:14.522 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:51.797188Z" 2022-09-12T19:17:14.522 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:14.522 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:14.523 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:17:14.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:14.523 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:17:14.523 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 0, 2022-09-12T19:17:14.524 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 0 2022-09-12T19:17:14.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:14.524 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:17:14Z" 2022-09-12T19:17:14.524 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:17:14.525 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14449053848353038014 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:03:01.269036Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004740", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:17:11.711910Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:04:56.879502Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:51.797188Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:17:14Z" } 2022-09-12T19:17:14.526 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:03:01.269036Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004740', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:17:11.711910Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:04:56.879502Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:51.797188Z'}}] 2022-09-12T19:17:14.527 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004740', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:14.528 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004740', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:14.529 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2022-09-12T19:17:14.530 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2022-09-12T19:17:14.531 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:14.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:14.533 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:14.583 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:14.583 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:14.652 INFO:teuthology.orchestra.run.smithi138.stdout:[ 2022-09-12T19:17:14.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:14.654 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-12T19:17:14.654 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:02:32.028847Z" 2022-09-12T19:17:14.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:14.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:14.656 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000004740", 2022-09-12T19:17:14.656 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:16:52.876058Z" 2022-09-12T19:17:14.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:14.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:14.658 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-12T19:17:14.658 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:04:34.689861Z" 2022-09-12T19:17:14.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:14.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:14.660 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:17:14.660 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:16:32.840203Z" 2022-09-12T19:17:14.661 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:14.662 INFO:teuthology.orchestra.run.smithi138.stdout:] 2022-09-12T19:17:14.664 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000004740', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:14.665 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2022-09-12T19:17:14.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:14.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:14.667 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:14.722 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:14.722 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:14.780 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:17:14.780 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:17:14.781 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:17:14.781 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:17:14.781 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:17:14.781 INFO:teuthology.orchestra.run.smithi035.stdout: "instance_id": 2819386190429732693 2022-09-12T19:17:14.782 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:14.782 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:17:14.782 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:14.783 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:17:14.783 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:14.783 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:14.783 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-12T19:17:14.784 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:14.784 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:14.784 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:14.785 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:02:32.028847Z" 2022-09-12T19:17:14.785 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:14.785 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:14.785 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:14.786 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:17:14.786 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:14.787 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:14.787 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000004740", 2022-09-12T19:17:14.788 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:14.788 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:14.789 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:14.790 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:52.876058Z" 2022-09-12T19:17:14.790 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:14.790 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:14.790 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:14.791 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:17:14.791 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:14.791 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:14.791 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-12T19:17:14.792 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:14.792 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:14.792 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:14.792 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:04:34.689861Z" 2022-09-12T19:17:14.793 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:14.793 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:14.793 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:14.793 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:17:14.794 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:14.794 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:14.794 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:17:14.795 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:14.795 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:14.795 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:14.796 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:32.840203Z" 2022-09-12T19:17:14.796 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:14.797 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:14.797 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:17:14.798 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:14.798 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:17:14.799 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 0, 2022-09-12T19:17:14.799 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 0 2022-09-12T19:17:14.801 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:14.801 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:17:14Z" 2022-09-12T19:17:14.801 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:17:14.802 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 2819386190429732693 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002486", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:02:32.028847Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004740", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:52.876058Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:04:34.689861Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:32.840203Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:17:14Z" } 2022-09-12T19:17:14.802 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:02:32.028847Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004740', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:52.876058Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:04:34.689861Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:32.840203Z'}}] 2022-09-12T19:17:14.803 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000004740', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:14.804 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000004740', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:14.804 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2022-09-12T19:17:14.805 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone3 2022-09-12T19:17:14.805 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:14.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:14.807 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:14.856 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:14.856 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:14.930 INFO:teuthology.orchestra.run.smithi138.stdout:[ 2022-09-12T19:17:14.931 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:14.931 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-12T19:17:14.931 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:02:32.028847Z" 2022-09-12T19:17:14.931 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:14.932 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:14.932 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000004740", 2022-09-12T19:17:14.932 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:16:52.876058Z" 2022-09-12T19:17:14.932 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:14.933 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:14.933 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-12T19:17:14.933 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:04:34.689861Z" 2022-09-12T19:17:14.934 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:14.934 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:14.934 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:17:14.934 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:16:32.840203Z" 2022-09-12T19:17:14.935 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:14.936 INFO:teuthology.orchestra.run.smithi138.stdout:] 2022-09-12T19:17:14.938 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000004740', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:14.939 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2022-09-12T19:17:14.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:14.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:14.941 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:14.998 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:14.998 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:15.045 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:17:15.046 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:17:15.046 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:17:15.047 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:17:15.047 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:17:15.047 INFO:teuthology.orchestra.run.smithi035.stdout: "instance_id": 16602476873656756125 2022-09-12T19:17:15.048 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:15.048 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:17:15.049 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:15.049 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:17:15.049 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:15.050 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:15.050 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-12T19:17:15.051 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:15.051 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:15.051 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:15.052 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:02:32.028847Z" 2022-09-12T19:17:15.052 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:15.052 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:15.052 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:15.053 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:17:15.053 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:15.053 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:15.053 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000004740", 2022-09-12T19:17:15.054 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:15.054 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:15.054 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:15.054 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:52.876058Z" 2022-09-12T19:17:15.055 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:15.056 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:15.056 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:15.056 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:17:15.056 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:15.057 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:15.057 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-12T19:17:15.057 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:15.057 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:15.058 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:15.058 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:04:34.689861Z" 2022-09-12T19:17:15.059 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:15.059 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:15.060 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:15.060 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:17:15.061 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:15.062 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:15.063 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:17:15.063 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:15.064 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:15.064 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:15.064 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:32.840203Z" 2022-09-12T19:17:15.065 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:15.065 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:15.065 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:17:15.066 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:15.066 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:17:15.066 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 0, 2022-09-12T19:17:15.067 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 0 2022-09-12T19:17:15.067 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:15.067 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:17:15Z" 2022-09-12T19:17:15.067 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:17:15.069 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16602476873656756125 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002486", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:02:32.028847Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004740", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:52.876058Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:04:34.689861Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:32.840203Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:17:15Z" } 2022-09-12T19:17:15.069 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:02:32.028847Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004740', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:52.876058Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:04:34.689861Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:32.840203Z'}}] 2022-09-12T19:17:15.070 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000004740', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:15.070 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000004740', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:15.071 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2022-09-12T19:17:15.071 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone4 2022-09-12T19:17:15.072 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone1 2022-09-12T19:17:15.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:15.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:15.074 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:15.125 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:15.125 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:15.192 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:17:15.193 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:15.193 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002768", 2022-09-12T19:17:15.194 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:02:41.932652Z" 2022-09-12T19:17:15.194 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:15.195 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:15.195 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000004599", 2022-09-12T19:17:15.196 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:17:11.917194Z" 2022-09-12T19:17:15.196 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:15.196 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:15.196 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-12T19:17:15.197 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:04:56.722572Z" 2022-09-12T19:17:15.197 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:15.197 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:15.197 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:17:15.198 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:16:31.931954Z" 2022-09-12T19:17:15.198 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:15.198 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:17:15.200 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000002768', 1: '00000000000000000000:00000000000000004599', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:15.200 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2022-09-12T19:17:15.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:15.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:15.202 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:15.255 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:15.255 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:15.312 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:17:15.312 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:17:15.313 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:17:15.313 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:17:15.313 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:17:15.314 INFO:teuthology.orchestra.run.smithi035.stdout: "instance_id": 9390097840922254989 2022-09-12T19:17:15.315 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:15.315 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:17:15.315 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:15.316 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:17:15.316 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:15.316 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:15.317 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002768", 2022-09-12T19:17:15.317 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:15.317 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:15.317 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:15.318 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:02:41.932652Z" 2022-09-12T19:17:15.318 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:15.319 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:15.319 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:15.319 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:17:15.319 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:15.320 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:15.320 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000004319", 2022-09-12T19:17:15.320 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:15.320 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:15.321 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:15.321 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:52.495139Z" 2022-09-12T19:17:15.321 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:15.321 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:15.322 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:15.322 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:17:15.322 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:15.322 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:15.323 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-12T19:17:15.323 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:15.323 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:15.323 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:15.324 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:04:56.722572Z" 2022-09-12T19:17:15.324 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:15.324 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:15.325 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:15.325 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:17:15.325 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:15.325 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:15.326 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:17:15.326 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:15.326 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:15.327 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:15.327 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:31.931954Z" 2022-09-12T19:17:15.327 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:15.327 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:15.328 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:17:15.328 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:15.328 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:17:15.329 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 0, 2022-09-12T19:17:15.329 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 0 2022-09-12T19:17:15.329 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:15.330 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:17:15Z" 2022-09-12T19:17:15.330 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:17:15.331 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 9390097840922254989 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002768", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:02:41.932652Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:52.495139Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006152", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:04:56.722572Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:31.931954Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:17:15Z" } 2022-09-12T19:17:15.332 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002768', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:02:41.932652Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:52.495139Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006152', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:04:56.722572Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:31.931954Z'}}] 2022-09-12T19:17:15.333 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002768', 1: '00000000000000000000:00000000000000004599', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:15.333 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002768', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:15.334 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone3: shard=1 master=00000000000000000000:00000000000000004599 target=00000000000000000000:00000000000000004319 2022-09-12T19:17:20.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:20.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:20.337 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:20.399 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:20.399 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:20.449 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:17:20.449 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:17:20.449 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:17:20.450 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:17:20.450 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:17:20.450 INFO:teuthology.orchestra.run.smithi035.stdout: "instance_id": 9390097840922254989 2022-09-12T19:17:20.450 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:20.451 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:17:20.451 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:20.451 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:17:20.452 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:20.452 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:20.452 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002768", 2022-09-12T19:17:20.452 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:20.453 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:20.453 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:20.453 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:02:41.932652Z" 2022-09-12T19:17:20.453 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:20.454 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:20.454 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:20.455 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:17:20.455 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:20.456 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:20.456 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000004319", 2022-09-12T19:17:20.457 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:20.457 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:20.458 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:20.458 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:52.495139Z" 2022-09-12T19:17:20.459 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:20.459 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:20.460 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:20.460 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:17:20.461 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:20.461 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:20.462 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-12T19:17:20.462 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:20.463 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:20.463 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:20.464 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:04:56.722572Z" 2022-09-12T19:17:20.464 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:20.464 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:20.465 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:20.465 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:17:20.466 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:20.466 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:20.467 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:17:20.467 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:20.467 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:20.467 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:20.468 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:31.931954Z" 2022-09-12T19:17:20.468 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:20.468 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:20.469 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:17:20.469 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:20.469 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:17:20.469 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 0, 2022-09-12T19:17:20.470 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 0 2022-09-12T19:17:20.470 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:20.470 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:17:20Z" 2022-09-12T19:17:20.470 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:17:20.472 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 9390097840922254989 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002768", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:02:41.932652Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:52.495139Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006152", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:04:56.722572Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:31.931954Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:17:20Z" } 2022-09-12T19:17:20.472 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002768', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:02:41.932652Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:52.495139Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006152', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:04:56.722572Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:31.931954Z'}}] 2022-09-12T19:17:20.473 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002768', 1: '00000000000000000000:00000000000000004599', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:20.473 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002768', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:20.474 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone3: shard=1 master=00000000000000000000:00000000000000004599 target=00000000000000000000:00000000000000004319 2022-09-12T19:17:25.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:25.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:25.478 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:25.542 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:25.542 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:25.602 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:17:25.602 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:17:25.602 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:17:25.603 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:17:25.603 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:17:25.603 INFO:teuthology.orchestra.run.smithi035.stdout: "instance_id": 9390097840922254989 2022-09-12T19:17:25.604 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:25.604 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:17:25.604 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:25.605 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:17:25.605 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:25.606 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:25.606 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002768", 2022-09-12T19:17:25.607 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:25.608 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:25.608 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:25.609 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:02:41.932652Z" 2022-09-12T19:17:25.609 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:25.611 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:25.611 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:25.612 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:17:25.612 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:25.612 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:25.612 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000004319", 2022-09-12T19:17:25.613 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:25.613 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:25.613 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:25.614 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:52.495139Z" 2022-09-12T19:17:25.614 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:25.614 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:25.615 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:25.615 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:17:25.615 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:25.615 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:25.615 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-12T19:17:25.616 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:25.616 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:25.616 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:25.616 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:04:56.722572Z" 2022-09-12T19:17:25.617 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:25.617 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:25.617 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:25.618 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:17:25.618 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:25.618 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:25.618 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:17:25.619 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:25.619 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:25.619 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:25.619 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:31.931954Z" 2022-09-12T19:17:25.620 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:25.620 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:25.620 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:17:25.620 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:25.620 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:17:25.621 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 0, 2022-09-12T19:17:25.621 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 0 2022-09-12T19:17:25.621 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:25.621 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:17:25Z" 2022-09-12T19:17:25.622 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:17:25.623 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 9390097840922254989 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002768", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:02:41.932652Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:52.495139Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006152", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:04:56.722572Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:31.931954Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:17:25Z" } 2022-09-12T19:17:25.624 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002768', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:02:41.932652Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:52.495139Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006152', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:04:56.722572Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:31.931954Z'}}] 2022-09-12T19:17:25.625 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002768', 1: '00000000000000000000:00000000000000004599', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:25.625 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002768', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:25.626 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone3: shard=1 master=00000000000000000000:00000000000000004599 target=00000000000000000000:00000000000000004319 2022-09-12T19:17:30.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:30.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:30.629 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:30.691 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:30.692 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:30.759 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:17:30.759 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:17:30.760 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:17:30.760 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:17:30.760 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:17:30.760 INFO:teuthology.orchestra.run.smithi035.stdout: "instance_id": 9390097840922254989 2022-09-12T19:17:30.761 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:30.761 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:17:30.761 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:30.762 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:17:30.762 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:30.763 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:30.763 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002768", 2022-09-12T19:17:30.763 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:30.763 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:30.764 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:30.764 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:02:41.932652Z" 2022-09-12T19:17:30.764 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:30.764 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:30.765 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:30.765 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:17:30.765 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:30.766 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:30.766 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000004319", 2022-09-12T19:17:30.766 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:30.766 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:30.767 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:30.767 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:52.495139Z" 2022-09-12T19:17:30.767 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:30.768 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:30.768 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:30.768 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:17:30.768 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:30.769 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:30.769 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-12T19:17:30.770 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:30.770 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:30.770 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:30.770 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:04:56.722572Z" 2022-09-12T19:17:30.771 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:30.771 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:30.771 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:30.771 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:17:30.772 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:30.772 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:30.772 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:17:30.773 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:30.773 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:30.773 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:30.773 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:31.931954Z" 2022-09-12T19:17:30.774 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:30.774 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:30.774 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:17:30.774 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:30.775 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:17:30.775 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 0, 2022-09-12T19:17:30.775 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 0 2022-09-12T19:17:30.775 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:30.776 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:17:30Z" 2022-09-12T19:17:30.776 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:17:30.777 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 9390097840922254989 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002768", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:02:41.932652Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:52.495139Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006152", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:04:56.722572Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:31.931954Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:17:30Z" } 2022-09-12T19:17:30.778 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002768', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:02:41.932652Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:52.495139Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006152', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:04:56.722572Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:31.931954Z'}}] 2022-09-12T19:17:30.779 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002768', 1: '00000000000000000000:00000000000000004599', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:30.779 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002768', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:30.780 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone3: shard=1 master=00000000000000000000:00000000000000004599 target=00000000000000000000:00000000000000004319 2022-09-12T19:17:35.782 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:35.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:35.784 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:35.847 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:35.847 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:35.907 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:17:35.908 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:17:35.908 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:17:35.909 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:17:35.909 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:17:35.909 INFO:teuthology.orchestra.run.smithi035.stdout: "instance_id": 9390097840922254989 2022-09-12T19:17:35.910 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:35.910 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:17:35.911 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:35.911 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:17:35.912 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:35.913 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:35.913 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002768", 2022-09-12T19:17:35.914 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:35.914 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:35.915 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:35.915 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:02:41.932652Z" 2022-09-12T19:17:35.915 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:35.915 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:35.916 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:35.916 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:17:35.916 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:35.916 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:35.917 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000004739", 2022-09-12T19:17:35.917 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:35.917 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:35.918 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:35.918 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:17:15.699790Z" 2022-09-12T19:17:35.918 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:35.918 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:35.919 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:35.919 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:17:35.919 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:35.919 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:35.920 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-12T19:17:35.920 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:35.920 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:35.921 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:35.921 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:04:56.722572Z" 2022-09-12T19:17:35.921 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:35.921 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:35.922 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:35.922 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:17:35.922 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:17:35.922 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:17:35.923 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:17:35.923 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:17:35.923 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:17:35.924 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:17:35.924 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:31.931954Z" 2022-09-12T19:17:35.924 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:35.925 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:35.925 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:17:35.925 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:35.925 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:17:35.926 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 0, 2022-09-12T19:17:35.926 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 0 2022-09-12T19:17:35.926 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:35.927 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:17:35Z" 2022-09-12T19:17:35.927 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:17:35.928 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 9390097840922254989 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002768", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:02:41.932652Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004739", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:17:15.699790Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006152", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:04:56.722572Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:31.931954Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:17:35Z" } 2022-09-12T19:17:35.929 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002768', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:02:41.932652Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004739', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:17:15.699790Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006152', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:04:56.722572Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:31.931954Z'}}] 2022-09-12T19:17:35.930 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002768', 1: '00000000000000000000:00000000000000004599', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:35.930 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002768', 1: '00000000000000000000:00000000000000004739', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:35.931 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2022-09-12T19:17:35.931 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone2 2022-09-12T19:17:35.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:35.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:35.933 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:35.984 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:35.984 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:36.054 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:17:36.054 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:36.055 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002768", 2022-09-12T19:17:36.055 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:02:41.932652Z" 2022-09-12T19:17:36.055 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:36.055 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:36.056 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000004739", 2022-09-12T19:17:36.056 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:17:15.699790Z" 2022-09-12T19:17:36.056 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:36.056 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:36.057 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-12T19:17:36.057 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:04:56.722572Z" 2022-09-12T19:17:36.057 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:36.058 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:36.058 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:17:36.058 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:16:31.931954Z" 2022-09-12T19:17:36.058 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:36.059 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:17:36.060 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000002768', 1: '00000000000000000000:00000000000000004739', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:36.061 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2022-09-12T19:17:36.061 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:36.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:36.062 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:36.122 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:36.122 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:36.199 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:17:36.200 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:17:36.200 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:17:36.201 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:17:36.201 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:17:36.201 INFO:teuthology.orchestra.run.smithi138.stdout: "instance_id": 15445976120917781840 2022-09-12T19:17:36.201 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:36.202 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:17:36.202 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:36.202 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:17:36.202 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:17:36.203 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:17:36.203 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000002768", 2022-09-12T19:17:36.203 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:17:36.203 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:17:36.204 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:17:36.204 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:02:41.932652Z" 2022-09-12T19:17:36.204 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:36.204 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:36.205 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:36.205 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:17:36.205 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:17:36.206 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:17:36.206 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000004739", 2022-09-12T19:17:36.206 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:17:36.206 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:17:36.207 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:17:36.207 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:17:15.699790Z" 2022-09-12T19:17:36.207 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:36.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:36.208 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:36.208 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:17:36.208 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:17:36.208 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:17:36.209 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-12T19:17:36.209 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:17:36.209 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:17:36.210 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:17:36.210 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:04:56.722572Z" 2022-09-12T19:17:36.210 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:36.210 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:36.211 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:36.211 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:17:36.211 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:17:36.211 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:17:36.212 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:17:36.212 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:17:36.212 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:17:36.213 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:17:36.213 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:31.931954Z" 2022-09-12T19:17:36.213 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:36.213 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:36.214 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:17:36.214 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:36.214 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:17:36.214 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 0, 2022-09-12T19:17:36.215 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 0 2022-09-12T19:17:36.215 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:36.215 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:17:36Z" 2022-09-12T19:17:36.216 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:17:36.217 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 15445976120917781840 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002768", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:02:41.932652Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004739", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:17:15.699790Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006152", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:04:56.722572Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:31.931954Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:17:36Z" } 2022-09-12T19:17:36.217 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002768', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:02:41.932652Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004739', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:17:15.699790Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006152', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:04:56.722572Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:31.931954Z'}}] 2022-09-12T19:17:36.218 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002768', 1: '00000000000000000000:00000000000000004739', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:36.219 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002768', 1: '00000000000000000000:00000000000000004739', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:17:36.219 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2022-09-12T19:17:36.220 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone4 2022-09-12T19:17:36.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:36.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:36.221 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:36.268 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:36.269 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:36.315 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:17:36.315 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:36.316 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-12T19:17:36.316 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:03:01.269036Z" 2022-09-12T19:17:36.316 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:36.317 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:36.317 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000004740", 2022-09-12T19:17:36.317 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:17:11.711910Z" 2022-09-12T19:17:36.318 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:36.318 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:36.318 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-12T19:17:36.319 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:04:56.879502Z" 2022-09-12T19:17:36.319 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:36.320 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:36.320 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:17:36.320 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:16:51.797188Z" 2022-09-12T19:17:36.320 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:36.321 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:17:36.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:17:36.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:17:36.326 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:17:36.372 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:36.373 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:36.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:36.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:36.426 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:36.477 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:36.478 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:36.544 INFO:teuthology.orchestra.run.smithi035.stdout:[] 2022-09-12T19:17:36.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:36.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:36.547 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:36.598 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:36.598 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:36.670 INFO:teuthology.orchestra.run.smithi035.stdout:[] 2022-09-12T19:17:36.672 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:36.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:36.673 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:36.724 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:36.724 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:36.777 INFO:teuthology.orchestra.run.smithi035.stdout:[] 2022-09-12T19:17:36.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:36.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:36.780 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:36.831 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:36.831 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:36.877 INFO:teuthology.orchestra.run.smithi035.stdout:[] 2022-09-12T19:17:36.879 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:36.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:36.881 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:36.932 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:36.933 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:37.005 INFO:teuthology.orchestra.run.smithi138.stdout:[ 2022-09-12T19:17:37.005 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:37.006 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-12T19:17:37.006 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:02:32.028847Z" 2022-09-12T19:17:37.006 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:37.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:37.007 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000004740", 2022-09-12T19:17:37.007 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:16:52.876058Z" 2022-09-12T19:17:37.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:37.008 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:37.008 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-12T19:17:37.008 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:04:34.689861Z" 2022-09-12T19:17:37.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:37.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:37.009 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:17:37.009 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:16:32.840203Z" 2022-09-12T19:17:37.009 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:37.010 INFO:teuthology.orchestra.run.smithi138.stdout:] 2022-09-12T19:17:37.010 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:17:37.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:17:37.011 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T19:17:37.056 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:37.057 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:38.115 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:38.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:38.120 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --shard-id 0 --max-entries 1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:38.176 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:38.176 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:38.234 INFO:teuthology.orchestra.run.smithi138.stdout:[] 2022-09-12T19:17:38.236 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:38.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:38.238 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --shard-id 1 --max-entries 1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:38.290 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:38.291 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:38.356 INFO:teuthology.orchestra.run.smithi138.stdout:[] 2022-09-12T19:17:38.357 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:38.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:38.359 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --shard-id 2 --max-entries 1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:38.407 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:38.408 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:38.453 INFO:teuthology.orchestra.run.smithi138.stdout:[] 2022-09-12T19:17:38.455 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:38.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:38.457 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --shard-id 3 --max-entries 1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:38.507 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:38.508 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:38.551 INFO:teuthology.orchestra.run.smithi138.stdout:[] 2022-09-12T19:17:38.552 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:38.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:38.554 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:38.609 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:38.609 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:38.682 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:17:38.683 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:38.683 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-12T19:17:38.683 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:03:01.269036Z" 2022-09-12T19:17:38.684 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:38.684 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:38.684 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000004740", 2022-09-12T19:17:38.685 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:17:11.711910Z" 2022-09-12T19:17:38.685 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:38.685 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:38.685 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-12T19:17:38.686 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:04:56.879502Z" 2022-09-12T19:17:38.686 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:17:38.686 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:17:38.687 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:17:38.687 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:16:51.797188Z" 2022-09-12T19:17:38.687 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:17:38.688 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:17:38.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:17:38.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:17:38.690 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:17:38.737 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:38.738 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:38.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:38.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:38.829 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:38.879 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:38.879 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:38.931 INFO:teuthology.orchestra.run.smithi035.stdout:[] 2022-09-12T19:17:38.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:38.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:38.934 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:38.986 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:38.987 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:39.035 INFO:teuthology.orchestra.run.smithi035.stdout:[] 2022-09-12T19:17:39.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:39.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:39.038 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:39.090 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:39.091 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:39.140 INFO:teuthology.orchestra.run.smithi035.stdout:[] 2022-09-12T19:17:39.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:39.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:17:39.143 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:17:39.193 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:39.194 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:39.253 INFO:teuthology.orchestra.run.smithi035.stdout:[] 2022-09-12T19:17:39.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... ok 2022-09-12T19:17:39.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:17:39.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:17:39.259 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T19:17:39.310 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:39.311 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:39.330 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:17:39.328+0000 7f112b047600 0 failed reading obj info from .rgw.root:zone_info.823fb3a5-4d8a-49dc-ae64-e05804bbea94: (2) No such file or directory 2022-09-12T19:17:39.331 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:17:39.328+0000 7f112b047600 0 WARNING: could not read zone params for zone id=823fb3a5-4d8a-49dc-ae64-e05804bbea94 name=test-zone3 2022-09-12T19:17:39.331 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:17:39.328+0000 7f112b047600 0 failed reading obj info from .rgw.root:zone_info.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47: (2) No such file or directory 2022-09-12T19:17:39.331 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:17:39.328+0000 7f112b047600 0 WARNING: could not read zone params for zone id=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47 name=test-zone1 2022-09-12T19:17:39.337 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:17:39.338 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:17:39.338 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone2", 2022-09-12T19:17:39.338 INFO:teuthology.orchestra.run.smithi138.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-09-12T19:17:39.339 INFO:teuthology.orchestra.run.smithi138.stdout: "control_pool": "test-zone2.rgw.control", 2022-09-12T19:17:39.339 INFO:teuthology.orchestra.run.smithi138.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-09-12T19:17:39.339 INFO:teuthology.orchestra.run.smithi138.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-09-12T19:17:39.339 INFO:teuthology.orchestra.run.smithi138.stdout: "log_pool": "test-zone2.rgw.log", 2022-09-12T19:17:39.340 INFO:teuthology.orchestra.run.smithi138.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-09-12T19:17:39.340 INFO:teuthology.orchestra.run.smithi138.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-09-12T19:17:39.340 INFO:teuthology.orchestra.run.smithi138.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-09-12T19:17:39.340 INFO:teuthology.orchestra.run.smithi138.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-09-12T19:17:39.341 INFO:teuthology.orchestra.run.smithi138.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-09-12T19:17:39.341 INFO:teuthology.orchestra.run.smithi138.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-09-12T19:17:39.341 INFO:teuthology.orchestra.run.smithi138.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-09-12T19:17:39.341 INFO:teuthology.orchestra.run.smithi138.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-09-12T19:17:39.342 INFO:teuthology.orchestra.run.smithi138.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-09-12T19:17:39.342 INFO:teuthology.orchestra.run.smithi138.stdout: "system_key": { 2022-09-12T19:17:39.342 INFO:teuthology.orchestra.run.smithi138.stdout: "access_key": "RDIEFLSOLOEHUH1F", 2022-09-12T19:17:39.342 INFO:teuthology.orchestra.run.smithi138.stdout: "secret_key": "LDCKRd1RecZNgErNcVDnUigDhC7QggCQ" 2022-09-12T19:17:39.343 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:39.343 INFO:teuthology.orchestra.run.smithi138.stdout: "placement_pools": [ 2022-09-12T19:17:39.343 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:39.343 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "default-placement", 2022-09-12T19:17:39.344 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:17:39.344 INFO:teuthology.orchestra.run.smithi138.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-09-12T19:17:39.344 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_classes": { 2022-09-12T19:17:39.344 INFO:teuthology.orchestra.run.smithi138.stdout: "STANDARD": { 2022-09-12T19:17:39.345 INFO:teuthology.orchestra.run.smithi138.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2022-09-12T19:17:39.345 INFO:teuthology.orchestra.run.smithi138.stdout: "compression_type": "random" 2022-09-12T19:17:39.345 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:39.345 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:39.346 INFO:teuthology.orchestra.run.smithi138.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-09-12T19:17:39.346 INFO:teuthology.orchestra.run.smithi138.stdout: "index_type": 0 2022-09-12T19:17:39.346 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:39.346 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:39.347 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:17:39.347 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:17:39.347 INFO:teuthology.orchestra.run.smithi138.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-09-12T19:17:39.347 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:17:39.349 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:17:39.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:17:39.350 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T19:17:39.398 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:17:39.399 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:17:39.453 INFO:teuthology.orchestra.run.smithi138.stderr:Sending period to new master zone fb29b4fb-1f78-4524-ab80-6a5e9e01ab47 2022-09-12T19:17:39.485 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:17:39.485 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:17:39.485 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 2, 2022-09-12T19:17:39.486 INFO:teuthology.orchestra.run.smithi138.stdout: "predecessor_uuid": "2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e", 2022-09-12T19:17:39.486 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": [ 2022-09-12T19:17:39.486 INFO:teuthology.orchestra.run.smithi138.stdout: "", 2022-09-12T19:17:39.486 INFO:teuthology.orchestra.run.smithi138.stdout: "", 2022-09-12T19:17:39.487 INFO:teuthology.orchestra.run.smithi138.stdout: "1_1663010125.771044_393.1", 2022-09-12T19:17:39.487 INFO:teuthology.orchestra.run.smithi138.stdout: "" 2022-09-12T19:17:39.487 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:17:39.488 INFO:teuthology.orchestra.run.smithi138.stdout: "period_map": { 2022-09-12T19:17:39.488 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:17:39.488 INFO:teuthology.orchestra.run.smithi138.stdout: "zonegroups": [ 2022-09-12T19:17:39.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:39.489 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "test-zonegroup", 2022-09-12T19:17:39.489 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zonegroup", 2022-09-12T19:17:39.490 INFO:teuthology.orchestra.run.smithi138.stdout: "api_name": "", 2022-09-12T19:17:39.490 INFO:teuthology.orchestra.run.smithi138.stdout: "is_master": true, 2022-09-12T19:17:39.490 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:17:39.490 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T19:17:39.491 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:17:39.491 INFO:teuthology.orchestra.run.smithi138.stdout: "hostnames": [], 2022-09-12T19:17:39.491 INFO:teuthology.orchestra.run.smithi138.stdout: "hostnames_s3website": [], 2022-09-12T19:17:39.491 INFO:teuthology.orchestra.run.smithi138.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:17:39.492 INFO:teuthology.orchestra.run.smithi138.stdout: "zones": [ 2022-09-12T19:17:39.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:39.492 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T19:17:39.492 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone3", 2022-09-12T19:17:39.493 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:17:39.493 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi035.front.sepia.ceph.com:8001" 2022-09-12T19:17:39.493 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:17:39.493 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T19:17:39.494 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T19:17:39.494 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:17:39.495 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T19:17:39.495 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "", 2022-09-12T19:17:39.496 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": true, 2022-09-12T19:17:39.496 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [], 2022-09-12T19:17:39.497 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T19:17:39.497 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T19:17:39.497 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:17:39.498 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:17:39.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:39.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:39.498 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "eb41eca3-59e3-4448-adac-b7c5067fb406", 2022-09-12T19:17:39.499 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone4", 2022-09-12T19:17:39.499 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:17:39.500 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi138.front.sepia.ceph.com:8001" 2022-09-12T19:17:39.500 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:17:39.500 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T19:17:39.500 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T19:17:39.501 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:17:39.502 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T19:17:39.502 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "pubsub", 2022-09-12T19:17:39.503 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": false, 2022-09-12T19:17:39.503 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [ 2022-09-12T19:17:39.503 INFO:teuthology.orchestra.run.smithi138.stdout: "test-zone1" 2022-09-12T19:17:39.504 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:17:39.504 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T19:17:39.505 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T19:17:39.505 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:17:39.505 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:17:39.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:39.506 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:39.506 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:17:39.506 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone1", 2022-09-12T19:17:39.507 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:17:39.507 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T19:17:39.507 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:17:39.507 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T19:17:39.508 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T19:17:39.508 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:17:39.508 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T19:17:39.508 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "", 2022-09-12T19:17:39.509 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": true, 2022-09-12T19:17:39.509 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [], 2022-09-12T19:17:39.509 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T19:17:39.510 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T19:17:39.510 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:17:39.510 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:17:39.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:39.511 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:39.511 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:17:39.512 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone2", 2022-09-12T19:17:39.512 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:17:39.512 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi138.front.sepia.ceph.com:8000" 2022-09-12T19:17:39.512 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:17:39.513 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T19:17:39.513 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T19:17:39.513 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:17:39.513 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T19:17:39.514 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "", 2022-09-12T19:17:39.514 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": false, 2022-09-12T19:17:39.514 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [], 2022-09-12T19:17:39.514 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T19:17:39.515 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T19:17:39.515 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:17:39.515 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:17:39.515 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:39.516 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:17:39.516 INFO:teuthology.orchestra.run.smithi138.stdout: "placement_targets": [ 2022-09-12T19:17:39.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:39.516 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "default-placement", 2022-09-12T19:17:39.518 INFO:teuthology.orchestra.run.smithi138.stdout: "tags": [], 2022-09-12T19:17:39.518 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_classes": [ 2022-09-12T19:17:39.518 INFO:teuthology.orchestra.run.smithi138.stdout: "STANDARD" 2022-09-12T19:17:39.518 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:17:39.519 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:39.519 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:17:39.519 INFO:teuthology.orchestra.run.smithi138.stdout: "default_placement": "default-placement", 2022-09-12T19:17:39.519 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:17:39.520 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_policy": { 2022-09-12T19:17:39.520 INFO:teuthology.orchestra.run.smithi138.stdout: "groups": [] 2022-09-12T19:17:39.520 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:39.520 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled_features": [ 2022-09-12T19:17:39.521 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:17:39.521 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:17:39.522 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:39.522 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:17:39.522 INFO:teuthology.orchestra.run.smithi138.stdout: "short_zone_ids": [ 2022-09-12T19:17:39.522 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:39.523 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T19:17:39.523 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 227168046 2022-09-12T19:17:39.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:39.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:39.523 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "eb41eca3-59e3-4448-adac-b7c5067fb406", 2022-09-12T19:17:39.524 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 2080702172 2022-09-12T19:17:39.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:39.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:39.524 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:17:39.525 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 422195086 2022-09-12T19:17:39.525 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:39.525 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:17:39.526 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:17:39.526 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 2881234289 2022-09-12T19:17:39.526 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:39.526 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:17:39.527 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:39.527 INFO:teuthology.orchestra.run.smithi138.stdout: "master_zonegroup": "test-zonegroup", 2022-09-12T19:17:39.527 INFO:teuthology.orchestra.run.smithi138.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:17:39.527 INFO:teuthology.orchestra.run.smithi138.stdout: "period_config": { 2022-09-12T19:17:39.528 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_quota": { 2022-09-12T19:17:39.528 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false, 2022-09-12T19:17:39.528 INFO:teuthology.orchestra.run.smithi138.stdout: "check_on_raw": false, 2022-09-12T19:17:39.528 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size": -1, 2022-09-12T19:17:39.529 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size_kb": 0, 2022-09-12T19:17:39.529 INFO:teuthology.orchestra.run.smithi138.stdout: "max_objects": -1 2022-09-12T19:17:39.529 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:39.529 INFO:teuthology.orchestra.run.smithi138.stdout: "user_quota": { 2022-09-12T19:17:39.530 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false, 2022-09-12T19:17:39.530 INFO:teuthology.orchestra.run.smithi138.stdout: "check_on_raw": false, 2022-09-12T19:17:39.530 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size": -1, 2022-09-12T19:17:39.530 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size_kb": 0, 2022-09-12T19:17:39.531 INFO:teuthology.orchestra.run.smithi138.stdout: "max_objects": -1 2022-09-12T19:17:39.531 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:39.531 INFO:teuthology.orchestra.run.smithi138.stdout: "user_ratelimit": { 2022-09-12T19:17:39.531 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_ops": 0, 2022-09-12T19:17:39.532 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_ops": 0, 2022-09-12T19:17:39.532 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_bytes": 0, 2022-09-12T19:17:39.532 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_bytes": 0, 2022-09-12T19:17:39.532 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false 2022-09-12T19:17:39.533 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:39.533 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_ratelimit": { 2022-09-12T19:17:39.533 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_ops": 0, 2022-09-12T19:17:39.533 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_ops": 0, 2022-09-12T19:17:39.534 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_bytes": 0, 2022-09-12T19:17:39.534 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_bytes": 0, 2022-09-12T19:17:39.534 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false 2022-09-12T19:17:39.534 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:39.535 INFO:teuthology.orchestra.run.smithi138.stdout: "anonymous_ratelimit": { 2022-09-12T19:17:39.535 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_ops": 0, 2022-09-12T19:17:39.535 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_ops": 0, 2022-09-12T19:17:39.535 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_bytes": 0, 2022-09-12T19:17:39.536 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_bytes": 0, 2022-09-12T19:17:39.536 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false 2022-09-12T19:17:39.536 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:17:39.537 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:17:39.537 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:17:39.537 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_name": "test-realm", 2022-09-12T19:17:39.537 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:17:39.538 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:17:39.538 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2022-09-12T19:18:39.540 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-49 2022-09-12T19:18:39.542 DEBUG:boto:path=/jdpiak-49/ 2022-09-12T19:18:39.543 DEBUG:boto:auth_path=/jdpiak-49/ 2022-09-12T19:18:39.544 DEBUG:boto:Method: PUT 2022-09-12T19:18:39.545 DEBUG:boto:Path: /jdpiak-49/ 2022-09-12T19:18:39.546 DEBUG:boto:Data: 2022-09-12T19:18:39.547 DEBUG:boto:Headers: {} 2022-09-12T19:18:39.548 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:18:39.549 DEBUG:boto:Port: 8000 2022-09-12T19:18:39.550 DEBUG:boto:Params: {} 2022-09-12T19:18:39.551 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:18:39.552 DEBUG:boto:Token: None 2022-09-12T19:18:39.552 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:18:39 GMT /jdpiak-49/ 2022-09-12T19:18:39.553 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MZMUp2nHv3byF5945fDMABid1LE= 2022-09-12T19:18:39.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:18:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MZMUp2nHv3byF5945fDMABid1LE=', 'Content-Length': '0'} 2022-09-12T19:18:39.579 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000020f5cfddd06d13a1-00631f860f-1544-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:18:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:18:39.581 DEBUG:boto:path=/jdpiak-49/testredirect 2022-09-12T19:18:39.581 DEBUG:boto:auth_path=/jdpiak-49/testredirect 2022-09-12T19:18:39.582 DEBUG:boto:Method: PUT 2022-09-12T19:18:39.583 DEBUG:boto:Path: /jdpiak-49/testredirect 2022-09-12T19:18:39.584 DEBUG:boto:Data: 2022-09-12T19:18:39.584 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-09-12T19:18:39.585 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:18:39.585 DEBUG:boto:Port: 8000 2022-09-12T19:18:39.586 DEBUG:boto:Params: {} 2022-09-12T19:18:39.587 DEBUG:boto:Token: None 2022-09-12T19:18:39.587 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Mon, 12 Sep 2022 19:18:39 GMT /jdpiak-49/testredirect 2022-09-12T19:18:39.588 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9pFUNDkk62sZQaF0y6zksJkM46E= 2022-09-12T19:18:39.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:18:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9pFUNDkk62sZQaF0y6zksJkM46E='} 2022-09-12T19:18:39.601 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e7ebcb60b33eeb6b-00631f860f-1544-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:18:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:18:39.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:18:39.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:18:39.604 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:18:39.662 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:18:39.662 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:18:39.710 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=38ca1f57-1358-496c-a99b-df49ee365a4f 2022-09-12T19:18:39.734 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:18:39.734 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:18:39.735 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:18:39.735 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:18:39.736 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:18:39.736 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:18:39.737 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:18:39.737 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:18:39.738 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:18:39.738 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:18:39.738 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:18:39.739 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:18:39.572818Z" 2022-09-12T19:18:39.739 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:18:39.739 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:18:39.740 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:18:39.740 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:18:39.740 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:18:39.740 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:18:39.742 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:18:39.743 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T19:18:39.744 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:18:39.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:18:39.746 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:18:39.802 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:18:39.802 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:18:39.833 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:18:39.830+0000 7f1fa6b3a600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:18:39.834 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:18:39.831+0000 7f1fa6b3a600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:18:39.850 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:18:39.851 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:18:39.851 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:18:39.851 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:18:39.852 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:18:39.852 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:18:39.852 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:18:39.853 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:18:39.853 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:18:39.854 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:18:39.854 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:18:39.855 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:18:39.855 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:18:39.856 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:18:39.856 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:18:39.856 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:18:39.857 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:18:39.857 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:18:39.858 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:18:39.858 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:18:39.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:18:39.859 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:18:39.860 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:18:39.860 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:18:39.861 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:18:39.861 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:18:39.862 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:18:39.862 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:18:39.863 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:18:39.863 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:18:39.864 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:18:39.864 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:18:39.864 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:18:39.865 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:18:39.865 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:18:39.866 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:18:39.866 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:18:39.867 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010212.371326_235.1", 2022-09-12T19:18:39.867 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:18:39.868 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:18:39.868 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:18:39.868 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:52.371326Z", 2022-09-12T19:18:39.869 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:18:39.869 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:18:39.869 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:18:39.869 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:18:39.870 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:18:39.870 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:18:39.870 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:18:39.871 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:18:39.871 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:18:39.871 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:18:39.871 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:18:39.872 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:18:39.872 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:18:39.872 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:18:39.873 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:18:39.873 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:18:39.873 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:18:39.874 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:18:39.874 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:18:39.874 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:18:39.875 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:18:39.875 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:18:39Z" 2022-09-12T19:18:39.875 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:18:39.876 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010212.371326_235.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:52.371326Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:18:39Z" } 2022-09-12T19:18:39.877 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010212.371326_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:52.371326Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:18:39.878 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:18:39.879 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010212.371326_235.1', 3: ''} 2022-09-12T19:18:39.879 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010319.572818_269.1 target=1_1663010212.371326_235.1 2022-09-12T19:18:44.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:18:44.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:18:44.883 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:18:44.942 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:18:44.942 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:18:44.971 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:18:44.968+0000 7fb1cdd9d600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:18:44.972 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:18:44.969+0000 7fb1cdd9d600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:18:44.989 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:18:44.990 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:18:44.990 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:18:44.991 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:18:44.991 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:18:44.992 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:18:44.992 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:18:44.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:18:44.993 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:18:44.994 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:18:44.994 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:18:44.995 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:18:44.995 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:18:44.996 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:18:44.996 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:18:44.997 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:18:44.997 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:18:44.998 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:18:44.998 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:18:44.998 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:18:44.999 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:18:45.000 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:18:45.000 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:18:45.001 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:18:45.001 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:18:45.002 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:18:45.003 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:18:45.003 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:18:45.004 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:18:45.004 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:18:45.004 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:18:45.005 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:18:45.005 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:18:45.005 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:18:45.006 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:18:45.006 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:18:45.006 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:18:45.006 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010212.371326_235.1", 2022-09-12T19:18:45.007 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:18:45.007 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:18:45.007 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:18:45.007 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:52.371326Z", 2022-09-12T19:18:45.008 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:18:45.009 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:18:45.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:18:45.010 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:18:45.010 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:18:45.011 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:18:45.011 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:18:45.011 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:18:45.012 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:18:45.012 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:18:45.013 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:18:45.013 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:18:45.013 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:18:45.013 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:18:45.014 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:18:45.014 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:18:45.014 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:18:45.015 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:18:45.015 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:18:45.016 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:18:45.016 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:18:45.017 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:18:44Z" 2022-09-12T19:18:45.018 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:18:45.020 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010212.371326_235.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:52.371326Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:18:44Z" } 2022-09-12T19:18:45.021 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010212.371326_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:52.371326Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:18:45.022 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:18:45.023 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010212.371326_235.1', 3: ''} 2022-09-12T19:18:45.024 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010319.572818_269.1 target=1_1663010212.371326_235.1 2022-09-12T19:18:50.026 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:18:50.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:18:50.028 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:18:50.086 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:18:50.087 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:18:50.136 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:18:50.133+0000 7ffa9963b600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:18:50.137 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:18:50.134+0000 7ffa9963b600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:18:50.151 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:18:50.151 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:18:50.152 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:18:50.153 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:18:50.153 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:18:50.154 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:18:50.154 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:18:50.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:18:50.155 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:18:50.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:18:50.155 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:18:50.156 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:18:50.156 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:18:50.156 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:18:50.156 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:18:50.157 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:18:50.157 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:18:50.157 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:18:50.157 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:18:50.158 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:18:50.158 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:18:50.158 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:18:50.159 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:18:50.159 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:18:50.159 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:18:50.159 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:18:50.160 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:18:50.160 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:18:50.160 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:18:50.160 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:18:50.161 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:18:50.161 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:18:50.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:18:50.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:18:50.162 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:18:50.162 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:18:50.162 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:18:50.162 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010212.371326_235.1", 2022-09-12T19:18:50.163 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:18:50.163 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:18:50.163 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:18:50.163 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:52.371326Z", 2022-09-12T19:18:50.164 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:18:50.164 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:18:50.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:18:50.164 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:18:50.165 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:18:50.165 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:18:50.165 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:18:50.165 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:18:50.166 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:18:50.166 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:18:50.166 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:18:50.166 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:18:50.167 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:18:50.167 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:18:50.167 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:18:50.167 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:18:50.168 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:18:50.168 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:18:50.168 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:18:50.168 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:18:50.169 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:18:50.169 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:18:50Z" 2022-09-12T19:18:50.169 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:18:50.171 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010212.371326_235.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:52.371326Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:18:50Z" } 2022-09-12T19:18:50.171 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010212.371326_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:52.371326Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:18:50.172 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:18:50.172 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010212.371326_235.1', 3: ''} 2022-09-12T19:18:50.173 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010319.572818_269.1 target=1_1663010212.371326_235.1 2022-09-12T19:18:55.174 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:18:55.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:18:55.176 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:18:55.234 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:18:55.235 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:18:55.268 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:18:55.265+0000 7f991e58a600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:18:55.269 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:18:55.266+0000 7f991e58a600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:18:55.292 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:18:55.293 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:18:55.293 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:18:55.294 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:18:55.294 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:18:55.294 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:18:55.295 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:18:55.295 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:18:55.295 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:18:55.295 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:18:55.296 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:18:55.297 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:18:55.297 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:18:55.297 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:18:55.298 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:18:55.298 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:18:55.299 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:18:55.299 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:18:55.300 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:18:55.300 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:18:55.301 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:18:55.301 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:18:55.302 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:18:55.302 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:18:55.303 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:18:55.303 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:18:55.304 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:18:55.304 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:18:55.304 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:18:55.305 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:18:55.305 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:18:55.305 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:18:55.306 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:18:55.306 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:18:55.306 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:18:55.307 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:18:55.307 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:18:55.307 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010212.371326_235.1", 2022-09-12T19:18:55.307 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:18:55.308 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:18:55.308 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:18:55.308 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:52.371326Z", 2022-09-12T19:18:55.309 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:18:55.309 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:18:55.309 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:18:55.309 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:18:55.310 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:18:55.310 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:18:55.310 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:18:55.310 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:18:55.311 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:18:55.311 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:18:55.311 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:18:55.312 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:18:55.312 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:18:55.312 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:18:55.312 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:18:55.313 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:18:55.313 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:18:55.313 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:18:55.313 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:18:55.314 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:18:55.314 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:18:55.315 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:18:55Z" 2022-09-12T19:18:55.315 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:18:55.316 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010212.371326_235.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:52.371326Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:18:55Z" } 2022-09-12T19:18:55.317 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010212.371326_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:52.371326Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:18:55.318 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:18:55.318 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010212.371326_235.1', 3: ''} 2022-09-12T19:18:55.319 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010319.572818_269.1 target=1_1663010212.371326_235.1 2022-09-12T19:19:00.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:19:00.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:19:00.322 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:19:00.380 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:19:00.380 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:19:00.434 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:19:00.430+0000 7fe42d058600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:19:00.435 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:19:00.432+0000 7fe42d058600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:19:00.455 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:19:00.456 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:19:00.456 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:19:00.457 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:19:00.457 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:19:00.457 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:19:00.458 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:19:00.458 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:00.459 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:19:00.459 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:00.460 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:19:00.460 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:00.460 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:00.461 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:19:00.461 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:00.461 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:19:00.462 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:00.462 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:19:00.463 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:00.463 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:00.463 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:00.464 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:00.464 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:19:00.464 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:00.464 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:00.465 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:19:00.465 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:00.465 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:00.466 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:00.466 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:19:00.466 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:00.467 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:00.467 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:00.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:00.468 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:19:00.468 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:00.469 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:00.469 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010212.371326_235.1", 2022-09-12T19:19:00.469 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:00.470 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:00.470 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:00.470 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:52.371326Z", 2022-09-12T19:19:00.471 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:19:00.471 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:00.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:00.471 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:00.472 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:19:00.472 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:00.472 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:00.472 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:19:00.473 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:00.473 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:00.474 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:00.474 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:19:00.475 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:00.475 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:00.476 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:00.476 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:19:00.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:00.477 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:19:00.477 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:19:00.478 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:19:00.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:00.479 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:19:00Z" 2022-09-12T19:19:00.479 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:19:00.481 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010212.371326_235.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:52.371326Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:19:00Z" } 2022-09-12T19:19:00.482 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010212.371326_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:52.371326Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:19:00.483 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:19:00.483 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010212.371326_235.1', 3: ''} 2022-09-12T19:19:00.484 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010319.572818_269.1 target=1_1663010212.371326_235.1 2022-09-12T19:19:05.485 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:19:05.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:19:05.487 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:19:05.544 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:19:05.545 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:19:05.595 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:19:05.593+0000 7f8659baa600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:19:05.596 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:19:05.594+0000 7f8659baa600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:19:05.619 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:19:05.619 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:19:05.620 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:19:05.620 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:19:05.621 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:19:05.621 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:19:05.621 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:19:05.622 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:05.622 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:19:05.622 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:05.624 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:19:05.625 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:05.625 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:05.627 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:19:05.627 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:05.628 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:19:05.628 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:05.629 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:19:05.629 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:05.630 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:05.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:05.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:05.631 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:19:05.632 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:05.632 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:05.632 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:19:05.633 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:05.633 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:05.633 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:05.633 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:19:05.634 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:05.634 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:05.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:05.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:05.635 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:19:05.636 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:05.636 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:05.636 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010212.371326_235.1", 2022-09-12T19:19:05.636 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:05.637 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:05.637 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:05.637 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:52.371326Z", 2022-09-12T19:19:05.637 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:19:05.638 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:05.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:05.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:05.639 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:19:05.639 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:05.639 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:05.640 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:19:05.640 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:05.640 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:05.640 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:05.641 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:19:05.641 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:05.641 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:05.641 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:05.642 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:19:05.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:05.643 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:19:05.643 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:19:05.643 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:19:05.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:05.644 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:19:05Z" 2022-09-12T19:19:05.644 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:19:05.645 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010212.371326_235.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:52.371326Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:19:05Z" } 2022-09-12T19:19:05.646 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010212.371326_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:52.371326Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:19:05.647 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:19:05.648 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010212.371326_235.1', 3: ''} 2022-09-12T19:19:05.648 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010319.572818_269.1 target=1_1663010212.371326_235.1 2022-09-12T19:19:10.649 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:19:10.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:19:10.651 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:19:10.708 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:19:10.709 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:19:10.745 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:19:10.737+0000 7f13a4172600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:19:10.746 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:19:10.738+0000 7f13a4172600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:19:10.758 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:19:10.758 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:19:10.759 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:19:10.759 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:19:10.759 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:19:10.759 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:19:10.760 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:19:10.760 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:10.760 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:19:10.761 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:10.761 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:19:10.761 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:10.761 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:10.762 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:19:10.762 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:10.762 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:19:10.762 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:10.763 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:19:10.763 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:10.763 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:10.763 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:10.764 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:10.764 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:19:10.764 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:10.765 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:10.765 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:19:10.765 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:10.765 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:10.766 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:10.766 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:19:10.766 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:10.766 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:10.767 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:10.767 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:10.767 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:19:10.767 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:10.768 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:10.768 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010212.371326_235.1", 2022-09-12T19:19:10.768 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:10.768 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:10.769 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:10.769 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:52.371326Z", 2022-09-12T19:19:10.769 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:19:10.769 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:10.770 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:10.770 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:10.770 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:19:10.770 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:10.771 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:10.771 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:19:10.771 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:10.771 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:10.772 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:10.772 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:19:10.772 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:10.772 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:10.773 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:10.773 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:19:10.773 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:10.774 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:19:10.774 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:19:10.774 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:19:10.775 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:10.775 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:19:10Z" 2022-09-12T19:19:10.775 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:19:10.776 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010212.371326_235.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:52.371326Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:19:10Z" } 2022-09-12T19:19:10.777 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010212.371326_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:52.371326Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:19:10.778 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:19:10.778 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010212.371326_235.1', 3: ''} 2022-09-12T19:19:10.778 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010319.572818_269.1 target=1_1663010212.371326_235.1 2022-09-12T19:19:15.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:19:15.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:19:15.783 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:19:15.840 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:19:15.840 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:19:15.887 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:19:15.884+0000 7f0273e75600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:19:15.888 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:19:15.886+0000 7f0273e75600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:19:15.912 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:19:15.913 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:19:15.913 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:19:15.913 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:19:15.914 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:19:15.915 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:19:15.916 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:19:15.916 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:15.917 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:19:15.917 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:15.918 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:19:15.918 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:15.919 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:15.920 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:19:15.921 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:15.921 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:19:15.922 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:15.922 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:19:15.923 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:15.923 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:15.923 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:15.923 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:15.924 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:19:15.924 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:15.924 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:15.924 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:19:15.925 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:15.925 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:15.925 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:15.925 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:19:15.926 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:15.926 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:15.926 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:15.927 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:15.927 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:19:15.927 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:15.927 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:15.928 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010212.371326_235.1", 2022-09-12T19:19:15.928 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:15.928 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:15.928 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:15.929 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:52.371326Z", 2022-09-12T19:19:15.929 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:19:15.929 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:15.929 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:15.930 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:15.930 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:19:15.930 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:15.931 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:15.931 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:19:15.931 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:15.931 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:15.932 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:15.932 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:19:15.932 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:15.932 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:15.933 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:15.933 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:19:15.933 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:15.933 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:19:15.934 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:19:15.934 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:19:15.934 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:15.934 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:19:15Z" 2022-09-12T19:19:15.935 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:19:15.935 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010212.371326_235.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:52.371326Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:19:15Z" } 2022-09-12T19:19:15.936 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010212.371326_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:52.371326Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:19:15.937 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:19:15.937 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010212.371326_235.1', 3: ''} 2022-09-12T19:19:15.937 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010319.572818_269.1 target=1_1663010212.371326_235.1 2022-09-12T19:19:20.942 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:19:20.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:19:20.943 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:19:21.002 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:19:21.003 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:19:21.034 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:19:21.031+0000 7fe0ff554600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:19:21.035 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:19:21.032+0000 7fe0ff554600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:19:21.048 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:19:21.049 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:19:21.049 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:19:21.050 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:19:21.050 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:19:21.050 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:19:21.051 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:19:21.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:21.051 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:19:21.051 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:21.052 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:19:21.052 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:21.052 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:21.052 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:19:21.053 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:21.053 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:19:21.054 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:21.054 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:19:21.055 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:21.055 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:21.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:21.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:21.056 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:19:21.057 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:21.057 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:21.058 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:19:21.058 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:21.059 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:21.059 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:21.059 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:19:21.060 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:21.060 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:21.061 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:21.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:21.062 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:19:21.062 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:21.062 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:21.063 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010212.371326_235.1", 2022-09-12T19:19:21.063 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:21.064 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:21.064 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:21.065 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:52.371326Z", 2022-09-12T19:19:21.065 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:19:21.066 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:21.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:21.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:21.067 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:19:21.067 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:21.067 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:21.068 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:19:21.068 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:21.069 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:21.069 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:21.069 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:19:21.070 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:21.070 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:21.070 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:21.070 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:19:21.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:21.071 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:19:21.071 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:19:21.071 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:19:21.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:21.072 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:19:21Z" 2022-09-12T19:19:21.072 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:19:21.073 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010212.371326_235.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:52.371326Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:19:21Z" } 2022-09-12T19:19:21.074 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010212.371326_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:52.371326Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:19:21.075 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:19:21.075 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010212.371326_235.1', 3: ''} 2022-09-12T19:19:21.076 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010319.572818_269.1 target=1_1663010212.371326_235.1 2022-09-12T19:19:26.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:19:26.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:19:26.079 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:19:26.139 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:19:26.139 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:19:26.182 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:19:26.180+0000 7f5e70f6f600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:19:26.183 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:19:26.181+0000 7f5e70f6f600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:19:26.203 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:19:26.204 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:19:26.204 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:19:26.204 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:19:26.205 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:19:26.205 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:19:26.205 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:19:26.205 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:26.206 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:19:26.206 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:26.206 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:19:26.207 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:26.207 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:26.207 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:19:26.208 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:26.208 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:19:26.208 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:26.209 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:19:26.210 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:26.210 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:26.210 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:26.211 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:26.211 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:19:26.211 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:26.211 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:26.212 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:19:26.212 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:26.212 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:26.213 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:26.214 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:19:26.214 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:26.215 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:26.215 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:26.216 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:26.216 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:19:26.217 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:26.217 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:26.217 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010212.371326_235.1", 2022-09-12T19:19:26.218 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:26.218 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:26.218 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:26.218 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:52.371326Z", 2022-09-12T19:19:26.219 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:19:26.219 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:26.219 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:26.219 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:26.220 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:19:26.220 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:26.221 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:26.221 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:19:26.221 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:26.221 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:26.221 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:26.222 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:19:26.222 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:26.222 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:26.223 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:26.223 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:19:26.223 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:26.223 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:19:26.223 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:19:26.224 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:19:26.224 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:26.224 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:19:26Z" 2022-09-12T19:19:26.224 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:19:26.226 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010212.371326_235.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:52.371326Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:19:26Z" } 2022-09-12T19:19:26.227 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010212.371326_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:52.371326Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:19:26.227 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:19:26.228 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010212.371326_235.1', 3: ''} 2022-09-12T19:19:26.228 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010319.572818_269.1 target=1_1663010212.371326_235.1 2022-09-12T19:19:31.230 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:19:31.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:19:31.231 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:19:31.289 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:19:31.290 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:19:31.328 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:19:31.323+0000 7fdff179d600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:19:31.328 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:19:31.324+0000 7fdff179d600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:19:31.340 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:19:31.341 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:19:31.342 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:19:31.342 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:19:31.343 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:19:31.343 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:19:31.343 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:19:31.343 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:31.344 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:19:31.344 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:31.344 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:19:31.344 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:31.345 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:31.345 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:19:31.345 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:31.345 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:19:31.346 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:31.346 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:19:31.346 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:31.346 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:31.347 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:31.347 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:31.347 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:19:31.348 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:31.348 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:31.348 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:19:31.348 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:31.349 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:31.349 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:31.349 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:19:31.350 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:31.350 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:31.350 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:31.350 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:31.351 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:19:31.351 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:31.351 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:31.351 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010212.371326_235.1", 2022-09-12T19:19:31.352 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:31.352 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:31.352 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:31.352 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:52.371326Z", 2022-09-12T19:19:31.353 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:19:31.353 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:31.353 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:31.353 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:31.354 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:19:31.354 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:31.354 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:31.354 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:19:31.355 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:31.356 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:31.356 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:31.356 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:19:31.357 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:31.357 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:31.357 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:31.358 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:19:31.358 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:31.358 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:19:31.358 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:19:31.359 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:19:31.359 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:31.359 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:19:31Z" 2022-09-12T19:19:31.359 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:19:31.361 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010212.371326_235.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:52.371326Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:19:31Z" } 2022-09-12T19:19:31.362 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010212.371326_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:52.371326Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:19:31.362 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:19:31.363 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010212.371326_235.1', 3: ''} 2022-09-12T19:19:31.363 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010319.572818_269.1 target=1_1663010212.371326_235.1 2022-09-12T19:19:36.364 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:19:36.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:19:36.367 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:19:36.424 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:19:36.425 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:19:36.457 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:19:36.456+0000 7f5eea454600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:19:36.458 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:19:36.457+0000 7f5eea454600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:19:36.478 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:19:36.478 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:19:36.478 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:19:36.479 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:19:36.479 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:19:36.479 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:19:36.479 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:19:36.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:36.480 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:19:36.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:36.480 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:19:36.481 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:36.481 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:36.481 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:19:36.481 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:36.482 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:19:36.482 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:36.482 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:19:36.483 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:36.483 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:36.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:36.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:36.484 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:19:36.484 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:36.484 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:36.484 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:19:36.485 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:36.485 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:36.485 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:36.485 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:19:36.486 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:36.486 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:36.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:36.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:36.487 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:19:36.487 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:36.487 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:36.487 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010212.371326_235.1", 2022-09-12T19:19:36.488 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:36.488 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:36.488 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:36.488 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:52.371326Z", 2022-09-12T19:19:36.489 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:19:36.489 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:36.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:36.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:36.490 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:19:36.490 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:36.490 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:36.491 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:19:36.491 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:36.491 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:36.491 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:36.492 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:19:36.492 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:36.492 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:36.492 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:36.493 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:19:36.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:36.493 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:19:36.493 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:19:36.494 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:19:36.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:36.495 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:19:36Z" 2022-09-12T19:19:36.495 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:19:36.497 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010212.371326_235.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:52.371326Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:19:36Z" } 2022-09-12T19:19:36.497 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010212.371326_235.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:52.371326Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:19:36.498 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:19:36.499 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010212.371326_235.1', 3: ''} 2022-09-12T19:19:36.499 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010319.572818_269.1 target=1_1663010212.371326_235.1 2022-09-12T19:19:41.501 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:19:41.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:19:41.504 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:19:41.562 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:19:41.563 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:19:41.592 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:19:41.589+0000 7f04ccfe1600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:19:41.593 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:19:41.590+0000 7f04ccfe1600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:19:41.607 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:19:41.608 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:19:41.609 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:19:41.609 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:19:41.610 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:19:41.610 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:19:41.610 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:19:41.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:41.612 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:19:41.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:41.613 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:19:41.613 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:41.613 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:41.614 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:19:41.614 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:41.615 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:19:41.615 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:41.615 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:19:41.615 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:41.616 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:41.616 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:41.616 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:41.617 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:19:41.620 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:41.621 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:41.621 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:19:41.621 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:41.621 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:41.622 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:41.622 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:19:41.622 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:41.623 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:41.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:41.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:41.623 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:19:41.624 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:41.624 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:41.624 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:19:41.625 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:41.625 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:41.625 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:41.625 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:19:41.626 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:19:41.626 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:41.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:41.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:41.627 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:19:41.627 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:41.627 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:41.628 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:19:41.628 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:41.628 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:41.628 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:41.629 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:19:41.629 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:41.629 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:41.629 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:41.630 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:19:41.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:41.630 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:19:41.630 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:19:41.631 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:19:41.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:41.631 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:19:41Z" 2022-09-12T19:19:41.631 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:19:41.633 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:19:41Z" } 2022-09-12T19:19:41.633 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:19:41.634 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:19:41.635 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:19:41.635 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T19:19:41.636 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T19:19:41.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:19:41.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:19:41.637 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:19:41.697 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:19:41.698 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:19:41.737 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:19:41.734+0000 7f079fd5a600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:19:41.739 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:19:41.736+0000 7f079fd5a600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:19:41.753 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:19:41.754 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:19:41.754 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:19:41.755 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:19:41.755 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:19:41.756 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:19:41.756 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:19:41.757 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:19:41.757 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:19:41.757 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:19:41.758 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:19:41.758 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:19:41.759 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:19:41.759 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:19:41.759 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:19:41.760 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:19:41.760 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:19:41.760 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:19:41.760 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:19:41.761 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:19:41.761 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:19:41.761 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:19:41.761 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:19:41.762 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:19:41.762 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:19:41.762 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:19:41.763 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:19:41.763 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:19:41.763 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:19:41.763 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:19:41.764 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:19:41.764 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:19:41.765 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:19:41.765 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:19:41.765 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:19:41.766 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:19:41.766 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:19:41.766 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:19:41.767 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:19:41.767 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:19:41.767 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:19:41.768 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:19:41.768 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:19:41.768 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:19:41.768 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:19:41.769 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:19:41.769 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:19:41.769 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:19:41.770 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:19:41.770 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:19:41.770 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:19:41.771 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:19:41.771 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:19:41.771 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:19:41.772 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:19:41.772 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:19:41.772 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:19:41.772 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:19:41.773 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:19:41.773 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:19:41.773 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:19:41.773 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:19:41.774 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:19:41.774 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:19:41Z" 2022-09-12T19:19:41.775 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:19:41.776 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:19:41Z" } 2022-09-12T19:19:41.777 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:19:41.778 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:19:41.779 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:19:41.780 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T19:19:41.781 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T19:19:41.782 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:19:41.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:19:41.784 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:19:41.835 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:19:41.836 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:19:41.866 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:19:41.864+0000 7f7627c95600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:19:41.867 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:19:41.865+0000 7f7627c95600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:19:41.887 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:19:41.888 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:19:41.889 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:19:41.889 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:19:41.890 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:19:41.890 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:19:41.891 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:19:41.891 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:41.892 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:19:41.892 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:41.893 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:19:41.893 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:41.894 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:41.894 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:19:41.895 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:41.895 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:19:41.896 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:41.896 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:19:41.897 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:41.897 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:41.898 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:41.898 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:41.899 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:19:41.899 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:41.900 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:41.900 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:19:41.900 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:41.901 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:41.901 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:41.901 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:19:41.902 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:41.902 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:41.902 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:41.902 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:41.903 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:19:41.903 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:41.903 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:41.904 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:19:41.904 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:41.904 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:41.904 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:41.905 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:19:41.905 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:19:41.905 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:41.905 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:41.906 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:41.906 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:19:41.906 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:41.907 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:19:41.907 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:19:41.907 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:19:41.908 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:19:41.908 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:19:41.908 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:19:41.908 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:19:41.909 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:41.909 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:41.909 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:19:41.910 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:41.910 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:19:41.910 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:19:41.911 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:19:41.911 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:41.911 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:19:41Z" 2022-09-12T19:19:41.912 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:19:41.913 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:19:41Z" } 2022-09-12T19:19:41.914 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:19:41.914 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:19:41.915 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:19:41.915 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T19:19:41.916 DEBUG:boto:path=/jdpiak-49/ 2022-09-12T19:19:41.917 DEBUG:boto:auth_path=/jdpiak-49/ 2022-09-12T19:19:41.917 DEBUG:boto:Method: HEAD 2022-09-12T19:19:41.918 DEBUG:boto:Path: /jdpiak-49/ 2022-09-12T19:19:41.918 DEBUG:boto:Data: 2022-09-12T19:19:41.919 DEBUG:boto:Headers: {} 2022-09-12T19:19:41.919 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:19:41.920 DEBUG:boto:Port: 8000 2022-09-12T19:19:41.920 DEBUG:boto:Params: {} 2022-09-12T19:19:41.921 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:19:41.922 DEBUG:boto:Token: None 2022-09-12T19:19:41.922 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:19:41 GMT /jdpiak-49/ 2022-09-12T19:19:41.923 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DH2kJpUEmbqrvX/7ABKpNehqyBE= 2022-09-12T19:19:41.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:19:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DH2kJpUEmbqrvX/7ABKpNehqyBE=', 'Content-Length': '0'} 2022-09-12T19:19:41.930 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001cd6e206cbc74a41-00631f864d-1518-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:19:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:19:41.931 DEBUG:boto:path=/jdpiak-49/testredirect 2022-09-12T19:19:41.933 DEBUG:boto:auth_path=/jdpiak-49/testredirect 2022-09-12T19:19:41.934 DEBUG:boto:Method: HEAD 2022-09-12T19:19:41.935 DEBUG:boto:Path: /jdpiak-49/testredirect 2022-09-12T19:19:41.936 DEBUG:boto:Data: 2022-09-12T19:19:41.937 DEBUG:boto:Headers: {} 2022-09-12T19:19:41.937 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:19:41.938 DEBUG:boto:Port: 8000 2022-09-12T19:19:41.939 DEBUG:boto:Params: {} 2022-09-12T19:19:41.939 DEBUG:boto:Token: None 2022-09-12T19:19:41.940 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:19:41 GMT /jdpiak-49/testredirect 2022-09-12T19:19:41.940 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2HWQAcllIbtLymjV5RyEMohkaSw= 2022-09-12T19:19:41.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:19:41 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2HWQAcllIbtLymjV5RyEMohkaSw=', 'Content-Length': '0'} 2022-09-12T19:19:41.943 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000008ca39303db85e6f5-00631f864d-1518-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:19:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:19:41.943 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:19:41.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:19:41.945 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T19:19:41.993 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:19:41.993 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:19:42.018 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:19:42.011+0000 7f74a697c600 0 failed reading obj info from .rgw.root:zone_info.823fb3a5-4d8a-49dc-ae64-e05804bbea94: (2) No such file or directory 2022-09-12T19:19:42.018 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:19:42.011+0000 7f74a697c600 0 WARNING: could not read zone params for zone id=823fb3a5-4d8a-49dc-ae64-e05804bbea94 name=test-zone3 2022-09-12T19:19:42.019 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:19:42.011+0000 7f74a697c600 0 failed reading obj info from .rgw.root:zone_info.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47: (2) No such file or directory 2022-09-12T19:19:42.019 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:19:42.011+0000 7f74a697c600 0 WARNING: could not read zone params for zone id=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47 name=test-zone1 2022-09-12T19:19:42.023 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:19:42.024 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:19:42.024 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone2", 2022-09-12T19:19:42.024 INFO:teuthology.orchestra.run.smithi138.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-09-12T19:19:42.025 INFO:teuthology.orchestra.run.smithi138.stdout: "control_pool": "test-zone2.rgw.control", 2022-09-12T19:19:42.025 INFO:teuthology.orchestra.run.smithi138.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-09-12T19:19:42.025 INFO:teuthology.orchestra.run.smithi138.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-09-12T19:19:42.025 INFO:teuthology.orchestra.run.smithi138.stdout: "log_pool": "test-zone2.rgw.log", 2022-09-12T19:19:42.026 INFO:teuthology.orchestra.run.smithi138.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-09-12T19:19:42.026 INFO:teuthology.orchestra.run.smithi138.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-09-12T19:19:42.026 INFO:teuthology.orchestra.run.smithi138.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-09-12T19:19:42.027 INFO:teuthology.orchestra.run.smithi138.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-09-12T19:19:42.027 INFO:teuthology.orchestra.run.smithi138.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-09-12T19:19:42.027 INFO:teuthology.orchestra.run.smithi138.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-09-12T19:19:42.027 INFO:teuthology.orchestra.run.smithi138.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-09-12T19:19:42.028 INFO:teuthology.orchestra.run.smithi138.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-09-12T19:19:42.028 INFO:teuthology.orchestra.run.smithi138.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-09-12T19:19:42.028 INFO:teuthology.orchestra.run.smithi138.stdout: "system_key": { 2022-09-12T19:19:42.028 INFO:teuthology.orchestra.run.smithi138.stdout: "access_key": "RDIEFLSOLOEHUH1F", 2022-09-12T19:19:42.029 INFO:teuthology.orchestra.run.smithi138.stdout: "secret_key": "LDCKRd1RecZNgErNcVDnUigDhC7QggCQ" 2022-09-12T19:19:42.029 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:42.029 INFO:teuthology.orchestra.run.smithi138.stdout: "placement_pools": [ 2022-09-12T19:19:42.030 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:42.030 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "default-placement", 2022-09-12T19:19:42.030 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:19:42.030 INFO:teuthology.orchestra.run.smithi138.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-09-12T19:19:42.031 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_classes": { 2022-09-12T19:19:42.031 INFO:teuthology.orchestra.run.smithi138.stdout: "STANDARD": { 2022-09-12T19:19:42.031 INFO:teuthology.orchestra.run.smithi138.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2022-09-12T19:19:42.031 INFO:teuthology.orchestra.run.smithi138.stdout: "compression_type": "random" 2022-09-12T19:19:42.032 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:42.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:42.032 INFO:teuthology.orchestra.run.smithi138.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-09-12T19:19:42.032 INFO:teuthology.orchestra.run.smithi138.stdout: "index_type": 0 2022-09-12T19:19:42.033 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:42.033 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:42.033 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:19:42.033 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:19:42.034 INFO:teuthology.orchestra.run.smithi138.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-09-12T19:19:42.034 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:19:42.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:19:42.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:19:42.037 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T19:19:42.087 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:19:42.087 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:19:42.149 INFO:teuthology.orchestra.run.smithi138.stderr:Sending period to new master zone fb29b4fb-1f78-4524-ab80-6a5e9e01ab47 2022-09-12T19:19:42.185 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:19:42.185 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:19:42.186 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 3, 2022-09-12T19:19:42.186 INFO:teuthology.orchestra.run.smithi138.stdout: "predecessor_uuid": "2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e", 2022-09-12T19:19:42.187 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": [ 2022-09-12T19:19:42.187 INFO:teuthology.orchestra.run.smithi138.stdout: "", 2022-09-12T19:19:42.188 INFO:teuthology.orchestra.run.smithi138.stdout: "", 2022-09-12T19:19:42.188 INFO:teuthology.orchestra.run.smithi138.stdout: "1_1663010125.771044_393.1", 2022-09-12T19:19:42.189 INFO:teuthology.orchestra.run.smithi138.stdout: "" 2022-09-12T19:19:42.189 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:19:42.189 INFO:teuthology.orchestra.run.smithi138.stdout: "period_map": { 2022-09-12T19:19:42.190 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:19:42.193 INFO:teuthology.orchestra.run.smithi138.stdout: "zonegroups": [ 2022-09-12T19:19:42.193 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:42.194 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "test-zonegroup", 2022-09-12T19:19:42.194 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zonegroup", 2022-09-12T19:19:42.195 INFO:teuthology.orchestra.run.smithi138.stdout: "api_name": "", 2022-09-12T19:19:42.195 INFO:teuthology.orchestra.run.smithi138.stdout: "is_master": true, 2022-09-12T19:19:42.195 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:19:42.195 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T19:19:42.196 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:19:42.196 INFO:teuthology.orchestra.run.smithi138.stdout: "hostnames": [], 2022-09-12T19:19:42.196 INFO:teuthology.orchestra.run.smithi138.stdout: "hostnames_s3website": [], 2022-09-12T19:19:42.196 INFO:teuthology.orchestra.run.smithi138.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:19:42.197 INFO:teuthology.orchestra.run.smithi138.stdout: "zones": [ 2022-09-12T19:19:42.197 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:42.197 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T19:19:42.197 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone3", 2022-09-12T19:19:42.198 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:19:42.198 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi035.front.sepia.ceph.com:8001" 2022-09-12T19:19:42.198 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:19:42.199 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T19:19:42.199 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T19:19:42.199 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:19:42.199 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T19:19:42.200 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "", 2022-09-12T19:19:42.200 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": true, 2022-09-12T19:19:42.200 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [], 2022-09-12T19:19:42.200 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T19:19:42.201 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T19:19:42.201 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:19:42.201 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:19:42.201 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:42.201 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:42.202 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "eb41eca3-59e3-4448-adac-b7c5067fb406", 2022-09-12T19:19:42.202 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone4", 2022-09-12T19:19:42.202 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:19:42.202 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi138.front.sepia.ceph.com:8001" 2022-09-12T19:19:42.203 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:19:42.203 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T19:19:42.203 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T19:19:42.203 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:19:42.204 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T19:19:42.204 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "pubsub", 2022-09-12T19:19:42.204 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": false, 2022-09-12T19:19:42.204 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [ 2022-09-12T19:19:42.205 INFO:teuthology.orchestra.run.smithi138.stdout: "test-zone1" 2022-09-12T19:19:42.205 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:19:42.205 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T19:19:42.205 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T19:19:42.206 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:19:42.206 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:19:42.206 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:42.206 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:42.207 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:19:42.207 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone1", 2022-09-12T19:19:42.207 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:19:42.208 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T19:19:42.208 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:19:42.209 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T19:19:42.209 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T19:19:42.209 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:19:42.209 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T19:19:42.210 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "", 2022-09-12T19:19:42.210 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": true, 2022-09-12T19:19:42.211 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [], 2022-09-12T19:19:42.211 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T19:19:42.211 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T19:19:42.211 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:19:42.212 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:19:42.212 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:42.212 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:42.212 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:19:42.213 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone2", 2022-09-12T19:19:42.213 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:19:42.213 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi138.front.sepia.ceph.com:8000" 2022-09-12T19:19:42.213 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:19:42.214 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T19:19:42.214 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T19:19:42.214 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:19:42.214 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T19:19:42.215 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "", 2022-09-12T19:19:42.215 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": false, 2022-09-12T19:19:42.215 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [], 2022-09-12T19:19:42.215 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:19:42.216 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T19:19:42.216 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:19:42.216 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:19:42.216 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:42.217 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:19:42.217 INFO:teuthology.orchestra.run.smithi138.stdout: "placement_targets": [ 2022-09-12T19:19:42.217 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:42.219 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "default-placement", 2022-09-12T19:19:42.219 INFO:teuthology.orchestra.run.smithi138.stdout: "tags": [], 2022-09-12T19:19:42.219 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_classes": [ 2022-09-12T19:19:42.220 INFO:teuthology.orchestra.run.smithi138.stdout: "STANDARD" 2022-09-12T19:19:42.220 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:19:42.220 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:42.220 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:19:42.221 INFO:teuthology.orchestra.run.smithi138.stdout: "default_placement": "default-placement", 2022-09-12T19:19:42.221 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:19:42.221 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_policy": { 2022-09-12T19:19:42.221 INFO:teuthology.orchestra.run.smithi138.stdout: "groups": [] 2022-09-12T19:19:42.222 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:42.222 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled_features": [ 2022-09-12T19:19:42.222 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:19:42.222 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:19:42.223 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:42.223 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:19:42.223 INFO:teuthology.orchestra.run.smithi138.stdout: "short_zone_ids": [ 2022-09-12T19:19:42.223 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:42.224 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T19:19:42.224 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 227168046 2022-09-12T19:19:42.224 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:42.224 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:42.225 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "eb41eca3-59e3-4448-adac-b7c5067fb406", 2022-09-12T19:19:42.225 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 2080702172 2022-09-12T19:19:42.225 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:42.225 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:42.226 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:19:42.226 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 422195086 2022-09-12T19:19:42.226 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:42.227 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:19:42.228 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:19:42.228 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 2881234289 2022-09-12T19:19:42.228 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:42.229 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:19:42.229 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:42.230 INFO:teuthology.orchestra.run.smithi138.stdout: "master_zonegroup": "test-zonegroup", 2022-09-12T19:19:42.230 INFO:teuthology.orchestra.run.smithi138.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:19:42.231 INFO:teuthology.orchestra.run.smithi138.stdout: "period_config": { 2022-09-12T19:19:42.231 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_quota": { 2022-09-12T19:19:42.232 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false, 2022-09-12T19:19:42.233 INFO:teuthology.orchestra.run.smithi138.stdout: "check_on_raw": false, 2022-09-12T19:19:42.233 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size": -1, 2022-09-12T19:19:42.234 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size_kb": 0, 2022-09-12T19:19:42.234 INFO:teuthology.orchestra.run.smithi138.stdout: "max_objects": -1 2022-09-12T19:19:42.235 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:42.235 INFO:teuthology.orchestra.run.smithi138.stdout: "user_quota": { 2022-09-12T19:19:42.236 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false, 2022-09-12T19:19:42.236 INFO:teuthology.orchestra.run.smithi138.stdout: "check_on_raw": false, 2022-09-12T19:19:42.237 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size": -1, 2022-09-12T19:19:42.237 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size_kb": 0, 2022-09-12T19:19:42.238 INFO:teuthology.orchestra.run.smithi138.stdout: "max_objects": -1 2022-09-12T19:19:42.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:42.238 INFO:teuthology.orchestra.run.smithi138.stdout: "user_ratelimit": { 2022-09-12T19:19:42.239 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_ops": 0, 2022-09-12T19:19:42.239 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_ops": 0, 2022-09-12T19:19:42.240 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_bytes": 0, 2022-09-12T19:19:42.240 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_bytes": 0, 2022-09-12T19:19:42.241 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false 2022-09-12T19:19:42.242 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:42.242 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_ratelimit": { 2022-09-12T19:19:42.243 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_ops": 0, 2022-09-12T19:19:42.243 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_ops": 0, 2022-09-12T19:19:42.243 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_bytes": 0, 2022-09-12T19:19:42.244 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_bytes": 0, 2022-09-12T19:19:42.244 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false 2022-09-12T19:19:42.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:42.244 INFO:teuthology.orchestra.run.smithi138.stdout: "anonymous_ratelimit": { 2022-09-12T19:19:42.245 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_ops": 0, 2022-09-12T19:19:42.245 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_ops": 0, 2022-09-12T19:19:42.245 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_bytes": 0, 2022-09-12T19:19:42.245 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_bytes": 0, 2022-09-12T19:19:42.246 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false 2022-09-12T19:19:42.246 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:19:42.246 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:19:42.247 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:19:42.247 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_name": "test-realm", 2022-09-12T19:19:42.247 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:19:42.247 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:19:42.248 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47" 2022-09-12T19:20:42.250 DEBUG:boto:path=/jdpiak-49/testredirect 2022-09-12T19:20:42.252 DEBUG:boto:auth_path=/jdpiak-49/testredirect 2022-09-12T19:20:42.253 DEBUG:boto:Method: HEAD 2022-09-12T19:20:42.253 DEBUG:boto:Path: /jdpiak-49/testredirect 2022-09-12T19:20:42.254 DEBUG:boto:Data: 2022-09-12T19:20:42.254 DEBUG:boto:Headers: {} 2022-09-12T19:20:42.255 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:20:42.255 DEBUG:boto:Port: 8000 2022-09-12T19:20:42.256 DEBUG:boto:Params: {} 2022-09-12T19:20:42.257 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:20:42.258 DEBUG:boto:Token: None 2022-09-12T19:20:42.258 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:20:42 GMT /jdpiak-49/testredirect 2022-09-12T19:20:42.259 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WkoRJPJ6kLSsxjBBGnCgHiJ9MR4= 2022-09-12T19:20:42.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:20:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WkoRJPJ6kLSsxjBBGnCgHiJ9MR4=', 'Content-Length': '0'} 2022-09-12T19:20:42.265 DEBUG:boto:Response headers: [('Location', 'http://smithi035.front.sepia.ceph.com:8000/jdpiak-49/testredirect?'), ('x-amz-request-id', 'tx0000047774a75b396f65d-00631f868a-1580-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:20:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:20:42.266 DEBUG:boto:Redirecting: http://smithi035.front.sepia.ceph.com/jdpiak-49/testredirect 2022-09-12T19:20:42.267 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': '8000'} 2022-09-12T19:20:42.267 DEBUG:boto:Token: None 2022-09-12T19:20:42.268 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:20:42 GMT /jdpiak-49/testredirect 2022-09-12T19:20:42.269 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WkoRJPJ6kLSsxjBBGnCgHiJ9MR4= 2022-09-12T19:20:42.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:20:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WkoRJPJ6kLSsxjBBGnCgHiJ9MR4=', 'Content-Length': '0'} 2022-09-12T19:20:42.277 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:18:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000007c3cbca67e19834e-00631f868a-1554-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:20:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:20:42.279 DEBUG:boto:path=/jdpiak-49/testredirect 2022-09-12T19:20:42.280 DEBUG:boto:auth_path=/jdpiak-49/testredirect 2022-09-12T19:20:42.281 DEBUG:boto:Method: GET 2022-09-12T19:20:42.282 DEBUG:boto:Path: /jdpiak-49/testredirect 2022-09-12T19:20:42.283 DEBUG:boto:Data: 2022-09-12T19:20:42.284 DEBUG:boto:Headers: {} 2022-09-12T19:20:42.285 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:20:42.286 DEBUG:boto:Port: 8000 2022-09-12T19:20:42.287 DEBUG:boto:Params: {} 2022-09-12T19:20:42.287 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:20:42.288 DEBUG:boto:Token: None 2022-09-12T19:20:42.290 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:20:42 GMT /jdpiak-49/testredirect 2022-09-12T19:20:42.291 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UuxH/VnteNBbrFQlLcvGD4qTfZI= 2022-09-12T19:20:42.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:20:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UuxH/VnteNBbrFQlLcvGD4qTfZI=', 'Content-Length': '0'} 2022-09-12T19:20:42.295 DEBUG:boto:Response headers: [('Location', 'http://smithi035.front.sepia.ceph.com:8000/jdpiak-49/testredirect?'), ('x-amz-request-id', 'tx00000da4cdd1d6f61736f-00631f868a-1580-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:20:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:20:42.296 DEBUG:boto:Redirecting: http://smithi035.front.sepia.ceph.com/jdpiak-49/testredirect 2022-09-12T19:20:42.297 DEBUG:boto:Token: None 2022-09-12T19:20:42.298 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:20:42 GMT /jdpiak-49/testredirect 2022-09-12T19:20:42.299 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UuxH/VnteNBbrFQlLcvGD4qTfZI= 2022-09-12T19:20:42.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:20:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UuxH/VnteNBbrFQlLcvGD4qTfZI=', 'Content-Length': '0'} 2022-09-12T19:20:42.303 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:18:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000002eb3bea87a860968-00631f868a-1554-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:20:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:20:42.304 DEBUG:boto:path=/jdpiak-49/testredirect 2022-09-12T19:20:42.305 DEBUG:boto:auth_path=/jdpiak-49/testredirect 2022-09-12T19:20:42.306 DEBUG:boto:Method: PUT 2022-09-12T19:20:42.307 DEBUG:boto:Path: /jdpiak-49/testredirect 2022-09-12T19:20:42.308 DEBUG:boto:Data: 2022-09-12T19:20:42.308 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-09-12T19:20:42.309 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:20:42.310 DEBUG:boto:Port: 8000 2022-09-12T19:20:42.311 DEBUG:boto:Params: {} 2022-09-12T19:20:42.312 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:20:42.313 DEBUG:boto:Token: None 2022-09-12T19:20:42.314 DEBUG:boto:StringToSign: PUT VpBzljOcorCZvRIkX5Nt3A== application/octet-stream Mon, 12 Sep 2022 19:20:42 GMT /jdpiak-49/testredirect 2022-09-12T19:20:42.315 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kjbykwTB2Uh0Um8slayL22G5OW0= 2022-09-12T19:20:42.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:20:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kjbykwTB2Uh0Um8slayL22G5OW0='} 2022-09-12T19:20:42.328 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001bed9116a3eb13f2-00631f868a-1554-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:20:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:20:42.330 DEBUG:boto:path=/jdpiak-49/testredirect 2022-09-12T19:20:42.330 DEBUG:boto:auth_path=/jdpiak-49/testredirect 2022-09-12T19:20:42.331 DEBUG:boto:Method: GET 2022-09-12T19:20:42.332 DEBUG:boto:Path: /jdpiak-49/testredirect 2022-09-12T19:20:42.333 DEBUG:boto:Data: 2022-09-12T19:20:42.334 DEBUG:boto:Headers: {} 2022-09-12T19:20:42.335 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:20:42.336 DEBUG:boto:Port: 8000 2022-09-12T19:20:42.337 DEBUG:boto:Params: {} 2022-09-12T19:20:42.338 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:20:42.339 DEBUG:boto:Token: None 2022-09-12T19:20:42.339 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:20:42 GMT /jdpiak-49/testredirect 2022-09-12T19:20:42.340 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UuxH/VnteNBbrFQlLcvGD4qTfZI= 2022-09-12T19:20:42.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:20:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UuxH/VnteNBbrFQlLcvGD4qTfZI=', 'Content-Length': '0'} 2022-09-12T19:20:42.345 DEBUG:boto:Response headers: [('Location', 'http://smithi035.front.sepia.ceph.com:8000/jdpiak-49/testredirect?'), ('x-amz-request-id', 'tx00000c1c70884dce79b0d-00631f868a-1580-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:20:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:20:42.346 DEBUG:boto:Redirecting: http://smithi035.front.sepia.ceph.com/jdpiak-49/testredirect 2022-09-12T19:20:42.347 DEBUG:boto:Token: None 2022-09-12T19:20:42.348 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:20:42 GMT /jdpiak-49/testredirect 2022-09-12T19:20:42.348 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UuxH/VnteNBbrFQlLcvGD4qTfZI= 2022-09-12T19:20:42.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:20:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UuxH/VnteNBbrFQlLcvGD4qTfZI=', 'Content-Length': '0'} 2022-09-12T19:20:42.352 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:20:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('x-amz-request-id', 'tx000009a597624e5537f8f-00631f868a-1554-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:20:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:20:42.354 DEBUG:boto:path=/jdpiak-49/testredirect 2022-09-12T19:20:42.355 DEBUG:boto:auth_path=/jdpiak-49/testredirect 2022-09-12T19:20:42.355 DEBUG:boto:Method: PUT 2022-09-12T19:20:42.356 DEBUG:boto:Path: /jdpiak-49/testredirect 2022-09-12T19:20:42.357 DEBUG:boto:Data: 2022-09-12T19:20:42.358 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-09-12T19:20:42.359 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:20:42.359 DEBUG:boto:Port: 8000 2022-09-12T19:20:42.360 DEBUG:boto:Params: {} 2022-09-12T19:20:42.360 DEBUG:boto:Token: None 2022-09-12T19:20:42.361 DEBUG:boto:StringToSign: PUT uk9S5NXZfBvPq4jGr+LM5g== application/octet-stream Mon, 12 Sep 2022 19:20:42 GMT /jdpiak-49/testredirect 2022-09-12T19:20:42.362 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JWeMxcLeYcRnMRlzU0JJnlrGpHE= 2022-09-12T19:20:42.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:20:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JWeMxcLeYcRnMRlzU0JJnlrGpHE='} 2022-09-12T19:20:42.368 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000ff0980bb95a27d2-00631f868a-1554-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:20:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:20:42.368 DEBUG:boto:path=/jdpiak-49/testredirect 2022-09-12T19:20:42.369 DEBUG:boto:auth_path=/jdpiak-49/testredirect 2022-09-12T19:20:42.369 DEBUG:boto:Method: GET 2022-09-12T19:20:42.370 DEBUG:boto:Path: /jdpiak-49/testredirect 2022-09-12T19:20:42.370 DEBUG:boto:Data: 2022-09-12T19:20:42.371 DEBUG:boto:Headers: {} 2022-09-12T19:20:42.371 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:20:42.372 DEBUG:boto:Port: 8000 2022-09-12T19:20:42.372 DEBUG:boto:Params: {} 2022-09-12T19:20:42.373 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:20:42.374 DEBUG:boto:Token: None 2022-09-12T19:20:42.375 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:20:42 GMT /jdpiak-49/testredirect 2022-09-12T19:20:42.376 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UuxH/VnteNBbrFQlLcvGD4qTfZI= 2022-09-12T19:20:42.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:20:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UuxH/VnteNBbrFQlLcvGD4qTfZI=', 'Content-Length': '0'} 2022-09-12T19:20:42.380 DEBUG:boto:Response headers: [('Location', 'http://smithi035.front.sepia.ceph.com:8000/jdpiak-49/testredirect?'), ('x-amz-request-id', 'tx0000008f9a3b6d6f0011b-00631f868a-1580-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:20:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:20:42.381 DEBUG:boto:Redirecting: http://smithi035.front.sepia.ceph.com/jdpiak-49/testredirect 2022-09-12T19:20:42.382 DEBUG:boto:Token: None 2022-09-12T19:20:42.383 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:20:42 GMT /jdpiak-49/testredirect 2022-09-12T19:20:42.384 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UuxH/VnteNBbrFQlLcvGD4qTfZI= 2022-09-12T19:20:42.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:20:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UuxH/VnteNBbrFQlLcvGD4qTfZI=', 'Content-Length': '0'} 2022-09-12T19:20:42.388 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:20:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('x-amz-request-id', 'tx000003cc80615ecefc600-00631f868a-1554-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:20:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:20:42.390 DEBUG:boto:path=/jdpiak-49/testredirect 2022-09-12T19:20:42.391 DEBUG:boto:auth_path=/jdpiak-49/testredirect 2022-09-12T19:20:42.392 DEBUG:boto:Method: PUT 2022-09-12T19:20:42.392 DEBUG:boto:Path: /jdpiak-49/testredirect 2022-09-12T19:20:42.393 DEBUG:boto:Data: 2022-09-12T19:20:42.394 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-09-12T19:20:42.395 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:20:42.396 DEBUG:boto:Port: 8000 2022-09-12T19:20:42.396 DEBUG:boto:Params: {} 2022-09-12T19:20:42.397 DEBUG:boto:Token: None 2022-09-12T19:20:42.397 DEBUG:boto:StringToSign: PUT 6aKmvaJMgaK7Z2rfDZvVeg== application/octet-stream Mon, 12 Sep 2022 19:20:42 GMT /jdpiak-49/testredirect 2022-09-12T19:20:42.398 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xU8a791zo4O9w89mk7PQj6+QVrs= 2022-09-12T19:20:42.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:20:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xU8a791zo4O9w89mk7PQj6+QVrs='} 2022-09-12T19:20:42.404 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a8240c6b8cc9533b-00631f868a-1554-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:20:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:20:42.405 DEBUG:boto:path=/jdpiak-49/testredirect 2022-09-12T19:20:42.406 DEBUG:boto:auth_path=/jdpiak-49/testredirect 2022-09-12T19:20:42.406 DEBUG:boto:Method: GET 2022-09-12T19:20:42.407 DEBUG:boto:Path: /jdpiak-49/testredirect 2022-09-12T19:20:42.407 DEBUG:boto:Data: 2022-09-12T19:20:42.408 DEBUG:boto:Headers: {} 2022-09-12T19:20:42.408 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:20:42.409 DEBUG:boto:Port: 8000 2022-09-12T19:20:42.410 DEBUG:boto:Params: {} 2022-09-12T19:20:42.410 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:20:42.411 DEBUG:boto:Token: None 2022-09-12T19:20:42.411 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:20:42 GMT /jdpiak-49/testredirect 2022-09-12T19:20:42.412 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UuxH/VnteNBbrFQlLcvGD4qTfZI= 2022-09-12T19:20:42.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:20:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UuxH/VnteNBbrFQlLcvGD4qTfZI=', 'Content-Length': '0'} 2022-09-12T19:20:42.416 DEBUG:boto:Response headers: [('Location', 'http://smithi035.front.sepia.ceph.com:8000/jdpiak-49/testredirect?'), ('x-amz-request-id', 'tx000005df94873ba23b485-00631f868a-1580-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:20:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:20:42.417 DEBUG:boto:Redirecting: http://smithi035.front.sepia.ceph.com/jdpiak-49/testredirect 2022-09-12T19:20:42.417 DEBUG:boto:Token: None 2022-09-12T19:20:42.418 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:20:42 GMT /jdpiak-49/testredirect 2022-09-12T19:20:42.419 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UuxH/VnteNBbrFQlLcvGD4qTfZI= 2022-09-12T19:20:42.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:20:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UuxH/VnteNBbrFQlLcvGD4qTfZI=', 'Content-Length': '0'} 2022-09-12T19:20:42.422 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:20:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('x-amz-request-id', 'tx000005b44f4c42b30b284-00631f868a-1554-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:20:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:20:42.423 DEBUG:boto:path=/jdpiak-49/testredirect 2022-09-12T19:20:42.423 DEBUG:boto:auth_path=/jdpiak-49/testredirect 2022-09-12T19:20:42.424 DEBUG:boto:Method: PUT 2022-09-12T19:20:42.425 DEBUG:boto:Path: /jdpiak-49/testredirect 2022-09-12T19:20:42.426 DEBUG:boto:Data: 2022-09-12T19:20:42.427 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-09-12T19:20:42.428 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:20:42.428 DEBUG:boto:Port: 8000 2022-09-12T19:20:42.429 DEBUG:boto:Params: {} 2022-09-12T19:20:42.430 DEBUG:boto:Token: None 2022-09-12T19:20:42.431 DEBUG:boto:StringToSign: PUT GbNgNQ5MAukCtgY+rsT5Xw== application/octet-stream Mon, 12 Sep 2022 19:20:42 GMT /jdpiak-49/testredirect 2022-09-12T19:20:42.432 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+58k6CGh/XojjQa/HIlNrVAn9eo= 2022-09-12T19:20:42.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:20:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+58k6CGh/XojjQa/HIlNrVAn9eo='} 2022-09-12T19:20:42.482 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009826a274404efac9-00631f868a-1554-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:20:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:20:42.483 DEBUG:boto:path=/jdpiak-49/testredirect 2022-09-12T19:20:42.484 DEBUG:boto:auth_path=/jdpiak-49/testredirect 2022-09-12T19:20:42.484 DEBUG:boto:Method: GET 2022-09-12T19:20:42.485 DEBUG:boto:Path: /jdpiak-49/testredirect 2022-09-12T19:20:42.485 DEBUG:boto:Data: 2022-09-12T19:20:42.486 DEBUG:boto:Headers: {} 2022-09-12T19:20:42.486 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:20:42.487 DEBUG:boto:Port: 8000 2022-09-12T19:20:42.487 DEBUG:boto:Params: {} 2022-09-12T19:20:42.488 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:20:42.489 DEBUG:boto:Token: None 2022-09-12T19:20:42.489 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:20:42 GMT /jdpiak-49/testredirect 2022-09-12T19:20:42.490 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UuxH/VnteNBbrFQlLcvGD4qTfZI= 2022-09-12T19:20:42.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:20:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UuxH/VnteNBbrFQlLcvGD4qTfZI=', 'Content-Length': '0'} 2022-09-12T19:20:42.493 DEBUG:boto:Response headers: [('Location', 'http://smithi035.front.sepia.ceph.com:8000/jdpiak-49/testredirect?'), ('x-amz-request-id', 'tx00000e583f97ce0d4910b-00631f868a-1580-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:20:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:20:42.494 DEBUG:boto:Redirecting: http://smithi035.front.sepia.ceph.com/jdpiak-49/testredirect 2022-09-12T19:20:42.495 DEBUG:boto:Token: None 2022-09-12T19:20:42.496 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:20:42 GMT /jdpiak-49/testredirect 2022-09-12T19:20:42.496 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UuxH/VnteNBbrFQlLcvGD4qTfZI= 2022-09-12T19:20:42.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:20:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UuxH/VnteNBbrFQlLcvGD4qTfZI=', 'Content-Length': '0'} 2022-09-12T19:20:42.499 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:20:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('x-amz-request-id', 'tx0000069026746158c6030-00631f868a-1554-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:20:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:20:42.500 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:20:42.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:20:42.502 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=true --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T19:20:42.560 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:20:42.560 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:20:42.583 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:20:42.580+0000 7fa3960f5600 0 failed reading obj info from .rgw.root:zone_info.823fb3a5-4d8a-49dc-ae64-e05804bbea94: (2) No such file or directory 2022-09-12T19:20:42.584 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:20:42.580+0000 7fa3960f5600 0 WARNING: could not read zone params for zone id=823fb3a5-4d8a-49dc-ae64-e05804bbea94 name=test-zone3 2022-09-12T19:20:42.585 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:20:42.581+0000 7fa3960f5600 0 failed reading obj info from .rgw.root:zone_info.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47: (2) No such file or directory 2022-09-12T19:20:42.585 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:20:42.581+0000 7fa3960f5600 0 WARNING: could not read zone params for zone id=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47 name=test-zone1 2022-09-12T19:20:42.589 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:20:42.590 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:20:42.590 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone2", 2022-09-12T19:20:42.591 INFO:teuthology.orchestra.run.smithi138.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-09-12T19:20:42.591 INFO:teuthology.orchestra.run.smithi138.stdout: "control_pool": "test-zone2.rgw.control", 2022-09-12T19:20:42.592 INFO:teuthology.orchestra.run.smithi138.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-09-12T19:20:42.592 INFO:teuthology.orchestra.run.smithi138.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-09-12T19:20:42.593 INFO:teuthology.orchestra.run.smithi138.stdout: "log_pool": "test-zone2.rgw.log", 2022-09-12T19:20:42.593 INFO:teuthology.orchestra.run.smithi138.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-09-12T19:20:42.594 INFO:teuthology.orchestra.run.smithi138.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-09-12T19:20:42.594 INFO:teuthology.orchestra.run.smithi138.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-09-12T19:20:42.595 INFO:teuthology.orchestra.run.smithi138.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-09-12T19:20:42.595 INFO:teuthology.orchestra.run.smithi138.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-09-12T19:20:42.596 INFO:teuthology.orchestra.run.smithi138.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-09-12T19:20:42.596 INFO:teuthology.orchestra.run.smithi138.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-09-12T19:20:42.597 INFO:teuthology.orchestra.run.smithi138.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-09-12T19:20:42.597 INFO:teuthology.orchestra.run.smithi138.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-09-12T19:20:42.598 INFO:teuthology.orchestra.run.smithi138.stdout: "system_key": { 2022-09-12T19:20:42.598 INFO:teuthology.orchestra.run.smithi138.stdout: "access_key": "RDIEFLSOLOEHUH1F", 2022-09-12T19:20:42.599 INFO:teuthology.orchestra.run.smithi138.stdout: "secret_key": "LDCKRd1RecZNgErNcVDnUigDhC7QggCQ" 2022-09-12T19:20:42.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:20:42.600 INFO:teuthology.orchestra.run.smithi138.stdout: "placement_pools": [ 2022-09-12T19:20:42.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:20:42.601 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "default-placement", 2022-09-12T19:20:42.601 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:20:42.602 INFO:teuthology.orchestra.run.smithi138.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-09-12T19:20:42.602 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_classes": { 2022-09-12T19:20:42.603 INFO:teuthology.orchestra.run.smithi138.stdout: "STANDARD": { 2022-09-12T19:20:42.603 INFO:teuthology.orchestra.run.smithi138.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2022-09-12T19:20:42.604 INFO:teuthology.orchestra.run.smithi138.stdout: "compression_type": "random" 2022-09-12T19:20:42.604 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:20:42.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:20:42.605 INFO:teuthology.orchestra.run.smithi138.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-09-12T19:20:42.606 INFO:teuthology.orchestra.run.smithi138.stdout: "index_type": 0 2022-09-12T19:20:42.606 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:20:42.607 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:20:42.607 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:20:42.608 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:20:42.608 INFO:teuthology.orchestra.run.smithi138.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-09-12T19:20:42.609 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:20:42.611 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:20:42.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:20:42.612 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T19:20:42.663 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:20:42.664 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:20:42.715 INFO:teuthology.orchestra.run.smithi138.stderr:Sending period to new master zone fb29b4fb-1f78-4524-ab80-6a5e9e01ab47 2022-09-12T19:20:42.745 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:20:42.746 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:20:42.746 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 4, 2022-09-12T19:20:42.746 INFO:teuthology.orchestra.run.smithi138.stdout: "predecessor_uuid": "2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e", 2022-09-12T19:20:42.747 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": [ 2022-09-12T19:20:42.747 INFO:teuthology.orchestra.run.smithi138.stdout: "", 2022-09-12T19:20:42.748 INFO:teuthology.orchestra.run.smithi138.stdout: "", 2022-09-12T19:20:42.748 INFO:teuthology.orchestra.run.smithi138.stdout: "1_1663010125.771044_393.1", 2022-09-12T19:20:42.749 INFO:teuthology.orchestra.run.smithi138.stdout: "" 2022-09-12T19:20:42.749 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:20:42.749 INFO:teuthology.orchestra.run.smithi138.stdout: "period_map": { 2022-09-12T19:20:42.750 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:20:42.751 INFO:teuthology.orchestra.run.smithi138.stdout: "zonegroups": [ 2022-09-12T19:20:42.751 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:20:42.752 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "test-zonegroup", 2022-09-12T19:20:42.752 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zonegroup", 2022-09-12T19:20:42.753 INFO:teuthology.orchestra.run.smithi138.stdout: "api_name": "", 2022-09-12T19:20:42.753 INFO:teuthology.orchestra.run.smithi138.stdout: "is_master": true, 2022-09-12T19:20:42.754 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:20:42.754 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T19:20:42.754 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:20:42.755 INFO:teuthology.orchestra.run.smithi138.stdout: "hostnames": [], 2022-09-12T19:20:42.755 INFO:teuthology.orchestra.run.smithi138.stdout: "hostnames_s3website": [], 2022-09-12T19:20:42.756 INFO:teuthology.orchestra.run.smithi138.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:20:42.756 INFO:teuthology.orchestra.run.smithi138.stdout: "zones": [ 2022-09-12T19:20:42.757 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:20:42.757 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T19:20:42.757 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone3", 2022-09-12T19:20:42.758 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:20:42.758 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi035.front.sepia.ceph.com:8001" 2022-09-12T19:20:42.759 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:20:42.759 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T19:20:42.760 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T19:20:42.760 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:20:42.761 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T19:20:42.761 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "", 2022-09-12T19:20:42.761 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": true, 2022-09-12T19:20:42.762 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [], 2022-09-12T19:20:42.762 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T19:20:42.762 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T19:20:42.763 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:20:42.763 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:20:42.764 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:20:42.764 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:20:42.764 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "eb41eca3-59e3-4448-adac-b7c5067fb406", 2022-09-12T19:20:42.765 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone4", 2022-09-12T19:20:42.765 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:20:42.766 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi138.front.sepia.ceph.com:8001" 2022-09-12T19:20:42.766 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:20:42.766 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T19:20:42.767 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T19:20:42.767 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:20:42.768 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T19:20:42.768 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "pubsub", 2022-09-12T19:20:42.769 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": false, 2022-09-12T19:20:42.769 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [ 2022-09-12T19:20:42.769 INFO:teuthology.orchestra.run.smithi138.stdout: "test-zone1" 2022-09-12T19:20:42.770 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:20:42.770 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T19:20:42.771 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T19:20:42.771 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:20:42.771 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:20:42.772 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:20:42.772 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:20:42.773 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:20:42.773 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone1", 2022-09-12T19:20:42.774 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:20:42.774 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T19:20:42.774 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:20:42.775 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T19:20:42.775 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T19:20:42.776 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:20:42.776 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T19:20:42.777 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "", 2022-09-12T19:20:42.778 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": true, 2022-09-12T19:20:42.779 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [], 2022-09-12T19:20:42.779 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T19:20:42.779 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T19:20:42.779 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:20:42.780 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:20:42.780 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:20:42.780 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:20:42.780 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:20:42.781 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone2", 2022-09-12T19:20:42.781 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:20:42.781 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi138.front.sepia.ceph.com:8000" 2022-09-12T19:20:42.781 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:20:42.782 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T19:20:42.782 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T19:20:42.782 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:20:42.782 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T19:20:42.783 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "", 2022-09-12T19:20:42.783 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": true, 2022-09-12T19:20:42.783 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [], 2022-09-12T19:20:42.783 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:20:42.784 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T19:20:42.784 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:20:42.784 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:20:42.784 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:20:42.785 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:20:42.785 INFO:teuthology.orchestra.run.smithi138.stdout: "placement_targets": [ 2022-09-12T19:20:42.785 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:20:42.787 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "default-placement", 2022-09-12T19:20:42.787 INFO:teuthology.orchestra.run.smithi138.stdout: "tags": [], 2022-09-12T19:20:42.787 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_classes": [ 2022-09-12T19:20:42.787 INFO:teuthology.orchestra.run.smithi138.stdout: "STANDARD" 2022-09-12T19:20:42.788 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:20:42.788 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:20:42.788 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:20:42.788 INFO:teuthology.orchestra.run.smithi138.stdout: "default_placement": "default-placement", 2022-09-12T19:20:42.789 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:20:42.789 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_policy": { 2022-09-12T19:20:42.789 INFO:teuthology.orchestra.run.smithi138.stdout: "groups": [] 2022-09-12T19:20:42.789 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:20:42.790 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled_features": [ 2022-09-12T19:20:42.790 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:20:42.790 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:20:42.790 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:20:42.791 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:20:42.791 INFO:teuthology.orchestra.run.smithi138.stdout: "short_zone_ids": [ 2022-09-12T19:20:42.791 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:20:42.791 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T19:20:42.792 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 227168046 2022-09-12T19:20:42.792 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:20:42.792 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:20:42.792 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "eb41eca3-59e3-4448-adac-b7c5067fb406", 2022-09-12T19:20:42.793 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 2080702172 2022-09-12T19:20:42.793 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:20:42.793 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:20:42.793 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:20:42.793 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 422195086 2022-09-12T19:20:42.794 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:20:42.794 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:20:42.794 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:20:42.794 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 2881234289 2022-09-12T19:20:42.795 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:20:42.795 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:20:42.795 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:20:42.795 INFO:teuthology.orchestra.run.smithi138.stdout: "master_zonegroup": "test-zonegroup", 2022-09-12T19:20:42.796 INFO:teuthology.orchestra.run.smithi138.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:20:42.796 INFO:teuthology.orchestra.run.smithi138.stdout: "period_config": { 2022-09-12T19:20:42.796 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_quota": { 2022-09-12T19:20:42.796 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false, 2022-09-12T19:20:42.797 INFO:teuthology.orchestra.run.smithi138.stdout: "check_on_raw": false, 2022-09-12T19:20:42.797 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size": -1, 2022-09-12T19:20:42.797 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size_kb": 0, 2022-09-12T19:20:42.797 INFO:teuthology.orchestra.run.smithi138.stdout: "max_objects": -1 2022-09-12T19:20:42.798 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:20:42.798 INFO:teuthology.orchestra.run.smithi138.stdout: "user_quota": { 2022-09-12T19:20:42.798 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false, 2022-09-12T19:20:42.799 INFO:teuthology.orchestra.run.smithi138.stdout: "check_on_raw": false, 2022-09-12T19:20:42.799 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size": -1, 2022-09-12T19:20:42.799 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size_kb": 0, 2022-09-12T19:20:42.799 INFO:teuthology.orchestra.run.smithi138.stdout: "max_objects": -1 2022-09-12T19:20:42.799 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:20:42.800 INFO:teuthology.orchestra.run.smithi138.stdout: "user_ratelimit": { 2022-09-12T19:20:42.800 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_ops": 0, 2022-09-12T19:20:42.800 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_ops": 0, 2022-09-12T19:20:42.800 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_bytes": 0, 2022-09-12T19:20:42.801 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_bytes": 0, 2022-09-12T19:20:42.801 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false 2022-09-12T19:20:42.801 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:20:42.801 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_ratelimit": { 2022-09-12T19:20:42.802 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_ops": 0, 2022-09-12T19:20:42.802 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_ops": 0, 2022-09-12T19:20:42.802 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_bytes": 0, 2022-09-12T19:20:42.802 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_bytes": 0, 2022-09-12T19:20:42.803 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false 2022-09-12T19:20:42.803 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:20:42.803 INFO:teuthology.orchestra.run.smithi138.stdout: "anonymous_ratelimit": { 2022-09-12T19:20:42.803 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_ops": 0, 2022-09-12T19:20:42.804 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_ops": 0, 2022-09-12T19:20:42.804 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_bytes": 0, 2022-09-12T19:20:42.804 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_bytes": 0, 2022-09-12T19:20:42.804 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false 2022-09-12T19:20:42.805 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:20:42.805 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:20:42.805 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:20:42.806 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_name": "test-realm", 2022-09-12T19:20:42.806 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:20:42.806 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:20:42.807 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to true 2022-09-12T19:21:42.809 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:21:42.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:21:42.812 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone= --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T19:21:42.870 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:21:42.870 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:21:42.892 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:21:42.890+0000 7fe5c2578600 0 failed reading obj info from .rgw.root:zone_info.823fb3a5-4d8a-49dc-ae64-e05804bbea94: (2) No such file or directory 2022-09-12T19:21:42.893 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:21:42.890+0000 7fe5c2578600 0 WARNING: could not read zone params for zone id=823fb3a5-4d8a-49dc-ae64-e05804bbea94 name=test-zone3 2022-09-12T19:21:42.893 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:21:42.890+0000 7fe5c2578600 0 failed reading obj info from .rgw.root:zone_info.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47: (2) No such file or directory 2022-09-12T19:21:42.894 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:21:42.890+0000 7fe5c2578600 0 WARNING: could not read zone params for zone id=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47 name=test-zone1 2022-09-12T19:21:42.901 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:21:42.901 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:21:42.902 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone2", 2022-09-12T19:21:42.902 INFO:teuthology.orchestra.run.smithi138.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-09-12T19:21:42.902 INFO:teuthology.orchestra.run.smithi138.stdout: "control_pool": "test-zone2.rgw.control", 2022-09-12T19:21:42.902 INFO:teuthology.orchestra.run.smithi138.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-09-12T19:21:42.903 INFO:teuthology.orchestra.run.smithi138.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-09-12T19:21:42.904 INFO:teuthology.orchestra.run.smithi138.stdout: "log_pool": "test-zone2.rgw.log", 2022-09-12T19:21:42.904 INFO:teuthology.orchestra.run.smithi138.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-09-12T19:21:42.905 INFO:teuthology.orchestra.run.smithi138.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-09-12T19:21:42.905 INFO:teuthology.orchestra.run.smithi138.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-09-12T19:21:42.905 INFO:teuthology.orchestra.run.smithi138.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-09-12T19:21:42.906 INFO:teuthology.orchestra.run.smithi138.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-09-12T19:21:42.906 INFO:teuthology.orchestra.run.smithi138.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-09-12T19:21:42.907 INFO:teuthology.orchestra.run.smithi138.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-09-12T19:21:42.907 INFO:teuthology.orchestra.run.smithi138.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-09-12T19:21:42.908 INFO:teuthology.orchestra.run.smithi138.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-09-12T19:21:42.908 INFO:teuthology.orchestra.run.smithi138.stdout: "system_key": { 2022-09-12T19:21:42.909 INFO:teuthology.orchestra.run.smithi138.stdout: "access_key": "RDIEFLSOLOEHUH1F", 2022-09-12T19:21:42.909 INFO:teuthology.orchestra.run.smithi138.stdout: "secret_key": "LDCKRd1RecZNgErNcVDnUigDhC7QggCQ" 2022-09-12T19:21:42.910 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:21:42.910 INFO:teuthology.orchestra.run.smithi138.stdout: "placement_pools": [ 2022-09-12T19:21:42.911 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:21:42.911 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "default-placement", 2022-09-12T19:21:42.912 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:21:42.912 INFO:teuthology.orchestra.run.smithi138.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-09-12T19:21:42.913 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_classes": { 2022-09-12T19:21:42.913 INFO:teuthology.orchestra.run.smithi138.stdout: "STANDARD": { 2022-09-12T19:21:42.914 INFO:teuthology.orchestra.run.smithi138.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2022-09-12T19:21:42.914 INFO:teuthology.orchestra.run.smithi138.stdout: "compression_type": "random" 2022-09-12T19:21:42.915 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:21:42.915 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:21:42.916 INFO:teuthology.orchestra.run.smithi138.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-09-12T19:21:42.916 INFO:teuthology.orchestra.run.smithi138.stdout: "index_type": 0 2022-09-12T19:21:42.917 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:21:42.917 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:21:42.918 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:21:42.918 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:21:42.919 INFO:teuthology.orchestra.run.smithi138.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-09-12T19:21:42.919 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:21:42.922 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:21:42.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:21:42.924 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T19:21:42.974 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:21:42.975 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:21:43.014 INFO:teuthology.orchestra.run.smithi138.stderr:Sending period to new master zone fb29b4fb-1f78-4524-ab80-6a5e9e01ab47 2022-09-12T19:21:43.044 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:21:43.044 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:21:43.045 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 5, 2022-09-12T19:21:43.045 INFO:teuthology.orchestra.run.smithi138.stdout: "predecessor_uuid": "2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e", 2022-09-12T19:21:43.046 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": [ 2022-09-12T19:21:43.046 INFO:teuthology.orchestra.run.smithi138.stdout: "", 2022-09-12T19:21:43.046 INFO:teuthology.orchestra.run.smithi138.stdout: "", 2022-09-12T19:21:43.047 INFO:teuthology.orchestra.run.smithi138.stdout: "1_1663010125.771044_393.1", 2022-09-12T19:21:43.047 INFO:teuthology.orchestra.run.smithi138.stdout: "" 2022-09-12T19:21:43.048 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:21:43.049 INFO:teuthology.orchestra.run.smithi138.stdout: "period_map": { 2022-09-12T19:21:43.049 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:21:43.049 INFO:teuthology.orchestra.run.smithi138.stdout: "zonegroups": [ 2022-09-12T19:21:43.050 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:21:43.050 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "test-zonegroup", 2022-09-12T19:21:43.051 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zonegroup", 2022-09-12T19:21:43.051 INFO:teuthology.orchestra.run.smithi138.stdout: "api_name": "", 2022-09-12T19:21:43.052 INFO:teuthology.orchestra.run.smithi138.stdout: "is_master": true, 2022-09-12T19:21:43.052 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:21:43.053 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T19:21:43.053 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:21:43.054 INFO:teuthology.orchestra.run.smithi138.stdout: "hostnames": [], 2022-09-12T19:21:43.054 INFO:teuthology.orchestra.run.smithi138.stdout: "hostnames_s3website": [], 2022-09-12T19:21:43.054 INFO:teuthology.orchestra.run.smithi138.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:21:43.054 INFO:teuthology.orchestra.run.smithi138.stdout: "zones": [ 2022-09-12T19:21:43.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:21:43.055 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T19:21:43.055 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone3", 2022-09-12T19:21:43.055 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:21:43.056 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi035.front.sepia.ceph.com:8001" 2022-09-12T19:21:43.056 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:21:43.056 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T19:21:43.057 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T19:21:43.057 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:21:43.057 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T19:21:43.057 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "", 2022-09-12T19:21:43.058 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": true, 2022-09-12T19:21:43.058 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [], 2022-09-12T19:21:43.058 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T19:21:43.058 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T19:21:43.059 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:21:43.059 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:21:43.059 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:21:43.060 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:21:43.060 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "eb41eca3-59e3-4448-adac-b7c5067fb406", 2022-09-12T19:21:43.060 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone4", 2022-09-12T19:21:43.060 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:21:43.061 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi138.front.sepia.ceph.com:8001" 2022-09-12T19:21:43.061 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:21:43.061 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T19:21:43.061 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T19:21:43.062 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:21:43.062 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T19:21:43.062 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "pubsub", 2022-09-12T19:21:43.063 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": false, 2022-09-12T19:21:43.063 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [ 2022-09-12T19:21:43.063 INFO:teuthology.orchestra.run.smithi138.stdout: "test-zone1" 2022-09-12T19:21:43.064 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:21:43.064 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T19:21:43.064 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T19:21:43.064 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:21:43.065 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:21:43.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:21:43.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:21:43.065 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:21:43.066 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone1", 2022-09-12T19:21:43.066 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:21:43.066 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T19:21:43.066 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:21:43.067 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T19:21:43.067 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T19:21:43.067 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:21:43.067 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T19:21:43.068 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "", 2022-09-12T19:21:43.068 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": true, 2022-09-12T19:21:43.068 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [], 2022-09-12T19:21:43.068 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T19:21:43.069 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T19:21:43.069 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:21:43.069 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:21:43.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:21:43.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:21:43.070 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:21:43.070 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone2", 2022-09-12T19:21:43.071 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:21:43.071 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi138.front.sepia.ceph.com:8000" 2022-09-12T19:21:43.071 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:21:43.071 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T19:21:43.072 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T19:21:43.072 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:21:43.072 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T19:21:43.072 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "", 2022-09-12T19:21:43.073 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": true, 2022-09-12T19:21:43.073 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [], 2022-09-12T19:21:43.073 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T19:21:43.073 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T19:21:43.074 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:21:43.074 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:21:43.074 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:21:43.074 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:21:43.075 INFO:teuthology.orchestra.run.smithi138.stdout: "placement_targets": [ 2022-09-12T19:21:43.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:21:43.075 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "default-placement", 2022-09-12T19:21:43.077 INFO:teuthology.orchestra.run.smithi138.stdout: "tags": [], 2022-09-12T19:21:43.077 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_classes": [ 2022-09-12T19:21:43.077 INFO:teuthology.orchestra.run.smithi138.stdout: "STANDARD" 2022-09-12T19:21:43.077 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:21:43.078 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:21:43.078 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:21:43.078 INFO:teuthology.orchestra.run.smithi138.stdout: "default_placement": "default-placement", 2022-09-12T19:21:43.079 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:21:43.079 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_policy": { 2022-09-12T19:21:43.079 INFO:teuthology.orchestra.run.smithi138.stdout: "groups": [] 2022-09-12T19:21:43.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:21:43.080 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled_features": [ 2022-09-12T19:21:43.080 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:21:43.080 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:21:43.080 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:21:43.081 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:21:43.081 INFO:teuthology.orchestra.run.smithi138.stdout: "short_zone_ids": [ 2022-09-12T19:21:43.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:21:43.081 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T19:21:43.082 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 227168046 2022-09-12T19:21:43.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:21:43.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:21:43.082 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "eb41eca3-59e3-4448-adac-b7c5067fb406", 2022-09-12T19:21:43.083 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 2080702172 2022-09-12T19:21:43.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:21:43.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:21:43.083 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:21:43.084 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 422195086 2022-09-12T19:21:43.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:21:43.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:21:43.084 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:21:43.085 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 2881234289 2022-09-12T19:21:43.085 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:21:43.085 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:21:43.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:21:43.086 INFO:teuthology.orchestra.run.smithi138.stdout: "master_zonegroup": "test-zonegroup", 2022-09-12T19:21:43.086 INFO:teuthology.orchestra.run.smithi138.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:21:43.086 INFO:teuthology.orchestra.run.smithi138.stdout: "period_config": { 2022-09-12T19:21:43.087 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_quota": { 2022-09-12T19:21:43.087 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false, 2022-09-12T19:21:43.087 INFO:teuthology.orchestra.run.smithi138.stdout: "check_on_raw": false, 2022-09-12T19:21:43.087 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size": -1, 2022-09-12T19:21:43.088 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size_kb": 0, 2022-09-12T19:21:43.088 INFO:teuthology.orchestra.run.smithi138.stdout: "max_objects": -1 2022-09-12T19:21:43.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:21:43.088 INFO:teuthology.orchestra.run.smithi138.stdout: "user_quota": { 2022-09-12T19:21:43.089 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false, 2022-09-12T19:21:43.089 INFO:teuthology.orchestra.run.smithi138.stdout: "check_on_raw": false, 2022-09-12T19:21:43.089 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size": -1, 2022-09-12T19:21:43.089 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size_kb": 0, 2022-09-12T19:21:43.090 INFO:teuthology.orchestra.run.smithi138.stdout: "max_objects": -1 2022-09-12T19:21:43.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:21:43.090 INFO:teuthology.orchestra.run.smithi138.stdout: "user_ratelimit": { 2022-09-12T19:21:43.091 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_ops": 0, 2022-09-12T19:21:43.091 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_ops": 0, 2022-09-12T19:21:43.091 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_bytes": 0, 2022-09-12T19:21:43.091 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_bytes": 0, 2022-09-12T19:21:43.092 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false 2022-09-12T19:21:43.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:21:43.092 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_ratelimit": { 2022-09-12T19:21:43.092 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_ops": 0, 2022-09-12T19:21:43.093 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_ops": 0, 2022-09-12T19:21:43.093 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_bytes": 0, 2022-09-12T19:21:43.093 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_bytes": 0, 2022-09-12T19:21:43.093 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false 2022-09-12T19:21:43.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:21:43.094 INFO:teuthology.orchestra.run.smithi138.stdout: "anonymous_ratelimit": { 2022-09-12T19:21:43.094 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_ops": 0, 2022-09-12T19:21:43.094 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_ops": 0, 2022-09-12T19:21:43.095 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_bytes": 0, 2022-09-12T19:21:43.095 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_bytes": 0, 2022-09-12T19:21:43.095 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false 2022-09-12T19:21:43.095 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:21:43.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:21:43.096 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:21:43.096 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_name": "test-realm", 2022-09-12T19:21:43.096 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:21:43.097 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:21:43.097 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "" 2022-09-12T19:22:43.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... ok 2022-09-12T19:22:43.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:22:43.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:22:43.104 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:22:43.163 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:22:43.164 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:22:43.200 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=38ca1f57-1358-496c-a99b-df49ee365a4f 2022-09-12T19:22:43.218 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:22:43.218 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:43.219 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:22:43.219 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:22:43.219 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:43.219 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:43.220 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:22:43.220 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:22:43.220 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:43.220 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:43.221 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:22:43.221 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:18:39.572818Z" 2022-09-12T19:22:43.221 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:43.221 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:43.222 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:22:43.222 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:22:43.222 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:43.223 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:22:43.224 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:22:43.224 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T19:22:43.225 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:22:43.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:22:43.226 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:22:43.285 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:22:43.285 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:22:43.316 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:22:43.313+0000 7f1c2650b600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:22:43.317 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:22:43.314+0000 7f1c2650b600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:22:43.332 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:22:43.333 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:22:43.333 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:22:43.334 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:22:43.334 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:22:43.335 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:22:43.335 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:22:43.336 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:43.336 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:22:43.337 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:22:43.338 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:22:43.338 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:22:43.339 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:22:43.339 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:22:43.339 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:22:43.340 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:22:43.340 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:22:43.341 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:22:43.341 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:22:43.342 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:22:43.342 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:43.342 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:22:43.343 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:22:43.343 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:22:43.344 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:22:43.344 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:22:43.345 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:22:43.345 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:22:43.346 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:22:43.346 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:22:43.346 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:22:43.347 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:22:43.347 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:43.348 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:22:43.348 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:22:43.349 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:22:43.349 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:22:43.350 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:22:43.350 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:22:43.350 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:22:43.351 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:22:43.351 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:22:43.352 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:22:43.352 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:22:43.353 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:43.353 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:22:43.353 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:22:43.354 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:22:43.354 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:22:43.355 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:22:43.355 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:22:43.356 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:22:43.356 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:22:43.356 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:22:43.357 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:22:43.357 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:22:43.358 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:22:43.358 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:22:43.359 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:43.359 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:22:43.360 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:22:43.360 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:22:43.360 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:43.361 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:22:43Z" 2022-09-12T19:22:43.361 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:22:43.363 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:22:43Z" } 2022-09-12T19:22:43.364 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:22:43.365 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:22:43.366 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:22:43.367 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T19:22:43.368 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T19:22:43.368 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:22:43.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:22:43.370 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:22:43.424 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:22:43.425 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:22:43.456 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:22:43.454+0000 7f8d173e9600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:22:43.457 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:22:43.455+0000 7f8d173e9600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:22:43.468 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:22:43.469 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:22:43.469 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:22:43.470 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:22:43.470 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:22:43.471 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:22:43.471 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:22:43.472 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:43.472 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:22:43.472 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:43.473 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:22:43.473 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:22:43.474 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:22:43.474 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:22:43.474 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:22:43.475 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:22:43.475 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:22:43.476 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:22:43.476 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:22:43.477 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:43.477 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:43.478 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:43.478 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:22:43.478 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:22:43.479 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:22:43.479 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:22:43.480 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:22:43.480 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:22:43.480 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:22:43.481 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:22:43.481 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:22:43.482 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:43.482 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:43.482 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:43.483 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:22:43.483 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:22:43.484 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:22:43.484 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:22:43.484 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:22:43.485 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:22:43.485 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:22:43.486 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:22:43.486 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:22:43.486 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:43.487 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:43.487 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:43.488 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:22:43.488 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:22:43.488 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:22:43.489 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:22:43.489 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:22:43.490 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:22:43.490 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:22:43.490 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:22:43.491 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:22:43.491 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:43.492 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:43.492 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:43.493 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:43.493 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:22:43.494 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:22:43.494 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:22:43.495 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:43.495 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:22:43Z" 2022-09-12T19:22:43.495 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:22:43.497 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:22:43Z" } 2022-09-12T19:22:43.498 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:22:43.499 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:22:43.500 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:22:43.501 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T19:22:43.501 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T19:22:43.502 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:22:43.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:22:43.504 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:22:43.557 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:22:43.557 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:22:43.590 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:22:43.587+0000 7f491ee70600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:22:43.591 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:22:43.589+0000 7f491ee70600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:22:43.609 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:22:43.609 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:22:43.610 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:22:43.610 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:22:43.610 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:22:43.610 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:22:43.611 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:22:43.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:43.611 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:22:43.611 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:22:43.612 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:22:43.612 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:22:43.612 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:22:43.612 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:22:43.613 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:22:43.613 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:22:43.613 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:22:43.613 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:22:43.614 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:22:43.614 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:22:43.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:43.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:22:43.615 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:22:43.615 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:22:43.616 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:22:43.616 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:22:43.616 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:22:43.616 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:22:43.617 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:22:43.617 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:22:43.617 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:22:43.617 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:22:43.618 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:43.618 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:22:43.618 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:22:43.619 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:22:43.619 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:22:43.619 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:22:43.619 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:22:43.620 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:22:43.620 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:22:43.620 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:22:43.620 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:22:43.620 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:22:43.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:43.621 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:22:43.621 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:22:43.621 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:22:43.622 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:22:43.622 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:22:43.622 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:22:43.623 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:22:43.623 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:22:43.623 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:22:43.623 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:22:43.624 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:22:43.624 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:22:43.624 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:22:43.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:43.625 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:22:43.625 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:22:43.625 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:22:43.625 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:43.625 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:22:43Z" 2022-09-12T19:22:43.626 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:22:43.627 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:22:43Z" } 2022-09-12T19:22:43.628 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:22:43.628 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:22:43.629 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:22:43.629 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T19:22:43.630 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'RDIEFLSOLOEHUH1F', '--secret', 'LDCKRd1RecZNgErNcVDnUigDhC7QggCQ', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:43.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'RDIEFLSOLOEHUH1F', '--secret', 'LDCKRd1RecZNgErNcVDnUigDhC7QggCQ', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:43.631 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key RDIEFLSOLOEHUH1F --secret LDCKRd1RecZNgErNcVDnUigDhC7QggCQ --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T19:22:43.683 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:22:43.683 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:22:43.709 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:22:43.706+0000 7fbcd8989600 0 failed reading obj info from .rgw.root:zone_info.823fb3a5-4d8a-49dc-ae64-e05804bbea94: (2) No such file or directory 2022-09-12T19:22:43.709 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:22:43.706+0000 7fbcd8989600 0 WARNING: could not read zone params for zone id=823fb3a5-4d8a-49dc-ae64-e05804bbea94 name=test-zone3 2022-09-12T19:22:43.710 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:22:43.706+0000 7fbcd8989600 0 failed reading obj info from .rgw.root:zone_info.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47: (2) No such file or directory 2022-09-12T19:22:43.710 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:22:43.706+0000 7fbcd8989600 0 WARNING: could not read zone params for zone id=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47 name=test-zone1 2022-09-12T19:22:43.712 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:22:43.712 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "29ceb551-a7cb-4153-9dbe-cb465b11b765", 2022-09-12T19:22:43.713 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "remove", 2022-09-12T19:22:43.713 INFO:teuthology.orchestra.run.smithi138.stdout: "domain_root": "remove.rgw.meta:root", 2022-09-12T19:22:43.713 INFO:teuthology.orchestra.run.smithi138.stdout: "control_pool": "remove.rgw.control", 2022-09-12T19:22:43.714 INFO:teuthology.orchestra.run.smithi138.stdout: "gc_pool": "remove.rgw.log:gc", 2022-09-12T19:22:43.714 INFO:teuthology.orchestra.run.smithi138.stdout: "lc_pool": "remove.rgw.log:lc", 2022-09-12T19:22:43.714 INFO:teuthology.orchestra.run.smithi138.stdout: "log_pool": "remove.rgw.log", 2022-09-12T19:22:43.715 INFO:teuthology.orchestra.run.smithi138.stdout: "intent_log_pool": "remove.rgw.log:intent", 2022-09-12T19:22:43.715 INFO:teuthology.orchestra.run.smithi138.stdout: "usage_log_pool": "remove.rgw.log:usage", 2022-09-12T19:22:43.715 INFO:teuthology.orchestra.run.smithi138.stdout: "roles_pool": "remove.rgw.meta:roles", 2022-09-12T19:22:43.716 INFO:teuthology.orchestra.run.smithi138.stdout: "reshard_pool": "remove.rgw.log:reshard", 2022-09-12T19:22:43.716 INFO:teuthology.orchestra.run.smithi138.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2022-09-12T19:22:43.716 INFO:teuthology.orchestra.run.smithi138.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2022-09-12T19:22:43.716 INFO:teuthology.orchestra.run.smithi138.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2022-09-12T19:22:43.717 INFO:teuthology.orchestra.run.smithi138.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2022-09-12T19:22:43.717 INFO:teuthology.orchestra.run.smithi138.stdout: "otp_pool": "remove.rgw.otp", 2022-09-12T19:22:43.717 INFO:teuthology.orchestra.run.smithi138.stdout: "system_key": { 2022-09-12T19:22:43.717 INFO:teuthology.orchestra.run.smithi138.stdout: "access_key": "RDIEFLSOLOEHUH1F", 2022-09-12T19:22:43.718 INFO:teuthology.orchestra.run.smithi138.stdout: "secret_key": "LDCKRd1RecZNgErNcVDnUigDhC7QggCQ" 2022-09-12T19:22:43.718 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:43.719 INFO:teuthology.orchestra.run.smithi138.stdout: "placement_pools": [ 2022-09-12T19:22:43.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:22:43.719 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "default-placement", 2022-09-12T19:22:43.720 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:22:43.720 INFO:teuthology.orchestra.run.smithi138.stdout: "index_pool": "remove.rgw.buckets.index", 2022-09-12T19:22:43.720 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_classes": { 2022-09-12T19:22:43.721 INFO:teuthology.orchestra.run.smithi138.stdout: "STANDARD": { 2022-09-12T19:22:43.721 INFO:teuthology.orchestra.run.smithi138.stdout: "data_pool": "remove.rgw.buckets.data" 2022-09-12T19:22:43.721 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:22:43.722 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:43.722 INFO:teuthology.orchestra.run.smithi138.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2022-09-12T19:22:43.722 INFO:teuthology.orchestra.run.smithi138.stdout: "index_type": 0 2022-09-12T19:22:43.722 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:22:43.723 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:22:43.723 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:22:43.723 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:22:43.723 INFO:teuthology.orchestra.run.smithi138.stdout: "notif_pool": "remove.rgw.log:notif" 2022-09-12T19:22:43.724 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:22:43.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:43.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:43.727 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T19:22:43.776 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:22:43.776 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:22:43.784 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:22:43.781+0000 7f054062d600 0 period (38ca1f57-1358-496c-a99b-df49ee365a4f does not have zone 29ceb551-a7cb-4153-9dbe-cb465b11b765 configured 2022-09-12T19:22:50.829 INFO:teuthology.orchestra.run.smithi138.stderr:Sending period to new master zone fb29b4fb-1f78-4524-ab80-6a5e9e01ab47 2022-09-12T19:22:50.865 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:22:50.866 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:22:50.867 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 6, 2022-09-12T19:22:50.867 INFO:teuthology.orchestra.run.smithi138.stdout: "predecessor_uuid": "2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e", 2022-09-12T19:22:50.867 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": [ 2022-09-12T19:22:50.868 INFO:teuthology.orchestra.run.smithi138.stdout: "", 2022-09-12T19:22:50.868 INFO:teuthology.orchestra.run.smithi138.stdout: "", 2022-09-12T19:22:50.869 INFO:teuthology.orchestra.run.smithi138.stdout: "1_1663010125.771044_393.1", 2022-09-12T19:22:50.870 INFO:teuthology.orchestra.run.smithi138.stdout: "" 2022-09-12T19:22:50.871 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:22:50.872 INFO:teuthology.orchestra.run.smithi138.stdout: "period_map": { 2022-09-12T19:22:50.872 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:22:50.873 INFO:teuthology.orchestra.run.smithi138.stdout: "zonegroups": [ 2022-09-12T19:22:50.874 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:22:50.874 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "test-zonegroup", 2022-09-12T19:22:50.875 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zonegroup", 2022-09-12T19:22:50.876 INFO:teuthology.orchestra.run.smithi138.stdout: "api_name": "", 2022-09-12T19:22:50.877 INFO:teuthology.orchestra.run.smithi138.stdout: "is_master": true, 2022-09-12T19:22:50.878 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:22:50.878 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T19:22:50.879 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:22:50.880 INFO:teuthology.orchestra.run.smithi138.stdout: "hostnames": [], 2022-09-12T19:22:50.881 INFO:teuthology.orchestra.run.smithi138.stdout: "hostnames_s3website": [], 2022-09-12T19:22:50.882 INFO:teuthology.orchestra.run.smithi138.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:22:50.883 INFO:teuthology.orchestra.run.smithi138.stdout: "zones": [ 2022-09-12T19:22:50.884 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:22:50.885 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "29ceb551-a7cb-4153-9dbe-cb465b11b765", 2022-09-12T19:22:50.886 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "remove", 2022-09-12T19:22:50.887 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [], 2022-09-12T19:22:50.888 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T19:22:50.888 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T19:22:50.889 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:22:50.890 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T19:22:50.890 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "", 2022-09-12T19:22:50.900 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": true, 2022-09-12T19:22:50.901 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [], 2022-09-12T19:22:50.901 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T19:22:50.902 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T19:22:50.902 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:22:50.903 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:22:50.904 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:50.904 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:22:50.905 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T19:22:50.906 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone3", 2022-09-12T19:22:50.906 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:22:50.907 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi035.front.sepia.ceph.com:8001" 2022-09-12T19:22:50.908 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:22:50.908 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T19:22:50.909 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T19:22:50.909 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:22:50.910 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T19:22:50.910 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "", 2022-09-12T19:22:50.911 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": true, 2022-09-12T19:22:50.911 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [], 2022-09-12T19:22:50.912 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T19:22:50.913 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T19:22:50.913 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:22:50.914 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:22:50.915 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:50.916 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:22:50.916 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "eb41eca3-59e3-4448-adac-b7c5067fb406", 2022-09-12T19:22:50.917 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone4", 2022-09-12T19:22:50.917 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:22:50.918 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi138.front.sepia.ceph.com:8001" 2022-09-12T19:22:50.918 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:22:50.919 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T19:22:50.919 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T19:22:50.920 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:22:50.920 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T19:22:50.921 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "pubsub", 2022-09-12T19:22:50.922 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": false, 2022-09-12T19:22:50.922 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [ 2022-09-12T19:22:50.923 INFO:teuthology.orchestra.run.smithi138.stdout: "test-zone1" 2022-09-12T19:22:50.924 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:22:50.924 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T19:22:50.925 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T19:22:50.925 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:22:50.926 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:22:50.927 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:50.927 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:22:50.928 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:22:50.929 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone1", 2022-09-12T19:22:50.930 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:22:50.931 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T19:22:50.932 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:22:50.933 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T19:22:50.934 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T19:22:50.935 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:22:50.935 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T19:22:50.936 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "", 2022-09-12T19:22:50.936 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": true, 2022-09-12T19:22:50.937 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [], 2022-09-12T19:22:50.938 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T19:22:50.939 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T19:22:50.940 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:22:50.940 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:22:50.941 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:50.941 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:22:50.942 INFO:teuthology.orchestra.run.smithi138.stdout: "id": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:22:50.942 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "test-zone2", 2022-09-12T19:22:50.943 INFO:teuthology.orchestra.run.smithi138.stdout: "endpoints": [ 2022-09-12T19:22:50.944 INFO:teuthology.orchestra.run.smithi138.stdout: "http://smithi138.front.sepia.ceph.com:8000" 2022-09-12T19:22:50.948 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:22:50.948 INFO:teuthology.orchestra.run.smithi138.stdout: "log_meta": false, 2022-09-12T19:22:50.949 INFO:teuthology.orchestra.run.smithi138.stdout: "log_data": true, 2022-09-12T19:22:50.950 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:22:50.951 INFO:teuthology.orchestra.run.smithi138.stdout: "read_only": false, 2022-09-12T19:22:50.952 INFO:teuthology.orchestra.run.smithi138.stdout: "tier_type": "", 2022-09-12T19:22:50.952 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from_all": true, 2022-09-12T19:22:50.953 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_from": [], 2022-09-12T19:22:50.955 INFO:teuthology.orchestra.run.smithi138.stdout: "redirect_zone": "", 2022-09-12T19:22:50.955 INFO:teuthology.orchestra.run.smithi138.stdout: "supported_features": [ 2022-09-12T19:22:50.956 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:22:50.956 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:22:50.957 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:22:50.958 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:22:50.958 INFO:teuthology.orchestra.run.smithi138.stdout: "placement_targets": [ 2022-09-12T19:22:50.959 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:22:50.959 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "default-placement", 2022-09-12T19:22:50.960 INFO:teuthology.orchestra.run.smithi138.stdout: "tags": [], 2022-09-12T19:22:50.961 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_classes": [ 2022-09-12T19:22:50.961 INFO:teuthology.orchestra.run.smithi138.stdout: "STANDARD" 2022-09-12T19:22:50.961 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:22:50.962 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:22:50.962 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:22:50.962 INFO:teuthology.orchestra.run.smithi138.stdout: "default_placement": "default-placement", 2022-09-12T19:22:50.963 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:22:50.963 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_policy": { 2022-09-12T19:22:50.963 INFO:teuthology.orchestra.run.smithi138.stdout: "groups": [] 2022-09-12T19:22:50.964 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:50.964 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled_features": [ 2022-09-12T19:22:50.964 INFO:teuthology.orchestra.run.smithi138.stdout: "resharding" 2022-09-12T19:22:50.965 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:22:50.965 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:22:50.965 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2022-09-12T19:22:50.966 INFO:teuthology.orchestra.run.smithi138.stdout: "short_zone_ids": [ 2022-09-12T19:22:50.966 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:22:50.966 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "29ceb551-a7cb-4153-9dbe-cb465b11b765", 2022-09-12T19:22:50.967 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 843534302 2022-09-12T19:22:50.967 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:50.967 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:22:50.968 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T19:22:50.968 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 227168046 2022-09-12T19:22:50.968 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:50.968 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:22:50.969 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "eb41eca3-59e3-4448-adac-b7c5067fb406", 2022-09-12T19:22:50.969 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 2080702172 2022-09-12T19:22:50.969 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:50.969 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:22:50.970 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:22:50.970 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 422195086 2022-09-12T19:22:50.970 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:50.970 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:22:50.971 INFO:teuthology.orchestra.run.smithi138.stdout: "key": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:22:50.971 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 2881234289 2022-09-12T19:22:50.971 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:22:50.971 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:22:50.972 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:50.972 INFO:teuthology.orchestra.run.smithi138.stdout: "master_zonegroup": "test-zonegroup", 2022-09-12T19:22:50.972 INFO:teuthology.orchestra.run.smithi138.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:22:50.972 INFO:teuthology.orchestra.run.smithi138.stdout: "period_config": { 2022-09-12T19:22:50.973 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_quota": { 2022-09-12T19:22:50.973 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false, 2022-09-12T19:22:50.973 INFO:teuthology.orchestra.run.smithi138.stdout: "check_on_raw": false, 2022-09-12T19:22:50.973 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size": -1, 2022-09-12T19:22:50.974 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size_kb": 0, 2022-09-12T19:22:50.974 INFO:teuthology.orchestra.run.smithi138.stdout: "max_objects": -1 2022-09-12T19:22:50.974 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:50.974 INFO:teuthology.orchestra.run.smithi138.stdout: "user_quota": { 2022-09-12T19:22:50.975 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false, 2022-09-12T19:22:50.975 INFO:teuthology.orchestra.run.smithi138.stdout: "check_on_raw": false, 2022-09-12T19:22:50.975 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size": -1, 2022-09-12T19:22:50.975 INFO:teuthology.orchestra.run.smithi138.stdout: "max_size_kb": 0, 2022-09-12T19:22:50.976 INFO:teuthology.orchestra.run.smithi138.stdout: "max_objects": -1 2022-09-12T19:22:50.976 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:50.976 INFO:teuthology.orchestra.run.smithi138.stdout: "user_ratelimit": { 2022-09-12T19:22:50.976 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_ops": 0, 2022-09-12T19:22:50.977 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_ops": 0, 2022-09-12T19:22:50.977 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_bytes": 0, 2022-09-12T19:22:50.977 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_bytes": 0, 2022-09-12T19:22:50.977 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false 2022-09-12T19:22:50.978 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:50.978 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket_ratelimit": { 2022-09-12T19:22:50.978 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_ops": 0, 2022-09-12T19:22:50.979 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_ops": 0, 2022-09-12T19:22:50.979 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_bytes": 0, 2022-09-12T19:22:50.979 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_bytes": 0, 2022-09-12T19:22:50.979 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false 2022-09-12T19:22:50.980 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:50.980 INFO:teuthology.orchestra.run.smithi138.stdout: "anonymous_ratelimit": { 2022-09-12T19:22:50.980 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_ops": 0, 2022-09-12T19:22:50.980 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_ops": 0, 2022-09-12T19:22:50.981 INFO:teuthology.orchestra.run.smithi138.stdout: "max_read_bytes": 0, 2022-09-12T19:22:50.981 INFO:teuthology.orchestra.run.smithi138.stdout: "max_write_bytes": 0, 2022-09-12T19:22:50.981 INFO:teuthology.orchestra.run.smithi138.stdout: "enabled": false 2022-09-12T19:22:50.981 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:22:50.982 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:50.982 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:22:50.982 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_name": "test-realm", 2022-09-12T19:22:50.982 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:22:50.983 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:22:50.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:50.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:50.985 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:22:51.045 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:22:51.046 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:22:51.062 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:22:51.059+0000 7fe5f8eff600 0 failed reading obj info from .rgw.root:zone_info.eb41eca3-59e3-4448-adac-b7c5067fb406: (2) No such file or directory 2022-09-12T19:22:51.062 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:22:51.059+0000 7fe5f8eff600 0 WARNING: could not read zone params for zone id=eb41eca3-59e3-4448-adac-b7c5067fb406 name=test-zone4 2022-09-12T19:22:51.063 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:22:51.061+0000 7fe5f8eff600 0 failed reading obj info from .rgw.root:zone_info.fdf3b910-2ea3-4164-a3f8-b3e146af1c44: (2) No such file or directory 2022-09-12T19:22:51.064 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:22:51.061+0000 7fe5f8eff600 0 WARNING: could not read zone params for zone id=fdf3b910-2ea3-4164-a3f8-b3e146af1c44 name=test-zone2 2022-09-12T19:22:51.072 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:22:51.073 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "test-zonegroup", 2022-09-12T19:22:51.073 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zonegroup", 2022-09-12T19:22:51.074 INFO:teuthology.orchestra.run.smithi035.stdout: "api_name": "", 2022-09-12T19:22:51.074 INFO:teuthology.orchestra.run.smithi035.stdout: "is_master": true, 2022-09-12T19:22:51.074 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T19:22:51.075 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T19:22:51.075 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.076 INFO:teuthology.orchestra.run.smithi035.stdout: "hostnames": [], 2022-09-12T19:22:51.076 INFO:teuthology.orchestra.run.smithi035.stdout: "hostnames_s3website": [], 2022-09-12T19:22:51.077 INFO:teuthology.orchestra.run.smithi035.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:22:51.077 INFO:teuthology.orchestra.run.smithi035.stdout: "zones": [ 2022-09-12T19:22:51.078 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.078 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T19:22:51.079 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone3", 2022-09-12T19:22:51.079 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T19:22:51.080 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi035.front.sepia.ceph.com:8001" 2022-09-12T19:22:51.080 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.080 INFO:teuthology.orchestra.run.smithi035.stdout: "log_meta": false, 2022-09-12T19:22:51.081 INFO:teuthology.orchestra.run.smithi035.stdout: "log_data": true, 2022-09-12T19:22:51.081 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:22:51.082 INFO:teuthology.orchestra.run.smithi035.stdout: "read_only": false, 2022-09-12T19:22:51.082 INFO:teuthology.orchestra.run.smithi035.stdout: "tier_type": "", 2022-09-12T19:22:51.083 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from_all": true, 2022-09-12T19:22:51.083 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from": [], 2022-09-12T19:22:51.084 INFO:teuthology.orchestra.run.smithi035.stdout: "redirect_zone": "", 2022-09-12T19:22:51.084 INFO:teuthology.orchestra.run.smithi035.stdout: "supported_features": [ 2022-09-12T19:22:51.085 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:22:51.085 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:51.086 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.086 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.087 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "eb41eca3-59e3-4448-adac-b7c5067fb406", 2022-09-12T19:22:51.087 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone4", 2022-09-12T19:22:51.088 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T19:22:51.088 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi138.front.sepia.ceph.com:8001" 2022-09-12T19:22:51.089 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.089 INFO:teuthology.orchestra.run.smithi035.stdout: "log_meta": false, 2022-09-12T19:22:51.090 INFO:teuthology.orchestra.run.smithi035.stdout: "log_data": true, 2022-09-12T19:22:51.090 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:22:51.091 INFO:teuthology.orchestra.run.smithi035.stdout: "read_only": false, 2022-09-12T19:22:51.091 INFO:teuthology.orchestra.run.smithi035.stdout: "tier_type": "pubsub", 2022-09-12T19:22:51.092 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from_all": false, 2022-09-12T19:22:51.092 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from": [ 2022-09-12T19:22:51.092 INFO:teuthology.orchestra.run.smithi035.stdout: "test-zone1" 2022-09-12T19:22:51.093 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.093 INFO:teuthology.orchestra.run.smithi035.stdout: "redirect_zone": "", 2022-09-12T19:22:51.093 INFO:teuthology.orchestra.run.smithi035.stdout: "supported_features": [ 2022-09-12T19:22:51.093 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:22:51.094 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:51.094 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.094 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.095 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:22:51.095 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone1", 2022-09-12T19:22:51.095 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T19:22:51.095 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T19:22:51.096 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.096 INFO:teuthology.orchestra.run.smithi035.stdout: "log_meta": false, 2022-09-12T19:22:51.096 INFO:teuthology.orchestra.run.smithi035.stdout: "log_data": true, 2022-09-12T19:22:51.096 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:22:51.097 INFO:teuthology.orchestra.run.smithi035.stdout: "read_only": false, 2022-09-12T19:22:51.097 INFO:teuthology.orchestra.run.smithi035.stdout: "tier_type": "", 2022-09-12T19:22:51.097 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from_all": true, 2022-09-12T19:22:51.097 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from": [], 2022-09-12T19:22:51.098 INFO:teuthology.orchestra.run.smithi035.stdout: "redirect_zone": "", 2022-09-12T19:22:51.098 INFO:teuthology.orchestra.run.smithi035.stdout: "supported_features": [ 2022-09-12T19:22:51.098 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:22:51.099 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:51.099 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.099 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.099 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:22:51.100 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone2", 2022-09-12T19:22:51.100 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T19:22:51.100 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi138.front.sepia.ceph.com:8000" 2022-09-12T19:22:51.100 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.101 INFO:teuthology.orchestra.run.smithi035.stdout: "log_meta": false, 2022-09-12T19:22:51.101 INFO:teuthology.orchestra.run.smithi035.stdout: "log_data": true, 2022-09-12T19:22:51.101 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:22:51.102 INFO:teuthology.orchestra.run.smithi035.stdout: "read_only": false, 2022-09-12T19:22:51.102 INFO:teuthology.orchestra.run.smithi035.stdout: "tier_type": "", 2022-09-12T19:22:51.102 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from_all": true, 2022-09-12T19:22:51.102 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from": [], 2022-09-12T19:22:51.103 INFO:teuthology.orchestra.run.smithi035.stdout: "redirect_zone": "", 2022-09-12T19:22:51.103 INFO:teuthology.orchestra.run.smithi035.stdout: "supported_features": [ 2022-09-12T19:22:51.103 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:22:51.103 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:51.104 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:51.104 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.104 INFO:teuthology.orchestra.run.smithi035.stdout: "placement_targets": [ 2022-09-12T19:22:51.104 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.105 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "default-placement", 2022-09-12T19:22:51.105 INFO:teuthology.orchestra.run.smithi035.stdout: "tags": [], 2022-09-12T19:22:51.105 INFO:teuthology.orchestra.run.smithi035.stdout: "storage_classes": [ 2022-09-12T19:22:51.105 INFO:teuthology.orchestra.run.smithi035.stdout: "STANDARD" 2022-09-12T19:22:51.106 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:51.106 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:51.106 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.106 INFO:teuthology.orchestra.run.smithi035.stdout: "default_placement": "default-placement", 2022-09-12T19:22:51.107 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:22:51.107 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_policy": { 2022-09-12T19:22:51.107 INFO:teuthology.orchestra.run.smithi035.stdout: "groups": [] 2022-09-12T19:22:51.107 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.108 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled_features": [ 2022-09-12T19:22:51.108 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:22:51.108 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:51.108 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:22:51.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:51.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:51.112 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:22:51.172 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:22:51.173 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:22:51.187 INFO:teuthology.orchestra.run.smithi035.stderr:zone name remove not found in zonegroup test-zonegroup 2022-09-12T19:22:51.191 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-09-12T19:22:51.192 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:51.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:51.194 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T19:22:51.250 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:22:51.251 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:22:51.286 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:22:51.283+0000 7f04d3449600 0 failed reading obj info from .rgw.root:zone_info.823fb3a5-4d8a-49dc-ae64-e05804bbea94: (2) No such file or directory 2022-09-12T19:22:51.287 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:22:51.283+0000 7f04d3449600 0 WARNING: could not read zone params for zone id=823fb3a5-4d8a-49dc-ae64-e05804bbea94 name=test-zone3 2022-09-12T19:22:51.288 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:22:51.284+0000 7f04d3449600 0 failed reading obj info from .rgw.root:zone_info.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47: (2) No such file or directory 2022-09-12T19:22:51.288 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:22:51.284+0000 7f04d3449600 0 WARNING: could not read zone params for zone id=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47 name=test-zone1 2022-09-12T19:22:51.290 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:22:51.287+0000 7f04d3449600 0 zone id 29ceb551-a7cb-4153-9dbe-cb465b11b765 is not a part of zonegroup default 2022-09-12T19:22:51.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:51.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:51.305 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:22:51.367 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:22:51.367 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:22:51.380 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:22:51.378+0000 7fa7d0b12600 0 WARNING: can't generate connection for zone 29ceb551-a7cb-4153-9dbe-cb465b11b765 id remove: no endpoints defined 2022-09-12T19:22:51.479 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:22:51.480 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:22:51.480 INFO:teuthology.orchestra.run.smithi035.stdout: "epoch": 7, 2022-09-12T19:22:51.480 INFO:teuthology.orchestra.run.smithi035.stdout: "predecessor_uuid": "2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e", 2022-09-12T19:22:51.481 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": [ 2022-09-12T19:22:51.481 INFO:teuthology.orchestra.run.smithi035.stdout: "", 2022-09-12T19:22:51.482 INFO:teuthology.orchestra.run.smithi035.stdout: "", 2022-09-12T19:22:51.482 INFO:teuthology.orchestra.run.smithi035.stdout: "1_1663010125.771044_393.1", 2022-09-12T19:22:51.482 INFO:teuthology.orchestra.run.smithi035.stdout: "" 2022-09-12T19:22:51.483 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.483 INFO:teuthology.orchestra.run.smithi035.stdout: "period_map": { 2022-09-12T19:22:51.483 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:22:51.484 INFO:teuthology.orchestra.run.smithi035.stdout: "zonegroups": [ 2022-09-12T19:22:51.484 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.485 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "test-zonegroup", 2022-09-12T19:22:51.485 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zonegroup", 2022-09-12T19:22:51.486 INFO:teuthology.orchestra.run.smithi035.stdout: "api_name": "", 2022-09-12T19:22:51.487 INFO:teuthology.orchestra.run.smithi035.stdout: "is_master": true, 2022-09-12T19:22:51.488 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T19:22:51.488 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T19:22:51.488 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.488 INFO:teuthology.orchestra.run.smithi035.stdout: "hostnames": [], 2022-09-12T19:22:51.489 INFO:teuthology.orchestra.run.smithi035.stdout: "hostnames_s3website": [], 2022-09-12T19:22:51.489 INFO:teuthology.orchestra.run.smithi035.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:22:51.489 INFO:teuthology.orchestra.run.smithi035.stdout: "zones": [ 2022-09-12T19:22:51.489 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.490 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T19:22:51.490 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone3", 2022-09-12T19:22:51.490 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T19:22:51.490 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi035.front.sepia.ceph.com:8001" 2022-09-12T19:22:51.491 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.491 INFO:teuthology.orchestra.run.smithi035.stdout: "log_meta": false, 2022-09-12T19:22:51.491 INFO:teuthology.orchestra.run.smithi035.stdout: "log_data": true, 2022-09-12T19:22:51.491 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:22:51.492 INFO:teuthology.orchestra.run.smithi035.stdout: "read_only": false, 2022-09-12T19:22:51.492 INFO:teuthology.orchestra.run.smithi035.stdout: "tier_type": "", 2022-09-12T19:22:51.492 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from_all": true, 2022-09-12T19:22:51.492 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from": [], 2022-09-12T19:22:51.493 INFO:teuthology.orchestra.run.smithi035.stdout: "redirect_zone": "", 2022-09-12T19:22:51.493 INFO:teuthology.orchestra.run.smithi035.stdout: "supported_features": [ 2022-09-12T19:22:51.493 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:22:51.493 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:51.494 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.494 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.494 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "eb41eca3-59e3-4448-adac-b7c5067fb406", 2022-09-12T19:22:51.494 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone4", 2022-09-12T19:22:51.495 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T19:22:51.495 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi138.front.sepia.ceph.com:8001" 2022-09-12T19:22:51.495 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.495 INFO:teuthology.orchestra.run.smithi035.stdout: "log_meta": false, 2022-09-12T19:22:51.496 INFO:teuthology.orchestra.run.smithi035.stdout: "log_data": true, 2022-09-12T19:22:51.496 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:22:51.496 INFO:teuthology.orchestra.run.smithi035.stdout: "read_only": false, 2022-09-12T19:22:51.496 INFO:teuthology.orchestra.run.smithi035.stdout: "tier_type": "pubsub", 2022-09-12T19:22:51.497 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from_all": false, 2022-09-12T19:22:51.497 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from": [ 2022-09-12T19:22:51.497 INFO:teuthology.orchestra.run.smithi035.stdout: "test-zone1" 2022-09-12T19:22:51.497 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.497 INFO:teuthology.orchestra.run.smithi035.stdout: "redirect_zone": "", 2022-09-12T19:22:51.498 INFO:teuthology.orchestra.run.smithi035.stdout: "supported_features": [ 2022-09-12T19:22:51.498 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:22:51.498 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:51.498 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.499 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.499 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:22:51.499 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone1", 2022-09-12T19:22:51.499 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T19:22:51.500 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T19:22:51.500 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.500 INFO:teuthology.orchestra.run.smithi035.stdout: "log_meta": false, 2022-09-12T19:22:51.500 INFO:teuthology.orchestra.run.smithi035.stdout: "log_data": true, 2022-09-12T19:22:51.501 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:22:51.501 INFO:teuthology.orchestra.run.smithi035.stdout: "read_only": false, 2022-09-12T19:22:51.501 INFO:teuthology.orchestra.run.smithi035.stdout: "tier_type": "", 2022-09-12T19:22:51.501 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from_all": true, 2022-09-12T19:22:51.502 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from": [], 2022-09-12T19:22:51.502 INFO:teuthology.orchestra.run.smithi035.stdout: "redirect_zone": "", 2022-09-12T19:22:51.502 INFO:teuthology.orchestra.run.smithi035.stdout: "supported_features": [ 2022-09-12T19:22:51.502 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:22:51.503 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:51.503 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.503 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.503 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:22:51.504 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone2", 2022-09-12T19:22:51.504 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T19:22:51.504 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi138.front.sepia.ceph.com:8000" 2022-09-12T19:22:51.504 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.505 INFO:teuthology.orchestra.run.smithi035.stdout: "log_meta": false, 2022-09-12T19:22:51.505 INFO:teuthology.orchestra.run.smithi035.stdout: "log_data": true, 2022-09-12T19:22:51.505 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:22:51.505 INFO:teuthology.orchestra.run.smithi035.stdout: "read_only": false, 2022-09-12T19:22:51.506 INFO:teuthology.orchestra.run.smithi035.stdout: "tier_type": "", 2022-09-12T19:22:51.506 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from_all": true, 2022-09-12T19:22:51.506 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from": [], 2022-09-12T19:22:51.506 INFO:teuthology.orchestra.run.smithi035.stdout: "redirect_zone": "", 2022-09-12T19:22:51.507 INFO:teuthology.orchestra.run.smithi035.stdout: "supported_features": [ 2022-09-12T19:22:51.507 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:22:51.507 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:51.507 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:51.508 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.508 INFO:teuthology.orchestra.run.smithi035.stdout: "placement_targets": [ 2022-09-12T19:22:51.508 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.508 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "default-placement", 2022-09-12T19:22:51.515 INFO:teuthology.orchestra.run.smithi035.stdout: "tags": [], 2022-09-12T19:22:51.515 INFO:teuthology.orchestra.run.smithi035.stdout: "storage_classes": [ 2022-09-12T19:22:51.515 INFO:teuthology.orchestra.run.smithi035.stdout: "STANDARD" 2022-09-12T19:22:51.516 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:51.516 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:51.516 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.516 INFO:teuthology.orchestra.run.smithi035.stdout: "default_placement": "default-placement", 2022-09-12T19:22:51.517 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:22:51.517 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_policy": { 2022-09-12T19:22:51.517 INFO:teuthology.orchestra.run.smithi035.stdout: "groups": [] 2022-09-12T19:22:51.517 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.518 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled_features": [ 2022-09-12T19:22:51.518 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:22:51.518 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:51.518 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:51.519 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.519 INFO:teuthology.orchestra.run.smithi035.stdout: "short_zone_ids": [ 2022-09-12T19:22:51.519 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.520 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T19:22:51.520 INFO:teuthology.orchestra.run.smithi035.stdout: "val": 227168046 2022-09-12T19:22:51.520 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.520 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.521 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "eb41eca3-59e3-4448-adac-b7c5067fb406", 2022-09-12T19:22:51.521 INFO:teuthology.orchestra.run.smithi035.stdout: "val": 2080702172 2022-09-12T19:22:51.521 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.521 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.522 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:22:51.522 INFO:teuthology.orchestra.run.smithi035.stdout: "val": 422195086 2022-09-12T19:22:51.522 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.522 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.523 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:22:51.523 INFO:teuthology.orchestra.run.smithi035.stdout: "val": 2881234289 2022-09-12T19:22:51.523 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:51.523 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:51.524 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.524 INFO:teuthology.orchestra.run.smithi035.stdout: "master_zonegroup": "test-zonegroup", 2022-09-12T19:22:51.524 INFO:teuthology.orchestra.run.smithi035.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:22:51.524 INFO:teuthology.orchestra.run.smithi035.stdout: "period_config": { 2022-09-12T19:22:51.525 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_quota": { 2022-09-12T19:22:51.525 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false, 2022-09-12T19:22:51.525 INFO:teuthology.orchestra.run.smithi035.stdout: "check_on_raw": false, 2022-09-12T19:22:51.525 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size": -1, 2022-09-12T19:22:51.526 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size_kb": 0, 2022-09-12T19:22:51.526 INFO:teuthology.orchestra.run.smithi035.stdout: "max_objects": -1 2022-09-12T19:22:51.526 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.526 INFO:teuthology.orchestra.run.smithi035.stdout: "user_quota": { 2022-09-12T19:22:51.527 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false, 2022-09-12T19:22:51.527 INFO:teuthology.orchestra.run.smithi035.stdout: "check_on_raw": false, 2022-09-12T19:22:51.527 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size": -1, 2022-09-12T19:22:51.528 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size_kb": 0, 2022-09-12T19:22:51.528 INFO:teuthology.orchestra.run.smithi035.stdout: "max_objects": -1 2022-09-12T19:22:51.528 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.528 INFO:teuthology.orchestra.run.smithi035.stdout: "user_ratelimit": { 2022-09-12T19:22:51.529 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_ops": 0, 2022-09-12T19:22:51.529 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_ops": 0, 2022-09-12T19:22:51.529 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_bytes": 0, 2022-09-12T19:22:51.529 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_bytes": 0, 2022-09-12T19:22:51.530 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false 2022-09-12T19:22:51.530 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.530 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_ratelimit": { 2022-09-12T19:22:51.530 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_ops": 0, 2022-09-12T19:22:51.531 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_ops": 0, 2022-09-12T19:22:51.531 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_bytes": 0, 2022-09-12T19:22:51.531 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_bytes": 0, 2022-09-12T19:22:51.531 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false 2022-09-12T19:22:51.532 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.532 INFO:teuthology.orchestra.run.smithi035.stdout: "anonymous_ratelimit": { 2022-09-12T19:22:51.532 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_ops": 0, 2022-09-12T19:22:51.532 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_ops": 0, 2022-09-12T19:22:51.533 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_bytes": 0, 2022-09-12T19:22:51.533 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_bytes": 0, 2022-09-12T19:22:51.533 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false 2022-09-12T19:22:51.533 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:51.534 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.534 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:22:51.534 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_name": "test-realm", 2022-09-12T19:22:51.534 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:22:51.535 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:22:51.536 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... ok 2022-09-12T19:22:51.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:51.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:51.538 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:22:51.593 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:22:51.594 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:22:51.631 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:22:51.631 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "4e46d9ae-24a4-426f-ad44-77c951440c18", 2022-09-12T19:22:51.632 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "remove_zg", 2022-09-12T19:22:51.632 INFO:teuthology.orchestra.run.smithi035.stdout: "api_name": "remove_zg", 2022-09-12T19:22:51.632 INFO:teuthology.orchestra.run.smithi035.stdout: "is_master": false, 2022-09-12T19:22:51.632 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [], 2022-09-12T19:22:51.633 INFO:teuthology.orchestra.run.smithi035.stdout: "hostnames": [], 2022-09-12T19:22:51.633 INFO:teuthology.orchestra.run.smithi035.stdout: "hostnames_s3website": [], 2022-09-12T19:22:51.633 INFO:teuthology.orchestra.run.smithi035.stdout: "master_zone": "", 2022-09-12T19:22:51.633 INFO:teuthology.orchestra.run.smithi035.stdout: "zones": [], 2022-09-12T19:22:51.634 INFO:teuthology.orchestra.run.smithi035.stdout: "placement_targets": [], 2022-09-12T19:22:51.634 INFO:teuthology.orchestra.run.smithi035.stdout: "default_placement": "", 2022-09-12T19:22:51.634 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:22:51.635 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_policy": { 2022-09-12T19:22:51.635 INFO:teuthology.orchestra.run.smithi035.stdout: "groups": [] 2022-09-12T19:22:51.636 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.636 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled_features": [ 2022-09-12T19:22:51.636 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:22:51.637 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:51.637 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:22:51.638 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:51.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:51.639 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:22:51.694 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:22:51.695 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:22:51.732 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:22:51.732 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "ad741acb-53d5-4810-a2b0-4448fda5b3ca", 2022-09-12T19:22:51.732 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "remove", 2022-09-12T19:22:51.733 INFO:teuthology.orchestra.run.smithi035.stdout: "domain_root": "remove.rgw.meta:root", 2022-09-12T19:22:51.733 INFO:teuthology.orchestra.run.smithi035.stdout: "control_pool": "remove.rgw.control", 2022-09-12T19:22:51.733 INFO:teuthology.orchestra.run.smithi035.stdout: "gc_pool": "remove.rgw.log:gc", 2022-09-12T19:22:51.734 INFO:teuthology.orchestra.run.smithi035.stdout: "lc_pool": "remove.rgw.log:lc", 2022-09-12T19:22:51.734 INFO:teuthology.orchestra.run.smithi035.stdout: "log_pool": "remove.rgw.log", 2022-09-12T19:22:51.734 INFO:teuthology.orchestra.run.smithi035.stdout: "intent_log_pool": "remove.rgw.log:intent", 2022-09-12T19:22:51.735 INFO:teuthology.orchestra.run.smithi035.stdout: "usage_log_pool": "remove.rgw.log:usage", 2022-09-12T19:22:51.735 INFO:teuthology.orchestra.run.smithi035.stdout: "roles_pool": "remove.rgw.meta:roles", 2022-09-12T19:22:51.735 INFO:teuthology.orchestra.run.smithi035.stdout: "reshard_pool": "remove.rgw.log:reshard", 2022-09-12T19:22:51.736 INFO:teuthology.orchestra.run.smithi035.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2022-09-12T19:22:51.736 INFO:teuthology.orchestra.run.smithi035.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2022-09-12T19:22:51.736 INFO:teuthology.orchestra.run.smithi035.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2022-09-12T19:22:51.737 INFO:teuthology.orchestra.run.smithi035.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2022-09-12T19:22:51.737 INFO:teuthology.orchestra.run.smithi035.stdout: "otp_pool": "remove.rgw.otp", 2022-09-12T19:22:51.737 INFO:teuthology.orchestra.run.smithi035.stdout: "system_key": { 2022-09-12T19:22:51.738 INFO:teuthology.orchestra.run.smithi035.stdout: "access_key": "", 2022-09-12T19:22:51.738 INFO:teuthology.orchestra.run.smithi035.stdout: "secret_key": "" 2022-09-12T19:22:51.738 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.739 INFO:teuthology.orchestra.run.smithi035.stdout: "placement_pools": [ 2022-09-12T19:22:51.739 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.739 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "default-placement", 2022-09-12T19:22:51.739 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:22:51.740 INFO:teuthology.orchestra.run.smithi035.stdout: "index_pool": "remove.rgw.buckets.index", 2022-09-12T19:22:51.740 INFO:teuthology.orchestra.run.smithi035.stdout: "storage_classes": { 2022-09-12T19:22:51.740 INFO:teuthology.orchestra.run.smithi035.stdout: "STANDARD": { 2022-09-12T19:22:51.740 INFO:teuthology.orchestra.run.smithi035.stdout: "data_pool": "remove.rgw.buckets.data" 2022-09-12T19:22:51.741 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:51.741 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.741 INFO:teuthology.orchestra.run.smithi035.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2022-09-12T19:22:51.741 INFO:teuthology.orchestra.run.smithi035.stdout: "index_type": 0 2022-09-12T19:22:51.742 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:51.742 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:51.742 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.743 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:22:51.743 INFO:teuthology.orchestra.run.smithi035.stdout: "notif_pool": "remove.rgw.log:notif" 2022-09-12T19:22:51.743 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:22:51.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:51.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:51.746 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:22:51.808 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:22:51.809 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:22:51.929 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:22:51.929 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:22:51.930 INFO:teuthology.orchestra.run.smithi035.stdout: "epoch": 8, 2022-09-12T19:22:51.930 INFO:teuthology.orchestra.run.smithi035.stdout: "predecessor_uuid": "2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e", 2022-09-12T19:22:51.930 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": [ 2022-09-12T19:22:51.931 INFO:teuthology.orchestra.run.smithi035.stdout: "", 2022-09-12T19:22:51.931 INFO:teuthology.orchestra.run.smithi035.stdout: "", 2022-09-12T19:22:51.932 INFO:teuthology.orchestra.run.smithi035.stdout: "1_1663010125.771044_393.1", 2022-09-12T19:22:51.932 INFO:teuthology.orchestra.run.smithi035.stdout: "" 2022-09-12T19:22:51.932 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.933 INFO:teuthology.orchestra.run.smithi035.stdout: "period_map": { 2022-09-12T19:22:51.933 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:22:51.933 INFO:teuthology.orchestra.run.smithi035.stdout: "zonegroups": [ 2022-09-12T19:22:51.934 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.934 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "4e46d9ae-24a4-426f-ad44-77c951440c18", 2022-09-12T19:22:51.934 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "remove_zg", 2022-09-12T19:22:51.935 INFO:teuthology.orchestra.run.smithi035.stdout: "api_name": "remove_zg", 2022-09-12T19:22:51.935 INFO:teuthology.orchestra.run.smithi035.stdout: "is_master": false, 2022-09-12T19:22:51.935 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [], 2022-09-12T19:22:51.935 INFO:teuthology.orchestra.run.smithi035.stdout: "hostnames": [], 2022-09-12T19:22:51.936 INFO:teuthology.orchestra.run.smithi035.stdout: "hostnames_s3website": [], 2022-09-12T19:22:51.936 INFO:teuthology.orchestra.run.smithi035.stdout: "master_zone": "ad741acb-53d5-4810-a2b0-4448fda5b3ca", 2022-09-12T19:22:51.936 INFO:teuthology.orchestra.run.smithi035.stdout: "zones": [ 2022-09-12T19:22:51.936 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.937 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "ad741acb-53d5-4810-a2b0-4448fda5b3ca", 2022-09-12T19:22:51.937 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "remove", 2022-09-12T19:22:51.937 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [], 2022-09-12T19:22:51.937 INFO:teuthology.orchestra.run.smithi035.stdout: "log_meta": false, 2022-09-12T19:22:51.938 INFO:teuthology.orchestra.run.smithi035.stdout: "log_data": false, 2022-09-12T19:22:51.938 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:22:51.938 INFO:teuthology.orchestra.run.smithi035.stdout: "read_only": false, 2022-09-12T19:22:51.938 INFO:teuthology.orchestra.run.smithi035.stdout: "tier_type": "", 2022-09-12T19:22:51.939 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from_all": true, 2022-09-12T19:22:51.939 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from": [], 2022-09-12T19:22:51.939 INFO:teuthology.orchestra.run.smithi035.stdout: "redirect_zone": "", 2022-09-12T19:22:51.940 INFO:teuthology.orchestra.run.smithi035.stdout: "supported_features": [ 2022-09-12T19:22:51.940 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:22:51.940 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:51.940 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:51.941 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.941 INFO:teuthology.orchestra.run.smithi035.stdout: "placement_targets": [ 2022-09-12T19:22:51.941 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.941 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "default-placement", 2022-09-12T19:22:51.942 INFO:teuthology.orchestra.run.smithi035.stdout: "tags": [], 2022-09-12T19:22:51.942 INFO:teuthology.orchestra.run.smithi035.stdout: "storage_classes": [ 2022-09-12T19:22:51.942 INFO:teuthology.orchestra.run.smithi035.stdout: "STANDARD" 2022-09-12T19:22:51.943 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:51.943 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:51.943 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.943 INFO:teuthology.orchestra.run.smithi035.stdout: "default_placement": "default-placement", 2022-09-12T19:22:51.944 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:22:51.944 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_policy": { 2022-09-12T19:22:51.944 INFO:teuthology.orchestra.run.smithi035.stdout: "groups": [] 2022-09-12T19:22:51.944 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.945 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled_features": [ 2022-09-12T19:22:51.945 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:22:51.945 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:51.945 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.945 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.946 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "test-zonegroup", 2022-09-12T19:22:51.946 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zonegroup", 2022-09-12T19:22:51.946 INFO:teuthology.orchestra.run.smithi035.stdout: "api_name": "", 2022-09-12T19:22:51.946 INFO:teuthology.orchestra.run.smithi035.stdout: "is_master": true, 2022-09-12T19:22:51.947 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T19:22:51.947 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T19:22:51.947 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.947 INFO:teuthology.orchestra.run.smithi035.stdout: "hostnames": [], 2022-09-12T19:22:51.948 INFO:teuthology.orchestra.run.smithi035.stdout: "hostnames_s3website": [], 2022-09-12T19:22:51.948 INFO:teuthology.orchestra.run.smithi035.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:22:51.948 INFO:teuthology.orchestra.run.smithi035.stdout: "zones": [ 2022-09-12T19:22:51.948 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.949 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T19:22:51.949 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone3", 2022-09-12T19:22:51.949 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T19:22:51.950 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi035.front.sepia.ceph.com:8001" 2022-09-12T19:22:51.950 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.951 INFO:teuthology.orchestra.run.smithi035.stdout: "log_meta": false, 2022-09-12T19:22:51.951 INFO:teuthology.orchestra.run.smithi035.stdout: "log_data": true, 2022-09-12T19:22:51.952 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:22:51.952 INFO:teuthology.orchestra.run.smithi035.stdout: "read_only": false, 2022-09-12T19:22:51.953 INFO:teuthology.orchestra.run.smithi035.stdout: "tier_type": "", 2022-09-12T19:22:51.953 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from_all": true, 2022-09-12T19:22:51.954 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from": [], 2022-09-12T19:22:51.954 INFO:teuthology.orchestra.run.smithi035.stdout: "redirect_zone": "", 2022-09-12T19:22:51.955 INFO:teuthology.orchestra.run.smithi035.stdout: "supported_features": [ 2022-09-12T19:22:51.955 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:22:51.956 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:51.956 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.956 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.957 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "eb41eca3-59e3-4448-adac-b7c5067fb406", 2022-09-12T19:22:51.957 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone4", 2022-09-12T19:22:51.958 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T19:22:51.958 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi138.front.sepia.ceph.com:8001" 2022-09-12T19:22:51.959 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.959 INFO:teuthology.orchestra.run.smithi035.stdout: "log_meta": false, 2022-09-12T19:22:51.960 INFO:teuthology.orchestra.run.smithi035.stdout: "log_data": true, 2022-09-12T19:22:51.960 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:22:51.961 INFO:teuthology.orchestra.run.smithi035.stdout: "read_only": false, 2022-09-12T19:22:51.961 INFO:teuthology.orchestra.run.smithi035.stdout: "tier_type": "pubsub", 2022-09-12T19:22:51.961 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from_all": false, 2022-09-12T19:22:51.961 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from": [ 2022-09-12T19:22:51.962 INFO:teuthology.orchestra.run.smithi035.stdout: "test-zone1" 2022-09-12T19:22:51.962 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.963 INFO:teuthology.orchestra.run.smithi035.stdout: "redirect_zone": "", 2022-09-12T19:22:51.963 INFO:teuthology.orchestra.run.smithi035.stdout: "supported_features": [ 2022-09-12T19:22:51.964 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:22:51.964 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:51.967 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.967 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.967 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:22:51.968 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone1", 2022-09-12T19:22:51.968 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T19:22:51.969 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T19:22:51.969 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.970 INFO:teuthology.orchestra.run.smithi035.stdout: "log_meta": false, 2022-09-12T19:22:51.970 INFO:teuthology.orchestra.run.smithi035.stdout: "log_data": true, 2022-09-12T19:22:51.970 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:22:51.971 INFO:teuthology.orchestra.run.smithi035.stdout: "read_only": false, 2022-09-12T19:22:51.971 INFO:teuthology.orchestra.run.smithi035.stdout: "tier_type": "", 2022-09-12T19:22:51.971 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from_all": true, 2022-09-12T19:22:51.972 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from": [], 2022-09-12T19:22:51.972 INFO:teuthology.orchestra.run.smithi035.stdout: "redirect_zone": "", 2022-09-12T19:22:51.972 INFO:teuthology.orchestra.run.smithi035.stdout: "supported_features": [ 2022-09-12T19:22:51.972 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:22:51.973 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:51.973 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.973 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.973 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:22:51.974 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone2", 2022-09-12T19:22:51.974 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T19:22:51.974 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi138.front.sepia.ceph.com:8000" 2022-09-12T19:22:51.975 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.975 INFO:teuthology.orchestra.run.smithi035.stdout: "log_meta": false, 2022-09-12T19:22:51.975 INFO:teuthology.orchestra.run.smithi035.stdout: "log_data": true, 2022-09-12T19:22:51.976 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:22:51.976 INFO:teuthology.orchestra.run.smithi035.stdout: "read_only": false, 2022-09-12T19:22:51.976 INFO:teuthology.orchestra.run.smithi035.stdout: "tier_type": "", 2022-09-12T19:22:51.977 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from_all": true, 2022-09-12T19:22:51.977 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from": [], 2022-09-12T19:22:51.977 INFO:teuthology.orchestra.run.smithi035.stdout: "redirect_zone": "", 2022-09-12T19:22:51.977 INFO:teuthology.orchestra.run.smithi035.stdout: "supported_features": [ 2022-09-12T19:22:51.978 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:22:51.978 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:51.978 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:51.979 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.979 INFO:teuthology.orchestra.run.smithi035.stdout: "placement_targets": [ 2022-09-12T19:22:51.979 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.979 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "default-placement", 2022-09-12T19:22:51.980 INFO:teuthology.orchestra.run.smithi035.stdout: "tags": [], 2022-09-12T19:22:51.980 INFO:teuthology.orchestra.run.smithi035.stdout: "storage_classes": [ 2022-09-12T19:22:51.980 INFO:teuthology.orchestra.run.smithi035.stdout: "STANDARD" 2022-09-12T19:22:51.980 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:51.981 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:51.981 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.981 INFO:teuthology.orchestra.run.smithi035.stdout: "default_placement": "default-placement", 2022-09-12T19:22:51.981 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:22:51.982 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_policy": { 2022-09-12T19:22:51.982 INFO:teuthology.orchestra.run.smithi035.stdout: "groups": [] 2022-09-12T19:22:51.983 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.983 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled_features": [ 2022-09-12T19:22:51.983 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:22:51.983 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:51.984 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:51.984 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:51.984 INFO:teuthology.orchestra.run.smithi035.stdout: "short_zone_ids": [ 2022-09-12T19:22:51.985 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.985 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T19:22:51.985 INFO:teuthology.orchestra.run.smithi035.stdout: "val": 227168046 2022-09-12T19:22:51.986 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.986 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.986 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "ad741acb-53d5-4810-a2b0-4448fda5b3ca", 2022-09-12T19:22:51.986 INFO:teuthology.orchestra.run.smithi035.stdout: "val": 937609069 2022-09-12T19:22:51.987 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.987 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.987 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "eb41eca3-59e3-4448-adac-b7c5067fb406", 2022-09-12T19:22:51.987 INFO:teuthology.orchestra.run.smithi035.stdout: "val": 2080702172 2022-09-12T19:22:51.988 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.988 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.988 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:22:51.988 INFO:teuthology.orchestra.run.smithi035.stdout: "val": 422195086 2022-09-12T19:22:51.989 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.989 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:51.989 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:22:51.989 INFO:teuthology.orchestra.run.smithi035.stdout: "val": 2881234289 2022-09-12T19:22:51.990 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:51.990 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:51.991 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.991 INFO:teuthology.orchestra.run.smithi035.stdout: "master_zonegroup": "test-zonegroup", 2022-09-12T19:22:51.991 INFO:teuthology.orchestra.run.smithi035.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:22:51.991 INFO:teuthology.orchestra.run.smithi035.stdout: "period_config": { 2022-09-12T19:22:51.992 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_quota": { 2022-09-12T19:22:51.992 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false, 2022-09-12T19:22:51.992 INFO:teuthology.orchestra.run.smithi035.stdout: "check_on_raw": false, 2022-09-12T19:22:51.992 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size": -1, 2022-09-12T19:22:51.993 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size_kb": 0, 2022-09-12T19:22:51.993 INFO:teuthology.orchestra.run.smithi035.stdout: "max_objects": -1 2022-09-12T19:22:51.993 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.994 INFO:teuthology.orchestra.run.smithi035.stdout: "user_quota": { 2022-09-12T19:22:51.994 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false, 2022-09-12T19:22:51.994 INFO:teuthology.orchestra.run.smithi035.stdout: "check_on_raw": false, 2022-09-12T19:22:51.995 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size": -1, 2022-09-12T19:22:51.995 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size_kb": 0, 2022-09-12T19:22:51.995 INFO:teuthology.orchestra.run.smithi035.stdout: "max_objects": -1 2022-09-12T19:22:51.996 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.996 INFO:teuthology.orchestra.run.smithi035.stdout: "user_ratelimit": { 2022-09-12T19:22:51.996 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_ops": 0, 2022-09-12T19:22:51.996 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_ops": 0, 2022-09-12T19:22:51.997 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_bytes": 0, 2022-09-12T19:22:51.997 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_bytes": 0, 2022-09-12T19:22:51.997 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false 2022-09-12T19:22:51.997 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.998 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_ratelimit": { 2022-09-12T19:22:51.998 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_ops": 0, 2022-09-12T19:22:51.998 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_ops": 0, 2022-09-12T19:22:51.998 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_bytes": 0, 2022-09-12T19:22:51.999 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_bytes": 0, 2022-09-12T19:22:51.999 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false 2022-09-12T19:22:51.999 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:51.999 INFO:teuthology.orchestra.run.smithi035.stdout: "anonymous_ratelimit": { 2022-09-12T19:22:52.000 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_ops": 0, 2022-09-12T19:22:52.000 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_ops": 0, 2022-09-12T19:22:52.000 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_bytes": 0, 2022-09-12T19:22:52.000 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_bytes": 0, 2022-09-12T19:22:52.001 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false 2022-09-12T19:22:52.001 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:52.001 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:52.001 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:22:52.002 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_name": "test-realm", 2022-09-12T19:22:52.002 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:22:52.002 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:22:52.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:52.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:52.004 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:22:52.064 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:22:52.065 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:22:52.103 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:22:52.101+0000 7f912500b600 0 zone id ad741acb-53d5-4810-a2b0-4448fda5b3ca is not a part of zonegroup test-zonegroup 2022-09-12T19:22:52.104 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:22:52.101+0000 7f912500b600 0 zone id ad741acb-53d5-4810-a2b0-4448fda5b3ca is not a part of zonegroup default 2022-09-12T19:22:52.118 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:52.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:52.120 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:22:52.174 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:22:52.175 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:22:52.207 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:22:52.204+0000 7ffaeb1cd600 0 ERROR: zonegroup remove_zg has a non existent master zone 2022-09-12T19:22:52.211 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-09-12T19:22:52.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:52.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:52.213 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:22:52.266 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:22:52.267 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:22:52.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:52.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:22:52.299 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:22:52.356 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:22:52.356 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:22:52.419 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:22:52.419 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:22:52.420 INFO:teuthology.orchestra.run.smithi035.stdout: "epoch": 9, 2022-09-12T19:22:52.421 INFO:teuthology.orchestra.run.smithi035.stdout: "predecessor_uuid": "2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e", 2022-09-12T19:22:52.422 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": [ 2022-09-12T19:22:52.422 INFO:teuthology.orchestra.run.smithi035.stdout: "", 2022-09-12T19:22:52.423 INFO:teuthology.orchestra.run.smithi035.stdout: "", 2022-09-12T19:22:52.424 INFO:teuthology.orchestra.run.smithi035.stdout: "1_1663010125.771044_393.1", 2022-09-12T19:22:52.424 INFO:teuthology.orchestra.run.smithi035.stdout: "" 2022-09-12T19:22:52.425 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:52.425 INFO:teuthology.orchestra.run.smithi035.stdout: "period_map": { 2022-09-12T19:22:52.426 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:22:52.427 INFO:teuthology.orchestra.run.smithi035.stdout: "zonegroups": [ 2022-09-12T19:22:52.427 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:52.428 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "test-zonegroup", 2022-09-12T19:22:52.428 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zonegroup", 2022-09-12T19:22:52.429 INFO:teuthology.orchestra.run.smithi035.stdout: "api_name": "", 2022-09-12T19:22:52.430 INFO:teuthology.orchestra.run.smithi035.stdout: "is_master": true, 2022-09-12T19:22:52.430 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T19:22:52.431 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T19:22:52.432 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:52.433 INFO:teuthology.orchestra.run.smithi035.stdout: "hostnames": [], 2022-09-12T19:22:52.433 INFO:teuthology.orchestra.run.smithi035.stdout: "hostnames_s3website": [], 2022-09-12T19:22:52.434 INFO:teuthology.orchestra.run.smithi035.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:22:52.435 INFO:teuthology.orchestra.run.smithi035.stdout: "zones": [ 2022-09-12T19:22:52.436 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:52.437 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T19:22:52.437 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone3", 2022-09-12T19:22:52.437 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T19:22:52.438 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi035.front.sepia.ceph.com:8001" 2022-09-12T19:22:52.439 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:52.439 INFO:teuthology.orchestra.run.smithi035.stdout: "log_meta": false, 2022-09-12T19:22:52.440 INFO:teuthology.orchestra.run.smithi035.stdout: "log_data": true, 2022-09-12T19:22:52.440 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:22:52.441 INFO:teuthology.orchestra.run.smithi035.stdout: "read_only": false, 2022-09-12T19:22:52.441 INFO:teuthology.orchestra.run.smithi035.stdout: "tier_type": "", 2022-09-12T19:22:52.442 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from_all": true, 2022-09-12T19:22:52.442 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from": [], 2022-09-12T19:22:52.443 INFO:teuthology.orchestra.run.smithi035.stdout: "redirect_zone": "", 2022-09-12T19:22:52.443 INFO:teuthology.orchestra.run.smithi035.stdout: "supported_features": [ 2022-09-12T19:22:52.444 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:22:52.444 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:52.445 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:52.445 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:52.446 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "eb41eca3-59e3-4448-adac-b7c5067fb406", 2022-09-12T19:22:52.446 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone4", 2022-09-12T19:22:52.447 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T19:22:52.447 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi138.front.sepia.ceph.com:8001" 2022-09-12T19:22:52.448 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:52.448 INFO:teuthology.orchestra.run.smithi035.stdout: "log_meta": false, 2022-09-12T19:22:52.449 INFO:teuthology.orchestra.run.smithi035.stdout: "log_data": true, 2022-09-12T19:22:52.449 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:22:52.450 INFO:teuthology.orchestra.run.smithi035.stdout: "read_only": false, 2022-09-12T19:22:52.450 INFO:teuthology.orchestra.run.smithi035.stdout: "tier_type": "pubsub", 2022-09-12T19:22:52.451 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from_all": false, 2022-09-12T19:22:52.451 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from": [ 2022-09-12T19:22:52.452 INFO:teuthology.orchestra.run.smithi035.stdout: "test-zone1" 2022-09-12T19:22:52.453 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:52.453 INFO:teuthology.orchestra.run.smithi035.stdout: "redirect_zone": "", 2022-09-12T19:22:52.454 INFO:teuthology.orchestra.run.smithi035.stdout: "supported_features": [ 2022-09-12T19:22:52.454 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:22:52.455 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:52.455 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:52.456 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:52.456 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:22:52.456 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone1", 2022-09-12T19:22:52.457 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T19:22:52.457 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi035.front.sepia.ceph.com:8000" 2022-09-12T19:22:52.457 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:52.458 INFO:teuthology.orchestra.run.smithi035.stdout: "log_meta": false, 2022-09-12T19:22:52.458 INFO:teuthology.orchestra.run.smithi035.stdout: "log_data": true, 2022-09-12T19:22:52.458 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:22:52.459 INFO:teuthology.orchestra.run.smithi035.stdout: "read_only": false, 2022-09-12T19:22:52.459 INFO:teuthology.orchestra.run.smithi035.stdout: "tier_type": "", 2022-09-12T19:22:52.459 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from_all": true, 2022-09-12T19:22:52.459 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from": [], 2022-09-12T19:22:52.460 INFO:teuthology.orchestra.run.smithi035.stdout: "redirect_zone": "", 2022-09-12T19:22:52.460 INFO:teuthology.orchestra.run.smithi035.stdout: "supported_features": [ 2022-09-12T19:22:52.460 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:22:52.460 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:52.461 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:52.461 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:52.461 INFO:teuthology.orchestra.run.smithi035.stdout: "id": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:22:52.461 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "test-zone2", 2022-09-12T19:22:52.462 INFO:teuthology.orchestra.run.smithi035.stdout: "endpoints": [ 2022-09-12T19:22:52.462 INFO:teuthology.orchestra.run.smithi035.stdout: "http://smithi138.front.sepia.ceph.com:8000" 2022-09-12T19:22:52.462 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:52.462 INFO:teuthology.orchestra.run.smithi035.stdout: "log_meta": false, 2022-09-12T19:22:52.463 INFO:teuthology.orchestra.run.smithi035.stdout: "log_data": true, 2022-09-12T19:22:52.463 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_index_max_shards": 11, 2022-09-12T19:22:52.463 INFO:teuthology.orchestra.run.smithi035.stdout: "read_only": false, 2022-09-12T19:22:52.463 INFO:teuthology.orchestra.run.smithi035.stdout: "tier_type": "", 2022-09-12T19:22:52.464 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from_all": true, 2022-09-12T19:22:52.464 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_from": [], 2022-09-12T19:22:52.464 INFO:teuthology.orchestra.run.smithi035.stdout: "redirect_zone": "", 2022-09-12T19:22:52.464 INFO:teuthology.orchestra.run.smithi035.stdout: "supported_features": [ 2022-09-12T19:22:52.464 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:22:52.465 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:52.465 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:52.465 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:52.465 INFO:teuthology.orchestra.run.smithi035.stdout: "placement_targets": [ 2022-09-12T19:22:52.466 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:52.466 INFO:teuthology.orchestra.run.smithi035.stdout: "name": "default-placement", 2022-09-12T19:22:52.468 INFO:teuthology.orchestra.run.smithi035.stdout: "tags": [], 2022-09-12T19:22:52.468 INFO:teuthology.orchestra.run.smithi035.stdout: "storage_classes": [ 2022-09-12T19:22:52.468 INFO:teuthology.orchestra.run.smithi035.stdout: "STANDARD" 2022-09-12T19:22:52.468 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:52.469 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:52.469 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:52.469 INFO:teuthology.orchestra.run.smithi035.stdout: "default_placement": "default-placement", 2022-09-12T19:22:52.470 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:22:52.470 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_policy": { 2022-09-12T19:22:52.470 INFO:teuthology.orchestra.run.smithi035.stdout: "groups": [] 2022-09-12T19:22:52.470 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:52.471 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled_features": [ 2022-09-12T19:22:52.471 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding" 2022-09-12T19:22:52.471 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:52.471 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:52.472 INFO:teuthology.orchestra.run.smithi035.stdout: ], 2022-09-12T19:22:52.472 INFO:teuthology.orchestra.run.smithi035.stdout: "short_zone_ids": [ 2022-09-12T19:22:52.472 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:52.472 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T19:22:52.473 INFO:teuthology.orchestra.run.smithi035.stdout: "val": 227168046 2022-09-12T19:22:52.473 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:52.473 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:52.473 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "eb41eca3-59e3-4448-adac-b7c5067fb406", 2022-09-12T19:22:52.474 INFO:teuthology.orchestra.run.smithi035.stdout: "val": 2080702172 2022-09-12T19:22:52.474 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:52.475 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:52.475 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:22:52.475 INFO:teuthology.orchestra.run.smithi035.stdout: "val": 422195086 2022-09-12T19:22:52.476 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:52.476 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:52.476 INFO:teuthology.orchestra.run.smithi035.stdout: "key": "fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:22:52.477 INFO:teuthology.orchestra.run.smithi035.stdout: "val": 2881234289 2022-09-12T19:22:52.477 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:52.477 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:22:52.478 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:52.478 INFO:teuthology.orchestra.run.smithi035.stdout: "master_zonegroup": "test-zonegroup", 2022-09-12T19:22:52.479 INFO:teuthology.orchestra.run.smithi035.stdout: "master_zone": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:22:52.479 INFO:teuthology.orchestra.run.smithi035.stdout: "period_config": { 2022-09-12T19:22:52.479 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_quota": { 2022-09-12T19:22:52.480 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false, 2022-09-12T19:22:52.480 INFO:teuthology.orchestra.run.smithi035.stdout: "check_on_raw": false, 2022-09-12T19:22:52.480 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size": -1, 2022-09-12T19:22:52.480 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size_kb": 0, 2022-09-12T19:22:52.481 INFO:teuthology.orchestra.run.smithi035.stdout: "max_objects": -1 2022-09-12T19:22:52.481 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:52.481 INFO:teuthology.orchestra.run.smithi035.stdout: "user_quota": { 2022-09-12T19:22:52.481 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false, 2022-09-12T19:22:52.482 INFO:teuthology.orchestra.run.smithi035.stdout: "check_on_raw": false, 2022-09-12T19:22:52.482 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size": -1, 2022-09-12T19:22:52.482 INFO:teuthology.orchestra.run.smithi035.stdout: "max_size_kb": 0, 2022-09-12T19:22:52.483 INFO:teuthology.orchestra.run.smithi035.stdout: "max_objects": -1 2022-09-12T19:22:52.483 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:52.483 INFO:teuthology.orchestra.run.smithi035.stdout: "user_ratelimit": { 2022-09-12T19:22:52.483 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_ops": 0, 2022-09-12T19:22:52.483 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_ops": 0, 2022-09-12T19:22:52.484 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_bytes": 0, 2022-09-12T19:22:52.484 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_bytes": 0, 2022-09-12T19:22:52.484 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false 2022-09-12T19:22:52.484 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:52.485 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket_ratelimit": { 2022-09-12T19:22:52.485 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_ops": 0, 2022-09-12T19:22:52.485 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_ops": 0, 2022-09-12T19:22:52.485 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_bytes": 0, 2022-09-12T19:22:52.486 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_bytes": 0, 2022-09-12T19:22:52.486 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false 2022-09-12T19:22:52.486 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:52.487 INFO:teuthology.orchestra.run.smithi035.stdout: "anonymous_ratelimit": { 2022-09-12T19:22:52.487 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_ops": 0, 2022-09-12T19:22:52.487 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_ops": 0, 2022-09-12T19:22:52.487 INFO:teuthology.orchestra.run.smithi035.stdout: "max_read_bytes": 0, 2022-09-12T19:22:52.488 INFO:teuthology.orchestra.run.smithi035.stdout: "max_write_bytes": 0, 2022-09-12T19:22:52.488 INFO:teuthology.orchestra.run.smithi035.stdout: "enabled": false 2022-09-12T19:22:52.488 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:52.488 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:52.489 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_id": "06f742c1-7106-48ed-a794-ceccfd2207bd", 2022-09-12T19:22:52.489 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_name": "test-realm", 2022-09-12T19:22:52.489 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:22:52.490 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:22:52.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2022-09-12T19:22:52.491 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-50 2022-09-12T19:22:52.492 DEBUG:boto:path=/jdpiak-50/ 2022-09-12T19:22:52.493 DEBUG:boto:auth_path=/jdpiak-50/ 2022-09-12T19:22:52.493 DEBUG:boto:Method: PUT 2022-09-12T19:22:52.494 DEBUG:boto:Path: /jdpiak-50/ 2022-09-12T19:22:52.494 DEBUG:boto:Data: 2022-09-12T19:22:52.495 DEBUG:boto:Headers: {} 2022-09-12T19:22:52.495 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:22:52.496 DEBUG:boto:Port: 8000 2022-09-12T19:22:52.496 DEBUG:boto:Params: {} 2022-09-12T19:22:52.497 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:22:52.498 DEBUG:boto:Token: None 2022-09-12T19:22:52.499 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:22:52 GMT /jdpiak-50/ 2022-09-12T19:22:52.500 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:U39UD7mYpUKBPreJENrm1keaWPc= 2022-09-12T19:22:52.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:22:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:U39UD7mYpUKBPreJENrm1keaWPc=', 'Content-Length': '0'} 2022-09-12T19:22:52.573 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003caf8273da7dd313-00631f870c-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:22:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:22:52.575 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jdpiak-51 2022-09-12T19:22:52.575 DEBUG:boto:path=/jdpiak-51/ 2022-09-12T19:22:52.576 DEBUG:boto:auth_path=/jdpiak-51/ 2022-09-12T19:22:52.577 DEBUG:boto:Method: PUT 2022-09-12T19:22:52.578 DEBUG:boto:Path: /jdpiak-51/ 2022-09-12T19:22:52.578 DEBUG:boto:Data: 2022-09-12T19:22:52.579 DEBUG:boto:Headers: {} 2022-09-12T19:22:52.580 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:22:52.581 DEBUG:boto:Port: 8000 2022-09-12T19:22:52.582 DEBUG:boto:Params: {} 2022-09-12T19:22:52.583 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:22:52.584 DEBUG:boto:Token: None 2022-09-12T19:22:52.585 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:22:52 GMT /jdpiak-51/ 2022-09-12T19:22:52.585 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ghCeNkpG+XPb1JjFZksKvr68XUY= 2022-09-12T19:22:52.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:22:52 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ghCeNkpG+XPb1JjFZksKvr68XUY=', 'Content-Length': '0'} 2022-09-12T19:22:53.137 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000577e312a5da45620-00631f870d-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:22:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:22:53.138 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jdpiak-52 2022-09-12T19:22:53.139 DEBUG:boto:path=/jdpiak-52/ 2022-09-12T19:22:53.139 DEBUG:boto:auth_path=/jdpiak-52/ 2022-09-12T19:22:53.140 DEBUG:boto:Method: PUT 2022-09-12T19:22:53.140 DEBUG:boto:Path: /jdpiak-52/ 2022-09-12T19:22:53.141 DEBUG:boto:Data: 2022-09-12T19:22:53.141 DEBUG:boto:Headers: {} 2022-09-12T19:22:53.142 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:22:53.142 DEBUG:boto:Port: 8001 2022-09-12T19:22:53.143 DEBUG:boto:Params: {} 2022-09-12T19:22:53.144 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T19:22:53.145 DEBUG:boto:Token: None 2022-09-12T19:22:53.145 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:22:53 GMT /jdpiak-52/ 2022-09-12T19:22:53.146 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:820IljT+0OVW+IiXR8dsxbUnXqY= 2022-09-12T19:22:53.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:22:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:820IljT+0OVW+IiXR8dsxbUnXqY=', 'Content-Length': '0'} 2022-09-12T19:22:53.185 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003629eb651b93750c-00631f870d-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:22:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:22:53.187 DEBUG:boto:path=/jdpiak-50/ 2022-09-12T19:22:53.188 DEBUG:boto:auth_path=/jdpiak-50/ 2022-09-12T19:22:53.189 DEBUG:boto:path=/jdpiak-50/?website 2022-09-12T19:22:53.190 DEBUG:boto:auth_path=/jdpiak-50/?website 2022-09-12T19:22:53.191 DEBUG:boto:Method: PUT 2022-09-12T19:22:53.191 DEBUG:boto:Path: /jdpiak-50/?website 2022-09-12T19:22:53.192 DEBUG:boto:Data: index.htmlerror.html 2022-09-12T19:22:53.193 DEBUG:boto:Headers: {} 2022-09-12T19:22:53.194 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:22:53.194 DEBUG:boto:Port: 8000 2022-09-12T19:22:53.195 DEBUG:boto:Params: {} 2022-09-12T19:22:53.196 DEBUG:boto:Token: None 2022-09-12T19:22:53.196 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:22:53 GMT /jdpiak-50/?website 2022-09-12T19:22:53.197 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LlCDou6qPbfSuNwHWZKKuNetWZE= 2022-09-12T19:22:53.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:22:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LlCDou6qPbfSuNwHWZKKuNetWZE=', 'Content-Length': '241'} 2022-09-12T19:22:53.199 DEBUG:boto:Response headers: [('Content-Length', '202'), ('x-amz-request-id', 'tx00000e420829d94401629-00631f870d-15c0-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:22:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:22:53.200 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2022-09-12T19:22:53.204 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-53 2022-09-12T19:22:53.204 DEBUG:boto:path=/jdpiak-53/ 2022-09-12T19:22:53.205 DEBUG:boto:auth_path=/jdpiak-53/ 2022-09-12T19:22:53.205 DEBUG:boto:Method: PUT 2022-09-12T19:22:53.206 DEBUG:boto:Path: /jdpiak-53/ 2022-09-12T19:22:53.206 DEBUG:boto:Data: 2022-09-12T19:22:53.207 DEBUG:boto:Headers: {} 2022-09-12T19:22:53.207 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:22:53.208 DEBUG:boto:Port: 8000 2022-09-12T19:22:53.208 DEBUG:boto:Params: {} 2022-09-12T19:22:53.209 DEBUG:boto:Token: None 2022-09-12T19:22:53.209 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:22:53 GMT /jdpiak-53/ 2022-09-12T19:22:53.210 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HaTW0q92KbBkjOaqvL1yMnyPxVM= 2022-09-12T19:22:53.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:22:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HaTW0q92KbBkjOaqvL1yMnyPxVM=', 'Content-Length': '0'} 2022-09-12T19:22:53.253 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000065930557983d0239-00631f870d-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:22:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:22:53.253 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jdpiak-54 2022-09-12T19:22:53.254 DEBUG:boto:path=/jdpiak-54/ 2022-09-12T19:22:53.254 DEBUG:boto:auth_path=/jdpiak-54/ 2022-09-12T19:22:53.255 DEBUG:boto:Method: PUT 2022-09-12T19:22:53.255 DEBUG:boto:Path: /jdpiak-54/ 2022-09-12T19:22:53.256 DEBUG:boto:Data: 2022-09-12T19:22:53.256 DEBUG:boto:Headers: {} 2022-09-12T19:22:53.257 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:22:53.257 DEBUG:boto:Port: 8000 2022-09-12T19:22:53.258 DEBUG:boto:Params: {} 2022-09-12T19:22:53.259 DEBUG:boto:Token: None 2022-09-12T19:22:53.259 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:22:53 GMT /jdpiak-54/ 2022-09-12T19:22:53.260 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1pNrlnJSxGWaz0UZukKU8/3kMKg= 2022-09-12T19:22:53.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:22:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1pNrlnJSxGWaz0UZukKU8/3kMKg=', 'Content-Length': '0'} 2022-09-12T19:22:53.304 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000044e12eec241ba177-00631f870d-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:22:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:22:53.305 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jdpiak-55 2022-09-12T19:22:53.305 DEBUG:boto:path=/jdpiak-55/ 2022-09-12T19:22:53.306 DEBUG:boto:auth_path=/jdpiak-55/ 2022-09-12T19:22:53.307 DEBUG:boto:Method: PUT 2022-09-12T19:22:53.307 DEBUG:boto:Path: /jdpiak-55/ 2022-09-12T19:22:53.308 DEBUG:boto:Data: 2022-09-12T19:22:53.308 DEBUG:boto:Headers: {} 2022-09-12T19:22:53.308 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:22:53.309 DEBUG:boto:Port: 8001 2022-09-12T19:22:53.309 DEBUG:boto:Params: {} 2022-09-12T19:22:53.310 DEBUG:boto:Token: None 2022-09-12T19:22:53.311 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:22:53 GMT /jdpiak-55/ 2022-09-12T19:22:53.311 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XO+YWNE6culbyBrwK1gyqYZCan4= 2022-09-12T19:22:53.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:22:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XO+YWNE6culbyBrwK1gyqYZCan4=', 'Content-Length': '0'} 2022-09-12T19:22:53.341 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000063db0cdaf4e776f9-00631f870d-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:22:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:22:53.342 DEBUG:boto:path=/jdpiak-53/ 2022-09-12T19:22:53.343 DEBUG:boto:auth_path=/jdpiak-53/ 2022-09-12T19:22:53.344 DEBUG:boto:path=/jdpiak-53/?policy 2022-09-12T19:22:53.345 DEBUG:boto:auth_path=/jdpiak-53/?policy 2022-09-12T19:22:53.346 DEBUG:boto:Method: PUT 2022-09-12T19:22:53.347 DEBUG:boto:Path: /jdpiak-53/?policy 2022-09-12T19:22:53.348 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2022-09-12T19:22:53.349 DEBUG:boto:Headers: {} 2022-09-12T19:22:53.350 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:22:53.351 DEBUG:boto:Port: 8000 2022-09-12T19:22:53.352 DEBUG:boto:Params: {} 2022-09-12T19:22:53.353 DEBUG:boto:Token: None 2022-09-12T19:22:53.354 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:22:53 GMT /jdpiak-53/?policy 2022-09-12T19:22:53.355 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L7ah6HTig+s0eMXt1hFPe4pAGYU= 2022-09-12T19:22:53.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:22:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L7ah6HTig+s0eMXt1hFPe4pAGYU=', 'Content-Length': '97'} 2022-09-12T19:22:53.363 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000066936eef98645130-00631f870d-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:22:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:22:53.365 DEBUG:boto:path=/jdpiak-53/ 2022-09-12T19:22:53.366 DEBUG:boto:auth_path=/jdpiak-53/ 2022-09-12T19:22:53.366 DEBUG:boto:path=/jdpiak-53/?policy 2022-09-12T19:22:53.368 DEBUG:boto:auth_path=/jdpiak-53/?policy 2022-09-12T19:22:53.368 DEBUG:boto:Method: GET 2022-09-12T19:22:53.370 DEBUG:boto:Path: /jdpiak-53/?policy 2022-09-12T19:22:53.371 DEBUG:boto:Data: 2022-09-12T19:22:53.371 DEBUG:boto:Headers: {} 2022-09-12T19:22:53.373 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:22:53.374 DEBUG:boto:Port: 8000 2022-09-12T19:22:53.375 DEBUG:boto:Params: {} 2022-09-12T19:22:53.376 DEBUG:boto:Token: None 2022-09-12T19:22:53.377 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:22:53 GMT /jdpiak-53/?policy 2022-09-12T19:22:53.378 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:arZ19ZHmI3oHvgsqumxP8xhdKdg= 2022-09-12T19:22:53.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:22:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:arZ19ZHmI3oHvgsqumxP8xhdKdg=', 'Content-Length': '0'} 2022-09-12T19:22:53.381 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000044085196ec28e561-00631f870d-15c0-test-zone1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Mon, 12 Sep 2022 19:22:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:22:53.382 DEBUG:boto:path=/jdpiak-54/ 2022-09-12T19:22:53.384 DEBUG:boto:auth_path=/jdpiak-54/ 2022-09-12T19:22:53.385 DEBUG:boto:path=/jdpiak-54/?policy 2022-09-12T19:22:53.385 DEBUG:boto:auth_path=/jdpiak-54/?policy 2022-09-12T19:22:53.386 DEBUG:boto:Method: PUT 2022-09-12T19:22:53.386 DEBUG:boto:Path: /jdpiak-54/?policy 2022-09-12T19:22:53.387 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2022-09-12T19:22:53.387 DEBUG:boto:Headers: {} 2022-09-12T19:22:53.388 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:22:53.389 DEBUG:boto:Port: 8000 2022-09-12T19:22:53.390 DEBUG:boto:Params: {} 2022-09-12T19:22:53.390 DEBUG:boto:Token: None 2022-09-12T19:22:53.391 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:22:53 GMT /jdpiak-54/?policy 2022-09-12T19:22:53.392 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IgLXlfHfZzfIDv9ZotEaVtptkKo= 2022-09-12T19:22:53.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:22:53 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IgLXlfHfZzfIDv9ZotEaVtptkKo=', 'Content-Length': '97'} 2022-09-12T19:22:54.403 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f7879d50f2b7bc73-00631f870d-15e1-test-zone2'), ('Date', 'Mon, 12 Sep 2022 19:22:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:22:54.404 DEBUG:boto:path=/jdpiak-54/ 2022-09-12T19:22:54.405 DEBUG:boto:auth_path=/jdpiak-54/ 2022-09-12T19:22:54.405 DEBUG:boto:path=/jdpiak-54/?policy 2022-09-12T19:22:54.406 DEBUG:boto:auth_path=/jdpiak-54/?policy 2022-09-12T19:22:54.407 DEBUG:boto:Method: GET 2022-09-12T19:22:54.407 DEBUG:boto:Path: /jdpiak-54/?policy 2022-09-12T19:22:54.408 DEBUG:boto:Data: 2022-09-12T19:22:54.409 DEBUG:boto:Headers: {} 2022-09-12T19:22:54.410 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:22:54.411 DEBUG:boto:Port: 8000 2022-09-12T19:22:54.412 DEBUG:boto:Params: {} 2022-09-12T19:22:54.413 DEBUG:boto:Token: None 2022-09-12T19:22:54.414 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:22:54 GMT /jdpiak-54/?policy 2022-09-12T19:22:54.415 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZdXK8+6zSe2dWZKJUym+hFeVhA0= 2022-09-12T19:22:54.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:22:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZdXK8+6zSe2dWZKJUym+hFeVhA0=', 'Content-Length': '0'} 2022-09-12T19:22:54.418 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a30dbc94297af338-00631f870e-15e1-test-zone2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Mon, 12 Sep 2022 19:22:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:22:54.418 DEBUG:boto:path=/jdpiak-55/ 2022-09-12T19:22:54.419 DEBUG:boto:auth_path=/jdpiak-55/ 2022-09-12T19:22:54.419 DEBUG:boto:path=/jdpiak-55/?policy 2022-09-12T19:22:54.420 DEBUG:boto:auth_path=/jdpiak-55/?policy 2022-09-12T19:22:54.420 DEBUG:boto:Method: PUT 2022-09-12T19:22:54.421 DEBUG:boto:Path: /jdpiak-55/?policy 2022-09-12T19:22:54.421 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2022-09-12T19:22:54.422 DEBUG:boto:Headers: {} 2022-09-12T19:22:54.422 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:22:54.423 DEBUG:boto:Port: 8001 2022-09-12T19:22:54.423 DEBUG:boto:Params: {} 2022-09-12T19:22:54.424 DEBUG:boto:Token: None 2022-09-12T19:22:54.425 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:22:54 GMT /jdpiak-55/?policy 2022-09-12T19:22:54.425 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TaUE5uoPI/r3MWyVmbBowbqM/Kk= 2022-09-12T19:22:54.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:22:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TaUE5uoPI/r3MWyVmbBowbqM/Kk=', 'Content-Length': '97'} 2022-09-12T19:22:55.438 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000105a0a494d4ae82b-00631f870e-15c4-test-zone3'), ('Date', 'Mon, 12 Sep 2022 19:22:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:22:55.439 DEBUG:boto:path=/jdpiak-55/ 2022-09-12T19:22:55.440 DEBUG:boto:auth_path=/jdpiak-55/ 2022-09-12T19:22:55.441 DEBUG:boto:path=/jdpiak-55/?policy 2022-09-12T19:22:55.442 DEBUG:boto:auth_path=/jdpiak-55/?policy 2022-09-12T19:22:55.443 DEBUG:boto:Method: GET 2022-09-12T19:22:55.443 DEBUG:boto:Path: /jdpiak-55/?policy 2022-09-12T19:22:55.444 DEBUG:boto:Data: 2022-09-12T19:22:55.445 DEBUG:boto:Headers: {} 2022-09-12T19:22:55.446 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:22:55.446 DEBUG:boto:Port: 8001 2022-09-12T19:22:55.447 DEBUG:boto:Params: {} 2022-09-12T19:22:55.448 DEBUG:boto:Token: None 2022-09-12T19:22:55.449 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:22:55 GMT /jdpiak-55/?policy 2022-09-12T19:22:55.450 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fQAwvT9Iporz/kJNtwvwJqwqG4U= 2022-09-12T19:22:55.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:22:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fQAwvT9Iporz/kJNtwvwJqwqG4U=', 'Content-Length': '0'} 2022-09-12T19:22:55.453 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003d6d42992aaa8f2d-00631f870f-15c4-test-zone3'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Mon, 12 Sep 2022 19:22:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:22:55.454 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2022-09-12T19:22:55.455 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-56 2022-09-12T19:22:55.456 DEBUG:boto:path=/jdpiak-56/ 2022-09-12T19:22:55.457 DEBUG:boto:auth_path=/jdpiak-56/ 2022-09-12T19:22:55.458 DEBUG:boto:Method: PUT 2022-09-12T19:22:55.458 DEBUG:boto:Path: /jdpiak-56/ 2022-09-12T19:22:55.459 DEBUG:boto:Data: 2022-09-12T19:22:55.459 DEBUG:boto:Headers: {} 2022-09-12T19:22:55.460 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:22:55.460 DEBUG:boto:Port: 8000 2022-09-12T19:22:55.461 DEBUG:boto:Params: {} 2022-09-12T19:22:55.461 DEBUG:boto:Token: None 2022-09-12T19:22:55.462 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:22:55 GMT /jdpiak-56/ 2022-09-12T19:22:55.463 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b3YrdNFUPrf/JzrliYaHhWwy1bQ= 2022-09-12T19:22:55.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:22:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b3YrdNFUPrf/JzrliYaHhWwy1bQ=', 'Content-Length': '0'} 2022-09-12T19:22:55.497 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b553487288284fc7-00631f870f-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:22:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:22:55.500 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jdpiak-57 2022-09-12T19:22:55.501 DEBUG:boto:path=/jdpiak-57/ 2022-09-12T19:22:55.502 DEBUG:boto:auth_path=/jdpiak-57/ 2022-09-12T19:22:55.503 DEBUG:boto:Method: PUT 2022-09-12T19:22:55.504 DEBUG:boto:Path: /jdpiak-57/ 2022-09-12T19:22:55.505 DEBUG:boto:Data: 2022-09-12T19:22:55.507 DEBUG:boto:Headers: {} 2022-09-12T19:22:55.508 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:22:55.508 DEBUG:boto:Port: 8000 2022-09-12T19:22:55.509 DEBUG:boto:Params: {} 2022-09-12T19:22:55.510 DEBUG:boto:Token: None 2022-09-12T19:22:55.511 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:22:55 GMT /jdpiak-57/ 2022-09-12T19:22:55.512 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XSmkhsQ3ex5PnnMUlx8MvSLSmTI= 2022-09-12T19:22:55.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:22:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XSmkhsQ3ex5PnnMUlx8MvSLSmTI=', 'Content-Length': '0'} 2022-09-12T19:22:55.548 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005d3b71255e164b91-00631f870f-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:22:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:22:55.548 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jdpiak-58 2022-09-12T19:22:55.549 DEBUG:boto:path=/jdpiak-58/ 2022-09-12T19:22:55.549 DEBUG:boto:auth_path=/jdpiak-58/ 2022-09-12T19:22:55.550 DEBUG:boto:Method: PUT 2022-09-12T19:22:55.550 DEBUG:boto:Path: /jdpiak-58/ 2022-09-12T19:22:55.551 DEBUG:boto:Data: 2022-09-12T19:22:55.551 DEBUG:boto:Headers: {} 2022-09-12T19:22:55.552 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:22:55.552 DEBUG:boto:Port: 8001 2022-09-12T19:22:55.553 DEBUG:boto:Params: {} 2022-09-12T19:22:55.554 DEBUG:boto:Token: None 2022-09-12T19:22:55.554 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:22:55 GMT /jdpiak-58/ 2022-09-12T19:22:55.555 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UUd+Md1EMFeI6XwZJAnzO6SKoGw= 2022-09-12T19:22:55.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:22:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UUd+Md1EMFeI6XwZJAnzO6SKoGw=', 'Content-Length': '0'} 2022-09-12T19:22:55.600 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bd0b3d9a65c8ad42-00631f870f-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:22:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:22:55.602 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:22:55.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:22:55.603 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:22:55.656 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:22:55.657 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:22:55.692 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=38ca1f57-1358-496c-a99b-df49ee365a4f 2022-09-12T19:22:55.707 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:22:55.708 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:55.708 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:22:55.708 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:22:55.709 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:55.709 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:55.709 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:22:55.709 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:22:55.710 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:55.710 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:55.710 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663010575.581422_409.1", 2022-09-12T19:22:55.710 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:22:55.581422Z" 2022-09-12T19:22:55.711 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:22:55.711 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:22:55.711 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:22:55.712 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:22:55.712 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:22:55.712 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:22:55.712 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:22:55.713 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T19:22:55.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:22:55.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:22:55.715 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:22:55.772 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:22:55.772 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:22:55.805 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:22:55.802+0000 7f32e40cd600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:22:55.806 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:22:55.803+0000 7f32e40cd600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:22:55.828 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:22:55.829 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:22:55.829 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:22:55.829 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:22:55.830 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:22:55.830 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:22:55.830 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:22:55.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:55.831 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:22:55.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:22:55.831 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:22:55.831 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:22:55.832 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:22:55.832 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:22:55.832 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:22:55.833 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:22:55.833 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:22:55.833 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:22:55.833 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:22:55.834 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:22:55.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:55.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:22:55.835 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:22:55.836 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:22:55.836 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:22:55.837 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:22:55.837 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:22:55.838 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:22:55.838 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:22:55.838 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:22:55.839 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:22:55.839 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:22:55.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:55.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:22:55.840 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:22:55.841 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:22:55.841 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:22:55.841 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:22:55.842 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:22:55.842 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:22:55.843 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:22:55.843 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:22:55.843 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:22:55.843 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:22:55.844 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:55.844 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:22:55.844 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:22:55.844 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:22:55.845 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:22:55.845 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:22:55.845 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:22:55.846 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:22:55.846 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:22:55.846 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:22:55.846 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:22:55.847 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:22:55.847 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:22:55.847 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:22:55.848 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:55.848 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:22:55.848 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:22:55.848 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:22:55.849 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:22:55.849 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:22:55Z" 2022-09-12T19:22:55.849 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:22:55.851 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:22:55Z" } 2022-09-12T19:22:55.852 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:22:55.853 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:22:55.854 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:22:55.855 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010319.572818_269.1 2022-09-12T19:23:00.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:23:00.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:23:00.858 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:23:00.916 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:23:00.917 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:23:00.959 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:23:00.956+0000 7f73a259c600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:23:00.960 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:23:00.957+0000 7f73a259c600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:23:00.974 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:23:00.974 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:23:00.975 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:23:00.975 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:23:00.976 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:23:00.976 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:23:00.977 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:23:00.977 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:00.978 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:23:00.978 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:00.979 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:23:00.979 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:00.979 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:00.980 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:23:00.980 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:00.981 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:23:00.981 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:00.982 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:23:00.982 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:00.983 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:00.983 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:00.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:00.984 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:23:00.985 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:00.985 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:00.986 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:23:00.986 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:00.987 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:00.987 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:00.988 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:23:00.988 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:00.989 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:00.989 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:00.990 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:00.991 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:23:00.991 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:00.992 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:00.992 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:23:00.993 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:00.993 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:00.994 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:00.994 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:23:00.995 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:23:00.995 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:00.995 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:00.996 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:00.996 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:23:00.997 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:00.997 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:00.998 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:23:00.999 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:00.999 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:01.000 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:01.000 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:23:01.000 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:01.001 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:01.001 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:01.002 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:23:01.002 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:01.003 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:23:01.003 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:23:01.004 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:23:01.004 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:01.005 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:23:00Z" 2022-09-12T19:23:01.005 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:23:01.008 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:23:00Z" } 2022-09-12T19:23:01.009 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:23:01.010 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:23:01.011 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:23:01.012 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010319.572818_269.1 2022-09-12T19:23:06.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:23:06.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:23:06.015 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:23:06.074 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:23:06.074 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:23:06.116 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:23:06.113+0000 7f9ad3e8d600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:23:06.117 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:23:06.115+0000 7f9ad3e8d600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:23:06.135 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:23:06.135 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:23:06.135 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:23:06.136 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:23:06.136 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:23:06.136 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:23:06.136 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:23:06.137 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:06.137 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:23:06.137 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:06.138 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:23:06.138 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:06.138 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:06.139 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:23:06.139 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:06.139 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:23:06.140 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:06.140 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:23:06.140 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:06.140 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:06.141 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:06.141 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:06.141 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:23:06.141 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:06.142 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:06.142 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:23:06.142 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:06.143 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:06.143 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:06.143 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:23:06.143 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:06.144 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:06.144 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:06.144 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:06.144 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:23:06.145 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:06.145 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:06.145 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:23:06.145 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:06.145 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:06.146 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:06.146 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:23:06.146 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:23:06.146 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:06.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:06.147 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:06.147 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:23:06.147 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:06.148 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:06.148 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:23:06.148 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:06.148 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:06.149 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:06.149 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:23:06.150 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:06.150 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:06.150 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:06.150 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:23:06.151 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:06.151 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:23:06.151 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:23:06.151 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:23:06.152 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:06.152 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:23:06Z" 2022-09-12T19:23:06.153 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:23:06.155 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:23:06Z" } 2022-09-12T19:23:06.156 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:23:06.157 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:23:06.158 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:23:06.159 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010319.572818_269.1 2022-09-12T19:23:11.161 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:23:11.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:23:11.163 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:23:11.259 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:23:11.259 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:23:11.296 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:23:11.293+0000 7f93588b1600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:23:11.297 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:23:11.294+0000 7f93588b1600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:23:11.312 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:23:11.312 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:23:11.312 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:23:11.312 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:23:11.313 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:23:11.313 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:23:11.313 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:23:11.314 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:11.314 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:23:11.314 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:11.314 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:23:11.315 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:11.315 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:11.315 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:23:11.315 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:11.316 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:23:11.316 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:11.316 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:23:11.316 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:11.317 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:11.317 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:11.317 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:11.317 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:23:11.318 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:11.318 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:11.318 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:23:11.318 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:11.319 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:11.319 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:11.319 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:23:11.320 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:11.320 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:11.320 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:11.320 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:11.321 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:23:11.321 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:11.321 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:11.321 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:23:11.322 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:11.322 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:11.322 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:11.322 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:23:11.323 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:23:11.323 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:11.323 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:11.323 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:11.324 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:23:11.324 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:11.324 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:11.325 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:23:11.326 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:11.327 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:11.327 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:11.328 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:23:11.328 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:11.329 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:11.329 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:11.329 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:23:11.330 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:11.330 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:23:11.331 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:23:11.331 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:23:11.331 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:11.332 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:23:11Z" 2022-09-12T19:23:11.332 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:23:11.335 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:23:11Z" } 2022-09-12T19:23:11.336 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:23:11.336 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:23:11.337 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:23:11.337 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010319.572818_269.1 2022-09-12T19:23:16.339 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:23:16.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:23:16.341 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:23:16.400 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:23:16.401 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:23:16.448 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:23:16.442+0000 7f31f4535600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:23:16.449 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:23:16.443+0000 7f31f4535600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:23:16.467 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:23:16.468 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:23:16.468 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:23:16.469 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:23:16.469 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:23:16.470 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:23:16.470 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:23:16.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:16.471 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:23:16.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:16.472 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:23:16.473 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:16.473 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:16.474 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:23:16.474 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:16.474 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:23:16.475 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:16.475 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:23:16.476 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:16.476 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:16.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:16.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:16.477 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:23:16.478 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:16.478 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:16.478 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:23:16.479 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:16.479 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:16.480 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:16.480 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:23:16.481 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:16.482 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:16.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:16.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:16.483 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:23:16.484 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:16.484 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:16.485 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:23:16.485 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:16.486 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:16.486 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:16.487 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:23:16.487 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:23:16.488 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:16.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:16.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:16.489 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:23:16.489 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:16.490 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:16.490 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:23:16.491 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:16.491 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:16.492 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:16.492 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:23:16.492 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:16.493 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:16.493 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:16.494 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:23:16.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:16.495 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:23:16.495 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:23:16.496 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:23:16.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:16.497 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:23:16Z" 2022-09-12T19:23:16.497 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:23:16.498 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:23:16Z" } 2022-09-12T19:23:16.499 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:23:16.500 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:23:16.501 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:23:16.502 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010319.572818_269.1 2022-09-12T19:23:21.504 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:23:21.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:23:21.505 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:23:21.563 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:23:21.564 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:23:21.605 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:23:21.602+0000 7f756de6a600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:23:21.606 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:23:21.603+0000 7f756de6a600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:23:21.623 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:23:21.624 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:23:21.624 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:23:21.625 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:23:21.625 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:23:21.626 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:23:21.626 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:23:21.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:21.627 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:23:21.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:21.627 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:23:21.628 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:21.628 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:21.628 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:23:21.628 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:21.629 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:23:21.629 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:21.629 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:23:21.630 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:21.630 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:21.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:21.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:21.631 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:23:21.631 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:21.631 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:21.632 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:23:21.632 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:21.632 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:21.633 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:21.633 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:23:21.633 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:21.634 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:21.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:21.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:21.635 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:23:21.635 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:21.636 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:21.636 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:23:21.636 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:21.636 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:21.637 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:21.637 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:23:21.637 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:23:21.637 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:21.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:21.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:21.638 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:23:21.638 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:21.639 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:21.639 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:23:21.639 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:21.639 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:21.640 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:21.640 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:23:21.640 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:21.641 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:21.642 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:21.642 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:23:21.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:21.643 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:23:21.644 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:23:21.644 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:23:21.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:21.645 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:23:21Z" 2022-09-12T19:23:21.646 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:23:21.648 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:23:21Z" } 2022-09-12T19:23:21.649 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:23:21.650 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:23:21.651 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:23:21.652 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010319.572818_269.1 2022-09-12T19:23:26.654 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:23:26.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:23:26.656 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:23:26.714 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:23:26.714 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:23:26.747 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:23:26.745+0000 7ff254357600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:23:26.748 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:23:26.746+0000 7ff254357600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:23:26.762 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:23:26.763 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:23:26.763 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:23:26.763 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:23:26.764 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:23:26.764 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:23:26.767 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:23:26.767 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:26.767 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:23:26.768 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:26.768 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:23:26.768 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:26.768 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:26.769 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:23:26.769 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:26.769 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:23:26.769 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:26.770 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:23:26.770 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:26.771 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:26.771 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:26.771 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:26.772 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:23:26.772 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:26.772 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:26.772 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:23:26.773 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:26.773 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:26.773 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:26.774 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:23:26.774 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:26.774 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:26.774 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:26.775 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:26.775 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:23:26.775 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:26.775 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:26.776 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:23:26.776 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:26.776 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:26.776 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:26.777 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:23:26.777 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:23:26.777 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:26.778 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:26.778 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:26.778 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:23:26.778 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:26.779 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:26.779 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:23:26.779 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:26.779 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:26.780 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:26.780 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:23:26.780 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:26.780 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:26.781 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:26.782 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:23:26.782 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:26.783 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:23:26.783 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:23:26.784 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:23:26.784 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:26.785 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:23:26Z" 2022-09-12T19:23:26.785 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:23:26.788 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:23:26Z" } 2022-09-12T19:23:26.789 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:23:26.790 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:23:26.791 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:23:26.792 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010319.572818_269.1 2022-09-12T19:23:31.794 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:23:31.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:23:31.796 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:23:31.855 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:23:31.855 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:23:31.903 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:23:31.901+0000 7fc998685600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:23:31.904 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:23:31.902+0000 7fc998685600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:23:31.921 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:23:31.921 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:23:31.922 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:23:31.922 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:23:31.923 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:23:31.923 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:23:31.923 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:23:31.924 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:31.924 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:23:31.924 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:31.925 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:23:31.925 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:31.926 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:31.926 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:23:31.926 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:31.927 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:23:31.927 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:31.928 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:23:31.928 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:31.928 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:31.929 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:31.929 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:31.930 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:23:31.930 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:31.930 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:31.931 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:23:31.931 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:31.931 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:31.932 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:31.932 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:23:31.932 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:31.933 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:31.933 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:31.934 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:31.934 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:23:31.934 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:31.935 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:31.935 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:23:31.935 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:31.936 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:31.936 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:31.937 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:23:31.937 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:23:31.937 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:31.938 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:31.938 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:31.938 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:23:31.939 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:31.939 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:31.939 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:23:31.940 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:31.940 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:31.941 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:31.941 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:23:31.941 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:31.942 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:31.942 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:31.942 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:23:31.943 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:31.943 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:23:31.943 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:23:31.944 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:23:31.944 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:31.945 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:23:31Z" 2022-09-12T19:23:31.945 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:23:31.947 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:23:31Z" } 2022-09-12T19:23:31.948 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:23:31.948 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:23:31.949 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:23:31.950 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010319.572818_269.1 2022-09-12T19:23:36.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:23:36.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:23:36.953 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:23:37.010 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:23:37.010 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:23:37.051 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:23:37.049+0000 7f8b1387f600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:23:37.053 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:23:37.050+0000 7f8b1387f600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:23:37.073 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:23:37.074 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:23:37.074 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:23:37.075 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:23:37.075 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:23:37.076 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:23:37.076 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:23:37.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:37.077 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:23:37.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:37.077 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:23:37.078 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:37.078 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:37.078 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:23:37.079 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:37.079 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:23:37.079 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:37.079 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:23:37.080 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:37.080 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:37.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:37.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:37.081 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:23:37.081 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:37.082 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:37.083 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:23:37.083 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:37.083 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:37.084 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:37.084 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:23:37.084 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:37.238 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:37.239 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:37.239 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:37.240 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:23:37.240 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:37.241 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:37.241 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:23:37.241 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:37.241 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:37.242 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:37.242 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:23:37.242 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:23:37.243 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:37.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:37.243 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:37.244 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:23:37.244 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:37.244 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:37.245 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:23:37.245 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:37.245 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:37.246 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:37.246 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:23:37.247 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:37.247 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:37.248 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:37.248 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:23:37.249 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:37.249 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:23:37.249 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:23:37.250 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:23:37.250 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:37.251 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:23:37Z" 2022-09-12T19:23:37.251 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:23:37.253 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:23:37Z" } 2022-09-12T19:23:37.254 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:23:37.255 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:23:37.256 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:23:37.257 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010319.572818_269.1 2022-09-12T19:23:42.259 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:23:42.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:23:42.261 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:23:42.318 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:23:42.318 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:23:42.352 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:23:42.349+0000 7fdf97cb3600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:23:42.353 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:23:42.350+0000 7fdf97cb3600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:23:42.367 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:23:42.367 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:23:42.368 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:23:42.368 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:23:42.369 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:23:42.369 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:23:42.370 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:23:42.371 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:42.371 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:23:42.372 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:42.372 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:23:42.373 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:42.373 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:42.374 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:23:42.374 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:42.375 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:23:42.375 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:42.376 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:23:42.377 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:42.377 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:42.378 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:42.378 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:42.379 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:23:42.380 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:42.380 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:42.381 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:23:42.381 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:42.382 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:42.382 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:42.383 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:23:42.383 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:42.384 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:42.384 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:42.385 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:42.385 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:23:42.386 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:42.387 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:42.387 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:23:42.387 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:42.388 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:42.388 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:42.388 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:23:42.388 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:23:42.389 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:42.389 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:42.389 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:42.389 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:23:42.390 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:42.390 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:42.390 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:23:42.390 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:42.391 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:42.391 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:42.391 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:23:42.391 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:42.392 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:42.392 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:42.392 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:23:42.392 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:42.393 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:23:42.393 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:23:42.393 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:23:42.393 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:42.394 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:23:42Z" 2022-09-12T19:23:42.394 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:23:42.395 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:23:42Z" } 2022-09-12T19:23:42.396 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:23:42.397 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:23:42.397 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:23:42.398 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010319.572818_269.1 2022-09-12T19:23:47.400 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:23:47.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:23:47.402 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:23:47.461 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:23:47.461 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:23:47.507 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:23:47.505+0000 7ff7f7db2600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:23:47.508 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:23:47.506+0000 7ff7f7db2600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:23:47.523 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:23:47.523 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:23:47.524 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:23:47.524 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:23:47.525 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:23:47.525 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:23:47.526 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:23:47.526 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:47.526 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:23:47.527 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:47.527 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:23:47.528 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:47.528 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:47.530 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:23:47.530 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:47.531 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:23:47.531 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:47.532 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:23:47.532 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:47.532 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:47.532 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:47.533 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:47.533 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:23:47.533 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:47.533 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:47.534 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:23:47.534 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:47.535 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:47.535 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:47.536 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:23:47.536 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:47.536 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:47.537 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:47.537 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:47.537 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:23:47.537 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:47.538 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:47.539 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:23:47.539 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:47.540 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:47.540 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:47.540 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:23:47.541 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:23:47.541 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:47.542 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:47.542 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:47.543 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:23:47.543 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:47.544 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:47.544 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:23:47.544 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:47.545 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:47.545 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:47.546 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:23:47.546 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:47.547 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:47.547 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:47.548 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:23:47.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:47.549 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:23:47.549 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:23:47.550 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:23:47.550 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:47.551 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:23:47Z" 2022-09-12T19:23:47.551 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:23:47.553 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:23:47Z" } 2022-09-12T19:23:47.554 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:23:47.555 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:23:47.555 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:23:47.556 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010319.572818_269.1 2022-09-12T19:23:52.558 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:23:52.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:23:52.559 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:23:52.616 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:23:52.616 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:23:52.657 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:23:52.654+0000 7ff40f233600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:23:52.658 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:23:52.655+0000 7ff40f233600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:23:52.671 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:23:52.672 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:23:52.672 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:23:52.672 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:23:52.672 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:23:52.673 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:23:52.673 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:23:52.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:52.673 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:23:52.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:52.674 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:23:52.674 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:52.675 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:52.675 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:23:52.675 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:52.676 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:23:52.676 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:52.676 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:23:52.677 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:52.677 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:52.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:52.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:52.678 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:23:52.679 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:52.679 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:52.679 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:23:52.680 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:52.680 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:52.681 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:52.681 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:23:52.681 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:52.682 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:52.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:52.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:52.683 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:23:52.683 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:52.684 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:52.684 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:23:52.685 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:52.685 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:52.685 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:52.686 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:23:52.686 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:23:52.687 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:52.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:52.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:52.688 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:23:52.688 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:52.689 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:52.689 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:23:52.689 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:52.690 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:52.690 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:52.690 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:23:52.690 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:52.691 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:52.691 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:52.691 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:23:52.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:52.692 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:23:52.692 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:23:52.692 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:23:52.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:52.693 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:23:52Z" 2022-09-12T19:23:52.693 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:23:52.695 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:23:52Z" } 2022-09-12T19:23:52.695 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:23:52.696 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:23:52.696 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:23:52.697 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010319.572818_269.1 2022-09-12T19:23:57.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:23:57.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:23:57.700 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:23:57.758 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:23:57.759 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:23:57.790 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:23:57.788+0000 7f81818ad600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:23:57.792 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:23:57.789+0000 7f81818ad600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:23:57.810 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:23:57.810 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:23:57.811 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:23:57.811 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:23:57.811 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:23:57.812 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:23:57.812 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:23:57.812 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:57.812 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:23:57.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:57.813 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:23:57.813 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:57.813 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:57.814 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:23:57.815 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:57.816 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:23:57.816 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:57.817 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:23:57.817 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:57.818 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:57.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:57.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:57.820 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:23:57.820 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:57.821 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:57.821 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:23:57.822 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:57.822 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:57.822 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:57.823 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:23:57.823 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:57.824 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:57.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:57.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:57.826 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:23:57.827 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:57.827 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:57.828 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:23:57.828 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:57.829 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:57.830 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:57.830 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:23:57.831 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:23:57.831 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:57.832 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:57.833 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:23:57.833 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:23:57.834 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:23:57.834 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:23:57.835 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:23:57.835 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:23:57.836 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:23:57.837 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:23:57.837 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:23:57.838 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:23:57.838 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:57.839 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:23:57.840 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:23:57.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:57.841 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:23:57.841 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:23:57.842 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:23:57.842 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:23:57.843 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:23:57Z" 2022-09-12T19:23:57.844 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:23:57.846 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:23:57Z" } 2022-09-12T19:23:57.848 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:23:57.849 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:23:57.850 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:23:57.851 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010319.572818_269.1 2022-09-12T19:24:02.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:24:02.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:24:02.854 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:24:02.912 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:24:02.913 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:24:02.945 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:24:02.942+0000 7f25d002d600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:24:02.945 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:24:02.943+0000 7f25d002d600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:24:02.959 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:24:02.960 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:24:02.960 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:24:02.961 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:24:02.961 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:24:02.961 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:24:02.962 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:24:02.962 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:02.962 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:24:02.962 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:02.963 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:24:02.963 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:02.964 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:02.964 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:24:02.964 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:02.964 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:24:02.965 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:02.965 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:24:02.966 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:02.966 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:02.966 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:02.967 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:02.967 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:24:02.968 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:02.968 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:02.969 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:24:02.969 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:02.969 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:02.970 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:02.970 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:24:02.971 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:02.972 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:02.972 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:02.972 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:02.973 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:24:02.974 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:02.974 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:02.974 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:24:02.976 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:02.976 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:02.976 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:02.977 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:24:02.978 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:24:02.978 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:02.978 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:02.979 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:02.980 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:24:02.980 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:02.981 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:02.981 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:24:02.982 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:02.982 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:02.983 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:02.983 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:24:02.983 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:02.984 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:02.984 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:02.985 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:24:02.985 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:02.985 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:24:02.986 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:24:02.986 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:24:02.987 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:02.987 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:24:02Z" 2022-09-12T19:24:02.988 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:24:02.989 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:24:02Z" } 2022-09-12T19:24:02.991 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:24:02.992 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:24:02.993 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:24:02.994 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010319.572818_269.1 2022-09-12T19:24:07.995 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:24:07.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:24:07.997 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:24:08.054 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:24:08.054 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:24:08.093 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:24:08.091+0000 7f9e8464e600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:24:08.094 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:24:08.092+0000 7f9e8464e600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:24:08.110 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:24:08.111 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:24:08.111 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:24:08.111 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:24:08.112 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:24:08.112 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:24:08.112 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:24:08.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:08.113 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:24:08.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:08.114 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:24:08.114 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:08.115 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:08.115 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:24:08.115 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:08.116 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:24:08.116 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:08.116 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:24:08.117 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:08.117 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:08.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:08.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:08.118 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:24:08.118 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:08.118 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:08.119 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:24:08.119 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:08.119 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:08.119 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:08.120 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:24:08.120 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:08.120 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:08.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:08.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:08.121 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:24:08.121 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:08.122 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:08.122 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:24:08.122 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:08.123 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:08.123 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:08.123 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:24:08.124 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:24:08.124 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:08.124 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:08.125 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:08.125 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:24:08.125 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:08.126 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:08.126 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:24:08.126 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:08.127 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:08.127 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:08.127 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:24:08.127 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:08.128 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:08.128 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:08.128 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:24:08.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:08.129 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:24:08.129 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:24:08.129 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:24:08.130 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:08.130 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:24:08Z" 2022-09-12T19:24:08.130 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:24:08.132 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:24:08Z" } 2022-09-12T19:24:08.132 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:24:08.133 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:24:08.134 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:24:08.134 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010319.572818_269.1 2022-09-12T19:24:13.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:24:13.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:24:13.141 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:24:13.200 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:24:13.200 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:24:13.241 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:24:13.239+0000 7f4115dbc600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:24:13.242 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:24:13.240+0000 7f4115dbc600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:24:13.259 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:24:13.259 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:24:13.260 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:24:13.260 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:24:13.260 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:24:13.260 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:24:13.261 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:24:13.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:13.261 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:24:13.261 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:13.262 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:24:13.262 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:13.263 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:13.263 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:24:13.263 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:13.264 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:24:13.264 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:13.265 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:24:13.265 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:13.265 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:13.265 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:13.266 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:13.266 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:24:13.266 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:13.266 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:13.267 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:24:13.267 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:13.267 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:13.267 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:13.268 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:24:13.268 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:13.268 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:13.268 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:13.268 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:13.269 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:24:13.269 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:13.269 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:13.269 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:24:13.270 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:13.270 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:13.270 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:13.271 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:24:13.271 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:24:13.271 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:13.271 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:13.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:13.272 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:24:13.272 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:13.272 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:13.273 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:24:13.273 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:13.273 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:13.274 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:13.274 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:24:13.274 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:13.274 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:13.275 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:13.275 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:24:13.275 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:13.275 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:24:13.276 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:24:13.276 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:24:13.276 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:13.276 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:24:13Z" 2022-09-12T19:24:13.277 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:24:13.278 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:24:13Z" } 2022-09-12T19:24:13.279 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:24:13.279 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:24:13.280 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:24:13.280 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010319.572818_269.1 2022-09-12T19:24:18.282 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:24:18.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:24:18.284 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:24:18.341 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:24:18.342 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:24:18.372 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:24:18.369+0000 7f61258a2600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:24:18.373 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:24:18.370+0000 7f61258a2600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:24:18.388 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:24:18.388 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:24:18.388 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:24:18.389 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:24:18.389 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:24:18.389 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:24:18.390 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:24:18.390 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:18.390 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:24:18.390 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:18.391 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:24:18.391 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:18.391 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:18.391 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:24:18.392 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:18.392 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:24:18.392 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:18.392 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:24:18.393 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:18.393 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:18.394 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:18.394 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:18.394 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:24:18.394 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:18.395 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:18.395 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:24:18.395 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:18.396 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:18.396 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:18.396 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:24:18.396 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:18.397 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:18.397 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:18.397 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:18.397 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:24:18.398 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:18.398 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:18.398 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:24:18.398 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:18.399 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:18.399 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:18.399 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:24:18.399 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:24:18.400 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:18.400 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:18.400 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:18.401 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:24:18.401 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:18.401 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:18.402 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:24:18.402 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:18.402 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:18.402 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:18.403 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:24:18.403 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:18.403 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:18.404 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:18.404 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:24:18.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:18.405 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:24:18.405 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:24:18.405 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:24:18.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:18.406 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:24:18Z" 2022-09-12T19:24:18.406 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:24:18.407 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:24:18Z" } 2022-09-12T19:24:18.408 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:24:18.409 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:24:18.409 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:24:18.410 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010319.572818_269.1 2022-09-12T19:24:23.411 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:24:23.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:24:23.413 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:24:23.469 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:24:23.470 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:24:23.520 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:24:23.517+0000 7f3b50e9b600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:24:23.520 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:24:23.518+0000 7f3b50e9b600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:24:23.539 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:24:23.540 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:24:23.540 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:24:23.540 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:24:23.541 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:24:23.541 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:24:23.542 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:24:23.542 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:23.543 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:24:23.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:23.544 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:24:23.544 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:23.545 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:23.545 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:24:23.545 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:23.546 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:24:23.546 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:23.547 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:24:23.547 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:23.548 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:23.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:23.549 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:23.549 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:24:23.549 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:23.550 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:23.551 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:24:23.551 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:23.552 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:23.552 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:23.552 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:24:23.553 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:23.553 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:23.554 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:23.554 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:23.555 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:24:23.555 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:23.556 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:23.556 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:24:23.557 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:23.557 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:23.558 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:23.558 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:24:23.559 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:24:23.559 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:23.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:23.560 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:23.560 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:24:23.560 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:23.560 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:23.561 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:24:23.561 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:23.561 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:23.561 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:23.562 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:24:23.562 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:23.562 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:23.562 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:23.563 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:24:23.563 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:23.563 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:24:23.564 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:24:23.564 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:24:23.564 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:23.564 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:24:23Z" 2022-09-12T19:24:23.564 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:24:23.566 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:24:23Z" } 2022-09-12T19:24:23.567 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:24:23.567 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:24:23.568 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:24:23.568 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010319.572818_269.1 2022-09-12T19:24:28.569 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:24:28.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:24:28.572 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:24:28.630 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:24:28.631 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:24:28.668 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:24:28.666+0000 7f66df3eb600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:24:28.669 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:24:28.667+0000 7f66df3eb600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:24:28.688 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:24:28.688 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:24:28.689 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:24:28.689 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:24:28.689 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:24:28.689 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:24:28.690 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:24:28.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:28.690 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:24:28.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:28.691 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:24:28.691 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:28.691 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:28.691 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:24:28.692 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:28.692 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:24:28.692 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:28.692 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:24:28.693 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:28.693 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:28.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:28.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:28.694 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:24:28.694 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:28.694 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:28.694 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:24:28.695 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:28.695 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:28.695 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:28.695 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:24:28.696 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:28.696 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:28.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:28.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:28.697 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:24:28.697 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:28.697 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:28.697 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:24:28.698 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:28.698 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:28.699 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:28.699 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:24:28.699 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:24:28.699 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:28.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:28.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:28.700 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:24:28.700 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:28.701 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:28.701 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:24:28.701 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:28.701 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:28.702 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:28.702 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:24:28.702 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:28.702 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:28.703 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:28.703 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:24:28.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:28.703 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:24:28.704 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:24:28.704 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:24:28.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:28.704 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:24:28Z" 2022-09-12T19:24:28.705 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:24:28.706 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:24:28Z" } 2022-09-12T19:24:28.707 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:24:28.707 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:24:28.708 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:24:28.708 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010319.572818_269.1 2022-09-12T19:24:33.710 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:24:33.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:24:33.711 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:24:33.768 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:24:33.768 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:24:33.812 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:24:33.809+0000 7f21ead52600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:24:33.813 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:24:33.810+0000 7f21ead52600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:24:33.826 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:24:33.827 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:24:33.827 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:24:33.827 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:24:33.827 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:24:33.828 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:24:33.828 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:24:33.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:33.829 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:24:33.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:33.829 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:24:33.829 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:34.183 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:34.183 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:24:34.183 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:34.184 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:24:34.184 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:34.184 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:24:34.185 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:34.185 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:34.185 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:34.186 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:34.186 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:24:34.186 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:34.187 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:34.187 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:24:34.187 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:34.188 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:34.188 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:34.188 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:24:34.188 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:34.189 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:34.189 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:34.189 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:34.189 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:24:34.190 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:34.190 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:34.190 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:24:34.191 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:34.191 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:34.191 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:34.191 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:24:34.192 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:24:34.192 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:34.192 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:34.193 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:34.193 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:24:34.193 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:34.193 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:34.194 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:24:34.194 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:34.195 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:34.195 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:34.195 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:24:34.196 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:34.196 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:34.196 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:34.197 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:24:34.197 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:34.197 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:24:34.197 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:24:34.198 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:24:34.198 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:34.199 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:24:33Z" 2022-09-12T19:24:34.199 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:24:34.201 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:24:33Z" } 2022-09-12T19:24:34.202 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:24:34.202 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:24:34.203 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:24:34.203 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010319.572818_269.1 2022-09-12T19:24:39.204 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:24:39.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:24:39.206 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:24:39.265 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:24:39.266 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:24:39.300 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:24:39.297+0000 7fe98dea9600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:24:39.300 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:24:39.298+0000 7fe98dea9600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:24:39.318 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:24:39.319 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:24:39.319 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:24:39.320 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:24:39.320 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:24:39.320 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:24:39.321 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:24:39.321 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:39.321 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:24:39.322 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:39.322 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:24:39.323 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:39.323 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:39.323 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:24:39.324 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:39.324 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:24:39.325 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:39.325 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:24:39.325 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:39.326 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:39.326 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:39.326 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:39.327 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:24:39.327 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:39.327 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:39.327 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:24:39.328 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:39.328 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:39.328 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:39.329 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:24:39.329 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:39.329 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:39.329 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:39.330 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:39.330 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:24:39.330 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:39.330 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:39.331 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:24:39.331 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:39.331 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:39.331 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:39.332 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:24:39.332 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:24:39.332 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:39.332 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:39.333 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:39.333 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:24:39.333 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:39.333 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:39.334 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:24:39.334 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:39.334 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:39.335 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:39.335 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:24:39.336 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:39.336 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:39.336 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:39.337 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:24:39.337 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:39.338 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:24:39.338 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:24:39.338 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:24:39.338 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:39.339 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:24:39Z" 2022-09-12T19:24:39.339 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:24:39.340 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:24:39Z" } 2022-09-12T19:24:39.341 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:24:39.342 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:24:39.342 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:24:39.343 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010319.572818_269.1 2022-09-12T19:24:44.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:24:44.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:24:44.347 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:24:44.405 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:24:44.405 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:24:44.448 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:24:44.445+0000 7f1648f17600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:24:44.449 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:24:44.447+0000 7f1648f17600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:24:44.467 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:24:44.468 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:24:44.469 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:24:44.469 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:24:44.470 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:24:44.470 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:24:44.471 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:24:44.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:44.472 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:24:44.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:44.473 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:24:44.474 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:44.474 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:44.475 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:24:44.475 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:44.476 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:24:44.476 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:44.477 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:24:44.477 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:44.478 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:44.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:44.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:44.480 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:24:44.481 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:44.481 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:44.482 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:24:44.482 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:44.483 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:44.484 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:44.484 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:24:44.485 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:44.485 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:44.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:44.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:44.486 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:24:44.486 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:44.486 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:44.487 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:24:44.487 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:44.487 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:44.488 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:44.488 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:24:44.488 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:24:44.489 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:44.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:44.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:44.490 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:24:44.490 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:44.490 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:44.491 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:24:44.491 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:44.491 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:44.492 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:44.492 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:24:44.492 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:44.492 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:44.493 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:44.493 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:24:44.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:44.493 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:24:44.494 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:24:44.494 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:24:44.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:44.494 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:24:44Z" 2022-09-12T19:24:44.495 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:24:44.497 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:24:44Z" } 2022-09-12T19:24:44.498 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:24:44.498 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:24:44.499 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:24:44.500 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010319.572818_269.1 2022-09-12T19:24:49.507 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:24:49.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:24:49.510 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:24:49.568 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:24:49.568 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:24:49.609 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:24:49.606+0000 7fc8a58c7600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:24:49.610 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:24:49.607+0000 7fc8a58c7600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:24:49.627 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:24:49.627 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:24:49.628 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:24:49.628 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:24:49.628 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:24:49.629 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:24:49.629 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:24:49.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:49.630 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:24:49.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:49.631 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:24:49.631 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:49.631 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:49.632 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:24:49.632 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:49.632 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:24:49.632 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:49.633 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:24:49.633 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:49.633 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:49.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:49.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:49.634 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:24:49.635 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:49.636 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:49.636 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:24:49.637 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:49.637 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:49.637 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:49.637 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:24:49.638 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:49.638 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:49.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:49.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:49.639 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:24:49.639 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:49.639 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:49.640 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010319.572818_269.1", 2022-09-12T19:24:49.640 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:49.641 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:49.641 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:49.641 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:18:39.572818Z", 2022-09-12T19:24:49.642 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:24:49.642 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:49.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:49.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:49.643 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:24:49.643 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:49.643 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:49.643 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:24:49.644 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:49.644 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:49.644 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:49.644 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:24:49.645 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:49.645 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:49.645 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:49.645 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:24:49.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:49.646 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:24:49.646 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:24:49.646 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:24:49.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:49.647 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:24:49Z" 2022-09-12T19:24:49.647 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:24:49.649 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010319.572818_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:18:39.572818Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:24:49Z" } 2022-09-12T19:24:49.649 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010319.572818_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:18:39.572818Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:24:49.650 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:24:49.650 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010319.572818_269.1', 3: ''} 2022-09-12T19:24:49.651 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010319.572818_269.1 2022-09-12T19:24:54.652 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:24:54.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:24:54.653 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:24:54.709 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:24:54.709 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:24:54.742 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:24:54.740+0000 7ffb56633600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:24:54.743 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:24:54.741+0000 7ffb56633600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:24:54.756 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:24:54.757 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:24:54.757 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:24:54.758 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:24:54.758 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:24:54.759 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:24:54.759 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:24:54.760 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:54.760 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:24:54.761 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:54.761 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:24:54.762 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:54.762 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:54.763 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:24:54.763 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:54.764 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:24:54.764 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:54.765 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:24:54.766 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:54.766 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:54.767 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:54.768 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:54.768 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:24:54.769 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:54.769 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:54.770 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:24:54.770 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:54.771 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:54.771 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:54.772 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:24:54.772 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:54.773 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:54.773 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:54.774 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:54.774 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:24:54.775 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:54.775 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:54.776 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010572.551013_355.1", 2022-09-12T19:24:54.776 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:54.777 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:54.777 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:54.778 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:22:52.551013Z", 2022-09-12T19:24:54.778 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:24:54.779 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:54.779 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:54.780 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:54.780 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:24:54.781 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:54.781 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:54.782 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:24:54.782 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:54.783 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:54.783 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:54.784 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:24:54.784 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:54.785 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:54.785 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:54.786 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:24:54.786 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:54.787 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:24:54.787 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:24:54.788 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:24:54.788 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:54.789 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:24:54Z" 2022-09-12T19:24:54.789 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:24:54.792 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010572.551013_355.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:22:52.551013Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:24:54Z" } 2022-09-12T19:24:54.793 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010572.551013_355.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:22:52.551013Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:24:54.794 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:24:54.795 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010572.551013_355.1', 3: ''} 2022-09-12T19:24:54.796 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010572.551013_355.1 2022-09-12T19:24:59.798 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:24:59.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:24:59.800 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:24:59.854 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:24:59.855 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:24:59.889 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:24:59.887+0000 7f613c5b1600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:24:59.890 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:24:59.888+0000 7f613c5b1600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:24:59.908 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:24:59.909 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:24:59.909 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:24:59.909 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:24:59.910 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:24:59.910 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:24:59.910 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:24:59.911 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:59.911 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:24:59.912 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:59.912 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:24:59.913 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:59.913 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:59.914 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:24:59.914 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:59.915 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:24:59.916 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:59.916 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:24:59.917 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:59.918 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:59.918 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:59.919 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:59.920 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:24:59.920 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:59.921 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:59.921 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:24:59.922 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:59.922 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:59.923 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:59.923 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:24:59.924 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:59.925 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:59.925 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:59.926 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:59.926 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:24:59.927 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:59.927 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:59.928 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010572.541800_354.1", 2022-09-12T19:24:59.928 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:59.929 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:59.929 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:59.930 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:22:52.541800Z", 2022-09-12T19:24:59.930 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:24:59.931 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:59.932 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:59.932 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:24:59.933 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:24:59.933 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:24:59.934 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:24:59.935 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:24:59.935 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:24:59.936 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:24:59.936 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:24:59.936 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:24:59.937 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:24:59.937 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:59.937 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:24:59.937 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:24:59.938 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:59.938 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:24:59.938 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:24:59.939 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:24:59.939 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:24:59.939 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:24:59Z" 2022-09-12T19:24:59.939 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:24:59.941 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010572.541800_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:22:52.541800Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:24:59Z" } 2022-09-12T19:24:59.942 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010572.541800_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:22:52.541800Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:24:59.944 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:24:59.945 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010572.541800_354.1', 3: ''} 2022-09-12T19:24:59.945 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010572.541800_354.1 2022-09-12T19:25:04.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:25:04.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:25:04.949 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:25:05.010 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:25:05.011 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:25:05.106 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:25:05.103+0000 7ffb2deb6600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:25:05.107 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:25:05.104+0000 7ffb2deb6600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:25:05.120 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:25:05.121 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:25:05.121 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:25:05.121 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:25:05.122 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:25:05.122 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:25:05.123 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:25:05.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:05.124 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:25:05.124 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:05.124 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:25:05.125 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:05.125 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:05.126 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:25:05.126 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:05.127 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:25:05.127 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:05.128 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:25:05.128 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:05.128 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:05.129 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:05.129 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:05.130 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:25:05.130 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:05.131 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:05.131 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:25:05.132 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:05.132 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:05.132 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:05.133 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:25:05.133 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:05.134 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:05.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:05.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:05.135 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:25:05.135 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:05.136 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:05.136 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010573.094162_367.1", 2022-09-12T19:25:05.137 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:05.137 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:05.138 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:05.138 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:22:53.094162Z", 2022-09-12T19:25:05.139 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:25:05.139 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:05.140 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:05.140 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:05.140 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:25:05.141 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:05.141 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:05.142 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:25:05.142 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:05.143 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:05.143 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:05.143 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:25:05.144 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:05.144 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:05.144 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:05.145 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:25:05.145 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:05.146 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:25:05.146 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:25:05.146 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:25:05.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:05.147 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:25:05Z" 2022-09-12T19:25:05.147 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:25:05.149 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010573.094162_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:22:53.094162Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:25:05Z" } 2022-09-12T19:25:05.150 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010573.094162_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:22:53.094162Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:25:05.151 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:25:05.152 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010573.094162_367.1', 3: ''} 2022-09-12T19:25:05.152 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010573.094162_367.1 2022-09-12T19:25:10.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:25:10.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:25:10.156 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:25:10.206 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:25:10.208 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:25:10.245 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:25:10.242+0000 7fdc43495600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:25:10.247 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:25:10.243+0000 7fdc43495600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:25:10.264 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:25:10.265 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:25:10.265 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:25:10.266 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:25:10.266 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:25:10.267 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:25:10.267 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:25:10.267 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:10.267 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:25:10.268 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:10.268 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:25:10.269 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:10.269 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:10.269 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:25:10.269 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:10.270 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:25:10.270 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:10.271 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:25:10.271 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:10.272 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:10.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:10.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:10.273 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:25:10.273 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:10.273 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:10.273 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:25:10.274 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:10.274 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:10.274 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:10.274 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:25:10.275 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:10.275 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:10.275 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:10.275 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:10.276 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:25:10.276 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:10.276 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:10.276 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010573.101538_369.1", 2022-09-12T19:25:10.277 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:10.277 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:10.277 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:10.277 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:22:53.101538Z", 2022-09-12T19:25:10.278 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:25:10.278 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:10.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:10.278 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:10.279 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:25:10.279 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:10.280 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:10.281 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:25:10.282 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:10.283 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:10.283 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:10.283 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:25:10.284 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:10.284 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:10.284 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:10.284 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:25:10.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:10.285 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:25:10.285 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:25:10.285 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:25:10.286 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:10.286 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:25:10Z" 2022-09-12T19:25:10.286 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:25:10.287 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010573.101538_369.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:22:53.101538Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:25:10Z" } 2022-09-12T19:25:10.287 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010573.101538_369.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:22:53.101538Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:25:10.288 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:25:10.288 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010573.101538_369.1', 3: ''} 2022-09-12T19:25:10.289 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010573.101538_369.1 2022-09-12T19:25:15.290 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:25:15.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:25:15.292 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:25:15.342 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:25:15.343 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:25:15.373 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:25:15.371+0000 7f7640e86600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:25:15.374 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:25:15.372+0000 7f7640e86600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:25:15.387 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:25:15.388 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:25:15.388 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:25:15.388 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:25:15.389 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:25:15.389 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:25:15.389 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:25:15.389 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:15.390 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:25:15.390 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:15.390 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:25:15.391 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:15.391 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:15.391 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:25:15.392 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:15.392 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:25:15.392 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:15.392 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:25:15.393 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:15.393 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:15.393 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:15.394 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:15.394 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:25:15.394 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:15.394 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:15.395 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:25:15.395 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:15.395 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:15.396 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:15.396 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:25:15.396 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:15.396 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:15.397 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:15.397 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:15.397 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:25:15.398 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:15.398 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:15.398 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010572.555375_356.1", 2022-09-12T19:25:15.398 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:15.399 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:15.399 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:15.399 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:22:52.555375Z", 2022-09-12T19:25:15.399 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:25:15.400 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:15.400 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:15.400 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:15.401 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:25:15.401 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:15.401 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:15.402 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:25:15.402 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:15.403 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:15.403 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:15.403 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:25:15.404 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:15.404 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:15.405 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:15.405 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:25:15.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:15.406 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:25:15.406 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:25:15.407 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:25:15.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:15.407 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:25:15Z" 2022-09-12T19:25:15.407 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:25:15.409 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010572.555375_356.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:22:52.555375Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:25:15Z" } 2022-09-12T19:25:15.409 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010572.555375_356.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:22:52.555375Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:25:15.410 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:25:15.411 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010572.555375_356.1', 3: ''} 2022-09-12T19:25:15.412 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010572.555375_356.1 2022-09-12T19:25:20.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:25:20.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:25:20.416 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:25:20.468 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:25:20.468 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:25:20.514 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:25:20.512+0000 7f4d5e722600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:25:20.519 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:25:20.517+0000 7f4d5e722600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:25:20.582 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:25:20.582 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:25:20.582 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:25:20.583 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:25:20.583 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:25:20.583 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:25:20.583 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:25:20.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:20.584 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:25:20.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:20.585 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:25:20.585 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:20.586 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:20.586 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:25:20.587 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:20.587 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:25:20.588 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:20.588 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:25:20.588 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:20.589 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:20.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:20.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:20.590 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:25:20.590 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:20.591 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:20.591 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:25:20.592 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:20.592 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:20.593 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:20.593 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:25:20.594 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:20.594 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:20.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:20.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:20.596 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:25:20.596 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:20.597 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:20.597 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010573.160868_371.1", 2022-09-12T19:25:20.598 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:20.598 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:20.599 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:20.599 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:22:53.160868Z", 2022-09-12T19:25:20.600 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:25:20.600 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:20.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:20.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:20.602 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:25:20.602 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:20.603 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:20.603 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:25:20.604 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:20.604 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:20.604 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:20.605 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:25:20.605 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:20.605 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:20.606 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:20.606 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:25:20.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:20.606 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:25:20.607 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:25:20.607 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:25:20.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:20.607 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:25:20Z" 2022-09-12T19:25:20.608 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:25:20.608 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010573.160868_371.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:22:53.160868Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:25:20Z" } 2022-09-12T19:25:20.609 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010573.160868_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:22:53.160868Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:25:20.610 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:25:20.610 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010573.160868_371.1', 3: ''} 2022-09-12T19:25:20.611 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010573.160868_371.1 2022-09-12T19:25:25.612 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:25:25.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:25:25.614 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:25:25.664 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:25:25.665 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:25:25.708 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:25:25.705+0000 7f9e148ee600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:25:25.709 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:25:25.706+0000 7f9e148ee600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:25:25.723 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:25:25.724 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:25:25.724 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:25:25.724 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:25:25.725 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:25:25.725 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:25:25.725 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:25:25.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:25.726 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:25:25.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:25.726 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:25:25.726 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:25.727 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:25.727 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:25:25.727 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:25.727 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:25:25.728 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:25.728 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:25:25.728 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:25.729 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:25.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:25.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:25.729 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:25:25.730 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:25.730 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:25.730 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:25:25.730 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:25.731 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:25.731 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:25.731 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:25:25.732 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:25.732 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:25.732 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:25.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:25.733 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:25:25.733 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:25.733 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:25.733 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010573.219875_376.1", 2022-09-12T19:25:25.734 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:25.734 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:25.734 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:25.735 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:22:53.219875Z", 2022-09-12T19:25:25.735 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:25:25.735 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:25.735 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:25.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:25.736 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:25:25.736 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:25.736 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:25.737 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:25:25.737 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:25.737 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:25.737 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:25.738 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:25:25.738 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:25.738 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:25.739 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:25.739 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:25:25.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:25.739 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:25:25.740 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:25:25.740 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:25:25.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:25.741 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:25:25Z" 2022-09-12T19:25:25.741 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:25:25.742 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010573.219875_376.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:22:53.219875Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:25:25Z" } 2022-09-12T19:25:25.743 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010573.219875_376.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:22:53.219875Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:25:25.743 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:25:25.744 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010573.219875_376.1', 3: ''} 2022-09-12T19:25:25.745 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010573.219875_376.1 2022-09-12T19:25:30.746 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:25:30.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:25:30.748 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:25:30.798 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:25:30.799 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:25:30.836 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:25:30.833+0000 7f68c3b5a600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:25:30.838 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:25:30.834+0000 7f68c3b5a600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:25:30.854 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:25:30.854 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:25:30.854 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:25:30.855 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:25:30.855 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:25:30.855 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:25:30.855 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:25:30.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:30.856 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:25:30.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:30.857 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:25:30.857 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:30.857 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:30.857 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:25:30.858 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:30.858 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:25:30.858 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:30.858 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:25:30.859 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:30.859 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:30.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:30.859 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:30.860 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:25:30.860 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:30.860 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:30.860 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:25:30.861 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:30.861 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:30.861 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:30.862 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:25:30.862 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:30.862 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:30.862 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:30.863 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:30.863 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:25:30.863 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:30.864 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:30.864 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010573.270088_383.1", 2022-09-12T19:25:30.864 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:30.864 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:30.865 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:30.865 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:22:53.270088Z", 2022-09-12T19:25:30.865 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:25:30.866 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:30.866 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:30.866 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:30.866 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:25:30.867 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:30.867 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:30.867 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:25:30.868 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:30.868 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:30.868 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:30.868 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:25:30.869 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:30.869 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:30.869 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:30.869 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:25:30.870 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:30.870 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:25:30.870 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:25:30.870 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:25:30.871 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:30.871 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:25:30Z" 2022-09-12T19:25:30.871 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:25:30.873 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010573.270088_383.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:22:53.270088Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:25:30Z" } 2022-09-12T19:25:30.873 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010573.270088_383.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:22:53.270088Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:25:30.874 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:25:30.875 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010573.270088_383.1', 3: ''} 2022-09-12T19:25:30.875 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010573.270088_383.1 2022-09-12T19:25:35.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:25:35.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:25:35.878 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:25:35.934 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:25:35.935 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:25:35.962 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:25:35.960+0000 7f64b76a6600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:25:35.963 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:25:35.961+0000 7f64b76a6600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:25:35.981 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:25:35.982 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:25:35.982 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:25:35.982 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:25:35.983 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:25:35.983 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:25:35.983 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:25:35.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:35.984 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:25:35.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:35.984 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:25:35.985 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:35.985 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:35.985 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:25:35.985 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:35.986 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:25:35.986 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:35.986 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:25:35.986 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:35.987 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:35.987 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:35.987 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:35.987 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:25:35.988 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:35.988 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:35.988 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:25:35.988 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:35.989 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:35.989 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:35.989 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:25:35.989 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:35.990 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:35.990 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:35.990 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:35.990 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:25:35.991 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:35.991 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:35.991 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010573.094162_367.1", 2022-09-12T19:25:35.991 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:35.992 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:35.992 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:35.992 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:22:53.094162Z", 2022-09-12T19:25:35.992 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:25:35.993 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:35.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:35.993 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:35.993 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:25:35.994 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:35.994 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:35.994 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:25:35.994 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:35.995 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:35.995 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:35.995 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:25:35.995 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:35.996 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:35.996 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:35.996 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:25:35.996 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:35.997 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:25:35.997 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:25:35.997 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:25:35.997 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:35.997 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:25:35Z" 2022-09-12T19:25:35.998 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:25:35.999 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010573.094162_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:22:53.094162Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:25:35Z" } 2022-09-12T19:25:36.000 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010573.094162_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:22:53.094162Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:25:36.000 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:25:36.001 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010573.094162_367.1', 3: ''} 2022-09-12T19:25:36.001 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010573.094162_367.1 2022-09-12T19:25:41.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:25:41.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:25:41.006 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:25:41.058 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:25:41.058 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:25:41.089 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:25:41.087+0000 7f7a049e2600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:25:41.090 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:25:41.088+0000 7f7a049e2600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:25:41.104 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:25:41.104 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:25:41.104 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:25:41.105 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:25:41.105 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:25:41.105 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:25:41.106 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:25:41.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:41.106 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:25:41.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:41.107 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:25:41.107 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:41.107 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:41.107 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:25:41.108 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:41.108 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:25:41.108 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:41.108 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:25:41.109 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:41.109 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:41.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:41.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:41.110 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:25:41.110 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:41.111 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:41.111 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:25:41.111 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:41.112 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:41.112 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:41.112 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:25:41.112 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:41.113 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:41.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:41.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:41.113 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:25:41.114 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:41.114 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:41.114 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010572.555375_356.1", 2022-09-12T19:25:41.114 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:41.115 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:41.115 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:41.115 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:22:52.555375Z", 2022-09-12T19:25:41.116 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:25:41.116 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:41.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:41.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:41.117 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:25:41.117 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:41.118 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:41.118 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:25:41.118 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:41.119 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:41.119 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:41.119 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:25:41.120 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:41.120 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:41.120 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:41.120 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:25:41.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:41.121 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:25:41.121 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:25:41.122 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:25:41.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:41.122 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:25:41Z" 2022-09-12T19:25:41.123 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:25:41.125 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010572.555375_356.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:22:52.555375Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:25:41Z" } 2022-09-12T19:25:41.126 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010572.555375_356.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:22:52.555375Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:25:41.127 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:25:41.127 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010572.555375_356.1', 3: ''} 2022-09-12T19:25:41.128 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010572.555375_356.1 2022-09-12T19:25:46.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:25:46.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:25:46.130 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:25:46.192 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:25:46.193 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:25:46.221 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:25:46.220+0000 7fabf22f3600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:25:46.222 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:25:46.221+0000 7fabf22f3600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:25:46.240 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:25:46.241 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:25:46.242 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:25:46.242 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:25:46.243 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:25:46.243 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:25:46.244 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:25:46.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:46.245 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:25:46.245 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:46.246 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:25:46.246 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:46.247 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:46.247 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:25:46.248 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:46.248 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:25:46.248 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:46.249 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:25:46.249 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:46.250 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:46.250 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:46.251 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:46.251 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:25:46.251 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:46.252 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:46.252 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:25:46.252 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:46.253 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:46.253 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:46.254 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:25:46.254 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:46.255 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:46.255 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:46.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:46.256 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:25:46.257 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:46.257 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:46.258 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010573.219875_376.1", 2022-09-12T19:25:46.258 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:46.259 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:46.259 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:46.260 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:22:53.219875Z", 2022-09-12T19:25:46.260 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:25:46.261 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:46.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:46.262 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:46.262 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:25:46.263 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:46.263 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:46.263 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:25:46.264 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:46.264 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:46.265 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:46.265 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:25:46.266 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:46.266 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:46.267 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:46.267 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:25:46.268 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:46.268 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:25:46.269 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:25:46.269 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:25:46.270 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:46.270 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:25:46Z" 2022-09-12T19:25:46.271 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:25:46.273 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010573.219875_376.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:22:53.219875Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:25:46Z" } 2022-09-12T19:25:46.274 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010573.219875_376.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:22:53.219875Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:25:46.275 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:25:46.276 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010573.219875_376.1', 3: ''} 2022-09-12T19:25:46.276 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010573.219875_376.1 2022-09-12T19:25:51.278 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:25:51.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:25:51.279 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:25:51.335 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:25:51.336 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:25:51.375 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:25:51.372+0000 7f4e038c5600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:25:51.376 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:25:51.374+0000 7f4e038c5600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:25:51.418 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:25:51.418 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:25:51.418 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:25:51.419 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:25:51.419 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:25:51.419 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:25:51.419 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:25:51.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:51.420 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:25:51.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:51.420 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:25:51.421 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:51.421 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:51.421 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:25:51.421 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:51.422 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:25:51.422 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:51.422 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:25:51.422 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:51.423 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:51.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:51.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:51.423 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:25:51.424 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:51.424 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:51.424 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:25:51.424 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:51.425 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:51.425 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:51.425 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:25:51.425 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:51.425 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:51.426 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:51.426 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:51.426 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:25:51.426 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:51.427 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:51.427 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010572.551013_355.1", 2022-09-12T19:25:51.427 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:51.427 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:51.428 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:51.428 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:22:52.551013Z", 2022-09-12T19:25:51.428 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:25:51.428 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:51.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:51.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:51.429 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:25:51.429 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:51.430 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:51.430 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:25:51.430 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:51.430 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:51.431 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:51.431 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:25:51.431 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:51.431 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:51.432 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:51.432 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:25:51.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:51.432 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:25:51.433 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:25:51.433 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:25:51.433 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:51.433 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:25:51Z" 2022-09-12T19:25:51.434 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:25:51.435 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010572.551013_355.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:22:52.551013Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:25:51Z" } 2022-09-12T19:25:51.436 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010572.551013_355.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:22:52.551013Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:25:51.436 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:25:51.437 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010572.551013_355.1', 3: ''} 2022-09-12T19:25:51.437 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010572.551013_355.1 2022-09-12T19:25:56.439 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:25:56.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:25:56.440 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:25:56.490 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:25:56.490 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:25:56.538 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:25:56.535+0000 7fd4e076f600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:25:56.539 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:25:56.536+0000 7fd4e076f600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:25:56.555 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:25:56.556 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:25:56.556 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:25:56.556 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:25:56.557 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:25:56.557 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:25:56.557 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:25:56.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:56.558 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:25:56.558 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:56.558 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:25:56.559 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:56.559 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:56.559 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:25:56.559 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:56.560 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:25:56.560 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:56.560 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:25:56.561 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:56.561 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:56.561 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:56.562 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:56.562 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:25:56.562 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:56.563 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:56.563 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:25:56.563 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:56.564 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:56.564 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:56.564 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:25:56.565 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:56.565 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:56.565 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:56.566 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:56.566 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:25:56.566 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:56.567 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:56.567 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010572.555375_356.1", 2022-09-12T19:25:56.567 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:56.567 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:56.568 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:56.568 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:22:52.555375Z", 2022-09-12T19:25:56.568 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:25:56.568 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:56.569 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:56.569 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:25:56.569 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:25:56.570 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:25:56.570 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:25:56.571 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:25:56.571 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:25:56.571 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:25:56.572 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:25:56.572 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:25:56.572 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:25:56.572 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:56.573 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:25:56.573 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:25:56.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:56.573 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:25:56.574 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:25:56.574 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:25:56.574 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:25:56.574 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:25:56Z" 2022-09-12T19:25:56.575 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:25:56.576 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010572.555375_356.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:22:52.555375Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:25:56Z" } 2022-09-12T19:25:56.577 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010572.555375_356.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:22:52.555375Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:25:56.578 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:25:56.578 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010572.555375_356.1', 3: ''} 2022-09-12T19:25:56.579 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010572.555375_356.1 2022-09-12T19:26:01.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:01.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:01.582 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:01.636 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:01.636 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:01.675 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:26:01.672+0000 7f2441964600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:26:01.676 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:26:01.673+0000 7f2441964600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:26:01.700 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:26:01.701 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:26:01.701 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:26:01.701 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:26:01.702 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:26:01.702 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:26:01.702 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:26:01.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:01.703 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:26:01.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:01.703 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:26:01.703 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:01.704 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:01.704 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:26:01.704 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:01.704 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:26:01.705 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:01.705 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:26:01.705 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:01.705 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:01.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:01.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:01.706 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:26:01.706 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:01.707 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:01.707 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:26:01.707 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:01.707 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:01.708 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:01.708 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:26:01.708 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:01.708 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:01.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:01.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:01.709 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:26:01.709 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:01.710 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:01.710 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010573.094162_367.1", 2022-09-12T19:26:01.710 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:01.711 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:01.711 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:01.711 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:22:53.094162Z", 2022-09-12T19:26:01.711 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:26:01.712 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:01.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:01.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:01.712 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:26:01.713 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:01.713 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:01.713 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:26:01.713 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:01.713 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:01.714 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:01.714 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:26:01.714 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:01.714 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:01.715 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:01.715 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:26:01.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:01.715 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:26:01.716 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:26:01.716 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:26:01.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:01.716 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:26:01Z" 2022-09-12T19:26:01.717 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:26:01.717 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010573.094162_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:22:53.094162Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:26:01Z" } 2022-09-12T19:26:01.718 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010573.094162_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:22:53.094162Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:26:01.718 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:26:01.719 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010573.094162_367.1', 3: ''} 2022-09-12T19:26:01.719 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010573.094162_367.1 2022-09-12T19:26:06.720 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:06.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:06.722 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:06.772 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:06.773 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:06.828 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:26:06.825+0000 7f9e2d59d600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:26:06.829 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:26:06.826+0000 7f9e2d59d600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:26:06.842 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:26:06.842 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:26:06.843 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:26:06.843 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:26:06.844 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:26:06.844 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:26:06.845 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:26:06.845 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:06.846 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:26:06.846 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:06.847 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:26:06.847 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:06.848 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:06.848 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:26:06.849 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:06.849 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:26:06.850 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:06.850 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:26:06.851 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:06.851 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:06.852 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:06.852 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:06.853 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:26:06.853 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:06.853 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:06.854 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:26:06.854 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:06.855 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:06.856 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:06.856 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:26:06.856 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:06.857 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:06.857 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:06.858 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:06.858 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:26:06.859 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:06.859 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:06.860 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010572.551013_355.1", 2022-09-12T19:26:06.860 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:06.860 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:06.861 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:06.861 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:22:52.551013Z", 2022-09-12T19:26:06.862 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:26:06.862 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:06.863 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:06.863 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:06.864 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:26:06.864 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:06.865 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:06.865 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:26:06.866 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:06.866 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:06.867 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:06.867 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:26:06.868 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:06.868 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:06.869 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:06.869 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:26:06.870 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:06.870 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:26:06.871 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:26:06.871 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:26:06.871 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:06.872 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:26:06Z" 2022-09-12T19:26:06.872 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:26:06.874 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010572.551013_355.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:22:52.551013Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:26:06Z" } 2022-09-12T19:26:06.875 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010572.551013_355.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:22:52.551013Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:26:06.875 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:26:06.876 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010572.551013_355.1', 3: ''} 2022-09-12T19:26:06.876 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010572.551013_355.1 2022-09-12T19:26:11.878 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:11.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:11.879 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:11.936 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:11.937 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:11.972 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:26:11.968+0000 7fd5b49a0600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:26:11.973 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:26:11.969+0000 7fd5b49a0600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:26:11.989 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:26:11.989 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:26:11.990 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:26:11.990 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:26:11.990 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:26:11.990 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:26:11.991 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:26:11.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:11.991 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:26:11.991 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:11.992 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:26:11.992 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:11.992 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:11.993 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:26:11.993 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:11.993 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:26:11.994 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:11.994 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:26:11.994 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:11.995 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:11.995 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:11.995 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:11.995 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:26:11.996 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:11.996 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:11.996 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:26:11.996 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:11.997 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:11.997 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:11.997 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:26:11.997 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:11.998 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:11.998 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:11.998 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:11.998 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:26:11.999 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:11.999 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:11.999 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010575.581422_409.1", 2022-09-12T19:26:11.999 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:12.000 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:12.000 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:12.000 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:22:55.581422Z", 2022-09-12T19:26:12.000 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:26:12.001 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:12.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:12.002 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:12.002 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:26:12.002 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:12.002 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:12.003 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:26:12.003 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:12.003 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:12.003 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:12.004 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:26:12.004 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:12.004 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:12.005 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:12.005 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:26:12.005 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:12.005 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:26:12.006 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:26:12.006 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:26:12.006 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:12.007 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:26:11Z" 2022-09-12T19:26:12.007 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:26:12.008 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010575.581422_409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:22:55.581422Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:26:11Z" } 2022-09-12T19:26:12.009 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010575.581422_409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:22:55.581422Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:26:12.010 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:26:12.010 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:26:12.011 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T19:26:12.011 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T19:26:12.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:12.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:12.013 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:12.075 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:12.076 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:12.121 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:26:12.113+0000 7f5b72b20600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:26:12.121 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:26:12.114+0000 7f5b72b20600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:26:12.131 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:26:12.132 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:26:12.132 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:26:12.133 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:26:12.133 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:26:12.134 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:26:12.134 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:26:12.135 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:12.135 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:26:12.136 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:12.136 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:26:12.137 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:26:12.137 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:26:12.138 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:26:12.138 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:26:12.138 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:26:12.139 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:26:12.139 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:26:12.140 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:26:12.140 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:12.141 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:12.141 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:12.141 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:26:12.142 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:26:12.142 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:26:12.143 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:26:12.143 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:26:12.144 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:26:12.144 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:26:12.144 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:26:12.145 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:26:12.145 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:12.146 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:12.146 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:12.147 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:26:12.147 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:26:12.148 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:26:12.148 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663010575.581422_409.1", 2022-09-12T19:26:12.149 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:26:12.149 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:26:12.150 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:26:12.150 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:22:55.581422Z", 2022-09-12T19:26:12.151 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:26:12.151 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:12.152 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:12.152 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:12.152 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:26:12.153 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:26:12.153 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:26:12.154 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:26:12.154 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:26:12.155 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:26:12.155 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:26:12.156 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:26:12.156 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:26:12.157 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:12.157 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:12.158 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:26:12.158 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:12.159 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:26:12.159 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:26:12.160 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:26:12.160 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:12.160 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:26:12Z" 2022-09-12T19:26:12.161 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:26:12.163 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010575.581422_409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:22:55.581422Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:26:12Z" } 2022-09-12T19:26:12.164 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010575.581422_409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:22:55.581422Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:26:12.165 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:26:12.166 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:26:12.167 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T19:26:12.168 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T19:26:12.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:12.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:12.171 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:12.224 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:12.225 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:12.276 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:26:12.274+0000 7fcd35b24600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:26:12.277 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:26:12.275+0000 7fcd35b24600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:26:12.297 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:26:12.298 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:26:12.298 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:26:12.298 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:26:12.299 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:26:12.299 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:26:12.299 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:26:12.299 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:12.300 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:26:12.300 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:12.300 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:26:12.300 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:12.301 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:12.301 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:26:12.301 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:12.301 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:26:12.302 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:12.302 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:26:12.302 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:12.302 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:12.303 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:12.303 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:12.303 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:26:12.303 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:12.304 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:12.304 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:26:12.304 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:12.304 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:12.305 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:12.305 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:26:12.305 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:12.306 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:12.306 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:12.306 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:12.306 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:26:12.306 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:12.307 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:12.307 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010572.551013_355.1", 2022-09-12T19:26:12.307 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:12.307 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:12.308 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:12.308 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:22:52.551013Z", 2022-09-12T19:26:12.308 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:26:12.308 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:12.309 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:12.309 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:12.309 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:26:12.309 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:12.310 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:12.310 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:26:12.310 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:12.310 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:12.311 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:12.311 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:26:12.311 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:12.311 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:12.312 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:12.312 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:26:12.312 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:12.312 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:26:12.312 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:26:12.313 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:26:12.313 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:12.313 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:26:12Z" 2022-09-12T19:26:12.313 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:26:12.315 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010572.551013_355.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:22:52.551013Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:26:12Z" } 2022-09-12T19:26:12.316 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010572.551013_355.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:22:52.551013Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:26:12.316 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:26:12.317 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010572.551013_355.1', 3: ''} 2022-09-12T19:26:12.317 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010572.551013_355.1 2022-09-12T19:26:17.318 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:17.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:17.320 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:17.371 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:17.371 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:17.406 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:26:17.403+0000 7f4ac74ce600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:26:17.406 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:26:17.404+0000 7f4ac74ce600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:26:17.418 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:26:17.418 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:26:17.419 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:26:17.419 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:26:17.419 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:26:17.419 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:26:17.420 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:26:17.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:17.420 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:26:17.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:17.421 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:26:17.421 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:17.421 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:17.421 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:26:17.422 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:17.422 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:26:17.422 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:17.423 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:26:17.423 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:17.423 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:17.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:17.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:17.424 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:26:17.424 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:17.424 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:17.425 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:26:17.425 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:17.425 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:17.426 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:17.426 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:26:17.426 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:17.427 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:17.427 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:17.427 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:17.427 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:26:17.428 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:17.428 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:17.428 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010573.160868_371.1", 2022-09-12T19:26:17.428 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:17.429 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:17.429 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:17.429 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:22:53.160868Z", 2022-09-12T19:26:17.429 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:26:17.430 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:17.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:17.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:17.430 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:26:17.431 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:17.431 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:17.431 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:26:17.431 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:17.432 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:17.432 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:17.432 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:26:17.432 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:17.432 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:17.433 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:17.433 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:26:17.433 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:17.433 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:26:17.434 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:26:17.434 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:26:17.434 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:17.434 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:26:17Z" 2022-09-12T19:26:17.435 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:26:17.436 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010573.160868_371.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:22:53.160868Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:26:17Z" } 2022-09-12T19:26:17.437 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010573.160868_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:22:53.160868Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:26:17.437 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:26:17.438 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010573.160868_371.1', 3: ''} 2022-09-12T19:26:17.438 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010573.160868_371.1 2022-09-12T19:26:22.440 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:22.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:22.442 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:22.499 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:22.499 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:22.529 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:26:22.526+0000 7ff719a23600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:26:22.530 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:26:22.528+0000 7ff719a23600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:26:22.543 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:26:22.543 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:26:22.544 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:26:22.544 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:26:22.545 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:26:22.545 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:26:22.546 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:26:22.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:22.547 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:26:22.548 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:22.548 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:26:22.549 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:22.549 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:22.550 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:26:22.550 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:22.551 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:26:22.551 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:22.552 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:26:22.552 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:22.553 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:22.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:22.554 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:22.554 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:26:22.555 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:22.555 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:22.556 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:26:22.556 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:22.557 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:22.557 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:22.558 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:26:22.558 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:22.559 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:22.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:22.560 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:22.560 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:26:22.561 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:22.561 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:22.562 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010572.555375_356.1", 2022-09-12T19:26:22.562 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:22.563 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:22.563 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:22.564 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:22:52.555375Z", 2022-09-12T19:26:22.564 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:26:22.565 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:22.565 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:22.566 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:22.566 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:26:22.567 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:22.567 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:22.568 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:26:22.568 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:22.569 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:22.569 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:22.570 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:26:22.570 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:22.571 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:22.571 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:22.571 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:26:22.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:22.572 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:26:22.572 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:26:22.572 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:26:22.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:22.573 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:26:22Z" 2022-09-12T19:26:22.573 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:26:22.575 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010572.555375_356.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:22:52.555375Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:26:22Z" } 2022-09-12T19:26:22.575 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010572.555375_356.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:22:52.555375Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:26:22.576 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:26:22.577 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010572.555375_356.1', 3: ''} 2022-09-12T19:26:22.577 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010572.555375_356.1 2022-09-12T19:26:27.579 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:27.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:27.582 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:27.649 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:27.650 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:27.688 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:26:27.686+0000 7fd2e7867600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:26:27.689 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:26:27.687+0000 7fd2e7867600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:26:27.703 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:26:27.703 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:26:27.704 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:26:27.704 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:26:27.704 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:26:27.704 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:26:27.705 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:26:27.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:27.705 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:26:27.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:27.706 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:26:27.706 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:27.706 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:27.707 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:26:27.707 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:27.707 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:26:27.707 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:27.708 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:26:27.708 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:27.708 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:27.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:27.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:27.709 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:26:27.709 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:27.709 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:27.710 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:26:27.710 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:27.710 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:27.710 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:27.711 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:26:27.711 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:27.711 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:27.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:27.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:27.712 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:26:27.712 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:27.712 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:27.713 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010573.099727_368.1", 2022-09-12T19:26:27.713 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:27.713 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:27.713 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:27.714 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:22:53.099727Z", 2022-09-12T19:26:27.714 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:26:27.714 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:27.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:27.715 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:27.715 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:26:27.715 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:27.715 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:27.716 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:26:27.716 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:27.716 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:27.716 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:27.717 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:26:27.717 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:27.717 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:27.717 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:27.718 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:26:27.718 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:27.718 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:26:27.718 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:26:27.719 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:26:27.719 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:27.719 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:26:27Z" 2022-09-12T19:26:27.719 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:26:27.721 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010573.099727_368.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:22:53.099727Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:26:27Z" } 2022-09-12T19:26:27.721 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010573.099727_368.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:22:53.099727Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:26:27.722 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:26:27.723 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010573.099727_368.1', 3: ''} 2022-09-12T19:26:27.723 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010573.099727_368.1 2022-09-12T19:26:32.724 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:32.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:32.726 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:32.780 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:32.781 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:32.828 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:26:32.826+0000 7f1b22097600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:26:32.829 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:26:32.827+0000 7f1b22097600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:26:32.848 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:26:32.848 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:26:32.849 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:26:32.849 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:26:32.849 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:26:32.849 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:26:32.850 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:26:32.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:32.850 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:26:32.851 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:32.851 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:26:32.851 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:32.851 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:32.852 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:26:32.852 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:32.852 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:26:32.852 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:32.853 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:26:32.853 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:32.853 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:32.853 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:32.854 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:32.854 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:26:32.854 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:32.854 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:32.855 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:26:32.855 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:32.855 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:32.855 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:32.856 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:26:32.856 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:32.856 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:32.857 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:32.857 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:32.857 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:26:32.857 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:32.858 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:32.858 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010573.160868_371.1", 2022-09-12T19:26:32.858 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:32.858 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:32.859 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:32.859 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:22:53.160868Z", 2022-09-12T19:26:32.859 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:26:32.859 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:32.860 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:32.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:32.860 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:26:32.860 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:32.861 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:32.861 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:26:32.861 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:32.861 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:32.862 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:32.862 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:26:32.862 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:32.863 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:32.863 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:32.863 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:26:32.863 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:32.864 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:26:32.864 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:26:32.864 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:26:32.864 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:32.864 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:26:32Z" 2022-09-12T19:26:32.865 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:26:32.866 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010573.160868_371.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:22:53.160868Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:26:32Z" } 2022-09-12T19:26:32.867 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010573.160868_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:22:53.160868Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:26:32.867 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:26:32.868 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010573.160868_371.1', 3: ''} 2022-09-12T19:26:32.868 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010573.160868_371.1 2022-09-12T19:26:37.870 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:37.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:37.872 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:37.931 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:37.931 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:37.960 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:26:37.957+0000 7f739e445600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:26:37.961 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:26:37.958+0000 7f739e445600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:26:37.976 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:26:37.977 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:26:37.977 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:26:37.977 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:26:37.978 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:26:37.978 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:26:37.978 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:26:37.979 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:37.979 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:26:37.979 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:37.980 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:26:37.980 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:37.981 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:37.981 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:26:37.981 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:37.982 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:26:37.982 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:37.982 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:26:37.983 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:37.983 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:37.983 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:37.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:37.984 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:26:37.985 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:37.985 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:37.985 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:26:37.986 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:37.986 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:37.986 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:37.987 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:26:37.987 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:37.987 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:37.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:37.988 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:37.988 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:26:37.988 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:37.989 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:37.989 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010573.105810_370.1", 2022-09-12T19:26:37.989 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:37.990 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:37.990 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:37.990 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:22:53.105810Z", 2022-09-12T19:26:37.990 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:26:37.991 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:37.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:37.991 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:37.991 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:26:37.992 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:37.992 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:37.992 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:26:37.992 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:37.993 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:37.993 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:37.993 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:26:37.993 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:37.994 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:37.994 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:37.994 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:26:37.994 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:37.995 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:26:37.995 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:26:37.995 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:26:37.995 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:37.996 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:26:37Z" 2022-09-12T19:26:37.996 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:26:37.997 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010573.105810_370.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:22:53.105810Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:26:37Z" } 2022-09-12T19:26:37.998 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010573.105810_370.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:22:53.105810Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:26:37.999 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:26:37.999 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010573.105810_370.1', 3: ''} 2022-09-12T19:26:38.000 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010573.105810_370.1 2022-09-12T19:26:43.000 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:43.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:43.002 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:43.061 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:43.061 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:43.093 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:26:43.090+0000 7f060ed74600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:26:43.094 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:26:43.092+0000 7f060ed74600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:26:43.107 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:26:43.108 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:26:43.108 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:26:43.109 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:26:43.109 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:26:43.110 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:26:43.110 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:26:43.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:43.111 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:26:43.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:43.111 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:26:43.111 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:43.112 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:43.112 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:26:43.112 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:43.113 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:26:43.113 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:43.114 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:26:43.114 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:43.115 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:43.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:43.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:43.116 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:26:43.117 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:43.117 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:43.118 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:26:43.118 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:43.119 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:43.119 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:43.120 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:26:43.120 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:43.121 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:43.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:43.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:43.122 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:26:43.123 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:43.123 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:43.124 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010573.160868_371.1", 2022-09-12T19:26:43.124 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:43.125 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:43.125 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:43.126 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:22:53.160868Z", 2022-09-12T19:26:43.126 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:26:43.127 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:43.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:43.128 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:43.128 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:26:43.129 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:43.129 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:43.130 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:26:43.130 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:43.131 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:43.131 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:43.132 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:26:43.132 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:43.133 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:43.133 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:43.134 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:26:43.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:43.135 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:26:43.135 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:26:43.136 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:26:43.136 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:43.137 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:26:43Z" 2022-09-12T19:26:43.137 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:26:43.139 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010573.160868_371.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:22:53.160868Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:26:43Z" } 2022-09-12T19:26:43.141 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010573.160868_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:22:53.160868Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:26:43.142 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:26:43.142 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010573.160868_371.1', 3: ''} 2022-09-12T19:26:43.143 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1663010575.581422_409.1 target=1_1663010573.160868_371.1 2022-09-12T19:26:48.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:48.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:48.147 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:48.205 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:48.205 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:48.253 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:26:48.250+0000 7fc94b1df600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:26:48.254 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:26:48.251+0000 7fc94b1df600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:26:48.275 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:26:48.275 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:26:48.276 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:26:48.276 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:26:48.276 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:26:48.277 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:26:48.277 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:26:48.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:48.278 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:26:48.278 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:48.279 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:26:48.279 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:48.279 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:48.280 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:26:48.280 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:48.280 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:26:48.281 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:48.281 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:26:48.282 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:48.282 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:48.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:48.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:48.283 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:26:48.284 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:48.284 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:48.284 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:26:48.285 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:48.285 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:48.285 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:48.286 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:26:48.286 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:48.286 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:48.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:48.287 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:48.287 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:26:48.288 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:48.288 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:48.288 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010575.581422_409.1", 2022-09-12T19:26:48.289 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:48.289 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:48.289 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:48.289 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:22:55.581422Z", 2022-09-12T19:26:48.290 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:26:48.290 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:48.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:48.291 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:48.291 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:26:48.291 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:48.292 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:26:48.292 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:26:48.293 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:48.293 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:48.294 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:48.294 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:26:48.294 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:26:48.295 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:48.295 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:48.296 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:26:48.296 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:48.297 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:26:48.297 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:26:48.298 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:26:48.298 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:48.299 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:26:48Z" 2022-09-12T19:26:48.299 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:26:48.300 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010575.581422_409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:22:55.581422Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:26:48Z" } 2022-09-12T19:26:48.301 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010575.581422_409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:22:55.581422Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:26:48.302 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:26:48.303 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010575.581422_409.1', 3: ''} 2022-09-12T19:26:48.304 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T19:26:48.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'jdpiak-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:26:48.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'jdpiak-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:26:48.306 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket jdpiak-56 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:26:48.370 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:48.371 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:48.461 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'jdpiak-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:26:48.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'jdpiak-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:26:48.464 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket jdpiak-57 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:26:48.519 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:48.520 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:48.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'jdpiak-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:26:48.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'jdpiak-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:26:48.599 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket jdpiak-58 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:26:48.652 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:48.652 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:48.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:48.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:48.739 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:48.787 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:48.787 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:48.844 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:26:48.845 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:0", 2022-09-12T19:26:48.845 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:6", 2022-09-12T19:26:48.846 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000078", 2022-09-12T19:26:48.846 INFO:teuthology.orchestra.run.smithi035.stdout: "meta.history", 2022-09-12T19:26:48.846 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:1", 2022-09-12T19:26:48.846 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:7", 2022-09-12T19:26:48.847 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:7", 2022-09-12T19:26:48.847 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3", 2022-09-12T19:26:48.847 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.23", 2022-09-12T19:26:48.847 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000070", 2022-09-12T19:26:48.848 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000104", 2022-09-12T19:26:48.848 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:0", 2022-09-12T19:26:48.848 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:3", 2022-09-12T19:26:48.848 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:7", 2022-09-12T19:26:48.849 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:1", 2022-09-12T19:26:48.849 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:2", 2022-09-12T19:26:48.849 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000026", 2022-09-12T19:26:48.849 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.fdf3b910-2ea3-4164-a3f8-b3e146af1c44.0", 2022-09-12T19:26:48.850 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:5", 2022-09-12T19:26:48.850 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:9", 2022-09-12T19:26:48.850 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:5", 2022-09-12T19:26:48.851 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.8", 2022-09-12T19:26:48.851 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:7", 2022-09-12T19:26:48.851 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000028", 2022-09-12T19:26:48.851 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:3", 2022-09-12T19:26:48.852 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:0", 2022-09-12T19:26:48.852 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:8", 2022-09-12T19:26:48.852 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:10", 2022-09-12T19:26:48.853 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:9", 2022-09-12T19:26:48.853 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:4", 2022-09-12T19:26:48.853 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:3", 2022-09-12T19:26:48.853 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:9", 2022-09-12T19:26:48.854 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:1", 2022-09-12T19:26:48.854 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:6", 2022-09-12T19:26:48.854 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000040", 2022-09-12T19:26:48.854 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:2", 2022-09-12T19:26:48.855 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.6", 2022-09-12T19:26:48.855 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:5", 2022-09-12T19:26:48.855 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:6", 2022-09-12T19:26:48.855 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10", 2022-09-12T19:26:48.856 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:0", 2022-09-12T19:26:48.856 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000015", 2022-09-12T19:26:48.856 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:5", 2022-09-12T19:26:48.856 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:1", 2022-09-12T19:26:48.857 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:2", 2022-09-12T19:26:48.857 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:4", 2022-09-12T19:26:48.857 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:0", 2022-09-12T19:26:48.857 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:1", 2022-09-12T19:26:48.858 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000069", 2022-09-12T19:26:48.858 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:4", 2022-09-12T19:26:48.858 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:1", 2022-09-12T19:26:48.859 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000095", 2022-09-12T19:26:48.859 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000003", 2022-09-12T19:26:48.859 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000047", 2022-09-12T19:26:48.859 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.fdf3b910-2ea3-4164-a3f8-b3e146af1c44.3", 2022-09-12T19:26:48.860 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:3", 2022-09-12T19:26:48.860 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38", 2022-09-12T19:26:48.860 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:6", 2022-09-12T19:26:48.860 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:0", 2022-09-12T19:26:48.861 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:2", 2022-09-12T19:26:48.861 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:8", 2022-09-12T19:26:48.861 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:5", 2022-09-12T19:26:48.861 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17", 2022-09-12T19:26:48.862 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:3", 2022-09-12T19:26:48.862 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:9", 2022-09-12T19:26:48.862 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000035", 2022-09-12T19:26:48.863 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:2", 2022-09-12T19:26:48.863 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:6", 2022-09-12T19:26:48.863 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000037", 2022-09-12T19:26:48.863 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19", 2022-09-12T19:26:48.864 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:5", 2022-09-12T19:26:48.864 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:1", 2022-09-12T19:26:48.864 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:5", 2022-09-12T19:26:48.864 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:0", 2022-09-12T19:26:48.865 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:10", 2022-09-12T19:26:48.865 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:8", 2022-09-12T19:26:48.865 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:2", 2022-09-12T19:26:48.866 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:9", 2022-09-12T19:26:48.866 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:10", 2022-09-12T19:26:48.866 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22", 2022-09-12T19:26:48.866 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:1", 2022-09-12T19:26:48.867 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:6", 2022-09-12T19:26:48.867 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:9", 2022-09-12T19:26:48.867 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000024", 2022-09-12T19:26:48.868 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:3", 2022-09-12T19:26:48.868 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:2", 2022-09-12T19:26:48.868 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000000", 2022-09-12T19:26:48.869 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:6", 2022-09-12T19:26:48.869 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000031", 2022-09-12T19:26:48.869 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:9", 2022-09-12T19:26:48.869 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:9", 2022-09-12T19:26:48.870 INFO:teuthology.orchestra.run.smithi035.stdout: "data_log.3", 2022-09-12T19:26:48.870 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:3", 2022-09-12T19:26:48.870 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:7", 2022-09-12T19:26:48.871 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:10", 2022-09-12T19:26:48.871 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:9", 2022-09-12T19:26:48.871 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:6", 2022-09-12T19:26:48.871 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:5", 2022-09-12T19:26:48.872 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:1", 2022-09-12T19:26:48.872 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:4", 2022-09-12T19:26:48.872 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:10", 2022-09-12T19:26:48.872 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.823fb3a5-4d8a-49dc-ae64-e05804bbea94.0.retry", 2022-09-12T19:26:48.873 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:3", 2022-09-12T19:26:48.873 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:4", 2022-09-12T19:26:48.873 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:5", 2022-09-12T19:26:48.873 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:4", 2022-09-12T19:26:48.874 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:2", 2022-09-12T19:26:48.874 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:9", 2022-09-12T19:26:48.874 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:3", 2022-09-12T19:26:48.874 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:6", 2022-09-12T19:26:48.875 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:0", 2022-09-12T19:26:48.875 INFO:teuthology.orchestra.run.smithi035.stdout: "data_loggenerations_metadata", 2022-09-12T19:26:48.875 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:4", 2022-09-12T19:26:48.875 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:7", 2022-09-12T19:26:48.876 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:6", 2022-09-12T19:26:48.876 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3", 2022-09-12T19:26:48.876 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:1", 2022-09-12T19:26:48.877 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:2", 2022-09-12T19:26:48.877 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:5", 2022-09-12T19:26:48.877 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:10", 2022-09-12T19:26:48.877 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:10", 2022-09-12T19:26:48.878 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:8", 2022-09-12T19:26:48.878 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:3", 2022-09-12T19:26:48.878 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:8", 2022-09-12T19:26:48.878 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:1", 2022-09-12T19:26:48.879 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:5", 2022-09-12T19:26:48.879 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000076", 2022-09-12T19:26:48.879 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:5", 2022-09-12T19:26:48.879 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000113", 2022-09-12T19:26:48.880 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:0", 2022-09-12T19:26:48.880 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000005", 2022-09-12T19:26:48.880 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:10", 2022-09-12T19:26:48.880 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000011", 2022-09-12T19:26:48.881 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000058", 2022-09-12T19:26:48.881 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:0", 2022-09-12T19:26:48.881 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:3", 2022-09-12T19:26:48.881 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000052", 2022-09-12T19:26:48.882 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:2", 2022-09-12T19:26:48.882 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1", 2022-09-12T19:26:48.882 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.21", 2022-09-12T19:26:48.882 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:2", 2022-09-12T19:26:48.883 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:1", 2022-09-12T19:26:48.883 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:7", 2022-09-12T19:26:48.883 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:0", 2022-09-12T19:26:48.884 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:4", 2022-09-12T19:26:48.884 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:8", 2022-09-12T19:26:48.884 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:9", 2022-09-12T19:26:48.884 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:1", 2022-09-12T19:26:48.885 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:1", 2022-09-12T19:26:48.885 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37", 2022-09-12T19:26:48.885 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:7", 2022-09-12T19:26:48.885 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:9", 2022-09-12T19:26:48.886 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000088", 2022-09-12T19:26:48.886 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:0", 2022-09-12T19:26:48.886 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36", 2022-09-12T19:26:48.886 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000080", 2022-09-12T19:26:48.887 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000090", 2022-09-12T19:26:48.887 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:10", 2022-09-12T19:26:48.887 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000110", 2022-09-12T19:26:48.887 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:2", 2022-09-12T19:26:48.888 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:1", 2022-09-12T19:26:48.888 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:7", 2022-09-12T19:26:48.888 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:5", 2022-09-12T19:26:48.888 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:26:48.889 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.fdf3b910-2ea3-4164-a3f8-b3e146af1c44.1", 2022-09-12T19:26:48.889 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000096", 2022-09-12T19:26:48.889 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:0", 2022-09-12T19:26:48.889 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:6", 2022-09-12T19:26:48.890 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:9", 2022-09-12T19:26:48.890 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:2", 2022-09-12T19:26:48.890 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000087", 2022-09-12T19:26:48.891 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:1", 2022-09-12T19:26:48.891 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:6", 2022-09-12T19:26:48.891 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000008", 2022-09-12T19:26:48.891 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:4", 2022-09-12T19:26:48.892 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:3", 2022-09-12T19:26:48.892 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:8", 2022-09-12T19:26:48.892 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:2", 2022-09-12T19:26:48.893 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:10", 2022-09-12T19:26:48.893 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:9", 2022-09-12T19:26:48.893 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:4", 2022-09-12T19:26:48.893 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:1", 2022-09-12T19:26:48.894 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:8", 2022-09-12T19:26:48.894 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:1", 2022-09-12T19:26:48.894 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:2", 2022-09-12T19:26:48.894 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000006", 2022-09-12T19:26:48.895 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000029", 2022-09-12T19:26:48.895 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:4", 2022-09-12T19:26:48.895 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:10", 2022-09-12T19:26:48.895 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000089", 2022-09-12T19:26:48.896 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000022", 2022-09-12T19:26:48.896 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:0", 2022-09-12T19:26:48.896 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:5", 2022-09-12T19:26:48.896 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:4", 2022-09-12T19:26:48.897 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38", 2022-09-12T19:26:48.897 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4", 2022-09-12T19:26:48.897 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:0", 2022-09-12T19:26:48.897 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.fdf3b910-2ea3-4164-a3f8-b3e146af1c44.2.retry", 2022-09-12T19:26:48.898 INFO:teuthology.orchestra.run.smithi035.stdout: "bilog.trim", 2022-09-12T19:26:48.898 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000016", 2022-09-12T19:26:48.898 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:3", 2022-09-12T19:26:48.899 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:0", 2022-09-12T19:26:48.899 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:2", 2022-09-12T19:26:48.899 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:6", 2022-09-12T19:26:48.899 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:8", 2022-09-12T19:26:48.900 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000041", 2022-09-12T19:26:48.900 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:7", 2022-09-12T19:26:48.900 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:2", 2022-09-12T19:26:48.900 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:9", 2022-09-12T19:26:48.901 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:1", 2022-09-12T19:26:48.901 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000018", 2022-09-12T19:26:48.901 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:7", 2022-09-12T19:26:48.901 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000092", 2022-09-12T19:26:48.902 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:5", 2022-09-12T19:26:48.902 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:4", 2022-09-12T19:26:48.902 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:2", 2022-09-12T19:26:48.903 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:4", 2022-09-12T19:26:48.903 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:9", 2022-09-12T19:26:48.903 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:4", 2022-09-12T19:26:48.903 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23", 2022-09-12T19:26:48.904 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:1", 2022-09-12T19:26:48.904 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:1", 2022-09-12T19:26:48.904 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:5", 2022-09-12T19:26:48.904 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:10", 2022-09-12T19:26:48.905 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.4", 2022-09-12T19:26:48.905 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:9", 2022-09-12T19:26:48.905 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:8", 2022-09-12T19:26:48.905 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:5", 2022-09-12T19:26:48.906 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:9", 2022-09-12T19:26:48.906 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:5", 2022-09-12T19:26:48.906 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:10", 2022-09-12T19:26:48.906 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:0", 2022-09-12T19:26:48.907 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21", 2022-09-12T19:26:48.907 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:1", 2022-09-12T19:26:48.907 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000014", 2022-09-12T19:26:48.907 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:7", 2022-09-12T19:26:48.908 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:5", 2022-09-12T19:26:48.908 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:9", 2022-09-12T19:26:48.908 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:7", 2022-09-12T19:26:48.908 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000112", 2022-09-12T19:26:48.909 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:3", 2022-09-12T19:26:48.909 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000007", 2022-09-12T19:26:48.909 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000021", 2022-09-12T19:26:48.910 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:0", 2022-09-12T19:26:48.910 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.16", 2022-09-12T19:26:48.910 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000064", 2022-09-12T19:26:48.910 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:5", 2022-09-12T19:26:48.911 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:0", 2022-09-12T19:26:48.911 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17", 2022-09-12T19:26:48.911 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:9", 2022-09-12T19:26:48.911 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12", 2022-09-12T19:26:48.912 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14", 2022-09-12T19:26:48.912 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000071", 2022-09-12T19:26:48.912 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:5", 2022-09-12T19:26:48.912 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:9", 2022-09-12T19:26:48.913 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000074", 2022-09-12T19:26:48.913 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:7", 2022-09-12T19:26:48.913 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:8", 2022-09-12T19:26:48.913 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:3", 2022-09-12T19:26:48.914 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:6", 2022-09-12T19:26:48.914 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:2", 2022-09-12T19:26:48.914 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:2", 2022-09-12T19:26:48.915 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:2", 2022-09-12T19:26:48.915 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:1", 2022-09-12T19:26:48.915 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000081", 2022-09-12T19:26:48.915 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000009", 2022-09-12T19:26:48.916 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:9", 2022-09-12T19:26:48.916 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:5", 2022-09-12T19:26:48.916 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000121", 2022-09-12T19:26:48.916 INFO:teuthology.orchestra.run.smithi035.stdout: "data_log.1", 2022-09-12T19:26:48.917 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:5", 2022-09-12T19:26:48.917 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:8", 2022-09-12T19:26:48.917 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:1", 2022-09-12T19:26:48.917 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:0", 2022-09-12T19:26:48.918 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:2", 2022-09-12T19:26:48.918 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:8", 2022-09-12T19:26:48.918 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:6", 2022-09-12T19:26:48.918 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.17", 2022-09-12T19:26:48.919 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000125", 2022-09-12T19:26:48.919 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000082", 2022-09-12T19:26:48.919 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:0", 2022-09-12T19:26:48.919 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16", 2022-09-12T19:26:48.920 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22", 2022-09-12T19:26:48.920 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:2", 2022-09-12T19:26:48.920 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:0", 2022-09-12T19:26:48.920 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:4", 2022-09-12T19:26:48.921 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000105", 2022-09-12T19:26:48.921 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000059", 2022-09-12T19:26:48.921 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:0", 2022-09-12T19:26:48.922 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:5", 2022-09-12T19:26:48.922 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:3", 2022-09-12T19:26:48.922 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:4", 2022-09-12T19:26:48.922 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:3", 2022-09-12T19:26:48.923 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:4", 2022-09-12T19:26:48.923 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000077", 2022-09-12T19:26:48.923 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000032", 2022-09-12T19:26:48.923 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:6", 2022-09-12T19:26:48.924 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:0", 2022-09-12T19:26:48.924 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.823fb3a5-4d8a-49dc-ae64-e05804bbea94.1.retry", 2022-09-12T19:26:48.924 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:10", 2022-09-12T19:26:48.924 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.24", 2022-09-12T19:26:48.925 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:10", 2022-09-12T19:26:48.925 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13", 2022-09-12T19:26:48.925 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:4", 2022-09-12T19:26:48.925 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:10", 2022-09-12T19:26:48.926 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000053", 2022-09-12T19:26:48.926 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:4", 2022-09-12T19:26:48.926 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000091", 2022-09-12T19:26:48.926 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:6", 2022-09-12T19:26:48.927 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:26:48.927 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000065", 2022-09-12T19:26:48.927 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:0", 2022-09-12T19:26:48.927 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:10", 2022-09-12T19:26:48.928 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000083", 2022-09-12T19:26:48.928 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000010", 2022-09-12T19:26:48.928 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:10", 2022-09-12T19:26:48.928 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:6", 2022-09-12T19:26:48.929 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:6", 2022-09-12T19:26:48.929 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:0", 2022-09-12T19:26:48.929 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:10", 2022-09-12T19:26:48.930 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:4", 2022-09-12T19:26:48.930 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21", 2022-09-12T19:26:48.930 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:10", 2022-09-12T19:26:48.930 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.7", 2022-09-12T19:26:48.931 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.3", 2022-09-12T19:26:48.931 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:8", 2022-09-12T19:26:48.931 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:26:48.931 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:3", 2022-09-12T19:26:48.932 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:6", 2022-09-12T19:26:48.932 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:0", 2022-09-12T19:26:48.932 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:1", 2022-09-12T19:26:48.932 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:1", 2022-09-12T19:26:48.933 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:6", 2022-09-12T19:26:48.933 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:7", 2022-09-12T19:26:48.933 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:5", 2022-09-12T19:26:48.933 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:0", 2022-09-12T19:26:48.934 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:7", 2022-09-12T19:26:48.934 INFO:teuthology.orchestra.run.smithi035.stdout: "mdlog.sync-status.shard.2", 2022-09-12T19:26:48.934 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:6", 2022-09-12T19:26:48.935 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.fdf3b910-2ea3-4164-a3f8-b3e146af1c44.2", 2022-09-12T19:26:48.935 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:0", 2022-09-12T19:26:48.935 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:3", 2022-09-12T19:26:48.935 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:0", 2022-09-12T19:26:48.936 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:2", 2022-09-12T19:26:48.936 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:3", 2022-09-12T19:26:48.936 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:10", 2022-09-12T19:26:48.936 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:6", 2022-09-12T19:26:48.937 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:4", 2022-09-12T19:26:48.937 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000045", 2022-09-12T19:26:48.937 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:8", 2022-09-12T19:26:48.937 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:10", 2022-09-12T19:26:48.938 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:1", 2022-09-12T19:26:48.938 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000002", 2022-09-12T19:26:48.938 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.12", 2022-09-12T19:26:48.939 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:2", 2022-09-12T19:26:48.939 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:10", 2022-09-12T19:26:48.939 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:10", 2022-09-12T19:26:48.939 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000116", 2022-09-12T19:26:48.940 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.25", 2022-09-12T19:26:48.940 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:7", 2022-09-12T19:26:48.940 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:2", 2022-09-12T19:26:48.940 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:3", 2022-09-12T19:26:48.941 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:1", 2022-09-12T19:26:48.941 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:5", 2022-09-12T19:26:48.941 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:3", 2022-09-12T19:26:48.941 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000034", 2022-09-12T19:26:48.942 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:5", 2022-09-12T19:26:48.942 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:26:48.942 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:2", 2022-09-12T19:26:48.942 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000101", 2022-09-12T19:26:48.943 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:7", 2022-09-12T19:26:48.943 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:5", 2022-09-12T19:26:48.943 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:0", 2022-09-12T19:26:48.943 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.13", 2022-09-12T19:26:48.944 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:7", 2022-09-12T19:26:48.944 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:6", 2022-09-12T19:26:48.944 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:8", 2022-09-12T19:26:48.944 INFO:teuthology.orchestra.run.smithi035.stdout: "mdlog.sync-status.shard.1", 2022-09-12T19:26:48.945 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:2", 2022-09-12T19:26:48.945 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:4", 2022-09-12T19:26:48.945 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:6", 2022-09-12T19:26:48.945 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:0", 2022-09-12T19:26:48.946 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:3", 2022-09-12T19:26:48.946 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:5", 2022-09-12T19:26:48.946 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:7", 2022-09-12T19:26:48.946 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:1", 2022-09-12T19:26:48.947 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:2", 2022-09-12T19:26:48.947 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:3", 2022-09-12T19:26:48.947 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000079", 2022-09-12T19:26:48.947 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000049", 2022-09-12T19:26:48.948 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:7", 2022-09-12T19:26:48.948 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:3", 2022-09-12T19:26:48.948 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000117", 2022-09-12T19:26:48.948 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000044", 2022-09-12T19:26:48.949 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:3", 2022-09-12T19:26:48.949 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:9", 2022-09-12T19:26:48.949 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.14", 2022-09-12T19:26:48.949 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:2", 2022-09-12T19:26:48.950 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:3", 2022-09-12T19:26:48.950 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000066", 2022-09-12T19:26:48.950 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:9", 2022-09-12T19:26:48.951 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11", 2022-09-12T19:26:48.951 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:8", 2022-09-12T19:26:48.951 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:4", 2022-09-12T19:26:48.951 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:7", 2022-09-12T19:26:48.952 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:8", 2022-09-12T19:26:48.952 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:7", 2022-09-12T19:26:48.952 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:9", 2022-09-12T19:26:48.952 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000068", 2022-09-12T19:26:48.953 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000085", 2022-09-12T19:26:48.953 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:1", 2022-09-12T19:26:48.953 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:8", 2022-09-12T19:26:48.953 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:2", 2022-09-12T19:26:48.954 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:4", 2022-09-12T19:26:48.954 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:10", 2022-09-12T19:26:48.954 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000073", 2022-09-12T19:26:48.954 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:8", 2022-09-12T19:26:48.955 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000038", 2022-09-12T19:26:48.955 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:3", 2022-09-12T19:26:48.955 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:6", 2022-09-12T19:26:48.955 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:1", 2022-09-12T19:26:48.956 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:10", 2022-09-12T19:26:48.956 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:5", 2022-09-12T19:26:48.956 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.10", 2022-09-12T19:26:48.956 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:2", 2022-09-12T19:26:48.957 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000118", 2022-09-12T19:26:48.957 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:8", 2022-09-12T19:26:48.957 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:6", 2022-09-12T19:26:48.957 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:4", 2022-09-12T19:26:48.958 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:1", 2022-09-12T19:26:48.958 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:2", 2022-09-12T19:26:48.958 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000036", 2022-09-12T19:26:48.958 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:9", 2022-09-12T19:26:48.959 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:2", 2022-09-12T19:26:48.959 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:8", 2022-09-12T19:26:48.959 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:0", 2022-09-12T19:26:48.959 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000103", 2022-09-12T19:26:48.960 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:10", 2022-09-12T19:26:48.960 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:2", 2022-09-12T19:26:48.960 INFO:teuthology.orchestra.run.smithi035.stdout: "mdlog.sync-status.shard.3", 2022-09-12T19:26:48.961 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:7", 2022-09-12T19:26:48.961 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:3", 2022-09-12T19:26:48.961 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:0", 2022-09-12T19:26:48.961 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:9", 2022-09-12T19:26:48.962 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.fdf3b910-2ea3-4164-a3f8-b3e146af1c44.1.retry", 2022-09-12T19:26:48.962 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000119", 2022-09-12T19:26:48.962 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:5", 2022-09-12T19:26:48.962 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000098", 2022-09-12T19:26:48.963 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:10", 2022-09-12T19:26:48.963 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:3", 2022-09-12T19:26:48.963 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:5", 2022-09-12T19:26:48.963 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:1", 2022-09-12T19:26:48.964 INFO:teuthology.orchestra.run.smithi035.stdout: "mdlog.sync-status.shard.0", 2022-09-12T19:26:48.964 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:6", 2022-09-12T19:26:48.964 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000027", 2022-09-12T19:26:48.964 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:6", 2022-09-12T19:26:48.965 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:0", 2022-09-12T19:26:48.965 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:3", 2022-09-12T19:26:48.965 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:7", 2022-09-12T19:26:48.965 INFO:teuthology.orchestra.run.smithi035.stdout: "mdlog.sync-status", 2022-09-12T19:26:48.966 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000019", 2022-09-12T19:26:48.966 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:8", 2022-09-12T19:26:48.966 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000039", 2022-09-12T19:26:48.967 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:26:48.967 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:9", 2022-09-12T19:26:48.967 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:9", 2022-09-12T19:26:48.967 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:6", 2022-09-12T19:26:48.967 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:8", 2022-09-12T19:26:48.968 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:8", 2022-09-12T19:26:48.968 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:7", 2022-09-12T19:26:48.968 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:8", 2022-09-12T19:26:48.969 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:8", 2022-09-12T19:26:48.969 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:9", 2022-09-12T19:26:48.969 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:7", 2022-09-12T19:26:48.969 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:1", 2022-09-12T19:26:48.970 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000100", 2022-09-12T19:26:48.970 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:8", 2022-09-12T19:26:48.970 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:8", 2022-09-12T19:26:48.970 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:9", 2022-09-12T19:26:48.971 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:2", 2022-09-12T19:26:48.971 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:4", 2022-09-12T19:26:48.971 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:4", 2022-09-12T19:26:48.971 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:6", 2022-09-12T19:26:48.972 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:6", 2022-09-12T19:26:48.972 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000093", 2022-09-12T19:26:48.972 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T19:26:48.972 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:8", 2022-09-12T19:26:48.973 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:3", 2022-09-12T19:26:48.973 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:10", 2022-09-12T19:26:48.973 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:6", 2022-09-12T19:26:48.973 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:4", 2022-09-12T19:26:48.974 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:9", 2022-09-12T19:26:48.974 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000004", 2022-09-12T19:26:48.974 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000063", 2022-09-12T19:26:48.974 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000122", 2022-09-12T19:26:48.975 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:8", 2022-09-12T19:26:48.975 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:6", 2022-09-12T19:26:48.975 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000057", 2022-09-12T19:26:48.976 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4", 2022-09-12T19:26:48.976 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13", 2022-09-12T19:26:48.976 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000054", 2022-09-12T19:26:48.976 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000114", 2022-09-12T19:26:48.977 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15", 2022-09-12T19:26:48.977 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:7", 2022-09-12T19:26:48.977 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:6", 2022-09-12T19:26:48.977 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.2", 2022-09-12T19:26:48.978 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:4", 2022-09-12T19:26:48.978 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:10", 2022-09-12T19:26:48.978 INFO:teuthology.orchestra.run.smithi035.stdout: "data_log.0", 2022-09-12T19:26:48.978 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:5", 2022-09-12T19:26:48.979 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:3", 2022-09-12T19:26:48.979 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:9", 2022-09-12T19:26:48.979 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:8", 2022-09-12T19:26:48.980 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:8", 2022-09-12T19:26:48.980 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000012", 2022-09-12T19:26:48.980 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:7", 2022-09-12T19:26:48.980 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:7", 2022-09-12T19:26:48.981 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:7", 2022-09-12T19:26:48.981 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:2", 2022-09-12T19:26:48.981 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:9", 2022-09-12T19:26:48.981 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:0", 2022-09-12T19:26:48.982 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:0", 2022-09-12T19:26:48.982 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000084", 2022-09-12T19:26:48.982 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000043", 2022-09-12T19:26:48.982 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5", 2022-09-12T19:26:48.983 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:4", 2022-09-12T19:26:48.983 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:3", 2022-09-12T19:26:48.983 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:3", 2022-09-12T19:26:48.983 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:7", 2022-09-12T19:26:48.984 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:0", 2022-09-12T19:26:48.984 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.11", 2022-09-12T19:26:48.984 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:0", 2022-09-12T19:26:48.985 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:6", 2022-09-12T19:26:48.985 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:0", 2022-09-12T19:26:48.985 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:4", 2022-09-12T19:26:48.985 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:2", 2022-09-12T19:26:48.986 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:8", 2022-09-12T19:26:48.986 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:2", 2022-09-12T19:26:48.986 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000111", 2022-09-12T19:26:48.986 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000048", 2022-09-12T19:26:48.987 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:9", 2022-09-12T19:26:48.987 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000020", 2022-09-12T19:26:48.987 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:1", 2022-09-12T19:26:48.987 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:10", 2022-09-12T19:26:48.988 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:6", 2022-09-12T19:26:48.988 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:8", 2022-09-12T19:26:48.988 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:0", 2022-09-12T19:26:48.988 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000099", 2022-09-12T19:26:48.989 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000056", 2022-09-12T19:26:48.989 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:7", 2022-09-12T19:26:48.989 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:9", 2022-09-12T19:26:48.989 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:4", 2022-09-12T19:26:48.990 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.19", 2022-09-12T19:26:48.990 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36", 2022-09-12T19:26:48.990 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:10", 2022-09-12T19:26:48.991 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:3", 2022-09-12T19:26:48.991 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:7", 2022-09-12T19:26:48.991 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:8", 2022-09-12T19:26:48.991 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15", 2022-09-12T19:26:48.992 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:6", 2022-09-12T19:26:48.992 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:6", 2022-09-12T19:26:48.992 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000072", 2022-09-12T19:26:48.992 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:1", 2022-09-12T19:26:48.993 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000062", 2022-09-12T19:26:48.993 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:9", 2022-09-12T19:26:48.993 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:1", 2022-09-12T19:26:48.993 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000115", 2022-09-12T19:26:48.994 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19", 2022-09-12T19:26:48.994 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:10", 2022-09-12T19:26:48.994 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.1", 2022-09-12T19:26:48.994 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000033", 2022-09-12T19:26:48.995 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:5", 2022-09-12T19:26:48.995 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:1", 2022-09-12T19:26:48.995 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:4", 2022-09-12T19:26:48.995 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.823fb3a5-4d8a-49dc-ae64-e05804bbea94.2", 2022-09-12T19:26:48.996 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:4", 2022-09-12T19:26:48.996 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:10", 2022-09-12T19:26:48.996 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:5", 2022-09-12T19:26:48.997 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:9", 2022-09-12T19:26:48.997 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:10", 2022-09-12T19:26:48.997 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:1", 2022-09-12T19:26:48.997 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000025", 2022-09-12T19:26:48.998 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:7", 2022-09-12T19:26:48.998 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000123", 2022-09-12T19:26:48.998 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11", 2022-09-12T19:26:48.998 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:4", 2022-09-12T19:26:48.999 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.18", 2022-09-12T19:26:48.999 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:6", 2022-09-12T19:26:48.999 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000108", 2022-09-12T19:26:48.999 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000094", 2022-09-12T19:26:49.000 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:1", 2022-09-12T19:26:49.000 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:5", 2022-09-12T19:26:49.000 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:8", 2022-09-12T19:26:49.000 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10", 2022-09-12T19:26:49.001 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.823fb3a5-4d8a-49dc-ae64-e05804bbea94.2.retry", 2022-09-12T19:26:49.001 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000060", 2022-09-12T19:26:49.001 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:7", 2022-09-12T19:26:49.001 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:4", 2022-09-12T19:26:49.002 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000109", 2022-09-12T19:26:49.002 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.15", 2022-09-12T19:26:49.002 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:5", 2022-09-12T19:26:49.003 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:8", 2022-09-12T19:26:49.003 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:3", 2022-09-12T19:26:49.003 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:7", 2022-09-12T19:26:49.004 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14", 2022-09-12T19:26:49.004 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:5", 2022-09-12T19:26:49.004 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:0", 2022-09-12T19:26:49.004 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000013", 2022-09-12T19:26:49.005 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:3", 2022-09-12T19:26:49.005 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:7", 2022-09-12T19:26:49.005 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5", 2022-09-12T19:26:49.005 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:8", 2022-09-12T19:26:49.006 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:7", 2022-09-12T19:26:49.006 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:8", 2022-09-12T19:26:49.006 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:6", 2022-09-12T19:26:49.007 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:26:49.007 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.823fb3a5-4d8a-49dc-ae64-e05804bbea94.0", 2022-09-12T19:26:49.007 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:10", 2022-09-12T19:26:49.007 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:2", 2022-09-12T19:26:49.008 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:4", 2022-09-12T19:26:49.008 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:8", 2022-09-12T19:26:49.008 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:3", 2022-09-12T19:26:49.008 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000051", 2022-09-12T19:26:49.009 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:3", 2022-09-12T19:26:49.009 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:2", 2022-09-12T19:26:49.009 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000106", 2022-09-12T19:26:49.009 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:9", 2022-09-12T19:26:49.010 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:0", 2022-09-12T19:26:49.010 INFO:teuthology.orchestra.run.smithi035.stdout: "data_log.0.yvq1UEd/bmQ.0", 2022-09-12T19:26:49.010 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:0", 2022-09-12T19:26:49.010 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:6", 2022-09-12T19:26:49.011 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37", 2022-09-12T19:26:49.011 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000017", 2022-09-12T19:26:49.011 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:4", 2022-09-12T19:26:49.011 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:9", 2022-09-12T19:26:49.012 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:4", 2022-09-12T19:26:49.012 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:9", 2022-09-12T19:26:49.012 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:0", 2022-09-12T19:26:49.012 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:5", 2022-09-12T19:26:49.013 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:5", 2022-09-12T19:26:49.013 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:5", 2022-09-12T19:26:49.013 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000023", 2022-09-12T19:26:49.013 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:0", 2022-09-12T19:26:49.014 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:8", 2022-09-12T19:26:49.014 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:2", 2022-09-12T19:26:49.014 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:9", 2022-09-12T19:26:49.014 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000107", 2022-09-12T19:26:49.015 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000046", 2022-09-12T19:26:49.015 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:3", 2022-09-12T19:26:49.015 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:10", 2022-09-12T19:26:49.016 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:1", 2022-09-12T19:26:49.016 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:7", 2022-09-12T19:26:49.016 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:6", 2022-09-12T19:26:49.016 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:2", 2022-09-12T19:26:49.017 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.26", 2022-09-12T19:26:49.017 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.22", 2022-09-12T19:26:49.017 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:9", 2022-09-12T19:26:49.017 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:10", 2022-09-12T19:26:49.018 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:9", 2022-09-12T19:26:49.018 INFO:teuthology.orchestra.run.smithi035.stdout: "data_log.2", 2022-09-12T19:26:49.018 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:4", 2022-09-12T19:26:49.018 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000067", 2022-09-12T19:26:49.019 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:10", 2022-09-12T19:26:49.019 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.823fb3a5-4d8a-49dc-ae64-e05804bbea94.1", 2022-09-12T19:26:49.019 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:3", 2022-09-12T19:26:49.019 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:2", 2022-09-12T19:26:49.020 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:5", 2022-09-12T19:26:49.020 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:3", 2022-09-12T19:26:49.020 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18", 2022-09-12T19:26:49.020 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:1", 2022-09-12T19:26:49.021 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:4", 2022-09-12T19:26:49.021 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:8", 2022-09-12T19:26:49.021 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:9", 2022-09-12T19:26:49.021 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:4", 2022-09-12T19:26:49.022 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:5", 2022-09-12T19:26:49.022 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:5", 2022-09-12T19:26:49.022 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.9", 2022-09-12T19:26:49.023 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000086", 2022-09-12T19:26:49.023 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:2", 2022-09-12T19:26:49.023 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.5", 2022-09-12T19:26:49.023 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:9", 2022-09-12T19:26:49.024 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:10", 2022-09-12T19:26:49.024 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:7", 2022-09-12T19:26:49.024 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:3", 2022-09-12T19:26:49.024 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:4", 2022-09-12T19:26:49.025 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16", 2022-09-12T19:26:49.025 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1", 2022-09-12T19:26:49.025 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:7", 2022-09-12T19:26:49.025 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:10", 2022-09-12T19:26:49.026 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.20", 2022-09-12T19:26:49.026 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:8", 2022-09-12T19:26:49.026 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:6", 2022-09-12T19:26:49.026 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:10", 2022-09-12T19:26:49.027 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:3", 2022-09-12T19:26:49.027 INFO:teuthology.orchestra.run.smithi035.stdout: "data_log.2.O8sQtFmx/wa.0", 2022-09-12T19:26:49.027 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:26:49.027 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12", 2022-09-12T19:26:49.028 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:6", 2022-09-12T19:26:49.028 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:4", 2022-09-12T19:26:49.028 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000050", 2022-09-12T19:26:49.028 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:3", 2022-09-12T19:26:49.029 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000102", 2022-09-12T19:26:49.029 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:7", 2022-09-12T19:26:49.029 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:8", 2022-09-12T19:26:49.029 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:10", 2022-09-12T19:26:49.030 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:4", 2022-09-12T19:26:49.030 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:8", 2022-09-12T19:26:49.030 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:4", 2022-09-12T19:26:49.031 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:5", 2022-09-12T19:26:49.031 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:7", 2022-09-12T19:26:49.031 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23", 2022-09-12T19:26:49.031 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:2", 2022-09-12T19:26:49.032 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000124", 2022-09-12T19:26:49.032 INFO:teuthology.orchestra.run.smithi035.stdout: "data_log.1.XGKHb5Ky4oH.0", 2022-09-12T19:26:49.032 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:5", 2022-09-12T19:26:49.032 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:1", 2022-09-12T19:26:49.033 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:7", 2022-09-12T19:26:49.033 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000001", 2022-09-12T19:26:49.033 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:2", 2022-09-12T19:26:49.033 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:1", 2022-09-12T19:26:49.034 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:6", 2022-09-12T19:26:49.034 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000126", 2022-09-12T19:26:49.034 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:4", 2022-09-12T19:26:49.034 INFO:teuthology.orchestra.run.smithi035.stdout: "meta.log.38ca1f57-1358-496c-a99b-df49ee365a4f.2", 2022-09-12T19:26:49.035 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:1", 2022-09-12T19:26:49.035 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000055", 2022-09-12T19:26:49.035 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:7", 2022-09-12T19:26:49.035 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000061", 2022-09-12T19:26:49.036 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:1", 2022-09-12T19:26:49.036 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:1", 2022-09-12T19:26:49.036 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:3", 2022-09-12T19:26:49.036 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:1", 2022-09-12T19:26:49.037 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:3", 2022-09-12T19:26:49.037 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:5", 2022-09-12T19:26:49.037 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:10", 2022-09-12T19:26:49.038 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.823fb3a5-4d8a-49dc-ae64-e05804bbea94.3", 2022-09-12T19:26:49.038 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.fdf3b910-2ea3-4164-a3f8-b3e146af1c44.0.retry", 2022-09-12T19:26:49.038 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000097", 2022-09-12T19:26:49.038 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:6", 2022-09-12T19:26:49.039 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:10", 2022-09-12T19:26:49.039 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:10", 2022-09-12T19:26:49.039 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:6", 2022-09-12T19:26:49.039 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:5", 2022-09-12T19:26:49.040 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000042", 2022-09-12T19:26:49.040 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:2", 2022-09-12T19:26:49.040 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:10", 2022-09-12T19:26:49.040 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:0", 2022-09-12T19:26:49.041 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:7", 2022-09-12T19:26:49.041 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:8", 2022-09-12T19:26:49.041 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:26:49.041 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:7", 2022-09-12T19:26:49.042 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000075", 2022-09-12T19:26:49.042 INFO:teuthology.orchestra.run.smithi035.stdout: "data_log.3.1q8wZghdNOa.0", 2022-09-12T19:26:49.042 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000120", 2022-09-12T19:26:49.043 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000030", 2022-09-12T19:26:49.043 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:26:49.043 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18", 2022-09-12T19:26:49.043 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:10", 2022-09-12T19:26:49.044 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:0", 2022-09-12T19:26:49.044 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:9", 2022-09-12T19:26:49.044 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:9", 2022-09-12T19:26:49.044 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:8" 2022-09-12T19:26:49.045 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:26:49.045 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T19:26:49.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:49.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:49.048 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:49.102 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:49.102 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:49.164 INFO:teuthology.orchestra.run.smithi138.stdout:[ 2022-09-12T19:26:49.165 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:0", 2022-09-12T19:26:49.165 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:4", 2022-09-12T19:26:49.165 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000078", 2022-09-12T19:26:49.166 INFO:teuthology.orchestra.run.smithi138.stdout: "meta.history", 2022-09-12T19:26:49.166 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:1", 2022-09-12T19:26:49.166 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:7", 2022-09-12T19:26:49.167 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:3", 2022-09-12T19:26:49.167 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:0", 2022-09-12T19:26:49.167 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:3", 2022-09-12T19:26:49.167 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000070", 2022-09-12T19:26:49.168 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000104", 2022-09-12T19:26:49.168 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:3", 2022-09-12T19:26:49.168 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:7", 2022-09-12T19:26:49.168 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:1", 2022-09-12T19:26:49.169 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:10", 2022-09-12T19:26:49.169 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000026", 2022-09-12T19:26:49.169 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:5", 2022-09-12T19:26:49.170 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:5", 2022-09-12T19:26:49.170 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:6", 2022-09-12T19:26:49.170 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:3", 2022-09-12T19:26:49.171 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:2", 2022-09-12T19:26:49.171 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.8", 2022-09-12T19:26:49.171 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:7", 2022-09-12T19:26:49.171 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000028", 2022-09-12T19:26:49.172 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:0", 2022-09-12T19:26:49.172 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1", 2022-09-12T19:26:49.172 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:6", 2022-09-12T19:26:49.172 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13", 2022-09-12T19:26:49.173 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:0", 2022-09-12T19:26:49.173 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:1", 2022-09-12T19:26:49.173 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:2", 2022-09-12T19:26:49.173 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:1", 2022-09-12T19:26:49.174 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:6", 2022-09-12T19:26:49.174 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:5", 2022-09-12T19:26:49.174 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000040", 2022-09-12T19:26:49.174 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:8", 2022-09-12T19:26:49.175 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:2", 2022-09-12T19:26:49.175 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.6", 2022-09-12T19:26:49.175 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:26:49.175 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:2", 2022-09-12T19:26:49.176 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:5", 2022-09-12T19:26:49.176 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:5", 2022-09-12T19:26:49.176 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:6", 2022-09-12T19:26:49.176 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10", 2022-09-12T19:26:49.177 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23", 2022-09-12T19:26:49.177 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000015", 2022-09-12T19:26:49.177 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:5", 2022-09-12T19:26:49.178 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:1", 2022-09-12T19:26:49.178 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:2", 2022-09-12T19:26:49.178 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:3", 2022-09-12T19:26:49.178 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:7", 2022-09-12T19:26:49.179 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000069", 2022-09-12T19:26:49.179 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:2", 2022-09-12T19:26:49.179 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:1", 2022-09-12T19:26:49.179 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000095", 2022-09-12T19:26:49.180 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000003", 2022-09-12T19:26:49.180 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000047", 2022-09-12T19:26:49.180 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:3", 2022-09-12T19:26:49.181 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:6", 2022-09-12T19:26:49.181 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:0", 2022-09-12T19:26:49.181 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:8", 2022-09-12T19:26:49.181 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17", 2022-09-12T19:26:49.182 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:9", 2022-09-12T19:26:49.182 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000035", 2022-09-12T19:26:49.182 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:6", 2022-09-12T19:26:49.182 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:10", 2022-09-12T19:26:49.183 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000037", 2022-09-12T19:26:49.183 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:0", 2022-09-12T19:26:49.183 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19", 2022-09-12T19:26:49.184 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:1", 2022-09-12T19:26:49.184 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:6", 2022-09-12T19:26:49.184 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:8", 2022-09-12T19:26:49.184 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.shard.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.0.retry", 2022-09-12T19:26:49.185 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:2", 2022-09-12T19:26:49.185 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:9", 2022-09-12T19:26:49.185 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:0", 2022-09-12T19:26:49.186 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:10", 2022-09-12T19:26:49.186 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22", 2022-09-12T19:26:49.186 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:6", 2022-09-12T19:26:49.186 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:9", 2022-09-12T19:26:49.187 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000024", 2022-09-12T19:26:49.187 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:1", 2022-09-12T19:26:49.187 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:0", 2022-09-12T19:26:49.187 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:5", 2022-09-12T19:26:49.188 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000000", 2022-09-12T19:26:49.188 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:6", 2022-09-12T19:26:49.188 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000031", 2022-09-12T19:26:49.189 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:0", 2022-09-12T19:26:49.189 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:9", 2022-09-12T19:26:49.189 INFO:teuthology.orchestra.run.smithi138.stdout: "data_log.3", 2022-09-12T19:26:49.189 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:8", 2022-09-12T19:26:49.190 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:3", 2022-09-12T19:26:49.190 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:2", 2022-09-12T19:26:49.190 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:7", 2022-09-12T19:26:49.190 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:9", 2022-09-12T19:26:49.191 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:1", 2022-09-12T19:26:49.191 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:10", 2022-09-12T19:26:49.191 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:6", 2022-09-12T19:26:49.191 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:5", 2022-09-12T19:26:49.192 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:1", 2022-09-12T19:26:49.192 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:10", 2022-09-12T19:26:49.192 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.shard.823fb3a5-4d8a-49dc-ae64-e05804bbea94.0.retry", 2022-09-12T19:26:49.192 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:4", 2022-09-12T19:26:49.193 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:5", 2022-09-12T19:26:49.193 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:4", 2022-09-12T19:26:49.193 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:9", 2022-09-12T19:26:49.193 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:9", 2022-09-12T19:26:49.194 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:3", 2022-09-12T19:26:49.194 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:5", 2022-09-12T19:26:49.194 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:3", 2022-09-12T19:26:49.195 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:6", 2022-09-12T19:26:49.195 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:3", 2022-09-12T19:26:49.195 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:7", 2022-09-12T19:26:49.195 INFO:teuthology.orchestra.run.smithi138.stdout: "data_loggenerations_metadata", 2022-09-12T19:26:49.196 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14", 2022-09-12T19:26:49.196 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3", 2022-09-12T19:26:49.196 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:2", 2022-09-12T19:26:49.196 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:5", 2022-09-12T19:26:49.197 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:10", 2022-09-12T19:26:49.197 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:10", 2022-09-12T19:26:49.197 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:3", 2022-09-12T19:26:49.197 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000076", 2022-09-12T19:26:49.198 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:1", 2022-09-12T19:26:49.198 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:5", 2022-09-12T19:26:49.198 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000113", 2022-09-12T19:26:49.199 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:0", 2022-09-12T19:26:49.199 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000005", 2022-09-12T19:26:49.199 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:10", 2022-09-12T19:26:49.199 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000011", 2022-09-12T19:26:49.200 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:9", 2022-09-12T19:26:49.200 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000058", 2022-09-12T19:26:49.200 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:10", 2022-09-12T19:26:49.200 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000052", 2022-09-12T19:26:49.201 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:9", 2022-09-12T19:26:49.201 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:2", 2022-09-12T19:26:49.201 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:1", 2022-09-12T19:26:49.202 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:9", 2022-09-12T19:26:49.202 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:0", 2022-09-12T19:26:49.202 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:1", 2022-09-12T19:26:49.202 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:7", 2022-09-12T19:26:49.203 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:9", 2022-09-12T19:26:49.203 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000088", 2022-09-12T19:26:49.203 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:0", 2022-09-12T19:26:49.203 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:9", 2022-09-12T19:26:49.204 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000080", 2022-09-12T19:26:49.204 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000090", 2022-09-12T19:26:49.204 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:10", 2022-09-12T19:26:49.204 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:10", 2022-09-12T19:26:49.205 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000110", 2022-09-12T19:26:49.205 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:26:49.205 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:4", 2022-09-12T19:26:49.205 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:26:49.206 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:3", 2022-09-12T19:26:49.206 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:2", 2022-09-12T19:26:49.206 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000096", 2022-09-12T19:26:49.207 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15", 2022-09-12T19:26:49.207 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:8", 2022-09-12T19:26:49.207 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:9", 2022-09-12T19:26:49.207 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:0", 2022-09-12T19:26:49.208 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000087", 2022-09-12T19:26:49.208 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:9", 2022-09-12T19:26:49.208 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:6", 2022-09-12T19:26:49.208 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000008", 2022-09-12T19:26:49.209 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:4", 2022-09-12T19:26:49.209 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:3", 2022-09-12T19:26:49.209 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:6", 2022-09-12T19:26:49.209 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:8", 2022-09-12T19:26:49.210 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:4", 2022-09-12T19:26:49.210 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:8", 2022-09-12T19:26:49.210 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:2", 2022-09-12T19:26:49.210 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:5", 2022-09-12T19:26:49.211 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:10", 2022-09-12T19:26:49.211 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:8", 2022-09-12T19:26:49.211 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:8", 2022-09-12T19:26:49.211 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:1", 2022-09-12T19:26:49.212 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000006", 2022-09-12T19:26:49.212 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000029", 2022-09-12T19:26:49.212 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000089", 2022-09-12T19:26:49.212 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000022", 2022-09-12T19:26:49.213 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21", 2022-09-12T19:26:49.213 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:5", 2022-09-12T19:26:49.213 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38", 2022-09-12T19:26:49.213 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4", 2022-09-12T19:26:49.214 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:0", 2022-09-12T19:26:49.214 INFO:teuthology.orchestra.run.smithi138.stdout: "bilog.trim", 2022-09-12T19:26:49.214 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000016", 2022-09-12T19:26:49.214 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:3", 2022-09-12T19:26:49.215 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:3", 2022-09-12T19:26:49.215 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:2", 2022-09-12T19:26:49.215 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:6", 2022-09-12T19:26:49.216 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:8", 2022-09-12T19:26:49.216 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000041", 2022-09-12T19:26:49.216 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:3", 2022-09-12T19:26:49.216 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:7", 2022-09-12T19:26:49.217 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:2", 2022-09-12T19:26:49.217 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:1", 2022-09-12T19:26:49.217 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:8", 2022-09-12T19:26:49.217 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000018", 2022-09-12T19:26:49.218 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:7", 2022-09-12T19:26:49.218 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000092", 2022-09-12T19:26:49.218 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:5", 2022-09-12T19:26:49.218 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:2", 2022-09-12T19:26:49.219 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:4", 2022-09-12T19:26:49.219 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:8", 2022-09-12T19:26:49.219 INFO:teuthology.orchestra.run.smithi138.stdout: "data_log.2.nTDbFBc1SfD.0", 2022-09-12T19:26:49.220 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22", 2022-09-12T19:26:49.220 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:3", 2022-09-12T19:26:49.220 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23", 2022-09-12T19:26:49.220 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:1", 2022-09-12T19:26:49.221 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:1", 2022-09-12T19:26:49.221 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:5", 2022-09-12T19:26:49.221 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.4", 2022-09-12T19:26:49.221 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:9", 2022-09-12T19:26:49.222 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:5", 2022-09-12T19:26:49.222 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:7", 2022-09-12T19:26:49.222 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:5", 2022-09-12T19:26:49.223 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5", 2022-09-12T19:26:49.223 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:10", 2022-09-12T19:26:49.223 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:0", 2022-09-12T19:26:49.223 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21", 2022-09-12T19:26:49.224 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:10", 2022-09-12T19:26:49.224 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000014", 2022-09-12T19:26:49.224 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:1", 2022-09-12T19:26:49.225 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:7", 2022-09-12T19:26:49.225 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:9", 2022-09-12T19:26:49.225 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:9", 2022-09-12T19:26:49.225 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:9", 2022-09-12T19:26:49.226 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000112", 2022-09-12T19:26:49.226 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:7", 2022-09-12T19:26:49.226 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:3", 2022-09-12T19:26:49.226 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000007", 2022-09-12T19:26:49.227 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000021", 2022-09-12T19:26:49.227 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:0", 2022-09-12T19:26:49.227 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000064", 2022-09-12T19:26:49.227 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:6", 2022-09-12T19:26:49.228 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:5", 2022-09-12T19:26:49.228 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:5", 2022-09-12T19:26:49.228 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:4", 2022-09-12T19:26:49.228 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:8", 2022-09-12T19:26:49.229 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:9", 2022-09-12T19:26:49.229 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000071", 2022-09-12T19:26:49.229 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:9", 2022-09-12T19:26:49.230 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000074", 2022-09-12T19:26:49.230 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16", 2022-09-12T19:26:49.230 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:8", 2022-09-12T19:26:49.230 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:0", 2022-09-12T19:26:49.231 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:2", 2022-09-12T19:26:49.231 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:5", 2022-09-12T19:26:49.231 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:10", 2022-09-12T19:26:49.231 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:7", 2022-09-12T19:26:49.232 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:10", 2022-09-12T19:26:49.232 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:0", 2022-09-12T19:26:49.233 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:6", 2022-09-12T19:26:49.233 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:2", 2022-09-12T19:26:49.233 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:2", 2022-09-12T19:26:49.234 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:2", 2022-09-12T19:26:49.234 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000081", 2022-09-12T19:26:49.234 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:8", 2022-09-12T19:26:49.234 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000009", 2022-09-12T19:26:49.235 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:9", 2022-09-12T19:26:49.235 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:0", 2022-09-12T19:26:49.235 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000121", 2022-09-12T19:26:49.235 INFO:teuthology.orchestra.run.smithi138.stdout: "data_log.1", 2022-09-12T19:26:49.236 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:9", 2022-09-12T19:26:49.236 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:5", 2022-09-12T19:26:49.236 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11", 2022-09-12T19:26:49.237 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:5", 2022-09-12T19:26:49.237 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:0", 2022-09-12T19:26:49.237 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:2", 2022-09-12T19:26:49.237 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:6", 2022-09-12T19:26:49.238 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000125", 2022-09-12T19:26:49.238 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000082", 2022-09-12T19:26:49.238 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:9", 2022-09-12T19:26:49.239 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16", 2022-09-12T19:26:49.239 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:0", 2022-09-12T19:26:49.239 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:9", 2022-09-12T19:26:49.239 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000105", 2022-09-12T19:26:49.240 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000059", 2022-09-12T19:26:49.240 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:0", 2022-09-12T19:26:49.240 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:4", 2022-09-12T19:26:49.240 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:5", 2022-09-12T19:26:49.241 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:3", 2022-09-12T19:26:49.241 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:2", 2022-09-12T19:26:49.241 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:4", 2022-09-12T19:26:49.241 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.shard.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.3", 2022-09-12T19:26:49.242 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000077", 2022-09-12T19:26:49.242 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:5", 2022-09-12T19:26:49.242 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000032", 2022-09-12T19:26:49.242 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:8", 2022-09-12T19:26:49.243 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:3", 2022-09-12T19:26:49.243 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:1", 2022-09-12T19:26:49.243 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.shard.823fb3a5-4d8a-49dc-ae64-e05804bbea94.1.retry", 2022-09-12T19:26:49.244 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:4", 2022-09-12T19:26:49.244 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:10", 2022-09-12T19:26:49.244 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13", 2022-09-12T19:26:49.245 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:2", 2022-09-12T19:26:49.245 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000053", 2022-09-12T19:26:49.245 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000091", 2022-09-12T19:26:49.245 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:6", 2022-09-12T19:26:49.246 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:26:49.246 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:8", 2022-09-12T19:26:49.246 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:8", 2022-09-12T19:26:49.246 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000065", 2022-09-12T19:26:49.247 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:5", 2022-09-12T19:26:49.247 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:10", 2022-09-12T19:26:49.247 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000083", 2022-09-12T19:26:49.247 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000010", 2022-09-12T19:26:49.248 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:1", 2022-09-12T19:26:49.248 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:0", 2022-09-12T19:26:49.248 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:8", 2022-09-12T19:26:49.249 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:6", 2022-09-12T19:26:49.249 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:10", 2022-09-12T19:26:49.249 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:9", 2022-09-12T19:26:49.249 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:8", 2022-09-12T19:26:49.250 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:10", 2022-09-12T19:26:49.250 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:7", 2022-09-12T19:26:49.250 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:0", 2022-09-12T19:26:49.250 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:10", 2022-09-12T19:26:49.251 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.7", 2022-09-12T19:26:49.251 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.3", 2022-09-12T19:26:49.251 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:6", 2022-09-12T19:26:49.251 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:4", 2022-09-12T19:26:49.252 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:1", 2022-09-12T19:26:49.252 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:10", 2022-09-12T19:26:49.252 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:9", 2022-09-12T19:26:49.253 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:4", 2022-09-12T19:26:49.253 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:6", 2022-09-12T19:26:49.253 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:7", 2022-09-12T19:26:49.253 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:3", 2022-09-12T19:26:49.254 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:5", 2022-09-12T19:26:49.254 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:0", 2022-09-12T19:26:49.254 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:2", 2022-09-12T19:26:49.254 INFO:teuthology.orchestra.run.smithi138.stdout: "mdlog.sync-status.shard.2", 2022-09-12T19:26:49.255 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:10", 2022-09-12T19:26:49.255 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:3", 2022-09-12T19:26:49.255 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:6", 2022-09-12T19:26:49.255 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:10", 2022-09-12T19:26:49.256 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:0", 2022-09-12T19:26:49.256 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:0", 2022-09-12T19:26:49.256 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:4", 2022-09-12T19:26:49.256 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:2", 2022-09-12T19:26:49.257 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000045", 2022-09-12T19:26:49.257 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:8", 2022-09-12T19:26:49.257 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:7", 2022-09-12T19:26:49.257 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:10", 2022-09-12T19:26:49.258 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000002", 2022-09-12T19:26:49.258 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.12", 2022-09-12T19:26:49.258 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:2", 2022-09-12T19:26:49.259 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:6", 2022-09-12T19:26:49.259 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:6", 2022-09-12T19:26:49.259 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:10", 2022-09-12T19:26:49.259 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:4", 2022-09-12T19:26:49.260 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000116", 2022-09-12T19:26:49.260 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.shard.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.2", 2022-09-12T19:26:49.260 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:7", 2022-09-12T19:26:49.260 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:7", 2022-09-12T19:26:49.261 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:5", 2022-09-12T19:26:49.261 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:3", 2022-09-12T19:26:49.261 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:5", 2022-09-12T19:26:49.261 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000034", 2022-09-12T19:26:49.262 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:2", 2022-09-12T19:26:49.262 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:7", 2022-09-12T19:26:49.262 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000101", 2022-09-12T19:26:49.262 INFO:teuthology.orchestra.run.smithi138.stdout: "data_log.0.jVqGZsHiXC8.0", 2022-09-12T19:26:49.263 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:8", 2022-09-12T19:26:49.263 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:0", 2022-09-12T19:26:49.263 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.13", 2022-09-12T19:26:49.263 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:7", 2022-09-12T19:26:49.264 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:6", 2022-09-12T19:26:49.264 INFO:teuthology.orchestra.run.smithi138.stdout: "mdlog.sync-status.shard.1", 2022-09-12T19:26:49.264 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:2", 2022-09-12T19:26:49.264 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:4", 2022-09-12T19:26:49.265 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:6", 2022-09-12T19:26:49.265 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:0", 2022-09-12T19:26:49.265 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.shard.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.2.retry", 2022-09-12T19:26:49.265 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:2", 2022-09-12T19:26:49.266 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:7", 2022-09-12T19:26:49.266 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:2", 2022-09-12T19:26:49.266 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:3", 2022-09-12T19:26:49.266 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000079", 2022-09-12T19:26:49.267 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000049", 2022-09-12T19:26:49.267 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000117", 2022-09-12T19:26:49.267 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000044", 2022-09-12T19:26:49.267 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:26:49.268 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:9", 2022-09-12T19:26:49.268 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:3", 2022-09-12T19:26:49.268 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:5", 2022-09-12T19:26:49.268 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000066", 2022-09-12T19:26:49.269 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:9", 2022-09-12T19:26:49.269 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11", 2022-09-12T19:26:49.269 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:8", 2022-09-12T19:26:49.269 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:8", 2022-09-12T19:26:49.270 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:26:49.270 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:7", 2022-09-12T19:26:49.270 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:4", 2022-09-12T19:26:49.271 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:4", 2022-09-12T19:26:49.271 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:7", 2022-09-12T19:26:49.271 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:8", 2022-09-12T19:26:49.271 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000068", 2022-09-12T19:26:49.272 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000085", 2022-09-12T19:26:49.272 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:0", 2022-09-12T19:26:49.272 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:1", 2022-09-12T19:26:49.272 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:9", 2022-09-12T19:26:49.273 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:4", 2022-09-12T19:26:49.273 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:4", 2022-09-12T19:26:49.273 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36", 2022-09-12T19:26:49.274 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:5", 2022-09-12T19:26:49.274 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000073", 2022-09-12T19:26:49.274 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:8", 2022-09-12T19:26:49.274 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:8", 2022-09-12T19:26:49.275 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000038", 2022-09-12T19:26:49.275 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:6", 2022-09-12T19:26:49.275 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:1", 2022-09-12T19:26:49.275 INFO:teuthology.orchestra.run.smithi138.stdout: "data_log.3.uKcoBtOeJ3y.0", 2022-09-12T19:26:49.276 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:10", 2022-09-12T19:26:49.276 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:0", 2022-09-12T19:26:49.276 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.10", 2022-09-12T19:26:49.277 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:2", 2022-09-12T19:26:49.277 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000118", 2022-09-12T19:26:49.277 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:8", 2022-09-12T19:26:49.277 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:6", 2022-09-12T19:26:49.278 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:10", 2022-09-12T19:26:49.278 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:6", 2022-09-12T19:26:49.278 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:4", 2022-09-12T19:26:49.278 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:1", 2022-09-12T19:26:49.279 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:2", 2022-09-12T19:26:49.279 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000036", 2022-09-12T19:26:49.279 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:9", 2022-09-12T19:26:49.279 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000103", 2022-09-12T19:26:49.280 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:10", 2022-09-12T19:26:49.280 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:8", 2022-09-12T19:26:49.280 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:2", 2022-09-12T19:26:49.281 INFO:teuthology.orchestra.run.smithi138.stdout: "mdlog.sync-status.shard.3", 2022-09-12T19:26:49.281 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:6", 2022-09-12T19:26:49.281 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:3", 2022-09-12T19:26:49.281 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:6", 2022-09-12T19:26:49.282 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000119", 2022-09-12T19:26:49.282 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000098", 2022-09-12T19:26:49.282 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:10", 2022-09-12T19:26:49.282 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:3", 2022-09-12T19:26:49.283 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:1", 2022-09-12T19:26:49.283 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:5", 2022-09-12T19:26:49.283 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:3", 2022-09-12T19:26:49.283 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38", 2022-09-12T19:26:49.284 INFO:teuthology.orchestra.run.smithi138.stdout: "mdlog.sync-status.shard.0", 2022-09-12T19:26:49.284 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:6", 2022-09-12T19:26:49.284 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000027", 2022-09-12T19:26:49.284 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:4", 2022-09-12T19:26:49.285 INFO:teuthology.orchestra.run.smithi138.stdout: "mdlog.sync-status", 2022-09-12T19:26:49.285 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000019", 2022-09-12T19:26:49.285 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:5", 2022-09-12T19:26:49.285 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:5", 2022-09-12T19:26:49.286 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000039", 2022-09-12T19:26:49.286 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.shard.823fb3a5-4d8a-49dc-ae64-e05804bbea94.3.retry", 2022-09-12T19:26:49.286 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:1", 2022-09-12T19:26:49.286 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:26:49.287 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:4", 2022-09-12T19:26:49.287 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:9", 2022-09-12T19:26:49.287 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:4", 2022-09-12T19:26:49.288 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:1", 2022-09-12T19:26:49.288 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:8", 2022-09-12T19:26:49.288 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:8", 2022-09-12T19:26:49.288 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:2", 2022-09-12T19:26:49.289 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:10", 2022-09-12T19:26:49.289 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:9", 2022-09-12T19:26:49.289 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:7", 2022-09-12T19:26:49.289 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:8", 2022-09-12T19:26:49.290 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:8", 2022-09-12T19:26:49.290 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:7", 2022-09-12T19:26:49.290 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:1", 2022-09-12T19:26:49.290 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000100", 2022-09-12T19:26:49.291 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:8", 2022-09-12T19:26:49.291 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:8", 2022-09-12T19:26:49.291 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:4", 2022-09-12T19:26:49.291 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:0", 2022-09-12T19:26:49.292 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:4", 2022-09-12T19:26:49.292 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:1", 2022-09-12T19:26:49.292 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:6", 2022-09-12T19:26:49.292 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:6", 2022-09-12T19:26:49.293 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000093", 2022-09-12T19:26:49.293 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94", 2022-09-12T19:26:49.293 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37", 2022-09-12T19:26:49.294 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:8", 2022-09-12T19:26:49.294 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:10", 2022-09-12T19:26:49.294 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:3", 2022-09-12T19:26:49.294 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:4", 2022-09-12T19:26:49.295 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:9", 2022-09-12T19:26:49.295 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:2", 2022-09-12T19:26:49.295 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000004", 2022-09-12T19:26:49.295 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000063", 2022-09-12T19:26:49.296 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:7", 2022-09-12T19:26:49.296 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000122", 2022-09-12T19:26:49.296 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:7", 2022-09-12T19:26:49.296 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000057", 2022-09-12T19:26:49.297 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000054", 2022-09-12T19:26:49.297 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000114", 2022-09-12T19:26:49.297 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:7", 2022-09-12T19:26:49.297 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15", 2022-09-12T19:26:49.298 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:7", 2022-09-12T19:26:49.298 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.2", 2022-09-12T19:26:49.298 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:4", 2022-09-12T19:26:49.299 INFO:teuthology.orchestra.run.smithi138.stdout: "data_log.0", 2022-09-12T19:26:49.299 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:3", 2022-09-12T19:26:49.299 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:7", 2022-09-12T19:26:49.299 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:8", 2022-09-12T19:26:49.300 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000012", 2022-09-12T19:26:49.300 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:7", 2022-09-12T19:26:49.300 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:10", 2022-09-12T19:26:49.300 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:7", 2022-09-12T19:26:49.301 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:9", 2022-09-12T19:26:49.301 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:9", 2022-09-12T19:26:49.301 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:6", 2022-09-12T19:26:49.302 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000084", 2022-09-12T19:26:49.302 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000043", 2022-09-12T19:26:49.302 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5", 2022-09-12T19:26:49.302 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:4", 2022-09-12T19:26:49.303 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:3", 2022-09-12T19:26:49.303 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:0", 2022-09-12T19:26:49.303 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:7", 2022-09-12T19:26:49.304 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:0", 2022-09-12T19:26:49.304 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.11", 2022-09-12T19:26:49.304 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:0", 2022-09-12T19:26:49.304 INFO:teuthology.orchestra.run.smithi138.stdout: "data_log.1.dYjs7wpS3UW.0", 2022-09-12T19:26:49.305 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:7", 2022-09-12T19:26:49.305 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:7", 2022-09-12T19:26:49.305 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:2", 2022-09-12T19:26:49.305 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:10", 2022-09-12T19:26:49.306 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:8", 2022-09-12T19:26:49.306 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:10", 2022-09-12T19:26:49.306 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:4", 2022-09-12T19:26:49.306 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:2", 2022-09-12T19:26:49.307 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000111", 2022-09-12T19:26:49.307 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000048", 2022-09-12T19:26:49.307 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:3", 2022-09-12T19:26:49.307 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000020", 2022-09-12T19:26:49.308 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:3", 2022-09-12T19:26:49.308 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:1", 2022-09-12T19:26:49.308 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10", 2022-09-12T19:26:49.308 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:0", 2022-09-12T19:26:49.309 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000099", 2022-09-12T19:26:49.309 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000056", 2022-09-12T19:26:49.309 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:9", 2022-09-12T19:26:49.309 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36", 2022-09-12T19:26:49.310 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:1", 2022-09-12T19:26:49.310 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:3", 2022-09-12T19:26:49.310 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:7", 2022-09-12T19:26:49.311 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:8", 2022-09-12T19:26:49.311 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:6", 2022-09-12T19:26:49.311 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:6", 2022-09-12T19:26:49.311 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000072", 2022-09-12T19:26:49.312 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000062", 2022-09-12T19:26:49.312 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:1", 2022-09-12T19:26:49.312 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000115", 2022-09-12T19:26:49.312 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.1", 2022-09-12T19:26:49.313 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000033", 2022-09-12T19:26:49.313 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:5", 2022-09-12T19:26:49.313 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:1", 2022-09-12T19:26:49.313 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:8", 2022-09-12T19:26:49.314 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:4", 2022-09-12T19:26:49.314 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:10", 2022-09-12T19:26:49.314 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:2", 2022-09-12T19:26:49.315 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.shard.823fb3a5-4d8a-49dc-ae64-e05804bbea94.2", 2022-09-12T19:26:49.315 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:4", 2022-09-12T19:26:49.315 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:9", 2022-09-12T19:26:49.315 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:10", 2022-09-12T19:26:49.316 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000025", 2022-09-12T19:26:49.316 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:9", 2022-09-12T19:26:49.316 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:7", 2022-09-12T19:26:49.316 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:8", 2022-09-12T19:26:49.317 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000123", 2022-09-12T19:26:49.317 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:2", 2022-09-12T19:26:49.317 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000108", 2022-09-12T19:26:49.317 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:6", 2022-09-12T19:26:49.318 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000094", 2022-09-12T19:26:49.318 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:5", 2022-09-12T19:26:49.318 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:1", 2022-09-12T19:26:49.318 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:1", 2022-09-12T19:26:49.319 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:6", 2022-09-12T19:26:49.319 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.shard.823fb3a5-4d8a-49dc-ae64-e05804bbea94.2.retry", 2022-09-12T19:26:49.319 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000060", 2022-09-12T19:26:49.319 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19", 2022-09-12T19:26:49.320 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:4", 2022-09-12T19:26:49.320 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000109", 2022-09-12T19:26:49.320 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:3", 2022-09-12T19:26:49.320 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:3", 2022-09-12T19:26:49.321 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:1", 2022-09-12T19:26:49.321 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14", 2022-09-12T19:26:49.321 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:5", 2022-09-12T19:26:49.321 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:0", 2022-09-12T19:26:49.322 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000013", 2022-09-12T19:26:49.322 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:1", 2022-09-12T19:26:49.322 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:1", 2022-09-12T19:26:49.322 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:8", 2022-09-12T19:26:49.323 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:8", 2022-09-12T19:26:49.323 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:6", 2022-09-12T19:26:49.323 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:3", 2022-09-12T19:26:49.323 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:6", 2022-09-12T19:26:49.324 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.shard.823fb3a5-4d8a-49dc-ae64-e05804bbea94.0", 2022-09-12T19:26:49.324 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:10", 2022-09-12T19:26:49.324 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:4", 2022-09-12T19:26:49.324 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:8", 2022-09-12T19:26:49.325 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000051", 2022-09-12T19:26:49.325 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:3", 2022-09-12T19:26:49.325 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000106", 2022-09-12T19:26:49.325 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:9", 2022-09-12T19:26:49.326 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:7", 2022-09-12T19:26:49.326 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:4", 2022-09-12T19:26:49.326 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:0", 2022-09-12T19:26:49.327 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:6", 2022-09-12T19:26:49.327 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37", 2022-09-12T19:26:49.327 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:10", 2022-09-12T19:26:49.327 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000017", 2022-09-12T19:26:49.328 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:4", 2022-09-12T19:26:49.328 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.shard.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.3.retry", 2022-09-12T19:26:49.328 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:0", 2022-09-12T19:26:49.329 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:4", 2022-09-12T19:26:49.329 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:9", 2022-09-12T19:26:49.329 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.shard.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.1", 2022-09-12T19:26:49.330 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:0", 2022-09-12T19:26:49.330 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:1", 2022-09-12T19:26:49.330 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17", 2022-09-12T19:26:49.330 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:5", 2022-09-12T19:26:49.331 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000023", 2022-09-12T19:26:49.331 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:0", 2022-09-12T19:26:49.331 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:7", 2022-09-12T19:26:49.331 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:8", 2022-09-12T19:26:49.332 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000107", 2022-09-12T19:26:49.332 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:10", 2022-09-12T19:26:49.332 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:0", 2022-09-12T19:26:49.333 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000046", 2022-09-12T19:26:49.333 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:3", 2022-09-12T19:26:49.333 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:1", 2022-09-12T19:26:49.334 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:2", 2022-09-12T19:26:49.334 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:0", 2022-09-12T19:26:49.335 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:2", 2022-09-12T19:26:49.335 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:3", 2022-09-12T19:26:49.335 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:1", 2022-09-12T19:26:49.335 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:0", 2022-09-12T19:26:49.336 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:5", 2022-09-12T19:26:49.336 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:26:49.336 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:9", 2022-09-12T19:26:49.336 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:3", 2022-09-12T19:26:49.337 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:6", 2022-09-12T19:26:49.337 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:9", 2022-09-12T19:26:49.337 INFO:teuthology.orchestra.run.smithi138.stdout: "data_log.2", 2022-09-12T19:26:49.338 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:10", 2022-09-12T19:26:49.338 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:1", 2022-09-12T19:26:49.338 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:4", 2022-09-12T19:26:49.338 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000067", 2022-09-12T19:26:49.339 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.shard.823fb3a5-4d8a-49dc-ae64-e05804bbea94.1", 2022-09-12T19:26:49.339 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:3", 2022-09-12T19:26:49.339 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:9", 2022-09-12T19:26:49.340 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:3", 2022-09-12T19:26:49.340 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:5", 2022-09-12T19:26:49.340 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:1", 2022-09-12T19:26:49.340 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:9", 2022-09-12T19:26:49.341 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:4", 2022-09-12T19:26:49.341 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:5", 2022-09-12T19:26:49.341 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:5", 2022-09-12T19:26:49.341 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.9", 2022-09-12T19:26:49.342 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000086", 2022-09-12T19:26:49.342 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:2", 2022-09-12T19:26:49.342 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.5", 2022-09-12T19:26:49.342 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:10", 2022-09-12T19:26:49.343 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:7", 2022-09-12T19:26:49.343 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:10", 2022-09-12T19:26:49.343 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:3", 2022-09-12T19:26:49.343 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:4", 2022-09-12T19:26:49.344 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:9", 2022-09-12T19:26:49.344 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:4", 2022-09-12T19:26:49.344 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1", 2022-09-12T19:26:49.344 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:6", 2022-09-12T19:26:49.345 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:7", 2022-09-12T19:26:49.345 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:10", 2022-09-12T19:26:49.345 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:8", 2022-09-12T19:26:49.345 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:6", 2022-09-12T19:26:49.346 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:2", 2022-09-12T19:26:49.346 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18", 2022-09-12T19:26:49.346 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:10", 2022-09-12T19:26:49.347 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:2", 2022-09-12T19:26:49.347 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:2", 2022-09-12T19:26:49.347 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:3", 2022-09-12T19:26:49.347 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12", 2022-09-12T19:26:49.348 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:2", 2022-09-12T19:26:49.348 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:1", 2022-09-12T19:26:49.348 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:26:49.348 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:4", 2022-09-12T19:26:49.349 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12", 2022-09-12T19:26:49.349 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:4", 2022-09-12T19:26:49.349 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:4", 2022-09-12T19:26:49.349 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:2", 2022-09-12T19:26:49.350 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:4", 2022-09-12T19:26:49.350 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000050", 2022-09-12T19:26:49.350 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000102", 2022-09-12T19:26:49.350 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:7", 2022-09-12T19:26:49.351 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:1", 2022-09-12T19:26:49.351 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:4", 2022-09-12T19:26:49.351 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:6", 2022-09-12T19:26:49.351 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:4", 2022-09-12T19:26:49.352 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.shard.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.1.retry", 2022-09-12T19:26:49.352 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:6", 2022-09-12T19:26:49.352 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:7", 2022-09-12T19:26:49.352 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:7", 2022-09-12T19:26:49.353 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:2", 2022-09-12T19:26:49.353 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000124", 2022-09-12T19:26:49.353 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:5", 2022-09-12T19:26:49.353 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:7", 2022-09-12T19:26:49.354 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:1", 2022-09-12T19:26:49.354 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:7", 2022-09-12T19:26:49.354 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:0", 2022-09-12T19:26:49.354 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000001", 2022-09-12T19:26:49.355 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4", 2022-09-12T19:26:49.355 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:9", 2022-09-12T19:26:49.355 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000126", 2022-09-12T19:26:49.355 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3", 2022-09-12T19:26:49.356 INFO:teuthology.orchestra.run.smithi138.stdout: "meta.log.38ca1f57-1358-496c-a99b-df49ee365a4f.2", 2022-09-12T19:26:49.356 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:6", 2022-09-12T19:26:49.356 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:1", 2022-09-12T19:26:49.357 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000055", 2022-09-12T19:26:49.357 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:7", 2022-09-12T19:26:49.357 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000061", 2022-09-12T19:26:49.357 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:5", 2022-09-12T19:26:49.358 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:8", 2022-09-12T19:26:49.358 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:5", 2022-09-12T19:26:49.358 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:9", 2022-09-12T19:26:49.358 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:3", 2022-09-12T19:26:49.359 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:5", 2022-09-12T19:26:49.359 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:10", 2022-09-12T19:26:49.359 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.shard.823fb3a5-4d8a-49dc-ae64-e05804bbea94.3", 2022-09-12T19:26:49.359 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:0", 2022-09-12T19:26:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000097", 2022-09-12T19:26:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.shard.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.0", 2022-09-12T19:26:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:4", 2022-09-12T19:26:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:2", 2022-09-12T19:26:49.361 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:5", 2022-09-12T19:26:49.361 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:3", 2022-09-12T19:26:49.361 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000042", 2022-09-12T19:26:49.361 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:3", 2022-09-12T19:26:49.362 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:1", 2022-09-12T19:26:49.362 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:9", 2022-09-12T19:26:49.362 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:8", 2022-09-12T19:26:49.362 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:7", 2022-09-12T19:26:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000075", 2022-09-12T19:26:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000120", 2022-09-12T19:26:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000030", 2022-09-12T19:26:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18", 2022-09-12T19:26:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:5", 2022-09-12T19:26:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:0", 2022-09-12T19:26:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:10", 2022-09-12T19:26:49.365 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:10", 2022-09-12T19:26:49.365 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.823fb3a5-4d8a-49dc-ae64-e05804bbea94:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:0" 2022-09-12T19:26:49.365 INFO:teuthology.orchestra.run.smithi138.stdout:] 2022-09-12T19:26:49.366 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T19:26:49.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:49.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:49.368 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:49.418 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:49.419 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:49.482 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:26:49.483 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:4", 2022-09-12T19:26:49.483 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:6", 2022-09-12T19:26:49.483 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000078", 2022-09-12T19:26:49.483 INFO:teuthology.orchestra.run.smithi035.stdout: "meta.history", 2022-09-12T19:26:49.484 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:7", 2022-09-12T19:26:49.484 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:3", 2022-09-12T19:26:49.484 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:0", 2022-09-12T19:26:49.484 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3", 2022-09-12T19:26:49.485 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:3", 2022-09-12T19:26:49.485 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000070", 2022-09-12T19:26:49.485 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000104", 2022-09-12T19:26:49.485 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:0", 2022-09-12T19:26:49.486 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:10", 2022-09-12T19:26:49.486 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:2", 2022-09-12T19:26:49.486 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000026", 2022-09-12T19:26:49.487 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.fdf3b910-2ea3-4164-a3f8-b3e146af1c44.0", 2022-09-12T19:26:49.487 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:5", 2022-09-12T19:26:49.487 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:6", 2022-09-12T19:26:49.487 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:9", 2022-09-12T19:26:49.488 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:5", 2022-09-12T19:26:49.488 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:3", 2022-09-12T19:26:49.488 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:2", 2022-09-12T19:26:49.488 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.8", 2022-09-12T19:26:49.489 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000028", 2022-09-12T19:26:49.489 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:3", 2022-09-12T19:26:49.489 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:8", 2022-09-12T19:26:49.489 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1", 2022-09-12T19:26:49.490 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:10", 2022-09-12T19:26:49.490 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:9", 2022-09-12T19:26:49.490 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:6", 2022-09-12T19:26:49.490 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13", 2022-09-12T19:26:49.491 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:4", 2022-09-12T19:26:49.491 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:0", 2022-09-12T19:26:49.491 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:3", 2022-09-12T19:26:49.491 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:1", 2022-09-12T19:26:49.491 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:2", 2022-09-12T19:26:49.492 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:9", 2022-09-12T19:26:49.492 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:5", 2022-09-12T19:26:49.492 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000040", 2022-09-12T19:26:49.492 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:8", 2022-09-12T19:26:49.493 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.6", 2022-09-12T19:26:49.493 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:26:49.493 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:2", 2022-09-12T19:26:49.493 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:5", 2022-09-12T19:26:49.494 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23", 2022-09-12T19:26:49.494 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:0", 2022-09-12T19:26:49.494 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000015", 2022-09-12T19:26:49.494 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:4", 2022-09-12T19:26:49.495 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:3", 2022-09-12T19:26:49.495 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:0", 2022-09-12T19:26:49.495 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:1", 2022-09-12T19:26:49.495 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:7", 2022-09-12T19:26:49.496 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000069", 2022-09-12T19:26:49.496 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:4", 2022-09-12T19:26:49.496 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:1", 2022-09-12T19:26:49.497 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:2", 2022-09-12T19:26:49.497 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:1", 2022-09-12T19:26:49.497 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000095", 2022-09-12T19:26:49.497 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000003", 2022-09-12T19:26:49.497 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000047", 2022-09-12T19:26:49.498 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.fdf3b910-2ea3-4164-a3f8-b3e146af1c44.3", 2022-09-12T19:26:49.498 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:6", 2022-09-12T19:26:49.498 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38", 2022-09-12T19:26:49.498 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:6", 2022-09-12T19:26:49.499 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:2", 2022-09-12T19:26:49.499 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:5", 2022-09-12T19:26:49.499 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:3", 2022-09-12T19:26:49.499 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:9", 2022-09-12T19:26:49.500 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:9", 2022-09-12T19:26:49.500 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000035", 2022-09-12T19:26:49.500 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:2", 2022-09-12T19:26:49.500 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:10", 2022-09-12T19:26:49.501 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000037", 2022-09-12T19:26:49.501 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:0", 2022-09-12T19:26:49.501 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:5", 2022-09-12T19:26:49.501 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:5", 2022-09-12T19:26:49.502 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:0", 2022-09-12T19:26:49.502 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:6", 2022-09-12T19:26:49.502 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:10", 2022-09-12T19:26:49.502 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.0.retry", 2022-09-12T19:26:49.503 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:10", 2022-09-12T19:26:49.503 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:0", 2022-09-12T19:26:49.503 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:10", 2022-09-12T19:26:49.503 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:1", 2022-09-12T19:26:49.504 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000024", 2022-09-12T19:26:49.504 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:3", 2022-09-12T19:26:49.504 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:1", 2022-09-12T19:26:49.504 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:2", 2022-09-12T19:26:49.505 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:0", 2022-09-12T19:26:49.505 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:5", 2022-09-12T19:26:49.505 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000000", 2022-09-12T19:26:49.505 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000031", 2022-09-12T19:26:49.506 INFO:teuthology.orchestra.run.smithi035.stdout: "meta.log.c0f7a512-c940-4d1f-8adf-3484630f876b.3", 2022-09-12T19:26:49.506 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:0", 2022-09-12T19:26:49.506 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:9", 2022-09-12T19:26:49.507 INFO:teuthology.orchestra.run.smithi035.stdout: "data_log.3", 2022-09-12T19:26:49.507 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:8", 2022-09-12T19:26:49.507 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:2", 2022-09-12T19:26:49.507 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:10", 2022-09-12T19:26:49.508 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:1", 2022-09-12T19:26:49.508 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:10", 2022-09-12T19:26:49.508 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:4", 2022-09-12T19:26:49.508 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:3", 2022-09-12T19:26:49.509 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:9", 2022-09-12T19:26:49.509 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:2", 2022-09-12T19:26:49.509 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:3", 2022-09-12T19:26:49.509 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:5", 2022-09-12T19:26:49.510 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:6", 2022-09-12T19:26:49.510 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:0", 2022-09-12T19:26:49.510 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:6", 2022-09-12T19:26:49.510 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:3", 2022-09-12T19:26:49.511 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:7", 2022-09-12T19:26:49.511 INFO:teuthology.orchestra.run.smithi035.stdout: "data_loggenerations_metadata", 2022-09-12T19:26:49.511 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14", 2022-09-12T19:26:49.511 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:4", 2022-09-12T19:26:49.512 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:7", 2022-09-12T19:26:49.512 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:6", 2022-09-12T19:26:49.512 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:1", 2022-09-12T19:26:49.513 INFO:teuthology.orchestra.run.smithi035.stdout: "data_log.0.WDlgquN6NAw.0", 2022-09-12T19:26:49.513 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:8", 2022-09-12T19:26:49.513 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:8", 2022-09-12T19:26:49.514 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:1", 2022-09-12T19:26:49.514 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:5", 2022-09-12T19:26:49.514 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000076", 2022-09-12T19:26:49.515 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:1", 2022-09-12T19:26:49.515 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000113", 2022-09-12T19:26:49.515 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000005", 2022-09-12T19:26:49.515 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000011", 2022-09-12T19:26:49.516 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:9", 2022-09-12T19:26:49.516 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000058", 2022-09-12T19:26:49.516 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:0", 2022-09-12T19:26:49.516 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:10", 2022-09-12T19:26:49.516 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:3", 2022-09-12T19:26:49.517 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000052", 2022-09-12T19:26:49.517 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:2", 2022-09-12T19:26:49.517 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1", 2022-09-12T19:26:49.518 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:9", 2022-09-12T19:26:49.518 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:9", 2022-09-12T19:26:49.518 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:7", 2022-09-12T19:26:49.519 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:4", 2022-09-12T19:26:49.519 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:8", 2022-09-12T19:26:49.519 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:9", 2022-09-12T19:26:49.519 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:1", 2022-09-12T19:26:49.520 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37", 2022-09-12T19:26:49.520 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000088", 2022-09-12T19:26:49.520 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:9", 2022-09-12T19:26:49.520 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36", 2022-09-12T19:26:49.521 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000080", 2022-09-12T19:26:49.521 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000090", 2022-09-12T19:26:49.521 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:10", 2022-09-12T19:26:49.521 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000110", 2022-09-12T19:26:49.522 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:2", 2022-09-12T19:26:49.522 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:26:49.522 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:4", 2022-09-12T19:26:49.522 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:1", 2022-09-12T19:26:49.523 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:7", 2022-09-12T19:26:49.523 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:5", 2022-09-12T19:26:49.523 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.fdf3b910-2ea3-4164-a3f8-b3e146af1c44.1", 2022-09-12T19:26:49.523 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:3", 2022-09-12T19:26:49.524 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:2", 2022-09-12T19:26:49.524 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000096", 2022-09-12T19:26:49.524 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:0", 2022-09-12T19:26:49.524 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:6", 2022-09-12T19:26:49.525 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15", 2022-09-12T19:26:49.525 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:8", 2022-09-12T19:26:49.525 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:0", 2022-09-12T19:26:49.525 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:2", 2022-09-12T19:26:49.526 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000087", 2022-09-12T19:26:49.526 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:1", 2022-09-12T19:26:49.526 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:9", 2022-09-12T19:26:49.527 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000008", 2022-09-12T19:26:49.527 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:8", 2022-09-12T19:26:49.527 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:6", 2022-09-12T19:26:49.527 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:8", 2022-09-12T19:26:49.528 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:4", 2022-09-12T19:26:49.528 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:8", 2022-09-12T19:26:49.528 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:5", 2022-09-12T19:26:49.528 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:9", 2022-09-12T19:26:49.529 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:4", 2022-09-12T19:26:49.529 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:8", 2022-09-12T19:26:49.529 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:1", 2022-09-12T19:26:49.529 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:2", 2022-09-12T19:26:49.530 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000006", 2022-09-12T19:26:49.530 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000029", 2022-09-12T19:26:49.530 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:4", 2022-09-12T19:26:49.530 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:10", 2022-09-12T19:26:49.531 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000089", 2022-09-12T19:26:49.531 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000022", 2022-09-12T19:26:49.531 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21", 2022-09-12T19:26:49.531 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:0", 2022-09-12T19:26:49.532 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:4", 2022-09-12T19:26:49.532 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.fdf3b910-2ea3-4164-a3f8-b3e146af1c44.2.retry", 2022-09-12T19:26:49.532 INFO:teuthology.orchestra.run.smithi035.stdout: "bilog.trim", 2022-09-12T19:26:49.532 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000016", 2022-09-12T19:26:49.533 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:3", 2022-09-12T19:26:49.533 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:0", 2022-09-12T19:26:49.533 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000041", 2022-09-12T19:26:49.533 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:3", 2022-09-12T19:26:49.533 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:9", 2022-09-12T19:26:49.534 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:8", 2022-09-12T19:26:49.534 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000018", 2022-09-12T19:26:49.534 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:7", 2022-09-12T19:26:49.534 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:7", 2022-09-12T19:26:49.535 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000092", 2022-09-12T19:26:49.535 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:4", 2022-09-12T19:26:49.535 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:9", 2022-09-12T19:26:49.535 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:8", 2022-09-12T19:26:49.536 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22", 2022-09-12T19:26:49.536 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:3", 2022-09-12T19:26:49.536 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:4", 2022-09-12T19:26:49.537 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:10", 2022-09-12T19:26:49.537 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.4", 2022-09-12T19:26:49.537 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:8", 2022-09-12T19:26:49.538 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:9", 2022-09-12T19:26:49.538 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:7", 2022-09-12T19:26:49.538 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5", 2022-09-12T19:26:49.538 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:1", 2022-09-12T19:26:49.539 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:10", 2022-09-12T19:26:49.539 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000014", 2022-09-12T19:26:49.539 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:1", 2022-09-12T19:26:49.539 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:5", 2022-09-12T19:26:49.540 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:9", 2022-09-12T19:26:49.540 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:9", 2022-09-12T19:26:49.540 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:7", 2022-09-12T19:26:49.540 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000112", 2022-09-12T19:26:49.541 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:7", 2022-09-12T19:26:49.541 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000007", 2022-09-12T19:26:49.541 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000021", 2022-09-12T19:26:49.541 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000064", 2022-09-12T19:26:49.542 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:6", 2022-09-12T19:26:49.542 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:0", 2022-09-12T19:26:49.542 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:5", 2022-09-12T19:26:49.542 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:4", 2022-09-12T19:26:49.543 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17", 2022-09-12T19:26:49.543 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:8", 2022-09-12T19:26:49.543 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12", 2022-09-12T19:26:49.543 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14", 2022-09-12T19:26:49.544 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000071", 2022-09-12T19:26:49.544 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:5", 2022-09-12T19:26:49.544 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000074", 2022-09-12T19:26:49.544 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:7", 2022-09-12T19:26:49.545 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16", 2022-09-12T19:26:49.545 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:0", 2022-09-12T19:26:49.545 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:2", 2022-09-12T19:26:49.546 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:5", 2022-09-12T19:26:49.546 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:10", 2022-09-12T19:26:49.546 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:3", 2022-09-12T19:26:49.547 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:7", 2022-09-12T19:26:49.547 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:10", 2022-09-12T19:26:49.547 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:0", 2022-09-12T19:26:49.547 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:2", 2022-09-12T19:26:49.548 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:2", 2022-09-12T19:26:49.548 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:1", 2022-09-12T19:26:49.548 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000081", 2022-09-12T19:26:49.548 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:8", 2022-09-12T19:26:49.549 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000009", 2022-09-12T19:26:49.549 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:5", 2022-09-12T19:26:49.549 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:0", 2022-09-12T19:26:49.549 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000121", 2022-09-12T19:26:49.550 INFO:teuthology.orchestra.run.smithi035.stdout: "data_log.1", 2022-09-12T19:26:49.550 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:9", 2022-09-12T19:26:49.550 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:5", 2022-09-12T19:26:49.551 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11", 2022-09-12T19:26:49.551 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:8", 2022-09-12T19:26:49.551 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:1", 2022-09-12T19:26:49.551 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:8", 2022-09-12T19:26:49.552 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000125", 2022-09-12T19:26:49.552 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000082", 2022-09-12T19:26:49.552 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:9", 2022-09-12T19:26:49.552 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:0", 2022-09-12T19:26:49.553 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22", 2022-09-12T19:26:49.553 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:0", 2022-09-12T19:26:49.553 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:2", 2022-09-12T19:26:49.553 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:0", 2022-09-12T19:26:49.554 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:9", 2022-09-12T19:26:49.554 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:4", 2022-09-12T19:26:49.554 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000105", 2022-09-12T19:26:49.555 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000059", 2022-09-12T19:26:49.555 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:4", 2022-09-12T19:26:49.555 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:2", 2022-09-12T19:26:49.555 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:4", 2022-09-12T19:26:49.556 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:3", 2022-09-12T19:26:49.556 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.3", 2022-09-12T19:26:49.556 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000077", 2022-09-12T19:26:49.556 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:5", 2022-09-12T19:26:49.557 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000032", 2022-09-12T19:26:49.557 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:8", 2022-09-12T19:26:49.557 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:3", 2022-09-12T19:26:49.557 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:6", 2022-09-12T19:26:49.558 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:0", 2022-09-12T19:26:49.558 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:1", 2022-09-12T19:26:49.558 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:10", 2022-09-12T19:26:49.558 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:4", 2022-09-12T19:26:49.559 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:4", 2022-09-12T19:26:49.559 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:10", 2022-09-12T19:26:49.559 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:2", 2022-09-12T19:26:49.559 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000053", 2022-09-12T19:26:49.560 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:4", 2022-09-12T19:26:49.560 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000091", 2022-09-12T19:26:49.560 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:8", 2022-09-12T19:26:49.561 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:8", 2022-09-12T19:26:49.561 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000065", 2022-09-12T19:26:49.561 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:0", 2022-09-12T19:26:49.561 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:5", 2022-09-12T19:26:49.562 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000083", 2022-09-12T19:26:49.562 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000010", 2022-09-12T19:26:49.562 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:1", 2022-09-12T19:26:49.563 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:0", 2022-09-12T19:26:49.563 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:8", 2022-09-12T19:26:49.563 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:10", 2022-09-12T19:26:49.563 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:6", 2022-09-12T19:26:49.564 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:0", 2022-09-12T19:26:49.564 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:9", 2022-09-12T19:26:49.564 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:8", 2022-09-12T19:26:49.564 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:10", 2022-09-12T19:26:49.565 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:7", 2022-09-12T19:26:49.565 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:4", 2022-09-12T19:26:49.565 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:0", 2022-09-12T19:26:49.565 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21", 2022-09-12T19:26:49.566 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.7", 2022-09-12T19:26:49.566 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.3", 2022-09-12T19:26:49.567 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:8", 2022-09-12T19:26:49.567 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:26:49.568 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:3", 2022-09-12T19:26:49.568 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:0", 2022-09-12T19:26:49.569 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:4", 2022-09-12T19:26:49.569 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:1", 2022-09-12T19:26:49.570 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:10", 2022-09-12T19:26:49.570 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:9", 2022-09-12T19:26:49.571 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:4", 2022-09-12T19:26:49.571 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:3", 2022-09-12T19:26:49.572 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:2", 2022-09-12T19:26:49.572 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:7", 2022-09-12T19:26:49.573 INFO:teuthology.orchestra.run.smithi035.stdout: "mdlog.sync-status.shard.2", 2022-09-12T19:26:49.574 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:10", 2022-09-12T19:26:49.574 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:3", 2022-09-12T19:26:49.575 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:6", 2022-09-12T19:26:49.575 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:6", 2022-09-12T19:26:49.576 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:10", 2022-09-12T19:26:49.576 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.fdf3b910-2ea3-4164-a3f8-b3e146af1c44.2", 2022-09-12T19:26:49.577 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:3", 2022-09-12T19:26:49.577 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:4", 2022-09-12T19:26:49.578 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:3", 2022-09-12T19:26:49.578 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:10", 2022-09-12T19:26:49.579 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:6", 2022-09-12T19:26:49.579 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:4", 2022-09-12T19:26:49.580 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000045", 2022-09-12T19:26:49.580 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:7", 2022-09-12T19:26:49.581 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:1", 2022-09-12T19:26:49.581 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000002", 2022-09-12T19:26:49.582 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:10", 2022-09-12T19:26:49.582 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:6", 2022-09-12T19:26:49.583 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:6", 2022-09-12T19:26:49.583 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:4", 2022-09-12T19:26:49.584 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000116", 2022-09-12T19:26:49.584 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.2", 2022-09-12T19:26:49.585 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:7", 2022-09-12T19:26:49.585 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:2", 2022-09-12T19:26:49.586 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:7", 2022-09-12T19:26:49.586 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:3", 2022-09-12T19:26:49.587 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:1", 2022-09-12T19:26:49.587 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:7", 2022-09-12T19:26:49.588 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:5", 2022-09-12T19:26:49.588 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000034", 2022-09-12T19:26:49.589 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:5", 2022-09-12T19:26:49.589 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44", 2022-09-12T19:26:49.590 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:7", 2022-09-12T19:26:49.590 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000101", 2022-09-12T19:26:49.591 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:8", 2022-09-12T19:26:49.591 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:7", 2022-09-12T19:26:49.592 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:5", 2022-09-12T19:26:49.592 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:8", 2022-09-12T19:26:49.593 INFO:teuthology.orchestra.run.smithi035.stdout: "mdlog.sync-status.shard.1", 2022-09-12T19:26:49.593 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:4", 2022-09-12T19:26:49.594 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:4", 2022-09-12T19:26:49.594 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:3", 2022-09-12T19:26:49.595 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:5", 2022-09-12T19:26:49.595 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.2.retry", 2022-09-12T19:26:49.595 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:2", 2022-09-12T19:26:49.595 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:1", 2022-09-12T19:26:49.596 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000079", 2022-09-12T19:26:49.596 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000049", 2022-09-12T19:26:49.596 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:7", 2022-09-12T19:26:49.597 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:3", 2022-09-12T19:26:49.597 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000117", 2022-09-12T19:26:49.597 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000044", 2022-09-12T19:26:49.597 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:3", 2022-09-12T19:26:49.598 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:26:49.598 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:9", 2022-09-12T19:26:49.598 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:9", 2022-09-12T19:26:49.598 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:2", 2022-09-12T19:26:49.599 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:5", 2022-09-12T19:26:49.599 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000066", 2022-09-12T19:26:49.599 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:8", 2022-09-12T19:26:49.600 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:26:49.600 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:7", 2022-09-12T19:26:49.600 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:7", 2022-09-12T19:26:49.600 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:8", 2022-09-12T19:26:49.601 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:4", 2022-09-12T19:26:49.601 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:8", 2022-09-12T19:26:49.601 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:9", 2022-09-12T19:26:49.601 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000068", 2022-09-12T19:26:49.602 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000085", 2022-09-12T19:26:49.602 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:0", 2022-09-12T19:26:49.602 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:9", 2022-09-12T19:26:49.602 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:8", 2022-09-12T19:26:49.603 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:4", 2022-09-12T19:26:49.603 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:2", 2022-09-12T19:26:49.603 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36", 2022-09-12T19:26:49.603 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:10", 2022-09-12T19:26:49.604 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:5", 2022-09-12T19:26:49.604 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000073", 2022-09-12T19:26:49.604 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:8", 2022-09-12T19:26:49.604 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000038", 2022-09-12T19:26:49.605 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:3", 2022-09-12T19:26:49.605 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:0", 2022-09-12T19:26:49.605 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:5", 2022-09-12T19:26:49.605 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.10", 2022-09-12T19:26:49.606 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000118", 2022-09-12T19:26:49.606 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:10", 2022-09-12T19:26:49.606 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:6", 2022-09-12T19:26:49.606 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000036", 2022-09-12T19:26:49.607 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:2", 2022-09-12T19:26:49.607 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:8", 2022-09-12T19:26:49.607 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:0", 2022-09-12T19:26:49.607 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000103", 2022-09-12T19:26:49.608 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:8", 2022-09-12T19:26:49.608 INFO:teuthology.orchestra.run.smithi035.stdout: "mdlog.sync-status.shard.3", 2022-09-12T19:26:49.608 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:7", 2022-09-12T19:26:49.608 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:6", 2022-09-12T19:26:49.609 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:0", 2022-09-12T19:26:49.609 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:9", 2022-09-12T19:26:49.609 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:6", 2022-09-12T19:26:49.610 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.fdf3b910-2ea3-4164-a3f8-b3e146af1c44.1.retry", 2022-09-12T19:26:49.610 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000119", 2022-09-12T19:26:49.610 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:5", 2022-09-12T19:26:49.610 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000098", 2022-09-12T19:26:49.611 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:1", 2022-09-12T19:26:49.611 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:5", 2022-09-12T19:26:49.611 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:5", 2022-09-12T19:26:49.611 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:3", 2022-09-12T19:26:49.612 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:1", 2022-09-12T19:26:49.612 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38", 2022-09-12T19:26:49.612 INFO:teuthology.orchestra.run.smithi035.stdout: "mdlog.sync-status.shard.0", 2022-09-12T19:26:49.612 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000027", 2022-09-12T19:26:49.613 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:6", 2022-09-12T19:26:49.613 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:0", 2022-09-12T19:26:49.613 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:4", 2022-09-12T19:26:49.613 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:3", 2022-09-12T19:26:49.614 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:7", 2022-09-12T19:26:49.614 INFO:teuthology.orchestra.run.smithi035.stdout: "mdlog.sync-status", 2022-09-12T19:26:49.614 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000019", 2022-09-12T19:26:49.614 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:5", 2022-09-12T19:26:49.615 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:5", 2022-09-12T19:26:49.615 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:8", 2022-09-12T19:26:49.615 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000039", 2022-09-12T19:26:49.615 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:1", 2022-09-12T19:26:49.616 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:4", 2022-09-12T19:26:49.616 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:9", 2022-09-12T19:26:49.616 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:6", 2022-09-12T19:26:49.616 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:4", 2022-09-12T19:26:49.617 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:1", 2022-09-12T19:26:49.617 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:2", 2022-09-12T19:26:49.617 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:10", 2022-09-12T19:26:49.617 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:9", 2022-09-12T19:26:49.618 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:9", 2022-09-12T19:26:49.618 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:7", 2022-09-12T19:26:49.618 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:7", 2022-09-12T19:26:49.618 INFO:teuthology.orchestra.run.smithi035.stdout: "meta.log.2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e.2", 2022-09-12T19:26:49.619 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000100", 2022-09-12T19:26:49.619 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:9", 2022-09-12T19:26:49.619 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:2", 2022-09-12T19:26:49.619 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:0", 2022-09-12T19:26:49.620 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:1", 2022-09-12T19:26:49.620 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:6", 2022-09-12T19:26:49.620 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:6", 2022-09-12T19:26:49.620 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000093", 2022-09-12T19:26:49.621 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37", 2022-09-12T19:26:49.621 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:3", 2022-09-12T19:26:49.621 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:6", 2022-09-12T19:26:49.621 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:3", 2022-09-12T19:26:49.622 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:2", 2022-09-12T19:26:49.622 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000004", 2022-09-12T19:26:49.622 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000063", 2022-09-12T19:26:49.622 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:7", 2022-09-12T19:26:49.623 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000122", 2022-09-12T19:26:49.623 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:8", 2022-09-12T19:26:49.623 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:6", 2022-09-12T19:26:49.623 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:7", 2022-09-12T19:26:49.624 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000057", 2022-09-12T19:26:49.624 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4", 2022-09-12T19:26:49.624 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13", 2022-09-12T19:26:49.624 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000054", 2022-09-12T19:26:49.625 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000114", 2022-09-12T19:26:49.625 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:7", 2022-09-12T19:26:49.625 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:6", 2022-09-12T19:26:49.625 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.2", 2022-09-12T19:26:49.626 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:10", 2022-09-12T19:26:49.626 INFO:teuthology.orchestra.run.smithi035.stdout: "data_log.0", 2022-09-12T19:26:49.626 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:5", 2022-09-12T19:26:49.627 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:9", 2022-09-12T19:26:49.627 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:7", 2022-09-12T19:26:49.627 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:8", 2022-09-12T19:26:49.627 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000012", 2022-09-12T19:26:49.628 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:10", 2022-09-12T19:26:49.628 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:7", 2022-09-12T19:26:49.628 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:2", 2022-09-12T19:26:49.628 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:9", 2022-09-12T19:26:49.628 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:0", 2022-09-12T19:26:49.629 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:6", 2022-09-12T19:26:49.629 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:0", 2022-09-12T19:26:49.629 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000084", 2022-09-12T19:26:49.629 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000043", 2022-09-12T19:26:49.630 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:3", 2022-09-12T19:26:49.630 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:0", 2022-09-12T19:26:49.630 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:7", 2022-09-12T19:26:49.631 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:7", 2022-09-12T19:26:49.631 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:6", 2022-09-12T19:26:49.631 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:2", 2022-09-12T19:26:49.631 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:10", 2022-09-12T19:26:49.632 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:8", 2022-09-12T19:26:49.632 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:10", 2022-09-12T19:26:49.632 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:0", 2022-09-12T19:26:49.632 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:2", 2022-09-12T19:26:49.633 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:8", 2022-09-12T19:26:49.633 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000111", 2022-09-12T19:26:49.633 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000048", 2022-09-12T19:26:49.633 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:9", 2022-09-12T19:26:49.634 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:3", 2022-09-12T19:26:49.634 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000020", 2022-09-12T19:26:49.634 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:3", 2022-09-12T19:26:49.634 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10", 2022-09-12T19:26:49.635 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:10", 2022-09-12T19:26:49.635 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:6", 2022-09-12T19:26:49.635 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:8", 2022-09-12T19:26:49.635 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000099", 2022-09-12T19:26:49.636 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000056", 2022-09-12T19:26:49.636 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:7", 2022-09-12T19:26:49.636 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:4", 2022-09-12T19:26:49.636 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:10", 2022-09-12T19:26:49.637 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:1", 2022-09-12T19:26:49.637 INFO:teuthology.orchestra.run.smithi035.stdout: "data_log.1.3w+wTS7t7vU.0", 2022-09-12T19:26:49.637 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15", 2022-09-12T19:26:49.637 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000072", 2022-09-12T19:26:49.638 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:1", 2022-09-12T19:26:49.638 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000062", 2022-09-12T19:26:49.638 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:9", 2022-09-12T19:26:49.638 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000115", 2022-09-12T19:26:49.639 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19", 2022-09-12T19:26:49.639 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:10", 2022-09-12T19:26:49.639 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.1", 2022-09-12T19:26:49.639 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000033", 2022-09-12T19:26:49.640 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:8", 2022-09-12T19:26:49.640 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:10", 2022-09-12T19:26:49.640 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:2", 2022-09-12T19:26:49.640 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:10", 2022-09-12T19:26:49.641 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:5", 2022-09-12T19:26:49.641 INFO:teuthology.orchestra.run.smithi035.stdout: "meta.log.c0f7a512-c940-4d1f-8adf-3484630f876b.2", 2022-09-12T19:26:49.641 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:1", 2022-09-12T19:26:49.641 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000025", 2022-09-12T19:26:49.642 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:9", 2022-09-12T19:26:49.642 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:8", 2022-09-12T19:26:49.642 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000123", 2022-09-12T19:26:49.642 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11", 2022-09-12T19:26:49.643 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:4", 2022-09-12T19:26:49.643 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:2", 2022-09-12T19:26:49.643 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:6", 2022-09-12T19:26:49.643 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000108", 2022-09-12T19:26:49.644 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:6", 2022-09-12T19:26:49.644 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000094", 2022-09-12T19:26:49.644 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:5", 2022-09-12T19:26:49.644 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:5", 2022-09-12T19:26:49.645 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:1", 2022-09-12T19:26:49.645 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:8", 2022-09-12T19:26:49.645 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:6", 2022-09-12T19:26:49.645 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10", 2022-09-12T19:26:49.646 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000060", 2022-09-12T19:26:49.646 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:7", 2022-09-12T19:26:49.646 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19", 2022-09-12T19:26:49.647 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:4", 2022-09-12T19:26:49.647 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:4", 2022-09-12T19:26:49.647 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000109", 2022-09-12T19:26:49.647 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:5", 2022-09-12T19:26:49.648 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:8", 2022-09-12T19:26:49.648 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:3", 2022-09-12T19:26:49.648 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:7", 2022-09-12T19:26:49.648 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:1", 2022-09-12T19:26:49.649 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000013", 2022-09-12T19:26:49.649 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:1", 2022-09-12T19:26:49.649 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:3", 2022-09-12T19:26:49.649 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:7", 2022-09-12T19:26:49.649 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:1", 2022-09-12T19:26:49.650 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5", 2022-09-12T19:26:49.650 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:7", 2022-09-12T19:26:49.650 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:6", 2022-09-12T19:26:49.651 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:3", 2022-09-12T19:26:49.651 INFO:teuthology.orchestra.run.smithi035.stdout: "data_log.3.fTsyz9ZnUPa.0", 2022-09-12T19:26:49.651 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:26:49.651 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:2", 2022-09-12T19:26:49.652 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:4", 2022-09-12T19:26:49.652 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:4", 2022-09-12T19:26:49.652 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:3", 2022-09-12T19:26:49.652 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000051", 2022-09-12T19:26:49.653 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:2", 2022-09-12T19:26:49.653 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000106", 2022-09-12T19:26:49.653 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:7", 2022-09-12T19:26:49.653 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:4", 2022-09-12T19:26:49.654 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:0", 2022-09-12T19:26:49.654 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:10", 2022-09-12T19:26:49.654 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000017", 2022-09-12T19:26:49.654 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:9", 2022-09-12T19:26:49.655 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.3.retry", 2022-09-12T19:26:49.655 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:0", 2022-09-12T19:26:49.655 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.1", 2022-09-12T19:26:49.655 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:1", 2022-09-12T19:26:49.656 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17", 2022-09-12T19:26:49.656 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:5", 2022-09-12T19:26:49.656 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:5", 2022-09-12T19:26:49.656 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000023", 2022-09-12T19:26:49.657 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:7", 2022-09-12T19:26:49.657 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:2", 2022-09-12T19:26:49.657 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:9", 2022-09-12T19:26:49.657 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000107", 2022-09-12T19:26:49.658 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:10", 2022-09-12T19:26:49.658 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:0", 2022-09-12T19:26:49.658 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000046", 2022-09-12T19:26:49.658 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:10", 2022-09-12T19:26:49.659 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:7", 2022-09-12T19:26:49.659 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:2", 2022-09-12T19:26:49.659 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:0", 2022-09-12T19:26:49.659 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:6", 2022-09-12T19:26:49.660 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:3", 2022-09-12T19:26:49.660 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:1", 2022-09-12T19:26:49.660 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:0", 2022-09-12T19:26:49.660 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:5", 2022-09-12T19:26:49.661 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:26:49.661 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:3", 2022-09-12T19:26:49.661 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:6", 2022-09-12T19:26:49.661 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:10", 2022-09-12T19:26:49.662 INFO:teuthology.orchestra.run.smithi035.stdout: "data_log.2", 2022-09-12T19:26:49.662 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:10", 2022-09-12T19:26:49.662 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:1", 2022-09-12T19:26:49.662 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000067", 2022-09-12T19:26:49.663 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:10", 2022-09-12T19:26:49.663 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:9", 2022-09-12T19:26:49.663 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:2", 2022-09-12T19:26:49.663 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:5", 2022-09-12T19:26:49.664 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:5", 2022-09-12T19:26:49.664 INFO:teuthology.orchestra.run.smithi035.stdout: "data_log.2.TAizdD4mJ/P.0", 2022-09-12T19:26:49.664 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18", 2022-09-12T19:26:49.664 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:4", 2022-09-12T19:26:49.665 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:8", 2022-09-12T19:26:49.665 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:5", 2022-09-12T19:26:49.665 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:5", 2022-09-12T19:26:49.665 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.9", 2022-09-12T19:26:49.666 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000086", 2022-09-12T19:26:49.666 INFO:teuthology.orchestra.run.smithi035.stdout: "meta.log.c0f7a512-c940-4d1f-8adf-3484630f876b.0", 2022-09-12T19:26:49.666 INFO:teuthology.orchestra.run.smithi035.stdout: "sync.error-log.5", 2022-09-12T19:26:49.667 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:9", 2022-09-12T19:26:49.667 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:7", 2022-09-12T19:26:49.667 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:10", 2022-09-12T19:26:49.667 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:7", 2022-09-12T19:26:49.667 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:4", 2022-09-12T19:26:49.668 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:9", 2022-09-12T19:26:49.668 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16", 2022-09-12T19:26:49.668 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:6", 2022-09-12T19:26:49.668 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:6", 2022-09-12T19:26:49.669 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:6", 2022-09-12T19:26:49.669 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:2", 2022-09-12T19:26:49.669 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18", 2022-09-12T19:26:49.670 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:2", 2022-09-12T19:26:49.670 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:3", 2022-09-12T19:26:49.670 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:2", 2022-09-12T19:26:49.670 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:3", 2022-09-12T19:26:49.671 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12", 2022-09-12T19:26:49.671 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:2", 2022-09-12T19:26:49.671 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:1", 2022-09-12T19:26:49.671 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:4", 2022-09-12T19:26:49.672 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:4", 2022-09-12T19:26:49.672 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:6", 2022-09-12T19:26:49.672 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:4", 2022-09-12T19:26:49.672 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:2", 2022-09-12T19:26:49.673 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000050", 2022-09-12T19:26:49.673 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:3", 2022-09-12T19:26:49.673 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000102", 2022-09-12T19:26:49.673 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:8", 2022-09-12T19:26:49.674 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:1", 2022-09-12T19:26:49.674 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:10", 2022-09-12T19:26:49.674 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:6", 2022-09-12T19:26:49.674 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:8", 2022-09-12T19:26:49.675 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:5", 2022-09-12T19:26:49.675 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:6", 2022-09-12T19:26:49.675 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:7", 2022-09-12T19:26:49.675 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23", 2022-09-12T19:26:49.676 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000124", 2022-09-12T19:26:49.676 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:7", 2022-09-12T19:26:49.676 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:0", 2022-09-12T19:26:49.676 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000001", 2022-09-12T19:26:49.677 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:2", 2022-09-12T19:26:49.677 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:1", 2022-09-12T19:26:49.677 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4", 2022-09-12T19:26:49.677 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:6", 2022-09-12T19:26:49.678 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:9", 2022-09-12T19:26:49.678 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000126", 2022-09-12T19:26:49.678 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3", 2022-09-12T19:26:49.678 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:4", 2022-09-12T19:26:49.679 INFO:teuthology.orchestra.run.smithi035.stdout: "meta.log.38ca1f57-1358-496c-a99b-df49ee365a4f.2", 2022-09-12T19:26:49.679 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:6", 2022-09-12T19:26:49.679 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:1", 2022-09-12T19:26:49.679 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:1", 2022-09-12T19:26:49.680 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000055", 2022-09-12T19:26:49.680 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000061", 2022-09-12T19:26:49.680 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:1", 2022-09-12T19:26:49.680 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:5", 2022-09-12T19:26:49.681 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:8", 2022-09-12T19:26:49.681 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:1", 2022-09-12T19:26:49.681 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:3", 2022-09-12T19:26:49.681 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:5", 2022-09-12T19:26:49.682 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:1", 2022-09-12T19:26:49.682 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:9", 2022-09-12T19:26:49.682 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.fdf3b910-2ea3-4164-a3f8-b3e146af1c44.0.retry", 2022-09-12T19:26:49.682 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:0", 2022-09-12T19:26:49.683 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000097", 2022-09-12T19:26:49.683 INFO:teuthology.orchestra.run.smithi035.stdout: "datalog.sync-status.shard.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.0", 2022-09-12T19:26:49.683 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:4", 2022-09-12T19:26:49.683 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:6", 2022-09-12T19:26:49.684 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:10", 2022-09-12T19:26:49.684 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:10", 2022-09-12T19:26:49.684 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:6", 2022-09-12T19:26:49.684 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:2", 2022-09-12T19:26:49.685 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:3", 2022-09-12T19:26:49.685 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000042", 2022-09-12T19:26:49.685 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:2", 2022-09-12T19:26:49.686 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:10", 2022-09-12T19:26:49.686 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:3", 2022-09-12T19:26:49.686 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:0", 2022-09-12T19:26:49.686 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:1", 2022-09-12T19:26:49.687 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:7", 2022-09-12T19:26:49.687 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:9", 2022-09-12T19:26:49.687 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:26:49.687 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000075", 2022-09-12T19:26:49.688 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000120", 2022-09-12T19:26:49.688 INFO:teuthology.orchestra.run.smithi035.stdout: "obj_delete_at_hint.0000000030", 2022-09-12T19:26:49.688 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.full-sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:26:49.689 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:5", 2022-09-12T19:26:49.689 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:0", 2022-09-12T19:26:49.689 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:10", 2022-09-12T19:26:49.689 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:9", 2022-09-12T19:26:49.690 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:9", 2022-09-12T19:26:49.690 INFO:teuthology.orchestra.run.smithi035.stdout: "bucket.sync-status.fdf3b910-2ea3-4164-a3f8-b3e146af1c44:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:8" 2022-09-12T19:26:49.690 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:26:49.691 INFO:teuthology.orchestra.run.smithi035.stdout: 2022-09-12T19:26:49.692 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:49.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:49.694 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:49.743 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:49.744 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:49.790 INFO:teuthology.orchestra.run.smithi138.stdout:[ 2022-09-12T19:26:49.790 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:4", 2022-09-12T19:26:49.791 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000078", 2022-09-12T19:26:49.791 INFO:teuthology.orchestra.run.smithi138.stdout: "meta.history", 2022-09-12T19:26:49.791 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:3", 2022-09-12T19:26:49.792 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:0", 2022-09-12T19:26:49.792 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:3", 2022-09-12T19:26:49.792 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000070", 2022-09-12T19:26:49.792 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000104", 2022-09-12T19:26:49.793 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:10", 2022-09-12T19:26:49.793 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000026", 2022-09-12T19:26:49.793 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:5", 2022-09-12T19:26:49.793 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:6", 2022-09-12T19:26:49.794 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:3", 2022-09-12T19:26:49.794 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:2", 2022-09-12T19:26:49.794 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.8", 2022-09-12T19:26:49.794 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000028", 2022-09-12T19:26:49.795 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1", 2022-09-12T19:26:49.795 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:6", 2022-09-12T19:26:49.795 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13", 2022-09-12T19:26:49.795 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:0", 2022-09-12T19:26:49.795 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:1", 2022-09-12T19:26:49.796 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:2", 2022-09-12T19:26:49.796 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:5", 2022-09-12T19:26:49.796 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000040", 2022-09-12T19:26:49.796 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:8", 2022-09-12T19:26:49.797 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.6", 2022-09-12T19:26:49.797 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26", 2022-09-12T19:26:49.797 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:2", 2022-09-12T19:26:49.797 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:5", 2022-09-12T19:26:49.798 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23", 2022-09-12T19:26:49.798 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000015", 2022-09-12T19:26:49.798 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:3", 2022-09-12T19:26:49.799 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:7", 2022-09-12T19:26:49.799 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000069", 2022-09-12T19:26:49.799 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:2", 2022-09-12T19:26:49.799 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:1", 2022-09-12T19:26:49.800 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000095", 2022-09-12T19:26:49.800 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000003", 2022-09-12T19:26:49.800 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000047", 2022-09-12T19:26:49.801 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:6", 2022-09-12T19:26:49.801 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:9", 2022-09-12T19:26:49.801 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000035", 2022-09-12T19:26:49.801 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:10", 2022-09-12T19:26:49.802 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000037", 2022-09-12T19:26:49.802 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:0", 2022-09-12T19:26:49.802 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:6", 2022-09-12T19:26:49.802 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.shard.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.0.retry", 2022-09-12T19:26:49.803 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:0", 2022-09-12T19:26:49.803 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:10", 2022-09-12T19:26:49.803 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000024", 2022-09-12T19:26:49.803 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:1", 2022-09-12T19:26:49.804 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:0", 2022-09-12T19:26:49.804 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:5", 2022-09-12T19:26:49.804 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000000", 2022-09-12T19:26:49.804 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000031", 2022-09-12T19:26:49.805 INFO:teuthology.orchestra.run.smithi138.stdout: "meta.log.c0f7a512-c940-4d1f-8adf-3484630f876b.3", 2022-09-12T19:26:49.805 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:0", 2022-09-12T19:26:49.805 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:8", 2022-09-12T19:26:49.806 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:2", 2022-09-12T19:26:49.806 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:1", 2022-09-12T19:26:49.806 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:10", 2022-09-12T19:26:49.806 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:9", 2022-09-12T19:26:49.807 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:3", 2022-09-12T19:26:49.807 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:5", 2022-09-12T19:26:49.807 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:6", 2022-09-12T19:26:49.807 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:3", 2022-09-12T19:26:49.808 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:7", 2022-09-12T19:26:49.808 INFO:teuthology.orchestra.run.smithi138.stdout: "data_loggenerations_metadata", 2022-09-12T19:26:49.808 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14", 2022-09-12T19:26:49.808 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000076", 2022-09-12T19:26:49.808 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:1", 2022-09-12T19:26:49.809 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000113", 2022-09-12T19:26:49.809 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000005", 2022-09-12T19:26:49.809 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000011", 2022-09-12T19:26:49.809 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:9", 2022-09-12T19:26:49.810 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000058", 2022-09-12T19:26:49.810 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:10", 2022-09-12T19:26:49.810 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000052", 2022-09-12T19:26:49.810 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:9", 2022-09-12T19:26:49.811 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:9", 2022-09-12T19:26:49.811 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000088", 2022-09-12T19:26:49.811 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:9", 2022-09-12T19:26:49.811 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000080", 2022-09-12T19:26:49.812 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000090", 2022-09-12T19:26:49.812 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:10", 2022-09-12T19:26:49.812 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000110", 2022-09-12T19:26:49.812 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20", 2022-09-12T19:26:49.813 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:4", 2022-09-12T19:26:49.813 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:3", 2022-09-12T19:26:49.813 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:2", 2022-09-12T19:26:49.814 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000096", 2022-09-12T19:26:49.814 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15", 2022-09-12T19:26:49.814 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:8", 2022-09-12T19:26:49.814 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:0", 2022-09-12T19:26:49.815 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000087", 2022-09-12T19:26:49.815 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:9", 2022-09-12T19:26:49.815 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000008", 2022-09-12T19:26:49.816 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:6", 2022-09-12T19:26:49.816 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:8", 2022-09-12T19:26:49.816 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:4", 2022-09-12T19:26:49.816 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:8", 2022-09-12T19:26:49.817 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:5", 2022-09-12T19:26:49.817 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:8", 2022-09-12T19:26:49.817 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000006", 2022-09-12T19:26:49.817 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000029", 2022-09-12T19:26:49.818 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000089", 2022-09-12T19:26:49.818 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000022", 2022-09-12T19:26:49.818 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21", 2022-09-12T19:26:49.818 INFO:teuthology.orchestra.run.smithi138.stdout: "bilog.trim", 2022-09-12T19:26:49.819 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000016", 2022-09-12T19:26:49.819 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:3", 2022-09-12T19:26:49.819 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000041", 2022-09-12T19:26:49.820 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:3", 2022-09-12T19:26:49.820 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:8", 2022-09-12T19:26:49.820 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000018", 2022-09-12T19:26:49.820 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:7", 2022-09-12T19:26:49.821 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000092", 2022-09-12T19:26:49.821 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:8", 2022-09-12T19:26:49.821 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22", 2022-09-12T19:26:49.821 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:3", 2022-09-12T19:26:49.822 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.4", 2022-09-12T19:26:49.822 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:7", 2022-09-12T19:26:49.822 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5", 2022-09-12T19:26:49.823 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:10", 2022-09-12T19:26:49.823 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000014", 2022-09-12T19:26:49.823 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:1", 2022-09-12T19:26:49.823 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:9", 2022-09-12T19:26:49.824 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:9", 2022-09-12T19:26:49.824 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000112", 2022-09-12T19:26:49.824 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:7", 2022-09-12T19:26:49.824 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000007", 2022-09-12T19:26:49.825 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000021", 2022-09-12T19:26:49.825 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.16", 2022-09-12T19:26:49.825 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000064", 2022-09-12T19:26:49.825 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:6", 2022-09-12T19:26:49.826 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:5", 2022-09-12T19:26:49.826 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:4", 2022-09-12T19:26:49.826 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:8", 2022-09-12T19:26:49.826 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000071", 2022-09-12T19:26:49.827 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000074", 2022-09-12T19:26:49.827 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16", 2022-09-12T19:26:49.827 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:0", 2022-09-12T19:26:49.827 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:2", 2022-09-12T19:26:49.828 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:5", 2022-09-12T19:26:49.828 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:10", 2022-09-12T19:26:49.828 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:7", 2022-09-12T19:26:49.828 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:10", 2022-09-12T19:26:49.829 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:0", 2022-09-12T19:26:49.829 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:2", 2022-09-12T19:26:49.829 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000081", 2022-09-12T19:26:49.829 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:8", 2022-09-12T19:26:49.830 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000009", 2022-09-12T19:26:49.830 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:0", 2022-09-12T19:26:49.830 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000121", 2022-09-12T19:26:49.830 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:9", 2022-09-12T19:26:49.831 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:5", 2022-09-12T19:26:49.831 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11", 2022-09-12T19:26:49.831 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.17", 2022-09-12T19:26:49.831 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000125", 2022-09-12T19:26:49.832 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000082", 2022-09-12T19:26:49.832 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:9", 2022-09-12T19:26:49.832 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:0", 2022-09-12T19:26:49.832 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:9", 2022-09-12T19:26:49.833 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000105", 2022-09-12T19:26:49.833 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000059", 2022-09-12T19:26:49.833 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:4", 2022-09-12T19:26:49.833 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:2", 2022-09-12T19:26:49.834 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.shard.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.3", 2022-09-12T19:26:49.834 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000077", 2022-09-12T19:26:49.834 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:5", 2022-09-12T19:26:49.834 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000032", 2022-09-12T19:26:49.835 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:8", 2022-09-12T19:26:49.835 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:3", 2022-09-12T19:26:49.835 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:1", 2022-09-12T19:26:49.835 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:4", 2022-09-12T19:26:49.836 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:2", 2022-09-12T19:26:49.836 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000053", 2022-09-12T19:26:49.836 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000091", 2022-09-12T19:26:49.836 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:8", 2022-09-12T19:26:49.837 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:8", 2022-09-12T19:26:49.837 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000065", 2022-09-12T19:26:49.837 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:5", 2022-09-12T19:26:49.837 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000083", 2022-09-12T19:26:49.838 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000010", 2022-09-12T19:26:49.838 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:1", 2022-09-12T19:26:49.838 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:0", 2022-09-12T19:26:49.838 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:8", 2022-09-12T19:26:49.839 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:9", 2022-09-12T19:26:49.839 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:8", 2022-09-12T19:26:49.839 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:10", 2022-09-12T19:26:49.839 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:7", 2022-09-12T19:26:49.840 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:0", 2022-09-12T19:26:49.840 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.7", 2022-09-12T19:26:49.840 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.3", 2022-09-12T19:26:49.841 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:4", 2022-09-12T19:26:49.841 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:10", 2022-09-12T19:26:49.841 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:9", 2022-09-12T19:26:49.841 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:4", 2022-09-12T19:26:49.842 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:3", 2022-09-12T19:26:49.842 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:2", 2022-09-12T19:26:49.842 INFO:teuthology.orchestra.run.smithi138.stdout: "mdlog.sync-status.shard.2", 2022-09-12T19:26:49.842 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:10", 2022-09-12T19:26:49.843 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:3", 2022-09-12T19:26:49.843 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:6", 2022-09-12T19:26:49.843 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:10", 2022-09-12T19:26:49.844 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:4", 2022-09-12T19:26:49.844 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000045", 2022-09-12T19:26:49.844 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:7", 2022-09-12T19:26:49.844 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000002", 2022-09-12T19:26:49.845 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.12", 2022-09-12T19:26:49.845 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:6", 2022-09-12T19:26:49.845 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:6", 2022-09-12T19:26:49.845 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:4", 2022-09-12T19:26:49.846 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000116", 2022-09-12T19:26:49.846 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.shard.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.2", 2022-09-12T19:26:49.846 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:7", 2022-09-12T19:26:49.846 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:7", 2022-09-12T19:26:49.847 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:5", 2022-09-12T19:26:49.847 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000034", 2022-09-12T19:26:49.848 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:7", 2022-09-12T19:26:49.848 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000101", 2022-09-12T19:26:49.848 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:8", 2022-09-12T19:26:49.848 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.13", 2022-09-12T19:26:49.849 INFO:teuthology.orchestra.run.smithi138.stdout: "mdlog.sync-status.shard.1", 2022-09-12T19:26:49.849 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:4", 2022-09-12T19:26:49.849 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.shard.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.2.retry", 2022-09-12T19:26:49.849 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:2", 2022-09-12T19:26:49.850 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000079", 2022-09-12T19:26:49.850 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000049", 2022-09-12T19:26:49.850 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000117", 2022-09-12T19:26:49.851 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000044", 2022-09-12T19:26:49.851 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24", 2022-09-12T19:26:49.851 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:9", 2022-09-12T19:26:49.851 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.14", 2022-09-12T19:26:49.852 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:5", 2022-09-12T19:26:49.852 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000066", 2022-09-12T19:26:49.852 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:8", 2022-09-12T19:26:49.852 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47", 2022-09-12T19:26:49.853 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:7", 2022-09-12T19:26:49.853 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:4", 2022-09-12T19:26:49.853 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:8", 2022-09-12T19:26:49.853 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000068", 2022-09-12T19:26:49.854 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000085", 2022-09-12T19:26:49.854 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:0", 2022-09-12T19:26:49.854 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:9", 2022-09-12T19:26:49.855 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:4", 2022-09-12T19:26:49.855 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36", 2022-09-12T19:26:49.855 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:5", 2022-09-12T19:26:49.855 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000073", 2022-09-12T19:26:49.856 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:8", 2022-09-12T19:26:49.856 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000038", 2022-09-12T19:26:49.856 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:0", 2022-09-12T19:26:49.856 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.10", 2022-09-12T19:26:49.857 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000118", 2022-09-12T19:26:49.857 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:10", 2022-09-12T19:26:49.857 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:6", 2022-09-12T19:26:49.857 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000036", 2022-09-12T19:26:49.858 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000103", 2022-09-12T19:26:49.858 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:8", 2022-09-12T19:26:49.858 INFO:teuthology.orchestra.run.smithi138.stdout: "mdlog.sync-status.shard.3", 2022-09-12T19:26:49.858 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:6", 2022-09-12T19:26:49.859 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:6", 2022-09-12T19:26:49.859 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000119", 2022-09-12T19:26:49.859 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000098", 2022-09-12T19:26:49.859 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:1", 2022-09-12T19:26:49.860 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:5", 2022-09-12T19:26:49.860 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:3", 2022-09-12T19:26:49.860 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38", 2022-09-12T19:26:49.860 INFO:teuthology.orchestra.run.smithi138.stdout: "mdlog.sync-status.shard.0", 2022-09-12T19:26:49.860 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000027", 2022-09-12T19:26:49.861 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:4", 2022-09-12T19:26:49.861 INFO:teuthology.orchestra.run.smithi138.stdout: "mdlog.sync-status", 2022-09-12T19:26:49.861 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000019", 2022-09-12T19:26:49.861 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:5", 2022-09-12T19:26:49.862 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:5", 2022-09-12T19:26:49.862 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000039", 2022-09-12T19:26:49.862 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:1", 2022-09-12T19:26:49.863 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:4", 2022-09-12T19:26:49.863 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:4", 2022-09-12T19:26:49.863 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:1", 2022-09-12T19:26:49.863 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:2", 2022-09-12T19:26:49.864 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10:10", 2022-09-12T19:26:49.864 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:9", 2022-09-12T19:26:49.864 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:7", 2022-09-12T19:26:49.865 INFO:teuthology.orchestra.run.smithi138.stdout: "meta.log.2e4f88cb-0d3e-4f1f-8a81-eb8a58b76e0e.2", 2022-09-12T19:26:49.865 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000100", 2022-09-12T19:26:49.865 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:0", 2022-09-12T19:26:49.866 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:1", 2022-09-12T19:26:49.866 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:6", 2022-09-12T19:26:49.866 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000093", 2022-09-12T19:26:49.866 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37", 2022-09-12T19:26:49.867 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:3", 2022-09-12T19:26:49.867 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:2", 2022-09-12T19:26:49.867 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000004", 2022-09-12T19:26:49.867 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000063", 2022-09-12T19:26:49.868 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:7", 2022-09-12T19:26:49.868 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000122", 2022-09-12T19:26:49.868 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:7", 2022-09-12T19:26:49.868 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000057", 2022-09-12T19:26:49.869 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000054", 2022-09-12T19:26:49.869 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000114", 2022-09-12T19:26:49.869 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:7", 2022-09-12T19:26:49.869 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.2", 2022-09-12T19:26:49.870 INFO:teuthology.orchestra.run.smithi138.stdout: "data_log.0", 2022-09-12T19:26:49.870 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:7", 2022-09-12T19:26:49.870 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000012", 2022-09-12T19:26:49.870 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:10", 2022-09-12T19:26:49.871 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:9", 2022-09-12T19:26:49.871 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:6", 2022-09-12T19:26:49.871 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000084", 2022-09-12T19:26:49.871 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000043", 2022-09-12T19:26:49.871 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:0", 2022-09-12T19:26:49.872 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.11", 2022-09-12T19:26:49.872 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:7", 2022-09-12T19:26:49.872 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:7", 2022-09-12T19:26:49.873 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:2", 2022-09-12T19:26:49.873 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:10", 2022-09-12T19:26:49.873 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:8", 2022-09-12T19:26:49.873 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:10", 2022-09-12T19:26:49.874 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000111", 2022-09-12T19:26:49.874 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000048", 2022-09-12T19:26:49.874 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:3", 2022-09-12T19:26:49.874 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000020", 2022-09-12T19:26:49.875 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:3", 2022-09-12T19:26:49.875 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-10:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.10", 2022-09-12T19:26:49.875 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000099", 2022-09-12T19:26:49.876 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000056", 2022-09-12T19:26:49.876 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:1", 2022-09-12T19:26:49.876 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000072", 2022-09-12T19:26:49.876 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000062", 2022-09-12T19:26:49.877 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000115", 2022-09-12T19:26:49.877 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.1", 2022-09-12T19:26:49.877 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000033", 2022-09-12T19:26:49.877 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:8", 2022-09-12T19:26:49.878 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:10", 2022-09-12T19:26:49.878 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:2", 2022-09-12T19:26:49.878 INFO:teuthology.orchestra.run.smithi138.stdout: "meta.log.c0f7a512-c940-4d1f-8adf-3484630f876b.2", 2022-09-12T19:26:49.878 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000025", 2022-09-12T19:26:49.879 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:9", 2022-09-12T19:26:49.879 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:8", 2022-09-12T19:26:49.879 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000123", 2022-09-12T19:26:49.880 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:2", 2022-09-12T19:26:49.880 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.18", 2022-09-12T19:26:49.880 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000108", 2022-09-12T19:26:49.880 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:6", 2022-09-12T19:26:49.881 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000094", 2022-09-12T19:26:49.881 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:5", 2022-09-12T19:26:49.881 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:1", 2022-09-12T19:26:49.882 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:6", 2022-09-12T19:26:49.882 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000060", 2022-09-12T19:26:49.882 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19", 2022-09-12T19:26:49.882 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:4", 2022-09-12T19:26:49.883 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000109", 2022-09-12T19:26:49.883 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.15", 2022-09-12T19:26:49.883 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:3", 2022-09-12T19:26:49.883 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:1", 2022-09-12T19:26:49.884 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000013", 2022-09-12T19:26:49.884 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:1", 2022-09-12T19:26:49.884 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:1", 2022-09-12T19:26:49.884 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:6", 2022-09-12T19:26:49.885 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:3", 2022-09-12T19:26:49.885 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:4", 2022-09-12T19:26:49.885 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000051", 2022-09-12T19:26:49.885 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000106", 2022-09-12T19:26:49.886 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:7", 2022-09-12T19:26:49.886 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:4", 2022-09-12T19:26:49.886 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:10", 2022-09-12T19:26:49.886 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000017", 2022-09-12T19:26:49.887 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:0", 2022-09-12T19:26:49.887 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.shard.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.1", 2022-09-12T19:26:49.887 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:1", 2022-09-12T19:26:49.887 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17", 2022-09-12T19:26:49.888 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000023", 2022-09-12T19:26:49.888 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:7", 2022-09-12T19:26:49.888 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000107", 2022-09-12T19:26:49.888 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:10", 2022-09-12T19:26:49.889 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:0", 2022-09-12T19:26:49.889 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000046", 2022-09-12T19:26:49.889 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:2", 2022-09-12T19:26:49.889 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:0", 2022-09-12T19:26:49.890 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:3", 2022-09-12T19:26:49.890 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3:1", 2022-09-12T19:26:49.890 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:0", 2022-09-12T19:26:49.890 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:5", 2022-09-12T19:26:49.891 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25", 2022-09-12T19:26:49.891 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:3", 2022-09-12T19:26:49.891 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:6", 2022-09-12T19:26:49.891 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:10", 2022-09-12T19:26:49.892 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:1", 2022-09-12T19:26:49.892 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000067", 2022-09-12T19:26:49.892 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:9", 2022-09-12T19:26:49.892 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:5", 2022-09-12T19:26:49.893 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-20:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.20:5", 2022-09-12T19:26:49.893 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.9", 2022-09-12T19:26:49.893 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000086", 2022-09-12T19:26:49.893 INFO:teuthology.orchestra.run.smithi138.stdout: "meta.log.c0f7a512-c940-4d1f-8adf-3484630f876b.0", 2022-09-12T19:26:49.894 INFO:teuthology.orchestra.run.smithi138.stdout: "sync.error-log.5", 2022-09-12T19:26:49.894 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:7", 2022-09-12T19:26:49.894 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:10", 2022-09-12T19:26:49.894 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:4", 2022-09-12T19:26:49.895 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:9", 2022-09-12T19:26:49.895 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:6", 2022-09-12T19:26:49.895 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-15:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.15:6", 2022-09-12T19:26:49.895 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-14:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.14:2", 2022-09-12T19:26:49.896 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18", 2022-09-12T19:26:49.896 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:2", 2022-09-12T19:26:49.896 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:2", 2022-09-12T19:26:49.896 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:3", 2022-09-12T19:26:49.897 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12", 2022-09-12T19:26:49.897 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:2", 2022-09-12T19:26:49.897 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:1", 2022-09-12T19:26:49.897 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-12:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.12:4", 2022-09-12T19:26:49.898 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-23:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.23:4", 2022-09-12T19:26:49.898 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4:4", 2022-09-12T19:26:49.898 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-48:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.5:2", 2022-09-12T19:26:49.898 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000050", 2022-09-12T19:26:49.899 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000102", 2022-09-12T19:26:49.899 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:1", 2022-09-12T19:26:49.899 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:6", 2022-09-12T19:26:49.899 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.shard.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.1.retry", 2022-09-12T19:26:49.900 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-13:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.13:6", 2022-09-12T19:26:49.900 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:7", 2022-09-12T19:26:49.900 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000124", 2022-09-12T19:26:49.900 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-37:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.37:7", 2022-09-12T19:26:49.901 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-21:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.21:0", 2022-09-12T19:26:49.901 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000001", 2022-09-12T19:26:49.901 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-47:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.4", 2022-09-12T19:26:49.902 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-22:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.22:9", 2022-09-12T19:26:49.902 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000126", 2022-09-12T19:26:49.902 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.full-sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-46:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.3", 2022-09-12T19:26:49.902 INFO:teuthology.orchestra.run.smithi138.stdout: "meta.log.38ca1f57-1358-496c-a99b-df49ee365a4f.2", 2022-09-12T19:26:49.903 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-16:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.16:6", 2022-09-12T19:26:49.903 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:1", 2022-09-12T19:26:49.903 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000055", 2022-09-12T19:26:49.903 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000061", 2022-09-12T19:26:49.904 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:5", 2022-09-12T19:26:49.904 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:8", 2022-09-12T19:26:49.904 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-17:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.17:5", 2022-09-12T19:26:49.904 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:9", 2022-09-12T19:26:49.905 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-36:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.36:0", 2022-09-12T19:26:49.905 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000097", 2022-09-12T19:26:49.905 INFO:teuthology.orchestra.run.smithi138.stdout: "datalog.sync-status.shard.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.0", 2022-09-12T19:26:49.905 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-18:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.18:4", 2022-09-12T19:26:49.906 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-26:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.26:2", 2022-09-12T19:26:49.906 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:3", 2022-09-12T19:26:49.906 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000042", 2022-09-12T19:26:49.906 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-24:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.24:3", 2022-09-12T19:26:49.907 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-49:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5446.1:1", 2022-09-12T19:26:49.907 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-19:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.19:9", 2022-09-12T19:26:49.907 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000075", 2022-09-12T19:26:49.907 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000120", 2022-09-12T19:26:49.908 INFO:teuthology.orchestra.run.smithi138.stdout: "obj_delete_at_hint.0000000030", 2022-09-12T19:26:49.908 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-11:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.11:5", 2022-09-12T19:26:49.908 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-25:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.25:0", 2022-09-12T19:26:49.908 INFO:teuthology.orchestra.run.smithi138.stdout: "bucket.sync-status.fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-38:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.38:10" 2022-09-12T19:26:49.909 INFO:teuthology.orchestra.run.smithi138.stdout:] 2022-09-12T19:26:49.909 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-09-12T19:26:49.911 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2022-09-12T19:26:49.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:49.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:49.913 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:49.968 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:49.969 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:50.030 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:26:50.030 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:50.030 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000003750", 2022-09-12T19:26:50.031 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:26:48.715838Z" 2022-09-12T19:26:50.031 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:50.031 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:50.031 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006000", 2022-09-12T19:26:50.032 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:26:48.716874Z" 2022-09-12T19:26:50.032 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:50.032 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:50.032 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000008115", 2022-09-12T19:26:50.033 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:26:48.717941Z" 2022-09-12T19:26:50.033 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:50.033 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:50.033 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-12T19:26:50.034 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:26:48.719052Z" 2022-09-12T19:26:50.034 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:50.034 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:26:50.035 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006000', 2: '00000000000000000000:00000000000000008115', 3: '00000000000000000000:00000000000000002060'} 2022-09-12T19:26:50.036 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2022-09-12T19:26:50.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:50.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:50.038 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:50.090 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:50.090 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:50.141 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:26:50.141 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:26:50.142 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:26:50.142 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:26:50.142 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:26:50.142 INFO:teuthology.orchestra.run.smithi138.stdout: "instance_id": 4124212413593488935 2022-09-12T19:26:50.143 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:50.143 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:26:50.143 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:50.143 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:26:50.144 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:50.144 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:26:50.144 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-12T19:26:50.145 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:50.145 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:50.145 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:50.145 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:03:01.269036Z" 2022-09-12T19:26:50.146 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:50.146 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:50.146 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:50.147 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:26:50.147 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:50.147 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:26:50.148 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000004740", 2022-09-12T19:26:50.148 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:50.148 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:50.149 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:50.149 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:17:11.711910Z" 2022-09-12T19:26:50.149 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:50.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:50.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:50.150 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:26:50.151 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:50.151 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:26:50.151 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000006855", 2022-09-12T19:26:50.152 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:50.152 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:50.152 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:50.153 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:20:42.909181Z" 2022-09-12T19:26:50.153 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:50.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:50.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:50.154 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:26:50.154 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:50.154 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:26:50.155 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:26:50.155 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:50.156 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:50.156 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:50.156 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:16:51.797188Z" 2022-09-12T19:26:50.156 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:50.157 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:50.157 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:26:50.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:50.158 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:26:50.158 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 0, 2022-09-12T19:26:50.158 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 0 2022-09-12T19:26:50.158 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:50.159 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:26:50Z" 2022-09-12T19:26:50.159 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:26:50.160 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 4124212413593488935 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:03:01.269036Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004740", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:17:11.711910Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006855", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:20:42.909181Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:51.797188Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:26:50Z" } 2022-09-12T19:26:50.161 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:03:01.269036Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004740', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:17:11.711910Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006855', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:20:42.909181Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:51.797188Z'}}] 2022-09-12T19:26:50.161 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006000', 2: '00000000000000000000:00000000000000008115', 3: '00000000000000000000:00000000000000002060'} 2022-09-12T19:26:50.162 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004740', 2: '00000000000000000000:00000000000000006855', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:26:50.162 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003750 target=00000000000000000000:00000000000000002909, shard=1 master=00000000000000000000:00000000000000006000 target=00000000000000000000:00000000000000004740, shard=2 master=00000000000000000000:00000000000000008115 target=00000000000000000000:00000000000000006855, shard=3 master=00000000000000000000:00000000000000002060 target=00000000000000000000:00000000000000000796 2022-09-12T19:26:55.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:55.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:55.165 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:55.221 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:55.221 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:55.277 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:26:55.278 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:26:55.278 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:26:55.278 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:26:55.279 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:26:55.279 INFO:teuthology.orchestra.run.smithi138.stdout: "instance_id": 4124212413593488935 2022-09-12T19:26:55.279 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:55.279 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:26:55.280 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:55.280 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:26:55.280 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:55.280 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:26:55.281 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000003750", 2022-09-12T19:26:55.281 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:55.281 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:55.281 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:55.282 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:26:48.715838Z" 2022-09-12T19:26:55.282 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:55.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:55.282 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:55.283 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:26:55.283 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:55.283 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:26:55.283 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000006000", 2022-09-12T19:26:55.284 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:55.284 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:55.284 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:55.284 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:26:48.716874Z" 2022-09-12T19:26:55.285 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:55.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:55.285 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:55.285 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:26:55.286 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:55.286 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:26:55.286 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000008115", 2022-09-12T19:26:55.286 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:55.287 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:55.287 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:55.287 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:26:48.717941Z" 2022-09-12T19:26:55.287 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:55.288 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:55.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:55.288 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:26:55.288 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:55.289 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:26:55.289 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-12T19:26:55.289 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:55.289 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:55.290 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:55.290 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:26:48.719052Z" 2022-09-12T19:26:55.290 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:55.290 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:55.291 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:26:55.291 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:55.291 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:26:55.291 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 0, 2022-09-12T19:26:55.291 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 0 2022-09-12T19:26:55.292 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:55.292 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:26:55Z" 2022-09-12T19:26:55.292 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:26:55.293 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 4124212413593488935 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003750", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:48.715838Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006000", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:48.716874Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008115", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:48.717941Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002060", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:48.719052Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:26:55Z" } 2022-09-12T19:26:55.294 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003750', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:48.715838Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006000', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:48.716874Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008115', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:48.717941Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:48.719052Z'}}] 2022-09-12T19:26:55.295 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006000', 2: '00000000000000000000:00000000000000008115', 3: '00000000000000000000:00000000000000002060'} 2022-09-12T19:26:55.295 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006000', 2: '00000000000000000000:00000000000000008115', 3: '00000000000000000000:00000000000000002060'} 2022-09-12T19:26:55.296 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2022-09-12T19:26:55.296 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone3 2022-09-12T19:26:55.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:55.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:55.298 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:55.356 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:55.356 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:55.413 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:26:55.414 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:55.414 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000003750", 2022-09-12T19:26:55.414 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:26:48.715838Z" 2022-09-12T19:26:55.415 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:55.415 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:55.415 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006000", 2022-09-12T19:26:55.415 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:26:48.716874Z" 2022-09-12T19:26:55.416 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:55.416 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:55.416 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000008115", 2022-09-12T19:26:55.416 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:26:48.717941Z" 2022-09-12T19:26:55.417 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:55.417 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:55.417 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-12T19:26:55.417 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:26:48.719052Z" 2022-09-12T19:26:55.418 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:55.418 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:26:55.428 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006000', 2: '00000000000000000000:00000000000000008115', 3: '00000000000000000000:00000000000000002060'} 2022-09-12T19:26:55.429 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2022-09-12T19:26:55.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:55.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:55.431 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:55.485 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:55.486 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:55.529 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:26:55.530 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:26:55.530 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:26:55.530 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:26:55.531 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:26:55.531 INFO:teuthology.orchestra.run.smithi035.stdout: "instance_id": 16681684026229800507 2022-09-12T19:26:55.531 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:55.532 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:26:55.532 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:55.532 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:26:55.533 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:26:55.533 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:26:55.533 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000003750", 2022-09-12T19:26:55.533 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:26:55.534 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:26:55.534 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:26:55.534 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:26:48.715838Z" 2022-09-12T19:26:55.535 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:55.535 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:55.535 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:55.535 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:26:55.536 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:26:55.536 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:26:55.536 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006000", 2022-09-12T19:26:55.536 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:26:55.537 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:26:55.537 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:26:55.537 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:26:48.716874Z" 2022-09-12T19:26:55.537 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:55.538 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:55.538 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:55.538 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:26:55.538 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:26:55.539 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:26:55.539 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000008115", 2022-09-12T19:26:55.539 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:26:55.539 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:26:55.540 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:26:55.540 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:26:48.717941Z" 2022-09-12T19:26:55.540 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:55.540 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:55.540 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:55.541 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:26:55.541 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:26:55.541 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:26:55.541 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-12T19:26:55.542 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:26:55.542 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:26:55.542 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:26:55.542 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:26:48.719052Z" 2022-09-12T19:26:55.543 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:55.543 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:55.543 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:26:55.544 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:55.544 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:26:55.545 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 0, 2022-09-12T19:26:55.545 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 0 2022-09-12T19:26:55.545 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:55.545 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:26:55Z" 2022-09-12T19:26:55.546 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:26:55.547 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16681684026229800507 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003750", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:48.715838Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006000", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:48.716874Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008115", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:48.717941Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002060", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:48.719052Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:26:55Z" } 2022-09-12T19:26:55.547 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003750', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:48.715838Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006000', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:48.716874Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008115', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:48.717941Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:48.719052Z'}}] 2022-09-12T19:26:55.548 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006000', 2: '00000000000000000000:00000000000000008115', 3: '00000000000000000000:00000000000000002060'} 2022-09-12T19:26:55.549 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006000', 2: '00000000000000000000:00000000000000008115', 3: '00000000000000000000:00000000000000002060'} 2022-09-12T19:26:55.549 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2022-09-12T19:26:55.550 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone4 2022-09-12T19:26:55.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:55.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:55.551 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:55.602 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:55.602 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:55.675 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:26:55.676 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:55.676 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000003750", 2022-09-12T19:26:55.677 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:26:48.715838Z" 2022-09-12T19:26:55.678 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:55.678 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:55.679 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006000", 2022-09-12T19:26:55.679 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:26:48.716874Z" 2022-09-12T19:26:55.679 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:55.680 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:55.680 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000008115", 2022-09-12T19:26:55.680 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:26:48.717941Z" 2022-09-12T19:26:55.681 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:55.681 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:55.681 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-12T19:26:55.681 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:26:48.719052Z" 2022-09-12T19:26:55.682 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:55.682 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:26:55.683 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006000', 2: '00000000000000000000:00000000000000008115', 3: '00000000000000000000:00000000000000002060'} 2022-09-12T19:26:55.684 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2022-09-12T19:26:55.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:55.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:55.685 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:55.738 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:55.739 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:55.785 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:26:55.785 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:26:55.786 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:26:55.786 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:26:55.786 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:26:55.786 INFO:teuthology.orchestra.run.smithi138.stdout: "instance_id": 14449053848353038014 2022-09-12T19:26:55.787 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:55.787 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:26:55.788 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:55.788 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:26:55.789 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:55.789 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:26:55.790 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000003750", 2022-09-12T19:26:55.791 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:55.791 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:55.792 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:55.792 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:26:48.715838Z" 2022-09-12T19:26:55.793 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:55.793 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:55.794 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:55.794 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:26:55.795 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:55.795 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:26:55.796 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000006000", 2022-09-12T19:26:55.796 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:55.797 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:55.797 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:55.798 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:26:48.716874Z" 2022-09-12T19:26:55.798 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:55.799 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:55.799 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:55.800 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:26:55.800 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:55.801 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:26:55.801 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000008115", 2022-09-12T19:26:55.802 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:55.802 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:55.803 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:55.803 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:26:48.717941Z" 2022-09-12T19:26:55.804 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:55.804 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:55.805 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:55.805 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:26:55.806 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:26:55.806 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:26:55.807 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-12T19:26:55.807 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:26:55.808 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:26:55.808 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:26:55.809 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:26:48.719052Z" 2022-09-12T19:26:55.809 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:55.810 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:55.811 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:26:55.811 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:55.812 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:26:55.812 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 0, 2022-09-12T19:26:55.813 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 0 2022-09-12T19:26:55.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:55.814 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:26:55Z" 2022-09-12T19:26:55.814 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:26:55.816 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14449053848353038014 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003750", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:48.715838Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006000", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:48.716874Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008115", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:48.717941Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002060", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:48.719052Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:26:55Z" } 2022-09-12T19:26:55.818 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003750', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:48.715838Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006000', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:48.716874Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008115', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:48.717941Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:48.719052Z'}}] 2022-09-12T19:26:55.819 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006000', 2: '00000000000000000000:00000000000000008115', 3: '00000000000000000000:00000000000000002060'} 2022-09-12T19:26:55.820 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006000', 2: '00000000000000000000:00000000000000008115', 3: '00000000000000000000:00000000000000002060'} 2022-09-12T19:26:55.820 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2022-09-12T19:26:55.821 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2022-09-12T19:26:55.822 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:55.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:55.824 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:55.874 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:55.874 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:55.941 INFO:teuthology.orchestra.run.smithi138.stdout:[ 2022-09-12T19:26:55.941 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:55.942 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000003327", 2022-09-12T19:26:55.942 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:26:52.516493Z" 2022-09-12T19:26:55.942 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:55.942 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:55.943 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000006000", 2022-09-12T19:26:55.943 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:26:52.517959Z" 2022-09-12T19:26:55.943 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:55.943 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:55.944 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000007835", 2022-09-12T19:26:55.944 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:26:52.519418Z" 2022-09-12T19:26:55.944 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:55.944 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:55.945 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-12T19:26:55.945 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:26:52.520864Z" 2022-09-12T19:26:55.945 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:55.945 INFO:teuthology.orchestra.run.smithi138.stdout:] 2022-09-12T19:26:55.947 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006000', 2: '00000000000000000000:00000000000000007835', 3: '00000000000000000000:00000000000000002060'} 2022-09-12T19:26:55.947 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2022-09-12T19:26:55.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:55.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:55.949 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:56.004 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:56.005 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:56.057 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:26:56.058 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:26:56.058 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:26:56.058 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:26:56.059 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:26:56.059 INFO:teuthology.orchestra.run.smithi035.stdout: "instance_id": 2819386190429732693 2022-09-12T19:26:56.060 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:56.060 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:26:56.060 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:56.061 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:26:56.061 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:26:56.061 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:26:56.061 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000003327", 2022-09-12T19:26:56.062 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:26:56.062 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:26:56.062 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:26:56.062 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:26:52.516493Z" 2022-09-12T19:26:56.063 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:56.063 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:56.063 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:56.063 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:26:56.064 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:26:56.064 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:26:56.064 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006000", 2022-09-12T19:26:56.064 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:26:56.065 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:26:56.065 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:26:56.065 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:26:52.517959Z" 2022-09-12T19:26:56.065 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:56.066 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:56.066 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:56.066 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:26:56.066 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:26:56.067 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:26:56.067 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000007835", 2022-09-12T19:26:56.067 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:26:56.067 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:26:56.068 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:26:56.068 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:26:52.519418Z" 2022-09-12T19:26:56.068 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:56.068 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:56.069 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:56.069 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:26:56.069 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:26:56.069 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:26:56.070 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-12T19:26:56.070 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:26:56.070 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:26:56.071 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:26:56.071 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:26:52.520864Z" 2022-09-12T19:26:56.071 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:56.071 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:56.072 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:26:56.072 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:56.072 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:26:56.072 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 0, 2022-09-12T19:26:56.073 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 0 2022-09-12T19:26:56.073 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:56.073 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:26:56Z" 2022-09-12T19:26:56.073 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:26:56.075 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 2819386190429732693 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003327", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:52.516493Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006000", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:52.517959Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007835", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:52.519418Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002060", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:52.520864Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:26:56Z" } 2022-09-12T19:26:56.075 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003327', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:52.516493Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006000', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:52.517959Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007835', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:52.519418Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:52.520864Z'}}] 2022-09-12T19:26:56.076 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006000', 2: '00000000000000000000:00000000000000007835', 3: '00000000000000000000:00000000000000002060'} 2022-09-12T19:26:56.076 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006000', 2: '00000000000000000000:00000000000000007835', 3: '00000000000000000000:00000000000000002060'} 2022-09-12T19:26:56.077 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2022-09-12T19:26:56.077 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone3 2022-09-12T19:26:56.078 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:56.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:56.079 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:56.128 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:56.128 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:56.179 INFO:teuthology.orchestra.run.smithi138.stdout:[ 2022-09-12T19:26:56.180 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:56.180 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000003327", 2022-09-12T19:26:56.180 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:26:52.516493Z" 2022-09-12T19:26:56.181 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:56.181 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:56.181 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000006000", 2022-09-12T19:26:56.181 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:26:52.517959Z" 2022-09-12T19:26:56.182 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:56.182 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:56.182 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000007835", 2022-09-12T19:26:56.182 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:26:52.519418Z" 2022-09-12T19:26:56.183 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:26:56.183 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:26:56.183 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-12T19:26:56.183 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T19:26:52.520864Z" 2022-09-12T19:26:56.184 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:26:56.184 INFO:teuthology.orchestra.run.smithi138.stdout:] 2022-09-12T19:26:56.185 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006000', 2: '00000000000000000000:00000000000000007835', 3: '00000000000000000000:00000000000000002060'} 2022-09-12T19:26:56.186 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2022-09-12T19:26:56.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:56.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:56.187 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:56.243 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:56.243 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:56.286 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:26:56.286 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:26:56.286 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:26:56.287 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:26:56.287 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:26:56.287 INFO:teuthology.orchestra.run.smithi035.stdout: "instance_id": 16602476873656756125 2022-09-12T19:26:56.288 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:56.288 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:26:56.288 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:56.288 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:26:56.289 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:26:56.289 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:26:56.289 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000003327", 2022-09-12T19:26:56.289 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:26:56.290 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:26:56.290 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:26:56.290 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:26:52.516493Z" 2022-09-12T19:26:56.291 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:56.291 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:56.292 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:56.292 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:26:56.293 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:26:56.293 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:26:56.293 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006000", 2022-09-12T19:26:56.293 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:26:56.294 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:26:56.294 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:26:56.295 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:26:52.517959Z" 2022-09-12T19:26:56.295 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:56.295 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:56.296 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:56.296 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:26:56.296 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:26:56.296 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:26:56.297 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000007835", 2022-09-12T19:26:56.297 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:26:56.297 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:26:56.298 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:26:56.298 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:26:52.519418Z" 2022-09-12T19:26:56.299 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:56.299 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:56.300 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:56.300 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:26:56.301 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:26:56.301 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:26:56.302 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-12T19:26:56.302 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:26:56.303 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:26:56.303 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:26:56.303 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:26:52.520864Z" 2022-09-12T19:26:56.304 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:56.304 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:56.304 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:26:56.305 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:56.305 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:26:56.305 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 0, 2022-09-12T19:26:56.306 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 0 2022-09-12T19:26:56.306 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:56.306 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:26:56Z" 2022-09-12T19:26:56.306 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:26:56.308 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16602476873656756125 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003327", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:52.516493Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006000", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:52.517959Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007835", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:52.519418Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002060", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:52.520864Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:26:56Z" } 2022-09-12T19:26:56.308 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003327', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:52.516493Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006000', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:52.517959Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007835', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:52.519418Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:52.520864Z'}}] 2022-09-12T19:26:56.309 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006000', 2: '00000000000000000000:00000000000000007835', 3: '00000000000000000000:00000000000000002060'} 2022-09-12T19:26:56.310 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006000', 2: '00000000000000000000:00000000000000007835', 3: '00000000000000000000:00000000000000002060'} 2022-09-12T19:26:56.310 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2022-09-12T19:26:56.311 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone4 2022-09-12T19:26:56.311 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone1 2022-09-12T19:26:56.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:56.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:56.313 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:56.363 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:56.363 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:56.409 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:26:56.410 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:56.410 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000003609", 2022-09-12T19:26:56.410 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:26:50.601083Z" 2022-09-12T19:26:56.410 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:56.411 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:56.411 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000005999", 2022-09-12T19:26:56.411 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:26:50.602329Z" 2022-09-12T19:26:56.412 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:56.412 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:56.412 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000007553", 2022-09-12T19:26:56.412 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:26:50.603550Z" 2022-09-12T19:26:56.413 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:56.413 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:56.413 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-12T19:26:56.413 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:26:50.604778Z" 2022-09-12T19:26:56.414 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:56.414 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:26:56.415 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000003609', 1: '00000000000000000000:00000000000000005999', 2: '00000000000000000000:00000000000000007553', 3: '00000000000000000000:00000000000000002060'} 2022-09-12T19:26:56.416 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2022-09-12T19:26:56.416 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:56.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:26:56.417 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:26:56.471 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:26:56.471 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:26:56.518 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:26:56.518 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:26:56.518 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:26:56.519 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:26:56.519 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:26:56.519 INFO:teuthology.orchestra.run.smithi035.stdout: "instance_id": 9390097840922254989 2022-09-12T19:26:56.519 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:56.520 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:26:56.520 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:56.520 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:26:56.520 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:26:56.521 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:26:56.521 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002768", 2022-09-12T19:26:56.521 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:26:56.522 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:26:56.522 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:26:56.522 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:02:41.932652Z" 2022-09-12T19:26:56.522 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:56.523 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:56.523 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:56.523 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:26:56.523 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:26:56.524 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:26:56.524 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000004739", 2022-09-12T19:26:56.524 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:26:56.525 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:26:56.525 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:26:56.525 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:17:15.699790Z" 2022-09-12T19:26:56.525 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:56.526 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:56.526 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:56.526 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:26:56.526 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:26:56.527 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:26:56.527 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006293", 2022-09-12T19:26:56.527 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:26:56.527 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:26:56.528 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:26:56.528 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:21:43.388813Z" 2022-09-12T19:26:56.528 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:56.528 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:56.529 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:26:56.529 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:26:56.529 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:26:56.529 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:26:56.530 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:26:56.530 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:26:56.530 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:26:56.530 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:26:56.531 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:31.931954Z" 2022-09-12T19:26:56.531 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:56.531 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:26:56.532 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:26:56.532 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:56.532 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:26:56.532 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 0, 2022-09-12T19:26:56.533 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 0 2022-09-12T19:26:56.533 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:26:56.533 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:26:56Z" 2022-09-12T19:26:56.533 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:26:56.535 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 9390097840922254989 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002768", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:02:41.932652Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004739", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:17:15.699790Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006293", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:21:43.388813Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:31.931954Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:26:56Z" } 2022-09-12T19:26:56.535 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002768', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:02:41.932652Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004739', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:17:15.699790Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006293', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:21:43.388813Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:31.931954Z'}}] 2022-09-12T19:26:56.536 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003609', 1: '00000000000000000000:00000000000000005999', 2: '00000000000000000000:00000000000000007553', 3: '00000000000000000000:00000000000000002060'} 2022-09-12T19:26:56.536 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002768', 1: '00000000000000000000:00000000000000004739', 2: '00000000000000000000:00000000000000006293', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:26:56.537 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone3: shard=0 master=00000000000000000000:00000000000000003609 target=00000000000000000000:00000000000000002768, shard=1 master=00000000000000000000:00000000000000005999 target=00000000000000000000:00000000000000004739, shard=2 master=00000000000000000000:00000000000000007553 target=00000000000000000000:00000000000000006293, shard=3 master=00000000000000000000:00000000000000002060 target=00000000000000000000:00000000000000000796 2022-09-12T19:27:01.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:27:01.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:27:01.540 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:27:01.601 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:27:01.601 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:27:01.665 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:27:01.665 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:27:01.665 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:27:01.666 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:27:01.666 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:27:01.666 INFO:teuthology.orchestra.run.smithi035.stdout: "instance_id": 9390097840922254989 2022-09-12T19:27:01.666 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:27:01.667 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:27:01.667 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:27:01.667 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:27:01.667 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:27:01.668 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:27:01.668 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002768", 2022-09-12T19:27:01.668 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:27:01.669 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:27:01.669 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:27:01.669 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:02:41.932652Z" 2022-09-12T19:27:01.669 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:27:01.670 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:27:01.670 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:27:01.670 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:27:01.671 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:27:01.671 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:27:01.671 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000004739", 2022-09-12T19:27:01.671 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:27:01.672 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:27:01.672 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:27:01.672 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:17:15.699790Z" 2022-09-12T19:27:01.672 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:27:01.673 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:27:01.673 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:27:01.673 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:27:01.673 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:27:01.674 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:27:01.674 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006293", 2022-09-12T19:27:01.674 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:27:01.674 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:27:01.675 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:27:01.675 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:21:43.388813Z" 2022-09-12T19:27:01.675 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:27:01.675 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:27:01.676 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:27:01.676 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:27:01.676 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:27:01.676 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:27:01.677 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-12T19:27:01.677 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:27:01.677 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:27:01.678 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:27:01.678 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:16:31.931954Z" 2022-09-12T19:27:01.678 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:27:01.678 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:27:01.679 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:27:01.679 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:27:01.679 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:27:01.679 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 0, 2022-09-12T19:27:01.680 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 0 2022-09-12T19:27:01.680 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:27:01.680 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:27:01Z" 2022-09-12T19:27:01.681 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:27:01.682 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 9390097840922254989 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002768", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:02:41.932652Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004739", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:17:15.699790Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006293", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:21:43.388813Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:16:31.931954Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:27:01Z" } 2022-09-12T19:27:01.683 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002768', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:02:41.932652Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004739', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:17:15.699790Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006293', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:21:43.388813Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:16:31.931954Z'}}] 2022-09-12T19:27:01.683 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003609', 1: '00000000000000000000:00000000000000005999', 2: '00000000000000000000:00000000000000007553', 3: '00000000000000000000:00000000000000002060'} 2022-09-12T19:27:01.684 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002768', 1: '00000000000000000000:00000000000000004739', 2: '00000000000000000000:00000000000000006293', 3: '00000000000000000000:00000000000000000796'} 2022-09-12T19:27:01.684 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone3: shard=0 master=00000000000000000000:00000000000000003609 target=00000000000000000000:00000000000000002768, shard=1 master=00000000000000000000:00000000000000005999 target=00000000000000000000:00000000000000004739, shard=2 master=00000000000000000000:00000000000000007553 target=00000000000000000000:00000000000000006293, shard=3 master=00000000000000000000:00000000000000002060 target=00000000000000000000:00000000000000000796 2022-09-12T19:27:06.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:27:06.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:27:06.687 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:27:06.749 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:27:06.749 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:27:06.804 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:27:06.805 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:27:06.805 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:27:06.806 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:27:06.806 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:27:06.807 INFO:teuthology.orchestra.run.smithi035.stdout: "instance_id": 9390097840922254989 2022-09-12T19:27:06.807 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:27:06.808 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:27:06.808 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:27:06.808 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:27:06.809 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:27:06.809 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:27:06.817 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000003609", 2022-09-12T19:27:06.818 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:27:06.818 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:27:06.818 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:27:06.819 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:26:50.601083Z" 2022-09-12T19:27:06.819 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:27:06.819 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:27:06.820 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:27:06.820 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:27:06.821 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:27:06.821 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:27:06.821 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000005999", 2022-09-12T19:27:06.821 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:27:06.822 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:27:06.822 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:27:06.822 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:26:50.602329Z" 2022-09-12T19:27:06.822 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:27:06.823 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:27:06.823 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:27:06.823 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:27:06.824 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:27:06.824 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:27:06.824 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000007553", 2022-09-12T19:27:06.825 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:27:06.825 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:27:06.825 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:27:06.825 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:26:50.603550Z" 2022-09-12T19:27:06.826 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:27:06.826 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:27:06.826 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:27:06.826 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:27:06.827 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:27:06.827 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T19:27:06.827 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-12T19:27:06.827 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:27:06.828 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:27:06.828 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:27:06.828 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:26:50.604778Z" 2022-09-12T19:27:06.828 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:27:06.829 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:27:06.829 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:27:06.829 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:27:06.830 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:27:06.830 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 0, 2022-09-12T19:27:06.830 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 0 2022-09-12T19:27:06.830 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:27:06.831 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:27:06Z" 2022-09-12T19:27:06.831 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:27:06.833 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 9390097840922254989 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003609", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:50.601083Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005999", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:50.602329Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007553", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:50.603550Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002060", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:50.604778Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:27:06Z" } 2022-09-12T19:27:06.833 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003609', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:50.601083Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005999', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:50.602329Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007553', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:50.603550Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:50.604778Z'}}] 2022-09-12T19:27:06.834 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003609', 1: '00000000000000000000:00000000000000005999', 2: '00000000000000000000:00000000000000007553', 3: '00000000000000000000:00000000000000002060'} 2022-09-12T19:27:06.835 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003609', 1: '00000000000000000000:00000000000000005999', 2: '00000000000000000000:00000000000000007553', 3: '00000000000000000000:00000000000000002060'} 2022-09-12T19:27:06.835 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2022-09-12T19:27:06.836 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone2 2022-09-12T19:27:06.836 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:27:06.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:27:06.837 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:27:06.889 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:27:06.889 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:27:06.937 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:27:06.937 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:27:06.938 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000003609", 2022-09-12T19:27:06.938 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:26:50.601083Z" 2022-09-12T19:27:06.938 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:27:06.939 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:27:06.939 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000005999", 2022-09-12T19:27:06.939 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:26:50.602329Z" 2022-09-12T19:27:06.940 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:27:06.940 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:27:06.940 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000007553", 2022-09-12T19:27:06.940 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:26:50.603550Z" 2022-09-12T19:27:06.941 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:27:06.941 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:27:06.941 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-12T19:27:06.941 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:26:50.604778Z" 2022-09-12T19:27:06.942 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:27:06.942 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:27:06.943 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000003609', 1: '00000000000000000000:00000000000000005999', 2: '00000000000000000000:00000000000000007553', 3: '00000000000000000000:00000000000000002060'} 2022-09-12T19:27:06.944 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2022-09-12T19:27:06.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:27:06.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:27:06.946 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:27:07.004 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:27:07.004 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:27:07.078 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:27:07.078 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:27:07.079 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:27:07.079 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:27:07.079 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:27:07.080 INFO:teuthology.orchestra.run.smithi138.stdout: "instance_id": 15445976120917781840 2022-09-12T19:27:07.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:27:07.080 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:27:07.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:27:07.081 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:27:07.081 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:27:07.081 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:27:07.082 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000003609", 2022-09-12T19:27:07.082 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:27:07.082 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:27:07.082 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:27:07.083 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:26:50.601083Z" 2022-09-12T19:27:07.083 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:27:07.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:27:07.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:27:07.084 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:27:07.084 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:27:07.084 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:27:07.085 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000005999", 2022-09-12T19:27:07.085 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:27:07.085 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:27:07.085 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:27:07.086 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:26:50.602329Z" 2022-09-12T19:27:07.086 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:27:07.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:27:07.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:27:07.087 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:27:07.087 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:27:07.087 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:27:07.087 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000007553", 2022-09-12T19:27:07.088 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:27:07.088 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:27:07.088 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:27:07.088 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:26:50.603550Z" 2022-09-12T19:27:07.089 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:27:07.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:27:07.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:27:07.089 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:27:07.090 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:27:07.090 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T19:27:07.090 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-12T19:27:07.090 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:27:07.091 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:27:07.091 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:27:07.091 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:26:50.604778Z" 2022-09-12T19:27:07.092 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:27:07.092 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:27:07.092 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:27:07.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:27:07.093 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:27:07.093 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 0, 2022-09-12T19:27:07.093 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 0 2022-09-12T19:27:07.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:27:07.094 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:27:07Z" 2022-09-12T19:27:07.094 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:27:07.095 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 15445976120917781840 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003609", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:50.601083Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005999", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:50.602329Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007553", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:50.603550Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002060", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:26:50.604778Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T19:27:07Z" } 2022-09-12T19:27:07.096 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003609', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:50.601083Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005999', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:50.602329Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007553', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:50.603550Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:26:50.604778Z'}}] 2022-09-12T19:27:07.096 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003609', 1: '00000000000000000000:00000000000000005999', 2: '00000000000000000000:00000000000000007553', 3: '00000000000000000000:00000000000000002060'} 2022-09-12T19:27:07.097 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003609', 1: '00000000000000000000:00000000000000005999', 2: '00000000000000000000:00000000000000007553', 3: '00000000000000000000:00000000000000002060'} 2022-09-12T19:27:07.097 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2022-09-12T19:27:07.098 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone4 2022-09-12T19:27:07.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... ok 2022-09-12T19:27:07.103 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-59 2022-09-12T19:27:07.104 DEBUG:boto:path=/jdpiak-59/ 2022-09-12T19:27:07.105 DEBUG:boto:auth_path=/jdpiak-59/ 2022-09-12T19:27:07.107 DEBUG:boto:Method: PUT 2022-09-12T19:27:07.108 DEBUG:boto:Path: /jdpiak-59/ 2022-09-12T19:27:07.108 DEBUG:boto:Data: 2022-09-12T19:27:07.109 DEBUG:boto:Headers: {} 2022-09-12T19:27:07.110 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:27:07.111 DEBUG:boto:Port: 8000 2022-09-12T19:27:07.112 DEBUG:boto:Params: {} 2022-09-12T19:27:07.114 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:27:07.115 DEBUG:boto:Token: None 2022-09-12T19:27:07.116 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-59/ 2022-09-12T19:27:07.117 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1Xr2GkeAxk2WQVKQnTU/OSQcqag= 2022-09-12T19:27:07.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1Xr2GkeAxk2WQVKQnTU/OSQcqag=', 'Content-Length': '0'} 2022-09-12T19:27:07.143 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d8ccf90a0eb389c2-00631f880b-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.144 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jdpiak-60 2022-09-12T19:27:07.145 DEBUG:boto:path=/jdpiak-60/ 2022-09-12T19:27:07.146 DEBUG:boto:auth_path=/jdpiak-60/ 2022-09-12T19:27:07.147 DEBUG:boto:Method: PUT 2022-09-12T19:27:07.147 DEBUG:boto:Path: /jdpiak-60/ 2022-09-12T19:27:07.148 DEBUG:boto:Data: 2022-09-12T19:27:07.148 DEBUG:boto:Headers: {} 2022-09-12T19:27:07.149 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:27:07.149 DEBUG:boto:Port: 8000 2022-09-12T19:27:07.150 DEBUG:boto:Params: {} 2022-09-12T19:27:07.151 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:27:07.151 DEBUG:boto:Token: None 2022-09-12T19:27:07.152 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-60/ 2022-09-12T19:27:07.153 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NOMie/1hP4Z3JED4sxZgTrs2jZQ= 2022-09-12T19:27:07.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NOMie/1hP4Z3JED4sxZgTrs2jZQ=', 'Content-Length': '0'} 2022-09-12T19:27:07.198 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d6197ff06fda791e-00631f880b-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.199 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jdpiak-61 2022-09-12T19:27:07.200 DEBUG:boto:path=/jdpiak-61/ 2022-09-12T19:27:07.200 DEBUG:boto:auth_path=/jdpiak-61/ 2022-09-12T19:27:07.201 DEBUG:boto:Method: PUT 2022-09-12T19:27:07.202 DEBUG:boto:Path: /jdpiak-61/ 2022-09-12T19:27:07.202 DEBUG:boto:Data: 2022-09-12T19:27:07.203 DEBUG:boto:Headers: {} 2022-09-12T19:27:07.203 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:27:07.204 DEBUG:boto:Port: 8001 2022-09-12T19:27:07.204 DEBUG:boto:Params: {} 2022-09-12T19:27:07.205 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T19:27:07.206 DEBUG:boto:Token: None 2022-09-12T19:27:07.206 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-61/ 2022-09-12T19:27:07.207 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D8+zE5aveX2QXQQ+JtHwTFmw3Gc= 2022-09-12T19:27:07.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D8+zE5aveX2QXQQ+JtHwTFmw3Gc=', 'Content-Length': '0'} 2022-09-12T19:27:07.261 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b662dd364fc99d86-00631f880b-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.262 DEBUG:boto:path=/jdpiak-59/ 2022-09-12T19:27:07.263 DEBUG:boto:auth_path=/jdpiak-59/ 2022-09-12T19:27:07.263 DEBUG:boto:Method: HEAD 2022-09-12T19:27:07.264 DEBUG:boto:Path: /jdpiak-59/ 2022-09-12T19:27:07.265 DEBUG:boto:Data: 2022-09-12T19:27:07.265 DEBUG:boto:Headers: {} 2022-09-12T19:27:07.266 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:27:07.266 DEBUG:boto:Port: 8000 2022-09-12T19:27:07.267 DEBUG:boto:Params: {} 2022-09-12T19:27:07.267 DEBUG:boto:Token: None 2022-09-12T19:27:07.268 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-59/ 2022-09-12T19:27:07.269 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LIoDd01a4CwEDwE3Ofkl+9QYvLM= 2022-09-12T19:27:07.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LIoDd01a4CwEDwE3Ofkl+9QYvLM=', 'Content-Length': '0'} 2022-09-12T19:27:07.273 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000542bdad7c5ab275f-00631f880b-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.275 DEBUG:boto:path=/jdpiak-59/obj1 2022-09-12T19:27:07.276 DEBUG:boto:auth_path=/jdpiak-59/obj1 2022-09-12T19:27:07.277 DEBUG:boto:Method: PUT 2022-09-12T19:27:07.278 DEBUG:boto:Path: /jdpiak-59/obj1 2022-09-12T19:27:07.279 DEBUG:boto:Data: 2022-09-12T19:27:07.280 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T19:27:07.281 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:27:07.282 DEBUG:boto:Port: 8000 2022-09-12T19:27:07.283 DEBUG:boto:Params: {} 2022-09-12T19:27:07.284 DEBUG:boto:Token: None 2022-09-12T19:27:07.285 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-59/obj1 2022-09-12T19:27:07.286 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zH7cz0nKiSlUtkik3tNJd/BOSvo= 2022-09-12T19:27:07.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zH7cz0nKiSlUtkik3tNJd/BOSvo='} 2022-09-12T19:27:07.294 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a81975e71eb40363-00631f880b-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.295 DEBUG:boto:path=/jdpiak-59/ 2022-09-12T19:27:07.296 DEBUG:boto:auth_path=/jdpiak-59/ 2022-09-12T19:27:07.297 DEBUG:boto:Method: HEAD 2022-09-12T19:27:07.298 DEBUG:boto:Path: /jdpiak-59/ 2022-09-12T19:27:07.299 DEBUG:boto:Data: 2022-09-12T19:27:07.300 DEBUG:boto:Headers: {} 2022-09-12T19:27:07.301 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:27:07.302 DEBUG:boto:Port: 8000 2022-09-12T19:27:07.303 DEBUG:boto:Params: {} 2022-09-12T19:27:07.304 DEBUG:boto:Token: None 2022-09-12T19:27:07.305 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-59/ 2022-09-12T19:27:07.306 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LIoDd01a4CwEDwE3Ofkl+9QYvLM= 2022-09-12T19:27:07.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LIoDd01a4CwEDwE3Ofkl+9QYvLM=', 'Content-Length': '0'} 2022-09-12T19:27:07.309 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a3bbe2b679700b2-00631f880b-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.311 DEBUG:boto:path=/jdpiak-59/obj2 2022-09-12T19:27:07.312 DEBUG:boto:auth_path=/jdpiak-59/obj2 2022-09-12T19:27:07.312 DEBUG:boto:Method: PUT 2022-09-12T19:27:07.313 DEBUG:boto:Path: /jdpiak-59/obj2 2022-09-12T19:27:07.314 DEBUG:boto:Data: 2022-09-12T19:27:07.315 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T19:27:07.316 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:27:07.317 DEBUG:boto:Port: 8000 2022-09-12T19:27:07.318 DEBUG:boto:Params: {} 2022-09-12T19:27:07.319 DEBUG:boto:Token: None 2022-09-12T19:27:07.320 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-59/obj2 2022-09-12T19:27:07.321 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:abSJTUhj9DqYjZt3lF4j8/QpPCI= 2022-09-12T19:27:07.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:abSJTUhj9DqYjZt3lF4j8/QpPCI='} 2022-09-12T19:27:07.328 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b70ab042614bc3aa-00631f880b-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.329 DEBUG:boto:path=/jdpiak-59/ 2022-09-12T19:27:07.330 DEBUG:boto:auth_path=/jdpiak-59/ 2022-09-12T19:27:07.331 DEBUG:boto:Method: HEAD 2022-09-12T19:27:07.332 DEBUG:boto:Path: /jdpiak-59/ 2022-09-12T19:27:07.333 DEBUG:boto:Data: 2022-09-12T19:27:07.334 DEBUG:boto:Headers: {} 2022-09-12T19:27:07.335 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:27:07.335 DEBUG:boto:Port: 8000 2022-09-12T19:27:07.336 DEBUG:boto:Params: {} 2022-09-12T19:27:07.337 DEBUG:boto:Token: None 2022-09-12T19:27:07.338 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-59/ 2022-09-12T19:27:07.339 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LIoDd01a4CwEDwE3Ofkl+9QYvLM= 2022-09-12T19:27:07.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LIoDd01a4CwEDwE3Ofkl+9QYvLM=', 'Content-Length': '0'} 2022-09-12T19:27:07.342 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba8b3f80adc22ad3-00631f880b-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.343 DEBUG:boto:path=/jdpiak-59/obj3 2022-09-12T19:27:07.344 DEBUG:boto:auth_path=/jdpiak-59/obj3 2022-09-12T19:27:07.344 DEBUG:boto:Method: PUT 2022-09-12T19:27:07.345 DEBUG:boto:Path: /jdpiak-59/obj3 2022-09-12T19:27:07.345 DEBUG:boto:Data: 2022-09-12T19:27:07.346 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T19:27:07.346 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:27:07.347 DEBUG:boto:Port: 8000 2022-09-12T19:27:07.347 DEBUG:boto:Params: {} 2022-09-12T19:27:07.348 DEBUG:boto:Token: None 2022-09-12T19:27:07.348 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-59/obj3 2022-09-12T19:27:07.349 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1a50ACfhMu38l/+CPdDAwVgEhxo= 2022-09-12T19:27:07.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1a50ACfhMu38l/+CPdDAwVgEhxo='} 2022-09-12T19:27:07.394 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e4cdb22565bc944-00631f880b-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.395 DEBUG:boto:path=/jdpiak-59/ 2022-09-12T19:27:07.395 DEBUG:boto:auth_path=/jdpiak-59/ 2022-09-12T19:27:07.396 DEBUG:boto:Method: HEAD 2022-09-12T19:27:07.396 DEBUG:boto:Path: /jdpiak-59/ 2022-09-12T19:27:07.397 DEBUG:boto:Data: 2022-09-12T19:27:07.397 DEBUG:boto:Headers: {} 2022-09-12T19:27:07.398 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:27:07.398 DEBUG:boto:Port: 8000 2022-09-12T19:27:07.399 DEBUG:boto:Params: {} 2022-09-12T19:27:07.399 DEBUG:boto:Token: None 2022-09-12T19:27:07.400 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-59/ 2022-09-12T19:27:07.401 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LIoDd01a4CwEDwE3Ofkl+9QYvLM= 2022-09-12T19:27:07.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LIoDd01a4CwEDwE3Ofkl+9QYvLM=', 'Content-Length': '0'} 2022-09-12T19:27:07.403 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004bc4053835567e7f-00631f880b-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.404 DEBUG:boto:path=/jdpiak-59/obj4 2022-09-12T19:27:07.405 DEBUG:boto:auth_path=/jdpiak-59/obj4 2022-09-12T19:27:07.405 DEBUG:boto:Method: PUT 2022-09-12T19:27:07.406 DEBUG:boto:Path: /jdpiak-59/obj4 2022-09-12T19:27:07.406 DEBUG:boto:Data: 2022-09-12T19:27:07.407 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T19:27:07.407 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:27:07.408 DEBUG:boto:Port: 8000 2022-09-12T19:27:07.408 DEBUG:boto:Params: {} 2022-09-12T19:27:07.409 DEBUG:boto:Token: None 2022-09-12T19:27:07.409 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-59/obj4 2022-09-12T19:27:07.410 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Jv4oek4+81Fmdzj0ASzkemtLYiU= 2022-09-12T19:27:07.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Jv4oek4+81Fmdzj0ASzkemtLYiU='} 2022-09-12T19:27:07.420 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000065788cfa04fe5e97-00631f880b-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.420 DEBUG:boto:path=/jdpiak-60/ 2022-09-12T19:27:07.421 DEBUG:boto:auth_path=/jdpiak-60/ 2022-09-12T19:27:07.421 DEBUG:boto:Method: HEAD 2022-09-12T19:27:07.422 DEBUG:boto:Path: /jdpiak-60/ 2022-09-12T19:27:07.422 DEBUG:boto:Data: 2022-09-12T19:27:07.423 DEBUG:boto:Headers: {} 2022-09-12T19:27:07.423 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:27:07.424 DEBUG:boto:Port: 8000 2022-09-12T19:27:07.424 DEBUG:boto:Params: {} 2022-09-12T19:27:07.425 DEBUG:boto:Token: None 2022-09-12T19:27:07.425 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-60/ 2022-09-12T19:27:07.426 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Lfw43h0Xtu5SOPqeW2DdrjdQF/8= 2022-09-12T19:27:07.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Lfw43h0Xtu5SOPqeW2DdrjdQF/8=', 'Content-Length': '0'} 2022-09-12T19:27:07.430 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ce16cd4b3dad3d8-00631f880b-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.431 DEBUG:boto:path=/jdpiak-60/obj1 2022-09-12T19:27:07.432 DEBUG:boto:auth_path=/jdpiak-60/obj1 2022-09-12T19:27:07.433 DEBUG:boto:Method: PUT 2022-09-12T19:27:07.434 DEBUG:boto:Path: /jdpiak-60/obj1 2022-09-12T19:27:07.435 DEBUG:boto:Data: 2022-09-12T19:27:07.436 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T19:27:07.437 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:27:07.437 DEBUG:boto:Port: 8000 2022-09-12T19:27:07.438 DEBUG:boto:Params: {} 2022-09-12T19:27:07.440 DEBUG:boto:Token: None 2022-09-12T19:27:07.441 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-60/obj1 2022-09-12T19:27:07.442 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MJgFSwwhfTtbeU/syOHan5R0vtQ= 2022-09-12T19:27:07.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MJgFSwwhfTtbeU/syOHan5R0vtQ='} 2022-09-12T19:27:07.453 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000050c6c1980e861763-00631f880b-15e1-test-zone2'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.454 DEBUG:boto:path=/jdpiak-60/ 2022-09-12T19:27:07.454 DEBUG:boto:auth_path=/jdpiak-60/ 2022-09-12T19:27:07.455 DEBUG:boto:Method: HEAD 2022-09-12T19:27:07.455 DEBUG:boto:Path: /jdpiak-60/ 2022-09-12T19:27:07.456 DEBUG:boto:Data: 2022-09-12T19:27:07.456 DEBUG:boto:Headers: {} 2022-09-12T19:27:07.457 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:27:07.457 DEBUG:boto:Port: 8000 2022-09-12T19:27:07.457 DEBUG:boto:Params: {} 2022-09-12T19:27:07.458 DEBUG:boto:Token: None 2022-09-12T19:27:07.459 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-60/ 2022-09-12T19:27:07.459 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Lfw43h0Xtu5SOPqeW2DdrjdQF/8= 2022-09-12T19:27:07.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Lfw43h0Xtu5SOPqeW2DdrjdQF/8=', 'Content-Length': '0'} 2022-09-12T19:27:07.462 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002fcd2b4e4e786485-00631f880b-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.463 DEBUG:boto:path=/jdpiak-60/obj2 2022-09-12T19:27:07.463 DEBUG:boto:auth_path=/jdpiak-60/obj2 2022-09-12T19:27:07.464 DEBUG:boto:Method: PUT 2022-09-12T19:27:07.464 DEBUG:boto:Path: /jdpiak-60/obj2 2022-09-12T19:27:07.465 DEBUG:boto:Data: 2022-09-12T19:27:07.465 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T19:27:07.466 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:27:07.466 DEBUG:boto:Port: 8000 2022-09-12T19:27:07.467 DEBUG:boto:Params: {} 2022-09-12T19:27:07.467 DEBUG:boto:Token: None 2022-09-12T19:27:07.468 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-60/obj2 2022-09-12T19:27:07.468 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:do4r/MfyAlJSZmCRTXmS/QzReGg= 2022-09-12T19:27:07.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:do4r/MfyAlJSZmCRTXmS/QzReGg='} 2022-09-12T19:27:07.483 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077cfd589fc5b1b07-00631f880b-15e1-test-zone2'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.483 DEBUG:boto:path=/jdpiak-60/ 2022-09-12T19:27:07.484 DEBUG:boto:auth_path=/jdpiak-60/ 2022-09-12T19:27:07.484 DEBUG:boto:Method: HEAD 2022-09-12T19:27:07.485 DEBUG:boto:Path: /jdpiak-60/ 2022-09-12T19:27:07.485 DEBUG:boto:Data: 2022-09-12T19:27:07.486 DEBUG:boto:Headers: {} 2022-09-12T19:27:07.486 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:27:07.487 DEBUG:boto:Port: 8000 2022-09-12T19:27:07.487 DEBUG:boto:Params: {} 2022-09-12T19:27:07.488 DEBUG:boto:Token: None 2022-09-12T19:27:07.488 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-60/ 2022-09-12T19:27:07.489 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Lfw43h0Xtu5SOPqeW2DdrjdQF/8= 2022-09-12T19:27:07.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Lfw43h0Xtu5SOPqeW2DdrjdQF/8=', 'Content-Length': '0'} 2022-09-12T19:27:07.492 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca6d12ce542a39ab-00631f880b-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.493 DEBUG:boto:path=/jdpiak-60/obj3 2022-09-12T19:27:07.493 DEBUG:boto:auth_path=/jdpiak-60/obj3 2022-09-12T19:27:07.494 DEBUG:boto:Method: PUT 2022-09-12T19:27:07.494 DEBUG:boto:Path: /jdpiak-60/obj3 2022-09-12T19:27:07.495 DEBUG:boto:Data: 2022-09-12T19:27:07.495 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T19:27:07.496 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:27:07.496 DEBUG:boto:Port: 8000 2022-09-12T19:27:07.497 DEBUG:boto:Params: {} 2022-09-12T19:27:07.497 DEBUG:boto:Token: None 2022-09-12T19:27:07.498 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-60/obj3 2022-09-12T19:27:07.498 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XXQLY+oiFeiznUDf6GzfvGBCd4w= 2022-09-12T19:27:07.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XXQLY+oiFeiznUDf6GzfvGBCd4w='} 2022-09-12T19:27:07.542 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006aa0c10d15432a5f-00631f880b-15e1-test-zone2'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.544 DEBUG:boto:path=/jdpiak-60/ 2022-09-12T19:27:07.544 DEBUG:boto:auth_path=/jdpiak-60/ 2022-09-12T19:27:07.545 DEBUG:boto:Method: HEAD 2022-09-12T19:27:07.546 DEBUG:boto:Path: /jdpiak-60/ 2022-09-12T19:27:07.547 DEBUG:boto:Data: 2022-09-12T19:27:07.548 DEBUG:boto:Headers: {} 2022-09-12T19:27:07.548 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:27:07.549 DEBUG:boto:Port: 8000 2022-09-12T19:27:07.550 DEBUG:boto:Params: {} 2022-09-12T19:27:07.551 DEBUG:boto:Token: None 2022-09-12T19:27:07.552 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-60/ 2022-09-12T19:27:07.553 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Lfw43h0Xtu5SOPqeW2DdrjdQF/8= 2022-09-12T19:27:07.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Lfw43h0Xtu5SOPqeW2DdrjdQF/8=', 'Content-Length': '0'} 2022-09-12T19:27:07.557 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000585f4f705134e943-00631f880b-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.558 DEBUG:boto:path=/jdpiak-60/obj4 2022-09-12T19:27:07.559 DEBUG:boto:auth_path=/jdpiak-60/obj4 2022-09-12T19:27:07.560 DEBUG:boto:Method: PUT 2022-09-12T19:27:07.560 DEBUG:boto:Path: /jdpiak-60/obj4 2022-09-12T19:27:07.561 DEBUG:boto:Data: 2022-09-12T19:27:07.562 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T19:27:07.563 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:27:07.564 DEBUG:boto:Port: 8000 2022-09-12T19:27:07.564 DEBUG:boto:Params: {} 2022-09-12T19:27:07.565 DEBUG:boto:Token: None 2022-09-12T19:27:07.566 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-60/obj4 2022-09-12T19:27:07.567 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pIgQ8FQPOQoM17yHdG/Pxg4Jerg= 2022-09-12T19:27:07.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pIgQ8FQPOQoM17yHdG/Pxg4Jerg='} 2022-09-12T19:27:07.576 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b0e3f5f7c8c7d18-00631f880b-15e1-test-zone2'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.577 DEBUG:boto:path=/jdpiak-61/ 2022-09-12T19:27:07.577 DEBUG:boto:auth_path=/jdpiak-61/ 2022-09-12T19:27:07.578 DEBUG:boto:Method: HEAD 2022-09-12T19:27:07.579 DEBUG:boto:Path: /jdpiak-61/ 2022-09-12T19:27:07.580 DEBUG:boto:Data: 2022-09-12T19:27:07.581 DEBUG:boto:Headers: {} 2022-09-12T19:27:07.581 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:27:07.582 DEBUG:boto:Port: 8001 2022-09-12T19:27:07.583 DEBUG:boto:Params: {} 2022-09-12T19:27:07.584 DEBUG:boto:Token: None 2022-09-12T19:27:07.585 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-61/ 2022-09-12T19:27:07.586 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vN9TQrDUKDekuLHAOJYF4F9y5To= 2022-09-12T19:27:07.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vN9TQrDUKDekuLHAOJYF4F9y5To=', 'Content-Length': '0'} 2022-09-12T19:27:07.589 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038aec9e5d3d7cbec-00631f880b-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.591 DEBUG:boto:path=/jdpiak-61/obj1 2022-09-12T19:27:07.591 DEBUG:boto:auth_path=/jdpiak-61/obj1 2022-09-12T19:27:07.592 DEBUG:boto:Method: PUT 2022-09-12T19:27:07.593 DEBUG:boto:Path: /jdpiak-61/obj1 2022-09-12T19:27:07.594 DEBUG:boto:Data: 2022-09-12T19:27:07.595 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T19:27:07.595 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:27:07.596 DEBUG:boto:Port: 8001 2022-09-12T19:27:07.597 DEBUG:boto:Params: {} 2022-09-12T19:27:07.598 DEBUG:boto:Token: None 2022-09-12T19:27:07.599 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-61/obj1 2022-09-12T19:27:07.600 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Rh7KG4QpzfN2L8Ca6njQFyCv1yA= 2022-09-12T19:27:07.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Rh7KG4QpzfN2L8Ca6njQFyCv1yA='} 2022-09-12T19:27:07.610 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fac166f1f3996b4c-00631f880b-15c4-test-zone3'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.611 DEBUG:boto:path=/jdpiak-61/ 2022-09-12T19:27:07.612 DEBUG:boto:auth_path=/jdpiak-61/ 2022-09-12T19:27:07.612 DEBUG:boto:Method: HEAD 2022-09-12T19:27:07.613 DEBUG:boto:Path: /jdpiak-61/ 2022-09-12T19:27:07.614 DEBUG:boto:Data: 2022-09-12T19:27:07.615 DEBUG:boto:Headers: {} 2022-09-12T19:27:07.615 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:27:07.616 DEBUG:boto:Port: 8001 2022-09-12T19:27:07.616 DEBUG:boto:Params: {} 2022-09-12T19:27:07.617 DEBUG:boto:Token: None 2022-09-12T19:27:07.617 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-61/ 2022-09-12T19:27:07.618 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vN9TQrDUKDekuLHAOJYF4F9y5To= 2022-09-12T19:27:07.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vN9TQrDUKDekuLHAOJYF4F9y5To=', 'Content-Length': '0'} 2022-09-12T19:27:07.621 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a792ab507e350e6-00631f880b-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.622 DEBUG:boto:path=/jdpiak-61/obj2 2022-09-12T19:27:07.623 DEBUG:boto:auth_path=/jdpiak-61/obj2 2022-09-12T19:27:07.624 DEBUG:boto:Method: PUT 2022-09-12T19:27:07.625 DEBUG:boto:Path: /jdpiak-61/obj2 2022-09-12T19:27:07.625 DEBUG:boto:Data: 2022-09-12T19:27:07.626 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T19:27:07.626 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:27:07.627 DEBUG:boto:Port: 8001 2022-09-12T19:27:07.627 DEBUG:boto:Params: {} 2022-09-12T19:27:07.628 DEBUG:boto:Token: None 2022-09-12T19:27:07.628 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-61/obj2 2022-09-12T19:27:07.629 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:voqMqQmsC+a4yrH+rGy89R3gQcg= 2022-09-12T19:27:07.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:voqMqQmsC+a4yrH+rGy89R3gQcg='} 2022-09-12T19:27:07.641 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aa61f0085e2677ea-00631f880b-15c4-test-zone3'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.641 DEBUG:boto:path=/jdpiak-61/ 2022-09-12T19:27:07.642 DEBUG:boto:auth_path=/jdpiak-61/ 2022-09-12T19:27:07.643 DEBUG:boto:Method: HEAD 2022-09-12T19:27:07.643 DEBUG:boto:Path: /jdpiak-61/ 2022-09-12T19:27:07.643 DEBUG:boto:Data: 2022-09-12T19:27:07.644 DEBUG:boto:Headers: {} 2022-09-12T19:27:07.644 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:27:07.645 DEBUG:boto:Port: 8001 2022-09-12T19:27:07.645 DEBUG:boto:Params: {} 2022-09-12T19:27:07.646 DEBUG:boto:Token: None 2022-09-12T19:27:07.647 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-61/ 2022-09-12T19:27:07.647 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vN9TQrDUKDekuLHAOJYF4F9y5To= 2022-09-12T19:27:07.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vN9TQrDUKDekuLHAOJYF4F9y5To=', 'Content-Length': '0'} 2022-09-12T19:27:07.650 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020c14488ddf04dd9-00631f880b-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.651 DEBUG:boto:path=/jdpiak-61/obj3 2022-09-12T19:27:07.651 DEBUG:boto:auth_path=/jdpiak-61/obj3 2022-09-12T19:27:07.652 DEBUG:boto:Method: PUT 2022-09-12T19:27:07.652 DEBUG:boto:Path: /jdpiak-61/obj3 2022-09-12T19:27:07.653 DEBUG:boto:Data: 2022-09-12T19:27:07.653 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T19:27:07.654 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:27:07.654 DEBUG:boto:Port: 8001 2022-09-12T19:27:07.655 DEBUG:boto:Params: {} 2022-09-12T19:27:07.655 DEBUG:boto:Token: None 2022-09-12T19:27:07.656 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-61/obj3 2022-09-12T19:27:07.656 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zlmMvGNjknekWMfsDWB2jp5HqtA= 2022-09-12T19:27:07.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zlmMvGNjknekWMfsDWB2jp5HqtA='} 2022-09-12T19:27:07.702 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000035ea158fdeb3a639-00631f880b-15c4-test-zone3'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.703 DEBUG:boto:path=/jdpiak-61/ 2022-09-12T19:27:07.703 DEBUG:boto:auth_path=/jdpiak-61/ 2022-09-12T19:27:07.704 DEBUG:boto:Method: HEAD 2022-09-12T19:27:07.704 DEBUG:boto:Path: /jdpiak-61/ 2022-09-12T19:27:07.705 DEBUG:boto:Data: 2022-09-12T19:27:07.706 DEBUG:boto:Headers: {} 2022-09-12T19:27:07.707 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:27:07.707 DEBUG:boto:Port: 8001 2022-09-12T19:27:07.708 DEBUG:boto:Params: {} 2022-09-12T19:27:07.708 DEBUG:boto:Token: None 2022-09-12T19:27:07.709 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-61/ 2022-09-12T19:27:07.709 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vN9TQrDUKDekuLHAOJYF4F9y5To= 2022-09-12T19:27:07.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vN9TQrDUKDekuLHAOJYF4F9y5To=', 'Content-Length': '0'} 2022-09-12T19:27:07.712 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a90a8209bd972dd3-00631f880b-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.713 DEBUG:boto:path=/jdpiak-61/obj4 2022-09-12T19:27:07.713 DEBUG:boto:auth_path=/jdpiak-61/obj4 2022-09-12T19:27:07.714 DEBUG:boto:Method: PUT 2022-09-12T19:27:07.715 DEBUG:boto:Path: /jdpiak-61/obj4 2022-09-12T19:27:07.715 DEBUG:boto:Data: 2022-09-12T19:27:07.716 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T19:27:07.716 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:27:07.717 DEBUG:boto:Port: 8001 2022-09-12T19:27:07.717 DEBUG:boto:Params: {} 2022-09-12T19:27:07.718 DEBUG:boto:Token: None 2022-09-12T19:27:07.718 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 19:27:07 GMT /jdpiak-61/obj4 2022-09-12T19:27:07.719 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xDgUsiVL6GlpCMk3RYvt+PsvIk0= 2022-09-12T19:27:07.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:27:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xDgUsiVL6GlpCMk3RYvt+PsvIk0='} 2022-09-12T19:27:07.726 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001733eb6ec67c3489-00631f880b-15c4-test-zone3'), ('Date', 'Mon, 12 Sep 2022 19:27:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:27:07.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:27:07.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:27:07.728 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:27:07.777 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:27:07.778 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:27:07.807 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=38ca1f57-1358-496c-a99b-df49ee365a4f 2022-09-12T19:27:07.823 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:27:07.823 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:27:07.823 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:27:07.823 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:27:07.824 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:27:07.824 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:27:07.824 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:27:07.824 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:27:07.825 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:27:07.825 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:27:07.825 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663010827.242912_474.1", 2022-09-12T19:27:07.826 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:27:07.242912Z" 2022-09-12T19:27:07.826 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:27:07.826 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:27:07.827 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:27:07.827 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:27:07.828 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:27:07.828 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:27:07.829 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1663010827.242912_474.1', 3: ''} 2022-09-12T19:27:07.830 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T19:27:07.831 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:27:07.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:27:07.833 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:27:07.884 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:27:07.884 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:27:07.928 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:27:07.926+0000 7ffba4da4600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:27:07.929 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:27:07.927+0000 7ffba4da4600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:27:07.944 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:27:07.945 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:27:07.945 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:27:07.946 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:27:07.946 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:27:07.946 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:27:07.947 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:27:07.947 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:27:07.948 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:27:07.948 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:27:07.949 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:27:07.949 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:27:07.949 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:27:07.950 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:27:07.950 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:27:07.951 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:27:07.951 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:27:07.951 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:27:07.951 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:27:07.952 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:27:07.952 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:27:07.952 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:27:07.952 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:27:07.953 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:27:07.953 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:27:07.953 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:27:07.954 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:27:07.954 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:27:07.954 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:27:07.954 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:27:07.955 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:27:07.955 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:27:07.955 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:27:07.955 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:27:07.956 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:27:07.956 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:27:07.956 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:27:07.956 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010827.230179_473.1", 2022-09-12T19:27:07.957 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:27:07.957 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:27:07.957 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:27:07.957 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:27:07.230179Z", 2022-09-12T19:27:07.958 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:27:07.958 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:27:07.959 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:27:07.959 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:27:07.959 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:27:07.959 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:27:07.959 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:27:07.960 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:27:07.960 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:27:07.960 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:27:07.960 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:27:07.961 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:27:07.961 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:27:07.961 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:27:07.961 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:27:07.962 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:27:07.962 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:27:07.962 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:27:07.963 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:27:07.963 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:27:07.963 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:27:07.963 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:27:07Z" 2022-09-12T19:27:07.964 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:27:07.965 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010827.230179_473.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:27:07.230179Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:27:07Z" } 2022-09-12T19:27:07.965 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010827.230179_473.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:27:07.230179Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:27:07.966 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010827.242912_474.1', 3: ''} 2022-09-12T19:27:07.967 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010827.230179_473.1', 3: ''} 2022-09-12T19:27:07.967 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663010827.242912_474.1 target=1_1663010827.230179_473.1 2022-09-12T19:27:12.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:27:12.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:27:12.971 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:27:13.027 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:27:13.027 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:27:13.066 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:27:13.065+0000 7fc4cbd88600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:27:13.067 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:27:13.066+0000 7fc4cbd88600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:27:13.082 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:27:13.082 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:27:13.083 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:27:13.083 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:27:13.084 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:27:13.084 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:27:13.085 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:27:13.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:27:13.086 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:27:13.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:27:13.087 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:27:13.087 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:27:13.088 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:27:13.088 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:27:13.088 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:27:13.088 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:27:13.089 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:27:13.089 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:27:13.090 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:27:13.091 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:27:13.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:27:13.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:27:13.092 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:27:13.093 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:27:13.093 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:27:13.094 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:27:13.094 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:27:13.095 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:27:13.095 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:27:13.096 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:27:13.096 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:27:13.097 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:27:13.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:27:13.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:27:13.099 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:27:13.099 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:27:13.100 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:27:13.100 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010827.242912_474.1", 2022-09-12T19:27:13.101 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:27:13.101 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:27:13.102 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:27:13.102 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:27:07.242912Z", 2022-09-12T19:27:13.103 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:27:13.103 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:27:13.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:27:13.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:27:13.104 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:27:13.105 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:27:13.105 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:27:13.106 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:27:13.106 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:27:13.107 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:27:13.108 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:27:13.108 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:27:13.109 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:27:13.109 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:27:13.110 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:27:13.110 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:27:13.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:27:13.111 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:27:13.112 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:27:13.112 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:27:13.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:27:13.113 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:27:13Z" 2022-09-12T19:27:13.114 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:27:13.116 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010827.242912_474.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:27:07.242912Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:27:13Z" } 2022-09-12T19:27:13.117 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010827.242912_474.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:27:07.242912Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:27:13.119 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010827.242912_474.1', 3: ''} 2022-09-12T19:27:13.120 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010827.242912_474.1', 3: ''} 2022-09-12T19:27:13.120 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T19:27:13.121 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T19:27:13.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:27:13.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:27:13.124 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:27:13.184 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:27:13.184 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:27:13.218 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:27:13.215+0000 7fac60460600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:27:13.219 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:27:13.217+0000 7fac60460600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:27:13.237 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:27:13.237 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:27:13.238 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:27:13.238 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:27:13.238 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:27:13.239 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:27:13.239 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:27:13.239 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:27:13.239 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:27:13.240 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:27:13.240 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:27:13.240 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:27:13.240 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:27:13.241 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:27:13.241 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:27:13.241 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:27:13.242 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:27:13.242 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:27:13.242 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:27:13.242 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:27:13.243 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:27:13.243 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:27:13.243 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:27:13.244 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:27:13.244 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:27:13.244 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:27:13.245 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:27:13.245 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:27:13.245 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:27:13.245 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:27:13.246 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:27:13.246 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:27:13.246 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:27:13.247 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:27:13.247 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:27:13.247 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:27:13.247 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:27:13.248 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663010827.242912_474.1", 2022-09-12T19:27:13.248 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:27:13.248 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:27:13.248 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:27:13.249 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:27:07.242912Z", 2022-09-12T19:27:13.249 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:27:13.249 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:27:13.249 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:27:13.250 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:27:13.250 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:27:13.250 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:27:13.250 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:27:13.251 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:27:13.251 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:27:13.251 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:27:13.251 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:27:13.252 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:27:13.252 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:27:13.252 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:27:13.252 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:27:13.253 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:27:13.253 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:27:13.254 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:27:13.255 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:27:13.255 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:27:13.255 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:27:13.255 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:27:13Z" 2022-09-12T19:27:13.256 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:27:13.257 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010827.242912_474.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:27:07.242912Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:27:13Z" } 2022-09-12T19:27:13.258 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010827.242912_474.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:27:07.242912Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:27:13.258 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010827.242912_474.1', 3: ''} 2022-09-12T19:27:13.259 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010827.242912_474.1', 3: ''} 2022-09-12T19:27:13.259 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T19:27:13.260 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T19:27:13.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:27:13.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:27:13.261 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:27:13.313 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:27:13.313 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:27:13.342 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:27:13.340+0000 7f9cf3d26600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:27:13.343 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:27:13.341+0000 7f9cf3d26600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:27:13.354 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:27:13.355 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:27:13.355 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:27:13.355 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:27:13.356 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:27:13.356 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:27:13.356 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:27:13.357 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:27:13.357 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:27:13.357 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:27:13.358 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:27:13.358 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:27:13.358 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:27:13.359 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:27:13.360 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:27:13.360 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:27:13.361 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:27:13.361 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:27:13.362 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:27:13.362 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:27:13.362 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:27:13.363 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:27:13.363 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:27:13.363 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:27:13.364 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:27:13.364 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:27:13.364 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:27:13.365 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:27:13.365 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:27:13.365 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:27:13.366 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:27:13.366 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:27:13.367 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:27:13.367 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:27:13.367 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:27:13.368 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:27:13.368 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:27:13.368 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663010827.242912_474.1", 2022-09-12T19:27:13.369 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:27:13.369 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:27:13.369 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:27:13.369 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:27:07.242912Z", 2022-09-12T19:27:13.370 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:27:13.370 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:27:13.370 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:27:13.370 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:27:13.371 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:27:13.371 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:27:13.371 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:27:13.372 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:27:13.372 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:27:13.373 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:27:13.373 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:27:13.374 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:27:13.374 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:27:13.374 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:27:13.374 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:27:13.375 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:27:13.375 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:27:13.375 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:27:13.375 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:27:13.376 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:27:13.376 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:27:13.376 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:27:13Z" 2022-09-12T19:27:13.377 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:27:13.379 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663010827.242912_474.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:27:07.242912Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:27:13Z" } 2022-09-12T19:27:13.379 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663010827.242912_474.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:27:07.242912Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:27:13.380 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663010827.242912_474.1', 3: ''} 2022-09-12T19:27:13.380 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663010827.242912_474.1', 3: ''} 2022-09-12T19:27:13.381 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T19:27:13.382 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-59 2022-09-12T19:27:13.383 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:27:13.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:27:13.384 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:27:13.439 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:27:13.440 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:27:13.487 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:27:13.485+0000 7f9d8818a600 1 waiting to reach incremental sync.. 2022-09-12T19:27:18.489 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:27:18.486+0000 7f9d8818a600 1 bucket sync caught up with source: 2022-09-12T19:27:18.490 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:27:18.490 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:27:18.491 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:27:18.486+0000 7f9d8818a600 0 bucket checkpoint complete 2022-09-12T19:27:18.508 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-59 2022-09-12T19:27:18.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:27:18.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:27:18.513 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:27:18.580 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:27:18.581 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:27:18.625 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:27:18.620+0000 7f8b50589600 1 bucket sync caught up with source: 2022-09-12T19:27:18.625 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:27:18.626 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:27:18.626 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:27:18.620+0000 7f8b50589600 0 bucket checkpoint complete 2022-09-12T19:27:18.639 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-59 2022-09-12T19:27:18.640 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:27:18.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:27:18.642 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:27:18.704 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:27:18.705 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:27:18.755 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:27:18.752+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:27:18.755 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:27:18.756 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:27:23.756 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:27:23.753+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:27:23.757 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:27:23.757 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:27:28.757 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:27:28.755+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:27:28.758 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:27:28.758 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:27:33.773 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:27:33.756+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:27:33.773 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:27:33.774 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:27:38.760 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:27:38.757+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:27:38.761 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:27:38.762 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:27:43.762 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:27:43.759+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:27:43.763 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:27:43.763 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:27:48.764 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:27:48.760+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:27:48.765 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:27:48.765 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:27:53.765 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:27:53.762+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:27:53.765 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:27:53.766 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:27:58.767 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:27:58.764+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:27:58.767 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:27:58.767 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:28:03.768 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:28:03.765+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:28:03.768 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:28:03.769 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:28:08.769 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:28:08.766+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:28:08.770 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:28:08.770 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:28:13.771 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:28:13.768+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:28:13.771 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:28:13.771 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:28:18.772 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:28:18.769+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:28:18.772 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:28:18.773 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:28:23.773 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:28:23.770+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:28:23.774 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:28:23.775 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:28:28.776 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:28:28.772+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:28:28.777 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:28:28.777 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:28:33.776 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:28:33.773+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:28:33.777 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:28:33.777 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:28:38.778 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:28:38.775+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:28:38.779 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:28:38.781 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:28:43.779 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:28:43.777+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:28:43.780 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:28:43.780 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:28:48.780 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:28:48.778+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:28:48.781 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:28:48.781 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:28:53.782 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:28:53.779+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:28:53.783 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:28:53.783 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:28:58.783 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:28:58.781+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:28:58.784 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:28:58.784 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:29:03.785 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:29:03.782+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:29:03.786 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:29:03.787 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:29:08.787 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:29:08.784+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:29:08.787 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:29:08.788 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:29:13.788 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:29:13.785+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:29:13.789 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:29:13.789 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:29:18.790 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:29:18.787+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:29:18.790 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:29:18.790 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:29:23.791 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:29:23.788+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:29:23.792 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:29:23.792 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:29:28.792 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:29:28.790+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:29:28.793 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:29:28.793 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:29:33.793 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:29:33.791+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:29:33.794 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:29:33.795 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:29:38.795 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:29:38.792+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:29:38.796 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:29:38.796 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:29:43.796 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:29:43.793+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:29:43.797 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:29:43.797 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:29:48.799 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:29:48.795+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:29:48.799 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:29:48.800 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:29:53.799 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:29:53.796+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:29:53.800 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:29:53.800 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:29:58.800 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:29:58.798+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:29:58.801 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:29:58.801 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:30:03.802 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:30:03.799+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:30:03.803 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:30:03.803 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:30:08.803 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:30:08.801+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:30:08.804 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:30:08.804 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:30:13.805 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:30:13.803+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:30:13.806 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:30:13.807 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:30:18.806 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:30:18.803+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:30:18.807 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:30:18.807 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:30:23.808 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:30:23.805+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:30:23.809 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:30:23.809 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:30:28.809 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:30:28.807+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:30:28.810 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:30:28.810 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:30:33.811 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:30:33.808+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:30:33.811 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:30:33.812 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:30:38.812 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:30:38.809+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:30:38.813 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:30:38.813 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:30:43.813 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:30:43.811+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:30:43.814 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:30:43.814 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:30:48.816 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:30:48.812+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:30:48.816 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:30:48.817 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:30:53.816 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:30:53.813+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:30:53.816 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:30:53.816 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:30:58.817 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:30:58.815+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:30:58.818 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:30:58.818 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:31:03.818 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:31:03.816+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:31:03.819 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:31:03.819 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:31:08.820 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:31:08.817+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:31:08.821 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:31:08.821 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:31:13.822 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:31:13.818+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:31:13.822 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:31:13.822 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:31:18.821 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:31:18.819+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:31:18.822 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:31:18.822 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:31:23.823 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:31:23.820+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:31:23.824 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:31:23.825 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:31:28.825 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:31:28.822+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:31:28.825 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:31:28.825 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:31:33.826 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:31:33.823+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:31:33.827 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:31:33.827 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:31:38.828 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:31:38.825+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:31:38.828 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:31:38.828 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:31:43.829 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:31:43.827+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:31:43.830 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:31:43.830 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:31:48.830 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:31:48.828+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:31:48.831 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:31:48.831 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:31:53.832 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:31:53.829+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:31:53.833 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:31:53.833 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:31:58.833 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:31:58.830+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:31:58.834 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:31:58.834 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:32:03.835 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:32:03.832+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:32:03.835 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:32:03.835 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:32:08.836 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:32:08.833+0000 7fd6c6e71600 1 waiting for incremental sync to catch up: 2022-09-12T19:32:08.837 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.58.5, , 00000000001.74.5, , , , , 00000000001.60.5, , , ] 2022-09-12T19:32:08.838 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.58.5, , 00000000001.74.5, , 00000000001.58.5, , , 00000000001.60.5, , , ] 2022-09-12T19:32:13.837 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:32:13.835+0000 7fd6c6e71600 0 bucket checkpoint timed out waiting for incremental sync to catch up 2022-09-12T19:32:13.838 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:32:13.835+0000 7fd6c6e71600 0 bucket sync checkpoint failed: (110) Connection timed out 2022-09-12T19:32:13.839 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:32:13.835+0000 7fd6c6e71600 -1 bucket sync checkpoint failed: (110) Connection timed out 2022-09-12T19:32:13.859 DEBUG:teuthology.orchestra.run:got remote process result: 110 2022-09-12T19:32:13.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... FAIL 2022-09-12T19:32:13.868 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-62 2022-09-12T19:32:13.869 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:32:13.870 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:32:13.870 DEBUG:boto:Method: PUT 2022-09-12T19:32:13.871 DEBUG:boto:Path: /jdpiak-62/ 2022-09-12T19:32:13.871 DEBUG:boto:Data: 2022-09-12T19:32:13.872 DEBUG:boto:Headers: {} 2022-09-12T19:32:13.872 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:32:13.873 DEBUG:boto:Port: 8000 2022-09-12T19:32:13.873 DEBUG:boto:Params: {} 2022-09-12T19:32:13.875 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:32:13.875 DEBUG:boto:Token: None 2022-09-12T19:32:13.876 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:32:13 GMT /jdpiak-62/ 2022-09-12T19:32:13.877 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:69ByRjBjywTw614cWsk003SU158= 2022-09-12T19:32:13.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:32:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:69ByRjBjywTw614cWsk003SU158=', 'Content-Length': '0'} 2022-09-12T19:32:13.910 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a2a40ed9f7088d08-00631f893d-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:13.911 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jdpiak-63 2022-09-12T19:32:13.912 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:32:13.912 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:32:13.913 DEBUG:boto:Method: PUT 2022-09-12T19:32:13.913 DEBUG:boto:Path: /jdpiak-63/ 2022-09-12T19:32:13.914 DEBUG:boto:Data: 2022-09-12T19:32:13.915 DEBUG:boto:Headers: {} 2022-09-12T19:32:13.915 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:32:13.916 DEBUG:boto:Port: 8000 2022-09-12T19:32:13.916 DEBUG:boto:Params: {} 2022-09-12T19:32:13.917 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:32:13.918 DEBUG:boto:Token: None 2022-09-12T19:32:13.918 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:32:13 GMT /jdpiak-63/ 2022-09-12T19:32:13.919 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6LS+Y9E//g4c7fKDnuu864neF78= 2022-09-12T19:32:13.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:32:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6LS+Y9E//g4c7fKDnuu864neF78=', 'Content-Length': '0'} 2022-09-12T19:32:13.965 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008b6d6454c7f8987b-00631f893d-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:13.966 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jdpiak-64 2022-09-12T19:32:13.967 DEBUG:boto:path=/jdpiak-64/ 2022-09-12T19:32:13.967 DEBUG:boto:auth_path=/jdpiak-64/ 2022-09-12T19:32:13.968 DEBUG:boto:Method: PUT 2022-09-12T19:32:13.969 DEBUG:boto:Path: /jdpiak-64/ 2022-09-12T19:32:13.969 DEBUG:boto:Data: 2022-09-12T19:32:13.970 DEBUG:boto:Headers: {} 2022-09-12T19:32:13.970 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:32:13.971 DEBUG:boto:Port: 8001 2022-09-12T19:32:13.971 DEBUG:boto:Params: {} 2022-09-12T19:32:13.972 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T19:32:13.973 DEBUG:boto:Token: None 2022-09-12T19:32:13.973 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:32:13 GMT /jdpiak-64/ 2022-09-12T19:32:13.974 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zegry1GBjWzSyH+PwiY4lSErlLY= 2022-09-12T19:32:13.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:32:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zegry1GBjWzSyH+PwiY4lSErlLY=', 'Content-Length': '0'} 2022-09-12T19:32:15.011 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001e3a313a84038b10-00631f893d-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:15.012 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:32:15.013 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:32:15.014 DEBUG:boto:Method: HEAD 2022-09-12T19:32:15.015 DEBUG:boto:Path: /jdpiak-62/ 2022-09-12T19:32:15.016 DEBUG:boto:Data: 2022-09-12T19:32:15.017 DEBUG:boto:Headers: {} 2022-09-12T19:32:15.018 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:32:15.019 DEBUG:boto:Port: 8000 2022-09-12T19:32:15.020 DEBUG:boto:Params: {} 2022-09-12T19:32:15.021 DEBUG:boto:Token: None 2022-09-12T19:32:15.021 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:32:15 GMT /jdpiak-62/ 2022-09-12T19:32:15.022 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KBBx7KXyPEJQ1qABuJudpRGBkD4= 2022-09-12T19:32:15.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:32:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KBBx7KXyPEJQ1qABuJudpRGBkD4=', 'Content-Length': '0'} 2022-09-12T19:32:15.027 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000323a487557edc948-00631f893f-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:15.028 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:32:15.029 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:32:15.030 DEBUG:boto:Method: PUT 2022-09-12T19:32:15.031 DEBUG:boto:Path: /jdpiak-62/obj1 2022-09-12T19:32:15.032 DEBUG:boto:Data: 2022-09-12T19:32:15.033 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T19:32:15.033 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:32:15.034 DEBUG:boto:Port: 8000 2022-09-12T19:32:15.034 DEBUG:boto:Params: {} 2022-09-12T19:32:15.035 DEBUG:boto:Token: None 2022-09-12T19:32:15.036 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 19:32:15 GMT /jdpiak-62/obj1 2022-09-12T19:32:15.037 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sLZyzT3CblKtg3e9muTEUPGoSOw= 2022-09-12T19:32:15.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:32:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sLZyzT3CblKtg3e9muTEUPGoSOw='} 2022-09-12T19:32:15.047 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000016f2aa034f2548b3-00631f893f-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:15.049 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:32:15.049 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:32:15.050 DEBUG:boto:Method: HEAD 2022-09-12T19:32:15.050 DEBUG:boto:Path: /jdpiak-62/ 2022-09-12T19:32:15.051 DEBUG:boto:Data: 2022-09-12T19:32:15.051 DEBUG:boto:Headers: {} 2022-09-12T19:32:15.052 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:32:15.052 DEBUG:boto:Port: 8000 2022-09-12T19:32:15.053 DEBUG:boto:Params: {} 2022-09-12T19:32:15.053 DEBUG:boto:Token: None 2022-09-12T19:32:15.054 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:32:15 GMT /jdpiak-62/ 2022-09-12T19:32:15.054 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KBBx7KXyPEJQ1qABuJudpRGBkD4= 2022-09-12T19:32:15.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:32:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KBBx7KXyPEJQ1qABuJudpRGBkD4=', 'Content-Length': '0'} 2022-09-12T19:32:15.061 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051f98296dff45f54-00631f893f-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:15.062 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:32:15.063 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:32:15.063 DEBUG:boto:Method: PUT 2022-09-12T19:32:15.064 DEBUG:boto:Path: /jdpiak-62/obj2 2022-09-12T19:32:15.064 DEBUG:boto:Data: 2022-09-12T19:32:15.065 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T19:32:15.065 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:32:15.066 DEBUG:boto:Port: 8000 2022-09-12T19:32:15.066 DEBUG:boto:Params: {} 2022-09-12T19:32:15.067 DEBUG:boto:Token: None 2022-09-12T19:32:15.068 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 19:32:15 GMT /jdpiak-62/obj2 2022-09-12T19:32:15.068 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CcoHV72Zj7aVaD4m9Jhd1uoqTPM= 2022-09-12T19:32:15.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:32:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CcoHV72Zj7aVaD4m9Jhd1uoqTPM='} 2022-09-12T19:32:15.079 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075ece5a44fff5041-00631f893f-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:15.080 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:32:15.081 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:32:15.082 DEBUG:boto:Method: HEAD 2022-09-12T19:32:15.083 DEBUG:boto:Path: /jdpiak-62/ 2022-09-12T19:32:15.084 DEBUG:boto:Data: 2022-09-12T19:32:15.085 DEBUG:boto:Headers: {} 2022-09-12T19:32:15.086 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:32:15.087 DEBUG:boto:Port: 8000 2022-09-12T19:32:15.087 DEBUG:boto:Params: {} 2022-09-12T19:32:15.089 DEBUG:boto:Token: None 2022-09-12T19:32:15.090 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:32:15 GMT /jdpiak-62/ 2022-09-12T19:32:15.091 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KBBx7KXyPEJQ1qABuJudpRGBkD4= 2022-09-12T19:32:15.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:32:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KBBx7KXyPEJQ1qABuJudpRGBkD4=', 'Content-Length': '0'} 2022-09-12T19:32:15.122 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fa3d137b96ae6c3-00631f893f-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:15.124 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:32:15.124 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:32:15.125 DEBUG:boto:Method: PUT 2022-09-12T19:32:15.126 DEBUG:boto:Path: /jdpiak-62/obj3 2022-09-12T19:32:15.127 DEBUG:boto:Data: 2022-09-12T19:32:15.128 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T19:32:15.129 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:32:15.130 DEBUG:boto:Port: 8000 2022-09-12T19:32:15.131 DEBUG:boto:Params: {} 2022-09-12T19:32:15.132 DEBUG:boto:Token: None 2022-09-12T19:32:15.133 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 19:32:15 GMT /jdpiak-62/obj3 2022-09-12T19:32:15.134 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zOnY5Wm+xrUImtfDeVGmMWYoQuI= 2022-09-12T19:32:15.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:32:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zOnY5Wm+xrUImtfDeVGmMWYoQuI='} 2022-09-12T19:32:15.183 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000ee7240916415689-00631f893f-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:15.184 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:32:15.185 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:32:15.186 DEBUG:boto:Method: HEAD 2022-09-12T19:32:15.187 DEBUG:boto:Path: /jdpiak-62/ 2022-09-12T19:32:15.188 DEBUG:boto:Data: 2022-09-12T19:32:15.189 DEBUG:boto:Headers: {} 2022-09-12T19:32:15.190 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:32:15.191 DEBUG:boto:Port: 8000 2022-09-12T19:32:15.193 DEBUG:boto:Params: {} 2022-09-12T19:32:15.195 DEBUG:boto:Token: None 2022-09-12T19:32:15.196 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:32:15 GMT /jdpiak-62/ 2022-09-12T19:32:15.197 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KBBx7KXyPEJQ1qABuJudpRGBkD4= 2022-09-12T19:32:15.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:32:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KBBx7KXyPEJQ1qABuJudpRGBkD4=', 'Content-Length': '0'} 2022-09-12T19:32:15.201 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2c1d75d9d408232-00631f893f-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:15.203 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:32:15.204 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:32:15.206 DEBUG:boto:Method: PUT 2022-09-12T19:32:15.207 DEBUG:boto:Path: /jdpiak-62/obj4 2022-09-12T19:32:15.207 DEBUG:boto:Data: 2022-09-12T19:32:15.208 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T19:32:15.209 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:32:15.211 DEBUG:boto:Port: 8000 2022-09-12T19:32:15.212 DEBUG:boto:Params: {} 2022-09-12T19:32:15.213 DEBUG:boto:Token: None 2022-09-12T19:32:15.214 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 19:32:15 GMT /jdpiak-62/obj4 2022-09-12T19:32:15.215 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/yRGR0yPFCGxjqt2fDboF5HDtBc= 2022-09-12T19:32:15.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:32:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/yRGR0yPFCGxjqt2fDboF5HDtBc='} 2022-09-12T19:32:15.228 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000826aa9f147cdc3fa-00631f893f-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:15.229 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:32:15.230 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:32:15.231 DEBUG:boto:Method: HEAD 2022-09-12T19:32:15.232 DEBUG:boto:Path: /jdpiak-63/ 2022-09-12T19:32:15.233 DEBUG:boto:Data: 2022-09-12T19:32:15.234 DEBUG:boto:Headers: {} 2022-09-12T19:32:15.235 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:32:15.236 DEBUG:boto:Port: 8000 2022-09-12T19:32:15.237 DEBUG:boto:Params: {} 2022-09-12T19:32:15.238 DEBUG:boto:Token: None 2022-09-12T19:32:15.239 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:32:15 GMT /jdpiak-63/ 2022-09-12T19:32:15.240 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bZxyk9UikTLm2LnkXwYCUvqVZtc= 2022-09-12T19:32:15.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:32:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bZxyk9UikTLm2LnkXwYCUvqVZtc=', 'Content-Length': '0'} 2022-09-12T19:32:15.245 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030b204ffac04e599-00631f893f-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:15.247 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:32:15.248 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:32:15.249 DEBUG:boto:Method: PUT 2022-09-12T19:32:15.250 DEBUG:boto:Path: /jdpiak-63/obj1 2022-09-12T19:32:15.251 DEBUG:boto:Data: 2022-09-12T19:32:15.252 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T19:32:15.253 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:32:15.254 DEBUG:boto:Port: 8000 2022-09-12T19:32:15.255 DEBUG:boto:Params: {} 2022-09-12T19:32:15.257 DEBUG:boto:Token: None 2022-09-12T19:32:15.258 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 19:32:15 GMT /jdpiak-63/obj1 2022-09-12T19:32:15.259 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:apYrQyglDH2zBbzomTh7cxKMaAM= 2022-09-12T19:32:15.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:32:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:apYrQyglDH2zBbzomTh7cxKMaAM='} 2022-09-12T19:32:15.272 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6522f5141fc30db-00631f893f-15e1-test-zone2'), ('Date', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:15.273 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:32:15.274 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:32:15.275 DEBUG:boto:Method: HEAD 2022-09-12T19:32:15.275 DEBUG:boto:Path: /jdpiak-63/ 2022-09-12T19:32:15.276 DEBUG:boto:Data: 2022-09-12T19:32:15.276 DEBUG:boto:Headers: {} 2022-09-12T19:32:15.277 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:32:15.277 DEBUG:boto:Port: 8000 2022-09-12T19:32:15.278 DEBUG:boto:Params: {} 2022-09-12T19:32:15.279 DEBUG:boto:Token: None 2022-09-12T19:32:15.279 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:32:15 GMT /jdpiak-63/ 2022-09-12T19:32:15.280 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bZxyk9UikTLm2LnkXwYCUvqVZtc= 2022-09-12T19:32:15.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:32:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bZxyk9UikTLm2LnkXwYCUvqVZtc=', 'Content-Length': '0'} 2022-09-12T19:32:15.284 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037bb6c8d04ba4438-00631f893f-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:15.288 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:32:15.288 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:32:15.289 DEBUG:boto:Method: PUT 2022-09-12T19:32:15.290 DEBUG:boto:Path: /jdpiak-63/obj2 2022-09-12T19:32:15.291 DEBUG:boto:Data: 2022-09-12T19:32:15.291 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T19:32:15.292 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:32:15.292 DEBUG:boto:Port: 8000 2022-09-12T19:32:15.293 DEBUG:boto:Params: {} 2022-09-12T19:32:15.293 DEBUG:boto:Token: None 2022-09-12T19:32:15.294 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 19:32:15 GMT /jdpiak-63/obj2 2022-09-12T19:32:15.295 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KT2jTqYneCQtX1JBppjphXvSRqM= 2022-09-12T19:32:15.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:32:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KT2jTqYneCQtX1JBppjphXvSRqM='} 2022-09-12T19:32:15.303 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eb68e1f237f3e98e-00631f893f-15e1-test-zone2'), ('Date', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:15.304 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:32:15.305 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:32:15.305 DEBUG:boto:Method: HEAD 2022-09-12T19:32:15.306 DEBUG:boto:Path: /jdpiak-63/ 2022-09-12T19:32:15.307 DEBUG:boto:Data: 2022-09-12T19:32:15.307 DEBUG:boto:Headers: {} 2022-09-12T19:32:15.308 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:32:15.308 DEBUG:boto:Port: 8000 2022-09-12T19:32:15.309 DEBUG:boto:Params: {} 2022-09-12T19:32:15.309 DEBUG:boto:Token: None 2022-09-12T19:32:15.310 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:32:15 GMT /jdpiak-63/ 2022-09-12T19:32:15.311 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bZxyk9UikTLm2LnkXwYCUvqVZtc= 2022-09-12T19:32:15.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:32:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bZxyk9UikTLm2LnkXwYCUvqVZtc=', 'Content-Length': '0'} 2022-09-12T19:32:15.314 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f756b204dc34cdf-00631f893f-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:15.316 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:32:15.316 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:32:15.317 DEBUG:boto:Method: PUT 2022-09-12T19:32:15.318 DEBUG:boto:Path: /jdpiak-63/obj3 2022-09-12T19:32:15.319 DEBUG:boto:Data: 2022-09-12T19:32:15.320 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T19:32:15.320 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:32:15.321 DEBUG:boto:Port: 8000 2022-09-12T19:32:15.322 DEBUG:boto:Params: {} 2022-09-12T19:32:15.323 DEBUG:boto:Token: None 2022-09-12T19:32:15.323 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 19:32:15 GMT /jdpiak-63/obj3 2022-09-12T19:32:15.324 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TZGHe8hJfpLN1w9nxRAMd2Bu97A= 2022-09-12T19:32:15.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:32:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TZGHe8hJfpLN1w9nxRAMd2Bu97A='} 2022-09-12T19:32:15.370 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000019790d6ccedd0449-00631f893f-15e1-test-zone2'), ('Date', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:15.372 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:32:15.372 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:32:15.373 DEBUG:boto:Method: HEAD 2022-09-12T19:32:15.374 DEBUG:boto:Path: /jdpiak-63/ 2022-09-12T19:32:15.375 DEBUG:boto:Data: 2022-09-12T19:32:15.376 DEBUG:boto:Headers: {} 2022-09-12T19:32:15.376 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:32:15.377 DEBUG:boto:Port: 8000 2022-09-12T19:32:15.378 DEBUG:boto:Params: {} 2022-09-12T19:32:15.379 DEBUG:boto:Token: None 2022-09-12T19:32:15.380 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:32:15 GMT /jdpiak-63/ 2022-09-12T19:32:15.381 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bZxyk9UikTLm2LnkXwYCUvqVZtc= 2022-09-12T19:32:15.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:32:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bZxyk9UikTLm2LnkXwYCUvqVZtc=', 'Content-Length': '0'} 2022-09-12T19:32:15.385 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000702f422e1c88669c-00631f893f-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:15.385 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:32:15.386 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:32:15.387 DEBUG:boto:Method: PUT 2022-09-12T19:32:15.387 DEBUG:boto:Path: /jdpiak-63/obj4 2022-09-12T19:32:15.388 DEBUG:boto:Data: 2022-09-12T19:32:15.389 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T19:32:15.390 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:32:15.391 DEBUG:boto:Port: 8000 2022-09-12T19:32:15.392 DEBUG:boto:Params: {} 2022-09-12T19:32:15.393 DEBUG:boto:Token: None 2022-09-12T19:32:15.394 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 19:32:15 GMT /jdpiak-63/obj4 2022-09-12T19:32:15.395 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SHOql0UqXL7VpNxinhShkUCJ76s= 2022-09-12T19:32:15.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:32:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SHOql0UqXL7VpNxinhShkUCJ76s='} 2022-09-12T19:32:15.405 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000043ba452ee5401701-00631f893f-15e1-test-zone2'), ('Date', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:15.407 DEBUG:boto:path=/jdpiak-64/ 2022-09-12T19:32:15.408 DEBUG:boto:auth_path=/jdpiak-64/ 2022-09-12T19:32:15.409 DEBUG:boto:Method: HEAD 2022-09-12T19:32:15.410 DEBUG:boto:Path: /jdpiak-64/ 2022-09-12T19:32:15.411 DEBUG:boto:Data: 2022-09-12T19:32:15.412 DEBUG:boto:Headers: {} 2022-09-12T19:32:15.413 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:32:15.414 DEBUG:boto:Port: 8001 2022-09-12T19:32:15.415 DEBUG:boto:Params: {} 2022-09-12T19:32:15.416 DEBUG:boto:Token: None 2022-09-12T19:32:15.418 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:32:15 GMT /jdpiak-64/ 2022-09-12T19:32:15.419 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:j9nCirPJ8tAxq4loo0pNblXfTrs= 2022-09-12T19:32:15.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:32:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:j9nCirPJ8tAxq4loo0pNblXfTrs=', 'Content-Length': '0'} 2022-09-12T19:32:15.423 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000948a4a89b4208dcc-00631f893f-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:15.425 DEBUG:boto:path=/jdpiak-64/obj1 2022-09-12T19:32:15.426 DEBUG:boto:auth_path=/jdpiak-64/obj1 2022-09-12T19:32:15.427 DEBUG:boto:Method: PUT 2022-09-12T19:32:15.428 DEBUG:boto:Path: /jdpiak-64/obj1 2022-09-12T19:32:15.429 DEBUG:boto:Data: 2022-09-12T19:32:15.430 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T19:32:15.431 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:32:15.431 DEBUG:boto:Port: 8001 2022-09-12T19:32:15.432 DEBUG:boto:Params: {} 2022-09-12T19:32:15.432 DEBUG:boto:Token: None 2022-09-12T19:32:15.433 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 19:32:15 GMT /jdpiak-64/obj1 2022-09-12T19:32:15.434 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QcliBT7QsMyeXMu1Cg+Xj52+srU= 2022-09-12T19:32:15.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:32:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QcliBT7QsMyeXMu1Cg+Xj52+srU='} 2022-09-12T19:32:15.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a84420ea808c3d12-00631f893f-15c4-test-zone3'), ('Date', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:15.443 DEBUG:boto:path=/jdpiak-64/ 2022-09-12T19:32:15.443 DEBUG:boto:auth_path=/jdpiak-64/ 2022-09-12T19:32:15.444 DEBUG:boto:Method: HEAD 2022-09-12T19:32:15.444 DEBUG:boto:Path: /jdpiak-64/ 2022-09-12T19:32:15.445 DEBUG:boto:Data: 2022-09-12T19:32:15.446 DEBUG:boto:Headers: {} 2022-09-12T19:32:15.446 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:32:15.447 DEBUG:boto:Port: 8001 2022-09-12T19:32:15.448 DEBUG:boto:Params: {} 2022-09-12T19:32:15.448 DEBUG:boto:Token: None 2022-09-12T19:32:15.449 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:32:15 GMT /jdpiak-64/ 2022-09-12T19:32:15.450 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:j9nCirPJ8tAxq4loo0pNblXfTrs= 2022-09-12T19:32:15.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:32:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:j9nCirPJ8tAxq4loo0pNblXfTrs=', 'Content-Length': '0'} 2022-09-12T19:32:15.454 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072741929fe7181c8-00631f893f-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:15.456 DEBUG:boto:path=/jdpiak-64/obj2 2022-09-12T19:32:15.457 DEBUG:boto:auth_path=/jdpiak-64/obj2 2022-09-12T19:32:15.458 DEBUG:boto:Method: PUT 2022-09-12T19:32:15.458 DEBUG:boto:Path: /jdpiak-64/obj2 2022-09-12T19:32:15.459 DEBUG:boto:Data: 2022-09-12T19:32:15.460 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T19:32:15.461 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:32:15.461 DEBUG:boto:Port: 8001 2022-09-12T19:32:15.462 DEBUG:boto:Params: {} 2022-09-12T19:32:15.463 DEBUG:boto:Token: None 2022-09-12T19:32:15.464 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 19:32:15 GMT /jdpiak-64/obj2 2022-09-12T19:32:15.464 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:axXXbFFaPbo1jZpdPovrborPDTM= 2022-09-12T19:32:15.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:32:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:axXXbFFaPbo1jZpdPovrborPDTM='} 2022-09-12T19:32:15.510 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000009b3bd9fc7fdba34-00631f893f-15c4-test-zone3'), ('Date', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:15.512 DEBUG:boto:path=/jdpiak-64/ 2022-09-12T19:32:15.513 DEBUG:boto:auth_path=/jdpiak-64/ 2022-09-12T19:32:15.513 DEBUG:boto:Method: HEAD 2022-09-12T19:32:15.514 DEBUG:boto:Path: /jdpiak-64/ 2022-09-12T19:32:15.514 DEBUG:boto:Data: 2022-09-12T19:32:15.515 DEBUG:boto:Headers: {} 2022-09-12T19:32:15.515 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:32:15.516 DEBUG:boto:Port: 8001 2022-09-12T19:32:15.516 DEBUG:boto:Params: {} 2022-09-12T19:32:15.517 DEBUG:boto:Token: None 2022-09-12T19:32:15.517 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:32:15 GMT /jdpiak-64/ 2022-09-12T19:32:15.518 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:j9nCirPJ8tAxq4loo0pNblXfTrs= 2022-09-12T19:32:15.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:32:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:j9nCirPJ8tAxq4loo0pNblXfTrs=', 'Content-Length': '0'} 2022-09-12T19:32:15.521 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003dd19a162070d0b6-00631f893f-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:15.522 DEBUG:boto:path=/jdpiak-64/obj3 2022-09-12T19:32:15.523 DEBUG:boto:auth_path=/jdpiak-64/obj3 2022-09-12T19:32:15.523 DEBUG:boto:Method: PUT 2022-09-12T19:32:15.524 DEBUG:boto:Path: /jdpiak-64/obj3 2022-09-12T19:32:15.525 DEBUG:boto:Data: 2022-09-12T19:32:15.525 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T19:32:15.526 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:32:15.526 DEBUG:boto:Port: 8001 2022-09-12T19:32:15.527 DEBUG:boto:Params: {} 2022-09-12T19:32:15.528 DEBUG:boto:Token: None 2022-09-12T19:32:15.528 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 19:32:15 GMT /jdpiak-64/obj3 2022-09-12T19:32:15.529 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tGz1c4xRAwAtL4CBbGWlxScjkuU= 2022-09-12T19:32:15.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:32:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tGz1c4xRAwAtL4CBbGWlxScjkuU='} 2022-09-12T19:32:15.536 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c6282be65616be3a-00631f893f-15c4-test-zone3'), ('Date', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:15.537 DEBUG:boto:path=/jdpiak-64/ 2022-09-12T19:32:15.537 DEBUG:boto:auth_path=/jdpiak-64/ 2022-09-12T19:32:15.538 DEBUG:boto:Method: HEAD 2022-09-12T19:32:15.538 DEBUG:boto:Path: /jdpiak-64/ 2022-09-12T19:32:15.539 DEBUG:boto:Data: 2022-09-12T19:32:15.539 DEBUG:boto:Headers: {} 2022-09-12T19:32:15.540 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:32:15.540 DEBUG:boto:Port: 8001 2022-09-12T19:32:15.541 DEBUG:boto:Params: {} 2022-09-12T19:32:15.541 DEBUG:boto:Token: None 2022-09-12T19:32:15.542 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:32:15 GMT /jdpiak-64/ 2022-09-12T19:32:15.542 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:j9nCirPJ8tAxq4loo0pNblXfTrs= 2022-09-12T19:32:15.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:32:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:j9nCirPJ8tAxq4loo0pNblXfTrs=', 'Content-Length': '0'} 2022-09-12T19:32:15.546 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7bb0337eeb7f38c-00631f893f-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:15.547 DEBUG:boto:path=/jdpiak-64/obj4 2022-09-12T19:32:15.548 DEBUG:boto:auth_path=/jdpiak-64/obj4 2022-09-12T19:32:15.548 DEBUG:boto:Method: PUT 2022-09-12T19:32:15.549 DEBUG:boto:Path: /jdpiak-64/obj4 2022-09-12T19:32:15.549 DEBUG:boto:Data: 2022-09-12T19:32:15.550 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-12T19:32:15.550 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:32:15.551 DEBUG:boto:Port: 8001 2022-09-12T19:32:15.551 DEBUG:boto:Params: {} 2022-09-12T19:32:15.552 DEBUG:boto:Token: None 2022-09-12T19:32:15.553 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Mon, 12 Sep 2022 19:32:15 GMT /jdpiak-64/obj4 2022-09-12T19:32:15.553 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F67P45k7o5dlGmgq1voeH62NxHY= 2022-09-12T19:32:15.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:32:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F67P45k7o5dlGmgq1voeH62NxHY='} 2022-09-12T19:32:15.598 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f1fbdaa053b11e3e-00631f893f-15c4-test-zone3'), ('Date', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:32:15.599 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:32:15.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:32:15.601 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:32:15.657 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:32:15.657 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:32:15.690 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=38ca1f57-1358-496c-a99b-df49ee365a4f 2022-09-12T19:32:15.706 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:32:15.707 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:32:15.708 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:32:15.708 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:32:15.708 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:32:15.709 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:32:15.709 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:32:15.709 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:32:15.709 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:32:15.710 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:32:15.710 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663011134.995774_533.1", 2022-09-12T19:32:15.710 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:32:14.995774Z" 2022-09-12T19:32:15.710 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:32:15.711 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:32:15.711 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:32:15.711 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:32:15.712 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:32:15.712 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:32:15.713 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1663011134.995774_533.1', 3: ''} 2022-09-12T19:32:15.714 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T19:32:15.715 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:32:15.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:32:15.718 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:32:15.776 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:32:15.777 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:32:15.819 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:32:15.817+0000 7f15c4180600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:32:15.821 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:32:15.818+0000 7f15c4180600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:32:15.837 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:32:15.837 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:32:15.838 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:32:15.838 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:32:15.839 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:32:15.839 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:32:15.839 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:32:15.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:32:15.840 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:32:15.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:32:15.840 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:32:15.841 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:32:15.841 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:32:15.841 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:32:15.842 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:32:15.842 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:32:15.842 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:32:15.843 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:32:15.843 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:32:15.843 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:32:15.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:32:15.844 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:32:15.844 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:32:15.844 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:32:15.844 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:32:15.845 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:32:15.845 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:32:15.845 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:32:15.845 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:32:15.846 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:32:15.846 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:32:15.846 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:32:15.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:32:15.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:32:15.848 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:32:15.848 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:32:15.848 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:32:15.849 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663011134.987281_530.1", 2022-09-12T19:32:15.849 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:32:15.849 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:32:15.849 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:32:15.850 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:32:14.987281Z", 2022-09-12T19:32:15.850 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:32:15.850 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:32:15.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:32:15.851 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:32:15.851 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:32:15.851 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:32:15.852 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:32:15.852 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:32:15.852 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:32:15.852 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:32:15.853 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:32:15.853 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:32:15.853 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:32:15.853 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:32:15.854 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:32:15.854 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:32:15.854 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:32:15.855 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:32:15.855 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:32:15.855 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:32:15.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:32:15.856 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:32:15Z" 2022-09-12T19:32:15.856 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:32:15.858 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011134.987281_530.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:32:14.987281Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:32:15Z" } 2022-09-12T19:32:15.858 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011134.987281_530.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:32:14.987281Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:32:15.859 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011134.995774_533.1', 3: ''} 2022-09-12T19:32:15.860 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011134.987281_530.1', 3: ''} 2022-09-12T19:32:15.860 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663011134.995774_533.1 target=1_1663011134.987281_530.1 2022-09-12T19:32:20.862 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:32:20.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:32:20.864 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:32:20.921 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:32:20.922 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:32:20.956 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:32:20.954+0000 7fe80b133600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:32:20.957 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:32:20.955+0000 7fe80b133600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:32:20.977 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:32:20.977 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:32:20.977 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:32:20.978 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:32:20.978 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:32:20.978 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:32:20.978 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:32:20.979 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:32:20.979 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:32:20.979 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:32:20.980 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:32:20.980 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:32:20.980 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:32:20.981 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:32:20.981 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:32:20.981 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:32:20.981 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:32:20.982 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:32:20.982 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:32:20.982 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:32:20.983 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:32:20.983 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:32:20.983 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:32:20.983 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:32:20.984 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:32:20.984 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:32:20.984 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:32:20.984 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:32:20.985 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:32:20.985 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:32:20.985 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:32:20.985 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:32:20.986 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:32:20.986 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:32:20.986 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:32:20.987 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:32:20.987 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:32:20.987 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663011134.995774_533.1", 2022-09-12T19:32:20.987 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:32:20.988 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:32:20.988 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:32:20.988 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:32:14.995774Z", 2022-09-12T19:32:20.988 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:32:20.989 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:32:20.989 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:32:20.989 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:32:20.989 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:32:20.990 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:32:20.990 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:32:20.990 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:32:20.990 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:32:20.991 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:32:20.991 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:32:20.991 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:32:20.991 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:32:20.992 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:32:20.992 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:32:20.992 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:32:20.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:32:20.993 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:32:20.993 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:32:20.993 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:32:20.994 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:32:20.994 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:32:20Z" 2022-09-12T19:32:20.994 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:32:20.995 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011134.995774_533.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:32:14.995774Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:32:20Z" } 2022-09-12T19:32:20.996 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011134.995774_533.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:32:14.995774Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:32:20.997 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011134.995774_533.1', 3: ''} 2022-09-12T19:32:20.997 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011134.995774_533.1', 3: ''} 2022-09-12T19:32:20.998 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T19:32:20.998 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T19:32:20.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:32:20.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:32:21.000 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:32:21.059 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:32:21.059 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:32:21.089 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:32:21.087+0000 7f9bb7be8600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:32:21.090 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:32:21.088+0000 7f9bb7be8600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:32:21.107 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:32:21.107 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:32:21.107 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:32:21.108 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:32:21.108 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:32:21.108 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:32:21.108 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:32:21.109 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:32:21.109 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:32:21.109 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:32:21.109 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:32:21.110 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:32:21.110 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:32:21.110 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:32:21.110 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:32:21.111 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:32:21.111 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:32:21.111 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:32:21.111 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:32:21.112 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:32:21.112 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:32:21.112 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:32:21.112 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:32:21.113 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:32:21.113 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:32:21.113 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:32:21.113 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:32:21.114 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:32:21.114 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:32:21.115 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:32:21.115 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:32:21.116 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:32:21.116 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:32:21.116 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:32:21.117 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:32:21.117 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:32:21.117 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:32:21.117 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663011134.995774_533.1", 2022-09-12T19:32:21.118 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:32:21.118 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:32:21.118 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:32:21.118 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:32:14.995774Z", 2022-09-12T19:32:21.119 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:32:21.119 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:32:21.119 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:32:21.119 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:32:21.120 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:32:21.120 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:32:21.120 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:32:21.121 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:32:21.121 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:32:21.121 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:32:21.121 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:32:21.122 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:32:21.122 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:32:21.122 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:32:21.122 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:32:21.123 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:32:21.123 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:32:21.123 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:32:21.123 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:32:21.124 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:32:21.124 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:32:21.124 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:32:21Z" 2022-09-12T19:32:21.124 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:32:21.126 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011134.995774_533.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:32:14.995774Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:32:21Z" } 2022-09-12T19:32:21.126 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011134.995774_533.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:32:14.995774Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:32:21.127 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011134.995774_533.1', 3: ''} 2022-09-12T19:32:21.127 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011134.995774_533.1', 3: ''} 2022-09-12T19:32:21.128 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T19:32:21.128 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T19:32:21.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:32:21.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:32:21.130 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:32:21.182 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:32:21.183 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:32:21.226 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:32:21.224+0000 7f2f7e47b600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:32:21.228 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:32:21.225+0000 7f2f7e47b600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:32:21.244 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:32:21.244 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:32:21.245 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:32:21.245 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:32:21.245 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:32:21.245 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:32:21.246 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:32:21.246 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:32:21.246 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:32:21.247 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:32:21.247 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:32:21.247 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:32:21.248 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:32:21.248 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:32:21.248 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:32:21.249 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:32:21.249 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:32:21.249 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:32:21.249 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:32:21.250 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:32:21.250 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:32:21.250 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:32:21.251 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:32:21.251 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:32:21.251 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:32:21.251 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:32:21.252 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:32:21.252 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:32:21.252 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:32:21.252 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:32:21.253 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:32:21.253 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:32:21.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:32:21.254 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:32:21.254 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:32:21.254 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:32:21.254 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:32:21.255 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663011134.995774_533.1", 2022-09-12T19:32:21.255 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:32:21.255 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:32:21.255 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:32:21.256 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:32:14.995774Z", 2022-09-12T19:32:21.256 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:32:21.256 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:32:21.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:32:21.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:32:21.257 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:32:21.257 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:32:21.257 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:32:21.257 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:32:21.258 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:32:21.258 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:32:21.258 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:32:21.259 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:32:21.259 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:32:21.259 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:32:21.259 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:32:21.259 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:32:21.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:32:21.260 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:32:21.260 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:32:21.260 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:32:21.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:32:21.261 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:32:21Z" 2022-09-12T19:32:21.261 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:32:21.263 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011134.995774_533.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:32:14.995774Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:32:21Z" } 2022-09-12T19:32:21.263 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011134.995774_533.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:32:14.995774Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:32:21.264 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011134.995774_533.1', 3: ''} 2022-09-12T19:32:21.264 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011134.995774_533.1', 3: ''} 2022-09-12T19:32:21.265 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T19:32:21.265 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-62 2022-09-12T19:32:21.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:32:21.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:32:21.267 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:32:21.323 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:32:21.324 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:32:21.362 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:32:21.359+0000 7f66c9dfa600 1 waiting to reach incremental sync.. 2022-09-12T19:32:26.362 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:32:26.360+0000 7f66c9dfa600 1 waiting to reach incremental sync.. 2022-09-12T19:32:31.363 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:32:31.361+0000 7f66c9dfa600 1 waiting to reach incremental sync.. 2022-09-12T19:32:36.365 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:32:36.362+0000 7f66c9dfa600 1 bucket sync caught up with source: 2022-09-12T19:32:36.365 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.184.5, , 00000000001.49.5, , 00000000001.26.5, , , 00000000001.56.5, , , ] 2022-09-12T19:32:36.366 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.184.5, , 00000000001.49.5, , 00000000001.26.5, , , 00000000001.56.5, , , ] 2022-09-12T19:32:36.366 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:32:36.362+0000 7f66c9dfa600 0 bucket checkpoint complete 2022-09-12T19:32:36.384 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-62 2022-09-12T19:32:36.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:32:36.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:32:36.387 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:32:36.451 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:32:36.452 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:32:36.502 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:32:36.498+0000 7fa214785600 1 bucket sync caught up with source: 2022-09-12T19:32:36.502 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [00000000001.184.5, , 00000000001.49.5, , 00000000001.26.5, , , 00000000001.56.5, , , ] 2022-09-12T19:32:36.503 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [00000000001.184.5, , 00000000001.49.5, , 00000000001.26.5, , , 00000000001.56.5, , , ] 2022-09-12T19:32:36.503 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:32:36.498+0000 7fa214785600 0 bucket checkpoint complete 2022-09-12T19:32:36.519 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-62 2022-09-12T19:32:36.520 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:32:36.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:32:36.521 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:32:36.587 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:32:36.587 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:32:36.642 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:32:36.639+0000 7fa7fc4d1600 1 waiting for incremental sync to catch up: 2022-09-12T19:32:36.642 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , 00000000001.26.5, , , 00000000001.56.5, , , ] 2022-09-12T19:32:36.643 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.184.5, , 00000000001.49.5, , 00000000001.26.5, , , 00000000001.56.5, , , ] 2022-09-12T19:32:41.643 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:32:41.640+0000 7fa7fc4d1600 1 waiting for incremental sync to catch up: 2022-09-12T19:32:41.643 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , 00000000001.26.5, , , 00000000001.56.5, , , ] 2022-09-12T19:32:41.644 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.184.5, , 00000000001.49.5, , 00000000001.26.5, , , 00000000001.56.5, , , ] 2022-09-12T19:32:46.644 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:32:46.642+0000 7fa7fc4d1600 1 waiting for incremental sync to catch up: 2022-09-12T19:32:46.645 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , 00000000001.26.5, , , 00000000001.56.5, , , ] 2022-09-12T19:32:46.645 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.184.5, , 00000000001.49.5, , 00000000001.26.5, , , 00000000001.56.5, , , ] 2022-09-12T19:32:51.646 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:32:51.643+0000 7fa7fc4d1600 1 waiting for incremental sync to catch up: 2022-09-12T19:32:51.646 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , 00000000001.26.5, , , 00000000001.56.5, , , ] 2022-09-12T19:32:51.647 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.184.5, , 00000000001.49.5, , 00000000001.26.5, , , 00000000001.56.5, , , ] 2022-09-12T19:32:56.647 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:32:56.645+0000 7fa7fc4d1600 1 waiting for incremental sync to catch up: 2022-09-12T19:32:56.648 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , 00000000001.26.5, , , 00000000001.56.5, , , ] 2022-09-12T19:32:56.649 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.184.5, , 00000000001.49.5, , 00000000001.26.5, , , 00000000001.56.5, , , ] 2022-09-12T19:33:01.649 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:33:01.646+0000 7fa7fc4d1600 1 waiting for incremental sync to catch up: 2022-09-12T19:33:01.649 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , 00000000001.26.5, , , 00000000001.56.5, , , ] 2022-09-12T19:33:01.650 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.184.5, , 00000000001.49.5, , 00000000001.26.5, , , 00000000001.56.5, , , ] 2022-09-12T19:33:06.651 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:33:06.648+0000 7fa7fc4d1600 1 waiting for incremental sync to catch up: 2022-09-12T19:33:06.651 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , 00000000001.26.5, , , 00000000001.56.5, , , ] 2022-09-12T19:33:06.652 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.184.5, , 00000000001.49.5, , 00000000001.26.5, , , 00000000001.56.5, , , ] 2022-09-12T19:33:11.652 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:33:11.650+0000 7fa7fc4d1600 1 waiting for incremental sync to catch up: 2022-09-12T19:33:11.652 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , 00000000001.26.5, , , 00000000001.56.5, , , ] 2022-09-12T19:33:11.653 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.184.5, , 00000000001.49.5, , 00000000001.26.5, , , 00000000001.56.5, , , ] 2022-09-12T19:33:16.653 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:33:16.651+0000 7fa7fc4d1600 1 bucket sync caught up with source: 2022-09-12T19:33:16.654 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.184.5, , 00000000001.49.5, , 00000000001.26.5, , , 00000000001.56.5, , , ] 2022-09-12T19:33:16.654 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.184.5, , 00000000001.49.5, , 00000000001.26.5, , , 00000000001.56.5, , , ] 2022-09-12T19:33:16.655 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:33:16.651+0000 7fa7fc4d1600 0 bucket checkpoint complete 2022-09-12T19:33:16.672 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-62 2022-09-12T19:33:16.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:33:16.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:33:16.674 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:33:16.740 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:33:16.740 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:33:16.782 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:33:16.779+0000 7f5d9cc36600 1 waiting to reach incremental sync.. 2022-09-12T19:33:21.783 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:33:21.780+0000 7f5d9cc36600 1 waiting to reach incremental sync.. 2022-09-12T19:33:26.783 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:33:26.781+0000 7f5d9cc36600 1 waiting to reach incremental sync.. 2022-09-12T19:33:31.784 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:33:31.782+0000 7f5d9cc36600 1 waiting to reach incremental sync.. 2022-09-12T19:33:36.785 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:33:36.783+0000 7f5d9cc36600 1 waiting to reach incremental sync.. 2022-09-12T19:33:41.786 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:33:41.783+0000 7f5d9cc36600 1 waiting to reach incremental sync.. 2022-09-12T19:33:46.786 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:33:46.783+0000 7f5d9cc36600 1 waiting to reach incremental sync.. 2022-09-12T19:33:51.786 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:33:51.784+0000 7f5d9cc36600 1 waiting to reach incremental sync.. 2022-09-12T19:33:56.788 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:33:56.786+0000 7f5d9cc36600 1 bucket sync caught up with source: 2022-09-12T19:33:56.789 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [00000000001.972.5, , 00000000001.923.5, , 00000000001.608.5, , , 00000000001.965.5, , , ] 2022-09-12T19:33:56.789 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [00000000001.972.5, , 00000000001.923.5, , 00000000001.608.5, , , 00000000001.965.5, , , ] 2022-09-12T19:33:56.790 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:33:56.786+0000 7f5d9cc36600 0 bucket checkpoint complete 2022-09-12T19:33:56.807 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-62 2022-09-12T19:33:56.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:33:56.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:33:56.809 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:33:56.873 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:33:56.874 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:33:56.916 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:33:56.913+0000 7f3e6dc03600 1 bucket sync caught up with source: 2022-09-12T19:33:56.917 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [00000000001.972.5, , 00000000001.923.5, , 00000000001.608.5, , , 00000000001.965.5, , , ] 2022-09-12T19:33:56.917 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [00000000001.972.5, , 00000000001.923.5, , 00000000001.608.5, , , 00000000001.965.5, , , ] 2022-09-12T19:33:56.917 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:33:56.913+0000 7f3e6dc03600 0 bucket checkpoint complete 2022-09-12T19:33:56.930 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-62 2022-09-12T19:33:56.930 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-62 2022-09-12T19:33:56.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:33:56.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:33:56.932 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:33:56.992 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:33:56.992 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:33:57.027 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:33:57.025+0000 7fbd18f91600 1 bucket sync caught up with source: 2022-09-12T19:33:57.028 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [00000000001.526.5, , 00000000001.389.5, , 00000000001.228.5, , , 00000000001.460.5, , , ] 2022-09-12T19:33:57.028 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [00000000001.526.5, , 00000000001.389.5, , 00000000001.228.5, , , 00000000001.460.5, , , ] 2022-09-12T19:33:57.028 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:33:57.025+0000 7fbd18f91600 0 bucket checkpoint complete 2022-09-12T19:33:57.041 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-62 2022-09-12T19:33:57.042 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:33:57.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:33:57.043 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:33:57.106 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:33:57.107 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:33:57.154 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:33:57.153+0000 7f4c05db4600 1 bucket sync caught up with source: 2022-09-12T19:33:57.155 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.526.5, , 00000000001.389.5, , 00000000001.228.5, , , 00000000001.460.5, , , ] 2022-09-12T19:33:57.155 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.526.5, , 00000000001.389.5, , 00000000001.228.5, , , 00000000001.460.5, , , ] 2022-09-12T19:33:57.155 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:33:57.153+0000 7f4c05db4600 0 bucket checkpoint complete 2022-09-12T19:33:57.169 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-62 2022-09-12T19:33:57.170 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-62 zones={test-zone2, test-zone1} 2022-09-12T19:33:57.171 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:57.172 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:57.173 DEBUG:boto:Method: HEAD 2022-09-12T19:33:57.173 DEBUG:boto:Path: /jdpiak-62/ 2022-09-12T19:33:57.174 DEBUG:boto:Data: 2022-09-12T19:33:57.174 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.175 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.175 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.176 DEBUG:boto:Params: {} 2022-09-12T19:33:57.177 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:33:57.177 DEBUG:boto:Token: None 2022-09-12T19:33:57.178 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/ 2022-09-12T19:33:57.179 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Z32xjIDd266o+o1YZYXJWhhxr5E= 2022-09-12T19:33:57.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Z32xjIDd266o+o1YZYXJWhhxr5E=', 'Content-Length': '0'} 2022-09-12T19:33:57.184 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c74d49a0290a15a-00631f89a5-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.185 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:57.185 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:57.186 DEBUG:boto:Method: HEAD 2022-09-12T19:33:57.186 DEBUG:boto:Path: /jdpiak-62/ 2022-09-12T19:33:57.187 DEBUG:boto:Data: 2022-09-12T19:33:57.187 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.188 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.189 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.189 DEBUG:boto:Params: {} 2022-09-12T19:33:57.190 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:33:57.190 DEBUG:boto:Token: None 2022-09-12T19:33:57.191 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/ 2022-09-12T19:33:57.192 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Z32xjIDd266o+o1YZYXJWhhxr5E= 2022-09-12T19:33:57.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Z32xjIDd266o+o1YZYXJWhhxr5E=', 'Content-Length': '0'} 2022-09-12T19:33:57.196 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000238428ea6524395d-00631f89a5-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.196 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:33:57.197 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:57.198 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:57.198 DEBUG:boto:path=/jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:57.199 DEBUG:boto:auth_path=/jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:57.199 DEBUG:boto:Method: GET 2022-09-12T19:33:57.200 DEBUG:boto:Path: /jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:57.201 DEBUG:boto:Data: 2022-09-12T19:33:57.201 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.202 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.203 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.203 DEBUG:boto:Params: {} 2022-09-12T19:33:57.204 DEBUG:boto:Token: None 2022-09-12T19:33:57.204 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/?versions 2022-09-12T19:33:57.205 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N7Nl0snh6VRogT01XVCrpATPub4= 2022-09-12T19:33:57.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N7Nl0snh6VRogT01XVCrpATPub4=', 'Content-Length': '0'} 2022-09-12T19:33:57.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e990b6049105e7d-00631f89a5-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.209 DEBUG:boto:b'jdpiak-62999falseobj1nulltrue2022-09-12T19:32:15.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-12T19:32:15.070Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-12T19:32:15.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-12T19:32:15.218Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:57.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-12T19:33:57.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-12T19:33:57.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-12T19:33:57.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-12T19:33:57.214 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:33:57.214 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:57.215 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:57.215 DEBUG:boto:path=/jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:57.216 DEBUG:boto:auth_path=/jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:57.216 DEBUG:boto:Method: GET 2022-09-12T19:33:57.217 DEBUG:boto:Path: /jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:57.217 DEBUG:boto:Data: 2022-09-12T19:33:57.218 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.218 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.219 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.219 DEBUG:boto:Params: {} 2022-09-12T19:33:57.220 DEBUG:boto:Token: None 2022-09-12T19:33:57.221 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/?versions 2022-09-12T19:33:57.221 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N7Nl0snh6VRogT01XVCrpATPub4= 2022-09-12T19:33:57.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N7Nl0snh6VRogT01XVCrpATPub4=', 'Content-Length': '0'} 2022-09-12T19:33:57.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6444024fc913806-00631f89a5-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.226 DEBUG:boto:b'jdpiak-62999falseobj1nulltrue2022-09-12T19:32:15.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-12T19:32:15.070Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-12T19:32:15.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-12T19:32:15.218Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:57.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-12T19:33:57.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-12T19:33:57.228 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-12T19:33:57.228 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-12T19:33:57.229 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:57.229 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:57.230 DEBUG:boto:path=/jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:57.230 DEBUG:boto:auth_path=/jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:57.231 DEBUG:boto:Method: GET 2022-09-12T19:33:57.231 DEBUG:boto:Path: /jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:57.232 DEBUG:boto:Data: 2022-09-12T19:33:57.232 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.233 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.233 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.234 DEBUG:boto:Params: {} 2022-09-12T19:33:57.235 DEBUG:boto:Token: None 2022-09-12T19:33:57.235 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/?versions 2022-09-12T19:33:57.236 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N7Nl0snh6VRogT01XVCrpATPub4= 2022-09-12T19:33:57.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N7Nl0snh6VRogT01XVCrpATPub4=', 'Content-Length': '0'} 2022-09-12T19:33:57.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ce5b273a4b7b34b-00631f89a5-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.240 DEBUG:boto:b'jdpiak-62999falseobj1nulltrue2022-09-12T19:32:15.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-12T19:32:15.070Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-12T19:32:15.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-12T19:32:15.218Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:57.241 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:57.241 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:57.242 DEBUG:boto:path=/jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:57.242 DEBUG:boto:auth_path=/jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:57.243 DEBUG:boto:Method: GET 2022-09-12T19:33:57.243 DEBUG:boto:Path: /jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:57.244 DEBUG:boto:Data: 2022-09-12T19:33:57.245 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.245 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.246 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.246 DEBUG:boto:Params: {} 2022-09-12T19:33:57.247 DEBUG:boto:Token: None 2022-09-12T19:33:57.247 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/?versions 2022-09-12T19:33:57.248 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N7Nl0snh6VRogT01XVCrpATPub4= 2022-09-12T19:33:57.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N7Nl0snh6VRogT01XVCrpATPub4=', 'Content-Length': '0'} 2022-09-12T19:33:57.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7374cc622483bf1-00631f89a5-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.252 DEBUG:boto:b'jdpiak-62999falseobj1nulltrue2022-09-12T19:32:15.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-12T19:32:15.070Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-12T19:32:15.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-12T19:32:15.218Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:57.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-12T19:33:57.253 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:57.254 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:57.255 DEBUG:boto:path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:57.255 DEBUG:boto:auth_path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:57.256 DEBUG:boto:Method: GET 2022-09-12T19:33:57.256 DEBUG:boto:Path: /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:57.257 DEBUG:boto:Data: 2022-09-12T19:33:57.257 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.258 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.259 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.259 DEBUG:boto:Params: {} 2022-09-12T19:33:57.260 DEBUG:boto:Token: None 2022-09-12T19:33:57.261 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:57.261 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XyVZx30721j4xV8pF4nWZrGf22Y= 2022-09-12T19:33:57.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XyVZx30721j4xV8pF4nWZrGf22Y=', 'Content-Length': '0'} 2022-09-12T19:33:57.264 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000125bb5d3bc8e5fc3-00631f89a5-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.265 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:57.265 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:57.266 DEBUG:boto:path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:57.267 DEBUG:boto:auth_path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:57.267 DEBUG:boto:Method: GET 2022-09-12T19:33:57.268 DEBUG:boto:Path: /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:57.268 DEBUG:boto:Data: 2022-09-12T19:33:57.269 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.269 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.270 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.270 DEBUG:boto:Params: {} 2022-09-12T19:33:57.271 DEBUG:boto:Token: None 2022-09-12T19:33:57.272 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:57.272 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XyVZx30721j4xV8pF4nWZrGf22Y= 2022-09-12T19:33:57.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XyVZx30721j4xV8pF4nWZrGf22Y=', 'Content-Length': '0'} 2022-09-12T19:33:57.275 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6399e3408a01551-00631f89a5-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.276 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:57.276 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:57.277 DEBUG:boto:path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:57.277 DEBUG:boto:auth_path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:57.278 DEBUG:boto:Method: HEAD 2022-09-12T19:33:57.278 DEBUG:boto:Path: /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:57.279 DEBUG:boto:Data: 2022-09-12T19:33:57.279 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.281 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.281 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.282 DEBUG:boto:Params: {} 2022-09-12T19:33:57.282 DEBUG:boto:Token: None 2022-09-12T19:33:57.283 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:57.284 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FILgvQL0So95OsflleI7MYirUZQ= 2022-09-12T19:33:57.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FILgvQL0So95OsflleI7MYirUZQ=', 'Content-Length': '0'} 2022-09-12T19:33:57.286 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059d32c7486cbd9e8-00631f89a5-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.287 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:57.287 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:57.288 DEBUG:boto:path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:57.288 DEBUG:boto:auth_path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:57.289 DEBUG:boto:Method: HEAD 2022-09-12T19:33:57.289 DEBUG:boto:Path: /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:57.290 DEBUG:boto:Data: 2022-09-12T19:33:57.290 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.291 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.291 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.292 DEBUG:boto:Params: {} 2022-09-12T19:33:57.292 DEBUG:boto:Token: None 2022-09-12T19:33:57.293 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:57.293 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FILgvQL0So95OsflleI7MYirUZQ= 2022-09-12T19:33:57.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FILgvQL0So95OsflleI7MYirUZQ=', 'Content-Length': '0'} 2022-09-12T19:33:57.296 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000330280cc06a25696-00631f89a5-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-12T19:33:57.297 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:57.297 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:57.298 DEBUG:boto:path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:57.299 DEBUG:boto:auth_path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:57.299 DEBUG:boto:Method: GET 2022-09-12T19:33:57.300 DEBUG:boto:Path: /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:57.301 DEBUG:boto:Data: 2022-09-12T19:33:57.301 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.302 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.302 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.303 DEBUG:boto:Params: {} 2022-09-12T19:33:57.304 DEBUG:boto:Token: None 2022-09-12T19:33:57.304 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:57.305 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XyVZx30721j4xV8pF4nWZrGf22Y= 2022-09-12T19:33:57.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XyVZx30721j4xV8pF4nWZrGf22Y=', 'Content-Length': '0'} 2022-09-12T19:33:57.308 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006230cc99dacf5dd4-00631f89a5-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.309 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:57.310 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:57.310 DEBUG:boto:path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:57.311 DEBUG:boto:auth_path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:57.312 DEBUG:boto:Method: GET 2022-09-12T19:33:57.312 DEBUG:boto:Path: /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:57.313 DEBUG:boto:Data: 2022-09-12T19:33:57.313 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.314 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.314 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.315 DEBUG:boto:Params: {} 2022-09-12T19:33:57.316 DEBUG:boto:Token: None 2022-09-12T19:33:57.316 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:57.317 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XyVZx30721j4xV8pF4nWZrGf22Y= 2022-09-12T19:33:57.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XyVZx30721j4xV8pF4nWZrGf22Y=', 'Content-Length': '0'} 2022-09-12T19:33:57.320 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d426e3e9b9e209be-00631f89a5-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.321 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:57.322 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:57.322 DEBUG:boto:path=/jdpiak-62/?prefix=obj1 2022-09-12T19:33:57.323 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj1 2022-09-12T19:33:57.323 DEBUG:boto:Method: GET 2022-09-12T19:33:57.324 DEBUG:boto:Path: /jdpiak-62/?prefix=obj1 2022-09-12T19:33:57.325 DEBUG:boto:Data: 2022-09-12T19:33:57.325 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.326 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.327 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.327 DEBUG:boto:Params: {} 2022-09-12T19:33:57.328 DEBUG:boto:Token: None 2022-09-12T19:33:57.329 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/ 2022-09-12T19:33:57.330 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rvA7ZotoTZdukHze7LoyLWvzeRA= 2022-09-12T19:33:57.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rvA7ZotoTZdukHze7LoyLWvzeRA=', 'Content-Length': '0'} 2022-09-12T19:33:57.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8060a6b2917840f-00631f89a5-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.333 DEBUG:boto:b'jdpiak-62obj11000falseobj12022-09-12T19:32:15.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:57.334 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:57.335 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:57.336 DEBUG:boto:path=/jdpiak-62/?prefix=obj1 2022-09-12T19:33:57.336 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj1 2022-09-12T19:33:57.337 DEBUG:boto:Method: GET 2022-09-12T19:33:57.338 DEBUG:boto:Path: /jdpiak-62/?prefix=obj1 2022-09-12T19:33:57.338 DEBUG:boto:Data: 2022-09-12T19:33:57.339 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.340 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.340 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.341 DEBUG:boto:Params: {} 2022-09-12T19:33:57.341 DEBUG:boto:Token: None 2022-09-12T19:33:57.342 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/ 2022-09-12T19:33:57.343 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rvA7ZotoTZdukHze7LoyLWvzeRA= 2022-09-12T19:33:57.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rvA7ZotoTZdukHze7LoyLWvzeRA=', 'Content-Length': '0'} 2022-09-12T19:33:57.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009030c389222a000b-00631f89a5-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.347 DEBUG:boto:b'jdpiak-62obj11000falseobj12022-09-12T19:32:15.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:57.348 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:57.348 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:57.349 DEBUG:boto:Method: HEAD 2022-09-12T19:33:57.350 DEBUG:boto:Path: /jdpiak-62/obj1 2022-09-12T19:33:57.350 DEBUG:boto:Data: 2022-09-12T19:33:57.351 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.351 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.352 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.352 DEBUG:boto:Params: {} 2022-09-12T19:33:57.353 DEBUG:boto:Token: None 2022-09-12T19:33:57.354 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj1 2022-09-12T19:33:57.354 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sGM/63zlx6aSfU5zAWTgeCXuvvk= 2022-09-12T19:33:57.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sGM/63zlx6aSfU5zAWTgeCXuvvk=', 'Content-Length': '0'} 2022-09-12T19:33:57.357 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000832f25068b95597b-00631f89a5-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.357 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:57.358 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:57.359 DEBUG:boto:Method: HEAD 2022-09-12T19:33:57.359 DEBUG:boto:Path: /jdpiak-62/obj1 2022-09-12T19:33:57.360 DEBUG:boto:Data: 2022-09-12T19:33:57.360 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.361 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.362 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.362 DEBUG:boto:Params: {} 2022-09-12T19:33:57.363 DEBUG:boto:Token: None 2022-09-12T19:33:57.363 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj1 2022-09-12T19:33:57.364 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sGM/63zlx6aSfU5zAWTgeCXuvvk= 2022-09-12T19:33:57.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sGM/63zlx6aSfU5zAWTgeCXuvvk=', 'Content-Length': '0'} 2022-09-12T19:33:57.366 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c4f518645a6c9a9-00631f89a5-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-12T19:33:57.367 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:57.368 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:57.369 DEBUG:boto:Method: GET 2022-09-12T19:33:57.369 DEBUG:boto:Path: /jdpiak-62/obj1 2022-09-12T19:33:57.370 DEBUG:boto:Data: 2022-09-12T19:33:57.371 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.371 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.372 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.372 DEBUG:boto:Params: {} 2022-09-12T19:33:57.373 DEBUG:boto:Token: None 2022-09-12T19:33:57.373 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj1 2022-09-12T19:33:57.374 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WL/MQdCws7bivat14xkaAWDqeFY= 2022-09-12T19:33:57.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WL/MQdCws7bivat14xkaAWDqeFY=', 'Content-Length': '0'} 2022-09-12T19:33:57.377 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001626036fdabfa6be-00631f89a5-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.378 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:57.379 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:57.379 DEBUG:boto:Method: GET 2022-09-12T19:33:57.380 DEBUG:boto:Path: /jdpiak-62/obj1 2022-09-12T19:33:57.380 DEBUG:boto:Data: 2022-09-12T19:33:57.381 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.381 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.382 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.382 DEBUG:boto:Params: {} 2022-09-12T19:33:57.383 DEBUG:boto:Token: None 2022-09-12T19:33:57.384 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj1 2022-09-12T19:33:57.384 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WL/MQdCws7bivat14xkaAWDqeFY= 2022-09-12T19:33:57.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WL/MQdCws7bivat14xkaAWDqeFY=', 'Content-Length': '0'} 2022-09-12T19:33:57.387 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003080d5dbf2ae1aa4-00631f89a5-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.430 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:57.431 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:57.431 DEBUG:boto:path=/jdpiak-62/?prefix=obj1 2022-09-12T19:33:57.432 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj1 2022-09-12T19:33:57.432 DEBUG:boto:Method: GET 2022-09-12T19:33:57.433 DEBUG:boto:Path: /jdpiak-62/?prefix=obj1 2022-09-12T19:33:57.433 DEBUG:boto:Data: 2022-09-12T19:33:57.434 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.434 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.435 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.435 DEBUG:boto:Params: {} 2022-09-12T19:33:57.436 DEBUG:boto:Token: None 2022-09-12T19:33:57.437 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/ 2022-09-12T19:33:57.437 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rvA7ZotoTZdukHze7LoyLWvzeRA= 2022-09-12T19:33:57.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rvA7ZotoTZdukHze7LoyLWvzeRA=', 'Content-Length': '0'} 2022-09-12T19:33:57.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ada7b13f02529c0-00631f89a5-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.441 DEBUG:boto:b'jdpiak-62obj11000falseobj12022-09-12T19:32:15.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:57.442 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:57.443 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:57.443 DEBUG:boto:path=/jdpiak-62/?prefix=obj1 2022-09-12T19:33:57.444 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj1 2022-09-12T19:33:57.444 DEBUG:boto:Method: GET 2022-09-12T19:33:57.445 DEBUG:boto:Path: /jdpiak-62/?prefix=obj1 2022-09-12T19:33:57.445 DEBUG:boto:Data: 2022-09-12T19:33:57.446 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.446 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.447 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.447 DEBUG:boto:Params: {} 2022-09-12T19:33:57.448 DEBUG:boto:Token: None 2022-09-12T19:33:57.449 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/ 2022-09-12T19:33:57.449 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rvA7ZotoTZdukHze7LoyLWvzeRA= 2022-09-12T19:33:57.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rvA7ZotoTZdukHze7LoyLWvzeRA=', 'Content-Length': '0'} 2022-09-12T19:33:57.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090dd49619c867ad1-00631f89a5-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.453 DEBUG:boto:b'jdpiak-62obj11000falseobj12022-09-12T19:32:15.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:57.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-12T19:33:57.454 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:57.455 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:57.456 DEBUG:boto:path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:57.456 DEBUG:boto:auth_path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:57.457 DEBUG:boto:Method: GET 2022-09-12T19:33:57.457 DEBUG:boto:Path: /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:57.458 DEBUG:boto:Data: 2022-09-12T19:33:57.458 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.459 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.459 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.460 DEBUG:boto:Params: {} 2022-09-12T19:33:57.460 DEBUG:boto:Token: None 2022-09-12T19:33:57.461 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:57.462 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2ItkI4g1TZ+EdQAvdYhp5jFAPlQ= 2022-09-12T19:33:57.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2ItkI4g1TZ+EdQAvdYhp5jFAPlQ=', 'Content-Length': '0'} 2022-09-12T19:33:57.464 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b83b0db979273f0-00631f89a5-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.465 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:57.466 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:57.466 DEBUG:boto:path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:57.467 DEBUG:boto:auth_path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:57.467 DEBUG:boto:Method: GET 2022-09-12T19:33:57.468 DEBUG:boto:Path: /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:57.468 DEBUG:boto:Data: 2022-09-12T19:33:57.469 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.469 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.470 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.471 DEBUG:boto:Params: {} 2022-09-12T19:33:57.472 DEBUG:boto:Token: None 2022-09-12T19:33:57.472 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:57.473 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2ItkI4g1TZ+EdQAvdYhp5jFAPlQ= 2022-09-12T19:33:57.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2ItkI4g1TZ+EdQAvdYhp5jFAPlQ=', 'Content-Length': '0'} 2022-09-12T19:33:57.475 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5e1c340ba2fcd9b-00631f89a5-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.476 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:57.477 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:57.477 DEBUG:boto:path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:57.478 DEBUG:boto:auth_path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:57.478 DEBUG:boto:Method: HEAD 2022-09-12T19:33:57.479 DEBUG:boto:Path: /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:57.479 DEBUG:boto:Data: 2022-09-12T19:33:57.480 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.481 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.481 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.482 DEBUG:boto:Params: {} 2022-09-12T19:33:57.482 DEBUG:boto:Token: None 2022-09-12T19:33:57.483 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:57.483 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UPY15F7xBOVCzMItlWfDS2OWwfU= 2022-09-12T19:33:57.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UPY15F7xBOVCzMItlWfDS2OWwfU=', 'Content-Length': '0'} 2022-09-12T19:33:57.486 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d090c4405e4b716-00631f89a5-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.486 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:57.487 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:57.487 DEBUG:boto:path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:57.488 DEBUG:boto:auth_path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:57.489 DEBUG:boto:Method: HEAD 2022-09-12T19:33:57.489 DEBUG:boto:Path: /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:57.490 DEBUG:boto:Data: 2022-09-12T19:33:57.491 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.491 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.492 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.492 DEBUG:boto:Params: {} 2022-09-12T19:33:57.493 DEBUG:boto:Token: None 2022-09-12T19:33:57.493 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:57.494 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UPY15F7xBOVCzMItlWfDS2OWwfU= 2022-09-12T19:33:57.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UPY15F7xBOVCzMItlWfDS2OWwfU=', 'Content-Length': '0'} 2022-09-12T19:33:57.496 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045085bd43b6954af-00631f89a5-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-12T19:33:57.498 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:57.498 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:57.499 DEBUG:boto:path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:57.500 DEBUG:boto:auth_path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:57.500 DEBUG:boto:Method: GET 2022-09-12T19:33:57.501 DEBUG:boto:Path: /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:57.501 DEBUG:boto:Data: 2022-09-12T19:33:57.502 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.502 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.503 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.503 DEBUG:boto:Params: {} 2022-09-12T19:33:57.504 DEBUG:boto:Token: None 2022-09-12T19:33:57.505 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:57.505 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2ItkI4g1TZ+EdQAvdYhp5jFAPlQ= 2022-09-12T19:33:57.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2ItkI4g1TZ+EdQAvdYhp5jFAPlQ=', 'Content-Length': '0'} 2022-09-12T19:33:57.508 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007741cd362509f7d3-00631f89a5-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.509 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:57.509 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:57.510 DEBUG:boto:path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:57.511 DEBUG:boto:auth_path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:57.511 DEBUG:boto:Method: GET 2022-09-12T19:33:57.512 DEBUG:boto:Path: /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:57.512 DEBUG:boto:Data: 2022-09-12T19:33:57.513 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.513 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.514 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.514 DEBUG:boto:Params: {} 2022-09-12T19:33:57.515 DEBUG:boto:Token: None 2022-09-12T19:33:57.516 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:57.516 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2ItkI4g1TZ+EdQAvdYhp5jFAPlQ= 2022-09-12T19:33:57.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2ItkI4g1TZ+EdQAvdYhp5jFAPlQ=', 'Content-Length': '0'} 2022-09-12T19:33:57.519 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a621d7cdd739417f-00631f89a5-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.520 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:57.520 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:57.521 DEBUG:boto:path=/jdpiak-62/?prefix=obj2 2022-09-12T19:33:57.521 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj2 2022-09-12T19:33:57.522 DEBUG:boto:Method: GET 2022-09-12T19:33:57.523 DEBUG:boto:Path: /jdpiak-62/?prefix=obj2 2022-09-12T19:33:57.523 DEBUG:boto:Data: 2022-09-12T19:33:57.524 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.524 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.525 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.525 DEBUG:boto:Params: {} 2022-09-12T19:33:57.526 DEBUG:boto:Token: None 2022-09-12T19:33:57.526 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/ 2022-09-12T19:33:57.527 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rvA7ZotoTZdukHze7LoyLWvzeRA= 2022-09-12T19:33:57.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rvA7ZotoTZdukHze7LoyLWvzeRA=', 'Content-Length': '0'} 2022-09-12T19:33:57.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2afc160b6c59f94-00631f89a5-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.531 DEBUG:boto:b'jdpiak-62obj21000falseobj22022-09-12T19:32:15.070Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:57.532 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:57.533 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:57.533 DEBUG:boto:path=/jdpiak-62/?prefix=obj2 2022-09-12T19:33:57.534 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj2 2022-09-12T19:33:57.534 DEBUG:boto:Method: GET 2022-09-12T19:33:57.544 DEBUG:boto:Path: /jdpiak-62/?prefix=obj2 2022-09-12T19:33:57.545 DEBUG:boto:Data: 2022-09-12T19:33:57.546 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.546 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.547 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.547 DEBUG:boto:Params: {} 2022-09-12T19:33:57.548 DEBUG:boto:Token: None 2022-09-12T19:33:57.549 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/ 2022-09-12T19:33:57.549 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rvA7ZotoTZdukHze7LoyLWvzeRA= 2022-09-12T19:33:57.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rvA7ZotoTZdukHze7LoyLWvzeRA=', 'Content-Length': '0'} 2022-09-12T19:33:57.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007690b6d9d3682ae0-00631f89a5-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.554 DEBUG:boto:b'jdpiak-62obj21000falseobj22022-09-12T19:32:15.070Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:57.555 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:57.555 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:57.556 DEBUG:boto:Method: HEAD 2022-09-12T19:33:57.556 DEBUG:boto:Path: /jdpiak-62/obj2 2022-09-12T19:33:57.557 DEBUG:boto:Data: 2022-09-12T19:33:57.557 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.558 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.558 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.559 DEBUG:boto:Params: {} 2022-09-12T19:33:57.559 DEBUG:boto:Token: None 2022-09-12T19:33:57.560 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj2 2022-09-12T19:33:57.560 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fox/ubF5VZV6MJwUBaaycs9xzHY= 2022-09-12T19:33:57.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fox/ubF5VZV6MJwUBaaycs9xzHY=', 'Content-Length': '0'} 2022-09-12T19:33:57.563 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000545ade3a61fd8988-00631f89a5-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.564 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:57.564 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:57.565 DEBUG:boto:Method: HEAD 2022-09-12T19:33:57.565 DEBUG:boto:Path: /jdpiak-62/obj2 2022-09-12T19:33:57.565 DEBUG:boto:Data: 2022-09-12T19:33:57.566 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.566 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.567 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.567 DEBUG:boto:Params: {} 2022-09-12T19:33:57.568 DEBUG:boto:Token: None 2022-09-12T19:33:57.569 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj2 2022-09-12T19:33:57.569 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fox/ubF5VZV6MJwUBaaycs9xzHY= 2022-09-12T19:33:57.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fox/ubF5VZV6MJwUBaaycs9xzHY=', 'Content-Length': '0'} 2022-09-12T19:33:57.572 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f457da86163b8980-00631f89a5-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-12T19:33:57.573 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:57.573 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:57.574 DEBUG:boto:Method: GET 2022-09-12T19:33:57.575 DEBUG:boto:Path: /jdpiak-62/obj2 2022-09-12T19:33:57.575 DEBUG:boto:Data: 2022-09-12T19:33:57.576 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.576 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.577 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.577 DEBUG:boto:Params: {} 2022-09-12T19:33:57.578 DEBUG:boto:Token: None 2022-09-12T19:33:57.578 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj2 2022-09-12T19:33:57.579 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pGtaPXwT1HX04zmisDe1NYSgS0A= 2022-09-12T19:33:57.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pGtaPXwT1HX04zmisDe1NYSgS0A=', 'Content-Length': '0'} 2022-09-12T19:33:57.581 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f622117c94e415f-00631f89a5-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.622 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:57.623 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:57.623 DEBUG:boto:Method: GET 2022-09-12T19:33:57.624 DEBUG:boto:Path: /jdpiak-62/obj2 2022-09-12T19:33:57.624 DEBUG:boto:Data: 2022-09-12T19:33:57.625 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.625 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.626 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.626 DEBUG:boto:Params: {} 2022-09-12T19:33:57.627 DEBUG:boto:Token: None 2022-09-12T19:33:57.628 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj2 2022-09-12T19:33:57.628 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pGtaPXwT1HX04zmisDe1NYSgS0A= 2022-09-12T19:33:57.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pGtaPXwT1HX04zmisDe1NYSgS0A=', 'Content-Length': '0'} 2022-09-12T19:33:57.631 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021c452ea392e5b92-00631f89a5-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.632 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:57.633 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:57.633 DEBUG:boto:path=/jdpiak-62/?prefix=obj2 2022-09-12T19:33:57.634 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj2 2022-09-12T19:33:57.634 DEBUG:boto:Method: GET 2022-09-12T19:33:57.635 DEBUG:boto:Path: /jdpiak-62/?prefix=obj2 2022-09-12T19:33:57.635 DEBUG:boto:Data: 2022-09-12T19:33:57.636 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.636 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.637 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.637 DEBUG:boto:Params: {} 2022-09-12T19:33:57.638 DEBUG:boto:Token: None 2022-09-12T19:33:57.638 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/ 2022-09-12T19:33:57.639 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rvA7ZotoTZdukHze7LoyLWvzeRA= 2022-09-12T19:33:57.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rvA7ZotoTZdukHze7LoyLWvzeRA=', 'Content-Length': '0'} 2022-09-12T19:33:57.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037ad74045a829335-00631f89a5-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.643 DEBUG:boto:b'jdpiak-62obj21000falseobj22022-09-12T19:32:15.070Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:57.644 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:57.644 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:57.645 DEBUG:boto:path=/jdpiak-62/?prefix=obj2 2022-09-12T19:33:57.645 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj2 2022-09-12T19:33:57.646 DEBUG:boto:Method: GET 2022-09-12T19:33:57.646 DEBUG:boto:Path: /jdpiak-62/?prefix=obj2 2022-09-12T19:33:57.647 DEBUG:boto:Data: 2022-09-12T19:33:57.647 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.648 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.648 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.649 DEBUG:boto:Params: {} 2022-09-12T19:33:57.649 DEBUG:boto:Token: None 2022-09-12T19:33:57.650 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/ 2022-09-12T19:33:57.651 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rvA7ZotoTZdukHze7LoyLWvzeRA= 2022-09-12T19:33:57.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rvA7ZotoTZdukHze7LoyLWvzeRA=', 'Content-Length': '0'} 2022-09-12T19:33:57.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086010f57c72a8ac9-00631f89a5-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.654 DEBUG:boto:b'jdpiak-62obj21000falseobj22022-09-12T19:32:15.070Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:57.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-12T19:33:57.656 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:57.656 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:57.657 DEBUG:boto:path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:57.657 DEBUG:boto:auth_path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:57.658 DEBUG:boto:Method: GET 2022-09-12T19:33:57.659 DEBUG:boto:Path: /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:57.659 DEBUG:boto:Data: 2022-09-12T19:33:57.660 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.660 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.661 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.661 DEBUG:boto:Params: {} 2022-09-12T19:33:57.662 DEBUG:boto:Token: None 2022-09-12T19:33:57.662 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:57.663 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5ZxUXWr0g7j8+HRGk3cnEp4UcWc= 2022-09-12T19:33:57.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5ZxUXWr0g7j8+HRGk3cnEp4UcWc=', 'Content-Length': '0'} 2022-09-12T19:33:57.665 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007de7353b35149f16-00631f89a5-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.666 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:57.667 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:57.668 DEBUG:boto:path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:57.668 DEBUG:boto:auth_path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:57.669 DEBUG:boto:Method: GET 2022-09-12T19:33:57.669 DEBUG:boto:Path: /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:57.670 DEBUG:boto:Data: 2022-09-12T19:33:57.670 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.671 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.671 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.672 DEBUG:boto:Params: {} 2022-09-12T19:33:57.672 DEBUG:boto:Token: None 2022-09-12T19:33:57.673 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:57.674 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5ZxUXWr0g7j8+HRGk3cnEp4UcWc= 2022-09-12T19:33:57.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5ZxUXWr0g7j8+HRGk3cnEp4UcWc=', 'Content-Length': '0'} 2022-09-12T19:33:57.676 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c3b563686473b0c-00631f89a5-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.718 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:57.719 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:57.719 DEBUG:boto:path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:57.720 DEBUG:boto:auth_path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:57.720 DEBUG:boto:Method: HEAD 2022-09-12T19:33:57.721 DEBUG:boto:Path: /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:57.721 DEBUG:boto:Data: 2022-09-12T19:33:57.722 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.722 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.723 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.723 DEBUG:boto:Params: {} 2022-09-12T19:33:57.724 DEBUG:boto:Token: None 2022-09-12T19:33:57.724 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:57.725 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fjqw4ZrReAvSKP7xTYTeBWHu1Vo= 2022-09-12T19:33:57.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fjqw4ZrReAvSKP7xTYTeBWHu1Vo=', 'Content-Length': '0'} 2022-09-12T19:33:57.727 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045e62245f8110887-00631f89a5-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.728 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:57.729 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:57.729 DEBUG:boto:path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:57.730 DEBUG:boto:auth_path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:57.730 DEBUG:boto:Method: HEAD 2022-09-12T19:33:57.731 DEBUG:boto:Path: /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:57.731 DEBUG:boto:Data: 2022-09-12T19:33:57.732 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.732 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.733 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.733 DEBUG:boto:Params: {} 2022-09-12T19:33:57.734 DEBUG:boto:Token: None 2022-09-12T19:33:57.734 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:57.735 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fjqw4ZrReAvSKP7xTYTeBWHu1Vo= 2022-09-12T19:33:57.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fjqw4ZrReAvSKP7xTYTeBWHu1Vo=', 'Content-Length': '0'} 2022-09-12T19:33:57.737 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002af0c78e9fb84a08-00631f89a5-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-12T19:33:57.739 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:57.739 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:57.740 DEBUG:boto:path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:57.740 DEBUG:boto:auth_path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:57.741 DEBUG:boto:Method: GET 2022-09-12T19:33:57.741 DEBUG:boto:Path: /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:57.742 DEBUG:boto:Data: 2022-09-12T19:33:57.742 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.743 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.743 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.744 DEBUG:boto:Params: {} 2022-09-12T19:33:57.744 DEBUG:boto:Token: None 2022-09-12T19:33:57.745 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:57.746 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5ZxUXWr0g7j8+HRGk3cnEp4UcWc= 2022-09-12T19:33:57.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5ZxUXWr0g7j8+HRGk3cnEp4UcWc=', 'Content-Length': '0'} 2022-09-12T19:33:57.748 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e97797a3af2e415f-00631f89a5-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.749 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:57.749 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:57.750 DEBUG:boto:path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:57.750 DEBUG:boto:auth_path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:57.751 DEBUG:boto:Method: GET 2022-09-12T19:33:57.751 DEBUG:boto:Path: /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:57.752 DEBUG:boto:Data: 2022-09-12T19:33:57.753 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.753 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.754 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.754 DEBUG:boto:Params: {} 2022-09-12T19:33:57.766 DEBUG:boto:Token: None 2022-09-12T19:33:57.767 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:57.767 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5ZxUXWr0g7j8+HRGk3cnEp4UcWc= 2022-09-12T19:33:57.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5ZxUXWr0g7j8+HRGk3cnEp4UcWc=', 'Content-Length': '0'} 2022-09-12T19:33:57.770 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a201332d718b021-00631f89a5-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.771 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:57.771 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:57.772 DEBUG:boto:path=/jdpiak-62/?prefix=obj3 2022-09-12T19:33:57.772 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj3 2022-09-12T19:33:57.773 DEBUG:boto:Method: GET 2022-09-12T19:33:57.774 DEBUG:boto:Path: /jdpiak-62/?prefix=obj3 2022-09-12T19:33:57.774 DEBUG:boto:Data: 2022-09-12T19:33:57.775 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.775 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.776 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.776 DEBUG:boto:Params: {} 2022-09-12T19:33:57.777 DEBUG:boto:Token: None 2022-09-12T19:33:57.777 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/ 2022-09-12T19:33:57.778 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rvA7ZotoTZdukHze7LoyLWvzeRA= 2022-09-12T19:33:57.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rvA7ZotoTZdukHze7LoyLWvzeRA=', 'Content-Length': '0'} 2022-09-12T19:33:57.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000716b14c094943db9-00631f89a5-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.782 DEBUG:boto:b'jdpiak-62obj31000falseobj32022-09-12T19:32:15.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:57.783 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:57.783 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:57.784 DEBUG:boto:path=/jdpiak-62/?prefix=obj3 2022-09-12T19:33:57.784 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj3 2022-09-12T19:33:57.785 DEBUG:boto:Method: GET 2022-09-12T19:33:57.785 DEBUG:boto:Path: /jdpiak-62/?prefix=obj3 2022-09-12T19:33:57.786 DEBUG:boto:Data: 2022-09-12T19:33:57.786 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.787 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.787 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.788 DEBUG:boto:Params: {} 2022-09-12T19:33:57.788 DEBUG:boto:Token: None 2022-09-12T19:33:57.789 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/ 2022-09-12T19:33:57.789 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rvA7ZotoTZdukHze7LoyLWvzeRA= 2022-09-12T19:33:57.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rvA7ZotoTZdukHze7LoyLWvzeRA=', 'Content-Length': '0'} 2022-09-12T19:33:57.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b9d1c3f60df06d4-00631f89a5-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.793 DEBUG:boto:b'jdpiak-62obj31000falseobj32022-09-12T19:32:15.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:57.794 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:57.795 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:57.795 DEBUG:boto:Method: HEAD 2022-09-12T19:33:57.796 DEBUG:boto:Path: /jdpiak-62/obj3 2022-09-12T19:33:57.796 DEBUG:boto:Data: 2022-09-12T19:33:57.797 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.797 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.798 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.798 DEBUG:boto:Params: {} 2022-09-12T19:33:57.799 DEBUG:boto:Token: None 2022-09-12T19:33:57.799 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj3 2022-09-12T19:33:57.800 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ye9NmSML0/4ddrn+5DXvIighRqA= 2022-09-12T19:33:57.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ye9NmSML0/4ddrn+5DXvIighRqA=', 'Content-Length': '0'} 2022-09-12T19:33:57.802 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021a777096c52a96a-00631f89a5-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.803 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:57.804 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:57.804 DEBUG:boto:Method: HEAD 2022-09-12T19:33:57.805 DEBUG:boto:Path: /jdpiak-62/obj3 2022-09-12T19:33:57.805 DEBUG:boto:Data: 2022-09-12T19:33:57.806 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.806 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.807 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.807 DEBUG:boto:Params: {} 2022-09-12T19:33:57.808 DEBUG:boto:Token: None 2022-09-12T19:33:57.808 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj3 2022-09-12T19:33:57.809 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ye9NmSML0/4ddrn+5DXvIighRqA= 2022-09-12T19:33:57.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ye9NmSML0/4ddrn+5DXvIighRqA=', 'Content-Length': '0'} 2022-09-12T19:33:57.812 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b1f07eaf8064ade-00631f89a5-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-12T19:33:57.813 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:57.813 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:57.814 DEBUG:boto:Method: GET 2022-09-12T19:33:57.815 DEBUG:boto:Path: /jdpiak-62/obj3 2022-09-12T19:33:57.815 DEBUG:boto:Data: 2022-09-12T19:33:57.816 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.816 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.817 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.817 DEBUG:boto:Params: {} 2022-09-12T19:33:57.818 DEBUG:boto:Token: None 2022-09-12T19:33:57.818 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj3 2022-09-12T19:33:57.819 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:auc2zj5J7i0S2k5G7reBGJix73I= 2022-09-12T19:33:57.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:auc2zj5J7i0S2k5G7reBGJix73I=', 'Content-Length': '0'} 2022-09-12T19:33:57.822 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003089daf212e3d509-00631f89a5-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.866 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:57.867 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:57.868 DEBUG:boto:Method: GET 2022-09-12T19:33:57.868 DEBUG:boto:Path: /jdpiak-62/obj3 2022-09-12T19:33:57.869 DEBUG:boto:Data: 2022-09-12T19:33:57.871 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.871 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.872 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.872 DEBUG:boto:Params: {} 2022-09-12T19:33:57.873 DEBUG:boto:Token: None 2022-09-12T19:33:57.874 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj3 2022-09-12T19:33:57.874 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:auc2zj5J7i0S2k5G7reBGJix73I= 2022-09-12T19:33:57.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:auc2zj5J7i0S2k5G7reBGJix73I=', 'Content-Length': '0'} 2022-09-12T19:33:57.878 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f82b3eddddd2b8b-00631f89a5-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.879 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:57.879 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:57.880 DEBUG:boto:path=/jdpiak-62/?prefix=obj3 2022-09-12T19:33:57.881 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj3 2022-09-12T19:33:57.881 DEBUG:boto:Method: GET 2022-09-12T19:33:57.882 DEBUG:boto:Path: /jdpiak-62/?prefix=obj3 2022-09-12T19:33:57.882 DEBUG:boto:Data: 2022-09-12T19:33:57.883 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.883 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.884 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.884 DEBUG:boto:Params: {} 2022-09-12T19:33:57.885 DEBUG:boto:Token: None 2022-09-12T19:33:57.886 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/ 2022-09-12T19:33:57.887 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rvA7ZotoTZdukHze7LoyLWvzeRA= 2022-09-12T19:33:57.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rvA7ZotoTZdukHze7LoyLWvzeRA=', 'Content-Length': '0'} 2022-09-12T19:33:57.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000df79112b8ccb6d2-00631f89a5-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.891 DEBUG:boto:b'jdpiak-62obj31000falseobj32022-09-12T19:32:15.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:57.892 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:57.893 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:57.893 DEBUG:boto:path=/jdpiak-62/?prefix=obj3 2022-09-12T19:33:57.894 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj3 2022-09-12T19:33:57.894 DEBUG:boto:Method: GET 2022-09-12T19:33:57.895 DEBUG:boto:Path: /jdpiak-62/?prefix=obj3 2022-09-12T19:33:57.895 DEBUG:boto:Data: 2022-09-12T19:33:57.896 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.896 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.897 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.897 DEBUG:boto:Params: {} 2022-09-12T19:33:57.898 DEBUG:boto:Token: None 2022-09-12T19:33:57.899 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/ 2022-09-12T19:33:57.899 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rvA7ZotoTZdukHze7LoyLWvzeRA= 2022-09-12T19:33:57.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rvA7ZotoTZdukHze7LoyLWvzeRA=', 'Content-Length': '0'} 2022-09-12T19:33:57.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009064eac56150c9f-00631f89a5-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.903 DEBUG:boto:b'jdpiak-62obj31000falseobj32022-09-12T19:32:15.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:57.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-12T19:33:57.905 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:33:57.905 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:33:57.906 DEBUG:boto:path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:57.906 DEBUG:boto:auth_path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:57.907 DEBUG:boto:Method: GET 2022-09-12T19:33:57.907 DEBUG:boto:Path: /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:57.908 DEBUG:boto:Data: 2022-09-12T19:33:57.908 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.909 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.909 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.910 DEBUG:boto:Params: {} 2022-09-12T19:33:57.911 DEBUG:boto:Token: None 2022-09-12T19:33:57.911 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:57.912 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zqHQuIcQrCnuywnaq7CGa7vXBVg= 2022-09-12T19:33:57.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zqHQuIcQrCnuywnaq7CGa7vXBVg=', 'Content-Length': '0'} 2022-09-12T19:33:57.915 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2113ffaccbcb365-00631f89a5-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.916 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:33:57.916 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:33:57.917 DEBUG:boto:path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:57.917 DEBUG:boto:auth_path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:57.918 DEBUG:boto:Method: GET 2022-09-12T19:33:57.918 DEBUG:boto:Path: /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:57.919 DEBUG:boto:Data: 2022-09-12T19:33:57.919 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.920 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.921 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.921 DEBUG:boto:Params: {} 2022-09-12T19:33:57.922 DEBUG:boto:Token: None 2022-09-12T19:33:57.922 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:57.923 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zqHQuIcQrCnuywnaq7CGa7vXBVg= 2022-09-12T19:33:57.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zqHQuIcQrCnuywnaq7CGa7vXBVg=', 'Content-Length': '0'} 2022-09-12T19:33:57.926 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000722ef17f072ade8-00631f89a5-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.966 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:33:57.967 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:33:57.968 DEBUG:boto:path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:57.968 DEBUG:boto:auth_path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:57.969 DEBUG:boto:Method: HEAD 2022-09-12T19:33:57.970 DEBUG:boto:Path: /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:57.970 DEBUG:boto:Data: 2022-09-12T19:33:57.971 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.971 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:57.972 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.972 DEBUG:boto:Params: {} 2022-09-12T19:33:57.973 DEBUG:boto:Token: None 2022-09-12T19:33:57.973 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:57.974 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Izue6RJnhNr1tM13pJey53cHIw4= 2022-09-12T19:33:57.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Izue6RJnhNr1tM13pJey53cHIw4=', 'Content-Length': '0'} 2022-09-12T19:33:57.977 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a091fccc1e9228dc-00631f89a5-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.978 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:33:57.978 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:33:57.979 DEBUG:boto:path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:57.979 DEBUG:boto:auth_path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:57.980 DEBUG:boto:Method: HEAD 2022-09-12T19:33:57.980 DEBUG:boto:Path: /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:57.981 DEBUG:boto:Data: 2022-09-12T19:33:57.981 DEBUG:boto:Headers: {} 2022-09-12T19:33:57.982 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:57.983 DEBUG:boto:Port: 8000 2022-09-12T19:33:57.984 DEBUG:boto:Params: {} 2022-09-12T19:33:57.985 DEBUG:boto:Token: None 2022-09-12T19:33:57.986 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:57 GMT /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:57.987 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Izue6RJnhNr1tM13pJey53cHIw4= 2022-09-12T19:33:57.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Izue6RJnhNr1tM13pJey53cHIw4=', 'Content-Length': '0'} 2022-09-12T19:33:57.991 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076dd0ac4f869ae1c-00631f89a5-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:57.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-12T19:33:57.994 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:33:57.995 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:33:57.996 DEBUG:boto:path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:57.997 DEBUG:boto:auth_path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:57.998 DEBUG:boto:Method: GET 2022-09-12T19:33:57.999 DEBUG:boto:Path: /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.000 DEBUG:boto:Data: 2022-09-12T19:33:58.001 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.002 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:58.003 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.004 DEBUG:boto:Params: {} 2022-09-12T19:33:58.005 DEBUG:boto:Token: None 2022-09-12T19:33:58.006 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.007 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sX2S6gZBYu4h1gFJkN6pUIo+LyY= 2022-09-12T19:33:58.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sX2S6gZBYu4h1gFJkN6pUIo+LyY=', 'Content-Length': '0'} 2022-09-12T19:33:58.011 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9464e54886bb3c0-00631f89a6-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.012 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:33:58.013 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:33:58.014 DEBUG:boto:path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.015 DEBUG:boto:auth_path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.016 DEBUG:boto:Method: GET 2022-09-12T19:33:58.017 DEBUG:boto:Path: /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.018 DEBUG:boto:Data: 2022-09-12T19:33:58.019 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.020 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.021 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.022 DEBUG:boto:Params: {} 2022-09-12T19:33:58.023 DEBUG:boto:Token: None 2022-09-12T19:33:58.024 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.026 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sX2S6gZBYu4h1gFJkN6pUIo+LyY= 2022-09-12T19:33:58.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sX2S6gZBYu4h1gFJkN6pUIo+LyY=', 'Content-Length': '0'} 2022-09-12T19:33:58.030 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057042b2dc94e6526-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.031 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:58.032 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:58.033 DEBUG:boto:path=/jdpiak-62/?prefix=obj4 2022-09-12T19:33:58.034 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj4 2022-09-12T19:33:58.035 DEBUG:boto:Method: GET 2022-09-12T19:33:58.036 DEBUG:boto:Path: /jdpiak-62/?prefix=obj4 2022-09-12T19:33:58.037 DEBUG:boto:Data: 2022-09-12T19:33:58.038 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.039 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:58.040 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.040 DEBUG:boto:Params: {} 2022-09-12T19:33:58.041 DEBUG:boto:Token: None 2022-09-12T19:33:58.041 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/ 2022-09-12T19:33:58.042 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY= 2022-09-12T19:33:58.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY=', 'Content-Length': '0'} 2022-09-12T19:33:58.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b09df57de270202-00631f89a6-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.090 DEBUG:boto:b'jdpiak-62obj41000falseobj42022-09-12T19:32:15.218Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:58.091 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:58.092 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:58.092 DEBUG:boto:path=/jdpiak-62/?prefix=obj4 2022-09-12T19:33:58.093 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj4 2022-09-12T19:33:58.093 DEBUG:boto:Method: GET 2022-09-12T19:33:58.094 DEBUG:boto:Path: /jdpiak-62/?prefix=obj4 2022-09-12T19:33:58.094 DEBUG:boto:Data: 2022-09-12T19:33:58.095 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.096 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.096 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.097 DEBUG:boto:Params: {} 2022-09-12T19:33:58.098 DEBUG:boto:Token: None 2022-09-12T19:33:58.098 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/ 2022-09-12T19:33:58.099 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY= 2022-09-12T19:33:58.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY=', 'Content-Length': '0'} 2022-09-12T19:33:58.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebbca5c0ad5d2f15-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.103 DEBUG:boto:b'jdpiak-62obj41000falseobj42022-09-12T19:32:15.218Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:58.104 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:33:58.105 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:33:58.106 DEBUG:boto:Method: HEAD 2022-09-12T19:33:58.107 DEBUG:boto:Path: /jdpiak-62/obj4 2022-09-12T19:33:58.107 DEBUG:boto:Data: 2022-09-12T19:33:58.108 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.108 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:58.109 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.109 DEBUG:boto:Params: {} 2022-09-12T19:33:58.110 DEBUG:boto:Token: None 2022-09-12T19:33:58.111 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj4 2022-09-12T19:33:58.111 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c/S57QfcSrndPoNiov5rD3VQ3Bc= 2022-09-12T19:33:58.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c/S57QfcSrndPoNiov5rD3VQ3Bc=', 'Content-Length': '0'} 2022-09-12T19:33:58.114 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000078caa8329ea9c42-00631f89a6-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.115 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:33:58.116 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:33:58.117 DEBUG:boto:Method: HEAD 2022-09-12T19:33:58.117 DEBUG:boto:Path: /jdpiak-62/obj4 2022-09-12T19:33:58.118 DEBUG:boto:Data: 2022-09-12T19:33:58.119 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.120 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.121 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.122 DEBUG:boto:Params: {} 2022-09-12T19:33:58.123 DEBUG:boto:Token: None 2022-09-12T19:33:58.124 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj4 2022-09-12T19:33:58.125 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c/S57QfcSrndPoNiov5rD3VQ3Bc= 2022-09-12T19:33:58.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c/S57QfcSrndPoNiov5rD3VQ3Bc=', 'Content-Length': '0'} 2022-09-12T19:33:58.129 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5b51ae0d31e01ad-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-12T19:33:58.130 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:33:58.131 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:33:58.131 DEBUG:boto:Method: GET 2022-09-12T19:33:58.132 DEBUG:boto:Path: /jdpiak-62/obj4 2022-09-12T19:33:58.133 DEBUG:boto:Data: 2022-09-12T19:33:58.134 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.134 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:58.135 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.135 DEBUG:boto:Params: {} 2022-09-12T19:33:58.136 DEBUG:boto:Token: None 2022-09-12T19:33:58.137 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj4 2022-09-12T19:33:58.137 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qwJMAUbCmRUAhCAtkaI+y/wLUog= 2022-09-12T19:33:58.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qwJMAUbCmRUAhCAtkaI+y/wLUog=', 'Content-Length': '0'} 2022-09-12T19:33:58.141 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5d94ce9a32f75e5-00631f89a6-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.142 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:33:58.143 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:33:58.144 DEBUG:boto:Method: GET 2022-09-12T19:33:58.145 DEBUG:boto:Path: /jdpiak-62/obj4 2022-09-12T19:33:58.146 DEBUG:boto:Data: 2022-09-12T19:33:58.147 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.149 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.151 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.152 DEBUG:boto:Params: {} 2022-09-12T19:33:58.153 DEBUG:boto:Token: None 2022-09-12T19:33:58.154 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj4 2022-09-12T19:33:58.155 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qwJMAUbCmRUAhCAtkaI+y/wLUog= 2022-09-12T19:33:58.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qwJMAUbCmRUAhCAtkaI+y/wLUog=', 'Content-Length': '0'} 2022-09-12T19:33:58.159 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd94f1bf04c67474-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.160 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:58.162 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:58.162 DEBUG:boto:path=/jdpiak-62/?prefix=obj4 2022-09-12T19:33:58.163 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj4 2022-09-12T19:33:58.163 DEBUG:boto:Method: GET 2022-09-12T19:33:58.164 DEBUG:boto:Path: /jdpiak-62/?prefix=obj4 2022-09-12T19:33:58.164 DEBUG:boto:Data: 2022-09-12T19:33:58.165 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.165 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:58.166 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.166 DEBUG:boto:Params: {} 2022-09-12T19:33:58.167 DEBUG:boto:Token: None 2022-09-12T19:33:58.168 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/ 2022-09-12T19:33:58.168 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY= 2022-09-12T19:33:58.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY=', 'Content-Length': '0'} 2022-09-12T19:33:58.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074efd0c085269955-00631f89a6-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.172 DEBUG:boto:b'jdpiak-62obj41000falseobj42022-09-12T19:32:15.218Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:58.173 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:58.174 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:58.174 DEBUG:boto:path=/jdpiak-62/?prefix=obj4 2022-09-12T19:33:58.175 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj4 2022-09-12T19:33:58.175 DEBUG:boto:Method: GET 2022-09-12T19:33:58.176 DEBUG:boto:Path: /jdpiak-62/?prefix=obj4 2022-09-12T19:33:58.176 DEBUG:boto:Data: 2022-09-12T19:33:58.177 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.177 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.178 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.179 DEBUG:boto:Params: {} 2022-09-12T19:33:58.179 DEBUG:boto:Token: None 2022-09-12T19:33:58.180 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/ 2022-09-12T19:33:58.180 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY= 2022-09-12T19:33:58.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY=', 'Content-Length': '0'} 2022-09-12T19:33:58.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bcca32a80a7b103-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.185 DEBUG:boto:b'jdpiak-62obj41000falseobj42022-09-12T19:32:15.218Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:58.185 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-62 zones={test-zone2, test-zone1} 2022-09-12T19:33:58.186 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-62 zones={test-zone3, test-zone1} 2022-09-12T19:33:58.187 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:58.187 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:58.188 DEBUG:boto:Method: HEAD 2022-09-12T19:33:58.188 DEBUG:boto:Path: /jdpiak-62/ 2022-09-12T19:33:58.189 DEBUG:boto:Data: 2022-09-12T19:33:58.189 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.190 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.190 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.191 DEBUG:boto:Params: {} 2022-09-12T19:33:58.191 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T19:33:58.192 DEBUG:boto:Token: None 2022-09-12T19:33:58.193 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/ 2022-09-12T19:33:58.194 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+XAJ/25uchm3d1rW3cs6SAtxu4o= 2022-09-12T19:33:58.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+XAJ/25uchm3d1rW3cs6SAtxu4o=', 'Content-Length': '0'} 2022-09-12T19:33:58.198 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c17b544209c7fb5-00631f89a6-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.199 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:58.199 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:58.200 DEBUG:boto:Method: HEAD 2022-09-12T19:33:58.201 DEBUG:boto:Path: /jdpiak-62/ 2022-09-12T19:33:58.201 DEBUG:boto:Data: 2022-09-12T19:33:58.202 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.202 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.203 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.203 DEBUG:boto:Params: {} 2022-09-12T19:33:58.204 DEBUG:boto:Token: None 2022-09-12T19:33:58.205 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/ 2022-09-12T19:33:58.205 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+XAJ/25uchm3d1rW3cs6SAtxu4o= 2022-09-12T19:33:58.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+XAJ/25uchm3d1rW3cs6SAtxu4o=', 'Content-Length': '0'} 2022-09-12T19:33:58.209 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000178a11fa751cbbb4-00631f89a6-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.209 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:33:58.210 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:58.211 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:58.211 DEBUG:boto:path=/jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:58.212 DEBUG:boto:auth_path=/jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:58.212 DEBUG:boto:Method: GET 2022-09-12T19:33:58.213 DEBUG:boto:Path: /jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:58.213 DEBUG:boto:Data: 2022-09-12T19:33:58.214 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.215 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.215 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.216 DEBUG:boto:Params: {} 2022-09-12T19:33:58.216 DEBUG:boto:Token: None 2022-09-12T19:33:58.217 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/?versions 2022-09-12T19:33:58.217 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:a2MklRgmftYgi6+UAgxKKyQctw8= 2022-09-12T19:33:58.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:a2MklRgmftYgi6+UAgxKKyQctw8=', 'Content-Length': '0'} 2022-09-12T19:33:58.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086c65839f47d848d-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.222 DEBUG:boto:b'jdpiak-62999falseobj1nulltrue2022-09-12T19:32:15.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-12T19:32:15.070Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-12T19:32:15.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-12T19:32:15.218Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:58.223 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-12T19:33:58.223 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-12T19:33:58.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-12T19:33:58.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-12T19:33:58.225 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:33:58.225 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:58.226 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:58.227 DEBUG:boto:path=/jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:58.227 DEBUG:boto:auth_path=/jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:58.228 DEBUG:boto:Method: GET 2022-09-12T19:33:58.228 DEBUG:boto:Path: /jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:58.229 DEBUG:boto:Data: 2022-09-12T19:33:58.229 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.230 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.230 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.231 DEBUG:boto:Params: {} 2022-09-12T19:33:58.232 DEBUG:boto:Token: None 2022-09-12T19:33:58.232 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/?versions 2022-09-12T19:33:58.233 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:a2MklRgmftYgi6+UAgxKKyQctw8= 2022-09-12T19:33:58.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:a2MklRgmftYgi6+UAgxKKyQctw8=', 'Content-Length': '0'} 2022-09-12T19:33:58.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000528387a0a86db120-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.237 DEBUG:boto:b'jdpiak-62999falseobj1nulltrue2022-09-12T19:32:15.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-12T19:32:15.070Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-12T19:32:15.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-12T19:32:15.218Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:58.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-12T19:33:58.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-12T19:33:58.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-12T19:33:58.240 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-12T19:33:58.240 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:58.241 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:58.242 DEBUG:boto:path=/jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:58.243 DEBUG:boto:auth_path=/jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:58.243 DEBUG:boto:Method: GET 2022-09-12T19:33:58.244 DEBUG:boto:Path: /jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:58.244 DEBUG:boto:Data: 2022-09-12T19:33:58.245 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.245 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.246 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.246 DEBUG:boto:Params: {} 2022-09-12T19:33:58.247 DEBUG:boto:Token: None 2022-09-12T19:33:58.248 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/?versions 2022-09-12T19:33:58.248 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:a2MklRgmftYgi6+UAgxKKyQctw8= 2022-09-12T19:33:58.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:a2MklRgmftYgi6+UAgxKKyQctw8=', 'Content-Length': '0'} 2022-09-12T19:33:58.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058cd3a4b3028de02-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.252 DEBUG:boto:b'jdpiak-62999falseobj1nulltrue2022-09-12T19:32:15.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-12T19:32:15.070Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-12T19:32:15.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-12T19:32:15.218Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:58.253 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:58.254 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:58.254 DEBUG:boto:path=/jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:58.255 DEBUG:boto:auth_path=/jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:58.255 DEBUG:boto:Method: GET 2022-09-12T19:33:58.256 DEBUG:boto:Path: /jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:58.256 DEBUG:boto:Data: 2022-09-12T19:33:58.257 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.257 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.258 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.259 DEBUG:boto:Params: {} 2022-09-12T19:33:58.259 DEBUG:boto:Token: None 2022-09-12T19:33:58.260 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/?versions 2022-09-12T19:33:58.260 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:a2MklRgmftYgi6+UAgxKKyQctw8= 2022-09-12T19:33:58.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:a2MklRgmftYgi6+UAgxKKyQctw8=', 'Content-Length': '0'} 2022-09-12T19:33:58.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d82e7ed9485d659b-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.265 DEBUG:boto:b'jdpiak-62999falseobj1nulltrue2022-09-12T19:32:15.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-12T19:32:15.070Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-12T19:32:15.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-12T19:32:15.218Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:58.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-12T19:33:58.266 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:58.267 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:58.267 DEBUG:boto:path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:58.268 DEBUG:boto:auth_path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:58.268 DEBUG:boto:Method: GET 2022-09-12T19:33:58.269 DEBUG:boto:Path: /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:58.269 DEBUG:boto:Data: 2022-09-12T19:33:58.270 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.271 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.271 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.272 DEBUG:boto:Params: {} 2022-09-12T19:33:58.272 DEBUG:boto:Token: None 2022-09-12T19:33:58.273 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:58.274 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oK/LGqsURMIhEF8fe7R7dWwx8Ys= 2022-09-12T19:33:58.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oK/LGqsURMIhEF8fe7R7dWwx8Ys=', 'Content-Length': '0'} 2022-09-12T19:33:58.276 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008824a30a9a07d7b9-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.277 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:58.278 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:58.278 DEBUG:boto:path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:58.279 DEBUG:boto:auth_path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:58.279 DEBUG:boto:Method: GET 2022-09-12T19:33:58.280 DEBUG:boto:Path: /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:58.281 DEBUG:boto:Data: 2022-09-12T19:33:58.281 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.282 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.282 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.283 DEBUG:boto:Params: {} 2022-09-12T19:33:58.284 DEBUG:boto:Token: None 2022-09-12T19:33:58.284 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:58.285 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oK/LGqsURMIhEF8fe7R7dWwx8Ys= 2022-09-12T19:33:58.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oK/LGqsURMIhEF8fe7R7dWwx8Ys=', 'Content-Length': '0'} 2022-09-12T19:33:58.288 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f888a3c2c34ff06f-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.289 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:58.289 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:58.290 DEBUG:boto:path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:58.290 DEBUG:boto:auth_path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:58.291 DEBUG:boto:Method: HEAD 2022-09-12T19:33:58.291 DEBUG:boto:Path: /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:58.292 DEBUG:boto:Data: 2022-09-12T19:33:58.292 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.293 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.293 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.294 DEBUG:boto:Params: {} 2022-09-12T19:33:58.294 DEBUG:boto:Token: None 2022-09-12T19:33:58.295 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:58.296 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uI1O5eGlVALZfkZE/Y7g94SW4B0= 2022-09-12T19:33:58.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uI1O5eGlVALZfkZE/Y7g94SW4B0=', 'Content-Length': '0'} 2022-09-12T19:33:58.298 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015ad4374ed9ab6b4-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.299 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:58.299 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:58.300 DEBUG:boto:path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:58.300 DEBUG:boto:auth_path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:58.301 DEBUG:boto:Method: HEAD 2022-09-12T19:33:58.301 DEBUG:boto:Path: /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:58.302 DEBUG:boto:Data: 2022-09-12T19:33:58.302 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.303 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.303 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.304 DEBUG:boto:Params: {} 2022-09-12T19:33:58.304 DEBUG:boto:Token: None 2022-09-12T19:33:58.305 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:58.306 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uI1O5eGlVALZfkZE/Y7g94SW4B0= 2022-09-12T19:33:58.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uI1O5eGlVALZfkZE/Y7g94SW4B0=', 'Content-Length': '0'} 2022-09-12T19:33:58.308 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ced7597d7d0e7b6e-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-12T19:33:58.310 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:58.310 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:58.311 DEBUG:boto:path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:58.311 DEBUG:boto:auth_path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:58.312 DEBUG:boto:Method: GET 2022-09-12T19:33:58.313 DEBUG:boto:Path: /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:58.313 DEBUG:boto:Data: 2022-09-12T19:33:58.314 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.314 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.315 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.316 DEBUG:boto:Params: {} 2022-09-12T19:33:58.316 DEBUG:boto:Token: None 2022-09-12T19:33:58.317 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:58.318 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oK/LGqsURMIhEF8fe7R7dWwx8Ys= 2022-09-12T19:33:58.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oK/LGqsURMIhEF8fe7R7dWwx8Ys=', 'Content-Length': '0'} 2022-09-12T19:33:58.320 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0fc17c5290b70ac-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.321 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:58.322 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:58.323 DEBUG:boto:path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:58.323 DEBUG:boto:auth_path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:58.324 DEBUG:boto:Method: GET 2022-09-12T19:33:58.324 DEBUG:boto:Path: /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:58.325 DEBUG:boto:Data: 2022-09-12T19:33:58.325 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.326 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.327 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.327 DEBUG:boto:Params: {} 2022-09-12T19:33:58.328 DEBUG:boto:Token: None 2022-09-12T19:33:58.328 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:58.329 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oK/LGqsURMIhEF8fe7R7dWwx8Ys= 2022-09-12T19:33:58.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oK/LGqsURMIhEF8fe7R7dWwx8Ys=', 'Content-Length': '0'} 2022-09-12T19:33:58.332 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd57c426e4c21bcd-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.333 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:58.334 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:58.334 DEBUG:boto:path=/jdpiak-62/?prefix=obj1 2022-09-12T19:33:58.335 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj1 2022-09-12T19:33:58.335 DEBUG:boto:Method: GET 2022-09-12T19:33:58.336 DEBUG:boto:Path: /jdpiak-62/?prefix=obj1 2022-09-12T19:33:58.337 DEBUG:boto:Data: 2022-09-12T19:33:58.337 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.338 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.338 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.339 DEBUG:boto:Params: {} 2022-09-12T19:33:58.340 DEBUG:boto:Token: None 2022-09-12T19:33:58.340 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/ 2022-09-12T19:33:58.341 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY= 2022-09-12T19:33:58.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY=', 'Content-Length': '0'} 2022-09-12T19:33:58.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b900da1469da5261-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.345 DEBUG:boto:b'jdpiak-62obj11000falseobj12022-09-12T19:32:15.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:58.346 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:58.346 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:58.347 DEBUG:boto:path=/jdpiak-62/?prefix=obj1 2022-09-12T19:33:58.348 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj1 2022-09-12T19:33:58.348 DEBUG:boto:Method: GET 2022-09-12T19:33:58.349 DEBUG:boto:Path: /jdpiak-62/?prefix=obj1 2022-09-12T19:33:58.350 DEBUG:boto:Data: 2022-09-12T19:33:58.350 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.351 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.351 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.352 DEBUG:boto:Params: {} 2022-09-12T19:33:58.352 DEBUG:boto:Token: None 2022-09-12T19:33:58.353 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/ 2022-09-12T19:33:58.353 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY= 2022-09-12T19:33:58.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY=', 'Content-Length': '0'} 2022-09-12T19:33:58.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7d656ea5271e90b-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.358 DEBUG:boto:b'jdpiak-62obj11000falseobj12022-09-12T19:32:15.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:58.359 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:58.359 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:58.360 DEBUG:boto:Method: HEAD 2022-09-12T19:33:58.360 DEBUG:boto:Path: /jdpiak-62/obj1 2022-09-12T19:33:58.361 DEBUG:boto:Data: 2022-09-12T19:33:58.361 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.362 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.362 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.363 DEBUG:boto:Params: {} 2022-09-12T19:33:58.363 DEBUG:boto:Token: None 2022-09-12T19:33:58.364 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj1 2022-09-12T19:33:58.364 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8WW8HlBlSbFLwVZnxLtKROQEW2w= 2022-09-12T19:33:58.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8WW8HlBlSbFLwVZnxLtKROQEW2w=', 'Content-Length': '0'} 2022-09-12T19:33:58.367 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa4910463c769abe-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.368 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:58.368 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:58.369 DEBUG:boto:Method: HEAD 2022-09-12T19:33:58.369 DEBUG:boto:Path: /jdpiak-62/obj1 2022-09-12T19:33:58.370 DEBUG:boto:Data: 2022-09-12T19:33:58.370 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.371 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.371 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.372 DEBUG:boto:Params: {} 2022-09-12T19:33:58.372 DEBUG:boto:Token: None 2022-09-12T19:33:58.373 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj1 2022-09-12T19:33:58.374 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8WW8HlBlSbFLwVZnxLtKROQEW2w= 2022-09-12T19:33:58.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8WW8HlBlSbFLwVZnxLtKROQEW2w=', 'Content-Length': '0'} 2022-09-12T19:33:58.376 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000528a082dfe17428e-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-12T19:33:58.378 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:58.378 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:58.379 DEBUG:boto:Method: GET 2022-09-12T19:33:58.379 DEBUG:boto:Path: /jdpiak-62/obj1 2022-09-12T19:33:58.380 DEBUG:boto:Data: 2022-09-12T19:33:58.380 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.381 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.381 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.382 DEBUG:boto:Params: {} 2022-09-12T19:33:58.383 DEBUG:boto:Token: None 2022-09-12T19:33:58.383 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj1 2022-09-12T19:33:58.384 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:utfxEaI7jJZyzUCnN7TQNdFjo+k= 2022-09-12T19:33:58.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:utfxEaI7jJZyzUCnN7TQNdFjo+k=', 'Content-Length': '0'} 2022-09-12T19:33:58.387 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000056909f8707719169-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.387 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:58.388 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:58.389 DEBUG:boto:Method: GET 2022-09-12T19:33:58.389 DEBUG:boto:Path: /jdpiak-62/obj1 2022-09-12T19:33:58.390 DEBUG:boto:Data: 2022-09-12T19:33:58.390 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.391 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.391 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.392 DEBUG:boto:Params: {} 2022-09-12T19:33:58.392 DEBUG:boto:Token: None 2022-09-12T19:33:58.393 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj1 2022-09-12T19:33:58.394 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:utfxEaI7jJZyzUCnN7TQNdFjo+k= 2022-09-12T19:33:58.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:utfxEaI7jJZyzUCnN7TQNdFjo+k=', 'Content-Length': '0'} 2022-09-12T19:33:58.396 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ed6d67359d9865b-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.397 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:58.398 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:58.398 DEBUG:boto:path=/jdpiak-62/?prefix=obj1 2022-09-12T19:33:58.399 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj1 2022-09-12T19:33:58.399 DEBUG:boto:Method: GET 2022-09-12T19:33:58.400 DEBUG:boto:Path: /jdpiak-62/?prefix=obj1 2022-09-12T19:33:58.400 DEBUG:boto:Data: 2022-09-12T19:33:58.401 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.401 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.402 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.402 DEBUG:boto:Params: {} 2022-09-12T19:33:58.403 DEBUG:boto:Token: None 2022-09-12T19:33:58.404 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/ 2022-09-12T19:33:58.405 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY= 2022-09-12T19:33:58.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY=', 'Content-Length': '0'} 2022-09-12T19:33:58.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006eb261f8c7b476c4-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.450 DEBUG:boto:b'jdpiak-62obj11000falseobj12022-09-12T19:32:15.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:58.451 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:58.452 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:58.452 DEBUG:boto:path=/jdpiak-62/?prefix=obj1 2022-09-12T19:33:58.453 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj1 2022-09-12T19:33:58.453 DEBUG:boto:Method: GET 2022-09-12T19:33:58.454 DEBUG:boto:Path: /jdpiak-62/?prefix=obj1 2022-09-12T19:33:58.454 DEBUG:boto:Data: 2022-09-12T19:33:58.455 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.455 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.456 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.456 DEBUG:boto:Params: {} 2022-09-12T19:33:58.457 DEBUG:boto:Token: None 2022-09-12T19:33:58.457 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/ 2022-09-12T19:33:58.458 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY= 2022-09-12T19:33:58.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY=', 'Content-Length': '0'} 2022-09-12T19:33:58.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076925924a9ed7590-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.463 DEBUG:boto:b'jdpiak-62obj11000falseobj12022-09-12T19:32:15.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:58.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-12T19:33:58.464 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:58.465 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:58.465 DEBUG:boto:path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:58.466 DEBUG:boto:auth_path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:58.466 DEBUG:boto:Method: GET 2022-09-12T19:33:58.467 DEBUG:boto:Path: /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:58.467 DEBUG:boto:Data: 2022-09-12T19:33:58.468 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.468 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.469 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.469 DEBUG:boto:Params: {} 2022-09-12T19:33:58.470 DEBUG:boto:Token: None 2022-09-12T19:33:58.471 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:58.471 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D/SevhxkicQrhm9+I7jjGnLZbCo= 2022-09-12T19:33:58.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D/SevhxkicQrhm9+I7jjGnLZbCo=', 'Content-Length': '0'} 2022-09-12T19:33:58.474 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df907bda8f28541d-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.475 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:58.476 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:58.476 DEBUG:boto:path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:58.477 DEBUG:boto:auth_path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:58.477 DEBUG:boto:Method: GET 2022-09-12T19:33:58.478 DEBUG:boto:Path: /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:58.478 DEBUG:boto:Data: 2022-09-12T19:33:58.479 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.480 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.480 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.481 DEBUG:boto:Params: {} 2022-09-12T19:33:58.482 DEBUG:boto:Token: None 2022-09-12T19:33:58.482 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:58.483 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D/SevhxkicQrhm9+I7jjGnLZbCo= 2022-09-12T19:33:58.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D/SevhxkicQrhm9+I7jjGnLZbCo=', 'Content-Length': '0'} 2022-09-12T19:33:58.486 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e40833da1d0c5bb-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.534 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:58.536 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:58.536 DEBUG:boto:path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:58.537 DEBUG:boto:auth_path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:58.538 DEBUG:boto:Method: HEAD 2022-09-12T19:33:58.539 DEBUG:boto:Path: /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:58.539 DEBUG:boto:Data: 2022-09-12T19:33:58.540 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.540 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.541 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.541 DEBUG:boto:Params: {} 2022-09-12T19:33:58.542 DEBUG:boto:Token: None 2022-09-12T19:33:58.542 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:58.543 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8FhH7hsj0JZMRgtxN4KxG6kTgGY= 2022-09-12T19:33:58.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8FhH7hsj0JZMRgtxN4KxG6kTgGY=', 'Content-Length': '0'} 2022-09-12T19:33:58.546 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000988dbbc3200fb909-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.547 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:58.548 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:58.549 DEBUG:boto:path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:58.550 DEBUG:boto:auth_path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:58.551 DEBUG:boto:Method: HEAD 2022-09-12T19:33:58.551 DEBUG:boto:Path: /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:58.552 DEBUG:boto:Data: 2022-09-12T19:33:58.553 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.553 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.554 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.555 DEBUG:boto:Params: {} 2022-09-12T19:33:58.556 DEBUG:boto:Token: None 2022-09-12T19:33:58.556 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:58.557 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8FhH7hsj0JZMRgtxN4KxG6kTgGY= 2022-09-12T19:33:58.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8FhH7hsj0JZMRgtxN4KxG6kTgGY=', 'Content-Length': '0'} 2022-09-12T19:33:58.561 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000115fe7e699644f99-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-12T19:33:58.562 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:58.563 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:58.563 DEBUG:boto:path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:58.564 DEBUG:boto:auth_path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:58.564 DEBUG:boto:Method: GET 2022-09-12T19:33:58.565 DEBUG:boto:Path: /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:58.566 DEBUG:boto:Data: 2022-09-12T19:33:58.567 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.568 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.569 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.570 DEBUG:boto:Params: {} 2022-09-12T19:33:58.571 DEBUG:boto:Token: None 2022-09-12T19:33:58.572 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:58.573 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D/SevhxkicQrhm9+I7jjGnLZbCo= 2022-09-12T19:33:58.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D/SevhxkicQrhm9+I7jjGnLZbCo=', 'Content-Length': '0'} 2022-09-12T19:33:58.577 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003edc5cdb6f37edb9-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.578 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:58.578 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:58.579 DEBUG:boto:path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:58.580 DEBUG:boto:auth_path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:58.580 DEBUG:boto:Method: GET 2022-09-12T19:33:58.581 DEBUG:boto:Path: /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:58.581 DEBUG:boto:Data: 2022-09-12T19:33:58.582 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.583 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.583 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.584 DEBUG:boto:Params: {} 2022-09-12T19:33:58.585 DEBUG:boto:Token: None 2022-09-12T19:33:58.585 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:58.586 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D/SevhxkicQrhm9+I7jjGnLZbCo= 2022-09-12T19:33:58.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D/SevhxkicQrhm9+I7jjGnLZbCo=', 'Content-Length': '0'} 2022-09-12T19:33:58.589 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c2fcd557be56ff5-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.590 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:58.591 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:58.591 DEBUG:boto:path=/jdpiak-62/?prefix=obj2 2022-09-12T19:33:58.592 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj2 2022-09-12T19:33:58.592 DEBUG:boto:Method: GET 2022-09-12T19:33:58.593 DEBUG:boto:Path: /jdpiak-62/?prefix=obj2 2022-09-12T19:33:58.593 DEBUG:boto:Data: 2022-09-12T19:33:58.594 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.595 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.595 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.596 DEBUG:boto:Params: {} 2022-09-12T19:33:58.596 DEBUG:boto:Token: None 2022-09-12T19:33:58.597 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/ 2022-09-12T19:33:58.597 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY= 2022-09-12T19:33:58.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY=', 'Content-Length': '0'} 2022-09-12T19:33:58.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5783c3b3b974727-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.602 DEBUG:boto:b'jdpiak-62obj21000falseobj22022-09-12T19:32:15.070Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:58.603 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:58.603 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:58.604 DEBUG:boto:path=/jdpiak-62/?prefix=obj2 2022-09-12T19:33:58.604 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj2 2022-09-12T19:33:58.605 DEBUG:boto:Method: GET 2022-09-12T19:33:58.605 DEBUG:boto:Path: /jdpiak-62/?prefix=obj2 2022-09-12T19:33:58.606 DEBUG:boto:Data: 2022-09-12T19:33:58.606 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.607 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.607 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.608 DEBUG:boto:Params: {} 2022-09-12T19:33:58.608 DEBUG:boto:Token: None 2022-09-12T19:33:58.609 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/ 2022-09-12T19:33:58.610 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY= 2022-09-12T19:33:58.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY=', 'Content-Length': '0'} 2022-09-12T19:33:58.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004695ad9a7f1b8d28-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.614 DEBUG:boto:b'jdpiak-62obj21000falseobj22022-09-12T19:32:15.070Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:58.615 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:58.615 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:58.616 DEBUG:boto:Method: HEAD 2022-09-12T19:33:58.617 DEBUG:boto:Path: /jdpiak-62/obj2 2022-09-12T19:33:58.617 DEBUG:boto:Data: 2022-09-12T19:33:58.618 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.618 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.619 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.619 DEBUG:boto:Params: {} 2022-09-12T19:33:58.620 DEBUG:boto:Token: None 2022-09-12T19:33:58.621 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj2 2022-09-12T19:33:58.622 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bjiEumRtA8JRzgZNU/ggSaRfAcg= 2022-09-12T19:33:58.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bjiEumRtA8JRzgZNU/ggSaRfAcg=', 'Content-Length': '0'} 2022-09-12T19:33:58.626 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000078d2ce9e4cac997e-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.627 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:58.627 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:58.628 DEBUG:boto:Method: HEAD 2022-09-12T19:33:58.629 DEBUG:boto:Path: /jdpiak-62/obj2 2022-09-12T19:33:58.630 DEBUG:boto:Data: 2022-09-12T19:33:58.630 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.631 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.631 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.632 DEBUG:boto:Params: {} 2022-09-12T19:33:58.632 DEBUG:boto:Token: None 2022-09-12T19:33:58.633 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj2 2022-09-12T19:33:58.634 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bjiEumRtA8JRzgZNU/ggSaRfAcg= 2022-09-12T19:33:58.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bjiEumRtA8JRzgZNU/ggSaRfAcg=', 'Content-Length': '0'} 2022-09-12T19:33:58.636 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ce9f96a36450e99-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-12T19:33:58.638 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:58.638 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:58.639 DEBUG:boto:Method: GET 2022-09-12T19:33:58.639 DEBUG:boto:Path: /jdpiak-62/obj2 2022-09-12T19:33:58.640 DEBUG:boto:Data: 2022-09-12T19:33:58.640 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.641 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.642 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.642 DEBUG:boto:Params: {} 2022-09-12T19:33:58.643 DEBUG:boto:Token: None 2022-09-12T19:33:58.643 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj2 2022-09-12T19:33:58.644 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:icXEv37ERRsSSepji3F1P0qr8h4= 2022-09-12T19:33:58.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:icXEv37ERRsSSepji3F1P0qr8h4=', 'Content-Length': '0'} 2022-09-12T19:33:58.647 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f915d7f02dfb493-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.647 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:58.648 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:58.648 DEBUG:boto:Method: GET 2022-09-12T19:33:58.649 DEBUG:boto:Path: /jdpiak-62/obj2 2022-09-12T19:33:58.650 DEBUG:boto:Data: 2022-09-12T19:33:58.650 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.651 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.651 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.652 DEBUG:boto:Params: {} 2022-09-12T19:33:58.652 DEBUG:boto:Token: None 2022-09-12T19:33:58.653 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj2 2022-09-12T19:33:58.654 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:icXEv37ERRsSSepji3F1P0qr8h4= 2022-09-12T19:33:58.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:icXEv37ERRsSSepji3F1P0qr8h4=', 'Content-Length': '0'} 2022-09-12T19:33:58.657 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f4d570f1fea87ed-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.658 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:58.658 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:58.659 DEBUG:boto:path=/jdpiak-62/?prefix=obj2 2022-09-12T19:33:58.659 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj2 2022-09-12T19:33:58.660 DEBUG:boto:Method: GET 2022-09-12T19:33:58.660 DEBUG:boto:Path: /jdpiak-62/?prefix=obj2 2022-09-12T19:33:58.661 DEBUG:boto:Data: 2022-09-12T19:33:58.661 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.662 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.663 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.663 DEBUG:boto:Params: {} 2022-09-12T19:33:58.664 DEBUG:boto:Token: None 2022-09-12T19:33:58.664 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/ 2022-09-12T19:33:58.665 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY= 2022-09-12T19:33:58.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY=', 'Content-Length': '0'} 2022-09-12T19:33:58.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b22b2f3399ac2c8-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.669 DEBUG:boto:b'jdpiak-62obj21000falseobj22022-09-12T19:32:15.070Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:58.670 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:58.670 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:58.671 DEBUG:boto:path=/jdpiak-62/?prefix=obj2 2022-09-12T19:33:58.672 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj2 2022-09-12T19:33:58.672 DEBUG:boto:Method: GET 2022-09-12T19:33:58.673 DEBUG:boto:Path: /jdpiak-62/?prefix=obj2 2022-09-12T19:33:58.673 DEBUG:boto:Data: 2022-09-12T19:33:58.674 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.674 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.675 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.675 DEBUG:boto:Params: {} 2022-09-12T19:33:58.676 DEBUG:boto:Token: None 2022-09-12T19:33:58.676 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/ 2022-09-12T19:33:58.677 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY= 2022-09-12T19:33:58.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY=', 'Content-Length': '0'} 2022-09-12T19:33:58.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000191b2cf9a9df49ab-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.682 DEBUG:boto:b'jdpiak-62obj21000falseobj22022-09-12T19:32:15.070Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:58.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-12T19:33:58.683 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:58.684 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:58.684 DEBUG:boto:path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:58.685 DEBUG:boto:auth_path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:58.685 DEBUG:boto:Method: GET 2022-09-12T19:33:58.686 DEBUG:boto:Path: /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:58.686 DEBUG:boto:Data: 2022-09-12T19:33:58.687 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.688 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.688 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.689 DEBUG:boto:Params: {} 2022-09-12T19:33:58.689 DEBUG:boto:Token: None 2022-09-12T19:33:58.690 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:58.691 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Kq1oB7QjjFfnNzJthRVFzKW1Yko= 2022-09-12T19:33:58.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Kq1oB7QjjFfnNzJthRVFzKW1Yko=', 'Content-Length': '0'} 2022-09-12T19:33:58.694 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa900bdf03b9479a-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.694 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:58.695 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:58.695 DEBUG:boto:path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:58.696 DEBUG:boto:auth_path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:58.696 DEBUG:boto:Method: GET 2022-09-12T19:33:58.697 DEBUG:boto:Path: /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:58.698 DEBUG:boto:Data: 2022-09-12T19:33:58.698 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.699 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.699 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.700 DEBUG:boto:Params: {} 2022-09-12T19:33:58.700 DEBUG:boto:Token: None 2022-09-12T19:33:58.701 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:58.702 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Kq1oB7QjjFfnNzJthRVFzKW1Yko= 2022-09-12T19:33:58.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Kq1oB7QjjFfnNzJthRVFzKW1Yko=', 'Content-Length': '0'} 2022-09-12T19:33:58.706 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fee19abe9f6a4fca-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.706 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:58.707 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:58.707 DEBUG:boto:path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:58.708 DEBUG:boto:auth_path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:58.708 DEBUG:boto:Method: HEAD 2022-09-12T19:33:58.709 DEBUG:boto:Path: /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:58.709 DEBUG:boto:Data: 2022-09-12T19:33:58.710 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.710 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.711 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.711 DEBUG:boto:Params: {} 2022-09-12T19:33:58.712 DEBUG:boto:Token: None 2022-09-12T19:33:58.713 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:58.713 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ovk0Myn8IrFOf7SKHJ/xvrAjvTY= 2022-09-12T19:33:58.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ovk0Myn8IrFOf7SKHJ/xvrAjvTY=', 'Content-Length': '0'} 2022-09-12T19:33:58.716 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e3b0aa27384a851-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.717 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:58.717 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:58.718 DEBUG:boto:path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:58.718 DEBUG:boto:auth_path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:58.719 DEBUG:boto:Method: HEAD 2022-09-12T19:33:58.719 DEBUG:boto:Path: /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:58.720 DEBUG:boto:Data: 2022-09-12T19:33:58.720 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.721 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.721 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.722 DEBUG:boto:Params: {} 2022-09-12T19:33:58.722 DEBUG:boto:Token: None 2022-09-12T19:33:58.723 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:58.723 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ovk0Myn8IrFOf7SKHJ/xvrAjvTY= 2022-09-12T19:33:58.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ovk0Myn8IrFOf7SKHJ/xvrAjvTY=', 'Content-Length': '0'} 2022-09-12T19:33:58.726 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000048373bada3825952-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-12T19:33:58.728 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:58.728 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:58.729 DEBUG:boto:path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:58.729 DEBUG:boto:auth_path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:58.730 DEBUG:boto:Method: GET 2022-09-12T19:33:58.731 DEBUG:boto:Path: /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:58.731 DEBUG:boto:Data: 2022-09-12T19:33:58.732 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.732 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.733 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.733 DEBUG:boto:Params: {} 2022-09-12T19:33:58.734 DEBUG:boto:Token: None 2022-09-12T19:33:58.735 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:58.735 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Kq1oB7QjjFfnNzJthRVFzKW1Yko= 2022-09-12T19:33:58.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Kq1oB7QjjFfnNzJthRVFzKW1Yko=', 'Content-Length': '0'} 2022-09-12T19:33:58.738 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c3d7d9a410ccd30-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.739 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:58.740 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:58.740 DEBUG:boto:path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:58.741 DEBUG:boto:auth_path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:58.741 DEBUG:boto:Method: GET 2022-09-12T19:33:58.742 DEBUG:boto:Path: /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:58.742 DEBUG:boto:Data: 2022-09-12T19:33:58.743 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.743 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.744 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.745 DEBUG:boto:Params: {} 2022-09-12T19:33:58.745 DEBUG:boto:Token: None 2022-09-12T19:33:58.746 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:58.747 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Kq1oB7QjjFfnNzJthRVFzKW1Yko= 2022-09-12T19:33:58.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Kq1oB7QjjFfnNzJthRVFzKW1Yko=', 'Content-Length': '0'} 2022-09-12T19:33:58.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ffd4e42e6451997e-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.750 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:58.751 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:58.751 DEBUG:boto:path=/jdpiak-62/?prefix=obj3 2022-09-12T19:33:58.752 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj3 2022-09-12T19:33:58.752 DEBUG:boto:Method: GET 2022-09-12T19:33:58.753 DEBUG:boto:Path: /jdpiak-62/?prefix=obj3 2022-09-12T19:33:58.754 DEBUG:boto:Data: 2022-09-12T19:33:58.755 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.755 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.756 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.757 DEBUG:boto:Params: {} 2022-09-12T19:33:58.757 DEBUG:boto:Token: None 2022-09-12T19:33:58.758 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/ 2022-09-12T19:33:58.758 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY= 2022-09-12T19:33:58.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY=', 'Content-Length': '0'} 2022-09-12T19:33:58.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008daffe0a83e12b69-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.806 DEBUG:boto:b'jdpiak-62obj31000falseobj32022-09-12T19:32:15.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:58.807 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:58.808 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:58.808 DEBUG:boto:path=/jdpiak-62/?prefix=obj3 2022-09-12T19:33:58.809 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj3 2022-09-12T19:33:58.809 DEBUG:boto:Method: GET 2022-09-12T19:33:58.810 DEBUG:boto:Path: /jdpiak-62/?prefix=obj3 2022-09-12T19:33:58.810 DEBUG:boto:Data: 2022-09-12T19:33:58.811 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.811 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.812 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.812 DEBUG:boto:Params: {} 2022-09-12T19:33:58.813 DEBUG:boto:Token: None 2022-09-12T19:33:58.814 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/ 2022-09-12T19:33:58.814 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY= 2022-09-12T19:33:58.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY=', 'Content-Length': '0'} 2022-09-12T19:33:58.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044befcd29a961975-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.818 DEBUG:boto:b'jdpiak-62obj31000falseobj32022-09-12T19:32:15.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:58.819 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:58.820 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:58.820 DEBUG:boto:Method: HEAD 2022-09-12T19:33:58.821 DEBUG:boto:Path: /jdpiak-62/obj3 2022-09-12T19:33:58.821 DEBUG:boto:Data: 2022-09-12T19:33:58.822 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.822 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.823 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.823 DEBUG:boto:Params: {} 2022-09-12T19:33:58.824 DEBUG:boto:Token: None 2022-09-12T19:33:58.824 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj3 2022-09-12T19:33:58.825 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2m8y8iY40hJdBqR+otjJgYdAsNU= 2022-09-12T19:33:58.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2m8y8iY40hJdBqR+otjJgYdAsNU=', 'Content-Length': '0'} 2022-09-12T19:33:58.827 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd0f6533c7405045-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.828 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:58.829 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:58.829 DEBUG:boto:Method: HEAD 2022-09-12T19:33:58.830 DEBUG:boto:Path: /jdpiak-62/obj3 2022-09-12T19:33:58.830 DEBUG:boto:Data: 2022-09-12T19:33:58.831 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.831 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.832 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.832 DEBUG:boto:Params: {} 2022-09-12T19:33:58.833 DEBUG:boto:Token: None 2022-09-12T19:33:58.833 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj3 2022-09-12T19:33:58.834 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2m8y8iY40hJdBqR+otjJgYdAsNU= 2022-09-12T19:33:58.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2m8y8iY40hJdBqR+otjJgYdAsNU=', 'Content-Length': '0'} 2022-09-12T19:33:58.836 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7b258215caa6e64-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-12T19:33:58.838 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:58.838 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:58.839 DEBUG:boto:Method: GET 2022-09-12T19:33:58.839 DEBUG:boto:Path: /jdpiak-62/obj3 2022-09-12T19:33:58.840 DEBUG:boto:Data: 2022-09-12T19:33:58.840 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.841 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.841 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.842 DEBUG:boto:Params: {} 2022-09-12T19:33:58.842 DEBUG:boto:Token: None 2022-09-12T19:33:58.843 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj3 2022-09-12T19:33:58.844 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YI9TrCxmJk3TZo1JznJKPEhuNE4= 2022-09-12T19:33:58.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YI9TrCxmJk3TZo1JznJKPEhuNE4=', 'Content-Length': '0'} 2022-09-12T19:33:58.847 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004af3cda1ecb4aa4e-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.847 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:58.848 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:58.848 DEBUG:boto:Method: GET 2022-09-12T19:33:58.849 DEBUG:boto:Path: /jdpiak-62/obj3 2022-09-12T19:33:58.849 DEBUG:boto:Data: 2022-09-12T19:33:58.850 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.851 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.851 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.852 DEBUG:boto:Params: {} 2022-09-12T19:33:58.852 DEBUG:boto:Token: None 2022-09-12T19:33:58.853 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj3 2022-09-12T19:33:58.853 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YI9TrCxmJk3TZo1JznJKPEhuNE4= 2022-09-12T19:33:58.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YI9TrCxmJk3TZo1JznJKPEhuNE4=', 'Content-Length': '0'} 2022-09-12T19:33:58.856 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000597d4029f60b6c87-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.857 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:58.858 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:58.858 DEBUG:boto:path=/jdpiak-62/?prefix=obj3 2022-09-12T19:33:58.859 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj3 2022-09-12T19:33:58.859 DEBUG:boto:Method: GET 2022-09-12T19:33:58.860 DEBUG:boto:Path: /jdpiak-62/?prefix=obj3 2022-09-12T19:33:58.861 DEBUG:boto:Data: 2022-09-12T19:33:58.861 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.862 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.862 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.863 DEBUG:boto:Params: {} 2022-09-12T19:33:58.864 DEBUG:boto:Token: None 2022-09-12T19:33:58.864 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/ 2022-09-12T19:33:58.865 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY= 2022-09-12T19:33:58.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY=', 'Content-Length': '0'} 2022-09-12T19:33:58.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fff66365123ce80-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.870 DEBUG:boto:b'jdpiak-62obj31000falseobj32022-09-12T19:32:15.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:58.871 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:58.872 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:58.872 DEBUG:boto:path=/jdpiak-62/?prefix=obj3 2022-09-12T19:33:58.873 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj3 2022-09-12T19:33:58.873 DEBUG:boto:Method: GET 2022-09-12T19:33:58.874 DEBUG:boto:Path: /jdpiak-62/?prefix=obj3 2022-09-12T19:33:58.875 DEBUG:boto:Data: 2022-09-12T19:33:58.875 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.876 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.877 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.877 DEBUG:boto:Params: {} 2022-09-12T19:33:58.878 DEBUG:boto:Token: None 2022-09-12T19:33:58.879 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/ 2022-09-12T19:33:58.880 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY= 2022-09-12T19:33:58.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rdHzry3NF5iNPeo6qF9heZNcXhY=', 'Content-Length': '0'} 2022-09-12T19:33:58.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c48b278e059edd3-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.884 DEBUG:boto:b'jdpiak-62obj31000falseobj32022-09-12T19:32:15.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:58.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-12T19:33:58.885 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:33:58.886 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:33:58.887 DEBUG:boto:path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.887 DEBUG:boto:auth_path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.888 DEBUG:boto:Method: GET 2022-09-12T19:33:58.889 DEBUG:boto:Path: /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.889 DEBUG:boto:Data: 2022-09-12T19:33:58.890 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.891 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.891 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.892 DEBUG:boto:Params: {} 2022-09-12T19:33:58.892 DEBUG:boto:Token: None 2022-09-12T19:33:58.893 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.894 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sX2S6gZBYu4h1gFJkN6pUIo+LyY= 2022-09-12T19:33:58.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sX2S6gZBYu4h1gFJkN6pUIo+LyY=', 'Content-Length': '0'} 2022-09-12T19:33:58.897 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047ead745f60c5c99-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.898 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:33:58.898 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:33:58.899 DEBUG:boto:path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.899 DEBUG:boto:auth_path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.900 DEBUG:boto:Method: GET 2022-09-12T19:33:58.900 DEBUG:boto:Path: /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.901 DEBUG:boto:Data: 2022-09-12T19:33:58.901 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.902 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.903 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.903 DEBUG:boto:Params: {} 2022-09-12T19:33:58.904 DEBUG:boto:Token: None 2022-09-12T19:33:58.904 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.905 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sX2S6gZBYu4h1gFJkN6pUIo+LyY= 2022-09-12T19:33:58.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sX2S6gZBYu4h1gFJkN6pUIo+LyY=', 'Content-Length': '0'} 2022-09-12T19:33:58.909 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd00280fb68e93ae-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.950 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:33:58.951 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:33:58.951 DEBUG:boto:path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.952 DEBUG:boto:auth_path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.953 DEBUG:boto:Method: HEAD 2022-09-12T19:33:58.953 DEBUG:boto:Path: /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.954 DEBUG:boto:Data: 2022-09-12T19:33:58.954 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.955 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.955 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.956 DEBUG:boto:Params: {} 2022-09-12T19:33:58.956 DEBUG:boto:Token: None 2022-09-12T19:33:58.957 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.958 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Mpfc/qVN1OTCWl7rqlGJCiaNxVg= 2022-09-12T19:33:58.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Mpfc/qVN1OTCWl7rqlGJCiaNxVg=', 'Content-Length': '0'} 2022-09-12T19:33:58.960 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d78e88243ee0b0b-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.961 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:33:58.962 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:33:58.963 DEBUG:boto:path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.963 DEBUG:boto:auth_path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.964 DEBUG:boto:Method: HEAD 2022-09-12T19:33:58.964 DEBUG:boto:Path: /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.965 DEBUG:boto:Data: 2022-09-12T19:33:58.965 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.966 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.966 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.967 DEBUG:boto:Params: {} 2022-09-12T19:33:58.967 DEBUG:boto:Token: None 2022-09-12T19:33:58.968 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.968 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Mpfc/qVN1OTCWl7rqlGJCiaNxVg= 2022-09-12T19:33:58.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Mpfc/qVN1OTCWl7rqlGJCiaNxVg=', 'Content-Length': '0'} 2022-09-12T19:33:58.971 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006dc1aa99d449d757-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-12T19:33:58.973 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:33:58.974 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:33:58.974 DEBUG:boto:path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.975 DEBUG:boto:auth_path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.976 DEBUG:boto:Method: GET 2022-09-12T19:33:58.976 DEBUG:boto:Path: /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.977 DEBUG:boto:Data: 2022-09-12T19:33:58.977 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.978 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:58.979 DEBUG:boto:Port: 8001 2022-09-12T19:33:58.979 DEBUG:boto:Params: {} 2022-09-12T19:33:58.980 DEBUG:boto:Token: None 2022-09-12T19:33:58.981 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.981 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sX2S6gZBYu4h1gFJkN6pUIo+LyY= 2022-09-12T19:33:58.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sX2S6gZBYu4h1gFJkN6pUIo+LyY=', 'Content-Length': '0'} 2022-09-12T19:33:58.985 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000309331c45d04a56c-00631f89a6-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:58.986 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:33:58.986 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:33:58.987 DEBUG:boto:path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.988 DEBUG:boto:auth_path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.988 DEBUG:boto:Method: GET 2022-09-12T19:33:58.989 DEBUG:boto:Path: /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.989 DEBUG:boto:Data: 2022-09-12T19:33:58.990 DEBUG:boto:Headers: {} 2022-09-12T19:33:58.990 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:58.991 DEBUG:boto:Port: 8000 2022-09-12T19:33:58.992 DEBUG:boto:Params: {} 2022-09-12T19:33:58.994 DEBUG:boto:Token: None 2022-09-12T19:33:58.994 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:58 GMT /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:58.995 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sX2S6gZBYu4h1gFJkN6pUIo+LyY= 2022-09-12T19:33:58.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sX2S6gZBYu4h1gFJkN6pUIo+LyY=', 'Content-Length': '0'} 2022-09-12T19:33:58.998 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007becf46e93953c72-00631f89a6-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.000 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:59.000 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:59.001 DEBUG:boto:path=/jdpiak-62/?prefix=obj4 2022-09-12T19:33:59.002 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj4 2022-09-12T19:33:59.003 DEBUG:boto:Method: GET 2022-09-12T19:33:59.004 DEBUG:boto:Path: /jdpiak-62/?prefix=obj4 2022-09-12T19:33:59.005 DEBUG:boto:Data: 2022-09-12T19:33:59.005 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.006 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.006 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.007 DEBUG:boto:Params: {} 2022-09-12T19:33:59.008 DEBUG:boto:Token: None 2022-09-12T19:33:59.008 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/ 2022-09-12T19:33:59.009 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo= 2022-09-12T19:33:59.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo=', 'Content-Length': '0'} 2022-09-12T19:33:59.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007586632cbd9a72cd-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.013 DEBUG:boto:b'jdpiak-62obj41000falseobj42022-09-12T19:32:15.218Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:59.015 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:59.015 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:59.016 DEBUG:boto:path=/jdpiak-62/?prefix=obj4 2022-09-12T19:33:59.017 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj4 2022-09-12T19:33:59.017 DEBUG:boto:Method: GET 2022-09-12T19:33:59.018 DEBUG:boto:Path: /jdpiak-62/?prefix=obj4 2022-09-12T19:33:59.018 DEBUG:boto:Data: 2022-09-12T19:33:59.019 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.019 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:59.020 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.021 DEBUG:boto:Params: {} 2022-09-12T19:33:59.021 DEBUG:boto:Token: None 2022-09-12T19:33:59.023 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/ 2022-09-12T19:33:59.023 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo= 2022-09-12T19:33:59.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo=', 'Content-Length': '0'} 2022-09-12T19:33:59.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000726da944b152b8ea-00631f89a7-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.029 DEBUG:boto:b'jdpiak-62obj41000falseobj42022-09-12T19:32:15.218Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:59.030 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:33:59.031 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:33:59.032 DEBUG:boto:Method: HEAD 2022-09-12T19:33:59.033 DEBUG:boto:Path: /jdpiak-62/obj4 2022-09-12T19:33:59.034 DEBUG:boto:Data: 2022-09-12T19:33:59.035 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.036 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.037 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.037 DEBUG:boto:Params: {} 2022-09-12T19:33:59.038 DEBUG:boto:Token: None 2022-09-12T19:33:59.039 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj4 2022-09-12T19:33:59.040 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZhgEZEk+6BmGX3YObORiZW818r8= 2022-09-12T19:33:59.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZhgEZEk+6BmGX3YObORiZW818r8=', 'Content-Length': '0'} 2022-09-12T19:33:59.043 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000510139500aafdacf-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.044 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:33:59.045 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:33:59.045 DEBUG:boto:Method: HEAD 2022-09-12T19:33:59.046 DEBUG:boto:Path: /jdpiak-62/obj4 2022-09-12T19:33:59.046 DEBUG:boto:Data: 2022-09-12T19:33:59.047 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.047 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:59.048 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.049 DEBUG:boto:Params: {} 2022-09-12T19:33:59.050 DEBUG:boto:Token: None 2022-09-12T19:33:59.050 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj4 2022-09-12T19:33:59.051 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZhgEZEk+6BmGX3YObORiZW818r8= 2022-09-12T19:33:59.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZhgEZEk+6BmGX3YObORiZW818r8=', 'Content-Length': '0'} 2022-09-12T19:33:59.054 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024bfc570c1784d02-00631f89a7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-12T19:33:59.055 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:33:59.055 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:33:59.056 DEBUG:boto:Method: GET 2022-09-12T19:33:59.057 DEBUG:boto:Path: /jdpiak-62/obj4 2022-09-12T19:33:59.057 DEBUG:boto:Data: 2022-09-12T19:33:59.058 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.059 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.060 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.061 DEBUG:boto:Params: {} 2022-09-12T19:33:59.062 DEBUG:boto:Token: None 2022-09-12T19:33:59.063 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj4 2022-09-12T19:33:59.063 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wSgf1vcqcDPYqJXU16alPeL/nnI= 2022-09-12T19:33:59.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wSgf1vcqcDPYqJXU16alPeL/nnI=', 'Content-Length': '0'} 2022-09-12T19:33:59.066 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006b34978ae6c2eda-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.106 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:33:59.107 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:33:59.107 DEBUG:boto:Method: GET 2022-09-12T19:33:59.108 DEBUG:boto:Path: /jdpiak-62/obj4 2022-09-12T19:33:59.109 DEBUG:boto:Data: 2022-09-12T19:33:59.109 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.110 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:59.110 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.111 DEBUG:boto:Params: {} 2022-09-12T19:33:59.112 DEBUG:boto:Token: None 2022-09-12T19:33:59.113 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj4 2022-09-12T19:33:59.114 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wSgf1vcqcDPYqJXU16alPeL/nnI= 2022-09-12T19:33:59.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wSgf1vcqcDPYqJXU16alPeL/nnI=', 'Content-Length': '0'} 2022-09-12T19:33:59.117 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006a0dfaabcf52f3b-00631f89a7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.118 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:59.118 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:59.119 DEBUG:boto:path=/jdpiak-62/?prefix=obj4 2022-09-12T19:33:59.119 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj4 2022-09-12T19:33:59.120 DEBUG:boto:Method: GET 2022-09-12T19:33:59.120 DEBUG:boto:Path: /jdpiak-62/?prefix=obj4 2022-09-12T19:33:59.121 DEBUG:boto:Data: 2022-09-12T19:33:59.122 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.123 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.123 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.124 DEBUG:boto:Params: {} 2022-09-12T19:33:59.124 DEBUG:boto:Token: None 2022-09-12T19:33:59.125 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/ 2022-09-12T19:33:59.126 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo= 2022-09-12T19:33:59.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo=', 'Content-Length': '0'} 2022-09-12T19:33:59.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001601684c14c76230-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.130 DEBUG:boto:b'jdpiak-62obj41000falseobj42022-09-12T19:32:15.218Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:59.131 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:59.131 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:59.132 DEBUG:boto:path=/jdpiak-62/?prefix=obj4 2022-09-12T19:33:59.133 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj4 2022-09-12T19:33:59.133 DEBUG:boto:Method: GET 2022-09-12T19:33:59.134 DEBUG:boto:Path: /jdpiak-62/?prefix=obj4 2022-09-12T19:33:59.134 DEBUG:boto:Data: 2022-09-12T19:33:59.135 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.135 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:33:59.136 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.136 DEBUG:boto:Params: {} 2022-09-12T19:33:59.137 DEBUG:boto:Token: None 2022-09-12T19:33:59.138 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/ 2022-09-12T19:33:59.138 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo= 2022-09-12T19:33:59.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo=', 'Content-Length': '0'} 2022-09-12T19:33:59.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d55b54fae8e66ce4-00631f89a7-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.143 DEBUG:boto:b'jdpiak-62obj41000falseobj42022-09-12T19:32:15.218Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:59.144 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-62 zones={test-zone3, test-zone1} 2022-09-12T19:33:59.144 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-62 zones={test-zone3, test-zone2} 2022-09-12T19:33:59.145 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:59.145 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:59.146 DEBUG:boto:Method: HEAD 2022-09-12T19:33:59.146 DEBUG:boto:Path: /jdpiak-62/ 2022-09-12T19:33:59.147 DEBUG:boto:Data: 2022-09-12T19:33:59.147 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.148 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.149 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.149 DEBUG:boto:Params: {} 2022-09-12T19:33:59.150 DEBUG:boto:Token: None 2022-09-12T19:33:59.151 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/ 2022-09-12T19:33:59.151 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dVmlNG++0mT8WdZdfQcwzBUw8+4= 2022-09-12T19:33:59.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dVmlNG++0mT8WdZdfQcwzBUw8+4=', 'Content-Length': '0'} 2022-09-12T19:33:59.154 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a7f8c69edbb3ea8-00631f89a7-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.155 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:59.156 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:59.156 DEBUG:boto:Method: HEAD 2022-09-12T19:33:59.157 DEBUG:boto:Path: /jdpiak-62/ 2022-09-12T19:33:59.158 DEBUG:boto:Data: 2022-09-12T19:33:59.158 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.159 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.159 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.160 DEBUG:boto:Params: {} 2022-09-12T19:33:59.160 DEBUG:boto:Token: None 2022-09-12T19:33:59.161 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/ 2022-09-12T19:33:59.161 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dVmlNG++0mT8WdZdfQcwzBUw8+4= 2022-09-12T19:33:59.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dVmlNG++0mT8WdZdfQcwzBUw8+4=', 'Content-Length': '0'} 2022-09-12T19:33:59.165 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001dc2af1b39e41dd-00631f89a7-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.166 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:33:59.166 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:59.167 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:59.167 DEBUG:boto:path=/jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:59.168 DEBUG:boto:auth_path=/jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:59.168 DEBUG:boto:Method: GET 2022-09-12T19:33:59.169 DEBUG:boto:Path: /jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:59.170 DEBUG:boto:Data: 2022-09-12T19:33:59.170 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.171 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.171 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.172 DEBUG:boto:Params: {} 2022-09-12T19:33:59.172 DEBUG:boto:Token: None 2022-09-12T19:33:59.173 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/?versions 2022-09-12T19:33:59.173 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GUwgyZp0D1CwSkjsvYLyvNBIcjY= 2022-09-12T19:33:59.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GUwgyZp0D1CwSkjsvYLyvNBIcjY=', 'Content-Length': '0'} 2022-09-12T19:33:59.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbbceeddb2dc180e-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.178 DEBUG:boto:b'jdpiak-62999falseobj1nulltrue2022-09-12T19:32:15.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-12T19:32:15.070Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-12T19:32:15.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-12T19:32:15.218Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:59.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-12T19:33:59.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-12T19:33:59.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-12T19:33:59.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-12T19:33:59.181 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:33:59.182 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:59.183 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:59.183 DEBUG:boto:path=/jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:59.184 DEBUG:boto:auth_path=/jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:59.185 DEBUG:boto:Method: GET 2022-09-12T19:33:59.186 DEBUG:boto:Path: /jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:59.186 DEBUG:boto:Data: 2022-09-12T19:33:59.187 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.187 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.188 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.188 DEBUG:boto:Params: {} 2022-09-12T19:33:59.189 DEBUG:boto:Token: None 2022-09-12T19:33:59.190 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/?versions 2022-09-12T19:33:59.190 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GUwgyZp0D1CwSkjsvYLyvNBIcjY= 2022-09-12T19:33:59.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GUwgyZp0D1CwSkjsvYLyvNBIcjY=', 'Content-Length': '0'} 2022-09-12T19:33:59.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ceb5aaa236d79d58-00631f89a7-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.195 DEBUG:boto:b'jdpiak-62999falseobj1nulltrue2022-09-12T19:32:15.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-12T19:32:15.070Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-12T19:32:15.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-12T19:32:15.218Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:59.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-12T19:33:59.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-12T19:33:59.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-12T19:33:59.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-12T19:33:59.198 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:59.199 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:59.200 DEBUG:boto:path=/jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:59.201 DEBUG:boto:auth_path=/jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:59.201 DEBUG:boto:Method: GET 2022-09-12T19:33:59.202 DEBUG:boto:Path: /jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:59.202 DEBUG:boto:Data: 2022-09-12T19:33:59.203 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.203 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.204 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.204 DEBUG:boto:Params: {} 2022-09-12T19:33:59.205 DEBUG:boto:Token: None 2022-09-12T19:33:59.206 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/?versions 2022-09-12T19:33:59.207 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GUwgyZp0D1CwSkjsvYLyvNBIcjY= 2022-09-12T19:33:59.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GUwgyZp0D1CwSkjsvYLyvNBIcjY=', 'Content-Length': '0'} 2022-09-12T19:33:59.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029c6d225d1029d7c-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.211 DEBUG:boto:b'jdpiak-62999falseobj1nulltrue2022-09-12T19:32:15.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-12T19:32:15.070Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-12T19:32:15.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-12T19:32:15.218Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:59.212 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:59.213 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:59.213 DEBUG:boto:path=/jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:59.214 DEBUG:boto:auth_path=/jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:59.215 DEBUG:boto:Method: GET 2022-09-12T19:33:59.216 DEBUG:boto:Path: /jdpiak-62/?versions&max-keys=999 2022-09-12T19:33:59.217 DEBUG:boto:Data: 2022-09-12T19:33:59.217 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.218 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.218 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.219 DEBUG:boto:Params: {} 2022-09-12T19:33:59.219 DEBUG:boto:Token: None 2022-09-12T19:33:59.220 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/?versions 2022-09-12T19:33:59.221 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GUwgyZp0D1CwSkjsvYLyvNBIcjY= 2022-09-12T19:33:59.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GUwgyZp0D1CwSkjsvYLyvNBIcjY=', 'Content-Length': '0'} 2022-09-12T19:33:59.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f696d60c7726142f-00631f89a7-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.225 DEBUG:boto:b'jdpiak-62999falseobj1nulltrue2022-09-12T19:32:15.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-12T19:32:15.070Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-12T19:32:15.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-12T19:32:15.218Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:59.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-12T19:33:59.226 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:59.227 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:59.228 DEBUG:boto:path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:59.229 DEBUG:boto:auth_path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:59.229 DEBUG:boto:Method: GET 2022-09-12T19:33:59.230 DEBUG:boto:Path: /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:59.230 DEBUG:boto:Data: 2022-09-12T19:33:59.231 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.231 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.232 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.232 DEBUG:boto:Params: {} 2022-09-12T19:33:59.233 DEBUG:boto:Token: None 2022-09-12T19:33:59.234 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:59.234 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iforMXIdsbdTZwD5Q/ExkSutxkY= 2022-09-12T19:33:59.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iforMXIdsbdTZwD5Q/ExkSutxkY=', 'Content-Length': '0'} 2022-09-12T19:33:59.237 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e190516b8d654514-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.278 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:59.279 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:59.280 DEBUG:boto:path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:59.280 DEBUG:boto:auth_path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:59.281 DEBUG:boto:Method: GET 2022-09-12T19:33:59.282 DEBUG:boto:Path: /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:59.282 DEBUG:boto:Data: 2022-09-12T19:33:59.283 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.283 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.284 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.284 DEBUG:boto:Params: {} 2022-09-12T19:33:59.285 DEBUG:boto:Token: None 2022-09-12T19:33:59.285 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:59.286 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iforMXIdsbdTZwD5Q/ExkSutxkY= 2022-09-12T19:33:59.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iforMXIdsbdTZwD5Q/ExkSutxkY=', 'Content-Length': '0'} 2022-09-12T19:33:59.289 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7417df85c681107-00631f89a7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.290 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:59.290 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:59.291 DEBUG:boto:path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:59.292 DEBUG:boto:auth_path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:59.292 DEBUG:boto:Method: HEAD 2022-09-12T19:33:59.293 DEBUG:boto:Path: /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:59.293 DEBUG:boto:Data: 2022-09-12T19:33:59.294 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.294 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.295 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.295 DEBUG:boto:Params: {} 2022-09-12T19:33:59.296 DEBUG:boto:Token: None 2022-09-12T19:33:59.296 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:59.297 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7kEmmgZ2m4bQYDkIvyXJv0uXvPg= 2022-09-12T19:33:59.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7kEmmgZ2m4bQYDkIvyXJv0uXvPg=', 'Content-Length': '0'} 2022-09-12T19:33:59.300 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c5a50724946ebdd-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.301 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:59.301 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:59.302 DEBUG:boto:path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:59.302 DEBUG:boto:auth_path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:59.303 DEBUG:boto:Method: HEAD 2022-09-12T19:33:59.303 DEBUG:boto:Path: /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:59.304 DEBUG:boto:Data: 2022-09-12T19:33:59.304 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.305 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.306 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.306 DEBUG:boto:Params: {} 2022-09-12T19:33:59.307 DEBUG:boto:Token: None 2022-09-12T19:33:59.307 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:59.308 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7kEmmgZ2m4bQYDkIvyXJv0uXvPg= 2022-09-12T19:33:59.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7kEmmgZ2m4bQYDkIvyXJv0uXvPg=', 'Content-Length': '0'} 2022-09-12T19:33:59.311 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000789d26834825371-00631f89a7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-12T19:33:59.312 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:59.313 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:59.313 DEBUG:boto:path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:59.314 DEBUG:boto:auth_path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:59.314 DEBUG:boto:Method: GET 2022-09-12T19:33:59.315 DEBUG:boto:Path: /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:59.316 DEBUG:boto:Data: 2022-09-12T19:33:59.316 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.317 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.317 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.318 DEBUG:boto:Params: {} 2022-09-12T19:33:59.319 DEBUG:boto:Token: None 2022-09-12T19:33:59.320 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:59.321 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iforMXIdsbdTZwD5Q/ExkSutxkY= 2022-09-12T19:33:59.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iforMXIdsbdTZwD5Q/ExkSutxkY=', 'Content-Length': '0'} 2022-09-12T19:33:59.325 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac5feaf7e8f991b0-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.326 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:59.327 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:59.328 DEBUG:boto:path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:59.328 DEBUG:boto:auth_path=/jdpiak-62/obj1?versionId=null 2022-09-12T19:33:59.329 DEBUG:boto:Method: GET 2022-09-12T19:33:59.330 DEBUG:boto:Path: /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:59.330 DEBUG:boto:Data: 2022-09-12T19:33:59.331 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.331 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.332 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.332 DEBUG:boto:Params: {} 2022-09-12T19:33:59.333 DEBUG:boto:Token: None 2022-09-12T19:33:59.334 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj1?versionId=null 2022-09-12T19:33:59.334 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iforMXIdsbdTZwD5Q/ExkSutxkY= 2022-09-12T19:33:59.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iforMXIdsbdTZwD5Q/ExkSutxkY=', 'Content-Length': '0'} 2022-09-12T19:33:59.337 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a0e87577775422b-00631f89a7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.378 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:59.379 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:59.380 DEBUG:boto:path=/jdpiak-62/?prefix=obj1 2022-09-12T19:33:59.380 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj1 2022-09-12T19:33:59.381 DEBUG:boto:Method: GET 2022-09-12T19:33:59.381 DEBUG:boto:Path: /jdpiak-62/?prefix=obj1 2022-09-12T19:33:59.382 DEBUG:boto:Data: 2022-09-12T19:33:59.383 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.384 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.385 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.386 DEBUG:boto:Params: {} 2022-09-12T19:33:59.386 DEBUG:boto:Token: None 2022-09-12T19:33:59.387 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/ 2022-09-12T19:33:59.388 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo= 2022-09-12T19:33:59.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo=', 'Content-Length': '0'} 2022-09-12T19:33:59.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019759b678a853e2e-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.393 DEBUG:boto:b'jdpiak-62obj11000falseobj12022-09-12T19:32:15.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:59.394 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:59.395 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:59.396 DEBUG:boto:path=/jdpiak-62/?prefix=obj1 2022-09-12T19:33:59.397 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj1 2022-09-12T19:33:59.397 DEBUG:boto:Method: GET 2022-09-12T19:33:59.398 DEBUG:boto:Path: /jdpiak-62/?prefix=obj1 2022-09-12T19:33:59.398 DEBUG:boto:Data: 2022-09-12T19:33:59.399 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.399 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.400 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.401 DEBUG:boto:Params: {} 2022-09-12T19:33:59.402 DEBUG:boto:Token: None 2022-09-12T19:33:59.404 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/ 2022-09-12T19:33:59.405 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo= 2022-09-12T19:33:59.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo=', 'Content-Length': '0'} 2022-09-12T19:33:59.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007125bb3503b5f89f-00631f89a7-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.409 DEBUG:boto:b'jdpiak-62obj11000falseobj12022-09-12T19:32:15.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:59.410 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:59.411 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:59.412 DEBUG:boto:Method: HEAD 2022-09-12T19:33:59.412 DEBUG:boto:Path: /jdpiak-62/obj1 2022-09-12T19:33:59.413 DEBUG:boto:Data: 2022-09-12T19:33:59.413 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.414 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.415 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.416 DEBUG:boto:Params: {} 2022-09-12T19:33:59.416 DEBUG:boto:Token: None 2022-09-12T19:33:59.417 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj1 2022-09-12T19:33:59.418 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0bhk8TcYXF2G9ei85NcyUHbWfCk= 2022-09-12T19:33:59.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0bhk8TcYXF2G9ei85NcyUHbWfCk=', 'Content-Length': '0'} 2022-09-12T19:33:59.420 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cbc15f61d9b819d1-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.421 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:59.422 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:59.422 DEBUG:boto:Method: HEAD 2022-09-12T19:33:59.423 DEBUG:boto:Path: /jdpiak-62/obj1 2022-09-12T19:33:59.424 DEBUG:boto:Data: 2022-09-12T19:33:59.424 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.425 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.425 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.426 DEBUG:boto:Params: {} 2022-09-12T19:33:59.426 DEBUG:boto:Token: None 2022-09-12T19:33:59.427 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj1 2022-09-12T19:33:59.427 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0bhk8TcYXF2G9ei85NcyUHbWfCk= 2022-09-12T19:33:59.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0bhk8TcYXF2G9ei85NcyUHbWfCk=', 'Content-Length': '0'} 2022-09-12T19:33:59.430 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080827a046a29a083-00631f89a7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-12T19:33:59.432 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:59.432 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:59.433 DEBUG:boto:Method: GET 2022-09-12T19:33:59.433 DEBUG:boto:Path: /jdpiak-62/obj1 2022-09-12T19:33:59.434 DEBUG:boto:Data: 2022-09-12T19:33:59.434 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.435 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.435 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.436 DEBUG:boto:Params: {} 2022-09-12T19:33:59.436 DEBUG:boto:Token: None 2022-09-12T19:33:59.437 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj1 2022-09-12T19:33:59.438 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E9O1ctxPwZFFUV0BwF3d+j3uTZc= 2022-09-12T19:33:59.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E9O1ctxPwZFFUV0BwF3d+j3uTZc=', 'Content-Length': '0'} 2022-09-12T19:33:59.440 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000268c397a65f2f7bf-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.441 DEBUG:boto:path=/jdpiak-62/obj1 2022-09-12T19:33:59.442 DEBUG:boto:auth_path=/jdpiak-62/obj1 2022-09-12T19:33:59.442 DEBUG:boto:Method: GET 2022-09-12T19:33:59.443 DEBUG:boto:Path: /jdpiak-62/obj1 2022-09-12T19:33:59.443 DEBUG:boto:Data: 2022-09-12T19:33:59.444 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.444 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.445 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.445 DEBUG:boto:Params: {} 2022-09-12T19:33:59.446 DEBUG:boto:Token: None 2022-09-12T19:33:59.447 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj1 2022-09-12T19:33:59.447 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E9O1ctxPwZFFUV0BwF3d+j3uTZc= 2022-09-12T19:33:59.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E9O1ctxPwZFFUV0BwF3d+j3uTZc=', 'Content-Length': '0'} 2022-09-12T19:33:59.450 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f511a63edf3018b9-00631f89a7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.451 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:59.451 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:59.452 DEBUG:boto:path=/jdpiak-62/?prefix=obj1 2022-09-12T19:33:59.452 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj1 2022-09-12T19:33:59.453 DEBUG:boto:Method: GET 2022-09-12T19:33:59.453 DEBUG:boto:Path: /jdpiak-62/?prefix=obj1 2022-09-12T19:33:59.454 DEBUG:boto:Data: 2022-09-12T19:33:59.454 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.455 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.455 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.456 DEBUG:boto:Params: {} 2022-09-12T19:33:59.456 DEBUG:boto:Token: None 2022-09-12T19:33:59.457 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/ 2022-09-12T19:33:59.458 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo= 2022-09-12T19:33:59.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo=', 'Content-Length': '0'} 2022-09-12T19:33:59.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6daf466065e33f2-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.462 DEBUG:boto:b'jdpiak-62obj11000falseobj12022-09-12T19:32:15.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:59.463 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:59.463 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:59.464 DEBUG:boto:path=/jdpiak-62/?prefix=obj1 2022-09-12T19:33:59.464 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj1 2022-09-12T19:33:59.465 DEBUG:boto:Method: GET 2022-09-12T19:33:59.465 DEBUG:boto:Path: /jdpiak-62/?prefix=obj1 2022-09-12T19:33:59.466 DEBUG:boto:Data: 2022-09-12T19:33:59.466 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.467 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.467 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.468 DEBUG:boto:Params: {} 2022-09-12T19:33:59.468 DEBUG:boto:Token: None 2022-09-12T19:33:59.469 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/ 2022-09-12T19:33:59.469 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo= 2022-09-12T19:33:59.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo=', 'Content-Length': '0'} 2022-09-12T19:33:59.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000240e3b5db47ea9ce-00631f89a7-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.514 DEBUG:boto:b'jdpiak-62obj11000falseobj12022-09-12T19:32:15.039Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:59.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-12T19:33:59.516 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:59.516 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:59.517 DEBUG:boto:path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:59.517 DEBUG:boto:auth_path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:59.518 DEBUG:boto:Method: GET 2022-09-12T19:33:59.518 DEBUG:boto:Path: /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:59.519 DEBUG:boto:Data: 2022-09-12T19:33:59.520 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.521 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.521 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.522 DEBUG:boto:Params: {} 2022-09-12T19:33:59.523 DEBUG:boto:Token: None 2022-09-12T19:33:59.523 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:59.524 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XCCw2ufE/jbI555LmP8KoSzOhWo= 2022-09-12T19:33:59.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XCCw2ufE/jbI555LmP8KoSzOhWo=', 'Content-Length': '0'} 2022-09-12T19:33:59.527 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005edba8f052f34eea-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.528 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:59.528 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:59.529 DEBUG:boto:path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:59.529 DEBUG:boto:auth_path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:59.530 DEBUG:boto:Method: GET 2022-09-12T19:33:59.531 DEBUG:boto:Path: /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:59.531 DEBUG:boto:Data: 2022-09-12T19:33:59.532 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.532 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.533 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.534 DEBUG:boto:Params: {} 2022-09-12T19:33:59.534 DEBUG:boto:Token: None 2022-09-12T19:33:59.535 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:59.536 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XCCw2ufE/jbI555LmP8KoSzOhWo= 2022-09-12T19:33:59.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XCCw2ufE/jbI555LmP8KoSzOhWo=', 'Content-Length': '0'} 2022-09-12T19:33:59.539 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4c81bea28249cdb-00631f89a7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.540 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:59.540 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:59.541 DEBUG:boto:path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:59.541 DEBUG:boto:auth_path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:59.542 DEBUG:boto:Method: HEAD 2022-09-12T19:33:59.542 DEBUG:boto:Path: /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:59.543 DEBUG:boto:Data: 2022-09-12T19:33:59.543 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.544 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.544 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.545 DEBUG:boto:Params: {} 2022-09-12T19:33:59.545 DEBUG:boto:Token: None 2022-09-12T19:33:59.546 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:59.546 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WJk7r2TOhBy/se4tO3QW0jQt/uI= 2022-09-12T19:33:59.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WJk7r2TOhBy/se4tO3QW0jQt/uI=', 'Content-Length': '0'} 2022-09-12T19:33:59.549 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f741a6c49154920e-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.550 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:59.551 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:59.552 DEBUG:boto:path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:59.553 DEBUG:boto:auth_path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:59.553 DEBUG:boto:Method: HEAD 2022-09-12T19:33:59.554 DEBUG:boto:Path: /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:59.555 DEBUG:boto:Data: 2022-09-12T19:33:59.556 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.556 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.557 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.558 DEBUG:boto:Params: {} 2022-09-12T19:33:59.559 DEBUG:boto:Token: None 2022-09-12T19:33:59.559 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:59.560 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WJk7r2TOhBy/se4tO3QW0jQt/uI= 2022-09-12T19:33:59.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WJk7r2TOhBy/se4tO3QW0jQt/uI=', 'Content-Length': '0'} 2022-09-12T19:33:59.563 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b35b23fc7bc561a6-00631f89a7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-12T19:33:59.564 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:59.565 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:59.565 DEBUG:boto:path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:59.566 DEBUG:boto:auth_path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:59.566 DEBUG:boto:Method: GET 2022-09-12T19:33:59.567 DEBUG:boto:Path: /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:59.567 DEBUG:boto:Data: 2022-09-12T19:33:59.568 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.568 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.569 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.570 DEBUG:boto:Params: {} 2022-09-12T19:33:59.570 DEBUG:boto:Token: None 2022-09-12T19:33:59.571 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:59.571 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XCCw2ufE/jbI555LmP8KoSzOhWo= 2022-09-12T19:33:59.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XCCw2ufE/jbI555LmP8KoSzOhWo=', 'Content-Length': '0'} 2022-09-12T19:33:59.574 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e0706e2a68f3b9a-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.618 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:59.619 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:59.619 DEBUG:boto:path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:59.620 DEBUG:boto:auth_path=/jdpiak-62/obj2?versionId=null 2022-09-12T19:33:59.620 DEBUG:boto:Method: GET 2022-09-12T19:33:59.621 DEBUG:boto:Path: /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:59.622 DEBUG:boto:Data: 2022-09-12T19:33:59.622 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.623 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.623 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.624 DEBUG:boto:Params: {} 2022-09-12T19:33:59.624 DEBUG:boto:Token: None 2022-09-12T19:33:59.625 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj2?versionId=null 2022-09-12T19:33:59.626 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XCCw2ufE/jbI555LmP8KoSzOhWo= 2022-09-12T19:33:59.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XCCw2ufE/jbI555LmP8KoSzOhWo=', 'Content-Length': '0'} 2022-09-12T19:33:59.629 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c20655ea53ceeb25-00631f89a7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.629 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:59.630 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:59.630 DEBUG:boto:path=/jdpiak-62/?prefix=obj2 2022-09-12T19:33:59.631 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj2 2022-09-12T19:33:59.631 DEBUG:boto:Method: GET 2022-09-12T19:33:59.632 DEBUG:boto:Path: /jdpiak-62/?prefix=obj2 2022-09-12T19:33:59.633 DEBUG:boto:Data: 2022-09-12T19:33:59.633 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.634 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.634 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.635 DEBUG:boto:Params: {} 2022-09-12T19:33:59.636 DEBUG:boto:Token: None 2022-09-12T19:33:59.636 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/ 2022-09-12T19:33:59.637 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo= 2022-09-12T19:33:59.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo=', 'Content-Length': '0'} 2022-09-12T19:33:59.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f398a5a7870bc86-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.641 DEBUG:boto:b'jdpiak-62obj21000falseobj22022-09-12T19:32:15.070Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:59.642 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:59.643 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:59.643 DEBUG:boto:path=/jdpiak-62/?prefix=obj2 2022-09-12T19:33:59.644 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj2 2022-09-12T19:33:59.645 DEBUG:boto:Method: GET 2022-09-12T19:33:59.645 DEBUG:boto:Path: /jdpiak-62/?prefix=obj2 2022-09-12T19:33:59.646 DEBUG:boto:Data: 2022-09-12T19:33:59.646 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.647 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.647 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.648 DEBUG:boto:Params: {} 2022-09-12T19:33:59.649 DEBUG:boto:Token: None 2022-09-12T19:33:59.649 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/ 2022-09-12T19:33:59.650 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo= 2022-09-12T19:33:59.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo=', 'Content-Length': '0'} 2022-09-12T19:33:59.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004fea4d6bff5b046-00631f89a7-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.654 DEBUG:boto:b'jdpiak-62obj21000falseobj22022-09-12T19:32:15.070Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:59.655 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:59.656 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:59.656 DEBUG:boto:Method: HEAD 2022-09-12T19:33:59.657 DEBUG:boto:Path: /jdpiak-62/obj2 2022-09-12T19:33:59.657 DEBUG:boto:Data: 2022-09-12T19:33:59.658 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.658 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.659 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.659 DEBUG:boto:Params: {} 2022-09-12T19:33:59.660 DEBUG:boto:Token: None 2022-09-12T19:33:59.660 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj2 2022-09-12T19:33:59.661 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2y4d0sUupPpZQ44tGGBqbBUi89s= 2022-09-12T19:33:59.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2y4d0sUupPpZQ44tGGBqbBUi89s=', 'Content-Length': '0'} 2022-09-12T19:33:59.663 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000743589997ce51250-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.664 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:59.665 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:59.665 DEBUG:boto:Method: HEAD 2022-09-12T19:33:59.666 DEBUG:boto:Path: /jdpiak-62/obj2 2022-09-12T19:33:59.666 DEBUG:boto:Data: 2022-09-12T19:33:59.667 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.667 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.668 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.668 DEBUG:boto:Params: {} 2022-09-12T19:33:59.669 DEBUG:boto:Token: None 2022-09-12T19:33:59.670 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj2 2022-09-12T19:33:59.670 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2y4d0sUupPpZQ44tGGBqbBUi89s= 2022-09-12T19:33:59.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2y4d0sUupPpZQ44tGGBqbBUi89s=', 'Content-Length': '0'} 2022-09-12T19:33:59.673 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f385f76bf26939b-00631f89a7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-12T19:33:59.674 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:59.675 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:59.676 DEBUG:boto:Method: GET 2022-09-12T19:33:59.676 DEBUG:boto:Path: /jdpiak-62/obj2 2022-09-12T19:33:59.677 DEBUG:boto:Data: 2022-09-12T19:33:59.678 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.678 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.679 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.680 DEBUG:boto:Params: {} 2022-09-12T19:33:59.681 DEBUG:boto:Token: None 2022-09-12T19:33:59.682 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj2 2022-09-12T19:33:59.683 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6WtU3nJJwv/OhfYR8G/yumoDDKg= 2022-09-12T19:33:59.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6WtU3nJJwv/OhfYR8G/yumoDDKg=', 'Content-Length': '0'} 2022-09-12T19:33:59.685 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cf657060dbdeb26-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.686 DEBUG:boto:path=/jdpiak-62/obj2 2022-09-12T19:33:59.687 DEBUG:boto:auth_path=/jdpiak-62/obj2 2022-09-12T19:33:59.688 DEBUG:boto:Method: GET 2022-09-12T19:33:59.688 DEBUG:boto:Path: /jdpiak-62/obj2 2022-09-12T19:33:59.689 DEBUG:boto:Data: 2022-09-12T19:33:59.689 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.690 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.690 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.691 DEBUG:boto:Params: {} 2022-09-12T19:33:59.692 DEBUG:boto:Token: None 2022-09-12T19:33:59.693 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj2 2022-09-12T19:33:59.693 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6WtU3nJJwv/OhfYR8G/yumoDDKg= 2022-09-12T19:33:59.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6WtU3nJJwv/OhfYR8G/yumoDDKg=', 'Content-Length': '0'} 2022-09-12T19:33:59.696 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fda7d6c6356f7f33-00631f89a7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.697 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:59.698 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:59.698 DEBUG:boto:path=/jdpiak-62/?prefix=obj2 2022-09-12T19:33:59.699 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj2 2022-09-12T19:33:59.699 DEBUG:boto:Method: GET 2022-09-12T19:33:59.700 DEBUG:boto:Path: /jdpiak-62/?prefix=obj2 2022-09-12T19:33:59.701 DEBUG:boto:Data: 2022-09-12T19:33:59.701 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.702 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.702 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.703 DEBUG:boto:Params: {} 2022-09-12T19:33:59.703 DEBUG:boto:Token: None 2022-09-12T19:33:59.704 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/ 2022-09-12T19:33:59.705 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo= 2022-09-12T19:33:59.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo=', 'Content-Length': '0'} 2022-09-12T19:33:59.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000553b76a410e7956d-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.709 DEBUG:boto:b'jdpiak-62obj21000falseobj22022-09-12T19:32:15.070Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:59.710 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:59.710 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:59.711 DEBUG:boto:path=/jdpiak-62/?prefix=obj2 2022-09-12T19:33:59.711 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj2 2022-09-12T19:33:59.712 DEBUG:boto:Method: GET 2022-09-12T19:33:59.712 DEBUG:boto:Path: /jdpiak-62/?prefix=obj2 2022-09-12T19:33:59.713 DEBUG:boto:Data: 2022-09-12T19:33:59.714 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.714 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.715 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.715 DEBUG:boto:Params: {} 2022-09-12T19:33:59.716 DEBUG:boto:Token: None 2022-09-12T19:33:59.717 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/ 2022-09-12T19:33:59.717 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo= 2022-09-12T19:33:59.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo=', 'Content-Length': '0'} 2022-09-12T19:33:59.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9484de26ec52c65-00631f89a7-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.722 DEBUG:boto:b'jdpiak-62obj21000falseobj22022-09-12T19:32:15.070Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:59.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-12T19:33:59.723 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:59.724 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:59.724 DEBUG:boto:path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:59.725 DEBUG:boto:auth_path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:59.725 DEBUG:boto:Method: GET 2022-09-12T19:33:59.726 DEBUG:boto:Path: /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:59.727 DEBUG:boto:Data: 2022-09-12T19:33:59.727 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.728 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.728 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.729 DEBUG:boto:Params: {} 2022-09-12T19:33:59.729 DEBUG:boto:Token: None 2022-09-12T19:33:59.730 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:59.730 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mqHL6JbaYriJcFUdHEVIoMYOb2k= 2022-09-12T19:33:59.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mqHL6JbaYriJcFUdHEVIoMYOb2k=', 'Content-Length': '0'} 2022-09-12T19:33:59.733 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db9f803a1abc722d-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.734 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:59.735 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:59.735 DEBUG:boto:path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:59.736 DEBUG:boto:auth_path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:59.737 DEBUG:boto:Method: GET 2022-09-12T19:33:59.737 DEBUG:boto:Path: /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:59.738 DEBUG:boto:Data: 2022-09-12T19:33:59.738 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.739 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.739 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.740 DEBUG:boto:Params: {} 2022-09-12T19:33:59.741 DEBUG:boto:Token: None 2022-09-12T19:33:59.741 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:59.742 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mqHL6JbaYriJcFUdHEVIoMYOb2k= 2022-09-12T19:33:59.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mqHL6JbaYriJcFUdHEVIoMYOb2k=', 'Content-Length': '0'} 2022-09-12T19:33:59.745 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005297e4b399f151b3-00631f89a7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.746 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:59.747 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:59.747 DEBUG:boto:path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:59.748 DEBUG:boto:auth_path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:59.748 DEBUG:boto:Method: HEAD 2022-09-12T19:33:59.749 DEBUG:boto:Path: /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:59.749 DEBUG:boto:Data: 2022-09-12T19:33:59.750 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.750 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.751 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.752 DEBUG:boto:Params: {} 2022-09-12T19:33:59.752 DEBUG:boto:Token: None 2022-09-12T19:33:59.753 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:59.754 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JQsNXGmflpeXe1amYzohAGKqZGE= 2022-09-12T19:33:59.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JQsNXGmflpeXe1amYzohAGKqZGE=', 'Content-Length': '0'} 2022-09-12T19:33:59.756 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed7b674acb8b0269-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.758 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:59.759 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:59.760 DEBUG:boto:path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:59.760 DEBUG:boto:auth_path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:59.761 DEBUG:boto:Method: HEAD 2022-09-12T19:33:59.761 DEBUG:boto:Path: /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:59.762 DEBUG:boto:Data: 2022-09-12T19:33:59.762 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.764 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.765 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.766 DEBUG:boto:Params: {} 2022-09-12T19:33:59.766 DEBUG:boto:Token: None 2022-09-12T19:33:59.767 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:59.768 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JQsNXGmflpeXe1amYzohAGKqZGE= 2022-09-12T19:33:59.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JQsNXGmflpeXe1amYzohAGKqZGE=', 'Content-Length': '0'} 2022-09-12T19:33:59.784 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003e417d1d81f5cb2-00631f89a7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-12T19:33:59.786 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:59.786 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:59.787 DEBUG:boto:path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:59.788 DEBUG:boto:auth_path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:59.788 DEBUG:boto:Method: GET 2022-09-12T19:33:59.789 DEBUG:boto:Path: /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:59.789 DEBUG:boto:Data: 2022-09-12T19:33:59.790 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.790 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.791 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.791 DEBUG:boto:Params: {} 2022-09-12T19:33:59.792 DEBUG:boto:Token: None 2022-09-12T19:33:59.793 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:59.793 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mqHL6JbaYriJcFUdHEVIoMYOb2k= 2022-09-12T19:33:59.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mqHL6JbaYriJcFUdHEVIoMYOb2k=', 'Content-Length': '0'} 2022-09-12T19:33:59.796 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000585c3a58bc85c9f5-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.797 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:59.798 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:59.799 DEBUG:boto:path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:59.799 DEBUG:boto:auth_path=/jdpiak-62/obj3?versionId=null 2022-09-12T19:33:59.800 DEBUG:boto:Method: GET 2022-09-12T19:33:59.800 DEBUG:boto:Path: /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:59.801 DEBUG:boto:Data: 2022-09-12T19:33:59.801 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.802 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.802 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.803 DEBUG:boto:Params: {} 2022-09-12T19:33:59.804 DEBUG:boto:Token: None 2022-09-12T19:33:59.804 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj3?versionId=null 2022-09-12T19:33:59.805 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mqHL6JbaYriJcFUdHEVIoMYOb2k= 2022-09-12T19:33:59.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mqHL6JbaYriJcFUdHEVIoMYOb2k=', 'Content-Length': '0'} 2022-09-12T19:33:59.808 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e9c501c762f7770-00631f89a7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.850 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:59.851 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:59.851 DEBUG:boto:path=/jdpiak-62/?prefix=obj3 2022-09-12T19:33:59.852 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj3 2022-09-12T19:33:59.853 DEBUG:boto:Method: GET 2022-09-12T19:33:59.853 DEBUG:boto:Path: /jdpiak-62/?prefix=obj3 2022-09-12T19:33:59.854 DEBUG:boto:Data: 2022-09-12T19:33:59.855 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.855 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.856 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.856 DEBUG:boto:Params: {} 2022-09-12T19:33:59.857 DEBUG:boto:Token: None 2022-09-12T19:33:59.858 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/ 2022-09-12T19:33:59.859 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo= 2022-09-12T19:33:59.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo=', 'Content-Length': '0'} 2022-09-12T19:33:59.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c74fafe3640d9f31-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.863 DEBUG:boto:b'jdpiak-62obj31000falseobj32022-09-12T19:32:15.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:59.865 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:59.865 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:59.866 DEBUG:boto:path=/jdpiak-62/?prefix=obj3 2022-09-12T19:33:59.866 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj3 2022-09-12T19:33:59.867 DEBUG:boto:Method: GET 2022-09-12T19:33:59.868 DEBUG:boto:Path: /jdpiak-62/?prefix=obj3 2022-09-12T19:33:59.868 DEBUG:boto:Data: 2022-09-12T19:33:59.869 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.870 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.870 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.871 DEBUG:boto:Params: {} 2022-09-12T19:33:59.871 DEBUG:boto:Token: None 2022-09-12T19:33:59.872 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/ 2022-09-12T19:33:59.873 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo= 2022-09-12T19:33:59.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo=', 'Content-Length': '0'} 2022-09-12T19:33:59.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069a3f2b46e983a47-00631f89a7-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.877 DEBUG:boto:b'jdpiak-62obj31000falseobj32022-09-12T19:32:15.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:59.878 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:59.878 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:59.879 DEBUG:boto:Method: HEAD 2022-09-12T19:33:59.879 DEBUG:boto:Path: /jdpiak-62/obj3 2022-09-12T19:33:59.880 DEBUG:boto:Data: 2022-09-12T19:33:59.881 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.881 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.882 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.882 DEBUG:boto:Params: {} 2022-09-12T19:33:59.883 DEBUG:boto:Token: None 2022-09-12T19:33:59.884 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj3 2022-09-12T19:33:59.884 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1iGnC1zzP5uD66eXA0FjTvyUlPA= 2022-09-12T19:33:59.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1iGnC1zzP5uD66eXA0FjTvyUlPA=', 'Content-Length': '0'} 2022-09-12T19:33:59.887 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000612fc19f87e7565-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.888 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:59.889 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:59.890 DEBUG:boto:Method: HEAD 2022-09-12T19:33:59.890 DEBUG:boto:Path: /jdpiak-62/obj3 2022-09-12T19:33:59.891 DEBUG:boto:Data: 2022-09-12T19:33:59.891 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.892 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.893 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.894 DEBUG:boto:Params: {} 2022-09-12T19:33:59.895 DEBUG:boto:Token: None 2022-09-12T19:33:59.896 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj3 2022-09-12T19:33:59.897 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1iGnC1zzP5uD66eXA0FjTvyUlPA= 2022-09-12T19:33:59.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1iGnC1zzP5uD66eXA0FjTvyUlPA=', 'Content-Length': '0'} 2022-09-12T19:33:59.900 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076fc01bd59be3b91-00631f89a7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-12T19:33:59.902 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:59.903 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:59.904 DEBUG:boto:Method: GET 2022-09-12T19:33:59.905 DEBUG:boto:Path: /jdpiak-62/obj3 2022-09-12T19:33:59.905 DEBUG:boto:Data: 2022-09-12T19:33:59.906 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.906 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.907 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.908 DEBUG:boto:Params: {} 2022-09-12T19:33:59.908 DEBUG:boto:Token: None 2022-09-12T19:33:59.909 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj3 2022-09-12T19:33:59.910 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KpNKtBIDtzHupcBKMlnWskSjntY= 2022-09-12T19:33:59.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KpNKtBIDtzHupcBKMlnWskSjntY=', 'Content-Length': '0'} 2022-09-12T19:33:59.913 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc938681807bf48d-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.914 DEBUG:boto:path=/jdpiak-62/obj3 2022-09-12T19:33:59.914 DEBUG:boto:auth_path=/jdpiak-62/obj3 2022-09-12T19:33:59.915 DEBUG:boto:Method: GET 2022-09-12T19:33:59.916 DEBUG:boto:Path: /jdpiak-62/obj3 2022-09-12T19:33:59.916 DEBUG:boto:Data: 2022-09-12T19:33:59.917 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.918 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.918 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.919 DEBUG:boto:Params: {} 2022-09-12T19:33:59.920 DEBUG:boto:Token: None 2022-09-12T19:33:59.920 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj3 2022-09-12T19:33:59.921 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KpNKtBIDtzHupcBKMlnWskSjntY= 2022-09-12T19:33:59.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KpNKtBIDtzHupcBKMlnWskSjntY=', 'Content-Length': '0'} 2022-09-12T19:33:59.924 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ecd656d7a46925dc-00631f89a7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.925 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:59.926 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:59.926 DEBUG:boto:path=/jdpiak-62/?prefix=obj3 2022-09-12T19:33:59.927 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj3 2022-09-12T19:33:59.927 DEBUG:boto:Method: GET 2022-09-12T19:33:59.928 DEBUG:boto:Path: /jdpiak-62/?prefix=obj3 2022-09-12T19:33:59.929 DEBUG:boto:Data: 2022-09-12T19:33:59.930 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.930 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.931 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.931 DEBUG:boto:Params: {} 2022-09-12T19:33:59.932 DEBUG:boto:Token: None 2022-09-12T19:33:59.933 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/ 2022-09-12T19:33:59.934 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo= 2022-09-12T19:33:59.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo=', 'Content-Length': '0'} 2022-09-12T19:33:59.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fd3ce9595f2e8d8-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.938 DEBUG:boto:b'jdpiak-62obj31000falseobj32022-09-12T19:32:15.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:59.939 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:33:59.940 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:33:59.940 DEBUG:boto:path=/jdpiak-62/?prefix=obj3 2022-09-12T19:33:59.941 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj3 2022-09-12T19:33:59.942 DEBUG:boto:Method: GET 2022-09-12T19:33:59.942 DEBUG:boto:Path: /jdpiak-62/?prefix=obj3 2022-09-12T19:33:59.943 DEBUG:boto:Data: 2022-09-12T19:33:59.943 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.944 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.945 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.945 DEBUG:boto:Params: {} 2022-09-12T19:33:59.946 DEBUG:boto:Token: None 2022-09-12T19:33:59.947 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/ 2022-09-12T19:33:59.948 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo= 2022-09-12T19:33:59.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X1VIDRmLSyK+KgNhzs8XlVDt0zo=', 'Content-Length': '0'} 2022-09-12T19:33:59.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005260afb4c9ddc46a-00631f89a7-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.952 DEBUG:boto:b'jdpiak-62obj31000falseobj32022-09-12T19:32:15.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:33:59.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-12T19:33:59.953 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:33:59.954 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:33:59.954 DEBUG:boto:path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:59.955 DEBUG:boto:auth_path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:59.956 DEBUG:boto:Method: GET 2022-09-12T19:33:59.956 DEBUG:boto:Path: /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:59.957 DEBUG:boto:Data: 2022-09-12T19:33:59.958 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.959 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.959 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.960 DEBUG:boto:Params: {} 2022-09-12T19:33:59.961 DEBUG:boto:Token: None 2022-09-12T19:33:59.961 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:59.962 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E8RQ4rYQ0FJcvRyVuHQ1d2o8J64= 2022-09-12T19:33:59.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E8RQ4rYQ0FJcvRyVuHQ1d2o8J64=', 'Content-Length': '0'} 2022-09-12T19:33:59.965 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a44235075c5c9f4-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.966 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:33:59.967 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:33:59.967 DEBUG:boto:path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:59.968 DEBUG:boto:auth_path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:59.969 DEBUG:boto:Method: GET 2022-09-12T19:33:59.969 DEBUG:boto:Path: /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:59.970 DEBUG:boto:Data: 2022-09-12T19:33:59.971 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.971 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.972 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.972 DEBUG:boto:Params: {} 2022-09-12T19:33:59.973 DEBUG:boto:Token: None 2022-09-12T19:33:59.973 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:59.974 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E8RQ4rYQ0FJcvRyVuHQ1d2o8J64= 2022-09-12T19:33:59.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E8RQ4rYQ0FJcvRyVuHQ1d2o8J64=', 'Content-Length': '0'} 2022-09-12T19:33:59.977 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7f433f3e3304e88-00631f89a7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.978 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:33:59.979 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:33:59.979 DEBUG:boto:path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:59.980 DEBUG:boto:auth_path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:59.980 DEBUG:boto:Method: HEAD 2022-09-12T19:33:59.981 DEBUG:boto:Path: /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:59.981 DEBUG:boto:Data: 2022-09-12T19:33:59.982 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.982 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:33:59.983 DEBUG:boto:Port: 8001 2022-09-12T19:33:59.984 DEBUG:boto:Params: {} 2022-09-12T19:33:59.984 DEBUG:boto:Token: None 2022-09-12T19:33:59.985 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:59.986 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gwUtM9V2rVzoIA0sdIZNrxmRHr4= 2022-09-12T19:33:59.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gwUtM9V2rVzoIA0sdIZNrxmRHr4=', 'Content-Length': '0'} 2022-09-12T19:33:59.989 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9f8ffed8cf33eb0-00631f89a7-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:33:59.990 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:33:59.990 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:33:59.991 DEBUG:boto:path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:59.992 DEBUG:boto:auth_path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:33:59.992 DEBUG:boto:Method: HEAD 2022-09-12T19:33:59.993 DEBUG:boto:Path: /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:59.993 DEBUG:boto:Data: 2022-09-12T19:33:59.994 DEBUG:boto:Headers: {} 2022-09-12T19:33:59.995 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:33:59.996 DEBUG:boto:Port: 8000 2022-09-12T19:33:59.996 DEBUG:boto:Params: {} 2022-09-12T19:33:59.997 DEBUG:boto:Token: None 2022-09-12T19:33:59.997 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:33:59 GMT /jdpiak-62/obj4?versionId=null 2022-09-12T19:33:59.998 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gwUtM9V2rVzoIA0sdIZNrxmRHr4= 2022-09-12T19:33:59.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:33:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gwUtM9V2rVzoIA0sdIZNrxmRHr4=', 'Content-Length': '0'} 2022-09-12T19:34:00.001 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b7fbc36b03ea5e2-00631f89a7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:33:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:00.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-12T19:34:00.002 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:34:00.003 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:34:00.003 DEBUG:boto:path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:34:00.004 DEBUG:boto:auth_path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:34:00.005 DEBUG:boto:Method: GET 2022-09-12T19:34:00.006 DEBUG:boto:Path: /jdpiak-62/obj4?versionId=null 2022-09-12T19:34:00.006 DEBUG:boto:Data: 2022-09-12T19:34:00.007 DEBUG:boto:Headers: {} 2022-09-12T19:34:00.008 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:00.008 DEBUG:boto:Port: 8001 2022-09-12T19:34:00.009 DEBUG:boto:Params: {} 2022-09-12T19:34:00.009 DEBUG:boto:Token: None 2022-09-12T19:34:00.010 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:00 GMT /jdpiak-62/obj4?versionId=null 2022-09-12T19:34:00.011 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qVJ9KAUEBV4ys2CIHJDQcl7sTAo= 2022-09-12T19:34:00.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qVJ9KAUEBV4ys2CIHJDQcl7sTAo=', 'Content-Length': '0'} 2022-09-12T19:34:00.014 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000276ceda4a3188ac9-00631f89a8-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:00.015 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:34:00.016 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:34:00.016 DEBUG:boto:path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:34:00.017 DEBUG:boto:auth_path=/jdpiak-62/obj4?versionId=null 2022-09-12T19:34:00.017 DEBUG:boto:Method: GET 2022-09-12T19:34:00.018 DEBUG:boto:Path: /jdpiak-62/obj4?versionId=null 2022-09-12T19:34:00.018 DEBUG:boto:Data: 2022-09-12T19:34:00.019 DEBUG:boto:Headers: {} 2022-09-12T19:34:00.019 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:00.020 DEBUG:boto:Port: 8000 2022-09-12T19:34:00.021 DEBUG:boto:Params: {} 2022-09-12T19:34:00.021 DEBUG:boto:Token: None 2022-09-12T19:34:00.022 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:00 GMT /jdpiak-62/obj4?versionId=null 2022-09-12T19:34:00.023 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qVJ9KAUEBV4ys2CIHJDQcl7sTAo= 2022-09-12T19:34:00.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qVJ9KAUEBV4ys2CIHJDQcl7sTAo=', 'Content-Length': '0'} 2022-09-12T19:34:00.026 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000705e86beda486bb7-00631f89a8-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:00.027 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:34:00.028 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:34:00.029 DEBUG:boto:path=/jdpiak-62/?prefix=obj4 2022-09-12T19:34:00.029 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj4 2022-09-12T19:34:00.030 DEBUG:boto:Method: GET 2022-09-12T19:34:00.031 DEBUG:boto:Path: /jdpiak-62/?prefix=obj4 2022-09-12T19:34:00.032 DEBUG:boto:Data: 2022-09-12T19:34:00.032 DEBUG:boto:Headers: {} 2022-09-12T19:34:00.033 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:00.033 DEBUG:boto:Port: 8001 2022-09-12T19:34:00.034 DEBUG:boto:Params: {} 2022-09-12T19:34:00.035 DEBUG:boto:Token: None 2022-09-12T19:34:00.035 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:00 GMT /jdpiak-62/ 2022-09-12T19:34:00.036 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R4ZlWMuJQ1jiOBrGoS2bVrXkroo= 2022-09-12T19:34:00.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R4ZlWMuJQ1jiOBrGoS2bVrXkroo=', 'Content-Length': '0'} 2022-09-12T19:34:00.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060737a4878f5dbfc-00631f89a8-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:00.082 DEBUG:boto:b'jdpiak-62obj41000falseobj42022-09-12T19:32:15.218Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:00.083 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:34:00.084 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:34:00.084 DEBUG:boto:path=/jdpiak-62/?prefix=obj4 2022-09-12T19:34:00.085 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj4 2022-09-12T19:34:00.085 DEBUG:boto:Method: GET 2022-09-12T19:34:00.086 DEBUG:boto:Path: /jdpiak-62/?prefix=obj4 2022-09-12T19:34:00.087 DEBUG:boto:Data: 2022-09-12T19:34:00.087 DEBUG:boto:Headers: {} 2022-09-12T19:34:00.088 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:00.088 DEBUG:boto:Port: 8000 2022-09-12T19:34:00.089 DEBUG:boto:Params: {} 2022-09-12T19:34:00.090 DEBUG:boto:Token: None 2022-09-12T19:34:00.090 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:00 GMT /jdpiak-62/ 2022-09-12T19:34:00.091 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R4ZlWMuJQ1jiOBrGoS2bVrXkroo= 2022-09-12T19:34:00.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R4ZlWMuJQ1jiOBrGoS2bVrXkroo=', 'Content-Length': '0'} 2022-09-12T19:34:00.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da02e208fe8361b6-00631f89a8-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:00.095 DEBUG:boto:b'jdpiak-62obj41000falseobj42022-09-12T19:32:15.218Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:00.096 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:34:00.097 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:34:00.097 DEBUG:boto:Method: HEAD 2022-09-12T19:34:00.098 DEBUG:boto:Path: /jdpiak-62/obj4 2022-09-12T19:34:00.098 DEBUG:boto:Data: 2022-09-12T19:34:00.099 DEBUG:boto:Headers: {} 2022-09-12T19:34:00.099 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:00.100 DEBUG:boto:Port: 8001 2022-09-12T19:34:00.101 DEBUG:boto:Params: {} 2022-09-12T19:34:00.101 DEBUG:boto:Token: None 2022-09-12T19:34:00.102 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:00 GMT /jdpiak-62/obj4 2022-09-12T19:34:00.102 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DkvytPEw20Eg6EjI3w5IL6f8als= 2022-09-12T19:34:00.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DkvytPEw20Eg6EjI3w5IL6f8als=', 'Content-Length': '0'} 2022-09-12T19:34:00.105 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000742697e752e51ecd-00631f89a8-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:00.106 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:34:00.107 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:34:00.107 DEBUG:boto:Method: HEAD 2022-09-12T19:34:00.108 DEBUG:boto:Path: /jdpiak-62/obj4 2022-09-12T19:34:00.108 DEBUG:boto:Data: 2022-09-12T19:34:00.109 DEBUG:boto:Headers: {} 2022-09-12T19:34:00.109 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:00.110 DEBUG:boto:Port: 8000 2022-09-12T19:34:00.110 DEBUG:boto:Params: {} 2022-09-12T19:34:00.111 DEBUG:boto:Token: None 2022-09-12T19:34:00.111 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:00 GMT /jdpiak-62/obj4 2022-09-12T19:34:00.112 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DkvytPEw20Eg6EjI3w5IL6f8als= 2022-09-12T19:34:00.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DkvytPEw20Eg6EjI3w5IL6f8als=', 'Content-Length': '0'} 2022-09-12T19:34:00.115 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028eed2788abd2d9d-00631f89a8-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:00.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-12T19:34:00.116 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:34:00.117 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:34:00.117 DEBUG:boto:Method: GET 2022-09-12T19:34:00.118 DEBUG:boto:Path: /jdpiak-62/obj4 2022-09-12T19:34:00.118 DEBUG:boto:Data: 2022-09-12T19:34:00.119 DEBUG:boto:Headers: {} 2022-09-12T19:34:00.119 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:00.120 DEBUG:boto:Port: 8001 2022-09-12T19:34:00.120 DEBUG:boto:Params: {} 2022-09-12T19:34:00.121 DEBUG:boto:Token: None 2022-09-12T19:34:00.122 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:00 GMT /jdpiak-62/obj4 2022-09-12T19:34:00.123 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LNzBEIjki77ByEyDkIavRlzZYFY= 2022-09-12T19:34:00.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LNzBEIjki77ByEyDkIavRlzZYFY=', 'Content-Length': '0'} 2022-09-12T19:34:00.125 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000865a04f1e68f57a8-00631f89a8-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:00.126 DEBUG:boto:path=/jdpiak-62/obj4 2022-09-12T19:34:00.127 DEBUG:boto:auth_path=/jdpiak-62/obj4 2022-09-12T19:34:00.127 DEBUG:boto:Method: GET 2022-09-12T19:34:00.128 DEBUG:boto:Path: /jdpiak-62/obj4 2022-09-12T19:34:00.128 DEBUG:boto:Data: 2022-09-12T19:34:00.129 DEBUG:boto:Headers: {} 2022-09-12T19:34:00.130 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:00.130 DEBUG:boto:Port: 8000 2022-09-12T19:34:00.131 DEBUG:boto:Params: {} 2022-09-12T19:34:00.132 DEBUG:boto:Token: None 2022-09-12T19:34:00.133 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:00 GMT /jdpiak-62/obj4 2022-09-12T19:34:00.134 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LNzBEIjki77ByEyDkIavRlzZYFY= 2022-09-12T19:34:00.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LNzBEIjki77ByEyDkIavRlzZYFY=', 'Content-Length': '0'} 2022-09-12T19:34:00.136 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022c9b6ce46dae886-00631f89a8-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:00.178 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:34:00.179 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:34:00.179 DEBUG:boto:path=/jdpiak-62/?prefix=obj4 2022-09-12T19:34:00.180 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj4 2022-09-12T19:34:00.180 DEBUG:boto:Method: GET 2022-09-12T19:34:00.181 DEBUG:boto:Path: /jdpiak-62/?prefix=obj4 2022-09-12T19:34:00.182 DEBUG:boto:Data: 2022-09-12T19:34:00.182 DEBUG:boto:Headers: {} 2022-09-12T19:34:00.183 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:00.183 DEBUG:boto:Port: 8001 2022-09-12T19:34:00.184 DEBUG:boto:Params: {} 2022-09-12T19:34:00.185 DEBUG:boto:Token: None 2022-09-12T19:34:00.185 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:00 GMT /jdpiak-62/ 2022-09-12T19:34:00.186 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R4ZlWMuJQ1jiOBrGoS2bVrXkroo= 2022-09-12T19:34:00.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R4ZlWMuJQ1jiOBrGoS2bVrXkroo=', 'Content-Length': '0'} 2022-09-12T19:34:00.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fdd1ee63cbcff97-00631f89a8-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:00.190 DEBUG:boto:b'jdpiak-62obj41000falseobj42022-09-12T19:32:15.218Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:00.191 DEBUG:boto:path=/jdpiak-62/ 2022-09-12T19:34:00.191 DEBUG:boto:auth_path=/jdpiak-62/ 2022-09-12T19:34:00.192 DEBUG:boto:path=/jdpiak-62/?prefix=obj4 2022-09-12T19:34:00.193 DEBUG:boto:auth_path=/jdpiak-62/?prefix=obj4 2022-09-12T19:34:00.193 DEBUG:boto:Method: GET 2022-09-12T19:34:00.194 DEBUG:boto:Path: /jdpiak-62/?prefix=obj4 2022-09-12T19:34:00.195 DEBUG:boto:Data: 2022-09-12T19:34:00.195 DEBUG:boto:Headers: {} 2022-09-12T19:34:00.196 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:00.196 DEBUG:boto:Port: 8000 2022-09-12T19:34:00.197 DEBUG:boto:Params: {} 2022-09-12T19:34:00.197 DEBUG:boto:Token: None 2022-09-12T19:34:00.198 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:00 GMT /jdpiak-62/ 2022-09-12T19:34:00.198 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R4ZlWMuJQ1jiOBrGoS2bVrXkroo= 2022-09-12T19:34:00.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R4ZlWMuJQ1jiOBrGoS2bVrXkroo=', 'Content-Length': '0'} 2022-09-12T19:34:00.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a859caab1cacd04d-00631f89a8-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:00.202 DEBUG:boto:b'jdpiak-62obj41000falseobj42022-09-12T19:32:15.218Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:00.203 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-62 zones={test-zone3, test-zone2} 2022-09-12T19:34:00.204 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-63 2022-09-12T19:34:00.205 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:34:00.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:34:00.206 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:34:00.269 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:34:00.269 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:34:00.338 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:34:00.336+0000 7f8583129600 1 bucket sync caught up with source: 2022-09-12T19:34:00.339 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.65.5, , 00000000001.41.5, , 00000000001.55.5, , , 00000000001.46.5, , , ] 2022-09-12T19:34:00.339 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.65.5, , 00000000001.41.5, , 00000000001.55.5, , , 00000000001.46.5, , , ] 2022-09-12T19:34:00.340 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:34:00.336+0000 7f8583129600 0 bucket checkpoint complete 2022-09-12T19:34:00.358 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-63 2022-09-12T19:34:00.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:34:00.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:34:00.362 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:34:00.429 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:34:00.429 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:34:00.479 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:34:00.476+0000 7facac153600 1 bucket sync caught up with source: 2022-09-12T19:34:00.480 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [00000000001.65.5, , 00000000001.41.5, , 00000000001.55.5, , , 00000000001.46.5, , , ] 2022-09-12T19:34:00.480 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [00000000001.65.5, , 00000000001.41.5, , 00000000001.55.5, , , 00000000001.46.5, , , ] 2022-09-12T19:34:00.480 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:34:00.476+0000 7facac153600 0 bucket checkpoint complete 2022-09-12T19:34:00.500 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-63 2022-09-12T19:34:00.501 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:34:00.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:34:00.502 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:34:00.560 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:34:00.561 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:34:00.614 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:34:00.613+0000 7fdbfb5a7600 1 bucket sync caught up with source: 2022-09-12T19:34:00.615 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.65.5, , 00000000001.41.5, , 00000000001.55.5, , , 00000000001.46.5, , , ] 2022-09-12T19:34:00.616 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.65.5, , 00000000001.41.5, , 00000000001.55.5, , , 00000000001.46.5, , , ] 2022-09-12T19:34:00.616 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:34:00.613+0000 7fdbfb5a7600 0 bucket checkpoint complete 2022-09-12T19:34:00.635 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-63 2022-09-12T19:34:00.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:34:00.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:34:00.637 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:34:00.697 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:34:00.697 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:34:00.744 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:34:00.741+0000 7f7cc60cb600 1 bucket sync caught up with source: 2022-09-12T19:34:00.744 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [00000000001.893.5, , 00000000001.416.5, , 00000000001.1127.5, , , 00000000001.491.5, , , ] 2022-09-12T19:34:00.745 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [00000000001.893.5, , 00000000001.416.5, , 00000000001.1127.5, , , 00000000001.491.5, , , ] 2022-09-12T19:34:00.745 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:34:00.741+0000 7f7cc60cb600 0 bucket checkpoint complete 2022-09-12T19:34:00.761 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-63 2022-09-12T19:34:00.761 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:34:00.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:34:00.764 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:34:00.822 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:34:00.823 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:34:00.873 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:34:00.870+0000 7f2bc7899600 1 bucket sync caught up with source: 2022-09-12T19:34:00.874 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [00000000001.893.5, , 00000000001.416.5, , 00000000001.1127.5, , , 00000000001.491.5, , , ] 2022-09-12T19:34:00.875 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [00000000001.893.5, , 00000000001.416.5, , 00000000001.1127.5, , , 00000000001.491.5, , , ] 2022-09-12T19:34:00.875 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:34:00.870+0000 7f2bc7899600 0 bucket checkpoint complete 2022-09-12T19:34:00.891 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-63 2022-09-12T19:34:00.892 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-63 2022-09-12T19:34:00.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:34:00.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:34:00.895 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:34:00.955 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:34:00.956 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:34:01.004 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:34:01.002+0000 7f7e590d0600 1 bucket sync caught up with source: 2022-09-12T19:34:01.005 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [00000000001.408.5, , 00000000001.287.5, , 00000000001.444.5, , , 00000000001.299.5, , , ] 2022-09-12T19:34:01.005 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [00000000001.408.5, , 00000000001.287.5, , 00000000001.444.5, , , 00000000001.299.5, , , ] 2022-09-12T19:34:01.006 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:34:01.002+0000 7f7e590d0600 0 bucket checkpoint complete 2022-09-12T19:34:01.024 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-63 2022-09-12T19:34:01.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:34:01.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:34:01.027 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:34:01.085 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:34:01.086 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:34:01.132 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:34:01.130+0000 7fdc98708600 1 bucket sync caught up with source: 2022-09-12T19:34:01.132 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.408.5, , 00000000001.287.5, , 00000000001.444.5, , , 00000000001.299.5, , , ] 2022-09-12T19:34:01.132 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.408.5, , 00000000001.287.5, , 00000000001.444.5, , , 00000000001.299.5, , , ] 2022-09-12T19:34:01.133 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:34:01.130+0000 7fdc98708600 0 bucket checkpoint complete 2022-09-12T19:34:01.146 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-63 2022-09-12T19:34:01.147 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-63 zones={test-zone2, test-zone1} 2022-09-12T19:34:01.148 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:01.150 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:01.151 DEBUG:boto:Method: HEAD 2022-09-12T19:34:01.152 DEBUG:boto:Path: /jdpiak-63/ 2022-09-12T19:34:01.153 DEBUG:boto:Data: 2022-09-12T19:34:01.154 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.154 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:01.155 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.155 DEBUG:boto:Params: {} 2022-09-12T19:34:01.156 DEBUG:boto:Token: None 2022-09-12T19:34:01.157 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/ 2022-09-12T19:34:01.157 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TFouXccECnhLswoKqX2t3AHFIPs= 2022-09-12T19:34:01.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TFouXccECnhLswoKqX2t3AHFIPs=', 'Content-Length': '0'} 2022-09-12T19:34:01.161 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec2fd0edbdbfb18d-00631f89a9-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.162 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:01.163 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:01.164 DEBUG:boto:Method: HEAD 2022-09-12T19:34:01.165 DEBUG:boto:Path: /jdpiak-63/ 2022-09-12T19:34:01.165 DEBUG:boto:Data: 2022-09-12T19:34:01.166 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.167 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:01.167 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.168 DEBUG:boto:Params: {} 2022-09-12T19:34:01.168 DEBUG:boto:Token: None 2022-09-12T19:34:01.169 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/ 2022-09-12T19:34:01.169 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TFouXccECnhLswoKqX2t3AHFIPs= 2022-09-12T19:34:01.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TFouXccECnhLswoKqX2t3AHFIPs=', 'Content-Length': '0'} 2022-09-12T19:34:01.173 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce06619e51cfb722-00631f89a9-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.174 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:34:01.175 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:01.175 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:01.176 DEBUG:boto:path=/jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:01.177 DEBUG:boto:auth_path=/jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:01.177 DEBUG:boto:Method: GET 2022-09-12T19:34:01.178 DEBUG:boto:Path: /jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:01.179 DEBUG:boto:Data: 2022-09-12T19:34:01.179 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.180 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:01.180 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.181 DEBUG:boto:Params: {} 2022-09-12T19:34:01.181 DEBUG:boto:Token: None 2022-09-12T19:34:01.182 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/?versions 2022-09-12T19:34:01.183 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NcD8eutfM4eveCzvHQpJS7K5Xc8= 2022-09-12T19:34:01.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NcD8eutfM4eveCzvHQpJS7K5Xc8=', 'Content-Length': '0'} 2022-09-12T19:34:01.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f978c86eeb97f24-00631f89a9-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.189 DEBUG:boto:b'jdpiak-63999falseobj1nulltrue2022-09-12T19:32:15.261Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-12T19:32:15.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-12T19:32:15.326Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-12T19:32:15.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:01.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-12T19:34:01.192 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-12T19:34:01.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-12T19:34:01.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-12T19:34:01.194 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:34:01.195 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:01.196 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:01.197 DEBUG:boto:path=/jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:01.198 DEBUG:boto:auth_path=/jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:01.199 DEBUG:boto:Method: GET 2022-09-12T19:34:01.200 DEBUG:boto:Path: /jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:01.201 DEBUG:boto:Data: 2022-09-12T19:34:01.202 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.203 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:01.204 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.205 DEBUG:boto:Params: {} 2022-09-12T19:34:01.206 DEBUG:boto:Token: None 2022-09-12T19:34:01.208 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/?versions 2022-09-12T19:34:01.208 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NcD8eutfM4eveCzvHQpJS7K5Xc8= 2022-09-12T19:34:01.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NcD8eutfM4eveCzvHQpJS7K5Xc8=', 'Content-Length': '0'} 2022-09-12T19:34:01.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b37871be0307e033-00631f89a9-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.213 DEBUG:boto:b'jdpiak-63999falseobj1nulltrue2022-09-12T19:32:15.261Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-12T19:32:15.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-12T19:32:15.326Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-12T19:32:15.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:01.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-12T19:34:01.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-12T19:34:01.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-12T19:34:01.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-12T19:34:01.216 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:01.216 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:01.217 DEBUG:boto:path=/jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:01.218 DEBUG:boto:auth_path=/jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:01.219 DEBUG:boto:Method: GET 2022-09-12T19:34:01.219 DEBUG:boto:Path: /jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:01.220 DEBUG:boto:Data: 2022-09-12T19:34:01.221 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.221 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:01.222 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.222 DEBUG:boto:Params: {} 2022-09-12T19:34:01.223 DEBUG:boto:Token: None 2022-09-12T19:34:01.223 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/?versions 2022-09-12T19:34:01.224 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NcD8eutfM4eveCzvHQpJS7K5Xc8= 2022-09-12T19:34:01.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NcD8eutfM4eveCzvHQpJS7K5Xc8=', 'Content-Length': '0'} 2022-09-12T19:34:01.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002da1efe28e90f113-00631f89a9-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.228 DEBUG:boto:b'jdpiak-63999falseobj1nulltrue2022-09-12T19:32:15.261Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-12T19:32:15.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-12T19:32:15.326Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-12T19:32:15.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:01.229 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:01.230 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:01.231 DEBUG:boto:path=/jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:01.231 DEBUG:boto:auth_path=/jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:01.232 DEBUG:boto:Method: GET 2022-09-12T19:34:01.232 DEBUG:boto:Path: /jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:01.233 DEBUG:boto:Data: 2022-09-12T19:34:01.233 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.234 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:01.234 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.235 DEBUG:boto:Params: {} 2022-09-12T19:34:01.236 DEBUG:boto:Token: None 2022-09-12T19:34:01.236 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/?versions 2022-09-12T19:34:01.237 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NcD8eutfM4eveCzvHQpJS7K5Xc8= 2022-09-12T19:34:01.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NcD8eutfM4eveCzvHQpJS7K5Xc8=', 'Content-Length': '0'} 2022-09-12T19:34:01.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ef592b2bd0e728f-00631f89a9-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.242 DEBUG:boto:b'jdpiak-63999falseobj1nulltrue2022-09-12T19:32:15.261Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-12T19:32:15.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-12T19:32:15.326Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-12T19:32:15.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:01.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-12T19:34:01.243 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:01.244 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:01.245 DEBUG:boto:path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:01.245 DEBUG:boto:auth_path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:01.246 DEBUG:boto:Method: GET 2022-09-12T19:34:01.247 DEBUG:boto:Path: /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:01.247 DEBUG:boto:Data: 2022-09-12T19:34:01.248 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.249 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:01.249 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.250 DEBUG:boto:Params: {} 2022-09-12T19:34:01.251 DEBUG:boto:Token: None 2022-09-12T19:34:01.251 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:01.252 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OPeqscoTYP/3yzzWB+yaZALxRLE= 2022-09-12T19:34:01.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OPeqscoTYP/3yzzWB+yaZALxRLE=', 'Content-Length': '0'} 2022-09-12T19:34:01.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005db36a80e55e32ff-00631f89a9-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.255 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:01.256 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:01.257 DEBUG:boto:path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:01.257 DEBUG:boto:auth_path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:01.258 DEBUG:boto:Method: GET 2022-09-12T19:34:01.258 DEBUG:boto:Path: /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:01.259 DEBUG:boto:Data: 2022-09-12T19:34:01.259 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.260 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:01.260 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.261 DEBUG:boto:Params: {} 2022-09-12T19:34:01.262 DEBUG:boto:Token: None 2022-09-12T19:34:01.262 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:01.263 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OPeqscoTYP/3yzzWB+yaZALxRLE= 2022-09-12T19:34:01.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OPeqscoTYP/3yzzWB+yaZALxRLE=', 'Content-Length': '0'} 2022-09-12T19:34:01.265 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000872baffaedb0a5bf-00631f89a9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.266 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:01.267 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:01.267 DEBUG:boto:path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:01.268 DEBUG:boto:auth_path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:01.268 DEBUG:boto:Method: HEAD 2022-09-12T19:34:01.269 DEBUG:boto:Path: /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:01.269 DEBUG:boto:Data: 2022-09-12T19:34:01.270 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.270 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:01.271 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.271 DEBUG:boto:Params: {} 2022-09-12T19:34:01.272 DEBUG:boto:Token: None 2022-09-12T19:34:01.273 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:01.273 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:95Bm6OEVYt4PfsD8TUnbH8/BcjY= 2022-09-12T19:34:01.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:95Bm6OEVYt4PfsD8TUnbH8/BcjY=', 'Content-Length': '0'} 2022-09-12T19:34:01.276 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009938d03e5cfe9646-00631f89a9-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.277 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:01.278 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:01.278 DEBUG:boto:path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:01.279 DEBUG:boto:auth_path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:01.279 DEBUG:boto:Method: HEAD 2022-09-12T19:34:01.280 DEBUG:boto:Path: /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:01.280 DEBUG:boto:Data: 2022-09-12T19:34:01.281 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.281 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:01.282 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.282 DEBUG:boto:Params: {} 2022-09-12T19:34:01.283 DEBUG:boto:Token: None 2022-09-12T19:34:01.283 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:01.284 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:95Bm6OEVYt4PfsD8TUnbH8/BcjY= 2022-09-12T19:34:01.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:95Bm6OEVYt4PfsD8TUnbH8/BcjY=', 'Content-Length': '0'} 2022-09-12T19:34:01.286 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032628202f901d4d8-00631f89a9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-12T19:34:01.288 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:01.288 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:01.289 DEBUG:boto:path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:01.289 DEBUG:boto:auth_path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:01.290 DEBUG:boto:Method: GET 2022-09-12T19:34:01.290 DEBUG:boto:Path: /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:01.291 DEBUG:boto:Data: 2022-09-12T19:34:01.291 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.292 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:01.292 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.293 DEBUG:boto:Params: {} 2022-09-12T19:34:01.294 DEBUG:boto:Token: None 2022-09-12T19:34:01.294 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:01.295 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OPeqscoTYP/3yzzWB+yaZALxRLE= 2022-09-12T19:34:01.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OPeqscoTYP/3yzzWB+yaZALxRLE=', 'Content-Length': '0'} 2022-09-12T19:34:01.297 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf731d20bf31201b-00631f89a9-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.298 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:01.299 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:01.299 DEBUG:boto:path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:01.300 DEBUG:boto:auth_path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:01.300 DEBUG:boto:Method: GET 2022-09-12T19:34:01.301 DEBUG:boto:Path: /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:01.301 DEBUG:boto:Data: 2022-09-12T19:34:01.302 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.303 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:01.303 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.304 DEBUG:boto:Params: {} 2022-09-12T19:34:01.304 DEBUG:boto:Token: None 2022-09-12T19:34:01.305 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:01.305 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OPeqscoTYP/3yzzWB+yaZALxRLE= 2022-09-12T19:34:01.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OPeqscoTYP/3yzzWB+yaZALxRLE=', 'Content-Length': '0'} 2022-09-12T19:34:01.308 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000309cd989d0a9350d-00631f89a9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.350 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:01.351 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:01.351 DEBUG:boto:path=/jdpiak-63/?prefix=obj1 2022-09-12T19:34:01.352 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj1 2022-09-12T19:34:01.353 DEBUG:boto:Method: GET 2022-09-12T19:34:01.353 DEBUG:boto:Path: /jdpiak-63/?prefix=obj1 2022-09-12T19:34:01.354 DEBUG:boto:Data: 2022-09-12T19:34:01.355 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.356 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:01.357 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.358 DEBUG:boto:Params: {} 2022-09-12T19:34:01.359 DEBUG:boto:Token: None 2022-09-12T19:34:01.360 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/ 2022-09-12T19:34:01.361 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0cGkHr8GdT40zzpJhuNW3JVTbv0= 2022-09-12T19:34:01.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0cGkHr8GdT40zzpJhuNW3JVTbv0=', 'Content-Length': '0'} 2022-09-12T19:34:01.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bb73606639ca81c-00631f89a9-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.367 DEBUG:boto:b'jdpiak-63obj11000falseobj12022-09-12T19:32:15.261Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:01.368 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:01.369 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:01.370 DEBUG:boto:path=/jdpiak-63/?prefix=obj1 2022-09-12T19:34:01.371 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj1 2022-09-12T19:34:01.372 DEBUG:boto:Method: GET 2022-09-12T19:34:01.373 DEBUG:boto:Path: /jdpiak-63/?prefix=obj1 2022-09-12T19:34:01.374 DEBUG:boto:Data: 2022-09-12T19:34:01.376 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.376 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:01.377 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.378 DEBUG:boto:Params: {} 2022-09-12T19:34:01.380 DEBUG:boto:Token: None 2022-09-12T19:34:01.381 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/ 2022-09-12T19:34:01.382 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0cGkHr8GdT40zzpJhuNW3JVTbv0= 2022-09-12T19:34:01.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0cGkHr8GdT40zzpJhuNW3JVTbv0=', 'Content-Length': '0'} 2022-09-12T19:34:01.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f14924462cdb39b-00631f89a9-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.387 DEBUG:boto:b'jdpiak-63obj11000falseobj12022-09-12T19:32:15.261Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:01.388 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:01.389 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:01.390 DEBUG:boto:Method: HEAD 2022-09-12T19:34:01.391 DEBUG:boto:Path: /jdpiak-63/obj1 2022-09-12T19:34:01.391 DEBUG:boto:Data: 2022-09-12T19:34:01.392 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.393 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:01.393 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.394 DEBUG:boto:Params: {} 2022-09-12T19:34:01.394 DEBUG:boto:Token: None 2022-09-12T19:34:01.395 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj1 2022-09-12T19:34:01.396 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qniUqO48yVuTXYtEcjqaREpMSxk= 2022-09-12T19:34:01.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qniUqO48yVuTXYtEcjqaREpMSxk=', 'Content-Length': '0'} 2022-09-12T19:34:01.398 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e39ef371a2892eec-00631f89a9-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.399 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:01.399 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:01.400 DEBUG:boto:Method: HEAD 2022-09-12T19:34:01.400 DEBUG:boto:Path: /jdpiak-63/obj1 2022-09-12T19:34:01.401 DEBUG:boto:Data: 2022-09-12T19:34:01.401 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.402 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:01.402 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.403 DEBUG:boto:Params: {} 2022-09-12T19:34:01.403 DEBUG:boto:Token: None 2022-09-12T19:34:01.404 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj1 2022-09-12T19:34:01.405 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qniUqO48yVuTXYtEcjqaREpMSxk= 2022-09-12T19:34:01.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qniUqO48yVuTXYtEcjqaREpMSxk=', 'Content-Length': '0'} 2022-09-12T19:34:01.407 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007fad64e1a2c09a89-00631f89a9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-12T19:34:01.409 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:01.410 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:01.410 DEBUG:boto:Method: GET 2022-09-12T19:34:01.411 DEBUG:boto:Path: /jdpiak-63/obj1 2022-09-12T19:34:01.412 DEBUG:boto:Data: 2022-09-12T19:34:01.413 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.413 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:01.414 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.414 DEBUG:boto:Params: {} 2022-09-12T19:34:01.415 DEBUG:boto:Token: None 2022-09-12T19:34:01.416 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj1 2022-09-12T19:34:01.416 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QeXx0TgQErxY3O40xroHicNpJnM= 2022-09-12T19:34:01.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QeXx0TgQErxY3O40xroHicNpJnM=', 'Content-Length': '0'} 2022-09-12T19:34:01.419 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003dac4873fce11016-00631f89a9-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.466 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:01.467 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:01.467 DEBUG:boto:Method: GET 2022-09-12T19:34:01.468 DEBUG:boto:Path: /jdpiak-63/obj1 2022-09-12T19:34:01.468 DEBUG:boto:Data: 2022-09-12T19:34:01.469 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.470 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:01.471 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.471 DEBUG:boto:Params: {} 2022-09-12T19:34:01.472 DEBUG:boto:Token: None 2022-09-12T19:34:01.472 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj1 2022-09-12T19:34:01.473 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QeXx0TgQErxY3O40xroHicNpJnM= 2022-09-12T19:34:01.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QeXx0TgQErxY3O40xroHicNpJnM=', 'Content-Length': '0'} 2022-09-12T19:34:01.476 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a2735597fa38b07-00631f89a9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.477 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:01.477 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:01.478 DEBUG:boto:path=/jdpiak-63/?prefix=obj1 2022-09-12T19:34:01.478 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj1 2022-09-12T19:34:01.479 DEBUG:boto:Method: GET 2022-09-12T19:34:01.480 DEBUG:boto:Path: /jdpiak-63/?prefix=obj1 2022-09-12T19:34:01.480 DEBUG:boto:Data: 2022-09-12T19:34:01.481 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.482 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:01.483 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.483 DEBUG:boto:Params: {} 2022-09-12T19:34:01.484 DEBUG:boto:Token: None 2022-09-12T19:34:01.484 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/ 2022-09-12T19:34:01.485 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0cGkHr8GdT40zzpJhuNW3JVTbv0= 2022-09-12T19:34:01.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0cGkHr8GdT40zzpJhuNW3JVTbv0=', 'Content-Length': '0'} 2022-09-12T19:34:01.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd843e7334329b98-00631f89a9-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.489 DEBUG:boto:b'jdpiak-63obj11000falseobj12022-09-12T19:32:15.261Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:01.490 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:01.491 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:01.491 DEBUG:boto:path=/jdpiak-63/?prefix=obj1 2022-09-12T19:34:01.492 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj1 2022-09-12T19:34:01.493 DEBUG:boto:Method: GET 2022-09-12T19:34:01.493 DEBUG:boto:Path: /jdpiak-63/?prefix=obj1 2022-09-12T19:34:01.494 DEBUG:boto:Data: 2022-09-12T19:34:01.495 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.496 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:01.496 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.497 DEBUG:boto:Params: {} 2022-09-12T19:34:01.498 DEBUG:boto:Token: None 2022-09-12T19:34:01.498 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/ 2022-09-12T19:34:01.499 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0cGkHr8GdT40zzpJhuNW3JVTbv0= 2022-09-12T19:34:01.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0cGkHr8GdT40zzpJhuNW3JVTbv0=', 'Content-Length': '0'} 2022-09-12T19:34:01.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2c53e8e9c0acbd8-00631f89a9-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.504 DEBUG:boto:b'jdpiak-63obj11000falseobj12022-09-12T19:32:15.261Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:01.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-12T19:34:01.506 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:01.507 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:01.508 DEBUG:boto:path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:01.508 DEBUG:boto:auth_path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:01.509 DEBUG:boto:Method: GET 2022-09-12T19:34:01.510 DEBUG:boto:Path: /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:01.510 DEBUG:boto:Data: 2022-09-12T19:34:01.511 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.511 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:01.512 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.513 DEBUG:boto:Params: {} 2022-09-12T19:34:01.514 DEBUG:boto:Token: None 2022-09-12T19:34:01.515 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:01.515 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MXP+gFTrakc9KHkRdP0Xk9yVX2I= 2022-09-12T19:34:01.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MXP+gFTrakc9KHkRdP0Xk9yVX2I=', 'Content-Length': '0'} 2022-09-12T19:34:01.519 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008bc58b007d3dca8e-00631f89a9-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.520 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:01.520 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:01.521 DEBUG:boto:path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:01.521 DEBUG:boto:auth_path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:01.522 DEBUG:boto:Method: GET 2022-09-12T19:34:01.523 DEBUG:boto:Path: /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:01.523 DEBUG:boto:Data: 2022-09-12T19:34:01.524 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.525 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:01.526 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.526 DEBUG:boto:Params: {} 2022-09-12T19:34:01.527 DEBUG:boto:Token: None 2022-09-12T19:34:01.528 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:01.529 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MXP+gFTrakc9KHkRdP0Xk9yVX2I= 2022-09-12T19:34:01.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MXP+gFTrakc9KHkRdP0Xk9yVX2I=', 'Content-Length': '0'} 2022-09-12T19:34:01.533 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0cead3f1e68b7e1-00631f89a9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.574 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:01.614 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:01.615 DEBUG:boto:path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:01.616 DEBUG:boto:auth_path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:01.617 DEBUG:boto:Method: HEAD 2022-09-12T19:34:01.618 DEBUG:boto:Path: /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:01.619 DEBUG:boto:Data: 2022-09-12T19:34:01.619 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.620 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:01.620 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.621 DEBUG:boto:Params: {} 2022-09-12T19:34:01.621 DEBUG:boto:Token: None 2022-09-12T19:34:01.622 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:01.623 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TqcQ01O7U7Zl9Ppv3YllIYUpzRs= 2022-09-12T19:34:01.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TqcQ01O7U7Zl9Ppv3YllIYUpzRs=', 'Content-Length': '0'} 2022-09-12T19:34:01.628 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3edddd97b4bd73e-00631f89a9-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.629 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:01.630 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:01.630 DEBUG:boto:path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:01.631 DEBUG:boto:auth_path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:01.631 DEBUG:boto:Method: HEAD 2022-09-12T19:34:01.632 DEBUG:boto:Path: /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:01.633 DEBUG:boto:Data: 2022-09-12T19:34:01.633 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.634 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:01.635 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.635 DEBUG:boto:Params: {} 2022-09-12T19:34:01.636 DEBUG:boto:Token: None 2022-09-12T19:34:01.637 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:01.637 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TqcQ01O7U7Zl9Ppv3YllIYUpzRs= 2022-09-12T19:34:01.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TqcQ01O7U7Zl9Ppv3YllIYUpzRs=', 'Content-Length': '0'} 2022-09-12T19:34:01.641 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee8504ef03fd80e1-00631f89a9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-12T19:34:01.643 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:01.644 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:01.644 DEBUG:boto:path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:01.645 DEBUG:boto:auth_path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:01.645 DEBUG:boto:Method: GET 2022-09-12T19:34:01.646 DEBUG:boto:Path: /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:01.647 DEBUG:boto:Data: 2022-09-12T19:34:01.647 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.648 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:01.648 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.649 DEBUG:boto:Params: {} 2022-09-12T19:34:01.650 DEBUG:boto:Token: None 2022-09-12T19:34:01.650 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:01.651 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MXP+gFTrakc9KHkRdP0Xk9yVX2I= 2022-09-12T19:34:01.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MXP+gFTrakc9KHkRdP0Xk9yVX2I=', 'Content-Length': '0'} 2022-09-12T19:34:01.654 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077632c6b42679c47-00631f89a9-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.655 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:01.655 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:01.656 DEBUG:boto:path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:01.656 DEBUG:boto:auth_path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:01.657 DEBUG:boto:Method: GET 2022-09-12T19:34:01.658 DEBUG:boto:Path: /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:01.658 DEBUG:boto:Data: 2022-09-12T19:34:01.659 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.659 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:01.660 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.660 DEBUG:boto:Params: {} 2022-09-12T19:34:01.661 DEBUG:boto:Token: None 2022-09-12T19:34:01.662 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:01.662 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MXP+gFTrakc9KHkRdP0Xk9yVX2I= 2022-09-12T19:34:01.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MXP+gFTrakc9KHkRdP0Xk9yVX2I=', 'Content-Length': '0'} 2022-09-12T19:34:01.665 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9f471f556fc5d65-00631f89a9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.666 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:01.667 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:01.667 DEBUG:boto:path=/jdpiak-63/?prefix=obj2 2022-09-12T19:34:01.668 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj2 2022-09-12T19:34:01.668 DEBUG:boto:Method: GET 2022-09-12T19:34:01.669 DEBUG:boto:Path: /jdpiak-63/?prefix=obj2 2022-09-12T19:34:01.670 DEBUG:boto:Data: 2022-09-12T19:34:01.670 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.671 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:01.672 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.672 DEBUG:boto:Params: {} 2022-09-12T19:34:01.673 DEBUG:boto:Token: None 2022-09-12T19:34:01.673 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/ 2022-09-12T19:34:01.674 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0cGkHr8GdT40zzpJhuNW3JVTbv0= 2022-09-12T19:34:01.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0cGkHr8GdT40zzpJhuNW3JVTbv0=', 'Content-Length': '0'} 2022-09-12T19:34:01.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4e0ca9850aa66e6-00631f89a9-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.718 DEBUG:boto:b'jdpiak-63obj21000falseobj22022-09-12T19:32:15.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:01.719 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:01.720 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:01.721 DEBUG:boto:path=/jdpiak-63/?prefix=obj2 2022-09-12T19:34:01.722 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj2 2022-09-12T19:34:01.723 DEBUG:boto:Method: GET 2022-09-12T19:34:01.723 DEBUG:boto:Path: /jdpiak-63/?prefix=obj2 2022-09-12T19:34:01.724 DEBUG:boto:Data: 2022-09-12T19:34:01.725 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.725 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:01.726 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.727 DEBUG:boto:Params: {} 2022-09-12T19:34:01.727 DEBUG:boto:Token: None 2022-09-12T19:34:01.728 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/ 2022-09-12T19:34:01.729 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0cGkHr8GdT40zzpJhuNW3JVTbv0= 2022-09-12T19:34:01.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0cGkHr8GdT40zzpJhuNW3JVTbv0=', 'Content-Length': '0'} 2022-09-12T19:34:01.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000674aa7190f1a194-00631f89a9-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.734 DEBUG:boto:b'jdpiak-63obj21000falseobj22022-09-12T19:32:15.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:01.735 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:01.736 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:01.736 DEBUG:boto:Method: HEAD 2022-09-12T19:34:01.737 DEBUG:boto:Path: /jdpiak-63/obj2 2022-09-12T19:34:01.738 DEBUG:boto:Data: 2022-09-12T19:34:01.739 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.740 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:01.741 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.741 DEBUG:boto:Params: {} 2022-09-12T19:34:01.742 DEBUG:boto:Token: None 2022-09-12T19:34:01.743 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj2 2022-09-12T19:34:01.743 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:49OUhteVt+OJfgj0QB45AHhBVlk= 2022-09-12T19:34:01.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:49OUhteVt+OJfgj0QB45AHhBVlk=', 'Content-Length': '0'} 2022-09-12T19:34:01.746 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000955d65c1f8155d1-00631f89a9-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.748 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:01.749 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:01.750 DEBUG:boto:Method: HEAD 2022-09-12T19:34:01.751 DEBUG:boto:Path: /jdpiak-63/obj2 2022-09-12T19:34:01.752 DEBUG:boto:Data: 2022-09-12T19:34:01.753 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.754 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:01.755 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.756 DEBUG:boto:Params: {} 2022-09-12T19:34:01.757 DEBUG:boto:Token: None 2022-09-12T19:34:01.758 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj2 2022-09-12T19:34:01.759 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:49OUhteVt+OJfgj0QB45AHhBVlk= 2022-09-12T19:34:01.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:49OUhteVt+OJfgj0QB45AHhBVlk=', 'Content-Length': '0'} 2022-09-12T19:34:01.763 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f16fb095f1b20d8-00631f89a9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-12T19:34:01.765 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:01.766 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:01.767 DEBUG:boto:Method: GET 2022-09-12T19:34:01.768 DEBUG:boto:Path: /jdpiak-63/obj2 2022-09-12T19:34:01.769 DEBUG:boto:Data: 2022-09-12T19:34:01.770 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.771 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:01.772 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.773 DEBUG:boto:Params: {} 2022-09-12T19:34:01.774 DEBUG:boto:Token: None 2022-09-12T19:34:01.775 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj2 2022-09-12T19:34:01.776 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xPa18CYB0CPHmtZOUWrgabWcUO4= 2022-09-12T19:34:01.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xPa18CYB0CPHmtZOUWrgabWcUO4=', 'Content-Length': '0'} 2022-09-12T19:34:01.779 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077397ee76290e7f4-00631f89a9-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.781 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:01.782 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:01.783 DEBUG:boto:Method: GET 2022-09-12T19:34:01.783 DEBUG:boto:Path: /jdpiak-63/obj2 2022-09-12T19:34:01.784 DEBUG:boto:Data: 2022-09-12T19:34:01.784 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.785 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:01.785 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.786 DEBUG:boto:Params: {} 2022-09-12T19:34:01.787 DEBUG:boto:Token: None 2022-09-12T19:34:01.787 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj2 2022-09-12T19:34:01.788 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xPa18CYB0CPHmtZOUWrgabWcUO4= 2022-09-12T19:34:01.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xPa18CYB0CPHmtZOUWrgabWcUO4=', 'Content-Length': '0'} 2022-09-12T19:34:01.803 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b6cd900a8823cb3b-00631f89a9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.804 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:01.805 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:01.806 DEBUG:boto:path=/jdpiak-63/?prefix=obj2 2022-09-12T19:34:01.806 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj2 2022-09-12T19:34:01.807 DEBUG:boto:Method: GET 2022-09-12T19:34:01.808 DEBUG:boto:Path: /jdpiak-63/?prefix=obj2 2022-09-12T19:34:01.809 DEBUG:boto:Data: 2022-09-12T19:34:01.809 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.810 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:01.810 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.811 DEBUG:boto:Params: {} 2022-09-12T19:34:01.811 DEBUG:boto:Token: None 2022-09-12T19:34:01.812 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/ 2022-09-12T19:34:01.813 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0cGkHr8GdT40zzpJhuNW3JVTbv0= 2022-09-12T19:34:01.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0cGkHr8GdT40zzpJhuNW3JVTbv0=', 'Content-Length': '0'} 2022-09-12T19:34:01.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054d0e112678ee471-00631f89a9-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.818 DEBUG:boto:b'jdpiak-63obj21000falseobj22022-09-12T19:32:15.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:01.820 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:01.821 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:01.823 DEBUG:boto:path=/jdpiak-63/?prefix=obj2 2022-09-12T19:34:01.824 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj2 2022-09-12T19:34:01.825 DEBUG:boto:Method: GET 2022-09-12T19:34:01.827 DEBUG:boto:Path: /jdpiak-63/?prefix=obj2 2022-09-12T19:34:01.828 DEBUG:boto:Data: 2022-09-12T19:34:01.829 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.830 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:01.831 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.832 DEBUG:boto:Params: {} 2022-09-12T19:34:01.833 DEBUG:boto:Token: None 2022-09-12T19:34:01.834 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/ 2022-09-12T19:34:01.835 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0cGkHr8GdT40zzpJhuNW3JVTbv0= 2022-09-12T19:34:01.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0cGkHr8GdT40zzpJhuNW3JVTbv0=', 'Content-Length': '0'} 2022-09-12T19:34:01.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d8298e6dc00d4e1-00631f89a9-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.841 DEBUG:boto:b'jdpiak-63obj21000falseobj22022-09-12T19:32:15.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:01.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-12T19:34:01.844 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:01.845 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:01.846 DEBUG:boto:path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:01.847 DEBUG:boto:auth_path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:01.848 DEBUG:boto:Method: GET 2022-09-12T19:34:01.849 DEBUG:boto:Path: /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:01.850 DEBUG:boto:Data: 2022-09-12T19:34:01.851 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.852 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:01.853 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.854 DEBUG:boto:Params: {} 2022-09-12T19:34:01.855 DEBUG:boto:Token: None 2022-09-12T19:34:01.857 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:01.858 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:C3OUlHjTFCGh4Cz4W/FGeG8/qWA= 2022-09-12T19:34:01.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:C3OUlHjTFCGh4Cz4W/FGeG8/qWA=', 'Content-Length': '0'} 2022-09-12T19:34:01.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000edfa990d86c5eca4-00631f89a9-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.862 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:01.862 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:01.863 DEBUG:boto:path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:01.863 DEBUG:boto:auth_path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:01.864 DEBUG:boto:Method: GET 2022-09-12T19:34:01.865 DEBUG:boto:Path: /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:01.865 DEBUG:boto:Data: 2022-09-12T19:34:01.866 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.867 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:01.868 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.868 DEBUG:boto:Params: {} 2022-09-12T19:34:01.869 DEBUG:boto:Token: None 2022-09-12T19:34:01.870 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:01.870 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:C3OUlHjTFCGh4Cz4W/FGeG8/qWA= 2022-09-12T19:34:01.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:C3OUlHjTFCGh4Cz4W/FGeG8/qWA=', 'Content-Length': '0'} 2022-09-12T19:34:01.873 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a14fc8147474c46e-00631f89a9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.874 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:01.875 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:01.875 DEBUG:boto:path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:01.878 DEBUG:boto:auth_path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:01.880 DEBUG:boto:Method: HEAD 2022-09-12T19:34:01.881 DEBUG:boto:Path: /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:01.882 DEBUG:boto:Data: 2022-09-12T19:34:01.883 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.884 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:01.885 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.886 DEBUG:boto:Params: {} 2022-09-12T19:34:01.887 DEBUG:boto:Token: None 2022-09-12T19:34:01.888 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:01.890 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UWjnJYd4OWhYaAKvlknpxXvHdMo= 2022-09-12T19:34:01.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UWjnJYd4OWhYaAKvlknpxXvHdMo=', 'Content-Length': '0'} 2022-09-12T19:34:01.894 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008407c7d425839240-00631f89a9-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.895 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:01.896 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:01.897 DEBUG:boto:path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:01.899 DEBUG:boto:auth_path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:01.900 DEBUG:boto:Method: HEAD 2022-09-12T19:34:01.900 DEBUG:boto:Path: /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:01.902 DEBUG:boto:Data: 2022-09-12T19:34:01.902 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.903 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:01.903 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.904 DEBUG:boto:Params: {} 2022-09-12T19:34:01.904 DEBUG:boto:Token: None 2022-09-12T19:34:01.905 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:01.906 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UWjnJYd4OWhYaAKvlknpxXvHdMo= 2022-09-12T19:34:01.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UWjnJYd4OWhYaAKvlknpxXvHdMo=', 'Content-Length': '0'} 2022-09-12T19:34:01.908 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ab7dd408f5fbc43-00631f89a9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-12T19:34:01.910 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:01.911 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:01.912 DEBUG:boto:path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:01.912 DEBUG:boto:auth_path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:01.913 DEBUG:boto:Method: GET 2022-09-12T19:34:01.913 DEBUG:boto:Path: /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:01.914 DEBUG:boto:Data: 2022-09-12T19:34:01.914 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.915 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:01.915 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.916 DEBUG:boto:Params: {} 2022-09-12T19:34:01.916 DEBUG:boto:Token: None 2022-09-12T19:34:01.917 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:01.918 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:C3OUlHjTFCGh4Cz4W/FGeG8/qWA= 2022-09-12T19:34:01.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:C3OUlHjTFCGh4Cz4W/FGeG8/qWA=', 'Content-Length': '0'} 2022-09-12T19:34:01.922 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ddd287a8da5674f-00631f89a9-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.923 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:01.924 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:01.924 DEBUG:boto:path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:01.925 DEBUG:boto:auth_path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:01.925 DEBUG:boto:Method: GET 2022-09-12T19:34:01.926 DEBUG:boto:Path: /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:01.926 DEBUG:boto:Data: 2022-09-12T19:34:01.927 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.927 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:01.928 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.928 DEBUG:boto:Params: {} 2022-09-12T19:34:01.929 DEBUG:boto:Token: None 2022-09-12T19:34:01.930 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:01.931 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:C3OUlHjTFCGh4Cz4W/FGeG8/qWA= 2022-09-12T19:34:01.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:C3OUlHjTFCGh4Cz4W/FGeG8/qWA=', 'Content-Length': '0'} 2022-09-12T19:34:01.934 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c4485ba3c3f833e-00631f89a9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.935 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:01.935 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:01.936 DEBUG:boto:path=/jdpiak-63/?prefix=obj3 2022-09-12T19:34:01.936 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj3 2022-09-12T19:34:01.937 DEBUG:boto:Method: GET 2022-09-12T19:34:01.937 DEBUG:boto:Path: /jdpiak-63/?prefix=obj3 2022-09-12T19:34:01.938 DEBUG:boto:Data: 2022-09-12T19:34:01.938 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.939 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:01.939 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.940 DEBUG:boto:Params: {} 2022-09-12T19:34:01.941 DEBUG:boto:Token: None 2022-09-12T19:34:01.941 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/ 2022-09-12T19:34:01.942 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0cGkHr8GdT40zzpJhuNW3JVTbv0= 2022-09-12T19:34:01.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0cGkHr8GdT40zzpJhuNW3JVTbv0=', 'Content-Length': '0'} 2022-09-12T19:34:01.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077b2f79b2e71e879-00631f89a9-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.946 DEBUG:boto:b'jdpiak-63obj31000falseobj32022-09-12T19:32:15.326Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:01.947 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:01.947 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:01.948 DEBUG:boto:path=/jdpiak-63/?prefix=obj3 2022-09-12T19:34:01.948 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj3 2022-09-12T19:34:01.949 DEBUG:boto:Method: GET 2022-09-12T19:34:01.949 DEBUG:boto:Path: /jdpiak-63/?prefix=obj3 2022-09-12T19:34:01.950 DEBUG:boto:Data: 2022-09-12T19:34:01.950 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.951 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:01.952 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.952 DEBUG:boto:Params: {} 2022-09-12T19:34:01.953 DEBUG:boto:Token: None 2022-09-12T19:34:01.953 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/ 2022-09-12T19:34:01.954 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0cGkHr8GdT40zzpJhuNW3JVTbv0= 2022-09-12T19:34:01.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0cGkHr8GdT40zzpJhuNW3JVTbv0=', 'Content-Length': '0'} 2022-09-12T19:34:01.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fd3fd930e09d760-00631f89a9-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.958 DEBUG:boto:b'jdpiak-63obj31000falseobj32022-09-12T19:32:15.326Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:01.959 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:01.960 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:01.960 DEBUG:boto:Method: HEAD 2022-09-12T19:34:01.961 DEBUG:boto:Path: /jdpiak-63/obj3 2022-09-12T19:34:01.961 DEBUG:boto:Data: 2022-09-12T19:34:01.962 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.962 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:01.963 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.963 DEBUG:boto:Params: {} 2022-09-12T19:34:01.964 DEBUG:boto:Token: None 2022-09-12T19:34:01.964 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj3 2022-09-12T19:34:01.965 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FupLINASgdHJmTILI/yYgVOvrt8= 2022-09-12T19:34:01.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FupLINASgdHJmTILI/yYgVOvrt8=', 'Content-Length': '0'} 2022-09-12T19:34:01.967 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041cabe0bc3dc1be7-00631f89a9-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.968 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:01.968 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:01.969 DEBUG:boto:Method: HEAD 2022-09-12T19:34:01.970 DEBUG:boto:Path: /jdpiak-63/obj3 2022-09-12T19:34:01.970 DEBUG:boto:Data: 2022-09-12T19:34:01.971 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.971 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:01.971 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.972 DEBUG:boto:Params: {} 2022-09-12T19:34:01.973 DEBUG:boto:Token: None 2022-09-12T19:34:01.973 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj3 2022-09-12T19:34:01.974 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FupLINASgdHJmTILI/yYgVOvrt8= 2022-09-12T19:34:01.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FupLINASgdHJmTILI/yYgVOvrt8=', 'Content-Length': '0'} 2022-09-12T19:34:01.976 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6833959c84b1ef3-00631f89a9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-12T19:34:01.978 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:01.978 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:01.979 DEBUG:boto:Method: GET 2022-09-12T19:34:01.979 DEBUG:boto:Path: /jdpiak-63/obj3 2022-09-12T19:34:01.980 DEBUG:boto:Data: 2022-09-12T19:34:01.980 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.981 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:01.981 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.982 DEBUG:boto:Params: {} 2022-09-12T19:34:01.982 DEBUG:boto:Token: None 2022-09-12T19:34:01.983 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj3 2022-09-12T19:34:01.984 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0YDPHZbh6ML1K04QBv7225PhFf8= 2022-09-12T19:34:01.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0YDPHZbh6ML1K04QBv7225PhFf8=', 'Content-Length': '0'} 2022-09-12T19:34:01.986 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000438511eb9ca3fc6d-00631f89a9-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.987 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:01.987 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:01.988 DEBUG:boto:Method: GET 2022-09-12T19:34:01.989 DEBUG:boto:Path: /jdpiak-63/obj3 2022-09-12T19:34:01.989 DEBUG:boto:Data: 2022-09-12T19:34:01.990 DEBUG:boto:Headers: {} 2022-09-12T19:34:01.990 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:01.991 DEBUG:boto:Port: 8000 2022-09-12T19:34:01.991 DEBUG:boto:Params: {} 2022-09-12T19:34:01.992 DEBUG:boto:Token: None 2022-09-12T19:34:01.992 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:01 GMT /jdpiak-63/obj3 2022-09-12T19:34:01.993 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0YDPHZbh6ML1K04QBv7225PhFf8= 2022-09-12T19:34:01.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0YDPHZbh6ML1K04QBv7225PhFf8=', 'Content-Length': '0'} 2022-09-12T19:34:01.996 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088c1634ffdbafcf1-00631f89a9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:01.997 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:01.998 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:01.998 DEBUG:boto:path=/jdpiak-63/?prefix=obj3 2022-09-12T19:34:01.999 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj3 2022-09-12T19:34:01.999 DEBUG:boto:Method: GET 2022-09-12T19:34:02.000 DEBUG:boto:Path: /jdpiak-63/?prefix=obj3 2022-09-12T19:34:02.000 DEBUG:boto:Data: 2022-09-12T19:34:02.001 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.001 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:02.002 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.002 DEBUG:boto:Params: {} 2022-09-12T19:34:02.003 DEBUG:boto:Token: None 2022-09-12T19:34:02.003 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/ 2022-09-12T19:34:02.004 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo= 2022-09-12T19:34:02.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo=', 'Content-Length': '0'} 2022-09-12T19:34:02.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000946b4c044eecaa95-00631f89aa-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.054 DEBUG:boto:b'jdpiak-63obj31000falseobj32022-09-12T19:32:15.326Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:02.057 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:02.058 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:02.059 DEBUG:boto:path=/jdpiak-63/?prefix=obj3 2022-09-12T19:34:02.059 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj3 2022-09-12T19:34:02.060 DEBUG:boto:Method: GET 2022-09-12T19:34:02.060 DEBUG:boto:Path: /jdpiak-63/?prefix=obj3 2022-09-12T19:34:02.061 DEBUG:boto:Data: 2022-09-12T19:34:02.061 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.062 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.063 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.064 DEBUG:boto:Params: {} 2022-09-12T19:34:02.065 DEBUG:boto:Token: None 2022-09-12T19:34:02.067 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/ 2022-09-12T19:34:02.068 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo= 2022-09-12T19:34:02.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo=', 'Content-Length': '0'} 2022-09-12T19:34:02.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa2fa8e895388876-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.074 DEBUG:boto:b'jdpiak-63obj31000falseobj32022-09-12T19:32:15.326Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:02.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-12T19:34:02.078 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:02.079 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:02.080 DEBUG:boto:path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:02.081 DEBUG:boto:auth_path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:02.082 DEBUG:boto:Method: GET 2022-09-12T19:34:02.083 DEBUG:boto:Path: /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:02.084 DEBUG:boto:Data: 2022-09-12T19:34:02.085 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.086 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:02.087 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.088 DEBUG:boto:Params: {} 2022-09-12T19:34:02.089 DEBUG:boto:Token: None 2022-09-12T19:34:02.090 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:02.091 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RjoMIyghjxtXxCQGwyaJFaAZl00= 2022-09-12T19:34:02.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RjoMIyghjxtXxCQGwyaJFaAZl00=', 'Content-Length': '0'} 2022-09-12T19:34:02.095 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003692bbc75c821971-00631f89aa-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.097 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:02.098 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:02.099 DEBUG:boto:path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:02.100 DEBUG:boto:auth_path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:02.101 DEBUG:boto:Method: GET 2022-09-12T19:34:02.102 DEBUG:boto:Path: /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:02.103 DEBUG:boto:Data: 2022-09-12T19:34:02.104 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.104 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.105 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.106 DEBUG:boto:Params: {} 2022-09-12T19:34:02.107 DEBUG:boto:Token: None 2022-09-12T19:34:02.109 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:02.110 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RjoMIyghjxtXxCQGwyaJFaAZl00= 2022-09-12T19:34:02.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RjoMIyghjxtXxCQGwyaJFaAZl00=', 'Content-Length': '0'} 2022-09-12T19:34:02.114 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a60c5c4ad2cf848b-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.115 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:02.116 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:02.117 DEBUG:boto:path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:02.118 DEBUG:boto:auth_path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:02.119 DEBUG:boto:Method: HEAD 2022-09-12T19:34:02.120 DEBUG:boto:Path: /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:02.120 DEBUG:boto:Data: 2022-09-12T19:34:02.121 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.121 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:02.122 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.122 DEBUG:boto:Params: {} 2022-09-12T19:34:02.123 DEBUG:boto:Token: None 2022-09-12T19:34:02.123 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:02.124 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P0rM/tPn9yvQ7eTIi5LhFZyGJCg= 2022-09-12T19:34:02.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P0rM/tPn9yvQ7eTIi5LhFZyGJCg=', 'Content-Length': '0'} 2022-09-12T19:34:02.126 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a4aec4b70945389-00631f89aa-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.127 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:02.127 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:02.128 DEBUG:boto:path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:02.128 DEBUG:boto:auth_path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:02.129 DEBUG:boto:Method: HEAD 2022-09-12T19:34:02.129 DEBUG:boto:Path: /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:02.130 DEBUG:boto:Data: 2022-09-12T19:34:02.130 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.131 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.131 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.132 DEBUG:boto:Params: {} 2022-09-12T19:34:02.132 DEBUG:boto:Token: None 2022-09-12T19:34:02.133 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:02.134 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P0rM/tPn9yvQ7eTIi5LhFZyGJCg= 2022-09-12T19:34:02.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P0rM/tPn9yvQ7eTIi5LhFZyGJCg=', 'Content-Length': '0'} 2022-09-12T19:34:02.136 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000bcaf167c2320c29-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-12T19:34:02.137 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:02.138 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:02.138 DEBUG:boto:path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:02.139 DEBUG:boto:auth_path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:02.139 DEBUG:boto:Method: GET 2022-09-12T19:34:02.140 DEBUG:boto:Path: /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:02.141 DEBUG:boto:Data: 2022-09-12T19:34:02.141 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.142 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:02.142 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.143 DEBUG:boto:Params: {} 2022-09-12T19:34:02.143 DEBUG:boto:Token: None 2022-09-12T19:34:02.144 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:02.145 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RjoMIyghjxtXxCQGwyaJFaAZl00= 2022-09-12T19:34:02.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RjoMIyghjxtXxCQGwyaJFaAZl00=', 'Content-Length': '0'} 2022-09-12T19:34:02.147 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089f0f9a3af692ccb-00631f89aa-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.148 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:02.149 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:02.149 DEBUG:boto:path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:02.150 DEBUG:boto:auth_path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:02.150 DEBUG:boto:Method: GET 2022-09-12T19:34:02.151 DEBUG:boto:Path: /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:02.151 DEBUG:boto:Data: 2022-09-12T19:34:02.153 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.153 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.154 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.154 DEBUG:boto:Params: {} 2022-09-12T19:34:02.155 DEBUG:boto:Token: None 2022-09-12T19:34:02.156 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:02.156 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RjoMIyghjxtXxCQGwyaJFaAZl00= 2022-09-12T19:34:02.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RjoMIyghjxtXxCQGwyaJFaAZl00=', 'Content-Length': '0'} 2022-09-12T19:34:02.159 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094c4979b9a6ac996-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.202 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:02.203 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:02.204 DEBUG:boto:path=/jdpiak-63/?prefix=obj4 2022-09-12T19:34:02.205 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj4 2022-09-12T19:34:02.206 DEBUG:boto:Method: GET 2022-09-12T19:34:02.207 DEBUG:boto:Path: /jdpiak-63/?prefix=obj4 2022-09-12T19:34:02.208 DEBUG:boto:Data: 2022-09-12T19:34:02.209 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.210 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:02.211 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.212 DEBUG:boto:Params: {} 2022-09-12T19:34:02.213 DEBUG:boto:Token: None 2022-09-12T19:34:02.214 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/ 2022-09-12T19:34:02.215 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo= 2022-09-12T19:34:02.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo=', 'Content-Length': '0'} 2022-09-12T19:34:02.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c937975d9c92f21-00631f89aa-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.220 DEBUG:boto:b'jdpiak-63obj41000falseobj42022-09-12T19:32:15.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:02.222 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:02.223 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:02.224 DEBUG:boto:path=/jdpiak-63/?prefix=obj4 2022-09-12T19:34:02.225 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj4 2022-09-12T19:34:02.226 DEBUG:boto:Method: GET 2022-09-12T19:34:02.227 DEBUG:boto:Path: /jdpiak-63/?prefix=obj4 2022-09-12T19:34:02.228 DEBUG:boto:Data: 2022-09-12T19:34:02.228 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.229 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.230 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.231 DEBUG:boto:Params: {} 2022-09-12T19:34:02.232 DEBUG:boto:Token: None 2022-09-12T19:34:02.233 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/ 2022-09-12T19:34:02.235 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo= 2022-09-12T19:34:02.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo=', 'Content-Length': '0'} 2022-09-12T19:34:02.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074306c185d518801-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.240 DEBUG:boto:b'jdpiak-63obj41000falseobj42022-09-12T19:32:15.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:02.241 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:02.242 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:02.243 DEBUG:boto:Method: HEAD 2022-09-12T19:34:02.243 DEBUG:boto:Path: /jdpiak-63/obj4 2022-09-12T19:34:02.244 DEBUG:boto:Data: 2022-09-12T19:34:02.245 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.245 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:02.246 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.246 DEBUG:boto:Params: {} 2022-09-12T19:34:02.247 DEBUG:boto:Token: None 2022-09-12T19:34:02.248 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj4 2022-09-12T19:34:02.248 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7LdDl8cQ5UkhPz2YmOA7X47SZRk= 2022-09-12T19:34:02.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7LdDl8cQ5UkhPz2YmOA7X47SZRk=', 'Content-Length': '0'} 2022-09-12T19:34:02.251 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ddd1f9d8918fb7bf-00631f89aa-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.252 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:02.252 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:02.253 DEBUG:boto:Method: HEAD 2022-09-12T19:34:02.253 DEBUG:boto:Path: /jdpiak-63/obj4 2022-09-12T19:34:02.254 DEBUG:boto:Data: 2022-09-12T19:34:02.254 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.255 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.256 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.256 DEBUG:boto:Params: {} 2022-09-12T19:34:02.257 DEBUG:boto:Token: None 2022-09-12T19:34:02.257 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj4 2022-09-12T19:34:02.258 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7LdDl8cQ5UkhPz2YmOA7X47SZRk= 2022-09-12T19:34:02.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7LdDl8cQ5UkhPz2YmOA7X47SZRk=', 'Content-Length': '0'} 2022-09-12T19:34:02.261 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083f10efefe5d2597-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-12T19:34:02.262 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:02.262 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:02.263 DEBUG:boto:Method: GET 2022-09-12T19:34:02.264 DEBUG:boto:Path: /jdpiak-63/obj4 2022-09-12T19:34:02.264 DEBUG:boto:Data: 2022-09-12T19:34:02.265 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.265 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:02.266 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.266 DEBUG:boto:Params: {} 2022-09-12T19:34:02.267 DEBUG:boto:Token: None 2022-09-12T19:34:02.268 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj4 2022-09-12T19:34:02.268 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mtZGHFVHbpCssSFgYbtOkGXaMWk= 2022-09-12T19:34:02.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mtZGHFVHbpCssSFgYbtOkGXaMWk=', 'Content-Length': '0'} 2022-09-12T19:34:02.271 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002125df9ef661649c-00631f89aa-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.314 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:02.315 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:02.315 DEBUG:boto:Method: GET 2022-09-12T19:34:02.316 DEBUG:boto:Path: /jdpiak-63/obj4 2022-09-12T19:34:02.317 DEBUG:boto:Data: 2022-09-12T19:34:02.317 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.318 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.318 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.319 DEBUG:boto:Params: {} 2022-09-12T19:34:02.320 DEBUG:boto:Token: None 2022-09-12T19:34:02.321 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj4 2022-09-12T19:34:02.321 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mtZGHFVHbpCssSFgYbtOkGXaMWk= 2022-09-12T19:34:02.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mtZGHFVHbpCssSFgYbtOkGXaMWk=', 'Content-Length': '0'} 2022-09-12T19:34:02.324 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a34facd7fa7d3324-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.325 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:02.326 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:02.327 DEBUG:boto:path=/jdpiak-63/?prefix=obj4 2022-09-12T19:34:02.327 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj4 2022-09-12T19:34:02.328 DEBUG:boto:Method: GET 2022-09-12T19:34:02.329 DEBUG:boto:Path: /jdpiak-63/?prefix=obj4 2022-09-12T19:34:02.329 DEBUG:boto:Data: 2022-09-12T19:34:02.330 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.331 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:02.331 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.332 DEBUG:boto:Params: {} 2022-09-12T19:34:02.332 DEBUG:boto:Token: None 2022-09-12T19:34:02.333 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/ 2022-09-12T19:34:02.334 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo= 2022-09-12T19:34:02.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo=', 'Content-Length': '0'} 2022-09-12T19:34:02.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000924189a15061cebf-00631f89aa-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.338 DEBUG:boto:b'jdpiak-63obj41000falseobj42022-09-12T19:32:15.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:02.339 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:02.339 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:02.340 DEBUG:boto:path=/jdpiak-63/?prefix=obj4 2022-09-12T19:34:02.340 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj4 2022-09-12T19:34:02.341 DEBUG:boto:Method: GET 2022-09-12T19:34:02.341 DEBUG:boto:Path: /jdpiak-63/?prefix=obj4 2022-09-12T19:34:02.342 DEBUG:boto:Data: 2022-09-12T19:34:02.343 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.343 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.344 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.344 DEBUG:boto:Params: {} 2022-09-12T19:34:02.345 DEBUG:boto:Token: None 2022-09-12T19:34:02.345 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/ 2022-09-12T19:34:02.346 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo= 2022-09-12T19:34:02.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo=', 'Content-Length': '0'} 2022-09-12T19:34:02.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e57998ed0e6b3906-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.350 DEBUG:boto:b'jdpiak-63obj41000falseobj42022-09-12T19:32:15.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:02.351 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-63 zones={test-zone2, test-zone1} 2022-09-12T19:34:02.352 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-63 zones={test-zone3, test-zone1} 2022-09-12T19:34:02.352 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:02.353 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:02.353 DEBUG:boto:Method: HEAD 2022-09-12T19:34:02.354 DEBUG:boto:Path: /jdpiak-63/ 2022-09-12T19:34:02.354 DEBUG:boto:Data: 2022-09-12T19:34:02.355 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.355 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:02.356 DEBUG:boto:Port: 8001 2022-09-12T19:34:02.356 DEBUG:boto:Params: {} 2022-09-12T19:34:02.357 DEBUG:boto:Token: None 2022-09-12T19:34:02.357 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/ 2022-09-12T19:34:02.358 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GqMI0a+vbaFsQu9w6qr/IbhBZsU= 2022-09-12T19:34:02.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GqMI0a+vbaFsQu9w6qr/IbhBZsU=', 'Content-Length': '0'} 2022-09-12T19:34:02.361 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b153710974f94c53-00631f89aa-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.362 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:02.363 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:02.363 DEBUG:boto:Method: HEAD 2022-09-12T19:34:02.364 DEBUG:boto:Path: /jdpiak-63/ 2022-09-12T19:34:02.364 DEBUG:boto:Data: 2022-09-12T19:34:02.365 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.365 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.366 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.366 DEBUG:boto:Params: {} 2022-09-12T19:34:02.367 DEBUG:boto:Token: None 2022-09-12T19:34:02.368 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/ 2022-09-12T19:34:02.368 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GqMI0a+vbaFsQu9w6qr/IbhBZsU= 2022-09-12T19:34:02.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GqMI0a+vbaFsQu9w6qr/IbhBZsU=', 'Content-Length': '0'} 2022-09-12T19:34:02.372 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007dabcaa93c5f1aa5-00631f89aa-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.373 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:34:02.374 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:02.374 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:02.375 DEBUG:boto:path=/jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:02.375 DEBUG:boto:auth_path=/jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:02.376 DEBUG:boto:Method: GET 2022-09-12T19:34:02.376 DEBUG:boto:Path: /jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:02.377 DEBUG:boto:Data: 2022-09-12T19:34:02.377 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.378 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:02.379 DEBUG:boto:Port: 8001 2022-09-12T19:34:02.379 DEBUG:boto:Params: {} 2022-09-12T19:34:02.380 DEBUG:boto:Token: None 2022-09-12T19:34:02.380 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/?versions 2022-09-12T19:34:02.381 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lc/koSxK4rwulI4xZFeSgqiBZw0= 2022-09-12T19:34:02.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lc/koSxK4rwulI4xZFeSgqiBZw0=', 'Content-Length': '0'} 2022-09-12T19:34:02.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000194d1c0f45a66695-00631f89aa-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.385 DEBUG:boto:b'jdpiak-63999falseobj1nulltrue2022-09-12T19:32:15.261Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-12T19:32:15.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-12T19:32:15.326Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-12T19:32:15.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:02.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-12T19:34:02.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-12T19:34:02.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-12T19:34:02.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-12T19:34:02.388 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:34:02.389 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:02.389 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:02.390 DEBUG:boto:path=/jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:02.391 DEBUG:boto:auth_path=/jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:02.391 DEBUG:boto:Method: GET 2022-09-12T19:34:02.392 DEBUG:boto:Path: /jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:02.392 DEBUG:boto:Data: 2022-09-12T19:34:02.393 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.393 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.394 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.395 DEBUG:boto:Params: {} 2022-09-12T19:34:02.395 DEBUG:boto:Token: None 2022-09-12T19:34:02.396 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/?versions 2022-09-12T19:34:02.397 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lc/koSxK4rwulI4xZFeSgqiBZw0= 2022-09-12T19:34:02.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lc/koSxK4rwulI4xZFeSgqiBZw0=', 'Content-Length': '0'} 2022-09-12T19:34:02.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090bc663e3ec9e992-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.401 DEBUG:boto:b'jdpiak-63999falseobj1nulltrue2022-09-12T19:32:15.261Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-12T19:32:15.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-12T19:32:15.326Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-12T19:32:15.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:02.402 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-12T19:34:02.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-12T19:34:02.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-12T19:34:02.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-12T19:34:02.404 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:02.405 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:02.406 DEBUG:boto:path=/jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:02.406 DEBUG:boto:auth_path=/jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:02.407 DEBUG:boto:Method: GET 2022-09-12T19:34:02.407 DEBUG:boto:Path: /jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:02.408 DEBUG:boto:Data: 2022-09-12T19:34:02.408 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.409 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:02.409 DEBUG:boto:Port: 8001 2022-09-12T19:34:02.410 DEBUG:boto:Params: {} 2022-09-12T19:34:02.411 DEBUG:boto:Token: None 2022-09-12T19:34:02.411 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/?versions 2022-09-12T19:34:02.412 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lc/koSxK4rwulI4xZFeSgqiBZw0= 2022-09-12T19:34:02.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lc/koSxK4rwulI4xZFeSgqiBZw0=', 'Content-Length': '0'} 2022-09-12T19:34:02.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a6de9ed827bdc90-00631f89aa-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.416 DEBUG:boto:b'jdpiak-63999falseobj1nulltrue2022-09-12T19:32:15.261Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-12T19:32:15.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-12T19:32:15.326Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-12T19:32:15.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:02.417 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:02.418 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:02.418 DEBUG:boto:path=/jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:02.419 DEBUG:boto:auth_path=/jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:02.419 DEBUG:boto:Method: GET 2022-09-12T19:34:02.420 DEBUG:boto:Path: /jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:02.420 DEBUG:boto:Data: 2022-09-12T19:34:02.421 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.421 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.422 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.422 DEBUG:boto:Params: {} 2022-09-12T19:34:02.423 DEBUG:boto:Token: None 2022-09-12T19:34:02.424 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/?versions 2022-09-12T19:34:02.424 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lc/koSxK4rwulI4xZFeSgqiBZw0= 2022-09-12T19:34:02.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lc/koSxK4rwulI4xZFeSgqiBZw0=', 'Content-Length': '0'} 2022-09-12T19:34:02.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cea13a5a3b7a5073-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.428 DEBUG:boto:b'jdpiak-63999falseobj1nulltrue2022-09-12T19:32:15.261Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-12T19:32:15.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-12T19:32:15.326Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-12T19:32:15.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:02.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-12T19:34:02.430 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:02.431 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:02.431 DEBUG:boto:path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:02.432 DEBUG:boto:auth_path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:02.432 DEBUG:boto:Method: GET 2022-09-12T19:34:02.433 DEBUG:boto:Path: /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:02.433 DEBUG:boto:Data: 2022-09-12T19:34:02.434 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.434 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:02.435 DEBUG:boto:Port: 8001 2022-09-12T19:34:02.435 DEBUG:boto:Params: {} 2022-09-12T19:34:02.436 DEBUG:boto:Token: None 2022-09-12T19:34:02.437 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:02.437 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZIMHJNaXJlKpWQaJicrlgc3yLUc= 2022-09-12T19:34:02.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZIMHJNaXJlKpWQaJicrlgc3yLUc=', 'Content-Length': '0'} 2022-09-12T19:34:02.440 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ff111581790b617-00631f89aa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.441 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:02.442 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:02.442 DEBUG:boto:path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:02.443 DEBUG:boto:auth_path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:02.443 DEBUG:boto:Method: GET 2022-09-12T19:34:02.444 DEBUG:boto:Path: /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:02.444 DEBUG:boto:Data: 2022-09-12T19:34:02.445 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.446 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.446 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.447 DEBUG:boto:Params: {} 2022-09-12T19:34:02.447 DEBUG:boto:Token: None 2022-09-12T19:34:02.448 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:02.448 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZIMHJNaXJlKpWQaJicrlgc3yLUc= 2022-09-12T19:34:02.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZIMHJNaXJlKpWQaJicrlgc3yLUc=', 'Content-Length': '0'} 2022-09-12T19:34:02.451 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d8be9a5e3023e69-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.494 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:02.495 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:02.495 DEBUG:boto:path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:02.496 DEBUG:boto:auth_path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:02.497 DEBUG:boto:Method: HEAD 2022-09-12T19:34:02.497 DEBUG:boto:Path: /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:02.498 DEBUG:boto:Data: 2022-09-12T19:34:02.498 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.499 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:02.500 DEBUG:boto:Port: 8001 2022-09-12T19:34:02.500 DEBUG:boto:Params: {} 2022-09-12T19:34:02.501 DEBUG:boto:Token: None 2022-09-12T19:34:02.501 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:02.502 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:O1JE/HwM0rV+VEYNwpQTbxWGMXs= 2022-09-12T19:34:02.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:O1JE/HwM0rV+VEYNwpQTbxWGMXs=', 'Content-Length': '0'} 2022-09-12T19:34:02.505 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f4cad98e90133a2-00631f89aa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.505 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:02.506 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:02.507 DEBUG:boto:path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:02.507 DEBUG:boto:auth_path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:02.508 DEBUG:boto:Method: HEAD 2022-09-12T19:34:02.508 DEBUG:boto:Path: /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:02.509 DEBUG:boto:Data: 2022-09-12T19:34:02.510 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.510 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.511 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.511 DEBUG:boto:Params: {} 2022-09-12T19:34:02.512 DEBUG:boto:Token: None 2022-09-12T19:34:02.512 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:02.513 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:O1JE/HwM0rV+VEYNwpQTbxWGMXs= 2022-09-12T19:34:02.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:O1JE/HwM0rV+VEYNwpQTbxWGMXs=', 'Content-Length': '0'} 2022-09-12T19:34:02.516 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c2e10eb37f9c521-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-12T19:34:02.517 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:02.517 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:02.518 DEBUG:boto:path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:02.519 DEBUG:boto:auth_path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:02.519 DEBUG:boto:Method: GET 2022-09-12T19:34:02.520 DEBUG:boto:Path: /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:02.520 DEBUG:boto:Data: 2022-09-12T19:34:02.521 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.521 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:02.522 DEBUG:boto:Port: 8001 2022-09-12T19:34:02.522 DEBUG:boto:Params: {} 2022-09-12T19:34:02.523 DEBUG:boto:Token: None 2022-09-12T19:34:02.524 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:02.524 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZIMHJNaXJlKpWQaJicrlgc3yLUc= 2022-09-12T19:34:02.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZIMHJNaXJlKpWQaJicrlgc3yLUc=', 'Content-Length': '0'} 2022-09-12T19:34:02.527 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007152be3e19467467-00631f89aa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.528 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:02.529 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:02.529 DEBUG:boto:path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:02.530 DEBUG:boto:auth_path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:02.530 DEBUG:boto:Method: GET 2022-09-12T19:34:02.531 DEBUG:boto:Path: /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:02.531 DEBUG:boto:Data: 2022-09-12T19:34:02.532 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.533 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.533 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.534 DEBUG:boto:Params: {} 2022-09-12T19:34:02.534 DEBUG:boto:Token: None 2022-09-12T19:34:02.535 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:02.536 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZIMHJNaXJlKpWQaJicrlgc3yLUc= 2022-09-12T19:34:02.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZIMHJNaXJlKpWQaJicrlgc3yLUc=', 'Content-Length': '0'} 2022-09-12T19:34:02.538 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006df18b30e20e082e-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.539 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:02.540 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:02.540 DEBUG:boto:path=/jdpiak-63/?prefix=obj1 2022-09-12T19:34:02.541 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj1 2022-09-12T19:34:02.541 DEBUG:boto:Method: GET 2022-09-12T19:34:02.542 DEBUG:boto:Path: /jdpiak-63/?prefix=obj1 2022-09-12T19:34:02.543 DEBUG:boto:Data: 2022-09-12T19:34:02.543 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.544 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:02.544 DEBUG:boto:Port: 8001 2022-09-12T19:34:02.545 DEBUG:boto:Params: {} 2022-09-12T19:34:02.554 DEBUG:boto:Token: None 2022-09-12T19:34:02.555 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/ 2022-09-12T19:34:02.555 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo= 2022-09-12T19:34:02.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo=', 'Content-Length': '0'} 2022-09-12T19:34:02.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080dd084a5b24931c-00631f89aa-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.560 DEBUG:boto:b'jdpiak-63obj11000falseobj12022-09-12T19:32:15.261Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:02.561 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:02.561 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:02.562 DEBUG:boto:path=/jdpiak-63/?prefix=obj1 2022-09-12T19:34:02.562 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj1 2022-09-12T19:34:02.563 DEBUG:boto:Method: GET 2022-09-12T19:34:02.563 DEBUG:boto:Path: /jdpiak-63/?prefix=obj1 2022-09-12T19:34:02.564 DEBUG:boto:Data: 2022-09-12T19:34:02.564 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.565 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.565 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.566 DEBUG:boto:Params: {} 2022-09-12T19:34:02.566 DEBUG:boto:Token: None 2022-09-12T19:34:02.567 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/ 2022-09-12T19:34:02.568 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo= 2022-09-12T19:34:02.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo=', 'Content-Length': '0'} 2022-09-12T19:34:02.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8fc5e49dd368e73-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.572 DEBUG:boto:b'jdpiak-63obj11000falseobj12022-09-12T19:32:15.261Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:02.572 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:02.573 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:02.573 DEBUG:boto:Method: HEAD 2022-09-12T19:34:02.574 DEBUG:boto:Path: /jdpiak-63/obj1 2022-09-12T19:34:02.575 DEBUG:boto:Data: 2022-09-12T19:34:02.575 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.576 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:02.576 DEBUG:boto:Port: 8001 2022-09-12T19:34:02.577 DEBUG:boto:Params: {} 2022-09-12T19:34:02.577 DEBUG:boto:Token: None 2022-09-12T19:34:02.578 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj1 2022-09-12T19:34:02.578 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OlMcGdo3SsZ9oH5tsJikWssRgLo= 2022-09-12T19:34:02.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OlMcGdo3SsZ9oH5tsJikWssRgLo=', 'Content-Length': '0'} 2022-09-12T19:34:02.581 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d984305504f4e72c-00631f89aa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.582 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:02.582 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:02.583 DEBUG:boto:Method: HEAD 2022-09-12T19:34:02.583 DEBUG:boto:Path: /jdpiak-63/obj1 2022-09-12T19:34:02.584 DEBUG:boto:Data: 2022-09-12T19:34:02.584 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.585 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.585 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.586 DEBUG:boto:Params: {} 2022-09-12T19:34:02.586 DEBUG:boto:Token: None 2022-09-12T19:34:02.587 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj1 2022-09-12T19:34:02.588 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OlMcGdo3SsZ9oH5tsJikWssRgLo= 2022-09-12T19:34:02.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OlMcGdo3SsZ9oH5tsJikWssRgLo=', 'Content-Length': '0'} 2022-09-12T19:34:02.590 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0190c5e589502b3-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-12T19:34:02.591 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:02.592 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:02.593 DEBUG:boto:Method: GET 2022-09-12T19:34:02.593 DEBUG:boto:Path: /jdpiak-63/obj1 2022-09-12T19:34:02.594 DEBUG:boto:Data: 2022-09-12T19:34:02.595 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.595 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:02.596 DEBUG:boto:Port: 8001 2022-09-12T19:34:02.596 DEBUG:boto:Params: {} 2022-09-12T19:34:02.597 DEBUG:boto:Token: None 2022-09-12T19:34:02.597 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj1 2022-09-12T19:34:02.598 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:feGk066zcJ+ZI4n863z7/0THWP4= 2022-09-12T19:34:02.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:feGk066zcJ+ZI4n863z7/0THWP4=', 'Content-Length': '0'} 2022-09-12T19:34:02.601 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000048d3444e2906f9d7-00631f89aa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.642 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:02.643 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:02.644 DEBUG:boto:Method: GET 2022-09-12T19:34:02.644 DEBUG:boto:Path: /jdpiak-63/obj1 2022-09-12T19:34:02.645 DEBUG:boto:Data: 2022-09-12T19:34:02.645 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.646 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.646 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.647 DEBUG:boto:Params: {} 2022-09-12T19:34:02.647 DEBUG:boto:Token: None 2022-09-12T19:34:02.648 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj1 2022-09-12T19:34:02.649 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:feGk066zcJ+ZI4n863z7/0THWP4= 2022-09-12T19:34:02.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:feGk066zcJ+ZI4n863z7/0THWP4=', 'Content-Length': '0'} 2022-09-12T19:34:02.652 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a46d9a239d91b04-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.653 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:02.653 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:02.654 DEBUG:boto:path=/jdpiak-63/?prefix=obj1 2022-09-12T19:34:02.654 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj1 2022-09-12T19:34:02.655 DEBUG:boto:Method: GET 2022-09-12T19:34:02.655 DEBUG:boto:Path: /jdpiak-63/?prefix=obj1 2022-09-12T19:34:02.656 DEBUG:boto:Data: 2022-09-12T19:34:02.656 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.657 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:02.657 DEBUG:boto:Port: 8001 2022-09-12T19:34:02.658 DEBUG:boto:Params: {} 2022-09-12T19:34:02.659 DEBUG:boto:Token: None 2022-09-12T19:34:02.659 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/ 2022-09-12T19:34:02.660 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo= 2022-09-12T19:34:02.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo=', 'Content-Length': '0'} 2022-09-12T19:34:02.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007eff9a4901d7be9c-00631f89aa-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.664 DEBUG:boto:b'jdpiak-63obj11000falseobj12022-09-12T19:32:15.261Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:02.665 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:02.665 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:02.666 DEBUG:boto:path=/jdpiak-63/?prefix=obj1 2022-09-12T19:34:02.666 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj1 2022-09-12T19:34:02.667 DEBUG:boto:Method: GET 2022-09-12T19:34:02.667 DEBUG:boto:Path: /jdpiak-63/?prefix=obj1 2022-09-12T19:34:02.668 DEBUG:boto:Data: 2022-09-12T19:34:02.668 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.669 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.669 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.670 DEBUG:boto:Params: {} 2022-09-12T19:34:02.670 DEBUG:boto:Token: None 2022-09-12T19:34:02.671 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/ 2022-09-12T19:34:02.672 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo= 2022-09-12T19:34:02.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo=', 'Content-Length': '0'} 2022-09-12T19:34:02.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b509528b19f1c410-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.676 DEBUG:boto:b'jdpiak-63obj11000falseobj12022-09-12T19:32:15.261Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:02.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-12T19:34:02.677 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:02.677 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:02.678 DEBUG:boto:path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:02.679 DEBUG:boto:auth_path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:02.679 DEBUG:boto:Method: GET 2022-09-12T19:34:02.680 DEBUG:boto:Path: /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:02.680 DEBUG:boto:Data: 2022-09-12T19:34:02.681 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.681 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:02.682 DEBUG:boto:Port: 8001 2022-09-12T19:34:02.682 DEBUG:boto:Params: {} 2022-09-12T19:34:02.683 DEBUG:boto:Token: None 2022-09-12T19:34:02.684 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:02.684 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BGyh1thTAhckLNyYxvbpYQiYKwA= 2022-09-12T19:34:02.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BGyh1thTAhckLNyYxvbpYQiYKwA=', 'Content-Length': '0'} 2022-09-12T19:34:02.687 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029188fa3729add36-00631f89aa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.688 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:02.688 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:02.689 DEBUG:boto:path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:02.690 DEBUG:boto:auth_path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:02.690 DEBUG:boto:Method: GET 2022-09-12T19:34:02.691 DEBUG:boto:Path: /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:02.691 DEBUG:boto:Data: 2022-09-12T19:34:02.692 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.692 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.693 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.693 DEBUG:boto:Params: {} 2022-09-12T19:34:02.694 DEBUG:boto:Token: None 2022-09-12T19:34:02.694 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:02.695 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BGyh1thTAhckLNyYxvbpYQiYKwA= 2022-09-12T19:34:02.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BGyh1thTAhckLNyYxvbpYQiYKwA=', 'Content-Length': '0'} 2022-09-12T19:34:02.698 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000582adb446a98d055-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.698 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:02.699 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:02.699 DEBUG:boto:path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:02.700 DEBUG:boto:auth_path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:02.700 DEBUG:boto:Method: HEAD 2022-09-12T19:34:02.701 DEBUG:boto:Path: /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:02.701 DEBUG:boto:Data: 2022-09-12T19:34:02.702 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.702 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:02.703 DEBUG:boto:Port: 8001 2022-09-12T19:34:02.703 DEBUG:boto:Params: {} 2022-09-12T19:34:02.704 DEBUG:boto:Token: None 2022-09-12T19:34:02.705 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:02.705 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+rnrDJLr32X2GyzAqghfVnEMwRE= 2022-09-12T19:34:02.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+rnrDJLr32X2GyzAqghfVnEMwRE=', 'Content-Length': '0'} 2022-09-12T19:34:02.708 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a3f2eb66b2d4e03-00631f89aa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.709 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:02.709 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:02.710 DEBUG:boto:path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:02.710 DEBUG:boto:auth_path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:02.711 DEBUG:boto:Method: HEAD 2022-09-12T19:34:02.711 DEBUG:boto:Path: /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:02.712 DEBUG:boto:Data: 2022-09-12T19:34:02.712 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.713 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.713 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.714 DEBUG:boto:Params: {} 2022-09-12T19:34:02.714 DEBUG:boto:Token: None 2022-09-12T19:34:02.715 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:02.716 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+rnrDJLr32X2GyzAqghfVnEMwRE= 2022-09-12T19:34:02.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+rnrDJLr32X2GyzAqghfVnEMwRE=', 'Content-Length': '0'} 2022-09-12T19:34:02.719 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085e85eeb9dc04778-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-12T19:34:02.720 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:02.720 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:02.721 DEBUG:boto:path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:02.722 DEBUG:boto:auth_path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:02.722 DEBUG:boto:Method: GET 2022-09-12T19:34:02.723 DEBUG:boto:Path: /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:02.723 DEBUG:boto:Data: 2022-09-12T19:34:02.724 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.724 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:02.725 DEBUG:boto:Port: 8001 2022-09-12T19:34:02.725 DEBUG:boto:Params: {} 2022-09-12T19:34:02.726 DEBUG:boto:Token: None 2022-09-12T19:34:02.726 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:02.727 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BGyh1thTAhckLNyYxvbpYQiYKwA= 2022-09-12T19:34:02.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BGyh1thTAhckLNyYxvbpYQiYKwA=', 'Content-Length': '0'} 2022-09-12T19:34:02.729 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046df88f6dfb50a0b-00631f89aa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.770 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:02.771 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:02.771 DEBUG:boto:path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:02.772 DEBUG:boto:auth_path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:02.772 DEBUG:boto:Method: GET 2022-09-12T19:34:02.773 DEBUG:boto:Path: /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:02.773 DEBUG:boto:Data: 2022-09-12T19:34:02.774 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.774 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.775 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.775 DEBUG:boto:Params: {} 2022-09-12T19:34:02.776 DEBUG:boto:Token: None 2022-09-12T19:34:02.777 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:02.777 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BGyh1thTAhckLNyYxvbpYQiYKwA= 2022-09-12T19:34:02.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BGyh1thTAhckLNyYxvbpYQiYKwA=', 'Content-Length': '0'} 2022-09-12T19:34:02.781 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055fb687e81264edb-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.781 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:02.782 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:02.782 DEBUG:boto:path=/jdpiak-63/?prefix=obj2 2022-09-12T19:34:02.783 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj2 2022-09-12T19:34:02.783 DEBUG:boto:Method: GET 2022-09-12T19:34:02.784 DEBUG:boto:Path: /jdpiak-63/?prefix=obj2 2022-09-12T19:34:02.785 DEBUG:boto:Data: 2022-09-12T19:34:02.785 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.787 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:02.787 DEBUG:boto:Port: 8001 2022-09-12T19:34:02.788 DEBUG:boto:Params: {} 2022-09-12T19:34:02.788 DEBUG:boto:Token: None 2022-09-12T19:34:02.789 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/ 2022-09-12T19:34:02.790 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo= 2022-09-12T19:34:02.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo=', 'Content-Length': '0'} 2022-09-12T19:34:02.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a46b2d325f4035e-00631f89aa-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.794 DEBUG:boto:b'jdpiak-63obj21000falseobj22022-09-12T19:32:15.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:02.795 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:02.795 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:02.796 DEBUG:boto:path=/jdpiak-63/?prefix=obj2 2022-09-12T19:34:02.796 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj2 2022-09-12T19:34:02.797 DEBUG:boto:Method: GET 2022-09-12T19:34:02.797 DEBUG:boto:Path: /jdpiak-63/?prefix=obj2 2022-09-12T19:34:02.798 DEBUG:boto:Data: 2022-09-12T19:34:02.798 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.799 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.800 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.800 DEBUG:boto:Params: {} 2022-09-12T19:34:02.801 DEBUG:boto:Token: None 2022-09-12T19:34:02.802 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/ 2022-09-12T19:34:02.802 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo= 2022-09-12T19:34:02.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo=', 'Content-Length': '0'} 2022-09-12T19:34:02.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea7416ccad3cff3c-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.807 DEBUG:boto:b'jdpiak-63obj21000falseobj22022-09-12T19:32:15.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:02.808 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:02.809 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:02.810 DEBUG:boto:Method: HEAD 2022-09-12T19:34:02.810 DEBUG:boto:Path: /jdpiak-63/obj2 2022-09-12T19:34:02.812 DEBUG:boto:Data: 2022-09-12T19:34:02.813 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.814 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:02.815 DEBUG:boto:Port: 8001 2022-09-12T19:34:02.816 DEBUG:boto:Params: {} 2022-09-12T19:34:02.817 DEBUG:boto:Token: None 2022-09-12T19:34:02.818 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj2 2022-09-12T19:34:02.819 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:K2ptiKzVXQSZOpjfEh46nHEkVj4= 2022-09-12T19:34:02.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:K2ptiKzVXQSZOpjfEh46nHEkVj4=', 'Content-Length': '0'} 2022-09-12T19:34:02.824 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001d2899ec34826d5-00631f89aa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.825 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:02.826 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:02.827 DEBUG:boto:Method: HEAD 2022-09-12T19:34:02.828 DEBUG:boto:Path: /jdpiak-63/obj2 2022-09-12T19:34:02.829 DEBUG:boto:Data: 2022-09-12T19:34:02.830 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.831 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.832 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.833 DEBUG:boto:Params: {} 2022-09-12T19:34:02.834 DEBUG:boto:Token: None 2022-09-12T19:34:02.835 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj2 2022-09-12T19:34:02.836 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:K2ptiKzVXQSZOpjfEh46nHEkVj4= 2022-09-12T19:34:02.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:K2ptiKzVXQSZOpjfEh46nHEkVj4=', 'Content-Length': '0'} 2022-09-12T19:34:02.840 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004530b7c9ee22cbf5-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-12T19:34:02.841 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:02.842 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:02.842 DEBUG:boto:Method: GET 2022-09-12T19:34:02.843 DEBUG:boto:Path: /jdpiak-63/obj2 2022-09-12T19:34:02.844 DEBUG:boto:Data: 2022-09-12T19:34:02.844 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.845 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:02.845 DEBUG:boto:Port: 8001 2022-09-12T19:34:02.846 DEBUG:boto:Params: {} 2022-09-12T19:34:02.846 DEBUG:boto:Token: None 2022-09-12T19:34:02.847 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj2 2022-09-12T19:34:02.848 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8AMaAokNd8maLc6wGtVDOqw0wc8= 2022-09-12T19:34:02.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8AMaAokNd8maLc6wGtVDOqw0wc8=', 'Content-Length': '0'} 2022-09-12T19:34:02.851 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf7b771a27a620d3-00631f89aa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.852 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:02.852 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:02.853 DEBUG:boto:Method: GET 2022-09-12T19:34:02.854 DEBUG:boto:Path: /jdpiak-63/obj2 2022-09-12T19:34:02.854 DEBUG:boto:Data: 2022-09-12T19:34:02.855 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.856 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.856 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.857 DEBUG:boto:Params: {} 2022-09-12T19:34:02.857 DEBUG:boto:Token: None 2022-09-12T19:34:02.858 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj2 2022-09-12T19:34:02.859 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8AMaAokNd8maLc6wGtVDOqw0wc8= 2022-09-12T19:34:02.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8AMaAokNd8maLc6wGtVDOqw0wc8=', 'Content-Length': '0'} 2022-09-12T19:34:02.862 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ff3b0c78fe8eb3e-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.863 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:02.863 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:02.864 DEBUG:boto:path=/jdpiak-63/?prefix=obj2 2022-09-12T19:34:02.865 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj2 2022-09-12T19:34:02.865 DEBUG:boto:Method: GET 2022-09-12T19:34:02.866 DEBUG:boto:Path: /jdpiak-63/?prefix=obj2 2022-09-12T19:34:02.866 DEBUG:boto:Data: 2022-09-12T19:34:02.867 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.867 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:02.868 DEBUG:boto:Port: 8001 2022-09-12T19:34:02.869 DEBUG:boto:Params: {} 2022-09-12T19:34:02.869 DEBUG:boto:Token: None 2022-09-12T19:34:02.870 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/ 2022-09-12T19:34:02.871 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo= 2022-09-12T19:34:02.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo=', 'Content-Length': '0'} 2022-09-12T19:34:02.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e82eef67a216eab6-00631f89aa-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.875 DEBUG:boto:b'jdpiak-63obj21000falseobj22022-09-12T19:32:15.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:02.876 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:02.876 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:02.877 DEBUG:boto:path=/jdpiak-63/?prefix=obj2 2022-09-12T19:34:02.877 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj2 2022-09-12T19:34:02.878 DEBUG:boto:Method: GET 2022-09-12T19:34:02.878 DEBUG:boto:Path: /jdpiak-63/?prefix=obj2 2022-09-12T19:34:02.879 DEBUG:boto:Data: 2022-09-12T19:34:02.879 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.880 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.880 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.881 DEBUG:boto:Params: {} 2022-09-12T19:34:02.881 DEBUG:boto:Token: None 2022-09-12T19:34:02.882 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/ 2022-09-12T19:34:02.883 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo= 2022-09-12T19:34:02.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8GFCWtOgcN1A/zD6SyX9LiVGTWo=', 'Content-Length': '0'} 2022-09-12T19:34:02.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000558c77319b9a2727-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.930 DEBUG:boto:b'jdpiak-63obj21000falseobj22022-09-12T19:32:15.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:02.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-12T19:34:02.932 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:02.933 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:02.933 DEBUG:boto:path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:02.934 DEBUG:boto:auth_path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:02.934 DEBUG:boto:Method: GET 2022-09-12T19:34:02.935 DEBUG:boto:Path: /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:02.935 DEBUG:boto:Data: 2022-09-12T19:34:02.936 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.937 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:02.937 DEBUG:boto:Port: 8001 2022-09-12T19:34:02.938 DEBUG:boto:Params: {} 2022-09-12T19:34:02.938 DEBUG:boto:Token: None 2022-09-12T19:34:02.939 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:02.940 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G0Vty69p77MZLyGA9Hx6UEOGV6U= 2022-09-12T19:34:02.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G0Vty69p77MZLyGA9Hx6UEOGV6U=', 'Content-Length': '0'} 2022-09-12T19:34:02.943 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086aaa2883e80e8fc-00631f89aa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.944 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:02.944 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:02.945 DEBUG:boto:path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:02.945 DEBUG:boto:auth_path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:02.946 DEBUG:boto:Method: GET 2022-09-12T19:34:02.946 DEBUG:boto:Path: /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:02.947 DEBUG:boto:Data: 2022-09-12T19:34:02.947 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.948 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.948 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.949 DEBUG:boto:Params: {} 2022-09-12T19:34:02.949 DEBUG:boto:Token: None 2022-09-12T19:34:02.950 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:02.951 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G0Vty69p77MZLyGA9Hx6UEOGV6U= 2022-09-12T19:34:02.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G0Vty69p77MZLyGA9Hx6UEOGV6U=', 'Content-Length': '0'} 2022-09-12T19:34:02.954 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c52116aeea491b1-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.954 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:02.955 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:02.955 DEBUG:boto:path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:02.956 DEBUG:boto:auth_path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:02.956 DEBUG:boto:Method: HEAD 2022-09-12T19:34:02.957 DEBUG:boto:Path: /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:02.957 DEBUG:boto:Data: 2022-09-12T19:34:02.958 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.958 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:02.959 DEBUG:boto:Port: 8001 2022-09-12T19:34:02.959 DEBUG:boto:Params: {} 2022-09-12T19:34:02.960 DEBUG:boto:Token: None 2022-09-12T19:34:02.961 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:02.961 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wsA8FFVmGiZClOqkIbAqFEHQHHw= 2022-09-12T19:34:02.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wsA8FFVmGiZClOqkIbAqFEHQHHw=', 'Content-Length': '0'} 2022-09-12T19:34:02.964 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c94d7f10d876fb0-00631f89aa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.964 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:02.965 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:02.965 DEBUG:boto:path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:02.966 DEBUG:boto:auth_path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:02.966 DEBUG:boto:Method: HEAD 2022-09-12T19:34:02.967 DEBUG:boto:Path: /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:02.967 DEBUG:boto:Data: 2022-09-12T19:34:02.968 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.968 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:02.969 DEBUG:boto:Port: 8000 2022-09-12T19:34:02.969 DEBUG:boto:Params: {} 2022-09-12T19:34:02.970 DEBUG:boto:Token: None 2022-09-12T19:34:02.971 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:02.971 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wsA8FFVmGiZClOqkIbAqFEHQHHw= 2022-09-12T19:34:02.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wsA8FFVmGiZClOqkIbAqFEHQHHw=', 'Content-Length': '0'} 2022-09-12T19:34:02.974 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd40b5dd11c85df7-00631f89aa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:02.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-12T19:34:02.975 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:02.976 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:02.976 DEBUG:boto:path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:02.977 DEBUG:boto:auth_path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:02.977 DEBUG:boto:Method: GET 2022-09-12T19:34:02.978 DEBUG:boto:Path: /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:02.978 DEBUG:boto:Data: 2022-09-12T19:34:02.979 DEBUG:boto:Headers: {} 2022-09-12T19:34:02.979 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:02.980 DEBUG:boto:Port: 8001 2022-09-12T19:34:02.981 DEBUG:boto:Params: {} 2022-09-12T19:34:02.981 DEBUG:boto:Token: None 2022-09-12T19:34:02.982 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:02 GMT /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:02.982 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G0Vty69p77MZLyGA9Hx6UEOGV6U= 2022-09-12T19:34:02.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G0Vty69p77MZLyGA9Hx6UEOGV6U=', 'Content-Length': '0'} 2022-09-12T19:34:02.985 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004071c1c0d708cbd7-00631f89aa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.026 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:03.027 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:03.027 DEBUG:boto:path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:03.028 DEBUG:boto:auth_path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:03.028 DEBUG:boto:Method: GET 2022-09-12T19:34:03.029 DEBUG:boto:Path: /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:03.030 DEBUG:boto:Data: 2022-09-12T19:34:03.030 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.031 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:03.031 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.032 DEBUG:boto:Params: {} 2022-09-12T19:34:03.033 DEBUG:boto:Token: None 2022-09-12T19:34:03.034 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:03.034 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:egni9J3iYwDQg880H06rNPA5gQE= 2022-09-12T19:34:03.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:egni9J3iYwDQg880H06rNPA5gQE=', 'Content-Length': '0'} 2022-09-12T19:34:03.037 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000230905b8be89b810-00631f89ab-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.038 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:03.039 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:03.039 DEBUG:boto:path=/jdpiak-63/?prefix=obj3 2022-09-12T19:34:03.040 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj3 2022-09-12T19:34:03.040 DEBUG:boto:Method: GET 2022-09-12T19:34:03.041 DEBUG:boto:Path: /jdpiak-63/?prefix=obj3 2022-09-12T19:34:03.042 DEBUG:boto:Data: 2022-09-12T19:34:03.043 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.043 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.044 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.044 DEBUG:boto:Params: {} 2022-09-12T19:34:03.045 DEBUG:boto:Token: None 2022-09-12T19:34:03.045 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/ 2022-09-12T19:34:03.046 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA= 2022-09-12T19:34:03.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA=', 'Content-Length': '0'} 2022-09-12T19:34:03.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000621bd6e31b03f37e-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.050 DEBUG:boto:b'jdpiak-63obj31000falseobj32022-09-12T19:32:15.326Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:03.051 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:03.052 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:03.052 DEBUG:boto:path=/jdpiak-63/?prefix=obj3 2022-09-12T19:34:03.053 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj3 2022-09-12T19:34:03.053 DEBUG:boto:Method: GET 2022-09-12T19:34:03.054 DEBUG:boto:Path: /jdpiak-63/?prefix=obj3 2022-09-12T19:34:03.055 DEBUG:boto:Data: 2022-09-12T19:34:03.055 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.056 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:03.056 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.057 DEBUG:boto:Params: {} 2022-09-12T19:34:03.057 DEBUG:boto:Token: None 2022-09-12T19:34:03.058 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/ 2022-09-12T19:34:03.058 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA= 2022-09-12T19:34:03.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA=', 'Content-Length': '0'} 2022-09-12T19:34:03.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dcda4531888fd54-00631f89ab-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.063 DEBUG:boto:b'jdpiak-63obj31000falseobj32022-09-12T19:32:15.326Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:03.064 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:03.065 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:03.066 DEBUG:boto:Method: HEAD 2022-09-12T19:34:03.067 DEBUG:boto:Path: /jdpiak-63/obj3 2022-09-12T19:34:03.068 DEBUG:boto:Data: 2022-09-12T19:34:03.069 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.070 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.071 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.072 DEBUG:boto:Params: {} 2022-09-12T19:34:03.073 DEBUG:boto:Token: None 2022-09-12T19:34:03.074 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj3 2022-09-12T19:34:03.075 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9fI4IcAaiUAy8vtINRo6Rl0qgNs= 2022-09-12T19:34:03.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9fI4IcAaiUAy8vtINRo6Rl0qgNs=', 'Content-Length': '0'} 2022-09-12T19:34:03.078 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a615a79f170b8882-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.079 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:03.079 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:03.080 DEBUG:boto:Method: HEAD 2022-09-12T19:34:03.080 DEBUG:boto:Path: /jdpiak-63/obj3 2022-09-12T19:34:03.081 DEBUG:boto:Data: 2022-09-12T19:34:03.081 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.082 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:03.082 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.083 DEBUG:boto:Params: {} 2022-09-12T19:34:03.084 DEBUG:boto:Token: None 2022-09-12T19:34:03.084 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj3 2022-09-12T19:34:03.085 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9fI4IcAaiUAy8vtINRo6Rl0qgNs= 2022-09-12T19:34:03.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9fI4IcAaiUAy8vtINRo6Rl0qgNs=', 'Content-Length': '0'} 2022-09-12T19:34:03.087 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085ea445f96b6f702-00631f89ab-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-12T19:34:03.089 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:03.089 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:03.090 DEBUG:boto:Method: GET 2022-09-12T19:34:03.090 DEBUG:boto:Path: /jdpiak-63/obj3 2022-09-12T19:34:03.091 DEBUG:boto:Data: 2022-09-12T19:34:03.091 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.092 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.093 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.093 DEBUG:boto:Params: {} 2022-09-12T19:34:03.094 DEBUG:boto:Token: None 2022-09-12T19:34:03.095 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj3 2022-09-12T19:34:03.095 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Hj7bMyEVNIocOrST97XE6hlM83I= 2022-09-12T19:34:03.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Hj7bMyEVNIocOrST97XE6hlM83I=', 'Content-Length': '0'} 2022-09-12T19:34:03.098 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f31865e7c14ed1d-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.099 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:03.100 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:03.101 DEBUG:boto:Method: GET 2022-09-12T19:34:03.101 DEBUG:boto:Path: /jdpiak-63/obj3 2022-09-12T19:34:03.102 DEBUG:boto:Data: 2022-09-12T19:34:03.102 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.106 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:03.106 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.107 DEBUG:boto:Params: {} 2022-09-12T19:34:03.108 DEBUG:boto:Token: None 2022-09-12T19:34:03.108 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj3 2022-09-12T19:34:03.109 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Hj7bMyEVNIocOrST97XE6hlM83I= 2022-09-12T19:34:03.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Hj7bMyEVNIocOrST97XE6hlM83I=', 'Content-Length': '0'} 2022-09-12T19:34:03.112 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000298856eac5e3cd2e-00631f89ab-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.113 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:03.113 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:03.114 DEBUG:boto:path=/jdpiak-63/?prefix=obj3 2022-09-12T19:34:03.115 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj3 2022-09-12T19:34:03.115 DEBUG:boto:Method: GET 2022-09-12T19:34:03.116 DEBUG:boto:Path: /jdpiak-63/?prefix=obj3 2022-09-12T19:34:03.116 DEBUG:boto:Data: 2022-09-12T19:34:03.117 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.117 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.118 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.119 DEBUG:boto:Params: {} 2022-09-12T19:34:03.119 DEBUG:boto:Token: None 2022-09-12T19:34:03.120 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/ 2022-09-12T19:34:03.120 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA= 2022-09-12T19:34:03.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA=', 'Content-Length': '0'} 2022-09-12T19:34:03.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee340a157c37b12f-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.166 DEBUG:boto:b'jdpiak-63obj31000falseobj32022-09-12T19:32:15.326Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:03.167 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:03.168 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:03.168 DEBUG:boto:path=/jdpiak-63/?prefix=obj3 2022-09-12T19:34:03.169 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj3 2022-09-12T19:34:03.170 DEBUG:boto:Method: GET 2022-09-12T19:34:03.170 DEBUG:boto:Path: /jdpiak-63/?prefix=obj3 2022-09-12T19:34:03.171 DEBUG:boto:Data: 2022-09-12T19:34:03.171 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.172 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:03.173 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.173 DEBUG:boto:Params: {} 2022-09-12T19:34:03.174 DEBUG:boto:Token: None 2022-09-12T19:34:03.175 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/ 2022-09-12T19:34:03.175 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA= 2022-09-12T19:34:03.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA=', 'Content-Length': '0'} 2022-09-12T19:34:03.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a441aeca17709a10-00631f89ab-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.179 DEBUG:boto:b'jdpiak-63obj31000falseobj32022-09-12T19:32:15.326Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:03.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-12T19:34:03.181 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:03.182 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:03.182 DEBUG:boto:path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:03.183 DEBUG:boto:auth_path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:03.183 DEBUG:boto:Method: GET 2022-09-12T19:34:03.184 DEBUG:boto:Path: /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:03.184 DEBUG:boto:Data: 2022-09-12T19:34:03.185 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.186 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.186 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.187 DEBUG:boto:Params: {} 2022-09-12T19:34:03.187 DEBUG:boto:Token: None 2022-09-12T19:34:03.188 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:03.189 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zlz33wLcQU1Dslujd9UkO63gDB8= 2022-09-12T19:34:03.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zlz33wLcQU1Dslujd9UkO63gDB8=', 'Content-Length': '0'} 2022-09-12T19:34:03.191 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5c98f4df09bc253-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.192 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:03.193 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:03.193 DEBUG:boto:path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:03.194 DEBUG:boto:auth_path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:03.194 DEBUG:boto:Method: GET 2022-09-12T19:34:03.195 DEBUG:boto:Path: /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:03.196 DEBUG:boto:Data: 2022-09-12T19:34:03.196 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.197 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:03.197 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.198 DEBUG:boto:Params: {} 2022-09-12T19:34:03.199 DEBUG:boto:Token: None 2022-09-12T19:34:03.199 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:03.200 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zlz33wLcQU1Dslujd9UkO63gDB8= 2022-09-12T19:34:03.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zlz33wLcQU1Dslujd9UkO63gDB8=', 'Content-Length': '0'} 2022-09-12T19:34:03.203 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae9f1291607e92dd-00631f89ab-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.246 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:03.247 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:03.247 DEBUG:boto:path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:03.248 DEBUG:boto:auth_path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:03.248 DEBUG:boto:Method: HEAD 2022-09-12T19:34:03.249 DEBUG:boto:Path: /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:03.249 DEBUG:boto:Data: 2022-09-12T19:34:03.250 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.250 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.251 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.251 DEBUG:boto:Params: {} 2022-09-12T19:34:03.252 DEBUG:boto:Token: None 2022-09-12T19:34:03.253 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:03.253 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5iVeYvlXjtcbIpjKdlHTF329lTE= 2022-09-12T19:34:03.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5iVeYvlXjtcbIpjKdlHTF329lTE=', 'Content-Length': '0'} 2022-09-12T19:34:03.256 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033b2145791e3601d-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.257 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:03.257 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:03.258 DEBUG:boto:path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:03.258 DEBUG:boto:auth_path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:03.259 DEBUG:boto:Method: HEAD 2022-09-12T19:34:03.259 DEBUG:boto:Path: /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:03.260 DEBUG:boto:Data: 2022-09-12T19:34:03.260 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.261 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:03.261 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.262 DEBUG:boto:Params: {} 2022-09-12T19:34:03.262 DEBUG:boto:Token: None 2022-09-12T19:34:03.263 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:03.263 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5iVeYvlXjtcbIpjKdlHTF329lTE= 2022-09-12T19:34:03.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5iVeYvlXjtcbIpjKdlHTF329lTE=', 'Content-Length': '0'} 2022-09-12T19:34:03.266 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f0939579103b175-00631f89ab-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-12T19:34:03.267 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:03.268 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:03.268 DEBUG:boto:path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:03.269 DEBUG:boto:auth_path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:03.269 DEBUG:boto:Method: GET 2022-09-12T19:34:03.270 DEBUG:boto:Path: /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:03.271 DEBUG:boto:Data: 2022-09-12T19:34:03.271 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.272 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.272 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.273 DEBUG:boto:Params: {} 2022-09-12T19:34:03.273 DEBUG:boto:Token: None 2022-09-12T19:34:03.274 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:03.275 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zlz33wLcQU1Dslujd9UkO63gDB8= 2022-09-12T19:34:03.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zlz33wLcQU1Dslujd9UkO63gDB8=', 'Content-Length': '0'} 2022-09-12T19:34:03.277 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000005ca69e840b8756-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.278 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:03.279 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:03.279 DEBUG:boto:path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:03.280 DEBUG:boto:auth_path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:03.280 DEBUG:boto:Method: GET 2022-09-12T19:34:03.281 DEBUG:boto:Path: /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:03.281 DEBUG:boto:Data: 2022-09-12T19:34:03.282 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.282 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:03.283 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.283 DEBUG:boto:Params: {} 2022-09-12T19:34:03.284 DEBUG:boto:Token: None 2022-09-12T19:34:03.285 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:03.285 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zlz33wLcQU1Dslujd9UkO63gDB8= 2022-09-12T19:34:03.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zlz33wLcQU1Dslujd9UkO63gDB8=', 'Content-Length': '0'} 2022-09-12T19:34:03.288 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004dfa9015c229fd4b-00631f89ab-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.289 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:03.289 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:03.290 DEBUG:boto:path=/jdpiak-63/?prefix=obj4 2022-09-12T19:34:03.290 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj4 2022-09-12T19:34:03.291 DEBUG:boto:Method: GET 2022-09-12T19:34:03.291 DEBUG:boto:Path: /jdpiak-63/?prefix=obj4 2022-09-12T19:34:03.292 DEBUG:boto:Data: 2022-09-12T19:34:03.293 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.293 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.294 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.294 DEBUG:boto:Params: {} 2022-09-12T19:34:03.295 DEBUG:boto:Token: None 2022-09-12T19:34:03.295 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/ 2022-09-12T19:34:03.296 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA= 2022-09-12T19:34:03.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA=', 'Content-Length': '0'} 2022-09-12T19:34:03.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aed6c6b8f1c8dd01-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.300 DEBUG:boto:b'jdpiak-63obj41000falseobj42022-09-12T19:32:15.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:03.301 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:03.301 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:03.302 DEBUG:boto:path=/jdpiak-63/?prefix=obj4 2022-09-12T19:34:03.303 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj4 2022-09-12T19:34:03.303 DEBUG:boto:Method: GET 2022-09-12T19:34:03.304 DEBUG:boto:Path: /jdpiak-63/?prefix=obj4 2022-09-12T19:34:03.304 DEBUG:boto:Data: 2022-09-12T19:34:03.305 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.305 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:03.306 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.306 DEBUG:boto:Params: {} 2022-09-12T19:34:03.307 DEBUG:boto:Token: None 2022-09-12T19:34:03.308 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/ 2022-09-12T19:34:03.308 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA= 2022-09-12T19:34:03.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA=', 'Content-Length': '0'} 2022-09-12T19:34:03.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054fd320921cc6f5f-00631f89ab-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.313 DEBUG:boto:b'jdpiak-63obj41000falseobj42022-09-12T19:32:15.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:03.314 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:03.314 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:03.315 DEBUG:boto:Method: HEAD 2022-09-12T19:34:03.315 DEBUG:boto:Path: /jdpiak-63/obj4 2022-09-12T19:34:03.316 DEBUG:boto:Data: 2022-09-12T19:34:03.316 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.317 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.317 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.318 DEBUG:boto:Params: {} 2022-09-12T19:34:03.318 DEBUG:boto:Token: None 2022-09-12T19:34:03.319 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj4 2022-09-12T19:34:03.320 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:47JkU+hZRL1YbMa5EdtQNzu7F7c= 2022-09-12T19:34:03.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:47JkU+hZRL1YbMa5EdtQNzu7F7c=', 'Content-Length': '0'} 2022-09-12T19:34:03.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000261db512b83f793d-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.323 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:03.324 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:03.324 DEBUG:boto:Method: HEAD 2022-09-12T19:34:03.325 DEBUG:boto:Path: /jdpiak-63/obj4 2022-09-12T19:34:03.325 DEBUG:boto:Data: 2022-09-12T19:34:03.326 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.326 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:03.327 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.327 DEBUG:boto:Params: {} 2022-09-12T19:34:03.328 DEBUG:boto:Token: None 2022-09-12T19:34:03.328 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj4 2022-09-12T19:34:03.329 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:47JkU+hZRL1YbMa5EdtQNzu7F7c= 2022-09-12T19:34:03.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:47JkU+hZRL1YbMa5EdtQNzu7F7c=', 'Content-Length': '0'} 2022-09-12T19:34:03.332 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000afcb7f52f2fd0880-00631f89ab-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-12T19:34:03.333 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:03.333 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:03.334 DEBUG:boto:Method: GET 2022-09-12T19:34:03.334 DEBUG:boto:Path: /jdpiak-63/obj4 2022-09-12T19:34:03.335 DEBUG:boto:Data: 2022-09-12T19:34:03.336 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.336 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.337 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.337 DEBUG:boto:Params: {} 2022-09-12T19:34:03.338 DEBUG:boto:Token: None 2022-09-12T19:34:03.338 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj4 2022-09-12T19:34:03.339 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:drTME+PQ6mRn5tzi+yfHS2H3qbE= 2022-09-12T19:34:03.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:drTME+PQ6mRn5tzi+yfHS2H3qbE=', 'Content-Length': '0'} 2022-09-12T19:34:03.342 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d397d34b5b061fda-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.382 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:03.383 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:03.383 DEBUG:boto:Method: GET 2022-09-12T19:34:03.384 DEBUG:boto:Path: /jdpiak-63/obj4 2022-09-12T19:34:03.384 DEBUG:boto:Data: 2022-09-12T19:34:03.385 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.385 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:03.386 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.387 DEBUG:boto:Params: {} 2022-09-12T19:34:03.387 DEBUG:boto:Token: None 2022-09-12T19:34:03.388 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj4 2022-09-12T19:34:03.388 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:drTME+PQ6mRn5tzi+yfHS2H3qbE= 2022-09-12T19:34:03.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:drTME+PQ6mRn5tzi+yfHS2H3qbE=', 'Content-Length': '0'} 2022-09-12T19:34:03.392 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2cdb2ebf20740d0-00631f89ab-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.392 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:03.393 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:03.393 DEBUG:boto:path=/jdpiak-63/?prefix=obj4 2022-09-12T19:34:03.394 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj4 2022-09-12T19:34:03.394 DEBUG:boto:Method: GET 2022-09-12T19:34:03.395 DEBUG:boto:Path: /jdpiak-63/?prefix=obj4 2022-09-12T19:34:03.396 DEBUG:boto:Data: 2022-09-12T19:34:03.396 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.397 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.397 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.398 DEBUG:boto:Params: {} 2022-09-12T19:34:03.398 DEBUG:boto:Token: None 2022-09-12T19:34:03.399 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/ 2022-09-12T19:34:03.399 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA= 2022-09-12T19:34:03.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA=', 'Content-Length': '0'} 2022-09-12T19:34:03.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1e0a450df07b60f-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.403 DEBUG:boto:b'jdpiak-63obj41000falseobj42022-09-12T19:32:15.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:03.404 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:03.405 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:03.405 DEBUG:boto:path=/jdpiak-63/?prefix=obj4 2022-09-12T19:34:03.406 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj4 2022-09-12T19:34:03.406 DEBUG:boto:Method: GET 2022-09-12T19:34:03.407 DEBUG:boto:Path: /jdpiak-63/?prefix=obj4 2022-09-12T19:34:03.408 DEBUG:boto:Data: 2022-09-12T19:34:03.408 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.409 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:34:03.409 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.410 DEBUG:boto:Params: {} 2022-09-12T19:34:03.410 DEBUG:boto:Token: None 2022-09-12T19:34:03.411 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/ 2022-09-12T19:34:03.411 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA= 2022-09-12T19:34:03.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA=', 'Content-Length': '0'} 2022-09-12T19:34:03.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e31d155708576b3e-00631f89ab-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.415 DEBUG:boto:b'jdpiak-63obj41000falseobj42022-09-12T19:32:15.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:03.416 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-63 zones={test-zone3, test-zone1} 2022-09-12T19:34:03.417 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-63 zones={test-zone3, test-zone2} 2022-09-12T19:34:03.417 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:03.418 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:03.418 DEBUG:boto:Method: HEAD 2022-09-12T19:34:03.419 DEBUG:boto:Path: /jdpiak-63/ 2022-09-12T19:34:03.419 DEBUG:boto:Data: 2022-09-12T19:34:03.420 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.420 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.421 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.421 DEBUG:boto:Params: {} 2022-09-12T19:34:03.422 DEBUG:boto:Token: None 2022-09-12T19:34:03.423 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/ 2022-09-12T19:34:03.423 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q/qV980JUqC53ud6sAOPk9nc1ZU= 2022-09-12T19:34:03.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q/qV980JUqC53ud6sAOPk9nc1ZU=', 'Content-Length': '0'} 2022-09-12T19:34:03.426 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3bf37d8aedda9d3-00631f89ab-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.427 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:03.427 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:03.428 DEBUG:boto:Method: HEAD 2022-09-12T19:34:03.428 DEBUG:boto:Path: /jdpiak-63/ 2022-09-12T19:34:03.429 DEBUG:boto:Data: 2022-09-12T19:34:03.429 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.430 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:03.430 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.431 DEBUG:boto:Params: {} 2022-09-12T19:34:03.432 DEBUG:boto:Token: None 2022-09-12T19:34:03.432 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/ 2022-09-12T19:34:03.433 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q/qV980JUqC53ud6sAOPk9nc1ZU= 2022-09-12T19:34:03.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q/qV980JUqC53ud6sAOPk9nc1ZU=', 'Content-Length': '0'} 2022-09-12T19:34:03.436 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f209ef830c77ee9-00631f89ab-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.437 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:34:03.437 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:03.438 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:03.438 DEBUG:boto:path=/jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:03.439 DEBUG:boto:auth_path=/jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:03.439 DEBUG:boto:Method: GET 2022-09-12T19:34:03.440 DEBUG:boto:Path: /jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:03.441 DEBUG:boto:Data: 2022-09-12T19:34:03.441 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.442 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.442 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.443 DEBUG:boto:Params: {} 2022-09-12T19:34:03.443 DEBUG:boto:Token: None 2022-09-12T19:34:03.444 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/?versions 2022-09-12T19:34:03.444 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3fTpVURfAyvacEl7QNdoQlLg1jg= 2022-09-12T19:34:03.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3fTpVURfAyvacEl7QNdoQlLg1jg=', 'Content-Length': '0'} 2022-09-12T19:34:03.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c5c8710ebdd0c3e-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.449 DEBUG:boto:b'jdpiak-63999falseobj1nulltrue2022-09-12T19:32:15.261Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-12T19:32:15.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-12T19:32:15.326Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-12T19:32:15.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:03.450 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-12T19:34:03.450 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-12T19:34:03.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-12T19:34:03.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-12T19:34:03.452 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:34:03.452 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:03.453 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:03.453 DEBUG:boto:path=/jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:03.454 DEBUG:boto:auth_path=/jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:03.455 DEBUG:boto:Method: GET 2022-09-12T19:34:03.455 DEBUG:boto:Path: /jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:03.456 DEBUG:boto:Data: 2022-09-12T19:34:03.457 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.457 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:03.458 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.458 DEBUG:boto:Params: {} 2022-09-12T19:34:03.459 DEBUG:boto:Token: None 2022-09-12T19:34:03.459 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/?versions 2022-09-12T19:34:03.460 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3fTpVURfAyvacEl7QNdoQlLg1jg= 2022-09-12T19:34:03.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3fTpVURfAyvacEl7QNdoQlLg1jg=', 'Content-Length': '0'} 2022-09-12T19:34:03.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca8839cd40f7fc7d-00631f89ab-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.464 DEBUG:boto:b'jdpiak-63999falseobj1nulltrue2022-09-12T19:32:15.261Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-12T19:32:15.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-12T19:32:15.326Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-12T19:32:15.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:03.465 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-12T19:34:03.466 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-12T19:34:03.466 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-12T19:34:03.467 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-12T19:34:03.467 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:03.468 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:03.469 DEBUG:boto:path=/jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:03.469 DEBUG:boto:auth_path=/jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:03.470 DEBUG:boto:Method: GET 2022-09-12T19:34:03.470 DEBUG:boto:Path: /jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:03.471 DEBUG:boto:Data: 2022-09-12T19:34:03.471 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.472 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.472 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.473 DEBUG:boto:Params: {} 2022-09-12T19:34:03.473 DEBUG:boto:Token: None 2022-09-12T19:34:03.474 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/?versions 2022-09-12T19:34:03.474 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3fTpVURfAyvacEl7QNdoQlLg1jg= 2022-09-12T19:34:03.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3fTpVURfAyvacEl7QNdoQlLg1jg=', 'Content-Length': '0'} 2022-09-12T19:34:03.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e0194e19ba5202b-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.479 DEBUG:boto:b'jdpiak-63999falseobj1nulltrue2022-09-12T19:32:15.261Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-12T19:32:15.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-12T19:32:15.326Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-12T19:32:15.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:03.480 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:03.480 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:03.481 DEBUG:boto:path=/jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:03.481 DEBUG:boto:auth_path=/jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:03.482 DEBUG:boto:Method: GET 2022-09-12T19:34:03.482 DEBUG:boto:Path: /jdpiak-63/?versions&max-keys=999 2022-09-12T19:34:03.483 DEBUG:boto:Data: 2022-09-12T19:34:03.483 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.484 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:03.484 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.485 DEBUG:boto:Params: {} 2022-09-12T19:34:03.485 DEBUG:boto:Token: None 2022-09-12T19:34:03.486 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/?versions 2022-09-12T19:34:03.486 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3fTpVURfAyvacEl7QNdoQlLg1jg= 2022-09-12T19:34:03.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3fTpVURfAyvacEl7QNdoQlLg1jg=', 'Content-Length': '0'} 2022-09-12T19:34:03.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d27ffe37d8198c98-00631f89ab-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.491 DEBUG:boto:b'jdpiak-63999falseobj1nulltrue2022-09-12T19:32:15.261Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-12T19:32:15.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-12T19:32:15.326Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-12T19:32:15.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:03.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-12T19:34:03.492 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:03.493 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:03.493 DEBUG:boto:path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:03.494 DEBUG:boto:auth_path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:03.494 DEBUG:boto:Method: GET 2022-09-12T19:34:03.495 DEBUG:boto:Path: /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:03.495 DEBUG:boto:Data: 2022-09-12T19:34:03.496 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.496 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.497 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.497 DEBUG:boto:Params: {} 2022-09-12T19:34:03.498 DEBUG:boto:Token: None 2022-09-12T19:34:03.499 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:03.499 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1wGzzZBJI3mJemdSZjuoLwu/RuQ= 2022-09-12T19:34:03.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1wGzzZBJI3mJemdSZjuoLwu/RuQ=', 'Content-Length': '0'} 2022-09-12T19:34:03.502 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f3781058d12d669-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.546 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:03.547 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:03.547 DEBUG:boto:path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:03.548 DEBUG:boto:auth_path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:03.549 DEBUG:boto:Method: GET 2022-09-12T19:34:03.549 DEBUG:boto:Path: /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:03.550 DEBUG:boto:Data: 2022-09-12T19:34:03.550 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.551 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:03.552 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.552 DEBUG:boto:Params: {} 2022-09-12T19:34:03.553 DEBUG:boto:Token: None 2022-09-12T19:34:03.553 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:03.554 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1wGzzZBJI3mJemdSZjuoLwu/RuQ= 2022-09-12T19:34:03.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1wGzzZBJI3mJemdSZjuoLwu/RuQ=', 'Content-Length': '0'} 2022-09-12T19:34:03.557 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000371d479b6a1c0147-00631f89ab-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.558 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:03.559 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:03.559 DEBUG:boto:path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:03.560 DEBUG:boto:auth_path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:03.560 DEBUG:boto:Method: HEAD 2022-09-12T19:34:03.561 DEBUG:boto:Path: /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:03.561 DEBUG:boto:Data: 2022-09-12T19:34:03.562 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.562 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.563 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.564 DEBUG:boto:Params: {} 2022-09-12T19:34:03.564 DEBUG:boto:Token: None 2022-09-12T19:34:03.565 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:03.565 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ho37FMW2NLCECbk9U7w6kMSyhGk= 2022-09-12T19:34:03.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ho37FMW2NLCECbk9U7w6kMSyhGk=', 'Content-Length': '0'} 2022-09-12T19:34:03.568 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6f7e93aa0061975-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.569 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:03.569 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:03.570 DEBUG:boto:path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:03.571 DEBUG:boto:auth_path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:03.571 DEBUG:boto:Method: HEAD 2022-09-12T19:34:03.572 DEBUG:boto:Path: /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:03.572 DEBUG:boto:Data: 2022-09-12T19:34:03.573 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.573 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:03.574 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.574 DEBUG:boto:Params: {} 2022-09-12T19:34:03.575 DEBUG:boto:Token: None 2022-09-12T19:34:03.575 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:03.576 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ho37FMW2NLCECbk9U7w6kMSyhGk= 2022-09-12T19:34:03.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ho37FMW2NLCECbk9U7w6kMSyhGk=', 'Content-Length': '0'} 2022-09-12T19:34:03.578 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006efbb8ba644c8f7e-00631f89ab-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-12T19:34:03.580 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:03.580 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:03.581 DEBUG:boto:path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:03.581 DEBUG:boto:auth_path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:03.582 DEBUG:boto:Method: GET 2022-09-12T19:34:03.583 DEBUG:boto:Path: /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:03.583 DEBUG:boto:Data: 2022-09-12T19:34:03.584 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.584 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.585 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.585 DEBUG:boto:Params: {} 2022-09-12T19:34:03.586 DEBUG:boto:Token: None 2022-09-12T19:34:03.586 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:03.587 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1wGzzZBJI3mJemdSZjuoLwu/RuQ= 2022-09-12T19:34:03.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1wGzzZBJI3mJemdSZjuoLwu/RuQ=', 'Content-Length': '0'} 2022-09-12T19:34:03.590 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d5f21a92bbaddde-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.591 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:03.591 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:03.592 DEBUG:boto:path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:03.592 DEBUG:boto:auth_path=/jdpiak-63/obj1?versionId=null 2022-09-12T19:34:03.593 DEBUG:boto:Method: GET 2022-09-12T19:34:03.593 DEBUG:boto:Path: /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:03.594 DEBUG:boto:Data: 2022-09-12T19:34:03.595 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.595 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:03.596 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.596 DEBUG:boto:Params: {} 2022-09-12T19:34:03.597 DEBUG:boto:Token: None 2022-09-12T19:34:03.598 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj1?versionId=null 2022-09-12T19:34:03.598 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1wGzzZBJI3mJemdSZjuoLwu/RuQ= 2022-09-12T19:34:03.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1wGzzZBJI3mJemdSZjuoLwu/RuQ=', 'Content-Length': '0'} 2022-09-12T19:34:03.601 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b63ccadf4965cb39-00631f89ab-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.642 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:03.643 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:03.643 DEBUG:boto:path=/jdpiak-63/?prefix=obj1 2022-09-12T19:34:03.644 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj1 2022-09-12T19:34:03.645 DEBUG:boto:Method: GET 2022-09-12T19:34:03.645 DEBUG:boto:Path: /jdpiak-63/?prefix=obj1 2022-09-12T19:34:03.646 DEBUG:boto:Data: 2022-09-12T19:34:03.646 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.647 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.647 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.648 DEBUG:boto:Params: {} 2022-09-12T19:34:03.649 DEBUG:boto:Token: None 2022-09-12T19:34:03.649 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/ 2022-09-12T19:34:03.650 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA= 2022-09-12T19:34:03.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA=', 'Content-Length': '0'} 2022-09-12T19:34:03.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d5518bc864408e9-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.654 DEBUG:boto:b'jdpiak-63obj11000falseobj12022-09-12T19:32:15.261Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:03.655 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:03.656 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:03.656 DEBUG:boto:path=/jdpiak-63/?prefix=obj1 2022-09-12T19:34:03.657 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj1 2022-09-12T19:34:03.657 DEBUG:boto:Method: GET 2022-09-12T19:34:03.658 DEBUG:boto:Path: /jdpiak-63/?prefix=obj1 2022-09-12T19:34:03.659 DEBUG:boto:Data: 2022-09-12T19:34:03.659 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.660 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:03.660 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.661 DEBUG:boto:Params: {} 2022-09-12T19:34:03.662 DEBUG:boto:Token: None 2022-09-12T19:34:03.662 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/ 2022-09-12T19:34:03.663 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA= 2022-09-12T19:34:03.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA=', 'Content-Length': '0'} 2022-09-12T19:34:03.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d835b5f9fba8f687-00631f89ab-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.667 DEBUG:boto:b'jdpiak-63obj11000falseobj12022-09-12T19:32:15.261Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:03.668 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:03.669 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:03.669 DEBUG:boto:Method: HEAD 2022-09-12T19:34:03.670 DEBUG:boto:Path: /jdpiak-63/obj1 2022-09-12T19:34:03.670 DEBUG:boto:Data: 2022-09-12T19:34:03.671 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.671 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.672 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.673 DEBUG:boto:Params: {} 2022-09-12T19:34:03.673 DEBUG:boto:Token: None 2022-09-12T19:34:03.674 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj1 2022-09-12T19:34:03.674 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ch4ZGkKA8QRxjRIjpLK5bg13kLc= 2022-09-12T19:34:03.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ch4ZGkKA8QRxjRIjpLK5bg13kLc=', 'Content-Length': '0'} 2022-09-12T19:34:03.677 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046bbcae88b1eefdb-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.678 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:03.679 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:03.679 DEBUG:boto:Method: HEAD 2022-09-12T19:34:03.680 DEBUG:boto:Path: /jdpiak-63/obj1 2022-09-12T19:34:03.680 DEBUG:boto:Data: 2022-09-12T19:34:03.681 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.681 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:03.682 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.682 DEBUG:boto:Params: {} 2022-09-12T19:34:03.683 DEBUG:boto:Token: None 2022-09-12T19:34:03.684 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj1 2022-09-12T19:34:03.684 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ch4ZGkKA8QRxjRIjpLK5bg13kLc= 2022-09-12T19:34:03.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ch4ZGkKA8QRxjRIjpLK5bg13kLc=', 'Content-Length': '0'} 2022-09-12T19:34:03.687 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000489b7f530f1b9810-00631f89ab-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-12T19:34:03.689 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:03.690 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:03.690 DEBUG:boto:Method: GET 2022-09-12T19:34:03.691 DEBUG:boto:Path: /jdpiak-63/obj1 2022-09-12T19:34:03.692 DEBUG:boto:Data: 2022-09-12T19:34:03.692 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.693 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.693 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.694 DEBUG:boto:Params: {} 2022-09-12T19:34:03.695 DEBUG:boto:Token: None 2022-09-12T19:34:03.695 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj1 2022-09-12T19:34:03.696 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:875YBdMgBdAa/CBEl1HBusPKutc= 2022-09-12T19:34:03.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:875YBdMgBdAa/CBEl1HBusPKutc=', 'Content-Length': '0'} 2022-09-12T19:34:03.699 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061ec15451d2221a1-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.700 DEBUG:boto:path=/jdpiak-63/obj1 2022-09-12T19:34:03.700 DEBUG:boto:auth_path=/jdpiak-63/obj1 2022-09-12T19:34:03.701 DEBUG:boto:Method: GET 2022-09-12T19:34:03.701 DEBUG:boto:Path: /jdpiak-63/obj1 2022-09-12T19:34:03.702 DEBUG:boto:Data: 2022-09-12T19:34:03.703 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.703 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:03.704 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.704 DEBUG:boto:Params: {} 2022-09-12T19:34:03.705 DEBUG:boto:Token: None 2022-09-12T19:34:03.706 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj1 2022-09-12T19:34:03.706 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:875YBdMgBdAa/CBEl1HBusPKutc= 2022-09-12T19:34:03.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:875YBdMgBdAa/CBEl1HBusPKutc=', 'Content-Length': '0'} 2022-09-12T19:34:03.709 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b563616f4998fffa-00631f89ab-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.710 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:03.711 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:03.712 DEBUG:boto:path=/jdpiak-63/?prefix=obj1 2022-09-12T19:34:03.712 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj1 2022-09-12T19:34:03.713 DEBUG:boto:Method: GET 2022-09-12T19:34:03.713 DEBUG:boto:Path: /jdpiak-63/?prefix=obj1 2022-09-12T19:34:03.714 DEBUG:boto:Data: 2022-09-12T19:34:03.715 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.715 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.716 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.717 DEBUG:boto:Params: {} 2022-09-12T19:34:03.717 DEBUG:boto:Token: None 2022-09-12T19:34:03.718 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/ 2022-09-12T19:34:03.719 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA= 2022-09-12T19:34:03.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA=', 'Content-Length': '0'} 2022-09-12T19:34:03.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfab80b62a736a0c-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.723 DEBUG:boto:b'jdpiak-63obj11000falseobj12022-09-12T19:32:15.261Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:03.724 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:03.725 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:03.726 DEBUG:boto:path=/jdpiak-63/?prefix=obj1 2022-09-12T19:34:03.726 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj1 2022-09-12T19:34:03.727 DEBUG:boto:Method: GET 2022-09-12T19:34:03.727 DEBUG:boto:Path: /jdpiak-63/?prefix=obj1 2022-09-12T19:34:03.728 DEBUG:boto:Data: 2022-09-12T19:34:03.728 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.729 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:03.729 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.730 DEBUG:boto:Params: {} 2022-09-12T19:34:03.731 DEBUG:boto:Token: None 2022-09-12T19:34:03.731 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/ 2022-09-12T19:34:03.732 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA= 2022-09-12T19:34:03.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA=', 'Content-Length': '0'} 2022-09-12T19:34:03.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f6209d59d5f69a4-00631f89ab-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.737 DEBUG:boto:b'jdpiak-63obj11000falseobj12022-09-12T19:32:15.261Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:03.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-12T19:34:03.738 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:03.739 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:03.739 DEBUG:boto:path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:03.740 DEBUG:boto:auth_path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:03.740 DEBUG:boto:Method: GET 2022-09-12T19:34:03.741 DEBUG:boto:Path: /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:03.741 DEBUG:boto:Data: 2022-09-12T19:34:03.742 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.742 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.743 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.743 DEBUG:boto:Params: {} 2022-09-12T19:34:03.744 DEBUG:boto:Token: None 2022-09-12T19:34:03.745 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:03.745 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QDwsZD7pZ1ObgYvMeTv3irTjBTo= 2022-09-12T19:34:03.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QDwsZD7pZ1ObgYvMeTv3irTjBTo=', 'Content-Length': '0'} 2022-09-12T19:34:03.748 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae5f5581b052bbfb-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.790 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:03.791 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:03.791 DEBUG:boto:path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:03.792 DEBUG:boto:auth_path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:03.792 DEBUG:boto:Method: GET 2022-09-12T19:34:03.793 DEBUG:boto:Path: /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:03.794 DEBUG:boto:Data: 2022-09-12T19:34:03.795 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.796 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:03.797 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.798 DEBUG:boto:Params: {} 2022-09-12T19:34:03.799 DEBUG:boto:Token: None 2022-09-12T19:34:03.800 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:03.801 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QDwsZD7pZ1ObgYvMeTv3irTjBTo= 2022-09-12T19:34:03.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QDwsZD7pZ1ObgYvMeTv3irTjBTo=', 'Content-Length': '0'} 2022-09-12T19:34:03.805 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d17c59552fb5fad-00631f89ab-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.806 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:03.807 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:03.808 DEBUG:boto:path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:03.809 DEBUG:boto:auth_path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:03.810 DEBUG:boto:Method: HEAD 2022-09-12T19:34:03.811 DEBUG:boto:Path: /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:03.812 DEBUG:boto:Data: 2022-09-12T19:34:03.813 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.814 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.815 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.816 DEBUG:boto:Params: {} 2022-09-12T19:34:03.817 DEBUG:boto:Token: None 2022-09-12T19:34:03.818 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:03.819 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jIs04oVPy5sgsTSpdhklNXv1464= 2022-09-12T19:34:03.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jIs04oVPy5sgsTSpdhklNXv1464=', 'Content-Length': '0'} 2022-09-12T19:34:03.823 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001799e4c3128c4339-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.824 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:03.825 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:03.827 DEBUG:boto:path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:03.828 DEBUG:boto:auth_path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:03.829 DEBUG:boto:Method: HEAD 2022-09-12T19:34:03.829 DEBUG:boto:Path: /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:03.830 DEBUG:boto:Data: 2022-09-12T19:34:03.831 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.832 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:03.833 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.834 DEBUG:boto:Params: {} 2022-09-12T19:34:03.844 DEBUG:boto:Token: None 2022-09-12T19:34:03.845 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:03.845 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jIs04oVPy5sgsTSpdhklNXv1464= 2022-09-12T19:34:03.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jIs04oVPy5sgsTSpdhklNXv1464=', 'Content-Length': '0'} 2022-09-12T19:34:03.849 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5f45acd831d25e9-00631f89ab-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-12T19:34:03.850 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:03.851 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:03.852 DEBUG:boto:path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:03.852 DEBUG:boto:auth_path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:03.853 DEBUG:boto:Method: GET 2022-09-12T19:34:03.854 DEBUG:boto:Path: /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:03.855 DEBUG:boto:Data: 2022-09-12T19:34:03.856 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.857 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.857 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.858 DEBUG:boto:Params: {} 2022-09-12T19:34:03.859 DEBUG:boto:Token: None 2022-09-12T19:34:03.859 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:03.860 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QDwsZD7pZ1ObgYvMeTv3irTjBTo= 2022-09-12T19:34:03.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QDwsZD7pZ1ObgYvMeTv3irTjBTo=', 'Content-Length': '0'} 2022-09-12T19:34:03.863 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c49647db3a393070-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.864 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:03.865 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:03.865 DEBUG:boto:path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:03.866 DEBUG:boto:auth_path=/jdpiak-63/obj2?versionId=null 2022-09-12T19:34:03.866 DEBUG:boto:Method: GET 2022-09-12T19:34:03.867 DEBUG:boto:Path: /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:03.867 DEBUG:boto:Data: 2022-09-12T19:34:03.868 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.869 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:03.870 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.870 DEBUG:boto:Params: {} 2022-09-12T19:34:03.871 DEBUG:boto:Token: None 2022-09-12T19:34:03.872 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj2?versionId=null 2022-09-12T19:34:03.873 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QDwsZD7pZ1ObgYvMeTv3irTjBTo= 2022-09-12T19:34:03.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QDwsZD7pZ1ObgYvMeTv3irTjBTo=', 'Content-Length': '0'} 2022-09-12T19:34:03.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c81f90ea004b82b-00631f89ab-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.877 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:03.878 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:03.879 DEBUG:boto:path=/jdpiak-63/?prefix=obj2 2022-09-12T19:34:03.880 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj2 2022-09-12T19:34:03.882 DEBUG:boto:Method: GET 2022-09-12T19:34:03.883 DEBUG:boto:Path: /jdpiak-63/?prefix=obj2 2022-09-12T19:34:03.884 DEBUG:boto:Data: 2022-09-12T19:34:03.885 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.886 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.887 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.888 DEBUG:boto:Params: {} 2022-09-12T19:34:03.889 DEBUG:boto:Token: None 2022-09-12T19:34:03.890 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/ 2022-09-12T19:34:03.891 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA= 2022-09-12T19:34:03.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA=', 'Content-Length': '0'} 2022-09-12T19:34:03.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a92ee2b5659323b1-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.897 DEBUG:boto:b'jdpiak-63obj21000falseobj22022-09-12T19:32:15.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:03.899 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:03.900 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:03.901 DEBUG:boto:path=/jdpiak-63/?prefix=obj2 2022-09-12T19:34:03.901 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj2 2022-09-12T19:34:03.902 DEBUG:boto:Method: GET 2022-09-12T19:34:03.902 DEBUG:boto:Path: /jdpiak-63/?prefix=obj2 2022-09-12T19:34:03.903 DEBUG:boto:Data: 2022-09-12T19:34:03.903 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.904 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:03.904 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.905 DEBUG:boto:Params: {} 2022-09-12T19:34:03.905 DEBUG:boto:Token: None 2022-09-12T19:34:03.906 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/ 2022-09-12T19:34:03.907 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA= 2022-09-12T19:34:03.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCgwOXI1JZ2VlDVQlQ9Tw19n2cA=', 'Content-Length': '0'} 2022-09-12T19:34:03.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcb88084f71c0436-00631f89ab-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.911 DEBUG:boto:b'jdpiak-63obj21000falseobj22022-09-12T19:32:15.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:03.911 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:03.912 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:03.912 DEBUG:boto:Method: HEAD 2022-09-12T19:34:03.913 DEBUG:boto:Path: /jdpiak-63/obj2 2022-09-12T19:34:03.913 DEBUG:boto:Data: 2022-09-12T19:34:03.914 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.915 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.915 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.916 DEBUG:boto:Params: {} 2022-09-12T19:34:03.916 DEBUG:boto:Token: None 2022-09-12T19:34:03.917 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj2 2022-09-12T19:34:03.917 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4Uv2FsV8hj+5QKFwNDte3AcuyXo= 2022-09-12T19:34:03.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4Uv2FsV8hj+5QKFwNDte3AcuyXo=', 'Content-Length': '0'} 2022-09-12T19:34:03.920 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b6d56fb891e07d8f-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.921 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:03.922 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:03.922 DEBUG:boto:Method: HEAD 2022-09-12T19:34:03.923 DEBUG:boto:Path: /jdpiak-63/obj2 2022-09-12T19:34:03.924 DEBUG:boto:Data: 2022-09-12T19:34:03.924 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.925 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:03.925 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.926 DEBUG:boto:Params: {} 2022-09-12T19:34:03.927 DEBUG:boto:Token: None 2022-09-12T19:34:03.927 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj2 2022-09-12T19:34:03.928 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4Uv2FsV8hj+5QKFwNDte3AcuyXo= 2022-09-12T19:34:03.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4Uv2FsV8hj+5QKFwNDte3AcuyXo=', 'Content-Length': '0'} 2022-09-12T19:34:03.931 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef9349c49f56a82d-00631f89ab-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-12T19:34:03.933 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:03.934 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:03.935 DEBUG:boto:Method: GET 2022-09-12T19:34:03.936 DEBUG:boto:Path: /jdpiak-63/obj2 2022-09-12T19:34:03.937 DEBUG:boto:Data: 2022-09-12T19:34:03.938 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.939 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:03.940 DEBUG:boto:Port: 8001 2022-09-12T19:34:03.941 DEBUG:boto:Params: {} 2022-09-12T19:34:03.942 DEBUG:boto:Token: None 2022-09-12T19:34:03.943 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj2 2022-09-12T19:34:03.944 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mZWiiv9Oeflf0ijXlhRd9BH7U3c= 2022-09-12T19:34:03.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mZWiiv9Oeflf0ijXlhRd9BH7U3c=', 'Content-Length': '0'} 2022-09-12T19:34:03.948 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006bcc5d1d3cba7832-00631f89ab-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:03.990 DEBUG:boto:path=/jdpiak-63/obj2 2022-09-12T19:34:03.991 DEBUG:boto:auth_path=/jdpiak-63/obj2 2022-09-12T19:34:03.992 DEBUG:boto:Method: GET 2022-09-12T19:34:03.992 DEBUG:boto:Path: /jdpiak-63/obj2 2022-09-12T19:34:03.993 DEBUG:boto:Data: 2022-09-12T19:34:03.993 DEBUG:boto:Headers: {} 2022-09-12T19:34:03.994 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:03.994 DEBUG:boto:Port: 8000 2022-09-12T19:34:03.995 DEBUG:boto:Params: {} 2022-09-12T19:34:03.995 DEBUG:boto:Token: None 2022-09-12T19:34:03.996 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:03 GMT /jdpiak-63/obj2 2022-09-12T19:34:03.997 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mZWiiv9Oeflf0ijXlhRd9BH7U3c= 2022-09-12T19:34:03.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mZWiiv9Oeflf0ijXlhRd9BH7U3c=', 'Content-Length': '0'} 2022-09-12T19:34:03.999 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a120d3fe60042f3-00631f89ab-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.000 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:04.001 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:04.001 DEBUG:boto:path=/jdpiak-63/?prefix=obj2 2022-09-12T19:34:04.002 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj2 2022-09-12T19:34:04.002 DEBUG:boto:Method: GET 2022-09-12T19:34:04.003 DEBUG:boto:Path: /jdpiak-63/?prefix=obj2 2022-09-12T19:34:04.003 DEBUG:boto:Data: 2022-09-12T19:34:04.004 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.004 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:04.005 DEBUG:boto:Port: 8001 2022-09-12T19:34:04.005 DEBUG:boto:Params: {} 2022-09-12T19:34:04.006 DEBUG:boto:Token: None 2022-09-12T19:34:04.007 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/ 2022-09-12T19:34:04.007 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:g+b9Suv2JJvbhKbCijOwk7Lzhxc= 2022-09-12T19:34:04.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:g+b9Suv2JJvbhKbCijOwk7Lzhxc=', 'Content-Length': '0'} 2022-09-12T19:34:04.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f728084bedd5199f-00631f89ac-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.012 DEBUG:boto:b'jdpiak-63obj21000falseobj22022-09-12T19:32:15.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:04.013 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:04.013 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:04.014 DEBUG:boto:path=/jdpiak-63/?prefix=obj2 2022-09-12T19:34:04.014 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj2 2022-09-12T19:34:04.015 DEBUG:boto:Method: GET 2022-09-12T19:34:04.016 DEBUG:boto:Path: /jdpiak-63/?prefix=obj2 2022-09-12T19:34:04.016 DEBUG:boto:Data: 2022-09-12T19:34:04.017 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.017 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:04.018 DEBUG:boto:Port: 8000 2022-09-12T19:34:04.018 DEBUG:boto:Params: {} 2022-09-12T19:34:04.019 DEBUG:boto:Token: None 2022-09-12T19:34:04.019 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/ 2022-09-12T19:34:04.020 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:g+b9Suv2JJvbhKbCijOwk7Lzhxc= 2022-09-12T19:34:04.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:g+b9Suv2JJvbhKbCijOwk7Lzhxc=', 'Content-Length': '0'} 2022-09-12T19:34:04.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000023a63b485e6739c-00631f89ac-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.027 DEBUG:boto:b'jdpiak-63obj21000falseobj22022-09-12T19:32:15.296Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:04.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-12T19:34:04.029 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:04.030 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:04.030 DEBUG:boto:path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:04.031 DEBUG:boto:auth_path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:04.032 DEBUG:boto:Method: GET 2022-09-12T19:34:04.033 DEBUG:boto:Path: /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:04.033 DEBUG:boto:Data: 2022-09-12T19:34:04.034 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.035 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:04.036 DEBUG:boto:Port: 8001 2022-09-12T19:34:04.036 DEBUG:boto:Params: {} 2022-09-12T19:34:04.037 DEBUG:boto:Token: None 2022-09-12T19:34:04.038 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:04.039 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:su5TW24Skw7bkCWUV1TQLTSE8Ro= 2022-09-12T19:34:04.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:su5TW24Skw7bkCWUV1TQLTSE8Ro=', 'Content-Length': '0'} 2022-09-12T19:34:04.042 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002866d649cd623a7b-00631f89ac-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.042 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:04.043 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:04.044 DEBUG:boto:path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:04.044 DEBUG:boto:auth_path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:04.045 DEBUG:boto:Method: GET 2022-09-12T19:34:04.046 DEBUG:boto:Path: /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:04.047 DEBUG:boto:Data: 2022-09-12T19:34:04.047 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.048 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:04.049 DEBUG:boto:Port: 8000 2022-09-12T19:34:04.049 DEBUG:boto:Params: {} 2022-09-12T19:34:04.050 DEBUG:boto:Token: None 2022-09-12T19:34:04.051 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:04.052 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:su5TW24Skw7bkCWUV1TQLTSE8Ro= 2022-09-12T19:34:04.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:su5TW24Skw7bkCWUV1TQLTSE8Ro=', 'Content-Length': '0'} 2022-09-12T19:34:04.055 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c51729f9d3302945-00631f89ac-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.056 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:04.056 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:04.057 DEBUG:boto:path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:04.057 DEBUG:boto:auth_path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:04.058 DEBUG:boto:Method: HEAD 2022-09-12T19:34:04.058 DEBUG:boto:Path: /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:04.059 DEBUG:boto:Data: 2022-09-12T19:34:04.059 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.060 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:04.061 DEBUG:boto:Port: 8001 2022-09-12T19:34:04.061 DEBUG:boto:Params: {} 2022-09-12T19:34:04.062 DEBUG:boto:Token: None 2022-09-12T19:34:04.063 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:04.063 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hEDUSMf/73Oz7gxWjVc/ew3Nwv4= 2022-09-12T19:34:04.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hEDUSMf/73Oz7gxWjVc/ew3Nwv4=', 'Content-Length': '0'} 2022-09-12T19:34:04.066 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a51711388686940-00631f89ac-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.067 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:04.068 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:04.068 DEBUG:boto:path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:04.069 DEBUG:boto:auth_path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:04.069 DEBUG:boto:Method: HEAD 2022-09-12T19:34:04.070 DEBUG:boto:Path: /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:04.071 DEBUG:boto:Data: 2022-09-12T19:34:04.071 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.072 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:04.072 DEBUG:boto:Port: 8000 2022-09-12T19:34:04.073 DEBUG:boto:Params: {} 2022-09-12T19:34:04.073 DEBUG:boto:Token: None 2022-09-12T19:34:04.074 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:04.075 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hEDUSMf/73Oz7gxWjVc/ew3Nwv4= 2022-09-12T19:34:04.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hEDUSMf/73Oz7gxWjVc/ew3Nwv4=', 'Content-Length': '0'} 2022-09-12T19:34:04.077 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5dd5a5d573cbf43-00631f89ac-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-12T19:34:04.079 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:04.079 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:04.080 DEBUG:boto:path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:04.080 DEBUG:boto:auth_path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:04.081 DEBUG:boto:Method: GET 2022-09-12T19:34:04.081 DEBUG:boto:Path: /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:04.082 DEBUG:boto:Data: 2022-09-12T19:34:04.082 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.083 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:04.083 DEBUG:boto:Port: 8001 2022-09-12T19:34:04.084 DEBUG:boto:Params: {} 2022-09-12T19:34:04.085 DEBUG:boto:Token: None 2022-09-12T19:34:04.085 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:04.086 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:su5TW24Skw7bkCWUV1TQLTSE8Ro= 2022-09-12T19:34:04.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:su5TW24Skw7bkCWUV1TQLTSE8Ro=', 'Content-Length': '0'} 2022-09-12T19:34:04.089 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090d9e032dcb747dd-00631f89ac-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.130 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:04.131 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:04.132 DEBUG:boto:path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:04.132 DEBUG:boto:auth_path=/jdpiak-63/obj3?versionId=null 2022-09-12T19:34:04.133 DEBUG:boto:Method: GET 2022-09-12T19:34:04.134 DEBUG:boto:Path: /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:04.134 DEBUG:boto:Data: 2022-09-12T19:34:04.135 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.135 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:04.136 DEBUG:boto:Port: 8000 2022-09-12T19:34:04.136 DEBUG:boto:Params: {} 2022-09-12T19:34:04.137 DEBUG:boto:Token: None 2022-09-12T19:34:04.138 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/obj3?versionId=null 2022-09-12T19:34:04.138 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:su5TW24Skw7bkCWUV1TQLTSE8Ro= 2022-09-12T19:34:04.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:su5TW24Skw7bkCWUV1TQLTSE8Ro=', 'Content-Length': '0'} 2022-09-12T19:34:04.141 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028803282cee80e0f-00631f89ac-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.142 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:04.142 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:04.143 DEBUG:boto:path=/jdpiak-63/?prefix=obj3 2022-09-12T19:34:04.143 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj3 2022-09-12T19:34:04.144 DEBUG:boto:Method: GET 2022-09-12T19:34:04.145 DEBUG:boto:Path: /jdpiak-63/?prefix=obj3 2022-09-12T19:34:04.145 DEBUG:boto:Data: 2022-09-12T19:34:04.146 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.146 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:04.147 DEBUG:boto:Port: 8001 2022-09-12T19:34:04.147 DEBUG:boto:Params: {} 2022-09-12T19:34:04.148 DEBUG:boto:Token: None 2022-09-12T19:34:04.148 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/ 2022-09-12T19:34:04.149 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:g+b9Suv2JJvbhKbCijOwk7Lzhxc= 2022-09-12T19:34:04.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:g+b9Suv2JJvbhKbCijOwk7Lzhxc=', 'Content-Length': '0'} 2022-09-12T19:34:04.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006870cd6a69ba93a8-00631f89ac-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.153 DEBUG:boto:b'jdpiak-63obj31000falseobj32022-09-12T19:32:15.326Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:04.154 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:04.154 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:04.155 DEBUG:boto:path=/jdpiak-63/?prefix=obj3 2022-09-12T19:34:04.155 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj3 2022-09-12T19:34:04.156 DEBUG:boto:Method: GET 2022-09-12T19:34:04.156 DEBUG:boto:Path: /jdpiak-63/?prefix=obj3 2022-09-12T19:34:04.157 DEBUG:boto:Data: 2022-09-12T19:34:04.157 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.158 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:04.159 DEBUG:boto:Port: 8000 2022-09-12T19:34:04.159 DEBUG:boto:Params: {} 2022-09-12T19:34:04.160 DEBUG:boto:Token: None 2022-09-12T19:34:04.160 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/ 2022-09-12T19:34:04.161 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:g+b9Suv2JJvbhKbCijOwk7Lzhxc= 2022-09-12T19:34:04.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:g+b9Suv2JJvbhKbCijOwk7Lzhxc=', 'Content-Length': '0'} 2022-09-12T19:34:04.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b35031d3deb0a866-00631f89ac-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.165 DEBUG:boto:b'jdpiak-63obj31000falseobj32022-09-12T19:32:15.326Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:04.166 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:04.166 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:04.167 DEBUG:boto:Method: HEAD 2022-09-12T19:34:04.167 DEBUG:boto:Path: /jdpiak-63/obj3 2022-09-12T19:34:04.168 DEBUG:boto:Data: 2022-09-12T19:34:04.168 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.169 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:04.169 DEBUG:boto:Port: 8001 2022-09-12T19:34:04.170 DEBUG:boto:Params: {} 2022-09-12T19:34:04.170 DEBUG:boto:Token: None 2022-09-12T19:34:04.171 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/obj3 2022-09-12T19:34:04.172 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZlAmUmaT60EiZe0uE+2/nxT5VBs= 2022-09-12T19:34:04.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZlAmUmaT60EiZe0uE+2/nxT5VBs=', 'Content-Length': '0'} 2022-09-12T19:34:04.174 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000416b8bbe7199e89e-00631f89ac-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.175 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:04.176 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:04.176 DEBUG:boto:Method: HEAD 2022-09-12T19:34:04.177 DEBUG:boto:Path: /jdpiak-63/obj3 2022-09-12T19:34:04.177 DEBUG:boto:Data: 2022-09-12T19:34:04.178 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.178 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:04.179 DEBUG:boto:Port: 8000 2022-09-12T19:34:04.179 DEBUG:boto:Params: {} 2022-09-12T19:34:04.180 DEBUG:boto:Token: None 2022-09-12T19:34:04.180 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/obj3 2022-09-12T19:34:04.181 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZlAmUmaT60EiZe0uE+2/nxT5VBs= 2022-09-12T19:34:04.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZlAmUmaT60EiZe0uE+2/nxT5VBs=', 'Content-Length': '0'} 2022-09-12T19:34:04.184 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a1e014167534b0f-00631f89ac-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-12T19:34:04.185 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:04.185 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:04.186 DEBUG:boto:Method: GET 2022-09-12T19:34:04.187 DEBUG:boto:Path: /jdpiak-63/obj3 2022-09-12T19:34:04.187 DEBUG:boto:Data: 2022-09-12T19:34:04.188 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.188 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:04.189 DEBUG:boto:Port: 8001 2022-09-12T19:34:04.189 DEBUG:boto:Params: {} 2022-09-12T19:34:04.190 DEBUG:boto:Token: None 2022-09-12T19:34:04.191 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/obj3 2022-09-12T19:34:04.191 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:o7l8dCyU/6g+riWmA7rc9n5LC2w= 2022-09-12T19:34:04.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:o7l8dCyU/6g+riWmA7rc9n5LC2w=', 'Content-Length': '0'} 2022-09-12T19:34:04.194 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000510c28c32660eb18-00631f89ac-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.195 DEBUG:boto:path=/jdpiak-63/obj3 2022-09-12T19:34:04.195 DEBUG:boto:auth_path=/jdpiak-63/obj3 2022-09-12T19:34:04.196 DEBUG:boto:Method: GET 2022-09-12T19:34:04.196 DEBUG:boto:Path: /jdpiak-63/obj3 2022-09-12T19:34:04.197 DEBUG:boto:Data: 2022-09-12T19:34:04.197 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.198 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:04.198 DEBUG:boto:Port: 8000 2022-09-12T19:34:04.199 DEBUG:boto:Params: {} 2022-09-12T19:34:04.199 DEBUG:boto:Token: None 2022-09-12T19:34:04.200 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/obj3 2022-09-12T19:34:04.201 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:o7l8dCyU/6g+riWmA7rc9n5LC2w= 2022-09-12T19:34:04.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:o7l8dCyU/6g+riWmA7rc9n5LC2w=', 'Content-Length': '0'} 2022-09-12T19:34:04.203 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b29ae098e1b7858-00631f89ac-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.204 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:04.205 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:04.205 DEBUG:boto:path=/jdpiak-63/?prefix=obj3 2022-09-12T19:34:04.206 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj3 2022-09-12T19:34:04.206 DEBUG:boto:Method: GET 2022-09-12T19:34:04.207 DEBUG:boto:Path: /jdpiak-63/?prefix=obj3 2022-09-12T19:34:04.207 DEBUG:boto:Data: 2022-09-12T19:34:04.208 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.208 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:04.209 DEBUG:boto:Port: 8001 2022-09-12T19:34:04.209 DEBUG:boto:Params: {} 2022-09-12T19:34:04.210 DEBUG:boto:Token: None 2022-09-12T19:34:04.211 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/ 2022-09-12T19:34:04.211 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:g+b9Suv2JJvbhKbCijOwk7Lzhxc= 2022-09-12T19:34:04.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:g+b9Suv2JJvbhKbCijOwk7Lzhxc=', 'Content-Length': '0'} 2022-09-12T19:34:04.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d06bc9016723bc2-00631f89ac-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.258 DEBUG:boto:b'jdpiak-63obj31000falseobj32022-09-12T19:32:15.326Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:04.259 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:04.260 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:04.260 DEBUG:boto:path=/jdpiak-63/?prefix=obj3 2022-09-12T19:34:04.261 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj3 2022-09-12T19:34:04.261 DEBUG:boto:Method: GET 2022-09-12T19:34:04.262 DEBUG:boto:Path: /jdpiak-63/?prefix=obj3 2022-09-12T19:34:04.262 DEBUG:boto:Data: 2022-09-12T19:34:04.263 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.263 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:04.264 DEBUG:boto:Port: 8000 2022-09-12T19:34:04.264 DEBUG:boto:Params: {} 2022-09-12T19:34:04.265 DEBUG:boto:Token: None 2022-09-12T19:34:04.266 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/ 2022-09-12T19:34:04.266 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:g+b9Suv2JJvbhKbCijOwk7Lzhxc= 2022-09-12T19:34:04.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:g+b9Suv2JJvbhKbCijOwk7Lzhxc=', 'Content-Length': '0'} 2022-09-12T19:34:04.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfc60ad006cb47cb-00631f89ac-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.271 DEBUG:boto:b'jdpiak-63obj31000falseobj32022-09-12T19:32:15.326Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:04.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-12T19:34:04.274 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:04.275 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:04.276 DEBUG:boto:path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:04.277 DEBUG:boto:auth_path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:04.278 DEBUG:boto:Method: GET 2022-09-12T19:34:04.279 DEBUG:boto:Path: /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:04.280 DEBUG:boto:Data: 2022-09-12T19:34:04.280 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.281 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:04.282 DEBUG:boto:Port: 8001 2022-09-12T19:34:04.283 DEBUG:boto:Params: {} 2022-09-12T19:34:04.284 DEBUG:boto:Token: None 2022-09-12T19:34:04.285 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:04.286 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s4T/DgE6VOtYbs+oj82cE7sAIE8= 2022-09-12T19:34:04.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s4T/DgE6VOtYbs+oj82cE7sAIE8=', 'Content-Length': '0'} 2022-09-12T19:34:04.289 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe818e2f5d393f6c-00631f89ac-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.290 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:04.290 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:04.291 DEBUG:boto:path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:04.291 DEBUG:boto:auth_path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:04.292 DEBUG:boto:Method: GET 2022-09-12T19:34:04.293 DEBUG:boto:Path: /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:04.294 DEBUG:boto:Data: 2022-09-12T19:34:04.294 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.295 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:04.295 DEBUG:boto:Port: 8000 2022-09-12T19:34:04.296 DEBUG:boto:Params: {} 2022-09-12T19:34:04.297 DEBUG:boto:Token: None 2022-09-12T19:34:04.297 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:04.298 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s4T/DgE6VOtYbs+oj82cE7sAIE8= 2022-09-12T19:34:04.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s4T/DgE6VOtYbs+oj82cE7sAIE8=', 'Content-Length': '0'} 2022-09-12T19:34:04.301 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a1bfcc03e6d9769c-00631f89ac-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.302 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:04.302 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:04.303 DEBUG:boto:path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:04.304 DEBUG:boto:auth_path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:04.304 DEBUG:boto:Method: HEAD 2022-09-12T19:34:04.305 DEBUG:boto:Path: /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:04.305 DEBUG:boto:Data: 2022-09-12T19:34:04.306 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.306 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:04.307 DEBUG:boto:Port: 8001 2022-09-12T19:34:04.307 DEBUG:boto:Params: {} 2022-09-12T19:34:04.308 DEBUG:boto:Token: None 2022-09-12T19:34:04.308 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:04.309 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GQ09nDfVqlcSpJWliPhotsmXmJk= 2022-09-12T19:34:04.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GQ09nDfVqlcSpJWliPhotsmXmJk=', 'Content-Length': '0'} 2022-09-12T19:34:04.312 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f87c85b48c6dd4b1-00631f89ac-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.313 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:04.314 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:04.314 DEBUG:boto:path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:04.315 DEBUG:boto:auth_path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:04.315 DEBUG:boto:Method: HEAD 2022-09-12T19:34:04.316 DEBUG:boto:Path: /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:04.316 DEBUG:boto:Data: 2022-09-12T19:34:04.317 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.317 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:04.318 DEBUG:boto:Port: 8000 2022-09-12T19:34:04.318 DEBUG:boto:Params: {} 2022-09-12T19:34:04.319 DEBUG:boto:Token: None 2022-09-12T19:34:04.319 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:04.320 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GQ09nDfVqlcSpJWliPhotsmXmJk= 2022-09-12T19:34:04.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GQ09nDfVqlcSpJWliPhotsmXmJk=', 'Content-Length': '0'} 2022-09-12T19:34:04.323 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011552ffd9a2eeb34-00631f89ac-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-12T19:34:04.324 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:04.325 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:04.325 DEBUG:boto:path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:04.326 DEBUG:boto:auth_path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:04.326 DEBUG:boto:Method: GET 2022-09-12T19:34:04.327 DEBUG:boto:Path: /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:04.328 DEBUG:boto:Data: 2022-09-12T19:34:04.329 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.329 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:04.330 DEBUG:boto:Port: 8001 2022-09-12T19:34:04.330 DEBUG:boto:Params: {} 2022-09-12T19:34:04.331 DEBUG:boto:Token: None 2022-09-12T19:34:04.332 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:04.333 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s4T/DgE6VOtYbs+oj82cE7sAIE8= 2022-09-12T19:34:04.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s4T/DgE6VOtYbs+oj82cE7sAIE8=', 'Content-Length': '0'} 2022-09-12T19:34:04.335 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e764d5537c52a0d-00631f89ac-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.336 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:04.337 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:04.337 DEBUG:boto:path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:04.338 DEBUG:boto:auth_path=/jdpiak-63/obj4?versionId=null 2022-09-12T19:34:04.339 DEBUG:boto:Method: GET 2022-09-12T19:34:04.339 DEBUG:boto:Path: /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:04.340 DEBUG:boto:Data: 2022-09-12T19:34:04.340 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.341 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:04.341 DEBUG:boto:Port: 8000 2022-09-12T19:34:04.342 DEBUG:boto:Params: {} 2022-09-12T19:34:04.342 DEBUG:boto:Token: None 2022-09-12T19:34:04.343 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/obj4?versionId=null 2022-09-12T19:34:04.344 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s4T/DgE6VOtYbs+oj82cE7sAIE8= 2022-09-12T19:34:04.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s4T/DgE6VOtYbs+oj82cE7sAIE8=', 'Content-Length': '0'} 2022-09-12T19:34:04.346 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e461b0fe01d2dfd1-00631f89ac-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.390 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:04.391 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:04.392 DEBUG:boto:path=/jdpiak-63/?prefix=obj4 2022-09-12T19:34:04.392 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj4 2022-09-12T19:34:04.393 DEBUG:boto:Method: GET 2022-09-12T19:34:04.393 DEBUG:boto:Path: /jdpiak-63/?prefix=obj4 2022-09-12T19:34:04.394 DEBUG:boto:Data: 2022-09-12T19:34:04.394 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.395 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:04.395 DEBUG:boto:Port: 8001 2022-09-12T19:34:04.396 DEBUG:boto:Params: {} 2022-09-12T19:34:04.396 DEBUG:boto:Token: None 2022-09-12T19:34:04.397 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/ 2022-09-12T19:34:04.398 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:g+b9Suv2JJvbhKbCijOwk7Lzhxc= 2022-09-12T19:34:04.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:g+b9Suv2JJvbhKbCijOwk7Lzhxc=', 'Content-Length': '0'} 2022-09-12T19:34:04.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2b2eaa2f5cca4ba-00631f89ac-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.402 DEBUG:boto:b'jdpiak-63obj41000falseobj42022-09-12T19:32:15.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:04.403 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:04.403 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:04.404 DEBUG:boto:path=/jdpiak-63/?prefix=obj4 2022-09-12T19:34:04.404 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj4 2022-09-12T19:34:04.405 DEBUG:boto:Method: GET 2022-09-12T19:34:04.405 DEBUG:boto:Path: /jdpiak-63/?prefix=obj4 2022-09-12T19:34:04.406 DEBUG:boto:Data: 2022-09-12T19:34:04.406 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.407 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:04.408 DEBUG:boto:Port: 8000 2022-09-12T19:34:04.408 DEBUG:boto:Params: {} 2022-09-12T19:34:04.409 DEBUG:boto:Token: None 2022-09-12T19:34:04.409 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/ 2022-09-12T19:34:04.410 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:g+b9Suv2JJvbhKbCijOwk7Lzhxc= 2022-09-12T19:34:04.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:g+b9Suv2JJvbhKbCijOwk7Lzhxc=', 'Content-Length': '0'} 2022-09-12T19:34:04.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d286f1f5e8f063a-00631f89ac-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.414 DEBUG:boto:b'jdpiak-63obj41000falseobj42022-09-12T19:32:15.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:04.415 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:04.415 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:04.416 DEBUG:boto:Method: HEAD 2022-09-12T19:34:04.416 DEBUG:boto:Path: /jdpiak-63/obj4 2022-09-12T19:34:04.417 DEBUG:boto:Data: 2022-09-12T19:34:04.417 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.418 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:04.418 DEBUG:boto:Port: 8001 2022-09-12T19:34:04.419 DEBUG:boto:Params: {} 2022-09-12T19:34:04.419 DEBUG:boto:Token: None 2022-09-12T19:34:04.420 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/obj4 2022-09-12T19:34:04.421 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:v6Jaom65RecOTBmP7GbDFcgHuB0= 2022-09-12T19:34:04.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:v6Jaom65RecOTBmP7GbDFcgHuB0=', 'Content-Length': '0'} 2022-09-12T19:34:04.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5e6a7b9fc8467b8-00631f89ac-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.424 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:04.424 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:04.425 DEBUG:boto:Method: HEAD 2022-09-12T19:34:04.425 DEBUG:boto:Path: /jdpiak-63/obj4 2022-09-12T19:34:04.426 DEBUG:boto:Data: 2022-09-12T19:34:04.426 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.427 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:04.427 DEBUG:boto:Port: 8000 2022-09-12T19:34:04.428 DEBUG:boto:Params: {} 2022-09-12T19:34:04.428 DEBUG:boto:Token: None 2022-09-12T19:34:04.429 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/obj4 2022-09-12T19:34:04.430 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:v6Jaom65RecOTBmP7GbDFcgHuB0= 2022-09-12T19:34:04.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:v6Jaom65RecOTBmP7GbDFcgHuB0=', 'Content-Length': '0'} 2022-09-12T19:34:04.432 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c1c53c17f0c8881-00631f89ac-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-12T19:34:04.433 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:04.434 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:04.435 DEBUG:boto:Method: GET 2022-09-12T19:34:04.435 DEBUG:boto:Path: /jdpiak-63/obj4 2022-09-12T19:34:04.436 DEBUG:boto:Data: 2022-09-12T19:34:04.436 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.437 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:04.437 DEBUG:boto:Port: 8001 2022-09-12T19:34:04.438 DEBUG:boto:Params: {} 2022-09-12T19:34:04.438 DEBUG:boto:Token: None 2022-09-12T19:34:04.439 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/obj4 2022-09-12T19:34:04.440 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F7oE+bbSHjCe1eyjqCBER44ByBQ= 2022-09-12T19:34:04.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F7oE+bbSHjCe1eyjqCBER44ByBQ=', 'Content-Length': '0'} 2022-09-12T19:34:04.442 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ef3232a7d48a3d5-00631f89ac-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.443 DEBUG:boto:path=/jdpiak-63/obj4 2022-09-12T19:34:04.444 DEBUG:boto:auth_path=/jdpiak-63/obj4 2022-09-12T19:34:04.444 DEBUG:boto:Method: GET 2022-09-12T19:34:04.445 DEBUG:boto:Path: /jdpiak-63/obj4 2022-09-12T19:34:04.446 DEBUG:boto:Data: 2022-09-12T19:34:04.446 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.447 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:04.447 DEBUG:boto:Port: 8000 2022-09-12T19:34:04.448 DEBUG:boto:Params: {} 2022-09-12T19:34:04.448 DEBUG:boto:Token: None 2022-09-12T19:34:04.449 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/obj4 2022-09-12T19:34:04.450 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F7oE+bbSHjCe1eyjqCBER44ByBQ= 2022-09-12T19:34:04.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F7oE+bbSHjCe1eyjqCBER44ByBQ=', 'Content-Length': '0'} 2022-09-12T19:34:04.452 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:32:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1c2438acabb714c-00631f89ac-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.454 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:04.455 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:04.456 DEBUG:boto:path=/jdpiak-63/?prefix=obj4 2022-09-12T19:34:04.457 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj4 2022-09-12T19:34:04.457 DEBUG:boto:Method: GET 2022-09-12T19:34:04.459 DEBUG:boto:Path: /jdpiak-63/?prefix=obj4 2022-09-12T19:34:04.459 DEBUG:boto:Data: 2022-09-12T19:34:04.460 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.461 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:34:04.462 DEBUG:boto:Port: 8001 2022-09-12T19:34:04.463 DEBUG:boto:Params: {} 2022-09-12T19:34:04.465 DEBUG:boto:Token: None 2022-09-12T19:34:04.466 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/ 2022-09-12T19:34:04.467 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:g+b9Suv2JJvbhKbCijOwk7Lzhxc= 2022-09-12T19:34:04.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:g+b9Suv2JJvbhKbCijOwk7Lzhxc=', 'Content-Length': '0'} 2022-09-12T19:34:04.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdf50666cb1e6567-00631f89ac-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.473 DEBUG:boto:b'jdpiak-63obj41000falseobj42022-09-12T19:32:15.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:04.474 DEBUG:boto:path=/jdpiak-63/ 2022-09-12T19:34:04.475 DEBUG:boto:auth_path=/jdpiak-63/ 2022-09-12T19:34:04.476 DEBUG:boto:path=/jdpiak-63/?prefix=obj4 2022-09-12T19:34:04.477 DEBUG:boto:auth_path=/jdpiak-63/?prefix=obj4 2022-09-12T19:34:04.478 DEBUG:boto:Method: GET 2022-09-12T19:34:04.479 DEBUG:boto:Path: /jdpiak-63/?prefix=obj4 2022-09-12T19:34:04.480 DEBUG:boto:Data: 2022-09-12T19:34:04.481 DEBUG:boto:Headers: {} 2022-09-12T19:34:04.482 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:34:04.483 DEBUG:boto:Port: 8000 2022-09-12T19:34:04.484 DEBUG:boto:Params: {} 2022-09-12T19:34:04.485 DEBUG:boto:Token: None 2022-09-12T19:34:04.486 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:34:04 GMT /jdpiak-63/ 2022-09-12T19:34:04.488 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:g+b9Suv2JJvbhKbCijOwk7Lzhxc= 2022-09-12T19:34:04.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:34:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:g+b9Suv2JJvbhKbCijOwk7Lzhxc=', 'Content-Length': '0'} 2022-09-12T19:34:04.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094f886b1bcc669cd-00631f89ac-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:34:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:34:04.494 DEBUG:boto:b'jdpiak-63obj41000falseobj42022-09-12T19:32:15.398Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:34:04.495 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-63 zones={test-zone3, test-zone2} 2022-09-12T19:34:04.496 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-64 2022-09-12T19:34:04.497 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:34:04.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:34:04.499 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:34:04.564 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:34:04.565 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:34:04.606 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:34:04.603+0000 7eff0c262600 1 bucket sync caught up with source: 2022-09-12T19:34:04.607 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:34:04.608 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:34:04.608 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:34:04.603+0000 7eff0c262600 0 bucket checkpoint complete 2022-09-12T19:34:04.621 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-64 2022-09-12T19:34:04.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:34:04.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:34:04.625 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:34:04.692 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:34:04.692 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:34:04.745 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:34:04.742+0000 7f6f5c578600 1 bucket sync caught up with source: 2022-09-12T19:34:04.745 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:34:04.746 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:34:04.746 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:34:04.742+0000 7f6f5c578600 0 bucket checkpoint complete 2022-09-12T19:34:04.764 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-64 2022-09-12T19:34:04.765 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:34:04.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:34:04.766 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:34:04.823 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:34:04.823 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:34:04.879 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:34:04.876+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:34:04.880 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:34:04.880 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:34:09.881 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:34:09.878+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:34:09.881 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:34:09.882 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:34:14.882 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:34:14.879+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:34:14.883 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:34:14.883 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:34:19.884 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:34:19.881+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:34:19.885 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:34:19.885 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:34:24.885 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:34:24.882+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:34:24.886 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:34:24.886 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:34:29.887 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:34:29.884+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:34:29.887 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:34:29.887 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:34:34.888 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:34:34.885+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:34:34.889 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:34:34.889 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:34:39.889 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:34:39.886+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:34:39.890 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:34:39.890 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:34:44.891 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:34:44.889+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:34:44.891 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:34:44.891 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:34:49.892 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:34:49.889+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:34:49.893 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:34:49.894 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:34:54.894 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:34:54.891+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:34:54.895 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:34:54.895 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:34:59.895 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:34:59.892+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:34:59.896 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:34:59.896 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:35:04.897 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:35:04.894+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:35:04.897 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:35:04.898 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:35:09.898 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:35:09.895+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:35:09.899 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:35:09.899 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:35:14.900 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:35:14.897+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:35:14.901 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:35:14.901 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:35:19.901 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:35:19.898+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:35:19.902 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:35:19.902 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:35:24.903 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:35:24.900+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:35:24.903 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:35:24.904 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:35:29.904 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:35:29.902+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:35:29.904 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:35:29.905 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:35:34.906 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:35:34.903+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:35:34.907 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:35:34.907 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:35:39.908 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:35:39.904+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:35:39.909 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:35:39.909 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:35:44.909 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:35:44.906+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:35:44.909 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:35:44.909 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:35:49.910 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:35:49.907+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:35:49.911 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:35:49.912 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:35:54.912 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:35:54.909+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:35:54.912 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:35:54.913 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:35:59.913 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:35:59.910+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:35:59.913 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:35:59.914 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:36:04.914 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:36:04.912+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:36:04.915 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:36:04.915 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:36:09.916 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:36:09.913+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:36:09.916 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:36:09.917 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:36:14.918 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:36:14.916+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:36:14.918 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:36:14.919 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:36:19.919 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:36:19.916+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:36:19.920 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:36:19.920 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:36:24.921 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:36:24.918+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:36:24.921 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:36:24.922 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:36:29.922 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:36:29.919+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:36:29.923 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:36:29.923 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:36:34.924 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:36:34.921+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:36:34.925 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:36:34.925 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:36:39.931 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:36:39.922+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:36:39.932 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:36:39.935 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:36:44.927 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:36:44.924+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:36:44.927 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:36:44.928 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:36:49.928 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:36:49.925+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:36:49.928 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:36:49.929 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:36:54.929 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:36:54.927+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:36:54.930 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:36:54.930 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:36:59.931 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:36:59.929+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:36:59.932 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:36:59.932 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:37:04.932 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:37:04.931+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:37:04.933 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:37:04.933 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:37:09.934 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:37:09.931+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:37:09.935 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:37:09.935 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:37:14.935 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:37:14.932+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:37:14.936 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:37:14.937 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:37:19.936 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:37:19.933+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:37:19.937 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:37:19.937 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:37:24.937 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:37:24.934+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:37:24.937 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:37:24.938 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:37:29.939 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:37:29.936+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:37:29.939 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:37:29.940 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:37:34.940 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:37:34.937+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:37:34.941 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:37:34.942 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:37:39.942 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:37:39.939+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:37:39.943 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:37:39.943 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:37:44.944 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:37:44.942+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:37:44.945 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:37:44.945 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:37:49.945 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:37:49.943+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:37:49.945 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:37:49.945 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:37:54.946 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:37:54.943+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:37:54.947 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:37:54.948 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:37:59.948 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:37:59.945+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:37:59.948 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:37:59.949 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:38:04.949 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:38:04.946+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:38:04.950 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:38:04.950 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:38:09.950 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:38:09.948+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:38:09.951 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:38:09.951 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:38:14.951 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:38:14.949+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:38:14.952 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:38:14.952 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:38:19.954 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:38:19.950+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:38:19.954 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:38:19.955 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:38:24.959 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:38:24.952+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:38:24.959 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:38:24.960 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:38:29.956 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:38:29.954+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:38:29.957 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:38:29.957 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:38:34.958 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:38:34.956+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:38:34.958 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:38:34.958 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:38:39.959 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:38:39.956+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:38:39.960 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:38:39.960 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:38:44.961 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:38:44.958+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:38:44.962 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:38:44.963 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:38:49.962 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:38:49.959+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:38:49.963 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:38:49.963 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:38:54.966 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:38:54.961+0000 7fecd3a21600 1 waiting for incremental sync to catch up: 2022-09-12T19:38:54.967 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.51.5, , 00000000001.45.5, , , , , 00000000001.78.5, , , ] 2022-09-12T19:38:54.968 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.51.5, , 00000000001.45.5, , 00000000001.53.5, , , 00000000001.78.5, , , ] 2022-09-12T19:38:59.965 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:38:59.962+0000 7fecd3a21600 0 bucket checkpoint timed out waiting for incremental sync to catch up 2022-09-12T19:38:59.965 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:38:59.962+0000 7fecd3a21600 0 bucket sync checkpoint failed: (110) Connection timed out 2022-09-12T19:38:59.966 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:38:59.962+0000 7fecd3a21600 -1 bucket sync checkpoint failed: (110) Connection timed out 2022-09-12T19:38:59.985 DEBUG:teuthology.orchestra.run:got remote process result: 110 2022-09-12T19:39:00.003 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... FAIL 2022-09-12T19:39:00.007 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-65 2022-09-12T19:39:00.008 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:00.010 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:00.012 DEBUG:boto:Method: PUT 2022-09-12T19:39:00.013 DEBUG:boto:Path: /jdpiak-65/ 2022-09-12T19:39:00.014 DEBUG:boto:Data: 2022-09-12T19:39:00.015 DEBUG:boto:Headers: {} 2022-09-12T19:39:00.015 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:00.016 DEBUG:boto:Port: 8000 2022-09-12T19:39:00.017 DEBUG:boto:Params: {} 2022-09-12T19:39:00.019 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:39:00.021 DEBUG:boto:Token: None 2022-09-12T19:39:00.022 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:39:00 GMT /jdpiak-65/ 2022-09-12T19:39:00.022 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2b6v3LVm6hk8KJzO+wTetA02BBo= 2022-09-12T19:39:00.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2b6v3LVm6hk8KJzO+wTetA02BBo=', 'Content-Length': '0'} 2022-09-12T19:39:00.053 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005c601ad6e1852c7f-00631f8ad4-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:39:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:00.054 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=jdpiak-66 2022-09-12T19:39:00.055 DEBUG:boto:path=/jdpiak-66/ 2022-09-12T19:39:00.055 DEBUG:boto:auth_path=/jdpiak-66/ 2022-09-12T19:39:00.056 DEBUG:boto:Method: PUT 2022-09-12T19:39:00.057 DEBUG:boto:Path: /jdpiak-66/ 2022-09-12T19:39:00.057 DEBUG:boto:Data: 2022-09-12T19:39:00.058 DEBUG:boto:Headers: {} 2022-09-12T19:39:00.059 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:00.059 DEBUG:boto:Port: 8000 2022-09-12T19:39:00.060 DEBUG:boto:Params: {} 2022-09-12T19:39:00.060 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:39:00.061 DEBUG:boto:Token: None 2022-09-12T19:39:00.062 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:39:00 GMT /jdpiak-66/ 2022-09-12T19:39:00.063 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/MhO7W5cr5ZCIVcac6O7KjeJeZ8= 2022-09-12T19:39:00.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/MhO7W5cr5ZCIVcac6O7KjeJeZ8=', 'Content-Length': '0'} 2022-09-12T19:39:00.111 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004ee598e8cb9ec5e0-00631f8ad4-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:39:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:00.112 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=jdpiak-67 2022-09-12T19:39:00.113 DEBUG:boto:path=/jdpiak-67/ 2022-09-12T19:39:00.114 DEBUG:boto:auth_path=/jdpiak-67/ 2022-09-12T19:39:00.114 DEBUG:boto:Method: PUT 2022-09-12T19:39:00.115 DEBUG:boto:Path: /jdpiak-67/ 2022-09-12T19:39:00.115 DEBUG:boto:Data: 2022-09-12T19:39:00.116 DEBUG:boto:Headers: {} 2022-09-12T19:39:00.116 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:39:00.117 DEBUG:boto:Port: 8001 2022-09-12T19:39:00.117 DEBUG:boto:Params: {} 2022-09-12T19:39:00.119 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T19:39:00.120 DEBUG:boto:Token: None 2022-09-12T19:39:00.121 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:39:00 GMT /jdpiak-67/ 2022-09-12T19:39:00.121 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wwjcJhzMh92znvwJfTqEOzlcBio= 2022-09-12T19:39:00.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wwjcJhzMh92znvwJfTqEOzlcBio=', 'Content-Length': '0'} 2022-09-12T19:39:01.171 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d68a56cb4e6aa5e0-00631f8ad4-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:39:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:01.172 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:01.173 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:01.173 DEBUG:boto:path=/jdpiak-65/MULTIPART?uploads 2022-09-12T19:39:01.174 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART?uploads 2022-09-12T19:39:01.174 DEBUG:boto:Method: POST 2022-09-12T19:39:01.175 DEBUG:boto:Path: /jdpiak-65/MULTIPART?uploads 2022-09-12T19:39:01.175 DEBUG:boto:Data: 2022-09-12T19:39:01.176 DEBUG:boto:Headers: {} 2022-09-12T19:39:01.177 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:01.178 DEBUG:boto:Port: 8000 2022-09-12T19:39:01.178 DEBUG:boto:Params: {} 2022-09-12T19:39:01.179 DEBUG:boto:Token: None 2022-09-12T19:39:01.180 DEBUG:boto:StringToSign: POST Mon, 12 Sep 2022 19:39:01 GMT /jdpiak-65/MULTIPART?uploads 2022-09-12T19:39:01.181 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J4mGZqMfgEW4BAO43f4vvsTAgrQ= 2022-09-12T19:39:01.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J4mGZqMfgEW4BAO43f4vvsTAgrQ=', 'Content-Length': '0'} 2022-09-12T19:39:02.516 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d8f13991a633c938-00631f8ad5-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '249'), ('Date', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:02.517 DEBUG:boto:b'jdpiak-65MULTIPART2~1IBFWJNLo9Mmw8DXB57rtPR_SoGTILQ' 2022-09-12T19:39:02.557 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:02.560 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:02.561 DEBUG:boto:path=/jdpiak-65/MULTIPART?uploadId=2~1IBFWJNLo9Mmw8DXB57rtPR_SoGTILQ&partNumber=1 2022-09-12T19:39:02.561 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART?uploadId=2~1IBFWJNLo9Mmw8DXB57rtPR_SoGTILQ&partNumber=1 2022-09-12T19:39:02.562 DEBUG:boto:Method: PUT 2022-09-12T19:39:02.562 DEBUG:boto:Path: /jdpiak-65/MULTIPART?uploadId=2~1IBFWJNLo9Mmw8DXB57rtPR_SoGTILQ&partNumber=1 2022-09-12T19:39:02.563 DEBUG:boto:Data: 2022-09-12T19:39:02.564 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2022-09-12T19:39:02.564 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:02.565 DEBUG:boto:Port: 8000 2022-09-12T19:39:02.565 DEBUG:boto:Params: {} 2022-09-12T19:39:02.566 DEBUG:boto:Token: None 2022-09-12T19:39:02.567 DEBUG:boto:StringToSign: PUT ebKBBg0ze5srhMzzkK3PdA== application/octet-stream Mon, 12 Sep 2022 19:39:02 GMT /jdpiak-65/MULTIPART?partNumber=1&uploadId=2~1IBFWJNLo9Mmw8DXB57rtPR_SoGTILQ 2022-09-12T19:39:02.567 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lAYIl3P9EsQCb7g7wP3gf0U5OQA= 2022-09-12T19:39:02.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:39:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lAYIl3P9EsQCb7g7wP3gf0U5OQA='} 2022-09-12T19:39:02.618 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000034f00e2224a0356c-00631f8ad6-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:02.660 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:02.660 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:02.661 DEBUG:boto:path=/jdpiak-65/MULTIPART?uploadId=2~1IBFWJNLo9Mmw8DXB57rtPR_SoGTILQ&partNumber=2 2022-09-12T19:39:02.661 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART?uploadId=2~1IBFWJNLo9Mmw8DXB57rtPR_SoGTILQ&partNumber=2 2022-09-12T19:39:02.662 DEBUG:boto:Method: PUT 2022-09-12T19:39:02.663 DEBUG:boto:Path: /jdpiak-65/MULTIPART?uploadId=2~1IBFWJNLo9Mmw8DXB57rtPR_SoGTILQ&partNumber=2 2022-09-12T19:39:02.663 DEBUG:boto:Data: 2022-09-12T19:39:02.664 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2022-09-12T19:39:02.664 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:02.665 DEBUG:boto:Port: 8000 2022-09-12T19:39:02.665 DEBUG:boto:Params: {} 2022-09-12T19:39:02.666 DEBUG:boto:Token: None 2022-09-12T19:39:02.666 DEBUG:boto:StringToSign: PUT dIQ6OrGTo4m87YmUAtmdXw== application/octet-stream Mon, 12 Sep 2022 19:39:02 GMT /jdpiak-65/MULTIPART?partNumber=2&uploadId=2~1IBFWJNLo9Mmw8DXB57rtPR_SoGTILQ 2022-09-12T19:39:02.667 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JSBuL60a1haAYYNMpXqWLDJWrVw= 2022-09-12T19:39:02.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:39:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JSBuL60a1haAYYNMpXqWLDJWrVw='} 2022-09-12T19:39:02.715 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000073a36af68cfacd9c-00631f8ad6-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:02.740 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:02.741 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:02.742 DEBUG:boto:path=/jdpiak-65/MULTIPART?uploadId=2~1IBFWJNLo9Mmw8DXB57rtPR_SoGTILQ&partNumber=3 2022-09-12T19:39:02.743 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART?uploadId=2~1IBFWJNLo9Mmw8DXB57rtPR_SoGTILQ&partNumber=3 2022-09-12T19:39:02.743 DEBUG:boto:Method: PUT 2022-09-12T19:39:02.744 DEBUG:boto:Path: /jdpiak-65/MULTIPART?uploadId=2~1IBFWJNLo9Mmw8DXB57rtPR_SoGTILQ&partNumber=3 2022-09-12T19:39:02.745 DEBUG:boto:Data: 2022-09-12T19:39:02.746 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2022-09-12T19:39:02.747 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:02.747 DEBUG:boto:Port: 8000 2022-09-12T19:39:02.748 DEBUG:boto:Params: {} 2022-09-12T19:39:02.749 DEBUG:boto:Token: None 2022-09-12T19:39:02.750 DEBUG:boto:StringToSign: PUT e4RW4edMN49Fhh9TYZ51tg== application/octet-stream Mon, 12 Sep 2022 19:39:02 GMT /jdpiak-65/MULTIPART?partNumber=3&uploadId=2~1IBFWJNLo9Mmw8DXB57rtPR_SoGTILQ 2022-09-12T19:39:02.751 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XqFqhS0YHBkljk7wbTTe/Pr4z7Q= 2022-09-12T19:39:02.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:39:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XqFqhS0YHBkljk7wbTTe/Pr4z7Q='} 2022-09-12T19:39:02.790 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007b60e56e3f6ba43d-00631f8ad6-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:02.811 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:02.812 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:02.812 DEBUG:boto:path=/jdpiak-65/MULTIPART?uploadId=2~1IBFWJNLo9Mmw8DXB57rtPR_SoGTILQ&partNumber=4 2022-09-12T19:39:02.813 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART?uploadId=2~1IBFWJNLo9Mmw8DXB57rtPR_SoGTILQ&partNumber=4 2022-09-12T19:39:02.813 DEBUG:boto:Method: PUT 2022-09-12T19:39:02.814 DEBUG:boto:Path: /jdpiak-65/MULTIPART?uploadId=2~1IBFWJNLo9Mmw8DXB57rtPR_SoGTILQ&partNumber=4 2022-09-12T19:39:02.814 DEBUG:boto:Data: 2022-09-12T19:39:02.815 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2022-09-12T19:39:02.815 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:02.816 DEBUG:boto:Port: 8000 2022-09-12T19:39:02.816 DEBUG:boto:Params: {} 2022-09-12T19:39:02.817 DEBUG:boto:Token: None 2022-09-12T19:39:02.818 DEBUG:boto:StringToSign: PUT l+jDi4yPcIlVvEiQwDQZTQ== application/octet-stream Mon, 12 Sep 2022 19:39:02 GMT /jdpiak-65/MULTIPART?partNumber=4&uploadId=2~1IBFWJNLo9Mmw8DXB57rtPR_SoGTILQ 2022-09-12T19:39:02.818 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:I1Is9TassqUAeNZc8ke6I1ATFvM= 2022-09-12T19:39:02.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:39:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:I1Is9TassqUAeNZc8ke6I1ATFvM='} 2022-09-12T19:39:02.853 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000095cb77786af59786-00631f8ad6-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:02.855 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:02.856 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:02.857 DEBUG:boto:path=/jdpiak-65/MULTIPART?uploadId=2~1IBFWJNLo9Mmw8DXB57rtPR_SoGTILQ 2022-09-12T19:39:02.858 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART?uploadId=2~1IBFWJNLo9Mmw8DXB57rtPR_SoGTILQ 2022-09-12T19:39:02.858 DEBUG:boto:Method: GET 2022-09-12T19:39:02.859 DEBUG:boto:Path: /jdpiak-65/MULTIPART?uploadId=2~1IBFWJNLo9Mmw8DXB57rtPR_SoGTILQ 2022-09-12T19:39:02.859 DEBUG:boto:Data: 2022-09-12T19:39:02.860 DEBUG:boto:Headers: {} 2022-09-12T19:39:02.860 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:02.861 DEBUG:boto:Port: 8000 2022-09-12T19:39:02.862 DEBUG:boto:Params: {} 2022-09-12T19:39:02.862 DEBUG:boto:Token: None 2022-09-12T19:39:02.863 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:02 GMT /jdpiak-65/MULTIPART?uploadId=2~1IBFWJNLo9Mmw8DXB57rtPR_SoGTILQ 2022-09-12T19:39:02.864 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:v2uGbH4knanTSbbsvuqPw5IiOfw= 2022-09-12T19:39:02.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:v2uGbH4knanTSbbsvuqPw5IiOfw=', 'Content-Length': '0'} 2022-09-12T19:39:02.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000958c5dbc0e8cb148-00631f8ad6-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:02.915 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:02.916 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:02.918 DEBUG:boto:path=/jdpiak-65/MULTIPART?uploadId=2~1IBFWJNLo9Mmw8DXB57rtPR_SoGTILQ 2022-09-12T19:39:02.919 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART?uploadId=2~1IBFWJNLo9Mmw8DXB57rtPR_SoGTILQ 2022-09-12T19:39:02.920 DEBUG:boto:Method: POST 2022-09-12T19:39:02.921 DEBUG:boto:Path: /jdpiak-65/MULTIPART?uploadId=2~1IBFWJNLo9Mmw8DXB57rtPR_SoGTILQ 2022-09-12T19:39:02.922 DEBUG:boto:Data: 1 "79b281060d337b9b2b84ccf390adcf74" 2 "74843a3ab193a389bced899402d99d5f" 3 "7b8456e1e74c378f45861f53619e75b6" 4 "97e8c38b8c8f708955bc4890c034194d" 2022-09-12T19:39:02.923 DEBUG:boto:Headers: {'Content-Type': 'text/xml'} 2022-09-12T19:39:02.923 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:02.924 DEBUG:boto:Port: 8000 2022-09-12T19:39:02.925 DEBUG:boto:Params: {} 2022-09-12T19:39:02.926 DEBUG:boto:Token: None 2022-09-12T19:39:02.927 DEBUG:boto:StringToSign: POST text/xml Mon, 12 Sep 2022 19:39:02 GMT /jdpiak-65/MULTIPART?uploadId=2~1IBFWJNLo9Mmw8DXB57rtPR_SoGTILQ 2022-09-12T19:39:02.928 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0p314MWkL6ZWDt3a6XFQYE4qs/s= 2022-09-12T19:39:02.930 DEBUG:boto:Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0p314MWkL6ZWDt3a6XFQYE4qs/s=', 'Content-Length': '460'} 2022-09-12T19:39:02.941 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000096a5fd087c78e406-00631f8ad6-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '291'), ('Date', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:02.943 DEBUG:boto:http://smithi035.front.sepia.ceph.com:8000/jdpiak-65/MULTIPARTjdpiak-65MULTIPART 2022-09-12T19:39:02.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:02.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:02.948 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:39:03.002 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:39:03.002 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:39:03.042 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=38ca1f57-1358-496c-a99b-df49ee365a4f 2022-09-12T19:39:03.055 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:39:03.056 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:03.056 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:39:03.056 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:39:03.057 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:03.057 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:03.057 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:39:03.058 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:39:03.058 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:03.059 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:03.059 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663011541.155823_601.1", 2022-09-12T19:39:03.060 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:39:01.155823Z" 2022-09-12T19:39:03.060 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:03.061 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:03.062 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:39:03.063 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:39:03.063 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:03.065 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:39:03.068 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1663011541.155823_601.1', 3: ''} 2022-09-12T19:39:03.069 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T19:39:03.070 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:03.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:03.073 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:39:03.134 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:39:03.135 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:39:03.198 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:39:03.177+0000 7f442ffc0600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:39:03.198 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:39:03.178+0000 7f442ffc0600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:39:03.200 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:39:03.201 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:39:03.201 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:39:03.201 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:39:03.202 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:39:03.202 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:39:03.202 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:39:03.203 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:03.203 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:39:03.203 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:39:03.204 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:39:03.204 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:39:03.204 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:39:03.205 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:39:03.205 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:39:03.205 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:39:03.206 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:39:03.207 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:39:03.207 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:39:03.207 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:03.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:03.208 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:39:03.208 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:39:03.208 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:39:03.208 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:39:03.209 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:39:03.209 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:39:03.209 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:39:03.210 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:39:03.210 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:39:03.210 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:39:03.210 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:03.211 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:03.211 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:39:03.211 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:39:03.211 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:39:03.212 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:39:03.212 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663011541.155823_601.1", 2022-09-12T19:39:03.212 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:39:03.212 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:39:03.213 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:39:03.213 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:39:01.155823Z", 2022-09-12T19:39:03.213 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:39:03.213 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:03.214 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:03.214 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:39:03.214 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:39:03.215 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:39:03.215 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:39:03.215 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:39:03.216 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:39:03.216 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:39:03.216 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:39:03.216 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:39:03.217 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:39:03.217 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:03.217 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:03.217 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:39:03.218 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:03.219 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:39:03.219 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:39:03.220 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:39:03.220 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:03.220 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:39:03Z" 2022-09-12T19:39:03.221 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:39:03.222 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011541.155823_601.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:39:01.155823Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:39:03Z" } 2022-09-12T19:39:03.223 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011541.155823_601.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:39:01.155823Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:39:03.224 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011541.155823_601.1', 3: ''} 2022-09-12T19:39:03.224 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011541.155823_601.1', 3: ''} 2022-09-12T19:39:03.225 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T19:39:03.225 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T19:39:03.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:03.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:03.227 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:39:03.280 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:39:03.280 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:39:03.314 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:39:03.311+0000 7f4dfefd3600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:39:03.315 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:39:03.312+0000 7f4dfefd3600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:39:03.327 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:39:03.328 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:39:03.328 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:39:03.328 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:39:03.329 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:39:03.329 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:39:03.329 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:39:03.329 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:03.330 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:39:03.330 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:03.330 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:39:03.330 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:39:03.331 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:39:03.331 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:39:03.331 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:39:03.331 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:39:03.332 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:39:03.332 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:39:03.332 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:39:03.333 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:03.333 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:03.334 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:03.334 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:39:03.335 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:39:03.335 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:39:03.335 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:39:03.336 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:39:03.337 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:39:03.337 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:39:03.338 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:39:03.338 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:39:03.339 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:03.339 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:03.340 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:03.340 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:39:03.341 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:39:03.341 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:39:03.341 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663011540.038907_590.1", 2022-09-12T19:39:03.342 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:39:03.342 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:39:03.343 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:39:03.343 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:39:00.038907Z", 2022-09-12T19:39:03.344 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:39:03.344 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:03.344 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:03.345 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:03.345 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:39:03.346 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:39:03.346 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:39:03.347 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:39:03.347 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:39:03.348 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:39:03.348 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:39:03.349 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:39:03.349 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:39:03.350 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:03.351 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:03.351 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:39:03.352 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:03.352 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:39:03.353 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:39:03.353 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:39:03.354 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:03.366 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:39:03Z" 2022-09-12T19:39:03.367 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:39:03.370 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011540.038907_590.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:39:00.038907Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:39:03Z" } 2022-09-12T19:39:03.371 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011540.038907_590.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:39:00.038907Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:39:03.372 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011541.155823_601.1', 3: ''} 2022-09-12T19:39:03.373 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011540.038907_590.1', 3: ''} 2022-09-12T19:39:03.374 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1663011541.155823_601.1 target=1_1663011540.038907_590.1 2022-09-12T19:39:08.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:08.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:08.381 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:39:08.453 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:39:08.454 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:39:08.483 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:39:08.478+0000 7fb9305c0600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:39:08.483 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:39:08.480+0000 7fb9305c0600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:39:08.504 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:39:08.505 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:39:08.506 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:39:08.506 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:39:08.507 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:39:08.507 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:39:08.512 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:39:08.513 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:08.513 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:39:08.514 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:08.515 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:39:08.516 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:39:08.517 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:39:08.517 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:39:08.518 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:39:08.519 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:39:08.519 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:39:08.519 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:39:08.520 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:39:08.520 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:08.520 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:08.520 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:08.521 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:39:08.521 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:39:08.522 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:39:08.522 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:39:08.523 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:39:08.523 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:39:08.523 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:39:08.524 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:39:08.524 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:39:08.524 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:08.524 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:08.525 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:08.526 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:39:08.526 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:39:08.527 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:39:08.528 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663011541.143382_598.1", 2022-09-12T19:39:08.528 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:39:08.529 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:39:08.529 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:39:08.529 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:39:01.143382Z", 2022-09-12T19:39:08.530 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:39:08.531 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:08.531 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:08.532 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:08.533 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:39:08.533 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:39:08.534 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:39:08.535 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:39:08.535 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:39:08.536 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:39:08.539 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:39:08.539 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:39:08.539 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:39:08.539 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:08.540 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:08.540 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:39:08.540 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:08.541 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:39:08.541 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:39:08.542 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:39:08.542 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:08.542 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:39:08Z" 2022-09-12T19:39:08.543 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:39:08.544 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011541.143382_598.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:39:01.143382Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:39:08Z" } 2022-09-12T19:39:08.553 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011541.143382_598.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:39:01.143382Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:39:08.554 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011541.155823_601.1', 3: ''} 2022-09-12T19:39:08.555 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011541.143382_598.1', 3: ''} 2022-09-12T19:39:08.557 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1663011541.155823_601.1 target=1_1663011541.143382_598.1 2022-09-12T19:39:13.558 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:13.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:13.560 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:39:13.622 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:39:13.622 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:39:13.656 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:39:13.654+0000 7f51a2cd4600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:39:13.658 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:39:13.655+0000 7f51a2cd4600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:39:13.674 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:39:13.674 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:39:13.675 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:39:13.675 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:39:13.675 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:39:13.676 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:39:13.676 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:39:13.676 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:13.676 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:39:13.677 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:13.677 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:39:13.677 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:39:13.678 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:39:13.678 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:39:13.679 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:39:13.679 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:39:13.679 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:39:13.680 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:39:13.680 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:39:13.680 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:13.680 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:13.681 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:13.681 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:39:13.681 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:39:13.681 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:39:13.682 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:39:13.682 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:39:13.682 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:39:13.682 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:39:13.683 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:39:13.683 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:39:13.683 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:13.683 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:13.684 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:13.684 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:39:13.684 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:39:13.685 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:39:13.685 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663011541.155823_601.1", 2022-09-12T19:39:13.685 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:39:13.685 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:39:13.686 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:39:13.686 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:39:01.155823Z", 2022-09-12T19:39:13.687 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:39:13.687 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:13.690 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:13.690 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:13.691 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:39:13.691 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:39:13.691 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:39:13.691 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:39:13.692 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:39:13.692 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:39:13.692 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:39:13.693 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:39:13.693 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:39:13.694 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:13.694 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:13.694 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:39:13.695 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:13.695 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:39:13.696 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:39:13.696 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:39:13.697 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:13.697 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:39:13Z" 2022-09-12T19:39:13.698 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:39:13.700 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011541.155823_601.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:39:01.155823Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:39:13Z" } 2022-09-12T19:39:13.701 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011541.155823_601.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:39:01.155823Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:39:13.702 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011541.155823_601.1', 3: ''} 2022-09-12T19:39:13.703 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011541.155823_601.1', 3: ''} 2022-09-12T19:39:13.704 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T19:39:13.705 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T19:39:13.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:13.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:13.708 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:39:13.767 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:39:13.767 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:39:13.805 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:39:13.804+0000 7f8116cd2600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:39:13.820 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:39:13.805+0000 7f8116cd2600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:39:13.824 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:39:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:39:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:39:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:39:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:39:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:39:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:39:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:39:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:39:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:39:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:39:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:39:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:39:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:39:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:39:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:39:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:39:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:39:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:39:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:39:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:39:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:39:13.831 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:39:13.831 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:39:13.831 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:39:13.831 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:39:13.832 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:39:13.832 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:39:13.833 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:13.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:13.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:39:13.834 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:39:13.834 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:39:13.835 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:39:13.835 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663011541.155823_601.1", 2022-09-12T19:39:13.835 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:39:13.835 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:39:13.836 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:39:13.836 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:39:01.155823Z", 2022-09-12T19:39:13.836 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:39:13.836 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:13.837 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:13.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:39:13.837 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:39:13.838 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:39:13.838 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:39:13.838 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:39:13.838 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:39:13.839 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:39:13.839 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:39:13.839 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:39:13.839 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:39:13.840 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:13.840 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:13.840 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:39:13.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:13.841 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:39:13.841 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:39:13.842 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:39:13.842 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:13.842 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:39:13Z" 2022-09-12T19:39:13.843 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:39:13.844 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011541.155823_601.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:39:01.155823Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:39:13Z" } 2022-09-12T19:39:13.845 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011541.155823_601.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:39:01.155823Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:39:13.845 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011541.155823_601.1', 3: ''} 2022-09-12T19:39:13.846 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011541.155823_601.1', 3: ''} 2022-09-12T19:39:13.846 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T19:39:13.847 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-65 2022-09-12T19:39:13.848 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:39:13.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:39:13.849 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:39:13.906 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:39:13.906 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:39:13.954 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:39:13.953+0000 7fe02ed08600 1 waiting to reach incremental sync.. 2022-09-12T19:39:18.958 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:39:18.955+0000 7fe02ed08600 1 bucket sync caught up with source: 2022-09-12T19:39:18.959 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000011.55.5, , , , , ] 2022-09-12T19:39:18.959 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000011.55.5, , , , , ] 2022-09-12T19:39:18.967 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:39:18.955+0000 7fe02ed08600 0 bucket checkpoint complete 2022-09-12T19:39:18.977 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-65 2022-09-12T19:39:18.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:39:18.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:39:18.979 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:39:19.052 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:39:19.053 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:39:19.099 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:39:19.090+0000 7f897649f600 1 bucket sync caught up with source: 2022-09-12T19:39:19.099 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000011.55.5, , , , , ] 2022-09-12T19:39:19.100 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000011.55.5, , , , , ] 2022-09-12T19:39:19.101 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:39:19.090+0000 7f897649f600 0 bucket checkpoint complete 2022-09-12T19:39:19.107 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-65 2022-09-12T19:39:19.108 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:39:19.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:39:19.109 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:39:19.182 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:39:19.183 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:39:19.218 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:39:19.215+0000 7fb7d7112600 1 bucket sync caught up with source: 2022-09-12T19:39:19.219 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000011.55.5, , , , , ] 2022-09-12T19:39:19.219 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000011.55.5, , , , , ] 2022-09-12T19:39:19.220 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:39:19.215+0000 7fb7d7112600 0 bucket checkpoint complete 2022-09-12T19:39:19.231 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-65 2022-09-12T19:39:19.233 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:39:19.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:39:19.235 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-65 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:39:19.295 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:39:19.296 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:39:19.335 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:39:19.333+0000 7f5d122d0600 1 waiting to reach incremental sync.. 2022-09-12T19:39:24.336 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:39:24.334+0000 7f5d122d0600 1 waiting to reach incremental sync.. 2022-09-12T19:39:29.338 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:39:29.335+0000 7f5d122d0600 1 waiting to reach incremental sync.. 2022-09-12T19:39:34.338 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:39:34.336+0000 7f5d122d0600 1 bucket sync caught up with source: 2022-09-12T19:39:34.338 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000001.556.5, , , , , ] 2022-09-12T19:39:34.339 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000001.556.5, , , , , ] 2022-09-12T19:39:34.339 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:39:34.336+0000 7f5d122d0600 0 bucket checkpoint complete 2022-09-12T19:39:34.356 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-65 2022-09-12T19:39:34.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:39:34.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:39:34.359 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-65 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:39:34.425 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:39:34.426 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:39:34.472 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:39:34.471+0000 7f63d8528600 1 bucket sync caught up with source: 2022-09-12T19:39:34.473 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000001.556.5, , , , , ] 2022-09-12T19:39:34.474 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000001.556.5, , , , , ] 2022-09-12T19:39:34.474 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:39:34.471+0000 7f63d8528600 0 bucket checkpoint complete 2022-09-12T19:39:34.490 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-65 2022-09-12T19:39:34.491 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-65 2022-09-12T19:39:34.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:39:34.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:39:34.493 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-65 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:39:34.552 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:39:34.552 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:39:34.600 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:39:34.598+0000 7f6b483ef600 1 bucket sync caught up with source: 2022-09-12T19:39:34.600 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000001.466.5, , , , , ] 2022-09-12T19:39:34.601 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000001.466.5, , , , , ] 2022-09-12T19:39:34.601 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:39:34.598+0000 7f6b483ef600 0 bucket checkpoint complete 2022-09-12T19:39:34.614 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-65 2022-09-12T19:39:34.615 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:39:34.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:39:34.616 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-65 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:39:34.685 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:39:34.686 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:39:34.740 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:39:34.733+0000 7f04ffa71600 1 bucket sync caught up with source: 2022-09-12T19:39:34.741 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000001.466.5, , , , , ] 2022-09-12T19:39:34.742 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000001.466.5, , , , , ] 2022-09-12T19:39:34.742 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:39:34.733+0000 7f04ffa71600 0 bucket checkpoint complete 2022-09-12T19:39:34.759 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-65 2022-09-12T19:39:34.761 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-65 zones={test-zone2, test-zone1} 2022-09-12T19:39:34.763 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:34.765 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:34.766 DEBUG:boto:Method: HEAD 2022-09-12T19:39:34.767 DEBUG:boto:Path: /jdpiak-65/ 2022-09-12T19:39:34.768 DEBUG:boto:Data: 2022-09-12T19:39:34.769 DEBUG:boto:Headers: {} 2022-09-12T19:39:34.770 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:34.771 DEBUG:boto:Port: 8000 2022-09-12T19:39:34.772 DEBUG:boto:Params: {} 2022-09-12T19:39:34.774 DEBUG:boto:Token: None 2022-09-12T19:39:34.775 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:39:34 GMT /jdpiak-65/ 2022-09-12T19:39:34.776 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uOZy9LclWZUBrnYMeRsAxSkXvVk= 2022-09-12T19:39:34.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uOZy9LclWZUBrnYMeRsAxSkXvVk=', 'Content-Length': '0'} 2022-09-12T19:39:34.782 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c87ec687543fd62a-00631f8af6-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:34.784 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:34.785 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:34.786 DEBUG:boto:Method: HEAD 2022-09-12T19:39:34.787 DEBUG:boto:Path: /jdpiak-65/ 2022-09-12T19:39:34.788 DEBUG:boto:Data: 2022-09-12T19:39:34.789 DEBUG:boto:Headers: {} 2022-09-12T19:39:34.790 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:34.791 DEBUG:boto:Port: 8000 2022-09-12T19:39:34.792 DEBUG:boto:Params: {} 2022-09-12T19:39:34.793 DEBUG:boto:Token: None 2022-09-12T19:39:34.794 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:39:34 GMT /jdpiak-65/ 2022-09-12T19:39:34.795 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uOZy9LclWZUBrnYMeRsAxSkXvVk= 2022-09-12T19:39:34.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uOZy9LclWZUBrnYMeRsAxSkXvVk=', 'Content-Length': '0'} 2022-09-12T19:39:34.800 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005099231d293d56a6-00631f8af6-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:34.801 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:39:34.802 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:34.803 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:34.804 DEBUG:boto:path=/jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:34.805 DEBUG:boto:auth_path=/jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:34.806 DEBUG:boto:Method: GET 2022-09-12T19:39:34.807 DEBUG:boto:Path: /jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:34.808 DEBUG:boto:Data: 2022-09-12T19:39:34.808 DEBUG:boto:Headers: {} 2022-09-12T19:39:34.809 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:34.809 DEBUG:boto:Port: 8000 2022-09-12T19:39:34.810 DEBUG:boto:Params: {} 2022-09-12T19:39:34.811 DEBUG:boto:Token: None 2022-09-12T19:39:34.811 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:34 GMT /jdpiak-65/?versions 2022-09-12T19:39:34.812 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KN5ot5Zu4qTfELo/ePk1j/7V7sI= 2022-09-12T19:39:34.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KN5ot5Zu4qTfELo/ePk1j/7V7sI=', 'Content-Length': '0'} 2022-09-12T19:39:34.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f06880a1f80bdd8f-00631f8af6-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:34.820 DEBUG:boto:b'jdpiak-65999falseMULTIPARTnulltrue2022-09-12T19:39:02.934Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:39:34.833 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2022-09-12T19:39:34.834 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:39:34.834 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:34.835 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:34.835 DEBUG:boto:path=/jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:34.836 DEBUG:boto:auth_path=/jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:34.836 DEBUG:boto:Method: GET 2022-09-12T19:39:34.837 DEBUG:boto:Path: /jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:34.837 DEBUG:boto:Data: 2022-09-12T19:39:34.838 DEBUG:boto:Headers: {} 2022-09-12T19:39:34.839 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:34.839 DEBUG:boto:Port: 8000 2022-09-12T19:39:34.840 DEBUG:boto:Params: {} 2022-09-12T19:39:34.840 DEBUG:boto:Token: None 2022-09-12T19:39:34.841 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:34 GMT /jdpiak-65/?versions 2022-09-12T19:39:34.841 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KN5ot5Zu4qTfELo/ePk1j/7V7sI= 2022-09-12T19:39:34.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KN5ot5Zu4qTfELo/ePk1j/7V7sI=', 'Content-Length': '0'} 2022-09-12T19:39:34.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b955af7beb93f923-00631f8af6-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:34.850 DEBUG:boto:b'jdpiak-65999falseMULTIPARTnulltrue2022-09-12T19:39:02.934Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:39:34.852 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2022-09-12T19:39:34.853 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:34.854 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:34.855 DEBUG:boto:path=/jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:34.856 DEBUG:boto:auth_path=/jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:34.857 DEBUG:boto:Method: GET 2022-09-12T19:39:34.859 DEBUG:boto:Path: /jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:34.860 DEBUG:boto:Data: 2022-09-12T19:39:34.861 DEBUG:boto:Headers: {} 2022-09-12T19:39:34.862 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:34.863 DEBUG:boto:Port: 8000 2022-09-12T19:39:34.864 DEBUG:boto:Params: {} 2022-09-12T19:39:34.865 DEBUG:boto:Token: None 2022-09-12T19:39:34.867 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:34 GMT /jdpiak-65/?versions 2022-09-12T19:39:34.868 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KN5ot5Zu4qTfELo/ePk1j/7V7sI= 2022-09-12T19:39:34.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KN5ot5Zu4qTfELo/ePk1j/7V7sI=', 'Content-Length': '0'} 2022-09-12T19:39:34.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a25ce406778bf11c-00631f8af6-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:34.884 DEBUG:boto:b'jdpiak-65999falseMULTIPARTnulltrue2022-09-12T19:39:02.934Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:39:34.885 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:34.887 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:34.888 DEBUG:boto:path=/jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:34.889 DEBUG:boto:auth_path=/jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:34.889 DEBUG:boto:Method: GET 2022-09-12T19:39:34.890 DEBUG:boto:Path: /jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:34.891 DEBUG:boto:Data: 2022-09-12T19:39:34.892 DEBUG:boto:Headers: {} 2022-09-12T19:39:34.893 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:34.894 DEBUG:boto:Port: 8000 2022-09-12T19:39:34.895 DEBUG:boto:Params: {} 2022-09-12T19:39:34.896 DEBUG:boto:Token: None 2022-09-12T19:39:34.897 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:34 GMT /jdpiak-65/?versions 2022-09-12T19:39:34.898 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KN5ot5Zu4qTfELo/ePk1j/7V7sI= 2022-09-12T19:39:34.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KN5ot5Zu4qTfELo/ePk1j/7V7sI=', 'Content-Length': '0'} 2022-09-12T19:39:34.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005563732812ae1fbe-00631f8af6-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:34.905 DEBUG:boto:b'jdpiak-65999falseMULTIPARTnulltrue2022-09-12T19:39:02.934Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:39:34.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2022-09-12T19:39:34.910 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:34.911 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:34.912 DEBUG:boto:path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:34.913 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:34.914 DEBUG:boto:Method: GET 2022-09-12T19:39:34.915 DEBUG:boto:Path: /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:34.917 DEBUG:boto:Data: 2022-09-12T19:39:34.918 DEBUG:boto:Headers: {} 2022-09-12T19:39:34.922 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:34.923 DEBUG:boto:Port: 8000 2022-09-12T19:39:34.924 DEBUG:boto:Params: {} 2022-09-12T19:39:34.925 DEBUG:boto:Token: None 2022-09-12T19:39:34.926 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:34 GMT /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:34.927 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tXj8VWLFoe2ra3bnQ56q3dJJBPQ= 2022-09-12T19:39:34.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tXj8VWLFoe2ra3bnQ56q3dJJBPQ=', 'Content-Length': '0'} 2022-09-12T19:39:34.942 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000e5b4f78180fa8db0-00631f8af6-15e1-test-zone2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:35.338 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:35.340 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:35.341 DEBUG:boto:path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:35.342 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:35.343 DEBUG:boto:Method: GET 2022-09-12T19:39:35.344 DEBUG:boto:Path: /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:35.346 DEBUG:boto:Data: 2022-09-12T19:39:35.347 DEBUG:boto:Headers: {} 2022-09-12T19:39:35.349 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:35.351 DEBUG:boto:Port: 8000 2022-09-12T19:39:35.352 DEBUG:boto:Params: {} 2022-09-12T19:39:35.353 DEBUG:boto:Token: None 2022-09-12T19:39:35.355 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:35 GMT /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:35.356 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3uJsVo7YipRQdrmJhENTknSLyf4= 2022-09-12T19:39:35.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3uJsVo7YipRQdrmJhENTknSLyf4=', 'Content-Length': '0'} 2022-09-12T19:39:35.365 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000054a2115be5bc7600-00631f8af7-15c0-test-zone1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:35.601 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:35.602 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:35.602 DEBUG:boto:path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:35.603 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:35.603 DEBUG:boto:Method: HEAD 2022-09-12T19:39:35.604 DEBUG:boto:Path: /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:35.604 DEBUG:boto:Data: 2022-09-12T19:39:35.605 DEBUG:boto:Headers: {} 2022-09-12T19:39:35.605 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:35.606 DEBUG:boto:Port: 8000 2022-09-12T19:39:35.606 DEBUG:boto:Params: {} 2022-09-12T19:39:35.607 DEBUG:boto:Token: None 2022-09-12T19:39:35.607 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:39:35 GMT /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:35.608 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:trBFDeSn1YgWIp6E/whooTk2TMM= 2022-09-12T19:39:35.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:trBFDeSn1YgWIp6E/whooTk2TMM=', 'Content-Length': '0'} 2022-09-12T19:39:35.611 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000937ada9cff63eb33-00631f8af7-15e1-test-zone2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:35.611 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:35.612 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:35.613 DEBUG:boto:path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:35.614 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:35.615 DEBUG:boto:Method: HEAD 2022-09-12T19:39:35.616 DEBUG:boto:Path: /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:35.617 DEBUG:boto:Data: 2022-09-12T19:39:35.617 DEBUG:boto:Headers: {} 2022-09-12T19:39:35.618 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:35.618 DEBUG:boto:Port: 8000 2022-09-12T19:39:35.619 DEBUG:boto:Params: {} 2022-09-12T19:39:35.620 DEBUG:boto:Token: None 2022-09-12T19:39:35.621 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:39:35 GMT /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:35.622 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:trBFDeSn1YgWIp6E/whooTk2TMM= 2022-09-12T19:39:35.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:trBFDeSn1YgWIp6E/whooTk2TMM=', 'Content-Length': '0'} 2022-09-12T19:39:35.627 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000008d8306f7f4734862-00631f8af7-15c0-test-zone1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:35.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2022-09-12T19:39:35.630 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:35.632 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:35.634 DEBUG:boto:path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:35.635 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:35.640 DEBUG:boto:Method: GET 2022-09-12T19:39:35.642 DEBUG:boto:Path: /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:35.643 DEBUG:boto:Data: 2022-09-12T19:39:35.644 DEBUG:boto:Headers: {} 2022-09-12T19:39:35.645 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:35.646 DEBUG:boto:Port: 8000 2022-09-12T19:39:35.646 DEBUG:boto:Params: {} 2022-09-12T19:39:35.648 DEBUG:boto:Token: None 2022-09-12T19:39:35.649 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:35 GMT /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:35.650 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3uJsVo7YipRQdrmJhENTknSLyf4= 2022-09-12T19:39:35.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3uJsVo7YipRQdrmJhENTknSLyf4=', 'Content-Length': '0'} 2022-09-12T19:39:35.654 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000c16ebeea95b8862f-00631f8af7-15e1-test-zone2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:35.823 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:35.824 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:35.825 DEBUG:boto:path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:35.825 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:35.826 DEBUG:boto:Method: GET 2022-09-12T19:39:35.826 DEBUG:boto:Path: /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:35.827 DEBUG:boto:Data: 2022-09-12T19:39:35.827 DEBUG:boto:Headers: {} 2022-09-12T19:39:35.828 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:35.829 DEBUG:boto:Port: 8000 2022-09-12T19:39:35.829 DEBUG:boto:Params: {} 2022-09-12T19:39:35.830 DEBUG:boto:Token: None 2022-09-12T19:39:35.831 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:35 GMT /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:35.831 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3uJsVo7YipRQdrmJhENTknSLyf4= 2022-09-12T19:39:35.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3uJsVo7YipRQdrmJhENTknSLyf4=', 'Content-Length': '0'} 2022-09-12T19:39:35.836 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000050bcd0c275693922-00631f8af7-15c0-test-zone1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:35.961 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:35.962 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:35.963 DEBUG:boto:path=/jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:35.964 DEBUG:boto:auth_path=/jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:35.964 DEBUG:boto:Method: GET 2022-09-12T19:39:35.965 DEBUG:boto:Path: /jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:35.965 DEBUG:boto:Data: 2022-09-12T19:39:35.966 DEBUG:boto:Headers: {} 2022-09-12T19:39:35.967 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:35.968 DEBUG:boto:Port: 8000 2022-09-12T19:39:35.968 DEBUG:boto:Params: {} 2022-09-12T19:39:35.969 DEBUG:boto:Token: None 2022-09-12T19:39:35.970 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:35 GMT /jdpiak-65/ 2022-09-12T19:39:35.971 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fIlgkdL4Kp0g/9SIl2KVVi70+tk= 2022-09-12T19:39:35.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fIlgkdL4Kp0g/9SIl2KVVi70+tk=', 'Content-Length': '0'} 2022-09-12T19:39:35.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ac86afa8caddb80-00631f8af7-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:35.977 DEBUG:boto:b'jdpiak-65MULTIPART1000falseMULTIPART2022-09-12T19:39:02.934Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:39:35.978 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:35.978 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:35.979 DEBUG:boto:path=/jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:35.979 DEBUG:boto:auth_path=/jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:35.980 DEBUG:boto:Method: GET 2022-09-12T19:39:35.980 DEBUG:boto:Path: /jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:35.981 DEBUG:boto:Data: 2022-09-12T19:39:35.981 DEBUG:boto:Headers: {} 2022-09-12T19:39:35.982 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:35.983 DEBUG:boto:Port: 8000 2022-09-12T19:39:35.984 DEBUG:boto:Params: {} 2022-09-12T19:39:35.985 DEBUG:boto:Token: None 2022-09-12T19:39:35.986 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:35 GMT /jdpiak-65/ 2022-09-12T19:39:35.987 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fIlgkdL4Kp0g/9SIl2KVVi70+tk= 2022-09-12T19:39:35.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fIlgkdL4Kp0g/9SIl2KVVi70+tk=', 'Content-Length': '0'} 2022-09-12T19:39:35.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a183ef5fb3a3c2f7-00631f8af7-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:39:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:35.993 DEBUG:boto:b'jdpiak-65MULTIPART1000falseMULTIPART2022-09-12T19:39:02.934Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:39:35.995 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:35.995 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:35.996 DEBUG:boto:Method: HEAD 2022-09-12T19:39:35.997 DEBUG:boto:Path: /jdpiak-65/MULTIPART 2022-09-12T19:39:35.997 DEBUG:boto:Data: 2022-09-12T19:39:35.998 DEBUG:boto:Headers: {} 2022-09-12T19:39:35.998 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:35.999 DEBUG:boto:Port: 8000 2022-09-12T19:39:35.999 DEBUG:boto:Params: {} 2022-09-12T19:39:36.000 DEBUG:boto:Token: None 2022-09-12T19:39:36.001 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:39:36 GMT /jdpiak-65/MULTIPART 2022-09-12T19:39:36.002 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2xOJGvpHIp9mPdzbvzd8lT6CZtA= 2022-09-12T19:39:36.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2xOJGvpHIp9mPdzbvzd8lT6CZtA=', 'Content-Length': '0'} 2022-09-12T19:39:36.005 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000004b7ee44fc18756f-00631f8af8-15e1-test-zone2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:36.005 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:36.006 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:36.007 DEBUG:boto:Method: HEAD 2022-09-12T19:39:36.007 DEBUG:boto:Path: /jdpiak-65/MULTIPART 2022-09-12T19:39:36.008 DEBUG:boto:Data: 2022-09-12T19:39:36.008 DEBUG:boto:Headers: {} 2022-09-12T19:39:36.009 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:36.010 DEBUG:boto:Port: 8000 2022-09-12T19:39:36.010 DEBUG:boto:Params: {} 2022-09-12T19:39:36.011 DEBUG:boto:Token: None 2022-09-12T19:39:36.012 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:39:36 GMT /jdpiak-65/MULTIPART 2022-09-12T19:39:36.012 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2xOJGvpHIp9mPdzbvzd8lT6CZtA= 2022-09-12T19:39:36.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2xOJGvpHIp9mPdzbvzd8lT6CZtA=', 'Content-Length': '0'} 2022-09-12T19:39:36.016 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000001f1be9aa46a42c58-00631f8af8-15c0-test-zone1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:36.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2022-09-12T19:39:36.017 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:36.018 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:36.019 DEBUG:boto:Method: GET 2022-09-12T19:39:36.020 DEBUG:boto:Path: /jdpiak-65/MULTIPART 2022-09-12T19:39:36.020 DEBUG:boto:Data: 2022-09-12T19:39:36.021 DEBUG:boto:Headers: {} 2022-09-12T19:39:36.021 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:36.022 DEBUG:boto:Port: 8000 2022-09-12T19:39:36.023 DEBUG:boto:Params: {} 2022-09-12T19:39:36.023 DEBUG:boto:Token: None 2022-09-12T19:39:36.024 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:36 GMT /jdpiak-65/MULTIPART 2022-09-12T19:39:36.025 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3WImrei1pvCAblZKntOtsPOzm/Y= 2022-09-12T19:39:36.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3WImrei1pvCAblZKntOtsPOzm/Y=', 'Content-Length': '0'} 2022-09-12T19:39:36.029 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000cd880832058757c3-00631f8af8-15e1-test-zone2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:36.152 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:36.153 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:36.153 DEBUG:boto:Method: GET 2022-09-12T19:39:36.154 DEBUG:boto:Path: /jdpiak-65/MULTIPART 2022-09-12T19:39:36.154 DEBUG:boto:Data: 2022-09-12T19:39:36.155 DEBUG:boto:Headers: {} 2022-09-12T19:39:36.156 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:36.156 DEBUG:boto:Port: 8000 2022-09-12T19:39:36.157 DEBUG:boto:Params: {} 2022-09-12T19:39:36.157 DEBUG:boto:Token: None 2022-09-12T19:39:36.158 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:36 GMT /jdpiak-65/MULTIPART 2022-09-12T19:39:36.159 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3WImrei1pvCAblZKntOtsPOzm/Y= 2022-09-12T19:39:36.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3WImrei1pvCAblZKntOtsPOzm/Y=', 'Content-Length': '0'} 2022-09-12T19:39:36.164 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000d819261455c4ed93-00631f8af8-15c0-test-zone1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:36.301 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:36.302 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:36.302 DEBUG:boto:path=/jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:36.303 DEBUG:boto:auth_path=/jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:36.304 DEBUG:boto:Method: GET 2022-09-12T19:39:36.304 DEBUG:boto:Path: /jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:36.305 DEBUG:boto:Data: 2022-09-12T19:39:36.305 DEBUG:boto:Headers: {} 2022-09-12T19:39:36.306 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:36.306 DEBUG:boto:Port: 8000 2022-09-12T19:39:36.307 DEBUG:boto:Params: {} 2022-09-12T19:39:36.307 DEBUG:boto:Token: None 2022-09-12T19:39:36.308 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:36 GMT /jdpiak-65/ 2022-09-12T19:39:36.309 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wgrlImQT631184roy1/LVQt/CZE= 2022-09-12T19:39:36.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wgrlImQT631184roy1/LVQt/CZE=', 'Content-Length': '0'} 2022-09-12T19:39:36.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a262756352a15fb-00631f8af8-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:36.316 DEBUG:boto:b'jdpiak-65MULTIPART1000falseMULTIPART2022-09-12T19:39:02.934Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:39:36.317 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:36.317 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:36.318 DEBUG:boto:path=/jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:36.318 DEBUG:boto:auth_path=/jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:36.319 DEBUG:boto:Method: GET 2022-09-12T19:39:36.319 DEBUG:boto:Path: /jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:36.320 DEBUG:boto:Data: 2022-09-12T19:39:36.320 DEBUG:boto:Headers: {} 2022-09-12T19:39:36.321 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:36.322 DEBUG:boto:Port: 8000 2022-09-12T19:39:36.322 DEBUG:boto:Params: {} 2022-09-12T19:39:36.323 DEBUG:boto:Token: None 2022-09-12T19:39:36.323 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:36 GMT /jdpiak-65/ 2022-09-12T19:39:36.324 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wgrlImQT631184roy1/LVQt/CZE= 2022-09-12T19:39:36.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wgrlImQT631184roy1/LVQt/CZE=', 'Content-Length': '0'} 2022-09-12T19:39:36.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c89ffcd8dca53e5b-00631f8af8-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:36.328 DEBUG:boto:b'jdpiak-65MULTIPART1000falseMULTIPART2022-09-12T19:39:02.934Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:39:36.329 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-65 zones={test-zone2, test-zone1} 2022-09-12T19:39:36.330 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-65 zones={test-zone3, test-zone1} 2022-09-12T19:39:36.330 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:36.331 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:36.331 DEBUG:boto:Method: HEAD 2022-09-12T19:39:36.332 DEBUG:boto:Path: /jdpiak-65/ 2022-09-12T19:39:36.333 DEBUG:boto:Data: 2022-09-12T19:39:36.333 DEBUG:boto:Headers: {} 2022-09-12T19:39:36.334 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:39:36.334 DEBUG:boto:Port: 8001 2022-09-12T19:39:36.335 DEBUG:boto:Params: {} 2022-09-12T19:39:36.335 DEBUG:boto:Token: None 2022-09-12T19:39:36.336 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:39:36 GMT /jdpiak-65/ 2022-09-12T19:39:36.337 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8KOtDBcX19BktssmmpJ7pQtwB3w= 2022-09-12T19:39:36.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8KOtDBcX19BktssmmpJ7pQtwB3w=', 'Content-Length': '0'} 2022-09-12T19:39:36.340 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000576b97b9d3bf6a08-00631f8af8-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:36.341 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:36.341 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:36.342 DEBUG:boto:Method: HEAD 2022-09-12T19:39:36.343 DEBUG:boto:Path: /jdpiak-65/ 2022-09-12T19:39:36.343 DEBUG:boto:Data: 2022-09-12T19:39:36.344 DEBUG:boto:Headers: {} 2022-09-12T19:39:36.344 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:36.345 DEBUG:boto:Port: 8000 2022-09-12T19:39:36.345 DEBUG:boto:Params: {} 2022-09-12T19:39:36.346 DEBUG:boto:Token: None 2022-09-12T19:39:36.347 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:39:36 GMT /jdpiak-65/ 2022-09-12T19:39:36.348 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8KOtDBcX19BktssmmpJ7pQtwB3w= 2022-09-12T19:39:36.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8KOtDBcX19BktssmmpJ7pQtwB3w=', 'Content-Length': '0'} 2022-09-12T19:39:36.352 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8fca392e0bee024-00631f8af8-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:36.353 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:39:36.354 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:36.354 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:36.355 DEBUG:boto:path=/jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:36.355 DEBUG:boto:auth_path=/jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:36.356 DEBUG:boto:Method: GET 2022-09-12T19:39:36.356 DEBUG:boto:Path: /jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:36.357 DEBUG:boto:Data: 2022-09-12T19:39:36.357 DEBUG:boto:Headers: {} 2022-09-12T19:39:36.358 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:39:36.358 DEBUG:boto:Port: 8001 2022-09-12T19:39:36.359 DEBUG:boto:Params: {} 2022-09-12T19:39:36.359 DEBUG:boto:Token: None 2022-09-12T19:39:36.360 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:36 GMT /jdpiak-65/?versions 2022-09-12T19:39:36.361 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N0fbDrgydEconjRujlym8LYixyw= 2022-09-12T19:39:36.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N0fbDrgydEconjRujlym8LYixyw=', 'Content-Length': '0'} 2022-09-12T19:39:36.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002512322486ebd55f-00631f8af8-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:36.365 DEBUG:boto:b'jdpiak-65999falseMULTIPARTnulltrue2022-09-12T19:39:02.934Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:39:36.366 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2022-09-12T19:39:36.366 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:39:36.367 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:36.368 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:36.368 DEBUG:boto:path=/jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:36.369 DEBUG:boto:auth_path=/jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:36.369 DEBUG:boto:Method: GET 2022-09-12T19:39:36.370 DEBUG:boto:Path: /jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:36.370 DEBUG:boto:Data: 2022-09-12T19:39:36.371 DEBUG:boto:Headers: {} 2022-09-12T19:39:36.372 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:36.372 DEBUG:boto:Port: 8000 2022-09-12T19:39:36.373 DEBUG:boto:Params: {} 2022-09-12T19:39:36.373 DEBUG:boto:Token: None 2022-09-12T19:39:36.374 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:36 GMT /jdpiak-65/?versions 2022-09-12T19:39:36.375 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N0fbDrgydEconjRujlym8LYixyw= 2022-09-12T19:39:36.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N0fbDrgydEconjRujlym8LYixyw=', 'Content-Length': '0'} 2022-09-12T19:39:36.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000185e80e8e748d8ff-00631f8af8-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:36.379 DEBUG:boto:b'jdpiak-65999falseMULTIPARTnulltrue2022-09-12T19:39:02.934Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:39:36.381 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2022-09-12T19:39:36.382 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:36.383 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:36.384 DEBUG:boto:path=/jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:36.385 DEBUG:boto:auth_path=/jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:36.385 DEBUG:boto:Method: GET 2022-09-12T19:39:36.386 DEBUG:boto:Path: /jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:36.387 DEBUG:boto:Data: 2022-09-12T19:39:36.387 DEBUG:boto:Headers: {} 2022-09-12T19:39:36.388 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:39:36.388 DEBUG:boto:Port: 8001 2022-09-12T19:39:36.389 DEBUG:boto:Params: {} 2022-09-12T19:39:36.390 DEBUG:boto:Token: None 2022-09-12T19:39:36.391 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:36 GMT /jdpiak-65/?versions 2022-09-12T19:39:36.392 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N0fbDrgydEconjRujlym8LYixyw= 2022-09-12T19:39:36.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N0fbDrgydEconjRujlym8LYixyw=', 'Content-Length': '0'} 2022-09-12T19:39:36.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8a9388858daa980-00631f8af8-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:36.396 DEBUG:boto:b'jdpiak-65999falseMULTIPARTnulltrue2022-09-12T19:39:02.934Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:39:36.397 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:36.398 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:36.398 DEBUG:boto:path=/jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:36.399 DEBUG:boto:auth_path=/jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:36.399 DEBUG:boto:Method: GET 2022-09-12T19:39:36.400 DEBUG:boto:Path: /jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:36.400 DEBUG:boto:Data: 2022-09-12T19:39:36.401 DEBUG:boto:Headers: {} 2022-09-12T19:39:36.401 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:36.402 DEBUG:boto:Port: 8000 2022-09-12T19:39:36.403 DEBUG:boto:Params: {} 2022-09-12T19:39:36.404 DEBUG:boto:Token: None 2022-09-12T19:39:36.405 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:36 GMT /jdpiak-65/?versions 2022-09-12T19:39:36.405 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N0fbDrgydEconjRujlym8LYixyw= 2022-09-12T19:39:36.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N0fbDrgydEconjRujlym8LYixyw=', 'Content-Length': '0'} 2022-09-12T19:39:36.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a49eada6c8e1c76-00631f8af8-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:36.450 DEBUG:boto:b'jdpiak-65999falseMULTIPARTnulltrue2022-09-12T19:39:02.934Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:39:36.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2022-09-12T19:39:36.452 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:36.453 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:36.454 DEBUG:boto:path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:36.454 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:36.455 DEBUG:boto:Method: GET 2022-09-12T19:39:36.457 DEBUG:boto:Path: /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:36.458 DEBUG:boto:Data: 2022-09-12T19:39:36.459 DEBUG:boto:Headers: {} 2022-09-12T19:39:36.460 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:39:36.462 DEBUG:boto:Port: 8001 2022-09-12T19:39:36.463 DEBUG:boto:Params: {} 2022-09-12T19:39:36.464 DEBUG:boto:Token: None 2022-09-12T19:39:36.465 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:36 GMT /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:36.466 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vcwxJl09bCgxJGE7RpQUQW4Vu14= 2022-09-12T19:39:36.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vcwxJl09bCgxJGE7RpQUQW4Vu14=', 'Content-Length': '0'} 2022-09-12T19:39:36.474 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000023c5ad037ead7580-00631f8af8-15c4-test-zone3'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:36.605 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:36.607 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:36.608 DEBUG:boto:path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:36.609 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:36.610 DEBUG:boto:Method: GET 2022-09-12T19:39:36.611 DEBUG:boto:Path: /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:36.612 DEBUG:boto:Data: 2022-09-12T19:39:36.613 DEBUG:boto:Headers: {} 2022-09-12T19:39:36.614 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:36.615 DEBUG:boto:Port: 8000 2022-09-12T19:39:36.616 DEBUG:boto:Params: {} 2022-09-12T19:39:36.618 DEBUG:boto:Token: None 2022-09-12T19:39:36.620 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:36 GMT /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:36.621 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vcwxJl09bCgxJGE7RpQUQW4Vu14= 2022-09-12T19:39:36.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vcwxJl09bCgxJGE7RpQUQW4Vu14=', 'Content-Length': '0'} 2022-09-12T19:39:36.627 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000004a5a6f454515cbce-00631f8af8-15c0-test-zone1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:36.766 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:36.767 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:36.767 DEBUG:boto:path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:36.768 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:36.768 DEBUG:boto:Method: HEAD 2022-09-12T19:39:36.769 DEBUG:boto:Path: /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:36.769 DEBUG:boto:Data: 2022-09-12T19:39:36.770 DEBUG:boto:Headers: {} 2022-09-12T19:39:36.770 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:39:36.771 DEBUG:boto:Port: 8001 2022-09-12T19:39:36.771 DEBUG:boto:Params: {} 2022-09-12T19:39:36.772 DEBUG:boto:Token: None 2022-09-12T19:39:36.773 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:39:36 GMT /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:36.773 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qgt80iyDMJPYchcxoReGddGpSqI= 2022-09-12T19:39:36.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qgt80iyDMJPYchcxoReGddGpSqI=', 'Content-Length': '0'} 2022-09-12T19:39:36.777 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000003ee9b49646f7fad5-00631f8af8-15c4-test-zone3'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:36.777 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:36.778 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:36.779 DEBUG:boto:path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:36.779 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:36.780 DEBUG:boto:Method: HEAD 2022-09-12T19:39:36.781 DEBUG:boto:Path: /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:36.781 DEBUG:boto:Data: 2022-09-12T19:39:36.782 DEBUG:boto:Headers: {} 2022-09-12T19:39:36.783 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:36.783 DEBUG:boto:Port: 8000 2022-09-12T19:39:36.784 DEBUG:boto:Params: {} 2022-09-12T19:39:36.784 DEBUG:boto:Token: None 2022-09-12T19:39:36.785 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:39:36 GMT /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:36.786 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qgt80iyDMJPYchcxoReGddGpSqI= 2022-09-12T19:39:36.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qgt80iyDMJPYchcxoReGddGpSqI=', 'Content-Length': '0'} 2022-09-12T19:39:36.789 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000001cce9c1b8d9bd34b-00631f8af8-15c0-test-zone1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:36.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2022-09-12T19:39:36.791 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:36.791 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:36.792 DEBUG:boto:path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:36.792 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:36.793 DEBUG:boto:Method: GET 2022-09-12T19:39:36.793 DEBUG:boto:Path: /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:36.794 DEBUG:boto:Data: 2022-09-12T19:39:36.794 DEBUG:boto:Headers: {} 2022-09-12T19:39:36.795 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:39:36.795 DEBUG:boto:Port: 8001 2022-09-12T19:39:36.796 DEBUG:boto:Params: {} 2022-09-12T19:39:36.797 DEBUG:boto:Token: None 2022-09-12T19:39:36.798 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:36 GMT /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:36.799 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vcwxJl09bCgxJGE7RpQUQW4Vu14= 2022-09-12T19:39:36.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vcwxJl09bCgxJGE7RpQUQW4Vu14=', 'Content-Length': '0'} 2022-09-12T19:39:36.803 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000f95cdbe619621466-00631f8af8-15c4-test-zone3'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:36.894 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:36.895 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:36.896 DEBUG:boto:path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:36.896 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:36.897 DEBUG:boto:Method: GET 2022-09-12T19:39:36.897 DEBUG:boto:Path: /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:36.898 DEBUG:boto:Data: 2022-09-12T19:39:36.898 DEBUG:boto:Headers: {} 2022-09-12T19:39:36.899 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:36.899 DEBUG:boto:Port: 8000 2022-09-12T19:39:36.900 DEBUG:boto:Params: {} 2022-09-12T19:39:36.901 DEBUG:boto:Token: None 2022-09-12T19:39:36.901 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:36 GMT /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:36.902 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vcwxJl09bCgxJGE7RpQUQW4Vu14= 2022-09-12T19:39:36.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vcwxJl09bCgxJGE7RpQUQW4Vu14=', 'Content-Length': '0'} 2022-09-12T19:39:36.907 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000d27eccc53b73a9fe-00631f8af8-15c0-test-zone1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:37.025 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:37.026 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:37.028 DEBUG:boto:path=/jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:37.029 DEBUG:boto:auth_path=/jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:37.030 DEBUG:boto:Method: GET 2022-09-12T19:39:37.031 DEBUG:boto:Path: /jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:37.032 DEBUG:boto:Data: 2022-09-12T19:39:37.033 DEBUG:boto:Headers: {} 2022-09-12T19:39:37.034 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:39:37.035 DEBUG:boto:Port: 8001 2022-09-12T19:39:37.036 DEBUG:boto:Params: {} 2022-09-12T19:39:37.037 DEBUG:boto:Token: None 2022-09-12T19:39:37.038 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:37 GMT /jdpiak-65/ 2022-09-12T19:39:37.039 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8hw6EuZw0NSZI+pKrtoX7A7YgSg= 2022-09-12T19:39:37.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8hw6EuZw0NSZI+pKrtoX7A7YgSg=', 'Content-Length': '0'} 2022-09-12T19:39:37.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f713c6358a287b89-00631f8af9-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:37.045 DEBUG:boto:b'jdpiak-65MULTIPART1000falseMULTIPART2022-09-12T19:39:02.934Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:39:37.047 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:37.047 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:37.048 DEBUG:boto:path=/jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:37.049 DEBUG:boto:auth_path=/jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:37.050 DEBUG:boto:Method: GET 2022-09-12T19:39:37.051 DEBUG:boto:Path: /jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:37.052 DEBUG:boto:Data: 2022-09-12T19:39:37.052 DEBUG:boto:Headers: {} 2022-09-12T19:39:37.053 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:37.053 DEBUG:boto:Port: 8000 2022-09-12T19:39:37.054 DEBUG:boto:Params: {} 2022-09-12T19:39:37.055 DEBUG:boto:Token: None 2022-09-12T19:39:37.055 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:37 GMT /jdpiak-65/ 2022-09-12T19:39:37.056 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8hw6EuZw0NSZI+pKrtoX7A7YgSg= 2022-09-12T19:39:37.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8hw6EuZw0NSZI+pKrtoX7A7YgSg=', 'Content-Length': '0'} 2022-09-12T19:39:37.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdf187a8a03fd3f0-00631f8af9-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:37.061 DEBUG:boto:b'jdpiak-65MULTIPART1000falseMULTIPART2022-09-12T19:39:02.934Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:39:37.062 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:37.063 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:37.064 DEBUG:boto:Method: HEAD 2022-09-12T19:39:37.064 DEBUG:boto:Path: /jdpiak-65/MULTIPART 2022-09-12T19:39:37.065 DEBUG:boto:Data: 2022-09-12T19:39:37.065 DEBUG:boto:Headers: {} 2022-09-12T19:39:37.066 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:39:37.066 DEBUG:boto:Port: 8001 2022-09-12T19:39:37.067 DEBUG:boto:Params: {} 2022-09-12T19:39:37.069 DEBUG:boto:Token: None 2022-09-12T19:39:37.069 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:39:37 GMT /jdpiak-65/MULTIPART 2022-09-12T19:39:37.070 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/cAWBiLPCbdf/xQbSPgvt3Mv4fY= 2022-09-12T19:39:37.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/cAWBiLPCbdf/xQbSPgvt3Mv4fY=', 'Content-Length': '0'} 2022-09-12T19:39:37.073 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000007a648eebd635da01-00631f8af9-15c4-test-zone3'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:37.075 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:37.075 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:37.076 DEBUG:boto:Method: HEAD 2022-09-12T19:39:37.077 DEBUG:boto:Path: /jdpiak-65/MULTIPART 2022-09-12T19:39:37.077 DEBUG:boto:Data: 2022-09-12T19:39:37.078 DEBUG:boto:Headers: {} 2022-09-12T19:39:37.078 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:37.079 DEBUG:boto:Port: 8000 2022-09-12T19:39:37.079 DEBUG:boto:Params: {} 2022-09-12T19:39:37.080 DEBUG:boto:Token: None 2022-09-12T19:39:37.081 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:39:37 GMT /jdpiak-65/MULTIPART 2022-09-12T19:39:37.082 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/cAWBiLPCbdf/xQbSPgvt3Mv4fY= 2022-09-12T19:39:37.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/cAWBiLPCbdf/xQbSPgvt3Mv4fY=', 'Content-Length': '0'} 2022-09-12T19:39:37.085 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000002d42649e0eaea164-00631f8af9-15c0-test-zone1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:37.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2022-09-12T19:39:37.087 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:37.088 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:37.089 DEBUG:boto:Method: GET 2022-09-12T19:39:37.090 DEBUG:boto:Path: /jdpiak-65/MULTIPART 2022-09-12T19:39:37.090 DEBUG:boto:Data: 2022-09-12T19:39:37.091 DEBUG:boto:Headers: {} 2022-09-12T19:39:37.092 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:39:37.092 DEBUG:boto:Port: 8001 2022-09-12T19:39:37.093 DEBUG:boto:Params: {} 2022-09-12T19:39:37.094 DEBUG:boto:Token: None 2022-09-12T19:39:37.095 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:37 GMT /jdpiak-65/MULTIPART 2022-09-12T19:39:37.095 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ss265E7XHcv4DJmEVaLTx9hghsI= 2022-09-12T19:39:37.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ss265E7XHcv4DJmEVaLTx9hghsI=', 'Content-Length': '0'} 2022-09-12T19:39:37.100 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000f25043949dcc511c-00631f8af9-15c4-test-zone3'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:37.216 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:37.216 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:37.217 DEBUG:boto:Method: GET 2022-09-12T19:39:37.217 DEBUG:boto:Path: /jdpiak-65/MULTIPART 2022-09-12T19:39:37.218 DEBUG:boto:Data: 2022-09-12T19:39:37.219 DEBUG:boto:Headers: {} 2022-09-12T19:39:37.219 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:37.220 DEBUG:boto:Port: 8000 2022-09-12T19:39:37.220 DEBUG:boto:Params: {} 2022-09-12T19:39:37.221 DEBUG:boto:Token: None 2022-09-12T19:39:37.222 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:37 GMT /jdpiak-65/MULTIPART 2022-09-12T19:39:37.223 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ss265E7XHcv4DJmEVaLTx9hghsI= 2022-09-12T19:39:37.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ss265E7XHcv4DJmEVaLTx9hghsI=', 'Content-Length': '0'} 2022-09-12T19:39:37.227 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000ec15f9dffd3b9f0c-00631f8af9-15c0-test-zone1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:37.373 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:37.374 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:37.374 DEBUG:boto:path=/jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:37.375 DEBUG:boto:auth_path=/jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:37.375 DEBUG:boto:Method: GET 2022-09-12T19:39:37.376 DEBUG:boto:Path: /jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:37.377 DEBUG:boto:Data: 2022-09-12T19:39:37.377 DEBUG:boto:Headers: {} 2022-09-12T19:39:37.378 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:39:37.378 DEBUG:boto:Port: 8001 2022-09-12T19:39:37.379 DEBUG:boto:Params: {} 2022-09-12T19:39:37.380 DEBUG:boto:Token: None 2022-09-12T19:39:37.380 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:37 GMT /jdpiak-65/ 2022-09-12T19:39:37.381 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8hw6EuZw0NSZI+pKrtoX7A7YgSg= 2022-09-12T19:39:37.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8hw6EuZw0NSZI+pKrtoX7A7YgSg=', 'Content-Length': '0'} 2022-09-12T19:39:37.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac77080543859e89-00631f8af9-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:37.387 DEBUG:boto:b'jdpiak-65MULTIPART1000falseMULTIPART2022-09-12T19:39:02.934Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:39:37.388 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:37.388 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:37.389 DEBUG:boto:path=/jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:37.389 DEBUG:boto:auth_path=/jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:37.390 DEBUG:boto:Method: GET 2022-09-12T19:39:37.391 DEBUG:boto:Path: /jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:37.392 DEBUG:boto:Data: 2022-09-12T19:39:37.393 DEBUG:boto:Headers: {} 2022-09-12T19:39:37.394 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:37.395 DEBUG:boto:Port: 8000 2022-09-12T19:39:37.396 DEBUG:boto:Params: {} 2022-09-12T19:39:37.397 DEBUG:boto:Token: None 2022-09-12T19:39:37.398 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:37 GMT /jdpiak-65/ 2022-09-12T19:39:37.399 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8hw6EuZw0NSZI+pKrtoX7A7YgSg= 2022-09-12T19:39:37.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8hw6EuZw0NSZI+pKrtoX7A7YgSg=', 'Content-Length': '0'} 2022-09-12T19:39:37.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008565e1b8b1f6fd34-00631f8af9-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:37.404 DEBUG:boto:b'jdpiak-65MULTIPART1000falseMULTIPART2022-09-12T19:39:02.934Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:39:37.406 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-65 zones={test-zone3, test-zone1} 2022-09-12T19:39:37.407 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-65 zones={test-zone3, test-zone2} 2022-09-12T19:39:37.407 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:37.408 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:37.409 DEBUG:boto:Method: HEAD 2022-09-12T19:39:37.410 DEBUG:boto:Path: /jdpiak-65/ 2022-09-12T19:39:37.411 DEBUG:boto:Data: 2022-09-12T19:39:37.412 DEBUG:boto:Headers: {} 2022-09-12T19:39:37.413 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:39:37.414 DEBUG:boto:Port: 8001 2022-09-12T19:39:37.415 DEBUG:boto:Params: {} 2022-09-12T19:39:37.416 DEBUG:boto:Token: None 2022-09-12T19:39:37.417 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:39:37 GMT /jdpiak-65/ 2022-09-12T19:39:37.418 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kpSvFAx2Zf5E+vmJ4nQOTmyLlVY= 2022-09-12T19:39:37.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kpSvFAx2Zf5E+vmJ4nQOTmyLlVY=', 'Content-Length': '0'} 2022-09-12T19:39:37.427 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060d096afb96908c7-00631f8af9-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:37.428 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:37.429 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:37.430 DEBUG:boto:Method: HEAD 2022-09-12T19:39:37.431 DEBUG:boto:Path: /jdpiak-65/ 2022-09-12T19:39:37.431 DEBUG:boto:Data: 2022-09-12T19:39:37.432 DEBUG:boto:Headers: {} 2022-09-12T19:39:37.433 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:37.433 DEBUG:boto:Port: 8000 2022-09-12T19:39:37.434 DEBUG:boto:Params: {} 2022-09-12T19:39:37.434 DEBUG:boto:Token: None 2022-09-12T19:39:37.435 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:39:37 GMT /jdpiak-65/ 2022-09-12T19:39:37.436 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kpSvFAx2Zf5E+vmJ4nQOTmyLlVY= 2022-09-12T19:39:37.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kpSvFAx2Zf5E+vmJ4nQOTmyLlVY=', 'Content-Length': '0'} 2022-09-12T19:39:37.439 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fac0854cffe0c299-00631f8af9-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:37.440 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:39:37.441 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:37.441 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:37.442 DEBUG:boto:path=/jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:37.443 DEBUG:boto:auth_path=/jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:37.444 DEBUG:boto:Method: GET 2022-09-12T19:39:37.445 DEBUG:boto:Path: /jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:37.446 DEBUG:boto:Data: 2022-09-12T19:39:37.447 DEBUG:boto:Headers: {} 2022-09-12T19:39:37.448 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:39:37.449 DEBUG:boto:Port: 8001 2022-09-12T19:39:37.450 DEBUG:boto:Params: {} 2022-09-12T19:39:37.451 DEBUG:boto:Token: None 2022-09-12T19:39:37.451 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:37 GMT /jdpiak-65/?versions 2022-09-12T19:39:37.452 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZYAMbmXF6FYU5fJyq9BF+K6sYcE= 2022-09-12T19:39:37.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZYAMbmXF6FYU5fJyq9BF+K6sYcE=', 'Content-Length': '0'} 2022-09-12T19:39:37.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de35c7244076b278-00631f8af9-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:37.457 DEBUG:boto:b'jdpiak-65999falseMULTIPARTnulltrue2022-09-12T19:39:02.934Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:39:37.458 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2022-09-12T19:39:37.459 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:39:37.459 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:37.460 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:37.460 DEBUG:boto:path=/jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:37.461 DEBUG:boto:auth_path=/jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:37.462 DEBUG:boto:Method: GET 2022-09-12T19:39:37.462 DEBUG:boto:Path: /jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:37.463 DEBUG:boto:Data: 2022-09-12T19:39:37.463 DEBUG:boto:Headers: {} 2022-09-12T19:39:37.464 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:37.464 DEBUG:boto:Port: 8000 2022-09-12T19:39:37.465 DEBUG:boto:Params: {} 2022-09-12T19:39:37.465 DEBUG:boto:Token: None 2022-09-12T19:39:37.466 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:37 GMT /jdpiak-65/?versions 2022-09-12T19:39:37.467 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZYAMbmXF6FYU5fJyq9BF+K6sYcE= 2022-09-12T19:39:37.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZYAMbmXF6FYU5fJyq9BF+K6sYcE=', 'Content-Length': '0'} 2022-09-12T19:39:37.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000028c487d05a52f65-00631f8af9-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:37.472 DEBUG:boto:b'jdpiak-65999falseMULTIPARTnulltrue2022-09-12T19:39:02.934Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:39:37.473 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2022-09-12T19:39:37.474 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:37.475 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:37.476 DEBUG:boto:path=/jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:37.477 DEBUG:boto:auth_path=/jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:37.478 DEBUG:boto:Method: GET 2022-09-12T19:39:37.479 DEBUG:boto:Path: /jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:37.480 DEBUG:boto:Data: 2022-09-12T19:39:37.481 DEBUG:boto:Headers: {} 2022-09-12T19:39:37.482 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:39:37.483 DEBUG:boto:Port: 8001 2022-09-12T19:39:37.484 DEBUG:boto:Params: {} 2022-09-12T19:39:37.486 DEBUG:boto:Token: None 2022-09-12T19:39:37.487 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:37 GMT /jdpiak-65/?versions 2022-09-12T19:39:37.488 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZYAMbmXF6FYU5fJyq9BF+K6sYcE= 2022-09-12T19:39:37.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZYAMbmXF6FYU5fJyq9BF+K6sYcE=', 'Content-Length': '0'} 2022-09-12T19:39:37.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008aef3f1396719030-00631f8af9-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:37.495 DEBUG:boto:b'jdpiak-65999falseMULTIPARTnulltrue2022-09-12T19:39:02.934Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:39:37.497 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:37.498 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:37.499 DEBUG:boto:path=/jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:37.501 DEBUG:boto:auth_path=/jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:37.502 DEBUG:boto:Method: GET 2022-09-12T19:39:37.502 DEBUG:boto:Path: /jdpiak-65/?versions&max-keys=999 2022-09-12T19:39:37.503 DEBUG:boto:Data: 2022-09-12T19:39:37.503 DEBUG:boto:Headers: {} 2022-09-12T19:39:37.504 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:37.504 DEBUG:boto:Port: 8000 2022-09-12T19:39:37.505 DEBUG:boto:Params: {} 2022-09-12T19:39:37.506 DEBUG:boto:Token: None 2022-09-12T19:39:37.506 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:37 GMT /jdpiak-65/?versions 2022-09-12T19:39:37.507 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZYAMbmXF6FYU5fJyq9BF+K6sYcE= 2022-09-12T19:39:37.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZYAMbmXF6FYU5fJyq9BF+K6sYcE=', 'Content-Length': '0'} 2022-09-12T19:39:37.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae5e28b5160f235d-00631f8af9-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:37.514 DEBUG:boto:b'jdpiak-65999falseMULTIPARTnulltrue2022-09-12T19:39:02.934Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:39:37.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2022-09-12T19:39:37.516 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:37.516 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:37.517 DEBUG:boto:path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:37.517 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:37.518 DEBUG:boto:Method: GET 2022-09-12T19:39:37.518 DEBUG:boto:Path: /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:37.519 DEBUG:boto:Data: 2022-09-12T19:39:37.520 DEBUG:boto:Headers: {} 2022-09-12T19:39:37.520 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:39:37.521 DEBUG:boto:Port: 8001 2022-09-12T19:39:37.522 DEBUG:boto:Params: {} 2022-09-12T19:39:37.522 DEBUG:boto:Token: None 2022-09-12T19:39:37.523 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:37 GMT /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:37.525 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7cDW+7DC1IjCmNH8FLwDyAAC1y8= 2022-09-12T19:39:37.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7cDW+7DC1IjCmNH8FLwDyAAC1y8=', 'Content-Length': '0'} 2022-09-12T19:39:37.530 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000021709f754c279c6a-00631f8af9-15c4-test-zone3'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:37.657 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:37.658 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:37.659 DEBUG:boto:path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:37.659 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:37.660 DEBUG:boto:Method: GET 2022-09-12T19:39:37.660 DEBUG:boto:Path: /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:37.661 DEBUG:boto:Data: 2022-09-12T19:39:37.662 DEBUG:boto:Headers: {} 2022-09-12T19:39:37.662 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:37.663 DEBUG:boto:Port: 8000 2022-09-12T19:39:37.664 DEBUG:boto:Params: {} 2022-09-12T19:39:37.664 DEBUG:boto:Token: None 2022-09-12T19:39:37.665 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:37 GMT /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:37.666 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7cDW+7DC1IjCmNH8FLwDyAAC1y8= 2022-09-12T19:39:37.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7cDW+7DC1IjCmNH8FLwDyAAC1y8=', 'Content-Length': '0'} 2022-09-12T19:39:37.671 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000d1df691ffd632b03-00631f8af9-15e1-test-zone2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:37.836 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:37.837 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:37.838 DEBUG:boto:path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:37.839 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:37.839 DEBUG:boto:Method: HEAD 2022-09-12T19:39:37.840 DEBUG:boto:Path: /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:37.841 DEBUG:boto:Data: 2022-09-12T19:39:37.841 DEBUG:boto:Headers: {} 2022-09-12T19:39:37.842 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:39:37.843 DEBUG:boto:Port: 8001 2022-09-12T19:39:37.843 DEBUG:boto:Params: {} 2022-09-12T19:39:37.844 DEBUG:boto:Token: None 2022-09-12T19:39:37.845 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:39:37 GMT /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:37.845 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Z9O9bYphNpz9eYI0fyq/mTTC2Gw= 2022-09-12T19:39:37.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Z9O9bYphNpz9eYI0fyq/mTTC2Gw=', 'Content-Length': '0'} 2022-09-12T19:39:37.848 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000003abb8066ece89250-00631f8af9-15c4-test-zone3'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:37.849 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:37.849 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:37.850 DEBUG:boto:path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:37.850 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:37.851 DEBUG:boto:Method: HEAD 2022-09-12T19:39:37.852 DEBUG:boto:Path: /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:37.852 DEBUG:boto:Data: 2022-09-12T19:39:37.853 DEBUG:boto:Headers: {} 2022-09-12T19:39:37.854 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:37.855 DEBUG:boto:Port: 8000 2022-09-12T19:39:37.856 DEBUG:boto:Params: {} 2022-09-12T19:39:37.857 DEBUG:boto:Token: None 2022-09-12T19:39:37.858 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:39:37 GMT /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:37.859 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Z9O9bYphNpz9eYI0fyq/mTTC2Gw= 2022-09-12T19:39:37.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Z9O9bYphNpz9eYI0fyq/mTTC2Gw=', 'Content-Length': '0'} 2022-09-12T19:39:37.862 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000a4784917982edd2f-00631f8af9-15e1-test-zone2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:37.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2022-09-12T19:39:37.863 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:37.864 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:37.864 DEBUG:boto:path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:37.865 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:37.865 DEBUG:boto:Method: GET 2022-09-12T19:39:37.866 DEBUG:boto:Path: /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:37.866 DEBUG:boto:Data: 2022-09-12T19:39:37.867 DEBUG:boto:Headers: {} 2022-09-12T19:39:37.867 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:39:37.868 DEBUG:boto:Port: 8001 2022-09-12T19:39:37.868 DEBUG:boto:Params: {} 2022-09-12T19:39:37.869 DEBUG:boto:Token: None 2022-09-12T19:39:37.870 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:37 GMT /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:37.871 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7cDW+7DC1IjCmNH8FLwDyAAC1y8= 2022-09-12T19:39:37.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7cDW+7DC1IjCmNH8FLwDyAAC1y8=', 'Content-Length': '0'} 2022-09-12T19:39:37.874 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000ee984ee9e86c5483-00631f8af9-15c4-test-zone3'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:37.990 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:37.991 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:37.991 DEBUG:boto:path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:37.992 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:37.992 DEBUG:boto:Method: GET 2022-09-12T19:39:37.993 DEBUG:boto:Path: /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:37.994 DEBUG:boto:Data: 2022-09-12T19:39:37.994 DEBUG:boto:Headers: {} 2022-09-12T19:39:37.995 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:37.995 DEBUG:boto:Port: 8000 2022-09-12T19:39:37.996 DEBUG:boto:Params: {} 2022-09-12T19:39:37.997 DEBUG:boto:Token: None 2022-09-12T19:39:37.997 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:37 GMT /jdpiak-65/MULTIPART?versionId=null 2022-09-12T19:39:37.998 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7cDW+7DC1IjCmNH8FLwDyAAC1y8= 2022-09-12T19:39:37.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7cDW+7DC1IjCmNH8FLwDyAAC1y8=', 'Content-Length': '0'} 2022-09-12T19:39:38.002 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000ba84bc54d6c0bc14-00631f8af9-15e1-test-zone2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:38.159 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:38.160 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:38.161 DEBUG:boto:path=/jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:38.161 DEBUG:boto:auth_path=/jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:38.162 DEBUG:boto:Method: GET 2022-09-12T19:39:38.162 DEBUG:boto:Path: /jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:38.163 DEBUG:boto:Data: 2022-09-12T19:39:38.163 DEBUG:boto:Headers: {} 2022-09-12T19:39:38.164 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:39:38.164 DEBUG:boto:Port: 8001 2022-09-12T19:39:38.165 DEBUG:boto:Params: {} 2022-09-12T19:39:38.166 DEBUG:boto:Token: None 2022-09-12T19:39:38.166 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:38 GMT /jdpiak-65/ 2022-09-12T19:39:38.167 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ekOov4CPrCJJ0CS1wka389DVix0= 2022-09-12T19:39:38.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ekOov4CPrCJJ0CS1wka389DVix0=', 'Content-Length': '0'} 2022-09-12T19:39:38.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c7b0795b0725239-00631f8afa-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:38.171 DEBUG:boto:b'jdpiak-65MULTIPART1000falseMULTIPART2022-09-12T19:39:02.934Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:39:38.172 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:38.173 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:38.173 DEBUG:boto:path=/jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:38.174 DEBUG:boto:auth_path=/jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:38.174 DEBUG:boto:Method: GET 2022-09-12T19:39:38.175 DEBUG:boto:Path: /jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:38.175 DEBUG:boto:Data: 2022-09-12T19:39:38.176 DEBUG:boto:Headers: {} 2022-09-12T19:39:38.176 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:38.177 DEBUG:boto:Port: 8000 2022-09-12T19:39:38.177 DEBUG:boto:Params: {} 2022-09-12T19:39:38.178 DEBUG:boto:Token: None 2022-09-12T19:39:38.179 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:38 GMT /jdpiak-65/ 2022-09-12T19:39:38.179 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ekOov4CPrCJJ0CS1wka389DVix0= 2022-09-12T19:39:38.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ekOov4CPrCJJ0CS1wka389DVix0=', 'Content-Length': '0'} 2022-09-12T19:39:38.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c252bad08197bb5-00631f8afa-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:38.184 DEBUG:boto:b'jdpiak-65MULTIPART1000falseMULTIPART2022-09-12T19:39:02.934Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:39:38.186 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:38.187 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:38.188 DEBUG:boto:Method: HEAD 2022-09-12T19:39:38.189 DEBUG:boto:Path: /jdpiak-65/MULTIPART 2022-09-12T19:39:38.190 DEBUG:boto:Data: 2022-09-12T19:39:38.190 DEBUG:boto:Headers: {} 2022-09-12T19:39:38.191 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:39:38.191 DEBUG:boto:Port: 8001 2022-09-12T19:39:38.192 DEBUG:boto:Params: {} 2022-09-12T19:39:38.192 DEBUG:boto:Token: None 2022-09-12T19:39:38.193 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:39:38 GMT /jdpiak-65/MULTIPART 2022-09-12T19:39:38.194 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7/b6aM1aGHZq+FQCFR9u5RXElKE= 2022-09-12T19:39:38.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7/b6aM1aGHZq+FQCFR9u5RXElKE=', 'Content-Length': '0'} 2022-09-12T19:39:38.197 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000b3f6dfd059e7298e-00631f8afa-15c4-test-zone3'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:38.197 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:38.198 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:38.198 DEBUG:boto:Method: HEAD 2022-09-12T19:39:38.199 DEBUG:boto:Path: /jdpiak-65/MULTIPART 2022-09-12T19:39:38.200 DEBUG:boto:Data: 2022-09-12T19:39:38.200 DEBUG:boto:Headers: {} 2022-09-12T19:39:38.201 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:38.201 DEBUG:boto:Port: 8000 2022-09-12T19:39:38.202 DEBUG:boto:Params: {} 2022-09-12T19:39:38.202 DEBUG:boto:Token: None 2022-09-12T19:39:38.203 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:39:38 GMT /jdpiak-65/MULTIPART 2022-09-12T19:39:38.204 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7/b6aM1aGHZq+FQCFR9u5RXElKE= 2022-09-12T19:39:38.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7/b6aM1aGHZq+FQCFR9u5RXElKE=', 'Content-Length': '0'} 2022-09-12T19:39:38.207 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000069626ef1553b1900-00631f8afa-15e1-test-zone2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:38.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2022-09-12T19:39:38.208 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:38.209 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:38.210 DEBUG:boto:Method: GET 2022-09-12T19:39:38.211 DEBUG:boto:Path: /jdpiak-65/MULTIPART 2022-09-12T19:39:38.212 DEBUG:boto:Data: 2022-09-12T19:39:38.213 DEBUG:boto:Headers: {} 2022-09-12T19:39:38.214 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:39:38.215 DEBUG:boto:Port: 8001 2022-09-12T19:39:38.215 DEBUG:boto:Params: {} 2022-09-12T19:39:38.216 DEBUG:boto:Token: None 2022-09-12T19:39:38.217 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:38 GMT /jdpiak-65/MULTIPART 2022-09-12T19:39:38.218 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qXMSSjHYsLg3cem3w//zAHWXWcc= 2022-09-12T19:39:38.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qXMSSjHYsLg3cem3w//zAHWXWcc=', 'Content-Length': '0'} 2022-09-12T19:39:38.222 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000003bf500cd343c24b3-00631f8afa-15c4-test-zone3'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:38.359 DEBUG:boto:path=/jdpiak-65/MULTIPART 2022-09-12T19:39:38.360 DEBUG:boto:auth_path=/jdpiak-65/MULTIPART 2022-09-12T19:39:38.360 DEBUG:boto:Method: GET 2022-09-12T19:39:38.361 DEBUG:boto:Path: /jdpiak-65/MULTIPART 2022-09-12T19:39:38.361 DEBUG:boto:Data: 2022-09-12T19:39:38.362 DEBUG:boto:Headers: {} 2022-09-12T19:39:38.362 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:38.363 DEBUG:boto:Port: 8000 2022-09-12T19:39:38.363 DEBUG:boto:Params: {} 2022-09-12T19:39:38.364 DEBUG:boto:Token: None 2022-09-12T19:39:38.365 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:38 GMT /jdpiak-65/MULTIPART 2022-09-12T19:39:38.365 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qXMSSjHYsLg3cem3w//zAHWXWcc= 2022-09-12T19:39:38.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qXMSSjHYsLg3cem3w//zAHWXWcc=', 'Content-Length': '0'} 2022-09-12T19:39:38.369 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000d01bf964f8b82c26-00631f8afa-15e1-test-zone2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:38.518 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:38.519 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:38.520 DEBUG:boto:path=/jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:38.520 DEBUG:boto:auth_path=/jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:38.521 DEBUG:boto:Method: GET 2022-09-12T19:39:38.521 DEBUG:boto:Path: /jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:38.522 DEBUG:boto:Data: 2022-09-12T19:39:38.523 DEBUG:boto:Headers: {} 2022-09-12T19:39:38.523 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:39:38.524 DEBUG:boto:Port: 8001 2022-09-12T19:39:38.524 DEBUG:boto:Params: {} 2022-09-12T19:39:38.525 DEBUG:boto:Token: None 2022-09-12T19:39:38.526 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:38 GMT /jdpiak-65/ 2022-09-12T19:39:38.526 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ekOov4CPrCJJ0CS1wka389DVix0= 2022-09-12T19:39:38.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ekOov4CPrCJJ0CS1wka389DVix0=', 'Content-Length': '0'} 2022-09-12T19:39:38.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbc9c64a88cf4570-00631f8afa-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:38.532 DEBUG:boto:b'jdpiak-65MULTIPART1000falseMULTIPART2022-09-12T19:39:02.934Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:39:38.533 DEBUG:boto:path=/jdpiak-65/ 2022-09-12T19:39:38.534 DEBUG:boto:auth_path=/jdpiak-65/ 2022-09-12T19:39:38.535 DEBUG:boto:path=/jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:38.536 DEBUG:boto:auth_path=/jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:38.536 DEBUG:boto:Method: GET 2022-09-12T19:39:38.537 DEBUG:boto:Path: /jdpiak-65/?prefix=MULTIPART 2022-09-12T19:39:38.538 DEBUG:boto:Data: 2022-09-12T19:39:38.539 DEBUG:boto:Headers: {} 2022-09-12T19:39:38.540 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:38.541 DEBUG:boto:Port: 8000 2022-09-12T19:39:38.541 DEBUG:boto:Params: {} 2022-09-12T19:39:38.543 DEBUG:boto:Token: None 2022-09-12T19:39:38.544 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:38 GMT /jdpiak-65/ 2022-09-12T19:39:38.546 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ekOov4CPrCJJ0CS1wka389DVix0= 2022-09-12T19:39:38.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ekOov4CPrCJJ0CS1wka389DVix0=', 'Content-Length': '0'} 2022-09-12T19:39:38.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f1896809221f306-00631f8afa-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:38.552 DEBUG:boto:b'jdpiak-65MULTIPART1000falseMULTIPART2022-09-12T19:39:02.934Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:39:38.553 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-65 zones={test-zone3, test-zone2} 2022-09-12T19:39:38.554 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ok 2022-09-12T19:39:38.555 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-68 2022-09-12T19:39:38.555 DEBUG:boto:path=/jdpiak-68/ 2022-09-12T19:39:38.556 DEBUG:boto:auth_path=/jdpiak-68/ 2022-09-12T19:39:38.557 DEBUG:boto:Method: PUT 2022-09-12T19:39:38.557 DEBUG:boto:Path: /jdpiak-68/ 2022-09-12T19:39:38.558 DEBUG:boto:Data: 2022-09-12T19:39:38.558 DEBUG:boto:Headers: {} 2022-09-12T19:39:38.559 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:38.559 DEBUG:boto:Port: 8000 2022-09-12T19:39:38.560 DEBUG:boto:Params: {} 2022-09-12T19:39:38.561 DEBUG:boto:Token: None 2022-09-12T19:39:38.562 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:39:38 GMT /jdpiak-68/ 2022-09-12T19:39:38.563 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OKNCx20tgM+lRGApkN+ydl0ctRE= 2022-09-12T19:39:38.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OKNCx20tgM+lRGApkN+ydl0ctRE=', 'Content-Length': '0'} 2022-09-12T19:39:38.586 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bf9eca35a562e75a-00631f8afa-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:38.587 DEBUG:boto:path=/jdpiak-68/testobj-sse-c 2022-09-12T19:39:38.587 DEBUG:boto:auth_path=/jdpiak-68/testobj-sse-c 2022-09-12T19:39:38.588 DEBUG:boto:Method: PUT 2022-09-12T19:39:38.588 DEBUG:boto:Path: /jdpiak-68/testobj-sse-c 2022-09-12T19:39:38.589 DEBUG:boto:Data: 2022-09-12T19:39:38.589 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-09-12T19:39:38.590 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:38.591 DEBUG:boto:Port: 8000 2022-09-12T19:39:38.591 DEBUG:boto:Params: {} 2022-09-12T19:39:38.592 DEBUG:boto:Token: None 2022-09-12T19:39:38.592 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Mon, 12 Sep 2022 19:39:38 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /jdpiak-68/testobj-sse-c 2022-09-12T19:39:38.593 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EYwQb/Hyv3OkCFoxCh0uIBbXVP8= 2022-09-12T19:39:38.593 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:39:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EYwQb/Hyv3OkCFoxCh0uIBbXVP8='} 2022-09-12T19:39:38.604 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx000008a52ae2f4b0ac002-00631f8afa-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:38.605 DEBUG:boto:path=/jdpiak-68/testobj-sse-kms 2022-09-12T19:39:38.606 DEBUG:boto:auth_path=/jdpiak-68/testobj-sse-kms 2022-09-12T19:39:38.607 DEBUG:boto:Method: PUT 2022-09-12T19:39:38.607 DEBUG:boto:Path: /jdpiak-68/testobj-sse-kms 2022-09-12T19:39:38.608 DEBUG:boto:Data: 2022-09-12T19:39:38.608 DEBUG:boto:Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-09-12T19:39:38.609 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:38.610 DEBUG:boto:Port: 8000 2022-09-12T19:39:38.611 DEBUG:boto:Params: {} 2022-09-12T19:39:38.612 DEBUG:boto:Token: None 2022-09-12T19:39:38.613 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Mon, 12 Sep 2022 19:39:38 GMT x-amz-server-side-encryption:aws:kms x-amz-server-side-encryption-aws-kms-key-id:testkey-1 /jdpiak-68/testobj-sse-kms 2022-09-12T19:39:38.613 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wEhOHwIu9SmOwUJ0cxNkkpBVLtA= 2022-09-12T19:39:38.614 DEBUG:boto:Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:39:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wEhOHwIu9SmOwUJ0cxNkkpBVLtA='} 2022-09-12T19:39:38.622 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6amRwaWFrLTY4L3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx00000e9a0bec9ca1c0485-00631f8afa-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:39:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:38.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:38.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:38.624 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:39:38.682 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:39:38.683 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:39:38.727 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=38ca1f57-1358-496c-a99b-df49ee365a4f 2022-09-12T19:39:38.750 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:39:38.751 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:38.751 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:39:38.751 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:39:38.751 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:38.752 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:38.752 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:39:38.752 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:39:38.753 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:38.753 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:38.753 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663011578.582406_606.1", 2022-09-12T19:39:38.754 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:39:38.582406Z" 2022-09-12T19:39:38.754 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:38.754 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:38.754 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:39:38.755 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:39:38.755 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:38.755 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:39:38.757 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1663011578.582406_606.1', 3: ''} 2022-09-12T19:39:38.757 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T19:39:38.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:38.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:38.760 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:39:38.818 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:39:38.819 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:39:38.850 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:39:38.847+0000 7fbdd6278600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:39:38.850 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:39:38.848+0000 7fbdd6278600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:39:38.871 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:39:38.872 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:39:38.873 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:39:38.873 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:39:38.873 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:39:38.874 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:39:38.874 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:39:38.874 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:38.874 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:39:38.875 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:39:38.875 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:39:38.876 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:39:38.876 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:39:38.877 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:39:38.877 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:39:38.878 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:39:38.878 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:39:38.878 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:39:38.879 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:39:38.879 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:38.880 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:38.880 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:39:38.881 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:39:38.881 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:39:38.882 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:39:38.882 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:39:38.883 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:39:38.883 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:39:38.884 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:39:38.884 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:39:38.885 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:39:38.885 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:38.886 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:38.886 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:39:38.887 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:39:38.887 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:39:38.888 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:39:38.888 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663011578.582406_606.1", 2022-09-12T19:39:38.889 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:39:38.889 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:39:38.890 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:39:38.890 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:39:38.582406Z", 2022-09-12T19:39:38.891 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:39:38.891 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:38.891 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:38.892 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:39:38.892 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:39:38.893 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:39:38.893 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:39:38.894 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:39:38.894 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:39:38.895 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:39:38.895 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:39:38.896 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:39:38.896 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:39:38.897 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:38.897 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:38.898 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:39:38.898 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:38.899 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:39:38.899 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:39:38.899 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:39:38.900 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:38.900 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:39:38Z" 2022-09-12T19:39:38.900 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:39:38.901 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011578.582406_606.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:39:38.582406Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:39:38Z" } 2022-09-12T19:39:38.902 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011578.582406_606.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:39:38.582406Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:39:38.903 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011578.582406_606.1', 3: ''} 2022-09-12T19:39:38.904 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011578.582406_606.1', 3: ''} 2022-09-12T19:39:38.905 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T19:39:38.905 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T19:39:38.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:38.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:38.908 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:39:38.964 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:39:38.965 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:39:39.008 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:39:39.005+0000 7f8d98581600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:39:39.009 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:39:39.007+0000 7f8d98581600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:39:39.025 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:39:39.025 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:39:39.026 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:39:39.026 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:39:39.027 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:39:39.027 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:39:39.028 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:39:39.028 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:39.028 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:39:39.029 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:39.029 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:39:39.030 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:39:39.030 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:39:39.031 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:39:39.031 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:39:39.031 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:39:39.032 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:39:39.032 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:39:39.033 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:39:39.033 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:39.033 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:39.034 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:39.034 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:39:39.035 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:39:39.035 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:39:39.035 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:39:39.036 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:39:39.036 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:39:39.037 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:39:39.037 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:39:39.037 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:39:39.038 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:39.038 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:39.039 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:39.039 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:39:39.040 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:39:39.040 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:39:39.040 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663011578.582406_606.1", 2022-09-12T19:39:39.041 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:39:39.041 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:39:39.042 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:39:39.042 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:39:38.582406Z", 2022-09-12T19:39:39.043 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:39:39.043 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:39.043 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:39.044 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:39.044 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:39:39.045 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:39:39.045 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:39:39.045 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:39:39.046 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:39:39.047 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:39:39.047 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:39:39.047 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:39:39.048 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:39:39.048 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:39.048 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:39.048 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:39:39.049 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:39.049 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:39:39.049 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:39:39.049 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:39:39.050 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:39.050 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:39:39Z" 2022-09-12T19:39:39.050 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:39:39.052 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011578.582406_606.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:39:38.582406Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:39:39Z" } 2022-09-12T19:39:39.052 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011578.582406_606.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:39:38.582406Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:39:39.053 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011578.582406_606.1', 3: ''} 2022-09-12T19:39:39.053 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011578.582406_606.1', 3: ''} 2022-09-12T19:39:39.054 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T19:39:39.054 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T19:39:39.055 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:39.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:39.056 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:39:39.108 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:39:39.109 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:39:39.148 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:39:39.145+0000 7f83437c6600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:39:39.149 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:39:39.146+0000 7f83437c6600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:39:39.164 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:39:39.165 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:39:39.165 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:39:39.170 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:39:39.170 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:39:39.170 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:39:39.171 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:39:39.171 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:39.171 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:39:39.172 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:39:39.172 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:39:39.172 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:39:39.173 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:39:39.173 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:39:39.173 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:39:39.173 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:39:39.174 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:39:39.174 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:39:39.174 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:39:39.174 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:39.175 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:39.175 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:39:39.175 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:39:39.175 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:39:39.176 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:39:39.176 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:39:39.176 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:39:39.176 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:39:39.177 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:39:39.177 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:39:39.177 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:39:39.177 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:39.178 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:39.178 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:39:39.178 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:39:39.178 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:39:39.179 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:39:39.179 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663011578.573938_603.1", 2022-09-12T19:39:39.179 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:39:39.179 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:39:39.180 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:39:39.180 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:39:38.573938Z", 2022-09-12T19:39:39.180 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:39:39.181 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:39.181 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:39.181 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:39:39.181 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:39:39.182 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:39:39.182 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:39:39.182 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:39:39.182 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:39:39.183 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:39:39.183 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:39:39.183 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:39:39.183 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:39:39.184 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:39.184 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:39.184 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:39:39.184 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:39.185 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:39:39.185 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:39:39.185 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:39:39.185 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:39.186 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:39:39Z" 2022-09-12T19:39:39.186 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:39:39.187 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011578.573938_603.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:39:38.573938Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:39:39Z" } 2022-09-12T19:39:39.188 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011578.573938_603.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:39:38.573938Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:39:39.189 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011578.582406_606.1', 3: ''} 2022-09-12T19:39:39.189 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011578.573938_603.1', 3: ''} 2022-09-12T19:39:39.190 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1663011578.582406_606.1 target=1_1663011578.573938_603.1 2022-09-12T19:39:44.191 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:44.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:44.193 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:39:44.250 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:39:44.250 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:39:44.331 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:39:44.329+0000 7f6612a51600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:39:44.332 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:39:44.330+0000 7f6612a51600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:39:44.345 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:39:44.346 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:39:44.346 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:39:44.346 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:39:44.347 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:39:44.347 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:39:44.347 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:39:44.348 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:44.348 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:39:44.348 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:39:44.348 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:39:44.349 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:39:44.349 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:39:44.349 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:39:44.349 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:39:44.350 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:39:44.350 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:39:44.350 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:39:44.351 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:39:44.351 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:44.351 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:44.351 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:39:44.352 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:39:44.352 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:39:44.352 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:39:44.352 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:39:44.353 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:39:44.353 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:39:44.353 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:39:44.353 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:39:44.354 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:39:44.354 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:44.354 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:44.355 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:39:44.355 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:39:44.355 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:39:44.355 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:39:44.356 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663011578.582406_606.1", 2022-09-12T19:39:44.356 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:39:44.356 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:39:44.356 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:39:44.357 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:39:38.582406Z", 2022-09-12T19:39:44.357 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:39:44.357 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:44.358 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:44.358 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:39:44.358 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:39:44.358 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:39:44.359 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:39:44.359 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:39:44.359 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:39:44.359 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:39:44.360 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:39:44.360 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:39:44.360 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:39:44.360 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:44.361 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:44.361 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:39:44.361 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:44.361 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:39:44.362 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:39:44.362 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:39:44.362 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:44.363 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:39:44Z" 2022-09-12T19:39:44.363 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:39:44.364 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011578.582406_606.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:39:38.582406Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:39:44Z" } 2022-09-12T19:39:44.365 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011578.582406_606.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:39:38.582406Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:39:44.365 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011578.582406_606.1', 3: ''} 2022-09-12T19:39:44.366 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011578.582406_606.1', 3: ''} 2022-09-12T19:39:44.366 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T19:39:44.367 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:39:44.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:39:44.368 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-68 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:39:44.429 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:39:44.430 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:39:44.464 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:39:44.462+0000 7f090552f600 1 waiting to reach incremental sync.. 2022-09-12T19:39:49.465 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:39:49.462+0000 7f090552f600 1 waiting to reach incremental sync.. 2022-09-12T19:39:54.466 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:39:54.464+0000 7f090552f600 1 bucket sync caught up with source: 2022-09-12T19:39:54.467 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.30.5, , , , 00000000001.58.5, , , , , ] 2022-09-12T19:39:54.467 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.30.5, , , , 00000000001.58.5, , , , , ] 2022-09-12T19:39:54.467 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:39:54.464+0000 7f090552f600 0 bucket checkpoint complete 2022-09-12T19:39:54.482 DEBUG:boto:path=/jdpiak-68/ 2022-09-12T19:39:54.483 DEBUG:boto:auth_path=/jdpiak-68/ 2022-09-12T19:39:54.484 DEBUG:boto:Method: HEAD 2022-09-12T19:39:54.485 DEBUG:boto:Path: /jdpiak-68/ 2022-09-12T19:39:54.487 DEBUG:boto:Data: 2022-09-12T19:39:54.488 DEBUG:boto:Headers: {} 2022-09-12T19:39:54.489 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:54.490 DEBUG:boto:Port: 8000 2022-09-12T19:39:54.491 DEBUG:boto:Params: {} 2022-09-12T19:39:54.493 DEBUG:boto:Token: None 2022-09-12T19:39:54.494 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:39:54 GMT /jdpiak-68/ 2022-09-12T19:39:54.495 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:biYtHJZMem/Y+E1XG2JDtfUj3+A= 2022-09-12T19:39:54.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:biYtHJZMem/Y+E1XG2JDtfUj3+A=', 'Content-Length': '0'} 2022-09-12T19:39:54.500 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '1024'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000078d3edfde00d899a-00631f8b0a-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:54.501 DEBUG:boto:path=/jdpiak-68/testobj-sse-c 2022-09-12T19:39:54.502 DEBUG:boto:auth_path=/jdpiak-68/testobj-sse-c 2022-09-12T19:39:54.503 DEBUG:boto:Method: HEAD 2022-09-12T19:39:54.504 DEBUG:boto:Path: /jdpiak-68/testobj-sse-c 2022-09-12T19:39:54.505 DEBUG:boto:Data: 2022-09-12T19:39:54.506 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2022-09-12T19:39:54.506 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:54.507 DEBUG:boto:Port: 8000 2022-09-12T19:39:54.508 DEBUG:boto:Params: {} 2022-09-12T19:39:54.509 DEBUG:boto:Token: None 2022-09-12T19:39:54.509 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:39:54 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /jdpiak-68/testobj-sse-c 2022-09-12T19:39:54.510 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P7DK5iTyMie8HKXzBdxK4R4joQY= 2022-09-12T19:39:54.511 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P7DK5iTyMie8HKXzBdxK4R4joQY=', 'Content-Length': '0'} 2022-09-12T19:39:54.513 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx0000052996911e11df086-00631f8b0a-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:54.514 DEBUG:boto:path=/jdpiak-68/testobj-sse-c 2022-09-12T19:39:54.515 DEBUG:boto:auth_path=/jdpiak-68/testobj-sse-c 2022-09-12T19:39:54.515 DEBUG:boto:Method: GET 2022-09-12T19:39:54.516 DEBUG:boto:Path: /jdpiak-68/testobj-sse-c 2022-09-12T19:39:54.517 DEBUG:boto:Data: 2022-09-12T19:39:54.517 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2022-09-12T19:39:54.518 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:54.518 DEBUG:boto:Port: 8000 2022-09-12T19:39:54.519 DEBUG:boto:Params: {} 2022-09-12T19:39:54.519 DEBUG:boto:Token: None 2022-09-12T19:39:54.520 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:54 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /jdpiak-68/testobj-sse-c 2022-09-12T19:39:54.520 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LrbEk3xPrJU3xUH9HMRtDADFPTM= 2022-09-12T19:39:54.522 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LrbEk3xPrJU3xUH9HMRtDADFPTM=', 'Content-Length': '0'} 2022-09-12T19:39:54.524 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000004d7acb5b3b8faf5f-00631f8b0a-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:54.525 DEBUG:boto:path=/jdpiak-68/testobj-sse-kms 2022-09-12T19:39:54.526 DEBUG:boto:auth_path=/jdpiak-68/testobj-sse-kms 2022-09-12T19:39:54.526 DEBUG:boto:Method: HEAD 2022-09-12T19:39:54.527 DEBUG:boto:Path: /jdpiak-68/testobj-sse-kms 2022-09-12T19:39:54.527 DEBUG:boto:Data: 2022-09-12T19:39:54.528 DEBUG:boto:Headers: {} 2022-09-12T19:39:54.528 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:54.529 DEBUG:boto:Port: 8000 2022-09-12T19:39:54.529 DEBUG:boto:Params: {} 2022-09-12T19:39:54.530 DEBUG:boto:Token: None 2022-09-12T19:39:54.530 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:39:54 GMT /jdpiak-68/testobj-sse-kms 2022-09-12T19:39:54.531 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UCElqA0tcVS8AQ+yAramTgiil5U= 2022-09-12T19:39:54.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UCElqA0tcVS8AQ+yAramTgiil5U=', 'Content-Length': '0'} 2022-09-12T19:39:54.533 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx0000037cc66c4b9ad407a-00631f8b0a-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:54.534 DEBUG:boto:path=/jdpiak-68/testobj-sse-kms 2022-09-12T19:39:54.535 DEBUG:boto:auth_path=/jdpiak-68/testobj-sse-kms 2022-09-12T19:39:54.535 DEBUG:boto:Method: GET 2022-09-12T19:39:54.536 DEBUG:boto:Path: /jdpiak-68/testobj-sse-kms 2022-09-12T19:39:54.536 DEBUG:boto:Data: 2022-09-12T19:39:54.537 DEBUG:boto:Headers: {} 2022-09-12T19:39:54.537 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:39:54.538 DEBUG:boto:Port: 8000 2022-09-12T19:39:54.538 DEBUG:boto:Params: {} 2022-09-12T19:39:54.539 DEBUG:boto:Token: None 2022-09-12T19:39:54.540 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:39:54 GMT /jdpiak-68/testobj-sse-kms 2022-09-12T19:39:54.540 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wsNbNxM2xxyGg7nNhbDwOTR6khI= 2022-09-12T19:39:54.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wsNbNxM2xxyGg7nNhbDwOTR6khI=', 'Content-Length': '0'} 2022-09-12T19:39:54.543 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000ffdb416e27e04e05-00631f8b0a-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:54.544 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ok 2022-09-12T19:39:54.545 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-69 2022-09-12T19:39:54.545 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:39:54.546 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:39:54.546 DEBUG:boto:Method: PUT 2022-09-12T19:39:54.547 DEBUG:boto:Path: /jdpiak-69/ 2022-09-12T19:39:54.547 DEBUG:boto:Data: 2022-09-12T19:39:54.548 DEBUG:boto:Headers: {} 2022-09-12T19:39:54.548 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:54.549 DEBUG:boto:Port: 8000 2022-09-12T19:39:54.549 DEBUG:boto:Params: {} 2022-09-12T19:39:54.550 DEBUG:boto:Token: None 2022-09-12T19:39:54.550 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:39:54 GMT /jdpiak-69/ 2022-09-12T19:39:54.551 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hz4n9PsI+Mghh/SRpvAPBZzbxuE= 2022-09-12T19:39:54.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hz4n9PsI+Mghh/SRpvAPBZzbxuE=', 'Content-Length': '0'} 2022-09-12T19:39:54.581 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008ee1b96b4f23f9ad-00631f8b0a-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:54.582 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:39:54.583 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:39:54.584 DEBUG:boto:Method: HEAD 2022-09-12T19:39:54.585 DEBUG:boto:Path: /jdpiak-69/ 2022-09-12T19:39:54.586 DEBUG:boto:Data: 2022-09-12T19:39:54.587 DEBUG:boto:Headers: {} 2022-09-12T19:39:54.588 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:54.589 DEBUG:boto:Port: 8000 2022-09-12T19:39:54.591 DEBUG:boto:Params: {} 2022-09-12T19:39:54.592 DEBUG:boto:Token: None 2022-09-12T19:39:54.593 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:39:54 GMT /jdpiak-69/ 2022-09-12T19:39:54.594 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ik88Ff0bh0gSN/JKzfT3aAH58gM= 2022-09-12T19:39:54.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ik88Ff0bh0gSN/JKzfT3aAH58gM=', 'Content-Length': '0'} 2022-09-12T19:39:54.599 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000106fea730552f0c0-00631f8b0a-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:54.600 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:39:54.601 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:39:54.602 DEBUG:boto:Method: PUT 2022-09-12T19:39:54.602 DEBUG:boto:Path: /jdpiak-69/a 2022-09-12T19:39:54.603 DEBUG:boto:Data: 2022-09-12T19:39:54.603 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:39:54.604 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:54.604 DEBUG:boto:Port: 8000 2022-09-12T19:39:54.605 DEBUG:boto:Params: {} 2022-09-12T19:39:54.606 DEBUG:boto:Token: None 2022-09-12T19:39:54.607 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:39:54 GMT /jdpiak-69/a 2022-09-12T19:39:54.607 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q0kvo+k8JZMNIEpylEBS7Q6o8jk= 2022-09-12T19:39:54.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:39:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q0kvo+k8JZMNIEpylEBS7Q6o8jk='} 2022-09-12T19:39:54.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e6c9056593f63f4-00631f8b0a-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:54.651 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:39:54.652 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:39:54.652 DEBUG:boto:Method: HEAD 2022-09-12T19:39:54.653 DEBUG:boto:Path: /jdpiak-69/ 2022-09-12T19:39:54.653 DEBUG:boto:Data: 2022-09-12T19:39:54.654 DEBUG:boto:Headers: {} 2022-09-12T19:39:54.654 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:54.655 DEBUG:boto:Port: 8000 2022-09-12T19:39:54.656 DEBUG:boto:Params: {} 2022-09-12T19:39:54.656 DEBUG:boto:Token: None 2022-09-12T19:39:54.662 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:39:54 GMT /jdpiak-69/ 2022-09-12T19:39:54.663 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ik88Ff0bh0gSN/JKzfT3aAH58gM= 2022-09-12T19:39:54.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ik88Ff0bh0gSN/JKzfT3aAH58gM=', 'Content-Length': '0'} 2022-09-12T19:39:54.668 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c90fb5a9289af43-00631f8b0a-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:54.669 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:39:54.670 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:39:54.671 DEBUG:boto:Method: PUT 2022-09-12T19:39:54.672 DEBUG:boto:Path: /jdpiak-69/b 2022-09-12T19:39:54.674 DEBUG:boto:Data: 2022-09-12T19:39:54.675 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:39:54.675 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:54.676 DEBUG:boto:Port: 8000 2022-09-12T19:39:54.676 DEBUG:boto:Params: {} 2022-09-12T19:39:54.677 DEBUG:boto:Token: None 2022-09-12T19:39:54.677 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:39:54 GMT /jdpiak-69/b 2022-09-12T19:39:54.678 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dIVdzZwnZz4ec4MJVNrTeO7WbbU= 2022-09-12T19:39:54.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:39:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dIVdzZwnZz4ec4MJVNrTeO7WbbU='} 2022-09-12T19:39:54.687 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001fb2c09222370403-00631f8b0a-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:54.688 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:39:54.689 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:39:54.690 DEBUG:boto:Method: HEAD 2022-09-12T19:39:54.691 DEBUG:boto:Path: /jdpiak-69/ 2022-09-12T19:39:54.692 DEBUG:boto:Data: 2022-09-12T19:39:54.693 DEBUG:boto:Headers: {} 2022-09-12T19:39:54.694 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:54.695 DEBUG:boto:Port: 8000 2022-09-12T19:39:54.696 DEBUG:boto:Params: {} 2022-09-12T19:39:54.697 DEBUG:boto:Token: None 2022-09-12T19:39:54.698 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:39:54 GMT /jdpiak-69/ 2022-09-12T19:39:54.699 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ik88Ff0bh0gSN/JKzfT3aAH58gM= 2022-09-12T19:39:54.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ik88Ff0bh0gSN/JKzfT3aAH58gM=', 'Content-Length': '0'} 2022-09-12T19:39:54.703 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000efcdb45114bf5a86-00631f8b0a-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:54.705 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:39:54.706 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:39:54.707 DEBUG:boto:Method: PUT 2022-09-12T19:39:54.708 DEBUG:boto:Path: /jdpiak-69/c 2022-09-12T19:39:54.709 DEBUG:boto:Data: 2022-09-12T19:39:54.710 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:39:54.710 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:54.711 DEBUG:boto:Port: 8000 2022-09-12T19:39:54.712 DEBUG:boto:Params: {} 2022-09-12T19:39:54.713 DEBUG:boto:Token: None 2022-09-12T19:39:54.714 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:39:54 GMT /jdpiak-69/c 2022-09-12T19:39:54.716 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DCvAgYOtSLJmLKRz1UH2hvhhHz0= 2022-09-12T19:39:54.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:39:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DCvAgYOtSLJmLKRz1UH2hvhhHz0='} 2022-09-12T19:39:54.762 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005418b0b3f1908ce3-00631f8b0a-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:54.764 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:39:54.765 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:39:54.766 DEBUG:boto:Method: HEAD 2022-09-12T19:39:54.768 DEBUG:boto:Path: /jdpiak-69/ 2022-09-12T19:39:54.769 DEBUG:boto:Data: 2022-09-12T19:39:54.770 DEBUG:boto:Headers: {} 2022-09-12T19:39:54.771 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:54.772 DEBUG:boto:Port: 8000 2022-09-12T19:39:54.774 DEBUG:boto:Params: {} 2022-09-12T19:39:54.775 DEBUG:boto:Token: None 2022-09-12T19:39:54.777 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:39:54 GMT /jdpiak-69/ 2022-09-12T19:39:54.779 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ik88Ff0bh0gSN/JKzfT3aAH58gM= 2022-09-12T19:39:54.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:39:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ik88Ff0bh0gSN/JKzfT3aAH58gM=', 'Content-Length': '0'} 2022-09-12T19:39:54.784 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa6d2acdc0799c8e-00631f8b0a-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:54.785 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:39:54.786 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:39:54.787 DEBUG:boto:Method: PUT 2022-09-12T19:39:54.788 DEBUG:boto:Path: /jdpiak-69/d 2022-09-12T19:39:54.789 DEBUG:boto:Data: 2022-09-12T19:39:54.790 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:39:54.791 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:39:54.792 DEBUG:boto:Port: 8000 2022-09-12T19:39:54.793 DEBUG:boto:Params: {} 2022-09-12T19:39:54.794 DEBUG:boto:Token: None 2022-09-12T19:39:54.795 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:39:54 GMT /jdpiak-69/d 2022-09-12T19:39:54.798 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vUH+XnbAPL0BJwBhlsRRJhhm5Ew= 2022-09-12T19:39:54.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:39:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vUH+XnbAPL0BJwBhlsRRJhhm5Ew='} 2022-09-12T19:39:54.810 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d8578da156cc0395-00631f8b0a-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:39:54.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:54.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:54.812 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:39:54.872 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:39:54.873 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:39:54.922 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=38ca1f57-1358-496c-a99b-df49ee365a4f 2022-09-12T19:39:54.947 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:39:54.948 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:54.949 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:39:54.949 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:39:54.949 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:54.950 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:54.950 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:39:54.951 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:39:54.952 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:54.952 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:54.953 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663011594.575283_610.1", 2022-09-12T19:39:54.953 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:39:54.575283Z" 2022-09-12T19:39:54.953 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:54.953 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:54.954 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:39:54.954 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:39:54.954 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:54.955 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:39:54.955 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1663011594.575283_610.1', 3: ''} 2022-09-12T19:39:54.956 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T19:39:54.957 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:54.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:54.960 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:39:55.011 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:39:55.012 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:39:55.053 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:39:55.050+0000 7f35edc86600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:39:55.054 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:39:55.051+0000 7f35edc86600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:39:55.072 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:39:55.073 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:39:55.073 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:39:55.073 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:39:55.073 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:39:55.075 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:39:55.075 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:39:55.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:55.077 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:39:55.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:39:55.077 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:39:55.078 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:39:55.078 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:39:55.078 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:39:55.079 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:39:55.079 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:39:55.079 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:39:55.080 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:39:55.080 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:39:55.080 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:55.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:55.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:39:55.081 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:39:55.081 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:39:55.081 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:39:55.086 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:39:55.087 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:39:55.087 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:39:55.088 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:39:55.089 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:39:55.090 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:39:55.091 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:55.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:55.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:39:55.092 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:39:55.093 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:39:55.093 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:39:55.097 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663011594.575283_610.1", 2022-09-12T19:39:55.097 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:39:55.097 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:39:55.098 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:39:55.098 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:39:54.575283Z", 2022-09-12T19:39:55.098 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:39:55.099 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:55.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:55.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:39:55.099 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:39:55.100 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:39:55.100 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:39:55.101 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:39:55.101 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:39:55.102 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:39:55.102 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:39:55.102 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:39:55.103 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:39:55.103 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:55.103 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:39:55.104 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:39:55.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:55.105 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:39:55.105 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:39:55.105 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:39:55.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:39:55.106 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:39:55Z" 2022-09-12T19:39:55.106 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:39:55.107 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011594.575283_610.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:39:54.575283Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:39:55Z" } 2022-09-12T19:39:55.108 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011594.575283_610.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:39:54.575283Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:39:55.109 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011594.575283_610.1', 3: ''} 2022-09-12T19:39:55.109 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011594.575283_610.1', 3: ''} 2022-09-12T19:39:55.110 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T19:39:55.110 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T19:39:55.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:55.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:39:55.113 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:39:55.176 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:39:55.177 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:39:55.216 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:39:55.213+0000 7fe84bd1a600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:39:55.222 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:39:55.214+0000 7fe84bd1a600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:39:55.232 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:39:55.233 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:39:55.233 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:39:55.233 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:39:55.233 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:39:55.234 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:39:55.234 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:39:55.235 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:55.235 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:39:55.236 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:55.236 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:39:55.237 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:39:55.237 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:39:55.237 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:39:55.238 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:39:55.238 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:39:55.238 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:39:55.238 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:39:55.239 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:39:55.239 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:55.239 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:55.239 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:55.240 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:39:55.240 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:39:55.241 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:39:55.241 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:39:55.241 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:39:55.242 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:39:55.242 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:39:55.242 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:39:55.242 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:39:55.243 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:55.243 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:55.243 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:55.244 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:39:55.244 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:39:55.244 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:39:55.244 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663011578.582406_606.1", 2022-09-12T19:39:55.245 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:39:55.245 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:39:55.245 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:39:55.245 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:39:38.582406Z", 2022-09-12T19:39:55.246 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:39:55.246 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:55.246 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:55.246 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:39:55.247 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:39:55.247 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:39:55.247 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:39:55.248 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:39:55.248 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:39:55.248 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:39:55.249 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:39:55.249 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:39:55.249 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:39:55.249 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:55.250 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:39:55.250 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:39:55.250 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:55.250 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:39:55.251 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:39:55.251 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:39:55.251 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:39:55.251 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:39:55Z" 2022-09-12T19:39:55.252 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:39:55.253 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011578.582406_606.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:39:38.582406Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:39:55Z" } 2022-09-12T19:39:55.254 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011578.582406_606.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:39:38.582406Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:39:55.254 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011594.575283_610.1', 3: ''} 2022-09-12T19:39:55.255 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011578.582406_606.1', 3: ''} 2022-09-12T19:39:55.255 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1663011594.575283_610.1 target=1_1663011578.582406_606.1 2022-09-12T19:40:00.257 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:40:00.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:40:00.260 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:40:00.327 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:40:00.328 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:40:00.353 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:40:00.350+0000 7fb70eadb600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:40:00.354 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:40:00.352+0000 7fb70eadb600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:40:00.368 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:40:00.368 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:40:00.369 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:40:00.369 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:40:00.369 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:40:00.369 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:40:00.370 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:40:00.370 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:40:00.370 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:40:00.370 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:40:00.371 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:40:00.371 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:40:00.371 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:40:00.372 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:40:00.372 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:40:00.372 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:40:00.372 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:40:00.373 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:40:00.373 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:40:00.373 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:40:00.373 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:40:00.373 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:40:00.374 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:40:00.374 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:40:00.374 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:40:00.375 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:40:00.375 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:40:00.375 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:40:00.375 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:40:00.376 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:40:00.376 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:40:00.376 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:40:00.376 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:40:00.377 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:40:00.377 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:40:00.377 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:40:00.377 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:40:00.378 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663011594.575283_610.1", 2022-09-12T19:40:00.378 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:40:00.378 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:40:00.379 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:40:00.379 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:39:54.575283Z", 2022-09-12T19:40:00.380 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:40:00.380 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:40:00.381 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:40:00.381 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:40:00.382 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:40:00.382 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:40:00.383 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:40:00.383 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:40:00.384 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:40:00.384 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:40:00.385 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:40:00.385 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:40:00.386 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:40:00.386 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:40:00.388 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:40:00.389 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:40:00.389 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:40:00.389 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:40:00.390 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:40:00.390 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:40:00.391 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:40:00.391 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:40:00Z" 2022-09-12T19:40:00.392 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:40:00.394 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011594.575283_610.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:39:54.575283Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:40:00Z" } 2022-09-12T19:40:00.398 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011594.575283_610.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:39:54.575283Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:40:00.402 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011594.575283_610.1', 3: ''} 2022-09-12T19:40:00.403 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011594.575283_610.1', 3: ''} 2022-09-12T19:40:00.405 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T19:40:00.406 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T19:40:00.407 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:40:00.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:40:00.409 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:40:00.471 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:40:00.471 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:40:00.503 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:40:00.500+0000 7faf7325c600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:40:00.503 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:40:00.501+0000 7faf7325c600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:40:00.520 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:40:00.521 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:40:00.522 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:40:00.522 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:40:00.523 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:40:00.523 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:40:00.524 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:40:00.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:40:00.525 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:40:00.525 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:40:00.526 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:40:00.526 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:40:00.526 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:40:00.527 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:40:00.527 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:40:00.528 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:40:00.528 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:40:00.529 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:40:00.529 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:40:00.530 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:40:00.530 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:40:00.531 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:40:00.531 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:40:00.532 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:40:00.532 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:40:00.533 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:40:00.533 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:40:00.533 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:40:00.534 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:40:00.541 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:40:00.542 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:40:00.543 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:40:00.543 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:40:00.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:40:00.544 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:40:00.544 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:40:00.548 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:40:00.548 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663011594.575283_610.1", 2022-09-12T19:40:00.548 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:40:00.548 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:40:00.549 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:40:00.549 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:39:54.575283Z", 2022-09-12T19:40:00.550 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:40:00.553 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:40:00.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:40:00.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:40:00.554 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:40:00.555 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:40:00.555 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:40:00.556 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:40:00.556 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:40:00.557 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:40:00.557 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:40:00.557 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:40:00.557 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:40:00.558 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:40:00.558 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:40:00.558 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:40:00.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:40:00.559 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:40:00.559 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:40:00.559 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:40:00.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:40:00.560 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:40:00Z" 2022-09-12T19:40:00.560 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:40:00.562 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011594.575283_610.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:39:54.575283Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:40:00Z" } 2022-09-12T19:40:00.562 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011594.575283_610.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:39:54.575283Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:40:00.563 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011594.575283_610.1', 3: ''} 2022-09-12T19:40:00.564 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011594.575283_610.1', 3: ''} 2022-09-12T19:40:00.565 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T19:40:00.565 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-69 2022-09-12T19:40:00.566 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:40:00.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:40:00.567 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:40:00.625 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:40:00.626 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:40:00.662 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:40:00.660+0000 7f37e2665600 1 waiting to reach incremental sync.. 2022-09-12T19:40:05.662 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:40:05.661+0000 7f37e2665600 1 waiting to reach incremental sync.. 2022-09-12T19:40:10.667 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:40:10.661+0000 7f37e2665600 1 waiting to reach incremental sync.. 2022-09-12T19:40:15.665 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:40:15.663+0000 7f37e2665600 1 bucket sync caught up with source: 2022-09-12T19:40:15.666 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.73.5, , , , 00000000001.52.5, , , 00000000001.56.5, , 00000000001.42.5] 2022-09-12T19:40:15.666 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.73.5, , , , 00000000001.52.5, , , 00000000001.56.5, , 00000000001.42.5] 2022-09-12T19:40:15.667 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:40:15.663+0000 7f37e2665600 0 bucket checkpoint complete 2022-09-12T19:40:15.684 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-69 2022-09-12T19:40:15.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:40:15.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:40:15.687 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:40:15.754 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:40:15.754 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:40:15.801 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:40:15.800+0000 7fb62fb79600 1 bucket sync caught up with source: 2022-09-12T19:40:15.802 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.73.5, , , , 00000000001.52.5, , , 00000000001.56.5, , 00000000001.42.5] 2022-09-12T19:40:15.803 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000001.73.5, , , , 00000000001.52.5, , , 00000000001.56.5, , 00000000001.42.5] 2022-09-12T19:40:15.803 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:40:15.800+0000 7fb62fb79600 0 bucket checkpoint complete 2022-09-12T19:40:15.821 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-69 2022-09-12T19:40:15.822 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:40:15.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:40:15.824 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:40:15.889 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:40:15.890 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:40:15.926 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:40:15.923+0000 7fa15d3c7600 1 bucket sync caught up with source: 2022-09-12T19:40:15.927 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.73.5, , , , 00000000001.52.5, , , 00000000001.56.5, , 00000000001.42.5] 2022-09-12T19:40:15.927 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.73.5, , , , 00000000001.52.5, , , 00000000001.56.5, , 00000000001.42.5] 2022-09-12T19:40:15.927 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:40:15.923+0000 7fa15d3c7600 0 bucket checkpoint complete 2022-09-12T19:40:15.945 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-69 2022-09-12T19:40:15.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:40:15.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:40:15.947 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-69 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:40:16.007 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:40:16.008 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:40:16.064 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:40:16.062+0000 7f2eafce4600 1 waiting to reach incremental sync.. 2022-09-12T19:40:21.065 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:40:21.063+0000 7f2eafce4600 1 waiting to reach incremental sync.. 2022-09-12T19:40:26.065 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:40:26.063+0000 7f2eafce4600 1 waiting to reach incremental sync.. 2022-09-12T19:40:31.065 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:40:31.063+0000 7f2eafce4600 1 waiting to reach incremental sync.. 2022-09-12T19:40:36.067 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:40:36.065+0000 7f2eafce4600 1 bucket sync caught up with source: 2022-09-12T19:40:36.068 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.350.5, , , , 00000000001.989.5, , , 00000000001.302.5, , 00000000001.830.5] 2022-09-12T19:40:36.069 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000001.350.5, , , , 00000000001.989.5, , , 00000000001.302.5, , 00000000001.830.5] 2022-09-12T19:40:36.069 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:40:36.065+0000 7f2eafce4600 0 bucket checkpoint complete 2022-09-12T19:40:36.090 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-69 2022-09-12T19:40:36.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:40:36.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:40:36.093 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-69 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:40:36.156 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:40:36.157 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:40:36.201 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:40:36.198+0000 7ff4a39fc600 1 bucket sync caught up with source: 2022-09-12T19:40:36.201 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.350.5, , , , 00000000001.989.5, , , 00000000001.302.5, , 00000000001.830.5] 2022-09-12T19:40:36.201 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000001.350.5, , , , 00000000001.989.5, , , 00000000001.302.5, , 00000000001.830.5] 2022-09-12T19:40:36.202 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:40:36.198+0000 7ff4a39fc600 0 bucket checkpoint complete 2022-09-12T19:40:36.217 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-69 2022-09-12T19:40:36.218 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-69 2022-09-12T19:40:36.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:40:36.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:40:36.221 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-69 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:40:36.281 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:40:36.282 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:40:36.333 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:40:36.330+0000 7f166c8a3600 1 bucket sync caught up with source: 2022-09-12T19:40:36.334 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.377.5, , , , 00000000001.520.5, , , 00000000001.226.5, , 00000000001.587.5] 2022-09-12T19:40:36.334 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000001.377.5, , , , 00000000001.520.5, , , 00000000001.226.5, , 00000000001.587.5] 2022-09-12T19:40:36.335 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:40:36.330+0000 7f166c8a3600 0 bucket checkpoint complete 2022-09-12T19:40:36.352 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-69 2022-09-12T19:40:36.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:40:36.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:40:36.355 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-69 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:40:36.420 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:40:36.420 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:40:36.459 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:40:36.456+0000 7fe25b7d9600 1 bucket sync caught up with source: 2022-09-12T19:40:36.459 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.377.5, , , , 00000000001.520.5, , , 00000000001.226.5, , 00000000001.587.5] 2022-09-12T19:40:36.459 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.377.5, , , , 00000000001.520.5, , , 00000000001.226.5, , 00000000001.587.5] 2022-09-12T19:40:36.460 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:40:36.456+0000 7fe25b7d9600 0 bucket checkpoint complete 2022-09-12T19:40:36.474 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-69 2022-09-12T19:40:36.475 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-69 zones={test-zone2, test-zone1} 2022-09-12T19:40:36.476 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:36.476 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:36.477 DEBUG:boto:Method: HEAD 2022-09-12T19:40:36.477 DEBUG:boto:Path: /jdpiak-69/ 2022-09-12T19:40:36.478 DEBUG:boto:Data: 2022-09-12T19:40:36.478 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.479 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:36.479 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.480 DEBUG:boto:Params: {} 2022-09-12T19:40:36.481 DEBUG:boto:Token: None 2022-09-12T19:40:36.481 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/ 2022-09-12T19:40:36.482 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ci8oHNgfHD/DrQR/04P1a9Oz/ac= 2022-09-12T19:40:36.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ci8oHNgfHD/DrQR/04P1a9Oz/ac=', 'Content-Length': '0'} 2022-09-12T19:40:36.485 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb3ee2edce50300d-00631f8b34-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.486 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:36.487 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:36.487 DEBUG:boto:Method: HEAD 2022-09-12T19:40:36.488 DEBUG:boto:Path: /jdpiak-69/ 2022-09-12T19:40:36.488 DEBUG:boto:Data: 2022-09-12T19:40:36.489 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.489 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:36.490 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.490 DEBUG:boto:Params: {} 2022-09-12T19:40:36.491 DEBUG:boto:Token: None 2022-09-12T19:40:36.491 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/ 2022-09-12T19:40:36.492 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ci8oHNgfHD/DrQR/04P1a9Oz/ac= 2022-09-12T19:40:36.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ci8oHNgfHD/DrQR/04P1a9Oz/ac=', 'Content-Length': '0'} 2022-09-12T19:40:36.497 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8782d8ab20dc387-00631f8b34-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.498 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:40:36.499 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:36.499 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:36.500 DEBUG:boto:path=/jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:36.500 DEBUG:boto:auth_path=/jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:36.501 DEBUG:boto:Method: GET 2022-09-12T19:40:36.501 DEBUG:boto:Path: /jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:36.502 DEBUG:boto:Data: 2022-09-12T19:40:36.503 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.504 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:36.504 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.505 DEBUG:boto:Params: {} 2022-09-12T19:40:36.507 DEBUG:boto:Token: None 2022-09-12T19:40:36.508 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/?versions 2022-09-12T19:40:36.509 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5/SPtSLm8eudjpKaC99zYvGchM8= 2022-09-12T19:40:36.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5/SPtSLm8eudjpKaC99zYvGchM8=', 'Content-Length': '0'} 2022-09-12T19:40:36.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a1b67f2b21ed8ad-00631f8b34-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.514 DEBUG:boto:b'jdpiak-69999falseanulltrue2022-09-12T19:39:54.609Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:39:54.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:39:54.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:39:54.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:36.516 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:40:36.516 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:40:36.517 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:40:36.518 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:40:36.520 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:40:36.521 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:36.521 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:36.522 DEBUG:boto:path=/jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:36.523 DEBUG:boto:auth_path=/jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:36.523 DEBUG:boto:Method: GET 2022-09-12T19:40:36.524 DEBUG:boto:Path: /jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:36.524 DEBUG:boto:Data: 2022-09-12T19:40:36.525 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.525 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:36.526 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.527 DEBUG:boto:Params: {} 2022-09-12T19:40:36.528 DEBUG:boto:Token: None 2022-09-12T19:40:36.529 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/?versions 2022-09-12T19:40:36.530 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5/SPtSLm8eudjpKaC99zYvGchM8= 2022-09-12T19:40:36.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5/SPtSLm8eudjpKaC99zYvGchM8=', 'Content-Length': '0'} 2022-09-12T19:40:36.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c72d9e636b40100e-00631f8b34-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.536 DEBUG:boto:b'jdpiak-69999falseanulltrue2022-09-12T19:39:54.609Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:39:54.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:39:54.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:39:54.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:36.537 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:40:36.538 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:40:36.539 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:40:36.539 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:40:36.540 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:36.541 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:36.542 DEBUG:boto:path=/jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:36.543 DEBUG:boto:auth_path=/jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:36.545 DEBUG:boto:Method: GET 2022-09-12T19:40:36.546 DEBUG:boto:Path: /jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:36.547 DEBUG:boto:Data: 2022-09-12T19:40:36.547 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.549 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:36.549 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.550 DEBUG:boto:Params: {} 2022-09-12T19:40:36.552 DEBUG:boto:Token: None 2022-09-12T19:40:36.553 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/?versions 2022-09-12T19:40:36.554 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5/SPtSLm8eudjpKaC99zYvGchM8= 2022-09-12T19:40:36.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5/SPtSLm8eudjpKaC99zYvGchM8=', 'Content-Length': '0'} 2022-09-12T19:40:36.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ef6ecaa2e7dc553-00631f8b34-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.560 DEBUG:boto:b'jdpiak-69999falseanulltrue2022-09-12T19:39:54.609Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:39:54.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:39:54.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:39:54.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:36.562 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:36.563 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:36.564 DEBUG:boto:path=/jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:36.567 DEBUG:boto:auth_path=/jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:36.568 DEBUG:boto:Method: GET 2022-09-12T19:40:36.569 DEBUG:boto:Path: /jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:36.570 DEBUG:boto:Data: 2022-09-12T19:40:36.571 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.572 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:36.573 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.574 DEBUG:boto:Params: {} 2022-09-12T19:40:36.575 DEBUG:boto:Token: None 2022-09-12T19:40:36.576 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/?versions 2022-09-12T19:40:36.577 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5/SPtSLm8eudjpKaC99zYvGchM8= 2022-09-12T19:40:36.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5/SPtSLm8eudjpKaC99zYvGchM8=', 'Content-Length': '0'} 2022-09-12T19:40:36.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f17aa921921adfc-00631f8b34-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.583 DEBUG:boto:b'jdpiak-69999falseanulltrue2022-09-12T19:39:54.609Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:39:54.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:39:54.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:39:54.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:36.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:40:36.584 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:36.585 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:36.585 DEBUG:boto:path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:36.586 DEBUG:boto:auth_path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:36.587 DEBUG:boto:Method: GET 2022-09-12T19:40:36.587 DEBUG:boto:Path: /jdpiak-69/a?versionId=null 2022-09-12T19:40:36.588 DEBUG:boto:Data: 2022-09-12T19:40:36.588 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.589 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:36.590 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.590 DEBUG:boto:Params: {} 2022-09-12T19:40:36.591 DEBUG:boto:Token: None 2022-09-12T19:40:36.591 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/a?versionId=null 2022-09-12T19:40:36.592 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5Ry/tUTDnTcPr66ng/Uzd8YBTIs= 2022-09-12T19:40:36.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5Ry/tUTDnTcPr66ng/Uzd8YBTIs=', 'Content-Length': '0'} 2022-09-12T19:40:36.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fd97055f956dd2d-00631f8b34-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.597 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:36.597 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:36.598 DEBUG:boto:path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:36.599 DEBUG:boto:auth_path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:36.599 DEBUG:boto:Method: GET 2022-09-12T19:40:36.600 DEBUG:boto:Path: /jdpiak-69/a?versionId=null 2022-09-12T19:40:36.601 DEBUG:boto:Data: 2022-09-12T19:40:36.602 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.603 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:36.604 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.605 DEBUG:boto:Params: {} 2022-09-12T19:40:36.606 DEBUG:boto:Token: None 2022-09-12T19:40:36.607 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/a?versionId=null 2022-09-12T19:40:36.608 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5Ry/tUTDnTcPr66ng/Uzd8YBTIs= 2022-09-12T19:40:36.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5Ry/tUTDnTcPr66ng/Uzd8YBTIs=', 'Content-Length': '0'} 2022-09-12T19:40:36.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009f7e4c1bbb7704e-00631f8b34-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.614 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:36.615 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:36.616 DEBUG:boto:path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:36.617 DEBUG:boto:auth_path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:36.622 DEBUG:boto:Method: HEAD 2022-09-12T19:40:36.623 DEBUG:boto:Path: /jdpiak-69/a?versionId=null 2022-09-12T19:40:36.623 DEBUG:boto:Data: 2022-09-12T19:40:36.624 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.625 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:36.627 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.628 DEBUG:boto:Params: {} 2022-09-12T19:40:36.630 DEBUG:boto:Token: None 2022-09-12T19:40:36.633 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/a?versionId=null 2022-09-12T19:40:36.635 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:v0zUnayRUQ9xIoVPww8Smm7xqIY= 2022-09-12T19:40:36.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:v0zUnayRUQ9xIoVPww8Smm7xqIY=', 'Content-Length': '0'} 2022-09-12T19:40:36.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e2b2cc2d0652726-00631f8b34-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.642 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:36.643 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:36.644 DEBUG:boto:path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:36.645 DEBUG:boto:auth_path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:36.646 DEBUG:boto:Method: HEAD 2022-09-12T19:40:36.647 DEBUG:boto:Path: /jdpiak-69/a?versionId=null 2022-09-12T19:40:36.648 DEBUG:boto:Data: 2022-09-12T19:40:36.648 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.649 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:36.650 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.650 DEBUG:boto:Params: {} 2022-09-12T19:40:36.651 DEBUG:boto:Token: None 2022-09-12T19:40:36.651 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/a?versionId=null 2022-09-12T19:40:36.652 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:v0zUnayRUQ9xIoVPww8Smm7xqIY= 2022-09-12T19:40:36.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:v0zUnayRUQ9xIoVPww8Smm7xqIY=', 'Content-Length': '0'} 2022-09-12T19:40:36.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d836f3379e5b22cf-00631f8b34-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:40:36.657 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:36.658 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:36.659 DEBUG:boto:path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:36.659 DEBUG:boto:auth_path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:36.660 DEBUG:boto:Method: GET 2022-09-12T19:40:36.660 DEBUG:boto:Path: /jdpiak-69/a?versionId=null 2022-09-12T19:40:36.661 DEBUG:boto:Data: 2022-09-12T19:40:36.661 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.662 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:36.663 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.663 DEBUG:boto:Params: {} 2022-09-12T19:40:36.664 DEBUG:boto:Token: None 2022-09-12T19:40:36.665 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/a?versionId=null 2022-09-12T19:40:36.665 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5Ry/tUTDnTcPr66ng/Uzd8YBTIs= 2022-09-12T19:40:36.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5Ry/tUTDnTcPr66ng/Uzd8YBTIs=', 'Content-Length': '0'} 2022-09-12T19:40:36.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fea16d1935816d31-00631f8b34-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.714 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:36.715 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:36.715 DEBUG:boto:path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:36.716 DEBUG:boto:auth_path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:36.716 DEBUG:boto:Method: GET 2022-09-12T19:40:36.717 DEBUG:boto:Path: /jdpiak-69/a?versionId=null 2022-09-12T19:40:36.718 DEBUG:boto:Data: 2022-09-12T19:40:36.718 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.719 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:36.719 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.720 DEBUG:boto:Params: {} 2022-09-12T19:40:36.723 DEBUG:boto:Token: None 2022-09-12T19:40:36.724 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/a?versionId=null 2022-09-12T19:40:36.725 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5Ry/tUTDnTcPr66ng/Uzd8YBTIs= 2022-09-12T19:40:36.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5Ry/tUTDnTcPr66ng/Uzd8YBTIs=', 'Content-Length': '0'} 2022-09-12T19:40:36.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd270bf148692995-00631f8b34-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.734 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:36.735 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:36.736 DEBUG:boto:path=/jdpiak-69/?prefix=a 2022-09-12T19:40:36.737 DEBUG:boto:auth_path=/jdpiak-69/?prefix=a 2022-09-12T19:40:36.738 DEBUG:boto:Method: GET 2022-09-12T19:40:36.739 DEBUG:boto:Path: /jdpiak-69/?prefix=a 2022-09-12T19:40:36.739 DEBUG:boto:Data: 2022-09-12T19:40:36.740 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.741 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:36.741 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.742 DEBUG:boto:Params: {} 2022-09-12T19:40:36.742 DEBUG:boto:Token: None 2022-09-12T19:40:36.743 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/ 2022-09-12T19:40:36.744 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cJY+C5R0CHejbwf/hYLivY82D8M= 2022-09-12T19:40:36.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cJY+C5R0CHejbwf/hYLivY82D8M=', 'Content-Length': '0'} 2022-09-12T19:40:36.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006db92705f67f1286-00631f8b34-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.748 DEBUG:boto:b'jdpiak-69a1000falsea2022-09-12T19:39:54.609Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:36.749 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:36.750 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:36.750 DEBUG:boto:path=/jdpiak-69/?prefix=a 2022-09-12T19:40:36.751 DEBUG:boto:auth_path=/jdpiak-69/?prefix=a 2022-09-12T19:40:36.751 DEBUG:boto:Method: GET 2022-09-12T19:40:36.752 DEBUG:boto:Path: /jdpiak-69/?prefix=a 2022-09-12T19:40:36.752 DEBUG:boto:Data: 2022-09-12T19:40:36.753 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.753 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:36.754 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.755 DEBUG:boto:Params: {} 2022-09-12T19:40:36.756 DEBUG:boto:Token: None 2022-09-12T19:40:36.757 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/ 2022-09-12T19:40:36.757 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cJY+C5R0CHejbwf/hYLivY82D8M= 2022-09-12T19:40:36.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cJY+C5R0CHejbwf/hYLivY82D8M=', 'Content-Length': '0'} 2022-09-12T19:40:36.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000613a5183f293867b-00631f8b34-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.762 DEBUG:boto:b'jdpiak-69a1000falsea2022-09-12T19:39:54.609Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:36.764 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:36.765 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:36.766 DEBUG:boto:Method: HEAD 2022-09-12T19:40:36.767 DEBUG:boto:Path: /jdpiak-69/a 2022-09-12T19:40:36.768 DEBUG:boto:Data: 2022-09-12T19:40:36.769 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.769 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:36.770 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.770 DEBUG:boto:Params: {} 2022-09-12T19:40:36.771 DEBUG:boto:Token: None 2022-09-12T19:40:36.771 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/a 2022-09-12T19:40:36.772 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8FyK8ZDpeylkF/rZ2jWemyl11Jc= 2022-09-12T19:40:36.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8FyK8ZDpeylkF/rZ2jWemyl11Jc=', 'Content-Length': '0'} 2022-09-12T19:40:36.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f38cd58f7813e95-00631f8b34-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.778 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:36.779 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:36.780 DEBUG:boto:Method: HEAD 2022-09-12T19:40:36.780 DEBUG:boto:Path: /jdpiak-69/a 2022-09-12T19:40:36.781 DEBUG:boto:Data: 2022-09-12T19:40:36.782 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.783 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:36.784 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.785 DEBUG:boto:Params: {} 2022-09-12T19:40:36.786 DEBUG:boto:Token: None 2022-09-12T19:40:36.787 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/a 2022-09-12T19:40:36.788 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8FyK8ZDpeylkF/rZ2jWemyl11Jc= 2022-09-12T19:40:36.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8FyK8ZDpeylkF/rZ2jWemyl11Jc=', 'Content-Length': '0'} 2022-09-12T19:40:36.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003614334c7817b461-00631f8b34-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:40:36.798 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:36.799 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:36.800 DEBUG:boto:Method: GET 2022-09-12T19:40:36.801 DEBUG:boto:Path: /jdpiak-69/a 2022-09-12T19:40:36.802 DEBUG:boto:Data: 2022-09-12T19:40:36.803 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.804 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:36.805 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.806 DEBUG:boto:Params: {} 2022-09-12T19:40:36.807 DEBUG:boto:Token: None 2022-09-12T19:40:36.808 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/a 2022-09-12T19:40:36.808 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L6TI6PKvZsn04OsWPudFcBsQwEI= 2022-09-12T19:40:36.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L6TI6PKvZsn04OsWPudFcBsQwEI=', 'Content-Length': '0'} 2022-09-12T19:40:36.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a03a5b0d2d78aa4c-00631f8b34-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.818 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:36.819 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:36.819 DEBUG:boto:Method: GET 2022-09-12T19:40:36.820 DEBUG:boto:Path: /jdpiak-69/a 2022-09-12T19:40:36.820 DEBUG:boto:Data: 2022-09-12T19:40:36.821 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.822 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:36.823 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.824 DEBUG:boto:Params: {} 2022-09-12T19:40:36.825 DEBUG:boto:Token: None 2022-09-12T19:40:36.826 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/a 2022-09-12T19:40:36.827 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L6TI6PKvZsn04OsWPudFcBsQwEI= 2022-09-12T19:40:36.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L6TI6PKvZsn04OsWPudFcBsQwEI=', 'Content-Length': '0'} 2022-09-12T19:40:36.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062dcb66605249922-00631f8b34-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.832 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:36.833 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:36.834 DEBUG:boto:path=/jdpiak-69/?prefix=a 2022-09-12T19:40:36.835 DEBUG:boto:auth_path=/jdpiak-69/?prefix=a 2022-09-12T19:40:36.836 DEBUG:boto:Method: GET 2022-09-12T19:40:36.836 DEBUG:boto:Path: /jdpiak-69/?prefix=a 2022-09-12T19:40:36.837 DEBUG:boto:Data: 2022-09-12T19:40:36.837 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.838 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:36.838 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.839 DEBUG:boto:Params: {} 2022-09-12T19:40:36.840 DEBUG:boto:Token: None 2022-09-12T19:40:36.840 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/ 2022-09-12T19:40:36.841 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cJY+C5R0CHejbwf/hYLivY82D8M= 2022-09-12T19:40:36.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cJY+C5R0CHejbwf/hYLivY82D8M=', 'Content-Length': '0'} 2022-09-12T19:40:36.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000211133155c1d2848-00631f8b34-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.890 DEBUG:boto:b'jdpiak-69a1000falsea2022-09-12T19:39:54.609Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:36.891 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:36.892 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:36.892 DEBUG:boto:path=/jdpiak-69/?prefix=a 2022-09-12T19:40:36.893 DEBUG:boto:auth_path=/jdpiak-69/?prefix=a 2022-09-12T19:40:36.894 DEBUG:boto:Method: GET 2022-09-12T19:40:36.895 DEBUG:boto:Path: /jdpiak-69/?prefix=a 2022-09-12T19:40:36.896 DEBUG:boto:Data: 2022-09-12T19:40:36.897 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.898 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:36.899 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.899 DEBUG:boto:Params: {} 2022-09-12T19:40:36.900 DEBUG:boto:Token: None 2022-09-12T19:40:36.901 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/ 2022-09-12T19:40:36.901 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cJY+C5R0CHejbwf/hYLivY82D8M= 2022-09-12T19:40:36.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cJY+C5R0CHejbwf/hYLivY82D8M=', 'Content-Length': '0'} 2022-09-12T19:40:36.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094faec8877949d50-00631f8b34-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.906 DEBUG:boto:b'jdpiak-69a1000falsea2022-09-12T19:39:54.609Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:36.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:40:36.908 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:36.908 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:36.909 DEBUG:boto:path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:36.909 DEBUG:boto:auth_path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:36.910 DEBUG:boto:Method: GET 2022-09-12T19:40:36.911 DEBUG:boto:Path: /jdpiak-69/b?versionId=null 2022-09-12T19:40:36.912 DEBUG:boto:Data: 2022-09-12T19:40:36.913 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.913 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:36.914 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.916 DEBUG:boto:Params: {} 2022-09-12T19:40:36.916 DEBUG:boto:Token: None 2022-09-12T19:40:36.917 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/b?versionId=null 2022-09-12T19:40:36.918 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CnP1SwOalD9d8AWQuKcaCuYKREI= 2022-09-12T19:40:36.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CnP1SwOalD9d8AWQuKcaCuYKREI=', 'Content-Length': '0'} 2022-09-12T19:40:36.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee3bd241f9e65394-00631f8b34-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.921 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:36.922 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:36.923 DEBUG:boto:path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:36.923 DEBUG:boto:auth_path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:36.924 DEBUG:boto:Method: GET 2022-09-12T19:40:36.924 DEBUG:boto:Path: /jdpiak-69/b?versionId=null 2022-09-12T19:40:36.925 DEBUG:boto:Data: 2022-09-12T19:40:36.925 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.926 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:36.926 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.927 DEBUG:boto:Params: {} 2022-09-12T19:40:36.927 DEBUG:boto:Token: None 2022-09-12T19:40:36.928 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/b?versionId=null 2022-09-12T19:40:36.929 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CnP1SwOalD9d8AWQuKcaCuYKREI= 2022-09-12T19:40:36.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CnP1SwOalD9d8AWQuKcaCuYKREI=', 'Content-Length': '0'} 2022-09-12T19:40:36.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000940f1f21894f8798-00631f8b34-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.932 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:36.933 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:36.934 DEBUG:boto:path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:36.934 DEBUG:boto:auth_path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:36.935 DEBUG:boto:Method: HEAD 2022-09-12T19:40:36.935 DEBUG:boto:Path: /jdpiak-69/b?versionId=null 2022-09-12T19:40:36.936 DEBUG:boto:Data: 2022-09-12T19:40:36.936 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.937 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:36.937 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.938 DEBUG:boto:Params: {} 2022-09-12T19:40:36.938 DEBUG:boto:Token: None 2022-09-12T19:40:36.939 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/b?versionId=null 2022-09-12T19:40:36.939 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KuaaxPxP1x9Zlx/BsPE1KoT08wM= 2022-09-12T19:40:36.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KuaaxPxP1x9Zlx/BsPE1KoT08wM=', 'Content-Length': '0'} 2022-09-12T19:40:36.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4e76fe5f92c755b-00631f8b34-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.943 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:36.943 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:36.944 DEBUG:boto:path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:36.944 DEBUG:boto:auth_path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:36.945 DEBUG:boto:Method: HEAD 2022-09-12T19:40:36.946 DEBUG:boto:Path: /jdpiak-69/b?versionId=null 2022-09-12T19:40:36.946 DEBUG:boto:Data: 2022-09-12T19:40:36.947 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.947 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:36.948 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.949 DEBUG:boto:Params: {} 2022-09-12T19:40:36.949 DEBUG:boto:Token: None 2022-09-12T19:40:36.950 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/b?versionId=null 2022-09-12T19:40:36.951 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KuaaxPxP1x9Zlx/BsPE1KoT08wM= 2022-09-12T19:40:36.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KuaaxPxP1x9Zlx/BsPE1KoT08wM=', 'Content-Length': '0'} 2022-09-12T19:40:36.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9710fbe35c07818-00631f8b34-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:40:36.955 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:36.956 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:36.957 DEBUG:boto:path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:36.957 DEBUG:boto:auth_path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:36.958 DEBUG:boto:Method: GET 2022-09-12T19:40:36.958 DEBUG:boto:Path: /jdpiak-69/b?versionId=null 2022-09-12T19:40:36.959 DEBUG:boto:Data: 2022-09-12T19:40:36.960 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.960 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:36.961 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.962 DEBUG:boto:Params: {} 2022-09-12T19:40:36.962 DEBUG:boto:Token: None 2022-09-12T19:40:36.963 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/b?versionId=null 2022-09-12T19:40:36.964 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CnP1SwOalD9d8AWQuKcaCuYKREI= 2022-09-12T19:40:36.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CnP1SwOalD9d8AWQuKcaCuYKREI=', 'Content-Length': '0'} 2022-09-12T19:40:36.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f49dbd136d5d6f2f-00631f8b34-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.969 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:36.970 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:36.971 DEBUG:boto:path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:36.971 DEBUG:boto:auth_path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:36.972 DEBUG:boto:Method: GET 2022-09-12T19:40:36.974 DEBUG:boto:Path: /jdpiak-69/b?versionId=null 2022-09-12T19:40:36.975 DEBUG:boto:Data: 2022-09-12T19:40:36.976 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.977 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:36.977 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.978 DEBUG:boto:Params: {} 2022-09-12T19:40:36.979 DEBUG:boto:Token: None 2022-09-12T19:40:36.980 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/b?versionId=null 2022-09-12T19:40:36.980 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CnP1SwOalD9d8AWQuKcaCuYKREI= 2022-09-12T19:40:36.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CnP1SwOalD9d8AWQuKcaCuYKREI=', 'Content-Length': '0'} 2022-09-12T19:40:36.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bf0d2fecf96df9d-00631f8b34-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:36.985 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:36.986 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:36.986 DEBUG:boto:path=/jdpiak-69/?prefix=b 2022-09-12T19:40:36.987 DEBUG:boto:auth_path=/jdpiak-69/?prefix=b 2022-09-12T19:40:36.988 DEBUG:boto:Method: GET 2022-09-12T19:40:36.989 DEBUG:boto:Path: /jdpiak-69/?prefix=b 2022-09-12T19:40:36.989 DEBUG:boto:Data: 2022-09-12T19:40:36.991 DEBUG:boto:Headers: {} 2022-09-12T19:40:36.992 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:36.992 DEBUG:boto:Port: 8000 2022-09-12T19:40:36.993 DEBUG:boto:Params: {} 2022-09-12T19:40:36.994 DEBUG:boto:Token: None 2022-09-12T19:40:36.995 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:36 GMT /jdpiak-69/ 2022-09-12T19:40:36.996 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cJY+C5R0CHejbwf/hYLivY82D8M= 2022-09-12T19:40:36.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cJY+C5R0CHejbwf/hYLivY82D8M=', 'Content-Length': '0'} 2022-09-12T19:40:37.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4e0001f08d34064-00631f8b34-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.002 DEBUG:boto:b'jdpiak-69b1000falseb2022-09-12T19:39:54.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:37.004 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:37.005 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:37.006 DEBUG:boto:path=/jdpiak-69/?prefix=b 2022-09-12T19:40:37.007 DEBUG:boto:auth_path=/jdpiak-69/?prefix=b 2022-09-12T19:40:37.008 DEBUG:boto:Method: GET 2022-09-12T19:40:37.009 DEBUG:boto:Path: /jdpiak-69/?prefix=b 2022-09-12T19:40:37.010 DEBUG:boto:Data: 2022-09-12T19:40:37.011 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.011 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:37.012 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.012 DEBUG:boto:Params: {} 2022-09-12T19:40:37.013 DEBUG:boto:Token: None 2022-09-12T19:40:37.013 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/ 2022-09-12T19:40:37.014 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XegTW0dwlaGb7CJVADmGlrtndys= 2022-09-12T19:40:37.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XegTW0dwlaGb7CJVADmGlrtndys=', 'Content-Length': '0'} 2022-09-12T19:40:37.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004818005a1f1f788f-00631f8b35-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.020 DEBUG:boto:b'jdpiak-69b1000falseb2022-09-12T19:39:54.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:37.021 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:37.022 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:37.023 DEBUG:boto:Method: HEAD 2022-09-12T19:40:37.024 DEBUG:boto:Path: /jdpiak-69/b 2022-09-12T19:40:37.025 DEBUG:boto:Data: 2022-09-12T19:40:37.026 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.026 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:37.027 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.028 DEBUG:boto:Params: {} 2022-09-12T19:40:37.029 DEBUG:boto:Token: None 2022-09-12T19:40:37.030 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/b 2022-09-12T19:40:37.032 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RGGmQeTeYv+P3jKWOHwRrmAXsG4= 2022-09-12T19:40:37.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RGGmQeTeYv+P3jKWOHwRrmAXsG4=', 'Content-Length': '0'} 2022-09-12T19:40:37.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038c3c2d7954b5650-00631f8b35-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.039 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:37.040 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:37.041 DEBUG:boto:Method: HEAD 2022-09-12T19:40:37.041 DEBUG:boto:Path: /jdpiak-69/b 2022-09-12T19:40:37.042 DEBUG:boto:Data: 2022-09-12T19:40:37.042 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.043 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:37.043 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.044 DEBUG:boto:Params: {} 2022-09-12T19:40:37.044 DEBUG:boto:Token: None 2022-09-12T19:40:37.045 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/b 2022-09-12T19:40:37.046 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RGGmQeTeYv+P3jKWOHwRrmAXsG4= 2022-09-12T19:40:37.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RGGmQeTeYv+P3jKWOHwRrmAXsG4=', 'Content-Length': '0'} 2022-09-12T19:40:37.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b486bb93a0f1797f-00631f8b35-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:40:37.050 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:37.050 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:37.051 DEBUG:boto:Method: GET 2022-09-12T19:40:37.051 DEBUG:boto:Path: /jdpiak-69/b 2022-09-12T19:40:37.052 DEBUG:boto:Data: 2022-09-12T19:40:37.053 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.053 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:37.054 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.054 DEBUG:boto:Params: {} 2022-09-12T19:40:37.055 DEBUG:boto:Token: None 2022-09-12T19:40:37.056 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/b 2022-09-12T19:40:37.056 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:a1q4xRx23PAUGT3DoPF0tYQ5sAY= 2022-09-12T19:40:37.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:a1q4xRx23PAUGT3DoPF0tYQ5sAY=', 'Content-Length': '0'} 2022-09-12T19:40:37.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003be4dff7eec5666b-00631f8b35-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.062 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:37.063 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:37.064 DEBUG:boto:Method: GET 2022-09-12T19:40:37.065 DEBUG:boto:Path: /jdpiak-69/b 2022-09-12T19:40:37.066 DEBUG:boto:Data: 2022-09-12T19:40:37.067 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.068 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:37.069 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.070 DEBUG:boto:Params: {} 2022-09-12T19:40:37.071 DEBUG:boto:Token: None 2022-09-12T19:40:37.072 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/b 2022-09-12T19:40:37.073 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:a1q4xRx23PAUGT3DoPF0tYQ5sAY= 2022-09-12T19:40:37.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:a1q4xRx23PAUGT3DoPF0tYQ5sAY=', 'Content-Length': '0'} 2022-09-12T19:40:37.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021b3537f0190c593-00631f8b35-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.122 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:37.123 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:37.124 DEBUG:boto:path=/jdpiak-69/?prefix=b 2022-09-12T19:40:37.124 DEBUG:boto:auth_path=/jdpiak-69/?prefix=b 2022-09-12T19:40:37.125 DEBUG:boto:Method: GET 2022-09-12T19:40:37.125 DEBUG:boto:Path: /jdpiak-69/?prefix=b 2022-09-12T19:40:37.126 DEBUG:boto:Data: 2022-09-12T19:40:37.126 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.127 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:37.127 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.128 DEBUG:boto:Params: {} 2022-09-12T19:40:37.128 DEBUG:boto:Token: None 2022-09-12T19:40:37.129 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/ 2022-09-12T19:40:37.130 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XegTW0dwlaGb7CJVADmGlrtndys= 2022-09-12T19:40:37.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XegTW0dwlaGb7CJVADmGlrtndys=', 'Content-Length': '0'} 2022-09-12T19:40:37.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dac344262854d783-00631f8b35-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.135 DEBUG:boto:b'jdpiak-69b1000falseb2022-09-12T19:39:54.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:37.136 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:37.137 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:37.138 DEBUG:boto:path=/jdpiak-69/?prefix=b 2022-09-12T19:40:37.139 DEBUG:boto:auth_path=/jdpiak-69/?prefix=b 2022-09-12T19:40:37.140 DEBUG:boto:Method: GET 2022-09-12T19:40:37.141 DEBUG:boto:Path: /jdpiak-69/?prefix=b 2022-09-12T19:40:37.142 DEBUG:boto:Data: 2022-09-12T19:40:37.143 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.144 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:37.145 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.146 DEBUG:boto:Params: {} 2022-09-12T19:40:37.147 DEBUG:boto:Token: None 2022-09-12T19:40:37.149 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/ 2022-09-12T19:40:37.150 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XegTW0dwlaGb7CJVADmGlrtndys= 2022-09-12T19:40:37.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XegTW0dwlaGb7CJVADmGlrtndys=', 'Content-Length': '0'} 2022-09-12T19:40:37.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc828eb00cc4c18c-00631f8b35-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.156 DEBUG:boto:b'jdpiak-69b1000falseb2022-09-12T19:39:54.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:37.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:40:37.159 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:37.160 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:37.161 DEBUG:boto:path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:37.162 DEBUG:boto:auth_path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:37.162 DEBUG:boto:Method: GET 2022-09-12T19:40:37.163 DEBUG:boto:Path: /jdpiak-69/c?versionId=null 2022-09-12T19:40:37.163 DEBUG:boto:Data: 2022-09-12T19:40:37.164 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.164 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:37.165 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.165 DEBUG:boto:Params: {} 2022-09-12T19:40:37.166 DEBUG:boto:Token: None 2022-09-12T19:40:37.167 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/c?versionId=null 2022-09-12T19:40:37.168 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ipc7G3kR1ErpBMzCTor0JExEfxU= 2022-09-12T19:40:37.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ipc7G3kR1ErpBMzCTor0JExEfxU=', 'Content-Length': '0'} 2022-09-12T19:40:37.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa091f8fd1cc4994-00631f8b35-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.171 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:37.172 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:37.173 DEBUG:boto:path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:37.173 DEBUG:boto:auth_path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:37.174 DEBUG:boto:Method: GET 2022-09-12T19:40:37.174 DEBUG:boto:Path: /jdpiak-69/c?versionId=null 2022-09-12T19:40:37.175 DEBUG:boto:Data: 2022-09-12T19:40:37.175 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.176 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:37.176 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.177 DEBUG:boto:Params: {} 2022-09-12T19:40:37.177 DEBUG:boto:Token: None 2022-09-12T19:40:37.178 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/c?versionId=null 2022-09-12T19:40:37.179 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ipc7G3kR1ErpBMzCTor0JExEfxU= 2022-09-12T19:40:37.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ipc7G3kR1ErpBMzCTor0JExEfxU=', 'Content-Length': '0'} 2022-09-12T19:40:37.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f4519fd8647f627-00631f8b35-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.182 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:37.183 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:37.184 DEBUG:boto:path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:37.184 DEBUG:boto:auth_path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:37.185 DEBUG:boto:Method: HEAD 2022-09-12T19:40:37.185 DEBUG:boto:Path: /jdpiak-69/c?versionId=null 2022-09-12T19:40:37.186 DEBUG:boto:Data: 2022-09-12T19:40:37.186 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.187 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:37.187 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.188 DEBUG:boto:Params: {} 2022-09-12T19:40:37.188 DEBUG:boto:Token: None 2022-09-12T19:40:37.189 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/c?versionId=null 2022-09-12T19:40:37.189 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Sd017EKUWMvmLuzmGWp/GCOIsZU= 2022-09-12T19:40:37.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Sd017EKUWMvmLuzmGWp/GCOIsZU=', 'Content-Length': '0'} 2022-09-12T19:40:37.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e9e771cd0c29607-00631f8b35-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.193 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:37.193 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:37.194 DEBUG:boto:path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:37.194 DEBUG:boto:auth_path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:37.195 DEBUG:boto:Method: HEAD 2022-09-12T19:40:37.195 DEBUG:boto:Path: /jdpiak-69/c?versionId=null 2022-09-12T19:40:37.196 DEBUG:boto:Data: 2022-09-12T19:40:37.196 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.197 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:37.197 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.198 DEBUG:boto:Params: {} 2022-09-12T19:40:37.198 DEBUG:boto:Token: None 2022-09-12T19:40:37.199 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/c?versionId=null 2022-09-12T19:40:37.199 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Sd017EKUWMvmLuzmGWp/GCOIsZU= 2022-09-12T19:40:37.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Sd017EKUWMvmLuzmGWp/GCOIsZU=', 'Content-Length': '0'} 2022-09-12T19:40:37.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052e24cc4c2b45b8f-00631f8b35-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:40:37.204 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:37.204 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:37.205 DEBUG:boto:path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:37.205 DEBUG:boto:auth_path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:37.208 DEBUG:boto:Method: GET 2022-09-12T19:40:37.208 DEBUG:boto:Path: /jdpiak-69/c?versionId=null 2022-09-12T19:40:37.209 DEBUG:boto:Data: 2022-09-12T19:40:37.209 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.210 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:37.210 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.211 DEBUG:boto:Params: {} 2022-09-12T19:40:37.212 DEBUG:boto:Token: None 2022-09-12T19:40:37.212 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/c?versionId=null 2022-09-12T19:40:37.213 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ipc7G3kR1ErpBMzCTor0JExEfxU= 2022-09-12T19:40:37.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ipc7G3kR1ErpBMzCTor0JExEfxU=', 'Content-Length': '0'} 2022-09-12T19:40:37.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003863d8b7e4c38a53-00631f8b35-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.261 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:37.261 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:37.262 DEBUG:boto:path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:37.263 DEBUG:boto:auth_path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:37.263 DEBUG:boto:Method: GET 2022-09-12T19:40:37.264 DEBUG:boto:Path: /jdpiak-69/c?versionId=null 2022-09-12T19:40:37.264 DEBUG:boto:Data: 2022-09-12T19:40:37.265 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.265 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:37.266 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.266 DEBUG:boto:Params: {} 2022-09-12T19:40:37.267 DEBUG:boto:Token: None 2022-09-12T19:40:37.268 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/c?versionId=null 2022-09-12T19:40:37.268 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ipc7G3kR1ErpBMzCTor0JExEfxU= 2022-09-12T19:40:37.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ipc7G3kR1ErpBMzCTor0JExEfxU=', 'Content-Length': '0'} 2022-09-12T19:40:37.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000318c40ce97b51d4f-00631f8b35-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.272 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:37.273 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:37.273 DEBUG:boto:path=/jdpiak-69/?prefix=c 2022-09-12T19:40:37.274 DEBUG:boto:auth_path=/jdpiak-69/?prefix=c 2022-09-12T19:40:37.275 DEBUG:boto:Method: GET 2022-09-12T19:40:37.275 DEBUG:boto:Path: /jdpiak-69/?prefix=c 2022-09-12T19:40:37.276 DEBUG:boto:Data: 2022-09-12T19:40:37.276 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.277 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:37.277 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.278 DEBUG:boto:Params: {} 2022-09-12T19:40:37.279 DEBUG:boto:Token: None 2022-09-12T19:40:37.279 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/ 2022-09-12T19:40:37.280 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XegTW0dwlaGb7CJVADmGlrtndys= 2022-09-12T19:40:37.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XegTW0dwlaGb7CJVADmGlrtndys=', 'Content-Length': '0'} 2022-09-12T19:40:37.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab892d1dc7bfe02a-00631f8b35-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.284 DEBUG:boto:b'jdpiak-69c1000falsec2022-09-12T19:39:54.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:37.285 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:37.286 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:37.286 DEBUG:boto:path=/jdpiak-69/?prefix=c 2022-09-12T19:40:37.287 DEBUG:boto:auth_path=/jdpiak-69/?prefix=c 2022-09-12T19:40:37.287 DEBUG:boto:Method: GET 2022-09-12T19:40:37.288 DEBUG:boto:Path: /jdpiak-69/?prefix=c 2022-09-12T19:40:37.289 DEBUG:boto:Data: 2022-09-12T19:40:37.290 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.290 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:37.291 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.292 DEBUG:boto:Params: {} 2022-09-12T19:40:37.292 DEBUG:boto:Token: None 2022-09-12T19:40:37.293 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/ 2022-09-12T19:40:37.294 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XegTW0dwlaGb7CJVADmGlrtndys= 2022-09-12T19:40:37.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XegTW0dwlaGb7CJVADmGlrtndys=', 'Content-Length': '0'} 2022-09-12T19:40:37.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000991e0b6fa28c06fa-00631f8b35-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.299 DEBUG:boto:b'jdpiak-69c1000falsec2022-09-12T19:39:54.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:37.300 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:37.301 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:37.302 DEBUG:boto:Method: HEAD 2022-09-12T19:40:37.302 DEBUG:boto:Path: /jdpiak-69/c 2022-09-12T19:40:37.303 DEBUG:boto:Data: 2022-09-12T19:40:37.303 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.304 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:37.304 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.305 DEBUG:boto:Params: {} 2022-09-12T19:40:37.306 DEBUG:boto:Token: None 2022-09-12T19:40:37.306 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/c 2022-09-12T19:40:37.307 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pX+SfQZBAISyV7tFg7IJ1k9T+/w= 2022-09-12T19:40:37.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pX+SfQZBAISyV7tFg7IJ1k9T+/w=', 'Content-Length': '0'} 2022-09-12T19:40:37.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a487a25af762430-00631f8b35-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.310 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:37.311 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:37.311 DEBUG:boto:Method: HEAD 2022-09-12T19:40:37.312 DEBUG:boto:Path: /jdpiak-69/c 2022-09-12T19:40:37.312 DEBUG:boto:Data: 2022-09-12T19:40:37.313 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.313 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:37.314 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.315 DEBUG:boto:Params: {} 2022-09-12T19:40:37.315 DEBUG:boto:Token: None 2022-09-12T19:40:37.316 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/c 2022-09-12T19:40:37.316 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pX+SfQZBAISyV7tFg7IJ1k9T+/w= 2022-09-12T19:40:37.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pX+SfQZBAISyV7tFg7IJ1k9T+/w=', 'Content-Length': '0'} 2022-09-12T19:40:37.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d81e8741180cc9f5-00631f8b35-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:40:37.320 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:37.321 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:37.321 DEBUG:boto:Method: GET 2022-09-12T19:40:37.322 DEBUG:boto:Path: /jdpiak-69/c 2022-09-12T19:40:37.323 DEBUG:boto:Data: 2022-09-12T19:40:37.323 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.324 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:37.324 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.325 DEBUG:boto:Params: {} 2022-09-12T19:40:37.325 DEBUG:boto:Token: None 2022-09-12T19:40:37.326 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/c 2022-09-12T19:40:37.327 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5ROWki6ZRbbv84xAWKqpbHWyI9Q= 2022-09-12T19:40:37.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5ROWki6ZRbbv84xAWKqpbHWyI9Q=', 'Content-Length': '0'} 2022-09-12T19:40:37.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d895a07aac62c334-00631f8b35-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.331 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:37.331 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:37.332 DEBUG:boto:Method: GET 2022-09-12T19:40:37.332 DEBUG:boto:Path: /jdpiak-69/c 2022-09-12T19:40:37.333 DEBUG:boto:Data: 2022-09-12T19:40:37.333 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.334 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:37.334 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.335 DEBUG:boto:Params: {} 2022-09-12T19:40:37.336 DEBUG:boto:Token: None 2022-09-12T19:40:37.336 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/c 2022-09-12T19:40:37.337 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5ROWki6ZRbbv84xAWKqpbHWyI9Q= 2022-09-12T19:40:37.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5ROWki6ZRbbv84xAWKqpbHWyI9Q=', 'Content-Length': '0'} 2022-09-12T19:40:37.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a3ceffa846b4cf1-00631f8b35-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.382 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:37.383 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:37.383 DEBUG:boto:path=/jdpiak-69/?prefix=c 2022-09-12T19:40:37.384 DEBUG:boto:auth_path=/jdpiak-69/?prefix=c 2022-09-12T19:40:37.385 DEBUG:boto:Method: GET 2022-09-12T19:40:37.385 DEBUG:boto:Path: /jdpiak-69/?prefix=c 2022-09-12T19:40:37.386 DEBUG:boto:Data: 2022-09-12T19:40:37.386 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.387 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:37.387 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.388 DEBUG:boto:Params: {} 2022-09-12T19:40:37.388 DEBUG:boto:Token: None 2022-09-12T19:40:37.389 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/ 2022-09-12T19:40:37.390 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XegTW0dwlaGb7CJVADmGlrtndys= 2022-09-12T19:40:37.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XegTW0dwlaGb7CJVADmGlrtndys=', 'Content-Length': '0'} 2022-09-12T19:40:37.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7e4f0d1af6b8a76-00631f8b35-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.393 DEBUG:boto:b'jdpiak-69c1000falsec2022-09-12T19:39:54.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:37.394 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:37.395 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:37.395 DEBUG:boto:path=/jdpiak-69/?prefix=c 2022-09-12T19:40:37.396 DEBUG:boto:auth_path=/jdpiak-69/?prefix=c 2022-09-12T19:40:37.397 DEBUG:boto:Method: GET 2022-09-12T19:40:37.397 DEBUG:boto:Path: /jdpiak-69/?prefix=c 2022-09-12T19:40:37.398 DEBUG:boto:Data: 2022-09-12T19:40:37.399 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.399 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:37.400 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.400 DEBUG:boto:Params: {} 2022-09-12T19:40:37.402 DEBUG:boto:Token: None 2022-09-12T19:40:37.403 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/ 2022-09-12T19:40:37.404 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XegTW0dwlaGb7CJVADmGlrtndys= 2022-09-12T19:40:37.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XegTW0dwlaGb7CJVADmGlrtndys=', 'Content-Length': '0'} 2022-09-12T19:40:37.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079c1236206f12575-00631f8b35-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.409 DEBUG:boto:b'jdpiak-69c1000falsec2022-09-12T19:39:54.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:37.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:40:37.411 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:37.411 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:37.412 DEBUG:boto:path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:37.412 DEBUG:boto:auth_path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:37.413 DEBUG:boto:Method: GET 2022-09-12T19:40:37.413 DEBUG:boto:Path: /jdpiak-69/d?versionId=null 2022-09-12T19:40:37.414 DEBUG:boto:Data: 2022-09-12T19:40:37.414 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.415 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:37.415 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.416 DEBUG:boto:Params: {} 2022-09-12T19:40:37.417 DEBUG:boto:Token: None 2022-09-12T19:40:37.417 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/d?versionId=null 2022-09-12T19:40:37.418 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:objwNvnSY7itgoET6OjnvaD1uMY= 2022-09-12T19:40:37.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:objwNvnSY7itgoET6OjnvaD1uMY=', 'Content-Length': '0'} 2022-09-12T19:40:37.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000073c8cbe138631fe-00631f8b35-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.422 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:37.422 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:37.423 DEBUG:boto:path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:37.423 DEBUG:boto:auth_path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:37.424 DEBUG:boto:Method: GET 2022-09-12T19:40:37.425 DEBUG:boto:Path: /jdpiak-69/d?versionId=null 2022-09-12T19:40:37.425 DEBUG:boto:Data: 2022-09-12T19:40:37.426 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.426 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:37.427 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.427 DEBUG:boto:Params: {} 2022-09-12T19:40:37.428 DEBUG:boto:Token: None 2022-09-12T19:40:37.429 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/d?versionId=null 2022-09-12T19:40:37.429 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:objwNvnSY7itgoET6OjnvaD1uMY= 2022-09-12T19:40:37.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:objwNvnSY7itgoET6OjnvaD1uMY=', 'Content-Length': '0'} 2022-09-12T19:40:37.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf6cc5a89e99c175-00631f8b35-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.433 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:37.433 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:37.434 DEBUG:boto:path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:37.434 DEBUG:boto:auth_path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:37.435 DEBUG:boto:Method: HEAD 2022-09-12T19:40:37.435 DEBUG:boto:Path: /jdpiak-69/d?versionId=null 2022-09-12T19:40:37.436 DEBUG:boto:Data: 2022-09-12T19:40:37.436 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.437 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:37.437 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.438 DEBUG:boto:Params: {} 2022-09-12T19:40:37.439 DEBUG:boto:Token: None 2022-09-12T19:40:37.439 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/d?versionId=null 2022-09-12T19:40:37.440 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dK4m7stiQDsETaVeljxr4E6EU7U= 2022-09-12T19:40:37.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dK4m7stiQDsETaVeljxr4E6EU7U=', 'Content-Length': '0'} 2022-09-12T19:40:37.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068a2c73a0b01f856-00631f8b35-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.443 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:37.444 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:37.444 DEBUG:boto:path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:37.445 DEBUG:boto:auth_path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:37.445 DEBUG:boto:Method: HEAD 2022-09-12T19:40:37.446 DEBUG:boto:Path: /jdpiak-69/d?versionId=null 2022-09-12T19:40:37.446 DEBUG:boto:Data: 2022-09-12T19:40:37.447 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.447 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:37.448 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.448 DEBUG:boto:Params: {} 2022-09-12T19:40:37.449 DEBUG:boto:Token: None 2022-09-12T19:40:37.449 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/d?versionId=null 2022-09-12T19:40:37.450 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dK4m7stiQDsETaVeljxr4E6EU7U= 2022-09-12T19:40:37.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dK4m7stiQDsETaVeljxr4E6EU7U=', 'Content-Length': '0'} 2022-09-12T19:40:37.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b18abce0d0c47f57-00631f8b35-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:40:37.455 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:37.455 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:37.456 DEBUG:boto:path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:37.456 DEBUG:boto:auth_path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:37.457 DEBUG:boto:Method: GET 2022-09-12T19:40:37.457 DEBUG:boto:Path: /jdpiak-69/d?versionId=null 2022-09-12T19:40:37.458 DEBUG:boto:Data: 2022-09-12T19:40:37.459 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.459 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:37.460 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.460 DEBUG:boto:Params: {} 2022-09-12T19:40:37.461 DEBUG:boto:Token: None 2022-09-12T19:40:37.461 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/d?versionId=null 2022-09-12T19:40:37.462 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:objwNvnSY7itgoET6OjnvaD1uMY= 2022-09-12T19:40:37.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:objwNvnSY7itgoET6OjnvaD1uMY=', 'Content-Length': '0'} 2022-09-12T19:40:37.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d491daf215368a7b-00631f8b35-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.506 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:37.507 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:37.507 DEBUG:boto:path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:37.508 DEBUG:boto:auth_path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:37.509 DEBUG:boto:Method: GET 2022-09-12T19:40:37.509 DEBUG:boto:Path: /jdpiak-69/d?versionId=null 2022-09-12T19:40:37.510 DEBUG:boto:Data: 2022-09-12T19:40:37.510 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.511 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:37.512 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.512 DEBUG:boto:Params: {} 2022-09-12T19:40:37.514 DEBUG:boto:Token: None 2022-09-12T19:40:37.516 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/d?versionId=null 2022-09-12T19:40:37.516 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:objwNvnSY7itgoET6OjnvaD1uMY= 2022-09-12T19:40:37.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:objwNvnSY7itgoET6OjnvaD1uMY=', 'Content-Length': '0'} 2022-09-12T19:40:37.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dbfc3fc5aba954e-00631f8b35-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.523 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:37.525 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:37.526 DEBUG:boto:path=/jdpiak-69/?prefix=d 2022-09-12T19:40:37.527 DEBUG:boto:auth_path=/jdpiak-69/?prefix=d 2022-09-12T19:40:37.528 DEBUG:boto:Method: GET 2022-09-12T19:40:37.529 DEBUG:boto:Path: /jdpiak-69/?prefix=d 2022-09-12T19:40:37.530 DEBUG:boto:Data: 2022-09-12T19:40:37.531 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.532 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:37.533 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.534 DEBUG:boto:Params: {} 2022-09-12T19:40:37.535 DEBUG:boto:Token: None 2022-09-12T19:40:37.536 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/ 2022-09-12T19:40:37.537 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XegTW0dwlaGb7CJVADmGlrtndys= 2022-09-12T19:40:37.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XegTW0dwlaGb7CJVADmGlrtndys=', 'Content-Length': '0'} 2022-09-12T19:40:37.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000377c49782386a28b-00631f8b35-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.543 DEBUG:boto:b'jdpiak-69d1000falsed2022-09-12T19:39:54.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:37.545 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:37.546 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:37.547 DEBUG:boto:path=/jdpiak-69/?prefix=d 2022-09-12T19:40:37.548 DEBUG:boto:auth_path=/jdpiak-69/?prefix=d 2022-09-12T19:40:37.549 DEBUG:boto:Method: GET 2022-09-12T19:40:37.550 DEBUG:boto:Path: /jdpiak-69/?prefix=d 2022-09-12T19:40:37.551 DEBUG:boto:Data: 2022-09-12T19:40:37.552 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.553 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:37.554 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.556 DEBUG:boto:Params: {} 2022-09-12T19:40:37.557 DEBUG:boto:Token: None 2022-09-12T19:40:37.559 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/ 2022-09-12T19:40:37.560 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XegTW0dwlaGb7CJVADmGlrtndys= 2022-09-12T19:40:37.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XegTW0dwlaGb7CJVADmGlrtndys=', 'Content-Length': '0'} 2022-09-12T19:40:37.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000309891221e5f0ef8-00631f8b35-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.565 DEBUG:boto:b'jdpiak-69d1000falsed2022-09-12T19:39:54.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:37.566 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:37.567 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:37.568 DEBUG:boto:Method: HEAD 2022-09-12T19:40:37.568 DEBUG:boto:Path: /jdpiak-69/d 2022-09-12T19:40:37.569 DEBUG:boto:Data: 2022-09-12T19:40:37.569 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.570 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:37.570 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.571 DEBUG:boto:Params: {} 2022-09-12T19:40:37.571 DEBUG:boto:Token: None 2022-09-12T19:40:37.572 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/d 2022-09-12T19:40:37.573 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E7CLe6vSpL5pbuxW1+cQ99rp8nY= 2022-09-12T19:40:37.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E7CLe6vSpL5pbuxW1+cQ99rp8nY=', 'Content-Length': '0'} 2022-09-12T19:40:37.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4b882ce6848e5e0-00631f8b35-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.577 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:37.578 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:37.578 DEBUG:boto:Method: HEAD 2022-09-12T19:40:37.579 DEBUG:boto:Path: /jdpiak-69/d 2022-09-12T19:40:37.579 DEBUG:boto:Data: 2022-09-12T19:40:37.580 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.580 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:37.581 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.582 DEBUG:boto:Params: {} 2022-09-12T19:40:37.583 DEBUG:boto:Token: None 2022-09-12T19:40:37.583 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/d 2022-09-12T19:40:37.584 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E7CLe6vSpL5pbuxW1+cQ99rp8nY= 2022-09-12T19:40:37.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E7CLe6vSpL5pbuxW1+cQ99rp8nY=', 'Content-Length': '0'} 2022-09-12T19:40:37.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e67a555e390f85b7-00631f8b35-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:40:37.588 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:37.589 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:37.589 DEBUG:boto:Method: GET 2022-09-12T19:40:37.590 DEBUG:boto:Path: /jdpiak-69/d 2022-09-12T19:40:37.591 DEBUG:boto:Data: 2022-09-12T19:40:37.591 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.592 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:37.592 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.593 DEBUG:boto:Params: {} 2022-09-12T19:40:37.594 DEBUG:boto:Token: None 2022-09-12T19:40:37.594 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/d 2022-09-12T19:40:37.595 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TBuNNb3nzRkLBLp0474cvGtsdMQ= 2022-09-12T19:40:37.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TBuNNb3nzRkLBLp0474cvGtsdMQ=', 'Content-Length': '0'} 2022-09-12T19:40:37.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d10aa8b7616e1274-00631f8b35-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.600 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:37.600 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:37.601 DEBUG:boto:Method: GET 2022-09-12T19:40:37.602 DEBUG:boto:Path: /jdpiak-69/d 2022-09-12T19:40:37.603 DEBUG:boto:Data: 2022-09-12T19:40:37.604 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.605 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:37.606 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.607 DEBUG:boto:Params: {} 2022-09-12T19:40:37.608 DEBUG:boto:Token: None 2022-09-12T19:40:37.609 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/d 2022-09-12T19:40:37.610 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TBuNNb3nzRkLBLp0474cvGtsdMQ= 2022-09-12T19:40:37.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TBuNNb3nzRkLBLp0474cvGtsdMQ=', 'Content-Length': '0'} 2022-09-12T19:40:37.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043ba042a43e7fe34-00631f8b35-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.615 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:37.616 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:37.617 DEBUG:boto:path=/jdpiak-69/?prefix=d 2022-09-12T19:40:37.619 DEBUG:boto:auth_path=/jdpiak-69/?prefix=d 2022-09-12T19:40:37.620 DEBUG:boto:Method: GET 2022-09-12T19:40:37.621 DEBUG:boto:Path: /jdpiak-69/?prefix=d 2022-09-12T19:40:37.622 DEBUG:boto:Data: 2022-09-12T19:40:37.623 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.623 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:37.624 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.625 DEBUG:boto:Params: {} 2022-09-12T19:40:37.625 DEBUG:boto:Token: None 2022-09-12T19:40:37.626 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/ 2022-09-12T19:40:37.627 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XegTW0dwlaGb7CJVADmGlrtndys= 2022-09-12T19:40:37.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XegTW0dwlaGb7CJVADmGlrtndys=', 'Content-Length': '0'} 2022-09-12T19:40:37.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000135e86ebf85e19bc-00631f8b35-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.674 DEBUG:boto:b'jdpiak-69d1000falsed2022-09-12T19:39:54.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:37.676 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:37.677 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:37.678 DEBUG:boto:path=/jdpiak-69/?prefix=d 2022-09-12T19:40:37.679 DEBUG:boto:auth_path=/jdpiak-69/?prefix=d 2022-09-12T19:40:37.680 DEBUG:boto:Method: GET 2022-09-12T19:40:37.681 DEBUG:boto:Path: /jdpiak-69/?prefix=d 2022-09-12T19:40:37.682 DEBUG:boto:Data: 2022-09-12T19:40:37.683 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.684 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:37.685 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.686 DEBUG:boto:Params: {} 2022-09-12T19:40:37.687 DEBUG:boto:Token: None 2022-09-12T19:40:37.688 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/ 2022-09-12T19:40:37.689 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XegTW0dwlaGb7CJVADmGlrtndys= 2022-09-12T19:40:37.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XegTW0dwlaGb7CJVADmGlrtndys=', 'Content-Length': '0'} 2022-09-12T19:40:37.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0d008d2d0a9ecab-00631f8b35-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.695 DEBUG:boto:b'jdpiak-69d1000falsed2022-09-12T19:39:54.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:37.696 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-69 zones={test-zone2, test-zone1} 2022-09-12T19:40:37.697 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-69 zones={test-zone3, test-zone1} 2022-09-12T19:40:37.698 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:37.699 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:37.700 DEBUG:boto:Method: HEAD 2022-09-12T19:40:37.701 DEBUG:boto:Path: /jdpiak-69/ 2022-09-12T19:40:37.702 DEBUG:boto:Data: 2022-09-12T19:40:37.702 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.703 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:37.704 DEBUG:boto:Port: 8001 2022-09-12T19:40:37.705 DEBUG:boto:Params: {} 2022-09-12T19:40:37.706 DEBUG:boto:Token: None 2022-09-12T19:40:37.707 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/ 2022-09-12T19:40:37.708 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lc0qUnRoJW6CiRgpV/O60/CfJI8= 2022-09-12T19:40:37.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lc0qUnRoJW6CiRgpV/O60/CfJI8=', 'Content-Length': '0'} 2022-09-12T19:40:37.716 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bceb6cf114cfe148-00631f8b35-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.717 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:37.718 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:37.719 DEBUG:boto:Method: HEAD 2022-09-12T19:40:37.720 DEBUG:boto:Path: /jdpiak-69/ 2022-09-12T19:40:37.721 DEBUG:boto:Data: 2022-09-12T19:40:37.722 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.723 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:37.724 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.725 DEBUG:boto:Params: {} 2022-09-12T19:40:37.726 DEBUG:boto:Token: None 2022-09-12T19:40:37.727 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/ 2022-09-12T19:40:37.728 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lc0qUnRoJW6CiRgpV/O60/CfJI8= 2022-09-12T19:40:37.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lc0qUnRoJW6CiRgpV/O60/CfJI8=', 'Content-Length': '0'} 2022-09-12T19:40:37.732 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd2913f1f3180d4b-00631f8b35-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.733 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:40:37.733 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:37.734 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:37.735 DEBUG:boto:path=/jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:37.735 DEBUG:boto:auth_path=/jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:37.736 DEBUG:boto:Method: GET 2022-09-12T19:40:37.736 DEBUG:boto:Path: /jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:37.737 DEBUG:boto:Data: 2022-09-12T19:40:37.738 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.738 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:37.739 DEBUG:boto:Port: 8001 2022-09-12T19:40:37.739 DEBUG:boto:Params: {} 2022-09-12T19:40:37.740 DEBUG:boto:Token: None 2022-09-12T19:40:37.740 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/?versions 2022-09-12T19:40:37.741 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5vkYXtnXPICu+/6Hop0Mp67epPs= 2022-09-12T19:40:37.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5vkYXtnXPICu+/6Hop0Mp67epPs=', 'Content-Length': '0'} 2022-09-12T19:40:37.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000087e925839799fcc-00631f8b35-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.746 DEBUG:boto:b'jdpiak-69999falseanulltrue2022-09-12T19:39:54.609Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:39:54.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:39:54.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:39:54.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:37.747 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:40:37.747 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:40:37.748 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:40:37.748 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:40:37.749 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:40:37.749 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:37.750 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:37.751 DEBUG:boto:path=/jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:37.751 DEBUG:boto:auth_path=/jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:37.752 DEBUG:boto:Method: GET 2022-09-12T19:40:37.752 DEBUG:boto:Path: /jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:37.753 DEBUG:boto:Data: 2022-09-12T19:40:37.753 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.754 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:37.754 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.755 DEBUG:boto:Params: {} 2022-09-12T19:40:37.756 DEBUG:boto:Token: None 2022-09-12T19:40:37.756 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/?versions 2022-09-12T19:40:37.757 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5vkYXtnXPICu+/6Hop0Mp67epPs= 2022-09-12T19:40:37.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5vkYXtnXPICu+/6Hop0Mp67epPs=', 'Content-Length': '0'} 2022-09-12T19:40:37.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095f5fabc386b555a-00631f8b35-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.771 DEBUG:boto:b'jdpiak-69999falseanulltrue2022-09-12T19:39:54.609Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:39:54.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:39:54.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:39:54.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:37.773 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:40:37.774 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:40:37.776 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:40:37.777 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:40:37.778 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:37.779 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:37.780 DEBUG:boto:path=/jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:37.781 DEBUG:boto:auth_path=/jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:37.782 DEBUG:boto:Method: GET 2022-09-12T19:40:37.783 DEBUG:boto:Path: /jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:37.784 DEBUG:boto:Data: 2022-09-12T19:40:37.784 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.785 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:37.786 DEBUG:boto:Port: 8001 2022-09-12T19:40:37.787 DEBUG:boto:Params: {} 2022-09-12T19:40:37.789 DEBUG:boto:Token: None 2022-09-12T19:40:37.790 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/?versions 2022-09-12T19:40:37.791 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5vkYXtnXPICu+/6Hop0Mp67epPs= 2022-09-12T19:40:37.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5vkYXtnXPICu+/6Hop0Mp67epPs=', 'Content-Length': '0'} 2022-09-12T19:40:37.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5717221e034d912-00631f8b35-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.797 DEBUG:boto:b'jdpiak-69999falseanulltrue2022-09-12T19:39:54.609Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:39:54.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:39:54.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:39:54.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:37.799 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:37.800 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:37.801 DEBUG:boto:path=/jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:37.802 DEBUG:boto:auth_path=/jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:37.803 DEBUG:boto:Method: GET 2022-09-12T19:40:37.804 DEBUG:boto:Path: /jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:37.806 DEBUG:boto:Data: 2022-09-12T19:40:37.807 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.808 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:37.809 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.810 DEBUG:boto:Params: {} 2022-09-12T19:40:37.812 DEBUG:boto:Token: None 2022-09-12T19:40:37.813 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/?versions 2022-09-12T19:40:37.815 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5vkYXtnXPICu+/6Hop0Mp67epPs= 2022-09-12T19:40:37.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5vkYXtnXPICu+/6Hop0Mp67epPs=', 'Content-Length': '0'} 2022-09-12T19:40:37.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1f1497488593099-00631f8b35-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.821 DEBUG:boto:b'jdpiak-69999falseanulltrue2022-09-12T19:39:54.609Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:39:54.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:39:54.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:39:54.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:37.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:40:37.823 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:37.824 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:37.825 DEBUG:boto:path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:37.825 DEBUG:boto:auth_path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:37.826 DEBUG:boto:Method: GET 2022-09-12T19:40:37.827 DEBUG:boto:Path: /jdpiak-69/a?versionId=null 2022-09-12T19:40:37.827 DEBUG:boto:Data: 2022-09-12T19:40:37.828 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.828 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:37.829 DEBUG:boto:Port: 8001 2022-09-12T19:40:37.830 DEBUG:boto:Params: {} 2022-09-12T19:40:37.830 DEBUG:boto:Token: None 2022-09-12T19:40:37.831 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/a?versionId=null 2022-09-12T19:40:37.832 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:waJ+qn+KXtOG43vQAX6R6fu11VI= 2022-09-12T19:40:37.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:waJ+qn+KXtOG43vQAX6R6fu11VI=', 'Content-Length': '0'} 2022-09-12T19:40:37.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceadb733a2f9b3df-00631f8b35-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.835 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:37.836 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:37.837 DEBUG:boto:path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:37.837 DEBUG:boto:auth_path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:37.838 DEBUG:boto:Method: GET 2022-09-12T19:40:37.839 DEBUG:boto:Path: /jdpiak-69/a?versionId=null 2022-09-12T19:40:37.839 DEBUG:boto:Data: 2022-09-12T19:40:37.840 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.840 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:37.841 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.841 DEBUG:boto:Params: {} 2022-09-12T19:40:37.842 DEBUG:boto:Token: None 2022-09-12T19:40:37.843 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/a?versionId=null 2022-09-12T19:40:37.844 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:waJ+qn+KXtOG43vQAX6R6fu11VI= 2022-09-12T19:40:37.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:waJ+qn+KXtOG43vQAX6R6fu11VI=', 'Content-Length': '0'} 2022-09-12T19:40:37.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a298a9dd1f24ec0-00631f8b35-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.895 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:37.896 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:37.897 DEBUG:boto:path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:37.897 DEBUG:boto:auth_path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:37.898 DEBUG:boto:Method: HEAD 2022-09-12T19:40:37.900 DEBUG:boto:Path: /jdpiak-69/a?versionId=null 2022-09-12T19:40:37.900 DEBUG:boto:Data: 2022-09-12T19:40:37.901 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.902 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:37.903 DEBUG:boto:Port: 8001 2022-09-12T19:40:37.904 DEBUG:boto:Params: {} 2022-09-12T19:40:37.905 DEBUG:boto:Token: None 2022-09-12T19:40:37.906 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/a?versionId=null 2022-09-12T19:40:37.907 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EvOoYWpoQzocWPrjyTm8oz7d96g= 2022-09-12T19:40:37.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EvOoYWpoQzocWPrjyTm8oz7d96g=', 'Content-Length': '0'} 2022-09-12T19:40:37.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000128be592f2ed390-00631f8b35-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.912 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:37.913 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:37.914 DEBUG:boto:path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:37.914 DEBUG:boto:auth_path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:37.915 DEBUG:boto:Method: HEAD 2022-09-12T19:40:37.916 DEBUG:boto:Path: /jdpiak-69/a?versionId=null 2022-09-12T19:40:37.917 DEBUG:boto:Data: 2022-09-12T19:40:37.917 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.918 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:37.919 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.919 DEBUG:boto:Params: {} 2022-09-12T19:40:37.920 DEBUG:boto:Token: None 2022-09-12T19:40:37.921 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/a?versionId=null 2022-09-12T19:40:37.921 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EvOoYWpoQzocWPrjyTm8oz7d96g= 2022-09-12T19:40:37.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EvOoYWpoQzocWPrjyTm8oz7d96g=', 'Content-Length': '0'} 2022-09-12T19:40:37.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000792f5db044a42b11-00631f8b35-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:40:37.928 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:37.929 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:37.930 DEBUG:boto:path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:37.931 DEBUG:boto:auth_path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:37.931 DEBUG:boto:Method: GET 2022-09-12T19:40:37.932 DEBUG:boto:Path: /jdpiak-69/a?versionId=null 2022-09-12T19:40:37.933 DEBUG:boto:Data: 2022-09-12T19:40:37.934 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.935 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:37.936 DEBUG:boto:Port: 8001 2022-09-12T19:40:37.937 DEBUG:boto:Params: {} 2022-09-12T19:40:37.938 DEBUG:boto:Token: None 2022-09-12T19:40:37.939 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/a?versionId=null 2022-09-12T19:40:37.939 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:waJ+qn+KXtOG43vQAX6R6fu11VI= 2022-09-12T19:40:37.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:waJ+qn+KXtOG43vQAX6R6fu11VI=', 'Content-Length': '0'} 2022-09-12T19:40:37.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000636942da99a4d0df-00631f8b35-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.944 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:37.945 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:37.945 DEBUG:boto:path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:37.946 DEBUG:boto:auth_path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:37.946 DEBUG:boto:Method: GET 2022-09-12T19:40:37.947 DEBUG:boto:Path: /jdpiak-69/a?versionId=null 2022-09-12T19:40:37.948 DEBUG:boto:Data: 2022-09-12T19:40:37.948 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.949 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:37.950 DEBUG:boto:Port: 8000 2022-09-12T19:40:37.950 DEBUG:boto:Params: {} 2022-09-12T19:40:37.951 DEBUG:boto:Token: None 2022-09-12T19:40:37.952 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/a?versionId=null 2022-09-12T19:40:37.953 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:waJ+qn+KXtOG43vQAX6R6fu11VI= 2022-09-12T19:40:37.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:waJ+qn+KXtOG43vQAX6R6fu11VI=', 'Content-Length': '0'} 2022-09-12T19:40:37.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a6b5c499344b3ae-00631f8b35-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:37.961 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:37.962 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:37.963 DEBUG:boto:path=/jdpiak-69/?prefix=a 2022-09-12T19:40:37.964 DEBUG:boto:auth_path=/jdpiak-69/?prefix=a 2022-09-12T19:40:37.965 DEBUG:boto:Method: GET 2022-09-12T19:40:37.966 DEBUG:boto:Path: /jdpiak-69/?prefix=a 2022-09-12T19:40:37.967 DEBUG:boto:Data: 2022-09-12T19:40:37.968 DEBUG:boto:Headers: {} 2022-09-12T19:40:37.969 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:37.970 DEBUG:boto:Port: 8001 2022-09-12T19:40:37.971 DEBUG:boto:Params: {} 2022-09-12T19:40:37.972 DEBUG:boto:Token: None 2022-09-12T19:40:37.974 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:37 GMT /jdpiak-69/ 2022-09-12T19:40:37.975 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XegTW0dwlaGb7CJVADmGlrtndys= 2022-09-12T19:40:37.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XegTW0dwlaGb7CJVADmGlrtndys=', 'Content-Length': '0'} 2022-09-12T19:40:37.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8683e93e62b413f-00631f8b35-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.022 DEBUG:boto:b'jdpiak-69a1000falsea2022-09-12T19:39:54.609Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:38.023 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:38.024 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:38.024 DEBUG:boto:path=/jdpiak-69/?prefix=a 2022-09-12T19:40:38.025 DEBUG:boto:auth_path=/jdpiak-69/?prefix=a 2022-09-12T19:40:38.026 DEBUG:boto:Method: GET 2022-09-12T19:40:38.026 DEBUG:boto:Path: /jdpiak-69/?prefix=a 2022-09-12T19:40:38.027 DEBUG:boto:Data: 2022-09-12T19:40:38.028 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.028 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:38.029 DEBUG:boto:Port: 8000 2022-09-12T19:40:38.030 DEBUG:boto:Params: {} 2022-09-12T19:40:38.030 DEBUG:boto:Token: None 2022-09-12T19:40:38.031 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/ 2022-09-12T19:40:38.032 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RXKEVo2XLrNW+jgFj2hOwAQZJAM= 2022-09-12T19:40:38.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RXKEVo2XLrNW+jgFj2hOwAQZJAM=', 'Content-Length': '0'} 2022-09-12T19:40:38.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028b8c3995180824a-00631f8b36-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.036 DEBUG:boto:b'jdpiak-69a1000falsea2022-09-12T19:39:54.609Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:38.037 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:38.038 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:38.038 DEBUG:boto:Method: HEAD 2022-09-12T19:40:38.039 DEBUG:boto:Path: /jdpiak-69/a 2022-09-12T19:40:38.040 DEBUG:boto:Data: 2022-09-12T19:40:38.040 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.041 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:38.041 DEBUG:boto:Port: 8001 2022-09-12T19:40:38.042 DEBUG:boto:Params: {} 2022-09-12T19:40:38.043 DEBUG:boto:Token: None 2022-09-12T19:40:38.044 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/a 2022-09-12T19:40:38.044 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wNcTVgT+KsQfd2AT2e340Wy4SfM= 2022-09-12T19:40:38.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wNcTVgT+KsQfd2AT2e340Wy4SfM=', 'Content-Length': '0'} 2022-09-12T19:40:38.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040af70bc79681349-00631f8b36-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.049 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:38.049 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:38.050 DEBUG:boto:Method: HEAD 2022-09-12T19:40:38.050 DEBUG:boto:Path: /jdpiak-69/a 2022-09-12T19:40:38.051 DEBUG:boto:Data: 2022-09-12T19:40:38.051 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.052 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:38.052 DEBUG:boto:Port: 8000 2022-09-12T19:40:38.053 DEBUG:boto:Params: {} 2022-09-12T19:40:38.053 DEBUG:boto:Token: None 2022-09-12T19:40:38.054 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/a 2022-09-12T19:40:38.055 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wNcTVgT+KsQfd2AT2e340Wy4SfM= 2022-09-12T19:40:38.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wNcTVgT+KsQfd2AT2e340Wy4SfM=', 'Content-Length': '0'} 2022-09-12T19:40:38.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc632832f23f2a4a-00631f8b36-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:40:38.060 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:38.060 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:38.061 DEBUG:boto:Method: GET 2022-09-12T19:40:38.061 DEBUG:boto:Path: /jdpiak-69/a 2022-09-12T19:40:38.062 DEBUG:boto:Data: 2022-09-12T19:40:38.063 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.063 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:38.064 DEBUG:boto:Port: 8001 2022-09-12T19:40:38.065 DEBUG:boto:Params: {} 2022-09-12T19:40:38.066 DEBUG:boto:Token: None 2022-09-12T19:40:38.066 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/a 2022-09-12T19:40:38.067 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1hkPIm7hKbJSAei3/szTJV1//TQ= 2022-09-12T19:40:38.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1hkPIm7hKbJSAei3/szTJV1//TQ=', 'Content-Length': '0'} 2022-09-12T19:40:38.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d66908cb39e019be-00631f8b36-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.071 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:38.072 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:38.073 DEBUG:boto:Method: GET 2022-09-12T19:40:38.074 DEBUG:boto:Path: /jdpiak-69/a 2022-09-12T19:40:38.074 DEBUG:boto:Data: 2022-09-12T19:40:38.075 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.075 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:38.076 DEBUG:boto:Port: 8000 2022-09-12T19:40:38.076 DEBUG:boto:Params: {} 2022-09-12T19:40:38.077 DEBUG:boto:Token: None 2022-09-12T19:40:38.078 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/a 2022-09-12T19:40:38.079 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1hkPIm7hKbJSAei3/szTJV1//TQ= 2022-09-12T19:40:38.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1hkPIm7hKbJSAei3/szTJV1//TQ=', 'Content-Length': '0'} 2022-09-12T19:40:38.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbdba10a30e7ba6b-00631f8b36-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.083 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:38.083 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:38.084 DEBUG:boto:path=/jdpiak-69/?prefix=a 2022-09-12T19:40:38.085 DEBUG:boto:auth_path=/jdpiak-69/?prefix=a 2022-09-12T19:40:38.085 DEBUG:boto:Method: GET 2022-09-12T19:40:38.086 DEBUG:boto:Path: /jdpiak-69/?prefix=a 2022-09-12T19:40:38.086 DEBUG:boto:Data: 2022-09-12T19:40:38.087 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.088 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:38.088 DEBUG:boto:Port: 8001 2022-09-12T19:40:38.089 DEBUG:boto:Params: {} 2022-09-12T19:40:38.090 DEBUG:boto:Token: None 2022-09-12T19:40:38.091 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/ 2022-09-12T19:40:38.092 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RXKEVo2XLrNW+jgFj2hOwAQZJAM= 2022-09-12T19:40:38.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RXKEVo2XLrNW+jgFj2hOwAQZJAM=', 'Content-Length': '0'} 2022-09-12T19:40:38.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000596225b70facee2d-00631f8b36-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.097 DEBUG:boto:b'jdpiak-69a1000falsea2022-09-12T19:39:54.609Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:38.099 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:38.102 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:38.103 DEBUG:boto:path=/jdpiak-69/?prefix=a 2022-09-12T19:40:38.104 DEBUG:boto:auth_path=/jdpiak-69/?prefix=a 2022-09-12T19:40:38.105 DEBUG:boto:Method: GET 2022-09-12T19:40:38.106 DEBUG:boto:Path: /jdpiak-69/?prefix=a 2022-09-12T19:40:38.107 DEBUG:boto:Data: 2022-09-12T19:40:38.108 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.109 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:38.110 DEBUG:boto:Port: 8000 2022-09-12T19:40:38.113 DEBUG:boto:Params: {} 2022-09-12T19:40:38.114 DEBUG:boto:Token: None 2022-09-12T19:40:38.116 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/ 2022-09-12T19:40:38.117 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RXKEVo2XLrNW+jgFj2hOwAQZJAM= 2022-09-12T19:40:38.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RXKEVo2XLrNW+jgFj2hOwAQZJAM=', 'Content-Length': '0'} 2022-09-12T19:40:38.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000367dfc0cb8663df3-00631f8b36-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.125 DEBUG:boto:b'jdpiak-69a1000falsea2022-09-12T19:39:54.609Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:38.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:40:38.128 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:38.129 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:38.130 DEBUG:boto:path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:38.131 DEBUG:boto:auth_path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:38.132 DEBUG:boto:Method: GET 2022-09-12T19:40:38.133 DEBUG:boto:Path: /jdpiak-69/b?versionId=null 2022-09-12T19:40:38.134 DEBUG:boto:Data: 2022-09-12T19:40:38.135 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.136 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:38.136 DEBUG:boto:Port: 8001 2022-09-12T19:40:38.137 DEBUG:boto:Params: {} 2022-09-12T19:40:38.138 DEBUG:boto:Token: None 2022-09-12T19:40:38.139 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/b?versionId=null 2022-09-12T19:40:38.140 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RzfTBuf+WUAKjH/piCYv+vP0MyU= 2022-09-12T19:40:38.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RzfTBuf+WUAKjH/piCYv+vP0MyU=', 'Content-Length': '0'} 2022-09-12T19:40:38.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d09a2a753051a5a-00631f8b36-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.145 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:38.145 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:38.146 DEBUG:boto:path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:38.147 DEBUG:boto:auth_path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:38.147 DEBUG:boto:Method: GET 2022-09-12T19:40:38.149 DEBUG:boto:Path: /jdpiak-69/b?versionId=null 2022-09-12T19:40:38.150 DEBUG:boto:Data: 2022-09-12T19:40:38.151 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.152 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:38.153 DEBUG:boto:Port: 8000 2022-09-12T19:40:38.154 DEBUG:boto:Params: {} 2022-09-12T19:40:38.155 DEBUG:boto:Token: None 2022-09-12T19:40:38.156 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/b?versionId=null 2022-09-12T19:40:38.157 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RzfTBuf+WUAKjH/piCYv+vP0MyU= 2022-09-12T19:40:38.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RzfTBuf+WUAKjH/piCYv+vP0MyU=', 'Content-Length': '0'} 2022-09-12T19:40:38.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066e6be825231495e-00631f8b36-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.204 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:38.205 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:38.205 DEBUG:boto:path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:38.206 DEBUG:boto:auth_path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:38.206 DEBUG:boto:Method: HEAD 2022-09-12T19:40:38.207 DEBUG:boto:Path: /jdpiak-69/b?versionId=null 2022-09-12T19:40:38.208 DEBUG:boto:Data: 2022-09-12T19:40:38.208 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.209 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:38.209 DEBUG:boto:Port: 8001 2022-09-12T19:40:38.210 DEBUG:boto:Params: {} 2022-09-12T19:40:38.210 DEBUG:boto:Token: None 2022-09-12T19:40:38.211 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/b?versionId=null 2022-09-12T19:40:38.212 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LwTWB0RrYxxYl5UvpwsrlhdWMeQ= 2022-09-12T19:40:38.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LwTWB0RrYxxYl5UvpwsrlhdWMeQ=', 'Content-Length': '0'} 2022-09-12T19:40:38.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000182ab48656b95cd1-00631f8b36-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.215 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:38.216 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:38.216 DEBUG:boto:path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:38.217 DEBUG:boto:auth_path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:38.217 DEBUG:boto:Method: HEAD 2022-09-12T19:40:38.218 DEBUG:boto:Path: /jdpiak-69/b?versionId=null 2022-09-12T19:40:38.219 DEBUG:boto:Data: 2022-09-12T19:40:38.219 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.220 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:38.220 DEBUG:boto:Port: 8000 2022-09-12T19:40:38.221 DEBUG:boto:Params: {} 2022-09-12T19:40:38.221 DEBUG:boto:Token: None 2022-09-12T19:40:38.223 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/b?versionId=null 2022-09-12T19:40:38.223 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LwTWB0RrYxxYl5UvpwsrlhdWMeQ= 2022-09-12T19:40:38.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LwTWB0RrYxxYl5UvpwsrlhdWMeQ=', 'Content-Length': '0'} 2022-09-12T19:40:38.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003813044ff0b4ebf0-00631f8b36-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:40:38.228 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:38.229 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:38.229 DEBUG:boto:path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:38.230 DEBUG:boto:auth_path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:38.230 DEBUG:boto:Method: GET 2022-09-12T19:40:38.231 DEBUG:boto:Path: /jdpiak-69/b?versionId=null 2022-09-12T19:40:38.231 DEBUG:boto:Data: 2022-09-12T19:40:38.232 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.232 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:38.233 DEBUG:boto:Port: 8001 2022-09-12T19:40:38.233 DEBUG:boto:Params: {} 2022-09-12T19:40:38.235 DEBUG:boto:Token: None 2022-09-12T19:40:38.235 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/b?versionId=null 2022-09-12T19:40:38.236 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RzfTBuf+WUAKjH/piCYv+vP0MyU= 2022-09-12T19:40:38.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RzfTBuf+WUAKjH/piCYv+vP0MyU=', 'Content-Length': '0'} 2022-09-12T19:40:38.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7e4565760333a24-00631f8b36-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.282 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:38.283 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:38.283 DEBUG:boto:path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:38.284 DEBUG:boto:auth_path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:38.285 DEBUG:boto:Method: GET 2022-09-12T19:40:38.286 DEBUG:boto:Path: /jdpiak-69/b?versionId=null 2022-09-12T19:40:38.286 DEBUG:boto:Data: 2022-09-12T19:40:38.287 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.288 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:38.289 DEBUG:boto:Port: 8000 2022-09-12T19:40:38.290 DEBUG:boto:Params: {} 2022-09-12T19:40:38.291 DEBUG:boto:Token: None 2022-09-12T19:40:38.292 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/b?versionId=null 2022-09-12T19:40:38.293 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RzfTBuf+WUAKjH/piCYv+vP0MyU= 2022-09-12T19:40:38.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RzfTBuf+WUAKjH/piCYv+vP0MyU=', 'Content-Length': '0'} 2022-09-12T19:40:38.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d74204cd01b19e5-00631f8b36-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.298 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:38.299 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:38.300 DEBUG:boto:path=/jdpiak-69/?prefix=b 2022-09-12T19:40:38.300 DEBUG:boto:auth_path=/jdpiak-69/?prefix=b 2022-09-12T19:40:38.301 DEBUG:boto:Method: GET 2022-09-12T19:40:38.301 DEBUG:boto:Path: /jdpiak-69/?prefix=b 2022-09-12T19:40:38.302 DEBUG:boto:Data: 2022-09-12T19:40:38.302 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.303 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:38.303 DEBUG:boto:Port: 8001 2022-09-12T19:40:38.304 DEBUG:boto:Params: {} 2022-09-12T19:40:38.305 DEBUG:boto:Token: None 2022-09-12T19:40:38.305 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/ 2022-09-12T19:40:38.306 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RXKEVo2XLrNW+jgFj2hOwAQZJAM= 2022-09-12T19:40:38.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RXKEVo2XLrNW+jgFj2hOwAQZJAM=', 'Content-Length': '0'} 2022-09-12T19:40:38.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb8602bbc825cbe2-00631f8b36-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.311 DEBUG:boto:b'jdpiak-69b1000falseb2022-09-12T19:39:54.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:38.313 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:38.314 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:38.315 DEBUG:boto:path=/jdpiak-69/?prefix=b 2022-09-12T19:40:38.316 DEBUG:boto:auth_path=/jdpiak-69/?prefix=b 2022-09-12T19:40:38.317 DEBUG:boto:Method: GET 2022-09-12T19:40:38.318 DEBUG:boto:Path: /jdpiak-69/?prefix=b 2022-09-12T19:40:38.319 DEBUG:boto:Data: 2022-09-12T19:40:38.320 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.321 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:38.322 DEBUG:boto:Port: 8000 2022-09-12T19:40:38.323 DEBUG:boto:Params: {} 2022-09-12T19:40:38.324 DEBUG:boto:Token: None 2022-09-12T19:40:38.325 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/ 2022-09-12T19:40:38.326 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RXKEVo2XLrNW+jgFj2hOwAQZJAM= 2022-09-12T19:40:38.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RXKEVo2XLrNW+jgFj2hOwAQZJAM=', 'Content-Length': '0'} 2022-09-12T19:40:38.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000962b9f8fbb453534-00631f8b36-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.331 DEBUG:boto:b'jdpiak-69b1000falseb2022-09-12T19:39:54.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:38.332 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:38.333 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:38.333 DEBUG:boto:Method: HEAD 2022-09-12T19:40:38.334 DEBUG:boto:Path: /jdpiak-69/b 2022-09-12T19:40:38.334 DEBUG:boto:Data: 2022-09-12T19:40:38.335 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.335 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:38.336 DEBUG:boto:Port: 8001 2022-09-12T19:40:38.336 DEBUG:boto:Params: {} 2022-09-12T19:40:38.337 DEBUG:boto:Token: None 2022-09-12T19:40:38.338 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/b 2022-09-12T19:40:38.339 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1NWlQVoUQLBptcTAuceO67+AmaA= 2022-09-12T19:40:38.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1NWlQVoUQLBptcTAuceO67+AmaA=', 'Content-Length': '0'} 2022-09-12T19:40:38.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000430dc0be6b9c3c4b-00631f8b36-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.343 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:38.343 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:38.344 DEBUG:boto:Method: HEAD 2022-09-12T19:40:38.344 DEBUG:boto:Path: /jdpiak-69/b 2022-09-12T19:40:38.345 DEBUG:boto:Data: 2022-09-12T19:40:38.345 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.346 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:38.346 DEBUG:boto:Port: 8000 2022-09-12T19:40:38.347 DEBUG:boto:Params: {} 2022-09-12T19:40:38.348 DEBUG:boto:Token: None 2022-09-12T19:40:38.348 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/b 2022-09-12T19:40:38.349 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1NWlQVoUQLBptcTAuceO67+AmaA= 2022-09-12T19:40:38.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1NWlQVoUQLBptcTAuceO67+AmaA=', 'Content-Length': '0'} 2022-09-12T19:40:38.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a809a5fa02bba1d1-00631f8b36-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:40:38.354 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:38.355 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:38.356 DEBUG:boto:Method: GET 2022-09-12T19:40:38.356 DEBUG:boto:Path: /jdpiak-69/b 2022-09-12T19:40:38.357 DEBUG:boto:Data: 2022-09-12T19:40:38.358 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.359 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:38.360 DEBUG:boto:Port: 8001 2022-09-12T19:40:38.361 DEBUG:boto:Params: {} 2022-09-12T19:40:38.362 DEBUG:boto:Token: None 2022-09-12T19:40:38.363 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/b 2022-09-12T19:40:38.364 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xUNhi2hzCHM98WBk4XjuWOEvO18= 2022-09-12T19:40:38.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xUNhi2hzCHM98WBk4XjuWOEvO18=', 'Content-Length': '0'} 2022-09-12T19:40:38.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096f3298f109096b6-00631f8b36-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.414 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:38.416 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:38.417 DEBUG:boto:Method: GET 2022-09-12T19:40:38.418 DEBUG:boto:Path: /jdpiak-69/b 2022-09-12T19:40:38.419 DEBUG:boto:Data: 2022-09-12T19:40:38.420 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.421 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:38.422 DEBUG:boto:Port: 8000 2022-09-12T19:40:38.423 DEBUG:boto:Params: {} 2022-09-12T19:40:38.424 DEBUG:boto:Token: None 2022-09-12T19:40:38.425 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/b 2022-09-12T19:40:38.426 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xUNhi2hzCHM98WBk4XjuWOEvO18= 2022-09-12T19:40:38.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xUNhi2hzCHM98WBk4XjuWOEvO18=', 'Content-Length': '0'} 2022-09-12T19:40:38.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b3d9d99677785a6-00631f8b36-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.432 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:38.433 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:38.433 DEBUG:boto:path=/jdpiak-69/?prefix=b 2022-09-12T19:40:38.434 DEBUG:boto:auth_path=/jdpiak-69/?prefix=b 2022-09-12T19:40:38.435 DEBUG:boto:Method: GET 2022-09-12T19:40:38.436 DEBUG:boto:Path: /jdpiak-69/?prefix=b 2022-09-12T19:40:38.437 DEBUG:boto:Data: 2022-09-12T19:40:38.438 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.439 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:38.440 DEBUG:boto:Port: 8001 2022-09-12T19:40:38.441 DEBUG:boto:Params: {} 2022-09-12T19:40:38.442 DEBUG:boto:Token: None 2022-09-12T19:40:38.443 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/ 2022-09-12T19:40:38.444 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RXKEVo2XLrNW+jgFj2hOwAQZJAM= 2022-09-12T19:40:38.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RXKEVo2XLrNW+jgFj2hOwAQZJAM=', 'Content-Length': '0'} 2022-09-12T19:40:38.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087d10c973be13f80-00631f8b36-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.448 DEBUG:boto:b'jdpiak-69b1000falseb2022-09-12T19:39:54.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:38.449 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:38.449 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:38.450 DEBUG:boto:path=/jdpiak-69/?prefix=b 2022-09-12T19:40:38.450 DEBUG:boto:auth_path=/jdpiak-69/?prefix=b 2022-09-12T19:40:38.451 DEBUG:boto:Method: GET 2022-09-12T19:40:38.452 DEBUG:boto:Path: /jdpiak-69/?prefix=b 2022-09-12T19:40:38.452 DEBUG:boto:Data: 2022-09-12T19:40:38.453 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.453 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:38.454 DEBUG:boto:Port: 8000 2022-09-12T19:40:38.454 DEBUG:boto:Params: {} 2022-09-12T19:40:38.455 DEBUG:boto:Token: None 2022-09-12T19:40:38.455 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/ 2022-09-12T19:40:38.456 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RXKEVo2XLrNW+jgFj2hOwAQZJAM= 2022-09-12T19:40:38.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RXKEVo2XLrNW+jgFj2hOwAQZJAM=', 'Content-Length': '0'} 2022-09-12T19:40:38.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e5d345d58bf5919-00631f8b36-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.502 DEBUG:boto:b'jdpiak-69b1000falseb2022-09-12T19:39:54.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:38.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:40:38.504 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:38.505 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:38.506 DEBUG:boto:path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:38.507 DEBUG:boto:auth_path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:38.508 DEBUG:boto:Method: GET 2022-09-12T19:40:38.509 DEBUG:boto:Path: /jdpiak-69/c?versionId=null 2022-09-12T19:40:38.510 DEBUG:boto:Data: 2022-09-12T19:40:38.511 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.513 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:38.514 DEBUG:boto:Port: 8001 2022-09-12T19:40:38.515 DEBUG:boto:Params: {} 2022-09-12T19:40:38.517 DEBUG:boto:Token: None 2022-09-12T19:40:38.518 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/c?versionId=null 2022-09-12T19:40:38.519 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fmOLkec5ayU2M72kQHY+if9YqXM= 2022-09-12T19:40:38.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fmOLkec5ayU2M72kQHY+if9YqXM=', 'Content-Length': '0'} 2022-09-12T19:40:38.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000604ca0f88c875c5a-00631f8b36-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.524 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:38.525 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:38.526 DEBUG:boto:path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:38.527 DEBUG:boto:auth_path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:38.528 DEBUG:boto:Method: GET 2022-09-12T19:40:38.530 DEBUG:boto:Path: /jdpiak-69/c?versionId=null 2022-09-12T19:40:38.531 DEBUG:boto:Data: 2022-09-12T19:40:38.532 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.533 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:38.533 DEBUG:boto:Port: 8000 2022-09-12T19:40:38.534 DEBUG:boto:Params: {} 2022-09-12T19:40:38.536 DEBUG:boto:Token: None 2022-09-12T19:40:38.537 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/c?versionId=null 2022-09-12T19:40:38.538 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fmOLkec5ayU2M72kQHY+if9YqXM= 2022-09-12T19:40:38.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fmOLkec5ayU2M72kQHY+if9YqXM=', 'Content-Length': '0'} 2022-09-12T19:40:38.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c71ef718678b752-00631f8b36-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.544 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:38.545 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:38.546 DEBUG:boto:path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:38.547 DEBUG:boto:auth_path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:38.549 DEBUG:boto:Method: HEAD 2022-09-12T19:40:38.550 DEBUG:boto:Path: /jdpiak-69/c?versionId=null 2022-09-12T19:40:38.551 DEBUG:boto:Data: 2022-09-12T19:40:38.552 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.553 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:38.554 DEBUG:boto:Port: 8001 2022-09-12T19:40:38.555 DEBUG:boto:Params: {} 2022-09-12T19:40:38.556 DEBUG:boto:Token: None 2022-09-12T19:40:38.561 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/c?versionId=null 2022-09-12T19:40:38.562 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:v0rGIqpRXd/2IgNi3zzZh82FWCk= 2022-09-12T19:40:38.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:v0rGIqpRXd/2IgNi3zzZh82FWCk=', 'Content-Length': '0'} 2022-09-12T19:40:38.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f450a9fbe6501ea-00631f8b36-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.567 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:38.568 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:38.569 DEBUG:boto:path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:38.569 DEBUG:boto:auth_path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:38.570 DEBUG:boto:Method: HEAD 2022-09-12T19:40:38.571 DEBUG:boto:Path: /jdpiak-69/c?versionId=null 2022-09-12T19:40:38.572 DEBUG:boto:Data: 2022-09-12T19:40:38.572 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.573 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:38.573 DEBUG:boto:Port: 8000 2022-09-12T19:40:38.574 DEBUG:boto:Params: {} 2022-09-12T19:40:38.575 DEBUG:boto:Token: None 2022-09-12T19:40:38.576 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/c?versionId=null 2022-09-12T19:40:38.577 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:v0rGIqpRXd/2IgNi3zzZh82FWCk= 2022-09-12T19:40:38.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:v0rGIqpRXd/2IgNi3zzZh82FWCk=', 'Content-Length': '0'} 2022-09-12T19:40:38.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d99154bf4263ed83-00631f8b36-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:40:38.585 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:38.585 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:38.586 DEBUG:boto:path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:38.586 DEBUG:boto:auth_path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:38.587 DEBUG:boto:Method: GET 2022-09-12T19:40:38.587 DEBUG:boto:Path: /jdpiak-69/c?versionId=null 2022-09-12T19:40:38.588 DEBUG:boto:Data: 2022-09-12T19:40:38.589 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.589 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:38.592 DEBUG:boto:Port: 8001 2022-09-12T19:40:38.593 DEBUG:boto:Params: {} 2022-09-12T19:40:38.593 DEBUG:boto:Token: None 2022-09-12T19:40:38.594 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/c?versionId=null 2022-09-12T19:40:38.594 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fmOLkec5ayU2M72kQHY+if9YqXM= 2022-09-12T19:40:38.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fmOLkec5ayU2M72kQHY+if9YqXM=', 'Content-Length': '0'} 2022-09-12T19:40:38.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002244ac900c283812-00631f8b36-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.598 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:38.599 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:38.599 DEBUG:boto:path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:38.600 DEBUG:boto:auth_path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:38.600 DEBUG:boto:Method: GET 2022-09-12T19:40:38.601 DEBUG:boto:Path: /jdpiak-69/c?versionId=null 2022-09-12T19:40:38.601 DEBUG:boto:Data: 2022-09-12T19:40:38.602 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.602 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:38.603 DEBUG:boto:Port: 8000 2022-09-12T19:40:38.603 DEBUG:boto:Params: {} 2022-09-12T19:40:38.604 DEBUG:boto:Token: None 2022-09-12T19:40:38.605 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/c?versionId=null 2022-09-12T19:40:38.606 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fmOLkec5ayU2M72kQHY+if9YqXM= 2022-09-12T19:40:38.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fmOLkec5ayU2M72kQHY+if9YqXM=', 'Content-Length': '0'} 2022-09-12T19:40:38.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a0edf572498bf5c-00631f8b36-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.655 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:38.656 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:38.657 DEBUG:boto:path=/jdpiak-69/?prefix=c 2022-09-12T19:40:38.658 DEBUG:boto:auth_path=/jdpiak-69/?prefix=c 2022-09-12T19:40:38.659 DEBUG:boto:Method: GET 2022-09-12T19:40:38.660 DEBUG:boto:Path: /jdpiak-69/?prefix=c 2022-09-12T19:40:38.661 DEBUG:boto:Data: 2022-09-12T19:40:38.662 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.664 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:38.665 DEBUG:boto:Port: 8001 2022-09-12T19:40:38.666 DEBUG:boto:Params: {} 2022-09-12T19:40:38.667 DEBUG:boto:Token: None 2022-09-12T19:40:38.668 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/ 2022-09-12T19:40:38.669 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RXKEVo2XLrNW+jgFj2hOwAQZJAM= 2022-09-12T19:40:38.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RXKEVo2XLrNW+jgFj2hOwAQZJAM=', 'Content-Length': '0'} 2022-09-12T19:40:38.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000580d94b06704ad6f-00631f8b36-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.676 DEBUG:boto:b'jdpiak-69c1000falsec2022-09-12T19:39:54.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:38.678 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:38.679 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:38.680 DEBUG:boto:path=/jdpiak-69/?prefix=c 2022-09-12T19:40:38.681 DEBUG:boto:auth_path=/jdpiak-69/?prefix=c 2022-09-12T19:40:38.682 DEBUG:boto:Method: GET 2022-09-12T19:40:38.683 DEBUG:boto:Path: /jdpiak-69/?prefix=c 2022-09-12T19:40:38.684 DEBUG:boto:Data: 2022-09-12T19:40:38.685 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.687 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:38.688 DEBUG:boto:Port: 8000 2022-09-12T19:40:38.689 DEBUG:boto:Params: {} 2022-09-12T19:40:38.690 DEBUG:boto:Token: None 2022-09-12T19:40:38.690 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/ 2022-09-12T19:40:38.691 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RXKEVo2XLrNW+jgFj2hOwAQZJAM= 2022-09-12T19:40:38.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RXKEVo2XLrNW+jgFj2hOwAQZJAM=', 'Content-Length': '0'} 2022-09-12T19:40:38.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a395456f2bb09411-00631f8b36-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.696 DEBUG:boto:b'jdpiak-69c1000falsec2022-09-12T19:39:54.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:38.697 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:38.698 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:38.699 DEBUG:boto:Method: HEAD 2022-09-12T19:40:38.700 DEBUG:boto:Path: /jdpiak-69/c 2022-09-12T19:40:38.701 DEBUG:boto:Data: 2022-09-12T19:40:38.702 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.703 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:38.705 DEBUG:boto:Port: 8001 2022-09-12T19:40:38.706 DEBUG:boto:Params: {} 2022-09-12T19:40:38.706 DEBUG:boto:Token: None 2022-09-12T19:40:38.707 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/c 2022-09-12T19:40:38.708 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eyuBd8IGVXtqDCeEVoWkWEO0JUY= 2022-09-12T19:40:38.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eyuBd8IGVXtqDCeEVoWkWEO0JUY=', 'Content-Length': '0'} 2022-09-12T19:40:38.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c652d164351a0ed2-00631f8b36-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.712 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:38.713 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:38.713 DEBUG:boto:Method: HEAD 2022-09-12T19:40:38.714 DEBUG:boto:Path: /jdpiak-69/c 2022-09-12T19:40:38.714 DEBUG:boto:Data: 2022-09-12T19:40:38.715 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.715 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:38.716 DEBUG:boto:Port: 8000 2022-09-12T19:40:38.716 DEBUG:boto:Params: {} 2022-09-12T19:40:38.717 DEBUG:boto:Token: None 2022-09-12T19:40:38.718 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/c 2022-09-12T19:40:38.719 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eyuBd8IGVXtqDCeEVoWkWEO0JUY= 2022-09-12T19:40:38.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eyuBd8IGVXtqDCeEVoWkWEO0JUY=', 'Content-Length': '0'} 2022-09-12T19:40:38.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea85cf4943c408cb-00631f8b36-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:40:38.725 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:38.726 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:38.728 DEBUG:boto:Method: GET 2022-09-12T19:40:38.729 DEBUG:boto:Path: /jdpiak-69/c 2022-09-12T19:40:38.730 DEBUG:boto:Data: 2022-09-12T19:40:38.731 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.732 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:38.733 DEBUG:boto:Port: 8001 2022-09-12T19:40:38.734 DEBUG:boto:Params: {} 2022-09-12T19:40:38.735 DEBUG:boto:Token: None 2022-09-12T19:40:38.737 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/c 2022-09-12T19:40:38.738 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AycfBhIWM7ogz+vmgNFhnIYO1Gg= 2022-09-12T19:40:38.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AycfBhIWM7ogz+vmgNFhnIYO1Gg=', 'Content-Length': '0'} 2022-09-12T19:40:38.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011e830767deaf9c5-00631f8b36-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.786 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:38.787 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:38.788 DEBUG:boto:Method: GET 2022-09-12T19:40:38.788 DEBUG:boto:Path: /jdpiak-69/c 2022-09-12T19:40:38.789 DEBUG:boto:Data: 2022-09-12T19:40:38.789 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.790 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:38.791 DEBUG:boto:Port: 8000 2022-09-12T19:40:38.792 DEBUG:boto:Params: {} 2022-09-12T19:40:38.792 DEBUG:boto:Token: None 2022-09-12T19:40:38.793 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/c 2022-09-12T19:40:38.793 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AycfBhIWM7ogz+vmgNFhnIYO1Gg= 2022-09-12T19:40:38.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AycfBhIWM7ogz+vmgNFhnIYO1Gg=', 'Content-Length': '0'} 2022-09-12T19:40:38.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000176876e5390a2880-00631f8b36-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.802 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:38.802 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:38.803 DEBUG:boto:path=/jdpiak-69/?prefix=c 2022-09-12T19:40:38.804 DEBUG:boto:auth_path=/jdpiak-69/?prefix=c 2022-09-12T19:40:38.804 DEBUG:boto:Method: GET 2022-09-12T19:40:38.805 DEBUG:boto:Path: /jdpiak-69/?prefix=c 2022-09-12T19:40:38.806 DEBUG:boto:Data: 2022-09-12T19:40:38.806 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.807 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:38.807 DEBUG:boto:Port: 8001 2022-09-12T19:40:38.808 DEBUG:boto:Params: {} 2022-09-12T19:40:38.808 DEBUG:boto:Token: None 2022-09-12T19:40:38.809 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/ 2022-09-12T19:40:38.810 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RXKEVo2XLrNW+jgFj2hOwAQZJAM= 2022-09-12T19:40:38.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RXKEVo2XLrNW+jgFj2hOwAQZJAM=', 'Content-Length': '0'} 2022-09-12T19:40:38.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b35339f16cb81970-00631f8b36-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.814 DEBUG:boto:b'jdpiak-69c1000falsec2022-09-12T19:39:54.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:38.815 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:38.816 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:38.816 DEBUG:boto:path=/jdpiak-69/?prefix=c 2022-09-12T19:40:38.817 DEBUG:boto:auth_path=/jdpiak-69/?prefix=c 2022-09-12T19:40:38.817 DEBUG:boto:Method: GET 2022-09-12T19:40:38.818 DEBUG:boto:Path: /jdpiak-69/?prefix=c 2022-09-12T19:40:38.818 DEBUG:boto:Data: 2022-09-12T19:40:38.819 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.820 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:38.820 DEBUG:boto:Port: 8000 2022-09-12T19:40:38.821 DEBUG:boto:Params: {} 2022-09-12T19:40:38.821 DEBUG:boto:Token: None 2022-09-12T19:40:38.822 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/ 2022-09-12T19:40:38.823 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RXKEVo2XLrNW+jgFj2hOwAQZJAM= 2022-09-12T19:40:38.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RXKEVo2XLrNW+jgFj2hOwAQZJAM=', 'Content-Length': '0'} 2022-09-12T19:40:38.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6ef3a0d4bdf7fd4-00631f8b36-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.870 DEBUG:boto:b'jdpiak-69c1000falsec2022-09-12T19:39:54.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:38.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:40:38.872 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:38.873 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:38.873 DEBUG:boto:path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:38.874 DEBUG:boto:auth_path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:38.874 DEBUG:boto:Method: GET 2022-09-12T19:40:38.875 DEBUG:boto:Path: /jdpiak-69/d?versionId=null 2022-09-12T19:40:38.875 DEBUG:boto:Data: 2022-09-12T19:40:38.876 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.877 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:38.877 DEBUG:boto:Port: 8001 2022-09-12T19:40:38.878 DEBUG:boto:Params: {} 2022-09-12T19:40:38.878 DEBUG:boto:Token: None 2022-09-12T19:40:38.879 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/d?versionId=null 2022-09-12T19:40:38.880 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D1CW6mih3HQPEaI/XbVnYhU7JyI= 2022-09-12T19:40:38.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D1CW6mih3HQPEaI/XbVnYhU7JyI=', 'Content-Length': '0'} 2022-09-12T19:40:38.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ecf8c17dcaf8aa9-00631f8b36-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.884 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:38.884 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:38.885 DEBUG:boto:path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:38.886 DEBUG:boto:auth_path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:38.886 DEBUG:boto:Method: GET 2022-09-12T19:40:38.887 DEBUG:boto:Path: /jdpiak-69/d?versionId=null 2022-09-12T19:40:38.887 DEBUG:boto:Data: 2022-09-12T19:40:38.888 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.888 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:38.889 DEBUG:boto:Port: 8000 2022-09-12T19:40:38.889 DEBUG:boto:Params: {} 2022-09-12T19:40:38.891 DEBUG:boto:Token: None 2022-09-12T19:40:38.891 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/d?versionId=null 2022-09-12T19:40:38.892 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D1CW6mih3HQPEaI/XbVnYhU7JyI= 2022-09-12T19:40:38.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D1CW6mih3HQPEaI/XbVnYhU7JyI=', 'Content-Length': '0'} 2022-09-12T19:40:38.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4c1dee8151e815f-00631f8b36-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.896 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:38.896 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:38.897 DEBUG:boto:path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:38.897 DEBUG:boto:auth_path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:38.898 DEBUG:boto:Method: HEAD 2022-09-12T19:40:38.898 DEBUG:boto:Path: /jdpiak-69/d?versionId=null 2022-09-12T19:40:38.899 DEBUG:boto:Data: 2022-09-12T19:40:38.900 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.900 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:38.901 DEBUG:boto:Port: 8001 2022-09-12T19:40:38.901 DEBUG:boto:Params: {} 2022-09-12T19:40:38.902 DEBUG:boto:Token: None 2022-09-12T19:40:38.903 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/d?versionId=null 2022-09-12T19:40:38.903 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rzk3wbCOE0AqXXLjKKBtn0ukmTk= 2022-09-12T19:40:38.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rzk3wbCOE0AqXXLjKKBtn0ukmTk=', 'Content-Length': '0'} 2022-09-12T19:40:38.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053f7667edef7fa6f-00631f8b36-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.907 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:38.907 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:38.908 DEBUG:boto:path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:38.909 DEBUG:boto:auth_path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:38.909 DEBUG:boto:Method: HEAD 2022-09-12T19:40:38.910 DEBUG:boto:Path: /jdpiak-69/d?versionId=null 2022-09-12T19:40:38.910 DEBUG:boto:Data: 2022-09-12T19:40:38.911 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.912 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:38.912 DEBUG:boto:Port: 8000 2022-09-12T19:40:38.913 DEBUG:boto:Params: {} 2022-09-12T19:40:38.914 DEBUG:boto:Token: None 2022-09-12T19:40:38.914 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/d?versionId=null 2022-09-12T19:40:38.915 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rzk3wbCOE0AqXXLjKKBtn0ukmTk= 2022-09-12T19:40:38.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rzk3wbCOE0AqXXLjKKBtn0ukmTk=', 'Content-Length': '0'} 2022-09-12T19:40:38.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8fa859eeace6c03-00631f8b36-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:40:38.919 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:38.920 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:38.920 DEBUG:boto:path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:38.921 DEBUG:boto:auth_path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:38.921 DEBUG:boto:Method: GET 2022-09-12T19:40:38.922 DEBUG:boto:Path: /jdpiak-69/d?versionId=null 2022-09-12T19:40:38.922 DEBUG:boto:Data: 2022-09-12T19:40:38.923 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.923 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:38.924 DEBUG:boto:Port: 8001 2022-09-12T19:40:38.924 DEBUG:boto:Params: {} 2022-09-12T19:40:38.925 DEBUG:boto:Token: None 2022-09-12T19:40:38.926 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/d?versionId=null 2022-09-12T19:40:38.926 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D1CW6mih3HQPEaI/XbVnYhU7JyI= 2022-09-12T19:40:38.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D1CW6mih3HQPEaI/XbVnYhU7JyI=', 'Content-Length': '0'} 2022-09-12T19:40:38.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d16d98019e80022-00631f8b36-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.975 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:38.977 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:38.978 DEBUG:boto:path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:38.979 DEBUG:boto:auth_path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:38.979 DEBUG:boto:Method: GET 2022-09-12T19:40:38.980 DEBUG:boto:Path: /jdpiak-69/d?versionId=null 2022-09-12T19:40:38.981 DEBUG:boto:Data: 2022-09-12T19:40:38.983 DEBUG:boto:Headers: {} 2022-09-12T19:40:38.984 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:38.985 DEBUG:boto:Port: 8000 2022-09-12T19:40:38.986 DEBUG:boto:Params: {} 2022-09-12T19:40:38.987 DEBUG:boto:Token: None 2022-09-12T19:40:38.988 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:38 GMT /jdpiak-69/d?versionId=null 2022-09-12T19:40:38.989 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D1CW6mih3HQPEaI/XbVnYhU7JyI= 2022-09-12T19:40:38.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D1CW6mih3HQPEaI/XbVnYhU7JyI=', 'Content-Length': '0'} 2022-09-12T19:40:38.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d696ea114181602-00631f8b36-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:38.994 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:38.995 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:38.996 DEBUG:boto:path=/jdpiak-69/?prefix=d 2022-09-12T19:40:38.996 DEBUG:boto:auth_path=/jdpiak-69/?prefix=d 2022-09-12T19:40:38.997 DEBUG:boto:Method: GET 2022-09-12T19:40:38.998 DEBUG:boto:Path: /jdpiak-69/?prefix=d 2022-09-12T19:40:38.998 DEBUG:boto:Data: 2022-09-12T19:40:38.999 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.000 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.001 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.001 DEBUG:boto:Params: {} 2022-09-12T19:40:39.002 DEBUG:boto:Token: None 2022-09-12T19:40:39.002 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/ 2022-09-12T19:40:39.003 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A= 2022-09-12T19:40:39.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A=', 'Content-Length': '0'} 2022-09-12T19:40:39.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a852e53a1345aa9c-00631f8b37-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.010 DEBUG:boto:b'jdpiak-69d1000falsed2022-09-12T19:39:54.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:39.013 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:39.014 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:39.020 DEBUG:boto:path=/jdpiak-69/?prefix=d 2022-09-12T19:40:39.021 DEBUG:boto:auth_path=/jdpiak-69/?prefix=d 2022-09-12T19:40:39.021 DEBUG:boto:Method: GET 2022-09-12T19:40:39.022 DEBUG:boto:Path: /jdpiak-69/?prefix=d 2022-09-12T19:40:39.022 DEBUG:boto:Data: 2022-09-12T19:40:39.023 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.023 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:39.024 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.024 DEBUG:boto:Params: {} 2022-09-12T19:40:39.025 DEBUG:boto:Token: None 2022-09-12T19:40:39.026 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/ 2022-09-12T19:40:39.026 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A= 2022-09-12T19:40:39.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A=', 'Content-Length': '0'} 2022-09-12T19:40:39.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a2f44324191f7cb-00631f8b37-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.031 DEBUG:boto:b'jdpiak-69d1000falsed2022-09-12T19:39:54.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:39.032 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:39.033 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:39.033 DEBUG:boto:Method: HEAD 2022-09-12T19:40:39.034 DEBUG:boto:Path: /jdpiak-69/d 2022-09-12T19:40:39.034 DEBUG:boto:Data: 2022-09-12T19:40:39.035 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.036 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.036 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.037 DEBUG:boto:Params: {} 2022-09-12T19:40:39.038 DEBUG:boto:Token: None 2022-09-12T19:40:39.038 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/d 2022-09-12T19:40:39.039 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:099gz3OUSImuDBuY1NocxAP+QyE= 2022-09-12T19:40:39.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:099gz3OUSImuDBuY1NocxAP+QyE=', 'Content-Length': '0'} 2022-09-12T19:40:39.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030170adcb138addc-00631f8b37-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.042 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:39.043 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:39.043 DEBUG:boto:Method: HEAD 2022-09-12T19:40:39.044 DEBUG:boto:Path: /jdpiak-69/d 2022-09-12T19:40:39.045 DEBUG:boto:Data: 2022-09-12T19:40:39.045 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.046 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:39.046 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.047 DEBUG:boto:Params: {} 2022-09-12T19:40:39.048 DEBUG:boto:Token: None 2022-09-12T19:40:39.048 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/d 2022-09-12T19:40:39.049 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:099gz3OUSImuDBuY1NocxAP+QyE= 2022-09-12T19:40:39.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:099gz3OUSImuDBuY1NocxAP+QyE=', 'Content-Length': '0'} 2022-09-12T19:40:39.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb079e4776362acb-00631f8b37-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:40:39.054 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:39.055 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:39.056 DEBUG:boto:Method: GET 2022-09-12T19:40:39.057 DEBUG:boto:Path: /jdpiak-69/d 2022-09-12T19:40:39.058 DEBUG:boto:Data: 2022-09-12T19:40:39.059 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.059 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.060 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.060 DEBUG:boto:Params: {} 2022-09-12T19:40:39.061 DEBUG:boto:Token: None 2022-09-12T19:40:39.063 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/d 2022-09-12T19:40:39.064 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/hSD1uOyC+cEgzPvhEL6FvQHQsA= 2022-09-12T19:40:39.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/hSD1uOyC+cEgzPvhEL6FvQHQsA=', 'Content-Length': '0'} 2022-09-12T19:40:39.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065be5f8ad3ed2e23-00631f8b37-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.069 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:39.070 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:39.071 DEBUG:boto:Method: GET 2022-09-12T19:40:39.071 DEBUG:boto:Path: /jdpiak-69/d 2022-09-12T19:40:39.072 DEBUG:boto:Data: 2022-09-12T19:40:39.073 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.073 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:39.074 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.074 DEBUG:boto:Params: {} 2022-09-12T19:40:39.075 DEBUG:boto:Token: None 2022-09-12T19:40:39.076 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/d 2022-09-12T19:40:39.078 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/hSD1uOyC+cEgzPvhEL6FvQHQsA= 2022-09-12T19:40:39.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/hSD1uOyC+cEgzPvhEL6FvQHQsA=', 'Content-Length': '0'} 2022-09-12T19:40:39.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000672ae5e2c60b0d09-00631f8b37-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.122 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:39.123 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:39.123 DEBUG:boto:path=/jdpiak-69/?prefix=d 2022-09-12T19:40:39.124 DEBUG:boto:auth_path=/jdpiak-69/?prefix=d 2022-09-12T19:40:39.125 DEBUG:boto:Method: GET 2022-09-12T19:40:39.125 DEBUG:boto:Path: /jdpiak-69/?prefix=d 2022-09-12T19:40:39.126 DEBUG:boto:Data: 2022-09-12T19:40:39.126 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.127 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.127 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.128 DEBUG:boto:Params: {} 2022-09-12T19:40:39.130 DEBUG:boto:Token: None 2022-09-12T19:40:39.131 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/ 2022-09-12T19:40:39.132 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A= 2022-09-12T19:40:39.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A=', 'Content-Length': '0'} 2022-09-12T19:40:39.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a0533b5aef770f3-00631f8b37-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.138 DEBUG:boto:b'jdpiak-69d1000falsed2022-09-12T19:39:54.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:39.139 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:39.141 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:39.141 DEBUG:boto:path=/jdpiak-69/?prefix=d 2022-09-12T19:40:39.142 DEBUG:boto:auth_path=/jdpiak-69/?prefix=d 2022-09-12T19:40:39.142 DEBUG:boto:Method: GET 2022-09-12T19:40:39.143 DEBUG:boto:Path: /jdpiak-69/?prefix=d 2022-09-12T19:40:39.143 DEBUG:boto:Data: 2022-09-12T19:40:39.144 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.144 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:39.145 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.146 DEBUG:boto:Params: {} 2022-09-12T19:40:39.146 DEBUG:boto:Token: None 2022-09-12T19:40:39.147 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/ 2022-09-12T19:40:39.148 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A= 2022-09-12T19:40:39.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A=', 'Content-Length': '0'} 2022-09-12T19:40:39.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc2f02b673fe0110-00631f8b37-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.152 DEBUG:boto:b'jdpiak-69d1000falsed2022-09-12T19:39:54.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:39.153 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-69 zones={test-zone3, test-zone1} 2022-09-12T19:40:39.154 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-69 zones={test-zone3, test-zone2} 2022-09-12T19:40:39.154 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:39.155 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:39.155 DEBUG:boto:Method: HEAD 2022-09-12T19:40:39.156 DEBUG:boto:Path: /jdpiak-69/ 2022-09-12T19:40:39.156 DEBUG:boto:Data: 2022-09-12T19:40:39.157 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.158 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.158 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.159 DEBUG:boto:Params: {} 2022-09-12T19:40:39.159 DEBUG:boto:Token: None 2022-09-12T19:40:39.160 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/ 2022-09-12T19:40:39.161 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VMtwTRRUgkDzHfs8iIFe6rcSpoA= 2022-09-12T19:40:39.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VMtwTRRUgkDzHfs8iIFe6rcSpoA=', 'Content-Length': '0'} 2022-09-12T19:40:39.165 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000665305688fc5bae8-00631f8b37-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.166 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:39.166 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:39.167 DEBUG:boto:Method: HEAD 2022-09-12T19:40:39.167 DEBUG:boto:Path: /jdpiak-69/ 2022-09-12T19:40:39.168 DEBUG:boto:Data: 2022-09-12T19:40:39.169 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.170 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:39.170 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.171 DEBUG:boto:Params: {} 2022-09-12T19:40:39.172 DEBUG:boto:Token: None 2022-09-12T19:40:39.173 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/ 2022-09-12T19:40:39.173 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VMtwTRRUgkDzHfs8iIFe6rcSpoA= 2022-09-12T19:40:39.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VMtwTRRUgkDzHfs8iIFe6rcSpoA=', 'Content-Length': '0'} 2022-09-12T19:40:39.177 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf45197d4607fbdf-00631f8b37-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.178 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:40:39.178 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:39.179 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:39.179 DEBUG:boto:path=/jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:39.180 DEBUG:boto:auth_path=/jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:39.181 DEBUG:boto:Method: GET 2022-09-12T19:40:39.181 DEBUG:boto:Path: /jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:39.182 DEBUG:boto:Data: 2022-09-12T19:40:39.183 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.184 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.184 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.185 DEBUG:boto:Params: {} 2022-09-12T19:40:39.186 DEBUG:boto:Token: None 2022-09-12T19:40:39.186 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/?versions 2022-09-12T19:40:39.187 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8YStvC5/ryBhhzvfvE9/kRhVroc= 2022-09-12T19:40:39.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8YStvC5/ryBhhzvfvE9/kRhVroc=', 'Content-Length': '0'} 2022-09-12T19:40:39.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000088971db4cfe44b5-00631f8b37-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.193 DEBUG:boto:b'jdpiak-69999falseanulltrue2022-09-12T19:39:54.609Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:39:54.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:39:54.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:39:54.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:39.194 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:40:39.194 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:40:39.195 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:40:39.195 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:40:39.196 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:40:39.197 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:39.198 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:39.199 DEBUG:boto:path=/jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:39.200 DEBUG:boto:auth_path=/jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:39.200 DEBUG:boto:Method: GET 2022-09-12T19:40:39.201 DEBUG:boto:Path: /jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:39.201 DEBUG:boto:Data: 2022-09-12T19:40:39.202 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.203 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:39.204 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.204 DEBUG:boto:Params: {} 2022-09-12T19:40:39.205 DEBUG:boto:Token: None 2022-09-12T19:40:39.205 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/?versions 2022-09-12T19:40:39.206 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8YStvC5/ryBhhzvfvE9/kRhVroc= 2022-09-12T19:40:39.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8YStvC5/ryBhhzvfvE9/kRhVroc=', 'Content-Length': '0'} 2022-09-12T19:40:39.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ded525b17c10e7c-00631f8b37-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.210 DEBUG:boto:b'jdpiak-69999falseanulltrue2022-09-12T19:39:54.609Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:39:54.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:39:54.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:39:54.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:39.211 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:40:39.212 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:40:39.212 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:40:39.213 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:40:39.213 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:39.214 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:39.215 DEBUG:boto:path=/jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:39.215 DEBUG:boto:auth_path=/jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:39.216 DEBUG:boto:Method: GET 2022-09-12T19:40:39.216 DEBUG:boto:Path: /jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:39.217 DEBUG:boto:Data: 2022-09-12T19:40:39.218 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.218 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.219 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.219 DEBUG:boto:Params: {} 2022-09-12T19:40:39.220 DEBUG:boto:Token: None 2022-09-12T19:40:39.220 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/?versions 2022-09-12T19:40:39.221 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8YStvC5/ryBhhzvfvE9/kRhVroc= 2022-09-12T19:40:39.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8YStvC5/ryBhhzvfvE9/kRhVroc=', 'Content-Length': '0'} 2022-09-12T19:40:39.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036b00796e2c5320a-00631f8b37-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.225 DEBUG:boto:b'jdpiak-69999falseanulltrue2022-09-12T19:39:54.609Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:39:54.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:39:54.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:39:54.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:39.227 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:39.227 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:39.228 DEBUG:boto:path=/jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:39.228 DEBUG:boto:auth_path=/jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:39.229 DEBUG:boto:Method: GET 2022-09-12T19:40:39.229 DEBUG:boto:Path: /jdpiak-69/?versions&max-keys=999 2022-09-12T19:40:39.230 DEBUG:boto:Data: 2022-09-12T19:40:39.230 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.231 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:39.232 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.232 DEBUG:boto:Params: {} 2022-09-12T19:40:39.233 DEBUG:boto:Token: None 2022-09-12T19:40:39.233 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/?versions 2022-09-12T19:40:39.234 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8YStvC5/ryBhhzvfvE9/kRhVroc= 2022-09-12T19:40:39.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8YStvC5/ryBhhzvfvE9/kRhVroc=', 'Content-Length': '0'} 2022-09-12T19:40:39.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c26131aaec85f2a7-00631f8b37-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.240 DEBUG:boto:b'jdpiak-69999falseanulltrue2022-09-12T19:39:54.609Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:39:54.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:39:54.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:39:54.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:39.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:40:39.242 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:39.243 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:39.243 DEBUG:boto:path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:39.244 DEBUG:boto:auth_path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:39.245 DEBUG:boto:Method: GET 2022-09-12T19:40:39.246 DEBUG:boto:Path: /jdpiak-69/a?versionId=null 2022-09-12T19:40:39.246 DEBUG:boto:Data: 2022-09-12T19:40:39.247 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.247 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.248 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.249 DEBUG:boto:Params: {} 2022-09-12T19:40:39.249 DEBUG:boto:Token: None 2022-09-12T19:40:39.250 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/a?versionId=null 2022-09-12T19:40:39.251 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ctdig10XBfJcOWmaT89aWSvzWMQ= 2022-09-12T19:40:39.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ctdig10XBfJcOWmaT89aWSvzWMQ=', 'Content-Length': '0'} 2022-09-12T19:40:39.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcfd2c61eee86d7e-00631f8b37-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.298 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:39.299 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:39.300 DEBUG:boto:path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:39.302 DEBUG:boto:auth_path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:39.303 DEBUG:boto:Method: GET 2022-09-12T19:40:39.303 DEBUG:boto:Path: /jdpiak-69/a?versionId=null 2022-09-12T19:40:39.304 DEBUG:boto:Data: 2022-09-12T19:40:39.305 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.307 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:39.308 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.309 DEBUG:boto:Params: {} 2022-09-12T19:40:39.310 DEBUG:boto:Token: None 2022-09-12T19:40:39.312 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/a?versionId=null 2022-09-12T19:40:39.313 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ctdig10XBfJcOWmaT89aWSvzWMQ= 2022-09-12T19:40:39.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ctdig10XBfJcOWmaT89aWSvzWMQ=', 'Content-Length': '0'} 2022-09-12T19:40:39.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001149b0fc34de7039-00631f8b37-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.317 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:39.318 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:39.319 DEBUG:boto:path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:39.319 DEBUG:boto:auth_path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:39.320 DEBUG:boto:Method: HEAD 2022-09-12T19:40:39.320 DEBUG:boto:Path: /jdpiak-69/a?versionId=null 2022-09-12T19:40:39.321 DEBUG:boto:Data: 2022-09-12T19:40:39.321 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.322 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.323 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.323 DEBUG:boto:Params: {} 2022-09-12T19:40:39.324 DEBUG:boto:Token: None 2022-09-12T19:40:39.325 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/a?versionId=null 2022-09-12T19:40:39.325 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kCGlGk/wHshoEDtcGUs0Aim41b4= 2022-09-12T19:40:39.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kCGlGk/wHshoEDtcGUs0Aim41b4=', 'Content-Length': '0'} 2022-09-12T19:40:39.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed2025067e40ccbe-00631f8b37-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.329 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:39.330 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:39.330 DEBUG:boto:path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:39.331 DEBUG:boto:auth_path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:39.331 DEBUG:boto:Method: HEAD 2022-09-12T19:40:39.332 DEBUG:boto:Path: /jdpiak-69/a?versionId=null 2022-09-12T19:40:39.332 DEBUG:boto:Data: 2022-09-12T19:40:39.333 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.333 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:39.334 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.335 DEBUG:boto:Params: {} 2022-09-12T19:40:39.335 DEBUG:boto:Token: None 2022-09-12T19:40:39.336 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/a?versionId=null 2022-09-12T19:40:39.336 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kCGlGk/wHshoEDtcGUs0Aim41b4= 2022-09-12T19:40:39.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kCGlGk/wHshoEDtcGUs0Aim41b4=', 'Content-Length': '0'} 2022-09-12T19:40:39.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd9543732eac698a-00631f8b37-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:40:39.340 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:39.341 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:39.342 DEBUG:boto:path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:39.342 DEBUG:boto:auth_path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:39.343 DEBUG:boto:Method: GET 2022-09-12T19:40:39.343 DEBUG:boto:Path: /jdpiak-69/a?versionId=null 2022-09-12T19:40:39.344 DEBUG:boto:Data: 2022-09-12T19:40:39.344 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.345 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.346 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.347 DEBUG:boto:Params: {} 2022-09-12T19:40:39.348 DEBUG:boto:Token: None 2022-09-12T19:40:39.349 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/a?versionId=null 2022-09-12T19:40:39.350 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ctdig10XBfJcOWmaT89aWSvzWMQ= 2022-09-12T19:40:39.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ctdig10XBfJcOWmaT89aWSvzWMQ=', 'Content-Length': '0'} 2022-09-12T19:40:39.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7b5a6990ec64372-00631f8b37-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.354 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:39.355 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:39.355 DEBUG:boto:path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:39.356 DEBUG:boto:auth_path=/jdpiak-69/a?versionId=null 2022-09-12T19:40:39.356 DEBUG:boto:Method: GET 2022-09-12T19:40:39.357 DEBUG:boto:Path: /jdpiak-69/a?versionId=null 2022-09-12T19:40:39.357 DEBUG:boto:Data: 2022-09-12T19:40:39.358 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.358 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:39.359 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.359 DEBUG:boto:Params: {} 2022-09-12T19:40:39.360 DEBUG:boto:Token: None 2022-09-12T19:40:39.361 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/a?versionId=null 2022-09-12T19:40:39.361 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ctdig10XBfJcOWmaT89aWSvzWMQ= 2022-09-12T19:40:39.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ctdig10XBfJcOWmaT89aWSvzWMQ=', 'Content-Length': '0'} 2022-09-12T19:40:39.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b259179ca4d8acdd-00631f8b37-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.365 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:39.365 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:39.366 DEBUG:boto:path=/jdpiak-69/?prefix=a 2022-09-12T19:40:39.367 DEBUG:boto:auth_path=/jdpiak-69/?prefix=a 2022-09-12T19:40:39.368 DEBUG:boto:Method: GET 2022-09-12T19:40:39.369 DEBUG:boto:Path: /jdpiak-69/?prefix=a 2022-09-12T19:40:39.370 DEBUG:boto:Data: 2022-09-12T19:40:39.371 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.371 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.372 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.373 DEBUG:boto:Params: {} 2022-09-12T19:40:39.374 DEBUG:boto:Token: None 2022-09-12T19:40:39.376 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/ 2022-09-12T19:40:39.377 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A= 2022-09-12T19:40:39.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A=', 'Content-Length': '0'} 2022-09-12T19:40:39.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e267bbfb8731c391-00631f8b37-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.381 DEBUG:boto:b'jdpiak-69a1000falsea2022-09-12T19:39:54.609Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:39.383 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:39.383 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:39.384 DEBUG:boto:path=/jdpiak-69/?prefix=a 2022-09-12T19:40:39.384 DEBUG:boto:auth_path=/jdpiak-69/?prefix=a 2022-09-12T19:40:39.385 DEBUG:boto:Method: GET 2022-09-12T19:40:39.385 DEBUG:boto:Path: /jdpiak-69/?prefix=a 2022-09-12T19:40:39.386 DEBUG:boto:Data: 2022-09-12T19:40:39.386 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.387 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:39.388 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.388 DEBUG:boto:Params: {} 2022-09-12T19:40:39.389 DEBUG:boto:Token: None 2022-09-12T19:40:39.389 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/ 2022-09-12T19:40:39.390 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A= 2022-09-12T19:40:39.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A=', 'Content-Length': '0'} 2022-09-12T19:40:39.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4e5eadb71a6fcb1-00631f8b37-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.394 DEBUG:boto:b'jdpiak-69a1000falsea2022-09-12T19:39:54.609Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:39.395 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:39.396 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:39.396 DEBUG:boto:Method: HEAD 2022-09-12T19:40:39.397 DEBUG:boto:Path: /jdpiak-69/a 2022-09-12T19:40:39.397 DEBUG:boto:Data: 2022-09-12T19:40:39.398 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.398 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.399 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.399 DEBUG:boto:Params: {} 2022-09-12T19:40:39.400 DEBUG:boto:Token: None 2022-09-12T19:40:39.401 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/a 2022-09-12T19:40:39.401 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vC8U2+L1QUACjOkrn4w6dXQH7fY= 2022-09-12T19:40:39.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vC8U2+L1QUACjOkrn4w6dXQH7fY=', 'Content-Length': '0'} 2022-09-12T19:40:39.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b1e720f0b2c5b74-00631f8b37-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.405 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:39.405 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:39.406 DEBUG:boto:Method: HEAD 2022-09-12T19:40:39.406 DEBUG:boto:Path: /jdpiak-69/a 2022-09-12T19:40:39.407 DEBUG:boto:Data: 2022-09-12T19:40:39.407 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.408 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:39.408 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.409 DEBUG:boto:Params: {} 2022-09-12T19:40:39.410 DEBUG:boto:Token: None 2022-09-12T19:40:39.410 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/a 2022-09-12T19:40:39.411 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vC8U2+L1QUACjOkrn4w6dXQH7fY= 2022-09-12T19:40:39.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vC8U2+L1QUACjOkrn4w6dXQH7fY=', 'Content-Length': '0'} 2022-09-12T19:40:39.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e56a31e2a83d106d-00631f8b37-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:40:39.415 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:39.415 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:39.416 DEBUG:boto:Method: GET 2022-09-12T19:40:39.416 DEBUG:boto:Path: /jdpiak-69/a 2022-09-12T19:40:39.417 DEBUG:boto:Data: 2022-09-12T19:40:39.418 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.418 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.419 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.419 DEBUG:boto:Params: {} 2022-09-12T19:40:39.420 DEBUG:boto:Token: None 2022-09-12T19:40:39.421 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/a 2022-09-12T19:40:39.421 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IcZ3/soeXsvBecaoJcUbBomTgnM= 2022-09-12T19:40:39.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IcZ3/soeXsvBecaoJcUbBomTgnM=', 'Content-Length': '0'} 2022-09-12T19:40:39.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcbd08e24a344e5a-00631f8b37-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.426 DEBUG:boto:path=/jdpiak-69/a 2022-09-12T19:40:39.427 DEBUG:boto:auth_path=/jdpiak-69/a 2022-09-12T19:40:39.427 DEBUG:boto:Method: GET 2022-09-12T19:40:39.428 DEBUG:boto:Path: /jdpiak-69/a 2022-09-12T19:40:39.429 DEBUG:boto:Data: 2022-09-12T19:40:39.429 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.430 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:39.430 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.431 DEBUG:boto:Params: {} 2022-09-12T19:40:39.431 DEBUG:boto:Token: None 2022-09-12T19:40:39.432 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/a 2022-09-12T19:40:39.433 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IcZ3/soeXsvBecaoJcUbBomTgnM= 2022-09-12T19:40:39.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IcZ3/soeXsvBecaoJcUbBomTgnM=', 'Content-Length': '0'} 2022-09-12T19:40:39.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ad9506af6fa6a4e-00631f8b37-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.442 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:39.442 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:39.443 DEBUG:boto:path=/jdpiak-69/?prefix=a 2022-09-12T19:40:39.443 DEBUG:boto:auth_path=/jdpiak-69/?prefix=a 2022-09-12T19:40:39.444 DEBUG:boto:Method: GET 2022-09-12T19:40:39.445 DEBUG:boto:Path: /jdpiak-69/?prefix=a 2022-09-12T19:40:39.445 DEBUG:boto:Data: 2022-09-12T19:40:39.446 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.446 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.447 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.447 DEBUG:boto:Params: {} 2022-09-12T19:40:39.448 DEBUG:boto:Token: None 2022-09-12T19:40:39.448 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/ 2022-09-12T19:40:39.449 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A= 2022-09-12T19:40:39.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A=', 'Content-Length': '0'} 2022-09-12T19:40:39.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e17ef81127a82159-00631f8b37-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.453 DEBUG:boto:b'jdpiak-69a1000falsea2022-09-12T19:39:54.609Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:39.454 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:39.455 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:39.456 DEBUG:boto:path=/jdpiak-69/?prefix=a 2022-09-12T19:40:39.456 DEBUG:boto:auth_path=/jdpiak-69/?prefix=a 2022-09-12T19:40:39.457 DEBUG:boto:Method: GET 2022-09-12T19:40:39.457 DEBUG:boto:Path: /jdpiak-69/?prefix=a 2022-09-12T19:40:39.458 DEBUG:boto:Data: 2022-09-12T19:40:39.459 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.459 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:39.460 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.460 DEBUG:boto:Params: {} 2022-09-12T19:40:39.461 DEBUG:boto:Token: None 2022-09-12T19:40:39.461 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/ 2022-09-12T19:40:39.462 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A= 2022-09-12T19:40:39.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A=', 'Content-Length': '0'} 2022-09-12T19:40:39.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005227c15a17704ce1-00631f8b37-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.466 DEBUG:boto:b'jdpiak-69a1000falsea2022-09-12T19:39:54.609Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:39.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:40:39.468 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:39.468 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:39.469 DEBUG:boto:path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:39.469 DEBUG:boto:auth_path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:39.470 DEBUG:boto:Method: GET 2022-09-12T19:40:39.471 DEBUG:boto:Path: /jdpiak-69/b?versionId=null 2022-09-12T19:40:39.471 DEBUG:boto:Data: 2022-09-12T19:40:39.472 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.472 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.473 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.473 DEBUG:boto:Params: {} 2022-09-12T19:40:39.474 DEBUG:boto:Token: None 2022-09-12T19:40:39.475 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/b?versionId=null 2022-09-12T19:40:39.475 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ujuCMekD0yAl5Jw3plI9qwcJqLA= 2022-09-12T19:40:39.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ujuCMekD0yAl5Jw3plI9qwcJqLA=', 'Content-Length': '0'} 2022-09-12T19:40:39.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc68aef287cb780b-00631f8b37-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.484 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:39.485 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:39.485 DEBUG:boto:path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:39.486 DEBUG:boto:auth_path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:39.487 DEBUG:boto:Method: GET 2022-09-12T19:40:39.488 DEBUG:boto:Path: /jdpiak-69/b?versionId=null 2022-09-12T19:40:39.490 DEBUG:boto:Data: 2022-09-12T19:40:39.491 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.492 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:39.493 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.495 DEBUG:boto:Params: {} 2022-09-12T19:40:39.496 DEBUG:boto:Token: None 2022-09-12T19:40:39.498 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/b?versionId=null 2022-09-12T19:40:39.499 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ujuCMekD0yAl5Jw3plI9qwcJqLA= 2022-09-12T19:40:39.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ujuCMekD0yAl5Jw3plI9qwcJqLA=', 'Content-Length': '0'} 2022-09-12T19:40:39.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ae657b6c5797425-00631f8b37-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.504 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:39.505 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:39.506 DEBUG:boto:path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:39.506 DEBUG:boto:auth_path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:39.507 DEBUG:boto:Method: HEAD 2022-09-12T19:40:39.507 DEBUG:boto:Path: /jdpiak-69/b?versionId=null 2022-09-12T19:40:39.508 DEBUG:boto:Data: 2022-09-12T19:40:39.509 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.509 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.511 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.512 DEBUG:boto:Params: {} 2022-09-12T19:40:39.512 DEBUG:boto:Token: None 2022-09-12T19:40:39.513 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/b?versionId=null 2022-09-12T19:40:39.514 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+T7RUMz/+umPv184WPHb9Ky9Z4Y= 2022-09-12T19:40:39.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+T7RUMz/+umPv184WPHb9Ky9Z4Y=', 'Content-Length': '0'} 2022-09-12T19:40:39.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062c2c6d0b80bd750-00631f8b37-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.517 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:39.518 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:39.519 DEBUG:boto:path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:39.519 DEBUG:boto:auth_path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:39.520 DEBUG:boto:Method: HEAD 2022-09-12T19:40:39.520 DEBUG:boto:Path: /jdpiak-69/b?versionId=null 2022-09-12T19:40:39.521 DEBUG:boto:Data: 2022-09-12T19:40:39.521 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.522 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:39.522 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.523 DEBUG:boto:Params: {} 2022-09-12T19:40:39.524 DEBUG:boto:Token: None 2022-09-12T19:40:39.524 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/b?versionId=null 2022-09-12T19:40:39.525 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+T7RUMz/+umPv184WPHb9Ky9Z4Y= 2022-09-12T19:40:39.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+T7RUMz/+umPv184WPHb9Ky9Z4Y=', 'Content-Length': '0'} 2022-09-12T19:40:39.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046843e4b0e1bb19b-00631f8b37-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:40:39.529 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:39.529 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:39.530 DEBUG:boto:path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:39.531 DEBUG:boto:auth_path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:39.531 DEBUG:boto:Method: GET 2022-09-12T19:40:39.532 DEBUG:boto:Path: /jdpiak-69/b?versionId=null 2022-09-12T19:40:39.532 DEBUG:boto:Data: 2022-09-12T19:40:39.533 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.534 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.534 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.535 DEBUG:boto:Params: {} 2022-09-12T19:40:39.535 DEBUG:boto:Token: None 2022-09-12T19:40:39.536 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/b?versionId=null 2022-09-12T19:40:39.537 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ujuCMekD0yAl5Jw3plI9qwcJqLA= 2022-09-12T19:40:39.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ujuCMekD0yAl5Jw3plI9qwcJqLA=', 'Content-Length': '0'} 2022-09-12T19:40:39.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005457137af024efa9-00631f8b37-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.540 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:39.541 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:39.542 DEBUG:boto:path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:39.542 DEBUG:boto:auth_path=/jdpiak-69/b?versionId=null 2022-09-12T19:40:39.543 DEBUG:boto:Method: GET 2022-09-12T19:40:39.543 DEBUG:boto:Path: /jdpiak-69/b?versionId=null 2022-09-12T19:40:39.544 DEBUG:boto:Data: 2022-09-12T19:40:39.544 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.545 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:39.546 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.546 DEBUG:boto:Params: {} 2022-09-12T19:40:39.547 DEBUG:boto:Token: None 2022-09-12T19:40:39.548 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/b?versionId=null 2022-09-12T19:40:39.549 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ujuCMekD0yAl5Jw3plI9qwcJqLA= 2022-09-12T19:40:39.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ujuCMekD0yAl5Jw3plI9qwcJqLA=', 'Content-Length': '0'} 2022-09-12T19:40:39.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bd05ceb29c9c9f1-00631f8b37-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.600 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:39.600 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:39.601 DEBUG:boto:path=/jdpiak-69/?prefix=b 2022-09-12T19:40:39.601 DEBUG:boto:auth_path=/jdpiak-69/?prefix=b 2022-09-12T19:40:39.602 DEBUG:boto:Method: GET 2022-09-12T19:40:39.602 DEBUG:boto:Path: /jdpiak-69/?prefix=b 2022-09-12T19:40:39.603 DEBUG:boto:Data: 2022-09-12T19:40:39.604 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.604 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.605 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.606 DEBUG:boto:Params: {} 2022-09-12T19:40:39.606 DEBUG:boto:Token: None 2022-09-12T19:40:39.607 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/ 2022-09-12T19:40:39.608 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A= 2022-09-12T19:40:39.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A=', 'Content-Length': '0'} 2022-09-12T19:40:39.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e936bf55edc01236-00631f8b37-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.612 DEBUG:boto:b'jdpiak-69b1000falseb2022-09-12T19:39:54.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:39.615 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:39.615 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:39.616 DEBUG:boto:path=/jdpiak-69/?prefix=b 2022-09-12T19:40:39.616 DEBUG:boto:auth_path=/jdpiak-69/?prefix=b 2022-09-12T19:40:39.617 DEBUG:boto:Method: GET 2022-09-12T19:40:39.617 DEBUG:boto:Path: /jdpiak-69/?prefix=b 2022-09-12T19:40:39.619 DEBUG:boto:Data: 2022-09-12T19:40:39.620 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.621 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:39.621 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.622 DEBUG:boto:Params: {} 2022-09-12T19:40:39.623 DEBUG:boto:Token: None 2022-09-12T19:40:39.624 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/ 2022-09-12T19:40:39.625 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A= 2022-09-12T19:40:39.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A=', 'Content-Length': '0'} 2022-09-12T19:40:39.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ebf113b490d20f2-00631f8b37-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.631 DEBUG:boto:b'jdpiak-69b1000falseb2022-09-12T19:39:54.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:39.633 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:39.634 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:39.635 DEBUG:boto:Method: HEAD 2022-09-12T19:40:39.636 DEBUG:boto:Path: /jdpiak-69/b 2022-09-12T19:40:39.636 DEBUG:boto:Data: 2022-09-12T19:40:39.637 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.637 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.638 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.639 DEBUG:boto:Params: {} 2022-09-12T19:40:39.640 DEBUG:boto:Token: None 2022-09-12T19:40:39.640 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/b 2022-09-12T19:40:39.641 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2yrPXRtfh7FREZSJmw16l/XB00I= 2022-09-12T19:40:39.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2yrPXRtfh7FREZSJmw16l/XB00I=', 'Content-Length': '0'} 2022-09-12T19:40:39.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c111310b1b1424b4-00631f8b37-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.645 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:39.645 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:39.646 DEBUG:boto:Method: HEAD 2022-09-12T19:40:39.646 DEBUG:boto:Path: /jdpiak-69/b 2022-09-12T19:40:39.647 DEBUG:boto:Data: 2022-09-12T19:40:39.647 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.648 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:39.649 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.649 DEBUG:boto:Params: {} 2022-09-12T19:40:39.650 DEBUG:boto:Token: None 2022-09-12T19:40:39.650 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/b 2022-09-12T19:40:39.651 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2yrPXRtfh7FREZSJmw16l/XB00I= 2022-09-12T19:40:39.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2yrPXRtfh7FREZSJmw16l/XB00I=', 'Content-Length': '0'} 2022-09-12T19:40:39.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fc3987be5923c22-00631f8b37-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:40:39.659 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:39.659 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:39.660 DEBUG:boto:Method: GET 2022-09-12T19:40:39.660 DEBUG:boto:Path: /jdpiak-69/b 2022-09-12T19:40:39.661 DEBUG:boto:Data: 2022-09-12T19:40:39.662 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.662 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.663 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.663 DEBUG:boto:Params: {} 2022-09-12T19:40:39.664 DEBUG:boto:Token: None 2022-09-12T19:40:39.665 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/b 2022-09-12T19:40:39.665 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WhkEN4Q1wmrpaJ0YSZGUZvjXgpA= 2022-09-12T19:40:39.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WhkEN4Q1wmrpaJ0YSZGUZvjXgpA=', 'Content-Length': '0'} 2022-09-12T19:40:39.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8928861d629636f-00631f8b37-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.726 DEBUG:boto:path=/jdpiak-69/b 2022-09-12T19:40:39.728 DEBUG:boto:auth_path=/jdpiak-69/b 2022-09-12T19:40:39.729 DEBUG:boto:Method: GET 2022-09-12T19:40:39.730 DEBUG:boto:Path: /jdpiak-69/b 2022-09-12T19:40:39.731 DEBUG:boto:Data: 2022-09-12T19:40:39.732 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.733 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:39.734 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.735 DEBUG:boto:Params: {} 2022-09-12T19:40:39.736 DEBUG:boto:Token: None 2022-09-12T19:40:39.737 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/b 2022-09-12T19:40:39.739 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WhkEN4Q1wmrpaJ0YSZGUZvjXgpA= 2022-09-12T19:40:39.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WhkEN4Q1wmrpaJ0YSZGUZvjXgpA=', 'Content-Length': '0'} 2022-09-12T19:40:39.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1f6d34ebf7e4c2a-00631f8b37-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.744 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:39.745 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:39.746 DEBUG:boto:path=/jdpiak-69/?prefix=b 2022-09-12T19:40:39.747 DEBUG:boto:auth_path=/jdpiak-69/?prefix=b 2022-09-12T19:40:39.748 DEBUG:boto:Method: GET 2022-09-12T19:40:39.749 DEBUG:boto:Path: /jdpiak-69/?prefix=b 2022-09-12T19:40:39.750 DEBUG:boto:Data: 2022-09-12T19:40:39.751 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.752 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.753 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.753 DEBUG:boto:Params: {} 2022-09-12T19:40:39.754 DEBUG:boto:Token: None 2022-09-12T19:40:39.755 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/ 2022-09-12T19:40:39.757 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A= 2022-09-12T19:40:39.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A=', 'Content-Length': '0'} 2022-09-12T19:40:39.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f12ed1faad04cc6-00631f8b37-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.772 DEBUG:boto:b'jdpiak-69b1000falseb2022-09-12T19:39:54.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:39.773 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:39.773 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:39.774 DEBUG:boto:path=/jdpiak-69/?prefix=b 2022-09-12T19:40:39.774 DEBUG:boto:auth_path=/jdpiak-69/?prefix=b 2022-09-12T19:40:39.775 DEBUG:boto:Method: GET 2022-09-12T19:40:39.775 DEBUG:boto:Path: /jdpiak-69/?prefix=b 2022-09-12T19:40:39.776 DEBUG:boto:Data: 2022-09-12T19:40:39.777 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.778 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:39.779 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.780 DEBUG:boto:Params: {} 2022-09-12T19:40:39.781 DEBUG:boto:Token: None 2022-09-12T19:40:39.782 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/ 2022-09-12T19:40:39.782 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A= 2022-09-12T19:40:39.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A=', 'Content-Length': '0'} 2022-09-12T19:40:39.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008071ce07dd55b3f5-00631f8b37-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.788 DEBUG:boto:b'jdpiak-69b1000falseb2022-09-12T19:39:54.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:39.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:40:39.791 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:39.792 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:39.793 DEBUG:boto:path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:39.794 DEBUG:boto:auth_path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:39.795 DEBUG:boto:Method: GET 2022-09-12T19:40:39.796 DEBUG:boto:Path: /jdpiak-69/c?versionId=null 2022-09-12T19:40:39.797 DEBUG:boto:Data: 2022-09-12T19:40:39.798 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.799 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.800 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.801 DEBUG:boto:Params: {} 2022-09-12T19:40:39.802 DEBUG:boto:Token: None 2022-09-12T19:40:39.803 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/c?versionId=null 2022-09-12T19:40:39.805 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P2KycDU3yP1GeKoeWFkyveGjwbE= 2022-09-12T19:40:39.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P2KycDU3yP1GeKoeWFkyveGjwbE=', 'Content-Length': '0'} 2022-09-12T19:40:39.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d77f402268b992a-00631f8b37-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.816 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:39.817 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:39.818 DEBUG:boto:path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:39.819 DEBUG:boto:auth_path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:39.819 DEBUG:boto:Method: GET 2022-09-12T19:40:39.820 DEBUG:boto:Path: /jdpiak-69/c?versionId=null 2022-09-12T19:40:39.821 DEBUG:boto:Data: 2022-09-12T19:40:39.822 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.823 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:39.824 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.825 DEBUG:boto:Params: {} 2022-09-12T19:40:39.826 DEBUG:boto:Token: None 2022-09-12T19:40:39.827 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/c?versionId=null 2022-09-12T19:40:39.828 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P2KycDU3yP1GeKoeWFkyveGjwbE= 2022-09-12T19:40:39.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P2KycDU3yP1GeKoeWFkyveGjwbE=', 'Content-Length': '0'} 2022-09-12T19:40:39.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4c336978560047c-00631f8b37-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.833 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:39.834 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:39.835 DEBUG:boto:path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:39.835 DEBUG:boto:auth_path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:39.836 DEBUG:boto:Method: HEAD 2022-09-12T19:40:39.837 DEBUG:boto:Path: /jdpiak-69/c?versionId=null 2022-09-12T19:40:39.838 DEBUG:boto:Data: 2022-09-12T19:40:39.839 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.840 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.840 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.841 DEBUG:boto:Params: {} 2022-09-12T19:40:39.842 DEBUG:boto:Token: None 2022-09-12T19:40:39.843 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/c?versionId=null 2022-09-12T19:40:39.844 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OpJYzlXhEgOFKmExF1wNNRDIBFk= 2022-09-12T19:40:39.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OpJYzlXhEgOFKmExF1wNNRDIBFk=', 'Content-Length': '0'} 2022-09-12T19:40:39.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000114c2133f4ff886c-00631f8b37-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.849 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:39.850 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:39.850 DEBUG:boto:path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:39.851 DEBUG:boto:auth_path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:39.851 DEBUG:boto:Method: HEAD 2022-09-12T19:40:39.852 DEBUG:boto:Path: /jdpiak-69/c?versionId=null 2022-09-12T19:40:39.852 DEBUG:boto:Data: 2022-09-12T19:40:39.853 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.854 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:39.854 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.855 DEBUG:boto:Params: {} 2022-09-12T19:40:39.856 DEBUG:boto:Token: None 2022-09-12T19:40:39.856 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/c?versionId=null 2022-09-12T19:40:39.857 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OpJYzlXhEgOFKmExF1wNNRDIBFk= 2022-09-12T19:40:39.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OpJYzlXhEgOFKmExF1wNNRDIBFk=', 'Content-Length': '0'} 2022-09-12T19:40:39.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd5a24e1dcf22556-00631f8b37-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:40:39.861 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:39.861 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:39.862 DEBUG:boto:path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:39.863 DEBUG:boto:auth_path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:39.863 DEBUG:boto:Method: GET 2022-09-12T19:40:39.864 DEBUG:boto:Path: /jdpiak-69/c?versionId=null 2022-09-12T19:40:39.864 DEBUG:boto:Data: 2022-09-12T19:40:39.865 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.865 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.866 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.866 DEBUG:boto:Params: {} 2022-09-12T19:40:39.867 DEBUG:boto:Token: None 2022-09-12T19:40:39.868 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/c?versionId=null 2022-09-12T19:40:39.868 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P2KycDU3yP1GeKoeWFkyveGjwbE= 2022-09-12T19:40:39.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P2KycDU3yP1GeKoeWFkyveGjwbE=', 'Content-Length': '0'} 2022-09-12T19:40:39.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040de25ca6d8e9385-00631f8b37-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.918 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:39.919 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:39.920 DEBUG:boto:path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:39.921 DEBUG:boto:auth_path=/jdpiak-69/c?versionId=null 2022-09-12T19:40:39.922 DEBUG:boto:Method: GET 2022-09-12T19:40:39.923 DEBUG:boto:Path: /jdpiak-69/c?versionId=null 2022-09-12T19:40:39.924 DEBUG:boto:Data: 2022-09-12T19:40:39.925 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.926 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:39.927 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.928 DEBUG:boto:Params: {} 2022-09-12T19:40:39.929 DEBUG:boto:Token: None 2022-09-12T19:40:39.931 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/c?versionId=null 2022-09-12T19:40:39.932 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P2KycDU3yP1GeKoeWFkyveGjwbE= 2022-09-12T19:40:39.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P2KycDU3yP1GeKoeWFkyveGjwbE=', 'Content-Length': '0'} 2022-09-12T19:40:39.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000026008a8d430fe4a-00631f8b37-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.938 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:39.939 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:39.940 DEBUG:boto:path=/jdpiak-69/?prefix=c 2022-09-12T19:40:39.941 DEBUG:boto:auth_path=/jdpiak-69/?prefix=c 2022-09-12T19:40:39.942 DEBUG:boto:Method: GET 2022-09-12T19:40:39.943 DEBUG:boto:Path: /jdpiak-69/?prefix=c 2022-09-12T19:40:39.944 DEBUG:boto:Data: 2022-09-12T19:40:39.944 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.945 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.945 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.946 DEBUG:boto:Params: {} 2022-09-12T19:40:39.947 DEBUG:boto:Token: None 2022-09-12T19:40:39.948 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/ 2022-09-12T19:40:39.948 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A= 2022-09-12T19:40:39.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A=', 'Content-Length': '0'} 2022-09-12T19:40:39.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002994d834afaa0a9f-00631f8b37-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.954 DEBUG:boto:b'jdpiak-69c1000falsec2022-09-12T19:39:54.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:39.955 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:39.956 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:39.957 DEBUG:boto:path=/jdpiak-69/?prefix=c 2022-09-12T19:40:39.958 DEBUG:boto:auth_path=/jdpiak-69/?prefix=c 2022-09-12T19:40:39.960 DEBUG:boto:Method: GET 2022-09-12T19:40:39.961 DEBUG:boto:Path: /jdpiak-69/?prefix=c 2022-09-12T19:40:39.962 DEBUG:boto:Data: 2022-09-12T19:40:39.963 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.964 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:39.965 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.966 DEBUG:boto:Params: {} 2022-09-12T19:40:39.967 DEBUG:boto:Token: None 2022-09-12T19:40:39.968 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/ 2022-09-12T19:40:39.970 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A= 2022-09-12T19:40:39.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jqKnWs86Kkh1CJC0GT8V2rvJB/A=', 'Content-Length': '0'} 2022-09-12T19:40:39.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e496de14c7446c08-00631f8b37-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.975 DEBUG:boto:b'jdpiak-69c1000falsec2022-09-12T19:39:54.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:39.977 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:39.978 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:39.979 DEBUG:boto:Method: HEAD 2022-09-12T19:40:39.980 DEBUG:boto:Path: /jdpiak-69/c 2022-09-12T19:40:39.981 DEBUG:boto:Data: 2022-09-12T19:40:39.982 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.982 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:39.983 DEBUG:boto:Port: 8001 2022-09-12T19:40:39.984 DEBUG:boto:Params: {} 2022-09-12T19:40:39.985 DEBUG:boto:Token: None 2022-09-12T19:40:39.986 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/c 2022-09-12T19:40:39.987 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wyyy3CsU7ZrcGjDVFOvb9YiRjNw= 2022-09-12T19:40:39.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wyyy3CsU7ZrcGjDVFOvb9YiRjNw=', 'Content-Length': '0'} 2022-09-12T19:40:39.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df6ce4c14e8c9a19-00631f8b37-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:39.993 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:39.994 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:39.994 DEBUG:boto:Method: HEAD 2022-09-12T19:40:39.995 DEBUG:boto:Path: /jdpiak-69/c 2022-09-12T19:40:39.995 DEBUG:boto:Data: 2022-09-12T19:40:39.996 DEBUG:boto:Headers: {} 2022-09-12T19:40:39.996 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:39.997 DEBUG:boto:Port: 8000 2022-09-12T19:40:39.997 DEBUG:boto:Params: {} 2022-09-12T19:40:39.998 DEBUG:boto:Token: None 2022-09-12T19:40:39.999 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:39 GMT /jdpiak-69/c 2022-09-12T19:40:39.999 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wyyy3CsU7ZrcGjDVFOvb9YiRjNw= 2022-09-12T19:40:40.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wyyy3CsU7ZrcGjDVFOvb9YiRjNw=', 'Content-Length': '0'} 2022-09-12T19:40:40.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000557df379aa00ebe8-00631f8b38-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:40:40.003 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:40.004 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:40.004 DEBUG:boto:Method: GET 2022-09-12T19:40:40.005 DEBUG:boto:Path: /jdpiak-69/c 2022-09-12T19:40:40.005 DEBUG:boto:Data: 2022-09-12T19:40:40.006 DEBUG:boto:Headers: {} 2022-09-12T19:40:40.007 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:40.007 DEBUG:boto:Port: 8001 2022-09-12T19:40:40.008 DEBUG:boto:Params: {} 2022-09-12T19:40:40.008 DEBUG:boto:Token: None 2022-09-12T19:40:40.009 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-69/c 2022-09-12T19:40:40.010 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:37E3U0gR9y2UDHrV+uV7bLPaDsA= 2022-09-12T19:40:40.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:37E3U0gR9y2UDHrV+uV7bLPaDsA=', 'Content-Length': '0'} 2022-09-12T19:40:40.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a1c8ca42839317c-00631f8b38-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.013 DEBUG:boto:path=/jdpiak-69/c 2022-09-12T19:40:40.014 DEBUG:boto:auth_path=/jdpiak-69/c 2022-09-12T19:40:40.015 DEBUG:boto:Method: GET 2022-09-12T19:40:40.015 DEBUG:boto:Path: /jdpiak-69/c 2022-09-12T19:40:40.016 DEBUG:boto:Data: 2022-09-12T19:40:40.016 DEBUG:boto:Headers: {} 2022-09-12T19:40:40.017 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:40.017 DEBUG:boto:Port: 8000 2022-09-12T19:40:40.018 DEBUG:boto:Params: {} 2022-09-12T19:40:40.019 DEBUG:boto:Token: None 2022-09-12T19:40:40.019 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-69/c 2022-09-12T19:40:40.020 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:37E3U0gR9y2UDHrV+uV7bLPaDsA= 2022-09-12T19:40:40.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:37E3U0gR9y2UDHrV+uV7bLPaDsA=', 'Content-Length': '0'} 2022-09-12T19:40:40.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006db5c7bf3bc432fc-00631f8b38-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.066 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:40.068 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:40.069 DEBUG:boto:path=/jdpiak-69/?prefix=c 2022-09-12T19:40:40.070 DEBUG:boto:auth_path=/jdpiak-69/?prefix=c 2022-09-12T19:40:40.071 DEBUG:boto:Method: GET 2022-09-12T19:40:40.072 DEBUG:boto:Path: /jdpiak-69/?prefix=c 2022-09-12T19:40:40.073 DEBUG:boto:Data: 2022-09-12T19:40:40.074 DEBUG:boto:Headers: {} 2022-09-12T19:40:40.075 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:40.076 DEBUG:boto:Port: 8001 2022-09-12T19:40:40.077 DEBUG:boto:Params: {} 2022-09-12T19:40:40.078 DEBUG:boto:Token: None 2022-09-12T19:40:40.079 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-69/ 2022-09-12T19:40:40.080 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NvtrUjv3Qou806Ce5prM0PCCmEY= 2022-09-12T19:40:40.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NvtrUjv3Qou806Ce5prM0PCCmEY=', 'Content-Length': '0'} 2022-09-12T19:40:40.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007df8c620b74df1ff-00631f8b38-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.084 DEBUG:boto:b'jdpiak-69c1000falsec2022-09-12T19:39:54.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:40.086 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:40.087 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:40.088 DEBUG:boto:path=/jdpiak-69/?prefix=c 2022-09-12T19:40:40.088 DEBUG:boto:auth_path=/jdpiak-69/?prefix=c 2022-09-12T19:40:40.089 DEBUG:boto:Method: GET 2022-09-12T19:40:40.089 DEBUG:boto:Path: /jdpiak-69/?prefix=c 2022-09-12T19:40:40.090 DEBUG:boto:Data: 2022-09-12T19:40:40.091 DEBUG:boto:Headers: {} 2022-09-12T19:40:40.091 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:40.092 DEBUG:boto:Port: 8000 2022-09-12T19:40:40.093 DEBUG:boto:Params: {} 2022-09-12T19:40:40.094 DEBUG:boto:Token: None 2022-09-12T19:40:40.095 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-69/ 2022-09-12T19:40:40.096 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NvtrUjv3Qou806Ce5prM0PCCmEY= 2022-09-12T19:40:40.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NvtrUjv3Qou806Ce5prM0PCCmEY=', 'Content-Length': '0'} 2022-09-12T19:40:40.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c80260b68830b5e6-00631f8b38-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.100 DEBUG:boto:b'jdpiak-69c1000falsec2022-09-12T19:39:54.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:40.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:40:40.102 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:40.102 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:40.103 DEBUG:boto:path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:40.103 DEBUG:boto:auth_path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:40.104 DEBUG:boto:Method: GET 2022-09-12T19:40:40.105 DEBUG:boto:Path: /jdpiak-69/d?versionId=null 2022-09-12T19:40:40.106 DEBUG:boto:Data: 2022-09-12T19:40:40.106 DEBUG:boto:Headers: {} 2022-09-12T19:40:40.107 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:40.107 DEBUG:boto:Port: 8001 2022-09-12T19:40:40.108 DEBUG:boto:Params: {} 2022-09-12T19:40:40.109 DEBUG:boto:Token: None 2022-09-12T19:40:40.109 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-69/d?versionId=null 2022-09-12T19:40:40.110 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:O97TPCniUYVApAIghonFGG8IT1U= 2022-09-12T19:40:40.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:O97TPCniUYVApAIghonFGG8IT1U=', 'Content-Length': '0'} 2022-09-12T19:40:40.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009eb1839435a774bf-00631f8b38-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.114 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:40.116 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:40.116 DEBUG:boto:path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:40.117 DEBUG:boto:auth_path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:40.117 DEBUG:boto:Method: GET 2022-09-12T19:40:40.118 DEBUG:boto:Path: /jdpiak-69/d?versionId=null 2022-09-12T19:40:40.119 DEBUG:boto:Data: 2022-09-12T19:40:40.119 DEBUG:boto:Headers: {} 2022-09-12T19:40:40.120 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:40.120 DEBUG:boto:Port: 8000 2022-09-12T19:40:40.121 DEBUG:boto:Params: {} 2022-09-12T19:40:40.122 DEBUG:boto:Token: None 2022-09-12T19:40:40.122 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-69/d?versionId=null 2022-09-12T19:40:40.123 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:O97TPCniUYVApAIghonFGG8IT1U= 2022-09-12T19:40:40.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:O97TPCniUYVApAIghonFGG8IT1U=', 'Content-Length': '0'} 2022-09-12T19:40:40.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faf77ecf3172aa0b-00631f8b38-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.128 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:40.129 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:40.130 DEBUG:boto:path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:40.130 DEBUG:boto:auth_path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:40.131 DEBUG:boto:Method: HEAD 2022-09-12T19:40:40.132 DEBUG:boto:Path: /jdpiak-69/d?versionId=null 2022-09-12T19:40:40.133 DEBUG:boto:Data: 2022-09-12T19:40:40.135 DEBUG:boto:Headers: {} 2022-09-12T19:40:40.136 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:40.136 DEBUG:boto:Port: 8001 2022-09-12T19:40:40.137 DEBUG:boto:Params: {} 2022-09-12T19:40:40.138 DEBUG:boto:Token: None 2022-09-12T19:40:40.140 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-69/d?versionId=null 2022-09-12T19:40:40.141 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ig3iXGMb738yh8qhhISacYh1zr0= 2022-09-12T19:40:40.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ig3iXGMb738yh8qhhISacYh1zr0=', 'Content-Length': '0'} 2022-09-12T19:40:40.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d76a83edc6505a3f-00631f8b38-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.150 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:40.151 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:40.152 DEBUG:boto:path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:40.153 DEBUG:boto:auth_path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:40.154 DEBUG:boto:Method: HEAD 2022-09-12T19:40:40.155 DEBUG:boto:Path: /jdpiak-69/d?versionId=null 2022-09-12T19:40:40.155 DEBUG:boto:Data: 2022-09-12T19:40:40.156 DEBUG:boto:Headers: {} 2022-09-12T19:40:40.157 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:40.158 DEBUG:boto:Port: 8000 2022-09-12T19:40:40.159 DEBUG:boto:Params: {} 2022-09-12T19:40:40.160 DEBUG:boto:Token: None 2022-09-12T19:40:40.161 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-69/d?versionId=null 2022-09-12T19:40:40.162 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ig3iXGMb738yh8qhhISacYh1zr0= 2022-09-12T19:40:40.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ig3iXGMb738yh8qhhISacYh1zr0=', 'Content-Length': '0'} 2022-09-12T19:40:40.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b89aebcca374578-00631f8b38-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:40:40.168 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:40.169 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:40.170 DEBUG:boto:path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:40.171 DEBUG:boto:auth_path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:40.172 DEBUG:boto:Method: GET 2022-09-12T19:40:40.173 DEBUG:boto:Path: /jdpiak-69/d?versionId=null 2022-09-12T19:40:40.173 DEBUG:boto:Data: 2022-09-12T19:40:40.174 DEBUG:boto:Headers: {} 2022-09-12T19:40:40.175 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:40.176 DEBUG:boto:Port: 8001 2022-09-12T19:40:40.177 DEBUG:boto:Params: {} 2022-09-12T19:40:40.177 DEBUG:boto:Token: None 2022-09-12T19:40:40.178 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-69/d?versionId=null 2022-09-12T19:40:40.179 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:O97TPCniUYVApAIghonFGG8IT1U= 2022-09-12T19:40:40.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:O97TPCniUYVApAIghonFGG8IT1U=', 'Content-Length': '0'} 2022-09-12T19:40:40.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060f7f7ca6b7baa18-00631f8b38-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.183 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:40.183 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:40.184 DEBUG:boto:path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:40.184 DEBUG:boto:auth_path=/jdpiak-69/d?versionId=null 2022-09-12T19:40:40.185 DEBUG:boto:Method: GET 2022-09-12T19:40:40.186 DEBUG:boto:Path: /jdpiak-69/d?versionId=null 2022-09-12T19:40:40.187 DEBUG:boto:Data: 2022-09-12T19:40:40.187 DEBUG:boto:Headers: {} 2022-09-12T19:40:40.188 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:40.189 DEBUG:boto:Port: 8000 2022-09-12T19:40:40.190 DEBUG:boto:Params: {} 2022-09-12T19:40:40.191 DEBUG:boto:Token: None 2022-09-12T19:40:40.192 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-69/d?versionId=null 2022-09-12T19:40:40.192 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:O97TPCniUYVApAIghonFGG8IT1U= 2022-09-12T19:40:40.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:O97TPCniUYVApAIghonFGG8IT1U=', 'Content-Length': '0'} 2022-09-12T19:40:40.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eda75aa2d5c56206-00631f8b38-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.238 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:40.239 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:40.239 DEBUG:boto:path=/jdpiak-69/?prefix=d 2022-09-12T19:40:40.240 DEBUG:boto:auth_path=/jdpiak-69/?prefix=d 2022-09-12T19:40:40.241 DEBUG:boto:Method: GET 2022-09-12T19:40:40.241 DEBUG:boto:Path: /jdpiak-69/?prefix=d 2022-09-12T19:40:40.242 DEBUG:boto:Data: 2022-09-12T19:40:40.242 DEBUG:boto:Headers: {} 2022-09-12T19:40:40.243 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:40.243 DEBUG:boto:Port: 8001 2022-09-12T19:40:40.244 DEBUG:boto:Params: {} 2022-09-12T19:40:40.245 DEBUG:boto:Token: None 2022-09-12T19:40:40.245 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-69/ 2022-09-12T19:40:40.246 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NvtrUjv3Qou806Ce5prM0PCCmEY= 2022-09-12T19:40:40.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NvtrUjv3Qou806Ce5prM0PCCmEY=', 'Content-Length': '0'} 2022-09-12T19:40:40.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d68c3965c6055e2-00631f8b38-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.251 DEBUG:boto:b'jdpiak-69d1000falsed2022-09-12T19:39:54.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:40.252 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:40.253 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:40.254 DEBUG:boto:path=/jdpiak-69/?prefix=d 2022-09-12T19:40:40.255 DEBUG:boto:auth_path=/jdpiak-69/?prefix=d 2022-09-12T19:40:40.256 DEBUG:boto:Method: GET 2022-09-12T19:40:40.257 DEBUG:boto:Path: /jdpiak-69/?prefix=d 2022-09-12T19:40:40.258 DEBUG:boto:Data: 2022-09-12T19:40:40.259 DEBUG:boto:Headers: {} 2022-09-12T19:40:40.260 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:40.261 DEBUG:boto:Port: 8000 2022-09-12T19:40:40.262 DEBUG:boto:Params: {} 2022-09-12T19:40:40.263 DEBUG:boto:Token: None 2022-09-12T19:40:40.264 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-69/ 2022-09-12T19:40:40.264 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NvtrUjv3Qou806Ce5prM0PCCmEY= 2022-09-12T19:40:40.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NvtrUjv3Qou806Ce5prM0PCCmEY=', 'Content-Length': '0'} 2022-09-12T19:40:40.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017f0779ef8b94b2a-00631f8b38-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.269 DEBUG:boto:b'jdpiak-69d1000falsed2022-09-12T19:39:54.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:40.270 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:40.271 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:40.272 DEBUG:boto:Method: HEAD 2022-09-12T19:40:40.273 DEBUG:boto:Path: /jdpiak-69/d 2022-09-12T19:40:40.273 DEBUG:boto:Data: 2022-09-12T19:40:40.274 DEBUG:boto:Headers: {} 2022-09-12T19:40:40.275 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:40.276 DEBUG:boto:Port: 8001 2022-09-12T19:40:40.278 DEBUG:boto:Params: {} 2022-09-12T19:40:40.280 DEBUG:boto:Token: None 2022-09-12T19:40:40.281 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-69/d 2022-09-12T19:40:40.283 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZjZHp9wiNoDMgywusu99veEzwFo= 2022-09-12T19:40:40.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZjZHp9wiNoDMgywusu99veEzwFo=', 'Content-Length': '0'} 2022-09-12T19:40:40.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000327ca56160a969c4-00631f8b38-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.291 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:40.292 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:40.292 DEBUG:boto:Method: HEAD 2022-09-12T19:40:40.293 DEBUG:boto:Path: /jdpiak-69/d 2022-09-12T19:40:40.294 DEBUG:boto:Data: 2022-09-12T19:40:40.295 DEBUG:boto:Headers: {} 2022-09-12T19:40:40.295 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:40.296 DEBUG:boto:Port: 8000 2022-09-12T19:40:40.297 DEBUG:boto:Params: {} 2022-09-12T19:40:40.298 DEBUG:boto:Token: None 2022-09-12T19:40:40.299 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-69/d 2022-09-12T19:40:40.300 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZjZHp9wiNoDMgywusu99veEzwFo= 2022-09-12T19:40:40.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZjZHp9wiNoDMgywusu99veEzwFo=', 'Content-Length': '0'} 2022-09-12T19:40:40.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e664ebc43dd5311d-00631f8b38-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:40:40.305 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:40.306 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:40.307 DEBUG:boto:Method: GET 2022-09-12T19:40:40.308 DEBUG:boto:Path: /jdpiak-69/d 2022-09-12T19:40:40.309 DEBUG:boto:Data: 2022-09-12T19:40:40.309 DEBUG:boto:Headers: {} 2022-09-12T19:40:40.310 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:40.311 DEBUG:boto:Port: 8001 2022-09-12T19:40:40.312 DEBUG:boto:Params: {} 2022-09-12T19:40:40.313 DEBUG:boto:Token: None 2022-09-12T19:40:40.314 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-69/d 2022-09-12T19:40:40.316 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gGhGpnvn9NNKpsX96+fLGhNHcDw= 2022-09-12T19:40:40.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gGhGpnvn9NNKpsX96+fLGhNHcDw=', 'Content-Length': '0'} 2022-09-12T19:40:40.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000360f8fc7971360ea-00631f8b38-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.322 DEBUG:boto:path=/jdpiak-69/d 2022-09-12T19:40:40.323 DEBUG:boto:auth_path=/jdpiak-69/d 2022-09-12T19:40:40.324 DEBUG:boto:Method: GET 2022-09-12T19:40:40.325 DEBUG:boto:Path: /jdpiak-69/d 2022-09-12T19:40:40.325 DEBUG:boto:Data: 2022-09-12T19:40:40.326 DEBUG:boto:Headers: {} 2022-09-12T19:40:40.327 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:40.328 DEBUG:boto:Port: 8000 2022-09-12T19:40:40.329 DEBUG:boto:Params: {} 2022-09-12T19:40:40.329 DEBUG:boto:Token: None 2022-09-12T19:40:40.330 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-69/d 2022-09-12T19:40:40.331 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gGhGpnvn9NNKpsX96+fLGhNHcDw= 2022-09-12T19:40:40.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gGhGpnvn9NNKpsX96+fLGhNHcDw=', 'Content-Length': '0'} 2022-09-12T19:40:40.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:39:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a203b14d38ce5aa1-00631f8b38-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.336 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:40.337 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:40.338 DEBUG:boto:path=/jdpiak-69/?prefix=d 2022-09-12T19:40:40.339 DEBUG:boto:auth_path=/jdpiak-69/?prefix=d 2022-09-12T19:40:40.340 DEBUG:boto:Method: GET 2022-09-12T19:40:40.341 DEBUG:boto:Path: /jdpiak-69/?prefix=d 2022-09-12T19:40:40.341 DEBUG:boto:Data: 2022-09-12T19:40:40.342 DEBUG:boto:Headers: {} 2022-09-12T19:40:40.343 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:40:40.344 DEBUG:boto:Port: 8001 2022-09-12T19:40:40.345 DEBUG:boto:Params: {} 2022-09-12T19:40:40.346 DEBUG:boto:Token: None 2022-09-12T19:40:40.347 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-69/ 2022-09-12T19:40:40.349 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NvtrUjv3Qou806Ce5prM0PCCmEY= 2022-09-12T19:40:40.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NvtrUjv3Qou806Ce5prM0PCCmEY=', 'Content-Length': '0'} 2022-09-12T19:40:40.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2b3b4647d4446a1-00631f8b38-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.356 DEBUG:boto:b'jdpiak-69d1000falsed2022-09-12T19:39:54.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:40.358 DEBUG:boto:path=/jdpiak-69/ 2022-09-12T19:40:40.359 DEBUG:boto:auth_path=/jdpiak-69/ 2022-09-12T19:40:40.360 DEBUG:boto:path=/jdpiak-69/?prefix=d 2022-09-12T19:40:40.361 DEBUG:boto:auth_path=/jdpiak-69/?prefix=d 2022-09-12T19:40:40.362 DEBUG:boto:Method: GET 2022-09-12T19:40:40.362 DEBUG:boto:Path: /jdpiak-69/?prefix=d 2022-09-12T19:40:40.363 DEBUG:boto:Data: 2022-09-12T19:40:40.364 DEBUG:boto:Headers: {} 2022-09-12T19:40:40.366 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:40:40.367 DEBUG:boto:Port: 8000 2022-09-12T19:40:40.369 DEBUG:boto:Params: {} 2022-09-12T19:40:40.371 DEBUG:boto:Token: None 2022-09-12T19:40:40.372 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-69/ 2022-09-12T19:40:40.373 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NvtrUjv3Qou806Ce5prM0PCCmEY= 2022-09-12T19:40:40.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NvtrUjv3Qou806Ce5prM0PCCmEY=', 'Content-Length': '0'} 2022-09-12T19:40:40.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035cc671925d852b4-00631f8b38-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.378 DEBUG:boto:b'jdpiak-69d1000falsed2022-09-12T19:39:54.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:40:40.379 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-69 zones={test-zone3, test-zone2} 2022-09-12T19:40:40.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:40:40.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:40:40.382 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:40:40.438 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:40:40.439 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:40:40.493 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-70 2022-09-12T19:40:40.494 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:40:40.495 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:40:40.496 DEBUG:boto:Method: PUT 2022-09-12T19:40:40.497 DEBUG:boto:Path: /jdpiak-70/ 2022-09-12T19:40:40.498 DEBUG:boto:Data: 2022-09-12T19:40:40.499 DEBUG:boto:Headers: {} 2022-09-12T19:40:40.499 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:40.500 DEBUG:boto:Port: 8000 2022-09-12T19:40:40.500 DEBUG:boto:Params: {} 2022-09-12T19:40:40.501 DEBUG:boto:Token: None 2022-09-12T19:40:40.502 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-70/ 2022-09-12T19:40:40.503 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yn+warbbph77KhEJIGLluEJynhw= 2022-09-12T19:40:40.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yn+warbbph77KhEJIGLluEJynhw=', 'Content-Length': '0'} 2022-09-12T19:40:40.526 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000afb1b7693a3c80c1-00631f8b38-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.527 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:40:40.528 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:40:40.529 DEBUG:boto:Method: HEAD 2022-09-12T19:40:40.530 DEBUG:boto:Path: /jdpiak-70/ 2022-09-12T19:40:40.530 DEBUG:boto:Data: 2022-09-12T19:40:40.531 DEBUG:boto:Headers: {} 2022-09-12T19:40:40.531 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:40.533 DEBUG:boto:Port: 8000 2022-09-12T19:40:40.534 DEBUG:boto:Params: {} 2022-09-12T19:40:40.536 DEBUG:boto:Token: None 2022-09-12T19:40:40.536 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-70/ 2022-09-12T19:40:40.537 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KyY3bwE2U7Vn6kDR6dAQHQExnh8= 2022-09-12T19:40:40.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KyY3bwE2U7Vn6kDR6dAQHQExnh8=', 'Content-Length': '0'} 2022-09-12T19:40:40.541 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006860176b57c4393b-00631f8b38-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.542 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:40:40.543 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:40:40.544 DEBUG:boto:Method: PUT 2022-09-12T19:40:40.544 DEBUG:boto:Path: /jdpiak-70/a 2022-09-12T19:40:40.545 DEBUG:boto:Data: 2022-09-12T19:40:40.546 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:40:40.546 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:40.547 DEBUG:boto:Port: 8000 2022-09-12T19:40:40.547 DEBUG:boto:Params: {} 2022-09-12T19:40:40.548 DEBUG:boto:Token: None 2022-09-12T19:40:40.549 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-70/a 2022-09-12T19:40:40.549 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e5CjILggVhgq7BJo/ZWFzP/GQHc= 2022-09-12T19:40:40.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e5CjILggVhgq7BJo/ZWFzP/GQHc='} 2022-09-12T19:40:40.559 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fea9247f2a95a2b5-00631f8b38-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.560 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:40:40.561 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:40:40.561 DEBUG:boto:Method: HEAD 2022-09-12T19:40:40.562 DEBUG:boto:Path: /jdpiak-70/ 2022-09-12T19:40:40.563 DEBUG:boto:Data: 2022-09-12T19:40:40.563 DEBUG:boto:Headers: {} 2022-09-12T19:40:40.564 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:40.564 DEBUG:boto:Port: 8000 2022-09-12T19:40:40.565 DEBUG:boto:Params: {} 2022-09-12T19:40:40.565 DEBUG:boto:Token: None 2022-09-12T19:40:40.566 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-70/ 2022-09-12T19:40:40.567 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KyY3bwE2U7Vn6kDR6dAQHQExnh8= 2022-09-12T19:40:40.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KyY3bwE2U7Vn6kDR6dAQHQExnh8=', 'Content-Length': '0'} 2022-09-12T19:40:40.570 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000236b513727954686-00631f8b38-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.571 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:40:40.572 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:40:40.572 DEBUG:boto:Method: PUT 2022-09-12T19:40:40.573 DEBUG:boto:Path: /jdpiak-70/b 2022-09-12T19:40:40.573 DEBUG:boto:Data: 2022-09-12T19:40:40.574 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:40:40.585 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:40.586 DEBUG:boto:Port: 8000 2022-09-12T19:40:40.586 DEBUG:boto:Params: {} 2022-09-12T19:40:40.587 DEBUG:boto:Token: None 2022-09-12T19:40:40.588 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-70/b 2022-09-12T19:40:40.589 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+qN+v4fI/LMJIniRH/z4kt5yJXo= 2022-09-12T19:40:40.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+qN+v4fI/LMJIniRH/z4kt5yJXo='} 2022-09-12T19:40:40.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a10fc9a3a2d7724-00631f8b38-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.636 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:40:40.637 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:40:40.637 DEBUG:boto:Method: HEAD 2022-09-12T19:40:40.638 DEBUG:boto:Path: /jdpiak-70/ 2022-09-12T19:40:40.638 DEBUG:boto:Data: 2022-09-12T19:40:40.639 DEBUG:boto:Headers: {} 2022-09-12T19:40:40.639 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:40.640 DEBUG:boto:Port: 8000 2022-09-12T19:40:40.641 DEBUG:boto:Params: {} 2022-09-12T19:40:40.642 DEBUG:boto:Token: None 2022-09-12T19:40:40.643 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-70/ 2022-09-12T19:40:40.644 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KyY3bwE2U7Vn6kDR6dAQHQExnh8= 2022-09-12T19:40:40.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KyY3bwE2U7Vn6kDR6dAQHQExnh8=', 'Content-Length': '0'} 2022-09-12T19:40:40.648 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a5e395316941266-00631f8b38-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.649 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:40:40.650 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:40:40.652 DEBUG:boto:Method: PUT 2022-09-12T19:40:40.653 DEBUG:boto:Path: /jdpiak-70/c 2022-09-12T19:40:40.654 DEBUG:boto:Data: 2022-09-12T19:40:40.655 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:40:40.656 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:40.656 DEBUG:boto:Port: 8000 2022-09-12T19:40:40.657 DEBUG:boto:Params: {} 2022-09-12T19:40:40.657 DEBUG:boto:Token: None 2022-09-12T19:40:40.658 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-70/c 2022-09-12T19:40:40.659 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LOr21cPM4S8pN592u+MYP0aUOW8= 2022-09-12T19:40:40.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LOr21cPM4S8pN592u+MYP0aUOW8='} 2022-09-12T19:40:40.665 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000080de14f5e4edff30-00631f8b38-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.667 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:40:40.668 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:40:40.668 DEBUG:boto:Method: HEAD 2022-09-12T19:40:40.669 DEBUG:boto:Path: /jdpiak-70/ 2022-09-12T19:40:40.670 DEBUG:boto:Data: 2022-09-12T19:40:40.671 DEBUG:boto:Headers: {} 2022-09-12T19:40:40.672 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:40.673 DEBUG:boto:Port: 8000 2022-09-12T19:40:40.674 DEBUG:boto:Params: {} 2022-09-12T19:40:40.675 DEBUG:boto:Token: None 2022-09-12T19:40:40.676 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-70/ 2022-09-12T19:40:40.677 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KyY3bwE2U7Vn6kDR6dAQHQExnh8= 2022-09-12T19:40:40.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KyY3bwE2U7Vn6kDR6dAQHQExnh8=', 'Content-Length': '0'} 2022-09-12T19:40:40.682 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e56db5ff75e4cfd-00631f8b38-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.684 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:40:40.685 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:40:40.686 DEBUG:boto:Method: PUT 2022-09-12T19:40:40.687 DEBUG:boto:Path: /jdpiak-70/d 2022-09-12T19:40:40.688 DEBUG:boto:Data: 2022-09-12T19:40:40.689 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:40:40.690 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:40:40.691 DEBUG:boto:Port: 8000 2022-09-12T19:40:40.692 DEBUG:boto:Params: {} 2022-09-12T19:40:40.693 DEBUG:boto:Token: None 2022-09-12T19:40:40.694 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:40:40 GMT /jdpiak-70/d 2022-09-12T19:40:40.695 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L1zK/2JmokElxOjur4Sz2lIwGhc= 2022-09-12T19:40:40.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:40:40 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L1zK/2JmokElxOjur4Sz2lIwGhc='} 2022-09-12T19:40:40.705 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006408069a85b4a702-00631f8b38-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:40:40.706 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:40:40.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:40:40.707 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:40:40.757 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:40:40.758 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:40:40.790 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=38ca1f57-1358-496c-a99b-df49ee365a4f 2022-09-12T19:40:40.811 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:40:40.812 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:40:40.812 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:40:40.812 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:40:40.813 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:40:40.813 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:40:40.813 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:40:40.813 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:40:40.814 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:40:40.814 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:40:40.814 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663011640.522951_623.1", 2022-09-12T19:40:40.814 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:40:40.522951Z" 2022-09-12T19:40:40.815 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:40:40.815 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:40:40.815 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:40:40.816 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:40:40.816 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:40:40.816 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:40:40.817 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1663011640.522951_623.1', 3: ''} 2022-09-12T19:40:40.818 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T19:40:40.819 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:40:40.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:40:40.821 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:40:40.877 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:40:40.878 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:40:40.921 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:40:40.919+0000 7f9305fd2600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:40:40.923 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:40:40.920+0000 7f9305fd2600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:40:40.943 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:40:40.943 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:40:40.943 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:40:40.944 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:40:40.944 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:40:40.944 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:40:40.944 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:40:40.945 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:40:40.945 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:40:40.945 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:40:40.946 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:40:40.946 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:40:40.946 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:40:40.946 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:40:40.947 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:40:40.947 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:40:40.947 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:40:40.947 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:40:40.948 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:40:40.948 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:40:40.948 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:40:40.948 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:40:40.949 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:40:40.949 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:40:40.949 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:40:40.949 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:40:40.950 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:40:40.950 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:40:40.950 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:40:40.950 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:40:40.951 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:40:40.951 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:40:40.952 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:40:40.952 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:40:40.952 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:40:40.952 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:40:40.953 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:40:40.953 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663011640.522951_623.1", 2022-09-12T19:40:40.953 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:40:40.953 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:40:40.954 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:40:40.954 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:40:40.522951Z", 2022-09-12T19:40:40.954 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:40:40.954 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:40:40.955 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:40:40.955 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:40:40.955 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:40:40.955 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:40:40.956 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:40:40.956 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:40:40.956 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:40:40.956 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:40:40.957 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:40:40.957 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:40:40.957 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:40:40.957 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:40:40.958 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:40:40.958 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:40:40.958 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:40:40.958 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:40:40.959 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:40:40.959 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:40:40.959 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:40:40.959 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:40:40Z" 2022-09-12T19:40:40.960 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:40:40.960 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011640.522951_623.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:40:40.522951Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:40:40Z" } 2022-09-12T19:40:40.961 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011640.522951_623.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:40:40.522951Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:40:40.961 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011640.522951_623.1', 3: ''} 2022-09-12T19:40:40.962 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011640.522951_623.1', 3: ''} 2022-09-12T19:40:40.962 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T19:40:40.963 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T19:40:40.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:40:40.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:40:40.965 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:40:41.022 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:40:41.023 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:40:41.065 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:40:41.062+0000 7f82012e1600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:40:41.065 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:40:41.063+0000 7f82012e1600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:40:41.084 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:40:41.085 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:40:41.086 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:40:41.086 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:40:41.087 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:40:41.087 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:40:41.088 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:40:41.088 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:40:41.089 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:40:41.089 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:40:41.090 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:40:41.090 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:40:41.091 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:40:41.091 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:40:41.092 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:40:41.092 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:40:41.097 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:40:41.098 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:40:41.098 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:40:41.099 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:40:41.099 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:40:41.100 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:40:41.100 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:40:41.101 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:40:41.101 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:40:41.102 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:40:41.102 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:40:41.103 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:40:41.103 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:40:41.104 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:40:41.104 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:40:41.105 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:40:41.105 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:40:41.106 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:40:41.106 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:40:41.107 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:40:41.107 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:40:41.108 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663011640.522951_623.1", 2022-09-12T19:40:41.108 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:40:41.109 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:40:41.109 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:40:41.110 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:40:40.522951Z", 2022-09-12T19:40:41.111 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:40:41.112 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:40:41.112 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:40:41.113 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:40:41.114 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:40:41.114 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:40:41.115 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:40:41.115 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:40:41.116 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:40:41.116 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:40:41.117 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:40:41.117 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:40:41.118 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:40:41.118 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:40:41.119 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:40:41.119 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:40:41.120 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:40:41.120 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:40:41.121 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:40:41.121 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:40:41.122 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:40:41.123 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:40:41Z" 2022-09-12T19:40:41.123 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:40:41.125 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011640.522951_623.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:40:40.522951Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:40:41Z" } 2022-09-12T19:40:41.127 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011640.522951_623.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:40:40.522951Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:40:41.128 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011640.522951_623.1', 3: ''} 2022-09-12T19:40:41.129 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011640.522951_623.1', 3: ''} 2022-09-12T19:40:41.130 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T19:40:41.131 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T19:40:41.132 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:40:41.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:40:41.135 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:40:41.195 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:40:41.196 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:40:41.230 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:40:41.227+0000 7f1f76529600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:40:41.231 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:40:41.229+0000 7f1f76529600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:40:41.249 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:40:41.250 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:40:41.250 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:40:41.251 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:40:41.251 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:40:41.251 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:40:41.251 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:40:41.252 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:40:41.252 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:40:41.252 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:40:41.252 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:40:41.253 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:40:41.253 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:40:41.254 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:40:41.254 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:40:41.254 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:40:41.255 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:40:41.255 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:40:41.255 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:40:41.255 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:40:41.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:40:41.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:40:41.257 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:40:41.257 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:40:41.258 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:40:41.258 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:40:41.258 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:40:41.259 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:40:41.259 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:40:41.259 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:40:41.259 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:40:41.260 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:40:41.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:40:41.260 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:40:41.260 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:40:41.261 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:40:41.261 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:40:41.261 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663011640.511778_620.1", 2022-09-12T19:40:41.261 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:40:41.262 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:40:41.262 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:40:41.262 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:40:40.511778Z", 2022-09-12T19:40:41.262 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:40:41.263 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:40:41.263 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:40:41.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:40:41.264 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:40:41.264 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:40:41.265 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:40:41.265 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:40:41.265 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:40:41.265 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:40:41.265 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:40:41.266 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:40:41.266 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:40:41.266 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:40:41.267 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:40:41.267 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:40:41.268 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:40:41.268 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:40:41.269 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:40:41.269 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:40:41.270 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:40:41.270 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:40:41Z" 2022-09-12T19:40:41.270 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:40:41.271 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011640.511778_620.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:40:40.511778Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:40:41Z" } 2022-09-12T19:40:41.271 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011640.511778_620.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:40:40.511778Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:40:41.272 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011640.522951_623.1', 3: ''} 2022-09-12T19:40:41.272 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011640.511778_620.1', 3: ''} 2022-09-12T19:40:41.273 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1663011640.522951_623.1 target=1_1663011640.511778_620.1 2022-09-12T19:40:46.276 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:40:46.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:40:46.279 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:40:46.339 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:40:46.340 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:40:46.384 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:40:46.383+0000 7f360a880600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:40:46.385 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:40:46.384+0000 7f360a880600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:40:46.405 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:40:46.405 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:40:46.405 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:40:46.406 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:40:46.406 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:40:46.406 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:40:46.407 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:40:46.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:40:46.407 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:40:46.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:40:46.408 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:40:46.408 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:40:46.409 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:40:46.409 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:40:46.409 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:40:46.409 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:40:46.410 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:40:46.410 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:40:46.411 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:40:46.411 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:40:46.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:40:46.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:40:46.412 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:40:46.412 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:40:46.413 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:40:46.413 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:40:46.413 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:40:46.414 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:40:46.414 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:40:46.414 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:40:46.415 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:40:46.415 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:40:46.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:40:46.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:40:46.416 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:40:46.416 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:40:46.416 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:40:46.416 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663011640.522951_623.1", 2022-09-12T19:40:46.417 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:40:46.417 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:40:46.417 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:40:46.418 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:40:40.522951Z", 2022-09-12T19:40:46.418 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:40:46.418 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:40:46.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:40:46.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:40:46.419 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:40:46.419 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:40:46.419 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:40:46.420 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:40:46.420 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:40:46.420 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:40:46.421 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:40:46.421 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:40:46.421 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:40:46.421 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:40:46.422 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:40:46.422 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:40:46.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:40:46.423 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:40:46.423 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:40:46.423 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:40:46.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:40:46.424 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:40:46Z" 2022-09-12T19:40:46.424 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:40:46.425 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011640.522951_623.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:40:40.522951Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:40:46Z" } 2022-09-12T19:40:46.426 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011640.522951_623.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:40:40.522951Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:40:46.427 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011640.522951_623.1', 3: ''} 2022-09-12T19:40:46.427 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011640.522951_623.1', 3: ''} 2022-09-12T19:40:46.428 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T19:40:46.428 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-70 2022-09-12T19:40:46.429 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:40:46.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:40:46.430 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:40:46.492 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:40:46.492 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:40:46.542 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:40:46.540+0000 7f202dd84600 1 waiting to reach incremental sync.. 2022-09-12T19:40:51.542 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:40:51.539+0000 7f202dd84600 1 waiting to reach incremental sync.. 2022-09-12T19:40:56.544 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:40:56.541+0000 7f202dd84600 1 bucket sync caught up with source: 2022-09-12T19:40:56.544 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.51.5, , , , 00000000001.55.5, , , 00000000001.60.5, , 00000000001.71.5] 2022-09-12T19:40:56.545 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.51.5, , , , 00000000001.55.5, , , 00000000001.60.5, , 00000000001.71.5] 2022-09-12T19:40:56.545 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:40:56.541+0000 7f202dd84600 0 bucket checkpoint complete 2022-09-12T19:40:56.563 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-70 2022-09-12T19:40:56.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:40:56.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:40:56.568 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:40:56.636 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:40:56.638 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:40:56.678 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:40:56.675+0000 7fed2649b600 1 bucket sync caught up with source: 2022-09-12T19:40:56.678 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.51.5, , , , 00000000001.55.5, , , 00000000001.60.5, , 00000000001.71.5] 2022-09-12T19:40:56.679 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000001.51.5, , , , 00000000001.55.5, , , 00000000001.60.5, , 00000000001.71.5] 2022-09-12T19:40:56.679 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:40:56.675+0000 7fed2649b600 0 bucket checkpoint complete 2022-09-12T19:40:56.693 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-70 2022-09-12T19:40:56.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:40:56.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:40:56.697 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:40:56.772 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:40:56.773 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:40:56.817 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:40:56.815+0000 7f7832f79600 1 bucket sync caught up with source: 2022-09-12T19:40:56.818 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.51.5, , , , 00000000001.55.5, , , 00000000001.60.5, , 00000000001.71.5] 2022-09-12T19:40:56.818 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.51.5, , , , 00000000001.55.5, , , 00000000001.60.5, , 00000000001.71.5] 2022-09-12T19:40:56.818 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:40:56.815+0000 7f7832f79600 0 bucket checkpoint complete 2022-09-12T19:40:56.839 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-70 2022-09-12T19:40:56.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:40:56.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:40:56.841 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-70 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:40:56.904 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:40:56.904 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:40:56.948 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:40:56.942+0000 7f0ac6440600 1 waiting to reach incremental sync.. 2022-09-12T19:41:01.948 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:41:01.943+0000 7f0ac6440600 1 waiting to reach incremental sync.. 2022-09-12T19:41:06.946 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:41:06.944+0000 7f0ac6440600 1 waiting to reach incremental sync.. 2022-09-12T19:41:11.946 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:41:11.945+0000 7f0ac6440600 1 waiting to reach incremental sync.. 2022-09-12T19:41:16.949 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:41:16.946+0000 7f0ac6440600 1 bucket sync caught up with source: 2022-09-12T19:41:16.950 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.902.5, , , , 00000000001.490.5, , , 00000000001.969.5, , 00000000001.899.5] 2022-09-12T19:41:16.950 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000001.902.5, , , , 00000000001.490.5, , , 00000000001.969.5, , 00000000001.899.5] 2022-09-12T19:41:16.951 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:41:16.946+0000 7f0ac6440600 0 bucket checkpoint complete 2022-09-12T19:41:16.968 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-70 2022-09-12T19:41:16.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:41:16.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:41:16.972 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-70 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:41:17.038 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:41:17.038 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:41:17.076 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:41:17.073+0000 7fee5aa8f600 1 bucket sync caught up with source: 2022-09-12T19:41:17.077 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.902.5, , , , 00000000001.490.5, , , 00000000001.969.5, , 00000000001.899.5] 2022-09-12T19:41:17.077 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000001.902.5, , , , 00000000001.490.5, , , 00000000001.969.5, , 00000000001.899.5] 2022-09-12T19:41:17.078 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:41:17.073+0000 7fee5aa8f600 0 bucket checkpoint complete 2022-09-12T19:41:17.092 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-70 2022-09-12T19:41:17.093 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-70 2022-09-12T19:41:17.094 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:41:17.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:41:17.097 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-70 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:41:17.161 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:41:17.166 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:41:17.199 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:41:17.195+0000 7f18514bc600 1 bucket sync caught up with source: 2022-09-12T19:41:17.200 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.427.5, , , , 00000000001.233.5, , , 00000000001.481.5, , 00000000001.431.5] 2022-09-12T19:41:17.200 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000001.427.5, , , , 00000000001.233.5, , , 00000000001.481.5, , 00000000001.431.5] 2022-09-12T19:41:17.200 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:41:17.195+0000 7f18514bc600 0 bucket checkpoint complete 2022-09-12T19:41:17.215 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-70 2022-09-12T19:41:17.216 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:41:17.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:41:17.219 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-70 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:41:17.286 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:41:17.287 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:41:17.323 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:41:17.321+0000 7f88b99d5600 1 bucket sync caught up with source: 2022-09-12T19:41:17.324 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.427.5, , , , 00000000001.233.5, , , 00000000001.481.5, , 00000000001.431.5] 2022-09-12T19:41:17.324 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.427.5, , , , 00000000001.233.5, , , 00000000001.481.5, , 00000000001.431.5] 2022-09-12T19:41:17.325 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:41:17.321+0000 7f88b99d5600 0 bucket checkpoint complete 2022-09-12T19:41:17.341 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-70 2022-09-12T19:41:17.342 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-70 zones={test-zone2, test-zone1} 2022-09-12T19:41:17.343 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:17.344 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:17.344 DEBUG:boto:Method: HEAD 2022-09-12T19:41:17.345 DEBUG:boto:Path: /jdpiak-70/ 2022-09-12T19:41:17.346 DEBUG:boto:Data: 2022-09-12T19:41:17.347 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.348 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:17.349 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.350 DEBUG:boto:Params: {} 2022-09-12T19:41:17.351 DEBUG:boto:Token: None 2022-09-12T19:41:17.352 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/ 2022-09-12T19:41:17.355 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:22Du55SWif8rDe/CrabHjPdGmlI= 2022-09-12T19:41:17.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:22Du55SWif8rDe/CrabHjPdGmlI=', 'Content-Length': '0'} 2022-09-12T19:41:17.360 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065de345ba707a5ae-00631f8b5d-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.362 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:17.363 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:17.364 DEBUG:boto:Method: HEAD 2022-09-12T19:41:17.365 DEBUG:boto:Path: /jdpiak-70/ 2022-09-12T19:41:17.366 DEBUG:boto:Data: 2022-09-12T19:41:17.367 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.368 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:17.368 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.370 DEBUG:boto:Params: {} 2022-09-12T19:41:17.370 DEBUG:boto:Token: None 2022-09-12T19:41:17.371 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/ 2022-09-12T19:41:17.371 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:22Du55SWif8rDe/CrabHjPdGmlI= 2022-09-12T19:41:17.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:22Du55SWif8rDe/CrabHjPdGmlI=', 'Content-Length': '0'} 2022-09-12T19:41:17.375 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e6bcf7f7658712db-00631f8b5d-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.375 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:41:17.376 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:17.376 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:17.377 DEBUG:boto:path=/jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:17.377 DEBUG:boto:auth_path=/jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:17.378 DEBUG:boto:Method: GET 2022-09-12T19:41:17.379 DEBUG:boto:Path: /jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:17.379 DEBUG:boto:Data: 2022-09-12T19:41:17.380 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.380 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:17.381 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.381 DEBUG:boto:Params: {} 2022-09-12T19:41:17.382 DEBUG:boto:Token: None 2022-09-12T19:41:17.383 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/?versions 2022-09-12T19:41:17.383 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iXO4Dkrmxov0V8eK7IrPOLgQzMQ= 2022-09-12T19:41:17.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iXO4Dkrmxov0V8eK7IrPOLgQzMQ=', 'Content-Length': '0'} 2022-09-12T19:41:17.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b2bc2637f1fdb40-00631f8b5d-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.387 DEBUG:boto:b'jdpiak-70999falseanulltrue2022-09-12T19:40:40.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:40:40.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:40:40.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:40:40.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:17.388 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:41:17.389 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:41:17.389 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:41:17.394 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:41:17.395 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:41:17.396 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:17.397 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:17.400 DEBUG:boto:path=/jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:17.402 DEBUG:boto:auth_path=/jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:17.407 DEBUG:boto:Method: GET 2022-09-12T19:41:17.407 DEBUG:boto:Path: /jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:17.408 DEBUG:boto:Data: 2022-09-12T19:41:17.408 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.409 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:17.409 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.410 DEBUG:boto:Params: {} 2022-09-12T19:41:17.411 DEBUG:boto:Token: None 2022-09-12T19:41:17.412 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/?versions 2022-09-12T19:41:17.412 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iXO4Dkrmxov0V8eK7IrPOLgQzMQ= 2022-09-12T19:41:17.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iXO4Dkrmxov0V8eK7IrPOLgQzMQ=', 'Content-Length': '0'} 2022-09-12T19:41:17.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d37124200456978-00631f8b5d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.417 DEBUG:boto:b'jdpiak-70999falseanulltrue2022-09-12T19:40:40.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:40:40.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:40:40.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:40:40.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:17.418 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:41:17.419 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:41:17.419 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:41:17.420 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:41:17.420 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:17.421 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:17.422 DEBUG:boto:path=/jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:17.422 DEBUG:boto:auth_path=/jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:17.423 DEBUG:boto:Method: GET 2022-09-12T19:41:17.423 DEBUG:boto:Path: /jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:17.424 DEBUG:boto:Data: 2022-09-12T19:41:17.424 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.425 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:17.425 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.426 DEBUG:boto:Params: {} 2022-09-12T19:41:17.426 DEBUG:boto:Token: None 2022-09-12T19:41:17.427 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/?versions 2022-09-12T19:41:17.428 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iXO4Dkrmxov0V8eK7IrPOLgQzMQ= 2022-09-12T19:41:17.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iXO4Dkrmxov0V8eK7IrPOLgQzMQ=', 'Content-Length': '0'} 2022-09-12T19:41:17.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fb8dde8a5007dab-00631f8b5d-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.432 DEBUG:boto:b'jdpiak-70999falseanulltrue2022-09-12T19:40:40.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:40:40.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:40:40.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:40:40.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:17.433 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:17.434 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:17.434 DEBUG:boto:path=/jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:17.435 DEBUG:boto:auth_path=/jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:17.435 DEBUG:boto:Method: GET 2022-09-12T19:41:17.436 DEBUG:boto:Path: /jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:17.436 DEBUG:boto:Data: 2022-09-12T19:41:17.437 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.437 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:17.438 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.438 DEBUG:boto:Params: {} 2022-09-12T19:41:17.439 DEBUG:boto:Token: None 2022-09-12T19:41:17.440 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/?versions 2022-09-12T19:41:17.440 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iXO4Dkrmxov0V8eK7IrPOLgQzMQ= 2022-09-12T19:41:17.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iXO4Dkrmxov0V8eK7IrPOLgQzMQ=', 'Content-Length': '0'} 2022-09-12T19:41:17.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054a8fe7e7315a980-00631f8b5d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.444 DEBUG:boto:b'jdpiak-70999falseanulltrue2022-09-12T19:40:40.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:40:40.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:40:40.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:40:40.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:17.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:41:17.446 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:17.447 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:17.447 DEBUG:boto:path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:17.448 DEBUG:boto:auth_path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:17.448 DEBUG:boto:Method: GET 2022-09-12T19:41:17.449 DEBUG:boto:Path: /jdpiak-70/a?versionId=null 2022-09-12T19:41:17.449 DEBUG:boto:Data: 2022-09-12T19:41:17.450 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.450 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:17.451 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.452 DEBUG:boto:Params: {} 2022-09-12T19:41:17.452 DEBUG:boto:Token: None 2022-09-12T19:41:17.453 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/a?versionId=null 2022-09-12T19:41:17.453 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:flvUYB/kucp4/E8G9lJAebfpwg0= 2022-09-12T19:41:17.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:flvUYB/kucp4/E8G9lJAebfpwg0=', 'Content-Length': '0'} 2022-09-12T19:41:17.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0cbb9a1bae505e6-00631f8b5d-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.498 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:17.499 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:17.500 DEBUG:boto:path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:17.501 DEBUG:boto:auth_path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:17.502 DEBUG:boto:Method: GET 2022-09-12T19:41:17.502 DEBUG:boto:Path: /jdpiak-70/a?versionId=null 2022-09-12T19:41:17.503 DEBUG:boto:Data: 2022-09-12T19:41:17.504 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.505 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:17.505 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.506 DEBUG:boto:Params: {} 2022-09-12T19:41:17.507 DEBUG:boto:Token: None 2022-09-12T19:41:17.507 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/a?versionId=null 2022-09-12T19:41:17.508 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:flvUYB/kucp4/E8G9lJAebfpwg0= 2022-09-12T19:41:17.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:flvUYB/kucp4/E8G9lJAebfpwg0=', 'Content-Length': '0'} 2022-09-12T19:41:17.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048dd1ad234a48809-00631f8b5d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.512 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:17.513 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:17.514 DEBUG:boto:path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:17.514 DEBUG:boto:auth_path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:17.515 DEBUG:boto:Method: HEAD 2022-09-12T19:41:17.515 DEBUG:boto:Path: /jdpiak-70/a?versionId=null 2022-09-12T19:41:17.516 DEBUG:boto:Data: 2022-09-12T19:41:17.517 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.517 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:17.518 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.518 DEBUG:boto:Params: {} 2022-09-12T19:41:17.519 DEBUG:boto:Token: None 2022-09-12T19:41:17.520 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/a?versionId=null 2022-09-12T19:41:17.520 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ogWkH0WT5vY3N3IMCT8Tm2y8Gg0= 2022-09-12T19:41:17.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ogWkH0WT5vY3N3IMCT8Tm2y8Gg0=', 'Content-Length': '0'} 2022-09-12T19:41:17.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a38a673118d29c1a-00631f8b5d-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.524 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:17.525 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:17.526 DEBUG:boto:path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:17.526 DEBUG:boto:auth_path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:17.527 DEBUG:boto:Method: HEAD 2022-09-12T19:41:17.527 DEBUG:boto:Path: /jdpiak-70/a?versionId=null 2022-09-12T19:41:17.528 DEBUG:boto:Data: 2022-09-12T19:41:17.529 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.529 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:17.530 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.531 DEBUG:boto:Params: {} 2022-09-12T19:41:17.532 DEBUG:boto:Token: None 2022-09-12T19:41:17.532 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/a?versionId=null 2022-09-12T19:41:17.533 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ogWkH0WT5vY3N3IMCT8Tm2y8Gg0= 2022-09-12T19:41:17.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ogWkH0WT5vY3N3IMCT8Tm2y8Gg0=', 'Content-Length': '0'} 2022-09-12T19:41:17.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d712a1fcb44ab73b-00631f8b5d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:41:17.537 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:17.539 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:17.539 DEBUG:boto:path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:17.540 DEBUG:boto:auth_path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:17.541 DEBUG:boto:Method: GET 2022-09-12T19:41:17.541 DEBUG:boto:Path: /jdpiak-70/a?versionId=null 2022-09-12T19:41:17.542 DEBUG:boto:Data: 2022-09-12T19:41:17.543 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.543 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:17.544 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.545 DEBUG:boto:Params: {} 2022-09-12T19:41:17.545 DEBUG:boto:Token: None 2022-09-12T19:41:17.546 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/a?versionId=null 2022-09-12T19:41:17.547 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:flvUYB/kucp4/E8G9lJAebfpwg0= 2022-09-12T19:41:17.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:flvUYB/kucp4/E8G9lJAebfpwg0=', 'Content-Length': '0'} 2022-09-12T19:41:17.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb9b55a757beda41-00631f8b5d-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.551 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:17.551 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:17.552 DEBUG:boto:path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:17.552 DEBUG:boto:auth_path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:17.553 DEBUG:boto:Method: GET 2022-09-12T19:41:17.554 DEBUG:boto:Path: /jdpiak-70/a?versionId=null 2022-09-12T19:41:17.554 DEBUG:boto:Data: 2022-09-12T19:41:17.556 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.557 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:17.558 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.558 DEBUG:boto:Params: {} 2022-09-12T19:41:17.559 DEBUG:boto:Token: None 2022-09-12T19:41:17.559 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/a?versionId=null 2022-09-12T19:41:17.560 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:flvUYB/kucp4/E8G9lJAebfpwg0= 2022-09-12T19:41:17.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:flvUYB/kucp4/E8G9lJAebfpwg0=', 'Content-Length': '0'} 2022-09-12T19:41:17.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006affccb4d3b33fab-00631f8b5d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.567 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:17.568 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:17.568 DEBUG:boto:path=/jdpiak-70/?prefix=a 2022-09-12T19:41:17.569 DEBUG:boto:auth_path=/jdpiak-70/?prefix=a 2022-09-12T19:41:17.570 DEBUG:boto:Method: GET 2022-09-12T19:41:17.570 DEBUG:boto:Path: /jdpiak-70/?prefix=a 2022-09-12T19:41:17.571 DEBUG:boto:Data: 2022-09-12T19:41:17.571 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.572 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:17.573 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.573 DEBUG:boto:Params: {} 2022-09-12T19:41:17.574 DEBUG:boto:Token: None 2022-09-12T19:41:17.575 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/ 2022-09-12T19:41:17.575 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xxa/J/HTz+t/e1SpVzxbWGZpYMs= 2022-09-12T19:41:17.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xxa/J/HTz+t/e1SpVzxbWGZpYMs=', 'Content-Length': '0'} 2022-09-12T19:41:17.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cf746ba28aca33e-00631f8b5d-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.579 DEBUG:boto:b'jdpiak-70a1000falsea2022-09-12T19:40:40.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:17.580 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:17.581 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:17.581 DEBUG:boto:path=/jdpiak-70/?prefix=a 2022-09-12T19:41:17.582 DEBUG:boto:auth_path=/jdpiak-70/?prefix=a 2022-09-12T19:41:17.583 DEBUG:boto:Method: GET 2022-09-12T19:41:17.583 DEBUG:boto:Path: /jdpiak-70/?prefix=a 2022-09-12T19:41:17.584 DEBUG:boto:Data: 2022-09-12T19:41:17.584 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.585 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:17.585 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.586 DEBUG:boto:Params: {} 2022-09-12T19:41:17.587 DEBUG:boto:Token: None 2022-09-12T19:41:17.588 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/ 2022-09-12T19:41:17.588 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xxa/J/HTz+t/e1SpVzxbWGZpYMs= 2022-09-12T19:41:17.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xxa/J/HTz+t/e1SpVzxbWGZpYMs=', 'Content-Length': '0'} 2022-09-12T19:41:17.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000898fde7ae2719933-00631f8b5d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.592 DEBUG:boto:b'jdpiak-70a1000falsea2022-09-12T19:40:40.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:17.593 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:17.594 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:17.594 DEBUG:boto:Method: HEAD 2022-09-12T19:41:17.595 DEBUG:boto:Path: /jdpiak-70/a 2022-09-12T19:41:17.596 DEBUG:boto:Data: 2022-09-12T19:41:17.597 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.597 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:17.598 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.598 DEBUG:boto:Params: {} 2022-09-12T19:41:17.599 DEBUG:boto:Token: None 2022-09-12T19:41:17.600 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/a 2022-09-12T19:41:17.600 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MDXZEC5imLX3iaTPj16swt1xwWc= 2022-09-12T19:41:17.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MDXZEC5imLX3iaTPj16swt1xwWc=', 'Content-Length': '0'} 2022-09-12T19:41:17.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe89d22c9a679f3c-00631f8b5d-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.604 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:17.605 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:17.605 DEBUG:boto:Method: HEAD 2022-09-12T19:41:17.606 DEBUG:boto:Path: /jdpiak-70/a 2022-09-12T19:41:17.607 DEBUG:boto:Data: 2022-09-12T19:41:17.607 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.608 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:17.608 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.609 DEBUG:boto:Params: {} 2022-09-12T19:41:17.609 DEBUG:boto:Token: None 2022-09-12T19:41:17.610 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/a 2022-09-12T19:41:17.610 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MDXZEC5imLX3iaTPj16swt1xwWc= 2022-09-12T19:41:17.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MDXZEC5imLX3iaTPj16swt1xwWc=', 'Content-Length': '0'} 2022-09-12T19:41:17.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fb7dec1222585bf-00631f8b5d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:41:17.615 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:17.616 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:17.616 DEBUG:boto:Method: GET 2022-09-12T19:41:17.617 DEBUG:boto:Path: /jdpiak-70/a 2022-09-12T19:41:17.618 DEBUG:boto:Data: 2022-09-12T19:41:17.618 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.619 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:17.619 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.620 DEBUG:boto:Params: {} 2022-09-12T19:41:17.621 DEBUG:boto:Token: None 2022-09-12T19:41:17.621 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/a 2022-09-12T19:41:17.622 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8+Gsz6B8vhingbAB0WfWNBCrwww= 2022-09-12T19:41:17.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8+Gsz6B8vhingbAB0WfWNBCrwww=', 'Content-Length': '0'} 2022-09-12T19:41:17.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c400cc2490b50996-00631f8b5d-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.626 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:17.635 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:17.636 DEBUG:boto:Method: GET 2022-09-12T19:41:17.637 DEBUG:boto:Path: /jdpiak-70/a 2022-09-12T19:41:17.637 DEBUG:boto:Data: 2022-09-12T19:41:17.638 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.638 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:17.639 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.639 DEBUG:boto:Params: {} 2022-09-12T19:41:17.640 DEBUG:boto:Token: None 2022-09-12T19:41:17.640 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/a 2022-09-12T19:41:17.641 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8+Gsz6B8vhingbAB0WfWNBCrwww= 2022-09-12T19:41:17.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8+Gsz6B8vhingbAB0WfWNBCrwww=', 'Content-Length': '0'} 2022-09-12T19:41:17.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fdfdef925d030e8-00631f8b5d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.645 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:17.645 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:17.646 DEBUG:boto:path=/jdpiak-70/?prefix=a 2022-09-12T19:41:17.646 DEBUG:boto:auth_path=/jdpiak-70/?prefix=a 2022-09-12T19:41:17.647 DEBUG:boto:Method: GET 2022-09-12T19:41:17.647 DEBUG:boto:Path: /jdpiak-70/?prefix=a 2022-09-12T19:41:17.648 DEBUG:boto:Data: 2022-09-12T19:41:17.648 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.649 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:17.649 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.650 DEBUG:boto:Params: {} 2022-09-12T19:41:17.650 DEBUG:boto:Token: None 2022-09-12T19:41:17.651 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/ 2022-09-12T19:41:17.652 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xxa/J/HTz+t/e1SpVzxbWGZpYMs= 2022-09-12T19:41:17.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xxa/J/HTz+t/e1SpVzxbWGZpYMs=', 'Content-Length': '0'} 2022-09-12T19:41:17.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbfaa91fdb03b561-00631f8b5d-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.656 DEBUG:boto:b'jdpiak-70a1000falsea2022-09-12T19:40:40.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:17.657 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:17.657 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:17.658 DEBUG:boto:path=/jdpiak-70/?prefix=a 2022-09-12T19:41:17.658 DEBUG:boto:auth_path=/jdpiak-70/?prefix=a 2022-09-12T19:41:17.659 DEBUG:boto:Method: GET 2022-09-12T19:41:17.659 DEBUG:boto:Path: /jdpiak-70/?prefix=a 2022-09-12T19:41:17.660 DEBUG:boto:Data: 2022-09-12T19:41:17.661 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.661 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:17.662 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.662 DEBUG:boto:Params: {} 2022-09-12T19:41:17.663 DEBUG:boto:Token: None 2022-09-12T19:41:17.663 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/ 2022-09-12T19:41:17.664 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xxa/J/HTz+t/e1SpVzxbWGZpYMs= 2022-09-12T19:41:17.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xxa/J/HTz+t/e1SpVzxbWGZpYMs=', 'Content-Length': '0'} 2022-09-12T19:41:17.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bc6393bb698068c-00631f8b5d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.668 DEBUG:boto:b'jdpiak-70a1000falsea2022-09-12T19:40:40.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:17.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:41:17.670 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:17.670 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:17.671 DEBUG:boto:path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:17.671 DEBUG:boto:auth_path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:17.672 DEBUG:boto:Method: GET 2022-09-12T19:41:17.672 DEBUG:boto:Path: /jdpiak-70/b?versionId=null 2022-09-12T19:41:17.673 DEBUG:boto:Data: 2022-09-12T19:41:17.674 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.674 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:17.675 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.675 DEBUG:boto:Params: {} 2022-09-12T19:41:17.676 DEBUG:boto:Token: None 2022-09-12T19:41:17.676 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/b?versionId=null 2022-09-12T19:41:17.677 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6823RPA+T400usV7Nlkzp8pMeBc= 2022-09-12T19:41:17.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6823RPA+T400usV7Nlkzp8pMeBc=', 'Content-Length': '0'} 2022-09-12T19:41:17.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006acacbaa24d892ec-00631f8b5d-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.722 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:17.723 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:17.724 DEBUG:boto:path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:17.724 DEBUG:boto:auth_path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:17.725 DEBUG:boto:Method: GET 2022-09-12T19:41:17.726 DEBUG:boto:Path: /jdpiak-70/b?versionId=null 2022-09-12T19:41:17.726 DEBUG:boto:Data: 2022-09-12T19:41:17.727 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.727 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:17.728 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.728 DEBUG:boto:Params: {} 2022-09-12T19:41:17.729 DEBUG:boto:Token: None 2022-09-12T19:41:17.730 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/b?versionId=null 2022-09-12T19:41:17.731 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6823RPA+T400usV7Nlkzp8pMeBc= 2022-09-12T19:41:17.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6823RPA+T400usV7Nlkzp8pMeBc=', 'Content-Length': '0'} 2022-09-12T19:41:17.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b982dafb95e4caac-00631f8b5d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.734 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:17.735 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:17.735 DEBUG:boto:path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:17.736 DEBUG:boto:auth_path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:17.736 DEBUG:boto:Method: HEAD 2022-09-12T19:41:17.737 DEBUG:boto:Path: /jdpiak-70/b?versionId=null 2022-09-12T19:41:17.737 DEBUG:boto:Data: 2022-09-12T19:41:17.738 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.738 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:17.739 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.739 DEBUG:boto:Params: {} 2022-09-12T19:41:17.740 DEBUG:boto:Token: None 2022-09-12T19:41:17.741 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/b?versionId=null 2022-09-12T19:41:17.741 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sclFOyr1aI36I0UYPQXg2/bYZpI= 2022-09-12T19:41:17.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sclFOyr1aI36I0UYPQXg2/bYZpI=', 'Content-Length': '0'} 2022-09-12T19:41:17.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006288975d60f20531-00631f8b5d-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.745 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:17.745 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:17.746 DEBUG:boto:path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:17.747 DEBUG:boto:auth_path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:17.747 DEBUG:boto:Method: HEAD 2022-09-12T19:41:17.748 DEBUG:boto:Path: /jdpiak-70/b?versionId=null 2022-09-12T19:41:17.748 DEBUG:boto:Data: 2022-09-12T19:41:17.749 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.749 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:17.750 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.751 DEBUG:boto:Params: {} 2022-09-12T19:41:17.751 DEBUG:boto:Token: None 2022-09-12T19:41:17.752 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/b?versionId=null 2022-09-12T19:41:17.753 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sclFOyr1aI36I0UYPQXg2/bYZpI= 2022-09-12T19:41:17.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sclFOyr1aI36I0UYPQXg2/bYZpI=', 'Content-Length': '0'} 2022-09-12T19:41:17.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e45f6e69d919a361-00631f8b5d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:41:17.756 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:17.757 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:17.758 DEBUG:boto:path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:17.758 DEBUG:boto:auth_path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:17.759 DEBUG:boto:Method: GET 2022-09-12T19:41:17.759 DEBUG:boto:Path: /jdpiak-70/b?versionId=null 2022-09-12T19:41:17.760 DEBUG:boto:Data: 2022-09-12T19:41:17.760 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.761 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:17.762 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.762 DEBUG:boto:Params: {} 2022-09-12T19:41:17.763 DEBUG:boto:Token: None 2022-09-12T19:41:17.763 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/b?versionId=null 2022-09-12T19:41:17.764 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6823RPA+T400usV7Nlkzp8pMeBc= 2022-09-12T19:41:17.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6823RPA+T400usV7Nlkzp8pMeBc=', 'Content-Length': '0'} 2022-09-12T19:41:17.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021f4bd1b8d846293-00631f8b5d-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.768 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:17.768 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:17.769 DEBUG:boto:path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:17.769 DEBUG:boto:auth_path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:17.770 DEBUG:boto:Method: GET 2022-09-12T19:41:17.771 DEBUG:boto:Path: /jdpiak-70/b?versionId=null 2022-09-12T19:41:17.771 DEBUG:boto:Data: 2022-09-12T19:41:17.772 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.772 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:17.773 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.773 DEBUG:boto:Params: {} 2022-09-12T19:41:17.774 DEBUG:boto:Token: None 2022-09-12T19:41:17.775 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/b?versionId=null 2022-09-12T19:41:17.775 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6823RPA+T400usV7Nlkzp8pMeBc= 2022-09-12T19:41:17.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6823RPA+T400usV7Nlkzp8pMeBc=', 'Content-Length': '0'} 2022-09-12T19:41:17.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a36bd4bd6b26192-00631f8b5d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.779 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:17.780 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:17.780 DEBUG:boto:path=/jdpiak-70/?prefix=b 2022-09-12T19:41:17.781 DEBUG:boto:auth_path=/jdpiak-70/?prefix=b 2022-09-12T19:41:17.781 DEBUG:boto:Method: GET 2022-09-12T19:41:17.782 DEBUG:boto:Path: /jdpiak-70/?prefix=b 2022-09-12T19:41:17.783 DEBUG:boto:Data: 2022-09-12T19:41:17.784 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.786 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:17.787 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.787 DEBUG:boto:Params: {} 2022-09-12T19:41:17.788 DEBUG:boto:Token: None 2022-09-12T19:41:17.789 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/ 2022-09-12T19:41:17.789 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xxa/J/HTz+t/e1SpVzxbWGZpYMs= 2022-09-12T19:41:17.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xxa/J/HTz+t/e1SpVzxbWGZpYMs=', 'Content-Length': '0'} 2022-09-12T19:41:17.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000826124d765f0fc6c-00631f8b5d-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.793 DEBUG:boto:b'jdpiak-70b1000falseb2022-09-12T19:40:40.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:17.794 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:17.795 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:17.795 DEBUG:boto:path=/jdpiak-70/?prefix=b 2022-09-12T19:41:17.796 DEBUG:boto:auth_path=/jdpiak-70/?prefix=b 2022-09-12T19:41:17.796 DEBUG:boto:Method: GET 2022-09-12T19:41:17.797 DEBUG:boto:Path: /jdpiak-70/?prefix=b 2022-09-12T19:41:17.798 DEBUG:boto:Data: 2022-09-12T19:41:17.798 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.799 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:17.799 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.800 DEBUG:boto:Params: {} 2022-09-12T19:41:17.800 DEBUG:boto:Token: None 2022-09-12T19:41:17.801 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/ 2022-09-12T19:41:17.802 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xxa/J/HTz+t/e1SpVzxbWGZpYMs= 2022-09-12T19:41:17.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xxa/J/HTz+t/e1SpVzxbWGZpYMs=', 'Content-Length': '0'} 2022-09-12T19:41:17.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1bbde7ce32bbf2a-00631f8b5d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.806 DEBUG:boto:b'jdpiak-70b1000falseb2022-09-12T19:40:40.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:17.807 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:17.808 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:17.808 DEBUG:boto:Method: HEAD 2022-09-12T19:41:17.809 DEBUG:boto:Path: /jdpiak-70/b 2022-09-12T19:41:17.809 DEBUG:boto:Data: 2022-09-12T19:41:17.810 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.810 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:17.811 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.811 DEBUG:boto:Params: {} 2022-09-12T19:41:17.812 DEBUG:boto:Token: None 2022-09-12T19:41:17.813 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/b 2022-09-12T19:41:17.813 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8xOFDhO7OXTvloTnm4Dz3u+Tfoc= 2022-09-12T19:41:17.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8xOFDhO7OXTvloTnm4Dz3u+Tfoc=', 'Content-Length': '0'} 2022-09-12T19:41:17.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047ed502dbaa1c606-00631f8b5d-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.817 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:17.817 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:17.818 DEBUG:boto:Method: HEAD 2022-09-12T19:41:17.818 DEBUG:boto:Path: /jdpiak-70/b 2022-09-12T19:41:17.819 DEBUG:boto:Data: 2022-09-12T19:41:17.819 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.820 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:17.820 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.821 DEBUG:boto:Params: {} 2022-09-12T19:41:17.822 DEBUG:boto:Token: None 2022-09-12T19:41:17.822 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/b 2022-09-12T19:41:17.823 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8xOFDhO7OXTvloTnm4Dz3u+Tfoc= 2022-09-12T19:41:17.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8xOFDhO7OXTvloTnm4Dz3u+Tfoc=', 'Content-Length': '0'} 2022-09-12T19:41:17.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e22a5f0f8ee8231a-00631f8b5d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:41:17.827 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:17.827 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:17.828 DEBUG:boto:Method: GET 2022-09-12T19:41:17.828 DEBUG:boto:Path: /jdpiak-70/b 2022-09-12T19:41:17.829 DEBUG:boto:Data: 2022-09-12T19:41:17.829 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.830 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:17.830 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.831 DEBUG:boto:Params: {} 2022-09-12T19:41:17.832 DEBUG:boto:Token: None 2022-09-12T19:41:17.832 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/b 2022-09-12T19:41:17.833 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xwo4viA7NqWNW4PLCtgjsBqVZuE= 2022-09-12T19:41:17.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xwo4viA7NqWNW4PLCtgjsBqVZuE=', 'Content-Length': '0'} 2022-09-12T19:41:17.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd523fcd506f6949-00631f8b5d-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.837 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:17.837 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:17.838 DEBUG:boto:Method: GET 2022-09-12T19:41:17.839 DEBUG:boto:Path: /jdpiak-70/b 2022-09-12T19:41:17.839 DEBUG:boto:Data: 2022-09-12T19:41:17.840 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.840 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:17.841 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.841 DEBUG:boto:Params: {} 2022-09-12T19:41:17.842 DEBUG:boto:Token: None 2022-09-12T19:41:17.843 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/b 2022-09-12T19:41:17.843 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xwo4viA7NqWNW4PLCtgjsBqVZuE= 2022-09-12T19:41:17.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xwo4viA7NqWNW4PLCtgjsBqVZuE=', 'Content-Length': '0'} 2022-09-12T19:41:17.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3de0cad1eaffd49-00631f8b5d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.847 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:17.847 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:17.848 DEBUG:boto:path=/jdpiak-70/?prefix=b 2022-09-12T19:41:17.848 DEBUG:boto:auth_path=/jdpiak-70/?prefix=b 2022-09-12T19:41:17.849 DEBUG:boto:Method: GET 2022-09-12T19:41:17.849 DEBUG:boto:Path: /jdpiak-70/?prefix=b 2022-09-12T19:41:17.850 DEBUG:boto:Data: 2022-09-12T19:41:17.851 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.851 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:17.852 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.852 DEBUG:boto:Params: {} 2022-09-12T19:41:17.853 DEBUG:boto:Token: None 2022-09-12T19:41:17.854 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/ 2022-09-12T19:41:17.854 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xxa/J/HTz+t/e1SpVzxbWGZpYMs= 2022-09-12T19:41:17.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xxa/J/HTz+t/e1SpVzxbWGZpYMs=', 'Content-Length': '0'} 2022-09-12T19:41:17.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b250f24577625caf-00631f8b5d-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.858 DEBUG:boto:b'jdpiak-70b1000falseb2022-09-12T19:40:40.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:17.859 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:17.860 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:17.860 DEBUG:boto:path=/jdpiak-70/?prefix=b 2022-09-12T19:41:17.861 DEBUG:boto:auth_path=/jdpiak-70/?prefix=b 2022-09-12T19:41:17.861 DEBUG:boto:Method: GET 2022-09-12T19:41:17.862 DEBUG:boto:Path: /jdpiak-70/?prefix=b 2022-09-12T19:41:17.862 DEBUG:boto:Data: 2022-09-12T19:41:17.863 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.863 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:17.864 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.866 DEBUG:boto:Params: {} 2022-09-12T19:41:17.867 DEBUG:boto:Token: None 2022-09-12T19:41:17.868 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/ 2022-09-12T19:41:17.869 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xxa/J/HTz+t/e1SpVzxbWGZpYMs= 2022-09-12T19:41:17.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xxa/J/HTz+t/e1SpVzxbWGZpYMs=', 'Content-Length': '0'} 2022-09-12T19:41:17.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000014447274e25c938-00631f8b5d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.918 DEBUG:boto:b'jdpiak-70b1000falseb2022-09-12T19:40:40.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:17.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:41:17.920 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:17.921 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:17.921 DEBUG:boto:path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:17.922 DEBUG:boto:auth_path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:17.923 DEBUG:boto:Method: GET 2022-09-12T19:41:17.923 DEBUG:boto:Path: /jdpiak-70/c?versionId=null 2022-09-12T19:41:17.924 DEBUG:boto:Data: 2022-09-12T19:41:17.925 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.925 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:17.926 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.926 DEBUG:boto:Params: {} 2022-09-12T19:41:17.927 DEBUG:boto:Token: None 2022-09-12T19:41:17.928 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/c?versionId=null 2022-09-12T19:41:17.929 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uMSHqryDa8WieFPmsVyRDkMUYnw= 2022-09-12T19:41:17.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uMSHqryDa8WieFPmsVyRDkMUYnw=', 'Content-Length': '0'} 2022-09-12T19:41:17.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000751ac7b10def0c86-00631f8b5d-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.933 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:17.934 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:17.935 DEBUG:boto:path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:17.935 DEBUG:boto:auth_path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:17.936 DEBUG:boto:Method: GET 2022-09-12T19:41:17.937 DEBUG:boto:Path: /jdpiak-70/c?versionId=null 2022-09-12T19:41:17.937 DEBUG:boto:Data: 2022-09-12T19:41:17.938 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.939 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:17.940 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.941 DEBUG:boto:Params: {} 2022-09-12T19:41:17.942 DEBUG:boto:Token: None 2022-09-12T19:41:17.942 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/c?versionId=null 2022-09-12T19:41:17.943 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uMSHqryDa8WieFPmsVyRDkMUYnw= 2022-09-12T19:41:17.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uMSHqryDa8WieFPmsVyRDkMUYnw=', 'Content-Length': '0'} 2022-09-12T19:41:17.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d95aad2e3ae1c75-00631f8b5d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.947 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:17.947 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:17.948 DEBUG:boto:path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:17.948 DEBUG:boto:auth_path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:17.949 DEBUG:boto:Method: HEAD 2022-09-12T19:41:17.949 DEBUG:boto:Path: /jdpiak-70/c?versionId=null 2022-09-12T19:41:17.950 DEBUG:boto:Data: 2022-09-12T19:41:17.951 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.952 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:17.953 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.953 DEBUG:boto:Params: {} 2022-09-12T19:41:17.954 DEBUG:boto:Token: None 2022-09-12T19:41:17.955 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/c?versionId=null 2022-09-12T19:41:17.955 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dsMsW+EEwF0zfhrRmia2SRi5eFE= 2022-09-12T19:41:17.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dsMsW+EEwF0zfhrRmia2SRi5eFE=', 'Content-Length': '0'} 2022-09-12T19:41:17.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f07abcb6520fb90f-00631f8b5d-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.959 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:17.959 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:17.960 DEBUG:boto:path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:17.961 DEBUG:boto:auth_path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:17.962 DEBUG:boto:Method: HEAD 2022-09-12T19:41:17.963 DEBUG:boto:Path: /jdpiak-70/c?versionId=null 2022-09-12T19:41:17.963 DEBUG:boto:Data: 2022-09-12T19:41:17.964 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.965 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:17.967 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.968 DEBUG:boto:Params: {} 2022-09-12T19:41:17.970 DEBUG:boto:Token: None 2022-09-12T19:41:17.971 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/c?versionId=null 2022-09-12T19:41:17.973 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dsMsW+EEwF0zfhrRmia2SRi5eFE= 2022-09-12T19:41:17.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dsMsW+EEwF0zfhrRmia2SRi5eFE=', 'Content-Length': '0'} 2022-09-12T19:41:17.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059c5a17914c2faa7-00631f8b5d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:17.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:41:17.979 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:17.980 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:17.982 DEBUG:boto:path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:17.983 DEBUG:boto:auth_path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:17.984 DEBUG:boto:Method: GET 2022-09-12T19:41:17.985 DEBUG:boto:Path: /jdpiak-70/c?versionId=null 2022-09-12T19:41:17.986 DEBUG:boto:Data: 2022-09-12T19:41:17.987 DEBUG:boto:Headers: {} 2022-09-12T19:41:17.988 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:17.989 DEBUG:boto:Port: 8000 2022-09-12T19:41:17.990 DEBUG:boto:Params: {} 2022-09-12T19:41:17.992 DEBUG:boto:Token: None 2022-09-12T19:41:17.993 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:17 GMT /jdpiak-70/c?versionId=null 2022-09-12T19:41:17.994 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uMSHqryDa8WieFPmsVyRDkMUYnw= 2022-09-12T19:41:17.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uMSHqryDa8WieFPmsVyRDkMUYnw=', 'Content-Length': '0'} 2022-09-12T19:41:17.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047ded5584d9703c0-00631f8b5d-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.000 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:18.001 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:18.002 DEBUG:boto:path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:18.003 DEBUG:boto:auth_path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:18.004 DEBUG:boto:Method: GET 2022-09-12T19:41:18.005 DEBUG:boto:Path: /jdpiak-70/c?versionId=null 2022-09-12T19:41:18.018 DEBUG:boto:Data: 2022-09-12T19:41:18.019 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.019 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:18.022 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.023 DEBUG:boto:Params: {} 2022-09-12T19:41:18.024 DEBUG:boto:Token: None 2022-09-12T19:41:18.025 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/c?versionId=null 2022-09-12T19:41:18.027 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8PFSNJRuBqAPHBP/WeebBSgNbyk= 2022-09-12T19:41:18.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8PFSNJRuBqAPHBP/WeebBSgNbyk=', 'Content-Length': '0'} 2022-09-12T19:41:18.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e83f2b46f56b97e2-00631f8b5e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.035 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:18.036 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:18.037 DEBUG:boto:path=/jdpiak-70/?prefix=c 2022-09-12T19:41:18.039 DEBUG:boto:auth_path=/jdpiak-70/?prefix=c 2022-09-12T19:41:18.040 DEBUG:boto:Method: GET 2022-09-12T19:41:18.042 DEBUG:boto:Path: /jdpiak-70/?prefix=c 2022-09-12T19:41:18.043 DEBUG:boto:Data: 2022-09-12T19:41:18.044 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.045 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:18.046 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.047 DEBUG:boto:Params: {} 2022-09-12T19:41:18.048 DEBUG:boto:Token: None 2022-09-12T19:41:18.050 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/ 2022-09-12T19:41:18.051 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ozp5XPjnYXHIax8kOFFlacixT5k= 2022-09-12T19:41:18.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ozp5XPjnYXHIax8kOFFlacixT5k=', 'Content-Length': '0'} 2022-09-12T19:41:18.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f26c69938d6046a-00631f8b5e-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.055 DEBUG:boto:b'jdpiak-70c1000falsec2022-09-12T19:40:40.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:18.056 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:18.058 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:18.059 DEBUG:boto:path=/jdpiak-70/?prefix=c 2022-09-12T19:41:18.060 DEBUG:boto:auth_path=/jdpiak-70/?prefix=c 2022-09-12T19:41:18.061 DEBUG:boto:Method: GET 2022-09-12T19:41:18.062 DEBUG:boto:Path: /jdpiak-70/?prefix=c 2022-09-12T19:41:18.062 DEBUG:boto:Data: 2022-09-12T19:41:18.063 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.064 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:18.065 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.065 DEBUG:boto:Params: {} 2022-09-12T19:41:18.066 DEBUG:boto:Token: None 2022-09-12T19:41:18.067 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/ 2022-09-12T19:41:18.068 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ozp5XPjnYXHIax8kOFFlacixT5k= 2022-09-12T19:41:18.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ozp5XPjnYXHIax8kOFFlacixT5k=', 'Content-Length': '0'} 2022-09-12T19:41:18.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdfbde914db59a9f-00631f8b5e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.114 DEBUG:boto:b'jdpiak-70c1000falsec2022-09-12T19:40:40.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:18.116 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:18.117 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:18.118 DEBUG:boto:Method: HEAD 2022-09-12T19:41:18.119 DEBUG:boto:Path: /jdpiak-70/c 2022-09-12T19:41:18.120 DEBUG:boto:Data: 2022-09-12T19:41:18.121 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.122 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:18.123 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.123 DEBUG:boto:Params: {} 2022-09-12T19:41:18.127 DEBUG:boto:Token: None 2022-09-12T19:41:18.128 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/c 2022-09-12T19:41:18.129 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2vnHvNaVZosSaU5nlNMcUWMduSk= 2022-09-12T19:41:18.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2vnHvNaVZosSaU5nlNMcUWMduSk=', 'Content-Length': '0'} 2022-09-12T19:41:18.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000605ae2b2dc488a2c-00631f8b5e-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.135 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:18.136 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:18.137 DEBUG:boto:Method: HEAD 2022-09-12T19:41:18.139 DEBUG:boto:Path: /jdpiak-70/c 2022-09-12T19:41:18.140 DEBUG:boto:Data: 2022-09-12T19:41:18.142 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.144 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:18.145 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.146 DEBUG:boto:Params: {} 2022-09-12T19:41:18.148 DEBUG:boto:Token: None 2022-09-12T19:41:18.150 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/c 2022-09-12T19:41:18.151 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2vnHvNaVZosSaU5nlNMcUWMduSk= 2022-09-12T19:41:18.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2vnHvNaVZosSaU5nlNMcUWMduSk=', 'Content-Length': '0'} 2022-09-12T19:41:18.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d0d82fdafefe4d0-00631f8b5e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:41:18.161 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:18.163 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:18.164 DEBUG:boto:Method: GET 2022-09-12T19:41:18.166 DEBUG:boto:Path: /jdpiak-70/c 2022-09-12T19:41:18.169 DEBUG:boto:Data: 2022-09-12T19:41:18.170 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.171 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:18.173 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.175 DEBUG:boto:Params: {} 2022-09-12T19:41:18.177 DEBUG:boto:Token: None 2022-09-12T19:41:18.179 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/c 2022-09-12T19:41:18.181 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JGnEtG4z+ECLRNs0krDJhV0YnUY= 2022-09-12T19:41:18.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JGnEtG4z+ECLRNs0krDJhV0YnUY=', 'Content-Length': '0'} 2022-09-12T19:41:18.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d10802ec05d3031e-00631f8b5e-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.188 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:18.189 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:18.190 DEBUG:boto:Method: GET 2022-09-12T19:41:18.191 DEBUG:boto:Path: /jdpiak-70/c 2022-09-12T19:41:18.192 DEBUG:boto:Data: 2022-09-12T19:41:18.193 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.194 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:18.195 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.197 DEBUG:boto:Params: {} 2022-09-12T19:41:18.198 DEBUG:boto:Token: None 2022-09-12T19:41:18.198 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/c 2022-09-12T19:41:18.200 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JGnEtG4z+ECLRNs0krDJhV0YnUY= 2022-09-12T19:41:18.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JGnEtG4z+ECLRNs0krDJhV0YnUY=', 'Content-Length': '0'} 2022-09-12T19:41:18.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e1744861acb02e5-00631f8b5e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.205 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:18.205 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:18.206 DEBUG:boto:path=/jdpiak-70/?prefix=c 2022-09-12T19:41:18.207 DEBUG:boto:auth_path=/jdpiak-70/?prefix=c 2022-09-12T19:41:18.208 DEBUG:boto:Method: GET 2022-09-12T19:41:18.208 DEBUG:boto:Path: /jdpiak-70/?prefix=c 2022-09-12T19:41:18.209 DEBUG:boto:Data: 2022-09-12T19:41:18.210 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.211 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:18.212 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.213 DEBUG:boto:Params: {} 2022-09-12T19:41:18.214 DEBUG:boto:Token: None 2022-09-12T19:41:18.218 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/ 2022-09-12T19:41:18.220 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ozp5XPjnYXHIax8kOFFlacixT5k= 2022-09-12T19:41:18.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ozp5XPjnYXHIax8kOFFlacixT5k=', 'Content-Length': '0'} 2022-09-12T19:41:18.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000943fd8f5067c796d-00631f8b5e-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.226 DEBUG:boto:b'jdpiak-70c1000falsec2022-09-12T19:40:40.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:18.228 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:18.229 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:18.230 DEBUG:boto:path=/jdpiak-70/?prefix=c 2022-09-12T19:41:18.231 DEBUG:boto:auth_path=/jdpiak-70/?prefix=c 2022-09-12T19:41:18.232 DEBUG:boto:Method: GET 2022-09-12T19:41:18.233 DEBUG:boto:Path: /jdpiak-70/?prefix=c 2022-09-12T19:41:18.234 DEBUG:boto:Data: 2022-09-12T19:41:18.236 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.237 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:18.238 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.244 DEBUG:boto:Params: {} 2022-09-12T19:41:18.246 DEBUG:boto:Token: None 2022-09-12T19:41:18.247 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/ 2022-09-12T19:41:18.248 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ozp5XPjnYXHIax8kOFFlacixT5k= 2022-09-12T19:41:18.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ozp5XPjnYXHIax8kOFFlacixT5k=', 'Content-Length': '0'} 2022-09-12T19:41:18.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017dc80433b5ff823-00631f8b5e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.259 DEBUG:boto:b'jdpiak-70c1000falsec2022-09-12T19:40:40.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:18.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:41:18.263 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:18.264 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:18.267 DEBUG:boto:path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:18.268 DEBUG:boto:auth_path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:18.270 DEBUG:boto:Method: GET 2022-09-12T19:41:18.271 DEBUG:boto:Path: /jdpiak-70/d?versionId=null 2022-09-12T19:41:18.274 DEBUG:boto:Data: 2022-09-12T19:41:18.275 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.276 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:18.280 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.281 DEBUG:boto:Params: {} 2022-09-12T19:41:18.282 DEBUG:boto:Token: None 2022-09-12T19:41:18.285 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/d?versionId=null 2022-09-12T19:41:18.286 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ozz/LR6JjeufhFfEc7mRpPgED7s= 2022-09-12T19:41:18.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ozz/LR6JjeufhFfEc7mRpPgED7s=', 'Content-Length': '0'} 2022-09-12T19:41:18.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052f0b7d227a6f45f-00631f8b5e-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.290 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:18.290 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:18.291 DEBUG:boto:path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:18.292 DEBUG:boto:auth_path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:18.292 DEBUG:boto:Method: GET 2022-09-12T19:41:18.293 DEBUG:boto:Path: /jdpiak-70/d?versionId=null 2022-09-12T19:41:18.293 DEBUG:boto:Data: 2022-09-12T19:41:18.294 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.294 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:18.295 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.296 DEBUG:boto:Params: {} 2022-09-12T19:41:18.297 DEBUG:boto:Token: None 2022-09-12T19:41:18.298 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/d?versionId=null 2022-09-12T19:41:18.298 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ozz/LR6JjeufhFfEc7mRpPgED7s= 2022-09-12T19:41:18.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ozz/LR6JjeufhFfEc7mRpPgED7s=', 'Content-Length': '0'} 2022-09-12T19:41:18.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050681859828e2f71-00631f8b5e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.303 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:18.303 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:18.304 DEBUG:boto:path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:18.304 DEBUG:boto:auth_path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:18.305 DEBUG:boto:Method: HEAD 2022-09-12T19:41:18.306 DEBUG:boto:Path: /jdpiak-70/d?versionId=null 2022-09-12T19:41:18.306 DEBUG:boto:Data: 2022-09-12T19:41:18.307 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.307 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:18.308 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.308 DEBUG:boto:Params: {} 2022-09-12T19:41:18.309 DEBUG:boto:Token: None 2022-09-12T19:41:18.310 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/d?versionId=null 2022-09-12T19:41:18.310 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:34oYmdFbhXnvjfV9EVQMUhAIXt8= 2022-09-12T19:41:18.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:34oYmdFbhXnvjfV9EVQMUhAIXt8=', 'Content-Length': '0'} 2022-09-12T19:41:18.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b87c2f7ee03a193-00631f8b5e-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.315 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:18.316 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:18.316 DEBUG:boto:path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:18.317 DEBUG:boto:auth_path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:18.317 DEBUG:boto:Method: HEAD 2022-09-12T19:41:18.318 DEBUG:boto:Path: /jdpiak-70/d?versionId=null 2022-09-12T19:41:18.319 DEBUG:boto:Data: 2022-09-12T19:41:18.319 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.320 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:18.320 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.321 DEBUG:boto:Params: {} 2022-09-12T19:41:18.321 DEBUG:boto:Token: None 2022-09-12T19:41:18.322 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/d?versionId=null 2022-09-12T19:41:18.323 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:34oYmdFbhXnvjfV9EVQMUhAIXt8= 2022-09-12T19:41:18.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:34oYmdFbhXnvjfV9EVQMUhAIXt8=', 'Content-Length': '0'} 2022-09-12T19:41:18.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000565fd13bffc58e89-00631f8b5e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:41:18.340 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:18.341 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:18.343 DEBUG:boto:path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:18.344 DEBUG:boto:auth_path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:18.346 DEBUG:boto:Method: GET 2022-09-12T19:41:18.347 DEBUG:boto:Path: /jdpiak-70/d?versionId=null 2022-09-12T19:41:18.349 DEBUG:boto:Data: 2022-09-12T19:41:18.349 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.350 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:18.350 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.351 DEBUG:boto:Params: {} 2022-09-12T19:41:18.352 DEBUG:boto:Token: None 2022-09-12T19:41:18.353 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/d?versionId=null 2022-09-12T19:41:18.354 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ozz/LR6JjeufhFfEc7mRpPgED7s= 2022-09-12T19:41:18.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ozz/LR6JjeufhFfEc7mRpPgED7s=', 'Content-Length': '0'} 2022-09-12T19:41:18.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f41eb18145b86079-00631f8b5e-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.402 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:18.403 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:18.404 DEBUG:boto:path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:18.405 DEBUG:boto:auth_path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:18.406 DEBUG:boto:Method: GET 2022-09-12T19:41:18.407 DEBUG:boto:Path: /jdpiak-70/d?versionId=null 2022-09-12T19:41:18.410 DEBUG:boto:Data: 2022-09-12T19:41:18.411 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.413 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:18.414 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.415 DEBUG:boto:Params: {} 2022-09-12T19:41:18.417 DEBUG:boto:Token: None 2022-09-12T19:41:18.418 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/d?versionId=null 2022-09-12T19:41:18.419 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ozz/LR6JjeufhFfEc7mRpPgED7s= 2022-09-12T19:41:18.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ozz/LR6JjeufhFfEc7mRpPgED7s=', 'Content-Length': '0'} 2022-09-12T19:41:18.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3d9dc07f8957e93-00631f8b5e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.424 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:18.425 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:18.426 DEBUG:boto:path=/jdpiak-70/?prefix=d 2022-09-12T19:41:18.427 DEBUG:boto:auth_path=/jdpiak-70/?prefix=d 2022-09-12T19:41:18.427 DEBUG:boto:Method: GET 2022-09-12T19:41:18.428 DEBUG:boto:Path: /jdpiak-70/?prefix=d 2022-09-12T19:41:18.429 DEBUG:boto:Data: 2022-09-12T19:41:18.429 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.430 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:18.431 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.431 DEBUG:boto:Params: {} 2022-09-12T19:41:18.432 DEBUG:boto:Token: None 2022-09-12T19:41:18.432 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/ 2022-09-12T19:41:18.433 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ozp5XPjnYXHIax8kOFFlacixT5k= 2022-09-12T19:41:18.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ozp5XPjnYXHIax8kOFFlacixT5k=', 'Content-Length': '0'} 2022-09-12T19:41:18.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000968d2a89c1954fdc-00631f8b5e-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.438 DEBUG:boto:b'jdpiak-70d1000falsed2022-09-12T19:40:40.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:18.439 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:18.440 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:18.440 DEBUG:boto:path=/jdpiak-70/?prefix=d 2022-09-12T19:41:18.441 DEBUG:boto:auth_path=/jdpiak-70/?prefix=d 2022-09-12T19:41:18.441 DEBUG:boto:Method: GET 2022-09-12T19:41:18.442 DEBUG:boto:Path: /jdpiak-70/?prefix=d 2022-09-12T19:41:18.442 DEBUG:boto:Data: 2022-09-12T19:41:18.443 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.444 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:18.444 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.445 DEBUG:boto:Params: {} 2022-09-12T19:41:18.446 DEBUG:boto:Token: None 2022-09-12T19:41:18.446 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/ 2022-09-12T19:41:18.447 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ozp5XPjnYXHIax8kOFFlacixT5k= 2022-09-12T19:41:18.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ozp5XPjnYXHIax8kOFFlacixT5k=', 'Content-Length': '0'} 2022-09-12T19:41:18.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b6b04157006eef5-00631f8b5e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.453 DEBUG:boto:b'jdpiak-70d1000falsed2022-09-12T19:40:40.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:18.463 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:18.464 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:18.464 DEBUG:boto:Method: HEAD 2022-09-12T19:41:18.465 DEBUG:boto:Path: /jdpiak-70/d 2022-09-12T19:41:18.466 DEBUG:boto:Data: 2022-09-12T19:41:18.466 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.467 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:18.467 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.468 DEBUG:boto:Params: {} 2022-09-12T19:41:18.469 DEBUG:boto:Token: None 2022-09-12T19:41:18.469 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/d 2022-09-12T19:41:18.470 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6wBhSjt4pndlpyGI4eL8B1BksZw= 2022-09-12T19:41:18.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6wBhSjt4pndlpyGI4eL8B1BksZw=', 'Content-Length': '0'} 2022-09-12T19:41:18.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d08d4f44988c6bc-00631f8b5e-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.474 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:18.475 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:18.475 DEBUG:boto:Method: HEAD 2022-09-12T19:41:18.476 DEBUG:boto:Path: /jdpiak-70/d 2022-09-12T19:41:18.476 DEBUG:boto:Data: 2022-09-12T19:41:18.477 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.477 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:18.478 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.478 DEBUG:boto:Params: {} 2022-09-12T19:41:18.479 DEBUG:boto:Token: None 2022-09-12T19:41:18.480 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/d 2022-09-12T19:41:18.480 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6wBhSjt4pndlpyGI4eL8B1BksZw= 2022-09-12T19:41:18.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6wBhSjt4pndlpyGI4eL8B1BksZw=', 'Content-Length': '0'} 2022-09-12T19:41:18.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5c9f417e44b9e98-00631f8b5e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:41:18.484 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:18.485 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:18.485 DEBUG:boto:Method: GET 2022-09-12T19:41:18.486 DEBUG:boto:Path: /jdpiak-70/d 2022-09-12T19:41:18.487 DEBUG:boto:Data: 2022-09-12T19:41:18.487 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.488 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:18.488 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.489 DEBUG:boto:Params: {} 2022-09-12T19:41:18.490 DEBUG:boto:Token: None 2022-09-12T19:41:18.490 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/d 2022-09-12T19:41:18.491 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:skKIj05GKGVGXD3HBjQRe73Kgd0= 2022-09-12T19:41:18.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:skKIj05GKGVGXD3HBjQRe73Kgd0=', 'Content-Length': '0'} 2022-09-12T19:41:18.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e57edef1c2c5c3d8-00631f8b5e-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.495 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:18.495 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:18.496 DEBUG:boto:Method: GET 2022-09-12T19:41:18.497 DEBUG:boto:Path: /jdpiak-70/d 2022-09-12T19:41:18.497 DEBUG:boto:Data: 2022-09-12T19:41:18.498 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.499 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:18.499 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.500 DEBUG:boto:Params: {} 2022-09-12T19:41:18.500 DEBUG:boto:Token: None 2022-09-12T19:41:18.501 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/d 2022-09-12T19:41:18.502 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:skKIj05GKGVGXD3HBjQRe73Kgd0= 2022-09-12T19:41:18.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:skKIj05GKGVGXD3HBjQRe73Kgd0=', 'Content-Length': '0'} 2022-09-12T19:41:18.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8bcef5c951c50eb-00631f8b5e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.506 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:18.507 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:18.508 DEBUG:boto:path=/jdpiak-70/?prefix=d 2022-09-12T19:41:18.508 DEBUG:boto:auth_path=/jdpiak-70/?prefix=d 2022-09-12T19:41:18.509 DEBUG:boto:Method: GET 2022-09-12T19:41:18.509 DEBUG:boto:Path: /jdpiak-70/?prefix=d 2022-09-12T19:41:18.510 DEBUG:boto:Data: 2022-09-12T19:41:18.511 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.511 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:18.512 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.512 DEBUG:boto:Params: {} 2022-09-12T19:41:18.513 DEBUG:boto:Token: None 2022-09-12T19:41:18.514 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/ 2022-09-12T19:41:18.514 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ozp5XPjnYXHIax8kOFFlacixT5k= 2022-09-12T19:41:18.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ozp5XPjnYXHIax8kOFFlacixT5k=', 'Content-Length': '0'} 2022-09-12T19:41:18.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000009751c71a96bac7-00631f8b5e-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.519 DEBUG:boto:b'jdpiak-70d1000falsed2022-09-12T19:40:40.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:18.520 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:18.521 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:18.521 DEBUG:boto:path=/jdpiak-70/?prefix=d 2022-09-12T19:41:18.522 DEBUG:boto:auth_path=/jdpiak-70/?prefix=d 2022-09-12T19:41:18.522 DEBUG:boto:Method: GET 2022-09-12T19:41:18.523 DEBUG:boto:Path: /jdpiak-70/?prefix=d 2022-09-12T19:41:18.524 DEBUG:boto:Data: 2022-09-12T19:41:18.525 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.525 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:18.526 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.527 DEBUG:boto:Params: {} 2022-09-12T19:41:18.527 DEBUG:boto:Token: None 2022-09-12T19:41:18.529 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/ 2022-09-12T19:41:18.530 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ozp5XPjnYXHIax8kOFFlacixT5k= 2022-09-12T19:41:18.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ozp5XPjnYXHIax8kOFFlacixT5k=', 'Content-Length': '0'} 2022-09-12T19:41:18.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e60ff2d0a9e09fe3-00631f8b5e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.536 DEBUG:boto:b'jdpiak-70d1000falsed2022-09-12T19:40:40.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:18.537 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-70 zones={test-zone2, test-zone1} 2022-09-12T19:41:18.537 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-70 zones={test-zone3, test-zone1} 2022-09-12T19:41:18.538 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:18.538 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:18.539 DEBUG:boto:Method: HEAD 2022-09-12T19:41:18.539 DEBUG:boto:Path: /jdpiak-70/ 2022-09-12T19:41:18.541 DEBUG:boto:Data: 2022-09-12T19:41:18.542 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.543 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:18.543 DEBUG:boto:Port: 8001 2022-09-12T19:41:18.544 DEBUG:boto:Params: {} 2022-09-12T19:41:18.545 DEBUG:boto:Token: None 2022-09-12T19:41:18.546 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/ 2022-09-12T19:41:18.547 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:txLUfUAH+YW7pMthzLG/L5UgLH0= 2022-09-12T19:41:18.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:txLUfUAH+YW7pMthzLG/L5UgLH0=', 'Content-Length': '0'} 2022-09-12T19:41:18.551 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2374144ca62f788-00631f8b5e-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.551 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:18.552 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:18.553 DEBUG:boto:Method: HEAD 2022-09-12T19:41:18.553 DEBUG:boto:Path: /jdpiak-70/ 2022-09-12T19:41:18.554 DEBUG:boto:Data: 2022-09-12T19:41:18.555 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.556 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:18.557 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.558 DEBUG:boto:Params: {} 2022-09-12T19:41:18.559 DEBUG:boto:Token: None 2022-09-12T19:41:18.561 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/ 2022-09-12T19:41:18.562 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:txLUfUAH+YW7pMthzLG/L5UgLH0= 2022-09-12T19:41:18.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:txLUfUAH+YW7pMthzLG/L5UgLH0=', 'Content-Length': '0'} 2022-09-12T19:41:18.567 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b568f01f2352355-00631f8b5e-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.568 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:41:18.569 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:18.571 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:18.572 DEBUG:boto:path=/jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:18.573 DEBUG:boto:auth_path=/jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:18.574 DEBUG:boto:Method: GET 2022-09-12T19:41:18.575 DEBUG:boto:Path: /jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:18.576 DEBUG:boto:Data: 2022-09-12T19:41:18.578 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.579 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:18.580 DEBUG:boto:Port: 8001 2022-09-12T19:41:18.581 DEBUG:boto:Params: {} 2022-09-12T19:41:18.582 DEBUG:boto:Token: None 2022-09-12T19:41:18.583 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/?versions 2022-09-12T19:41:18.584 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7oMnOSRpgOPrVuA2BTql4s7ZkB0= 2022-09-12T19:41:18.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7oMnOSRpgOPrVuA2BTql4s7ZkB0=', 'Content-Length': '0'} 2022-09-12T19:41:18.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8d8ccd3df7e7c13-00631f8b5e-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.588 DEBUG:boto:b'jdpiak-70999falseanulltrue2022-09-12T19:40:40.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:40:40.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:40:40.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:40:40.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:18.589 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:41:18.589 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:41:18.590 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:41:18.590 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:41:18.591 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:41:18.591 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:18.592 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:18.593 DEBUG:boto:path=/jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:18.593 DEBUG:boto:auth_path=/jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:18.594 DEBUG:boto:Method: GET 2022-09-12T19:41:18.594 DEBUG:boto:Path: /jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:18.595 DEBUG:boto:Data: 2022-09-12T19:41:18.595 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.597 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:18.598 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.599 DEBUG:boto:Params: {} 2022-09-12T19:41:18.599 DEBUG:boto:Token: None 2022-09-12T19:41:18.600 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/?versions 2022-09-12T19:41:18.601 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7oMnOSRpgOPrVuA2BTql4s7ZkB0= 2022-09-12T19:41:18.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7oMnOSRpgOPrVuA2BTql4s7ZkB0=', 'Content-Length': '0'} 2022-09-12T19:41:18.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009aa60dd5ad16ec08-00631f8b5e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.607 DEBUG:boto:b'jdpiak-70999falseanulltrue2022-09-12T19:40:40.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:40:40.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:40:40.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:40:40.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:18.610 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:41:18.611 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:41:18.612 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:41:18.613 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:41:18.614 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:18.616 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:18.617 DEBUG:boto:path=/jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:18.618 DEBUG:boto:auth_path=/jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:18.619 DEBUG:boto:Method: GET 2022-09-12T19:41:18.621 DEBUG:boto:Path: /jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:18.622 DEBUG:boto:Data: 2022-09-12T19:41:18.623 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.624 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:18.625 DEBUG:boto:Port: 8001 2022-09-12T19:41:18.626 DEBUG:boto:Params: {} 2022-09-12T19:41:18.627 DEBUG:boto:Token: None 2022-09-12T19:41:18.629 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/?versions 2022-09-12T19:41:18.630 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7oMnOSRpgOPrVuA2BTql4s7ZkB0= 2022-09-12T19:41:18.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7oMnOSRpgOPrVuA2BTql4s7ZkB0=', 'Content-Length': '0'} 2022-09-12T19:41:18.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012040fecc95335a3-00631f8b5e-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.636 DEBUG:boto:b'jdpiak-70999falseanulltrue2022-09-12T19:40:40.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:40:40.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:40:40.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:40:40.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:18.638 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:18.639 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:18.640 DEBUG:boto:path=/jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:18.641 DEBUG:boto:auth_path=/jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:18.643 DEBUG:boto:Method: GET 2022-09-12T19:41:18.644 DEBUG:boto:Path: /jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:18.645 DEBUG:boto:Data: 2022-09-12T19:41:18.646 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.647 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:18.648 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.649 DEBUG:boto:Params: {} 2022-09-12T19:41:18.650 DEBUG:boto:Token: None 2022-09-12T19:41:18.652 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/?versions 2022-09-12T19:41:18.653 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7oMnOSRpgOPrVuA2BTql4s7ZkB0= 2022-09-12T19:41:18.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7oMnOSRpgOPrVuA2BTql4s7ZkB0=', 'Content-Length': '0'} 2022-09-12T19:41:18.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000416486e05814e66b-00631f8b5e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.659 DEBUG:boto:b'jdpiak-70999falseanulltrue2022-09-12T19:40:40.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:40:40.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:40:40.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:40:40.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:18.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:41:18.662 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:18.663 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:18.664 DEBUG:boto:path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:18.665 DEBUG:boto:auth_path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:18.666 DEBUG:boto:Method: GET 2022-09-12T19:41:18.666 DEBUG:boto:Path: /jdpiak-70/a?versionId=null 2022-09-12T19:41:18.667 DEBUG:boto:Data: 2022-09-12T19:41:18.667 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.668 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:18.668 DEBUG:boto:Port: 8001 2022-09-12T19:41:18.669 DEBUG:boto:Params: {} 2022-09-12T19:41:18.670 DEBUG:boto:Token: None 2022-09-12T19:41:18.670 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/a?versionId=null 2022-09-12T19:41:18.671 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RMMmyz77x5r8IB/ReqLBPXlN0+Y= 2022-09-12T19:41:18.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RMMmyz77x5r8IB/ReqLBPXlN0+Y=', 'Content-Length': '0'} 2022-09-12T19:41:18.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005da2ae33502c1c07-00631f8b5e-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.675 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:18.675 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:18.676 DEBUG:boto:path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:18.676 DEBUG:boto:auth_path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:18.677 DEBUG:boto:Method: GET 2022-09-12T19:41:18.678 DEBUG:boto:Path: /jdpiak-70/a?versionId=null 2022-09-12T19:41:18.678 DEBUG:boto:Data: 2022-09-12T19:41:18.679 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.679 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:18.680 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.680 DEBUG:boto:Params: {} 2022-09-12T19:41:18.681 DEBUG:boto:Token: None 2022-09-12T19:41:18.682 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/a?versionId=null 2022-09-12T19:41:18.682 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RMMmyz77x5r8IB/ReqLBPXlN0+Y= 2022-09-12T19:41:18.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RMMmyz77x5r8IB/ReqLBPXlN0+Y=', 'Content-Length': '0'} 2022-09-12T19:41:18.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ac0df17cde9cff8-00631f8b5e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.688 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:18.688 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:18.689 DEBUG:boto:path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:18.689 DEBUG:boto:auth_path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:18.690 DEBUG:boto:Method: HEAD 2022-09-12T19:41:18.691 DEBUG:boto:Path: /jdpiak-70/a?versionId=null 2022-09-12T19:41:18.692 DEBUG:boto:Data: 2022-09-12T19:41:18.693 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.694 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:18.695 DEBUG:boto:Port: 8001 2022-09-12T19:41:18.696 DEBUG:boto:Params: {} 2022-09-12T19:41:18.697 DEBUG:boto:Token: None 2022-09-12T19:41:18.699 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/a?versionId=null 2022-09-12T19:41:18.700 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/Ni3JvDFszGgAJ5klnXkRJ2naxo= 2022-09-12T19:41:18.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/Ni3JvDFszGgAJ5klnXkRJ2naxo=', 'Content-Length': '0'} 2022-09-12T19:41:18.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d463b4916a3c10e7-00631f8b5e-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.705 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:18.706 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:18.707 DEBUG:boto:path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:18.708 DEBUG:boto:auth_path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:18.709 DEBUG:boto:Method: HEAD 2022-09-12T19:41:18.709 DEBUG:boto:Path: /jdpiak-70/a?versionId=null 2022-09-12T19:41:18.710 DEBUG:boto:Data: 2022-09-12T19:41:18.711 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.712 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:18.713 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.713 DEBUG:boto:Params: {} 2022-09-12T19:41:18.714 DEBUG:boto:Token: None 2022-09-12T19:41:18.715 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/a?versionId=null 2022-09-12T19:41:18.716 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/Ni3JvDFszGgAJ5klnXkRJ2naxo= 2022-09-12T19:41:18.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/Ni3JvDFszGgAJ5klnXkRJ2naxo=', 'Content-Length': '0'} 2022-09-12T19:41:18.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fddbe0ade58bd2d3-00631f8b5e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:41:18.720 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:18.721 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:18.721 DEBUG:boto:path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:18.722 DEBUG:boto:auth_path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:18.722 DEBUG:boto:Method: GET 2022-09-12T19:41:18.723 DEBUG:boto:Path: /jdpiak-70/a?versionId=null 2022-09-12T19:41:18.723 DEBUG:boto:Data: 2022-09-12T19:41:18.724 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.725 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:18.726 DEBUG:boto:Port: 8001 2022-09-12T19:41:18.727 DEBUG:boto:Params: {} 2022-09-12T19:41:18.729 DEBUG:boto:Token: None 2022-09-12T19:41:18.730 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/a?versionId=null 2022-09-12T19:41:18.735 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RMMmyz77x5r8IB/ReqLBPXlN0+Y= 2022-09-12T19:41:18.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RMMmyz77x5r8IB/ReqLBPXlN0+Y=', 'Content-Length': '0'} 2022-09-12T19:41:18.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0218a50f37bde35-00631f8b5e-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.740 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:18.741 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:18.742 DEBUG:boto:path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:18.743 DEBUG:boto:auth_path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:18.744 DEBUG:boto:Method: GET 2022-09-12T19:41:18.745 DEBUG:boto:Path: /jdpiak-70/a?versionId=null 2022-09-12T19:41:18.746 DEBUG:boto:Data: 2022-09-12T19:41:18.747 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.748 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:18.749 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.750 DEBUG:boto:Params: {} 2022-09-12T19:41:18.751 DEBUG:boto:Token: None 2022-09-12T19:41:18.752 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/a?versionId=null 2022-09-12T19:41:18.753 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RMMmyz77x5r8IB/ReqLBPXlN0+Y= 2022-09-12T19:41:18.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RMMmyz77x5r8IB/ReqLBPXlN0+Y=', 'Content-Length': '0'} 2022-09-12T19:41:18.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f21c1e8adfc5bfa8-00631f8b5e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.759 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:18.760 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:18.760 DEBUG:boto:path=/jdpiak-70/?prefix=a 2022-09-12T19:41:18.761 DEBUG:boto:auth_path=/jdpiak-70/?prefix=a 2022-09-12T19:41:18.762 DEBUG:boto:Method: GET 2022-09-12T19:41:18.762 DEBUG:boto:Path: /jdpiak-70/?prefix=a 2022-09-12T19:41:18.763 DEBUG:boto:Data: 2022-09-12T19:41:18.763 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.764 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:18.764 DEBUG:boto:Port: 8001 2022-09-12T19:41:18.765 DEBUG:boto:Params: {} 2022-09-12T19:41:18.766 DEBUG:boto:Token: None 2022-09-12T19:41:18.767 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/ 2022-09-12T19:41:18.768 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ozp5XPjnYXHIax8kOFFlacixT5k= 2022-09-12T19:41:18.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ozp5XPjnYXHIax8kOFFlacixT5k=', 'Content-Length': '0'} 2022-09-12T19:41:18.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b45069074519df4-00631f8b5e-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.818 DEBUG:boto:b'jdpiak-70a1000falsea2022-09-12T19:40:40.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:18.820 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:18.820 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:18.821 DEBUG:boto:path=/jdpiak-70/?prefix=a 2022-09-12T19:41:18.821 DEBUG:boto:auth_path=/jdpiak-70/?prefix=a 2022-09-12T19:41:18.822 DEBUG:boto:Method: GET 2022-09-12T19:41:18.822 DEBUG:boto:Path: /jdpiak-70/?prefix=a 2022-09-12T19:41:18.823 DEBUG:boto:Data: 2022-09-12T19:41:18.824 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.824 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:18.825 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.825 DEBUG:boto:Params: {} 2022-09-12T19:41:18.826 DEBUG:boto:Token: None 2022-09-12T19:41:18.827 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/ 2022-09-12T19:41:18.827 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ozp5XPjnYXHIax8kOFFlacixT5k= 2022-09-12T19:41:18.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ozp5XPjnYXHIax8kOFFlacixT5k=', 'Content-Length': '0'} 2022-09-12T19:41:18.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e08a9f9357be11f2-00631f8b5e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.839 DEBUG:boto:b'jdpiak-70a1000falsea2022-09-12T19:40:40.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:18.844 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:18.845 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:18.846 DEBUG:boto:Method: HEAD 2022-09-12T19:41:18.847 DEBUG:boto:Path: /jdpiak-70/a 2022-09-12T19:41:18.848 DEBUG:boto:Data: 2022-09-12T19:41:18.849 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.850 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:18.851 DEBUG:boto:Port: 8001 2022-09-12T19:41:18.852 DEBUG:boto:Params: {} 2022-09-12T19:41:18.853 DEBUG:boto:Token: None 2022-09-12T19:41:18.853 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/a 2022-09-12T19:41:18.854 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fqdhU4p+WK62tv/aRtEBax1JLsc= 2022-09-12T19:41:18.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fqdhU4p+WK62tv/aRtEBax1JLsc=', 'Content-Length': '0'} 2022-09-12T19:41:18.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8cfc1409bf59f15-00631f8b5e-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.857 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:18.858 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:18.859 DEBUG:boto:Method: HEAD 2022-09-12T19:41:18.860 DEBUG:boto:Path: /jdpiak-70/a 2022-09-12T19:41:18.860 DEBUG:boto:Data: 2022-09-12T19:41:18.861 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.861 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:18.862 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.862 DEBUG:boto:Params: {} 2022-09-12T19:41:18.863 DEBUG:boto:Token: None 2022-09-12T19:41:18.864 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/a 2022-09-12T19:41:18.865 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fqdhU4p+WK62tv/aRtEBax1JLsc= 2022-09-12T19:41:18.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fqdhU4p+WK62tv/aRtEBax1JLsc=', 'Content-Length': '0'} 2022-09-12T19:41:18.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0e8b4c037c648be-00631f8b5e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:41:18.870 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:18.871 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:18.871 DEBUG:boto:Method: GET 2022-09-12T19:41:18.872 DEBUG:boto:Path: /jdpiak-70/a 2022-09-12T19:41:18.873 DEBUG:boto:Data: 2022-09-12T19:41:18.873 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.874 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:18.875 DEBUG:boto:Port: 8001 2022-09-12T19:41:18.875 DEBUG:boto:Params: {} 2022-09-12T19:41:18.876 DEBUG:boto:Token: None 2022-09-12T19:41:18.877 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/a 2022-09-12T19:41:18.878 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mEDzP4cJ0sC3tTZ413AxpX7IuCI= 2022-09-12T19:41:18.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mEDzP4cJ0sC3tTZ413AxpX7IuCI=', 'Content-Length': '0'} 2022-09-12T19:41:18.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056ed8a2bae5fd47b-00631f8b5e-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.887 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:18.889 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:18.890 DEBUG:boto:Method: GET 2022-09-12T19:41:18.891 DEBUG:boto:Path: /jdpiak-70/a 2022-09-12T19:41:18.892 DEBUG:boto:Data: 2022-09-12T19:41:18.893 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.894 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:18.895 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.897 DEBUG:boto:Params: {} 2022-09-12T19:41:18.898 DEBUG:boto:Token: None 2022-09-12T19:41:18.899 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/a 2022-09-12T19:41:18.900 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mEDzP4cJ0sC3tTZ413AxpX7IuCI= 2022-09-12T19:41:18.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mEDzP4cJ0sC3tTZ413AxpX7IuCI=', 'Content-Length': '0'} 2022-09-12T19:41:18.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037c87212b2516f02-00631f8b5e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.946 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:18.948 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:18.949 DEBUG:boto:path=/jdpiak-70/?prefix=a 2022-09-12T19:41:18.950 DEBUG:boto:auth_path=/jdpiak-70/?prefix=a 2022-09-12T19:41:18.951 DEBUG:boto:Method: GET 2022-09-12T19:41:18.952 DEBUG:boto:Path: /jdpiak-70/?prefix=a 2022-09-12T19:41:18.953 DEBUG:boto:Data: 2022-09-12T19:41:18.954 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.955 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:18.956 DEBUG:boto:Port: 8001 2022-09-12T19:41:18.957 DEBUG:boto:Params: {} 2022-09-12T19:41:18.958 DEBUG:boto:Token: None 2022-09-12T19:41:18.960 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/ 2022-09-12T19:41:18.961 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ozp5XPjnYXHIax8kOFFlacixT5k= 2022-09-12T19:41:18.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ozp5XPjnYXHIax8kOFFlacixT5k=', 'Content-Length': '0'} 2022-09-12T19:41:18.965 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9ab7a7c2b428d40-00631f8b5e-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.967 DEBUG:boto:b'jdpiak-70a1000falsea2022-09-12T19:40:40.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:18.968 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:18.969 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:18.970 DEBUG:boto:path=/jdpiak-70/?prefix=a 2022-09-12T19:41:18.971 DEBUG:boto:auth_path=/jdpiak-70/?prefix=a 2022-09-12T19:41:18.972 DEBUG:boto:Method: GET 2022-09-12T19:41:18.974 DEBUG:boto:Path: /jdpiak-70/?prefix=a 2022-09-12T19:41:18.975 DEBUG:boto:Data: 2022-09-12T19:41:18.976 DEBUG:boto:Headers: {} 2022-09-12T19:41:18.977 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:18.978 DEBUG:boto:Port: 8000 2022-09-12T19:41:18.979 DEBUG:boto:Params: {} 2022-09-12T19:41:18.981 DEBUG:boto:Token: None 2022-09-12T19:41:18.982 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:18 GMT /jdpiak-70/ 2022-09-12T19:41:18.983 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ozp5XPjnYXHIax8kOFFlacixT5k= 2022-09-12T19:41:18.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ozp5XPjnYXHIax8kOFFlacixT5k=', 'Content-Length': '0'} 2022-09-12T19:41:18.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf0979dc321c3272-00631f8b5e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:18.999 DEBUG:boto:b'jdpiak-70a1000falsea2022-09-12T19:40:40.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:19.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:41:19.001 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:19.002 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:19.003 DEBUG:boto:path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:19.003 DEBUG:boto:auth_path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:19.004 DEBUG:boto:Method: GET 2022-09-12T19:41:19.006 DEBUG:boto:Path: /jdpiak-70/b?versionId=null 2022-09-12T19:41:19.006 DEBUG:boto:Data: 2022-09-12T19:41:19.007 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.008 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:19.009 DEBUG:boto:Port: 8001 2022-09-12T19:41:19.010 DEBUG:boto:Params: {} 2022-09-12T19:41:19.011 DEBUG:boto:Token: None 2022-09-12T19:41:19.011 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/b?versionId=null 2022-09-12T19:41:19.012 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p+ueCgXF9obYfn+BB3vCAasHH3I= 2022-09-12T19:41:19.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p+ueCgXF9obYfn+BB3vCAasHH3I=', 'Content-Length': '0'} 2022-09-12T19:41:19.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0461a7b93d44052-00631f8b5f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.016 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:19.017 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:19.017 DEBUG:boto:path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:19.018 DEBUG:boto:auth_path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:19.018 DEBUG:boto:Method: GET 2022-09-12T19:41:19.019 DEBUG:boto:Path: /jdpiak-70/b?versionId=null 2022-09-12T19:41:19.020 DEBUG:boto:Data: 2022-09-12T19:41:19.020 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.021 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:19.023 DEBUG:boto:Port: 8000 2022-09-12T19:41:19.024 DEBUG:boto:Params: {} 2022-09-12T19:41:19.025 DEBUG:boto:Token: None 2022-09-12T19:41:19.026 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/b?versionId=null 2022-09-12T19:41:19.026 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p+ueCgXF9obYfn+BB3vCAasHH3I= 2022-09-12T19:41:19.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p+ueCgXF9obYfn+BB3vCAasHH3I=', 'Content-Length': '0'} 2022-09-12T19:41:19.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004977d3f68e1bbab5-00631f8b5f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.032 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:19.033 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:19.033 DEBUG:boto:path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:19.036 DEBUG:boto:auth_path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:19.037 DEBUG:boto:Method: HEAD 2022-09-12T19:41:19.037 DEBUG:boto:Path: /jdpiak-70/b?versionId=null 2022-09-12T19:41:19.038 DEBUG:boto:Data: 2022-09-12T19:41:19.039 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.040 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:19.041 DEBUG:boto:Port: 8001 2022-09-12T19:41:19.042 DEBUG:boto:Params: {} 2022-09-12T19:41:19.043 DEBUG:boto:Token: None 2022-09-12T19:41:19.046 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/b?versionId=null 2022-09-12T19:41:19.047 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hw+Di19Bc9+tGWe3bxOe+KzJ5mk= 2022-09-12T19:41:19.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hw+Di19Bc9+tGWe3bxOe+KzJ5mk=', 'Content-Length': '0'} 2022-09-12T19:41:19.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e3fac42b390302c-00631f8b5f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.053 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:19.054 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:19.055 DEBUG:boto:path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:19.056 DEBUG:boto:auth_path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:19.057 DEBUG:boto:Method: HEAD 2022-09-12T19:41:19.059 DEBUG:boto:Path: /jdpiak-70/b?versionId=null 2022-09-12T19:41:19.059 DEBUG:boto:Data: 2022-09-12T19:41:19.061 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.062 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:19.063 DEBUG:boto:Port: 8000 2022-09-12T19:41:19.064 DEBUG:boto:Params: {} 2022-09-12T19:41:19.065 DEBUG:boto:Token: None 2022-09-12T19:41:19.066 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/b?versionId=null 2022-09-12T19:41:19.067 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hw+Di19Bc9+tGWe3bxOe+KzJ5mk= 2022-09-12T19:41:19.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hw+Di19Bc9+tGWe3bxOe+KzJ5mk=', 'Content-Length': '0'} 2022-09-12T19:41:19.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000828c564e0cbd2dff-00631f8b5f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:41:19.073 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:19.073 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:19.074 DEBUG:boto:path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:19.075 DEBUG:boto:auth_path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:19.076 DEBUG:boto:Method: GET 2022-09-12T19:41:19.076 DEBUG:boto:Path: /jdpiak-70/b?versionId=null 2022-09-12T19:41:19.077 DEBUG:boto:Data: 2022-09-12T19:41:19.078 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.078 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:19.079 DEBUG:boto:Port: 8001 2022-09-12T19:41:19.080 DEBUG:boto:Params: {} 2022-09-12T19:41:19.081 DEBUG:boto:Token: None 2022-09-12T19:41:19.082 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/b?versionId=null 2022-09-12T19:41:19.082 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p+ueCgXF9obYfn+BB3vCAasHH3I= 2022-09-12T19:41:19.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p+ueCgXF9obYfn+BB3vCAasHH3I=', 'Content-Length': '0'} 2022-09-12T19:41:19.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f781d1c279e0f9bc-00631f8b5f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.087 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:19.088 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:19.089 DEBUG:boto:path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:19.089 DEBUG:boto:auth_path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:19.090 DEBUG:boto:Method: GET 2022-09-12T19:41:19.090 DEBUG:boto:Path: /jdpiak-70/b?versionId=null 2022-09-12T19:41:19.091 DEBUG:boto:Data: 2022-09-12T19:41:19.092 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.092 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:19.093 DEBUG:boto:Port: 8000 2022-09-12T19:41:19.094 DEBUG:boto:Params: {} 2022-09-12T19:41:19.094 DEBUG:boto:Token: None 2022-09-12T19:41:19.095 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/b?versionId=null 2022-09-12T19:41:19.096 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p+ueCgXF9obYfn+BB3vCAasHH3I= 2022-09-12T19:41:19.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p+ueCgXF9obYfn+BB3vCAasHH3I=', 'Content-Length': '0'} 2022-09-12T19:41:19.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3b20ca46df24880-00631f8b5f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.146 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:19.148 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:19.149 DEBUG:boto:path=/jdpiak-70/?prefix=b 2022-09-12T19:41:19.150 DEBUG:boto:auth_path=/jdpiak-70/?prefix=b 2022-09-12T19:41:19.151 DEBUG:boto:Method: GET 2022-09-12T19:41:19.152 DEBUG:boto:Path: /jdpiak-70/?prefix=b 2022-09-12T19:41:19.152 DEBUG:boto:Data: 2022-09-12T19:41:19.153 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.154 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:19.154 DEBUG:boto:Port: 8001 2022-09-12T19:41:19.155 DEBUG:boto:Params: {} 2022-09-12T19:41:19.155 DEBUG:boto:Token: None 2022-09-12T19:41:19.156 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/ 2022-09-12T19:41:19.157 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+SP1i7jLHSDPlBnbY/C0eNU9Gk= 2022-09-12T19:41:19.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+SP1i7jLHSDPlBnbY/C0eNU9Gk=', 'Content-Length': '0'} 2022-09-12T19:41:19.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad8e39d972b9c495-00631f8b5f-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.162 DEBUG:boto:b'jdpiak-70b1000falseb2022-09-12T19:40:40.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:19.173 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:19.185 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:19.187 DEBUG:boto:path=/jdpiak-70/?prefix=b 2022-09-12T19:41:19.187 DEBUG:boto:auth_path=/jdpiak-70/?prefix=b 2022-09-12T19:41:19.188 DEBUG:boto:Method: GET 2022-09-12T19:41:19.189 DEBUG:boto:Path: /jdpiak-70/?prefix=b 2022-09-12T19:41:19.189 DEBUG:boto:Data: 2022-09-12T19:41:19.190 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.190 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:19.191 DEBUG:boto:Port: 8000 2022-09-12T19:41:19.192 DEBUG:boto:Params: {} 2022-09-12T19:41:19.193 DEBUG:boto:Token: None 2022-09-12T19:41:19.193 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/ 2022-09-12T19:41:19.194 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+SP1i7jLHSDPlBnbY/C0eNU9Gk= 2022-09-12T19:41:19.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+SP1i7jLHSDPlBnbY/C0eNU9Gk=', 'Content-Length': '0'} 2022-09-12T19:41:19.198 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd1e6295b73e1089-00631f8b5f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.200 DEBUG:boto:b'jdpiak-70b1000falseb2022-09-12T19:40:40.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:19.201 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:19.202 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:19.203 DEBUG:boto:Method: HEAD 2022-09-12T19:41:19.205 DEBUG:boto:Path: /jdpiak-70/b 2022-09-12T19:41:19.206 DEBUG:boto:Data: 2022-09-12T19:41:19.207 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.208 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:19.209 DEBUG:boto:Port: 8001 2022-09-12T19:41:19.210 DEBUG:boto:Params: {} 2022-09-12T19:41:19.211 DEBUG:boto:Token: None 2022-09-12T19:41:19.212 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/b 2022-09-12T19:41:19.213 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lba3jSOkXc9QzUDstJ6fmxIf8vI= 2022-09-12T19:41:19.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lba3jSOkXc9QzUDstJ6fmxIf8vI=', 'Content-Length': '0'} 2022-09-12T19:41:19.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000525d06cb695f7467-00631f8b5f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.217 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:19.217 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:19.218 DEBUG:boto:Method: HEAD 2022-09-12T19:41:19.218 DEBUG:boto:Path: /jdpiak-70/b 2022-09-12T19:41:19.219 DEBUG:boto:Data: 2022-09-12T19:41:19.219 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.220 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:19.221 DEBUG:boto:Port: 8000 2022-09-12T19:41:19.221 DEBUG:boto:Params: {} 2022-09-12T19:41:19.222 DEBUG:boto:Token: None 2022-09-12T19:41:19.223 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/b 2022-09-12T19:41:19.223 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lba3jSOkXc9QzUDstJ6fmxIf8vI= 2022-09-12T19:41:19.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lba3jSOkXc9QzUDstJ6fmxIf8vI=', 'Content-Length': '0'} 2022-09-12T19:41:19.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d95d8802b1a64951-00631f8b5f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:41:19.227 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:19.228 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:19.229 DEBUG:boto:Method: GET 2022-09-12T19:41:19.229 DEBUG:boto:Path: /jdpiak-70/b 2022-09-12T19:41:19.230 DEBUG:boto:Data: 2022-09-12T19:41:19.230 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.231 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:19.232 DEBUG:boto:Port: 8001 2022-09-12T19:41:19.232 DEBUG:boto:Params: {} 2022-09-12T19:41:19.233 DEBUG:boto:Token: None 2022-09-12T19:41:19.233 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/b 2022-09-12T19:41:19.234 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cfkOcBJ9gyP4eNn1C2L3yU3SSfI= 2022-09-12T19:41:19.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cfkOcBJ9gyP4eNn1C2L3yU3SSfI=', 'Content-Length': '0'} 2022-09-12T19:41:19.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000100344484ce32888-00631f8b5f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.238 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:19.239 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:19.239 DEBUG:boto:Method: GET 2022-09-12T19:41:19.240 DEBUG:boto:Path: /jdpiak-70/b 2022-09-12T19:41:19.240 DEBUG:boto:Data: 2022-09-12T19:41:19.241 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.241 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:19.242 DEBUG:boto:Port: 8000 2022-09-12T19:41:19.243 DEBUG:boto:Params: {} 2022-09-12T19:41:19.243 DEBUG:boto:Token: None 2022-09-12T19:41:19.244 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/b 2022-09-12T19:41:19.245 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cfkOcBJ9gyP4eNn1C2L3yU3SSfI= 2022-09-12T19:41:19.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cfkOcBJ9gyP4eNn1C2L3yU3SSfI=', 'Content-Length': '0'} 2022-09-12T19:41:19.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0557b8376ed64b2-00631f8b5f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.249 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:19.250 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:19.251 DEBUG:boto:path=/jdpiak-70/?prefix=b 2022-09-12T19:41:19.252 DEBUG:boto:auth_path=/jdpiak-70/?prefix=b 2022-09-12T19:41:19.254 DEBUG:boto:Method: GET 2022-09-12T19:41:19.255 DEBUG:boto:Path: /jdpiak-70/?prefix=b 2022-09-12T19:41:19.256 DEBUG:boto:Data: 2022-09-12T19:41:19.257 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.258 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:19.259 DEBUG:boto:Port: 8001 2022-09-12T19:41:19.260 DEBUG:boto:Params: {} 2022-09-12T19:41:19.261 DEBUG:boto:Token: None 2022-09-12T19:41:19.263 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/ 2022-09-12T19:41:19.264 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+SP1i7jLHSDPlBnbY/C0eNU9Gk= 2022-09-12T19:41:19.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+SP1i7jLHSDPlBnbY/C0eNU9Gk=', 'Content-Length': '0'} 2022-09-12T19:41:19.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6de12543f95f6d0-00631f8b5f-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.271 DEBUG:boto:b'jdpiak-70b1000falseb2022-09-12T19:40:40.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:19.272 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:19.273 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:19.275 DEBUG:boto:path=/jdpiak-70/?prefix=b 2022-09-12T19:41:19.276 DEBUG:boto:auth_path=/jdpiak-70/?prefix=b 2022-09-12T19:41:19.277 DEBUG:boto:Method: GET 2022-09-12T19:41:19.279 DEBUG:boto:Path: /jdpiak-70/?prefix=b 2022-09-12T19:41:19.280 DEBUG:boto:Data: 2022-09-12T19:41:19.281 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.282 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:19.283 DEBUG:boto:Port: 8000 2022-09-12T19:41:19.284 DEBUG:boto:Params: {} 2022-09-12T19:41:19.285 DEBUG:boto:Token: None 2022-09-12T19:41:19.287 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/ 2022-09-12T19:41:19.288 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+SP1i7jLHSDPlBnbY/C0eNU9Gk= 2022-09-12T19:41:19.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+SP1i7jLHSDPlBnbY/C0eNU9Gk=', 'Content-Length': '0'} 2022-09-12T19:41:19.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f1cd1911d931ea8-00631f8b5f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.295 DEBUG:boto:b'jdpiak-70b1000falseb2022-09-12T19:40:40.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:19.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:41:19.299 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:19.300 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:19.301 DEBUG:boto:path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:19.302 DEBUG:boto:auth_path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:19.302 DEBUG:boto:Method: GET 2022-09-12T19:41:19.303 DEBUG:boto:Path: /jdpiak-70/c?versionId=null 2022-09-12T19:41:19.304 DEBUG:boto:Data: 2022-09-12T19:41:19.305 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.306 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:19.307 DEBUG:boto:Port: 8001 2022-09-12T19:41:19.307 DEBUG:boto:Params: {} 2022-09-12T19:41:19.309 DEBUG:boto:Token: None 2022-09-12T19:41:19.309 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/c?versionId=null 2022-09-12T19:41:19.310 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bKA5DUSAUt8CYHkfdJ9ZWwtY5E8= 2022-09-12T19:41:19.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bKA5DUSAUt8CYHkfdJ9ZWwtY5E8=', 'Content-Length': '0'} 2022-09-12T19:41:19.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a93e9d31768c7f99-00631f8b5f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.315 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:19.316 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:19.316 DEBUG:boto:path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:19.317 DEBUG:boto:auth_path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:19.318 DEBUG:boto:Method: GET 2022-09-12T19:41:19.319 DEBUG:boto:Path: /jdpiak-70/c?versionId=null 2022-09-12T19:41:19.319 DEBUG:boto:Data: 2022-09-12T19:41:19.320 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.321 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:19.321 DEBUG:boto:Port: 8000 2022-09-12T19:41:19.322 DEBUG:boto:Params: {} 2022-09-12T19:41:19.323 DEBUG:boto:Token: None 2022-09-12T19:41:19.323 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/c?versionId=null 2022-09-12T19:41:19.324 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bKA5DUSAUt8CYHkfdJ9ZWwtY5E8= 2022-09-12T19:41:19.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bKA5DUSAUt8CYHkfdJ9ZWwtY5E8=', 'Content-Length': '0'} 2022-09-12T19:41:19.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4d391e708d42871-00631f8b5f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.329 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:19.330 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:19.331 DEBUG:boto:path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:19.332 DEBUG:boto:auth_path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:19.333 DEBUG:boto:Method: HEAD 2022-09-12T19:41:19.334 DEBUG:boto:Path: /jdpiak-70/c?versionId=null 2022-09-12T19:41:19.335 DEBUG:boto:Data: 2022-09-12T19:41:19.336 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.337 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:19.337 DEBUG:boto:Port: 8001 2022-09-12T19:41:19.338 DEBUG:boto:Params: {} 2022-09-12T19:41:19.339 DEBUG:boto:Token: None 2022-09-12T19:41:19.340 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/c?versionId=null 2022-09-12T19:41:19.340 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qO4XqXAOuoWnHXYebyME+k3bijA= 2022-09-12T19:41:19.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qO4XqXAOuoWnHXYebyME+k3bijA=', 'Content-Length': '0'} 2022-09-12T19:41:19.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acd73728977269af-00631f8b5f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.344 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:19.345 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:19.346 DEBUG:boto:path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:19.346 DEBUG:boto:auth_path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:19.347 DEBUG:boto:Method: HEAD 2022-09-12T19:41:19.348 DEBUG:boto:Path: /jdpiak-70/c?versionId=null 2022-09-12T19:41:19.348 DEBUG:boto:Data: 2022-09-12T19:41:19.349 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.350 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:19.350 DEBUG:boto:Port: 8000 2022-09-12T19:41:19.351 DEBUG:boto:Params: {} 2022-09-12T19:41:19.351 DEBUG:boto:Token: None 2022-09-12T19:41:19.352 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/c?versionId=null 2022-09-12T19:41:19.353 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qO4XqXAOuoWnHXYebyME+k3bijA= 2022-09-12T19:41:19.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qO4XqXAOuoWnHXYebyME+k3bijA=', 'Content-Length': '0'} 2022-09-12T19:41:19.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c00c7657e44ff66-00631f8b5f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:41:19.357 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:19.358 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:19.359 DEBUG:boto:path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:19.360 DEBUG:boto:auth_path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:19.361 DEBUG:boto:Method: GET 2022-09-12T19:41:19.362 DEBUG:boto:Path: /jdpiak-70/c?versionId=null 2022-09-12T19:41:19.364 DEBUG:boto:Data: 2022-09-12T19:41:19.365 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.366 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:19.367 DEBUG:boto:Port: 8001 2022-09-12T19:41:19.368 DEBUG:boto:Params: {} 2022-09-12T19:41:19.369 DEBUG:boto:Token: None 2022-09-12T19:41:19.371 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/c?versionId=null 2022-09-12T19:41:19.372 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bKA5DUSAUt8CYHkfdJ9ZWwtY5E8= 2022-09-12T19:41:19.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bKA5DUSAUt8CYHkfdJ9ZWwtY5E8=', 'Content-Length': '0'} 2022-09-12T19:41:19.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1200a34787e417d-00631f8b5f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.379 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:19.381 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:19.382 DEBUG:boto:path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:19.384 DEBUG:boto:auth_path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:19.385 DEBUG:boto:Method: GET 2022-09-12T19:41:19.386 DEBUG:boto:Path: /jdpiak-70/c?versionId=null 2022-09-12T19:41:19.387 DEBUG:boto:Data: 2022-09-12T19:41:19.388 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.390 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:19.391 DEBUG:boto:Port: 8000 2022-09-12T19:41:19.392 DEBUG:boto:Params: {} 2022-09-12T19:41:19.394 DEBUG:boto:Token: None 2022-09-12T19:41:19.395 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/c?versionId=null 2022-09-12T19:41:19.396 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bKA5DUSAUt8CYHkfdJ9ZWwtY5E8= 2022-09-12T19:41:19.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bKA5DUSAUt8CYHkfdJ9ZWwtY5E8=', 'Content-Length': '0'} 2022-09-12T19:41:19.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3786716dd9104de-00631f8b5f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.403 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:19.404 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:19.405 DEBUG:boto:path=/jdpiak-70/?prefix=c 2022-09-12T19:41:19.414 DEBUG:boto:auth_path=/jdpiak-70/?prefix=c 2022-09-12T19:41:19.416 DEBUG:boto:Method: GET 2022-09-12T19:41:19.417 DEBUG:boto:Path: /jdpiak-70/?prefix=c 2022-09-12T19:41:19.418 DEBUG:boto:Data: 2022-09-12T19:41:19.419 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.422 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:19.422 DEBUG:boto:Port: 8001 2022-09-12T19:41:19.423 DEBUG:boto:Params: {} 2022-09-12T19:41:19.424 DEBUG:boto:Token: None 2022-09-12T19:41:19.424 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/ 2022-09-12T19:41:19.425 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+SP1i7jLHSDPlBnbY/C0eNU9Gk= 2022-09-12T19:41:19.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+SP1i7jLHSDPlBnbY/C0eNU9Gk=', 'Content-Length': '0'} 2022-09-12T19:41:19.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4ee27658b030a34-00631f8b5f-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.430 DEBUG:boto:b'jdpiak-70c1000falsec2022-09-12T19:40:40.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:19.431 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:19.432 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:19.432 DEBUG:boto:path=/jdpiak-70/?prefix=c 2022-09-12T19:41:19.433 DEBUG:boto:auth_path=/jdpiak-70/?prefix=c 2022-09-12T19:41:19.434 DEBUG:boto:Method: GET 2022-09-12T19:41:19.434 DEBUG:boto:Path: /jdpiak-70/?prefix=c 2022-09-12T19:41:19.435 DEBUG:boto:Data: 2022-09-12T19:41:19.436 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.436 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:19.437 DEBUG:boto:Port: 8000 2022-09-12T19:41:19.437 DEBUG:boto:Params: {} 2022-09-12T19:41:19.438 DEBUG:boto:Token: None 2022-09-12T19:41:19.439 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/ 2022-09-12T19:41:19.440 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+SP1i7jLHSDPlBnbY/C0eNU9Gk= 2022-09-12T19:41:19.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+SP1i7jLHSDPlBnbY/C0eNU9Gk=', 'Content-Length': '0'} 2022-09-12T19:41:19.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcf5735eb8ffc579-00631f8b5f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.445 DEBUG:boto:b'jdpiak-70c1000falsec2022-09-12T19:40:40.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:19.446 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:19.446 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:19.447 DEBUG:boto:Method: HEAD 2022-09-12T19:41:19.448 DEBUG:boto:Path: /jdpiak-70/c 2022-09-12T19:41:19.448 DEBUG:boto:Data: 2022-09-12T19:41:19.449 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.449 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:19.450 DEBUG:boto:Port: 8001 2022-09-12T19:41:19.450 DEBUG:boto:Params: {} 2022-09-12T19:41:19.454 DEBUG:boto:Token: None 2022-09-12T19:41:19.455 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/c 2022-09-12T19:41:19.456 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WgAuRBjCei16q6KdifLF4hweDNw= 2022-09-12T19:41:19.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WgAuRBjCei16q6KdifLF4hweDNw=', 'Content-Length': '0'} 2022-09-12T19:41:19.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035017dd715d41ed0-00631f8b5f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.461 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:19.461 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:19.462 DEBUG:boto:Method: HEAD 2022-09-12T19:41:19.463 DEBUG:boto:Path: /jdpiak-70/c 2022-09-12T19:41:19.463 DEBUG:boto:Data: 2022-09-12T19:41:19.464 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.465 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:19.466 DEBUG:boto:Port: 8000 2022-09-12T19:41:19.466 DEBUG:boto:Params: {} 2022-09-12T19:41:19.467 DEBUG:boto:Token: None 2022-09-12T19:41:19.468 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/c 2022-09-12T19:41:19.468 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WgAuRBjCei16q6KdifLF4hweDNw= 2022-09-12T19:41:19.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WgAuRBjCei16q6KdifLF4hweDNw=', 'Content-Length': '0'} 2022-09-12T19:41:19.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e91e888602b3c9fd-00631f8b5f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:41:19.474 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:19.477 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:19.478 DEBUG:boto:Method: GET 2022-09-12T19:41:19.479 DEBUG:boto:Path: /jdpiak-70/c 2022-09-12T19:41:19.480 DEBUG:boto:Data: 2022-09-12T19:41:19.481 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.482 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:19.483 DEBUG:boto:Port: 8001 2022-09-12T19:41:19.484 DEBUG:boto:Params: {} 2022-09-12T19:41:19.485 DEBUG:boto:Token: None 2022-09-12T19:41:19.486 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/c 2022-09-12T19:41:19.487 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zlZePdcNq6j2VRvYXgpGpu3QfBI= 2022-09-12T19:41:19.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zlZePdcNq6j2VRvYXgpGpu3QfBI=', 'Content-Length': '0'} 2022-09-12T19:41:19.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c6ba0392a1ad075-00631f8b5f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.493 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:19.494 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:19.495 DEBUG:boto:Method: GET 2022-09-12T19:41:19.496 DEBUG:boto:Path: /jdpiak-70/c 2022-09-12T19:41:19.497 DEBUG:boto:Data: 2022-09-12T19:41:19.498 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.499 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:19.500 DEBUG:boto:Port: 8000 2022-09-12T19:41:19.502 DEBUG:boto:Params: {} 2022-09-12T19:41:19.503 DEBUG:boto:Token: None 2022-09-12T19:41:19.504 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/c 2022-09-12T19:41:19.505 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zlZePdcNq6j2VRvYXgpGpu3QfBI= 2022-09-12T19:41:19.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zlZePdcNq6j2VRvYXgpGpu3QfBI=', 'Content-Length': '0'} 2022-09-12T19:41:19.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000453323ea2c2a6d37-00631f8b5f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.516 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:19.527 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:19.528 DEBUG:boto:path=/jdpiak-70/?prefix=c 2022-09-12T19:41:19.529 DEBUG:boto:auth_path=/jdpiak-70/?prefix=c 2022-09-12T19:41:19.532 DEBUG:boto:Method: GET 2022-09-12T19:41:19.533 DEBUG:boto:Path: /jdpiak-70/?prefix=c 2022-09-12T19:41:19.534 DEBUG:boto:Data: 2022-09-12T19:41:19.536 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.537 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:19.538 DEBUG:boto:Port: 8001 2022-09-12T19:41:19.539 DEBUG:boto:Params: {} 2022-09-12T19:41:19.541 DEBUG:boto:Token: None 2022-09-12T19:41:19.542 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/ 2022-09-12T19:41:19.544 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+SP1i7jLHSDPlBnbY/C0eNU9Gk= 2022-09-12T19:41:19.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+SP1i7jLHSDPlBnbY/C0eNU9Gk=', 'Content-Length': '0'} 2022-09-12T19:41:19.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007733921158a8f012-00631f8b5f-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.554 DEBUG:boto:b'jdpiak-70c1000falsec2022-09-12T19:40:40.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:19.556 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:19.559 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:19.560 DEBUG:boto:path=/jdpiak-70/?prefix=c 2022-09-12T19:41:19.561 DEBUG:boto:auth_path=/jdpiak-70/?prefix=c 2022-09-12T19:41:19.562 DEBUG:boto:Method: GET 2022-09-12T19:41:19.564 DEBUG:boto:Path: /jdpiak-70/?prefix=c 2022-09-12T19:41:19.566 DEBUG:boto:Data: 2022-09-12T19:41:19.567 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.568 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:19.574 DEBUG:boto:Port: 8000 2022-09-12T19:41:19.576 DEBUG:boto:Params: {} 2022-09-12T19:41:19.578 DEBUG:boto:Token: None 2022-09-12T19:41:19.579 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/ 2022-09-12T19:41:19.581 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+SP1i7jLHSDPlBnbY/C0eNU9Gk= 2022-09-12T19:41:19.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+SP1i7jLHSDPlBnbY/C0eNU9Gk=', 'Content-Length': '0'} 2022-09-12T19:41:19.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee7185b784415442-00631f8b5f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.590 DEBUG:boto:b'jdpiak-70c1000falsec2022-09-12T19:40:40.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:19.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:41:19.592 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:19.593 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:19.594 DEBUG:boto:path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:19.595 DEBUG:boto:auth_path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:19.597 DEBUG:boto:Method: GET 2022-09-12T19:41:19.598 DEBUG:boto:Path: /jdpiak-70/d?versionId=null 2022-09-12T19:41:19.598 DEBUG:boto:Data: 2022-09-12T19:41:19.599 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.600 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:19.601 DEBUG:boto:Port: 8001 2022-09-12T19:41:19.602 DEBUG:boto:Params: {} 2022-09-12T19:41:19.604 DEBUG:boto:Token: None 2022-09-12T19:41:19.605 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/d?versionId=null 2022-09-12T19:41:19.606 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DjHZRtLJkGl3/uoRak3WJ6oIWos= 2022-09-12T19:41:19.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DjHZRtLJkGl3/uoRak3WJ6oIWos=', 'Content-Length': '0'} 2022-09-12T19:41:19.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d681a1f2d75b95c2-00631f8b5f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.616 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:19.617 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:19.618 DEBUG:boto:path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:19.621 DEBUG:boto:auth_path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:19.622 DEBUG:boto:Method: GET 2022-09-12T19:41:19.624 DEBUG:boto:Path: /jdpiak-70/d?versionId=null 2022-09-12T19:41:19.625 DEBUG:boto:Data: 2022-09-12T19:41:19.626 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.631 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:19.632 DEBUG:boto:Port: 8000 2022-09-12T19:41:19.633 DEBUG:boto:Params: {} 2022-09-12T19:41:19.634 DEBUG:boto:Token: None 2022-09-12T19:41:19.635 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/d?versionId=null 2022-09-12T19:41:19.637 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DjHZRtLJkGl3/uoRak3WJ6oIWos= 2022-09-12T19:41:19.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DjHZRtLJkGl3/uoRak3WJ6oIWos=', 'Content-Length': '0'} 2022-09-12T19:41:19.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d693dc7bb094192-00631f8b5f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.645 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:19.647 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:19.648 DEBUG:boto:path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:19.649 DEBUG:boto:auth_path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:19.650 DEBUG:boto:Method: HEAD 2022-09-12T19:41:19.651 DEBUG:boto:Path: /jdpiak-70/d?versionId=null 2022-09-12T19:41:19.652 DEBUG:boto:Data: 2022-09-12T19:41:19.653 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.653 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:19.655 DEBUG:boto:Port: 8001 2022-09-12T19:41:19.656 DEBUG:boto:Params: {} 2022-09-12T19:41:19.657 DEBUG:boto:Token: None 2022-09-12T19:41:19.658 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/d?versionId=null 2022-09-12T19:41:19.660 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KgKt3EU/ByPa9BNjdffy0ynsejI= 2022-09-12T19:41:19.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KgKt3EU/ByPa9BNjdffy0ynsejI=', 'Content-Length': '0'} 2022-09-12T19:41:19.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000890bf655bc854d54-00631f8b5f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.668 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:19.669 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:19.674 DEBUG:boto:path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:19.675 DEBUG:boto:auth_path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:19.676 DEBUG:boto:Method: HEAD 2022-09-12T19:41:19.677 DEBUG:boto:Path: /jdpiak-70/d?versionId=null 2022-09-12T19:41:19.678 DEBUG:boto:Data: 2022-09-12T19:41:19.679 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.680 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:19.681 DEBUG:boto:Port: 8000 2022-09-12T19:41:19.681 DEBUG:boto:Params: {} 2022-09-12T19:41:19.682 DEBUG:boto:Token: None 2022-09-12T19:41:19.685 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/d?versionId=null 2022-09-12T19:41:19.686 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KgKt3EU/ByPa9BNjdffy0ynsejI= 2022-09-12T19:41:19.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KgKt3EU/ByPa9BNjdffy0ynsejI=', 'Content-Length': '0'} 2022-09-12T19:41:19.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1096e1180108eef-00631f8b5f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:41:19.693 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:19.694 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:19.697 DEBUG:boto:path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:19.698 DEBUG:boto:auth_path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:19.699 DEBUG:boto:Method: GET 2022-09-12T19:41:19.700 DEBUG:boto:Path: /jdpiak-70/d?versionId=null 2022-09-12T19:41:19.701 DEBUG:boto:Data: 2022-09-12T19:41:19.703 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.704 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:19.705 DEBUG:boto:Port: 8001 2022-09-12T19:41:19.706 DEBUG:boto:Params: {} 2022-09-12T19:41:19.708 DEBUG:boto:Token: None 2022-09-12T19:41:19.709 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/d?versionId=null 2022-09-12T19:41:19.710 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DjHZRtLJkGl3/uoRak3WJ6oIWos= 2022-09-12T19:41:19.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DjHZRtLJkGl3/uoRak3WJ6oIWos=', 'Content-Length': '0'} 2022-09-12T19:41:19.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ced89ddd23691d8a-00631f8b5f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.720 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:19.721 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:19.721 DEBUG:boto:path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:19.722 DEBUG:boto:auth_path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:19.722 DEBUG:boto:Method: GET 2022-09-12T19:41:19.723 DEBUG:boto:Path: /jdpiak-70/d?versionId=null 2022-09-12T19:41:19.724 DEBUG:boto:Data: 2022-09-12T19:41:19.724 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.725 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:19.725 DEBUG:boto:Port: 8000 2022-09-12T19:41:19.726 DEBUG:boto:Params: {} 2022-09-12T19:41:19.727 DEBUG:boto:Token: None 2022-09-12T19:41:19.727 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/d?versionId=null 2022-09-12T19:41:19.728 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DjHZRtLJkGl3/uoRak3WJ6oIWos= 2022-09-12T19:41:19.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DjHZRtLJkGl3/uoRak3WJ6oIWos=', 'Content-Length': '0'} 2022-09-12T19:41:19.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000213f830d1ca6a6-00631f8b5f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.749 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:19.750 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:19.750 DEBUG:boto:path=/jdpiak-70/?prefix=d 2022-09-12T19:41:19.751 DEBUG:boto:auth_path=/jdpiak-70/?prefix=d 2022-09-12T19:41:19.752 DEBUG:boto:Method: GET 2022-09-12T19:41:19.753 DEBUG:boto:Path: /jdpiak-70/?prefix=d 2022-09-12T19:41:19.753 DEBUG:boto:Data: 2022-09-12T19:41:19.754 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.754 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:19.755 DEBUG:boto:Port: 8001 2022-09-12T19:41:19.756 DEBUG:boto:Params: {} 2022-09-12T19:41:19.757 DEBUG:boto:Token: None 2022-09-12T19:41:19.757 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/ 2022-09-12T19:41:19.758 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+SP1i7jLHSDPlBnbY/C0eNU9Gk= 2022-09-12T19:41:19.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+SP1i7jLHSDPlBnbY/C0eNU9Gk=', 'Content-Length': '0'} 2022-09-12T19:41:19.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e98ad6220db90d84-00631f8b5f-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.777 DEBUG:boto:b'jdpiak-70d1000falsed2022-09-12T19:40:40.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:19.784 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:19.785 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:19.791 DEBUG:boto:path=/jdpiak-70/?prefix=d 2022-09-12T19:41:19.791 DEBUG:boto:auth_path=/jdpiak-70/?prefix=d 2022-09-12T19:41:19.797 DEBUG:boto:Method: GET 2022-09-12T19:41:19.797 DEBUG:boto:Path: /jdpiak-70/?prefix=d 2022-09-12T19:41:19.798 DEBUG:boto:Data: 2022-09-12T19:41:19.799 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.799 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:19.800 DEBUG:boto:Port: 8000 2022-09-12T19:41:19.801 DEBUG:boto:Params: {} 2022-09-12T19:41:19.802 DEBUG:boto:Token: None 2022-09-12T19:41:19.802 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/ 2022-09-12T19:41:19.803 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+SP1i7jLHSDPlBnbY/C0eNU9Gk= 2022-09-12T19:41:19.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+SP1i7jLHSDPlBnbY/C0eNU9Gk=', 'Content-Length': '0'} 2022-09-12T19:41:19.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a54c8a1df2e37ab-00631f8b5f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.834 DEBUG:boto:b'jdpiak-70d1000falsed2022-09-12T19:40:40.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:19.841 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:19.844 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:19.845 DEBUG:boto:Method: HEAD 2022-09-12T19:41:19.846 DEBUG:boto:Path: /jdpiak-70/d 2022-09-12T19:41:19.852 DEBUG:boto:Data: 2022-09-12T19:41:19.853 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.854 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:19.859 DEBUG:boto:Port: 8001 2022-09-12T19:41:19.860 DEBUG:boto:Params: {} 2022-09-12T19:41:19.861 DEBUG:boto:Token: None 2022-09-12T19:41:19.862 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/d 2022-09-12T19:41:19.863 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DAMGXX5iOoaGDFC+XyTQd9sG2yo= 2022-09-12T19:41:19.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DAMGXX5iOoaGDFC+XyTQd9sG2yo=', 'Content-Length': '0'} 2022-09-12T19:41:19.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078d55ff21dc897c9-00631f8b5f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.873 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:19.875 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:19.877 DEBUG:boto:Method: HEAD 2022-09-12T19:41:19.877 DEBUG:boto:Path: /jdpiak-70/d 2022-09-12T19:41:19.878 DEBUG:boto:Data: 2022-09-12T19:41:19.878 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.879 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:19.880 DEBUG:boto:Port: 8000 2022-09-12T19:41:19.881 DEBUG:boto:Params: {} 2022-09-12T19:41:19.881 DEBUG:boto:Token: None 2022-09-12T19:41:19.882 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/d 2022-09-12T19:41:19.883 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DAMGXX5iOoaGDFC+XyTQd9sG2yo= 2022-09-12T19:41:19.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DAMGXX5iOoaGDFC+XyTQd9sG2yo=', 'Content-Length': '0'} 2022-09-12T19:41:19.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0b4061333294386-00631f8b5f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:41:19.903 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:19.905 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:19.908 DEBUG:boto:Method: GET 2022-09-12T19:41:19.909 DEBUG:boto:Path: /jdpiak-70/d 2022-09-12T19:41:19.911 DEBUG:boto:Data: 2022-09-12T19:41:19.913 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.913 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:19.914 DEBUG:boto:Port: 8001 2022-09-12T19:41:19.915 DEBUG:boto:Params: {} 2022-09-12T19:41:19.916 DEBUG:boto:Token: None 2022-09-12T19:41:19.916 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/d 2022-09-12T19:41:19.918 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1re3xU0lTbjUDjcQonFj66shyGY= 2022-09-12T19:41:19.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1re3xU0lTbjUDjcQonFj66shyGY=', 'Content-Length': '0'} 2022-09-12T19:41:19.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fce96c648e187ee-00631f8b5f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.924 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:19.926 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:19.927 DEBUG:boto:Method: GET 2022-09-12T19:41:19.928 DEBUG:boto:Path: /jdpiak-70/d 2022-09-12T19:41:19.929 DEBUG:boto:Data: 2022-09-12T19:41:19.929 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.930 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:19.930 DEBUG:boto:Port: 8000 2022-09-12T19:41:19.931 DEBUG:boto:Params: {} 2022-09-12T19:41:19.932 DEBUG:boto:Token: None 2022-09-12T19:41:19.933 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/d 2022-09-12T19:41:19.934 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1re3xU0lTbjUDjcQonFj66shyGY= 2022-09-12T19:41:19.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1re3xU0lTbjUDjcQonFj66shyGY=', 'Content-Length': '0'} 2022-09-12T19:41:19.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ef62f8b8d7fdae2-00631f8b5f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:19.947 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:19.948 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:19.948 DEBUG:boto:path=/jdpiak-70/?prefix=d 2022-09-12T19:41:19.949 DEBUG:boto:auth_path=/jdpiak-70/?prefix=d 2022-09-12T19:41:19.949 DEBUG:boto:Method: GET 2022-09-12T19:41:19.950 DEBUG:boto:Path: /jdpiak-70/?prefix=d 2022-09-12T19:41:19.951 DEBUG:boto:Data: 2022-09-12T19:41:19.952 DEBUG:boto:Headers: {} 2022-09-12T19:41:19.952 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:19.953 DEBUG:boto:Port: 8001 2022-09-12T19:41:19.953 DEBUG:boto:Params: {} 2022-09-12T19:41:19.954 DEBUG:boto:Token: None 2022-09-12T19:41:19.955 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:19 GMT /jdpiak-70/ 2022-09-12T19:41:19.955 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+SP1i7jLHSDPlBnbY/C0eNU9Gk= 2022-09-12T19:41:19.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+SP1i7jLHSDPlBnbY/C0eNU9Gk=', 'Content-Length': '0'} 2022-09-12T19:41:19.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000115f1b61b0812226-00631f8b5f-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.007 DEBUG:boto:b'jdpiak-70d1000falsed2022-09-12T19:40:40.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:20.008 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:20.009 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:20.009 DEBUG:boto:path=/jdpiak-70/?prefix=d 2022-09-12T19:41:20.011 DEBUG:boto:auth_path=/jdpiak-70/?prefix=d 2022-09-12T19:41:20.012 DEBUG:boto:Method: GET 2022-09-12T19:41:20.013 DEBUG:boto:Path: /jdpiak-70/?prefix=d 2022-09-12T19:41:20.014 DEBUG:boto:Data: 2022-09-12T19:41:20.015 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.016 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:20.018 DEBUG:boto:Port: 8000 2022-09-12T19:41:20.019 DEBUG:boto:Params: {} 2022-09-12T19:41:20.020 DEBUG:boto:Token: None 2022-09-12T19:41:20.021 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/ 2022-09-12T19:41:20.022 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AyDaFiScgaFSQrzv9+yCcDyHDFE= 2022-09-12T19:41:20.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AyDaFiScgaFSQrzv9+yCcDyHDFE=', 'Content-Length': '0'} 2022-09-12T19:41:20.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee91c9621035232c-00631f8b60-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.028 DEBUG:boto:b'jdpiak-70d1000falsed2022-09-12T19:40:40.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:20.030 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-70 zones={test-zone3, test-zone1} 2022-09-12T19:41:20.030 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-70 zones={test-zone3, test-zone2} 2022-09-12T19:41:20.032 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:20.033 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:20.034 DEBUG:boto:Method: HEAD 2022-09-12T19:41:20.036 DEBUG:boto:Path: /jdpiak-70/ 2022-09-12T19:41:20.037 DEBUG:boto:Data: 2022-09-12T19:41:20.037 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.038 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:20.039 DEBUG:boto:Port: 8001 2022-09-12T19:41:20.039 DEBUG:boto:Params: {} 2022-09-12T19:41:20.040 DEBUG:boto:Token: None 2022-09-12T19:41:20.041 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/ 2022-09-12T19:41:20.043 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x/2zMU5BOFExU6QysA9NaCHa33E= 2022-09-12T19:41:20.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x/2zMU5BOFExU6QysA9NaCHa33E=', 'Content-Length': '0'} 2022-09-12T19:41:20.048 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b4bc53ae1d4500df-00631f8b60-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.049 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:20.050 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:20.051 DEBUG:boto:Method: HEAD 2022-09-12T19:41:20.052 DEBUG:boto:Path: /jdpiak-70/ 2022-09-12T19:41:20.053 DEBUG:boto:Data: 2022-09-12T19:41:20.054 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.055 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:20.056 DEBUG:boto:Port: 8000 2022-09-12T19:41:20.056 DEBUG:boto:Params: {} 2022-09-12T19:41:20.057 DEBUG:boto:Token: None 2022-09-12T19:41:20.059 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/ 2022-09-12T19:41:20.060 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x/2zMU5BOFExU6QysA9NaCHa33E= 2022-09-12T19:41:20.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x/2zMU5BOFExU6QysA9NaCHa33E=', 'Content-Length': '0'} 2022-09-12T19:41:20.065 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027d11a5982ee1682-00631f8b60-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.066 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:41:20.067 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:20.068 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:20.069 DEBUG:boto:path=/jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:20.070 DEBUG:boto:auth_path=/jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:20.071 DEBUG:boto:Method: GET 2022-09-12T19:41:20.072 DEBUG:boto:Path: /jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:20.073 DEBUG:boto:Data: 2022-09-12T19:41:20.074 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.075 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:20.077 DEBUG:boto:Port: 8001 2022-09-12T19:41:20.078 DEBUG:boto:Params: {} 2022-09-12T19:41:20.079 DEBUG:boto:Token: None 2022-09-12T19:41:20.080 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/?versions 2022-09-12T19:41:20.081 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RRWGShH2KArlfdJeHYcduhjD9Os= 2022-09-12T19:41:20.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RRWGShH2KArlfdJeHYcduhjD9Os=', 'Content-Length': '0'} 2022-09-12T19:41:20.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053a3c05b78c5c956-00631f8b60-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.087 DEBUG:boto:b'jdpiak-70999falseanulltrue2022-09-12T19:40:40.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:40:40.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:40:40.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:40:40.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:20.089 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:41:20.090 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:41:20.091 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:41:20.092 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:41:20.093 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:41:20.094 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:20.095 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:20.096 DEBUG:boto:path=/jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:20.097 DEBUG:boto:auth_path=/jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:20.098 DEBUG:boto:Method: GET 2022-09-12T19:41:20.099 DEBUG:boto:Path: /jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:20.100 DEBUG:boto:Data: 2022-09-12T19:41:20.101 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.102 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:20.103 DEBUG:boto:Port: 8000 2022-09-12T19:41:20.104 DEBUG:boto:Params: {} 2022-09-12T19:41:20.105 DEBUG:boto:Token: None 2022-09-12T19:41:20.106 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/?versions 2022-09-12T19:41:20.107 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RRWGShH2KArlfdJeHYcduhjD9Os= 2022-09-12T19:41:20.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RRWGShH2KArlfdJeHYcduhjD9Os=', 'Content-Length': '0'} 2022-09-12T19:41:20.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083207ae5864948e2-00631f8b60-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.113 DEBUG:boto:b'jdpiak-70999falseanulltrue2022-09-12T19:40:40.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:40:40.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:40:40.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:40:40.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:20.115 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:41:20.116 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:41:20.117 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:41:20.118 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:41:20.119 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:20.120 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:20.121 DEBUG:boto:path=/jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:20.121 DEBUG:boto:auth_path=/jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:20.122 DEBUG:boto:Method: GET 2022-09-12T19:41:20.123 DEBUG:boto:Path: /jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:20.124 DEBUG:boto:Data: 2022-09-12T19:41:20.125 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.126 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:20.127 DEBUG:boto:Port: 8001 2022-09-12T19:41:20.128 DEBUG:boto:Params: {} 2022-09-12T19:41:20.129 DEBUG:boto:Token: None 2022-09-12T19:41:20.130 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/?versions 2022-09-12T19:41:20.132 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RRWGShH2KArlfdJeHYcduhjD9Os= 2022-09-12T19:41:20.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RRWGShH2KArlfdJeHYcduhjD9Os=', 'Content-Length': '0'} 2022-09-12T19:41:20.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000deb6427024b23af4-00631f8b60-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.137 DEBUG:boto:b'jdpiak-70999falseanulltrue2022-09-12T19:40:40.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:40:40.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:40:40.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:40:40.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:20.139 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:20.140 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:20.141 DEBUG:boto:path=/jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:20.142 DEBUG:boto:auth_path=/jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:20.143 DEBUG:boto:Method: GET 2022-09-12T19:41:20.144 DEBUG:boto:Path: /jdpiak-70/?versions&max-keys=999 2022-09-12T19:41:20.145 DEBUG:boto:Data: 2022-09-12T19:41:20.146 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.147 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:20.148 DEBUG:boto:Port: 8000 2022-09-12T19:41:20.149 DEBUG:boto:Params: {} 2022-09-12T19:41:20.150 DEBUG:boto:Token: None 2022-09-12T19:41:20.152 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/?versions 2022-09-12T19:41:20.153 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RRWGShH2KArlfdJeHYcduhjD9Os= 2022-09-12T19:41:20.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RRWGShH2KArlfdJeHYcduhjD9Os=', 'Content-Length': '0'} 2022-09-12T19:41:20.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000876da52b413915e2-00631f8b60-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.159 DEBUG:boto:b'jdpiak-70999falseanulltrue2022-09-12T19:40:40.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:40:40.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:40:40.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:40:40.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:20.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:41:20.162 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:20.164 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:20.165 DEBUG:boto:path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:20.166 DEBUG:boto:auth_path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:20.167 DEBUG:boto:Method: GET 2022-09-12T19:41:20.168 DEBUG:boto:Path: /jdpiak-70/a?versionId=null 2022-09-12T19:41:20.169 DEBUG:boto:Data: 2022-09-12T19:41:20.170 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.171 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:20.172 DEBUG:boto:Port: 8001 2022-09-12T19:41:20.173 DEBUG:boto:Params: {} 2022-09-12T19:41:20.174 DEBUG:boto:Token: None 2022-09-12T19:41:20.175 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/a?versionId=null 2022-09-12T19:41:20.177 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MHXCjaUdTipiiB7ujCdbo2RRf7Y= 2022-09-12T19:41:20.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MHXCjaUdTipiiB7ujCdbo2RRf7Y=', 'Content-Length': '0'} 2022-09-12T19:41:20.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000241d7e55082cabe9-00631f8b60-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.182 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:20.183 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:20.184 DEBUG:boto:path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:20.185 DEBUG:boto:auth_path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:20.186 DEBUG:boto:Method: GET 2022-09-12T19:41:20.187 DEBUG:boto:Path: /jdpiak-70/a?versionId=null 2022-09-12T19:41:20.188 DEBUG:boto:Data: 2022-09-12T19:41:20.189 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.190 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:20.191 DEBUG:boto:Port: 8000 2022-09-12T19:41:20.192 DEBUG:boto:Params: {} 2022-09-12T19:41:20.193 DEBUG:boto:Token: None 2022-09-12T19:41:20.194 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/a?versionId=null 2022-09-12T19:41:20.195 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MHXCjaUdTipiiB7ujCdbo2RRf7Y= 2022-09-12T19:41:20.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MHXCjaUdTipiiB7ujCdbo2RRf7Y=', 'Content-Length': '0'} 2022-09-12T19:41:20.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001206d44f349e1ff9-00631f8b60-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.203 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:20.204 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:20.204 DEBUG:boto:path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:20.205 DEBUG:boto:auth_path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:20.207 DEBUG:boto:Method: HEAD 2022-09-12T19:41:20.208 DEBUG:boto:Path: /jdpiak-70/a?versionId=null 2022-09-12T19:41:20.209 DEBUG:boto:Data: 2022-09-12T19:41:20.209 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.210 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:20.210 DEBUG:boto:Port: 8001 2022-09-12T19:41:20.211 DEBUG:boto:Params: {} 2022-09-12T19:41:20.212 DEBUG:boto:Token: None 2022-09-12T19:41:20.212 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/a?versionId=null 2022-09-12T19:41:20.213 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CN9BKz2DpLq98B4TItJ/B7JLLEw= 2022-09-12T19:41:20.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CN9BKz2DpLq98B4TItJ/B7JLLEw=', 'Content-Length': '0'} 2022-09-12T19:41:20.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055062639b9581893-00631f8b60-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.219 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:20.219 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:20.220 DEBUG:boto:path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:20.220 DEBUG:boto:auth_path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:20.221 DEBUG:boto:Method: HEAD 2022-09-12T19:41:20.221 DEBUG:boto:Path: /jdpiak-70/a?versionId=null 2022-09-12T19:41:20.222 DEBUG:boto:Data: 2022-09-12T19:41:20.223 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.223 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:20.224 DEBUG:boto:Port: 8000 2022-09-12T19:41:20.224 DEBUG:boto:Params: {} 2022-09-12T19:41:20.225 DEBUG:boto:Token: None 2022-09-12T19:41:20.226 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/a?versionId=null 2022-09-12T19:41:20.228 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CN9BKz2DpLq98B4TItJ/B7JLLEw= 2022-09-12T19:41:20.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CN9BKz2DpLq98B4TItJ/B7JLLEw=', 'Content-Length': '0'} 2022-09-12T19:41:20.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e1164c0979749e0-00631f8b60-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:41:20.235 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:20.236 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:20.237 DEBUG:boto:path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:20.237 DEBUG:boto:auth_path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:20.238 DEBUG:boto:Method: GET 2022-09-12T19:41:20.239 DEBUG:boto:Path: /jdpiak-70/a?versionId=null 2022-09-12T19:41:20.240 DEBUG:boto:Data: 2022-09-12T19:41:20.241 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.242 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:20.244 DEBUG:boto:Port: 8001 2022-09-12T19:41:20.244 DEBUG:boto:Params: {} 2022-09-12T19:41:20.246 DEBUG:boto:Token: None 2022-09-12T19:41:20.247 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/a?versionId=null 2022-09-12T19:41:20.248 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MHXCjaUdTipiiB7ujCdbo2RRf7Y= 2022-09-12T19:41:20.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MHXCjaUdTipiiB7ujCdbo2RRf7Y=', 'Content-Length': '0'} 2022-09-12T19:41:20.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3f6c7a60d8311e4-00631f8b60-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.257 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:20.258 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:20.259 DEBUG:boto:path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:20.262 DEBUG:boto:auth_path=/jdpiak-70/a?versionId=null 2022-09-12T19:41:20.263 DEBUG:boto:Method: GET 2022-09-12T19:41:20.264 DEBUG:boto:Path: /jdpiak-70/a?versionId=null 2022-09-12T19:41:20.265 DEBUG:boto:Data: 2022-09-12T19:41:20.267 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.268 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:20.270 DEBUG:boto:Port: 8000 2022-09-12T19:41:20.273 DEBUG:boto:Params: {} 2022-09-12T19:41:20.274 DEBUG:boto:Token: None 2022-09-12T19:41:20.276 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/a?versionId=null 2022-09-12T19:41:20.277 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MHXCjaUdTipiiB7ujCdbo2RRf7Y= 2022-09-12T19:41:20.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MHXCjaUdTipiiB7ujCdbo2RRf7Y=', 'Content-Length': '0'} 2022-09-12T19:41:20.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8fa2ef05e5a6421-00631f8b60-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.289 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:20.290 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:20.294 DEBUG:boto:path=/jdpiak-70/?prefix=a 2022-09-12T19:41:20.295 DEBUG:boto:auth_path=/jdpiak-70/?prefix=a 2022-09-12T19:41:20.296 DEBUG:boto:Method: GET 2022-09-12T19:41:20.297 DEBUG:boto:Path: /jdpiak-70/?prefix=a 2022-09-12T19:41:20.299 DEBUG:boto:Data: 2022-09-12T19:41:20.300 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.301 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:20.302 DEBUG:boto:Port: 8001 2022-09-12T19:41:20.303 DEBUG:boto:Params: {} 2022-09-12T19:41:20.304 DEBUG:boto:Token: None 2022-09-12T19:41:20.306 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/ 2022-09-12T19:41:20.307 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AyDaFiScgaFSQrzv9+yCcDyHDFE= 2022-09-12T19:41:20.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AyDaFiScgaFSQrzv9+yCcDyHDFE=', 'Content-Length': '0'} 2022-09-12T19:41:20.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d4f93e5f1b13b43-00631f8b60-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.314 DEBUG:boto:b'jdpiak-70a1000falsea2022-09-12T19:40:40.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:20.316 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:20.317 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:20.318 DEBUG:boto:path=/jdpiak-70/?prefix=a 2022-09-12T19:41:20.319 DEBUG:boto:auth_path=/jdpiak-70/?prefix=a 2022-09-12T19:41:20.319 DEBUG:boto:Method: GET 2022-09-12T19:41:20.320 DEBUG:boto:Path: /jdpiak-70/?prefix=a 2022-09-12T19:41:20.320 DEBUG:boto:Data: 2022-09-12T19:41:20.321 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.321 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:20.322 DEBUG:boto:Port: 8000 2022-09-12T19:41:20.323 DEBUG:boto:Params: {} 2022-09-12T19:41:20.324 DEBUG:boto:Token: None 2022-09-12T19:41:20.325 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/ 2022-09-12T19:41:20.327 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AyDaFiScgaFSQrzv9+yCcDyHDFE= 2022-09-12T19:41:20.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AyDaFiScgaFSQrzv9+yCcDyHDFE=', 'Content-Length': '0'} 2022-09-12T19:41:20.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9f88b9d514c6dea-00631f8b60-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.334 DEBUG:boto:b'jdpiak-70a1000falsea2022-09-12T19:40:40.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:20.336 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:20.338 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:20.339 DEBUG:boto:Method: HEAD 2022-09-12T19:41:20.340 DEBUG:boto:Path: /jdpiak-70/a 2022-09-12T19:41:20.341 DEBUG:boto:Data: 2022-09-12T19:41:20.341 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.343 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:20.344 DEBUG:boto:Port: 8001 2022-09-12T19:41:20.345 DEBUG:boto:Params: {} 2022-09-12T19:41:20.346 DEBUG:boto:Token: None 2022-09-12T19:41:20.347 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/a 2022-09-12T19:41:20.348 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OMywAmCvUDLfss9WrRep8WG/a2c= 2022-09-12T19:41:20.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OMywAmCvUDLfss9WrRep8WG/a2c=', 'Content-Length': '0'} 2022-09-12T19:41:20.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c42e148649d9718-00631f8b60-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.355 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:20.356 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:20.357 DEBUG:boto:Method: HEAD 2022-09-12T19:41:20.358 DEBUG:boto:Path: /jdpiak-70/a 2022-09-12T19:41:20.359 DEBUG:boto:Data: 2022-09-12T19:41:20.359 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.360 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:20.360 DEBUG:boto:Port: 8000 2022-09-12T19:41:20.361 DEBUG:boto:Params: {} 2022-09-12T19:41:20.362 DEBUG:boto:Token: None 2022-09-12T19:41:20.362 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/a 2022-09-12T19:41:20.363 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OMywAmCvUDLfss9WrRep8WG/a2c= 2022-09-12T19:41:20.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OMywAmCvUDLfss9WrRep8WG/a2c=', 'Content-Length': '0'} 2022-09-12T19:41:20.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005eba87de55738994-00631f8b60-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:41:20.370 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:20.370 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:20.371 DEBUG:boto:Method: GET 2022-09-12T19:41:20.372 DEBUG:boto:Path: /jdpiak-70/a 2022-09-12T19:41:20.372 DEBUG:boto:Data: 2022-09-12T19:41:20.373 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.373 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:20.374 DEBUG:boto:Port: 8001 2022-09-12T19:41:20.375 DEBUG:boto:Params: {} 2022-09-12T19:41:20.376 DEBUG:boto:Token: None 2022-09-12T19:41:20.378 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/a 2022-09-12T19:41:20.379 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZHU/8xFzp56rurgC8SCwA4itqdA= 2022-09-12T19:41:20.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZHU/8xFzp56rurgC8SCwA4itqdA=', 'Content-Length': '0'} 2022-09-12T19:41:20.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8bae45948e33d2b-00631f8b60-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.385 DEBUG:boto:path=/jdpiak-70/a 2022-09-12T19:41:20.386 DEBUG:boto:auth_path=/jdpiak-70/a 2022-09-12T19:41:20.387 DEBUG:boto:Method: GET 2022-09-12T19:41:20.387 DEBUG:boto:Path: /jdpiak-70/a 2022-09-12T19:41:20.388 DEBUG:boto:Data: 2022-09-12T19:41:20.389 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.389 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:20.390 DEBUG:boto:Port: 8000 2022-09-12T19:41:20.391 DEBUG:boto:Params: {} 2022-09-12T19:41:20.391 DEBUG:boto:Token: None 2022-09-12T19:41:20.392 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/a 2022-09-12T19:41:20.393 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZHU/8xFzp56rurgC8SCwA4itqdA= 2022-09-12T19:41:20.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZHU/8xFzp56rurgC8SCwA4itqdA=', 'Content-Length': '0'} 2022-09-12T19:41:20.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019ffe1f7f4999601-00631f8b60-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.397 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:20.398 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:20.399 DEBUG:boto:path=/jdpiak-70/?prefix=a 2022-09-12T19:41:20.399 DEBUG:boto:auth_path=/jdpiak-70/?prefix=a 2022-09-12T19:41:20.400 DEBUG:boto:Method: GET 2022-09-12T19:41:20.400 DEBUG:boto:Path: /jdpiak-70/?prefix=a 2022-09-12T19:41:20.401 DEBUG:boto:Data: 2022-09-12T19:41:20.401 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.402 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:20.402 DEBUG:boto:Port: 8001 2022-09-12T19:41:20.403 DEBUG:boto:Params: {} 2022-09-12T19:41:20.404 DEBUG:boto:Token: None 2022-09-12T19:41:20.404 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/ 2022-09-12T19:41:20.405 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AyDaFiScgaFSQrzv9+yCcDyHDFE= 2022-09-12T19:41:20.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AyDaFiScgaFSQrzv9+yCcDyHDFE=', 'Content-Length': '0'} 2022-09-12T19:41:20.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c75a9361799cdba8-00631f8b60-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.409 DEBUG:boto:b'jdpiak-70a1000falsea2022-09-12T19:40:40.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:20.410 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:20.411 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:20.411 DEBUG:boto:path=/jdpiak-70/?prefix=a 2022-09-12T19:41:20.412 DEBUG:boto:auth_path=/jdpiak-70/?prefix=a 2022-09-12T19:41:20.412 DEBUG:boto:Method: GET 2022-09-12T19:41:20.413 DEBUG:boto:Path: /jdpiak-70/?prefix=a 2022-09-12T19:41:20.413 DEBUG:boto:Data: 2022-09-12T19:41:20.414 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.414 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:20.415 DEBUG:boto:Port: 8000 2022-09-12T19:41:20.416 DEBUG:boto:Params: {} 2022-09-12T19:41:20.417 DEBUG:boto:Token: None 2022-09-12T19:41:20.417 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/ 2022-09-12T19:41:20.418 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AyDaFiScgaFSQrzv9+yCcDyHDFE= 2022-09-12T19:41:20.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AyDaFiScgaFSQrzv9+yCcDyHDFE=', 'Content-Length': '0'} 2022-09-12T19:41:20.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000225a14c686ac3b69-00631f8b60-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.422 DEBUG:boto:b'jdpiak-70a1000falsea2022-09-12T19:40:40.552Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:20.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:41:20.423 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:20.424 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:20.425 DEBUG:boto:path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:20.425 DEBUG:boto:auth_path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:20.426 DEBUG:boto:Method: GET 2022-09-12T19:41:20.427 DEBUG:boto:Path: /jdpiak-70/b?versionId=null 2022-09-12T19:41:20.427 DEBUG:boto:Data: 2022-09-12T19:41:20.428 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.429 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:20.429 DEBUG:boto:Port: 8001 2022-09-12T19:41:20.430 DEBUG:boto:Params: {} 2022-09-12T19:41:20.430 DEBUG:boto:Token: None 2022-09-12T19:41:20.431 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/b?versionId=null 2022-09-12T19:41:20.432 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1/+5rqNh190ctvt0j7LizYpVjoU= 2022-09-12T19:41:20.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1/+5rqNh190ctvt0j7LizYpVjoU=', 'Content-Length': '0'} 2022-09-12T19:41:20.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078874929118ce4b9-00631f8b60-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.441 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:20.441 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:20.442 DEBUG:boto:path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:20.443 DEBUG:boto:auth_path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:20.444 DEBUG:boto:Method: GET 2022-09-12T19:41:20.445 DEBUG:boto:Path: /jdpiak-70/b?versionId=null 2022-09-12T19:41:20.446 DEBUG:boto:Data: 2022-09-12T19:41:20.447 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.448 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:20.449 DEBUG:boto:Port: 8000 2022-09-12T19:41:20.450 DEBUG:boto:Params: {} 2022-09-12T19:41:20.451 DEBUG:boto:Token: None 2022-09-12T19:41:20.452 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/b?versionId=null 2022-09-12T19:41:20.452 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1/+5rqNh190ctvt0j7LizYpVjoU= 2022-09-12T19:41:20.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1/+5rqNh190ctvt0j7LizYpVjoU=', 'Content-Length': '0'} 2022-09-12T19:41:20.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c5b6bb93a8dd154-00631f8b60-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.457 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:20.458 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:20.458 DEBUG:boto:path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:20.459 DEBUG:boto:auth_path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:20.460 DEBUG:boto:Method: HEAD 2022-09-12T19:41:20.461 DEBUG:boto:Path: /jdpiak-70/b?versionId=null 2022-09-12T19:41:20.461 DEBUG:boto:Data: 2022-09-12T19:41:20.462 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.463 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:20.463 DEBUG:boto:Port: 8001 2022-09-12T19:41:20.464 DEBUG:boto:Params: {} 2022-09-12T19:41:20.464 DEBUG:boto:Token: None 2022-09-12T19:41:20.465 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/b?versionId=null 2022-09-12T19:41:20.465 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KrrP1ZT7G46lPOkHNyGvte5MpYY= 2022-09-12T19:41:20.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KrrP1ZT7G46lPOkHNyGvte5MpYY=', 'Content-Length': '0'} 2022-09-12T19:41:20.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090ab5298fccc8b29-00631f8b60-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.470 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:20.471 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:20.472 DEBUG:boto:path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:20.473 DEBUG:boto:auth_path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:20.474 DEBUG:boto:Method: HEAD 2022-09-12T19:41:20.475 DEBUG:boto:Path: /jdpiak-70/b?versionId=null 2022-09-12T19:41:20.476 DEBUG:boto:Data: 2022-09-12T19:41:20.477 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.478 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:20.479 DEBUG:boto:Port: 8000 2022-09-12T19:41:20.480 DEBUG:boto:Params: {} 2022-09-12T19:41:20.481 DEBUG:boto:Token: None 2022-09-12T19:41:20.482 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/b?versionId=null 2022-09-12T19:41:20.483 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KrrP1ZT7G46lPOkHNyGvte5MpYY= 2022-09-12T19:41:20.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KrrP1ZT7G46lPOkHNyGvte5MpYY=', 'Content-Length': '0'} 2022-09-12T19:41:20.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012f9f7fae62774ab-00631f8b60-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:41:20.487 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:20.488 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:20.489 DEBUG:boto:path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:20.489 DEBUG:boto:auth_path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:20.490 DEBUG:boto:Method: GET 2022-09-12T19:41:20.490 DEBUG:boto:Path: /jdpiak-70/b?versionId=null 2022-09-12T19:41:20.491 DEBUG:boto:Data: 2022-09-12T19:41:20.492 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.493 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:20.493 DEBUG:boto:Port: 8001 2022-09-12T19:41:20.494 DEBUG:boto:Params: {} 2022-09-12T19:41:20.494 DEBUG:boto:Token: None 2022-09-12T19:41:20.495 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/b?versionId=null 2022-09-12T19:41:20.496 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1/+5rqNh190ctvt0j7LizYpVjoU= 2022-09-12T19:41:20.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1/+5rqNh190ctvt0j7LizYpVjoU=', 'Content-Length': '0'} 2022-09-12T19:41:20.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059f6d6e451526e6d-00631f8b60-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.542 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:20.543 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:20.544 DEBUG:boto:path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:20.544 DEBUG:boto:auth_path=/jdpiak-70/b?versionId=null 2022-09-12T19:41:20.545 DEBUG:boto:Method: GET 2022-09-12T19:41:20.546 DEBUG:boto:Path: /jdpiak-70/b?versionId=null 2022-09-12T19:41:20.547 DEBUG:boto:Data: 2022-09-12T19:41:20.547 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.548 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:20.548 DEBUG:boto:Port: 8000 2022-09-12T19:41:20.549 DEBUG:boto:Params: {} 2022-09-12T19:41:20.550 DEBUG:boto:Token: None 2022-09-12T19:41:20.550 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/b?versionId=null 2022-09-12T19:41:20.551 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1/+5rqNh190ctvt0j7LizYpVjoU= 2022-09-12T19:41:20.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1/+5rqNh190ctvt0j7LizYpVjoU=', 'Content-Length': '0'} 2022-09-12T19:41:20.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b831604e82b4717-00631f8b60-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.558 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:20.558 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:20.559 DEBUG:boto:path=/jdpiak-70/?prefix=b 2022-09-12T19:41:20.560 DEBUG:boto:auth_path=/jdpiak-70/?prefix=b 2022-09-12T19:41:20.560 DEBUG:boto:Method: GET 2022-09-12T19:41:20.561 DEBUG:boto:Path: /jdpiak-70/?prefix=b 2022-09-12T19:41:20.562 DEBUG:boto:Data: 2022-09-12T19:41:20.563 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.564 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:20.565 DEBUG:boto:Port: 8001 2022-09-12T19:41:20.566 DEBUG:boto:Params: {} 2022-09-12T19:41:20.567 DEBUG:boto:Token: None 2022-09-12T19:41:20.569 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/ 2022-09-12T19:41:20.570 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AyDaFiScgaFSQrzv9+yCcDyHDFE= 2022-09-12T19:41:20.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AyDaFiScgaFSQrzv9+yCcDyHDFE=', 'Content-Length': '0'} 2022-09-12T19:41:20.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000934b71222eb6c4b-00631f8b60-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.575 DEBUG:boto:b'jdpiak-70b1000falseb2022-09-12T19:40:40.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:20.576 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:20.576 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:20.577 DEBUG:boto:path=/jdpiak-70/?prefix=b 2022-09-12T19:41:20.578 DEBUG:boto:auth_path=/jdpiak-70/?prefix=b 2022-09-12T19:41:20.579 DEBUG:boto:Method: GET 2022-09-12T19:41:20.579 DEBUG:boto:Path: /jdpiak-70/?prefix=b 2022-09-12T19:41:20.580 DEBUG:boto:Data: 2022-09-12T19:41:20.580 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.581 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:20.582 DEBUG:boto:Port: 8000 2022-09-12T19:41:20.583 DEBUG:boto:Params: {} 2022-09-12T19:41:20.584 DEBUG:boto:Token: None 2022-09-12T19:41:20.585 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/ 2022-09-12T19:41:20.587 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AyDaFiScgaFSQrzv9+yCcDyHDFE= 2022-09-12T19:41:20.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AyDaFiScgaFSQrzv9+yCcDyHDFE=', 'Content-Length': '0'} 2022-09-12T19:41:20.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000199da65813b1795c-00631f8b60-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.593 DEBUG:boto:b'jdpiak-70b1000falseb2022-09-12T19:40:40.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:20.594 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:20.594 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:20.595 DEBUG:boto:Method: HEAD 2022-09-12T19:41:20.595 DEBUG:boto:Path: /jdpiak-70/b 2022-09-12T19:41:20.596 DEBUG:boto:Data: 2022-09-12T19:41:20.597 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.597 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:20.598 DEBUG:boto:Port: 8001 2022-09-12T19:41:20.599 DEBUG:boto:Params: {} 2022-09-12T19:41:20.599 DEBUG:boto:Token: None 2022-09-12T19:41:20.600 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/b 2022-09-12T19:41:20.600 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/gilgR6filwJkyRJjEcZng+znQ4= 2022-09-12T19:41:20.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/gilgR6filwJkyRJjEcZng+znQ4=', 'Content-Length': '0'} 2022-09-12T19:41:20.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a2efadc3bf4ccc0-00631f8b60-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.604 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:20.606 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:20.606 DEBUG:boto:Method: HEAD 2022-09-12T19:41:20.607 DEBUG:boto:Path: /jdpiak-70/b 2022-09-12T19:41:20.608 DEBUG:boto:Data: 2022-09-12T19:41:20.608 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.609 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:20.610 DEBUG:boto:Port: 8000 2022-09-12T19:41:20.610 DEBUG:boto:Params: {} 2022-09-12T19:41:20.611 DEBUG:boto:Token: None 2022-09-12T19:41:20.611 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/b 2022-09-12T19:41:20.612 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/gilgR6filwJkyRJjEcZng+znQ4= 2022-09-12T19:41:20.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/gilgR6filwJkyRJjEcZng+znQ4=', 'Content-Length': '0'} 2022-09-12T19:41:20.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6aed1aa63d5f56c-00631f8b60-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:41:20.617 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:20.617 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:20.618 DEBUG:boto:Method: GET 2022-09-12T19:41:20.619 DEBUG:boto:Path: /jdpiak-70/b 2022-09-12T19:41:20.619 DEBUG:boto:Data: 2022-09-12T19:41:20.620 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.620 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:20.621 DEBUG:boto:Port: 8001 2022-09-12T19:41:20.621 DEBUG:boto:Params: {} 2022-09-12T19:41:20.622 DEBUG:boto:Token: None 2022-09-12T19:41:20.623 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/b 2022-09-12T19:41:20.623 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+Xhiq3Bp/JLwqXTnRomfASfTX8k= 2022-09-12T19:41:20.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+Xhiq3Bp/JLwqXTnRomfASfTX8k=', 'Content-Length': '0'} 2022-09-12T19:41:20.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1ac27c5542000f7-00631f8b60-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.628 DEBUG:boto:path=/jdpiak-70/b 2022-09-12T19:41:20.629 DEBUG:boto:auth_path=/jdpiak-70/b 2022-09-12T19:41:20.629 DEBUG:boto:Method: GET 2022-09-12T19:41:20.631 DEBUG:boto:Path: /jdpiak-70/b 2022-09-12T19:41:20.634 DEBUG:boto:Data: 2022-09-12T19:41:20.634 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.635 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:20.636 DEBUG:boto:Port: 8000 2022-09-12T19:41:20.637 DEBUG:boto:Params: {} 2022-09-12T19:41:20.638 DEBUG:boto:Token: None 2022-09-12T19:41:20.638 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/b 2022-09-12T19:41:20.639 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+Xhiq3Bp/JLwqXTnRomfASfTX8k= 2022-09-12T19:41:20.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+Xhiq3Bp/JLwqXTnRomfASfTX8k=', 'Content-Length': '0'} 2022-09-12T19:41:20.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8447a4eae643ffa-00631f8b60-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.691 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:20.692 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:20.693 DEBUG:boto:path=/jdpiak-70/?prefix=b 2022-09-12T19:41:20.693 DEBUG:boto:auth_path=/jdpiak-70/?prefix=b 2022-09-12T19:41:20.694 DEBUG:boto:Method: GET 2022-09-12T19:41:20.695 DEBUG:boto:Path: /jdpiak-70/?prefix=b 2022-09-12T19:41:20.695 DEBUG:boto:Data: 2022-09-12T19:41:20.696 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.696 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:20.697 DEBUG:boto:Port: 8001 2022-09-12T19:41:20.697 DEBUG:boto:Params: {} 2022-09-12T19:41:20.698 DEBUG:boto:Token: None 2022-09-12T19:41:20.699 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/ 2022-09-12T19:41:20.699 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AyDaFiScgaFSQrzv9+yCcDyHDFE= 2022-09-12T19:41:20.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AyDaFiScgaFSQrzv9+yCcDyHDFE=', 'Content-Length': '0'} 2022-09-12T19:41:20.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072df9b34a11078bf-00631f8b60-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.705 DEBUG:boto:b'jdpiak-70b1000falseb2022-09-12T19:40:40.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:20.706 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:20.707 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:20.708 DEBUG:boto:path=/jdpiak-70/?prefix=b 2022-09-12T19:41:20.709 DEBUG:boto:auth_path=/jdpiak-70/?prefix=b 2022-09-12T19:41:20.710 DEBUG:boto:Method: GET 2022-09-12T19:41:20.711 DEBUG:boto:Path: /jdpiak-70/?prefix=b 2022-09-12T19:41:20.712 DEBUG:boto:Data: 2022-09-12T19:41:20.713 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.714 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:20.715 DEBUG:boto:Port: 8000 2022-09-12T19:41:20.716 DEBUG:boto:Params: {} 2022-09-12T19:41:20.717 DEBUG:boto:Token: None 2022-09-12T19:41:20.718 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/ 2022-09-12T19:41:20.719 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AyDaFiScgaFSQrzv9+yCcDyHDFE= 2022-09-12T19:41:20.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AyDaFiScgaFSQrzv9+yCcDyHDFE=', 'Content-Length': '0'} 2022-09-12T19:41:20.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059b81b14a514f5b2-00631f8b60-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.724 DEBUG:boto:b'jdpiak-70b1000falseb2022-09-12T19:40:40.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:20.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:41:20.727 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:20.728 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:20.729 DEBUG:boto:path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:20.730 DEBUG:boto:auth_path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:20.731 DEBUG:boto:Method: GET 2022-09-12T19:41:20.731 DEBUG:boto:Path: /jdpiak-70/c?versionId=null 2022-09-12T19:41:20.732 DEBUG:boto:Data: 2022-09-12T19:41:20.732 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.733 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:20.734 DEBUG:boto:Port: 8001 2022-09-12T19:41:20.734 DEBUG:boto:Params: {} 2022-09-12T19:41:20.735 DEBUG:boto:Token: None 2022-09-12T19:41:20.736 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/c?versionId=null 2022-09-12T19:41:20.737 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:njTWExhPgmaFIg+5s6y+cOMp3Ds= 2022-09-12T19:41:20.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:njTWExhPgmaFIg+5s6y+cOMp3Ds=', 'Content-Length': '0'} 2022-09-12T19:41:20.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003058817bc3d896bb-00631f8b60-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.743 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:20.744 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:20.745 DEBUG:boto:path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:20.747 DEBUG:boto:auth_path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:20.747 DEBUG:boto:Method: GET 2022-09-12T19:41:20.748 DEBUG:boto:Path: /jdpiak-70/c?versionId=null 2022-09-12T19:41:20.749 DEBUG:boto:Data: 2022-09-12T19:41:20.750 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.750 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:20.751 DEBUG:boto:Port: 8000 2022-09-12T19:41:20.752 DEBUG:boto:Params: {} 2022-09-12T19:41:20.752 DEBUG:boto:Token: None 2022-09-12T19:41:20.753 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/c?versionId=null 2022-09-12T19:41:20.754 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:njTWExhPgmaFIg+5s6y+cOMp3Ds= 2022-09-12T19:41:20.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:njTWExhPgmaFIg+5s6y+cOMp3Ds=', 'Content-Length': '0'} 2022-09-12T19:41:20.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ca2c0250f61278d-00631f8b60-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.758 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:20.759 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:20.760 DEBUG:boto:path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:20.760 DEBUG:boto:auth_path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:20.761 DEBUG:boto:Method: HEAD 2022-09-12T19:41:20.761 DEBUG:boto:Path: /jdpiak-70/c?versionId=null 2022-09-12T19:41:20.762 DEBUG:boto:Data: 2022-09-12T19:41:20.762 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.763 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:20.763 DEBUG:boto:Port: 8001 2022-09-12T19:41:20.764 DEBUG:boto:Params: {} 2022-09-12T19:41:20.765 DEBUG:boto:Token: None 2022-09-12T19:41:20.766 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/c?versionId=null 2022-09-12T19:41:20.767 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rGSH45IRePwYO2kj4suP8hcqtQ8= 2022-09-12T19:41:20.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rGSH45IRePwYO2kj4suP8hcqtQ8=', 'Content-Length': '0'} 2022-09-12T19:41:20.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a58a518621b5d9b9-00631f8b60-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.771 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:20.772 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:20.773 DEBUG:boto:path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:20.773 DEBUG:boto:auth_path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:20.774 DEBUG:boto:Method: HEAD 2022-09-12T19:41:20.776 DEBUG:boto:Path: /jdpiak-70/c?versionId=null 2022-09-12T19:41:20.777 DEBUG:boto:Data: 2022-09-12T19:41:20.777 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.778 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:20.778 DEBUG:boto:Port: 8000 2022-09-12T19:41:20.779 DEBUG:boto:Params: {} 2022-09-12T19:41:20.780 DEBUG:boto:Token: None 2022-09-12T19:41:20.781 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/c?versionId=null 2022-09-12T19:41:20.781 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rGSH45IRePwYO2kj4suP8hcqtQ8= 2022-09-12T19:41:20.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rGSH45IRePwYO2kj4suP8hcqtQ8=', 'Content-Length': '0'} 2022-09-12T19:41:20.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000544b90d308f5330d-00631f8b60-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:41:20.786 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:20.786 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:20.787 DEBUG:boto:path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:20.788 DEBUG:boto:auth_path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:20.788 DEBUG:boto:Method: GET 2022-09-12T19:41:20.789 DEBUG:boto:Path: /jdpiak-70/c?versionId=null 2022-09-12T19:41:20.790 DEBUG:boto:Data: 2022-09-12T19:41:20.790 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.791 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:20.792 DEBUG:boto:Port: 8001 2022-09-12T19:41:20.792 DEBUG:boto:Params: {} 2022-09-12T19:41:20.793 DEBUG:boto:Token: None 2022-09-12T19:41:20.794 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/c?versionId=null 2022-09-12T19:41:20.795 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:njTWExhPgmaFIg+5s6y+cOMp3Ds= 2022-09-12T19:41:20.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:njTWExhPgmaFIg+5s6y+cOMp3Ds=', 'Content-Length': '0'} 2022-09-12T19:41:20.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec9e50b0e911ab02-00631f8b60-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.800 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:20.801 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:20.802 DEBUG:boto:path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:20.803 DEBUG:boto:auth_path=/jdpiak-70/c?versionId=null 2022-09-12T19:41:20.804 DEBUG:boto:Method: GET 2022-09-12T19:41:20.805 DEBUG:boto:Path: /jdpiak-70/c?versionId=null 2022-09-12T19:41:20.806 DEBUG:boto:Data: 2022-09-12T19:41:20.807 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.809 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:20.810 DEBUG:boto:Port: 8000 2022-09-12T19:41:20.811 DEBUG:boto:Params: {} 2022-09-12T19:41:20.812 DEBUG:boto:Token: None 2022-09-12T19:41:20.814 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/c?versionId=null 2022-09-12T19:41:20.815 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:njTWExhPgmaFIg+5s6y+cOMp3Ds= 2022-09-12T19:41:20.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:njTWExhPgmaFIg+5s6y+cOMp3Ds=', 'Content-Length': '0'} 2022-09-12T19:41:20.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f68f8a31393915da-00631f8b60-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.820 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:20.821 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:20.821 DEBUG:boto:path=/jdpiak-70/?prefix=c 2022-09-12T19:41:20.822 DEBUG:boto:auth_path=/jdpiak-70/?prefix=c 2022-09-12T19:41:20.823 DEBUG:boto:Method: GET 2022-09-12T19:41:20.823 DEBUG:boto:Path: /jdpiak-70/?prefix=c 2022-09-12T19:41:20.824 DEBUG:boto:Data: 2022-09-12T19:41:20.824 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.825 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:20.826 DEBUG:boto:Port: 8001 2022-09-12T19:41:20.826 DEBUG:boto:Params: {} 2022-09-12T19:41:20.827 DEBUG:boto:Token: None 2022-09-12T19:41:20.828 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/ 2022-09-12T19:41:20.828 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AyDaFiScgaFSQrzv9+yCcDyHDFE= 2022-09-12T19:41:20.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AyDaFiScgaFSQrzv9+yCcDyHDFE=', 'Content-Length': '0'} 2022-09-12T19:41:20.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040bb9cf4378d8386-00631f8b60-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.833 DEBUG:boto:b'jdpiak-70c1000falsec2022-09-12T19:40:40.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:20.834 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:20.835 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:20.835 DEBUG:boto:path=/jdpiak-70/?prefix=c 2022-09-12T19:41:20.836 DEBUG:boto:auth_path=/jdpiak-70/?prefix=c 2022-09-12T19:41:20.836 DEBUG:boto:Method: GET 2022-09-12T19:41:20.837 DEBUG:boto:Path: /jdpiak-70/?prefix=c 2022-09-12T19:41:20.837 DEBUG:boto:Data: 2022-09-12T19:41:20.838 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.838 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:20.839 DEBUG:boto:Port: 8000 2022-09-12T19:41:20.840 DEBUG:boto:Params: {} 2022-09-12T19:41:20.840 DEBUG:boto:Token: None 2022-09-12T19:41:20.841 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/ 2022-09-12T19:41:20.842 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AyDaFiScgaFSQrzv9+yCcDyHDFE= 2022-09-12T19:41:20.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AyDaFiScgaFSQrzv9+yCcDyHDFE=', 'Content-Length': '0'} 2022-09-12T19:41:20.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089c1e7698b7a39d7-00631f8b60-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.890 DEBUG:boto:b'jdpiak-70c1000falsec2022-09-12T19:40:40.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:20.892 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:20.893 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:20.894 DEBUG:boto:Method: HEAD 2022-09-12T19:41:20.895 DEBUG:boto:Path: /jdpiak-70/c 2022-09-12T19:41:20.896 DEBUG:boto:Data: 2022-09-12T19:41:20.896 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.898 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:20.899 DEBUG:boto:Port: 8001 2022-09-12T19:41:20.900 DEBUG:boto:Params: {} 2022-09-12T19:41:20.901 DEBUG:boto:Token: None 2022-09-12T19:41:20.901 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/c 2022-09-12T19:41:20.902 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WMYRDZbILvrewksMLlsmadEkj+o= 2022-09-12T19:41:20.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WMYRDZbILvrewksMLlsmadEkj+o=', 'Content-Length': '0'} 2022-09-12T19:41:20.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037120298a289855d-00631f8b60-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.907 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:20.908 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:20.908 DEBUG:boto:Method: HEAD 2022-09-12T19:41:20.909 DEBUG:boto:Path: /jdpiak-70/c 2022-09-12T19:41:20.909 DEBUG:boto:Data: 2022-09-12T19:41:20.910 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.911 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:20.912 DEBUG:boto:Port: 8000 2022-09-12T19:41:20.913 DEBUG:boto:Params: {} 2022-09-12T19:41:20.914 DEBUG:boto:Token: None 2022-09-12T19:41:20.915 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/c 2022-09-12T19:41:20.916 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WMYRDZbILvrewksMLlsmadEkj+o= 2022-09-12T19:41:20.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WMYRDZbILvrewksMLlsmadEkj+o=', 'Content-Length': '0'} 2022-09-12T19:41:20.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3e98a56634fbac5-00631f8b60-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:41:20.921 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:20.922 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:20.923 DEBUG:boto:Method: GET 2022-09-12T19:41:20.924 DEBUG:boto:Path: /jdpiak-70/c 2022-09-12T19:41:20.925 DEBUG:boto:Data: 2022-09-12T19:41:20.926 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.926 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:20.927 DEBUG:boto:Port: 8001 2022-09-12T19:41:20.927 DEBUG:boto:Params: {} 2022-09-12T19:41:20.928 DEBUG:boto:Token: None 2022-09-12T19:41:20.929 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/c 2022-09-12T19:41:20.930 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rtnoADIlRWzPT0rq2wZ8FI3LPTs= 2022-09-12T19:41:20.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rtnoADIlRWzPT0rq2wZ8FI3LPTs=', 'Content-Length': '0'} 2022-09-12T19:41:20.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3396ee01259615d-00631f8b60-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:20.986 DEBUG:boto:path=/jdpiak-70/c 2022-09-12T19:41:20.987 DEBUG:boto:auth_path=/jdpiak-70/c 2022-09-12T19:41:20.990 DEBUG:boto:Method: GET 2022-09-12T19:41:20.991 DEBUG:boto:Path: /jdpiak-70/c 2022-09-12T19:41:20.992 DEBUG:boto:Data: 2022-09-12T19:41:20.992 DEBUG:boto:Headers: {} 2022-09-12T19:41:20.993 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:20.993 DEBUG:boto:Port: 8000 2022-09-12T19:41:20.994 DEBUG:boto:Params: {} 2022-09-12T19:41:20.995 DEBUG:boto:Token: None 2022-09-12T19:41:20.995 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:20 GMT /jdpiak-70/c 2022-09-12T19:41:20.996 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rtnoADIlRWzPT0rq2wZ8FI3LPTs= 2022-09-12T19:41:20.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rtnoADIlRWzPT0rq2wZ8FI3LPTs=', 'Content-Length': '0'} 2022-09-12T19:41:21.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eab3477695680ea1-00631f8b61-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:21.005 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:21.006 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:21.007 DEBUG:boto:path=/jdpiak-70/?prefix=c 2022-09-12T19:41:21.008 DEBUG:boto:auth_path=/jdpiak-70/?prefix=c 2022-09-12T19:41:21.009 DEBUG:boto:Method: GET 2022-09-12T19:41:21.010 DEBUG:boto:Path: /jdpiak-70/?prefix=c 2022-09-12T19:41:21.011 DEBUG:boto:Data: 2022-09-12T19:41:21.012 DEBUG:boto:Headers: {} 2022-09-12T19:41:21.012 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:21.014 DEBUG:boto:Port: 8001 2022-09-12T19:41:21.014 DEBUG:boto:Params: {} 2022-09-12T19:41:21.015 DEBUG:boto:Token: None 2022-09-12T19:41:21.016 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:21 GMT /jdpiak-70/ 2022-09-12T19:41:21.017 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JJSnSCdzi7fwRnnm1eqKvh37gyY= 2022-09-12T19:41:21.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JJSnSCdzi7fwRnnm1eqKvh37gyY=', 'Content-Length': '0'} 2022-09-12T19:41:21.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ca6779a775407ca-00631f8b61-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:21.028 DEBUG:boto:b'jdpiak-70c1000falsec2022-09-12T19:40:40.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:21.029 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:21.030 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:21.031 DEBUG:boto:path=/jdpiak-70/?prefix=c 2022-09-12T19:41:21.032 DEBUG:boto:auth_path=/jdpiak-70/?prefix=c 2022-09-12T19:41:21.033 DEBUG:boto:Method: GET 2022-09-12T19:41:21.034 DEBUG:boto:Path: /jdpiak-70/?prefix=c 2022-09-12T19:41:21.035 DEBUG:boto:Data: 2022-09-12T19:41:21.036 DEBUG:boto:Headers: {} 2022-09-12T19:41:21.036 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:21.037 DEBUG:boto:Port: 8000 2022-09-12T19:41:21.038 DEBUG:boto:Params: {} 2022-09-12T19:41:21.039 DEBUG:boto:Token: None 2022-09-12T19:41:21.040 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:21 GMT /jdpiak-70/ 2022-09-12T19:41:21.041 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JJSnSCdzi7fwRnnm1eqKvh37gyY= 2022-09-12T19:41:21.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JJSnSCdzi7fwRnnm1eqKvh37gyY=', 'Content-Length': '0'} 2022-09-12T19:41:21.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015da5be50e251742-00631f8b61-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:21.049 DEBUG:boto:b'jdpiak-70c1000falsec2022-09-12T19:40:40.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:21.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:41:21.050 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:21.051 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:21.051 DEBUG:boto:path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:21.052 DEBUG:boto:auth_path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:21.053 DEBUG:boto:Method: GET 2022-09-12T19:41:21.053 DEBUG:boto:Path: /jdpiak-70/d?versionId=null 2022-09-12T19:41:21.054 DEBUG:boto:Data: 2022-09-12T19:41:21.054 DEBUG:boto:Headers: {} 2022-09-12T19:41:21.055 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:21.055 DEBUG:boto:Port: 8001 2022-09-12T19:41:21.056 DEBUG:boto:Params: {} 2022-09-12T19:41:21.057 DEBUG:boto:Token: None 2022-09-12T19:41:21.057 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:21 GMT /jdpiak-70/d?versionId=null 2022-09-12T19:41:21.058 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dhDjvjtiHg5kJPKk4DAfyuaYniU= 2022-09-12T19:41:21.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dhDjvjtiHg5kJPKk4DAfyuaYniU=', 'Content-Length': '0'} 2022-09-12T19:41:21.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043125a7a479be0a8-00631f8b61-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:21.073 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:21.074 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:21.074 DEBUG:boto:path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:21.075 DEBUG:boto:auth_path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:21.075 DEBUG:boto:Method: GET 2022-09-12T19:41:21.076 DEBUG:boto:Path: /jdpiak-70/d?versionId=null 2022-09-12T19:41:21.077 DEBUG:boto:Data: 2022-09-12T19:41:21.078 DEBUG:boto:Headers: {} 2022-09-12T19:41:21.078 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:21.079 DEBUG:boto:Port: 8000 2022-09-12T19:41:21.079 DEBUG:boto:Params: {} 2022-09-12T19:41:21.080 DEBUG:boto:Token: None 2022-09-12T19:41:21.081 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:21 GMT /jdpiak-70/d?versionId=null 2022-09-12T19:41:21.081 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dhDjvjtiHg5kJPKk4DAfyuaYniU= 2022-09-12T19:41:21.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dhDjvjtiHg5kJPKk4DAfyuaYniU=', 'Content-Length': '0'} 2022-09-12T19:41:21.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec3af7e9f5ec2c16-00631f8b61-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:21.134 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:21.135 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:21.136 DEBUG:boto:path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:21.137 DEBUG:boto:auth_path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:21.140 DEBUG:boto:Method: HEAD 2022-09-12T19:41:21.141 DEBUG:boto:Path: /jdpiak-70/d?versionId=null 2022-09-12T19:41:21.142 DEBUG:boto:Data: 2022-09-12T19:41:21.143 DEBUG:boto:Headers: {} 2022-09-12T19:41:21.144 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:21.145 DEBUG:boto:Port: 8001 2022-09-12T19:41:21.146 DEBUG:boto:Params: {} 2022-09-12T19:41:21.147 DEBUG:boto:Token: None 2022-09-12T19:41:21.148 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:21 GMT /jdpiak-70/d?versionId=null 2022-09-12T19:41:21.149 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9cE6H8un2NIedwbxq7qKy8JERIw= 2022-09-12T19:41:21.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9cE6H8un2NIedwbxq7qKy8JERIw=', 'Content-Length': '0'} 2022-09-12T19:41:21.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023c55ae5d4905ed3-00631f8b61-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:21.160 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:21.161 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:21.162 DEBUG:boto:path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:21.163 DEBUG:boto:auth_path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:21.164 DEBUG:boto:Method: HEAD 2022-09-12T19:41:21.165 DEBUG:boto:Path: /jdpiak-70/d?versionId=null 2022-09-12T19:41:21.165 DEBUG:boto:Data: 2022-09-12T19:41:21.178 DEBUG:boto:Headers: {} 2022-09-12T19:41:21.190 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:21.191 DEBUG:boto:Port: 8000 2022-09-12T19:41:21.192 DEBUG:boto:Params: {} 2022-09-12T19:41:21.193 DEBUG:boto:Token: None 2022-09-12T19:41:21.197 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:21 GMT /jdpiak-70/d?versionId=null 2022-09-12T19:41:21.198 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9cE6H8un2NIedwbxq7qKy8JERIw= 2022-09-12T19:41:21.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9cE6H8un2NIedwbxq7qKy8JERIw=', 'Content-Length': '0'} 2022-09-12T19:41:21.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d120730a8e430788-00631f8b61-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:21.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:41:21.204 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:21.205 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:21.206 DEBUG:boto:path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:21.207 DEBUG:boto:auth_path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:21.208 DEBUG:boto:Method: GET 2022-09-12T19:41:21.209 DEBUG:boto:Path: /jdpiak-70/d?versionId=null 2022-09-12T19:41:21.211 DEBUG:boto:Data: 2022-09-12T19:41:21.212 DEBUG:boto:Headers: {} 2022-09-12T19:41:21.213 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:21.214 DEBUG:boto:Port: 8001 2022-09-12T19:41:21.215 DEBUG:boto:Params: {} 2022-09-12T19:41:21.216 DEBUG:boto:Token: None 2022-09-12T19:41:21.217 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:21 GMT /jdpiak-70/d?versionId=null 2022-09-12T19:41:21.218 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dhDjvjtiHg5kJPKk4DAfyuaYniU= 2022-09-12T19:41:21.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dhDjvjtiHg5kJPKk4DAfyuaYniU=', 'Content-Length': '0'} 2022-09-12T19:41:21.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000405f8c07004757a6-00631f8b61-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:21.224 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:21.225 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:21.226 DEBUG:boto:path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:21.227 DEBUG:boto:auth_path=/jdpiak-70/d?versionId=null 2022-09-12T19:41:21.228 DEBUG:boto:Method: GET 2022-09-12T19:41:21.229 DEBUG:boto:Path: /jdpiak-70/d?versionId=null 2022-09-12T19:41:21.230 DEBUG:boto:Data: 2022-09-12T19:41:21.231 DEBUG:boto:Headers: {} 2022-09-12T19:41:21.232 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:21.233 DEBUG:boto:Port: 8000 2022-09-12T19:41:21.234 DEBUG:boto:Params: {} 2022-09-12T19:41:21.235 DEBUG:boto:Token: None 2022-09-12T19:41:21.236 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:21 GMT /jdpiak-70/d?versionId=null 2022-09-12T19:41:21.237 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dhDjvjtiHg5kJPKk4DAfyuaYniU= 2022-09-12T19:41:21.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dhDjvjtiHg5kJPKk4DAfyuaYniU=', 'Content-Length': '0'} 2022-09-12T19:41:21.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068d885a81afd4e59-00631f8b61-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:21.243 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:21.244 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:21.245 DEBUG:boto:path=/jdpiak-70/?prefix=d 2022-09-12T19:41:21.245 DEBUG:boto:auth_path=/jdpiak-70/?prefix=d 2022-09-12T19:41:21.246 DEBUG:boto:Method: GET 2022-09-12T19:41:21.247 DEBUG:boto:Path: /jdpiak-70/?prefix=d 2022-09-12T19:41:21.248 DEBUG:boto:Data: 2022-09-12T19:41:21.249 DEBUG:boto:Headers: {} 2022-09-12T19:41:21.250 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:21.251 DEBUG:boto:Port: 8001 2022-09-12T19:41:21.252 DEBUG:boto:Params: {} 2022-09-12T19:41:21.254 DEBUG:boto:Token: None 2022-09-12T19:41:21.255 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:21 GMT /jdpiak-70/ 2022-09-12T19:41:21.256 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JJSnSCdzi7fwRnnm1eqKvh37gyY= 2022-09-12T19:41:21.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JJSnSCdzi7fwRnnm1eqKvh37gyY=', 'Content-Length': '0'} 2022-09-12T19:41:21.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000170cee8ac7ce6e79-00631f8b61-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:21.262 DEBUG:boto:b'jdpiak-70d1000falsed2022-09-12T19:40:40.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:21.264 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:21.265 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:21.267 DEBUG:boto:path=/jdpiak-70/?prefix=d 2022-09-12T19:41:21.268 DEBUG:boto:auth_path=/jdpiak-70/?prefix=d 2022-09-12T19:41:21.269 DEBUG:boto:Method: GET 2022-09-12T19:41:21.270 DEBUG:boto:Path: /jdpiak-70/?prefix=d 2022-09-12T19:41:21.271 DEBUG:boto:Data: 2022-09-12T19:41:21.273 DEBUG:boto:Headers: {} 2022-09-12T19:41:21.274 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:21.275 DEBUG:boto:Port: 8000 2022-09-12T19:41:21.276 DEBUG:boto:Params: {} 2022-09-12T19:41:21.278 DEBUG:boto:Token: None 2022-09-12T19:41:21.279 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:21 GMT /jdpiak-70/ 2022-09-12T19:41:21.280 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JJSnSCdzi7fwRnnm1eqKvh37gyY= 2022-09-12T19:41:21.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JJSnSCdzi7fwRnnm1eqKvh37gyY=', 'Content-Length': '0'} 2022-09-12T19:41:21.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fbf621c1c885a8c-00631f8b61-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:21.287 DEBUG:boto:b'jdpiak-70d1000falsed2022-09-12T19:40:40.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:21.289 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:21.290 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:21.291 DEBUG:boto:Method: HEAD 2022-09-12T19:41:21.292 DEBUG:boto:Path: /jdpiak-70/d 2022-09-12T19:41:21.293 DEBUG:boto:Data: 2022-09-12T19:41:21.295 DEBUG:boto:Headers: {} 2022-09-12T19:41:21.297 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:21.298 DEBUG:boto:Port: 8001 2022-09-12T19:41:21.299 DEBUG:boto:Params: {} 2022-09-12T19:41:21.300 DEBUG:boto:Token: None 2022-09-12T19:41:21.301 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:21 GMT /jdpiak-70/d 2022-09-12T19:41:21.302 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BgSgcpZx0AveAb+fCm3CfTR9Z30= 2022-09-12T19:41:21.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BgSgcpZx0AveAb+fCm3CfTR9Z30=', 'Content-Length': '0'} 2022-09-12T19:41:21.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c419f43d777f6bea-00631f8b61-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:21.306 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:21.307 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:21.307 DEBUG:boto:Method: HEAD 2022-09-12T19:41:21.308 DEBUG:boto:Path: /jdpiak-70/d 2022-09-12T19:41:21.308 DEBUG:boto:Data: 2022-09-12T19:41:21.309 DEBUG:boto:Headers: {} 2022-09-12T19:41:21.309 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:21.310 DEBUG:boto:Port: 8000 2022-09-12T19:41:21.310 DEBUG:boto:Params: {} 2022-09-12T19:41:21.311 DEBUG:boto:Token: None 2022-09-12T19:41:21.311 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:21 GMT /jdpiak-70/d 2022-09-12T19:41:21.312 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BgSgcpZx0AveAb+fCm3CfTR9Z30= 2022-09-12T19:41:21.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BgSgcpZx0AveAb+fCm3CfTR9Z30=', 'Content-Length': '0'} 2022-09-12T19:41:21.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b34cfd1e39ce642-00631f8b61-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:21.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:41:21.316 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:21.317 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:21.317 DEBUG:boto:Method: GET 2022-09-12T19:41:21.320 DEBUG:boto:Path: /jdpiak-70/d 2022-09-12T19:41:21.320 DEBUG:boto:Data: 2022-09-12T19:41:21.321 DEBUG:boto:Headers: {} 2022-09-12T19:41:21.322 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:21.322 DEBUG:boto:Port: 8001 2022-09-12T19:41:21.323 DEBUG:boto:Params: {} 2022-09-12T19:41:21.324 DEBUG:boto:Token: None 2022-09-12T19:41:21.325 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:21 GMT /jdpiak-70/d 2022-09-12T19:41:21.326 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wIA8SIKyYyi9rjLCzvhCtW3shZM= 2022-09-12T19:41:21.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wIA8SIKyYyi9rjLCzvhCtW3shZM=', 'Content-Length': '0'} 2022-09-12T19:41:21.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bff256597e3f9472-00631f8b61-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:21.330 DEBUG:boto:path=/jdpiak-70/d 2022-09-12T19:41:21.331 DEBUG:boto:auth_path=/jdpiak-70/d 2022-09-12T19:41:21.332 DEBUG:boto:Method: GET 2022-09-12T19:41:21.332 DEBUG:boto:Path: /jdpiak-70/d 2022-09-12T19:41:21.333 DEBUG:boto:Data: 2022-09-12T19:41:21.333 DEBUG:boto:Headers: {} 2022-09-12T19:41:21.334 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:21.335 DEBUG:boto:Port: 8000 2022-09-12T19:41:21.335 DEBUG:boto:Params: {} 2022-09-12T19:41:21.336 DEBUG:boto:Token: None 2022-09-12T19:41:21.337 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:21 GMT /jdpiak-70/d 2022-09-12T19:41:21.337 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wIA8SIKyYyi9rjLCzvhCtW3shZM= 2022-09-12T19:41:21.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wIA8SIKyYyi9rjLCzvhCtW3shZM=', 'Content-Length': '0'} 2022-09-12T19:41:21.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:40:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009809d13678e7c4bf-00631f8b61-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:21.341 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:21.342 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:21.342 DEBUG:boto:path=/jdpiak-70/?prefix=d 2022-09-12T19:41:21.343 DEBUG:boto:auth_path=/jdpiak-70/?prefix=d 2022-09-12T19:41:21.344 DEBUG:boto:Method: GET 2022-09-12T19:41:21.344 DEBUG:boto:Path: /jdpiak-70/?prefix=d 2022-09-12T19:41:21.345 DEBUG:boto:Data: 2022-09-12T19:41:21.345 DEBUG:boto:Headers: {} 2022-09-12T19:41:21.346 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:21.346 DEBUG:boto:Port: 8001 2022-09-12T19:41:21.347 DEBUG:boto:Params: {} 2022-09-12T19:41:21.348 DEBUG:boto:Token: None 2022-09-12T19:41:21.348 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:21 GMT /jdpiak-70/ 2022-09-12T19:41:21.349 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JJSnSCdzi7fwRnnm1eqKvh37gyY= 2022-09-12T19:41:21.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JJSnSCdzi7fwRnnm1eqKvh37gyY=', 'Content-Length': '0'} 2022-09-12T19:41:21.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059f1f2ac4f1fce38-00631f8b61-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:21.355 DEBUG:boto:b'jdpiak-70d1000falsed2022-09-12T19:40:40.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:21.356 DEBUG:boto:path=/jdpiak-70/ 2022-09-12T19:41:21.357 DEBUG:boto:auth_path=/jdpiak-70/ 2022-09-12T19:41:21.358 DEBUG:boto:path=/jdpiak-70/?prefix=d 2022-09-12T19:41:21.359 DEBUG:boto:auth_path=/jdpiak-70/?prefix=d 2022-09-12T19:41:21.360 DEBUG:boto:Method: GET 2022-09-12T19:41:21.362 DEBUG:boto:Path: /jdpiak-70/?prefix=d 2022-09-12T19:41:21.364 DEBUG:boto:Data: 2022-09-12T19:41:21.365 DEBUG:boto:Headers: {} 2022-09-12T19:41:21.366 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:21.367 DEBUG:boto:Port: 8000 2022-09-12T19:41:21.368 DEBUG:boto:Params: {} 2022-09-12T19:41:21.369 DEBUG:boto:Token: None 2022-09-12T19:41:21.371 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:21 GMT /jdpiak-70/ 2022-09-12T19:41:21.373 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JJSnSCdzi7fwRnnm1eqKvh37gyY= 2022-09-12T19:41:21.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JJSnSCdzi7fwRnnm1eqKvh37gyY=', 'Content-Length': '0'} 2022-09-12T19:41:21.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000025694d07b2df9d1-00631f8b61-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:21.380 DEBUG:boto:b'jdpiak-70d1000falsed2022-09-12T19:40:40.698Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:21.381 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-70 zones={test-zone3, test-zone2} 2022-09-12T19:41:21.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:41:21.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:41:21.383 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:41:21.441 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:41:21.441 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:41:21.513 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'jdpiak-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:41:21.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'jdpiak-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:41:21.517 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket jdpiak-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:41:21.569 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:41:21.569 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:41:21.624 INFO:teuthology.orchestra.run.smithi035.stdout:[] 2022-09-12T19:41:21.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'jdpiak-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:41:21.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'jdpiak-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:41:21.629 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket jdpiak-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:41:21.678 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:41:21.679 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:41:21.727 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:41:21.727 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:21.728 INFO:teuthology.orchestra.run.smithi035.stdout: "op_id": "1#00000000001.73.5", 2022-09-12T19:41:21.728 INFO:teuthology.orchestra.run.smithi035.stdout: "op_tag": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5568.6803971837588890612", 2022-09-12T19:41:21.728 INFO:teuthology.orchestra.run.smithi035.stdout: "op": "write", 2022-09-12T19:41:21.728 INFO:teuthology.orchestra.run.smithi035.stdout: "object": "a", 2022-09-12T19:41:21.729 INFO:teuthology.orchestra.run.smithi035.stdout: "instance": "", 2022-09-12T19:41:21.729 INFO:teuthology.orchestra.run.smithi035.stdout: "state": "complete", 2022-09-12T19:41:21.729 INFO:teuthology.orchestra.run.smithi035.stdout: "index_ver": 1, 2022-09-12T19:41:21.729 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:39:54.609900560Z", 2022-09-12T19:41:21.730 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": { 2022-09-12T19:41:21.731 INFO:teuthology.orchestra.run.smithi035.stdout: "pool": 9, 2022-09-12T19:41:21.732 INFO:teuthology.orchestra.run.smithi035.stdout: "epoch": 7 2022-09-12T19:41:21.732 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:21.734 INFO:teuthology.orchestra.run.smithi035.stdout: "bilog_flags": 0, 2022-09-12T19:41:21.734 INFO:teuthology.orchestra.run.smithi035.stdout: "versioned": false, 2022-09-12T19:41:21.734 INFO:teuthology.orchestra.run.smithi035.stdout: "owner": "", 2022-09-12T19:41:21.735 INFO:teuthology.orchestra.run.smithi035.stdout: "owner_display_name": "", 2022-09-12T19:41:21.735 INFO:teuthology.orchestra.run.smithi035.stdout: "zones_trace": [ 2022-09-12T19:41:21.736 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:21.736 INFO:teuthology.orchestra.run.smithi035.stdout: "entry": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-69:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.20" 2022-09-12T19:41:21.736 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:21.737 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:41:21.737 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:21.738 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:21.738 INFO:teuthology.orchestra.run.smithi035.stdout: "op_id": "5#00000000001.52.5", 2022-09-12T19:41:21.739 INFO:teuthology.orchestra.run.smithi035.stdout: "op_tag": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5568.15589084359251723157", 2022-09-12T19:41:21.739 INFO:teuthology.orchestra.run.smithi035.stdout: "op": "write", 2022-09-12T19:41:21.739 INFO:teuthology.orchestra.run.smithi035.stdout: "object": "d", 2022-09-12T19:41:21.739 INFO:teuthology.orchestra.run.smithi035.stdout: "instance": "", 2022-09-12T19:41:21.740 INFO:teuthology.orchestra.run.smithi035.stdout: "state": "complete", 2022-09-12T19:41:21.740 INFO:teuthology.orchestra.run.smithi035.stdout: "index_ver": 1, 2022-09-12T19:41:21.740 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:39:54.800278109Z", 2022-09-12T19:41:21.741 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": { 2022-09-12T19:41:21.741 INFO:teuthology.orchestra.run.smithi035.stdout: "pool": 9, 2022-09-12T19:41:21.741 INFO:teuthology.orchestra.run.smithi035.stdout: "epoch": 2 2022-09-12T19:41:21.742 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:21.742 INFO:teuthology.orchestra.run.smithi035.stdout: "bilog_flags": 0, 2022-09-12T19:41:21.742 INFO:teuthology.orchestra.run.smithi035.stdout: "versioned": false, 2022-09-12T19:41:21.742 INFO:teuthology.orchestra.run.smithi035.stdout: "owner": "", 2022-09-12T19:41:21.743 INFO:teuthology.orchestra.run.smithi035.stdout: "owner_display_name": "", 2022-09-12T19:41:21.743 INFO:teuthology.orchestra.run.smithi035.stdout: "zones_trace": [ 2022-09-12T19:41:21.743 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:21.744 INFO:teuthology.orchestra.run.smithi035.stdout: "entry": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-69:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.20" 2022-09-12T19:41:21.744 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:21.744 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:41:21.744 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:21.745 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:21.745 INFO:teuthology.orchestra.run.smithi035.stdout: "op_id": "8#00000000001.56.5", 2022-09-12T19:41:21.745 INFO:teuthology.orchestra.run.smithi035.stdout: "op_tag": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5568.6059787585525419235", 2022-09-12T19:41:21.746 INFO:teuthology.orchestra.run.smithi035.stdout: "op": "write", 2022-09-12T19:41:21.746 INFO:teuthology.orchestra.run.smithi035.stdout: "object": "c", 2022-09-12T19:41:21.746 INFO:teuthology.orchestra.run.smithi035.stdout: "instance": "", 2022-09-12T19:41:21.747 INFO:teuthology.orchestra.run.smithi035.stdout: "state": "complete", 2022-09-12T19:41:21.747 INFO:teuthology.orchestra.run.smithi035.stdout: "index_ver": 1, 2022-09-12T19:41:21.747 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:39:54.718500331Z", 2022-09-12T19:41:21.747 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": { 2022-09-12T19:41:21.748 INFO:teuthology.orchestra.run.smithi035.stdout: "pool": 9, 2022-09-12T19:41:21.748 INFO:teuthology.orchestra.run.smithi035.stdout: "epoch": 4 2022-09-12T19:41:21.748 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:21.749 INFO:teuthology.orchestra.run.smithi035.stdout: "bilog_flags": 0, 2022-09-12T19:41:21.749 INFO:teuthology.orchestra.run.smithi035.stdout: "versioned": false, 2022-09-12T19:41:21.749 INFO:teuthology.orchestra.run.smithi035.stdout: "owner": "", 2022-09-12T19:41:21.749 INFO:teuthology.orchestra.run.smithi035.stdout: "owner_display_name": "", 2022-09-12T19:41:21.750 INFO:teuthology.orchestra.run.smithi035.stdout: "zones_trace": [ 2022-09-12T19:41:21.750 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:21.750 INFO:teuthology.orchestra.run.smithi035.stdout: "entry": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-69:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.20" 2022-09-12T19:41:21.750 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:21.751 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:41:21.751 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:21.751 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:21.751 INFO:teuthology.orchestra.run.smithi035.stdout: "op_id": "10#00000000001.42.5", 2022-09-12T19:41:21.752 INFO:teuthology.orchestra.run.smithi035.stdout: "op_tag": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5568.2284099694902051843", 2022-09-12T19:41:21.752 INFO:teuthology.orchestra.run.smithi035.stdout: "op": "write", 2022-09-12T19:41:21.752 INFO:teuthology.orchestra.run.smithi035.stdout: "object": "b", 2022-09-12T19:41:21.753 INFO:teuthology.orchestra.run.smithi035.stdout: "instance": "", 2022-09-12T19:41:21.753 INFO:teuthology.orchestra.run.smithi035.stdout: "state": "complete", 2022-09-12T19:41:21.753 INFO:teuthology.orchestra.run.smithi035.stdout: "index_ver": 1, 2022-09-12T19:41:21.753 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:39:54.681535688Z", 2022-09-12T19:41:21.754 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": { 2022-09-12T19:41:21.754 INFO:teuthology.orchestra.run.smithi035.stdout: "pool": 9, 2022-09-12T19:41:21.754 INFO:teuthology.orchestra.run.smithi035.stdout: "epoch": 4 2022-09-12T19:41:21.754 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:21.755 INFO:teuthology.orchestra.run.smithi035.stdout: "bilog_flags": 0, 2022-09-12T19:41:21.755 INFO:teuthology.orchestra.run.smithi035.stdout: "versioned": false, 2022-09-12T19:41:21.755 INFO:teuthology.orchestra.run.smithi035.stdout: "owner": "", 2022-09-12T19:41:21.755 INFO:teuthology.orchestra.run.smithi035.stdout: "owner_display_name": "", 2022-09-12T19:41:21.756 INFO:teuthology.orchestra.run.smithi035.stdout: "zones_trace": [ 2022-09-12T19:41:21.756 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:21.756 INFO:teuthology.orchestra.run.smithi035.stdout: "entry": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-69:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.20" 2022-09-12T19:41:21.756 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:21.757 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:41:21.757 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:21.757 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:41:21.758 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:41:21.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:41:21.759 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:41:21.819 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:41:21.819 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:41:21.869 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:21.866+0000 7f54f1494700 -1 req 6287798145486596601 0.001999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-4:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.4,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-4:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.4,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:21.872 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:21.867+0000 7f5527500700 -1 req 3921761895803084264 0.001999959s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-52:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.3,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-52:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.3,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:21.872 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:21.867+0000 7fa818726700 -1 req 2234610641449644889 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-4:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.4,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-4:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.4,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:21.873 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:21.868+0000 7fa848f87700 -1 req 14119514846456437193 0.002999933s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-52:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.3,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-52:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.3,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:21.876 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:21.868+0000 7fe7fe5eb700 -1 req 17333100259511335395 0.002999933s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-52:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.3,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-52:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.3,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:21.877 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:21.868+0000 7fe7f9de2700 -1 req 17253890950847836201 0.002999933s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-4:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.4,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-4:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.4,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:21.880 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:21.874+0000 7fa819728700 -1 req 1185691107918271063 0.002999934s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-67:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.18,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-67:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.18,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:21.891 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:21.873+0000 7f54c7c41700 -1 req 16843784934666270584 0.001999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-67:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.18,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-67:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.18,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:21.891 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:21.875+0000 7fe7e45b7700 -1 req 4817527428695863012 0.003999912s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-67:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.18,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-67:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.18,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:21.892 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:21.879+0000 7fa7cd690700 -1 req 13296433006157738642 0.001999955s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-43:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.2,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-43:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.2,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:21.893 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:21.879+0000 7fe7dada4700 -1 req 9404161119157160964 0.001999955s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-43:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.2,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-43:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.2,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:21.893 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:21.878+0000 7f54d1c55700 -1 req 6274063692521070792 0.000999980s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-43:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.2,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-43:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.2,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:21.924 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:21.919+0000 7f54dd46c700 -1 req 17920836732723677240 0.000999980s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:21.926 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:21.920+0000 7fa7e46be700 -1 req 8808178108912249593 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:21.926 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:21.920+0000 7fe7d759d700 -1 req 3935791226350829215 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:21.981 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:21.978+0000 7fe7ba563700 -1 req 14502218853493579401 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-3:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.3,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-3:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.3,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:21.981 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:21.977+0000 7f54c3438700 -1 req 14122031416337441027 0.001999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-3:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.3,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-3:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.3,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:21.982 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:21.978+0000 7fa7f26da700 -1 req 4569217526211422763 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-3:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.3,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-3:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.3,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:21.983 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:21.980+0000 7fa7b8e67700 -1 req 13178787167443973016 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-42:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.1,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-42:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.1,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:21.984 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:21.980+0000 7f55104d2700 -1 req 16187434415363580524 0.001999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-42:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.1,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-42:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.1,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:21.984 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:21.980+0000 7f5528502700 -1 req 9099214416269627052 0.001999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:21.985 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:21.980+0000 7fe8ad749700 -1 req 16411952175360258015 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-42:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.1,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-42:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.1,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:21.985 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:21.981+0000 7fe8ab745700 -1 req 213811568287643603 0.002999934s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:21.986 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:21.981+0000 7fa7bc66e700 -1 req 5487265607558391658 0.002999934s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.080 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:22.077+0000 7fe87feee700 -1 req 14350041783023577623 0.000999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-55:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.6,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-55:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.6,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:22.081 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:22.078+0000 7f54ae40e700 -1 req 15101402988043930121 0.001999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-55:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.6,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-55:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.6,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:22.082 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:22.079+0000 7fa88c80e700 -1 req 3779318349661905472 0.002999933s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-55:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.6,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-55:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.6,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.112 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:22.109+0000 7fe87aee4700 -1 req 3494098291729294338 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-50:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.1,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-50:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.1,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:22.118 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:22.110+0000 7fa89e031700 -1 req 9898197519735905748 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-50:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.1,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-50:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.1,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.119 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:22.109+0000 7f557659e700 -1 req 3313982029909003954 0.000999980s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-50:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.1,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-50:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.1,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:22.129 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:22.124+0000 7fe864eb8700 -1 req 11343607457010921561 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-40:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.40,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-40:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.40,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:22.131 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:22.125+0000 7fa801ef9700 -1 req 16373845280187042530 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-40:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.40,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-40:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.40,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.132 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:22.124+0000 7f5484bbb700 -1 req 13669286473713857541 0.001999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-40:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.40,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-40:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.40,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:22.132 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:22.125+0000 7f5578da3700 -1 req 2857885045238146091 0.000999980s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:22.133 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:22.126+0000 7fe8616b1700 -1 req 10662037368849296044 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:22.134 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:22.126+0000 7fa8567a2700 -1 req 13840756958839861624 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.171 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:22.168+0000 7f5561574700 -1 req 6485870212650629249 0.001999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-44:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.1,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-44:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.1,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:22.172 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:22.168+0000 7fe852693700 -1 req 8556813359115742896 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-44:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.1,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-44:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.1,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:22.173 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:22.168+0000 7fa898826700 -1 req 9472706795275587018 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-44:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.1,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-44:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.1,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.214 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:22.212+0000 7f5542d37700 -1 req 9397234526098125101 0.001999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-32:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.32,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-32:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.32,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:22.215 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:22.212+0000 7fe83e66b700 -1 req 2632268001234515206 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-32:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.32,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-32:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.32,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:22.216 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:22.212+0000 7fa85e7b2700 -1 req 11279329579410509446 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-32:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.32,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-32:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.32,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.225 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:22.222+0000 7fe831e52700 -1 req 15496734027160411521 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-54:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.5,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-54:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.5,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:22.226 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:22.223+0000 7f556a586700 -1 req 8338797822466066679 0.001999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-54:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.5,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-54:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.5,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:22.226 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:22.223+0000 7f552fd11700 -1 req 498141111469856544 0.001999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-7:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.7,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-7:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.7,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:22.227 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:22.223+0000 7fa87bfed700 -1 req 7233009794707691408 0.000999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-54:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.5,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-54:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.5,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.227 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:22.223+0000 7fa848786700 -1 req 14519393750362560945 0.000999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-7:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.7,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-7:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.7,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.228 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:22.223+0000 7fe82de4a700 -1 req 11812951437814446057 0.000999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-7:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.7,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-7:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.7,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:22.259 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:22.256+0000 7f554b548700 -1 req 11130149981483004928 0.000999980s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-2:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.2,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-2:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.2,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:22.260 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:22.257+0000 7fa838766700 -1 req 17058101603650478405 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-2:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.2,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-2:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.2,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.260 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:22.257+0000 7fe821631700 -1 req 14157182695899505481 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-2:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.2,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-2:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.2,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:22.269 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:22.267+0000 7f5573598700 -1 req 1702662693076425377 0.001999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-5:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.5,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-5:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.5,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:22.269 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:22.267+0000 7f551f4f0700 -1 req 14017518777707337564 0.001999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-27:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.27,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-27:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.27,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:22.270 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:22.267+0000 7fa8a403d700 -1 req 6988195629968387545 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-5:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.5,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-5:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.5,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.271 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:22.267+0000 7fe819e22700 -1 req 5291796961636200329 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-5:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.5,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-5:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.5,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:22.271 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:22.268+0000 7fe81b625700 -1 req 18379852735302966108 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-27:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.27,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-27:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.27,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:22.272 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:22.268+0000 7fa86efd3700 -1 req 18115929246089009266 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-27:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.27,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-27:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.27,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.303 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:22.301+0000 7f54f0c93700 -1 req 10983537954424745735 0.001999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-6:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.6,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-6:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.6,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:22.305 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:22.301+0000 7fa82b74c700 -1 req 1189495458658507172 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-6:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.6,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-6:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.6,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.306 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:22.301+0000 7fe809601700 -1 req 2471213077991255516 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-6:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.6,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-6:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.6,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:22.315 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:22.312+0000 7f553cd2b700 -1 req 75004906506358268 0.001999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-34:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.34,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-34:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.34,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:22.316 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:22.313+0000 7fa8747de700 -1 req 14857784573031928412 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-34:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.34,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-34:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.34,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.317 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:22.313+0000 7fe80ce08700 -1 req 1068581379807627781 0.002999934s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-34:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.34,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-34:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.34,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:22.348 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:22.345+0000 7f5559564700 -1 req 7567903342021614948 0.001999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-45:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.2,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-45:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.2,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:22.349 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:22.345+0000 7fe7fb5e5700 -1 req 2652238256493309044 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-45:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.2,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-45:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.2,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:22.349 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:22.345+0000 7fa82e752700 -1 req 15915601138663122699 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-45:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.2,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-45:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.2,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.350 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:22.348+0000 7f54f7ca1700 -1 req 7711186758410008834 0.001999959s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-33:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.33,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-33:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.33,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:22.352 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:22.348+0000 7fe7f85df700 -1 req 2046998795587777221 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-33:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.33,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-33:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.33,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:22.353 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:22.348+0000 7fa817724700 -1 req 6722991987816807484 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-33:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.33,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-33:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.33,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.362 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:22.358+0000 7f553ad27700 -1 req 545545968487482584 0.001999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-28:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.28,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-28:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.28,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:22.363 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:22.358+0000 7fe7f45d7700 -1 req 2305950544225161312 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-28:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.28,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-28:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.28,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:22.364 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:22.358+0000 7fa829f49700 -1 req 15826075508407846761 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-28:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.28,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-28:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.28,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.365 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:22.363+0000 7f54fccab700 -1 req 13886705426147688547 0.000999980s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-66:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.17,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-66:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.17,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:22.367 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:22.363+0000 7fe7e6dbc700 -1 req 13138185956936948136 0.000999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-66:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.17,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-66:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.17,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:22.367 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:22.363+0000 7fa8016f8700 -1 req 10269362809268331428 0.000999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-66:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.17,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-66:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.17,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.391 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:22.388+0000 7f54d3c59700 -1 req 7713121087230659431 0.000999980s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-31:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.31,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-31:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.31,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:22.392 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:22.389+0000 7fa7df6b4700 -1 req 2893076319848647217 0.001999955s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-31:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.31,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-31:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.31,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.393 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:22.389+0000 7fe7e25b3700 -1 req 9171917777820339470 0.001999955s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-31:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.31,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-31:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.31,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:22.394 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:22.391+0000 7f54e447a700 -1 req 9060056517194642519 0.000999980s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-39:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.39,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-39:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.39,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:22.395 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:22.391+0000 7fe7dbda6700 -1 req 9039773671769876389 0.000999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-39:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.39,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-39:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.39,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:22.395 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:22.391+0000 7fa800ef7700 -1 req 5357030762555808111 0.000999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-39:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.39,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-39:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.39,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.410 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:22.406+0000 7f54d645e700 -1 req 13288039696059208379 0.000999980s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-51:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.2,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-51:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.2,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:22.411 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:22.406+0000 7fe7d058f700 -1 req 15299305888007261341 0.000999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-51:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.2,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-51:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.2,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:22.411 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:22.407+0000 7fa7e6ec3700 -1 req 10936600874545827579 0.001999955s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-51:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.2,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-51:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.2,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.436 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:22.433+0000 7f54d9464700 -1 req 7356732439076185849 0.001999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-8:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.8,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-8:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.8,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:22.437 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:22.433+0000 7fa7dd6b0700 -1 req 5911610326991901526 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-8:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.8,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-8:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.8,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.438 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:22.433+0000 7fe7c857f700 -1 req 10317361359966209748 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-8:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.8,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-8:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.8,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:22.479 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:22.477+0000 7f54c543c700 -1 req 13849219781380555044 0.001999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-41:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.41,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-41:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.41,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:22.481 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:22.477+0000 7fe7bbd66700 -1 req 16227928757658045506 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-41:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.41,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-41:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.41,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:22.490 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:22.477+0000 7fa7ba66a700 -1 req 6083315332052375926 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-41:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.41,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-41:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.41,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.557 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:22.555+0000 7f548f3d0700 -1 req 6750639439083038781 0.001999959s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-1:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.1,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-1:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.1,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:22.563 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:22.556+0000 7fe88bf06700 -1 req 16312334905748100458 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-1:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.1,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-1:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.1,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:22.564 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:22.556+0000 7fa7ea6ca700 -1 req 2280436842966663904 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-1:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.1,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-1:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.1,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.603 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:22.601+0000 7fe87deea700 -1 req 2834438648175849493 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-30:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.30,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-30:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.30,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:22.604 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:22.601+0000 7f5492bd7700 -1 req 3121905182709990198 0.001999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-30:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.30,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-30:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.30,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:22.604 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:22.601+0000 7fa875fe1700 -1 req 9142096059655327533 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-30:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.30,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-30:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.30,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.629 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:41:22.626+0000 7fe86aec4700 -1 req 4937325949781339369 0.000999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-53:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.4,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-53:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.4,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:41:22.629 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:41:22.627+0000 7fa8ae051700 -1 req 7935493515749676810 0.001999956s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-53:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.4,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-53:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.4,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:41:22.630 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:41:22.626+0000 7f54a23f6700 -1 req 17262406941204744995 0.000999980s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-53:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.4,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-53:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.4,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:41:22.695 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'jdpiak-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:41:22.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'jdpiak-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:41:22.698 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket jdpiak-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:41:22.751 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:41:22.751 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:41:22.795 INFO:teuthology.orchestra.run.smithi035.stdout:[] 2022-09-12T19:41:22.798 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ok 2022-09-12T19:41:22.800 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=jdpiak-71 2022-09-12T19:41:22.801 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:22.802 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:22.803 DEBUG:boto:Method: PUT 2022-09-12T19:41:22.804 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:41:22.805 DEBUG:boto:Data: 2022-09-12T19:41:22.807 DEBUG:boto:Headers: {} 2022-09-12T19:41:22.808 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:22.809 DEBUG:boto:Port: 8000 2022-09-12T19:41:22.810 DEBUG:boto:Params: {} 2022-09-12T19:41:22.811 DEBUG:boto:Token: None 2022-09-12T19:41:22.812 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:41:22 GMT /jdpiak-71/ 2022-09-12T19:41:22.813 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9OMTB+x3WAMLufXoKBT4MhzQz/A= 2022-09-12T19:41:22.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9OMTB+x3WAMLufXoKBT4MhzQz/A=', 'Content-Length': '0'} 2022-09-12T19:41:22.834 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000044bde5710b51034-00631f8b62-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:22.835 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:22.836 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:22.837 DEBUG:boto:Method: HEAD 2022-09-12T19:41:22.838 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:41:22.839 DEBUG:boto:Data: 2022-09-12T19:41:22.840 DEBUG:boto:Headers: {} 2022-09-12T19:41:22.841 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:22.842 DEBUG:boto:Port: 8000 2022-09-12T19:41:22.843 DEBUG:boto:Params: {} 2022-09-12T19:41:22.844 DEBUG:boto:Token: None 2022-09-12T19:41:22.845 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:22 GMT /jdpiak-71/ 2022-09-12T19:41:22.846 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Jb582410Ad5Pk4IkKcB7t9THPqI= 2022-09-12T19:41:22.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Jb582410Ad5Pk4IkKcB7t9THPqI=', 'Content-Length': '0'} 2022-09-12T19:41:22.855 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060da3b93ab935eb9-00631f8b62-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:22.856 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:22.857 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:22.857 DEBUG:boto:Method: PUT 2022-09-12T19:41:22.858 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:41:22.859 DEBUG:boto:Data: 2022-09-12T19:41:22.859 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:41:22.860 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:22.860 DEBUG:boto:Port: 8000 2022-09-12T19:41:22.861 DEBUG:boto:Params: {} 2022-09-12T19:41:22.861 DEBUG:boto:Token: None 2022-09-12T19:41:22.862 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:41:22 GMT /jdpiak-71/a 2022-09-12T19:41:22.863 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T/ShhKKIKSMRgNjedalIaTw+mqU= 2022-09-12T19:41:22.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:41:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T/ShhKKIKSMRgNjedalIaTw+mqU='} 2022-09-12T19:41:22.871 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003b3f4b9583fa19c8-00631f8b62-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:22.872 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:22.872 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:22.873 DEBUG:boto:Method: HEAD 2022-09-12T19:41:22.874 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:41:22.874 DEBUG:boto:Data: 2022-09-12T19:41:22.875 DEBUG:boto:Headers: {} 2022-09-12T19:41:22.876 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:22.877 DEBUG:boto:Port: 8000 2022-09-12T19:41:22.877 DEBUG:boto:Params: {} 2022-09-12T19:41:22.878 DEBUG:boto:Token: None 2022-09-12T19:41:22.878 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:22 GMT /jdpiak-71/ 2022-09-12T19:41:22.879 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Jb582410Ad5Pk4IkKcB7t9THPqI= 2022-09-12T19:41:22.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Jb582410Ad5Pk4IkKcB7t9THPqI=', 'Content-Length': '0'} 2022-09-12T19:41:22.882 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c512fd7f2f389e58-00631f8b62-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:22.884 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:22.885 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:22.886 DEBUG:boto:Method: PUT 2022-09-12T19:41:22.888 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:41:22.889 DEBUG:boto:Data: 2022-09-12T19:41:22.890 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:41:22.891 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:22.892 DEBUG:boto:Port: 8000 2022-09-12T19:41:22.893 DEBUG:boto:Params: {} 2022-09-12T19:41:22.894 DEBUG:boto:Token: None 2022-09-12T19:41:22.895 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:41:22 GMT /jdpiak-71/b 2022-09-12T19:41:22.896 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CxOWJ9hDX2bmrVue/YVmU2efejw= 2022-09-12T19:41:22.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:41:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CxOWJ9hDX2bmrVue/YVmU2efejw='} 2022-09-12T19:41:22.947 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005fb0279a4d30eae8-00631f8b62-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:22.948 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:22.949 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:22.950 DEBUG:boto:Method: HEAD 2022-09-12T19:41:22.950 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:41:22.951 DEBUG:boto:Data: 2022-09-12T19:41:22.952 DEBUG:boto:Headers: {} 2022-09-12T19:41:22.953 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:22.954 DEBUG:boto:Port: 8000 2022-09-12T19:41:22.954 DEBUG:boto:Params: {} 2022-09-12T19:41:22.955 DEBUG:boto:Token: None 2022-09-12T19:41:22.956 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:22 GMT /jdpiak-71/ 2022-09-12T19:41:22.957 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Jb582410Ad5Pk4IkKcB7t9THPqI= 2022-09-12T19:41:22.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Jb582410Ad5Pk4IkKcB7t9THPqI=', 'Content-Length': '0'} 2022-09-12T19:41:22.962 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0668918d5cddc4c-00631f8b62-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:22.963 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:22.964 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:22.965 DEBUG:boto:Method: PUT 2022-09-12T19:41:22.966 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:41:22.967 DEBUG:boto:Data: 2022-09-12T19:41:22.968 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:41:22.968 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:22.969 DEBUG:boto:Port: 8000 2022-09-12T19:41:22.970 DEBUG:boto:Params: {} 2022-09-12T19:41:22.971 DEBUG:boto:Token: None 2022-09-12T19:41:22.972 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:41:22 GMT /jdpiak-71/c 2022-09-12T19:41:22.973 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fxKyqXYRRHAwo4aC7IOOcbLVSRw= 2022-09-12T19:41:22.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:41:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fxKyqXYRRHAwo4aC7IOOcbLVSRw='} 2022-09-12T19:41:22.981 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000504a65ebc7aa932c-00631f8b62-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:22.983 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:22.984 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:22.986 DEBUG:boto:Method: HEAD 2022-09-12T19:41:22.988 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:41:22.989 DEBUG:boto:Data: 2022-09-12T19:41:22.990 DEBUG:boto:Headers: {} 2022-09-12T19:41:22.991 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:22.992 DEBUG:boto:Port: 8000 2022-09-12T19:41:22.993 DEBUG:boto:Params: {} 2022-09-12T19:41:22.994 DEBUG:boto:Token: None 2022-09-12T19:41:22.996 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:22 GMT /jdpiak-71/ 2022-09-12T19:41:22.997 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Jb582410Ad5Pk4IkKcB7t9THPqI= 2022-09-12T19:41:22.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Jb582410Ad5Pk4IkKcB7t9THPqI=', 'Content-Length': '0'} 2022-09-12T19:41:23.002 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005c139defa77edc6b-00631f8b62-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:23.003 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:23.004 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:23.005 DEBUG:boto:Method: PUT 2022-09-12T19:41:23.007 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:41:23.008 DEBUG:boto:Data: 2022-09-12T19:41:23.009 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:41:23.010 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:23.011 DEBUG:boto:Port: 8000 2022-09-12T19:41:23.012 DEBUG:boto:Params: {} 2022-09-12T19:41:23.013 DEBUG:boto:Token: None 2022-09-12T19:41:23.014 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:41:23 GMT /jdpiak-71/d 2022-09-12T19:41:23.015 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vgWmWpkqo4PLjlSGy7IlVemXXc0= 2022-09-12T19:41:23.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:41:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vgWmWpkqo4PLjlSGy7IlVemXXc0='} 2022-09-12T19:41:23.025 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000beab642ed1d1d35a-00631f8b63-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:23.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:41:23.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:41:23.028 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:41:23.079 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:41:23.080 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:41:23.122 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=38ca1f57-1358-496c-a99b-df49ee365a4f 2022-09-12T19:41:23.143 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:41:23.143 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:23.143 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:41:23.144 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:41:23.144 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:23.145 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:23.145 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:41:23.145 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:41:23.145 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:23.146 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:23.146 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663011682.829175_632.1", 2022-09-12T19:41:23.146 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:41:22.829175Z" 2022-09-12T19:41:23.146 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:23.147 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:23.147 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:41:23.147 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:41:23.148 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:23.148 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:41:23.149 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1663011682.829175_632.1', 3: ''} 2022-09-12T19:41:23.150 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T19:41:23.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:41:23.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:41:23.152 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:41:23.214 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:41:23.214 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:41:23.244 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:41:23.240+0000 7f3bc8dbe600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:41:23.244 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:41:23.241+0000 7f3bc8dbe600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:41:23.257 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:41:23.266 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:41:23.266 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:41:23.267 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:41:23.267 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:41:23.267 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:41:23.267 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:41:23.268 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:41:23.268 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:41:23.268 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:41:23.268 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:41:23.269 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:41:23.269 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:41:23.269 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:41:23.269 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:41:23.274 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:41:23.275 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:41:23.275 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:41:23.276 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:41:23.276 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:41:23.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:41:23.280 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:41:23.280 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:41:23.281 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:41:23.282 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:41:23.282 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:41:23.283 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:41:23.283 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:41:23.284 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:41:23.284 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:41:23.286 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:41:23.286 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:41:23.286 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:41:23.287 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:41:23.287 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:41:23.287 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:41:23.288 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:41:23.288 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663011682.822045_629.1", 2022-09-12T19:41:23.288 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:41:23.288 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:41:23.289 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:41:23.289 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:41:22.822045Z", 2022-09-12T19:41:23.289 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:41:23.289 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:41:23.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:41:23.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:41:23.290 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:41:23.290 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:41:23.291 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:41:23.291 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:41:23.299 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:41:23.300 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:41:23.300 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:41:23.300 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:41:23.306 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:41:23.308 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:41:23.308 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:41:23.309 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:41:23.310 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:41:23.310 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:41:23.311 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:41:23.311 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:41:23.311 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:41:23.311 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:41:23Z" 2022-09-12T19:41:23.312 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:41:23.313 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011682.822045_629.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:41:22.822045Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:41:23Z" } 2022-09-12T19:41:23.409 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011682.822045_629.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:41:22.822045Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:41:23.410 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011682.829175_632.1', 3: ''} 2022-09-12T19:41:23.411 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011682.822045_629.1', 3: ''} 2022-09-12T19:41:23.411 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663011682.829175_632.1 target=1_1663011682.822045_629.1 2022-09-12T19:41:28.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:41:28.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:41:28.415 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:41:28.470 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:41:28.471 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:41:28.510 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:41:28.507+0000 7fab10837600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:41:28.515 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:41:28.508+0000 7fab10837600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:41:28.529 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:41:28.530 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:41:28.531 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:41:28.531 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:41:28.532 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:41:28.532 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:41:28.533 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:41:28.533 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:41:28.534 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:41:28.534 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:41:28.535 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:41:28.535 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:41:28.536 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:41:28.536 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:41:28.537 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:41:28.538 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:41:28.538 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:41:28.539 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:41:28.539 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:41:28.540 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:41:28.540 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:41:28.541 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:41:28.541 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:41:28.542 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:41:28.543 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:41:28.543 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:41:28.544 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:41:28.544 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:41:28.545 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:41:28.545 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:41:28.545 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:41:28.546 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:41:28.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:41:28.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:41:28.547 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:41:28.547 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:41:28.547 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:41:28.547 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663011682.829175_632.1", 2022-09-12T19:41:28.548 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:41:28.548 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:41:28.548 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:41:28.550 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:41:22.829175Z", 2022-09-12T19:41:28.550 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:41:28.550 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:41:28.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:41:28.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:41:28.551 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:41:28.551 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:41:28.552 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:41:28.552 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:41:28.552 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:41:28.552 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:41:28.553 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:41:28.553 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:41:28.553 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:41:28.553 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:41:28.554 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:41:28.554 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:41:28.554 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:41:28.555 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:41:28.555 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:41:28.556 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:41:28.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:41:28.557 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:41:28Z" 2022-09-12T19:41:28.557 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:41:28.559 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011682.829175_632.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:41:22.829175Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:41:28Z" } 2022-09-12T19:41:28.559 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011682.829175_632.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:41:22.829175Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:41:28.560 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011682.829175_632.1', 3: ''} 2022-09-12T19:41:28.561 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011682.829175_632.1', 3: ''} 2022-09-12T19:41:28.562 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T19:41:28.562 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T19:41:28.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:41:28.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:41:28.564 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:41:28.624 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:41:28.625 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:41:28.659 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:41:28.656+0000 7f6755861600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:41:28.661 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:41:28.657+0000 7f6755861600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:41:28.673 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:41:28.674 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:41:28.675 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:41:28.675 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:41:28.675 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:41:28.676 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:41:28.677 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:41:28.677 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:28.677 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:41:28.678 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:28.678 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:41:28.679 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:41:28.679 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:41:28.679 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:41:28.680 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:41:28.680 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:41:28.680 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:41:28.681 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:41:28.681 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:41:28.681 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:28.681 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:28.682 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:28.682 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:41:28.682 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:41:28.682 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:41:28.683 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:41:28.683 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:41:28.683 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:41:28.683 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:41:28.684 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:41:28.684 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:41:28.685 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:28.685 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:28.685 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:28.686 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:41:28.686 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:41:28.687 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:41:28.687 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663011682.829175_632.1", 2022-09-12T19:41:28.687 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:41:28.688 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:41:28.688 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:41:28.688 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:41:22.829175Z", 2022-09-12T19:41:28.689 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:41:28.689 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:28.689 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:28.690 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:28.690 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:41:28.691 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:41:28.691 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:41:28.691 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:41:28.691 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:41:28.692 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:41:28.692 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:41:28.692 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:41:28.692 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:41:28.693 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:28.693 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:28.693 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:41:28.693 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:28.694 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:41:28.694 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:41:28.694 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:41:28.695 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:28.695 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:41:28Z" 2022-09-12T19:41:28.695 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:41:28.697 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011682.829175_632.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:41:22.829175Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:41:28Z" } 2022-09-12T19:41:28.698 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011682.829175_632.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:41:22.829175Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:41:28.698 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011682.829175_632.1', 3: ''} 2022-09-12T19:41:28.699 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011682.829175_632.1', 3: ''} 2022-09-12T19:41:28.700 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T19:41:28.700 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T19:41:28.701 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:41:28.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:41:28.703 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:41:28.755 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:41:28.755 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:41:28.795 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:41:28.788+0000 7f154e664600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:41:28.795 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:41:28.789+0000 7f154e664600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:41:28.804 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:41:28.805 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:41:28.805 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:41:28.805 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:41:28.805 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:41:28.806 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:41:28.806 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:41:28.806 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:41:28.807 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:41:28.807 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:41:28.807 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:41:28.807 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:41:28.808 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:41:28.808 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:41:28.808 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:41:28.808 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:41:28.809 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:41:28.809 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:41:28.809 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:41:28.809 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:41:28.812 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:41:28.812 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:41:28.813 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:41:28.813 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:41:28.813 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:41:28.813 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:41:28.814 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:41:28.814 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:41:28.814 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:41:28.815 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:41:28.815 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:41:28.815 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:41:28.815 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:41:28.815 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:41:28.816 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:41:28.816 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:41:28.816 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:41:28.817 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663011682.829175_632.1", 2022-09-12T19:41:28.817 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:41:28.817 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:41:28.817 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:41:28.818 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:41:22.829175Z", 2022-09-12T19:41:28.818 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:41:28.818 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:41:28.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:41:28.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:41:28.819 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:41:28.820 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:41:28.820 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:41:28.820 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:41:28.821 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:41:28.822 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:41:28.822 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:41:28.823 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:41:28.823 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:41:28.824 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:41:28.824 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:41:28.825 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:41:28.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:41:28.826 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:41:28.826 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:41:28.827 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:41:28.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:41:28.838 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:41:28Z" 2022-09-12T19:41:28.839 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:41:28.839 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011682.829175_632.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:41:22.829175Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:41:28Z" } 2022-09-12T19:41:28.840 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011682.829175_632.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:41:22.829175Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:41:28.840 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011682.829175_632.1', 3: ''} 2022-09-12T19:41:28.841 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011682.829175_632.1', 3: ''} 2022-09-12T19:41:28.842 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T19:41:28.842 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-71 2022-09-12T19:41:28.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:41:28.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:41:28.844 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-71 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:41:28.909 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:41:28.910 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:41:28.937 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:41:28.935+0000 7ff4507e9600 1 waiting to reach incremental sync.. 2022-09-12T19:41:33.938 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:41:33.936+0000 7ff4507e9600 1 waiting to reach incremental sync.. 2022-09-12T19:41:38.940 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:41:38.937+0000 7ff4507e9600 1 bucket sync caught up with source: 2022-09-12T19:41:38.940 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.52.5, , , , 00000000001.89.5, , , 00000000001.60.5, , 00000000001.41.5] 2022-09-12T19:41:38.941 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.52.5, , , , 00000000001.89.5, , , 00000000001.60.5, , 00000000001.41.5] 2022-09-12T19:41:38.941 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:41:38.937+0000 7ff4507e9600 0 bucket checkpoint complete 2022-09-12T19:41:38.959 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-71 2022-09-12T19:41:38.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:41:38.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:41:38.963 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-71 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:41:39.029 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:41:39.030 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:41:39.083 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:41:39.080+0000 7fe6ce3d9600 1 bucket sync caught up with source: 2022-09-12T19:41:39.083 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.52.5, , , , 00000000001.89.5, , , 00000000001.60.5, , 00000000001.41.5] 2022-09-12T19:41:39.084 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000001.52.5, , , , 00000000001.89.5, , , 00000000001.60.5, , 00000000001.41.5] 2022-09-12T19:41:39.084 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:41:39.080+0000 7fe6ce3d9600 0 bucket checkpoint complete 2022-09-12T19:41:39.102 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-71 2022-09-12T19:41:39.103 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:41:39.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:41:39.104 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-71 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:41:39.169 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:41:39.169 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:41:39.207 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:41:39.205+0000 7fa860305600 1 bucket sync caught up with source: 2022-09-12T19:41:39.208 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.52.5, , , , 00000000001.89.5, , , 00000000001.60.5, , 00000000001.41.5] 2022-09-12T19:41:39.210 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.52.5, , , , 00000000001.89.5, , , 00000000001.60.5, , 00000000001.41.5] 2022-09-12T19:41:39.210 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:41:39.205+0000 7fa860305600 0 bucket checkpoint complete 2022-09-12T19:41:39.221 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-71 2022-09-12T19:41:39.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:41:39.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:41:39.225 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-71 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:41:39.287 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:41:39.288 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:41:39.323 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:41:39.321+0000 7f1a0ccc4600 1 waiting to reach incremental sync.. 2022-09-12T19:41:44.324 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:41:44.322+0000 7f1a0ccc4600 1 waiting to reach incremental sync.. 2022-09-12T19:41:49.324 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:41:49.322+0000 7f1a0ccc4600 1 waiting to reach incremental sync.. 2022-09-12T19:41:54.327 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:41:54.324+0000 7f1a0ccc4600 1 bucket sync caught up with source: 2022-09-12T19:41:54.327 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.886.5, , , , 00000000001.1376.5, , , 00000000001.1137.5, , 00000000001.510.5] 2022-09-12T19:41:54.327 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000001.886.5, , , , 00000000001.1376.5, , , 00000000001.1137.5, , 00000000001.510.5] 2022-09-12T19:41:54.328 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:41:54.324+0000 7f1a0ccc4600 0 bucket checkpoint complete 2022-09-12T19:41:54.347 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-71 2022-09-12T19:41:54.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:41:54.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:41:54.351 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-71 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:41:54.413 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:41:54.414 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:41:54.458 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:41:54.454+0000 7fb3223d8600 1 bucket sync caught up with source: 2022-09-12T19:41:54.459 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.886.5, , , , 00000000001.1376.5, , , 00000000001.1137.5, , 00000000001.510.5] 2022-09-12T19:41:54.459 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000001.886.5, , , , 00000000001.1376.5, , , 00000000001.1137.5, , 00000000001.510.5] 2022-09-12T19:41:54.459 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:41:54.454+0000 7fb3223d8600 0 bucket checkpoint complete 2022-09-12T19:41:54.471 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-71 2022-09-12T19:41:54.471 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-71 2022-09-12T19:41:54.472 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:41:54.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:41:54.473 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-71 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:41:54.531 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:41:54.532 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:41:54.569 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:41:54.566+0000 7f47d1917600 1 bucket sync caught up with source: 2022-09-12T19:41:54.571 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.496.5, , , , 00000000001.744.5, , , 00000000001.447.5, , 00000000001.242.5] 2022-09-12T19:41:54.571 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000001.496.5, , , , 00000000001.744.5, , , 00000000001.447.5, , 00000000001.242.5] 2022-09-12T19:41:54.572 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:41:54.566+0000 7f47d1917600 0 bucket checkpoint complete 2022-09-12T19:41:54.583 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-71 2022-09-12T19:41:54.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:41:54.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:41:54.585 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-71 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:41:54.650 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:41:54.651 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:41:54.710 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:41:54.705+0000 7ff6c5e8a600 1 bucket sync caught up with source: 2022-09-12T19:41:54.711 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.496.5, , , , 00000000001.744.5, , , 00000000001.447.5, , 00000000001.242.5] 2022-09-12T19:41:54.712 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.496.5, , , , 00000000001.744.5, , , 00000000001.447.5, , 00000000001.242.5] 2022-09-12T19:41:54.712 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:41:54.705+0000 7ff6c5e8a600 0 bucket checkpoint complete 2022-09-12T19:41:54.728 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-71 2022-09-12T19:41:54.729 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-71 zones={test-zone2, test-zone1} 2022-09-12T19:41:54.732 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:54.733 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:54.734 DEBUG:boto:Method: HEAD 2022-09-12T19:41:54.737 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:41:54.738 DEBUG:boto:Data: 2022-09-12T19:41:54.739 DEBUG:boto:Headers: {} 2022-09-12T19:41:54.740 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:54.741 DEBUG:boto:Port: 8000 2022-09-12T19:41:54.742 DEBUG:boto:Params: {} 2022-09-12T19:41:54.743 DEBUG:boto:Token: None 2022-09-12T19:41:54.744 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:54 GMT /jdpiak-71/ 2022-09-12T19:41:54.746 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AOTPJ6hdUYWzdlvpbmgfjxlKIGM= 2022-09-12T19:41:54.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AOTPJ6hdUYWzdlvpbmgfjxlKIGM=', 'Content-Length': '0'} 2022-09-12T19:41:54.750 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b2d6a30f8781247-00631f8b82-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:41:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:54.752 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:54.752 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:54.753 DEBUG:boto:Method: HEAD 2022-09-12T19:41:54.755 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:41:54.756 DEBUG:boto:Data: 2022-09-12T19:41:54.757 DEBUG:boto:Headers: {} 2022-09-12T19:41:54.758 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:54.759 DEBUG:boto:Port: 8000 2022-09-12T19:41:54.761 DEBUG:boto:Params: {} 2022-09-12T19:41:54.761 DEBUG:boto:Token: None 2022-09-12T19:41:54.762 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:54 GMT /jdpiak-71/ 2022-09-12T19:41:54.763 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AOTPJ6hdUYWzdlvpbmgfjxlKIGM= 2022-09-12T19:41:54.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AOTPJ6hdUYWzdlvpbmgfjxlKIGM=', 'Content-Length': '0'} 2022-09-12T19:41:54.766 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b5a2d23ad2ae686-00631f8b82-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:41:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:54.767 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:41:54.767 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:54.768 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:54.768 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:54.769 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:54.769 DEBUG:boto:Method: GET 2022-09-12T19:41:54.770 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:54.773 DEBUG:boto:Data: 2022-09-12T19:41:54.773 DEBUG:boto:Headers: {} 2022-09-12T19:41:54.774 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:54.774 DEBUG:boto:Port: 8000 2022-09-12T19:41:54.775 DEBUG:boto:Params: {} 2022-09-12T19:41:54.776 DEBUG:boto:Token: None 2022-09-12T19:41:54.776 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:54 GMT /jdpiak-71/?versions 2022-09-12T19:41:54.777 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XmZpYfy/3hsSQJkwo4tzOldWlpw= 2022-09-12T19:41:54.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XmZpYfy/3hsSQJkwo4tzOldWlpw=', 'Content-Length': '0'} 2022-09-12T19:41:54.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043b6082c38ce2cc4-00631f8b82-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:54.781 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:54.791 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:41:54.792 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:41:54.793 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:41:54.794 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:41:54.795 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:41:54.796 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:54.797 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:54.798 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:54.799 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:54.801 DEBUG:boto:Method: GET 2022-09-12T19:41:54.802 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:54.803 DEBUG:boto:Data: 2022-09-12T19:41:54.804 DEBUG:boto:Headers: {} 2022-09-12T19:41:54.805 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:54.807 DEBUG:boto:Port: 8000 2022-09-12T19:41:54.808 DEBUG:boto:Params: {} 2022-09-12T19:41:54.809 DEBUG:boto:Token: None 2022-09-12T19:41:54.810 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:54 GMT /jdpiak-71/?versions 2022-09-12T19:41:54.812 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XmZpYfy/3hsSQJkwo4tzOldWlpw= 2022-09-12T19:41:54.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XmZpYfy/3hsSQJkwo4tzOldWlpw=', 'Content-Length': '0'} 2022-09-12T19:41:54.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d92bd54bb4d4fb38-00631f8b82-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:54.819 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:54.820 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:41:54.821 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:41:54.822 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:41:54.822 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:41:54.823 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:54.824 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:54.825 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:54.826 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:54.827 DEBUG:boto:Method: GET 2022-09-12T19:41:54.828 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:54.829 DEBUG:boto:Data: 2022-09-12T19:41:54.830 DEBUG:boto:Headers: {} 2022-09-12T19:41:54.831 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:54.831 DEBUG:boto:Port: 8000 2022-09-12T19:41:54.832 DEBUG:boto:Params: {} 2022-09-12T19:41:54.834 DEBUG:boto:Token: None 2022-09-12T19:41:54.835 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:54 GMT /jdpiak-71/?versions 2022-09-12T19:41:54.836 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XmZpYfy/3hsSQJkwo4tzOldWlpw= 2022-09-12T19:41:54.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XmZpYfy/3hsSQJkwo4tzOldWlpw=', 'Content-Length': '0'} 2022-09-12T19:41:54.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098f0745dc1102597-00631f8b82-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:54.843 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:54.844 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:54.845 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:54.846 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:54.847 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:54.848 DEBUG:boto:Method: GET 2022-09-12T19:41:54.849 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:54.850 DEBUG:boto:Data: 2022-09-12T19:41:54.851 DEBUG:boto:Headers: {} 2022-09-12T19:41:54.852 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:54.853 DEBUG:boto:Port: 8000 2022-09-12T19:41:54.854 DEBUG:boto:Params: {} 2022-09-12T19:41:54.855 DEBUG:boto:Token: None 2022-09-12T19:41:54.856 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:54 GMT /jdpiak-71/?versions 2022-09-12T19:41:54.857 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XmZpYfy/3hsSQJkwo4tzOldWlpw= 2022-09-12T19:41:54.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XmZpYfy/3hsSQJkwo4tzOldWlpw=', 'Content-Length': '0'} 2022-09-12T19:41:54.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d709b5f820f178a-00631f8b82-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:54.864 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:54.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:41:54.867 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:54.869 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:54.870 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:54.871 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:54.872 DEBUG:boto:Method: GET 2022-09-12T19:41:54.873 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:41:54.875 DEBUG:boto:Data: 2022-09-12T19:41:54.876 DEBUG:boto:Headers: {} 2022-09-12T19:41:54.877 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:54.879 DEBUG:boto:Port: 8000 2022-09-12T19:41:54.880 DEBUG:boto:Params: {} 2022-09-12T19:41:54.881 DEBUG:boto:Token: None 2022-09-12T19:41:54.883 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:54 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:41:54.885 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NOfrjnKsyrlKBIivUQzI1Yw3CnM= 2022-09-12T19:41:54.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NOfrjnKsyrlKBIivUQzI1Yw3CnM=', 'Content-Length': '0'} 2022-09-12T19:41:54.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb0a3378c94c9905-00631f8b82-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:54.896 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:54.897 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:54.898 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:54.899 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:54.900 DEBUG:boto:Method: GET 2022-09-12T19:41:54.902 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:41:54.903 DEBUG:boto:Data: 2022-09-12T19:41:54.905 DEBUG:boto:Headers: {} 2022-09-12T19:41:54.906 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:54.907 DEBUG:boto:Port: 8000 2022-09-12T19:41:54.908 DEBUG:boto:Params: {} 2022-09-12T19:41:54.909 DEBUG:boto:Token: None 2022-09-12T19:41:54.911 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:54 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:41:54.912 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NOfrjnKsyrlKBIivUQzI1Yw3CnM= 2022-09-12T19:41:54.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NOfrjnKsyrlKBIivUQzI1Yw3CnM=', 'Content-Length': '0'} 2022-09-12T19:41:54.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a22b1c82f0107db-00631f8b82-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:54.918 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:54.919 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:54.920 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:54.921 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:54.922 DEBUG:boto:Method: HEAD 2022-09-12T19:41:54.923 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:41:54.924 DEBUG:boto:Data: 2022-09-12T19:41:54.925 DEBUG:boto:Headers: {} 2022-09-12T19:41:54.926 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:54.927 DEBUG:boto:Port: 8000 2022-09-12T19:41:54.928 DEBUG:boto:Params: {} 2022-09-12T19:41:54.929 DEBUG:boto:Token: None 2022-09-12T19:41:54.930 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:54 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:41:54.931 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sF/TCn4lDZ3ecXMHre/oYuKyi/c= 2022-09-12T19:41:54.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sF/TCn4lDZ3ecXMHre/oYuKyi/c=', 'Content-Length': '0'} 2022-09-12T19:41:54.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007521f3703b64b069-00631f8b82-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:54.937 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:54.938 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:54.939 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:54.940 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:54.941 DEBUG:boto:Method: HEAD 2022-09-12T19:41:54.942 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:41:54.943 DEBUG:boto:Data: 2022-09-12T19:41:54.944 DEBUG:boto:Headers: {} 2022-09-12T19:41:54.945 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:54.946 DEBUG:boto:Port: 8000 2022-09-12T19:41:54.947 DEBUG:boto:Params: {} 2022-09-12T19:41:54.948 DEBUG:boto:Token: None 2022-09-12T19:41:54.951 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:54 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:41:54.952 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sF/TCn4lDZ3ecXMHre/oYuKyi/c= 2022-09-12T19:41:54.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sF/TCn4lDZ3ecXMHre/oYuKyi/c=', 'Content-Length': '0'} 2022-09-12T19:41:54.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e28e09e42c22426d-00631f8b82-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:54.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:41:54.959 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:54.960 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:54.961 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:54.962 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:54.964 DEBUG:boto:Method: GET 2022-09-12T19:41:54.965 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:41:54.965 DEBUG:boto:Data: 2022-09-12T19:41:54.967 DEBUG:boto:Headers: {} 2022-09-12T19:41:54.968 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:54.969 DEBUG:boto:Port: 8000 2022-09-12T19:41:54.970 DEBUG:boto:Params: {} 2022-09-12T19:41:54.971 DEBUG:boto:Token: None 2022-09-12T19:41:54.972 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:54 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:41:54.973 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NOfrjnKsyrlKBIivUQzI1Yw3CnM= 2022-09-12T19:41:54.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NOfrjnKsyrlKBIivUQzI1Yw3CnM=', 'Content-Length': '0'} 2022-09-12T19:41:54.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ad5e59d16561055-00631f8b82-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:54.978 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:54.979 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:54.980 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:54.981 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:54.983 DEBUG:boto:Method: GET 2022-09-12T19:41:54.984 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:41:54.984 DEBUG:boto:Data: 2022-09-12T19:41:54.985 DEBUG:boto:Headers: {} 2022-09-12T19:41:54.986 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:54.987 DEBUG:boto:Port: 8000 2022-09-12T19:41:54.988 DEBUG:boto:Params: {} 2022-09-12T19:41:54.988 DEBUG:boto:Token: None 2022-09-12T19:41:54.989 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:54 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:41:54.990 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NOfrjnKsyrlKBIivUQzI1Yw3CnM= 2022-09-12T19:41:54.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NOfrjnKsyrlKBIivUQzI1Yw3CnM=', 'Content-Length': '0'} 2022-09-12T19:41:54.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000427c93bff79f404b-00631f8b82-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:54.995 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:54.995 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:54.996 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:41:54.996 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:41:54.997 DEBUG:boto:Method: GET 2022-09-12T19:41:54.997 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:41:54.998 DEBUG:boto:Data: 2022-09-12T19:41:54.999 DEBUG:boto:Headers: {} 2022-09-12T19:41:54.999 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:55.000 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.000 DEBUG:boto:Params: {} 2022-09-12T19:41:55.001 DEBUG:boto:Token: None 2022-09-12T19:41:55.002 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/ 2022-09-12T19:41:55.002 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw= 2022-09-12T19:41:55.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw=', 'Content-Length': '0'} 2022-09-12T19:41:55.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000157d255040455276-00631f8b83-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.007 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:55.008 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:55.009 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:55.010 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:41:55.010 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:41:55.011 DEBUG:boto:Method: GET 2022-09-12T19:41:55.011 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:41:55.012 DEBUG:boto:Data: 2022-09-12T19:41:55.012 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.013 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:55.013 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.014 DEBUG:boto:Params: {} 2022-09-12T19:41:55.015 DEBUG:boto:Token: None 2022-09-12T19:41:55.015 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/ 2022-09-12T19:41:55.016 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw= 2022-09-12T19:41:55.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw=', 'Content-Length': '0'} 2022-09-12T19:41:55.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f6ae1b266654fa0-00631f8b83-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.020 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:55.021 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:55.021 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:55.022 DEBUG:boto:Method: HEAD 2022-09-12T19:41:55.022 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:41:55.023 DEBUG:boto:Data: 2022-09-12T19:41:55.023 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.024 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:55.024 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.025 DEBUG:boto:Params: {} 2022-09-12T19:41:55.025 DEBUG:boto:Token: None 2022-09-12T19:41:55.026 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/a 2022-09-12T19:41:55.027 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WP0qZj0p6uOz8gen0hlzBHs8qbU= 2022-09-12T19:41:55.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WP0qZj0p6uOz8gen0hlzBHs8qbU=', 'Content-Length': '0'} 2022-09-12T19:41:55.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056e3fc87c821f709-00631f8b83-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.031 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:55.032 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:55.033 DEBUG:boto:Method: HEAD 2022-09-12T19:41:55.034 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:41:55.034 DEBUG:boto:Data: 2022-09-12T19:41:55.035 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.036 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:55.037 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.038 DEBUG:boto:Params: {} 2022-09-12T19:41:55.039 DEBUG:boto:Token: None 2022-09-12T19:41:55.040 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/a 2022-09-12T19:41:55.041 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WP0qZj0p6uOz8gen0hlzBHs8qbU= 2022-09-12T19:41:55.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WP0qZj0p6uOz8gen0hlzBHs8qbU=', 'Content-Length': '0'} 2022-09-12T19:41:55.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000106d0e4e8dd8a0b0-00631f8b83-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:41:55.047 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:55.048 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:55.049 DEBUG:boto:Method: GET 2022-09-12T19:41:55.049 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:41:55.050 DEBUG:boto:Data: 2022-09-12T19:41:55.051 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.052 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:55.052 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.053 DEBUG:boto:Params: {} 2022-09-12T19:41:55.054 DEBUG:boto:Token: None 2022-09-12T19:41:55.055 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/a 2022-09-12T19:41:55.056 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:I0QkRg4ZvyPVnaotaFGAQx2X2F8= 2022-09-12T19:41:55.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:I0QkRg4ZvyPVnaotaFGAQx2X2F8=', 'Content-Length': '0'} 2022-09-12T19:41:55.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af2d66ed5ece0a07-00631f8b83-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.102 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:55.103 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:55.103 DEBUG:boto:Method: GET 2022-09-12T19:41:55.104 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:41:55.104 DEBUG:boto:Data: 2022-09-12T19:41:55.105 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.106 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:55.106 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.107 DEBUG:boto:Params: {} 2022-09-12T19:41:55.107 DEBUG:boto:Token: None 2022-09-12T19:41:55.108 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/a 2022-09-12T19:41:55.109 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:I0QkRg4ZvyPVnaotaFGAQx2X2F8= 2022-09-12T19:41:55.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:I0QkRg4ZvyPVnaotaFGAQx2X2F8=', 'Content-Length': '0'} 2022-09-12T19:41:55.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf42828f720c9110-00631f8b83-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.119 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:55.120 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:55.120 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:41:55.121 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:41:55.121 DEBUG:boto:Method: GET 2022-09-12T19:41:55.122 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:41:55.122 DEBUG:boto:Data: 2022-09-12T19:41:55.123 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.124 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:55.124 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.125 DEBUG:boto:Params: {} 2022-09-12T19:41:55.125 DEBUG:boto:Token: None 2022-09-12T19:41:55.126 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/ 2022-09-12T19:41:55.136 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw= 2022-09-12T19:41:55.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw=', 'Content-Length': '0'} 2022-09-12T19:41:55.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081198ac8e369076b-00631f8b83-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.142 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:55.144 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:55.145 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:55.146 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:41:55.147 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:41:55.148 DEBUG:boto:Method: GET 2022-09-12T19:41:55.149 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:41:55.150 DEBUG:boto:Data: 2022-09-12T19:41:55.151 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.152 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:55.153 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.155 DEBUG:boto:Params: {} 2022-09-12T19:41:55.156 DEBUG:boto:Token: None 2022-09-12T19:41:55.158 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/ 2022-09-12T19:41:55.160 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw= 2022-09-12T19:41:55.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw=', 'Content-Length': '0'} 2022-09-12T19:41:55.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009db29e497ca3cb0b-00631f8b83-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.167 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:55.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:41:55.171 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:55.172 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:55.173 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:55.174 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:55.175 DEBUG:boto:Method: GET 2022-09-12T19:41:55.176 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:41:55.177 DEBUG:boto:Data: 2022-09-12T19:41:55.178 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.179 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:55.180 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.181 DEBUG:boto:Params: {} 2022-09-12T19:41:55.183 DEBUG:boto:Token: None 2022-09-12T19:41:55.184 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:41:55.185 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c3ZXnhDxmrCVYW5m/kyqG4I05BY= 2022-09-12T19:41:55.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c3ZXnhDxmrCVYW5m/kyqG4I05BY=', 'Content-Length': '0'} 2022-09-12T19:41:55.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d34e911c4915de0-00631f8b83-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.196 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:55.197 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:55.198 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:55.200 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:55.201 DEBUG:boto:Method: GET 2022-09-12T19:41:55.203 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:41:55.204 DEBUG:boto:Data: 2022-09-12T19:41:55.205 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.207 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:55.207 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.208 DEBUG:boto:Params: {} 2022-09-12T19:41:55.209 DEBUG:boto:Token: None 2022-09-12T19:41:55.210 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:41:55.211 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c3ZXnhDxmrCVYW5m/kyqG4I05BY= 2022-09-12T19:41:55.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c3ZXnhDxmrCVYW5m/kyqG4I05BY=', 'Content-Length': '0'} 2022-09-12T19:41:55.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006036208d024f3ab0-00631f8b83-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.215 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:55.216 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:55.216 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:55.217 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:55.217 DEBUG:boto:Method: HEAD 2022-09-12T19:41:55.218 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:41:55.218 DEBUG:boto:Data: 2022-09-12T19:41:55.219 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.219 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:55.220 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.220 DEBUG:boto:Params: {} 2022-09-12T19:41:55.221 DEBUG:boto:Token: None 2022-09-12T19:41:55.221 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:41:55.222 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EJ5GUdRI5YoFKDR44A43Jx/3lIg= 2022-09-12T19:41:55.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EJ5GUdRI5YoFKDR44A43Jx/3lIg=', 'Content-Length': '0'} 2022-09-12T19:41:55.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4f7640e87b9b237-00631f8b83-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.225 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:55.226 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:55.226 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:55.227 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:55.227 DEBUG:boto:Method: HEAD 2022-09-12T19:41:55.228 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:41:55.228 DEBUG:boto:Data: 2022-09-12T19:41:55.229 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.229 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:55.230 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.230 DEBUG:boto:Params: {} 2022-09-12T19:41:55.231 DEBUG:boto:Token: None 2022-09-12T19:41:55.232 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:41:55.234 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EJ5GUdRI5YoFKDR44A43Jx/3lIg= 2022-09-12T19:41:55.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EJ5GUdRI5YoFKDR44A43Jx/3lIg=', 'Content-Length': '0'} 2022-09-12T19:41:55.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6464d0729c0e96b-00631f8b83-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:41:55.241 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:55.242 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:55.244 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:55.245 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:55.245 DEBUG:boto:Method: GET 2022-09-12T19:41:55.246 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:41:55.247 DEBUG:boto:Data: 2022-09-12T19:41:55.247 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.248 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:55.249 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.249 DEBUG:boto:Params: {} 2022-09-12T19:41:55.250 DEBUG:boto:Token: None 2022-09-12T19:41:55.251 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:41:55.251 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c3ZXnhDxmrCVYW5m/kyqG4I05BY= 2022-09-12T19:41:55.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c3ZXnhDxmrCVYW5m/kyqG4I05BY=', 'Content-Length': '0'} 2022-09-12T19:41:55.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb52bab19b762cda-00631f8b83-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.256 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:55.257 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:55.258 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:55.258 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:55.259 DEBUG:boto:Method: GET 2022-09-12T19:41:55.259 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:41:55.260 DEBUG:boto:Data: 2022-09-12T19:41:55.261 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.261 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:55.262 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.263 DEBUG:boto:Params: {} 2022-09-12T19:41:55.264 DEBUG:boto:Token: None 2022-09-12T19:41:55.264 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:41:55.265 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c3ZXnhDxmrCVYW5m/kyqG4I05BY= 2022-09-12T19:41:55.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c3ZXnhDxmrCVYW5m/kyqG4I05BY=', 'Content-Length': '0'} 2022-09-12T19:41:55.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c55e8f6bf0bd1428-00631f8b83-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.270 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:55.270 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:55.271 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:41:55.272 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:41:55.273 DEBUG:boto:Method: GET 2022-09-12T19:41:55.274 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:41:55.275 DEBUG:boto:Data: 2022-09-12T19:41:55.276 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.304 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:55.305 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.305 DEBUG:boto:Params: {} 2022-09-12T19:41:55.306 DEBUG:boto:Token: None 2022-09-12T19:41:55.307 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/ 2022-09-12T19:41:55.308 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw= 2022-09-12T19:41:55.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw=', 'Content-Length': '0'} 2022-09-12T19:41:55.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000045e00bcdd8cf21d-00631f8b83-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.311 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:55.313 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:55.313 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:55.314 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:41:55.315 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:41:55.316 DEBUG:boto:Method: GET 2022-09-12T19:41:55.317 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:41:55.317 DEBUG:boto:Data: 2022-09-12T19:41:55.318 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.319 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:55.320 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.320 DEBUG:boto:Params: {} 2022-09-12T19:41:55.321 DEBUG:boto:Token: None 2022-09-12T19:41:55.321 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/ 2022-09-12T19:41:55.322 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw= 2022-09-12T19:41:55.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw=', 'Content-Length': '0'} 2022-09-12T19:41:55.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf53b3f4434b8754-00631f8b83-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.326 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:55.327 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:55.328 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:55.329 DEBUG:boto:Method: HEAD 2022-09-12T19:41:55.329 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:41:55.330 DEBUG:boto:Data: 2022-09-12T19:41:55.330 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.331 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:55.331 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.332 DEBUG:boto:Params: {} 2022-09-12T19:41:55.332 DEBUG:boto:Token: None 2022-09-12T19:41:55.333 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/b 2022-09-12T19:41:55.334 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PWmSbkhPwC81cgBwW4KyyxrUWV0= 2022-09-12T19:41:55.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PWmSbkhPwC81cgBwW4KyyxrUWV0=', 'Content-Length': '0'} 2022-09-12T19:41:55.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090692b398f874aca-00631f8b83-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.337 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:55.338 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:55.339 DEBUG:boto:Method: HEAD 2022-09-12T19:41:55.339 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:41:55.340 DEBUG:boto:Data: 2022-09-12T19:41:55.340 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.341 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:55.342 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.342 DEBUG:boto:Params: {} 2022-09-12T19:41:55.343 DEBUG:boto:Token: None 2022-09-12T19:41:55.343 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/b 2022-09-12T19:41:55.344 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PWmSbkhPwC81cgBwW4KyyxrUWV0= 2022-09-12T19:41:55.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PWmSbkhPwC81cgBwW4KyyxrUWV0=', 'Content-Length': '0'} 2022-09-12T19:41:55.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca88f2ea1fe755e0-00631f8b83-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:41:55.353 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:55.354 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:55.355 DEBUG:boto:Method: GET 2022-09-12T19:41:55.356 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:41:55.357 DEBUG:boto:Data: 2022-09-12T19:41:55.358 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.359 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:55.360 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.362 DEBUG:boto:Params: {} 2022-09-12T19:41:55.363 DEBUG:boto:Token: None 2022-09-12T19:41:55.364 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/b 2022-09-12T19:41:55.365 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9EZ+MykpnJ/PEG0NxTeyF7YwVBo= 2022-09-12T19:41:55.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9EZ+MykpnJ/PEG0NxTeyF7YwVBo=', 'Content-Length': '0'} 2022-09-12T19:41:55.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc14015e9a7ed092-00631f8b83-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.410 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:55.412 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:55.413 DEBUG:boto:Method: GET 2022-09-12T19:41:55.414 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:41:55.415 DEBUG:boto:Data: 2022-09-12T19:41:55.416 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.417 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:55.419 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.420 DEBUG:boto:Params: {} 2022-09-12T19:41:55.421 DEBUG:boto:Token: None 2022-09-12T19:41:55.422 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/b 2022-09-12T19:41:55.423 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9EZ+MykpnJ/PEG0NxTeyF7YwVBo= 2022-09-12T19:41:55.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9EZ+MykpnJ/PEG0NxTeyF7YwVBo=', 'Content-Length': '0'} 2022-09-12T19:41:55.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043a2e0b26611556c-00631f8b83-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.435 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:55.436 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:55.437 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:41:55.438 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:41:55.439 DEBUG:boto:Method: GET 2022-09-12T19:41:55.440 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:41:55.441 DEBUG:boto:Data: 2022-09-12T19:41:55.443 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.444 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:55.445 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.446 DEBUG:boto:Params: {} 2022-09-12T19:41:55.447 DEBUG:boto:Token: None 2022-09-12T19:41:55.448 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/ 2022-09-12T19:41:55.449 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw= 2022-09-12T19:41:55.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw=', 'Content-Length': '0'} 2022-09-12T19:41:55.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034e1e2dce063646c-00631f8b83-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.459 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:55.460 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:55.461 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:55.462 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:41:55.463 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:41:55.464 DEBUG:boto:Method: GET 2022-09-12T19:41:55.465 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:41:55.466 DEBUG:boto:Data: 2022-09-12T19:41:55.467 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.468 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:55.468 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.469 DEBUG:boto:Params: {} 2022-09-12T19:41:55.471 DEBUG:boto:Token: None 2022-09-12T19:41:55.472 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/ 2022-09-12T19:41:55.473 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw= 2022-09-12T19:41:55.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw=', 'Content-Length': '0'} 2022-09-12T19:41:55.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ada0ec7d89b0831-00631f8b83-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.480 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:55.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:41:55.483 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:55.484 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:55.485 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:55.485 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:55.486 DEBUG:boto:Method: GET 2022-09-12T19:41:55.487 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:41:55.488 DEBUG:boto:Data: 2022-09-12T19:41:55.489 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.490 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:55.491 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.491 DEBUG:boto:Params: {} 2022-09-12T19:41:55.492 DEBUG:boto:Token: None 2022-09-12T19:41:55.492 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:41:55.493 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KxXPi+YdFr0Ssqp3c1beLxuGs/Y= 2022-09-12T19:41:55.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KxXPi+YdFr0Ssqp3c1beLxuGs/Y=', 'Content-Length': '0'} 2022-09-12T19:41:55.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0fed200ef21a7f8-00631f8b83-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.497 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:55.498 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:55.498 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:55.499 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:55.500 DEBUG:boto:Method: GET 2022-09-12T19:41:55.501 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:41:55.502 DEBUG:boto:Data: 2022-09-12T19:41:55.503 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.504 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:55.505 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.506 DEBUG:boto:Params: {} 2022-09-12T19:41:55.507 DEBUG:boto:Token: None 2022-09-12T19:41:55.508 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:41:55.509 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KxXPi+YdFr0Ssqp3c1beLxuGs/Y= 2022-09-12T19:41:55.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KxXPi+YdFr0Ssqp3c1beLxuGs/Y=', 'Content-Length': '0'} 2022-09-12T19:41:55.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019db02c00bb767ab-00631f8b83-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.558 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:55.559 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:55.560 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:55.561 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:55.562 DEBUG:boto:Method: HEAD 2022-09-12T19:41:55.563 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:41:55.564 DEBUG:boto:Data: 2022-09-12T19:41:55.565 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.566 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:55.566 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.567 DEBUG:boto:Params: {} 2022-09-12T19:41:55.568 DEBUG:boto:Token: None 2022-09-12T19:41:55.569 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:41:55.570 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Qau0l59Ib9DDF/2laBsOnNBm7a4= 2022-09-12T19:41:55.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Qau0l59Ib9DDF/2laBsOnNBm7a4=', 'Content-Length': '0'} 2022-09-12T19:41:55.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c394d1c3b06a0cf-00631f8b83-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.577 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:55.578 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:55.579 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:55.580 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:55.581 DEBUG:boto:Method: HEAD 2022-09-12T19:41:55.582 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:41:55.583 DEBUG:boto:Data: 2022-09-12T19:41:55.584 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.585 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:55.585 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.589 DEBUG:boto:Params: {} 2022-09-12T19:41:55.590 DEBUG:boto:Token: None 2022-09-12T19:41:55.591 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:41:55.591 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Qau0l59Ib9DDF/2laBsOnNBm7a4= 2022-09-12T19:41:55.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Qau0l59Ib9DDF/2laBsOnNBm7a4=', 'Content-Length': '0'} 2022-09-12T19:41:55.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000213a6bbe800645f2-00631f8b83-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:41:55.596 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:55.596 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:55.597 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:55.597 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:55.598 DEBUG:boto:Method: GET 2022-09-12T19:41:55.599 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:41:55.602 DEBUG:boto:Data: 2022-09-12T19:41:55.603 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.604 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:55.605 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.606 DEBUG:boto:Params: {} 2022-09-12T19:41:55.607 DEBUG:boto:Token: None 2022-09-12T19:41:55.608 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:41:55.609 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KxXPi+YdFr0Ssqp3c1beLxuGs/Y= 2022-09-12T19:41:55.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KxXPi+YdFr0Ssqp3c1beLxuGs/Y=', 'Content-Length': '0'} 2022-09-12T19:41:55.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e81d677ea479664a-00631f8b83-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.615 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:55.616 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:55.617 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:55.618 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:55.619 DEBUG:boto:Method: GET 2022-09-12T19:41:55.619 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:41:55.620 DEBUG:boto:Data: 2022-09-12T19:41:55.620 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.621 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:55.621 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.622 DEBUG:boto:Params: {} 2022-09-12T19:41:55.622 DEBUG:boto:Token: None 2022-09-12T19:41:55.623 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:41:55.624 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KxXPi+YdFr0Ssqp3c1beLxuGs/Y= 2022-09-12T19:41:55.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KxXPi+YdFr0Ssqp3c1beLxuGs/Y=', 'Content-Length': '0'} 2022-09-12T19:41:55.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050aaaa30a66d5cdb-00631f8b83-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.631 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:55.632 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:55.633 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:41:55.634 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:41:55.635 DEBUG:boto:Method: GET 2022-09-12T19:41:55.636 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:41:55.637 DEBUG:boto:Data: 2022-09-12T19:41:55.638 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.638 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:55.639 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.640 DEBUG:boto:Params: {} 2022-09-12T19:41:55.641 DEBUG:boto:Token: None 2022-09-12T19:41:55.642 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/ 2022-09-12T19:41:55.643 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw= 2022-09-12T19:41:55.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw=', 'Content-Length': '0'} 2022-09-12T19:41:55.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001feda2e1d50b1dba-00631f8b83-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.647 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:55.649 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:55.650 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:55.651 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:41:55.652 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:41:55.653 DEBUG:boto:Method: GET 2022-09-12T19:41:55.654 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:41:55.655 DEBUG:boto:Data: 2022-09-12T19:41:55.656 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.656 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:55.658 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.659 DEBUG:boto:Params: {} 2022-09-12T19:41:55.660 DEBUG:boto:Token: None 2022-09-12T19:41:55.661 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/ 2022-09-12T19:41:55.662 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw= 2022-09-12T19:41:55.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw=', 'Content-Length': '0'} 2022-09-12T19:41:55.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be57a0065c3a65cf-00631f8b83-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.710 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:55.712 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:55.713 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:55.714 DEBUG:boto:Method: HEAD 2022-09-12T19:41:55.715 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:41:55.716 DEBUG:boto:Data: 2022-09-12T19:41:55.717 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.718 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:55.718 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.719 DEBUG:boto:Params: {} 2022-09-12T19:41:55.720 DEBUG:boto:Token: None 2022-09-12T19:41:55.721 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/c 2022-09-12T19:41:55.723 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AGkIY0RRe5jPSuMVT6uu8LSH7RY= 2022-09-12T19:41:55.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AGkIY0RRe5jPSuMVT6uu8LSH7RY=', 'Content-Length': '0'} 2022-09-12T19:41:55.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fffe2be3101763a-00631f8b83-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.728 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:55.729 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:55.729 DEBUG:boto:Method: HEAD 2022-09-12T19:41:55.730 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:41:55.731 DEBUG:boto:Data: 2022-09-12T19:41:55.731 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.732 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:55.732 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.733 DEBUG:boto:Params: {} 2022-09-12T19:41:55.735 DEBUG:boto:Token: None 2022-09-12T19:41:55.736 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/c 2022-09-12T19:41:55.737 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AGkIY0RRe5jPSuMVT6uu8LSH7RY= 2022-09-12T19:41:55.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AGkIY0RRe5jPSuMVT6uu8LSH7RY=', 'Content-Length': '0'} 2022-09-12T19:41:55.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a000fb86c1040bd-00631f8b83-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:41:55.753 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:55.754 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:55.755 DEBUG:boto:Method: GET 2022-09-12T19:41:55.756 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:41:55.757 DEBUG:boto:Data: 2022-09-12T19:41:55.758 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.759 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:55.759 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.760 DEBUG:boto:Params: {} 2022-09-12T19:41:55.761 DEBUG:boto:Token: None 2022-09-12T19:41:55.763 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/c 2022-09-12T19:41:55.764 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fOQqNAw5YhWohn0jSd2cOeyaJs8= 2022-09-12T19:41:55.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fOQqNAw5YhWohn0jSd2cOeyaJs8=', 'Content-Length': '0'} 2022-09-12T19:41:55.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c18f0be2a671ae00-00631f8b83-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.817 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:55.818 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:55.820 DEBUG:boto:Method: GET 2022-09-12T19:41:55.821 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:41:55.822 DEBUG:boto:Data: 2022-09-12T19:41:55.823 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.824 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:55.825 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.826 DEBUG:boto:Params: {} 2022-09-12T19:41:55.827 DEBUG:boto:Token: None 2022-09-12T19:41:55.828 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/c 2022-09-12T19:41:55.829 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fOQqNAw5YhWohn0jSd2cOeyaJs8= 2022-09-12T19:41:55.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fOQqNAw5YhWohn0jSd2cOeyaJs8=', 'Content-Length': '0'} 2022-09-12T19:41:55.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac8deade00383c5b-00631f8b83-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.833 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:55.834 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:55.834 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:41:55.835 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:41:55.836 DEBUG:boto:Method: GET 2022-09-12T19:41:55.836 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:41:55.837 DEBUG:boto:Data: 2022-09-12T19:41:55.837 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.839 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:55.839 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.840 DEBUG:boto:Params: {} 2022-09-12T19:41:55.841 DEBUG:boto:Token: None 2022-09-12T19:41:55.841 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/ 2022-09-12T19:41:55.842 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw= 2022-09-12T19:41:55.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw=', 'Content-Length': '0'} 2022-09-12T19:41:55.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b9d127d5bdffbf0-00631f8b83-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.847 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:55.848 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:55.848 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:55.849 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:41:55.849 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:41:55.850 DEBUG:boto:Method: GET 2022-09-12T19:41:55.851 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:41:55.851 DEBUG:boto:Data: 2022-09-12T19:41:55.852 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.853 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:55.853 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.854 DEBUG:boto:Params: {} 2022-09-12T19:41:55.854 DEBUG:boto:Token: None 2022-09-12T19:41:55.855 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/ 2022-09-12T19:41:55.856 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw= 2022-09-12T19:41:55.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw=', 'Content-Length': '0'} 2022-09-12T19:41:55.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009aeb8b1a50472f9c-00631f8b83-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.861 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:55.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:41:55.862 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:55.863 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:55.863 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:55.864 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:55.864 DEBUG:boto:Method: GET 2022-09-12T19:41:55.865 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:41:55.866 DEBUG:boto:Data: 2022-09-12T19:41:55.866 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.867 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:55.867 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.868 DEBUG:boto:Params: {} 2022-09-12T19:41:55.868 DEBUG:boto:Token: None 2022-09-12T19:41:55.869 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:41:55.870 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2j1WQj+vKb5wWtGFDlgTXIFPTnw= 2022-09-12T19:41:55.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2j1WQj+vKb5wWtGFDlgTXIFPTnw=', 'Content-Length': '0'} 2022-09-12T19:41:55.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076fcbea6baa05391-00631f8b83-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.873 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:55.874 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:55.875 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:55.875 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:55.876 DEBUG:boto:Method: GET 2022-09-12T19:41:55.876 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:41:55.877 DEBUG:boto:Data: 2022-09-12T19:41:55.877 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.878 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:55.878 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.879 DEBUG:boto:Params: {} 2022-09-12T19:41:55.879 DEBUG:boto:Token: None 2022-09-12T19:41:55.880 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:41:55.881 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2j1WQj+vKb5wWtGFDlgTXIFPTnw= 2022-09-12T19:41:55.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2j1WQj+vKb5wWtGFDlgTXIFPTnw=', 'Content-Length': '0'} 2022-09-12T19:41:55.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edf3e7b81817e966-00631f8b83-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.885 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:55.886 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:55.886 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:55.887 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:55.887 DEBUG:boto:Method: HEAD 2022-09-12T19:41:55.888 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:41:55.888 DEBUG:boto:Data: 2022-09-12T19:41:55.889 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.889 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:55.890 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.890 DEBUG:boto:Params: {} 2022-09-12T19:41:55.891 DEBUG:boto:Token: None 2022-09-12T19:41:55.892 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:41:55.892 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gLw3qeJcmMqXTDblp2Wz32c1Fq0= 2022-09-12T19:41:55.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gLw3qeJcmMqXTDblp2Wz32c1Fq0=', 'Content-Length': '0'} 2022-09-12T19:41:55.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a9dbe36a9f0e6ea-00631f8b83-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.896 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:55.896 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:55.897 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:55.897 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:55.898 DEBUG:boto:Method: HEAD 2022-09-12T19:41:55.898 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:41:55.899 DEBUG:boto:Data: 2022-09-12T19:41:55.900 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.900 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:55.901 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.901 DEBUG:boto:Params: {} 2022-09-12T19:41:55.902 DEBUG:boto:Token: None 2022-09-12T19:41:55.903 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:41:55.903 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gLw3qeJcmMqXTDblp2Wz32c1Fq0= 2022-09-12T19:41:55.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gLw3qeJcmMqXTDblp2Wz32c1Fq0=', 'Content-Length': '0'} 2022-09-12T19:41:55.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000329f41bfa0aeb713-00631f8b83-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:41:55.909 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:55.909 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:55.910 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:55.911 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:55.912 DEBUG:boto:Method: GET 2022-09-12T19:41:55.912 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:41:55.913 DEBUG:boto:Data: 2022-09-12T19:41:55.914 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.915 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:55.918 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.919 DEBUG:boto:Params: {} 2022-09-12T19:41:55.920 DEBUG:boto:Token: None 2022-09-12T19:41:55.922 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:41:55.923 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2j1WQj+vKb5wWtGFDlgTXIFPTnw= 2022-09-12T19:41:55.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2j1WQj+vKb5wWtGFDlgTXIFPTnw=', 'Content-Length': '0'} 2022-09-12T19:41:55.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bec133f781737ce5-00631f8b83-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.971 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:55.972 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:55.972 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:55.973 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:55.974 DEBUG:boto:Method: GET 2022-09-12T19:41:55.974 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:41:55.975 DEBUG:boto:Data: 2022-09-12T19:41:55.975 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.976 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:55.976 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.977 DEBUG:boto:Params: {} 2022-09-12T19:41:55.978 DEBUG:boto:Token: None 2022-09-12T19:41:55.978 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:41:55.979 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2j1WQj+vKb5wWtGFDlgTXIFPTnw= 2022-09-12T19:41:55.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2j1WQj+vKb5wWtGFDlgTXIFPTnw=', 'Content-Length': '0'} 2022-09-12T19:41:55.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a31c28250eb13e31-00631f8b83-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.983 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:55.984 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:55.985 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:41:55.985 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:41:55.986 DEBUG:boto:Method: GET 2022-09-12T19:41:55.986 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:41:55.987 DEBUG:boto:Data: 2022-09-12T19:41:55.988 DEBUG:boto:Headers: {} 2022-09-12T19:41:55.988 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:55.989 DEBUG:boto:Port: 8000 2022-09-12T19:41:55.989 DEBUG:boto:Params: {} 2022-09-12T19:41:55.990 DEBUG:boto:Token: None 2022-09-12T19:41:55.991 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:55 GMT /jdpiak-71/ 2022-09-12T19:41:55.992 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw= 2022-09-12T19:41:55.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zusq46TYMTLVVMay+BBQbw5Spnw=', 'Content-Length': '0'} 2022-09-12T19:41:55.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e172b24cdc83146-00631f8b83-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:55.997 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:55.998 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:55.999 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:56.000 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:41:56.000 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:41:56.001 DEBUG:boto:Method: GET 2022-09-12T19:41:56.001 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:41:56.002 DEBUG:boto:Data: 2022-09-12T19:41:56.004 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.005 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:56.006 DEBUG:boto:Port: 8000 2022-09-12T19:41:56.007 DEBUG:boto:Params: {} 2022-09-12T19:41:56.008 DEBUG:boto:Token: None 2022-09-12T19:41:56.009 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/ 2022-09-12T19:41:56.011 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V1Dp4GcM63vxx/k1zSckYHOHDxI= 2022-09-12T19:41:56.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V1Dp4GcM63vxx/k1zSckYHOHDxI=', 'Content-Length': '0'} 2022-09-12T19:41:56.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba7d631791f677c5-00631f8b84-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.019 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:56.020 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:56.021 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:56.022 DEBUG:boto:Method: HEAD 2022-09-12T19:41:56.023 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:41:56.024 DEBUG:boto:Data: 2022-09-12T19:41:56.026 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.029 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:56.030 DEBUG:boto:Port: 8000 2022-09-12T19:41:56.031 DEBUG:boto:Params: {} 2022-09-12T19:41:56.032 DEBUG:boto:Token: None 2022-09-12T19:41:56.033 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/d 2022-09-12T19:41:56.034 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/vYOAgr/AHcZbtpl6v6N/5c5Kqo= 2022-09-12T19:41:56.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/vYOAgr/AHcZbtpl6v6N/5c5Kqo=', 'Content-Length': '0'} 2022-09-12T19:41:56.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6862d8dec1ca40d-00631f8b84-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.040 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:56.041 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:56.042 DEBUG:boto:Method: HEAD 2022-09-12T19:41:56.043 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:41:56.043 DEBUG:boto:Data: 2022-09-12T19:41:56.044 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.044 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:56.045 DEBUG:boto:Port: 8000 2022-09-12T19:41:56.045 DEBUG:boto:Params: {} 2022-09-12T19:41:56.046 DEBUG:boto:Token: None 2022-09-12T19:41:56.047 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/d 2022-09-12T19:41:56.049 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/vYOAgr/AHcZbtpl6v6N/5c5Kqo= 2022-09-12T19:41:56.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/vYOAgr/AHcZbtpl6v6N/5c5Kqo=', 'Content-Length': '0'} 2022-09-12T19:41:56.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b813d877dd84c4b1-00631f8b84-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:41:56.059 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:56.060 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:56.060 DEBUG:boto:Method: GET 2022-09-12T19:41:56.061 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:41:56.062 DEBUG:boto:Data: 2022-09-12T19:41:56.062 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.063 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:56.064 DEBUG:boto:Port: 8000 2022-09-12T19:41:56.064 DEBUG:boto:Params: {} 2022-09-12T19:41:56.065 DEBUG:boto:Token: None 2022-09-12T19:41:56.065 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/d 2022-09-12T19:41:56.066 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3AFc7Ce51pBC4URnY+agAWJpGl4= 2022-09-12T19:41:56.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3AFc7Ce51pBC4URnY+agAWJpGl4=', 'Content-Length': '0'} 2022-09-12T19:41:56.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d8c86c65398e59d-00631f8b84-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.081 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:56.082 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:56.083 DEBUG:boto:Method: GET 2022-09-12T19:41:56.083 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:41:56.084 DEBUG:boto:Data: 2022-09-12T19:41:56.085 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.085 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:56.086 DEBUG:boto:Port: 8000 2022-09-12T19:41:56.087 DEBUG:boto:Params: {} 2022-09-12T19:41:56.087 DEBUG:boto:Token: None 2022-09-12T19:41:56.088 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/d 2022-09-12T19:41:56.089 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3AFc7Ce51pBC4URnY+agAWJpGl4= 2022-09-12T19:41:56.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3AFc7Ce51pBC4URnY+agAWJpGl4=', 'Content-Length': '0'} 2022-09-12T19:41:56.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c85b896fc1cd26bb-00631f8b84-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.093 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:56.095 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:56.096 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:41:56.097 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:41:56.098 DEBUG:boto:Method: GET 2022-09-12T19:41:56.099 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:41:56.100 DEBUG:boto:Data: 2022-09-12T19:41:56.101 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.102 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:56.104 DEBUG:boto:Port: 8000 2022-09-12T19:41:56.105 DEBUG:boto:Params: {} 2022-09-12T19:41:56.106 DEBUG:boto:Token: None 2022-09-12T19:41:56.107 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/ 2022-09-12T19:41:56.109 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V1Dp4GcM63vxx/k1zSckYHOHDxI= 2022-09-12T19:41:56.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V1Dp4GcM63vxx/k1zSckYHOHDxI=', 'Content-Length': '0'} 2022-09-12T19:41:56.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000592a8b5d0b9e41f0-00631f8b84-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.124 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:56.126 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:56.127 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:56.132 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:41:56.133 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:41:56.134 DEBUG:boto:Method: GET 2022-09-12T19:41:56.135 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:41:56.136 DEBUG:boto:Data: 2022-09-12T19:41:56.139 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.140 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:56.141 DEBUG:boto:Port: 8000 2022-09-12T19:41:56.142 DEBUG:boto:Params: {} 2022-09-12T19:41:56.144 DEBUG:boto:Token: None 2022-09-12T19:41:56.145 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/ 2022-09-12T19:41:56.146 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V1Dp4GcM63vxx/k1zSckYHOHDxI= 2022-09-12T19:41:56.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V1Dp4GcM63vxx/k1zSckYHOHDxI=', 'Content-Length': '0'} 2022-09-12T19:41:56.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002755c5b6c2f835e5-00631f8b84-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.154 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:56.156 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-71 zones={test-zone2, test-zone1} 2022-09-12T19:41:56.158 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-71 zones={test-zone3, test-zone1} 2022-09-12T19:41:56.159 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:56.160 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:56.166 DEBUG:boto:Method: HEAD 2022-09-12T19:41:56.167 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:41:56.168 DEBUG:boto:Data: 2022-09-12T19:41:56.169 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.172 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:56.173 DEBUG:boto:Port: 8001 2022-09-12T19:41:56.175 DEBUG:boto:Params: {} 2022-09-12T19:41:56.176 DEBUG:boto:Token: None 2022-09-12T19:41:56.177 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/ 2022-09-12T19:41:56.178 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rOVYywr/ltuPFuZvoO7EJcX1h08= 2022-09-12T19:41:56.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rOVYywr/ltuPFuZvoO7EJcX1h08=', 'Content-Length': '0'} 2022-09-12T19:41:56.190 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ed219278b9c35d5-00631f8b84-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.191 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:56.192 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:56.193 DEBUG:boto:Method: HEAD 2022-09-12T19:41:56.193 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:41:56.194 DEBUG:boto:Data: 2022-09-12T19:41:56.195 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.195 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:56.196 DEBUG:boto:Port: 8000 2022-09-12T19:41:56.197 DEBUG:boto:Params: {} 2022-09-12T19:41:56.197 DEBUG:boto:Token: None 2022-09-12T19:41:56.198 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/ 2022-09-12T19:41:56.199 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rOVYywr/ltuPFuZvoO7EJcX1h08= 2022-09-12T19:41:56.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rOVYywr/ltuPFuZvoO7EJcX1h08=', 'Content-Length': '0'} 2022-09-12T19:41:56.204 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e16f0ce5fdb725db-00631f8b84-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.205 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:41:56.206 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:56.207 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:56.208 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:56.209 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:56.209 DEBUG:boto:Method: GET 2022-09-12T19:41:56.210 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:56.211 DEBUG:boto:Data: 2022-09-12T19:41:56.212 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.213 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:56.214 DEBUG:boto:Port: 8001 2022-09-12T19:41:56.215 DEBUG:boto:Params: {} 2022-09-12T19:41:56.216 DEBUG:boto:Token: None 2022-09-12T19:41:56.217 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/?versions 2022-09-12T19:41:56.218 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CbtKAqJEXgZEnb13BKb817NDSIE= 2022-09-12T19:41:56.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CbtKAqJEXgZEnb13BKb817NDSIE=', 'Content-Length': '0'} 2022-09-12T19:41:56.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bad64a39a9f0f63-00631f8b84-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.225 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:56.226 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:41:56.228 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:41:56.228 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:41:56.229 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:41:56.229 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:41:56.230 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:56.231 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:56.232 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:56.233 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:56.235 DEBUG:boto:Method: GET 2022-09-12T19:41:56.236 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:56.237 DEBUG:boto:Data: 2022-09-12T19:41:56.238 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.239 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:56.240 DEBUG:boto:Port: 8000 2022-09-12T19:41:56.241 DEBUG:boto:Params: {} 2022-09-12T19:41:56.242 DEBUG:boto:Token: None 2022-09-12T19:41:56.243 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/?versions 2022-09-12T19:41:56.244 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CbtKAqJEXgZEnb13BKb817NDSIE= 2022-09-12T19:41:56.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CbtKAqJEXgZEnb13BKb817NDSIE=', 'Content-Length': '0'} 2022-09-12T19:41:56.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a50126d549521db-00631f8b84-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.251 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:56.253 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:41:56.253 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:41:56.254 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:41:56.254 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:41:56.255 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:56.258 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:56.259 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:56.259 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:56.260 DEBUG:boto:Method: GET 2022-09-12T19:41:56.261 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:56.262 DEBUG:boto:Data: 2022-09-12T19:41:56.263 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.264 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:56.266 DEBUG:boto:Port: 8001 2022-09-12T19:41:56.267 DEBUG:boto:Params: {} 2022-09-12T19:41:56.268 DEBUG:boto:Token: None 2022-09-12T19:41:56.269 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/?versions 2022-09-12T19:41:56.271 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CbtKAqJEXgZEnb13BKb817NDSIE= 2022-09-12T19:41:56.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CbtKAqJEXgZEnb13BKb817NDSIE=', 'Content-Length': '0'} 2022-09-12T19:41:56.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f48b2da1e8a98ef-00631f8b84-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.279 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:56.280 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:56.281 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:56.281 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:56.282 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:56.282 DEBUG:boto:Method: GET 2022-09-12T19:41:56.283 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:56.283 DEBUG:boto:Data: 2022-09-12T19:41:56.284 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.284 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:56.285 DEBUG:boto:Port: 8000 2022-09-12T19:41:56.286 DEBUG:boto:Params: {} 2022-09-12T19:41:56.287 DEBUG:boto:Token: None 2022-09-12T19:41:56.288 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/?versions 2022-09-12T19:41:56.289 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CbtKAqJEXgZEnb13BKb817NDSIE= 2022-09-12T19:41:56.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CbtKAqJEXgZEnb13BKb817NDSIE=', 'Content-Length': '0'} 2022-09-12T19:41:56.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000581879fefd8221ab-00631f8b84-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.296 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:56.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:41:56.299 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:56.300 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:56.301 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:56.301 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:56.302 DEBUG:boto:Method: GET 2022-09-12T19:41:56.303 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:41:56.304 DEBUG:boto:Data: 2022-09-12T19:41:56.304 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.305 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:56.306 DEBUG:boto:Port: 8001 2022-09-12T19:41:56.306 DEBUG:boto:Params: {} 2022-09-12T19:41:56.307 DEBUG:boto:Token: None 2022-09-12T19:41:56.307 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:41:56.309 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YfCZMIj4wuC4zDeRW1Kx+CU5y8g= 2022-09-12T19:41:56.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YfCZMIj4wuC4zDeRW1Kx+CU5y8g=', 'Content-Length': '0'} 2022-09-12T19:41:56.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c8d85f0ac73699a-00631f8b84-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.313 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:56.314 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:56.314 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:56.315 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:56.316 DEBUG:boto:Method: GET 2022-09-12T19:41:56.317 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:41:56.317 DEBUG:boto:Data: 2022-09-12T19:41:56.318 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.318 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:56.319 DEBUG:boto:Port: 8000 2022-09-12T19:41:56.319 DEBUG:boto:Params: {} 2022-09-12T19:41:56.320 DEBUG:boto:Token: None 2022-09-12T19:41:56.321 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:41:56.322 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YfCZMIj4wuC4zDeRW1Kx+CU5y8g= 2022-09-12T19:41:56.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YfCZMIj4wuC4zDeRW1Kx+CU5y8g=', 'Content-Length': '0'} 2022-09-12T19:41:56.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e142632595f09288-00631f8b84-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.366 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:56.390 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:56.391 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:56.392 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:56.393 DEBUG:boto:Method: HEAD 2022-09-12T19:41:56.394 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:41:56.394 DEBUG:boto:Data: 2022-09-12T19:41:56.395 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.396 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:56.396 DEBUG:boto:Port: 8001 2022-09-12T19:41:56.397 DEBUG:boto:Params: {} 2022-09-12T19:41:56.397 DEBUG:boto:Token: None 2022-09-12T19:41:56.399 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:41:56.399 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QPqRRhs/eLi/l+zg31deg9BHTzM= 2022-09-12T19:41:56.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QPqRRhs/eLi/l+zg31deg9BHTzM=', 'Content-Length': '0'} 2022-09-12T19:41:56.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003874b2da89423fdd-00631f8b84-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.405 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:56.406 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:56.406 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:56.407 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:56.407 DEBUG:boto:Method: HEAD 2022-09-12T19:41:56.408 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:41:56.408 DEBUG:boto:Data: 2022-09-12T19:41:56.409 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.409 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:56.410 DEBUG:boto:Port: 8000 2022-09-12T19:41:56.411 DEBUG:boto:Params: {} 2022-09-12T19:41:56.411 DEBUG:boto:Token: None 2022-09-12T19:41:56.412 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:41:56.414 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QPqRRhs/eLi/l+zg31deg9BHTzM= 2022-09-12T19:41:56.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QPqRRhs/eLi/l+zg31deg9BHTzM=', 'Content-Length': '0'} 2022-09-12T19:41:56.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b30dfb0be994e3d-00631f8b84-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:41:56.420 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:56.421 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:56.421 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:56.422 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:56.423 DEBUG:boto:Method: GET 2022-09-12T19:41:56.424 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:41:56.427 DEBUG:boto:Data: 2022-09-12T19:41:56.428 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.429 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:56.430 DEBUG:boto:Port: 8001 2022-09-12T19:41:56.431 DEBUG:boto:Params: {} 2022-09-12T19:41:56.431 DEBUG:boto:Token: None 2022-09-12T19:41:56.432 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:41:56.433 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YfCZMIj4wuC4zDeRW1Kx+CU5y8g= 2022-09-12T19:41:56.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YfCZMIj4wuC4zDeRW1Kx+CU5y8g=', 'Content-Length': '0'} 2022-09-12T19:41:56.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c26d7ec7bb8a1416-00631f8b84-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.438 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:56.439 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:56.440 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:56.440 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:56.441 DEBUG:boto:Method: GET 2022-09-12T19:41:56.442 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:41:56.443 DEBUG:boto:Data: 2022-09-12T19:41:56.443 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.444 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:56.444 DEBUG:boto:Port: 8000 2022-09-12T19:41:56.445 DEBUG:boto:Params: {} 2022-09-12T19:41:56.445 DEBUG:boto:Token: None 2022-09-12T19:41:56.446 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:41:56.447 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YfCZMIj4wuC4zDeRW1Kx+CU5y8g= 2022-09-12T19:41:56.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YfCZMIj4wuC4zDeRW1Kx+CU5y8g=', 'Content-Length': '0'} 2022-09-12T19:41:56.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005718367310ddf62c-00631f8b84-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.456 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:56.456 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:56.457 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:41:56.458 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:41:56.460 DEBUG:boto:Method: GET 2022-09-12T19:41:56.464 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:41:56.465 DEBUG:boto:Data: 2022-09-12T19:41:56.466 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.467 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:56.468 DEBUG:boto:Port: 8001 2022-09-12T19:41:56.469 DEBUG:boto:Params: {} 2022-09-12T19:41:56.471 DEBUG:boto:Token: None 2022-09-12T19:41:56.472 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/ 2022-09-12T19:41:56.473 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V1Dp4GcM63vxx/k1zSckYHOHDxI= 2022-09-12T19:41:56.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V1Dp4GcM63vxx/k1zSckYHOHDxI=', 'Content-Length': '0'} 2022-09-12T19:41:56.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4f56ee746b4fa2a-00631f8b84-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.522 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:56.524 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:56.525 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:56.526 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:41:56.527 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:41:56.528 DEBUG:boto:Method: GET 2022-09-12T19:41:56.529 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:41:56.530 DEBUG:boto:Data: 2022-09-12T19:41:56.531 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.532 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:56.533 DEBUG:boto:Port: 8000 2022-09-12T19:41:56.534 DEBUG:boto:Params: {} 2022-09-12T19:41:56.535 DEBUG:boto:Token: None 2022-09-12T19:41:56.536 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/ 2022-09-12T19:41:56.537 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V1Dp4GcM63vxx/k1zSckYHOHDxI= 2022-09-12T19:41:56.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V1Dp4GcM63vxx/k1zSckYHOHDxI=', 'Content-Length': '0'} 2022-09-12T19:41:56.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c944af3f538b4425-00631f8b84-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.548 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:56.549 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:56.550 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:56.551 DEBUG:boto:Method: HEAD 2022-09-12T19:41:56.551 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:41:56.552 DEBUG:boto:Data: 2022-09-12T19:41:56.552 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.553 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:56.553 DEBUG:boto:Port: 8001 2022-09-12T19:41:56.554 DEBUG:boto:Params: {} 2022-09-12T19:41:56.555 DEBUG:boto:Token: None 2022-09-12T19:41:56.555 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/a 2022-09-12T19:41:56.556 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:slSVyHUPjes+3LRMVRWOsNUUqcw= 2022-09-12T19:41:56.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:slSVyHUPjes+3LRMVRWOsNUUqcw=', 'Content-Length': '0'} 2022-09-12T19:41:56.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040e1935f2f9f7f9b-00631f8b84-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.560 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:56.562 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:56.563 DEBUG:boto:Method: HEAD 2022-09-12T19:41:56.564 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:41:56.565 DEBUG:boto:Data: 2022-09-12T19:41:56.567 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.568 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:56.568 DEBUG:boto:Port: 8000 2022-09-12T19:41:56.569 DEBUG:boto:Params: {} 2022-09-12T19:41:56.570 DEBUG:boto:Token: None 2022-09-12T19:41:56.570 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/a 2022-09-12T19:41:56.571 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:slSVyHUPjes+3LRMVRWOsNUUqcw= 2022-09-12T19:41:56.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:slSVyHUPjes+3LRMVRWOsNUUqcw=', 'Content-Length': '0'} 2022-09-12T19:41:56.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2ba07483b83ad82-00631f8b84-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:41:56.588 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:56.589 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:56.590 DEBUG:boto:Method: GET 2022-09-12T19:41:56.591 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:41:56.591 DEBUG:boto:Data: 2022-09-12T19:41:56.592 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.593 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:56.593 DEBUG:boto:Port: 8001 2022-09-12T19:41:56.594 DEBUG:boto:Params: {} 2022-09-12T19:41:56.594 DEBUG:boto:Token: None 2022-09-12T19:41:56.595 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/a 2022-09-12T19:41:56.596 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PiOcB/HmTcKMBBXjdsqRYoaO0q0= 2022-09-12T19:41:56.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PiOcB/HmTcKMBBXjdsqRYoaO0q0=', 'Content-Length': '0'} 2022-09-12T19:41:56.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b3d994d4543119f-00631f8b84-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.600 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:56.602 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:56.603 DEBUG:boto:Method: GET 2022-09-12T19:41:56.604 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:41:56.605 DEBUG:boto:Data: 2022-09-12T19:41:56.606 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.607 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:56.615 DEBUG:boto:Port: 8000 2022-09-12T19:41:56.616 DEBUG:boto:Params: {} 2022-09-12T19:41:56.617 DEBUG:boto:Token: None 2022-09-12T19:41:56.624 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/a 2022-09-12T19:41:56.626 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PiOcB/HmTcKMBBXjdsqRYoaO0q0= 2022-09-12T19:41:56.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PiOcB/HmTcKMBBXjdsqRYoaO0q0=', 'Content-Length': '0'} 2022-09-12T19:41:56.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003d2a707665d2297-00631f8b84-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.630 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:56.630 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:56.631 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:41:56.632 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:41:56.632 DEBUG:boto:Method: GET 2022-09-12T19:41:56.633 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:41:56.634 DEBUG:boto:Data: 2022-09-12T19:41:56.635 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.635 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:56.636 DEBUG:boto:Port: 8001 2022-09-12T19:41:56.636 DEBUG:boto:Params: {} 2022-09-12T19:41:56.637 DEBUG:boto:Token: None 2022-09-12T19:41:56.637 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/ 2022-09-12T19:41:56.638 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V1Dp4GcM63vxx/k1zSckYHOHDxI= 2022-09-12T19:41:56.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V1Dp4GcM63vxx/k1zSckYHOHDxI=', 'Content-Length': '0'} 2022-09-12T19:41:56.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e13d81b9152ce3a6-00631f8b84-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.682 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:56.683 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:56.684 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:56.684 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:41:56.685 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:41:56.685 DEBUG:boto:Method: GET 2022-09-12T19:41:56.686 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:41:56.687 DEBUG:boto:Data: 2022-09-12T19:41:56.688 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.688 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:56.689 DEBUG:boto:Port: 8000 2022-09-12T19:41:56.690 DEBUG:boto:Params: {} 2022-09-12T19:41:56.690 DEBUG:boto:Token: None 2022-09-12T19:41:56.692 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/ 2022-09-12T19:41:56.692 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V1Dp4GcM63vxx/k1zSckYHOHDxI= 2022-09-12T19:41:56.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V1Dp4GcM63vxx/k1zSckYHOHDxI=', 'Content-Length': '0'} 2022-09-12T19:41:56.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005248aa79783ece9e-00631f8b84-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.698 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:56.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:41:56.700 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:56.701 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:56.702 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:56.703 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:56.704 DEBUG:boto:Method: GET 2022-09-12T19:41:56.705 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:41:56.707 DEBUG:boto:Data: 2022-09-12T19:41:56.708 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.709 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:56.710 DEBUG:boto:Port: 8001 2022-09-12T19:41:56.711 DEBUG:boto:Params: {} 2022-09-12T19:41:56.713 DEBUG:boto:Token: None 2022-09-12T19:41:56.714 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:41:56.715 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:URo4gh2F8lwyqCaK0N/Zjfwciew= 2022-09-12T19:41:56.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:URo4gh2F8lwyqCaK0N/Zjfwciew=', 'Content-Length': '0'} 2022-09-12T19:41:56.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb91097b4fca7251-00631f8b84-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.722 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:56.722 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:56.723 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:56.723 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:56.724 DEBUG:boto:Method: GET 2022-09-12T19:41:56.729 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:41:56.730 DEBUG:boto:Data: 2022-09-12T19:41:56.731 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.732 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:56.734 DEBUG:boto:Port: 8000 2022-09-12T19:41:56.736 DEBUG:boto:Params: {} 2022-09-12T19:41:56.737 DEBUG:boto:Token: None 2022-09-12T19:41:56.738 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:41:56.739 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:URo4gh2F8lwyqCaK0N/Zjfwciew= 2022-09-12T19:41:56.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:URo4gh2F8lwyqCaK0N/Zjfwciew=', 'Content-Length': '0'} 2022-09-12T19:41:56.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005eb12bee88b367d0-00631f8b84-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.745 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:56.747 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:56.749 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:56.750 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:56.751 DEBUG:boto:Method: HEAD 2022-09-12T19:41:56.751 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:41:56.752 DEBUG:boto:Data: 2022-09-12T19:41:56.753 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.754 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:56.755 DEBUG:boto:Port: 8001 2022-09-12T19:41:56.756 DEBUG:boto:Params: {} 2022-09-12T19:41:56.756 DEBUG:boto:Token: None 2022-09-12T19:41:56.757 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:41:56.758 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TrcTWZ+w1G8d9j41oTdHv8EcBSY= 2022-09-12T19:41:56.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TrcTWZ+w1G8d9j41oTdHv8EcBSY=', 'Content-Length': '0'} 2022-09-12T19:41:56.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a31ebfbfe317d8b-00631f8b84-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.762 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:56.762 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:56.763 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:56.764 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:56.764 DEBUG:boto:Method: HEAD 2022-09-12T19:41:56.765 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:41:56.765 DEBUG:boto:Data: 2022-09-12T19:41:56.766 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.767 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:56.767 DEBUG:boto:Port: 8000 2022-09-12T19:41:56.768 DEBUG:boto:Params: {} 2022-09-12T19:41:56.769 DEBUG:boto:Token: None 2022-09-12T19:41:56.770 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:41:56.770 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TrcTWZ+w1G8d9j41oTdHv8EcBSY= 2022-09-12T19:41:56.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TrcTWZ+w1G8d9j41oTdHv8EcBSY=', 'Content-Length': '0'} 2022-09-12T19:41:56.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000696c693c7332155e-00631f8b84-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:41:56.775 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:56.775 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:56.776 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:56.778 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:56.778 DEBUG:boto:Method: GET 2022-09-12T19:41:56.779 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:41:56.779 DEBUG:boto:Data: 2022-09-12T19:41:56.780 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.781 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:56.781 DEBUG:boto:Port: 8001 2022-09-12T19:41:56.782 DEBUG:boto:Params: {} 2022-09-12T19:41:56.782 DEBUG:boto:Token: None 2022-09-12T19:41:56.783 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:41:56.785 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:URo4gh2F8lwyqCaK0N/Zjfwciew= 2022-09-12T19:41:56.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:URo4gh2F8lwyqCaK0N/Zjfwciew=', 'Content-Length': '0'} 2022-09-12T19:41:56.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3a0762388d91633-00631f8b84-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.792 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:56.793 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:56.794 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:56.795 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:56.795 DEBUG:boto:Method: GET 2022-09-12T19:41:56.796 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:41:56.797 DEBUG:boto:Data: 2022-09-12T19:41:56.798 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.799 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:56.800 DEBUG:boto:Port: 8000 2022-09-12T19:41:56.801 DEBUG:boto:Params: {} 2022-09-12T19:41:56.802 DEBUG:boto:Token: None 2022-09-12T19:41:56.804 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:41:56.805 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:URo4gh2F8lwyqCaK0N/Zjfwciew= 2022-09-12T19:41:56.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:URo4gh2F8lwyqCaK0N/Zjfwciew=', 'Content-Length': '0'} 2022-09-12T19:41:56.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b9e214206bb131d-00631f8b84-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.859 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:56.860 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:56.861 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:41:56.863 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:41:56.864 DEBUG:boto:Method: GET 2022-09-12T19:41:56.865 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:41:56.866 DEBUG:boto:Data: 2022-09-12T19:41:56.868 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.869 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:56.870 DEBUG:boto:Port: 8001 2022-09-12T19:41:56.871 DEBUG:boto:Params: {} 2022-09-12T19:41:56.872 DEBUG:boto:Token: None 2022-09-12T19:41:56.873 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/ 2022-09-12T19:41:56.874 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V1Dp4GcM63vxx/k1zSckYHOHDxI= 2022-09-12T19:41:56.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V1Dp4GcM63vxx/k1zSckYHOHDxI=', 'Content-Length': '0'} 2022-09-12T19:41:56.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069d17985c7be25df-00631f8b84-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.878 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:56.879 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:56.880 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:56.881 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:41:56.882 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:41:56.882 DEBUG:boto:Method: GET 2022-09-12T19:41:56.883 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:41:56.884 DEBUG:boto:Data: 2022-09-12T19:41:56.884 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.885 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:56.885 DEBUG:boto:Port: 8000 2022-09-12T19:41:56.886 DEBUG:boto:Params: {} 2022-09-12T19:41:56.886 DEBUG:boto:Token: None 2022-09-12T19:41:56.887 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/ 2022-09-12T19:41:56.888 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V1Dp4GcM63vxx/k1zSckYHOHDxI= 2022-09-12T19:41:56.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V1Dp4GcM63vxx/k1zSckYHOHDxI=', 'Content-Length': '0'} 2022-09-12T19:41:56.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3fff3aec1b26788-00631f8b84-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.892 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:56.893 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:56.894 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:56.895 DEBUG:boto:Method: HEAD 2022-09-12T19:41:56.896 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:41:56.897 DEBUG:boto:Data: 2022-09-12T19:41:56.898 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.899 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:56.900 DEBUG:boto:Port: 8001 2022-09-12T19:41:56.901 DEBUG:boto:Params: {} 2022-09-12T19:41:56.902 DEBUG:boto:Token: None 2022-09-12T19:41:56.903 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/b 2022-09-12T19:41:56.904 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Z0SIw4OzCbM+2M0iB+MjPMw2F48= 2022-09-12T19:41:56.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Z0SIw4OzCbM+2M0iB+MjPMw2F48=', 'Content-Length': '0'} 2022-09-12T19:41:56.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c406dab92beb3e09-00631f8b84-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.909 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:56.911 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:56.912 DEBUG:boto:Method: HEAD 2022-09-12T19:41:56.913 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:41:56.913 DEBUG:boto:Data: 2022-09-12T19:41:56.914 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.915 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:56.916 DEBUG:boto:Port: 8000 2022-09-12T19:41:56.917 DEBUG:boto:Params: {} 2022-09-12T19:41:56.918 DEBUG:boto:Token: None 2022-09-12T19:41:56.919 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/b 2022-09-12T19:41:56.919 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Z0SIw4OzCbM+2M0iB+MjPMw2F48= 2022-09-12T19:41:56.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Z0SIw4OzCbM+2M0iB+MjPMw2F48=', 'Content-Length': '0'} 2022-09-12T19:41:56.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000696b970087632c8-00631f8b84-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:41:56.924 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:56.925 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:56.926 DEBUG:boto:Method: GET 2022-09-12T19:41:56.927 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:41:56.928 DEBUG:boto:Data: 2022-09-12T19:41:56.929 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.930 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:56.931 DEBUG:boto:Port: 8001 2022-09-12T19:41:56.932 DEBUG:boto:Params: {} 2022-09-12T19:41:56.933 DEBUG:boto:Token: None 2022-09-12T19:41:56.934 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/b 2022-09-12T19:41:56.935 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dqBFEZHYU9dOJ0/4j3VON6dwrvI= 2022-09-12T19:41:56.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dqBFEZHYU9dOJ0/4j3VON6dwrvI=', 'Content-Length': '0'} 2022-09-12T19:41:56.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cd187998cf66652-00631f8b84-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.941 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:56.942 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:56.944 DEBUG:boto:Method: GET 2022-09-12T19:41:56.945 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:41:56.946 DEBUG:boto:Data: 2022-09-12T19:41:56.947 DEBUG:boto:Headers: {} 2022-09-12T19:41:56.948 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:56.949 DEBUG:boto:Port: 8000 2022-09-12T19:41:56.950 DEBUG:boto:Params: {} 2022-09-12T19:41:56.951 DEBUG:boto:Token: None 2022-09-12T19:41:56.952 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:56 GMT /jdpiak-71/b 2022-09-12T19:41:56.953 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dqBFEZHYU9dOJ0/4j3VON6dwrvI= 2022-09-12T19:41:56.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dqBFEZHYU9dOJ0/4j3VON6dwrvI=', 'Content-Length': '0'} 2022-09-12T19:41:56.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074d77991898058a6-00631f8b84-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:56.999 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:57.000 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:57.001 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:41:57.001 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:41:57.003 DEBUG:boto:Method: GET 2022-09-12T19:41:57.004 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:41:57.006 DEBUG:boto:Data: 2022-09-12T19:41:57.008 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.009 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:57.010 DEBUG:boto:Port: 8001 2022-09-12T19:41:57.011 DEBUG:boto:Params: {} 2022-09-12T19:41:57.012 DEBUG:boto:Token: None 2022-09-12T19:41:57.012 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/ 2022-09-12T19:41:57.013 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eA3Y3X2ThrgukTYYUs1eBSokV7M= 2022-09-12T19:41:57.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eA3Y3X2ThrgukTYYUs1eBSokV7M=', 'Content-Length': '0'} 2022-09-12T19:41:57.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6081dd2859ec260-00631f8b85-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.020 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:57.022 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:57.023 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:57.024 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:41:57.025 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:41:57.028 DEBUG:boto:Method: GET 2022-09-12T19:41:57.031 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:41:57.032 DEBUG:boto:Data: 2022-09-12T19:41:57.033 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.034 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:57.035 DEBUG:boto:Port: 8000 2022-09-12T19:41:57.036 DEBUG:boto:Params: {} 2022-09-12T19:41:57.037 DEBUG:boto:Token: None 2022-09-12T19:41:57.038 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/ 2022-09-12T19:41:57.039 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eA3Y3X2ThrgukTYYUs1eBSokV7M= 2022-09-12T19:41:57.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eA3Y3X2ThrgukTYYUs1eBSokV7M=', 'Content-Length': '0'} 2022-09-12T19:41:57.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcfa2a29e3e7214b-00631f8b85-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.046 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:57.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:41:57.049 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:57.050 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:57.051 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:57.051 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:57.053 DEBUG:boto:Method: GET 2022-09-12T19:41:57.054 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:41:57.055 DEBUG:boto:Data: 2022-09-12T19:41:57.056 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.057 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:57.058 DEBUG:boto:Port: 8001 2022-09-12T19:41:57.059 DEBUG:boto:Params: {} 2022-09-12T19:41:57.060 DEBUG:boto:Token: None 2022-09-12T19:41:57.061 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:41:57.062 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kr6FrSL++ufFJllJ706fhp0rjUM= 2022-09-12T19:41:57.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kr6FrSL++ufFJllJ706fhp0rjUM=', 'Content-Length': '0'} 2022-09-12T19:41:57.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f70a9ea85845e5be-00631f8b85-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.071 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:57.071 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:57.072 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:57.072 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:57.073 DEBUG:boto:Method: GET 2022-09-12T19:41:57.074 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:41:57.074 DEBUG:boto:Data: 2022-09-12T19:41:57.075 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.076 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:57.077 DEBUG:boto:Port: 8000 2022-09-12T19:41:57.078 DEBUG:boto:Params: {} 2022-09-12T19:41:57.079 DEBUG:boto:Token: None 2022-09-12T19:41:57.080 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:41:57.082 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kr6FrSL++ufFJllJ706fhp0rjUM= 2022-09-12T19:41:57.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kr6FrSL++ufFJllJ706fhp0rjUM=', 'Content-Length': '0'} 2022-09-12T19:41:57.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000366c303717f2d004-00631f8b85-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.094 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:57.095 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:57.095 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:57.096 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:57.097 DEBUG:boto:Method: HEAD 2022-09-12T19:41:57.098 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:41:57.099 DEBUG:boto:Data: 2022-09-12T19:41:57.100 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.101 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:57.102 DEBUG:boto:Port: 8001 2022-09-12T19:41:57.103 DEBUG:boto:Params: {} 2022-09-12T19:41:57.104 DEBUG:boto:Token: None 2022-09-12T19:41:57.105 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:41:57.106 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f1KeqIH6IVyFpv1VTHXrx2xx+UI= 2022-09-12T19:41:57.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f1KeqIH6IVyFpv1VTHXrx2xx+UI=', 'Content-Length': '0'} 2022-09-12T19:41:57.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088abb2835dfd3c3e-00631f8b85-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.111 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:57.112 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:57.113 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:57.114 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:57.115 DEBUG:boto:Method: HEAD 2022-09-12T19:41:57.116 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:41:57.116 DEBUG:boto:Data: 2022-09-12T19:41:57.117 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.122 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:57.123 DEBUG:boto:Port: 8000 2022-09-12T19:41:57.124 DEBUG:boto:Params: {} 2022-09-12T19:41:57.125 DEBUG:boto:Token: None 2022-09-12T19:41:57.127 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:41:57.129 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f1KeqIH6IVyFpv1VTHXrx2xx+UI= 2022-09-12T19:41:57.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f1KeqIH6IVyFpv1VTHXrx2xx+UI=', 'Content-Length': '0'} 2022-09-12T19:41:57.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000157fab080b6ce003-00631f8b85-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:41:57.141 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:57.142 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:57.143 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:57.144 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:57.144 DEBUG:boto:Method: GET 2022-09-12T19:41:57.145 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:41:57.146 DEBUG:boto:Data: 2022-09-12T19:41:57.147 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.148 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:57.148 DEBUG:boto:Port: 8001 2022-09-12T19:41:57.149 DEBUG:boto:Params: {} 2022-09-12T19:41:57.150 DEBUG:boto:Token: None 2022-09-12T19:41:57.150 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:41:57.151 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kr6FrSL++ufFJllJ706fhp0rjUM= 2022-09-12T19:41:57.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kr6FrSL++ufFJllJ706fhp0rjUM=', 'Content-Length': '0'} 2022-09-12T19:41:57.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbcb578ed69950bf-00631f8b85-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.155 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:57.156 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:57.156 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:57.157 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:57.158 DEBUG:boto:Method: GET 2022-09-12T19:41:57.158 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:41:57.159 DEBUG:boto:Data: 2022-09-12T19:41:57.159 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.160 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:57.160 DEBUG:boto:Port: 8000 2022-09-12T19:41:57.161 DEBUG:boto:Params: {} 2022-09-12T19:41:57.162 DEBUG:boto:Token: None 2022-09-12T19:41:57.162 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:41:57.163 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kr6FrSL++ufFJllJ706fhp0rjUM= 2022-09-12T19:41:57.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kr6FrSL++ufFJllJ706fhp0rjUM=', 'Content-Length': '0'} 2022-09-12T19:41:57.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058f091c850251f48-00631f8b85-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.172 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:57.173 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:57.174 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:41:57.174 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:41:57.175 DEBUG:boto:Method: GET 2022-09-12T19:41:57.176 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:41:57.176 DEBUG:boto:Data: 2022-09-12T19:41:57.177 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.178 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:57.178 DEBUG:boto:Port: 8001 2022-09-12T19:41:57.179 DEBUG:boto:Params: {} 2022-09-12T19:41:57.179 DEBUG:boto:Token: None 2022-09-12T19:41:57.180 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/ 2022-09-12T19:41:57.181 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eA3Y3X2ThrgukTYYUs1eBSokV7M= 2022-09-12T19:41:57.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eA3Y3X2ThrgukTYYUs1eBSokV7M=', 'Content-Length': '0'} 2022-09-12T19:41:57.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002002a4f89b2f7d76-00631f8b85-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.186 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:57.187 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:57.188 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:57.189 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:41:57.190 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:41:57.190 DEBUG:boto:Method: GET 2022-09-12T19:41:57.195 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:41:57.196 DEBUG:boto:Data: 2022-09-12T19:41:57.198 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.199 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:57.200 DEBUG:boto:Port: 8000 2022-09-12T19:41:57.201 DEBUG:boto:Params: {} 2022-09-12T19:41:57.202 DEBUG:boto:Token: None 2022-09-12T19:41:57.203 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/ 2022-09-12T19:41:57.204 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eA3Y3X2ThrgukTYYUs1eBSokV7M= 2022-09-12T19:41:57.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eA3Y3X2ThrgukTYYUs1eBSokV7M=', 'Content-Length': '0'} 2022-09-12T19:41:57.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee1a12d72b313d83-00631f8b85-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.210 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:57.211 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:57.212 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:57.213 DEBUG:boto:Method: HEAD 2022-09-12T19:41:57.213 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:41:57.214 DEBUG:boto:Data: 2022-09-12T19:41:57.215 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.216 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:57.217 DEBUG:boto:Port: 8001 2022-09-12T19:41:57.218 DEBUG:boto:Params: {} 2022-09-12T19:41:57.219 DEBUG:boto:Token: None 2022-09-12T19:41:57.220 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/c 2022-09-12T19:41:57.221 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jmpIoVOvNNTcXWtixUjb6M+XU88= 2022-09-12T19:41:57.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jmpIoVOvNNTcXWtixUjb6M+XU88=', 'Content-Length': '0'} 2022-09-12T19:41:57.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001881e0518b54e6d8-00631f8b85-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.226 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:57.227 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:57.228 DEBUG:boto:Method: HEAD 2022-09-12T19:41:57.228 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:41:57.229 DEBUG:boto:Data: 2022-09-12T19:41:57.230 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.231 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:57.232 DEBUG:boto:Port: 8000 2022-09-12T19:41:57.233 DEBUG:boto:Params: {} 2022-09-12T19:41:57.234 DEBUG:boto:Token: None 2022-09-12T19:41:57.236 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/c 2022-09-12T19:41:57.237 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jmpIoVOvNNTcXWtixUjb6M+XU88= 2022-09-12T19:41:57.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jmpIoVOvNNTcXWtixUjb6M+XU88=', 'Content-Length': '0'} 2022-09-12T19:41:57.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afe49f1ad583906a-00631f8b85-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:41:57.242 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:57.243 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:57.244 DEBUG:boto:Method: GET 2022-09-12T19:41:57.244 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:41:57.245 DEBUG:boto:Data: 2022-09-12T19:41:57.245 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.246 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:57.246 DEBUG:boto:Port: 8001 2022-09-12T19:41:57.247 DEBUG:boto:Params: {} 2022-09-12T19:41:57.247 DEBUG:boto:Token: None 2022-09-12T19:41:57.248 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/c 2022-09-12T19:41:57.250 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gCLw5QFWtDlkSfiu8ukrc6pvK6w= 2022-09-12T19:41:57.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gCLw5QFWtDlkSfiu8ukrc6pvK6w=', 'Content-Length': '0'} 2022-09-12T19:41:57.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c140d1b708109a9d-00631f8b85-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.254 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:57.255 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:57.255 DEBUG:boto:Method: GET 2022-09-12T19:41:57.256 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:41:57.256 DEBUG:boto:Data: 2022-09-12T19:41:57.257 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.257 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:57.258 DEBUG:boto:Port: 8000 2022-09-12T19:41:57.259 DEBUG:boto:Params: {} 2022-09-12T19:41:57.260 DEBUG:boto:Token: None 2022-09-12T19:41:57.261 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/c 2022-09-12T19:41:57.263 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gCLw5QFWtDlkSfiu8ukrc6pvK6w= 2022-09-12T19:41:57.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gCLw5QFWtDlkSfiu8ukrc6pvK6w=', 'Content-Length': '0'} 2022-09-12T19:41:57.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065fe46039955b85f-00631f8b85-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.269 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:57.270 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:57.271 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:41:57.272 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:41:57.273 DEBUG:boto:Method: GET 2022-09-12T19:41:57.273 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:41:57.274 DEBUG:boto:Data: 2022-09-12T19:41:57.275 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.276 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:57.277 DEBUG:boto:Port: 8001 2022-09-12T19:41:57.278 DEBUG:boto:Params: {} 2022-09-12T19:41:57.280 DEBUG:boto:Token: None 2022-09-12T19:41:57.281 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/ 2022-09-12T19:41:57.282 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eA3Y3X2ThrgukTYYUs1eBSokV7M= 2022-09-12T19:41:57.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eA3Y3X2ThrgukTYYUs1eBSokV7M=', 'Content-Length': '0'} 2022-09-12T19:41:57.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5773fa53a0d81ca-00631f8b85-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.289 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:57.290 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:57.292 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:57.293 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:41:57.295 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:41:57.296 DEBUG:boto:Method: GET 2022-09-12T19:41:57.297 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:41:57.298 DEBUG:boto:Data: 2022-09-12T19:41:57.299 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.300 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:57.301 DEBUG:boto:Port: 8000 2022-09-12T19:41:57.302 DEBUG:boto:Params: {} 2022-09-12T19:41:57.304 DEBUG:boto:Token: None 2022-09-12T19:41:57.305 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/ 2022-09-12T19:41:57.306 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eA3Y3X2ThrgukTYYUs1eBSokV7M= 2022-09-12T19:41:57.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eA3Y3X2ThrgukTYYUs1eBSokV7M=', 'Content-Length': '0'} 2022-09-12T19:41:57.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cee3551f2248dca9-00631f8b85-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.319 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:57.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:41:57.321 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:57.322 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:57.323 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:57.324 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:57.325 DEBUG:boto:Method: GET 2022-09-12T19:41:57.326 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:41:57.327 DEBUG:boto:Data: 2022-09-12T19:41:57.328 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.329 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:57.330 DEBUG:boto:Port: 8001 2022-09-12T19:41:57.331 DEBUG:boto:Params: {} 2022-09-12T19:41:57.332 DEBUG:boto:Token: None 2022-09-12T19:41:57.333 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:41:57.334 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sfUcxTsoTzaSxvHi0no9BhELRm4= 2022-09-12T19:41:57.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sfUcxTsoTzaSxvHi0no9BhELRm4=', 'Content-Length': '0'} 2022-09-12T19:41:57.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000299e54780d0217c-00631f8b85-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.340 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:57.341 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:57.345 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:57.346 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:57.348 DEBUG:boto:Method: GET 2022-09-12T19:41:57.349 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:41:57.350 DEBUG:boto:Data: 2022-09-12T19:41:57.351 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.352 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:57.352 DEBUG:boto:Port: 8000 2022-09-12T19:41:57.353 DEBUG:boto:Params: {} 2022-09-12T19:41:57.354 DEBUG:boto:Token: None 2022-09-12T19:41:57.355 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:41:57.355 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sfUcxTsoTzaSxvHi0no9BhELRm4= 2022-09-12T19:41:57.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sfUcxTsoTzaSxvHi0no9BhELRm4=', 'Content-Length': '0'} 2022-09-12T19:41:57.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e87e54d259efbb50-00631f8b85-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.402 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:57.405 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:57.405 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:57.406 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:57.407 DEBUG:boto:Method: HEAD 2022-09-12T19:41:57.407 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:41:57.408 DEBUG:boto:Data: 2022-09-12T19:41:57.409 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.409 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:57.410 DEBUG:boto:Port: 8001 2022-09-12T19:41:57.410 DEBUG:boto:Params: {} 2022-09-12T19:41:57.411 DEBUG:boto:Token: None 2022-09-12T19:41:57.412 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:41:57.412 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+TlJR4mvJKtMb4yhfP3ufB5rFaw= 2022-09-12T19:41:57.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+TlJR4mvJKtMb4yhfP3ufB5rFaw=', 'Content-Length': '0'} 2022-09-12T19:41:57.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000745cb1f111db8243-00631f8b85-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.418 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:57.422 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:57.423 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:57.424 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:57.430 DEBUG:boto:Method: HEAD 2022-09-12T19:41:57.431 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:41:57.432 DEBUG:boto:Data: 2022-09-12T19:41:57.433 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.434 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:57.435 DEBUG:boto:Port: 8000 2022-09-12T19:41:57.436 DEBUG:boto:Params: {} 2022-09-12T19:41:57.437 DEBUG:boto:Token: None 2022-09-12T19:41:57.438 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:41:57.440 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+TlJR4mvJKtMb4yhfP3ufB5rFaw= 2022-09-12T19:41:57.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+TlJR4mvJKtMb4yhfP3ufB5rFaw=', 'Content-Length': '0'} 2022-09-12T19:41:57.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025c8b5fe21263f79-00631f8b85-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:41:57.448 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:57.448 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:57.449 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:57.449 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:57.450 DEBUG:boto:Method: GET 2022-09-12T19:41:57.451 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:41:57.452 DEBUG:boto:Data: 2022-09-12T19:41:57.453 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.454 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:57.455 DEBUG:boto:Port: 8001 2022-09-12T19:41:57.455 DEBUG:boto:Params: {} 2022-09-12T19:41:57.456 DEBUG:boto:Token: None 2022-09-12T19:41:57.457 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:41:57.458 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sfUcxTsoTzaSxvHi0no9BhELRm4= 2022-09-12T19:41:57.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sfUcxTsoTzaSxvHi0no9BhELRm4=', 'Content-Length': '0'} 2022-09-12T19:41:57.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007530ba72cf807eb1-00631f8b85-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.469 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:57.470 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:57.471 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:57.472 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:57.476 DEBUG:boto:Method: GET 2022-09-12T19:41:57.478 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:41:57.482 DEBUG:boto:Data: 2022-09-12T19:41:57.483 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.496 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:57.497 DEBUG:boto:Port: 8000 2022-09-12T19:41:57.498 DEBUG:boto:Params: {} 2022-09-12T19:41:57.499 DEBUG:boto:Token: None 2022-09-12T19:41:57.500 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:41:57.501 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sfUcxTsoTzaSxvHi0no9BhELRm4= 2022-09-12T19:41:57.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sfUcxTsoTzaSxvHi0no9BhELRm4=', 'Content-Length': '0'} 2022-09-12T19:41:57.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c008466fbe8b1cc-00631f8b85-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.515 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:57.516 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:57.516 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:41:57.517 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:41:57.517 DEBUG:boto:Method: GET 2022-09-12T19:41:57.518 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:41:57.518 DEBUG:boto:Data: 2022-09-12T19:41:57.519 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.519 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:57.520 DEBUG:boto:Port: 8001 2022-09-12T19:41:57.521 DEBUG:boto:Params: {} 2022-09-12T19:41:57.521 DEBUG:boto:Token: None 2022-09-12T19:41:57.522 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/ 2022-09-12T19:41:57.523 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eA3Y3X2ThrgukTYYUs1eBSokV7M= 2022-09-12T19:41:57.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eA3Y3X2ThrgukTYYUs1eBSokV7M=', 'Content-Length': '0'} 2022-09-12T19:41:57.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051583b18a9bff89d-00631f8b85-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.527 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:57.528 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:57.529 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:57.530 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:41:57.530 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:41:57.531 DEBUG:boto:Method: GET 2022-09-12T19:41:57.531 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:41:57.532 DEBUG:boto:Data: 2022-09-12T19:41:57.533 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.533 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:57.534 DEBUG:boto:Port: 8000 2022-09-12T19:41:57.534 DEBUG:boto:Params: {} 2022-09-12T19:41:57.535 DEBUG:boto:Token: None 2022-09-12T19:41:57.539 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/ 2022-09-12T19:41:57.541 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eA3Y3X2ThrgukTYYUs1eBSokV7M= 2022-09-12T19:41:57.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eA3Y3X2ThrgukTYYUs1eBSokV7M=', 'Content-Length': '0'} 2022-09-12T19:41:57.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c206b43f842ff05d-00631f8b85-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.551 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:57.553 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:57.554 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:57.555 DEBUG:boto:Method: HEAD 2022-09-12T19:41:57.556 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:41:57.556 DEBUG:boto:Data: 2022-09-12T19:41:57.557 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.557 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:57.558 DEBUG:boto:Port: 8001 2022-09-12T19:41:57.559 DEBUG:boto:Params: {} 2022-09-12T19:41:57.559 DEBUG:boto:Token: None 2022-09-12T19:41:57.560 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/d 2022-09-12T19:41:57.561 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XRHGK7bafNcQ2Ux98nCnZe7Kc5I= 2022-09-12T19:41:57.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XRHGK7bafNcQ2Ux98nCnZe7Kc5I=', 'Content-Length': '0'} 2022-09-12T19:41:57.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023c6a09e7b9c314f-00631f8b85-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.565 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:57.566 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:57.566 DEBUG:boto:Method: HEAD 2022-09-12T19:41:57.567 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:41:57.570 DEBUG:boto:Data: 2022-09-12T19:41:57.571 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.572 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:57.573 DEBUG:boto:Port: 8000 2022-09-12T19:41:57.576 DEBUG:boto:Params: {} 2022-09-12T19:41:57.577 DEBUG:boto:Token: None 2022-09-12T19:41:57.578 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/d 2022-09-12T19:41:57.579 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XRHGK7bafNcQ2Ux98nCnZe7Kc5I= 2022-09-12T19:41:57.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XRHGK7bafNcQ2Ux98nCnZe7Kc5I=', 'Content-Length': '0'} 2022-09-12T19:41:57.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077246f22956889ec-00631f8b85-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:41:57.588 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:57.589 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:57.590 DEBUG:boto:Method: GET 2022-09-12T19:41:57.591 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:41:57.592 DEBUG:boto:Data: 2022-09-12T19:41:57.593 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.594 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:57.594 DEBUG:boto:Port: 8001 2022-09-12T19:41:57.595 DEBUG:boto:Params: {} 2022-09-12T19:41:57.596 DEBUG:boto:Token: None 2022-09-12T19:41:57.596 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/d 2022-09-12T19:41:57.597 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Z7RS2YvgpNdolKeTNE7NXOrQRyU= 2022-09-12T19:41:57.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Z7RS2YvgpNdolKeTNE7NXOrQRyU=', 'Content-Length': '0'} 2022-09-12T19:41:57.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000bfcf295e741f44-00631f8b85-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.642 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:57.643 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:57.644 DEBUG:boto:Method: GET 2022-09-12T19:41:57.645 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:41:57.645 DEBUG:boto:Data: 2022-09-12T19:41:57.646 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.647 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:57.648 DEBUG:boto:Port: 8000 2022-09-12T19:41:57.649 DEBUG:boto:Params: {} 2022-09-12T19:41:57.650 DEBUG:boto:Token: None 2022-09-12T19:41:57.651 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/d 2022-09-12T19:41:57.652 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Z7RS2YvgpNdolKeTNE7NXOrQRyU= 2022-09-12T19:41:57.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Z7RS2YvgpNdolKeTNE7NXOrQRyU=', 'Content-Length': '0'} 2022-09-12T19:41:57.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fa61ac865b91c0a-00631f8b85-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.656 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:57.657 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:57.658 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:41:57.658 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:41:57.659 DEBUG:boto:Method: GET 2022-09-12T19:41:57.659 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:41:57.660 DEBUG:boto:Data: 2022-09-12T19:41:57.661 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.661 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:57.662 DEBUG:boto:Port: 8001 2022-09-12T19:41:57.662 DEBUG:boto:Params: {} 2022-09-12T19:41:57.663 DEBUG:boto:Token: None 2022-09-12T19:41:57.665 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/ 2022-09-12T19:41:57.666 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eA3Y3X2ThrgukTYYUs1eBSokV7M= 2022-09-12T19:41:57.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eA3Y3X2ThrgukTYYUs1eBSokV7M=', 'Content-Length': '0'} 2022-09-12T19:41:57.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6e11d5b8f1b17b9-00631f8b85-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.671 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:57.672 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:57.672 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:57.673 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:41:57.673 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:41:57.674 DEBUG:boto:Method: GET 2022-09-12T19:41:57.675 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:41:57.675 DEBUG:boto:Data: 2022-09-12T19:41:57.676 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.676 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:57.677 DEBUG:boto:Port: 8000 2022-09-12T19:41:57.677 DEBUG:boto:Params: {} 2022-09-12T19:41:57.679 DEBUG:boto:Token: None 2022-09-12T19:41:57.680 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/ 2022-09-12T19:41:57.680 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eA3Y3X2ThrgukTYYUs1eBSokV7M= 2022-09-12T19:41:57.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eA3Y3X2ThrgukTYYUs1eBSokV7M=', 'Content-Length': '0'} 2022-09-12T19:41:57.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006502683a6e09c043-00631f8b85-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.685 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:57.686 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-71 zones={test-zone3, test-zone1} 2022-09-12T19:41:57.686 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-71 zones={test-zone3, test-zone2} 2022-09-12T19:41:57.687 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:57.687 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:57.688 DEBUG:boto:Method: HEAD 2022-09-12T19:41:57.689 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:41:57.689 DEBUG:boto:Data: 2022-09-12T19:41:57.690 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.691 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:57.693 DEBUG:boto:Port: 8001 2022-09-12T19:41:57.694 DEBUG:boto:Params: {} 2022-09-12T19:41:57.695 DEBUG:boto:Token: None 2022-09-12T19:41:57.696 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/ 2022-09-12T19:41:57.697 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zeFYAyfc6Prj3fiFpw5b3IkVmhw= 2022-09-12T19:41:57.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zeFYAyfc6Prj3fiFpw5b3IkVmhw=', 'Content-Length': '0'} 2022-09-12T19:41:57.701 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004970d698786f6382-00631f8b85-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.703 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:57.703 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:57.704 DEBUG:boto:Method: HEAD 2022-09-12T19:41:57.704 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:41:57.705 DEBUG:boto:Data: 2022-09-12T19:41:57.705 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.706 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:57.706 DEBUG:boto:Port: 8000 2022-09-12T19:41:57.707 DEBUG:boto:Params: {} 2022-09-12T19:41:57.707 DEBUG:boto:Token: None 2022-09-12T19:41:57.708 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/ 2022-09-12T19:41:57.709 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zeFYAyfc6Prj3fiFpw5b3IkVmhw= 2022-09-12T19:41:57.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zeFYAyfc6Prj3fiFpw5b3IkVmhw=', 'Content-Length': '0'} 2022-09-12T19:41:57.712 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ec9883626f5b4e5-00631f8b85-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.713 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:41:57.713 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:57.714 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:57.715 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:57.715 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:57.716 DEBUG:boto:Method: GET 2022-09-12T19:41:57.716 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:57.717 DEBUG:boto:Data: 2022-09-12T19:41:57.717 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.718 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:57.719 DEBUG:boto:Port: 8001 2022-09-12T19:41:57.719 DEBUG:boto:Params: {} 2022-09-12T19:41:57.720 DEBUG:boto:Token: None 2022-09-12T19:41:57.721 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/?versions 2022-09-12T19:41:57.722 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kdvylg47pRHKLYOCXyJ6w+7mUNY= 2022-09-12T19:41:57.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kdvylg47pRHKLYOCXyJ6w+7mUNY=', 'Content-Length': '0'} 2022-09-12T19:41:57.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000407b1c1722f72c62-00631f8b85-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.728 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:57.729 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:41:57.731 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:41:57.732 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:41:57.732 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:41:57.733 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:41:57.734 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:57.735 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:57.736 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:57.737 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:57.738 DEBUG:boto:Method: GET 2022-09-12T19:41:57.739 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:57.740 DEBUG:boto:Data: 2022-09-12T19:41:57.741 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.742 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:57.743 DEBUG:boto:Port: 8000 2022-09-12T19:41:57.743 DEBUG:boto:Params: {} 2022-09-12T19:41:57.744 DEBUG:boto:Token: None 2022-09-12T19:41:57.745 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/?versions 2022-09-12T19:41:57.745 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kdvylg47pRHKLYOCXyJ6w+7mUNY= 2022-09-12T19:41:57.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kdvylg47pRHKLYOCXyJ6w+7mUNY=', 'Content-Length': '0'} 2022-09-12T19:41:57.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c203d70a0c299e63-00631f8b85-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.749 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:57.750 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:41:57.751 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:41:57.752 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:41:57.752 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:41:57.753 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:57.753 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:57.768 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:57.769 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:57.770 DEBUG:boto:Method: GET 2022-09-12T19:41:57.771 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:57.771 DEBUG:boto:Data: 2022-09-12T19:41:57.772 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.773 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:57.773 DEBUG:boto:Port: 8001 2022-09-12T19:41:57.774 DEBUG:boto:Params: {} 2022-09-12T19:41:57.774 DEBUG:boto:Token: None 2022-09-12T19:41:57.775 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/?versions 2022-09-12T19:41:57.776 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kdvylg47pRHKLYOCXyJ6w+7mUNY= 2022-09-12T19:41:57.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kdvylg47pRHKLYOCXyJ6w+7mUNY=', 'Content-Length': '0'} 2022-09-12T19:41:57.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000126b8e51dceb6e3f-00631f8b85-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.790 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:57.791 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:57.792 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:57.793 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:57.794 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:57.795 DEBUG:boto:Method: GET 2022-09-12T19:41:57.796 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:41:57.797 DEBUG:boto:Data: 2022-09-12T19:41:57.798 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.799 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:57.800 DEBUG:boto:Port: 8000 2022-09-12T19:41:57.801 DEBUG:boto:Params: {} 2022-09-12T19:41:57.802 DEBUG:boto:Token: None 2022-09-12T19:41:57.803 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/?versions 2022-09-12T19:41:57.803 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kdvylg47pRHKLYOCXyJ6w+7mUNY= 2022-09-12T19:41:57.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kdvylg47pRHKLYOCXyJ6w+7mUNY=', 'Content-Length': '0'} 2022-09-12T19:41:57.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4c3bbec69c22510-00631f8b85-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.809 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:57.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:41:57.812 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:57.813 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:57.814 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:57.815 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:57.817 DEBUG:boto:Method: GET 2022-09-12T19:41:57.818 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:41:57.819 DEBUG:boto:Data: 2022-09-12T19:41:57.819 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.820 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:57.820 DEBUG:boto:Port: 8001 2022-09-12T19:41:57.821 DEBUG:boto:Params: {} 2022-09-12T19:41:57.822 DEBUG:boto:Token: None 2022-09-12T19:41:57.823 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:41:57.823 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uP3Oy55wTYI80BokeIzmF0uZkCI= 2022-09-12T19:41:57.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uP3Oy55wTYI80BokeIzmF0uZkCI=', 'Content-Length': '0'} 2022-09-12T19:41:57.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d121aa472de0f486-00631f8b85-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.827 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:57.828 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:57.828 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:57.829 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:57.831 DEBUG:boto:Method: GET 2022-09-12T19:41:57.832 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:41:57.832 DEBUG:boto:Data: 2022-09-12T19:41:57.833 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.834 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:57.835 DEBUG:boto:Port: 8000 2022-09-12T19:41:57.837 DEBUG:boto:Params: {} 2022-09-12T19:41:57.838 DEBUG:boto:Token: None 2022-09-12T19:41:57.839 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:41:57.840 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uP3Oy55wTYI80BokeIzmF0uZkCI= 2022-09-12T19:41:57.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uP3Oy55wTYI80BokeIzmF0uZkCI=', 'Content-Length': '0'} 2022-09-12T19:41:57.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cb22daf78e26250-00631f8b85-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.846 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:57.847 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:57.848 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:57.848 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:57.849 DEBUG:boto:Method: HEAD 2022-09-12T19:41:57.849 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:41:57.850 DEBUG:boto:Data: 2022-09-12T19:41:57.850 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.851 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:57.851 DEBUG:boto:Port: 8001 2022-09-12T19:41:57.852 DEBUG:boto:Params: {} 2022-09-12T19:41:57.852 DEBUG:boto:Token: None 2022-09-12T19:41:57.853 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:41:57.853 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wixY5C70qz0RSCewTKoyGaE9uF4= 2022-09-12T19:41:57.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wixY5C70qz0RSCewTKoyGaE9uF4=', 'Content-Length': '0'} 2022-09-12T19:41:57.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010532520c408f0ef-00631f8b85-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.859 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:57.863 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:57.865 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:57.866 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:57.867 DEBUG:boto:Method: HEAD 2022-09-12T19:41:57.867 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:41:57.868 DEBUG:boto:Data: 2022-09-12T19:41:57.869 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.871 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:57.871 DEBUG:boto:Port: 8000 2022-09-12T19:41:57.872 DEBUG:boto:Params: {} 2022-09-12T19:41:57.873 DEBUG:boto:Token: None 2022-09-12T19:41:57.875 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:41:57.876 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wixY5C70qz0RSCewTKoyGaE9uF4= 2022-09-12T19:41:57.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wixY5C70qz0RSCewTKoyGaE9uF4=', 'Content-Length': '0'} 2022-09-12T19:41:57.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e443c50aa99504c-00631f8b85-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:41:57.887 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:57.888 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:57.889 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:57.889 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:57.896 DEBUG:boto:Method: GET 2022-09-12T19:41:57.897 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:41:57.898 DEBUG:boto:Data: 2022-09-12T19:41:57.899 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.900 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:57.901 DEBUG:boto:Port: 8001 2022-09-12T19:41:57.902 DEBUG:boto:Params: {} 2022-09-12T19:41:57.903 DEBUG:boto:Token: None 2022-09-12T19:41:57.904 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:41:57.905 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uP3Oy55wTYI80BokeIzmF0uZkCI= 2022-09-12T19:41:57.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uP3Oy55wTYI80BokeIzmF0uZkCI=', 'Content-Length': '0'} 2022-09-12T19:41:57.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dafe40fce8c3b140-00631f8b85-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.911 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:57.912 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:57.913 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:57.913 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:41:57.914 DEBUG:boto:Method: GET 2022-09-12T19:41:57.914 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:41:57.915 DEBUG:boto:Data: 2022-09-12T19:41:57.915 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.916 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:57.917 DEBUG:boto:Port: 8000 2022-09-12T19:41:57.918 DEBUG:boto:Params: {} 2022-09-12T19:41:57.918 DEBUG:boto:Token: None 2022-09-12T19:41:57.919 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:41:57.920 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uP3Oy55wTYI80BokeIzmF0uZkCI= 2022-09-12T19:41:57.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uP3Oy55wTYI80BokeIzmF0uZkCI=', 'Content-Length': '0'} 2022-09-12T19:41:57.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d2afdccee4fb103-00631f8b85-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.968 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:57.969 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:57.970 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:41:57.971 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:41:57.972 DEBUG:boto:Method: GET 2022-09-12T19:41:57.973 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:41:57.974 DEBUG:boto:Data: 2022-09-12T19:41:57.975 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.976 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:57.977 DEBUG:boto:Port: 8001 2022-09-12T19:41:57.978 DEBUG:boto:Params: {} 2022-09-12T19:41:57.979 DEBUG:boto:Token: None 2022-09-12T19:41:57.980 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:57 GMT /jdpiak-71/ 2022-09-12T19:41:57.981 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eA3Y3X2ThrgukTYYUs1eBSokV7M= 2022-09-12T19:41:57.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eA3Y3X2ThrgukTYYUs1eBSokV7M=', 'Content-Length': '0'} 2022-09-12T19:41:57.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003698f6e90e9155a6-00631f8b85-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:57.990 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:57.992 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:57.993 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:57.994 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:41:57.994 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:41:57.995 DEBUG:boto:Method: GET 2022-09-12T19:41:57.996 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:41:57.997 DEBUG:boto:Data: 2022-09-12T19:41:57.998 DEBUG:boto:Headers: {} 2022-09-12T19:41:57.999 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:58.000 DEBUG:boto:Port: 8000 2022-09-12T19:41:58.001 DEBUG:boto:Params: {} 2022-09-12T19:41:58.007 DEBUG:boto:Token: None 2022-09-12T19:41:58.008 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/ 2022-09-12T19:41:58.015 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg= 2022-09-12T19:41:58.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg=', 'Content-Length': '0'} 2022-09-12T19:41:58.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8e301d8fb934e26-00631f8b86-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.021 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:58.022 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:58.023 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:58.024 DEBUG:boto:Method: HEAD 2022-09-12T19:41:58.026 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:41:58.028 DEBUG:boto:Data: 2022-09-12T19:41:58.029 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.030 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:58.031 DEBUG:boto:Port: 8001 2022-09-12T19:41:58.032 DEBUG:boto:Params: {} 2022-09-12T19:41:58.033 DEBUG:boto:Token: None 2022-09-12T19:41:58.034 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/a 2022-09-12T19:41:58.035 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8b/XrWkJ/24UXu6XXEyT46SAcLU= 2022-09-12T19:41:58.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8b/XrWkJ/24UXu6XXEyT46SAcLU=', 'Content-Length': '0'} 2022-09-12T19:41:58.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f6b1d1da4dd8905-00631f8b86-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.063 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:58.065 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:58.066 DEBUG:boto:Method: HEAD 2022-09-12T19:41:58.067 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:41:58.068 DEBUG:boto:Data: 2022-09-12T19:41:58.070 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.070 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:58.071 DEBUG:boto:Port: 8000 2022-09-12T19:41:58.072 DEBUG:boto:Params: {} 2022-09-12T19:41:58.074 DEBUG:boto:Token: None 2022-09-12T19:41:58.075 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/a 2022-09-12T19:41:58.077 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8b/XrWkJ/24UXu6XXEyT46SAcLU= 2022-09-12T19:41:58.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8b/XrWkJ/24UXu6XXEyT46SAcLU=', 'Content-Length': '0'} 2022-09-12T19:41:58.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa081f1c6e486657-00631f8b86-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:41:58.083 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:58.083 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:58.084 DEBUG:boto:Method: GET 2022-09-12T19:41:58.085 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:41:58.085 DEBUG:boto:Data: 2022-09-12T19:41:58.086 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.087 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:58.087 DEBUG:boto:Port: 8001 2022-09-12T19:41:58.088 DEBUG:boto:Params: {} 2022-09-12T19:41:58.088 DEBUG:boto:Token: None 2022-09-12T19:41:58.089 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/a 2022-09-12T19:41:58.090 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tWEhNE9h6jAi9TajMGxhvrsnMIQ= 2022-09-12T19:41:58.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tWEhNE9h6jAi9TajMGxhvrsnMIQ=', 'Content-Length': '0'} 2022-09-12T19:41:58.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079319b6beae0f5c6-00631f8b86-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.094 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:41:58.095 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:41:58.096 DEBUG:boto:Method: GET 2022-09-12T19:41:58.096 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:41:58.097 DEBUG:boto:Data: 2022-09-12T19:41:58.097 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.098 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:58.099 DEBUG:boto:Port: 8000 2022-09-12T19:41:58.099 DEBUG:boto:Params: {} 2022-09-12T19:41:58.100 DEBUG:boto:Token: None 2022-09-12T19:41:58.101 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/a 2022-09-12T19:41:58.101 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tWEhNE9h6jAi9TajMGxhvrsnMIQ= 2022-09-12T19:41:58.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tWEhNE9h6jAi9TajMGxhvrsnMIQ=', 'Content-Length': '0'} 2022-09-12T19:41:58.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e81631d2f1ed2030-00631f8b86-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.105 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:58.106 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:58.106 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:41:58.107 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:41:58.107 DEBUG:boto:Method: GET 2022-09-12T19:41:58.108 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:41:58.108 DEBUG:boto:Data: 2022-09-12T19:41:58.109 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.110 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:58.110 DEBUG:boto:Port: 8001 2022-09-12T19:41:58.111 DEBUG:boto:Params: {} 2022-09-12T19:41:58.111 DEBUG:boto:Token: None 2022-09-12T19:41:58.112 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/ 2022-09-12T19:41:58.113 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg= 2022-09-12T19:41:58.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg=', 'Content-Length': '0'} 2022-09-12T19:41:58.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075531a21706672c5-00631f8b86-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.119 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:58.120 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:58.121 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:58.122 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:41:58.123 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:41:58.124 DEBUG:boto:Method: GET 2022-09-12T19:41:58.125 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:41:58.126 DEBUG:boto:Data: 2022-09-12T19:41:58.127 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.128 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:58.129 DEBUG:boto:Port: 8000 2022-09-12T19:41:58.130 DEBUG:boto:Params: {} 2022-09-12T19:41:58.131 DEBUG:boto:Token: None 2022-09-12T19:41:58.132 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/ 2022-09-12T19:41:58.133 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg= 2022-09-12T19:41:58.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg=', 'Content-Length': '0'} 2022-09-12T19:41:58.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a90cf1397d30e1b6-00631f8b86-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.138 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:58.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:41:58.139 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:58.140 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:58.141 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:58.142 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:58.143 DEBUG:boto:Method: GET 2022-09-12T19:41:58.144 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:41:58.145 DEBUG:boto:Data: 2022-09-12T19:41:58.146 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.147 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:58.148 DEBUG:boto:Port: 8001 2022-09-12T19:41:58.149 DEBUG:boto:Params: {} 2022-09-12T19:41:58.151 DEBUG:boto:Token: None 2022-09-12T19:41:58.152 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:41:58.153 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X6q3VPo+d5hOdhYTBLK28sr+SSc= 2022-09-12T19:41:58.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X6q3VPo+d5hOdhYTBLK28sr+SSc=', 'Content-Length': '0'} 2022-09-12T19:41:58.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fc7b9e5e3ddd753-00631f8b86-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.157 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:58.157 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:58.158 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:58.158 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:58.159 DEBUG:boto:Method: GET 2022-09-12T19:41:58.159 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:41:58.160 DEBUG:boto:Data: 2022-09-12T19:41:58.161 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.161 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:58.162 DEBUG:boto:Port: 8000 2022-09-12T19:41:58.162 DEBUG:boto:Params: {} 2022-09-12T19:41:58.163 DEBUG:boto:Token: None 2022-09-12T19:41:58.164 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:41:58.165 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X6q3VPo+d5hOdhYTBLK28sr+SSc= 2022-09-12T19:41:58.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X6q3VPo+d5hOdhYTBLK28sr+SSc=', 'Content-Length': '0'} 2022-09-12T19:41:58.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa7c4a99021dae4b-00631f8b86-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.169 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:58.171 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:58.172 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:58.173 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:58.174 DEBUG:boto:Method: HEAD 2022-09-12T19:41:58.175 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:41:58.176 DEBUG:boto:Data: 2022-09-12T19:41:58.177 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.178 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:58.179 DEBUG:boto:Port: 8001 2022-09-12T19:41:58.180 DEBUG:boto:Params: {} 2022-09-12T19:41:58.181 DEBUG:boto:Token: None 2022-09-12T19:41:58.183 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:41:58.184 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pGQWdSMUII8uTV3mxpJLsEfU0Cc= 2022-09-12T19:41:58.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pGQWdSMUII8uTV3mxpJLsEfU0Cc=', 'Content-Length': '0'} 2022-09-12T19:41:58.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015f8e77938666dcf-00631f8b86-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.189 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:58.190 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:58.191 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:58.192 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:58.193 DEBUG:boto:Method: HEAD 2022-09-12T19:41:58.194 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:41:58.195 DEBUG:boto:Data: 2022-09-12T19:41:58.196 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.197 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:58.198 DEBUG:boto:Port: 8000 2022-09-12T19:41:58.199 DEBUG:boto:Params: {} 2022-09-12T19:41:58.200 DEBUG:boto:Token: None 2022-09-12T19:41:58.201 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:41:58.202 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pGQWdSMUII8uTV3mxpJLsEfU0Cc= 2022-09-12T19:41:58.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pGQWdSMUII8uTV3mxpJLsEfU0Cc=', 'Content-Length': '0'} 2022-09-12T19:41:58.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a450b004c967a2e-00631f8b86-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:41:58.208 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:58.209 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:58.210 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:58.211 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:58.211 DEBUG:boto:Method: GET 2022-09-12T19:41:58.212 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:41:58.212 DEBUG:boto:Data: 2022-09-12T19:41:58.213 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.214 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:58.214 DEBUG:boto:Port: 8001 2022-09-12T19:41:58.215 DEBUG:boto:Params: {} 2022-09-12T19:41:58.215 DEBUG:boto:Token: None 2022-09-12T19:41:58.216 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:41:58.217 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X6q3VPo+d5hOdhYTBLK28sr+SSc= 2022-09-12T19:41:58.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X6q3VPo+d5hOdhYTBLK28sr+SSc=', 'Content-Length': '0'} 2022-09-12T19:41:58.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4a53c11a5840e39-00631f8b86-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.221 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:58.222 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:58.222 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:58.223 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:41:58.223 DEBUG:boto:Method: GET 2022-09-12T19:41:58.224 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:41:58.224 DEBUG:boto:Data: 2022-09-12T19:41:58.225 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.225 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:58.226 DEBUG:boto:Port: 8000 2022-09-12T19:41:58.227 DEBUG:boto:Params: {} 2022-09-12T19:41:58.227 DEBUG:boto:Token: None 2022-09-12T19:41:58.228 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:41:58.229 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X6q3VPo+d5hOdhYTBLK28sr+SSc= 2022-09-12T19:41:58.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X6q3VPo+d5hOdhYTBLK28sr+SSc=', 'Content-Length': '0'} 2022-09-12T19:41:58.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000178f4ee224cb0eca-00631f8b86-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.233 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:58.234 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:58.234 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:41:58.235 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:41:58.235 DEBUG:boto:Method: GET 2022-09-12T19:41:58.236 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:41:58.236 DEBUG:boto:Data: 2022-09-12T19:41:58.237 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.237 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:58.238 DEBUG:boto:Port: 8001 2022-09-12T19:41:58.239 DEBUG:boto:Params: {} 2022-09-12T19:41:58.239 DEBUG:boto:Token: None 2022-09-12T19:41:58.240 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/ 2022-09-12T19:41:58.240 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg= 2022-09-12T19:41:58.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg=', 'Content-Length': '0'} 2022-09-12T19:41:58.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cde5a8048960427c-00631f8b86-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.245 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:58.245 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:58.246 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:58.247 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:41:58.247 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:41:58.248 DEBUG:boto:Method: GET 2022-09-12T19:41:58.249 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:41:58.249 DEBUG:boto:Data: 2022-09-12T19:41:58.250 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.250 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:58.251 DEBUG:boto:Port: 8000 2022-09-12T19:41:58.251 DEBUG:boto:Params: {} 2022-09-12T19:41:58.252 DEBUG:boto:Token: None 2022-09-12T19:41:58.253 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/ 2022-09-12T19:41:58.253 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg= 2022-09-12T19:41:58.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg=', 'Content-Length': '0'} 2022-09-12T19:41:58.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000372939928a6157b9-00631f8b86-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.258 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:58.259 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:58.259 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:58.260 DEBUG:boto:Method: HEAD 2022-09-12T19:41:58.260 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:41:58.261 DEBUG:boto:Data: 2022-09-12T19:41:58.262 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.262 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:58.263 DEBUG:boto:Port: 8001 2022-09-12T19:41:58.263 DEBUG:boto:Params: {} 2022-09-12T19:41:58.264 DEBUG:boto:Token: None 2022-09-12T19:41:58.264 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/b 2022-09-12T19:41:58.265 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BvdU04yYadrOXTdObOaoPKHSeiI= 2022-09-12T19:41:58.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BvdU04yYadrOXTdObOaoPKHSeiI=', 'Content-Length': '0'} 2022-09-12T19:41:58.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dfe6aebfc436549-00631f8b86-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.269 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:58.269 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:58.270 DEBUG:boto:Method: HEAD 2022-09-12T19:41:58.270 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:41:58.271 DEBUG:boto:Data: 2022-09-12T19:41:58.271 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.272 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:58.273 DEBUG:boto:Port: 8000 2022-09-12T19:41:58.273 DEBUG:boto:Params: {} 2022-09-12T19:41:58.274 DEBUG:boto:Token: None 2022-09-12T19:41:58.274 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/b 2022-09-12T19:41:58.275 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BvdU04yYadrOXTdObOaoPKHSeiI= 2022-09-12T19:41:58.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BvdU04yYadrOXTdObOaoPKHSeiI=', 'Content-Length': '0'} 2022-09-12T19:41:58.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f76b979413ae5cb4-00631f8b86-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:41:58.279 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:58.280 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:58.280 DEBUG:boto:Method: GET 2022-09-12T19:41:58.281 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:41:58.281 DEBUG:boto:Data: 2022-09-12T19:41:58.282 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.283 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:58.283 DEBUG:boto:Port: 8001 2022-09-12T19:41:58.284 DEBUG:boto:Params: {} 2022-09-12T19:41:58.285 DEBUG:boto:Token: None 2022-09-12T19:41:58.286 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/b 2022-09-12T19:41:58.287 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m1ZwDOu6RSQ9pDdf4+u5Jp26y60= 2022-09-12T19:41:58.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m1ZwDOu6RSQ9pDdf4+u5Jp26y60=', 'Content-Length': '0'} 2022-09-12T19:41:58.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000008878667d0f41ad-00631f8b86-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.290 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:41:58.291 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:41:58.292 DEBUG:boto:Method: GET 2022-09-12T19:41:58.292 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:41:58.293 DEBUG:boto:Data: 2022-09-12T19:41:58.293 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.294 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:58.294 DEBUG:boto:Port: 8000 2022-09-12T19:41:58.295 DEBUG:boto:Params: {} 2022-09-12T19:41:58.296 DEBUG:boto:Token: None 2022-09-12T19:41:58.296 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/b 2022-09-12T19:41:58.297 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m1ZwDOu6RSQ9pDdf4+u5Jp26y60= 2022-09-12T19:41:58.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m1ZwDOu6RSQ9pDdf4+u5Jp26y60=', 'Content-Length': '0'} 2022-09-12T19:41:58.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000491cb4f152e4f240-00631f8b86-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.301 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:58.302 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:58.302 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:41:58.303 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:41:58.303 DEBUG:boto:Method: GET 2022-09-12T19:41:58.304 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:41:58.305 DEBUG:boto:Data: 2022-09-12T19:41:58.305 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.306 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:58.306 DEBUG:boto:Port: 8001 2022-09-12T19:41:58.307 DEBUG:boto:Params: {} 2022-09-12T19:41:58.308 DEBUG:boto:Token: None 2022-09-12T19:41:58.308 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/ 2022-09-12T19:41:58.309 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg= 2022-09-12T19:41:58.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg=', 'Content-Length': '0'} 2022-09-12T19:41:58.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000963d6f6dddc9e572-00631f8b86-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.315 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:58.316 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:58.317 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:58.318 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:41:58.319 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:41:58.320 DEBUG:boto:Method: GET 2022-09-12T19:41:58.321 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:41:58.322 DEBUG:boto:Data: 2022-09-12T19:41:58.322 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.323 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:58.324 DEBUG:boto:Port: 8000 2022-09-12T19:41:58.326 DEBUG:boto:Params: {} 2022-09-12T19:41:58.327 DEBUG:boto:Token: None 2022-09-12T19:41:58.328 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/ 2022-09-12T19:41:58.328 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg= 2022-09-12T19:41:58.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg=', 'Content-Length': '0'} 2022-09-12T19:41:58.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006210f63386e53238-00631f8b86-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.333 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:58.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:41:58.335 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:58.336 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:58.336 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:58.337 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:58.338 DEBUG:boto:Method: GET 2022-09-12T19:41:58.338 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:41:58.339 DEBUG:boto:Data: 2022-09-12T19:41:58.340 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.340 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:58.341 DEBUG:boto:Port: 8001 2022-09-12T19:41:58.342 DEBUG:boto:Params: {} 2022-09-12T19:41:58.343 DEBUG:boto:Token: None 2022-09-12T19:41:58.344 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:41:58.345 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JbSOz0jCzd+wMeg31yLIh4t07S0= 2022-09-12T19:41:58.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JbSOz0jCzd+wMeg31yLIh4t07S0=', 'Content-Length': '0'} 2022-09-12T19:41:58.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c43dec886a0b6c0-00631f8b86-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.350 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:58.351 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:58.351 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:58.352 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:58.353 DEBUG:boto:Method: GET 2022-09-12T19:41:58.353 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:41:58.354 DEBUG:boto:Data: 2022-09-12T19:41:58.355 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.355 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:58.357 DEBUG:boto:Port: 8000 2022-09-12T19:41:58.357 DEBUG:boto:Params: {} 2022-09-12T19:41:58.358 DEBUG:boto:Token: None 2022-09-12T19:41:58.359 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:41:58.359 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JbSOz0jCzd+wMeg31yLIh4t07S0= 2022-09-12T19:41:58.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JbSOz0jCzd+wMeg31yLIh4t07S0=', 'Content-Length': '0'} 2022-09-12T19:41:58.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000b43782ec801597-00631f8b86-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.364 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:58.364 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:58.365 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:58.366 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:58.367 DEBUG:boto:Method: HEAD 2022-09-12T19:41:58.368 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:41:58.368 DEBUG:boto:Data: 2022-09-12T19:41:58.369 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.370 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:58.370 DEBUG:boto:Port: 8001 2022-09-12T19:41:58.371 DEBUG:boto:Params: {} 2022-09-12T19:41:58.372 DEBUG:boto:Token: None 2022-09-12T19:41:58.372 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:41:58.374 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wPLuJeACcMGku5OVxhEwuq8szDM= 2022-09-12T19:41:58.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wPLuJeACcMGku5OVxhEwuq8szDM=', 'Content-Length': '0'} 2022-09-12T19:41:58.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060d66ab93699300c-00631f8b86-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.380 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:58.381 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:58.382 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:58.383 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:58.384 DEBUG:boto:Method: HEAD 2022-09-12T19:41:58.385 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:41:58.386 DEBUG:boto:Data: 2022-09-12T19:41:58.387 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.387 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:58.389 DEBUG:boto:Port: 8000 2022-09-12T19:41:58.390 DEBUG:boto:Params: {} 2022-09-12T19:41:58.391 DEBUG:boto:Token: None 2022-09-12T19:41:58.391 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:41:58.392 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wPLuJeACcMGku5OVxhEwuq8szDM= 2022-09-12T19:41:58.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wPLuJeACcMGku5OVxhEwuq8szDM=', 'Content-Length': '0'} 2022-09-12T19:41:58.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa54ed32b1f02400-00631f8b86-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:41:58.396 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:58.397 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:58.397 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:58.398 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:58.398 DEBUG:boto:Method: GET 2022-09-12T19:41:58.399 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:41:58.400 DEBUG:boto:Data: 2022-09-12T19:41:58.400 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.401 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:58.401 DEBUG:boto:Port: 8001 2022-09-12T19:41:58.402 DEBUG:boto:Params: {} 2022-09-12T19:41:58.403 DEBUG:boto:Token: None 2022-09-12T19:41:58.403 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:41:58.404 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JbSOz0jCzd+wMeg31yLIh4t07S0= 2022-09-12T19:41:58.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JbSOz0jCzd+wMeg31yLIh4t07S0=', 'Content-Length': '0'} 2022-09-12T19:41:58.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdbc72b4c72f9db9-00631f8b86-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.407 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:58.408 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:58.409 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:58.409 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:41:58.410 DEBUG:boto:Method: GET 2022-09-12T19:41:58.410 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:41:58.411 DEBUG:boto:Data: 2022-09-12T19:41:58.411 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.412 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:58.412 DEBUG:boto:Port: 8000 2022-09-12T19:41:58.413 DEBUG:boto:Params: {} 2022-09-12T19:41:58.414 DEBUG:boto:Token: None 2022-09-12T19:41:58.414 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:41:58.415 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JbSOz0jCzd+wMeg31yLIh4t07S0= 2022-09-12T19:41:58.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JbSOz0jCzd+wMeg31yLIh4t07S0=', 'Content-Length': '0'} 2022-09-12T19:41:58.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007caac7586d61b133-00631f8b86-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.419 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:58.420 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:58.421 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:41:58.421 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:41:58.422 DEBUG:boto:Method: GET 2022-09-12T19:41:58.422 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:41:58.423 DEBUG:boto:Data: 2022-09-12T19:41:58.423 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.424 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:58.425 DEBUG:boto:Port: 8001 2022-09-12T19:41:58.425 DEBUG:boto:Params: {} 2022-09-12T19:41:58.426 DEBUG:boto:Token: None 2022-09-12T19:41:58.426 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/ 2022-09-12T19:41:58.427 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg= 2022-09-12T19:41:58.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg=', 'Content-Length': '0'} 2022-09-12T19:41:58.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fd9a630b4f58fe5-00631f8b86-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.431 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:58.432 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:58.432 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:58.433 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:41:58.433 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:41:58.434 DEBUG:boto:Method: GET 2022-09-12T19:41:58.435 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:41:58.435 DEBUG:boto:Data: 2022-09-12T19:41:58.436 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.436 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:58.437 DEBUG:boto:Port: 8000 2022-09-12T19:41:58.437 DEBUG:boto:Params: {} 2022-09-12T19:41:58.438 DEBUG:boto:Token: None 2022-09-12T19:41:58.439 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/ 2022-09-12T19:41:58.439 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg= 2022-09-12T19:41:58.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg=', 'Content-Length': '0'} 2022-09-12T19:41:58.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092043753dd8a5b99-00631f8b86-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.443 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:58.444 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:58.444 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:58.445 DEBUG:boto:Method: HEAD 2022-09-12T19:41:58.446 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:41:58.446 DEBUG:boto:Data: 2022-09-12T19:41:58.447 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.447 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:58.448 DEBUG:boto:Port: 8001 2022-09-12T19:41:58.449 DEBUG:boto:Params: {} 2022-09-12T19:41:58.449 DEBUG:boto:Token: None 2022-09-12T19:41:58.450 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/c 2022-09-12T19:41:58.450 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bNAvh2RqdhQBTJ/sc/HY5FJszuA= 2022-09-12T19:41:58.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bNAvh2RqdhQBTJ/sc/HY5FJszuA=', 'Content-Length': '0'} 2022-09-12T19:41:58.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065cf908ba4f8db7f-00631f8b86-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.454 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:58.454 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:58.455 DEBUG:boto:Method: HEAD 2022-09-12T19:41:58.455 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:41:58.456 DEBUG:boto:Data: 2022-09-12T19:41:58.456 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.457 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:58.457 DEBUG:boto:Port: 8000 2022-09-12T19:41:58.458 DEBUG:boto:Params: {} 2022-09-12T19:41:58.459 DEBUG:boto:Token: None 2022-09-12T19:41:58.459 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/c 2022-09-12T19:41:58.460 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bNAvh2RqdhQBTJ/sc/HY5FJszuA= 2022-09-12T19:41:58.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bNAvh2RqdhQBTJ/sc/HY5FJszuA=', 'Content-Length': '0'} 2022-09-12T19:41:58.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3af5da10d9e2f38-00631f8b86-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:41:58.464 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:58.464 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:58.465 DEBUG:boto:Method: GET 2022-09-12T19:41:58.465 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:41:58.466 DEBUG:boto:Data: 2022-09-12T19:41:58.467 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.467 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:58.468 DEBUG:boto:Port: 8001 2022-09-12T19:41:58.468 DEBUG:boto:Params: {} 2022-09-12T19:41:58.469 DEBUG:boto:Token: None 2022-09-12T19:41:58.469 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/c 2022-09-12T19:41:58.470 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uQt1XKAekmskVZspltnOgIGFe/w= 2022-09-12T19:41:58.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uQt1XKAekmskVZspltnOgIGFe/w=', 'Content-Length': '0'} 2022-09-12T19:41:58.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024f1211411a174f8-00631f8b86-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.474 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:41:58.474 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:41:58.475 DEBUG:boto:Method: GET 2022-09-12T19:41:58.475 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:41:58.476 DEBUG:boto:Data: 2022-09-12T19:41:58.477 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.477 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:58.478 DEBUG:boto:Port: 8000 2022-09-12T19:41:58.478 DEBUG:boto:Params: {} 2022-09-12T19:41:58.479 DEBUG:boto:Token: None 2022-09-12T19:41:58.479 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/c 2022-09-12T19:41:58.480 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uQt1XKAekmskVZspltnOgIGFe/w= 2022-09-12T19:41:58.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uQt1XKAekmskVZspltnOgIGFe/w=', 'Content-Length': '0'} 2022-09-12T19:41:58.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c40761a271211ca-00631f8b86-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.530 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:58.531 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:58.532 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:41:58.532 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:41:58.533 DEBUG:boto:Method: GET 2022-09-12T19:41:58.533 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:41:58.534 DEBUG:boto:Data: 2022-09-12T19:41:58.535 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.535 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:58.536 DEBUG:boto:Port: 8001 2022-09-12T19:41:58.536 DEBUG:boto:Params: {} 2022-09-12T19:41:58.537 DEBUG:boto:Token: None 2022-09-12T19:41:58.538 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/ 2022-09-12T19:41:58.538 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg= 2022-09-12T19:41:58.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg=', 'Content-Length': '0'} 2022-09-12T19:41:58.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061c98140e1650ab0-00631f8b86-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.554 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:58.555 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:58.555 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:58.556 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:41:58.557 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:41:58.557 DEBUG:boto:Method: GET 2022-09-12T19:41:58.558 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:41:58.558 DEBUG:boto:Data: 2022-09-12T19:41:58.559 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.560 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:58.560 DEBUG:boto:Port: 8000 2022-09-12T19:41:58.561 DEBUG:boto:Params: {} 2022-09-12T19:41:58.561 DEBUG:boto:Token: None 2022-09-12T19:41:58.562 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/ 2022-09-12T19:41:58.563 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg= 2022-09-12T19:41:58.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg=', 'Content-Length': '0'} 2022-09-12T19:41:58.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ac042dafdcab793-00631f8b86-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.583 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:58.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:41:58.584 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:58.585 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:58.585 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:58.586 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:58.587 DEBUG:boto:Method: GET 2022-09-12T19:41:58.588 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:41:58.588 DEBUG:boto:Data: 2022-09-12T19:41:58.589 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.589 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:58.599 DEBUG:boto:Port: 8001 2022-09-12T19:41:58.599 DEBUG:boto:Params: {} 2022-09-12T19:41:58.600 DEBUG:boto:Token: None 2022-09-12T19:41:58.601 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:41:58.601 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qvBNL5RDNt8mSvnMIPl38tu/aio= 2022-09-12T19:41:58.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qvBNL5RDNt8mSvnMIPl38tu/aio=', 'Content-Length': '0'} 2022-09-12T19:41:58.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adcf996cf987dffb-00631f8b86-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.607 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:58.608 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:58.608 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:58.609 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:58.609 DEBUG:boto:Method: GET 2022-09-12T19:41:58.610 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:41:58.611 DEBUG:boto:Data: 2022-09-12T19:41:58.611 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.612 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:58.612 DEBUG:boto:Port: 8000 2022-09-12T19:41:58.613 DEBUG:boto:Params: {} 2022-09-12T19:41:58.614 DEBUG:boto:Token: None 2022-09-12T19:41:58.615 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:41:58.615 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qvBNL5RDNt8mSvnMIPl38tu/aio= 2022-09-12T19:41:58.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qvBNL5RDNt8mSvnMIPl38tu/aio=', 'Content-Length': '0'} 2022-09-12T19:41:58.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a94abf90afa50c1b-00631f8b86-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.619 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:58.620 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:58.621 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:58.621 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:58.622 DEBUG:boto:Method: HEAD 2022-09-12T19:41:58.623 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:41:58.623 DEBUG:boto:Data: 2022-09-12T19:41:58.624 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.625 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:58.625 DEBUG:boto:Port: 8001 2022-09-12T19:41:58.626 DEBUG:boto:Params: {} 2022-09-12T19:41:58.626 DEBUG:boto:Token: None 2022-09-12T19:41:58.627 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:41:58.629 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3PGVcCoiOrxQYHfmafnOK8dCUsY= 2022-09-12T19:41:58.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3PGVcCoiOrxQYHfmafnOK8dCUsY=', 'Content-Length': '0'} 2022-09-12T19:41:58.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb743928a8db4e52-00631f8b86-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.633 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:58.634 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:58.634 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:58.635 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:58.635 DEBUG:boto:Method: HEAD 2022-09-12T19:41:58.636 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:41:58.636 DEBUG:boto:Data: 2022-09-12T19:41:58.637 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.638 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:58.639 DEBUG:boto:Port: 8000 2022-09-12T19:41:58.639 DEBUG:boto:Params: {} 2022-09-12T19:41:58.640 DEBUG:boto:Token: None 2022-09-12T19:41:58.641 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:41:58.641 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3PGVcCoiOrxQYHfmafnOK8dCUsY= 2022-09-12T19:41:58.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3PGVcCoiOrxQYHfmafnOK8dCUsY=', 'Content-Length': '0'} 2022-09-12T19:41:58.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9882ccb504ce76a-00631f8b86-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:41:58.648 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:58.648 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:58.649 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:58.649 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:58.650 DEBUG:boto:Method: GET 2022-09-12T19:41:58.651 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:41:58.651 DEBUG:boto:Data: 2022-09-12T19:41:58.652 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.652 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:58.653 DEBUG:boto:Port: 8001 2022-09-12T19:41:58.654 DEBUG:boto:Params: {} 2022-09-12T19:41:58.656 DEBUG:boto:Token: None 2022-09-12T19:41:58.657 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:41:58.657 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qvBNL5RDNt8mSvnMIPl38tu/aio= 2022-09-12T19:41:58.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qvBNL5RDNt8mSvnMIPl38tu/aio=', 'Content-Length': '0'} 2022-09-12T19:41:58.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dab96722996229f6-00631f8b86-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.702 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:58.703 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:58.704 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:58.706 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:41:58.707 DEBUG:boto:Method: GET 2022-09-12T19:41:58.708 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:41:58.709 DEBUG:boto:Data: 2022-09-12T19:41:58.710 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.711 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:58.713 DEBUG:boto:Port: 8000 2022-09-12T19:41:58.714 DEBUG:boto:Params: {} 2022-09-12T19:41:58.715 DEBUG:boto:Token: None 2022-09-12T19:41:58.716 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:41:58.717 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qvBNL5RDNt8mSvnMIPl38tu/aio= 2022-09-12T19:41:58.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qvBNL5RDNt8mSvnMIPl38tu/aio=', 'Content-Length': '0'} 2022-09-12T19:41:58.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016a582e13dd19290-00631f8b86-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.723 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:58.724 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:58.725 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:41:58.726 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:41:58.727 DEBUG:boto:Method: GET 2022-09-12T19:41:58.728 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:41:58.729 DEBUG:boto:Data: 2022-09-12T19:41:58.730 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.731 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:58.732 DEBUG:boto:Port: 8001 2022-09-12T19:41:58.733 DEBUG:boto:Params: {} 2022-09-12T19:41:58.734 DEBUG:boto:Token: None 2022-09-12T19:41:58.734 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/ 2022-09-12T19:41:58.735 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg= 2022-09-12T19:41:58.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg=', 'Content-Length': '0'} 2022-09-12T19:41:58.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000518ab7dc0e3013d4-00631f8b86-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.741 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:58.743 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:58.744 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:58.745 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:41:58.746 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:41:58.748 DEBUG:boto:Method: GET 2022-09-12T19:41:58.749 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:41:58.750 DEBUG:boto:Data: 2022-09-12T19:41:58.751 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.752 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:58.753 DEBUG:boto:Port: 8000 2022-09-12T19:41:58.754 DEBUG:boto:Params: {} 2022-09-12T19:41:58.755 DEBUG:boto:Token: None 2022-09-12T19:41:58.757 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/ 2022-09-12T19:41:58.758 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg= 2022-09-12T19:41:58.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg=', 'Content-Length': '0'} 2022-09-12T19:41:58.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5936569298376e9-00631f8b86-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.771 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:58.772 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:58.772 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:58.773 DEBUG:boto:Method: HEAD 2022-09-12T19:41:58.774 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:41:58.774 DEBUG:boto:Data: 2022-09-12T19:41:58.775 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.775 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:58.776 DEBUG:boto:Port: 8001 2022-09-12T19:41:58.776 DEBUG:boto:Params: {} 2022-09-12T19:41:58.777 DEBUG:boto:Token: None 2022-09-12T19:41:58.778 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/d 2022-09-12T19:41:58.787 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4JEc/26tIpdr9qFbXSnkimoKKM0= 2022-09-12T19:41:58.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4JEc/26tIpdr9qFbXSnkimoKKM0=', 'Content-Length': '0'} 2022-09-12T19:41:58.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a13db9f7320d979-00631f8b86-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.798 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:58.799 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:58.800 DEBUG:boto:Method: HEAD 2022-09-12T19:41:58.800 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:41:58.801 DEBUG:boto:Data: 2022-09-12T19:41:58.802 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.802 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:58.803 DEBUG:boto:Port: 8000 2022-09-12T19:41:58.803 DEBUG:boto:Params: {} 2022-09-12T19:41:58.804 DEBUG:boto:Token: None 2022-09-12T19:41:58.805 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/d 2022-09-12T19:41:58.805 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4JEc/26tIpdr9qFbXSnkimoKKM0= 2022-09-12T19:41:58.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4JEc/26tIpdr9qFbXSnkimoKKM0=', 'Content-Length': '0'} 2022-09-12T19:41:58.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e264598f4ad9cab0-00631f8b86-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:41:58.810 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:58.810 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:58.811 DEBUG:boto:Method: GET 2022-09-12T19:41:58.812 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:41:58.812 DEBUG:boto:Data: 2022-09-12T19:41:58.813 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.814 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:58.815 DEBUG:boto:Port: 8001 2022-09-12T19:41:58.816 DEBUG:boto:Params: {} 2022-09-12T19:41:58.817 DEBUG:boto:Token: None 2022-09-12T19:41:58.818 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/d 2022-09-12T19:41:58.819 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8ZeBaay36zohDFVxHIQ2HVvQ5rI= 2022-09-12T19:41:58.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8ZeBaay36zohDFVxHIQ2HVvQ5rI=', 'Content-Length': '0'} 2022-09-12T19:41:58.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c4a7738c88a06bd-00631f8b86-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.825 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:41:58.830 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:41:58.831 DEBUG:boto:Method: GET 2022-09-12T19:41:58.831 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:41:58.832 DEBUG:boto:Data: 2022-09-12T19:41:58.832 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.833 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:58.834 DEBUG:boto:Port: 8000 2022-09-12T19:41:58.834 DEBUG:boto:Params: {} 2022-09-12T19:41:58.835 DEBUG:boto:Token: None 2022-09-12T19:41:58.836 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/d 2022-09-12T19:41:58.836 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8ZeBaay36zohDFVxHIQ2HVvQ5rI= 2022-09-12T19:41:58.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8ZeBaay36zohDFVxHIQ2HVvQ5rI=', 'Content-Length': '0'} 2022-09-12T19:41:58.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033864a5b9cdf0918-00631f8b86-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.886 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:58.887 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:58.888 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:41:58.888 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:41:58.889 DEBUG:boto:Method: GET 2022-09-12T19:41:58.890 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:41:58.891 DEBUG:boto:Data: 2022-09-12T19:41:58.891 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.892 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:41:58.892 DEBUG:boto:Port: 8001 2022-09-12T19:41:58.893 DEBUG:boto:Params: {} 2022-09-12T19:41:58.894 DEBUG:boto:Token: None 2022-09-12T19:41:58.894 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/ 2022-09-12T19:41:58.895 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg= 2022-09-12T19:41:58.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg=', 'Content-Length': '0'} 2022-09-12T19:41:58.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f3ccbf703f3b245-00631f8b86-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.910 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:58.911 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:58.912 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:58.912 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:41:58.913 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:41:58.915 DEBUG:boto:Method: GET 2022-09-12T19:41:58.916 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:41:58.917 DEBUG:boto:Data: 2022-09-12T19:41:58.918 DEBUG:boto:Headers: {} 2022-09-12T19:41:58.919 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:41:58.920 DEBUG:boto:Port: 8000 2022-09-12T19:41:58.921 DEBUG:boto:Params: {} 2022-09-12T19:41:58.922 DEBUG:boto:Token: None 2022-09-12T19:41:58.923 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:41:58 GMT /jdpiak-71/ 2022-09-12T19:41:58.925 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg= 2022-09-12T19:41:58.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q3LpkIlLQZ8BbtpN+vUBVx/Zxdg=', 'Content-Length': '0'} 2022-09-12T19:41:58.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000946d052f312d3205-00631f8b86-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:41:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:58.931 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:41:58.932 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-71 zones={test-zone3, test-zone2} 2022-09-12T19:41:58.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'jdpiak-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:41:58.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'jdpiak-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:41:58.936 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket jdpiak-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:41:58.997 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:41:58.997 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:41:59.046 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:41:59.047 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:41:59.047 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding": "None", 2022-09-12T19:41:59.047 INFO:teuthology.orchestra.run.smithi035.stdout: "current_index": { 2022-09-12T19:41:59.047 INFO:teuthology.orchestra.run.smithi035.stdout: "gen": 0, 2022-09-12T19:41:59.048 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:41:59.048 INFO:teuthology.orchestra.run.smithi035.stdout: "type": "Normal", 2022-09-12T19:41:59.049 INFO:teuthology.orchestra.run.smithi035.stdout: "normal": { 2022-09-12T19:41:59.049 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 11, 2022-09-12T19:41:59.049 INFO:teuthology.orchestra.run.smithi035.stdout: "hash_type": "Mod" 2022-09-12T19:41:59.049 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:59.050 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:59.050 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:59.050 INFO:teuthology.orchestra.run.smithi035.stdout: "logs": [ 2022-09-12T19:41:59.051 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:59.051 INFO:teuthology.orchestra.run.smithi035.stdout: "gen": 0, 2022-09-12T19:41:59.051 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:41:59.053 INFO:teuthology.orchestra.run.smithi035.stdout: "type": "InIndex", 2022-09-12T19:41:59.053 INFO:teuthology.orchestra.run.smithi035.stdout: "in_index": { 2022-09-12T19:41:59.053 INFO:teuthology.orchestra.run.smithi035.stdout: "gen": 0, 2022-09-12T19:41:59.053 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:41:59.054 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 11, 2022-09-12T19:41:59.054 INFO:teuthology.orchestra.run.smithi035.stdout: "hash_type": "Mod" 2022-09-12T19:41:59.054 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:59.055 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:59.055 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:59.055 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:59.055 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:41:59.056 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:59.056 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:41:59.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'jdpiak-71', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:41:59.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'jdpiak-71', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:41:59.062 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket jdpiak-71 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:41:59.117 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:41:59.117 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:41:59.166 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:41:59.167 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:59.167 INFO:teuthology.orchestra.run.smithi035.stdout: "op_id": "1#00000000001.52.5", 2022-09-12T19:41:59.168 INFO:teuthology.orchestra.run.smithi035.stdout: "op_tag": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5568.4269214077306935752", 2022-09-12T19:41:59.169 INFO:teuthology.orchestra.run.smithi035.stdout: "op": "write", 2022-09-12T19:41:59.169 INFO:teuthology.orchestra.run.smithi035.stdout: "object": "a", 2022-09-12T19:41:59.169 INFO:teuthology.orchestra.run.smithi035.stdout: "instance": "", 2022-09-12T19:41:59.170 INFO:teuthology.orchestra.run.smithi035.stdout: "state": "complete", 2022-09-12T19:41:59.171 INFO:teuthology.orchestra.run.smithi035.stdout: "index_ver": 1, 2022-09-12T19:41:59.171 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:41:22.865493560Z", 2022-09-12T19:41:59.172 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": { 2022-09-12T19:41:59.172 INFO:teuthology.orchestra.run.smithi035.stdout: "pool": 9, 2022-09-12T19:41:59.173 INFO:teuthology.orchestra.run.smithi035.stdout: "epoch": 10 2022-09-12T19:41:59.173 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:59.174 INFO:teuthology.orchestra.run.smithi035.stdout: "bilog_flags": 0, 2022-09-12T19:41:59.175 INFO:teuthology.orchestra.run.smithi035.stdout: "versioned": false, 2022-09-12T19:41:59.175 INFO:teuthology.orchestra.run.smithi035.stdout: "owner": "", 2022-09-12T19:41:59.176 INFO:teuthology.orchestra.run.smithi035.stdout: "owner_display_name": "", 2022-09-12T19:41:59.176 INFO:teuthology.orchestra.run.smithi035.stdout: "zones_trace": [ 2022-09-12T19:41:59.176 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:59.177 INFO:teuthology.orchestra.run.smithi035.stdout: "entry": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-71:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.22" 2022-09-12T19:41:59.177 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:59.180 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:41:59.180 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:59.181 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:59.181 INFO:teuthology.orchestra.run.smithi035.stdout: "op_id": "5#00000000001.89.5", 2022-09-12T19:41:59.182 INFO:teuthology.orchestra.run.smithi035.stdout: "op_tag": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5568.13739185240475292506", 2022-09-12T19:41:59.182 INFO:teuthology.orchestra.run.smithi035.stdout: "op": "write", 2022-09-12T19:41:59.182 INFO:teuthology.orchestra.run.smithi035.stdout: "object": "d", 2022-09-12T19:41:59.183 INFO:teuthology.orchestra.run.smithi035.stdout: "instance": "", 2022-09-12T19:41:59.183 INFO:teuthology.orchestra.run.smithi035.stdout: "state": "complete", 2022-09-12T19:41:59.184 INFO:teuthology.orchestra.run.smithi035.stdout: "index_ver": 1, 2022-09-12T19:41:59.184 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:41:23.018165401Z", 2022-09-12T19:41:59.184 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": { 2022-09-12T19:41:59.185 INFO:teuthology.orchestra.run.smithi035.stdout: "pool": 9, 2022-09-12T19:41:59.185 INFO:teuthology.orchestra.run.smithi035.stdout: "epoch": 3 2022-09-12T19:41:59.186 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:59.187 INFO:teuthology.orchestra.run.smithi035.stdout: "bilog_flags": 0, 2022-09-12T19:41:59.188 INFO:teuthology.orchestra.run.smithi035.stdout: "versioned": false, 2022-09-12T19:41:59.188 INFO:teuthology.orchestra.run.smithi035.stdout: "owner": "", 2022-09-12T19:41:59.188 INFO:teuthology.orchestra.run.smithi035.stdout: "owner_display_name": "", 2022-09-12T19:41:59.189 INFO:teuthology.orchestra.run.smithi035.stdout: "zones_trace": [ 2022-09-12T19:41:59.193 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:59.194 INFO:teuthology.orchestra.run.smithi035.stdout: "entry": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-71:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.22" 2022-09-12T19:41:59.194 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:59.195 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:41:59.195 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:59.195 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:59.196 INFO:teuthology.orchestra.run.smithi035.stdout: "op_id": "8#00000000001.60.5", 2022-09-12T19:41:59.196 INFO:teuthology.orchestra.run.smithi035.stdout: "op_tag": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5568.5785548734652388140", 2022-09-12T19:41:59.196 INFO:teuthology.orchestra.run.smithi035.stdout: "op": "write", 2022-09-12T19:41:59.198 INFO:teuthology.orchestra.run.smithi035.stdout: "object": "c", 2022-09-12T19:41:59.199 INFO:teuthology.orchestra.run.smithi035.stdout: "instance": "", 2022-09-12T19:41:59.199 INFO:teuthology.orchestra.run.smithi035.stdout: "state": "complete", 2022-09-12T19:41:59.199 INFO:teuthology.orchestra.run.smithi035.stdout: "index_ver": 1, 2022-09-12T19:41:59.200 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:41:22.975964102Z", 2022-09-12T19:41:59.200 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": { 2022-09-12T19:41:59.200 INFO:teuthology.orchestra.run.smithi035.stdout: "pool": 9, 2022-09-12T19:41:59.200 INFO:teuthology.orchestra.run.smithi035.stdout: "epoch": 5 2022-09-12T19:41:59.201 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:59.201 INFO:teuthology.orchestra.run.smithi035.stdout: "bilog_flags": 0, 2022-09-12T19:41:59.201 INFO:teuthology.orchestra.run.smithi035.stdout: "versioned": false, 2022-09-12T19:41:59.202 INFO:teuthology.orchestra.run.smithi035.stdout: "owner": "", 2022-09-12T19:41:59.204 INFO:teuthology.orchestra.run.smithi035.stdout: "owner_display_name": "", 2022-09-12T19:41:59.204 INFO:teuthology.orchestra.run.smithi035.stdout: "zones_trace": [ 2022-09-12T19:41:59.204 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:59.205 INFO:teuthology.orchestra.run.smithi035.stdout: "entry": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-71:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.22" 2022-09-12T19:41:59.205 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:59.205 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:41:59.205 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:59.206 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:59.206 INFO:teuthology.orchestra.run.smithi035.stdout: "op_id": "10#00000000001.41.5", 2022-09-12T19:41:59.206 INFO:teuthology.orchestra.run.smithi035.stdout: "op_tag": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5568.6895054573177727720", 2022-09-12T19:41:59.207 INFO:teuthology.orchestra.run.smithi035.stdout: "op": "write", 2022-09-12T19:41:59.207 INFO:teuthology.orchestra.run.smithi035.stdout: "object": "b", 2022-09-12T19:41:59.208 INFO:teuthology.orchestra.run.smithi035.stdout: "instance": "", 2022-09-12T19:41:59.210 INFO:teuthology.orchestra.run.smithi035.stdout: "state": "complete", 2022-09-12T19:41:59.210 INFO:teuthology.orchestra.run.smithi035.stdout: "index_ver": 1, 2022-09-12T19:41:59.211 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:41:22.899433728Z", 2022-09-12T19:41:59.212 INFO:teuthology.orchestra.run.smithi035.stdout: "ver": { 2022-09-12T19:41:59.212 INFO:teuthology.orchestra.run.smithi035.stdout: "pool": 9, 2022-09-12T19:41:59.214 INFO:teuthology.orchestra.run.smithi035.stdout: "epoch": 14 2022-09-12T19:41:59.214 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:59.215 INFO:teuthology.orchestra.run.smithi035.stdout: "bilog_flags": 0, 2022-09-12T19:41:59.216 INFO:teuthology.orchestra.run.smithi035.stdout: "versioned": false, 2022-09-12T19:41:59.216 INFO:teuthology.orchestra.run.smithi035.stdout: "owner": "", 2022-09-12T19:41:59.217 INFO:teuthology.orchestra.run.smithi035.stdout: "owner_display_name": "", 2022-09-12T19:41:59.217 INFO:teuthology.orchestra.run.smithi035.stdout: "zones_trace": [ 2022-09-12T19:41:59.228 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:59.228 INFO:teuthology.orchestra.run.smithi035.stdout: "entry": "fb29b4fb-1f78-4524-ab80-6a5e9e01ab47:jdpiak-71:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.22" 2022-09-12T19:41:59.229 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:59.230 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:41:59.230 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:59.231 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:41:59.233 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'jdpiak-71', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:41:59.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'jdpiak-71', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:41:59.236 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket jdpiak-71 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:41:59.288 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:41:59.289 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:41:59.352 INFO:teuthology.orchestra.run.smithi035.stdout:tenant: 2022-09-12T19:41:59.353 INFO:teuthology.orchestra.run.smithi035.stdout:bucket name: jdpiak-71 2022-09-12T19:41:59.358 INFO:teuthology.orchestra.run.smithi035.stdout:total entries: 4 2022-09-12T19:41:59.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'jdpiak-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:41:59.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'jdpiak-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:41:59.427 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket jdpiak-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:41:59.477 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:41:59.478 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:41:59.527 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:41:59.528 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:41:59.528 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding": "None", 2022-09-12T19:41:59.528 INFO:teuthology.orchestra.run.smithi035.stdout: "current_index": { 2022-09-12T19:41:59.528 INFO:teuthology.orchestra.run.smithi035.stdout: "gen": 1, 2022-09-12T19:41:59.529 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:41:59.529 INFO:teuthology.orchestra.run.smithi035.stdout: "type": "Normal", 2022-09-12T19:41:59.529 INFO:teuthology.orchestra.run.smithi035.stdout: "normal": { 2022-09-12T19:41:59.531 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 3, 2022-09-12T19:41:59.531 INFO:teuthology.orchestra.run.smithi035.stdout: "hash_type": "Mod" 2022-09-12T19:41:59.534 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:59.534 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:59.541 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:59.542 INFO:teuthology.orchestra.run.smithi035.stdout: "logs": [ 2022-09-12T19:41:59.542 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:59.543 INFO:teuthology.orchestra.run.smithi035.stdout: "gen": 0, 2022-09-12T19:41:59.546 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:41:59.546 INFO:teuthology.orchestra.run.smithi035.stdout: "type": "InIndex", 2022-09-12T19:41:59.546 INFO:teuthology.orchestra.run.smithi035.stdout: "in_index": { 2022-09-12T19:41:59.547 INFO:teuthology.orchestra.run.smithi035.stdout: "gen": 0, 2022-09-12T19:41:59.547 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:41:59.547 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 11, 2022-09-12T19:41:59.548 INFO:teuthology.orchestra.run.smithi035.stdout: "hash_type": "Mod" 2022-09-12T19:41:59.548 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:59.548 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:59.549 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:59.549 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:41:59.549 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:41:59.549 INFO:teuthology.orchestra.run.smithi035.stdout: "gen": 1, 2022-09-12T19:41:59.550 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:41:59.550 INFO:teuthology.orchestra.run.smithi035.stdout: "type": "InIndex", 2022-09-12T19:41:59.550 INFO:teuthology.orchestra.run.smithi035.stdout: "in_index": { 2022-09-12T19:41:59.551 INFO:teuthology.orchestra.run.smithi035.stdout: "gen": 1, 2022-09-12T19:41:59.551 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:41:59.551 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 3, 2022-09-12T19:41:59.552 INFO:teuthology.orchestra.run.smithi035.stdout: "hash_type": "Mod" 2022-09-12T19:41:59.552 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:59.552 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:59.553 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:59.553 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:59.554 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:41:59.555 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:41:59.555 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:41:59.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'jdpiak-71', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:41:59.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'jdpiak-71', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:41:59.558 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket jdpiak-71 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:41:59.610 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:41:59.610 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:41:59.678 INFO:teuthology.orchestra.run.smithi035.stdout:[] 2022-09-12T19:41:59.680 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:59.681 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:59.694 DEBUG:boto:Method: HEAD 2022-09-12T19:41:59.695 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:41:59.696 DEBUG:boto:Data: 2022-09-12T19:41:59.697 DEBUG:boto:Headers: {} 2022-09-12T19:41:59.700 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:59.701 DEBUG:boto:Port: 8000 2022-09-12T19:41:59.702 DEBUG:boto:Params: {} 2022-09-12T19:41:59.703 DEBUG:boto:Token: None 2022-09-12T19:41:59.704 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:59 GMT /jdpiak-71/ 2022-09-12T19:41:59.705 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eHUWDhsAA04hyMkz5u84Rhks8nQ= 2022-09-12T19:41:59.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eHUWDhsAA04hyMkz5u84Rhks8nQ=', 'Content-Length': '0'} 2022-09-12T19:41:59.709 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea0fed04cee79305-00631f8b87-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:59.710 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:41:59.711 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:41:59.712 DEBUG:boto:Method: PUT 2022-09-12T19:41:59.712 DEBUG:boto:Path: /jdpiak-71/e 2022-09-12T19:41:59.713 DEBUG:boto:Data: 2022-09-12T19:41:59.713 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:41:59.714 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:59.714 DEBUG:boto:Port: 8000 2022-09-12T19:41:59.715 DEBUG:boto:Params: {} 2022-09-12T19:41:59.729 DEBUG:boto:Token: None 2022-09-12T19:41:59.731 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:41:59 GMT /jdpiak-71/e 2022-09-12T19:41:59.732 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m/CfwY+7VyG7lOMXvgcIED5chto= 2022-09-12T19:41:59.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:41:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m/CfwY+7VyG7lOMXvgcIED5chto='} 2022-09-12T19:41:59.743 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000017a7dd7b3cdc921d-00631f8b87-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:59.744 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:59.744 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:59.745 DEBUG:boto:Method: HEAD 2022-09-12T19:41:59.746 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:41:59.746 DEBUG:boto:Data: 2022-09-12T19:41:59.747 DEBUG:boto:Headers: {} 2022-09-12T19:41:59.748 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:59.749 DEBUG:boto:Port: 8000 2022-09-12T19:41:59.750 DEBUG:boto:Params: {} 2022-09-12T19:41:59.751 DEBUG:boto:Token: None 2022-09-12T19:41:59.752 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:59 GMT /jdpiak-71/ 2022-09-12T19:41:59.753 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eHUWDhsAA04hyMkz5u84Rhks8nQ= 2022-09-12T19:41:59.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eHUWDhsAA04hyMkz5u84Rhks8nQ=', 'Content-Length': '0'} 2022-09-12T19:41:59.755 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000007a788ecf188a2-00631f8b87-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:59.756 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:41:59.757 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:41:59.758 DEBUG:boto:Method: PUT 2022-09-12T19:41:59.759 DEBUG:boto:Path: /jdpiak-71/f 2022-09-12T19:41:59.760 DEBUG:boto:Data: 2022-09-12T19:41:59.761 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:41:59.762 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:59.763 DEBUG:boto:Port: 8000 2022-09-12T19:41:59.764 DEBUG:boto:Params: {} 2022-09-12T19:41:59.765 DEBUG:boto:Token: None 2022-09-12T19:41:59.766 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:41:59 GMT /jdpiak-71/f 2022-09-12T19:41:59.767 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ursa4hxsSV64oIcMa5X8YdUJjls= 2022-09-12T19:41:59.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:41:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ursa4hxsSV64oIcMa5X8YdUJjls='} 2022-09-12T19:41:59.819 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d66cfe9b41ce97e2-00631f8b87-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:59.820 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:59.822 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:59.823 DEBUG:boto:Method: HEAD 2022-09-12T19:41:59.824 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:41:59.825 DEBUG:boto:Data: 2022-09-12T19:41:59.826 DEBUG:boto:Headers: {} 2022-09-12T19:41:59.828 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:59.831 DEBUG:boto:Port: 8000 2022-09-12T19:41:59.832 DEBUG:boto:Params: {} 2022-09-12T19:41:59.833 DEBUG:boto:Token: None 2022-09-12T19:41:59.836 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:59 GMT /jdpiak-71/ 2022-09-12T19:41:59.837 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eHUWDhsAA04hyMkz5u84Rhks8nQ= 2022-09-12T19:41:59.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eHUWDhsAA04hyMkz5u84Rhks8nQ=', 'Content-Length': '0'} 2022-09-12T19:41:59.842 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb1ab741e1831fe6-00631f8b87-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:59.844 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:41:59.845 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:41:59.845 DEBUG:boto:Method: PUT 2022-09-12T19:41:59.846 DEBUG:boto:Path: /jdpiak-71/g 2022-09-12T19:41:59.846 DEBUG:boto:Data: 2022-09-12T19:41:59.847 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:41:59.847 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:59.848 DEBUG:boto:Port: 8000 2022-09-12T19:41:59.848 DEBUG:boto:Params: {} 2022-09-12T19:41:59.849 DEBUG:boto:Token: None 2022-09-12T19:41:59.850 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:41:59 GMT /jdpiak-71/g 2022-09-12T19:41:59.850 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nlk44fn0xQLNR57h76LjjhGLTIs= 2022-09-12T19:41:59.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:41:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nlk44fn0xQLNR57h76LjjhGLTIs='} 2022-09-12T19:41:59.862 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008bd1759319b89073-00631f8b87-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:59.863 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:41:59.864 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:41:59.865 DEBUG:boto:Method: HEAD 2022-09-12T19:41:59.866 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:41:59.867 DEBUG:boto:Data: 2022-09-12T19:41:59.867 DEBUG:boto:Headers: {} 2022-09-12T19:41:59.868 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:59.869 DEBUG:boto:Port: 8000 2022-09-12T19:41:59.870 DEBUG:boto:Params: {} 2022-09-12T19:41:59.872 DEBUG:boto:Token: None 2022-09-12T19:41:59.873 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:41:59 GMT /jdpiak-71/ 2022-09-12T19:41:59.874 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eHUWDhsAA04hyMkz5u84Rhks8nQ= 2022-09-12T19:41:59.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:41:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eHUWDhsAA04hyMkz5u84Rhks8nQ=', 'Content-Length': '0'} 2022-09-12T19:41:59.878 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000664b35b48e3a94d8-00631f8b87-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:59.880 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:41:59.881 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:41:59.882 DEBUG:boto:Method: PUT 2022-09-12T19:41:59.883 DEBUG:boto:Path: /jdpiak-71/h 2022-09-12T19:41:59.884 DEBUG:boto:Data: 2022-09-12T19:41:59.885 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:41:59.886 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:41:59.887 DEBUG:boto:Port: 8000 2022-09-12T19:41:59.888 DEBUG:boto:Params: {} 2022-09-12T19:41:59.889 DEBUG:boto:Token: None 2022-09-12T19:41:59.890 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:41:59 GMT /jdpiak-71/h 2022-09-12T19:41:59.891 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IcmfpjIj5y9xL2Qrg9vnAaUMPFU= 2022-09-12T19:41:59.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:41:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IcmfpjIj5y9xL2Qrg9vnAaUMPFU='} 2022-09-12T19:41:59.938 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a00d86912fdfe53-00631f8b87-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:41:59.939 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-71 2022-09-12T19:41:59.940 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:41:59.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:41:59.942 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-71 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:42:00.007 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:42:00.008 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:42:00.045 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:42:00.042+0000 7f0b6814f600 1 waiting to reach latest gen 1, on 0.. 2022-09-12T19:42:05.045 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:42:05.042+0000 7f0b6814f600 1 waiting to reach latest gen 1, on 0.. 2022-09-12T19:42:10.045 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:42:10.043+0000 7f0b6814f600 1 waiting to reach latest gen 1, on 0.. 2022-09-12T19:42:15.047 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:42:15.045+0000 7f0b6814f600 1 waiting for incremental sync to catch up: 2022-09-12T19:42:15.048 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , ] 2022-09-12T19:42:15.048 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:42:20.048 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:42:20.046+0000 7f0b6814f600 1 waiting for incremental sync to catch up: 2022-09-12T19:42:20.048 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , ] 2022-09-12T19:42:20.049 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:42:25.049 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:42:25.046+0000 7f0b6814f600 1 waiting for incremental sync to catch up: 2022-09-12T19:42:25.050 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , ] 2022-09-12T19:42:25.051 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:42:30.051 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:42:30.047+0000 7f0b6814f600 1 waiting for incremental sync to catch up: 2022-09-12T19:42:30.052 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , ] 2022-09-12T19:42:30.052 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:42:35.051 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:42:35.048+0000 7f0b6814f600 1 waiting for incremental sync to catch up: 2022-09-12T19:42:35.052 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:42:35.052 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:42:40.053 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:42:40.049+0000 7f0b6814f600 1 waiting for incremental sync to catch up: 2022-09-12T19:42:40.054 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:42:40.054 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:42:45.053 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:42:45.050+0000 7f0b6814f600 1 waiting for incremental sync to catch up: 2022-09-12T19:42:45.054 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:42:45.054 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:42:50.054 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:42:50.051+0000 7f0b6814f600 1 waiting for incremental sync to catch up: 2022-09-12T19:42:50.055 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:42:50.056 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:42:55.055 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:42:55.052+0000 7f0b6814f600 1 waiting for incremental sync to catch up: 2022-09-12T19:42:55.055 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:42:55.055 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:00.056 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:43:00.054+0000 7f0b6814f600 1 waiting for incremental sync to catch up: 2022-09-12T19:43:00.056 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:00.056 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:05.057 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:43:05.055+0000 7f0b6814f600 1 waiting for incremental sync to catch up: 2022-09-12T19:43:05.057 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:05.057 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:10.058 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:43:10.055+0000 7f0b6814f600 1 waiting for incremental sync to catch up: 2022-09-12T19:43:10.059 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:10.059 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:15.061 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:43:15.056+0000 7f0b6814f600 1 waiting for incremental sync to catch up: 2022-09-12T19:43:15.062 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:15.063 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:20.061 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:43:20.057+0000 7f0b6814f600 1 waiting for incremental sync to catch up: 2022-09-12T19:43:20.061 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:20.062 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:25.063 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:43:25.058+0000 7f0b6814f600 1 waiting for incremental sync to catch up: 2022-09-12T19:43:25.070 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:25.070 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:30.062 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:43:30.059+0000 7f0b6814f600 1 waiting for incremental sync to catch up: 2022-09-12T19:43:30.063 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:30.063 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:35.063 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:43:35.060+0000 7f0b6814f600 1 waiting for incremental sync to catch up: 2022-09-12T19:43:35.063 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:35.064 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:40.064 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:43:40.061+0000 7f0b6814f600 1 waiting for incremental sync to catch up: 2022-09-12T19:43:40.065 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:40.066 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:45.065 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:43:45.062+0000 7f0b6814f600 1 waiting for incremental sync to catch up: 2022-09-12T19:43:45.065 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:45.066 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:50.068 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:43:50.064+0000 7f0b6814f600 1 waiting for incremental sync to catch up: 2022-09-12T19:43:50.068 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:50.068 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:55.067 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:43:55.064+0000 7f0b6814f600 1 bucket sync caught up with source: 2022-09-12T19:43:55.068 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:55.068 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:55.069 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:43:55.064+0000 7f0b6814f600 0 bucket checkpoint complete 2022-09-12T19:43:55.087 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-71 2022-09-12T19:43:55.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:43:55.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:43:55.092 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-71 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:43:55.162 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:43:55.162 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:43:55.204 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:43:55.201+0000 7f5b902a4600 1 bucket sync caught up with source: 2022-09-12T19:43:55.204 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:55.204 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:55.205 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:43:55.201+0000 7f5b902a4600 0 bucket checkpoint complete 2022-09-12T19:43:55.224 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-71 2022-09-12T19:43:55.226 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:43:55.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:43:55.227 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-71 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:43:55.290 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:43:55.291 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:43:55.335 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:43:55.333+0000 7f6e429da600 1 bucket sync caught up with source: 2022-09-12T19:43:55.336 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:55.336 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.76.5, 00000000002.41.5, 00000000002.64.5] 2022-09-12T19:43:55.337 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:43:55.333+0000 7f6e429da600 0 bucket checkpoint complete 2022-09-12T19:43:55.354 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-71 2022-09-12T19:43:55.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:43:55.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:43:55.357 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-71 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:43:55.431 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:43:55.431 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:43:55.464 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:43:55.461+0000 7f39c7854600 1 bucket sync caught up with source: 2022-09-12T19:43:55.464 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.890.5, , , 00000000001.976.5, 00000000001.1376.5, , , 00000000002.1141.5, , 00000000002.514.5] 2022-09-12T19:43:55.465 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.890.5, , , 00000000001.976.5, 00000000001.1376.5, , , 00000000002.1141.5, , 00000000002.514.5] 2022-09-12T19:43:55.465 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:43:55.461+0000 7f39c7854600 0 bucket checkpoint complete 2022-09-12T19:43:55.479 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-71 2022-09-12T19:43:55.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:43:55.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:43:55.482 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-71 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:43:55.542 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:43:55.542 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:43:55.580 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:43:55.577+0000 7f086c499600 1 bucket sync caught up with source: 2022-09-12T19:43:55.580 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.890.5, , , 00000000001.976.5, 00000000001.1376.5, , , 00000000002.1141.5, , 00000000002.514.5] 2022-09-12T19:43:55.580 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.890.5, , , 00000000001.976.5, 00000000001.1376.5, , , 00000000002.1141.5, , 00000000002.514.5] 2022-09-12T19:43:55.581 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:43:55.577+0000 7f086c499600 0 bucket checkpoint complete 2022-09-12T19:43:55.595 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-71 2022-09-12T19:43:55.597 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-71 2022-09-12T19:43:55.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:43:55.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:43:55.600 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-71 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:43:55.659 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:43:55.660 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:43:55.714 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:43:55.711+0000 7f1ed19c3600 1 bucket sync caught up with source: 2022-09-12T19:43:55.714 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.500.5, , , 00000000001.486.5, 00000000001.744.5, , , 00000000002.451.5, , 00000000002.246.5] 2022-09-12T19:43:55.715 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.500.5, , , 00000000001.486.5, 00000000001.744.5, , , 00000000002.451.5, , 00000000002.246.5] 2022-09-12T19:43:55.715 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:43:55.711+0000 7f1ed19c3600 0 bucket checkpoint complete 2022-09-12T19:43:55.728 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-71 2022-09-12T19:43:55.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:43:55.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:43:55.732 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-71 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:43:55.796 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:43:55.796 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:43:55.846 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:43:55.844+0000 7fef5ec7c600 1 bucket sync caught up with source: 2022-09-12T19:43:55.847 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.500.5, , , 00000000001.486.5, 00000000001.744.5, , , 00000000002.451.5, , 00000000002.246.5] 2022-09-12T19:43:55.847 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000002.500.5, , , 00000000001.486.5, 00000000001.744.5, , , 00000000002.451.5, , 00000000002.246.5] 2022-09-12T19:43:55.848 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:43:55.844+0000 7fef5ec7c600 0 bucket checkpoint complete 2022-09-12T19:43:55.867 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-71 2022-09-12T19:43:55.868 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-71 zones={test-zone2, test-zone1} 2022-09-12T19:43:55.882 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:55.886 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:55.887 DEBUG:boto:Method: HEAD 2022-09-12T19:43:55.888 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:43:55.888 DEBUG:boto:Data: 2022-09-12T19:43:55.889 DEBUG:boto:Headers: {} 2022-09-12T19:43:55.889 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:55.890 DEBUG:boto:Port: 8000 2022-09-12T19:43:55.890 DEBUG:boto:Params: {} 2022-09-12T19:43:55.892 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:43:55.892 DEBUG:boto:Token: None 2022-09-12T19:43:55.893 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:55 GMT /jdpiak-71/ 2022-09-12T19:43:55.894 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YrBb6sRFfoz06qFOff5giMYbeTU= 2022-09-12T19:43:55.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YrBb6sRFfoz06qFOff5giMYbeTU=', 'Content-Length': '0'} 2022-09-12T19:43:55.904 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014f7f216708dd529-00631f8bfb-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:43:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:55.906 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:55.906 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:55.907 DEBUG:boto:Method: HEAD 2022-09-12T19:43:55.908 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:43:55.909 DEBUG:boto:Data: 2022-09-12T19:43:55.910 DEBUG:boto:Headers: {} 2022-09-12T19:43:55.911 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:55.913 DEBUG:boto:Port: 8000 2022-09-12T19:43:55.914 DEBUG:boto:Params: {} 2022-09-12T19:43:55.915 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:43:55.916 DEBUG:boto:Token: None 2022-09-12T19:43:55.917 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:55 GMT /jdpiak-71/ 2022-09-12T19:43:55.918 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YrBb6sRFfoz06qFOff5giMYbeTU= 2022-09-12T19:43:55.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YrBb6sRFfoz06qFOff5giMYbeTU=', 'Content-Length': '0'} 2022-09-12T19:43:55.923 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d29ca69e8a8eadd6-00631f8bfb-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:43:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:55.925 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:43:55.926 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:55.927 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:55.928 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:43:55.929 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:43:55.931 DEBUG:boto:Method: GET 2022-09-12T19:43:55.932 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:43:55.933 DEBUG:boto:Data: 2022-09-12T19:43:55.935 DEBUG:boto:Headers: {} 2022-09-12T19:43:55.936 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:55.937 DEBUG:boto:Port: 8000 2022-09-12T19:43:55.938 DEBUG:boto:Params: {} 2022-09-12T19:43:55.939 DEBUG:boto:Token: None 2022-09-12T19:43:55.940 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:55 GMT /jdpiak-71/?versions 2022-09-12T19:43:55.942 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PxkB6R2U0HY0p7MKpwujloycX5k= 2022-09-12T19:43:55.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PxkB6R2U0HY0p7MKpwujloycX5k=', 'Content-Length': '0'} 2022-09-12T19:43:55.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce378f958a8c65ba-00631f8bfb-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:55.947 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:55.951 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:43:55.952 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:43:55.953 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:43:55.954 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:43:55.954 DEBUG:tasks.rgw_multi.zone_rados:o=e 2022-09-12T19:43:55.955 DEBUG:tasks.rgw_multi.zone_rados:o=f 2022-09-12T19:43:55.956 DEBUG:tasks.rgw_multi.zone_rados:o=g 2022-09-12T19:43:55.956 DEBUG:tasks.rgw_multi.zone_rados:o=h 2022-09-12T19:43:55.957 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:43:55.957 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:55.958 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:55.960 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:43:55.961 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:43:55.962 DEBUG:boto:Method: GET 2022-09-12T19:43:55.963 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:43:55.964 DEBUG:boto:Data: 2022-09-12T19:43:55.965 DEBUG:boto:Headers: {} 2022-09-12T19:43:55.967 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:55.967 DEBUG:boto:Port: 8000 2022-09-12T19:43:55.968 DEBUG:boto:Params: {} 2022-09-12T19:43:55.970 DEBUG:boto:Token: None 2022-09-12T19:43:55.972 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:55 GMT /jdpiak-71/?versions 2022-09-12T19:43:55.973 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PxkB6R2U0HY0p7MKpwujloycX5k= 2022-09-12T19:43:55.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PxkB6R2U0HY0p7MKpwujloycX5k=', 'Content-Length': '0'} 2022-09-12T19:43:55.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7f83d81d53e6989-00631f8bfb-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:55.978 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:55.981 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:43:55.982 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:43:55.983 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:43:55.984 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:43:55.985 DEBUG:tasks.rgw_multi.zone_rados:o=e 2022-09-12T19:43:55.986 DEBUG:tasks.rgw_multi.zone_rados:o=f 2022-09-12T19:43:55.987 DEBUG:tasks.rgw_multi.zone_rados:o=g 2022-09-12T19:43:55.989 DEBUG:tasks.rgw_multi.zone_rados:o=h 2022-09-12T19:43:55.990 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:55.991 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:55.993 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:43:55.994 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:43:55.995 DEBUG:boto:Method: GET 2022-09-12T19:43:55.996 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:43:55.997 DEBUG:boto:Data: 2022-09-12T19:43:55.998 DEBUG:boto:Headers: {} 2022-09-12T19:43:55.999 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:56.000 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.001 DEBUG:boto:Params: {} 2022-09-12T19:43:56.003 DEBUG:boto:Token: None 2022-09-12T19:43:56.005 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/?versions 2022-09-12T19:43:56.006 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LS3yYpmJyyGGPpK3S77hShaSNx4= 2022-09-12T19:43:56.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LS3yYpmJyyGGPpK3S77hShaSNx4=', 'Content-Length': '0'} 2022-09-12T19:43:56.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c68a3a911ff07d0d-00631f8bfc-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.011 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:56.012 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:56.013 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:56.014 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:43:56.015 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:43:56.016 DEBUG:boto:Method: GET 2022-09-12T19:43:56.016 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:43:56.017 DEBUG:boto:Data: 2022-09-12T19:43:56.017 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.018 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:56.018 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.019 DEBUG:boto:Params: {} 2022-09-12T19:43:56.020 DEBUG:boto:Token: None 2022-09-12T19:43:56.020 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/?versions 2022-09-12T19:43:56.021 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LS3yYpmJyyGGPpK3S77hShaSNx4= 2022-09-12T19:43:56.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LS3yYpmJyyGGPpK3S77hShaSNx4=', 'Content-Length': '0'} 2022-09-12T19:43:56.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4243c73bd6bed90-00631f8bfc-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.029 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:56.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:43:56.031 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:43:56.032 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:43:56.032 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:43:56.033 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:43:56.034 DEBUG:boto:Method: GET 2022-09-12T19:43:56.035 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:43:56.036 DEBUG:boto:Data: 2022-09-12T19:43:56.036 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.037 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:56.037 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.047 DEBUG:boto:Params: {} 2022-09-12T19:43:56.048 DEBUG:boto:Token: None 2022-09-12T19:43:56.049 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:43:56.050 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9+RaF01/s0DJQZJ6mYFii8aSTWg= 2022-09-12T19:43:56.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9+RaF01/s0DJQZJ6mYFii8aSTWg=', 'Content-Length': '0'} 2022-09-12T19:43:56.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041ae32a520465ca9-00631f8bfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.056 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:43:56.056 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:43:56.057 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:43:56.057 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:43:56.058 DEBUG:boto:Method: GET 2022-09-12T19:43:56.058 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:43:56.059 DEBUG:boto:Data: 2022-09-12T19:43:56.060 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.060 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:56.061 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.061 DEBUG:boto:Params: {} 2022-09-12T19:43:56.062 DEBUG:boto:Token: None 2022-09-12T19:43:56.062 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:43:56.063 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9+RaF01/s0DJQZJ6mYFii8aSTWg= 2022-09-12T19:43:56.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9+RaF01/s0DJQZJ6mYFii8aSTWg=', 'Content-Length': '0'} 2022-09-12T19:43:56.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9ab23493e899b62-00631f8bfc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.067 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:43:56.068 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:43:56.068 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:43:56.069 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:43:56.069 DEBUG:boto:Method: HEAD 2022-09-12T19:43:56.070 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:43:56.070 DEBUG:boto:Data: 2022-09-12T19:43:56.071 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.071 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:56.072 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.072 DEBUG:boto:Params: {} 2022-09-12T19:43:56.073 DEBUG:boto:Token: None 2022-09-12T19:43:56.074 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:43:56.074 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tQVKixcueP+IXjHTdJYaMpah6rw= 2022-09-12T19:43:56.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tQVKixcueP+IXjHTdJYaMpah6rw=', 'Content-Length': '0'} 2022-09-12T19:43:56.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000809ef7c986c20c38-00631f8bfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.077 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:43:56.078 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:43:56.079 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:43:56.079 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:43:56.080 DEBUG:boto:Method: HEAD 2022-09-12T19:43:56.080 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:43:56.081 DEBUG:boto:Data: 2022-09-12T19:43:56.081 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.082 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:56.082 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.083 DEBUG:boto:Params: {} 2022-09-12T19:43:56.083 DEBUG:boto:Token: None 2022-09-12T19:43:56.084 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:43:56.085 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tQVKixcueP+IXjHTdJYaMpah6rw= 2022-09-12T19:43:56.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tQVKixcueP+IXjHTdJYaMpah6rw=', 'Content-Length': '0'} 2022-09-12T19:43:56.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d95439b46be37b16-00631f8bfc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:43:56.089 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:43:56.089 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:43:56.090 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:43:56.090 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:43:56.091 DEBUG:boto:Method: GET 2022-09-12T19:43:56.091 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:43:56.092 DEBUG:boto:Data: 2022-09-12T19:43:56.092 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.093 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:56.093 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.094 DEBUG:boto:Params: {} 2022-09-12T19:43:56.095 DEBUG:boto:Token: None 2022-09-12T19:43:56.095 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:43:56.096 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9+RaF01/s0DJQZJ6mYFii8aSTWg= 2022-09-12T19:43:56.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9+RaF01/s0DJQZJ6mYFii8aSTWg=', 'Content-Length': '0'} 2022-09-12T19:43:56.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035cc4c1697fe3c7b-00631f8bfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.099 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:43:56.100 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:43:56.100 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:43:56.101 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:43:56.102 DEBUG:boto:Method: GET 2022-09-12T19:43:56.103 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:43:56.104 DEBUG:boto:Data: 2022-09-12T19:43:56.105 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.106 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:56.107 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.108 DEBUG:boto:Params: {} 2022-09-12T19:43:56.109 DEBUG:boto:Token: None 2022-09-12T19:43:56.110 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:43:56.111 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9+RaF01/s0DJQZJ6mYFii8aSTWg= 2022-09-12T19:43:56.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9+RaF01/s0DJQZJ6mYFii8aSTWg=', 'Content-Length': '0'} 2022-09-12T19:43:56.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000945d5023a2fdb87c-00631f8bfc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.117 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:56.117 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:56.118 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:43:56.118 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:43:56.119 DEBUG:boto:Method: GET 2022-09-12T19:43:56.120 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:43:56.121 DEBUG:boto:Data: 2022-09-12T19:43:56.122 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.122 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:56.124 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.124 DEBUG:boto:Params: {} 2022-09-12T19:43:56.126 DEBUG:boto:Token: None 2022-09-12T19:43:56.127 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/ 2022-09-12T19:43:56.128 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PL3WIkRWNfzFsTd2AO455l0S+MQ= 2022-09-12T19:43:56.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PL3WIkRWNfzFsTd2AO455l0S+MQ=', 'Content-Length': '0'} 2022-09-12T19:43:56.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f84918a52ce504a-00631f8bfc-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.134 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:56.136 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:56.137 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:56.138 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:43:56.139 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:43:56.141 DEBUG:boto:Method: GET 2022-09-12T19:43:56.142 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:43:56.143 DEBUG:boto:Data: 2022-09-12T19:43:56.144 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.145 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:56.147 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.148 DEBUG:boto:Params: {} 2022-09-12T19:43:56.149 DEBUG:boto:Token: None 2022-09-12T19:43:56.150 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/ 2022-09-12T19:43:56.150 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PL3WIkRWNfzFsTd2AO455l0S+MQ= 2022-09-12T19:43:56.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PL3WIkRWNfzFsTd2AO455l0S+MQ=', 'Content-Length': '0'} 2022-09-12T19:43:56.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfe6031dec18b6d2-00631f8bfc-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.155 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:56.156 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:43:56.157 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:43:56.158 DEBUG:boto:Method: HEAD 2022-09-12T19:43:56.158 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:43:56.159 DEBUG:boto:Data: 2022-09-12T19:43:56.160 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.160 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:56.161 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.161 DEBUG:boto:Params: {} 2022-09-12T19:43:56.162 DEBUG:boto:Token: None 2022-09-12T19:43:56.163 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/a 2022-09-12T19:43:56.163 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Wed/YaH4fTu4cvD4JNugRtn/6nE= 2022-09-12T19:43:56.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Wed/YaH4fTu4cvD4JNugRtn/6nE=', 'Content-Length': '0'} 2022-09-12T19:43:56.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd7ffc6c652de024-00631f8bfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.167 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:43:56.167 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:43:56.168 DEBUG:boto:Method: HEAD 2022-09-12T19:43:56.169 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:43:56.169 DEBUG:boto:Data: 2022-09-12T19:43:56.170 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.171 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:56.171 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.172 DEBUG:boto:Params: {} 2022-09-12T19:43:56.172 DEBUG:boto:Token: None 2022-09-12T19:43:56.173 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/a 2022-09-12T19:43:56.173 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Wed/YaH4fTu4cvD4JNugRtn/6nE= 2022-09-12T19:43:56.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Wed/YaH4fTu4cvD4JNugRtn/6nE=', 'Content-Length': '0'} 2022-09-12T19:43:56.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c00d36ea46f59df-00631f8bfc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:43:56.182 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:43:56.183 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:43:56.184 DEBUG:boto:Method: GET 2022-09-12T19:43:56.185 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:43:56.186 DEBUG:boto:Data: 2022-09-12T19:43:56.187 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.188 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:56.189 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.191 DEBUG:boto:Params: {} 2022-09-12T19:43:56.192 DEBUG:boto:Token: None 2022-09-12T19:43:56.193 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/a 2022-09-12T19:43:56.194 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9AN0A30kKzjvI0szAOkrD/ZcCZ4= 2022-09-12T19:43:56.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9AN0A30kKzjvI0szAOkrD/ZcCZ4=', 'Content-Length': '0'} 2022-09-12T19:43:56.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f9163bb86127790-00631f8bfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.197 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:43:56.198 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:43:56.199 DEBUG:boto:Method: GET 2022-09-12T19:43:56.199 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:43:56.200 DEBUG:boto:Data: 2022-09-12T19:43:56.201 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.201 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:56.202 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.203 DEBUG:boto:Params: {} 2022-09-12T19:43:56.203 DEBUG:boto:Token: None 2022-09-12T19:43:56.204 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/a 2022-09-12T19:43:56.205 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9AN0A30kKzjvI0szAOkrD/ZcCZ4= 2022-09-12T19:43:56.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9AN0A30kKzjvI0szAOkrD/ZcCZ4=', 'Content-Length': '0'} 2022-09-12T19:43:56.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6cc156545f4ef20-00631f8bfc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.254 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:56.255 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:56.256 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:43:56.257 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:43:56.258 DEBUG:boto:Method: GET 2022-09-12T19:43:56.258 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:43:56.260 DEBUG:boto:Data: 2022-09-12T19:43:56.262 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.263 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:56.264 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.265 DEBUG:boto:Params: {} 2022-09-12T19:43:56.266 DEBUG:boto:Token: None 2022-09-12T19:43:56.267 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/ 2022-09-12T19:43:56.268 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PL3WIkRWNfzFsTd2AO455l0S+MQ= 2022-09-12T19:43:56.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PL3WIkRWNfzFsTd2AO455l0S+MQ=', 'Content-Length': '0'} 2022-09-12T19:43:56.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bcf7c8a56b6965d-00631f8bfc-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.273 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:56.275 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:56.276 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:56.277 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:43:56.278 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:43:56.279 DEBUG:boto:Method: GET 2022-09-12T19:43:56.280 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:43:56.281 DEBUG:boto:Data: 2022-09-12T19:43:56.282 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.283 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:56.284 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.284 DEBUG:boto:Params: {} 2022-09-12T19:43:56.285 DEBUG:boto:Token: None 2022-09-12T19:43:56.287 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/ 2022-09-12T19:43:56.288 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PL3WIkRWNfzFsTd2AO455l0S+MQ= 2022-09-12T19:43:56.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PL3WIkRWNfzFsTd2AO455l0S+MQ=', 'Content-Length': '0'} 2022-09-12T19:43:56.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1764cb395756dba-00631f8bfc-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.295 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:56.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:43:56.296 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:43:56.297 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:43:56.298 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:43:56.298 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:43:56.299 DEBUG:boto:Method: GET 2022-09-12T19:43:56.300 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:43:56.300 DEBUG:boto:Data: 2022-09-12T19:43:56.301 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.302 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:56.303 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.303 DEBUG:boto:Params: {} 2022-09-12T19:43:56.304 DEBUG:boto:Token: None 2022-09-12T19:43:56.305 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:43:56.306 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W28AXn3O7u97/ga4TBlWEt/VVxg= 2022-09-12T19:43:56.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W28AXn3O7u97/ga4TBlWEt/VVxg=', 'Content-Length': '0'} 2022-09-12T19:43:56.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0dae24d8116cbae-00631f8bfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.311 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:43:56.312 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:43:56.312 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:43:56.313 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:43:56.313 DEBUG:boto:Method: GET 2022-09-12T19:43:56.314 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:43:56.315 DEBUG:boto:Data: 2022-09-12T19:43:56.315 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.316 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:56.317 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.317 DEBUG:boto:Params: {} 2022-09-12T19:43:56.318 DEBUG:boto:Token: None 2022-09-12T19:43:56.319 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:43:56.320 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W28AXn3O7u97/ga4TBlWEt/VVxg= 2022-09-12T19:43:56.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W28AXn3O7u97/ga4TBlWEt/VVxg=', 'Content-Length': '0'} 2022-09-12T19:43:56.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1dfabf7bfab15dd-00631f8bfc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.324 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:43:56.324 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:43:56.325 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:43:56.326 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:43:56.326 DEBUG:boto:Method: HEAD 2022-09-12T19:43:56.327 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:43:56.328 DEBUG:boto:Data: 2022-09-12T19:43:56.328 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.329 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:56.329 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.330 DEBUG:boto:Params: {} 2022-09-12T19:43:56.331 DEBUG:boto:Token: None 2022-09-12T19:43:56.332 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:43:56.333 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uoJIJGEjF/JNB7vt8rWhihojBfY= 2022-09-12T19:43:56.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uoJIJGEjF/JNB7vt8rWhihojBfY=', 'Content-Length': '0'} 2022-09-12T19:43:56.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060463b39449cf478-00631f8bfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.337 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:43:56.338 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:43:56.339 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:43:56.340 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:43:56.341 DEBUG:boto:Method: HEAD 2022-09-12T19:43:56.342 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:43:56.343 DEBUG:boto:Data: 2022-09-12T19:43:56.344 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.345 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:56.346 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.347 DEBUG:boto:Params: {} 2022-09-12T19:43:56.349 DEBUG:boto:Token: None 2022-09-12T19:43:56.350 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:43:56.351 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uoJIJGEjF/JNB7vt8rWhihojBfY= 2022-09-12T19:43:56.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uoJIJGEjF/JNB7vt8rWhihojBfY=', 'Content-Length': '0'} 2022-09-12T19:43:56.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dab822d57325054-00631f8bfc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:43:56.359 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:43:56.360 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:43:56.361 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:43:56.362 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:43:56.363 DEBUG:boto:Method: GET 2022-09-12T19:43:56.364 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:43:56.366 DEBUG:boto:Data: 2022-09-12T19:43:56.367 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.368 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:56.369 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.370 DEBUG:boto:Params: {} 2022-09-12T19:43:56.371 DEBUG:boto:Token: None 2022-09-12T19:43:56.373 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:43:56.375 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W28AXn3O7u97/ga4TBlWEt/VVxg= 2022-09-12T19:43:56.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W28AXn3O7u97/ga4TBlWEt/VVxg=', 'Content-Length': '0'} 2022-09-12T19:43:56.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8c46bc68bf1194d-00631f8bfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.380 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:43:56.381 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:43:56.382 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:43:56.383 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:43:56.384 DEBUG:boto:Method: GET 2022-09-12T19:43:56.385 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:43:56.386 DEBUG:boto:Data: 2022-09-12T19:43:56.387 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.388 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:56.389 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.390 DEBUG:boto:Params: {} 2022-09-12T19:43:56.391 DEBUG:boto:Token: None 2022-09-12T19:43:56.393 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:43:56.394 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W28AXn3O7u97/ga4TBlWEt/VVxg= 2022-09-12T19:43:56.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W28AXn3O7u97/ga4TBlWEt/VVxg=', 'Content-Length': '0'} 2022-09-12T19:43:56.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031527c30f765af0d-00631f8bfc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.401 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:56.402 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:56.403 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:43:56.404 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:43:56.405 DEBUG:boto:Method: GET 2022-09-12T19:43:56.406 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:43:56.407 DEBUG:boto:Data: 2022-09-12T19:43:56.409 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.410 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:56.411 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.412 DEBUG:boto:Params: {} 2022-09-12T19:43:56.414 DEBUG:boto:Token: None 2022-09-12T19:43:56.415 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/ 2022-09-12T19:43:56.417 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PL3WIkRWNfzFsTd2AO455l0S+MQ= 2022-09-12T19:43:56.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PL3WIkRWNfzFsTd2AO455l0S+MQ=', 'Content-Length': '0'} 2022-09-12T19:43:56.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1518303d79e62c3-00631f8bfc-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.466 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:56.468 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:56.471 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:56.473 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:43:56.474 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:43:56.476 DEBUG:boto:Method: GET 2022-09-12T19:43:56.478 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:43:56.481 DEBUG:boto:Data: 2022-09-12T19:43:56.485 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.486 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:56.487 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.488 DEBUG:boto:Params: {} 2022-09-12T19:43:56.489 DEBUG:boto:Token: None 2022-09-12T19:43:56.490 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/ 2022-09-12T19:43:56.491 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PL3WIkRWNfzFsTd2AO455l0S+MQ= 2022-09-12T19:43:56.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PL3WIkRWNfzFsTd2AO455l0S+MQ=', 'Content-Length': '0'} 2022-09-12T19:43:56.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb98da6c71f7767a-00631f8bfc-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.501 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:56.503 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:43:56.504 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:43:56.505 DEBUG:boto:Method: HEAD 2022-09-12T19:43:56.506 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:43:56.507 DEBUG:boto:Data: 2022-09-12T19:43:56.508 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.509 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:56.509 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.520 DEBUG:boto:Params: {} 2022-09-12T19:43:56.521 DEBUG:boto:Token: None 2022-09-12T19:43:56.523 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/b 2022-09-12T19:43:56.524 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fBSTgluuBfx3BFWckk/CFsWSZ/Q= 2022-09-12T19:43:56.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fBSTgluuBfx3BFWckk/CFsWSZ/Q=', 'Content-Length': '0'} 2022-09-12T19:43:56.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004666901ad566518-00631f8bfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.530 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:43:56.531 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:43:56.532 DEBUG:boto:Method: HEAD 2022-09-12T19:43:56.533 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:43:56.534 DEBUG:boto:Data: 2022-09-12T19:43:56.534 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.535 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:56.536 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.537 DEBUG:boto:Params: {} 2022-09-12T19:43:56.538 DEBUG:boto:Token: None 2022-09-12T19:43:56.539 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/b 2022-09-12T19:43:56.540 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fBSTgluuBfx3BFWckk/CFsWSZ/Q= 2022-09-12T19:43:56.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fBSTgluuBfx3BFWckk/CFsWSZ/Q=', 'Content-Length': '0'} 2022-09-12T19:43:56.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4a5103b2f08f0c7-00631f8bfc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:43:56.546 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:43:56.547 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:43:56.548 DEBUG:boto:Method: GET 2022-09-12T19:43:56.549 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:43:56.550 DEBUG:boto:Data: 2022-09-12T19:43:56.551 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.552 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:56.553 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.554 DEBUG:boto:Params: {} 2022-09-12T19:43:56.555 DEBUG:boto:Token: None 2022-09-12T19:43:56.555 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/b 2022-09-12T19:43:56.556 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GbLgCYfAnJpmGDLnagfgRL6cBxw= 2022-09-12T19:43:56.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GbLgCYfAnJpmGDLnagfgRL6cBxw=', 'Content-Length': '0'} 2022-09-12T19:43:56.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc63b07bf8234645-00631f8bfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.561 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:43:56.562 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:43:56.563 DEBUG:boto:Method: GET 2022-09-12T19:43:56.563 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:43:56.564 DEBUG:boto:Data: 2022-09-12T19:43:56.564 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.565 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:56.566 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.566 DEBUG:boto:Params: {} 2022-09-12T19:43:56.567 DEBUG:boto:Token: None 2022-09-12T19:43:56.568 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/b 2022-09-12T19:43:56.568 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GbLgCYfAnJpmGDLnagfgRL6cBxw= 2022-09-12T19:43:56.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GbLgCYfAnJpmGDLnagfgRL6cBxw=', 'Content-Length': '0'} 2022-09-12T19:43:56.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075d6aba2e9e3c73b-00631f8bfc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.572 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:56.573 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:56.573 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:43:56.574 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:43:56.575 DEBUG:boto:Method: GET 2022-09-12T19:43:56.575 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:43:56.576 DEBUG:boto:Data: 2022-09-12T19:43:56.576 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.577 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:56.578 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.578 DEBUG:boto:Params: {} 2022-09-12T19:43:56.579 DEBUG:boto:Token: None 2022-09-12T19:43:56.580 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/ 2022-09-12T19:43:56.580 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PL3WIkRWNfzFsTd2AO455l0S+MQ= 2022-09-12T19:43:56.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PL3WIkRWNfzFsTd2AO455l0S+MQ=', 'Content-Length': '0'} 2022-09-12T19:43:56.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065447eb252a36591-00631f8bfc-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.584 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:56.585 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:56.586 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:56.587 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:43:56.587 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:43:56.590 DEBUG:boto:Method: GET 2022-09-12T19:43:56.591 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:43:56.592 DEBUG:boto:Data: 2022-09-12T19:43:56.593 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.594 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:56.595 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.596 DEBUG:boto:Params: {} 2022-09-12T19:43:56.596 DEBUG:boto:Token: None 2022-09-12T19:43:56.597 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/ 2022-09-12T19:43:56.597 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PL3WIkRWNfzFsTd2AO455l0S+MQ= 2022-09-12T19:43:56.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PL3WIkRWNfzFsTd2AO455l0S+MQ=', 'Content-Length': '0'} 2022-09-12T19:43:56.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c7357c37f4230e8-00631f8bfc-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.602 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:56.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:43:56.604 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:43:56.604 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:43:56.605 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:43:56.605 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:43:56.606 DEBUG:boto:Method: GET 2022-09-12T19:43:56.607 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:43:56.607 DEBUG:boto:Data: 2022-09-12T19:43:56.608 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.608 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:56.609 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.610 DEBUG:boto:Params: {} 2022-09-12T19:43:56.611 DEBUG:boto:Token: None 2022-09-12T19:43:56.612 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:43:56.613 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TkJzFHqwmtbOEFpKyMGRCRTYuJ0= 2022-09-12T19:43:56.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TkJzFHqwmtbOEFpKyMGRCRTYuJ0=', 'Content-Length': '0'} 2022-09-12T19:43:56.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000690c29e516f5ff30-00631f8bfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.621 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:43:56.622 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:43:56.622 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:43:56.623 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:43:56.624 DEBUG:boto:Method: GET 2022-09-12T19:43:56.624 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:43:56.625 DEBUG:boto:Data: 2022-09-12T19:43:56.625 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.626 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:56.627 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.628 DEBUG:boto:Params: {} 2022-09-12T19:43:56.629 DEBUG:boto:Token: None 2022-09-12T19:43:56.630 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:43:56.631 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TkJzFHqwmtbOEFpKyMGRCRTYuJ0= 2022-09-12T19:43:56.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TkJzFHqwmtbOEFpKyMGRCRTYuJ0=', 'Content-Length': '0'} 2022-09-12T19:43:56.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecfe3c471a5986c0-00631f8bfc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.649 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:43:56.650 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:43:56.650 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:43:56.651 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:43:56.651 DEBUG:boto:Method: HEAD 2022-09-12T19:43:56.652 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:43:56.652 DEBUG:boto:Data: 2022-09-12T19:43:56.653 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.654 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:56.656 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.657 DEBUG:boto:Params: {} 2022-09-12T19:43:56.658 DEBUG:boto:Token: None 2022-09-12T19:43:56.660 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:43:56.661 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:odtWs+wS8pDL2W2wFmPZYHyOER8= 2022-09-12T19:43:56.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:odtWs+wS8pDL2W2wFmPZYHyOER8=', 'Content-Length': '0'} 2022-09-12T19:43:56.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000775119e87db53956-00631f8bfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.667 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:43:56.669 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:43:56.670 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:43:56.671 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:43:56.672 DEBUG:boto:Method: HEAD 2022-09-12T19:43:56.673 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:43:56.674 DEBUG:boto:Data: 2022-09-12T19:43:56.675 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.676 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:56.677 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.680 DEBUG:boto:Params: {} 2022-09-12T19:43:56.681 DEBUG:boto:Token: None 2022-09-12T19:43:56.682 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:43:56.683 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:odtWs+wS8pDL2W2wFmPZYHyOER8= 2022-09-12T19:43:56.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:odtWs+wS8pDL2W2wFmPZYHyOER8=', 'Content-Length': '0'} 2022-09-12T19:43:56.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb8c43d21d4371fa-00631f8bfc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:43:56.694 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:43:56.694 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:43:56.695 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:43:56.696 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:43:56.696 DEBUG:boto:Method: GET 2022-09-12T19:43:56.697 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:43:56.697 DEBUG:boto:Data: 2022-09-12T19:43:56.698 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.699 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:56.699 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.700 DEBUG:boto:Params: {} 2022-09-12T19:43:56.701 DEBUG:boto:Token: None 2022-09-12T19:43:56.702 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:43:56.704 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TkJzFHqwmtbOEFpKyMGRCRTYuJ0= 2022-09-12T19:43:56.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TkJzFHqwmtbOEFpKyMGRCRTYuJ0=', 'Content-Length': '0'} 2022-09-12T19:43:56.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000374109fe40eadb0a-00631f8bfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.709 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:43:56.710 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:43:56.711 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:43:56.712 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:43:56.714 DEBUG:boto:Method: GET 2022-09-12T19:43:56.715 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:43:56.716 DEBUG:boto:Data: 2022-09-12T19:43:56.717 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.718 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:56.719 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.720 DEBUG:boto:Params: {} 2022-09-12T19:43:56.721 DEBUG:boto:Token: None 2022-09-12T19:43:56.722 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:43:56.724 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TkJzFHqwmtbOEFpKyMGRCRTYuJ0= 2022-09-12T19:43:56.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TkJzFHqwmtbOEFpKyMGRCRTYuJ0=', 'Content-Length': '0'} 2022-09-12T19:43:56.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dae49312bd58ad82-00631f8bfc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.774 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:56.776 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:56.777 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:43:56.777 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:43:56.790 DEBUG:boto:Method: GET 2022-09-12T19:43:56.792 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:43:56.793 DEBUG:boto:Data: 2022-09-12T19:43:56.794 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.795 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:56.796 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.798 DEBUG:boto:Params: {} 2022-09-12T19:43:56.799 DEBUG:boto:Token: None 2022-09-12T19:43:56.800 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/ 2022-09-12T19:43:56.801 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PL3WIkRWNfzFsTd2AO455l0S+MQ= 2022-09-12T19:43:56.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PL3WIkRWNfzFsTd2AO455l0S+MQ=', 'Content-Length': '0'} 2022-09-12T19:43:56.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7467634e8df09af-00631f8bfc-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.807 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:56.808 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:56.808 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:56.809 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:43:56.809 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:43:56.810 DEBUG:boto:Method: GET 2022-09-12T19:43:56.810 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:43:56.811 DEBUG:boto:Data: 2022-09-12T19:43:56.812 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.813 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:56.815 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.816 DEBUG:boto:Params: {} 2022-09-12T19:43:56.817 DEBUG:boto:Token: None 2022-09-12T19:43:56.818 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/ 2022-09-12T19:43:56.819 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PL3WIkRWNfzFsTd2AO455l0S+MQ= 2022-09-12T19:43:56.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PL3WIkRWNfzFsTd2AO455l0S+MQ=', 'Content-Length': '0'} 2022-09-12T19:43:56.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a79bf90d43aaad9-00631f8bfc-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.829 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:56.830 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:43:56.831 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:43:56.832 DEBUG:boto:Method: HEAD 2022-09-12T19:43:56.833 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:43:56.834 DEBUG:boto:Data: 2022-09-12T19:43:56.835 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.836 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:56.837 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.838 DEBUG:boto:Params: {} 2022-09-12T19:43:56.839 DEBUG:boto:Token: None 2022-09-12T19:43:56.840 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/c 2022-09-12T19:43:56.841 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sdpaOrL1u0iRZswCVO6HxXmK4U0= 2022-09-12T19:43:56.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sdpaOrL1u0iRZswCVO6HxXmK4U0=', 'Content-Length': '0'} 2022-09-12T19:43:56.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e009da0c4d70e368-00631f8bfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.847 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:43:56.848 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:43:56.849 DEBUG:boto:Method: HEAD 2022-09-12T19:43:56.850 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:43:56.851 DEBUG:boto:Data: 2022-09-12T19:43:56.852 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.853 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:56.854 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.855 DEBUG:boto:Params: {} 2022-09-12T19:43:56.857 DEBUG:boto:Token: None 2022-09-12T19:43:56.858 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/c 2022-09-12T19:43:56.859 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sdpaOrL1u0iRZswCVO6HxXmK4U0= 2022-09-12T19:43:56.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sdpaOrL1u0iRZswCVO6HxXmK4U0=', 'Content-Length': '0'} 2022-09-12T19:43:56.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b7a7d3a7f7e766d-00631f8bfc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:43:56.865 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:43:56.866 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:43:56.867 DEBUG:boto:Method: GET 2022-09-12T19:43:56.868 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:43:56.870 DEBUG:boto:Data: 2022-09-12T19:43:56.871 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.872 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:56.873 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.874 DEBUG:boto:Params: {} 2022-09-12T19:43:56.875 DEBUG:boto:Token: None 2022-09-12T19:43:56.876 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/c 2022-09-12T19:43:56.877 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0a9MfAxT8gEbuUeEg7sZ092T+io= 2022-09-12T19:43:56.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0a9MfAxT8gEbuUeEg7sZ092T+io=', 'Content-Length': '0'} 2022-09-12T19:43:56.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9e290fe0bb29ede-00631f8bfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.883 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:43:56.885 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:43:56.886 DEBUG:boto:Method: GET 2022-09-12T19:43:56.887 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:43:56.889 DEBUG:boto:Data: 2022-09-12T19:43:56.890 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.891 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:56.892 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.893 DEBUG:boto:Params: {} 2022-09-12T19:43:56.894 DEBUG:boto:Token: None 2022-09-12T19:43:56.897 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/c 2022-09-12T19:43:56.898 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0a9MfAxT8gEbuUeEg7sZ092T+io= 2022-09-12T19:43:56.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0a9MfAxT8gEbuUeEg7sZ092T+io=', 'Content-Length': '0'} 2022-09-12T19:43:56.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ec0cc7a2c5e3c84-00631f8bfc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.908 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:56.909 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:56.910 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:43:56.911 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:43:56.913 DEBUG:boto:Method: GET 2022-09-12T19:43:56.914 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:43:56.915 DEBUG:boto:Data: 2022-09-12T19:43:56.917 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.918 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:56.919 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.920 DEBUG:boto:Params: {} 2022-09-12T19:43:56.922 DEBUG:boto:Token: None 2022-09-12T19:43:56.923 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/ 2022-09-12T19:43:56.924 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PL3WIkRWNfzFsTd2AO455l0S+MQ= 2022-09-12T19:43:56.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PL3WIkRWNfzFsTd2AO455l0S+MQ=', 'Content-Length': '0'} 2022-09-12T19:43:56.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e20e0b0087db243-00631f8bfc-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.932 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:56.934 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:56.936 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:56.938 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:43:56.939 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:43:56.940 DEBUG:boto:Method: GET 2022-09-12T19:43:56.941 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:43:56.942 DEBUG:boto:Data: 2022-09-12T19:43:56.944 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.945 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:56.946 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.947 DEBUG:boto:Params: {} 2022-09-12T19:43:56.948 DEBUG:boto:Token: None 2022-09-12T19:43:56.949 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/ 2022-09-12T19:43:56.950 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PL3WIkRWNfzFsTd2AO455l0S+MQ= 2022-09-12T19:43:56.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PL3WIkRWNfzFsTd2AO455l0S+MQ=', 'Content-Length': '0'} 2022-09-12T19:43:56.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0af4bf4e68d684e-00631f8bfc-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.957 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:56.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:43:56.959 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:43:56.960 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:43:56.962 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:43:56.963 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:43:56.964 DEBUG:boto:Method: GET 2022-09-12T19:43:56.965 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:43:56.966 DEBUG:boto:Data: 2022-09-12T19:43:56.967 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.968 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:56.969 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.970 DEBUG:boto:Params: {} 2022-09-12T19:43:56.971 DEBUG:boto:Token: None 2022-09-12T19:43:56.973 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:43:56.975 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3Bq//R5DJHWVFpdPHo/3D71TERQ= 2022-09-12T19:43:56.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3Bq//R5DJHWVFpdPHo/3D71TERQ=', 'Content-Length': '0'} 2022-09-12T19:43:56.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021e8e9707cc8b219-00631f8bfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:56.981 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:43:56.983 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:43:56.984 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:43:56.985 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:43:56.986 DEBUG:boto:Method: GET 2022-09-12T19:43:56.987 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:43:56.988 DEBUG:boto:Data: 2022-09-12T19:43:56.989 DEBUG:boto:Headers: {} 2022-09-12T19:43:56.990 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:56.991 DEBUG:boto:Port: 8000 2022-09-12T19:43:56.992 DEBUG:boto:Params: {} 2022-09-12T19:43:56.993 DEBUG:boto:Token: None 2022-09-12T19:43:56.994 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:56 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:43:56.995 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3Bq//R5DJHWVFpdPHo/3D71TERQ= 2022-09-12T19:43:56.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3Bq//R5DJHWVFpdPHo/3D71TERQ=', 'Content-Length': '0'} 2022-09-12T19:43:56.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012f70d89e757dcfe-00631f8bfc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.001 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:43:57.001 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:43:57.003 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:43:57.004 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:43:57.005 DEBUG:boto:Method: HEAD 2022-09-12T19:43:57.008 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:43:57.009 DEBUG:boto:Data: 2022-09-12T19:43:57.010 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.011 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:57.012 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.014 DEBUG:boto:Params: {} 2022-09-12T19:43:57.015 DEBUG:boto:Token: None 2022-09-12T19:43:57.016 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:43:57.018 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hxMLfDTSqk9yKL4pQyE+OmPia7s= 2022-09-12T19:43:57.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hxMLfDTSqk9yKL4pQyE+OmPia7s=', 'Content-Length': '0'} 2022-09-12T19:43:57.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004585e2479b82a28-00631f8bfd-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.023 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:43:57.025 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:43:57.026 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:43:57.027 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:43:57.027 DEBUG:boto:Method: HEAD 2022-09-12T19:43:57.028 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:43:57.029 DEBUG:boto:Data: 2022-09-12T19:43:57.030 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.031 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:57.031 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.032 DEBUG:boto:Params: {} 2022-09-12T19:43:57.032 DEBUG:boto:Token: None 2022-09-12T19:43:57.033 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:43:57.034 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hxMLfDTSqk9yKL4pQyE+OmPia7s= 2022-09-12T19:43:57.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hxMLfDTSqk9yKL4pQyE+OmPia7s=', 'Content-Length': '0'} 2022-09-12T19:43:57.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000938c83d68e174264-00631f8bfd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:43:57.038 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:43:57.039 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:43:57.039 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:43:57.040 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:43:57.040 DEBUG:boto:Method: GET 2022-09-12T19:43:57.041 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:43:57.041 DEBUG:boto:Data: 2022-09-12T19:43:57.042 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.043 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:57.044 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.045 DEBUG:boto:Params: {} 2022-09-12T19:43:57.046 DEBUG:boto:Token: None 2022-09-12T19:43:57.047 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:43:57.048 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zh+0qbcZoi1bDBCnGgL4xDCu680= 2022-09-12T19:43:57.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zh+0qbcZoi1bDBCnGgL4xDCu680=', 'Content-Length': '0'} 2022-09-12T19:43:57.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b4bee13bec76b6e-00631f8bfd-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.055 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:43:57.056 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:43:57.057 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:43:57.058 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:43:57.059 DEBUG:boto:Method: GET 2022-09-12T19:43:57.060 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:43:57.061 DEBUG:boto:Data: 2022-09-12T19:43:57.061 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.062 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:57.062 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.063 DEBUG:boto:Params: {} 2022-09-12T19:43:57.064 DEBUG:boto:Token: None 2022-09-12T19:43:57.064 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:43:57.065 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zh+0qbcZoi1bDBCnGgL4xDCu680= 2022-09-12T19:43:57.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zh+0qbcZoi1bDBCnGgL4xDCu680=', 'Content-Length': '0'} 2022-09-12T19:43:57.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbc65012e2504000-00631f8bfd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.110 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:57.111 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:57.113 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:43:57.114 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:43:57.115 DEBUG:boto:Method: GET 2022-09-12T19:43:57.116 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:43:57.117 DEBUG:boto:Data: 2022-09-12T19:43:57.118 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.119 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:57.120 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.121 DEBUG:boto:Params: {} 2022-09-12T19:43:57.122 DEBUG:boto:Token: None 2022-09-12T19:43:57.123 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/ 2022-09-12T19:43:57.125 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kQpDECX8kZmZfUFYqcSjY6YruHA= 2022-09-12T19:43:57.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kQpDECX8kZmZfUFYqcSjY6YruHA=', 'Content-Length': '0'} 2022-09-12T19:43:57.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022b4c8220aabe6b9-00631f8bfd-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.131 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:57.133 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:57.134 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:57.135 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:43:57.136 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:43:57.137 DEBUG:boto:Method: GET 2022-09-12T19:43:57.138 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:43:57.140 DEBUG:boto:Data: 2022-09-12T19:43:57.141 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.141 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:57.142 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.143 DEBUG:boto:Params: {} 2022-09-12T19:43:57.143 DEBUG:boto:Token: None 2022-09-12T19:43:57.144 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/ 2022-09-12T19:43:57.145 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kQpDECX8kZmZfUFYqcSjY6YruHA= 2022-09-12T19:43:57.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kQpDECX8kZmZfUFYqcSjY6YruHA=', 'Content-Length': '0'} 2022-09-12T19:43:57.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d2dffddf93aefc0-00631f8bfd-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.148 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:57.152 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:43:57.152 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:43:57.154 DEBUG:boto:Method: HEAD 2022-09-12T19:43:57.155 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:43:57.156 DEBUG:boto:Data: 2022-09-12T19:43:57.157 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.158 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:57.159 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.160 DEBUG:boto:Params: {} 2022-09-12T19:43:57.161 DEBUG:boto:Token: None 2022-09-12T19:43:57.163 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/d 2022-09-12T19:43:57.164 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EIaS8ZjiZdOPmNFYh3q6oQEzTJk= 2022-09-12T19:43:57.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EIaS8ZjiZdOPmNFYh3q6oQEzTJk=', 'Content-Length': '0'} 2022-09-12T19:43:57.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a8cc56e15b9aedc-00631f8bfd-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.169 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:43:57.170 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:43:57.171 DEBUG:boto:Method: HEAD 2022-09-12T19:43:57.172 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:43:57.174 DEBUG:boto:Data: 2022-09-12T19:43:57.175 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.176 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:57.177 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.178 DEBUG:boto:Params: {} 2022-09-12T19:43:57.179 DEBUG:boto:Token: None 2022-09-12T19:43:57.180 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/d 2022-09-12T19:43:57.181 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EIaS8ZjiZdOPmNFYh3q6oQEzTJk= 2022-09-12T19:43:57.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EIaS8ZjiZdOPmNFYh3q6oQEzTJk=', 'Content-Length': '0'} 2022-09-12T19:43:57.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cca3bd3ff12fd6e5-00631f8bfd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:43:57.188 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:43:57.189 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:43:57.190 DEBUG:boto:Method: GET 2022-09-12T19:43:57.191 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:43:57.192 DEBUG:boto:Data: 2022-09-12T19:43:57.193 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.195 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:57.195 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.196 DEBUG:boto:Params: {} 2022-09-12T19:43:57.197 DEBUG:boto:Token: None 2022-09-12T19:43:57.198 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/d 2022-09-12T19:43:57.199 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2ATYs9ClPfQHY9xwi1tWesRApPU= 2022-09-12T19:43:57.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2ATYs9ClPfQHY9xwi1tWesRApPU=', 'Content-Length': '0'} 2022-09-12T19:43:57.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010cfbfd5ab38071b-00631f8bfd-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.203 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:43:57.203 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:43:57.204 DEBUG:boto:Method: GET 2022-09-12T19:43:57.205 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:43:57.205 DEBUG:boto:Data: 2022-09-12T19:43:57.206 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.208 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:57.208 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.209 DEBUG:boto:Params: {} 2022-09-12T19:43:57.209 DEBUG:boto:Token: None 2022-09-12T19:43:57.210 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/d 2022-09-12T19:43:57.211 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2ATYs9ClPfQHY9xwi1tWesRApPU= 2022-09-12T19:43:57.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2ATYs9ClPfQHY9xwi1tWesRApPU=', 'Content-Length': '0'} 2022-09-12T19:43:57.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f62a5a72d48acce1-00631f8bfd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.214 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:57.215 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:57.215 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:43:57.216 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:43:57.216 DEBUG:boto:Method: GET 2022-09-12T19:43:57.217 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:43:57.217 DEBUG:boto:Data: 2022-09-12T19:43:57.218 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.218 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:57.219 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.219 DEBUG:boto:Params: {} 2022-09-12T19:43:57.220 DEBUG:boto:Token: None 2022-09-12T19:43:57.221 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/ 2022-09-12T19:43:57.221 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kQpDECX8kZmZfUFYqcSjY6YruHA= 2022-09-12T19:43:57.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kQpDECX8kZmZfUFYqcSjY6YruHA=', 'Content-Length': '0'} 2022-09-12T19:43:57.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf930b5bbf54805a-00631f8bfd-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.226 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:57.227 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:57.227 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:57.228 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:43:57.228 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:43:57.229 DEBUG:boto:Method: GET 2022-09-12T19:43:57.229 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:43:57.230 DEBUG:boto:Data: 2022-09-12T19:43:57.230 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.231 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:57.231 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.232 DEBUG:boto:Params: {} 2022-09-12T19:43:57.232 DEBUG:boto:Token: None 2022-09-12T19:43:57.233 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/ 2022-09-12T19:43:57.234 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kQpDECX8kZmZfUFYqcSjY6YruHA= 2022-09-12T19:43:57.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kQpDECX8kZmZfUFYqcSjY6YruHA=', 'Content-Length': '0'} 2022-09-12T19:43:57.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026c35a572bb34d34-00631f8bfd-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.237 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:57.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:43:57.239 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:43:57.239 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:43:57.240 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:43:57.240 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:43:57.241 DEBUG:boto:Method: GET 2022-09-12T19:43:57.242 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:43:57.242 DEBUG:boto:Data: 2022-09-12T19:43:57.243 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.243 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:57.244 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.244 DEBUG:boto:Params: {} 2022-09-12T19:43:57.245 DEBUG:boto:Token: None 2022-09-12T19:43:57.246 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:43:57.246 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lhbdSrXE/zz0yXmFG2VNIXXJ8rY= 2022-09-12T19:43:57.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lhbdSrXE/zz0yXmFG2VNIXXJ8rY=', 'Content-Length': '0'} 2022-09-12T19:43:57.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017836c6d0c797846-00631f8bfd-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.250 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:43:57.251 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:43:57.251 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:43:57.252 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:43:57.252 DEBUG:boto:Method: GET 2022-09-12T19:43:57.253 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:43:57.254 DEBUG:boto:Data: 2022-09-12T19:43:57.254 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.255 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:57.255 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.256 DEBUG:boto:Params: {} 2022-09-12T19:43:57.257 DEBUG:boto:Token: None 2022-09-12T19:43:57.257 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:43:57.258 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lhbdSrXE/zz0yXmFG2VNIXXJ8rY= 2022-09-12T19:43:57.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lhbdSrXE/zz0yXmFG2VNIXXJ8rY=', 'Content-Length': '0'} 2022-09-12T19:43:57.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073e97540f87de6ed-00631f8bfd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.262 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:43:57.263 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:43:57.264 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:43:57.264 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:43:57.264 DEBUG:boto:Method: HEAD 2022-09-12T19:43:57.265 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:43:57.266 DEBUG:boto:Data: 2022-09-12T19:43:57.266 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.267 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:57.267 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.268 DEBUG:boto:Params: {} 2022-09-12T19:43:57.268 DEBUG:boto:Token: None 2022-09-12T19:43:57.269 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:43:57.269 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WcM8Vk1862P9HmL4WjAU3nno9e8= 2022-09-12T19:43:57.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WcM8Vk1862P9HmL4WjAU3nno9e8=', 'Content-Length': '0'} 2022-09-12T19:43:57.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e6200b0ceb01d29-00631f8bfd-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.273 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:43:57.274 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:43:57.275 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:43:57.275 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:43:57.276 DEBUG:boto:Method: HEAD 2022-09-12T19:43:57.276 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:43:57.277 DEBUG:boto:Data: 2022-09-12T19:43:57.278 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.279 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:57.280 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.281 DEBUG:boto:Params: {} 2022-09-12T19:43:57.282 DEBUG:boto:Token: None 2022-09-12T19:43:57.283 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:43:57.284 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WcM8Vk1862P9HmL4WjAU3nno9e8= 2022-09-12T19:43:57.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WcM8Vk1862P9HmL4WjAU3nno9e8=', 'Content-Length': '0'} 2022-09-12T19:43:57.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ee1f91c6fcbe8a7-00631f8bfd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:43:57.290 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:43:57.291 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:43:57.292 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:43:57.293 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:43:57.294 DEBUG:boto:Method: GET 2022-09-12T19:43:57.295 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:43:57.296 DEBUG:boto:Data: 2022-09-12T19:43:57.296 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.297 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:57.297 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.298 DEBUG:boto:Params: {} 2022-09-12T19:43:57.299 DEBUG:boto:Token: None 2022-09-12T19:43:57.299 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:43:57.300 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lhbdSrXE/zz0yXmFG2VNIXXJ8rY= 2022-09-12T19:43:57.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lhbdSrXE/zz0yXmFG2VNIXXJ8rY=', 'Content-Length': '0'} 2022-09-12T19:43:57.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076cc13d94527cb78-00631f8bfd-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.304 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:43:57.304 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:43:57.305 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:43:57.305 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:43:57.306 DEBUG:boto:Method: GET 2022-09-12T19:43:57.307 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:43:57.307 DEBUG:boto:Data: 2022-09-12T19:43:57.308 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.308 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:57.309 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.309 DEBUG:boto:Params: {} 2022-09-12T19:43:57.310 DEBUG:boto:Token: None 2022-09-12T19:43:57.311 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:43:57.311 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lhbdSrXE/zz0yXmFG2VNIXXJ8rY= 2022-09-12T19:43:57.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lhbdSrXE/zz0yXmFG2VNIXXJ8rY=', 'Content-Length': '0'} 2022-09-12T19:43:57.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000247936ba920bea1e-00631f8bfd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.316 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:57.319 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:57.320 DEBUG:boto:path=/jdpiak-71/?prefix=e 2022-09-12T19:43:57.321 DEBUG:boto:auth_path=/jdpiak-71/?prefix=e 2022-09-12T19:43:57.321 DEBUG:boto:Method: GET 2022-09-12T19:43:57.322 DEBUG:boto:Path: /jdpiak-71/?prefix=e 2022-09-12T19:43:57.323 DEBUG:boto:Data: 2022-09-12T19:43:57.324 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.324 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:57.325 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.326 DEBUG:boto:Params: {} 2022-09-12T19:43:57.331 DEBUG:boto:Token: None 2022-09-12T19:43:57.332 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/ 2022-09-12T19:43:57.332 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kQpDECX8kZmZfUFYqcSjY6YruHA= 2022-09-12T19:43:57.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kQpDECX8kZmZfUFYqcSjY6YruHA=', 'Content-Length': '0'} 2022-09-12T19:43:57.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d953bba32b3c572c-00631f8bfd-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.378 DEBUG:boto:b'jdpiak-71e1000falsee2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:57.379 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:57.380 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:57.380 DEBUG:boto:path=/jdpiak-71/?prefix=e 2022-09-12T19:43:57.381 DEBUG:boto:auth_path=/jdpiak-71/?prefix=e 2022-09-12T19:43:57.381 DEBUG:boto:Method: GET 2022-09-12T19:43:57.382 DEBUG:boto:Path: /jdpiak-71/?prefix=e 2022-09-12T19:43:57.382 DEBUG:boto:Data: 2022-09-12T19:43:57.383 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.383 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:57.384 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.385 DEBUG:boto:Params: {} 2022-09-12T19:43:57.385 DEBUG:boto:Token: None 2022-09-12T19:43:57.386 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/ 2022-09-12T19:43:57.387 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kQpDECX8kZmZfUFYqcSjY6YruHA= 2022-09-12T19:43:57.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kQpDECX8kZmZfUFYqcSjY6YruHA=', 'Content-Length': '0'} 2022-09-12T19:43:57.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009f2da17a5cd5d25-00631f8bfd-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.391 DEBUG:boto:b'jdpiak-71e1000falsee2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:57.392 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:43:57.392 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:43:57.393 DEBUG:boto:Method: HEAD 2022-09-12T19:43:57.393 DEBUG:boto:Path: /jdpiak-71/e 2022-09-12T19:43:57.394 DEBUG:boto:Data: 2022-09-12T19:43:57.394 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.395 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:57.396 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.396 DEBUG:boto:Params: {} 2022-09-12T19:43:57.397 DEBUG:boto:Token: None 2022-09-12T19:43:57.397 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/e 2022-09-12T19:43:57.398 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Vp6GU8Pu+qltxaMKAulN6o/O3IQ= 2022-09-12T19:43:57.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Vp6GU8Pu+qltxaMKAulN6o/O3IQ=', 'Content-Length': '0'} 2022-09-12T19:43:57.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000373987fd6738c3b8-00631f8bfd-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.405 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:43:57.406 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:43:57.407 DEBUG:boto:Method: HEAD 2022-09-12T19:43:57.408 DEBUG:boto:Path: /jdpiak-71/e 2022-09-12T19:43:57.410 DEBUG:boto:Data: 2022-09-12T19:43:57.412 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.413 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:57.414 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.416 DEBUG:boto:Params: {} 2022-09-12T19:43:57.417 DEBUG:boto:Token: None 2022-09-12T19:43:57.418 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/e 2022-09-12T19:43:57.419 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Vp6GU8Pu+qltxaMKAulN6o/O3IQ= 2022-09-12T19:43:57.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Vp6GU8Pu+qltxaMKAulN6o/O3IQ=', 'Content-Length': '0'} 2022-09-12T19:43:57.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abbaa1f724d6f3bc-00631f8bfd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:43:57.426 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:43:57.427 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:43:57.428 DEBUG:boto:Method: GET 2022-09-12T19:43:57.429 DEBUG:boto:Path: /jdpiak-71/e 2022-09-12T19:43:57.430 DEBUG:boto:Data: 2022-09-12T19:43:57.431 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.432 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:57.433 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.434 DEBUG:boto:Params: {} 2022-09-12T19:43:57.436 DEBUG:boto:Token: None 2022-09-12T19:43:57.437 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/e 2022-09-12T19:43:57.438 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2yolHXrsdWO3+H1/h8f+cpmpGfs= 2022-09-12T19:43:57.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2yolHXrsdWO3+H1/h8f+cpmpGfs=', 'Content-Length': '0'} 2022-09-12T19:43:57.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000835b9b1758d9ebed-00631f8bfd-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.443 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:43:57.444 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:43:57.445 DEBUG:boto:Method: GET 2022-09-12T19:43:57.447 DEBUG:boto:Path: /jdpiak-71/e 2022-09-12T19:43:57.448 DEBUG:boto:Data: 2022-09-12T19:43:57.449 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.451 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:57.452 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.453 DEBUG:boto:Params: {} 2022-09-12T19:43:57.454 DEBUG:boto:Token: None 2022-09-12T19:43:57.455 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/e 2022-09-12T19:43:57.457 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2yolHXrsdWO3+H1/h8f+cpmpGfs= 2022-09-12T19:43:57.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2yolHXrsdWO3+H1/h8f+cpmpGfs=', 'Content-Length': '0'} 2022-09-12T19:43:57.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004ff4d57166026ec-00631f8bfd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.462 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:57.463 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:57.464 DEBUG:boto:path=/jdpiak-71/?prefix=e 2022-09-12T19:43:57.464 DEBUG:boto:auth_path=/jdpiak-71/?prefix=e 2022-09-12T19:43:57.465 DEBUG:boto:Method: GET 2022-09-12T19:43:57.466 DEBUG:boto:Path: /jdpiak-71/?prefix=e 2022-09-12T19:43:57.467 DEBUG:boto:Data: 2022-09-12T19:43:57.467 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.468 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:57.469 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.482 DEBUG:boto:Params: {} 2022-09-12T19:43:57.483 DEBUG:boto:Token: None 2022-09-12T19:43:57.484 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/ 2022-09-12T19:43:57.484 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kQpDECX8kZmZfUFYqcSjY6YruHA= 2022-09-12T19:43:57.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kQpDECX8kZmZfUFYqcSjY6YruHA=', 'Content-Length': '0'} 2022-09-12T19:43:57.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e50208863cff3c2-00631f8bfd-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.492 DEBUG:boto:b'jdpiak-71e1000falsee2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:57.493 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:57.494 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:57.495 DEBUG:boto:path=/jdpiak-71/?prefix=e 2022-09-12T19:43:57.496 DEBUG:boto:auth_path=/jdpiak-71/?prefix=e 2022-09-12T19:43:57.497 DEBUG:boto:Method: GET 2022-09-12T19:43:57.498 DEBUG:boto:Path: /jdpiak-71/?prefix=e 2022-09-12T19:43:57.499 DEBUG:boto:Data: 2022-09-12T19:43:57.500 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.501 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:57.501 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.502 DEBUG:boto:Params: {} 2022-09-12T19:43:57.503 DEBUG:boto:Token: None 2022-09-12T19:43:57.503 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/ 2022-09-12T19:43:57.504 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kQpDECX8kZmZfUFYqcSjY6YruHA= 2022-09-12T19:43:57.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kQpDECX8kZmZfUFYqcSjY6YruHA=', 'Content-Length': '0'} 2022-09-12T19:43:57.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005db54b3d5be99efe-00631f8bfd-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.513 DEBUG:boto:b'jdpiak-71e1000falsee2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:57.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:43:57.516 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:43:57.517 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:43:57.517 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:43:57.518 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:43:57.519 DEBUG:boto:Method: GET 2022-09-12T19:43:57.520 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:43:57.521 DEBUG:boto:Data: 2022-09-12T19:43:57.522 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.523 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:57.524 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.525 DEBUG:boto:Params: {} 2022-09-12T19:43:57.526 DEBUG:boto:Token: None 2022-09-12T19:43:57.527 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:43:57.528 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CGnH1aEDWhwfptub41wCCYKOmMo= 2022-09-12T19:43:57.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CGnH1aEDWhwfptub41wCCYKOmMo=', 'Content-Length': '0'} 2022-09-12T19:43:57.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2ed8f2648efdff2-00631f8bfd-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.578 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:43:57.579 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:43:57.580 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:43:57.581 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:43:57.581 DEBUG:boto:Method: GET 2022-09-12T19:43:57.582 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:43:57.583 DEBUG:boto:Data: 2022-09-12T19:43:57.583 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.584 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:57.585 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.587 DEBUG:boto:Params: {} 2022-09-12T19:43:57.588 DEBUG:boto:Token: None 2022-09-12T19:43:57.590 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:43:57.591 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CGnH1aEDWhwfptub41wCCYKOmMo= 2022-09-12T19:43:57.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CGnH1aEDWhwfptub41wCCYKOmMo=', 'Content-Length': '0'} 2022-09-12T19:43:57.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c3f0cd5cdcb446c-00631f8bfd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.599 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:43:57.600 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:43:57.601 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:43:57.602 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:43:57.603 DEBUG:boto:Method: HEAD 2022-09-12T19:43:57.604 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:43:57.605 DEBUG:boto:Data: 2022-09-12T19:43:57.606 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.607 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:57.608 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.611 DEBUG:boto:Params: {} 2022-09-12T19:43:57.612 DEBUG:boto:Token: None 2022-09-12T19:43:57.613 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:43:57.615 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YYZA7wL5+bb8jsXLTck0hWYq5is= 2022-09-12T19:43:57.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YYZA7wL5+bb8jsXLTck0hWYq5is=', 'Content-Length': '0'} 2022-09-12T19:43:57.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cdc173b17839ff5-00631f8bfd-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.620 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:43:57.621 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:43:57.622 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:43:57.623 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:43:57.624 DEBUG:boto:Method: HEAD 2022-09-12T19:43:57.625 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:43:57.626 DEBUG:boto:Data: 2022-09-12T19:43:57.627 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.628 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:57.628 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.629 DEBUG:boto:Params: {} 2022-09-12T19:43:57.629 DEBUG:boto:Token: None 2022-09-12T19:43:57.630 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:43:57.631 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YYZA7wL5+bb8jsXLTck0hWYq5is= 2022-09-12T19:43:57.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YYZA7wL5+bb8jsXLTck0hWYq5is=', 'Content-Length': '0'} 2022-09-12T19:43:57.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f85c1b826cb78c1c-00631f8bfd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:43:57.635 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:43:57.635 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:43:57.636 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:43:57.637 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:43:57.637 DEBUG:boto:Method: GET 2022-09-12T19:43:57.638 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:43:57.638 DEBUG:boto:Data: 2022-09-12T19:43:57.639 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.639 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:57.640 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.640 DEBUG:boto:Params: {} 2022-09-12T19:43:57.641 DEBUG:boto:Token: None 2022-09-12T19:43:57.642 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:43:57.642 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CGnH1aEDWhwfptub41wCCYKOmMo= 2022-09-12T19:43:57.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CGnH1aEDWhwfptub41wCCYKOmMo=', 'Content-Length': '0'} 2022-09-12T19:43:57.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e594f7df97eac2bb-00631f8bfd-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.650 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:43:57.651 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:43:57.651 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:43:57.652 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:43:57.652 DEBUG:boto:Method: GET 2022-09-12T19:43:57.653 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:43:57.654 DEBUG:boto:Data: 2022-09-12T19:43:57.654 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.655 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:57.655 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.656 DEBUG:boto:Params: {} 2022-09-12T19:43:57.656 DEBUG:boto:Token: None 2022-09-12T19:43:57.657 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:43:57.658 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CGnH1aEDWhwfptub41wCCYKOmMo= 2022-09-12T19:43:57.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CGnH1aEDWhwfptub41wCCYKOmMo=', 'Content-Length': '0'} 2022-09-12T19:43:57.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000580111637c365220-00631f8bfd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.661 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:57.662 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:57.662 DEBUG:boto:path=/jdpiak-71/?prefix=f 2022-09-12T19:43:57.663 DEBUG:boto:auth_path=/jdpiak-71/?prefix=f 2022-09-12T19:43:57.664 DEBUG:boto:Method: GET 2022-09-12T19:43:57.664 DEBUG:boto:Path: /jdpiak-71/?prefix=f 2022-09-12T19:43:57.665 DEBUG:boto:Data: 2022-09-12T19:43:57.665 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.666 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:57.666 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.667 DEBUG:boto:Params: {} 2022-09-12T19:43:57.667 DEBUG:boto:Token: None 2022-09-12T19:43:57.668 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/ 2022-09-12T19:43:57.669 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kQpDECX8kZmZfUFYqcSjY6YruHA= 2022-09-12T19:43:57.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kQpDECX8kZmZfUFYqcSjY6YruHA=', 'Content-Length': '0'} 2022-09-12T19:43:57.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cacb88a945c75c5a-00631f8bfd-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.714 DEBUG:boto:b'jdpiak-71f1000falsef2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:57.716 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:57.717 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:57.719 DEBUG:boto:path=/jdpiak-71/?prefix=f 2022-09-12T19:43:57.720 DEBUG:boto:auth_path=/jdpiak-71/?prefix=f 2022-09-12T19:43:57.720 DEBUG:boto:Method: GET 2022-09-12T19:43:57.721 DEBUG:boto:Path: /jdpiak-71/?prefix=f 2022-09-12T19:43:57.723 DEBUG:boto:Data: 2022-09-12T19:43:57.724 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.725 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:57.726 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.728 DEBUG:boto:Params: {} 2022-09-12T19:43:57.730 DEBUG:boto:Token: None 2022-09-12T19:43:57.731 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/ 2022-09-12T19:43:57.732 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kQpDECX8kZmZfUFYqcSjY6YruHA= 2022-09-12T19:43:57.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kQpDECX8kZmZfUFYqcSjY6YruHA=', 'Content-Length': '0'} 2022-09-12T19:43:57.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003668fc9bf39d9433-00631f8bfd-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.739 DEBUG:boto:b'jdpiak-71f1000falsef2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:57.741 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:43:57.742 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:43:57.743 DEBUG:boto:Method: HEAD 2022-09-12T19:43:57.743 DEBUG:boto:Path: /jdpiak-71/f 2022-09-12T19:43:57.744 DEBUG:boto:Data: 2022-09-12T19:43:57.745 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.745 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:57.746 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.747 DEBUG:boto:Params: {} 2022-09-12T19:43:57.747 DEBUG:boto:Token: None 2022-09-12T19:43:57.748 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/f 2022-09-12T19:43:57.749 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AcNjUDMXaCiIIfqKhtWN/22UdTc= 2022-09-12T19:43:57.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AcNjUDMXaCiIIfqKhtWN/22UdTc=', 'Content-Length': '0'} 2022-09-12T19:43:57.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d16113c1332ee00b-00631f8bfd-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.753 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:43:57.753 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:43:57.754 DEBUG:boto:Method: HEAD 2022-09-12T19:43:57.754 DEBUG:boto:Path: /jdpiak-71/f 2022-09-12T19:43:57.755 DEBUG:boto:Data: 2022-09-12T19:43:57.756 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.756 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:57.757 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.758 DEBUG:boto:Params: {} 2022-09-12T19:43:57.759 DEBUG:boto:Token: None 2022-09-12T19:43:57.760 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/f 2022-09-12T19:43:57.761 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AcNjUDMXaCiIIfqKhtWN/22UdTc= 2022-09-12T19:43:57.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AcNjUDMXaCiIIfqKhtWN/22UdTc=', 'Content-Length': '0'} 2022-09-12T19:43:57.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069590750827c0312-00631f8bfd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:43:57.768 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:43:57.769 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:43:57.770 DEBUG:boto:Method: GET 2022-09-12T19:43:57.771 DEBUG:boto:Path: /jdpiak-71/f 2022-09-12T19:43:57.772 DEBUG:boto:Data: 2022-09-12T19:43:57.773 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.774 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:57.775 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.776 DEBUG:boto:Params: {} 2022-09-12T19:43:57.777 DEBUG:boto:Token: None 2022-09-12T19:43:57.778 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/f 2022-09-12T19:43:57.779 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JIn2QIo+p16AaIYhY2IQ5jE7vjM= 2022-09-12T19:43:57.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JIn2QIo+p16AaIYhY2IQ5jE7vjM=', 'Content-Length': '0'} 2022-09-12T19:43:57.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014f26b1d9ae36629-00631f8bfd-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.798 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:43:57.798 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:43:57.799 DEBUG:boto:Method: GET 2022-09-12T19:43:57.801 DEBUG:boto:Path: /jdpiak-71/f 2022-09-12T19:43:57.802 DEBUG:boto:Data: 2022-09-12T19:43:57.802 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.807 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:57.808 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.808 DEBUG:boto:Params: {} 2022-09-12T19:43:57.809 DEBUG:boto:Token: None 2022-09-12T19:43:57.810 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/f 2022-09-12T19:43:57.811 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JIn2QIo+p16AaIYhY2IQ5jE7vjM= 2022-09-12T19:43:57.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JIn2QIo+p16AaIYhY2IQ5jE7vjM=', 'Content-Length': '0'} 2022-09-12T19:43:57.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000170c711587770a01-00631f8bfd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.815 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:57.817 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:57.818 DEBUG:boto:path=/jdpiak-71/?prefix=f 2022-09-12T19:43:57.819 DEBUG:boto:auth_path=/jdpiak-71/?prefix=f 2022-09-12T19:43:57.819 DEBUG:boto:Method: GET 2022-09-12T19:43:57.820 DEBUG:boto:Path: /jdpiak-71/?prefix=f 2022-09-12T19:43:57.820 DEBUG:boto:Data: 2022-09-12T19:43:57.821 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.821 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:57.822 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.823 DEBUG:boto:Params: {} 2022-09-12T19:43:57.823 DEBUG:boto:Token: None 2022-09-12T19:43:57.824 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/ 2022-09-12T19:43:57.825 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kQpDECX8kZmZfUFYqcSjY6YruHA= 2022-09-12T19:43:57.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kQpDECX8kZmZfUFYqcSjY6YruHA=', 'Content-Length': '0'} 2022-09-12T19:43:57.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb4f59f9dffe514c-00631f8bfd-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.832 DEBUG:boto:b'jdpiak-71f1000falsef2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:57.834 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:57.834 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:57.835 DEBUG:boto:path=/jdpiak-71/?prefix=f 2022-09-12T19:43:57.835 DEBUG:boto:auth_path=/jdpiak-71/?prefix=f 2022-09-12T19:43:57.836 DEBUG:boto:Method: GET 2022-09-12T19:43:57.837 DEBUG:boto:Path: /jdpiak-71/?prefix=f 2022-09-12T19:43:57.837 DEBUG:boto:Data: 2022-09-12T19:43:57.838 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.839 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:57.839 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.840 DEBUG:boto:Params: {} 2022-09-12T19:43:57.841 DEBUG:boto:Token: None 2022-09-12T19:43:57.841 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/ 2022-09-12T19:43:57.842 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kQpDECX8kZmZfUFYqcSjY6YruHA= 2022-09-12T19:43:57.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kQpDECX8kZmZfUFYqcSjY6YruHA=', 'Content-Length': '0'} 2022-09-12T19:43:57.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b65ed01d3bea0948-00631f8bfd-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.846 DEBUG:boto:b'jdpiak-71f1000falsef2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:57.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:43:57.848 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:43:57.848 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:43:57.849 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:43:57.850 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:43:57.851 DEBUG:boto:Method: GET 2022-09-12T19:43:57.852 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:43:57.853 DEBUG:boto:Data: 2022-09-12T19:43:57.853 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.854 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:57.854 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.855 DEBUG:boto:Params: {} 2022-09-12T19:43:57.855 DEBUG:boto:Token: None 2022-09-12T19:43:57.856 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:43:57.857 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:K0UCE5zOtcHzvvNILTTy4GDxhI4= 2022-09-12T19:43:57.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:K0UCE5zOtcHzvvNILTTy4GDxhI4=', 'Content-Length': '0'} 2022-09-12T19:43:57.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007da04d39d7dcc9de-00631f8bfd-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.903 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:43:57.904 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:43:57.904 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:43:57.905 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:43:57.906 DEBUG:boto:Method: GET 2022-09-12T19:43:57.906 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:43:57.907 DEBUG:boto:Data: 2022-09-12T19:43:57.911 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.911 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:57.912 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.916 DEBUG:boto:Params: {} 2022-09-12T19:43:57.917 DEBUG:boto:Token: None 2022-09-12T19:43:57.918 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:43:57.918 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:K0UCE5zOtcHzvvNILTTy4GDxhI4= 2022-09-12T19:43:57.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:K0UCE5zOtcHzvvNILTTy4GDxhI4=', 'Content-Length': '0'} 2022-09-12T19:43:57.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b20be2e50c8d8903-00631f8bfd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.927 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:43:57.928 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:43:57.928 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:43:57.929 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:43:57.929 DEBUG:boto:Method: HEAD 2022-09-12T19:43:57.937 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:43:57.937 DEBUG:boto:Data: 2022-09-12T19:43:57.938 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.938 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:57.939 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.939 DEBUG:boto:Params: {} 2022-09-12T19:43:57.940 DEBUG:boto:Token: None 2022-09-12T19:43:57.941 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:43:57.941 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bbpK5qWfXcn1uI1nMx6LobWDXQU= 2022-09-12T19:43:57.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bbpK5qWfXcn1uI1nMx6LobWDXQU=', 'Content-Length': '0'} 2022-09-12T19:43:57.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000512c3dc11a67872-00631f8bfd-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.958 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:43:57.959 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:43:57.959 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:43:57.960 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:43:57.960 DEBUG:boto:Method: HEAD 2022-09-12T19:43:57.961 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:43:57.961 DEBUG:boto:Data: 2022-09-12T19:43:57.962 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.962 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:57.963 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.964 DEBUG:boto:Params: {} 2022-09-12T19:43:57.964 DEBUG:boto:Token: None 2022-09-12T19:43:57.965 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:43:57.965 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bbpK5qWfXcn1uI1nMx6LobWDXQU= 2022-09-12T19:43:57.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bbpK5qWfXcn1uI1nMx6LobWDXQU=', 'Content-Length': '0'} 2022-09-12T19:43:57.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097de3e532024e44f-00631f8bfd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:43:57.971 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:43:57.972 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:43:57.972 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:43:57.973 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:43:57.973 DEBUG:boto:Method: GET 2022-09-12T19:43:57.974 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:43:57.975 DEBUG:boto:Data: 2022-09-12T19:43:57.975 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.976 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:57.977 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.977 DEBUG:boto:Params: {} 2022-09-12T19:43:57.982 DEBUG:boto:Token: None 2022-09-12T19:43:57.983 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:43:57.984 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:K0UCE5zOtcHzvvNILTTy4GDxhI4= 2022-09-12T19:43:57.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:K0UCE5zOtcHzvvNILTTy4GDxhI4=', 'Content-Length': '0'} 2022-09-12T19:43:57.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005951034e461d40c-00631f8bfd-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:57.990 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:43:57.991 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:43:57.991 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:43:57.992 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:43:57.992 DEBUG:boto:Method: GET 2022-09-12T19:43:57.993 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:43:57.994 DEBUG:boto:Data: 2022-09-12T19:43:57.994 DEBUG:boto:Headers: {} 2022-09-12T19:43:57.995 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:57.995 DEBUG:boto:Port: 8000 2022-09-12T19:43:57.996 DEBUG:boto:Params: {} 2022-09-12T19:43:57.997 DEBUG:boto:Token: None 2022-09-12T19:43:57.997 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:57 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:43:57.998 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:K0UCE5zOtcHzvvNILTTy4GDxhI4= 2022-09-12T19:43:57.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:K0UCE5zOtcHzvvNILTTy4GDxhI4=', 'Content-Length': '0'} 2022-09-12T19:43:58.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094c34fec725d83ff-00631f8bfd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.007 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:58.007 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:58.008 DEBUG:boto:path=/jdpiak-71/?prefix=g 2022-09-12T19:43:58.008 DEBUG:boto:auth_path=/jdpiak-71/?prefix=g 2022-09-12T19:43:58.009 DEBUG:boto:Method: GET 2022-09-12T19:43:58.009 DEBUG:boto:Path: /jdpiak-71/?prefix=g 2022-09-12T19:43:58.010 DEBUG:boto:Data: 2022-09-12T19:43:58.011 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.011 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:58.012 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.012 DEBUG:boto:Params: {} 2022-09-12T19:43:58.013 DEBUG:boto:Token: None 2022-09-12T19:43:58.014 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/ 2022-09-12T19:43:58.027 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc= 2022-09-12T19:43:58.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc=', 'Content-Length': '0'} 2022-09-12T19:43:58.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f71c81860488d257-00631f8bfe-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.034 DEBUG:boto:b'jdpiak-71g1000falseg2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:58.035 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:58.036 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:58.036 DEBUG:boto:path=/jdpiak-71/?prefix=g 2022-09-12T19:43:58.037 DEBUG:boto:auth_path=/jdpiak-71/?prefix=g 2022-09-12T19:43:58.038 DEBUG:boto:Method: GET 2022-09-12T19:43:58.038 DEBUG:boto:Path: /jdpiak-71/?prefix=g 2022-09-12T19:43:58.039 DEBUG:boto:Data: 2022-09-12T19:43:58.040 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.040 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:58.041 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.042 DEBUG:boto:Params: {} 2022-09-12T19:43:58.042 DEBUG:boto:Token: None 2022-09-12T19:43:58.043 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/ 2022-09-12T19:43:58.044 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc= 2022-09-12T19:43:58.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc=', 'Content-Length': '0'} 2022-09-12T19:43:58.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ded07fa519bdb08-00631f8bfe-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.048 DEBUG:boto:b'jdpiak-71g1000falseg2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:58.048 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:43:58.049 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:43:58.050 DEBUG:boto:Method: HEAD 2022-09-12T19:43:58.050 DEBUG:boto:Path: /jdpiak-71/g 2022-09-12T19:43:58.051 DEBUG:boto:Data: 2022-09-12T19:43:58.051 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.052 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:58.053 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.053 DEBUG:boto:Params: {} 2022-09-12T19:43:58.054 DEBUG:boto:Token: None 2022-09-12T19:43:58.054 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/g 2022-09-12T19:43:58.055 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BmgfaJYZmLlcnXdEd/ZXfoe5bw8= 2022-09-12T19:43:58.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BmgfaJYZmLlcnXdEd/ZXfoe5bw8=', 'Content-Length': '0'} 2022-09-12T19:43:58.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000781837671b5b7c33-00631f8bfe-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.059 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:43:58.060 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:43:58.061 DEBUG:boto:Method: HEAD 2022-09-12T19:43:58.061 DEBUG:boto:Path: /jdpiak-71/g 2022-09-12T19:43:58.062 DEBUG:boto:Data: 2022-09-12T19:43:58.062 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.063 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:58.063 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.064 DEBUG:boto:Params: {} 2022-09-12T19:43:58.064 DEBUG:boto:Token: None 2022-09-12T19:43:58.065 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/g 2022-09-12T19:43:58.066 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BmgfaJYZmLlcnXdEd/ZXfoe5bw8= 2022-09-12T19:43:58.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BmgfaJYZmLlcnXdEd/ZXfoe5bw8=', 'Content-Length': '0'} 2022-09-12T19:43:58.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042d1fcd84516e992-00631f8bfe-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:43:58.071 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:43:58.071 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:43:58.072 DEBUG:boto:Method: GET 2022-09-12T19:43:58.073 DEBUG:boto:Path: /jdpiak-71/g 2022-09-12T19:43:58.073 DEBUG:boto:Data: 2022-09-12T19:43:58.074 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.074 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:58.075 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.076 DEBUG:boto:Params: {} 2022-09-12T19:43:58.077 DEBUG:boto:Token: None 2022-09-12T19:43:58.078 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/g 2022-09-12T19:43:58.078 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EyPeivIYqem/sL/Hz6LHpKMDkR0= 2022-09-12T19:43:58.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EyPeivIYqem/sL/Hz6LHpKMDkR0=', 'Content-Length': '0'} 2022-09-12T19:43:58.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004839c9eef30afcc4-00631f8bfe-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.122 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:43:58.123 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:43:58.124 DEBUG:boto:Method: GET 2022-09-12T19:43:58.124 DEBUG:boto:Path: /jdpiak-71/g 2022-09-12T19:43:58.125 DEBUG:boto:Data: 2022-09-12T19:43:58.125 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.126 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:58.126 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.127 DEBUG:boto:Params: {} 2022-09-12T19:43:58.128 DEBUG:boto:Token: None 2022-09-12T19:43:58.129 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/g 2022-09-12T19:43:58.129 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EyPeivIYqem/sL/Hz6LHpKMDkR0= 2022-09-12T19:43:58.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EyPeivIYqem/sL/Hz6LHpKMDkR0=', 'Content-Length': '0'} 2022-09-12T19:43:58.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059a2fac64f71736b-00631f8bfe-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.134 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:58.134 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:58.135 DEBUG:boto:path=/jdpiak-71/?prefix=g 2022-09-12T19:43:58.136 DEBUG:boto:auth_path=/jdpiak-71/?prefix=g 2022-09-12T19:43:58.136 DEBUG:boto:Method: GET 2022-09-12T19:43:58.137 DEBUG:boto:Path: /jdpiak-71/?prefix=g 2022-09-12T19:43:58.138 DEBUG:boto:Data: 2022-09-12T19:43:58.138 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.139 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:58.140 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.140 DEBUG:boto:Params: {} 2022-09-12T19:43:58.141 DEBUG:boto:Token: None 2022-09-12T19:43:58.142 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/ 2022-09-12T19:43:58.142 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc= 2022-09-12T19:43:58.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc=', 'Content-Length': '0'} 2022-09-12T19:43:58.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055cb1900ba821be3-00631f8bfe-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.148 DEBUG:boto:b'jdpiak-71g1000falseg2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:58.149 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:58.150 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:58.151 DEBUG:boto:path=/jdpiak-71/?prefix=g 2022-09-12T19:43:58.154 DEBUG:boto:auth_path=/jdpiak-71/?prefix=g 2022-09-12T19:43:58.156 DEBUG:boto:Method: GET 2022-09-12T19:43:58.158 DEBUG:boto:Path: /jdpiak-71/?prefix=g 2022-09-12T19:43:58.159 DEBUG:boto:Data: 2022-09-12T19:43:58.159 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.160 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:58.161 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.161 DEBUG:boto:Params: {} 2022-09-12T19:43:58.162 DEBUG:boto:Token: None 2022-09-12T19:43:58.163 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/ 2022-09-12T19:43:58.164 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc= 2022-09-12T19:43:58.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc=', 'Content-Length': '0'} 2022-09-12T19:43:58.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060709ca580b7a381-00631f8bfe-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.168 DEBUG:boto:b'jdpiak-71g1000falseg2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:58.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:43:58.169 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:43:58.170 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:43:58.171 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:43:58.171 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:43:58.172 DEBUG:boto:Method: GET 2022-09-12T19:43:58.173 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:43:58.174 DEBUG:boto:Data: 2022-09-12T19:43:58.174 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.175 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:58.175 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.176 DEBUG:boto:Params: {} 2022-09-12T19:43:58.177 DEBUG:boto:Token: None 2022-09-12T19:43:58.177 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:43:58.178 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2XTGrmYjnFaYbFXcUmZPNSgr9Tw= 2022-09-12T19:43:58.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2XTGrmYjnFaYbFXcUmZPNSgr9Tw=', 'Content-Length': '0'} 2022-09-12T19:43:58.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b7034c3f9703c82-00631f8bfe-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.182 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:43:58.183 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:43:58.183 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:43:58.184 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:43:58.184 DEBUG:boto:Method: GET 2022-09-12T19:43:58.185 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:43:58.185 DEBUG:boto:Data: 2022-09-12T19:43:58.186 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.187 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:58.187 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.188 DEBUG:boto:Params: {} 2022-09-12T19:43:58.189 DEBUG:boto:Token: None 2022-09-12T19:43:58.190 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:43:58.190 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2XTGrmYjnFaYbFXcUmZPNSgr9Tw= 2022-09-12T19:43:58.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2XTGrmYjnFaYbFXcUmZPNSgr9Tw=', 'Content-Length': '0'} 2022-09-12T19:43:58.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f900d6c268276d45-00631f8bfe-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.194 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:43:58.195 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:43:58.196 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:43:58.196 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:43:58.197 DEBUG:boto:Method: HEAD 2022-09-12T19:43:58.198 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:43:58.198 DEBUG:boto:Data: 2022-09-12T19:43:58.199 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.199 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:58.200 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.200 DEBUG:boto:Params: {} 2022-09-12T19:43:58.201 DEBUG:boto:Token: None 2022-09-12T19:43:58.202 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:43:58.202 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MvGq2IT7fby84bWlkujYZLzZ6Q0= 2022-09-12T19:43:58.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MvGq2IT7fby84bWlkujYZLzZ6Q0=', 'Content-Length': '0'} 2022-09-12T19:43:58.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bc55d3e65c43d4a-00631f8bfe-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.206 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:43:58.207 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:43:58.207 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:43:58.208 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:43:58.208 DEBUG:boto:Method: HEAD 2022-09-12T19:43:58.209 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:43:58.209 DEBUG:boto:Data: 2022-09-12T19:43:58.210 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.210 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:58.211 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.211 DEBUG:boto:Params: {} 2022-09-12T19:43:58.212 DEBUG:boto:Token: None 2022-09-12T19:43:58.213 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:43:58.214 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MvGq2IT7fby84bWlkujYZLzZ6Q0= 2022-09-12T19:43:58.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MvGq2IT7fby84bWlkujYZLzZ6Q0=', 'Content-Length': '0'} 2022-09-12T19:43:58.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d482c9065ac398c-00631f8bfe-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:43:58.218 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:43:58.219 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:43:58.219 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:43:58.220 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:43:58.220 DEBUG:boto:Method: GET 2022-09-12T19:43:58.221 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:43:58.221 DEBUG:boto:Data: 2022-09-12T19:43:58.222 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.222 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:58.223 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.223 DEBUG:boto:Params: {} 2022-09-12T19:43:58.225 DEBUG:boto:Token: None 2022-09-12T19:43:58.225 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:43:58.226 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2XTGrmYjnFaYbFXcUmZPNSgr9Tw= 2022-09-12T19:43:58.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2XTGrmYjnFaYbFXcUmZPNSgr9Tw=', 'Content-Length': '0'} 2022-09-12T19:43:58.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e75a303306bfc72f-00631f8bfe-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.230 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:43:58.231 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:43:58.232 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:43:58.232 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:43:58.233 DEBUG:boto:Method: GET 2022-09-12T19:43:58.233 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:43:58.234 DEBUG:boto:Data: 2022-09-12T19:43:58.235 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.236 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:58.236 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.237 DEBUG:boto:Params: {} 2022-09-12T19:43:58.238 DEBUG:boto:Token: None 2022-09-12T19:43:58.238 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:43:58.239 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2XTGrmYjnFaYbFXcUmZPNSgr9Tw= 2022-09-12T19:43:58.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2XTGrmYjnFaYbFXcUmZPNSgr9Tw=', 'Content-Length': '0'} 2022-09-12T19:43:58.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000308166173ca5fcb2-00631f8bfe-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.243 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:58.244 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:58.244 DEBUG:boto:path=/jdpiak-71/?prefix=h 2022-09-12T19:43:58.245 DEBUG:boto:auth_path=/jdpiak-71/?prefix=h 2022-09-12T19:43:58.246 DEBUG:boto:Method: GET 2022-09-12T19:43:58.246 DEBUG:boto:Path: /jdpiak-71/?prefix=h 2022-09-12T19:43:58.247 DEBUG:boto:Data: 2022-09-12T19:43:58.248 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.248 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:58.249 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.249 DEBUG:boto:Params: {} 2022-09-12T19:43:58.250 DEBUG:boto:Token: None 2022-09-12T19:43:58.250 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/ 2022-09-12T19:43:58.251 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc= 2022-09-12T19:43:58.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc=', 'Content-Length': '0'} 2022-09-12T19:43:58.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006feb990ce96d9fcd-00631f8bfe-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.255 DEBUG:boto:b'jdpiak-71h1000falseh2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:58.257 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:58.258 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:58.259 DEBUG:boto:path=/jdpiak-71/?prefix=h 2022-09-12T19:43:58.260 DEBUG:boto:auth_path=/jdpiak-71/?prefix=h 2022-09-12T19:43:58.261 DEBUG:boto:Method: GET 2022-09-12T19:43:58.263 DEBUG:boto:Path: /jdpiak-71/?prefix=h 2022-09-12T19:43:58.264 DEBUG:boto:Data: 2022-09-12T19:43:58.265 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.266 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:58.267 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.268 DEBUG:boto:Params: {} 2022-09-12T19:43:58.270 DEBUG:boto:Token: None 2022-09-12T19:43:58.271 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/ 2022-09-12T19:43:58.272 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc= 2022-09-12T19:43:58.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc=', 'Content-Length': '0'} 2022-09-12T19:43:58.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013c1efd873cb73ff-00631f8bfe-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.279 DEBUG:boto:b'jdpiak-71h1000falseh2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:58.280 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:43:58.281 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:43:58.281 DEBUG:boto:Method: HEAD 2022-09-12T19:43:58.283 DEBUG:boto:Path: /jdpiak-71/h 2022-09-12T19:43:58.284 DEBUG:boto:Data: 2022-09-12T19:43:58.284 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.285 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:58.285 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.286 DEBUG:boto:Params: {} 2022-09-12T19:43:58.286 DEBUG:boto:Token: None 2022-09-12T19:43:58.287 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/h 2022-09-12T19:43:58.288 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4/3PpzFzxuMn/SohOlO9sIhxUYY= 2022-09-12T19:43:58.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4/3PpzFzxuMn/SohOlO9sIhxUYY=', 'Content-Length': '0'} 2022-09-12T19:43:58.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013719746bf3f651e-00631f8bfe-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.291 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:43:58.292 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:43:58.292 DEBUG:boto:Method: HEAD 2022-09-12T19:43:58.293 DEBUG:boto:Path: /jdpiak-71/h 2022-09-12T19:43:58.293 DEBUG:boto:Data: 2022-09-12T19:43:58.294 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.294 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:58.295 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.295 DEBUG:boto:Params: {} 2022-09-12T19:43:58.296 DEBUG:boto:Token: None 2022-09-12T19:43:58.297 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/h 2022-09-12T19:43:58.297 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4/3PpzFzxuMn/SohOlO9sIhxUYY= 2022-09-12T19:43:58.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4/3PpzFzxuMn/SohOlO9sIhxUYY=', 'Content-Length': '0'} 2022-09-12T19:43:58.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0a400507b940b25-00631f8bfe-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:43:58.301 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:43:58.302 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:43:58.303 DEBUG:boto:Method: GET 2022-09-12T19:43:58.303 DEBUG:boto:Path: /jdpiak-71/h 2022-09-12T19:43:58.304 DEBUG:boto:Data: 2022-09-12T19:43:58.304 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.305 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:58.305 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.306 DEBUG:boto:Params: {} 2022-09-12T19:43:58.307 DEBUG:boto:Token: None 2022-09-12T19:43:58.307 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/h 2022-09-12T19:43:58.308 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bxkG4J6uuou3V6H6C+4eepaYYT4= 2022-09-12T19:43:58.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bxkG4J6uuou3V6H6C+4eepaYYT4=', 'Content-Length': '0'} 2022-09-12T19:43:58.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001b7af95fdf2c6c5-00631f8bfe-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.354 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:43:58.355 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:43:58.356 DEBUG:boto:Method: GET 2022-09-12T19:43:58.356 DEBUG:boto:Path: /jdpiak-71/h 2022-09-12T19:43:58.357 DEBUG:boto:Data: 2022-09-12T19:43:58.357 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.358 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:58.358 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.359 DEBUG:boto:Params: {} 2022-09-12T19:43:58.359 DEBUG:boto:Token: None 2022-09-12T19:43:58.360 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/h 2022-09-12T19:43:58.362 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bxkG4J6uuou3V6H6C+4eepaYYT4= 2022-09-12T19:43:58.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bxkG4J6uuou3V6H6C+4eepaYYT4=', 'Content-Length': '0'} 2022-09-12T19:43:58.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7149899d60d6170-00631f8bfe-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.366 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:58.367 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:58.367 DEBUG:boto:path=/jdpiak-71/?prefix=h 2022-09-12T19:43:58.368 DEBUG:boto:auth_path=/jdpiak-71/?prefix=h 2022-09-12T19:43:58.368 DEBUG:boto:Method: GET 2022-09-12T19:43:58.369 DEBUG:boto:Path: /jdpiak-71/?prefix=h 2022-09-12T19:43:58.369 DEBUG:boto:Data: 2022-09-12T19:43:58.370 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.371 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:43:58.372 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.373 DEBUG:boto:Params: {} 2022-09-12T19:43:58.373 DEBUG:boto:Token: None 2022-09-12T19:43:58.374 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/ 2022-09-12T19:43:58.375 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc= 2022-09-12T19:43:58.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc=', 'Content-Length': '0'} 2022-09-12T19:43:58.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095d324be5ad87fca-00631f8bfe-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.379 DEBUG:boto:b'jdpiak-71h1000falseh2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:58.380 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:58.382 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:58.383 DEBUG:boto:path=/jdpiak-71/?prefix=h 2022-09-12T19:43:58.384 DEBUG:boto:auth_path=/jdpiak-71/?prefix=h 2022-09-12T19:43:58.385 DEBUG:boto:Method: GET 2022-09-12T19:43:58.386 DEBUG:boto:Path: /jdpiak-71/?prefix=h 2022-09-12T19:43:58.387 DEBUG:boto:Data: 2022-09-12T19:43:58.388 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.389 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:58.390 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.391 DEBUG:boto:Params: {} 2022-09-12T19:43:58.392 DEBUG:boto:Token: None 2022-09-12T19:43:58.393 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/ 2022-09-12T19:43:58.393 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc= 2022-09-12T19:43:58.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc=', 'Content-Length': '0'} 2022-09-12T19:43:58.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c81b6ccd609df9a6-00631f8bfe-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.399 DEBUG:boto:b'jdpiak-71h1000falseh2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:58.400 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-71 zones={test-zone2, test-zone1} 2022-09-12T19:43:58.401 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-71 zones={test-zone3, test-zone1} 2022-09-12T19:43:58.402 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:58.403 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:58.404 DEBUG:boto:Method: HEAD 2022-09-12T19:43:58.405 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:43:58.406 DEBUG:boto:Data: 2022-09-12T19:43:58.407 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.408 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:58.409 DEBUG:boto:Port: 8001 2022-09-12T19:43:58.410 DEBUG:boto:Params: {} 2022-09-12T19:43:58.411 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T19:43:58.412 DEBUG:boto:Token: None 2022-09-12T19:43:58.413 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/ 2022-09-12T19:43:58.414 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DjzWBh2VeUjH4yTYK5YzyJ/prCc= 2022-09-12T19:43:58.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DjzWBh2VeUjH4yTYK5YzyJ/prCc=', 'Content-Length': '0'} 2022-09-12T19:43:58.419 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000149f1b2860e3ea02-00631f8bfe-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.421 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:58.421 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:58.422 DEBUG:boto:Method: HEAD 2022-09-12T19:43:58.423 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:43:58.424 DEBUG:boto:Data: 2022-09-12T19:43:58.425 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.426 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:58.427 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.428 DEBUG:boto:Params: {} 2022-09-12T19:43:58.429 DEBUG:boto:Token: None 2022-09-12T19:43:58.430 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/ 2022-09-12T19:43:58.431 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DjzWBh2VeUjH4yTYK5YzyJ/prCc= 2022-09-12T19:43:58.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DjzWBh2VeUjH4yTYK5YzyJ/prCc=', 'Content-Length': '0'} 2022-09-12T19:43:58.438 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000061e6b4d25b39675d-00631f8bfe-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.439 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:43:58.440 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:58.441 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:58.442 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:43:58.443 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:43:58.443 DEBUG:boto:Method: GET 2022-09-12T19:43:58.444 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:43:58.444 DEBUG:boto:Data: 2022-09-12T19:43:58.445 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.446 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:58.446 DEBUG:boto:Port: 8001 2022-09-12T19:43:58.447 DEBUG:boto:Params: {} 2022-09-12T19:43:58.447 DEBUG:boto:Token: None 2022-09-12T19:43:58.448 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/?versions 2022-09-12T19:43:58.448 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:i+uvBwSc+yRRylqG0zUalm0YFgE= 2022-09-12T19:43:58.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:i+uvBwSc+yRRylqG0zUalm0YFgE=', 'Content-Length': '0'} 2022-09-12T19:43:58.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076061480ba41b198-00631f8bfe-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.453 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:58.454 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:43:58.455 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:43:58.455 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:43:58.456 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:43:58.456 DEBUG:tasks.rgw_multi.zone_rados:o=e 2022-09-12T19:43:58.457 DEBUG:tasks.rgw_multi.zone_rados:o=f 2022-09-12T19:43:58.457 DEBUG:tasks.rgw_multi.zone_rados:o=g 2022-09-12T19:43:58.458 DEBUG:tasks.rgw_multi.zone_rados:o=h 2022-09-12T19:43:58.458 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:43:58.459 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:58.459 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:58.460 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:43:58.460 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:43:58.461 DEBUG:boto:Method: GET 2022-09-12T19:43:58.461 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:43:58.462 DEBUG:boto:Data: 2022-09-12T19:43:58.462 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.463 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:58.464 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.464 DEBUG:boto:Params: {} 2022-09-12T19:43:58.465 DEBUG:boto:Token: None 2022-09-12T19:43:58.466 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/?versions 2022-09-12T19:43:58.466 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:i+uvBwSc+yRRylqG0zUalm0YFgE= 2022-09-12T19:43:58.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:i+uvBwSc+yRRylqG0zUalm0YFgE=', 'Content-Length': '0'} 2022-09-12T19:43:58.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfc0badbcd580d98-00631f8bfe-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.472 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:58.473 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:43:58.474 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:43:58.475 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:43:58.475 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:43:58.476 DEBUG:tasks.rgw_multi.zone_rados:o=e 2022-09-12T19:43:58.477 DEBUG:tasks.rgw_multi.zone_rados:o=f 2022-09-12T19:43:58.478 DEBUG:tasks.rgw_multi.zone_rados:o=g 2022-09-12T19:43:58.479 DEBUG:tasks.rgw_multi.zone_rados:o=h 2022-09-12T19:43:58.480 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:58.481 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:58.481 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:43:58.482 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:43:58.483 DEBUG:boto:Method: GET 2022-09-12T19:43:58.485 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:43:58.486 DEBUG:boto:Data: 2022-09-12T19:43:58.487 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.487 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:58.493 DEBUG:boto:Port: 8001 2022-09-12T19:43:58.496 DEBUG:boto:Params: {} 2022-09-12T19:43:58.497 DEBUG:boto:Token: None 2022-09-12T19:43:58.498 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/?versions 2022-09-12T19:43:58.498 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:i+uvBwSc+yRRylqG0zUalm0YFgE= 2022-09-12T19:43:58.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:i+uvBwSc+yRRylqG0zUalm0YFgE=', 'Content-Length': '0'} 2022-09-12T19:43:58.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3de3ca9774baf4e-00631f8bfe-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.507 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:58.509 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:58.509 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:58.510 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:43:58.510 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:43:58.511 DEBUG:boto:Method: GET 2022-09-12T19:43:58.511 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:43:58.512 DEBUG:boto:Data: 2022-09-12T19:43:58.512 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.513 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:58.514 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.518 DEBUG:boto:Params: {} 2022-09-12T19:43:58.518 DEBUG:boto:Token: None 2022-09-12T19:43:58.520 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/?versions 2022-09-12T19:43:58.520 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:i+uvBwSc+yRRylqG0zUalm0YFgE= 2022-09-12T19:43:58.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:i+uvBwSc+yRRylqG0zUalm0YFgE=', 'Content-Length': '0'} 2022-09-12T19:43:58.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000411d4e9151ae324c-00631f8bfe-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.530 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:58.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:43:58.535 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:43:58.536 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:43:58.537 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:43:58.538 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:43:58.540 DEBUG:boto:Method: GET 2022-09-12T19:43:58.541 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:43:58.542 DEBUG:boto:Data: 2022-09-12T19:43:58.543 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.544 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:58.545 DEBUG:boto:Port: 8001 2022-09-12T19:43:58.547 DEBUG:boto:Params: {} 2022-09-12T19:43:58.548 DEBUG:boto:Token: None 2022-09-12T19:43:58.548 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:43:58.549 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/NGo93OWbA3cKfR1S0cT5Feh9/8= 2022-09-12T19:43:58.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/NGo93OWbA3cKfR1S0cT5Feh9/8=', 'Content-Length': '0'} 2022-09-12T19:43:58.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2738699f3491edf-00631f8bfe-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.554 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:43:58.554 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:43:58.555 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:43:58.555 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:43:58.556 DEBUG:boto:Method: GET 2022-09-12T19:43:58.557 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:43:58.557 DEBUG:boto:Data: 2022-09-12T19:43:58.558 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.558 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:58.559 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.559 DEBUG:boto:Params: {} 2022-09-12T19:43:58.560 DEBUG:boto:Token: None 2022-09-12T19:43:58.561 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:43:58.561 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/NGo93OWbA3cKfR1S0cT5Feh9/8= 2022-09-12T19:43:58.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/NGo93OWbA3cKfR1S0cT5Feh9/8=', 'Content-Length': '0'} 2022-09-12T19:43:58.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000255c59d30ea9999c-00631f8bfe-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.565 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:43:58.566 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:43:58.566 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:43:58.567 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:43:58.567 DEBUG:boto:Method: HEAD 2022-09-12T19:43:58.568 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:43:58.568 DEBUG:boto:Data: 2022-09-12T19:43:58.569 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.569 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:58.570 DEBUG:boto:Port: 8001 2022-09-12T19:43:58.570 DEBUG:boto:Params: {} 2022-09-12T19:43:58.571 DEBUG:boto:Token: None 2022-09-12T19:43:58.572 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:43:58.572 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:17SY6kmM+Jxe0JsVBIBIlEDGbHA= 2022-09-12T19:43:58.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:17SY6kmM+Jxe0JsVBIBIlEDGbHA=', 'Content-Length': '0'} 2022-09-12T19:43:58.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048deb30405fc1c7f-00631f8bfe-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.576 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:43:58.576 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:43:58.577 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:43:58.577 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:43:58.578 DEBUG:boto:Method: HEAD 2022-09-12T19:43:58.578 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:43:58.579 DEBUG:boto:Data: 2022-09-12T19:43:58.579 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.580 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:58.580 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.581 DEBUG:boto:Params: {} 2022-09-12T19:43:58.581 DEBUG:boto:Token: None 2022-09-12T19:43:58.582 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:43:58.583 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:17SY6kmM+Jxe0JsVBIBIlEDGbHA= 2022-09-12T19:43:58.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:17SY6kmM+Jxe0JsVBIBIlEDGbHA=', 'Content-Length': '0'} 2022-09-12T19:43:58.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3867bac962bba49-00631f8bfe-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:43:58.587 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:43:58.587 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:43:58.588 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:43:58.588 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:43:58.589 DEBUG:boto:Method: GET 2022-09-12T19:43:58.589 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:43:58.590 DEBUG:boto:Data: 2022-09-12T19:43:58.591 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.591 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:58.592 DEBUG:boto:Port: 8001 2022-09-12T19:43:58.592 DEBUG:boto:Params: {} 2022-09-12T19:43:58.593 DEBUG:boto:Token: None 2022-09-12T19:43:58.593 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:43:58.594 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/NGo93OWbA3cKfR1S0cT5Feh9/8= 2022-09-12T19:43:58.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/NGo93OWbA3cKfR1S0cT5Feh9/8=', 'Content-Length': '0'} 2022-09-12T19:43:58.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c80ab1774bec9c4-00631f8bfe-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.638 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:43:58.639 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:43:58.639 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:43:58.640 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:43:58.641 DEBUG:boto:Method: GET 2022-09-12T19:43:58.642 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:43:58.642 DEBUG:boto:Data: 2022-09-12T19:43:58.643 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.643 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:58.644 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.644 DEBUG:boto:Params: {} 2022-09-12T19:43:58.645 DEBUG:boto:Token: None 2022-09-12T19:43:58.646 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:43:58.646 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/NGo93OWbA3cKfR1S0cT5Feh9/8= 2022-09-12T19:43:58.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/NGo93OWbA3cKfR1S0cT5Feh9/8=', 'Content-Length': '0'} 2022-09-12T19:43:58.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e21d21cc44c5669-00631f8bfe-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.650 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:58.651 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:58.651 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:43:58.652 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:43:58.652 DEBUG:boto:Method: GET 2022-09-12T19:43:58.653 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:43:58.653 DEBUG:boto:Data: 2022-09-12T19:43:58.654 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.654 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:58.655 DEBUG:boto:Port: 8001 2022-09-12T19:43:58.655 DEBUG:boto:Params: {} 2022-09-12T19:43:58.656 DEBUG:boto:Token: None 2022-09-12T19:43:58.657 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/ 2022-09-12T19:43:58.657 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc= 2022-09-12T19:43:58.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc=', 'Content-Length': '0'} 2022-09-12T19:43:58.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021820ddcbcb07ce7-00631f8bfe-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.662 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:58.663 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:58.664 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:58.664 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:43:58.665 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:43:58.666 DEBUG:boto:Method: GET 2022-09-12T19:43:58.667 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:43:58.667 DEBUG:boto:Data: 2022-09-12T19:43:58.669 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.670 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:58.670 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.671 DEBUG:boto:Params: {} 2022-09-12T19:43:58.673 DEBUG:boto:Token: None 2022-09-12T19:43:58.674 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/ 2022-09-12T19:43:58.676 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc= 2022-09-12T19:43:58.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc=', 'Content-Length': '0'} 2022-09-12T19:43:58.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000572aa50884c86df6-00631f8bfe-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.682 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:58.683 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:43:58.684 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:43:58.685 DEBUG:boto:Method: HEAD 2022-09-12T19:43:58.686 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:43:58.688 DEBUG:boto:Data: 2022-09-12T19:43:58.688 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.689 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:58.690 DEBUG:boto:Port: 8001 2022-09-12T19:43:58.691 DEBUG:boto:Params: {} 2022-09-12T19:43:58.692 DEBUG:boto:Token: None 2022-09-12T19:43:58.693 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/a 2022-09-12T19:43:58.694 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pMuS9Crqv1Mf4rw43LZyI7ixUTQ= 2022-09-12T19:43:58.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pMuS9Crqv1Mf4rw43LZyI7ixUTQ=', 'Content-Length': '0'} 2022-09-12T19:43:58.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f56d3715d4c5fc3d-00631f8bfe-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.701 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:43:58.702 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:43:58.703 DEBUG:boto:Method: HEAD 2022-09-12T19:43:58.704 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:43:58.705 DEBUG:boto:Data: 2022-09-12T19:43:58.706 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.707 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:58.707 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.708 DEBUG:boto:Params: {} 2022-09-12T19:43:58.710 DEBUG:boto:Token: None 2022-09-12T19:43:58.711 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/a 2022-09-12T19:43:58.712 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pMuS9Crqv1Mf4rw43LZyI7ixUTQ= 2022-09-12T19:43:58.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pMuS9Crqv1Mf4rw43LZyI7ixUTQ=', 'Content-Length': '0'} 2022-09-12T19:43:58.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5a154facf226d2c-00631f8bfe-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:43:58.719 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:43:58.720 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:43:58.721 DEBUG:boto:Method: GET 2022-09-12T19:43:58.722 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:43:58.723 DEBUG:boto:Data: 2022-09-12T19:43:58.724 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.725 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:58.726 DEBUG:boto:Port: 8001 2022-09-12T19:43:58.727 DEBUG:boto:Params: {} 2022-09-12T19:43:58.728 DEBUG:boto:Token: None 2022-09-12T19:43:58.729 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/a 2022-09-12T19:43:58.730 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oDy2UsqJCZDBR41CkGC1jRIUV1A= 2022-09-12T19:43:58.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oDy2UsqJCZDBR41CkGC1jRIUV1A=', 'Content-Length': '0'} 2022-09-12T19:43:58.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000362eda29a331d98-00631f8bfe-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.736 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:43:58.737 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:43:58.738 DEBUG:boto:Method: GET 2022-09-12T19:43:58.739 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:43:58.740 DEBUG:boto:Data: 2022-09-12T19:43:58.741 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.741 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:58.742 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.743 DEBUG:boto:Params: {} 2022-09-12T19:43:58.743 DEBUG:boto:Token: None 2022-09-12T19:43:58.744 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/a 2022-09-12T19:43:58.745 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oDy2UsqJCZDBR41CkGC1jRIUV1A= 2022-09-12T19:43:58.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oDy2UsqJCZDBR41CkGC1jRIUV1A=', 'Content-Length': '0'} 2022-09-12T19:43:58.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8c01ba295aedec1-00631f8bfe-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.748 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:58.749 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:58.750 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:43:58.750 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:43:58.751 DEBUG:boto:Method: GET 2022-09-12T19:43:58.751 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:43:58.752 DEBUG:boto:Data: 2022-09-12T19:43:58.752 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.753 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:58.753 DEBUG:boto:Port: 8001 2022-09-12T19:43:58.754 DEBUG:boto:Params: {} 2022-09-12T19:43:58.755 DEBUG:boto:Token: None 2022-09-12T19:43:58.756 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/ 2022-09-12T19:43:58.756 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc= 2022-09-12T19:43:58.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc=', 'Content-Length': '0'} 2022-09-12T19:43:58.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d08953da8d9eb4b-00631f8bfe-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.762 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:58.764 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:58.764 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:58.765 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:43:58.766 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:43:58.767 DEBUG:boto:Method: GET 2022-09-12T19:43:58.767 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:43:58.768 DEBUG:boto:Data: 2022-09-12T19:43:58.768 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.769 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:58.769 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.770 DEBUG:boto:Params: {} 2022-09-12T19:43:58.771 DEBUG:boto:Token: None 2022-09-12T19:43:58.771 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/ 2022-09-12T19:43:58.772 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc= 2022-09-12T19:43:58.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc=', 'Content-Length': '0'} 2022-09-12T19:43:58.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000470b804f6c006a6-00631f8bfe-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.788 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:58.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:43:58.791 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:43:58.792 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:43:58.793 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:43:58.794 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:43:58.795 DEBUG:boto:Method: GET 2022-09-12T19:43:58.797 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:43:58.798 DEBUG:boto:Data: 2022-09-12T19:43:58.799 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.799 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:58.800 DEBUG:boto:Port: 8001 2022-09-12T19:43:58.800 DEBUG:boto:Params: {} 2022-09-12T19:43:58.801 DEBUG:boto:Token: None 2022-09-12T19:43:58.802 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:43:58.802 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3LraIoMVHyZxXKRNumyJTlMxxPg= 2022-09-12T19:43:58.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3LraIoMVHyZxXKRNumyJTlMxxPg=', 'Content-Length': '0'} 2022-09-12T19:43:58.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cbe1cbd77aba32f-00631f8bfe-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.807 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:43:58.808 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:43:58.809 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:43:58.824 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:43:58.825 DEBUG:boto:Method: GET 2022-09-12T19:43:58.827 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:43:58.828 DEBUG:boto:Data: 2022-09-12T19:43:58.829 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.830 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:58.831 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.832 DEBUG:boto:Params: {} 2022-09-12T19:43:58.834 DEBUG:boto:Token: None 2022-09-12T19:43:58.836 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:43:58.837 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3LraIoMVHyZxXKRNumyJTlMxxPg= 2022-09-12T19:43:58.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3LraIoMVHyZxXKRNumyJTlMxxPg=', 'Content-Length': '0'} 2022-09-12T19:43:58.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016f1b217d0001c1f-00631f8bfe-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.843 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:43:58.847 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:43:58.848 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:43:58.849 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:43:58.850 DEBUG:boto:Method: HEAD 2022-09-12T19:43:58.851 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:43:58.852 DEBUG:boto:Data: 2022-09-12T19:43:58.853 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.855 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:58.856 DEBUG:boto:Port: 8001 2022-09-12T19:43:58.857 DEBUG:boto:Params: {} 2022-09-12T19:43:58.858 DEBUG:boto:Token: None 2022-09-12T19:43:58.860 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:43:58.861 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3lMfxa8TrqCCfnn1O/hb6HW4To4= 2022-09-12T19:43:58.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3lMfxa8TrqCCfnn1O/hb6HW4To4=', 'Content-Length': '0'} 2022-09-12T19:43:58.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ddc663b6a086d2a-00631f8bfe-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.868 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:43:58.869 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:43:58.870 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:43:58.872 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:43:58.873 DEBUG:boto:Method: HEAD 2022-09-12T19:43:58.874 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:43:58.875 DEBUG:boto:Data: 2022-09-12T19:43:58.876 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.877 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:58.878 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.879 DEBUG:boto:Params: {} 2022-09-12T19:43:58.881 DEBUG:boto:Token: None 2022-09-12T19:43:58.882 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:43:58.883 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3lMfxa8TrqCCfnn1O/hb6HW4To4= 2022-09-12T19:43:58.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3lMfxa8TrqCCfnn1O/hb6HW4To4=', 'Content-Length': '0'} 2022-09-12T19:43:58.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ed5c876cf6345dd-00631f8bfe-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:43:58.897 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:43:58.898 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:43:58.899 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:43:58.900 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:43:58.901 DEBUG:boto:Method: GET 2022-09-12T19:43:58.902 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:43:58.903 DEBUG:boto:Data: 2022-09-12T19:43:58.906 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.908 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:58.909 DEBUG:boto:Port: 8001 2022-09-12T19:43:58.910 DEBUG:boto:Params: {} 2022-09-12T19:43:58.911 DEBUG:boto:Token: None 2022-09-12T19:43:58.912 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:43:58.913 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3LraIoMVHyZxXKRNumyJTlMxxPg= 2022-09-12T19:43:58.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3LraIoMVHyZxXKRNumyJTlMxxPg=', 'Content-Length': '0'} 2022-09-12T19:43:58.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c9aa95ec962ac94-00631f8bfe-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.919 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:43:58.920 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:43:58.921 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:43:58.923 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:43:58.926 DEBUG:boto:Method: GET 2022-09-12T19:43:58.928 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:43:58.929 DEBUG:boto:Data: 2022-09-12T19:43:58.930 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.931 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:58.935 DEBUG:boto:Port: 8000 2022-09-12T19:43:58.936 DEBUG:boto:Params: {} 2022-09-12T19:43:58.936 DEBUG:boto:Token: None 2022-09-12T19:43:58.937 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:43:58.938 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3LraIoMVHyZxXKRNumyJTlMxxPg= 2022-09-12T19:43:58.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3LraIoMVHyZxXKRNumyJTlMxxPg=', 'Content-Length': '0'} 2022-09-12T19:43:58.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039c808d41057c7d8-00631f8bfe-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:58.942 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:58.943 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:58.943 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:43:58.944 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:43:58.945 DEBUG:boto:Method: GET 2022-09-12T19:43:58.945 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:43:58.946 DEBUG:boto:Data: 2022-09-12T19:43:58.947 DEBUG:boto:Headers: {} 2022-09-12T19:43:58.947 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:58.948 DEBUG:boto:Port: 8001 2022-09-12T19:43:58.949 DEBUG:boto:Params: {} 2022-09-12T19:43:58.949 DEBUG:boto:Token: None 2022-09-12T19:43:58.950 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:58 GMT /jdpiak-71/ 2022-09-12T19:43:58.951 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc= 2022-09-12T19:43:58.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2IYkQLzl6+iq8f7gxjUQCWCJIuc=', 'Content-Length': '0'} 2022-09-12T19:43:58.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086e87fe078a72e51-00631f8bfe-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.002 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:59.004 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:59.005 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:59.006 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:43:59.007 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:43:59.008 DEBUG:boto:Method: GET 2022-09-12T19:43:59.008 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:43:59.009 DEBUG:boto:Data: 2022-09-12T19:43:59.010 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.011 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:59.011 DEBUG:boto:Port: 8000 2022-09-12T19:43:59.012 DEBUG:boto:Params: {} 2022-09-12T19:43:59.012 DEBUG:boto:Token: None 2022-09-12T19:43:59.013 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/ 2022-09-12T19:43:59.014 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak= 2022-09-12T19:43:59.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak=', 'Content-Length': '0'} 2022-09-12T19:43:59.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000081f4471590fe5a4-00631f8bff-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.021 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:59.023 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:43:59.024 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:43:59.025 DEBUG:boto:Method: HEAD 2022-09-12T19:43:59.030 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:43:59.031 DEBUG:boto:Data: 2022-09-12T19:43:59.032 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.033 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:59.035 DEBUG:boto:Port: 8001 2022-09-12T19:43:59.036 DEBUG:boto:Params: {} 2022-09-12T19:43:59.037 DEBUG:boto:Token: None 2022-09-12T19:43:59.038 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/b 2022-09-12T19:43:59.040 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zeeBmrLFxij5EY6+5p3KaN83jNU= 2022-09-12T19:43:59.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zeeBmrLFxij5EY6+5p3KaN83jNU=', 'Content-Length': '0'} 2022-09-12T19:43:59.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036c5f0479f2cc6a5-00631f8bff-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.046 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:43:59.047 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:43:59.048 DEBUG:boto:Method: HEAD 2022-09-12T19:43:59.048 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:43:59.049 DEBUG:boto:Data: 2022-09-12T19:43:59.050 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.051 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:59.052 DEBUG:boto:Port: 8000 2022-09-12T19:43:59.053 DEBUG:boto:Params: {} 2022-09-12T19:43:59.053 DEBUG:boto:Token: None 2022-09-12T19:43:59.054 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/b 2022-09-12T19:43:59.055 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zeeBmrLFxij5EY6+5p3KaN83jNU= 2022-09-12T19:43:59.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zeeBmrLFxij5EY6+5p3KaN83jNU=', 'Content-Length': '0'} 2022-09-12T19:43:59.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006931b31ed7081675-00631f8bff-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:43:59.064 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:43:59.064 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:43:59.065 DEBUG:boto:Method: GET 2022-09-12T19:43:59.065 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:43:59.066 DEBUG:boto:Data: 2022-09-12T19:43:59.066 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.067 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:59.067 DEBUG:boto:Port: 8001 2022-09-12T19:43:59.068 DEBUG:boto:Params: {} 2022-09-12T19:43:59.069 DEBUG:boto:Token: None 2022-09-12T19:43:59.069 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/b 2022-09-12T19:43:59.070 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LKIJH5/vh3gumJhOCnPUkkgD9LM= 2022-09-12T19:43:59.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LKIJH5/vh3gumJhOCnPUkkgD9LM=', 'Content-Length': '0'} 2022-09-12T19:43:59.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004c947bce1e4788b-00631f8bff-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.074 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:43:59.075 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:43:59.076 DEBUG:boto:Method: GET 2022-09-12T19:43:59.077 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:43:59.078 DEBUG:boto:Data: 2022-09-12T19:43:59.079 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.080 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:59.081 DEBUG:boto:Port: 8000 2022-09-12T19:43:59.082 DEBUG:boto:Params: {} 2022-09-12T19:43:59.083 DEBUG:boto:Token: None 2022-09-12T19:43:59.084 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/b 2022-09-12T19:43:59.086 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LKIJH5/vh3gumJhOCnPUkkgD9LM= 2022-09-12T19:43:59.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LKIJH5/vh3gumJhOCnPUkkgD9LM=', 'Content-Length': '0'} 2022-09-12T19:43:59.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b891cde7f6860792-00631f8bff-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.091 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:59.092 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:59.093 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:43:59.094 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:43:59.095 DEBUG:boto:Method: GET 2022-09-12T19:43:59.096 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:43:59.096 DEBUG:boto:Data: 2022-09-12T19:43:59.097 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.098 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:59.099 DEBUG:boto:Port: 8001 2022-09-12T19:43:59.100 DEBUG:boto:Params: {} 2022-09-12T19:43:59.101 DEBUG:boto:Token: None 2022-09-12T19:43:59.102 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/ 2022-09-12T19:43:59.103 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak= 2022-09-12T19:43:59.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak=', 'Content-Length': '0'} 2022-09-12T19:43:59.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b72cf39cf3c0993-00631f8bff-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.110 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:59.111 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:59.112 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:59.113 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:43:59.114 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:43:59.115 DEBUG:boto:Method: GET 2022-09-12T19:43:59.116 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:43:59.117 DEBUG:boto:Data: 2022-09-12T19:43:59.118 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.119 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:59.120 DEBUG:boto:Port: 8000 2022-09-12T19:43:59.120 DEBUG:boto:Params: {} 2022-09-12T19:43:59.121 DEBUG:boto:Token: None 2022-09-12T19:43:59.122 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/ 2022-09-12T19:43:59.123 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak= 2022-09-12T19:43:59.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak=', 'Content-Length': '0'} 2022-09-12T19:43:59.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003da8100352c72e0d-00631f8bff-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.132 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:59.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:43:59.135 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:43:59.136 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:43:59.137 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:43:59.139 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:43:59.140 DEBUG:boto:Method: GET 2022-09-12T19:43:59.141 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:43:59.142 DEBUG:boto:Data: 2022-09-12T19:43:59.143 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.144 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:59.145 DEBUG:boto:Port: 8001 2022-09-12T19:43:59.146 DEBUG:boto:Params: {} 2022-09-12T19:43:59.147 DEBUG:boto:Token: None 2022-09-12T19:43:59.148 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:43:59.150 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RcYRuanXnCexKIQcITrZymZHdRA= 2022-09-12T19:43:59.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RcYRuanXnCexKIQcITrZymZHdRA=', 'Content-Length': '0'} 2022-09-12T19:43:59.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024dd7e31ff9ab990-00631f8bff-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.156 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:43:59.158 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:43:59.159 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:43:59.160 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:43:59.161 DEBUG:boto:Method: GET 2022-09-12T19:43:59.162 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:43:59.163 DEBUG:boto:Data: 2022-09-12T19:43:59.164 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.166 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:59.167 DEBUG:boto:Port: 8000 2022-09-12T19:43:59.168 DEBUG:boto:Params: {} 2022-09-12T19:43:59.169 DEBUG:boto:Token: None 2022-09-12T19:43:59.171 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:43:59.173 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RcYRuanXnCexKIQcITrZymZHdRA= 2022-09-12T19:43:59.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RcYRuanXnCexKIQcITrZymZHdRA=', 'Content-Length': '0'} 2022-09-12T19:43:59.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccc21c6a04a9d728-00631f8bff-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.179 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:43:59.180 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:43:59.181 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:43:59.182 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:43:59.182 DEBUG:boto:Method: HEAD 2022-09-12T19:43:59.183 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:43:59.184 DEBUG:boto:Data: 2022-09-12T19:43:59.185 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.186 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:59.186 DEBUG:boto:Port: 8001 2022-09-12T19:43:59.187 DEBUG:boto:Params: {} 2022-09-12T19:43:59.188 DEBUG:boto:Token: None 2022-09-12T19:43:59.188 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:43:59.189 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8+Bb0M7eIiQ19JcMewQlWe+nXGU= 2022-09-12T19:43:59.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8+Bb0M7eIiQ19JcMewQlWe+nXGU=', 'Content-Length': '0'} 2022-09-12T19:43:59.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfb54ebab1b544e3-00631f8bff-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.193 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:43:59.194 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:43:59.195 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:43:59.196 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:43:59.197 DEBUG:boto:Method: HEAD 2022-09-12T19:43:59.197 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:43:59.198 DEBUG:boto:Data: 2022-09-12T19:43:59.199 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.200 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:59.200 DEBUG:boto:Port: 8000 2022-09-12T19:43:59.201 DEBUG:boto:Params: {} 2022-09-12T19:43:59.201 DEBUG:boto:Token: None 2022-09-12T19:43:59.202 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:43:59.203 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8+Bb0M7eIiQ19JcMewQlWe+nXGU= 2022-09-12T19:43:59.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8+Bb0M7eIiQ19JcMewQlWe+nXGU=', 'Content-Length': '0'} 2022-09-12T19:43:59.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c39e252878d6b03-00631f8bff-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:43:59.207 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:43:59.208 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:43:59.209 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:43:59.210 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:43:59.211 DEBUG:boto:Method: GET 2022-09-12T19:43:59.212 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:43:59.213 DEBUG:boto:Data: 2022-09-12T19:43:59.215 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.216 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:59.217 DEBUG:boto:Port: 8001 2022-09-12T19:43:59.218 DEBUG:boto:Params: {} 2022-09-12T19:43:59.219 DEBUG:boto:Token: None 2022-09-12T19:43:59.221 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:43:59.222 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RcYRuanXnCexKIQcITrZymZHdRA= 2022-09-12T19:43:59.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RcYRuanXnCexKIQcITrZymZHdRA=', 'Content-Length': '0'} 2022-09-12T19:43:59.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d90f2e2d830810c9-00631f8bff-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.231 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:43:59.232 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:43:59.234 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:43:59.236 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:43:59.237 DEBUG:boto:Method: GET 2022-09-12T19:43:59.238 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:43:59.239 DEBUG:boto:Data: 2022-09-12T19:43:59.241 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.242 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:59.243 DEBUG:boto:Port: 8000 2022-09-12T19:43:59.244 DEBUG:boto:Params: {} 2022-09-12T19:43:59.245 DEBUG:boto:Token: None 2022-09-12T19:43:59.247 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:43:59.248 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RcYRuanXnCexKIQcITrZymZHdRA= 2022-09-12T19:43:59.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RcYRuanXnCexKIQcITrZymZHdRA=', 'Content-Length': '0'} 2022-09-12T19:43:59.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae6cfd4d31225f5c-00631f8bff-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.254 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:59.255 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:59.257 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:43:59.258 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:43:59.259 DEBUG:boto:Method: GET 2022-09-12T19:43:59.260 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:43:59.261 DEBUG:boto:Data: 2022-09-12T19:43:59.262 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.263 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:59.264 DEBUG:boto:Port: 8001 2022-09-12T19:43:59.265 DEBUG:boto:Params: {} 2022-09-12T19:43:59.266 DEBUG:boto:Token: None 2022-09-12T19:43:59.267 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/ 2022-09-12T19:43:59.269 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak= 2022-09-12T19:43:59.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak=', 'Content-Length': '0'} 2022-09-12T19:43:59.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bfa8c4ecc3a60c7-00631f8bff-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.275 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:59.277 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:59.278 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:59.279 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:43:59.280 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:43:59.281 DEBUG:boto:Method: GET 2022-09-12T19:43:59.282 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:43:59.283 DEBUG:boto:Data: 2022-09-12T19:43:59.284 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.285 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:59.286 DEBUG:boto:Port: 8000 2022-09-12T19:43:59.287 DEBUG:boto:Params: {} 2022-09-12T19:43:59.288 DEBUG:boto:Token: None 2022-09-12T19:43:59.289 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/ 2022-09-12T19:43:59.290 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak= 2022-09-12T19:43:59.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak=', 'Content-Length': '0'} 2022-09-12T19:43:59.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b799c1a1c08aa858-00631f8bff-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.295 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:59.296 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:43:59.297 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:43:59.297 DEBUG:boto:Method: HEAD 2022-09-12T19:43:59.298 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:43:59.298 DEBUG:boto:Data: 2022-09-12T19:43:59.299 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.300 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:59.300 DEBUG:boto:Port: 8001 2022-09-12T19:43:59.301 DEBUG:boto:Params: {} 2022-09-12T19:43:59.301 DEBUG:boto:Token: None 2022-09-12T19:43:59.302 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/c 2022-09-12T19:43:59.303 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L0uXWaszpy6zLNvL78vmFBx5WU0= 2022-09-12T19:43:59.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L0uXWaszpy6zLNvL78vmFBx5WU0=', 'Content-Length': '0'} 2022-09-12T19:43:59.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0d1ffb792929ab2-00631f8bff-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.307 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:43:59.307 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:43:59.308 DEBUG:boto:Method: HEAD 2022-09-12T19:43:59.308 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:43:59.309 DEBUG:boto:Data: 2022-09-12T19:43:59.309 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.310 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:59.310 DEBUG:boto:Port: 8000 2022-09-12T19:43:59.311 DEBUG:boto:Params: {} 2022-09-12T19:43:59.312 DEBUG:boto:Token: None 2022-09-12T19:43:59.312 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/c 2022-09-12T19:43:59.313 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L0uXWaszpy6zLNvL78vmFBx5WU0= 2022-09-12T19:43:59.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L0uXWaszpy6zLNvL78vmFBx5WU0=', 'Content-Length': '0'} 2022-09-12T19:43:59.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083a990a4d40a4fa4-00631f8bff-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:43:59.322 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:43:59.323 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:43:59.324 DEBUG:boto:Method: GET 2022-09-12T19:43:59.325 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:43:59.326 DEBUG:boto:Data: 2022-09-12T19:43:59.327 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.328 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:59.329 DEBUG:boto:Port: 8001 2022-09-12T19:43:59.329 DEBUG:boto:Params: {} 2022-09-12T19:43:59.330 DEBUG:boto:Token: None 2022-09-12T19:43:59.331 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/c 2022-09-12T19:43:59.331 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dBPtpdprUPeQo4OMaOwUTjmLMk4= 2022-09-12T19:43:59.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dBPtpdprUPeQo4OMaOwUTjmLMk4=', 'Content-Length': '0'} 2022-09-12T19:43:59.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035f004d169d953ae-00631f8bff-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.336 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:43:59.337 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:43:59.338 DEBUG:boto:Method: GET 2022-09-12T19:43:59.339 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:43:59.340 DEBUG:boto:Data: 2022-09-12T19:43:59.341 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.342 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:59.343 DEBUG:boto:Port: 8000 2022-09-12T19:43:59.344 DEBUG:boto:Params: {} 2022-09-12T19:43:59.345 DEBUG:boto:Token: None 2022-09-12T19:43:59.346 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/c 2022-09-12T19:43:59.347 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dBPtpdprUPeQo4OMaOwUTjmLMk4= 2022-09-12T19:43:59.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dBPtpdprUPeQo4OMaOwUTjmLMk4=', 'Content-Length': '0'} 2022-09-12T19:43:59.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4a6b2e665bf2527-00631f8bff-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.392 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:59.394 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:59.395 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:43:59.396 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:43:59.397 DEBUG:boto:Method: GET 2022-09-12T19:43:59.398 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:43:59.399 DEBUG:boto:Data: 2022-09-12T19:43:59.400 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.401 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:59.403 DEBUG:boto:Port: 8001 2022-09-12T19:43:59.404 DEBUG:boto:Params: {} 2022-09-12T19:43:59.405 DEBUG:boto:Token: None 2022-09-12T19:43:59.407 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/ 2022-09-12T19:43:59.408 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak= 2022-09-12T19:43:59.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak=', 'Content-Length': '0'} 2022-09-12T19:43:59.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029b38960ed33071c-00631f8bff-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.415 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:59.417 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:59.418 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:59.419 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:43:59.420 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:43:59.421 DEBUG:boto:Method: GET 2022-09-12T19:43:59.422 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:43:59.425 DEBUG:boto:Data: 2022-09-12T19:43:59.426 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.427 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:59.428 DEBUG:boto:Port: 8000 2022-09-12T19:43:59.429 DEBUG:boto:Params: {} 2022-09-12T19:43:59.431 DEBUG:boto:Token: None 2022-09-12T19:43:59.432 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/ 2022-09-12T19:43:59.434 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak= 2022-09-12T19:43:59.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak=', 'Content-Length': '0'} 2022-09-12T19:43:59.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5eed0732bb5d1a2-00631f8bff-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.440 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:59.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:43:59.442 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:43:59.442 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:43:59.443 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:43:59.444 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:43:59.444 DEBUG:boto:Method: GET 2022-09-12T19:43:59.446 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:43:59.446 DEBUG:boto:Data: 2022-09-12T19:43:59.447 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.448 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:59.448 DEBUG:boto:Port: 8001 2022-09-12T19:43:59.449 DEBUG:boto:Params: {} 2022-09-12T19:43:59.449 DEBUG:boto:Token: None 2022-09-12T19:43:59.450 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:43:59.451 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5z4vYmIeqHrFEkONJoOA5sTLHUg= 2022-09-12T19:43:59.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5z4vYmIeqHrFEkONJoOA5sTLHUg=', 'Content-Length': '0'} 2022-09-12T19:43:59.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6e711417ca239d2-00631f8bff-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.494 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:43:59.495 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:43:59.496 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:43:59.498 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:43:59.500 DEBUG:boto:Method: GET 2022-09-12T19:43:59.500 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:43:59.501 DEBUG:boto:Data: 2022-09-12T19:43:59.502 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.503 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:59.503 DEBUG:boto:Port: 8000 2022-09-12T19:43:59.504 DEBUG:boto:Params: {} 2022-09-12T19:43:59.504 DEBUG:boto:Token: None 2022-09-12T19:43:59.505 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:43:59.506 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5z4vYmIeqHrFEkONJoOA5sTLHUg= 2022-09-12T19:43:59.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5z4vYmIeqHrFEkONJoOA5sTLHUg=', 'Content-Length': '0'} 2022-09-12T19:43:59.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3199438dc41cef8-00631f8bff-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.511 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:43:59.511 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:43:59.512 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:43:59.512 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:43:59.513 DEBUG:boto:Method: HEAD 2022-09-12T19:43:59.513 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:43:59.530 DEBUG:boto:Data: 2022-09-12T19:43:59.531 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.532 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:59.532 DEBUG:boto:Port: 8001 2022-09-12T19:43:59.533 DEBUG:boto:Params: {} 2022-09-12T19:43:59.533 DEBUG:boto:Token: None 2022-09-12T19:43:59.538 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:43:59.540 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PIXhYIerI9zI5sTkjwONIFVpwNo= 2022-09-12T19:43:59.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PIXhYIerI9zI5sTkjwONIFVpwNo=', 'Content-Length': '0'} 2022-09-12T19:43:59.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095a85181645f38fc-00631f8bff-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.544 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:43:59.545 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:43:59.547 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:43:59.548 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:43:59.549 DEBUG:boto:Method: HEAD 2022-09-12T19:43:59.550 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:43:59.551 DEBUG:boto:Data: 2022-09-12T19:43:59.552 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.553 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:59.554 DEBUG:boto:Port: 8000 2022-09-12T19:43:59.555 DEBUG:boto:Params: {} 2022-09-12T19:43:59.556 DEBUG:boto:Token: None 2022-09-12T19:43:59.557 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:43:59.558 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PIXhYIerI9zI5sTkjwONIFVpwNo= 2022-09-12T19:43:59.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PIXhYIerI9zI5sTkjwONIFVpwNo=', 'Content-Length': '0'} 2022-09-12T19:43:59.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c740765e1537ae0a-00631f8bff-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:43:59.564 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:43:59.578 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:43:59.580 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:43:59.581 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:43:59.582 DEBUG:boto:Method: GET 2022-09-12T19:43:59.583 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:43:59.584 DEBUG:boto:Data: 2022-09-12T19:43:59.586 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.587 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:59.588 DEBUG:boto:Port: 8001 2022-09-12T19:43:59.589 DEBUG:boto:Params: {} 2022-09-12T19:43:59.591 DEBUG:boto:Token: None 2022-09-12T19:43:59.592 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:43:59.593 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5z4vYmIeqHrFEkONJoOA5sTLHUg= 2022-09-12T19:43:59.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5z4vYmIeqHrFEkONJoOA5sTLHUg=', 'Content-Length': '0'} 2022-09-12T19:43:59.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6d02a623ddf1e9f-00631f8bff-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.599 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:43:59.600 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:43:59.602 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:43:59.603 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:43:59.604 DEBUG:boto:Method: GET 2022-09-12T19:43:59.605 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:43:59.606 DEBUG:boto:Data: 2022-09-12T19:43:59.607 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.608 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:59.609 DEBUG:boto:Port: 8000 2022-09-12T19:43:59.610 DEBUG:boto:Params: {} 2022-09-12T19:43:59.611 DEBUG:boto:Token: None 2022-09-12T19:43:59.612 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:43:59.613 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5z4vYmIeqHrFEkONJoOA5sTLHUg= 2022-09-12T19:43:59.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5z4vYmIeqHrFEkONJoOA5sTLHUg=', 'Content-Length': '0'} 2022-09-12T19:43:59.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8b8b187683e7255-00631f8bff-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.620 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:59.621 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:59.623 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:43:59.624 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:43:59.625 DEBUG:boto:Method: GET 2022-09-12T19:43:59.626 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:43:59.627 DEBUG:boto:Data: 2022-09-12T19:43:59.628 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.629 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:59.630 DEBUG:boto:Port: 8001 2022-09-12T19:43:59.631 DEBUG:boto:Params: {} 2022-09-12T19:43:59.632 DEBUG:boto:Token: None 2022-09-12T19:43:59.634 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/ 2022-09-12T19:43:59.635 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak= 2022-09-12T19:43:59.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak=', 'Content-Length': '0'} 2022-09-12T19:43:59.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b54cb94cab17c59-00631f8bff-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.641 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:59.643 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:59.644 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:59.645 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:43:59.646 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:43:59.647 DEBUG:boto:Method: GET 2022-09-12T19:43:59.648 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:43:59.648 DEBUG:boto:Data: 2022-09-12T19:43:59.649 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.650 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:59.651 DEBUG:boto:Port: 8000 2022-09-12T19:43:59.652 DEBUG:boto:Params: {} 2022-09-12T19:43:59.653 DEBUG:boto:Token: None 2022-09-12T19:43:59.654 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/ 2022-09-12T19:43:59.655 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak= 2022-09-12T19:43:59.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak=', 'Content-Length': '0'} 2022-09-12T19:43:59.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6446bcd92f409a3-00631f8bff-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.662 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:59.663 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:43:59.664 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:43:59.665 DEBUG:boto:Method: HEAD 2022-09-12T19:43:59.666 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:43:59.667 DEBUG:boto:Data: 2022-09-12T19:43:59.668 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.669 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:59.670 DEBUG:boto:Port: 8001 2022-09-12T19:43:59.671 DEBUG:boto:Params: {} 2022-09-12T19:43:59.672 DEBUG:boto:Token: None 2022-09-12T19:43:59.673 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/d 2022-09-12T19:43:59.674 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:miAMh1TQC1AGMnxpIbRL5J6HXyk= 2022-09-12T19:43:59.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:miAMh1TQC1AGMnxpIbRL5J6HXyk=', 'Content-Length': '0'} 2022-09-12T19:43:59.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000379161a774ffad36-00631f8bff-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.687 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:43:59.688 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:43:59.689 DEBUG:boto:Method: HEAD 2022-09-12T19:43:59.690 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:43:59.691 DEBUG:boto:Data: 2022-09-12T19:43:59.692 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.693 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:59.694 DEBUG:boto:Port: 8000 2022-09-12T19:43:59.695 DEBUG:boto:Params: {} 2022-09-12T19:43:59.696 DEBUG:boto:Token: None 2022-09-12T19:43:59.697 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/d 2022-09-12T19:43:59.698 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:miAMh1TQC1AGMnxpIbRL5J6HXyk= 2022-09-12T19:43:59.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:miAMh1TQC1AGMnxpIbRL5J6HXyk=', 'Content-Length': '0'} 2022-09-12T19:43:59.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005809ded9ffe559ea-00631f8bff-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:43:59.705 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:43:59.706 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:43:59.707 DEBUG:boto:Method: GET 2022-09-12T19:43:59.708 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:43:59.709 DEBUG:boto:Data: 2022-09-12T19:43:59.709 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.710 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:59.711 DEBUG:boto:Port: 8001 2022-09-12T19:43:59.712 DEBUG:boto:Params: {} 2022-09-12T19:43:59.714 DEBUG:boto:Token: None 2022-09-12T19:43:59.715 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/d 2022-09-12T19:43:59.716 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IHuuCWSGP+SrLa7Z4kXAfFR/MiA= 2022-09-12T19:43:59.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IHuuCWSGP+SrLa7Z4kXAfFR/MiA=', 'Content-Length': '0'} 2022-09-12T19:43:59.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060b951704a3f85d4-00631f8bff-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.721 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:43:59.722 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:43:59.723 DEBUG:boto:Method: GET 2022-09-12T19:43:59.724 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:43:59.725 DEBUG:boto:Data: 2022-09-12T19:43:59.726 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.727 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:59.728 DEBUG:boto:Port: 8000 2022-09-12T19:43:59.729 DEBUG:boto:Params: {} 2022-09-12T19:43:59.730 DEBUG:boto:Token: None 2022-09-12T19:43:59.731 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/d 2022-09-12T19:43:59.732 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IHuuCWSGP+SrLa7Z4kXAfFR/MiA= 2022-09-12T19:43:59.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IHuuCWSGP+SrLa7Z4kXAfFR/MiA=', 'Content-Length': '0'} 2022-09-12T19:43:59.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fd205e66bbb6ce3-00631f8bff-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.739 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:59.740 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:59.741 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:43:59.742 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:43:59.743 DEBUG:boto:Method: GET 2022-09-12T19:43:59.745 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:43:59.746 DEBUG:boto:Data: 2022-09-12T19:43:59.747 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.748 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:59.749 DEBUG:boto:Port: 8001 2022-09-12T19:43:59.750 DEBUG:boto:Params: {} 2022-09-12T19:43:59.751 DEBUG:boto:Token: None 2022-09-12T19:43:59.752 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/ 2022-09-12T19:43:59.752 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak= 2022-09-12T19:43:59.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak=', 'Content-Length': '0'} 2022-09-12T19:43:59.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbc0f338023ec3f6-00631f8bff-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.756 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:59.757 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:59.758 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:59.758 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:43:59.759 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:43:59.759 DEBUG:boto:Method: GET 2022-09-12T19:43:59.760 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:43:59.760 DEBUG:boto:Data: 2022-09-12T19:43:59.761 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.761 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:59.762 DEBUG:boto:Port: 8000 2022-09-12T19:43:59.763 DEBUG:boto:Params: {} 2022-09-12T19:43:59.763 DEBUG:boto:Token: None 2022-09-12T19:43:59.764 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/ 2022-09-12T19:43:59.764 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak= 2022-09-12T19:43:59.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak=', 'Content-Length': '0'} 2022-09-12T19:43:59.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcf7275418946af2-00631f8bff-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.768 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:59.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:43:59.770 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:43:59.770 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:43:59.771 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:43:59.771 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:43:59.772 DEBUG:boto:Method: GET 2022-09-12T19:43:59.772 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:43:59.773 DEBUG:boto:Data: 2022-09-12T19:43:59.773 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.774 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:59.774 DEBUG:boto:Port: 8001 2022-09-12T19:43:59.775 DEBUG:boto:Params: {} 2022-09-12T19:43:59.775 DEBUG:boto:Token: None 2022-09-12T19:43:59.776 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:43:59.777 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SvrqNQStZ3u874t54m/KDXbW1kw= 2022-09-12T19:43:59.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SvrqNQStZ3u874t54m/KDXbW1kw=', 'Content-Length': '0'} 2022-09-12T19:43:59.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a99e0617e4cec61c-00631f8bff-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.781 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:43:59.782 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:43:59.782 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:43:59.783 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:43:59.783 DEBUG:boto:Method: GET 2022-09-12T19:43:59.784 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:43:59.784 DEBUG:boto:Data: 2022-09-12T19:43:59.785 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.785 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:59.786 DEBUG:boto:Port: 8000 2022-09-12T19:43:59.786 DEBUG:boto:Params: {} 2022-09-12T19:43:59.787 DEBUG:boto:Token: None 2022-09-12T19:43:59.787 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:43:59.788 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SvrqNQStZ3u874t54m/KDXbW1kw= 2022-09-12T19:43:59.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SvrqNQStZ3u874t54m/KDXbW1kw=', 'Content-Length': '0'} 2022-09-12T19:43:59.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049bf02408f5efa62-00631f8bff-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.792 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:43:59.792 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:43:59.793 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:43:59.793 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:43:59.794 DEBUG:boto:Method: HEAD 2022-09-12T19:43:59.794 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:43:59.795 DEBUG:boto:Data: 2022-09-12T19:43:59.795 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.796 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:59.796 DEBUG:boto:Port: 8001 2022-09-12T19:43:59.797 DEBUG:boto:Params: {} 2022-09-12T19:43:59.798 DEBUG:boto:Token: None 2022-09-12T19:43:59.798 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:43:59.799 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+NbqnDm7wpeTdjUMOYZzEDNxtDU= 2022-09-12T19:43:59.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+NbqnDm7wpeTdjUMOYZzEDNxtDU=', 'Content-Length': '0'} 2022-09-12T19:43:59.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4439a97b3a93e08-00631f8bff-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.802 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:43:59.803 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:43:59.803 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:43:59.804 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:43:59.804 DEBUG:boto:Method: HEAD 2022-09-12T19:43:59.805 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:43:59.805 DEBUG:boto:Data: 2022-09-12T19:43:59.806 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.806 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:59.807 DEBUG:boto:Port: 8000 2022-09-12T19:43:59.808 DEBUG:boto:Params: {} 2022-09-12T19:43:59.808 DEBUG:boto:Token: None 2022-09-12T19:43:59.809 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:43:59.810 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+NbqnDm7wpeTdjUMOYZzEDNxtDU= 2022-09-12T19:43:59.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+NbqnDm7wpeTdjUMOYZzEDNxtDU=', 'Content-Length': '0'} 2022-09-12T19:43:59.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000276cc9e710c43d82-00631f8bff-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:43:59.820 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:43:59.821 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:43:59.822 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:43:59.823 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:43:59.824 DEBUG:boto:Method: GET 2022-09-12T19:43:59.825 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:43:59.826 DEBUG:boto:Data: 2022-09-12T19:43:59.827 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.828 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:59.829 DEBUG:boto:Port: 8001 2022-09-12T19:43:59.830 DEBUG:boto:Params: {} 2022-09-12T19:43:59.831 DEBUG:boto:Token: None 2022-09-12T19:43:59.832 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:43:59.833 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SvrqNQStZ3u874t54m/KDXbW1kw= 2022-09-12T19:43:59.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SvrqNQStZ3u874t54m/KDXbW1kw=', 'Content-Length': '0'} 2022-09-12T19:43:59.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001767013a7967a410-00631f8bff-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.837 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:43:59.837 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:43:59.838 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:43:59.839 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:43:59.839 DEBUG:boto:Method: GET 2022-09-12T19:43:59.840 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:43:59.840 DEBUG:boto:Data: 2022-09-12T19:43:59.841 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.841 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:59.842 DEBUG:boto:Port: 8000 2022-09-12T19:43:59.843 DEBUG:boto:Params: {} 2022-09-12T19:43:59.843 DEBUG:boto:Token: None 2022-09-12T19:43:59.844 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:43:59.845 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SvrqNQStZ3u874t54m/KDXbW1kw= 2022-09-12T19:43:59.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SvrqNQStZ3u874t54m/KDXbW1kw=', 'Content-Length': '0'} 2022-09-12T19:43:59.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000882cef2f5b6487ce-00631f8bff-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.850 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:59.851 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:59.852 DEBUG:boto:path=/jdpiak-71/?prefix=e 2022-09-12T19:43:59.853 DEBUG:boto:auth_path=/jdpiak-71/?prefix=e 2022-09-12T19:43:59.855 DEBUG:boto:Method: GET 2022-09-12T19:43:59.856 DEBUG:boto:Path: /jdpiak-71/?prefix=e 2022-09-12T19:43:59.857 DEBUG:boto:Data: 2022-09-12T19:43:59.858 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.859 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:59.860 DEBUG:boto:Port: 8001 2022-09-12T19:43:59.861 DEBUG:boto:Params: {} 2022-09-12T19:43:59.862 DEBUG:boto:Token: None 2022-09-12T19:43:59.863 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/ 2022-09-12T19:43:59.865 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak= 2022-09-12T19:43:59.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak=', 'Content-Length': '0'} 2022-09-12T19:43:59.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095752862a78a78d1-00631f8bff-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.871 DEBUG:boto:b'jdpiak-71e1000falsee2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:59.873 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:59.874 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:59.876 DEBUG:boto:path=/jdpiak-71/?prefix=e 2022-09-12T19:43:59.877 DEBUG:boto:auth_path=/jdpiak-71/?prefix=e 2022-09-12T19:43:59.878 DEBUG:boto:Method: GET 2022-09-12T19:43:59.879 DEBUG:boto:Path: /jdpiak-71/?prefix=e 2022-09-12T19:43:59.880 DEBUG:boto:Data: 2022-09-12T19:43:59.881 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.883 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:59.884 DEBUG:boto:Port: 8000 2022-09-12T19:43:59.886 DEBUG:boto:Params: {} 2022-09-12T19:43:59.888 DEBUG:boto:Token: None 2022-09-12T19:43:59.889 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/ 2022-09-12T19:43:59.890 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak= 2022-09-12T19:43:59.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tYyAD5nL4Remh6kLKX22+w+f4ak=', 'Content-Length': '0'} 2022-09-12T19:43:59.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be67c35ea2fae6f5-00631f8bff-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.895 DEBUG:boto:b'jdpiak-71e1000falsee2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:43:59.896 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:43:59.896 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:43:59.897 DEBUG:boto:Method: HEAD 2022-09-12T19:43:59.897 DEBUG:boto:Path: /jdpiak-71/e 2022-09-12T19:43:59.898 DEBUG:boto:Data: 2022-09-12T19:43:59.898 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.899 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:59.899 DEBUG:boto:Port: 8001 2022-09-12T19:43:59.900 DEBUG:boto:Params: {} 2022-09-12T19:43:59.901 DEBUG:boto:Token: None 2022-09-12T19:43:59.901 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/e 2022-09-12T19:43:59.902 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PrgpVeCrM5UgpHUqb278JKhzJB0= 2022-09-12T19:43:59.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PrgpVeCrM5UgpHUqb278JKhzJB0=', 'Content-Length': '0'} 2022-09-12T19:43:59.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000898343156b47bd4f-00631f8bff-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.906 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:43:59.906 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:43:59.907 DEBUG:boto:Method: HEAD 2022-09-12T19:43:59.908 DEBUG:boto:Path: /jdpiak-71/e 2022-09-12T19:43:59.908 DEBUG:boto:Data: 2022-09-12T19:43:59.909 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.909 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:59.910 DEBUG:boto:Port: 8000 2022-09-12T19:43:59.910 DEBUG:boto:Params: {} 2022-09-12T19:43:59.911 DEBUG:boto:Token: None 2022-09-12T19:43:59.912 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/e 2022-09-12T19:43:59.912 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PrgpVeCrM5UgpHUqb278JKhzJB0= 2022-09-12T19:43:59.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PrgpVeCrM5UgpHUqb278JKhzJB0=', 'Content-Length': '0'} 2022-09-12T19:43:59.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af87b90530ebcbae-00631f8bff-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:43:59.918 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:43:59.919 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:43:59.920 DEBUG:boto:Method: GET 2022-09-12T19:43:59.921 DEBUG:boto:Path: /jdpiak-71/e 2022-09-12T19:43:59.923 DEBUG:boto:Data: 2022-09-12T19:43:59.924 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.925 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:43:59.926 DEBUG:boto:Port: 8001 2022-09-12T19:43:59.927 DEBUG:boto:Params: {} 2022-09-12T19:43:59.928 DEBUG:boto:Token: None 2022-09-12T19:43:59.929 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/e 2022-09-12T19:43:59.930 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ad9325zxMgMeBtL1IJNtj8c36II= 2022-09-12T19:43:59.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ad9325zxMgMeBtL1IJNtj8c36II=', 'Content-Length': '0'} 2022-09-12T19:43:59.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b99f5e90357812dc-00631f8bff-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.937 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:43:59.937 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:43:59.938 DEBUG:boto:Method: GET 2022-09-12T19:43:59.939 DEBUG:boto:Path: /jdpiak-71/e 2022-09-12T19:43:59.940 DEBUG:boto:Data: 2022-09-12T19:43:59.941 DEBUG:boto:Headers: {} 2022-09-12T19:43:59.945 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:43:59.945 DEBUG:boto:Port: 8000 2022-09-12T19:43:59.946 DEBUG:boto:Params: {} 2022-09-12T19:43:59.947 DEBUG:boto:Token: None 2022-09-12T19:43:59.947 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:43:59 GMT /jdpiak-71/e 2022-09-12T19:43:59.948 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ad9325zxMgMeBtL1IJNtj8c36II= 2022-09-12T19:43:59.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:43:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ad9325zxMgMeBtL1IJNtj8c36II=', 'Content-Length': '0'} 2022-09-12T19:43:59.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7e722e74fb00bc7-00631f8bff-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:43:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:43:59.994 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:43:59.995 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:43:59.996 DEBUG:boto:path=/jdpiak-71/?prefix=e 2022-09-12T19:43:59.998 DEBUG:boto:auth_path=/jdpiak-71/?prefix=e 2022-09-12T19:43:59.999 DEBUG:boto:Method: GET 2022-09-12T19:44:00.000 DEBUG:boto:Path: /jdpiak-71/?prefix=e 2022-09-12T19:44:00.001 DEBUG:boto:Data: 2022-09-12T19:44:00.002 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.003 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:00.004 DEBUG:boto:Port: 8001 2022-09-12T19:44:00.005 DEBUG:boto:Params: {} 2022-09-12T19:44:00.007 DEBUG:boto:Token: None 2022-09-12T19:44:00.008 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/ 2022-09-12T19:44:00.009 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qco2s+Dq4a6c9K8izrotvRNNb4s= 2022-09-12T19:44:00.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qco2s+Dq4a6c9K8izrotvRNNb4s=', 'Content-Length': '0'} 2022-09-12T19:44:00.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000471f7084a5aba359-00631f8c00-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.020 DEBUG:boto:b'jdpiak-71e1000falsee2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:00.021 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:00.025 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:00.026 DEBUG:boto:path=/jdpiak-71/?prefix=e 2022-09-12T19:44:00.027 DEBUG:boto:auth_path=/jdpiak-71/?prefix=e 2022-09-12T19:44:00.027 DEBUG:boto:Method: GET 2022-09-12T19:44:00.028 DEBUG:boto:Path: /jdpiak-71/?prefix=e 2022-09-12T19:44:00.029 DEBUG:boto:Data: 2022-09-12T19:44:00.030 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.031 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:00.032 DEBUG:boto:Port: 8000 2022-09-12T19:44:00.033 DEBUG:boto:Params: {} 2022-09-12T19:44:00.034 DEBUG:boto:Token: None 2022-09-12T19:44:00.035 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/ 2022-09-12T19:44:00.036 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qco2s+Dq4a6c9K8izrotvRNNb4s= 2022-09-12T19:44:00.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qco2s+Dq4a6c9K8izrotvRNNb4s=', 'Content-Length': '0'} 2022-09-12T19:44:00.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b46f834b388428b-00631f8c00-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.043 DEBUG:boto:b'jdpiak-71e1000falsee2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:00.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:44:00.045 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:00.046 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:00.047 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:00.047 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:00.048 DEBUG:boto:Method: GET 2022-09-12T19:44:00.049 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:00.049 DEBUG:boto:Data: 2022-09-12T19:44:00.050 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.051 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:00.052 DEBUG:boto:Port: 8001 2022-09-12T19:44:00.052 DEBUG:boto:Params: {} 2022-09-12T19:44:00.053 DEBUG:boto:Token: None 2022-09-12T19:44:00.053 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:00.054 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:j92KKfkmhmTuv3rsoIruZ+VjLm0= 2022-09-12T19:44:00.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:j92KKfkmhmTuv3rsoIruZ+VjLm0=', 'Content-Length': '0'} 2022-09-12T19:44:00.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cfcd5f4a2924525-00631f8c00-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.104 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:00.106 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:00.107 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:00.108 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:00.109 DEBUG:boto:Method: GET 2022-09-12T19:44:00.110 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:00.111 DEBUG:boto:Data: 2022-09-12T19:44:00.112 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.113 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:00.115 DEBUG:boto:Port: 8000 2022-09-12T19:44:00.116 DEBUG:boto:Params: {} 2022-09-12T19:44:00.117 DEBUG:boto:Token: None 2022-09-12T19:44:00.118 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:00.120 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:j92KKfkmhmTuv3rsoIruZ+VjLm0= 2022-09-12T19:44:00.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:j92KKfkmhmTuv3rsoIruZ+VjLm0=', 'Content-Length': '0'} 2022-09-12T19:44:00.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e1651673e1cf5ef-00631f8c00-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.129 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:00.130 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:00.131 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:00.132 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:00.133 DEBUG:boto:Method: HEAD 2022-09-12T19:44:00.134 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:00.135 DEBUG:boto:Data: 2022-09-12T19:44:00.137 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.138 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:00.138 DEBUG:boto:Port: 8001 2022-09-12T19:44:00.139 DEBUG:boto:Params: {} 2022-09-12T19:44:00.143 DEBUG:boto:Token: None 2022-09-12T19:44:00.144 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:00.146 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vrgkmkCdMZfdFIn/ZxicK0RdPNQ= 2022-09-12T19:44:00.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vrgkmkCdMZfdFIn/ZxicK0RdPNQ=', 'Content-Length': '0'} 2022-09-12T19:44:00.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005153cbd1e386bab8-00631f8c00-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.153 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:00.154 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:00.156 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:00.157 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:00.158 DEBUG:boto:Method: HEAD 2022-09-12T19:44:00.159 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:00.160 DEBUG:boto:Data: 2022-09-12T19:44:00.162 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.163 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:00.164 DEBUG:boto:Port: 8000 2022-09-12T19:44:00.165 DEBUG:boto:Params: {} 2022-09-12T19:44:00.166 DEBUG:boto:Token: None 2022-09-12T19:44:00.167 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:00.169 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vrgkmkCdMZfdFIn/ZxicK0RdPNQ= 2022-09-12T19:44:00.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vrgkmkCdMZfdFIn/ZxicK0RdPNQ=', 'Content-Length': '0'} 2022-09-12T19:44:00.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006233aa4bd315894b-00631f8c00-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:44:00.175 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:00.177 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:00.178 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:00.179 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:00.180 DEBUG:boto:Method: GET 2022-09-12T19:44:00.181 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:00.182 DEBUG:boto:Data: 2022-09-12T19:44:00.182 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.183 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:00.184 DEBUG:boto:Port: 8001 2022-09-12T19:44:00.184 DEBUG:boto:Params: {} 2022-09-12T19:44:00.185 DEBUG:boto:Token: None 2022-09-12T19:44:00.186 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:00.187 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:j92KKfkmhmTuv3rsoIruZ+VjLm0= 2022-09-12T19:44:00.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:j92KKfkmhmTuv3rsoIruZ+VjLm0=', 'Content-Length': '0'} 2022-09-12T19:44:00.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007541e8489eebfef9-00631f8c00-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.190 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:00.191 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:00.192 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:00.193 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:00.193 DEBUG:boto:Method: GET 2022-09-12T19:44:00.194 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:00.194 DEBUG:boto:Data: 2022-09-12T19:44:00.195 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.196 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:00.196 DEBUG:boto:Port: 8000 2022-09-12T19:44:00.197 DEBUG:boto:Params: {} 2022-09-12T19:44:00.198 DEBUG:boto:Token: None 2022-09-12T19:44:00.198 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:00.199 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:j92KKfkmhmTuv3rsoIruZ+VjLm0= 2022-09-12T19:44:00.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:j92KKfkmhmTuv3rsoIruZ+VjLm0=', 'Content-Length': '0'} 2022-09-12T19:44:00.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cd8bfe6c0dc139b-00631f8c00-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.204 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:00.205 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:00.206 DEBUG:boto:path=/jdpiak-71/?prefix=f 2022-09-12T19:44:00.207 DEBUG:boto:auth_path=/jdpiak-71/?prefix=f 2022-09-12T19:44:00.207 DEBUG:boto:Method: GET 2022-09-12T19:44:00.208 DEBUG:boto:Path: /jdpiak-71/?prefix=f 2022-09-12T19:44:00.209 DEBUG:boto:Data: 2022-09-12T19:44:00.210 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.211 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:00.211 DEBUG:boto:Port: 8001 2022-09-12T19:44:00.212 DEBUG:boto:Params: {} 2022-09-12T19:44:00.213 DEBUG:boto:Token: None 2022-09-12T19:44:00.213 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/ 2022-09-12T19:44:00.214 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qco2s+Dq4a6c9K8izrotvRNNb4s= 2022-09-12T19:44:00.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qco2s+Dq4a6c9K8izrotvRNNb4s=', 'Content-Length': '0'} 2022-09-12T19:44:00.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e1645d1582b260a-00631f8c00-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.220 DEBUG:boto:b'jdpiak-71f1000falsef2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:00.222 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:00.223 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:00.224 DEBUG:boto:path=/jdpiak-71/?prefix=f 2022-09-12T19:44:00.225 DEBUG:boto:auth_path=/jdpiak-71/?prefix=f 2022-09-12T19:44:00.226 DEBUG:boto:Method: GET 2022-09-12T19:44:00.227 DEBUG:boto:Path: /jdpiak-71/?prefix=f 2022-09-12T19:44:00.227 DEBUG:boto:Data: 2022-09-12T19:44:00.228 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.229 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:00.229 DEBUG:boto:Port: 8000 2022-09-12T19:44:00.230 DEBUG:boto:Params: {} 2022-09-12T19:44:00.231 DEBUG:boto:Token: None 2022-09-12T19:44:00.231 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/ 2022-09-12T19:44:00.232 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qco2s+Dq4a6c9K8izrotvRNNb4s= 2022-09-12T19:44:00.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qco2s+Dq4a6c9K8izrotvRNNb4s=', 'Content-Length': '0'} 2022-09-12T19:44:00.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c7a3f80c9d7ca56-00631f8c00-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.236 DEBUG:boto:b'jdpiak-71f1000falsef2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:00.237 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:00.237 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:00.238 DEBUG:boto:Method: HEAD 2022-09-12T19:44:00.238 DEBUG:boto:Path: /jdpiak-71/f 2022-09-12T19:44:00.239 DEBUG:boto:Data: 2022-09-12T19:44:00.240 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.240 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:00.241 DEBUG:boto:Port: 8001 2022-09-12T19:44:00.241 DEBUG:boto:Params: {} 2022-09-12T19:44:00.242 DEBUG:boto:Token: None 2022-09-12T19:44:00.243 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/f 2022-09-12T19:44:00.244 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/ucV8vB7sUmWLxPwcn57IfJH5gM= 2022-09-12T19:44:00.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/ucV8vB7sUmWLxPwcn57IfJH5gM=', 'Content-Length': '0'} 2022-09-12T19:44:00.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba0233b2651a1515-00631f8c00-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.259 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:00.260 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:00.261 DEBUG:boto:Method: HEAD 2022-09-12T19:44:00.262 DEBUG:boto:Path: /jdpiak-71/f 2022-09-12T19:44:00.263 DEBUG:boto:Data: 2022-09-12T19:44:00.264 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.265 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:00.266 DEBUG:boto:Port: 8000 2022-09-12T19:44:00.270 DEBUG:boto:Params: {} 2022-09-12T19:44:00.271 DEBUG:boto:Token: None 2022-09-12T19:44:00.272 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/f 2022-09-12T19:44:00.273 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/ucV8vB7sUmWLxPwcn57IfJH5gM= 2022-09-12T19:44:00.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/ucV8vB7sUmWLxPwcn57IfJH5gM=', 'Content-Length': '0'} 2022-09-12T19:44:00.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f88e9b6e52c64a88-00631f8c00-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:44:00.287 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:00.288 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:00.289 DEBUG:boto:Method: GET 2022-09-12T19:44:00.290 DEBUG:boto:Path: /jdpiak-71/f 2022-09-12T19:44:00.291 DEBUG:boto:Data: 2022-09-12T19:44:00.292 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.293 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:00.294 DEBUG:boto:Port: 8001 2022-09-12T19:44:00.296 DEBUG:boto:Params: {} 2022-09-12T19:44:00.297 DEBUG:boto:Token: None 2022-09-12T19:44:00.298 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/f 2022-09-12T19:44:00.299 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oYVgURHUoH5n314Sj7FFfX74lcw= 2022-09-12T19:44:00.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oYVgURHUoH5n314Sj7FFfX74lcw=', 'Content-Length': '0'} 2022-09-12T19:44:00.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e388fe49dfbcd945-00631f8c00-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.350 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:00.351 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:00.352 DEBUG:boto:Method: GET 2022-09-12T19:44:00.353 DEBUG:boto:Path: /jdpiak-71/f 2022-09-12T19:44:00.354 DEBUG:boto:Data: 2022-09-12T19:44:00.356 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.358 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:00.362 DEBUG:boto:Port: 8000 2022-09-12T19:44:00.366 DEBUG:boto:Params: {} 2022-09-12T19:44:00.367 DEBUG:boto:Token: None 2022-09-12T19:44:00.368 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/f 2022-09-12T19:44:00.369 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oYVgURHUoH5n314Sj7FFfX74lcw= 2022-09-12T19:44:00.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oYVgURHUoH5n314Sj7FFfX74lcw=', 'Content-Length': '0'} 2022-09-12T19:44:00.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007111aad4b987db25-00631f8c00-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.375 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:00.375 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:00.376 DEBUG:boto:path=/jdpiak-71/?prefix=f 2022-09-12T19:44:00.377 DEBUG:boto:auth_path=/jdpiak-71/?prefix=f 2022-09-12T19:44:00.382 DEBUG:boto:Method: GET 2022-09-12T19:44:00.383 DEBUG:boto:Path: /jdpiak-71/?prefix=f 2022-09-12T19:44:00.383 DEBUG:boto:Data: 2022-09-12T19:44:00.384 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.384 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:00.385 DEBUG:boto:Port: 8001 2022-09-12T19:44:00.385 DEBUG:boto:Params: {} 2022-09-12T19:44:00.386 DEBUG:boto:Token: None 2022-09-12T19:44:00.387 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/ 2022-09-12T19:44:00.388 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qco2s+Dq4a6c9K8izrotvRNNb4s= 2022-09-12T19:44:00.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qco2s+Dq4a6c9K8izrotvRNNb4s=', 'Content-Length': '0'} 2022-09-12T19:44:00.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a8c51edba7d3826-00631f8c00-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.392 DEBUG:boto:b'jdpiak-71f1000falsef2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:00.393 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:00.394 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:00.396 DEBUG:boto:path=/jdpiak-71/?prefix=f 2022-09-12T19:44:00.397 DEBUG:boto:auth_path=/jdpiak-71/?prefix=f 2022-09-12T19:44:00.398 DEBUG:boto:Method: GET 2022-09-12T19:44:00.399 DEBUG:boto:Path: /jdpiak-71/?prefix=f 2022-09-12T19:44:00.400 DEBUG:boto:Data: 2022-09-12T19:44:00.401 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.402 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:00.403 DEBUG:boto:Port: 8000 2022-09-12T19:44:00.404 DEBUG:boto:Params: {} 2022-09-12T19:44:00.405 DEBUG:boto:Token: None 2022-09-12T19:44:00.406 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/ 2022-09-12T19:44:00.407 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qco2s+Dq4a6c9K8izrotvRNNb4s= 2022-09-12T19:44:00.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qco2s+Dq4a6c9K8izrotvRNNb4s=', 'Content-Length': '0'} 2022-09-12T19:44:00.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b78c08d137008a85-00631f8c00-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.413 DEBUG:boto:b'jdpiak-71f1000falsef2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:00.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:44:00.415 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:00.416 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:00.417 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:00.418 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:00.419 DEBUG:boto:Method: GET 2022-09-12T19:44:00.420 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:00.421 DEBUG:boto:Data: 2022-09-12T19:44:00.421 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.422 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:00.423 DEBUG:boto:Port: 8001 2022-09-12T19:44:00.424 DEBUG:boto:Params: {} 2022-09-12T19:44:00.425 DEBUG:boto:Token: None 2022-09-12T19:44:00.427 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:00.427 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZncrG2V9QOgM8Jk+rkr2HdVcYaw= 2022-09-12T19:44:00.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZncrG2V9QOgM8Jk+rkr2HdVcYaw=', 'Content-Length': '0'} 2022-09-12T19:44:00.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce5ff566425385a8-00631f8c00-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.432 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:00.433 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:00.433 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:00.434 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:00.435 DEBUG:boto:Method: GET 2022-09-12T19:44:00.435 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:00.436 DEBUG:boto:Data: 2022-09-12T19:44:00.436 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.437 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:00.438 DEBUG:boto:Port: 8000 2022-09-12T19:44:00.438 DEBUG:boto:Params: {} 2022-09-12T19:44:00.439 DEBUG:boto:Token: None 2022-09-12T19:44:00.440 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:00.440 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZncrG2V9QOgM8Jk+rkr2HdVcYaw= 2022-09-12T19:44:00.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZncrG2V9QOgM8Jk+rkr2HdVcYaw=', 'Content-Length': '0'} 2022-09-12T19:44:00.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005eeac95a94e92d43-00631f8c00-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.445 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:00.445 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:00.446 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:00.446 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:00.447 DEBUG:boto:Method: HEAD 2022-09-12T19:44:00.447 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:00.448 DEBUG:boto:Data: 2022-09-12T19:44:00.449 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.450 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:00.451 DEBUG:boto:Port: 8001 2022-09-12T19:44:00.452 DEBUG:boto:Params: {} 2022-09-12T19:44:00.453 DEBUG:boto:Token: None 2022-09-12T19:44:00.454 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:00.456 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/qvmCYm4OuZhe8xI1qCcsyR/4UU= 2022-09-12T19:44:00.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/qvmCYm4OuZhe8xI1qCcsyR/4UU=', 'Content-Length': '0'} 2022-09-12T19:44:00.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbcbebc12298f841-00631f8c00-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.471 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:00.472 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:00.473 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:00.474 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:00.476 DEBUG:boto:Method: HEAD 2022-09-12T19:44:00.477 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:00.478 DEBUG:boto:Data: 2022-09-12T19:44:00.480 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.481 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:00.483 DEBUG:boto:Port: 8000 2022-09-12T19:44:00.483 DEBUG:boto:Params: {} 2022-09-12T19:44:00.485 DEBUG:boto:Token: None 2022-09-12T19:44:00.486 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:00.488 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/qvmCYm4OuZhe8xI1qCcsyR/4UU= 2022-09-12T19:44:00.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/qvmCYm4OuZhe8xI1qCcsyR/4UU=', 'Content-Length': '0'} 2022-09-12T19:44:00.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bd1061b808e41d1-00631f8c00-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:44:00.501 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:00.502 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:00.503 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:00.503 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:00.504 DEBUG:boto:Method: GET 2022-09-12T19:44:00.504 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:00.505 DEBUG:boto:Data: 2022-09-12T19:44:00.505 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.506 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:00.507 DEBUG:boto:Port: 8001 2022-09-12T19:44:00.508 DEBUG:boto:Params: {} 2022-09-12T19:44:00.509 DEBUG:boto:Token: None 2022-09-12T19:44:00.510 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:00.511 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZncrG2V9QOgM8Jk+rkr2HdVcYaw= 2022-09-12T19:44:00.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZncrG2V9QOgM8Jk+rkr2HdVcYaw=', 'Content-Length': '0'} 2022-09-12T19:44:00.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000970ba2b41da18a81-00631f8c00-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.517 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:00.517 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:00.518 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:00.518 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:00.519 DEBUG:boto:Method: GET 2022-09-12T19:44:00.520 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:00.521 DEBUG:boto:Data: 2022-09-12T19:44:00.521 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.522 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:00.523 DEBUG:boto:Port: 8000 2022-09-12T19:44:00.523 DEBUG:boto:Params: {} 2022-09-12T19:44:00.524 DEBUG:boto:Token: None 2022-09-12T19:44:00.525 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:00.525 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZncrG2V9QOgM8Jk+rkr2HdVcYaw= 2022-09-12T19:44:00.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZncrG2V9QOgM8Jk+rkr2HdVcYaw=', 'Content-Length': '0'} 2022-09-12T19:44:00.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040e8514024bc85ac-00631f8c00-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.532 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:00.532 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:00.533 DEBUG:boto:path=/jdpiak-71/?prefix=g 2022-09-12T19:44:00.534 DEBUG:boto:auth_path=/jdpiak-71/?prefix=g 2022-09-12T19:44:00.536 DEBUG:boto:Method: GET 2022-09-12T19:44:00.536 DEBUG:boto:Path: /jdpiak-71/?prefix=g 2022-09-12T19:44:00.537 DEBUG:boto:Data: 2022-09-12T19:44:00.537 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.538 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:00.539 DEBUG:boto:Port: 8001 2022-09-12T19:44:00.539 DEBUG:boto:Params: {} 2022-09-12T19:44:00.540 DEBUG:boto:Token: None 2022-09-12T19:44:00.541 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/ 2022-09-12T19:44:00.542 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qco2s+Dq4a6c9K8izrotvRNNb4s= 2022-09-12T19:44:00.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qco2s+Dq4a6c9K8izrotvRNNb4s=', 'Content-Length': '0'} 2022-09-12T19:44:00.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9afb7db77e439c9-00631f8c00-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.547 DEBUG:boto:b'jdpiak-71g1000falseg2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:00.549 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:00.549 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:00.550 DEBUG:boto:path=/jdpiak-71/?prefix=g 2022-09-12T19:44:00.551 DEBUG:boto:auth_path=/jdpiak-71/?prefix=g 2022-09-12T19:44:00.552 DEBUG:boto:Method: GET 2022-09-12T19:44:00.552 DEBUG:boto:Path: /jdpiak-71/?prefix=g 2022-09-12T19:44:00.553 DEBUG:boto:Data: 2022-09-12T19:44:00.554 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.554 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:00.555 DEBUG:boto:Port: 8000 2022-09-12T19:44:00.557 DEBUG:boto:Params: {} 2022-09-12T19:44:00.557 DEBUG:boto:Token: None 2022-09-12T19:44:00.558 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/ 2022-09-12T19:44:00.559 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qco2s+Dq4a6c9K8izrotvRNNb4s= 2022-09-12T19:44:00.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qco2s+Dq4a6c9K8izrotvRNNb4s=', 'Content-Length': '0'} 2022-09-12T19:44:00.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c9e98508dcaf06d-00631f8c00-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.569 DEBUG:boto:b'jdpiak-71g1000falseg2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:00.570 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:00.571 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:00.572 DEBUG:boto:Method: HEAD 2022-09-12T19:44:00.573 DEBUG:boto:Path: /jdpiak-71/g 2022-09-12T19:44:00.574 DEBUG:boto:Data: 2022-09-12T19:44:00.575 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.576 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:00.578 DEBUG:boto:Port: 8001 2022-09-12T19:44:00.579 DEBUG:boto:Params: {} 2022-09-12T19:44:00.581 DEBUG:boto:Token: None 2022-09-12T19:44:00.582 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/g 2022-09-12T19:44:00.583 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nW1EnDWDpg917G4dtIem1KYdqPc= 2022-09-12T19:44:00.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nW1EnDWDpg917G4dtIem1KYdqPc=', 'Content-Length': '0'} 2022-09-12T19:44:00.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2aa9799d0e71981-00631f8c00-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.591 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:00.592 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:00.593 DEBUG:boto:Method: HEAD 2022-09-12T19:44:00.594 DEBUG:boto:Path: /jdpiak-71/g 2022-09-12T19:44:00.595 DEBUG:boto:Data: 2022-09-12T19:44:00.596 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.598 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:00.599 DEBUG:boto:Port: 8000 2022-09-12T19:44:00.600 DEBUG:boto:Params: {} 2022-09-12T19:44:00.601 DEBUG:boto:Token: None 2022-09-12T19:44:00.603 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/g 2022-09-12T19:44:00.604 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nW1EnDWDpg917G4dtIem1KYdqPc= 2022-09-12T19:44:00.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nW1EnDWDpg917G4dtIem1KYdqPc=', 'Content-Length': '0'} 2022-09-12T19:44:00.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e96bc6019f846d5-00631f8c00-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:44:00.610 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:00.611 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:00.613 DEBUG:boto:Method: GET 2022-09-12T19:44:00.614 DEBUG:boto:Path: /jdpiak-71/g 2022-09-12T19:44:00.615 DEBUG:boto:Data: 2022-09-12T19:44:00.616 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.617 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:00.618 DEBUG:boto:Port: 8001 2022-09-12T19:44:00.619 DEBUG:boto:Params: {} 2022-09-12T19:44:00.621 DEBUG:boto:Token: None 2022-09-12T19:44:00.622 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/g 2022-09-12T19:44:00.623 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/Fu3Ws3BedkCNxgfB6TF+cavTgA= 2022-09-12T19:44:00.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/Fu3Ws3BedkCNxgfB6TF+cavTgA=', 'Content-Length': '0'} 2022-09-12T19:44:00.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b04c82021ca75c9a-00631f8c00-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.629 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:00.630 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:00.631 DEBUG:boto:Method: GET 2022-09-12T19:44:00.632 DEBUG:boto:Path: /jdpiak-71/g 2022-09-12T19:44:00.633 DEBUG:boto:Data: 2022-09-12T19:44:00.635 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.636 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:00.638 DEBUG:boto:Port: 8000 2022-09-12T19:44:00.639 DEBUG:boto:Params: {} 2022-09-12T19:44:00.640 DEBUG:boto:Token: None 2022-09-12T19:44:00.641 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/g 2022-09-12T19:44:00.642 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/Fu3Ws3BedkCNxgfB6TF+cavTgA= 2022-09-12T19:44:00.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/Fu3Ws3BedkCNxgfB6TF+cavTgA=', 'Content-Length': '0'} 2022-09-12T19:44:00.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0e13b7f428fe1eb-00631f8c00-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.648 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:00.649 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:00.649 DEBUG:boto:path=/jdpiak-71/?prefix=g 2022-09-12T19:44:00.650 DEBUG:boto:auth_path=/jdpiak-71/?prefix=g 2022-09-12T19:44:00.650 DEBUG:boto:Method: GET 2022-09-12T19:44:00.651 DEBUG:boto:Path: /jdpiak-71/?prefix=g 2022-09-12T19:44:00.652 DEBUG:boto:Data: 2022-09-12T19:44:00.653 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.654 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:00.654 DEBUG:boto:Port: 8001 2022-09-12T19:44:00.655 DEBUG:boto:Params: {} 2022-09-12T19:44:00.657 DEBUG:boto:Token: None 2022-09-12T19:44:00.658 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/ 2022-09-12T19:44:00.659 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qco2s+Dq4a6c9K8izrotvRNNb4s= 2022-09-12T19:44:00.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qco2s+Dq4a6c9K8izrotvRNNb4s=', 'Content-Length': '0'} 2022-09-12T19:44:00.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd0d93141832d089-00631f8c00-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.710 DEBUG:boto:b'jdpiak-71g1000falseg2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:00.713 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:00.714 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:00.715 DEBUG:boto:path=/jdpiak-71/?prefix=g 2022-09-12T19:44:00.716 DEBUG:boto:auth_path=/jdpiak-71/?prefix=g 2022-09-12T19:44:00.717 DEBUG:boto:Method: GET 2022-09-12T19:44:00.718 DEBUG:boto:Path: /jdpiak-71/?prefix=g 2022-09-12T19:44:00.719 DEBUG:boto:Data: 2022-09-12T19:44:00.720 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.721 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:00.722 DEBUG:boto:Port: 8000 2022-09-12T19:44:00.724 DEBUG:boto:Params: {} 2022-09-12T19:44:00.725 DEBUG:boto:Token: None 2022-09-12T19:44:00.727 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/ 2022-09-12T19:44:00.729 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qco2s+Dq4a6c9K8izrotvRNNb4s= 2022-09-12T19:44:00.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qco2s+Dq4a6c9K8izrotvRNNb4s=', 'Content-Length': '0'} 2022-09-12T19:44:00.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088697890e7316461-00631f8c00-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.734 DEBUG:boto:b'jdpiak-71g1000falseg2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:00.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:44:00.737 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:00.739 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:00.740 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:00.741 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:00.742 DEBUG:boto:Method: GET 2022-09-12T19:44:00.744 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:00.745 DEBUG:boto:Data: 2022-09-12T19:44:00.746 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.747 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:00.749 DEBUG:boto:Port: 8001 2022-09-12T19:44:00.750 DEBUG:boto:Params: {} 2022-09-12T19:44:00.751 DEBUG:boto:Token: None 2022-09-12T19:44:00.753 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:00.755 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HayhzjNS0tNcJ1n8Aw0ltYmJ5aI= 2022-09-12T19:44:00.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HayhzjNS0tNcJ1n8Aw0ltYmJ5aI=', 'Content-Length': '0'} 2022-09-12T19:44:00.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039e640418f70d61e-00631f8c00-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.764 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:00.765 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:00.766 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:00.767 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:00.768 DEBUG:boto:Method: GET 2022-09-12T19:44:00.769 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:00.770 DEBUG:boto:Data: 2022-09-12T19:44:00.772 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.773 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:00.775 DEBUG:boto:Port: 8000 2022-09-12T19:44:00.776 DEBUG:boto:Params: {} 2022-09-12T19:44:00.777 DEBUG:boto:Token: None 2022-09-12T19:44:00.778 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:00.780 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HayhzjNS0tNcJ1n8Aw0ltYmJ5aI= 2022-09-12T19:44:00.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HayhzjNS0tNcJ1n8Aw0ltYmJ5aI=', 'Content-Length': '0'} 2022-09-12T19:44:00.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000314c04f2a6823b75-00631f8c00-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.786 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:00.787 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:00.788 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:00.789 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:00.790 DEBUG:boto:Method: HEAD 2022-09-12T19:44:00.791 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:00.791 DEBUG:boto:Data: 2022-09-12T19:44:00.792 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.793 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:00.794 DEBUG:boto:Port: 8001 2022-09-12T19:44:00.795 DEBUG:boto:Params: {} 2022-09-12T19:44:00.796 DEBUG:boto:Token: None 2022-09-12T19:44:00.797 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:00.798 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WCvYlLAvvuscKyFEZqAshENXIGM= 2022-09-12T19:44:00.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WCvYlLAvvuscKyFEZqAshENXIGM=', 'Content-Length': '0'} 2022-09-12T19:44:00.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b33fc275711743f-00631f8c00-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.803 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:00.804 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:00.805 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:00.806 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:00.807 DEBUG:boto:Method: HEAD 2022-09-12T19:44:00.808 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:00.809 DEBUG:boto:Data: 2022-09-12T19:44:00.810 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.811 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:00.812 DEBUG:boto:Port: 8000 2022-09-12T19:44:00.813 DEBUG:boto:Params: {} 2022-09-12T19:44:00.814 DEBUG:boto:Token: None 2022-09-12T19:44:00.815 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:00.816 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WCvYlLAvvuscKyFEZqAshENXIGM= 2022-09-12T19:44:00.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WCvYlLAvvuscKyFEZqAshENXIGM=', 'Content-Length': '0'} 2022-09-12T19:44:00.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e7917cec6323cee-00631f8c00-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:44:00.822 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:00.824 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:00.825 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:00.826 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:00.827 DEBUG:boto:Method: GET 2022-09-12T19:44:00.828 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:00.829 DEBUG:boto:Data: 2022-09-12T19:44:00.830 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.831 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:00.832 DEBUG:boto:Port: 8001 2022-09-12T19:44:00.833 DEBUG:boto:Params: {} 2022-09-12T19:44:00.835 DEBUG:boto:Token: None 2022-09-12T19:44:00.836 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:00.837 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HayhzjNS0tNcJ1n8Aw0ltYmJ5aI= 2022-09-12T19:44:00.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HayhzjNS0tNcJ1n8Aw0ltYmJ5aI=', 'Content-Length': '0'} 2022-09-12T19:44:00.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e6bb1f44733ebce-00631f8c00-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.843 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:00.844 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:00.846 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:00.847 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:00.848 DEBUG:boto:Method: GET 2022-09-12T19:44:00.849 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:00.850 DEBUG:boto:Data: 2022-09-12T19:44:00.851 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.853 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:00.854 DEBUG:boto:Port: 8000 2022-09-12T19:44:00.855 DEBUG:boto:Params: {} 2022-09-12T19:44:00.856 DEBUG:boto:Token: None 2022-09-12T19:44:00.857 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:00.859 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HayhzjNS0tNcJ1n8Aw0ltYmJ5aI= 2022-09-12T19:44:00.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HayhzjNS0tNcJ1n8Aw0ltYmJ5aI=', 'Content-Length': '0'} 2022-09-12T19:44:00.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098bdbda4bee6a8e9-00631f8c00-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.869 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:00.870 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:00.871 DEBUG:boto:path=/jdpiak-71/?prefix=h 2022-09-12T19:44:00.872 DEBUG:boto:auth_path=/jdpiak-71/?prefix=h 2022-09-12T19:44:00.872 DEBUG:boto:Method: GET 2022-09-12T19:44:00.873 DEBUG:boto:Path: /jdpiak-71/?prefix=h 2022-09-12T19:44:00.873 DEBUG:boto:Data: 2022-09-12T19:44:00.874 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.874 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:00.875 DEBUG:boto:Port: 8001 2022-09-12T19:44:00.876 DEBUG:boto:Params: {} 2022-09-12T19:44:00.876 DEBUG:boto:Token: None 2022-09-12T19:44:00.877 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/ 2022-09-12T19:44:00.878 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qco2s+Dq4a6c9K8izrotvRNNb4s= 2022-09-12T19:44:00.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qco2s+Dq4a6c9K8izrotvRNNb4s=', 'Content-Length': '0'} 2022-09-12T19:44:00.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c479930901822e3-00631f8c00-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.882 DEBUG:boto:b'jdpiak-71h1000falseh2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:00.883 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:00.883 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:00.884 DEBUG:boto:path=/jdpiak-71/?prefix=h 2022-09-12T19:44:00.884 DEBUG:boto:auth_path=/jdpiak-71/?prefix=h 2022-09-12T19:44:00.885 DEBUG:boto:Method: GET 2022-09-12T19:44:00.886 DEBUG:boto:Path: /jdpiak-71/?prefix=h 2022-09-12T19:44:00.886 DEBUG:boto:Data: 2022-09-12T19:44:00.887 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.888 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:00.888 DEBUG:boto:Port: 8000 2022-09-12T19:44:00.889 DEBUG:boto:Params: {} 2022-09-12T19:44:00.889 DEBUG:boto:Token: None 2022-09-12T19:44:00.890 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/ 2022-09-12T19:44:00.891 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qco2s+Dq4a6c9K8izrotvRNNb4s= 2022-09-12T19:44:00.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qco2s+Dq4a6c9K8izrotvRNNb4s=', 'Content-Length': '0'} 2022-09-12T19:44:00.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eaa0435354cab1ca-00631f8c00-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.937 DEBUG:boto:b'jdpiak-71h1000falseh2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:00.941 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:00.942 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:00.943 DEBUG:boto:Method: HEAD 2022-09-12T19:44:00.944 DEBUG:boto:Path: /jdpiak-71/h 2022-09-12T19:44:00.945 DEBUG:boto:Data: 2022-09-12T19:44:00.945 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.946 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:00.947 DEBUG:boto:Port: 8001 2022-09-12T19:44:00.947 DEBUG:boto:Params: {} 2022-09-12T19:44:00.948 DEBUG:boto:Token: None 2022-09-12T19:44:00.948 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/h 2022-09-12T19:44:00.949 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:phjIZ0HSswz0XrzDte97/tNYjoA= 2022-09-12T19:44:00.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:phjIZ0HSswz0XrzDte97/tNYjoA=', 'Content-Length': '0'} 2022-09-12T19:44:00.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089b74d571b9238f3-00631f8c00-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.953 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:00.954 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:00.954 DEBUG:boto:Method: HEAD 2022-09-12T19:44:00.955 DEBUG:boto:Path: /jdpiak-71/h 2022-09-12T19:44:00.956 DEBUG:boto:Data: 2022-09-12T19:44:00.956 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.957 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:00.957 DEBUG:boto:Port: 8000 2022-09-12T19:44:00.958 DEBUG:boto:Params: {} 2022-09-12T19:44:00.958 DEBUG:boto:Token: None 2022-09-12T19:44:00.959 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/h 2022-09-12T19:44:00.960 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:phjIZ0HSswz0XrzDte97/tNYjoA= 2022-09-12T19:44:00.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:phjIZ0HSswz0XrzDte97/tNYjoA=', 'Content-Length': '0'} 2022-09-12T19:44:00.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000563fcf5b958f2203-00631f8c00-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:44:00.966 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:00.966 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:00.967 DEBUG:boto:Method: GET 2022-09-12T19:44:00.968 DEBUG:boto:Path: /jdpiak-71/h 2022-09-12T19:44:00.969 DEBUG:boto:Data: 2022-09-12T19:44:00.970 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.971 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:00.972 DEBUG:boto:Port: 8001 2022-09-12T19:44:00.972 DEBUG:boto:Params: {} 2022-09-12T19:44:00.974 DEBUG:boto:Token: None 2022-09-12T19:44:00.976 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/h 2022-09-12T19:44:00.976 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nSjrs6kP9Hbke64oZbNy2o18C/0= 2022-09-12T19:44:00.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nSjrs6kP9Hbke64oZbNy2o18C/0=', 'Content-Length': '0'} 2022-09-12T19:44:00.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f053ad3449ef5c2-00631f8c00-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:00.980 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:00.981 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:00.982 DEBUG:boto:Method: GET 2022-09-12T19:44:00.982 DEBUG:boto:Path: /jdpiak-71/h 2022-09-12T19:44:00.984 DEBUG:boto:Data: 2022-09-12T19:44:00.986 DEBUG:boto:Headers: {} 2022-09-12T19:44:00.987 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:00.988 DEBUG:boto:Port: 8000 2022-09-12T19:44:00.990 DEBUG:boto:Params: {} 2022-09-12T19:44:00.991 DEBUG:boto:Token: None 2022-09-12T19:44:00.992 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:00 GMT /jdpiak-71/h 2022-09-12T19:44:00.994 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nSjrs6kP9Hbke64oZbNy2o18C/0= 2022-09-12T19:44:00.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nSjrs6kP9Hbke64oZbNy2o18C/0=', 'Content-Length': '0'} 2022-09-12T19:44:00.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4d9e74b485b4505-00631f8c00-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.001 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:01.002 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:01.003 DEBUG:boto:path=/jdpiak-71/?prefix=h 2022-09-12T19:44:01.004 DEBUG:boto:auth_path=/jdpiak-71/?prefix=h 2022-09-12T19:44:01.006 DEBUG:boto:Method: GET 2022-09-12T19:44:01.010 DEBUG:boto:Path: /jdpiak-71/?prefix=h 2022-09-12T19:44:01.011 DEBUG:boto:Data: 2022-09-12T19:44:01.012 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.012 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:01.013 DEBUG:boto:Port: 8001 2022-09-12T19:44:01.014 DEBUG:boto:Params: {} 2022-09-12T19:44:01.014 DEBUG:boto:Token: None 2022-09-12T19:44:01.015 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/ 2022-09-12T19:44:01.016 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Oa3VHEgZbrxcildJS3kFk/o1uI= 2022-09-12T19:44:01.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Oa3VHEgZbrxcildJS3kFk/o1uI=', 'Content-Length': '0'} 2022-09-12T19:44:01.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b30e63081ee0b1e-00631f8c01-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.020 DEBUG:boto:b'jdpiak-71h1000falseh2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:01.021 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:01.022 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:01.023 DEBUG:boto:path=/jdpiak-71/?prefix=h 2022-09-12T19:44:01.023 DEBUG:boto:auth_path=/jdpiak-71/?prefix=h 2022-09-12T19:44:01.024 DEBUG:boto:Method: GET 2022-09-12T19:44:01.025 DEBUG:boto:Path: /jdpiak-71/?prefix=h 2022-09-12T19:44:01.025 DEBUG:boto:Data: 2022-09-12T19:44:01.026 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.027 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:01.027 DEBUG:boto:Port: 8000 2022-09-12T19:44:01.028 DEBUG:boto:Params: {} 2022-09-12T19:44:01.029 DEBUG:boto:Token: None 2022-09-12T19:44:01.029 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/ 2022-09-12T19:44:01.030 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Oa3VHEgZbrxcildJS3kFk/o1uI= 2022-09-12T19:44:01.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Oa3VHEgZbrxcildJS3kFk/o1uI=', 'Content-Length': '0'} 2022-09-12T19:44:01.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064783ad610de3328-00631f8c01-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.074 DEBUG:boto:b'jdpiak-71h1000falseh2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:01.075 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-71 zones={test-zone3, test-zone1} 2022-09-12T19:44:01.076 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-71 zones={test-zone3, test-zone2} 2022-09-12T19:44:01.076 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:01.077 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:01.079 DEBUG:boto:Method: HEAD 2022-09-12T19:44:01.079 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:44:01.080 DEBUG:boto:Data: 2022-09-12T19:44:01.080 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.081 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:01.081 DEBUG:boto:Port: 8001 2022-09-12T19:44:01.082 DEBUG:boto:Params: {} 2022-09-12T19:44:01.083 DEBUG:boto:Token: None 2022-09-12T19:44:01.083 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/ 2022-09-12T19:44:01.084 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8aVCpiYuJaMrEkDXWS9+bdWkWCE= 2022-09-12T19:44:01.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8aVCpiYuJaMrEkDXWS9+bdWkWCE=', 'Content-Length': '0'} 2022-09-12T19:44:01.090 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059459e826e3dfb63-00631f8c01-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.092 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:01.093 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:01.094 DEBUG:boto:Method: HEAD 2022-09-12T19:44:01.095 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:44:01.096 DEBUG:boto:Data: 2022-09-12T19:44:01.097 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.097 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:01.098 DEBUG:boto:Port: 8000 2022-09-12T19:44:01.098 DEBUG:boto:Params: {} 2022-09-12T19:44:01.099 DEBUG:boto:Token: None 2022-09-12T19:44:01.100 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/ 2022-09-12T19:44:01.100 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8aVCpiYuJaMrEkDXWS9+bdWkWCE= 2022-09-12T19:44:01.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8aVCpiYuJaMrEkDXWS9+bdWkWCE=', 'Content-Length': '0'} 2022-09-12T19:44:01.104 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004177494b76fdda6c-00631f8c01-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.105 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:44:01.106 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:01.107 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:01.108 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:01.109 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:01.110 DEBUG:boto:Method: GET 2022-09-12T19:44:01.111 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:01.112 DEBUG:boto:Data: 2022-09-12T19:44:01.113 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.114 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:01.115 DEBUG:boto:Port: 8001 2022-09-12T19:44:01.116 DEBUG:boto:Params: {} 2022-09-12T19:44:01.117 DEBUG:boto:Token: None 2022-09-12T19:44:01.118 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/?versions 2022-09-12T19:44:01.119 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vnhtqEHKMqxtCbBthMOmo8iilVY= 2022-09-12T19:44:01.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vnhtqEHKMqxtCbBthMOmo8iilVY=', 'Content-Length': '0'} 2022-09-12T19:44:01.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3f7b79a28929c7c-00631f8c01-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.124 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:01.126 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:44:01.126 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:44:01.127 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:44:01.128 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:44:01.129 DEBUG:tasks.rgw_multi.zone_rados:o=e 2022-09-12T19:44:01.130 DEBUG:tasks.rgw_multi.zone_rados:o=f 2022-09-12T19:44:01.131 DEBUG:tasks.rgw_multi.zone_rados:o=g 2022-09-12T19:44:01.132 DEBUG:tasks.rgw_multi.zone_rados:o=h 2022-09-12T19:44:01.133 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:44:01.134 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:01.136 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:01.137 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:01.138 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:01.139 DEBUG:boto:Method: GET 2022-09-12T19:44:01.139 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:01.140 DEBUG:boto:Data: 2022-09-12T19:44:01.140 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.141 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:01.141 DEBUG:boto:Port: 8000 2022-09-12T19:44:01.142 DEBUG:boto:Params: {} 2022-09-12T19:44:01.142 DEBUG:boto:Token: None 2022-09-12T19:44:01.143 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/?versions 2022-09-12T19:44:01.144 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vnhtqEHKMqxtCbBthMOmo8iilVY= 2022-09-12T19:44:01.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vnhtqEHKMqxtCbBthMOmo8iilVY=', 'Content-Length': '0'} 2022-09-12T19:44:01.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa63dad659dbc178-00631f8c01-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.150 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:01.153 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:44:01.153 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:44:01.154 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:44:01.155 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:44:01.162 DEBUG:tasks.rgw_multi.zone_rados:o=e 2022-09-12T19:44:01.163 DEBUG:tasks.rgw_multi.zone_rados:o=f 2022-09-12T19:44:01.164 DEBUG:tasks.rgw_multi.zone_rados:o=g 2022-09-12T19:44:01.165 DEBUG:tasks.rgw_multi.zone_rados:o=h 2022-09-12T19:44:01.166 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:01.167 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:01.168 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:01.168 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:01.169 DEBUG:boto:Method: GET 2022-09-12T19:44:01.170 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:01.171 DEBUG:boto:Data: 2022-09-12T19:44:01.172 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.173 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:01.174 DEBUG:boto:Port: 8001 2022-09-12T19:44:01.175 DEBUG:boto:Params: {} 2022-09-12T19:44:01.176 DEBUG:boto:Token: None 2022-09-12T19:44:01.177 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/?versions 2022-09-12T19:44:01.178 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vnhtqEHKMqxtCbBthMOmo8iilVY= 2022-09-12T19:44:01.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vnhtqEHKMqxtCbBthMOmo8iilVY=', 'Content-Length': '0'} 2022-09-12T19:44:01.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007be1990c40896a6d-00631f8c01-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.183 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:01.186 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:01.187 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:01.188 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:01.189 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:01.190 DEBUG:boto:Method: GET 2022-09-12T19:44:01.191 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:01.192 DEBUG:boto:Data: 2022-09-12T19:44:01.192 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.194 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:01.195 DEBUG:boto:Port: 8000 2022-09-12T19:44:01.196 DEBUG:boto:Params: {} 2022-09-12T19:44:01.197 DEBUG:boto:Token: None 2022-09-12T19:44:01.198 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/?versions 2022-09-12T19:44:01.199 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vnhtqEHKMqxtCbBthMOmo8iilVY= 2022-09-12T19:44:01.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vnhtqEHKMqxtCbBthMOmo8iilVY=', 'Content-Length': '0'} 2022-09-12T19:44:01.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d1d79cdffd9ff0f-00631f8c01-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.205 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:01.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:44:01.208 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:01.209 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:01.210 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:01.211 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:01.212 DEBUG:boto:Method: GET 2022-09-12T19:44:01.213 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:44:01.214 DEBUG:boto:Data: 2022-09-12T19:44:01.215 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.215 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:01.216 DEBUG:boto:Port: 8001 2022-09-12T19:44:01.217 DEBUG:boto:Params: {} 2022-09-12T19:44:01.219 DEBUG:boto:Token: None 2022-09-12T19:44:01.220 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:44:01.221 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DyXUCuL12kLjhBUuzZCAp6Rl0vA= 2022-09-12T19:44:01.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DyXUCuL12kLjhBUuzZCAp6Rl0vA=', 'Content-Length': '0'} 2022-09-12T19:44:01.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0a4b232d6ce5e1f-00631f8c01-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.227 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:01.228 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:01.229 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:01.230 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:01.231 DEBUG:boto:Method: GET 2022-09-12T19:44:01.232 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:44:01.233 DEBUG:boto:Data: 2022-09-12T19:44:01.235 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.236 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:01.237 DEBUG:boto:Port: 8000 2022-09-12T19:44:01.238 DEBUG:boto:Params: {} 2022-09-12T19:44:01.239 DEBUG:boto:Token: None 2022-09-12T19:44:01.240 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:44:01.241 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DyXUCuL12kLjhBUuzZCAp6Rl0vA= 2022-09-12T19:44:01.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DyXUCuL12kLjhBUuzZCAp6Rl0vA=', 'Content-Length': '0'} 2022-09-12T19:44:01.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc1d6341aa4ce959-00631f8c01-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.246 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:01.247 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:01.248 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:01.249 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:01.250 DEBUG:boto:Method: HEAD 2022-09-12T19:44:01.251 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:44:01.251 DEBUG:boto:Data: 2022-09-12T19:44:01.252 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.253 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:01.254 DEBUG:boto:Port: 8001 2022-09-12T19:44:01.255 DEBUG:boto:Params: {} 2022-09-12T19:44:01.256 DEBUG:boto:Token: None 2022-09-12T19:44:01.257 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:44:01.258 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ReCDBrHk4dFE0df32sbCyvJyEmA= 2022-09-12T19:44:01.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ReCDBrHk4dFE0df32sbCyvJyEmA=', 'Content-Length': '0'} 2022-09-12T19:44:01.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a0626df2bae04a3-00631f8c01-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.263 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:01.265 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:01.265 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:01.266 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:01.267 DEBUG:boto:Method: HEAD 2022-09-12T19:44:01.268 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:44:01.269 DEBUG:boto:Data: 2022-09-12T19:44:01.270 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.270 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:01.271 DEBUG:boto:Port: 8000 2022-09-12T19:44:01.272 DEBUG:boto:Params: {} 2022-09-12T19:44:01.273 DEBUG:boto:Token: None 2022-09-12T19:44:01.274 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:44:01.275 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ReCDBrHk4dFE0df32sbCyvJyEmA= 2022-09-12T19:44:01.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ReCDBrHk4dFE0df32sbCyvJyEmA=', 'Content-Length': '0'} 2022-09-12T19:44:01.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c82839e4714e8ae-00631f8c01-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:44:01.281 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:01.283 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:01.284 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:01.285 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:01.286 DEBUG:boto:Method: GET 2022-09-12T19:44:01.287 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:44:01.288 DEBUG:boto:Data: 2022-09-12T19:44:01.289 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.289 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:01.290 DEBUG:boto:Port: 8001 2022-09-12T19:44:01.291 DEBUG:boto:Params: {} 2022-09-12T19:44:01.292 DEBUG:boto:Token: None 2022-09-12T19:44:01.293 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:44:01.293 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DyXUCuL12kLjhBUuzZCAp6Rl0vA= 2022-09-12T19:44:01.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DyXUCuL12kLjhBUuzZCAp6Rl0vA=', 'Content-Length': '0'} 2022-09-12T19:44:01.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063ab232de27a2220-00631f8c01-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.298 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:01.299 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:01.299 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:01.300 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:01.301 DEBUG:boto:Method: GET 2022-09-12T19:44:01.302 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:44:01.303 DEBUG:boto:Data: 2022-09-12T19:44:01.304 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.305 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:01.305 DEBUG:boto:Port: 8000 2022-09-12T19:44:01.306 DEBUG:boto:Params: {} 2022-09-12T19:44:01.307 DEBUG:boto:Token: None 2022-09-12T19:44:01.308 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:44:01.308 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DyXUCuL12kLjhBUuzZCAp6Rl0vA= 2022-09-12T19:44:01.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DyXUCuL12kLjhBUuzZCAp6Rl0vA=', 'Content-Length': '0'} 2022-09-12T19:44:01.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac1ecf53050224bb-00631f8c01-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.312 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:01.313 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:01.314 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:44:01.315 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:44:01.316 DEBUG:boto:Method: GET 2022-09-12T19:44:01.317 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:44:01.318 DEBUG:boto:Data: 2022-09-12T19:44:01.319 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.319 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:01.320 DEBUG:boto:Port: 8001 2022-09-12T19:44:01.322 DEBUG:boto:Params: {} 2022-09-12T19:44:01.323 DEBUG:boto:Token: None 2022-09-12T19:44:01.324 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/ 2022-09-12T19:44:01.324 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Oa3VHEgZbrxcildJS3kFk/o1uI= 2022-09-12T19:44:01.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Oa3VHEgZbrxcildJS3kFk/o1uI=', 'Content-Length': '0'} 2022-09-12T19:44:01.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041809c6fbf3b9083-00631f8c01-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.329 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:01.334 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:01.336 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:01.337 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:44:01.338 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:44:01.339 DEBUG:boto:Method: GET 2022-09-12T19:44:01.340 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:44:01.341 DEBUG:boto:Data: 2022-09-12T19:44:01.342 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.343 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:01.345 DEBUG:boto:Port: 8000 2022-09-12T19:44:01.346 DEBUG:boto:Params: {} 2022-09-12T19:44:01.347 DEBUG:boto:Token: None 2022-09-12T19:44:01.349 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/ 2022-09-12T19:44:01.351 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Oa3VHEgZbrxcildJS3kFk/o1uI= 2022-09-12T19:44:01.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Oa3VHEgZbrxcildJS3kFk/o1uI=', 'Content-Length': '0'} 2022-09-12T19:44:01.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a24de6b14793f5d5-00631f8c01-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.360 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:01.362 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:01.363 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:01.364 DEBUG:boto:Method: HEAD 2022-09-12T19:44:01.365 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:44:01.366 DEBUG:boto:Data: 2022-09-12T19:44:01.367 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.368 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:01.369 DEBUG:boto:Port: 8001 2022-09-12T19:44:01.370 DEBUG:boto:Params: {} 2022-09-12T19:44:01.371 DEBUG:boto:Token: None 2022-09-12T19:44:01.372 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/a 2022-09-12T19:44:01.373 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tGMbS4+Ig9/xA+1FiBGqOP2qnf4= 2022-09-12T19:44:01.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tGMbS4+Ig9/xA+1FiBGqOP2qnf4=', 'Content-Length': '0'} 2022-09-12T19:44:01.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a798148b448cb71-00631f8c01-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.377 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:01.378 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:01.379 DEBUG:boto:Method: HEAD 2022-09-12T19:44:01.379 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:44:01.380 DEBUG:boto:Data: 2022-09-12T19:44:01.381 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.381 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:01.382 DEBUG:boto:Port: 8000 2022-09-12T19:44:01.383 DEBUG:boto:Params: {} 2022-09-12T19:44:01.384 DEBUG:boto:Token: None 2022-09-12T19:44:01.384 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/a 2022-09-12T19:44:01.385 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tGMbS4+Ig9/xA+1FiBGqOP2qnf4= 2022-09-12T19:44:01.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tGMbS4+Ig9/xA+1FiBGqOP2qnf4=', 'Content-Length': '0'} 2022-09-12T19:44:01.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a83db4a7b54aa8ad-00631f8c01-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:44:01.389 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:01.390 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:01.390 DEBUG:boto:Method: GET 2022-09-12T19:44:01.391 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:44:01.392 DEBUG:boto:Data: 2022-09-12T19:44:01.392 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.393 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:01.393 DEBUG:boto:Port: 8001 2022-09-12T19:44:01.394 DEBUG:boto:Params: {} 2022-09-12T19:44:01.395 DEBUG:boto:Token: None 2022-09-12T19:44:01.396 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/a 2022-09-12T19:44:01.396 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SnGHsCZ+LcOhXO6yDcU5US1ri50= 2022-09-12T19:44:01.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SnGHsCZ+LcOhXO6yDcU5US1ri50=', 'Content-Length': '0'} 2022-09-12T19:44:01.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c301ffe664a3248-00631f8c01-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.403 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:01.403 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:01.405 DEBUG:boto:Method: GET 2022-09-12T19:44:01.405 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:44:01.407 DEBUG:boto:Data: 2022-09-12T19:44:01.408 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.408 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:01.409 DEBUG:boto:Port: 8000 2022-09-12T19:44:01.409 DEBUG:boto:Params: {} 2022-09-12T19:44:01.410 DEBUG:boto:Token: None 2022-09-12T19:44:01.411 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/a 2022-09-12T19:44:01.412 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SnGHsCZ+LcOhXO6yDcU5US1ri50= 2022-09-12T19:44:01.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SnGHsCZ+LcOhXO6yDcU5US1ri50=', 'Content-Length': '0'} 2022-09-12T19:44:01.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004bbbd4ef0246c57-00631f8c01-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.462 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:01.463 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:01.464 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:44:01.465 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:44:01.467 DEBUG:boto:Method: GET 2022-09-12T19:44:01.470 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:44:01.470 DEBUG:boto:Data: 2022-09-12T19:44:01.471 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.472 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:01.473 DEBUG:boto:Port: 8001 2022-09-12T19:44:01.473 DEBUG:boto:Params: {} 2022-09-12T19:44:01.474 DEBUG:boto:Token: None 2022-09-12T19:44:01.475 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/ 2022-09-12T19:44:01.475 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Oa3VHEgZbrxcildJS3kFk/o1uI= 2022-09-12T19:44:01.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Oa3VHEgZbrxcildJS3kFk/o1uI=', 'Content-Length': '0'} 2022-09-12T19:44:01.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcea15d5a4da44f3-00631f8c01-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.484 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:01.485 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:01.489 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:01.490 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:44:01.490 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:44:01.491 DEBUG:boto:Method: GET 2022-09-12T19:44:01.491 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:44:01.492 DEBUG:boto:Data: 2022-09-12T19:44:01.493 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.493 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:01.494 DEBUG:boto:Port: 8000 2022-09-12T19:44:01.494 DEBUG:boto:Params: {} 2022-09-12T19:44:01.495 DEBUG:boto:Token: None 2022-09-12T19:44:01.496 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/ 2022-09-12T19:44:01.497 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Oa3VHEgZbrxcildJS3kFk/o1uI= 2022-09-12T19:44:01.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Oa3VHEgZbrxcildJS3kFk/o1uI=', 'Content-Length': '0'} 2022-09-12T19:44:01.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000077465eb37cd0039-00631f8c01-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.502 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:01.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:44:01.503 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:01.504 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:01.505 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:01.505 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:01.507 DEBUG:boto:Method: GET 2022-09-12T19:44:01.507 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:44:01.508 DEBUG:boto:Data: 2022-09-12T19:44:01.508 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.509 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:01.510 DEBUG:boto:Port: 8001 2022-09-12T19:44:01.510 DEBUG:boto:Params: {} 2022-09-12T19:44:01.511 DEBUG:boto:Token: None 2022-09-12T19:44:01.512 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:44:01.512 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KBApgsAzf2xXl8w4WMXJBLGBX9s= 2022-09-12T19:44:01.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KBApgsAzf2xXl8w4WMXJBLGBX9s=', 'Content-Length': '0'} 2022-09-12T19:44:01.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4135796736c1ae7-00631f8c01-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.516 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:01.517 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:01.517 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:01.518 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:01.518 DEBUG:boto:Method: GET 2022-09-12T19:44:01.519 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:44:01.520 DEBUG:boto:Data: 2022-09-12T19:44:01.521 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.521 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:01.522 DEBUG:boto:Port: 8000 2022-09-12T19:44:01.522 DEBUG:boto:Params: {} 2022-09-12T19:44:01.523 DEBUG:boto:Token: None 2022-09-12T19:44:01.523 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:44:01.543 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KBApgsAzf2xXl8w4WMXJBLGBX9s= 2022-09-12T19:44:01.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KBApgsAzf2xXl8w4WMXJBLGBX9s=', 'Content-Length': '0'} 2022-09-12T19:44:01.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dabaf838e71b8a7-00631f8c01-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.551 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:01.552 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:01.552 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:01.556 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:01.556 DEBUG:boto:Method: HEAD 2022-09-12T19:44:01.557 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:44:01.557 DEBUG:boto:Data: 2022-09-12T19:44:01.558 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.558 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:01.559 DEBUG:boto:Port: 8001 2022-09-12T19:44:01.560 DEBUG:boto:Params: {} 2022-09-12T19:44:01.560 DEBUG:boto:Token: None 2022-09-12T19:44:01.561 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:44:01.562 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:y7fdtPs6gomAhoMzKe3uujlxeek= 2022-09-12T19:44:01.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:y7fdtPs6gomAhoMzKe3uujlxeek=', 'Content-Length': '0'} 2022-09-12T19:44:01.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ac188eb6502c5ba-00631f8c01-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.566 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:01.566 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:01.567 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:01.568 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:01.569 DEBUG:boto:Method: HEAD 2022-09-12T19:44:01.570 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:44:01.571 DEBUG:boto:Data: 2022-09-12T19:44:01.571 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.572 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:01.572 DEBUG:boto:Port: 8000 2022-09-12T19:44:01.573 DEBUG:boto:Params: {} 2022-09-12T19:44:01.573 DEBUG:boto:Token: None 2022-09-12T19:44:01.575 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:44:01.575 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:y7fdtPs6gomAhoMzKe3uujlxeek= 2022-09-12T19:44:01.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:y7fdtPs6gomAhoMzKe3uujlxeek=', 'Content-Length': '0'} 2022-09-12T19:44:01.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e2c7afa1bca2cec-00631f8c01-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:44:01.590 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:01.591 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:01.592 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:01.593 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:01.594 DEBUG:boto:Method: GET 2022-09-12T19:44:01.595 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:44:01.596 DEBUG:boto:Data: 2022-09-12T19:44:01.597 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.598 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:01.599 DEBUG:boto:Port: 8001 2022-09-12T19:44:01.600 DEBUG:boto:Params: {} 2022-09-12T19:44:01.601 DEBUG:boto:Token: None 2022-09-12T19:44:01.602 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:44:01.603 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KBApgsAzf2xXl8w4WMXJBLGBX9s= 2022-09-12T19:44:01.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KBApgsAzf2xXl8w4WMXJBLGBX9s=', 'Content-Length': '0'} 2022-09-12T19:44:01.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc19347f4529397c-00631f8c01-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.608 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:01.609 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:01.610 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:01.611 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:01.612 DEBUG:boto:Method: GET 2022-09-12T19:44:01.613 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:44:01.614 DEBUG:boto:Data: 2022-09-12T19:44:01.615 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.615 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:01.616 DEBUG:boto:Port: 8000 2022-09-12T19:44:01.616 DEBUG:boto:Params: {} 2022-09-12T19:44:01.617 DEBUG:boto:Token: None 2022-09-12T19:44:01.617 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:44:01.618 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KBApgsAzf2xXl8w4WMXJBLGBX9s= 2022-09-12T19:44:01.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KBApgsAzf2xXl8w4WMXJBLGBX9s=', 'Content-Length': '0'} 2022-09-12T19:44:01.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005e6b70fc45e20ae-00631f8c01-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.662 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:01.663 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:01.664 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:44:01.665 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:44:01.665 DEBUG:boto:Method: GET 2022-09-12T19:44:01.666 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:44:01.666 DEBUG:boto:Data: 2022-09-12T19:44:01.667 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.667 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:01.668 DEBUG:boto:Port: 8001 2022-09-12T19:44:01.669 DEBUG:boto:Params: {} 2022-09-12T19:44:01.669 DEBUG:boto:Token: None 2022-09-12T19:44:01.670 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/ 2022-09-12T19:44:01.671 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Oa3VHEgZbrxcildJS3kFk/o1uI= 2022-09-12T19:44:01.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Oa3VHEgZbrxcildJS3kFk/o1uI=', 'Content-Length': '0'} 2022-09-12T19:44:01.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f5cd279708a5b9f-00631f8c01-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.675 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:01.676 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:01.677 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:01.677 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:44:01.678 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:44:01.679 DEBUG:boto:Method: GET 2022-09-12T19:44:01.680 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:44:01.681 DEBUG:boto:Data: 2022-09-12T19:44:01.681 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.682 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:01.682 DEBUG:boto:Port: 8000 2022-09-12T19:44:01.683 DEBUG:boto:Params: {} 2022-09-12T19:44:01.684 DEBUG:boto:Token: None 2022-09-12T19:44:01.684 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/ 2022-09-12T19:44:01.685 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Oa3VHEgZbrxcildJS3kFk/o1uI= 2022-09-12T19:44:01.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Oa3VHEgZbrxcildJS3kFk/o1uI=', 'Content-Length': '0'} 2022-09-12T19:44:01.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ab631e87434c3c1-00631f8c01-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.693 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:01.695 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:01.696 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:01.697 DEBUG:boto:Method: HEAD 2022-09-12T19:44:01.698 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:44:01.700 DEBUG:boto:Data: 2022-09-12T19:44:01.701 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.703 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:01.704 DEBUG:boto:Port: 8001 2022-09-12T19:44:01.705 DEBUG:boto:Params: {} 2022-09-12T19:44:01.706 DEBUG:boto:Token: None 2022-09-12T19:44:01.708 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/b 2022-09-12T19:44:01.709 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9HIt3tm7sYiL2+derXWgAt4ES24= 2022-09-12T19:44:01.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9HIt3tm7sYiL2+derXWgAt4ES24=', 'Content-Length': '0'} 2022-09-12T19:44:01.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c9e38899cad1006-00631f8c01-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.715 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:01.716 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:01.718 DEBUG:boto:Method: HEAD 2022-09-12T19:44:01.719 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:44:01.719 DEBUG:boto:Data: 2022-09-12T19:44:01.720 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.721 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:01.721 DEBUG:boto:Port: 8000 2022-09-12T19:44:01.722 DEBUG:boto:Params: {} 2022-09-12T19:44:01.723 DEBUG:boto:Token: None 2022-09-12T19:44:01.723 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/b 2022-09-12T19:44:01.724 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9HIt3tm7sYiL2+derXWgAt4ES24= 2022-09-12T19:44:01.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9HIt3tm7sYiL2+derXWgAt4ES24=', 'Content-Length': '0'} 2022-09-12T19:44:01.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cf968f2e0ce04cb-00631f8c01-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:44:01.728 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:01.729 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:01.729 DEBUG:boto:Method: GET 2022-09-12T19:44:01.730 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:44:01.731 DEBUG:boto:Data: 2022-09-12T19:44:01.731 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.732 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:01.733 DEBUG:boto:Port: 8001 2022-09-12T19:44:01.733 DEBUG:boto:Params: {} 2022-09-12T19:44:01.734 DEBUG:boto:Token: None 2022-09-12T19:44:01.735 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/b 2022-09-12T19:44:01.735 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1gdAmVg0CFLn1p72ul8WLlsXq7I= 2022-09-12T19:44:01.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1gdAmVg0CFLn1p72ul8WLlsXq7I=', 'Content-Length': '0'} 2022-09-12T19:44:01.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002322c7f1745f07a5-00631f8c01-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.782 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:01.783 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:01.785 DEBUG:boto:Method: GET 2022-09-12T19:44:01.785 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:44:01.786 DEBUG:boto:Data: 2022-09-12T19:44:01.787 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.787 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:01.788 DEBUG:boto:Port: 8000 2022-09-12T19:44:01.788 DEBUG:boto:Params: {} 2022-09-12T19:44:01.789 DEBUG:boto:Token: None 2022-09-12T19:44:01.790 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/b 2022-09-12T19:44:01.791 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1gdAmVg0CFLn1p72ul8WLlsXq7I= 2022-09-12T19:44:01.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1gdAmVg0CFLn1p72ul8WLlsXq7I=', 'Content-Length': '0'} 2022-09-12T19:44:01.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006aa7e3f1be2e2df3-00631f8c01-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.795 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:01.796 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:01.797 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:44:01.797 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:44:01.798 DEBUG:boto:Method: GET 2022-09-12T19:44:01.799 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:44:01.800 DEBUG:boto:Data: 2022-09-12T19:44:01.800 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.801 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:01.801 DEBUG:boto:Port: 8001 2022-09-12T19:44:01.802 DEBUG:boto:Params: {} 2022-09-12T19:44:01.803 DEBUG:boto:Token: None 2022-09-12T19:44:01.804 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/ 2022-09-12T19:44:01.805 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Oa3VHEgZbrxcildJS3kFk/o1uI= 2022-09-12T19:44:01.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Oa3VHEgZbrxcildJS3kFk/o1uI=', 'Content-Length': '0'} 2022-09-12T19:44:01.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cabf58128574f87-00631f8c01-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.810 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:01.811 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:01.812 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:01.813 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:44:01.814 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:44:01.814 DEBUG:boto:Method: GET 2022-09-12T19:44:01.815 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:44:01.815 DEBUG:boto:Data: 2022-09-12T19:44:01.816 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.817 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:01.817 DEBUG:boto:Port: 8000 2022-09-12T19:44:01.818 DEBUG:boto:Params: {} 2022-09-12T19:44:01.819 DEBUG:boto:Token: None 2022-09-12T19:44:01.820 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/ 2022-09-12T19:44:01.821 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Oa3VHEgZbrxcildJS3kFk/o1uI= 2022-09-12T19:44:01.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Oa3VHEgZbrxcildJS3kFk/o1uI=', 'Content-Length': '0'} 2022-09-12T19:44:01.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083ab53fa3b8186dd-00631f8c01-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.870 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:01.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:44:01.872 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:01.872 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:01.873 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:01.874 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:01.875 DEBUG:boto:Method: GET 2022-09-12T19:44:01.875 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:44:01.876 DEBUG:boto:Data: 2022-09-12T19:44:01.876 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.877 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:01.877 DEBUG:boto:Port: 8001 2022-09-12T19:44:01.878 DEBUG:boto:Params: {} 2022-09-12T19:44:01.879 DEBUG:boto:Token: None 2022-09-12T19:44:01.879 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:44:01.880 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LZtDJNLmT7w1EuWc80YD3OBtSwQ= 2022-09-12T19:44:01.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LZtDJNLmT7w1EuWc80YD3OBtSwQ=', 'Content-Length': '0'} 2022-09-12T19:44:01.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b278ed8178a7ec6-00631f8c01-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.884 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:01.884 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:01.885 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:01.885 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:01.886 DEBUG:boto:Method: GET 2022-09-12T19:44:01.887 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:44:01.887 DEBUG:boto:Data: 2022-09-12T19:44:01.888 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.888 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:01.889 DEBUG:boto:Port: 8000 2022-09-12T19:44:01.889 DEBUG:boto:Params: {} 2022-09-12T19:44:01.890 DEBUG:boto:Token: None 2022-09-12T19:44:01.891 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:44:01.892 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LZtDJNLmT7w1EuWc80YD3OBtSwQ= 2022-09-12T19:44:01.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LZtDJNLmT7w1EuWc80YD3OBtSwQ=', 'Content-Length': '0'} 2022-09-12T19:44:01.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae1d1fd8bbb9bddd-00631f8c01-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.896 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:01.897 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:01.897 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:01.898 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:01.899 DEBUG:boto:Method: HEAD 2022-09-12T19:44:01.899 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:44:01.900 DEBUG:boto:Data: 2022-09-12T19:44:01.900 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.901 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:01.901 DEBUG:boto:Port: 8001 2022-09-12T19:44:01.902 DEBUG:boto:Params: {} 2022-09-12T19:44:01.903 DEBUG:boto:Token: None 2022-09-12T19:44:01.904 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:44:01.904 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Ns7IyVwgl2N6JT0oIHNx0a6Dgs= 2022-09-12T19:44:01.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Ns7IyVwgl2N6JT0oIHNx0a6Dgs=', 'Content-Length': '0'} 2022-09-12T19:44:01.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006520638fce50e5f3-00631f8c01-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.913 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:01.913 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:01.914 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:01.915 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:01.916 DEBUG:boto:Method: HEAD 2022-09-12T19:44:01.917 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:44:01.917 DEBUG:boto:Data: 2022-09-12T19:44:01.918 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.919 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:01.919 DEBUG:boto:Port: 8000 2022-09-12T19:44:01.920 DEBUG:boto:Params: {} 2022-09-12T19:44:01.921 DEBUG:boto:Token: None 2022-09-12T19:44:01.921 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:44:01.922 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Ns7IyVwgl2N6JT0oIHNx0a6Dgs= 2022-09-12T19:44:01.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Ns7IyVwgl2N6JT0oIHNx0a6Dgs=', 'Content-Length': '0'} 2022-09-12T19:44:01.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099d3457d13a37675-00631f8c01-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:44:01.927 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:01.927 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:01.928 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:01.929 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:01.931 DEBUG:boto:Method: GET 2022-09-12T19:44:01.932 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:44:01.933 DEBUG:boto:Data: 2022-09-12T19:44:01.934 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.935 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:01.936 DEBUG:boto:Port: 8001 2022-09-12T19:44:01.937 DEBUG:boto:Params: {} 2022-09-12T19:44:01.938 DEBUG:boto:Token: None 2022-09-12T19:44:01.939 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:44:01.940 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LZtDJNLmT7w1EuWc80YD3OBtSwQ= 2022-09-12T19:44:01.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LZtDJNLmT7w1EuWc80YD3OBtSwQ=', 'Content-Length': '0'} 2022-09-12T19:44:01.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b42ec59a04bc439-00631f8c01-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.960 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:01.967 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:01.968 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:01.969 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:01.970 DEBUG:boto:Method: GET 2022-09-12T19:44:01.970 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:44:01.971 DEBUG:boto:Data: 2022-09-12T19:44:01.971 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.972 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:01.973 DEBUG:boto:Port: 8000 2022-09-12T19:44:01.974 DEBUG:boto:Params: {} 2022-09-12T19:44:01.975 DEBUG:boto:Token: None 2022-09-12T19:44:01.975 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:44:01.976 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LZtDJNLmT7w1EuWc80YD3OBtSwQ= 2022-09-12T19:44:01.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LZtDJNLmT7w1EuWc80YD3OBtSwQ=', 'Content-Length': '0'} 2022-09-12T19:44:01.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c7fed889387d16d-00631f8c01-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:01.980 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:01.981 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:01.981 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:44:01.982 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:44:01.987 DEBUG:boto:Method: GET 2022-09-12T19:44:01.987 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:44:01.988 DEBUG:boto:Data: 2022-09-12T19:44:01.989 DEBUG:boto:Headers: {} 2022-09-12T19:44:01.990 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:01.992 DEBUG:boto:Port: 8001 2022-09-12T19:44:01.994 DEBUG:boto:Params: {} 2022-09-12T19:44:01.997 DEBUG:boto:Token: None 2022-09-12T19:44:01.998 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:01 GMT /jdpiak-71/ 2022-09-12T19:44:02.000 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Oa3VHEgZbrxcildJS3kFk/o1uI= 2022-09-12T19:44:02.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Oa3VHEgZbrxcildJS3kFk/o1uI=', 'Content-Length': '0'} 2022-09-12T19:44:02.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065cffc90dc5ea3f1-00631f8c02-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.046 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:02.047 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:02.048 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:02.049 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:44:02.050 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:44:02.052 DEBUG:boto:Method: GET 2022-09-12T19:44:02.055 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:44:02.056 DEBUG:boto:Data: 2022-09-12T19:44:02.057 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.058 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:02.058 DEBUG:boto:Port: 8000 2022-09-12T19:44:02.059 DEBUG:boto:Params: {} 2022-09-12T19:44:02.060 DEBUG:boto:Token: None 2022-09-12T19:44:02.061 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/ 2022-09-12T19:44:02.064 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk= 2022-09-12T19:44:02.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk=', 'Content-Length': '0'} 2022-09-12T19:44:02.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067221141beaf3376-00631f8c02-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.070 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:02.074 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:02.075 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:02.076 DEBUG:boto:Method: HEAD 2022-09-12T19:44:02.076 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:44:02.078 DEBUG:boto:Data: 2022-09-12T19:44:02.078 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.079 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:02.080 DEBUG:boto:Port: 8001 2022-09-12T19:44:02.080 DEBUG:boto:Params: {} 2022-09-12T19:44:02.081 DEBUG:boto:Token: None 2022-09-12T19:44:02.082 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/c 2022-09-12T19:44:02.082 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4zf8YTXlBKGtjG2Su1sTbF4q0hk= 2022-09-12T19:44:02.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4zf8YTXlBKGtjG2Su1sTbF4q0hk=', 'Content-Length': '0'} 2022-09-12T19:44:02.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002297a98f1b9bd71d-00631f8c02-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.087 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:02.088 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:02.088 DEBUG:boto:Method: HEAD 2022-09-12T19:44:02.089 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:44:02.091 DEBUG:boto:Data: 2022-09-12T19:44:02.091 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.092 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:02.092 DEBUG:boto:Port: 8000 2022-09-12T19:44:02.093 DEBUG:boto:Params: {} 2022-09-12T19:44:02.093 DEBUG:boto:Token: None 2022-09-12T19:44:02.094 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/c 2022-09-12T19:44:02.095 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4zf8YTXlBKGtjG2Su1sTbF4q0hk= 2022-09-12T19:44:02.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4zf8YTXlBKGtjG2Su1sTbF4q0hk=', 'Content-Length': '0'} 2022-09-12T19:44:02.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030b8542fe85d9b25-00631f8c02-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:44:02.100 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:02.101 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:02.101 DEBUG:boto:Method: GET 2022-09-12T19:44:02.102 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:44:02.102 DEBUG:boto:Data: 2022-09-12T19:44:02.103 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.103 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:02.104 DEBUG:boto:Port: 8001 2022-09-12T19:44:02.105 DEBUG:boto:Params: {} 2022-09-12T19:44:02.106 DEBUG:boto:Token: None 2022-09-12T19:44:02.107 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/c 2022-09-12T19:44:02.107 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:I0U+RmaaijcwXvj5+nLoB5Qyqk0= 2022-09-12T19:44:02.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:I0U+RmaaijcwXvj5+nLoB5Qyqk0=', 'Content-Length': '0'} 2022-09-12T19:44:02.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0c88e027aa7bade-00631f8c02-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.128 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:02.129 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:02.130 DEBUG:boto:Method: GET 2022-09-12T19:44:02.131 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:44:02.132 DEBUG:boto:Data: 2022-09-12T19:44:02.133 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.134 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:02.135 DEBUG:boto:Port: 8000 2022-09-12T19:44:02.136 DEBUG:boto:Params: {} 2022-09-12T19:44:02.137 DEBUG:boto:Token: None 2022-09-12T19:44:02.139 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/c 2022-09-12T19:44:02.140 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:I0U+RmaaijcwXvj5+nLoB5Qyqk0= 2022-09-12T19:44:02.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:I0U+RmaaijcwXvj5+nLoB5Qyqk0=', 'Content-Length': '0'} 2022-09-12T19:44:02.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b28d59d8f1d8c810-00631f8c02-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.156 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:02.157 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:02.162 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:44:02.166 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:44:02.168 DEBUG:boto:Method: GET 2022-09-12T19:44:02.169 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:44:02.171 DEBUG:boto:Data: 2022-09-12T19:44:02.173 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.175 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:02.176 DEBUG:boto:Port: 8001 2022-09-12T19:44:02.177 DEBUG:boto:Params: {} 2022-09-12T19:44:02.178 DEBUG:boto:Token: None 2022-09-12T19:44:02.179 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/ 2022-09-12T19:44:02.181 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk= 2022-09-12T19:44:02.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk=', 'Content-Length': '0'} 2022-09-12T19:44:02.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c753df9909ad4a2c-00631f8c02-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.186 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:02.188 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:02.189 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:02.189 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:44:02.190 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:44:02.192 DEBUG:boto:Method: GET 2022-09-12T19:44:02.193 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:44:02.194 DEBUG:boto:Data: 2022-09-12T19:44:02.195 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.196 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:02.196 DEBUG:boto:Port: 8000 2022-09-12T19:44:02.197 DEBUG:boto:Params: {} 2022-09-12T19:44:02.199 DEBUG:boto:Token: None 2022-09-12T19:44:02.200 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/ 2022-09-12T19:44:02.200 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk= 2022-09-12T19:44:02.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk=', 'Content-Length': '0'} 2022-09-12T19:44:02.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f06d57fcf3a5aee2-00631f8c02-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.205 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:02.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:44:02.207 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:02.208 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:02.209 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:02.210 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:02.211 DEBUG:boto:Method: GET 2022-09-12T19:44:02.212 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:44:02.213 DEBUG:boto:Data: 2022-09-12T19:44:02.214 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.214 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:02.216 DEBUG:boto:Port: 8001 2022-09-12T19:44:02.217 DEBUG:boto:Params: {} 2022-09-12T19:44:02.218 DEBUG:boto:Token: None 2022-09-12T19:44:02.219 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:44:02.220 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jqLmdeM7IirnE8QT/dzlrO8f2ME= 2022-09-12T19:44:02.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jqLmdeM7IirnE8QT/dzlrO8f2ME=', 'Content-Length': '0'} 2022-09-12T19:44:02.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a35b6f99028ed27-00631f8c02-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.228 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:02.229 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:02.230 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:02.231 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:02.232 DEBUG:boto:Method: GET 2022-09-12T19:44:02.233 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:44:02.233 DEBUG:boto:Data: 2022-09-12T19:44:02.235 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.236 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:02.237 DEBUG:boto:Port: 8000 2022-09-12T19:44:02.238 DEBUG:boto:Params: {} 2022-09-12T19:44:02.239 DEBUG:boto:Token: None 2022-09-12T19:44:02.241 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:44:02.242 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jqLmdeM7IirnE8QT/dzlrO8f2ME= 2022-09-12T19:44:02.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jqLmdeM7IirnE8QT/dzlrO8f2ME=', 'Content-Length': '0'} 2022-09-12T19:44:02.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000263e5ff8f924d380-00631f8c02-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.246 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:02.246 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:02.247 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:02.248 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:02.249 DEBUG:boto:Method: HEAD 2022-09-12T19:44:02.250 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:44:02.250 DEBUG:boto:Data: 2022-09-12T19:44:02.251 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.251 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:02.252 DEBUG:boto:Port: 8001 2022-09-12T19:44:02.253 DEBUG:boto:Params: {} 2022-09-12T19:44:02.254 DEBUG:boto:Token: None 2022-09-12T19:44:02.254 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:44:02.255 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AiGrpXFiSg0+JYHwtcWWm+hPhPU= 2022-09-12T19:44:02.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AiGrpXFiSg0+JYHwtcWWm+hPhPU=', 'Content-Length': '0'} 2022-09-12T19:44:02.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000485ee67d7b626b4-00631f8c02-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.263 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:02.264 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:02.264 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:02.266 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:02.266 DEBUG:boto:Method: HEAD 2022-09-12T19:44:02.267 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:44:02.268 DEBUG:boto:Data: 2022-09-12T19:44:02.269 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.270 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:02.271 DEBUG:boto:Port: 8000 2022-09-12T19:44:02.272 DEBUG:boto:Params: {} 2022-09-12T19:44:02.273 DEBUG:boto:Token: None 2022-09-12T19:44:02.274 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:44:02.275 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AiGrpXFiSg0+JYHwtcWWm+hPhPU= 2022-09-12T19:44:02.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AiGrpXFiSg0+JYHwtcWWm+hPhPU=', 'Content-Length': '0'} 2022-09-12T19:44:02.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060ab38e50f9d4c98-00631f8c02-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:44:02.281 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:02.282 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:02.283 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:02.283 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:02.284 DEBUG:boto:Method: GET 2022-09-12T19:44:02.284 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:44:02.285 DEBUG:boto:Data: 2022-09-12T19:44:02.285 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.286 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:02.287 DEBUG:boto:Port: 8001 2022-09-12T19:44:02.288 DEBUG:boto:Params: {} 2022-09-12T19:44:02.289 DEBUG:boto:Token: None 2022-09-12T19:44:02.294 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:44:02.295 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jqLmdeM7IirnE8QT/dzlrO8f2ME= 2022-09-12T19:44:02.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jqLmdeM7IirnE8QT/dzlrO8f2ME=', 'Content-Length': '0'} 2022-09-12T19:44:02.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058131f673e9a6b5b-00631f8c02-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.342 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:02.343 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:02.343 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:02.344 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:02.344 DEBUG:boto:Method: GET 2022-09-12T19:44:02.345 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:44:02.345 DEBUG:boto:Data: 2022-09-12T19:44:02.346 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.346 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:02.347 DEBUG:boto:Port: 8000 2022-09-12T19:44:02.348 DEBUG:boto:Params: {} 2022-09-12T19:44:02.348 DEBUG:boto:Token: None 2022-09-12T19:44:02.349 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:44:02.350 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jqLmdeM7IirnE8QT/dzlrO8f2ME= 2022-09-12T19:44:02.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jqLmdeM7IirnE8QT/dzlrO8f2ME=', 'Content-Length': '0'} 2022-09-12T19:44:02.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9e849b8dae26a7b-00631f8c02-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.353 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:02.354 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:02.355 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:44:02.355 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:44:02.356 DEBUG:boto:Method: GET 2022-09-12T19:44:02.356 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:44:02.357 DEBUG:boto:Data: 2022-09-12T19:44:02.357 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.358 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:02.358 DEBUG:boto:Port: 8001 2022-09-12T19:44:02.359 DEBUG:boto:Params: {} 2022-09-12T19:44:02.360 DEBUG:boto:Token: None 2022-09-12T19:44:02.360 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/ 2022-09-12T19:44:02.361 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk= 2022-09-12T19:44:02.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk=', 'Content-Length': '0'} 2022-09-12T19:44:02.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086203558c09e0a33-00631f8c02-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.365 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:02.366 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:02.366 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:02.367 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:44:02.368 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:44:02.369 DEBUG:boto:Method: GET 2022-09-12T19:44:02.369 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:44:02.370 DEBUG:boto:Data: 2022-09-12T19:44:02.371 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.371 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:02.372 DEBUG:boto:Port: 8000 2022-09-12T19:44:02.373 DEBUG:boto:Params: {} 2022-09-12T19:44:02.373 DEBUG:boto:Token: None 2022-09-12T19:44:02.374 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/ 2022-09-12T19:44:02.374 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk= 2022-09-12T19:44:02.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk=', 'Content-Length': '0'} 2022-09-12T19:44:02.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b61f5dacb21e91b-00631f8c02-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.378 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:02.379 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:02.380 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:02.380 DEBUG:boto:Method: HEAD 2022-09-12T19:44:02.381 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:44:02.381 DEBUG:boto:Data: 2022-09-12T19:44:02.382 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.383 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:02.383 DEBUG:boto:Port: 8001 2022-09-12T19:44:02.384 DEBUG:boto:Params: {} 2022-09-12T19:44:02.384 DEBUG:boto:Token: None 2022-09-12T19:44:02.385 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/d 2022-09-12T19:44:02.385 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x3wQvgVXqwBnCjZ47ZR67iYSchc= 2022-09-12T19:44:02.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x3wQvgVXqwBnCjZ47ZR67iYSchc=', 'Content-Length': '0'} 2022-09-12T19:44:02.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012d19b4fb310e86e-00631f8c02-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.390 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:02.390 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:02.391 DEBUG:boto:Method: HEAD 2022-09-12T19:44:02.391 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:44:02.392 DEBUG:boto:Data: 2022-09-12T19:44:02.392 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.393 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:02.393 DEBUG:boto:Port: 8000 2022-09-12T19:44:02.394 DEBUG:boto:Params: {} 2022-09-12T19:44:02.395 DEBUG:boto:Token: None 2022-09-12T19:44:02.395 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/d 2022-09-12T19:44:02.396 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x3wQvgVXqwBnCjZ47ZR67iYSchc= 2022-09-12T19:44:02.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x3wQvgVXqwBnCjZ47ZR67iYSchc=', 'Content-Length': '0'} 2022-09-12T19:44:02.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001b2e566891a0685-00631f8c02-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:44:02.400 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:02.401 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:02.402 DEBUG:boto:Method: GET 2022-09-12T19:44:02.402 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:44:02.403 DEBUG:boto:Data: 2022-09-12T19:44:02.403 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.404 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:02.405 DEBUG:boto:Port: 8001 2022-09-12T19:44:02.405 DEBUG:boto:Params: {} 2022-09-12T19:44:02.406 DEBUG:boto:Token: None 2022-09-12T19:44:02.406 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/d 2022-09-12T19:44:02.407 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HlzjleCFS2n/2J7/oMbNA+rtYLA= 2022-09-12T19:44:02.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HlzjleCFS2n/2J7/oMbNA+rtYLA=', 'Content-Length': '0'} 2022-09-12T19:44:02.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e2563cb58e1627d-00631f8c02-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.414 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:02.415 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:02.417 DEBUG:boto:Method: GET 2022-09-12T19:44:02.418 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:44:02.419 DEBUG:boto:Data: 2022-09-12T19:44:02.421 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.422 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:02.423 DEBUG:boto:Port: 8000 2022-09-12T19:44:02.424 DEBUG:boto:Params: {} 2022-09-12T19:44:02.425 DEBUG:boto:Token: None 2022-09-12T19:44:02.426 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/d 2022-09-12T19:44:02.428 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HlzjleCFS2n/2J7/oMbNA+rtYLA= 2022-09-12T19:44:02.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HlzjleCFS2n/2J7/oMbNA+rtYLA=', 'Content-Length': '0'} 2022-09-12T19:44:02.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000be6b6e9804aa9da-00631f8c02-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.475 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:02.476 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:02.477 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:44:02.478 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:44:02.479 DEBUG:boto:Method: GET 2022-09-12T19:44:02.480 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:44:02.481 DEBUG:boto:Data: 2022-09-12T19:44:02.482 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.484 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:02.485 DEBUG:boto:Port: 8001 2022-09-12T19:44:02.486 DEBUG:boto:Params: {} 2022-09-12T19:44:02.487 DEBUG:boto:Token: None 2022-09-12T19:44:02.488 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/ 2022-09-12T19:44:02.489 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk= 2022-09-12T19:44:02.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk=', 'Content-Length': '0'} 2022-09-12T19:44:02.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000602de978cbbfd13a-00631f8c02-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.499 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:02.501 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:02.502 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:02.503 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:44:02.504 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:44:02.504 DEBUG:boto:Method: GET 2022-09-12T19:44:02.505 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:44:02.506 DEBUG:boto:Data: 2022-09-12T19:44:02.507 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.508 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:02.509 DEBUG:boto:Port: 8000 2022-09-12T19:44:02.510 DEBUG:boto:Params: {} 2022-09-12T19:44:02.511 DEBUG:boto:Token: None 2022-09-12T19:44:02.512 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/ 2022-09-12T19:44:02.514 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk= 2022-09-12T19:44:02.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk=', 'Content-Length': '0'} 2022-09-12T19:44:02.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8a4874e10ced121-00631f8c02-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.520 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:02.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:44:02.522 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:02.523 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:02.524 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:02.525 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:02.525 DEBUG:boto:Method: GET 2022-09-12T19:44:02.526 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:44:02.527 DEBUG:boto:Data: 2022-09-12T19:44:02.528 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.529 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:02.530 DEBUG:boto:Port: 8001 2022-09-12T19:44:02.531 DEBUG:boto:Params: {} 2022-09-12T19:44:02.532 DEBUG:boto:Token: None 2022-09-12T19:44:02.533 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:44:02.534 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Bqlce6w8dnmJkT8OVXgn7eeQPPQ= 2022-09-12T19:44:02.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Bqlce6w8dnmJkT8OVXgn7eeQPPQ=', 'Content-Length': '0'} 2022-09-12T19:44:02.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e48346ff3b53b8e-00631f8c02-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.539 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:02.541 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:02.542 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:02.543 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:02.544 DEBUG:boto:Method: GET 2022-09-12T19:44:02.544 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:44:02.545 DEBUG:boto:Data: 2022-09-12T19:44:02.545 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.546 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:02.546 DEBUG:boto:Port: 8000 2022-09-12T19:44:02.547 DEBUG:boto:Params: {} 2022-09-12T19:44:02.548 DEBUG:boto:Token: None 2022-09-12T19:44:02.549 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:44:02.550 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Bqlce6w8dnmJkT8OVXgn7eeQPPQ= 2022-09-12T19:44:02.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Bqlce6w8dnmJkT8OVXgn7eeQPPQ=', 'Content-Length': '0'} 2022-09-12T19:44:02.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a244c82a1a58355-00631f8c02-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.556 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:02.556 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:02.557 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:02.558 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:02.559 DEBUG:boto:Method: HEAD 2022-09-12T19:44:02.560 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:44:02.561 DEBUG:boto:Data: 2022-09-12T19:44:02.562 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.563 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:02.564 DEBUG:boto:Port: 8001 2022-09-12T19:44:02.565 DEBUG:boto:Params: {} 2022-09-12T19:44:02.566 DEBUG:boto:Token: None 2022-09-12T19:44:02.568 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:44:02.569 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WC2NsBqHD+rjYCKVSmLJsaxPrpg= 2022-09-12T19:44:02.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WC2NsBqHD+rjYCKVSmLJsaxPrpg=', 'Content-Length': '0'} 2022-09-12T19:44:02.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000539f09d40e01afa6-00631f8c02-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.575 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:02.576 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:02.576 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:02.578 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:02.579 DEBUG:boto:Method: HEAD 2022-09-12T19:44:02.580 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:44:02.580 DEBUG:boto:Data: 2022-09-12T19:44:02.581 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.581 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:02.582 DEBUG:boto:Port: 8000 2022-09-12T19:44:02.582 DEBUG:boto:Params: {} 2022-09-12T19:44:02.583 DEBUG:boto:Token: None 2022-09-12T19:44:02.583 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:44:02.584 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WC2NsBqHD+rjYCKVSmLJsaxPrpg= 2022-09-12T19:44:02.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WC2NsBqHD+rjYCKVSmLJsaxPrpg=', 'Content-Length': '0'} 2022-09-12T19:44:02.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a55259b46db80271-00631f8c02-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:44:02.588 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:02.589 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:02.589 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:02.590 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:02.590 DEBUG:boto:Method: GET 2022-09-12T19:44:02.591 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:44:02.591 DEBUG:boto:Data: 2022-09-12T19:44:02.592 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.592 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:02.593 DEBUG:boto:Port: 8001 2022-09-12T19:44:02.593 DEBUG:boto:Params: {} 2022-09-12T19:44:02.594 DEBUG:boto:Token: None 2022-09-12T19:44:02.595 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:44:02.595 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Bqlce6w8dnmJkT8OVXgn7eeQPPQ= 2022-09-12T19:44:02.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Bqlce6w8dnmJkT8OVXgn7eeQPPQ=', 'Content-Length': '0'} 2022-09-12T19:44:02.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fbb113b85e521f4-00631f8c02-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.600 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:02.600 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:02.601 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:02.601 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:02.602 DEBUG:boto:Method: GET 2022-09-12T19:44:02.602 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:44:02.603 DEBUG:boto:Data: 2022-09-12T19:44:02.604 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.604 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:02.605 DEBUG:boto:Port: 8000 2022-09-12T19:44:02.605 DEBUG:boto:Params: {} 2022-09-12T19:44:02.606 DEBUG:boto:Token: None 2022-09-12T19:44:02.607 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:44:02.607 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Bqlce6w8dnmJkT8OVXgn7eeQPPQ= 2022-09-12T19:44:02.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Bqlce6w8dnmJkT8OVXgn7eeQPPQ=', 'Content-Length': '0'} 2022-09-12T19:44:02.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006efcd346f534219c-00631f8c02-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.654 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:02.655 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:02.656 DEBUG:boto:path=/jdpiak-71/?prefix=e 2022-09-12T19:44:02.656 DEBUG:boto:auth_path=/jdpiak-71/?prefix=e 2022-09-12T19:44:02.657 DEBUG:boto:Method: GET 2022-09-12T19:44:02.658 DEBUG:boto:Path: /jdpiak-71/?prefix=e 2022-09-12T19:44:02.658 DEBUG:boto:Data: 2022-09-12T19:44:02.659 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.660 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:02.661 DEBUG:boto:Port: 8001 2022-09-12T19:44:02.662 DEBUG:boto:Params: {} 2022-09-12T19:44:02.663 DEBUG:boto:Token: None 2022-09-12T19:44:02.663 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/ 2022-09-12T19:44:02.664 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk= 2022-09-12T19:44:02.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk=', 'Content-Length': '0'} 2022-09-12T19:44:02.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f002810628c3260e-00631f8c02-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.670 DEBUG:boto:b'jdpiak-71e1000falsee2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:02.672 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:02.673 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:02.673 DEBUG:boto:path=/jdpiak-71/?prefix=e 2022-09-12T19:44:02.674 DEBUG:boto:auth_path=/jdpiak-71/?prefix=e 2022-09-12T19:44:02.676 DEBUG:boto:Method: GET 2022-09-12T19:44:02.677 DEBUG:boto:Path: /jdpiak-71/?prefix=e 2022-09-12T19:44:02.678 DEBUG:boto:Data: 2022-09-12T19:44:02.679 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.680 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:02.682 DEBUG:boto:Port: 8000 2022-09-12T19:44:02.683 DEBUG:boto:Params: {} 2022-09-12T19:44:02.684 DEBUG:boto:Token: None 2022-09-12T19:44:02.685 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/ 2022-09-12T19:44:02.686 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk= 2022-09-12T19:44:02.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk=', 'Content-Length': '0'} 2022-09-12T19:44:02.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c02c3d2bb01a46ae-00631f8c02-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.693 DEBUG:boto:b'jdpiak-71e1000falsee2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:02.694 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:02.695 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:02.696 DEBUG:boto:Method: HEAD 2022-09-12T19:44:02.698 DEBUG:boto:Path: /jdpiak-71/e 2022-09-12T19:44:02.699 DEBUG:boto:Data: 2022-09-12T19:44:02.700 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.701 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:02.702 DEBUG:boto:Port: 8001 2022-09-12T19:44:02.703 DEBUG:boto:Params: {} 2022-09-12T19:44:02.705 DEBUG:boto:Token: None 2022-09-12T19:44:02.706 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/e 2022-09-12T19:44:02.707 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1xXkVz7ShzKerxBtAduEXs7XcrA= 2022-09-12T19:44:02.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1xXkVz7ShzKerxBtAduEXs7XcrA=', 'Content-Length': '0'} 2022-09-12T19:44:02.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f41e28e4ffa8cdbb-00631f8c02-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.712 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:02.713 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:02.714 DEBUG:boto:Method: HEAD 2022-09-12T19:44:02.715 DEBUG:boto:Path: /jdpiak-71/e 2022-09-12T19:44:02.716 DEBUG:boto:Data: 2022-09-12T19:44:02.717 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.718 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:02.719 DEBUG:boto:Port: 8000 2022-09-12T19:44:02.721 DEBUG:boto:Params: {} 2022-09-12T19:44:02.722 DEBUG:boto:Token: None 2022-09-12T19:44:02.723 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/e 2022-09-12T19:44:02.724 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1xXkVz7ShzKerxBtAduEXs7XcrA= 2022-09-12T19:44:02.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1xXkVz7ShzKerxBtAduEXs7XcrA=', 'Content-Length': '0'} 2022-09-12T19:44:02.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdab791c128e6e13-00631f8c02-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:44:02.729 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:02.730 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:02.730 DEBUG:boto:Method: GET 2022-09-12T19:44:02.731 DEBUG:boto:Path: /jdpiak-71/e 2022-09-12T19:44:02.733 DEBUG:boto:Data: 2022-09-12T19:44:02.735 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.735 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:02.736 DEBUG:boto:Port: 8001 2022-09-12T19:44:02.736 DEBUG:boto:Params: {} 2022-09-12T19:44:02.737 DEBUG:boto:Token: None 2022-09-12T19:44:02.738 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/e 2022-09-12T19:44:02.738 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MK+N7asZoVIdPRIVUejM1UhBMHI= 2022-09-12T19:44:02.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MK+N7asZoVIdPRIVUejM1UhBMHI=', 'Content-Length': '0'} 2022-09-12T19:44:02.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023247451500fb9c1-00631f8c02-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.742 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:02.744 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:02.744 DEBUG:boto:Method: GET 2022-09-12T19:44:02.745 DEBUG:boto:Path: /jdpiak-71/e 2022-09-12T19:44:02.745 DEBUG:boto:Data: 2022-09-12T19:44:02.746 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.746 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:02.747 DEBUG:boto:Port: 8000 2022-09-12T19:44:02.747 DEBUG:boto:Params: {} 2022-09-12T19:44:02.748 DEBUG:boto:Token: None 2022-09-12T19:44:02.749 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/e 2022-09-12T19:44:02.749 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MK+N7asZoVIdPRIVUejM1UhBMHI= 2022-09-12T19:44:02.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MK+N7asZoVIdPRIVUejM1UhBMHI=', 'Content-Length': '0'} 2022-09-12T19:44:02.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e84d7a684c0c5a7a-00631f8c02-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.755 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:02.756 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:02.756 DEBUG:boto:path=/jdpiak-71/?prefix=e 2022-09-12T19:44:02.757 DEBUG:boto:auth_path=/jdpiak-71/?prefix=e 2022-09-12T19:44:02.757 DEBUG:boto:Method: GET 2022-09-12T19:44:02.758 DEBUG:boto:Path: /jdpiak-71/?prefix=e 2022-09-12T19:44:02.758 DEBUG:boto:Data: 2022-09-12T19:44:02.759 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.759 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:02.760 DEBUG:boto:Port: 8001 2022-09-12T19:44:02.760 DEBUG:boto:Params: {} 2022-09-12T19:44:02.761 DEBUG:boto:Token: None 2022-09-12T19:44:02.761 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/ 2022-09-12T19:44:02.762 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk= 2022-09-12T19:44:02.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk=', 'Content-Length': '0'} 2022-09-12T19:44:02.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cb2c6ad6759ee64-00631f8c02-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.770 DEBUG:boto:b'jdpiak-71e1000falsee2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:02.772 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:02.773 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:02.775 DEBUG:boto:path=/jdpiak-71/?prefix=e 2022-09-12T19:44:02.776 DEBUG:boto:auth_path=/jdpiak-71/?prefix=e 2022-09-12T19:44:02.777 DEBUG:boto:Method: GET 2022-09-12T19:44:02.779 DEBUG:boto:Path: /jdpiak-71/?prefix=e 2022-09-12T19:44:02.781 DEBUG:boto:Data: 2022-09-12T19:44:02.782 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.783 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:02.784 DEBUG:boto:Port: 8000 2022-09-12T19:44:02.785 DEBUG:boto:Params: {} 2022-09-12T19:44:02.786 DEBUG:boto:Token: None 2022-09-12T19:44:02.787 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/ 2022-09-12T19:44:02.788 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk= 2022-09-12T19:44:02.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk=', 'Content-Length': '0'} 2022-09-12T19:44:02.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f20ef713a0e6804-00631f8c02-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.834 DEBUG:boto:b'jdpiak-71e1000falsee2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:02.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:44:02.836 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:02.837 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:02.837 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:02.838 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:02.839 DEBUG:boto:Method: GET 2022-09-12T19:44:02.839 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:02.840 DEBUG:boto:Data: 2022-09-12T19:44:02.841 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.841 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:02.842 DEBUG:boto:Port: 8001 2022-09-12T19:44:02.842 DEBUG:boto:Params: {} 2022-09-12T19:44:02.843 DEBUG:boto:Token: None 2022-09-12T19:44:02.844 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:02.844 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MO5JoonmCO96t2mJO/ELTbGB1gs= 2022-09-12T19:44:02.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MO5JoonmCO96t2mJO/ELTbGB1gs=', 'Content-Length': '0'} 2022-09-12T19:44:02.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018fae14299ca7ffe-00631f8c02-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.850 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:02.851 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:02.852 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:02.852 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:02.853 DEBUG:boto:Method: GET 2022-09-12T19:44:02.854 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:02.855 DEBUG:boto:Data: 2022-09-12T19:44:02.855 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.856 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:02.856 DEBUG:boto:Port: 8000 2022-09-12T19:44:02.857 DEBUG:boto:Params: {} 2022-09-12T19:44:02.857 DEBUG:boto:Token: None 2022-09-12T19:44:02.858 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:02.859 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MO5JoonmCO96t2mJO/ELTbGB1gs= 2022-09-12T19:44:02.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MO5JoonmCO96t2mJO/ELTbGB1gs=', 'Content-Length': '0'} 2022-09-12T19:44:02.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0dd428eb907e17a-00631f8c02-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.863 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:02.864 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:02.865 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:02.865 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:02.866 DEBUG:boto:Method: HEAD 2022-09-12T19:44:02.866 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:02.867 DEBUG:boto:Data: 2022-09-12T19:44:02.867 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.868 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:02.868 DEBUG:boto:Port: 8001 2022-09-12T19:44:02.869 DEBUG:boto:Params: {} 2022-09-12T19:44:02.869 DEBUG:boto:Token: None 2022-09-12T19:44:02.870 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:02.871 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:g4iu0X+N5RThWIC9VXGPNQRoonY= 2022-09-12T19:44:02.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:g4iu0X+N5RThWIC9VXGPNQRoonY=', 'Content-Length': '0'} 2022-09-12T19:44:02.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a852869062b9911d-00631f8c02-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.876 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:02.876 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:02.877 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:02.877 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:02.878 DEBUG:boto:Method: HEAD 2022-09-12T19:44:02.878 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:02.879 DEBUG:boto:Data: 2022-09-12T19:44:02.879 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.880 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:02.880 DEBUG:boto:Port: 8000 2022-09-12T19:44:02.881 DEBUG:boto:Params: {} 2022-09-12T19:44:02.881 DEBUG:boto:Token: None 2022-09-12T19:44:02.882 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:02.883 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:g4iu0X+N5RThWIC9VXGPNQRoonY= 2022-09-12T19:44:02.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:g4iu0X+N5RThWIC9VXGPNQRoonY=', 'Content-Length': '0'} 2022-09-12T19:44:02.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efeea67e7a32a44f-00631f8c02-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:44:02.887 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:02.888 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:02.888 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:02.889 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:02.890 DEBUG:boto:Method: GET 2022-09-12T19:44:02.890 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:02.891 DEBUG:boto:Data: 2022-09-12T19:44:02.891 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.892 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:02.893 DEBUG:boto:Port: 8001 2022-09-12T19:44:02.893 DEBUG:boto:Params: {} 2022-09-12T19:44:02.894 DEBUG:boto:Token: None 2022-09-12T19:44:02.895 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:02.895 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MO5JoonmCO96t2mJO/ELTbGB1gs= 2022-09-12T19:44:02.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MO5JoonmCO96t2mJO/ELTbGB1gs=', 'Content-Length': '0'} 2022-09-12T19:44:02.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c7023463a78bbb4-00631f8c02-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.946 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:02.947 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:02.947 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:02.948 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:02.948 DEBUG:boto:Method: GET 2022-09-12T19:44:02.949 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:02.949 DEBUG:boto:Data: 2022-09-12T19:44:02.950 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.951 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:02.951 DEBUG:boto:Port: 8000 2022-09-12T19:44:02.952 DEBUG:boto:Params: {} 2022-09-12T19:44:02.952 DEBUG:boto:Token: None 2022-09-12T19:44:02.953 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:02.954 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MO5JoonmCO96t2mJO/ELTbGB1gs= 2022-09-12T19:44:02.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MO5JoonmCO96t2mJO/ELTbGB1gs=', 'Content-Length': '0'} 2022-09-12T19:44:02.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009662451cc1c8372e-00631f8c02-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.958 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:02.958 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:02.959 DEBUG:boto:path=/jdpiak-71/?prefix=f 2022-09-12T19:44:02.959 DEBUG:boto:auth_path=/jdpiak-71/?prefix=f 2022-09-12T19:44:02.960 DEBUG:boto:Method: GET 2022-09-12T19:44:02.961 DEBUG:boto:Path: /jdpiak-71/?prefix=f 2022-09-12T19:44:02.961 DEBUG:boto:Data: 2022-09-12T19:44:02.962 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.962 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:02.963 DEBUG:boto:Port: 8001 2022-09-12T19:44:02.963 DEBUG:boto:Params: {} 2022-09-12T19:44:02.964 DEBUG:boto:Token: None 2022-09-12T19:44:02.965 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/ 2022-09-12T19:44:02.965 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk= 2022-09-12T19:44:02.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk=', 'Content-Length': '0'} 2022-09-12T19:44:02.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003ca7562b2979b82-00631f8c02-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.970 DEBUG:boto:b'jdpiak-71f1000falsef2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:02.971 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:02.972 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:02.972 DEBUG:boto:path=/jdpiak-71/?prefix=f 2022-09-12T19:44:02.973 DEBUG:boto:auth_path=/jdpiak-71/?prefix=f 2022-09-12T19:44:02.973 DEBUG:boto:Method: GET 2022-09-12T19:44:02.974 DEBUG:boto:Path: /jdpiak-71/?prefix=f 2022-09-12T19:44:02.975 DEBUG:boto:Data: 2022-09-12T19:44:02.975 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.976 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:02.976 DEBUG:boto:Port: 8000 2022-09-12T19:44:02.977 DEBUG:boto:Params: {} 2022-09-12T19:44:02.977 DEBUG:boto:Token: None 2022-09-12T19:44:02.978 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/ 2022-09-12T19:44:02.979 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk= 2022-09-12T19:44:02.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9AcNma6S2wmfdVLYourcSf3Zfvk=', 'Content-Length': '0'} 2022-09-12T19:44:02.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f3693a393540b7c-00631f8c02-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.983 DEBUG:boto:b'jdpiak-71f1000falsef2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:02.984 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:02.984 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:02.985 DEBUG:boto:Method: HEAD 2022-09-12T19:44:02.985 DEBUG:boto:Path: /jdpiak-71/f 2022-09-12T19:44:02.986 DEBUG:boto:Data: 2022-09-12T19:44:02.986 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.987 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:02.987 DEBUG:boto:Port: 8001 2022-09-12T19:44:02.988 DEBUG:boto:Params: {} 2022-09-12T19:44:02.988 DEBUG:boto:Token: None 2022-09-12T19:44:02.989 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/f 2022-09-12T19:44:02.990 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:grWshceLOuyM8FfguUeYDZgn704= 2022-09-12T19:44:02.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:grWshceLOuyM8FfguUeYDZgn704=', 'Content-Length': '0'} 2022-09-12T19:44:02.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3c606726c8d8c40-00631f8c02-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:02.993 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:02.994 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:02.994 DEBUG:boto:Method: HEAD 2022-09-12T19:44:02.995 DEBUG:boto:Path: /jdpiak-71/f 2022-09-12T19:44:02.995 DEBUG:boto:Data: 2022-09-12T19:44:02.996 DEBUG:boto:Headers: {} 2022-09-12T19:44:02.997 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:02.997 DEBUG:boto:Port: 8000 2022-09-12T19:44:02.998 DEBUG:boto:Params: {} 2022-09-12T19:44:02.998 DEBUG:boto:Token: None 2022-09-12T19:44:02.999 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:02 GMT /jdpiak-71/f 2022-09-12T19:44:03.000 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:grWshceLOuyM8FfguUeYDZgn704= 2022-09-12T19:44:03.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:grWshceLOuyM8FfguUeYDZgn704=', 'Content-Length': '0'} 2022-09-12T19:44:03.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065a5f051cbf2dc90-00631f8c02-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:44:03.004 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:03.004 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:03.005 DEBUG:boto:Method: GET 2022-09-12T19:44:03.006 DEBUG:boto:Path: /jdpiak-71/f 2022-09-12T19:44:03.006 DEBUG:boto:Data: 2022-09-12T19:44:03.007 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.007 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:03.008 DEBUG:boto:Port: 8001 2022-09-12T19:44:03.008 DEBUG:boto:Params: {} 2022-09-12T19:44:03.009 DEBUG:boto:Token: None 2022-09-12T19:44:03.009 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/f 2022-09-12T19:44:03.010 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2WrMWCS67AQvHPMaUGeFLUho6Xc= 2022-09-12T19:44:03.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2WrMWCS67AQvHPMaUGeFLUho6Xc=', 'Content-Length': '0'} 2022-09-12T19:44:03.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e80f2050d209018-00631f8c03-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.014 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:03.015 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:03.015 DEBUG:boto:Method: GET 2022-09-12T19:44:03.016 DEBUG:boto:Path: /jdpiak-71/f 2022-09-12T19:44:03.016 DEBUG:boto:Data: 2022-09-12T19:44:03.017 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.017 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:03.018 DEBUG:boto:Port: 8000 2022-09-12T19:44:03.018 DEBUG:boto:Params: {} 2022-09-12T19:44:03.019 DEBUG:boto:Token: None 2022-09-12T19:44:03.020 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/f 2022-09-12T19:44:03.021 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2WrMWCS67AQvHPMaUGeFLUho6Xc= 2022-09-12T19:44:03.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2WrMWCS67AQvHPMaUGeFLUho6Xc=', 'Content-Length': '0'} 2022-09-12T19:44:03.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bc3c8b9f650faf5-00631f8c03-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.067 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:03.068 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:03.069 DEBUG:boto:path=/jdpiak-71/?prefix=f 2022-09-12T19:44:03.070 DEBUG:boto:auth_path=/jdpiak-71/?prefix=f 2022-09-12T19:44:03.071 DEBUG:boto:Method: GET 2022-09-12T19:44:03.072 DEBUG:boto:Path: /jdpiak-71/?prefix=f 2022-09-12T19:44:03.073 DEBUG:boto:Data: 2022-09-12T19:44:03.074 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.075 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:03.076 DEBUG:boto:Port: 8001 2022-09-12T19:44:03.077 DEBUG:boto:Params: {} 2022-09-12T19:44:03.078 DEBUG:boto:Token: None 2022-09-12T19:44:03.079 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/ 2022-09-12T19:44:03.080 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6wwu8KM8iSTAJzrBGhLpsD5okNI= 2022-09-12T19:44:03.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6wwu8KM8iSTAJzrBGhLpsD5okNI=', 'Content-Length': '0'} 2022-09-12T19:44:03.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df1e4883fddff781-00631f8c03-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.085 DEBUG:boto:b'jdpiak-71f1000falsef2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:03.086 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:03.087 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:03.087 DEBUG:boto:path=/jdpiak-71/?prefix=f 2022-09-12T19:44:03.088 DEBUG:boto:auth_path=/jdpiak-71/?prefix=f 2022-09-12T19:44:03.088 DEBUG:boto:Method: GET 2022-09-12T19:44:03.089 DEBUG:boto:Path: /jdpiak-71/?prefix=f 2022-09-12T19:44:03.090 DEBUG:boto:Data: 2022-09-12T19:44:03.090 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.091 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:03.091 DEBUG:boto:Port: 8000 2022-09-12T19:44:03.092 DEBUG:boto:Params: {} 2022-09-12T19:44:03.093 DEBUG:boto:Token: None 2022-09-12T19:44:03.094 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/ 2022-09-12T19:44:03.095 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6wwu8KM8iSTAJzrBGhLpsD5okNI= 2022-09-12T19:44:03.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6wwu8KM8iSTAJzrBGhLpsD5okNI=', 'Content-Length': '0'} 2022-09-12T19:44:03.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012a13ff86658fc17-00631f8c03-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.099 DEBUG:boto:b'jdpiak-71f1000falsef2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:03.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:44:03.101 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:03.102 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:03.102 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:03.105 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:03.106 DEBUG:boto:Method: GET 2022-09-12T19:44:03.107 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:03.107 DEBUG:boto:Data: 2022-09-12T19:44:03.108 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.108 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:03.109 DEBUG:boto:Port: 8001 2022-09-12T19:44:03.110 DEBUG:boto:Params: {} 2022-09-12T19:44:03.111 DEBUG:boto:Token: None 2022-09-12T19:44:03.112 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:03.112 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vkK4ndSev4jiJpOGfP2O/OL32U0= 2022-09-12T19:44:03.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vkK4ndSev4jiJpOGfP2O/OL32U0=', 'Content-Length': '0'} 2022-09-12T19:44:03.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061a7cf2a19860c8d-00631f8c03-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.120 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:03.121 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:03.121 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:03.122 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:03.123 DEBUG:boto:Method: GET 2022-09-12T19:44:03.123 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:03.124 DEBUG:boto:Data: 2022-09-12T19:44:03.124 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.125 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:03.126 DEBUG:boto:Port: 8000 2022-09-12T19:44:03.126 DEBUG:boto:Params: {} 2022-09-12T19:44:03.127 DEBUG:boto:Token: None 2022-09-12T19:44:03.127 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:03.128 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vkK4ndSev4jiJpOGfP2O/OL32U0= 2022-09-12T19:44:03.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vkK4ndSev4jiJpOGfP2O/OL32U0=', 'Content-Length': '0'} 2022-09-12T19:44:03.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7f4a8383ed001a7-00631f8c03-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.132 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:03.132 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:03.133 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:03.133 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:03.134 DEBUG:boto:Method: HEAD 2022-09-12T19:44:03.134 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:03.135 DEBUG:boto:Data: 2022-09-12T19:44:03.135 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.136 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:03.136 DEBUG:boto:Port: 8001 2022-09-12T19:44:03.137 DEBUG:boto:Params: {} 2022-09-12T19:44:03.138 DEBUG:boto:Token: None 2022-09-12T19:44:03.138 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:03.139 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vj4dmI3bd5h8dRZTMVXX9Y5OOiA= 2022-09-12T19:44:03.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vj4dmI3bd5h8dRZTMVXX9Y5OOiA=', 'Content-Length': '0'} 2022-09-12T19:44:03.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008559e6e68d270674-00631f8c03-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.142 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:03.143 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:03.143 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:03.144 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:03.144 DEBUG:boto:Method: HEAD 2022-09-12T19:44:03.145 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:03.146 DEBUG:boto:Data: 2022-09-12T19:44:03.146 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.147 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:03.147 DEBUG:boto:Port: 8000 2022-09-12T19:44:03.148 DEBUG:boto:Params: {} 2022-09-12T19:44:03.148 DEBUG:boto:Token: None 2022-09-12T19:44:03.149 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:03.149 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vj4dmI3bd5h8dRZTMVXX9Y5OOiA= 2022-09-12T19:44:03.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vj4dmI3bd5h8dRZTMVXX9Y5OOiA=', 'Content-Length': '0'} 2022-09-12T19:44:03.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd180240b21b5d10-00631f8c03-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:44:03.154 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:03.154 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:03.155 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:03.156 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:03.156 DEBUG:boto:Method: GET 2022-09-12T19:44:03.157 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:03.157 DEBUG:boto:Data: 2022-09-12T19:44:03.158 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.159 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:03.159 DEBUG:boto:Port: 8001 2022-09-12T19:44:03.160 DEBUG:boto:Params: {} 2022-09-12T19:44:03.160 DEBUG:boto:Token: None 2022-09-12T19:44:03.161 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:03.162 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vkK4ndSev4jiJpOGfP2O/OL32U0= 2022-09-12T19:44:03.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vkK4ndSev4jiJpOGfP2O/OL32U0=', 'Content-Length': '0'} 2022-09-12T19:44:03.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000352188d584b78802-00631f8c03-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.166 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:03.166 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:03.167 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:03.168 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:03.168 DEBUG:boto:Method: GET 2022-09-12T19:44:03.169 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:03.170 DEBUG:boto:Data: 2022-09-12T19:44:03.170 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.171 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:03.171 DEBUG:boto:Port: 8000 2022-09-12T19:44:03.172 DEBUG:boto:Params: {} 2022-09-12T19:44:03.172 DEBUG:boto:Token: None 2022-09-12T19:44:03.173 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:03.174 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vkK4ndSev4jiJpOGfP2O/OL32U0= 2022-09-12T19:44:03.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vkK4ndSev4jiJpOGfP2O/OL32U0=', 'Content-Length': '0'} 2022-09-12T19:44:03.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037d193b4612acc3a-00631f8c03-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.218 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:03.219 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:03.220 DEBUG:boto:path=/jdpiak-71/?prefix=g 2022-09-12T19:44:03.221 DEBUG:boto:auth_path=/jdpiak-71/?prefix=g 2022-09-12T19:44:03.222 DEBUG:boto:Method: GET 2022-09-12T19:44:03.223 DEBUG:boto:Path: /jdpiak-71/?prefix=g 2022-09-12T19:44:03.223 DEBUG:boto:Data: 2022-09-12T19:44:03.224 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.225 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:03.225 DEBUG:boto:Port: 8001 2022-09-12T19:44:03.226 DEBUG:boto:Params: {} 2022-09-12T19:44:03.226 DEBUG:boto:Token: None 2022-09-12T19:44:03.227 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/ 2022-09-12T19:44:03.228 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6wwu8KM8iSTAJzrBGhLpsD5okNI= 2022-09-12T19:44:03.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6wwu8KM8iSTAJzrBGhLpsD5okNI=', 'Content-Length': '0'} 2022-09-12T19:44:03.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dae22601fed57395-00631f8c03-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.232 DEBUG:boto:b'jdpiak-71g1000falseg2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:03.233 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:03.234 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:03.234 DEBUG:boto:path=/jdpiak-71/?prefix=g 2022-09-12T19:44:03.235 DEBUG:boto:auth_path=/jdpiak-71/?prefix=g 2022-09-12T19:44:03.235 DEBUG:boto:Method: GET 2022-09-12T19:44:03.236 DEBUG:boto:Path: /jdpiak-71/?prefix=g 2022-09-12T19:44:03.236 DEBUG:boto:Data: 2022-09-12T19:44:03.237 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.237 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:03.238 DEBUG:boto:Port: 8000 2022-09-12T19:44:03.239 DEBUG:boto:Params: {} 2022-09-12T19:44:03.239 DEBUG:boto:Token: None 2022-09-12T19:44:03.240 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/ 2022-09-12T19:44:03.240 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6wwu8KM8iSTAJzrBGhLpsD5okNI= 2022-09-12T19:44:03.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6wwu8KM8iSTAJzrBGhLpsD5okNI=', 'Content-Length': '0'} 2022-09-12T19:44:03.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017b43dc215b47a0a-00631f8c03-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.248 DEBUG:boto:b'jdpiak-71g1000falseg2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:03.250 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:03.251 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:03.252 DEBUG:boto:Method: HEAD 2022-09-12T19:44:03.253 DEBUG:boto:Path: /jdpiak-71/g 2022-09-12T19:44:03.254 DEBUG:boto:Data: 2022-09-12T19:44:03.255 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.256 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:03.256 DEBUG:boto:Port: 8001 2022-09-12T19:44:03.257 DEBUG:boto:Params: {} 2022-09-12T19:44:03.258 DEBUG:boto:Token: None 2022-09-12T19:44:03.259 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/g 2022-09-12T19:44:03.260 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ltKOGWOzANkHiwhmClhTJQB6VTQ= 2022-09-12T19:44:03.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ltKOGWOzANkHiwhmClhTJQB6VTQ=', 'Content-Length': '0'} 2022-09-12T19:44:03.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0d455ace53b5fa4-00631f8c03-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.266 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:03.267 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:03.268 DEBUG:boto:Method: HEAD 2022-09-12T19:44:03.269 DEBUG:boto:Path: /jdpiak-71/g 2022-09-12T19:44:03.271 DEBUG:boto:Data: 2022-09-12T19:44:03.272 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.275 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:03.276 DEBUG:boto:Port: 8000 2022-09-12T19:44:03.277 DEBUG:boto:Params: {} 2022-09-12T19:44:03.278 DEBUG:boto:Token: None 2022-09-12T19:44:03.279 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/g 2022-09-12T19:44:03.280 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ltKOGWOzANkHiwhmClhTJQB6VTQ= 2022-09-12T19:44:03.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ltKOGWOzANkHiwhmClhTJQB6VTQ=', 'Content-Length': '0'} 2022-09-12T19:44:03.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000981e02c7541b4bbe-00631f8c03-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:44:03.286 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:03.287 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:03.288 DEBUG:boto:Method: GET 2022-09-12T19:44:03.289 DEBUG:boto:Path: /jdpiak-71/g 2022-09-12T19:44:03.290 DEBUG:boto:Data: 2022-09-12T19:44:03.291 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.292 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:03.294 DEBUG:boto:Port: 8001 2022-09-12T19:44:03.295 DEBUG:boto:Params: {} 2022-09-12T19:44:03.297 DEBUG:boto:Token: None 2022-09-12T19:44:03.298 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/g 2022-09-12T19:44:03.299 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EqwBCGRjmgENaGGjDEPs5eefV+k= 2022-09-12T19:44:03.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EqwBCGRjmgENaGGjDEPs5eefV+k=', 'Content-Length': '0'} 2022-09-12T19:44:03.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6899f383044b842-00631f8c03-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.346 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:03.348 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:03.349 DEBUG:boto:Method: GET 2022-09-12T19:44:03.350 DEBUG:boto:Path: /jdpiak-71/g 2022-09-12T19:44:03.351 DEBUG:boto:Data: 2022-09-12T19:44:03.352 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.353 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:03.354 DEBUG:boto:Port: 8000 2022-09-12T19:44:03.355 DEBUG:boto:Params: {} 2022-09-12T19:44:03.357 DEBUG:boto:Token: None 2022-09-12T19:44:03.358 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/g 2022-09-12T19:44:03.359 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EqwBCGRjmgENaGGjDEPs5eefV+k= 2022-09-12T19:44:03.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EqwBCGRjmgENaGGjDEPs5eefV+k=', 'Content-Length': '0'} 2022-09-12T19:44:03.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5179468a9c5cac5-00631f8c03-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.366 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:03.367 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:03.368 DEBUG:boto:path=/jdpiak-71/?prefix=g 2022-09-12T19:44:03.373 DEBUG:boto:auth_path=/jdpiak-71/?prefix=g 2022-09-12T19:44:03.375 DEBUG:boto:Method: GET 2022-09-12T19:44:03.376 DEBUG:boto:Path: /jdpiak-71/?prefix=g 2022-09-12T19:44:03.377 DEBUG:boto:Data: 2022-09-12T19:44:03.379 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.379 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:03.380 DEBUG:boto:Port: 8001 2022-09-12T19:44:03.381 DEBUG:boto:Params: {} 2022-09-12T19:44:03.383 DEBUG:boto:Token: None 2022-09-12T19:44:03.385 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/ 2022-09-12T19:44:03.386 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6wwu8KM8iSTAJzrBGhLpsD5okNI= 2022-09-12T19:44:03.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6wwu8KM8iSTAJzrBGhLpsD5okNI=', 'Content-Length': '0'} 2022-09-12T19:44:03.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8f0e17c3906ae61-00631f8c03-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.395 DEBUG:boto:b'jdpiak-71g1000falseg2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:03.397 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:03.399 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:03.400 DEBUG:boto:path=/jdpiak-71/?prefix=g 2022-09-12T19:44:03.402 DEBUG:boto:auth_path=/jdpiak-71/?prefix=g 2022-09-12T19:44:03.404 DEBUG:boto:Method: GET 2022-09-12T19:44:03.405 DEBUG:boto:Path: /jdpiak-71/?prefix=g 2022-09-12T19:44:03.406 DEBUG:boto:Data: 2022-09-12T19:44:03.407 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.408 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:03.409 DEBUG:boto:Port: 8000 2022-09-12T19:44:03.410 DEBUG:boto:Params: {} 2022-09-12T19:44:03.411 DEBUG:boto:Token: None 2022-09-12T19:44:03.412 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/ 2022-09-12T19:44:03.413 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6wwu8KM8iSTAJzrBGhLpsD5okNI= 2022-09-12T19:44:03.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6wwu8KM8iSTAJzrBGhLpsD5okNI=', 'Content-Length': '0'} 2022-09-12T19:44:03.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f38d3502a8f5662d-00631f8c03-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.420 DEBUG:boto:b'jdpiak-71g1000falseg2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:03.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:44:03.423 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:03.425 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:03.426 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:03.427 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:03.428 DEBUG:boto:Method: GET 2022-09-12T19:44:03.429 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:03.430 DEBUG:boto:Data: 2022-09-12T19:44:03.430 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.431 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:03.432 DEBUG:boto:Port: 8001 2022-09-12T19:44:03.433 DEBUG:boto:Params: {} 2022-09-12T19:44:03.435 DEBUG:boto:Token: None 2022-09-12T19:44:03.436 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:03.437 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ML8eHVMJZ9LVYUBuBYmvHcTvRtg= 2022-09-12T19:44:03.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ML8eHVMJZ9LVYUBuBYmvHcTvRtg=', 'Content-Length': '0'} 2022-09-12T19:44:03.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000537864363c163778-00631f8c03-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.452 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:03.453 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:03.454 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:03.454 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:03.455 DEBUG:boto:Method: GET 2022-09-12T19:44:03.455 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:03.457 DEBUG:boto:Data: 2022-09-12T19:44:03.458 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.459 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:03.460 DEBUG:boto:Port: 8000 2022-09-12T19:44:03.461 DEBUG:boto:Params: {} 2022-09-12T19:44:03.462 DEBUG:boto:Token: None 2022-09-12T19:44:03.463 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:03.464 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ML8eHVMJZ9LVYUBuBYmvHcTvRtg= 2022-09-12T19:44:03.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ML8eHVMJZ9LVYUBuBYmvHcTvRtg=', 'Content-Length': '0'} 2022-09-12T19:44:03.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070c39b05b0e05bff-00631f8c03-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.472 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:03.473 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:03.474 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:03.476 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:03.477 DEBUG:boto:Method: HEAD 2022-09-12T19:44:03.479 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:03.480 DEBUG:boto:Data: 2022-09-12T19:44:03.482 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.483 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:03.484 DEBUG:boto:Port: 8001 2022-09-12T19:44:03.485 DEBUG:boto:Params: {} 2022-09-12T19:44:03.487 DEBUG:boto:Token: None 2022-09-12T19:44:03.488 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:03.491 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hjRb7AnX0rMNuq9thhDudE0Lk7E= 2022-09-12T19:44:03.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hjRb7AnX0rMNuq9thhDudE0Lk7E=', 'Content-Length': '0'} 2022-09-12T19:44:03.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b48c0237559dd0ed-00631f8c03-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.503 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:03.504 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:03.506 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:03.507 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:03.508 DEBUG:boto:Method: HEAD 2022-09-12T19:44:03.509 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:03.511 DEBUG:boto:Data: 2022-09-12T19:44:03.512 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.513 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:03.514 DEBUG:boto:Port: 8000 2022-09-12T19:44:03.516 DEBUG:boto:Params: {} 2022-09-12T19:44:03.518 DEBUG:boto:Token: None 2022-09-12T19:44:03.519 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:03.520 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hjRb7AnX0rMNuq9thhDudE0Lk7E= 2022-09-12T19:44:03.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hjRb7AnX0rMNuq9thhDudE0Lk7E=', 'Content-Length': '0'} 2022-09-12T19:44:03.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fcdf336f4fecb5a-00631f8c03-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:44:03.528 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:03.529 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:03.530 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:03.531 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:03.532 DEBUG:boto:Method: GET 2022-09-12T19:44:03.533 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:03.534 DEBUG:boto:Data: 2022-09-12T19:44:03.534 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.535 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:03.536 DEBUG:boto:Port: 8001 2022-09-12T19:44:03.536 DEBUG:boto:Params: {} 2022-09-12T19:44:03.537 DEBUG:boto:Token: None 2022-09-12T19:44:03.538 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:03.538 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ML8eHVMJZ9LVYUBuBYmvHcTvRtg= 2022-09-12T19:44:03.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ML8eHVMJZ9LVYUBuBYmvHcTvRtg=', 'Content-Length': '0'} 2022-09-12T19:44:03.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fe50905ebd51d49-00631f8c03-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.545 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:03.545 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:03.546 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:03.547 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:03.548 DEBUG:boto:Method: GET 2022-09-12T19:44:03.549 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:03.550 DEBUG:boto:Data: 2022-09-12T19:44:03.551 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.551 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:03.552 DEBUG:boto:Port: 8000 2022-09-12T19:44:03.553 DEBUG:boto:Params: {} 2022-09-12T19:44:03.554 DEBUG:boto:Token: None 2022-09-12T19:44:03.555 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:03.566 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ML8eHVMJZ9LVYUBuBYmvHcTvRtg= 2022-09-12T19:44:03.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ML8eHVMJZ9LVYUBuBYmvHcTvRtg=', 'Content-Length': '0'} 2022-09-12T19:44:03.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2ab4cb54dc7d16c-00631f8c03-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.572 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:03.573 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:03.574 DEBUG:boto:path=/jdpiak-71/?prefix=h 2022-09-12T19:44:03.575 DEBUG:boto:auth_path=/jdpiak-71/?prefix=h 2022-09-12T19:44:03.576 DEBUG:boto:Method: GET 2022-09-12T19:44:03.578 DEBUG:boto:Path: /jdpiak-71/?prefix=h 2022-09-12T19:44:03.579 DEBUG:boto:Data: 2022-09-12T19:44:03.580 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.581 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:03.582 DEBUG:boto:Port: 8001 2022-09-12T19:44:03.583 DEBUG:boto:Params: {} 2022-09-12T19:44:03.585 DEBUG:boto:Token: None 2022-09-12T19:44:03.587 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/ 2022-09-12T19:44:03.588 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6wwu8KM8iSTAJzrBGhLpsD5okNI= 2022-09-12T19:44:03.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6wwu8KM8iSTAJzrBGhLpsD5okNI=', 'Content-Length': '0'} 2022-09-12T19:44:03.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bca1405d0a8f008e-00631f8c03-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.604 DEBUG:boto:b'jdpiak-71h1000falseh2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:03.605 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:03.606 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:03.607 DEBUG:boto:path=/jdpiak-71/?prefix=h 2022-09-12T19:44:03.608 DEBUG:boto:auth_path=/jdpiak-71/?prefix=h 2022-09-12T19:44:03.610 DEBUG:boto:Method: GET 2022-09-12T19:44:03.611 DEBUG:boto:Path: /jdpiak-71/?prefix=h 2022-09-12T19:44:03.612 DEBUG:boto:Data: 2022-09-12T19:44:03.613 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.614 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:03.615 DEBUG:boto:Port: 8000 2022-09-12T19:44:03.618 DEBUG:boto:Params: {} 2022-09-12T19:44:03.619 DEBUG:boto:Token: None 2022-09-12T19:44:03.621 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/ 2022-09-12T19:44:03.622 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6wwu8KM8iSTAJzrBGhLpsD5okNI= 2022-09-12T19:44:03.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6wwu8KM8iSTAJzrBGhLpsD5okNI=', 'Content-Length': '0'} 2022-09-12T19:44:03.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bb1788bf1fe332d-00631f8c03-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.628 DEBUG:boto:b'jdpiak-71h1000falseh2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:03.629 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:03.635 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:03.636 DEBUG:boto:Method: HEAD 2022-09-12T19:44:03.637 DEBUG:boto:Path: /jdpiak-71/h 2022-09-12T19:44:03.638 DEBUG:boto:Data: 2022-09-12T19:44:03.639 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.641 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:03.641 DEBUG:boto:Port: 8001 2022-09-12T19:44:03.642 DEBUG:boto:Params: {} 2022-09-12T19:44:03.644 DEBUG:boto:Token: None 2022-09-12T19:44:03.647 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/h 2022-09-12T19:44:03.648 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:harXN335uBLTu2fyj7mcH7Tssw0= 2022-09-12T19:44:03.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:harXN335uBLTu2fyj7mcH7Tssw0=', 'Content-Length': '0'} 2022-09-12T19:44:03.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062a7051afbe5f83d-00631f8c03-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.654 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:03.656 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:03.657 DEBUG:boto:Method: HEAD 2022-09-12T19:44:03.658 DEBUG:boto:Path: /jdpiak-71/h 2022-09-12T19:44:03.659 DEBUG:boto:Data: 2022-09-12T19:44:03.660 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.661 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:03.661 DEBUG:boto:Port: 8000 2022-09-12T19:44:03.662 DEBUG:boto:Params: {} 2022-09-12T19:44:03.662 DEBUG:boto:Token: None 2022-09-12T19:44:03.663 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/h 2022-09-12T19:44:03.664 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:harXN335uBLTu2fyj7mcH7Tssw0= 2022-09-12T19:44:03.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:harXN335uBLTu2fyj7mcH7Tssw0=', 'Content-Length': '0'} 2022-09-12T19:44:03.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8670a1187f3ae7d-00631f8c03-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:44:03.673 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:03.674 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:03.675 DEBUG:boto:Method: GET 2022-09-12T19:44:03.676 DEBUG:boto:Path: /jdpiak-71/h 2022-09-12T19:44:03.677 DEBUG:boto:Data: 2022-09-12T19:44:03.678 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.679 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:03.680 DEBUG:boto:Port: 8001 2022-09-12T19:44:03.681 DEBUG:boto:Params: {} 2022-09-12T19:44:03.682 DEBUG:boto:Token: None 2022-09-12T19:44:03.684 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/h 2022-09-12T19:44:03.685 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9aO+rMYTuIjJJXGpXcP/GtqdHT4= 2022-09-12T19:44:03.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9aO+rMYTuIjJJXGpXcP/GtqdHT4=', 'Content-Length': '0'} 2022-09-12T19:44:03.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b543ab4006371b6-00631f8c03-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.690 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:03.691 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:03.692 DEBUG:boto:Method: GET 2022-09-12T19:44:03.693 DEBUG:boto:Path: /jdpiak-71/h 2022-09-12T19:44:03.695 DEBUG:boto:Data: 2022-09-12T19:44:03.695 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.697 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:03.698 DEBUG:boto:Port: 8000 2022-09-12T19:44:03.699 DEBUG:boto:Params: {} 2022-09-12T19:44:03.700 DEBUG:boto:Token: None 2022-09-12T19:44:03.701 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/h 2022-09-12T19:44:03.702 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9aO+rMYTuIjJJXGpXcP/GtqdHT4= 2022-09-12T19:44:03.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9aO+rMYTuIjJJXGpXcP/GtqdHT4=', 'Content-Length': '0'} 2022-09-12T19:44:03.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e345f6c0ff81d20c-00631f8c03-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.754 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:03.755 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:03.763 DEBUG:boto:path=/jdpiak-71/?prefix=h 2022-09-12T19:44:03.765 DEBUG:boto:auth_path=/jdpiak-71/?prefix=h 2022-09-12T19:44:03.767 DEBUG:boto:Method: GET 2022-09-12T19:44:03.768 DEBUG:boto:Path: /jdpiak-71/?prefix=h 2022-09-12T19:44:03.770 DEBUG:boto:Data: 2022-09-12T19:44:03.771 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.772 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:03.772 DEBUG:boto:Port: 8001 2022-09-12T19:44:03.773 DEBUG:boto:Params: {} 2022-09-12T19:44:03.776 DEBUG:boto:Token: None 2022-09-12T19:44:03.777 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/ 2022-09-12T19:44:03.778 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6wwu8KM8iSTAJzrBGhLpsD5okNI= 2022-09-12T19:44:03.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6wwu8KM8iSTAJzrBGhLpsD5okNI=', 'Content-Length': '0'} 2022-09-12T19:44:03.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008bddb532e1d299b-00631f8c03-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.795 DEBUG:boto:b'jdpiak-71h1000falseh2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:03.797 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:03.799 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:03.800 DEBUG:boto:path=/jdpiak-71/?prefix=h 2022-09-12T19:44:03.811 DEBUG:boto:auth_path=/jdpiak-71/?prefix=h 2022-09-12T19:44:03.814 DEBUG:boto:Method: GET 2022-09-12T19:44:03.816 DEBUG:boto:Path: /jdpiak-71/?prefix=h 2022-09-12T19:44:03.816 DEBUG:boto:Data: 2022-09-12T19:44:03.817 DEBUG:boto:Headers: {} 2022-09-12T19:44:03.819 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:03.820 DEBUG:boto:Port: 8000 2022-09-12T19:44:03.820 DEBUG:boto:Params: {} 2022-09-12T19:44:03.821 DEBUG:boto:Token: None 2022-09-12T19:44:03.822 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:03 GMT /jdpiak-71/ 2022-09-12T19:44:03.823 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6wwu8KM8iSTAJzrBGhLpsD5okNI= 2022-09-12T19:44:03.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6wwu8KM8iSTAJzrBGhLpsD5okNI=', 'Content-Length': '0'} 2022-09-12T19:44:03.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9a6715bb62258bf-00631f8c03-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:03.833 DEBUG:boto:b'jdpiak-71h1000falseh2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:03.834 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-71 zones={test-zone3, test-zone2} 2022-09-12T19:44:03.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'jdpiak-71', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:44:03.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'jdpiak-71', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:44:03.837 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket jdpiak-71 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:44:03.897 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:44:03.898 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:44:03.949 INFO:teuthology.orchestra.run.smithi035.stdout:tenant: 2022-09-12T19:44:03.950 INFO:teuthology.orchestra.run.smithi035.stdout:bucket name: jdpiak-71 2022-09-12T19:44:03.951 INFO:teuthology.orchestra.run.smithi035.stdout:total entries: 8 2022-09-12T19:44:03.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'jdpiak-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:44:03.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'jdpiak-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:44:03.998 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket jdpiak-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:44:04.047 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:44:04.048 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:44:04.110 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:44:04.111 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:44:04.111 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding": "None", 2022-09-12T19:44:04.111 INFO:teuthology.orchestra.run.smithi035.stdout: "current_index": { 2022-09-12T19:44:04.112 INFO:teuthology.orchestra.run.smithi035.stdout: "gen": 2, 2022-09-12T19:44:04.112 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:44:04.113 INFO:teuthology.orchestra.run.smithi035.stdout: "type": "Normal", 2022-09-12T19:44:04.113 INFO:teuthology.orchestra.run.smithi035.stdout: "normal": { 2022-09-12T19:44:04.116 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 3, 2022-09-12T19:44:04.116 INFO:teuthology.orchestra.run.smithi035.stdout: "hash_type": "Mod" 2022-09-12T19:44:04.116 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:04.117 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:04.117 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:44:04.118 INFO:teuthology.orchestra.run.smithi035.stdout: "logs": [ 2022-09-12T19:44:04.118 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:44:04.119 INFO:teuthology.orchestra.run.smithi035.stdout: "gen": 0, 2022-09-12T19:44:04.119 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:44:04.119 INFO:teuthology.orchestra.run.smithi035.stdout: "type": "InIndex", 2022-09-12T19:44:04.120 INFO:teuthology.orchestra.run.smithi035.stdout: "in_index": { 2022-09-12T19:44:04.120 INFO:teuthology.orchestra.run.smithi035.stdout: "gen": 0, 2022-09-12T19:44:04.121 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:44:04.121 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 11, 2022-09-12T19:44:04.121 INFO:teuthology.orchestra.run.smithi035.stdout: "hash_type": "Mod" 2022-09-12T19:44:04.122 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:04.122 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:04.123 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:04.123 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:44:04.123 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:44:04.124 INFO:teuthology.orchestra.run.smithi035.stdout: "gen": 1, 2022-09-12T19:44:04.124 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:44:04.125 INFO:teuthology.orchestra.run.smithi035.stdout: "type": "InIndex", 2022-09-12T19:44:04.125 INFO:teuthology.orchestra.run.smithi035.stdout: "in_index": { 2022-09-12T19:44:04.126 INFO:teuthology.orchestra.run.smithi035.stdout: "gen": 1, 2022-09-12T19:44:04.127 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:44:04.127 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 3, 2022-09-12T19:44:04.128 INFO:teuthology.orchestra.run.smithi035.stdout: "hash_type": "Mod" 2022-09-12T19:44:04.128 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:04.128 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:04.129 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:04.129 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:44:04.130 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:44:04.130 INFO:teuthology.orchestra.run.smithi035.stdout: "gen": 2, 2022-09-12T19:44:04.130 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:44:04.131 INFO:teuthology.orchestra.run.smithi035.stdout: "type": "InIndex", 2022-09-12T19:44:04.131 INFO:teuthology.orchestra.run.smithi035.stdout: "in_index": { 2022-09-12T19:44:04.132 INFO:teuthology.orchestra.run.smithi035.stdout: "gen": 2, 2022-09-12T19:44:04.132 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:44:04.132 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 3, 2022-09-12T19:44:04.133 INFO:teuthology.orchestra.run.smithi035.stdout: "hash_type": "Mod" 2022-09-12T19:44:04.133 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:04.134 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:04.134 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:04.135 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:04.135 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:44:04.135 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:04.136 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:44:04.138 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-71 2022-09-12T19:44:04.139 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:44:04.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:44:04.142 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-71 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:44:04.206 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:44:04.207 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:44:04.244 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:44:04.241+0000 7f7e04199600 1 waiting to reach latest gen 2, on 1.. 2022-09-12T19:44:09.246 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:44:09.242+0000 7f7e04199600 1 waiting to reach latest gen 2, on 1.. 2022-09-12T19:44:14.245 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:44:14.242+0000 7f7e04199600 1 waiting to reach latest gen 2, on 1.. 2022-09-12T19:44:19.246 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:44:19.243+0000 7f7e04199600 1 bucket sync caught up with empty source 2022-09-12T19:44:19.247 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:44:19.243+0000 7f7e04199600 0 bucket checkpoint complete 2022-09-12T19:44:19.269 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-71 2022-09-12T19:44:19.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:44:19.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:44:19.274 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-71 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:44:19.342 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:44:19.342 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:44:19.400 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:44:19.394+0000 7f762e42b600 1 bucket sync caught up with empty source 2022-09-12T19:44:19.401 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:44:19.394+0000 7f762e42b600 0 bucket checkpoint complete 2022-09-12T19:44:19.414 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-71 2022-09-12T19:44:19.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:44:19.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:44:19.418 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-71 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:44:19.484 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:44:19.485 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:44:19.533 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:44:19.528+0000 7f893b4e2600 1 bucket sync caught up with empty source 2022-09-12T19:44:19.534 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:44:19.528+0000 7f893b4e2600 0 bucket checkpoint complete 2022-09-12T19:44:19.544 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-71 2022-09-12T19:44:19.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:44:19.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:44:19.548 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-71 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:44:19.608 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:44:19.609 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:44:19.655 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:44:19.652+0000 7fc289dc5600 1 bucket sync caught up with source: 2022-09-12T19:44:19.655 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.890.5, , , 00000000001.976.5, 00000000001.1376.5, , , 00000000002.1141.5, , 00000000002.514.5] 2022-09-12T19:44:19.655 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.890.5, , , 00000000001.976.5, 00000000001.1376.5, , , 00000000002.1141.5, , 00000000002.514.5] 2022-09-12T19:44:19.656 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:44:19.652+0000 7fc289dc5600 0 bucket checkpoint complete 2022-09-12T19:44:19.669 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-71 2022-09-12T19:44:19.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:44:19.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:44:19.673 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-71 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:44:19.732 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:44:19.733 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:44:19.790 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:44:19.780+0000 7ff5e2791600 1 bucket sync caught up with source: 2022-09-12T19:44:19.791 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.890.5, , , 00000000001.976.5, 00000000001.1376.5, , , 00000000002.1141.5, , 00000000002.514.5] 2022-09-12T19:44:19.791 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.890.5, , , 00000000001.976.5, 00000000001.1376.5, , , 00000000002.1141.5, , 00000000002.514.5] 2022-09-12T19:44:19.791 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:44:19.780+0000 7ff5e2791600 0 bucket checkpoint complete 2022-09-12T19:44:19.819 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-71 2022-09-12T19:44:19.821 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-71 2022-09-12T19:44:19.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:44:19.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:44:19.823 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-71 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:44:19.881 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:44:19.882 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:44:19.940 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:44:19.938+0000 7f6e56c0d600 1 bucket sync caught up with source: 2022-09-12T19:44:19.941 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.500.5, , , 00000000001.486.5, 00000000001.744.5, , , 00000000002.451.5, , 00000000002.246.5] 2022-09-12T19:44:19.942 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.500.5, , , 00000000001.486.5, 00000000001.744.5, , , 00000000002.451.5, , 00000000002.246.5] 2022-09-12T19:44:19.942 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:44:19.938+0000 7f6e56c0d600 0 bucket checkpoint complete 2022-09-12T19:44:19.955 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-71 2022-09-12T19:44:19.956 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:44:19.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-71', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:44:19.958 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-71 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:44:20.018 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:44:20.019 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:44:20.062 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:44:20.060+0000 7f5e1e1b8600 1 bucket sync caught up with source: 2022-09-12T19:44:20.063 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.500.5, , , 00000000001.486.5, 00000000001.744.5, , , 00000000002.451.5, , 00000000002.246.5] 2022-09-12T19:44:20.063 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000002.500.5, , , 00000000001.486.5, 00000000001.744.5, , , 00000000002.451.5, , 00000000002.246.5] 2022-09-12T19:44:20.063 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:44:20.060+0000 7f5e1e1b8600 0 bucket checkpoint complete 2022-09-12T19:44:20.080 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-71 2022-09-12T19:44:20.081 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-71 zones={test-zone2, test-zone1} 2022-09-12T19:44:20.082 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:20.083 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:20.083 DEBUG:boto:Method: HEAD 2022-09-12T19:44:20.084 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:44:20.084 DEBUG:boto:Data: 2022-09-12T19:44:20.085 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.085 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:20.086 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.087 DEBUG:boto:Params: {} 2022-09-12T19:44:20.087 DEBUG:boto:Token: None 2022-09-12T19:44:20.088 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/ 2022-09-12T19:44:20.089 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jv4aoir011aGk2ga+P8no6LfwKA= 2022-09-12T19:44:20.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jv4aoir011aGk2ga+P8no6LfwKA=', 'Content-Length': '0'} 2022-09-12T19:44:20.092 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000967db00151680ad7-00631f8c14-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.093 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:20.093 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:20.094 DEBUG:boto:Method: HEAD 2022-09-12T19:44:20.095 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:44:20.095 DEBUG:boto:Data: 2022-09-12T19:44:20.096 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.096 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:20.097 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.097 DEBUG:boto:Params: {} 2022-09-12T19:44:20.098 DEBUG:boto:Token: None 2022-09-12T19:44:20.098 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/ 2022-09-12T19:44:20.099 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jv4aoir011aGk2ga+P8no6LfwKA= 2022-09-12T19:44:20.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jv4aoir011aGk2ga+P8no6LfwKA=', 'Content-Length': '0'} 2022-09-12T19:44:20.101 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1b3e3922b76e5ca-00631f8c14-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.102 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:44:20.103 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:20.104 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:20.105 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:20.105 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:20.106 DEBUG:boto:Method: GET 2022-09-12T19:44:20.106 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:20.107 DEBUG:boto:Data: 2022-09-12T19:44:20.107 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.108 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:20.110 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.111 DEBUG:boto:Params: {} 2022-09-12T19:44:20.111 DEBUG:boto:Token: None 2022-09-12T19:44:20.112 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/?versions 2022-09-12T19:44:20.113 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mlVNeudzfyiFzXog+SZtwjFpZ0o= 2022-09-12T19:44:20.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mlVNeudzfyiFzXog+SZtwjFpZ0o=', 'Content-Length': '0'} 2022-09-12T19:44:20.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e98e5cb55ca52d4-00631f8c14-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.128 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:20.129 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:44:20.130 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:44:20.131 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:44:20.131 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:44:20.134 DEBUG:tasks.rgw_multi.zone_rados:o=e 2022-09-12T19:44:20.135 DEBUG:tasks.rgw_multi.zone_rados:o=f 2022-09-12T19:44:20.138 DEBUG:tasks.rgw_multi.zone_rados:o=g 2022-09-12T19:44:20.139 DEBUG:tasks.rgw_multi.zone_rados:o=h 2022-09-12T19:44:20.139 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:44:20.140 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:20.141 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:20.141 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:20.142 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:20.143 DEBUG:boto:Method: GET 2022-09-12T19:44:20.144 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:20.144 DEBUG:boto:Data: 2022-09-12T19:44:20.145 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.145 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:20.146 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.147 DEBUG:boto:Params: {} 2022-09-12T19:44:20.148 DEBUG:boto:Token: None 2022-09-12T19:44:20.149 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/?versions 2022-09-12T19:44:20.150 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mlVNeudzfyiFzXog+SZtwjFpZ0o= 2022-09-12T19:44:20.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mlVNeudzfyiFzXog+SZtwjFpZ0o=', 'Content-Length': '0'} 2022-09-12T19:44:20.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d06e85b74e62f3d4-00631f8c14-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.154 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:20.155 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:44:20.155 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:44:20.156 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:44:20.157 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:44:20.158 DEBUG:tasks.rgw_multi.zone_rados:o=e 2022-09-12T19:44:20.158 DEBUG:tasks.rgw_multi.zone_rados:o=f 2022-09-12T19:44:20.159 DEBUG:tasks.rgw_multi.zone_rados:o=g 2022-09-12T19:44:20.160 DEBUG:tasks.rgw_multi.zone_rados:o=h 2022-09-12T19:44:20.160 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:20.161 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:20.162 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:20.162 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:20.163 DEBUG:boto:Method: GET 2022-09-12T19:44:20.164 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:20.164 DEBUG:boto:Data: 2022-09-12T19:44:20.166 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.167 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:20.167 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.168 DEBUG:boto:Params: {} 2022-09-12T19:44:20.169 DEBUG:boto:Token: None 2022-09-12T19:44:20.170 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/?versions 2022-09-12T19:44:20.171 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mlVNeudzfyiFzXog+SZtwjFpZ0o= 2022-09-12T19:44:20.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mlVNeudzfyiFzXog+SZtwjFpZ0o=', 'Content-Length': '0'} 2022-09-12T19:44:20.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000beffa7fe88e0ddd1-00631f8c14-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.176 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:20.178 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:20.179 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:20.180 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:20.181 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:20.181 DEBUG:boto:Method: GET 2022-09-12T19:44:20.182 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:20.183 DEBUG:boto:Data: 2022-09-12T19:44:20.183 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.184 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:20.185 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.185 DEBUG:boto:Params: {} 2022-09-12T19:44:20.186 DEBUG:boto:Token: None 2022-09-12T19:44:20.186 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/?versions 2022-09-12T19:44:20.187 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mlVNeudzfyiFzXog+SZtwjFpZ0o= 2022-09-12T19:44:20.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mlVNeudzfyiFzXog+SZtwjFpZ0o=', 'Content-Length': '0'} 2022-09-12T19:44:20.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085e3d74a4b096e44-00631f8c14-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.191 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:20.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:44:20.193 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:20.194 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:20.194 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:20.195 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:20.195 DEBUG:boto:Method: GET 2022-09-12T19:44:20.196 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:44:20.196 DEBUG:boto:Data: 2022-09-12T19:44:20.197 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.197 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:20.198 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.199 DEBUG:boto:Params: {} 2022-09-12T19:44:20.200 DEBUG:boto:Token: None 2022-09-12T19:44:20.201 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:44:20.201 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CPpL604hJEsV2gneXsQNLsKn4UU= 2022-09-12T19:44:20.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CPpL604hJEsV2gneXsQNLsKn4UU=', 'Content-Length': '0'} 2022-09-12T19:44:20.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dd51064a6fcd9a2-00631f8c14-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.205 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:20.205 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:20.206 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:20.207 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:20.207 DEBUG:boto:Method: GET 2022-09-12T19:44:20.208 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:44:20.209 DEBUG:boto:Data: 2022-09-12T19:44:20.209 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.210 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:20.210 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.211 DEBUG:boto:Params: {} 2022-09-12T19:44:20.212 DEBUG:boto:Token: None 2022-09-12T19:44:20.212 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:44:20.213 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CPpL604hJEsV2gneXsQNLsKn4UU= 2022-09-12T19:44:20.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CPpL604hJEsV2gneXsQNLsKn4UU=', 'Content-Length': '0'} 2022-09-12T19:44:20.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f07a9ddd423dc355-00631f8c14-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.217 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:20.217 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:20.218 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:20.218 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:20.219 DEBUG:boto:Method: HEAD 2022-09-12T19:44:20.219 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:44:20.220 DEBUG:boto:Data: 2022-09-12T19:44:20.221 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.221 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:20.222 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.222 DEBUG:boto:Params: {} 2022-09-12T19:44:20.223 DEBUG:boto:Token: None 2022-09-12T19:44:20.224 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:44:20.225 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4axXDMpzpk/VB+WHZ4XOshD4Kws= 2022-09-12T19:44:20.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4axXDMpzpk/VB+WHZ4XOshD4Kws=', 'Content-Length': '0'} 2022-09-12T19:44:20.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0967eafee571f23-00631f8c14-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.228 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:20.229 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:20.229 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:20.230 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:20.230 DEBUG:boto:Method: HEAD 2022-09-12T19:44:20.231 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:44:20.232 DEBUG:boto:Data: 2022-09-12T19:44:20.232 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.233 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:20.233 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.234 DEBUG:boto:Params: {} 2022-09-12T19:44:20.235 DEBUG:boto:Token: None 2022-09-12T19:44:20.235 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:44:20.236 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4axXDMpzpk/VB+WHZ4XOshD4Kws= 2022-09-12T19:44:20.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4axXDMpzpk/VB+WHZ4XOshD4Kws=', 'Content-Length': '0'} 2022-09-12T19:44:20.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005148719f11168eee-00631f8c14-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:44:20.241 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:20.242 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:20.242 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:20.243 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:20.243 DEBUG:boto:Method: GET 2022-09-12T19:44:20.244 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:44:20.245 DEBUG:boto:Data: 2022-09-12T19:44:20.245 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.246 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:20.247 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.247 DEBUG:boto:Params: {} 2022-09-12T19:44:20.248 DEBUG:boto:Token: None 2022-09-12T19:44:20.249 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:44:20.249 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CPpL604hJEsV2gneXsQNLsKn4UU= 2022-09-12T19:44:20.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CPpL604hJEsV2gneXsQNLsKn4UU=', 'Content-Length': '0'} 2022-09-12T19:44:20.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000439d1961ef6786a1-00631f8c14-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.253 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:20.253 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:20.254 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:20.255 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:20.256 DEBUG:boto:Method: GET 2022-09-12T19:44:20.256 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:44:20.257 DEBUG:boto:Data: 2022-09-12T19:44:20.257 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.258 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:20.259 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.259 DEBUG:boto:Params: {} 2022-09-12T19:44:20.260 DEBUG:boto:Token: None 2022-09-12T19:44:20.261 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:44:20.261 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CPpL604hJEsV2gneXsQNLsKn4UU= 2022-09-12T19:44:20.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CPpL604hJEsV2gneXsQNLsKn4UU=', 'Content-Length': '0'} 2022-09-12T19:44:20.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd7512d4ac7e46b0-00631f8c14-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.265 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:20.266 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:20.266 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:44:20.267 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:44:20.268 DEBUG:boto:Method: GET 2022-09-12T19:44:20.268 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:44:20.269 DEBUG:boto:Data: 2022-09-12T19:44:20.269 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.270 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:20.270 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.271 DEBUG:boto:Params: {} 2022-09-12T19:44:20.272 DEBUG:boto:Token: None 2022-09-12T19:44:20.272 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/ 2022-09-12T19:44:20.273 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FPJUJ9fOOn2DPpKikjkMTDo+DNs= 2022-09-12T19:44:20.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FPJUJ9fOOn2DPpKikjkMTDo+DNs=', 'Content-Length': '0'} 2022-09-12T19:44:20.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000340028089c88a497-00631f8c14-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.277 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:20.278 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:20.279 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:20.279 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:44:20.280 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:44:20.281 DEBUG:boto:Method: GET 2022-09-12T19:44:20.281 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:44:20.282 DEBUG:boto:Data: 2022-09-12T19:44:20.283 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.284 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:20.284 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.285 DEBUG:boto:Params: {} 2022-09-12T19:44:20.285 DEBUG:boto:Token: None 2022-09-12T19:44:20.286 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/ 2022-09-12T19:44:20.287 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FPJUJ9fOOn2DPpKikjkMTDo+DNs= 2022-09-12T19:44:20.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FPJUJ9fOOn2DPpKikjkMTDo+DNs=', 'Content-Length': '0'} 2022-09-12T19:44:20.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d25ba5c7b4566a6d-00631f8c14-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.290 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:20.291 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:20.292 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:20.292 DEBUG:boto:Method: HEAD 2022-09-12T19:44:20.293 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:44:20.293 DEBUG:boto:Data: 2022-09-12T19:44:20.294 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.295 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:20.295 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.296 DEBUG:boto:Params: {} 2022-09-12T19:44:20.296 DEBUG:boto:Token: None 2022-09-12T19:44:20.297 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/a 2022-09-12T19:44:20.298 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MGhMDMclRlh1Z86thHS6p97UKVY= 2022-09-12T19:44:20.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MGhMDMclRlh1Z86thHS6p97UKVY=', 'Content-Length': '0'} 2022-09-12T19:44:20.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000616a4cb00df9e70a-00631f8c14-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.301 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:20.302 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:20.303 DEBUG:boto:Method: HEAD 2022-09-12T19:44:20.303 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:44:20.304 DEBUG:boto:Data: 2022-09-12T19:44:20.305 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.305 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:20.306 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.306 DEBUG:boto:Params: {} 2022-09-12T19:44:20.307 DEBUG:boto:Token: None 2022-09-12T19:44:20.307 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/a 2022-09-12T19:44:20.308 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MGhMDMclRlh1Z86thHS6p97UKVY= 2022-09-12T19:44:20.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MGhMDMclRlh1Z86thHS6p97UKVY=', 'Content-Length': '0'} 2022-09-12T19:44:20.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004758a692fcdeaf45-00631f8c14-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:44:20.312 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:20.313 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:20.313 DEBUG:boto:Method: GET 2022-09-12T19:44:20.314 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:44:20.315 DEBUG:boto:Data: 2022-09-12T19:44:20.315 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.316 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:20.316 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.317 DEBUG:boto:Params: {} 2022-09-12T19:44:20.317 DEBUG:boto:Token: None 2022-09-12T19:44:20.318 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/a 2022-09-12T19:44:20.319 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JzAHGAWyK1J7ILIlDMklPrmuoCs= 2022-09-12T19:44:20.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JzAHGAWyK1J7ILIlDMklPrmuoCs=', 'Content-Length': '0'} 2022-09-12T19:44:20.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5520d2de6792079-00631f8c14-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.324 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:20.325 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:20.326 DEBUG:boto:Method: GET 2022-09-12T19:44:20.327 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:44:20.328 DEBUG:boto:Data: 2022-09-12T19:44:20.329 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.330 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:20.331 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.332 DEBUG:boto:Params: {} 2022-09-12T19:44:20.333 DEBUG:boto:Token: None 2022-09-12T19:44:20.334 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/a 2022-09-12T19:44:20.334 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JzAHGAWyK1J7ILIlDMklPrmuoCs= 2022-09-12T19:44:20.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JzAHGAWyK1J7ILIlDMklPrmuoCs=', 'Content-Length': '0'} 2022-09-12T19:44:20.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009edbb47a58df4966-00631f8c14-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.378 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:20.379 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:20.380 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:44:20.380 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:44:20.381 DEBUG:boto:Method: GET 2022-09-12T19:44:20.381 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:44:20.382 DEBUG:boto:Data: 2022-09-12T19:44:20.383 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.383 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:20.384 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.385 DEBUG:boto:Params: {} 2022-09-12T19:44:20.386 DEBUG:boto:Token: None 2022-09-12T19:44:20.387 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/ 2022-09-12T19:44:20.387 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FPJUJ9fOOn2DPpKikjkMTDo+DNs= 2022-09-12T19:44:20.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FPJUJ9fOOn2DPpKikjkMTDo+DNs=', 'Content-Length': '0'} 2022-09-12T19:44:20.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009078878cf036b57d-00631f8c14-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.391 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:20.392 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:20.393 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:20.393 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:44:20.394 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:44:20.394 DEBUG:boto:Method: GET 2022-09-12T19:44:20.395 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:44:20.396 DEBUG:boto:Data: 2022-09-12T19:44:20.397 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.397 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:20.398 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.399 DEBUG:boto:Params: {} 2022-09-12T19:44:20.399 DEBUG:boto:Token: None 2022-09-12T19:44:20.400 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/ 2022-09-12T19:44:20.401 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FPJUJ9fOOn2DPpKikjkMTDo+DNs= 2022-09-12T19:44:20.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FPJUJ9fOOn2DPpKikjkMTDo+DNs=', 'Content-Length': '0'} 2022-09-12T19:44:20.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d01efaf2037dfb3-00631f8c14-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.405 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:20.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:44:20.406 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:20.407 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:20.408 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:20.409 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:20.411 DEBUG:boto:Method: GET 2022-09-12T19:44:20.412 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:44:20.413 DEBUG:boto:Data: 2022-09-12T19:44:20.414 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.415 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:20.416 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.417 DEBUG:boto:Params: {} 2022-09-12T19:44:20.418 DEBUG:boto:Token: None 2022-09-12T19:44:20.419 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:44:20.420 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cHByYp1VpRwmbxlyilUGxK5JPLk= 2022-09-12T19:44:20.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cHByYp1VpRwmbxlyilUGxK5JPLk=', 'Content-Length': '0'} 2022-09-12T19:44:20.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bd352402e2d2656-00631f8c14-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.426 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:20.427 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:20.428 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:20.429 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:20.430 DEBUG:boto:Method: GET 2022-09-12T19:44:20.431 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:44:20.432 DEBUG:boto:Data: 2022-09-12T19:44:20.432 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.433 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:20.433 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.434 DEBUG:boto:Params: {} 2022-09-12T19:44:20.435 DEBUG:boto:Token: None 2022-09-12T19:44:20.435 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:44:20.436 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cHByYp1VpRwmbxlyilUGxK5JPLk= 2022-09-12T19:44:20.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cHByYp1VpRwmbxlyilUGxK5JPLk=', 'Content-Length': '0'} 2022-09-12T19:44:20.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad25cbefe44b822b-00631f8c14-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.440 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:20.441 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:20.442 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:20.443 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:20.444 DEBUG:boto:Method: HEAD 2022-09-12T19:44:20.444 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:44:20.445 DEBUG:boto:Data: 2022-09-12T19:44:20.446 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.447 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:20.448 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.449 DEBUG:boto:Params: {} 2022-09-12T19:44:20.449 DEBUG:boto:Token: None 2022-09-12T19:44:20.450 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:44:20.451 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D4FB+k1j+fzYD8I0oalTAXDTSMc= 2022-09-12T19:44:20.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D4FB+k1j+fzYD8I0oalTAXDTSMc=', 'Content-Length': '0'} 2022-09-12T19:44:20.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9b2e4b298c02e5a-00631f8c14-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.454 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:20.455 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:20.456 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:20.456 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:20.457 DEBUG:boto:Method: HEAD 2022-09-12T19:44:20.457 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:44:20.458 DEBUG:boto:Data: 2022-09-12T19:44:20.459 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.459 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:20.460 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.460 DEBUG:boto:Params: {} 2022-09-12T19:44:20.461 DEBUG:boto:Token: None 2022-09-12T19:44:20.462 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:44:20.463 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D4FB+k1j+fzYD8I0oalTAXDTSMc= 2022-09-12T19:44:20.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D4FB+k1j+fzYD8I0oalTAXDTSMc=', 'Content-Length': '0'} 2022-09-12T19:44:20.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002efbc9ea6d1c10c6-00631f8c14-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:44:20.467 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:20.468 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:20.469 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:20.470 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:20.471 DEBUG:boto:Method: GET 2022-09-12T19:44:20.472 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:44:20.472 DEBUG:boto:Data: 2022-09-12T19:44:20.473 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.474 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:20.475 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.475 DEBUG:boto:Params: {} 2022-09-12T19:44:20.476 DEBUG:boto:Token: None 2022-09-12T19:44:20.477 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:44:20.478 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cHByYp1VpRwmbxlyilUGxK5JPLk= 2022-09-12T19:44:20.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cHByYp1VpRwmbxlyilUGxK5JPLk=', 'Content-Length': '0'} 2022-09-12T19:44:20.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063695ce92701c490-00631f8c14-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.522 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:20.523 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:20.523 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:20.524 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:20.525 DEBUG:boto:Method: GET 2022-09-12T19:44:20.525 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:44:20.526 DEBUG:boto:Data: 2022-09-12T19:44:20.527 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.527 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:20.528 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.529 DEBUG:boto:Params: {} 2022-09-12T19:44:20.530 DEBUG:boto:Token: None 2022-09-12T19:44:20.530 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:44:20.531 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cHByYp1VpRwmbxlyilUGxK5JPLk= 2022-09-12T19:44:20.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cHByYp1VpRwmbxlyilUGxK5JPLk=', 'Content-Length': '0'} 2022-09-12T19:44:20.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d1d06de44e93dec-00631f8c14-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.535 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:20.536 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:20.537 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:44:20.538 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:44:20.538 DEBUG:boto:Method: GET 2022-09-12T19:44:20.539 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:44:20.539 DEBUG:boto:Data: 2022-09-12T19:44:20.540 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.540 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:20.541 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.541 DEBUG:boto:Params: {} 2022-09-12T19:44:20.542 DEBUG:boto:Token: None 2022-09-12T19:44:20.543 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/ 2022-09-12T19:44:20.543 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FPJUJ9fOOn2DPpKikjkMTDo+DNs= 2022-09-12T19:44:20.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FPJUJ9fOOn2DPpKikjkMTDo+DNs=', 'Content-Length': '0'} 2022-09-12T19:44:20.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fabdd1a19d2db4c-00631f8c14-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.549 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:20.550 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:20.551 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:20.552 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:44:20.552 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:44:20.553 DEBUG:boto:Method: GET 2022-09-12T19:44:20.553 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:44:20.554 DEBUG:boto:Data: 2022-09-12T19:44:20.554 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.555 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:20.555 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.556 DEBUG:boto:Params: {} 2022-09-12T19:44:20.557 DEBUG:boto:Token: None 2022-09-12T19:44:20.557 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/ 2022-09-12T19:44:20.559 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FPJUJ9fOOn2DPpKikjkMTDo+DNs= 2022-09-12T19:44:20.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FPJUJ9fOOn2DPpKikjkMTDo+DNs=', 'Content-Length': '0'} 2022-09-12T19:44:20.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b16aaedce054660e-00631f8c14-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.563 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:20.564 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:20.564 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:20.565 DEBUG:boto:Method: HEAD 2022-09-12T19:44:20.565 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:44:20.566 DEBUG:boto:Data: 2022-09-12T19:44:20.566 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.567 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:20.567 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.568 DEBUG:boto:Params: {} 2022-09-12T19:44:20.569 DEBUG:boto:Token: None 2022-09-12T19:44:20.570 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/b 2022-09-12T19:44:20.570 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A6FRUyeZnY0hQwNwjIXl4/bE8I4= 2022-09-12T19:44:20.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A6FRUyeZnY0hQwNwjIXl4/bE8I4=', 'Content-Length': '0'} 2022-09-12T19:44:20.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e6cf6fd2de495a4-00631f8c14-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.575 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:20.576 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:20.577 DEBUG:boto:Method: HEAD 2022-09-12T19:44:20.578 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:44:20.580 DEBUG:boto:Data: 2022-09-12T19:44:20.581 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.582 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:20.583 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.584 DEBUG:boto:Params: {} 2022-09-12T19:44:20.585 DEBUG:boto:Token: None 2022-09-12T19:44:20.586 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/b 2022-09-12T19:44:20.588 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A6FRUyeZnY0hQwNwjIXl4/bE8I4= 2022-09-12T19:44:20.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A6FRUyeZnY0hQwNwjIXl4/bE8I4=', 'Content-Length': '0'} 2022-09-12T19:44:20.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ddc7ea1568a95a5-00631f8c14-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:44:20.594 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:20.594 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:20.595 DEBUG:boto:Method: GET 2022-09-12T19:44:20.595 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:44:20.596 DEBUG:boto:Data: 2022-09-12T19:44:20.597 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.597 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:20.598 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.598 DEBUG:boto:Params: {} 2022-09-12T19:44:20.599 DEBUG:boto:Token: None 2022-09-12T19:44:20.600 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/b 2022-09-12T19:44:20.600 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zvD5zAsav3Nvv6Ji1/UR51b3G6k= 2022-09-12T19:44:20.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zvD5zAsav3Nvv6Ji1/UR51b3G6k=', 'Content-Length': '0'} 2022-09-12T19:44:20.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a7667db3d5fb4be-00631f8c14-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.604 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:20.605 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:20.605 DEBUG:boto:Method: GET 2022-09-12T19:44:20.606 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:44:20.607 DEBUG:boto:Data: 2022-09-12T19:44:20.607 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.608 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:20.608 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.609 DEBUG:boto:Params: {} 2022-09-12T19:44:20.609 DEBUG:boto:Token: None 2022-09-12T19:44:20.610 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/b 2022-09-12T19:44:20.611 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zvD5zAsav3Nvv6Ji1/UR51b3G6k= 2022-09-12T19:44:20.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zvD5zAsav3Nvv6Ji1/UR51b3G6k=', 'Content-Length': '0'} 2022-09-12T19:44:20.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000474947019a859aa6-00631f8c14-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.617 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:20.617 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:20.619 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:44:20.620 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:44:20.620 DEBUG:boto:Method: GET 2022-09-12T19:44:20.621 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:44:20.622 DEBUG:boto:Data: 2022-09-12T19:44:20.622 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.623 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:20.624 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.625 DEBUG:boto:Params: {} 2022-09-12T19:44:20.626 DEBUG:boto:Token: None 2022-09-12T19:44:20.627 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/ 2022-09-12T19:44:20.628 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FPJUJ9fOOn2DPpKikjkMTDo+DNs= 2022-09-12T19:44:20.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FPJUJ9fOOn2DPpKikjkMTDo+DNs=', 'Content-Length': '0'} 2022-09-12T19:44:20.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd29c41f4d7242f3-00631f8c14-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.634 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:20.635 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:20.635 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:20.636 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:44:20.636 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:44:20.637 DEBUG:boto:Method: GET 2022-09-12T19:44:20.637 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:44:20.638 DEBUG:boto:Data: 2022-09-12T19:44:20.638 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.639 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:20.639 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.640 DEBUG:boto:Params: {} 2022-09-12T19:44:20.641 DEBUG:boto:Token: None 2022-09-12T19:44:20.641 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/ 2022-09-12T19:44:20.642 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FPJUJ9fOOn2DPpKikjkMTDo+DNs= 2022-09-12T19:44:20.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FPJUJ9fOOn2DPpKikjkMTDo+DNs=', 'Content-Length': '0'} 2022-09-12T19:44:20.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000413638f118d11226-00631f8c14-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.690 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:20.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:44:20.693 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:20.694 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:20.695 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:20.695 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:20.696 DEBUG:boto:Method: GET 2022-09-12T19:44:20.696 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:44:20.697 DEBUG:boto:Data: 2022-09-12T19:44:20.697 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.698 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:20.698 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.699 DEBUG:boto:Params: {} 2022-09-12T19:44:20.700 DEBUG:boto:Token: None 2022-09-12T19:44:20.700 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:44:20.701 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rSCxsx2I3zneXuqQvp4UDshE7wQ= 2022-09-12T19:44:20.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rSCxsx2I3zneXuqQvp4UDshE7wQ=', 'Content-Length': '0'} 2022-09-12T19:44:20.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015e951ffba680c20-00631f8c14-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.707 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:20.708 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:20.709 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:20.709 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:20.710 DEBUG:boto:Method: GET 2022-09-12T19:44:20.711 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:44:20.712 DEBUG:boto:Data: 2022-09-12T19:44:20.712 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.713 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:20.713 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.714 DEBUG:boto:Params: {} 2022-09-12T19:44:20.714 DEBUG:boto:Token: None 2022-09-12T19:44:20.715 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:44:20.716 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rSCxsx2I3zneXuqQvp4UDshE7wQ= 2022-09-12T19:44:20.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rSCxsx2I3zneXuqQvp4UDshE7wQ=', 'Content-Length': '0'} 2022-09-12T19:44:20.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058305105376e9d25-00631f8c14-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.720 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:20.720 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:20.721 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:20.721 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:20.722 DEBUG:boto:Method: HEAD 2022-09-12T19:44:20.723 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:44:20.723 DEBUG:boto:Data: 2022-09-12T19:44:20.724 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.724 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:20.725 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.726 DEBUG:boto:Params: {} 2022-09-12T19:44:20.727 DEBUG:boto:Token: None 2022-09-12T19:44:20.727 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:44:20.729 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mnzvMkHblFT12r2fGBfv46Mn3Wo= 2022-09-12T19:44:20.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mnzvMkHblFT12r2fGBfv46Mn3Wo=', 'Content-Length': '0'} 2022-09-12T19:44:20.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032928141967e989a-00631f8c14-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.732 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:20.733 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:20.734 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:20.735 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:20.735 DEBUG:boto:Method: HEAD 2022-09-12T19:44:20.736 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:44:20.736 DEBUG:boto:Data: 2022-09-12T19:44:20.737 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.737 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:20.738 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.739 DEBUG:boto:Params: {} 2022-09-12T19:44:20.739 DEBUG:boto:Token: None 2022-09-12T19:44:20.740 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:44:20.741 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mnzvMkHblFT12r2fGBfv46Mn3Wo= 2022-09-12T19:44:20.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mnzvMkHblFT12r2fGBfv46Mn3Wo=', 'Content-Length': '0'} 2022-09-12T19:44:20.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3e4ed57e5a40d46-00631f8c14-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:44:20.746 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:20.746 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:20.747 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:20.748 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:20.748 DEBUG:boto:Method: GET 2022-09-12T19:44:20.749 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:44:20.750 DEBUG:boto:Data: 2022-09-12T19:44:20.751 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.751 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:20.752 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.753 DEBUG:boto:Params: {} 2022-09-12T19:44:20.754 DEBUG:boto:Token: None 2022-09-12T19:44:20.755 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:44:20.755 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rSCxsx2I3zneXuqQvp4UDshE7wQ= 2022-09-12T19:44:20.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rSCxsx2I3zneXuqQvp4UDshE7wQ=', 'Content-Length': '0'} 2022-09-12T19:44:20.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fa878053c1f9f57-00631f8c14-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.759 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:20.760 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:20.760 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:20.761 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:20.761 DEBUG:boto:Method: GET 2022-09-12T19:44:20.762 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:44:20.763 DEBUG:boto:Data: 2022-09-12T19:44:20.764 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.765 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:20.765 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.766 DEBUG:boto:Params: {} 2022-09-12T19:44:20.767 DEBUG:boto:Token: None 2022-09-12T19:44:20.768 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:44:20.769 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rSCxsx2I3zneXuqQvp4UDshE7wQ= 2022-09-12T19:44:20.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rSCxsx2I3zneXuqQvp4UDshE7wQ=', 'Content-Length': '0'} 2022-09-12T19:44:20.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003577bad7393902ef-00631f8c14-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.773 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:20.774 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:20.775 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:44:20.775 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:44:20.776 DEBUG:boto:Method: GET 2022-09-12T19:44:20.776 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:44:20.777 DEBUG:boto:Data: 2022-09-12T19:44:20.777 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.778 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:20.779 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.780 DEBUG:boto:Params: {} 2022-09-12T19:44:20.780 DEBUG:boto:Token: None 2022-09-12T19:44:20.781 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/ 2022-09-12T19:44:20.782 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FPJUJ9fOOn2DPpKikjkMTDo+DNs= 2022-09-12T19:44:20.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FPJUJ9fOOn2DPpKikjkMTDo+DNs=', 'Content-Length': '0'} 2022-09-12T19:44:20.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ea2717efb092cc8-00631f8c14-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.826 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:20.828 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:20.828 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:20.830 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:44:20.831 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:44:20.832 DEBUG:boto:Method: GET 2022-09-12T19:44:20.833 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:44:20.833 DEBUG:boto:Data: 2022-09-12T19:44:20.834 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.835 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:20.836 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.837 DEBUG:boto:Params: {} 2022-09-12T19:44:20.837 DEBUG:boto:Token: None 2022-09-12T19:44:20.839 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/ 2022-09-12T19:44:20.840 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FPJUJ9fOOn2DPpKikjkMTDo+DNs= 2022-09-12T19:44:20.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FPJUJ9fOOn2DPpKikjkMTDo+DNs=', 'Content-Length': '0'} 2022-09-12T19:44:20.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5c03efc49d0e6a4-00631f8c14-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.845 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:20.847 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:20.848 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:20.849 DEBUG:boto:Method: HEAD 2022-09-12T19:44:20.849 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:44:20.850 DEBUG:boto:Data: 2022-09-12T19:44:20.851 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.852 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:20.853 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.855 DEBUG:boto:Params: {} 2022-09-12T19:44:20.856 DEBUG:boto:Token: None 2022-09-12T19:44:20.857 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/c 2022-09-12T19:44:20.858 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tEjKno/SguCzMa5o6JumSgxtXGs= 2022-09-12T19:44:20.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tEjKno/SguCzMa5o6JumSgxtXGs=', 'Content-Length': '0'} 2022-09-12T19:44:20.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9c3555e8728c423-00631f8c14-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.863 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:20.864 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:20.865 DEBUG:boto:Method: HEAD 2022-09-12T19:44:20.866 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:44:20.867 DEBUG:boto:Data: 2022-09-12T19:44:20.867 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.868 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:20.869 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.871 DEBUG:boto:Params: {} 2022-09-12T19:44:20.873 DEBUG:boto:Token: None 2022-09-12T19:44:20.874 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/c 2022-09-12T19:44:20.875 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tEjKno/SguCzMa5o6JumSgxtXGs= 2022-09-12T19:44:20.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tEjKno/SguCzMa5o6JumSgxtXGs=', 'Content-Length': '0'} 2022-09-12T19:44:20.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057b4d68a64104418-00631f8c14-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:44:20.882 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:20.883 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:20.883 DEBUG:boto:Method: GET 2022-09-12T19:44:20.884 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:44:20.885 DEBUG:boto:Data: 2022-09-12T19:44:20.885 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.886 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:20.886 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.887 DEBUG:boto:Params: {} 2022-09-12T19:44:20.887 DEBUG:boto:Token: None 2022-09-12T19:44:20.888 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/c 2022-09-12T19:44:20.889 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+CpCdV+Y2KP6we5m4hPSvdTuLZY= 2022-09-12T19:44:20.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+CpCdV+Y2KP6we5m4hPSvdTuLZY=', 'Content-Length': '0'} 2022-09-12T19:44:20.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a0fe6402b6df9b7-00631f8c14-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.892 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:20.893 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:20.893 DEBUG:boto:Method: GET 2022-09-12T19:44:20.894 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:44:20.894 DEBUG:boto:Data: 2022-09-12T19:44:20.895 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.895 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:20.896 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.896 DEBUG:boto:Params: {} 2022-09-12T19:44:20.897 DEBUG:boto:Token: None 2022-09-12T19:44:20.898 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/c 2022-09-12T19:44:20.898 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+CpCdV+Y2KP6we5m4hPSvdTuLZY= 2022-09-12T19:44:20.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+CpCdV+Y2KP6we5m4hPSvdTuLZY=', 'Content-Length': '0'} 2022-09-12T19:44:20.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000171c3df021dc10ba-00631f8c14-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.902 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:20.903 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:20.903 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:44:20.904 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:44:20.905 DEBUG:boto:Method: GET 2022-09-12T19:44:20.905 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:44:20.906 DEBUG:boto:Data: 2022-09-12T19:44:20.907 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.907 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:20.908 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.908 DEBUG:boto:Params: {} 2022-09-12T19:44:20.909 DEBUG:boto:Token: None 2022-09-12T19:44:20.910 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/ 2022-09-12T19:44:20.911 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FPJUJ9fOOn2DPpKikjkMTDo+DNs= 2022-09-12T19:44:20.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FPJUJ9fOOn2DPpKikjkMTDo+DNs=', 'Content-Length': '0'} 2022-09-12T19:44:20.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf35dc09d52d8a36-00631f8c14-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.916 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:20.917 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:20.920 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:20.921 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:44:20.922 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:44:20.923 DEBUG:boto:Method: GET 2022-09-12T19:44:20.924 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:44:20.926 DEBUG:boto:Data: 2022-09-12T19:44:20.927 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.928 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:20.929 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.931 DEBUG:boto:Params: {} 2022-09-12T19:44:20.933 DEBUG:boto:Token: None 2022-09-12T19:44:20.934 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/ 2022-09-12T19:44:20.936 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FPJUJ9fOOn2DPpKikjkMTDo+DNs= 2022-09-12T19:44:20.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FPJUJ9fOOn2DPpKikjkMTDo+DNs=', 'Content-Length': '0'} 2022-09-12T19:44:20.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8ec6800cd9212b2-00631f8c14-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.952 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:20.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:44:20.955 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:20.956 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:20.957 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:20.958 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:20.959 DEBUG:boto:Method: GET 2022-09-12T19:44:20.961 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:44:20.962 DEBUG:boto:Data: 2022-09-12T19:44:20.963 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.964 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:20.965 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.966 DEBUG:boto:Params: {} 2022-09-12T19:44:20.967 DEBUG:boto:Token: None 2022-09-12T19:44:20.969 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:44:20.970 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q+mfh6tpsiuQeP8MPQ1f6TjYKvo= 2022-09-12T19:44:20.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q+mfh6tpsiuQeP8MPQ1f6TjYKvo=', 'Content-Length': '0'} 2022-09-12T19:44:20.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e343c6075ff08e45-00631f8c14-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.976 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:20.977 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:20.978 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:20.980 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:20.982 DEBUG:boto:Method: GET 2022-09-12T19:44:20.983 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:44:20.984 DEBUG:boto:Data: 2022-09-12T19:44:20.985 DEBUG:boto:Headers: {} 2022-09-12T19:44:20.986 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:20.987 DEBUG:boto:Port: 8000 2022-09-12T19:44:20.988 DEBUG:boto:Params: {} 2022-09-12T19:44:20.990 DEBUG:boto:Token: None 2022-09-12T19:44:20.991 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:20 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:44:20.992 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q+mfh6tpsiuQeP8MPQ1f6TjYKvo= 2022-09-12T19:44:20.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q+mfh6tpsiuQeP8MPQ1f6TjYKvo=', 'Content-Length': '0'} 2022-09-12T19:44:20.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000362ef6a2b1c96efe-00631f8c14-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:20.998 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:20.999 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:21.000 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:21.001 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:21.002 DEBUG:boto:Method: HEAD 2022-09-12T19:44:21.003 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:44:21.004 DEBUG:boto:Data: 2022-09-12T19:44:21.005 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.005 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:21.006 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.007 DEBUG:boto:Params: {} 2022-09-12T19:44:21.007 DEBUG:boto:Token: None 2022-09-12T19:44:21.008 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:44:21.009 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TrOjl0aAUnQ8Sy8OAWX6qrmWAh0= 2022-09-12T19:44:21.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TrOjl0aAUnQ8Sy8OAWX6qrmWAh0=', 'Content-Length': '0'} 2022-09-12T19:44:21.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000970931355cd9006e-00631f8c15-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.012 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:21.013 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:21.013 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:21.014 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:21.015 DEBUG:boto:Method: HEAD 2022-09-12T19:44:21.015 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:44:21.016 DEBUG:boto:Data: 2022-09-12T19:44:21.017 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.017 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:21.022 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.023 DEBUG:boto:Params: {} 2022-09-12T19:44:21.024 DEBUG:boto:Token: None 2022-09-12T19:44:21.025 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:44:21.026 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TrOjl0aAUnQ8Sy8OAWX6qrmWAh0= 2022-09-12T19:44:21.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TrOjl0aAUnQ8Sy8OAWX6qrmWAh0=', 'Content-Length': '0'} 2022-09-12T19:44:21.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027621ede182931dd-00631f8c15-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:44:21.037 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:21.038 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:21.040 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:21.040 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:21.042 DEBUG:boto:Method: GET 2022-09-12T19:44:21.043 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:44:21.044 DEBUG:boto:Data: 2022-09-12T19:44:21.045 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.046 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:21.047 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.048 DEBUG:boto:Params: {} 2022-09-12T19:44:21.050 DEBUG:boto:Token: None 2022-09-12T19:44:21.051 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:44:21.054 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xer6xMC6MiR1g72P0x5zGpuLcuU= 2022-09-12T19:44:21.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xer6xMC6MiR1g72P0x5zGpuLcuU=', 'Content-Length': '0'} 2022-09-12T19:44:21.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c59f04c0ff697dfc-00631f8c15-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.060 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:21.062 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:21.063 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:21.064 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:21.065 DEBUG:boto:Method: GET 2022-09-12T19:44:21.065 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:44:21.066 DEBUG:boto:Data: 2022-09-12T19:44:21.067 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.067 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:21.068 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.068 DEBUG:boto:Params: {} 2022-09-12T19:44:21.069 DEBUG:boto:Token: None 2022-09-12T19:44:21.069 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:44:21.071 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xer6xMC6MiR1g72P0x5zGpuLcuU= 2022-09-12T19:44:21.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xer6xMC6MiR1g72P0x5zGpuLcuU=', 'Content-Length': '0'} 2022-09-12T19:44:21.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fc9d263d7004ec0-00631f8c15-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.076 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:21.077 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:21.077 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:44:21.078 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:44:21.079 DEBUG:boto:Method: GET 2022-09-12T19:44:21.079 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:44:21.080 DEBUG:boto:Data: 2022-09-12T19:44:21.080 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.081 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:21.082 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.082 DEBUG:boto:Params: {} 2022-09-12T19:44:21.083 DEBUG:boto:Token: None 2022-09-12T19:44:21.084 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/ 2022-09-12T19:44:21.084 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q9+ZV3fhH3dUpjHNyO/LWQG/FSA= 2022-09-12T19:44:21.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q9+ZV3fhH3dUpjHNyO/LWQG/FSA=', 'Content-Length': '0'} 2022-09-12T19:44:21.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c3478e40107fc7c-00631f8c15-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.090 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:21.092 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:21.093 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:21.093 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:44:21.095 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:44:21.096 DEBUG:boto:Method: GET 2022-09-12T19:44:21.096 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:44:21.097 DEBUG:boto:Data: 2022-09-12T19:44:21.098 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.099 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:21.099 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.100 DEBUG:boto:Params: {} 2022-09-12T19:44:21.100 DEBUG:boto:Token: None 2022-09-12T19:44:21.101 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/ 2022-09-12T19:44:21.102 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q9+ZV3fhH3dUpjHNyO/LWQG/FSA= 2022-09-12T19:44:21.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q9+ZV3fhH3dUpjHNyO/LWQG/FSA=', 'Content-Length': '0'} 2022-09-12T19:44:21.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c0080f5ba23356c-00631f8c15-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.147 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:21.149 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:21.150 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:21.151 DEBUG:boto:Method: HEAD 2022-09-12T19:44:21.151 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:44:21.152 DEBUG:boto:Data: 2022-09-12T19:44:21.153 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.154 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:21.155 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.156 DEBUG:boto:Params: {} 2022-09-12T19:44:21.157 DEBUG:boto:Token: None 2022-09-12T19:44:21.160 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/d 2022-09-12T19:44:21.161 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mtCCfMjvAtUhlW9GBK5ygYdnHZA= 2022-09-12T19:44:21.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mtCCfMjvAtUhlW9GBK5ygYdnHZA=', 'Content-Length': '0'} 2022-09-12T19:44:21.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abbcdcb049889838-00631f8c15-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.166 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:21.167 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:21.168 DEBUG:boto:Method: HEAD 2022-09-12T19:44:21.170 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:44:21.171 DEBUG:boto:Data: 2022-09-12T19:44:21.172 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.173 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:21.174 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.175 DEBUG:boto:Params: {} 2022-09-12T19:44:21.177 DEBUG:boto:Token: None 2022-09-12T19:44:21.178 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/d 2022-09-12T19:44:21.179 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mtCCfMjvAtUhlW9GBK5ygYdnHZA= 2022-09-12T19:44:21.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mtCCfMjvAtUhlW9GBK5ygYdnHZA=', 'Content-Length': '0'} 2022-09-12T19:44:21.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b31bee851138cd8-00631f8c15-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:44:21.190 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:21.191 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:21.192 DEBUG:boto:Method: GET 2022-09-12T19:44:21.193 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:44:21.194 DEBUG:boto:Data: 2022-09-12T19:44:21.195 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.196 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:21.198 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.199 DEBUG:boto:Params: {} 2022-09-12T19:44:21.200 DEBUG:boto:Token: None 2022-09-12T19:44:21.202 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/d 2022-09-12T19:44:21.203 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Wvw5XxXW0riIU9Gq8MQnXif1m7Y= 2022-09-12T19:44:21.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Wvw5XxXW0riIU9Gq8MQnXif1m7Y=', 'Content-Length': '0'} 2022-09-12T19:44:21.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e7b3936bde6c459-00631f8c15-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.209 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:21.210 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:21.211 DEBUG:boto:Method: GET 2022-09-12T19:44:21.213 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:44:21.214 DEBUG:boto:Data: 2022-09-12T19:44:21.215 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.216 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:21.217 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.218 DEBUG:boto:Params: {} 2022-09-12T19:44:21.219 DEBUG:boto:Token: None 2022-09-12T19:44:21.221 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/d 2022-09-12T19:44:21.222 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Wvw5XxXW0riIU9Gq8MQnXif1m7Y= 2022-09-12T19:44:21.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Wvw5XxXW0riIU9Gq8MQnXif1m7Y=', 'Content-Length': '0'} 2022-09-12T19:44:21.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8696b5a206acff3-00631f8c15-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.228 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:21.229 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:21.230 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:44:21.235 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:44:21.236 DEBUG:boto:Method: GET 2022-09-12T19:44:21.237 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:44:21.238 DEBUG:boto:Data: 2022-09-12T19:44:21.239 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.240 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:21.241 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.242 DEBUG:boto:Params: {} 2022-09-12T19:44:21.243 DEBUG:boto:Token: None 2022-09-12T19:44:21.244 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/ 2022-09-12T19:44:21.246 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q9+ZV3fhH3dUpjHNyO/LWQG/FSA= 2022-09-12T19:44:21.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q9+ZV3fhH3dUpjHNyO/LWQG/FSA=', 'Content-Length': '0'} 2022-09-12T19:44:21.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004a5ab131e6e72e8-00631f8c15-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.252 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:21.253 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:21.254 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:21.255 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:44:21.256 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:44:21.257 DEBUG:boto:Method: GET 2022-09-12T19:44:21.257 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:44:21.258 DEBUG:boto:Data: 2022-09-12T19:44:21.258 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.259 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:21.259 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.260 DEBUG:boto:Params: {} 2022-09-12T19:44:21.260 DEBUG:boto:Token: None 2022-09-12T19:44:21.261 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/ 2022-09-12T19:44:21.262 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q9+ZV3fhH3dUpjHNyO/LWQG/FSA= 2022-09-12T19:44:21.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q9+ZV3fhH3dUpjHNyO/LWQG/FSA=', 'Content-Length': '0'} 2022-09-12T19:44:21.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aff8c51faacadb4b-00631f8c15-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.268 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:21.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:44:21.272 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:21.273 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:21.274 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:21.275 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:21.276 DEBUG:boto:Method: GET 2022-09-12T19:44:21.278 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:44:21.279 DEBUG:boto:Data: 2022-09-12T19:44:21.281 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.282 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:21.283 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.284 DEBUG:boto:Params: {} 2022-09-12T19:44:21.286 DEBUG:boto:Token: None 2022-09-12T19:44:21.287 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:44:21.289 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Pnll73F8ByXLXBHk3vqd7s0TDOI= 2022-09-12T19:44:21.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Pnll73F8ByXLXBHk3vqd7s0TDOI=', 'Content-Length': '0'} 2022-09-12T19:44:21.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007432c20c8ffa3c8b-00631f8c15-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.295 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:21.297 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:21.298 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:21.299 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:21.299 DEBUG:boto:Method: GET 2022-09-12T19:44:21.300 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:44:21.301 DEBUG:boto:Data: 2022-09-12T19:44:21.301 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.302 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:21.302 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.303 DEBUG:boto:Params: {} 2022-09-12T19:44:21.304 DEBUG:boto:Token: None 2022-09-12T19:44:21.305 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:44:21.305 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Pnll73F8ByXLXBHk3vqd7s0TDOI= 2022-09-12T19:44:21.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Pnll73F8ByXLXBHk3vqd7s0TDOI=', 'Content-Length': '0'} 2022-09-12T19:44:21.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018c0e6405d95eb2f-00631f8c15-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.311 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:21.312 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:21.313 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:21.314 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:21.315 DEBUG:boto:Method: HEAD 2022-09-12T19:44:21.316 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:44:21.317 DEBUG:boto:Data: 2022-09-12T19:44:21.318 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.319 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:21.320 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.321 DEBUG:boto:Params: {} 2022-09-12T19:44:21.322 DEBUG:boto:Token: None 2022-09-12T19:44:21.323 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:44:21.324 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UL8FATZq9UzsvuDfIacBYVHEkvs= 2022-09-12T19:44:21.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UL8FATZq9UzsvuDfIacBYVHEkvs=', 'Content-Length': '0'} 2022-09-12T19:44:21.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d3e1b451bea8d5d-00631f8c15-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.329 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:21.330 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:21.330 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:21.331 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:21.332 DEBUG:boto:Method: HEAD 2022-09-12T19:44:21.332 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:44:21.333 DEBUG:boto:Data: 2022-09-12T19:44:21.335 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.336 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:21.336 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.337 DEBUG:boto:Params: {} 2022-09-12T19:44:21.338 DEBUG:boto:Token: None 2022-09-12T19:44:21.339 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:44:21.340 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UL8FATZq9UzsvuDfIacBYVHEkvs= 2022-09-12T19:44:21.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UL8FATZq9UzsvuDfIacBYVHEkvs=', 'Content-Length': '0'} 2022-09-12T19:44:21.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0d955641588419e-00631f8c15-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:44:21.346 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:21.347 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:21.348 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:21.349 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:21.350 DEBUG:boto:Method: GET 2022-09-12T19:44:21.351 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:44:21.352 DEBUG:boto:Data: 2022-09-12T19:44:21.353 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.353 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:21.355 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.356 DEBUG:boto:Params: {} 2022-09-12T19:44:21.357 DEBUG:boto:Token: None 2022-09-12T19:44:21.358 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:44:21.359 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Pnll73F8ByXLXBHk3vqd7s0TDOI= 2022-09-12T19:44:21.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Pnll73F8ByXLXBHk3vqd7s0TDOI=', 'Content-Length': '0'} 2022-09-12T19:44:21.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a5c06f2f62a20fe-00631f8c15-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.364 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:21.371 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:21.372 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:21.373 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:21.375 DEBUG:boto:Method: GET 2022-09-12T19:44:21.376 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:44:21.378 DEBUG:boto:Data: 2022-09-12T19:44:21.378 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.379 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:21.380 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.380 DEBUG:boto:Params: {} 2022-09-12T19:44:21.381 DEBUG:boto:Token: None 2022-09-12T19:44:21.381 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:44:21.382 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Pnll73F8ByXLXBHk3vqd7s0TDOI= 2022-09-12T19:44:21.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Pnll73F8ByXLXBHk3vqd7s0TDOI=', 'Content-Length': '0'} 2022-09-12T19:44:21.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff922547a9567719-00631f8c15-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.396 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:21.397 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:21.398 DEBUG:boto:path=/jdpiak-71/?prefix=e 2022-09-12T19:44:21.399 DEBUG:boto:auth_path=/jdpiak-71/?prefix=e 2022-09-12T19:44:21.400 DEBUG:boto:Method: GET 2022-09-12T19:44:21.400 DEBUG:boto:Path: /jdpiak-71/?prefix=e 2022-09-12T19:44:21.401 DEBUG:boto:Data: 2022-09-12T19:44:21.401 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.402 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:21.403 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.404 DEBUG:boto:Params: {} 2022-09-12T19:44:21.405 DEBUG:boto:Token: None 2022-09-12T19:44:21.406 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/ 2022-09-12T19:44:21.408 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q9+ZV3fhH3dUpjHNyO/LWQG/FSA= 2022-09-12T19:44:21.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q9+ZV3fhH3dUpjHNyO/LWQG/FSA=', 'Content-Length': '0'} 2022-09-12T19:44:21.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0e7effc72dca933-00631f8c15-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.415 DEBUG:boto:b'jdpiak-71e1000falsee2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:21.416 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:21.417 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:21.417 DEBUG:boto:path=/jdpiak-71/?prefix=e 2022-09-12T19:44:21.418 DEBUG:boto:auth_path=/jdpiak-71/?prefix=e 2022-09-12T19:44:21.419 DEBUG:boto:Method: GET 2022-09-12T19:44:21.420 DEBUG:boto:Path: /jdpiak-71/?prefix=e 2022-09-12T19:44:21.421 DEBUG:boto:Data: 2022-09-12T19:44:21.421 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.422 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:21.422 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.423 DEBUG:boto:Params: {} 2022-09-12T19:44:21.424 DEBUG:boto:Token: None 2022-09-12T19:44:21.424 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/ 2022-09-12T19:44:21.426 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q9+ZV3fhH3dUpjHNyO/LWQG/FSA= 2022-09-12T19:44:21.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q9+ZV3fhH3dUpjHNyO/LWQG/FSA=', 'Content-Length': '0'} 2022-09-12T19:44:21.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c963e1d6537328eb-00631f8c15-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.431 DEBUG:boto:b'jdpiak-71e1000falsee2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:21.432 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:21.432 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:21.433 DEBUG:boto:Method: HEAD 2022-09-12T19:44:21.433 DEBUG:boto:Path: /jdpiak-71/e 2022-09-12T19:44:21.434 DEBUG:boto:Data: 2022-09-12T19:44:21.434 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.435 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:21.435 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.436 DEBUG:boto:Params: {} 2022-09-12T19:44:21.437 DEBUG:boto:Token: None 2022-09-12T19:44:21.438 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/e 2022-09-12T19:44:21.439 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ERbI07qzoieChHDJnA0F/DIjits= 2022-09-12T19:44:21.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ERbI07qzoieChHDJnA0F/DIjits=', 'Content-Length': '0'} 2022-09-12T19:44:21.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fec53a5d11e48efe-00631f8c15-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.443 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:21.443 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:21.444 DEBUG:boto:Method: HEAD 2022-09-12T19:44:21.444 DEBUG:boto:Path: /jdpiak-71/e 2022-09-12T19:44:21.445 DEBUG:boto:Data: 2022-09-12T19:44:21.445 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.446 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:21.447 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.447 DEBUG:boto:Params: {} 2022-09-12T19:44:21.448 DEBUG:boto:Token: None 2022-09-12T19:44:21.449 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/e 2022-09-12T19:44:21.450 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ERbI07qzoieChHDJnA0F/DIjits= 2022-09-12T19:44:21.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ERbI07qzoieChHDJnA0F/DIjits=', 'Content-Length': '0'} 2022-09-12T19:44:21.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011b9f23046af275a-00631f8c15-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:44:21.454 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:21.455 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:21.456 DEBUG:boto:Method: GET 2022-09-12T19:44:21.456 DEBUG:boto:Path: /jdpiak-71/e 2022-09-12T19:44:21.457 DEBUG:boto:Data: 2022-09-12T19:44:21.458 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.459 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:21.461 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.463 DEBUG:boto:Params: {} 2022-09-12T19:44:21.464 DEBUG:boto:Token: None 2022-09-12T19:44:21.465 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/e 2022-09-12T19:44:21.467 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+dArjRfZQXM5eUkVa10a5G8NVFI= 2022-09-12T19:44:21.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+dArjRfZQXM5eUkVa10a5G8NVFI=', 'Content-Length': '0'} 2022-09-12T19:44:21.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b704851c376ed8a8-00631f8c15-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.515 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:21.517 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:21.519 DEBUG:boto:Method: GET 2022-09-12T19:44:21.520 DEBUG:boto:Path: /jdpiak-71/e 2022-09-12T19:44:21.521 DEBUG:boto:Data: 2022-09-12T19:44:21.522 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.523 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:21.524 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.525 DEBUG:boto:Params: {} 2022-09-12T19:44:21.526 DEBUG:boto:Token: None 2022-09-12T19:44:21.528 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/e 2022-09-12T19:44:21.530 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+dArjRfZQXM5eUkVa10a5G8NVFI= 2022-09-12T19:44:21.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+dArjRfZQXM5eUkVa10a5G8NVFI=', 'Content-Length': '0'} 2022-09-12T19:44:21.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1ab892ce2cb1590-00631f8c15-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.552 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:21.553 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:21.553 DEBUG:boto:path=/jdpiak-71/?prefix=e 2022-09-12T19:44:21.566 DEBUG:boto:auth_path=/jdpiak-71/?prefix=e 2022-09-12T19:44:21.567 DEBUG:boto:Method: GET 2022-09-12T19:44:21.567 DEBUG:boto:Path: /jdpiak-71/?prefix=e 2022-09-12T19:44:21.568 DEBUG:boto:Data: 2022-09-12T19:44:21.569 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.569 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:21.570 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.571 DEBUG:boto:Params: {} 2022-09-12T19:44:21.572 DEBUG:boto:Token: None 2022-09-12T19:44:21.582 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/ 2022-09-12T19:44:21.583 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q9+ZV3fhH3dUpjHNyO/LWQG/FSA= 2022-09-12T19:44:21.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q9+ZV3fhH3dUpjHNyO/LWQG/FSA=', 'Content-Length': '0'} 2022-09-12T19:44:21.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e96b84d17e266ef-00631f8c15-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.597 DEBUG:boto:b'jdpiak-71e1000falsee2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:21.598 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:21.599 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:21.600 DEBUG:boto:path=/jdpiak-71/?prefix=e 2022-09-12T19:44:21.600 DEBUG:boto:auth_path=/jdpiak-71/?prefix=e 2022-09-12T19:44:21.601 DEBUG:boto:Method: GET 2022-09-12T19:44:21.601 DEBUG:boto:Path: /jdpiak-71/?prefix=e 2022-09-12T19:44:21.602 DEBUG:boto:Data: 2022-09-12T19:44:21.603 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.603 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:21.604 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.604 DEBUG:boto:Params: {} 2022-09-12T19:44:21.605 DEBUG:boto:Token: None 2022-09-12T19:44:21.606 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/ 2022-09-12T19:44:21.606 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q9+ZV3fhH3dUpjHNyO/LWQG/FSA= 2022-09-12T19:44:21.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q9+ZV3fhH3dUpjHNyO/LWQG/FSA=', 'Content-Length': '0'} 2022-09-12T19:44:21.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d8c5b7a7d62d2ed-00631f8c15-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.611 DEBUG:boto:b'jdpiak-71e1000falsee2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:21.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:44:21.612 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:21.613 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:21.618 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:21.619 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:21.619 DEBUG:boto:Method: GET 2022-09-12T19:44:21.620 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:21.621 DEBUG:boto:Data: 2022-09-12T19:44:21.621 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.622 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:21.623 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.623 DEBUG:boto:Params: {} 2022-09-12T19:44:21.624 DEBUG:boto:Token: None 2022-09-12T19:44:21.625 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:21.625 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HHYB5dAIPJg/zv7X3EJS1XpQVCU= 2022-09-12T19:44:21.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HHYB5dAIPJg/zv7X3EJS1XpQVCU=', 'Content-Length': '0'} 2022-09-12T19:44:21.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c26a8076b29cdc9-00631f8c15-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.647 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:21.648 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:21.649 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:21.649 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:21.650 DEBUG:boto:Method: GET 2022-09-12T19:44:21.651 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:21.651 DEBUG:boto:Data: 2022-09-12T19:44:21.652 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.652 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:21.653 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.658 DEBUG:boto:Params: {} 2022-09-12T19:44:21.659 DEBUG:boto:Token: None 2022-09-12T19:44:21.660 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:21.661 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HHYB5dAIPJg/zv7X3EJS1XpQVCU= 2022-09-12T19:44:21.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HHYB5dAIPJg/zv7X3EJS1XpQVCU=', 'Content-Length': '0'} 2022-09-12T19:44:21.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026a00e6991a4d1e9-00631f8c15-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.669 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:21.670 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:21.671 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:21.671 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:21.672 DEBUG:boto:Method: HEAD 2022-09-12T19:44:21.672 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:21.673 DEBUG:boto:Data: 2022-09-12T19:44:21.673 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.674 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:21.674 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.675 DEBUG:boto:Params: {} 2022-09-12T19:44:21.675 DEBUG:boto:Token: None 2022-09-12T19:44:21.676 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:21.677 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xes6MR+9ISfWi1j6PGNNa/ohBdU= 2022-09-12T19:44:21.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xes6MR+9ISfWi1j6PGNNa/ohBdU=', 'Content-Length': '0'} 2022-09-12T19:44:21.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a9ddc207947a670-00631f8c15-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.695 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:21.696 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:21.697 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:21.698 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:21.699 DEBUG:boto:Method: HEAD 2022-09-12T19:44:21.700 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:21.701 DEBUG:boto:Data: 2022-09-12T19:44:21.701 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.704 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:21.705 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.706 DEBUG:boto:Params: {} 2022-09-12T19:44:21.707 DEBUG:boto:Token: None 2022-09-12T19:44:21.708 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:21.709 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xes6MR+9ISfWi1j6PGNNa/ohBdU= 2022-09-12T19:44:21.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xes6MR+9ISfWi1j6PGNNa/ohBdU=', 'Content-Length': '0'} 2022-09-12T19:44:21.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0dd20bfc7b180b3-00631f8c15-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:44:21.723 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:21.724 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:21.724 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:21.725 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:21.725 DEBUG:boto:Method: GET 2022-09-12T19:44:21.727 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:21.727 DEBUG:boto:Data: 2022-09-12T19:44:21.728 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.728 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:21.729 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.729 DEBUG:boto:Params: {} 2022-09-12T19:44:21.730 DEBUG:boto:Token: None 2022-09-12T19:44:21.731 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:21.732 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HHYB5dAIPJg/zv7X3EJS1XpQVCU= 2022-09-12T19:44:21.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HHYB5dAIPJg/zv7X3EJS1XpQVCU=', 'Content-Length': '0'} 2022-09-12T19:44:21.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032211611ebdc4deb-00631f8c15-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.750 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:21.751 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:21.752 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:21.753 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:21.755 DEBUG:boto:Method: GET 2022-09-12T19:44:21.767 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:21.768 DEBUG:boto:Data: 2022-09-12T19:44:21.769 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.770 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:21.771 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.772 DEBUG:boto:Params: {} 2022-09-12T19:44:21.774 DEBUG:boto:Token: None 2022-09-12T19:44:21.775 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:21.777 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HHYB5dAIPJg/zv7X3EJS1XpQVCU= 2022-09-12T19:44:21.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HHYB5dAIPJg/zv7X3EJS1XpQVCU=', 'Content-Length': '0'} 2022-09-12T19:44:21.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c086cda7a2377ee3-00631f8c15-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.783 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:21.784 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:21.786 DEBUG:boto:path=/jdpiak-71/?prefix=f 2022-09-12T19:44:21.787 DEBUG:boto:auth_path=/jdpiak-71/?prefix=f 2022-09-12T19:44:21.788 DEBUG:boto:Method: GET 2022-09-12T19:44:21.789 DEBUG:boto:Path: /jdpiak-71/?prefix=f 2022-09-12T19:44:21.790 DEBUG:boto:Data: 2022-09-12T19:44:21.791 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.793 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:21.794 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.795 DEBUG:boto:Params: {} 2022-09-12T19:44:21.796 DEBUG:boto:Token: None 2022-09-12T19:44:21.798 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/ 2022-09-12T19:44:21.799 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q9+ZV3fhH3dUpjHNyO/LWQG/FSA= 2022-09-12T19:44:21.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q9+ZV3fhH3dUpjHNyO/LWQG/FSA=', 'Content-Length': '0'} 2022-09-12T19:44:21.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e465dd2ba5a2ed7c-00631f8c15-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.807 DEBUG:boto:b'jdpiak-71f1000falsef2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:21.812 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:21.813 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:21.815 DEBUG:boto:path=/jdpiak-71/?prefix=f 2022-09-12T19:44:21.815 DEBUG:boto:auth_path=/jdpiak-71/?prefix=f 2022-09-12T19:44:21.816 DEBUG:boto:Method: GET 2022-09-12T19:44:21.817 DEBUG:boto:Path: /jdpiak-71/?prefix=f 2022-09-12T19:44:21.818 DEBUG:boto:Data: 2022-09-12T19:44:21.819 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.820 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:21.821 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.822 DEBUG:boto:Params: {} 2022-09-12T19:44:21.823 DEBUG:boto:Token: None 2022-09-12T19:44:21.823 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/ 2022-09-12T19:44:21.824 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q9+ZV3fhH3dUpjHNyO/LWQG/FSA= 2022-09-12T19:44:21.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q9+ZV3fhH3dUpjHNyO/LWQG/FSA=', 'Content-Length': '0'} 2022-09-12T19:44:21.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ebc905d6062fb4c-00631f8c15-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.829 DEBUG:boto:b'jdpiak-71f1000falsef2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:21.830 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:21.831 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:21.832 DEBUG:boto:Method: HEAD 2022-09-12T19:44:21.833 DEBUG:boto:Path: /jdpiak-71/f 2022-09-12T19:44:21.834 DEBUG:boto:Data: 2022-09-12T19:44:21.835 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.835 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:21.836 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.837 DEBUG:boto:Params: {} 2022-09-12T19:44:21.838 DEBUG:boto:Token: None 2022-09-12T19:44:21.839 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/f 2022-09-12T19:44:21.840 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rYKvakbrp61Da0YTjdtcGDC7ycI= 2022-09-12T19:44:21.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rYKvakbrp61Da0YTjdtcGDC7ycI=', 'Content-Length': '0'} 2022-09-12T19:44:21.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e15f29eb57f9183c-00631f8c15-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.846 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:21.847 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:21.848 DEBUG:boto:Method: HEAD 2022-09-12T19:44:21.849 DEBUG:boto:Path: /jdpiak-71/f 2022-09-12T19:44:21.849 DEBUG:boto:Data: 2022-09-12T19:44:21.850 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.851 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:21.852 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.853 DEBUG:boto:Params: {} 2022-09-12T19:44:21.854 DEBUG:boto:Token: None 2022-09-12T19:44:21.862 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/f 2022-09-12T19:44:21.864 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rYKvakbrp61Da0YTjdtcGDC7ycI= 2022-09-12T19:44:21.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rYKvakbrp61Da0YTjdtcGDC7ycI=', 'Content-Length': '0'} 2022-09-12T19:44:21.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000023320754c6384b9-00631f8c15-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:44:21.871 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:21.872 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:21.873 DEBUG:boto:Method: GET 2022-09-12T19:44:21.874 DEBUG:boto:Path: /jdpiak-71/f 2022-09-12T19:44:21.874 DEBUG:boto:Data: 2022-09-12T19:44:21.875 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.877 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:21.878 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.879 DEBUG:boto:Params: {} 2022-09-12T19:44:21.880 DEBUG:boto:Token: None 2022-09-12T19:44:21.881 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/f 2022-09-12T19:44:21.882 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5DMfoJUPrP5ICHG+cEWb9CnNKpM= 2022-09-12T19:44:21.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5DMfoJUPrP5ICHG+cEWb9CnNKpM=', 'Content-Length': '0'} 2022-09-12T19:44:21.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002247588feaf1ae6f-00631f8c15-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.887 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:21.887 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:21.888 DEBUG:boto:Method: GET 2022-09-12T19:44:21.889 DEBUG:boto:Path: /jdpiak-71/f 2022-09-12T19:44:21.889 DEBUG:boto:Data: 2022-09-12T19:44:21.890 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.891 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:21.891 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.892 DEBUG:boto:Params: {} 2022-09-12T19:44:21.892 DEBUG:boto:Token: None 2022-09-12T19:44:21.893 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/f 2022-09-12T19:44:21.894 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5DMfoJUPrP5ICHG+cEWb9CnNKpM= 2022-09-12T19:44:21.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5DMfoJUPrP5ICHG+cEWb9CnNKpM=', 'Content-Length': '0'} 2022-09-12T19:44:21.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000839f4b188f7b9122-00631f8c15-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.900 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:21.901 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:21.902 DEBUG:boto:path=/jdpiak-71/?prefix=f 2022-09-12T19:44:21.903 DEBUG:boto:auth_path=/jdpiak-71/?prefix=f 2022-09-12T19:44:21.904 DEBUG:boto:Method: GET 2022-09-12T19:44:21.905 DEBUG:boto:Path: /jdpiak-71/?prefix=f 2022-09-12T19:44:21.906 DEBUG:boto:Data: 2022-09-12T19:44:21.907 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.908 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:21.909 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.910 DEBUG:boto:Params: {} 2022-09-12T19:44:21.911 DEBUG:boto:Token: None 2022-09-12T19:44:21.912 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/ 2022-09-12T19:44:21.913 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q9+ZV3fhH3dUpjHNyO/LWQG/FSA= 2022-09-12T19:44:21.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q9+ZV3fhH3dUpjHNyO/LWQG/FSA=', 'Content-Length': '0'} 2022-09-12T19:44:21.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009c594896e1f97eb-00631f8c15-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.919 DEBUG:boto:b'jdpiak-71f1000falsef2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:21.921 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:21.922 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:21.922 DEBUG:boto:path=/jdpiak-71/?prefix=f 2022-09-12T19:44:21.923 DEBUG:boto:auth_path=/jdpiak-71/?prefix=f 2022-09-12T19:44:21.924 DEBUG:boto:Method: GET 2022-09-12T19:44:21.925 DEBUG:boto:Path: /jdpiak-71/?prefix=f 2022-09-12T19:44:21.925 DEBUG:boto:Data: 2022-09-12T19:44:21.926 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.926 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:21.927 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.927 DEBUG:boto:Params: {} 2022-09-12T19:44:21.928 DEBUG:boto:Token: None 2022-09-12T19:44:21.928 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/ 2022-09-12T19:44:21.929 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q9+ZV3fhH3dUpjHNyO/LWQG/FSA= 2022-09-12T19:44:21.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q9+ZV3fhH3dUpjHNyO/LWQG/FSA=', 'Content-Length': '0'} 2022-09-12T19:44:21.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dafbd607ab55676c-00631f8c15-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.939 DEBUG:boto:b'jdpiak-71f1000falsef2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:21.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:44:21.942 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:21.943 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:21.944 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:21.945 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:21.946 DEBUG:boto:Method: GET 2022-09-12T19:44:21.947 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:21.948 DEBUG:boto:Data: 2022-09-12T19:44:21.949 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.950 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:21.951 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.952 DEBUG:boto:Params: {} 2022-09-12T19:44:21.953 DEBUG:boto:Token: None 2022-09-12T19:44:21.954 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:21.955 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AM0TNbIe1jqRRcfWG8g9P9FlQ9g= 2022-09-12T19:44:21.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AM0TNbIe1jqRRcfWG8g9P9FlQ9g=', 'Content-Length': '0'} 2022-09-12T19:44:21.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019179030cf991c8d-00631f8c15-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.961 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:21.962 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:21.963 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:21.964 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:21.965 DEBUG:boto:Method: GET 2022-09-12T19:44:21.966 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:21.967 DEBUG:boto:Data: 2022-09-12T19:44:21.968 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.969 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:21.970 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.971 DEBUG:boto:Params: {} 2022-09-12T19:44:21.972 DEBUG:boto:Token: None 2022-09-12T19:44:21.973 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:21.974 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AM0TNbIe1jqRRcfWG8g9P9FlQ9g= 2022-09-12T19:44:21.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AM0TNbIe1jqRRcfWG8g9P9FlQ9g=', 'Content-Length': '0'} 2022-09-12T19:44:21.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000734eaa5515e87a16-00631f8c15-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:21.980 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:21.981 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:21.982 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:21.983 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:21.985 DEBUG:boto:Method: HEAD 2022-09-12T19:44:21.987 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:21.990 DEBUG:boto:Data: 2022-09-12T19:44:21.991 DEBUG:boto:Headers: {} 2022-09-12T19:44:21.991 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:21.993 DEBUG:boto:Port: 8000 2022-09-12T19:44:21.994 DEBUG:boto:Params: {} 2022-09-12T19:44:21.997 DEBUG:boto:Token: None 2022-09-12T19:44:21.999 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:21 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:22.000 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QOp7OSn0gIQjp8MPsrk7phyqsSs= 2022-09-12T19:44:22.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QOp7OSn0gIQjp8MPsrk7phyqsSs=', 'Content-Length': '0'} 2022-09-12T19:44:22.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014a4c4e7c863f9e3-00631f8c16-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.005 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:22.006 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:22.006 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:22.007 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:22.007 DEBUG:boto:Method: HEAD 2022-09-12T19:44:22.008 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:22.008 DEBUG:boto:Data: 2022-09-12T19:44:22.009 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.009 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:22.010 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.010 DEBUG:boto:Params: {} 2022-09-12T19:44:22.011 DEBUG:boto:Token: None 2022-09-12T19:44:22.011 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:22.012 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:adKEz6M0iq9dHjr98bRlofc9mms= 2022-09-12T19:44:22.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:adKEz6M0iq9dHjr98bRlofc9mms=', 'Content-Length': '0'} 2022-09-12T19:44:22.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f119dcce6d7b48a2-00631f8c16-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:44:22.017 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:22.017 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:22.018 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:22.018 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:22.019 DEBUG:boto:Method: GET 2022-09-12T19:44:22.019 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:22.020 DEBUG:boto:Data: 2022-09-12T19:44:22.021 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.021 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:22.022 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.022 DEBUG:boto:Params: {} 2022-09-12T19:44:22.023 DEBUG:boto:Token: None 2022-09-12T19:44:22.024 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:22.025 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R9PI7/0b9wEQAN0kmvpRN95Yzn8= 2022-09-12T19:44:22.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R9PI7/0b9wEQAN0kmvpRN95Yzn8=', 'Content-Length': '0'} 2022-09-12T19:44:22.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000713890002c0b5c60-00631f8c16-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.028 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:22.029 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:22.029 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:22.030 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:22.030 DEBUG:boto:Method: GET 2022-09-12T19:44:22.031 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:22.031 DEBUG:boto:Data: 2022-09-12T19:44:22.032 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.032 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:22.033 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.033 DEBUG:boto:Params: {} 2022-09-12T19:44:22.034 DEBUG:boto:Token: None 2022-09-12T19:44:22.035 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:22.035 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R9PI7/0b9wEQAN0kmvpRN95Yzn8= 2022-09-12T19:44:22.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R9PI7/0b9wEQAN0kmvpRN95Yzn8=', 'Content-Length': '0'} 2022-09-12T19:44:22.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c352cdf7d5607bf-00631f8c16-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.039 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:22.040 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:22.040 DEBUG:boto:path=/jdpiak-71/?prefix=g 2022-09-12T19:44:22.041 DEBUG:boto:auth_path=/jdpiak-71/?prefix=g 2022-09-12T19:44:22.041 DEBUG:boto:Method: GET 2022-09-12T19:44:22.042 DEBUG:boto:Path: /jdpiak-71/?prefix=g 2022-09-12T19:44:22.042 DEBUG:boto:Data: 2022-09-12T19:44:22.043 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.043 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:22.044 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.045 DEBUG:boto:Params: {} 2022-09-12T19:44:22.045 DEBUG:boto:Token: None 2022-09-12T19:44:22.046 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/ 2022-09-12T19:44:22.047 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:txDIz3Z9DmfEzM8qY0P9JibO+v4= 2022-09-12T19:44:22.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:txDIz3Z9DmfEzM8qY0P9JibO+v4=', 'Content-Length': '0'} 2022-09-12T19:44:22.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000698bf6b2a1b45b9a-00631f8c16-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.051 DEBUG:boto:b'jdpiak-71g1000falseg2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:22.052 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:22.053 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:22.053 DEBUG:boto:path=/jdpiak-71/?prefix=g 2022-09-12T19:44:22.054 DEBUG:boto:auth_path=/jdpiak-71/?prefix=g 2022-09-12T19:44:22.055 DEBUG:boto:Method: GET 2022-09-12T19:44:22.055 DEBUG:boto:Path: /jdpiak-71/?prefix=g 2022-09-12T19:44:22.056 DEBUG:boto:Data: 2022-09-12T19:44:22.056 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.057 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:22.057 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.058 DEBUG:boto:Params: {} 2022-09-12T19:44:22.058 DEBUG:boto:Token: None 2022-09-12T19:44:22.059 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/ 2022-09-12T19:44:22.060 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:txDIz3Z9DmfEzM8qY0P9JibO+v4= 2022-09-12T19:44:22.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:txDIz3Z9DmfEzM8qY0P9JibO+v4=', 'Content-Length': '0'} 2022-09-12T19:44:22.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea9ccaf88d33981e-00631f8c16-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.064 DEBUG:boto:b'jdpiak-71g1000falseg2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:22.065 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:22.066 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:22.067 DEBUG:boto:Method: HEAD 2022-09-12T19:44:22.068 DEBUG:boto:Path: /jdpiak-71/g 2022-09-12T19:44:22.069 DEBUG:boto:Data: 2022-09-12T19:44:22.069 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.070 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:22.071 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.072 DEBUG:boto:Params: {} 2022-09-12T19:44:22.073 DEBUG:boto:Token: None 2022-09-12T19:44:22.074 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/g 2022-09-12T19:44:22.075 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OZWfoCZYExtuaUSs4XAb2TOiSnM= 2022-09-12T19:44:22.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OZWfoCZYExtuaUSs4XAb2TOiSnM=', 'Content-Length': '0'} 2022-09-12T19:44:22.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000957a0c33a9884640-00631f8c16-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.080 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:22.081 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:22.082 DEBUG:boto:Method: HEAD 2022-09-12T19:44:22.083 DEBUG:boto:Path: /jdpiak-71/g 2022-09-12T19:44:22.083 DEBUG:boto:Data: 2022-09-12T19:44:22.084 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.085 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:22.086 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.087 DEBUG:boto:Params: {} 2022-09-12T19:44:22.088 DEBUG:boto:Token: None 2022-09-12T19:44:22.089 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/g 2022-09-12T19:44:22.090 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OZWfoCZYExtuaUSs4XAb2TOiSnM= 2022-09-12T19:44:22.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OZWfoCZYExtuaUSs4XAb2TOiSnM=', 'Content-Length': '0'} 2022-09-12T19:44:22.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d7f11ae4cc930e9-00631f8c16-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:44:22.098 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:22.099 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:22.100 DEBUG:boto:Method: GET 2022-09-12T19:44:22.101 DEBUG:boto:Path: /jdpiak-71/g 2022-09-12T19:44:22.102 DEBUG:boto:Data: 2022-09-12T19:44:22.103 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.104 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:22.105 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.106 DEBUG:boto:Params: {} 2022-09-12T19:44:22.107 DEBUG:boto:Token: None 2022-09-12T19:44:22.108 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/g 2022-09-12T19:44:22.109 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XXQidop02KqT13Vr6TCaPPHjrsY= 2022-09-12T19:44:22.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XXQidop02KqT13Vr6TCaPPHjrsY=', 'Content-Length': '0'} 2022-09-12T19:44:22.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003db69491c4ce6acd-00631f8c16-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.159 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:22.160 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:22.161 DEBUG:boto:Method: GET 2022-09-12T19:44:22.163 DEBUG:boto:Path: /jdpiak-71/g 2022-09-12T19:44:22.164 DEBUG:boto:Data: 2022-09-12T19:44:22.165 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.166 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:22.167 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.168 DEBUG:boto:Params: {} 2022-09-12T19:44:22.169 DEBUG:boto:Token: None 2022-09-12T19:44:22.170 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/g 2022-09-12T19:44:22.171 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XXQidop02KqT13Vr6TCaPPHjrsY= 2022-09-12T19:44:22.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XXQidop02KqT13Vr6TCaPPHjrsY=', 'Content-Length': '0'} 2022-09-12T19:44:22.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0f88ee438fbb2cd-00631f8c16-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.176 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:22.177 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:22.177 DEBUG:boto:path=/jdpiak-71/?prefix=g 2022-09-12T19:44:22.178 DEBUG:boto:auth_path=/jdpiak-71/?prefix=g 2022-09-12T19:44:22.180 DEBUG:boto:Method: GET 2022-09-12T19:44:22.181 DEBUG:boto:Path: /jdpiak-71/?prefix=g 2022-09-12T19:44:22.181 DEBUG:boto:Data: 2022-09-12T19:44:22.182 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.183 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:22.184 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.185 DEBUG:boto:Params: {} 2022-09-12T19:44:22.187 DEBUG:boto:Token: None 2022-09-12T19:44:22.188 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/ 2022-09-12T19:44:22.189 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:txDIz3Z9DmfEzM8qY0P9JibO+v4= 2022-09-12T19:44:22.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:txDIz3Z9DmfEzM8qY0P9JibO+v4=', 'Content-Length': '0'} 2022-09-12T19:44:22.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf5b54720c9fd212-00631f8c16-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.193 DEBUG:boto:b'jdpiak-71g1000falseg2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:22.194 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:22.195 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:22.195 DEBUG:boto:path=/jdpiak-71/?prefix=g 2022-09-12T19:44:22.196 DEBUG:boto:auth_path=/jdpiak-71/?prefix=g 2022-09-12T19:44:22.196 DEBUG:boto:Method: GET 2022-09-12T19:44:22.197 DEBUG:boto:Path: /jdpiak-71/?prefix=g 2022-09-12T19:44:22.198 DEBUG:boto:Data: 2022-09-12T19:44:22.198 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.199 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:22.199 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.200 DEBUG:boto:Params: {} 2022-09-12T19:44:22.201 DEBUG:boto:Token: None 2022-09-12T19:44:22.201 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/ 2022-09-12T19:44:22.202 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:txDIz3Z9DmfEzM8qY0P9JibO+v4= 2022-09-12T19:44:22.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:txDIz3Z9DmfEzM8qY0P9JibO+v4=', 'Content-Length': '0'} 2022-09-12T19:44:22.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be158bbd78bf2d71-00631f8c16-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.206 DEBUG:boto:b'jdpiak-71g1000falseg2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:22.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:44:22.208 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:22.208 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:22.209 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:22.209 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:22.210 DEBUG:boto:Method: GET 2022-09-12T19:44:22.210 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:22.211 DEBUG:boto:Data: 2022-09-12T19:44:22.212 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.212 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:22.213 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.213 DEBUG:boto:Params: {} 2022-09-12T19:44:22.214 DEBUG:boto:Token: None 2022-09-12T19:44:22.215 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:22.215 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KLjjcRbdGhufPlQvNj2j/u5byI8= 2022-09-12T19:44:22.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KLjjcRbdGhufPlQvNj2j/u5byI8=', 'Content-Length': '0'} 2022-09-12T19:44:22.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fe64e4d12a6ba76-00631f8c16-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.219 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:22.220 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:22.220 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:22.221 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:22.222 DEBUG:boto:Method: GET 2022-09-12T19:44:22.223 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:22.223 DEBUG:boto:Data: 2022-09-12T19:44:22.224 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.224 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:22.225 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.226 DEBUG:boto:Params: {} 2022-09-12T19:44:22.226 DEBUG:boto:Token: None 2022-09-12T19:44:22.227 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:22.228 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KLjjcRbdGhufPlQvNj2j/u5byI8= 2022-09-12T19:44:22.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KLjjcRbdGhufPlQvNj2j/u5byI8=', 'Content-Length': '0'} 2022-09-12T19:44:22.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a355b2be913fdb4b-00631f8c16-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.232 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:22.233 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:22.233 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:22.234 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:22.234 DEBUG:boto:Method: HEAD 2022-09-12T19:44:22.235 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:22.236 DEBUG:boto:Data: 2022-09-12T19:44:22.236 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.237 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:22.237 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.238 DEBUG:boto:Params: {} 2022-09-12T19:44:22.238 DEBUG:boto:Token: None 2022-09-12T19:44:22.239 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:22.240 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:58kEciVBIyrIPVW8quq1VptPLek= 2022-09-12T19:44:22.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:58kEciVBIyrIPVW8quq1VptPLek=', 'Content-Length': '0'} 2022-09-12T19:44:22.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3c7f1e12895c9d1-00631f8c16-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.245 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:22.246 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:22.247 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:22.249 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:22.250 DEBUG:boto:Method: HEAD 2022-09-12T19:44:22.251 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:22.252 DEBUG:boto:Data: 2022-09-12T19:44:22.253 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.254 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:22.255 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.256 DEBUG:boto:Params: {} 2022-09-12T19:44:22.257 DEBUG:boto:Token: None 2022-09-12T19:44:22.258 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:22.260 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:58kEciVBIyrIPVW8quq1VptPLek= 2022-09-12T19:44:22.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:58kEciVBIyrIPVW8quq1VptPLek=', 'Content-Length': '0'} 2022-09-12T19:44:22.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f108caf52c7d0ed-00631f8c16-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:44:22.266 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:22.267 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:22.267 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:22.268 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:22.271 DEBUG:boto:Method: GET 2022-09-12T19:44:22.271 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:22.272 DEBUG:boto:Data: 2022-09-12T19:44:22.273 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.274 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:22.274 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.275 DEBUG:boto:Params: {} 2022-09-12T19:44:22.275 DEBUG:boto:Token: None 2022-09-12T19:44:22.276 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:22.277 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KLjjcRbdGhufPlQvNj2j/u5byI8= 2022-09-12T19:44:22.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KLjjcRbdGhufPlQvNj2j/u5byI8=', 'Content-Length': '0'} 2022-09-12T19:44:22.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000451d8e841a7a0ba5-00631f8c16-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.281 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:22.281 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:22.282 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:22.282 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:22.283 DEBUG:boto:Method: GET 2022-09-12T19:44:22.283 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:22.285 DEBUG:boto:Data: 2022-09-12T19:44:22.285 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.286 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:22.286 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.287 DEBUG:boto:Params: {} 2022-09-12T19:44:22.288 DEBUG:boto:Token: None 2022-09-12T19:44:22.288 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:22.289 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KLjjcRbdGhufPlQvNj2j/u5byI8= 2022-09-12T19:44:22.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KLjjcRbdGhufPlQvNj2j/u5byI8=', 'Content-Length': '0'} 2022-09-12T19:44:22.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4c1190c8fa6704d-00631f8c16-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.296 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:22.297 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:22.297 DEBUG:boto:path=/jdpiak-71/?prefix=h 2022-09-12T19:44:22.298 DEBUG:boto:auth_path=/jdpiak-71/?prefix=h 2022-09-12T19:44:22.298 DEBUG:boto:Method: GET 2022-09-12T19:44:22.299 DEBUG:boto:Path: /jdpiak-71/?prefix=h 2022-09-12T19:44:22.299 DEBUG:boto:Data: 2022-09-12T19:44:22.300 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.300 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:22.301 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.301 DEBUG:boto:Params: {} 2022-09-12T19:44:22.303 DEBUG:boto:Token: None 2022-09-12T19:44:22.304 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/ 2022-09-12T19:44:22.304 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:txDIz3Z9DmfEzM8qY0P9JibO+v4= 2022-09-12T19:44:22.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:txDIz3Z9DmfEzM8qY0P9JibO+v4=', 'Content-Length': '0'} 2022-09-12T19:44:22.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db7664ac85c0d0f6-00631f8c16-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.309 DEBUG:boto:b'jdpiak-71h1000falseh2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:22.310 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:22.311 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:22.311 DEBUG:boto:path=/jdpiak-71/?prefix=h 2022-09-12T19:44:22.313 DEBUG:boto:auth_path=/jdpiak-71/?prefix=h 2022-09-12T19:44:22.313 DEBUG:boto:Method: GET 2022-09-12T19:44:22.314 DEBUG:boto:Path: /jdpiak-71/?prefix=h 2022-09-12T19:44:22.315 DEBUG:boto:Data: 2022-09-12T19:44:22.316 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.316 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:22.317 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.317 DEBUG:boto:Params: {} 2022-09-12T19:44:22.318 DEBUG:boto:Token: None 2022-09-12T19:44:22.320 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/ 2022-09-12T19:44:22.321 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:txDIz3Z9DmfEzM8qY0P9JibO+v4= 2022-09-12T19:44:22.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:txDIz3Z9DmfEzM8qY0P9JibO+v4=', 'Content-Length': '0'} 2022-09-12T19:44:22.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003aada53dd1c13d68-00631f8c16-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.372 DEBUG:boto:b'jdpiak-71h1000falseh2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:22.374 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:22.376 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:22.378 DEBUG:boto:Method: HEAD 2022-09-12T19:44:22.378 DEBUG:boto:Path: /jdpiak-71/h 2022-09-12T19:44:22.379 DEBUG:boto:Data: 2022-09-12T19:44:22.379 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.380 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:22.381 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.381 DEBUG:boto:Params: {} 2022-09-12T19:44:22.382 DEBUG:boto:Token: None 2022-09-12T19:44:22.382 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/h 2022-09-12T19:44:22.383 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aH/rp/mDkNFyGcB5oOf6a4HLT78= 2022-09-12T19:44:22.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aH/rp/mDkNFyGcB5oOf6a4HLT78=', 'Content-Length': '0'} 2022-09-12T19:44:22.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0bfc2dbb827686f-00631f8c16-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.387 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:22.387 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:22.388 DEBUG:boto:Method: HEAD 2022-09-12T19:44:22.388 DEBUG:boto:Path: /jdpiak-71/h 2022-09-12T19:44:22.389 DEBUG:boto:Data: 2022-09-12T19:44:22.389 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.390 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:22.390 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.391 DEBUG:boto:Params: {} 2022-09-12T19:44:22.391 DEBUG:boto:Token: None 2022-09-12T19:44:22.392 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/h 2022-09-12T19:44:22.393 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aH/rp/mDkNFyGcB5oOf6a4HLT78= 2022-09-12T19:44:22.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aH/rp/mDkNFyGcB5oOf6a4HLT78=', 'Content-Length': '0'} 2022-09-12T19:44:22.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000886f6abf73d73158-00631f8c16-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:44:22.397 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:22.398 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:22.399 DEBUG:boto:Method: GET 2022-09-12T19:44:22.400 DEBUG:boto:Path: /jdpiak-71/h 2022-09-12T19:44:22.401 DEBUG:boto:Data: 2022-09-12T19:44:22.402 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.403 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:22.403 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.404 DEBUG:boto:Params: {} 2022-09-12T19:44:22.404 DEBUG:boto:Token: None 2022-09-12T19:44:22.405 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/h 2022-09-12T19:44:22.406 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lNxabEAmN46G9aURNdkn+6+wr7s= 2022-09-12T19:44:22.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lNxabEAmN46G9aURNdkn+6+wr7s=', 'Content-Length': '0'} 2022-09-12T19:44:22.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000407e8b4ce5ddb25c-00631f8c16-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.416 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:22.417 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:22.418 DEBUG:boto:Method: GET 2022-09-12T19:44:22.419 DEBUG:boto:Path: /jdpiak-71/h 2022-09-12T19:44:22.420 DEBUG:boto:Data: 2022-09-12T19:44:22.421 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.422 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:22.423 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.424 DEBUG:boto:Params: {} 2022-09-12T19:44:22.426 DEBUG:boto:Token: None 2022-09-12T19:44:22.428 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/h 2022-09-12T19:44:22.429 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lNxabEAmN46G9aURNdkn+6+wr7s= 2022-09-12T19:44:22.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lNxabEAmN46G9aURNdkn+6+wr7s=', 'Content-Length': '0'} 2022-09-12T19:44:22.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6b2cbee4e0cc708-00631f8c16-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.436 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:22.437 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:22.438 DEBUG:boto:path=/jdpiak-71/?prefix=h 2022-09-12T19:44:22.439 DEBUG:boto:auth_path=/jdpiak-71/?prefix=h 2022-09-12T19:44:22.440 DEBUG:boto:Method: GET 2022-09-12T19:44:22.441 DEBUG:boto:Path: /jdpiak-71/?prefix=h 2022-09-12T19:44:22.442 DEBUG:boto:Data: 2022-09-12T19:44:22.443 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.444 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:22.445 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.446 DEBUG:boto:Params: {} 2022-09-12T19:44:22.447 DEBUG:boto:Token: None 2022-09-12T19:44:22.449 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/ 2022-09-12T19:44:22.450 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:txDIz3Z9DmfEzM8qY0P9JibO+v4= 2022-09-12T19:44:22.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:txDIz3Z9DmfEzM8qY0P9JibO+v4=', 'Content-Length': '0'} 2022-09-12T19:44:22.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6e481a46c68038d-00631f8c16-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.499 DEBUG:boto:b'jdpiak-71h1000falseh2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:22.501 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:22.502 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:22.503 DEBUG:boto:path=/jdpiak-71/?prefix=h 2022-09-12T19:44:22.504 DEBUG:boto:auth_path=/jdpiak-71/?prefix=h 2022-09-12T19:44:22.505 DEBUG:boto:Method: GET 2022-09-12T19:44:22.507 DEBUG:boto:Path: /jdpiak-71/?prefix=h 2022-09-12T19:44:22.508 DEBUG:boto:Data: 2022-09-12T19:44:22.509 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.510 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:22.511 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.512 DEBUG:boto:Params: {} 2022-09-12T19:44:22.513 DEBUG:boto:Token: None 2022-09-12T19:44:22.514 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/ 2022-09-12T19:44:22.515 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:txDIz3Z9DmfEzM8qY0P9JibO+v4= 2022-09-12T19:44:22.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:txDIz3Z9DmfEzM8qY0P9JibO+v4=', 'Content-Length': '0'} 2022-09-12T19:44:22.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e17b01611f63396-00631f8c16-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.534 DEBUG:boto:b'jdpiak-71h1000falseh2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:22.535 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-71 zones={test-zone2, test-zone1} 2022-09-12T19:44:22.536 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-71 zones={test-zone3, test-zone1} 2022-09-12T19:44:22.537 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:22.538 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:22.539 DEBUG:boto:Method: HEAD 2022-09-12T19:44:22.541 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:44:22.542 DEBUG:boto:Data: 2022-09-12T19:44:22.544 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.545 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:22.562 DEBUG:boto:Port: 8001 2022-09-12T19:44:22.564 DEBUG:boto:Params: {} 2022-09-12T19:44:22.570 DEBUG:boto:Token: None 2022-09-12T19:44:22.570 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/ 2022-09-12T19:44:22.571 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WgTfYzLAZF6LNhaVZsQAXJLX9pw= 2022-09-12T19:44:22.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WgTfYzLAZF6LNhaVZsQAXJLX9pw=', 'Content-Length': '0'} 2022-09-12T19:44:22.581 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002fd12801d9ef4aec-00631f8c16-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.583 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:22.584 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:22.586 DEBUG:boto:Method: HEAD 2022-09-12T19:44:22.587 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:44:22.589 DEBUG:boto:Data: 2022-09-12T19:44:22.590 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.591 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:22.593 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.599 DEBUG:boto:Params: {} 2022-09-12T19:44:22.600 DEBUG:boto:Token: None 2022-09-12T19:44:22.601 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/ 2022-09-12T19:44:22.602 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WgTfYzLAZF6LNhaVZsQAXJLX9pw= 2022-09-12T19:44:22.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WgTfYzLAZF6LNhaVZsQAXJLX9pw=', 'Content-Length': '0'} 2022-09-12T19:44:22.610 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042288579dc050c13-00631f8c16-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.613 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:44:22.615 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:22.617 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:22.619 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:22.620 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:22.622 DEBUG:boto:Method: GET 2022-09-12T19:44:22.623 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:22.624 DEBUG:boto:Data: 2022-09-12T19:44:22.625 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.625 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:22.626 DEBUG:boto:Port: 8001 2022-09-12T19:44:22.628 DEBUG:boto:Params: {} 2022-09-12T19:44:22.629 DEBUG:boto:Token: None 2022-09-12T19:44:22.631 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/?versions 2022-09-12T19:44:22.632 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ajnOvybxSxl6rSCSpc/4erN/Jl0= 2022-09-12T19:44:22.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ajnOvybxSxl6rSCSpc/4erN/Jl0=', 'Content-Length': '0'} 2022-09-12T19:44:22.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b1abd6084a48370-00631f8c16-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.646 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:22.651 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:44:22.652 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:44:22.653 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:44:22.654 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:44:22.656 DEBUG:tasks.rgw_multi.zone_rados:o=e 2022-09-12T19:44:22.657 DEBUG:tasks.rgw_multi.zone_rados:o=f 2022-09-12T19:44:22.659 DEBUG:tasks.rgw_multi.zone_rados:o=g 2022-09-12T19:44:22.660 DEBUG:tasks.rgw_multi.zone_rados:o=h 2022-09-12T19:44:22.664 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:44:22.664 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:22.665 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:22.666 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:22.666 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:22.667 DEBUG:boto:Method: GET 2022-09-12T19:44:22.667 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:22.669 DEBUG:boto:Data: 2022-09-12T19:44:22.670 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.672 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:22.674 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.676 DEBUG:boto:Params: {} 2022-09-12T19:44:22.680 DEBUG:boto:Token: None 2022-09-12T19:44:22.683 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/?versions 2022-09-12T19:44:22.686 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ajnOvybxSxl6rSCSpc/4erN/Jl0= 2022-09-12T19:44:22.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ajnOvybxSxl6rSCSpc/4erN/Jl0=', 'Content-Length': '0'} 2022-09-12T19:44:22.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099a4bf0d37658b8b-00631f8c16-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.694 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:22.697 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:44:22.698 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:44:22.699 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:44:22.700 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:44:22.701 DEBUG:tasks.rgw_multi.zone_rados:o=e 2022-09-12T19:44:22.702 DEBUG:tasks.rgw_multi.zone_rados:o=f 2022-09-12T19:44:22.703 DEBUG:tasks.rgw_multi.zone_rados:o=g 2022-09-12T19:44:22.704 DEBUG:tasks.rgw_multi.zone_rados:o=h 2022-09-12T19:44:22.705 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:22.710 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:22.711 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:22.711 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:22.712 DEBUG:boto:Method: GET 2022-09-12T19:44:22.712 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:22.713 DEBUG:boto:Data: 2022-09-12T19:44:22.713 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.714 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:22.715 DEBUG:boto:Port: 8001 2022-09-12T19:44:22.716 DEBUG:boto:Params: {} 2022-09-12T19:44:22.717 DEBUG:boto:Token: None 2022-09-12T19:44:22.717 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/?versions 2022-09-12T19:44:22.718 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ajnOvybxSxl6rSCSpc/4erN/Jl0= 2022-09-12T19:44:22.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ajnOvybxSxl6rSCSpc/4erN/Jl0=', 'Content-Length': '0'} 2022-09-12T19:44:22.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5f0d07ed251101e-00631f8c16-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.726 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:22.728 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:22.729 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:22.731 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:22.732 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:22.733 DEBUG:boto:Method: GET 2022-09-12T19:44:22.734 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:22.735 DEBUG:boto:Data: 2022-09-12T19:44:22.736 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.737 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:22.738 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.739 DEBUG:boto:Params: {} 2022-09-12T19:44:22.740 DEBUG:boto:Token: None 2022-09-12T19:44:22.740 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/?versions 2022-09-12T19:44:22.741 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ajnOvybxSxl6rSCSpc/4erN/Jl0= 2022-09-12T19:44:22.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ajnOvybxSxl6rSCSpc/4erN/Jl0=', 'Content-Length': '0'} 2022-09-12T19:44:22.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000858ab4e794ad15f0-00631f8c16-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.746 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:22.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:44:22.748 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:22.749 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:22.749 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:22.750 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:22.751 DEBUG:boto:Method: GET 2022-09-12T19:44:22.752 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:44:22.752 DEBUG:boto:Data: 2022-09-12T19:44:22.753 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.753 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:22.754 DEBUG:boto:Port: 8001 2022-09-12T19:44:22.755 DEBUG:boto:Params: {} 2022-09-12T19:44:22.755 DEBUG:boto:Token: None 2022-09-12T19:44:22.756 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:44:22.757 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cqm2xPyJu95yY4XM3Inwq4mc46o= 2022-09-12T19:44:22.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cqm2xPyJu95yY4XM3Inwq4mc46o=', 'Content-Length': '0'} 2022-09-12T19:44:22.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f799cf87c7ac656c-00631f8c16-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.766 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:22.767 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:22.768 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:22.769 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:22.770 DEBUG:boto:Method: GET 2022-09-12T19:44:22.771 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:44:22.772 DEBUG:boto:Data: 2022-09-12T19:44:22.773 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.774 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:22.774 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.775 DEBUG:boto:Params: {} 2022-09-12T19:44:22.775 DEBUG:boto:Token: None 2022-09-12T19:44:22.776 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:44:22.777 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cqm2xPyJu95yY4XM3Inwq4mc46o= 2022-09-12T19:44:22.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cqm2xPyJu95yY4XM3Inwq4mc46o=', 'Content-Length': '0'} 2022-09-12T19:44:22.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000809835cd8f8a0559-00631f8c16-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.786 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:22.787 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:22.788 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:22.789 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:22.790 DEBUG:boto:Method: HEAD 2022-09-12T19:44:22.793 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:44:22.794 DEBUG:boto:Data: 2022-09-12T19:44:22.795 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.796 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:22.797 DEBUG:boto:Port: 8001 2022-09-12T19:44:22.798 DEBUG:boto:Params: {} 2022-09-12T19:44:22.798 DEBUG:boto:Token: None 2022-09-12T19:44:22.799 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:44:22.800 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+ZCLYwCD3if3BwY7Ixt2i33fvm0= 2022-09-12T19:44:22.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+ZCLYwCD3if3BwY7Ixt2i33fvm0=', 'Content-Length': '0'} 2022-09-12T19:44:22.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8e84dc0b2f3bc2e-00631f8c16-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.804 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:22.805 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:22.806 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:22.807 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:22.808 DEBUG:boto:Method: HEAD 2022-09-12T19:44:22.808 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:44:22.809 DEBUG:boto:Data: 2022-09-12T19:44:22.809 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.810 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:22.810 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.811 DEBUG:boto:Params: {} 2022-09-12T19:44:22.812 DEBUG:boto:Token: None 2022-09-12T19:44:22.814 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:44:22.815 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+ZCLYwCD3if3BwY7Ixt2i33fvm0= 2022-09-12T19:44:22.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+ZCLYwCD3if3BwY7Ixt2i33fvm0=', 'Content-Length': '0'} 2022-09-12T19:44:22.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a26d49966041bb27-00631f8c16-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:44:22.824 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:22.825 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:22.826 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:22.827 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:22.828 DEBUG:boto:Method: GET 2022-09-12T19:44:22.830 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:44:22.831 DEBUG:boto:Data: 2022-09-12T19:44:22.832 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.833 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:22.834 DEBUG:boto:Port: 8001 2022-09-12T19:44:22.835 DEBUG:boto:Params: {} 2022-09-12T19:44:22.836 DEBUG:boto:Token: None 2022-09-12T19:44:22.838 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:44:22.839 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cqm2xPyJu95yY4XM3Inwq4mc46o= 2022-09-12T19:44:22.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cqm2xPyJu95yY4XM3Inwq4mc46o=', 'Content-Length': '0'} 2022-09-12T19:44:22.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be9f06f0fa882561-00631f8c16-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.846 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:22.847 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:22.848 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:22.849 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:22.850 DEBUG:boto:Method: GET 2022-09-12T19:44:22.852 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:44:22.853 DEBUG:boto:Data: 2022-09-12T19:44:22.856 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.858 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:22.858 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.859 DEBUG:boto:Params: {} 2022-09-12T19:44:22.859 DEBUG:boto:Token: None 2022-09-12T19:44:22.860 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:44:22.861 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cqm2xPyJu95yY4XM3Inwq4mc46o= 2022-09-12T19:44:22.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cqm2xPyJu95yY4XM3Inwq4mc46o=', 'Content-Length': '0'} 2022-09-12T19:44:22.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008da20a4adc1db780-00631f8c16-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.865 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:22.866 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:22.866 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:44:22.867 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:44:22.867 DEBUG:boto:Method: GET 2022-09-12T19:44:22.868 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:44:22.868 DEBUG:boto:Data: 2022-09-12T19:44:22.869 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.870 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:22.870 DEBUG:boto:Port: 8001 2022-09-12T19:44:22.871 DEBUG:boto:Params: {} 2022-09-12T19:44:22.872 DEBUG:boto:Token: None 2022-09-12T19:44:22.873 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/ 2022-09-12T19:44:22.873 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:txDIz3Z9DmfEzM8qY0P9JibO+v4= 2022-09-12T19:44:22.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:txDIz3Z9DmfEzM8qY0P9JibO+v4=', 'Content-Length': '0'} 2022-09-12T19:44:22.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4561fee533f5eac-00631f8c16-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.878 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:22.879 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:22.880 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:22.880 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:44:22.881 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:44:22.881 DEBUG:boto:Method: GET 2022-09-12T19:44:22.882 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:44:22.883 DEBUG:boto:Data: 2022-09-12T19:44:22.883 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.884 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:22.884 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.885 DEBUG:boto:Params: {} 2022-09-12T19:44:22.886 DEBUG:boto:Token: None 2022-09-12T19:44:22.886 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/ 2022-09-12T19:44:22.887 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:txDIz3Z9DmfEzM8qY0P9JibO+v4= 2022-09-12T19:44:22.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:txDIz3Z9DmfEzM8qY0P9JibO+v4=', 'Content-Length': '0'} 2022-09-12T19:44:22.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b84b107a2f243d23-00631f8c16-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.891 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:22.892 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:22.893 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:22.893 DEBUG:boto:Method: HEAD 2022-09-12T19:44:22.894 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:44:22.894 DEBUG:boto:Data: 2022-09-12T19:44:22.895 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.895 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:22.896 DEBUG:boto:Port: 8001 2022-09-12T19:44:22.897 DEBUG:boto:Params: {} 2022-09-12T19:44:22.897 DEBUG:boto:Token: None 2022-09-12T19:44:22.898 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/a 2022-09-12T19:44:22.899 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RecARlYt6uepQ9Yke8ArppBp/VY= 2022-09-12T19:44:22.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RecARlYt6uepQ9Yke8ArppBp/VY=', 'Content-Length': '0'} 2022-09-12T19:44:22.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000095921d450138e84-00631f8c16-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.902 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:22.903 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:22.903 DEBUG:boto:Method: HEAD 2022-09-12T19:44:22.904 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:44:22.905 DEBUG:boto:Data: 2022-09-12T19:44:22.905 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.906 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:22.906 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.907 DEBUG:boto:Params: {} 2022-09-12T19:44:22.908 DEBUG:boto:Token: None 2022-09-12T19:44:22.909 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/a 2022-09-12T19:44:22.910 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RecARlYt6uepQ9Yke8ArppBp/VY= 2022-09-12T19:44:22.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RecARlYt6uepQ9Yke8ArppBp/VY=', 'Content-Length': '0'} 2022-09-12T19:44:22.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3f73d9caa023e3c-00631f8c16-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:44:22.913 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:22.914 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:22.915 DEBUG:boto:Method: GET 2022-09-12T19:44:22.916 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:44:22.916 DEBUG:boto:Data: 2022-09-12T19:44:22.917 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.917 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:22.918 DEBUG:boto:Port: 8001 2022-09-12T19:44:22.919 DEBUG:boto:Params: {} 2022-09-12T19:44:22.919 DEBUG:boto:Token: None 2022-09-12T19:44:22.921 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/a 2022-09-12T19:44:22.922 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dFedKnZkMEpdicm6c7R5WKDZyKs= 2022-09-12T19:44:22.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dFedKnZkMEpdicm6c7R5WKDZyKs=', 'Content-Length': '0'} 2022-09-12T19:44:22.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047349287fc2f42f2-00631f8c16-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.926 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:22.927 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:22.927 DEBUG:boto:Method: GET 2022-09-12T19:44:22.928 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:44:22.928 DEBUG:boto:Data: 2022-09-12T19:44:22.929 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.929 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:22.930 DEBUG:boto:Port: 8000 2022-09-12T19:44:22.930 DEBUG:boto:Params: {} 2022-09-12T19:44:22.931 DEBUG:boto:Token: None 2022-09-12T19:44:22.932 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/a 2022-09-12T19:44:22.933 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dFedKnZkMEpdicm6c7R5WKDZyKs= 2022-09-12T19:44:22.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dFedKnZkMEpdicm6c7R5WKDZyKs=', 'Content-Length': '0'} 2022-09-12T19:44:22.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c140d5e60a39744-00631f8c16-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.978 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:22.980 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:22.981 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:44:22.982 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:44:22.983 DEBUG:boto:Method: GET 2022-09-12T19:44:22.984 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:44:22.986 DEBUG:boto:Data: 2022-09-12T19:44:22.987 DEBUG:boto:Headers: {} 2022-09-12T19:44:22.987 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:22.988 DEBUG:boto:Port: 8001 2022-09-12T19:44:22.988 DEBUG:boto:Params: {} 2022-09-12T19:44:22.989 DEBUG:boto:Token: None 2022-09-12T19:44:22.990 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:22 GMT /jdpiak-71/ 2022-09-12T19:44:22.991 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:txDIz3Z9DmfEzM8qY0P9JibO+v4= 2022-09-12T19:44:22.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:txDIz3Z9DmfEzM8qY0P9JibO+v4=', 'Content-Length': '0'} 2022-09-12T19:44:22.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020c53a68e5e7009d-00631f8c16-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:22.995 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:22.996 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:22.997 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:22.997 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:44:22.998 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:44:22.999 DEBUG:boto:Method: GET 2022-09-12T19:44:23.000 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:44:23.001 DEBUG:boto:Data: 2022-09-12T19:44:23.002 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.004 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:23.004 DEBUG:boto:Port: 8000 2022-09-12T19:44:23.005 DEBUG:boto:Params: {} 2022-09-12T19:44:23.006 DEBUG:boto:Token: None 2022-09-12T19:44:23.007 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/ 2022-09-12T19:44:23.007 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tJNykwaMYzGVMWYYIkwbemZSRyo= 2022-09-12T19:44:23.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tJNykwaMYzGVMWYYIkwbemZSRyo=', 'Content-Length': '0'} 2022-09-12T19:44:23.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007341bbc9741a2336-00631f8c17-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.012 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:23.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:44:23.013 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:23.014 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:23.015 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:23.015 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:23.016 DEBUG:boto:Method: GET 2022-09-12T19:44:23.016 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:44:23.017 DEBUG:boto:Data: 2022-09-12T19:44:23.017 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.018 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:23.019 DEBUG:boto:Port: 8001 2022-09-12T19:44:23.019 DEBUG:boto:Params: {} 2022-09-12T19:44:23.020 DEBUG:boto:Token: None 2022-09-12T19:44:23.021 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:44:23.021 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EjwgCri8LGXu40s6of87cQlMYdY= 2022-09-12T19:44:23.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EjwgCri8LGXu40s6of87cQlMYdY=', 'Content-Length': '0'} 2022-09-12T19:44:23.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000861448b32312f1d0-00631f8c17-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.066 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:23.068 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:23.069 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:23.070 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:23.071 DEBUG:boto:Method: GET 2022-09-12T19:44:23.072 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:44:23.073 DEBUG:boto:Data: 2022-09-12T19:44:23.074 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.075 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:23.076 DEBUG:boto:Port: 8000 2022-09-12T19:44:23.077 DEBUG:boto:Params: {} 2022-09-12T19:44:23.079 DEBUG:boto:Token: None 2022-09-12T19:44:23.080 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:44:23.082 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EjwgCri8LGXu40s6of87cQlMYdY= 2022-09-12T19:44:23.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EjwgCri8LGXu40s6of87cQlMYdY=', 'Content-Length': '0'} 2022-09-12T19:44:23.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035a4c4b25248ed6b-00631f8c17-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.087 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:23.088 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:23.089 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:23.089 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:23.090 DEBUG:boto:Method: HEAD 2022-09-12T19:44:23.090 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:44:23.091 DEBUG:boto:Data: 2022-09-12T19:44:23.092 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.093 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:23.094 DEBUG:boto:Port: 8001 2022-09-12T19:44:23.095 DEBUG:boto:Params: {} 2022-09-12T19:44:23.096 DEBUG:boto:Token: None 2022-09-12T19:44:23.098 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:44:23.099 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LG79ekup9YjDw+WRNXBNoyeTMmA= 2022-09-12T19:44:23.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LG79ekup9YjDw+WRNXBNoyeTMmA=', 'Content-Length': '0'} 2022-09-12T19:44:23.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000466209fa03731f0e-00631f8c17-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.105 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:23.106 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:23.107 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:23.108 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:23.109 DEBUG:boto:Method: HEAD 2022-09-12T19:44:23.111 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:44:23.112 DEBUG:boto:Data: 2022-09-12T19:44:23.115 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.115 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:23.116 DEBUG:boto:Port: 8000 2022-09-12T19:44:23.116 DEBUG:boto:Params: {} 2022-09-12T19:44:23.117 DEBUG:boto:Token: None 2022-09-12T19:44:23.118 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:44:23.119 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LG79ekup9YjDw+WRNXBNoyeTMmA= 2022-09-12T19:44:23.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LG79ekup9YjDw+WRNXBNoyeTMmA=', 'Content-Length': '0'} 2022-09-12T19:44:23.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080e93fdd1f1cdca7-00631f8c17-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:44:23.128 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:23.129 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:23.131 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:23.132 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:23.134 DEBUG:boto:Method: GET 2022-09-12T19:44:23.135 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:44:23.136 DEBUG:boto:Data: 2022-09-12T19:44:23.137 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.139 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:23.140 DEBUG:boto:Port: 8001 2022-09-12T19:44:23.141 DEBUG:boto:Params: {} 2022-09-12T19:44:23.142 DEBUG:boto:Token: None 2022-09-12T19:44:23.143 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:44:23.144 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EjwgCri8LGXu40s6of87cQlMYdY= 2022-09-12T19:44:23.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EjwgCri8LGXu40s6of87cQlMYdY=', 'Content-Length': '0'} 2022-09-12T19:44:23.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008394640df81a688e-00631f8c17-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.150 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:23.151 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:23.151 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:23.152 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:23.153 DEBUG:boto:Method: GET 2022-09-12T19:44:23.153 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:44:23.154 DEBUG:boto:Data: 2022-09-12T19:44:23.155 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.155 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:23.156 DEBUG:boto:Port: 8000 2022-09-12T19:44:23.157 DEBUG:boto:Params: {} 2022-09-12T19:44:23.157 DEBUG:boto:Token: None 2022-09-12T19:44:23.158 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:44:23.159 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EjwgCri8LGXu40s6of87cQlMYdY= 2022-09-12T19:44:23.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EjwgCri8LGXu40s6of87cQlMYdY=', 'Content-Length': '0'} 2022-09-12T19:44:23.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7fa0e340390771d-00631f8c17-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.221 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:23.222 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:23.223 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:44:23.223 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:44:23.224 DEBUG:boto:Method: GET 2022-09-12T19:44:23.225 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:44:23.226 DEBUG:boto:Data: 2022-09-12T19:44:23.227 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.228 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:23.229 DEBUG:boto:Port: 8001 2022-09-12T19:44:23.230 DEBUG:boto:Params: {} 2022-09-12T19:44:23.231 DEBUG:boto:Token: None 2022-09-12T19:44:23.232 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/ 2022-09-12T19:44:23.233 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tJNykwaMYzGVMWYYIkwbemZSRyo= 2022-09-12T19:44:23.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tJNykwaMYzGVMWYYIkwbemZSRyo=', 'Content-Length': '0'} 2022-09-12T19:44:23.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e497b998b7d86c31-00631f8c17-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.240 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:23.241 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:23.242 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:23.242 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:44:23.243 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:44:23.243 DEBUG:boto:Method: GET 2022-09-12T19:44:23.244 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:44:23.245 DEBUG:boto:Data: 2022-09-12T19:44:23.246 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.246 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:23.247 DEBUG:boto:Port: 8000 2022-09-12T19:44:23.248 DEBUG:boto:Params: {} 2022-09-12T19:44:23.249 DEBUG:boto:Token: None 2022-09-12T19:44:23.249 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/ 2022-09-12T19:44:23.250 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tJNykwaMYzGVMWYYIkwbemZSRyo= 2022-09-12T19:44:23.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tJNykwaMYzGVMWYYIkwbemZSRyo=', 'Content-Length': '0'} 2022-09-12T19:44:23.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b88a8333b9305383-00631f8c17-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.254 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:23.256 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:23.257 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:23.259 DEBUG:boto:Method: HEAD 2022-09-12T19:44:23.260 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:44:23.261 DEBUG:boto:Data: 2022-09-12T19:44:23.262 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.263 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:23.264 DEBUG:boto:Port: 8001 2022-09-12T19:44:23.264 DEBUG:boto:Params: {} 2022-09-12T19:44:23.265 DEBUG:boto:Token: None 2022-09-12T19:44:23.266 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/b 2022-09-12T19:44:23.267 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wVItWTUx4fgD9EVjCTBqRA4CKhU= 2022-09-12T19:44:23.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wVItWTUx4fgD9EVjCTBqRA4CKhU=', 'Content-Length': '0'} 2022-09-12T19:44:23.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c96cdf18efacd57-00631f8c17-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.272 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:23.273 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:23.274 DEBUG:boto:Method: HEAD 2022-09-12T19:44:23.275 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:44:23.276 DEBUG:boto:Data: 2022-09-12T19:44:23.277 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.278 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:23.279 DEBUG:boto:Port: 8000 2022-09-12T19:44:23.280 DEBUG:boto:Params: {} 2022-09-12T19:44:23.281 DEBUG:boto:Token: None 2022-09-12T19:44:23.282 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/b 2022-09-12T19:44:23.283 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wVItWTUx4fgD9EVjCTBqRA4CKhU= 2022-09-12T19:44:23.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wVItWTUx4fgD9EVjCTBqRA4CKhU=', 'Content-Length': '0'} 2022-09-12T19:44:23.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035a539060df1480e-00631f8c17-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:44:23.289 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:23.290 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:23.291 DEBUG:boto:Method: GET 2022-09-12T19:44:23.292 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:44:23.293 DEBUG:boto:Data: 2022-09-12T19:44:23.294 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.295 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:23.296 DEBUG:boto:Port: 8001 2022-09-12T19:44:23.297 DEBUG:boto:Params: {} 2022-09-12T19:44:23.298 DEBUG:boto:Token: None 2022-09-12T19:44:23.300 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/b 2022-09-12T19:44:23.301 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gfEdMJSOSWvkvY3BZWIZQ8fVjIY= 2022-09-12T19:44:23.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gfEdMJSOSWvkvY3BZWIZQ8fVjIY=', 'Content-Length': '0'} 2022-09-12T19:44:23.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000531962ac02d5e58d-00631f8c17-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.306 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:23.307 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:23.307 DEBUG:boto:Method: GET 2022-09-12T19:44:23.308 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:44:23.308 DEBUG:boto:Data: 2022-09-12T19:44:23.309 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.309 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:23.310 DEBUG:boto:Port: 8000 2022-09-12T19:44:23.311 DEBUG:boto:Params: {} 2022-09-12T19:44:23.312 DEBUG:boto:Token: None 2022-09-12T19:44:23.312 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/b 2022-09-12T19:44:23.313 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gfEdMJSOSWvkvY3BZWIZQ8fVjIY= 2022-09-12T19:44:23.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gfEdMJSOSWvkvY3BZWIZQ8fVjIY=', 'Content-Length': '0'} 2022-09-12T19:44:23.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070d39bca65e56402-00631f8c17-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.320 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:23.321 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:23.323 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:44:23.324 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:44:23.325 DEBUG:boto:Method: GET 2022-09-12T19:44:23.326 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:44:23.326 DEBUG:boto:Data: 2022-09-12T19:44:23.327 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.328 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:23.329 DEBUG:boto:Port: 8001 2022-09-12T19:44:23.329 DEBUG:boto:Params: {} 2022-09-12T19:44:23.330 DEBUG:boto:Token: None 2022-09-12T19:44:23.331 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/ 2022-09-12T19:44:23.333 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tJNykwaMYzGVMWYYIkwbemZSRyo= 2022-09-12T19:44:23.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tJNykwaMYzGVMWYYIkwbemZSRyo=', 'Content-Length': '0'} 2022-09-12T19:44:23.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca07765f37be342d-00631f8c17-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.343 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:23.344 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:23.345 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:23.345 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:44:23.346 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:44:23.346 DEBUG:boto:Method: GET 2022-09-12T19:44:23.347 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:44:23.348 DEBUG:boto:Data: 2022-09-12T19:44:23.348 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.349 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:23.350 DEBUG:boto:Port: 8000 2022-09-12T19:44:23.350 DEBUG:boto:Params: {} 2022-09-12T19:44:23.351 DEBUG:boto:Token: None 2022-09-12T19:44:23.351 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/ 2022-09-12T19:44:23.352 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tJNykwaMYzGVMWYYIkwbemZSRyo= 2022-09-12T19:44:23.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tJNykwaMYzGVMWYYIkwbemZSRyo=', 'Content-Length': '0'} 2022-09-12T19:44:23.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c175a4001ef0beed-00631f8c17-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.356 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:23.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:44:23.357 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:23.358 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:23.359 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:23.359 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:23.360 DEBUG:boto:Method: GET 2022-09-12T19:44:23.360 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:44:23.361 DEBUG:boto:Data: 2022-09-12T19:44:23.361 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.362 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:23.363 DEBUG:boto:Port: 8001 2022-09-12T19:44:23.363 DEBUG:boto:Params: {} 2022-09-12T19:44:23.364 DEBUG:boto:Token: None 2022-09-12T19:44:23.364 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:44:23.365 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tBBfHy6+ZJuj7XQu4XgzTcxY/AU= 2022-09-12T19:44:23.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tBBfHy6+ZJuj7XQu4XgzTcxY/AU=', 'Content-Length': '0'} 2022-09-12T19:44:23.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5c6f7215bfc5bd4-00631f8c17-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.369 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:23.370 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:23.371 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:23.371 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:23.372 DEBUG:boto:Method: GET 2022-09-12T19:44:23.372 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:44:23.373 DEBUG:boto:Data: 2022-09-12T19:44:23.373 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.374 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:23.375 DEBUG:boto:Port: 8000 2022-09-12T19:44:23.375 DEBUG:boto:Params: {} 2022-09-12T19:44:23.376 DEBUG:boto:Token: None 2022-09-12T19:44:23.376 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:44:23.377 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tBBfHy6+ZJuj7XQu4XgzTcxY/AU= 2022-09-12T19:44:23.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tBBfHy6+ZJuj7XQu4XgzTcxY/AU=', 'Content-Length': '0'} 2022-09-12T19:44:23.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d2684f1832c5ea2-00631f8c17-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.422 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:23.424 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:23.425 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:23.425 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:23.426 DEBUG:boto:Method: HEAD 2022-09-12T19:44:23.427 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:44:23.427 DEBUG:boto:Data: 2022-09-12T19:44:23.428 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.428 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:23.429 DEBUG:boto:Port: 8001 2022-09-12T19:44:23.430 DEBUG:boto:Params: {} 2022-09-12T19:44:23.431 DEBUG:boto:Token: None 2022-09-12T19:44:23.432 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:44:23.433 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JIYhgUblqR4IvqGVYEZVm6Wcg/4= 2022-09-12T19:44:23.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JIYhgUblqR4IvqGVYEZVm6Wcg/4=', 'Content-Length': '0'} 2022-09-12T19:44:23.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faa8c5ed52159266-00631f8c17-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.437 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:23.437 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:23.438 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:23.439 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:23.440 DEBUG:boto:Method: HEAD 2022-09-12T19:44:23.441 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:44:23.442 DEBUG:boto:Data: 2022-09-12T19:44:23.442 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.443 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:23.444 DEBUG:boto:Port: 8000 2022-09-12T19:44:23.445 DEBUG:boto:Params: {} 2022-09-12T19:44:23.446 DEBUG:boto:Token: None 2022-09-12T19:44:23.447 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:44:23.448 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JIYhgUblqR4IvqGVYEZVm6Wcg/4= 2022-09-12T19:44:23.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JIYhgUblqR4IvqGVYEZVm6Wcg/4=', 'Content-Length': '0'} 2022-09-12T19:44:23.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000335da57f44c2cd57-00631f8c17-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:44:23.453 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:23.453 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:23.454 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:23.455 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:23.455 DEBUG:boto:Method: GET 2022-09-12T19:44:23.456 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:44:23.456 DEBUG:boto:Data: 2022-09-12T19:44:23.457 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.457 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:23.458 DEBUG:boto:Port: 8001 2022-09-12T19:44:23.459 DEBUG:boto:Params: {} 2022-09-12T19:44:23.460 DEBUG:boto:Token: None 2022-09-12T19:44:23.462 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:44:23.463 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tBBfHy6+ZJuj7XQu4XgzTcxY/AU= 2022-09-12T19:44:23.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tBBfHy6+ZJuj7XQu4XgzTcxY/AU=', 'Content-Length': '0'} 2022-09-12T19:44:23.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dcf9283f5fa9871-00631f8c17-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.468 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:23.469 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:23.469 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:23.470 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:23.471 DEBUG:boto:Method: GET 2022-09-12T19:44:23.472 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:44:23.473 DEBUG:boto:Data: 2022-09-12T19:44:23.474 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.475 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:23.476 DEBUG:boto:Port: 8000 2022-09-12T19:44:23.477 DEBUG:boto:Params: {} 2022-09-12T19:44:23.478 DEBUG:boto:Token: None 2022-09-12T19:44:23.479 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:44:23.480 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tBBfHy6+ZJuj7XQu4XgzTcxY/AU= 2022-09-12T19:44:23.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tBBfHy6+ZJuj7XQu4XgzTcxY/AU=', 'Content-Length': '0'} 2022-09-12T19:44:23.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f91e1d0aeb94d7d8-00631f8c17-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.485 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:23.486 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:23.487 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:44:23.488 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:44:23.489 DEBUG:boto:Method: GET 2022-09-12T19:44:23.490 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:44:23.491 DEBUG:boto:Data: 2022-09-12T19:44:23.492 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.493 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:23.494 DEBUG:boto:Port: 8001 2022-09-12T19:44:23.495 DEBUG:boto:Params: {} 2022-09-12T19:44:23.495 DEBUG:boto:Token: None 2022-09-12T19:44:23.496 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/ 2022-09-12T19:44:23.497 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tJNykwaMYzGVMWYYIkwbemZSRyo= 2022-09-12T19:44:23.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tJNykwaMYzGVMWYYIkwbemZSRyo=', 'Content-Length': '0'} 2022-09-12T19:44:23.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd6472d9a5d19079-00631f8c17-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.550 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:23.552 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:23.553 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:23.553 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:44:23.558 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:44:23.559 DEBUG:boto:Method: GET 2022-09-12T19:44:23.560 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:44:23.561 DEBUG:boto:Data: 2022-09-12T19:44:23.562 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.563 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:23.565 DEBUG:boto:Port: 8000 2022-09-12T19:44:23.566 DEBUG:boto:Params: {} 2022-09-12T19:44:23.567 DEBUG:boto:Token: None 2022-09-12T19:44:23.569 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/ 2022-09-12T19:44:23.570 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tJNykwaMYzGVMWYYIkwbemZSRyo= 2022-09-12T19:44:23.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tJNykwaMYzGVMWYYIkwbemZSRyo=', 'Content-Length': '0'} 2022-09-12T19:44:23.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0d2659e86d4e200-00631f8c17-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.577 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:23.579 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:23.580 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:23.581 DEBUG:boto:Method: HEAD 2022-09-12T19:44:23.586 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:44:23.587 DEBUG:boto:Data: 2022-09-12T19:44:23.588 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.588 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:23.589 DEBUG:boto:Port: 8001 2022-09-12T19:44:23.590 DEBUG:boto:Params: {} 2022-09-12T19:44:23.591 DEBUG:boto:Token: None 2022-09-12T19:44:23.592 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/c 2022-09-12T19:44:23.592 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ONDyxWinodU2AlWYm8lsxikz6VQ= 2022-09-12T19:44:23.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ONDyxWinodU2AlWYm8lsxikz6VQ=', 'Content-Length': '0'} 2022-09-12T19:44:23.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050596680f94cfd4d-00631f8c17-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.599 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:23.600 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:23.602 DEBUG:boto:Method: HEAD 2022-09-12T19:44:23.604 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:44:23.605 DEBUG:boto:Data: 2022-09-12T19:44:23.605 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.606 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:23.608 DEBUG:boto:Port: 8000 2022-09-12T19:44:23.608 DEBUG:boto:Params: {} 2022-09-12T19:44:23.609 DEBUG:boto:Token: None 2022-09-12T19:44:23.611 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/c 2022-09-12T19:44:23.612 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ONDyxWinodU2AlWYm8lsxikz6VQ= 2022-09-12T19:44:23.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ONDyxWinodU2AlWYm8lsxikz6VQ=', 'Content-Length': '0'} 2022-09-12T19:44:23.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c011eb30308ae160-00631f8c17-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:44:23.619 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:23.621 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:23.622 DEBUG:boto:Method: GET 2022-09-12T19:44:23.623 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:44:23.624 DEBUG:boto:Data: 2022-09-12T19:44:23.625 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.626 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:23.628 DEBUG:boto:Port: 8001 2022-09-12T19:44:23.629 DEBUG:boto:Params: {} 2022-09-12T19:44:23.630 DEBUG:boto:Token: None 2022-09-12T19:44:23.631 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/c 2022-09-12T19:44:23.633 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8Ki9XevwnAVRHYyvcUGJLnjl+is= 2022-09-12T19:44:23.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8Ki9XevwnAVRHYyvcUGJLnjl+is=', 'Content-Length': '0'} 2022-09-12T19:44:23.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079ad5067dc403b1d-00631f8c17-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.642 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:23.643 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:23.645 DEBUG:boto:Method: GET 2022-09-12T19:44:23.646 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:44:23.648 DEBUG:boto:Data: 2022-09-12T19:44:23.649 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.650 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:23.651 DEBUG:boto:Port: 8000 2022-09-12T19:44:23.652 DEBUG:boto:Params: {} 2022-09-12T19:44:23.653 DEBUG:boto:Token: None 2022-09-12T19:44:23.654 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/c 2022-09-12T19:44:23.655 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8Ki9XevwnAVRHYyvcUGJLnjl+is= 2022-09-12T19:44:23.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8Ki9XevwnAVRHYyvcUGJLnjl+is=', 'Content-Length': '0'} 2022-09-12T19:44:23.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077e7a2f13158718b-00631f8c17-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.664 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:23.667 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:23.669 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:44:23.670 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:44:23.671 DEBUG:boto:Method: GET 2022-09-12T19:44:23.672 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:44:23.673 DEBUG:boto:Data: 2022-09-12T19:44:23.674 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.675 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:23.676 DEBUG:boto:Port: 8001 2022-09-12T19:44:23.677 DEBUG:boto:Params: {} 2022-09-12T19:44:23.679 DEBUG:boto:Token: None 2022-09-12T19:44:23.680 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/ 2022-09-12T19:44:23.681 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tJNykwaMYzGVMWYYIkwbemZSRyo= 2022-09-12T19:44:23.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tJNykwaMYzGVMWYYIkwbemZSRyo=', 'Content-Length': '0'} 2022-09-12T19:44:23.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c138c5766777ccca-00631f8c17-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.692 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:23.692 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:23.693 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:23.694 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:44:23.695 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:44:23.695 DEBUG:boto:Method: GET 2022-09-12T19:44:23.696 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:44:23.697 DEBUG:boto:Data: 2022-09-12T19:44:23.697 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.698 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:23.699 DEBUG:boto:Port: 8000 2022-09-12T19:44:23.700 DEBUG:boto:Params: {} 2022-09-12T19:44:23.701 DEBUG:boto:Token: None 2022-09-12T19:44:23.702 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/ 2022-09-12T19:44:23.703 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tJNykwaMYzGVMWYYIkwbemZSRyo= 2022-09-12T19:44:23.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tJNykwaMYzGVMWYYIkwbemZSRyo=', 'Content-Length': '0'} 2022-09-12T19:44:23.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052742dfcc91f1d4e-00631f8c17-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.709 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:23.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:44:23.711 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:23.712 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:23.713 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:23.714 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:23.715 DEBUG:boto:Method: GET 2022-09-12T19:44:23.716 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:44:23.716 DEBUG:boto:Data: 2022-09-12T19:44:23.717 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.718 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:23.719 DEBUG:boto:Port: 8001 2022-09-12T19:44:23.720 DEBUG:boto:Params: {} 2022-09-12T19:44:23.721 DEBUG:boto:Token: None 2022-09-12T19:44:23.723 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:44:23.724 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tb5j7Qsz1QldXKa4HZBQ4Qtq1AI= 2022-09-12T19:44:23.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tb5j7Qsz1QldXKa4HZBQ4Qtq1AI=', 'Content-Length': '0'} 2022-09-12T19:44:23.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000d43f5bd9c66572-00631f8c17-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.729 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:23.729 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:23.730 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:23.730 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:23.731 DEBUG:boto:Method: GET 2022-09-12T19:44:23.731 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:44:23.732 DEBUG:boto:Data: 2022-09-12T19:44:23.733 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.734 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:23.734 DEBUG:boto:Port: 8000 2022-09-12T19:44:23.735 DEBUG:boto:Params: {} 2022-09-12T19:44:23.736 DEBUG:boto:Token: None 2022-09-12T19:44:23.736 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:44:23.737 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tb5j7Qsz1QldXKa4HZBQ4Qtq1AI= 2022-09-12T19:44:23.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tb5j7Qsz1QldXKa4HZBQ4Qtq1AI=', 'Content-Length': '0'} 2022-09-12T19:44:23.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003777731ca86fd993-00631f8c17-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.782 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:23.783 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:23.783 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:23.784 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:23.784 DEBUG:boto:Method: HEAD 2022-09-12T19:44:23.786 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:44:23.787 DEBUG:boto:Data: 2022-09-12T19:44:23.788 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.789 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:23.790 DEBUG:boto:Port: 8001 2022-09-12T19:44:23.791 DEBUG:boto:Params: {} 2022-09-12T19:44:23.792 DEBUG:boto:Token: None 2022-09-12T19:44:23.794 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:44:23.795 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yKVIu8zFncMSUa+E0NrGMlwLeV4= 2022-09-12T19:44:23.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yKVIu8zFncMSUa+E0NrGMlwLeV4=', 'Content-Length': '0'} 2022-09-12T19:44:23.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4c33841263a4c64-00631f8c17-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.800 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:23.801 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:23.803 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:23.804 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:23.804 DEBUG:boto:Method: HEAD 2022-09-12T19:44:23.805 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:44:23.807 DEBUG:boto:Data: 2022-09-12T19:44:23.807 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.808 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:23.809 DEBUG:boto:Port: 8000 2022-09-12T19:44:23.810 DEBUG:boto:Params: {} 2022-09-12T19:44:23.811 DEBUG:boto:Token: None 2022-09-12T19:44:23.812 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:44:23.813 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yKVIu8zFncMSUa+E0NrGMlwLeV4= 2022-09-12T19:44:23.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yKVIu8zFncMSUa+E0NrGMlwLeV4=', 'Content-Length': '0'} 2022-09-12T19:44:23.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031654453193ceef4-00631f8c17-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:44:23.819 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:23.820 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:23.821 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:23.822 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:23.823 DEBUG:boto:Method: GET 2022-09-12T19:44:23.823 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:44:23.824 DEBUG:boto:Data: 2022-09-12T19:44:23.825 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.827 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:23.829 DEBUG:boto:Port: 8001 2022-09-12T19:44:23.830 DEBUG:boto:Params: {} 2022-09-12T19:44:23.831 DEBUG:boto:Token: None 2022-09-12T19:44:23.832 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:44:23.833 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tb5j7Qsz1QldXKa4HZBQ4Qtq1AI= 2022-09-12T19:44:23.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tb5j7Qsz1QldXKa4HZBQ4Qtq1AI=', 'Content-Length': '0'} 2022-09-12T19:44:23.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6b035427a343be5-00631f8c17-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.882 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:23.883 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:23.884 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:23.885 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:23.886 DEBUG:boto:Method: GET 2022-09-12T19:44:23.887 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:44:23.887 DEBUG:boto:Data: 2022-09-12T19:44:23.888 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.888 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:23.889 DEBUG:boto:Port: 8000 2022-09-12T19:44:23.889 DEBUG:boto:Params: {} 2022-09-12T19:44:23.890 DEBUG:boto:Token: None 2022-09-12T19:44:23.891 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:44:23.891 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tb5j7Qsz1QldXKa4HZBQ4Qtq1AI= 2022-09-12T19:44:23.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tb5j7Qsz1QldXKa4HZBQ4Qtq1AI=', 'Content-Length': '0'} 2022-09-12T19:44:23.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e7544232fc5f8c8-00631f8c17-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.896 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:23.896 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:23.897 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:44:23.898 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:44:23.898 DEBUG:boto:Method: GET 2022-09-12T19:44:23.899 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:44:23.900 DEBUG:boto:Data: 2022-09-12T19:44:23.900 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.901 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:23.902 DEBUG:boto:Port: 8001 2022-09-12T19:44:23.903 DEBUG:boto:Params: {} 2022-09-12T19:44:23.906 DEBUG:boto:Token: None 2022-09-12T19:44:23.907 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/ 2022-09-12T19:44:23.908 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tJNykwaMYzGVMWYYIkwbemZSRyo= 2022-09-12T19:44:23.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tJNykwaMYzGVMWYYIkwbemZSRyo=', 'Content-Length': '0'} 2022-09-12T19:44:23.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fad8001d0a82b2d1-00631f8c17-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.912 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:23.914 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:23.915 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:23.916 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:44:23.916 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:44:23.917 DEBUG:boto:Method: GET 2022-09-12T19:44:23.918 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:44:23.919 DEBUG:boto:Data: 2022-09-12T19:44:23.920 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.920 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:23.921 DEBUG:boto:Port: 8000 2022-09-12T19:44:23.922 DEBUG:boto:Params: {} 2022-09-12T19:44:23.923 DEBUG:boto:Token: None 2022-09-12T19:44:23.925 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/ 2022-09-12T19:44:23.926 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tJNykwaMYzGVMWYYIkwbemZSRyo= 2022-09-12T19:44:23.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tJNykwaMYzGVMWYYIkwbemZSRyo=', 'Content-Length': '0'} 2022-09-12T19:44:23.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078485e224d18a118-00631f8c17-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.929 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:23.930 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:23.931 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:23.931 DEBUG:boto:Method: HEAD 2022-09-12T19:44:23.932 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:44:23.932 DEBUG:boto:Data: 2022-09-12T19:44:23.933 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.934 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:23.935 DEBUG:boto:Port: 8001 2022-09-12T19:44:23.936 DEBUG:boto:Params: {} 2022-09-12T19:44:23.937 DEBUG:boto:Token: None 2022-09-12T19:44:23.938 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/d 2022-09-12T19:44:23.938 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X5Kiq6sl1Z7I6qg1tXypib/uGeY= 2022-09-12T19:44:23.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X5Kiq6sl1Z7I6qg1tXypib/uGeY=', 'Content-Length': '0'} 2022-09-12T19:44:23.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fd146f509cee839-00631f8c17-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.943 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:23.943 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:23.944 DEBUG:boto:Method: HEAD 2022-09-12T19:44:23.945 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:44:23.945 DEBUG:boto:Data: 2022-09-12T19:44:23.946 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.946 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:23.947 DEBUG:boto:Port: 8000 2022-09-12T19:44:23.948 DEBUG:boto:Params: {} 2022-09-12T19:44:23.949 DEBUG:boto:Token: None 2022-09-12T19:44:23.950 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/d 2022-09-12T19:44:23.951 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X5Kiq6sl1Z7I6qg1tXypib/uGeY= 2022-09-12T19:44:23.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X5Kiq6sl1Z7I6qg1tXypib/uGeY=', 'Content-Length': '0'} 2022-09-12T19:44:23.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bccd75fcb6c760fc-00631f8c17-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:44:23.957 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:23.958 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:23.959 DEBUG:boto:Method: GET 2022-09-12T19:44:23.960 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:44:23.961 DEBUG:boto:Data: 2022-09-12T19:44:23.962 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.962 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:23.963 DEBUG:boto:Port: 8001 2022-09-12T19:44:23.963 DEBUG:boto:Params: {} 2022-09-12T19:44:23.964 DEBUG:boto:Token: None 2022-09-12T19:44:23.965 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/d 2022-09-12T19:44:23.965 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KevhTs5jZIotWrULI8z/wfjaoaU= 2022-09-12T19:44:23.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KevhTs5jZIotWrULI8z/wfjaoaU=', 'Content-Length': '0'} 2022-09-12T19:44:23.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000762bda8b99845047-00631f8c17-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:23.970 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:23.971 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:23.972 DEBUG:boto:Method: GET 2022-09-12T19:44:23.973 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:44:23.974 DEBUG:boto:Data: 2022-09-12T19:44:23.975 DEBUG:boto:Headers: {} 2022-09-12T19:44:23.975 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:23.976 DEBUG:boto:Port: 8000 2022-09-12T19:44:23.976 DEBUG:boto:Params: {} 2022-09-12T19:44:23.977 DEBUG:boto:Token: None 2022-09-12T19:44:23.978 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:23 GMT /jdpiak-71/d 2022-09-12T19:44:23.979 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KevhTs5jZIotWrULI8z/wfjaoaU= 2022-09-12T19:44:23.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KevhTs5jZIotWrULI8z/wfjaoaU=', 'Content-Length': '0'} 2022-09-12T19:44:23.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051e247bef620d4cc-00631f8c17-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.026 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:24.027 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:24.028 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:44:24.028 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:44:24.029 DEBUG:boto:Method: GET 2022-09-12T19:44:24.030 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:44:24.030 DEBUG:boto:Data: 2022-09-12T19:44:24.031 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.032 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:24.033 DEBUG:boto:Port: 8001 2022-09-12T19:44:24.033 DEBUG:boto:Params: {} 2022-09-12T19:44:24.034 DEBUG:boto:Token: None 2022-09-12T19:44:24.035 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/ 2022-09-12T19:44:24.036 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UUH86dsZSKOYz6cLTgXXf5KuVlo= 2022-09-12T19:44:24.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UUH86dsZSKOYz6cLTgXXf5KuVlo=', 'Content-Length': '0'} 2022-09-12T19:44:24.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba31980fb328f0cd-00631f8c18-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.041 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:24.042 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:24.043 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:24.044 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:44:24.045 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:44:24.046 DEBUG:boto:Method: GET 2022-09-12T19:44:24.047 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:44:24.048 DEBUG:boto:Data: 2022-09-12T19:44:24.049 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.049 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:24.050 DEBUG:boto:Port: 8000 2022-09-12T19:44:24.051 DEBUG:boto:Params: {} 2022-09-12T19:44:24.052 DEBUG:boto:Token: None 2022-09-12T19:44:24.053 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/ 2022-09-12T19:44:24.053 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UUH86dsZSKOYz6cLTgXXf5KuVlo= 2022-09-12T19:44:24.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UUH86dsZSKOYz6cLTgXXf5KuVlo=', 'Content-Length': '0'} 2022-09-12T19:44:24.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd01261258d3db9e-00631f8c18-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.058 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:24.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:44:24.060 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:24.061 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:24.062 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:24.063 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:24.064 DEBUG:boto:Method: GET 2022-09-12T19:44:24.065 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:44:24.066 DEBUG:boto:Data: 2022-09-12T19:44:24.067 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.068 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:24.069 DEBUG:boto:Port: 8001 2022-09-12T19:44:24.070 DEBUG:boto:Params: {} 2022-09-12T19:44:24.072 DEBUG:boto:Token: None 2022-09-12T19:44:24.073 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:44:24.074 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCAMkxrh58WsGjVRS7BB1Xfml5o= 2022-09-12T19:44:24.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCAMkxrh58WsGjVRS7BB1Xfml5o=', 'Content-Length': '0'} 2022-09-12T19:44:24.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f0937ec10483f4a-00631f8c18-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.122 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:24.123 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:24.124 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:24.125 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:24.127 DEBUG:boto:Method: GET 2022-09-12T19:44:24.127 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:44:24.128 DEBUG:boto:Data: 2022-09-12T19:44:24.129 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.130 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:24.131 DEBUG:boto:Port: 8000 2022-09-12T19:44:24.132 DEBUG:boto:Params: {} 2022-09-12T19:44:24.132 DEBUG:boto:Token: None 2022-09-12T19:44:24.133 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:44:24.133 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCAMkxrh58WsGjVRS7BB1Xfml5o= 2022-09-12T19:44:24.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCAMkxrh58WsGjVRS7BB1Xfml5o=', 'Content-Length': '0'} 2022-09-12T19:44:24.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036f558716f2e60fa-00631f8c18-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.138 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:24.139 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:24.140 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:24.140 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:24.141 DEBUG:boto:Method: HEAD 2022-09-12T19:44:24.141 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:44:24.142 DEBUG:boto:Data: 2022-09-12T19:44:24.143 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.143 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:24.144 DEBUG:boto:Port: 8001 2022-09-12T19:44:24.145 DEBUG:boto:Params: {} 2022-09-12T19:44:24.146 DEBUG:boto:Token: None 2022-09-12T19:44:24.147 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:44:24.148 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3sqBkMY+2y1/RAdMk+v4s8nmwxM= 2022-09-12T19:44:24.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3sqBkMY+2y1/RAdMk+v4s8nmwxM=', 'Content-Length': '0'} 2022-09-12T19:44:24.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000703a47d8c39e8b16-00631f8c18-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.152 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:24.153 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:24.153 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:24.154 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:24.155 DEBUG:boto:Method: HEAD 2022-09-12T19:44:24.156 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:44:24.157 DEBUG:boto:Data: 2022-09-12T19:44:24.158 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.159 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:24.160 DEBUG:boto:Port: 8000 2022-09-12T19:44:24.161 DEBUG:boto:Params: {} 2022-09-12T19:44:24.162 DEBUG:boto:Token: None 2022-09-12T19:44:24.163 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:44:24.164 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3sqBkMY+2y1/RAdMk+v4s8nmwxM= 2022-09-12T19:44:24.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3sqBkMY+2y1/RAdMk+v4s8nmwxM=', 'Content-Length': '0'} 2022-09-12T19:44:24.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a427606b99c2005-00631f8c18-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:44:24.168 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:24.169 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:24.170 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:24.171 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:24.171 DEBUG:boto:Method: GET 2022-09-12T19:44:24.172 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:44:24.173 DEBUG:boto:Data: 2022-09-12T19:44:24.174 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.174 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:24.175 DEBUG:boto:Port: 8001 2022-09-12T19:44:24.176 DEBUG:boto:Params: {} 2022-09-12T19:44:24.176 DEBUG:boto:Token: None 2022-09-12T19:44:24.177 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:44:24.178 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCAMkxrh58WsGjVRS7BB1Xfml5o= 2022-09-12T19:44:24.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCAMkxrh58WsGjVRS7BB1Xfml5o=', 'Content-Length': '0'} 2022-09-12T19:44:24.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099a47bd494ebf210-00631f8c18-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.183 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:24.184 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:24.185 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:24.186 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:24.187 DEBUG:boto:Method: GET 2022-09-12T19:44:24.188 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:44:24.189 DEBUG:boto:Data: 2022-09-12T19:44:24.190 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.191 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:24.192 DEBUG:boto:Port: 8000 2022-09-12T19:44:24.193 DEBUG:boto:Params: {} 2022-09-12T19:44:24.195 DEBUG:boto:Token: None 2022-09-12T19:44:24.196 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:44:24.197 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCAMkxrh58WsGjVRS7BB1Xfml5o= 2022-09-12T19:44:24.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCAMkxrh58WsGjVRS7BB1Xfml5o=', 'Content-Length': '0'} 2022-09-12T19:44:24.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005103fddef611d928-00631f8c18-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.202 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:24.203 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:24.203 DEBUG:boto:path=/jdpiak-71/?prefix=e 2022-09-12T19:44:24.204 DEBUG:boto:auth_path=/jdpiak-71/?prefix=e 2022-09-12T19:44:24.204 DEBUG:boto:Method: GET 2022-09-12T19:44:24.205 DEBUG:boto:Path: /jdpiak-71/?prefix=e 2022-09-12T19:44:24.205 DEBUG:boto:Data: 2022-09-12T19:44:24.206 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.206 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:24.207 DEBUG:boto:Port: 8001 2022-09-12T19:44:24.208 DEBUG:boto:Params: {} 2022-09-12T19:44:24.208 DEBUG:boto:Token: None 2022-09-12T19:44:24.209 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/ 2022-09-12T19:44:24.210 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UUH86dsZSKOYz6cLTgXXf5KuVlo= 2022-09-12T19:44:24.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UUH86dsZSKOYz6cLTgXXf5KuVlo=', 'Content-Length': '0'} 2022-09-12T19:44:24.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be29029c7433a6fb-00631f8c18-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.214 DEBUG:boto:b'jdpiak-71e1000falsee2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:24.215 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:24.215 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:24.216 DEBUG:boto:path=/jdpiak-71/?prefix=e 2022-09-12T19:44:24.216 DEBUG:boto:auth_path=/jdpiak-71/?prefix=e 2022-09-12T19:44:24.221 DEBUG:boto:Method: GET 2022-09-12T19:44:24.224 DEBUG:boto:Path: /jdpiak-71/?prefix=e 2022-09-12T19:44:24.227 DEBUG:boto:Data: 2022-09-12T19:44:24.229 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.230 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:24.230 DEBUG:boto:Port: 8000 2022-09-12T19:44:24.231 DEBUG:boto:Params: {} 2022-09-12T19:44:24.232 DEBUG:boto:Token: None 2022-09-12T19:44:24.236 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/ 2022-09-12T19:44:24.237 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UUH86dsZSKOYz6cLTgXXf5KuVlo= 2022-09-12T19:44:24.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UUH86dsZSKOYz6cLTgXXf5KuVlo=', 'Content-Length': '0'} 2022-09-12T19:44:24.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8df2ee825875387-00631f8c18-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.248 DEBUG:boto:b'jdpiak-71e1000falsee2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:24.249 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:24.250 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:24.251 DEBUG:boto:Method: HEAD 2022-09-12T19:44:24.252 DEBUG:boto:Path: /jdpiak-71/e 2022-09-12T19:44:24.252 DEBUG:boto:Data: 2022-09-12T19:44:24.253 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.255 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:24.257 DEBUG:boto:Port: 8001 2022-09-12T19:44:24.258 DEBUG:boto:Params: {} 2022-09-12T19:44:24.259 DEBUG:boto:Token: None 2022-09-12T19:44:24.260 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/e 2022-09-12T19:44:24.261 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EUd/gR3POBDwJwE7X6vH2xtV8aI= 2022-09-12T19:44:24.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EUd/gR3POBDwJwE7X6vH2xtV8aI=', 'Content-Length': '0'} 2022-09-12T19:44:24.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059819ada31b5f511-00631f8c18-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.265 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:24.268 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:24.269 DEBUG:boto:Method: HEAD 2022-09-12T19:44:24.270 DEBUG:boto:Path: /jdpiak-71/e 2022-09-12T19:44:24.271 DEBUG:boto:Data: 2022-09-12T19:44:24.271 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.272 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:24.273 DEBUG:boto:Port: 8000 2022-09-12T19:44:24.274 DEBUG:boto:Params: {} 2022-09-12T19:44:24.276 DEBUG:boto:Token: None 2022-09-12T19:44:24.277 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/e 2022-09-12T19:44:24.278 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EUd/gR3POBDwJwE7X6vH2xtV8aI= 2022-09-12T19:44:24.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EUd/gR3POBDwJwE7X6vH2xtV8aI=', 'Content-Length': '0'} 2022-09-12T19:44:24.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051e69d312f1addc4-00631f8c18-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:44:24.284 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:24.285 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:24.285 DEBUG:boto:Method: GET 2022-09-12T19:44:24.286 DEBUG:boto:Path: /jdpiak-71/e 2022-09-12T19:44:24.287 DEBUG:boto:Data: 2022-09-12T19:44:24.288 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.289 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:24.290 DEBUG:boto:Port: 8001 2022-09-12T19:44:24.291 DEBUG:boto:Params: {} 2022-09-12T19:44:24.292 DEBUG:boto:Token: None 2022-09-12T19:44:24.293 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/e 2022-09-12T19:44:24.294 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pDaJdh/neOEU1UdsRxIp01AOZOg= 2022-09-12T19:44:24.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pDaJdh/neOEU1UdsRxIp01AOZOg=', 'Content-Length': '0'} 2022-09-12T19:44:24.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000857063af1584842-00631f8c18-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.304 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:24.305 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:24.306 DEBUG:boto:Method: GET 2022-09-12T19:44:24.307 DEBUG:boto:Path: /jdpiak-71/e 2022-09-12T19:44:24.308 DEBUG:boto:Data: 2022-09-12T19:44:24.309 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.311 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:24.312 DEBUG:boto:Port: 8000 2022-09-12T19:44:24.313 DEBUG:boto:Params: {} 2022-09-12T19:44:24.315 DEBUG:boto:Token: None 2022-09-12T19:44:24.316 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/e 2022-09-12T19:44:24.318 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pDaJdh/neOEU1UdsRxIp01AOZOg= 2022-09-12T19:44:24.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pDaJdh/neOEU1UdsRxIp01AOZOg=', 'Content-Length': '0'} 2022-09-12T19:44:24.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0b666abeb46742b-00631f8c18-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.324 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:24.325 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:24.326 DEBUG:boto:path=/jdpiak-71/?prefix=e 2022-09-12T19:44:24.327 DEBUG:boto:auth_path=/jdpiak-71/?prefix=e 2022-09-12T19:44:24.328 DEBUG:boto:Method: GET 2022-09-12T19:44:24.330 DEBUG:boto:Path: /jdpiak-71/?prefix=e 2022-09-12T19:44:24.331 DEBUG:boto:Data: 2022-09-12T19:44:24.332 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.333 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:24.334 DEBUG:boto:Port: 8001 2022-09-12T19:44:24.335 DEBUG:boto:Params: {} 2022-09-12T19:44:24.336 DEBUG:boto:Token: None 2022-09-12T19:44:24.338 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/ 2022-09-12T19:44:24.340 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UUH86dsZSKOYz6cLTgXXf5KuVlo= 2022-09-12T19:44:24.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UUH86dsZSKOYz6cLTgXXf5KuVlo=', 'Content-Length': '0'} 2022-09-12T19:44:24.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c6e55acadbca9d2-00631f8c18-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.346 DEBUG:boto:b'jdpiak-71e1000falsee2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:24.348 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:24.349 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:24.351 DEBUG:boto:path=/jdpiak-71/?prefix=e 2022-09-12T19:44:24.352 DEBUG:boto:auth_path=/jdpiak-71/?prefix=e 2022-09-12T19:44:24.353 DEBUG:boto:Method: GET 2022-09-12T19:44:24.354 DEBUG:boto:Path: /jdpiak-71/?prefix=e 2022-09-12T19:44:24.356 DEBUG:boto:Data: 2022-09-12T19:44:24.357 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.358 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:24.363 DEBUG:boto:Port: 8000 2022-09-12T19:44:24.364 DEBUG:boto:Params: {} 2022-09-12T19:44:24.365 DEBUG:boto:Token: None 2022-09-12T19:44:24.366 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/ 2022-09-12T19:44:24.367 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UUH86dsZSKOYz6cLTgXXf5KuVlo= 2022-09-12T19:44:24.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UUH86dsZSKOYz6cLTgXXf5KuVlo=', 'Content-Length': '0'} 2022-09-12T19:44:24.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000648e2ff453a567ea-00631f8c18-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.418 DEBUG:boto:b'jdpiak-71e1000falsee2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:24.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:44:24.420 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:24.421 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:24.421 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:24.422 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:24.423 DEBUG:boto:Method: GET 2022-09-12T19:44:24.423 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:24.424 DEBUG:boto:Data: 2022-09-12T19:44:24.424 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.425 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:24.425 DEBUG:boto:Port: 8001 2022-09-12T19:44:24.426 DEBUG:boto:Params: {} 2022-09-12T19:44:24.427 DEBUG:boto:Token: None 2022-09-12T19:44:24.428 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:24.428 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sAvz4R5s0GYfkzuRv7jn4Go8HgE= 2022-09-12T19:44:24.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sAvz4R5s0GYfkzuRv7jn4Go8HgE=', 'Content-Length': '0'} 2022-09-12T19:44:24.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b20355f52a1b6c4-00631f8c18-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.433 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:24.434 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:24.435 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:24.436 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:24.437 DEBUG:boto:Method: GET 2022-09-12T19:44:24.438 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:24.439 DEBUG:boto:Data: 2022-09-12T19:44:24.440 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.441 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:24.442 DEBUG:boto:Port: 8000 2022-09-12T19:44:24.443 DEBUG:boto:Params: {} 2022-09-12T19:44:24.445 DEBUG:boto:Token: None 2022-09-12T19:44:24.446 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:24.447 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sAvz4R5s0GYfkzuRv7jn4Go8HgE= 2022-09-12T19:44:24.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sAvz4R5s0GYfkzuRv7jn4Go8HgE=', 'Content-Length': '0'} 2022-09-12T19:44:24.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8fb94d1d5e397b3-00631f8c18-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.452 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:24.452 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:24.453 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:24.454 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:24.454 DEBUG:boto:Method: HEAD 2022-09-12T19:44:24.455 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:24.455 DEBUG:boto:Data: 2022-09-12T19:44:24.456 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.457 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:24.458 DEBUG:boto:Port: 8001 2022-09-12T19:44:24.459 DEBUG:boto:Params: {} 2022-09-12T19:44:24.460 DEBUG:boto:Token: None 2022-09-12T19:44:24.461 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:24.463 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wleK16HaAX8L5mCTJgTUMW2vcAQ= 2022-09-12T19:44:24.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wleK16HaAX8L5mCTJgTUMW2vcAQ=', 'Content-Length': '0'} 2022-09-12T19:44:24.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f84ee1a22447314f-00631f8c18-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.469 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:24.470 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:24.471 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:24.471 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:24.472 DEBUG:boto:Method: HEAD 2022-09-12T19:44:24.473 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:24.475 DEBUG:boto:Data: 2022-09-12T19:44:24.476 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.477 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:24.478 DEBUG:boto:Port: 8000 2022-09-12T19:44:24.479 DEBUG:boto:Params: {} 2022-09-12T19:44:24.480 DEBUG:boto:Token: None 2022-09-12T19:44:24.481 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:24.482 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wleK16HaAX8L5mCTJgTUMW2vcAQ= 2022-09-12T19:44:24.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wleK16HaAX8L5mCTJgTUMW2vcAQ=', 'Content-Length': '0'} 2022-09-12T19:44:24.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1091d9cb6cd5db3-00631f8c18-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:44:24.493 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:24.495 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:24.496 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:24.497 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:24.498 DEBUG:boto:Method: GET 2022-09-12T19:44:24.500 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:24.501 DEBUG:boto:Data: 2022-09-12T19:44:24.502 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.504 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:24.505 DEBUG:boto:Port: 8001 2022-09-12T19:44:24.506 DEBUG:boto:Params: {} 2022-09-12T19:44:24.508 DEBUG:boto:Token: None 2022-09-12T19:44:24.509 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:24.510 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sAvz4R5s0GYfkzuRv7jn4Go8HgE= 2022-09-12T19:44:24.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sAvz4R5s0GYfkzuRv7jn4Go8HgE=', 'Content-Length': '0'} 2022-09-12T19:44:24.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093ab101438af8670-00631f8c18-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.517 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:24.518 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:24.519 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:24.521 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:24.523 DEBUG:boto:Method: GET 2022-09-12T19:44:24.524 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:24.525 DEBUG:boto:Data: 2022-09-12T19:44:24.526 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.527 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:24.530 DEBUG:boto:Port: 8000 2022-09-12T19:44:24.531 DEBUG:boto:Params: {} 2022-09-12T19:44:24.533 DEBUG:boto:Token: None 2022-09-12T19:44:24.535 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:24.537 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sAvz4R5s0GYfkzuRv7jn4Go8HgE= 2022-09-12T19:44:24.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sAvz4R5s0GYfkzuRv7jn4Go8HgE=', 'Content-Length': '0'} 2022-09-12T19:44:24.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044dafd3af6e3b147-00631f8c18-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.582 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:24.583 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:24.584 DEBUG:boto:path=/jdpiak-71/?prefix=f 2022-09-12T19:44:24.585 DEBUG:boto:auth_path=/jdpiak-71/?prefix=f 2022-09-12T19:44:24.585 DEBUG:boto:Method: GET 2022-09-12T19:44:24.586 DEBUG:boto:Path: /jdpiak-71/?prefix=f 2022-09-12T19:44:24.587 DEBUG:boto:Data: 2022-09-12T19:44:24.587 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.588 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:24.588 DEBUG:boto:Port: 8001 2022-09-12T19:44:24.589 DEBUG:boto:Params: {} 2022-09-12T19:44:24.589 DEBUG:boto:Token: None 2022-09-12T19:44:24.590 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/ 2022-09-12T19:44:24.591 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UUH86dsZSKOYz6cLTgXXf5KuVlo= 2022-09-12T19:44:24.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UUH86dsZSKOYz6cLTgXXf5KuVlo=', 'Content-Length': '0'} 2022-09-12T19:44:24.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b00cf02702a4220b-00631f8c18-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.595 DEBUG:boto:b'jdpiak-71f1000falsef2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:24.596 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:24.596 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:24.597 DEBUG:boto:path=/jdpiak-71/?prefix=f 2022-09-12T19:44:24.598 DEBUG:boto:auth_path=/jdpiak-71/?prefix=f 2022-09-12T19:44:24.598 DEBUG:boto:Method: GET 2022-09-12T19:44:24.599 DEBUG:boto:Path: /jdpiak-71/?prefix=f 2022-09-12T19:44:24.600 DEBUG:boto:Data: 2022-09-12T19:44:24.601 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.602 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:24.603 DEBUG:boto:Port: 8000 2022-09-12T19:44:24.604 DEBUG:boto:Params: {} 2022-09-12T19:44:24.604 DEBUG:boto:Token: None 2022-09-12T19:44:24.605 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/ 2022-09-12T19:44:24.606 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UUH86dsZSKOYz6cLTgXXf5KuVlo= 2022-09-12T19:44:24.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UUH86dsZSKOYz6cLTgXXf5KuVlo=', 'Content-Length': '0'} 2022-09-12T19:44:24.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045a29d89d7a03444-00631f8c18-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.610 DEBUG:boto:b'jdpiak-71f1000falsef2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:24.611 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:24.612 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:24.613 DEBUG:boto:Method: HEAD 2022-09-12T19:44:24.613 DEBUG:boto:Path: /jdpiak-71/f 2022-09-12T19:44:24.614 DEBUG:boto:Data: 2022-09-12T19:44:24.614 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.615 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:24.615 DEBUG:boto:Port: 8001 2022-09-12T19:44:24.616 DEBUG:boto:Params: {} 2022-09-12T19:44:24.616 DEBUG:boto:Token: None 2022-09-12T19:44:24.617 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/f 2022-09-12T19:44:24.617 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4+FbONbeWGEIeMV/B+nHhqzQ4ow= 2022-09-12T19:44:24.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4+FbONbeWGEIeMV/B+nHhqzQ4ow=', 'Content-Length': '0'} 2022-09-12T19:44:24.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000851a1928d46d5346-00631f8c18-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.625 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:24.626 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:24.627 DEBUG:boto:Method: HEAD 2022-09-12T19:44:24.627 DEBUG:boto:Path: /jdpiak-71/f 2022-09-12T19:44:24.628 DEBUG:boto:Data: 2022-09-12T19:44:24.628 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.629 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:24.629 DEBUG:boto:Port: 8000 2022-09-12T19:44:24.630 DEBUG:boto:Params: {} 2022-09-12T19:44:24.631 DEBUG:boto:Token: None 2022-09-12T19:44:24.632 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/f 2022-09-12T19:44:24.633 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4+FbONbeWGEIeMV/B+nHhqzQ4ow= 2022-09-12T19:44:24.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4+FbONbeWGEIeMV/B+nHhqzQ4ow=', 'Content-Length': '0'} 2022-09-12T19:44:24.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e09e3783e43d163-00631f8c18-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:44:24.639 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:24.640 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:24.641 DEBUG:boto:Method: GET 2022-09-12T19:44:24.641 DEBUG:boto:Path: /jdpiak-71/f 2022-09-12T19:44:24.642 DEBUG:boto:Data: 2022-09-12T19:44:24.642 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.643 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:24.643 DEBUG:boto:Port: 8001 2022-09-12T19:44:24.644 DEBUG:boto:Params: {} 2022-09-12T19:44:24.645 DEBUG:boto:Token: None 2022-09-12T19:44:24.646 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/f 2022-09-12T19:44:24.646 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:baFDe2i3b7Be9Px78qCd4lldwy0= 2022-09-12T19:44:24.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:baFDe2i3b7Be9Px78qCd4lldwy0=', 'Content-Length': '0'} 2022-09-12T19:44:24.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086e83e4156ff97b5-00631f8c18-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.698 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:24.699 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:24.700 DEBUG:boto:Method: GET 2022-09-12T19:44:24.701 DEBUG:boto:Path: /jdpiak-71/f 2022-09-12T19:44:24.702 DEBUG:boto:Data: 2022-09-12T19:44:24.703 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.704 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:24.705 DEBUG:boto:Port: 8000 2022-09-12T19:44:24.706 DEBUG:boto:Params: {} 2022-09-12T19:44:24.707 DEBUG:boto:Token: None 2022-09-12T19:44:24.709 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/f 2022-09-12T19:44:24.710 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:baFDe2i3b7Be9Px78qCd4lldwy0= 2022-09-12T19:44:24.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:baFDe2i3b7Be9Px78qCd4lldwy0=', 'Content-Length': '0'} 2022-09-12T19:44:24.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d0f556ebf28ebbb-00631f8c18-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.717 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:24.720 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:24.721 DEBUG:boto:path=/jdpiak-71/?prefix=f 2022-09-12T19:44:24.722 DEBUG:boto:auth_path=/jdpiak-71/?prefix=f 2022-09-12T19:44:24.723 DEBUG:boto:Method: GET 2022-09-12T19:44:24.724 DEBUG:boto:Path: /jdpiak-71/?prefix=f 2022-09-12T19:44:24.725 DEBUG:boto:Data: 2022-09-12T19:44:24.726 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.727 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:24.728 DEBUG:boto:Port: 8001 2022-09-12T19:44:24.729 DEBUG:boto:Params: {} 2022-09-12T19:44:24.731 DEBUG:boto:Token: None 2022-09-12T19:44:24.732 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/ 2022-09-12T19:44:24.734 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UUH86dsZSKOYz6cLTgXXf5KuVlo= 2022-09-12T19:44:24.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UUH86dsZSKOYz6cLTgXXf5KuVlo=', 'Content-Length': '0'} 2022-09-12T19:44:24.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063eb7ca962de3295-00631f8c18-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.747 DEBUG:boto:b'jdpiak-71f1000falsef2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:24.749 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:24.749 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:24.750 DEBUG:boto:path=/jdpiak-71/?prefix=f 2022-09-12T19:44:24.751 DEBUG:boto:auth_path=/jdpiak-71/?prefix=f 2022-09-12T19:44:24.752 DEBUG:boto:Method: GET 2022-09-12T19:44:24.753 DEBUG:boto:Path: /jdpiak-71/?prefix=f 2022-09-12T19:44:24.755 DEBUG:boto:Data: 2022-09-12T19:44:24.756 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.759 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:24.760 DEBUG:boto:Port: 8000 2022-09-12T19:44:24.761 DEBUG:boto:Params: {} 2022-09-12T19:44:24.763 DEBUG:boto:Token: None 2022-09-12T19:44:24.765 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/ 2022-09-12T19:44:24.766 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UUH86dsZSKOYz6cLTgXXf5KuVlo= 2022-09-12T19:44:24.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UUH86dsZSKOYz6cLTgXXf5KuVlo=', 'Content-Length': '0'} 2022-09-12T19:44:24.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b565d5ab8be0200-00631f8c18-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.782 DEBUG:boto:b'jdpiak-71f1000falsef2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:24.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:44:24.784 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:24.784 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:24.785 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:24.786 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:24.786 DEBUG:boto:Method: GET 2022-09-12T19:44:24.787 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:24.788 DEBUG:boto:Data: 2022-09-12T19:44:24.788 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.789 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:24.789 DEBUG:boto:Port: 8001 2022-09-12T19:44:24.790 DEBUG:boto:Params: {} 2022-09-12T19:44:24.791 DEBUG:boto:Token: None 2022-09-12T19:44:24.791 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:24.792 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sPyMcqEEeIzchY4CfAhmbSz/zzY= 2022-09-12T19:44:24.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sPyMcqEEeIzchY4CfAhmbSz/zzY=', 'Content-Length': '0'} 2022-09-12T19:44:24.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005171dfe7507f898e-00631f8c18-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.801 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:24.802 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:24.803 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:24.804 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:24.805 DEBUG:boto:Method: GET 2022-09-12T19:44:24.806 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:24.807 DEBUG:boto:Data: 2022-09-12T19:44:24.809 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.810 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:24.811 DEBUG:boto:Port: 8000 2022-09-12T19:44:24.812 DEBUG:boto:Params: {} 2022-09-12T19:44:24.813 DEBUG:boto:Token: None 2022-09-12T19:44:24.814 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:24.815 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sPyMcqEEeIzchY4CfAhmbSz/zzY= 2022-09-12T19:44:24.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sPyMcqEEeIzchY4CfAhmbSz/zzY=', 'Content-Length': '0'} 2022-09-12T19:44:24.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc5019b92c54ce98-00631f8c18-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.822 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:24.822 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:24.823 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:24.824 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:24.825 DEBUG:boto:Method: HEAD 2022-09-12T19:44:24.826 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:24.827 DEBUG:boto:Data: 2022-09-12T19:44:24.828 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.829 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:24.830 DEBUG:boto:Port: 8001 2022-09-12T19:44:24.830 DEBUG:boto:Params: {} 2022-09-12T19:44:24.831 DEBUG:boto:Token: None 2022-09-12T19:44:24.832 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:24.833 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:juLquqqQG2Nec+aw/dF5XUG5WJ0= 2022-09-12T19:44:24.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:juLquqqQG2Nec+aw/dF5XUG5WJ0=', 'Content-Length': '0'} 2022-09-12T19:44:24.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d545d1f060a02c45-00631f8c18-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.853 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:24.854 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:24.855 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:24.856 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:24.857 DEBUG:boto:Method: HEAD 2022-09-12T19:44:24.858 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:24.859 DEBUG:boto:Data: 2022-09-12T19:44:24.861 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.863 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:24.864 DEBUG:boto:Port: 8000 2022-09-12T19:44:24.866 DEBUG:boto:Params: {} 2022-09-12T19:44:24.868 DEBUG:boto:Token: None 2022-09-12T19:44:24.869 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:24.871 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:juLquqqQG2Nec+aw/dF5XUG5WJ0= 2022-09-12T19:44:24.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:juLquqqQG2Nec+aw/dF5XUG5WJ0=', 'Content-Length': '0'} 2022-09-12T19:44:24.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa8a55213590934f-00631f8c18-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:44:24.879 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:24.880 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:24.881 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:24.883 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:24.884 DEBUG:boto:Method: GET 2022-09-12T19:44:24.886 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:24.887 DEBUG:boto:Data: 2022-09-12T19:44:24.888 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.889 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:24.890 DEBUG:boto:Port: 8001 2022-09-12T19:44:24.891 DEBUG:boto:Params: {} 2022-09-12T19:44:24.892 DEBUG:boto:Token: None 2022-09-12T19:44:24.894 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:24.895 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sPyMcqEEeIzchY4CfAhmbSz/zzY= 2022-09-12T19:44:24.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sPyMcqEEeIzchY4CfAhmbSz/zzY=', 'Content-Length': '0'} 2022-09-12T19:44:24.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013d44231d93b0e71-00631f8c18-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.902 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:24.902 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:24.903 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:24.904 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:24.904 DEBUG:boto:Method: GET 2022-09-12T19:44:24.905 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:24.906 DEBUG:boto:Data: 2022-09-12T19:44:24.906 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.907 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:24.907 DEBUG:boto:Port: 8000 2022-09-12T19:44:24.908 DEBUG:boto:Params: {} 2022-09-12T19:44:24.909 DEBUG:boto:Token: None 2022-09-12T19:44:24.909 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:24.910 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sPyMcqEEeIzchY4CfAhmbSz/zzY= 2022-09-12T19:44:24.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sPyMcqEEeIzchY4CfAhmbSz/zzY=', 'Content-Length': '0'} 2022-09-12T19:44:24.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b898545f4f380ca4-00631f8c18-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.954 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:24.956 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:24.957 DEBUG:boto:path=/jdpiak-71/?prefix=g 2022-09-12T19:44:24.958 DEBUG:boto:auth_path=/jdpiak-71/?prefix=g 2022-09-12T19:44:24.960 DEBUG:boto:Method: GET 2022-09-12T19:44:24.961 DEBUG:boto:Path: /jdpiak-71/?prefix=g 2022-09-12T19:44:24.962 DEBUG:boto:Data: 2022-09-12T19:44:24.963 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.964 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:24.965 DEBUG:boto:Port: 8001 2022-09-12T19:44:24.966 DEBUG:boto:Params: {} 2022-09-12T19:44:24.967 DEBUG:boto:Token: None 2022-09-12T19:44:24.969 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/ 2022-09-12T19:44:24.970 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UUH86dsZSKOYz6cLTgXXf5KuVlo= 2022-09-12T19:44:24.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UUH86dsZSKOYz6cLTgXXf5KuVlo=', 'Content-Length': '0'} 2022-09-12T19:44:24.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000133a2e05d45eb51b-00631f8c18-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.976 DEBUG:boto:b'jdpiak-71g1000falseg2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:24.977 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:24.978 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:24.978 DEBUG:boto:path=/jdpiak-71/?prefix=g 2022-09-12T19:44:24.979 DEBUG:boto:auth_path=/jdpiak-71/?prefix=g 2022-09-12T19:44:24.980 DEBUG:boto:Method: GET 2022-09-12T19:44:24.980 DEBUG:boto:Path: /jdpiak-71/?prefix=g 2022-09-12T19:44:24.981 DEBUG:boto:Data: 2022-09-12T19:44:24.982 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.982 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:24.983 DEBUG:boto:Port: 8000 2022-09-12T19:44:24.984 DEBUG:boto:Params: {} 2022-09-12T19:44:24.985 DEBUG:boto:Token: None 2022-09-12T19:44:24.985 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/ 2022-09-12T19:44:24.986 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UUH86dsZSKOYz6cLTgXXf5KuVlo= 2022-09-12T19:44:24.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UUH86dsZSKOYz6cLTgXXf5KuVlo=', 'Content-Length': '0'} 2022-09-12T19:44:24.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da1e714320c5b83c-00631f8c18-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:24.991 DEBUG:boto:b'jdpiak-71g1000falseg2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:24.992 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:24.992 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:24.993 DEBUG:boto:Method: HEAD 2022-09-12T19:44:24.994 DEBUG:boto:Path: /jdpiak-71/g 2022-09-12T19:44:24.994 DEBUG:boto:Data: 2022-09-12T19:44:24.995 DEBUG:boto:Headers: {} 2022-09-12T19:44:24.996 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:24.996 DEBUG:boto:Port: 8001 2022-09-12T19:44:24.997 DEBUG:boto:Params: {} 2022-09-12T19:44:24.997 DEBUG:boto:Token: None 2022-09-12T19:44:24.998 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:24 GMT /jdpiak-71/g 2022-09-12T19:44:24.999 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pLr+mVdh7yz0ntgU8VMkF4ocXXo= 2022-09-12T19:44:24.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pLr+mVdh7yz0ntgU8VMkF4ocXXo=', 'Content-Length': '0'} 2022-09-12T19:44:25.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eefbf74e84666f76-00631f8c19-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.003 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:25.003 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:25.004 DEBUG:boto:Method: HEAD 2022-09-12T19:44:25.005 DEBUG:boto:Path: /jdpiak-71/g 2022-09-12T19:44:25.005 DEBUG:boto:Data: 2022-09-12T19:44:25.006 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.007 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:25.007 DEBUG:boto:Port: 8000 2022-09-12T19:44:25.008 DEBUG:boto:Params: {} 2022-09-12T19:44:25.008 DEBUG:boto:Token: None 2022-09-12T19:44:25.009 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/g 2022-09-12T19:44:25.010 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5vU8jT06W3TXs8m5x+Q9NwGhCWw= 2022-09-12T19:44:25.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5vU8jT06W3TXs8m5x+Q9NwGhCWw=', 'Content-Length': '0'} 2022-09-12T19:44:25.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f4a852c2db2b34c-00631f8c19-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:44:25.016 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:25.016 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:25.017 DEBUG:boto:Method: GET 2022-09-12T19:44:25.018 DEBUG:boto:Path: /jdpiak-71/g 2022-09-12T19:44:25.019 DEBUG:boto:Data: 2022-09-12T19:44:25.019 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.020 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:25.020 DEBUG:boto:Port: 8001 2022-09-12T19:44:25.021 DEBUG:boto:Params: {} 2022-09-12T19:44:25.021 DEBUG:boto:Token: None 2022-09-12T19:44:25.022 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/g 2022-09-12T19:44:25.024 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WCh+jOKyUWfMIg9A669wAciY5SY= 2022-09-12T19:44:25.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WCh+jOKyUWfMIg9A669wAciY5SY=', 'Content-Length': '0'} 2022-09-12T19:44:25.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000509111ad85fc3de3-00631f8c19-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.038 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:25.042 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:25.043 DEBUG:boto:Method: GET 2022-09-12T19:44:25.044 DEBUG:boto:Path: /jdpiak-71/g 2022-09-12T19:44:25.046 DEBUG:boto:Data: 2022-09-12T19:44:25.048 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.049 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:25.051 DEBUG:boto:Port: 8000 2022-09-12T19:44:25.053 DEBUG:boto:Params: {} 2022-09-12T19:44:25.054 DEBUG:boto:Token: None 2022-09-12T19:44:25.055 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/g 2022-09-12T19:44:25.056 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WCh+jOKyUWfMIg9A669wAciY5SY= 2022-09-12T19:44:25.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WCh+jOKyUWfMIg9A669wAciY5SY=', 'Content-Length': '0'} 2022-09-12T19:44:25.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020d205caf5acf0fe-00631f8c19-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.069 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:25.071 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:25.072 DEBUG:boto:path=/jdpiak-71/?prefix=g 2022-09-12T19:44:25.072 DEBUG:boto:auth_path=/jdpiak-71/?prefix=g 2022-09-12T19:44:25.073 DEBUG:boto:Method: GET 2022-09-12T19:44:25.073 DEBUG:boto:Path: /jdpiak-71/?prefix=g 2022-09-12T19:44:25.074 DEBUG:boto:Data: 2022-09-12T19:44:25.075 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.075 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:25.076 DEBUG:boto:Port: 8001 2022-09-12T19:44:25.076 DEBUG:boto:Params: {} 2022-09-12T19:44:25.077 DEBUG:boto:Token: None 2022-09-12T19:44:25.078 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/ 2022-09-12T19:44:25.079 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h1YvV+wDNgQtxx6anMPU0H+fpwc= 2022-09-12T19:44:25.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h1YvV+wDNgQtxx6anMPU0H+fpwc=', 'Content-Length': '0'} 2022-09-12T19:44:25.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000564734f660b0d1d9-00631f8c19-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.086 DEBUG:boto:b'jdpiak-71g1000falseg2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:25.087 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:25.088 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:25.089 DEBUG:boto:path=/jdpiak-71/?prefix=g 2022-09-12T19:44:25.092 DEBUG:boto:auth_path=/jdpiak-71/?prefix=g 2022-09-12T19:44:25.093 DEBUG:boto:Method: GET 2022-09-12T19:44:25.097 DEBUG:boto:Path: /jdpiak-71/?prefix=g 2022-09-12T19:44:25.099 DEBUG:boto:Data: 2022-09-12T19:44:25.100 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.103 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:25.104 DEBUG:boto:Port: 8000 2022-09-12T19:44:25.106 DEBUG:boto:Params: {} 2022-09-12T19:44:25.107 DEBUG:boto:Token: None 2022-09-12T19:44:25.109 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/ 2022-09-12T19:44:25.110 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h1YvV+wDNgQtxx6anMPU0H+fpwc= 2022-09-12T19:44:25.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h1YvV+wDNgQtxx6anMPU0H+fpwc=', 'Content-Length': '0'} 2022-09-12T19:44:25.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc83830a8a15b336-00631f8c19-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.126 DEBUG:boto:b'jdpiak-71g1000falseg2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:25.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:44:25.131 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:25.132 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:25.132 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:25.133 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:25.134 DEBUG:boto:Method: GET 2022-09-12T19:44:25.135 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:25.136 DEBUG:boto:Data: 2022-09-12T19:44:25.137 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.138 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:25.139 DEBUG:boto:Port: 8001 2022-09-12T19:44:25.140 DEBUG:boto:Params: {} 2022-09-12T19:44:25.141 DEBUG:boto:Token: None 2022-09-12T19:44:25.142 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:25.143 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bXt7VoenSXkg/Y+UGluMXAyFjQE= 2022-09-12T19:44:25.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bXt7VoenSXkg/Y+UGluMXAyFjQE=', 'Content-Length': '0'} 2022-09-12T19:44:25.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000421545917cc0daf0-00631f8c19-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.151 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:25.153 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:25.154 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:25.155 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:25.156 DEBUG:boto:Method: GET 2022-09-12T19:44:25.157 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:25.158 DEBUG:boto:Data: 2022-09-12T19:44:25.159 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.160 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:25.162 DEBUG:boto:Port: 8000 2022-09-12T19:44:25.163 DEBUG:boto:Params: {} 2022-09-12T19:44:25.164 DEBUG:boto:Token: None 2022-09-12T19:44:25.165 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:25.166 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bXt7VoenSXkg/Y+UGluMXAyFjQE= 2022-09-12T19:44:25.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bXt7VoenSXkg/Y+UGluMXAyFjQE=', 'Content-Length': '0'} 2022-09-12T19:44:25.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024182a35cc963c48-00631f8c19-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.173 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:25.174 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:25.175 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:25.176 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:25.177 DEBUG:boto:Method: HEAD 2022-09-12T19:44:25.178 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:25.180 DEBUG:boto:Data: 2022-09-12T19:44:25.180 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.181 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:25.182 DEBUG:boto:Port: 8001 2022-09-12T19:44:25.183 DEBUG:boto:Params: {} 2022-09-12T19:44:25.185 DEBUG:boto:Token: None 2022-09-12T19:44:25.186 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:25.187 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R72DSVYINlglps0ruVTGH30NuMM= 2022-09-12T19:44:25.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R72DSVYINlglps0ruVTGH30NuMM=', 'Content-Length': '0'} 2022-09-12T19:44:25.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c2b7f3566ac8b1a-00631f8c19-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.192 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:25.193 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:25.194 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:25.194 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:25.195 DEBUG:boto:Method: HEAD 2022-09-12T19:44:25.196 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:25.197 DEBUG:boto:Data: 2022-09-12T19:44:25.198 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.199 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:25.200 DEBUG:boto:Port: 8000 2022-09-12T19:44:25.201 DEBUG:boto:Params: {} 2022-09-12T19:44:25.202 DEBUG:boto:Token: None 2022-09-12T19:44:25.203 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:25.204 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R72DSVYINlglps0ruVTGH30NuMM= 2022-09-12T19:44:25.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R72DSVYINlglps0ruVTGH30NuMM=', 'Content-Length': '0'} 2022-09-12T19:44:25.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f05bd1007688a27-00631f8c19-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:44:25.210 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:25.211 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:25.212 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:25.212 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:25.213 DEBUG:boto:Method: GET 2022-09-12T19:44:25.214 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:25.215 DEBUG:boto:Data: 2022-09-12T19:44:25.215 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.216 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:25.216 DEBUG:boto:Port: 8001 2022-09-12T19:44:25.217 DEBUG:boto:Params: {} 2022-09-12T19:44:25.218 DEBUG:boto:Token: None 2022-09-12T19:44:25.218 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:25.219 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bXt7VoenSXkg/Y+UGluMXAyFjQE= 2022-09-12T19:44:25.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bXt7VoenSXkg/Y+UGluMXAyFjQE=', 'Content-Length': '0'} 2022-09-12T19:44:25.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b97ebb0cf85c8437-00631f8c19-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.270 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:25.271 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:25.271 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:25.272 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:25.273 DEBUG:boto:Method: GET 2022-09-12T19:44:25.273 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:25.274 DEBUG:boto:Data: 2022-09-12T19:44:25.275 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.275 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:25.276 DEBUG:boto:Port: 8000 2022-09-12T19:44:25.276 DEBUG:boto:Params: {} 2022-09-12T19:44:25.277 DEBUG:boto:Token: None 2022-09-12T19:44:25.277 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:25.278 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bXt7VoenSXkg/Y+UGluMXAyFjQE= 2022-09-12T19:44:25.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bXt7VoenSXkg/Y+UGluMXAyFjQE=', 'Content-Length': '0'} 2022-09-12T19:44:25.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000832855938d758936-00631f8c19-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.282 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:25.282 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:25.283 DEBUG:boto:path=/jdpiak-71/?prefix=h 2022-09-12T19:44:25.284 DEBUG:boto:auth_path=/jdpiak-71/?prefix=h 2022-09-12T19:44:25.284 DEBUG:boto:Method: GET 2022-09-12T19:44:25.285 DEBUG:boto:Path: /jdpiak-71/?prefix=h 2022-09-12T19:44:25.285 DEBUG:boto:Data: 2022-09-12T19:44:25.286 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.287 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:25.287 DEBUG:boto:Port: 8001 2022-09-12T19:44:25.288 DEBUG:boto:Params: {} 2022-09-12T19:44:25.288 DEBUG:boto:Token: None 2022-09-12T19:44:25.289 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/ 2022-09-12T19:44:25.289 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h1YvV+wDNgQtxx6anMPU0H+fpwc= 2022-09-12T19:44:25.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h1YvV+wDNgQtxx6anMPU0H+fpwc=', 'Content-Length': '0'} 2022-09-12T19:44:25.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012479199e83238ea-00631f8c19-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.294 DEBUG:boto:b'jdpiak-71h1000falseh2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:25.297 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:25.298 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:25.299 DEBUG:boto:path=/jdpiak-71/?prefix=h 2022-09-12T19:44:25.300 DEBUG:boto:auth_path=/jdpiak-71/?prefix=h 2022-09-12T19:44:25.301 DEBUG:boto:Method: GET 2022-09-12T19:44:25.302 DEBUG:boto:Path: /jdpiak-71/?prefix=h 2022-09-12T19:44:25.303 DEBUG:boto:Data: 2022-09-12T19:44:25.304 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.305 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:25.306 DEBUG:boto:Port: 8000 2022-09-12T19:44:25.307 DEBUG:boto:Params: {} 2022-09-12T19:44:25.308 DEBUG:boto:Token: None 2022-09-12T19:44:25.309 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/ 2022-09-12T19:44:25.310 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h1YvV+wDNgQtxx6anMPU0H+fpwc= 2022-09-12T19:44:25.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h1YvV+wDNgQtxx6anMPU0H+fpwc=', 'Content-Length': '0'} 2022-09-12T19:44:25.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096ed5d3b9754cc0d-00631f8c19-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.316 DEBUG:boto:b'jdpiak-71h1000falseh2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:25.317 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:25.318 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:25.319 DEBUG:boto:Method: HEAD 2022-09-12T19:44:25.320 DEBUG:boto:Path: /jdpiak-71/h 2022-09-12T19:44:25.321 DEBUG:boto:Data: 2022-09-12T19:44:25.321 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.322 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:25.322 DEBUG:boto:Port: 8001 2022-09-12T19:44:25.323 DEBUG:boto:Params: {} 2022-09-12T19:44:25.323 DEBUG:boto:Token: None 2022-09-12T19:44:25.324 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/h 2022-09-12T19:44:25.324 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tcy+EEkvPtUILKeUExgRorywEJM= 2022-09-12T19:44:25.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tcy+EEkvPtUILKeUExgRorywEJM=', 'Content-Length': '0'} 2022-09-12T19:44:25.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019b20a2d08ac5403-00631f8c19-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.331 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:25.332 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:25.333 DEBUG:boto:Method: HEAD 2022-09-12T19:44:25.334 DEBUG:boto:Path: /jdpiak-71/h 2022-09-12T19:44:25.335 DEBUG:boto:Data: 2022-09-12T19:44:25.335 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.336 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:25.336 DEBUG:boto:Port: 8000 2022-09-12T19:44:25.336 DEBUG:boto:Params: {} 2022-09-12T19:44:25.337 DEBUG:boto:Token: None 2022-09-12T19:44:25.338 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/h 2022-09-12T19:44:25.339 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tcy+EEkvPtUILKeUExgRorywEJM= 2022-09-12T19:44:25.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tcy+EEkvPtUILKeUExgRorywEJM=', 'Content-Length': '0'} 2022-09-12T19:44:25.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027c2e1353793e466-00631f8c19-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:44:25.344 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:25.344 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:25.345 DEBUG:boto:Method: GET 2022-09-12T19:44:25.346 DEBUG:boto:Path: /jdpiak-71/h 2022-09-12T19:44:25.346 DEBUG:boto:Data: 2022-09-12T19:44:25.347 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.348 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:25.349 DEBUG:boto:Port: 8001 2022-09-12T19:44:25.351 DEBUG:boto:Params: {} 2022-09-12T19:44:25.352 DEBUG:boto:Token: None 2022-09-12T19:44:25.353 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/h 2022-09-12T19:44:25.355 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P+haFC+/edKFILxzA0xcMoq6UP4= 2022-09-12T19:44:25.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P+haFC+/edKFILxzA0xcMoq6UP4=', 'Content-Length': '0'} 2022-09-12T19:44:25.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ae55aeda3d576a1-00631f8c19-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.360 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:25.362 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:25.363 DEBUG:boto:Method: GET 2022-09-12T19:44:25.364 DEBUG:boto:Path: /jdpiak-71/h 2022-09-12T19:44:25.365 DEBUG:boto:Data: 2022-09-12T19:44:25.366 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.367 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:25.368 DEBUG:boto:Port: 8000 2022-09-12T19:44:25.369 DEBUG:boto:Params: {} 2022-09-12T19:44:25.370 DEBUG:boto:Token: None 2022-09-12T19:44:25.371 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/h 2022-09-12T19:44:25.372 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P+haFC+/edKFILxzA0xcMoq6UP4= 2022-09-12T19:44:25.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P+haFC+/edKFILxzA0xcMoq6UP4=', 'Content-Length': '0'} 2022-09-12T19:44:25.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c1fde01cfd180aa-00631f8c19-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.377 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:25.378 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:25.378 DEBUG:boto:path=/jdpiak-71/?prefix=h 2022-09-12T19:44:25.379 DEBUG:boto:auth_path=/jdpiak-71/?prefix=h 2022-09-12T19:44:25.379 DEBUG:boto:Method: GET 2022-09-12T19:44:25.380 DEBUG:boto:Path: /jdpiak-71/?prefix=h 2022-09-12T19:44:25.381 DEBUG:boto:Data: 2022-09-12T19:44:25.381 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.382 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:25.383 DEBUG:boto:Port: 8001 2022-09-12T19:44:25.383 DEBUG:boto:Params: {} 2022-09-12T19:44:25.384 DEBUG:boto:Token: None 2022-09-12T19:44:25.385 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/ 2022-09-12T19:44:25.385 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h1YvV+wDNgQtxx6anMPU0H+fpwc= 2022-09-12T19:44:25.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h1YvV+wDNgQtxx6anMPU0H+fpwc=', 'Content-Length': '0'} 2022-09-12T19:44:25.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090e320981b579418-00631f8c19-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.389 DEBUG:boto:b'jdpiak-71h1000falseh2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:25.390 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:25.391 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:25.392 DEBUG:boto:path=/jdpiak-71/?prefix=h 2022-09-12T19:44:25.392 DEBUG:boto:auth_path=/jdpiak-71/?prefix=h 2022-09-12T19:44:25.393 DEBUG:boto:Method: GET 2022-09-12T19:44:25.393 DEBUG:boto:Path: /jdpiak-71/?prefix=h 2022-09-12T19:44:25.394 DEBUG:boto:Data: 2022-09-12T19:44:25.394 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.395 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:25.396 DEBUG:boto:Port: 8000 2022-09-12T19:44:25.396 DEBUG:boto:Params: {} 2022-09-12T19:44:25.397 DEBUG:boto:Token: None 2022-09-12T19:44:25.398 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/ 2022-09-12T19:44:25.398 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h1YvV+wDNgQtxx6anMPU0H+fpwc= 2022-09-12T19:44:25.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h1YvV+wDNgQtxx6anMPU0H+fpwc=', 'Content-Length': '0'} 2022-09-12T19:44:25.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ff4bad11857a7e6-00631f8c19-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.403 DEBUG:boto:b'jdpiak-71h1000falseh2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:25.404 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-71 zones={test-zone3, test-zone1} 2022-09-12T19:44:25.405 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-71 zones={test-zone3, test-zone2} 2022-09-12T19:44:25.406 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:25.407 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:25.408 DEBUG:boto:Method: HEAD 2022-09-12T19:44:25.409 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:44:25.409 DEBUG:boto:Data: 2022-09-12T19:44:25.410 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.411 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:25.412 DEBUG:boto:Port: 8001 2022-09-12T19:44:25.413 DEBUG:boto:Params: {} 2022-09-12T19:44:25.414 DEBUG:boto:Token: None 2022-09-12T19:44:25.415 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/ 2022-09-12T19:44:25.416 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zcjsh6HDf6Zsw5zA2MHb/OZYBug= 2022-09-12T19:44:25.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zcjsh6HDf6Zsw5zA2MHb/OZYBug=', 'Content-Length': '0'} 2022-09-12T19:44:25.419 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000049012aeb334dfa68-00631f8c19-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.420 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:25.420 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:25.421 DEBUG:boto:Method: HEAD 2022-09-12T19:44:25.422 DEBUG:boto:Path: /jdpiak-71/ 2022-09-12T19:44:25.423 DEBUG:boto:Data: 2022-09-12T19:44:25.424 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.425 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:25.425 DEBUG:boto:Port: 8000 2022-09-12T19:44:25.426 DEBUG:boto:Params: {} 2022-09-12T19:44:25.426 DEBUG:boto:Token: None 2022-09-12T19:44:25.427 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/ 2022-09-12T19:44:25.428 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zcjsh6HDf6Zsw5zA2MHb/OZYBug= 2022-09-12T19:44:25.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zcjsh6HDf6Zsw5zA2MHb/OZYBug=', 'Content-Length': '0'} 2022-09-12T19:44:25.433 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f11d5f93ba7ff25-00631f8c19-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.434 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:44:25.435 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:25.437 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:25.438 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:25.439 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:25.441 DEBUG:boto:Method: GET 2022-09-12T19:44:25.442 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:25.443 DEBUG:boto:Data: 2022-09-12T19:44:25.445 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.446 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:25.446 DEBUG:boto:Port: 8001 2022-09-12T19:44:25.447 DEBUG:boto:Params: {} 2022-09-12T19:44:25.448 DEBUG:boto:Token: None 2022-09-12T19:44:25.448 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/?versions 2022-09-12T19:44:25.449 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:so9G+PK72bQ+ntsMIXoniOHuqFc= 2022-09-12T19:44:25.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:so9G+PK72bQ+ntsMIXoniOHuqFc=', 'Content-Length': '0'} 2022-09-12T19:44:25.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000546ba8fb7e9a30a-00631f8c19-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.455 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:25.457 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:44:25.458 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:44:25.459 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:44:25.459 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:44:25.460 DEBUG:tasks.rgw_multi.zone_rados:o=e 2022-09-12T19:44:25.461 DEBUG:tasks.rgw_multi.zone_rados:o=f 2022-09-12T19:44:25.462 DEBUG:tasks.rgw_multi.zone_rados:o=g 2022-09-12T19:44:25.462 DEBUG:tasks.rgw_multi.zone_rados:o=h 2022-09-12T19:44:25.463 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:44:25.464 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:25.465 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:25.465 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:25.466 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:25.467 DEBUG:boto:Method: GET 2022-09-12T19:44:25.467 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:25.468 DEBUG:boto:Data: 2022-09-12T19:44:25.468 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.469 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:25.470 DEBUG:boto:Port: 8000 2022-09-12T19:44:25.470 DEBUG:boto:Params: {} 2022-09-12T19:44:25.471 DEBUG:boto:Token: None 2022-09-12T19:44:25.472 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/?versions 2022-09-12T19:44:25.472 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:so9G+PK72bQ+ntsMIXoniOHuqFc= 2022-09-12T19:44:25.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:so9G+PK72bQ+ntsMIXoniOHuqFc=', 'Content-Length': '0'} 2022-09-12T19:44:25.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055464d0a9fc00299-00631f8c19-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.478 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:25.480 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:44:25.481 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:44:25.481 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:44:25.482 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:44:25.483 DEBUG:tasks.rgw_multi.zone_rados:o=e 2022-09-12T19:44:25.484 DEBUG:tasks.rgw_multi.zone_rados:o=f 2022-09-12T19:44:25.485 DEBUG:tasks.rgw_multi.zone_rados:o=g 2022-09-12T19:44:25.486 DEBUG:tasks.rgw_multi.zone_rados:o=h 2022-09-12T19:44:25.487 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:25.488 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:25.489 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:25.490 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:25.490 DEBUG:boto:Method: GET 2022-09-12T19:44:25.491 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:25.492 DEBUG:boto:Data: 2022-09-12T19:44:25.493 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.494 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:25.495 DEBUG:boto:Port: 8001 2022-09-12T19:44:25.496 DEBUG:boto:Params: {} 2022-09-12T19:44:25.497 DEBUG:boto:Token: None 2022-09-12T19:44:25.498 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/?versions 2022-09-12T19:44:25.500 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:so9G+PK72bQ+ntsMIXoniOHuqFc= 2022-09-12T19:44:25.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:so9G+PK72bQ+ntsMIXoniOHuqFc=', 'Content-Length': '0'} 2022-09-12T19:44:25.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e678d766754413e-00631f8c19-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.507 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:25.508 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:25.509 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:25.510 DEBUG:boto:path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:25.510 DEBUG:boto:auth_path=/jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:25.511 DEBUG:boto:Method: GET 2022-09-12T19:44:25.512 DEBUG:boto:Path: /jdpiak-71/?versions&max-keys=999 2022-09-12T19:44:25.513 DEBUG:boto:Data: 2022-09-12T19:44:25.514 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.515 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:25.516 DEBUG:boto:Port: 8000 2022-09-12T19:44:25.517 DEBUG:boto:Params: {} 2022-09-12T19:44:25.518 DEBUG:boto:Token: None 2022-09-12T19:44:25.519 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/?versions 2022-09-12T19:44:25.520 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:so9G+PK72bQ+ntsMIXoniOHuqFc= 2022-09-12T19:44:25.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:so9G+PK72bQ+ntsMIXoniOHuqFc=', 'Content-Length': '0'} 2022-09-12T19:44:25.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000681651fd0269dd24-00631f8c19-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.524 DEBUG:boto:b'jdpiak-71999falseanulltrue2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:25.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:44:25.528 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:25.529 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:25.529 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:25.530 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:25.531 DEBUG:boto:Method: GET 2022-09-12T19:44:25.532 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:44:25.533 DEBUG:boto:Data: 2022-09-12T19:44:25.534 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.535 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:25.536 DEBUG:boto:Port: 8001 2022-09-12T19:44:25.537 DEBUG:boto:Params: {} 2022-09-12T19:44:25.538 DEBUG:boto:Token: None 2022-09-12T19:44:25.539 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:44:25.540 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4v+QDufTKeQHQxWwoy1jfiogmGA= 2022-09-12T19:44:25.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4v+QDufTKeQHQxWwoy1jfiogmGA=', 'Content-Length': '0'} 2022-09-12T19:44:25.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d164075736af0b6a-00631f8c19-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.544 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:25.545 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:25.545 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:25.546 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:25.547 DEBUG:boto:Method: GET 2022-09-12T19:44:25.547 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:44:25.548 DEBUG:boto:Data: 2022-09-12T19:44:25.548 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.549 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:25.550 DEBUG:boto:Port: 8000 2022-09-12T19:44:25.550 DEBUG:boto:Params: {} 2022-09-12T19:44:25.552 DEBUG:boto:Token: None 2022-09-12T19:44:25.554 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:44:25.555 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4v+QDufTKeQHQxWwoy1jfiogmGA= 2022-09-12T19:44:25.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4v+QDufTKeQHQxWwoy1jfiogmGA=', 'Content-Length': '0'} 2022-09-12T19:44:25.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ddab7a6ad35028d-00631f8c19-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.561 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:25.562 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:25.563 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:25.564 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:25.564 DEBUG:boto:Method: HEAD 2022-09-12T19:44:25.565 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:44:25.565 DEBUG:boto:Data: 2022-09-12T19:44:25.566 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.566 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:25.567 DEBUG:boto:Port: 8001 2022-09-12T19:44:25.567 DEBUG:boto:Params: {} 2022-09-12T19:44:25.568 DEBUG:boto:Token: None 2022-09-12T19:44:25.569 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:44:25.569 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RmlzkuYmEiKRuUBDTF1harlX0/Y= 2022-09-12T19:44:25.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RmlzkuYmEiKRuUBDTF1harlX0/Y=', 'Content-Length': '0'} 2022-09-12T19:44:25.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ddddbc609992fb3-00631f8c19-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.573 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:25.574 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:25.575 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:25.575 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:25.576 DEBUG:boto:Method: HEAD 2022-09-12T19:44:25.576 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:44:25.577 DEBUG:boto:Data: 2022-09-12T19:44:25.578 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.579 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:25.579 DEBUG:boto:Port: 8000 2022-09-12T19:44:25.580 DEBUG:boto:Params: {} 2022-09-12T19:44:25.581 DEBUG:boto:Token: None 2022-09-12T19:44:25.582 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:44:25.582 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RmlzkuYmEiKRuUBDTF1harlX0/Y= 2022-09-12T19:44:25.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RmlzkuYmEiKRuUBDTF1harlX0/Y=', 'Content-Length': '0'} 2022-09-12T19:44:25.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006868929cb1422a13-00631f8c19-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:44:25.587 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:25.587 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:25.588 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:25.589 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:25.589 DEBUG:boto:Method: GET 2022-09-12T19:44:25.590 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:44:25.592 DEBUG:boto:Data: 2022-09-12T19:44:25.593 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.593 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:25.594 DEBUG:boto:Port: 8001 2022-09-12T19:44:25.595 DEBUG:boto:Params: {} 2022-09-12T19:44:25.597 DEBUG:boto:Token: None 2022-09-12T19:44:25.598 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:44:25.599 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4v+QDufTKeQHQxWwoy1jfiogmGA= 2022-09-12T19:44:25.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4v+QDufTKeQHQxWwoy1jfiogmGA=', 'Content-Length': '0'} 2022-09-12T19:44:25.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4545b35cb11c5c5-00631f8c19-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.603 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:25.604 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:25.604 DEBUG:boto:path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:25.605 DEBUG:boto:auth_path=/jdpiak-71/a?versionId=null 2022-09-12T19:44:25.606 DEBUG:boto:Method: GET 2022-09-12T19:44:25.607 DEBUG:boto:Path: /jdpiak-71/a?versionId=null 2022-09-12T19:44:25.607 DEBUG:boto:Data: 2022-09-12T19:44:25.608 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.609 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:25.610 DEBUG:boto:Port: 8000 2022-09-12T19:44:25.611 DEBUG:boto:Params: {} 2022-09-12T19:44:25.611 DEBUG:boto:Token: None 2022-09-12T19:44:25.612 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/a?versionId=null 2022-09-12T19:44:25.613 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4v+QDufTKeQHQxWwoy1jfiogmGA= 2022-09-12T19:44:25.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4v+QDufTKeQHQxWwoy1jfiogmGA=', 'Content-Length': '0'} 2022-09-12T19:44:25.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045fd7d5d726e63ea-00631f8c19-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.617 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:25.617 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:25.618 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:44:25.618 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:44:25.619 DEBUG:boto:Method: GET 2022-09-12T19:44:25.619 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:44:25.620 DEBUG:boto:Data: 2022-09-12T19:44:25.620 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.621 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:25.621 DEBUG:boto:Port: 8001 2022-09-12T19:44:25.622 DEBUG:boto:Params: {} 2022-09-12T19:44:25.623 DEBUG:boto:Token: None 2022-09-12T19:44:25.623 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/ 2022-09-12T19:44:25.624 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h1YvV+wDNgQtxx6anMPU0H+fpwc= 2022-09-12T19:44:25.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h1YvV+wDNgQtxx6anMPU0H+fpwc=', 'Content-Length': '0'} 2022-09-12T19:44:25.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edd525d1d1fb2acb-00631f8c19-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.674 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:25.677 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:25.679 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:25.680 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:44:25.682 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:44:25.684 DEBUG:boto:Method: GET 2022-09-12T19:44:25.685 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:44:25.687 DEBUG:boto:Data: 2022-09-12T19:44:25.689 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.690 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:25.692 DEBUG:boto:Port: 8000 2022-09-12T19:44:25.693 DEBUG:boto:Params: {} 2022-09-12T19:44:25.694 DEBUG:boto:Token: None 2022-09-12T19:44:25.695 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/ 2022-09-12T19:44:25.697 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h1YvV+wDNgQtxx6anMPU0H+fpwc= 2022-09-12T19:44:25.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h1YvV+wDNgQtxx6anMPU0H+fpwc=', 'Content-Length': '0'} 2022-09-12T19:44:25.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6952f823e729dc2-00631f8c19-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.704 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:25.706 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:25.707 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:25.709 DEBUG:boto:Method: HEAD 2022-09-12T19:44:25.709 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:44:25.711 DEBUG:boto:Data: 2022-09-12T19:44:25.712 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.712 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:25.714 DEBUG:boto:Port: 8001 2022-09-12T19:44:25.715 DEBUG:boto:Params: {} 2022-09-12T19:44:25.717 DEBUG:boto:Token: None 2022-09-12T19:44:25.718 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/a 2022-09-12T19:44:25.719 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UHSuw+dJMX26sAEPhjbDUKozBbY= 2022-09-12T19:44:25.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UHSuw+dJMX26sAEPhjbDUKozBbY=', 'Content-Length': '0'} 2022-09-12T19:44:25.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3557856934bd1a4-00631f8c19-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.723 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:25.724 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:25.725 DEBUG:boto:Method: HEAD 2022-09-12T19:44:25.725 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:44:25.726 DEBUG:boto:Data: 2022-09-12T19:44:25.726 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.727 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:25.728 DEBUG:boto:Port: 8000 2022-09-12T19:44:25.728 DEBUG:boto:Params: {} 2022-09-12T19:44:25.729 DEBUG:boto:Token: None 2022-09-12T19:44:25.730 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/a 2022-09-12T19:44:25.730 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UHSuw+dJMX26sAEPhjbDUKozBbY= 2022-09-12T19:44:25.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UHSuw+dJMX26sAEPhjbDUKozBbY=', 'Content-Length': '0'} 2022-09-12T19:44:25.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003776e9ec6214600d-00631f8c19-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:44:25.735 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:25.736 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:25.737 DEBUG:boto:Method: GET 2022-09-12T19:44:25.737 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:44:25.738 DEBUG:boto:Data: 2022-09-12T19:44:25.739 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.739 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:25.740 DEBUG:boto:Port: 8001 2022-09-12T19:44:25.741 DEBUG:boto:Params: {} 2022-09-12T19:44:25.742 DEBUG:boto:Token: None 2022-09-12T19:44:25.743 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/a 2022-09-12T19:44:25.743 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H9JWqaYPkLqVXorjnBKBZBPCYhM= 2022-09-12T19:44:25.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H9JWqaYPkLqVXorjnBKBZBPCYhM=', 'Content-Length': '0'} 2022-09-12T19:44:25.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e02ceae1c936a60-00631f8c19-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.747 DEBUG:boto:path=/jdpiak-71/a 2022-09-12T19:44:25.748 DEBUG:boto:auth_path=/jdpiak-71/a 2022-09-12T19:44:25.749 DEBUG:boto:Method: GET 2022-09-12T19:44:25.750 DEBUG:boto:Path: /jdpiak-71/a 2022-09-12T19:44:25.751 DEBUG:boto:Data: 2022-09-12T19:44:25.751 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.752 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:25.753 DEBUG:boto:Port: 8000 2022-09-12T19:44:25.754 DEBUG:boto:Params: {} 2022-09-12T19:44:25.755 DEBUG:boto:Token: None 2022-09-12T19:44:25.756 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/a 2022-09-12T19:44:25.756 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H9JWqaYPkLqVXorjnBKBZBPCYhM= 2022-09-12T19:44:25.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H9JWqaYPkLqVXorjnBKBZBPCYhM=', 'Content-Length': '0'} 2022-09-12T19:44:25.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab43f4c9c7ea9b9b-00631f8c19-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.763 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:25.764 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:25.765 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:44:25.766 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:44:25.767 DEBUG:boto:Method: GET 2022-09-12T19:44:25.768 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:44:25.769 DEBUG:boto:Data: 2022-09-12T19:44:25.771 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.771 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:25.772 DEBUG:boto:Port: 8001 2022-09-12T19:44:25.772 DEBUG:boto:Params: {} 2022-09-12T19:44:25.773 DEBUG:boto:Token: None 2022-09-12T19:44:25.774 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/ 2022-09-12T19:44:25.775 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h1YvV+wDNgQtxx6anMPU0H+fpwc= 2022-09-12T19:44:25.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h1YvV+wDNgQtxx6anMPU0H+fpwc=', 'Content-Length': '0'} 2022-09-12T19:44:25.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9c2257aeb4f81df-00631f8c19-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.779 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:25.780 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:25.781 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:25.781 DEBUG:boto:path=/jdpiak-71/?prefix=a 2022-09-12T19:44:25.782 DEBUG:boto:auth_path=/jdpiak-71/?prefix=a 2022-09-12T19:44:25.782 DEBUG:boto:Method: GET 2022-09-12T19:44:25.783 DEBUG:boto:Path: /jdpiak-71/?prefix=a 2022-09-12T19:44:25.784 DEBUG:boto:Data: 2022-09-12T19:44:25.784 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.785 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:25.785 DEBUG:boto:Port: 8000 2022-09-12T19:44:25.786 DEBUG:boto:Params: {} 2022-09-12T19:44:25.786 DEBUG:boto:Token: None 2022-09-12T19:44:25.787 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/ 2022-09-12T19:44:25.788 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h1YvV+wDNgQtxx6anMPU0H+fpwc= 2022-09-12T19:44:25.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h1YvV+wDNgQtxx6anMPU0H+fpwc=', 'Content-Length': '0'} 2022-09-12T19:44:25.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb0bf564500a7baa-00631f8c19-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.792 DEBUG:boto:b'jdpiak-71a1000falsea2022-09-12T19:41:22.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:25.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:44:25.794 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:25.795 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:25.796 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:25.796 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:25.797 DEBUG:boto:Method: GET 2022-09-12T19:44:25.798 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:44:25.798 DEBUG:boto:Data: 2022-09-12T19:44:25.799 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.800 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:25.800 DEBUG:boto:Port: 8001 2022-09-12T19:44:25.801 DEBUG:boto:Params: {} 2022-09-12T19:44:25.802 DEBUG:boto:Token: None 2022-09-12T19:44:25.802 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:44:25.803 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:23Z35Cbkl+cr2ltd5REMU4sxzdM= 2022-09-12T19:44:25.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:23Z35Cbkl+cr2ltd5REMU4sxzdM=', 'Content-Length': '0'} 2022-09-12T19:44:25.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc292be9b562e1f5-00631f8c19-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.852 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:25.855 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:25.855 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:25.856 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:25.858 DEBUG:boto:Method: GET 2022-09-12T19:44:25.858 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:44:25.859 DEBUG:boto:Data: 2022-09-12T19:44:25.860 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.861 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:25.861 DEBUG:boto:Port: 8000 2022-09-12T19:44:25.862 DEBUG:boto:Params: {} 2022-09-12T19:44:25.863 DEBUG:boto:Token: None 2022-09-12T19:44:25.863 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:44:25.865 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:23Z35Cbkl+cr2ltd5REMU4sxzdM= 2022-09-12T19:44:25.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:23Z35Cbkl+cr2ltd5REMU4sxzdM=', 'Content-Length': '0'} 2022-09-12T19:44:25.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027970770ba7763ec-00631f8c19-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.871 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:25.872 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:25.873 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:25.873 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:25.874 DEBUG:boto:Method: HEAD 2022-09-12T19:44:25.874 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:44:25.875 DEBUG:boto:Data: 2022-09-12T19:44:25.876 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.876 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:25.877 DEBUG:boto:Port: 8001 2022-09-12T19:44:25.878 DEBUG:boto:Params: {} 2022-09-12T19:44:25.878 DEBUG:boto:Token: None 2022-09-12T19:44:25.879 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:44:25.880 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:buhf0seBoC8AXYNSAM1BdpoYCIc= 2022-09-12T19:44:25.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:buhf0seBoC8AXYNSAM1BdpoYCIc=', 'Content-Length': '0'} 2022-09-12T19:44:25.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062a94151a15b4380-00631f8c19-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.894 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:25.895 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:25.896 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:25.897 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:25.897 DEBUG:boto:Method: HEAD 2022-09-12T19:44:25.899 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:44:25.899 DEBUG:boto:Data: 2022-09-12T19:44:25.900 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.900 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:25.901 DEBUG:boto:Port: 8000 2022-09-12T19:44:25.902 DEBUG:boto:Params: {} 2022-09-12T19:44:25.902 DEBUG:boto:Token: None 2022-09-12T19:44:25.903 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:44:25.904 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:buhf0seBoC8AXYNSAM1BdpoYCIc= 2022-09-12T19:44:25.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:buhf0seBoC8AXYNSAM1BdpoYCIc=', 'Content-Length': '0'} 2022-09-12T19:44:25.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f14cb6888198669a-00631f8c19-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:44:25.909 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:25.911 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:25.912 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:25.914 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:25.915 DEBUG:boto:Method: GET 2022-09-12T19:44:25.916 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:44:25.918 DEBUG:boto:Data: 2022-09-12T19:44:25.918 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.920 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:25.920 DEBUG:boto:Port: 8001 2022-09-12T19:44:25.921 DEBUG:boto:Params: {} 2022-09-12T19:44:25.922 DEBUG:boto:Token: None 2022-09-12T19:44:25.923 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:44:25.924 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:23Z35Cbkl+cr2ltd5REMU4sxzdM= 2022-09-12T19:44:25.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:23Z35Cbkl+cr2ltd5REMU4sxzdM=', 'Content-Length': '0'} 2022-09-12T19:44:25.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2ed8ebca8b972e4-00631f8c19-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.929 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:25.930 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:25.931 DEBUG:boto:path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:25.932 DEBUG:boto:auth_path=/jdpiak-71/b?versionId=null 2022-09-12T19:44:25.932 DEBUG:boto:Method: GET 2022-09-12T19:44:25.933 DEBUG:boto:Path: /jdpiak-71/b?versionId=null 2022-09-12T19:44:25.934 DEBUG:boto:Data: 2022-09-12T19:44:25.936 DEBUG:boto:Headers: {} 2022-09-12T19:44:25.937 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:25.938 DEBUG:boto:Port: 8000 2022-09-12T19:44:25.940 DEBUG:boto:Params: {} 2022-09-12T19:44:25.942 DEBUG:boto:Token: None 2022-09-12T19:44:25.944 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:25 GMT /jdpiak-71/b?versionId=null 2022-09-12T19:44:25.945 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:23Z35Cbkl+cr2ltd5REMU4sxzdM= 2022-09-12T19:44:25.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:23Z35Cbkl+cr2ltd5REMU4sxzdM=', 'Content-Length': '0'} 2022-09-12T19:44:25.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b18cdbfbe74c4e27-00631f8c19-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:25.990 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:25.992 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:25.995 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:44:25.996 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:44:25.998 DEBUG:boto:Method: GET 2022-09-12T19:44:25.999 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:44:26.002 DEBUG:boto:Data: 2022-09-12T19:44:26.003 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.005 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.006 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.007 DEBUG:boto:Params: {} 2022-09-12T19:44:26.009 DEBUG:boto:Token: None 2022-09-12T19:44:26.010 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/ 2022-09-12T19:44:26.012 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw= 2022-09-12T19:44:26.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw=', 'Content-Length': '0'} 2022-09-12T19:44:26.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b92c9c075ac20aa-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.017 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:26.019 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:26.020 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:26.022 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:44:26.023 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:44:26.025 DEBUG:boto:Method: GET 2022-09-12T19:44:26.026 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:44:26.028 DEBUG:boto:Data: 2022-09-12T19:44:26.029 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.030 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.031 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.033 DEBUG:boto:Params: {} 2022-09-12T19:44:26.034 DEBUG:boto:Token: None 2022-09-12T19:44:26.036 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/ 2022-09-12T19:44:26.038 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw= 2022-09-12T19:44:26.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw=', 'Content-Length': '0'} 2022-09-12T19:44:26.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001376b9fb617fd80-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.045 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:26.046 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:26.048 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:26.049 DEBUG:boto:Method: HEAD 2022-09-12T19:44:26.051 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:44:26.052 DEBUG:boto:Data: 2022-09-12T19:44:26.053 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.055 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.056 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.057 DEBUG:boto:Params: {} 2022-09-12T19:44:26.058 DEBUG:boto:Token: None 2022-09-12T19:44:26.059 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/b 2022-09-12T19:44:26.061 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SQ/8o05X/nAgthl1UmPzIs3Evc4= 2022-09-12T19:44:26.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SQ/8o05X/nAgthl1UmPzIs3Evc4=', 'Content-Length': '0'} 2022-09-12T19:44:26.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f9aa338c439bb8b-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.070 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:26.070 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:26.071 DEBUG:boto:Method: HEAD 2022-09-12T19:44:26.072 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:44:26.073 DEBUG:boto:Data: 2022-09-12T19:44:26.074 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.075 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.076 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.077 DEBUG:boto:Params: {} 2022-09-12T19:44:26.078 DEBUG:boto:Token: None 2022-09-12T19:44:26.079 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/b 2022-09-12T19:44:26.079 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SQ/8o05X/nAgthl1UmPzIs3Evc4= 2022-09-12T19:44:26.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SQ/8o05X/nAgthl1UmPzIs3Evc4=', 'Content-Length': '0'} 2022-09-12T19:44:26.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056395079c0c1104e-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:44:26.088 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:26.089 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:26.089 DEBUG:boto:Method: GET 2022-09-12T19:44:26.090 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:44:26.090 DEBUG:boto:Data: 2022-09-12T19:44:26.091 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.092 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.092 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.093 DEBUG:boto:Params: {} 2022-09-12T19:44:26.094 DEBUG:boto:Token: None 2022-09-12T19:44:26.096 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/b 2022-09-12T19:44:26.097 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MwfcLTnQDyaLX+w83+0mYMvKAwA= 2022-09-12T19:44:26.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MwfcLTnQDyaLX+w83+0mYMvKAwA=', 'Content-Length': '0'} 2022-09-12T19:44:26.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050c1ed9c94a71e2b-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.101 DEBUG:boto:path=/jdpiak-71/b 2022-09-12T19:44:26.102 DEBUG:boto:auth_path=/jdpiak-71/b 2022-09-12T19:44:26.102 DEBUG:boto:Method: GET 2022-09-12T19:44:26.103 DEBUG:boto:Path: /jdpiak-71/b 2022-09-12T19:44:26.104 DEBUG:boto:Data: 2022-09-12T19:44:26.104 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.105 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.106 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.106 DEBUG:boto:Params: {} 2022-09-12T19:44:26.107 DEBUG:boto:Token: None 2022-09-12T19:44:26.108 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/b 2022-09-12T19:44:26.108 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MwfcLTnQDyaLX+w83+0mYMvKAwA= 2022-09-12T19:44:26.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MwfcLTnQDyaLX+w83+0mYMvKAwA=', 'Content-Length': '0'} 2022-09-12T19:44:26.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0ce38dea2ba14f4-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.114 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:26.115 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:26.116 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:44:26.116 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:44:26.117 DEBUG:boto:Method: GET 2022-09-12T19:44:26.118 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:44:26.118 DEBUG:boto:Data: 2022-09-12T19:44:26.119 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.120 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.120 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.121 DEBUG:boto:Params: {} 2022-09-12T19:44:26.122 DEBUG:boto:Token: None 2022-09-12T19:44:26.123 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/ 2022-09-12T19:44:26.123 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw= 2022-09-12T19:44:26.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw=', 'Content-Length': '0'} 2022-09-12T19:44:26.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017441f8195d0b972-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.128 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:26.129 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:26.130 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:26.130 DEBUG:boto:path=/jdpiak-71/?prefix=b 2022-09-12T19:44:26.131 DEBUG:boto:auth_path=/jdpiak-71/?prefix=b 2022-09-12T19:44:26.132 DEBUG:boto:Method: GET 2022-09-12T19:44:26.133 DEBUG:boto:Path: /jdpiak-71/?prefix=b 2022-09-12T19:44:26.133 DEBUG:boto:Data: 2022-09-12T19:44:26.134 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.135 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.136 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.136 DEBUG:boto:Params: {} 2022-09-12T19:44:26.137 DEBUG:boto:Token: None 2022-09-12T19:44:26.137 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/ 2022-09-12T19:44:26.138 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw= 2022-09-12T19:44:26.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw=', 'Content-Length': '0'} 2022-09-12T19:44:26.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a5b53765a1c2070-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.142 DEBUG:boto:b'jdpiak-71b1000falseb2022-09-12T19:41:22.899Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:26.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:44:26.144 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:26.144 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:26.145 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:26.146 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:26.146 DEBUG:boto:Method: GET 2022-09-12T19:44:26.147 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:44:26.147 DEBUG:boto:Data: 2022-09-12T19:44:26.148 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.148 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.149 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.150 DEBUG:boto:Params: {} 2022-09-12T19:44:26.150 DEBUG:boto:Token: None 2022-09-12T19:44:26.151 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:44:26.152 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6faZW4UcVLSR4+PaTXhI84k1lIk= 2022-09-12T19:44:26.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6faZW4UcVLSR4+PaTXhI84k1lIk=', 'Content-Length': '0'} 2022-09-12T19:44:26.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000632d195ebf065ec9-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.198 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:26.200 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:26.200 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:26.201 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:26.203 DEBUG:boto:Method: GET 2022-09-12T19:44:26.203 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:44:26.204 DEBUG:boto:Data: 2022-09-12T19:44:26.205 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.206 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.207 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.208 DEBUG:boto:Params: {} 2022-09-12T19:44:26.209 DEBUG:boto:Token: None 2022-09-12T19:44:26.210 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:44:26.211 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6faZW4UcVLSR4+PaTXhI84k1lIk= 2022-09-12T19:44:26.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6faZW4UcVLSR4+PaTXhI84k1lIk=', 'Content-Length': '0'} 2022-09-12T19:44:26.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000266f2592fc5bc036-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.215 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:26.216 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:26.216 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:26.217 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:26.217 DEBUG:boto:Method: HEAD 2022-09-12T19:44:26.218 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:44:26.218 DEBUG:boto:Data: 2022-09-12T19:44:26.219 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.219 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.220 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.220 DEBUG:boto:Params: {} 2022-09-12T19:44:26.221 DEBUG:boto:Token: None 2022-09-12T19:44:26.222 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:44:26.222 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fmb7hOzGbR0XUf6hwl/A7jtvrZo= 2022-09-12T19:44:26.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fmb7hOzGbR0XUf6hwl/A7jtvrZo=', 'Content-Length': '0'} 2022-09-12T19:44:26.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb04723f75c2e596-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.226 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:26.227 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:26.227 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:26.228 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:26.229 DEBUG:boto:Method: HEAD 2022-09-12T19:44:26.230 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:44:26.230 DEBUG:boto:Data: 2022-09-12T19:44:26.231 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.232 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.232 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.233 DEBUG:boto:Params: {} 2022-09-12T19:44:26.233 DEBUG:boto:Token: None 2022-09-12T19:44:26.234 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:44:26.235 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fmb7hOzGbR0XUf6hwl/A7jtvrZo= 2022-09-12T19:44:26.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fmb7hOzGbR0XUf6hwl/A7jtvrZo=', 'Content-Length': '0'} 2022-09-12T19:44:26.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dd288f25af21b06-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:44:26.241 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:26.246 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:26.247 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:26.248 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:26.249 DEBUG:boto:Method: GET 2022-09-12T19:44:26.250 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:44:26.251 DEBUG:boto:Data: 2022-09-12T19:44:26.253 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.254 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.255 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.256 DEBUG:boto:Params: {} 2022-09-12T19:44:26.258 DEBUG:boto:Token: None 2022-09-12T19:44:26.259 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:44:26.261 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6faZW4UcVLSR4+PaTXhI84k1lIk= 2022-09-12T19:44:26.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6faZW4UcVLSR4+PaTXhI84k1lIk=', 'Content-Length': '0'} 2022-09-12T19:44:26.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000615f909a8ca08848-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.267 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:26.268 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:26.270 DEBUG:boto:path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:26.271 DEBUG:boto:auth_path=/jdpiak-71/c?versionId=null 2022-09-12T19:44:26.272 DEBUG:boto:Method: GET 2022-09-12T19:44:26.273 DEBUG:boto:Path: /jdpiak-71/c?versionId=null 2022-09-12T19:44:26.274 DEBUG:boto:Data: 2022-09-12T19:44:26.275 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.276 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.278 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.279 DEBUG:boto:Params: {} 2022-09-12T19:44:26.280 DEBUG:boto:Token: None 2022-09-12T19:44:26.281 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/c?versionId=null 2022-09-12T19:44:26.282 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6faZW4UcVLSR4+PaTXhI84k1lIk= 2022-09-12T19:44:26.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6faZW4UcVLSR4+PaTXhI84k1lIk=', 'Content-Length': '0'} 2022-09-12T19:44:26.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009971df6b20410083-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.289 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:26.290 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:26.291 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:44:26.292 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:44:26.293 DEBUG:boto:Method: GET 2022-09-12T19:44:26.294 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:44:26.295 DEBUG:boto:Data: 2022-09-12T19:44:26.296 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.297 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.298 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.299 DEBUG:boto:Params: {} 2022-09-12T19:44:26.300 DEBUG:boto:Token: None 2022-09-12T19:44:26.301 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/ 2022-09-12T19:44:26.303 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw= 2022-09-12T19:44:26.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw=', 'Content-Length': '0'} 2022-09-12T19:44:26.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f04eaebbba476c0-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.307 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:26.309 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:26.309 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:26.310 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:44:26.311 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:44:26.311 DEBUG:boto:Method: GET 2022-09-12T19:44:26.312 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:44:26.313 DEBUG:boto:Data: 2022-09-12T19:44:26.313 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.314 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.315 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.316 DEBUG:boto:Params: {} 2022-09-12T19:44:26.317 DEBUG:boto:Token: None 2022-09-12T19:44:26.318 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/ 2022-09-12T19:44:26.318 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw= 2022-09-12T19:44:26.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw=', 'Content-Length': '0'} 2022-09-12T19:44:26.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ef2e938bf8ff6df-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.322 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:26.323 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:26.324 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:26.325 DEBUG:boto:Method: HEAD 2022-09-12T19:44:26.326 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:44:26.327 DEBUG:boto:Data: 2022-09-12T19:44:26.328 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.329 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.329 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.330 DEBUG:boto:Params: {} 2022-09-12T19:44:26.331 DEBUG:boto:Token: None 2022-09-12T19:44:26.332 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/c 2022-09-12T19:44:26.333 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PJVNrupUVBdchMaETWCnvs07XxE= 2022-09-12T19:44:26.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PJVNrupUVBdchMaETWCnvs07XxE=', 'Content-Length': '0'} 2022-09-12T19:44:26.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c74dacf75bf9c151-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.338 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:26.339 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:26.340 DEBUG:boto:Method: HEAD 2022-09-12T19:44:26.341 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:44:26.342 DEBUG:boto:Data: 2022-09-12T19:44:26.342 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.343 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.344 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.345 DEBUG:boto:Params: {} 2022-09-12T19:44:26.345 DEBUG:boto:Token: None 2022-09-12T19:44:26.346 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/c 2022-09-12T19:44:26.347 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PJVNrupUVBdchMaETWCnvs07XxE= 2022-09-12T19:44:26.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PJVNrupUVBdchMaETWCnvs07XxE=', 'Content-Length': '0'} 2022-09-12T19:44:26.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033d07dcc2aa47969-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:44:26.351 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:26.351 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:26.352 DEBUG:boto:Method: GET 2022-09-12T19:44:26.353 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:44:26.353 DEBUG:boto:Data: 2022-09-12T19:44:26.354 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.354 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.355 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.356 DEBUG:boto:Params: {} 2022-09-12T19:44:26.356 DEBUG:boto:Token: None 2022-09-12T19:44:26.357 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/c 2022-09-12T19:44:26.358 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cCxbdEPaodgKJHd8ANZzz8dU8n4= 2022-09-12T19:44:26.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cCxbdEPaodgKJHd8ANZzz8dU8n4=', 'Content-Length': '0'} 2022-09-12T19:44:26.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9ee9fd1dac5749c-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.362 DEBUG:boto:path=/jdpiak-71/c 2022-09-12T19:44:26.363 DEBUG:boto:auth_path=/jdpiak-71/c 2022-09-12T19:44:26.365 DEBUG:boto:Method: GET 2022-09-12T19:44:26.366 DEBUG:boto:Path: /jdpiak-71/c 2022-09-12T19:44:26.367 DEBUG:boto:Data: 2022-09-12T19:44:26.368 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.369 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.370 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.371 DEBUG:boto:Params: {} 2022-09-12T19:44:26.373 DEBUG:boto:Token: None 2022-09-12T19:44:26.374 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/c 2022-09-12T19:44:26.375 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cCxbdEPaodgKJHd8ANZzz8dU8n4= 2022-09-12T19:44:26.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cCxbdEPaodgKJHd8ANZzz8dU8n4=', 'Content-Length': '0'} 2022-09-12T19:44:26.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4a423c195c11080-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.382 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:26.383 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:26.384 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:44:26.385 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:44:26.386 DEBUG:boto:Method: GET 2022-09-12T19:44:26.387 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:44:26.388 DEBUG:boto:Data: 2022-09-12T19:44:26.389 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.390 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.391 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.392 DEBUG:boto:Params: {} 2022-09-12T19:44:26.394 DEBUG:boto:Token: None 2022-09-12T19:44:26.395 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/ 2022-09-12T19:44:26.396 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw= 2022-09-12T19:44:26.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw=', 'Content-Length': '0'} 2022-09-12T19:44:26.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006043a297efc23146-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.402 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:26.403 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:26.404 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:26.405 DEBUG:boto:path=/jdpiak-71/?prefix=c 2022-09-12T19:44:26.405 DEBUG:boto:auth_path=/jdpiak-71/?prefix=c 2022-09-12T19:44:26.406 DEBUG:boto:Method: GET 2022-09-12T19:44:26.407 DEBUG:boto:Path: /jdpiak-71/?prefix=c 2022-09-12T19:44:26.407 DEBUG:boto:Data: 2022-09-12T19:44:26.408 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.408 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.409 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.409 DEBUG:boto:Params: {} 2022-09-12T19:44:26.410 DEBUG:boto:Token: None 2022-09-12T19:44:26.411 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/ 2022-09-12T19:44:26.412 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw= 2022-09-12T19:44:26.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw=', 'Content-Length': '0'} 2022-09-12T19:44:26.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c601c21022d3a437-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.416 DEBUG:boto:b'jdpiak-71c1000falsec2022-09-12T19:41:22.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:26.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:44:26.418 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:26.419 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:26.420 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:26.423 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:26.424 DEBUG:boto:Method: GET 2022-09-12T19:44:26.425 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:44:26.426 DEBUG:boto:Data: 2022-09-12T19:44:26.427 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.428 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.429 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.429 DEBUG:boto:Params: {} 2022-09-12T19:44:26.430 DEBUG:boto:Token: None 2022-09-12T19:44:26.431 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:44:26.431 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R3x64CLoegacXlRE+CaB2ViSazk= 2022-09-12T19:44:26.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R3x64CLoegacXlRE+CaB2ViSazk=', 'Content-Length': '0'} 2022-09-12T19:44:26.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000441c491b29973f43-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.439 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:26.440 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:26.441 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:26.442 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:26.443 DEBUG:boto:Method: GET 2022-09-12T19:44:26.444 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:44:26.444 DEBUG:boto:Data: 2022-09-12T19:44:26.445 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.446 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.446 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.447 DEBUG:boto:Params: {} 2022-09-12T19:44:26.447 DEBUG:boto:Token: None 2022-09-12T19:44:26.448 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:44:26.449 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R3x64CLoegacXlRE+CaB2ViSazk= 2022-09-12T19:44:26.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R3x64CLoegacXlRE+CaB2ViSazk=', 'Content-Length': '0'} 2022-09-12T19:44:26.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004725b8995e5ef6cc-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.459 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:26.460 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:26.460 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:26.461 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:26.461 DEBUG:boto:Method: HEAD 2022-09-12T19:44:26.462 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:44:26.462 DEBUG:boto:Data: 2022-09-12T19:44:26.463 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.463 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.464 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.465 DEBUG:boto:Params: {} 2022-09-12T19:44:26.465 DEBUG:boto:Token: None 2022-09-12T19:44:26.466 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:44:26.467 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+xWKdE2fN1RKcSJ9Df2eP3UdF9E= 2022-09-12T19:44:26.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+xWKdE2fN1RKcSJ9Df2eP3UdF9E=', 'Content-Length': '0'} 2022-09-12T19:44:26.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000808493a0fde22a7-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.470 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:26.470 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:26.471 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:26.471 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:26.472 DEBUG:boto:Method: HEAD 2022-09-12T19:44:26.472 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:44:26.473 DEBUG:boto:Data: 2022-09-12T19:44:26.473 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.474 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.474 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.475 DEBUG:boto:Params: {} 2022-09-12T19:44:26.476 DEBUG:boto:Token: None 2022-09-12T19:44:26.476 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:44:26.477 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+xWKdE2fN1RKcSJ9Df2eP3UdF9E= 2022-09-12T19:44:26.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+xWKdE2fN1RKcSJ9Df2eP3UdF9E=', 'Content-Length': '0'} 2022-09-12T19:44:26.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef437a823135b722-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:44:26.481 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:26.482 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:26.482 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:26.483 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:26.483 DEBUG:boto:Method: GET 2022-09-12T19:44:26.484 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:44:26.484 DEBUG:boto:Data: 2022-09-12T19:44:26.485 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.486 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.486 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.487 DEBUG:boto:Params: {} 2022-09-12T19:44:26.487 DEBUG:boto:Token: None 2022-09-12T19:44:26.488 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:44:26.489 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R3x64CLoegacXlRE+CaB2ViSazk= 2022-09-12T19:44:26.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R3x64CLoegacXlRE+CaB2ViSazk=', 'Content-Length': '0'} 2022-09-12T19:44:26.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000638966f695699518-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.534 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:26.535 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:26.536 DEBUG:boto:path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:26.537 DEBUG:boto:auth_path=/jdpiak-71/d?versionId=null 2022-09-12T19:44:26.538 DEBUG:boto:Method: GET 2022-09-12T19:44:26.539 DEBUG:boto:Path: /jdpiak-71/d?versionId=null 2022-09-12T19:44:26.542 DEBUG:boto:Data: 2022-09-12T19:44:26.543 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.545 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.546 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.547 DEBUG:boto:Params: {} 2022-09-12T19:44:26.548 DEBUG:boto:Token: None 2022-09-12T19:44:26.549 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/d?versionId=null 2022-09-12T19:44:26.550 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R3x64CLoegacXlRE+CaB2ViSazk= 2022-09-12T19:44:26.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R3x64CLoegacXlRE+CaB2ViSazk=', 'Content-Length': '0'} 2022-09-12T19:44:26.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000246d52aff8ea9f89-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.556 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:26.557 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:26.559 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:44:26.560 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:44:26.561 DEBUG:boto:Method: GET 2022-09-12T19:44:26.562 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:44:26.563 DEBUG:boto:Data: 2022-09-12T19:44:26.564 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.565 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.566 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.567 DEBUG:boto:Params: {} 2022-09-12T19:44:26.569 DEBUG:boto:Token: None 2022-09-12T19:44:26.570 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/ 2022-09-12T19:44:26.571 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw= 2022-09-12T19:44:26.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw=', 'Content-Length': '0'} 2022-09-12T19:44:26.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e290ae2686f3c529-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.578 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:26.580 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:26.581 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:26.581 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:44:26.582 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:44:26.583 DEBUG:boto:Method: GET 2022-09-12T19:44:26.583 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:44:26.584 DEBUG:boto:Data: 2022-09-12T19:44:26.584 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.585 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.586 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.586 DEBUG:boto:Params: {} 2022-09-12T19:44:26.587 DEBUG:boto:Token: None 2022-09-12T19:44:26.587 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/ 2022-09-12T19:44:26.588 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw= 2022-09-12T19:44:26.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw=', 'Content-Length': '0'} 2022-09-12T19:44:26.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc96213096651deb-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.592 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:26.594 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:26.595 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:26.595 DEBUG:boto:Method: HEAD 2022-09-12T19:44:26.596 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:44:26.596 DEBUG:boto:Data: 2022-09-12T19:44:26.597 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.597 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.598 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.598 DEBUG:boto:Params: {} 2022-09-12T19:44:26.599 DEBUG:boto:Token: None 2022-09-12T19:44:26.599 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/d 2022-09-12T19:44:26.600 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0vpBWHTowbLR8jfFPCgw2+N5fuQ= 2022-09-12T19:44:26.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0vpBWHTowbLR8jfFPCgw2+N5fuQ=', 'Content-Length': '0'} 2022-09-12T19:44:26.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000988fc264aa0a483a-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.604 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:26.604 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:26.605 DEBUG:boto:Method: HEAD 2022-09-12T19:44:26.605 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:44:26.606 DEBUG:boto:Data: 2022-09-12T19:44:26.606 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.607 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.607 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.608 DEBUG:boto:Params: {} 2022-09-12T19:44:26.609 DEBUG:boto:Token: None 2022-09-12T19:44:26.609 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/d 2022-09-12T19:44:26.610 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0vpBWHTowbLR8jfFPCgw2+N5fuQ= 2022-09-12T19:44:26.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0vpBWHTowbLR8jfFPCgw2+N5fuQ=', 'Content-Length': '0'} 2022-09-12T19:44:26.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051fa2cbcb736beab-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:44:26.614 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:26.614 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:26.615 DEBUG:boto:Method: GET 2022-09-12T19:44:26.616 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:44:26.616 DEBUG:boto:Data: 2022-09-12T19:44:26.617 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.617 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.618 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.618 DEBUG:boto:Params: {} 2022-09-12T19:44:26.619 DEBUG:boto:Token: None 2022-09-12T19:44:26.620 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/d 2022-09-12T19:44:26.620 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Jw2yg/cMfFHPaW1bfF1wphNUQ3M= 2022-09-12T19:44:26.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Jw2yg/cMfFHPaW1bfF1wphNUQ3M=', 'Content-Length': '0'} 2022-09-12T19:44:26.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f890894c500da6b6-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.627 DEBUG:boto:path=/jdpiak-71/d 2022-09-12T19:44:26.628 DEBUG:boto:auth_path=/jdpiak-71/d 2022-09-12T19:44:26.628 DEBUG:boto:Method: GET 2022-09-12T19:44:26.629 DEBUG:boto:Path: /jdpiak-71/d 2022-09-12T19:44:26.629 DEBUG:boto:Data: 2022-09-12T19:44:26.630 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.630 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.631 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.632 DEBUG:boto:Params: {} 2022-09-12T19:44:26.632 DEBUG:boto:Token: None 2022-09-12T19:44:26.633 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/d 2022-09-12T19:44:26.634 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Jw2yg/cMfFHPaW1bfF1wphNUQ3M= 2022-09-12T19:44:26.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Jw2yg/cMfFHPaW1bfF1wphNUQ3M=', 'Content-Length': '0'} 2022-09-12T19:44:26.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000879edad67de95c28-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.638 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:26.638 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:26.639 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:44:26.639 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:44:26.640 DEBUG:boto:Method: GET 2022-09-12T19:44:26.641 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:44:26.641 DEBUG:boto:Data: 2022-09-12T19:44:26.642 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.642 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.643 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.644 DEBUG:boto:Params: {} 2022-09-12T19:44:26.645 DEBUG:boto:Token: None 2022-09-12T19:44:26.645 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/ 2022-09-12T19:44:26.646 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw= 2022-09-12T19:44:26.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw=', 'Content-Length': '0'} 2022-09-12T19:44:26.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e133daf4a20c214f-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.650 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:26.651 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:26.652 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:26.652 DEBUG:boto:path=/jdpiak-71/?prefix=d 2022-09-12T19:44:26.653 DEBUG:boto:auth_path=/jdpiak-71/?prefix=d 2022-09-12T19:44:26.653 DEBUG:boto:Method: GET 2022-09-12T19:44:26.654 DEBUG:boto:Path: /jdpiak-71/?prefix=d 2022-09-12T19:44:26.655 DEBUG:boto:Data: 2022-09-12T19:44:26.655 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.656 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.657 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.657 DEBUG:boto:Params: {} 2022-09-12T19:44:26.658 DEBUG:boto:Token: None 2022-09-12T19:44:26.659 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/ 2022-09-12T19:44:26.660 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw= 2022-09-12T19:44:26.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw=', 'Content-Length': '0'} 2022-09-12T19:44:26.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a5a4ea5745b21ac-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.664 DEBUG:boto:b'jdpiak-71d1000falsed2022-09-12T19:41:23.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:26.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:44:26.666 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:26.666 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:26.667 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:26.668 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:26.668 DEBUG:boto:Method: GET 2022-09-12T19:44:26.669 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:44:26.669 DEBUG:boto:Data: 2022-09-12T19:44:26.670 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.670 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.671 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.671 DEBUG:boto:Params: {} 2022-09-12T19:44:26.672 DEBUG:boto:Token: None 2022-09-12T19:44:26.672 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:44:26.673 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ru8WYFKCRcqVQC+r37cWBlB9RW8= 2022-09-12T19:44:26.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ru8WYFKCRcqVQC+r37cWBlB9RW8=', 'Content-Length': '0'} 2022-09-12T19:44:26.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d736a23b2ae9145-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.677 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:26.677 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:26.678 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:26.679 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:26.679 DEBUG:boto:Method: GET 2022-09-12T19:44:26.680 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:44:26.680 DEBUG:boto:Data: 2022-09-12T19:44:26.681 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.681 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.682 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.683 DEBUG:boto:Params: {} 2022-09-12T19:44:26.683 DEBUG:boto:Token: None 2022-09-12T19:44:26.684 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:44:26.684 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ru8WYFKCRcqVQC+r37cWBlB9RW8= 2022-09-12T19:44:26.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ru8WYFKCRcqVQC+r37cWBlB9RW8=', 'Content-Length': '0'} 2022-09-12T19:44:26.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df4b49c212854f41-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.730 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:26.731 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:26.731 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:26.732 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:26.733 DEBUG:boto:Method: HEAD 2022-09-12T19:44:26.733 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:44:26.734 DEBUG:boto:Data: 2022-09-12T19:44:26.734 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.735 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.735 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.736 DEBUG:boto:Params: {} 2022-09-12T19:44:26.737 DEBUG:boto:Token: None 2022-09-12T19:44:26.738 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:44:26.738 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vc0oPFQjH/kOIa0VfegfXaxobhM= 2022-09-12T19:44:26.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vc0oPFQjH/kOIa0VfegfXaxobhM=', 'Content-Length': '0'} 2022-09-12T19:44:26.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd976c4f6a3560da-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.743 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:26.743 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:26.744 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:26.744 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:26.745 DEBUG:boto:Method: HEAD 2022-09-12T19:44:26.745 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:44:26.746 DEBUG:boto:Data: 2022-09-12T19:44:26.746 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.747 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.747 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.748 DEBUG:boto:Params: {} 2022-09-12T19:44:26.748 DEBUG:boto:Token: None 2022-09-12T19:44:26.749 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:44:26.750 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vc0oPFQjH/kOIa0VfegfXaxobhM= 2022-09-12T19:44:26.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vc0oPFQjH/kOIa0VfegfXaxobhM=', 'Content-Length': '0'} 2022-09-12T19:44:26.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5ab44ec67ca5d85-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:44:26.754 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:26.755 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:26.755 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:26.756 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:26.756 DEBUG:boto:Method: GET 2022-09-12T19:44:26.757 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:44:26.758 DEBUG:boto:Data: 2022-09-12T19:44:26.758 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.759 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.759 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.760 DEBUG:boto:Params: {} 2022-09-12T19:44:26.760 DEBUG:boto:Token: None 2022-09-12T19:44:26.761 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:44:26.762 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ru8WYFKCRcqVQC+r37cWBlB9RW8= 2022-09-12T19:44:26.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ru8WYFKCRcqVQC+r37cWBlB9RW8=', 'Content-Length': '0'} 2022-09-12T19:44:26.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038b563983666910c-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.765 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:26.766 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:26.766 DEBUG:boto:path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:26.767 DEBUG:boto:auth_path=/jdpiak-71/e?versionId=null 2022-09-12T19:44:26.767 DEBUG:boto:Method: GET 2022-09-12T19:44:26.768 DEBUG:boto:Path: /jdpiak-71/e?versionId=null 2022-09-12T19:44:26.769 DEBUG:boto:Data: 2022-09-12T19:44:26.769 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.770 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.770 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.771 DEBUG:boto:Params: {} 2022-09-12T19:44:26.771 DEBUG:boto:Token: None 2022-09-12T19:44:26.772 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/e?versionId=null 2022-09-12T19:44:26.773 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ru8WYFKCRcqVQC+r37cWBlB9RW8= 2022-09-12T19:44:26.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ru8WYFKCRcqVQC+r37cWBlB9RW8=', 'Content-Length': '0'} 2022-09-12T19:44:26.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcfa066951914a26-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.776 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:26.777 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:26.777 DEBUG:boto:path=/jdpiak-71/?prefix=e 2022-09-12T19:44:26.778 DEBUG:boto:auth_path=/jdpiak-71/?prefix=e 2022-09-12T19:44:26.778 DEBUG:boto:Method: GET 2022-09-12T19:44:26.779 DEBUG:boto:Path: /jdpiak-71/?prefix=e 2022-09-12T19:44:26.780 DEBUG:boto:Data: 2022-09-12T19:44:26.780 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.781 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.781 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.782 DEBUG:boto:Params: {} 2022-09-12T19:44:26.782 DEBUG:boto:Token: None 2022-09-12T19:44:26.783 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/ 2022-09-12T19:44:26.783 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw= 2022-09-12T19:44:26.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw=', 'Content-Length': '0'} 2022-09-12T19:44:26.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbbd70f1d591dd6d-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.788 DEBUG:boto:b'jdpiak-71e1000falsee2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:26.789 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:26.790 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:26.790 DEBUG:boto:path=/jdpiak-71/?prefix=e 2022-09-12T19:44:26.791 DEBUG:boto:auth_path=/jdpiak-71/?prefix=e 2022-09-12T19:44:26.792 DEBUG:boto:Method: GET 2022-09-12T19:44:26.792 DEBUG:boto:Path: /jdpiak-71/?prefix=e 2022-09-12T19:44:26.793 DEBUG:boto:Data: 2022-09-12T19:44:26.794 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.794 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.795 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.796 DEBUG:boto:Params: {} 2022-09-12T19:44:26.796 DEBUG:boto:Token: None 2022-09-12T19:44:26.797 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/ 2022-09-12T19:44:26.797 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw= 2022-09-12T19:44:26.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw=', 'Content-Length': '0'} 2022-09-12T19:44:26.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003503dd8f4d7f33a8-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.802 DEBUG:boto:b'jdpiak-71e1000falsee2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:26.803 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:26.805 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:26.805 DEBUG:boto:Method: HEAD 2022-09-12T19:44:26.806 DEBUG:boto:Path: /jdpiak-71/e 2022-09-12T19:44:26.807 DEBUG:boto:Data: 2022-09-12T19:44:26.808 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.809 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.809 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.810 DEBUG:boto:Params: {} 2022-09-12T19:44:26.810 DEBUG:boto:Token: None 2022-09-12T19:44:26.811 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/e 2022-09-12T19:44:26.812 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:57GWxfJLhNCUu0fSC+N5z+I8DFo= 2022-09-12T19:44:26.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:57GWxfJLhNCUu0fSC+N5z+I8DFo=', 'Content-Length': '0'} 2022-09-12T19:44:26.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000296c75a4f98e1e7b-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.817 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:26.818 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:26.819 DEBUG:boto:Method: HEAD 2022-09-12T19:44:26.820 DEBUG:boto:Path: /jdpiak-71/e 2022-09-12T19:44:26.821 DEBUG:boto:Data: 2022-09-12T19:44:26.822 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.823 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.823 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.824 DEBUG:boto:Params: {} 2022-09-12T19:44:26.825 DEBUG:boto:Token: None 2022-09-12T19:44:26.826 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/e 2022-09-12T19:44:26.828 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:57GWxfJLhNCUu0fSC+N5z+I8DFo= 2022-09-12T19:44:26.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:57GWxfJLhNCUu0fSC+N5z+I8DFo=', 'Content-Length': '0'} 2022-09-12T19:44:26.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5c3d22d3f2bd7e1-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:44:26.833 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:26.834 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:26.835 DEBUG:boto:Method: GET 2022-09-12T19:44:26.836 DEBUG:boto:Path: /jdpiak-71/e 2022-09-12T19:44:26.837 DEBUG:boto:Data: 2022-09-12T19:44:26.838 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.840 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.840 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.841 DEBUG:boto:Params: {} 2022-09-12T19:44:26.841 DEBUG:boto:Token: None 2022-09-12T19:44:26.842 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/e 2022-09-12T19:44:26.843 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SrD5Z4moKklYS6Ekatc3zi4xF5U= 2022-09-12T19:44:26.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SrD5Z4moKklYS6Ekatc3zi4xF5U=', 'Content-Length': '0'} 2022-09-12T19:44:26.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c86a1b25ba936766-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.887 DEBUG:boto:path=/jdpiak-71/e 2022-09-12T19:44:26.888 DEBUG:boto:auth_path=/jdpiak-71/e 2022-09-12T19:44:26.889 DEBUG:boto:Method: GET 2022-09-12T19:44:26.890 DEBUG:boto:Path: /jdpiak-71/e 2022-09-12T19:44:26.891 DEBUG:boto:Data: 2022-09-12T19:44:26.892 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.893 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.894 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.895 DEBUG:boto:Params: {} 2022-09-12T19:44:26.896 DEBUG:boto:Token: None 2022-09-12T19:44:26.897 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/e 2022-09-12T19:44:26.898 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SrD5Z4moKklYS6Ekatc3zi4xF5U= 2022-09-12T19:44:26.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SrD5Z4moKklYS6Ekatc3zi4xF5U=', 'Content-Length': '0'} 2022-09-12T19:44:26.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b616e2e958bddbf6-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.905 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:26.907 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:26.908 DEBUG:boto:path=/jdpiak-71/?prefix=e 2022-09-12T19:44:26.909 DEBUG:boto:auth_path=/jdpiak-71/?prefix=e 2022-09-12T19:44:26.910 DEBUG:boto:Method: GET 2022-09-12T19:44:26.910 DEBUG:boto:Path: /jdpiak-71/?prefix=e 2022-09-12T19:44:26.911 DEBUG:boto:Data: 2022-09-12T19:44:26.912 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.913 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.915 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.916 DEBUG:boto:Params: {} 2022-09-12T19:44:26.917 DEBUG:boto:Token: None 2022-09-12T19:44:26.919 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/ 2022-09-12T19:44:26.920 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw= 2022-09-12T19:44:26.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw=', 'Content-Length': '0'} 2022-09-12T19:44:26.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a71e342bc3700fae-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.933 DEBUG:boto:b'jdpiak-71e1000falsee2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:26.935 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:26.936 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:26.937 DEBUG:boto:path=/jdpiak-71/?prefix=e 2022-09-12T19:44:26.938 DEBUG:boto:auth_path=/jdpiak-71/?prefix=e 2022-09-12T19:44:26.938 DEBUG:boto:Method: GET 2022-09-12T19:44:26.939 DEBUG:boto:Path: /jdpiak-71/?prefix=e 2022-09-12T19:44:26.940 DEBUG:boto:Data: 2022-09-12T19:44:26.941 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.942 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.943 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.944 DEBUG:boto:Params: {} 2022-09-12T19:44:26.945 DEBUG:boto:Token: None 2022-09-12T19:44:26.946 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/ 2022-09-12T19:44:26.948 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw= 2022-09-12T19:44:26.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EvBv9I/tWfbe10TfJkaHrlHrGAw=', 'Content-Length': '0'} 2022-09-12T19:44:26.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6e82e530ed59bfa-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.954 DEBUG:boto:b'jdpiak-71e1000falsee2022-09-12T19:41:59.733Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:26.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:44:26.956 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:26.957 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:26.958 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:26.959 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:26.960 DEBUG:boto:Method: GET 2022-09-12T19:44:26.961 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:26.962 DEBUG:boto:Data: 2022-09-12T19:44:26.963 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.964 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:26.965 DEBUG:boto:Port: 8001 2022-09-12T19:44:26.966 DEBUG:boto:Params: {} 2022-09-12T19:44:26.967 DEBUG:boto:Token: None 2022-09-12T19:44:26.968 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:26.969 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:U+VmV0EasATq5EAIpC/BagXOmEE= 2022-09-12T19:44:26.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:U+VmV0EasATq5EAIpC/BagXOmEE=', 'Content-Length': '0'} 2022-09-12T19:44:26.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afb894aebf73075f-00631f8c1a-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.975 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:26.976 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:26.977 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:26.978 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:26.979 DEBUG:boto:Method: GET 2022-09-12T19:44:26.980 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:26.981 DEBUG:boto:Data: 2022-09-12T19:44:26.982 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.983 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:26.984 DEBUG:boto:Port: 8000 2022-09-12T19:44:26.985 DEBUG:boto:Params: {} 2022-09-12T19:44:26.987 DEBUG:boto:Token: None 2022-09-12T19:44:26.988 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:26 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:26.989 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:U+VmV0EasATq5EAIpC/BagXOmEE= 2022-09-12T19:44:26.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:U+VmV0EasATq5EAIpC/BagXOmEE=', 'Content-Length': '0'} 2022-09-12T19:44:26.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4e87ac6a20acdd0-00631f8c1a-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:26.993 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:26.994 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:26.994 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:26.995 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:26.995 DEBUG:boto:Method: HEAD 2022-09-12T19:44:26.996 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:26.997 DEBUG:boto:Data: 2022-09-12T19:44:26.998 DEBUG:boto:Headers: {} 2022-09-12T19:44:26.999 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:27.000 DEBUG:boto:Port: 8001 2022-09-12T19:44:27.000 DEBUG:boto:Params: {} 2022-09-12T19:44:27.001 DEBUG:boto:Token: None 2022-09-12T19:44:27.002 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:27.003 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:g+mF1YlQ/Q3wAlVdXxu4TbbOFRM= 2022-09-12T19:44:27.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:g+mF1YlQ/Q3wAlVdXxu4TbbOFRM=', 'Content-Length': '0'} 2022-09-12T19:44:27.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d9812065f2feffa-00631f8c1b-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.007 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:27.007 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:27.008 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:27.008 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:27.009 DEBUG:boto:Method: HEAD 2022-09-12T19:44:27.009 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:27.010 DEBUG:boto:Data: 2022-09-12T19:44:27.010 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.011 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:27.011 DEBUG:boto:Port: 8000 2022-09-12T19:44:27.012 DEBUG:boto:Params: {} 2022-09-12T19:44:27.012 DEBUG:boto:Token: None 2022-09-12T19:44:27.013 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:27.014 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:g+mF1YlQ/Q3wAlVdXxu4TbbOFRM= 2022-09-12T19:44:27.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:g+mF1YlQ/Q3wAlVdXxu4TbbOFRM=', 'Content-Length': '0'} 2022-09-12T19:44:27.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050fe3d9483022419-00631f8c1b-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:44:27.018 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:27.018 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:27.019 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:27.019 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:27.020 DEBUG:boto:Method: GET 2022-09-12T19:44:27.021 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:27.021 DEBUG:boto:Data: 2022-09-12T19:44:27.022 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.022 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:27.023 DEBUG:boto:Port: 8001 2022-09-12T19:44:27.023 DEBUG:boto:Params: {} 2022-09-12T19:44:27.024 DEBUG:boto:Token: None 2022-09-12T19:44:27.025 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:27.025 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p9CkQrwHSGzuVhfMNmb8UxrIm7c= 2022-09-12T19:44:27.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p9CkQrwHSGzuVhfMNmb8UxrIm7c=', 'Content-Length': '0'} 2022-09-12T19:44:27.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a39ec84d8c822a6-00631f8c1b-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.032 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:27.033 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:27.034 DEBUG:boto:path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:27.035 DEBUG:boto:auth_path=/jdpiak-71/f?versionId=null 2022-09-12T19:44:27.036 DEBUG:boto:Method: GET 2022-09-12T19:44:27.037 DEBUG:boto:Path: /jdpiak-71/f?versionId=null 2022-09-12T19:44:27.038 DEBUG:boto:Data: 2022-09-12T19:44:27.039 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.040 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:27.041 DEBUG:boto:Port: 8000 2022-09-12T19:44:27.041 DEBUG:boto:Params: {} 2022-09-12T19:44:27.042 DEBUG:boto:Token: None 2022-09-12T19:44:27.043 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/f?versionId=null 2022-09-12T19:44:27.044 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p9CkQrwHSGzuVhfMNmb8UxrIm7c= 2022-09-12T19:44:27.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p9CkQrwHSGzuVhfMNmb8UxrIm7c=', 'Content-Length': '0'} 2022-09-12T19:44:27.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f6e234811e4bec2-00631f8c1b-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.049 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:27.049 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:27.050 DEBUG:boto:path=/jdpiak-71/?prefix=f 2022-09-12T19:44:27.052 DEBUG:boto:auth_path=/jdpiak-71/?prefix=f 2022-09-12T19:44:27.053 DEBUG:boto:Method: GET 2022-09-12T19:44:27.053 DEBUG:boto:Path: /jdpiak-71/?prefix=f 2022-09-12T19:44:27.054 DEBUG:boto:Data: 2022-09-12T19:44:27.054 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.055 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:27.055 DEBUG:boto:Port: 8001 2022-09-12T19:44:27.056 DEBUG:boto:Params: {} 2022-09-12T19:44:27.056 DEBUG:boto:Token: None 2022-09-12T19:44:27.057 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/ 2022-09-12T19:44:27.058 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GNc7FbEZFk/obzvOBAf77v0Fbl0= 2022-09-12T19:44:27.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GNc7FbEZFk/obzvOBAf77v0Fbl0=', 'Content-Length': '0'} 2022-09-12T19:44:27.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006efc95d355b14a5-00631f8c1b-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.065 DEBUG:boto:b'jdpiak-71f1000falsef2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:27.066 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:27.067 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:27.068 DEBUG:boto:path=/jdpiak-71/?prefix=f 2022-09-12T19:44:27.068 DEBUG:boto:auth_path=/jdpiak-71/?prefix=f 2022-09-12T19:44:27.069 DEBUG:boto:Method: GET 2022-09-12T19:44:27.069 DEBUG:boto:Path: /jdpiak-71/?prefix=f 2022-09-12T19:44:27.070 DEBUG:boto:Data: 2022-09-12T19:44:27.071 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.071 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:27.072 DEBUG:boto:Port: 8000 2022-09-12T19:44:27.072 DEBUG:boto:Params: {} 2022-09-12T19:44:27.073 DEBUG:boto:Token: None 2022-09-12T19:44:27.074 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/ 2022-09-12T19:44:27.075 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GNc7FbEZFk/obzvOBAf77v0Fbl0= 2022-09-12T19:44:27.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GNc7FbEZFk/obzvOBAf77v0Fbl0=', 'Content-Length': '0'} 2022-09-12T19:44:27.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095d12cc1d8609b4b-00631f8c1b-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.079 DEBUG:boto:b'jdpiak-71f1000falsef2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:27.080 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:27.081 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:27.082 DEBUG:boto:Method: HEAD 2022-09-12T19:44:27.082 DEBUG:boto:Path: /jdpiak-71/f 2022-09-12T19:44:27.083 DEBUG:boto:Data: 2022-09-12T19:44:27.084 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.085 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:27.086 DEBUG:boto:Port: 8001 2022-09-12T19:44:27.087 DEBUG:boto:Params: {} 2022-09-12T19:44:27.088 DEBUG:boto:Token: None 2022-09-12T19:44:27.089 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/f 2022-09-12T19:44:27.090 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pzkjMOc9YNPXljJvqmpe7KmFBIc= 2022-09-12T19:44:27.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pzkjMOc9YNPXljJvqmpe7KmFBIc=', 'Content-Length': '0'} 2022-09-12T19:44:27.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5bc62b96efd4451-00631f8c1b-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.096 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:27.097 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:27.097 DEBUG:boto:Method: HEAD 2022-09-12T19:44:27.098 DEBUG:boto:Path: /jdpiak-71/f 2022-09-12T19:44:27.099 DEBUG:boto:Data: 2022-09-12T19:44:27.100 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.101 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:27.102 DEBUG:boto:Port: 8000 2022-09-12T19:44:27.103 DEBUG:boto:Params: {} 2022-09-12T19:44:27.103 DEBUG:boto:Token: None 2022-09-12T19:44:27.104 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/f 2022-09-12T19:44:27.105 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pzkjMOc9YNPXljJvqmpe7KmFBIc= 2022-09-12T19:44:27.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pzkjMOc9YNPXljJvqmpe7KmFBIc=', 'Content-Length': '0'} 2022-09-12T19:44:27.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b24ee3afe96d49c-00631f8c1b-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:44:27.109 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:27.110 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:27.111 DEBUG:boto:Method: GET 2022-09-12T19:44:27.112 DEBUG:boto:Path: /jdpiak-71/f 2022-09-12T19:44:27.113 DEBUG:boto:Data: 2022-09-12T19:44:27.113 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.114 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:27.114 DEBUG:boto:Port: 8001 2022-09-12T19:44:27.115 DEBUG:boto:Params: {} 2022-09-12T19:44:27.115 DEBUG:boto:Token: None 2022-09-12T19:44:27.117 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/f 2022-09-12T19:44:27.118 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HOO0XO++XXSJbKxhhOU0MOLopAQ= 2022-09-12T19:44:27.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HOO0XO++XXSJbKxhhOU0MOLopAQ=', 'Content-Length': '0'} 2022-09-12T19:44:27.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069cb29100abec362-00631f8c1b-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.166 DEBUG:boto:path=/jdpiak-71/f 2022-09-12T19:44:27.168 DEBUG:boto:auth_path=/jdpiak-71/f 2022-09-12T19:44:27.169 DEBUG:boto:Method: GET 2022-09-12T19:44:27.170 DEBUG:boto:Path: /jdpiak-71/f 2022-09-12T19:44:27.171 DEBUG:boto:Data: 2022-09-12T19:44:27.173 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.174 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:27.175 DEBUG:boto:Port: 8000 2022-09-12T19:44:27.176 DEBUG:boto:Params: {} 2022-09-12T19:44:27.177 DEBUG:boto:Token: None 2022-09-12T19:44:27.179 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/f 2022-09-12T19:44:27.180 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HOO0XO++XXSJbKxhhOU0MOLopAQ= 2022-09-12T19:44:27.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HOO0XO++XXSJbKxhhOU0MOLopAQ=', 'Content-Length': '0'} 2022-09-12T19:44:27.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f8498de568f63fa-00631f8c1b-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.186 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:27.187 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:27.188 DEBUG:boto:path=/jdpiak-71/?prefix=f 2022-09-12T19:44:27.189 DEBUG:boto:auth_path=/jdpiak-71/?prefix=f 2022-09-12T19:44:27.190 DEBUG:boto:Method: GET 2022-09-12T19:44:27.191 DEBUG:boto:Path: /jdpiak-71/?prefix=f 2022-09-12T19:44:27.192 DEBUG:boto:Data: 2022-09-12T19:44:27.193 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.195 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:27.196 DEBUG:boto:Port: 8001 2022-09-12T19:44:27.197 DEBUG:boto:Params: {} 2022-09-12T19:44:27.198 DEBUG:boto:Token: None 2022-09-12T19:44:27.200 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/ 2022-09-12T19:44:27.201 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GNc7FbEZFk/obzvOBAf77v0Fbl0= 2022-09-12T19:44:27.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GNc7FbEZFk/obzvOBAf77v0Fbl0=', 'Content-Length': '0'} 2022-09-12T19:44:27.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b768d1c7a7aad25-00631f8c1b-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.207 DEBUG:boto:b'jdpiak-71f1000falsef2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:27.209 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:27.211 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:27.212 DEBUG:boto:path=/jdpiak-71/?prefix=f 2022-09-12T19:44:27.213 DEBUG:boto:auth_path=/jdpiak-71/?prefix=f 2022-09-12T19:44:27.214 DEBUG:boto:Method: GET 2022-09-12T19:44:27.215 DEBUG:boto:Path: /jdpiak-71/?prefix=f 2022-09-12T19:44:27.216 DEBUG:boto:Data: 2022-09-12T19:44:27.217 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.218 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:27.219 DEBUG:boto:Port: 8000 2022-09-12T19:44:27.220 DEBUG:boto:Params: {} 2022-09-12T19:44:27.221 DEBUG:boto:Token: None 2022-09-12T19:44:27.222 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/ 2022-09-12T19:44:27.223 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GNc7FbEZFk/obzvOBAf77v0Fbl0= 2022-09-12T19:44:27.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GNc7FbEZFk/obzvOBAf77v0Fbl0=', 'Content-Length': '0'} 2022-09-12T19:44:27.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5a2de9fd90301c0-00631f8c1b-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.230 DEBUG:boto:b'jdpiak-71f1000falsef2022-09-12T19:41:59.770Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:27.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:44:27.232 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:27.233 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:27.234 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:27.235 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:27.236 DEBUG:boto:Method: GET 2022-09-12T19:44:27.237 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:27.238 DEBUG:boto:Data: 2022-09-12T19:44:27.240 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.241 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:27.243 DEBUG:boto:Port: 8001 2022-09-12T19:44:27.244 DEBUG:boto:Params: {} 2022-09-12T19:44:27.245 DEBUG:boto:Token: None 2022-09-12T19:44:27.247 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:27.248 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aFQNJ8Nj95euo1iwo5XKkcYQ8lE= 2022-09-12T19:44:27.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aFQNJ8Nj95euo1iwo5XKkcYQ8lE=', 'Content-Length': '0'} 2022-09-12T19:44:27.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0dae5eec2f2d8bf-00631f8c1b-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.255 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:27.256 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:27.257 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:27.258 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:27.259 DEBUG:boto:Method: GET 2022-09-12T19:44:27.260 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:27.262 DEBUG:boto:Data: 2022-09-12T19:44:27.263 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.264 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:27.266 DEBUG:boto:Port: 8000 2022-09-12T19:44:27.267 DEBUG:boto:Params: {} 2022-09-12T19:44:27.268 DEBUG:boto:Token: None 2022-09-12T19:44:27.270 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:27.271 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aFQNJ8Nj95euo1iwo5XKkcYQ8lE= 2022-09-12T19:44:27.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aFQNJ8Nj95euo1iwo5XKkcYQ8lE=', 'Content-Length': '0'} 2022-09-12T19:44:27.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004da8e6abe0de07f9-00631f8c1b-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.278 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:27.279 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:27.280 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:27.281 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:27.282 DEBUG:boto:Method: HEAD 2022-09-12T19:44:27.283 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:27.284 DEBUG:boto:Data: 2022-09-12T19:44:27.285 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.286 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:27.286 DEBUG:boto:Port: 8001 2022-09-12T19:44:27.287 DEBUG:boto:Params: {} 2022-09-12T19:44:27.289 DEBUG:boto:Token: None 2022-09-12T19:44:27.290 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:27.291 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QQuVAIhXQAc9PrMl/RdHDhbeYmE= 2022-09-12T19:44:27.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QQuVAIhXQAc9PrMl/RdHDhbeYmE=', 'Content-Length': '0'} 2022-09-12T19:44:27.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2cb75aa4c09075f-00631f8c1b-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.296 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:27.297 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:27.297 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:27.298 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:27.299 DEBUG:boto:Method: HEAD 2022-09-12T19:44:27.299 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:27.300 DEBUG:boto:Data: 2022-09-12T19:44:27.301 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.301 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:27.302 DEBUG:boto:Port: 8000 2022-09-12T19:44:27.303 DEBUG:boto:Params: {} 2022-09-12T19:44:27.303 DEBUG:boto:Token: None 2022-09-12T19:44:27.304 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:27.304 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QQuVAIhXQAc9PrMl/RdHDhbeYmE= 2022-09-12T19:44:27.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QQuVAIhXQAc9PrMl/RdHDhbeYmE=', 'Content-Length': '0'} 2022-09-12T19:44:27.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007543132fb836dc66-00631f8c1b-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:44:27.310 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:27.311 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:27.312 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:27.313 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:27.314 DEBUG:boto:Method: GET 2022-09-12T19:44:27.315 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:27.316 DEBUG:boto:Data: 2022-09-12T19:44:27.317 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.318 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:27.319 DEBUG:boto:Port: 8001 2022-09-12T19:44:27.320 DEBUG:boto:Params: {} 2022-09-12T19:44:27.321 DEBUG:boto:Token: None 2022-09-12T19:44:27.325 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:27.326 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aFQNJ8Nj95euo1iwo5XKkcYQ8lE= 2022-09-12T19:44:27.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aFQNJ8Nj95euo1iwo5XKkcYQ8lE=', 'Content-Length': '0'} 2022-09-12T19:44:27.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094379dd24dd8b9ec-00631f8c1b-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.333 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:27.334 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:27.335 DEBUG:boto:path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:27.336 DEBUG:boto:auth_path=/jdpiak-71/g?versionId=null 2022-09-12T19:44:27.337 DEBUG:boto:Method: GET 2022-09-12T19:44:27.338 DEBUG:boto:Path: /jdpiak-71/g?versionId=null 2022-09-12T19:44:27.340 DEBUG:boto:Data: 2022-09-12T19:44:27.341 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.341 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:27.342 DEBUG:boto:Port: 8000 2022-09-12T19:44:27.343 DEBUG:boto:Params: {} 2022-09-12T19:44:27.344 DEBUG:boto:Token: None 2022-09-12T19:44:27.345 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/g?versionId=null 2022-09-12T19:44:27.347 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aFQNJ8Nj95euo1iwo5XKkcYQ8lE= 2022-09-12T19:44:27.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aFQNJ8Nj95euo1iwo5XKkcYQ8lE=', 'Content-Length': '0'} 2022-09-12T19:44:27.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b28aab5e91d3a96-00631f8c1b-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.356 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:27.358 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:27.361 DEBUG:boto:path=/jdpiak-71/?prefix=g 2022-09-12T19:44:27.361 DEBUG:boto:auth_path=/jdpiak-71/?prefix=g 2022-09-12T19:44:27.362 DEBUG:boto:Method: GET 2022-09-12T19:44:27.364 DEBUG:boto:Path: /jdpiak-71/?prefix=g 2022-09-12T19:44:27.364 DEBUG:boto:Data: 2022-09-12T19:44:27.366 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.366 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:27.367 DEBUG:boto:Port: 8001 2022-09-12T19:44:27.367 DEBUG:boto:Params: {} 2022-09-12T19:44:27.368 DEBUG:boto:Token: None 2022-09-12T19:44:27.369 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/ 2022-09-12T19:44:27.370 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GNc7FbEZFk/obzvOBAf77v0Fbl0= 2022-09-12T19:44:27.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GNc7FbEZFk/obzvOBAf77v0Fbl0=', 'Content-Length': '0'} 2022-09-12T19:44:27.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f5ae6e8d471c53a-00631f8c1b-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.423 DEBUG:boto:b'jdpiak-71g1000falseg2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:27.425 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:27.426 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:27.427 DEBUG:boto:path=/jdpiak-71/?prefix=g 2022-09-12T19:44:27.428 DEBUG:boto:auth_path=/jdpiak-71/?prefix=g 2022-09-12T19:44:27.429 DEBUG:boto:Method: GET 2022-09-12T19:44:27.430 DEBUG:boto:Path: /jdpiak-71/?prefix=g 2022-09-12T19:44:27.431 DEBUG:boto:Data: 2022-09-12T19:44:27.432 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.433 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:27.434 DEBUG:boto:Port: 8000 2022-09-12T19:44:27.436 DEBUG:boto:Params: {} 2022-09-12T19:44:27.437 DEBUG:boto:Token: None 2022-09-12T19:44:27.438 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/ 2022-09-12T19:44:27.439 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GNc7FbEZFk/obzvOBAf77v0Fbl0= 2022-09-12T19:44:27.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GNc7FbEZFk/obzvOBAf77v0Fbl0=', 'Content-Length': '0'} 2022-09-12T19:44:27.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4df630e1846995b-00631f8c1b-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.447 DEBUG:boto:b'jdpiak-71g1000falseg2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:27.448 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:27.449 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:27.451 DEBUG:boto:Method: HEAD 2022-09-12T19:44:27.452 DEBUG:boto:Path: /jdpiak-71/g 2022-09-12T19:44:27.453 DEBUG:boto:Data: 2022-09-12T19:44:27.455 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.456 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:27.458 DEBUG:boto:Port: 8001 2022-09-12T19:44:27.459 DEBUG:boto:Params: {} 2022-09-12T19:44:27.461 DEBUG:boto:Token: None 2022-09-12T19:44:27.462 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/g 2022-09-12T19:44:27.463 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:338pf/nEuc5/1QSj1uUrbN9WXTA= 2022-09-12T19:44:27.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:338pf/nEuc5/1QSj1uUrbN9WXTA=', 'Content-Length': '0'} 2022-09-12T19:44:27.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6fcb37934e5d682-00631f8c1b-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.471 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:27.472 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:27.473 DEBUG:boto:Method: HEAD 2022-09-12T19:44:27.475 DEBUG:boto:Path: /jdpiak-71/g 2022-09-12T19:44:27.476 DEBUG:boto:Data: 2022-09-12T19:44:27.477 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.479 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:27.480 DEBUG:boto:Port: 8000 2022-09-12T19:44:27.481 DEBUG:boto:Params: {} 2022-09-12T19:44:27.482 DEBUG:boto:Token: None 2022-09-12T19:44:27.483 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/g 2022-09-12T19:44:27.484 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:338pf/nEuc5/1QSj1uUrbN9WXTA= 2022-09-12T19:44:27.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:338pf/nEuc5/1QSj1uUrbN9WXTA=', 'Content-Length': '0'} 2022-09-12T19:44:27.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a6b789ce1c9f39a-00631f8c1b-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:44:27.488 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:27.489 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:27.489 DEBUG:boto:Method: GET 2022-09-12T19:44:27.490 DEBUG:boto:Path: /jdpiak-71/g 2022-09-12T19:44:27.491 DEBUG:boto:Data: 2022-09-12T19:44:27.491 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.492 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:27.493 DEBUG:boto:Port: 8001 2022-09-12T19:44:27.493 DEBUG:boto:Params: {} 2022-09-12T19:44:27.494 DEBUG:boto:Token: None 2022-09-12T19:44:27.494 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/g 2022-09-12T19:44:27.495 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5SVQlLFKramaseky7f86x9PcHPQ= 2022-09-12T19:44:27.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5SVQlLFKramaseky7f86x9PcHPQ=', 'Content-Length': '0'} 2022-09-12T19:44:27.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007861a35cc2d3d54-00631f8c1b-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.499 DEBUG:boto:path=/jdpiak-71/g 2022-09-12T19:44:27.500 DEBUG:boto:auth_path=/jdpiak-71/g 2022-09-12T19:44:27.501 DEBUG:boto:Method: GET 2022-09-12T19:44:27.502 DEBUG:boto:Path: /jdpiak-71/g 2022-09-12T19:44:27.503 DEBUG:boto:Data: 2022-09-12T19:44:27.504 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.504 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:27.505 DEBUG:boto:Port: 8000 2022-09-12T19:44:27.505 DEBUG:boto:Params: {} 2022-09-12T19:44:27.506 DEBUG:boto:Token: None 2022-09-12T19:44:27.507 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/g 2022-09-12T19:44:27.507 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5SVQlLFKramaseky7f86x9PcHPQ= 2022-09-12T19:44:27.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5SVQlLFKramaseky7f86x9PcHPQ=', 'Content-Length': '0'} 2022-09-12T19:44:27.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005076d0b88f2817f6-00631f8c1b-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.511 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:27.512 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:27.512 DEBUG:boto:path=/jdpiak-71/?prefix=g 2022-09-12T19:44:27.513 DEBUG:boto:auth_path=/jdpiak-71/?prefix=g 2022-09-12T19:44:27.514 DEBUG:boto:Method: GET 2022-09-12T19:44:27.514 DEBUG:boto:Path: /jdpiak-71/?prefix=g 2022-09-12T19:44:27.515 DEBUG:boto:Data: 2022-09-12T19:44:27.515 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.516 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:27.516 DEBUG:boto:Port: 8001 2022-09-12T19:44:27.517 DEBUG:boto:Params: {} 2022-09-12T19:44:27.518 DEBUG:boto:Token: None 2022-09-12T19:44:27.518 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/ 2022-09-12T19:44:27.519 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GNc7FbEZFk/obzvOBAf77v0Fbl0= 2022-09-12T19:44:27.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GNc7FbEZFk/obzvOBAf77v0Fbl0=', 'Content-Length': '0'} 2022-09-12T19:44:27.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058dcf3e7c0af9acf-00631f8c1b-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.534 DEBUG:boto:b'jdpiak-71g1000falseg2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:27.535 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:27.535 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:27.536 DEBUG:boto:path=/jdpiak-71/?prefix=g 2022-09-12T19:44:27.536 DEBUG:boto:auth_path=/jdpiak-71/?prefix=g 2022-09-12T19:44:27.537 DEBUG:boto:Method: GET 2022-09-12T19:44:27.538 DEBUG:boto:Path: /jdpiak-71/?prefix=g 2022-09-12T19:44:27.538 DEBUG:boto:Data: 2022-09-12T19:44:27.539 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.540 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:27.540 DEBUG:boto:Port: 8000 2022-09-12T19:44:27.541 DEBUG:boto:Params: {} 2022-09-12T19:44:27.542 DEBUG:boto:Token: None 2022-09-12T19:44:27.543 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/ 2022-09-12T19:44:27.544 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GNc7FbEZFk/obzvOBAf77v0Fbl0= 2022-09-12T19:44:27.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GNc7FbEZFk/obzvOBAf77v0Fbl0=', 'Content-Length': '0'} 2022-09-12T19:44:27.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c50bf2f012db5df-00631f8c1b-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.548 DEBUG:boto:b'jdpiak-71g1000falseg2022-09-12T19:41:59.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:27.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:44:27.549 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:27.550 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:27.551 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:27.552 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:27.553 DEBUG:boto:Method: GET 2022-09-12T19:44:27.554 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:27.556 DEBUG:boto:Data: 2022-09-12T19:44:27.557 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.558 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:27.559 DEBUG:boto:Port: 8001 2022-09-12T19:44:27.560 DEBUG:boto:Params: {} 2022-09-12T19:44:27.561 DEBUG:boto:Token: None 2022-09-12T19:44:27.562 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:27.563 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wZ2dNXFE59yKXe+4N1KV+3R/lYQ= 2022-09-12T19:44:27.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wZ2dNXFE59yKXe+4N1KV+3R/lYQ=', 'Content-Length': '0'} 2022-09-12T19:44:27.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afaf835818f4c0d5-00631f8c1b-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.568 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:27.569 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:27.570 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:27.571 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:27.572 DEBUG:boto:Method: GET 2022-09-12T19:44:27.573 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:27.574 DEBUG:boto:Data: 2022-09-12T19:44:27.575 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.575 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:27.576 DEBUG:boto:Port: 8000 2022-09-12T19:44:27.576 DEBUG:boto:Params: {} 2022-09-12T19:44:27.577 DEBUG:boto:Token: None 2022-09-12T19:44:27.577 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:27.578 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wZ2dNXFE59yKXe+4N1KV+3R/lYQ= 2022-09-12T19:44:27.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wZ2dNXFE59yKXe+4N1KV+3R/lYQ=', 'Content-Length': '0'} 2022-09-12T19:44:27.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032cd368632390c23-00631f8c1b-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.628 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:27.629 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:27.629 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:27.630 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:27.630 DEBUG:boto:Method: HEAD 2022-09-12T19:44:27.631 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:27.631 DEBUG:boto:Data: 2022-09-12T19:44:27.632 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.632 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:27.633 DEBUG:boto:Port: 8001 2022-09-12T19:44:27.633 DEBUG:boto:Params: {} 2022-09-12T19:44:27.634 DEBUG:boto:Token: None 2022-09-12T19:44:27.635 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:27.636 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LLOhfQRqls8AU73G5wfVwuB7d2U= 2022-09-12T19:44:27.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LLOhfQRqls8AU73G5wfVwuB7d2U=', 'Content-Length': '0'} 2022-09-12T19:44:27.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d013eb7f5747b747-00631f8c1b-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.649 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:27.650 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:27.651 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:27.651 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:27.652 DEBUG:boto:Method: HEAD 2022-09-12T19:44:27.652 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:27.653 DEBUG:boto:Data: 2022-09-12T19:44:27.653 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.655 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:27.655 DEBUG:boto:Port: 8000 2022-09-12T19:44:27.656 DEBUG:boto:Params: {} 2022-09-12T19:44:27.656 DEBUG:boto:Token: None 2022-09-12T19:44:27.658 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:27.659 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LLOhfQRqls8AU73G5wfVwuB7d2U= 2022-09-12T19:44:27.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LLOhfQRqls8AU73G5wfVwuB7d2U=', 'Content-Length': '0'} 2022-09-12T19:44:27.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030d076cc490db1fb-00631f8c1b-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:44:27.665 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:27.666 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:27.668 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:27.669 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:27.670 DEBUG:boto:Method: GET 2022-09-12T19:44:27.671 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:27.672 DEBUG:boto:Data: 2022-09-12T19:44:27.673 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.674 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:27.675 DEBUG:boto:Port: 8001 2022-09-12T19:44:27.676 DEBUG:boto:Params: {} 2022-09-12T19:44:27.677 DEBUG:boto:Token: None 2022-09-12T19:44:27.678 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:27 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:27.679 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wZ2dNXFE59yKXe+4N1KV+3R/lYQ= 2022-09-12T19:44:27.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wZ2dNXFE59yKXe+4N1KV+3R/lYQ=', 'Content-Length': '0'} 2022-09-12T19:44:27.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086203c6ac43e4b67-00631f8c1b-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:27.690 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:27.691 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:27.692 DEBUG:boto:path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:27.693 DEBUG:boto:auth_path=/jdpiak-71/h?versionId=null 2022-09-12T19:44:27.694 DEBUG:boto:Method: GET 2022-09-12T19:44:27.695 DEBUG:boto:Path: /jdpiak-71/h?versionId=null 2022-09-12T19:44:27.696 DEBUG:boto:Data: 2022-09-12T19:44:27.698 DEBUG:boto:Headers: {} 2022-09-12T19:44:27.700 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:27.701 DEBUG:boto:Port: 8000 2022-09-12T19:44:28.298 DEBUG:boto:Params: {} 2022-09-12T19:44:28.300 DEBUG:boto:Token: None 2022-09-12T19:44:28.301 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:28 GMT /jdpiak-71/h?versionId=null 2022-09-12T19:44:28.302 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ny17JPVNCPXzkWfCTX35aruCSC4= 2022-09-12T19:44:28.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ny17JPVNCPXzkWfCTX35aruCSC4=', 'Content-Length': '0'} 2022-09-12T19:44:28.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009c8320fc57cb2a9-00631f8c1c-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:28.312 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:28.313 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:28.314 DEBUG:boto:path=/jdpiak-71/?prefix=h 2022-09-12T19:44:28.315 DEBUG:boto:auth_path=/jdpiak-71/?prefix=h 2022-09-12T19:44:28.315 DEBUG:boto:Method: GET 2022-09-12T19:44:28.316 DEBUG:boto:Path: /jdpiak-71/?prefix=h 2022-09-12T19:44:28.317 DEBUG:boto:Data: 2022-09-12T19:44:28.318 DEBUG:boto:Headers: {} 2022-09-12T19:44:28.319 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:28.320 DEBUG:boto:Port: 8001 2022-09-12T19:44:28.321 DEBUG:boto:Params: {} 2022-09-12T19:44:28.322 DEBUG:boto:Token: None 2022-09-12T19:44:28.323 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:28 GMT /jdpiak-71/ 2022-09-12T19:44:28.323 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IUrJ53LY6UPEeY8Bt3OSAG8essY= 2022-09-12T19:44:28.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IUrJ53LY6UPEeY8Bt3OSAG8essY=', 'Content-Length': '0'} 2022-09-12T19:44:28.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047c31e6b16f8dc22-00631f8c1c-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:28.331 DEBUG:boto:b'jdpiak-71h1000falseh2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:28.333 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:28.334 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:28.335 DEBUG:boto:path=/jdpiak-71/?prefix=h 2022-09-12T19:44:28.336 DEBUG:boto:auth_path=/jdpiak-71/?prefix=h 2022-09-12T19:44:28.337 DEBUG:boto:Method: GET 2022-09-12T19:44:28.338 DEBUG:boto:Path: /jdpiak-71/?prefix=h 2022-09-12T19:44:28.339 DEBUG:boto:Data: 2022-09-12T19:44:28.340 DEBUG:boto:Headers: {} 2022-09-12T19:44:28.341 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:28.342 DEBUG:boto:Port: 8000 2022-09-12T19:44:28.343 DEBUG:boto:Params: {} 2022-09-12T19:44:28.344 DEBUG:boto:Token: None 2022-09-12T19:44:28.345 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:28 GMT /jdpiak-71/ 2022-09-12T19:44:28.346 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IUrJ53LY6UPEeY8Bt3OSAG8essY= 2022-09-12T19:44:28.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IUrJ53LY6UPEeY8Bt3OSAG8essY=', 'Content-Length': '0'} 2022-09-12T19:44:28.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f713ec609dc1ac8-00631f8c1c-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:28.364 DEBUG:boto:b'jdpiak-71h1000falseh2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:28.365 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:28.366 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:28.367 DEBUG:boto:Method: HEAD 2022-09-12T19:44:28.368 DEBUG:boto:Path: /jdpiak-71/h 2022-09-12T19:44:28.369 DEBUG:boto:Data: 2022-09-12T19:44:28.371 DEBUG:boto:Headers: {} 2022-09-12T19:44:28.371 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:28.372 DEBUG:boto:Port: 8001 2022-09-12T19:44:28.373 DEBUG:boto:Params: {} 2022-09-12T19:44:28.384 DEBUG:boto:Token: None 2022-09-12T19:44:28.385 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:28 GMT /jdpiak-71/h 2022-09-12T19:44:28.386 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F3wHbZEOAx96y0ht/n7mam6HVH4= 2022-09-12T19:44:28.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F3wHbZEOAx96y0ht/n7mam6HVH4=', 'Content-Length': '0'} 2022-09-12T19:44:28.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5b360c47061e194-00631f8c1c-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:28.395 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:28.396 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:28.398 DEBUG:boto:Method: HEAD 2022-09-12T19:44:28.399 DEBUG:boto:Path: /jdpiak-71/h 2022-09-12T19:44:28.400 DEBUG:boto:Data: 2022-09-12T19:44:28.400 DEBUG:boto:Headers: {} 2022-09-12T19:44:28.401 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:28.402 DEBUG:boto:Port: 8000 2022-09-12T19:44:28.403 DEBUG:boto:Params: {} 2022-09-12T19:44:28.404 DEBUG:boto:Token: None 2022-09-12T19:44:28.404 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:28 GMT /jdpiak-71/h 2022-09-12T19:44:28.405 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F3wHbZEOAx96y0ht/n7mam6HVH4= 2022-09-12T19:44:28.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F3wHbZEOAx96y0ht/n7mam6HVH4=', 'Content-Length': '0'} 2022-09-12T19:44:28.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f74853237d7ba03-00631f8c1c-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:28.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:44:28.412 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:28.413 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:28.414 DEBUG:boto:Method: GET 2022-09-12T19:44:28.415 DEBUG:boto:Path: /jdpiak-71/h 2022-09-12T19:44:28.416 DEBUG:boto:Data: 2022-09-12T19:44:28.417 DEBUG:boto:Headers: {} 2022-09-12T19:44:28.419 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:28.433 DEBUG:boto:Port: 8001 2022-09-12T19:44:28.435 DEBUG:boto:Params: {} 2022-09-12T19:44:28.436 DEBUG:boto:Token: None 2022-09-12T19:44:28.438 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:28 GMT /jdpiak-71/h 2022-09-12T19:44:28.440 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:w/77RO8AtGCEbEPAGziJS5maAQs= 2022-09-12T19:44:28.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:w/77RO8AtGCEbEPAGziJS5maAQs=', 'Content-Length': '0'} 2022-09-12T19:44:28.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000142380667cf7e76f-00631f8c1c-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:28.446 DEBUG:boto:path=/jdpiak-71/h 2022-09-12T19:44:28.447 DEBUG:boto:auth_path=/jdpiak-71/h 2022-09-12T19:44:28.448 DEBUG:boto:Method: GET 2022-09-12T19:44:28.449 DEBUG:boto:Path: /jdpiak-71/h 2022-09-12T19:44:28.450 DEBUG:boto:Data: 2022-09-12T19:44:28.451 DEBUG:boto:Headers: {} 2022-09-12T19:44:28.451 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:28.452 DEBUG:boto:Port: 8000 2022-09-12T19:44:28.452 DEBUG:boto:Params: {} 2022-09-12T19:44:28.455 DEBUG:boto:Token: None 2022-09-12T19:44:28.456 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:28 GMT /jdpiak-71/h 2022-09-12T19:44:28.457 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:w/77RO8AtGCEbEPAGziJS5maAQs= 2022-09-12T19:44:28.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:w/77RO8AtGCEbEPAGziJS5maAQs=', 'Content-Length': '0'} 2022-09-12T19:44:28.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:41:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d48eaf2b74e54f97-00631f8c1c-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:44:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:28.483 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:28.486 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:28.487 DEBUG:boto:path=/jdpiak-71/?prefix=h 2022-09-12T19:44:28.488 DEBUG:boto:auth_path=/jdpiak-71/?prefix=h 2022-09-12T19:44:28.491 DEBUG:boto:Method: GET 2022-09-12T19:44:28.492 DEBUG:boto:Path: /jdpiak-71/?prefix=h 2022-09-12T19:44:28.496 DEBUG:boto:Data: 2022-09-12T19:44:28.497 DEBUG:boto:Headers: {} 2022-09-12T19:44:28.497 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:44:28.498 DEBUG:boto:Port: 8001 2022-09-12T19:44:28.499 DEBUG:boto:Params: {} 2022-09-12T19:44:28.500 DEBUG:boto:Token: None 2022-09-12T19:44:28.502 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:28 GMT /jdpiak-71/ 2022-09-12T19:44:28.504 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IUrJ53LY6UPEeY8Bt3OSAG8essY= 2022-09-12T19:44:28.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IUrJ53LY6UPEeY8Bt3OSAG8essY=', 'Content-Length': '0'} 2022-09-12T19:44:28.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfa74c22c2556104-00631f8c1c-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:28.516 DEBUG:boto:b'jdpiak-71h1000falseh2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:28.518 DEBUG:boto:path=/jdpiak-71/ 2022-09-12T19:44:28.520 DEBUG:boto:auth_path=/jdpiak-71/ 2022-09-12T19:44:28.521 DEBUG:boto:path=/jdpiak-71/?prefix=h 2022-09-12T19:44:28.522 DEBUG:boto:auth_path=/jdpiak-71/?prefix=h 2022-09-12T19:44:28.523 DEBUG:boto:Method: GET 2022-09-12T19:44:28.524 DEBUG:boto:Path: /jdpiak-71/?prefix=h 2022-09-12T19:44:28.525 DEBUG:boto:Data: 2022-09-12T19:44:28.527 DEBUG:boto:Headers: {} 2022-09-12T19:44:28.528 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:44:28.531 DEBUG:boto:Port: 8000 2022-09-12T19:44:28.532 DEBUG:boto:Params: {} 2022-09-12T19:44:28.533 DEBUG:boto:Token: None 2022-09-12T19:44:28.535 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:44:28 GMT /jdpiak-71/ 2022-09-12T19:44:28.536 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IUrJ53LY6UPEeY8Bt3OSAG8essY= 2022-09-12T19:44:28.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IUrJ53LY6UPEeY8Bt3OSAG8essY=', 'Content-Length': '0'} 2022-09-12T19:44:28.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c24a19f414d7040-00631f8c1c-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:44:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:28.553 DEBUG:boto:b'jdpiak-71h1000falseh2022-09-12T19:41:59.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:44:28.555 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-71 zones={test-zone3, test-zone2} 2022-09-12T19:44:28.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:44:28.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:44:28.558 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:44:28.614 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:44:28.615 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:44:28.664 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:44:28.661+0000 7f555fd71700 -1 req 788267409067706670 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-52:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.3,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-52:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.3,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:44:28.665 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:44:28.661+0000 7f5569584700 -1 req 5008516476720346018 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-4:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.4,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-4:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.4,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:44:28.669 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:44:28.661+0000 7fe85a6a3700 -1 req 3481597893456650844 0.000999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-52:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.3,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-52:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.3,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:44:28.669 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:44:28.661+0000 7fe862eb4700 -1 req 17683018336328001276 0.000999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-4:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.4,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-4:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.4,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:44:28.671 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:44:28.661+0000 7fa8607b6700 -1 req 5449482080855922072 0.000999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-4:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.4,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-4:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.4,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:44:28.671 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:44:28.661+0000 7fa85afab700 -1 req 7273231801732192389 0.000999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-52:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.3,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-52:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.3,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:44:28.682 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:44:28.680+0000 7fe844677700 -1 req 18411014525963224064 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-67:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.18,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-67:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.18,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:44:28.683 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:44:28.680+0000 7fa8737dc700 -1 req 13576508449727310525 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-67:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.18,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-67:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.18,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:44:28.688 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:44:28.680+0000 7f553c52a700 -1 req 8056759146867230466 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-67:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.18,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-67:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.18,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:44:28.722 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:44:28.717+0000 7f5529d05700 -1 req 9722266663283765858 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:44:28.723 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:44:28.717+0000 7f554fd51700 -1 req 11295630102142497545 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-43:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.2,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-43:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.2,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:44:28.724 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:44:28.717+0000 7fe83ae64700 -1 req 16777125238543296285 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-43:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.2,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-43:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.2,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:44:28.726 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:44:28.717+0000 7fa859fa9700 -1 req 9807472027289943355 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-43:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.2,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-43:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.2,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:44:28.726 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:44:28.717+0000 7fa8597a8700 -1 req 14923109142353183145 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:44:28.727 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:44:28.717+0000 7fe830e50700 -1 req 18092057751312024512 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-29:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.29,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:44:28.768 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:44:28.766+0000 7f54ef490700 -1 req 14515641313298275658 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-3:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.3,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-3:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.3,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:44:28.771 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:44:28.765+0000 7fe826e3c700 -1 req 8346497723995776901 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-3:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.3,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-3:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.3,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:44:28.772 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:44:28.765+0000 7fa853f9d700 -1 req 13683110116668214152 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-3:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.3,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-3:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.3,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:44:28.772 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:44:28.769+0000 7fa84a78a700 -1 req 16396755729856234406 0.000999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-42:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.1,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-42:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.1,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:44:28.773 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:44:28.769+0000 7fe819e22700 -1 req 933626195664019640 0.000999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-42:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.1,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-42:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.1,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:44:28.774 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:44:28.770+0000 7f55054bc700 -1 req 15205775253159405980 0.000999980s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-42:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.1,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-42:fdf3b910-2ea3-4164-a3f8-b3e146af1c44.5166.1,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:44:28.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'jdpiak-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:44:28.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'jdpiak-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:44:28.839 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket jdpiak-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:44:28.890 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:44:28.892 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:44:28.932 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:44:28.933 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:44:28.933 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding": "None", 2022-09-12T19:44:28.933 INFO:teuthology.orchestra.run.smithi035.stdout: "current_index": { 2022-09-12T19:44:28.934 INFO:teuthology.orchestra.run.smithi035.stdout: "gen": 2, 2022-09-12T19:44:28.934 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:44:28.935 INFO:teuthology.orchestra.run.smithi035.stdout: "type": "Normal", 2022-09-12T19:44:28.936 INFO:teuthology.orchestra.run.smithi035.stdout: "normal": { 2022-09-12T19:44:28.936 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 3, 2022-09-12T19:44:28.936 INFO:teuthology.orchestra.run.smithi035.stdout: "hash_type": "Mod" 2022-09-12T19:44:28.937 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:28.937 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:28.937 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:44:28.938 INFO:teuthology.orchestra.run.smithi035.stdout: "logs": [ 2022-09-12T19:44:28.938 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:44:28.938 INFO:teuthology.orchestra.run.smithi035.stdout: "gen": 1, 2022-09-12T19:44:28.939 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:44:28.940 INFO:teuthology.orchestra.run.smithi035.stdout: "type": "InIndex", 2022-09-12T19:44:28.941 INFO:teuthology.orchestra.run.smithi035.stdout: "in_index": { 2022-09-12T19:44:28.941 INFO:teuthology.orchestra.run.smithi035.stdout: "gen": 1, 2022-09-12T19:44:28.941 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:44:28.942 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 3, 2022-09-12T19:44:28.942 INFO:teuthology.orchestra.run.smithi035.stdout: "hash_type": "Mod" 2022-09-12T19:44:28.943 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:28.943 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:28.944 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:28.944 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:44:28.945 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:44:28.945 INFO:teuthology.orchestra.run.smithi035.stdout: "gen": 2, 2022-09-12T19:44:28.945 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:44:28.946 INFO:teuthology.orchestra.run.smithi035.stdout: "type": "InIndex", 2022-09-12T19:44:28.946 INFO:teuthology.orchestra.run.smithi035.stdout: "in_index": { 2022-09-12T19:44:28.946 INFO:teuthology.orchestra.run.smithi035.stdout: "gen": 2, 2022-09-12T19:44:28.947 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:44:28.947 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 3, 2022-09-12T19:44:28.947 INFO:teuthology.orchestra.run.smithi035.stdout: "hash_type": "Mod" 2022-09-12T19:44:28.948 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:28.949 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:28.949 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:28.950 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:28.950 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:44:28.951 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:28.951 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:44:28.952 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:44:28.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:44:28.954 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:44:29.001 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:44:29.001 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:44:29.052 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:44:29.047+0000 7f54e3c79700 -1 req 13239632555482607308 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:44:29.054 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:44:29.047+0000 7fa887804700 -1 req 13140504920268751463 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:44:29.054 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:44:29.048+0000 7fe811611700 -1 req 1878381531103824677 0.000999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-35:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.35,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:44:29.104 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:44:29.100+0000 7fe806dfc700 -1 req 16119963911117021580 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-55:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.6,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-55:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.6,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:44:29.105 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:44:29.101+0000 7f54d0c53700 -1 req 9063156803468827607 0.000999980s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-55:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.6,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-55:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.6,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:44:29.105 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:44:29.100+0000 7fa86f7d4700 -1 req 8338819054803045490 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-55:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.6,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-55:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.6,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:44:29.109 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:44:29.108+0000 7f54e547c700 -1 req 9161416075399441148 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-50:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.1,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-50:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.1,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:44:29.110 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:44:29.107+0000 7fe7d7d9e700 -1 req 16070529455716148504 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-50:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.1,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-50:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.1,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:44:29.111 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:44:29.107+0000 7fa843f7d700 -1 req 17157517478522636363 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-50:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.1,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-50:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.1,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:44:29.114 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:44:29.112+0000 7f54cb448700 -1 req 15704209290133569889 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-40:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.40,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-40:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.40,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:44:29.115 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:44:29.111+0000 7fe7d4597700 -1 req 8525597043418983270 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-40:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.40,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-40:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.40,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:44:29.116 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:44:29.112+0000 7fa7d0696700 -1 req 2750358567964609861 0.000999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-40:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.40,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-40:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.40,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:44:29.147 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:44:29.144+0000 7f54b5c1d700 -1 req 14533509380363510154 0.000999980s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:44:29.148 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:44:29.143+0000 7fe7d659b700 -1 req 15890454810307136807 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:44:29.150 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:44:29.143+0000 7fa805700700 -1 req 862153141417915858 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-9:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.4344.9,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:44:29.158 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:44:29.157+0000 7f54bd42c700 -1 req 10918677286003881221 0.000999980s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-44:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.1,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-44:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.1,z=823fb3a5-4d8a-49dc-ae64-e05804bbea94,az=0}} returned ret=-2 2022-09-12T19:44:29.160 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:44:29.156+0000 7fa81ef33700 -1 req 11739496317960105135 0.000999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-44:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.1,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-44:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.1,z=fdf3b910-2ea3-4164-a3f8-b3e146af1c44,az=0}} returned ret=-2 2022-09-12T19:44:29.160 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T19:44:29.156+0000 7fe7c3d76700 -1 req 14587692751693919089 0.000999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=jdpiak-44:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.1,z=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47,az=0},d={b=jdpiak-44:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5172.1,z=eb41eca3-59e3-4448-adac-b7c5067fb406,az=0}} returned ret=-2 2022-09-12T19:44:29.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'jdpiak-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:44:29.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'jdpiak-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:44:29.239 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket jdpiak-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:44:29.298 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:44:29.299 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:44:29.339 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:44:29.340 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:44:29.340 INFO:teuthology.orchestra.run.smithi035.stdout: "resharding": "None", 2022-09-12T19:44:29.341 INFO:teuthology.orchestra.run.smithi035.stdout: "current_index": { 2022-09-12T19:44:29.341 INFO:teuthology.orchestra.run.smithi035.stdout: "gen": 2, 2022-09-12T19:44:29.341 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:44:29.342 INFO:teuthology.orchestra.run.smithi035.stdout: "type": "Normal", 2022-09-12T19:44:29.342 INFO:teuthology.orchestra.run.smithi035.stdout: "normal": { 2022-09-12T19:44:29.343 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 3, 2022-09-12T19:44:29.343 INFO:teuthology.orchestra.run.smithi035.stdout: "hash_type": "Mod" 2022-09-12T19:44:29.344 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:29.344 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:29.345 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:44:29.345 INFO:teuthology.orchestra.run.smithi035.stdout: "logs": [ 2022-09-12T19:44:29.346 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:44:29.346 INFO:teuthology.orchestra.run.smithi035.stdout: "gen": 1, 2022-09-12T19:44:29.347 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:44:29.347 INFO:teuthology.orchestra.run.smithi035.stdout: "type": "InIndex", 2022-09-12T19:44:29.348 INFO:teuthology.orchestra.run.smithi035.stdout: "in_index": { 2022-09-12T19:44:29.349 INFO:teuthology.orchestra.run.smithi035.stdout: "gen": 1, 2022-09-12T19:44:29.349 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:44:29.350 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 3, 2022-09-12T19:44:29.350 INFO:teuthology.orchestra.run.smithi035.stdout: "hash_type": "Mod" 2022-09-12T19:44:29.351 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:29.351 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:29.352 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:29.352 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:44:29.353 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:44:29.353 INFO:teuthology.orchestra.run.smithi035.stdout: "gen": 2, 2022-09-12T19:44:29.354 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:44:29.354 INFO:teuthology.orchestra.run.smithi035.stdout: "type": "InIndex", 2022-09-12T19:44:29.355 INFO:teuthology.orchestra.run.smithi035.stdout: "in_index": { 2022-09-12T19:44:29.356 INFO:teuthology.orchestra.run.smithi035.stdout: "gen": 2, 2022-09-12T19:44:29.356 INFO:teuthology.orchestra.run.smithi035.stdout: "layout": { 2022-09-12T19:44:29.357 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 3, 2022-09-12T19:44:29.357 INFO:teuthology.orchestra.run.smithi035.stdout: "hash_type": "Mod" 2022-09-12T19:44:29.358 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:29.359 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:29.359 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:29.360 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:29.360 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:44:29.361 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:29.361 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:44:29.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... FAIL 2022-09-12T19:44:29.384 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:44:29.385 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:44:29.386 DEBUG:boto:Method: PUT 2022-09-12T19:44:29.387 DEBUG:boto:Path: /jdpiak-72/ 2022-09-12T19:44:29.388 DEBUG:boto:Data: 2022-09-12T19:44:29.389 DEBUG:boto:Headers: {} 2022-09-12T19:44:29.391 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:29.392 DEBUG:boto:Port: 8000 2022-09-12T19:44:29.393 DEBUG:boto:Params: {} 2022-09-12T19:44:29.395 DEBUG:boto:Token: None 2022-09-12T19:44:29.396 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:44:29 GMT /jdpiak-72/ 2022-09-12T19:44:29.397 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F0HTuELC9FLCHAhKQYqISlc4pow= 2022-09-12T19:44:29.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F0HTuELC9FLCHAhKQYqISlc4pow=', 'Content-Length': '0'} 2022-09-12T19:44:29.420 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c6d236db0ad948bb-00631f8c1d-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:44:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:29.422 DEBUG:rgw_multi.tests:created bucket=jdpiak-72 2022-09-12T19:44:29.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:44:29.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:44:29.426 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:44:29.478 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:44:29.479 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:44:29.510 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=38ca1f57-1358-496c-a99b-df49ee365a4f 2022-09-12T19:44:29.530 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:44:29.531 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:44:29.532 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:44:29.532 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:44:29.533 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:44:29.534 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:44:29.535 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:44:29.536 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:44:29.537 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:44:29.537 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:44:29.538 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663011869.416287_676.1", 2022-09-12T19:44:29.538 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:44:29.416287Z" 2022-09-12T19:44:29.539 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:44:29.539 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:44:29.540 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:44:29.540 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:44:29.540 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:29.541 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:44:29.543 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1663011869.416287_676.1', 3: ''} 2022-09-12T19:44:29.545 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T19:44:29.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:44:29.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:44:29.548 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:44:29.610 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:44:29.617 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:44:29.650 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:44:29.647+0000 7fde3d338600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:44:29.651 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:44:29.648+0000 7fde3d338600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:44:29.671 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:44:29.671 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:44:29.672 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:44:29.672 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:44:29.673 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:44:29.673 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:44:29.674 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:44:29.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:44:29.675 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:44:29.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:44:29.676 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:44:29.677 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:44:29.677 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:44:29.678 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:44:29.678 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:44:29.679 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:44:29.680 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:44:29.680 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:44:29.681 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:44:29.681 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:44:29.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:44:29.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:44:29.682 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:44:29.682 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:44:29.682 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:44:29.683 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:44:29.683 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:44:29.683 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:44:29.683 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:44:29.684 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:44:29.684 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:44:29.684 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:44:29.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:44:29.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:44:29.685 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:44:29.685 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:44:29.685 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:44:29.686 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663011869.416287_676.1", 2022-09-12T19:44:29.686 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:44:29.686 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:44:29.686 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:44:29.687 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:44:29.416287Z", 2022-09-12T19:44:29.687 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:44:29.687 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:44:29.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:44:29.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:44:29.688 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:44:29.688 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:44:29.689 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:44:29.689 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:44:29.689 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:44:29.690 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:44:29.690 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:44:29.690 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:44:29.691 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:44:29.691 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:44:29.691 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:44:29.691 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:44:29.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:44:29.692 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:44:29.692 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:44:29.693 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:44:29.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:44:29.693 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:44:29Z" 2022-09-12T19:44:29.694 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:44:29.694 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011869.416287_676.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:44:29.416287Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:44:29Z" } 2022-09-12T19:44:29.695 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011869.416287_676.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:44:29.416287Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:44:29.695 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011869.416287_676.1', 3: ''} 2022-09-12T19:44:29.696 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011869.416287_676.1', 3: ''} 2022-09-12T19:44:29.696 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T19:44:29.697 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T19:44:29.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:44:29.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:44:29.699 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:44:29.764 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:44:29.765 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:44:29.810 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:44:29.808+0000 7f7f98458600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:44:29.811 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:44:29.809+0000 7f7f98458600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:44:29.839 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:44:29.840 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:44:29.841 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:44:29.842 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:44:29.843 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:44:29.845 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:44:29.845 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:44:29.846 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:44:29.847 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:44:29.847 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:44:29.848 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:44:29.849 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:44:29.850 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:44:29.850 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:44:29.851 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:44:29.851 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:44:29.852 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:44:29.852 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:44:29.852 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:44:29.852 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:29.853 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:44:29.853 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:44:29.853 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:44:29.854 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:44:29.854 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:44:29.855 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:44:29.855 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:44:29.855 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:44:29.856 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:44:29.856 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:44:29.856 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:44:29.857 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:29.858 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:44:29.859 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:44:29.860 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:44:29.861 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:44:29.862 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:44:29.863 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663011869.416287_676.1", 2022-09-12T19:44:29.863 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:44:29.864 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:44:29.864 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:44:29.865 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:44:29.416287Z", 2022-09-12T19:44:29.866 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:44:29.867 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:29.867 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:44:29.867 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:44:29.868 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:44:29.868 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:44:29.869 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:44:29.869 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:44:29.870 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:44:29.870 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:44:29.871 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:44:29.871 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:44:29.871 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:44:29.872 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:29.872 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:44:29.872 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:44:29.872 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:44:29.873 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:44:29.873 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:44:29.873 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:44:29.874 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:44:29.874 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:44:29Z" 2022-09-12T19:44:29.874 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:44:29.875 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011869.416287_676.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:44:29.416287Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:44:29Z" } 2022-09-12T19:44:29.876 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011869.416287_676.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:44:29.416287Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:44:29.876 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011869.416287_676.1', 3: ''} 2022-09-12T19:44:29.877 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011869.416287_676.1', 3: ''} 2022-09-12T19:44:29.877 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T19:44:29.878 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T19:44:29.878 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:44:29.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:44:29.880 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:44:29.932 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:44:29.934 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:44:29.980 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:44:29.969+0000 7f9cf1270600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:44:29.981 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:44:29.970+0000 7f9cf1270600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:44:29.991 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:44:29.992 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:44:29.992 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:44:29.993 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:44:29.993 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:44:29.994 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:44:29.994 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:44:29.995 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:44:29.995 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:44:29.996 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:44:29.996 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:44:29.996 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:44:29.997 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:44:29.997 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:44:29.998 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:44:29.998 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:44:29.999 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:44:29.999 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:44:30.000 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:44:30.000 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:44:30.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:44:30.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:44:30.002 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:44:30.002 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:44:30.003 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:44:30.003 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:44:30.004 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:44:30.004 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:44:30.005 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:44:30.005 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:44:30.006 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:44:30.006 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:44:30.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:44:30.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:44:30.008 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:44:30.009 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:44:30.009 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:44:30.010 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663011868.672853_671.1", 2022-09-12T19:44:30.010 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:44:30.011 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:44:30.011 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:44:30.012 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:44:28.672853Z", 2022-09-12T19:44:30.012 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:44:30.013 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:44:30.013 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:44:30.013 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:44:30.014 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:44:30.014 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:44:30.015 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:44:30.015 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:44:30.016 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:44:30.016 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:44:30.017 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:44:30.017 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:44:30.018 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:44:30.018 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:44:30.019 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:44:30.020 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:44:30.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:44:30.021 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:44:30.022 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:44:30.023 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:44:30.023 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:44:30.024 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:44:29Z" 2022-09-12T19:44:30.024 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:44:30.025 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011868.672853_671.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:44:28.672853Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:44:29Z" } 2022-09-12T19:44:30.026 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011868.672853_671.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:44:28.672853Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:44:30.027 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011869.416287_676.1', 3: ''} 2022-09-12T19:44:30.028 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011868.672853_671.1', 3: ''} 2022-09-12T19:44:30.029 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1663011869.416287_676.1 target=1_1663011868.672853_671.1 2022-09-12T19:44:35.031 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:44:35.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:44:35.034 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:44:35.092 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:44:35.093 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:44:35.142 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:44:35.140+0000 7f532a01d600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:44:35.143 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:44:35.142+0000 7f532a01d600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:44:35.164 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:44:35.165 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:44:35.165 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:44:35.165 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:44:35.166 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:44:35.166 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:44:35.166 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:44:35.167 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:44:35.167 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:44:35.167 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:44:35.167 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:44:35.168 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:44:35.168 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:44:35.168 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:44:35.168 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:44:35.169 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:44:35.169 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:44:35.169 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:44:35.170 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:44:35.170 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:44:35.170 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:44:35.170 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:44:35.171 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:44:35.171 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:44:35.171 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:44:35.172 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:44:35.172 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:44:35.172 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:44:35.172 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:44:35.173 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:44:35.173 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:44:35.173 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:44:35.173 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:44:35.174 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:44:35.174 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:44:35.174 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:44:35.175 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:44:35.175 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663011869.416287_676.1", 2022-09-12T19:44:35.175 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:44:35.176 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:44:35.176 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:44:35.177 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:44:29.416287Z", 2022-09-12T19:44:35.177 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:44:35.178 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:44:35.178 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:44:35.178 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:44:35.178 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:44:35.179 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:44:35.179 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:44:35.179 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:44:35.179 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:44:35.180 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:44:35.180 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:44:35.180 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:44:35.181 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:44:35.181 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:44:35.181 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:44:35.181 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:44:35.182 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:44:35.182 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:44:35.182 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:44:35.182 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:44:35.183 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:44:35.183 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:44:35Z" 2022-09-12T19:44:35.183 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:44:35.183 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663011869.416287_676.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:44:29.416287Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:44:35Z" } 2022-09-12T19:44:35.185 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663011869.416287_676.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:44:29.416287Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:44:35.185 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663011869.416287_676.1', 3: ''} 2022-09-12T19:44:35.186 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663011869.416287_676.1', 3: ''} 2022-09-12T19:44:35.186 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T19:44:35.187 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:44:35.188 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:44:35.188 DEBUG:boto:Method: HEAD 2022-09-12T19:44:35.189 DEBUG:boto:Path: /jdpiak-72/ 2022-09-12T19:44:35.189 DEBUG:boto:Data: 2022-09-12T19:44:35.190 DEBUG:boto:Headers: {} 2022-09-12T19:44:35.190 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:35.191 DEBUG:boto:Port: 8000 2022-09-12T19:44:35.191 DEBUG:boto:Params: {} 2022-09-12T19:44:35.192 DEBUG:boto:Token: None 2022-09-12T19:44:35.193 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:35 GMT /jdpiak-72/ 2022-09-12T19:44:35.194 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:evmWZQyMemG/YbJ7x7qwI/GtX5M= 2022-09-12T19:44:35.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:evmWZQyMemG/YbJ7x7qwI/GtX5M=', 'Content-Length': '0'} 2022-09-12T19:44:35.197 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002699ed9eed17b7da-00631f8c23-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:35.198 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:44:35.199 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:44:35.199 DEBUG:boto:Method: PUT 2022-09-12T19:44:35.200 DEBUG:boto:Path: /jdpiak-72/a 2022-09-12T19:44:35.200 DEBUG:boto:Data: 2022-09-12T19:44:35.201 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:44:35.201 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:35.202 DEBUG:boto:Port: 8000 2022-09-12T19:44:35.202 DEBUG:boto:Params: {} 2022-09-12T19:44:35.203 DEBUG:boto:Token: None 2022-09-12T19:44:35.203 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:44:35 GMT /jdpiak-72/a 2022-09-12T19:44:35.204 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4cR/9cnijSQ7hOFgG/SRiFtBITo= 2022-09-12T19:44:35.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:44:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4cR/9cnijSQ7hOFgG/SRiFtBITo='} 2022-09-12T19:44:35.214 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008faa8c38cd101262-00631f8c23-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:35.215 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:44:35.216 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:44:35.217 DEBUG:boto:Method: HEAD 2022-09-12T19:44:35.218 DEBUG:boto:Path: /jdpiak-72/ 2022-09-12T19:44:35.219 DEBUG:boto:Data: 2022-09-12T19:44:35.220 DEBUG:boto:Headers: {} 2022-09-12T19:44:35.220 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:35.221 DEBUG:boto:Port: 8000 2022-09-12T19:44:35.222 DEBUG:boto:Params: {} 2022-09-12T19:44:35.223 DEBUG:boto:Token: None 2022-09-12T19:44:35.223 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:35 GMT /jdpiak-72/ 2022-09-12T19:44:35.224 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:evmWZQyMemG/YbJ7x7qwI/GtX5M= 2022-09-12T19:44:35.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:evmWZQyMemG/YbJ7x7qwI/GtX5M=', 'Content-Length': '0'} 2022-09-12T19:44:35.228 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012650bb706345c6a-00631f8c23-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:35.228 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:44:35.229 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:44:35.230 DEBUG:boto:Method: PUT 2022-09-12T19:44:35.230 DEBUG:boto:Path: /jdpiak-72/b 2022-09-12T19:44:35.231 DEBUG:boto:Data: 2022-09-12T19:44:35.232 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:44:35.232 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:35.233 DEBUG:boto:Port: 8000 2022-09-12T19:44:35.233 DEBUG:boto:Params: {} 2022-09-12T19:44:35.234 DEBUG:boto:Token: None 2022-09-12T19:44:35.235 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:44:35 GMT /jdpiak-72/b 2022-09-12T19:44:35.235 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2IBzgm4yFT2f9VvioI0kT3DOs/Y= 2022-09-12T19:44:35.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:44:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2IBzgm4yFT2f9VvioI0kT3DOs/Y='} 2022-09-12T19:44:35.282 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f59ead7eb2a9f1ca-00631f8c23-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:35.283 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:44:35.284 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:44:35.284 DEBUG:boto:Method: HEAD 2022-09-12T19:44:35.285 DEBUG:boto:Path: /jdpiak-72/ 2022-09-12T19:44:35.285 DEBUG:boto:Data: 2022-09-12T19:44:35.286 DEBUG:boto:Headers: {} 2022-09-12T19:44:35.286 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:35.287 DEBUG:boto:Port: 8000 2022-09-12T19:44:35.287 DEBUG:boto:Params: {} 2022-09-12T19:44:35.288 DEBUG:boto:Token: None 2022-09-12T19:44:35.288 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:35 GMT /jdpiak-72/ 2022-09-12T19:44:35.289 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:evmWZQyMemG/YbJ7x7qwI/GtX5M= 2022-09-12T19:44:35.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:evmWZQyMemG/YbJ7x7qwI/GtX5M=', 'Content-Length': '0'} 2022-09-12T19:44:35.292 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a11ced8ddfeec5d9-00631f8c23-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:35.293 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:44:35.294 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:44:35.294 DEBUG:boto:Method: PUT 2022-09-12T19:44:35.295 DEBUG:boto:Path: /jdpiak-72/c 2022-09-12T19:44:35.295 DEBUG:boto:Data: 2022-09-12T19:44:35.296 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:44:35.296 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:35.297 DEBUG:boto:Port: 8000 2022-09-12T19:44:35.297 DEBUG:boto:Params: {} 2022-09-12T19:44:35.298 DEBUG:boto:Token: None 2022-09-12T19:44:35.299 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:44:35 GMT /jdpiak-72/c 2022-09-12T19:44:35.300 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5vymWZvylOzmbPCKbpKekgfyHxQ= 2022-09-12T19:44:35.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:44:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5vymWZvylOzmbPCKbpKekgfyHxQ='} 2022-09-12T19:44:35.307 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bd9685c3d3f8c650-00631f8c23-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:35.308 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:44:35.309 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:44:35.310 DEBUG:boto:Method: HEAD 2022-09-12T19:44:35.310 DEBUG:boto:Path: /jdpiak-72/ 2022-09-12T19:44:35.312 DEBUG:boto:Data: 2022-09-12T19:44:35.312 DEBUG:boto:Headers: {} 2022-09-12T19:44:35.313 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:35.314 DEBUG:boto:Port: 8000 2022-09-12T19:44:35.315 DEBUG:boto:Params: {} 2022-09-12T19:44:35.316 DEBUG:boto:Token: None 2022-09-12T19:44:35.317 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:44:35 GMT /jdpiak-72/ 2022-09-12T19:44:35.318 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:evmWZQyMemG/YbJ7x7qwI/GtX5M= 2022-09-12T19:44:35.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:44:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:evmWZQyMemG/YbJ7x7qwI/GtX5M=', 'Content-Length': '0'} 2022-09-12T19:44:35.322 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b9dc11d856c1bdea-00631f8c23-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:35.323 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:44:35.324 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:44:35.324 DEBUG:boto:Method: PUT 2022-09-12T19:44:35.325 DEBUG:boto:Path: /jdpiak-72/d 2022-09-12T19:44:35.326 DEBUG:boto:Data: 2022-09-12T19:44:35.327 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:44:35.328 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:44:35.328 DEBUG:boto:Port: 8000 2022-09-12T19:44:35.329 DEBUG:boto:Params: {} 2022-09-12T19:44:35.330 DEBUG:boto:Token: None 2022-09-12T19:44:35.331 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:44:35 GMT /jdpiak-72/d 2022-09-12T19:44:35.332 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UCCxx8yTwyzS9KYYu90DcP+idck= 2022-09-12T19:44:35.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:44:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UCCxx8yTwyzS9KYYu90DcP+idck='} 2022-09-12T19:44:35.340 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007957c768fe6c240f-00631f8c23-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:44:35.341 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-72 2022-09-12T19:44:35.341 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:44:35.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:44:35.343 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-72 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:44:35.401 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:44:35.402 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:44:35.438 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:44:35.436+0000 7f5a6dc08600 1 waiting to reach incremental sync.. 2022-09-12T19:44:40.438 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:44:40.436+0000 7f5a6dc08600 1 waiting to reach incremental sync.. 2022-09-12T19:44:45.439 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:44:45.436+0000 7f5a6dc08600 1 waiting to reach incremental sync.. 2022-09-12T19:44:50.440 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:44:50.437+0000 7f5a6dc08600 1 waiting to reach incremental sync.. 2022-09-12T19:44:55.443 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:44:55.438+0000 7f5a6dc08600 1 bucket sync caught up with source: 2022-09-12T19:44:55.443 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.36.5, , , , 00000000001.39.5, , , 00000000001.59.5, , 00000000001.53.5] 2022-09-12T19:44:55.444 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.39.5, , , 00000000001.59.5, , 00000000001.53.5] 2022-09-12T19:44:55.444 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:44:55.438+0000 7f5a6dc08600 0 bucket checkpoint complete 2022-09-12T19:44:55.455 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-72 2022-09-12T19:44:55.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:44:55.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:44:55.458 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-72 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:44:55.525 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:44:55.525 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:44:55.564 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:44:55.561+0000 7f1328997600 1 bucket sync caught up with source: 2022-09-12T19:44:55.564 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.36.5, , , , 00000000001.39.5, , , 00000000001.59.5, , 00000000001.53.5] 2022-09-12T19:44:55.565 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.39.5, , , 00000000001.59.5, , 00000000001.53.5] 2022-09-12T19:44:55.566 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:44:55.561+0000 7f1328997600 0 bucket checkpoint complete 2022-09-12T19:44:55.578 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-72 2022-09-12T19:44:55.580 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:44:55.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:44:55.582 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-72 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:44:55.640 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:44:55.641 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:44:55.677 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:44:55.675+0000 7f2940fc7600 1 bucket sync caught up with source: 2022-09-12T19:44:55.678 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.36.5, , , , 00000000001.39.5, , , 00000000001.59.5, , 00000000001.53.5] 2022-09-12T19:44:55.678 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.39.5, , , 00000000001.59.5, , 00000000001.53.5] 2022-09-12T19:44:55.678 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:44:55.675+0000 7f2940fc7600 0 bucket checkpoint complete 2022-09-12T19:44:55.691 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-72 2022-09-12T19:44:55.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:44:55.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:44:55.694 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-72 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:44:55.753 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:44:55.753 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:44:55.789 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:44:55.786+0000 7f0900407600 1 waiting to reach incremental sync.. 2022-09-12T19:45:00.789 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:45:00.787+0000 7f0900407600 1 waiting to reach incremental sync.. 2022-09-12T19:45:05.791 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:45:05.788+0000 7f0900407600 1 waiting to reach incremental sync.. 2022-09-12T19:45:10.791 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:45:10.788+0000 7f0900407600 1 waiting to reach incremental sync.. 2022-09-12T19:45:15.798 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:45:15.790+0000 7f0900407600 1 bucket sync caught up with source: 2022-09-12T19:45:15.799 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.617.5, , , , 00000000001.629.5, , , 00000000001.906.5, , 00000000001.496.5] 2022-09-12T19:45:15.800 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000001.617.5, , , , 00000000001.629.5, , , 00000000001.906.5, , 00000000001.496.5] 2022-09-12T19:45:15.801 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:45:15.790+0000 7f0900407600 0 bucket checkpoint complete 2022-09-12T19:45:15.823 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-72 2022-09-12T19:45:15.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:45:15.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:45:15.826 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-72 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:45:15.902 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:45:15.902 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:45:15.949 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:45:15.946+0000 7f97f15e7600 1 bucket sync caught up with source: 2022-09-12T19:45:15.950 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.617.5, , , , 00000000001.629.5, , , 00000000001.906.5, , 00000000001.496.5] 2022-09-12T19:45:15.950 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000001.617.5, , , , 00000000001.629.5, , , 00000000001.906.5, , 00000000001.496.5] 2022-09-12T19:45:15.951 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:45:15.946+0000 7f97f15e7600 0 bucket checkpoint complete 2022-09-12T19:45:15.966 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-72 2022-09-12T19:45:15.967 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-72 2022-09-12T19:45:15.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:45:15.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:45:15.969 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-72 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:45:16.040 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:45:16.041 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:45:16.085 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:45:16.082+0000 7fd15ed54600 1 bucket sync caught up with source: 2022-09-12T19:45:16.086 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.254.5, , , , 00000000001.281.5, , , 00000000001.433.5, , 00000000001.321.5] 2022-09-12T19:45:16.086 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000001.254.5, , , , 00000000001.281.5, , , 00000000001.433.5, , 00000000001.321.5] 2022-09-12T19:45:16.086 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:45:16.082+0000 7fd15ed54600 0 bucket checkpoint complete 2022-09-12T19:45:16.099 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-72 2022-09-12T19:45:16.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:45:16.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:45:16.102 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-72 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:45:16.167 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:45:16.168 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:45:16.218 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:45:16.215+0000 7ff6c85cf600 1 bucket sync caught up with source: 2022-09-12T19:45:16.220 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.254.5, , , , 00000000001.281.5, , , 00000000001.433.5, , 00000000001.321.5] 2022-09-12T19:45:16.221 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.254.5, , , , 00000000001.281.5, , , 00000000001.433.5, , 00000000001.321.5] 2022-09-12T19:45:16.223 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:45:16.215+0000 7ff6c85cf600 0 bucket checkpoint complete 2022-09-12T19:45:16.239 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-72 2022-09-12T19:45:16.240 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-72 zones={test-zone2, test-zone1} 2022-09-12T19:45:16.240 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:16.241 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:16.242 DEBUG:boto:Method: HEAD 2022-09-12T19:45:16.242 DEBUG:boto:Path: /jdpiak-72/ 2022-09-12T19:45:16.243 DEBUG:boto:Data: 2022-09-12T19:45:16.243 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.244 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:16.245 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.245 DEBUG:boto:Params: {} 2022-09-12T19:45:16.246 DEBUG:boto:Token: None 2022-09-12T19:45:16.247 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/ 2022-09-12T19:45:16.247 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nT/xLW73psBraaQHd/iVt+DTi6U= 2022-09-12T19:45:16.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nT/xLW73psBraaQHd/iVt+DTi6U=', 'Content-Length': '0'} 2022-09-12T19:45:16.257 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ccc4c71be2c7c15-00631f8c4c-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.257 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:16.258 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:16.259 DEBUG:boto:Method: HEAD 2022-09-12T19:45:16.259 DEBUG:boto:Path: /jdpiak-72/ 2022-09-12T19:45:16.260 DEBUG:boto:Data: 2022-09-12T19:45:16.261 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.261 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:16.262 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.262 DEBUG:boto:Params: {} 2022-09-12T19:45:16.263 DEBUG:boto:Token: None 2022-09-12T19:45:16.264 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/ 2022-09-12T19:45:16.270 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nT/xLW73psBraaQHd/iVt+DTi6U= 2022-09-12T19:45:16.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nT/xLW73psBraaQHd/iVt+DTi6U=', 'Content-Length': '0'} 2022-09-12T19:45:16.281 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000333318a572628827-00631f8c4c-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.282 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:45:16.283 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:16.284 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:16.284 DEBUG:boto:path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:16.285 DEBUG:boto:auth_path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:16.286 DEBUG:boto:Method: GET 2022-09-12T19:45:16.286 DEBUG:boto:Path: /jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:16.287 DEBUG:boto:Data: 2022-09-12T19:45:16.288 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.288 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:16.289 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.289 DEBUG:boto:Params: {} 2022-09-12T19:45:16.290 DEBUG:boto:Token: None 2022-09-12T19:45:16.291 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/?versions 2022-09-12T19:45:16.292 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ekjhPssd3DrN+IpWL4k3uSLVfAo= 2022-09-12T19:45:16.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ekjhPssd3DrN+IpWL4k3uSLVfAo=', 'Content-Length': '0'} 2022-09-12T19:45:16.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d88acb4a9e299349-00631f8c4c-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.304 DEBUG:boto:b'jdpiak-72999falseanulltrue2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:16.306 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:45:16.307 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:45:16.309 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:45:16.309 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:45:16.310 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:45:16.310 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:16.311 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:16.312 DEBUG:boto:path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:16.313 DEBUG:boto:auth_path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:16.313 DEBUG:boto:Method: GET 2022-09-12T19:45:16.314 DEBUG:boto:Path: /jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:16.314 DEBUG:boto:Data: 2022-09-12T19:45:16.315 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.316 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:16.316 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.317 DEBUG:boto:Params: {} 2022-09-12T19:45:16.318 DEBUG:boto:Token: None 2022-09-12T19:45:16.318 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/?versions 2022-09-12T19:45:16.319 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ekjhPssd3DrN+IpWL4k3uSLVfAo= 2022-09-12T19:45:16.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ekjhPssd3DrN+IpWL4k3uSLVfAo=', 'Content-Length': '0'} 2022-09-12T19:45:16.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081bd91e602772f1b-00631f8c4c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.331 DEBUG:boto:b'jdpiak-72999falseanulltrue2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:16.332 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:45:16.333 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:45:16.333 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:45:16.334 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:45:16.334 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:16.335 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:16.335 DEBUG:boto:path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:16.336 DEBUG:boto:auth_path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:16.336 DEBUG:boto:Method: GET 2022-09-12T19:45:16.337 DEBUG:boto:Path: /jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:16.337 DEBUG:boto:Data: 2022-09-12T19:45:16.338 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.338 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:16.339 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.340 DEBUG:boto:Params: {} 2022-09-12T19:45:16.341 DEBUG:boto:Token: None 2022-09-12T19:45:16.342 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/?versions 2022-09-12T19:45:16.343 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ekjhPssd3DrN+IpWL4k3uSLVfAo= 2022-09-12T19:45:16.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ekjhPssd3DrN+IpWL4k3uSLVfAo=', 'Content-Length': '0'} 2022-09-12T19:45:16.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026b5c80735e81206-00631f8c4c-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.352 DEBUG:boto:b'jdpiak-72999falseanulltrue2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:16.353 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:16.354 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:16.354 DEBUG:boto:path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:16.355 DEBUG:boto:auth_path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:16.356 DEBUG:boto:Method: GET 2022-09-12T19:45:16.356 DEBUG:boto:Path: /jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:16.358 DEBUG:boto:Data: 2022-09-12T19:45:16.358 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.359 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:16.360 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.360 DEBUG:boto:Params: {} 2022-09-12T19:45:16.361 DEBUG:boto:Token: None 2022-09-12T19:45:16.362 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/?versions 2022-09-12T19:45:16.363 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ekjhPssd3DrN+IpWL4k3uSLVfAo= 2022-09-12T19:45:16.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ekjhPssd3DrN+IpWL4k3uSLVfAo=', 'Content-Length': '0'} 2022-09-12T19:45:16.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dda668e439da57f-00631f8c4c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.373 DEBUG:boto:b'jdpiak-72999falseanulltrue2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:16.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:45:16.376 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:16.377 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:16.379 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:16.379 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:16.380 DEBUG:boto:Method: GET 2022-09-12T19:45:16.380 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:45:16.381 DEBUG:boto:Data: 2022-09-12T19:45:16.382 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.383 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:16.384 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.385 DEBUG:boto:Params: {} 2022-09-12T19:45:16.387 DEBUG:boto:Token: None 2022-09-12T19:45:16.388 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:45:16.390 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7sy11XXWtxo1DKAm982S1f7gJAM= 2022-09-12T19:45:16.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7sy11XXWtxo1DKAm982S1f7gJAM=', 'Content-Length': '0'} 2022-09-12T19:45:16.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e52af407394ddc39-00631f8c4c-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.397 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:16.398 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:16.399 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:16.399 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:16.400 DEBUG:boto:Method: GET 2022-09-12T19:45:16.401 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:45:16.401 DEBUG:boto:Data: 2022-09-12T19:45:16.402 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.403 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:16.404 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.404 DEBUG:boto:Params: {} 2022-09-12T19:45:16.405 DEBUG:boto:Token: None 2022-09-12T19:45:16.406 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:45:16.407 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7sy11XXWtxo1DKAm982S1f7gJAM= 2022-09-12T19:45:16.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7sy11XXWtxo1DKAm982S1f7gJAM=', 'Content-Length': '0'} 2022-09-12T19:45:16.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddfeb77e838a3a43-00631f8c4c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.466 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:16.467 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:16.468 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:16.469 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:16.470 DEBUG:boto:Method: HEAD 2022-09-12T19:45:16.471 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:45:16.472 DEBUG:boto:Data: 2022-09-12T19:45:16.473 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.474 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:16.475 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.476 DEBUG:boto:Params: {} 2022-09-12T19:45:16.477 DEBUG:boto:Token: None 2022-09-12T19:45:16.478 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:45:16.479 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rxWrk4B2StHs+QzDiq1Q4XYj2GQ= 2022-09-12T19:45:16.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rxWrk4B2StHs+QzDiq1Q4XYj2GQ=', 'Content-Length': '0'} 2022-09-12T19:45:16.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b64dc585fe9172eb-00631f8c4c-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.488 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:16.489 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:16.490 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:16.490 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:16.491 DEBUG:boto:Method: HEAD 2022-09-12T19:45:16.491 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:45:16.492 DEBUG:boto:Data: 2022-09-12T19:45:16.492 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.493 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:16.494 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.494 DEBUG:boto:Params: {} 2022-09-12T19:45:16.495 DEBUG:boto:Token: None 2022-09-12T19:45:16.496 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:45:16.496 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rxWrk4B2StHs+QzDiq1Q4XYj2GQ= 2022-09-12T19:45:16.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rxWrk4B2StHs+QzDiq1Q4XYj2GQ=', 'Content-Length': '0'} 2022-09-12T19:45:16.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000146fa010072ccb3c-00631f8c4c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:45:16.509 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:16.510 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:16.511 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:16.512 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:16.513 DEBUG:boto:Method: GET 2022-09-12T19:45:16.514 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:45:16.515 DEBUG:boto:Data: 2022-09-12T19:45:16.516 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.517 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:16.518 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.519 DEBUG:boto:Params: {} 2022-09-12T19:45:16.520 DEBUG:boto:Token: None 2022-09-12T19:45:16.521 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:45:16.522 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7sy11XXWtxo1DKAm982S1f7gJAM= 2022-09-12T19:45:16.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7sy11XXWtxo1DKAm982S1f7gJAM=', 'Content-Length': '0'} 2022-09-12T19:45:16.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000310972b7026f837e-00631f8c4c-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.566 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:16.567 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:16.569 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:16.570 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:16.571 DEBUG:boto:Method: GET 2022-09-12T19:45:16.571 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:45:16.572 DEBUG:boto:Data: 2022-09-12T19:45:16.572 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.573 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:16.574 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.574 DEBUG:boto:Params: {} 2022-09-12T19:45:16.575 DEBUG:boto:Token: None 2022-09-12T19:45:16.576 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:45:16.576 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7sy11XXWtxo1DKAm982S1f7gJAM= 2022-09-12T19:45:16.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7sy11XXWtxo1DKAm982S1f7gJAM=', 'Content-Length': '0'} 2022-09-12T19:45:16.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a49305524e3b35b3-00631f8c4c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.586 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:16.587 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:16.587 DEBUG:boto:path=/jdpiak-72/?prefix=a 2022-09-12T19:45:16.588 DEBUG:boto:auth_path=/jdpiak-72/?prefix=a 2022-09-12T19:45:16.589 DEBUG:boto:Method: GET 2022-09-12T19:45:16.589 DEBUG:boto:Path: /jdpiak-72/?prefix=a 2022-09-12T19:45:16.590 DEBUG:boto:Data: 2022-09-12T19:45:16.591 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.591 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:16.592 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.593 DEBUG:boto:Params: {} 2022-09-12T19:45:16.593 DEBUG:boto:Token: None 2022-09-12T19:45:16.594 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/ 2022-09-12T19:45:16.595 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6+wvvbnvg7sTFFG7G37YaYI2xls= 2022-09-12T19:45:16.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6+wvvbnvg7sTFFG7G37YaYI2xls=', 'Content-Length': '0'} 2022-09-12T19:45:16.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085529ac34f14d9a9-00631f8c4c-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.601 DEBUG:boto:b'jdpiak-72a1000falsea2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:16.603 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:16.604 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:16.605 DEBUG:boto:path=/jdpiak-72/?prefix=a 2022-09-12T19:45:16.606 DEBUG:boto:auth_path=/jdpiak-72/?prefix=a 2022-09-12T19:45:16.607 DEBUG:boto:Method: GET 2022-09-12T19:45:16.608 DEBUG:boto:Path: /jdpiak-72/?prefix=a 2022-09-12T19:45:16.609 DEBUG:boto:Data: 2022-09-12T19:45:16.610 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.611 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:16.612 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.612 DEBUG:boto:Params: {} 2022-09-12T19:45:16.613 DEBUG:boto:Token: None 2022-09-12T19:45:16.615 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/ 2022-09-12T19:45:16.616 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6+wvvbnvg7sTFFG7G37YaYI2xls= 2022-09-12T19:45:16.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6+wvvbnvg7sTFFG7G37YaYI2xls=', 'Content-Length': '0'} 2022-09-12T19:45:16.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a35e256e666b127-00631f8c4c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.626 DEBUG:boto:b'jdpiak-72a1000falsea2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:16.627 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:16.628 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:16.629 DEBUG:boto:Method: HEAD 2022-09-12T19:45:16.630 DEBUG:boto:Path: /jdpiak-72/a 2022-09-12T19:45:16.631 DEBUG:boto:Data: 2022-09-12T19:45:16.632 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.632 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:16.633 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.634 DEBUG:boto:Params: {} 2022-09-12T19:45:16.634 DEBUG:boto:Token: None 2022-09-12T19:45:16.635 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/a 2022-09-12T19:45:16.635 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A3R3t78RAvnnjUA3Ov5TgkTwy0M= 2022-09-12T19:45:16.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A3R3t78RAvnnjUA3Ov5TgkTwy0M=', 'Content-Length': '0'} 2022-09-12T19:45:16.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef60413ef339e0e2-00631f8c4c-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.643 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:16.644 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:16.645 DEBUG:boto:Method: HEAD 2022-09-12T19:45:16.646 DEBUG:boto:Path: /jdpiak-72/a 2022-09-12T19:45:16.647 DEBUG:boto:Data: 2022-09-12T19:45:16.648 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.649 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:16.650 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.651 DEBUG:boto:Params: {} 2022-09-12T19:45:16.653 DEBUG:boto:Token: None 2022-09-12T19:45:16.654 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/a 2022-09-12T19:45:16.655 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A3R3t78RAvnnjUA3Ov5TgkTwy0M= 2022-09-12T19:45:16.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A3R3t78RAvnnjUA3Ov5TgkTwy0M=', 'Content-Length': '0'} 2022-09-12T19:45:16.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3f6e9a3bef0e922-00631f8c4c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:45:16.665 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:16.666 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:16.667 DEBUG:boto:Method: GET 2022-09-12T19:45:16.668 DEBUG:boto:Path: /jdpiak-72/a 2022-09-12T19:45:16.669 DEBUG:boto:Data: 2022-09-12T19:45:16.670 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.671 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:16.672 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.673 DEBUG:boto:Params: {} 2022-09-12T19:45:16.675 DEBUG:boto:Token: None 2022-09-12T19:45:16.676 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/a 2022-09-12T19:45:16.677 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rtUHONSPx/8Te/w6ChGwCLP9EFA= 2022-09-12T19:45:16.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rtUHONSPx/8Te/w6ChGwCLP9EFA=', 'Content-Length': '0'} 2022-09-12T19:45:16.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001194a1f1a3d5b7fd-00631f8c4c-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.684 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:16.685 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:16.686 DEBUG:boto:Method: GET 2022-09-12T19:45:16.687 DEBUG:boto:Path: /jdpiak-72/a 2022-09-12T19:45:16.688 DEBUG:boto:Data: 2022-09-12T19:45:16.689 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.690 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:16.691 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.692 DEBUG:boto:Params: {} 2022-09-12T19:45:16.694 DEBUG:boto:Token: None 2022-09-12T19:45:16.695 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/a 2022-09-12T19:45:16.696 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rtUHONSPx/8Te/w6ChGwCLP9EFA= 2022-09-12T19:45:16.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rtUHONSPx/8Te/w6ChGwCLP9EFA=', 'Content-Length': '0'} 2022-09-12T19:45:16.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec7ab122241f5e87-00631f8c4c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.747 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:16.750 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:16.752 DEBUG:boto:path=/jdpiak-72/?prefix=a 2022-09-12T19:45:16.754 DEBUG:boto:auth_path=/jdpiak-72/?prefix=a 2022-09-12T19:45:16.755 DEBUG:boto:Method: GET 2022-09-12T19:45:16.756 DEBUG:boto:Path: /jdpiak-72/?prefix=a 2022-09-12T19:45:16.758 DEBUG:boto:Data: 2022-09-12T19:45:16.759 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.762 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:16.764 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.766 DEBUG:boto:Params: {} 2022-09-12T19:45:16.767 DEBUG:boto:Token: None 2022-09-12T19:45:16.767 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/ 2022-09-12T19:45:16.769 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6+wvvbnvg7sTFFG7G37YaYI2xls= 2022-09-12T19:45:16.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6+wvvbnvg7sTFFG7G37YaYI2xls=', 'Content-Length': '0'} 2022-09-12T19:45:16.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088427ce36922db92-00631f8c4c-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.775 DEBUG:boto:b'jdpiak-72a1000falsea2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:16.777 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:16.778 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:16.779 DEBUG:boto:path=/jdpiak-72/?prefix=a 2022-09-12T19:45:16.780 DEBUG:boto:auth_path=/jdpiak-72/?prefix=a 2022-09-12T19:45:16.781 DEBUG:boto:Method: GET 2022-09-12T19:45:16.783 DEBUG:boto:Path: /jdpiak-72/?prefix=a 2022-09-12T19:45:16.784 DEBUG:boto:Data: 2022-09-12T19:45:16.785 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.787 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:16.788 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.789 DEBUG:boto:Params: {} 2022-09-12T19:45:16.791 DEBUG:boto:Token: None 2022-09-12T19:45:16.791 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/ 2022-09-12T19:45:16.792 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6+wvvbnvg7sTFFG7G37YaYI2xls= 2022-09-12T19:45:16.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6+wvvbnvg7sTFFG7G37YaYI2xls=', 'Content-Length': '0'} 2022-09-12T19:45:16.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a89262b0a2d3929e-00631f8c4c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.801 DEBUG:boto:b'jdpiak-72a1000falsea2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:16.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:45:16.803 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:16.804 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:16.805 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:16.806 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:16.807 DEBUG:boto:Method: GET 2022-09-12T19:45:16.808 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:45:16.809 DEBUG:boto:Data: 2022-09-12T19:45:16.809 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.810 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:16.811 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.812 DEBUG:boto:Params: {} 2022-09-12T19:45:16.813 DEBUG:boto:Token: None 2022-09-12T19:45:16.814 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:45:16.815 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tw4vNmpiBKHsd/TWcDck1zQY9ss= 2022-09-12T19:45:16.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tw4vNmpiBKHsd/TWcDck1zQY9ss=', 'Content-Length': '0'} 2022-09-12T19:45:16.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7df8cd10bb07135-00631f8c4c-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.822 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:16.824 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:16.825 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:16.826 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:16.827 DEBUG:boto:Method: GET 2022-09-12T19:45:16.828 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:45:16.830 DEBUG:boto:Data: 2022-09-12T19:45:16.831 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.832 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:16.833 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.834 DEBUG:boto:Params: {} 2022-09-12T19:45:16.835 DEBUG:boto:Token: None 2022-09-12T19:45:16.836 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:45:16.837 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tw4vNmpiBKHsd/TWcDck1zQY9ss= 2022-09-12T19:45:16.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tw4vNmpiBKHsd/TWcDck1zQY9ss=', 'Content-Length': '0'} 2022-09-12T19:45:16.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d681c25f67fc7a9f-00631f8c4c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.856 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:16.857 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:16.858 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:16.859 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:16.860 DEBUG:boto:Method: HEAD 2022-09-12T19:45:16.861 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:45:16.863 DEBUG:boto:Data: 2022-09-12T19:45:16.864 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.865 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:16.866 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.867 DEBUG:boto:Params: {} 2022-09-12T19:45:16.868 DEBUG:boto:Token: None 2022-09-12T19:45:16.869 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:45:16.870 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:d5utiVFS/TAwPURVHBuFSHfHtlo= 2022-09-12T19:45:16.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:d5utiVFS/TAwPURVHBuFSHfHtlo=', 'Content-Length': '0'} 2022-09-12T19:45:16.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025039f068d6cf4a4-00631f8c4c-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.876 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:16.877 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:16.878 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:16.879 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:16.880 DEBUG:boto:Method: HEAD 2022-09-12T19:45:16.880 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:45:16.881 DEBUG:boto:Data: 2022-09-12T19:45:16.883 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.883 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:16.884 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.885 DEBUG:boto:Params: {} 2022-09-12T19:45:16.887 DEBUG:boto:Token: None 2022-09-12T19:45:16.888 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:45:16.889 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:d5utiVFS/TAwPURVHBuFSHfHtlo= 2022-09-12T19:45:16.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:d5utiVFS/TAwPURVHBuFSHfHtlo=', 'Content-Length': '0'} 2022-09-12T19:45:16.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e279d525cf50184-00631f8c4c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:45:16.896 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:16.897 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:16.898 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:16.899 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:16.900 DEBUG:boto:Method: GET 2022-09-12T19:45:16.901 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:45:16.902 DEBUG:boto:Data: 2022-09-12T19:45:16.904 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.906 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:16.907 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.908 DEBUG:boto:Params: {} 2022-09-12T19:45:16.909 DEBUG:boto:Token: None 2022-09-12T19:45:16.910 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:45:16.911 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tw4vNmpiBKHsd/TWcDck1zQY9ss= 2022-09-12T19:45:16.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tw4vNmpiBKHsd/TWcDck1zQY9ss=', 'Content-Length': '0'} 2022-09-12T19:45:16.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4596ec7058d346a-00631f8c4c-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.917 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:16.918 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:16.919 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:16.921 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:16.922 DEBUG:boto:Method: GET 2022-09-12T19:45:16.924 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:45:16.925 DEBUG:boto:Data: 2022-09-12T19:45:16.926 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.927 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:16.928 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.929 DEBUG:boto:Params: {} 2022-09-12T19:45:16.930 DEBUG:boto:Token: None 2022-09-12T19:45:16.931 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:45:16.932 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tw4vNmpiBKHsd/TWcDck1zQY9ss= 2022-09-12T19:45:16.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tw4vNmpiBKHsd/TWcDck1zQY9ss=', 'Content-Length': '0'} 2022-09-12T19:45:16.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffdb10dd8ef852c0-00631f8c4c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.938 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:16.939 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:16.940 DEBUG:boto:path=/jdpiak-72/?prefix=b 2022-09-12T19:45:16.940 DEBUG:boto:auth_path=/jdpiak-72/?prefix=b 2022-09-12T19:45:16.941 DEBUG:boto:Method: GET 2022-09-12T19:45:16.942 DEBUG:boto:Path: /jdpiak-72/?prefix=b 2022-09-12T19:45:16.943 DEBUG:boto:Data: 2022-09-12T19:45:16.943 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.944 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:16.945 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.947 DEBUG:boto:Params: {} 2022-09-12T19:45:16.948 DEBUG:boto:Token: None 2022-09-12T19:45:16.948 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/ 2022-09-12T19:45:16.949 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6+wvvbnvg7sTFFG7G37YaYI2xls= 2022-09-12T19:45:16.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6+wvvbnvg7sTFFG7G37YaYI2xls=', 'Content-Length': '0'} 2022-09-12T19:45:16.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000627cb32ec7be4434-00631f8c4c-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.955 DEBUG:boto:b'jdpiak-72b1000falseb2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:16.956 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:16.957 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:16.958 DEBUG:boto:path=/jdpiak-72/?prefix=b 2022-09-12T19:45:16.961 DEBUG:boto:auth_path=/jdpiak-72/?prefix=b 2022-09-12T19:45:16.962 DEBUG:boto:Method: GET 2022-09-12T19:45:16.963 DEBUG:boto:Path: /jdpiak-72/?prefix=b 2022-09-12T19:45:16.964 DEBUG:boto:Data: 2022-09-12T19:45:16.965 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.966 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:16.967 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.968 DEBUG:boto:Params: {} 2022-09-12T19:45:16.968 DEBUG:boto:Token: None 2022-09-12T19:45:16.969 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/ 2022-09-12T19:45:16.970 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6+wvvbnvg7sTFFG7G37YaYI2xls= 2022-09-12T19:45:16.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6+wvvbnvg7sTFFG7G37YaYI2xls=', 'Content-Length': '0'} 2022-09-12T19:45:16.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010801e50d48c88fd-00631f8c4c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.975 DEBUG:boto:b'jdpiak-72b1000falseb2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:16.976 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:16.977 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:16.977 DEBUG:boto:Method: HEAD 2022-09-12T19:45:16.978 DEBUG:boto:Path: /jdpiak-72/b 2022-09-12T19:45:16.979 DEBUG:boto:Data: 2022-09-12T19:45:16.979 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.980 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:16.981 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.981 DEBUG:boto:Params: {} 2022-09-12T19:45:16.982 DEBUG:boto:Token: None 2022-09-12T19:45:16.983 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/b 2022-09-12T19:45:16.983 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fqIIkzwDtLm1BjF8uy+P0zaOrrM= 2022-09-12T19:45:16.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fqIIkzwDtLm1BjF8uy+P0zaOrrM=', 'Content-Length': '0'} 2022-09-12T19:45:16.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1a91f5f85f4473d-00631f8c4c-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.987 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:16.988 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:16.989 DEBUG:boto:Method: HEAD 2022-09-12T19:45:16.989 DEBUG:boto:Path: /jdpiak-72/b 2022-09-12T19:45:16.990 DEBUG:boto:Data: 2022-09-12T19:45:16.991 DEBUG:boto:Headers: {} 2022-09-12T19:45:16.991 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:16.992 DEBUG:boto:Port: 8000 2022-09-12T19:45:16.993 DEBUG:boto:Params: {} 2022-09-12T19:45:16.993 DEBUG:boto:Token: None 2022-09-12T19:45:16.994 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:16 GMT /jdpiak-72/b 2022-09-12T19:45:16.995 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fqIIkzwDtLm1BjF8uy+P0zaOrrM= 2022-09-12T19:45:16.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fqIIkzwDtLm1BjF8uy+P0zaOrrM=', 'Content-Length': '0'} 2022-09-12T19:45:16.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e1237a063bb85fa-00631f8c4c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:16.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:45:16.999 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:17.000 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:17.000 DEBUG:boto:Method: GET 2022-09-12T19:45:17.001 DEBUG:boto:Path: /jdpiak-72/b 2022-09-12T19:45:17.001 DEBUG:boto:Data: 2022-09-12T19:45:17.002 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.002 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:17.003 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.004 DEBUG:boto:Params: {} 2022-09-12T19:45:17.005 DEBUG:boto:Token: None 2022-09-12T19:45:17.006 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/b 2022-09-12T19:45:17.007 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7Q0KYjuvKypw6peYcTXX6ztKVnc= 2022-09-12T19:45:17.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7Q0KYjuvKypw6peYcTXX6ztKVnc=', 'Content-Length': '0'} 2022-09-12T19:45:17.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de5e49e60715c4b6-00631f8c4d-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.014 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:17.015 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:17.016 DEBUG:boto:Method: GET 2022-09-12T19:45:17.017 DEBUG:boto:Path: /jdpiak-72/b 2022-09-12T19:45:17.018 DEBUG:boto:Data: 2022-09-12T19:45:17.019 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.020 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:17.021 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.021 DEBUG:boto:Params: {} 2022-09-12T19:45:17.022 DEBUG:boto:Token: None 2022-09-12T19:45:17.023 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/b 2022-09-12T19:45:17.024 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7Q0KYjuvKypw6peYcTXX6ztKVnc= 2022-09-12T19:45:17.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7Q0KYjuvKypw6peYcTXX6ztKVnc=', 'Content-Length': '0'} 2022-09-12T19:45:17.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033d0c2074da2a51d-00631f8c4d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.033 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:17.034 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:17.034 DEBUG:boto:path=/jdpiak-72/?prefix=b 2022-09-12T19:45:17.035 DEBUG:boto:auth_path=/jdpiak-72/?prefix=b 2022-09-12T19:45:17.036 DEBUG:boto:Method: GET 2022-09-12T19:45:17.037 DEBUG:boto:Path: /jdpiak-72/?prefix=b 2022-09-12T19:45:17.037 DEBUG:boto:Data: 2022-09-12T19:45:17.038 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.038 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:17.039 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.039 DEBUG:boto:Params: {} 2022-09-12T19:45:17.040 DEBUG:boto:Token: None 2022-09-12T19:45:17.041 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/ 2022-09-12T19:45:17.041 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WMpeNN3/3wPBDFQvZYQCZpQUUB4= 2022-09-12T19:45:17.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WMpeNN3/3wPBDFQvZYQCZpQUUB4=', 'Content-Length': '0'} 2022-09-12T19:45:17.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006855582d54cc3744-00631f8c4d-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.096 DEBUG:boto:b'jdpiak-72b1000falseb2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:17.097 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:17.098 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:17.099 DEBUG:boto:path=/jdpiak-72/?prefix=b 2022-09-12T19:45:17.099 DEBUG:boto:auth_path=/jdpiak-72/?prefix=b 2022-09-12T19:45:17.100 DEBUG:boto:Method: GET 2022-09-12T19:45:17.100 DEBUG:boto:Path: /jdpiak-72/?prefix=b 2022-09-12T19:45:17.101 DEBUG:boto:Data: 2022-09-12T19:45:17.101 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.102 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:17.103 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.103 DEBUG:boto:Params: {} 2022-09-12T19:45:17.104 DEBUG:boto:Token: None 2022-09-12T19:45:17.105 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/ 2022-09-12T19:45:17.106 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WMpeNN3/3wPBDFQvZYQCZpQUUB4= 2022-09-12T19:45:17.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WMpeNN3/3wPBDFQvZYQCZpQUUB4=', 'Content-Length': '0'} 2022-09-12T19:45:17.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbcff95fb6bd0396-00631f8c4d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.116 DEBUG:boto:b'jdpiak-72b1000falseb2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:17.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:45:17.117 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:17.118 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:17.119 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:17.119 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:17.120 DEBUG:boto:Method: GET 2022-09-12T19:45:17.120 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:45:17.121 DEBUG:boto:Data: 2022-09-12T19:45:17.122 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.122 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:17.123 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.124 DEBUG:boto:Params: {} 2022-09-12T19:45:17.124 DEBUG:boto:Token: None 2022-09-12T19:45:17.125 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:45:17.126 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xjFAGHme2MrmpXZFuiY8RIMrCXs= 2022-09-12T19:45:17.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xjFAGHme2MrmpXZFuiY8RIMrCXs=', 'Content-Length': '0'} 2022-09-12T19:45:17.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b1f9dde44ed3dc2-00631f8c4d-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.132 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:17.132 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:17.133 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:17.133 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:17.134 DEBUG:boto:Method: GET 2022-09-12T19:45:17.134 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:45:17.135 DEBUG:boto:Data: 2022-09-12T19:45:17.135 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.136 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:17.137 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.137 DEBUG:boto:Params: {} 2022-09-12T19:45:17.138 DEBUG:boto:Token: None 2022-09-12T19:45:17.138 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:45:17.139 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xjFAGHme2MrmpXZFuiY8RIMrCXs= 2022-09-12T19:45:17.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xjFAGHme2MrmpXZFuiY8RIMrCXs=', 'Content-Length': '0'} 2022-09-12T19:45:17.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fe09c18e6cfb5a3-00631f8c4d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.146 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:17.147 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:17.147 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:17.148 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:17.149 DEBUG:boto:Method: HEAD 2022-09-12T19:45:17.149 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:45:17.150 DEBUG:boto:Data: 2022-09-12T19:45:17.150 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.151 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:17.151 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.152 DEBUG:boto:Params: {} 2022-09-12T19:45:17.152 DEBUG:boto:Token: None 2022-09-12T19:45:17.153 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:45:17.154 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p+LinvvMQh3/vFOs3uW3SEHyTr8= 2022-09-12T19:45:17.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p+LinvvMQh3/vFOs3uW3SEHyTr8=', 'Content-Length': '0'} 2022-09-12T19:45:17.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001812b59a8856e880-00631f8c4d-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.161 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:17.162 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:17.163 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:17.163 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:17.164 DEBUG:boto:Method: HEAD 2022-09-12T19:45:17.165 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:45:17.166 DEBUG:boto:Data: 2022-09-12T19:45:17.167 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.168 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:17.169 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.169 DEBUG:boto:Params: {} 2022-09-12T19:45:17.170 DEBUG:boto:Token: None 2022-09-12T19:45:17.171 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:45:17.172 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p+LinvvMQh3/vFOs3uW3SEHyTr8= 2022-09-12T19:45:17.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p+LinvvMQh3/vFOs3uW3SEHyTr8=', 'Content-Length': '0'} 2022-09-12T19:45:17.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbc0d02f10479c12-00631f8c4d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:45:17.179 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:17.180 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:17.181 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:17.181 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:17.182 DEBUG:boto:Method: GET 2022-09-12T19:45:17.182 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:45:17.183 DEBUG:boto:Data: 2022-09-12T19:45:17.183 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.184 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:17.184 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.185 DEBUG:boto:Params: {} 2022-09-12T19:45:17.186 DEBUG:boto:Token: None 2022-09-12T19:45:17.186 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:45:17.187 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xjFAGHme2MrmpXZFuiY8RIMrCXs= 2022-09-12T19:45:17.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xjFAGHme2MrmpXZFuiY8RIMrCXs=', 'Content-Length': '0'} 2022-09-12T19:45:17.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003dd01a06db79ddb9-00631f8c4d-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.240 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:17.241 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:17.242 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:17.243 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:17.244 DEBUG:boto:Method: GET 2022-09-12T19:45:17.244 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:45:17.245 DEBUG:boto:Data: 2022-09-12T19:45:17.246 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.247 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:17.247 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.248 DEBUG:boto:Params: {} 2022-09-12T19:45:17.249 DEBUG:boto:Token: None 2022-09-12T19:45:17.250 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:45:17.251 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xjFAGHme2MrmpXZFuiY8RIMrCXs= 2022-09-12T19:45:17.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xjFAGHme2MrmpXZFuiY8RIMrCXs=', 'Content-Length': '0'} 2022-09-12T19:45:17.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e68803debd4fb86a-00631f8c4d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.267 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:17.270 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:17.270 DEBUG:boto:path=/jdpiak-72/?prefix=c 2022-09-12T19:45:17.271 DEBUG:boto:auth_path=/jdpiak-72/?prefix=c 2022-09-12T19:45:17.272 DEBUG:boto:Method: GET 2022-09-12T19:45:17.273 DEBUG:boto:Path: /jdpiak-72/?prefix=c 2022-09-12T19:45:17.274 DEBUG:boto:Data: 2022-09-12T19:45:17.275 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.276 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:17.277 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.278 DEBUG:boto:Params: {} 2022-09-12T19:45:17.279 DEBUG:boto:Token: None 2022-09-12T19:45:17.280 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/ 2022-09-12T19:45:17.281 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WMpeNN3/3wPBDFQvZYQCZpQUUB4= 2022-09-12T19:45:17.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WMpeNN3/3wPBDFQvZYQCZpQUUB4=', 'Content-Length': '0'} 2022-09-12T19:45:17.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7a09bccc42ac065-00631f8c4d-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.287 DEBUG:boto:b'jdpiak-72c1000falsec2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:17.289 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:17.290 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:17.291 DEBUG:boto:path=/jdpiak-72/?prefix=c 2022-09-12T19:45:17.298 DEBUG:boto:auth_path=/jdpiak-72/?prefix=c 2022-09-12T19:45:17.299 DEBUG:boto:Method: GET 2022-09-12T19:45:17.300 DEBUG:boto:Path: /jdpiak-72/?prefix=c 2022-09-12T19:45:17.301 DEBUG:boto:Data: 2022-09-12T19:45:17.303 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.305 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:17.307 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.310 DEBUG:boto:Params: {} 2022-09-12T19:45:17.311 DEBUG:boto:Token: None 2022-09-12T19:45:17.312 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/ 2022-09-12T19:45:17.312 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WMpeNN3/3wPBDFQvZYQCZpQUUB4= 2022-09-12T19:45:17.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WMpeNN3/3wPBDFQvZYQCZpQUUB4=', 'Content-Length': '0'} 2022-09-12T19:45:17.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8a54353713bfb39-00631f8c4d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.318 DEBUG:boto:b'jdpiak-72c1000falsec2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:17.319 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:17.320 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:17.320 DEBUG:boto:Method: HEAD 2022-09-12T19:45:17.321 DEBUG:boto:Path: /jdpiak-72/c 2022-09-12T19:45:17.322 DEBUG:boto:Data: 2022-09-12T19:45:17.322 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.323 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:17.324 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.324 DEBUG:boto:Params: {} 2022-09-12T19:45:17.325 DEBUG:boto:Token: None 2022-09-12T19:45:17.326 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/c 2022-09-12T19:45:17.327 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N5i6gU2xQd0I7Sdpw/I2fbyJM1M= 2022-09-12T19:45:17.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N5i6gU2xQd0I7Sdpw/I2fbyJM1M=', 'Content-Length': '0'} 2022-09-12T19:45:17.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8a64956179b9329-00631f8c4d-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.333 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:17.334 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:17.334 DEBUG:boto:Method: HEAD 2022-09-12T19:45:17.335 DEBUG:boto:Path: /jdpiak-72/c 2022-09-12T19:45:17.335 DEBUG:boto:Data: 2022-09-12T19:45:17.336 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.337 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:17.337 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.338 DEBUG:boto:Params: {} 2022-09-12T19:45:17.338 DEBUG:boto:Token: None 2022-09-12T19:45:17.339 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/c 2022-09-12T19:45:17.340 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N5i6gU2xQd0I7Sdpw/I2fbyJM1M= 2022-09-12T19:45:17.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N5i6gU2xQd0I7Sdpw/I2fbyJM1M=', 'Content-Length': '0'} 2022-09-12T19:45:17.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000352ba6da3500c47f-00631f8c4d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:45:17.347 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:17.348 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:17.349 DEBUG:boto:Method: GET 2022-09-12T19:45:17.350 DEBUG:boto:Path: /jdpiak-72/c 2022-09-12T19:45:17.350 DEBUG:boto:Data: 2022-09-12T19:45:17.351 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.352 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:17.352 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.353 DEBUG:boto:Params: {} 2022-09-12T19:45:17.354 DEBUG:boto:Token: None 2022-09-12T19:45:17.354 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/c 2022-09-12T19:45:17.355 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l+zymUu8dcOKUsS8F5pJuCIwHeQ= 2022-09-12T19:45:17.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l+zymUu8dcOKUsS8F5pJuCIwHeQ=', 'Content-Length': '0'} 2022-09-12T19:45:17.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9e44df6efe6a9da-00631f8c4d-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.361 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:17.362 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:17.363 DEBUG:boto:Method: GET 2022-09-12T19:45:17.364 DEBUG:boto:Path: /jdpiak-72/c 2022-09-12T19:45:17.365 DEBUG:boto:Data: 2022-09-12T19:45:17.366 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.367 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:17.367 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.368 DEBUG:boto:Params: {} 2022-09-12T19:45:17.369 DEBUG:boto:Token: None 2022-09-12T19:45:17.370 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/c 2022-09-12T19:45:17.371 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l+zymUu8dcOKUsS8F5pJuCIwHeQ= 2022-09-12T19:45:17.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l+zymUu8dcOKUsS8F5pJuCIwHeQ=', 'Content-Length': '0'} 2022-09-12T19:45:17.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5d9526ea60316bf-00631f8c4d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.379 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:17.381 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:17.381 DEBUG:boto:path=/jdpiak-72/?prefix=c 2022-09-12T19:45:17.382 DEBUG:boto:auth_path=/jdpiak-72/?prefix=c 2022-09-12T19:45:17.383 DEBUG:boto:Method: GET 2022-09-12T19:45:17.384 DEBUG:boto:Path: /jdpiak-72/?prefix=c 2022-09-12T19:45:17.384 DEBUG:boto:Data: 2022-09-12T19:45:17.385 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.385 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:17.386 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.388 DEBUG:boto:Params: {} 2022-09-12T19:45:17.389 DEBUG:boto:Token: None 2022-09-12T19:45:17.390 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/ 2022-09-12T19:45:17.391 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WMpeNN3/3wPBDFQvZYQCZpQUUB4= 2022-09-12T19:45:17.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WMpeNN3/3wPBDFQvZYQCZpQUUB4=', 'Content-Length': '0'} 2022-09-12T19:45:17.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061ecacb96ace4529-00631f8c4d-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.407 DEBUG:boto:b'jdpiak-72c1000falsec2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:17.409 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:17.411 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:17.412 DEBUG:boto:path=/jdpiak-72/?prefix=c 2022-09-12T19:45:17.413 DEBUG:boto:auth_path=/jdpiak-72/?prefix=c 2022-09-12T19:45:17.414 DEBUG:boto:Method: GET 2022-09-12T19:45:17.415 DEBUG:boto:Path: /jdpiak-72/?prefix=c 2022-09-12T19:45:17.416 DEBUG:boto:Data: 2022-09-12T19:45:17.417 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.418 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:17.420 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.421 DEBUG:boto:Params: {} 2022-09-12T19:45:17.423 DEBUG:boto:Token: None 2022-09-12T19:45:17.424 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/ 2022-09-12T19:45:17.426 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WMpeNN3/3wPBDFQvZYQCZpQUUB4= 2022-09-12T19:45:17.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WMpeNN3/3wPBDFQvZYQCZpQUUB4=', 'Content-Length': '0'} 2022-09-12T19:45:17.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3d0e9e7a2187d2c-00631f8c4d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.435 DEBUG:boto:b'jdpiak-72c1000falsec2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:17.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:45:17.439 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:17.440 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:17.440 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:17.441 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:17.442 DEBUG:boto:Method: GET 2022-09-12T19:45:17.443 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:45:17.444 DEBUG:boto:Data: 2022-09-12T19:45:17.445 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.445 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:17.446 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.447 DEBUG:boto:Params: {} 2022-09-12T19:45:17.448 DEBUG:boto:Token: None 2022-09-12T19:45:17.449 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:45:17.450 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fzOzXASnZmN9hMlEb3cQYMUYlyM= 2022-09-12T19:45:17.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fzOzXASnZmN9hMlEb3cQYMUYlyM=', 'Content-Length': '0'} 2022-09-12T19:45:17.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2a8fafd7d4af9d0-00631f8c4d-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.457 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:17.458 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:17.459 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:17.460 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:17.461 DEBUG:boto:Method: GET 2022-09-12T19:45:17.461 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:45:17.462 DEBUG:boto:Data: 2022-09-12T19:45:17.463 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.463 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:17.464 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.465 DEBUG:boto:Params: {} 2022-09-12T19:45:17.465 DEBUG:boto:Token: None 2022-09-12T19:45:17.466 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:45:17.467 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fzOzXASnZmN9hMlEb3cQYMUYlyM= 2022-09-12T19:45:17.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fzOzXASnZmN9hMlEb3cQYMUYlyM=', 'Content-Length': '0'} 2022-09-12T19:45:17.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000873631a5d1146092-00631f8c4d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.472 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:17.472 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:17.473 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:17.473 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:17.474 DEBUG:boto:Method: HEAD 2022-09-12T19:45:17.474 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:45:17.475 DEBUG:boto:Data: 2022-09-12T19:45:17.475 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.476 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:17.477 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.477 DEBUG:boto:Params: {} 2022-09-12T19:45:17.479 DEBUG:boto:Token: None 2022-09-12T19:45:17.480 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:45:17.481 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4pynHb9KMlxMcEOn1tLI2n0Vrg8= 2022-09-12T19:45:17.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4pynHb9KMlxMcEOn1tLI2n0Vrg8=', 'Content-Length': '0'} 2022-09-12T19:45:17.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9c888cc31faaa4f-00631f8c4d-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.489 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:17.490 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:17.491 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:17.492 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:17.493 DEBUG:boto:Method: HEAD 2022-09-12T19:45:17.494 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:45:17.495 DEBUG:boto:Data: 2022-09-12T19:45:17.497 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.498 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:17.499 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.500 DEBUG:boto:Params: {} 2022-09-12T19:45:17.501 DEBUG:boto:Token: None 2022-09-12T19:45:17.503 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:45:17.504 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4pynHb9KMlxMcEOn1tLI2n0Vrg8= 2022-09-12T19:45:17.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4pynHb9KMlxMcEOn1tLI2n0Vrg8=', 'Content-Length': '0'} 2022-09-12T19:45:17.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082a4f0b4aadc4339-00631f8c4d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:45:17.513 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:17.515 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:17.516 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:17.517 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:17.518 DEBUG:boto:Method: GET 2022-09-12T19:45:17.519 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:45:17.520 DEBUG:boto:Data: 2022-09-12T19:45:17.521 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.522 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:17.523 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.525 DEBUG:boto:Params: {} 2022-09-12T19:45:17.527 DEBUG:boto:Token: None 2022-09-12T19:45:17.528 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:45:17.530 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fzOzXASnZmN9hMlEb3cQYMUYlyM= 2022-09-12T19:45:17.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fzOzXASnZmN9hMlEb3cQYMUYlyM=', 'Content-Length': '0'} 2022-09-12T19:45:17.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000771f1d875b1ca219-00631f8c4d-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.542 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:17.544 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:17.544 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:17.546 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:17.547 DEBUG:boto:Method: GET 2022-09-12T19:45:17.548 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:45:17.549 DEBUG:boto:Data: 2022-09-12T19:45:17.551 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.552 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:17.553 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.554 DEBUG:boto:Params: {} 2022-09-12T19:45:17.555 DEBUG:boto:Token: None 2022-09-12T19:45:17.556 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:45:17.558 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fzOzXASnZmN9hMlEb3cQYMUYlyM= 2022-09-12T19:45:17.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fzOzXASnZmN9hMlEb3cQYMUYlyM=', 'Content-Length': '0'} 2022-09-12T19:45:17.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009075e41015ca7c58-00631f8c4d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.565 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:17.566 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:17.567 DEBUG:boto:path=/jdpiak-72/?prefix=d 2022-09-12T19:45:17.568 DEBUG:boto:auth_path=/jdpiak-72/?prefix=d 2022-09-12T19:45:17.569 DEBUG:boto:Method: GET 2022-09-12T19:45:17.570 DEBUG:boto:Path: /jdpiak-72/?prefix=d 2022-09-12T19:45:17.572 DEBUG:boto:Data: 2022-09-12T19:45:17.573 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.574 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:17.575 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.576 DEBUG:boto:Params: {} 2022-09-12T19:45:17.577 DEBUG:boto:Token: None 2022-09-12T19:45:17.578 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/ 2022-09-12T19:45:17.579 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WMpeNN3/3wPBDFQvZYQCZpQUUB4= 2022-09-12T19:45:17.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WMpeNN3/3wPBDFQvZYQCZpQUUB4=', 'Content-Length': '0'} 2022-09-12T19:45:17.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abbba641aa5450c6-00631f8c4d-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.588 DEBUG:boto:b'jdpiak-72d1000falsed2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:17.589 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:17.591 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:17.591 DEBUG:boto:path=/jdpiak-72/?prefix=d 2022-09-12T19:45:17.593 DEBUG:boto:auth_path=/jdpiak-72/?prefix=d 2022-09-12T19:45:17.594 DEBUG:boto:Method: GET 2022-09-12T19:45:17.595 DEBUG:boto:Path: /jdpiak-72/?prefix=d 2022-09-12T19:45:17.597 DEBUG:boto:Data: 2022-09-12T19:45:17.598 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.599 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:17.600 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.601 DEBUG:boto:Params: {} 2022-09-12T19:45:17.602 DEBUG:boto:Token: None 2022-09-12T19:45:17.604 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/ 2022-09-12T19:45:17.605 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WMpeNN3/3wPBDFQvZYQCZpQUUB4= 2022-09-12T19:45:17.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WMpeNN3/3wPBDFQvZYQCZpQUUB4=', 'Content-Length': '0'} 2022-09-12T19:45:17.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031856c4596f992ae-00631f8c4d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.613 DEBUG:boto:b'jdpiak-72d1000falsed2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:17.615 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:17.616 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:17.617 DEBUG:boto:Method: HEAD 2022-09-12T19:45:17.618 DEBUG:boto:Path: /jdpiak-72/d 2022-09-12T19:45:17.619 DEBUG:boto:Data: 2022-09-12T19:45:17.620 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.621 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:17.623 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.624 DEBUG:boto:Params: {} 2022-09-12T19:45:17.625 DEBUG:boto:Token: None 2022-09-12T19:45:17.626 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/d 2022-09-12T19:45:17.627 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Vs25yOk+MaRBoVwmhza1wD5Ba8= 2022-09-12T19:45:17.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Vs25yOk+MaRBoVwmhza1wD5Ba8=', 'Content-Length': '0'} 2022-09-12T19:45:17.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a022dc1020d55e4-00631f8c4d-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.635 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:17.636 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:17.637 DEBUG:boto:Method: HEAD 2022-09-12T19:45:17.638 DEBUG:boto:Path: /jdpiak-72/d 2022-09-12T19:45:17.639 DEBUG:boto:Data: 2022-09-12T19:45:17.640 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.642 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:17.643 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.644 DEBUG:boto:Params: {} 2022-09-12T19:45:17.645 DEBUG:boto:Token: None 2022-09-12T19:45:17.646 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/d 2022-09-12T19:45:17.648 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Vs25yOk+MaRBoVwmhza1wD5Ba8= 2022-09-12T19:45:17.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Vs25yOk+MaRBoVwmhza1wD5Ba8=', 'Content-Length': '0'} 2022-09-12T19:45:17.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031c6c3939989b97d-00631f8c4d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:45:17.655 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:17.656 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:17.657 DEBUG:boto:Method: GET 2022-09-12T19:45:17.658 DEBUG:boto:Path: /jdpiak-72/d 2022-09-12T19:45:17.659 DEBUG:boto:Data: 2022-09-12T19:45:17.660 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.661 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:17.663 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.664 DEBUG:boto:Params: {} 2022-09-12T19:45:17.665 DEBUG:boto:Token: None 2022-09-12T19:45:17.666 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/d 2022-09-12T19:45:17.667 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Eh21ZBvzCaxJ2fFgfJRxS/Qmtg0= 2022-09-12T19:45:17.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Eh21ZBvzCaxJ2fFgfJRxS/Qmtg0=', 'Content-Length': '0'} 2022-09-12T19:45:17.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e0711e02c311ac9-00631f8c4d-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.677 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:17.678 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:17.679 DEBUG:boto:Method: GET 2022-09-12T19:45:17.680 DEBUG:boto:Path: /jdpiak-72/d 2022-09-12T19:45:17.680 DEBUG:boto:Data: 2022-09-12T19:45:17.681 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.681 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:17.682 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.683 DEBUG:boto:Params: {} 2022-09-12T19:45:17.684 DEBUG:boto:Token: None 2022-09-12T19:45:17.685 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/d 2022-09-12T19:45:17.685 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Eh21ZBvzCaxJ2fFgfJRxS/Qmtg0= 2022-09-12T19:45:17.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Eh21ZBvzCaxJ2fFgfJRxS/Qmtg0=', 'Content-Length': '0'} 2022-09-12T19:45:17.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ed787ce14cd0303-00631f8c4d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.693 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:17.694 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:17.694 DEBUG:boto:path=/jdpiak-72/?prefix=d 2022-09-12T19:45:17.695 DEBUG:boto:auth_path=/jdpiak-72/?prefix=d 2022-09-12T19:45:17.696 DEBUG:boto:Method: GET 2022-09-12T19:45:17.696 DEBUG:boto:Path: /jdpiak-72/?prefix=d 2022-09-12T19:45:17.697 DEBUG:boto:Data: 2022-09-12T19:45:17.697 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.698 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:17.699 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.700 DEBUG:boto:Params: {} 2022-09-12T19:45:17.700 DEBUG:boto:Token: None 2022-09-12T19:45:17.701 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/ 2022-09-12T19:45:17.702 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WMpeNN3/3wPBDFQvZYQCZpQUUB4= 2022-09-12T19:45:17.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WMpeNN3/3wPBDFQvZYQCZpQUUB4=', 'Content-Length': '0'} 2022-09-12T19:45:17.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5d955bb27b4305c-00631f8c4d-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.708 DEBUG:boto:b'jdpiak-72d1000falsed2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:17.710 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:17.711 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:17.712 DEBUG:boto:path=/jdpiak-72/?prefix=d 2022-09-12T19:45:17.713 DEBUG:boto:auth_path=/jdpiak-72/?prefix=d 2022-09-12T19:45:17.714 DEBUG:boto:Method: GET 2022-09-12T19:45:17.715 DEBUG:boto:Path: /jdpiak-72/?prefix=d 2022-09-12T19:45:17.716 DEBUG:boto:Data: 2022-09-12T19:45:17.717 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.718 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:17.719 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.720 DEBUG:boto:Params: {} 2022-09-12T19:45:17.721 DEBUG:boto:Token: None 2022-09-12T19:45:17.722 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/ 2022-09-12T19:45:17.723 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WMpeNN3/3wPBDFQvZYQCZpQUUB4= 2022-09-12T19:45:17.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WMpeNN3/3wPBDFQvZYQCZpQUUB4=', 'Content-Length': '0'} 2022-09-12T19:45:17.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044bf14002679151c-00631f8c4d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.730 DEBUG:boto:b'jdpiak-72d1000falsed2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:17.731 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-72 zones={test-zone2, test-zone1} 2022-09-12T19:45:17.731 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-72 zones={test-zone3, test-zone1} 2022-09-12T19:45:17.732 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:17.733 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:17.733 DEBUG:boto:Method: HEAD 2022-09-12T19:45:17.734 DEBUG:boto:Path: /jdpiak-72/ 2022-09-12T19:45:17.735 DEBUG:boto:Data: 2022-09-12T19:45:17.735 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.736 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:17.737 DEBUG:boto:Port: 8001 2022-09-12T19:45:17.737 DEBUG:boto:Params: {} 2022-09-12T19:45:17.738 DEBUG:boto:Token: None 2022-09-12T19:45:17.739 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/ 2022-09-12T19:45:17.740 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SR+ZQiqluVxeZRvUf+CpTzlXgdA= 2022-09-12T19:45:17.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SR+ZQiqluVxeZRvUf+CpTzlXgdA=', 'Content-Length': '0'} 2022-09-12T19:45:17.745 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e10236272c5d5535-00631f8c4d-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.746 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:17.747 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:17.747 DEBUG:boto:Method: HEAD 2022-09-12T19:45:17.748 DEBUG:boto:Path: /jdpiak-72/ 2022-09-12T19:45:17.748 DEBUG:boto:Data: 2022-09-12T19:45:17.749 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.750 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:17.750 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.751 DEBUG:boto:Params: {} 2022-09-12T19:45:17.751 DEBUG:boto:Token: None 2022-09-12T19:45:17.752 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/ 2022-09-12T19:45:17.753 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SR+ZQiqluVxeZRvUf+CpTzlXgdA= 2022-09-12T19:45:17.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SR+ZQiqluVxeZRvUf+CpTzlXgdA=', 'Content-Length': '0'} 2022-09-12T19:45:17.757 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092880d0dfa0451e4-00631f8c4d-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.758 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:45:17.759 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:17.759 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:17.760 DEBUG:boto:path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:17.761 DEBUG:boto:auth_path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:17.761 DEBUG:boto:Method: GET 2022-09-12T19:45:17.762 DEBUG:boto:Path: /jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:17.762 DEBUG:boto:Data: 2022-09-12T19:45:17.763 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.764 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:17.765 DEBUG:boto:Port: 8001 2022-09-12T19:45:17.766 DEBUG:boto:Params: {} 2022-09-12T19:45:17.766 DEBUG:boto:Token: None 2022-09-12T19:45:17.767 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/?versions 2022-09-12T19:45:17.767 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mVFE4hLo4J3FMSbilaZhdSl3Zoc= 2022-09-12T19:45:17.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mVFE4hLo4J3FMSbilaZhdSl3Zoc=', 'Content-Length': '0'} 2022-09-12T19:45:17.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ccfe5acf6a0acae-00631f8c4d-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.777 DEBUG:boto:b'jdpiak-72999falseanulltrue2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:17.780 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:45:17.781 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:45:17.784 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:45:17.790 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:45:17.791 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:45:17.792 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:17.793 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:17.795 DEBUG:boto:path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:17.795 DEBUG:boto:auth_path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:17.796 DEBUG:boto:Method: GET 2022-09-12T19:45:17.797 DEBUG:boto:Path: /jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:17.798 DEBUG:boto:Data: 2022-09-12T19:45:17.800 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.801 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:17.802 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.803 DEBUG:boto:Params: {} 2022-09-12T19:45:17.805 DEBUG:boto:Token: None 2022-09-12T19:45:17.806 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/?versions 2022-09-12T19:45:17.807 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mVFE4hLo4J3FMSbilaZhdSl3Zoc= 2022-09-12T19:45:17.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mVFE4hLo4J3FMSbilaZhdSl3Zoc=', 'Content-Length': '0'} 2022-09-12T19:45:17.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000814b546eec00576-00631f8c4d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.814 DEBUG:boto:b'jdpiak-72999falseanulltrue2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:17.816 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:45:17.817 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:45:17.817 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:45:17.818 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:45:17.819 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:17.820 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:17.821 DEBUG:boto:path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:17.822 DEBUG:boto:auth_path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:17.823 DEBUG:boto:Method: GET 2022-09-12T19:45:17.824 DEBUG:boto:Path: /jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:17.825 DEBUG:boto:Data: 2022-09-12T19:45:17.826 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.827 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:17.828 DEBUG:boto:Port: 8001 2022-09-12T19:45:17.829 DEBUG:boto:Params: {} 2022-09-12T19:45:17.830 DEBUG:boto:Token: None 2022-09-12T19:45:17.830 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/?versions 2022-09-12T19:45:17.832 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mVFE4hLo4J3FMSbilaZhdSl3Zoc= 2022-09-12T19:45:17.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mVFE4hLo4J3FMSbilaZhdSl3Zoc=', 'Content-Length': '0'} 2022-09-12T19:45:17.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000639adfc80b49b639-00631f8c4d-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.839 DEBUG:boto:b'jdpiak-72999falseanulltrue2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:17.841 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:17.841 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:17.842 DEBUG:boto:path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:17.843 DEBUG:boto:auth_path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:17.843 DEBUG:boto:Method: GET 2022-09-12T19:45:17.844 DEBUG:boto:Path: /jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:17.845 DEBUG:boto:Data: 2022-09-12T19:45:17.846 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.847 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:17.849 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.851 DEBUG:boto:Params: {} 2022-09-12T19:45:17.852 DEBUG:boto:Token: None 2022-09-12T19:45:17.853 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/?versions 2022-09-12T19:45:17.854 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mVFE4hLo4J3FMSbilaZhdSl3Zoc= 2022-09-12T19:45:17.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mVFE4hLo4J3FMSbilaZhdSl3Zoc=', 'Content-Length': '0'} 2022-09-12T19:45:17.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfcc33f60fa14cbe-00631f8c4d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.859 DEBUG:boto:b'jdpiak-72999falseanulltrue2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:17.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:45:17.861 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:17.862 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:17.863 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:17.863 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:17.864 DEBUG:boto:Method: GET 2022-09-12T19:45:17.864 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:45:17.865 DEBUG:boto:Data: 2022-09-12T19:45:17.866 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.867 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:17.867 DEBUG:boto:Port: 8001 2022-09-12T19:45:17.868 DEBUG:boto:Params: {} 2022-09-12T19:45:17.868 DEBUG:boto:Token: None 2022-09-12T19:45:17.869 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:45:17.870 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:anfwMpyhPCXFde1+5VPIzR7cmcw= 2022-09-12T19:45:17.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:anfwMpyhPCXFde1+5VPIzR7cmcw=', 'Content-Length': '0'} 2022-09-12T19:45:17.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1662efbd197f01f-00631f8c4d-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.875 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:17.876 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:17.877 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:17.877 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:17.878 DEBUG:boto:Method: GET 2022-09-12T19:45:17.879 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:45:17.879 DEBUG:boto:Data: 2022-09-12T19:45:17.880 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.881 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:17.883 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.884 DEBUG:boto:Params: {} 2022-09-12T19:45:17.885 DEBUG:boto:Token: None 2022-09-12T19:45:17.886 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:45:17.888 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:anfwMpyhPCXFde1+5VPIzR7cmcw= 2022-09-12T19:45:17.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:anfwMpyhPCXFde1+5VPIzR7cmcw=', 'Content-Length': '0'} 2022-09-12T19:45:17.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c68f8bedda1c574f-00631f8c4d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.935 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:17.937 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:17.938 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:17.939 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:17.940 DEBUG:boto:Method: HEAD 2022-09-12T19:45:17.941 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:45:17.941 DEBUG:boto:Data: 2022-09-12T19:45:17.943 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.944 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:17.946 DEBUG:boto:Port: 8001 2022-09-12T19:45:17.947 DEBUG:boto:Params: {} 2022-09-12T19:45:17.948 DEBUG:boto:Token: None 2022-09-12T19:45:17.949 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:45:17.950 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:860I1H0dGVgpsJGG/ShuO++AfOc= 2022-09-12T19:45:17.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:860I1H0dGVgpsJGG/ShuO++AfOc=', 'Content-Length': '0'} 2022-09-12T19:45:17.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000095a3e79620e86dd-00631f8c4d-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.954 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:17.955 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:17.956 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:17.957 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:17.958 DEBUG:boto:Method: HEAD 2022-09-12T19:45:17.959 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:45:17.960 DEBUG:boto:Data: 2022-09-12T19:45:17.961 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.962 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:17.963 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.964 DEBUG:boto:Params: {} 2022-09-12T19:45:17.965 DEBUG:boto:Token: None 2022-09-12T19:45:17.966 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:45:17.967 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:860I1H0dGVgpsJGG/ShuO++AfOc= 2022-09-12T19:45:17.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:860I1H0dGVgpsJGG/ShuO++AfOc=', 'Content-Length': '0'} 2022-09-12T19:45:17.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8ff2ad30cb02b88-00631f8c4d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:45:17.974 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:17.975 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:17.976 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:17.976 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:17.977 DEBUG:boto:Method: GET 2022-09-12T19:45:17.978 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:45:17.979 DEBUG:boto:Data: 2022-09-12T19:45:17.980 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.981 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:17.982 DEBUG:boto:Port: 8001 2022-09-12T19:45:17.982 DEBUG:boto:Params: {} 2022-09-12T19:45:17.983 DEBUG:boto:Token: None 2022-09-12T19:45:17.984 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:45:17.984 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:anfwMpyhPCXFde1+5VPIzR7cmcw= 2022-09-12T19:45:17.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:anfwMpyhPCXFde1+5VPIzR7cmcw=', 'Content-Length': '0'} 2022-09-12T19:45:17.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035ed9e09ae8f8781-00631f8c4d-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:17.988 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:17.989 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:17.989 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:17.990 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:17.990 DEBUG:boto:Method: GET 2022-09-12T19:45:17.991 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:45:17.991 DEBUG:boto:Data: 2022-09-12T19:45:17.992 DEBUG:boto:Headers: {} 2022-09-12T19:45:17.993 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:17.993 DEBUG:boto:Port: 8000 2022-09-12T19:45:17.994 DEBUG:boto:Params: {} 2022-09-12T19:45:17.994 DEBUG:boto:Token: None 2022-09-12T19:45:17.995 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:17 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:45:17.996 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:anfwMpyhPCXFde1+5VPIzR7cmcw= 2022-09-12T19:45:17.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:anfwMpyhPCXFde1+5VPIzR7cmcw=', 'Content-Length': '0'} 2022-09-12T19:45:17.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6790a068f20f153-00631f8c4d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.000 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:18.000 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:18.001 DEBUG:boto:path=/jdpiak-72/?prefix=a 2022-09-12T19:45:18.001 DEBUG:boto:auth_path=/jdpiak-72/?prefix=a 2022-09-12T19:45:18.002 DEBUG:boto:Method: GET 2022-09-12T19:45:18.003 DEBUG:boto:Path: /jdpiak-72/?prefix=a 2022-09-12T19:45:18.004 DEBUG:boto:Data: 2022-09-12T19:45:18.004 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.005 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.005 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.006 DEBUG:boto:Params: {} 2022-09-12T19:45:18.006 DEBUG:boto:Token: None 2022-09-12T19:45:18.007 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/ 2022-09-12T19:45:18.008 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk= 2022-09-12T19:45:18.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk=', 'Content-Length': '0'} 2022-09-12T19:45:18.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008049b770ce2fe726-00631f8c4e-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.012 DEBUG:boto:b'jdpiak-72a1000falsea2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:18.013 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:18.013 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:18.014 DEBUG:boto:path=/jdpiak-72/?prefix=a 2022-09-12T19:45:18.015 DEBUG:boto:auth_path=/jdpiak-72/?prefix=a 2022-09-12T19:45:18.015 DEBUG:boto:Method: GET 2022-09-12T19:45:18.016 DEBUG:boto:Path: /jdpiak-72/?prefix=a 2022-09-12T19:45:18.016 DEBUG:boto:Data: 2022-09-12T19:45:18.018 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.018 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:18.019 DEBUG:boto:Port: 8000 2022-09-12T19:45:18.019 DEBUG:boto:Params: {} 2022-09-12T19:45:18.020 DEBUG:boto:Token: None 2022-09-12T19:45:18.021 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/ 2022-09-12T19:45:18.021 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk= 2022-09-12T19:45:18.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk=', 'Content-Length': '0'} 2022-09-12T19:45:18.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6c0fca835165ceb-00631f8c4e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.066 DEBUG:boto:b'jdpiak-72a1000falsea2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:18.067 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:18.068 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:18.068 DEBUG:boto:Method: HEAD 2022-09-12T19:45:18.069 DEBUG:boto:Path: /jdpiak-72/a 2022-09-12T19:45:18.069 DEBUG:boto:Data: 2022-09-12T19:45:18.070 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.070 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.071 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.071 DEBUG:boto:Params: {} 2022-09-12T19:45:18.072 DEBUG:boto:Token: None 2022-09-12T19:45:18.072 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/a 2022-09-12T19:45:18.073 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MEPIGmMIO3fA4ABOkpoBjLYIumY= 2022-09-12T19:45:18.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MEPIGmMIO3fA4ABOkpoBjLYIumY=', 'Content-Length': '0'} 2022-09-12T19:45:18.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bc1c847cf8231f0-00631f8c4e-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.077 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:18.077 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:18.078 DEBUG:boto:Method: HEAD 2022-09-12T19:45:18.078 DEBUG:boto:Path: /jdpiak-72/a 2022-09-12T19:45:18.079 DEBUG:boto:Data: 2022-09-12T19:45:18.079 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.080 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:18.080 DEBUG:boto:Port: 8000 2022-09-12T19:45:18.081 DEBUG:boto:Params: {} 2022-09-12T19:45:18.081 DEBUG:boto:Token: None 2022-09-12T19:45:18.082 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/a 2022-09-12T19:45:18.083 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MEPIGmMIO3fA4ABOkpoBjLYIumY= 2022-09-12T19:45:18.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MEPIGmMIO3fA4ABOkpoBjLYIumY=', 'Content-Length': '0'} 2022-09-12T19:45:18.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb1ff593d1eb3a72-00631f8c4e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:45:18.087 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:18.087 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:18.088 DEBUG:boto:Method: GET 2022-09-12T19:45:18.088 DEBUG:boto:Path: /jdpiak-72/a 2022-09-12T19:45:18.089 DEBUG:boto:Data: 2022-09-12T19:45:18.090 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.090 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.091 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.091 DEBUG:boto:Params: {} 2022-09-12T19:45:18.092 DEBUG:boto:Token: None 2022-09-12T19:45:18.093 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/a 2022-09-12T19:45:18.093 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XXBRFyUbiCfoiJF+rWys4ScF/Ww= 2022-09-12T19:45:18.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XXBRFyUbiCfoiJF+rWys4ScF/Ww=', 'Content-Length': '0'} 2022-09-12T19:45:18.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad7e302b3922ac93-00631f8c4e-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.097 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:18.097 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:18.098 DEBUG:boto:Method: GET 2022-09-12T19:45:18.099 DEBUG:boto:Path: /jdpiak-72/a 2022-09-12T19:45:18.099 DEBUG:boto:Data: 2022-09-12T19:45:18.100 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.100 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:18.101 DEBUG:boto:Port: 8000 2022-09-12T19:45:18.101 DEBUG:boto:Params: {} 2022-09-12T19:45:18.102 DEBUG:boto:Token: None 2022-09-12T19:45:18.102 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/a 2022-09-12T19:45:18.103 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XXBRFyUbiCfoiJF+rWys4ScF/Ww= 2022-09-12T19:45:18.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XXBRFyUbiCfoiJF+rWys4ScF/Ww=', 'Content-Length': '0'} 2022-09-12T19:45:18.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052aec2b05425f3ed-00631f8c4e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.107 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:18.107 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:18.108 DEBUG:boto:path=/jdpiak-72/?prefix=a 2022-09-12T19:45:18.108 DEBUG:boto:auth_path=/jdpiak-72/?prefix=a 2022-09-12T19:45:18.109 DEBUG:boto:Method: GET 2022-09-12T19:45:18.110 DEBUG:boto:Path: /jdpiak-72/?prefix=a 2022-09-12T19:45:18.110 DEBUG:boto:Data: 2022-09-12T19:45:18.111 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.111 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.112 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.112 DEBUG:boto:Params: {} 2022-09-12T19:45:18.113 DEBUG:boto:Token: None 2022-09-12T19:45:18.113 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/ 2022-09-12T19:45:18.114 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk= 2022-09-12T19:45:18.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk=', 'Content-Length': '0'} 2022-09-12T19:45:18.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050148f85ab6425d3-00631f8c4e-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.118 DEBUG:boto:b'jdpiak-72a1000falsea2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:18.119 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:18.120 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:18.120 DEBUG:boto:path=/jdpiak-72/?prefix=a 2022-09-12T19:45:18.121 DEBUG:boto:auth_path=/jdpiak-72/?prefix=a 2022-09-12T19:45:18.121 DEBUG:boto:Method: GET 2022-09-12T19:45:18.122 DEBUG:boto:Path: /jdpiak-72/?prefix=a 2022-09-12T19:45:18.122 DEBUG:boto:Data: 2022-09-12T19:45:18.123 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.123 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:18.124 DEBUG:boto:Port: 8000 2022-09-12T19:45:18.125 DEBUG:boto:Params: {} 2022-09-12T19:45:18.125 DEBUG:boto:Token: None 2022-09-12T19:45:18.126 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/ 2022-09-12T19:45:18.127 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk= 2022-09-12T19:45:18.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk=', 'Content-Length': '0'} 2022-09-12T19:45:18.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004aa1f2f972273bc5-00631f8c4e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.131 DEBUG:boto:b'jdpiak-72a1000falsea2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:18.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:45:18.133 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:18.133 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:18.134 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:18.134 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:18.135 DEBUG:boto:Method: GET 2022-09-12T19:45:18.136 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:45:18.136 DEBUG:boto:Data: 2022-09-12T19:45:18.137 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.137 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.138 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.138 DEBUG:boto:Params: {} 2022-09-12T19:45:18.140 DEBUG:boto:Token: None 2022-09-12T19:45:18.140 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:45:18.141 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nJn/GdsqFI/yRwEUfpTOiMxeCYM= 2022-09-12T19:45:18.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nJn/GdsqFI/yRwEUfpTOiMxeCYM=', 'Content-Length': '0'} 2022-09-12T19:45:18.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083410331356f5156-00631f8c4e-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.145 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:18.145 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:18.146 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:18.146 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:18.147 DEBUG:boto:Method: GET 2022-09-12T19:45:18.147 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:45:18.148 DEBUG:boto:Data: 2022-09-12T19:45:18.148 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.149 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:18.149 DEBUG:boto:Port: 8000 2022-09-12T19:45:18.150 DEBUG:boto:Params: {} 2022-09-12T19:45:18.150 DEBUG:boto:Token: None 2022-09-12T19:45:18.151 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:45:18.152 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nJn/GdsqFI/yRwEUfpTOiMxeCYM= 2022-09-12T19:45:18.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nJn/GdsqFI/yRwEUfpTOiMxeCYM=', 'Content-Length': '0'} 2022-09-12T19:45:18.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041926ea3e2f2ef9d-00631f8c4e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.198 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:18.199 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:18.200 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:18.201 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:18.201 DEBUG:boto:Method: HEAD 2022-09-12T19:45:18.202 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:45:18.202 DEBUG:boto:Data: 2022-09-12T19:45:18.203 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.203 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.204 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.205 DEBUG:boto:Params: {} 2022-09-12T19:45:18.206 DEBUG:boto:Token: None 2022-09-12T19:45:18.207 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:45:18.207 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R/TBV8ulsAx9atrOa9MtZON2gmA= 2022-09-12T19:45:18.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R/TBV8ulsAx9atrOa9MtZON2gmA=', 'Content-Length': '0'} 2022-09-12T19:45:18.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008209ee082baf373d-00631f8c4e-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.211 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:18.212 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:18.212 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:18.213 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:18.213 DEBUG:boto:Method: HEAD 2022-09-12T19:45:18.214 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:45:18.214 DEBUG:boto:Data: 2022-09-12T19:45:18.215 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.215 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:18.216 DEBUG:boto:Port: 8000 2022-09-12T19:45:18.217 DEBUG:boto:Params: {} 2022-09-12T19:45:18.217 DEBUG:boto:Token: None 2022-09-12T19:45:18.218 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:45:18.219 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R/TBV8ulsAx9atrOa9MtZON2gmA= 2022-09-12T19:45:18.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R/TBV8ulsAx9atrOa9MtZON2gmA=', 'Content-Length': '0'} 2022-09-12T19:45:18.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf8665c142eed3ca-00631f8c4e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:45:18.226 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:18.227 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:18.228 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:18.229 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:18.230 DEBUG:boto:Method: GET 2022-09-12T19:45:18.230 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:45:18.231 DEBUG:boto:Data: 2022-09-12T19:45:18.232 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.232 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.233 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.233 DEBUG:boto:Params: {} 2022-09-12T19:45:18.234 DEBUG:boto:Token: None 2022-09-12T19:45:18.234 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:45:18.235 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nJn/GdsqFI/yRwEUfpTOiMxeCYM= 2022-09-12T19:45:18.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nJn/GdsqFI/yRwEUfpTOiMxeCYM=', 'Content-Length': '0'} 2022-09-12T19:45:18.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b1c14c734c075a0-00631f8c4e-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.239 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:18.240 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:18.240 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:18.241 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:18.241 DEBUG:boto:Method: GET 2022-09-12T19:45:18.242 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:45:18.242 DEBUG:boto:Data: 2022-09-12T19:45:18.243 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.244 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:18.244 DEBUG:boto:Port: 8000 2022-09-12T19:45:18.245 DEBUG:boto:Params: {} 2022-09-12T19:45:18.245 DEBUG:boto:Token: None 2022-09-12T19:45:18.246 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:45:18.247 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nJn/GdsqFI/yRwEUfpTOiMxeCYM= 2022-09-12T19:45:18.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nJn/GdsqFI/yRwEUfpTOiMxeCYM=', 'Content-Length': '0'} 2022-09-12T19:45:18.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078e6e13697fdc182-00631f8c4e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.251 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:18.252 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:18.253 DEBUG:boto:path=/jdpiak-72/?prefix=b 2022-09-12T19:45:18.254 DEBUG:boto:auth_path=/jdpiak-72/?prefix=b 2022-09-12T19:45:18.255 DEBUG:boto:Method: GET 2022-09-12T19:45:18.256 DEBUG:boto:Path: /jdpiak-72/?prefix=b 2022-09-12T19:45:18.257 DEBUG:boto:Data: 2022-09-12T19:45:18.258 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.260 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.260 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.261 DEBUG:boto:Params: {} 2022-09-12T19:45:18.262 DEBUG:boto:Token: None 2022-09-12T19:45:18.263 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/ 2022-09-12T19:45:18.264 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk= 2022-09-12T19:45:18.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk=', 'Content-Length': '0'} 2022-09-12T19:45:18.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000872098e4bdab75b-00631f8c4e-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.310 DEBUG:boto:b'jdpiak-72b1000falseb2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:18.311 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:18.312 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:18.313 DEBUG:boto:path=/jdpiak-72/?prefix=b 2022-09-12T19:45:18.313 DEBUG:boto:auth_path=/jdpiak-72/?prefix=b 2022-09-12T19:45:18.314 DEBUG:boto:Method: GET 2022-09-12T19:45:18.314 DEBUG:boto:Path: /jdpiak-72/?prefix=b 2022-09-12T19:45:18.315 DEBUG:boto:Data: 2022-09-12T19:45:18.315 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.316 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:18.317 DEBUG:boto:Port: 8000 2022-09-12T19:45:18.317 DEBUG:boto:Params: {} 2022-09-12T19:45:18.318 DEBUG:boto:Token: None 2022-09-12T19:45:18.319 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/ 2022-09-12T19:45:18.320 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk= 2022-09-12T19:45:18.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk=', 'Content-Length': '0'} 2022-09-12T19:45:18.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000271d1e88350b16ba-00631f8c4e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.324 DEBUG:boto:b'jdpiak-72b1000falseb2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:18.325 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:18.325 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:18.326 DEBUG:boto:Method: HEAD 2022-09-12T19:45:18.327 DEBUG:boto:Path: /jdpiak-72/b 2022-09-12T19:45:18.327 DEBUG:boto:Data: 2022-09-12T19:45:18.328 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.328 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.329 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.330 DEBUG:boto:Params: {} 2022-09-12T19:45:18.331 DEBUG:boto:Token: None 2022-09-12T19:45:18.331 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/b 2022-09-12T19:45:18.332 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aA9dlpUsFeJlzOxUg3QHxl3ceWs= 2022-09-12T19:45:18.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aA9dlpUsFeJlzOxUg3QHxl3ceWs=', 'Content-Length': '0'} 2022-09-12T19:45:18.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090148ba42967a8bc-00631f8c4e-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.337 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:18.339 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:18.339 DEBUG:boto:Method: HEAD 2022-09-12T19:45:18.340 DEBUG:boto:Path: /jdpiak-72/b 2022-09-12T19:45:18.340 DEBUG:boto:Data: 2022-09-12T19:45:18.341 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.342 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:18.342 DEBUG:boto:Port: 8000 2022-09-12T19:45:18.343 DEBUG:boto:Params: {} 2022-09-12T19:45:18.344 DEBUG:boto:Token: None 2022-09-12T19:45:18.344 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/b 2022-09-12T19:45:18.345 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aA9dlpUsFeJlzOxUg3QHxl3ceWs= 2022-09-12T19:45:18.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aA9dlpUsFeJlzOxUg3QHxl3ceWs=', 'Content-Length': '0'} 2022-09-12T19:45:18.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6638ac588cbe573-00631f8c4e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:45:18.349 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:18.349 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:18.350 DEBUG:boto:Method: GET 2022-09-12T19:45:18.351 DEBUG:boto:Path: /jdpiak-72/b 2022-09-12T19:45:18.351 DEBUG:boto:Data: 2022-09-12T19:45:18.352 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.352 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.353 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.353 DEBUG:boto:Params: {} 2022-09-12T19:45:18.354 DEBUG:boto:Token: None 2022-09-12T19:45:18.355 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/b 2022-09-12T19:45:18.355 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cUMiQvxMHtp+j/XI/ENTyO+oUtE= 2022-09-12T19:45:18.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cUMiQvxMHtp+j/XI/ENTyO+oUtE=', 'Content-Length': '0'} 2022-09-12T19:45:18.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060cdc731486b52b3-00631f8c4e-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.359 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:18.360 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:18.361 DEBUG:boto:Method: GET 2022-09-12T19:45:18.361 DEBUG:boto:Path: /jdpiak-72/b 2022-09-12T19:45:18.362 DEBUG:boto:Data: 2022-09-12T19:45:18.362 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.363 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:18.363 DEBUG:boto:Port: 8000 2022-09-12T19:45:18.364 DEBUG:boto:Params: {} 2022-09-12T19:45:18.365 DEBUG:boto:Token: None 2022-09-12T19:45:18.366 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/b 2022-09-12T19:45:18.366 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cUMiQvxMHtp+j/XI/ENTyO+oUtE= 2022-09-12T19:45:18.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cUMiQvxMHtp+j/XI/ENTyO+oUtE=', 'Content-Length': '0'} 2022-09-12T19:45:18.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c629fb74c3d80b8-00631f8c4e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.371 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:18.372 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:18.373 DEBUG:boto:path=/jdpiak-72/?prefix=b 2022-09-12T19:45:18.374 DEBUG:boto:auth_path=/jdpiak-72/?prefix=b 2022-09-12T19:45:18.375 DEBUG:boto:Method: GET 2022-09-12T19:45:18.376 DEBUG:boto:Path: /jdpiak-72/?prefix=b 2022-09-12T19:45:18.377 DEBUG:boto:Data: 2022-09-12T19:45:18.378 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.379 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.380 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.381 DEBUG:boto:Params: {} 2022-09-12T19:45:18.382 DEBUG:boto:Token: None 2022-09-12T19:45:18.382 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/ 2022-09-12T19:45:18.383 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk= 2022-09-12T19:45:18.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk=', 'Content-Length': '0'} 2022-09-12T19:45:18.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5c0e15d5d171736-00631f8c4e-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.388 DEBUG:boto:b'jdpiak-72b1000falseb2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:18.390 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:18.391 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:18.392 DEBUG:boto:path=/jdpiak-72/?prefix=b 2022-09-12T19:45:18.393 DEBUG:boto:auth_path=/jdpiak-72/?prefix=b 2022-09-12T19:45:18.394 DEBUG:boto:Method: GET 2022-09-12T19:45:18.395 DEBUG:boto:Path: /jdpiak-72/?prefix=b 2022-09-12T19:45:18.395 DEBUG:boto:Data: 2022-09-12T19:45:18.396 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.397 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:18.398 DEBUG:boto:Port: 8000 2022-09-12T19:45:18.399 DEBUG:boto:Params: {} 2022-09-12T19:45:18.400 DEBUG:boto:Token: None 2022-09-12T19:45:18.400 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/ 2022-09-12T19:45:18.401 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk= 2022-09-12T19:45:18.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk=', 'Content-Length': '0'} 2022-09-12T19:45:18.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070950dccd1279ffb-00631f8c4e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.406 DEBUG:boto:b'jdpiak-72b1000falseb2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:18.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:45:18.407 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:18.408 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:18.409 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:18.409 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:18.410 DEBUG:boto:Method: GET 2022-09-12T19:45:18.410 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:45:18.411 DEBUG:boto:Data: 2022-09-12T19:45:18.411 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.412 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.413 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.413 DEBUG:boto:Params: {} 2022-09-12T19:45:18.414 DEBUG:boto:Token: None 2022-09-12T19:45:18.415 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:45:18.415 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:98IOj1NCUJZK+dAwiDehZY+php4= 2022-09-12T19:45:18.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:98IOj1NCUJZK+dAwiDehZY+php4=', 'Content-Length': '0'} 2022-09-12T19:45:18.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc95a79e980ca62f-00631f8c4e-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.466 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:18.467 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:18.467 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:18.468 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:18.468 DEBUG:boto:Method: GET 2022-09-12T19:45:18.470 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:45:18.470 DEBUG:boto:Data: 2022-09-12T19:45:18.471 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.471 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:18.472 DEBUG:boto:Port: 8000 2022-09-12T19:45:18.472 DEBUG:boto:Params: {} 2022-09-12T19:45:18.473 DEBUG:boto:Token: None 2022-09-12T19:45:18.474 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:45:18.474 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:98IOj1NCUJZK+dAwiDehZY+php4= 2022-09-12T19:45:18.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:98IOj1NCUJZK+dAwiDehZY+php4=', 'Content-Length': '0'} 2022-09-12T19:45:18.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007be0c4f153c2499e-00631f8c4e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.478 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:18.479 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:18.479 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:18.480 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:18.480 DEBUG:boto:Method: HEAD 2022-09-12T19:45:18.481 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:45:18.481 DEBUG:boto:Data: 2022-09-12T19:45:18.482 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.482 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.483 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.483 DEBUG:boto:Params: {} 2022-09-12T19:45:18.484 DEBUG:boto:Token: None 2022-09-12T19:45:18.484 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:45:18.485 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rAKK7+MADicK+Xu6yOG8i7CCsKE= 2022-09-12T19:45:18.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rAKK7+MADicK+Xu6yOG8i7CCsKE=', 'Content-Length': '0'} 2022-09-12T19:45:18.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e090fbf51adec50-00631f8c4e-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.489 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:18.489 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:18.490 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:18.490 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:18.491 DEBUG:boto:Method: HEAD 2022-09-12T19:45:18.491 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:45:18.492 DEBUG:boto:Data: 2022-09-12T19:45:18.492 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.493 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:18.493 DEBUG:boto:Port: 8000 2022-09-12T19:45:18.494 DEBUG:boto:Params: {} 2022-09-12T19:45:18.494 DEBUG:boto:Token: None 2022-09-12T19:45:18.495 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:45:18.496 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rAKK7+MADicK+Xu6yOG8i7CCsKE= 2022-09-12T19:45:18.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rAKK7+MADicK+Xu6yOG8i7CCsKE=', 'Content-Length': '0'} 2022-09-12T19:45:18.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e2aa36be9f4d7fe-00631f8c4e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:45:18.500 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:18.500 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:18.501 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:18.501 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:18.502 DEBUG:boto:Method: GET 2022-09-12T19:45:18.502 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:45:18.503 DEBUG:boto:Data: 2022-09-12T19:45:18.504 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.504 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.505 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.505 DEBUG:boto:Params: {} 2022-09-12T19:45:18.506 DEBUG:boto:Token: None 2022-09-12T19:45:18.507 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:45:18.507 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:98IOj1NCUJZK+dAwiDehZY+php4= 2022-09-12T19:45:18.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:98IOj1NCUJZK+dAwiDehZY+php4=', 'Content-Length': '0'} 2022-09-12T19:45:18.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004531dbe283efc48b-00631f8c4e-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.511 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:18.512 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:18.512 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:18.513 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:18.513 DEBUG:boto:Method: GET 2022-09-12T19:45:18.514 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:45:18.514 DEBUG:boto:Data: 2022-09-12T19:45:18.515 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.515 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:18.516 DEBUG:boto:Port: 8000 2022-09-12T19:45:18.516 DEBUG:boto:Params: {} 2022-09-12T19:45:18.517 DEBUG:boto:Token: None 2022-09-12T19:45:18.518 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:45:18.519 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:98IOj1NCUJZK+dAwiDehZY+php4= 2022-09-12T19:45:18.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:98IOj1NCUJZK+dAwiDehZY+php4=', 'Content-Length': '0'} 2022-09-12T19:45:18.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043c31cf3ee3844fc-00631f8c4e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.566 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:18.567 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:18.567 DEBUG:boto:path=/jdpiak-72/?prefix=c 2022-09-12T19:45:18.568 DEBUG:boto:auth_path=/jdpiak-72/?prefix=c 2022-09-12T19:45:18.568 DEBUG:boto:Method: GET 2022-09-12T19:45:18.569 DEBUG:boto:Path: /jdpiak-72/?prefix=c 2022-09-12T19:45:18.570 DEBUG:boto:Data: 2022-09-12T19:45:18.571 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.571 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.572 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.572 DEBUG:boto:Params: {} 2022-09-12T19:45:18.573 DEBUG:boto:Token: None 2022-09-12T19:45:18.574 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/ 2022-09-12T19:45:18.574 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk= 2022-09-12T19:45:18.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk=', 'Content-Length': '0'} 2022-09-12T19:45:18.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af8126ccd232df78-00631f8c4e-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.579 DEBUG:boto:b'jdpiak-72c1000falsec2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:18.580 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:18.581 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:18.582 DEBUG:boto:path=/jdpiak-72/?prefix=c 2022-09-12T19:45:18.583 DEBUG:boto:auth_path=/jdpiak-72/?prefix=c 2022-09-12T19:45:18.584 DEBUG:boto:Method: GET 2022-09-12T19:45:18.584 DEBUG:boto:Path: /jdpiak-72/?prefix=c 2022-09-12T19:45:18.585 DEBUG:boto:Data: 2022-09-12T19:45:18.586 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.599 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:18.601 DEBUG:boto:Port: 8000 2022-09-12T19:45:18.602 DEBUG:boto:Params: {} 2022-09-12T19:45:18.603 DEBUG:boto:Token: None 2022-09-12T19:45:18.604 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/ 2022-09-12T19:45:18.605 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk= 2022-09-12T19:45:18.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk=', 'Content-Length': '0'} 2022-09-12T19:45:18.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000539fc9c77072c68a-00631f8c4e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.611 DEBUG:boto:b'jdpiak-72c1000falsec2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:18.612 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:18.612 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:18.613 DEBUG:boto:Method: HEAD 2022-09-12T19:45:18.613 DEBUG:boto:Path: /jdpiak-72/c 2022-09-12T19:45:18.614 DEBUG:boto:Data: 2022-09-12T19:45:18.614 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.615 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.616 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.616 DEBUG:boto:Params: {} 2022-09-12T19:45:18.617 DEBUG:boto:Token: None 2022-09-12T19:45:18.617 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/c 2022-09-12T19:45:18.618 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+c9EWFhRgiTw8D1rqKjuzhMJQPU= 2022-09-12T19:45:18.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+c9EWFhRgiTw8D1rqKjuzhMJQPU=', 'Content-Length': '0'} 2022-09-12T19:45:18.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000486ff09d387d3547-00631f8c4e-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.621 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:18.622 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:18.623 DEBUG:boto:Method: HEAD 2022-09-12T19:45:18.623 DEBUG:boto:Path: /jdpiak-72/c 2022-09-12T19:45:18.624 DEBUG:boto:Data: 2022-09-12T19:45:18.624 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.625 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:18.625 DEBUG:boto:Port: 8000 2022-09-12T19:45:18.626 DEBUG:boto:Params: {} 2022-09-12T19:45:18.627 DEBUG:boto:Token: None 2022-09-12T19:45:18.627 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/c 2022-09-12T19:45:18.628 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+c9EWFhRgiTw8D1rqKjuzhMJQPU= 2022-09-12T19:45:18.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+c9EWFhRgiTw8D1rqKjuzhMJQPU=', 'Content-Length': '0'} 2022-09-12T19:45:18.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001483692a2ccd3ad4-00631f8c4e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:45:18.632 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:18.632 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:18.633 DEBUG:boto:Method: GET 2022-09-12T19:45:18.633 DEBUG:boto:Path: /jdpiak-72/c 2022-09-12T19:45:18.634 DEBUG:boto:Data: 2022-09-12T19:45:18.635 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.635 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.636 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.636 DEBUG:boto:Params: {} 2022-09-12T19:45:18.637 DEBUG:boto:Token: None 2022-09-12T19:45:18.637 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/c 2022-09-12T19:45:18.638 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dckWiW+YSufcUnY0NrJVuzkRHSA= 2022-09-12T19:45:18.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dckWiW+YSufcUnY0NrJVuzkRHSA=', 'Content-Length': '0'} 2022-09-12T19:45:18.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f82849f05996d95-00631f8c4e-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.642 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:18.642 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:18.643 DEBUG:boto:Method: GET 2022-09-12T19:45:18.643 DEBUG:boto:Path: /jdpiak-72/c 2022-09-12T19:45:18.644 DEBUG:boto:Data: 2022-09-12T19:45:18.644 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.645 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:18.646 DEBUG:boto:Port: 8000 2022-09-12T19:45:18.646 DEBUG:boto:Params: {} 2022-09-12T19:45:18.647 DEBUG:boto:Token: None 2022-09-12T19:45:18.647 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/c 2022-09-12T19:45:18.648 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dckWiW+YSufcUnY0NrJVuzkRHSA= 2022-09-12T19:45:18.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dckWiW+YSufcUnY0NrJVuzkRHSA=', 'Content-Length': '0'} 2022-09-12T19:45:18.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a60514fe9739048c-00631f8c4e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.652 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:18.652 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:18.653 DEBUG:boto:path=/jdpiak-72/?prefix=c 2022-09-12T19:45:18.653 DEBUG:boto:auth_path=/jdpiak-72/?prefix=c 2022-09-12T19:45:18.654 DEBUG:boto:Method: GET 2022-09-12T19:45:18.655 DEBUG:boto:Path: /jdpiak-72/?prefix=c 2022-09-12T19:45:18.655 DEBUG:boto:Data: 2022-09-12T19:45:18.656 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.656 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.657 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.657 DEBUG:boto:Params: {} 2022-09-12T19:45:18.658 DEBUG:boto:Token: None 2022-09-12T19:45:18.659 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/ 2022-09-12T19:45:18.659 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk= 2022-09-12T19:45:18.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk=', 'Content-Length': '0'} 2022-09-12T19:45:18.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001728688b5f2df43a-00631f8c4e-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.663 DEBUG:boto:b'jdpiak-72c1000falsec2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:18.664 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:18.665 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:18.665 DEBUG:boto:path=/jdpiak-72/?prefix=c 2022-09-12T19:45:18.666 DEBUG:boto:auth_path=/jdpiak-72/?prefix=c 2022-09-12T19:45:18.666 DEBUG:boto:Method: GET 2022-09-12T19:45:18.667 DEBUG:boto:Path: /jdpiak-72/?prefix=c 2022-09-12T19:45:18.668 DEBUG:boto:Data: 2022-09-12T19:45:18.668 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.669 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:18.669 DEBUG:boto:Port: 8000 2022-09-12T19:45:18.670 DEBUG:boto:Params: {} 2022-09-12T19:45:18.670 DEBUG:boto:Token: None 2022-09-12T19:45:18.671 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/ 2022-09-12T19:45:18.672 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk= 2022-09-12T19:45:18.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk=', 'Content-Length': '0'} 2022-09-12T19:45:18.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008772cfbbe77f96d2-00631f8c4e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.676 DEBUG:boto:b'jdpiak-72c1000falsec2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:18.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:45:18.677 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:18.678 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:18.679 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:18.679 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:18.680 DEBUG:boto:Method: GET 2022-09-12T19:45:18.680 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:45:18.681 DEBUG:boto:Data: 2022-09-12T19:45:18.681 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.682 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.682 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.683 DEBUG:boto:Params: {} 2022-09-12T19:45:18.684 DEBUG:boto:Token: None 2022-09-12T19:45:18.684 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:45:18.685 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p/hsaxLKJrlbxfJeRCEUF4nBsT4= 2022-09-12T19:45:18.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p/hsaxLKJrlbxfJeRCEUF4nBsT4=', 'Content-Length': '0'} 2022-09-12T19:45:18.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042da425183082b3a-00631f8c4e-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.689 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:18.689 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:18.690 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:18.690 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:18.691 DEBUG:boto:Method: GET 2022-09-12T19:45:18.691 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:45:18.692 DEBUG:boto:Data: 2022-09-12T19:45:18.693 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.693 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:18.694 DEBUG:boto:Port: 8000 2022-09-12T19:45:18.694 DEBUG:boto:Params: {} 2022-09-12T19:45:18.695 DEBUG:boto:Token: None 2022-09-12T19:45:18.696 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:45:18.696 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p/hsaxLKJrlbxfJeRCEUF4nBsT4= 2022-09-12T19:45:18.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p/hsaxLKJrlbxfJeRCEUF4nBsT4=', 'Content-Length': '0'} 2022-09-12T19:45:18.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026885cedeaf9519e-00631f8c4e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.742 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:18.743 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:18.743 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:18.744 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:18.744 DEBUG:boto:Method: HEAD 2022-09-12T19:45:18.745 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:45:18.745 DEBUG:boto:Data: 2022-09-12T19:45:18.746 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.746 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.747 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.748 DEBUG:boto:Params: {} 2022-09-12T19:45:18.748 DEBUG:boto:Token: None 2022-09-12T19:45:18.749 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:45:18.749 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fX03PZZcRT5svRra4cmoB7swsmM= 2022-09-12T19:45:18.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fX03PZZcRT5svRra4cmoB7swsmM=', 'Content-Length': '0'} 2022-09-12T19:45:18.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a547354736f493c-00631f8c4e-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.755 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:18.756 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:18.757 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:18.758 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:18.759 DEBUG:boto:Method: HEAD 2022-09-12T19:45:18.760 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:45:18.761 DEBUG:boto:Data: 2022-09-12T19:45:18.762 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.763 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:18.764 DEBUG:boto:Port: 8000 2022-09-12T19:45:18.765 DEBUG:boto:Params: {} 2022-09-12T19:45:18.766 DEBUG:boto:Token: None 2022-09-12T19:45:18.767 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:45:18.768 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fX03PZZcRT5svRra4cmoB7swsmM= 2022-09-12T19:45:18.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fX03PZZcRT5svRra4cmoB7swsmM=', 'Content-Length': '0'} 2022-09-12T19:45:18.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029c6d6b05356cd05-00631f8c4e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:45:18.774 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:18.775 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:18.776 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:18.777 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:18.778 DEBUG:boto:Method: GET 2022-09-12T19:45:18.779 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:45:18.780 DEBUG:boto:Data: 2022-09-12T19:45:18.782 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.783 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.783 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.784 DEBUG:boto:Params: {} 2022-09-12T19:45:18.785 DEBUG:boto:Token: None 2022-09-12T19:45:18.785 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:45:18.786 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p/hsaxLKJrlbxfJeRCEUF4nBsT4= 2022-09-12T19:45:18.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p/hsaxLKJrlbxfJeRCEUF4nBsT4=', 'Content-Length': '0'} 2022-09-12T19:45:18.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d12818977a300952-00631f8c4e-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.790 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:18.790 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:18.791 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:18.792 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:18.792 DEBUG:boto:Method: GET 2022-09-12T19:45:18.793 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:45:18.793 DEBUG:boto:Data: 2022-09-12T19:45:18.794 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.795 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:18.795 DEBUG:boto:Port: 8000 2022-09-12T19:45:18.796 DEBUG:boto:Params: {} 2022-09-12T19:45:18.796 DEBUG:boto:Token: None 2022-09-12T19:45:18.797 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:45:18.798 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p/hsaxLKJrlbxfJeRCEUF4nBsT4= 2022-09-12T19:45:18.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p/hsaxLKJrlbxfJeRCEUF4nBsT4=', 'Content-Length': '0'} 2022-09-12T19:45:18.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e08fa454eaa07666-00631f8c4e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.804 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:18.805 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:18.806 DEBUG:boto:path=/jdpiak-72/?prefix=d 2022-09-12T19:45:18.807 DEBUG:boto:auth_path=/jdpiak-72/?prefix=d 2022-09-12T19:45:18.807 DEBUG:boto:Method: GET 2022-09-12T19:45:18.808 DEBUG:boto:Path: /jdpiak-72/?prefix=d 2022-09-12T19:45:18.808 DEBUG:boto:Data: 2022-09-12T19:45:18.809 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.810 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.810 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.811 DEBUG:boto:Params: {} 2022-09-12T19:45:18.811 DEBUG:boto:Token: None 2022-09-12T19:45:18.813 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/ 2022-09-12T19:45:18.814 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk= 2022-09-12T19:45:18.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk=', 'Content-Length': '0'} 2022-09-12T19:45:18.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fcb10863a2913a5-00631f8c4e-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.819 DEBUG:boto:b'jdpiak-72d1000falsed2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:18.820 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:18.821 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:18.822 DEBUG:boto:path=/jdpiak-72/?prefix=d 2022-09-12T19:45:18.822 DEBUG:boto:auth_path=/jdpiak-72/?prefix=d 2022-09-12T19:45:18.823 DEBUG:boto:Method: GET 2022-09-12T19:45:18.823 DEBUG:boto:Path: /jdpiak-72/?prefix=d 2022-09-12T19:45:18.824 DEBUG:boto:Data: 2022-09-12T19:45:18.825 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.825 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:18.826 DEBUG:boto:Port: 8000 2022-09-12T19:45:18.827 DEBUG:boto:Params: {} 2022-09-12T19:45:18.827 DEBUG:boto:Token: None 2022-09-12T19:45:18.828 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/ 2022-09-12T19:45:18.829 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk= 2022-09-12T19:45:18.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk=', 'Content-Length': '0'} 2022-09-12T19:45:18.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e499f7b6de47badf-00631f8c4e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.834 DEBUG:boto:b'jdpiak-72d1000falsed2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:18.835 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:18.836 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:18.837 DEBUG:boto:Method: HEAD 2022-09-12T19:45:18.838 DEBUG:boto:Path: /jdpiak-72/d 2022-09-12T19:45:18.838 DEBUG:boto:Data: 2022-09-12T19:45:18.839 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.839 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.840 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.840 DEBUG:boto:Params: {} 2022-09-12T19:45:18.841 DEBUG:boto:Token: None 2022-09-12T19:45:18.842 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/d 2022-09-12T19:45:18.843 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uOrRwF2BPyGsLy81gPulQhPCRh0= 2022-09-12T19:45:18.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uOrRwF2BPyGsLy81gPulQhPCRh0=', 'Content-Length': '0'} 2022-09-12T19:45:18.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a514047476139db1-00631f8c4e-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.846 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:18.847 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:18.848 DEBUG:boto:Method: HEAD 2022-09-12T19:45:18.848 DEBUG:boto:Path: /jdpiak-72/d 2022-09-12T19:45:18.849 DEBUG:boto:Data: 2022-09-12T19:45:18.849 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.850 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:18.850 DEBUG:boto:Port: 8000 2022-09-12T19:45:18.851 DEBUG:boto:Params: {} 2022-09-12T19:45:18.852 DEBUG:boto:Token: None 2022-09-12T19:45:18.853 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/d 2022-09-12T19:45:18.853 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uOrRwF2BPyGsLy81gPulQhPCRh0= 2022-09-12T19:45:18.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uOrRwF2BPyGsLy81gPulQhPCRh0=', 'Content-Length': '0'} 2022-09-12T19:45:18.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c57b46832abac5eb-00631f8c4e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:45:18.860 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:18.860 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:18.861 DEBUG:boto:Method: GET 2022-09-12T19:45:18.862 DEBUG:boto:Path: /jdpiak-72/d 2022-09-12T19:45:18.864 DEBUG:boto:Data: 2022-09-12T19:45:18.865 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.865 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.866 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.867 DEBUG:boto:Params: {} 2022-09-12T19:45:18.868 DEBUG:boto:Token: None 2022-09-12T19:45:18.869 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/d 2022-09-12T19:45:18.870 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W4SeVeZnD4ZbbcCOzrGM+g1uDIo= 2022-09-12T19:45:18.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W4SeVeZnD4ZbbcCOzrGM+g1uDIo=', 'Content-Length': '0'} 2022-09-12T19:45:18.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000797ad6e8797cdd8e-00631f8c4e-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.918 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:18.920 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:18.921 DEBUG:boto:Method: GET 2022-09-12T19:45:18.922 DEBUG:boto:Path: /jdpiak-72/d 2022-09-12T19:45:18.923 DEBUG:boto:Data: 2022-09-12T19:45:18.924 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.925 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:18.927 DEBUG:boto:Port: 8000 2022-09-12T19:45:18.928 DEBUG:boto:Params: {} 2022-09-12T19:45:18.929 DEBUG:boto:Token: None 2022-09-12T19:45:18.931 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/d 2022-09-12T19:45:18.931 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W4SeVeZnD4ZbbcCOzrGM+g1uDIo= 2022-09-12T19:45:18.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W4SeVeZnD4ZbbcCOzrGM+g1uDIo=', 'Content-Length': '0'} 2022-09-12T19:45:18.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ba3c2fd4a2080f7-00631f8c4e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.936 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:18.937 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:18.938 DEBUG:boto:path=/jdpiak-72/?prefix=d 2022-09-12T19:45:18.939 DEBUG:boto:auth_path=/jdpiak-72/?prefix=d 2022-09-12T19:45:18.940 DEBUG:boto:Method: GET 2022-09-12T19:45:18.941 DEBUG:boto:Path: /jdpiak-72/?prefix=d 2022-09-12T19:45:18.942 DEBUG:boto:Data: 2022-09-12T19:45:18.943 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.943 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.944 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.945 DEBUG:boto:Params: {} 2022-09-12T19:45:18.947 DEBUG:boto:Token: None 2022-09-12T19:45:18.949 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/ 2022-09-12T19:45:18.951 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk= 2022-09-12T19:45:18.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk=', 'Content-Length': '0'} 2022-09-12T19:45:18.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008936b60628f80805-00631f8c4e-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.958 DEBUG:boto:b'jdpiak-72d1000falsed2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:18.960 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:18.962 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:18.963 DEBUG:boto:path=/jdpiak-72/?prefix=d 2022-09-12T19:45:18.964 DEBUG:boto:auth_path=/jdpiak-72/?prefix=d 2022-09-12T19:45:18.966 DEBUG:boto:Method: GET 2022-09-12T19:45:18.967 DEBUG:boto:Path: /jdpiak-72/?prefix=d 2022-09-12T19:45:18.967 DEBUG:boto:Data: 2022-09-12T19:45:18.968 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.969 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:18.970 DEBUG:boto:Port: 8000 2022-09-12T19:45:18.971 DEBUG:boto:Params: {} 2022-09-12T19:45:18.972 DEBUG:boto:Token: None 2022-09-12T19:45:18.973 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/ 2022-09-12T19:45:18.973 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk= 2022-09-12T19:45:18.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rOQUrtx1/meudY/mpCeH1K67fUk=', 'Content-Length': '0'} 2022-09-12T19:45:18.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d98a2e37179af086-00631f8c4e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:18.979 DEBUG:boto:b'jdpiak-72d1000falsed2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:18.980 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-72 zones={test-zone3, test-zone1} 2022-09-12T19:45:18.981 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-72 zones={test-zone3, test-zone2} 2022-09-12T19:45:18.981 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:18.983 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:18.985 DEBUG:boto:Method: HEAD 2022-09-12T19:45:18.986 DEBUG:boto:Path: /jdpiak-72/ 2022-09-12T19:45:18.987 DEBUG:boto:Data: 2022-09-12T19:45:18.988 DEBUG:boto:Headers: {} 2022-09-12T19:45:18.990 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:18.992 DEBUG:boto:Port: 8001 2022-09-12T19:45:18.994 DEBUG:boto:Params: {} 2022-09-12T19:45:18.996 DEBUG:boto:Token: None 2022-09-12T19:45:18.996 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:18 GMT /jdpiak-72/ 2022-09-12T19:45:18.997 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IguS8/LjxtBZxXmtrwZtkW1bok8= 2022-09-12T19:45:18.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IguS8/LjxtBZxXmtrwZtkW1bok8=', 'Content-Length': '0'} 2022-09-12T19:45:19.001 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000dd9428daddbf541-00631f8c4e-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.017 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:19.018 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:19.019 DEBUG:boto:Method: HEAD 2022-09-12T19:45:19.020 DEBUG:boto:Path: /jdpiak-72/ 2022-09-12T19:45:19.021 DEBUG:boto:Data: 2022-09-12T19:45:19.022 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.023 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:19.024 DEBUG:boto:Port: 8000 2022-09-12T19:45:19.025 DEBUG:boto:Params: {} 2022-09-12T19:45:19.026 DEBUG:boto:Token: None 2022-09-12T19:45:19.027 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/ 2022-09-12T19:45:19.028 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:esBOIVEMRybve2vI4GpcxKUmdhE= 2022-09-12T19:45:19.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:esBOIVEMRybve2vI4GpcxKUmdhE=', 'Content-Length': '0'} 2022-09-12T19:45:19.032 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030d8b46602f50ce2-00631f8c4f-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.033 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:45:19.035 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:19.035 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:19.036 DEBUG:boto:path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:19.037 DEBUG:boto:auth_path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:19.038 DEBUG:boto:Method: GET 2022-09-12T19:45:19.039 DEBUG:boto:Path: /jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:19.040 DEBUG:boto:Data: 2022-09-12T19:45:19.041 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.042 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:19.043 DEBUG:boto:Port: 8001 2022-09-12T19:45:19.044 DEBUG:boto:Params: {} 2022-09-12T19:45:19.045 DEBUG:boto:Token: None 2022-09-12T19:45:19.046 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/?versions 2022-09-12T19:45:19.046 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WA2VIAH+t990mqZjt5AxIDS3QMk= 2022-09-12T19:45:19.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WA2VIAH+t990mqZjt5AxIDS3QMk=', 'Content-Length': '0'} 2022-09-12T19:45:19.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ef2316c614f3d71-00631f8c4f-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.052 DEBUG:boto:b'jdpiak-72999falseanulltrue2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:19.053 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:45:19.054 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:45:19.054 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:45:19.054 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:45:19.055 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:45:19.055 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:19.056 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:19.057 DEBUG:boto:path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:19.057 DEBUG:boto:auth_path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:19.058 DEBUG:boto:Method: GET 2022-09-12T19:45:19.058 DEBUG:boto:Path: /jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:19.059 DEBUG:boto:Data: 2022-09-12T19:45:19.059 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.060 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:19.060 DEBUG:boto:Port: 8000 2022-09-12T19:45:19.061 DEBUG:boto:Params: {} 2022-09-12T19:45:19.061 DEBUG:boto:Token: None 2022-09-12T19:45:19.062 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/?versions 2022-09-12T19:45:19.063 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WA2VIAH+t990mqZjt5AxIDS3QMk= 2022-09-12T19:45:19.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WA2VIAH+t990mqZjt5AxIDS3QMk=', 'Content-Length': '0'} 2022-09-12T19:45:19.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5908e75f4b35e21-00631f8c4f-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.072 DEBUG:boto:b'jdpiak-72999falseanulltrue2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:19.073 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:45:19.074 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:45:19.074 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:45:19.075 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:45:19.076 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:19.076 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:19.077 DEBUG:boto:path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:19.077 DEBUG:boto:auth_path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:19.078 DEBUG:boto:Method: GET 2022-09-12T19:45:19.078 DEBUG:boto:Path: /jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:19.079 DEBUG:boto:Data: 2022-09-12T19:45:19.079 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.080 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:19.080 DEBUG:boto:Port: 8001 2022-09-12T19:45:19.081 DEBUG:boto:Params: {} 2022-09-12T19:45:19.082 DEBUG:boto:Token: None 2022-09-12T19:45:19.083 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/?versions 2022-09-12T19:45:19.083 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WA2VIAH+t990mqZjt5AxIDS3QMk= 2022-09-12T19:45:19.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WA2VIAH+t990mqZjt5AxIDS3QMk=', 'Content-Length': '0'} 2022-09-12T19:45:19.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007418e51dcc725400-00631f8c4f-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.087 DEBUG:boto:b'jdpiak-72999falseanulltrue2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:19.089 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:19.089 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:19.090 DEBUG:boto:path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:19.090 DEBUG:boto:auth_path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:19.091 DEBUG:boto:Method: GET 2022-09-12T19:45:19.091 DEBUG:boto:Path: /jdpiak-72/?versions&max-keys=999 2022-09-12T19:45:19.092 DEBUG:boto:Data: 2022-09-12T19:45:19.092 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.093 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:19.094 DEBUG:boto:Port: 8000 2022-09-12T19:45:19.094 DEBUG:boto:Params: {} 2022-09-12T19:45:19.095 DEBUG:boto:Token: None 2022-09-12T19:45:19.096 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/?versions 2022-09-12T19:45:19.096 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WA2VIAH+t990mqZjt5AxIDS3QMk= 2022-09-12T19:45:19.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WA2VIAH+t990mqZjt5AxIDS3QMk=', 'Content-Length': '0'} 2022-09-12T19:45:19.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002277c9e542f2387-00631f8c4f-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.100 DEBUG:boto:b'jdpiak-72999falseanulltrue2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:19.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:45:19.102 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:19.102 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:19.103 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:19.103 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:19.104 DEBUG:boto:Method: GET 2022-09-12T19:45:19.105 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:45:19.105 DEBUG:boto:Data: 2022-09-12T19:45:19.106 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.106 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:19.107 DEBUG:boto:Port: 8001 2022-09-12T19:45:19.108 DEBUG:boto:Params: {} 2022-09-12T19:45:19.108 DEBUG:boto:Token: None 2022-09-12T19:45:19.109 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:45:19.109 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F5mx+dluR01yT48xYZVOzK7CxSA= 2022-09-12T19:45:19.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F5mx+dluR01yT48xYZVOzK7CxSA=', 'Content-Length': '0'} 2022-09-12T19:45:19.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000189dcd335d9f0649-00631f8c4f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.158 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:19.159 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:19.159 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:19.160 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:19.160 DEBUG:boto:Method: GET 2022-09-12T19:45:19.161 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:45:19.162 DEBUG:boto:Data: 2022-09-12T19:45:19.162 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.163 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:19.163 DEBUG:boto:Port: 8000 2022-09-12T19:45:19.164 DEBUG:boto:Params: {} 2022-09-12T19:45:19.164 DEBUG:boto:Token: None 2022-09-12T19:45:19.165 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:45:19.166 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F5mx+dluR01yT48xYZVOzK7CxSA= 2022-09-12T19:45:19.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F5mx+dluR01yT48xYZVOzK7CxSA=', 'Content-Length': '0'} 2022-09-12T19:45:19.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019e0e8cc92244cce-00631f8c4f-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.169 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:19.170 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:19.171 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:19.171 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:19.172 DEBUG:boto:Method: HEAD 2022-09-12T19:45:19.172 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:45:19.173 DEBUG:boto:Data: 2022-09-12T19:45:19.173 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.174 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:19.174 DEBUG:boto:Port: 8001 2022-09-12T19:45:19.175 DEBUG:boto:Params: {} 2022-09-12T19:45:19.175 DEBUG:boto:Token: None 2022-09-12T19:45:19.176 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:45:19.176 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eGuQUzS6ITHr1RweoJZ4HMoTbkI= 2022-09-12T19:45:19.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eGuQUzS6ITHr1RweoJZ4HMoTbkI=', 'Content-Length': '0'} 2022-09-12T19:45:19.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052a2fe8e4b61dc0d-00631f8c4f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.180 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:19.180 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:19.181 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:19.181 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:19.182 DEBUG:boto:Method: HEAD 2022-09-12T19:45:19.182 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:45:19.183 DEBUG:boto:Data: 2022-09-12T19:45:19.183 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.184 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:19.184 DEBUG:boto:Port: 8000 2022-09-12T19:45:19.185 DEBUG:boto:Params: {} 2022-09-12T19:45:19.185 DEBUG:boto:Token: None 2022-09-12T19:45:19.186 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:45:19.187 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eGuQUzS6ITHr1RweoJZ4HMoTbkI= 2022-09-12T19:45:19.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eGuQUzS6ITHr1RweoJZ4HMoTbkI=', 'Content-Length': '0'} 2022-09-12T19:45:19.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008e8a92321ce17e7-00631f8c4f-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:45:19.191 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:19.191 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:19.192 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:19.192 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:19.193 DEBUG:boto:Method: GET 2022-09-12T19:45:19.194 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:45:19.194 DEBUG:boto:Data: 2022-09-12T19:45:19.195 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.195 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:19.196 DEBUG:boto:Port: 8001 2022-09-12T19:45:19.196 DEBUG:boto:Params: {} 2022-09-12T19:45:19.197 DEBUG:boto:Token: None 2022-09-12T19:45:19.197 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:45:19.198 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F5mx+dluR01yT48xYZVOzK7CxSA= 2022-09-12T19:45:19.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F5mx+dluR01yT48xYZVOzK7CxSA=', 'Content-Length': '0'} 2022-09-12T19:45:19.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b46ff43f725bdf20-00631f8c4f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.201 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:19.202 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:19.203 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:19.203 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:45:19.204 DEBUG:boto:Method: GET 2022-09-12T19:45:19.204 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:45:19.205 DEBUG:boto:Data: 2022-09-12T19:45:19.205 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.206 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:19.206 DEBUG:boto:Port: 8000 2022-09-12T19:45:19.207 DEBUG:boto:Params: {} 2022-09-12T19:45:19.207 DEBUG:boto:Token: None 2022-09-12T19:45:19.208 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:45:19.209 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F5mx+dluR01yT48xYZVOzK7CxSA= 2022-09-12T19:45:19.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F5mx+dluR01yT48xYZVOzK7CxSA=', 'Content-Length': '0'} 2022-09-12T19:45:19.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d97eaaef30196750-00631f8c4f-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.212 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:19.213 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:19.213 DEBUG:boto:path=/jdpiak-72/?prefix=a 2022-09-12T19:45:19.214 DEBUG:boto:auth_path=/jdpiak-72/?prefix=a 2022-09-12T19:45:19.214 DEBUG:boto:Method: GET 2022-09-12T19:45:19.215 DEBUG:boto:Path: /jdpiak-72/?prefix=a 2022-09-12T19:45:19.216 DEBUG:boto:Data: 2022-09-12T19:45:19.216 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.217 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:19.217 DEBUG:boto:Port: 8001 2022-09-12T19:45:19.218 DEBUG:boto:Params: {} 2022-09-12T19:45:19.218 DEBUG:boto:Token: None 2022-09-12T19:45:19.219 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/ 2022-09-12T19:45:19.220 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OU4Q3EDnxTzpJA48xkI457y5F+E= 2022-09-12T19:45:19.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OU4Q3EDnxTzpJA48xkI457y5F+E=', 'Content-Length': '0'} 2022-09-12T19:45:19.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f9fe438b601e809-00631f8c4f-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.224 DEBUG:boto:b'jdpiak-72a1000falsea2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:19.224 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:19.225 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:19.226 DEBUG:boto:path=/jdpiak-72/?prefix=a 2022-09-12T19:45:19.226 DEBUG:boto:auth_path=/jdpiak-72/?prefix=a 2022-09-12T19:45:19.227 DEBUG:boto:Method: GET 2022-09-12T19:45:19.228 DEBUG:boto:Path: /jdpiak-72/?prefix=a 2022-09-12T19:45:19.228 DEBUG:boto:Data: 2022-09-12T19:45:19.229 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.229 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:19.230 DEBUG:boto:Port: 8000 2022-09-12T19:45:19.230 DEBUG:boto:Params: {} 2022-09-12T19:45:19.231 DEBUG:boto:Token: None 2022-09-12T19:45:19.231 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/ 2022-09-12T19:45:19.232 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OU4Q3EDnxTzpJA48xkI457y5F+E= 2022-09-12T19:45:19.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OU4Q3EDnxTzpJA48xkI457y5F+E=', 'Content-Length': '0'} 2022-09-12T19:45:19.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099e922aaca23efd8-00631f8c4f-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.236 DEBUG:boto:b'jdpiak-72a1000falsea2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:19.238 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:19.238 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:19.239 DEBUG:boto:Method: HEAD 2022-09-12T19:45:19.239 DEBUG:boto:Path: /jdpiak-72/a 2022-09-12T19:45:19.240 DEBUG:boto:Data: 2022-09-12T19:45:19.240 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.241 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:19.241 DEBUG:boto:Port: 8001 2022-09-12T19:45:19.242 DEBUG:boto:Params: {} 2022-09-12T19:45:19.243 DEBUG:boto:Token: None 2022-09-12T19:45:19.243 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/a 2022-09-12T19:45:19.244 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c0zhqg/xlVkq3KLxYx+N/sRlwyU= 2022-09-12T19:45:19.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c0zhqg/xlVkq3KLxYx+N/sRlwyU=', 'Content-Length': '0'} 2022-09-12T19:45:19.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b248585e8e277d0-00631f8c4f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.248 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:19.248 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:19.249 DEBUG:boto:Method: HEAD 2022-09-12T19:45:19.249 DEBUG:boto:Path: /jdpiak-72/a 2022-09-12T19:45:19.250 DEBUG:boto:Data: 2022-09-12T19:45:19.251 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.251 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:19.252 DEBUG:boto:Port: 8000 2022-09-12T19:45:19.252 DEBUG:boto:Params: {} 2022-09-12T19:45:19.253 DEBUG:boto:Token: None 2022-09-12T19:45:19.254 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/a 2022-09-12T19:45:19.255 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c0zhqg/xlVkq3KLxYx+N/sRlwyU= 2022-09-12T19:45:19.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c0zhqg/xlVkq3KLxYx+N/sRlwyU=', 'Content-Length': '0'} 2022-09-12T19:45:19.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbfc03374c822926-00631f8c4f-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:45:19.259 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:19.259 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:19.260 DEBUG:boto:Method: GET 2022-09-12T19:45:19.261 DEBUG:boto:Path: /jdpiak-72/a 2022-09-12T19:45:19.261 DEBUG:boto:Data: 2022-09-12T19:45:19.262 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.262 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:19.263 DEBUG:boto:Port: 8001 2022-09-12T19:45:19.264 DEBUG:boto:Params: {} 2022-09-12T19:45:19.265 DEBUG:boto:Token: None 2022-09-12T19:45:19.265 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/a 2022-09-12T19:45:19.266 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BQERg4w+Oy3QTcEchDtDT3otE1Y= 2022-09-12T19:45:19.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BQERg4w+Oy3QTcEchDtDT3otE1Y=', 'Content-Length': '0'} 2022-09-12T19:45:19.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa022413bd1723b5-00631f8c4f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.314 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:45:19.315 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:45:19.316 DEBUG:boto:Method: GET 2022-09-12T19:45:19.316 DEBUG:boto:Path: /jdpiak-72/a 2022-09-12T19:45:19.317 DEBUG:boto:Data: 2022-09-12T19:45:19.317 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.318 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:19.319 DEBUG:boto:Port: 8000 2022-09-12T19:45:19.320 DEBUG:boto:Params: {} 2022-09-12T19:45:19.321 DEBUG:boto:Token: None 2022-09-12T19:45:19.322 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/a 2022-09-12T19:45:19.323 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BQERg4w+Oy3QTcEchDtDT3otE1Y= 2022-09-12T19:45:19.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BQERg4w+Oy3QTcEchDtDT3otE1Y=', 'Content-Length': '0'} 2022-09-12T19:45:19.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a19745571e5ba69-00631f8c4f-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.326 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:19.327 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:19.327 DEBUG:boto:path=/jdpiak-72/?prefix=a 2022-09-12T19:45:19.328 DEBUG:boto:auth_path=/jdpiak-72/?prefix=a 2022-09-12T19:45:19.328 DEBUG:boto:Method: GET 2022-09-12T19:45:19.329 DEBUG:boto:Path: /jdpiak-72/?prefix=a 2022-09-12T19:45:19.329 DEBUG:boto:Data: 2022-09-12T19:45:19.330 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.331 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:19.331 DEBUG:boto:Port: 8001 2022-09-12T19:45:19.332 DEBUG:boto:Params: {} 2022-09-12T19:45:19.333 DEBUG:boto:Token: None 2022-09-12T19:45:19.333 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/ 2022-09-12T19:45:19.334 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OU4Q3EDnxTzpJA48xkI457y5F+E= 2022-09-12T19:45:19.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OU4Q3EDnxTzpJA48xkI457y5F+E=', 'Content-Length': '0'} 2022-09-12T19:45:19.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d9a12f8a63fa976-00631f8c4f-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.338 DEBUG:boto:b'jdpiak-72a1000falsea2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:19.340 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:19.340 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:19.341 DEBUG:boto:path=/jdpiak-72/?prefix=a 2022-09-12T19:45:19.341 DEBUG:boto:auth_path=/jdpiak-72/?prefix=a 2022-09-12T19:45:19.342 DEBUG:boto:Method: GET 2022-09-12T19:45:19.342 DEBUG:boto:Path: /jdpiak-72/?prefix=a 2022-09-12T19:45:19.343 DEBUG:boto:Data: 2022-09-12T19:45:19.343 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.344 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:19.345 DEBUG:boto:Port: 8000 2022-09-12T19:45:19.345 DEBUG:boto:Params: {} 2022-09-12T19:45:19.346 DEBUG:boto:Token: None 2022-09-12T19:45:19.350 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/ 2022-09-12T19:45:19.350 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OU4Q3EDnxTzpJA48xkI457y5F+E= 2022-09-12T19:45:19.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OU4Q3EDnxTzpJA48xkI457y5F+E=', 'Content-Length': '0'} 2022-09-12T19:45:19.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000040274136a1b8c40-00631f8c4f-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.360 DEBUG:boto:b'jdpiak-72a1000falsea2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:19.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:45:19.362 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:19.363 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:19.364 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:19.365 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:19.366 DEBUG:boto:Method: GET 2022-09-12T19:45:19.367 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:45:19.368 DEBUG:boto:Data: 2022-09-12T19:45:19.369 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.370 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:19.371 DEBUG:boto:Port: 8001 2022-09-12T19:45:19.372 DEBUG:boto:Params: {} 2022-09-12T19:45:19.373 DEBUG:boto:Token: None 2022-09-12T19:45:19.374 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:45:19.375 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KoT79yZ2EQGGlz9WNV0ft7CxTHs= 2022-09-12T19:45:19.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KoT79yZ2EQGGlz9WNV0ft7CxTHs=', 'Content-Length': '0'} 2022-09-12T19:45:19.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067f549691179a63a-00631f8c4f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.381 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:19.381 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:19.382 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:19.383 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:19.384 DEBUG:boto:Method: GET 2022-09-12T19:45:19.385 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:45:19.386 DEBUG:boto:Data: 2022-09-12T19:45:19.387 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.388 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:19.389 DEBUG:boto:Port: 8000 2022-09-12T19:45:19.389 DEBUG:boto:Params: {} 2022-09-12T19:45:19.390 DEBUG:boto:Token: None 2022-09-12T19:45:19.391 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:45:19.391 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KoT79yZ2EQGGlz9WNV0ft7CxTHs= 2022-09-12T19:45:19.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KoT79yZ2EQGGlz9WNV0ft7CxTHs=', 'Content-Length': '0'} 2022-09-12T19:45:19.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b638b17daf5e7de-00631f8c4f-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.395 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:19.396 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:19.396 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:19.397 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:19.397 DEBUG:boto:Method: HEAD 2022-09-12T19:45:19.398 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:45:19.399 DEBUG:boto:Data: 2022-09-12T19:45:19.399 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.400 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:19.400 DEBUG:boto:Port: 8001 2022-09-12T19:45:19.401 DEBUG:boto:Params: {} 2022-09-12T19:45:19.401 DEBUG:boto:Token: None 2022-09-12T19:45:19.402 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:45:19.403 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FsC0jqPoC7XMjnShCoSqQagB1uo= 2022-09-12T19:45:19.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FsC0jqPoC7XMjnShCoSqQagB1uo=', 'Content-Length': '0'} 2022-09-12T19:45:19.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e4079b43cecf98b-00631f8c4f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.407 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:19.408 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:19.409 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:19.410 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:19.411 DEBUG:boto:Method: HEAD 2022-09-12T19:45:19.412 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:45:19.413 DEBUG:boto:Data: 2022-09-12T19:45:19.414 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.415 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:19.416 DEBUG:boto:Port: 8000 2022-09-12T19:45:19.417 DEBUG:boto:Params: {} 2022-09-12T19:45:19.418 DEBUG:boto:Token: None 2022-09-12T19:45:19.419 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:45:19.420 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FsC0jqPoC7XMjnShCoSqQagB1uo= 2022-09-12T19:45:19.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FsC0jqPoC7XMjnShCoSqQagB1uo=', 'Content-Length': '0'} 2022-09-12T19:45:19.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015cb3fd1ce6940b5-00631f8c4f-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:45:19.428 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:19.429 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:19.430 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:19.431 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:19.432 DEBUG:boto:Method: GET 2022-09-12T19:45:19.433 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:45:19.434 DEBUG:boto:Data: 2022-09-12T19:45:19.435 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.436 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:19.437 DEBUG:boto:Port: 8001 2022-09-12T19:45:19.438 DEBUG:boto:Params: {} 2022-09-12T19:45:19.439 DEBUG:boto:Token: None 2022-09-12T19:45:19.440 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:45:19.441 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KoT79yZ2EQGGlz9WNV0ft7CxTHs= 2022-09-12T19:45:19.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KoT79yZ2EQGGlz9WNV0ft7CxTHs=', 'Content-Length': '0'} 2022-09-12T19:45:19.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000606b647ad378eff-00631f8c4f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.447 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:19.448 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:19.449 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:19.450 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:45:19.451 DEBUG:boto:Method: GET 2022-09-12T19:45:19.452 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:45:19.453 DEBUG:boto:Data: 2022-09-12T19:45:19.454 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.456 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:19.457 DEBUG:boto:Port: 8000 2022-09-12T19:45:19.457 DEBUG:boto:Params: {} 2022-09-12T19:45:19.458 DEBUG:boto:Token: None 2022-09-12T19:45:19.459 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:45:19.459 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KoT79yZ2EQGGlz9WNV0ft7CxTHs= 2022-09-12T19:45:19.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KoT79yZ2EQGGlz9WNV0ft7CxTHs=', 'Content-Length': '0'} 2022-09-12T19:45:19.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cabb99dc99ff124-00631f8c4f-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.510 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:19.511 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:19.512 DEBUG:boto:path=/jdpiak-72/?prefix=b 2022-09-12T19:45:19.513 DEBUG:boto:auth_path=/jdpiak-72/?prefix=b 2022-09-12T19:45:19.514 DEBUG:boto:Method: GET 2022-09-12T19:45:19.516 DEBUG:boto:Path: /jdpiak-72/?prefix=b 2022-09-12T19:45:19.517 DEBUG:boto:Data: 2022-09-12T19:45:19.518 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.519 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:19.521 DEBUG:boto:Port: 8001 2022-09-12T19:45:19.522 DEBUG:boto:Params: {} 2022-09-12T19:45:19.524 DEBUG:boto:Token: None 2022-09-12T19:45:19.525 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/ 2022-09-12T19:45:19.527 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OU4Q3EDnxTzpJA48xkI457y5F+E= 2022-09-12T19:45:19.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OU4Q3EDnxTzpJA48xkI457y5F+E=', 'Content-Length': '0'} 2022-09-12T19:45:19.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cca895c2ed6175c8-00631f8c4f-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.533 DEBUG:boto:b'jdpiak-72b1000falseb2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:19.534 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:19.535 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:19.537 DEBUG:boto:path=/jdpiak-72/?prefix=b 2022-09-12T19:45:19.537 DEBUG:boto:auth_path=/jdpiak-72/?prefix=b 2022-09-12T19:45:19.538 DEBUG:boto:Method: GET 2022-09-12T19:45:19.539 DEBUG:boto:Path: /jdpiak-72/?prefix=b 2022-09-12T19:45:19.539 DEBUG:boto:Data: 2022-09-12T19:45:19.540 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.540 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:19.541 DEBUG:boto:Port: 8000 2022-09-12T19:45:19.541 DEBUG:boto:Params: {} 2022-09-12T19:45:19.542 DEBUG:boto:Token: None 2022-09-12T19:45:19.543 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/ 2022-09-12T19:45:19.543 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OU4Q3EDnxTzpJA48xkI457y5F+E= 2022-09-12T19:45:19.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OU4Q3EDnxTzpJA48xkI457y5F+E=', 'Content-Length': '0'} 2022-09-12T19:45:19.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1d1b5fc16c2489c-00631f8c4f-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.549 DEBUG:boto:b'jdpiak-72b1000falseb2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:19.550 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:19.551 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:19.552 DEBUG:boto:Method: HEAD 2022-09-12T19:45:19.553 DEBUG:boto:Path: /jdpiak-72/b 2022-09-12T19:45:19.553 DEBUG:boto:Data: 2022-09-12T19:45:19.554 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.555 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:19.556 DEBUG:boto:Port: 8001 2022-09-12T19:45:19.556 DEBUG:boto:Params: {} 2022-09-12T19:45:19.557 DEBUG:boto:Token: None 2022-09-12T19:45:19.558 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/b 2022-09-12T19:45:19.559 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cMe4RAxdrEN7BYbfWTz5XwODCFA= 2022-09-12T19:45:19.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cMe4RAxdrEN7BYbfWTz5XwODCFA=', 'Content-Length': '0'} 2022-09-12T19:45:19.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4cf955c5d71f6d5-00631f8c4f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.563 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:19.563 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:19.564 DEBUG:boto:Method: HEAD 2022-09-12T19:45:19.564 DEBUG:boto:Path: /jdpiak-72/b 2022-09-12T19:45:19.565 DEBUG:boto:Data: 2022-09-12T19:45:19.566 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.567 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:19.569 DEBUG:boto:Port: 8000 2022-09-12T19:45:19.570 DEBUG:boto:Params: {} 2022-09-12T19:45:19.571 DEBUG:boto:Token: None 2022-09-12T19:45:19.572 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/b 2022-09-12T19:45:19.573 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cMe4RAxdrEN7BYbfWTz5XwODCFA= 2022-09-12T19:45:19.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cMe4RAxdrEN7BYbfWTz5XwODCFA=', 'Content-Length': '0'} 2022-09-12T19:45:19.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9edce232f3d30ca-00631f8c4f-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:45:19.580 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:19.581 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:19.582 DEBUG:boto:Method: GET 2022-09-12T19:45:19.583 DEBUG:boto:Path: /jdpiak-72/b 2022-09-12T19:45:19.584 DEBUG:boto:Data: 2022-09-12T19:45:19.584 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.585 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:19.585 DEBUG:boto:Port: 8001 2022-09-12T19:45:19.586 DEBUG:boto:Params: {} 2022-09-12T19:45:19.587 DEBUG:boto:Token: None 2022-09-12T19:45:19.588 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/b 2022-09-12T19:45:19.588 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tbMxoBHX4gBJfd19NGwzvNRVISQ= 2022-09-12T19:45:19.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tbMxoBHX4gBJfd19NGwzvNRVISQ=', 'Content-Length': '0'} 2022-09-12T19:45:19.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093cdb96b7f5bfeb9-00631f8c4f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.592 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:45:19.593 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:45:19.594 DEBUG:boto:Method: GET 2022-09-12T19:45:19.594 DEBUG:boto:Path: /jdpiak-72/b 2022-09-12T19:45:19.595 DEBUG:boto:Data: 2022-09-12T19:45:19.595 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.596 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:19.597 DEBUG:boto:Port: 8000 2022-09-12T19:45:19.598 DEBUG:boto:Params: {} 2022-09-12T19:45:19.601 DEBUG:boto:Token: None 2022-09-12T19:45:19.603 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/b 2022-09-12T19:45:19.604 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tbMxoBHX4gBJfd19NGwzvNRVISQ= 2022-09-12T19:45:19.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tbMxoBHX4gBJfd19NGwzvNRVISQ=', 'Content-Length': '0'} 2022-09-12T19:45:19.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e21a144eb415ab0-00631f8c4f-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.610 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:19.611 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:19.612 DEBUG:boto:path=/jdpiak-72/?prefix=b 2022-09-12T19:45:19.613 DEBUG:boto:auth_path=/jdpiak-72/?prefix=b 2022-09-12T19:45:19.614 DEBUG:boto:Method: GET 2022-09-12T19:45:19.615 DEBUG:boto:Path: /jdpiak-72/?prefix=b 2022-09-12T19:45:19.616 DEBUG:boto:Data: 2022-09-12T19:45:19.617 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.618 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:19.619 DEBUG:boto:Port: 8001 2022-09-12T19:45:19.620 DEBUG:boto:Params: {} 2022-09-12T19:45:19.621 DEBUG:boto:Token: None 2022-09-12T19:45:19.623 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/ 2022-09-12T19:45:19.624 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OU4Q3EDnxTzpJA48xkI457y5F+E= 2022-09-12T19:45:19.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OU4Q3EDnxTzpJA48xkI457y5F+E=', 'Content-Length': '0'} 2022-09-12T19:45:19.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000722a289a12f847dc-00631f8c4f-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.630 DEBUG:boto:b'jdpiak-72b1000falseb2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:19.631 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:19.632 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:19.633 DEBUG:boto:path=/jdpiak-72/?prefix=b 2022-09-12T19:45:19.634 DEBUG:boto:auth_path=/jdpiak-72/?prefix=b 2022-09-12T19:45:19.635 DEBUG:boto:Method: GET 2022-09-12T19:45:19.637 DEBUG:boto:Path: /jdpiak-72/?prefix=b 2022-09-12T19:45:19.638 DEBUG:boto:Data: 2022-09-12T19:45:19.640 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.641 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:19.641 DEBUG:boto:Port: 8000 2022-09-12T19:45:19.642 DEBUG:boto:Params: {} 2022-09-12T19:45:19.643 DEBUG:boto:Token: None 2022-09-12T19:45:19.644 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/ 2022-09-12T19:45:19.645 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OU4Q3EDnxTzpJA48xkI457y5F+E= 2022-09-12T19:45:19.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OU4Q3EDnxTzpJA48xkI457y5F+E=', 'Content-Length': '0'} 2022-09-12T19:45:19.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043bad3a0ae4d99f6-00631f8c4f-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.649 DEBUG:boto:b'jdpiak-72b1000falseb2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:19.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:45:19.650 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:19.651 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:19.652 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:19.653 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:19.654 DEBUG:boto:Method: GET 2022-09-12T19:45:19.655 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:45:19.656 DEBUG:boto:Data: 2022-09-12T19:45:19.657 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.659 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:19.659 DEBUG:boto:Port: 8001 2022-09-12T19:45:19.660 DEBUG:boto:Params: {} 2022-09-12T19:45:19.661 DEBUG:boto:Token: None 2022-09-12T19:45:19.663 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:45:19.664 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+BYSTi/NRP/D44yOfweALiI3XRQ= 2022-09-12T19:45:19.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+BYSTi/NRP/D44yOfweALiI3XRQ=', 'Content-Length': '0'} 2022-09-12T19:45:19.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c6e9c60d1ff35da-00631f8c4f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.668 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:19.669 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:19.669 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:19.670 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:19.671 DEBUG:boto:Method: GET 2022-09-12T19:45:19.671 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:45:19.672 DEBUG:boto:Data: 2022-09-12T19:45:19.672 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.673 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:19.673 DEBUG:boto:Port: 8000 2022-09-12T19:45:19.674 DEBUG:boto:Params: {} 2022-09-12T19:45:19.675 DEBUG:boto:Token: None 2022-09-12T19:45:19.676 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:45:19.677 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+BYSTi/NRP/D44yOfweALiI3XRQ= 2022-09-12T19:45:19.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+BYSTi/NRP/D44yOfweALiI3XRQ=', 'Content-Length': '0'} 2022-09-12T19:45:19.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030f48970c905432d-00631f8c4f-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.680 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:19.681 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:19.681 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:19.682 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:19.682 DEBUG:boto:Method: HEAD 2022-09-12T19:45:19.683 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:45:19.684 DEBUG:boto:Data: 2022-09-12T19:45:19.684 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.685 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:19.686 DEBUG:boto:Port: 8001 2022-09-12T19:45:19.686 DEBUG:boto:Params: {} 2022-09-12T19:45:19.687 DEBUG:boto:Token: None 2022-09-12T19:45:19.687 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:45:19.688 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h07olN4gW9zTA9wL6vTA8JRk/GI= 2022-09-12T19:45:19.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h07olN4gW9zTA9wL6vTA8JRk/GI=', 'Content-Length': '0'} 2022-09-12T19:45:19.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b19db5c70ba0b46-00631f8c4f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.692 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:19.693 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:19.694 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:19.695 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:19.695 DEBUG:boto:Method: HEAD 2022-09-12T19:45:19.696 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:45:19.697 DEBUG:boto:Data: 2022-09-12T19:45:19.697 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.698 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:19.698 DEBUG:boto:Port: 8000 2022-09-12T19:45:19.699 DEBUG:boto:Params: {} 2022-09-12T19:45:19.699 DEBUG:boto:Token: None 2022-09-12T19:45:19.700 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:45:19.701 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h07olN4gW9zTA9wL6vTA8JRk/GI= 2022-09-12T19:45:19.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h07olN4gW9zTA9wL6vTA8JRk/GI=', 'Content-Length': '0'} 2022-09-12T19:45:19.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093ec6ad1bd02b9fa-00631f8c4f-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:45:19.705 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:19.706 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:19.707 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:19.708 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:19.709 DEBUG:boto:Method: GET 2022-09-12T19:45:19.710 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:45:19.711 DEBUG:boto:Data: 2022-09-12T19:45:19.712 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.713 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:19.714 DEBUG:boto:Port: 8001 2022-09-12T19:45:19.715 DEBUG:boto:Params: {} 2022-09-12T19:45:19.716 DEBUG:boto:Token: None 2022-09-12T19:45:19.717 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:45:19.718 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+BYSTi/NRP/D44yOfweALiI3XRQ= 2022-09-12T19:45:19.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+BYSTi/NRP/D44yOfweALiI3XRQ=', 'Content-Length': '0'} 2022-09-12T19:45:19.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008062e4ccb7ad17d8-00631f8c4f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.767 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:19.768 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:19.769 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:19.770 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:45:19.771 DEBUG:boto:Method: GET 2022-09-12T19:45:19.772 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:45:19.773 DEBUG:boto:Data: 2022-09-12T19:45:19.775 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.776 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:19.777 DEBUG:boto:Port: 8000 2022-09-12T19:45:19.778 DEBUG:boto:Params: {} 2022-09-12T19:45:19.779 DEBUG:boto:Token: None 2022-09-12T19:45:19.780 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:45:19.781 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+BYSTi/NRP/D44yOfweALiI3XRQ= 2022-09-12T19:45:19.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+BYSTi/NRP/D44yOfweALiI3XRQ=', 'Content-Length': '0'} 2022-09-12T19:45:19.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000400f6fdbeab4f040-00631f8c4f-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.791 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:19.792 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:19.794 DEBUG:boto:path=/jdpiak-72/?prefix=c 2022-09-12T19:45:19.795 DEBUG:boto:auth_path=/jdpiak-72/?prefix=c 2022-09-12T19:45:19.795 DEBUG:boto:Method: GET 2022-09-12T19:45:19.796 DEBUG:boto:Path: /jdpiak-72/?prefix=c 2022-09-12T19:45:19.797 DEBUG:boto:Data: 2022-09-12T19:45:19.801 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.803 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:19.803 DEBUG:boto:Port: 8001 2022-09-12T19:45:19.804 DEBUG:boto:Params: {} 2022-09-12T19:45:19.804 DEBUG:boto:Token: None 2022-09-12T19:45:19.805 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/ 2022-09-12T19:45:19.806 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OU4Q3EDnxTzpJA48xkI457y5F+E= 2022-09-12T19:45:19.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OU4Q3EDnxTzpJA48xkI457y5F+E=', 'Content-Length': '0'} 2022-09-12T19:45:19.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd265bd5ee1870c7-00631f8c4f-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.812 DEBUG:boto:b'jdpiak-72c1000falsec2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:19.813 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:19.814 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:19.815 DEBUG:boto:path=/jdpiak-72/?prefix=c 2022-09-12T19:45:19.815 DEBUG:boto:auth_path=/jdpiak-72/?prefix=c 2022-09-12T19:45:19.816 DEBUG:boto:Method: GET 2022-09-12T19:45:19.816 DEBUG:boto:Path: /jdpiak-72/?prefix=c 2022-09-12T19:45:19.817 DEBUG:boto:Data: 2022-09-12T19:45:19.819 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.820 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:19.820 DEBUG:boto:Port: 8000 2022-09-12T19:45:19.821 DEBUG:boto:Params: {} 2022-09-12T19:45:19.822 DEBUG:boto:Token: None 2022-09-12T19:45:19.822 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/ 2022-09-12T19:45:19.823 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OU4Q3EDnxTzpJA48xkI457y5F+E= 2022-09-12T19:45:19.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OU4Q3EDnxTzpJA48xkI457y5F+E=', 'Content-Length': '0'} 2022-09-12T19:45:19.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8ba4d055b4730d3-00631f8c4f-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.870 DEBUG:boto:b'jdpiak-72c1000falsec2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:19.872 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:19.872 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:19.873 DEBUG:boto:Method: HEAD 2022-09-12T19:45:19.874 DEBUG:boto:Path: /jdpiak-72/c 2022-09-12T19:45:19.875 DEBUG:boto:Data: 2022-09-12T19:45:19.875 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.876 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:19.877 DEBUG:boto:Port: 8001 2022-09-12T19:45:19.877 DEBUG:boto:Params: {} 2022-09-12T19:45:19.878 DEBUG:boto:Token: None 2022-09-12T19:45:19.878 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/c 2022-09-12T19:45:19.879 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dwqza/ymlDCpY68kcCMww/vb4+g= 2022-09-12T19:45:19.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dwqza/ymlDCpY68kcCMww/vb4+g=', 'Content-Length': '0'} 2022-09-12T19:45:19.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bafab4ee572567a-00631f8c4f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.884 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:19.885 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:19.885 DEBUG:boto:Method: HEAD 2022-09-12T19:45:19.886 DEBUG:boto:Path: /jdpiak-72/c 2022-09-12T19:45:19.886 DEBUG:boto:Data: 2022-09-12T19:45:19.887 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.887 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:19.888 DEBUG:boto:Port: 8000 2022-09-12T19:45:19.888 DEBUG:boto:Params: {} 2022-09-12T19:45:19.889 DEBUG:boto:Token: None 2022-09-12T19:45:19.890 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/c 2022-09-12T19:45:19.891 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dwqza/ymlDCpY68kcCMww/vb4+g= 2022-09-12T19:45:19.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dwqza/ymlDCpY68kcCMww/vb4+g=', 'Content-Length': '0'} 2022-09-12T19:45:19.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2dfdbeba242cf6c-00631f8c4f-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:45:19.896 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:19.897 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:19.897 DEBUG:boto:Method: GET 2022-09-12T19:45:19.898 DEBUG:boto:Path: /jdpiak-72/c 2022-09-12T19:45:19.899 DEBUG:boto:Data: 2022-09-12T19:45:19.900 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.901 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:19.901 DEBUG:boto:Port: 8001 2022-09-12T19:45:19.902 DEBUG:boto:Params: {} 2022-09-12T19:45:19.903 DEBUG:boto:Token: None 2022-09-12T19:45:19.904 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/c 2022-09-12T19:45:19.904 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mPeLeT4b6C+ikKsgrNqmdVEui+k= 2022-09-12T19:45:19.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mPeLeT4b6C+ikKsgrNqmdVEui+k=', 'Content-Length': '0'} 2022-09-12T19:45:19.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a99d18846b18305-00631f8c4f-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.908 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:45:19.909 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:45:19.909 DEBUG:boto:Method: GET 2022-09-12T19:45:19.910 DEBUG:boto:Path: /jdpiak-72/c 2022-09-12T19:45:19.911 DEBUG:boto:Data: 2022-09-12T19:45:19.912 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.913 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:19.913 DEBUG:boto:Port: 8000 2022-09-12T19:45:19.914 DEBUG:boto:Params: {} 2022-09-12T19:45:19.914 DEBUG:boto:Token: None 2022-09-12T19:45:19.915 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/c 2022-09-12T19:45:19.916 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mPeLeT4b6C+ikKsgrNqmdVEui+k= 2022-09-12T19:45:19.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mPeLeT4b6C+ikKsgrNqmdVEui+k=', 'Content-Length': '0'} 2022-09-12T19:45:19.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000132258689b2b5cfb-00631f8c4f-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.920 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:19.921 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:19.921 DEBUG:boto:path=/jdpiak-72/?prefix=c 2022-09-12T19:45:19.922 DEBUG:boto:auth_path=/jdpiak-72/?prefix=c 2022-09-12T19:45:19.923 DEBUG:boto:Method: GET 2022-09-12T19:45:19.924 DEBUG:boto:Path: /jdpiak-72/?prefix=c 2022-09-12T19:45:19.925 DEBUG:boto:Data: 2022-09-12T19:45:19.925 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.926 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:19.926 DEBUG:boto:Port: 8001 2022-09-12T19:45:19.927 DEBUG:boto:Params: {} 2022-09-12T19:45:19.928 DEBUG:boto:Token: None 2022-09-12T19:45:19.928 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/ 2022-09-12T19:45:19.929 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OU4Q3EDnxTzpJA48xkI457y5F+E= 2022-09-12T19:45:19.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OU4Q3EDnxTzpJA48xkI457y5F+E=', 'Content-Length': '0'} 2022-09-12T19:45:19.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee497363f74b4d76-00631f8c4f-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.934 DEBUG:boto:b'jdpiak-72c1000falsec2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:19.935 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:19.935 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:19.936 DEBUG:boto:path=/jdpiak-72/?prefix=c 2022-09-12T19:45:19.936 DEBUG:boto:auth_path=/jdpiak-72/?prefix=c 2022-09-12T19:45:19.937 DEBUG:boto:Method: GET 2022-09-12T19:45:19.938 DEBUG:boto:Path: /jdpiak-72/?prefix=c 2022-09-12T19:45:19.938 DEBUG:boto:Data: 2022-09-12T19:45:19.939 DEBUG:boto:Headers: {} 2022-09-12T19:45:19.940 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:19.940 DEBUG:boto:Port: 8000 2022-09-12T19:45:19.941 DEBUG:boto:Params: {} 2022-09-12T19:45:19.942 DEBUG:boto:Token: None 2022-09-12T19:45:19.942 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:19 GMT /jdpiak-72/ 2022-09-12T19:45:19.943 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OU4Q3EDnxTzpJA48xkI457y5F+E= 2022-09-12T19:45:19.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OU4Q3EDnxTzpJA48xkI457y5F+E=', 'Content-Length': '0'} 2022-09-12T19:45:19.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bbc4d1a1fe399bb-00631f8c4f-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:19.994 DEBUG:boto:b'jdpiak-72c1000falsec2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:19.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:45:19.996 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:19.997 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:19.997 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:19.998 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:19.999 DEBUG:boto:Method: GET 2022-09-12T19:45:19.999 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:45:20.000 DEBUG:boto:Data: 2022-09-12T19:45:20.000 DEBUG:boto:Headers: {} 2022-09-12T19:45:20.001 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:20.001 DEBUG:boto:Port: 8001 2022-09-12T19:45:20.002 DEBUG:boto:Params: {} 2022-09-12T19:45:20.002 DEBUG:boto:Token: None 2022-09-12T19:45:20.003 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:20 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:45:20.004 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5UjkdylLW0FyxJ2tM4j+bxB+9/E= 2022-09-12T19:45:20.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5UjkdylLW0FyxJ2tM4j+bxB+9/E=', 'Content-Length': '0'} 2022-09-12T19:45:20.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abd5b372069bf266-00631f8c50-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:20.008 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:20.008 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:20.009 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:20.010 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:20.010 DEBUG:boto:Method: GET 2022-09-12T19:45:20.011 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:45:20.011 DEBUG:boto:Data: 2022-09-12T19:45:20.012 DEBUG:boto:Headers: {} 2022-09-12T19:45:20.013 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:20.013 DEBUG:boto:Port: 8000 2022-09-12T19:45:20.014 DEBUG:boto:Params: {} 2022-09-12T19:45:20.014 DEBUG:boto:Token: None 2022-09-12T19:45:20.015 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:20 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:45:20.016 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5UjkdylLW0FyxJ2tM4j+bxB+9/E= 2022-09-12T19:45:20.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5UjkdylLW0FyxJ2tM4j+bxB+9/E=', 'Content-Length': '0'} 2022-09-12T19:45:20.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a58fae3261c5388-00631f8c50-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:20.020 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:20.020 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:20.021 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:20.021 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:20.022 DEBUG:boto:Method: HEAD 2022-09-12T19:45:20.022 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:45:20.023 DEBUG:boto:Data: 2022-09-12T19:45:20.024 DEBUG:boto:Headers: {} 2022-09-12T19:45:20.024 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:20.025 DEBUG:boto:Port: 8001 2022-09-12T19:45:20.025 DEBUG:boto:Params: {} 2022-09-12T19:45:20.026 DEBUG:boto:Token: None 2022-09-12T19:45:20.027 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:20 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:45:20.027 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Wq5iEZIKhXsZ2iAXMo1Gz8R77iM= 2022-09-12T19:45:20.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Wq5iEZIKhXsZ2iAXMo1Gz8R77iM=', 'Content-Length': '0'} 2022-09-12T19:45:20.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087fdd5feb3e36e19-00631f8c50-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:20.031 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:20.031 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:20.032 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:20.032 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:20.033 DEBUG:boto:Method: HEAD 2022-09-12T19:45:20.033 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:45:20.034 DEBUG:boto:Data: 2022-09-12T19:45:20.034 DEBUG:boto:Headers: {} 2022-09-12T19:45:20.035 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:20.036 DEBUG:boto:Port: 8000 2022-09-12T19:45:20.036 DEBUG:boto:Params: {} 2022-09-12T19:45:20.037 DEBUG:boto:Token: None 2022-09-12T19:45:20.038 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:20 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:45:20.039 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Wq5iEZIKhXsZ2iAXMo1Gz8R77iM= 2022-09-12T19:45:20.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Wq5iEZIKhXsZ2iAXMo1Gz8R77iM=', 'Content-Length': '0'} 2022-09-12T19:45:20.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000656ee5079142baf4-00631f8c50-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:20.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:45:20.043 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:20.043 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:20.044 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:20.044 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:20.045 DEBUG:boto:Method: GET 2022-09-12T19:45:20.045 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:45:20.046 DEBUG:boto:Data: 2022-09-12T19:45:20.047 DEBUG:boto:Headers: {} 2022-09-12T19:45:20.047 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:20.048 DEBUG:boto:Port: 8001 2022-09-12T19:45:20.048 DEBUG:boto:Params: {} 2022-09-12T19:45:20.049 DEBUG:boto:Token: None 2022-09-12T19:45:20.050 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:20 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:45:20.050 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5UjkdylLW0FyxJ2tM4j+bxB+9/E= 2022-09-12T19:45:20.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5UjkdylLW0FyxJ2tM4j+bxB+9/E=', 'Content-Length': '0'} 2022-09-12T19:45:20.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e388f9c0dd8b484-00631f8c50-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:20.054 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:20.055 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:20.055 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:20.056 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:45:20.056 DEBUG:boto:Method: GET 2022-09-12T19:45:20.057 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:45:20.057 DEBUG:boto:Data: 2022-09-12T19:45:20.058 DEBUG:boto:Headers: {} 2022-09-12T19:45:20.059 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:20.059 DEBUG:boto:Port: 8000 2022-09-12T19:45:20.060 DEBUG:boto:Params: {} 2022-09-12T19:45:20.061 DEBUG:boto:Token: None 2022-09-12T19:45:20.062 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:20 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:45:20.063 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5UjkdylLW0FyxJ2tM4j+bxB+9/E= 2022-09-12T19:45:20.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5UjkdylLW0FyxJ2tM4j+bxB+9/E=', 'Content-Length': '0'} 2022-09-12T19:45:20.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000253e29ce935d6ab0-00631f8c50-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:20.069 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:20.070 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:20.071 DEBUG:boto:path=/jdpiak-72/?prefix=d 2022-09-12T19:45:20.072 DEBUG:boto:auth_path=/jdpiak-72/?prefix=d 2022-09-12T19:45:20.073 DEBUG:boto:Method: GET 2022-09-12T19:45:20.074 DEBUG:boto:Path: /jdpiak-72/?prefix=d 2022-09-12T19:45:20.075 DEBUG:boto:Data: 2022-09-12T19:45:20.076 DEBUG:boto:Headers: {} 2022-09-12T19:45:20.077 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:20.078 DEBUG:boto:Port: 8001 2022-09-12T19:45:20.079 DEBUG:boto:Params: {} 2022-09-12T19:45:20.080 DEBUG:boto:Token: None 2022-09-12T19:45:20.081 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:20 GMT /jdpiak-72/ 2022-09-12T19:45:20.082 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xlAWUA9vZzX0S/oTs6/PcbXORz4= 2022-09-12T19:45:20.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xlAWUA9vZzX0S/oTs6/PcbXORz4=', 'Content-Length': '0'} 2022-09-12T19:45:20.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef047a28e1d21228-00631f8c50-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:20.086 DEBUG:boto:b'jdpiak-72d1000falsed2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:20.088 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:20.089 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:20.090 DEBUG:boto:path=/jdpiak-72/?prefix=d 2022-09-12T19:45:20.091 DEBUG:boto:auth_path=/jdpiak-72/?prefix=d 2022-09-12T19:45:20.092 DEBUG:boto:Method: GET 2022-09-12T19:45:20.093 DEBUG:boto:Path: /jdpiak-72/?prefix=d 2022-09-12T19:45:20.094 DEBUG:boto:Data: 2022-09-12T19:45:20.095 DEBUG:boto:Headers: {} 2022-09-12T19:45:20.096 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:20.097 DEBUG:boto:Port: 8000 2022-09-12T19:45:20.098 DEBUG:boto:Params: {} 2022-09-12T19:45:20.099 DEBUG:boto:Token: None 2022-09-12T19:45:20.100 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:20 GMT /jdpiak-72/ 2022-09-12T19:45:20.101 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xlAWUA9vZzX0S/oTs6/PcbXORz4= 2022-09-12T19:45:20.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xlAWUA9vZzX0S/oTs6/PcbXORz4=', 'Content-Length': '0'} 2022-09-12T19:45:20.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ef294de544eae46-00631f8c50-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:20.149 DEBUG:boto:b'jdpiak-72d1000falsed2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:20.151 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:20.152 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:20.153 DEBUG:boto:Method: HEAD 2022-09-12T19:45:20.155 DEBUG:boto:Path: /jdpiak-72/d 2022-09-12T19:45:20.156 DEBUG:boto:Data: 2022-09-12T19:45:20.157 DEBUG:boto:Headers: {} 2022-09-12T19:45:20.159 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:20.160 DEBUG:boto:Port: 8001 2022-09-12T19:45:20.161 DEBUG:boto:Params: {} 2022-09-12T19:45:20.162 DEBUG:boto:Token: None 2022-09-12T19:45:20.163 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:20 GMT /jdpiak-72/d 2022-09-12T19:45:20.164 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QA/Pn0IBFG9PBORdzaQ/DvOvFrc= 2022-09-12T19:45:20.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QA/Pn0IBFG9PBORdzaQ/DvOvFrc=', 'Content-Length': '0'} 2022-09-12T19:45:20.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052a26bed802f1682-00631f8c50-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:20.169 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:20.170 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:20.171 DEBUG:boto:Method: HEAD 2022-09-12T19:45:20.172 DEBUG:boto:Path: /jdpiak-72/d 2022-09-12T19:45:20.173 DEBUG:boto:Data: 2022-09-12T19:45:20.174 DEBUG:boto:Headers: {} 2022-09-12T19:45:20.175 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:20.176 DEBUG:boto:Port: 8000 2022-09-12T19:45:20.177 DEBUG:boto:Params: {} 2022-09-12T19:45:20.178 DEBUG:boto:Token: None 2022-09-12T19:45:20.179 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:20 GMT /jdpiak-72/d 2022-09-12T19:45:20.181 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QA/Pn0IBFG9PBORdzaQ/DvOvFrc= 2022-09-12T19:45:20.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QA/Pn0IBFG9PBORdzaQ/DvOvFrc=', 'Content-Length': '0'} 2022-09-12T19:45:20.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000997a42b46d7c0ea5-00631f8c50-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:20.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:45:20.187 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:20.188 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:20.189 DEBUG:boto:Method: GET 2022-09-12T19:45:20.190 DEBUG:boto:Path: /jdpiak-72/d 2022-09-12T19:45:20.191 DEBUG:boto:Data: 2022-09-12T19:45:20.192 DEBUG:boto:Headers: {} 2022-09-12T19:45:20.193 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:20.194 DEBUG:boto:Port: 8001 2022-09-12T19:45:20.195 DEBUG:boto:Params: {} 2022-09-12T19:45:20.196 DEBUG:boto:Token: None 2022-09-12T19:45:20.197 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:20 GMT /jdpiak-72/d 2022-09-12T19:45:20.199 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WmpV4IhuyCHhnsA1X49jngmKquk= 2022-09-12T19:45:20.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WmpV4IhuyCHhnsA1X49jngmKquk=', 'Content-Length': '0'} 2022-09-12T19:45:20.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010df7d683dc70caa-00631f8c50-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:20.204 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:45:20.205 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:45:20.206 DEBUG:boto:Method: GET 2022-09-12T19:45:20.207 DEBUG:boto:Path: /jdpiak-72/d 2022-09-12T19:45:20.208 DEBUG:boto:Data: 2022-09-12T19:45:20.209 DEBUG:boto:Headers: {} 2022-09-12T19:45:20.210 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:20.211 DEBUG:boto:Port: 8000 2022-09-12T19:45:20.212 DEBUG:boto:Params: {} 2022-09-12T19:45:20.213 DEBUG:boto:Token: None 2022-09-12T19:45:20.214 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:20 GMT /jdpiak-72/d 2022-09-12T19:45:20.214 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WmpV4IhuyCHhnsA1X49jngmKquk= 2022-09-12T19:45:20.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WmpV4IhuyCHhnsA1X49jngmKquk=', 'Content-Length': '0'} 2022-09-12T19:45:20.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000906ccdd1ecc85463-00631f8c50-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:20.218 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:20.219 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:20.219 DEBUG:boto:path=/jdpiak-72/?prefix=d 2022-09-12T19:45:20.220 DEBUG:boto:auth_path=/jdpiak-72/?prefix=d 2022-09-12T19:45:20.220 DEBUG:boto:Method: GET 2022-09-12T19:45:20.221 DEBUG:boto:Path: /jdpiak-72/?prefix=d 2022-09-12T19:45:20.221 DEBUG:boto:Data: 2022-09-12T19:45:20.222 DEBUG:boto:Headers: {} 2022-09-12T19:45:20.223 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:45:20.223 DEBUG:boto:Port: 8001 2022-09-12T19:45:20.224 DEBUG:boto:Params: {} 2022-09-12T19:45:20.224 DEBUG:boto:Token: None 2022-09-12T19:45:20.225 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:20 GMT /jdpiak-72/ 2022-09-12T19:45:20.226 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xlAWUA9vZzX0S/oTs6/PcbXORz4= 2022-09-12T19:45:20.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xlAWUA9vZzX0S/oTs6/PcbXORz4=', 'Content-Length': '0'} 2022-09-12T19:45:20.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047cdd49395483d62-00631f8c50-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:20.274 DEBUG:boto:b'jdpiak-72d1000falsed2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:20.275 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:20.276 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:20.277 DEBUG:boto:path=/jdpiak-72/?prefix=d 2022-09-12T19:45:20.278 DEBUG:boto:auth_path=/jdpiak-72/?prefix=d 2022-09-12T19:45:20.279 DEBUG:boto:Method: GET 2022-09-12T19:45:20.280 DEBUG:boto:Path: /jdpiak-72/?prefix=d 2022-09-12T19:45:20.281 DEBUG:boto:Data: 2022-09-12T19:45:20.282 DEBUG:boto:Headers: {} 2022-09-12T19:45:20.282 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:45:20.283 DEBUG:boto:Port: 8000 2022-09-12T19:45:20.284 DEBUG:boto:Params: {} 2022-09-12T19:45:20.284 DEBUG:boto:Token: None 2022-09-12T19:45:20.285 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:45:20 GMT /jdpiak-72/ 2022-09-12T19:45:20.286 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xlAWUA9vZzX0S/oTs6/PcbXORz4= 2022-09-12T19:45:20.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xlAWUA9vZzX0S/oTs6/PcbXORz4=', 'Content-Length': '0'} 2022-09-12T19:45:20.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009802235288b3d4cf-00631f8c50-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:20.289 DEBUG:boto:b'jdpiak-72d1000falsed2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:45:20.290 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-72 zones={test-zone3, test-zone2} 2022-09-12T19:45:20.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'jdpiak-72', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:45:20.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'jdpiak-72', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:45:20.292 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket jdpiak-72 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:45:20.355 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:45:20.356 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:45:20.404 INFO:teuthology.orchestra.run.smithi035.stdout:tenant: 2022-09-12T19:45:20.404 INFO:teuthology.orchestra.run.smithi035.stdout:bucket name: jdpiak-72 2022-09-12T19:45:20.409 INFO:teuthology.orchestra.run.smithi035.stdout:total entries: 4 2022-09-12T19:45:20.459 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'reshard', '--bucket', 'jdpiak-72', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:45:20.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'reshard', '--bucket', 'jdpiak-72', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:45:20.461 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket reshard --bucket jdpiak-72 --num-shards 3 --yes-i-really-mean-it --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T19:45:20.521 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:45:20.521 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:45:20.580 INFO:teuthology.orchestra.run.smithi138.stdout:tenant: 2022-09-12T19:45:20.581 INFO:teuthology.orchestra.run.smithi138.stdout:bucket name: jdpiak-72 2022-09-12T19:45:20.585 INFO:teuthology.orchestra.run.smithi138.stdout:total entries: 4 2022-09-12T19:45:20.641 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'jdpiak-72', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:45:20.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'jdpiak-72', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:45:20.643 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket jdpiak-72 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:45:20.701 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:45:20.701 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:45:20.771 INFO:teuthology.orchestra.run.smithi035.stdout:tenant: 2022-09-12T19:45:20.772 INFO:teuthology.orchestra.run.smithi035.stdout:bucket name: jdpiak-72 2022-09-12T19:45:20.773 INFO:teuthology.orchestra.run.smithi035.stdout:total entries: 4 2022-09-12T19:45:20.800 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:20.801 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:20.802 DEBUG:boto:Method: HEAD 2022-09-12T19:45:20.802 DEBUG:boto:Path: /jdpiak-72/ 2022-09-12T19:45:20.803 DEBUG:boto:Data: 2022-09-12T19:45:20.803 DEBUG:boto:Headers: {} 2022-09-12T19:45:20.804 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:20.804 DEBUG:boto:Port: 8000 2022-09-12T19:45:20.805 DEBUG:boto:Params: {} 2022-09-12T19:45:20.806 DEBUG:boto:Token: None 2022-09-12T19:45:20.807 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:20 GMT /jdpiak-72/ 2022-09-12T19:45:20.808 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hExb21Hyhdlje5PTzz6MOv2+1xU= 2022-09-12T19:45:20.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hExb21Hyhdlje5PTzz6MOv2+1xU=', 'Content-Length': '0'} 2022-09-12T19:45:20.811 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072314c3dbc45814c-00631f8c50-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:20.812 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:45:20.813 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:45:20.813 DEBUG:boto:Method: PUT 2022-09-12T19:45:20.814 DEBUG:boto:Path: /jdpiak-72/e 2022-09-12T19:45:20.814 DEBUG:boto:Data: 2022-09-12T19:45:20.815 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:45:20.815 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:20.816 DEBUG:boto:Port: 8000 2022-09-12T19:45:20.817 DEBUG:boto:Params: {} 2022-09-12T19:45:20.818 DEBUG:boto:Token: None 2022-09-12T19:45:20.820 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:45:20 GMT /jdpiak-72/e 2022-09-12T19:45:20.821 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8yuuOgy0R6bt40qLuHY9Q3+v+jM= 2022-09-12T19:45:20.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:45:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8yuuOgy0R6bt40qLuHY9Q3+v+jM='} 2022-09-12T19:45:20.866 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001d386e9152e4ca2e-00631f8c50-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:20.868 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:20.869 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:20.870 DEBUG:boto:Method: HEAD 2022-09-12T19:45:20.871 DEBUG:boto:Path: /jdpiak-72/ 2022-09-12T19:45:20.872 DEBUG:boto:Data: 2022-09-12T19:45:20.872 DEBUG:boto:Headers: {} 2022-09-12T19:45:20.873 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:20.874 DEBUG:boto:Port: 8000 2022-09-12T19:45:20.875 DEBUG:boto:Params: {} 2022-09-12T19:45:20.875 DEBUG:boto:Token: None 2022-09-12T19:45:20.876 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:20 GMT /jdpiak-72/ 2022-09-12T19:45:20.876 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hExb21Hyhdlje5PTzz6MOv2+1xU= 2022-09-12T19:45:20.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hExb21Hyhdlje5PTzz6MOv2+1xU=', 'Content-Length': '0'} 2022-09-12T19:45:20.880 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000be2c451fa03038b-00631f8c50-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:20.880 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:45:20.881 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:45:20.882 DEBUG:boto:Method: PUT 2022-09-12T19:45:20.883 DEBUG:boto:Path: /jdpiak-72/f 2022-09-12T19:45:20.884 DEBUG:boto:Data: 2022-09-12T19:45:20.885 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:45:20.885 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:20.886 DEBUG:boto:Port: 8000 2022-09-12T19:45:20.887 DEBUG:boto:Params: {} 2022-09-12T19:45:20.887 DEBUG:boto:Token: None 2022-09-12T19:45:20.888 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:45:20 GMT /jdpiak-72/f 2022-09-12T19:45:20.889 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Lw1J/KiIqb7lR2jSC8gEu4CSQx0= 2022-09-12T19:45:20.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:45:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Lw1J/KiIqb7lR2jSC8gEu4CSQx0='} 2022-09-12T19:45:20.896 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007caec377f88cb0d2-00631f8c50-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:20.896 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:20.897 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:20.898 DEBUG:boto:Method: HEAD 2022-09-12T19:45:20.898 DEBUG:boto:Path: /jdpiak-72/ 2022-09-12T19:45:20.899 DEBUG:boto:Data: 2022-09-12T19:45:20.899 DEBUG:boto:Headers: {} 2022-09-12T19:45:20.900 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:20.900 DEBUG:boto:Port: 8000 2022-09-12T19:45:20.901 DEBUG:boto:Params: {} 2022-09-12T19:45:20.901 DEBUG:boto:Token: None 2022-09-12T19:45:20.902 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:20 GMT /jdpiak-72/ 2022-09-12T19:45:20.903 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hExb21Hyhdlje5PTzz6MOv2+1xU= 2022-09-12T19:45:20.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hExb21Hyhdlje5PTzz6MOv2+1xU=', 'Content-Length': '0'} 2022-09-12T19:45:20.907 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001df65b73ba032bea-00631f8c50-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:20.908 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:45:20.909 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:45:20.909 DEBUG:boto:Method: PUT 2022-09-12T19:45:20.910 DEBUG:boto:Path: /jdpiak-72/g 2022-09-12T19:45:20.911 DEBUG:boto:Data: 2022-09-12T19:45:20.912 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:45:20.913 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:20.914 DEBUG:boto:Port: 8000 2022-09-12T19:45:20.914 DEBUG:boto:Params: {} 2022-09-12T19:45:20.915 DEBUG:boto:Token: None 2022-09-12T19:45:20.916 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:45:20 GMT /jdpiak-72/g 2022-09-12T19:45:20.917 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kbxdan/MSTAzlSRY+3Ggv0QBKHo= 2022-09-12T19:45:20.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:45:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kbxdan/MSTAzlSRY+3Ggv0QBKHo='} 2022-09-12T19:45:20.966 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db712a6fd77105f4-00631f8c50-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:20.967 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:45:20.968 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:45:20.968 DEBUG:boto:Method: HEAD 2022-09-12T19:45:20.969 DEBUG:boto:Path: /jdpiak-72/ 2022-09-12T19:45:20.969 DEBUG:boto:Data: 2022-09-12T19:45:20.970 DEBUG:boto:Headers: {} 2022-09-12T19:45:20.971 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:20.971 DEBUG:boto:Port: 8000 2022-09-12T19:45:20.972 DEBUG:boto:Params: {} 2022-09-12T19:45:20.972 DEBUG:boto:Token: None 2022-09-12T19:45:20.973 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:45:20 GMT /jdpiak-72/ 2022-09-12T19:45:20.973 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hExb21Hyhdlje5PTzz6MOv2+1xU= 2022-09-12T19:45:20.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:45:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hExb21Hyhdlje5PTzz6MOv2+1xU=', 'Content-Length': '0'} 2022-09-12T19:45:20.976 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5bac5359aea298f-00631f8c50-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:20.977 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:45:20.977 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:45:20.978 DEBUG:boto:Method: PUT 2022-09-12T19:45:20.979 DEBUG:boto:Path: /jdpiak-72/h 2022-09-12T19:45:20.979 DEBUG:boto:Data: 2022-09-12T19:45:20.980 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:45:20.980 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:45:20.981 DEBUG:boto:Port: 8000 2022-09-12T19:45:20.981 DEBUG:boto:Params: {} 2022-09-12T19:45:20.982 DEBUG:boto:Token: None 2022-09-12T19:45:20.983 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:45:20 GMT /jdpiak-72/h 2022-09-12T19:45:20.983 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VD9WtZV+regodVgAlvo5Fav4Qz8= 2022-09-12T19:45:20.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:45:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VD9WtZV+regodVgAlvo5Fav4Qz8='} 2022-09-12T19:45:20.990 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000085aefb672f3f879-00631f8c50-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:45:20.991 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-72 2022-09-12T19:45:20.992 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:45:20.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:45:20.994 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-72 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:45:21.052 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:45:21.053 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:45:21.089 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:45:21.087+0000 7f714bcc5600 1 waiting to reach latest gen 2, on 0.. 2022-09-12T19:45:26.089 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:45:26.087+0000 7f714bcc5600 1 waiting to reach latest gen 2, on 0.. 2022-09-12T19:45:31.090 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:45:31.087+0000 7f714bcc5600 1 waiting to reach latest gen 2, on 0.. 2022-09-12T19:45:36.093 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:45:36.088+0000 7f714bcc5600 1 waiting to reach latest gen 2, on 1.. 2022-09-12T19:45:41.091 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:45:41.089+0000 7f714bcc5600 1 waiting to reach latest gen 2, on 1.. 2022-09-12T19:45:46.092 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:45:46.090+0000 7f714bcc5600 1 waiting to reach latest gen 2, on 1.. 2022-09-12T19:45:51.099 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:45:51.090+0000 7f714bcc5600 1 waiting to reach latest gen 2, on 1.. 2022-09-12T19:45:56.094 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:45:56.091+0000 7f714bcc5600 1 waiting to reach latest gen 2, on 1.. 2022-09-12T19:46:01.094 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:46:01.092+0000 7f714bcc5600 1 waiting to reach latest gen 2, on 1.. 2022-09-12T19:46:06.095 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:46:06.093+0000 7f714bcc5600 1 waiting to reach latest gen 2, on 1.. 2022-09-12T19:46:11.095 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:46:11.093+0000 7f714bcc5600 1 waiting to reach latest gen 2, on 1.. 2022-09-12T19:46:16.097 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:46:16.094+0000 7f714bcc5600 1 waiting for incremental sync to catch up: 2022-09-12T19:46:16.098 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , ] 2022-09-12T19:46:16.098 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:46:21.098 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:46:21.095+0000 7f714bcc5600 1 waiting for incremental sync to catch up: 2022-09-12T19:46:21.098 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , ] 2022-09-12T19:46:21.098 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:46:26.100 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:46:26.096+0000 7f714bcc5600 1 waiting for incremental sync to catch up: 2022-09-12T19:46:26.101 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , ] 2022-09-12T19:46:26.101 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:46:31.099 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:46:31.097+0000 7f714bcc5600 1 waiting for incremental sync to catch up: 2022-09-12T19:46:31.100 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , ] 2022-09-12T19:46:31.101 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:46:36.100 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:46:36.098+0000 7f714bcc5600 1 waiting for incremental sync to catch up: 2022-09-12T19:46:36.101 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , ] 2022-09-12T19:46:36.102 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:46:41.101 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:46:41.098+0000 7f714bcc5600 1 waiting for incremental sync to catch up: 2022-09-12T19:46:41.102 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , ] 2022-09-12T19:46:41.103 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:46:46.102 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:46:46.099+0000 7f714bcc5600 1 waiting for incremental sync to catch up: 2022-09-12T19:46:46.103 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , ] 2022-09-12T19:46:46.103 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:46:51.103 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:46:51.101+0000 7f714bcc5600 1 waiting for incremental sync to catch up: 2022-09-12T19:46:51.103 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , ] 2022-09-12T19:46:51.103 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:46:56.104 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:46:56.101+0000 7f714bcc5600 1 waiting for incremental sync to catch up: 2022-09-12T19:46:56.104 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , ] 2022-09-12T19:46:56.105 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:47:01.105 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:47:01.102+0000 7f714bcc5600 1 waiting for incremental sync to catch up: 2022-09-12T19:47:01.105 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , ] 2022-09-12T19:47:01.106 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:47:06.105 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:47:06.103+0000 7f714bcc5600 1 waiting for incremental sync to catch up: 2022-09-12T19:47:06.106 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , ] 2022-09-12T19:47:06.106 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:47:11.106 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:47:11.104+0000 7f714bcc5600 1 waiting for incremental sync to catch up: 2022-09-12T19:47:11.107 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , ] 2022-09-12T19:47:11.107 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:47:16.107 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:47:16.105+0000 7f714bcc5600 1 waiting for incremental sync to catch up: 2022-09-12T19:47:16.108 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , ] 2022-09-12T19:47:16.108 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:47:21.108 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:47:21.105+0000 7f714bcc5600 1 waiting for incremental sync to catch up: 2022-09-12T19:47:21.109 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , ] 2022-09-12T19:47:21.109 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:47:26.108 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:47:26.106+0000 7f714bcc5600 1 waiting for incremental sync to catch up: 2022-09-12T19:47:26.109 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , ] 2022-09-12T19:47:26.109 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:47:31.109 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:47:31.107+0000 7f714bcc5600 1 waiting for incremental sync to catch up: 2022-09-12T19:47:31.110 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , ] 2022-09-12T19:47:31.110 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:47:36.110 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:47:36.109+0000 7f714bcc5600 1 bucket sync caught up with source: 2022-09-12T19:47:36.111 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:47:36.111 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:47:36.111 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:47:36.109+0000 7f714bcc5600 0 bucket checkpoint complete 2022-09-12T19:47:36.134 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-72 2022-09-12T19:47:36.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:47:36.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:47:36.138 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-72 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:47:36.205 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:47:36.205 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:47:36.244 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:47:36.242+0000 7f615d274600 1 waiting for incremental sync to catch up: 2022-09-12T19:47:36.245 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.85.5, ] 2022-09-12T19:47:36.245 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:47:41.245 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:47:41.243+0000 7f615d274600 1 waiting for incremental sync to catch up: 2022-09-12T19:47:41.246 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.85.5, ] 2022-09-12T19:47:41.246 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:47:46.246 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:47:46.244+0000 7f615d274600 1 waiting for incremental sync to catch up: 2022-09-12T19:47:46.247 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.85.5, ] 2022-09-12T19:47:46.247 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:47:51.247 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:47:51.246+0000 7f615d274600 1 waiting for incremental sync to catch up: 2022-09-12T19:47:51.248 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.85.5, ] 2022-09-12T19:47:51.248 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:47:56.248 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:47:56.247+0000 7f615d274600 1 bucket sync caught up with source: 2022-09-12T19:47:56.249 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:47:56.249 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:47:56.250 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:47:56.247+0000 7f615d274600 0 bucket checkpoint complete 2022-09-12T19:47:56.271 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-72 2022-09-12T19:47:56.272 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:47:56.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:47:56.273 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-72 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:47:56.337 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:47:56.338 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:47:56.396 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:47:56.393+0000 7f531f476600 1 bucket sync caught up with source: 2022-09-12T19:47:56.396 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:47:56.396 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.60.5, 00000000002.85.5, 00000000002.54.5] 2022-09-12T19:47:56.397 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:47:56.393+0000 7f531f476600 0 bucket checkpoint complete 2022-09-12T19:47:56.414 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-72 2022-09-12T19:47:56.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:47:56.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:47:56.416 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-72 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:47:56.481 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:47:56.481 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:47:56.525 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:47:56.524+0000 7fabc768d600 1 bucket sync caught up with source: 2022-09-12T19:47:56.526 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [00000000003.808.5, 00000000002.916.5, 00000000002.595.5] 2022-09-12T19:47:56.526 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [00000000003.808.5, 00000000002.916.5, 00000000002.595.5] 2022-09-12T19:47:56.527 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:47:56.524+0000 7fabc768d600 0 bucket checkpoint complete 2022-09-12T19:47:56.541 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-72 2022-09-12T19:47:56.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:47:56.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:47:56.543 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-72 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:47:56.601 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:47:56.602 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:47:56.638 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:47:56.637+0000 7f797a260600 1 bucket sync caught up with source: 2022-09-12T19:47:56.639 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [00000000003.808.5, 00000000002.916.5, 00000000002.595.5] 2022-09-12T19:47:56.639 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [00000000003.808.5, 00000000002.916.5, 00000000002.595.5] 2022-09-12T19:47:56.639 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:47:56.637+0000 7f797a260600 0 bucket checkpoint complete 2022-09-12T19:47:56.658 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-72 2022-09-12T19:47:56.659 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-72 2022-09-12T19:47:56.660 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:47:56.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:47:56.662 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-72 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:47:56.720 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:47:56.721 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:47:56.770 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:47:56.768+0000 7f23ffde3600 1 waiting for incremental sync to catch up: 2022-09-12T19:47:56.770 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.262.5, , , , 00000000001.281.5, , , 00000000002.447.5, , 00000000001.321.5] 2022-09-12T19:47:56.771 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.262.5, , , 00000000001.306.5, 00000000001.281.5, , , 00000000002.447.5, , 00000000002.329.5] 2022-09-12T19:48:01.772 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:48:01.770+0000 7f23ffde3600 1 waiting for incremental sync to catch up: 2022-09-12T19:48:01.773 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.262.5, , , , 00000000001.281.5, , , 00000000002.447.5, , 00000000001.321.5] 2022-09-12T19:48:01.773 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.262.5, , , 00000000001.306.5, 00000000001.281.5, , , 00000000002.447.5, , 00000000002.329.5] 2022-09-12T19:48:06.773 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:48:06.770+0000 7f23ffde3600 1 bucket sync caught up with source: 2022-09-12T19:48:06.774 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.262.5, , , 00000000001.306.5, 00000000001.281.5, , , 00000000002.447.5, , 00000000002.329.5] 2022-09-12T19:48:06.775 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.262.5, , , 00000000001.306.5, 00000000001.281.5, , , 00000000002.447.5, , 00000000002.329.5] 2022-09-12T19:48:06.775 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:48:06.770+0000 7f23ffde3600 0 bucket checkpoint complete 2022-09-12T19:48:06.795 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-72 2022-09-12T19:48:06.797 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:48:06.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-72', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:48:06.798 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-72 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:48:06.862 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:48:06.863 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:48:06.907 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:48:06.904+0000 7f96c6ba9600 1 bucket sync caught up with source: 2022-09-12T19:48:06.908 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.262.5, , , 00000000001.306.5, 00000000001.281.5, , , 00000000002.447.5, , 00000000002.329.5] 2022-09-12T19:48:06.909 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000002.262.5, , , 00000000001.306.5, 00000000001.281.5, , , 00000000002.447.5, , 00000000002.329.5] 2022-09-12T19:48:06.909 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:48:06.904+0000 7f96c6ba9600 0 bucket checkpoint complete 2022-09-12T19:48:06.928 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-72 2022-09-12T19:48:06.929 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-72 zones={test-zone2, test-zone1} 2022-09-12T19:48:06.930 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:06.930 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:06.931 DEBUG:boto:Method: HEAD 2022-09-12T19:48:06.932 DEBUG:boto:Path: /jdpiak-72/ 2022-09-12T19:48:06.935 DEBUG:boto:Data: 2022-09-12T19:48:06.935 DEBUG:boto:Headers: {} 2022-09-12T19:48:06.936 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:06.936 DEBUG:boto:Port: 8000 2022-09-12T19:48:06.937 DEBUG:boto:Params: {} 2022-09-12T19:48:06.938 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:48:06.939 DEBUG:boto:Token: None 2022-09-12T19:48:06.940 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:06 GMT /jdpiak-72/ 2022-09-12T19:48:06.940 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/YdAiFz7FJrjx4ZQDemiVNTN3qc= 2022-09-12T19:48:06.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/YdAiFz7FJrjx4ZQDemiVNTN3qc=', 'Content-Length': '0'} 2022-09-12T19:48:06.946 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c745842aacbd50ac-00631f8cf6-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:48:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:06.947 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:06.948 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:06.948 DEBUG:boto:Method: HEAD 2022-09-12T19:48:06.949 DEBUG:boto:Path: /jdpiak-72/ 2022-09-12T19:48:06.951 DEBUG:boto:Data: 2022-09-12T19:48:06.952 DEBUG:boto:Headers: {} 2022-09-12T19:48:06.953 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:06.955 DEBUG:boto:Port: 8000 2022-09-12T19:48:06.955 DEBUG:boto:Params: {} 2022-09-12T19:48:06.957 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:48:06.958 DEBUG:boto:Token: None 2022-09-12T19:48:06.959 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:06 GMT /jdpiak-72/ 2022-09-12T19:48:06.960 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/YdAiFz7FJrjx4ZQDemiVNTN3qc= 2022-09-12T19:48:06.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/YdAiFz7FJrjx4ZQDemiVNTN3qc=', 'Content-Length': '0'} 2022-09-12T19:48:06.965 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097067d947a25b0e1-00631f8cf6-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:48:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:06.966 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:48:06.970 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:06.971 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:06.972 DEBUG:boto:path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:06.973 DEBUG:boto:auth_path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:06.974 DEBUG:boto:Method: GET 2022-09-12T19:48:06.975 DEBUG:boto:Path: /jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:06.976 DEBUG:boto:Data: 2022-09-12T19:48:06.977 DEBUG:boto:Headers: {} 2022-09-12T19:48:06.978 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:06.979 DEBUG:boto:Port: 8000 2022-09-12T19:48:06.980 DEBUG:boto:Params: {} 2022-09-12T19:48:06.981 DEBUG:boto:Token: None 2022-09-12T19:48:06.982 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:06 GMT /jdpiak-72/?versions 2022-09-12T19:48:06.983 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gQdljSM+mv5WV83SaEglDiE7T+I= 2022-09-12T19:48:06.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gQdljSM+mv5WV83SaEglDiE7T+I=', 'Content-Length': '0'} 2022-09-12T19:48:06.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092ba430eba2a7162-00631f8cf6-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:06.988 DEBUG:boto:b'jdpiak-72999falseanulltrue2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:45:20.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:45:20.891Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:45:20.920Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:45:20.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:06.991 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:48:06.992 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:48:06.993 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:48:06.994 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:48:06.995 DEBUG:tasks.rgw_multi.zone_rados:o=e 2022-09-12T19:48:06.996 DEBUG:tasks.rgw_multi.zone_rados:o=f 2022-09-12T19:48:06.996 DEBUG:tasks.rgw_multi.zone_rados:o=g 2022-09-12T19:48:06.997 DEBUG:tasks.rgw_multi.zone_rados:o=h 2022-09-12T19:48:06.998 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:48:06.999 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:07.000 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:07.001 DEBUG:boto:path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:07.002 DEBUG:boto:auth_path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:07.003 DEBUG:boto:Method: GET 2022-09-12T19:48:07.003 DEBUG:boto:Path: /jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:07.004 DEBUG:boto:Data: 2022-09-12T19:48:07.005 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.005 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:07.006 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.007 DEBUG:boto:Params: {} 2022-09-12T19:48:07.008 DEBUG:boto:Token: None 2022-09-12T19:48:07.009 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/?versions 2022-09-12T19:48:07.010 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0R5kqRtjsEfM4P11gRA5m+EoULg= 2022-09-12T19:48:07.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0R5kqRtjsEfM4P11gRA5m+EoULg=', 'Content-Length': '0'} 2022-09-12T19:48:07.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb839bb9acbe515c-00631f8cf7-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.016 DEBUG:boto:b'jdpiak-72999falseanulltrue2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:45:20.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:45:20.891Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:45:20.920Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:45:20.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:07.018 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:48:07.018 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:48:07.019 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:48:07.019 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:48:07.020 DEBUG:tasks.rgw_multi.zone_rados:o=e 2022-09-12T19:48:07.020 DEBUG:tasks.rgw_multi.zone_rados:o=f 2022-09-12T19:48:07.021 DEBUG:tasks.rgw_multi.zone_rados:o=g 2022-09-12T19:48:07.021 DEBUG:tasks.rgw_multi.zone_rados:o=h 2022-09-12T19:48:07.022 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:07.022 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:07.023 DEBUG:boto:path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:07.024 DEBUG:boto:auth_path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:07.024 DEBUG:boto:Method: GET 2022-09-12T19:48:07.025 DEBUG:boto:Path: /jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:07.025 DEBUG:boto:Data: 2022-09-12T19:48:07.026 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.026 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:07.027 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.027 DEBUG:boto:Params: {} 2022-09-12T19:48:07.028 DEBUG:boto:Token: None 2022-09-12T19:48:07.029 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/?versions 2022-09-12T19:48:07.029 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0R5kqRtjsEfM4P11gRA5m+EoULg= 2022-09-12T19:48:07.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0R5kqRtjsEfM4P11gRA5m+EoULg=', 'Content-Length': '0'} 2022-09-12T19:48:07.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cd56eabcd54c36a-00631f8cf7-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.034 DEBUG:boto:b'jdpiak-72999falseanulltrue2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:45:20.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:45:20.891Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:45:20.920Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:45:20.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:07.035 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:07.036 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:07.036 DEBUG:boto:path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:07.037 DEBUG:boto:auth_path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:07.037 DEBUG:boto:Method: GET 2022-09-12T19:48:07.038 DEBUG:boto:Path: /jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:07.038 DEBUG:boto:Data: 2022-09-12T19:48:07.039 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.039 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:07.040 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.041 DEBUG:boto:Params: {} 2022-09-12T19:48:07.042 DEBUG:boto:Token: None 2022-09-12T19:48:07.043 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/?versions 2022-09-12T19:48:07.044 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0R5kqRtjsEfM4P11gRA5m+EoULg= 2022-09-12T19:48:07.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0R5kqRtjsEfM4P11gRA5m+EoULg=', 'Content-Length': '0'} 2022-09-12T19:48:07.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa5c0c3d58e1be36-00631f8cf7-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.049 DEBUG:boto:b'jdpiak-72999falseanulltrue2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:45:20.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:45:20.891Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:45:20.920Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:45:20.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:07.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:48:07.053 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:07.054 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:07.055 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:07.056 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:07.057 DEBUG:boto:Method: GET 2022-09-12T19:48:07.058 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:48:07.058 DEBUG:boto:Data: 2022-09-12T19:48:07.059 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.060 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:07.061 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.062 DEBUG:boto:Params: {} 2022-09-12T19:48:07.063 DEBUG:boto:Token: None 2022-09-12T19:48:07.064 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:48:07.069 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A50rIq5cH6gt2xcy4JG4q36bL04= 2022-09-12T19:48:07.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A50rIq5cH6gt2xcy4JG4q36bL04=', 'Content-Length': '0'} 2022-09-12T19:48:07.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c861d416fbd271ad-00631f8cf7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.076 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:07.076 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:07.077 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:07.079 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:07.080 DEBUG:boto:Method: GET 2022-09-12T19:48:07.081 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:48:07.082 DEBUG:boto:Data: 2022-09-12T19:48:07.083 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.084 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:07.084 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.085 DEBUG:boto:Params: {} 2022-09-12T19:48:07.086 DEBUG:boto:Token: None 2022-09-12T19:48:07.087 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:48:07.087 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A50rIq5cH6gt2xcy4JG4q36bL04= 2022-09-12T19:48:07.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A50rIq5cH6gt2xcy4JG4q36bL04=', 'Content-Length': '0'} 2022-09-12T19:48:07.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff0898ed9ec1d688-00631f8cf7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.092 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:07.093 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:07.094 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:07.095 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:07.096 DEBUG:boto:Method: HEAD 2022-09-12T19:48:07.097 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:48:07.098 DEBUG:boto:Data: 2022-09-12T19:48:07.099 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.100 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:07.101 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.101 DEBUG:boto:Params: {} 2022-09-12T19:48:07.102 DEBUG:boto:Token: None 2022-09-12T19:48:07.103 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:48:07.104 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1zhAdGsec/FCsQCc2wi/ANMkAzQ= 2022-09-12T19:48:07.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1zhAdGsec/FCsQCc2wi/ANMkAzQ=', 'Content-Length': '0'} 2022-09-12T19:48:07.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051a74c73367c4c97-00631f8cf7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.109 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:07.109 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:07.110 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:07.110 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:07.111 DEBUG:boto:Method: HEAD 2022-09-12T19:48:07.111 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:48:07.112 DEBUG:boto:Data: 2022-09-12T19:48:07.112 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.113 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:07.113 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.113 DEBUG:boto:Params: {} 2022-09-12T19:48:07.114 DEBUG:boto:Token: None 2022-09-12T19:48:07.115 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:48:07.115 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1zhAdGsec/FCsQCc2wi/ANMkAzQ= 2022-09-12T19:48:07.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1zhAdGsec/FCsQCc2wi/ANMkAzQ=', 'Content-Length': '0'} 2022-09-12T19:48:07.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097c6f7de305ce90b-00631f8cf7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:48:07.121 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:07.122 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:07.123 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:07.123 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:07.124 DEBUG:boto:Method: GET 2022-09-12T19:48:07.124 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:48:07.125 DEBUG:boto:Data: 2022-09-12T19:48:07.125 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.126 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:07.127 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.127 DEBUG:boto:Params: {} 2022-09-12T19:48:07.128 DEBUG:boto:Token: None 2022-09-12T19:48:07.128 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:48:07.129 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A50rIq5cH6gt2xcy4JG4q36bL04= 2022-09-12T19:48:07.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A50rIq5cH6gt2xcy4JG4q36bL04=', 'Content-Length': '0'} 2022-09-12T19:48:07.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da35e3d7a72ed4c1-00631f8cf7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.132 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:07.133 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:07.133 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:07.134 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:07.135 DEBUG:boto:Method: GET 2022-09-12T19:48:07.135 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:48:07.136 DEBUG:boto:Data: 2022-09-12T19:48:07.136 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.137 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:07.137 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.138 DEBUG:boto:Params: {} 2022-09-12T19:48:07.138 DEBUG:boto:Token: None 2022-09-12T19:48:07.139 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:48:07.140 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A50rIq5cH6gt2xcy4JG4q36bL04= 2022-09-12T19:48:07.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A50rIq5cH6gt2xcy4JG4q36bL04=', 'Content-Length': '0'} 2022-09-12T19:48:07.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0fbf78ef8873399-00631f8cf7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.191 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:07.192 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:07.193 DEBUG:boto:path=/jdpiak-72/?prefix=a 2022-09-12T19:48:07.194 DEBUG:boto:auth_path=/jdpiak-72/?prefix=a 2022-09-12T19:48:07.195 DEBUG:boto:Method: GET 2022-09-12T19:48:07.195 DEBUG:boto:Path: /jdpiak-72/?prefix=a 2022-09-12T19:48:07.196 DEBUG:boto:Data: 2022-09-12T19:48:07.197 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.197 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:07.198 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.198 DEBUG:boto:Params: {} 2022-09-12T19:48:07.199 DEBUG:boto:Token: None 2022-09-12T19:48:07.199 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/ 2022-09-12T19:48:07.200 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ToHM+D25Vadjv5gWWBdw4Swr/rE= 2022-09-12T19:48:07.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ToHM+D25Vadjv5gWWBdw4Swr/rE=', 'Content-Length': '0'} 2022-09-12T19:48:07.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066449f51ad5e7333-00631f8cf7-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.204 DEBUG:boto:b'jdpiak-72a1000falsea2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:07.206 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:07.207 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:07.208 DEBUG:boto:path=/jdpiak-72/?prefix=a 2022-09-12T19:48:07.209 DEBUG:boto:auth_path=/jdpiak-72/?prefix=a 2022-09-12T19:48:07.210 DEBUG:boto:Method: GET 2022-09-12T19:48:07.211 DEBUG:boto:Path: /jdpiak-72/?prefix=a 2022-09-12T19:48:07.212 DEBUG:boto:Data: 2022-09-12T19:48:07.213 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.214 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:07.215 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.216 DEBUG:boto:Params: {} 2022-09-12T19:48:07.217 DEBUG:boto:Token: None 2022-09-12T19:48:07.217 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/ 2022-09-12T19:48:07.218 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ToHM+D25Vadjv5gWWBdw4Swr/rE= 2022-09-12T19:48:07.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ToHM+D25Vadjv5gWWBdw4Swr/rE=', 'Content-Length': '0'} 2022-09-12T19:48:07.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003296612c9332938c-00631f8cf7-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.227 DEBUG:boto:b'jdpiak-72a1000falsea2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:07.229 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:07.230 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:07.231 DEBUG:boto:Method: HEAD 2022-09-12T19:48:07.231 DEBUG:boto:Path: /jdpiak-72/a 2022-09-12T19:48:07.232 DEBUG:boto:Data: 2022-09-12T19:48:07.233 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.234 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:07.235 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.242 DEBUG:boto:Params: {} 2022-09-12T19:48:07.243 DEBUG:boto:Token: None 2022-09-12T19:48:07.244 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/a 2022-09-12T19:48:07.244 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+AE3BjizEdILNKfHQOfqhSsp3AI= 2022-09-12T19:48:07.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+AE3BjizEdILNKfHQOfqhSsp3AI=', 'Content-Length': '0'} 2022-09-12T19:48:07.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4074e84c74d874c-00631f8cf7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.250 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:07.252 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:07.253 DEBUG:boto:Method: HEAD 2022-09-12T19:48:07.254 DEBUG:boto:Path: /jdpiak-72/a 2022-09-12T19:48:07.255 DEBUG:boto:Data: 2022-09-12T19:48:07.256 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.257 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:07.258 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.259 DEBUG:boto:Params: {} 2022-09-12T19:48:07.260 DEBUG:boto:Token: None 2022-09-12T19:48:07.261 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/a 2022-09-12T19:48:07.261 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+AE3BjizEdILNKfHQOfqhSsp3AI= 2022-09-12T19:48:07.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+AE3BjizEdILNKfHQOfqhSsp3AI=', 'Content-Length': '0'} 2022-09-12T19:48:07.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aca5da78c6bf8e9c-00631f8cf7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:48:07.266 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:07.267 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:07.268 DEBUG:boto:Method: GET 2022-09-12T19:48:07.269 DEBUG:boto:Path: /jdpiak-72/a 2022-09-12T19:48:07.270 DEBUG:boto:Data: 2022-09-12T19:48:07.271 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.272 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:07.273 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.275 DEBUG:boto:Params: {} 2022-09-12T19:48:07.275 DEBUG:boto:Token: None 2022-09-12T19:48:07.276 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/a 2022-09-12T19:48:07.277 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:I2WADhJk957XXfSuxY0RL6u1Ho4= 2022-09-12T19:48:07.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:I2WADhJk957XXfSuxY0RL6u1Ho4=', 'Content-Length': '0'} 2022-09-12T19:48:07.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087fc8c868aebc968-00631f8cf7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.281 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:07.281 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:07.282 DEBUG:boto:Method: GET 2022-09-12T19:48:07.283 DEBUG:boto:Path: /jdpiak-72/a 2022-09-12T19:48:07.283 DEBUG:boto:Data: 2022-09-12T19:48:07.284 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.284 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:07.285 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.286 DEBUG:boto:Params: {} 2022-09-12T19:48:07.286 DEBUG:boto:Token: None 2022-09-12T19:48:07.288 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/a 2022-09-12T19:48:07.289 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:I2WADhJk957XXfSuxY0RL6u1Ho4= 2022-09-12T19:48:07.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:I2WADhJk957XXfSuxY0RL6u1Ho4=', 'Content-Length': '0'} 2022-09-12T19:48:07.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb85afe19da95176-00631f8cf7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.293 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:07.294 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:07.294 DEBUG:boto:path=/jdpiak-72/?prefix=a 2022-09-12T19:48:07.295 DEBUG:boto:auth_path=/jdpiak-72/?prefix=a 2022-09-12T19:48:07.295 DEBUG:boto:Method: GET 2022-09-12T19:48:07.296 DEBUG:boto:Path: /jdpiak-72/?prefix=a 2022-09-12T19:48:07.297 DEBUG:boto:Data: 2022-09-12T19:48:07.298 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.299 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:07.299 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.300 DEBUG:boto:Params: {} 2022-09-12T19:48:07.301 DEBUG:boto:Token: None 2022-09-12T19:48:07.302 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/ 2022-09-12T19:48:07.303 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ToHM+D25Vadjv5gWWBdw4Swr/rE= 2022-09-12T19:48:07.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ToHM+D25Vadjv5gWWBdw4Swr/rE=', 'Content-Length': '0'} 2022-09-12T19:48:07.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fdda1c39fd97796-00631f8cf7-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.319 DEBUG:boto:b'jdpiak-72a1000falsea2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:07.320 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:07.321 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:07.322 DEBUG:boto:path=/jdpiak-72/?prefix=a 2022-09-12T19:48:07.323 DEBUG:boto:auth_path=/jdpiak-72/?prefix=a 2022-09-12T19:48:07.324 DEBUG:boto:Method: GET 2022-09-12T19:48:07.324 DEBUG:boto:Path: /jdpiak-72/?prefix=a 2022-09-12T19:48:07.325 DEBUG:boto:Data: 2022-09-12T19:48:07.326 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.326 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:07.327 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.328 DEBUG:boto:Params: {} 2022-09-12T19:48:07.328 DEBUG:boto:Token: None 2022-09-12T19:48:07.329 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/ 2022-09-12T19:48:07.330 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ToHM+D25Vadjv5gWWBdw4Swr/rE= 2022-09-12T19:48:07.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ToHM+D25Vadjv5gWWBdw4Swr/rE=', 'Content-Length': '0'} 2022-09-12T19:48:07.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f16e6f80ccff0812-00631f8cf7-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.345 DEBUG:boto:b'jdpiak-72a1000falsea2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:07.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:48:07.350 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:07.351 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:07.352 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:07.354 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:07.360 DEBUG:boto:Method: GET 2022-09-12T19:48:07.361 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:48:07.363 DEBUG:boto:Data: 2022-09-12T19:48:07.364 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.365 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:07.366 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.373 DEBUG:boto:Params: {} 2022-09-12T19:48:07.374 DEBUG:boto:Token: None 2022-09-12T19:48:07.375 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:48:07.376 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eRswLXThjHa9e/+SjggZCTE79Jc= 2022-09-12T19:48:07.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eRswLXThjHa9e/+SjggZCTE79Jc=', 'Content-Length': '0'} 2022-09-12T19:48:07.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013ab34294b2ea652-00631f8cf7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.382 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:07.383 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:07.384 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:07.385 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:07.386 DEBUG:boto:Method: GET 2022-09-12T19:48:07.387 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:48:07.388 DEBUG:boto:Data: 2022-09-12T19:48:07.389 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.398 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:07.399 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.400 DEBUG:boto:Params: {} 2022-09-12T19:48:07.403 DEBUG:boto:Token: None 2022-09-12T19:48:07.405 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:48:07.405 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eRswLXThjHa9e/+SjggZCTE79Jc= 2022-09-12T19:48:07.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eRswLXThjHa9e/+SjggZCTE79Jc=', 'Content-Length': '0'} 2022-09-12T19:48:07.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004621d26194700337-00631f8cf7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.416 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:07.417 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:07.419 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:07.420 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:07.421 DEBUG:boto:Method: HEAD 2022-09-12T19:48:07.422 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:48:07.423 DEBUG:boto:Data: 2022-09-12T19:48:07.427 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.429 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:07.431 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.432 DEBUG:boto:Params: {} 2022-09-12T19:48:07.433 DEBUG:boto:Token: None 2022-09-12T19:48:07.434 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:48:07.435 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CAkIKBkDgrz6/dN8F84KGFf1gf0= 2022-09-12T19:48:07.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CAkIKBkDgrz6/dN8F84KGFf1gf0=', 'Content-Length': '0'} 2022-09-12T19:48:07.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d52624f22207fcd-00631f8cf7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.441 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:07.441 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:07.443 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:07.444 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:07.445 DEBUG:boto:Method: HEAD 2022-09-12T19:48:07.446 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:48:07.448 DEBUG:boto:Data: 2022-09-12T19:48:07.448 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.449 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:07.450 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.451 DEBUG:boto:Params: {} 2022-09-12T19:48:07.453 DEBUG:boto:Token: None 2022-09-12T19:48:07.453 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:48:07.454 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CAkIKBkDgrz6/dN8F84KGFf1gf0= 2022-09-12T19:48:07.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CAkIKBkDgrz6/dN8F84KGFf1gf0=', 'Content-Length': '0'} 2022-09-12T19:48:07.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a7872ce09925f69-00631f8cf7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:48:07.466 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:07.467 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:07.468 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:07.468 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:07.469 DEBUG:boto:Method: GET 2022-09-12T19:48:07.469 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:48:07.470 DEBUG:boto:Data: 2022-09-12T19:48:07.470 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.471 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:07.472 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.472 DEBUG:boto:Params: {} 2022-09-12T19:48:07.473 DEBUG:boto:Token: None 2022-09-12T19:48:07.473 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:48:07.474 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eRswLXThjHa9e/+SjggZCTE79Jc= 2022-09-12T19:48:07.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eRswLXThjHa9e/+SjggZCTE79Jc=', 'Content-Length': '0'} 2022-09-12T19:48:07.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f224694aa06b598b-00631f8cf7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.481 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:07.482 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:07.483 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:07.485 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:07.485 DEBUG:boto:Method: GET 2022-09-12T19:48:07.486 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:48:07.487 DEBUG:boto:Data: 2022-09-12T19:48:07.489 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.490 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:07.491 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.492 DEBUG:boto:Params: {} 2022-09-12T19:48:07.493 DEBUG:boto:Token: None 2022-09-12T19:48:07.494 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:48:07.495 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eRswLXThjHa9e/+SjggZCTE79Jc= 2022-09-12T19:48:07.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eRswLXThjHa9e/+SjggZCTE79Jc=', 'Content-Length': '0'} 2022-09-12T19:48:07.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bfe32286b2700a2-00631f8cf7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.550 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:07.551 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:07.551 DEBUG:boto:path=/jdpiak-72/?prefix=b 2022-09-12T19:48:07.552 DEBUG:boto:auth_path=/jdpiak-72/?prefix=b 2022-09-12T19:48:07.553 DEBUG:boto:Method: GET 2022-09-12T19:48:07.553 DEBUG:boto:Path: /jdpiak-72/?prefix=b 2022-09-12T19:48:07.554 DEBUG:boto:Data: 2022-09-12T19:48:07.554 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.555 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:07.555 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.556 DEBUG:boto:Params: {} 2022-09-12T19:48:07.556 DEBUG:boto:Token: None 2022-09-12T19:48:07.557 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/ 2022-09-12T19:48:07.558 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ToHM+D25Vadjv5gWWBdw4Swr/rE= 2022-09-12T19:48:07.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ToHM+D25Vadjv5gWWBdw4Swr/rE=', 'Content-Length': '0'} 2022-09-12T19:48:07.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2c2a0fd04a768ce-00631f8cf7-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.561 DEBUG:boto:b'jdpiak-72b1000falseb2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:07.562 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:07.563 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:07.564 DEBUG:boto:path=/jdpiak-72/?prefix=b 2022-09-12T19:48:07.565 DEBUG:boto:auth_path=/jdpiak-72/?prefix=b 2022-09-12T19:48:07.565 DEBUG:boto:Method: GET 2022-09-12T19:48:07.566 DEBUG:boto:Path: /jdpiak-72/?prefix=b 2022-09-12T19:48:07.566 DEBUG:boto:Data: 2022-09-12T19:48:07.567 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.567 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:07.568 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.568 DEBUG:boto:Params: {} 2022-09-12T19:48:07.569 DEBUG:boto:Token: None 2022-09-12T19:48:07.570 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/ 2022-09-12T19:48:07.570 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ToHM+D25Vadjv5gWWBdw4Swr/rE= 2022-09-12T19:48:07.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ToHM+D25Vadjv5gWWBdw4Swr/rE=', 'Content-Length': '0'} 2022-09-12T19:48:07.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5b5c4750a1676fd-00631f8cf7-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.574 DEBUG:boto:b'jdpiak-72b1000falseb2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:07.575 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:07.575 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:07.576 DEBUG:boto:Method: HEAD 2022-09-12T19:48:07.577 DEBUG:boto:Path: /jdpiak-72/b 2022-09-12T19:48:07.577 DEBUG:boto:Data: 2022-09-12T19:48:07.578 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.579 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:07.579 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.580 DEBUG:boto:Params: {} 2022-09-12T19:48:07.581 DEBUG:boto:Token: None 2022-09-12T19:48:07.582 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/b 2022-09-12T19:48:07.583 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZcEL/GhFLSGor0bJMfPTX2NmIpQ= 2022-09-12T19:48:07.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZcEL/GhFLSGor0bJMfPTX2NmIpQ=', 'Content-Length': '0'} 2022-09-12T19:48:07.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000654ddf1ca1afc3e4-00631f8cf7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.588 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:07.589 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:07.590 DEBUG:boto:Method: HEAD 2022-09-12T19:48:07.591 DEBUG:boto:Path: /jdpiak-72/b 2022-09-12T19:48:07.591 DEBUG:boto:Data: 2022-09-12T19:48:07.592 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.592 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:07.593 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.593 DEBUG:boto:Params: {} 2022-09-12T19:48:07.594 DEBUG:boto:Token: None 2022-09-12T19:48:07.594 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/b 2022-09-12T19:48:07.595 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZcEL/GhFLSGor0bJMfPTX2NmIpQ= 2022-09-12T19:48:07.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZcEL/GhFLSGor0bJMfPTX2NmIpQ=', 'Content-Length': '0'} 2022-09-12T19:48:07.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc6a8928faff75e6-00631f8cf7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:48:07.600 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:07.601 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:07.601 DEBUG:boto:Method: GET 2022-09-12T19:48:07.602 DEBUG:boto:Path: /jdpiak-72/b 2022-09-12T19:48:07.603 DEBUG:boto:Data: 2022-09-12T19:48:07.603 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.604 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:07.605 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.605 DEBUG:boto:Params: {} 2022-09-12T19:48:07.606 DEBUG:boto:Token: None 2022-09-12T19:48:07.607 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/b 2022-09-12T19:48:07.607 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ox6VQwDyBNSZjsaPZFM4kL2h4tg= 2022-09-12T19:48:07.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ox6VQwDyBNSZjsaPZFM4kL2h4tg=', 'Content-Length': '0'} 2022-09-12T19:48:07.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcd630a170d02631-00631f8cf7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.611 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:07.611 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:07.612 DEBUG:boto:Method: GET 2022-09-12T19:48:07.613 DEBUG:boto:Path: /jdpiak-72/b 2022-09-12T19:48:07.613 DEBUG:boto:Data: 2022-09-12T19:48:07.614 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.614 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:07.615 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.615 DEBUG:boto:Params: {} 2022-09-12T19:48:07.616 DEBUG:boto:Token: None 2022-09-12T19:48:07.617 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/b 2022-09-12T19:48:07.618 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ox6VQwDyBNSZjsaPZFM4kL2h4tg= 2022-09-12T19:48:07.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ox6VQwDyBNSZjsaPZFM4kL2h4tg=', 'Content-Length': '0'} 2022-09-12T19:48:07.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088d6e07682fbdda8-00631f8cf7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.621 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:07.622 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:07.623 DEBUG:boto:path=/jdpiak-72/?prefix=b 2022-09-12T19:48:07.623 DEBUG:boto:auth_path=/jdpiak-72/?prefix=b 2022-09-12T19:48:07.624 DEBUG:boto:Method: GET 2022-09-12T19:48:07.624 DEBUG:boto:Path: /jdpiak-72/?prefix=b 2022-09-12T19:48:07.625 DEBUG:boto:Data: 2022-09-12T19:48:07.625 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.626 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:07.626 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.627 DEBUG:boto:Params: {} 2022-09-12T19:48:07.628 DEBUG:boto:Token: None 2022-09-12T19:48:07.628 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/ 2022-09-12T19:48:07.629 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ToHM+D25Vadjv5gWWBdw4Swr/rE= 2022-09-12T19:48:07.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ToHM+D25Vadjv5gWWBdw4Swr/rE=', 'Content-Length': '0'} 2022-09-12T19:48:07.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012a9e477550585d6-00631f8cf7-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.633 DEBUG:boto:b'jdpiak-72b1000falseb2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:07.634 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:07.635 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:07.635 DEBUG:boto:path=/jdpiak-72/?prefix=b 2022-09-12T19:48:07.636 DEBUG:boto:auth_path=/jdpiak-72/?prefix=b 2022-09-12T19:48:07.636 DEBUG:boto:Method: GET 2022-09-12T19:48:07.637 DEBUG:boto:Path: /jdpiak-72/?prefix=b 2022-09-12T19:48:07.637 DEBUG:boto:Data: 2022-09-12T19:48:07.638 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.638 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:07.639 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.640 DEBUG:boto:Params: {} 2022-09-12T19:48:07.640 DEBUG:boto:Token: None 2022-09-12T19:48:07.641 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/ 2022-09-12T19:48:07.642 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ToHM+D25Vadjv5gWWBdw4Swr/rE= 2022-09-12T19:48:07.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ToHM+D25Vadjv5gWWBdw4Swr/rE=', 'Content-Length': '0'} 2022-09-12T19:48:07.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019893b499b025d3c-00631f8cf7-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.645 DEBUG:boto:b'jdpiak-72b1000falseb2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:07.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:48:07.647 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:07.647 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:07.648 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:07.648 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:07.649 DEBUG:boto:Method: GET 2022-09-12T19:48:07.650 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:48:07.650 DEBUG:boto:Data: 2022-09-12T19:48:07.651 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.651 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:07.652 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.653 DEBUG:boto:Params: {} 2022-09-12T19:48:07.653 DEBUG:boto:Token: None 2022-09-12T19:48:07.654 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:48:07.655 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fR2UiQvb0UnGjwiBcszIeyVyL9c= 2022-09-12T19:48:07.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fR2UiQvb0UnGjwiBcszIeyVyL9c=', 'Content-Length': '0'} 2022-09-12T19:48:07.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcf8bef269a0189f-00631f8cf7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.658 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:07.659 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:07.659 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:07.660 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:07.661 DEBUG:boto:Method: GET 2022-09-12T19:48:07.661 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:48:07.662 DEBUG:boto:Data: 2022-09-12T19:48:07.662 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.663 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:07.664 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.664 DEBUG:boto:Params: {} 2022-09-12T19:48:07.665 DEBUG:boto:Token: None 2022-09-12T19:48:07.666 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:48:07.666 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fR2UiQvb0UnGjwiBcszIeyVyL9c= 2022-09-12T19:48:07.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fR2UiQvb0UnGjwiBcszIeyVyL9c=', 'Content-Length': '0'} 2022-09-12T19:48:07.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc9518ad0ebd5d67-00631f8cf7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.670 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:07.671 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:07.671 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:07.672 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:07.672 DEBUG:boto:Method: HEAD 2022-09-12T19:48:07.673 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:48:07.674 DEBUG:boto:Data: 2022-09-12T19:48:07.675 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.676 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:07.676 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.677 DEBUG:boto:Params: {} 2022-09-12T19:48:07.678 DEBUG:boto:Token: None 2022-09-12T19:48:07.679 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:48:07.679 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LzUlcI1jT9ZTUFoGjQVBsqvXcAQ= 2022-09-12T19:48:07.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LzUlcI1jT9ZTUFoGjQVBsqvXcAQ=', 'Content-Length': '0'} 2022-09-12T19:48:07.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d2a2549df4601bb-00631f8cf7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.683 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:07.684 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:07.685 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:07.685 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:07.686 DEBUG:boto:Method: HEAD 2022-09-12T19:48:07.687 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:48:07.687 DEBUG:boto:Data: 2022-09-12T19:48:07.688 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.688 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:07.689 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.690 DEBUG:boto:Params: {} 2022-09-12T19:48:07.691 DEBUG:boto:Token: None 2022-09-12T19:48:07.691 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:48:07.692 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LzUlcI1jT9ZTUFoGjQVBsqvXcAQ= 2022-09-12T19:48:07.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LzUlcI1jT9ZTUFoGjQVBsqvXcAQ=', 'Content-Length': '0'} 2022-09-12T19:48:07.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af5a9c948e54017b-00631f8cf7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:48:07.704 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:07.704 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:07.705 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:07.705 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:07.706 DEBUG:boto:Method: GET 2022-09-12T19:48:07.706 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:48:07.707 DEBUG:boto:Data: 2022-09-12T19:48:07.708 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.708 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:07.709 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.710 DEBUG:boto:Params: {} 2022-09-12T19:48:07.710 DEBUG:boto:Token: None 2022-09-12T19:48:07.711 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:48:07.712 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fR2UiQvb0UnGjwiBcszIeyVyL9c= 2022-09-12T19:48:07.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fR2UiQvb0UnGjwiBcszIeyVyL9c=', 'Content-Length': '0'} 2022-09-12T19:48:07.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc5421f93db7f11c-00631f8cf7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.716 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:07.716 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:07.717 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:07.717 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:07.719 DEBUG:boto:Method: GET 2022-09-12T19:48:07.719 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:48:07.720 DEBUG:boto:Data: 2022-09-12T19:48:07.720 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.721 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:07.721 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.722 DEBUG:boto:Params: {} 2022-09-12T19:48:07.723 DEBUG:boto:Token: None 2022-09-12T19:48:07.723 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:48:07.724 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fR2UiQvb0UnGjwiBcszIeyVyL9c= 2022-09-12T19:48:07.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fR2UiQvb0UnGjwiBcszIeyVyL9c=', 'Content-Length': '0'} 2022-09-12T19:48:07.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000388374331a766d01-00631f8cf7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.776 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:07.777 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:07.778 DEBUG:boto:path=/jdpiak-72/?prefix=c 2022-09-12T19:48:07.778 DEBUG:boto:auth_path=/jdpiak-72/?prefix=c 2022-09-12T19:48:07.779 DEBUG:boto:Method: GET 2022-09-12T19:48:07.779 DEBUG:boto:Path: /jdpiak-72/?prefix=c 2022-09-12T19:48:07.780 DEBUG:boto:Data: 2022-09-12T19:48:07.781 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.781 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:07.782 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.782 DEBUG:boto:Params: {} 2022-09-12T19:48:07.783 DEBUG:boto:Token: None 2022-09-12T19:48:07.783 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/ 2022-09-12T19:48:07.784 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ToHM+D25Vadjv5gWWBdw4Swr/rE= 2022-09-12T19:48:07.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ToHM+D25Vadjv5gWWBdw4Swr/rE=', 'Content-Length': '0'} 2022-09-12T19:48:07.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066544a23b64f49f7-00631f8cf7-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.788 DEBUG:boto:b'jdpiak-72c1000falsec2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:07.789 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:07.789 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:07.790 DEBUG:boto:path=/jdpiak-72/?prefix=c 2022-09-12T19:48:07.790 DEBUG:boto:auth_path=/jdpiak-72/?prefix=c 2022-09-12T19:48:07.791 DEBUG:boto:Method: GET 2022-09-12T19:48:07.791 DEBUG:boto:Path: /jdpiak-72/?prefix=c 2022-09-12T19:48:07.792 DEBUG:boto:Data: 2022-09-12T19:48:07.792 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.793 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:07.793 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.794 DEBUG:boto:Params: {} 2022-09-12T19:48:07.794 DEBUG:boto:Token: None 2022-09-12T19:48:07.795 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/ 2022-09-12T19:48:07.795 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ToHM+D25Vadjv5gWWBdw4Swr/rE= 2022-09-12T19:48:07.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ToHM+D25Vadjv5gWWBdw4Swr/rE=', 'Content-Length': '0'} 2022-09-12T19:48:07.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f89147b031ce943-00631f8cf7-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.799 DEBUG:boto:b'jdpiak-72c1000falsec2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:07.800 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:07.800 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:07.801 DEBUG:boto:Method: HEAD 2022-09-12T19:48:07.801 DEBUG:boto:Path: /jdpiak-72/c 2022-09-12T19:48:07.802 DEBUG:boto:Data: 2022-09-12T19:48:07.802 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.803 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:07.803 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.804 DEBUG:boto:Params: {} 2022-09-12T19:48:07.805 DEBUG:boto:Token: None 2022-09-12T19:48:07.805 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/c 2022-09-12T19:48:07.806 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p2VuIxmJ4I147Q3BO+WzRhguttA= 2022-09-12T19:48:07.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p2VuIxmJ4I147Q3BO+WzRhguttA=', 'Content-Length': '0'} 2022-09-12T19:48:07.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000300876df37472e06-00631f8cf7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.809 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:07.810 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:07.810 DEBUG:boto:Method: HEAD 2022-09-12T19:48:07.811 DEBUG:boto:Path: /jdpiak-72/c 2022-09-12T19:48:07.811 DEBUG:boto:Data: 2022-09-12T19:48:07.812 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.812 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:07.813 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.813 DEBUG:boto:Params: {} 2022-09-12T19:48:07.814 DEBUG:boto:Token: None 2022-09-12T19:48:07.814 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/c 2022-09-12T19:48:07.815 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p2VuIxmJ4I147Q3BO+WzRhguttA= 2022-09-12T19:48:07.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p2VuIxmJ4I147Q3BO+WzRhguttA=', 'Content-Length': '0'} 2022-09-12T19:48:07.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000562e31fa0ceebf42-00631f8cf7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:48:07.819 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:07.820 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:07.820 DEBUG:boto:Method: GET 2022-09-12T19:48:07.821 DEBUG:boto:Path: /jdpiak-72/c 2022-09-12T19:48:07.821 DEBUG:boto:Data: 2022-09-12T19:48:07.822 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.822 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:07.823 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.823 DEBUG:boto:Params: {} 2022-09-12T19:48:07.824 DEBUG:boto:Token: None 2022-09-12T19:48:07.825 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/c 2022-09-12T19:48:07.825 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DG1LmqD//8tf5l2TOAj0upbCfi8= 2022-09-12T19:48:07.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DG1LmqD//8tf5l2TOAj0upbCfi8=', 'Content-Length': '0'} 2022-09-12T19:48:07.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049910ed07e844bd3-00631f8cf7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.870 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:07.871 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:07.872 DEBUG:boto:Method: GET 2022-09-12T19:48:07.873 DEBUG:boto:Path: /jdpiak-72/c 2022-09-12T19:48:07.874 DEBUG:boto:Data: 2022-09-12T19:48:07.874 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.875 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:07.875 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.876 DEBUG:boto:Params: {} 2022-09-12T19:48:07.877 DEBUG:boto:Token: None 2022-09-12T19:48:07.878 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/c 2022-09-12T19:48:07.879 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DG1LmqD//8tf5l2TOAj0upbCfi8= 2022-09-12T19:48:07.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DG1LmqD//8tf5l2TOAj0upbCfi8=', 'Content-Length': '0'} 2022-09-12T19:48:07.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e4c619d58093759-00631f8cf7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.887 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:07.888 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:07.889 DEBUG:boto:path=/jdpiak-72/?prefix=c 2022-09-12T19:48:07.889 DEBUG:boto:auth_path=/jdpiak-72/?prefix=c 2022-09-12T19:48:07.890 DEBUG:boto:Method: GET 2022-09-12T19:48:07.891 DEBUG:boto:Path: /jdpiak-72/?prefix=c 2022-09-12T19:48:07.892 DEBUG:boto:Data: 2022-09-12T19:48:07.893 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.893 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:07.894 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.895 DEBUG:boto:Params: {} 2022-09-12T19:48:07.896 DEBUG:boto:Token: None 2022-09-12T19:48:07.897 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/ 2022-09-12T19:48:07.897 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ToHM+D25Vadjv5gWWBdw4Swr/rE= 2022-09-12T19:48:07.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ToHM+D25Vadjv5gWWBdw4Swr/rE=', 'Content-Length': '0'} 2022-09-12T19:48:07.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4cf82a2f3d8f95d-00631f8cf7-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.901 DEBUG:boto:b'jdpiak-72c1000falsec2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:07.902 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:07.902 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:07.903 DEBUG:boto:path=/jdpiak-72/?prefix=c 2022-09-12T19:48:07.904 DEBUG:boto:auth_path=/jdpiak-72/?prefix=c 2022-09-12T19:48:07.904 DEBUG:boto:Method: GET 2022-09-12T19:48:07.905 DEBUG:boto:Path: /jdpiak-72/?prefix=c 2022-09-12T19:48:07.905 DEBUG:boto:Data: 2022-09-12T19:48:07.906 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.906 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:07.907 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.907 DEBUG:boto:Params: {} 2022-09-12T19:48:07.908 DEBUG:boto:Token: None 2022-09-12T19:48:07.909 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/ 2022-09-12T19:48:07.910 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ToHM+D25Vadjv5gWWBdw4Swr/rE= 2022-09-12T19:48:07.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ToHM+D25Vadjv5gWWBdw4Swr/rE=', 'Content-Length': '0'} 2022-09-12T19:48:07.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000956d356213e4f924-00631f8cf7-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.958 DEBUG:boto:b'jdpiak-72c1000falsec2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:07.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:48:07.960 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:07.961 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:07.961 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:07.963 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:07.964 DEBUG:boto:Method: GET 2022-09-12T19:48:07.965 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:48:07.966 DEBUG:boto:Data: 2022-09-12T19:48:07.967 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.967 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:07.968 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.969 DEBUG:boto:Params: {} 2022-09-12T19:48:07.970 DEBUG:boto:Token: None 2022-09-12T19:48:07.972 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:48:07.973 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QVwM40yLxZs+KlH04Y2rxfYVymQ= 2022-09-12T19:48:07.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QVwM40yLxZs+KlH04Y2rxfYVymQ=', 'Content-Length': '0'} 2022-09-12T19:48:07.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c662f395fb56815-00631f8cf7-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.979 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:07.979 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:07.980 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:07.980 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:07.981 DEBUG:boto:Method: GET 2022-09-12T19:48:07.983 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:48:07.983 DEBUG:boto:Data: 2022-09-12T19:48:07.984 DEBUG:boto:Headers: {} 2022-09-12T19:48:07.986 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:07.987 DEBUG:boto:Port: 8000 2022-09-12T19:48:07.987 DEBUG:boto:Params: {} 2022-09-12T19:48:07.989 DEBUG:boto:Token: None 2022-09-12T19:48:07.990 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:07 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:48:07.992 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QVwM40yLxZs+KlH04Y2rxfYVymQ= 2022-09-12T19:48:07.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QVwM40yLxZs+KlH04Y2rxfYVymQ=', 'Content-Length': '0'} 2022-09-12T19:48:07.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020ec9e3099593096-00631f8cf7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:07.998 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:07.999 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:08.000 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:08.001 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:08.002 DEBUG:boto:Method: HEAD 2022-09-12T19:48:08.003 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:48:08.003 DEBUG:boto:Data: 2022-09-12T19:48:08.004 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.005 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.005 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.006 DEBUG:boto:Params: {} 2022-09-12T19:48:08.007 DEBUG:boto:Token: None 2022-09-12T19:48:08.008 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:48:08.009 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2zVQGKYq/zb9ydhZ2p6r2eYSgvk= 2022-09-12T19:48:08.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2zVQGKYq/zb9ydhZ2p6r2eYSgvk=', 'Content-Length': '0'} 2022-09-12T19:48:08.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040af97d60d75aece-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.014 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:08.016 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:08.017 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:08.017 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:08.018 DEBUG:boto:Method: HEAD 2022-09-12T19:48:08.019 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:48:08.020 DEBUG:boto:Data: 2022-09-12T19:48:08.021 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.021 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.022 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.023 DEBUG:boto:Params: {} 2022-09-12T19:48:08.023 DEBUG:boto:Token: None 2022-09-12T19:48:08.024 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:48:08.025 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2zVQGKYq/zb9ydhZ2p6r2eYSgvk= 2022-09-12T19:48:08.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2zVQGKYq/zb9ydhZ2p6r2eYSgvk=', 'Content-Length': '0'} 2022-09-12T19:48:08.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa76fa8da89d9171-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:48:08.030 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:08.031 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:08.032 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:08.033 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:08.034 DEBUG:boto:Method: GET 2022-09-12T19:48:08.035 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:48:08.035 DEBUG:boto:Data: 2022-09-12T19:48:08.036 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.037 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.038 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.039 DEBUG:boto:Params: {} 2022-09-12T19:48:08.040 DEBUG:boto:Token: None 2022-09-12T19:48:08.040 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:48:08.041 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Gzz/ToVCgYP2pIpglocS34+w1Ns= 2022-09-12T19:48:08.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Gzz/ToVCgYP2pIpglocS34+w1Ns=', 'Content-Length': '0'} 2022-09-12T19:48:08.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099427775505a1136-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.045 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:08.046 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:08.046 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:08.047 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:08.047 DEBUG:boto:Method: GET 2022-09-12T19:48:08.048 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:48:08.048 DEBUG:boto:Data: 2022-09-12T19:48:08.049 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.049 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.050 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.051 DEBUG:boto:Params: {} 2022-09-12T19:48:08.051 DEBUG:boto:Token: None 2022-09-12T19:48:08.052 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:48:08.052 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Gzz/ToVCgYP2pIpglocS34+w1Ns= 2022-09-12T19:48:08.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Gzz/ToVCgYP2pIpglocS34+w1Ns=', 'Content-Length': '0'} 2022-09-12T19:48:08.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a74eb62123f64157-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.057 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:08.057 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:08.058 DEBUG:boto:path=/jdpiak-72/?prefix=d 2022-09-12T19:48:08.058 DEBUG:boto:auth_path=/jdpiak-72/?prefix=d 2022-09-12T19:48:08.059 DEBUG:boto:Method: GET 2022-09-12T19:48:08.059 DEBUG:boto:Path: /jdpiak-72/?prefix=d 2022-09-12T19:48:08.060 DEBUG:boto:Data: 2022-09-12T19:48:08.060 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.061 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.061 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.062 DEBUG:boto:Params: {} 2022-09-12T19:48:08.063 DEBUG:boto:Token: None 2022-09-12T19:48:08.063 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/ 2022-09-12T19:48:08.064 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8= 2022-09-12T19:48:08.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8=', 'Content-Length': '0'} 2022-09-12T19:48:08.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092fe2a090cfd23eb-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.067 DEBUG:boto:b'jdpiak-72d1000falsed2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:08.068 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:08.069 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:08.069 DEBUG:boto:path=/jdpiak-72/?prefix=d 2022-09-12T19:48:08.070 DEBUG:boto:auth_path=/jdpiak-72/?prefix=d 2022-09-12T19:48:08.070 DEBUG:boto:Method: GET 2022-09-12T19:48:08.071 DEBUG:boto:Path: /jdpiak-72/?prefix=d 2022-09-12T19:48:08.072 DEBUG:boto:Data: 2022-09-12T19:48:08.072 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.073 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.074 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.075 DEBUG:boto:Params: {} 2022-09-12T19:48:08.075 DEBUG:boto:Token: None 2022-09-12T19:48:08.076 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/ 2022-09-12T19:48:08.077 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8= 2022-09-12T19:48:08.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8=', 'Content-Length': '0'} 2022-09-12T19:48:08.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d16ea09a262ca1d3-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.080 DEBUG:boto:b'jdpiak-72d1000falsed2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:08.081 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:08.081 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:08.082 DEBUG:boto:Method: HEAD 2022-09-12T19:48:08.083 DEBUG:boto:Path: /jdpiak-72/d 2022-09-12T19:48:08.083 DEBUG:boto:Data: 2022-09-12T19:48:08.084 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.084 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.085 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.085 DEBUG:boto:Params: {} 2022-09-12T19:48:08.086 DEBUG:boto:Token: None 2022-09-12T19:48:08.086 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/d 2022-09-12T19:48:08.087 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vrAc+c7uyLYpG0VN9AnXWn1dQhw= 2022-09-12T19:48:08.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vrAc+c7uyLYpG0VN9AnXWn1dQhw=', 'Content-Length': '0'} 2022-09-12T19:48:08.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005909967f24ca2148-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.090 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:08.091 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:08.092 DEBUG:boto:Method: HEAD 2022-09-12T19:48:08.092 DEBUG:boto:Path: /jdpiak-72/d 2022-09-12T19:48:08.092 DEBUG:boto:Data: 2022-09-12T19:48:08.093 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.094 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.094 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.095 DEBUG:boto:Params: {} 2022-09-12T19:48:08.095 DEBUG:boto:Token: None 2022-09-12T19:48:08.096 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/d 2022-09-12T19:48:08.097 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vrAc+c7uyLYpG0VN9AnXWn1dQhw= 2022-09-12T19:48:08.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vrAc+c7uyLYpG0VN9AnXWn1dQhw=', 'Content-Length': '0'} 2022-09-12T19:48:08.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002daefe18409c6d17-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:48:08.101 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:08.101 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:08.102 DEBUG:boto:Method: GET 2022-09-12T19:48:08.102 DEBUG:boto:Path: /jdpiak-72/d 2022-09-12T19:48:08.103 DEBUG:boto:Data: 2022-09-12T19:48:08.103 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.104 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.105 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.106 DEBUG:boto:Params: {} 2022-09-12T19:48:08.108 DEBUG:boto:Token: None 2022-09-12T19:48:08.109 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/d 2022-09-12T19:48:08.110 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OJpg/IZWPHia1Dyg1e2IfCGB7Gc= 2022-09-12T19:48:08.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OJpg/IZWPHia1Dyg1e2IfCGB7Gc=', 'Content-Length': '0'} 2022-09-12T19:48:08.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000681bbd4cc9e490a4-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.115 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:08.116 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:08.116 DEBUG:boto:Method: GET 2022-09-12T19:48:08.117 DEBUG:boto:Path: /jdpiak-72/d 2022-09-12T19:48:08.117 DEBUG:boto:Data: 2022-09-12T19:48:08.118 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.119 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.119 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.120 DEBUG:boto:Params: {} 2022-09-12T19:48:08.120 DEBUG:boto:Token: None 2022-09-12T19:48:08.121 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/d 2022-09-12T19:48:08.122 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OJpg/IZWPHia1Dyg1e2IfCGB7Gc= 2022-09-12T19:48:08.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OJpg/IZWPHia1Dyg1e2IfCGB7Gc=', 'Content-Length': '0'} 2022-09-12T19:48:08.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000598e869a52155a9e-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.126 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:08.126 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:08.127 DEBUG:boto:path=/jdpiak-72/?prefix=d 2022-09-12T19:48:08.127 DEBUG:boto:auth_path=/jdpiak-72/?prefix=d 2022-09-12T19:48:08.128 DEBUG:boto:Method: GET 2022-09-12T19:48:08.129 DEBUG:boto:Path: /jdpiak-72/?prefix=d 2022-09-12T19:48:08.129 DEBUG:boto:Data: 2022-09-12T19:48:08.130 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.130 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.131 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.131 DEBUG:boto:Params: {} 2022-09-12T19:48:08.132 DEBUG:boto:Token: None 2022-09-12T19:48:08.132 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/ 2022-09-12T19:48:08.133 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8= 2022-09-12T19:48:08.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8=', 'Content-Length': '0'} 2022-09-12T19:48:08.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000663a95700769337a-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.179 DEBUG:boto:b'jdpiak-72d1000falsed2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:08.180 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:08.181 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:08.181 DEBUG:boto:path=/jdpiak-72/?prefix=d 2022-09-12T19:48:08.182 DEBUG:boto:auth_path=/jdpiak-72/?prefix=d 2022-09-12T19:48:08.182 DEBUG:boto:Method: GET 2022-09-12T19:48:08.183 DEBUG:boto:Path: /jdpiak-72/?prefix=d 2022-09-12T19:48:08.183 DEBUG:boto:Data: 2022-09-12T19:48:08.184 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.185 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.185 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.186 DEBUG:boto:Params: {} 2022-09-12T19:48:08.186 DEBUG:boto:Token: None 2022-09-12T19:48:08.187 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/ 2022-09-12T19:48:08.188 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8= 2022-09-12T19:48:08.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8=', 'Content-Length': '0'} 2022-09-12T19:48:08.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa4d1ec2d10c803f-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.192 DEBUG:boto:b'jdpiak-72d1000falsed2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:08.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:48:08.194 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:08.194 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:08.195 DEBUG:boto:path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:08.196 DEBUG:boto:auth_path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:08.196 DEBUG:boto:Method: GET 2022-09-12T19:48:08.197 DEBUG:boto:Path: /jdpiak-72/e?versionId=null 2022-09-12T19:48:08.198 DEBUG:boto:Data: 2022-09-12T19:48:08.199 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.200 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.201 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.201 DEBUG:boto:Params: {} 2022-09-12T19:48:08.202 DEBUG:boto:Token: None 2022-09-12T19:48:08.203 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/e?versionId=null 2022-09-12T19:48:08.204 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nsm91olXgPi+LnQP+eJlcOxWnvA= 2022-09-12T19:48:08.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nsm91olXgPi+LnQP+eJlcOxWnvA=', 'Content-Length': '0'} 2022-09-12T19:48:08.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ce478d443e00a36-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.209 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:08.209 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:08.210 DEBUG:boto:path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:08.211 DEBUG:boto:auth_path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:08.212 DEBUG:boto:Method: GET 2022-09-12T19:48:08.213 DEBUG:boto:Path: /jdpiak-72/e?versionId=null 2022-09-12T19:48:08.213 DEBUG:boto:Data: 2022-09-12T19:48:08.214 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.215 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.216 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.217 DEBUG:boto:Params: {} 2022-09-12T19:48:08.218 DEBUG:boto:Token: None 2022-09-12T19:48:08.219 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/e?versionId=null 2022-09-12T19:48:08.220 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nsm91olXgPi+LnQP+eJlcOxWnvA= 2022-09-12T19:48:08.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nsm91olXgPi+LnQP+eJlcOxWnvA=', 'Content-Length': '0'} 2022-09-12T19:48:08.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075812fd892d55eda-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.225 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:08.226 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:08.227 DEBUG:boto:path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:08.227 DEBUG:boto:auth_path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:08.228 DEBUG:boto:Method: HEAD 2022-09-12T19:48:08.228 DEBUG:boto:Path: /jdpiak-72/e?versionId=null 2022-09-12T19:48:08.229 DEBUG:boto:Data: 2022-09-12T19:48:08.229 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.230 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.230 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.231 DEBUG:boto:Params: {} 2022-09-12T19:48:08.232 DEBUG:boto:Token: None 2022-09-12T19:48:08.232 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/e?versionId=null 2022-09-12T19:48:08.233 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jEyw4sr/C2flfn9eQG24LFdyrco= 2022-09-12T19:48:08.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jEyw4sr/C2flfn9eQG24LFdyrco=', 'Content-Length': '0'} 2022-09-12T19:48:08.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034b58938a1b24d31-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.236 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:08.237 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:08.237 DEBUG:boto:path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:08.238 DEBUG:boto:auth_path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:08.238 DEBUG:boto:Method: HEAD 2022-09-12T19:48:08.239 DEBUG:boto:Path: /jdpiak-72/e?versionId=null 2022-09-12T19:48:08.239 DEBUG:boto:Data: 2022-09-12T19:48:08.240 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.240 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.241 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.241 DEBUG:boto:Params: {} 2022-09-12T19:48:08.242 DEBUG:boto:Token: None 2022-09-12T19:48:08.242 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/e?versionId=null 2022-09-12T19:48:08.244 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jEyw4sr/C2flfn9eQG24LFdyrco= 2022-09-12T19:48:08.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jEyw4sr/C2flfn9eQG24LFdyrco=', 'Content-Length': '0'} 2022-09-12T19:48:08.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031255f7a54a9fe9e-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:48:08.249 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:08.249 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:08.250 DEBUG:boto:path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:08.250 DEBUG:boto:auth_path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:08.251 DEBUG:boto:Method: GET 2022-09-12T19:48:08.252 DEBUG:boto:Path: /jdpiak-72/e?versionId=null 2022-09-12T19:48:08.252 DEBUG:boto:Data: 2022-09-12T19:48:08.253 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.253 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.254 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.256 DEBUG:boto:Params: {} 2022-09-12T19:48:08.257 DEBUG:boto:Token: None 2022-09-12T19:48:08.258 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/e?versionId=null 2022-09-12T19:48:08.259 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nsm91olXgPi+LnQP+eJlcOxWnvA= 2022-09-12T19:48:08.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nsm91olXgPi+LnQP+eJlcOxWnvA=', 'Content-Length': '0'} 2022-09-12T19:48:08.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc40a91e813f5332-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.263 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:08.264 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:08.265 DEBUG:boto:path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:08.266 DEBUG:boto:auth_path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:08.267 DEBUG:boto:Method: GET 2022-09-12T19:48:08.267 DEBUG:boto:Path: /jdpiak-72/e?versionId=null 2022-09-12T19:48:08.268 DEBUG:boto:Data: 2022-09-12T19:48:08.268 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.269 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.269 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.270 DEBUG:boto:Params: {} 2022-09-12T19:48:08.271 DEBUG:boto:Token: None 2022-09-12T19:48:08.271 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/e?versionId=null 2022-09-12T19:48:08.272 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nsm91olXgPi+LnQP+eJlcOxWnvA= 2022-09-12T19:48:08.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nsm91olXgPi+LnQP+eJlcOxWnvA=', 'Content-Length': '0'} 2022-09-12T19:48:08.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025747cd1ef8c346b-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.276 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:08.276 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:08.277 DEBUG:boto:path=/jdpiak-72/?prefix=e 2022-09-12T19:48:08.277 DEBUG:boto:auth_path=/jdpiak-72/?prefix=e 2022-09-12T19:48:08.278 DEBUG:boto:Method: GET 2022-09-12T19:48:08.279 DEBUG:boto:Path: /jdpiak-72/?prefix=e 2022-09-12T19:48:08.279 DEBUG:boto:Data: 2022-09-12T19:48:08.280 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.281 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.281 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.282 DEBUG:boto:Params: {} 2022-09-12T19:48:08.282 DEBUG:boto:Token: None 2022-09-12T19:48:08.283 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/ 2022-09-12T19:48:08.284 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8= 2022-09-12T19:48:08.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8=', 'Content-Length': '0'} 2022-09-12T19:48:08.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d40ec617dba6f4f1-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.288 DEBUG:boto:b'jdpiak-72e1000falsee2022-09-12T19:45:20.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:08.289 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:08.290 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:08.290 DEBUG:boto:path=/jdpiak-72/?prefix=e 2022-09-12T19:48:08.291 DEBUG:boto:auth_path=/jdpiak-72/?prefix=e 2022-09-12T19:48:08.292 DEBUG:boto:Method: GET 2022-09-12T19:48:08.292 DEBUG:boto:Path: /jdpiak-72/?prefix=e 2022-09-12T19:48:08.293 DEBUG:boto:Data: 2022-09-12T19:48:08.293 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.294 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.294 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.295 DEBUG:boto:Params: {} 2022-09-12T19:48:08.296 DEBUG:boto:Token: None 2022-09-12T19:48:08.296 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/ 2022-09-12T19:48:08.297 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8= 2022-09-12T19:48:08.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8=', 'Content-Length': '0'} 2022-09-12T19:48:08.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006532159161851ef4-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.342 DEBUG:boto:b'jdpiak-72e1000falsee2022-09-12T19:45:20.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:08.343 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:08.344 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:08.344 DEBUG:boto:Method: HEAD 2022-09-12T19:48:08.345 DEBUG:boto:Path: /jdpiak-72/e 2022-09-12T19:48:08.345 DEBUG:boto:Data: 2022-09-12T19:48:08.346 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.346 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.347 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.347 DEBUG:boto:Params: {} 2022-09-12T19:48:08.348 DEBUG:boto:Token: None 2022-09-12T19:48:08.349 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/e 2022-09-12T19:48:08.350 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lGtikky6kmfRAobEdBKt2mNniI4= 2022-09-12T19:48:08.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lGtikky6kmfRAobEdBKt2mNniI4=', 'Content-Length': '0'} 2022-09-12T19:48:08.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2fdb084aeac2c4c-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.355 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:08.355 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:08.356 DEBUG:boto:Method: HEAD 2022-09-12T19:48:08.356 DEBUG:boto:Path: /jdpiak-72/e 2022-09-12T19:48:08.357 DEBUG:boto:Data: 2022-09-12T19:48:08.357 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.358 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.358 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.359 DEBUG:boto:Params: {} 2022-09-12T19:48:08.360 DEBUG:boto:Token: None 2022-09-12T19:48:08.361 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/e 2022-09-12T19:48:08.361 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lGtikky6kmfRAobEdBKt2mNniI4= 2022-09-12T19:48:08.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lGtikky6kmfRAobEdBKt2mNniI4=', 'Content-Length': '0'} 2022-09-12T19:48:08.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025cbffa2d0c8b689-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:48:08.365 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:08.366 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:08.367 DEBUG:boto:Method: GET 2022-09-12T19:48:08.367 DEBUG:boto:Path: /jdpiak-72/e 2022-09-12T19:48:08.368 DEBUG:boto:Data: 2022-09-12T19:48:08.368 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.369 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.369 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.370 DEBUG:boto:Params: {} 2022-09-12T19:48:08.370 DEBUG:boto:Token: None 2022-09-12T19:48:08.371 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/e 2022-09-12T19:48:08.372 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lujflw34B7XWMd1yq7vED1MBo5c= 2022-09-12T19:48:08.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lujflw34B7XWMd1yq7vED1MBo5c=', 'Content-Length': '0'} 2022-09-12T19:48:08.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c5927b6d11a8bd0-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.375 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:08.376 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:08.376 DEBUG:boto:Method: GET 2022-09-12T19:48:08.377 DEBUG:boto:Path: /jdpiak-72/e 2022-09-12T19:48:08.377 DEBUG:boto:Data: 2022-09-12T19:48:08.378 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.379 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.380 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.380 DEBUG:boto:Params: {} 2022-09-12T19:48:08.381 DEBUG:boto:Token: None 2022-09-12T19:48:08.381 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/e 2022-09-12T19:48:08.382 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lujflw34B7XWMd1yq7vED1MBo5c= 2022-09-12T19:48:08.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lujflw34B7XWMd1yq7vED1MBo5c=', 'Content-Length': '0'} 2022-09-12T19:48:08.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000634ce6088df2860e-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.387 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:08.388 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:08.388 DEBUG:boto:path=/jdpiak-72/?prefix=e 2022-09-12T19:48:08.389 DEBUG:boto:auth_path=/jdpiak-72/?prefix=e 2022-09-12T19:48:08.389 DEBUG:boto:Method: GET 2022-09-12T19:48:08.390 DEBUG:boto:Path: /jdpiak-72/?prefix=e 2022-09-12T19:48:08.391 DEBUG:boto:Data: 2022-09-12T19:48:08.391 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.392 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.392 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.393 DEBUG:boto:Params: {} 2022-09-12T19:48:08.393 DEBUG:boto:Token: None 2022-09-12T19:48:08.394 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/ 2022-09-12T19:48:08.395 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8= 2022-09-12T19:48:08.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8=', 'Content-Length': '0'} 2022-09-12T19:48:08.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e418a9928e0b8af-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.399 DEBUG:boto:b'jdpiak-72e1000falsee2022-09-12T19:45:20.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:08.400 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:08.400 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:08.401 DEBUG:boto:path=/jdpiak-72/?prefix=e 2022-09-12T19:48:08.401 DEBUG:boto:auth_path=/jdpiak-72/?prefix=e 2022-09-12T19:48:08.402 DEBUG:boto:Method: GET 2022-09-12T19:48:08.403 DEBUG:boto:Path: /jdpiak-72/?prefix=e 2022-09-12T19:48:08.403 DEBUG:boto:Data: 2022-09-12T19:48:08.404 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.404 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.405 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.406 DEBUG:boto:Params: {} 2022-09-12T19:48:08.406 DEBUG:boto:Token: None 2022-09-12T19:48:08.407 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/ 2022-09-12T19:48:08.408 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8= 2022-09-12T19:48:08.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8=', 'Content-Length': '0'} 2022-09-12T19:48:08.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa629b33d5f04bff-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.424 DEBUG:boto:b'jdpiak-72e1000falsee2022-09-12T19:45:20.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:08.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:48:08.426 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:08.426 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:08.427 DEBUG:boto:path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:08.428 DEBUG:boto:auth_path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:08.428 DEBUG:boto:Method: GET 2022-09-12T19:48:08.429 DEBUG:boto:Path: /jdpiak-72/f?versionId=null 2022-09-12T19:48:08.430 DEBUG:boto:Data: 2022-09-12T19:48:08.431 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.432 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.433 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.434 DEBUG:boto:Params: {} 2022-09-12T19:48:08.435 DEBUG:boto:Token: None 2022-09-12T19:48:08.436 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/f?versionId=null 2022-09-12T19:48:08.436 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fBo6UK3e6mvAIUezPUJO0Al0WN0= 2022-09-12T19:48:08.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fBo6UK3e6mvAIUezPUJO0Al0WN0=', 'Content-Length': '0'} 2022-09-12T19:48:08.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b4d570f37e9dfb3-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.440 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:08.441 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:08.441 DEBUG:boto:path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:08.442 DEBUG:boto:auth_path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:08.442 DEBUG:boto:Method: GET 2022-09-12T19:48:08.443 DEBUG:boto:Path: /jdpiak-72/f?versionId=null 2022-09-12T19:48:08.443 DEBUG:boto:Data: 2022-09-12T19:48:08.444 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.444 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.445 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.446 DEBUG:boto:Params: {} 2022-09-12T19:48:08.447 DEBUG:boto:Token: None 2022-09-12T19:48:08.447 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/f?versionId=null 2022-09-12T19:48:08.448 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fBo6UK3e6mvAIUezPUJO0Al0WN0= 2022-09-12T19:48:08.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fBo6UK3e6mvAIUezPUJO0Al0WN0=', 'Content-Length': '0'} 2022-09-12T19:48:08.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f2c68431c30bd94-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.498 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:08.499 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:08.500 DEBUG:boto:path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:08.501 DEBUG:boto:auth_path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:08.502 DEBUG:boto:Method: HEAD 2022-09-12T19:48:08.504 DEBUG:boto:Path: /jdpiak-72/f?versionId=null 2022-09-12T19:48:08.505 DEBUG:boto:Data: 2022-09-12T19:48:08.506 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.507 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.508 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.509 DEBUG:boto:Params: {} 2022-09-12T19:48:08.510 DEBUG:boto:Token: None 2022-09-12T19:48:08.511 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/f?versionId=null 2022-09-12T19:48:08.512 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ErYAQ1Kz2lGBnqvN/nJey5MzVTk= 2022-09-12T19:48:08.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ErYAQ1Kz2lGBnqvN/nJey5MzVTk=', 'Content-Length': '0'} 2022-09-12T19:48:08.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b7fc8a7969fbf7c-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.518 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:08.518 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:08.519 DEBUG:boto:path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:08.519 DEBUG:boto:auth_path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:08.520 DEBUG:boto:Method: HEAD 2022-09-12T19:48:08.520 DEBUG:boto:Path: /jdpiak-72/f?versionId=null 2022-09-12T19:48:08.521 DEBUG:boto:Data: 2022-09-12T19:48:08.521 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.522 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.522 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.523 DEBUG:boto:Params: {} 2022-09-12T19:48:08.523 DEBUG:boto:Token: None 2022-09-12T19:48:08.524 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/f?versionId=null 2022-09-12T19:48:08.524 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ErYAQ1Kz2lGBnqvN/nJey5MzVTk= 2022-09-12T19:48:08.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ErYAQ1Kz2lGBnqvN/nJey5MzVTk=', 'Content-Length': '0'} 2022-09-12T19:48:08.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd57386a7d183166-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:48:08.528 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:08.529 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:08.530 DEBUG:boto:path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:08.530 DEBUG:boto:auth_path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:08.531 DEBUG:boto:Method: GET 2022-09-12T19:48:08.531 DEBUG:boto:Path: /jdpiak-72/f?versionId=null 2022-09-12T19:48:08.532 DEBUG:boto:Data: 2022-09-12T19:48:08.532 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.533 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.533 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.534 DEBUG:boto:Params: {} 2022-09-12T19:48:08.534 DEBUG:boto:Token: None 2022-09-12T19:48:08.535 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/f?versionId=null 2022-09-12T19:48:08.536 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fBo6UK3e6mvAIUezPUJO0Al0WN0= 2022-09-12T19:48:08.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fBo6UK3e6mvAIUezPUJO0Al0WN0=', 'Content-Length': '0'} 2022-09-12T19:48:08.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009725da488fa69909-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.539 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:08.540 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:08.540 DEBUG:boto:path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:08.541 DEBUG:boto:auth_path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:08.541 DEBUG:boto:Method: GET 2022-09-12T19:48:08.542 DEBUG:boto:Path: /jdpiak-72/f?versionId=null 2022-09-12T19:48:08.542 DEBUG:boto:Data: 2022-09-12T19:48:08.543 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.543 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.544 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.544 DEBUG:boto:Params: {} 2022-09-12T19:48:08.545 DEBUG:boto:Token: None 2022-09-12T19:48:08.546 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/f?versionId=null 2022-09-12T19:48:08.546 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fBo6UK3e6mvAIUezPUJO0Al0WN0= 2022-09-12T19:48:08.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fBo6UK3e6mvAIUezPUJO0Al0WN0=', 'Content-Length': '0'} 2022-09-12T19:48:08.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b930bd68790604da-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.550 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:08.551 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:08.551 DEBUG:boto:path=/jdpiak-72/?prefix=f 2022-09-12T19:48:08.552 DEBUG:boto:auth_path=/jdpiak-72/?prefix=f 2022-09-12T19:48:08.552 DEBUG:boto:Method: GET 2022-09-12T19:48:08.553 DEBUG:boto:Path: /jdpiak-72/?prefix=f 2022-09-12T19:48:08.553 DEBUG:boto:Data: 2022-09-12T19:48:08.554 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.554 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.555 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.555 DEBUG:boto:Params: {} 2022-09-12T19:48:08.556 DEBUG:boto:Token: None 2022-09-12T19:48:08.557 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/ 2022-09-12T19:48:08.557 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8= 2022-09-12T19:48:08.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8=', 'Content-Length': '0'} 2022-09-12T19:48:08.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d07b43f75774a57-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.602 DEBUG:boto:b'jdpiak-72f1000falsef2022-09-12T19:45:20.891Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:08.603 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:08.604 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:08.604 DEBUG:boto:path=/jdpiak-72/?prefix=f 2022-09-12T19:48:08.605 DEBUG:boto:auth_path=/jdpiak-72/?prefix=f 2022-09-12T19:48:08.605 DEBUG:boto:Method: GET 2022-09-12T19:48:08.606 DEBUG:boto:Path: /jdpiak-72/?prefix=f 2022-09-12T19:48:08.606 DEBUG:boto:Data: 2022-09-12T19:48:08.607 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.607 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.608 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.609 DEBUG:boto:Params: {} 2022-09-12T19:48:08.609 DEBUG:boto:Token: None 2022-09-12T19:48:08.610 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/ 2022-09-12T19:48:08.610 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8= 2022-09-12T19:48:08.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8=', 'Content-Length': '0'} 2022-09-12T19:48:08.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031e46f6ed10eb98f-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.614 DEBUG:boto:b'jdpiak-72f1000falsef2022-09-12T19:45:20.891Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:08.615 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:08.615 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:08.617 DEBUG:boto:Method: HEAD 2022-09-12T19:48:08.617 DEBUG:boto:Path: /jdpiak-72/f 2022-09-12T19:48:08.627 DEBUG:boto:Data: 2022-09-12T19:48:08.628 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.628 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.630 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.630 DEBUG:boto:Params: {} 2022-09-12T19:48:08.632 DEBUG:boto:Token: None 2022-09-12T19:48:08.632 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/f 2022-09-12T19:48:08.633 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RcVNGktWfA/cFZNdcX45zl5AqXU= 2022-09-12T19:48:08.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RcVNGktWfA/cFZNdcX45zl5AqXU=', 'Content-Length': '0'} 2022-09-12T19:48:08.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001b45d3dc9d9afac-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.638 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:08.639 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:08.640 DEBUG:boto:Method: HEAD 2022-09-12T19:48:08.641 DEBUG:boto:Path: /jdpiak-72/f 2022-09-12T19:48:08.642 DEBUG:boto:Data: 2022-09-12T19:48:08.643 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.644 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.645 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.646 DEBUG:boto:Params: {} 2022-09-12T19:48:08.647 DEBUG:boto:Token: None 2022-09-12T19:48:08.648 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/f 2022-09-12T19:48:08.649 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RcVNGktWfA/cFZNdcX45zl5AqXU= 2022-09-12T19:48:08.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RcVNGktWfA/cFZNdcX45zl5AqXU=', 'Content-Length': '0'} 2022-09-12T19:48:08.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a709eda25d7c55f-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:48:08.655 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:08.656 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:08.657 DEBUG:boto:Method: GET 2022-09-12T19:48:08.657 DEBUG:boto:Path: /jdpiak-72/f 2022-09-12T19:48:08.658 DEBUG:boto:Data: 2022-09-12T19:48:08.659 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.660 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.661 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.662 DEBUG:boto:Params: {} 2022-09-12T19:48:08.663 DEBUG:boto:Token: None 2022-09-12T19:48:08.664 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/f 2022-09-12T19:48:08.664 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5RVamJ77HecrEB2HlnL51PhwQi0= 2022-09-12T19:48:08.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5RVamJ77HecrEB2HlnL51PhwQi0=', 'Content-Length': '0'} 2022-09-12T19:48:08.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ba722a9850e261a-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.668 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:08.669 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:08.670 DEBUG:boto:Method: GET 2022-09-12T19:48:08.670 DEBUG:boto:Path: /jdpiak-72/f 2022-09-12T19:48:08.671 DEBUG:boto:Data: 2022-09-12T19:48:08.671 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.672 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.673 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.673 DEBUG:boto:Params: {} 2022-09-12T19:48:08.674 DEBUG:boto:Token: None 2022-09-12T19:48:08.675 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/f 2022-09-12T19:48:08.675 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5RVamJ77HecrEB2HlnL51PhwQi0= 2022-09-12T19:48:08.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5RVamJ77HecrEB2HlnL51PhwQi0=', 'Content-Length': '0'} 2022-09-12T19:48:08.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013ed08460dc08c52-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.681 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:08.682 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:08.683 DEBUG:boto:path=/jdpiak-72/?prefix=f 2022-09-12T19:48:08.684 DEBUG:boto:auth_path=/jdpiak-72/?prefix=f 2022-09-12T19:48:08.684 DEBUG:boto:Method: GET 2022-09-12T19:48:08.685 DEBUG:boto:Path: /jdpiak-72/?prefix=f 2022-09-12T19:48:08.686 DEBUG:boto:Data: 2022-09-12T19:48:08.687 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.687 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.688 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.688 DEBUG:boto:Params: {} 2022-09-12T19:48:08.689 DEBUG:boto:Token: None 2022-09-12T19:48:08.690 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/ 2022-09-12T19:48:08.690 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8= 2022-09-12T19:48:08.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8=', 'Content-Length': '0'} 2022-09-12T19:48:08.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0036240dc450fa5-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.694 DEBUG:boto:b'jdpiak-72f1000falsef2022-09-12T19:45:20.891Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:08.694 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:08.695 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:08.696 DEBUG:boto:path=/jdpiak-72/?prefix=f 2022-09-12T19:48:08.697 DEBUG:boto:auth_path=/jdpiak-72/?prefix=f 2022-09-12T19:48:08.698 DEBUG:boto:Method: GET 2022-09-12T19:48:08.698 DEBUG:boto:Path: /jdpiak-72/?prefix=f 2022-09-12T19:48:08.699 DEBUG:boto:Data: 2022-09-12T19:48:08.699 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.700 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.701 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.701 DEBUG:boto:Params: {} 2022-09-12T19:48:08.702 DEBUG:boto:Token: None 2022-09-12T19:48:08.702 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/ 2022-09-12T19:48:08.703 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8= 2022-09-12T19:48:08.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8=', 'Content-Length': '0'} 2022-09-12T19:48:08.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000516e16dfdf8896df-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.709 DEBUG:boto:b'jdpiak-72f1000falsef2022-09-12T19:45:20.891Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:08.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:48:08.710 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:08.711 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:08.711 DEBUG:boto:path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:08.712 DEBUG:boto:auth_path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:08.713 DEBUG:boto:Method: GET 2022-09-12T19:48:08.713 DEBUG:boto:Path: /jdpiak-72/g?versionId=null 2022-09-12T19:48:08.714 DEBUG:boto:Data: 2022-09-12T19:48:08.715 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.715 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.716 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.716 DEBUG:boto:Params: {} 2022-09-12T19:48:08.717 DEBUG:boto:Token: None 2022-09-12T19:48:08.717 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/g?versionId=null 2022-09-12T19:48:08.718 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:n0s7DKBk0VNuWmkZHSnch0vzKEw= 2022-09-12T19:48:08.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:n0s7DKBk0VNuWmkZHSnch0vzKEw=', 'Content-Length': '0'} 2022-09-12T19:48:08.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047b8bb784355e8d9-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.722 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:08.722 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:08.723 DEBUG:boto:path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:08.724 DEBUG:boto:auth_path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:08.724 DEBUG:boto:Method: GET 2022-09-12T19:48:08.725 DEBUG:boto:Path: /jdpiak-72/g?versionId=null 2022-09-12T19:48:08.725 DEBUG:boto:Data: 2022-09-12T19:48:08.727 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.727 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.728 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.728 DEBUG:boto:Params: {} 2022-09-12T19:48:08.729 DEBUG:boto:Token: None 2022-09-12T19:48:08.729 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/g?versionId=null 2022-09-12T19:48:08.730 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:n0s7DKBk0VNuWmkZHSnch0vzKEw= 2022-09-12T19:48:08.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:n0s7DKBk0VNuWmkZHSnch0vzKEw=', 'Content-Length': '0'} 2022-09-12T19:48:08.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a24fd7afc68999d-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.734 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:08.734 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:08.735 DEBUG:boto:path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:08.735 DEBUG:boto:auth_path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:08.736 DEBUG:boto:Method: HEAD 2022-09-12T19:48:08.736 DEBUG:boto:Path: /jdpiak-72/g?versionId=null 2022-09-12T19:48:08.737 DEBUG:boto:Data: 2022-09-12T19:48:08.737 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.738 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.738 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.739 DEBUG:boto:Params: {} 2022-09-12T19:48:08.739 DEBUG:boto:Token: None 2022-09-12T19:48:08.740 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/g?versionId=null 2022-09-12T19:48:08.740 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ne95PuFSefU1qXKJf5BFZMGlihc= 2022-09-12T19:48:08.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ne95PuFSefU1qXKJf5BFZMGlihc=', 'Content-Length': '0'} 2022-09-12T19:48:08.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006391ac93536cc9dd-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.744 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:08.744 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:08.745 DEBUG:boto:path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:08.746 DEBUG:boto:auth_path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:08.746 DEBUG:boto:Method: HEAD 2022-09-12T19:48:08.747 DEBUG:boto:Path: /jdpiak-72/g?versionId=null 2022-09-12T19:48:08.747 DEBUG:boto:Data: 2022-09-12T19:48:08.748 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.748 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.749 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.749 DEBUG:boto:Params: {} 2022-09-12T19:48:08.750 DEBUG:boto:Token: None 2022-09-12T19:48:08.750 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/g?versionId=null 2022-09-12T19:48:08.751 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ne95PuFSefU1qXKJf5BFZMGlihc= 2022-09-12T19:48:08.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ne95PuFSefU1qXKJf5BFZMGlihc=', 'Content-Length': '0'} 2022-09-12T19:48:08.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000255b3fad7df6da44-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:48:08.755 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:08.755 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:08.756 DEBUG:boto:path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:08.757 DEBUG:boto:auth_path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:08.757 DEBUG:boto:Method: GET 2022-09-12T19:48:08.758 DEBUG:boto:Path: /jdpiak-72/g?versionId=null 2022-09-12T19:48:08.758 DEBUG:boto:Data: 2022-09-12T19:48:08.759 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.759 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.760 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.760 DEBUG:boto:Params: {} 2022-09-12T19:48:08.761 DEBUG:boto:Token: None 2022-09-12T19:48:08.762 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/g?versionId=null 2022-09-12T19:48:08.762 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:n0s7DKBk0VNuWmkZHSnch0vzKEw= 2022-09-12T19:48:08.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:n0s7DKBk0VNuWmkZHSnch0vzKEw=', 'Content-Length': '0'} 2022-09-12T19:48:08.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2fdb619f188643e-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.766 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:08.767 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:08.768 DEBUG:boto:path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:08.768 DEBUG:boto:auth_path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:08.769 DEBUG:boto:Method: GET 2022-09-12T19:48:08.769 DEBUG:boto:Path: /jdpiak-72/g?versionId=null 2022-09-12T19:48:08.770 DEBUG:boto:Data: 2022-09-12T19:48:08.770 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.771 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.771 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.772 DEBUG:boto:Params: {} 2022-09-12T19:48:08.773 DEBUG:boto:Token: None 2022-09-12T19:48:08.773 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/g?versionId=null 2022-09-12T19:48:08.774 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:n0s7DKBk0VNuWmkZHSnch0vzKEw= 2022-09-12T19:48:08.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:n0s7DKBk0VNuWmkZHSnch0vzKEw=', 'Content-Length': '0'} 2022-09-12T19:48:08.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae004ee474787b38-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.778 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:08.779 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:08.780 DEBUG:boto:path=/jdpiak-72/?prefix=g 2022-09-12T19:48:08.780 DEBUG:boto:auth_path=/jdpiak-72/?prefix=g 2022-09-12T19:48:08.781 DEBUG:boto:Method: GET 2022-09-12T19:48:08.781 DEBUG:boto:Path: /jdpiak-72/?prefix=g 2022-09-12T19:48:08.782 DEBUG:boto:Data: 2022-09-12T19:48:08.782 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.783 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.783 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.784 DEBUG:boto:Params: {} 2022-09-12T19:48:08.785 DEBUG:boto:Token: None 2022-09-12T19:48:08.785 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/ 2022-09-12T19:48:08.786 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8= 2022-09-12T19:48:08.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8=', 'Content-Length': '0'} 2022-09-12T19:48:08.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ee376b69381c04c-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.789 DEBUG:boto:b'jdpiak-72g1000falseg2022-09-12T19:45:20.920Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:08.790 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:08.791 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:08.791 DEBUG:boto:path=/jdpiak-72/?prefix=g 2022-09-12T19:48:08.792 DEBUG:boto:auth_path=/jdpiak-72/?prefix=g 2022-09-12T19:48:08.793 DEBUG:boto:Method: GET 2022-09-12T19:48:08.793 DEBUG:boto:Path: /jdpiak-72/?prefix=g 2022-09-12T19:48:08.794 DEBUG:boto:Data: 2022-09-12T19:48:08.794 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.795 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.795 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.796 DEBUG:boto:Params: {} 2022-09-12T19:48:08.797 DEBUG:boto:Token: None 2022-09-12T19:48:08.797 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/ 2022-09-12T19:48:08.798 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8= 2022-09-12T19:48:08.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8=', 'Content-Length': '0'} 2022-09-12T19:48:08.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c8d9dd698b4b207-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.803 DEBUG:boto:b'jdpiak-72g1000falseg2022-09-12T19:45:20.920Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:08.804 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:08.805 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:08.806 DEBUG:boto:Method: HEAD 2022-09-12T19:48:08.806 DEBUG:boto:Path: /jdpiak-72/g 2022-09-12T19:48:08.807 DEBUG:boto:Data: 2022-09-12T19:48:08.807 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.808 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.808 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.809 DEBUG:boto:Params: {} 2022-09-12T19:48:08.809 DEBUG:boto:Token: None 2022-09-12T19:48:08.810 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/g 2022-09-12T19:48:08.811 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zo9GJZR+1ZFI0hX0a0BEh2yOX1c= 2022-09-12T19:48:08.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zo9GJZR+1ZFI0hX0a0BEh2yOX1c=', 'Content-Length': '0'} 2022-09-12T19:48:08.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000163fc66710f6673e-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.814 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:08.814 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:08.815 DEBUG:boto:Method: HEAD 2022-09-12T19:48:08.816 DEBUG:boto:Path: /jdpiak-72/g 2022-09-12T19:48:08.816 DEBUG:boto:Data: 2022-09-12T19:48:08.817 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.817 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.818 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.818 DEBUG:boto:Params: {} 2022-09-12T19:48:08.819 DEBUG:boto:Token: None 2022-09-12T19:48:08.819 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/g 2022-09-12T19:48:08.820 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zo9GJZR+1ZFI0hX0a0BEh2yOX1c= 2022-09-12T19:48:08.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zo9GJZR+1ZFI0hX0a0BEh2yOX1c=', 'Content-Length': '0'} 2022-09-12T19:48:08.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edff59d976ce0ba1-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:48:08.824 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:08.824 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:08.825 DEBUG:boto:Method: GET 2022-09-12T19:48:08.826 DEBUG:boto:Path: /jdpiak-72/g 2022-09-12T19:48:08.826 DEBUG:boto:Data: 2022-09-12T19:48:08.827 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.827 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.828 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.829 DEBUG:boto:Params: {} 2022-09-12T19:48:08.829 DEBUG:boto:Token: None 2022-09-12T19:48:08.830 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/g 2022-09-12T19:48:08.831 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:peOIa2lTLnh7KdqLX1wz+PB44nk= 2022-09-12T19:48:08.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:peOIa2lTLnh7KdqLX1wz+PB44nk=', 'Content-Length': '0'} 2022-09-12T19:48:08.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aea816427ffd3710-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.834 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:08.835 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:08.835 DEBUG:boto:Method: GET 2022-09-12T19:48:08.836 DEBUG:boto:Path: /jdpiak-72/g 2022-09-12T19:48:08.836 DEBUG:boto:Data: 2022-09-12T19:48:08.837 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.837 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.838 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.838 DEBUG:boto:Params: {} 2022-09-12T19:48:08.839 DEBUG:boto:Token: None 2022-09-12T19:48:08.840 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/g 2022-09-12T19:48:08.840 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:peOIa2lTLnh7KdqLX1wz+PB44nk= 2022-09-12T19:48:08.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:peOIa2lTLnh7KdqLX1wz+PB44nk=', 'Content-Length': '0'} 2022-09-12T19:48:08.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cabe4ebffd6b857-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.886 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:08.887 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:08.888 DEBUG:boto:path=/jdpiak-72/?prefix=g 2022-09-12T19:48:08.889 DEBUG:boto:auth_path=/jdpiak-72/?prefix=g 2022-09-12T19:48:08.890 DEBUG:boto:Method: GET 2022-09-12T19:48:08.891 DEBUG:boto:Path: /jdpiak-72/?prefix=g 2022-09-12T19:48:08.892 DEBUG:boto:Data: 2022-09-12T19:48:08.893 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.894 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.895 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.896 DEBUG:boto:Params: {} 2022-09-12T19:48:08.897 DEBUG:boto:Token: None 2022-09-12T19:48:08.898 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/ 2022-09-12T19:48:08.899 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8= 2022-09-12T19:48:08.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8=', 'Content-Length': '0'} 2022-09-12T19:48:08.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1907176ddd3c6c9-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.904 DEBUG:boto:b'jdpiak-72g1000falseg2022-09-12T19:45:20.920Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:08.906 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:08.907 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:08.908 DEBUG:boto:path=/jdpiak-72/?prefix=g 2022-09-12T19:48:08.909 DEBUG:boto:auth_path=/jdpiak-72/?prefix=g 2022-09-12T19:48:08.910 DEBUG:boto:Method: GET 2022-09-12T19:48:08.911 DEBUG:boto:Path: /jdpiak-72/?prefix=g 2022-09-12T19:48:08.912 DEBUG:boto:Data: 2022-09-12T19:48:08.913 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.913 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.914 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.915 DEBUG:boto:Params: {} 2022-09-12T19:48:08.917 DEBUG:boto:Token: None 2022-09-12T19:48:08.918 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/ 2022-09-12T19:48:08.919 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8= 2022-09-12T19:48:08.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ekNXu7xfBoJVm/kl9WdloXhTR8=', 'Content-Length': '0'} 2022-09-12T19:48:08.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009450f10acac0b1db-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.924 DEBUG:boto:b'jdpiak-72g1000falseg2022-09-12T19:45:20.920Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:08.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:48:08.927 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:08.928 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:08.929 DEBUG:boto:path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:08.930 DEBUG:boto:auth_path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:08.930 DEBUG:boto:Method: GET 2022-09-12T19:48:08.931 DEBUG:boto:Path: /jdpiak-72/h?versionId=null 2022-09-12T19:48:08.932 DEBUG:boto:Data: 2022-09-12T19:48:08.933 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.934 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.935 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.936 DEBUG:boto:Params: {} 2022-09-12T19:48:08.937 DEBUG:boto:Token: None 2022-09-12T19:48:08.938 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/h?versionId=null 2022-09-12T19:48:08.940 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WgXswSqelN2LbUrKVh0NnRgMP9E= 2022-09-12T19:48:08.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WgXswSqelN2LbUrKVh0NnRgMP9E=', 'Content-Length': '0'} 2022-09-12T19:48:08.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e6ec943fd0667b5-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.945 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:08.946 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:08.947 DEBUG:boto:path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:08.948 DEBUG:boto:auth_path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:08.949 DEBUG:boto:Method: GET 2022-09-12T19:48:08.950 DEBUG:boto:Path: /jdpiak-72/h?versionId=null 2022-09-12T19:48:08.951 DEBUG:boto:Data: 2022-09-12T19:48:08.952 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.953 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.954 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.955 DEBUG:boto:Params: {} 2022-09-12T19:48:08.956 DEBUG:boto:Token: None 2022-09-12T19:48:08.957 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/h?versionId=null 2022-09-12T19:48:08.958 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WgXswSqelN2LbUrKVh0NnRgMP9E= 2022-09-12T19:48:08.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WgXswSqelN2LbUrKVh0NnRgMP9E=', 'Content-Length': '0'} 2022-09-12T19:48:08.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ca3c24813c1a17c-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.964 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:08.965 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:08.965 DEBUG:boto:path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:08.966 DEBUG:boto:auth_path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:08.966 DEBUG:boto:Method: HEAD 2022-09-12T19:48:08.967 DEBUG:boto:Path: /jdpiak-72/h?versionId=null 2022-09-12T19:48:08.968 DEBUG:boto:Data: 2022-09-12T19:48:08.968 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.969 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.969 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.970 DEBUG:boto:Params: {} 2022-09-12T19:48:08.970 DEBUG:boto:Token: None 2022-09-12T19:48:08.971 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/h?versionId=null 2022-09-12T19:48:08.971 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NeHvPdqSXjiJiQWS2eYboMaDrLo= 2022-09-12T19:48:08.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NeHvPdqSXjiJiQWS2eYboMaDrLo=', 'Content-Length': '0'} 2022-09-12T19:48:08.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad5e8c78ff3944ff-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.975 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:08.975 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:08.976 DEBUG:boto:path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:08.977 DEBUG:boto:auth_path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:08.977 DEBUG:boto:Method: HEAD 2022-09-12T19:48:08.978 DEBUG:boto:Path: /jdpiak-72/h?versionId=null 2022-09-12T19:48:08.978 DEBUG:boto:Data: 2022-09-12T19:48:08.979 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.979 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:08.980 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.980 DEBUG:boto:Params: {} 2022-09-12T19:48:08.981 DEBUG:boto:Token: None 2022-09-12T19:48:08.981 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/h?versionId=null 2022-09-12T19:48:08.982 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NeHvPdqSXjiJiQWS2eYboMaDrLo= 2022-09-12T19:48:08.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NeHvPdqSXjiJiQWS2eYboMaDrLo=', 'Content-Length': '0'} 2022-09-12T19:48:08.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2d6fdc0b601e16c-00631f8cf8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:08.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:48:08.987 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:08.988 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:08.988 DEBUG:boto:path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:08.989 DEBUG:boto:auth_path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:08.989 DEBUG:boto:Method: GET 2022-09-12T19:48:08.990 DEBUG:boto:Path: /jdpiak-72/h?versionId=null 2022-09-12T19:48:08.991 DEBUG:boto:Data: 2022-09-12T19:48:08.992 DEBUG:boto:Headers: {} 2022-09-12T19:48:08.993 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:08.994 DEBUG:boto:Port: 8000 2022-09-12T19:48:08.995 DEBUG:boto:Params: {} 2022-09-12T19:48:08.996 DEBUG:boto:Token: None 2022-09-12T19:48:08.997 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:08 GMT /jdpiak-72/h?versionId=null 2022-09-12T19:48:08.998 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WgXswSqelN2LbUrKVh0NnRgMP9E= 2022-09-12T19:48:08.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WgXswSqelN2LbUrKVh0NnRgMP9E=', 'Content-Length': '0'} 2022-09-12T19:48:09.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081503747ed7be159-00631f8cf8-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.003 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:09.004 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:09.004 DEBUG:boto:path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:09.005 DEBUG:boto:auth_path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:09.005 DEBUG:boto:Method: GET 2022-09-12T19:48:09.006 DEBUG:boto:Path: /jdpiak-72/h?versionId=null 2022-09-12T19:48:09.007 DEBUG:boto:Data: 2022-09-12T19:48:09.007 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.008 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.009 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.009 DEBUG:boto:Params: {} 2022-09-12T19:48:09.010 DEBUG:boto:Token: None 2022-09-12T19:48:09.011 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/h?versionId=null 2022-09-12T19:48:09.012 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gM2Qf+U1/IpoS+4jO9maVXessCU= 2022-09-12T19:48:09.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gM2Qf+U1/IpoS+4jO9maVXessCU=', 'Content-Length': '0'} 2022-09-12T19:48:09.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1e2408f9273d1d0-00631f8cf9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.062 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:09.063 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:09.064 DEBUG:boto:path=/jdpiak-72/?prefix=h 2022-09-12T19:48:09.064 DEBUG:boto:auth_path=/jdpiak-72/?prefix=h 2022-09-12T19:48:09.065 DEBUG:boto:Method: GET 2022-09-12T19:48:09.066 DEBUG:boto:Path: /jdpiak-72/?prefix=h 2022-09-12T19:48:09.066 DEBUG:boto:Data: 2022-09-12T19:48:09.067 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.067 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:09.068 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.068 DEBUG:boto:Params: {} 2022-09-12T19:48:09.069 DEBUG:boto:Token: None 2022-09-12T19:48:09.070 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/ 2022-09-12T19:48:09.071 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk= 2022-09-12T19:48:09.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk=', 'Content-Length': '0'} 2022-09-12T19:48:09.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5e6f5d14491116d-00631f8cf9-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.077 DEBUG:boto:b'jdpiak-72h1000falseh2022-09-12T19:45:20.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:09.079 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:09.080 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:09.081 DEBUG:boto:path=/jdpiak-72/?prefix=h 2022-09-12T19:48:09.082 DEBUG:boto:auth_path=/jdpiak-72/?prefix=h 2022-09-12T19:48:09.083 DEBUG:boto:Method: GET 2022-09-12T19:48:09.084 DEBUG:boto:Path: /jdpiak-72/?prefix=h 2022-09-12T19:48:09.085 DEBUG:boto:Data: 2022-09-12T19:48:09.086 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.087 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.087 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.088 DEBUG:boto:Params: {} 2022-09-12T19:48:09.089 DEBUG:boto:Token: None 2022-09-12T19:48:09.090 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/ 2022-09-12T19:48:09.091 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk= 2022-09-12T19:48:09.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk=', 'Content-Length': '0'} 2022-09-12T19:48:09.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8bd75f05a21fa4c-00631f8cf9-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.095 DEBUG:boto:b'jdpiak-72h1000falseh2022-09-12T19:45:20.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:09.096 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:09.097 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:09.097 DEBUG:boto:Method: HEAD 2022-09-12T19:48:09.098 DEBUG:boto:Path: /jdpiak-72/h 2022-09-12T19:48:09.098 DEBUG:boto:Data: 2022-09-12T19:48:09.099 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.100 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:09.100 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.101 DEBUG:boto:Params: {} 2022-09-12T19:48:09.101 DEBUG:boto:Token: None 2022-09-12T19:48:09.102 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/h 2022-09-12T19:48:09.103 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1gVLI9SD/F4gYavtyiU4ZThdkZs= 2022-09-12T19:48:09.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1gVLI9SD/F4gYavtyiU4ZThdkZs=', 'Content-Length': '0'} 2022-09-12T19:48:09.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005848009ddd2268d2-00631f8cf9-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.107 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:09.107 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:09.108 DEBUG:boto:Method: HEAD 2022-09-12T19:48:09.108 DEBUG:boto:Path: /jdpiak-72/h 2022-09-12T19:48:09.109 DEBUG:boto:Data: 2022-09-12T19:48:09.109 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.110 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.111 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.111 DEBUG:boto:Params: {} 2022-09-12T19:48:09.113 DEBUG:boto:Token: None 2022-09-12T19:48:09.114 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/h 2022-09-12T19:48:09.115 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1gVLI9SD/F4gYavtyiU4ZThdkZs= 2022-09-12T19:48:09.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1gVLI9SD/F4gYavtyiU4ZThdkZs=', 'Content-Length': '0'} 2022-09-12T19:48:09.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b3178108cbf8096-00631f8cf9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:48:09.121 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:09.121 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:09.122 DEBUG:boto:Method: GET 2022-09-12T19:48:09.122 DEBUG:boto:Path: /jdpiak-72/h 2022-09-12T19:48:09.123 DEBUG:boto:Data: 2022-09-12T19:48:09.123 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.124 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:09.124 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.125 DEBUG:boto:Params: {} 2022-09-12T19:48:09.126 DEBUG:boto:Token: None 2022-09-12T19:48:09.126 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/h 2022-09-12T19:48:09.127 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IAo+RVX2I+mwN2Czf67cnMq/kCo= 2022-09-12T19:48:09.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IAo+RVX2I+mwN2Czf67cnMq/kCo=', 'Content-Length': '0'} 2022-09-12T19:48:09.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026691937e2495b3c-00631f8cf9-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.132 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:09.133 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:09.134 DEBUG:boto:Method: GET 2022-09-12T19:48:09.134 DEBUG:boto:Path: /jdpiak-72/h 2022-09-12T19:48:09.135 DEBUG:boto:Data: 2022-09-12T19:48:09.135 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.136 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.136 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.137 DEBUG:boto:Params: {} 2022-09-12T19:48:09.137 DEBUG:boto:Token: None 2022-09-12T19:48:09.138 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/h 2022-09-12T19:48:09.139 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IAo+RVX2I+mwN2Czf67cnMq/kCo= 2022-09-12T19:48:09.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IAo+RVX2I+mwN2Czf67cnMq/kCo=', 'Content-Length': '0'} 2022-09-12T19:48:09.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d89847cd08c616ea-00631f8cf9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.143 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:09.143 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:09.144 DEBUG:boto:path=/jdpiak-72/?prefix=h 2022-09-12T19:48:09.144 DEBUG:boto:auth_path=/jdpiak-72/?prefix=h 2022-09-12T19:48:09.145 DEBUG:boto:Method: GET 2022-09-12T19:48:09.145 DEBUG:boto:Path: /jdpiak-72/?prefix=h 2022-09-12T19:48:09.146 DEBUG:boto:Data: 2022-09-12T19:48:09.147 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.147 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:09.148 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.148 DEBUG:boto:Params: {} 2022-09-12T19:48:09.149 DEBUG:boto:Token: None 2022-09-12T19:48:09.149 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/ 2022-09-12T19:48:09.150 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk= 2022-09-12T19:48:09.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk=', 'Content-Length': '0'} 2022-09-12T19:48:09.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097e9c21607e5c3b9-00631f8cf9-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.154 DEBUG:boto:b'jdpiak-72h1000falseh2022-09-12T19:45:20.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:09.155 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:09.156 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:09.157 DEBUG:boto:path=/jdpiak-72/?prefix=h 2022-09-12T19:48:09.158 DEBUG:boto:auth_path=/jdpiak-72/?prefix=h 2022-09-12T19:48:09.159 DEBUG:boto:Method: GET 2022-09-12T19:48:09.160 DEBUG:boto:Path: /jdpiak-72/?prefix=h 2022-09-12T19:48:09.161 DEBUG:boto:Data: 2022-09-12T19:48:09.161 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.162 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.163 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.164 DEBUG:boto:Params: {} 2022-09-12T19:48:09.165 DEBUG:boto:Token: None 2022-09-12T19:48:09.165 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/ 2022-09-12T19:48:09.166 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk= 2022-09-12T19:48:09.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk=', 'Content-Length': '0'} 2022-09-12T19:48:09.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b88b0aa27ea504f4-00631f8cf9-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.171 DEBUG:boto:b'jdpiak-72h1000falseh2022-09-12T19:45:20.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:09.172 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-72 zones={test-zone2, test-zone1} 2022-09-12T19:48:09.173 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-72 zones={test-zone3, test-zone1} 2022-09-12T19:48:09.174 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:09.175 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:09.176 DEBUG:boto:Method: HEAD 2022-09-12T19:48:09.176 DEBUG:boto:Path: /jdpiak-72/ 2022-09-12T19:48:09.177 DEBUG:boto:Data: 2022-09-12T19:48:09.177 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.178 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:09.178 DEBUG:boto:Port: 8001 2022-09-12T19:48:09.179 DEBUG:boto:Params: {} 2022-09-12T19:48:09.180 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T19:48:09.180 DEBUG:boto:Token: None 2022-09-12T19:48:09.181 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/ 2022-09-12T19:48:09.181 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rcDpEVNWaYRFte6sELBRxUqJDy8= 2022-09-12T19:48:09.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rcDpEVNWaYRFte6sELBRxUqJDy8=', 'Content-Length': '0'} 2022-09-12T19:48:09.186 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df60fea0b4597328-00631f8cf9-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.187 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:09.187 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:09.188 DEBUG:boto:Method: HEAD 2022-09-12T19:48:09.188 DEBUG:boto:Path: /jdpiak-72/ 2022-09-12T19:48:09.189 DEBUG:boto:Data: 2022-09-12T19:48:09.189 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.190 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.190 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.191 DEBUG:boto:Params: {} 2022-09-12T19:48:09.191 DEBUG:boto:Token: None 2022-09-12T19:48:09.192 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/ 2022-09-12T19:48:09.193 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rcDpEVNWaYRFte6sELBRxUqJDy8= 2022-09-12T19:48:09.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rcDpEVNWaYRFte6sELBRxUqJDy8=', 'Content-Length': '0'} 2022-09-12T19:48:09.195 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c51d0459d3a0690b-00631f8cf9-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.196 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:48:09.197 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:09.197 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:09.198 DEBUG:boto:path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:09.198 DEBUG:boto:auth_path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:09.199 DEBUG:boto:Method: GET 2022-09-12T19:48:09.199 DEBUG:boto:Path: /jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:09.200 DEBUG:boto:Data: 2022-09-12T19:48:09.200 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.201 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:09.201 DEBUG:boto:Port: 8001 2022-09-12T19:48:09.202 DEBUG:boto:Params: {} 2022-09-12T19:48:09.203 DEBUG:boto:Token: None 2022-09-12T19:48:09.203 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/?versions 2022-09-12T19:48:09.204 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e+mz8OTcuyXMbAt8pB0mWWDL2Mk= 2022-09-12T19:48:09.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e+mz8OTcuyXMbAt8pB0mWWDL2Mk=', 'Content-Length': '0'} 2022-09-12T19:48:09.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029074ec01969b0e1-00631f8cf9-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.209 DEBUG:boto:b'jdpiak-72999falseanulltrue2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:45:20.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:45:20.891Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:45:20.920Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:45:20.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:09.211 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:48:09.212 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:48:09.213 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:48:09.214 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:48:09.215 DEBUG:tasks.rgw_multi.zone_rados:o=e 2022-09-12T19:48:09.216 DEBUG:tasks.rgw_multi.zone_rados:o=f 2022-09-12T19:48:09.216 DEBUG:tasks.rgw_multi.zone_rados:o=g 2022-09-12T19:48:09.217 DEBUG:tasks.rgw_multi.zone_rados:o=h 2022-09-12T19:48:09.218 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:48:09.219 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:09.219 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:09.220 DEBUG:boto:path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:09.221 DEBUG:boto:auth_path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:09.221 DEBUG:boto:Method: GET 2022-09-12T19:48:09.222 DEBUG:boto:Path: /jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:09.222 DEBUG:boto:Data: 2022-09-12T19:48:09.223 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.223 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.224 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.224 DEBUG:boto:Params: {} 2022-09-12T19:48:09.225 DEBUG:boto:Token: None 2022-09-12T19:48:09.226 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/?versions 2022-09-12T19:48:09.227 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e+mz8OTcuyXMbAt8pB0mWWDL2Mk= 2022-09-12T19:48:09.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e+mz8OTcuyXMbAt8pB0mWWDL2Mk=', 'Content-Length': '0'} 2022-09-12T19:48:09.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060094301fff90f54-00631f8cf9-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.231 DEBUG:boto:b'jdpiak-72999falseanulltrue2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:45:20.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:45:20.891Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:45:20.920Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:45:20.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:09.233 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:48:09.234 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:48:09.235 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:48:09.236 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:48:09.236 DEBUG:tasks.rgw_multi.zone_rados:o=e 2022-09-12T19:48:09.237 DEBUG:tasks.rgw_multi.zone_rados:o=f 2022-09-12T19:48:09.239 DEBUG:tasks.rgw_multi.zone_rados:o=g 2022-09-12T19:48:09.239 DEBUG:tasks.rgw_multi.zone_rados:o=h 2022-09-12T19:48:09.241 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:09.241 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:09.242 DEBUG:boto:path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:09.242 DEBUG:boto:auth_path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:09.243 DEBUG:boto:Method: GET 2022-09-12T19:48:09.243 DEBUG:boto:Path: /jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:09.244 DEBUG:boto:Data: 2022-09-12T19:48:09.244 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.245 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:09.245 DEBUG:boto:Port: 8001 2022-09-12T19:48:09.246 DEBUG:boto:Params: {} 2022-09-12T19:48:09.246 DEBUG:boto:Token: None 2022-09-12T19:48:09.247 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/?versions 2022-09-12T19:48:09.248 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e+mz8OTcuyXMbAt8pB0mWWDL2Mk= 2022-09-12T19:48:09.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e+mz8OTcuyXMbAt8pB0mWWDL2Mk=', 'Content-Length': '0'} 2022-09-12T19:48:09.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e91554e14a9e323-00631f8cf9-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.252 DEBUG:boto:b'jdpiak-72999falseanulltrue2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:45:20.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:45:20.891Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:45:20.920Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:45:20.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:09.253 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:09.254 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:09.254 DEBUG:boto:path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:09.255 DEBUG:boto:auth_path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:09.255 DEBUG:boto:Method: GET 2022-09-12T19:48:09.256 DEBUG:boto:Path: /jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:09.257 DEBUG:boto:Data: 2022-09-12T19:48:09.257 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.258 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.258 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.259 DEBUG:boto:Params: {} 2022-09-12T19:48:09.260 DEBUG:boto:Token: None 2022-09-12T19:48:09.260 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/?versions 2022-09-12T19:48:09.261 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e+mz8OTcuyXMbAt8pB0mWWDL2Mk= 2022-09-12T19:48:09.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e+mz8OTcuyXMbAt8pB0mWWDL2Mk=', 'Content-Length': '0'} 2022-09-12T19:48:09.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d62267d10115618c-00631f8cf9-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.265 DEBUG:boto:b'jdpiak-72999falseanulltrue2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:45:20.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:45:20.891Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:45:20.920Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:45:20.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:09.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:48:09.267 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:09.268 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:09.268 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:09.269 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:09.269 DEBUG:boto:Method: GET 2022-09-12T19:48:09.270 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:48:09.270 DEBUG:boto:Data: 2022-09-12T19:48:09.271 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.272 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:09.272 DEBUG:boto:Port: 8001 2022-09-12T19:48:09.273 DEBUG:boto:Params: {} 2022-09-12T19:48:09.273 DEBUG:boto:Token: None 2022-09-12T19:48:09.274 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:48:09.275 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2i2GVV+g11kyJYhuiCsfT9WIzxU= 2022-09-12T19:48:09.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2i2GVV+g11kyJYhuiCsfT9WIzxU=', 'Content-Length': '0'} 2022-09-12T19:48:09.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002db894a5ab6c5a88-00631f8cf9-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.279 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:09.279 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:09.280 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:09.280 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:09.281 DEBUG:boto:Method: GET 2022-09-12T19:48:09.281 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:48:09.283 DEBUG:boto:Data: 2022-09-12T19:48:09.283 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.284 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.286 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.286 DEBUG:boto:Params: {} 2022-09-12T19:48:09.287 DEBUG:boto:Token: None 2022-09-12T19:48:09.288 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:48:09.289 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2i2GVV+g11kyJYhuiCsfT9WIzxU= 2022-09-12T19:48:09.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2i2GVV+g11kyJYhuiCsfT9WIzxU=', 'Content-Length': '0'} 2022-09-12T19:48:09.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e15c1a37c60db3d6-00631f8cf9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.338 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:09.339 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:09.340 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:09.341 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:09.342 DEBUG:boto:Method: HEAD 2022-09-12T19:48:09.342 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:48:09.343 DEBUG:boto:Data: 2022-09-12T19:48:09.343 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.344 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:09.344 DEBUG:boto:Port: 8001 2022-09-12T19:48:09.345 DEBUG:boto:Params: {} 2022-09-12T19:48:09.346 DEBUG:boto:Token: None 2022-09-12T19:48:09.346 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:48:09.347 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UELjtlmWVvE6xdu3HxQ5UMZXNy8= 2022-09-12T19:48:09.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UELjtlmWVvE6xdu3HxQ5UMZXNy8=', 'Content-Length': '0'} 2022-09-12T19:48:09.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003162dcd8dd9797ae-00631f8cf9-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.351 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:09.351 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:09.352 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:09.352 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:09.353 DEBUG:boto:Method: HEAD 2022-09-12T19:48:09.353 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:48:09.354 DEBUG:boto:Data: 2022-09-12T19:48:09.354 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.355 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.355 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.356 DEBUG:boto:Params: {} 2022-09-12T19:48:09.356 DEBUG:boto:Token: None 2022-09-12T19:48:09.357 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:48:09.357 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UELjtlmWVvE6xdu3HxQ5UMZXNy8= 2022-09-12T19:48:09.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UELjtlmWVvE6xdu3HxQ5UMZXNy8=', 'Content-Length': '0'} 2022-09-12T19:48:09.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ec675fb33be90e3-00631f8cf9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:48:09.362 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:09.362 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:09.363 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:09.363 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:09.377 DEBUG:boto:Method: GET 2022-09-12T19:48:09.378 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:48:09.379 DEBUG:boto:Data: 2022-09-12T19:48:09.379 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.380 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:09.380 DEBUG:boto:Port: 8001 2022-09-12T19:48:09.381 DEBUG:boto:Params: {} 2022-09-12T19:48:09.381 DEBUG:boto:Token: None 2022-09-12T19:48:09.382 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:48:09.383 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2i2GVV+g11kyJYhuiCsfT9WIzxU= 2022-09-12T19:48:09.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2i2GVV+g11kyJYhuiCsfT9WIzxU=', 'Content-Length': '0'} 2022-09-12T19:48:09.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001967c241f06648c9-00631f8cf9-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.386 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:09.387 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:09.388 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:09.388 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:09.389 DEBUG:boto:Method: GET 2022-09-12T19:48:09.389 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:48:09.390 DEBUG:boto:Data: 2022-09-12T19:48:09.390 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.391 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.391 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.392 DEBUG:boto:Params: {} 2022-09-12T19:48:09.392 DEBUG:boto:Token: None 2022-09-12T19:48:09.393 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:48:09.394 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2i2GVV+g11kyJYhuiCsfT9WIzxU= 2022-09-12T19:48:09.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2i2GVV+g11kyJYhuiCsfT9WIzxU=', 'Content-Length': '0'} 2022-09-12T19:48:09.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000075c4e2a5cae5f56-00631f8cf9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.397 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:09.398 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:09.398 DEBUG:boto:path=/jdpiak-72/?prefix=a 2022-09-12T19:48:09.399 DEBUG:boto:auth_path=/jdpiak-72/?prefix=a 2022-09-12T19:48:09.399 DEBUG:boto:Method: GET 2022-09-12T19:48:09.400 DEBUG:boto:Path: /jdpiak-72/?prefix=a 2022-09-12T19:48:09.400 DEBUG:boto:Data: 2022-09-12T19:48:09.401 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.401 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:09.402 DEBUG:boto:Port: 8001 2022-09-12T19:48:09.403 DEBUG:boto:Params: {} 2022-09-12T19:48:09.403 DEBUG:boto:Token: None 2022-09-12T19:48:09.404 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/ 2022-09-12T19:48:09.404 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk= 2022-09-12T19:48:09.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk=', 'Content-Length': '0'} 2022-09-12T19:48:09.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001290f9268d670d52-00631f8cf9-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.409 DEBUG:boto:b'jdpiak-72a1000falsea2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:09.409 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:09.410 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:09.411 DEBUG:boto:path=/jdpiak-72/?prefix=a 2022-09-12T19:48:09.411 DEBUG:boto:auth_path=/jdpiak-72/?prefix=a 2022-09-12T19:48:09.412 DEBUG:boto:Method: GET 2022-09-12T19:48:09.412 DEBUG:boto:Path: /jdpiak-72/?prefix=a 2022-09-12T19:48:09.413 DEBUG:boto:Data: 2022-09-12T19:48:09.413 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.414 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.414 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.415 DEBUG:boto:Params: {} 2022-09-12T19:48:09.416 DEBUG:boto:Token: None 2022-09-12T19:48:09.416 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/ 2022-09-12T19:48:09.417 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk= 2022-09-12T19:48:09.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk=', 'Content-Length': '0'} 2022-09-12T19:48:09.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c6b09645cc07988-00631f8cf9-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.421 DEBUG:boto:b'jdpiak-72a1000falsea2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:09.421 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:09.422 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:09.423 DEBUG:boto:Method: HEAD 2022-09-12T19:48:09.423 DEBUG:boto:Path: /jdpiak-72/a 2022-09-12T19:48:09.424 DEBUG:boto:Data: 2022-09-12T19:48:09.424 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.425 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:09.425 DEBUG:boto:Port: 8001 2022-09-12T19:48:09.426 DEBUG:boto:Params: {} 2022-09-12T19:48:09.426 DEBUG:boto:Token: None 2022-09-12T19:48:09.427 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/a 2022-09-12T19:48:09.427 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5WGsi+0LLFgXJkPxmbrZwOLCumE= 2022-09-12T19:48:09.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5WGsi+0LLFgXJkPxmbrZwOLCumE=', 'Content-Length': '0'} 2022-09-12T19:48:09.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000574e9afff60b2e26-00631f8cf9-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.431 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:09.431 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:09.432 DEBUG:boto:Method: HEAD 2022-09-12T19:48:09.432 DEBUG:boto:Path: /jdpiak-72/a 2022-09-12T19:48:09.433 DEBUG:boto:Data: 2022-09-12T19:48:09.433 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.434 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.434 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.435 DEBUG:boto:Params: {} 2022-09-12T19:48:09.435 DEBUG:boto:Token: None 2022-09-12T19:48:09.436 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/a 2022-09-12T19:48:09.436 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5WGsi+0LLFgXJkPxmbrZwOLCumE= 2022-09-12T19:48:09.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5WGsi+0LLFgXJkPxmbrZwOLCumE=', 'Content-Length': '0'} 2022-09-12T19:48:09.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000531cd50b475bd3ac-00631f8cf9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:48:09.440 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:09.441 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:09.441 DEBUG:boto:Method: GET 2022-09-12T19:48:09.442 DEBUG:boto:Path: /jdpiak-72/a 2022-09-12T19:48:09.443 DEBUG:boto:Data: 2022-09-12T19:48:09.443 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.444 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:09.444 DEBUG:boto:Port: 8001 2022-09-12T19:48:09.445 DEBUG:boto:Params: {} 2022-09-12T19:48:09.445 DEBUG:boto:Token: None 2022-09-12T19:48:09.446 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/a 2022-09-12T19:48:09.447 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G/yRHF4I5HeZIANnKRkd+u5tHLc= 2022-09-12T19:48:09.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G/yRHF4I5HeZIANnKRkd+u5tHLc=', 'Content-Length': '0'} 2022-09-12T19:48:09.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f711de9d10187ff2-00631f8cf9-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.490 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:09.491 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:09.492 DEBUG:boto:Method: GET 2022-09-12T19:48:09.492 DEBUG:boto:Path: /jdpiak-72/a 2022-09-12T19:48:09.493 DEBUG:boto:Data: 2022-09-12T19:48:09.493 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.494 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.495 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.496 DEBUG:boto:Params: {} 2022-09-12T19:48:09.497 DEBUG:boto:Token: None 2022-09-12T19:48:09.498 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/a 2022-09-12T19:48:09.499 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G/yRHF4I5HeZIANnKRkd+u5tHLc= 2022-09-12T19:48:09.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G/yRHF4I5HeZIANnKRkd+u5tHLc=', 'Content-Length': '0'} 2022-09-12T19:48:09.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cc5bf0e01efd065-00631f8cf9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.503 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:09.504 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:09.504 DEBUG:boto:path=/jdpiak-72/?prefix=a 2022-09-12T19:48:09.505 DEBUG:boto:auth_path=/jdpiak-72/?prefix=a 2022-09-12T19:48:09.505 DEBUG:boto:Method: GET 2022-09-12T19:48:09.506 DEBUG:boto:Path: /jdpiak-72/?prefix=a 2022-09-12T19:48:09.507 DEBUG:boto:Data: 2022-09-12T19:48:09.507 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.508 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:09.508 DEBUG:boto:Port: 8001 2022-09-12T19:48:09.509 DEBUG:boto:Params: {} 2022-09-12T19:48:09.510 DEBUG:boto:Token: None 2022-09-12T19:48:09.511 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/ 2022-09-12T19:48:09.512 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk= 2022-09-12T19:48:09.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk=', 'Content-Length': '0'} 2022-09-12T19:48:09.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad1a27aeebe0c455-00631f8cf9-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.518 DEBUG:boto:b'jdpiak-72a1000falsea2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:09.519 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:09.520 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:09.521 DEBUG:boto:path=/jdpiak-72/?prefix=a 2022-09-12T19:48:09.522 DEBUG:boto:auth_path=/jdpiak-72/?prefix=a 2022-09-12T19:48:09.522 DEBUG:boto:Method: GET 2022-09-12T19:48:09.523 DEBUG:boto:Path: /jdpiak-72/?prefix=a 2022-09-12T19:48:09.524 DEBUG:boto:Data: 2022-09-12T19:48:09.524 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.525 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.526 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.526 DEBUG:boto:Params: {} 2022-09-12T19:48:09.527 DEBUG:boto:Token: None 2022-09-12T19:48:09.528 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/ 2022-09-12T19:48:09.528 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk= 2022-09-12T19:48:09.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk=', 'Content-Length': '0'} 2022-09-12T19:48:09.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d2ff9477c6f0b59-00631f8cf9-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.533 DEBUG:boto:b'jdpiak-72a1000falsea2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:09.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:48:09.535 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:09.536 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:09.537 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:09.537 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:09.538 DEBUG:boto:Method: GET 2022-09-12T19:48:09.539 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:48:09.540 DEBUG:boto:Data: 2022-09-12T19:48:09.541 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.542 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:09.543 DEBUG:boto:Port: 8001 2022-09-12T19:48:09.544 DEBUG:boto:Params: {} 2022-09-12T19:48:09.545 DEBUG:boto:Token: None 2022-09-12T19:48:09.546 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:48:09.547 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5YZfpg9Y1A8C202ktYndOl9HVN4= 2022-09-12T19:48:09.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5YZfpg9Y1A8C202ktYndOl9HVN4=', 'Content-Length': '0'} 2022-09-12T19:48:09.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043cacf5e60be574d-00631f8cf9-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.552 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:09.552 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:09.553 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:09.554 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:09.554 DEBUG:boto:Method: GET 2022-09-12T19:48:09.555 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:48:09.555 DEBUG:boto:Data: 2022-09-12T19:48:09.556 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.556 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.557 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.557 DEBUG:boto:Params: {} 2022-09-12T19:48:09.558 DEBUG:boto:Token: None 2022-09-12T19:48:09.559 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:48:09.559 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5YZfpg9Y1A8C202ktYndOl9HVN4= 2022-09-12T19:48:09.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5YZfpg9Y1A8C202ktYndOl9HVN4=', 'Content-Length': '0'} 2022-09-12T19:48:09.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edc5d989f98645eb-00631f8cf9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.563 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:09.563 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:09.564 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:09.564 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:09.565 DEBUG:boto:Method: HEAD 2022-09-12T19:48:09.565 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:48:09.566 DEBUG:boto:Data: 2022-09-12T19:48:09.567 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.567 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:09.568 DEBUG:boto:Port: 8001 2022-09-12T19:48:09.568 DEBUG:boto:Params: {} 2022-09-12T19:48:09.569 DEBUG:boto:Token: None 2022-09-12T19:48:09.569 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:48:09.570 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KXoukC8xi/wZKQz9lBxQzo9nFig= 2022-09-12T19:48:09.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KXoukC8xi/wZKQz9lBxQzo9nFig=', 'Content-Length': '0'} 2022-09-12T19:48:09.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044392ec4fb77a085-00631f8cf9-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.573 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:09.574 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:09.574 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:09.575 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:09.575 DEBUG:boto:Method: HEAD 2022-09-12T19:48:09.576 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:48:09.576 DEBUG:boto:Data: 2022-09-12T19:48:09.577 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.577 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.578 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.578 DEBUG:boto:Params: {} 2022-09-12T19:48:09.579 DEBUG:boto:Token: None 2022-09-12T19:48:09.580 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:48:09.580 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KXoukC8xi/wZKQz9lBxQzo9nFig= 2022-09-12T19:48:09.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KXoukC8xi/wZKQz9lBxQzo9nFig=', 'Content-Length': '0'} 2022-09-12T19:48:09.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fb5326b432eedee-00631f8cf9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:48:09.585 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:09.585 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:09.586 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:09.586 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:09.587 DEBUG:boto:Method: GET 2022-09-12T19:48:09.587 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:48:09.588 DEBUG:boto:Data: 2022-09-12T19:48:09.588 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.589 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:09.589 DEBUG:boto:Port: 8001 2022-09-12T19:48:09.590 DEBUG:boto:Params: {} 2022-09-12T19:48:09.591 DEBUG:boto:Token: None 2022-09-12T19:48:09.591 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:48:09.592 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5YZfpg9Y1A8C202ktYndOl9HVN4= 2022-09-12T19:48:09.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5YZfpg9Y1A8C202ktYndOl9HVN4=', 'Content-Length': '0'} 2022-09-12T19:48:09.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa7b06a5bbf95865-00631f8cf9-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.596 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:09.597 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:09.598 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:09.598 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:09.599 DEBUG:boto:Method: GET 2022-09-12T19:48:09.599 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:48:09.600 DEBUG:boto:Data: 2022-09-12T19:48:09.600 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.601 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.601 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.602 DEBUG:boto:Params: {} 2022-09-12T19:48:09.603 DEBUG:boto:Token: None 2022-09-12T19:48:09.603 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:48:09.604 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5YZfpg9Y1A8C202ktYndOl9HVN4= 2022-09-12T19:48:09.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5YZfpg9Y1A8C202ktYndOl9HVN4=', 'Content-Length': '0'} 2022-09-12T19:48:09.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca8d2c41ca77afaf-00631f8cf9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.658 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:09.659 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:09.660 DEBUG:boto:path=/jdpiak-72/?prefix=b 2022-09-12T19:48:09.661 DEBUG:boto:auth_path=/jdpiak-72/?prefix=b 2022-09-12T19:48:09.661 DEBUG:boto:Method: GET 2022-09-12T19:48:09.662 DEBUG:boto:Path: /jdpiak-72/?prefix=b 2022-09-12T19:48:09.663 DEBUG:boto:Data: 2022-09-12T19:48:09.663 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.664 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:09.664 DEBUG:boto:Port: 8001 2022-09-12T19:48:09.665 DEBUG:boto:Params: {} 2022-09-12T19:48:09.666 DEBUG:boto:Token: None 2022-09-12T19:48:09.666 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/ 2022-09-12T19:48:09.667 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk= 2022-09-12T19:48:09.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk=', 'Content-Length': '0'} 2022-09-12T19:48:09.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d85d3a8c40072d9-00631f8cf9-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.672 DEBUG:boto:b'jdpiak-72b1000falseb2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:09.673 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:09.674 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:09.674 DEBUG:boto:path=/jdpiak-72/?prefix=b 2022-09-12T19:48:09.675 DEBUG:boto:auth_path=/jdpiak-72/?prefix=b 2022-09-12T19:48:09.676 DEBUG:boto:Method: GET 2022-09-12T19:48:09.677 DEBUG:boto:Path: /jdpiak-72/?prefix=b 2022-09-12T19:48:09.678 DEBUG:boto:Data: 2022-09-12T19:48:09.679 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.680 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.680 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.681 DEBUG:boto:Params: {} 2022-09-12T19:48:09.682 DEBUG:boto:Token: None 2022-09-12T19:48:09.683 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/ 2022-09-12T19:48:09.684 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk= 2022-09-12T19:48:09.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk=', 'Content-Length': '0'} 2022-09-12T19:48:09.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071ae30097ef07c18-00631f8cf9-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.689 DEBUG:boto:b'jdpiak-72b1000falseb2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:09.690 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:09.690 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:09.691 DEBUG:boto:Method: HEAD 2022-09-12T19:48:09.691 DEBUG:boto:Path: /jdpiak-72/b 2022-09-12T19:48:09.692 DEBUG:boto:Data: 2022-09-12T19:48:09.692 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.693 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:09.693 DEBUG:boto:Port: 8001 2022-09-12T19:48:09.694 DEBUG:boto:Params: {} 2022-09-12T19:48:09.695 DEBUG:boto:Token: None 2022-09-12T19:48:09.696 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/b 2022-09-12T19:48:09.697 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/YNs2kanCoM6EvClfmWlDynoZBE= 2022-09-12T19:48:09.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/YNs2kanCoM6EvClfmWlDynoZBE=', 'Content-Length': '0'} 2022-09-12T19:48:09.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2d50e0ed7cad406-00631f8cf9-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.702 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:09.703 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:09.704 DEBUG:boto:Method: HEAD 2022-09-12T19:48:09.704 DEBUG:boto:Path: /jdpiak-72/b 2022-09-12T19:48:09.705 DEBUG:boto:Data: 2022-09-12T19:48:09.706 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.707 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.707 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.708 DEBUG:boto:Params: {} 2022-09-12T19:48:09.708 DEBUG:boto:Token: None 2022-09-12T19:48:09.709 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/b 2022-09-12T19:48:09.710 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/YNs2kanCoM6EvClfmWlDynoZBE= 2022-09-12T19:48:09.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/YNs2kanCoM6EvClfmWlDynoZBE=', 'Content-Length': '0'} 2022-09-12T19:48:09.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ab6efbfd592f7bd-00631f8cf9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:48:09.714 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:09.715 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:09.716 DEBUG:boto:Method: GET 2022-09-12T19:48:09.717 DEBUG:boto:Path: /jdpiak-72/b 2022-09-12T19:48:09.718 DEBUG:boto:Data: 2022-09-12T19:48:09.719 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.720 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:09.721 DEBUG:boto:Port: 8001 2022-09-12T19:48:09.721 DEBUG:boto:Params: {} 2022-09-12T19:48:09.722 DEBUG:boto:Token: None 2022-09-12T19:48:09.723 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/b 2022-09-12T19:48:09.724 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f10kzxc1UsdR9sjk74m70xAU4+w= 2022-09-12T19:48:09.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f10kzxc1UsdR9sjk74m70xAU4+w=', 'Content-Length': '0'} 2022-09-12T19:48:09.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c882ae57d50089f3-00631f8cf9-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.770 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:09.772 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:09.773 DEBUG:boto:Method: GET 2022-09-12T19:48:09.775 DEBUG:boto:Path: /jdpiak-72/b 2022-09-12T19:48:09.776 DEBUG:boto:Data: 2022-09-12T19:48:09.778 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.779 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.780 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.781 DEBUG:boto:Params: {} 2022-09-12T19:48:09.782 DEBUG:boto:Token: None 2022-09-12T19:48:09.783 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/b 2022-09-12T19:48:09.785 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f10kzxc1UsdR9sjk74m70xAU4+w= 2022-09-12T19:48:09.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f10kzxc1UsdR9sjk74m70xAU4+w=', 'Content-Length': '0'} 2022-09-12T19:48:09.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003478dd0f840f8e19-00631f8cf9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.790 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:09.790 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:09.791 DEBUG:boto:path=/jdpiak-72/?prefix=b 2022-09-12T19:48:09.791 DEBUG:boto:auth_path=/jdpiak-72/?prefix=b 2022-09-12T19:48:09.792 DEBUG:boto:Method: GET 2022-09-12T19:48:09.792 DEBUG:boto:Path: /jdpiak-72/?prefix=b 2022-09-12T19:48:09.793 DEBUG:boto:Data: 2022-09-12T19:48:09.793 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.794 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:09.795 DEBUG:boto:Port: 8001 2022-09-12T19:48:09.795 DEBUG:boto:Params: {} 2022-09-12T19:48:09.796 DEBUG:boto:Token: None 2022-09-12T19:48:09.796 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/ 2022-09-12T19:48:09.797 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk= 2022-09-12T19:48:09.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk=', 'Content-Length': '0'} 2022-09-12T19:48:09.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dcadaabaa89fc66-00631f8cf9-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.801 DEBUG:boto:b'jdpiak-72b1000falseb2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:09.802 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:09.803 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:09.804 DEBUG:boto:path=/jdpiak-72/?prefix=b 2022-09-12T19:48:09.804 DEBUG:boto:auth_path=/jdpiak-72/?prefix=b 2022-09-12T19:48:09.805 DEBUG:boto:Method: GET 2022-09-12T19:48:09.805 DEBUG:boto:Path: /jdpiak-72/?prefix=b 2022-09-12T19:48:09.806 DEBUG:boto:Data: 2022-09-12T19:48:09.807 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.807 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.808 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.808 DEBUG:boto:Params: {} 2022-09-12T19:48:09.809 DEBUG:boto:Token: None 2022-09-12T19:48:09.810 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/ 2022-09-12T19:48:09.810 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk= 2022-09-12T19:48:09.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk=', 'Content-Length': '0'} 2022-09-12T19:48:09.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9fb6f754cbad8f5-00631f8cf9-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.814 DEBUG:boto:b'jdpiak-72b1000falseb2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:09.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:48:09.816 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:09.816 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:09.817 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:09.817 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:09.818 DEBUG:boto:Method: GET 2022-09-12T19:48:09.818 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:48:09.819 DEBUG:boto:Data: 2022-09-12T19:48:09.820 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.820 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:09.821 DEBUG:boto:Port: 8001 2022-09-12T19:48:09.821 DEBUG:boto:Params: {} 2022-09-12T19:48:09.822 DEBUG:boto:Token: None 2022-09-12T19:48:09.822 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:48:09.823 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lSuFk+Q/NTGRpELqgXNpkI3qO3I= 2022-09-12T19:48:09.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lSuFk+Q/NTGRpELqgXNpkI3qO3I=', 'Content-Length': '0'} 2022-09-12T19:48:09.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041d3e2ea745e9fc1-00631f8cf9-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.827 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:09.828 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:09.828 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:09.829 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:09.830 DEBUG:boto:Method: GET 2022-09-12T19:48:09.831 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:48:09.832 DEBUG:boto:Data: 2022-09-12T19:48:09.833 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.834 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.835 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.836 DEBUG:boto:Params: {} 2022-09-12T19:48:09.837 DEBUG:boto:Token: None 2022-09-12T19:48:09.838 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:48:09.839 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lSuFk+Q/NTGRpELqgXNpkI3qO3I= 2022-09-12T19:48:09.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lSuFk+Q/NTGRpELqgXNpkI3qO3I=', 'Content-Length': '0'} 2022-09-12T19:48:09.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6daaa70c070e702-00631f8cf9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.845 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:09.846 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:09.846 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:09.847 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:09.847 DEBUG:boto:Method: HEAD 2022-09-12T19:48:09.848 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:48:09.849 DEBUG:boto:Data: 2022-09-12T19:48:09.849 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.850 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:09.850 DEBUG:boto:Port: 8001 2022-09-12T19:48:09.851 DEBUG:boto:Params: {} 2022-09-12T19:48:09.852 DEBUG:boto:Token: None 2022-09-12T19:48:09.853 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:48:09.853 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e4sJ4qsn6QBBnNNxOz6L09r3tTo= 2022-09-12T19:48:09.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e4sJ4qsn6QBBnNNxOz6L09r3tTo=', 'Content-Length': '0'} 2022-09-12T19:48:09.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b580318f1070dec-00631f8cf9-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.857 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:09.858 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:09.859 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:09.860 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:09.861 DEBUG:boto:Method: HEAD 2022-09-12T19:48:09.862 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:48:09.863 DEBUG:boto:Data: 2022-09-12T19:48:09.863 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.864 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.866 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.867 DEBUG:boto:Params: {} 2022-09-12T19:48:09.868 DEBUG:boto:Token: None 2022-09-12T19:48:09.869 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:48:09.870 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e4sJ4qsn6QBBnNNxOz6L09r3tTo= 2022-09-12T19:48:09.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e4sJ4qsn6QBBnNNxOz6L09r3tTo=', 'Content-Length': '0'} 2022-09-12T19:48:09.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c195b2298f79f97-00631f8cf9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:48:09.875 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:09.876 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:09.876 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:09.877 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:09.878 DEBUG:boto:Method: GET 2022-09-12T19:48:09.878 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:48:09.879 DEBUG:boto:Data: 2022-09-12T19:48:09.880 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.880 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:09.881 DEBUG:boto:Port: 8001 2022-09-12T19:48:09.881 DEBUG:boto:Params: {} 2022-09-12T19:48:09.882 DEBUG:boto:Token: None 2022-09-12T19:48:09.883 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:48:09.883 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lSuFk+Q/NTGRpELqgXNpkI3qO3I= 2022-09-12T19:48:09.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lSuFk+Q/NTGRpELqgXNpkI3qO3I=', 'Content-Length': '0'} 2022-09-12T19:48:09.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce5b2d59ce938e2c-00631f8cf9-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.887 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:09.888 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:09.889 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:09.889 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:09.890 DEBUG:boto:Method: GET 2022-09-12T19:48:09.890 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:48:09.891 DEBUG:boto:Data: 2022-09-12T19:48:09.891 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.892 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.893 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.893 DEBUG:boto:Params: {} 2022-09-12T19:48:09.894 DEBUG:boto:Token: None 2022-09-12T19:48:09.895 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:48:09.895 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lSuFk+Q/NTGRpELqgXNpkI3qO3I= 2022-09-12T19:48:09.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lSuFk+Q/NTGRpELqgXNpkI3qO3I=', 'Content-Length': '0'} 2022-09-12T19:48:09.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005de446eef44dcc1-00631f8cf9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.900 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:09.901 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:09.902 DEBUG:boto:path=/jdpiak-72/?prefix=c 2022-09-12T19:48:09.903 DEBUG:boto:auth_path=/jdpiak-72/?prefix=c 2022-09-12T19:48:09.904 DEBUG:boto:Method: GET 2022-09-12T19:48:09.905 DEBUG:boto:Path: /jdpiak-72/?prefix=c 2022-09-12T19:48:09.906 DEBUG:boto:Data: 2022-09-12T19:48:09.907 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.908 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:09.909 DEBUG:boto:Port: 8001 2022-09-12T19:48:09.910 DEBUG:boto:Params: {} 2022-09-12T19:48:09.910 DEBUG:boto:Token: None 2022-09-12T19:48:09.911 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/ 2022-09-12T19:48:09.912 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk= 2022-09-12T19:48:09.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk=', 'Content-Length': '0'} 2022-09-12T19:48:09.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9db98621876fe28-00631f8cf9-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.958 DEBUG:boto:b'jdpiak-72c1000falsec2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:09.959 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:09.960 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:09.961 DEBUG:boto:path=/jdpiak-72/?prefix=c 2022-09-12T19:48:09.962 DEBUG:boto:auth_path=/jdpiak-72/?prefix=c 2022-09-12T19:48:09.963 DEBUG:boto:Method: GET 2022-09-12T19:48:09.964 DEBUG:boto:Path: /jdpiak-72/?prefix=c 2022-09-12T19:48:09.965 DEBUG:boto:Data: 2022-09-12T19:48:09.966 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.967 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.968 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.969 DEBUG:boto:Params: {} 2022-09-12T19:48:09.970 DEBUG:boto:Token: None 2022-09-12T19:48:09.971 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/ 2022-09-12T19:48:09.972 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk= 2022-09-12T19:48:09.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Fi3mBq+PCYcmpGO9J6EG7Ccqgk=', 'Content-Length': '0'} 2022-09-12T19:48:09.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdc729e56605f067-00631f8cf9-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.977 DEBUG:boto:b'jdpiak-72c1000falsec2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:09.979 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:09.980 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:09.981 DEBUG:boto:Method: HEAD 2022-09-12T19:48:09.982 DEBUG:boto:Path: /jdpiak-72/c 2022-09-12T19:48:09.983 DEBUG:boto:Data: 2022-09-12T19:48:09.984 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.985 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:09.985 DEBUG:boto:Port: 8001 2022-09-12T19:48:09.986 DEBUG:boto:Params: {} 2022-09-12T19:48:09.987 DEBUG:boto:Token: None 2022-09-12T19:48:09.987 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/c 2022-09-12T19:48:09.988 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DCovh3vJvXclMOuQ/57bKZlYIrE= 2022-09-12T19:48:09.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DCovh3vJvXclMOuQ/57bKZlYIrE=', 'Content-Length': '0'} 2022-09-12T19:48:09.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080076963cd14b020-00631f8cf9-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:09.992 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:09.992 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:09.993 DEBUG:boto:Method: HEAD 2022-09-12T19:48:09.994 DEBUG:boto:Path: /jdpiak-72/c 2022-09-12T19:48:09.995 DEBUG:boto:Data: 2022-09-12T19:48:09.996 DEBUG:boto:Headers: {} 2022-09-12T19:48:09.997 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:09.997 DEBUG:boto:Port: 8000 2022-09-12T19:48:09.998 DEBUG:boto:Params: {} 2022-09-12T19:48:09.998 DEBUG:boto:Token: None 2022-09-12T19:48:09.999 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:09 GMT /jdpiak-72/c 2022-09-12T19:48:10.000 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DCovh3vJvXclMOuQ/57bKZlYIrE= 2022-09-12T19:48:10.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DCovh3vJvXclMOuQ/57bKZlYIrE=', 'Content-Length': '0'} 2022-09-12T19:48:10.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f30e5bb49b17ab1f-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:48:10.004 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:10.005 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:10.005 DEBUG:boto:Method: GET 2022-09-12T19:48:10.006 DEBUG:boto:Path: /jdpiak-72/c 2022-09-12T19:48:10.007 DEBUG:boto:Data: 2022-09-12T19:48:10.007 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.008 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.008 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.009 DEBUG:boto:Params: {} 2022-09-12T19:48:10.010 DEBUG:boto:Token: None 2022-09-12T19:48:10.010 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/c 2022-09-12T19:48:10.011 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:irp5pKvlSNKD8lUUCQLiUaNawuY= 2022-09-12T19:48:10.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:irp5pKvlSNKD8lUUCQLiUaNawuY=', 'Content-Length': '0'} 2022-09-12T19:48:10.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003082c356efc2b134-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.016 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:10.017 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:10.018 DEBUG:boto:Method: GET 2022-09-12T19:48:10.019 DEBUG:boto:Path: /jdpiak-72/c 2022-09-12T19:48:10.020 DEBUG:boto:Data: 2022-09-12T19:48:10.021 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.022 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.022 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.023 DEBUG:boto:Params: {} 2022-09-12T19:48:10.024 DEBUG:boto:Token: None 2022-09-12T19:48:10.024 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/c 2022-09-12T19:48:10.025 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:irp5pKvlSNKD8lUUCQLiUaNawuY= 2022-09-12T19:48:10.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:irp5pKvlSNKD8lUUCQLiUaNawuY=', 'Content-Length': '0'} 2022-09-12T19:48:10.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052b80874158728f9-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.029 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:10.030 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:10.031 DEBUG:boto:path=/jdpiak-72/?prefix=c 2022-09-12T19:48:10.031 DEBUG:boto:auth_path=/jdpiak-72/?prefix=c 2022-09-12T19:48:10.032 DEBUG:boto:Method: GET 2022-09-12T19:48:10.032 DEBUG:boto:Path: /jdpiak-72/?prefix=c 2022-09-12T19:48:10.033 DEBUG:boto:Data: 2022-09-12T19:48:10.033 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.034 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.034 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.035 DEBUG:boto:Params: {} 2022-09-12T19:48:10.036 DEBUG:boto:Token: None 2022-09-12T19:48:10.036 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/ 2022-09-12T19:48:10.037 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo= 2022-09-12T19:48:10.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo=', 'Content-Length': '0'} 2022-09-12T19:48:10.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f018f9aa2efe333-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.042 DEBUG:boto:b'jdpiak-72c1000falsec2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:10.043 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:10.043 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:10.044 DEBUG:boto:path=/jdpiak-72/?prefix=c 2022-09-12T19:48:10.044 DEBUG:boto:auth_path=/jdpiak-72/?prefix=c 2022-09-12T19:48:10.045 DEBUG:boto:Method: GET 2022-09-12T19:48:10.046 DEBUG:boto:Path: /jdpiak-72/?prefix=c 2022-09-12T19:48:10.047 DEBUG:boto:Data: 2022-09-12T19:48:10.047 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.048 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.048 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.049 DEBUG:boto:Params: {} 2022-09-12T19:48:10.050 DEBUG:boto:Token: None 2022-09-12T19:48:10.051 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/ 2022-09-12T19:48:10.051 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo= 2022-09-12T19:48:10.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo=', 'Content-Length': '0'} 2022-09-12T19:48:10.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f28ac4251e7f6c9d-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.055 DEBUG:boto:b'jdpiak-72c1000falsec2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:10.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:48:10.057 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:10.057 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:10.058 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:10.058 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:10.059 DEBUG:boto:Method: GET 2022-09-12T19:48:10.059 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:48:10.060 DEBUG:boto:Data: 2022-09-12T19:48:10.060 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.061 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.061 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.062 DEBUG:boto:Params: {} 2022-09-12T19:48:10.063 DEBUG:boto:Token: None 2022-09-12T19:48:10.063 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:48:10.064 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zn2wMY5pxEiyRPfXVAA5BLH16dg= 2022-09-12T19:48:10.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zn2wMY5pxEiyRPfXVAA5BLH16dg=', 'Content-Length': '0'} 2022-09-12T19:48:10.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ecbaf4a61959556-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.067 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:10.068 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:10.068 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:10.069 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:10.069 DEBUG:boto:Method: GET 2022-09-12T19:48:10.070 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:48:10.071 DEBUG:boto:Data: 2022-09-12T19:48:10.071 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.072 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.072 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.073 DEBUG:boto:Params: {} 2022-09-12T19:48:10.073 DEBUG:boto:Token: None 2022-09-12T19:48:10.074 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:48:10.075 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zn2wMY5pxEiyRPfXVAA5BLH16dg= 2022-09-12T19:48:10.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zn2wMY5pxEiyRPfXVAA5BLH16dg=', 'Content-Length': '0'} 2022-09-12T19:48:10.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004579440ca2e15ef8-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.122 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:10.123 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:10.123 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:10.124 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:10.125 DEBUG:boto:Method: HEAD 2022-09-12T19:48:10.125 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:48:10.126 DEBUG:boto:Data: 2022-09-12T19:48:10.126 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.127 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.128 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.128 DEBUG:boto:Params: {} 2022-09-12T19:48:10.129 DEBUG:boto:Token: None 2022-09-12T19:48:10.129 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:48:10.130 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ks5AmC8jQkyvTWKnffaKUJfanNY= 2022-09-12T19:48:10.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ks5AmC8jQkyvTWKnffaKUJfanNY=', 'Content-Length': '0'} 2022-09-12T19:48:10.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eda70a912cb506d1-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.134 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:10.134 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:10.135 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:10.135 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:10.136 DEBUG:boto:Method: HEAD 2022-09-12T19:48:10.136 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:48:10.137 DEBUG:boto:Data: 2022-09-12T19:48:10.137 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.138 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.138 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.139 DEBUG:boto:Params: {} 2022-09-12T19:48:10.140 DEBUG:boto:Token: None 2022-09-12T19:48:10.140 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:48:10.141 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ks5AmC8jQkyvTWKnffaKUJfanNY= 2022-09-12T19:48:10.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ks5AmC8jQkyvTWKnffaKUJfanNY=', 'Content-Length': '0'} 2022-09-12T19:48:10.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dc407736a7caf41-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:48:10.145 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:10.145 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:10.146 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:10.146 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:10.147 DEBUG:boto:Method: GET 2022-09-12T19:48:10.148 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:48:10.148 DEBUG:boto:Data: 2022-09-12T19:48:10.149 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.149 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.150 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.150 DEBUG:boto:Params: {} 2022-09-12T19:48:10.151 DEBUG:boto:Token: None 2022-09-12T19:48:10.152 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:48:10.152 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zn2wMY5pxEiyRPfXVAA5BLH16dg= 2022-09-12T19:48:10.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zn2wMY5pxEiyRPfXVAA5BLH16dg=', 'Content-Length': '0'} 2022-09-12T19:48:10.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c41300dee6f3756c-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.156 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:10.157 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:10.157 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:10.158 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:10.159 DEBUG:boto:Method: GET 2022-09-12T19:48:10.159 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:48:10.160 DEBUG:boto:Data: 2022-09-12T19:48:10.161 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.161 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.162 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.162 DEBUG:boto:Params: {} 2022-09-12T19:48:10.163 DEBUG:boto:Token: None 2022-09-12T19:48:10.164 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:48:10.164 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zn2wMY5pxEiyRPfXVAA5BLH16dg= 2022-09-12T19:48:10.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zn2wMY5pxEiyRPfXVAA5BLH16dg=', 'Content-Length': '0'} 2022-09-12T19:48:10.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daa8dade9cd2d23d-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.169 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:10.169 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:10.170 DEBUG:boto:path=/jdpiak-72/?prefix=d 2022-09-12T19:48:10.171 DEBUG:boto:auth_path=/jdpiak-72/?prefix=d 2022-09-12T19:48:10.172 DEBUG:boto:Method: GET 2022-09-12T19:48:10.172 DEBUG:boto:Path: /jdpiak-72/?prefix=d 2022-09-12T19:48:10.173 DEBUG:boto:Data: 2022-09-12T19:48:10.173 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.174 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.174 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.175 DEBUG:boto:Params: {} 2022-09-12T19:48:10.176 DEBUG:boto:Token: None 2022-09-12T19:48:10.177 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/ 2022-09-12T19:48:10.178 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo= 2022-09-12T19:48:10.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo=', 'Content-Length': '0'} 2022-09-12T19:48:10.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1fccc93efc5f49a-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.183 DEBUG:boto:b'jdpiak-72d1000falsed2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:10.184 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:10.184 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:10.185 DEBUG:boto:path=/jdpiak-72/?prefix=d 2022-09-12T19:48:10.185 DEBUG:boto:auth_path=/jdpiak-72/?prefix=d 2022-09-12T19:48:10.186 DEBUG:boto:Method: GET 2022-09-12T19:48:10.186 DEBUG:boto:Path: /jdpiak-72/?prefix=d 2022-09-12T19:48:10.188 DEBUG:boto:Data: 2022-09-12T19:48:10.188 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.189 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.189 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.190 DEBUG:boto:Params: {} 2022-09-12T19:48:10.191 DEBUG:boto:Token: None 2022-09-12T19:48:10.191 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/ 2022-09-12T19:48:10.192 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo= 2022-09-12T19:48:10.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo=', 'Content-Length': '0'} 2022-09-12T19:48:10.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000829a2b9024dd9921-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.196 DEBUG:boto:b'jdpiak-72d1000falsed2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:10.197 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:10.197 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:10.198 DEBUG:boto:Method: HEAD 2022-09-12T19:48:10.198 DEBUG:boto:Path: /jdpiak-72/d 2022-09-12T19:48:10.199 DEBUG:boto:Data: 2022-09-12T19:48:10.199 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.200 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.200 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.201 DEBUG:boto:Params: {} 2022-09-12T19:48:10.201 DEBUG:boto:Token: None 2022-09-12T19:48:10.202 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/d 2022-09-12T19:48:10.203 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lcvCiwTq3hX9FuG3FTQbdmS2MyM= 2022-09-12T19:48:10.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lcvCiwTq3hX9FuG3FTQbdmS2MyM=', 'Content-Length': '0'} 2022-09-12T19:48:10.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a62a92ba51664f65-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.206 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:10.207 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:10.207 DEBUG:boto:Method: HEAD 2022-09-12T19:48:10.208 DEBUG:boto:Path: /jdpiak-72/d 2022-09-12T19:48:10.208 DEBUG:boto:Data: 2022-09-12T19:48:10.209 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.209 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.210 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.211 DEBUG:boto:Params: {} 2022-09-12T19:48:10.212 DEBUG:boto:Token: None 2022-09-12T19:48:10.213 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/d 2022-09-12T19:48:10.214 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lcvCiwTq3hX9FuG3FTQbdmS2MyM= 2022-09-12T19:48:10.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lcvCiwTq3hX9FuG3FTQbdmS2MyM=', 'Content-Length': '0'} 2022-09-12T19:48:10.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c6e00864211bba9-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:48:10.220 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:10.221 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:10.222 DEBUG:boto:Method: GET 2022-09-12T19:48:10.222 DEBUG:boto:Path: /jdpiak-72/d 2022-09-12T19:48:10.223 DEBUG:boto:Data: 2022-09-12T19:48:10.224 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.224 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.225 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.226 DEBUG:boto:Params: {} 2022-09-12T19:48:10.227 DEBUG:boto:Token: None 2022-09-12T19:48:10.227 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/d 2022-09-12T19:48:10.228 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kswaDfDsVttrQXwrk1POT0Nzdeo= 2022-09-12T19:48:10.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kswaDfDsVttrQXwrk1POT0Nzdeo=', 'Content-Length': '0'} 2022-09-12T19:48:10.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7be8e029fc4247a-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.274 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:10.275 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:10.275 DEBUG:boto:Method: GET 2022-09-12T19:48:10.276 DEBUG:boto:Path: /jdpiak-72/d 2022-09-12T19:48:10.277 DEBUG:boto:Data: 2022-09-12T19:48:10.277 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.278 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.278 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.279 DEBUG:boto:Params: {} 2022-09-12T19:48:10.279 DEBUG:boto:Token: None 2022-09-12T19:48:10.280 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/d 2022-09-12T19:48:10.281 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kswaDfDsVttrQXwrk1POT0Nzdeo= 2022-09-12T19:48:10.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kswaDfDsVttrQXwrk1POT0Nzdeo=', 'Content-Length': '0'} 2022-09-12T19:48:10.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e1082738388268e-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.285 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:10.285 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:10.286 DEBUG:boto:path=/jdpiak-72/?prefix=d 2022-09-12T19:48:10.287 DEBUG:boto:auth_path=/jdpiak-72/?prefix=d 2022-09-12T19:48:10.288 DEBUG:boto:Method: GET 2022-09-12T19:48:10.289 DEBUG:boto:Path: /jdpiak-72/?prefix=d 2022-09-12T19:48:10.290 DEBUG:boto:Data: 2022-09-12T19:48:10.291 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.292 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.293 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.294 DEBUG:boto:Params: {} 2022-09-12T19:48:10.295 DEBUG:boto:Token: None 2022-09-12T19:48:10.296 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/ 2022-09-12T19:48:10.297 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo= 2022-09-12T19:48:10.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo=', 'Content-Length': '0'} 2022-09-12T19:48:10.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cc68e39cc2dde77-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.303 DEBUG:boto:b'jdpiak-72d1000falsed2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:10.304 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:10.305 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:10.306 DEBUG:boto:path=/jdpiak-72/?prefix=d 2022-09-12T19:48:10.307 DEBUG:boto:auth_path=/jdpiak-72/?prefix=d 2022-09-12T19:48:10.308 DEBUG:boto:Method: GET 2022-09-12T19:48:10.309 DEBUG:boto:Path: /jdpiak-72/?prefix=d 2022-09-12T19:48:10.310 DEBUG:boto:Data: 2022-09-12T19:48:10.311 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.312 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.313 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.314 DEBUG:boto:Params: {} 2022-09-12T19:48:10.315 DEBUG:boto:Token: None 2022-09-12T19:48:10.316 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/ 2022-09-12T19:48:10.317 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo= 2022-09-12T19:48:10.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo=', 'Content-Length': '0'} 2022-09-12T19:48:10.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e870f1abfc54a047-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.323 DEBUG:boto:b'jdpiak-72d1000falsed2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:10.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:48:10.325 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:10.326 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:10.327 DEBUG:boto:path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:10.328 DEBUG:boto:auth_path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:10.328 DEBUG:boto:Method: GET 2022-09-12T19:48:10.329 DEBUG:boto:Path: /jdpiak-72/e?versionId=null 2022-09-12T19:48:10.329 DEBUG:boto:Data: 2022-09-12T19:48:10.330 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.330 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.331 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.331 DEBUG:boto:Params: {} 2022-09-12T19:48:10.332 DEBUG:boto:Token: None 2022-09-12T19:48:10.333 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/e?versionId=null 2022-09-12T19:48:10.334 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CaquW3j9yqJSgx/32/Ld0lARwSA= 2022-09-12T19:48:10.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CaquW3j9yqJSgx/32/Ld0lARwSA=', 'Content-Length': '0'} 2022-09-12T19:48:10.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f83baa38c95e303c-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.338 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:10.339 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:10.340 DEBUG:boto:path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:10.341 DEBUG:boto:auth_path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:10.341 DEBUG:boto:Method: GET 2022-09-12T19:48:10.342 DEBUG:boto:Path: /jdpiak-72/e?versionId=null 2022-09-12T19:48:10.343 DEBUG:boto:Data: 2022-09-12T19:48:10.344 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.345 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.347 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.347 DEBUG:boto:Params: {} 2022-09-12T19:48:10.348 DEBUG:boto:Token: None 2022-09-12T19:48:10.349 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/e?versionId=null 2022-09-12T19:48:10.351 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CaquW3j9yqJSgx/32/Ld0lARwSA= 2022-09-12T19:48:10.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CaquW3j9yqJSgx/32/Ld0lARwSA=', 'Content-Length': '0'} 2022-09-12T19:48:10.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3ce6456d704efe5-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.360 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:10.361 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:10.362 DEBUG:boto:path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:10.363 DEBUG:boto:auth_path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:10.364 DEBUG:boto:Method: HEAD 2022-09-12T19:48:10.365 DEBUG:boto:Path: /jdpiak-72/e?versionId=null 2022-09-12T19:48:10.366 DEBUG:boto:Data: 2022-09-12T19:48:10.367 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.367 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.368 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.368 DEBUG:boto:Params: {} 2022-09-12T19:48:10.369 DEBUG:boto:Token: None 2022-09-12T19:48:10.369 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/e?versionId=null 2022-09-12T19:48:10.370 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7Pf0rq2+Hts3KIRK4rHNrusf5t0= 2022-09-12T19:48:10.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7Pf0rq2+Hts3KIRK4rHNrusf5t0=', 'Content-Length': '0'} 2022-09-12T19:48:10.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da298f30d44ce76e-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.374 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:10.375 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:10.376 DEBUG:boto:path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:10.377 DEBUG:boto:auth_path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:10.378 DEBUG:boto:Method: HEAD 2022-09-12T19:48:10.379 DEBUG:boto:Path: /jdpiak-72/e?versionId=null 2022-09-12T19:48:10.379 DEBUG:boto:Data: 2022-09-12T19:48:10.380 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.381 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.382 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.382 DEBUG:boto:Params: {} 2022-09-12T19:48:10.383 DEBUG:boto:Token: None 2022-09-12T19:48:10.384 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/e?versionId=null 2022-09-12T19:48:10.384 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7Pf0rq2+Hts3KIRK4rHNrusf5t0= 2022-09-12T19:48:10.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7Pf0rq2+Hts3KIRK4rHNrusf5t0=', 'Content-Length': '0'} 2022-09-12T19:48:10.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004aa1f6d8f423b8e0-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:48:10.390 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:10.391 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:10.392 DEBUG:boto:path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:10.393 DEBUG:boto:auth_path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:10.394 DEBUG:boto:Method: GET 2022-09-12T19:48:10.394 DEBUG:boto:Path: /jdpiak-72/e?versionId=null 2022-09-12T19:48:10.395 DEBUG:boto:Data: 2022-09-12T19:48:10.396 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.397 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.398 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.398 DEBUG:boto:Params: {} 2022-09-12T19:48:10.399 DEBUG:boto:Token: None 2022-09-12T19:48:10.399 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/e?versionId=null 2022-09-12T19:48:10.400 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CaquW3j9yqJSgx/32/Ld0lARwSA= 2022-09-12T19:48:10.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CaquW3j9yqJSgx/32/Ld0lARwSA=', 'Content-Length': '0'} 2022-09-12T19:48:10.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076e37c94c6840443-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.404 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:10.405 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:10.405 DEBUG:boto:path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:10.406 DEBUG:boto:auth_path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:10.406 DEBUG:boto:Method: GET 2022-09-12T19:48:10.407 DEBUG:boto:Path: /jdpiak-72/e?versionId=null 2022-09-12T19:48:10.407 DEBUG:boto:Data: 2022-09-12T19:48:10.408 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.408 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.409 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.410 DEBUG:boto:Params: {} 2022-09-12T19:48:10.410 DEBUG:boto:Token: None 2022-09-12T19:48:10.411 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/e?versionId=null 2022-09-12T19:48:10.411 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CaquW3j9yqJSgx/32/Ld0lARwSA= 2022-09-12T19:48:10.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CaquW3j9yqJSgx/32/Ld0lARwSA=', 'Content-Length': '0'} 2022-09-12T19:48:10.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd05e781a92f40a8-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.462 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:10.463 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:10.463 DEBUG:boto:path=/jdpiak-72/?prefix=e 2022-09-12T19:48:10.464 DEBUG:boto:auth_path=/jdpiak-72/?prefix=e 2022-09-12T19:48:10.464 DEBUG:boto:Method: GET 2022-09-12T19:48:10.465 DEBUG:boto:Path: /jdpiak-72/?prefix=e 2022-09-12T19:48:10.466 DEBUG:boto:Data: 2022-09-12T19:48:10.466 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.467 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.467 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.471 DEBUG:boto:Params: {} 2022-09-12T19:48:10.472 DEBUG:boto:Token: None 2022-09-12T19:48:10.472 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/ 2022-09-12T19:48:10.473 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo= 2022-09-12T19:48:10.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo=', 'Content-Length': '0'} 2022-09-12T19:48:10.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001e6a05a10b8ab9d-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.477 DEBUG:boto:b'jdpiak-72e1000falsee2022-09-12T19:45:20.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:10.478 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:10.479 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:10.480 DEBUG:boto:path=/jdpiak-72/?prefix=e 2022-09-12T19:48:10.480 DEBUG:boto:auth_path=/jdpiak-72/?prefix=e 2022-09-12T19:48:10.481 DEBUG:boto:Method: GET 2022-09-12T19:48:10.481 DEBUG:boto:Path: /jdpiak-72/?prefix=e 2022-09-12T19:48:10.482 DEBUG:boto:Data: 2022-09-12T19:48:10.482 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.483 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.483 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.484 DEBUG:boto:Params: {} 2022-09-12T19:48:10.484 DEBUG:boto:Token: None 2022-09-12T19:48:10.485 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/ 2022-09-12T19:48:10.486 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo= 2022-09-12T19:48:10.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo=', 'Content-Length': '0'} 2022-09-12T19:48:10.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1dd3c70bfe4615c-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.490 DEBUG:boto:b'jdpiak-72e1000falsee2022-09-12T19:45:20.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:10.491 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:10.492 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:10.492 DEBUG:boto:Method: HEAD 2022-09-12T19:48:10.493 DEBUG:boto:Path: /jdpiak-72/e 2022-09-12T19:48:10.493 DEBUG:boto:Data: 2022-09-12T19:48:10.494 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.494 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.495 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.495 DEBUG:boto:Params: {} 2022-09-12T19:48:10.496 DEBUG:boto:Token: None 2022-09-12T19:48:10.496 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/e 2022-09-12T19:48:10.497 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lSfQ5r2A5qOtzU2XeCaGDemMnKw= 2022-09-12T19:48:10.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lSfQ5r2A5qOtzU2XeCaGDemMnKw=', 'Content-Length': '0'} 2022-09-12T19:48:10.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fb86baf8a6a9310-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.501 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:10.501 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:10.502 DEBUG:boto:Method: HEAD 2022-09-12T19:48:10.503 DEBUG:boto:Path: /jdpiak-72/e 2022-09-12T19:48:10.503 DEBUG:boto:Data: 2022-09-12T19:48:10.504 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.504 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.505 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.505 DEBUG:boto:Params: {} 2022-09-12T19:48:10.506 DEBUG:boto:Token: None 2022-09-12T19:48:10.506 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/e 2022-09-12T19:48:10.507 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lSfQ5r2A5qOtzU2XeCaGDemMnKw= 2022-09-12T19:48:10.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lSfQ5r2A5qOtzU2XeCaGDemMnKw=', 'Content-Length': '0'} 2022-09-12T19:48:10.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ade9b857dee71104-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:48:10.511 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:10.512 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:10.512 DEBUG:boto:Method: GET 2022-09-12T19:48:10.513 DEBUG:boto:Path: /jdpiak-72/e 2022-09-12T19:48:10.513 DEBUG:boto:Data: 2022-09-12T19:48:10.514 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.514 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.515 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.515 DEBUG:boto:Params: {} 2022-09-12T19:48:10.516 DEBUG:boto:Token: None 2022-09-12T19:48:10.517 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/e 2022-09-12T19:48:10.517 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f508KjRc314ulM+mJHM+EHQ5oUY= 2022-09-12T19:48:10.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f508KjRc314ulM+mJHM+EHQ5oUY=', 'Content-Length': '0'} 2022-09-12T19:48:10.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000087a3218faf8e60-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.521 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:10.521 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:10.522 DEBUG:boto:Method: GET 2022-09-12T19:48:10.523 DEBUG:boto:Path: /jdpiak-72/e 2022-09-12T19:48:10.523 DEBUG:boto:Data: 2022-09-12T19:48:10.524 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.524 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.525 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.525 DEBUG:boto:Params: {} 2022-09-12T19:48:10.526 DEBUG:boto:Token: None 2022-09-12T19:48:10.526 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/e 2022-09-12T19:48:10.527 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f508KjRc314ulM+mJHM+EHQ5oUY= 2022-09-12T19:48:10.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f508KjRc314ulM+mJHM+EHQ5oUY=', 'Content-Length': '0'} 2022-09-12T19:48:10.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006310beb4346ee9e3-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.531 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:10.531 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:10.532 DEBUG:boto:path=/jdpiak-72/?prefix=e 2022-09-12T19:48:10.532 DEBUG:boto:auth_path=/jdpiak-72/?prefix=e 2022-09-12T19:48:10.533 DEBUG:boto:Method: GET 2022-09-12T19:48:10.533 DEBUG:boto:Path: /jdpiak-72/?prefix=e 2022-09-12T19:48:10.534 DEBUG:boto:Data: 2022-09-12T19:48:10.534 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.535 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.536 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.536 DEBUG:boto:Params: {} 2022-09-12T19:48:10.537 DEBUG:boto:Token: None 2022-09-12T19:48:10.537 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/ 2022-09-12T19:48:10.538 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo= 2022-09-12T19:48:10.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo=', 'Content-Length': '0'} 2022-09-12T19:48:10.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071e3c51b79b5c983-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.542 DEBUG:boto:b'jdpiak-72e1000falsee2022-09-12T19:45:20.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:10.543 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:10.544 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:10.545 DEBUG:boto:path=/jdpiak-72/?prefix=e 2022-09-12T19:48:10.545 DEBUG:boto:auth_path=/jdpiak-72/?prefix=e 2022-09-12T19:48:10.546 DEBUG:boto:Method: GET 2022-09-12T19:48:10.546 DEBUG:boto:Path: /jdpiak-72/?prefix=e 2022-09-12T19:48:10.547 DEBUG:boto:Data: 2022-09-12T19:48:10.548 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.548 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.549 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.550 DEBUG:boto:Params: {} 2022-09-12T19:48:10.551 DEBUG:boto:Token: None 2022-09-12T19:48:10.552 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/ 2022-09-12T19:48:10.553 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo= 2022-09-12T19:48:10.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo=', 'Content-Length': '0'} 2022-09-12T19:48:10.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ba924c7ecb9549c-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.559 DEBUG:boto:b'jdpiak-72e1000falsee2022-09-12T19:45:20.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:10.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:48:10.561 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:10.561 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:10.562 DEBUG:boto:path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:10.563 DEBUG:boto:auth_path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:10.563 DEBUG:boto:Method: GET 2022-09-12T19:48:10.564 DEBUG:boto:Path: /jdpiak-72/f?versionId=null 2022-09-12T19:48:10.565 DEBUG:boto:Data: 2022-09-12T19:48:10.565 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.566 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.567 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.567 DEBUG:boto:Params: {} 2022-09-12T19:48:10.568 DEBUG:boto:Token: None 2022-09-12T19:48:10.569 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/f?versionId=null 2022-09-12T19:48:10.569 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GiS+jNn0vW33SUJ0RogBmMgrk3w= 2022-09-12T19:48:10.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GiS+jNn0vW33SUJ0RogBmMgrk3w=', 'Content-Length': '0'} 2022-09-12T19:48:10.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e726a6ab98792c0-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.573 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:10.574 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:10.574 DEBUG:boto:path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:10.575 DEBUG:boto:auth_path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:10.575 DEBUG:boto:Method: GET 2022-09-12T19:48:10.576 DEBUG:boto:Path: /jdpiak-72/f?versionId=null 2022-09-12T19:48:10.576 DEBUG:boto:Data: 2022-09-12T19:48:10.577 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.577 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.578 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.579 DEBUG:boto:Params: {} 2022-09-12T19:48:10.580 DEBUG:boto:Token: None 2022-09-12T19:48:10.580 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/f?versionId=null 2022-09-12T19:48:10.581 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GiS+jNn0vW33SUJ0RogBmMgrk3w= 2022-09-12T19:48:10.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GiS+jNn0vW33SUJ0RogBmMgrk3w=', 'Content-Length': '0'} 2022-09-12T19:48:10.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000715dad3a34ba7d4a-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.585 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:10.585 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:10.586 DEBUG:boto:path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:10.587 DEBUG:boto:auth_path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:10.587 DEBUG:boto:Method: HEAD 2022-09-12T19:48:10.588 DEBUG:boto:Path: /jdpiak-72/f?versionId=null 2022-09-12T19:48:10.589 DEBUG:boto:Data: 2022-09-12T19:48:10.590 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.591 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.592 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.593 DEBUG:boto:Params: {} 2022-09-12T19:48:10.594 DEBUG:boto:Token: None 2022-09-12T19:48:10.595 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/f?versionId=null 2022-09-12T19:48:10.596 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bPtJ6Via9M7XjDckjjxIJz6ztT8= 2022-09-12T19:48:10.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bPtJ6Via9M7XjDckjjxIJz6ztT8=', 'Content-Length': '0'} 2022-09-12T19:48:10.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e57ed1463bc30dc5-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.601 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:10.601 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:10.602 DEBUG:boto:path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:10.603 DEBUG:boto:auth_path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:10.604 DEBUG:boto:Method: HEAD 2022-09-12T19:48:10.605 DEBUG:boto:Path: /jdpiak-72/f?versionId=null 2022-09-12T19:48:10.606 DEBUG:boto:Data: 2022-09-12T19:48:10.607 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.608 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.609 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.610 DEBUG:boto:Params: {} 2022-09-12T19:48:10.611 DEBUG:boto:Token: None 2022-09-12T19:48:10.612 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/f?versionId=null 2022-09-12T19:48:10.613 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bPtJ6Via9M7XjDckjjxIJz6ztT8= 2022-09-12T19:48:10.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bPtJ6Via9M7XjDckjjxIJz6ztT8=', 'Content-Length': '0'} 2022-09-12T19:48:10.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067ecb9962d671a33-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:48:10.617 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:10.618 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:10.619 DEBUG:boto:path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:10.619 DEBUG:boto:auth_path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:10.620 DEBUG:boto:Method: GET 2022-09-12T19:48:10.620 DEBUG:boto:Path: /jdpiak-72/f?versionId=null 2022-09-12T19:48:10.621 DEBUG:boto:Data: 2022-09-12T19:48:10.621 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.622 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.623 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.623 DEBUG:boto:Params: {} 2022-09-12T19:48:10.624 DEBUG:boto:Token: None 2022-09-12T19:48:10.624 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/f?versionId=null 2022-09-12T19:48:10.625 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GiS+jNn0vW33SUJ0RogBmMgrk3w= 2022-09-12T19:48:10.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GiS+jNn0vW33SUJ0RogBmMgrk3w=', 'Content-Length': '0'} 2022-09-12T19:48:10.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000858b0d20c1ebc16a-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.629 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:10.629 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:10.630 DEBUG:boto:path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:10.630 DEBUG:boto:auth_path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:10.631 DEBUG:boto:Method: GET 2022-09-12T19:48:10.631 DEBUG:boto:Path: /jdpiak-72/f?versionId=null 2022-09-12T19:48:10.632 DEBUG:boto:Data: 2022-09-12T19:48:10.632 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.633 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.633 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.634 DEBUG:boto:Params: {} 2022-09-12T19:48:10.635 DEBUG:boto:Token: None 2022-09-12T19:48:10.635 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/f?versionId=null 2022-09-12T19:48:10.636 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GiS+jNn0vW33SUJ0RogBmMgrk3w= 2022-09-12T19:48:10.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GiS+jNn0vW33SUJ0RogBmMgrk3w=', 'Content-Length': '0'} 2022-09-12T19:48:10.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000992dbe9e2fc69dfb-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.639 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:10.640 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:10.641 DEBUG:boto:path=/jdpiak-72/?prefix=f 2022-09-12T19:48:10.641 DEBUG:boto:auth_path=/jdpiak-72/?prefix=f 2022-09-12T19:48:10.642 DEBUG:boto:Method: GET 2022-09-12T19:48:10.643 DEBUG:boto:Path: /jdpiak-72/?prefix=f 2022-09-12T19:48:10.643 DEBUG:boto:Data: 2022-09-12T19:48:10.644 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.644 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.645 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.645 DEBUG:boto:Params: {} 2022-09-12T19:48:10.647 DEBUG:boto:Token: None 2022-09-12T19:48:10.648 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/ 2022-09-12T19:48:10.649 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo= 2022-09-12T19:48:10.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo=', 'Content-Length': '0'} 2022-09-12T19:48:10.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023e10c630864e5b7-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.654 DEBUG:boto:b'jdpiak-72f1000falsef2022-09-12T19:45:20.891Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:10.655 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:10.656 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:10.656 DEBUG:boto:path=/jdpiak-72/?prefix=f 2022-09-12T19:48:10.657 DEBUG:boto:auth_path=/jdpiak-72/?prefix=f 2022-09-12T19:48:10.658 DEBUG:boto:Method: GET 2022-09-12T19:48:10.658 DEBUG:boto:Path: /jdpiak-72/?prefix=f 2022-09-12T19:48:10.659 DEBUG:boto:Data: 2022-09-12T19:48:10.659 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.660 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.660 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.661 DEBUG:boto:Params: {} 2022-09-12T19:48:10.662 DEBUG:boto:Token: None 2022-09-12T19:48:10.663 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/ 2022-09-12T19:48:10.664 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo= 2022-09-12T19:48:10.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo=', 'Content-Length': '0'} 2022-09-12T19:48:10.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cb4ffbbcb3033f9-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.710 DEBUG:boto:b'jdpiak-72f1000falsef2022-09-12T19:45:20.891Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:10.711 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:10.712 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:10.712 DEBUG:boto:Method: HEAD 2022-09-12T19:48:10.713 DEBUG:boto:Path: /jdpiak-72/f 2022-09-12T19:48:10.713 DEBUG:boto:Data: 2022-09-12T19:48:10.714 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.714 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.715 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.715 DEBUG:boto:Params: {} 2022-09-12T19:48:10.716 DEBUG:boto:Token: None 2022-09-12T19:48:10.716 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/f 2022-09-12T19:48:10.717 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qqflndXk6iB2TZt2JsowzLeok7w= 2022-09-12T19:48:10.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qqflndXk6iB2TZt2JsowzLeok7w=', 'Content-Length': '0'} 2022-09-12T19:48:10.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c0ebe4c47a49674-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.721 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:10.721 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:10.722 DEBUG:boto:Method: HEAD 2022-09-12T19:48:10.722 DEBUG:boto:Path: /jdpiak-72/f 2022-09-12T19:48:10.723 DEBUG:boto:Data: 2022-09-12T19:48:10.723 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.724 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.724 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.725 DEBUG:boto:Params: {} 2022-09-12T19:48:10.726 DEBUG:boto:Token: None 2022-09-12T19:48:10.726 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/f 2022-09-12T19:48:10.727 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qqflndXk6iB2TZt2JsowzLeok7w= 2022-09-12T19:48:10.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qqflndXk6iB2TZt2JsowzLeok7w=', 'Content-Length': '0'} 2022-09-12T19:48:10.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f4c2a00c584721b-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:48:10.731 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:10.731 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:10.732 DEBUG:boto:Method: GET 2022-09-12T19:48:10.732 DEBUG:boto:Path: /jdpiak-72/f 2022-09-12T19:48:10.733 DEBUG:boto:Data: 2022-09-12T19:48:10.733 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.734 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.735 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.735 DEBUG:boto:Params: {} 2022-09-12T19:48:10.736 DEBUG:boto:Token: None 2022-09-12T19:48:10.736 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/f 2022-09-12T19:48:10.737 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4OKEN/NUay2++ijC4lNJGEgXa9g= 2022-09-12T19:48:10.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4OKEN/NUay2++ijC4lNJGEgXa9g=', 'Content-Length': '0'} 2022-09-12T19:48:10.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a98a0c0826124d67-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.740 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:10.741 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:10.742 DEBUG:boto:Method: GET 2022-09-12T19:48:10.742 DEBUG:boto:Path: /jdpiak-72/f 2022-09-12T19:48:10.743 DEBUG:boto:Data: 2022-09-12T19:48:10.743 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.744 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.744 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.745 DEBUG:boto:Params: {} 2022-09-12T19:48:10.745 DEBUG:boto:Token: None 2022-09-12T19:48:10.746 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/f 2022-09-12T19:48:10.747 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4OKEN/NUay2++ijC4lNJGEgXa9g= 2022-09-12T19:48:10.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4OKEN/NUay2++ijC4lNJGEgXa9g=', 'Content-Length': '0'} 2022-09-12T19:48:10.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081a915fd012322ac-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.750 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:10.751 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:10.751 DEBUG:boto:path=/jdpiak-72/?prefix=f 2022-09-12T19:48:10.752 DEBUG:boto:auth_path=/jdpiak-72/?prefix=f 2022-09-12T19:48:10.752 DEBUG:boto:Method: GET 2022-09-12T19:48:10.753 DEBUG:boto:Path: /jdpiak-72/?prefix=f 2022-09-12T19:48:10.754 DEBUG:boto:Data: 2022-09-12T19:48:10.754 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.755 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.755 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.756 DEBUG:boto:Params: {} 2022-09-12T19:48:10.756 DEBUG:boto:Token: None 2022-09-12T19:48:10.757 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/ 2022-09-12T19:48:10.757 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo= 2022-09-12T19:48:10.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo=', 'Content-Length': '0'} 2022-09-12T19:48:10.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ad099955ee5390d-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.762 DEBUG:boto:b'jdpiak-72f1000falsef2022-09-12T19:45:20.891Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:10.763 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:10.763 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:10.764 DEBUG:boto:path=/jdpiak-72/?prefix=f 2022-09-12T19:48:10.764 DEBUG:boto:auth_path=/jdpiak-72/?prefix=f 2022-09-12T19:48:10.765 DEBUG:boto:Method: GET 2022-09-12T19:48:10.765 DEBUG:boto:Path: /jdpiak-72/?prefix=f 2022-09-12T19:48:10.766 DEBUG:boto:Data: 2022-09-12T19:48:10.766 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.767 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.767 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.768 DEBUG:boto:Params: {} 2022-09-12T19:48:10.769 DEBUG:boto:Token: None 2022-09-12T19:48:10.769 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/ 2022-09-12T19:48:10.770 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo= 2022-09-12T19:48:10.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo=', 'Content-Length': '0'} 2022-09-12T19:48:10.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000052dad91df05e784-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.773 DEBUG:boto:b'jdpiak-72f1000falsef2022-09-12T19:45:20.891Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:10.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:48:10.775 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:10.775 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:10.776 DEBUG:boto:path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:10.777 DEBUG:boto:auth_path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:10.778 DEBUG:boto:Method: GET 2022-09-12T19:48:10.778 DEBUG:boto:Path: /jdpiak-72/g?versionId=null 2022-09-12T19:48:10.779 DEBUG:boto:Data: 2022-09-12T19:48:10.780 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.780 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.781 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.781 DEBUG:boto:Params: {} 2022-09-12T19:48:10.782 DEBUG:boto:Token: None 2022-09-12T19:48:10.782 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/g?versionId=null 2022-09-12T19:48:10.783 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FjghIPYG4GgQpw73e9hkHP579Lc= 2022-09-12T19:48:10.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FjghIPYG4GgQpw73e9hkHP579Lc=', 'Content-Length': '0'} 2022-09-12T19:48:10.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ad0431f728d34d1-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.787 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:10.788 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:10.788 DEBUG:boto:path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:10.789 DEBUG:boto:auth_path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:10.789 DEBUG:boto:Method: GET 2022-09-12T19:48:10.790 DEBUG:boto:Path: /jdpiak-72/g?versionId=null 2022-09-12T19:48:10.790 DEBUG:boto:Data: 2022-09-12T19:48:10.791 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.791 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.792 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.792 DEBUG:boto:Params: {} 2022-09-12T19:48:10.793 DEBUG:boto:Token: None 2022-09-12T19:48:10.794 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/g?versionId=null 2022-09-12T19:48:10.794 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FjghIPYG4GgQpw73e9hkHP579Lc= 2022-09-12T19:48:10.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FjghIPYG4GgQpw73e9hkHP579Lc=', 'Content-Length': '0'} 2022-09-12T19:48:10.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009eee2609ce508b74-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.798 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:10.799 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:10.800 DEBUG:boto:path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:10.800 DEBUG:boto:auth_path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:10.801 DEBUG:boto:Method: HEAD 2022-09-12T19:48:10.802 DEBUG:boto:Path: /jdpiak-72/g?versionId=null 2022-09-12T19:48:10.802 DEBUG:boto:Data: 2022-09-12T19:48:10.803 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.803 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.804 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.804 DEBUG:boto:Params: {} 2022-09-12T19:48:10.805 DEBUG:boto:Token: None 2022-09-12T19:48:10.805 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/g?versionId=null 2022-09-12T19:48:10.806 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E2y1sKuUsHCJFNM16MKVq8VRbA0= 2022-09-12T19:48:10.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E2y1sKuUsHCJFNM16MKVq8VRbA0=', 'Content-Length': '0'} 2022-09-12T19:48:10.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d163695865b09ab-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.810 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:10.811 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:10.811 DEBUG:boto:path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:10.812 DEBUG:boto:auth_path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:10.813 DEBUG:boto:Method: HEAD 2022-09-12T19:48:10.814 DEBUG:boto:Path: /jdpiak-72/g?versionId=null 2022-09-12T19:48:10.814 DEBUG:boto:Data: 2022-09-12T19:48:10.815 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.816 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.817 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.818 DEBUG:boto:Params: {} 2022-09-12T19:48:10.819 DEBUG:boto:Token: None 2022-09-12T19:48:10.820 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/g?versionId=null 2022-09-12T19:48:10.821 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E2y1sKuUsHCJFNM16MKVq8VRbA0= 2022-09-12T19:48:10.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E2y1sKuUsHCJFNM16MKVq8VRbA0=', 'Content-Length': '0'} 2022-09-12T19:48:10.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005352c5e17130f3bf-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:48:10.825 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:10.826 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:10.826 DEBUG:boto:path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:10.827 DEBUG:boto:auth_path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:10.827 DEBUG:boto:Method: GET 2022-09-12T19:48:10.828 DEBUG:boto:Path: /jdpiak-72/g?versionId=null 2022-09-12T19:48:10.828 DEBUG:boto:Data: 2022-09-12T19:48:10.829 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.830 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.830 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.831 DEBUG:boto:Params: {} 2022-09-12T19:48:10.832 DEBUG:boto:Token: None 2022-09-12T19:48:10.833 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/g?versionId=null 2022-09-12T19:48:10.834 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FjghIPYG4GgQpw73e9hkHP579Lc= 2022-09-12T19:48:10.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FjghIPYG4GgQpw73e9hkHP579Lc=', 'Content-Length': '0'} 2022-09-12T19:48:10.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005587ad3226da04c7-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.837 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:10.838 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:10.839 DEBUG:boto:path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:10.839 DEBUG:boto:auth_path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:10.840 DEBUG:boto:Method: GET 2022-09-12T19:48:10.840 DEBUG:boto:Path: /jdpiak-72/g?versionId=null 2022-09-12T19:48:10.841 DEBUG:boto:Data: 2022-09-12T19:48:10.841 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.842 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.842 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.843 DEBUG:boto:Params: {} 2022-09-12T19:48:10.843 DEBUG:boto:Token: None 2022-09-12T19:48:10.844 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/g?versionId=null 2022-09-12T19:48:10.844 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FjghIPYG4GgQpw73e9hkHP579Lc= 2022-09-12T19:48:10.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FjghIPYG4GgQpw73e9hkHP579Lc=', 'Content-Length': '0'} 2022-09-12T19:48:10.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035deaccc84cb532a-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.848 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:10.848 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:10.849 DEBUG:boto:path=/jdpiak-72/?prefix=g 2022-09-12T19:48:10.849 DEBUG:boto:auth_path=/jdpiak-72/?prefix=g 2022-09-12T19:48:10.850 DEBUG:boto:Method: GET 2022-09-12T19:48:10.851 DEBUG:boto:Path: /jdpiak-72/?prefix=g 2022-09-12T19:48:10.851 DEBUG:boto:Data: 2022-09-12T19:48:10.852 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.852 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.853 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.853 DEBUG:boto:Params: {} 2022-09-12T19:48:10.854 DEBUG:boto:Token: None 2022-09-12T19:48:10.855 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/ 2022-09-12T19:48:10.855 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo= 2022-09-12T19:48:10.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo=', 'Content-Length': '0'} 2022-09-12T19:48:10.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056fd0b43a1ea2bd5-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.859 DEBUG:boto:b'jdpiak-72g1000falseg2022-09-12T19:45:20.920Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:10.860 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:10.861 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:10.861 DEBUG:boto:path=/jdpiak-72/?prefix=g 2022-09-12T19:48:10.862 DEBUG:boto:auth_path=/jdpiak-72/?prefix=g 2022-09-12T19:48:10.862 DEBUG:boto:Method: GET 2022-09-12T19:48:10.863 DEBUG:boto:Path: /jdpiak-72/?prefix=g 2022-09-12T19:48:10.864 DEBUG:boto:Data: 2022-09-12T19:48:10.864 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.865 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.865 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.866 DEBUG:boto:Params: {} 2022-09-12T19:48:10.867 DEBUG:boto:Token: None 2022-09-12T19:48:10.867 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/ 2022-09-12T19:48:10.868 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo= 2022-09-12T19:48:10.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo=', 'Content-Length': '0'} 2022-09-12T19:48:10.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc326fe255d8a0ea-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.871 DEBUG:boto:b'jdpiak-72g1000falseg2022-09-12T19:45:20.920Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:10.872 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:10.873 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:10.873 DEBUG:boto:Method: HEAD 2022-09-12T19:48:10.874 DEBUG:boto:Path: /jdpiak-72/g 2022-09-12T19:48:10.874 DEBUG:boto:Data: 2022-09-12T19:48:10.875 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.875 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.876 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.876 DEBUG:boto:Params: {} 2022-09-12T19:48:10.877 DEBUG:boto:Token: None 2022-09-12T19:48:10.878 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/g 2022-09-12T19:48:10.878 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SH6uWq8aTHJmUd3F/a1ODZlWvRk= 2022-09-12T19:48:10.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SH6uWq8aTHJmUd3F/a1ODZlWvRk=', 'Content-Length': '0'} 2022-09-12T19:48:10.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007540368a407a640c-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.882 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:10.882 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:10.883 DEBUG:boto:Method: HEAD 2022-09-12T19:48:10.883 DEBUG:boto:Path: /jdpiak-72/g 2022-09-12T19:48:10.884 DEBUG:boto:Data: 2022-09-12T19:48:10.884 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.885 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.885 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.886 DEBUG:boto:Params: {} 2022-09-12T19:48:10.887 DEBUG:boto:Token: None 2022-09-12T19:48:10.887 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/g 2022-09-12T19:48:10.888 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SH6uWq8aTHJmUd3F/a1ODZlWvRk= 2022-09-12T19:48:10.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SH6uWq8aTHJmUd3F/a1ODZlWvRk=', 'Content-Length': '0'} 2022-09-12T19:48:10.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c41b570cad9b17c-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:48:10.892 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:10.892 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:10.893 DEBUG:boto:Method: GET 2022-09-12T19:48:10.893 DEBUG:boto:Path: /jdpiak-72/g 2022-09-12T19:48:10.894 DEBUG:boto:Data: 2022-09-12T19:48:10.895 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.895 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.896 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.896 DEBUG:boto:Params: {} 2022-09-12T19:48:10.897 DEBUG:boto:Token: None 2022-09-12T19:48:10.897 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/g 2022-09-12T19:48:10.898 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ofrEuWRffGXKYA1OgdxwaLpeRu0= 2022-09-12T19:48:10.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ofrEuWRffGXKYA1OgdxwaLpeRu0=', 'Content-Length': '0'} 2022-09-12T19:48:10.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e271be00291ea668-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.946 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:10.946 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:10.947 DEBUG:boto:Method: GET 2022-09-12T19:48:10.948 DEBUG:boto:Path: /jdpiak-72/g 2022-09-12T19:48:10.948 DEBUG:boto:Data: 2022-09-12T19:48:10.949 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.949 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.950 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.951 DEBUG:boto:Params: {} 2022-09-12T19:48:10.951 DEBUG:boto:Token: None 2022-09-12T19:48:10.952 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/g 2022-09-12T19:48:10.953 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ofrEuWRffGXKYA1OgdxwaLpeRu0= 2022-09-12T19:48:10.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ofrEuWRffGXKYA1OgdxwaLpeRu0=', 'Content-Length': '0'} 2022-09-12T19:48:10.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009aab08613abcbae0-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.956 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:10.957 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:10.958 DEBUG:boto:path=/jdpiak-72/?prefix=g 2022-09-12T19:48:10.958 DEBUG:boto:auth_path=/jdpiak-72/?prefix=g 2022-09-12T19:48:10.959 DEBUG:boto:Method: GET 2022-09-12T19:48:10.959 DEBUG:boto:Path: /jdpiak-72/?prefix=g 2022-09-12T19:48:10.960 DEBUG:boto:Data: 2022-09-12T19:48:10.960 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.961 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.961 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.962 DEBUG:boto:Params: {} 2022-09-12T19:48:10.963 DEBUG:boto:Token: None 2022-09-12T19:48:10.963 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/ 2022-09-12T19:48:10.964 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo= 2022-09-12T19:48:10.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo=', 'Content-Length': '0'} 2022-09-12T19:48:10.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009edd943d4eeac0c6-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.968 DEBUG:boto:b'jdpiak-72g1000falseg2022-09-12T19:45:20.920Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:10.969 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:10.970 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:10.970 DEBUG:boto:path=/jdpiak-72/?prefix=g 2022-09-12T19:48:10.971 DEBUG:boto:auth_path=/jdpiak-72/?prefix=g 2022-09-12T19:48:10.971 DEBUG:boto:Method: GET 2022-09-12T19:48:10.972 DEBUG:boto:Path: /jdpiak-72/?prefix=g 2022-09-12T19:48:10.972 DEBUG:boto:Data: 2022-09-12T19:48:10.973 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.974 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.974 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.975 DEBUG:boto:Params: {} 2022-09-12T19:48:10.975 DEBUG:boto:Token: None 2022-09-12T19:48:10.976 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/ 2022-09-12T19:48:10.976 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo= 2022-09-12T19:48:10.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UzAhBcIKHsiRL1PjECMwDlI4cNo=', 'Content-Length': '0'} 2022-09-12T19:48:10.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a4cac38364abc66-00631f8cfa-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.980 DEBUG:boto:b'jdpiak-72g1000falseg2022-09-12T19:45:20.920Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:10.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:48:10.981 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:10.982 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:10.983 DEBUG:boto:path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:10.983 DEBUG:boto:auth_path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:10.984 DEBUG:boto:Method: GET 2022-09-12T19:48:10.984 DEBUG:boto:Path: /jdpiak-72/h?versionId=null 2022-09-12T19:48:10.985 DEBUG:boto:Data: 2022-09-12T19:48:10.985 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.986 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:10.986 DEBUG:boto:Port: 8001 2022-09-12T19:48:10.987 DEBUG:boto:Params: {} 2022-09-12T19:48:10.988 DEBUG:boto:Token: None 2022-09-12T19:48:10.988 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/h?versionId=null 2022-09-12T19:48:10.989 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:71j/8BqLYTqs7Fxiv0CmndKA104= 2022-09-12T19:48:10.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:71j/8BqLYTqs7Fxiv0CmndKA104=', 'Content-Length': '0'} 2022-09-12T19:48:10.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c0dc57aa80b722b-00631f8cfa-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:10.993 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:10.993 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:10.994 DEBUG:boto:path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:10.994 DEBUG:boto:auth_path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:10.995 DEBUG:boto:Method: GET 2022-09-12T19:48:10.995 DEBUG:boto:Path: /jdpiak-72/h?versionId=null 2022-09-12T19:48:10.996 DEBUG:boto:Data: 2022-09-12T19:48:10.996 DEBUG:boto:Headers: {} 2022-09-12T19:48:10.997 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:10.997 DEBUG:boto:Port: 8000 2022-09-12T19:48:10.998 DEBUG:boto:Params: {} 2022-09-12T19:48:10.998 DEBUG:boto:Token: None 2022-09-12T19:48:10.999 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:10 GMT /jdpiak-72/h?versionId=null 2022-09-12T19:48:11.000 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:71j/8BqLYTqs7Fxiv0CmndKA104= 2022-09-12T19:48:11.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:71j/8BqLYTqs7Fxiv0CmndKA104=', 'Content-Length': '0'} 2022-09-12T19:48:11.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7bf733d75528188-00631f8cfb-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.003 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:11.004 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:11.005 DEBUG:boto:path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:11.005 DEBUG:boto:auth_path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:11.006 DEBUG:boto:Method: HEAD 2022-09-12T19:48:11.006 DEBUG:boto:Path: /jdpiak-72/h?versionId=null 2022-09-12T19:48:11.007 DEBUG:boto:Data: 2022-09-12T19:48:11.007 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.008 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.008 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.009 DEBUG:boto:Params: {} 2022-09-12T19:48:11.010 DEBUG:boto:Token: None 2022-09-12T19:48:11.010 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/h?versionId=null 2022-09-12T19:48:11.011 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:llYUwZTd2jT4tGPz+NeJb7P6Mho= 2022-09-12T19:48:11.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:llYUwZTd2jT4tGPz+NeJb7P6Mho=', 'Content-Length': '0'} 2022-09-12T19:48:11.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061a2db748f61b991-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.014 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:11.015 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:11.015 DEBUG:boto:path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:11.016 DEBUG:boto:auth_path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:11.016 DEBUG:boto:Method: HEAD 2022-09-12T19:48:11.017 DEBUG:boto:Path: /jdpiak-72/h?versionId=null 2022-09-12T19:48:11.017 DEBUG:boto:Data: 2022-09-12T19:48:11.018 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.019 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:11.020 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.020 DEBUG:boto:Params: {} 2022-09-12T19:48:11.021 DEBUG:boto:Token: None 2022-09-12T19:48:11.021 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/h?versionId=null 2022-09-12T19:48:11.022 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:llYUwZTd2jT4tGPz+NeJb7P6Mho= 2022-09-12T19:48:11.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:llYUwZTd2jT4tGPz+NeJb7P6Mho=', 'Content-Length': '0'} 2022-09-12T19:48:11.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bbc95abe51a9568-00631f8cfb-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:48:11.027 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:11.027 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:11.028 DEBUG:boto:path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:11.028 DEBUG:boto:auth_path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:11.029 DEBUG:boto:Method: GET 2022-09-12T19:48:11.030 DEBUG:boto:Path: /jdpiak-72/h?versionId=null 2022-09-12T19:48:11.031 DEBUG:boto:Data: 2022-09-12T19:48:11.031 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.032 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.032 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.033 DEBUG:boto:Params: {} 2022-09-12T19:48:11.034 DEBUG:boto:Token: None 2022-09-12T19:48:11.034 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/h?versionId=null 2022-09-12T19:48:11.035 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E4FJ6/tnvLEZVB+rRJdajVrAFPI= 2022-09-12T19:48:11.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E4FJ6/tnvLEZVB+rRJdajVrAFPI=', 'Content-Length': '0'} 2022-09-12T19:48:11.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043c398a4ada79927-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.039 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:11.039 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:11.040 DEBUG:boto:path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:11.040 DEBUG:boto:auth_path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:11.041 DEBUG:boto:Method: GET 2022-09-12T19:48:11.041 DEBUG:boto:Path: /jdpiak-72/h?versionId=null 2022-09-12T19:48:11.042 DEBUG:boto:Data: 2022-09-12T19:48:11.043 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.043 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:11.044 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.044 DEBUG:boto:Params: {} 2022-09-12T19:48:11.045 DEBUG:boto:Token: None 2022-09-12T19:48:11.045 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/h?versionId=null 2022-09-12T19:48:11.046 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E4FJ6/tnvLEZVB+rRJdajVrAFPI= 2022-09-12T19:48:11.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E4FJ6/tnvLEZVB+rRJdajVrAFPI=', 'Content-Length': '0'} 2022-09-12T19:48:11.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9b8eb214fc8e7e0-00631f8cfb-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.090 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:11.091 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:11.091 DEBUG:boto:path=/jdpiak-72/?prefix=h 2022-09-12T19:48:11.092 DEBUG:boto:auth_path=/jdpiak-72/?prefix=h 2022-09-12T19:48:11.092 DEBUG:boto:Method: GET 2022-09-12T19:48:11.093 DEBUG:boto:Path: /jdpiak-72/?prefix=h 2022-09-12T19:48:11.093 DEBUG:boto:Data: 2022-09-12T19:48:11.094 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.094 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.095 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.095 DEBUG:boto:Params: {} 2022-09-12T19:48:11.096 DEBUG:boto:Token: None 2022-09-12T19:48:11.096 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/ 2022-09-12T19:48:11.097 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE= 2022-09-12T19:48:11.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE=', 'Content-Length': '0'} 2022-09-12T19:48:11.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000855bf3ce9a597986-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.101 DEBUG:boto:b'jdpiak-72h1000falseh2022-09-12T19:45:20.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:11.102 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:11.102 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:11.103 DEBUG:boto:path=/jdpiak-72/?prefix=h 2022-09-12T19:48:11.104 DEBUG:boto:auth_path=/jdpiak-72/?prefix=h 2022-09-12T19:48:11.104 DEBUG:boto:Method: GET 2022-09-12T19:48:11.105 DEBUG:boto:Path: /jdpiak-72/?prefix=h 2022-09-12T19:48:11.105 DEBUG:boto:Data: 2022-09-12T19:48:11.106 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.107 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:11.107 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.108 DEBUG:boto:Params: {} 2022-09-12T19:48:11.108 DEBUG:boto:Token: None 2022-09-12T19:48:11.109 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/ 2022-09-12T19:48:11.109 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE= 2022-09-12T19:48:11.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE=', 'Content-Length': '0'} 2022-09-12T19:48:11.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fadbee4b8e8757c6-00631f8cfb-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.113 DEBUG:boto:b'jdpiak-72h1000falseh2022-09-12T19:45:20.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:11.114 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:11.114 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:11.115 DEBUG:boto:Method: HEAD 2022-09-12T19:48:11.115 DEBUG:boto:Path: /jdpiak-72/h 2022-09-12T19:48:11.116 DEBUG:boto:Data: 2022-09-12T19:48:11.116 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.117 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.117 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.118 DEBUG:boto:Params: {} 2022-09-12T19:48:11.118 DEBUG:boto:Token: None 2022-09-12T19:48:11.119 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/h 2022-09-12T19:48:11.120 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7K2eZEMl2Xve5N6AozBo3V7hBJY= 2022-09-12T19:48:11.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7K2eZEMl2Xve5N6AozBo3V7hBJY=', 'Content-Length': '0'} 2022-09-12T19:48:11.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037cd9ebb59aa913c-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.123 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:11.123 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:11.124 DEBUG:boto:Method: HEAD 2022-09-12T19:48:11.124 DEBUG:boto:Path: /jdpiak-72/h 2022-09-12T19:48:11.125 DEBUG:boto:Data: 2022-09-12T19:48:11.125 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.126 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:11.126 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.127 DEBUG:boto:Params: {} 2022-09-12T19:48:11.127 DEBUG:boto:Token: None 2022-09-12T19:48:11.128 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/h 2022-09-12T19:48:11.128 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7K2eZEMl2Xve5N6AozBo3V7hBJY= 2022-09-12T19:48:11.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7K2eZEMl2Xve5N6AozBo3V7hBJY=', 'Content-Length': '0'} 2022-09-12T19:48:11.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073c3590c6d0ae219-00631f8cfb-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:48:11.132 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:11.132 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:11.133 DEBUG:boto:Method: GET 2022-09-12T19:48:11.134 DEBUG:boto:Path: /jdpiak-72/h 2022-09-12T19:48:11.134 DEBUG:boto:Data: 2022-09-12T19:48:11.135 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.135 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.136 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.136 DEBUG:boto:Params: {} 2022-09-12T19:48:11.137 DEBUG:boto:Token: None 2022-09-12T19:48:11.137 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/h 2022-09-12T19:48:11.138 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VesghZoh4Gza+iEi2z1MVqFO7Bg= 2022-09-12T19:48:11.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VesghZoh4Gza+iEi2z1MVqFO7Bg=', 'Content-Length': '0'} 2022-09-12T19:48:11.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af09017bb45f60ed-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.186 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:11.187 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:11.187 DEBUG:boto:Method: GET 2022-09-12T19:48:11.188 DEBUG:boto:Path: /jdpiak-72/h 2022-09-12T19:48:11.188 DEBUG:boto:Data: 2022-09-12T19:48:11.189 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.190 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:11.190 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.191 DEBUG:boto:Params: {} 2022-09-12T19:48:11.191 DEBUG:boto:Token: None 2022-09-12T19:48:11.192 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/h 2022-09-12T19:48:11.193 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VesghZoh4Gza+iEi2z1MVqFO7Bg= 2022-09-12T19:48:11.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VesghZoh4Gza+iEi2z1MVqFO7Bg=', 'Content-Length': '0'} 2022-09-12T19:48:11.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b70331f1898a5df-00631f8cfb-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.197 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:11.198 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:11.198 DEBUG:boto:path=/jdpiak-72/?prefix=h 2022-09-12T19:48:11.199 DEBUG:boto:auth_path=/jdpiak-72/?prefix=h 2022-09-12T19:48:11.199 DEBUG:boto:Method: GET 2022-09-12T19:48:11.200 DEBUG:boto:Path: /jdpiak-72/?prefix=h 2022-09-12T19:48:11.200 DEBUG:boto:Data: 2022-09-12T19:48:11.201 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.201 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.202 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.202 DEBUG:boto:Params: {} 2022-09-12T19:48:11.203 DEBUG:boto:Token: None 2022-09-12T19:48:11.204 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/ 2022-09-12T19:48:11.204 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE= 2022-09-12T19:48:11.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE=', 'Content-Length': '0'} 2022-09-12T19:48:11.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000812bf7ddd68e4e90-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.209 DEBUG:boto:b'jdpiak-72h1000falseh2022-09-12T19:45:20.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:11.210 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:11.210 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:11.211 DEBUG:boto:path=/jdpiak-72/?prefix=h 2022-09-12T19:48:11.211 DEBUG:boto:auth_path=/jdpiak-72/?prefix=h 2022-09-12T19:48:11.212 DEBUG:boto:Method: GET 2022-09-12T19:48:11.212 DEBUG:boto:Path: /jdpiak-72/?prefix=h 2022-09-12T19:48:11.213 DEBUG:boto:Data: 2022-09-12T19:48:11.214 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.214 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:11.215 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.215 DEBUG:boto:Params: {} 2022-09-12T19:48:11.216 DEBUG:boto:Token: None 2022-09-12T19:48:11.216 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/ 2022-09-12T19:48:11.217 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE= 2022-09-12T19:48:11.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE=', 'Content-Length': '0'} 2022-09-12T19:48:11.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000879cc4f1ec1b1a99-00631f8cfb-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.222 DEBUG:boto:b'jdpiak-72h1000falseh2022-09-12T19:45:20.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:11.223 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-72 zones={test-zone3, test-zone1} 2022-09-12T19:48:11.223 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-72 zones={test-zone3, test-zone2} 2022-09-12T19:48:11.224 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:11.224 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:11.225 DEBUG:boto:Method: HEAD 2022-09-12T19:48:11.225 DEBUG:boto:Path: /jdpiak-72/ 2022-09-12T19:48:11.226 DEBUG:boto:Data: 2022-09-12T19:48:11.226 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.227 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.227 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.228 DEBUG:boto:Params: {} 2022-09-12T19:48:11.228 DEBUG:boto:Token: None 2022-09-12T19:48:11.229 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/ 2022-09-12T19:48:11.230 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0GqHpiE2gYNtL64Inlx0jwHyWeE= 2022-09-12T19:48:11.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0GqHpiE2gYNtL64Inlx0jwHyWeE=', 'Content-Length': '0'} 2022-09-12T19:48:11.233 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000965f8a5e5b6ec09a-00631f8cfb-15c4-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.234 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:11.234 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:11.235 DEBUG:boto:Method: HEAD 2022-09-12T19:48:11.235 DEBUG:boto:Path: /jdpiak-72/ 2022-09-12T19:48:11.236 DEBUG:boto:Data: 2022-09-12T19:48:11.236 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.237 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:11.238 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.238 DEBUG:boto:Params: {} 2022-09-12T19:48:11.239 DEBUG:boto:Token: None 2022-09-12T19:48:11.239 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/ 2022-09-12T19:48:11.240 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0GqHpiE2gYNtL64Inlx0jwHyWeE= 2022-09-12T19:48:11.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0GqHpiE2gYNtL64Inlx0jwHyWeE=', 'Content-Length': '0'} 2022-09-12T19:48:11.243 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee5b9f4448ffddee-00631f8cfb-15e1-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.244 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T19:48:11.244 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:11.245 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:11.245 DEBUG:boto:path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:11.246 DEBUG:boto:auth_path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:11.247 DEBUG:boto:Method: GET 2022-09-12T19:48:11.247 DEBUG:boto:Path: /jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:11.248 DEBUG:boto:Data: 2022-09-12T19:48:11.248 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.249 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.249 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.250 DEBUG:boto:Params: {} 2022-09-12T19:48:11.251 DEBUG:boto:Token: None 2022-09-12T19:48:11.251 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/?versions 2022-09-12T19:48:11.252 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u6Rj/ru3R26GWTSY5Nh098a5Ss0= 2022-09-12T19:48:11.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u6Rj/ru3R26GWTSY5Nh098a5Ss0=', 'Content-Length': '0'} 2022-09-12T19:48:11.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000837f3df01f7e58a-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.256 DEBUG:boto:b'jdpiak-72999falseanulltrue2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:45:20.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:45:20.891Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:45:20.920Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:45:20.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:11.257 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:48:11.258 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:48:11.258 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:48:11.259 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:48:11.259 DEBUG:tasks.rgw_multi.zone_rados:o=e 2022-09-12T19:48:11.259 DEBUG:tasks.rgw_multi.zone_rados:o=f 2022-09-12T19:48:11.260 DEBUG:tasks.rgw_multi.zone_rados:o=g 2022-09-12T19:48:11.260 DEBUG:tasks.rgw_multi.zone_rados:o=h 2022-09-12T19:48:11.261 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T19:48:11.261 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:11.262 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:11.263 DEBUG:boto:path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:11.263 DEBUG:boto:auth_path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:11.264 DEBUG:boto:Method: GET 2022-09-12T19:48:11.264 DEBUG:boto:Path: /jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:11.265 DEBUG:boto:Data: 2022-09-12T19:48:11.265 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.266 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:11.266 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.267 DEBUG:boto:Params: {} 2022-09-12T19:48:11.267 DEBUG:boto:Token: None 2022-09-12T19:48:11.268 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/?versions 2022-09-12T19:48:11.269 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u6Rj/ru3R26GWTSY5Nh098a5Ss0= 2022-09-12T19:48:11.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u6Rj/ru3R26GWTSY5Nh098a5Ss0=', 'Content-Length': '0'} 2022-09-12T19:48:11.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053aa4c9dd4705c16-00631f8cfb-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.273 DEBUG:boto:b'jdpiak-72999falseanulltrue2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:45:20.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:45:20.891Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:45:20.920Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:45:20.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:11.275 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-12T19:48:11.276 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-12T19:48:11.278 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-12T19:48:11.279 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-12T19:48:11.279 DEBUG:tasks.rgw_multi.zone_rados:o=e 2022-09-12T19:48:11.280 DEBUG:tasks.rgw_multi.zone_rados:o=f 2022-09-12T19:48:11.281 DEBUG:tasks.rgw_multi.zone_rados:o=g 2022-09-12T19:48:11.283 DEBUG:tasks.rgw_multi.zone_rados:o=h 2022-09-12T19:48:11.285 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:11.286 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:11.288 DEBUG:boto:path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:11.290 DEBUG:boto:auth_path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:11.291 DEBUG:boto:Method: GET 2022-09-12T19:48:11.293 DEBUG:boto:Path: /jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:11.294 DEBUG:boto:Data: 2022-09-12T19:48:11.295 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.296 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.297 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.297 DEBUG:boto:Params: {} 2022-09-12T19:48:11.298 DEBUG:boto:Token: None 2022-09-12T19:48:11.299 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/?versions 2022-09-12T19:48:11.299 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u6Rj/ru3R26GWTSY5Nh098a5Ss0= 2022-09-12T19:48:11.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u6Rj/ru3R26GWTSY5Nh098a5Ss0=', 'Content-Length': '0'} 2022-09-12T19:48:11.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1483392d1463697-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.303 DEBUG:boto:b'jdpiak-72999falseanulltrue2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:45:20.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:45:20.891Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:45:20.920Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:45:20.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:11.305 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:11.305 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:11.306 DEBUG:boto:path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:11.306 DEBUG:boto:auth_path=/jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:11.307 DEBUG:boto:Method: GET 2022-09-12T19:48:11.307 DEBUG:boto:Path: /jdpiak-72/?versions&max-keys=999 2022-09-12T19:48:11.308 DEBUG:boto:Data: 2022-09-12T19:48:11.308 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.309 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:11.309 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.310 DEBUG:boto:Params: {} 2022-09-12T19:48:11.310 DEBUG:boto:Token: None 2022-09-12T19:48:11.311 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/?versions 2022-09-12T19:48:11.312 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u6Rj/ru3R26GWTSY5Nh098a5Ss0= 2022-09-12T19:48:11.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u6Rj/ru3R26GWTSY5Nh098a5Ss0=', 'Content-Length': '0'} 2022-09-12T19:48:11.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000739101d59bdc86ff-00631f8cfb-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.315 DEBUG:boto:b'jdpiak-72999falseanulltrue2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-12T19:45:20.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-12T19:45:20.891Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-12T19:45:20.920Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-12T19:45:20.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:11.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:48:11.317 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:11.318 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:11.318 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:11.319 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:11.319 DEBUG:boto:Method: GET 2022-09-12T19:48:11.320 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:48:11.320 DEBUG:boto:Data: 2022-09-12T19:48:11.321 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.321 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.322 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.322 DEBUG:boto:Params: {} 2022-09-12T19:48:11.323 DEBUG:boto:Token: None 2022-09-12T19:48:11.324 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:48:11.324 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0NK3SDzEXMXKSho/nAAGkj29gtE= 2022-09-12T19:48:11.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0NK3SDzEXMXKSho/nAAGkj29gtE=', 'Content-Length': '0'} 2022-09-12T19:48:11.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2af4ea81bf01d28-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.328 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:11.329 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:11.329 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:11.330 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:11.330 DEBUG:boto:Method: GET 2022-09-12T19:48:11.331 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:48:11.331 DEBUG:boto:Data: 2022-09-12T19:48:11.332 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.333 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:11.333 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.334 DEBUG:boto:Params: {} 2022-09-12T19:48:11.334 DEBUG:boto:Token: None 2022-09-12T19:48:11.335 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:48:11.336 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0NK3SDzEXMXKSho/nAAGkj29gtE= 2022-09-12T19:48:11.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0NK3SDzEXMXKSho/nAAGkj29gtE=', 'Content-Length': '0'} 2022-09-12T19:48:11.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a8bdbb55ddc11e7-00631f8cfb-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.382 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:11.383 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:11.384 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:11.384 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:11.385 DEBUG:boto:Method: HEAD 2022-09-12T19:48:11.385 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:48:11.386 DEBUG:boto:Data: 2022-09-12T19:48:11.386 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.387 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.387 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.388 DEBUG:boto:Params: {} 2022-09-12T19:48:11.388 DEBUG:boto:Token: None 2022-09-12T19:48:11.389 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:48:11.390 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s5uf/ZdC4IvCfW+D+QWuZGxri98= 2022-09-12T19:48:11.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s5uf/ZdC4IvCfW+D+QWuZGxri98=', 'Content-Length': '0'} 2022-09-12T19:48:11.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ef00474c3a98a34-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.395 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:11.395 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:11.396 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:11.397 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:11.398 DEBUG:boto:Method: HEAD 2022-09-12T19:48:11.398 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:48:11.399 DEBUG:boto:Data: 2022-09-12T19:48:11.400 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.401 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:11.401 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.402 DEBUG:boto:Params: {} 2022-09-12T19:48:11.404 DEBUG:boto:Token: None 2022-09-12T19:48:11.405 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:48:11.405 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s5uf/ZdC4IvCfW+D+QWuZGxri98= 2022-09-12T19:48:11.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s5uf/ZdC4IvCfW+D+QWuZGxri98=', 'Content-Length': '0'} 2022-09-12T19:48:11.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c474789c636baf5-00631f8cfb-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:48:11.410 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:11.411 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:11.411 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:11.412 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:11.412 DEBUG:boto:Method: GET 2022-09-12T19:48:11.413 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:48:11.413 DEBUG:boto:Data: 2022-09-12T19:48:11.415 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.415 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.416 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.416 DEBUG:boto:Params: {} 2022-09-12T19:48:11.417 DEBUG:boto:Token: None 2022-09-12T19:48:11.417 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:48:11.418 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0NK3SDzEXMXKSho/nAAGkj29gtE= 2022-09-12T19:48:11.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0NK3SDzEXMXKSho/nAAGkj29gtE=', 'Content-Length': '0'} 2022-09-12T19:48:11.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a22cc8d4cbb97e76-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.422 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:11.422 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:11.423 DEBUG:boto:path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:11.423 DEBUG:boto:auth_path=/jdpiak-72/a?versionId=null 2022-09-12T19:48:11.424 DEBUG:boto:Method: GET 2022-09-12T19:48:11.424 DEBUG:boto:Path: /jdpiak-72/a?versionId=null 2022-09-12T19:48:11.425 DEBUG:boto:Data: 2022-09-12T19:48:11.425 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.426 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:11.427 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.427 DEBUG:boto:Params: {} 2022-09-12T19:48:11.428 DEBUG:boto:Token: None 2022-09-12T19:48:11.428 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/a?versionId=null 2022-09-12T19:48:11.429 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0NK3SDzEXMXKSho/nAAGkj29gtE= 2022-09-12T19:48:11.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0NK3SDzEXMXKSho/nAAGkj29gtE=', 'Content-Length': '0'} 2022-09-12T19:48:11.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057657d27b26305ce-00631f8cfb-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.433 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:11.434 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:11.434 DEBUG:boto:path=/jdpiak-72/?prefix=a 2022-09-12T19:48:11.435 DEBUG:boto:auth_path=/jdpiak-72/?prefix=a 2022-09-12T19:48:11.436 DEBUG:boto:Method: GET 2022-09-12T19:48:11.436 DEBUG:boto:Path: /jdpiak-72/?prefix=a 2022-09-12T19:48:11.437 DEBUG:boto:Data: 2022-09-12T19:48:11.437 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.438 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.438 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.439 DEBUG:boto:Params: {} 2022-09-12T19:48:11.439 DEBUG:boto:Token: None 2022-09-12T19:48:11.440 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/ 2022-09-12T19:48:11.441 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE= 2022-09-12T19:48:11.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE=', 'Content-Length': '0'} 2022-09-12T19:48:11.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008abae1723b64dacb-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.445 DEBUG:boto:b'jdpiak-72a1000falsea2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:11.446 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:11.447 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:11.447 DEBUG:boto:path=/jdpiak-72/?prefix=a 2022-09-12T19:48:11.448 DEBUG:boto:auth_path=/jdpiak-72/?prefix=a 2022-09-12T19:48:11.448 DEBUG:boto:Method: GET 2022-09-12T19:48:11.449 DEBUG:boto:Path: /jdpiak-72/?prefix=a 2022-09-12T19:48:11.450 DEBUG:boto:Data: 2022-09-12T19:48:11.450 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.451 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:11.451 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.452 DEBUG:boto:Params: {} 2022-09-12T19:48:11.452 DEBUG:boto:Token: None 2022-09-12T19:48:11.453 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/ 2022-09-12T19:48:11.453 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE= 2022-09-12T19:48:11.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE=', 'Content-Length': '0'} 2022-09-12T19:48:11.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b518e40016038f2-00631f8cfb-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.458 DEBUG:boto:b'jdpiak-72a1000falsea2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:11.459 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:11.459 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:11.460 DEBUG:boto:Method: HEAD 2022-09-12T19:48:11.461 DEBUG:boto:Path: /jdpiak-72/a 2022-09-12T19:48:11.462 DEBUG:boto:Data: 2022-09-12T19:48:11.462 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.463 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.463 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.464 DEBUG:boto:Params: {} 2022-09-12T19:48:11.465 DEBUG:boto:Token: None 2022-09-12T19:48:11.465 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/a 2022-09-12T19:48:11.466 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LkwfDq7GF0bbYz4BXAinFPJatXQ= 2022-09-12T19:48:11.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LkwfDq7GF0bbYz4BXAinFPJatXQ=', 'Content-Length': '0'} 2022-09-12T19:48:11.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050c5778e6543d423-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.470 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:11.470 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:11.471 DEBUG:boto:Method: HEAD 2022-09-12T19:48:11.471 DEBUG:boto:Path: /jdpiak-72/a 2022-09-12T19:48:11.472 DEBUG:boto:Data: 2022-09-12T19:48:11.472 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.473 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:11.474 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.474 DEBUG:boto:Params: {} 2022-09-12T19:48:11.475 DEBUG:boto:Token: None 2022-09-12T19:48:11.476 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/a 2022-09-12T19:48:11.476 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LkwfDq7GF0bbYz4BXAinFPJatXQ= 2022-09-12T19:48:11.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LkwfDq7GF0bbYz4BXAinFPJatXQ=', 'Content-Length': '0'} 2022-09-12T19:48:11.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d3cc4d56c4c82a1-00631f8cfb-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-12T19:48:11.481 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:11.482 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:11.483 DEBUG:boto:Method: GET 2022-09-12T19:48:11.483 DEBUG:boto:Path: /jdpiak-72/a 2022-09-12T19:48:11.484 DEBUG:boto:Data: 2022-09-12T19:48:11.485 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.486 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.487 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.488 DEBUG:boto:Params: {} 2022-09-12T19:48:11.489 DEBUG:boto:Token: None 2022-09-12T19:48:11.489 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/a 2022-09-12T19:48:11.490 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Gi3Uk3GVy2Fc/UVjN4tf7sijEdE= 2022-09-12T19:48:11.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Gi3Uk3GVy2Fc/UVjN4tf7sijEdE=', 'Content-Length': '0'} 2022-09-12T19:48:11.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b69e9eaf751075bd-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.494 DEBUG:boto:path=/jdpiak-72/a 2022-09-12T19:48:11.494 DEBUG:boto:auth_path=/jdpiak-72/a 2022-09-12T19:48:11.495 DEBUG:boto:Method: GET 2022-09-12T19:48:11.496 DEBUG:boto:Path: /jdpiak-72/a 2022-09-12T19:48:11.496 DEBUG:boto:Data: 2022-09-12T19:48:11.497 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.497 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:11.498 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.498 DEBUG:boto:Params: {} 2022-09-12T19:48:11.499 DEBUG:boto:Token: None 2022-09-12T19:48:11.500 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/a 2022-09-12T19:48:11.500 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Gi3Uk3GVy2Fc/UVjN4tf7sijEdE= 2022-09-12T19:48:11.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Gi3Uk3GVy2Fc/UVjN4tf7sijEdE=', 'Content-Length': '0'} 2022-09-12T19:48:11.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001146a17023074dbe-00631f8cfb-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.504 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:11.505 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:11.505 DEBUG:boto:path=/jdpiak-72/?prefix=a 2022-09-12T19:48:11.506 DEBUG:boto:auth_path=/jdpiak-72/?prefix=a 2022-09-12T19:48:11.506 DEBUG:boto:Method: GET 2022-09-12T19:48:11.507 DEBUG:boto:Path: /jdpiak-72/?prefix=a 2022-09-12T19:48:11.508 DEBUG:boto:Data: 2022-09-12T19:48:11.508 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.509 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.509 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.510 DEBUG:boto:Params: {} 2022-09-12T19:48:11.511 DEBUG:boto:Token: None 2022-09-12T19:48:11.511 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/ 2022-09-12T19:48:11.512 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE= 2022-09-12T19:48:11.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE=', 'Content-Length': '0'} 2022-09-12T19:48:11.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fba6959afcb96411-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.516 DEBUG:boto:b'jdpiak-72a1000falsea2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:11.517 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:11.518 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:11.518 DEBUG:boto:path=/jdpiak-72/?prefix=a 2022-09-12T19:48:11.519 DEBUG:boto:auth_path=/jdpiak-72/?prefix=a 2022-09-12T19:48:11.520 DEBUG:boto:Method: GET 2022-09-12T19:48:11.520 DEBUG:boto:Path: /jdpiak-72/?prefix=a 2022-09-12T19:48:11.521 DEBUG:boto:Data: 2022-09-12T19:48:11.521 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.522 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:11.522 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.523 DEBUG:boto:Params: {} 2022-09-12T19:48:11.523 DEBUG:boto:Token: None 2022-09-12T19:48:11.524 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/ 2022-09-12T19:48:11.525 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE= 2022-09-12T19:48:11.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE=', 'Content-Length': '0'} 2022-09-12T19:48:11.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004783d0001f8b6ff3-00631f8cfb-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.528 DEBUG:boto:b'jdpiak-72a1000falsea2022-09-12T19:44:35.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:11.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:48:11.530 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:11.530 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:11.531 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:11.531 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:11.532 DEBUG:boto:Method: GET 2022-09-12T19:48:11.532 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:48:11.533 DEBUG:boto:Data: 2022-09-12T19:48:11.534 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.534 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.535 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.535 DEBUG:boto:Params: {} 2022-09-12T19:48:11.536 DEBUG:boto:Token: None 2022-09-12T19:48:11.536 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:48:11.537 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8bYkfwczqb7GMDkXJXvX/CvcJ8c= 2022-09-12T19:48:11.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8bYkfwczqb7GMDkXJXvX/CvcJ8c=', 'Content-Length': '0'} 2022-09-12T19:48:11.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7b4de4c280c313d-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.586 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:11.587 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:11.587 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:11.588 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:11.588 DEBUG:boto:Method: GET 2022-09-12T19:48:11.589 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:48:11.590 DEBUG:boto:Data: 2022-09-12T19:48:11.590 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.591 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:11.591 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.592 DEBUG:boto:Params: {} 2022-09-12T19:48:11.592 DEBUG:boto:Token: None 2022-09-12T19:48:11.593 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:48:11.594 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8bYkfwczqb7GMDkXJXvX/CvcJ8c= 2022-09-12T19:48:11.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8bYkfwczqb7GMDkXJXvX/CvcJ8c=', 'Content-Length': '0'} 2022-09-12T19:48:11.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c8dc2aeeb9a8495-00631f8cfb-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.598 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:11.599 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:11.599 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:11.600 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:11.600 DEBUG:boto:Method: HEAD 2022-09-12T19:48:11.601 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:48:11.601 DEBUG:boto:Data: 2022-09-12T19:48:11.602 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.602 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.603 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.603 DEBUG:boto:Params: {} 2022-09-12T19:48:11.604 DEBUG:boto:Token: None 2022-09-12T19:48:11.605 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:48:11.605 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Qvne02HCcKC3gW0ucoLPWCmpwzc= 2022-09-12T19:48:11.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Qvne02HCcKC3gW0ucoLPWCmpwzc=', 'Content-Length': '0'} 2022-09-12T19:48:11.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000baf10b5ce280acd3-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.609 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:11.609 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:11.611 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:11.612 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:11.612 DEBUG:boto:Method: HEAD 2022-09-12T19:48:11.613 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:48:11.613 DEBUG:boto:Data: 2022-09-12T19:48:11.614 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.615 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:11.615 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.616 DEBUG:boto:Params: {} 2022-09-12T19:48:11.617 DEBUG:boto:Token: None 2022-09-12T19:48:11.617 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:48:11.618 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Qvne02HCcKC3gW0ucoLPWCmpwzc= 2022-09-12T19:48:11.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Qvne02HCcKC3gW0ucoLPWCmpwzc=', 'Content-Length': '0'} 2022-09-12T19:48:11.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000163c79a75fbc396-00631f8cfb-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:48:11.623 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:11.624 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:11.624 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:11.625 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:11.625 DEBUG:boto:Method: GET 2022-09-12T19:48:11.626 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:48:11.627 DEBUG:boto:Data: 2022-09-12T19:48:11.628 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.628 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.629 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.630 DEBUG:boto:Params: {} 2022-09-12T19:48:11.631 DEBUG:boto:Token: None 2022-09-12T19:48:11.632 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:48:11.633 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8bYkfwczqb7GMDkXJXvX/CvcJ8c= 2022-09-12T19:48:11.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8bYkfwczqb7GMDkXJXvX/CvcJ8c=', 'Content-Length': '0'} 2022-09-12T19:48:11.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002de187692b7c91b5-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.638 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:11.639 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:11.640 DEBUG:boto:path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:11.641 DEBUG:boto:auth_path=/jdpiak-72/b?versionId=null 2022-09-12T19:48:11.642 DEBUG:boto:Method: GET 2022-09-12T19:48:11.643 DEBUG:boto:Path: /jdpiak-72/b?versionId=null 2022-09-12T19:48:11.644 DEBUG:boto:Data: 2022-09-12T19:48:11.645 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.646 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:11.647 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.649 DEBUG:boto:Params: {} 2022-09-12T19:48:11.650 DEBUG:boto:Token: None 2022-09-12T19:48:11.651 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/b?versionId=null 2022-09-12T19:48:11.652 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8bYkfwczqb7GMDkXJXvX/CvcJ8c= 2022-09-12T19:48:11.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8bYkfwczqb7GMDkXJXvX/CvcJ8c=', 'Content-Length': '0'} 2022-09-12T19:48:11.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b997c9ee525bdb2-00631f8cfb-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.657 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:11.657 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:11.658 DEBUG:boto:path=/jdpiak-72/?prefix=b 2022-09-12T19:48:11.659 DEBUG:boto:auth_path=/jdpiak-72/?prefix=b 2022-09-12T19:48:11.659 DEBUG:boto:Method: GET 2022-09-12T19:48:11.660 DEBUG:boto:Path: /jdpiak-72/?prefix=b 2022-09-12T19:48:11.660 DEBUG:boto:Data: 2022-09-12T19:48:11.661 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.661 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.662 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.663 DEBUG:boto:Params: {} 2022-09-12T19:48:11.663 DEBUG:boto:Token: None 2022-09-12T19:48:11.664 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/ 2022-09-12T19:48:11.665 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE= 2022-09-12T19:48:11.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE=', 'Content-Length': '0'} 2022-09-12T19:48:11.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069d593416218ca3c-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.669 DEBUG:boto:b'jdpiak-72b1000falseb2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:11.671 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:11.672 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:11.673 DEBUG:boto:path=/jdpiak-72/?prefix=b 2022-09-12T19:48:11.673 DEBUG:boto:auth_path=/jdpiak-72/?prefix=b 2022-09-12T19:48:11.675 DEBUG:boto:Method: GET 2022-09-12T19:48:11.676 DEBUG:boto:Path: /jdpiak-72/?prefix=b 2022-09-12T19:48:11.677 DEBUG:boto:Data: 2022-09-12T19:48:11.678 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.678 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:11.679 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.680 DEBUG:boto:Params: {} 2022-09-12T19:48:11.681 DEBUG:boto:Token: None 2022-09-12T19:48:11.682 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/ 2022-09-12T19:48:11.682 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE= 2022-09-12T19:48:11.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE=', 'Content-Length': '0'} 2022-09-12T19:48:11.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b746fa8e4ce1c70-00631f8cfb-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.687 DEBUG:boto:b'jdpiak-72b1000falseb2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:11.688 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:11.689 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:11.689 DEBUG:boto:Method: HEAD 2022-09-12T19:48:11.690 DEBUG:boto:Path: /jdpiak-72/b 2022-09-12T19:48:11.691 DEBUG:boto:Data: 2022-09-12T19:48:11.692 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.692 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.693 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.693 DEBUG:boto:Params: {} 2022-09-12T19:48:11.694 DEBUG:boto:Token: None 2022-09-12T19:48:11.694 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/b 2022-09-12T19:48:11.695 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zzhbe9OILaZHeP8gt+EnFWiNGaU= 2022-09-12T19:48:11.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zzhbe9OILaZHeP8gt+EnFWiNGaU=', 'Content-Length': '0'} 2022-09-12T19:48:11.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9d2c32df0cfb0ae-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.699 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:11.700 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:11.700 DEBUG:boto:Method: HEAD 2022-09-12T19:48:11.701 DEBUG:boto:Path: /jdpiak-72/b 2022-09-12T19:48:11.701 DEBUG:boto:Data: 2022-09-12T19:48:11.702 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.703 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:11.704 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.705 DEBUG:boto:Params: {} 2022-09-12T19:48:11.706 DEBUG:boto:Token: None 2022-09-12T19:48:11.707 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/b 2022-09-12T19:48:11.708 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zzhbe9OILaZHeP8gt+EnFWiNGaU= 2022-09-12T19:48:11.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zzhbe9OILaZHeP8gt+EnFWiNGaU=', 'Content-Length': '0'} 2022-09-12T19:48:11.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c53de111ae125e5d-00631f8cfb-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-12T19:48:11.713 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:11.714 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:11.714 DEBUG:boto:Method: GET 2022-09-12T19:48:11.715 DEBUG:boto:Path: /jdpiak-72/b 2022-09-12T19:48:11.715 DEBUG:boto:Data: 2022-09-12T19:48:11.716 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.716 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.717 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.717 DEBUG:boto:Params: {} 2022-09-12T19:48:11.718 DEBUG:boto:Token: None 2022-09-12T19:48:11.719 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/b 2022-09-12T19:48:11.719 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7zIoee4vl8oeFxmHeM9iaePzO5Q= 2022-09-12T19:48:11.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7zIoee4vl8oeFxmHeM9iaePzO5Q=', 'Content-Length': '0'} 2022-09-12T19:48:11.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed8ed2a241158ea2-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.723 DEBUG:boto:path=/jdpiak-72/b 2022-09-12T19:48:11.723 DEBUG:boto:auth_path=/jdpiak-72/b 2022-09-12T19:48:11.724 DEBUG:boto:Method: GET 2022-09-12T19:48:11.725 DEBUG:boto:Path: /jdpiak-72/b 2022-09-12T19:48:11.725 DEBUG:boto:Data: 2022-09-12T19:48:11.726 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.726 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:11.727 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.727 DEBUG:boto:Params: {} 2022-09-12T19:48:11.728 DEBUG:boto:Token: None 2022-09-12T19:48:11.729 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/b 2022-09-12T19:48:11.729 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7zIoee4vl8oeFxmHeM9iaePzO5Q= 2022-09-12T19:48:11.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7zIoee4vl8oeFxmHeM9iaePzO5Q=', 'Content-Length': '0'} 2022-09-12T19:48:11.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9afb98498bd3974-00631f8cfb-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.778 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:11.779 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:11.780 DEBUG:boto:path=/jdpiak-72/?prefix=b 2022-09-12T19:48:11.781 DEBUG:boto:auth_path=/jdpiak-72/?prefix=b 2022-09-12T19:48:11.781 DEBUG:boto:Method: GET 2022-09-12T19:48:11.782 DEBUG:boto:Path: /jdpiak-72/?prefix=b 2022-09-12T19:48:11.782 DEBUG:boto:Data: 2022-09-12T19:48:11.783 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.783 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.784 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.784 DEBUG:boto:Params: {} 2022-09-12T19:48:11.785 DEBUG:boto:Token: None 2022-09-12T19:48:11.786 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/ 2022-09-12T19:48:11.786 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE= 2022-09-12T19:48:11.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE=', 'Content-Length': '0'} 2022-09-12T19:48:11.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8f57dca9658cd89-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.791 DEBUG:boto:b'jdpiak-72b1000falseb2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:11.791 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:11.792 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:11.793 DEBUG:boto:path=/jdpiak-72/?prefix=b 2022-09-12T19:48:11.793 DEBUG:boto:auth_path=/jdpiak-72/?prefix=b 2022-09-12T19:48:11.794 DEBUG:boto:Method: GET 2022-09-12T19:48:11.794 DEBUG:boto:Path: /jdpiak-72/?prefix=b 2022-09-12T19:48:11.795 DEBUG:boto:Data: 2022-09-12T19:48:11.795 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.796 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:11.796 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.797 DEBUG:boto:Params: {} 2022-09-12T19:48:11.798 DEBUG:boto:Token: None 2022-09-12T19:48:11.798 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/ 2022-09-12T19:48:11.799 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE= 2022-09-12T19:48:11.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE=', 'Content-Length': '0'} 2022-09-12T19:48:11.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f276ccbee120b70d-00631f8cfb-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.802 DEBUG:boto:b'jdpiak-72b1000falseb2022-09-12T19:44:35.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:11.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:48:11.804 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:11.804 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:11.805 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:11.805 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:11.806 DEBUG:boto:Method: GET 2022-09-12T19:48:11.806 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:48:11.807 DEBUG:boto:Data: 2022-09-12T19:48:11.808 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.808 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.809 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.810 DEBUG:boto:Params: {} 2022-09-12T19:48:11.811 DEBUG:boto:Token: None 2022-09-12T19:48:11.812 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:48:11.813 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uafWNPvjs+hSOhlq5HKXogmN+/s= 2022-09-12T19:48:11.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uafWNPvjs+hSOhlq5HKXogmN+/s=', 'Content-Length': '0'} 2022-09-12T19:48:11.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004679ae5ea36811b3-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.817 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:11.817 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:11.818 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:11.818 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:11.819 DEBUG:boto:Method: GET 2022-09-12T19:48:11.820 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:48:11.820 DEBUG:boto:Data: 2022-09-12T19:48:11.821 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.822 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:11.822 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.823 DEBUG:boto:Params: {} 2022-09-12T19:48:11.824 DEBUG:boto:Token: None 2022-09-12T19:48:11.825 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:48:11.825 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uafWNPvjs+hSOhlq5HKXogmN+/s= 2022-09-12T19:48:11.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uafWNPvjs+hSOhlq5HKXogmN+/s=', 'Content-Length': '0'} 2022-09-12T19:48:11.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6860cccbb4f109c-00631f8cfb-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.830 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:11.830 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:11.831 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:11.831 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:11.832 DEBUG:boto:Method: HEAD 2022-09-12T19:48:11.832 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:48:11.833 DEBUG:boto:Data: 2022-09-12T19:48:11.833 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.834 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.834 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.835 DEBUG:boto:Params: {} 2022-09-12T19:48:11.835 DEBUG:boto:Token: None 2022-09-12T19:48:11.836 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:48:11.837 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KdxmmRd3qzE7zVX+V90NxF3mvaU= 2022-09-12T19:48:11.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KdxmmRd3qzE7zVX+V90NxF3mvaU=', 'Content-Length': '0'} 2022-09-12T19:48:11.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dce550173262038-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.841 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:11.842 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:11.842 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:11.843 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:11.843 DEBUG:boto:Method: HEAD 2022-09-12T19:48:11.844 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:48:11.844 DEBUG:boto:Data: 2022-09-12T19:48:11.845 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.846 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:11.847 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.847 DEBUG:boto:Params: {} 2022-09-12T19:48:11.848 DEBUG:boto:Token: None 2022-09-12T19:48:11.849 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:48:11.849 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KdxmmRd3qzE7zVX+V90NxF3mvaU= 2022-09-12T19:48:11.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KdxmmRd3qzE7zVX+V90NxF3mvaU=', 'Content-Length': '0'} 2022-09-12T19:48:11.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bf20649ed2e0db4-00631f8cfb-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:48:11.855 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:11.855 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:11.856 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:11.856 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:11.857 DEBUG:boto:Method: GET 2022-09-12T19:48:11.857 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:48:11.858 DEBUG:boto:Data: 2022-09-12T19:48:11.858 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.859 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.859 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.860 DEBUG:boto:Params: {} 2022-09-12T19:48:11.861 DEBUG:boto:Token: None 2022-09-12T19:48:11.861 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:48:11.862 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uafWNPvjs+hSOhlq5HKXogmN+/s= 2022-09-12T19:48:11.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uafWNPvjs+hSOhlq5HKXogmN+/s=', 'Content-Length': '0'} 2022-09-12T19:48:11.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002aae629392dc73c-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.866 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:11.866 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:11.867 DEBUG:boto:path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:11.867 DEBUG:boto:auth_path=/jdpiak-72/c?versionId=null 2022-09-12T19:48:11.868 DEBUG:boto:Method: GET 2022-09-12T19:48:11.868 DEBUG:boto:Path: /jdpiak-72/c?versionId=null 2022-09-12T19:48:11.869 DEBUG:boto:Data: 2022-09-12T19:48:11.870 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.871 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:11.872 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.873 DEBUG:boto:Params: {} 2022-09-12T19:48:11.874 DEBUG:boto:Token: None 2022-09-12T19:48:11.875 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/c?versionId=null 2022-09-12T19:48:11.876 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uafWNPvjs+hSOhlq5HKXogmN+/s= 2022-09-12T19:48:11.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uafWNPvjs+hSOhlq5HKXogmN+/s=', 'Content-Length': '0'} 2022-09-12T19:48:11.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c092248b72f50731-00631f8cfb-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.881 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:11.882 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:11.883 DEBUG:boto:path=/jdpiak-72/?prefix=c 2022-09-12T19:48:11.884 DEBUG:boto:auth_path=/jdpiak-72/?prefix=c 2022-09-12T19:48:11.885 DEBUG:boto:Method: GET 2022-09-12T19:48:11.885 DEBUG:boto:Path: /jdpiak-72/?prefix=c 2022-09-12T19:48:11.886 DEBUG:boto:Data: 2022-09-12T19:48:11.886 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.887 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.887 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.888 DEBUG:boto:Params: {} 2022-09-12T19:48:11.889 DEBUG:boto:Token: None 2022-09-12T19:48:11.889 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/ 2022-09-12T19:48:11.890 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE= 2022-09-12T19:48:11.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE=', 'Content-Length': '0'} 2022-09-12T19:48:11.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e079e6e9446f98a-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.895 DEBUG:boto:b'jdpiak-72c1000falsec2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:11.896 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:11.897 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:11.898 DEBUG:boto:path=/jdpiak-72/?prefix=c 2022-09-12T19:48:11.899 DEBUG:boto:auth_path=/jdpiak-72/?prefix=c 2022-09-12T19:48:11.899 DEBUG:boto:Method: GET 2022-09-12T19:48:11.900 DEBUG:boto:Path: /jdpiak-72/?prefix=c 2022-09-12T19:48:11.900 DEBUG:boto:Data: 2022-09-12T19:48:11.901 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.901 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:11.902 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.902 DEBUG:boto:Params: {} 2022-09-12T19:48:11.903 DEBUG:boto:Token: None 2022-09-12T19:48:11.904 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/ 2022-09-12T19:48:11.904 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE= 2022-09-12T19:48:11.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZY4eJxkgy2BkpttHXp+1usIFHE=', 'Content-Length': '0'} 2022-09-12T19:48:11.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014f158d50e3903b3-00631f8cfb-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.908 DEBUG:boto:b'jdpiak-72c1000falsec2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:11.908 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:11.909 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:11.909 DEBUG:boto:Method: HEAD 2022-09-12T19:48:11.910 DEBUG:boto:Path: /jdpiak-72/c 2022-09-12T19:48:11.910 DEBUG:boto:Data: 2022-09-12T19:48:11.911 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.911 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.912 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.912 DEBUG:boto:Params: {} 2022-09-12T19:48:11.913 DEBUG:boto:Token: None 2022-09-12T19:48:11.914 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/c 2022-09-12T19:48:11.914 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ua7swn8CkiN5TPP/qoXsXtHHXjY= 2022-09-12T19:48:11.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ua7swn8CkiN5TPP/qoXsXtHHXjY=', 'Content-Length': '0'} 2022-09-12T19:48:11.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b40b2429bebcb802-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.918 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:11.918 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:11.919 DEBUG:boto:Method: HEAD 2022-09-12T19:48:11.919 DEBUG:boto:Path: /jdpiak-72/c 2022-09-12T19:48:11.920 DEBUG:boto:Data: 2022-09-12T19:48:11.920 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.921 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:11.921 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.922 DEBUG:boto:Params: {} 2022-09-12T19:48:11.922 DEBUG:boto:Token: None 2022-09-12T19:48:11.923 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/c 2022-09-12T19:48:11.923 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ua7swn8CkiN5TPP/qoXsXtHHXjY= 2022-09-12T19:48:11.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ua7swn8CkiN5TPP/qoXsXtHHXjY=', 'Content-Length': '0'} 2022-09-12T19:48:11.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aed674dac7cc98d5-00631f8cfb-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-12T19:48:11.927 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:11.928 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:11.928 DEBUG:boto:Method: GET 2022-09-12T19:48:11.929 DEBUG:boto:Path: /jdpiak-72/c 2022-09-12T19:48:11.929 DEBUG:boto:Data: 2022-09-12T19:48:11.930 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.931 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:11.931 DEBUG:boto:Port: 8001 2022-09-12T19:48:11.932 DEBUG:boto:Params: {} 2022-09-12T19:48:11.932 DEBUG:boto:Token: None 2022-09-12T19:48:11.933 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/c 2022-09-12T19:48:11.933 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zn5Le1noHp6LXhUQnSGKCznaI+E= 2022-09-12T19:48:11.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zn5Le1noHp6LXhUQnSGKCznaI+E=', 'Content-Length': '0'} 2022-09-12T19:48:11.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000032798022c3b3b76-00631f8cfb-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.937 DEBUG:boto:path=/jdpiak-72/c 2022-09-12T19:48:11.938 DEBUG:boto:auth_path=/jdpiak-72/c 2022-09-12T19:48:11.938 DEBUG:boto:Method: GET 2022-09-12T19:48:11.939 DEBUG:boto:Path: /jdpiak-72/c 2022-09-12T19:48:11.939 DEBUG:boto:Data: 2022-09-12T19:48:11.940 DEBUG:boto:Headers: {} 2022-09-12T19:48:11.940 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:11.941 DEBUG:boto:Port: 8000 2022-09-12T19:48:11.941 DEBUG:boto:Params: {} 2022-09-12T19:48:11.942 DEBUG:boto:Token: None 2022-09-12T19:48:11.942 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:11 GMT /jdpiak-72/c 2022-09-12T19:48:11.943 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zn5Le1noHp6LXhUQnSGKCznaI+E= 2022-09-12T19:48:11.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zn5Le1noHp6LXhUQnSGKCznaI+E=', 'Content-Length': '0'} 2022-09-12T19:48:11.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c030feb4208311a8-00631f8cfb-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:11.994 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:11.995 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:11.996 DEBUG:boto:path=/jdpiak-72/?prefix=c 2022-09-12T19:48:11.997 DEBUG:boto:auth_path=/jdpiak-72/?prefix=c 2022-09-12T19:48:11.998 DEBUG:boto:Method: GET 2022-09-12T19:48:11.999 DEBUG:boto:Path: /jdpiak-72/?prefix=c 2022-09-12T19:48:12.000 DEBUG:boto:Data: 2022-09-12T19:48:12.001 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.002 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.003 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.004 DEBUG:boto:Params: {} 2022-09-12T19:48:12.005 DEBUG:boto:Token: None 2022-09-12T19:48:12.006 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/ 2022-09-12T19:48:12.007 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s= 2022-09-12T19:48:12.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s=', 'Content-Length': '0'} 2022-09-12T19:48:12.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9bca08376ad2057-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.016 DEBUG:boto:b'jdpiak-72c1000falsec2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:12.017 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:12.018 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:12.019 DEBUG:boto:path=/jdpiak-72/?prefix=c 2022-09-12T19:48:12.020 DEBUG:boto:auth_path=/jdpiak-72/?prefix=c 2022-09-12T19:48:12.020 DEBUG:boto:Method: GET 2022-09-12T19:48:12.021 DEBUG:boto:Path: /jdpiak-72/?prefix=c 2022-09-12T19:48:12.022 DEBUG:boto:Data: 2022-09-12T19:48:12.022 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.023 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.023 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.024 DEBUG:boto:Params: {} 2022-09-12T19:48:12.025 DEBUG:boto:Token: None 2022-09-12T19:48:12.026 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/ 2022-09-12T19:48:12.026 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s= 2022-09-12T19:48:12.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s=', 'Content-Length': '0'} 2022-09-12T19:48:12.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c401135fb53cd1a4-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.031 DEBUG:boto:b'jdpiak-72c1000falsec2022-09-12T19:44:35.301Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:12.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:48:12.034 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:12.035 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:12.036 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:12.037 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:12.038 DEBUG:boto:Method: GET 2022-09-12T19:48:12.038 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:48:12.039 DEBUG:boto:Data: 2022-09-12T19:48:12.041 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.042 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.043 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.044 DEBUG:boto:Params: {} 2022-09-12T19:48:12.045 DEBUG:boto:Token: None 2022-09-12T19:48:12.046 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:48:12.047 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h8NTkvPmCh3/X6b8eJPeIFv2cAk= 2022-09-12T19:48:12.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h8NTkvPmCh3/X6b8eJPeIFv2cAk=', 'Content-Length': '0'} 2022-09-12T19:48:12.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000050a669dc7858f7-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.094 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:12.095 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:12.096 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:12.097 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:12.098 DEBUG:boto:Method: GET 2022-09-12T19:48:12.101 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:48:12.102 DEBUG:boto:Data: 2022-09-12T19:48:12.105 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.106 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.106 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.107 DEBUG:boto:Params: {} 2022-09-12T19:48:12.108 DEBUG:boto:Token: None 2022-09-12T19:48:12.109 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:48:12.110 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h8NTkvPmCh3/X6b8eJPeIFv2cAk= 2022-09-12T19:48:12.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h8NTkvPmCh3/X6b8eJPeIFv2cAk=', 'Content-Length': '0'} 2022-09-12T19:48:12.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000514798f521f8efc9-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.115 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:12.115 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:12.116 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:12.117 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:12.117 DEBUG:boto:Method: HEAD 2022-09-12T19:48:12.118 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:48:12.118 DEBUG:boto:Data: 2022-09-12T19:48:12.119 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.119 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.120 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.121 DEBUG:boto:Params: {} 2022-09-12T19:48:12.122 DEBUG:boto:Token: None 2022-09-12T19:48:12.122 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:48:12.123 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EBYsneshgSuItmgSmfNLe0/86AA= 2022-09-12T19:48:12.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EBYsneshgSuItmgSmfNLe0/86AA=', 'Content-Length': '0'} 2022-09-12T19:48:12.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c788431e67bf5f79-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.127 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:12.128 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:12.129 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:12.130 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:12.131 DEBUG:boto:Method: HEAD 2022-09-12T19:48:12.132 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:48:12.133 DEBUG:boto:Data: 2022-09-12T19:48:12.133 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.134 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.135 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.136 DEBUG:boto:Params: {} 2022-09-12T19:48:12.137 DEBUG:boto:Token: None 2022-09-12T19:48:12.138 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:48:12.139 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EBYsneshgSuItmgSmfNLe0/86AA= 2022-09-12T19:48:12.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EBYsneshgSuItmgSmfNLe0/86AA=', 'Content-Length': '0'} 2022-09-12T19:48:12.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000beb4cfb5a690fd6a-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:48:12.145 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:12.146 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:12.147 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:12.148 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:12.149 DEBUG:boto:Method: GET 2022-09-12T19:48:12.149 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:48:12.150 DEBUG:boto:Data: 2022-09-12T19:48:12.151 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.152 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.153 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.154 DEBUG:boto:Params: {} 2022-09-12T19:48:12.155 DEBUG:boto:Token: None 2022-09-12T19:48:12.156 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:48:12.157 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h8NTkvPmCh3/X6b8eJPeIFv2cAk= 2022-09-12T19:48:12.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h8NTkvPmCh3/X6b8eJPeIFv2cAk=', 'Content-Length': '0'} 2022-09-12T19:48:12.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007168a4d5549aa2c9-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.162 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:12.164 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:12.164 DEBUG:boto:path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:12.165 DEBUG:boto:auth_path=/jdpiak-72/d?versionId=null 2022-09-12T19:48:12.165 DEBUG:boto:Method: GET 2022-09-12T19:48:12.166 DEBUG:boto:Path: /jdpiak-72/d?versionId=null 2022-09-12T19:48:12.168 DEBUG:boto:Data: 2022-09-12T19:48:12.168 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.169 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.170 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.171 DEBUG:boto:Params: {} 2022-09-12T19:48:12.172 DEBUG:boto:Token: None 2022-09-12T19:48:12.173 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/d?versionId=null 2022-09-12T19:48:12.174 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h8NTkvPmCh3/X6b8eJPeIFv2cAk= 2022-09-12T19:48:12.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h8NTkvPmCh3/X6b8eJPeIFv2cAk=', 'Content-Length': '0'} 2022-09-12T19:48:12.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dabe706aae96b0cb-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.184 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:12.185 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:12.185 DEBUG:boto:path=/jdpiak-72/?prefix=d 2022-09-12T19:48:12.186 DEBUG:boto:auth_path=/jdpiak-72/?prefix=d 2022-09-12T19:48:12.187 DEBUG:boto:Method: GET 2022-09-12T19:48:12.188 DEBUG:boto:Path: /jdpiak-72/?prefix=d 2022-09-12T19:48:12.189 DEBUG:boto:Data: 2022-09-12T19:48:12.190 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.191 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.193 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.193 DEBUG:boto:Params: {} 2022-09-12T19:48:12.194 DEBUG:boto:Token: None 2022-09-12T19:48:12.195 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/ 2022-09-12T19:48:12.195 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s= 2022-09-12T19:48:12.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s=', 'Content-Length': '0'} 2022-09-12T19:48:12.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025d070d097fbf7da-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.200 DEBUG:boto:b'jdpiak-72d1000falsed2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:12.202 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:12.203 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:12.204 DEBUG:boto:path=/jdpiak-72/?prefix=d 2022-09-12T19:48:12.205 DEBUG:boto:auth_path=/jdpiak-72/?prefix=d 2022-09-12T19:48:12.206 DEBUG:boto:Method: GET 2022-09-12T19:48:12.206 DEBUG:boto:Path: /jdpiak-72/?prefix=d 2022-09-12T19:48:12.207 DEBUG:boto:Data: 2022-09-12T19:48:12.207 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.208 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.209 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.209 DEBUG:boto:Params: {} 2022-09-12T19:48:12.210 DEBUG:boto:Token: None 2022-09-12T19:48:12.211 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/ 2022-09-12T19:48:12.212 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s= 2022-09-12T19:48:12.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s=', 'Content-Length': '0'} 2022-09-12T19:48:12.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9f8938ff5602392-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.258 DEBUG:boto:b'jdpiak-72d1000falsed2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:12.259 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:12.260 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:12.260 DEBUG:boto:Method: HEAD 2022-09-12T19:48:12.261 DEBUG:boto:Path: /jdpiak-72/d 2022-09-12T19:48:12.262 DEBUG:boto:Data: 2022-09-12T19:48:12.263 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.264 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.264 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.265 DEBUG:boto:Params: {} 2022-09-12T19:48:12.266 DEBUG:boto:Token: None 2022-09-12T19:48:12.266 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/d 2022-09-12T19:48:12.267 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eINKiOXwTVtYx0yj4CR+EEK/u0k= 2022-09-12T19:48:12.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eINKiOXwTVtYx0yj4CR+EEK/u0k=', 'Content-Length': '0'} 2022-09-12T19:48:12.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000573d7d2caed59617-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.271 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:12.271 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:12.272 DEBUG:boto:Method: HEAD 2022-09-12T19:48:12.272 DEBUG:boto:Path: /jdpiak-72/d 2022-09-12T19:48:12.273 DEBUG:boto:Data: 2022-09-12T19:48:12.273 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.274 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.275 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.275 DEBUG:boto:Params: {} 2022-09-12T19:48:12.276 DEBUG:boto:Token: None 2022-09-12T19:48:12.277 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/d 2022-09-12T19:48:12.277 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eINKiOXwTVtYx0yj4CR+EEK/u0k= 2022-09-12T19:48:12.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eINKiOXwTVtYx0yj4CR+EEK/u0k=', 'Content-Length': '0'} 2022-09-12T19:48:12.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000648fdda8b004fb99-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-12T19:48:12.283 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:12.284 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:12.285 DEBUG:boto:Method: GET 2022-09-12T19:48:12.286 DEBUG:boto:Path: /jdpiak-72/d 2022-09-12T19:48:12.287 DEBUG:boto:Data: 2022-09-12T19:48:12.288 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.290 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.291 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.292 DEBUG:boto:Params: {} 2022-09-12T19:48:12.293 DEBUG:boto:Token: None 2022-09-12T19:48:12.294 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/d 2022-09-12T19:48:12.296 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7MDbVkeJzsu2CVBmqTcmlx0h4WU= 2022-09-12T19:48:12.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7MDbVkeJzsu2CVBmqTcmlx0h4WU=', 'Content-Length': '0'} 2022-09-12T19:48:12.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1463c7fef466426-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.303 DEBUG:boto:path=/jdpiak-72/d 2022-09-12T19:48:12.305 DEBUG:boto:auth_path=/jdpiak-72/d 2022-09-12T19:48:12.307 DEBUG:boto:Method: GET 2022-09-12T19:48:12.309 DEBUG:boto:Path: /jdpiak-72/d 2022-09-12T19:48:12.310 DEBUG:boto:Data: 2022-09-12T19:48:12.313 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.315 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.316 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.318 DEBUG:boto:Params: {} 2022-09-12T19:48:12.319 DEBUG:boto:Token: None 2022-09-12T19:48:12.320 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/d 2022-09-12T19:48:12.321 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7MDbVkeJzsu2CVBmqTcmlx0h4WU= 2022-09-12T19:48:12.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7MDbVkeJzsu2CVBmqTcmlx0h4WU=', 'Content-Length': '0'} 2022-09-12T19:48:12.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:44:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b42e29aa34ad10cb-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.326 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:12.327 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:12.328 DEBUG:boto:path=/jdpiak-72/?prefix=d 2022-09-12T19:48:12.329 DEBUG:boto:auth_path=/jdpiak-72/?prefix=d 2022-09-12T19:48:12.330 DEBUG:boto:Method: GET 2022-09-12T19:48:12.331 DEBUG:boto:Path: /jdpiak-72/?prefix=d 2022-09-12T19:48:12.332 DEBUG:boto:Data: 2022-09-12T19:48:12.332 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.333 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.334 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.334 DEBUG:boto:Params: {} 2022-09-12T19:48:12.335 DEBUG:boto:Token: None 2022-09-12T19:48:12.336 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/ 2022-09-12T19:48:12.337 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s= 2022-09-12T19:48:12.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s=', 'Content-Length': '0'} 2022-09-12T19:48:12.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b022d712be2364b4-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.341 DEBUG:boto:b'jdpiak-72d1000falsed2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:12.343 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:12.344 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:12.345 DEBUG:boto:path=/jdpiak-72/?prefix=d 2022-09-12T19:48:12.346 DEBUG:boto:auth_path=/jdpiak-72/?prefix=d 2022-09-12T19:48:12.347 DEBUG:boto:Method: GET 2022-09-12T19:48:12.348 DEBUG:boto:Path: /jdpiak-72/?prefix=d 2022-09-12T19:48:12.348 DEBUG:boto:Data: 2022-09-12T19:48:12.349 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.350 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.351 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.351 DEBUG:boto:Params: {} 2022-09-12T19:48:12.352 DEBUG:boto:Token: None 2022-09-12T19:48:12.353 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/ 2022-09-12T19:48:12.354 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s= 2022-09-12T19:48:12.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s=', 'Content-Length': '0'} 2022-09-12T19:48:12.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d12b6ab02e284dea-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.360 DEBUG:boto:b'jdpiak-72d1000falsed2022-09-12T19:44:35.334Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:12.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:48:12.364 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:12.365 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:12.366 DEBUG:boto:path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:12.367 DEBUG:boto:auth_path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:12.369 DEBUG:boto:Method: GET 2022-09-12T19:48:12.370 DEBUG:boto:Path: /jdpiak-72/e?versionId=null 2022-09-12T19:48:12.371 DEBUG:boto:Data: 2022-09-12T19:48:12.372 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.374 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.375 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.375 DEBUG:boto:Params: {} 2022-09-12T19:48:12.376 DEBUG:boto:Token: None 2022-09-12T19:48:12.377 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/e?versionId=null 2022-09-12T19:48:12.378 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+i6hcQyQn+dfyc+2s0x0+v7z5+g= 2022-09-12T19:48:12.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+i6hcQyQn+dfyc+2s0x0+v7z5+g=', 'Content-Length': '0'} 2022-09-12T19:48:12.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054d2fe0d7cffb601-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.383 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:12.384 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:12.385 DEBUG:boto:path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:12.387 DEBUG:boto:auth_path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:12.388 DEBUG:boto:Method: GET 2022-09-12T19:48:12.389 DEBUG:boto:Path: /jdpiak-72/e?versionId=null 2022-09-12T19:48:12.390 DEBUG:boto:Data: 2022-09-12T19:48:12.391 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.392 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.393 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.394 DEBUG:boto:Params: {} 2022-09-12T19:48:12.395 DEBUG:boto:Token: None 2022-09-12T19:48:12.396 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/e?versionId=null 2022-09-12T19:48:12.397 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+i6hcQyQn+dfyc+2s0x0+v7z5+g= 2022-09-12T19:48:12.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+i6hcQyQn+dfyc+2s0x0+v7z5+g=', 'Content-Length': '0'} 2022-09-12T19:48:12.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000717a733cf65f4346-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.402 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:12.403 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:12.403 DEBUG:boto:path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:12.404 DEBUG:boto:auth_path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:12.405 DEBUG:boto:Method: HEAD 2022-09-12T19:48:12.405 DEBUG:boto:Path: /jdpiak-72/e?versionId=null 2022-09-12T19:48:12.406 DEBUG:boto:Data: 2022-09-12T19:48:12.407 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.408 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.409 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.410 DEBUG:boto:Params: {} 2022-09-12T19:48:12.411 DEBUG:boto:Token: None 2022-09-12T19:48:12.412 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/e?versionId=null 2022-09-12T19:48:12.413 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2DMxembnPUB94jzInrYmxCX675A= 2022-09-12T19:48:12.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2DMxembnPUB94jzInrYmxCX675A=', 'Content-Length': '0'} 2022-09-12T19:48:12.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc2f3a309070b362-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.418 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:12.419 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:12.420 DEBUG:boto:path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:12.421 DEBUG:boto:auth_path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:12.422 DEBUG:boto:Method: HEAD 2022-09-12T19:48:12.422 DEBUG:boto:Path: /jdpiak-72/e?versionId=null 2022-09-12T19:48:12.424 DEBUG:boto:Data: 2022-09-12T19:48:12.424 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.425 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.425 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.426 DEBUG:boto:Params: {} 2022-09-12T19:48:12.426 DEBUG:boto:Token: None 2022-09-12T19:48:12.427 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/e?versionId=null 2022-09-12T19:48:12.427 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2DMxembnPUB94jzInrYmxCX675A= 2022-09-12T19:48:12.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2DMxembnPUB94jzInrYmxCX675A=', 'Content-Length': '0'} 2022-09-12T19:48:12.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023308944824662b1-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:48:12.432 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:12.433 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:12.434 DEBUG:boto:path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:12.435 DEBUG:boto:auth_path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:12.436 DEBUG:boto:Method: GET 2022-09-12T19:48:12.437 DEBUG:boto:Path: /jdpiak-72/e?versionId=null 2022-09-12T19:48:12.438 DEBUG:boto:Data: 2022-09-12T19:48:12.439 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.440 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.442 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.442 DEBUG:boto:Params: {} 2022-09-12T19:48:12.443 DEBUG:boto:Token: None 2022-09-12T19:48:12.456 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/e?versionId=null 2022-09-12T19:48:12.457 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+i6hcQyQn+dfyc+2s0x0+v7z5+g= 2022-09-12T19:48:12.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+i6hcQyQn+dfyc+2s0x0+v7z5+g=', 'Content-Length': '0'} 2022-09-12T19:48:12.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec25743b6d51d00f-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.461 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:12.462 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:12.463 DEBUG:boto:path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:12.464 DEBUG:boto:auth_path=/jdpiak-72/e?versionId=null 2022-09-12T19:48:12.465 DEBUG:boto:Method: GET 2022-09-12T19:48:12.465 DEBUG:boto:Path: /jdpiak-72/e?versionId=null 2022-09-12T19:48:12.466 DEBUG:boto:Data: 2022-09-12T19:48:12.466 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.467 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.468 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.468 DEBUG:boto:Params: {} 2022-09-12T19:48:12.469 DEBUG:boto:Token: None 2022-09-12T19:48:12.470 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/e?versionId=null 2022-09-12T19:48:12.471 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+i6hcQyQn+dfyc+2s0x0+v7z5+g= 2022-09-12T19:48:12.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+i6hcQyQn+dfyc+2s0x0+v7z5+g=', 'Content-Length': '0'} 2022-09-12T19:48:12.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000587dc00f6da2d458-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.518 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:12.519 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:12.519 DEBUG:boto:path=/jdpiak-72/?prefix=e 2022-09-12T19:48:12.521 DEBUG:boto:auth_path=/jdpiak-72/?prefix=e 2022-09-12T19:48:12.521 DEBUG:boto:Method: GET 2022-09-12T19:48:12.522 DEBUG:boto:Path: /jdpiak-72/?prefix=e 2022-09-12T19:48:12.522 DEBUG:boto:Data: 2022-09-12T19:48:12.523 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.524 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.524 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.525 DEBUG:boto:Params: {} 2022-09-12T19:48:12.526 DEBUG:boto:Token: None 2022-09-12T19:48:12.527 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/ 2022-09-12T19:48:12.527 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s= 2022-09-12T19:48:12.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s=', 'Content-Length': '0'} 2022-09-12T19:48:12.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037d0e83e893bda1d-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.532 DEBUG:boto:b'jdpiak-72e1000falsee2022-09-12T19:45:20.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:12.534 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:12.534 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:12.535 DEBUG:boto:path=/jdpiak-72/?prefix=e 2022-09-12T19:48:12.535 DEBUG:boto:auth_path=/jdpiak-72/?prefix=e 2022-09-12T19:48:12.536 DEBUG:boto:Method: GET 2022-09-12T19:48:12.536 DEBUG:boto:Path: /jdpiak-72/?prefix=e 2022-09-12T19:48:12.537 DEBUG:boto:Data: 2022-09-12T19:48:12.537 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.538 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.539 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.539 DEBUG:boto:Params: {} 2022-09-12T19:48:12.540 DEBUG:boto:Token: None 2022-09-12T19:48:12.541 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/ 2022-09-12T19:48:12.542 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s= 2022-09-12T19:48:12.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s=', 'Content-Length': '0'} 2022-09-12T19:48:12.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000792ab04dde90a513-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.546 DEBUG:boto:b'jdpiak-72e1000falsee2022-09-12T19:45:20.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:12.547 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:12.547 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:12.548 DEBUG:boto:Method: HEAD 2022-09-12T19:48:12.548 DEBUG:boto:Path: /jdpiak-72/e 2022-09-12T19:48:12.549 DEBUG:boto:Data: 2022-09-12T19:48:12.550 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.551 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.551 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.552 DEBUG:boto:Params: {} 2022-09-12T19:48:12.553 DEBUG:boto:Token: None 2022-09-12T19:48:12.554 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/e 2022-09-12T19:48:12.554 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:r2xGw8G8HJnoGI8ensWar/GZFoc= 2022-09-12T19:48:12.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:r2xGw8G8HJnoGI8ensWar/GZFoc=', 'Content-Length': '0'} 2022-09-12T19:48:12.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca8e60c268aedffb-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.559 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:12.560 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:12.561 DEBUG:boto:Method: HEAD 2022-09-12T19:48:12.562 DEBUG:boto:Path: /jdpiak-72/e 2022-09-12T19:48:12.562 DEBUG:boto:Data: 2022-09-12T19:48:12.564 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.565 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.566 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.567 DEBUG:boto:Params: {} 2022-09-12T19:48:12.568 DEBUG:boto:Token: None 2022-09-12T19:48:12.569 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/e 2022-09-12T19:48:12.569 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:r2xGw8G8HJnoGI8ensWar/GZFoc= 2022-09-12T19:48:12.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:r2xGw8G8HJnoGI8ensWar/GZFoc=', 'Content-Length': '0'} 2022-09-12T19:48:12.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e25a19578906d41b-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-12T19:48:12.573 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:12.574 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:12.575 DEBUG:boto:Method: GET 2022-09-12T19:48:12.576 DEBUG:boto:Path: /jdpiak-72/e 2022-09-12T19:48:12.576 DEBUG:boto:Data: 2022-09-12T19:48:12.577 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.578 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.578 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.579 DEBUG:boto:Params: {} 2022-09-12T19:48:12.580 DEBUG:boto:Token: None 2022-09-12T19:48:12.581 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/e 2022-09-12T19:48:12.581 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Iei8sCxNlMEF5fhByQoHU5yxmf8= 2022-09-12T19:48:12.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Iei8sCxNlMEF5fhByQoHU5yxmf8=', 'Content-Length': '0'} 2022-09-12T19:48:12.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4a174f3d5877100-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.586 DEBUG:boto:path=/jdpiak-72/e 2022-09-12T19:48:12.587 DEBUG:boto:auth_path=/jdpiak-72/e 2022-09-12T19:48:12.588 DEBUG:boto:Method: GET 2022-09-12T19:48:12.589 DEBUG:boto:Path: /jdpiak-72/e 2022-09-12T19:48:12.590 DEBUG:boto:Data: 2022-09-12T19:48:12.591 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.592 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.593 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.594 DEBUG:boto:Params: {} 2022-09-12T19:48:12.595 DEBUG:boto:Token: None 2022-09-12T19:48:12.596 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/e 2022-09-12T19:48:12.597 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Iei8sCxNlMEF5fhByQoHU5yxmf8= 2022-09-12T19:48:12.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Iei8sCxNlMEF5fhByQoHU5yxmf8=', 'Content-Length': '0'} 2022-09-12T19:48:12.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b10f72d0963aa493-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.602 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:12.603 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:12.604 DEBUG:boto:path=/jdpiak-72/?prefix=e 2022-09-12T19:48:12.605 DEBUG:boto:auth_path=/jdpiak-72/?prefix=e 2022-09-12T19:48:12.606 DEBUG:boto:Method: GET 2022-09-12T19:48:12.607 DEBUG:boto:Path: /jdpiak-72/?prefix=e 2022-09-12T19:48:12.608 DEBUG:boto:Data: 2022-09-12T19:48:12.609 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.610 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.611 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.612 DEBUG:boto:Params: {} 2022-09-12T19:48:12.613 DEBUG:boto:Token: None 2022-09-12T19:48:12.614 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/ 2022-09-12T19:48:12.615 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s= 2022-09-12T19:48:12.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s=', 'Content-Length': '0'} 2022-09-12T19:48:12.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006536ffc866cdca0-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.620 DEBUG:boto:b'jdpiak-72e1000falsee2022-09-12T19:45:20.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:12.621 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:12.621 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:12.622 DEBUG:boto:path=/jdpiak-72/?prefix=e 2022-09-12T19:48:12.622 DEBUG:boto:auth_path=/jdpiak-72/?prefix=e 2022-09-12T19:48:12.623 DEBUG:boto:Method: GET 2022-09-12T19:48:12.623 DEBUG:boto:Path: /jdpiak-72/?prefix=e 2022-09-12T19:48:12.624 DEBUG:boto:Data: 2022-09-12T19:48:12.625 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.625 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.627 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.628 DEBUG:boto:Params: {} 2022-09-12T19:48:12.629 DEBUG:boto:Token: None 2022-09-12T19:48:12.630 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/ 2022-09-12T19:48:12.630 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s= 2022-09-12T19:48:12.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s=', 'Content-Length': '0'} 2022-09-12T19:48:12.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002855c3350baf38fd-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.634 DEBUG:boto:b'jdpiak-72e1000falsee2022-09-12T19:45:20.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:12.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:48:12.636 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:12.636 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:12.637 DEBUG:boto:path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:12.637 DEBUG:boto:auth_path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:12.638 DEBUG:boto:Method: GET 2022-09-12T19:48:12.639 DEBUG:boto:Path: /jdpiak-72/f?versionId=null 2022-09-12T19:48:12.639 DEBUG:boto:Data: 2022-09-12T19:48:12.640 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.640 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.641 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.641 DEBUG:boto:Params: {} 2022-09-12T19:48:12.642 DEBUG:boto:Token: None 2022-09-12T19:48:12.642 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/f?versionId=null 2022-09-12T19:48:12.643 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ta204NALyIt/LjAdk8/FUEJNyTQ= 2022-09-12T19:48:12.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ta204NALyIt/LjAdk8/FUEJNyTQ=', 'Content-Length': '0'} 2022-09-12T19:48:12.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097cdb3282425ea76-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.690 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:12.691 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:12.692 DEBUG:boto:path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:12.693 DEBUG:boto:auth_path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:12.695 DEBUG:boto:Method: GET 2022-09-12T19:48:12.696 DEBUG:boto:Path: /jdpiak-72/f?versionId=null 2022-09-12T19:48:12.697 DEBUG:boto:Data: 2022-09-12T19:48:12.698 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.699 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.700 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.701 DEBUG:boto:Params: {} 2022-09-12T19:48:12.702 DEBUG:boto:Token: None 2022-09-12T19:48:12.703 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/f?versionId=null 2022-09-12T19:48:12.704 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ta204NALyIt/LjAdk8/FUEJNyTQ= 2022-09-12T19:48:12.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ta204NALyIt/LjAdk8/FUEJNyTQ=', 'Content-Length': '0'} 2022-09-12T19:48:12.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a46b879d0d7d4039-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.710 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:12.711 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:12.712 DEBUG:boto:path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:12.713 DEBUG:boto:auth_path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:12.714 DEBUG:boto:Method: HEAD 2022-09-12T19:48:12.715 DEBUG:boto:Path: /jdpiak-72/f?versionId=null 2022-09-12T19:48:12.716 DEBUG:boto:Data: 2022-09-12T19:48:12.717 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.717 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.718 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.719 DEBUG:boto:Params: {} 2022-09-12T19:48:12.719 DEBUG:boto:Token: None 2022-09-12T19:48:12.720 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/f?versionId=null 2022-09-12T19:48:12.721 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x0aQN4X+JL33caYYAVycnPIDmQ0= 2022-09-12T19:48:12.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x0aQN4X+JL33caYYAVycnPIDmQ0=', 'Content-Length': '0'} 2022-09-12T19:48:12.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054699bf1f1554722-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.724 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:12.725 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:12.725 DEBUG:boto:path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:12.726 DEBUG:boto:auth_path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:12.727 DEBUG:boto:Method: HEAD 2022-09-12T19:48:12.728 DEBUG:boto:Path: /jdpiak-72/f?versionId=null 2022-09-12T19:48:12.728 DEBUG:boto:Data: 2022-09-12T19:48:12.729 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.729 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.730 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.731 DEBUG:boto:Params: {} 2022-09-12T19:48:12.731 DEBUG:boto:Token: None 2022-09-12T19:48:12.732 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/f?versionId=null 2022-09-12T19:48:12.732 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x0aQN4X+JL33caYYAVycnPIDmQ0= 2022-09-12T19:48:12.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x0aQN4X+JL33caYYAVycnPIDmQ0=', 'Content-Length': '0'} 2022-09-12T19:48:12.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d3bde573616826f-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:48:12.736 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:12.737 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:12.737 DEBUG:boto:path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:12.738 DEBUG:boto:auth_path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:12.738 DEBUG:boto:Method: GET 2022-09-12T19:48:12.739 DEBUG:boto:Path: /jdpiak-72/f?versionId=null 2022-09-12T19:48:12.739 DEBUG:boto:Data: 2022-09-12T19:48:12.740 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.741 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.741 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.742 DEBUG:boto:Params: {} 2022-09-12T19:48:12.742 DEBUG:boto:Token: None 2022-09-12T19:48:12.743 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/f?versionId=null 2022-09-12T19:48:12.744 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ta204NALyIt/LjAdk8/FUEJNyTQ= 2022-09-12T19:48:12.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ta204NALyIt/LjAdk8/FUEJNyTQ=', 'Content-Length': '0'} 2022-09-12T19:48:12.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbb74f1b907bc4c7-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.748 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:12.749 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:12.749 DEBUG:boto:path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:12.750 DEBUG:boto:auth_path=/jdpiak-72/f?versionId=null 2022-09-12T19:48:12.750 DEBUG:boto:Method: GET 2022-09-12T19:48:12.751 DEBUG:boto:Path: /jdpiak-72/f?versionId=null 2022-09-12T19:48:12.751 DEBUG:boto:Data: 2022-09-12T19:48:12.752 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.752 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.753 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.754 DEBUG:boto:Params: {} 2022-09-12T19:48:12.755 DEBUG:boto:Token: None 2022-09-12T19:48:12.756 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/f?versionId=null 2022-09-12T19:48:12.757 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ta204NALyIt/LjAdk8/FUEJNyTQ= 2022-09-12T19:48:12.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ta204NALyIt/LjAdk8/FUEJNyTQ=', 'Content-Length': '0'} 2022-09-12T19:48:12.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb7f4d19b7441990-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.761 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:12.762 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:12.763 DEBUG:boto:path=/jdpiak-72/?prefix=f 2022-09-12T19:48:12.763 DEBUG:boto:auth_path=/jdpiak-72/?prefix=f 2022-09-12T19:48:12.764 DEBUG:boto:Method: GET 2022-09-12T19:48:12.765 DEBUG:boto:Path: /jdpiak-72/?prefix=f 2022-09-12T19:48:12.766 DEBUG:boto:Data: 2022-09-12T19:48:12.767 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.767 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.768 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.769 DEBUG:boto:Params: {} 2022-09-12T19:48:12.769 DEBUG:boto:Token: None 2022-09-12T19:48:12.771 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/ 2022-09-12T19:48:12.772 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s= 2022-09-12T19:48:12.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s=', 'Content-Length': '0'} 2022-09-12T19:48:12.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000189f197b8807eb1a-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.776 DEBUG:boto:b'jdpiak-72f1000falsef2022-09-12T19:45:20.891Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:12.777 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:12.777 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:12.778 DEBUG:boto:path=/jdpiak-72/?prefix=f 2022-09-12T19:48:12.778 DEBUG:boto:auth_path=/jdpiak-72/?prefix=f 2022-09-12T19:48:12.779 DEBUG:boto:Method: GET 2022-09-12T19:48:12.780 DEBUG:boto:Path: /jdpiak-72/?prefix=f 2022-09-12T19:48:12.780 DEBUG:boto:Data: 2022-09-12T19:48:12.781 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.781 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.782 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.782 DEBUG:boto:Params: {} 2022-09-12T19:48:12.783 DEBUG:boto:Token: None 2022-09-12T19:48:12.784 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/ 2022-09-12T19:48:12.784 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s= 2022-09-12T19:48:12.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s=', 'Content-Length': '0'} 2022-09-12T19:48:12.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddbc04254e7bd745-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.788 DEBUG:boto:b'jdpiak-72f1000falsef2022-09-12T19:45:20.891Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:12.789 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:12.789 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:12.790 DEBUG:boto:Method: HEAD 2022-09-12T19:48:12.790 DEBUG:boto:Path: /jdpiak-72/f 2022-09-12T19:48:12.791 DEBUG:boto:Data: 2022-09-12T19:48:12.791 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.792 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.792 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.793 DEBUG:boto:Params: {} 2022-09-12T19:48:12.793 DEBUG:boto:Token: None 2022-09-12T19:48:12.794 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/f 2022-09-12T19:48:12.794 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H6OVjGEGVwtnhpXL37ANDKeY59c= 2022-09-12T19:48:12.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H6OVjGEGVwtnhpXL37ANDKeY59c=', 'Content-Length': '0'} 2022-09-12T19:48:12.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8f7ecc775d7e6cc-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.798 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:12.798 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:12.799 DEBUG:boto:Method: HEAD 2022-09-12T19:48:12.799 DEBUG:boto:Path: /jdpiak-72/f 2022-09-12T19:48:12.800 DEBUG:boto:Data: 2022-09-12T19:48:12.800 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.801 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.801 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.802 DEBUG:boto:Params: {} 2022-09-12T19:48:12.802 DEBUG:boto:Token: None 2022-09-12T19:48:12.803 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/f 2022-09-12T19:48:12.803 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H6OVjGEGVwtnhpXL37ANDKeY59c= 2022-09-12T19:48:12.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H6OVjGEGVwtnhpXL37ANDKeY59c=', 'Content-Length': '0'} 2022-09-12T19:48:12.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b471ebdef4a411bd-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-12T19:48:12.808 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:12.809 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:12.810 DEBUG:boto:Method: GET 2022-09-12T19:48:12.810 DEBUG:boto:Path: /jdpiak-72/f 2022-09-12T19:48:12.811 DEBUG:boto:Data: 2022-09-12T19:48:12.811 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.812 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.813 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.813 DEBUG:boto:Params: {} 2022-09-12T19:48:12.814 DEBUG:boto:Token: None 2022-09-12T19:48:12.814 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/f 2022-09-12T19:48:12.815 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GI7ejn++31c6V3hhfyxE1/uUcqw= 2022-09-12T19:48:12.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GI7ejn++31c6V3hhfyxE1/uUcqw=', 'Content-Length': '0'} 2022-09-12T19:48:12.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006eb6b8e701716194-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.862 DEBUG:boto:path=/jdpiak-72/f 2022-09-12T19:48:12.863 DEBUG:boto:auth_path=/jdpiak-72/f 2022-09-12T19:48:12.864 DEBUG:boto:Method: GET 2022-09-12T19:48:12.864 DEBUG:boto:Path: /jdpiak-72/f 2022-09-12T19:48:12.865 DEBUG:boto:Data: 2022-09-12T19:48:12.865 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.866 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.867 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.867 DEBUG:boto:Params: {} 2022-09-12T19:48:12.868 DEBUG:boto:Token: None 2022-09-12T19:48:12.869 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/f 2022-09-12T19:48:12.869 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GI7ejn++31c6V3hhfyxE1/uUcqw= 2022-09-12T19:48:12.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GI7ejn++31c6V3hhfyxE1/uUcqw=', 'Content-Length': '0'} 2022-09-12T19:48:12.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013177b9589078ef2-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.873 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:12.874 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:12.874 DEBUG:boto:path=/jdpiak-72/?prefix=f 2022-09-12T19:48:12.875 DEBUG:boto:auth_path=/jdpiak-72/?prefix=f 2022-09-12T19:48:12.877 DEBUG:boto:Method: GET 2022-09-12T19:48:12.878 DEBUG:boto:Path: /jdpiak-72/?prefix=f 2022-09-12T19:48:12.878 DEBUG:boto:Data: 2022-09-12T19:48:12.879 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.880 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.881 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.882 DEBUG:boto:Params: {} 2022-09-12T19:48:12.883 DEBUG:boto:Token: None 2022-09-12T19:48:12.884 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/ 2022-09-12T19:48:12.885 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s= 2022-09-12T19:48:12.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s=', 'Content-Length': '0'} 2022-09-12T19:48:12.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c25a207f4cb1b98d-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.890 DEBUG:boto:b'jdpiak-72f1000falsef2022-09-12T19:45:20.891Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:12.892 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:12.893 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:12.894 DEBUG:boto:path=/jdpiak-72/?prefix=f 2022-09-12T19:48:12.894 DEBUG:boto:auth_path=/jdpiak-72/?prefix=f 2022-09-12T19:48:12.895 DEBUG:boto:Method: GET 2022-09-12T19:48:12.896 DEBUG:boto:Path: /jdpiak-72/?prefix=f 2022-09-12T19:48:12.897 DEBUG:boto:Data: 2022-09-12T19:48:12.898 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.898 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.899 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.899 DEBUG:boto:Params: {} 2022-09-12T19:48:12.900 DEBUG:boto:Token: None 2022-09-12T19:48:12.901 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/ 2022-09-12T19:48:12.901 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s= 2022-09-12T19:48:12.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s=', 'Content-Length': '0'} 2022-09-12T19:48:12.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf86986adcea3549-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.905 DEBUG:boto:b'jdpiak-72f1000falsef2022-09-12T19:45:20.891Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:12.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:48:12.906 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:12.907 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:12.908 DEBUG:boto:path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:12.908 DEBUG:boto:auth_path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:12.909 DEBUG:boto:Method: GET 2022-09-12T19:48:12.909 DEBUG:boto:Path: /jdpiak-72/g?versionId=null 2022-09-12T19:48:12.910 DEBUG:boto:Data: 2022-09-12T19:48:12.910 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.911 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.911 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.912 DEBUG:boto:Params: {} 2022-09-12T19:48:12.913 DEBUG:boto:Token: None 2022-09-12T19:48:12.913 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/g?versionId=null 2022-09-12T19:48:12.914 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YAXQr9++d2zHtYu7PLhAFGivkwU= 2022-09-12T19:48:12.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YAXQr9++d2zHtYu7PLhAFGivkwU=', 'Content-Length': '0'} 2022-09-12T19:48:12.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009731f25fc3799661-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.917 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:12.918 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:12.919 DEBUG:boto:path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:12.920 DEBUG:boto:auth_path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:12.921 DEBUG:boto:Method: GET 2022-09-12T19:48:12.922 DEBUG:boto:Path: /jdpiak-72/g?versionId=null 2022-09-12T19:48:12.922 DEBUG:boto:Data: 2022-09-12T19:48:12.923 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.923 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.924 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.924 DEBUG:boto:Params: {} 2022-09-12T19:48:12.925 DEBUG:boto:Token: None 2022-09-12T19:48:12.925 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/g?versionId=null 2022-09-12T19:48:12.926 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YAXQr9++d2zHtYu7PLhAFGivkwU= 2022-09-12T19:48:12.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YAXQr9++d2zHtYu7PLhAFGivkwU=', 'Content-Length': '0'} 2022-09-12T19:48:12.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000832cfda1f0cd4746-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.930 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:12.930 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:12.931 DEBUG:boto:path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:12.931 DEBUG:boto:auth_path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:12.932 DEBUG:boto:Method: HEAD 2022-09-12T19:48:12.932 DEBUG:boto:Path: /jdpiak-72/g?versionId=null 2022-09-12T19:48:12.933 DEBUG:boto:Data: 2022-09-12T19:48:12.933 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.934 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.934 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.935 DEBUG:boto:Params: {} 2022-09-12T19:48:12.935 DEBUG:boto:Token: None 2022-09-12T19:48:12.936 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/g?versionId=null 2022-09-12T19:48:12.937 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2M/5nhEEP3Whcns/tKoSoRhi5oM= 2022-09-12T19:48:12.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2M/5nhEEP3Whcns/tKoSoRhi5oM=', 'Content-Length': '0'} 2022-09-12T19:48:12.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084b31cc683823889-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.940 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:12.941 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:12.941 DEBUG:boto:path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:12.942 DEBUG:boto:auth_path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:12.942 DEBUG:boto:Method: HEAD 2022-09-12T19:48:12.943 DEBUG:boto:Path: /jdpiak-72/g?versionId=null 2022-09-12T19:48:12.943 DEBUG:boto:Data: 2022-09-12T19:48:12.944 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.944 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.944 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.945 DEBUG:boto:Params: {} 2022-09-12T19:48:12.946 DEBUG:boto:Token: None 2022-09-12T19:48:12.946 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/g?versionId=null 2022-09-12T19:48:12.947 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2M/5nhEEP3Whcns/tKoSoRhi5oM= 2022-09-12T19:48:12.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2M/5nhEEP3Whcns/tKoSoRhi5oM=', 'Content-Length': '0'} 2022-09-12T19:48:12.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be8ba1ed883c6970-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:48:12.951 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:12.951 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:12.952 DEBUG:boto:path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:12.952 DEBUG:boto:auth_path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:12.953 DEBUG:boto:Method: GET 2022-09-12T19:48:12.954 DEBUG:boto:Path: /jdpiak-72/g?versionId=null 2022-09-12T19:48:12.954 DEBUG:boto:Data: 2022-09-12T19:48:12.955 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.955 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.956 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.956 DEBUG:boto:Params: {} 2022-09-12T19:48:12.957 DEBUG:boto:Token: None 2022-09-12T19:48:12.957 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/g?versionId=null 2022-09-12T19:48:12.958 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YAXQr9++d2zHtYu7PLhAFGivkwU= 2022-09-12T19:48:12.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YAXQr9++d2zHtYu7PLhAFGivkwU=', 'Content-Length': '0'} 2022-09-12T19:48:12.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000146221099b45bfcd-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.962 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:12.962 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:12.963 DEBUG:boto:path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:12.963 DEBUG:boto:auth_path=/jdpiak-72/g?versionId=null 2022-09-12T19:48:12.964 DEBUG:boto:Method: GET 2022-09-12T19:48:12.964 DEBUG:boto:Path: /jdpiak-72/g?versionId=null 2022-09-12T19:48:12.965 DEBUG:boto:Data: 2022-09-12T19:48:12.965 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.966 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.966 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.967 DEBUG:boto:Params: {} 2022-09-12T19:48:12.968 DEBUG:boto:Token: None 2022-09-12T19:48:12.968 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/g?versionId=null 2022-09-12T19:48:12.969 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YAXQr9++d2zHtYu7PLhAFGivkwU= 2022-09-12T19:48:12.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YAXQr9++d2zHtYu7PLhAFGivkwU=', 'Content-Length': '0'} 2022-09-12T19:48:12.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cb07e0cbb868cd7-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.972 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:12.973 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:12.973 DEBUG:boto:path=/jdpiak-72/?prefix=g 2022-09-12T19:48:12.974 DEBUG:boto:auth_path=/jdpiak-72/?prefix=g 2022-09-12T19:48:12.974 DEBUG:boto:Method: GET 2022-09-12T19:48:12.975 DEBUG:boto:Path: /jdpiak-72/?prefix=g 2022-09-12T19:48:12.976 DEBUG:boto:Data: 2022-09-12T19:48:12.976 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.977 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:12.977 DEBUG:boto:Port: 8001 2022-09-12T19:48:12.978 DEBUG:boto:Params: {} 2022-09-12T19:48:12.979 DEBUG:boto:Token: None 2022-09-12T19:48:12.979 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/ 2022-09-12T19:48:12.980 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s= 2022-09-12T19:48:12.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s=', 'Content-Length': '0'} 2022-09-12T19:48:12.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000367c90227dc40ca8-00631f8cfc-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.985 DEBUG:boto:b'jdpiak-72g1000falseg2022-09-12T19:45:20.920Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:12.986 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:12.987 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:12.988 DEBUG:boto:path=/jdpiak-72/?prefix=g 2022-09-12T19:48:12.988 DEBUG:boto:auth_path=/jdpiak-72/?prefix=g 2022-09-12T19:48:12.989 DEBUG:boto:Method: GET 2022-09-12T19:48:12.989 DEBUG:boto:Path: /jdpiak-72/?prefix=g 2022-09-12T19:48:12.990 DEBUG:boto:Data: 2022-09-12T19:48:12.990 DEBUG:boto:Headers: {} 2022-09-12T19:48:12.991 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:12.992 DEBUG:boto:Port: 8000 2022-09-12T19:48:12.992 DEBUG:boto:Params: {} 2022-09-12T19:48:12.993 DEBUG:boto:Token: None 2022-09-12T19:48:12.994 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:12 GMT /jdpiak-72/ 2022-09-12T19:48:12.994 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s= 2022-09-12T19:48:12.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AcYtfdWEpoD0AQG/kBfCl065/8s=', 'Content-Length': '0'} 2022-09-12T19:48:12.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e87d9c65fb5b2818-00631f8cfc-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:12.998 DEBUG:boto:b'jdpiak-72g1000falseg2022-09-12T19:45:20.920Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:12.999 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:12.999 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:13.000 DEBUG:boto:Method: HEAD 2022-09-12T19:48:13.000 DEBUG:boto:Path: /jdpiak-72/g 2022-09-12T19:48:13.001 DEBUG:boto:Data: 2022-09-12T19:48:13.002 DEBUG:boto:Headers: {} 2022-09-12T19:48:13.002 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:13.003 DEBUG:boto:Port: 8001 2022-09-12T19:48:13.003 DEBUG:boto:Params: {} 2022-09-12T19:48:13.004 DEBUG:boto:Token: None 2022-09-12T19:48:13.004 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:13 GMT /jdpiak-72/g 2022-09-12T19:48:13.005 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ea1pUuASb7elZ/qHdO1zY0ZriX0= 2022-09-12T19:48:13.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ea1pUuASb7elZ/qHdO1zY0ZriX0=', 'Content-Length': '0'} 2022-09-12T19:48:13.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1525bbee6c50be1-00631f8cfd-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:13.008 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:13.009 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:13.009 DEBUG:boto:Method: HEAD 2022-09-12T19:48:13.010 DEBUG:boto:Path: /jdpiak-72/g 2022-09-12T19:48:13.010 DEBUG:boto:Data: 2022-09-12T19:48:13.011 DEBUG:boto:Headers: {} 2022-09-12T19:48:13.011 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:13.012 DEBUG:boto:Port: 8000 2022-09-12T19:48:13.012 DEBUG:boto:Params: {} 2022-09-12T19:48:13.013 DEBUG:boto:Token: None 2022-09-12T19:48:13.014 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:13 GMT /jdpiak-72/g 2022-09-12T19:48:13.014 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ea1pUuASb7elZ/qHdO1zY0ZriX0= 2022-09-12T19:48:13.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ea1pUuASb7elZ/qHdO1zY0ZriX0=', 'Content-Length': '0'} 2022-09-12T19:48:13.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b822240ab252d63e-00631f8cfd-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:13.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-12T19:48:13.018 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:13.019 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:13.019 DEBUG:boto:Method: GET 2022-09-12T19:48:13.020 DEBUG:boto:Path: /jdpiak-72/g 2022-09-12T19:48:13.020 DEBUG:boto:Data: 2022-09-12T19:48:13.021 DEBUG:boto:Headers: {} 2022-09-12T19:48:13.021 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:13.022 DEBUG:boto:Port: 8001 2022-09-12T19:48:13.022 DEBUG:boto:Params: {} 2022-09-12T19:48:13.023 DEBUG:boto:Token: None 2022-09-12T19:48:13.023 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:13 GMT /jdpiak-72/g 2022-09-12T19:48:13.024 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Oa8AUlt64ryV4FoXtowNBwn4cD0= 2022-09-12T19:48:13.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Oa8AUlt64ryV4FoXtowNBwn4cD0=', 'Content-Length': '0'} 2022-09-12T19:48:13.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfdb81ed08f38d43-00631f8cfd-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:13.027 DEBUG:boto:path=/jdpiak-72/g 2022-09-12T19:48:13.028 DEBUG:boto:auth_path=/jdpiak-72/g 2022-09-12T19:48:13.028 DEBUG:boto:Method: GET 2022-09-12T19:48:13.029 DEBUG:boto:Path: /jdpiak-72/g 2022-09-12T19:48:13.029 DEBUG:boto:Data: 2022-09-12T19:48:13.030 DEBUG:boto:Headers: {} 2022-09-12T19:48:13.031 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:13.031 DEBUG:boto:Port: 8000 2022-09-12T19:48:13.032 DEBUG:boto:Params: {} 2022-09-12T19:48:13.033 DEBUG:boto:Token: None 2022-09-12T19:48:13.033 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:13 GMT /jdpiak-72/g 2022-09-12T19:48:13.034 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Oa8AUlt64ryV4FoXtowNBwn4cD0= 2022-09-12T19:48:13.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Oa8AUlt64ryV4FoXtowNBwn4cD0=', 'Content-Length': '0'} 2022-09-12T19:48:13.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad7c25fd2d563145-00631f8cfd-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:13.038 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:13.039 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:13.039 DEBUG:boto:path=/jdpiak-72/?prefix=g 2022-09-12T19:48:13.040 DEBUG:boto:auth_path=/jdpiak-72/?prefix=g 2022-09-12T19:48:13.040 DEBUG:boto:Method: GET 2022-09-12T19:48:13.041 DEBUG:boto:Path: /jdpiak-72/?prefix=g 2022-09-12T19:48:13.041 DEBUG:boto:Data: 2022-09-12T19:48:13.042 DEBUG:boto:Headers: {} 2022-09-12T19:48:13.042 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:13.043 DEBUG:boto:Port: 8001 2022-09-12T19:48:13.043 DEBUG:boto:Params: {} 2022-09-12T19:48:13.044 DEBUG:boto:Token: None 2022-09-12T19:48:13.045 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:13 GMT /jdpiak-72/ 2022-09-12T19:48:13.045 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ntUE5lnNMPHHBQVmvNHDFSnp+nc= 2022-09-12T19:48:13.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ntUE5lnNMPHHBQVmvNHDFSnp+nc=', 'Content-Length': '0'} 2022-09-12T19:48:13.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000057f2ea62ba2b545-00631f8cfd-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:13.094 DEBUG:boto:b'jdpiak-72g1000falseg2022-09-12T19:45:20.920Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:13.095 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:13.096 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:13.096 DEBUG:boto:path=/jdpiak-72/?prefix=g 2022-09-12T19:48:13.097 DEBUG:boto:auth_path=/jdpiak-72/?prefix=g 2022-09-12T19:48:13.098 DEBUG:boto:Method: GET 2022-09-12T19:48:13.099 DEBUG:boto:Path: /jdpiak-72/?prefix=g 2022-09-12T19:48:13.100 DEBUG:boto:Data: 2022-09-12T19:48:13.101 DEBUG:boto:Headers: {} 2022-09-12T19:48:13.102 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:13.103 DEBUG:boto:Port: 8000 2022-09-12T19:48:13.104 DEBUG:boto:Params: {} 2022-09-12T19:48:13.105 DEBUG:boto:Token: None 2022-09-12T19:48:13.106 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:13 GMT /jdpiak-72/ 2022-09-12T19:48:13.107 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ntUE5lnNMPHHBQVmvNHDFSnp+nc= 2022-09-12T19:48:13.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ntUE5lnNMPHHBQVmvNHDFSnp+nc=', 'Content-Length': '0'} 2022-09-12T19:48:13.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000277451aeed6cd55a-00631f8cfd-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:13.112 DEBUG:boto:b'jdpiak-72g1000falseg2022-09-12T19:45:20.920Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:13.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:48:13.113 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:13.114 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:13.114 DEBUG:boto:path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:13.115 DEBUG:boto:auth_path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:13.115 DEBUG:boto:Method: GET 2022-09-12T19:48:13.116 DEBUG:boto:Path: /jdpiak-72/h?versionId=null 2022-09-12T19:48:13.116 DEBUG:boto:Data: 2022-09-12T19:48:13.117 DEBUG:boto:Headers: {} 2022-09-12T19:48:13.117 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:13.118 DEBUG:boto:Port: 8001 2022-09-12T19:48:13.119 DEBUG:boto:Params: {} 2022-09-12T19:48:13.120 DEBUG:boto:Token: None 2022-09-12T19:48:13.121 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:13 GMT /jdpiak-72/h?versionId=null 2022-09-12T19:48:13.122 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3SdZqV+5RvosGVBq44EYMl+PcB4= 2022-09-12T19:48:13.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3SdZqV+5RvosGVBq44EYMl+PcB4=', 'Content-Length': '0'} 2022-09-12T19:48:13.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002fa5713f6ecac7c-00631f8cfd-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:13.126 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:13.127 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:13.128 DEBUG:boto:path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:13.129 DEBUG:boto:auth_path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:13.129 DEBUG:boto:Method: GET 2022-09-12T19:48:13.130 DEBUG:boto:Path: /jdpiak-72/h?versionId=null 2022-09-12T19:48:13.130 DEBUG:boto:Data: 2022-09-12T19:48:13.131 DEBUG:boto:Headers: {} 2022-09-12T19:48:13.132 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:13.132 DEBUG:boto:Port: 8000 2022-09-12T19:48:13.133 DEBUG:boto:Params: {} 2022-09-12T19:48:13.134 DEBUG:boto:Token: None 2022-09-12T19:48:13.134 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:13 GMT /jdpiak-72/h?versionId=null 2022-09-12T19:48:13.135 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3SdZqV+5RvosGVBq44EYMl+PcB4= 2022-09-12T19:48:13.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3SdZqV+5RvosGVBq44EYMl+PcB4=', 'Content-Length': '0'} 2022-09-12T19:48:13.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001eda5d8ca8fe930e-00631f8cfd-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:13.186 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:13.187 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:13.188 DEBUG:boto:path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:13.189 DEBUG:boto:auth_path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:13.190 DEBUG:boto:Method: HEAD 2022-09-12T19:48:13.191 DEBUG:boto:Path: /jdpiak-72/h?versionId=null 2022-09-12T19:48:13.192 DEBUG:boto:Data: 2022-09-12T19:48:13.193 DEBUG:boto:Headers: {} 2022-09-12T19:48:13.194 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:13.195 DEBUG:boto:Port: 8001 2022-09-12T19:48:13.196 DEBUG:boto:Params: {} 2022-09-12T19:48:13.197 DEBUG:boto:Token: None 2022-09-12T19:48:13.198 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:13 GMT /jdpiak-72/h?versionId=null 2022-09-12T19:48:13.199 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X21nVE6XvPRFXjNdVIsSXuNl4Tg= 2022-09-12T19:48:13.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X21nVE6XvPRFXjNdVIsSXuNl4Tg=', 'Content-Length': '0'} 2022-09-12T19:48:13.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012d5d25c5c747935-00631f8cfd-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:13.203 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:13.203 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:13.204 DEBUG:boto:path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:13.204 DEBUG:boto:auth_path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:13.205 DEBUG:boto:Method: HEAD 2022-09-12T19:48:13.205 DEBUG:boto:Path: /jdpiak-72/h?versionId=null 2022-09-12T19:48:13.206 DEBUG:boto:Data: 2022-09-12T19:48:13.207 DEBUG:boto:Headers: {} 2022-09-12T19:48:13.207 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:13.208 DEBUG:boto:Port: 8000 2022-09-12T19:48:13.209 DEBUG:boto:Params: {} 2022-09-12T19:48:13.210 DEBUG:boto:Token: None 2022-09-12T19:48:13.211 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:13 GMT /jdpiak-72/h?versionId=null 2022-09-12T19:48:13.212 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X21nVE6XvPRFXjNdVIsSXuNl4Tg= 2022-09-12T19:48:13.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X21nVE6XvPRFXjNdVIsSXuNl4Tg=', 'Content-Length': '0'} 2022-09-12T19:48:13.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed909202a6854209-00631f8cfd-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:13.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:48:13.217 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:13.218 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:13.218 DEBUG:boto:path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:13.219 DEBUG:boto:auth_path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:13.219 DEBUG:boto:Method: GET 2022-09-12T19:48:13.220 DEBUG:boto:Path: /jdpiak-72/h?versionId=null 2022-09-12T19:48:13.220 DEBUG:boto:Data: 2022-09-12T19:48:13.221 DEBUG:boto:Headers: {} 2022-09-12T19:48:13.222 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:13.222 DEBUG:boto:Port: 8001 2022-09-12T19:48:13.223 DEBUG:boto:Params: {} 2022-09-12T19:48:13.224 DEBUG:boto:Token: None 2022-09-12T19:48:13.226 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:13 GMT /jdpiak-72/h?versionId=null 2022-09-12T19:48:13.227 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3SdZqV+5RvosGVBq44EYMl+PcB4= 2022-09-12T19:48:13.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3SdZqV+5RvosGVBq44EYMl+PcB4=', 'Content-Length': '0'} 2022-09-12T19:48:13.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005007af0810cb08ff-00631f8cfd-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:13.232 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:13.233 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:13.234 DEBUG:boto:path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:13.235 DEBUG:boto:auth_path=/jdpiak-72/h?versionId=null 2022-09-12T19:48:13.236 DEBUG:boto:Method: GET 2022-09-12T19:48:13.236 DEBUG:boto:Path: /jdpiak-72/h?versionId=null 2022-09-12T19:48:13.237 DEBUG:boto:Data: 2022-09-12T19:48:13.238 DEBUG:boto:Headers: {} 2022-09-12T19:48:13.238 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:13.239 DEBUG:boto:Port: 8000 2022-09-12T19:48:13.239 DEBUG:boto:Params: {} 2022-09-12T19:48:13.240 DEBUG:boto:Token: None 2022-09-12T19:48:13.240 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:13 GMT /jdpiak-72/h?versionId=null 2022-09-12T19:48:13.241 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3SdZqV+5RvosGVBq44EYMl+PcB4= 2022-09-12T19:48:13.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3SdZqV+5RvosGVBq44EYMl+PcB4=', 'Content-Length': '0'} 2022-09-12T19:48:13.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000172d5b32e8c7cf2a-00631f8cfd-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:13.245 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:13.245 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:13.246 DEBUG:boto:path=/jdpiak-72/?prefix=h 2022-09-12T19:48:13.246 DEBUG:boto:auth_path=/jdpiak-72/?prefix=h 2022-09-12T19:48:13.247 DEBUG:boto:Method: GET 2022-09-12T19:48:13.247 DEBUG:boto:Path: /jdpiak-72/?prefix=h 2022-09-12T19:48:13.248 DEBUG:boto:Data: 2022-09-12T19:48:13.248 DEBUG:boto:Headers: {} 2022-09-12T19:48:13.249 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:13.250 DEBUG:boto:Port: 8001 2022-09-12T19:48:13.250 DEBUG:boto:Params: {} 2022-09-12T19:48:13.251 DEBUG:boto:Token: None 2022-09-12T19:48:13.251 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:13 GMT /jdpiak-72/ 2022-09-12T19:48:13.252 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ntUE5lnNMPHHBQVmvNHDFSnp+nc= 2022-09-12T19:48:13.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ntUE5lnNMPHHBQVmvNHDFSnp+nc=', 'Content-Length': '0'} 2022-09-12T19:48:13.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f6e27959419a5a9-00631f8cfd-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:13.258 DEBUG:boto:b'jdpiak-72h1000falseh2022-09-12T19:45:20.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:13.259 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:13.259 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:13.260 DEBUG:boto:path=/jdpiak-72/?prefix=h 2022-09-12T19:48:13.260 DEBUG:boto:auth_path=/jdpiak-72/?prefix=h 2022-09-12T19:48:13.261 DEBUG:boto:Method: GET 2022-09-12T19:48:13.262 DEBUG:boto:Path: /jdpiak-72/?prefix=h 2022-09-12T19:48:13.262 DEBUG:boto:Data: 2022-09-12T19:48:13.263 DEBUG:boto:Headers: {} 2022-09-12T19:48:13.264 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:13.264 DEBUG:boto:Port: 8000 2022-09-12T19:48:13.265 DEBUG:boto:Params: {} 2022-09-12T19:48:13.265 DEBUG:boto:Token: None 2022-09-12T19:48:13.266 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:13 GMT /jdpiak-72/ 2022-09-12T19:48:13.267 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ntUE5lnNMPHHBQVmvNHDFSnp+nc= 2022-09-12T19:48:13.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ntUE5lnNMPHHBQVmvNHDFSnp+nc=', 'Content-Length': '0'} 2022-09-12T19:48:13.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006221552b6c394c53-00631f8cfd-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:13.270 DEBUG:boto:b'jdpiak-72h1000falseh2022-09-12T19:45:20.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:13.271 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:13.272 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:13.272 DEBUG:boto:Method: HEAD 2022-09-12T19:48:13.273 DEBUG:boto:Path: /jdpiak-72/h 2022-09-12T19:48:13.273 DEBUG:boto:Data: 2022-09-12T19:48:13.274 DEBUG:boto:Headers: {} 2022-09-12T19:48:13.274 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:13.275 DEBUG:boto:Port: 8001 2022-09-12T19:48:13.275 DEBUG:boto:Params: {} 2022-09-12T19:48:13.276 DEBUG:boto:Token: None 2022-09-12T19:48:13.276 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:13 GMT /jdpiak-72/h 2022-09-12T19:48:13.277 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NO+qzs4uEHWkIh+Kvz2Iisyhcnc= 2022-09-12T19:48:13.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NO+qzs4uEHWkIh+Kvz2Iisyhcnc=', 'Content-Length': '0'} 2022-09-12T19:48:13.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fc01aa33b2a2273-00631f8cfd-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:13.281 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:13.281 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:13.282 DEBUG:boto:Method: HEAD 2022-09-12T19:48:13.283 DEBUG:boto:Path: /jdpiak-72/h 2022-09-12T19:48:13.283 DEBUG:boto:Data: 2022-09-12T19:48:13.284 DEBUG:boto:Headers: {} 2022-09-12T19:48:13.285 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:13.285 DEBUG:boto:Port: 8000 2022-09-12T19:48:13.286 DEBUG:boto:Params: {} 2022-09-12T19:48:13.286 DEBUG:boto:Token: None 2022-09-12T19:48:13.287 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:13 GMT /jdpiak-72/h 2022-09-12T19:48:13.288 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NO+qzs4uEHWkIh+Kvz2Iisyhcnc= 2022-09-12T19:48:13.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NO+qzs4uEHWkIh+Kvz2Iisyhcnc=', 'Content-Length': '0'} 2022-09-12T19:48:13.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000619a5ea4350ef3c3-00631f8cfd-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:13.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-12T19:48:13.292 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:13.293 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:13.293 DEBUG:boto:Method: GET 2022-09-12T19:48:13.294 DEBUG:boto:Path: /jdpiak-72/h 2022-09-12T19:48:13.294 DEBUG:boto:Data: 2022-09-12T19:48:13.295 DEBUG:boto:Headers: {} 2022-09-12T19:48:13.295 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:13.296 DEBUG:boto:Port: 8001 2022-09-12T19:48:13.296 DEBUG:boto:Params: {} 2022-09-12T19:48:13.297 DEBUG:boto:Token: None 2022-09-12T19:48:13.298 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:13 GMT /jdpiak-72/h 2022-09-12T19:48:13.298 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qdfM8PtApZJ+AP+lU6AfybrGGJ8= 2022-09-12T19:48:13.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qdfM8PtApZJ+AP+lU6AfybrGGJ8=', 'Content-Length': '0'} 2022-09-12T19:48:13.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffad8bf5b6cd2f81-00631f8cfd-15c4-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:13.342 DEBUG:boto:path=/jdpiak-72/h 2022-09-12T19:48:13.343 DEBUG:boto:auth_path=/jdpiak-72/h 2022-09-12T19:48:13.344 DEBUG:boto:Method: GET 2022-09-12T19:48:13.344 DEBUG:boto:Path: /jdpiak-72/h 2022-09-12T19:48:13.345 DEBUG:boto:Data: 2022-09-12T19:48:13.345 DEBUG:boto:Headers: {} 2022-09-12T19:48:13.346 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:13.347 DEBUG:boto:Port: 8000 2022-09-12T19:48:13.347 DEBUG:boto:Params: {} 2022-09-12T19:48:13.348 DEBUG:boto:Token: None 2022-09-12T19:48:13.348 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:13 GMT /jdpiak-72/h 2022-09-12T19:48:13.349 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qdfM8PtApZJ+AP+lU6AfybrGGJ8= 2022-09-12T19:48:13.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qdfM8PtApZJ+AP+lU6AfybrGGJ8=', 'Content-Length': '0'} 2022-09-12T19:48:13.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 19:45:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4be233d8b027574-00631f8cfd-15e1-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 19:48:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:13.353 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:13.353 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:13.354 DEBUG:boto:path=/jdpiak-72/?prefix=h 2022-09-12T19:48:13.355 DEBUG:boto:auth_path=/jdpiak-72/?prefix=h 2022-09-12T19:48:13.355 DEBUG:boto:Method: GET 2022-09-12T19:48:13.356 DEBUG:boto:Path: /jdpiak-72/?prefix=h 2022-09-12T19:48:13.356 DEBUG:boto:Data: 2022-09-12T19:48:13.357 DEBUG:boto:Headers: {} 2022-09-12T19:48:13.358 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:13.358 DEBUG:boto:Port: 8001 2022-09-12T19:48:13.359 DEBUG:boto:Params: {} 2022-09-12T19:48:13.359 DEBUG:boto:Token: None 2022-09-12T19:48:13.360 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:13 GMT /jdpiak-72/ 2022-09-12T19:48:13.361 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ntUE5lnNMPHHBQVmvNHDFSnp+nc= 2022-09-12T19:48:13.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ntUE5lnNMPHHBQVmvNHDFSnp+nc=', 'Content-Length': '0'} 2022-09-12T19:48:13.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a132b20509c2b6fb-00631f8cfd-15c4-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:13.367 DEBUG:boto:b'jdpiak-72h1000falseh2022-09-12T19:45:20.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:13.367 DEBUG:boto:path=/jdpiak-72/ 2022-09-12T19:48:13.368 DEBUG:boto:auth_path=/jdpiak-72/ 2022-09-12T19:48:13.369 DEBUG:boto:path=/jdpiak-72/?prefix=h 2022-09-12T19:48:13.369 DEBUG:boto:auth_path=/jdpiak-72/?prefix=h 2022-09-12T19:48:13.370 DEBUG:boto:Method: GET 2022-09-12T19:48:13.370 DEBUG:boto:Path: /jdpiak-72/?prefix=h 2022-09-12T19:48:13.371 DEBUG:boto:Data: 2022-09-12T19:48:13.371 DEBUG:boto:Headers: {} 2022-09-12T19:48:13.372 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:13.372 DEBUG:boto:Port: 8000 2022-09-12T19:48:13.373 DEBUG:boto:Params: {} 2022-09-12T19:48:13.373 DEBUG:boto:Token: None 2022-09-12T19:48:13.374 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:48:13 GMT /jdpiak-72/ 2022-09-12T19:48:13.375 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ntUE5lnNMPHHBQVmvNHDFSnp+nc= 2022-09-12T19:48:13.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ntUE5lnNMPHHBQVmvNHDFSnp+nc=', 'Content-Length': '0'} 2022-09-12T19:48:13.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000baf429adfbeae340-00631f8cfd-15e1-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:48:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:13.379 DEBUG:boto:b'jdpiak-72h1000falseh2022-09-12T19:45:20.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T19:48:13.380 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-72 zones={test-zone3, test-zone2} 2022-09-12T19:48:13.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... ok 2022-09-12T19:48:13.384 DEBUG:boto:path=/jdpiak-73/ 2022-09-12T19:48:13.385 DEBUG:boto:auth_path=/jdpiak-73/ 2022-09-12T19:48:13.385 DEBUG:boto:Method: PUT 2022-09-12T19:48:13.386 DEBUG:boto:Path: /jdpiak-73/ 2022-09-12T19:48:13.386 DEBUG:boto:Data: 2022-09-12T19:48:13.387 DEBUG:boto:Headers: {} 2022-09-12T19:48:13.387 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:13.388 DEBUG:boto:Port: 8000 2022-09-12T19:48:13.389 DEBUG:boto:Params: {} 2022-09-12T19:48:13.389 DEBUG:boto:Token: None 2022-09-12T19:48:13.390 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:48:13 GMT /jdpiak-73/ 2022-09-12T19:48:13.391 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G7Ba/ZwoJTHDuiRa7p8sdbaQoe4= 2022-09-12T19:48:13.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G7Ba/ZwoJTHDuiRa7p8sdbaQoe4=', 'Content-Length': '0'} 2022-09-12T19:48:13.414 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dd968af016a0aecd-00631f8cfd-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:48:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:13.415 DEBUG:rgw_multi.tests:created bucket=jdpiak-73 2022-09-12T19:48:13.416 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:48:13.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:48:13.417 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:48:13.475 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:48:13.476 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:48:13.516 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=38ca1f57-1358-496c-a99b-df49ee365a4f 2022-09-12T19:48:13.530 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:48:13.531 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:48:13.531 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:48:13.531 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:48:13.532 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:48:13.532 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:48:13.532 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:48:13.533 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:48:13.533 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:48:13.533 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:48:13.533 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663012093.411077_700.1", 2022-09-12T19:48:13.534 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:48:13.411077Z" 2022-09-12T19:48:13.534 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:48:13.534 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:48:13.535 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:48:13.535 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:48:13.535 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:48:13.536 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:48:13.537 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1663012093.411077_700.1', 3: ''} 2022-09-12T19:48:13.538 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T19:48:13.538 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:48:13.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:48:13.541 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:48:13.598 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:48:13.599 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:48:13.631 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:48:13.629+0000 7fd990705600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:48:13.632 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:48:13.630+0000 7fd990705600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:48:13.646 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:48:13.647 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:48:13.648 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:48:13.648 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:48:13.649 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:48:13.649 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:48:13.650 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:48:13.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:48:13.651 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:48:13.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:48:13.652 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:48:13.652 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:48:13.653 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:48:13.653 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:48:13.654 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:48:13.654 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:48:13.655 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:48:13.655 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:48:13.656 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:48:13.657 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:48:13.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:48:13.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:48:13.658 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:48:13.659 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:48:13.659 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:48:13.660 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:48:13.661 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:48:13.661 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:48:13.661 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:48:13.662 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:48:13.662 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:48:13.663 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:48:13.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:48:13.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:48:13.664 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:48:13.664 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:48:13.669 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:48:13.669 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663012093.402005_697.1", 2022-09-12T19:48:13.670 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:48:13.670 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:48:13.671 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:48:13.671 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:48:13.402005Z", 2022-09-12T19:48:13.671 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:48:13.671 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:48:13.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:48:13.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:48:13.673 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:48:13.673 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:48:13.674 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:48:13.674 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:48:13.675 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:48:13.675 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:48:13.675 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:48:13.675 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:48:13.676 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:48:13.676 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:48:13.676 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:48:13.676 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:48:13.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:48:13.677 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:48:13.678 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:48:13.678 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:48:13.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:48:13.679 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:48:13Z" 2022-09-12T19:48:13.680 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:48:13.682 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663012093.402005_697.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:48:13.402005Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:48:13Z" } 2022-09-12T19:48:13.683 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663012093.402005_697.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:48:13.402005Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:48:13.684 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663012093.411077_700.1', 3: ''} 2022-09-12T19:48:13.685 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663012093.402005_697.1', 3: ''} 2022-09-12T19:48:13.686 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663012093.411077_700.1 target=1_1663012093.402005_697.1 2022-09-12T19:48:18.687 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:48:18.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:48:18.689 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:48:18.746 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:48:18.746 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:48:18.790 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:48:18.787+0000 7faf1eb31600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:48:18.791 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:48:18.789+0000 7faf1eb31600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:48:18.809 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:48:18.809 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:48:18.809 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:48:18.810 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:48:18.810 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:48:18.810 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:48:18.811 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:48:18.811 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:48:18.811 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:48:18.812 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:48:18.812 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:48:18.812 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:48:18.813 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:48:18.813 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:48:18.813 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:48:18.814 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:48:18.814 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:48:18.814 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:48:18.814 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:48:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:48:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:48:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:48:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:48:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:48:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:48:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:48:18.817 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:48:18.817 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:48:18.817 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:48:18.817 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:48:18.818 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:48:18.818 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:48:18.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:48:18.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:48:18.819 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:48:18.820 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:48:18.820 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:48:18.820 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663012093.411077_700.1", 2022-09-12T19:48:18.821 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:48:18.821 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:48:18.821 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:48:18.821 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:48:13.411077Z", 2022-09-12T19:48:18.822 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:48:18.822 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:48:18.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:48:18.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:48:18.823 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:48:18.823 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:48:18.823 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:48:18.824 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:48:18.824 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:48:18.824 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:48:18.824 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:48:18.825 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:48:18.825 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:48:18.825 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:48:18.825 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:48:18.826 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:48:18.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:48:18.826 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:48:18.827 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:48:18.827 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:48:18.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:48:18.827 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:48:18Z" 2022-09-12T19:48:18.828 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:48:18.829 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663012093.411077_700.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:48:13.411077Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:48:18Z" } 2022-09-12T19:48:18.830 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663012093.411077_700.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:48:13.411077Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:48:18.831 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663012093.411077_700.1', 3: ''} 2022-09-12T19:48:18.831 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663012093.411077_700.1', 3: ''} 2022-09-12T19:48:18.832 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T19:48:18.832 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T19:48:18.833 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:48:18.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:48:18.834 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:48:18.890 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:48:18.890 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:48:18.935 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:48:18.933+0000 7f8f0526d600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:48:18.936 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:48:18.934+0000 7f8f0526d600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:48:18.953 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:48:18.954 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:48:18.954 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:48:18.954 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:48:18.955 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:48:18.955 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:48:18.955 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:48:18.955 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:48:18.956 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:48:18.956 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:48:18.956 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:48:18.956 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:48:18.957 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:48:18.957 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:48:18.957 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:48:18.957 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:48:18.958 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:48:18.958 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:48:18.958 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:48:18.958 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:48:18.959 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:48:18.959 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:48:18.959 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:48:18.959 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:48:18.960 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:48:18.960 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:48:18.960 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:48:18.960 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:48:18.961 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:48:18.961 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:48:18.961 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:48:18.961 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:48:18.962 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:48:18.962 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:48:18.962 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:48:18.963 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:48:18.963 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:48:18.963 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663012093.411077_700.1", 2022-09-12T19:48:18.963 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:48:18.964 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:48:18.964 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:48:18.964 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:48:13.411077Z", 2022-09-12T19:48:18.964 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:48:18.965 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:48:18.965 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:48:18.965 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:48:18.965 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:48:18.966 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:48:18.966 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:48:18.966 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:48:18.967 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:48:18.967 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:48:18.967 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:48:18.968 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:48:18.968 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:48:18.968 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:48:18.968 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:48:18.969 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:48:18.969 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:48:18.969 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:48:18.969 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:48:18.970 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:48:18.970 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:48:18.970 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:48:18Z" 2022-09-12T19:48:18.970 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:48:18.971 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663012093.411077_700.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:48:13.411077Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:48:18Z" } 2022-09-12T19:48:18.971 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663012093.411077_700.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:48:13.411077Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:48:18.972 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663012093.411077_700.1', 3: ''} 2022-09-12T19:48:18.973 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663012093.411077_700.1', 3: ''} 2022-09-12T19:48:18.973 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T19:48:18.974 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T19:48:18.974 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:48:18.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:48:18.975 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:48:19.028 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:48:19.028 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:48:19.061 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:48:19.058+0000 7f5da2d84600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:48:19.062 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:48:19.059+0000 7f5da2d84600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:48:19.073 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:48:19.074 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:48:19.074 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:48:19.075 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:48:19.075 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:48:19.075 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:48:19.075 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:48:19.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:48:19.076 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:48:19.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:48:19.076 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:48:19.077 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:48:19.077 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:48:19.077 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:48:19.077 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:48:19.078 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:48:19.078 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:48:19.079 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:48:19.079 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:48:19.079 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:48:19.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:48:19.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:48:19.080 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:48:19.080 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:48:19.080 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:48:19.081 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:48:19.081 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:48:19.081 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:48:19.081 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:48:19.082 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:48:19.082 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:48:19.082 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:48:19.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:48:19.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:48:19.083 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:48:19.083 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:48:19.084 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:48:19.084 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663012093.411077_700.1", 2022-09-12T19:48:19.084 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:48:19.085 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:48:19.085 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:48:19.085 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:48:13.411077Z", 2022-09-12T19:48:19.085 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:48:19.086 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:48:19.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:48:19.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:48:19.086 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:48:19.087 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:48:19.087 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:48:19.087 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:48:19.088 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:48:19.088 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:48:19.088 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:48:19.088 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:48:19.089 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:48:19.089 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:48:19.089 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:48:19.090 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:48:19.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:48:19.090 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:48:19.090 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:48:19.091 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:48:19.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:48:19.092 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:48:19Z" 2022-09-12T19:48:19.092 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:48:19.093 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663012093.411077_700.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:48:13.411077Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:48:19Z" } 2022-09-12T19:48:19.094 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663012093.411077_700.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:48:13.411077Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:48:19.095 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663012093.411077_700.1', 3: ''} 2022-09-12T19:48:19.095 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663012093.411077_700.1', 3: ''} 2022-09-12T19:48:19.095 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T19:48:19.096 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2022-09-12T19:48:19.097 INFO:teuthology.orchestra.run:waiting for 300 2022-09-12T19:48:19.098 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:48:19.096+0000 7fa8deada700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 108977) UID: 0 2022-09-12T19:48:19.098 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T19:48:19.096+0000 7fa8e2d45600 -1 shutting down 2022-09-12T19:48:24.800 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2022-09-12T19:48:25.200 INFO:tasks.rgw.c2.client.0:Stopped 2022-09-12T19:48:25.201 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2022-09-12T19:48:25.201 INFO:teuthology.orchestra.run:waiting for 300 2022-09-12T19:48:25.203 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:48:25.200+0000 7f55ae035700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 125274) UID: 0 2022-09-12T19:48:25.203 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T19:48:25.200+0000 7f55b22a0600 -1 shutting down 2022-09-12T19:48:30.260 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2022-09-12T19:48:30.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2022-09-12T19:48:31.303 INFO:tasks.rgw.c1.client.1:Stopped 2022-09-12T19:48:31.304 DEBUG:boto:path=/jdpiak-73/ 2022-09-12T19:48:31.305 DEBUG:boto:auth_path=/jdpiak-73/ 2022-09-12T19:48:31.305 DEBUG:boto:Method: HEAD 2022-09-12T19:48:31.306 DEBUG:boto:Path: /jdpiak-73/ 2022-09-12T19:48:31.307 DEBUG:boto:Data: 2022-09-12T19:48:31.307 DEBUG:boto:Headers: {} 2022-09-12T19:48:31.308 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:31.308 DEBUG:boto:Port: 8000 2022-09-12T19:48:31.309 DEBUG:boto:Params: {} 2022-09-12T19:48:31.309 DEBUG:boto:Token: None 2022-09-12T19:48:31.310 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:31 GMT /jdpiak-73/ 2022-09-12T19:48:31.310 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hQWCqiqV9P3Oos+bGVpvMeQWABc= 2022-09-12T19:48:31.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hQWCqiqV9P3Oos+bGVpvMeQWABc=', 'Content-Length': '0'} 2022-09-12T19:48:31.318 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e407246b91dcb64-00631f8d0f-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:48:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:31.319 DEBUG:boto:path=/jdpiak-73/a 2022-09-12T19:48:31.321 DEBUG:boto:auth_path=/jdpiak-73/a 2022-09-12T19:48:31.321 DEBUG:boto:Method: PUT 2022-09-12T19:48:31.322 DEBUG:boto:Path: /jdpiak-73/a 2022-09-12T19:48:31.323 DEBUG:boto:Data: 2022-09-12T19:48:31.324 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:48:31.325 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:31.326 DEBUG:boto:Port: 8000 2022-09-12T19:48:31.326 DEBUG:boto:Params: {} 2022-09-12T19:48:31.327 DEBUG:boto:Token: None 2022-09-12T19:48:31.328 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:48:31 GMT /jdpiak-73/a 2022-09-12T19:48:31.329 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GRcPOQxcrP0keNrzjAsdReaOr6k= 2022-09-12T19:48:31.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:48:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GRcPOQxcrP0keNrzjAsdReaOr6k='} 2022-09-12T19:48:31.339 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b3a43f42f9f2659-00631f8d0f-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:48:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:31.340 DEBUG:boto:path=/jdpiak-73/ 2022-09-12T19:48:31.341 DEBUG:boto:auth_path=/jdpiak-73/ 2022-09-12T19:48:31.342 DEBUG:boto:Method: HEAD 2022-09-12T19:48:31.343 DEBUG:boto:Path: /jdpiak-73/ 2022-09-12T19:48:31.344 DEBUG:boto:Data: 2022-09-12T19:48:31.345 DEBUG:boto:Headers: {} 2022-09-12T19:48:31.346 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:31.347 DEBUG:boto:Port: 8000 2022-09-12T19:48:31.347 DEBUG:boto:Params: {} 2022-09-12T19:48:31.348 DEBUG:boto:Token: None 2022-09-12T19:48:31.349 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:31 GMT /jdpiak-73/ 2022-09-12T19:48:31.350 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hQWCqiqV9P3Oos+bGVpvMeQWABc= 2022-09-12T19:48:31.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hQWCqiqV9P3Oos+bGVpvMeQWABc=', 'Content-Length': '0'} 2022-09-12T19:48:31.355 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000048217866fa4ad35b-00631f8d0f-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:48:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:31.356 DEBUG:boto:path=/jdpiak-73/b 2022-09-12T19:48:31.357 DEBUG:boto:auth_path=/jdpiak-73/b 2022-09-12T19:48:31.358 DEBUG:boto:Method: PUT 2022-09-12T19:48:31.359 DEBUG:boto:Path: /jdpiak-73/b 2022-09-12T19:48:31.360 DEBUG:boto:Data: 2022-09-12T19:48:31.361 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:48:31.362 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:31.363 DEBUG:boto:Port: 8000 2022-09-12T19:48:31.363 DEBUG:boto:Params: {} 2022-09-12T19:48:31.364 DEBUG:boto:Token: None 2022-09-12T19:48:31.365 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:48:31 GMT /jdpiak-73/b 2022-09-12T19:48:31.366 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p92Rbm+MIFwIaSYdzMToOB3T4hM= 2022-09-12T19:48:31.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:48:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p92Rbm+MIFwIaSYdzMToOB3T4hM='} 2022-09-12T19:48:31.414 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da016db43f9083c8-00631f8d0f-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:48:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:31.415 DEBUG:boto:path=/jdpiak-73/ 2022-09-12T19:48:31.416 DEBUG:boto:auth_path=/jdpiak-73/ 2022-09-12T19:48:31.416 DEBUG:boto:Method: HEAD 2022-09-12T19:48:31.417 DEBUG:boto:Path: /jdpiak-73/ 2022-09-12T19:48:31.417 DEBUG:boto:Data: 2022-09-12T19:48:31.418 DEBUG:boto:Headers: {} 2022-09-12T19:48:31.418 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:31.419 DEBUG:boto:Port: 8000 2022-09-12T19:48:31.419 DEBUG:boto:Params: {} 2022-09-12T19:48:31.420 DEBUG:boto:Token: None 2022-09-12T19:48:31.420 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:31 GMT /jdpiak-73/ 2022-09-12T19:48:31.421 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hQWCqiqV9P3Oos+bGVpvMeQWABc= 2022-09-12T19:48:31.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hQWCqiqV9P3Oos+bGVpvMeQWABc=', 'Content-Length': '0'} 2022-09-12T19:48:31.424 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1f482a28f8ab4ca-00631f8d0f-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:48:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:31.425 DEBUG:boto:path=/jdpiak-73/c 2022-09-12T19:48:31.426 DEBUG:boto:auth_path=/jdpiak-73/c 2022-09-12T19:48:31.426 DEBUG:boto:Method: PUT 2022-09-12T19:48:31.427 DEBUG:boto:Path: /jdpiak-73/c 2022-09-12T19:48:31.427 DEBUG:boto:Data: 2022-09-12T19:48:31.428 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:48:31.428 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:31.429 DEBUG:boto:Port: 8000 2022-09-12T19:48:31.429 DEBUG:boto:Params: {} 2022-09-12T19:48:31.430 DEBUG:boto:Token: None 2022-09-12T19:48:31.431 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:48:31 GMT /jdpiak-73/c 2022-09-12T19:48:31.431 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QpRHAU/9ReZluL4yvOFdoHDw9zU= 2022-09-12T19:48:31.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:48:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QpRHAU/9ReZluL4yvOFdoHDw9zU='} 2022-09-12T19:48:31.439 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000debef3bf19f0e8d3-00631f8d0f-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:48:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:31.440 DEBUG:boto:path=/jdpiak-73/ 2022-09-12T19:48:31.441 DEBUG:boto:auth_path=/jdpiak-73/ 2022-09-12T19:48:31.441 DEBUG:boto:Method: HEAD 2022-09-12T19:48:31.442 DEBUG:boto:Path: /jdpiak-73/ 2022-09-12T19:48:31.442 DEBUG:boto:Data: 2022-09-12T19:48:31.443 DEBUG:boto:Headers: {} 2022-09-12T19:48:31.443 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:31.444 DEBUG:boto:Port: 8000 2022-09-12T19:48:31.444 DEBUG:boto:Params: {} 2022-09-12T19:48:31.445 DEBUG:boto:Token: None 2022-09-12T19:48:31.445 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:31 GMT /jdpiak-73/ 2022-09-12T19:48:31.446 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hQWCqiqV9P3Oos+bGVpvMeQWABc= 2022-09-12T19:48:31.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hQWCqiqV9P3Oos+bGVpvMeQWABc=', 'Content-Length': '0'} 2022-09-12T19:48:31.449 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060d852fa30ff95d8-00631f8d0f-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:48:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:31.450 DEBUG:boto:path=/jdpiak-73/d 2022-09-12T19:48:31.451 DEBUG:boto:auth_path=/jdpiak-73/d 2022-09-12T19:48:31.451 DEBUG:boto:Method: PUT 2022-09-12T19:48:31.452 DEBUG:boto:Path: /jdpiak-73/d 2022-09-12T19:48:31.452 DEBUG:boto:Data: 2022-09-12T19:48:31.452 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:48:31.453 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:31.453 DEBUG:boto:Port: 8000 2022-09-12T19:48:31.454 DEBUG:boto:Params: {} 2022-09-12T19:48:31.455 DEBUG:boto:Token: None 2022-09-12T19:48:31.455 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:48:31 GMT /jdpiak-73/d 2022-09-12T19:48:31.456 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LVHNPX31ZIU/++TA1QrDD9grA68= 2022-09-12T19:48:31.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:48:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LVHNPX31ZIU/++TA1QrDD9grA68='} 2022-09-12T19:48:31.464 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f15efb00dfe9db02-00631f8d0f-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:48:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:31.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'jdpiak-73', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:48:31.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'jdpiak-73', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T19:48:31.466 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket jdpiak-73 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T19:48:31.525 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:48:31.526 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:48:31.596 INFO:teuthology.orchestra.run.smithi035.stdout:tenant: 2022-09-12T19:48:31.597 INFO:teuthology.orchestra.run.smithi035.stdout:bucket name: jdpiak-73 2022-09-12T19:48:31.601 INFO:teuthology.orchestra.run.smithi035.stdout:total entries: 4 2022-09-12T19:48:31.669 DEBUG:boto:path=/jdpiak-73/ 2022-09-12T19:48:31.669 DEBUG:boto:auth_path=/jdpiak-73/ 2022-09-12T19:48:31.670 DEBUG:boto:Method: HEAD 2022-09-12T19:48:31.670 DEBUG:boto:Path: /jdpiak-73/ 2022-09-12T19:48:31.671 DEBUG:boto:Data: 2022-09-12T19:48:31.671 DEBUG:boto:Headers: {} 2022-09-12T19:48:31.672 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:31.673 DEBUG:boto:Port: 8000 2022-09-12T19:48:31.673 DEBUG:boto:Params: {} 2022-09-12T19:48:31.674 DEBUG:boto:Token: None 2022-09-12T19:48:31.674 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:31 GMT /jdpiak-73/ 2022-09-12T19:48:31.675 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hQWCqiqV9P3Oos+bGVpvMeQWABc= 2022-09-12T19:48:31.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hQWCqiqV9P3Oos+bGVpvMeQWABc=', 'Content-Length': '0'} 2022-09-12T19:48:31.678 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090e6a5bde97e8318-00631f8d0f-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:48:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:31.678 DEBUG:boto:path=/jdpiak-73/e 2022-09-12T19:48:31.679 DEBUG:boto:auth_path=/jdpiak-73/e 2022-09-12T19:48:31.679 DEBUG:boto:Method: PUT 2022-09-12T19:48:31.680 DEBUG:boto:Path: /jdpiak-73/e 2022-09-12T19:48:31.681 DEBUG:boto:Data: 2022-09-12T19:48:31.681 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:48:31.682 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:31.682 DEBUG:boto:Port: 8000 2022-09-12T19:48:31.683 DEBUG:boto:Params: {} 2022-09-12T19:48:31.683 DEBUG:boto:Token: None 2022-09-12T19:48:31.684 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:48:31 GMT /jdpiak-73/e 2022-09-12T19:48:31.684 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q9Unx+RD5SGHHP5atCVq1qaJ+Co= 2022-09-12T19:48:31.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:48:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q9Unx+RD5SGHHP5atCVq1qaJ+Co='} 2022-09-12T19:48:31.693 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000392f966761b92e5f-00631f8d0f-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:48:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:31.694 DEBUG:boto:path=/jdpiak-73/ 2022-09-12T19:48:31.695 DEBUG:boto:auth_path=/jdpiak-73/ 2022-09-12T19:48:31.695 DEBUG:boto:Method: HEAD 2022-09-12T19:48:31.696 DEBUG:boto:Path: /jdpiak-73/ 2022-09-12T19:48:31.697 DEBUG:boto:Data: 2022-09-12T19:48:31.697 DEBUG:boto:Headers: {} 2022-09-12T19:48:31.698 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:31.698 DEBUG:boto:Port: 8000 2022-09-12T19:48:31.699 DEBUG:boto:Params: {} 2022-09-12T19:48:31.699 DEBUG:boto:Token: None 2022-09-12T19:48:31.700 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:31 GMT /jdpiak-73/ 2022-09-12T19:48:31.701 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hQWCqiqV9P3Oos+bGVpvMeQWABc= 2022-09-12T19:48:31.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hQWCqiqV9P3Oos+bGVpvMeQWABc=', 'Content-Length': '0'} 2022-09-12T19:48:31.704 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9766b869544a9fc-00631f8d0f-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:48:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:31.705 DEBUG:boto:path=/jdpiak-73/f 2022-09-12T19:48:31.706 DEBUG:boto:auth_path=/jdpiak-73/f 2022-09-12T19:48:31.707 DEBUG:boto:Method: PUT 2022-09-12T19:48:31.708 DEBUG:boto:Path: /jdpiak-73/f 2022-09-12T19:48:31.709 DEBUG:boto:Data: 2022-09-12T19:48:31.710 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:48:31.711 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:31.711 DEBUG:boto:Port: 8000 2022-09-12T19:48:31.712 DEBUG:boto:Params: {} 2022-09-12T19:48:31.713 DEBUG:boto:Token: None 2022-09-12T19:48:31.714 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:48:31 GMT /jdpiak-73/f 2022-09-12T19:48:31.715 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BCKR/4b7052QBt5FCpw71l9z3kQ= 2022-09-12T19:48:31.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:48:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BCKR/4b7052QBt5FCpw71l9z3kQ='} 2022-09-12T19:48:31.758 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000026c8cb2df603f060-00631f8d0f-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:48:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:31.760 DEBUG:boto:path=/jdpiak-73/ 2022-09-12T19:48:31.761 DEBUG:boto:auth_path=/jdpiak-73/ 2022-09-12T19:48:31.761 DEBUG:boto:Method: HEAD 2022-09-12T19:48:31.762 DEBUG:boto:Path: /jdpiak-73/ 2022-09-12T19:48:31.762 DEBUG:boto:Data: 2022-09-12T19:48:31.763 DEBUG:boto:Headers: {} 2022-09-12T19:48:31.763 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:31.764 DEBUG:boto:Port: 8000 2022-09-12T19:48:31.764 DEBUG:boto:Params: {} 2022-09-12T19:48:31.765 DEBUG:boto:Token: None 2022-09-12T19:48:31.766 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:31 GMT /jdpiak-73/ 2022-09-12T19:48:31.766 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hQWCqiqV9P3Oos+bGVpvMeQWABc= 2022-09-12T19:48:31.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hQWCqiqV9P3Oos+bGVpvMeQWABc=', 'Content-Length': '0'} 2022-09-12T19:48:31.769 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000012cf3ecd7d0cab7-00631f8d0f-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:48:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:31.770 DEBUG:boto:path=/jdpiak-73/g 2022-09-12T19:48:31.770 DEBUG:boto:auth_path=/jdpiak-73/g 2022-09-12T19:48:31.771 DEBUG:boto:Method: PUT 2022-09-12T19:48:31.772 DEBUG:boto:Path: /jdpiak-73/g 2022-09-12T19:48:31.772 DEBUG:boto:Data: 2022-09-12T19:48:31.773 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:48:31.773 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:31.774 DEBUG:boto:Port: 8000 2022-09-12T19:48:31.775 DEBUG:boto:Params: {} 2022-09-12T19:48:31.775 DEBUG:boto:Token: None 2022-09-12T19:48:31.776 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:48:31 GMT /jdpiak-73/g 2022-09-12T19:48:31.776 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sSUOBIZkl0VZsm8QDIU4TB6yUWc= 2022-09-12T19:48:31.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:48:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sSUOBIZkl0VZsm8QDIU4TB6yUWc='} 2022-09-12T19:48:31.784 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a8d57ba38bfd2221-00631f8d0f-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:48:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:31.785 DEBUG:boto:path=/jdpiak-73/ 2022-09-12T19:48:31.785 DEBUG:boto:auth_path=/jdpiak-73/ 2022-09-12T19:48:31.786 DEBUG:boto:Method: HEAD 2022-09-12T19:48:31.786 DEBUG:boto:Path: /jdpiak-73/ 2022-09-12T19:48:31.787 DEBUG:boto:Data: 2022-09-12T19:48:31.787 DEBUG:boto:Headers: {} 2022-09-12T19:48:31.788 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:31.788 DEBUG:boto:Port: 8000 2022-09-12T19:48:31.789 DEBUG:boto:Params: {} 2022-09-12T19:48:31.790 DEBUG:boto:Token: None 2022-09-12T19:48:31.790 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:48:31 GMT /jdpiak-73/ 2022-09-12T19:48:31.791 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hQWCqiqV9P3Oos+bGVpvMeQWABc= 2022-09-12T19:48:31.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:48:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hQWCqiqV9P3Oos+bGVpvMeQWABc=', 'Content-Length': '0'} 2022-09-12T19:48:31.793 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023c04feabe7cc140-00631f8d0f-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:48:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:31.794 DEBUG:boto:path=/jdpiak-73/h 2022-09-12T19:48:31.795 DEBUG:boto:auth_path=/jdpiak-73/h 2022-09-12T19:48:31.795 DEBUG:boto:Method: PUT 2022-09-12T19:48:31.796 DEBUG:boto:Path: /jdpiak-73/h 2022-09-12T19:48:31.796 DEBUG:boto:Data: 2022-09-12T19:48:31.797 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:48:31.797 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:48:31.798 DEBUG:boto:Port: 8000 2022-09-12T19:48:31.798 DEBUG:boto:Params: {} 2022-09-12T19:48:31.799 DEBUG:boto:Token: None 2022-09-12T19:48:31.799 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:48:31 GMT /jdpiak-73/h 2022-09-12T19:48:31.800 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mboTdvYH+wWhReibO70d2ePZ8nA= 2022-09-12T19:48:31.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:48:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mboTdvYH+wWhReibO70d2ePZ8nA='} 2022-09-12T19:48:31.806 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e0369df2c2a077b7-00631f8d0f-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:48:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:48:31.807 INFO:tasks.rgw.c2.client.0:Restarting daemon 2022-09-12T19:48:31.808 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2022-09-12T19:48:31.812 INFO:tasks.rgw.c2.client.0:Started 2022-09-12T19:48:31.813 DEBUG:teuthology.orchestra.run.smithi138:> curl http://smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:31.841 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-12T19:48:31.843 INFO:teuthology.orchestra.run.smithi138.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-12T19:48:31.843 INFO:teuthology.orchestra.run.smithi138.stderr: Dload Upload Total Spent Left Speed 2022-09-12T19:48:31.844 INFO:teuthology.orchestra.run.smithi138.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi138.front.sepia.ceph.com port 8000: Connection refused 2022-09-12T19:48:32.844 DEBUG:teuthology.orchestra.run.smithi138:> curl http://smithi138.front.sepia.ceph.com:8000 2022-09-12T19:48:32.868 INFO:teuthology.orchestra.run.smithi138.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-12T19:48:32.869 INFO:teuthology.orchestra.run.smithi138.stderr: Dload Upload Total Spent Left Speed 2022-09-12T19:48:32.869 INFO:teuthology.orchestra.run.smithi138.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-12T19:48:32.871 INFO:teuthology.orchestra.run.smithi138.stdout:anonymous 2022-09-12T19:48:32.871 INFO:tasks.rgw.c1.client.1:Restarting daemon 2022-09-12T19:48:32.872 DEBUG:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2022-09-12T19:48:32.876 INFO:tasks.rgw.c1.client.1:Started 2022-09-12T19:48:32.876 DEBUG:teuthology.orchestra.run.smithi035:> curl http://smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:32.904 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-12T19:48:32.905 INFO:teuthology.orchestra.run.smithi035.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-12T19:48:32.906 INFO:teuthology.orchestra.run.smithi035.stderr: Dload Upload Total Spent Left Speed 2022-09-12T19:48:32.906 INFO:teuthology.orchestra.run.smithi035.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi035.front.sepia.ceph.com port 8001: Connection refused 2022-09-12T19:48:33.908 DEBUG:teuthology.orchestra.run.smithi035:> curl http://smithi035.front.sepia.ceph.com:8001 2022-09-12T19:48:33.932 INFO:teuthology.orchestra.run.smithi035.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-12T19:48:33.933 INFO:teuthology.orchestra.run.smithi035.stderr: Dload Upload Total Spent Left Speed 2022-09-12T19:48:33.934 INFO:teuthology.orchestra.run.smithi035.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-12T19:48:33.935 INFO:teuthology.orchestra.run.smithi035.stdout:anonymous 2022-09-12T19:48:33.935 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-73 2022-09-12T19:48:33.936 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:48:33.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:48:33.937 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-73 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:48:33.997 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:48:33.997 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:48:34.032 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:48:34.029+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:48:35.965 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2022-09-12T19:48:39.032 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:48:39.030+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:48:44.033 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:48:44.031+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:48:49.033 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:48:49.031+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:48:54.034 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:48:54.032+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:48:59.035 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:48:59.032+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:49:04.035 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:49:04.033+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:49:09.036 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:49:09.034+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:49:14.037 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:49:14.034+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:49:19.037 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:49:19.034+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:49:24.037 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:49:24.035+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:49:29.038 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:49:29.036+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:49:34.039 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:49:34.036+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:49:39.039 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:49:39.037+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:49:44.040 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:49:44.038+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:49:49.040 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:49:49.038+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:49:54.041 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:49:54.039+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:49:59.041 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:49:59.039+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:50:04.044 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:50:04.040+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:50:09.043 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:50:09.040+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:50:14.043 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:50:14.041+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:50:19.044 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:50:19.041+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:50:24.045 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:50:24.042+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:50:29.045 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:50:29.043+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:50:34.046 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:50:34.044+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:50:39.046 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:50:39.045+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:50:44.047 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:50:44.045+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:50:49.047 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:50:49.045+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:50:54.048 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:50:54.046+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:50:59.049 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:50:59.047+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:51:04.050 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:51:04.047+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:51:09.050 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:51:09.048+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:51:14.051 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:51:14.049+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:51:19.051 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:51:19.049+0000 7f5824b3b600 1 waiting to reach incremental sync.. 2022-09-12T19:51:24.053 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:51:24.051+0000 7f5824b3b600 1 bucket sync caught up with source: 2022-09-12T19:51:24.054 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:51:24.054 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:51:24.055 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:51:24.051+0000 7f5824b3b600 0 bucket checkpoint complete 2022-09-12T19:51:24.073 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-73 2022-09-12T19:51:24.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:51:24.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:51:24.077 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-73 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:51:24.147 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:51:24.147 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:51:24.206 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:51:24.205+0000 7f11138f4600 1 waiting to reach incremental sync.. 2022-09-12T19:51:29.207 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:51:29.205+0000 7f11138f4600 1 bucket sync caught up with source: 2022-09-12T19:51:29.208 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:51:29.208 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:51:29.208 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:51:29.205+0000 7f11138f4600 0 bucket checkpoint complete 2022-09-12T19:51:29.225 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-73 2022-09-12T19:51:29.226 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:51:29.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:51:29.227 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-73 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:51:29.292 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:51:29.293 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:51:29.331 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:51:29.328+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:51:29.331 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:51:29.332 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:51:34.332 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:51:34.329+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:51:34.332 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:51:34.333 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:51:39.333 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:51:39.330+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:51:39.333 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:51:39.334 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:51:44.334 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:51:44.331+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:51:44.335 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:51:44.335 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:51:49.334 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:51:49.332+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:51:49.335 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:51:49.335 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:51:54.336 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:51:54.333+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:51:54.336 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:51:54.336 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:51:59.337 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:51:59.334+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:51:59.338 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:51:59.338 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:52:04.338 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:52:04.335+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:52:04.338 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:52:04.338 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:52:09.338 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:52:09.337+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:52:09.339 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:52:09.339 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:52:14.339 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:52:14.337+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:52:14.340 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:52:14.340 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:52:19.340 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:52:19.338+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:52:19.341 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:52:19.341 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:52:24.341 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:52:24.339+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:52:24.342 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:52:24.343 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:52:29.342 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:52:29.340+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:52:29.343 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:52:29.343 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:52:34.343 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:52:34.341+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:52:34.344 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:52:34.344 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:52:39.344 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:52:39.342+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:52:39.345 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:52:39.345 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:52:44.345 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:52:44.343+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:52:44.346 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:52:44.346 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:52:49.347 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:52:49.344+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:52:49.347 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:52:49.348 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:52:54.347 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:52:54.345+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:52:54.347 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:52:54.347 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:52:59.348 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:52:59.345+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:52:59.348 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:52:59.348 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:53:04.349 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:53:04.346+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:53:04.349 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:53:04.350 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:53:09.350 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:53:09.347+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:53:09.351 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:53:09.351 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:53:14.351 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:53:14.348+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:53:14.351 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:53:14.351 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:53:19.351 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:53:19.349+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:53:19.352 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:53:19.352 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:53:24.352 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:53:24.350+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:53:24.353 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:53:24.353 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:53:29.356 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:53:29.351+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:53:29.357 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:53:29.357 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:53:34.355 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:53:34.352+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:53:34.355 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:53:34.356 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:53:39.355 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:53:39.354+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:53:39.356 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:53:39.356 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:53:44.356 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:53:44.354+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:53:44.356 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:53:44.357 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:53:49.357 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:53:49.354+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:53:49.357 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:53:49.357 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:53:54.360 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:53:54.355+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:53:54.361 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:53:54.362 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:53:59.359 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:53:59.356+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:53:59.360 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:53:59.360 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:54:04.360 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:54:04.357+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:54:04.361 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:54:04.361 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:54:09.361 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:54:09.358+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:54:09.362 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:54:09.362 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:54:14.366 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:54:14.359+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:54:14.366 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:54:14.367 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:54:19.362 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:54:19.360+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:54:19.365 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:54:19.366 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:54:24.363 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:54:24.362+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:54:24.364 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:54:24.364 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:54:29.364 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:54:29.363+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:54:29.365 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:54:29.366 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:54:34.365 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:54:34.363+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:54:34.366 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:54:34.366 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:54:39.366 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:54:39.364+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:54:39.367 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:54:39.367 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:54:44.368 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:54:44.365+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:54:44.369 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:54:44.369 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:54:49.368 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:54:49.366+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:54:49.369 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:54:49.370 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:54:54.369 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:54:54.367+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:54:54.369 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:54:54.370 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:54:59.370 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:54:59.368+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:54:59.371 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:54:59.371 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:55:04.371 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:55:04.369+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:55:04.371 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:55:04.372 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:55:09.372 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:55:09.370+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:55:09.373 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:55:09.373 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:55:14.373 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:55:14.371+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:55:14.374 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:55:14.374 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:55:19.374 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:55:19.371+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:55:19.374 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:55:19.374 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:55:24.375 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:55:24.372+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:55:24.376 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:55:24.377 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:55:29.376 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:55:29.373+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:55:29.377 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:55:29.377 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:55:34.377 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:55:34.374+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:55:34.378 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:55:34.378 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:55:39.378 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:55:39.375+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:55:39.378 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:55:39.379 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:55:44.378 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:55:44.376+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:55:44.379 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:55:44.379 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:55:49.379 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:55:49.377+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:55:49.380 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:55:49.381 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:55:54.380 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:55:54.379+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:55:54.381 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:55:54.382 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:55:59.382 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:55:59.380+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:55:59.383 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:55:59.383 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:56:04.383 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:56:04.380+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:56:04.383 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:56:04.384 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:56:09.383 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:56:09.381+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:56:09.384 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:56:09.384 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:56:14.384 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:56:14.382+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:56:14.385 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:56:14.386 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:56:19.385 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:56:19.383+0000 7f67a90d4600 1 waiting for incremental sync to catch up: 2022-09-12T19:56:19.386 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:56:19.386 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000003.65.5, 00000000002.40.5, 00000000002.65.5] 2022-09-12T19:56:24.386 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:56:24.384+0000 7f67a90d4600 0 bucket checkpoint timed out waiting for incremental sync to catch up 2022-09-12T19:56:24.387 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:56:24.384+0000 7f67a90d4600 0 bucket sync checkpoint failed: (110) Connection timed out 2022-09-12T19:56:24.387 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:56:24.384+0000 7f67a90d4600 -1 bucket sync checkpoint failed: (110) Connection timed out 2022-09-12T19:56:24.403 DEBUG:teuthology.orchestra.run:got remote process result: 110 2022-09-12T19:56:24.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... FAIL 2022-09-12T19:56:24.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:56:24.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:56:24.413 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:56:24.470 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:56:24.470 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:56:24.503 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=38ca1f57-1358-496c-a99b-df49ee365a4f 2022-09-12T19:56:24.517 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:56:24.518 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:56:24.518 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:56:24.519 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:56:24.519 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:56:24.521 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:56:24.521 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:56:24.522 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:56:24.522 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:56:24.523 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:56:24.523 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663012111.609883_706.1", 2022-09-12T19:56:24.524 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:48:31.609883Z" 2022-09-12T19:56:24.524 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:56:24.525 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:56:24.525 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:56:24.526 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:56:24.526 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:56:24.526 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:56:24.528 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1663012111.609883_706.1', 3: ''} 2022-09-12T19:56:24.528 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T19:56:24.529 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:56:24.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:56:24.530 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:56:24.589 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:56:24.590 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:56:24.644 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:56:24.642+0000 7fc5c9c3e600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:56:24.645 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:56:24.643+0000 7fc5c9c3e600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:56:24.668 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:56:24.669 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:56:24.669 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:56:24.669 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:56:24.669 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:56:24.670 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:56:24.670 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:56:24.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:24.671 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:56:24.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:56:24.671 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:56:24.671 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:56:24.672 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:56:24.672 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:56:24.672 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:56:24.672 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:56:24.673 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:56:24.673 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:56:24.673 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:56:24.674 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:56:24.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:24.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:56:24.675 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:56:24.676 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:56:24.676 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:56:24.676 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:56:24.676 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:56:24.677 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:56:24.677 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:56:24.678 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:56:24.678 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:56:24.678 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:56:24.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:24.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:56:24.679 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:56:24.680 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:56:24.680 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:56:24.680 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663012111.609883_706.1", 2022-09-12T19:56:24.681 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:56:24.681 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:56:24.682 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:56:24.682 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:48:31.609883Z", 2022-09-12T19:56:24.683 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:56:24.683 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:56:24.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:24.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:56:24.685 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:56:24.685 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:56:24.686 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:56:24.686 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:56:24.687 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:56:24.687 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:56:24.688 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:56:24.688 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:56:24.688 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:56:24.689 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:56:24.689 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:56:24.690 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:56:24.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:24.691 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:56:24.691 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:56:24.692 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:56:24.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:24.692 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:56:24Z" 2022-09-12T19:56:24.693 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:56:24.695 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663012111.609883_706.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:48:31.609883Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:56:24Z" } 2022-09-12T19:56:24.697 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663012111.609883_706.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:48:31.609883Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:56:24.697 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663012111.609883_706.1', 3: ''} 2022-09-12T19:56:24.698 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663012111.609883_706.1', 3: ''} 2022-09-12T19:56:24.699 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T19:56:24.700 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T19:56:24.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:56:24.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:56:24.702 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:56:24.757 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:56:24.758 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:56:24.785 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:56:24.785+0000 7f9fd9cfb600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:56:24.786 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:56:24.786+0000 7f9fd9cfb600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:56:24.799 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:56:24.800 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:56:24.800 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:56:24.801 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:56:24.801 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:56:24.802 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:56:24.803 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:56:24.803 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:56:24.804 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:56:24.804 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:56:24.805 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:56:24.805 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:56:24.805 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:56:24.806 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:56:24.806 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:56:24.807 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:56:24.807 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:56:24.808 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:56:24.808 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:56:24.809 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:56:24.809 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:56:24.810 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:56:24.810 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:56:24.810 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:56:24.811 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:56:24.811 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:56:24.811 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:56:24.811 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:56:24.812 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:56:24.812 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:56:24.812 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:56:24.813 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:56:24.813 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:56:24.813 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:56:24.813 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:56:24.814 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:56:24.814 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:56:24.814 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663012111.609883_706.1", 2022-09-12T19:56:24.815 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:56:24.815 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:56:24.815 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:56:24.815 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:48:31.609883Z", 2022-09-12T19:56:24.816 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:56:24.816 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:56:24.816 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:56:24.816 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:56:24.817 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:56:24.817 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:56:24.817 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:56:24.818 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:56:24.818 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:56:24.818 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:56:24.818 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:56:24.819 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:56:24.819 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:56:24.819 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:56:24.819 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:56:24.820 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:56:24.820 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:56:24.820 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:56:24.821 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:56:24.821 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:56:24.821 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:56:24.822 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:56:24Z" 2022-09-12T19:56:24.822 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:56:24.823 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663012111.609883_706.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:48:31.609883Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:56:24Z" } 2022-09-12T19:56:24.824 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663012111.609883_706.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:48:31.609883Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:56:24.825 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663012111.609883_706.1', 3: ''} 2022-09-12T19:56:24.826 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663012111.609883_706.1', 3: ''} 2022-09-12T19:56:24.826 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T19:56:24.827 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T19:56:24.827 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:56:24.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:56:24.828 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:56:24.879 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:56:24.879 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:56:24.916 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:56:24.914+0000 7f39cc177600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:56:24.917 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:56:24.915+0000 7f39cc177600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:56:24.929 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:56:24.930 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:56:24.930 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:56:24.931 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:56:24.931 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:56:24.932 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:56:24.932 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:56:24.933 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:24.933 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:56:24.934 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:56:24.934 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:56:24.935 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:56:24.935 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:56:24.935 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:56:24.936 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:56:24.936 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:56:24.936 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:56:24.936 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:56:24.937 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:56:24.937 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:56:24.937 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:24.937 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:56:24.938 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:56:24.938 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:56:24.938 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:56:24.939 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:56:24.939 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:56:24.939 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:56:24.939 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:56:24.939 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:56:24.940 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:56:24.940 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:56:24.940 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:24.941 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:56:24.941 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:56:24.941 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:56:24.941 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:56:24.942 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663012111.609883_706.1", 2022-09-12T19:56:24.942 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:56:24.942 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:56:24.942 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:56:24.943 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:48:31.609883Z", 2022-09-12T19:56:24.943 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:56:24.943 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:56:24.943 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:24.944 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:56:24.944 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:56:24.944 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:56:24.944 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:56:24.945 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:56:24.945 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:56:24.945 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:56:24.946 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:56:24.946 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:56:24.947 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:56:24.947 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:56:24.948 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:56:24.949 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:56:24.949 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:24.950 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:56:24.950 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:56:24.951 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:56:24.951 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:24.952 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:56:24Z" 2022-09-12T19:56:24.952 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:56:24.954 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663012111.609883_706.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:48:31.609883Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:56:24Z" } 2022-09-12T19:56:24.955 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663012111.609883_706.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:48:31.609883Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:56:24.956 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663012111.609883_706.1', 3: ''} 2022-09-12T19:56:24.957 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663012111.609883_706.1', 3: ''} 2022-09-12T19:56:24.957 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T19:56:24.959 DEBUG:boto:path=/ 2022-09-12T19:56:24.960 DEBUG:boto:auth_path=/ 2022-09-12T19:56:24.961 DEBUG:boto:Method: GET 2022-09-12T19:56:24.962 DEBUG:boto:Path: / 2022-09-12T19:56:24.963 DEBUG:boto:Data: 2022-09-12T19:56:24.964 DEBUG:boto:Headers: {} 2022-09-12T19:56:24.964 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:56:24.965 DEBUG:boto:Port: 8000 2022-09-12T19:56:24.966 DEBUG:boto:Params: {} 2022-09-12T19:56:24.968 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:56:24.969 DEBUG:boto:Token: None 2022-09-12T19:56:24.971 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:56:24 GMT / 2022-09-12T19:56:24.972 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:afKIBT5fZF3mQaUBCqql7oCym3o= 2022-09-12T19:56:24.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:56:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:afKIBT5fZF3mQaUBCqql7oCym3o=', 'Content-Length': '0'} 2022-09-12T19:56:24.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066968eabb288b18b-00631f8ee8-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:56:24.985 DEBUG:boto:path=/ 2022-09-12T19:56:24.986 DEBUG:boto:auth_path=/ 2022-09-12T19:56:24.987 DEBUG:boto:Method: GET 2022-09-12T19:56:24.987 DEBUG:boto:Path: / 2022-09-12T19:56:24.988 DEBUG:boto:Data: 2022-09-12T19:56:24.988 DEBUG:boto:Headers: {} 2022-09-12T19:56:24.989 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T19:56:24.989 DEBUG:boto:Port: 8000 2022-09-12T19:56:24.990 DEBUG:boto:Params: {} 2022-09-12T19:56:24.992 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T19:56:24.993 DEBUG:boto:Token: None 2022-09-12T19:56:24.993 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:56:24 GMT / 2022-09-12T19:56:24.994 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:afKIBT5fZF3mQaUBCqql7oCym3o= 2022-09-12T19:56:24.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:56:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:afKIBT5fZF3mQaUBCqql7oCym3o=', 'Content-Length': '0'} 2022-09-12T19:56:25.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ba56f6505d43662-00631f8ee8-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:56:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:56:25.002 DEBUG:boto:path=/ 2022-09-12T19:56:25.003 DEBUG:boto:auth_path=/ 2022-09-12T19:56:25.004 DEBUG:boto:Method: GET 2022-09-12T19:56:25.004 DEBUG:boto:Path: / 2022-09-12T19:56:25.005 DEBUG:boto:Data: 2022-09-12T19:56:25.005 DEBUG:boto:Headers: {} 2022-09-12T19:56:25.006 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T19:56:25.006 DEBUG:boto:Port: 8001 2022-09-12T19:56:25.007 DEBUG:boto:Params: {} 2022-09-12T19:56:25.007 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T19:56:25.008 DEBUG:boto:Token: None 2022-09-12T19:56:25.008 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 19:56:25 GMT / 2022-09-12T19:56:25.009 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8P4Yi6WJ8nmS5zkFROk2gfh5uKc= 2022-09-12T19:56:25.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:56:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8P4Yi6WJ8nmS5zkFROk2gfh5uKc=', 'Content-Length': '0'} 2022-09-12T19:56:25.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003358a576b44c715d-00631f8ee9-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:56:25.018 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... ok 2022-09-12T19:56:25.018 DEBUG:boto:path=/jdpiak-74/ 2022-09-12T19:56:25.019 DEBUG:boto:auth_path=/jdpiak-74/ 2022-09-12T19:56:25.020 DEBUG:boto:Method: PUT 2022-09-12T19:56:25.020 DEBUG:boto:Path: /jdpiak-74/ 2022-09-12T19:56:25.021 DEBUG:boto:Data: 2022-09-12T19:56:25.021 DEBUG:boto:Headers: {} 2022-09-12T19:56:25.022 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:56:25.023 DEBUG:boto:Port: 8000 2022-09-12T19:56:25.023 DEBUG:boto:Params: {} 2022-09-12T19:56:25.024 DEBUG:boto:Token: None 2022-09-12T19:56:25.024 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 19:56:25 GMT /jdpiak-74/ 2022-09-12T19:56:25.025 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:i1NGAJp/P39ZQ4ifFbyZmqM69PE= 2022-09-12T19:56:25.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:56:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:i1NGAJp/P39ZQ4ifFbyZmqM69PE=', 'Content-Length': '0'} 2022-09-12T19:56:25.044 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a9471d0f526fce28-00631f8ee9-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:56:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:56:25.045 DEBUG:rgw_multi.tests:created bucket=jdpiak-74 2022-09-12T19:56:25.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:56:25.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:56:25.046 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:56:25.099 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:56:25.099 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:56:25.141 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=38ca1f57-1358-496c-a99b-df49ee365a4f 2022-09-12T19:56:25.158 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T19:56:25.159 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:56:25.159 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:56:25.159 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:56:25.160 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:56:25.160 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:56:25.160 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:56:25.160 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:56:25.161 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:56:25.161 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:56:25.161 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663012585.041391_737.1", 2022-09-12T19:56:25.161 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T19:56:25.041391Z" 2022-09-12T19:56:25.162 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:56:25.162 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:56:25.162 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:56:25.162 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T19:56:25.163 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:56:25.163 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T19:56:25.164 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1663012585.041391_737.1', 3: ''} 2022-09-12T19:56:25.165 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T19:56:25.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:56:25.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:56:25.167 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:56:25.217 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:56:25.217 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:56:25.266 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:56:25.263+0000 7fc2d7e9a600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:56:25.266 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:56:25.264+0000 7fc2d7e9a600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:56:25.280 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:56:25.280 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:56:25.281 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:56:25.281 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:56:25.281 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:56:25.281 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:56:25.282 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:56:25.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:25.282 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:56:25.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:56:25.283 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:56:25.283 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:56:25.284 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:56:25.284 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:56:25.284 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:56:25.284 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:56:25.285 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:56:25.285 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:56:25.285 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:56:25.285 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:56:25.286 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:25.287 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:56:25.287 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:56:25.290 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:56:25.290 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:56:25.291 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:56:25.291 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:56:25.291 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:56:25.291 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:56:25.292 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:56:25.292 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:56:25.292 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:56:25.293 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:25.293 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:56:25.294 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:56:25.294 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:56:25.295 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:56:25.295 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663012585.033555_734.1", 2022-09-12T19:56:25.296 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:56:25.296 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:56:25.297 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:56:25.297 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:56:25.033555Z", 2022-09-12T19:56:25.298 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:56:25.299 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:56:25.299 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:25.300 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:56:25.301 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:56:25.301 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:56:25.302 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:56:25.302 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:56:25.303 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:56:25.304 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:56:25.305 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:56:25.305 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:56:25.306 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:56:25.306 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:56:25.307 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:56:25.307 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:56:25.308 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:25.308 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:56:25.309 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:56:25.309 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:56:25.310 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:25.310 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:56:25Z" 2022-09-12T19:56:25.310 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:56:25.313 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663012585.033555_734.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:56:25.033555Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:56:25Z" } 2022-09-12T19:56:25.314 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663012585.033555_734.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:56:25.033555Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:56:25.315 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663012585.041391_737.1', 3: ''} 2022-09-12T19:56:25.316 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663012585.033555_734.1', 3: ''} 2022-09-12T19:56:25.316 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663012585.041391_737.1 target=1_1663012585.033555_734.1 2022-09-12T19:56:30.318 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:56:30.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:56:30.320 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:56:30.378 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:56:30.379 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:56:30.417 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:56:30.415+0000 7fc0432a9600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:56:30.418 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:56:30.416+0000 7fc0432a9600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:56:30.436 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:56:30.437 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:56:30.437 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:56:30.438 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:56:30.438 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:56:30.438 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:56:30.439 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:56:30.439 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:30.440 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:56:30.440 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:56:30.440 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:56:30.441 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:56:30.441 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:56:30.442 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:56:30.442 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:56:30.442 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:56:30.443 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:56:30.443 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:56:30.443 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:56:30.444 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:56:30.444 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:30.444 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:56:30.444 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:56:30.445 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:56:30.445 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:56:30.445 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:56:30.445 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:56:30.446 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:56:30.447 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:56:30.447 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:56:30.447 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:56:30.447 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:56:30.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:30.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:56:30.449 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:56:30.449 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:56:30.449 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:56:30.450 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663012585.041391_737.1", 2022-09-12T19:56:30.450 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:56:30.451 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:56:30.451 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:56:30.452 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:56:25.041391Z", 2022-09-12T19:56:30.452 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:56:30.452 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:56:30.453 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:30.454 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:56:30.454 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:56:30.454 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:56:30.455 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:56:30.455 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:56:30.455 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:56:30.455 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:56:30.456 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:56:30.456 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:56:30.456 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:56:30.456 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:56:30.457 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:56:30.457 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:56:30.457 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:30.457 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:56:30.458 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:56:30.458 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:56:30.459 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:30.459 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:56:30Z" 2022-09-12T19:56:30.459 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:56:30.461 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663012585.041391_737.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:56:25.041391Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:56:30Z" } 2022-09-12T19:56:30.463 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663012585.041391_737.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:56:25.041391Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:56:30.464 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663012585.041391_737.1', 3: ''} 2022-09-12T19:56:30.464 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663012585.041391_737.1', 3: ''} 2022-09-12T19:56:30.465 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T19:56:30.466 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T19:56:30.467 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:56:30.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:56:30.469 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:56:30.529 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:56:30.530 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:56:30.563 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:56:30.562+0000 7f1d63c66600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:56:30.564 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:56:30.564+0000 7f1d63c66600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:56:30.579 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:56:30.579 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T19:56:30.580 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T19:56:30.580 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T19:56:30.580 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T19:56:30.581 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:56:30.581 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:56:30.581 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:56:30.582 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T19:56:30.582 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:56:30.583 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T19:56:30.583 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:56:30.583 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:56:30.583 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:56:30.584 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:56:30.584 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T19:56:30.585 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:56:30.585 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:56:30.586 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:56:30.586 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:56:30.587 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:56:30.587 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:56:30.588 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T19:56:30.588 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:56:30.589 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:56:30.589 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T19:56:30.590 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:56:30.590 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:56:30.591 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:56:30.591 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T19:56:30.592 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:56:30.593 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:56:30.593 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:56:30.593 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:56:30.594 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T19:56:30.594 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:56:30.594 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:56:30.595 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663012585.041391_737.1", 2022-09-12T19:56:30.595 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:56:30.596 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:56:30.596 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:56:30.596 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T19:56:25.041391Z", 2022-09-12T19:56:30.596 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T19:56:30.597 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:56:30.597 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:56:30.597 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T19:56:30.597 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T19:56:30.598 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T19:56:30.598 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T19:56:30.599 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:56:30.599 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T19:56:30.599 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T19:56:30.599 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T19:56:30.600 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:56:30.600 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T19:56:30.600 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:56:30.601 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T19:56:30.601 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:56:30.601 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:56:30.601 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T19:56:30.602 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T19:56:30.602 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T19:56:30.602 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T19:56:30.603 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T19:56:30Z" 2022-09-12T19:56:30.603 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:56:30.605 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663012585.041391_737.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:56:25.041391Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:56:30Z" } 2022-09-12T19:56:30.606 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663012585.041391_737.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:56:25.041391Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:56:30.606 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663012585.041391_737.1', 3: ''} 2022-09-12T19:56:30.607 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663012585.041391_737.1', 3: ''} 2022-09-12T19:56:30.607 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T19:56:30.608 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T19:56:30.609 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:56:30.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:56:30.610 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:56:30.662 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:56:30.663 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:56:30.695 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:56:30.693+0000 7f5b3ff25600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:56:30.696 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:56:30.694+0000 7f5b3ff25600 20 RGW-SYNC:meta: read sync status 2022-09-12T19:56:30.708 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T19:56:30.709 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T19:56:30.709 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T19:56:30.709 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T19:56:30.710 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T19:56:30.710 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T19:56:30.711 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:56:30.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:30.712 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T19:56:30.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:56:30.713 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T19:56:30.713 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:56:30.714 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:56:30.717 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T19:56:30.717 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:56:30.718 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T19:56:30.718 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:56:30.719 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T19:56:30.719 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:56:30.720 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:56:30.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:30.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:56:30.721 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T19:56:30.721 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:56:30.722 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:56:30.722 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T19:56:30.723 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:56:30.723 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:56:30.724 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:56:30.724 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T19:56:30.725 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:56:30.725 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:56:30.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:30.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:56:30.727 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T19:56:30.727 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:56:30.727 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:56:30.728 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663012585.041391_737.1", 2022-09-12T19:56:30.728 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:56:30.728 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:56:30.729 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:56:30.729 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T19:56:25.041391Z", 2022-09-12T19:56:30.729 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T19:56:30.730 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:56:30.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:30.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T19:56:30.731 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T19:56:30.731 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T19:56:30.731 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T19:56:30.732 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T19:56:30.732 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T19:56:30.732 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T19:56:30.733 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T19:56:30.733 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T19:56:30.733 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T19:56:30.733 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:56:30.734 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T19:56:30.734 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T19:56:30.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:30.734 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T19:56:30.735 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T19:56:30.735 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T19:56:30.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T19:56:30.736 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T19:56:30Z" 2022-09-12T19:56:30.736 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T19:56:30.739 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663012585.041391_737.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T19:56:25.041391Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T19:56:30Z" } 2022-09-12T19:56:30.740 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663012585.041391_737.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T19:56:25.041391Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T19:56:30.741 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663012585.041391_737.1', 3: ''} 2022-09-12T19:56:30.741 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663012585.041391_737.1', 3: ''} 2022-09-12T19:56:30.742 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T19:56:30.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:56:30.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T19:56:30.744 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --num-shards 11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T19:56:30.803 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:56:30.803 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:56:30.846 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T19:56:30.847 INFO:teuthology.orchestra.run.smithi035.stdout: "objs": [ 2022-09-12T19:56:30.847 INFO:teuthology.orchestra.run.smithi035.stdout: "obj00000000000000000001", 2022-09-12T19:56:30.847 INFO:teuthology.orchestra.run.smithi035.stdout: "obj00000000000000000005", 2022-09-12T19:56:30.847 INFO:teuthology.orchestra.run.smithi035.stdout: "obj00000000000000000012", 2022-09-12T19:56:30.848 INFO:teuthology.orchestra.run.smithi035.stdout: "obj00000000000000000004", 2022-09-12T19:56:30.848 INFO:teuthology.orchestra.run.smithi035.stdout: "obj00000000000000000011", 2022-09-12T19:56:30.848 INFO:teuthology.orchestra.run.smithi035.stdout: "obj00000000000000000000", 2022-09-12T19:56:30.848 INFO:teuthology.orchestra.run.smithi035.stdout: "obj00000000000000000010", 2022-09-12T19:56:30.849 INFO:teuthology.orchestra.run.smithi035.stdout: "obj00000000000000000003", 2022-09-12T19:56:30.849 INFO:teuthology.orchestra.run.smithi035.stdout: "obj00000000000000000013", 2022-09-12T19:56:30.849 INFO:teuthology.orchestra.run.smithi035.stdout: "obj00000000000000000002", 2022-09-12T19:56:30.849 INFO:teuthology.orchestra.run.smithi035.stdout: "obj00000000000000000006" 2022-09-12T19:56:30.850 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T19:56:30.850 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T19:56:30.852 DEBUG:boto:path=/jdpiak-74/ 2022-09-12T19:56:30.852 DEBUG:boto:auth_path=/jdpiak-74/ 2022-09-12T19:56:30.853 DEBUG:boto:Method: HEAD 2022-09-12T19:56:30.854 DEBUG:boto:Path: /jdpiak-74/ 2022-09-12T19:56:30.854 DEBUG:boto:Data: 2022-09-12T19:56:30.855 DEBUG:boto:Headers: {} 2022-09-12T19:56:30.855 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:56:30.856 DEBUG:boto:Port: 8000 2022-09-12T19:56:30.856 DEBUG:boto:Params: {} 2022-09-12T19:56:30.857 DEBUG:boto:Token: None 2022-09-12T19:56:30.858 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:56:30 GMT /jdpiak-74/ 2022-09-12T19:56:30.859 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TolvXpGo+rttsaw0WBQAYVufwNQ= 2022-09-12T19:56:30.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:56:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TolvXpGo+rttsaw0WBQAYVufwNQ=', 'Content-Length': '0'} 2022-09-12T19:56:30.862 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000baf5011b6a663168-00631f8eee-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:56:30.863 DEBUG:boto:path=/jdpiak-74/obj00000000000000000001 2022-09-12T19:56:30.865 DEBUG:boto:auth_path=/jdpiak-74/obj00000000000000000001 2022-09-12T19:56:30.865 DEBUG:boto:Method: PUT 2022-09-12T19:56:30.866 DEBUG:boto:Path: /jdpiak-74/obj00000000000000000001 2022-09-12T19:56:30.866 DEBUG:boto:Data: 2022-09-12T19:56:30.867 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:56:30.867 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:56:30.868 DEBUG:boto:Port: 8000 2022-09-12T19:56:30.868 DEBUG:boto:Params: {} 2022-09-12T19:56:30.869 DEBUG:boto:Token: None 2022-09-12T19:56:30.869 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:56:30 GMT /jdpiak-74/obj00000000000000000001 2022-09-12T19:56:30.870 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hGT1lGyNEPcIXYHfMoEbJlnP9VA= 2022-09-12T19:56:30.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:56:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hGT1lGyNEPcIXYHfMoEbJlnP9VA='} 2022-09-12T19:56:30.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c25336adc10bee77-00631f8eee-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:56:30.879 DEBUG:boto:path=/jdpiak-74/ 2022-09-12T19:56:30.879 DEBUG:boto:auth_path=/jdpiak-74/ 2022-09-12T19:56:30.880 DEBUG:boto:Method: HEAD 2022-09-12T19:56:30.880 DEBUG:boto:Path: /jdpiak-74/ 2022-09-12T19:56:30.881 DEBUG:boto:Data: 2022-09-12T19:56:30.881 DEBUG:boto:Headers: {} 2022-09-12T19:56:30.882 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:56:30.882 DEBUG:boto:Port: 8000 2022-09-12T19:56:30.883 DEBUG:boto:Params: {} 2022-09-12T19:56:30.883 DEBUG:boto:Token: None 2022-09-12T19:56:30.884 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:56:30 GMT /jdpiak-74/ 2022-09-12T19:56:30.885 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TolvXpGo+rttsaw0WBQAYVufwNQ= 2022-09-12T19:56:30.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:56:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TolvXpGo+rttsaw0WBQAYVufwNQ=', 'Content-Length': '0'} 2022-09-12T19:56:30.889 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc28bccdfc676fdc-00631f8eee-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:56:30.890 DEBUG:boto:path=/jdpiak-74/obj00000000000000000006 2022-09-12T19:56:30.890 DEBUG:boto:auth_path=/jdpiak-74/obj00000000000000000006 2022-09-12T19:56:30.891 DEBUG:boto:Method: PUT 2022-09-12T19:56:30.891 DEBUG:boto:Path: /jdpiak-74/obj00000000000000000006 2022-09-12T19:56:30.892 DEBUG:boto:Data: 2022-09-12T19:56:30.892 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:56:30.893 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:56:30.893 DEBUG:boto:Port: 8000 2022-09-12T19:56:30.894 DEBUG:boto:Params: {} 2022-09-12T19:56:30.895 DEBUG:boto:Token: None 2022-09-12T19:56:30.895 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:56:30 GMT /jdpiak-74/obj00000000000000000006 2022-09-12T19:56:30.896 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QyA0RwiIFsvrZCzJw5dx36kmM/c= 2022-09-12T19:56:30.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:56:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QyA0RwiIFsvrZCzJw5dx36kmM/c='} 2022-09-12T19:56:30.904 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005eb877fdc6bb112d-00631f8eee-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:56:30.905 DEBUG:boto:path=/jdpiak-74/ 2022-09-12T19:56:30.906 DEBUG:boto:auth_path=/jdpiak-74/ 2022-09-12T19:56:30.907 DEBUG:boto:Method: HEAD 2022-09-12T19:56:30.908 DEBUG:boto:Path: /jdpiak-74/ 2022-09-12T19:56:30.909 DEBUG:boto:Data: 2022-09-12T19:56:30.909 DEBUG:boto:Headers: {} 2022-09-12T19:56:30.910 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:56:30.910 DEBUG:boto:Port: 8000 2022-09-12T19:56:30.911 DEBUG:boto:Params: {} 2022-09-12T19:56:30.911 DEBUG:boto:Token: None 2022-09-12T19:56:30.912 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:56:30 GMT /jdpiak-74/ 2022-09-12T19:56:30.912 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TolvXpGo+rttsaw0WBQAYVufwNQ= 2022-09-12T19:56:30.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:56:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TolvXpGo+rttsaw0WBQAYVufwNQ=', 'Content-Length': '0'} 2022-09-12T19:56:30.916 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca555236a5bf0748-00631f8eee-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:56:30.917 DEBUG:boto:path=/jdpiak-74/obj00000000000000000000 2022-09-12T19:56:30.917 DEBUG:boto:auth_path=/jdpiak-74/obj00000000000000000000 2022-09-12T19:56:30.918 DEBUG:boto:Method: PUT 2022-09-12T19:56:30.919 DEBUG:boto:Path: /jdpiak-74/obj00000000000000000000 2022-09-12T19:56:30.919 DEBUG:boto:Data: 2022-09-12T19:56:30.920 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:56:30.921 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:56:30.921 DEBUG:boto:Port: 8000 2022-09-12T19:56:30.922 DEBUG:boto:Params: {} 2022-09-12T19:56:30.923 DEBUG:boto:Token: None 2022-09-12T19:56:30.924 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:56:30 GMT /jdpiak-74/obj00000000000000000000 2022-09-12T19:56:30.925 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:giO/YAKM67JEw72lYnjJ0+NaoK0= 2022-09-12T19:56:30.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:56:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:giO/YAKM67JEw72lYnjJ0+NaoK0='} 2022-09-12T19:56:30.932 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000063d814f06f2e8c07-00631f8eee-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:56:30.933 DEBUG:boto:path=/jdpiak-74/ 2022-09-12T19:56:30.934 DEBUG:boto:auth_path=/jdpiak-74/ 2022-09-12T19:56:30.935 DEBUG:boto:Method: HEAD 2022-09-12T19:56:30.935 DEBUG:boto:Path: /jdpiak-74/ 2022-09-12T19:56:30.936 DEBUG:boto:Data: 2022-09-12T19:56:30.936 DEBUG:boto:Headers: {} 2022-09-12T19:56:30.937 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:56:30.938 DEBUG:boto:Port: 8000 2022-09-12T19:56:30.939 DEBUG:boto:Params: {} 2022-09-12T19:56:30.940 DEBUG:boto:Token: None 2022-09-12T19:56:30.940 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:56:30 GMT /jdpiak-74/ 2022-09-12T19:56:30.941 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TolvXpGo+rttsaw0WBQAYVufwNQ= 2022-09-12T19:56:30.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:56:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TolvXpGo+rttsaw0WBQAYVufwNQ=', 'Content-Length': '0'} 2022-09-12T19:56:30.945 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008fd2d82677908db2-00631f8eee-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:56:30.947 DEBUG:boto:path=/jdpiak-74/obj00000000000000000005 2022-09-12T19:56:30.948 DEBUG:boto:auth_path=/jdpiak-74/obj00000000000000000005 2022-09-12T19:56:30.948 DEBUG:boto:Method: PUT 2022-09-12T19:56:30.949 DEBUG:boto:Path: /jdpiak-74/obj00000000000000000005 2022-09-12T19:56:30.950 DEBUG:boto:Data: 2022-09-12T19:56:30.951 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:56:30.952 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:56:30.953 DEBUG:boto:Port: 8000 2022-09-12T19:56:30.953 DEBUG:boto:Params: {} 2022-09-12T19:56:30.954 DEBUG:boto:Token: None 2022-09-12T19:56:30.955 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:56:30 GMT /jdpiak-74/obj00000000000000000005 2022-09-12T19:56:30.955 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kzI995LFMS3irPOftm/vxRmfjIk= 2022-09-12T19:56:30.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:56:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kzI995LFMS3irPOftm/vxRmfjIk='} 2022-09-12T19:56:30.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bd7f0a84491da75d-00631f8eee-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:56:31.000 DEBUG:boto:path=/jdpiak-74/ 2022-09-12T19:56:31.001 DEBUG:boto:auth_path=/jdpiak-74/ 2022-09-12T19:56:31.002 DEBUG:boto:Method: HEAD 2022-09-12T19:56:31.002 DEBUG:boto:Path: /jdpiak-74/ 2022-09-12T19:56:31.004 DEBUG:boto:Data: 2022-09-12T19:56:31.005 DEBUG:boto:Headers: {} 2022-09-12T19:56:31.005 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:56:31.006 DEBUG:boto:Port: 8000 2022-09-12T19:56:31.007 DEBUG:boto:Params: {} 2022-09-12T19:56:31.008 DEBUG:boto:Token: None 2022-09-12T19:56:31.009 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:56:31 GMT /jdpiak-74/ 2022-09-12T19:56:31.010 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oO8bOmdwCR3gpjs9xep4LWSXodE= 2022-09-12T19:56:31.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:56:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oO8bOmdwCR3gpjs9xep4LWSXodE=', 'Content-Length': '0'} 2022-09-12T19:56:31.014 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea26b49dfda0d1fc-00631f8eef-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:56:31.016 DEBUG:boto:path=/jdpiak-74/obj00000000000000000013 2022-09-12T19:56:31.016 DEBUG:boto:auth_path=/jdpiak-74/obj00000000000000000013 2022-09-12T19:56:31.017 DEBUG:boto:Method: PUT 2022-09-12T19:56:31.018 DEBUG:boto:Path: /jdpiak-74/obj00000000000000000013 2022-09-12T19:56:31.019 DEBUG:boto:Data: 2022-09-12T19:56:31.020 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:56:31.020 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:56:31.021 DEBUG:boto:Port: 8000 2022-09-12T19:56:31.021 DEBUG:boto:Params: {} 2022-09-12T19:56:31.022 DEBUG:boto:Token: None 2022-09-12T19:56:31.023 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:56:31 GMT /jdpiak-74/obj00000000000000000013 2022-09-12T19:56:31.023 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:duvxlYE3MPrj6RLz+/YD3FAc+CU= 2022-09-12T19:56:31.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:56:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:duvxlYE3MPrj6RLz+/YD3FAc+CU='} 2022-09-12T19:56:31.031 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c326090b5fa296d6-00631f8eef-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:56:31.032 DEBUG:boto:path=/jdpiak-74/ 2022-09-12T19:56:31.033 DEBUG:boto:auth_path=/jdpiak-74/ 2022-09-12T19:56:31.034 DEBUG:boto:Method: HEAD 2022-09-12T19:56:31.034 DEBUG:boto:Path: /jdpiak-74/ 2022-09-12T19:56:31.036 DEBUG:boto:Data: 2022-09-12T19:56:31.037 DEBUG:boto:Headers: {} 2022-09-12T19:56:31.037 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:56:31.038 DEBUG:boto:Port: 8000 2022-09-12T19:56:31.040 DEBUG:boto:Params: {} 2022-09-12T19:56:31.041 DEBUG:boto:Token: None 2022-09-12T19:56:31.041 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:56:31 GMT /jdpiak-74/ 2022-09-12T19:56:31.042 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oO8bOmdwCR3gpjs9xep4LWSXodE= 2022-09-12T19:56:31.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:56:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oO8bOmdwCR3gpjs9xep4LWSXodE=', 'Content-Length': '0'} 2022-09-12T19:56:31.045 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004be24ac53badc060-00631f8eef-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:56:31.047 DEBUG:boto:path=/jdpiak-74/obj00000000000000000011 2022-09-12T19:56:31.048 DEBUG:boto:auth_path=/jdpiak-74/obj00000000000000000011 2022-09-12T19:56:31.049 DEBUG:boto:Method: PUT 2022-09-12T19:56:31.050 DEBUG:boto:Path: /jdpiak-74/obj00000000000000000011 2022-09-12T19:56:31.051 DEBUG:boto:Data: 2022-09-12T19:56:31.051 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:56:31.052 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:56:31.052 DEBUG:boto:Port: 8000 2022-09-12T19:56:31.053 DEBUG:boto:Params: {} 2022-09-12T19:56:31.053 DEBUG:boto:Token: None 2022-09-12T19:56:31.054 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:56:31 GMT /jdpiak-74/obj00000000000000000011 2022-09-12T19:56:31.054 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bMB7j/hcInP1w8oz6kCDkssbEeo= 2022-09-12T19:56:31.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:56:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bMB7j/hcInP1w8oz6kCDkssbEeo='} 2022-09-12T19:56:31.062 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eabf3bddb6ed6973-00631f8eef-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:56:31.063 DEBUG:boto:path=/jdpiak-74/ 2022-09-12T19:56:31.064 DEBUG:boto:auth_path=/jdpiak-74/ 2022-09-12T19:56:31.065 DEBUG:boto:Method: HEAD 2022-09-12T19:56:31.065 DEBUG:boto:Path: /jdpiak-74/ 2022-09-12T19:56:31.066 DEBUG:boto:Data: 2022-09-12T19:56:31.067 DEBUG:boto:Headers: {} 2022-09-12T19:56:31.068 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:56:31.069 DEBUG:boto:Port: 8000 2022-09-12T19:56:31.069 DEBUG:boto:Params: {} 2022-09-12T19:56:31.070 DEBUG:boto:Token: None 2022-09-12T19:56:31.071 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:56:31 GMT /jdpiak-74/ 2022-09-12T19:56:31.072 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oO8bOmdwCR3gpjs9xep4LWSXodE= 2022-09-12T19:56:31.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:56:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oO8bOmdwCR3gpjs9xep4LWSXodE=', 'Content-Length': '0'} 2022-09-12T19:56:31.076 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d53babb42c206ae3-00631f8eef-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:56:31.077 DEBUG:boto:path=/jdpiak-74/obj00000000000000000002 2022-09-12T19:56:31.078 DEBUG:boto:auth_path=/jdpiak-74/obj00000000000000000002 2022-09-12T19:56:31.079 DEBUG:boto:Method: PUT 2022-09-12T19:56:31.080 DEBUG:boto:Path: /jdpiak-74/obj00000000000000000002 2022-09-12T19:56:31.081 DEBUG:boto:Data: 2022-09-12T19:56:31.082 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:56:31.083 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:56:31.083 DEBUG:boto:Port: 8000 2022-09-12T19:56:31.084 DEBUG:boto:Params: {} 2022-09-12T19:56:31.085 DEBUG:boto:Token: None 2022-09-12T19:56:31.086 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:56:31 GMT /jdpiak-74/obj00000000000000000002 2022-09-12T19:56:31.087 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B7pII0JpCq7s+7jBZFdnkv0LBMY= 2022-09-12T19:56:31.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:56:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B7pII0JpCq7s+7jBZFdnkv0LBMY='} 2022-09-12T19:56:31.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000094dbd39e84d7ab8c-00631f8eef-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:56:31.136 DEBUG:boto:path=/jdpiak-74/ 2022-09-12T19:56:31.137 DEBUG:boto:auth_path=/jdpiak-74/ 2022-09-12T19:56:31.137 DEBUG:boto:Method: HEAD 2022-09-12T19:56:31.138 DEBUG:boto:Path: /jdpiak-74/ 2022-09-12T19:56:31.138 DEBUG:boto:Data: 2022-09-12T19:56:31.139 DEBUG:boto:Headers: {} 2022-09-12T19:56:31.139 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:56:31.140 DEBUG:boto:Port: 8000 2022-09-12T19:56:31.140 DEBUG:boto:Params: {} 2022-09-12T19:56:31.141 DEBUG:boto:Token: None 2022-09-12T19:56:31.141 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:56:31 GMT /jdpiak-74/ 2022-09-12T19:56:31.142 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oO8bOmdwCR3gpjs9xep4LWSXodE= 2022-09-12T19:56:31.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:56:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oO8bOmdwCR3gpjs9xep4LWSXodE=', 'Content-Length': '0'} 2022-09-12T19:56:31.145 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4c6c4e55839d5fe-00631f8eef-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:56:31.145 DEBUG:boto:path=/jdpiak-74/obj00000000000000000004 2022-09-12T19:56:31.146 DEBUG:boto:auth_path=/jdpiak-74/obj00000000000000000004 2022-09-12T19:56:31.147 DEBUG:boto:Method: PUT 2022-09-12T19:56:31.147 DEBUG:boto:Path: /jdpiak-74/obj00000000000000000004 2022-09-12T19:56:31.148 DEBUG:boto:Data: 2022-09-12T19:56:31.148 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:56:31.149 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:56:31.149 DEBUG:boto:Port: 8000 2022-09-12T19:56:31.149 DEBUG:boto:Params: {} 2022-09-12T19:56:31.150 DEBUG:boto:Token: None 2022-09-12T19:56:31.151 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:56:31 GMT /jdpiak-74/obj00000000000000000004 2022-09-12T19:56:31.151 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cZVd4WTlDWX99LXJDvEvpr42R7s= 2022-09-12T19:56:31.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:56:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cZVd4WTlDWX99LXJDvEvpr42R7s='} 2022-09-12T19:56:31.158 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a8b4b57c3d1da414-00631f8eef-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:56:31.159 DEBUG:boto:path=/jdpiak-74/ 2022-09-12T19:56:31.159 DEBUG:boto:auth_path=/jdpiak-74/ 2022-09-12T19:56:31.160 DEBUG:boto:Method: HEAD 2022-09-12T19:56:31.160 DEBUG:boto:Path: /jdpiak-74/ 2022-09-12T19:56:31.161 DEBUG:boto:Data: 2022-09-12T19:56:31.161 DEBUG:boto:Headers: {} 2022-09-12T19:56:31.162 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:56:31.162 DEBUG:boto:Port: 8000 2022-09-12T19:56:31.163 DEBUG:boto:Params: {} 2022-09-12T19:56:31.163 DEBUG:boto:Token: None 2022-09-12T19:56:31.164 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:56:31 GMT /jdpiak-74/ 2022-09-12T19:56:31.165 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oO8bOmdwCR3gpjs9xep4LWSXodE= 2022-09-12T19:56:31.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:56:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oO8bOmdwCR3gpjs9xep4LWSXodE=', 'Content-Length': '0'} 2022-09-12T19:56:31.176 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059fd1aeb87f1ad79-00631f8eef-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:56:31.177 DEBUG:boto:path=/jdpiak-74/obj00000000000000000003 2022-09-12T19:56:31.178 DEBUG:boto:auth_path=/jdpiak-74/obj00000000000000000003 2022-09-12T19:56:31.178 DEBUG:boto:Method: PUT 2022-09-12T19:56:31.179 DEBUG:boto:Path: /jdpiak-74/obj00000000000000000003 2022-09-12T19:56:31.179 DEBUG:boto:Data: 2022-09-12T19:56:31.180 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:56:31.180 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:56:31.181 DEBUG:boto:Port: 8000 2022-09-12T19:56:31.181 DEBUG:boto:Params: {} 2022-09-12T19:56:31.182 DEBUG:boto:Token: None 2022-09-12T19:56:31.183 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:56:31 GMT /jdpiak-74/obj00000000000000000003 2022-09-12T19:56:31.183 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5PlugwnGd2VBEV/dPnPQm5h8ECY= 2022-09-12T19:56:31.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:56:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5PlugwnGd2VBEV/dPnPQm5h8ECY='} 2022-09-12T19:56:31.190 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c9fb201864073040-00631f8eef-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:56:31.192 DEBUG:boto:path=/jdpiak-74/ 2022-09-12T19:56:31.192 DEBUG:boto:auth_path=/jdpiak-74/ 2022-09-12T19:56:31.193 DEBUG:boto:Method: HEAD 2022-09-12T19:56:31.193 DEBUG:boto:Path: /jdpiak-74/ 2022-09-12T19:56:31.194 DEBUG:boto:Data: 2022-09-12T19:56:31.194 DEBUG:boto:Headers: {} 2022-09-12T19:56:31.195 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:56:31.195 DEBUG:boto:Port: 8000 2022-09-12T19:56:31.196 DEBUG:boto:Params: {} 2022-09-12T19:56:31.196 DEBUG:boto:Token: None 2022-09-12T19:56:31.197 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 19:56:31 GMT /jdpiak-74/ 2022-09-12T19:56:31.198 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oO8bOmdwCR3gpjs9xep4LWSXodE= 2022-09-12T19:56:31.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 19:56:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oO8bOmdwCR3gpjs9xep4LWSXodE=', 'Content-Length': '0'} 2022-09-12T19:56:31.203 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035f3c246d4e13829-00631f8eef-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:56:31.204 DEBUG:boto:path=/jdpiak-74/obj00000000000000000012 2022-09-12T19:56:31.205 DEBUG:boto:auth_path=/jdpiak-74/obj00000000000000000012 2022-09-12T19:56:31.206 DEBUG:boto:Method: PUT 2022-09-12T19:56:31.207 DEBUG:boto:Path: /jdpiak-74/obj00000000000000000012 2022-09-12T19:56:31.208 DEBUG:boto:Data: 2022-09-12T19:56:31.209 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T19:56:31.210 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T19:56:31.211 DEBUG:boto:Port: 8000 2022-09-12T19:56:31.213 DEBUG:boto:Params: {} 2022-09-12T19:56:31.214 DEBUG:boto:Token: None 2022-09-12T19:56:31.215 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 19:56:31 GMT /jdpiak-74/obj00000000000000000012 2022-09-12T19:56:31.215 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bbfuhbk3q+zS66Ka+UGzXY5GWFY= 2022-09-12T19:56:31.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 19:56:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bbfuhbk3q+zS66Ka+UGzXY5GWFY='} 2022-09-12T19:56:31.263 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f11ab4b25c49ed9-00631f8eef-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T19:56:31.264 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-74 2022-09-12T19:56:31.264 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:56:31.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:56:31.268 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:56:31.326 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:56:31.327 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:56:31.360 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:56:31.358+0000 7fc510d79600 1 waiting to reach incremental sync.. 2022-09-12T19:56:36.361 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:56:36.358+0000 7fc510d79600 1 waiting to reach incremental sync.. 2022-09-12T19:56:41.361 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:56:41.360+0000 7fc510d79600 1 waiting to reach incremental sync.. 2022-09-12T19:56:46.362 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:56:46.360+0000 7fc510d79600 1 waiting to reach incremental sync.. 2022-09-12T19:56:51.363 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:56:51.360+0000 7fc510d79600 1 waiting to reach incremental sync.. 2022-09-12T19:56:56.363 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:56:56.361+0000 7fc510d79600 1 waiting to reach incremental sync.. 2022-09-12T19:57:01.364 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:57:01.362+0000 7fc510d79600 1 waiting to reach incremental sync.. 2022-09-12T19:57:06.365 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:57:06.363+0000 7fc510d79600 1 waiting to reach incremental sync.. 2022-09-12T19:57:11.366 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:57:11.363+0000 7fc510d79600 1 waiting to reach incremental sync.. 2022-09-12T19:57:16.367 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:57:16.364+0000 7fc510d79600 1 waiting to reach incremental sync.. 2022-09-12T19:57:21.370 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:57:21.365+0000 7fc510d79600 1 waiting to reach incremental sync.. 2022-09-12T19:57:26.371 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:57:26.368+0000 7fc510d79600 1 bucket sync caught up with source: 2022-09-12T19:57:26.372 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:57:26.372 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:57:26.373 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:57:26.368+0000 7fc510d79600 0 bucket checkpoint complete 2022-09-12T19:57:26.388 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-74 2022-09-12T19:57:26.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:57:26.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:57:26.391 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T19:57:26.458 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:57:26.458 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:57:26.499 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:57:26.498+0000 7f0a38d62600 1 waiting to reach incremental sync.. 2022-09-12T19:57:31.500 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:57:31.498+0000 7f0a38d62600 1 waiting to reach incremental sync.. 2022-09-12T19:57:36.501 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:57:36.499+0000 7f0a38d62600 1 waiting to reach incremental sync.. 2022-09-12T19:57:41.502 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:57:41.500+0000 7f0a38d62600 1 waiting to reach incremental sync.. 2022-09-12T19:57:46.502 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:57:46.500+0000 7f0a38d62600 1 waiting to reach incremental sync.. 2022-09-12T19:57:51.503 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:57:51.501+0000 7f0a38d62600 1 waiting to reach incremental sync.. 2022-09-12T19:57:56.503 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:57:56.502+0000 7f0a38d62600 1 waiting to reach incremental sync.. 2022-09-12T19:58:01.504 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:58:01.503+0000 7f0a38d62600 1 waiting to reach incremental sync.. 2022-09-12T19:58:06.506 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:58:06.505+0000 7f0a38d62600 1 bucket sync caught up with source: 2022-09-12T19:58:06.507 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:58:06.507 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:58:06.508 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T19:58:06.505+0000 7f0a38d62600 0 bucket checkpoint complete 2022-09-12T19:58:06.527 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-74 2022-09-12T19:58:06.528 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:58:06.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T19:58:06.530 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T19:58:06.596 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T19:58:06.596 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T19:58:06.655 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:58:06.653+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T19:58:06.656 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:58:06.656 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:58:11.657 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:58:11.655+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T19:58:11.657 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:58:11.658 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:58:16.657 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:58:16.655+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T19:58:16.658 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:58:16.659 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:58:21.659 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:58:21.656+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T19:58:21.660 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:58:21.660 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:58:26.660 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:58:26.658+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T19:58:26.661 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:58:26.661 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:58:31.662 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:58:31.659+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T19:58:31.662 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:58:31.663 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:58:36.664 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:58:36.661+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T19:58:36.664 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:58:36.665 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:58:41.665 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:58:41.662+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T19:58:41.666 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:58:41.666 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:58:46.666 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:58:46.664+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T19:58:46.667 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:58:46.667 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:58:51.668 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:58:51.665+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T19:58:51.668 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:58:51.669 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:58:56.669 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:58:56.668+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T19:58:56.670 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:58:56.670 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:59:01.671 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:59:01.669+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T19:59:01.671 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:59:01.672 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:59:06.672 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:59:06.670+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T19:59:06.673 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:59:06.673 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:59:11.673 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:59:11.671+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T19:59:11.674 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:59:11.674 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:59:16.675 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:59:16.673+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T19:59:16.676 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:59:16.676 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:59:21.678 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:59:21.675+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T19:59:21.678 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:59:21.678 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:59:26.679 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:59:26.677+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T19:59:26.680 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:59:26.680 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:59:31.681 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:59:31.679+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T19:59:31.682 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:59:31.682 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:59:36.683 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:59:36.680+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T19:59:36.683 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:59:36.683 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:59:41.684 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:59:41.682+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T19:59:41.684 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:59:41.685 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:59:46.684 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:59:46.683+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T19:59:46.685 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:59:46.685 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:59:51.686 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:59:51.683+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T19:59:51.687 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:59:51.687 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:59:56.688 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T19:59:56.685+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T19:59:56.688 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T19:59:56.689 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:00:01.689 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:00:01.686+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:00:01.689 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:00:01.689 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:00:06.690 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:00:06.688+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:00:06.691 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:00:06.692 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:00:11.692 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:00:11.689+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:00:11.693 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:00:11.693 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:00:16.693 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:00:16.691+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:00:16.694 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:00:16.694 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:00:21.695 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:00:21.692+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:00:21.695 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:00:21.695 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:00:26.696 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:00:26.695+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:00:26.697 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:00:26.697 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:00:31.697 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:00:31.696+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:00:31.698 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:00:31.698 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:00:36.699 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:00:36.697+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:00:36.700 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:00:36.701 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:00:41.700 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:00:41.698+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:00:41.701 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:00:41.701 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:00:46.702 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:00:46.699+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:00:46.703 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:00:46.703 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:00:51.703 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:00:51.701+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:00:51.704 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:00:51.704 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:00:56.705 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:00:56.702+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:00:56.706 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:00:56.706 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:01:01.706 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:01:01.704+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:01:01.707 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:01:01.707 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:01:06.708 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:01:06.706+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:01:06.709 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:01:06.709 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:01:11.709 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:01:11.708+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:01:11.710 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:01:11.710 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:01:16.712 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:01:16.709+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:01:16.713 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:01:16.714 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:01:21.712 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:01:21.710+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:01:21.713 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:01:21.713 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:01:26.714 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:01:26.711+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:01:26.714 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:01:26.714 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:01:31.715 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:01:31.713+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:01:31.715 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:01:31.716 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:01:36.716 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:01:36.714+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:01:36.717 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:01:36.717 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:01:41.718 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:01:41.716+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:01:41.719 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:01:41.719 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:01:46.719 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:01:46.717+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:01:46.720 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:01:46.720 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:01:51.721 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:01:51.719+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:01:51.722 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:01:51.722 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:01:56.722 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:01:56.721+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:01:56.723 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:01:56.723 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:02:01.724 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:02:01.723+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:02:01.725 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:02:01.725 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:02:06.726 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:02:06.723+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:02:06.727 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:02:06.727 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:02:11.727 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:02:11.725+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:02:11.728 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:02:11.728 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:02:16.729 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:02:16.726+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:02:16.729 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:02:16.729 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:02:21.730 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:02:21.728+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:02:21.731 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:02:21.731 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:02:26.732 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:02:26.729+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:02:26.733 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:02:26.733 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:02:31.733 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:02:31.731+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:02:31.734 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:02:31.734 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:02:36.735 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:02:36.732+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:02:36.735 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:02:36.735 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:02:41.736 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:02:41.734+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:02:41.736 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:02:41.737 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:02:46.737 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:02:46.736+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:02:46.738 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:02:46.738 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:02:51.739 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:02:51.736+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:02:51.739 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:02:51.740 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:02:56.740 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:02:56.738+0000 7f2fcae7b600 1 waiting for incremental sync to catch up: 2022-09-12T20:02:56.741 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:02:56.741 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.72.5, 00000000001.68.5, 00000000001.90.5, 00000000001.57.5, 00000000001.59.5, 00000000001.59.5, , 00000000001.32.5, 00000000001.57.5, 00000000001.44.5, 00000000001.54.5] 2022-09-12T20:03:01.741 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:03:01.739+0000 7f2fcae7b600 0 bucket checkpoint timed out waiting for incremental sync to catch up 2022-09-12T20:03:01.742 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:03:01.739+0000 7f2fcae7b600 0 bucket sync checkpoint failed: (110) Connection timed out 2022-09-12T20:03:01.742 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:03:01.739+0000 7f2fcae7b600 -1 bucket sync checkpoint failed: (110) Connection timed out 2022-09-12T20:03:01.763 DEBUG:teuthology.orchestra.run:got remote process result: 110 2022-09-12T20:03:01.768 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... FAIL 2022-09-12T20:03:01.770 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:01.771 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:01.771 DEBUG:boto:Method: PUT 2022-09-12T20:03:01.772 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:03:01.772 DEBUG:boto:Data: 2022-09-12T20:03:01.773 DEBUG:boto:Headers: {} 2022-09-12T20:03:01.773 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:01.774 DEBUG:boto:Port: 8000 2022-09-12T20:03:01.774 DEBUG:boto:Params: {} 2022-09-12T20:03:01.776 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T20:03:01.776 DEBUG:boto:Token: None 2022-09-12T20:03:01.777 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 20:03:01 GMT /jdpiak-75/ 2022-09-12T20:03:01.778 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B0HBW3eVTrGzu738QJa1mQRxGj8= 2022-09-12T20:03:01.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B0HBW3eVTrGzu738QJa1mQRxGj8=', 'Content-Length': '0'} 2022-09-12T20:03:01.810 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000372bf1c0ef6a60ad-00631f9075-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:03:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:01.811 DEBUG:rgw_multi.tests:created bucket=jdpiak-75 2022-09-12T20:03:01.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:03:01.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:03:01.814 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:03:01.874 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:03:01.874 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:03:01.927 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=38ca1f57-1358-496c-a99b-df49ee365a4f 2022-09-12T20:03:01.941 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T20:03:01.942 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:03:01.942 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T20:03:01.942 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T20:03:01.943 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:03:01.943 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:03:01.943 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T20:03:01.943 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T20:03:01.944 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:03:01.944 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:03:01.945 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663012981.806440_775.1", 2022-09-12T20:03:01.945 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:03:01.806440Z" 2022-09-12T20:03:01.945 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:03:01.945 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:03:01.946 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T20:03:01.946 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T20:03:01.947 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:03:01.947 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T20:03:01.949 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1663012981.806440_775.1', 3: ''} 2022-09-12T20:03:01.950 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T20:03:01.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:03:01.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:03:01.952 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:03:02.011 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:03:02.012 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:03:02.048 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:03:02.046+0000 7fce788c4600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:03:02.049 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:03:02.047+0000 7fce788c4600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:03:02.064 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:03:02.064 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:03:02.065 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:03:02.065 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:03:02.066 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:03:02.066 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:03:02.067 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:03:02.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:03:02.068 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:03:02.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:03:02.069 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:03:02.069 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:03:02.070 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:03:02.070 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:03:02.071 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:03:02.071 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:03:02.071 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:03:02.072 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:03:02.072 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:03:02.073 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:03:02.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:03:02.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:03:02.075 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:03:02.075 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:03:02.076 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:03:02.077 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:03:02.077 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:03:02.079 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:03:02.080 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:03:02.080 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:03:02.081 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:03:02.081 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:03:02.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:03:02.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:03:02.083 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:03:02.083 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:03:02.084 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:03:02.084 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663012981.806440_775.1", 2022-09-12T20:03:02.085 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:03:02.086 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:03:02.086 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:03:02.086 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:03:01.806440Z", 2022-09-12T20:03:02.087 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:03:02.087 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:03:02.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:03:02.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:03:02.089 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:03:02.089 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:03:02.089 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:03:02.090 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T20:03:02.090 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:03:02.091 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:03:02.091 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:03:02.092 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T20:03:02.092 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:03:02.092 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:03:02.093 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:03:02.093 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:03:02.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:03:02.094 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:03:02.095 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:03:02.096 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:03:02.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:03:02.097 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:03:02Z" 2022-09-12T20:03:02.097 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:03:02.100 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663012981.806440_775.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:03:01.806440Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:03:02Z" } 2022-09-12T20:03:02.101 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663012981.806440_775.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:03:01.806440Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T20:03:02.103 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663012981.806440_775.1', 3: ''} 2022-09-12T20:03:02.104 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663012981.806440_775.1', 3: ''} 2022-09-12T20:03:02.104 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T20:03:02.105 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T20:03:02.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:03:02.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:03:02.108 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:03:02.172 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:03:02.173 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:03:02.208 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:03:02.206+0000 7f5b933ba600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:03:02.210 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:03:02.207+0000 7f5b933ba600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:03:02.231 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T20:03:02.231 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T20:03:02.231 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T20:03:02.232 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T20:03:02.232 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T20:03:02.232 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:03:02.233 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T20:03:02.233 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:03:02.235 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T20:03:02.235 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:03:02.236 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T20:03:02.236 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:03:02.237 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T20:03:02.238 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:03:02.238 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:03:02.238 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T20:03:02.239 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:03:02.239 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:03:02.240 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T20:03:02.241 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:03:02.241 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:03:02.241 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:03:02.242 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T20:03:02.243 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:03:02.243 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T20:03:02.243 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T20:03:02.244 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:03:02.244 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:03:02.245 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:03:02.246 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T20:03:02.246 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T20:03:02.247 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:03:02.247 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:03:02.247 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:03:02.247 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T20:03:02.248 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:03:02.248 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T20:03:02.248 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663012981.806440_775.1", 2022-09-12T20:03:02.249 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:03:02.249 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:03:02.249 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:03:02.250 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T20:03:01.806440Z", 2022-09-12T20:03:02.250 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T20:03:02.250 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:03:02.251 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:03:02.251 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:03:02.252 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T20:03:02.252 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:03:02.252 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T20:03:02.253 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T20:03:02.253 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:03:02.253 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:03:02.253 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:03:02.254 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T20:03:02.254 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T20:03:02.254 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:03:02.255 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:03:02.255 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T20:03:02.255 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:03:02.256 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T20:03:02.257 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T20:03:02.257 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T20:03:02.259 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:03:02.259 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T20:03:02Z" 2022-09-12T20:03:02.260 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T20:03:02.260 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663012981.806440_775.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:03:01.806440Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:03:02Z" } 2022-09-12T20:03:02.262 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663012981.806440_775.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:03:01.806440Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T20:03:02.263 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663012981.806440_775.1', 3: ''} 2022-09-12T20:03:02.263 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663012981.806440_775.1', 3: ''} 2022-09-12T20:03:02.264 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T20:03:02.265 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T20:03:02.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:03:02.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:03:02.268 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:03:02.322 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:03:02.323 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:03:02.350 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:03:02.347+0000 7f31a4356600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:03:02.351 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:03:02.348+0000 7f31a4356600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:03:02.363 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:03:02.363 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:03:02.364 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:03:02.364 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:03:02.365 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:03:02.365 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:03:02.365 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:03:02.366 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:03:02.366 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:03:02.367 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:03:02.367 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:03:02.368 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:03:02.368 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:03:02.369 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:03:02.369 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:03:02.370 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:03:02.371 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:03:02.372 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:03:02.372 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:03:02.373 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:03:02.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:03:02.374 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:03:02.375 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:03:02.375 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:03:02.376 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:03:02.376 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:03:02.377 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:03:02.377 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:03:02.378 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:03:02.378 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:03:02.379 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:03:02.380 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:03:02.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:03:02.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:03:02.381 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:03:02.381 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:03:02.381 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:03:02.382 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663012981.806440_775.1", 2022-09-12T20:03:02.382 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:03:02.382 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:03:02.383 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:03:02.383 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:03:01.806440Z", 2022-09-12T20:03:02.383 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:03:02.383 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:03:02.384 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:03:02.384 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:03:02.384 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:03:02.385 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:03:02.385 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:03:02.385 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T20:03:02.385 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:03:02.386 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:03:02.386 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:03:02.386 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T20:03:02.386 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:03:02.387 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:03:02.387 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:03:02.387 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:03:02.388 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:03:02.388 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:03:02.388 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:03:02.389 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:03:02.389 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:03:02.389 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:03:02Z" 2022-09-12T20:03:02.389 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:03:02.391 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663012981.806440_775.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:03:01.806440Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:03:02Z" } 2022-09-12T20:03:02.392 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663012981.806440_775.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:03:01.806440Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T20:03:02.392 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663012981.806440_775.1', 3: ''} 2022-09-12T20:03:02.393 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663012981.806440_775.1', 3: ''} 2022-09-12T20:03:02.393 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T20:03:02.394 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:02.395 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:02.395 DEBUG:boto:Method: HEAD 2022-09-12T20:03:02.396 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:03:02.397 DEBUG:boto:Data: 2022-09-12T20:03:02.398 DEBUG:boto:Headers: {} 2022-09-12T20:03:02.399 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:02.399 DEBUG:boto:Port: 8000 2022-09-12T20:03:02.400 DEBUG:boto:Params: {} 2022-09-12T20:03:02.401 DEBUG:boto:Token: None 2022-09-12T20:03:02.402 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:02 GMT /jdpiak-75/ 2022-09-12T20:03:02.403 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VulGBWkh5WCqQqliULE7N5Lib78= 2022-09-12T20:03:02.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VulGBWkh5WCqQqliULE7N5Lib78=', 'Content-Length': '0'} 2022-09-12T20:03:02.406 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afbb5b0f2bc3c0c7-00631f9076-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:02.408 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:02.408 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:02.409 DEBUG:boto:Method: PUT 2022-09-12T20:03:02.410 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:03:02.410 DEBUG:boto:Data: 2022-09-12T20:03:02.411 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:03:02.411 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:02.412 DEBUG:boto:Port: 8000 2022-09-12T20:03:02.412 DEBUG:boto:Params: {} 2022-09-12T20:03:02.413 DEBUG:boto:Token: None 2022-09-12T20:03:02.414 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:03:02 GMT /jdpiak-75/obj11 2022-09-12T20:03:02.415 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oWDfz3cW/WzXGVsiE31gl9ziqio= 2022-09-12T20:03:02.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:03:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oWDfz3cW/WzXGVsiE31gl9ziqio='} 2022-09-12T20:03:02.424 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054495aebe6bc6945-00631f9076-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:02.425 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:02.426 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:02.427 DEBUG:boto:Method: HEAD 2022-09-12T20:03:02.427 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:03:02.428 DEBUG:boto:Data: 2022-09-12T20:03:02.428 DEBUG:boto:Headers: {} 2022-09-12T20:03:02.429 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:02.429 DEBUG:boto:Port: 8000 2022-09-12T20:03:02.430 DEBUG:boto:Params: {} 2022-09-12T20:03:02.431 DEBUG:boto:Token: None 2022-09-12T20:03:02.431 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:02 GMT /jdpiak-75/ 2022-09-12T20:03:02.432 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VulGBWkh5WCqQqliULE7N5Lib78= 2022-09-12T20:03:02.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VulGBWkh5WCqQqliULE7N5Lib78=', 'Content-Length': '0'} 2022-09-12T20:03:02.435 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ca19e677ac71f6f-00631f9076-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:02.436 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:02.436 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:02.437 DEBUG:boto:Method: PUT 2022-09-12T20:03:02.438 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:03:02.438 DEBUG:boto:Data: 2022-09-12T20:03:02.439 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:03:02.439 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:02.440 DEBUG:boto:Port: 8000 2022-09-12T20:03:02.440 DEBUG:boto:Params: {} 2022-09-12T20:03:02.441 DEBUG:boto:Token: None 2022-09-12T20:03:02.442 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:03:02 GMT /jdpiak-75/obj22 2022-09-12T20:03:02.443 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4I4RSOUZQ3S6yN5yync2VBiGfLQ= 2022-09-12T20:03:02.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:03:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4I4RSOUZQ3S6yN5yync2VBiGfLQ='} 2022-09-12T20:03:02.486 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003dcdd9d133f7edc0-00631f9076-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:02.488 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:02.488 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:02.489 DEBUG:boto:Method: HEAD 2022-09-12T20:03:02.489 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:03:02.490 DEBUG:boto:Data: 2022-09-12T20:03:02.490 DEBUG:boto:Headers: {} 2022-09-12T20:03:02.491 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:02.491 DEBUG:boto:Port: 8000 2022-09-12T20:03:02.492 DEBUG:boto:Params: {} 2022-09-12T20:03:02.493 DEBUG:boto:Token: None 2022-09-12T20:03:02.493 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:02 GMT /jdpiak-75/ 2022-09-12T20:03:02.494 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VulGBWkh5WCqQqliULE7N5Lib78= 2022-09-12T20:03:02.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VulGBWkh5WCqQqliULE7N5Lib78=', 'Content-Length': '0'} 2022-09-12T20:03:02.497 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8bcd2a366614808-00631f9076-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:02.498 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:02.499 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:02.499 DEBUG:boto:Method: PUT 2022-09-12T20:03:02.500 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:03:02.501 DEBUG:boto:Data: 2022-09-12T20:03:02.502 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:03:02.503 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:02.504 DEBUG:boto:Port: 8000 2022-09-12T20:03:02.505 DEBUG:boto:Params: {} 2022-09-12T20:03:02.507 DEBUG:boto:Token: None 2022-09-12T20:03:02.508 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:03:02 GMT /jdpiak-75/obj33 2022-09-12T20:03:02.512 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:heNtny6TKANdme95jbS4sCD9Oow= 2022-09-12T20:03:02.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:03:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:heNtny6TKANdme95jbS4sCD9Oow='} 2022-09-12T20:03:02.519 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000045e2aa7bb0c3c87a-00631f9076-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:02.521 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:02.522 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:02.522 DEBUG:boto:Method: HEAD 2022-09-12T20:03:02.523 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:03:02.524 DEBUG:boto:Data: 2022-09-12T20:03:02.525 DEBUG:boto:Headers: {} 2022-09-12T20:03:02.526 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:02.526 DEBUG:boto:Port: 8000 2022-09-12T20:03:02.527 DEBUG:boto:Params: {} 2022-09-12T20:03:02.528 DEBUG:boto:Token: None 2022-09-12T20:03:02.528 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:02 GMT /jdpiak-75/ 2022-09-12T20:03:02.529 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VulGBWkh5WCqQqliULE7N5Lib78= 2022-09-12T20:03:02.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VulGBWkh5WCqQqliULE7N5Lib78=', 'Content-Length': '0'} 2022-09-12T20:03:02.532 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000791937ede8eab941-00631f9076-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:02.534 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:02.535 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:02.535 DEBUG:boto:Method: PUT 2022-09-12T20:03:02.536 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:03:02.537 DEBUG:boto:Data: 2022-09-12T20:03:02.538 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:03:02.539 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:02.540 DEBUG:boto:Port: 8000 2022-09-12T20:03:02.540 DEBUG:boto:Params: {} 2022-09-12T20:03:02.541 DEBUG:boto:Token: None 2022-09-12T20:03:02.542 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:03:02 GMT /jdpiak-75/obj44 2022-09-12T20:03:02.542 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:58RSVo3Nya+MswjB0LACdzJRZxs= 2022-09-12T20:03:02.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:03:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:58RSVo3Nya+MswjB0LACdzJRZxs='} 2022-09-12T20:03:02.552 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e0e7425b59d75e0-00631f9076-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:02.553 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:02.554 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:02.555 DEBUG:boto:Method: HEAD 2022-09-12T20:03:02.556 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:03:02.556 DEBUG:boto:Data: 2022-09-12T20:03:02.557 DEBUG:boto:Headers: {} 2022-09-12T20:03:02.557 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:02.558 DEBUG:boto:Port: 8000 2022-09-12T20:03:02.559 DEBUG:boto:Params: {} 2022-09-12T20:03:02.560 DEBUG:boto:Token: None 2022-09-12T20:03:02.560 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:02 GMT /jdpiak-75/ 2022-09-12T20:03:02.561 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VulGBWkh5WCqQqliULE7N5Lib78= 2022-09-12T20:03:02.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VulGBWkh5WCqQqliULE7N5Lib78=', 'Content-Length': '0'} 2022-09-12T20:03:02.565 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001332adec7ba6a134-00631f9076-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:02.566 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:02.567 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:02.568 DEBUG:boto:Method: PUT 2022-09-12T20:03:02.568 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:03:02.569 DEBUG:boto:Data: 2022-09-12T20:03:02.570 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:03:02.570 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:02.571 DEBUG:boto:Port: 8000 2022-09-12T20:03:02.572 DEBUG:boto:Params: {} 2022-09-12T20:03:02.573 DEBUG:boto:Token: None 2022-09-12T20:03:02.574 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:03:02 GMT /jdpiak-75/obj55 2022-09-12T20:03:02.575 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B7xOTPcSc+AaQrlOfEhd+uelQkI= 2022-09-12T20:03:02.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:03:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B7xOTPcSc+AaQrlOfEhd+uelQkI='} 2022-09-12T20:03:02.618 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000030fc06c9205386a4-00631f9076-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:02.619 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-75 2022-09-12T20:03:02.620 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:03:02.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:03:02.621 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T20:03:02.680 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:03:02.680 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:03:02.730 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:03:02.728+0000 7faaed643600 1 waiting to reach incremental sync.. 2022-09-12T20:03:07.732 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:03:07.730+0000 7faaed643600 1 bucket sync caught up with source: 2022-09-12T20:03:07.733 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000001.37.5, 00000000002.65.5, 00000000002.71.5] 2022-09-12T20:03:07.734 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000001.37.5, 00000000002.65.5, 00000000002.71.5] 2022-09-12T20:03:07.734 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:03:07.730+0000 7faaed643600 0 bucket checkpoint complete 2022-09-12T20:03:07.750 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-75 2022-09-12T20:03:07.751 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:03:07.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:03:07.752 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:03:07.819 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:03:07.820 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:03:07.861 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:03:07.860+0000 7fe5ca6b6600 1 bucket sync caught up with source: 2022-09-12T20:03:07.862 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000001.37.5, 00000000002.65.5, 00000000002.71.5] 2022-09-12T20:03:07.862 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000001.37.5, 00000000002.65.5, 00000000002.71.5] 2022-09-12T20:03:07.862 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:03:07.860+0000 7fe5ca6b6600 0 bucket checkpoint complete 2022-09-12T20:03:07.876 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-75 2022-09-12T20:03:07.877 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:03:07.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:03:07.878 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T20:03:07.940 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:03:07.941 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:03:07.975 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:03:07.973+0000 7f84e90bb600 1 waiting to reach incremental sync.. 2022-09-12T20:03:12.975 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:03:12.973+0000 7f84e90bb600 1 waiting to reach incremental sync.. 2022-09-12T20:03:17.978 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:03:17.975+0000 7f84e90bb600 1 bucket sync caught up with source: 2022-09-12T20:03:17.978 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000001.37.5, 00000000002.65.5, 00000000002.71.5] 2022-09-12T20:03:17.979 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000001.37.5, 00000000002.65.5, 00000000002.71.5] 2022-09-12T20:03:17.979 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:03:17.975+0000 7f84e90bb600 0 bucket checkpoint complete 2022-09-12T20:03:17.995 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-75 2022-09-12T20:03:17.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:03:17.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:03:17.997 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:03:18.064 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:03:18.064 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:03:18.116 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:03:18.114+0000 7f35aa3b3600 1 bucket sync caught up with source: 2022-09-12T20:03:18.117 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000001.544.5, 00000000002.905.5, 00000000002.570.5] 2022-09-12T20:03:18.117 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000001.544.5, 00000000002.905.5, 00000000002.570.5] 2022-09-12T20:03:18.118 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:03:18.114+0000 7f35aa3b3600 0 bucket checkpoint complete 2022-09-12T20:03:18.133 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-75 2022-09-12T20:03:18.133 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:03:18.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:03:18.135 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:03:18.193 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:03:18.193 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:03:19.231 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:03:19.229+0000 7faf5e82b600 1 bucket sync caught up with source: 2022-09-12T20:03:19.232 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000001.544.5, 00000000002.905.5, 00000000002.570.5] 2022-09-12T20:03:19.232 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000001.544.5, 00000000002.905.5, 00000000002.570.5] 2022-09-12T20:03:19.232 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:03:19.229+0000 7faf5e82b600 0 bucket checkpoint complete 2022-09-12T20:03:19.252 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-75 2022-09-12T20:03:19.252 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-75 2022-09-12T20:03:19.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:03:19.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:03:19.254 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:03:19.317 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:03:19.318 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:03:19.376 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:03:19.374+0000 7f570bd47600 1 waiting to reach incremental sync.. 2022-09-12T20:03:24.377 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:03:24.376+0000 7f570bd47600 1 bucket sync caught up with source: 2022-09-12T20:03:24.378 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000001.314.5, 00000000002.521.5, 00000000002.484.5] 2022-09-12T20:03:24.378 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000001.314.5, 00000000002.521.5, 00000000002.484.5] 2022-09-12T20:03:24.379 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:03:24.376+0000 7f570bd47600 0 bucket checkpoint complete 2022-09-12T20:03:24.394 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-75 2022-09-12T20:03:24.395 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:03:24.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:03:24.396 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T20:03:24.459 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:03:24.460 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:03:24.501 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:03:24.499+0000 7fae1868e600 1 bucket sync caught up with source: 2022-09-12T20:03:24.501 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000001.314.5, 00000000002.521.5, 00000000002.484.5] 2022-09-12T20:03:24.501 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000001.314.5, 00000000002.521.5, 00000000002.484.5] 2022-09-12T20:03:24.502 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:03:24.499+0000 7fae1868e600 0 bucket checkpoint complete 2022-09-12T20:03:24.516 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-75 2022-09-12T20:03:24.517 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-75 zones={test-zone2, test-zone1} 2022-09-12T20:03:24.517 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:24.518 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:24.518 DEBUG:boto:Method: HEAD 2022-09-12T20:03:24.519 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:03:24.519 DEBUG:boto:Data: 2022-09-12T20:03:24.520 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.520 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:24.521 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.521 DEBUG:boto:Params: {} 2022-09-12T20:03:24.522 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T20:03:24.523 DEBUG:boto:Token: None 2022-09-12T20:03:24.523 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/ 2022-09-12T20:03:24.524 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NYlcDAe60qFGMJSV4qltXqZTMk8= 2022-09-12T20:03:24.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NYlcDAe60qFGMJSV4qltXqZTMk8=', 'Content-Length': '0'} 2022-09-12T20:03:24.528 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5dfd20dfb329961-00631f908c-189d-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.529 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:24.529 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:24.530 DEBUG:boto:Method: HEAD 2022-09-12T20:03:24.530 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:03:24.531 DEBUG:boto:Data: 2022-09-12T20:03:24.531 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.532 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:24.532 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.533 DEBUG:boto:Params: {} 2022-09-12T20:03:24.534 DEBUG:boto:Token: None 2022-09-12T20:03:24.534 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/ 2022-09-12T20:03:24.535 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NYlcDAe60qFGMJSV4qltXqZTMk8= 2022-09-12T20:03:24.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NYlcDAe60qFGMJSV4qltXqZTMk8=', 'Content-Length': '0'} 2022-09-12T20:03:24.538 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077ca94c24ab04ad7-00631f908c-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.539 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T20:03:24.540 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:24.540 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:24.541 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:24.541 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:24.542 DEBUG:boto:Method: GET 2022-09-12T20:03:24.542 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:24.543 DEBUG:boto:Data: 2022-09-12T20:03:24.543 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.544 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:24.544 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.545 DEBUG:boto:Params: {} 2022-09-12T20:03:24.545 DEBUG:boto:Token: None 2022-09-12T20:03:24.546 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/?versions 2022-09-12T20:03:24.546 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hdAsTY88CHA2zlLudOHnF+OA6Z8= 2022-09-12T20:03:24.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hdAsTY88CHA2zlLudOHnF+OA6Z8=', 'Content-Length': '0'} 2022-09-12T20:03:24.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003670b7b5ea97f7bf-00631f908c-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.550 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:24.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:03:24.552 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:03:24.552 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:03:24.553 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:03:24.553 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:03:24.554 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T20:03:24.555 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:24.555 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:24.556 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:24.556 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:24.557 DEBUG:boto:Method: GET 2022-09-12T20:03:24.557 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:24.558 DEBUG:boto:Data: 2022-09-12T20:03:24.558 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.559 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:24.559 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.560 DEBUG:boto:Params: {} 2022-09-12T20:03:24.560 DEBUG:boto:Token: None 2022-09-12T20:03:24.561 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/?versions 2022-09-12T20:03:24.561 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hdAsTY88CHA2zlLudOHnF+OA6Z8= 2022-09-12T20:03:24.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hdAsTY88CHA2zlLudOHnF+OA6Z8=', 'Content-Length': '0'} 2022-09-12T20:03:24.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1c71502a34f0d5a-00631f908c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.565 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:24.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:03:24.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:03:24.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:03:24.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:03:24.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:03:24.569 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:24.569 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:24.570 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:24.570 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:24.571 DEBUG:boto:Method: GET 2022-09-12T20:03:24.571 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:24.572 DEBUG:boto:Data: 2022-09-12T20:03:24.572 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.573 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:24.573 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.574 DEBUG:boto:Params: {} 2022-09-12T20:03:24.575 DEBUG:boto:Token: None 2022-09-12T20:03:24.575 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/?versions 2022-09-12T20:03:24.576 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hdAsTY88CHA2zlLudOHnF+OA6Z8= 2022-09-12T20:03:24.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hdAsTY88CHA2zlLudOHnF+OA6Z8=', 'Content-Length': '0'} 2022-09-12T20:03:24.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef4ef7c0159ea51a-00631f908c-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.579 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:24.580 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:24.581 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:24.581 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:24.582 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:24.582 DEBUG:boto:Method: GET 2022-09-12T20:03:24.583 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:24.583 DEBUG:boto:Data: 2022-09-12T20:03:24.584 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.584 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:24.585 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.585 DEBUG:boto:Params: {} 2022-09-12T20:03:24.586 DEBUG:boto:Token: None 2022-09-12T20:03:24.586 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/?versions 2022-09-12T20:03:24.587 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hdAsTY88CHA2zlLudOHnF+OA6Z8= 2022-09-12T20:03:24.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hdAsTY88CHA2zlLudOHnF+OA6Z8=', 'Content-Length': '0'} 2022-09-12T20:03:24.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d779e6322a0bcf9c-00631f908c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.591 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:24.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:03:24.592 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:24.593 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:24.593 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:24.594 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:24.594 DEBUG:boto:Method: GET 2022-09-12T20:03:24.595 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:24.596 DEBUG:boto:Data: 2022-09-12T20:03:24.596 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.597 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:24.597 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.598 DEBUG:boto:Params: {} 2022-09-12T20:03:24.598 DEBUG:boto:Token: None 2022-09-12T20:03:24.599 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:24.599 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fq4xj5IVtoqwufk3LN9YPcam03A= 2022-09-12T20:03:24.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fq4xj5IVtoqwufk3LN9YPcam03A=', 'Content-Length': '0'} 2022-09-12T20:03:24.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba874a34a6175ed7-00631f908c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.603 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:24.603 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:24.604 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:24.604 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:24.605 DEBUG:boto:Method: GET 2022-09-12T20:03:24.605 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:24.606 DEBUG:boto:Data: 2022-09-12T20:03:24.606 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.607 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:24.607 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.608 DEBUG:boto:Params: {} 2022-09-12T20:03:24.608 DEBUG:boto:Token: None 2022-09-12T20:03:24.609 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:24.610 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fq4xj5IVtoqwufk3LN9YPcam03A= 2022-09-12T20:03:24.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fq4xj5IVtoqwufk3LN9YPcam03A=', 'Content-Length': '0'} 2022-09-12T20:03:24.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bdbac0db68cad56-00631f908c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.613 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:24.614 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:24.614 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:24.615 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:24.615 DEBUG:boto:Method: HEAD 2022-09-12T20:03:24.616 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:24.616 DEBUG:boto:Data: 2022-09-12T20:03:24.617 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.617 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:24.618 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.618 DEBUG:boto:Params: {} 2022-09-12T20:03:24.619 DEBUG:boto:Token: None 2022-09-12T20:03:24.619 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:24.620 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jy3QT8ITuhOb9bEGJLKFb4cTNqY= 2022-09-12T20:03:24.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jy3QT8ITuhOb9bEGJLKFb4cTNqY=', 'Content-Length': '0'} 2022-09-12T20:03:24.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebb950a0f7d77577-00631f908c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.623 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:24.623 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:24.624 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:24.624 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:24.625 DEBUG:boto:Method: HEAD 2022-09-12T20:03:24.625 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:24.626 DEBUG:boto:Data: 2022-09-12T20:03:24.626 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.627 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:24.627 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.628 DEBUG:boto:Params: {} 2022-09-12T20:03:24.628 DEBUG:boto:Token: None 2022-09-12T20:03:24.629 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:24.629 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jy3QT8ITuhOb9bEGJLKFb4cTNqY= 2022-09-12T20:03:24.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jy3QT8ITuhOb9bEGJLKFb4cTNqY=', 'Content-Length': '0'} 2022-09-12T20:03:24.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000656a691b448ce312-00631f908c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:03:24.633 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:24.633 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:24.634 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:24.635 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:24.635 DEBUG:boto:Method: GET 2022-09-12T20:03:24.636 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:24.636 DEBUG:boto:Data: 2022-09-12T20:03:24.637 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.637 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:24.638 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.638 DEBUG:boto:Params: {} 2022-09-12T20:03:24.639 DEBUG:boto:Token: None 2022-09-12T20:03:24.639 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:24.640 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fq4xj5IVtoqwufk3LN9YPcam03A= 2022-09-12T20:03:24.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fq4xj5IVtoqwufk3LN9YPcam03A=', 'Content-Length': '0'} 2022-09-12T20:03:24.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009c605ac2824f364-00631f908c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.686 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:24.687 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:24.687 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:24.688 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:24.688 DEBUG:boto:Method: GET 2022-09-12T20:03:24.689 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:24.689 DEBUG:boto:Data: 2022-09-12T20:03:24.690 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.690 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:24.691 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.691 DEBUG:boto:Params: {} 2022-09-12T20:03:24.692 DEBUG:boto:Token: None 2022-09-12T20:03:24.693 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:24.693 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fq4xj5IVtoqwufk3LN9YPcam03A= 2022-09-12T20:03:24.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fq4xj5IVtoqwufk3LN9YPcam03A=', 'Content-Length': '0'} 2022-09-12T20:03:24.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec72984f7d147a26-00631f908c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.697 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:24.697 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:24.698 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:03:24.699 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:03:24.699 DEBUG:boto:Method: GET 2022-09-12T20:03:24.700 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:03:24.700 DEBUG:boto:Data: 2022-09-12T20:03:24.701 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.701 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:24.702 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.702 DEBUG:boto:Params: {} 2022-09-12T20:03:24.703 DEBUG:boto:Token: None 2022-09-12T20:03:24.703 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/ 2022-09-12T20:03:24.704 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fJZNImFB1tiZg1JGZp+aM+xBTyw= 2022-09-12T20:03:24.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fJZNImFB1tiZg1JGZp+aM+xBTyw=', 'Content-Length': '0'} 2022-09-12T20:03:24.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c62fb0d131476b30-00631f908c-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.708 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:24.709 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:24.709 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:24.710 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:03:24.710 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:03:24.711 DEBUG:boto:Method: GET 2022-09-12T20:03:24.711 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:03:24.712 DEBUG:boto:Data: 2022-09-12T20:03:24.712 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.713 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:24.713 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.714 DEBUG:boto:Params: {} 2022-09-12T20:03:24.714 DEBUG:boto:Token: None 2022-09-12T20:03:24.715 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/ 2022-09-12T20:03:24.716 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fJZNImFB1tiZg1JGZp+aM+xBTyw= 2022-09-12T20:03:24.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fJZNImFB1tiZg1JGZp+aM+xBTyw=', 'Content-Length': '0'} 2022-09-12T20:03:24.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043e69c27a292a629-00631f908c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.719 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:24.720 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:24.721 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:24.721 DEBUG:boto:Method: HEAD 2022-09-12T20:03:24.722 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:03:24.722 DEBUG:boto:Data: 2022-09-12T20:03:24.723 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.723 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:24.724 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.724 DEBUG:boto:Params: {} 2022-09-12T20:03:24.725 DEBUG:boto:Token: None 2022-09-12T20:03:24.725 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/obj11 2022-09-12T20:03:24.726 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PwQeTUHSWonJrd3jucx7Vet8w50= 2022-09-12T20:03:24.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PwQeTUHSWonJrd3jucx7Vet8w50=', 'Content-Length': '0'} 2022-09-12T20:03:24.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f04341a4de0c3147-00631f908c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.729 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:24.729 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:24.730 DEBUG:boto:Method: HEAD 2022-09-12T20:03:24.730 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:03:24.731 DEBUG:boto:Data: 2022-09-12T20:03:24.731 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.732 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:24.732 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.733 DEBUG:boto:Params: {} 2022-09-12T20:03:24.733 DEBUG:boto:Token: None 2022-09-12T20:03:24.734 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/obj11 2022-09-12T20:03:24.734 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PwQeTUHSWonJrd3jucx7Vet8w50= 2022-09-12T20:03:24.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PwQeTUHSWonJrd3jucx7Vet8w50=', 'Content-Length': '0'} 2022-09-12T20:03:24.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000081e91b37d65425d-00631f908c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:03:24.738 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:24.739 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:24.739 DEBUG:boto:Method: GET 2022-09-12T20:03:24.740 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:03:24.741 DEBUG:boto:Data: 2022-09-12T20:03:24.741 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.742 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:24.742 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.743 DEBUG:boto:Params: {} 2022-09-12T20:03:24.743 DEBUG:boto:Token: None 2022-09-12T20:03:24.744 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/obj11 2022-09-12T20:03:24.745 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lH47aYQASI5SHFRD35CXvsC5cpI= 2022-09-12T20:03:24.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lH47aYQASI5SHFRD35CXvsC5cpI=', 'Content-Length': '0'} 2022-09-12T20:03:24.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002871d0feeed9937e-00631f908c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.749 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:24.749 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:24.750 DEBUG:boto:Method: GET 2022-09-12T20:03:24.751 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:03:24.751 DEBUG:boto:Data: 2022-09-12T20:03:24.752 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.752 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:24.753 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.753 DEBUG:boto:Params: {} 2022-09-12T20:03:24.754 DEBUG:boto:Token: None 2022-09-12T20:03:24.754 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/obj11 2022-09-12T20:03:24.755 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lH47aYQASI5SHFRD35CXvsC5cpI= 2022-09-12T20:03:24.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lH47aYQASI5SHFRD35CXvsC5cpI=', 'Content-Length': '0'} 2022-09-12T20:03:24.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094a99a16c2e2b3ae-00631f908c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.798 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:24.799 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:24.799 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:03:24.800 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:03:24.800 DEBUG:boto:Method: GET 2022-09-12T20:03:24.801 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:03:24.801 DEBUG:boto:Data: 2022-09-12T20:03:24.802 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.802 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:24.803 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.803 DEBUG:boto:Params: {} 2022-09-12T20:03:24.804 DEBUG:boto:Token: None 2022-09-12T20:03:24.805 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/ 2022-09-12T20:03:24.805 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fJZNImFB1tiZg1JGZp+aM+xBTyw= 2022-09-12T20:03:24.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fJZNImFB1tiZg1JGZp+aM+xBTyw=', 'Content-Length': '0'} 2022-09-12T20:03:24.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019f9d475d9f02867-00631f908c-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.809 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:24.810 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:24.810 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:24.811 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:03:24.811 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:03:24.812 DEBUG:boto:Method: GET 2022-09-12T20:03:24.812 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:03:24.813 DEBUG:boto:Data: 2022-09-12T20:03:24.813 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.814 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:24.814 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.815 DEBUG:boto:Params: {} 2022-09-12T20:03:24.816 DEBUG:boto:Token: None 2022-09-12T20:03:24.816 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/ 2022-09-12T20:03:24.817 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fJZNImFB1tiZg1JGZp+aM+xBTyw= 2022-09-12T20:03:24.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fJZNImFB1tiZg1JGZp+aM+xBTyw=', 'Content-Length': '0'} 2022-09-12T20:03:24.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f5c24ea97880348-00631f908c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.821 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:24.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:03:24.822 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:24.823 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:24.823 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:24.824 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:24.824 DEBUG:boto:Method: GET 2022-09-12T20:03:24.825 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:24.825 DEBUG:boto:Data: 2022-09-12T20:03:24.826 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.827 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:24.827 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.828 DEBUG:boto:Params: {} 2022-09-12T20:03:24.828 DEBUG:boto:Token: None 2022-09-12T20:03:24.829 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:24.830 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:edFI4frblUjwiw9rVTzEpl1Q/b0= 2022-09-12T20:03:24.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:edFI4frblUjwiw9rVTzEpl1Q/b0=', 'Content-Length': '0'} 2022-09-12T20:03:24.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086da9749d69cad9c-00631f908c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.833 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:24.834 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:24.834 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:24.835 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:24.835 DEBUG:boto:Method: GET 2022-09-12T20:03:24.836 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:24.836 DEBUG:boto:Data: 2022-09-12T20:03:24.837 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.838 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:24.838 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.839 DEBUG:boto:Params: {} 2022-09-12T20:03:24.839 DEBUG:boto:Token: None 2022-09-12T20:03:24.840 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:24.840 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:edFI4frblUjwiw9rVTzEpl1Q/b0= 2022-09-12T20:03:24.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:edFI4frblUjwiw9rVTzEpl1Q/b0=', 'Content-Length': '0'} 2022-09-12T20:03:24.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a0499693fd55f46-00631f908c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.844 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:24.844 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:24.845 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:24.845 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:24.846 DEBUG:boto:Method: HEAD 2022-09-12T20:03:24.847 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:24.847 DEBUG:boto:Data: 2022-09-12T20:03:24.848 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.848 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:24.849 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.849 DEBUG:boto:Params: {} 2022-09-12T20:03:24.850 DEBUG:boto:Token: None 2022-09-12T20:03:24.850 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:24.851 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4GCr26RLqhoW7B9evt8P8vL/r6I= 2022-09-12T20:03:24.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4GCr26RLqhoW7B9evt8P8vL/r6I=', 'Content-Length': '0'} 2022-09-12T20:03:24.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ebf878965baf89f-00631f908c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.854 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:24.854 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:24.855 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:24.855 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:24.856 DEBUG:boto:Method: HEAD 2022-09-12T20:03:24.857 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:24.857 DEBUG:boto:Data: 2022-09-12T20:03:24.858 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.858 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:24.859 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.859 DEBUG:boto:Params: {} 2022-09-12T20:03:24.860 DEBUG:boto:Token: None 2022-09-12T20:03:24.860 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:24.861 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4GCr26RLqhoW7B9evt8P8vL/r6I= 2022-09-12T20:03:24.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4GCr26RLqhoW7B9evt8P8vL/r6I=', 'Content-Length': '0'} 2022-09-12T20:03:24.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098e24d1dfee30d08-00631f908c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:03:24.864 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:24.865 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:24.865 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:24.866 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:24.866 DEBUG:boto:Method: GET 2022-09-12T20:03:24.867 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:24.867 DEBUG:boto:Data: 2022-09-12T20:03:24.868 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.868 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:24.869 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.869 DEBUG:boto:Params: {} 2022-09-12T20:03:24.870 DEBUG:boto:Token: None 2022-09-12T20:03:24.871 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:24.871 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:edFI4frblUjwiw9rVTzEpl1Q/b0= 2022-09-12T20:03:24.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:edFI4frblUjwiw9rVTzEpl1Q/b0=', 'Content-Length': '0'} 2022-09-12T20:03:24.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dd8dfa31b21b007-00631f908c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.874 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:24.875 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:24.875 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:24.876 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:24.877 DEBUG:boto:Method: GET 2022-09-12T20:03:24.877 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:24.878 DEBUG:boto:Data: 2022-09-12T20:03:24.878 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.879 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:24.879 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.880 DEBUG:boto:Params: {} 2022-09-12T20:03:24.880 DEBUG:boto:Token: None 2022-09-12T20:03:24.881 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:24.881 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:edFI4frblUjwiw9rVTzEpl1Q/b0= 2022-09-12T20:03:24.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:edFI4frblUjwiw9rVTzEpl1Q/b0=', 'Content-Length': '0'} 2022-09-12T20:03:24.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000551d463e156965d2-00631f908c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.885 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:24.886 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:24.886 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:03:24.887 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:03:24.887 DEBUG:boto:Method: GET 2022-09-12T20:03:24.888 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:03:24.888 DEBUG:boto:Data: 2022-09-12T20:03:24.889 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.889 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:24.890 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.890 DEBUG:boto:Params: {} 2022-09-12T20:03:24.891 DEBUG:boto:Token: None 2022-09-12T20:03:24.891 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/ 2022-09-12T20:03:24.892 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fJZNImFB1tiZg1JGZp+aM+xBTyw= 2022-09-12T20:03:24.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fJZNImFB1tiZg1JGZp+aM+xBTyw=', 'Content-Length': '0'} 2022-09-12T20:03:24.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f022b7e52e41896-00631f908c-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.896 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:24.896 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:24.897 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:24.897 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:03:24.898 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:03:24.898 DEBUG:boto:Method: GET 2022-09-12T20:03:24.899 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:03:24.899 DEBUG:boto:Data: 2022-09-12T20:03:24.900 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.900 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:24.901 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.901 DEBUG:boto:Params: {} 2022-09-12T20:03:24.902 DEBUG:boto:Token: None 2022-09-12T20:03:24.903 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/ 2022-09-12T20:03:24.903 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fJZNImFB1tiZg1JGZp+aM+xBTyw= 2022-09-12T20:03:24.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fJZNImFB1tiZg1JGZp+aM+xBTyw=', 'Content-Length': '0'} 2022-09-12T20:03:24.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e551967309a63b7-00631f908c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.907 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:24.908 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:24.909 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:24.909 DEBUG:boto:Method: HEAD 2022-09-12T20:03:24.910 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:03:24.910 DEBUG:boto:Data: 2022-09-12T20:03:24.911 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.911 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:24.912 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.912 DEBUG:boto:Params: {} 2022-09-12T20:03:24.913 DEBUG:boto:Token: None 2022-09-12T20:03:24.913 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/obj22 2022-09-12T20:03:24.914 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Pw2aYwRgxBBBEVfXalyb780CfhA= 2022-09-12T20:03:24.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Pw2aYwRgxBBBEVfXalyb780CfhA=', 'Content-Length': '0'} 2022-09-12T20:03:24.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8b266186dd37b58-00631f908c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.917 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:24.917 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:24.918 DEBUG:boto:Method: HEAD 2022-09-12T20:03:24.918 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:03:24.919 DEBUG:boto:Data: 2022-09-12T20:03:24.919 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.920 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:24.920 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.921 DEBUG:boto:Params: {} 2022-09-12T20:03:24.921 DEBUG:boto:Token: None 2022-09-12T20:03:24.922 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/obj22 2022-09-12T20:03:24.923 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Pw2aYwRgxBBBEVfXalyb780CfhA= 2022-09-12T20:03:24.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Pw2aYwRgxBBBEVfXalyb780CfhA=', 'Content-Length': '0'} 2022-09-12T20:03:24.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cc2fd9cd8266e99-00631f908c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:03:24.927 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:24.927 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:24.928 DEBUG:boto:Method: GET 2022-09-12T20:03:24.928 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:03:24.929 DEBUG:boto:Data: 2022-09-12T20:03:24.929 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.930 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:24.930 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.931 DEBUG:boto:Params: {} 2022-09-12T20:03:24.931 DEBUG:boto:Token: None 2022-09-12T20:03:24.932 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/obj22 2022-09-12T20:03:24.932 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cmw1oUenBCPjpn0xznWzE0essfw= 2022-09-12T20:03:24.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cmw1oUenBCPjpn0xznWzE0essfw=', 'Content-Length': '0'} 2022-09-12T20:03:24.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069a2a99245cc6ecc-00631f908c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.978 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:24.979 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:24.979 DEBUG:boto:Method: GET 2022-09-12T20:03:24.980 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:03:24.980 DEBUG:boto:Data: 2022-09-12T20:03:24.981 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.981 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:24.982 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.982 DEBUG:boto:Params: {} 2022-09-12T20:03:24.983 DEBUG:boto:Token: None 2022-09-12T20:03:24.984 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/obj22 2022-09-12T20:03:24.984 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cmw1oUenBCPjpn0xznWzE0essfw= 2022-09-12T20:03:24.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cmw1oUenBCPjpn0xznWzE0essfw=', 'Content-Length': '0'} 2022-09-12T20:03:24.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b97889584b36042-00631f908c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.988 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:24.988 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:24.989 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:03:24.990 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:03:24.990 DEBUG:boto:Method: GET 2022-09-12T20:03:24.991 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:03:24.991 DEBUG:boto:Data: 2022-09-12T20:03:24.992 DEBUG:boto:Headers: {} 2022-09-12T20:03:24.992 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:24.993 DEBUG:boto:Port: 8000 2022-09-12T20:03:24.993 DEBUG:boto:Params: {} 2022-09-12T20:03:24.994 DEBUG:boto:Token: None 2022-09-12T20:03:24.995 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:24 GMT /jdpiak-75/ 2022-09-12T20:03:24.995 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fJZNImFB1tiZg1JGZp+aM+xBTyw= 2022-09-12T20:03:24.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fJZNImFB1tiZg1JGZp+aM+xBTyw=', 'Content-Length': '0'} 2022-09-12T20:03:24.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001889a317063e4477-00631f908c-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:24.999 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:25.000 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:25.001 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:25.001 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:03:25.002 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:03:25.002 DEBUG:boto:Method: GET 2022-09-12T20:03:25.003 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:03:25.003 DEBUG:boto:Data: 2022-09-12T20:03:25.004 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.004 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.005 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.006 DEBUG:boto:Params: {} 2022-09-12T20:03:25.006 DEBUG:boto:Token: None 2022-09-12T20:03:25.007 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/ 2022-09-12T20:03:25.007 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk= 2022-09-12T20:03:25.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk=', 'Content-Length': '0'} 2022-09-12T20:03:25.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032b119534727c3ee-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.054 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:25.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:03:25.056 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:25.056 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:25.057 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:25.057 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:25.058 DEBUG:boto:Method: GET 2022-09-12T20:03:25.058 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:25.059 DEBUG:boto:Data: 2022-09-12T20:03:25.059 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.060 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:25.061 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.061 DEBUG:boto:Params: {} 2022-09-12T20:03:25.062 DEBUG:boto:Token: None 2022-09-12T20:03:25.062 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:25.063 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ejOF98d4fnck5WIRoZThGR5JHYY= 2022-09-12T20:03:25.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ejOF98d4fnck5WIRoZThGR5JHYY=', 'Content-Length': '0'} 2022-09-12T20:03:25.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005df8d2eb7ddce18f-00631f908d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.067 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:25.067 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:25.068 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:25.068 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:25.069 DEBUG:boto:Method: GET 2022-09-12T20:03:25.069 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:25.070 DEBUG:boto:Data: 2022-09-12T20:03:25.070 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.071 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.071 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.072 DEBUG:boto:Params: {} 2022-09-12T20:03:25.073 DEBUG:boto:Token: None 2022-09-12T20:03:25.073 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:25.074 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ejOF98d4fnck5WIRoZThGR5JHYY= 2022-09-12T20:03:25.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ejOF98d4fnck5WIRoZThGR5JHYY=', 'Content-Length': '0'} 2022-09-12T20:03:25.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f0fc05909fdd512-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.077 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:25.078 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:25.079 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:25.079 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:25.080 DEBUG:boto:Method: HEAD 2022-09-12T20:03:25.080 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:25.081 DEBUG:boto:Data: 2022-09-12T20:03:25.081 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.082 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:25.082 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.083 DEBUG:boto:Params: {} 2022-09-12T20:03:25.083 DEBUG:boto:Token: None 2022-09-12T20:03:25.084 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:25.084 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+j+2gBfxL8CtHXXbYChsonUGI/A= 2022-09-12T20:03:25.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+j+2gBfxL8CtHXXbYChsonUGI/A=', 'Content-Length': '0'} 2022-09-12T20:03:25.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddc372641e0bf283-00631f908d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.088 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:25.088 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:25.089 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:25.089 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:25.090 DEBUG:boto:Method: HEAD 2022-09-12T20:03:25.090 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:25.091 DEBUG:boto:Data: 2022-09-12T20:03:25.091 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.092 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.092 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.093 DEBUG:boto:Params: {} 2022-09-12T20:03:25.094 DEBUG:boto:Token: None 2022-09-12T20:03:25.094 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:25.095 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+j+2gBfxL8CtHXXbYChsonUGI/A= 2022-09-12T20:03:25.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+j+2gBfxL8CtHXXbYChsonUGI/A=', 'Content-Length': '0'} 2022-09-12T20:03:25.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000124bc817914c31b5-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:03:25.099 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:25.099 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:25.100 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:25.100 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:25.101 DEBUG:boto:Method: GET 2022-09-12T20:03:25.101 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:25.102 DEBUG:boto:Data: 2022-09-12T20:03:25.102 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.103 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:25.103 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.104 DEBUG:boto:Params: {} 2022-09-12T20:03:25.105 DEBUG:boto:Token: None 2022-09-12T20:03:25.105 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:25.106 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ejOF98d4fnck5WIRoZThGR5JHYY= 2022-09-12T20:03:25.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ejOF98d4fnck5WIRoZThGR5JHYY=', 'Content-Length': '0'} 2022-09-12T20:03:25.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025b1d6396607c16f-00631f908d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.109 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:25.110 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:25.110 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:25.111 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:25.111 DEBUG:boto:Method: GET 2022-09-12T20:03:25.112 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:25.112 DEBUG:boto:Data: 2022-09-12T20:03:25.113 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.113 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.114 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.114 DEBUG:boto:Params: {} 2022-09-12T20:03:25.115 DEBUG:boto:Token: None 2022-09-12T20:03:25.116 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:25.116 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ejOF98d4fnck5WIRoZThGR5JHYY= 2022-09-12T20:03:25.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ejOF98d4fnck5WIRoZThGR5JHYY=', 'Content-Length': '0'} 2022-09-12T20:03:25.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dba0b975bb606fe1-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.120 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:25.120 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:25.121 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:03:25.121 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:03:25.122 DEBUG:boto:Method: GET 2022-09-12T20:03:25.123 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:03:25.123 DEBUG:boto:Data: 2022-09-12T20:03:25.124 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.124 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:25.125 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.125 DEBUG:boto:Params: {} 2022-09-12T20:03:25.126 DEBUG:boto:Token: None 2022-09-12T20:03:25.126 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/ 2022-09-12T20:03:25.127 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk= 2022-09-12T20:03:25.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk=', 'Content-Length': '0'} 2022-09-12T20:03:25.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf106ddc42faba79-00631f908d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.131 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:25.132 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:25.132 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:25.133 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:03:25.133 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:03:25.134 DEBUG:boto:Method: GET 2022-09-12T20:03:25.134 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:03:25.135 DEBUG:boto:Data: 2022-09-12T20:03:25.135 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.136 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.137 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.137 DEBUG:boto:Params: {} 2022-09-12T20:03:25.138 DEBUG:boto:Token: None 2022-09-12T20:03:25.138 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/ 2022-09-12T20:03:25.139 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk= 2022-09-12T20:03:25.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk=', 'Content-Length': '0'} 2022-09-12T20:03:25.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000755de488f917eb3b-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.143 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:25.144 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:25.145 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:25.145 DEBUG:boto:Method: HEAD 2022-09-12T20:03:25.146 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:03:25.146 DEBUG:boto:Data: 2022-09-12T20:03:25.147 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.147 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:25.148 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.148 DEBUG:boto:Params: {} 2022-09-12T20:03:25.149 DEBUG:boto:Token: None 2022-09-12T20:03:25.149 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj33 2022-09-12T20:03:25.150 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dHfUhm6CrdKCJ9eYZlsvzOvqxoo= 2022-09-12T20:03:25.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dHfUhm6CrdKCJ9eYZlsvzOvqxoo=', 'Content-Length': '0'} 2022-09-12T20:03:25.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000566dd86db182390c-00631f908d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.153 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:25.154 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:25.154 DEBUG:boto:Method: HEAD 2022-09-12T20:03:25.155 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:03:25.155 DEBUG:boto:Data: 2022-09-12T20:03:25.156 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.156 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.157 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.157 DEBUG:boto:Params: {} 2022-09-12T20:03:25.158 DEBUG:boto:Token: None 2022-09-12T20:03:25.159 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj33 2022-09-12T20:03:25.159 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dHfUhm6CrdKCJ9eYZlsvzOvqxoo= 2022-09-12T20:03:25.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dHfUhm6CrdKCJ9eYZlsvzOvqxoo=', 'Content-Length': '0'} 2022-09-12T20:03:25.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d08abe27c76cd5f6-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:03:25.163 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:25.164 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:25.164 DEBUG:boto:Method: GET 2022-09-12T20:03:25.165 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:03:25.165 DEBUG:boto:Data: 2022-09-12T20:03:25.166 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.166 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:25.167 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.167 DEBUG:boto:Params: {} 2022-09-12T20:03:25.168 DEBUG:boto:Token: None 2022-09-12T20:03:25.169 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj33 2022-09-12T20:03:25.169 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0JqwJo5Iv9Ctb203NmMyj2ZqrZY= 2022-09-12T20:03:25.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0JqwJo5Iv9Ctb203NmMyj2ZqrZY=', 'Content-Length': '0'} 2022-09-12T20:03:25.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a5a1cf23adcd247-00631f908d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.218 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:25.219 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:25.219 DEBUG:boto:Method: GET 2022-09-12T20:03:25.220 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:03:25.221 DEBUG:boto:Data: 2022-09-12T20:03:25.221 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.222 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.222 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.223 DEBUG:boto:Params: {} 2022-09-12T20:03:25.224 DEBUG:boto:Token: None 2022-09-12T20:03:25.225 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj33 2022-09-12T20:03:25.225 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0JqwJo5Iv9Ctb203NmMyj2ZqrZY= 2022-09-12T20:03:25.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0JqwJo5Iv9Ctb203NmMyj2ZqrZY=', 'Content-Length': '0'} 2022-09-12T20:03:25.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cd4b8344fc0cdf5-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.229 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:25.229 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:25.230 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:03:25.230 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:03:25.231 DEBUG:boto:Method: GET 2022-09-12T20:03:25.232 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:03:25.232 DEBUG:boto:Data: 2022-09-12T20:03:25.233 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.234 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:25.234 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.235 DEBUG:boto:Params: {} 2022-09-12T20:03:25.235 DEBUG:boto:Token: None 2022-09-12T20:03:25.236 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/ 2022-09-12T20:03:25.237 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk= 2022-09-12T20:03:25.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk=', 'Content-Length': '0'} 2022-09-12T20:03:25.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099ba40a73eeed025-00631f908d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.241 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:25.242 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:25.242 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:25.243 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:03:25.243 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:03:25.244 DEBUG:boto:Method: GET 2022-09-12T20:03:25.244 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:03:25.245 DEBUG:boto:Data: 2022-09-12T20:03:25.245 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.246 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.246 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.247 DEBUG:boto:Params: {} 2022-09-12T20:03:25.248 DEBUG:boto:Token: None 2022-09-12T20:03:25.248 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/ 2022-09-12T20:03:25.249 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk= 2022-09-12T20:03:25.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk=', 'Content-Length': '0'} 2022-09-12T20:03:25.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cbe274c5d144337-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.253 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:25.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:03:25.255 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:25.256 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:25.256 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:25.257 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:25.258 DEBUG:boto:Method: GET 2022-09-12T20:03:25.258 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:25.259 DEBUG:boto:Data: 2022-09-12T20:03:25.259 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.260 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:25.260 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.261 DEBUG:boto:Params: {} 2022-09-12T20:03:25.262 DEBUG:boto:Token: None 2022-09-12T20:03:25.262 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:25.263 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hOPLKogvkVuduX32qlZ2MWUOm3o= 2022-09-12T20:03:25.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hOPLKogvkVuduX32qlZ2MWUOm3o=', 'Content-Length': '0'} 2022-09-12T20:03:25.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007efcc5f28e5e5bbe-00631f908d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.266 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:25.267 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:25.268 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:25.268 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:25.269 DEBUG:boto:Method: GET 2022-09-12T20:03:25.269 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:25.270 DEBUG:boto:Data: 2022-09-12T20:03:25.270 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.271 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.271 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.272 DEBUG:boto:Params: {} 2022-09-12T20:03:25.273 DEBUG:boto:Token: None 2022-09-12T20:03:25.273 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:25.274 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hOPLKogvkVuduX32qlZ2MWUOm3o= 2022-09-12T20:03:25.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hOPLKogvkVuduX32qlZ2MWUOm3o=', 'Content-Length': '0'} 2022-09-12T20:03:25.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7b32d952a212546-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.278 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:25.278 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:25.279 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:25.279 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:25.280 DEBUG:boto:Method: HEAD 2022-09-12T20:03:25.280 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:25.281 DEBUG:boto:Data: 2022-09-12T20:03:25.281 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.282 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:25.282 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.283 DEBUG:boto:Params: {} 2022-09-12T20:03:25.283 DEBUG:boto:Token: None 2022-09-12T20:03:25.284 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:25.285 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xUQ5XnRDMf130864UcMAbnxd/O4= 2022-09-12T20:03:25.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xUQ5XnRDMf130864UcMAbnxd/O4=', 'Content-Length': '0'} 2022-09-12T20:03:25.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042d883411dc17ef3-00631f908d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.288 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:25.288 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:25.289 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:25.289 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:25.290 DEBUG:boto:Method: HEAD 2022-09-12T20:03:25.290 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:25.291 DEBUG:boto:Data: 2022-09-12T20:03:25.291 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.292 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.292 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.293 DEBUG:boto:Params: {} 2022-09-12T20:03:25.293 DEBUG:boto:Token: None 2022-09-12T20:03:25.294 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:25.295 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xUQ5XnRDMf130864UcMAbnxd/O4= 2022-09-12T20:03:25.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xUQ5XnRDMf130864UcMAbnxd/O4=', 'Content-Length': '0'} 2022-09-12T20:03:25.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d510bb60a6bbf66e-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:03:25.298 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:25.299 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:25.300 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:25.300 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:25.301 DEBUG:boto:Method: GET 2022-09-12T20:03:25.301 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:25.302 DEBUG:boto:Data: 2022-09-12T20:03:25.302 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.303 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:25.303 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.304 DEBUG:boto:Params: {} 2022-09-12T20:03:25.304 DEBUG:boto:Token: None 2022-09-12T20:03:25.305 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:25.306 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hOPLKogvkVuduX32qlZ2MWUOm3o= 2022-09-12T20:03:25.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hOPLKogvkVuduX32qlZ2MWUOm3o=', 'Content-Length': '0'} 2022-09-12T20:03:25.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cfb253184a1c63e-00631f908d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.309 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:25.310 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:25.310 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:25.311 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:25.311 DEBUG:boto:Method: GET 2022-09-12T20:03:25.312 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:25.312 DEBUG:boto:Data: 2022-09-12T20:03:25.313 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.314 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.314 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.315 DEBUG:boto:Params: {} 2022-09-12T20:03:25.315 DEBUG:boto:Token: None 2022-09-12T20:03:25.316 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:25.317 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hOPLKogvkVuduX32qlZ2MWUOm3o= 2022-09-12T20:03:25.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hOPLKogvkVuduX32qlZ2MWUOm3o=', 'Content-Length': '0'} 2022-09-12T20:03:25.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a661fb923c2ddea-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.320 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:25.321 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:25.321 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:03:25.322 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:03:25.322 DEBUG:boto:Method: GET 2022-09-12T20:03:25.323 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:03:25.324 DEBUG:boto:Data: 2022-09-12T20:03:25.324 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.325 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:25.325 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.326 DEBUG:boto:Params: {} 2022-09-12T20:03:25.326 DEBUG:boto:Token: None 2022-09-12T20:03:25.327 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/ 2022-09-12T20:03:25.327 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk= 2022-09-12T20:03:25.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk=', 'Content-Length': '0'} 2022-09-12T20:03:25.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd63c1330ab6a6f7-00631f908d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.331 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:25.332 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:25.333 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:25.333 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:03:25.334 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:03:25.334 DEBUG:boto:Method: GET 2022-09-12T20:03:25.335 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:03:25.335 DEBUG:boto:Data: 2022-09-12T20:03:25.336 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.336 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.337 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.337 DEBUG:boto:Params: {} 2022-09-12T20:03:25.338 DEBUG:boto:Token: None 2022-09-12T20:03:25.339 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/ 2022-09-12T20:03:25.339 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk= 2022-09-12T20:03:25.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk=', 'Content-Length': '0'} 2022-09-12T20:03:25.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3b1d71b327eaa15-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.344 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:25.344 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:25.345 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:25.345 DEBUG:boto:Method: HEAD 2022-09-12T20:03:25.346 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:03:25.347 DEBUG:boto:Data: 2022-09-12T20:03:25.347 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.348 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:25.348 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.349 DEBUG:boto:Params: {} 2022-09-12T20:03:25.349 DEBUG:boto:Token: None 2022-09-12T20:03:25.350 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj44 2022-09-12T20:03:25.350 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fz5hnYQ7+eTWNhP1JlDn0k4ON3o= 2022-09-12T20:03:25.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fz5hnYQ7+eTWNhP1JlDn0k4ON3o=', 'Content-Length': '0'} 2022-09-12T20:03:25.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033a289241929c8aa-00631f908d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.353 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:25.354 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:25.355 DEBUG:boto:Method: HEAD 2022-09-12T20:03:25.355 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:03:25.356 DEBUG:boto:Data: 2022-09-12T20:03:25.356 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.357 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.357 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.358 DEBUG:boto:Params: {} 2022-09-12T20:03:25.358 DEBUG:boto:Token: None 2022-09-12T20:03:25.359 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj44 2022-09-12T20:03:25.359 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fz5hnYQ7+eTWNhP1JlDn0k4ON3o= 2022-09-12T20:03:25.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fz5hnYQ7+eTWNhP1JlDn0k4ON3o=', 'Content-Length': '0'} 2022-09-12T20:03:25.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c25f84a590806034-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:03:25.363 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:25.364 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:25.364 DEBUG:boto:Method: GET 2022-09-12T20:03:25.365 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:03:25.365 DEBUG:boto:Data: 2022-09-12T20:03:25.366 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.367 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:25.367 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.368 DEBUG:boto:Params: {} 2022-09-12T20:03:25.368 DEBUG:boto:Token: None 2022-09-12T20:03:25.369 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj44 2022-09-12T20:03:25.369 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b0t8+xfBrIGr6P1HQA2EzuRzBNQ= 2022-09-12T20:03:25.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b0t8+xfBrIGr6P1HQA2EzuRzBNQ=', 'Content-Length': '0'} 2022-09-12T20:03:25.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003580fa99f785f8b1-00631f908d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.373 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:25.374 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:25.374 DEBUG:boto:Method: GET 2022-09-12T20:03:25.375 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:03:25.375 DEBUG:boto:Data: 2022-09-12T20:03:25.376 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.376 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.377 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.377 DEBUG:boto:Params: {} 2022-09-12T20:03:25.378 DEBUG:boto:Token: None 2022-09-12T20:03:25.379 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj44 2022-09-12T20:03:25.379 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b0t8+xfBrIGr6P1HQA2EzuRzBNQ= 2022-09-12T20:03:25.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b0t8+xfBrIGr6P1HQA2EzuRzBNQ=', 'Content-Length': '0'} 2022-09-12T20:03:25.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076deb586d3465032-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.383 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:25.383 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:25.384 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:03:25.384 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:03:25.385 DEBUG:boto:Method: GET 2022-09-12T20:03:25.385 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:03:25.386 DEBUG:boto:Data: 2022-09-12T20:03:25.386 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.387 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:25.387 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.388 DEBUG:boto:Params: {} 2022-09-12T20:03:25.389 DEBUG:boto:Token: None 2022-09-12T20:03:25.389 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/ 2022-09-12T20:03:25.390 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk= 2022-09-12T20:03:25.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk=', 'Content-Length': '0'} 2022-09-12T20:03:25.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e97f3dd39b84e90-00631f908d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.393 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:25.394 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:25.395 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:25.395 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:03:25.396 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:03:25.396 DEBUG:boto:Method: GET 2022-09-12T20:03:25.397 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:03:25.397 DEBUG:boto:Data: 2022-09-12T20:03:25.398 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.399 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.399 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.400 DEBUG:boto:Params: {} 2022-09-12T20:03:25.400 DEBUG:boto:Token: None 2022-09-12T20:03:25.401 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/ 2022-09-12T20:03:25.401 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk= 2022-09-12T20:03:25.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk=', 'Content-Length': '0'} 2022-09-12T20:03:25.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc72a7a5c4f18889-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.446 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:25.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:03:25.448 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:25.448 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:25.449 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:25.449 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:25.450 DEBUG:boto:Method: GET 2022-09-12T20:03:25.451 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:25.451 DEBUG:boto:Data: 2022-09-12T20:03:25.452 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.452 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:25.453 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.453 DEBUG:boto:Params: {} 2022-09-12T20:03:25.454 DEBUG:boto:Token: None 2022-09-12T20:03:25.455 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:25.455 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OPO8j6Tf0OE9RvbE5cF1Ka03ovs= 2022-09-12T20:03:25.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OPO8j6Tf0OE9RvbE5cF1Ka03ovs=', 'Content-Length': '0'} 2022-09-12T20:03:25.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047fa63946f5716da-00631f908d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.459 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:25.460 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:25.460 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:25.461 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:25.461 DEBUG:boto:Method: GET 2022-09-12T20:03:25.462 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:25.463 DEBUG:boto:Data: 2022-09-12T20:03:25.463 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.464 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.464 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.465 DEBUG:boto:Params: {} 2022-09-12T20:03:25.465 DEBUG:boto:Token: None 2022-09-12T20:03:25.466 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:25.467 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OPO8j6Tf0OE9RvbE5cF1Ka03ovs= 2022-09-12T20:03:25.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OPO8j6Tf0OE9RvbE5cF1Ka03ovs=', 'Content-Length': '0'} 2022-09-12T20:03:25.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060079ee6a8858cdf-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.471 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:25.471 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:25.472 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:25.472 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:25.473 DEBUG:boto:Method: HEAD 2022-09-12T20:03:25.473 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:25.474 DEBUG:boto:Data: 2022-09-12T20:03:25.474 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.475 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:25.475 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.476 DEBUG:boto:Params: {} 2022-09-12T20:03:25.476 DEBUG:boto:Token: None 2022-09-12T20:03:25.477 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:25.477 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:muT0ZAL/5VsMrsqf9jG2tD3jQdQ= 2022-09-12T20:03:25.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:muT0ZAL/5VsMrsqf9jG2tD3jQdQ=', 'Content-Length': '0'} 2022-09-12T20:03:25.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f1ef5bc36b39418-00631f908d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.481 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:25.481 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:25.482 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:25.482 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:25.483 DEBUG:boto:Method: HEAD 2022-09-12T20:03:25.483 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:25.484 DEBUG:boto:Data: 2022-09-12T20:03:25.484 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.485 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.485 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.486 DEBUG:boto:Params: {} 2022-09-12T20:03:25.486 DEBUG:boto:Token: None 2022-09-12T20:03:25.487 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:25.488 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:muT0ZAL/5VsMrsqf9jG2tD3jQdQ= 2022-09-12T20:03:25.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:muT0ZAL/5VsMrsqf9jG2tD3jQdQ=', 'Content-Length': '0'} 2022-09-12T20:03:25.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cfa3f990ee25ea8-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:03:25.491 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:25.492 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:25.493 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:25.493 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:25.494 DEBUG:boto:Method: GET 2022-09-12T20:03:25.494 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:25.495 DEBUG:boto:Data: 2022-09-12T20:03:25.495 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.496 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:25.496 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.497 DEBUG:boto:Params: {} 2022-09-12T20:03:25.497 DEBUG:boto:Token: None 2022-09-12T20:03:25.498 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:25.499 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OPO8j6Tf0OE9RvbE5cF1Ka03ovs= 2022-09-12T20:03:25.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OPO8j6Tf0OE9RvbE5cF1Ka03ovs=', 'Content-Length': '0'} 2022-09-12T20:03:25.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f01b2d39d0038930-00631f908d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.542 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:25.543 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:25.544 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:25.544 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:25.545 DEBUG:boto:Method: GET 2022-09-12T20:03:25.546 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:25.547 DEBUG:boto:Data: 2022-09-12T20:03:25.547 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.548 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.549 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.549 DEBUG:boto:Params: {} 2022-09-12T20:03:25.550 DEBUG:boto:Token: None 2022-09-12T20:03:25.550 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:25.551 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OPO8j6Tf0OE9RvbE5cF1Ka03ovs= 2022-09-12T20:03:25.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OPO8j6Tf0OE9RvbE5cF1Ka03ovs=', 'Content-Length': '0'} 2022-09-12T20:03:25.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d4da72ad4fe0532-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.555 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:25.556 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:25.556 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:03:25.557 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:03:25.557 DEBUG:boto:Method: GET 2022-09-12T20:03:25.558 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:03:25.558 DEBUG:boto:Data: 2022-09-12T20:03:25.559 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.560 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:25.560 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.561 DEBUG:boto:Params: {} 2022-09-12T20:03:25.561 DEBUG:boto:Token: None 2022-09-12T20:03:25.562 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/ 2022-09-12T20:03:25.563 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk= 2022-09-12T20:03:25.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk=', 'Content-Length': '0'} 2022-09-12T20:03:25.567 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048fb588db6f952db-00631f908d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.568 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:25.569 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:25.569 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:25.570 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:03:25.571 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:03:25.571 DEBUG:boto:Method: GET 2022-09-12T20:03:25.572 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:03:25.573 DEBUG:boto:Data: 2022-09-12T20:03:25.574 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.574 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.575 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.575 DEBUG:boto:Params: {} 2022-09-12T20:03:25.576 DEBUG:boto:Token: None 2022-09-12T20:03:25.577 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/ 2022-09-12T20:03:25.577 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk= 2022-09-12T20:03:25.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk=', 'Content-Length': '0'} 2022-09-12T20:03:25.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef2d40fa4b58bf61-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.622 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:25.623 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:25.624 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:25.624 DEBUG:boto:Method: HEAD 2022-09-12T20:03:25.625 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:03:25.626 DEBUG:boto:Data: 2022-09-12T20:03:25.627 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.627 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:25.628 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.628 DEBUG:boto:Params: {} 2022-09-12T20:03:25.629 DEBUG:boto:Token: None 2022-09-12T20:03:25.630 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj55 2022-09-12T20:03:25.630 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:owMFMH5LUPYTCyq6sFIP46dDAUI= 2022-09-12T20:03:25.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:owMFMH5LUPYTCyq6sFIP46dDAUI=', 'Content-Length': '0'} 2022-09-12T20:03:25.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a79a55a36c7224de-00631f908d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.634 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:25.635 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:25.635 DEBUG:boto:Method: HEAD 2022-09-12T20:03:25.636 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:03:25.636 DEBUG:boto:Data: 2022-09-12T20:03:25.637 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.637 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.638 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.638 DEBUG:boto:Params: {} 2022-09-12T20:03:25.639 DEBUG:boto:Token: None 2022-09-12T20:03:25.639 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj55 2022-09-12T20:03:25.640 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:owMFMH5LUPYTCyq6sFIP46dDAUI= 2022-09-12T20:03:25.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:owMFMH5LUPYTCyq6sFIP46dDAUI=', 'Content-Length': '0'} 2022-09-12T20:03:25.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd5d09d9f8f2ca27-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:03:25.644 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:25.645 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:25.646 DEBUG:boto:Method: GET 2022-09-12T20:03:25.646 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:03:25.647 DEBUG:boto:Data: 2022-09-12T20:03:25.647 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.648 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:25.648 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.649 DEBUG:boto:Params: {} 2022-09-12T20:03:25.650 DEBUG:boto:Token: None 2022-09-12T20:03:25.651 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj55 2022-09-12T20:03:25.651 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LkO//AW5fVKwWAzoglQCeEkYW88= 2022-09-12T20:03:25.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LkO//AW5fVKwWAzoglQCeEkYW88=', 'Content-Length': '0'} 2022-09-12T20:03:25.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b40f27bf4dbb41bd-00631f908d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.656 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:25.656 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:25.657 DEBUG:boto:Method: GET 2022-09-12T20:03:25.658 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:03:25.658 DEBUG:boto:Data: 2022-09-12T20:03:25.659 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.659 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.660 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.660 DEBUG:boto:Params: {} 2022-09-12T20:03:25.661 DEBUG:boto:Token: None 2022-09-12T20:03:25.662 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj55 2022-09-12T20:03:25.662 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LkO//AW5fVKwWAzoglQCeEkYW88= 2022-09-12T20:03:25.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LkO//AW5fVKwWAzoglQCeEkYW88=', 'Content-Length': '0'} 2022-09-12T20:03:25.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab25aeaa414837e1-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.667 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:25.667 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:25.668 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:03:25.669 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:03:25.669 DEBUG:boto:Method: GET 2022-09-12T20:03:25.670 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:03:25.670 DEBUG:boto:Data: 2022-09-12T20:03:25.671 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.672 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:25.672 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.673 DEBUG:boto:Params: {} 2022-09-12T20:03:25.673 DEBUG:boto:Token: None 2022-09-12T20:03:25.674 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/ 2022-09-12T20:03:25.675 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk= 2022-09-12T20:03:25.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk=', 'Content-Length': '0'} 2022-09-12T20:03:25.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc339e12d0d33f52-00631f908d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.679 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:25.679 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:25.680 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:25.681 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:03:25.681 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:03:25.682 DEBUG:boto:Method: GET 2022-09-12T20:03:25.683 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:03:25.684 DEBUG:boto:Data: 2022-09-12T20:03:25.684 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.685 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.686 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.686 DEBUG:boto:Params: {} 2022-09-12T20:03:25.687 DEBUG:boto:Token: None 2022-09-12T20:03:25.688 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/ 2022-09-12T20:03:25.689 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk= 2022-09-12T20:03:25.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk=', 'Content-Length': '0'} 2022-09-12T20:03:25.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003232d2a5047c097b-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.734 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:25.735 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-75 zones={test-zone2, test-zone1} 2022-09-12T20:03:25.736 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-75 zones={test-zone3, test-zone1} 2022-09-12T20:03:25.736 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:25.737 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:25.737 DEBUG:boto:Method: HEAD 2022-09-12T20:03:25.738 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:03:25.738 DEBUG:boto:Data: 2022-09-12T20:03:25.739 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.739 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:25.740 DEBUG:boto:Port: 8001 2022-09-12T20:03:25.740 DEBUG:boto:Params: {} 2022-09-12T20:03:25.741 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T20:03:25.742 DEBUG:boto:Token: None 2022-09-12T20:03:25.742 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/ 2022-09-12T20:03:25.743 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:C2JuogwgqLafNUscEofiye7bkdU= 2022-09-12T20:03:25.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:C2JuogwgqLafNUscEofiye7bkdU=', 'Content-Length': '0'} 2022-09-12T20:03:25.747 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b88ba570ce218b1-00631f908d-1826-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.748 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:25.748 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:25.749 DEBUG:boto:Method: HEAD 2022-09-12T20:03:25.750 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:03:25.751 DEBUG:boto:Data: 2022-09-12T20:03:25.752 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.752 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.753 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.753 DEBUG:boto:Params: {} 2022-09-12T20:03:25.754 DEBUG:boto:Token: None 2022-09-12T20:03:25.754 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/ 2022-09-12T20:03:25.755 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:C2JuogwgqLafNUscEofiye7bkdU= 2022-09-12T20:03:25.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:C2JuogwgqLafNUscEofiye7bkdU=', 'Content-Length': '0'} 2022-09-12T20:03:25.759 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af665f249649ef27-00631f908d-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.759 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T20:03:25.760 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:25.760 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:25.761 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:25.761 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:25.762 DEBUG:boto:Method: GET 2022-09-12T20:03:25.763 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:25.764 DEBUG:boto:Data: 2022-09-12T20:03:25.765 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.766 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:25.767 DEBUG:boto:Port: 8001 2022-09-12T20:03:25.768 DEBUG:boto:Params: {} 2022-09-12T20:03:25.769 DEBUG:boto:Token: None 2022-09-12T20:03:25.770 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/?versions 2022-09-12T20:03:25.771 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BsieWgJzFCOkfEEAYNXcZ8iYVMY= 2022-09-12T20:03:25.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BsieWgJzFCOkfEEAYNXcZ8iYVMY=', 'Content-Length': '0'} 2022-09-12T20:03:25.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddd52ba034f531ed-00631f908d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.777 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:25.778 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:03:25.778 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:03:25.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:03:25.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:03:25.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:03:25.781 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T20:03:25.781 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:25.782 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:25.783 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:25.783 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:25.784 DEBUG:boto:Method: GET 2022-09-12T20:03:25.784 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:25.785 DEBUG:boto:Data: 2022-09-12T20:03:25.786 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.787 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.787 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.788 DEBUG:boto:Params: {} 2022-09-12T20:03:25.788 DEBUG:boto:Token: None 2022-09-12T20:03:25.789 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/?versions 2022-09-12T20:03:25.790 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BsieWgJzFCOkfEEAYNXcZ8iYVMY= 2022-09-12T20:03:25.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BsieWgJzFCOkfEEAYNXcZ8iYVMY=', 'Content-Length': '0'} 2022-09-12T20:03:25.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d46b594ed5bea836-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.801 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:25.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:03:25.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:03:25.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:03:25.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:03:25.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:03:25.805 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:25.805 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:25.806 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:25.807 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:25.808 DEBUG:boto:Method: GET 2022-09-12T20:03:25.808 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:25.809 DEBUG:boto:Data: 2022-09-12T20:03:25.810 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.811 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:25.812 DEBUG:boto:Port: 8001 2022-09-12T20:03:25.813 DEBUG:boto:Params: {} 2022-09-12T20:03:25.814 DEBUG:boto:Token: None 2022-09-12T20:03:25.815 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/?versions 2022-09-12T20:03:25.816 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BsieWgJzFCOkfEEAYNXcZ8iYVMY= 2022-09-12T20:03:25.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BsieWgJzFCOkfEEAYNXcZ8iYVMY=', 'Content-Length': '0'} 2022-09-12T20:03:25.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de4ff3b32cd4c0e9-00631f908d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.821 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:25.822 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:25.823 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:25.824 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:25.825 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:25.826 DEBUG:boto:Method: GET 2022-09-12T20:03:25.827 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:25.827 DEBUG:boto:Data: 2022-09-12T20:03:25.828 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.829 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.829 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.830 DEBUG:boto:Params: {} 2022-09-12T20:03:25.831 DEBUG:boto:Token: None 2022-09-12T20:03:25.831 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/?versions 2022-09-12T20:03:25.832 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BsieWgJzFCOkfEEAYNXcZ8iYVMY= 2022-09-12T20:03:25.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BsieWgJzFCOkfEEAYNXcZ8iYVMY=', 'Content-Length': '0'} 2022-09-12T20:03:25.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050c9c361994eb825-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.836 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:25.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:03:25.838 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:25.839 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:25.840 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:25.841 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:25.842 DEBUG:boto:Method: GET 2022-09-12T20:03:25.842 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:25.843 DEBUG:boto:Data: 2022-09-12T20:03:25.844 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.845 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:25.846 DEBUG:boto:Port: 8001 2022-09-12T20:03:25.847 DEBUG:boto:Params: {} 2022-09-12T20:03:25.848 DEBUG:boto:Token: None 2022-09-12T20:03:25.849 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:25.850 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:y7R4VaGXDLjkD5SE7krGtVaFkGM= 2022-09-12T20:03:25.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:y7R4VaGXDLjkD5SE7krGtVaFkGM=', 'Content-Length': '0'} 2022-09-12T20:03:25.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df4daa874aba1735-00631f908d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.854 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:25.854 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:25.855 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:25.855 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:25.856 DEBUG:boto:Method: GET 2022-09-12T20:03:25.857 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:25.857 DEBUG:boto:Data: 2022-09-12T20:03:25.858 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.858 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.859 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.860 DEBUG:boto:Params: {} 2022-09-12T20:03:25.860 DEBUG:boto:Token: None 2022-09-12T20:03:25.861 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:25.861 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:y7R4VaGXDLjkD5SE7krGtVaFkGM= 2022-09-12T20:03:25.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:y7R4VaGXDLjkD5SE7krGtVaFkGM=', 'Content-Length': '0'} 2022-09-12T20:03:25.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098767be9b3aaf792-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.906 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:25.907 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:25.907 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:25.908 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:25.908 DEBUG:boto:Method: HEAD 2022-09-12T20:03:25.909 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:25.909 DEBUG:boto:Data: 2022-09-12T20:03:25.910 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.910 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:25.911 DEBUG:boto:Port: 8001 2022-09-12T20:03:25.911 DEBUG:boto:Params: {} 2022-09-12T20:03:25.912 DEBUG:boto:Token: None 2022-09-12T20:03:25.913 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:25.913 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mJ60GP1ZD+zZfSsGwvy9pqHwo+s= 2022-09-12T20:03:25.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mJ60GP1ZD+zZfSsGwvy9pqHwo+s=', 'Content-Length': '0'} 2022-09-12T20:03:25.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ece1cf040638a845-00631f908d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.916 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:25.917 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:25.917 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:25.918 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:25.918 DEBUG:boto:Method: HEAD 2022-09-12T20:03:25.919 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:25.919 DEBUG:boto:Data: 2022-09-12T20:03:25.920 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.921 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.921 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.922 DEBUG:boto:Params: {} 2022-09-12T20:03:25.922 DEBUG:boto:Token: None 2022-09-12T20:03:25.923 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:25.923 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mJ60GP1ZD+zZfSsGwvy9pqHwo+s= 2022-09-12T20:03:25.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mJ60GP1ZD+zZfSsGwvy9pqHwo+s=', 'Content-Length': '0'} 2022-09-12T20:03:25.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccebb9045bd42b7c-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:03:25.927 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:25.928 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:25.929 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:25.929 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:25.930 DEBUG:boto:Method: GET 2022-09-12T20:03:25.930 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:25.931 DEBUG:boto:Data: 2022-09-12T20:03:25.932 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.933 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:25.933 DEBUG:boto:Port: 8001 2022-09-12T20:03:25.934 DEBUG:boto:Params: {} 2022-09-12T20:03:25.935 DEBUG:boto:Token: None 2022-09-12T20:03:25.936 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:25.937 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:y7R4VaGXDLjkD5SE7krGtVaFkGM= 2022-09-12T20:03:25.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:y7R4VaGXDLjkD5SE7krGtVaFkGM=', 'Content-Length': '0'} 2022-09-12T20:03:25.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dd4c96282920b68-00631f908d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.941 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:25.942 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:25.942 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:25.943 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:25.944 DEBUG:boto:Method: GET 2022-09-12T20:03:25.944 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:25.945 DEBUG:boto:Data: 2022-09-12T20:03:25.945 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.946 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.946 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.947 DEBUG:boto:Params: {} 2022-09-12T20:03:25.947 DEBUG:boto:Token: None 2022-09-12T20:03:25.948 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:25.948 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:y7R4VaGXDLjkD5SE7krGtVaFkGM= 2022-09-12T20:03:25.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:y7R4VaGXDLjkD5SE7krGtVaFkGM=', 'Content-Length': '0'} 2022-09-12T20:03:25.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3884a25635a08ce-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.952 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:25.953 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:25.953 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:03:25.954 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:03:25.954 DEBUG:boto:Method: GET 2022-09-12T20:03:25.955 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:03:25.955 DEBUG:boto:Data: 2022-09-12T20:03:25.956 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.956 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:25.957 DEBUG:boto:Port: 8001 2022-09-12T20:03:25.957 DEBUG:boto:Params: {} 2022-09-12T20:03:25.958 DEBUG:boto:Token: None 2022-09-12T20:03:25.959 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/ 2022-09-12T20:03:25.959 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk= 2022-09-12T20:03:25.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk=', 'Content-Length': '0'} 2022-09-12T20:03:25.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7e67e8b1b647f00-00631f908d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.963 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:25.965 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:25.966 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:25.967 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:03:25.968 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:03:25.969 DEBUG:boto:Method: GET 2022-09-12T20:03:25.969 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:03:25.971 DEBUG:boto:Data: 2022-09-12T20:03:25.971 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.972 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.972 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.973 DEBUG:boto:Params: {} 2022-09-12T20:03:25.973 DEBUG:boto:Token: None 2022-09-12T20:03:25.974 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/ 2022-09-12T20:03:25.975 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk= 2022-09-12T20:03:25.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:03wN5bmaiYBoXWCwWWgt6+8JnSk=', 'Content-Length': '0'} 2022-09-12T20:03:25.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e1ba4f35968b69b-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.978 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:25.979 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:25.980 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:25.980 DEBUG:boto:Method: HEAD 2022-09-12T20:03:25.981 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:03:25.981 DEBUG:boto:Data: 2022-09-12T20:03:25.982 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.982 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:25.983 DEBUG:boto:Port: 8001 2022-09-12T20:03:25.983 DEBUG:boto:Params: {} 2022-09-12T20:03:25.984 DEBUG:boto:Token: None 2022-09-12T20:03:25.985 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj11 2022-09-12T20:03:25.985 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jtdaI3ZRZws9iFi1sqmN7daZ2p4= 2022-09-12T20:03:25.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jtdaI3ZRZws9iFi1sqmN7daZ2p4=', 'Content-Length': '0'} 2022-09-12T20:03:25.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008231078db90be87c-00631f908d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.989 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:25.989 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:25.990 DEBUG:boto:Method: HEAD 2022-09-12T20:03:25.990 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:03:25.991 DEBUG:boto:Data: 2022-09-12T20:03:25.991 DEBUG:boto:Headers: {} 2022-09-12T20:03:25.992 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:25.993 DEBUG:boto:Port: 8000 2022-09-12T20:03:25.993 DEBUG:boto:Params: {} 2022-09-12T20:03:25.994 DEBUG:boto:Token: None 2022-09-12T20:03:25.994 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:25 GMT /jdpiak-75/obj11 2022-09-12T20:03:25.995 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jtdaI3ZRZws9iFi1sqmN7daZ2p4= 2022-09-12T20:03:25.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jtdaI3ZRZws9iFi1sqmN7daZ2p4=', 'Content-Length': '0'} 2022-09-12T20:03:25.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0b8b8883bd6457b-00631f908d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:25.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:03:25.999 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:26.000 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:26.000 DEBUG:boto:Method: GET 2022-09-12T20:03:26.001 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:03:26.001 DEBUG:boto:Data: 2022-09-12T20:03:26.002 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.002 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.003 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.004 DEBUG:boto:Params: {} 2022-09-12T20:03:26.004 DEBUG:boto:Token: None 2022-09-12T20:03:26.005 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj11 2022-09-12T20:03:26.006 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A4oE5COYVQRt60/bWA66oDj/nkI= 2022-09-12T20:03:26.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A4oE5COYVQRt60/bWA66oDj/nkI=', 'Content-Length': '0'} 2022-09-12T20:03:26.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcc085fa9438fa5b-00631f908e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.010 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:26.010 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:26.011 DEBUG:boto:Method: GET 2022-09-12T20:03:26.011 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:03:26.012 DEBUG:boto:Data: 2022-09-12T20:03:26.013 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.013 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.014 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.014 DEBUG:boto:Params: {} 2022-09-12T20:03:26.015 DEBUG:boto:Token: None 2022-09-12T20:03:26.016 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj11 2022-09-12T20:03:26.016 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A4oE5COYVQRt60/bWA66oDj/nkI= 2022-09-12T20:03:26.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A4oE5COYVQRt60/bWA66oDj/nkI=', 'Content-Length': '0'} 2022-09-12T20:03:26.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caabc66efc358976-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.066 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:26.067 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:26.068 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:03:26.068 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:03:26.069 DEBUG:boto:Method: GET 2022-09-12T20:03:26.069 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:03:26.070 DEBUG:boto:Data: 2022-09-12T20:03:26.070 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.071 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.071 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.072 DEBUG:boto:Params: {} 2022-09-12T20:03:26.073 DEBUG:boto:Token: None 2022-09-12T20:03:26.073 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/ 2022-09-12T20:03:26.074 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc= 2022-09-12T20:03:26.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc=', 'Content-Length': '0'} 2022-09-12T20:03:26.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000807186a6b8ba6eeb-00631f908e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.079 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:26.080 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:26.080 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:26.081 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:03:26.081 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:03:26.082 DEBUG:boto:Method: GET 2022-09-12T20:03:26.082 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:03:26.083 DEBUG:boto:Data: 2022-09-12T20:03:26.083 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.084 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.085 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.085 DEBUG:boto:Params: {} 2022-09-12T20:03:26.086 DEBUG:boto:Token: None 2022-09-12T20:03:26.087 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/ 2022-09-12T20:03:26.087 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc= 2022-09-12T20:03:26.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc=', 'Content-Length': '0'} 2022-09-12T20:03:26.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d756c96ba8318ed-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.092 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:26.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:03:26.093 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:26.094 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:26.095 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:26.095 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:26.096 DEBUG:boto:Method: GET 2022-09-12T20:03:26.096 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:26.097 DEBUG:boto:Data: 2022-09-12T20:03:26.097 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.098 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.099 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.100 DEBUG:boto:Params: {} 2022-09-12T20:03:26.100 DEBUG:boto:Token: None 2022-09-12T20:03:26.101 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:26.101 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S4v0BJ6fd0xL6ZzKc5NVbZ5xrFE= 2022-09-12T20:03:26.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S4v0BJ6fd0xL6ZzKc5NVbZ5xrFE=', 'Content-Length': '0'} 2022-09-12T20:03:26.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c665985d56633fc-00631f908e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.105 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:26.105 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:26.106 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:26.107 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:26.107 DEBUG:boto:Method: GET 2022-09-12T20:03:26.108 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:26.108 DEBUG:boto:Data: 2022-09-12T20:03:26.109 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.109 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.110 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.111 DEBUG:boto:Params: {} 2022-09-12T20:03:26.111 DEBUG:boto:Token: None 2022-09-12T20:03:26.112 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:26.112 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S4v0BJ6fd0xL6ZzKc5NVbZ5xrFE= 2022-09-12T20:03:26.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S4v0BJ6fd0xL6ZzKc5NVbZ5xrFE=', 'Content-Length': '0'} 2022-09-12T20:03:26.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000692fc72cdabd0c7f-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.116 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:26.116 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:26.117 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:26.117 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:26.118 DEBUG:boto:Method: HEAD 2022-09-12T20:03:26.119 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:26.119 DEBUG:boto:Data: 2022-09-12T20:03:26.120 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.120 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.121 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.121 DEBUG:boto:Params: {} 2022-09-12T20:03:26.122 DEBUG:boto:Token: None 2022-09-12T20:03:26.123 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:26.123 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fdj0nIBAlo0ZShEs6A9aDG4CpIg= 2022-09-12T20:03:26.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fdj0nIBAlo0ZShEs6A9aDG4CpIg=', 'Content-Length': '0'} 2022-09-12T20:03:26.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3641c05d54a950e-00631f908e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.127 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:26.127 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:26.128 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:26.129 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:26.129 DEBUG:boto:Method: HEAD 2022-09-12T20:03:26.130 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:26.130 DEBUG:boto:Data: 2022-09-12T20:03:26.131 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.131 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.132 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.132 DEBUG:boto:Params: {} 2022-09-12T20:03:26.133 DEBUG:boto:Token: None 2022-09-12T20:03:26.134 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:26.134 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fdj0nIBAlo0ZShEs6A9aDG4CpIg= 2022-09-12T20:03:26.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fdj0nIBAlo0ZShEs6A9aDG4CpIg=', 'Content-Length': '0'} 2022-09-12T20:03:26.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8e41e5b8d7c5360-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:03:26.138 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:26.139 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:26.140 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:26.140 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:26.141 DEBUG:boto:Method: GET 2022-09-12T20:03:26.141 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:26.142 DEBUG:boto:Data: 2022-09-12T20:03:26.143 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.143 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.144 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.144 DEBUG:boto:Params: {} 2022-09-12T20:03:26.145 DEBUG:boto:Token: None 2022-09-12T20:03:26.145 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:26.146 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S4v0BJ6fd0xL6ZzKc5NVbZ5xrFE= 2022-09-12T20:03:26.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S4v0BJ6fd0xL6ZzKc5NVbZ5xrFE=', 'Content-Length': '0'} 2022-09-12T20:03:26.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb221feb0e354811-00631f908e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.194 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:26.195 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:26.195 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:26.196 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:26.197 DEBUG:boto:Method: GET 2022-09-12T20:03:26.197 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:26.198 DEBUG:boto:Data: 2022-09-12T20:03:26.198 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.199 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.199 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.200 DEBUG:boto:Params: {} 2022-09-12T20:03:26.200 DEBUG:boto:Token: None 2022-09-12T20:03:26.201 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:26.202 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S4v0BJ6fd0xL6ZzKc5NVbZ5xrFE= 2022-09-12T20:03:26.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S4v0BJ6fd0xL6ZzKc5NVbZ5xrFE=', 'Content-Length': '0'} 2022-09-12T20:03:26.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006234112807d61a89-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.205 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:26.206 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:26.207 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:03:26.207 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:03:26.208 DEBUG:boto:Method: GET 2022-09-12T20:03:26.208 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:03:26.209 DEBUG:boto:Data: 2022-09-12T20:03:26.209 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.210 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.210 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.211 DEBUG:boto:Params: {} 2022-09-12T20:03:26.211 DEBUG:boto:Token: None 2022-09-12T20:03:26.212 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/ 2022-09-12T20:03:26.213 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc= 2022-09-12T20:03:26.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc=', 'Content-Length': '0'} 2022-09-12T20:03:26.216 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c1367478b76d635-00631f908e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.217 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:26.218 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:26.218 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:26.219 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:03:26.219 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:03:26.220 DEBUG:boto:Method: GET 2022-09-12T20:03:26.220 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:03:26.221 DEBUG:boto:Data: 2022-09-12T20:03:26.222 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.222 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.223 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.223 DEBUG:boto:Params: {} 2022-09-12T20:03:26.224 DEBUG:boto:Token: None 2022-09-12T20:03:26.225 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/ 2022-09-12T20:03:26.225 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc= 2022-09-12T20:03:26.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc=', 'Content-Length': '0'} 2022-09-12T20:03:26.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088bc9ae709cb5392-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.229 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:26.230 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:26.231 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:26.231 DEBUG:boto:Method: HEAD 2022-09-12T20:03:26.232 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:03:26.232 DEBUG:boto:Data: 2022-09-12T20:03:26.233 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.233 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.234 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.234 DEBUG:boto:Params: {} 2022-09-12T20:03:26.235 DEBUG:boto:Token: None 2022-09-12T20:03:26.235 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj22 2022-09-12T20:03:26.236 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cqAGp4JXAK3Ut+7YuyEzCrfdU0o= 2022-09-12T20:03:26.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cqAGp4JXAK3Ut+7YuyEzCrfdU0o=', 'Content-Length': '0'} 2022-09-12T20:03:26.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012b8b3c96d646999-00631f908e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.239 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:26.240 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:26.241 DEBUG:boto:Method: HEAD 2022-09-12T20:03:26.241 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:03:26.242 DEBUG:boto:Data: 2022-09-12T20:03:26.242 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.243 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.243 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.244 DEBUG:boto:Params: {} 2022-09-12T20:03:26.244 DEBUG:boto:Token: None 2022-09-12T20:03:26.245 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj22 2022-09-12T20:03:26.245 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cqAGp4JXAK3Ut+7YuyEzCrfdU0o= 2022-09-12T20:03:26.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cqAGp4JXAK3Ut+7YuyEzCrfdU0o=', 'Content-Length': '0'} 2022-09-12T20:03:26.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000472c17a6a2da593a-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:03:26.249 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:26.250 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:26.250 DEBUG:boto:Method: GET 2022-09-12T20:03:26.251 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:03:26.251 DEBUG:boto:Data: 2022-09-12T20:03:26.252 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.253 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.253 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.254 DEBUG:boto:Params: {} 2022-09-12T20:03:26.254 DEBUG:boto:Token: None 2022-09-12T20:03:26.255 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj22 2022-09-12T20:03:26.255 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TS91hehG2qbikk2e3hWYuhjJcrM= 2022-09-12T20:03:26.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TS91hehG2qbikk2e3hWYuhjJcrM=', 'Content-Length': '0'} 2022-09-12T20:03:26.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc504ae0c58a3a83-00631f908e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.259 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:26.259 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:26.260 DEBUG:boto:Method: GET 2022-09-12T20:03:26.260 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:03:26.261 DEBUG:boto:Data: 2022-09-12T20:03:26.261 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.262 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.263 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.263 DEBUG:boto:Params: {} 2022-09-12T20:03:26.264 DEBUG:boto:Token: None 2022-09-12T20:03:26.264 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj22 2022-09-12T20:03:26.265 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TS91hehG2qbikk2e3hWYuhjJcrM= 2022-09-12T20:03:26.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TS91hehG2qbikk2e3hWYuhjJcrM=', 'Content-Length': '0'} 2022-09-12T20:03:26.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b971a5fcc2cd3d8a-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.268 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:26.269 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:26.269 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:03:26.270 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:03:26.270 DEBUG:boto:Method: GET 2022-09-12T20:03:26.271 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:03:26.272 DEBUG:boto:Data: 2022-09-12T20:03:26.272 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.273 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.273 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.274 DEBUG:boto:Params: {} 2022-09-12T20:03:26.274 DEBUG:boto:Token: None 2022-09-12T20:03:26.275 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/ 2022-09-12T20:03:26.276 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc= 2022-09-12T20:03:26.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc=', 'Content-Length': '0'} 2022-09-12T20:03:26.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b03085e3c29d8b5-00631f908e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.322 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:26.323 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:26.324 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:26.324 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:03:26.325 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:03:26.325 DEBUG:boto:Method: GET 2022-09-12T20:03:26.326 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:03:26.326 DEBUG:boto:Data: 2022-09-12T20:03:26.327 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.327 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.328 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.328 DEBUG:boto:Params: {} 2022-09-12T20:03:26.329 DEBUG:boto:Token: None 2022-09-12T20:03:26.329 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/ 2022-09-12T20:03:26.330 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc= 2022-09-12T20:03:26.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc=', 'Content-Length': '0'} 2022-09-12T20:03:26.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbd4c9182e6d7f08-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.334 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:26.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:03:26.336 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:26.336 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:26.337 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:26.337 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:26.338 DEBUG:boto:Method: GET 2022-09-12T20:03:26.339 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:26.339 DEBUG:boto:Data: 2022-09-12T20:03:26.340 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.340 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.341 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.341 DEBUG:boto:Params: {} 2022-09-12T20:03:26.342 DEBUG:boto:Token: None 2022-09-12T20:03:26.343 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:26.343 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MPqsYxYrLqevVVTsffReMP0qUAQ= 2022-09-12T20:03:26.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MPqsYxYrLqevVVTsffReMP0qUAQ=', 'Content-Length': '0'} 2022-09-12T20:03:26.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3dcb2726d1ff8df-00631f908e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.347 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:26.348 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:26.349 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:26.349 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:26.350 DEBUG:boto:Method: GET 2022-09-12T20:03:26.350 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:26.351 DEBUG:boto:Data: 2022-09-12T20:03:26.351 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.352 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.353 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.353 DEBUG:boto:Params: {} 2022-09-12T20:03:26.354 DEBUG:boto:Token: None 2022-09-12T20:03:26.355 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:26.355 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MPqsYxYrLqevVVTsffReMP0qUAQ= 2022-09-12T20:03:26.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MPqsYxYrLqevVVTsffReMP0qUAQ=', 'Content-Length': '0'} 2022-09-12T20:03:26.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056f53f94a26ebe37-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.402 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:26.403 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:26.403 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:26.404 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:26.404 DEBUG:boto:Method: HEAD 2022-09-12T20:03:26.405 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:26.405 DEBUG:boto:Data: 2022-09-12T20:03:26.406 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.406 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.407 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.407 DEBUG:boto:Params: {} 2022-09-12T20:03:26.408 DEBUG:boto:Token: None 2022-09-12T20:03:26.409 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:26.409 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VubiWWLjXJXwRDPZI8FPQmmnIlw= 2022-09-12T20:03:26.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VubiWWLjXJXwRDPZI8FPQmmnIlw=', 'Content-Length': '0'} 2022-09-12T20:03:26.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022323267124ded25-00631f908e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.413 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:26.413 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:26.414 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:26.414 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:26.415 DEBUG:boto:Method: HEAD 2022-09-12T20:03:26.415 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:26.416 DEBUG:boto:Data: 2022-09-12T20:03:26.416 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.417 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.417 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.418 DEBUG:boto:Params: {} 2022-09-12T20:03:26.418 DEBUG:boto:Token: None 2022-09-12T20:03:26.419 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:26.420 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VubiWWLjXJXwRDPZI8FPQmmnIlw= 2022-09-12T20:03:26.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VubiWWLjXJXwRDPZI8FPQmmnIlw=', 'Content-Length': '0'} 2022-09-12T20:03:26.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055cfe79ab19ea26c-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:03:26.423 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:26.424 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:26.424 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:26.425 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:26.425 DEBUG:boto:Method: GET 2022-09-12T20:03:26.426 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:26.426 DEBUG:boto:Data: 2022-09-12T20:03:26.427 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.427 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.428 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.429 DEBUG:boto:Params: {} 2022-09-12T20:03:26.429 DEBUG:boto:Token: None 2022-09-12T20:03:26.430 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:26.431 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MPqsYxYrLqevVVTsffReMP0qUAQ= 2022-09-12T20:03:26.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MPqsYxYrLqevVVTsffReMP0qUAQ=', 'Content-Length': '0'} 2022-09-12T20:03:26.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059f41b5c6a91a11a-00631f908e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.434 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:26.435 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:26.435 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:26.436 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:26.436 DEBUG:boto:Method: GET 2022-09-12T20:03:26.437 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:26.438 DEBUG:boto:Data: 2022-09-12T20:03:26.438 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.439 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.439 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.440 DEBUG:boto:Params: {} 2022-09-12T20:03:26.441 DEBUG:boto:Token: None 2022-09-12T20:03:26.441 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:26.442 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MPqsYxYrLqevVVTsffReMP0qUAQ= 2022-09-12T20:03:26.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MPqsYxYrLqevVVTsffReMP0qUAQ=', 'Content-Length': '0'} 2022-09-12T20:03:26.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5685431534178ab-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.445 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:26.446 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:26.447 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:03:26.447 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:03:26.448 DEBUG:boto:Method: GET 2022-09-12T20:03:26.448 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:03:26.449 DEBUG:boto:Data: 2022-09-12T20:03:26.449 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.450 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.450 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.451 DEBUG:boto:Params: {} 2022-09-12T20:03:26.451 DEBUG:boto:Token: None 2022-09-12T20:03:26.452 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/ 2022-09-12T20:03:26.452 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc= 2022-09-12T20:03:26.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc=', 'Content-Length': '0'} 2022-09-12T20:03:26.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000481d9b9a7f305aba-00631f908e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.456 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:26.457 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:26.458 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:26.458 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:03:26.459 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:03:26.460 DEBUG:boto:Method: GET 2022-09-12T20:03:26.460 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:03:26.461 DEBUG:boto:Data: 2022-09-12T20:03:26.461 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.462 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.462 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.463 DEBUG:boto:Params: {} 2022-09-12T20:03:26.463 DEBUG:boto:Token: None 2022-09-12T20:03:26.464 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/ 2022-09-12T20:03:26.464 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc= 2022-09-12T20:03:26.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc=', 'Content-Length': '0'} 2022-09-12T20:03:26.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088668608e43bb10e-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.469 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:26.470 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:26.470 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:26.471 DEBUG:boto:Method: HEAD 2022-09-12T20:03:26.471 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:03:26.472 DEBUG:boto:Data: 2022-09-12T20:03:26.472 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.473 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.473 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.474 DEBUG:boto:Params: {} 2022-09-12T20:03:26.474 DEBUG:boto:Token: None 2022-09-12T20:03:26.475 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj33 2022-09-12T20:03:26.476 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zHBxYR71e2diTtx1FKTEWqFeX+o= 2022-09-12T20:03:26.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zHBxYR71e2diTtx1FKTEWqFeX+o=', 'Content-Length': '0'} 2022-09-12T20:03:26.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089b7bfb8d63b2da2-00631f908e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.479 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:26.479 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:26.480 DEBUG:boto:Method: HEAD 2022-09-12T20:03:26.480 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:03:26.481 DEBUG:boto:Data: 2022-09-12T20:03:26.481 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.482 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.482 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.483 DEBUG:boto:Params: {} 2022-09-12T20:03:26.483 DEBUG:boto:Token: None 2022-09-12T20:03:26.484 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj33 2022-09-12T20:03:26.485 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zHBxYR71e2diTtx1FKTEWqFeX+o= 2022-09-12T20:03:26.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zHBxYR71e2diTtx1FKTEWqFeX+o=', 'Content-Length': '0'} 2022-09-12T20:03:26.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025e398be6af44cba-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:03:26.488 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:26.489 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:26.489 DEBUG:boto:Method: GET 2022-09-12T20:03:26.490 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:03:26.490 DEBUG:boto:Data: 2022-09-12T20:03:26.491 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.491 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.492 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.492 DEBUG:boto:Params: {} 2022-09-12T20:03:26.493 DEBUG:boto:Token: None 2022-09-12T20:03:26.494 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj33 2022-09-12T20:03:26.494 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L+1uG/7b5aITIdDBB1UJfDoQPYw= 2022-09-12T20:03:26.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L+1uG/7b5aITIdDBB1UJfDoQPYw=', 'Content-Length': '0'} 2022-09-12T20:03:26.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c54502d9567787a-00631f908e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.498 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:26.499 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:26.499 DEBUG:boto:Method: GET 2022-09-12T20:03:26.500 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:03:26.501 DEBUG:boto:Data: 2022-09-12T20:03:26.501 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.502 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.502 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.503 DEBUG:boto:Params: {} 2022-09-12T20:03:26.503 DEBUG:boto:Token: None 2022-09-12T20:03:26.504 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj33 2022-09-12T20:03:26.505 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L+1uG/7b5aITIdDBB1UJfDoQPYw= 2022-09-12T20:03:26.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L+1uG/7b5aITIdDBB1UJfDoQPYw=', 'Content-Length': '0'} 2022-09-12T20:03:26.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab596a24c92c18ff-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.509 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:26.510 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:26.511 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:03:26.511 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:03:26.512 DEBUG:boto:Method: GET 2022-09-12T20:03:26.512 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:03:26.513 DEBUG:boto:Data: 2022-09-12T20:03:26.513 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.514 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.514 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.515 DEBUG:boto:Params: {} 2022-09-12T20:03:26.515 DEBUG:boto:Token: None 2022-09-12T20:03:26.516 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/ 2022-09-12T20:03:26.517 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc= 2022-09-12T20:03:26.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc=', 'Content-Length': '0'} 2022-09-12T20:03:26.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072c4977bc659fbcf-00631f908e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.521 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:26.522 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:26.528 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:26.529 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:03:26.530 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:03:26.531 DEBUG:boto:Method: GET 2022-09-12T20:03:26.531 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:03:26.532 DEBUG:boto:Data: 2022-09-12T20:03:26.532 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.533 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.533 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.534 DEBUG:boto:Params: {} 2022-09-12T20:03:26.534 DEBUG:boto:Token: None 2022-09-12T20:03:26.535 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/ 2022-09-12T20:03:26.536 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc= 2022-09-12T20:03:26.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc=', 'Content-Length': '0'} 2022-09-12T20:03:26.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2227728e0f3d4e2-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.541 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:26.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:03:26.542 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:26.543 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:26.543 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:26.544 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:26.544 DEBUG:boto:Method: GET 2022-09-12T20:03:26.545 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:26.546 DEBUG:boto:Data: 2022-09-12T20:03:26.546 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.547 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.547 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.548 DEBUG:boto:Params: {} 2022-09-12T20:03:26.548 DEBUG:boto:Token: None 2022-09-12T20:03:26.549 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:26.550 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KLEYwklEruB2e3hlfZ31UofbNY8= 2022-09-12T20:03:26.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KLEYwklEruB2e3hlfZ31UofbNY8=', 'Content-Length': '0'} 2022-09-12T20:03:26.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b0ba9eac952bc52-00631f908e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.554 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:26.554 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:26.555 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:26.555 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:26.556 DEBUG:boto:Method: GET 2022-09-12T20:03:26.556 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:26.557 DEBUG:boto:Data: 2022-09-12T20:03:26.558 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.558 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.559 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.559 DEBUG:boto:Params: {} 2022-09-12T20:03:26.560 DEBUG:boto:Token: None 2022-09-12T20:03:26.560 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:26.561 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KLEYwklEruB2e3hlfZ31UofbNY8= 2022-09-12T20:03:26.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KLEYwklEruB2e3hlfZ31UofbNY8=', 'Content-Length': '0'} 2022-09-12T20:03:26.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086c5f6c24280efdc-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.606 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:26.607 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:26.607 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:26.608 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:26.608 DEBUG:boto:Method: HEAD 2022-09-12T20:03:26.609 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:26.609 DEBUG:boto:Data: 2022-09-12T20:03:26.610 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.610 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.611 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.611 DEBUG:boto:Params: {} 2022-09-12T20:03:26.612 DEBUG:boto:Token: None 2022-09-12T20:03:26.613 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:26.613 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sgliiV8FSKV2kE64Sxmi0a6ZS1k= 2022-09-12T20:03:26.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sgliiV8FSKV2kE64Sxmi0a6ZS1k=', 'Content-Length': '0'} 2022-09-12T20:03:26.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a257d4985f2fefa5-00631f908e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.617 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:26.617 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:26.618 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:26.618 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:26.619 DEBUG:boto:Method: HEAD 2022-09-12T20:03:26.619 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:26.620 DEBUG:boto:Data: 2022-09-12T20:03:26.620 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.621 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.621 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.622 DEBUG:boto:Params: {} 2022-09-12T20:03:26.622 DEBUG:boto:Token: None 2022-09-12T20:03:26.623 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:26.624 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sgliiV8FSKV2kE64Sxmi0a6ZS1k= 2022-09-12T20:03:26.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sgliiV8FSKV2kE64Sxmi0a6ZS1k=', 'Content-Length': '0'} 2022-09-12T20:03:26.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc540f9db44dd029-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:03:26.627 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:26.628 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:26.629 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:26.629 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:26.630 DEBUG:boto:Method: GET 2022-09-12T20:03:26.630 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:26.631 DEBUG:boto:Data: 2022-09-12T20:03:26.631 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.632 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.632 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.633 DEBUG:boto:Params: {} 2022-09-12T20:03:26.634 DEBUG:boto:Token: None 2022-09-12T20:03:26.634 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:26.635 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KLEYwklEruB2e3hlfZ31UofbNY8= 2022-09-12T20:03:26.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KLEYwklEruB2e3hlfZ31UofbNY8=', 'Content-Length': '0'} 2022-09-12T20:03:26.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000398fe950c7fea2c1-00631f908e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.638 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:26.639 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:26.639 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:26.640 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:26.640 DEBUG:boto:Method: GET 2022-09-12T20:03:26.641 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:26.642 DEBUG:boto:Data: 2022-09-12T20:03:26.642 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.643 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.643 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.644 DEBUG:boto:Params: {} 2022-09-12T20:03:26.644 DEBUG:boto:Token: None 2022-09-12T20:03:26.645 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:26.646 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KLEYwklEruB2e3hlfZ31UofbNY8= 2022-09-12T20:03:26.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KLEYwklEruB2e3hlfZ31UofbNY8=', 'Content-Length': '0'} 2022-09-12T20:03:26.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004bc90bddc9315c8-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.649 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:26.650 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:26.650 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:03:26.651 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:03:26.651 DEBUG:boto:Method: GET 2022-09-12T20:03:26.652 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:03:26.652 DEBUG:boto:Data: 2022-09-12T20:03:26.653 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.654 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.654 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.655 DEBUG:boto:Params: {} 2022-09-12T20:03:26.655 DEBUG:boto:Token: None 2022-09-12T20:03:26.656 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/ 2022-09-12T20:03:26.657 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc= 2022-09-12T20:03:26.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc=', 'Content-Length': '0'} 2022-09-12T20:03:26.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6e7e03906c7cadf-00631f908e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.661 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:26.662 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:26.662 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:26.663 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:03:26.663 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:03:26.664 DEBUG:boto:Method: GET 2022-09-12T20:03:26.664 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:03:26.665 DEBUG:boto:Data: 2022-09-12T20:03:26.665 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.666 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.666 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.667 DEBUG:boto:Params: {} 2022-09-12T20:03:26.668 DEBUG:boto:Token: None 2022-09-12T20:03:26.668 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/ 2022-09-12T20:03:26.669 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc= 2022-09-12T20:03:26.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc=', 'Content-Length': '0'} 2022-09-12T20:03:26.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3ff6ea07d7585d6-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.673 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:26.674 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:26.674 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:26.675 DEBUG:boto:Method: HEAD 2022-09-12T20:03:26.675 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:03:26.676 DEBUG:boto:Data: 2022-09-12T20:03:26.676 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.677 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.677 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.678 DEBUG:boto:Params: {} 2022-09-12T20:03:26.678 DEBUG:boto:Token: None 2022-09-12T20:03:26.679 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj44 2022-09-12T20:03:26.680 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OJmXxda+SupphXSQOQ1WjzyzSMo= 2022-09-12T20:03:26.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OJmXxda+SupphXSQOQ1WjzyzSMo=', 'Content-Length': '0'} 2022-09-12T20:03:26.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d60b58a62da83fac-00631f908e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.683 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:26.683 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:26.684 DEBUG:boto:Method: HEAD 2022-09-12T20:03:26.684 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:03:26.685 DEBUG:boto:Data: 2022-09-12T20:03:26.685 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.686 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.686 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.687 DEBUG:boto:Params: {} 2022-09-12T20:03:26.687 DEBUG:boto:Token: None 2022-09-12T20:03:26.688 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj44 2022-09-12T20:03:26.689 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OJmXxda+SupphXSQOQ1WjzyzSMo= 2022-09-12T20:03:26.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OJmXxda+SupphXSQOQ1WjzyzSMo=', 'Content-Length': '0'} 2022-09-12T20:03:26.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043060e8af932a086-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:03:26.693 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:26.693 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:26.694 DEBUG:boto:Method: GET 2022-09-12T20:03:26.694 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:03:26.695 DEBUG:boto:Data: 2022-09-12T20:03:26.695 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.696 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.696 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.697 DEBUG:boto:Params: {} 2022-09-12T20:03:26.697 DEBUG:boto:Token: None 2022-09-12T20:03:26.698 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj44 2022-09-12T20:03:26.699 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YWz0/CIeMo8o5uARuTA7ZEtMWZE= 2022-09-12T20:03:26.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YWz0/CIeMo8o5uARuTA7ZEtMWZE=', 'Content-Length': '0'} 2022-09-12T20:03:26.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b1e1d9d3fde92ae-00631f908e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.742 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:26.743 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:26.744 DEBUG:boto:Method: GET 2022-09-12T20:03:26.746 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:03:26.747 DEBUG:boto:Data: 2022-09-12T20:03:26.748 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.749 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.750 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.750 DEBUG:boto:Params: {} 2022-09-12T20:03:26.751 DEBUG:boto:Token: None 2022-09-12T20:03:26.752 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj44 2022-09-12T20:03:26.753 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YWz0/CIeMo8o5uARuTA7ZEtMWZE= 2022-09-12T20:03:26.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YWz0/CIeMo8o5uARuTA7ZEtMWZE=', 'Content-Length': '0'} 2022-09-12T20:03:26.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfad87e954b92b82-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.758 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:26.758 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:26.759 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:03:26.759 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:03:26.760 DEBUG:boto:Method: GET 2022-09-12T20:03:26.760 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:03:26.761 DEBUG:boto:Data: 2022-09-12T20:03:26.761 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.762 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.763 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.763 DEBUG:boto:Params: {} 2022-09-12T20:03:26.764 DEBUG:boto:Token: None 2022-09-12T20:03:26.764 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/ 2022-09-12T20:03:26.765 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc= 2022-09-12T20:03:26.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc=', 'Content-Length': '0'} 2022-09-12T20:03:26.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d8d71fd8c385dcc-00631f908e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.769 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:26.770 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:26.771 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:26.771 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:03:26.772 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:03:26.772 DEBUG:boto:Method: GET 2022-09-12T20:03:26.773 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:03:26.774 DEBUG:boto:Data: 2022-09-12T20:03:26.774 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.775 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.775 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.776 DEBUG:boto:Params: {} 2022-09-12T20:03:26.776 DEBUG:boto:Token: None 2022-09-12T20:03:26.777 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/ 2022-09-12T20:03:26.777 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc= 2022-09-12T20:03:26.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc=', 'Content-Length': '0'} 2022-09-12T20:03:26.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064f11f2890a66afd-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.782 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:26.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:03:26.783 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:26.784 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:26.784 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:26.785 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:26.785 DEBUG:boto:Method: GET 2022-09-12T20:03:26.786 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:26.787 DEBUG:boto:Data: 2022-09-12T20:03:26.787 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.788 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.788 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.789 DEBUG:boto:Params: {} 2022-09-12T20:03:26.789 DEBUG:boto:Token: None 2022-09-12T20:03:26.790 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:26.791 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JlHJLrYqgdC0POiQR72MkryirEY= 2022-09-12T20:03:26.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JlHJLrYqgdC0POiQR72MkryirEY=', 'Content-Length': '0'} 2022-09-12T20:03:26.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c9cf38695fddf9e-00631f908e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.795 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:26.796 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:26.796 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:26.797 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:26.797 DEBUG:boto:Method: GET 2022-09-12T20:03:26.798 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:26.798 DEBUG:boto:Data: 2022-09-12T20:03:26.799 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.800 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.800 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.801 DEBUG:boto:Params: {} 2022-09-12T20:03:26.801 DEBUG:boto:Token: None 2022-09-12T20:03:26.802 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:26.803 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JlHJLrYqgdC0POiQR72MkryirEY= 2022-09-12T20:03:26.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JlHJLrYqgdC0POiQR72MkryirEY=', 'Content-Length': '0'} 2022-09-12T20:03:26.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077d4281033d5c96d-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.806 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:26.807 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:26.807 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:26.808 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:26.808 DEBUG:boto:Method: HEAD 2022-09-12T20:03:26.809 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:26.809 DEBUG:boto:Data: 2022-09-12T20:03:26.810 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.810 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.811 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.811 DEBUG:boto:Params: {} 2022-09-12T20:03:26.812 DEBUG:boto:Token: None 2022-09-12T20:03:26.813 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:26.813 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y6Ssgk9J1ZT6IOmb1yyLyaoWj+Y= 2022-09-12T20:03:26.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y6Ssgk9J1ZT6IOmb1yyLyaoWj+Y=', 'Content-Length': '0'} 2022-09-12T20:03:26.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071c25336d2c2efd4-00631f908e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.816 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:26.817 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:26.817 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:26.818 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:26.819 DEBUG:boto:Method: HEAD 2022-09-12T20:03:26.819 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:26.820 DEBUG:boto:Data: 2022-09-12T20:03:26.820 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.821 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.821 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.821 DEBUG:boto:Params: {} 2022-09-12T20:03:26.822 DEBUG:boto:Token: None 2022-09-12T20:03:26.823 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:26.823 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y6Ssgk9J1ZT6IOmb1yyLyaoWj+Y= 2022-09-12T20:03:26.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y6Ssgk9J1ZT6IOmb1yyLyaoWj+Y=', 'Content-Length': '0'} 2022-09-12T20:03:26.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2a3147696a058c5-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:03:26.827 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:26.828 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:26.828 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:26.829 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:26.829 DEBUG:boto:Method: GET 2022-09-12T20:03:26.830 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:26.830 DEBUG:boto:Data: 2022-09-12T20:03:26.831 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.831 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.832 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.832 DEBUG:boto:Params: {} 2022-09-12T20:03:26.833 DEBUG:boto:Token: None 2022-09-12T20:03:26.834 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:26.834 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JlHJLrYqgdC0POiQR72MkryirEY= 2022-09-12T20:03:26.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JlHJLrYqgdC0POiQR72MkryirEY=', 'Content-Length': '0'} 2022-09-12T20:03:26.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001db0bc880590776a-00631f908e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.838 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:26.839 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:26.839 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:26.840 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:26.840 DEBUG:boto:Method: GET 2022-09-12T20:03:26.841 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:26.841 DEBUG:boto:Data: 2022-09-12T20:03:26.842 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.842 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.843 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.843 DEBUG:boto:Params: {} 2022-09-12T20:03:26.844 DEBUG:boto:Token: None 2022-09-12T20:03:26.845 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:26.845 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JlHJLrYqgdC0POiQR72MkryirEY= 2022-09-12T20:03:26.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JlHJLrYqgdC0POiQR72MkryirEY=', 'Content-Length': '0'} 2022-09-12T20:03:26.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b954d18379bda5cd-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.849 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:26.849 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:26.850 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:03:26.850 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:03:26.851 DEBUG:boto:Method: GET 2022-09-12T20:03:26.851 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:03:26.852 DEBUG:boto:Data: 2022-09-12T20:03:26.852 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.853 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.853 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.854 DEBUG:boto:Params: {} 2022-09-12T20:03:26.854 DEBUG:boto:Token: None 2022-09-12T20:03:26.855 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/ 2022-09-12T20:03:26.856 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc= 2022-09-12T20:03:26.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc=', 'Content-Length': '0'} 2022-09-12T20:03:26.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa17ba97cf5a31f3-00631f908e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.860 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:26.861 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:26.861 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:26.862 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:03:26.862 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:03:26.863 DEBUG:boto:Method: GET 2022-09-12T20:03:26.863 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:03:26.864 DEBUG:boto:Data: 2022-09-12T20:03:26.864 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.865 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.865 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.866 DEBUG:boto:Params: {} 2022-09-12T20:03:26.866 DEBUG:boto:Token: None 2022-09-12T20:03:26.867 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/ 2022-09-12T20:03:26.868 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc= 2022-09-12T20:03:26.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc=', 'Content-Length': '0'} 2022-09-12T20:03:26.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d314c6c425bb29a8-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.872 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:26.873 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:26.873 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:26.874 DEBUG:boto:Method: HEAD 2022-09-12T20:03:26.874 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:03:26.875 DEBUG:boto:Data: 2022-09-12T20:03:26.875 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.876 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.876 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.877 DEBUG:boto:Params: {} 2022-09-12T20:03:26.877 DEBUG:boto:Token: None 2022-09-12T20:03:26.878 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj55 2022-09-12T20:03:26.879 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5wYyfNidIjwhL5Aw/PoTKB9YtQQ= 2022-09-12T20:03:26.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5wYyfNidIjwhL5Aw/PoTKB9YtQQ=', 'Content-Length': '0'} 2022-09-12T20:03:26.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010736d575251b451-00631f908e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.882 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:26.883 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:26.883 DEBUG:boto:Method: HEAD 2022-09-12T20:03:26.884 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:03:26.884 DEBUG:boto:Data: 2022-09-12T20:03:26.885 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.885 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.886 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.886 DEBUG:boto:Params: {} 2022-09-12T20:03:26.887 DEBUG:boto:Token: None 2022-09-12T20:03:26.887 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj55 2022-09-12T20:03:26.888 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5wYyfNidIjwhL5Aw/PoTKB9YtQQ= 2022-09-12T20:03:26.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5wYyfNidIjwhL5Aw/PoTKB9YtQQ=', 'Content-Length': '0'} 2022-09-12T20:03:26.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000655ee2545b401880-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:03:26.892 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:26.892 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:26.893 DEBUG:boto:Method: GET 2022-09-12T20:03:26.893 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:03:26.894 DEBUG:boto:Data: 2022-09-12T20:03:26.895 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.895 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.896 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.896 DEBUG:boto:Params: {} 2022-09-12T20:03:26.897 DEBUG:boto:Token: None 2022-09-12T20:03:26.897 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj55 2022-09-12T20:03:26.898 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:n4IBQvikFZGNYVN4zD4J72qHkFE= 2022-09-12T20:03:26.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:n4IBQvikFZGNYVN4zD4J72qHkFE=', 'Content-Length': '0'} 2022-09-12T20:03:26.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000575e7dce1bb227c2-00631f908e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.942 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:26.943 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:26.943 DEBUG:boto:Method: GET 2022-09-12T20:03:26.944 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:03:26.944 DEBUG:boto:Data: 2022-09-12T20:03:26.945 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.945 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.946 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.946 DEBUG:boto:Params: {} 2022-09-12T20:03:26.947 DEBUG:boto:Token: None 2022-09-12T20:03:26.948 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/obj55 2022-09-12T20:03:26.948 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:n4IBQvikFZGNYVN4zD4J72qHkFE= 2022-09-12T20:03:26.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:n4IBQvikFZGNYVN4zD4J72qHkFE=', 'Content-Length': '0'} 2022-09-12T20:03:26.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d88e1358ccfb9158-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.953 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:26.954 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:26.955 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:03:26.956 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:03:26.957 DEBUG:boto:Method: GET 2022-09-12T20:03:26.958 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:03:26.959 DEBUG:boto:Data: 2022-09-12T20:03:26.960 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.961 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.962 DEBUG:boto:Port: 8001 2022-09-12T20:03:26.963 DEBUG:boto:Params: {} 2022-09-12T20:03:26.963 DEBUG:boto:Token: None 2022-09-12T20:03:26.964 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/ 2022-09-12T20:03:26.965 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc= 2022-09-12T20:03:26.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc=', 'Content-Length': '0'} 2022-09-12T20:03:26.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000806ebb1049bd91e7-00631f908e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.970 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:26.971 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:26.972 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:26.973 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:03:26.974 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:03:26.975 DEBUG:boto:Method: GET 2022-09-12T20:03:26.976 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:03:26.977 DEBUG:boto:Data: 2022-09-12T20:03:26.978 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.979 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:26.980 DEBUG:boto:Port: 8000 2022-09-12T20:03:26.981 DEBUG:boto:Params: {} 2022-09-12T20:03:26.982 DEBUG:boto:Token: None 2022-09-12T20:03:26.983 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:26 GMT /jdpiak-75/ 2022-09-12T20:03:26.984 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc= 2022-09-12T20:03:26.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PwnbeV+ZJaG047cKY9Lpr+w3BDc=', 'Content-Length': '0'} 2022-09-12T20:03:26.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ed03982959f1a89-00631f908e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:26.990 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:26.992 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-75 zones={test-zone3, test-zone1} 2022-09-12T20:03:26.992 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-75 zones={test-zone3, test-zone2} 2022-09-12T20:03:26.993 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:26.994 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:26.995 DEBUG:boto:Method: HEAD 2022-09-12T20:03:26.996 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:03:26.997 DEBUG:boto:Data: 2022-09-12T20:03:26.997 DEBUG:boto:Headers: {} 2022-09-12T20:03:26.998 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:26.999 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.000 DEBUG:boto:Params: {} 2022-09-12T20:03:27.001 DEBUG:boto:Token: None 2022-09-12T20:03:27.002 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/ 2022-09-12T20:03:27.003 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qUmHtyFho4ozMfZKagXVwIY37wA= 2022-09-12T20:03:27.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qUmHtyFho4ozMfZKagXVwIY37wA=', 'Content-Length': '0'} 2022-09-12T20:03:27.007 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bee6bc96c13699a7-00631f908f-1826-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.008 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:27.009 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:27.010 DEBUG:boto:Method: HEAD 2022-09-12T20:03:27.011 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:03:27.012 DEBUG:boto:Data: 2022-09-12T20:03:27.013 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.013 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.014 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.015 DEBUG:boto:Params: {} 2022-09-12T20:03:27.016 DEBUG:boto:Token: None 2022-09-12T20:03:27.017 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/ 2022-09-12T20:03:27.018 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qUmHtyFho4ozMfZKagXVwIY37wA= 2022-09-12T20:03:27.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qUmHtyFho4ozMfZKagXVwIY37wA=', 'Content-Length': '0'} 2022-09-12T20:03:27.021 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b4321b043a51e2d3-00631f908f-189d-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.022 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T20:03:27.023 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:27.023 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:27.024 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:27.024 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:27.025 DEBUG:boto:Method: GET 2022-09-12T20:03:27.025 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:27.026 DEBUG:boto:Data: 2022-09-12T20:03:27.026 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.027 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.027 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.028 DEBUG:boto:Params: {} 2022-09-12T20:03:27.028 DEBUG:boto:Token: None 2022-09-12T20:03:27.029 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/?versions 2022-09-12T20:03:27.030 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ocYeboHvnvRRg0jy+dqd3Pxo0lw= 2022-09-12T20:03:27.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ocYeboHvnvRRg0jy+dqd3Pxo0lw=', 'Content-Length': '0'} 2022-09-12T20:03:27.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb89726a8b4b666e-00631f908f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.034 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:27.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:03:27.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:03:27.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:03:27.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:03:27.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:03:27.037 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T20:03:27.038 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:27.038 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:27.039 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:27.039 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:27.040 DEBUG:boto:Method: GET 2022-09-12T20:03:27.041 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:27.041 DEBUG:boto:Data: 2022-09-12T20:03:27.042 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.042 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.043 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.043 DEBUG:boto:Params: {} 2022-09-12T20:03:27.044 DEBUG:boto:Token: None 2022-09-12T20:03:27.044 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/?versions 2022-09-12T20:03:27.045 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ocYeboHvnvRRg0jy+dqd3Pxo0lw= 2022-09-12T20:03:27.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ocYeboHvnvRRg0jy+dqd3Pxo0lw=', 'Content-Length': '0'} 2022-09-12T20:03:27.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000771e7aa1079197f7-00631f908f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.049 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:27.050 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:03:27.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:03:27.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:03:27.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:03:27.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:03:27.053 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:27.053 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:27.054 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:27.054 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:27.055 DEBUG:boto:Method: GET 2022-09-12T20:03:27.055 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:27.056 DEBUG:boto:Data: 2022-09-12T20:03:27.056 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.057 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.057 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.058 DEBUG:boto:Params: {} 2022-09-12T20:03:27.058 DEBUG:boto:Token: None 2022-09-12T20:03:27.059 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/?versions 2022-09-12T20:03:27.060 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ocYeboHvnvRRg0jy+dqd3Pxo0lw= 2022-09-12T20:03:27.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ocYeboHvnvRRg0jy+dqd3Pxo0lw=', 'Content-Length': '0'} 2022-09-12T20:03:27.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096474c046c84ac7d-00631f908f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.064 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:27.065 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:27.066 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:27.067 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:27.068 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:27.069 DEBUG:boto:Method: GET 2022-09-12T20:03:27.070 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:03:27.071 DEBUG:boto:Data: 2022-09-12T20:03:27.072 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.073 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.074 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.075 DEBUG:boto:Params: {} 2022-09-12T20:03:27.076 DEBUG:boto:Token: None 2022-09-12T20:03:27.077 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/?versions 2022-09-12T20:03:27.078 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ocYeboHvnvRRg0jy+dqd3Pxo0lw= 2022-09-12T20:03:27.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ocYeboHvnvRRg0jy+dqd3Pxo0lw=', 'Content-Length': '0'} 2022-09-12T20:03:27.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb754335b14cdd15-00631f908f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.084 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:27.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:03:27.087 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:27.088 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:27.089 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:27.090 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:27.091 DEBUG:boto:Method: GET 2022-09-12T20:03:27.092 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:27.093 DEBUG:boto:Data: 2022-09-12T20:03:27.094 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.095 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.096 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.097 DEBUG:boto:Params: {} 2022-09-12T20:03:27.098 DEBUG:boto:Token: None 2022-09-12T20:03:27.099 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:27.100 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7bIXtEt14cVp4kgtEBddTdLW8qk= 2022-09-12T20:03:27.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7bIXtEt14cVp4kgtEBddTdLW8qk=', 'Content-Length': '0'} 2022-09-12T20:03:27.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076ae2add6e497e81-00631f908f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.106 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:27.107 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:27.108 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:27.109 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:27.110 DEBUG:boto:Method: GET 2022-09-12T20:03:27.111 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:27.112 DEBUG:boto:Data: 2022-09-12T20:03:27.113 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.114 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.115 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.116 DEBUG:boto:Params: {} 2022-09-12T20:03:27.117 DEBUG:boto:Token: None 2022-09-12T20:03:27.118 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:27.119 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7bIXtEt14cVp4kgtEBddTdLW8qk= 2022-09-12T20:03:27.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7bIXtEt14cVp4kgtEBddTdLW8qk=', 'Content-Length': '0'} 2022-09-12T20:03:27.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8ed6ed3b45bf448-00631f908f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.170 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:27.171 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:27.172 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:27.172 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:27.173 DEBUG:boto:Method: HEAD 2022-09-12T20:03:27.173 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:27.174 DEBUG:boto:Data: 2022-09-12T20:03:27.174 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.175 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.175 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.176 DEBUG:boto:Params: {} 2022-09-12T20:03:27.177 DEBUG:boto:Token: None 2022-09-12T20:03:27.177 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:27.178 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s/7WUw+HSQNuHsNCfsUIz4PF+Ag= 2022-09-12T20:03:27.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s/7WUw+HSQNuHsNCfsUIz4PF+Ag=', 'Content-Length': '0'} 2022-09-12T20:03:27.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca68b74f98160b96-00631f908f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.183 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:27.183 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:27.184 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:27.184 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:27.185 DEBUG:boto:Method: HEAD 2022-09-12T20:03:27.185 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:27.186 DEBUG:boto:Data: 2022-09-12T20:03:27.186 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.187 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.187 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.188 DEBUG:boto:Params: {} 2022-09-12T20:03:27.188 DEBUG:boto:Token: None 2022-09-12T20:03:27.189 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:27.190 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s/7WUw+HSQNuHsNCfsUIz4PF+Ag= 2022-09-12T20:03:27.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s/7WUw+HSQNuHsNCfsUIz4PF+Ag=', 'Content-Length': '0'} 2022-09-12T20:03:27.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f4a86286e334c04-00631f908f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:03:27.194 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:27.195 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:27.196 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:27.197 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:27.199 DEBUG:boto:Method: GET 2022-09-12T20:03:27.200 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:27.201 DEBUG:boto:Data: 2022-09-12T20:03:27.202 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.203 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.204 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.205 DEBUG:boto:Params: {} 2022-09-12T20:03:27.206 DEBUG:boto:Token: None 2022-09-12T20:03:27.207 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:27.208 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7bIXtEt14cVp4kgtEBddTdLW8qk= 2022-09-12T20:03:27.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7bIXtEt14cVp4kgtEBddTdLW8qk=', 'Content-Length': '0'} 2022-09-12T20:03:27.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f91e867ba1a82583-00631f908f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.213 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:27.215 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:27.216 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:27.217 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:03:27.218 DEBUG:boto:Method: GET 2022-09-12T20:03:27.219 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:27.219 DEBUG:boto:Data: 2022-09-12T20:03:27.220 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.221 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.222 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.223 DEBUG:boto:Params: {} 2022-09-12T20:03:27.224 DEBUG:boto:Token: None 2022-09-12T20:03:27.226 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:03:27.227 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7bIXtEt14cVp4kgtEBddTdLW8qk= 2022-09-12T20:03:27.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7bIXtEt14cVp4kgtEBddTdLW8qk=', 'Content-Length': '0'} 2022-09-12T20:03:27.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000026475921933f64e-00631f908f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.230 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:27.231 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:27.232 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:03:27.232 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:03:27.233 DEBUG:boto:Method: GET 2022-09-12T20:03:27.234 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:03:27.234 DEBUG:boto:Data: 2022-09-12T20:03:27.235 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.235 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.236 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.236 DEBUG:boto:Params: {} 2022-09-12T20:03:27.237 DEBUG:boto:Token: None 2022-09-12T20:03:27.238 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/ 2022-09-12T20:03:27.239 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg= 2022-09-12T20:03:27.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg=', 'Content-Length': '0'} 2022-09-12T20:03:27.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d4da0829194831c-00631f908f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.243 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:27.244 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:27.245 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:27.245 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:03:27.246 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:03:27.246 DEBUG:boto:Method: GET 2022-09-12T20:03:27.247 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:03:27.247 DEBUG:boto:Data: 2022-09-12T20:03:27.248 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.249 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.249 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.250 DEBUG:boto:Params: {} 2022-09-12T20:03:27.251 DEBUG:boto:Token: None 2022-09-12T20:03:27.251 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/ 2022-09-12T20:03:27.252 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg= 2022-09-12T20:03:27.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg=', 'Content-Length': '0'} 2022-09-12T20:03:27.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9c3e2b02fa34861-00631f908f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.256 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:27.257 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:27.257 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:27.258 DEBUG:boto:Method: HEAD 2022-09-12T20:03:27.258 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:03:27.259 DEBUG:boto:Data: 2022-09-12T20:03:27.260 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.260 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.261 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.261 DEBUG:boto:Params: {} 2022-09-12T20:03:27.262 DEBUG:boto:Token: None 2022-09-12T20:03:27.262 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj11 2022-09-12T20:03:27.263 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+OoLWH5Q09CjzmGepi5aDqlkLoM= 2022-09-12T20:03:27.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+OoLWH5Q09CjzmGepi5aDqlkLoM=', 'Content-Length': '0'} 2022-09-12T20:03:27.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008f813a189c19c08-00631f908f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.267 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:27.267 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:27.268 DEBUG:boto:Method: HEAD 2022-09-12T20:03:27.268 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:03:27.269 DEBUG:boto:Data: 2022-09-12T20:03:27.269 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.270 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.270 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.271 DEBUG:boto:Params: {} 2022-09-12T20:03:27.272 DEBUG:boto:Token: None 2022-09-12T20:03:27.272 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj11 2022-09-12T20:03:27.273 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+OoLWH5Q09CjzmGepi5aDqlkLoM= 2022-09-12T20:03:27.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+OoLWH5Q09CjzmGepi5aDqlkLoM=', 'Content-Length': '0'} 2022-09-12T20:03:27.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bc1fd0a4a90e940-00631f908f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:03:27.277 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:27.278 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:27.278 DEBUG:boto:Method: GET 2022-09-12T20:03:27.279 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:03:27.279 DEBUG:boto:Data: 2022-09-12T20:03:27.280 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.280 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.281 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.282 DEBUG:boto:Params: {} 2022-09-12T20:03:27.283 DEBUG:boto:Token: None 2022-09-12T20:03:27.283 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj11 2022-09-12T20:03:27.284 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:33tTqEN9V/JoBNsZQVze6mhCuKE= 2022-09-12T20:03:27.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:33tTqEN9V/JoBNsZQVze6mhCuKE=', 'Content-Length': '0'} 2022-09-12T20:03:27.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030e496f01b5126d2-00631f908f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.330 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:03:27.331 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:03:27.332 DEBUG:boto:Method: GET 2022-09-12T20:03:27.333 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:03:27.334 DEBUG:boto:Data: 2022-09-12T20:03:27.335 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.336 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.337 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.338 DEBUG:boto:Params: {} 2022-09-12T20:03:27.339 DEBUG:boto:Token: None 2022-09-12T20:03:27.340 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj11 2022-09-12T20:03:27.341 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:33tTqEN9V/JoBNsZQVze6mhCuKE= 2022-09-12T20:03:27.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:33tTqEN9V/JoBNsZQVze6mhCuKE=', 'Content-Length': '0'} 2022-09-12T20:03:27.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029ea206bec455182-00631f908f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.347 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:27.348 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:27.349 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:03:27.350 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:03:27.351 DEBUG:boto:Method: GET 2022-09-12T20:03:27.352 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:03:27.353 DEBUG:boto:Data: 2022-09-12T20:03:27.354 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.355 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.356 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.357 DEBUG:boto:Params: {} 2022-09-12T20:03:27.358 DEBUG:boto:Token: None 2022-09-12T20:03:27.359 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/ 2022-09-12T20:03:27.360 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg= 2022-09-12T20:03:27.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg=', 'Content-Length': '0'} 2022-09-12T20:03:27.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d9052e226229f6b-00631f908f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.366 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:27.368 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:27.369 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:27.370 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:03:27.371 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:03:27.372 DEBUG:boto:Method: GET 2022-09-12T20:03:27.373 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:03:27.373 DEBUG:boto:Data: 2022-09-12T20:03:27.374 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.374 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.375 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.375 DEBUG:boto:Params: {} 2022-09-12T20:03:27.376 DEBUG:boto:Token: None 2022-09-12T20:03:27.377 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/ 2022-09-12T20:03:27.377 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg= 2022-09-12T20:03:27.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg=', 'Content-Length': '0'} 2022-09-12T20:03:27.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca5de25209f6c1c4-00631f908f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.381 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:27.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:03:27.383 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:27.383 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:27.384 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:27.384 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:27.385 DEBUG:boto:Method: GET 2022-09-12T20:03:27.386 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:27.386 DEBUG:boto:Data: 2022-09-12T20:03:27.387 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.387 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.388 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.388 DEBUG:boto:Params: {} 2022-09-12T20:03:27.389 DEBUG:boto:Token: None 2022-09-12T20:03:27.390 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:27.390 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9RTV5uBJu7IVEzd9IXcP+Rz4Ixc= 2022-09-12T20:03:27.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9RTV5uBJu7IVEzd9IXcP+Rz4Ixc=', 'Content-Length': '0'} 2022-09-12T20:03:27.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c489f0b4660f0a1d-00631f908f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.394 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:27.394 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:27.395 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:27.396 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:27.396 DEBUG:boto:Method: GET 2022-09-12T20:03:27.397 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:27.397 DEBUG:boto:Data: 2022-09-12T20:03:27.398 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.398 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.399 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.399 DEBUG:boto:Params: {} 2022-09-12T20:03:27.400 DEBUG:boto:Token: None 2022-09-12T20:03:27.401 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:27.401 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9RTV5uBJu7IVEzd9IXcP+Rz4Ixc= 2022-09-12T20:03:27.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9RTV5uBJu7IVEzd9IXcP+Rz4Ixc=', 'Content-Length': '0'} 2022-09-12T20:03:27.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dee568f5d323d65-00631f908f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.406 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:27.407 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:27.407 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:27.408 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:27.408 DEBUG:boto:Method: HEAD 2022-09-12T20:03:27.409 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:27.410 DEBUG:boto:Data: 2022-09-12T20:03:27.411 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.411 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.412 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.412 DEBUG:boto:Params: {} 2022-09-12T20:03:27.414 DEBUG:boto:Token: None 2022-09-12T20:03:27.415 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:27.416 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rX7/ipKTVcyy5VLhJBS7AXa+TIQ= 2022-09-12T20:03:27.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rX7/ipKTVcyy5VLhJBS7AXa+TIQ=', 'Content-Length': '0'} 2022-09-12T20:03:27.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e3019ea772b2ff5-00631f908f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.419 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:27.420 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:27.420 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:27.421 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:27.421 DEBUG:boto:Method: HEAD 2022-09-12T20:03:27.422 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:27.422 DEBUG:boto:Data: 2022-09-12T20:03:27.423 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.423 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.424 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.424 DEBUG:boto:Params: {} 2022-09-12T20:03:27.425 DEBUG:boto:Token: None 2022-09-12T20:03:27.425 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:27.426 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rX7/ipKTVcyy5VLhJBS7AXa+TIQ= 2022-09-12T20:03:27.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rX7/ipKTVcyy5VLhJBS7AXa+TIQ=', 'Content-Length': '0'} 2022-09-12T20:03:27.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3349a78d843a90c-00631f908f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:03:27.430 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:27.430 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:27.431 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:27.431 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:27.432 DEBUG:boto:Method: GET 2022-09-12T20:03:27.432 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:27.433 DEBUG:boto:Data: 2022-09-12T20:03:27.433 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.434 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.435 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.435 DEBUG:boto:Params: {} 2022-09-12T20:03:27.436 DEBUG:boto:Token: None 2022-09-12T20:03:27.436 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:27.437 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9RTV5uBJu7IVEzd9IXcP+Rz4Ixc= 2022-09-12T20:03:27.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9RTV5uBJu7IVEzd9IXcP+Rz4Ixc=', 'Content-Length': '0'} 2022-09-12T20:03:27.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b14d7e0567e60125-00631f908f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.441 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:27.441 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:27.442 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:27.442 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:03:27.443 DEBUG:boto:Method: GET 2022-09-12T20:03:27.443 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:27.444 DEBUG:boto:Data: 2022-09-12T20:03:27.444 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.445 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.446 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.446 DEBUG:boto:Params: {} 2022-09-12T20:03:27.447 DEBUG:boto:Token: None 2022-09-12T20:03:27.447 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:03:27.448 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9RTV5uBJu7IVEzd9IXcP+Rz4Ixc= 2022-09-12T20:03:27.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9RTV5uBJu7IVEzd9IXcP+Rz4Ixc=', 'Content-Length': '0'} 2022-09-12T20:03:27.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3ab3f9c7110f2c1-00631f908f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.451 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:27.452 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:27.452 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:03:27.453 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:03:27.453 DEBUG:boto:Method: GET 2022-09-12T20:03:27.454 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:03:27.454 DEBUG:boto:Data: 2022-09-12T20:03:27.455 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.455 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.456 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.457 DEBUG:boto:Params: {} 2022-09-12T20:03:27.457 DEBUG:boto:Token: None 2022-09-12T20:03:27.458 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/ 2022-09-12T20:03:27.458 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg= 2022-09-12T20:03:27.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg=', 'Content-Length': '0'} 2022-09-12T20:03:27.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027b53b7eb7d59a17-00631f908f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.462 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:27.463 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:27.464 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:27.464 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:03:27.465 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:03:27.465 DEBUG:boto:Method: GET 2022-09-12T20:03:27.466 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:03:27.466 DEBUG:boto:Data: 2022-09-12T20:03:27.467 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.467 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.468 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.469 DEBUG:boto:Params: {} 2022-09-12T20:03:27.469 DEBUG:boto:Token: None 2022-09-12T20:03:27.470 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/ 2022-09-12T20:03:27.470 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg= 2022-09-12T20:03:27.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg=', 'Content-Length': '0'} 2022-09-12T20:03:27.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b69d84c7647f8f8-00631f908f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.514 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:27.515 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:27.516 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:27.516 DEBUG:boto:Method: HEAD 2022-09-12T20:03:27.517 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:03:27.517 DEBUG:boto:Data: 2022-09-12T20:03:27.518 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.518 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.519 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.519 DEBUG:boto:Params: {} 2022-09-12T20:03:27.520 DEBUG:boto:Token: None 2022-09-12T20:03:27.520 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj22 2022-09-12T20:03:27.521 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tgDvEk8GDV0fDiKd5SQndlJuK88= 2022-09-12T20:03:27.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tgDvEk8GDV0fDiKd5SQndlJuK88=', 'Content-Length': '0'} 2022-09-12T20:03:27.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c30d76e745bfec55-00631f908f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.524 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:27.525 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:27.525 DEBUG:boto:Method: HEAD 2022-09-12T20:03:27.526 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:03:27.527 DEBUG:boto:Data: 2022-09-12T20:03:27.527 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.528 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.528 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.528 DEBUG:boto:Params: {} 2022-09-12T20:03:27.529 DEBUG:boto:Token: None 2022-09-12T20:03:27.530 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj22 2022-09-12T20:03:27.530 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tgDvEk8GDV0fDiKd5SQndlJuK88= 2022-09-12T20:03:27.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tgDvEk8GDV0fDiKd5SQndlJuK88=', 'Content-Length': '0'} 2022-09-12T20:03:27.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000303d5b1257577fa0-00631f908f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:03:27.534 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:27.535 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:27.535 DEBUG:boto:Method: GET 2022-09-12T20:03:27.536 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:03:27.537 DEBUG:boto:Data: 2022-09-12T20:03:27.537 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.538 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.538 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.539 DEBUG:boto:Params: {} 2022-09-12T20:03:27.540 DEBUG:boto:Token: None 2022-09-12T20:03:27.540 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj22 2022-09-12T20:03:27.541 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CvMLlyckOFm4P7akPKqITlT3Vp8= 2022-09-12T20:03:27.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CvMLlyckOFm4P7akPKqITlT3Vp8=', 'Content-Length': '0'} 2022-09-12T20:03:27.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f185206770141e4a-00631f908f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.544 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:03:27.545 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:03:27.546 DEBUG:boto:Method: GET 2022-09-12T20:03:27.546 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:03:27.547 DEBUG:boto:Data: 2022-09-12T20:03:27.547 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.548 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.548 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.549 DEBUG:boto:Params: {} 2022-09-12T20:03:27.549 DEBUG:boto:Token: None 2022-09-12T20:03:27.550 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj22 2022-09-12T20:03:27.551 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CvMLlyckOFm4P7akPKqITlT3Vp8= 2022-09-12T20:03:27.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CvMLlyckOFm4P7akPKqITlT3Vp8=', 'Content-Length': '0'} 2022-09-12T20:03:27.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004aa42bfa95f73911-00631f908f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.554 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:27.555 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:27.555 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:03:27.556 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:03:27.556 DEBUG:boto:Method: GET 2022-09-12T20:03:27.557 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:03:27.557 DEBUG:boto:Data: 2022-09-12T20:03:27.558 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.559 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.559 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.560 DEBUG:boto:Params: {} 2022-09-12T20:03:27.560 DEBUG:boto:Token: None 2022-09-12T20:03:27.561 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/ 2022-09-12T20:03:27.562 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg= 2022-09-12T20:03:27.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg=', 'Content-Length': '0'} 2022-09-12T20:03:27.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000359536d0582f4407-00631f908f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.566 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:27.567 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:27.568 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:27.568 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:03:27.569 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:03:27.569 DEBUG:boto:Method: GET 2022-09-12T20:03:27.570 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:03:27.571 DEBUG:boto:Data: 2022-09-12T20:03:27.571 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.572 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.572 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.573 DEBUG:boto:Params: {} 2022-09-12T20:03:27.573 DEBUG:boto:Token: None 2022-09-12T20:03:27.574 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/ 2022-09-12T20:03:27.575 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg= 2022-09-12T20:03:27.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg=', 'Content-Length': '0'} 2022-09-12T20:03:27.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcbd2c411c57c258-00631f908f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.579 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:27.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:03:27.580 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:27.581 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:27.581 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:27.582 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:27.582 DEBUG:boto:Method: GET 2022-09-12T20:03:27.583 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:27.583 DEBUG:boto:Data: 2022-09-12T20:03:27.584 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.585 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.585 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.586 DEBUG:boto:Params: {} 2022-09-12T20:03:27.586 DEBUG:boto:Token: None 2022-09-12T20:03:27.587 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:27.588 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/Dxnyc1/n4lpLf2zn6cG8PUBPRs= 2022-09-12T20:03:27.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/Dxnyc1/n4lpLf2zn6cG8PUBPRs=', 'Content-Length': '0'} 2022-09-12T20:03:27.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d018865ab3da8b99-00631f908f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.634 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:27.635 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:27.635 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:27.636 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:27.636 DEBUG:boto:Method: GET 2022-09-12T20:03:27.637 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:27.638 DEBUG:boto:Data: 2022-09-12T20:03:27.639 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.639 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.640 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.640 DEBUG:boto:Params: {} 2022-09-12T20:03:27.641 DEBUG:boto:Token: None 2022-09-12T20:03:27.641 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:27.642 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/Dxnyc1/n4lpLf2zn6cG8PUBPRs= 2022-09-12T20:03:27.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/Dxnyc1/n4lpLf2zn6cG8PUBPRs=', 'Content-Length': '0'} 2022-09-12T20:03:27.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f94edeee6ce44e4d-00631f908f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.647 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:27.647 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:27.648 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:27.648 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:27.649 DEBUG:boto:Method: HEAD 2022-09-12T20:03:27.649 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:27.650 DEBUG:boto:Data: 2022-09-12T20:03:27.650 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.651 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.651 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.652 DEBUG:boto:Params: {} 2022-09-12T20:03:27.653 DEBUG:boto:Token: None 2022-09-12T20:03:27.653 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:27.654 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/fLpZaO0NYtjOVNjH8v6THfX1UE= 2022-09-12T20:03:27.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/fLpZaO0NYtjOVNjH8v6THfX1UE=', 'Content-Length': '0'} 2022-09-12T20:03:27.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b48d2d686f30fea-00631f908f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.658 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:27.658 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:27.659 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:27.659 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:27.660 DEBUG:boto:Method: HEAD 2022-09-12T20:03:27.660 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:27.661 DEBUG:boto:Data: 2022-09-12T20:03:27.661 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.662 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.662 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.663 DEBUG:boto:Params: {} 2022-09-12T20:03:27.664 DEBUG:boto:Token: None 2022-09-12T20:03:27.665 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:27.666 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/fLpZaO0NYtjOVNjH8v6THfX1UE= 2022-09-12T20:03:27.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/fLpZaO0NYtjOVNjH8v6THfX1UE=', 'Content-Length': '0'} 2022-09-12T20:03:27.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cce1db92cca91423-00631f908f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:03:27.670 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:27.670 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:27.671 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:27.671 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:27.672 DEBUG:boto:Method: GET 2022-09-12T20:03:27.673 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:27.674 DEBUG:boto:Data: 2022-09-12T20:03:27.674 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.675 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.676 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.676 DEBUG:boto:Params: {} 2022-09-12T20:03:27.677 DEBUG:boto:Token: None 2022-09-12T20:03:27.678 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:27.678 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/Dxnyc1/n4lpLf2zn6cG8PUBPRs= 2022-09-12T20:03:27.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/Dxnyc1/n4lpLf2zn6cG8PUBPRs=', 'Content-Length': '0'} 2022-09-12T20:03:27.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca5f34c8378af9e1-00631f908f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.682 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:27.682 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:27.683 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:27.683 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:03:27.684 DEBUG:boto:Method: GET 2022-09-12T20:03:27.685 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:27.686 DEBUG:boto:Data: 2022-09-12T20:03:27.686 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.687 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.687 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.688 DEBUG:boto:Params: {} 2022-09-12T20:03:27.688 DEBUG:boto:Token: None 2022-09-12T20:03:27.689 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:03:27.690 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/Dxnyc1/n4lpLf2zn6cG8PUBPRs= 2022-09-12T20:03:27.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/Dxnyc1/n4lpLf2zn6cG8PUBPRs=', 'Content-Length': '0'} 2022-09-12T20:03:27.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046758f140633e6a6-00631f908f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.693 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:27.694 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:27.695 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:03:27.695 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:03:27.696 DEBUG:boto:Method: GET 2022-09-12T20:03:27.697 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:03:27.698 DEBUG:boto:Data: 2022-09-12T20:03:27.698 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.699 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.699 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.700 DEBUG:boto:Params: {} 2022-09-12T20:03:27.700 DEBUG:boto:Token: None 2022-09-12T20:03:27.701 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/ 2022-09-12T20:03:27.701 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg= 2022-09-12T20:03:27.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg=', 'Content-Length': '0'} 2022-09-12T20:03:27.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cd6a413cc02196c-00631f908f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.706 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:27.707 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:27.708 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:27.708 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:03:27.709 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:03:27.709 DEBUG:boto:Method: GET 2022-09-12T20:03:27.710 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:03:27.710 DEBUG:boto:Data: 2022-09-12T20:03:27.711 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.711 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.712 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.712 DEBUG:boto:Params: {} 2022-09-12T20:03:27.713 DEBUG:boto:Token: None 2022-09-12T20:03:27.713 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/ 2022-09-12T20:03:27.714 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg= 2022-09-12T20:03:27.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg=', 'Content-Length': '0'} 2022-09-12T20:03:27.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d77ff9f8c7f348a-00631f908f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.718 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:27.719 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:27.720 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:27.720 DEBUG:boto:Method: HEAD 2022-09-12T20:03:27.721 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:03:27.721 DEBUG:boto:Data: 2022-09-12T20:03:27.722 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.722 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.723 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.724 DEBUG:boto:Params: {} 2022-09-12T20:03:27.724 DEBUG:boto:Token: None 2022-09-12T20:03:27.725 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj33 2022-09-12T20:03:27.725 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:llAS5guqh9e+r4nlwVR2VR3h9AI= 2022-09-12T20:03:27.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:llAS5guqh9e+r4nlwVR2VR3h9AI=', 'Content-Length': '0'} 2022-09-12T20:03:27.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fd8a9fed39f657f-00631f908f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.729 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:27.730 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:27.730 DEBUG:boto:Method: HEAD 2022-09-12T20:03:27.731 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:03:27.732 DEBUG:boto:Data: 2022-09-12T20:03:27.732 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.733 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.733 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.734 DEBUG:boto:Params: {} 2022-09-12T20:03:27.734 DEBUG:boto:Token: None 2022-09-12T20:03:27.735 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj33 2022-09-12T20:03:27.735 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:llAS5guqh9e+r4nlwVR2VR3h9AI= 2022-09-12T20:03:27.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:llAS5guqh9e+r4nlwVR2VR3h9AI=', 'Content-Length': '0'} 2022-09-12T20:03:27.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8164d452000b733-00631f908f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:03:27.739 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:27.740 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:27.741 DEBUG:boto:Method: GET 2022-09-12T20:03:27.741 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:03:27.742 DEBUG:boto:Data: 2022-09-12T20:03:27.742 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.743 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.743 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.744 DEBUG:boto:Params: {} 2022-09-12T20:03:27.745 DEBUG:boto:Token: None 2022-09-12T20:03:27.745 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj33 2022-09-12T20:03:27.746 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KIm/R8mFfnHUXeazHmthMiwDzuw= 2022-09-12T20:03:27.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KIm/R8mFfnHUXeazHmthMiwDzuw=', 'Content-Length': '0'} 2022-09-12T20:03:27.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012c986098dd7f02a-00631f908f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.750 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:03:27.750 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:03:27.751 DEBUG:boto:Method: GET 2022-09-12T20:03:27.751 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:03:27.752 DEBUG:boto:Data: 2022-09-12T20:03:27.752 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.753 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.754 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.754 DEBUG:boto:Params: {} 2022-09-12T20:03:27.755 DEBUG:boto:Token: None 2022-09-12T20:03:27.755 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj33 2022-09-12T20:03:27.756 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KIm/R8mFfnHUXeazHmthMiwDzuw= 2022-09-12T20:03:27.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KIm/R8mFfnHUXeazHmthMiwDzuw=', 'Content-Length': '0'} 2022-09-12T20:03:27.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079f0bdd170bad861-00631f908f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.802 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:27.803 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:27.803 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:03:27.804 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:03:27.804 DEBUG:boto:Method: GET 2022-09-12T20:03:27.805 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:03:27.806 DEBUG:boto:Data: 2022-09-12T20:03:27.806 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.807 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.807 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.808 DEBUG:boto:Params: {} 2022-09-12T20:03:27.808 DEBUG:boto:Token: None 2022-09-12T20:03:27.809 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/ 2022-09-12T20:03:27.810 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg= 2022-09-12T20:03:27.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg=', 'Content-Length': '0'} 2022-09-12T20:03:27.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da027e394b53593b-00631f908f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.814 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:27.815 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:27.815 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:27.816 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:03:27.816 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:03:27.817 DEBUG:boto:Method: GET 2022-09-12T20:03:27.817 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:03:27.818 DEBUG:boto:Data: 2022-09-12T20:03:27.818 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.819 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.819 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.820 DEBUG:boto:Params: {} 2022-09-12T20:03:27.821 DEBUG:boto:Token: None 2022-09-12T20:03:27.821 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/ 2022-09-12T20:03:27.822 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg= 2022-09-12T20:03:27.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg=', 'Content-Length': '0'} 2022-09-12T20:03:27.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7e13e1569e4c31c-00631f908f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.826 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:27.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:03:27.828 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:27.828 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:27.829 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:27.829 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:27.830 DEBUG:boto:Method: GET 2022-09-12T20:03:27.830 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:27.831 DEBUG:boto:Data: 2022-09-12T20:03:27.831 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.832 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.833 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.834 DEBUG:boto:Params: {} 2022-09-12T20:03:27.834 DEBUG:boto:Token: None 2022-09-12T20:03:27.835 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:27.835 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UPHQ5MET7UbaDjlumF91TA1FKkI= 2022-09-12T20:03:27.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UPHQ5MET7UbaDjlumF91TA1FKkI=', 'Content-Length': '0'} 2022-09-12T20:03:27.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af668c7b2d8a4da8-00631f908f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.840 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:27.840 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:27.841 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:27.842 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:27.842 DEBUG:boto:Method: GET 2022-09-12T20:03:27.843 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:27.844 DEBUG:boto:Data: 2022-09-12T20:03:27.845 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.846 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.846 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.847 DEBUG:boto:Params: {} 2022-09-12T20:03:27.848 DEBUG:boto:Token: None 2022-09-12T20:03:27.849 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:27.850 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UPHQ5MET7UbaDjlumF91TA1FKkI= 2022-09-12T20:03:27.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UPHQ5MET7UbaDjlumF91TA1FKkI=', 'Content-Length': '0'} 2022-09-12T20:03:27.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1555e87720f6cc1-00631f908f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.855 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:27.855 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:27.856 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:27.856 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:27.857 DEBUG:boto:Method: HEAD 2022-09-12T20:03:27.857 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:27.858 DEBUG:boto:Data: 2022-09-12T20:03:27.858 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.859 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.860 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.861 DEBUG:boto:Params: {} 2022-09-12T20:03:27.862 DEBUG:boto:Token: None 2022-09-12T20:03:27.863 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:27.864 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EW0XmhLbD5YSkv6dFNoz9v8A8q8= 2022-09-12T20:03:27.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EW0XmhLbD5YSkv6dFNoz9v8A8q8=', 'Content-Length': '0'} 2022-09-12T20:03:27.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000751cf1a4aa06c76b-00631f908f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.868 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:27.869 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:27.869 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:27.870 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:27.871 DEBUG:boto:Method: HEAD 2022-09-12T20:03:27.872 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:27.872 DEBUG:boto:Data: 2022-09-12T20:03:27.873 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.873 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.874 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.874 DEBUG:boto:Params: {} 2022-09-12T20:03:27.875 DEBUG:boto:Token: None 2022-09-12T20:03:27.876 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:27.876 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EW0XmhLbD5YSkv6dFNoz9v8A8q8= 2022-09-12T20:03:27.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EW0XmhLbD5YSkv6dFNoz9v8A8q8=', 'Content-Length': '0'} 2022-09-12T20:03:27.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0100ac69a2a7941-00631f908f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:03:27.880 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:27.881 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:27.881 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:27.882 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:27.882 DEBUG:boto:Method: GET 2022-09-12T20:03:27.883 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:27.883 DEBUG:boto:Data: 2022-09-12T20:03:27.884 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.884 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.885 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.886 DEBUG:boto:Params: {} 2022-09-12T20:03:27.886 DEBUG:boto:Token: None 2022-09-12T20:03:27.887 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:27.888 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UPHQ5MET7UbaDjlumF91TA1FKkI= 2022-09-12T20:03:27.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UPHQ5MET7UbaDjlumF91TA1FKkI=', 'Content-Length': '0'} 2022-09-12T20:03:27.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001be97d2c0c8c10ff-00631f908f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.934 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:27.935 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:27.935 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:27.936 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:03:27.937 DEBUG:boto:Method: GET 2022-09-12T20:03:27.937 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:27.938 DEBUG:boto:Data: 2022-09-12T20:03:27.938 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.939 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.940 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.940 DEBUG:boto:Params: {} 2022-09-12T20:03:27.941 DEBUG:boto:Token: None 2022-09-12T20:03:27.941 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:03:27.942 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UPHQ5MET7UbaDjlumF91TA1FKkI= 2022-09-12T20:03:27.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UPHQ5MET7UbaDjlumF91TA1FKkI=', 'Content-Length': '0'} 2022-09-12T20:03:27.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b16fbe2b78a6677-00631f908f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.946 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:27.946 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:27.947 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:03:27.948 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:03:27.948 DEBUG:boto:Method: GET 2022-09-12T20:03:27.949 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:03:27.949 DEBUG:boto:Data: 2022-09-12T20:03:27.950 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.950 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.951 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.952 DEBUG:boto:Params: {} 2022-09-12T20:03:27.952 DEBUG:boto:Token: None 2022-09-12T20:03:27.953 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/ 2022-09-12T20:03:27.954 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg= 2022-09-12T20:03:27.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg=', 'Content-Length': '0'} 2022-09-12T20:03:27.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000950bd76aa4aa28b5-00631f908f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.958 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:27.959 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:27.959 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:27.960 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:03:27.960 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:03:27.961 DEBUG:boto:Method: GET 2022-09-12T20:03:27.961 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:03:27.962 DEBUG:boto:Data: 2022-09-12T20:03:27.962 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.963 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.963 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.964 DEBUG:boto:Params: {} 2022-09-12T20:03:27.964 DEBUG:boto:Token: None 2022-09-12T20:03:27.965 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/ 2022-09-12T20:03:27.966 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg= 2022-09-12T20:03:27.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XK3/gDIJb4xp29hZDYBivQ5/Ztg=', 'Content-Length': '0'} 2022-09-12T20:03:27.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b82e9d5f1817e52-00631f908f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.969 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:27.970 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:27.971 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:27.972 DEBUG:boto:Method: HEAD 2022-09-12T20:03:27.972 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:03:27.973 DEBUG:boto:Data: 2022-09-12T20:03:27.974 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.974 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.975 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.975 DEBUG:boto:Params: {} 2022-09-12T20:03:27.976 DEBUG:boto:Token: None 2022-09-12T20:03:27.976 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj44 2022-09-12T20:03:27.977 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CfXReISgUA3wuxzgRigCWpktexM= 2022-09-12T20:03:27.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CfXReISgUA3wuxzgRigCWpktexM=', 'Content-Length': '0'} 2022-09-12T20:03:27.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e540991d5309d6ab-00631f908f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.981 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:27.981 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:27.982 DEBUG:boto:Method: HEAD 2022-09-12T20:03:27.982 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:03:27.983 DEBUG:boto:Data: 2022-09-12T20:03:27.984 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.984 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:27.985 DEBUG:boto:Port: 8000 2022-09-12T20:03:27.985 DEBUG:boto:Params: {} 2022-09-12T20:03:27.986 DEBUG:boto:Token: None 2022-09-12T20:03:27.986 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj44 2022-09-12T20:03:27.987 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CfXReISgUA3wuxzgRigCWpktexM= 2022-09-12T20:03:27.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CfXReISgUA3wuxzgRigCWpktexM=', 'Content-Length': '0'} 2022-09-12T20:03:27.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ca990d22e2003cf-00631f908f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:27.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:03:27.991 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:27.992 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:27.993 DEBUG:boto:Method: GET 2022-09-12T20:03:27.993 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:03:27.994 DEBUG:boto:Data: 2022-09-12T20:03:27.995 DEBUG:boto:Headers: {} 2022-09-12T20:03:27.996 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:27.997 DEBUG:boto:Port: 8001 2022-09-12T20:03:27.997 DEBUG:boto:Params: {} 2022-09-12T20:03:27.998 DEBUG:boto:Token: None 2022-09-12T20:03:27.998 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:27 GMT /jdpiak-75/obj44 2022-09-12T20:03:27.999 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dtiDhYN8h6PwZZGSM0AFuMECri4= 2022-09-12T20:03:28.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dtiDhYN8h6PwZZGSM0AFuMECri4=', 'Content-Length': '0'} 2022-09-12T20:03:28.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ed9a76f4b9ae336-00631f9090-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.003 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:03:28.003 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:03:28.004 DEBUG:boto:Method: GET 2022-09-12T20:03:28.005 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:03:28.005 DEBUG:boto:Data: 2022-09-12T20:03:28.006 DEBUG:boto:Headers: {} 2022-09-12T20:03:28.007 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:28.007 DEBUG:boto:Port: 8000 2022-09-12T20:03:28.008 DEBUG:boto:Params: {} 2022-09-12T20:03:28.008 DEBUG:boto:Token: None 2022-09-12T20:03:28.009 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/obj44 2022-09-12T20:03:28.010 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NCng9lQdVlfznrRneu9DgG7JgCM= 2022-09-12T20:03:28.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NCng9lQdVlfznrRneu9DgG7JgCM=', 'Content-Length': '0'} 2022-09-12T20:03:28.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c862c1dc75e34b5-00631f9090-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.013 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:28.014 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:28.014 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:03:28.015 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:03:28.016 DEBUG:boto:Method: GET 2022-09-12T20:03:28.016 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:03:28.017 DEBUG:boto:Data: 2022-09-12T20:03:28.018 DEBUG:boto:Headers: {} 2022-09-12T20:03:28.019 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:28.019 DEBUG:boto:Port: 8001 2022-09-12T20:03:28.020 DEBUG:boto:Params: {} 2022-09-12T20:03:28.021 DEBUG:boto:Token: None 2022-09-12T20:03:28.021 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/ 2022-09-12T20:03:28.022 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xwiJNY10Fp++N3JCfFskuRWPy+c= 2022-09-12T20:03:28.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xwiJNY10Fp++N3JCfFskuRWPy+c=', 'Content-Length': '0'} 2022-09-12T20:03:28.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b847bc9ffa5ed54-00631f9090-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.027 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:28.028 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:28.029 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:28.029 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:03:28.030 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:03:28.030 DEBUG:boto:Method: GET 2022-09-12T20:03:28.031 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:03:28.031 DEBUG:boto:Data: 2022-09-12T20:03:28.032 DEBUG:boto:Headers: {} 2022-09-12T20:03:28.033 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:28.033 DEBUG:boto:Port: 8000 2022-09-12T20:03:28.034 DEBUG:boto:Params: {} 2022-09-12T20:03:28.034 DEBUG:boto:Token: None 2022-09-12T20:03:28.036 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/ 2022-09-12T20:03:28.037 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xwiJNY10Fp++N3JCfFskuRWPy+c= 2022-09-12T20:03:28.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xwiJNY10Fp++N3JCfFskuRWPy+c=', 'Content-Length': '0'} 2022-09-12T20:03:28.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5a607b8f7f5b495-00631f9090-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.041 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:28.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:03:28.042 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:28.044 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:28.044 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:28.045 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:28.045 DEBUG:boto:Method: GET 2022-09-12T20:03:28.046 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:28.046 DEBUG:boto:Data: 2022-09-12T20:03:28.047 DEBUG:boto:Headers: {} 2022-09-12T20:03:28.048 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:28.048 DEBUG:boto:Port: 8001 2022-09-12T20:03:28.049 DEBUG:boto:Params: {} 2022-09-12T20:03:28.049 DEBUG:boto:Token: None 2022-09-12T20:03:28.050 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:28.051 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TQcxGWjM7u1NYdZ+zeU34BbjJrY= 2022-09-12T20:03:28.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TQcxGWjM7u1NYdZ+zeU34BbjJrY=', 'Content-Length': '0'} 2022-09-12T20:03:28.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087bf92c1c57bc737-00631f9090-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.055 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:28.055 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:28.056 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:28.057 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:28.057 DEBUG:boto:Method: GET 2022-09-12T20:03:28.058 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:28.059 DEBUG:boto:Data: 2022-09-12T20:03:28.059 DEBUG:boto:Headers: {} 2022-09-12T20:03:28.060 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:28.060 DEBUG:boto:Port: 8000 2022-09-12T20:03:28.061 DEBUG:boto:Params: {} 2022-09-12T20:03:28.062 DEBUG:boto:Token: None 2022-09-12T20:03:28.063 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:28.063 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TQcxGWjM7u1NYdZ+zeU34BbjJrY= 2022-09-12T20:03:28.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TQcxGWjM7u1NYdZ+zeU34BbjJrY=', 'Content-Length': '0'} 2022-09-12T20:03:28.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba793f5c727b72b3-00631f9090-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.067 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:28.067 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:28.068 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:28.068 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:28.069 DEBUG:boto:Method: HEAD 2022-09-12T20:03:28.070 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:28.071 DEBUG:boto:Data: 2022-09-12T20:03:28.071 DEBUG:boto:Headers: {} 2022-09-12T20:03:28.072 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:28.072 DEBUG:boto:Port: 8001 2022-09-12T20:03:28.073 DEBUG:boto:Params: {} 2022-09-12T20:03:28.073 DEBUG:boto:Token: None 2022-09-12T20:03:28.074 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:28.075 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tp3zUkYoNMQuwSQT5UfZAsaQN2E= 2022-09-12T20:03:28.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tp3zUkYoNMQuwSQT5UfZAsaQN2E=', 'Content-Length': '0'} 2022-09-12T20:03:28.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091668ec482b632fc-00631f9090-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.079 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:28.079 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:28.080 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:28.080 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:28.081 DEBUG:boto:Method: HEAD 2022-09-12T20:03:28.081 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:28.082 DEBUG:boto:Data: 2022-09-12T20:03:28.082 DEBUG:boto:Headers: {} 2022-09-12T20:03:28.083 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:28.083 DEBUG:boto:Port: 8000 2022-09-12T20:03:28.084 DEBUG:boto:Params: {} 2022-09-12T20:03:28.085 DEBUG:boto:Token: None 2022-09-12T20:03:28.085 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:28.086 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tp3zUkYoNMQuwSQT5UfZAsaQN2E= 2022-09-12T20:03:28.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tp3zUkYoNMQuwSQT5UfZAsaQN2E=', 'Content-Length': '0'} 2022-09-12T20:03:28.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003296726a1c0a07d3-00631f9090-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:03:28.090 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:28.091 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:28.091 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:28.092 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:28.092 DEBUG:boto:Method: GET 2022-09-12T20:03:28.093 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:28.093 DEBUG:boto:Data: 2022-09-12T20:03:28.094 DEBUG:boto:Headers: {} 2022-09-12T20:03:28.095 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:28.095 DEBUG:boto:Port: 8001 2022-09-12T20:03:28.096 DEBUG:boto:Params: {} 2022-09-12T20:03:28.097 DEBUG:boto:Token: None 2022-09-12T20:03:28.098 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:28.098 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TQcxGWjM7u1NYdZ+zeU34BbjJrY= 2022-09-12T20:03:28.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TQcxGWjM7u1NYdZ+zeU34BbjJrY=', 'Content-Length': '0'} 2022-09-12T20:03:28.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000808df8304f0c9d47-00631f9090-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.142 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:28.143 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:28.143 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:28.144 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:03:28.145 DEBUG:boto:Method: GET 2022-09-12T20:03:28.145 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:28.146 DEBUG:boto:Data: 2022-09-12T20:03:28.147 DEBUG:boto:Headers: {} 2022-09-12T20:03:28.148 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:28.149 DEBUG:boto:Port: 8000 2022-09-12T20:03:28.150 DEBUG:boto:Params: {} 2022-09-12T20:03:28.150 DEBUG:boto:Token: None 2022-09-12T20:03:28.151 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:03:28.152 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TQcxGWjM7u1NYdZ+zeU34BbjJrY= 2022-09-12T20:03:28.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TQcxGWjM7u1NYdZ+zeU34BbjJrY=', 'Content-Length': '0'} 2022-09-12T20:03:28.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a13068f1ddb49cd5-00631f9090-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.156 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:28.156 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:28.157 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:03:28.157 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:03:28.158 DEBUG:boto:Method: GET 2022-09-12T20:03:28.158 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:03:28.159 DEBUG:boto:Data: 2022-09-12T20:03:28.160 DEBUG:boto:Headers: {} 2022-09-12T20:03:28.160 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:28.161 DEBUG:boto:Port: 8001 2022-09-12T20:03:28.161 DEBUG:boto:Params: {} 2022-09-12T20:03:28.162 DEBUG:boto:Token: None 2022-09-12T20:03:28.162 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/ 2022-09-12T20:03:28.163 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xwiJNY10Fp++N3JCfFskuRWPy+c= 2022-09-12T20:03:28.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xwiJNY10Fp++N3JCfFskuRWPy+c=', 'Content-Length': '0'} 2022-09-12T20:03:28.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c86ba1cd02ec0a53-00631f9090-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.168 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:28.169 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:28.170 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:28.170 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:03:28.171 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:03:28.172 DEBUG:boto:Method: GET 2022-09-12T20:03:28.173 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:03:28.173 DEBUG:boto:Data: 2022-09-12T20:03:28.174 DEBUG:boto:Headers: {} 2022-09-12T20:03:28.175 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:28.176 DEBUG:boto:Port: 8000 2022-09-12T20:03:28.176 DEBUG:boto:Params: {} 2022-09-12T20:03:28.177 DEBUG:boto:Token: None 2022-09-12T20:03:28.177 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/ 2022-09-12T20:03:28.178 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xwiJNY10Fp++N3JCfFskuRWPy+c= 2022-09-12T20:03:28.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xwiJNY10Fp++N3JCfFskuRWPy+c=', 'Content-Length': '0'} 2022-09-12T20:03:28.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ab18b279188d38a-00631f9090-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.226 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:28.227 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:28.228 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:28.228 DEBUG:boto:Method: HEAD 2022-09-12T20:03:28.229 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:03:28.229 DEBUG:boto:Data: 2022-09-12T20:03:28.230 DEBUG:boto:Headers: {} 2022-09-12T20:03:28.230 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:28.231 DEBUG:boto:Port: 8001 2022-09-12T20:03:28.231 DEBUG:boto:Params: {} 2022-09-12T20:03:28.232 DEBUG:boto:Token: None 2022-09-12T20:03:28.233 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/obj55 2022-09-12T20:03:28.234 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qd6oakvDAY1MPbXQn380MP1Kmmo= 2022-09-12T20:03:28.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qd6oakvDAY1MPbXQn380MP1Kmmo=', 'Content-Length': '0'} 2022-09-12T20:03:28.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd82082511c4ff0a-00631f9090-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.239 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:28.240 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:28.240 DEBUG:boto:Method: HEAD 2022-09-12T20:03:28.241 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:03:28.242 DEBUG:boto:Data: 2022-09-12T20:03:28.242 DEBUG:boto:Headers: {} 2022-09-12T20:03:28.243 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:28.243 DEBUG:boto:Port: 8000 2022-09-12T20:03:28.244 DEBUG:boto:Params: {} 2022-09-12T20:03:28.245 DEBUG:boto:Token: None 2022-09-12T20:03:28.245 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/obj55 2022-09-12T20:03:28.246 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qd6oakvDAY1MPbXQn380MP1Kmmo= 2022-09-12T20:03:28.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qd6oakvDAY1MPbXQn380MP1Kmmo=', 'Content-Length': '0'} 2022-09-12T20:03:28.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c91f96074cea51cf-00631f9090-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:03:28.250 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:28.250 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:28.251 DEBUG:boto:Method: GET 2022-09-12T20:03:28.252 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:03:28.252 DEBUG:boto:Data: 2022-09-12T20:03:28.253 DEBUG:boto:Headers: {} 2022-09-12T20:03:28.253 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:28.254 DEBUG:boto:Port: 8001 2022-09-12T20:03:28.254 DEBUG:boto:Params: {} 2022-09-12T20:03:28.255 DEBUG:boto:Token: None 2022-09-12T20:03:28.256 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/obj55 2022-09-12T20:03:28.256 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7bmp8VybEpjF44lse3SFA7e4yfA= 2022-09-12T20:03:28.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7bmp8VybEpjF44lse3SFA7e4yfA=', 'Content-Length': '0'} 2022-09-12T20:03:28.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc8fc0ae112ad324-00631f9090-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.260 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:03:28.260 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:03:28.261 DEBUG:boto:Method: GET 2022-09-12T20:03:28.261 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:03:28.262 DEBUG:boto:Data: 2022-09-12T20:03:28.262 DEBUG:boto:Headers: {} 2022-09-12T20:03:28.263 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:28.264 DEBUG:boto:Port: 8000 2022-09-12T20:03:28.265 DEBUG:boto:Params: {} 2022-09-12T20:03:28.265 DEBUG:boto:Token: None 2022-09-12T20:03:28.266 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/obj55 2022-09-12T20:03:28.267 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7bmp8VybEpjF44lse3SFA7e4yfA= 2022-09-12T20:03:28.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7bmp8VybEpjF44lse3SFA7e4yfA=', 'Content-Length': '0'} 2022-09-12T20:03:28.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006431d737f78b8d34-00631f9090-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.270 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:28.271 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:28.272 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:03:28.273 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:03:28.274 DEBUG:boto:Method: GET 2022-09-12T20:03:28.275 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:03:28.276 DEBUG:boto:Data: 2022-09-12T20:03:28.276 DEBUG:boto:Headers: {} 2022-09-12T20:03:28.277 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:03:28.277 DEBUG:boto:Port: 8001 2022-09-12T20:03:28.278 DEBUG:boto:Params: {} 2022-09-12T20:03:28.279 DEBUG:boto:Token: None 2022-09-12T20:03:28.279 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/ 2022-09-12T20:03:28.281 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xwiJNY10Fp++N3JCfFskuRWPy+c= 2022-09-12T20:03:28.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xwiJNY10Fp++N3JCfFskuRWPy+c=', 'Content-Length': '0'} 2022-09-12T20:03:28.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000438057ddb0110a5c-00631f9090-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.286 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:28.287 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:28.288 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:28.288 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:03:28.289 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:03:28.289 DEBUG:boto:Method: GET 2022-09-12T20:03:28.290 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:03:28.290 DEBUG:boto:Data: 2022-09-12T20:03:28.291 DEBUG:boto:Headers: {} 2022-09-12T20:03:28.291 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:03:28.292 DEBUG:boto:Port: 8000 2022-09-12T20:03:28.293 DEBUG:boto:Params: {} 2022-09-12T20:03:28.293 DEBUG:boto:Token: None 2022-09-12T20:03:28.294 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/ 2022-09-12T20:03:28.294 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xwiJNY10Fp++N3JCfFskuRWPy+c= 2022-09-12T20:03:28.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xwiJNY10Fp++N3JCfFskuRWPy+c=', 'Content-Length': '0'} 2022-09-12T20:03:28.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000257cfef1cc9ac23-00631f9090-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.299 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:03:28.299 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-75 zones={test-zone3, test-zone2} 2022-09-12T20:03:28.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:03:28.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:03:28.301 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T20:03:28.363 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:03:28.364 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:03:28.428 INFO:teuthology.orchestra.run.smithi035.stdout:tenant: 2022-09-12T20:03:28.428 INFO:teuthology.orchestra.run.smithi035.stdout:bucket name: jdpiak-75 2022-09-12T20:03:28.433 INFO:teuthology.orchestra.run.smithi035.stdout:total entries: 5 2022-09-12T20:03:28.478 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:28.479 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:28.479 DEBUG:boto:Method: HEAD 2022-09-12T20:03:28.480 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:03:28.480 DEBUG:boto:Data: 2022-09-12T20:03:28.481 DEBUG:boto:Headers: {} 2022-09-12T20:03:28.482 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:28.483 DEBUG:boto:Port: 8000 2022-09-12T20:03:28.483 DEBUG:boto:Params: {} 2022-09-12T20:03:28.484 DEBUG:boto:Token: None 2022-09-12T20:03:28.484 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/ 2022-09-12T20:03:28.485 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iZM59Y2c3mJ4C/wPhPJsnKfBey8= 2022-09-12T20:03:28.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iZM59Y2c3mJ4C/wPhPJsnKfBey8=', 'Content-Length': '0'} 2022-09-12T20:03:28.489 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000919d05f0f1706b91-00631f9090-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.490 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:03:28.490 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:03:28.491 DEBUG:boto:Method: PUT 2022-09-12T20:03:28.491 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:03:28.492 DEBUG:boto:Data: 2022-09-12T20:03:28.492 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:03:28.493 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:28.493 DEBUG:boto:Port: 8000 2022-09-12T20:03:28.494 DEBUG:boto:Params: {} 2022-09-12T20:03:28.495 DEBUG:boto:Token: None 2022-09-12T20:03:28.495 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/obj17 2022-09-12T20:03:28.496 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AoYAZ6a6gELfWi9cwk1DYQW5LKM= 2022-09-12T20:03:28.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AoYAZ6a6gELfWi9cwk1DYQW5LKM='} 2022-09-12T20:03:28.504 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009bed9bbc2962e579-00631f9090-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.504 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:28.505 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:28.505 DEBUG:boto:Method: HEAD 2022-09-12T20:03:28.506 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:03:28.506 DEBUG:boto:Data: 2022-09-12T20:03:28.507 DEBUG:boto:Headers: {} 2022-09-12T20:03:28.507 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:28.508 DEBUG:boto:Port: 8000 2022-09-12T20:03:28.508 DEBUG:boto:Params: {} 2022-09-12T20:03:28.509 DEBUG:boto:Token: None 2022-09-12T20:03:28.510 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/ 2022-09-12T20:03:28.510 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iZM59Y2c3mJ4C/wPhPJsnKfBey8= 2022-09-12T20:03:28.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iZM59Y2c3mJ4C/wPhPJsnKfBey8=', 'Content-Length': '0'} 2022-09-12T20:03:28.514 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006de38dfcb570f972-00631f9090-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.515 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:03:28.516 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:03:28.516 DEBUG:boto:Method: PUT 2022-09-12T20:03:28.517 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:03:28.517 DEBUG:boto:Data: 2022-09-12T20:03:28.518 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:03:28.519 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:28.519 DEBUG:boto:Port: 8000 2022-09-12T20:03:28.520 DEBUG:boto:Params: {} 2022-09-12T20:03:28.520 DEBUG:boto:Token: None 2022-09-12T20:03:28.521 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/obj34 2022-09-12T20:03:28.522 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fSxnTzh3WgxZ87eeht9xpEt53dU= 2022-09-12T20:03:28.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fSxnTzh3WgxZ87eeht9xpEt53dU='} 2022-09-12T20:03:28.529 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000574c0b5325c860c7-00631f9090-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.530 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:28.530 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:28.531 DEBUG:boto:Method: HEAD 2022-09-12T20:03:28.531 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:03:28.532 DEBUG:boto:Data: 2022-09-12T20:03:28.533 DEBUG:boto:Headers: {} 2022-09-12T20:03:28.534 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:28.534 DEBUG:boto:Port: 8000 2022-09-12T20:03:28.535 DEBUG:boto:Params: {} 2022-09-12T20:03:28.535 DEBUG:boto:Token: None 2022-09-12T20:03:28.536 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/ 2022-09-12T20:03:28.537 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iZM59Y2c3mJ4C/wPhPJsnKfBey8= 2022-09-12T20:03:28.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iZM59Y2c3mJ4C/wPhPJsnKfBey8=', 'Content-Length': '0'} 2022-09-12T20:03:28.540 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d452231d67de550-00631f9090-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.540 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:03:28.541 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:03:28.541 DEBUG:boto:Method: PUT 2022-09-12T20:03:28.542 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:03:28.549 DEBUG:boto:Data: 2022-09-12T20:03:28.550 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:03:28.551 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:28.551 DEBUG:boto:Port: 8000 2022-09-12T20:03:28.552 DEBUG:boto:Params: {} 2022-09-12T20:03:28.552 DEBUG:boto:Token: None 2022-09-12T20:03:28.553 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/obj51 2022-09-12T20:03:28.553 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oev1BwI1XwMi40Z+GCQCdWCmpZE= 2022-09-12T20:03:28.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oev1BwI1XwMi40Z+GCQCdWCmpZE='} 2022-09-12T20:03:28.602 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e6543c05af2e109-00631f9090-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.603 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:28.604 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:28.604 DEBUG:boto:Method: HEAD 2022-09-12T20:03:28.605 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:03:28.605 DEBUG:boto:Data: 2022-09-12T20:03:28.606 DEBUG:boto:Headers: {} 2022-09-12T20:03:28.606 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:28.607 DEBUG:boto:Port: 8000 2022-09-12T20:03:28.607 DEBUG:boto:Params: {} 2022-09-12T20:03:28.608 DEBUG:boto:Token: None 2022-09-12T20:03:28.608 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/ 2022-09-12T20:03:28.609 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iZM59Y2c3mJ4C/wPhPJsnKfBey8= 2022-09-12T20:03:28.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iZM59Y2c3mJ4C/wPhPJsnKfBey8=', 'Content-Length': '0'} 2022-09-12T20:03:28.613 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d343d23cc63ca12-00631f9090-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.613 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:03:28.614 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:03:28.615 DEBUG:boto:Method: PUT 2022-09-12T20:03:28.615 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:03:28.616 DEBUG:boto:Data: 2022-09-12T20:03:28.616 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:03:28.617 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:28.617 DEBUG:boto:Port: 8000 2022-09-12T20:03:28.618 DEBUG:boto:Params: {} 2022-09-12T20:03:28.619 DEBUG:boto:Token: None 2022-09-12T20:03:28.619 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/obj68 2022-09-12T20:03:28.620 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rVmZFi810Ot5Ig1yZTrIzPAhbkU= 2022-09-12T20:03:28.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rVmZFi810Ot5Ig1yZTrIzPAhbkU='} 2022-09-12T20:03:28.626 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f1500b7b37070c36-00631f9090-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.626 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:03:28.627 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:03:28.627 DEBUG:boto:Method: HEAD 2022-09-12T20:03:28.628 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:03:28.628 DEBUG:boto:Data: 2022-09-12T20:03:28.629 DEBUG:boto:Headers: {} 2022-09-12T20:03:28.629 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:28.630 DEBUG:boto:Port: 8000 2022-09-12T20:03:28.631 DEBUG:boto:Params: {} 2022-09-12T20:03:28.631 DEBUG:boto:Token: None 2022-09-12T20:03:28.632 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/ 2022-09-12T20:03:28.632 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iZM59Y2c3mJ4C/wPhPJsnKfBey8= 2022-09-12T20:03:28.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iZM59Y2c3mJ4C/wPhPJsnKfBey8=', 'Content-Length': '0'} 2022-09-12T20:03:28.635 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d249b16c7f1b370d-00631f9090-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.636 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:03:28.637 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:03:28.637 DEBUG:boto:Method: PUT 2022-09-12T20:03:28.638 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:03:28.638 DEBUG:boto:Data: 2022-09-12T20:03:28.639 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:03:28.639 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:03:28.640 DEBUG:boto:Port: 8000 2022-09-12T20:03:28.640 DEBUG:boto:Params: {} 2022-09-12T20:03:28.641 DEBUG:boto:Token: None 2022-09-12T20:03:28.642 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:03:28 GMT /jdpiak-75/obj85 2022-09-12T20:03:28.642 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:53xrQQRX16SZSx3W+wPlTMQY5Tk= 2022-09-12T20:03:28.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:03:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:53xrQQRX16SZSx3W+wPlTMQY5Tk='} 2022-09-12T20:03:28.686 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0892386b4d4daa5-00631f9090-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:03:28.687 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-75 2022-09-12T20:03:28.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:03:28.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:03:28.689 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T20:03:28.752 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:03:28.753 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:03:28.792 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:03:28.791+0000 7f20b8d32600 1 waiting to reach latest gen 1, on 0.. 2022-09-12T20:03:33.793 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:03:33.792+0000 7f20b8d32600 1 waiting to reach latest gen 1, on 0.. 2022-09-12T20:03:38.794 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:03:38.791+0000 7f20b8d32600 1 waiting to reach latest gen 1, on 0.. 2022-09-12T20:03:43.796 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:03:43.793+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:03:43.796 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , ] 2022-09-12T20:03:43.797 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:03:48.797 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:03:48.795+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:03:48.798 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , ] 2022-09-12T20:03:48.798 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:03:53.799 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:03:53.797+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:03:53.799 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , ] 2022-09-12T20:03:53.800 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:03:58.801 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:03:58.798+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:03:58.801 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , ] 2022-09-12T20:03:58.802 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:04:03.802 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:04:03.800+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:04:03.803 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , ] 2022-09-12T20:04:03.803 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:04:08.804 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:04:08.802+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:04:08.804 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , ] 2022-09-12T20:04:08.805 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:04:13.805 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:04:13.804+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:04:13.806 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , ] 2022-09-12T20:04:13.806 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:04:18.807 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:04:18.806+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:04:18.808 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , ] 2022-09-12T20:04:18.808 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:04:23.809 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:04:23.807+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:04:23.809 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , 00000000001.70.5, , , , , , , 00000000001.36.5, , , ] 2022-09-12T20:04:23.810 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:04:28.811 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:04:28.808+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:04:28.811 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , 00000000001.70.5, , , , , , , 00000000001.36.5, , , ] 2022-09-12T20:04:28.811 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:04:33.812 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:04:33.810+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:04:33.813 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , 00000000001.70.5, , , , , , , 00000000001.36.5, , , ] 2022-09-12T20:04:33.813 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:04:38.814 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:04:38.812+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:04:38.814 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , 00000000001.70.5, , , , , , , 00000000001.36.5, , , ] 2022-09-12T20:04:38.815 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:04:43.816 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:04:43.813+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:04:43.816 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , , 00000000001.36.5, , , ] 2022-09-12T20:04:43.817 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:04:48.817 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:04:48.815+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:04:48.818 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , , 00000000001.36.5, , , ] 2022-09-12T20:04:48.818 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:04:53.819 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:04:53.816+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:04:53.819 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , , 00000000001.36.5, , , ] 2022-09-12T20:04:53.820 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:04:58.820 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:04:58.819+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:04:58.821 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , , 00000000001.36.5, , , ] 2022-09-12T20:04:58.821 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:05:03.822 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:05:03.821+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:05:03.823 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , , 00000000001.36.5, , , ] 2022-09-12T20:05:03.823 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:05:08.824 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:05:08.821+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:05:08.824 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , , 00000000001.36.5, , , ] 2022-09-12T20:05:08.824 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:05:13.825 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:05:13.823+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:05:13.826 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , , 00000000001.36.5, , , ] 2022-09-12T20:05:13.826 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:05:18.827 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:05:18.825+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:05:18.827 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , , 00000000001.36.5, , , ] 2022-09-12T20:05:18.828 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:05:23.829 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:05:23.826+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:05:23.829 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , , 00000000001.36.5, , , ] 2022-09-12T20:05:23.829 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:05:28.830 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:05:28.828+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:05:28.831 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , , 00000000001.36.5, , , ] 2022-09-12T20:05:28.831 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:05:33.832 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:05:33.830+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:05:33.832 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , , 00000000001.36.5, , , ] 2022-09-12T20:05:33.833 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:05:38.834 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:05:38.831+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:05:38.834 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , , 00000000001.36.5, , , ] 2022-09-12T20:05:38.835 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:05:43.836 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:05:43.833+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:05:43.836 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , , 00000000001.36.5, , , ] 2022-09-12T20:05:43.837 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:05:48.837 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:05:48.836+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:05:48.838 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , , 00000000001.36.5, , , ] 2022-09-12T20:05:48.838 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:05:53.839 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:05:53.837+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:05:53.839 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , , 00000000001.36.5, , , ] 2022-09-12T20:05:53.840 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:05:58.840 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:05:58.838+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:05:58.841 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , , 00000000001.36.5, , , ] 2022-09-12T20:05:58.841 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:06:03.842 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:06:03.840+0000 7f20b8d32600 1 waiting for incremental sync to catch up: 2022-09-12T20:06:03.843 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , , 00000000001.36.5, , , ] 2022-09-12T20:06:03.843 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:06:08.844 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:06:08.842+0000 7f20b8d32600 1 bucket sync caught up with source: 2022-09-12T20:06:08.845 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:06:08.846 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:06:08.846 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:06:08.842+0000 7f20b8d32600 0 bucket checkpoint complete 2022-09-12T20:06:08.864 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-75 2022-09-12T20:06:08.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:06:08.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:06:08.869 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:06:08.940 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:06:08.940 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:06:08.981 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:06:08.979+0000 7fd94ae35600 1 bucket sync caught up with source: 2022-09-12T20:06:08.981 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:06:08.981 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:06:08.982 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:06:08.979+0000 7fd94ae35600 0 bucket checkpoint complete 2022-09-12T20:06:08.996 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-75 2022-09-12T20:06:08.996 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:06:08.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:06:08.998 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T20:06:09.065 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:06:09.065 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:06:09.116 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:06:09.114+0000 7f410a20e600 1 bucket sync caught up with source: 2022-09-12T20:06:09.117 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:06:09.117 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000001.84.5, , , , , , 00000000001.70.5, , 00000000001.71.5, , , , 00000000001.43.5, 00000000001.36.5, , , ] 2022-09-12T20:06:09.117 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:06:09.114+0000 7f410a20e600 0 bucket checkpoint complete 2022-09-12T20:06:09.132 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-75 2022-09-12T20:06:09.133 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:06:09.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:06:09.134 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:06:09.195 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:06:09.195 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:06:09.238 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:06:09.237+0000 7f4371b50600 1 bucket sync caught up with source: 2022-09-12T20:06:09.239 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.635.5, 00000000001.831.5, , , 00000000001.911.5, , , 00000000001.544.5, 00000000003.913.5, 00000000003.574.5] 2022-09-12T20:06:09.239 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000001.635.5, 00000000001.831.5, , , 00000000001.911.5, , , 00000000001.544.5, 00000000003.913.5, 00000000003.574.5] 2022-09-12T20:06:09.239 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:06:09.237+0000 7f4371b50600 0 bucket checkpoint complete 2022-09-12T20:06:09.256 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-75 2022-09-12T20:06:09.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:06:09.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:06:09.258 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:06:09.315 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:06:09.316 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:06:09.358 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:06:09.357+0000 7f0d60a35600 1 bucket sync caught up with source: 2022-09-12T20:06:09.359 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.635.5, 00000000001.831.5, , , 00000000001.911.5, , , 00000000001.544.5, 00000000003.913.5, 00000000003.574.5] 2022-09-12T20:06:09.359 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000001.635.5, 00000000001.831.5, , , 00000000001.911.5, , , 00000000001.544.5, 00000000003.913.5, 00000000003.574.5] 2022-09-12T20:06:09.360 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:06:09.357+0000 7f0d60a35600 0 bucket checkpoint complete 2022-09-12T20:06:09.372 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-75 2022-09-12T20:06:09.373 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-75 2022-09-12T20:06:09.373 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:06:09.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:06:09.374 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:06:09.432 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:06:09.432 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:06:09.470 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:06:09.468+0000 7f2c2d929600 1 bucket sync caught up with source: 2022-09-12T20:06:09.471 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000001.292.5, 00000000001.331.5, , , 00000000001.527.5, , , 00000000001.314.5, 00000000003.529.5, 00000000003.488.5] 2022-09-12T20:06:09.471 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000001.292.5, 00000000001.331.5, , , 00000000001.527.5, , , 00000000001.314.5, 00000000003.529.5, 00000000003.488.5] 2022-09-12T20:06:09.471 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:06:09.468+0000 7f2c2d929600 0 bucket checkpoint complete 2022-09-12T20:06:09.484 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-75 2022-09-12T20:06:09.485 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:06:09.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:06:09.486 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T20:06:09.547 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:06:09.548 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:06:09.597 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:06:09.595+0000 7fdb1c1e8600 1 bucket sync caught up with source: 2022-09-12T20:06:09.598 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000001.292.5, 00000000001.331.5, , , 00000000001.527.5, , , 00000000001.314.5, 00000000003.529.5, 00000000003.488.5] 2022-09-12T20:06:09.598 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000001.292.5, 00000000001.331.5, , , 00000000001.527.5, , , 00000000001.314.5, 00000000003.529.5, 00000000003.488.5] 2022-09-12T20:06:09.598 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:06:09.595+0000 7fdb1c1e8600 0 bucket checkpoint complete 2022-09-12T20:06:09.616 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-75 2022-09-12T20:06:09.616 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-75 zones={test-zone2, test-zone1} 2022-09-12T20:06:09.617 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:09.618 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:09.618 DEBUG:boto:Method: HEAD 2022-09-12T20:06:09.619 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:06:09.619 DEBUG:boto:Data: 2022-09-12T20:06:09.620 DEBUG:boto:Headers: {} 2022-09-12T20:06:09.620 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:09.621 DEBUG:boto:Port: 8000 2022-09-12T20:06:09.622 DEBUG:boto:Params: {} 2022-09-12T20:06:09.623 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T20:06:09.624 DEBUG:boto:Token: None 2022-09-12T20:06:09.625 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:09 GMT /jdpiak-75/ 2022-09-12T20:06:09.626 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iPYo9lrIL7gdc/iem+LoQavj+Ns= 2022-09-12T20:06:09.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iPYo9lrIL7gdc/iem+LoQavj+Ns=', 'Content-Length': '0'} 2022-09-12T20:06:09.631 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c585025428c7531-00631f9131-189d-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:06:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:09.632 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:09.633 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:09.633 DEBUG:boto:Method: HEAD 2022-09-12T20:06:09.634 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:06:09.634 DEBUG:boto:Data: 2022-09-12T20:06:09.635 DEBUG:boto:Headers: {} 2022-09-12T20:06:09.635 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:09.636 DEBUG:boto:Port: 8000 2022-09-12T20:06:09.636 DEBUG:boto:Params: {} 2022-09-12T20:06:09.637 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T20:06:09.638 DEBUG:boto:Token: None 2022-09-12T20:06:09.638 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:09 GMT /jdpiak-75/ 2022-09-12T20:06:09.639 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iPYo9lrIL7gdc/iem+LoQavj+Ns= 2022-09-12T20:06:09.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iPYo9lrIL7gdc/iem+LoQavj+Ns=', 'Content-Length': '0'} 2022-09-12T20:06:09.644 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ced33df3e1c7292b-00631f9131-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:06:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:09.644 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T20:06:09.645 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:09.646 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:09.646 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:09.647 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:09.647 DEBUG:boto:Method: GET 2022-09-12T20:06:09.648 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:09.648 DEBUG:boto:Data: 2022-09-12T20:06:09.649 DEBUG:boto:Headers: {} 2022-09-12T20:06:09.649 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:09.650 DEBUG:boto:Port: 8000 2022-09-12T20:06:09.650 DEBUG:boto:Params: {} 2022-09-12T20:06:09.651 DEBUG:boto:Token: None 2022-09-12T20:06:09.652 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:09 GMT /jdpiak-75/?versions 2022-09-12T20:06:09.652 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9xBN+CYGwJbOjxARfo76v5cYFQA= 2022-09-12T20:06:09.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9xBN+CYGwJbOjxARfo76v5cYFQA=', 'Content-Length': '0'} 2022-09-12T20:06:09.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4f2cff84e9fb240-00631f9131-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:09.656 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:09.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:06:09.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:06:09.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:06:09.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:06:09.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:06:09.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:06:09.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:06:09.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:06:09.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:06:09.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:06:09.663 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T20:06:09.663 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:09.664 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:09.664 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:09.665 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:09.665 DEBUG:boto:Method: GET 2022-09-12T20:06:09.666 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:09.667 DEBUG:boto:Data: 2022-09-12T20:06:09.667 DEBUG:boto:Headers: {} 2022-09-12T20:06:09.668 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:09.668 DEBUG:boto:Port: 8000 2022-09-12T20:06:09.669 DEBUG:boto:Params: {} 2022-09-12T20:06:09.669 DEBUG:boto:Token: None 2022-09-12T20:06:09.670 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:09 GMT /jdpiak-75/?versions 2022-09-12T20:06:09.671 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9xBN+CYGwJbOjxARfo76v5cYFQA= 2022-09-12T20:06:09.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9xBN+CYGwJbOjxARfo76v5cYFQA=', 'Content-Length': '0'} 2022-09-12T20:06:09.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7bfe7ef512b6366-00631f9131-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:09.675 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:09.676 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:06:09.676 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:06:09.677 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:06:09.677 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:06:09.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:06:09.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:06:09.679 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:06:09.679 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:06:09.679 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:06:09.680 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:06:09.680 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:09.681 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:09.682 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:09.682 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:09.683 DEBUG:boto:Method: GET 2022-09-12T20:06:09.683 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:09.684 DEBUG:boto:Data: 2022-09-12T20:06:09.684 DEBUG:boto:Headers: {} 2022-09-12T20:06:09.685 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:09.685 DEBUG:boto:Port: 8000 2022-09-12T20:06:09.686 DEBUG:boto:Params: {} 2022-09-12T20:06:09.686 DEBUG:boto:Token: None 2022-09-12T20:06:09.687 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:09 GMT /jdpiak-75/?versions 2022-09-12T20:06:09.687 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9xBN+CYGwJbOjxARfo76v5cYFQA= 2022-09-12T20:06:09.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9xBN+CYGwJbOjxARfo76v5cYFQA=', 'Content-Length': '0'} 2022-09-12T20:06:09.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002514085f7813f36f-00631f9131-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:09.692 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:09.693 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:09.694 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:09.694 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:09.695 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:09.695 DEBUG:boto:Method: GET 2022-09-12T20:06:09.696 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:09.696 DEBUG:boto:Data: 2022-09-12T20:06:09.697 DEBUG:boto:Headers: {} 2022-09-12T20:06:09.697 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:09.698 DEBUG:boto:Port: 8000 2022-09-12T20:06:09.699 DEBUG:boto:Params: {} 2022-09-12T20:06:09.699 DEBUG:boto:Token: None 2022-09-12T20:06:09.700 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:09 GMT /jdpiak-75/?versions 2022-09-12T20:06:09.700 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9xBN+CYGwJbOjxARfo76v5cYFQA= 2022-09-12T20:06:09.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9xBN+CYGwJbOjxARfo76v5cYFQA=', 'Content-Length': '0'} 2022-09-12T20:06:09.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5468c59c03a0b45-00631f9131-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:09.704 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:09.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:06:09.706 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:09.707 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:09.708 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:09.708 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:09.709 DEBUG:boto:Method: GET 2022-09-12T20:06:09.709 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:09.710 DEBUG:boto:Data: 2022-09-12T20:06:09.710 DEBUG:boto:Headers: {} 2022-09-12T20:06:09.711 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:09.711 DEBUG:boto:Port: 8000 2022-09-12T20:06:09.712 DEBUG:boto:Params: {} 2022-09-12T20:06:09.713 DEBUG:boto:Token: None 2022-09-12T20:06:09.713 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:09 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:09.714 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZoBZQupzoFNg/jYN8sVojg2cq1g= 2022-09-12T20:06:09.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZoBZQupzoFNg/jYN8sVojg2cq1g=', 'Content-Length': '0'} 2022-09-12T20:06:09.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e44e080decb3893-00631f9131-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:09.758 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:09.759 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:09.760 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:09.760 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:09.761 DEBUG:boto:Method: GET 2022-09-12T20:06:09.761 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:09.762 DEBUG:boto:Data: 2022-09-12T20:06:09.763 DEBUG:boto:Headers: {} 2022-09-12T20:06:09.763 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:09.764 DEBUG:boto:Port: 8000 2022-09-12T20:06:09.764 DEBUG:boto:Params: {} 2022-09-12T20:06:09.765 DEBUG:boto:Token: None 2022-09-12T20:06:09.766 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:09 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:09.766 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZoBZQupzoFNg/jYN8sVojg2cq1g= 2022-09-12T20:06:09.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZoBZQupzoFNg/jYN8sVojg2cq1g=', 'Content-Length': '0'} 2022-09-12T20:06:09.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f51f88c4fe1e2583-00631f9131-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:09.770 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:09.771 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:09.771 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:09.772 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:09.773 DEBUG:boto:Method: HEAD 2022-09-12T20:06:09.773 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:09.774 DEBUG:boto:Data: 2022-09-12T20:06:09.775 DEBUG:boto:Headers: {} 2022-09-12T20:06:09.775 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:09.776 DEBUG:boto:Port: 8000 2022-09-12T20:06:09.776 DEBUG:boto:Params: {} 2022-09-12T20:06:09.777 DEBUG:boto:Token: None 2022-09-12T20:06:09.777 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:09 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:09.778 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UmGG5FlWlJ14RIBBIxj3p4Z0uEg= 2022-09-12T20:06:09.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UmGG5FlWlJ14RIBBIxj3p4Z0uEg=', 'Content-Length': '0'} 2022-09-12T20:06:09.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a55d968eb13ecca-00631f9131-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:09.781 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:09.782 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:09.783 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:09.783 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:09.784 DEBUG:boto:Method: HEAD 2022-09-12T20:06:09.784 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:09.785 DEBUG:boto:Data: 2022-09-12T20:06:09.785 DEBUG:boto:Headers: {} 2022-09-12T20:06:09.786 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:09.786 DEBUG:boto:Port: 8000 2022-09-12T20:06:09.787 DEBUG:boto:Params: {} 2022-09-12T20:06:09.787 DEBUG:boto:Token: None 2022-09-12T20:06:09.788 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:09 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:09.788 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UmGG5FlWlJ14RIBBIxj3p4Z0uEg= 2022-09-12T20:06:09.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UmGG5FlWlJ14RIBBIxj3p4Z0uEg=', 'Content-Length': '0'} 2022-09-12T20:06:09.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb908209ca53d489-00631f9131-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:09.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:06:09.792 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:09.793 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:09.794 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:09.794 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:09.795 DEBUG:boto:Method: GET 2022-09-12T20:06:09.795 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:09.796 DEBUG:boto:Data: 2022-09-12T20:06:09.796 DEBUG:boto:Headers: {} 2022-09-12T20:06:09.797 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:09.797 DEBUG:boto:Port: 8000 2022-09-12T20:06:09.798 DEBUG:boto:Params: {} 2022-09-12T20:06:09.799 DEBUG:boto:Token: None 2022-09-12T20:06:09.799 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:09 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:09.800 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZoBZQupzoFNg/jYN8sVojg2cq1g= 2022-09-12T20:06:09.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZoBZQupzoFNg/jYN8sVojg2cq1g=', 'Content-Length': '0'} 2022-09-12T20:06:09.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000383a8c37b6aded29-00631f9131-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:09.803 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:09.804 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:09.805 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:09.805 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:09.806 DEBUG:boto:Method: GET 2022-09-12T20:06:09.807 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:09.807 DEBUG:boto:Data: 2022-09-12T20:06:09.808 DEBUG:boto:Headers: {} 2022-09-12T20:06:09.809 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:09.810 DEBUG:boto:Port: 8000 2022-09-12T20:06:09.811 DEBUG:boto:Params: {} 2022-09-12T20:06:09.811 DEBUG:boto:Token: None 2022-09-12T20:06:09.812 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:09 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:09.813 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZoBZQupzoFNg/jYN8sVojg2cq1g= 2022-09-12T20:06:09.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZoBZQupzoFNg/jYN8sVojg2cq1g=', 'Content-Length': '0'} 2022-09-12T20:06:09.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1455a0b1b236162-00631f9131-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:09.817 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:09.818 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:09.819 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:06:09.820 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:06:09.820 DEBUG:boto:Method: GET 2022-09-12T20:06:09.821 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:06:09.821 DEBUG:boto:Data: 2022-09-12T20:06:09.822 DEBUG:boto:Headers: {} 2022-09-12T20:06:09.823 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:09.823 DEBUG:boto:Port: 8000 2022-09-12T20:06:09.824 DEBUG:boto:Params: {} 2022-09-12T20:06:09.825 DEBUG:boto:Token: None 2022-09-12T20:06:09.825 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:09 GMT /jdpiak-75/ 2022-09-12T20:06:09.826 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1S7rFgsG7FILzNY6LM0Gk9Xe4mY= 2022-09-12T20:06:09.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1S7rFgsG7FILzNY6LM0Gk9Xe4mY=', 'Content-Length': '0'} 2022-09-12T20:06:09.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdb0326f7ec17915-00631f9131-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:09.830 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:09.831 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:09.831 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:09.832 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:06:09.832 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:06:09.833 DEBUG:boto:Method: GET 2022-09-12T20:06:09.833 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:06:09.834 DEBUG:boto:Data: 2022-09-12T20:06:09.834 DEBUG:boto:Headers: {} 2022-09-12T20:06:09.835 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:09.835 DEBUG:boto:Port: 8000 2022-09-12T20:06:09.836 DEBUG:boto:Params: {} 2022-09-12T20:06:09.837 DEBUG:boto:Token: None 2022-09-12T20:06:09.837 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:09 GMT /jdpiak-75/ 2022-09-12T20:06:09.838 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1S7rFgsG7FILzNY6LM0Gk9Xe4mY= 2022-09-12T20:06:09.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1S7rFgsG7FILzNY6LM0Gk9Xe4mY=', 'Content-Length': '0'} 2022-09-12T20:06:09.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eae1b83a6688f379-00631f9131-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:09.843 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:09.845 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:09.846 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:09.847 DEBUG:boto:Method: HEAD 2022-09-12T20:06:09.848 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:06:09.848 DEBUG:boto:Data: 2022-09-12T20:06:09.849 DEBUG:boto:Headers: {} 2022-09-12T20:06:09.858 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:09.858 DEBUG:boto:Port: 8000 2022-09-12T20:06:09.859 DEBUG:boto:Params: {} 2022-09-12T20:06:09.859 DEBUG:boto:Token: None 2022-09-12T20:06:09.860 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:09 GMT /jdpiak-75/obj11 2022-09-12T20:06:09.861 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HrU8otNmxEVNC0F1ArRGj+W9AXU= 2022-09-12T20:06:09.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HrU8otNmxEVNC0F1ArRGj+W9AXU=', 'Content-Length': '0'} 2022-09-12T20:06:09.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010578710836aceae-00631f9131-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:09.864 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:09.864 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:09.865 DEBUG:boto:Method: HEAD 2022-09-12T20:06:09.865 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:06:09.866 DEBUG:boto:Data: 2022-09-12T20:06:09.867 DEBUG:boto:Headers: {} 2022-09-12T20:06:09.867 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:09.868 DEBUG:boto:Port: 8000 2022-09-12T20:06:09.868 DEBUG:boto:Params: {} 2022-09-12T20:06:09.869 DEBUG:boto:Token: None 2022-09-12T20:06:09.869 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:09 GMT /jdpiak-75/obj11 2022-09-12T20:06:09.870 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HrU8otNmxEVNC0F1ArRGj+W9AXU= 2022-09-12T20:06:09.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HrU8otNmxEVNC0F1ArRGj+W9AXU=', 'Content-Length': '0'} 2022-09-12T20:06:09.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006584f280e0878b04-00631f9131-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:09.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:06:09.874 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:09.874 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:09.875 DEBUG:boto:Method: GET 2022-09-12T20:06:09.875 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:06:09.876 DEBUG:boto:Data: 2022-09-12T20:06:09.876 DEBUG:boto:Headers: {} 2022-09-12T20:06:09.877 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:09.878 DEBUG:boto:Port: 8000 2022-09-12T20:06:09.878 DEBUG:boto:Params: {} 2022-09-12T20:06:09.879 DEBUG:boto:Token: None 2022-09-12T20:06:09.879 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:09 GMT /jdpiak-75/obj11 2022-09-12T20:06:09.880 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sR6z5ZX1I22T/69m8dsYmOxI/6c= 2022-09-12T20:06:09.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sR6z5ZX1I22T/69m8dsYmOxI/6c=', 'Content-Length': '0'} 2022-09-12T20:06:09.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e661801dfcc2211e-00631f9131-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:09.883 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:09.884 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:09.884 DEBUG:boto:Method: GET 2022-09-12T20:06:09.885 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:06:09.885 DEBUG:boto:Data: 2022-09-12T20:06:09.886 DEBUG:boto:Headers: {} 2022-09-12T20:06:09.886 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:09.887 DEBUG:boto:Port: 8000 2022-09-12T20:06:09.887 DEBUG:boto:Params: {} 2022-09-12T20:06:09.888 DEBUG:boto:Token: None 2022-09-12T20:06:09.889 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:09 GMT /jdpiak-75/obj11 2022-09-12T20:06:09.889 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sR6z5ZX1I22T/69m8dsYmOxI/6c= 2022-09-12T20:06:09.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sR6z5ZX1I22T/69m8dsYmOxI/6c=', 'Content-Length': '0'} 2022-09-12T20:06:09.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032a1a9595c64779a-00631f9131-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:09.934 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:09.935 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:09.935 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:06:09.936 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:06:09.936 DEBUG:boto:Method: GET 2022-09-12T20:06:09.937 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:06:09.937 DEBUG:boto:Data: 2022-09-12T20:06:09.938 DEBUG:boto:Headers: {} 2022-09-12T20:06:09.939 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:09.939 DEBUG:boto:Port: 8000 2022-09-12T20:06:09.940 DEBUG:boto:Params: {} 2022-09-12T20:06:09.940 DEBUG:boto:Token: None 2022-09-12T20:06:09.941 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:09 GMT /jdpiak-75/ 2022-09-12T20:06:09.942 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1S7rFgsG7FILzNY6LM0Gk9Xe4mY= 2022-09-12T20:06:09.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1S7rFgsG7FILzNY6LM0Gk9Xe4mY=', 'Content-Length': '0'} 2022-09-12T20:06:09.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070728e6ea0017306-00631f9131-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:09.946 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:09.947 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:09.947 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:09.948 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:06:09.948 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:06:09.949 DEBUG:boto:Method: GET 2022-09-12T20:06:09.949 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:06:09.950 DEBUG:boto:Data: 2022-09-12T20:06:09.951 DEBUG:boto:Headers: {} 2022-09-12T20:06:09.951 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:09.952 DEBUG:boto:Port: 8000 2022-09-12T20:06:09.952 DEBUG:boto:Params: {} 2022-09-12T20:06:09.953 DEBUG:boto:Token: None 2022-09-12T20:06:09.953 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:09 GMT /jdpiak-75/ 2022-09-12T20:06:09.954 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1S7rFgsG7FILzNY6LM0Gk9Xe4mY= 2022-09-12T20:06:09.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1S7rFgsG7FILzNY6LM0Gk9Xe4mY=', 'Content-Length': '0'} 2022-09-12T20:06:09.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060acdea467e7f2cf-00631f9131-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:09.958 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:09.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:06:09.960 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:09.960 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:09.961 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:09.961 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:09.962 DEBUG:boto:Method: GET 2022-09-12T20:06:09.963 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:09.963 DEBUG:boto:Data: 2022-09-12T20:06:09.964 DEBUG:boto:Headers: {} 2022-09-12T20:06:09.964 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:09.965 DEBUG:boto:Port: 8000 2022-09-12T20:06:09.965 DEBUG:boto:Params: {} 2022-09-12T20:06:09.966 DEBUG:boto:Token: None 2022-09-12T20:06:09.966 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:09 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:09.967 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5heHm24IIrG5rqAq/j2tRvcN6EI= 2022-09-12T20:06:09.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5heHm24IIrG5rqAq/j2tRvcN6EI=', 'Content-Length': '0'} 2022-09-12T20:06:09.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf0bbd0dbb25a133-00631f9131-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:09.970 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:09.971 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:09.972 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:09.972 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:09.973 DEBUG:boto:Method: GET 2022-09-12T20:06:09.973 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:09.974 DEBUG:boto:Data: 2022-09-12T20:06:09.974 DEBUG:boto:Headers: {} 2022-09-12T20:06:09.975 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:09.975 DEBUG:boto:Port: 8000 2022-09-12T20:06:09.976 DEBUG:boto:Params: {} 2022-09-12T20:06:09.976 DEBUG:boto:Token: None 2022-09-12T20:06:09.977 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:09 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:09.978 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5heHm24IIrG5rqAq/j2tRvcN6EI= 2022-09-12T20:06:09.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5heHm24IIrG5rqAq/j2tRvcN6EI=', 'Content-Length': '0'} 2022-09-12T20:06:09.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001100e684e0390489-00631f9131-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:09.981 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:09.982 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:09.982 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:09.983 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:09.983 DEBUG:boto:Method: HEAD 2022-09-12T20:06:09.984 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:09.984 DEBUG:boto:Data: 2022-09-12T20:06:09.985 DEBUG:boto:Headers: {} 2022-09-12T20:06:09.985 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:09.986 DEBUG:boto:Port: 8000 2022-09-12T20:06:09.986 DEBUG:boto:Params: {} 2022-09-12T20:06:09.987 DEBUG:boto:Token: None 2022-09-12T20:06:09.987 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:09 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:09.988 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oN6m/Dc9pnU0sHikF3mJd64uiOw= 2022-09-12T20:06:09.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oN6m/Dc9pnU0sHikF3mJd64uiOw=', 'Content-Length': '0'} 2022-09-12T20:06:09.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac2f46d6f6ed3cf4-00631f9131-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:09.991 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:09.992 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:09.992 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:09.993 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:09.993 DEBUG:boto:Method: HEAD 2022-09-12T20:06:09.994 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:09.994 DEBUG:boto:Data: 2022-09-12T20:06:09.995 DEBUG:boto:Headers: {} 2022-09-12T20:06:09.995 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:09.996 DEBUG:boto:Port: 8000 2022-09-12T20:06:09.996 DEBUG:boto:Params: {} 2022-09-12T20:06:09.997 DEBUG:boto:Token: None 2022-09-12T20:06:09.997 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:09 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:09.998 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oN6m/Dc9pnU0sHikF3mJd64uiOw= 2022-09-12T20:06:09.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oN6m/Dc9pnU0sHikF3mJd64uiOw=', 'Content-Length': '0'} 2022-09-12T20:06:10.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084c9a2b3908d3622-00631f9131-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:06:10.002 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:10.002 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:10.003 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:10.003 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:10.004 DEBUG:boto:Method: GET 2022-09-12T20:06:10.005 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:10.005 DEBUG:boto:Data: 2022-09-12T20:06:10.006 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.006 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.007 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.007 DEBUG:boto:Params: {} 2022-09-12T20:06:10.008 DEBUG:boto:Token: None 2022-09-12T20:06:10.008 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:10.009 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yvdIqIJ2PkzKxYz3eBBeia+rhmY= 2022-09-12T20:06:10.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yvdIqIJ2PkzKxYz3eBBeia+rhmY=', 'Content-Length': '0'} 2022-09-12T20:06:10.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae9b382c669fc2a7-00631f9132-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.054 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:10.055 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:10.055 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:10.056 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:10.056 DEBUG:boto:Method: GET 2022-09-12T20:06:10.057 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:10.058 DEBUG:boto:Data: 2022-09-12T20:06:10.058 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.059 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.059 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.060 DEBUG:boto:Params: {} 2022-09-12T20:06:10.060 DEBUG:boto:Token: None 2022-09-12T20:06:10.061 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:10.062 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yvdIqIJ2PkzKxYz3eBBeia+rhmY= 2022-09-12T20:06:10.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yvdIqIJ2PkzKxYz3eBBeia+rhmY=', 'Content-Length': '0'} 2022-09-12T20:06:10.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bc1d310b50b2139-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.065 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:10.066 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:10.066 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:06:10.067 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:06:10.068 DEBUG:boto:Method: GET 2022-09-12T20:06:10.068 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:06:10.069 DEBUG:boto:Data: 2022-09-12T20:06:10.069 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.070 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.070 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.071 DEBUG:boto:Params: {} 2022-09-12T20:06:10.071 DEBUG:boto:Token: None 2022-09-12T20:06:10.072 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/ 2022-09-12T20:06:10.073 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A= 2022-09-12T20:06:10.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A=', 'Content-Length': '0'} 2022-09-12T20:06:10.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000665afadaa4ac03e4-00631f9132-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.076 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:10.077 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:10.078 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:10.078 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:06:10.079 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:06:10.079 DEBUG:boto:Method: GET 2022-09-12T20:06:10.080 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:06:10.080 DEBUG:boto:Data: 2022-09-12T20:06:10.081 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.081 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.082 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.082 DEBUG:boto:Params: {} 2022-09-12T20:06:10.083 DEBUG:boto:Token: None 2022-09-12T20:06:10.084 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/ 2022-09-12T20:06:10.084 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A= 2022-09-12T20:06:10.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A=', 'Content-Length': '0'} 2022-09-12T20:06:10.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2aa2fe7fbb8f9a3-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.089 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:10.089 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:10.090 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:10.091 DEBUG:boto:Method: HEAD 2022-09-12T20:06:10.091 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:06:10.092 DEBUG:boto:Data: 2022-09-12T20:06:10.092 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.093 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.093 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.094 DEBUG:boto:Params: {} 2022-09-12T20:06:10.094 DEBUG:boto:Token: None 2022-09-12T20:06:10.095 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj17 2022-09-12T20:06:10.095 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TAky5zoVLnr2JiL+NOlg73imZhA= 2022-09-12T20:06:10.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TAky5zoVLnr2JiL+NOlg73imZhA=', 'Content-Length': '0'} 2022-09-12T20:06:10.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d423761277af02ef-00631f9132-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.099 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:10.099 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:10.100 DEBUG:boto:Method: HEAD 2022-09-12T20:06:10.100 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:06:10.101 DEBUG:boto:Data: 2022-09-12T20:06:10.101 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.102 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.102 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.103 DEBUG:boto:Params: {} 2022-09-12T20:06:10.103 DEBUG:boto:Token: None 2022-09-12T20:06:10.104 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj17 2022-09-12T20:06:10.105 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TAky5zoVLnr2JiL+NOlg73imZhA= 2022-09-12T20:06:10.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TAky5zoVLnr2JiL+NOlg73imZhA=', 'Content-Length': '0'} 2022-09-12T20:06:10.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec38e0e5e38a78ec-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:06:10.108 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:10.109 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:10.109 DEBUG:boto:Method: GET 2022-09-12T20:06:10.110 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:06:10.111 DEBUG:boto:Data: 2022-09-12T20:06:10.111 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.112 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.112 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.113 DEBUG:boto:Params: {} 2022-09-12T20:06:10.113 DEBUG:boto:Token: None 2022-09-12T20:06:10.114 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj17 2022-09-12T20:06:10.114 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XyJFuKjWHCoh+UoFHgsrtkQ04VM= 2022-09-12T20:06:10.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XyJFuKjWHCoh+UoFHgsrtkQ04VM=', 'Content-Length': '0'} 2022-09-12T20:06:10.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c09f20cf1be663b-00631f9132-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.118 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:10.118 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:10.119 DEBUG:boto:Method: GET 2022-09-12T20:06:10.119 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:06:10.120 DEBUG:boto:Data: 2022-09-12T20:06:10.121 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.121 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.122 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.122 DEBUG:boto:Params: {} 2022-09-12T20:06:10.123 DEBUG:boto:Token: None 2022-09-12T20:06:10.123 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj17 2022-09-12T20:06:10.124 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XyJFuKjWHCoh+UoFHgsrtkQ04VM= 2022-09-12T20:06:10.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XyJFuKjWHCoh+UoFHgsrtkQ04VM=', 'Content-Length': '0'} 2022-09-12T20:06:10.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e12cc8c101ccfad-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.128 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:10.128 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:10.129 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:06:10.129 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:06:10.130 DEBUG:boto:Method: GET 2022-09-12T20:06:10.130 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:06:10.131 DEBUG:boto:Data: 2022-09-12T20:06:10.131 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.132 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.132 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.133 DEBUG:boto:Params: {} 2022-09-12T20:06:10.133 DEBUG:boto:Token: None 2022-09-12T20:06:10.134 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/ 2022-09-12T20:06:10.135 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A= 2022-09-12T20:06:10.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A=', 'Content-Length': '0'} 2022-09-12T20:06:10.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066769162caad1f50-00631f9132-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.138 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:10.139 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:10.140 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:10.140 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:06:10.141 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:06:10.141 DEBUG:boto:Method: GET 2022-09-12T20:06:10.142 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:06:10.142 DEBUG:boto:Data: 2022-09-12T20:06:10.143 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.143 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.144 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.144 DEBUG:boto:Params: {} 2022-09-12T20:06:10.145 DEBUG:boto:Token: None 2022-09-12T20:06:10.146 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/ 2022-09-12T20:06:10.146 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A= 2022-09-12T20:06:10.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A=', 'Content-Length': '0'} 2022-09-12T20:06:10.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cfeb20ab6bb7792-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.194 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:10.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:06:10.196 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:10.196 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:10.197 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:10.197 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:10.198 DEBUG:boto:Method: GET 2022-09-12T20:06:10.198 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:10.199 DEBUG:boto:Data: 2022-09-12T20:06:10.199 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.200 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.200 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.201 DEBUG:boto:Params: {} 2022-09-12T20:06:10.202 DEBUG:boto:Token: None 2022-09-12T20:06:10.202 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:10.203 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wRcHUfZScehgnqX8erRGf7XkphM= 2022-09-12T20:06:10.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wRcHUfZScehgnqX8erRGf7XkphM=', 'Content-Length': '0'} 2022-09-12T20:06:10.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ac2a093346ef6c5-00631f9132-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.207 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:10.207 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:10.208 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:10.208 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:10.209 DEBUG:boto:Method: GET 2022-09-12T20:06:10.209 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:10.210 DEBUG:boto:Data: 2022-09-12T20:06:10.210 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.211 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.211 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.212 DEBUG:boto:Params: {} 2022-09-12T20:06:10.213 DEBUG:boto:Token: None 2022-09-12T20:06:10.213 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:10.214 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wRcHUfZScehgnqX8erRGf7XkphM= 2022-09-12T20:06:10.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wRcHUfZScehgnqX8erRGf7XkphM=', 'Content-Length': '0'} 2022-09-12T20:06:10.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000632479a1d2ca7b1e-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.218 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:10.218 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:10.219 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:10.219 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:10.220 DEBUG:boto:Method: HEAD 2022-09-12T20:06:10.220 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:10.221 DEBUG:boto:Data: 2022-09-12T20:06:10.221 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.222 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.222 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.223 DEBUG:boto:Params: {} 2022-09-12T20:06:10.224 DEBUG:boto:Token: None 2022-09-12T20:06:10.224 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:10.225 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4olcAvslO8W1sl/W4lIccoKXX0Q= 2022-09-12T20:06:10.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4olcAvslO8W1sl/W4lIccoKXX0Q=', 'Content-Length': '0'} 2022-09-12T20:06:10.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015e0bf15d4bfd673-00631f9132-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.228 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:10.229 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:10.229 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:10.230 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:10.230 DEBUG:boto:Method: HEAD 2022-09-12T20:06:10.231 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:10.231 DEBUG:boto:Data: 2022-09-12T20:06:10.232 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.232 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.233 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.233 DEBUG:boto:Params: {} 2022-09-12T20:06:10.234 DEBUG:boto:Token: None 2022-09-12T20:06:10.234 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:10.235 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4olcAvslO8W1sl/W4lIccoKXX0Q= 2022-09-12T20:06:10.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4olcAvslO8W1sl/W4lIccoKXX0Q=', 'Content-Length': '0'} 2022-09-12T20:06:10.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb431deaf9b9bacb-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:06:10.239 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:10.239 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:10.240 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:10.240 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:10.241 DEBUG:boto:Method: GET 2022-09-12T20:06:10.241 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:10.242 DEBUG:boto:Data: 2022-09-12T20:06:10.243 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.243 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.244 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.244 DEBUG:boto:Params: {} 2022-09-12T20:06:10.245 DEBUG:boto:Token: None 2022-09-12T20:06:10.245 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:10.246 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wRcHUfZScehgnqX8erRGf7XkphM= 2022-09-12T20:06:10.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wRcHUfZScehgnqX8erRGf7XkphM=', 'Content-Length': '0'} 2022-09-12T20:06:10.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058da592eeac23652-00631f9132-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.249 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:10.250 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:10.251 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:10.251 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:10.252 DEBUG:boto:Method: GET 2022-09-12T20:06:10.252 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:10.253 DEBUG:boto:Data: 2022-09-12T20:06:10.253 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.254 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.254 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.255 DEBUG:boto:Params: {} 2022-09-12T20:06:10.256 DEBUG:boto:Token: None 2022-09-12T20:06:10.256 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:10.257 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wRcHUfZScehgnqX8erRGf7XkphM= 2022-09-12T20:06:10.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wRcHUfZScehgnqX8erRGf7XkphM=', 'Content-Length': '0'} 2022-09-12T20:06:10.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ca7aa6ad4adcc09-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.260 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:10.261 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:10.262 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:06:10.262 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:06:10.263 DEBUG:boto:Method: GET 2022-09-12T20:06:10.263 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:06:10.264 DEBUG:boto:Data: 2022-09-12T20:06:10.264 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.265 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.265 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.266 DEBUG:boto:Params: {} 2022-09-12T20:06:10.267 DEBUG:boto:Token: None 2022-09-12T20:06:10.267 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/ 2022-09-12T20:06:10.268 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A= 2022-09-12T20:06:10.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A=', 'Content-Length': '0'} 2022-09-12T20:06:10.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d24837eebd82e045-00631f9132-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.272 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:10.273 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:10.274 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:10.274 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:06:10.275 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:06:10.275 DEBUG:boto:Method: GET 2022-09-12T20:06:10.276 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:06:10.276 DEBUG:boto:Data: 2022-09-12T20:06:10.277 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.277 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.278 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.278 DEBUG:boto:Params: {} 2022-09-12T20:06:10.279 DEBUG:boto:Token: None 2022-09-12T20:06:10.280 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/ 2022-09-12T20:06:10.280 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A= 2022-09-12T20:06:10.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A=', 'Content-Length': '0'} 2022-09-12T20:06:10.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000386a91065704fc8b-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.285 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:10.286 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:10.286 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:10.287 DEBUG:boto:Method: HEAD 2022-09-12T20:06:10.287 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:06:10.288 DEBUG:boto:Data: 2022-09-12T20:06:10.288 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.289 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.289 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.290 DEBUG:boto:Params: {} 2022-09-12T20:06:10.290 DEBUG:boto:Token: None 2022-09-12T20:06:10.291 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj22 2022-09-12T20:06:10.292 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uKPHeBNw9JuuLNKn7rABEuAGoE0= 2022-09-12T20:06:10.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uKPHeBNw9JuuLNKn7rABEuAGoE0=', 'Content-Length': '0'} 2022-09-12T20:06:10.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f934188ab1a20aef-00631f9132-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.295 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:10.295 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:10.296 DEBUG:boto:Method: HEAD 2022-09-12T20:06:10.297 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:06:10.297 DEBUG:boto:Data: 2022-09-12T20:06:10.298 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.298 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.299 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.299 DEBUG:boto:Params: {} 2022-09-12T20:06:10.300 DEBUG:boto:Token: None 2022-09-12T20:06:10.301 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj22 2022-09-12T20:06:10.301 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uKPHeBNw9JuuLNKn7rABEuAGoE0= 2022-09-12T20:06:10.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uKPHeBNw9JuuLNKn7rABEuAGoE0=', 'Content-Length': '0'} 2022-09-12T20:06:10.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008894f0d0b24aa568-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:06:10.305 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:10.306 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:10.306 DEBUG:boto:Method: GET 2022-09-12T20:06:10.307 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:06:10.307 DEBUG:boto:Data: 2022-09-12T20:06:10.308 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.309 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.309 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.310 DEBUG:boto:Params: {} 2022-09-12T20:06:10.310 DEBUG:boto:Token: None 2022-09-12T20:06:10.311 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj22 2022-09-12T20:06:10.311 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bNZH4SEzzDcS55nYtmO1o+JXnfA= 2022-09-12T20:06:10.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bNZH4SEzzDcS55nYtmO1o+JXnfA=', 'Content-Length': '0'} 2022-09-12T20:06:10.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3c3df5f11eee3a1-00631f9132-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.358 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:10.359 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:10.359 DEBUG:boto:Method: GET 2022-09-12T20:06:10.360 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:06:10.360 DEBUG:boto:Data: 2022-09-12T20:06:10.361 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.362 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.362 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.363 DEBUG:boto:Params: {} 2022-09-12T20:06:10.364 DEBUG:boto:Token: None 2022-09-12T20:06:10.364 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj22 2022-09-12T20:06:10.365 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bNZH4SEzzDcS55nYtmO1o+JXnfA= 2022-09-12T20:06:10.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bNZH4SEzzDcS55nYtmO1o+JXnfA=', 'Content-Length': '0'} 2022-09-12T20:06:10.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076ad961aa1feb522-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.368 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:10.369 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:10.370 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:06:10.370 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:06:10.371 DEBUG:boto:Method: GET 2022-09-12T20:06:10.371 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:06:10.372 DEBUG:boto:Data: 2022-09-12T20:06:10.372 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.373 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.374 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.374 DEBUG:boto:Params: {} 2022-09-12T20:06:10.375 DEBUG:boto:Token: None 2022-09-12T20:06:10.376 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/ 2022-09-12T20:06:10.376 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A= 2022-09-12T20:06:10.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A=', 'Content-Length': '0'} 2022-09-12T20:06:10.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013561f08a8c6a770-00631f9132-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.380 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:10.381 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:10.382 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:10.382 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:06:10.383 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:06:10.383 DEBUG:boto:Method: GET 2022-09-12T20:06:10.384 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:06:10.384 DEBUG:boto:Data: 2022-09-12T20:06:10.385 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.385 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.386 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.387 DEBUG:boto:Params: {} 2022-09-12T20:06:10.387 DEBUG:boto:Token: None 2022-09-12T20:06:10.388 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/ 2022-09-12T20:06:10.389 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A= 2022-09-12T20:06:10.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A=', 'Content-Length': '0'} 2022-09-12T20:06:10.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032b69d511b4aa3ba-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.394 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:10.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:06:10.395 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:10.396 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:10.397 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:10.397 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:10.398 DEBUG:boto:Method: GET 2022-09-12T20:06:10.398 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:10.399 DEBUG:boto:Data: 2022-09-12T20:06:10.399 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.400 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.400 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.401 DEBUG:boto:Params: {} 2022-09-12T20:06:10.401 DEBUG:boto:Token: None 2022-09-12T20:06:10.402 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:10.403 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GP3WvfL5YjX2iBxvtN7Wl09/lag= 2022-09-12T20:06:10.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GP3WvfL5YjX2iBxvtN7Wl09/lag=', 'Content-Length': '0'} 2022-09-12T20:06:10.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5ab1ef7f45292cf-00631f9132-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.406 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:10.407 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:10.407 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:10.408 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:10.408 DEBUG:boto:Method: GET 2022-09-12T20:06:10.409 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:10.409 DEBUG:boto:Data: 2022-09-12T20:06:10.410 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.410 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.411 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.411 DEBUG:boto:Params: {} 2022-09-12T20:06:10.412 DEBUG:boto:Token: None 2022-09-12T20:06:10.413 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:10.413 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GP3WvfL5YjX2iBxvtN7Wl09/lag= 2022-09-12T20:06:10.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GP3WvfL5YjX2iBxvtN7Wl09/lag=', 'Content-Length': '0'} 2022-09-12T20:06:10.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7b617a928c10b11-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.417 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:10.417 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:10.418 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:10.418 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:10.419 DEBUG:boto:Method: HEAD 2022-09-12T20:06:10.419 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:10.420 DEBUG:boto:Data: 2022-09-12T20:06:10.421 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.421 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.422 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.422 DEBUG:boto:Params: {} 2022-09-12T20:06:10.423 DEBUG:boto:Token: None 2022-09-12T20:06:10.423 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:10.424 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eOaZvT/Tehb2q27J5OL7M472SzE= 2022-09-12T20:06:10.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eOaZvT/Tehb2q27J5OL7M472SzE=', 'Content-Length': '0'} 2022-09-12T20:06:10.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009820cd71429edee5-00631f9132-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.427 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:10.427 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:10.428 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:10.428 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:10.429 DEBUG:boto:Method: HEAD 2022-09-12T20:06:10.429 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:10.430 DEBUG:boto:Data: 2022-09-12T20:06:10.430 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.431 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.431 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.432 DEBUG:boto:Params: {} 2022-09-12T20:06:10.433 DEBUG:boto:Token: None 2022-09-12T20:06:10.433 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:10.434 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eOaZvT/Tehb2q27J5OL7M472SzE= 2022-09-12T20:06:10.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eOaZvT/Tehb2q27J5OL7M472SzE=', 'Content-Length': '0'} 2022-09-12T20:06:10.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9413397aa432146-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:06:10.437 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:10.438 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:10.438 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:10.439 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:10.440 DEBUG:boto:Method: GET 2022-09-12T20:06:10.440 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:10.441 DEBUG:boto:Data: 2022-09-12T20:06:10.441 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.442 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.442 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.443 DEBUG:boto:Params: {} 2022-09-12T20:06:10.443 DEBUG:boto:Token: None 2022-09-12T20:06:10.444 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:10.445 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GP3WvfL5YjX2iBxvtN7Wl09/lag= 2022-09-12T20:06:10.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GP3WvfL5YjX2iBxvtN7Wl09/lag=', 'Content-Length': '0'} 2022-09-12T20:06:10.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f8810c89dedd239-00631f9132-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.448 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:10.448 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:10.449 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:10.450 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:10.450 DEBUG:boto:Method: GET 2022-09-12T20:06:10.451 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:10.451 DEBUG:boto:Data: 2022-09-12T20:06:10.452 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.452 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.453 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.453 DEBUG:boto:Params: {} 2022-09-12T20:06:10.454 DEBUG:boto:Token: None 2022-09-12T20:06:10.454 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:10.455 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GP3WvfL5YjX2iBxvtN7Wl09/lag= 2022-09-12T20:06:10.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GP3WvfL5YjX2iBxvtN7Wl09/lag=', 'Content-Length': '0'} 2022-09-12T20:06:10.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004af6465cb132ffb9-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.458 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:10.459 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:10.459 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:06:10.460 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:06:10.461 DEBUG:boto:Method: GET 2022-09-12T20:06:10.461 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:06:10.462 DEBUG:boto:Data: 2022-09-12T20:06:10.462 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.463 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.463 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.464 DEBUG:boto:Params: {} 2022-09-12T20:06:10.465 DEBUG:boto:Token: None 2022-09-12T20:06:10.465 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/ 2022-09-12T20:06:10.466 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A= 2022-09-12T20:06:10.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A=', 'Content-Length': '0'} 2022-09-12T20:06:10.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000033d0a7a2c01ba3c-00631f9132-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.470 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:10.471 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:10.472 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:10.472 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:06:10.473 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:06:10.473 DEBUG:boto:Method: GET 2022-09-12T20:06:10.474 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:06:10.474 DEBUG:boto:Data: 2022-09-12T20:06:10.475 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.476 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.476 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.477 DEBUG:boto:Params: {} 2022-09-12T20:06:10.477 DEBUG:boto:Token: None 2022-09-12T20:06:10.478 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/ 2022-09-12T20:06:10.479 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A= 2022-09-12T20:06:10.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A=', 'Content-Length': '0'} 2022-09-12T20:06:10.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4ed2a8c0957ad6c-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.483 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:10.484 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:10.485 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:10.485 DEBUG:boto:Method: HEAD 2022-09-12T20:06:10.486 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:06:10.486 DEBUG:boto:Data: 2022-09-12T20:06:10.487 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.487 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.488 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.488 DEBUG:boto:Params: {} 2022-09-12T20:06:10.489 DEBUG:boto:Token: None 2022-09-12T20:06:10.490 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj33 2022-09-12T20:06:10.490 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oa5iKihD2jqWWzs1F2df2aNH2os= 2022-09-12T20:06:10.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oa5iKihD2jqWWzs1F2df2aNH2os=', 'Content-Length': '0'} 2022-09-12T20:06:10.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068c51d75270f1687-00631f9132-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.494 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:10.494 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:10.495 DEBUG:boto:Method: HEAD 2022-09-12T20:06:10.495 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:06:10.496 DEBUG:boto:Data: 2022-09-12T20:06:10.496 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.497 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.497 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.498 DEBUG:boto:Params: {} 2022-09-12T20:06:10.498 DEBUG:boto:Token: None 2022-09-12T20:06:10.499 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj33 2022-09-12T20:06:10.500 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oa5iKihD2jqWWzs1F2df2aNH2os= 2022-09-12T20:06:10.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oa5iKihD2jqWWzs1F2df2aNH2os=', 'Content-Length': '0'} 2022-09-12T20:06:10.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dca1cf86941e650-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:06:10.504 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:10.504 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:10.505 DEBUG:boto:Method: GET 2022-09-12T20:06:10.505 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:06:10.506 DEBUG:boto:Data: 2022-09-12T20:06:10.507 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.507 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.508 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.508 DEBUG:boto:Params: {} 2022-09-12T20:06:10.509 DEBUG:boto:Token: None 2022-09-12T20:06:10.510 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj33 2022-09-12T20:06:10.510 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:y1o10IDxD6H1nm6K1mQrYtWKIXY= 2022-09-12T20:06:10.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:y1o10IDxD6H1nm6K1mQrYtWKIXY=', 'Content-Length': '0'} 2022-09-12T20:06:10.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001460a5b5d8bc5f29-00631f9132-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.514 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:10.514 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:10.515 DEBUG:boto:Method: GET 2022-09-12T20:06:10.515 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:06:10.516 DEBUG:boto:Data: 2022-09-12T20:06:10.516 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.517 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.517 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.518 DEBUG:boto:Params: {} 2022-09-12T20:06:10.518 DEBUG:boto:Token: None 2022-09-12T20:06:10.519 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj33 2022-09-12T20:06:10.520 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:y1o10IDxD6H1nm6K1mQrYtWKIXY= 2022-09-12T20:06:10.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:y1o10IDxD6H1nm6K1mQrYtWKIXY=', 'Content-Length': '0'} 2022-09-12T20:06:10.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000135efda2353fcdfe-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.566 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:10.567 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:10.567 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:06:10.568 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:06:10.568 DEBUG:boto:Method: GET 2022-09-12T20:06:10.569 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:06:10.570 DEBUG:boto:Data: 2022-09-12T20:06:10.570 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.571 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.571 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.572 DEBUG:boto:Params: {} 2022-09-12T20:06:10.572 DEBUG:boto:Token: None 2022-09-12T20:06:10.573 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/ 2022-09-12T20:06:10.574 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A= 2022-09-12T20:06:10.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A=', 'Content-Length': '0'} 2022-09-12T20:06:10.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7a703fc0d53e469-00631f9132-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.578 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:10.579 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:10.579 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:10.580 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:06:10.580 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:06:10.581 DEBUG:boto:Method: GET 2022-09-12T20:06:10.581 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:06:10.582 DEBUG:boto:Data: 2022-09-12T20:06:10.582 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.583 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.583 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.584 DEBUG:boto:Params: {} 2022-09-12T20:06:10.585 DEBUG:boto:Token: None 2022-09-12T20:06:10.586 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/ 2022-09-12T20:06:10.586 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A= 2022-09-12T20:06:10.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A=', 'Content-Length': '0'} 2022-09-12T20:06:10.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000539b372798188445-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.591 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:10.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:06:10.592 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:10.593 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:10.594 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:10.594 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:10.595 DEBUG:boto:Method: GET 2022-09-12T20:06:10.595 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:10.596 DEBUG:boto:Data: 2022-09-12T20:06:10.596 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.597 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.598 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.598 DEBUG:boto:Params: {} 2022-09-12T20:06:10.599 DEBUG:boto:Token: None 2022-09-12T20:06:10.599 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:10.600 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k0lGBfgULb1svgh/ZtPdZyADUtA= 2022-09-12T20:06:10.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k0lGBfgULb1svgh/ZtPdZyADUtA=', 'Content-Length': '0'} 2022-09-12T20:06:10.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000117b7753bbc0cddd-00631f9132-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.604 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:10.604 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:10.605 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:10.605 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:10.606 DEBUG:boto:Method: GET 2022-09-12T20:06:10.607 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:10.607 DEBUG:boto:Data: 2022-09-12T20:06:10.608 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.608 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.609 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.609 DEBUG:boto:Params: {} 2022-09-12T20:06:10.610 DEBUG:boto:Token: None 2022-09-12T20:06:10.611 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:10.611 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k0lGBfgULb1svgh/ZtPdZyADUtA= 2022-09-12T20:06:10.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k0lGBfgULb1svgh/ZtPdZyADUtA=', 'Content-Length': '0'} 2022-09-12T20:06:10.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9d91784afcc5535-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.615 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:10.615 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:10.616 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:10.616 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:10.617 DEBUG:boto:Method: HEAD 2022-09-12T20:06:10.618 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:10.618 DEBUG:boto:Data: 2022-09-12T20:06:10.619 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.619 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.620 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.620 DEBUG:boto:Params: {} 2022-09-12T20:06:10.621 DEBUG:boto:Token: None 2022-09-12T20:06:10.621 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:10.622 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cWjMtPdYyZlaelvTTFfny1nK/wQ= 2022-09-12T20:06:10.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cWjMtPdYyZlaelvTTFfny1nK/wQ=', 'Content-Length': '0'} 2022-09-12T20:06:10.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e32e9d1005dc51ca-00631f9132-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.625 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:10.626 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:10.626 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:10.627 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:10.628 DEBUG:boto:Method: HEAD 2022-09-12T20:06:10.628 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:10.629 DEBUG:boto:Data: 2022-09-12T20:06:10.629 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.630 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.630 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.631 DEBUG:boto:Params: {} 2022-09-12T20:06:10.631 DEBUG:boto:Token: None 2022-09-12T20:06:10.632 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:10.632 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cWjMtPdYyZlaelvTTFfny1nK/wQ= 2022-09-12T20:06:10.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cWjMtPdYyZlaelvTTFfny1nK/wQ=', 'Content-Length': '0'} 2022-09-12T20:06:10.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f6ff7aba21c66ae-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:06:10.636 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:10.637 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:10.638 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:10.638 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:10.639 DEBUG:boto:Method: GET 2022-09-12T20:06:10.639 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:10.640 DEBUG:boto:Data: 2022-09-12T20:06:10.640 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.641 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.642 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.642 DEBUG:boto:Params: {} 2022-09-12T20:06:10.643 DEBUG:boto:Token: None 2022-09-12T20:06:10.643 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:10.644 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k0lGBfgULb1svgh/ZtPdZyADUtA= 2022-09-12T20:06:10.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k0lGBfgULb1svgh/ZtPdZyADUtA=', 'Content-Length': '0'} 2022-09-12T20:06:10.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000349c6860bea0884d-00631f9132-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.694 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:10.695 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:10.695 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:10.696 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:10.696 DEBUG:boto:Method: GET 2022-09-12T20:06:10.697 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:10.697 DEBUG:boto:Data: 2022-09-12T20:06:10.698 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.699 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.699 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.700 DEBUG:boto:Params: {} 2022-09-12T20:06:10.700 DEBUG:boto:Token: None 2022-09-12T20:06:10.701 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:10.702 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k0lGBfgULb1svgh/ZtPdZyADUtA= 2022-09-12T20:06:10.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k0lGBfgULb1svgh/ZtPdZyADUtA=', 'Content-Length': '0'} 2022-09-12T20:06:10.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069b88efe808745c0-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.705 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:10.706 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:10.706 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:06:10.707 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:06:10.707 DEBUG:boto:Method: GET 2022-09-12T20:06:10.708 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:06:10.708 DEBUG:boto:Data: 2022-09-12T20:06:10.709 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.709 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.710 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.710 DEBUG:boto:Params: {} 2022-09-12T20:06:10.711 DEBUG:boto:Token: None 2022-09-12T20:06:10.712 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/ 2022-09-12T20:06:10.712 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A= 2022-09-12T20:06:10.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A=', 'Content-Length': '0'} 2022-09-12T20:06:10.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000636f8ba1d2b3af6b-00631f9132-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.716 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:10.717 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:10.718 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:10.718 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:06:10.719 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:06:10.719 DEBUG:boto:Method: GET 2022-09-12T20:06:10.720 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:06:10.720 DEBUG:boto:Data: 2022-09-12T20:06:10.721 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.721 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.722 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.722 DEBUG:boto:Params: {} 2022-09-12T20:06:10.723 DEBUG:boto:Token: None 2022-09-12T20:06:10.724 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/ 2022-09-12T20:06:10.724 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A= 2022-09-12T20:06:10.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A=', 'Content-Length': '0'} 2022-09-12T20:06:10.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000843b6990a035d5a-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.729 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:10.730 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:10.730 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:10.731 DEBUG:boto:Method: HEAD 2022-09-12T20:06:10.731 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:06:10.732 DEBUG:boto:Data: 2022-09-12T20:06:10.732 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.733 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.733 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.734 DEBUG:boto:Params: {} 2022-09-12T20:06:10.734 DEBUG:boto:Token: None 2022-09-12T20:06:10.735 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj34 2022-09-12T20:06:10.735 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GE8mLr77AfY6wjsmQwB8I/Q06RA= 2022-09-12T20:06:10.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GE8mLr77AfY6wjsmQwB8I/Q06RA=', 'Content-Length': '0'} 2022-09-12T20:06:10.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031dd18800d734b71-00631f9132-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.739 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:10.740 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:10.740 DEBUG:boto:Method: HEAD 2022-09-12T20:06:10.741 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:06:10.741 DEBUG:boto:Data: 2022-09-12T20:06:10.742 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.742 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.743 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.743 DEBUG:boto:Params: {} 2022-09-12T20:06:10.744 DEBUG:boto:Token: None 2022-09-12T20:06:10.744 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj34 2022-09-12T20:06:10.745 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GE8mLr77AfY6wjsmQwB8I/Q06RA= 2022-09-12T20:06:10.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GE8mLr77AfY6wjsmQwB8I/Q06RA=', 'Content-Length': '0'} 2022-09-12T20:06:10.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003100de3c9bf9e88c-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:06:10.749 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:10.749 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:10.750 DEBUG:boto:Method: GET 2022-09-12T20:06:10.751 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:06:10.751 DEBUG:boto:Data: 2022-09-12T20:06:10.752 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.752 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.753 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.753 DEBUG:boto:Params: {} 2022-09-12T20:06:10.754 DEBUG:boto:Token: None 2022-09-12T20:06:10.754 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj34 2022-09-12T20:06:10.755 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UNrbb2V9R0mJAlreL77L8HhDkJ8= 2022-09-12T20:06:10.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UNrbb2V9R0mJAlreL77L8HhDkJ8=', 'Content-Length': '0'} 2022-09-12T20:06:10.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c628055bc252bab1-00631f9132-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.758 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:10.759 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:10.759 DEBUG:boto:Method: GET 2022-09-12T20:06:10.760 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:06:10.760 DEBUG:boto:Data: 2022-09-12T20:06:10.761 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.761 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.762 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.762 DEBUG:boto:Params: {} 2022-09-12T20:06:10.763 DEBUG:boto:Token: None 2022-09-12T20:06:10.764 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj34 2022-09-12T20:06:10.764 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UNrbb2V9R0mJAlreL77L8HhDkJ8= 2022-09-12T20:06:10.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UNrbb2V9R0mJAlreL77L8HhDkJ8=', 'Content-Length': '0'} 2022-09-12T20:06:10.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4ab47407efbf45d-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.768 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:10.768 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:10.769 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:06:10.769 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:06:10.770 DEBUG:boto:Method: GET 2022-09-12T20:06:10.770 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:06:10.771 DEBUG:boto:Data: 2022-09-12T20:06:10.772 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.772 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.773 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.773 DEBUG:boto:Params: {} 2022-09-12T20:06:10.774 DEBUG:boto:Token: None 2022-09-12T20:06:10.774 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/ 2022-09-12T20:06:10.775 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A= 2022-09-12T20:06:10.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A=', 'Content-Length': '0'} 2022-09-12T20:06:10.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb0bc28ef5f92d91-00631f9132-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.779 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:10.780 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:10.780 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:10.781 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:06:10.781 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:06:10.782 DEBUG:boto:Method: GET 2022-09-12T20:06:10.782 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:06:10.783 DEBUG:boto:Data: 2022-09-12T20:06:10.783 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.784 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.785 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.785 DEBUG:boto:Params: {} 2022-09-12T20:06:10.786 DEBUG:boto:Token: None 2022-09-12T20:06:10.786 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/ 2022-09-12T20:06:10.787 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A= 2022-09-12T20:06:10.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A=', 'Content-Length': '0'} 2022-09-12T20:06:10.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6a997abf5a69a74-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.791 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:10.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:06:10.793 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:10.793 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:10.794 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:10.794 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:10.795 DEBUG:boto:Method: GET 2022-09-12T20:06:10.795 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:10.796 DEBUG:boto:Data: 2022-09-12T20:06:10.796 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.797 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.797 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.798 DEBUG:boto:Params: {} 2022-09-12T20:06:10.798 DEBUG:boto:Token: None 2022-09-12T20:06:10.799 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:10.799 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u8N0WepYkom3c0vwWFtZ1SbWUUQ= 2022-09-12T20:06:10.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u8N0WepYkom3c0vwWFtZ1SbWUUQ=', 'Content-Length': '0'} 2022-09-12T20:06:10.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5b8bcddf00feff1-00631f9132-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.846 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:10.847 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:10.847 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:10.848 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:10.848 DEBUG:boto:Method: GET 2022-09-12T20:06:10.849 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:10.849 DEBUG:boto:Data: 2022-09-12T20:06:10.850 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.851 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.851 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.852 DEBUG:boto:Params: {} 2022-09-12T20:06:10.852 DEBUG:boto:Token: None 2022-09-12T20:06:10.853 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:10.853 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u8N0WepYkom3c0vwWFtZ1SbWUUQ= 2022-09-12T20:06:10.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u8N0WepYkom3c0vwWFtZ1SbWUUQ=', 'Content-Length': '0'} 2022-09-12T20:06:10.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f8bdd5710b1d1be-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.857 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:10.858 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:10.858 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:10.859 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:10.859 DEBUG:boto:Method: HEAD 2022-09-12T20:06:10.860 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:10.860 DEBUG:boto:Data: 2022-09-12T20:06:10.861 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.861 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.862 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.863 DEBUG:boto:Params: {} 2022-09-12T20:06:10.863 DEBUG:boto:Token: None 2022-09-12T20:06:10.864 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:10.864 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nR0BSGpuCbM5LRPZFOilRd/cxu0= 2022-09-12T20:06:10.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nR0BSGpuCbM5LRPZFOilRd/cxu0=', 'Content-Length': '0'} 2022-09-12T20:06:10.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7f6d5d01cc69bb9-00631f9132-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.868 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:10.868 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:10.869 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:10.869 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:10.870 DEBUG:boto:Method: HEAD 2022-09-12T20:06:10.870 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:10.871 DEBUG:boto:Data: 2022-09-12T20:06:10.871 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.872 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.872 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.873 DEBUG:boto:Params: {} 2022-09-12T20:06:10.873 DEBUG:boto:Token: None 2022-09-12T20:06:10.874 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:10.875 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nR0BSGpuCbM5LRPZFOilRd/cxu0= 2022-09-12T20:06:10.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nR0BSGpuCbM5LRPZFOilRd/cxu0=', 'Content-Length': '0'} 2022-09-12T20:06:10.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c35e4d73bae7d6c5-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:06:10.879 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:10.879 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:10.880 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:10.880 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:10.881 DEBUG:boto:Method: GET 2022-09-12T20:06:10.881 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:10.882 DEBUG:boto:Data: 2022-09-12T20:06:10.882 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.883 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.884 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.884 DEBUG:boto:Params: {} 2022-09-12T20:06:10.885 DEBUG:boto:Token: None 2022-09-12T20:06:10.885 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:10.886 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u8N0WepYkom3c0vwWFtZ1SbWUUQ= 2022-09-12T20:06:10.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u8N0WepYkom3c0vwWFtZ1SbWUUQ=', 'Content-Length': '0'} 2022-09-12T20:06:10.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015ef0534a6c1b6d2-00631f9132-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.889 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:10.890 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:10.891 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:10.891 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:10.892 DEBUG:boto:Method: GET 2022-09-12T20:06:10.892 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:10.893 DEBUG:boto:Data: 2022-09-12T20:06:10.893 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.894 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.894 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.895 DEBUG:boto:Params: {} 2022-09-12T20:06:10.895 DEBUG:boto:Token: None 2022-09-12T20:06:10.896 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:10.897 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u8N0WepYkom3c0vwWFtZ1SbWUUQ= 2022-09-12T20:06:10.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u8N0WepYkom3c0vwWFtZ1SbWUUQ=', 'Content-Length': '0'} 2022-09-12T20:06:10.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005363904d0da07963-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.900 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:10.901 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:10.901 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:06:10.902 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:06:10.902 DEBUG:boto:Method: GET 2022-09-12T20:06:10.903 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:06:10.903 DEBUG:boto:Data: 2022-09-12T20:06:10.904 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.905 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.905 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.906 DEBUG:boto:Params: {} 2022-09-12T20:06:10.906 DEBUG:boto:Token: None 2022-09-12T20:06:10.907 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/ 2022-09-12T20:06:10.907 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A= 2022-09-12T20:06:10.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A=', 'Content-Length': '0'} 2022-09-12T20:06:10.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbbb7a0933a7f798-00631f9132-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.911 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:10.912 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:10.913 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:10.913 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:06:10.914 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:06:10.914 DEBUG:boto:Method: GET 2022-09-12T20:06:10.915 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:06:10.915 DEBUG:boto:Data: 2022-09-12T20:06:10.916 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.916 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.917 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.917 DEBUG:boto:Params: {} 2022-09-12T20:06:10.918 DEBUG:boto:Token: None 2022-09-12T20:06:10.919 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/ 2022-09-12T20:06:10.919 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A= 2022-09-12T20:06:10.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f/ecfwVKk/Dt1ukTLlcDKpeZA2A=', 'Content-Length': '0'} 2022-09-12T20:06:10.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4179959024d943b-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.924 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:10.925 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:10.926 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:10.926 DEBUG:boto:Method: HEAD 2022-09-12T20:06:10.927 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:06:10.927 DEBUG:boto:Data: 2022-09-12T20:06:10.928 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.928 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.929 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.929 DEBUG:boto:Params: {} 2022-09-12T20:06:10.930 DEBUG:boto:Token: None 2022-09-12T20:06:10.930 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj44 2022-09-12T20:06:10.931 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kGb1EXkDhrWiv2yqcD5bhuJvw7c= 2022-09-12T20:06:10.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kGb1EXkDhrWiv2yqcD5bhuJvw7c=', 'Content-Length': '0'} 2022-09-12T20:06:10.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088f26a2022636543-00631f9132-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.934 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:10.935 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:10.935 DEBUG:boto:Method: HEAD 2022-09-12T20:06:10.936 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:06:10.936 DEBUG:boto:Data: 2022-09-12T20:06:10.937 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.937 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.938 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.938 DEBUG:boto:Params: {} 2022-09-12T20:06:10.939 DEBUG:boto:Token: None 2022-09-12T20:06:10.939 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj44 2022-09-12T20:06:10.940 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kGb1EXkDhrWiv2yqcD5bhuJvw7c= 2022-09-12T20:06:10.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kGb1EXkDhrWiv2yqcD5bhuJvw7c=', 'Content-Length': '0'} 2022-09-12T20:06:10.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0afa3b6d38f6edb-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:06:10.944 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:10.944 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:10.945 DEBUG:boto:Method: GET 2022-09-12T20:06:10.945 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:06:10.946 DEBUG:boto:Data: 2022-09-12T20:06:10.947 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.947 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:10.948 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.948 DEBUG:boto:Params: {} 2022-09-12T20:06:10.949 DEBUG:boto:Token: None 2022-09-12T20:06:10.949 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj44 2022-09-12T20:06:10.950 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SqotvQIFalF6RUSqkIsbxWiLZA4= 2022-09-12T20:06:10.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SqotvQIFalF6RUSqkIsbxWiLZA4=', 'Content-Length': '0'} 2022-09-12T20:06:10.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061724a8ea73224e3-00631f9132-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:10.953 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:10.954 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:10.954 DEBUG:boto:Method: GET 2022-09-12T20:06:10.955 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:06:10.955 DEBUG:boto:Data: 2022-09-12T20:06:10.956 DEBUG:boto:Headers: {} 2022-09-12T20:06:10.956 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:10.957 DEBUG:boto:Port: 8000 2022-09-12T20:06:10.957 DEBUG:boto:Params: {} 2022-09-12T20:06:10.958 DEBUG:boto:Token: None 2022-09-12T20:06:10.959 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:10 GMT /jdpiak-75/obj44 2022-09-12T20:06:10.959 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SqotvQIFalF6RUSqkIsbxWiLZA4= 2022-09-12T20:06:10.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SqotvQIFalF6RUSqkIsbxWiLZA4=', 'Content-Length': '0'} 2022-09-12T20:06:10.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000325494bcd4906c8e-00631f9132-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.002 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:11.003 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:11.003 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:06:11.004 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:06:11.004 DEBUG:boto:Method: GET 2022-09-12T20:06:11.005 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:06:11.006 DEBUG:boto:Data: 2022-09-12T20:06:11.006 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.007 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.007 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.008 DEBUG:boto:Params: {} 2022-09-12T20:06:11.009 DEBUG:boto:Token: None 2022-09-12T20:06:11.009 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/ 2022-09-12T20:06:11.010 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4= 2022-09-12T20:06:11.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4=', 'Content-Length': '0'} 2022-09-12T20:06:11.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009354d25a7e2b8053-00631f9133-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.014 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:11.015 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:11.015 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:11.016 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:06:11.017 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:06:11.017 DEBUG:boto:Method: GET 2022-09-12T20:06:11.018 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:06:11.018 DEBUG:boto:Data: 2022-09-12T20:06:11.019 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.019 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.020 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.020 DEBUG:boto:Params: {} 2022-09-12T20:06:11.021 DEBUG:boto:Token: None 2022-09-12T20:06:11.033 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/ 2022-09-12T20:06:11.034 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4= 2022-09-12T20:06:11.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4=', 'Content-Length': '0'} 2022-09-12T20:06:11.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b7433f4062e96fe-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.038 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:11.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:06:11.040 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:11.040 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:11.041 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:11.042 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:11.042 DEBUG:boto:Method: GET 2022-09-12T20:06:11.043 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:11.043 DEBUG:boto:Data: 2022-09-12T20:06:11.044 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.044 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.045 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.045 DEBUG:boto:Params: {} 2022-09-12T20:06:11.046 DEBUG:boto:Token: None 2022-09-12T20:06:11.047 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:11.047 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N5bpr8YliasoL7eN+UDUmM0dqpE= 2022-09-12T20:06:11.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N5bpr8YliasoL7eN+UDUmM0dqpE=', 'Content-Length': '0'} 2022-09-12T20:06:11.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f3d508007ec7f58-00631f9133-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.051 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:11.052 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:11.052 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:11.053 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:11.053 DEBUG:boto:Method: GET 2022-09-12T20:06:11.054 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:11.054 DEBUG:boto:Data: 2022-09-12T20:06:11.055 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.056 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.056 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.057 DEBUG:boto:Params: {} 2022-09-12T20:06:11.057 DEBUG:boto:Token: None 2022-09-12T20:06:11.058 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:11.059 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N5bpr8YliasoL7eN+UDUmM0dqpE= 2022-09-12T20:06:11.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N5bpr8YliasoL7eN+UDUmM0dqpE=', 'Content-Length': '0'} 2022-09-12T20:06:11.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036d6a0442e0e1cd9-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.062 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:11.063 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:11.064 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:11.064 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:11.065 DEBUG:boto:Method: HEAD 2022-09-12T20:06:11.065 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:11.066 DEBUG:boto:Data: 2022-09-12T20:06:11.066 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.067 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.067 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.068 DEBUG:boto:Params: {} 2022-09-12T20:06:11.068 DEBUG:boto:Token: None 2022-09-12T20:06:11.069 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:11.069 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6NQgsXJe6a/bhIuqSFVp8aJnVd0= 2022-09-12T20:06:11.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6NQgsXJe6a/bhIuqSFVp8aJnVd0=', 'Content-Length': '0'} 2022-09-12T20:06:11.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a92c15f0f5fb6561-00631f9133-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.073 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:11.074 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:11.074 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:11.075 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:11.075 DEBUG:boto:Method: HEAD 2022-09-12T20:06:11.076 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:11.076 DEBUG:boto:Data: 2022-09-12T20:06:11.077 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.077 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.078 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.078 DEBUG:boto:Params: {} 2022-09-12T20:06:11.079 DEBUG:boto:Token: None 2022-09-12T20:06:11.079 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:11.080 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6NQgsXJe6a/bhIuqSFVp8aJnVd0= 2022-09-12T20:06:11.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6NQgsXJe6a/bhIuqSFVp8aJnVd0=', 'Content-Length': '0'} 2022-09-12T20:06:11.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000241d1353d14bf8f9-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:06:11.084 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:11.084 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:11.085 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:11.085 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:11.086 DEBUG:boto:Method: GET 2022-09-12T20:06:11.087 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:11.087 DEBUG:boto:Data: 2022-09-12T20:06:11.088 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.088 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.089 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.089 DEBUG:boto:Params: {} 2022-09-12T20:06:11.090 DEBUG:boto:Token: None 2022-09-12T20:06:11.090 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:11.091 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N5bpr8YliasoL7eN+UDUmM0dqpE= 2022-09-12T20:06:11.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N5bpr8YliasoL7eN+UDUmM0dqpE=', 'Content-Length': '0'} 2022-09-12T20:06:11.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6af62aa6e33838b-00631f9133-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.134 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:11.135 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:11.135 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:11.136 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:11.136 DEBUG:boto:Method: GET 2022-09-12T20:06:11.137 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:11.137 DEBUG:boto:Data: 2022-09-12T20:06:11.138 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.138 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.139 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.140 DEBUG:boto:Params: {} 2022-09-12T20:06:11.140 DEBUG:boto:Token: None 2022-09-12T20:06:11.141 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:11.141 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N5bpr8YliasoL7eN+UDUmM0dqpE= 2022-09-12T20:06:11.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N5bpr8YliasoL7eN+UDUmM0dqpE=', 'Content-Length': '0'} 2022-09-12T20:06:11.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a32815192c6498c5-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.145 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:11.146 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:11.146 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:06:11.147 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:06:11.147 DEBUG:boto:Method: GET 2022-09-12T20:06:11.148 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:06:11.148 DEBUG:boto:Data: 2022-09-12T20:06:11.149 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.149 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.150 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.150 DEBUG:boto:Params: {} 2022-09-12T20:06:11.151 DEBUG:boto:Token: None 2022-09-12T20:06:11.152 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/ 2022-09-12T20:06:11.152 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4= 2022-09-12T20:06:11.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4=', 'Content-Length': '0'} 2022-09-12T20:06:11.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c387f38596ed0e24-00631f9133-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.156 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:11.157 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:11.158 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:11.158 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:06:11.159 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:06:11.159 DEBUG:boto:Method: GET 2022-09-12T20:06:11.160 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:06:11.160 DEBUG:boto:Data: 2022-09-12T20:06:11.161 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.161 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.162 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.163 DEBUG:boto:Params: {} 2022-09-12T20:06:11.163 DEBUG:boto:Token: None 2022-09-12T20:06:11.164 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/ 2022-09-12T20:06:11.164 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4= 2022-09-12T20:06:11.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4=', 'Content-Length': '0'} 2022-09-12T20:06:11.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000436483b3ea74381b-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.169 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:11.170 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:11.171 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:11.171 DEBUG:boto:Method: HEAD 2022-09-12T20:06:11.172 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:06:11.172 DEBUG:boto:Data: 2022-09-12T20:06:11.173 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.173 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.174 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.174 DEBUG:boto:Params: {} 2022-09-12T20:06:11.175 DEBUG:boto:Token: None 2022-09-12T20:06:11.176 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj51 2022-09-12T20:06:11.176 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DCbY+5OiNwyRLE20fhnE3xKuKnE= 2022-09-12T20:06:11.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DCbY+5OiNwyRLE20fhnE3xKuKnE=', 'Content-Length': '0'} 2022-09-12T20:06:11.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8235ebb33b5ccb5-00631f9133-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.179 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:11.180 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:11.181 DEBUG:boto:Method: HEAD 2022-09-12T20:06:11.181 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:06:11.182 DEBUG:boto:Data: 2022-09-12T20:06:11.182 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.183 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.183 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.184 DEBUG:boto:Params: {} 2022-09-12T20:06:11.184 DEBUG:boto:Token: None 2022-09-12T20:06:11.185 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj51 2022-09-12T20:06:11.185 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DCbY+5OiNwyRLE20fhnE3xKuKnE= 2022-09-12T20:06:11.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DCbY+5OiNwyRLE20fhnE3xKuKnE=', 'Content-Length': '0'} 2022-09-12T20:06:11.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087da412df473b830-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:06:11.190 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:11.190 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:11.191 DEBUG:boto:Method: GET 2022-09-12T20:06:11.191 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:06:11.192 DEBUG:boto:Data: 2022-09-12T20:06:11.192 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.193 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.193 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.194 DEBUG:boto:Params: {} 2022-09-12T20:06:11.195 DEBUG:boto:Token: None 2022-09-12T20:06:11.195 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj51 2022-09-12T20:06:11.196 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dLgp3sjAbB2ZByGCCD/zNALcyCk= 2022-09-12T20:06:11.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dLgp3sjAbB2ZByGCCD/zNALcyCk=', 'Content-Length': '0'} 2022-09-12T20:06:11.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b662c7c244c90dfe-00631f9133-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.242 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:11.243 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:11.243 DEBUG:boto:Method: GET 2022-09-12T20:06:11.244 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:06:11.244 DEBUG:boto:Data: 2022-09-12T20:06:11.245 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.246 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.246 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.247 DEBUG:boto:Params: {} 2022-09-12T20:06:11.247 DEBUG:boto:Token: None 2022-09-12T20:06:11.248 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj51 2022-09-12T20:06:11.249 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dLgp3sjAbB2ZByGCCD/zNALcyCk= 2022-09-12T20:06:11.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dLgp3sjAbB2ZByGCCD/zNALcyCk=', 'Content-Length': '0'} 2022-09-12T20:06:11.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f96d5fd491ab42e8-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.255 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:11.256 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:11.257 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:06:11.257 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:06:11.258 DEBUG:boto:Method: GET 2022-09-12T20:06:11.259 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:06:11.259 DEBUG:boto:Data: 2022-09-12T20:06:11.260 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.260 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.261 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.261 DEBUG:boto:Params: {} 2022-09-12T20:06:11.262 DEBUG:boto:Token: None 2022-09-12T20:06:11.262 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/ 2022-09-12T20:06:11.263 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4= 2022-09-12T20:06:11.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4=', 'Content-Length': '0'} 2022-09-12T20:06:11.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a557a6982598a6b9-00631f9133-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.268 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:11.269 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:11.270 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:11.271 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:06:11.272 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:06:11.274 DEBUG:boto:Method: GET 2022-09-12T20:06:11.274 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:06:11.275 DEBUG:boto:Data: 2022-09-12T20:06:11.275 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.276 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.276 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.277 DEBUG:boto:Params: {} 2022-09-12T20:06:11.277 DEBUG:boto:Token: None 2022-09-12T20:06:11.278 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/ 2022-09-12T20:06:11.279 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4= 2022-09-12T20:06:11.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4=', 'Content-Length': '0'} 2022-09-12T20:06:11.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d59f88ee6345c46-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.284 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:11.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:06:11.285 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:11.285 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:11.286 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:11.287 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:11.287 DEBUG:boto:Method: GET 2022-09-12T20:06:11.288 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:11.288 DEBUG:boto:Data: 2022-09-12T20:06:11.289 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.289 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.290 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.291 DEBUG:boto:Params: {} 2022-09-12T20:06:11.291 DEBUG:boto:Token: None 2022-09-12T20:06:11.292 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:11.292 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7eDwg9CMd826YNoslxw2Lfpiusk= 2022-09-12T20:06:11.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7eDwg9CMd826YNoslxw2Lfpiusk=', 'Content-Length': '0'} 2022-09-12T20:06:11.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a797247ac95eb8f-00631f9133-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.296 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:11.296 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:11.297 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:11.298 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:11.298 DEBUG:boto:Method: GET 2022-09-12T20:06:11.299 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:11.299 DEBUG:boto:Data: 2022-09-12T20:06:11.300 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.301 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.301 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.302 DEBUG:boto:Params: {} 2022-09-12T20:06:11.302 DEBUG:boto:Token: None 2022-09-12T20:06:11.303 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:11.303 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7eDwg9CMd826YNoslxw2Lfpiusk= 2022-09-12T20:06:11.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7eDwg9CMd826YNoslxw2Lfpiusk=', 'Content-Length': '0'} 2022-09-12T20:06:11.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfe6b80a28e1b6a1-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.350 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:11.351 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:11.351 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:11.352 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:11.352 DEBUG:boto:Method: HEAD 2022-09-12T20:06:11.353 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:11.353 DEBUG:boto:Data: 2022-09-12T20:06:11.354 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.355 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.355 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.356 DEBUG:boto:Params: {} 2022-09-12T20:06:11.356 DEBUG:boto:Token: None 2022-09-12T20:06:11.357 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:11.357 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BehK2QtrqZgvO3ei4lkkeKYylnw= 2022-09-12T20:06:11.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BehK2QtrqZgvO3ei4lkkeKYylnw=', 'Content-Length': '0'} 2022-09-12T20:06:11.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000146644943aa82700-00631f9133-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.361 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:11.361 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:11.362 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:11.362 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:11.363 DEBUG:boto:Method: HEAD 2022-09-12T20:06:11.363 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:11.364 DEBUG:boto:Data: 2022-09-12T20:06:11.364 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.365 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.365 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.366 DEBUG:boto:Params: {} 2022-09-12T20:06:11.366 DEBUG:boto:Token: None 2022-09-12T20:06:11.367 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:11.367 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BehK2QtrqZgvO3ei4lkkeKYylnw= 2022-09-12T20:06:11.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BehK2QtrqZgvO3ei4lkkeKYylnw=', 'Content-Length': '0'} 2022-09-12T20:06:11.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cae601e6597b2f9-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:06:11.372 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:11.372 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:11.373 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:11.374 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:11.374 DEBUG:boto:Method: GET 2022-09-12T20:06:11.375 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:11.375 DEBUG:boto:Data: 2022-09-12T20:06:11.376 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.376 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.377 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.377 DEBUG:boto:Params: {} 2022-09-12T20:06:11.378 DEBUG:boto:Token: None 2022-09-12T20:06:11.378 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:11.379 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7eDwg9CMd826YNoslxw2Lfpiusk= 2022-09-12T20:06:11.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7eDwg9CMd826YNoslxw2Lfpiusk=', 'Content-Length': '0'} 2022-09-12T20:06:11.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9bb135beafe784b-00631f9133-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.383 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:11.383 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:11.384 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:11.384 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:11.385 DEBUG:boto:Method: GET 2022-09-12T20:06:11.385 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:11.386 DEBUG:boto:Data: 2022-09-12T20:06:11.386 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.387 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.387 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.388 DEBUG:boto:Params: {} 2022-09-12T20:06:11.389 DEBUG:boto:Token: None 2022-09-12T20:06:11.389 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:11.390 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7eDwg9CMd826YNoslxw2Lfpiusk= 2022-09-12T20:06:11.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7eDwg9CMd826YNoslxw2Lfpiusk=', 'Content-Length': '0'} 2022-09-12T20:06:11.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024333b5dae30a9f4-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.393 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:11.394 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:11.394 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:06:11.395 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:06:11.395 DEBUG:boto:Method: GET 2022-09-12T20:06:11.396 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:06:11.397 DEBUG:boto:Data: 2022-09-12T20:06:11.397 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.398 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.399 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.400 DEBUG:boto:Params: {} 2022-09-12T20:06:11.401 DEBUG:boto:Token: None 2022-09-12T20:06:11.402 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/ 2022-09-12T20:06:11.403 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4= 2022-09-12T20:06:11.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4=', 'Content-Length': '0'} 2022-09-12T20:06:11.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ceea0c3a13ec9b7e-00631f9133-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.450 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:11.451 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:11.452 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:11.452 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:06:11.453 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:06:11.453 DEBUG:boto:Method: GET 2022-09-12T20:06:11.454 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:06:11.454 DEBUG:boto:Data: 2022-09-12T20:06:11.455 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.456 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.456 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.457 DEBUG:boto:Params: {} 2022-09-12T20:06:11.457 DEBUG:boto:Token: None 2022-09-12T20:06:11.458 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/ 2022-09-12T20:06:11.458 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4= 2022-09-12T20:06:11.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4=', 'Content-Length': '0'} 2022-09-12T20:06:11.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbf77948989c8a29-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.463 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:11.463 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:11.464 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:11.465 DEBUG:boto:Method: HEAD 2022-09-12T20:06:11.465 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:06:11.466 DEBUG:boto:Data: 2022-09-12T20:06:11.467 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.467 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.468 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.468 DEBUG:boto:Params: {} 2022-09-12T20:06:11.469 DEBUG:boto:Token: None 2022-09-12T20:06:11.469 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj55 2022-09-12T20:06:11.470 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4LfoaHyj805sWEo2u/411ThTjIU= 2022-09-12T20:06:11.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4LfoaHyj805sWEo2u/411ThTjIU=', 'Content-Length': '0'} 2022-09-12T20:06:11.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f750d629db1008a-00631f9133-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.474 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:11.474 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:11.475 DEBUG:boto:Method: HEAD 2022-09-12T20:06:11.475 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:06:11.476 DEBUG:boto:Data: 2022-09-12T20:06:11.476 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.477 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.477 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.478 DEBUG:boto:Params: {} 2022-09-12T20:06:11.478 DEBUG:boto:Token: None 2022-09-12T20:06:11.479 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj55 2022-09-12T20:06:11.479 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4LfoaHyj805sWEo2u/411ThTjIU= 2022-09-12T20:06:11.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4LfoaHyj805sWEo2u/411ThTjIU=', 'Content-Length': '0'} 2022-09-12T20:06:11.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e163c5cf3526093-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:06:11.483 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:11.484 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:11.484 DEBUG:boto:Method: GET 2022-09-12T20:06:11.485 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:06:11.485 DEBUG:boto:Data: 2022-09-12T20:06:11.486 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.487 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.487 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.488 DEBUG:boto:Params: {} 2022-09-12T20:06:11.488 DEBUG:boto:Token: None 2022-09-12T20:06:11.489 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj55 2022-09-12T20:06:11.490 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iG+40NDGeUEXa/4NO1B2q6KY/Zc= 2022-09-12T20:06:11.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iG+40NDGeUEXa/4NO1B2q6KY/Zc=', 'Content-Length': '0'} 2022-09-12T20:06:11.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b9c073c92cf0f13-00631f9133-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.493 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:11.494 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:11.495 DEBUG:boto:Method: GET 2022-09-12T20:06:11.496 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:06:11.497 DEBUG:boto:Data: 2022-09-12T20:06:11.498 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.499 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.500 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.501 DEBUG:boto:Params: {} 2022-09-12T20:06:11.502 DEBUG:boto:Token: None 2022-09-12T20:06:11.503 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj55 2022-09-12T20:06:11.504 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iG+40NDGeUEXa/4NO1B2q6KY/Zc= 2022-09-12T20:06:11.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iG+40NDGeUEXa/4NO1B2q6KY/Zc=', 'Content-Length': '0'} 2022-09-12T20:06:11.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e293040d39958d4c-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.550 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:11.551 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:11.551 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:06:11.552 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:06:11.552 DEBUG:boto:Method: GET 2022-09-12T20:06:11.553 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:06:11.553 DEBUG:boto:Data: 2022-09-12T20:06:11.554 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.554 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.555 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.555 DEBUG:boto:Params: {} 2022-09-12T20:06:11.556 DEBUG:boto:Token: None 2022-09-12T20:06:11.557 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/ 2022-09-12T20:06:11.557 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4= 2022-09-12T20:06:11.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4=', 'Content-Length': '0'} 2022-09-12T20:06:11.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060f953e09dffcc1b-00631f9133-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.562 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:11.563 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:11.563 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:11.564 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:06:11.564 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:06:11.565 DEBUG:boto:Method: GET 2022-09-12T20:06:11.565 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:06:11.566 DEBUG:boto:Data: 2022-09-12T20:06:11.566 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.567 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.567 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.568 DEBUG:boto:Params: {} 2022-09-12T20:06:11.568 DEBUG:boto:Token: None 2022-09-12T20:06:11.569 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/ 2022-09-12T20:06:11.570 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4= 2022-09-12T20:06:11.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4=', 'Content-Length': '0'} 2022-09-12T20:06:11.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009f97c040c943109-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.574 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:11.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:06:11.575 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:11.576 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:11.576 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:11.577 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:11.577 DEBUG:boto:Method: GET 2022-09-12T20:06:11.578 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:11.579 DEBUG:boto:Data: 2022-09-12T20:06:11.579 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.580 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.580 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.581 DEBUG:boto:Params: {} 2022-09-12T20:06:11.581 DEBUG:boto:Token: None 2022-09-12T20:06:11.582 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:11.582 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MJoU+8nd/DgxqvG3MNobcGcBJBI= 2022-09-12T20:06:11.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MJoU+8nd/DgxqvG3MNobcGcBJBI=', 'Content-Length': '0'} 2022-09-12T20:06:11.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1a7c29193116590-00631f9133-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.586 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:11.587 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:11.587 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:11.588 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:11.588 DEBUG:boto:Method: GET 2022-09-12T20:06:11.589 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:11.589 DEBUG:boto:Data: 2022-09-12T20:06:11.590 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.590 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.591 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.591 DEBUG:boto:Params: {} 2022-09-12T20:06:11.592 DEBUG:boto:Token: None 2022-09-12T20:06:11.592 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:11.593 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MJoU+8nd/DgxqvG3MNobcGcBJBI= 2022-09-12T20:06:11.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MJoU+8nd/DgxqvG3MNobcGcBJBI=', 'Content-Length': '0'} 2022-09-12T20:06:11.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070507d95a7898929-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.596 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:11.597 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:11.597 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:11.598 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:11.598 DEBUG:boto:Method: HEAD 2022-09-12T20:06:11.599 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:11.599 DEBUG:boto:Data: 2022-09-12T20:06:11.600 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.600 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.601 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.601 DEBUG:boto:Params: {} 2022-09-12T20:06:11.602 DEBUG:boto:Token: None 2022-09-12T20:06:11.603 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:11.603 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:313YT+AIe3cQhDGYvvFtQ+PZLT0= 2022-09-12T20:06:11.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:313YT+AIe3cQhDGYvvFtQ+PZLT0=', 'Content-Length': '0'} 2022-09-12T20:06:11.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acc4015b3d5cb3b4-00631f9133-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.606 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:11.607 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:11.607 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:11.608 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:11.608 DEBUG:boto:Method: HEAD 2022-09-12T20:06:11.609 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:11.609 DEBUG:boto:Data: 2022-09-12T20:06:11.610 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.610 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.611 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.611 DEBUG:boto:Params: {} 2022-09-12T20:06:11.612 DEBUG:boto:Token: None 2022-09-12T20:06:11.613 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:11.613 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:313YT+AIe3cQhDGYvvFtQ+PZLT0= 2022-09-12T20:06:11.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:313YT+AIe3cQhDGYvvFtQ+PZLT0=', 'Content-Length': '0'} 2022-09-12T20:06:11.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016de2d13cd4a9ce3-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:06:11.617 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:11.618 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:11.618 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:11.619 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:11.619 DEBUG:boto:Method: GET 2022-09-12T20:06:11.620 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:11.620 DEBUG:boto:Data: 2022-09-12T20:06:11.621 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.621 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.622 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.622 DEBUG:boto:Params: {} 2022-09-12T20:06:11.623 DEBUG:boto:Token: None 2022-09-12T20:06:11.624 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:11.624 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MJoU+8nd/DgxqvG3MNobcGcBJBI= 2022-09-12T20:06:11.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MJoU+8nd/DgxqvG3MNobcGcBJBI=', 'Content-Length': '0'} 2022-09-12T20:06:11.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f56f7a4cfdd28f25-00631f9133-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.628 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:11.628 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:11.629 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:11.629 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:11.630 DEBUG:boto:Method: GET 2022-09-12T20:06:11.631 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:11.631 DEBUG:boto:Data: 2022-09-12T20:06:11.632 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.632 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.633 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.633 DEBUG:boto:Params: {} 2022-09-12T20:06:11.634 DEBUG:boto:Token: None 2022-09-12T20:06:11.634 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:11.635 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MJoU+8nd/DgxqvG3MNobcGcBJBI= 2022-09-12T20:06:11.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MJoU+8nd/DgxqvG3MNobcGcBJBI=', 'Content-Length': '0'} 2022-09-12T20:06:11.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bda1321f4d416b4e-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.638 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:11.639 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:11.639 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:06:11.640 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:06:11.640 DEBUG:boto:Method: GET 2022-09-12T20:06:11.641 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:06:11.641 DEBUG:boto:Data: 2022-09-12T20:06:11.642 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.642 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.643 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.644 DEBUG:boto:Params: {} 2022-09-12T20:06:11.644 DEBUG:boto:Token: None 2022-09-12T20:06:11.645 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/ 2022-09-12T20:06:11.645 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4= 2022-09-12T20:06:11.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4=', 'Content-Length': '0'} 2022-09-12T20:06:11.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a60b3e31c01c6f8-00631f9133-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.649 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:11.650 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:11.650 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:11.651 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:06:11.652 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:06:11.652 DEBUG:boto:Method: GET 2022-09-12T20:06:11.653 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:06:11.653 DEBUG:boto:Data: 2022-09-12T20:06:11.654 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.654 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.655 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.655 DEBUG:boto:Params: {} 2022-09-12T20:06:11.656 DEBUG:boto:Token: None 2022-09-12T20:06:11.656 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/ 2022-09-12T20:06:11.657 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4= 2022-09-12T20:06:11.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4=', 'Content-Length': '0'} 2022-09-12T20:06:11.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fa5c68f7a883caf-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.662 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:11.662 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:11.663 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:11.663 DEBUG:boto:Method: HEAD 2022-09-12T20:06:11.664 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:06:11.664 DEBUG:boto:Data: 2022-09-12T20:06:11.665 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.665 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.666 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.666 DEBUG:boto:Params: {} 2022-09-12T20:06:11.667 DEBUG:boto:Token: None 2022-09-12T20:06:11.667 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj68 2022-09-12T20:06:11.668 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aNkT+b/gJsTb7hiN06/0KL8Bb5k= 2022-09-12T20:06:11.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aNkT+b/gJsTb7hiN06/0KL8Bb5k=', 'Content-Length': '0'} 2022-09-12T20:06:11.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c23e5b810edcce05-00631f9133-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.671 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:11.672 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:11.672 DEBUG:boto:Method: HEAD 2022-09-12T20:06:11.673 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:06:11.673 DEBUG:boto:Data: 2022-09-12T20:06:11.674 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.674 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.675 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.675 DEBUG:boto:Params: {} 2022-09-12T20:06:11.676 DEBUG:boto:Token: None 2022-09-12T20:06:11.676 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj68 2022-09-12T20:06:11.677 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aNkT+b/gJsTb7hiN06/0KL8Bb5k= 2022-09-12T20:06:11.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aNkT+b/gJsTb7hiN06/0KL8Bb5k=', 'Content-Length': '0'} 2022-09-12T20:06:11.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8d50a0af1bc598d-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:06:11.681 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:11.681 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:11.682 DEBUG:boto:Method: GET 2022-09-12T20:06:11.682 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:06:11.683 DEBUG:boto:Data: 2022-09-12T20:06:11.683 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.684 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.685 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.685 DEBUG:boto:Params: {} 2022-09-12T20:06:11.686 DEBUG:boto:Token: None 2022-09-12T20:06:11.686 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj68 2022-09-12T20:06:11.687 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wgIFnNsusJdBlVrt6ri/9oevwHs= 2022-09-12T20:06:11.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wgIFnNsusJdBlVrt6ri/9oevwHs=', 'Content-Length': '0'} 2022-09-12T20:06:11.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd7fd3a370434b73-00631f9133-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.730 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:11.731 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:11.731 DEBUG:boto:Method: GET 2022-09-12T20:06:11.732 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:06:11.732 DEBUG:boto:Data: 2022-09-12T20:06:11.733 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.733 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.734 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.734 DEBUG:boto:Params: {} 2022-09-12T20:06:11.735 DEBUG:boto:Token: None 2022-09-12T20:06:11.736 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj68 2022-09-12T20:06:11.736 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wgIFnNsusJdBlVrt6ri/9oevwHs= 2022-09-12T20:06:11.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wgIFnNsusJdBlVrt6ri/9oevwHs=', 'Content-Length': '0'} 2022-09-12T20:06:11.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005709050ec6338335-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.740 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:11.741 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:11.741 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:06:11.742 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:06:11.742 DEBUG:boto:Method: GET 2022-09-12T20:06:11.743 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:06:11.743 DEBUG:boto:Data: 2022-09-12T20:06:11.744 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.745 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.745 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.746 DEBUG:boto:Params: {} 2022-09-12T20:06:11.746 DEBUG:boto:Token: None 2022-09-12T20:06:11.747 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/ 2022-09-12T20:06:11.747 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4= 2022-09-12T20:06:11.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4=', 'Content-Length': '0'} 2022-09-12T20:06:11.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2e218c5c80aedda-00631f9133-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.751 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:11.752 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:11.753 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:11.753 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:06:11.754 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:06:11.754 DEBUG:boto:Method: GET 2022-09-12T20:06:11.755 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:06:11.755 DEBUG:boto:Data: 2022-09-12T20:06:11.756 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.757 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.757 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.758 DEBUG:boto:Params: {} 2022-09-12T20:06:11.758 DEBUG:boto:Token: None 2022-09-12T20:06:11.759 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/ 2022-09-12T20:06:11.759 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4= 2022-09-12T20:06:11.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4=', 'Content-Length': '0'} 2022-09-12T20:06:11.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007103a7bd89e63171-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.764 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:11.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:06:11.766 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:11.766 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:11.767 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:11.767 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:11.768 DEBUG:boto:Method: GET 2022-09-12T20:06:11.768 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:11.769 DEBUG:boto:Data: 2022-09-12T20:06:11.769 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.770 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.771 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.771 DEBUG:boto:Params: {} 2022-09-12T20:06:11.772 DEBUG:boto:Token: None 2022-09-12T20:06:11.772 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:11.773 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/QJwtlTG5NJ4uzSVEb75x5IiwXU= 2022-09-12T20:06:11.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/QJwtlTG5NJ4uzSVEb75x5IiwXU=', 'Content-Length': '0'} 2022-09-12T20:06:11.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a9098328bae2295-00631f9133-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.777 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:11.777 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:11.778 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:11.778 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:11.779 DEBUG:boto:Method: GET 2022-09-12T20:06:11.779 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:11.780 DEBUG:boto:Data: 2022-09-12T20:06:11.780 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.781 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.782 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.782 DEBUG:boto:Params: {} 2022-09-12T20:06:11.783 DEBUG:boto:Token: None 2022-09-12T20:06:11.783 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:11.784 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/QJwtlTG5NJ4uzSVEb75x5IiwXU= 2022-09-12T20:06:11.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/QJwtlTG5NJ4uzSVEb75x5IiwXU=', 'Content-Length': '0'} 2022-09-12T20:06:11.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000221fc897a0d62897-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.788 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:11.789 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:11.789 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:11.790 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:11.790 DEBUG:boto:Method: HEAD 2022-09-12T20:06:11.791 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:11.791 DEBUG:boto:Data: 2022-09-12T20:06:11.792 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.792 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.793 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.793 DEBUG:boto:Params: {} 2022-09-12T20:06:11.794 DEBUG:boto:Token: None 2022-09-12T20:06:11.795 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:11.795 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:90F9c9vRqDdrTRM2GFqJjltcDXo= 2022-09-12T20:06:11.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:90F9c9vRqDdrTRM2GFqJjltcDXo=', 'Content-Length': '0'} 2022-09-12T20:06:11.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000365fdb25c4da860e-00631f9133-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.798 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:11.799 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:11.799 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:11.800 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:11.800 DEBUG:boto:Method: HEAD 2022-09-12T20:06:11.801 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:11.801 DEBUG:boto:Data: 2022-09-12T20:06:11.802 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.803 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.803 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.804 DEBUG:boto:Params: {} 2022-09-12T20:06:11.804 DEBUG:boto:Token: None 2022-09-12T20:06:11.805 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:11.805 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:90F9c9vRqDdrTRM2GFqJjltcDXo= 2022-09-12T20:06:11.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:90F9c9vRqDdrTRM2GFqJjltcDXo=', 'Content-Length': '0'} 2022-09-12T20:06:11.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063b1016a64e55523-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:06:11.809 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:11.810 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:11.811 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:11.811 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:11.812 DEBUG:boto:Method: GET 2022-09-12T20:06:11.814 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:11.815 DEBUG:boto:Data: 2022-09-12T20:06:11.816 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.816 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.817 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.817 DEBUG:boto:Params: {} 2022-09-12T20:06:11.818 DEBUG:boto:Token: None 2022-09-12T20:06:11.818 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:11.819 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/QJwtlTG5NJ4uzSVEb75x5IiwXU= 2022-09-12T20:06:11.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/QJwtlTG5NJ4uzSVEb75x5IiwXU=', 'Content-Length': '0'} 2022-09-12T20:06:11.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af768a62e1176ac9-00631f9133-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.823 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:11.823 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:11.824 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:11.824 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:11.825 DEBUG:boto:Method: GET 2022-09-12T20:06:11.825 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:11.826 DEBUG:boto:Data: 2022-09-12T20:06:11.826 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.827 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.828 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.828 DEBUG:boto:Params: {} 2022-09-12T20:06:11.829 DEBUG:boto:Token: None 2022-09-12T20:06:11.830 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:11.831 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/QJwtlTG5NJ4uzSVEb75x5IiwXU= 2022-09-12T20:06:11.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/QJwtlTG5NJ4uzSVEb75x5IiwXU=', 'Content-Length': '0'} 2022-09-12T20:06:11.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d7d81471f711032-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.882 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:11.883 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:11.883 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:06:11.884 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:06:11.884 DEBUG:boto:Method: GET 2022-09-12T20:06:11.885 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:06:11.885 DEBUG:boto:Data: 2022-09-12T20:06:11.886 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.887 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.887 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.888 DEBUG:boto:Params: {} 2022-09-12T20:06:11.888 DEBUG:boto:Token: None 2022-09-12T20:06:11.889 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/ 2022-09-12T20:06:11.890 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4= 2022-09-12T20:06:11.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4=', 'Content-Length': '0'} 2022-09-12T20:06:11.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6148953a4383497-00631f9133-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.894 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:11.895 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:11.896 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:11.896 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:06:11.897 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:06:11.898 DEBUG:boto:Method: GET 2022-09-12T20:06:11.898 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:06:11.899 DEBUG:boto:Data: 2022-09-12T20:06:11.899 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.900 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.900 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.901 DEBUG:boto:Params: {} 2022-09-12T20:06:11.901 DEBUG:boto:Token: None 2022-09-12T20:06:11.902 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/ 2022-09-12T20:06:11.903 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4= 2022-09-12T20:06:11.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4=', 'Content-Length': '0'} 2022-09-12T20:06:11.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8a5fb6b4fa57a2a-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.907 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:11.908 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:11.909 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:11.909 DEBUG:boto:Method: HEAD 2022-09-12T20:06:11.910 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:06:11.911 DEBUG:boto:Data: 2022-09-12T20:06:11.911 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.912 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.912 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.913 DEBUG:boto:Params: {} 2022-09-12T20:06:11.914 DEBUG:boto:Token: None 2022-09-12T20:06:11.914 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj85 2022-09-12T20:06:11.915 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zvolsQh/6yuuT42jSHBjK6hdXiU= 2022-09-12T20:06:11.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zvolsQh/6yuuT42jSHBjK6hdXiU=', 'Content-Length': '0'} 2022-09-12T20:06:11.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f225ca3dd56dfa70-00631f9133-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.919 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:11.919 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:11.920 DEBUG:boto:Method: HEAD 2022-09-12T20:06:11.920 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:06:11.921 DEBUG:boto:Data: 2022-09-12T20:06:11.921 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.922 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.922 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.923 DEBUG:boto:Params: {} 2022-09-12T20:06:11.923 DEBUG:boto:Token: None 2022-09-12T20:06:11.924 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj85 2022-09-12T20:06:11.924 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zvolsQh/6yuuT42jSHBjK6hdXiU= 2022-09-12T20:06:11.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zvolsQh/6yuuT42jSHBjK6hdXiU=', 'Content-Length': '0'} 2022-09-12T20:06:11.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003551539d074c3c5e-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:06:11.928 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:11.929 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:11.929 DEBUG:boto:Method: GET 2022-09-12T20:06:11.930 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:06:11.931 DEBUG:boto:Data: 2022-09-12T20:06:11.931 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.932 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.932 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.933 DEBUG:boto:Params: {} 2022-09-12T20:06:11.933 DEBUG:boto:Token: None 2022-09-12T20:06:11.934 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj85 2022-09-12T20:06:11.934 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2j7X65dhN9mTh37L+0Mmt5Ru+Qc= 2022-09-12T20:06:11.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2j7X65dhN9mTh37L+0Mmt5Ru+Qc=', 'Content-Length': '0'} 2022-09-12T20:06:11.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8fde96867d8bca6-00631f9133-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.939 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:11.939 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:11.940 DEBUG:boto:Method: GET 2022-09-12T20:06:11.940 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:06:11.941 DEBUG:boto:Data: 2022-09-12T20:06:11.941 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.942 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.942 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.943 DEBUG:boto:Params: {} 2022-09-12T20:06:11.943 DEBUG:boto:Token: None 2022-09-12T20:06:11.944 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/obj85 2022-09-12T20:06:11.945 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2j7X65dhN9mTh37L+0Mmt5Ru+Qc= 2022-09-12T20:06:11.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2j7X65dhN9mTh37L+0Mmt5Ru+Qc=', 'Content-Length': '0'} 2022-09-12T20:06:11.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c024a31b4a114c61-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.948 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:11.949 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:11.949 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:06:11.950 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:06:11.950 DEBUG:boto:Method: GET 2022-09-12T20:06:11.951 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:06:11.952 DEBUG:boto:Data: 2022-09-12T20:06:11.952 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.953 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:11.953 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.954 DEBUG:boto:Params: {} 2022-09-12T20:06:11.954 DEBUG:boto:Token: None 2022-09-12T20:06:11.955 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/ 2022-09-12T20:06:11.955 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4= 2022-09-12T20:06:11.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4=', 'Content-Length': '0'} 2022-09-12T20:06:11.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087926cef1e1420b0-00631f9133-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.959 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:11.960 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:11.961 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:11.961 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:06:11.962 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:06:11.962 DEBUG:boto:Method: GET 2022-09-12T20:06:11.963 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:06:11.963 DEBUG:boto:Data: 2022-09-12T20:06:11.964 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.964 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.965 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.965 DEBUG:boto:Params: {} 2022-09-12T20:06:11.966 DEBUG:boto:Token: None 2022-09-12T20:06:11.967 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/ 2022-09-12T20:06:11.967 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4= 2022-09-12T20:06:11.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mMP+mb2u1z3aIFLVLIuE9SJfh+4=', 'Content-Length': '0'} 2022-09-12T20:06:11.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b83aea2ee6e088bf-00631f9133-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.972 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:11.973 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-75 zones={test-zone2, test-zone1} 2022-09-12T20:06:11.973 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-75 zones={test-zone3, test-zone1} 2022-09-12T20:06:11.974 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:11.974 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:11.975 DEBUG:boto:Method: HEAD 2022-09-12T20:06:11.975 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:06:11.976 DEBUG:boto:Data: 2022-09-12T20:06:11.976 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.977 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:11.978 DEBUG:boto:Port: 8001 2022-09-12T20:06:11.978 DEBUG:boto:Params: {} 2022-09-12T20:06:11.979 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T20:06:11.980 DEBUG:boto:Token: None 2022-09-12T20:06:11.980 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/ 2022-09-12T20:06:11.981 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ldKoMpirBa8rAAdmA2o0wEB3UGI= 2022-09-12T20:06:11.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ldKoMpirBa8rAAdmA2o0wEB3UGI=', 'Content-Length': '0'} 2022-09-12T20:06:11.985 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012fcceeadaf6660e-00631f9133-1826-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.986 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:11.987 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:11.988 DEBUG:boto:Method: HEAD 2022-09-12T20:06:11.988 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:06:11.989 DEBUG:boto:Data: 2022-09-12T20:06:11.989 DEBUG:boto:Headers: {} 2022-09-12T20:06:11.990 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:11.990 DEBUG:boto:Port: 8000 2022-09-12T20:06:11.991 DEBUG:boto:Params: {} 2022-09-12T20:06:11.991 DEBUG:boto:Token: None 2022-09-12T20:06:11.992 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:11 GMT /jdpiak-75/ 2022-09-12T20:06:11.992 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ldKoMpirBa8rAAdmA2o0wEB3UGI= 2022-09-12T20:06:11.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ldKoMpirBa8rAAdmA2o0wEB3UGI=', 'Content-Length': '0'} 2022-09-12T20:06:11.996 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f12edc42b25b06b-00631f9133-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:11.997 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T20:06:11.997 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:11.998 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:11.999 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:11.999 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:11.999 DEBUG:boto:Method: GET 2022-09-12T20:06:12.000 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:12.001 DEBUG:boto:Data: 2022-09-12T20:06:12.001 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.002 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.002 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.003 DEBUG:boto:Params: {} 2022-09-12T20:06:12.004 DEBUG:boto:Token: None 2022-09-12T20:06:12.004 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/?versions 2022-09-12T20:06:12.005 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+wkiWrV8qQzLE7HzwegQ/BnSMq0= 2022-09-12T20:06:12.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+wkiWrV8qQzLE7HzwegQ/BnSMq0=', 'Content-Length': '0'} 2022-09-12T20:06:12.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e34461692fa09e9a-00631f9134-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.009 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:12.010 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:06:12.011 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:06:12.011 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:06:12.012 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:06:12.012 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:06:12.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:06:12.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:06:12.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:06:12.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:06:12.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:06:12.015 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T20:06:12.016 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:12.016 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:12.017 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:12.017 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:12.018 DEBUG:boto:Method: GET 2022-09-12T20:06:12.018 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:12.019 DEBUG:boto:Data: 2022-09-12T20:06:12.020 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.020 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.021 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.021 DEBUG:boto:Params: {} 2022-09-12T20:06:12.022 DEBUG:boto:Token: None 2022-09-12T20:06:12.023 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/?versions 2022-09-12T20:06:12.023 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+wkiWrV8qQzLE7HzwegQ/BnSMq0= 2022-09-12T20:06:12.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+wkiWrV8qQzLE7HzwegQ/BnSMq0=', 'Content-Length': '0'} 2022-09-12T20:06:12.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000980bfd1c15fee598-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.028 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:12.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:06:12.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:06:12.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:06:12.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:06:12.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:06:12.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:06:12.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:06:12.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:06:12.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:06:12.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:06:12.036 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:12.036 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:12.037 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:12.037 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:12.038 DEBUG:boto:Method: GET 2022-09-12T20:06:12.038 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:12.039 DEBUG:boto:Data: 2022-09-12T20:06:12.039 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.040 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.040 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.041 DEBUG:boto:Params: {} 2022-09-12T20:06:12.041 DEBUG:boto:Token: None 2022-09-12T20:06:12.042 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/?versions 2022-09-12T20:06:12.042 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+wkiWrV8qQzLE7HzwegQ/BnSMq0= 2022-09-12T20:06:12.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+wkiWrV8qQzLE7HzwegQ/BnSMq0=', 'Content-Length': '0'} 2022-09-12T20:06:12.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a075c642a6a3a07-00631f9134-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.047 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:12.048 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:12.049 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:12.049 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:12.050 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:12.050 DEBUG:boto:Method: GET 2022-09-12T20:06:12.051 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:12.051 DEBUG:boto:Data: 2022-09-12T20:06:12.052 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.052 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.053 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.053 DEBUG:boto:Params: {} 2022-09-12T20:06:12.054 DEBUG:boto:Token: None 2022-09-12T20:06:12.054 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/?versions 2022-09-12T20:06:12.055 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+wkiWrV8qQzLE7HzwegQ/BnSMq0= 2022-09-12T20:06:12.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+wkiWrV8qQzLE7HzwegQ/BnSMq0=', 'Content-Length': '0'} 2022-09-12T20:06:12.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039b0cb78a53a7a73-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.060 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:12.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:06:12.062 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:12.062 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:12.063 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:12.063 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:12.064 DEBUG:boto:Method: GET 2022-09-12T20:06:12.064 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:12.065 DEBUG:boto:Data: 2022-09-12T20:06:12.065 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.066 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.066 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.067 DEBUG:boto:Params: {} 2022-09-12T20:06:12.067 DEBUG:boto:Token: None 2022-09-12T20:06:12.068 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:12.069 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LxMIDfDdgKPd7ybx9hBYYK2iy2Q= 2022-09-12T20:06:12.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LxMIDfDdgKPd7ybx9hBYYK2iy2Q=', 'Content-Length': '0'} 2022-09-12T20:06:12.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000812f001064a2d38-00631f9134-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.072 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:12.073 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:12.073 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:12.074 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:12.074 DEBUG:boto:Method: GET 2022-09-12T20:06:12.075 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:12.076 DEBUG:boto:Data: 2022-09-12T20:06:12.076 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.077 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.077 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.078 DEBUG:boto:Params: {} 2022-09-12T20:06:12.078 DEBUG:boto:Token: None 2022-09-12T20:06:12.079 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:12.079 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LxMIDfDdgKPd7ybx9hBYYK2iy2Q= 2022-09-12T20:06:12.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LxMIDfDdgKPd7ybx9hBYYK2iy2Q=', 'Content-Length': '0'} 2022-09-12T20:06:12.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001c0024442a9b719-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.126 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:12.127 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:12.127 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:12.128 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:12.128 DEBUG:boto:Method: HEAD 2022-09-12T20:06:12.129 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:12.129 DEBUG:boto:Data: 2022-09-12T20:06:12.130 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.130 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.131 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.131 DEBUG:boto:Params: {} 2022-09-12T20:06:12.132 DEBUG:boto:Token: None 2022-09-12T20:06:12.132 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:12.133 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ppSoG7O+OuFEIm/MhZ6MO4i7Mek= 2022-09-12T20:06:12.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ppSoG7O+OuFEIm/MhZ6MO4i7Mek=', 'Content-Length': '0'} 2022-09-12T20:06:12.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bfd965f1b421d88-00631f9134-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.136 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:12.137 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:12.137 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:12.138 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:12.138 DEBUG:boto:Method: HEAD 2022-09-12T20:06:12.139 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:12.139 DEBUG:boto:Data: 2022-09-12T20:06:12.140 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.140 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.141 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.141 DEBUG:boto:Params: {} 2022-09-12T20:06:12.142 DEBUG:boto:Token: None 2022-09-12T20:06:12.143 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:12.143 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ppSoG7O+OuFEIm/MhZ6MO4i7Mek= 2022-09-12T20:06:12.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ppSoG7O+OuFEIm/MhZ6MO4i7Mek=', 'Content-Length': '0'} 2022-09-12T20:06:12.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbf43bc74f3432be-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:06:12.147 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:12.148 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:12.149 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:12.149 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:12.150 DEBUG:boto:Method: GET 2022-09-12T20:06:12.150 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:12.151 DEBUG:boto:Data: 2022-09-12T20:06:12.151 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.152 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.152 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.153 DEBUG:boto:Params: {} 2022-09-12T20:06:12.153 DEBUG:boto:Token: None 2022-09-12T20:06:12.154 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:12.155 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LxMIDfDdgKPd7ybx9hBYYK2iy2Q= 2022-09-12T20:06:12.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LxMIDfDdgKPd7ybx9hBYYK2iy2Q=', 'Content-Length': '0'} 2022-09-12T20:06:12.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c214ff1bd8a02803-00631f9134-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.158 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:12.159 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:12.159 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:12.160 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:12.161 DEBUG:boto:Method: GET 2022-09-12T20:06:12.161 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:12.162 DEBUG:boto:Data: 2022-09-12T20:06:12.162 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.163 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.163 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.164 DEBUG:boto:Params: {} 2022-09-12T20:06:12.164 DEBUG:boto:Token: None 2022-09-12T20:06:12.165 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:12.165 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LxMIDfDdgKPd7ybx9hBYYK2iy2Q= 2022-09-12T20:06:12.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LxMIDfDdgKPd7ybx9hBYYK2iy2Q=', 'Content-Length': '0'} 2022-09-12T20:06:12.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b5689fa6fa0d8aa-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.169 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:12.170 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:12.170 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:06:12.171 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:06:12.171 DEBUG:boto:Method: GET 2022-09-12T20:06:12.172 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:06:12.172 DEBUG:boto:Data: 2022-09-12T20:06:12.173 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.173 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.174 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.174 DEBUG:boto:Params: {} 2022-09-12T20:06:12.175 DEBUG:boto:Token: None 2022-09-12T20:06:12.175 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/ 2022-09-12T20:06:12.176 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0= 2022-09-12T20:06:12.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0=', 'Content-Length': '0'} 2022-09-12T20:06:12.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095ccb3458ceac7c9-00631f9134-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.180 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:12.181 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:12.182 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:12.182 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:06:12.183 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:06:12.184 DEBUG:boto:Method: GET 2022-09-12T20:06:12.185 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:06:12.186 DEBUG:boto:Data: 2022-09-12T20:06:12.187 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.188 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.189 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.190 DEBUG:boto:Params: {} 2022-09-12T20:06:12.191 DEBUG:boto:Token: None 2022-09-12T20:06:12.192 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/ 2022-09-12T20:06:12.193 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0= 2022-09-12T20:06:12.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0=', 'Content-Length': '0'} 2022-09-12T20:06:12.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060acf4c67341bea3-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.198 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:12.199 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:12.200 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:12.200 DEBUG:boto:Method: HEAD 2022-09-12T20:06:12.201 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:06:12.201 DEBUG:boto:Data: 2022-09-12T20:06:12.202 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.202 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.203 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.203 DEBUG:boto:Params: {} 2022-09-12T20:06:12.204 DEBUG:boto:Token: None 2022-09-12T20:06:12.204 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj11 2022-09-12T20:06:12.205 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:onFq1WHtxEaq5Cb0OMHcdlYOfo8= 2022-09-12T20:06:12.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:onFq1WHtxEaq5Cb0OMHcdlYOfo8=', 'Content-Length': '0'} 2022-09-12T20:06:12.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a90744d1145391fb-00631f9134-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.208 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:12.209 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:12.209 DEBUG:boto:Method: HEAD 2022-09-12T20:06:12.210 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:06:12.210 DEBUG:boto:Data: 2022-09-12T20:06:12.211 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.211 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.212 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.212 DEBUG:boto:Params: {} 2022-09-12T20:06:12.213 DEBUG:boto:Token: None 2022-09-12T20:06:12.213 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj11 2022-09-12T20:06:12.214 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:onFq1WHtxEaq5Cb0OMHcdlYOfo8= 2022-09-12T20:06:12.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:onFq1WHtxEaq5Cb0OMHcdlYOfo8=', 'Content-Length': '0'} 2022-09-12T20:06:12.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba9b18264e337b3a-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:06:12.218 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:12.219 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:12.219 DEBUG:boto:Method: GET 2022-09-12T20:06:12.220 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:06:12.220 DEBUG:boto:Data: 2022-09-12T20:06:12.221 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.221 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.222 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.222 DEBUG:boto:Params: {} 2022-09-12T20:06:12.223 DEBUG:boto:Token: None 2022-09-12T20:06:12.223 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj11 2022-09-12T20:06:12.224 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z0mCEHNcs8bfPkViZ4Q4io99CmA= 2022-09-12T20:06:12.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z0mCEHNcs8bfPkViZ4Q4io99CmA=', 'Content-Length': '0'} 2022-09-12T20:06:12.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f2dd72cb85e04b4-00631f9134-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.227 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:12.228 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:12.229 DEBUG:boto:Method: GET 2022-09-12T20:06:12.229 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:06:12.230 DEBUG:boto:Data: 2022-09-12T20:06:12.230 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.231 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.231 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.232 DEBUG:boto:Params: {} 2022-09-12T20:06:12.233 DEBUG:boto:Token: None 2022-09-12T20:06:12.233 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj11 2022-09-12T20:06:12.234 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z0mCEHNcs8bfPkViZ4Q4io99CmA= 2022-09-12T20:06:12.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z0mCEHNcs8bfPkViZ4Q4io99CmA=', 'Content-Length': '0'} 2022-09-12T20:06:12.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e72735306190af1c-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.237 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:12.238 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:12.238 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:06:12.239 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:06:12.239 DEBUG:boto:Method: GET 2022-09-12T20:06:12.240 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:06:12.241 DEBUG:boto:Data: 2022-09-12T20:06:12.241 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.242 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.242 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.243 DEBUG:boto:Params: {} 2022-09-12T20:06:12.244 DEBUG:boto:Token: None 2022-09-12T20:06:12.244 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/ 2022-09-12T20:06:12.245 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0= 2022-09-12T20:06:12.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0=', 'Content-Length': '0'} 2022-09-12T20:06:12.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000308d162ebb22ea09-00631f9134-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.290 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:12.291 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:12.292 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:12.293 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:06:12.294 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:06:12.294 DEBUG:boto:Method: GET 2022-09-12T20:06:12.295 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:06:12.296 DEBUG:boto:Data: 2022-09-12T20:06:12.296 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.297 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.298 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.298 DEBUG:boto:Params: {} 2022-09-12T20:06:12.299 DEBUG:boto:Token: None 2022-09-12T20:06:12.300 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/ 2022-09-12T20:06:12.301 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0= 2022-09-12T20:06:12.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0=', 'Content-Length': '0'} 2022-09-12T20:06:12.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0ef893138747dd7-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.305 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:12.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:06:12.308 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:12.309 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:12.310 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:12.311 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:12.312 DEBUG:boto:Method: GET 2022-09-12T20:06:12.312 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:12.313 DEBUG:boto:Data: 2022-09-12T20:06:12.314 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.315 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.316 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.317 DEBUG:boto:Params: {} 2022-09-12T20:06:12.318 DEBUG:boto:Token: None 2022-09-12T20:06:12.319 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:12.320 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wfPCK1pxIYUZ4DDSk+86nA55T80= 2022-09-12T20:06:12.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wfPCK1pxIYUZ4DDSk+86nA55T80=', 'Content-Length': '0'} 2022-09-12T20:06:12.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036c5c1bb874dae2d-00631f9134-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.323 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:12.324 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:12.324 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:12.325 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:12.325 DEBUG:boto:Method: GET 2022-09-12T20:06:12.326 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:12.327 DEBUG:boto:Data: 2022-09-12T20:06:12.328 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.328 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.329 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.329 DEBUG:boto:Params: {} 2022-09-12T20:06:12.330 DEBUG:boto:Token: None 2022-09-12T20:06:12.330 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:12.331 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wfPCK1pxIYUZ4DDSk+86nA55T80= 2022-09-12T20:06:12.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wfPCK1pxIYUZ4DDSk+86nA55T80=', 'Content-Length': '0'} 2022-09-12T20:06:12.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007292da71c43d38b8-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.335 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:12.336 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:12.336 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:12.337 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:12.337 DEBUG:boto:Method: HEAD 2022-09-12T20:06:12.338 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:12.339 DEBUG:boto:Data: 2022-09-12T20:06:12.339 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.340 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.340 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.341 DEBUG:boto:Params: {} 2022-09-12T20:06:12.341 DEBUG:boto:Token: None 2022-09-12T20:06:12.342 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:12.342 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1cuHaKzTJgonVivIRz2zTI9mJFA= 2022-09-12T20:06:12.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1cuHaKzTJgonVivIRz2zTI9mJFA=', 'Content-Length': '0'} 2022-09-12T20:06:12.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9efbbe82c77babe-00631f9134-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.346 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:12.346 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:12.347 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:12.347 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:12.348 DEBUG:boto:Method: HEAD 2022-09-12T20:06:12.349 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:12.349 DEBUG:boto:Data: 2022-09-12T20:06:12.350 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.350 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.351 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.351 DEBUG:boto:Params: {} 2022-09-12T20:06:12.352 DEBUG:boto:Token: None 2022-09-12T20:06:12.352 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:12.353 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1cuHaKzTJgonVivIRz2zTI9mJFA= 2022-09-12T20:06:12.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1cuHaKzTJgonVivIRz2zTI9mJFA=', 'Content-Length': '0'} 2022-09-12T20:06:12.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031d0c78f3c78d460-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:06:12.357 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:12.357 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:12.358 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:12.359 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:12.359 DEBUG:boto:Method: GET 2022-09-12T20:06:12.360 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:12.360 DEBUG:boto:Data: 2022-09-12T20:06:12.361 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.361 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.362 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.363 DEBUG:boto:Params: {} 2022-09-12T20:06:12.364 DEBUG:boto:Token: None 2022-09-12T20:06:12.365 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:12.365 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wfPCK1pxIYUZ4DDSk+86nA55T80= 2022-09-12T20:06:12.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wfPCK1pxIYUZ4DDSk+86nA55T80=', 'Content-Length': '0'} 2022-09-12T20:06:12.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d1c325157e8acc1-00631f9134-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.369 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:12.369 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:12.370 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:12.370 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:12.371 DEBUG:boto:Method: GET 2022-09-12T20:06:12.371 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:12.372 DEBUG:boto:Data: 2022-09-12T20:06:12.372 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.373 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.373 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.374 DEBUG:boto:Params: {} 2022-09-12T20:06:12.375 DEBUG:boto:Token: None 2022-09-12T20:06:12.375 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:12.376 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wfPCK1pxIYUZ4DDSk+86nA55T80= 2022-09-12T20:06:12.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wfPCK1pxIYUZ4DDSk+86nA55T80=', 'Content-Length': '0'} 2022-09-12T20:06:12.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7f498f956feae80-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.422 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:12.423 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:12.423 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:06:12.424 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:06:12.424 DEBUG:boto:Method: GET 2022-09-12T20:06:12.425 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:06:12.425 DEBUG:boto:Data: 2022-09-12T20:06:12.426 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.426 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.427 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.427 DEBUG:boto:Params: {} 2022-09-12T20:06:12.428 DEBUG:boto:Token: None 2022-09-12T20:06:12.429 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/ 2022-09-12T20:06:12.429 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0= 2022-09-12T20:06:12.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0=', 'Content-Length': '0'} 2022-09-12T20:06:12.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c1d74e573570264-00631f9134-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.434 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:12.435 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:12.436 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:12.436 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:06:12.437 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:06:12.437 DEBUG:boto:Method: GET 2022-09-12T20:06:12.438 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:06:12.438 DEBUG:boto:Data: 2022-09-12T20:06:12.439 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.440 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.440 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.441 DEBUG:boto:Params: {} 2022-09-12T20:06:12.441 DEBUG:boto:Token: None 2022-09-12T20:06:12.442 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/ 2022-09-12T20:06:12.442 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0= 2022-09-12T20:06:12.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0=', 'Content-Length': '0'} 2022-09-12T20:06:12.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc9f3d7cc2822fec-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.447 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:12.448 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:12.448 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:12.449 DEBUG:boto:Method: HEAD 2022-09-12T20:06:12.449 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:06:12.450 DEBUG:boto:Data: 2022-09-12T20:06:12.451 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.451 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.451 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.452 DEBUG:boto:Params: {} 2022-09-12T20:06:12.453 DEBUG:boto:Token: None 2022-09-12T20:06:12.453 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj17 2022-09-12T20:06:12.454 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vEFmO/zoElbably/XZhQNuTqMCQ= 2022-09-12T20:06:12.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vEFmO/zoElbably/XZhQNuTqMCQ=', 'Content-Length': '0'} 2022-09-12T20:06:12.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4a8a07d02e22839-00631f9134-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.457 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:12.457 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:12.458 DEBUG:boto:Method: HEAD 2022-09-12T20:06:12.458 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:06:12.459 DEBUG:boto:Data: 2022-09-12T20:06:12.459 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.460 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.461 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.461 DEBUG:boto:Params: {} 2022-09-12T20:06:12.462 DEBUG:boto:Token: None 2022-09-12T20:06:12.462 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj17 2022-09-12T20:06:12.463 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vEFmO/zoElbably/XZhQNuTqMCQ= 2022-09-12T20:06:12.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vEFmO/zoElbably/XZhQNuTqMCQ=', 'Content-Length': '0'} 2022-09-12T20:06:12.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf1f715356174dde-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:06:12.467 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:12.467 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:12.468 DEBUG:boto:Method: GET 2022-09-12T20:06:12.468 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:06:12.469 DEBUG:boto:Data: 2022-09-12T20:06:12.469 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.470 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.470 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.471 DEBUG:boto:Params: {} 2022-09-12T20:06:12.472 DEBUG:boto:Token: None 2022-09-12T20:06:12.472 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj17 2022-09-12T20:06:12.473 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YQ6eIW3xgtLrd/shhdjkiolX0TQ= 2022-09-12T20:06:12.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YQ6eIW3xgtLrd/shhdjkiolX0TQ=', 'Content-Length': '0'} 2022-09-12T20:06:12.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000001d35ee7b0aa28f-00631f9134-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.476 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:12.477 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:12.477 DEBUG:boto:Method: GET 2022-09-12T20:06:12.478 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:06:12.479 DEBUG:boto:Data: 2022-09-12T20:06:12.479 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.480 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.480 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.481 DEBUG:boto:Params: {} 2022-09-12T20:06:12.482 DEBUG:boto:Token: None 2022-09-12T20:06:12.482 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj17 2022-09-12T20:06:12.483 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YQ6eIW3xgtLrd/shhdjkiolX0TQ= 2022-09-12T20:06:12.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YQ6eIW3xgtLrd/shhdjkiolX0TQ=', 'Content-Length': '0'} 2022-09-12T20:06:12.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d8dbf5393e3b76f-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.487 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:12.487 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:12.488 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:06:12.488 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:06:12.489 DEBUG:boto:Method: GET 2022-09-12T20:06:12.490 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:06:12.491 DEBUG:boto:Data: 2022-09-12T20:06:12.492 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.493 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.493 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.494 DEBUG:boto:Params: {} 2022-09-12T20:06:12.495 DEBUG:boto:Token: None 2022-09-12T20:06:12.496 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/ 2022-09-12T20:06:12.497 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0= 2022-09-12T20:06:12.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0=', 'Content-Length': '0'} 2022-09-12T20:06:12.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cfbf4a28f05e890-00631f9134-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.502 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:12.503 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:12.504 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:12.505 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:06:12.506 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:06:12.507 DEBUG:boto:Method: GET 2022-09-12T20:06:12.507 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:06:12.508 DEBUG:boto:Data: 2022-09-12T20:06:12.508 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.509 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.509 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.510 DEBUG:boto:Params: {} 2022-09-12T20:06:12.511 DEBUG:boto:Token: None 2022-09-12T20:06:12.512 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/ 2022-09-12T20:06:12.513 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0= 2022-09-12T20:06:12.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0=', 'Content-Length': '0'} 2022-09-12T20:06:12.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6c787e13039187b-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.558 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:12.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:06:12.560 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:12.561 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:12.562 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:12.563 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:12.564 DEBUG:boto:Method: GET 2022-09-12T20:06:12.565 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:12.566 DEBUG:boto:Data: 2022-09-12T20:06:12.567 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.568 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.569 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.569 DEBUG:boto:Params: {} 2022-09-12T20:06:12.571 DEBUG:boto:Token: None 2022-09-12T20:06:12.572 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:12.573 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6uBd7WP0/2/qmfPk9px7RMW2ZO0= 2022-09-12T20:06:12.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6uBd7WP0/2/qmfPk9px7RMW2ZO0=', 'Content-Length': '0'} 2022-09-12T20:06:12.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afe804461343cfdb-00631f9134-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.578 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:12.579 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:12.579 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:12.580 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:12.581 DEBUG:boto:Method: GET 2022-09-12T20:06:12.582 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:12.583 DEBUG:boto:Data: 2022-09-12T20:06:12.584 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.585 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.586 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.587 DEBUG:boto:Params: {} 2022-09-12T20:06:12.588 DEBUG:boto:Token: None 2022-09-12T20:06:12.589 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:12.590 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6uBd7WP0/2/qmfPk9px7RMW2ZO0= 2022-09-12T20:06:12.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6uBd7WP0/2/qmfPk9px7RMW2ZO0=', 'Content-Length': '0'} 2022-09-12T20:06:12.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091acfe22220f3ccb-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.595 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:12.595 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:12.596 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:12.597 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:12.598 DEBUG:boto:Method: HEAD 2022-09-12T20:06:12.599 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:12.600 DEBUG:boto:Data: 2022-09-12T20:06:12.600 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.601 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.602 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.603 DEBUG:boto:Params: {} 2022-09-12T20:06:12.604 DEBUG:boto:Token: None 2022-09-12T20:06:12.605 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:12.606 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qs00WWl34/l2Tj37jhI1scuTOpk= 2022-09-12T20:06:12.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qs00WWl34/l2Tj37jhI1scuTOpk=', 'Content-Length': '0'} 2022-09-12T20:06:12.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053258b01befd9076-00631f9134-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.610 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:12.611 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:12.612 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:12.613 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:12.614 DEBUG:boto:Method: HEAD 2022-09-12T20:06:12.614 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:12.615 DEBUG:boto:Data: 2022-09-12T20:06:12.616 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.617 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.618 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.619 DEBUG:boto:Params: {} 2022-09-12T20:06:12.620 DEBUG:boto:Token: None 2022-09-12T20:06:12.621 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:12.621 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qs00WWl34/l2Tj37jhI1scuTOpk= 2022-09-12T20:06:12.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qs00WWl34/l2Tj37jhI1scuTOpk=', 'Content-Length': '0'} 2022-09-12T20:06:12.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef5b69c87791f51c-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:06:12.627 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:12.627 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:12.628 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:12.628 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:12.629 DEBUG:boto:Method: GET 2022-09-12T20:06:12.629 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:12.630 DEBUG:boto:Data: 2022-09-12T20:06:12.631 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.631 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.632 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.632 DEBUG:boto:Params: {} 2022-09-12T20:06:12.633 DEBUG:boto:Token: None 2022-09-12T20:06:12.633 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:12.634 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6uBd7WP0/2/qmfPk9px7RMW2ZO0= 2022-09-12T20:06:12.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6uBd7WP0/2/qmfPk9px7RMW2ZO0=', 'Content-Length': '0'} 2022-09-12T20:06:12.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005514e1a2c780fffc-00631f9134-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.638 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:12.638 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:12.639 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:12.640 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:12.640 DEBUG:boto:Method: GET 2022-09-12T20:06:12.641 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:12.641 DEBUG:boto:Data: 2022-09-12T20:06:12.642 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.642 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.643 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.643 DEBUG:boto:Params: {} 2022-09-12T20:06:12.644 DEBUG:boto:Token: None 2022-09-12T20:06:12.644 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:12.645 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6uBd7WP0/2/qmfPk9px7RMW2ZO0= 2022-09-12T20:06:12.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6uBd7WP0/2/qmfPk9px7RMW2ZO0=', 'Content-Length': '0'} 2022-09-12T20:06:12.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6566c67ce770439-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.649 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:12.649 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:12.650 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:06:12.650 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:06:12.651 DEBUG:boto:Method: GET 2022-09-12T20:06:12.651 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:06:12.652 DEBUG:boto:Data: 2022-09-12T20:06:12.652 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.653 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.653 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.654 DEBUG:boto:Params: {} 2022-09-12T20:06:12.655 DEBUG:boto:Token: None 2022-09-12T20:06:12.655 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/ 2022-09-12T20:06:12.656 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0= 2022-09-12T20:06:12.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0=', 'Content-Length': '0'} 2022-09-12T20:06:12.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db3ef19a95d7992f-00631f9134-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.660 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:12.661 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:12.662 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:12.662 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:06:12.663 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:06:12.664 DEBUG:boto:Method: GET 2022-09-12T20:06:12.664 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:06:12.665 DEBUG:boto:Data: 2022-09-12T20:06:12.665 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.666 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.666 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.667 DEBUG:boto:Params: {} 2022-09-12T20:06:12.667 DEBUG:boto:Token: None 2022-09-12T20:06:12.668 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/ 2022-09-12T20:06:12.669 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0= 2022-09-12T20:06:12.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0=', 'Content-Length': '0'} 2022-09-12T20:06:12.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be596124428e927d-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.674 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:12.674 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:12.675 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:12.676 DEBUG:boto:Method: HEAD 2022-09-12T20:06:12.676 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:06:12.677 DEBUG:boto:Data: 2022-09-12T20:06:12.677 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.678 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.678 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.679 DEBUG:boto:Params: {} 2022-09-12T20:06:12.679 DEBUG:boto:Token: None 2022-09-12T20:06:12.680 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj22 2022-09-12T20:06:12.681 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7VGn4g1oZVYwfBQUk5D/E5krHgk= 2022-09-12T20:06:12.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7VGn4g1oZVYwfBQUk5D/E5krHgk=', 'Content-Length': '0'} 2022-09-12T20:06:12.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009866fb304050d563-00631f9134-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.684 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:12.685 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:12.685 DEBUG:boto:Method: HEAD 2022-09-12T20:06:12.686 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:06:12.686 DEBUG:boto:Data: 2022-09-12T20:06:12.687 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.687 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.688 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.688 DEBUG:boto:Params: {} 2022-09-12T20:06:12.689 DEBUG:boto:Token: None 2022-09-12T20:06:12.689 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj22 2022-09-12T20:06:12.690 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7VGn4g1oZVYwfBQUk5D/E5krHgk= 2022-09-12T20:06:12.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7VGn4g1oZVYwfBQUk5D/E5krHgk=', 'Content-Length': '0'} 2022-09-12T20:06:12.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db6b51d757ea6059-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:06:12.694 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:12.694 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:12.695 DEBUG:boto:Method: GET 2022-09-12T20:06:12.695 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:06:12.696 DEBUG:boto:Data: 2022-09-12T20:06:12.696 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.697 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.697 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.698 DEBUG:boto:Params: {} 2022-09-12T20:06:12.698 DEBUG:boto:Token: None 2022-09-12T20:06:12.699 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj22 2022-09-12T20:06:12.700 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lVRJGaYnl5nLpFh9gpZT9CzRNnM= 2022-09-12T20:06:12.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lVRJGaYnl5nLpFh9gpZT9CzRNnM=', 'Content-Length': '0'} 2022-09-12T20:06:12.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060b67c77d672cc7c-00631f9134-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.746 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:12.747 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:12.747 DEBUG:boto:Method: GET 2022-09-12T20:06:12.748 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:06:12.748 DEBUG:boto:Data: 2022-09-12T20:06:12.749 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.749 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.750 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.751 DEBUG:boto:Params: {} 2022-09-12T20:06:12.751 DEBUG:boto:Token: None 2022-09-12T20:06:12.752 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj22 2022-09-12T20:06:12.752 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lVRJGaYnl5nLpFh9gpZT9CzRNnM= 2022-09-12T20:06:12.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lVRJGaYnl5nLpFh9gpZT9CzRNnM=', 'Content-Length': '0'} 2022-09-12T20:06:12.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c82cc4a6459527cc-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.756 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:12.756 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:12.757 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:06:12.757 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:06:12.758 DEBUG:boto:Method: GET 2022-09-12T20:06:12.759 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:06:12.759 DEBUG:boto:Data: 2022-09-12T20:06:12.760 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.760 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.761 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.761 DEBUG:boto:Params: {} 2022-09-12T20:06:12.762 DEBUG:boto:Token: None 2022-09-12T20:06:12.762 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/ 2022-09-12T20:06:12.763 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0= 2022-09-12T20:06:12.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0=', 'Content-Length': '0'} 2022-09-12T20:06:12.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030ec55aa9427b843-00631f9134-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.767 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:12.768 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:12.769 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:12.769 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:06:12.770 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:06:12.770 DEBUG:boto:Method: GET 2022-09-12T20:06:12.771 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:06:12.771 DEBUG:boto:Data: 2022-09-12T20:06:12.772 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.772 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.773 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.773 DEBUG:boto:Params: {} 2022-09-12T20:06:12.774 DEBUG:boto:Token: None 2022-09-12T20:06:12.775 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/ 2022-09-12T20:06:12.775 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0= 2022-09-12T20:06:12.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0=', 'Content-Length': '0'} 2022-09-12T20:06:12.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3ce0e92ab089797-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.780 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:12.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:06:12.781 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:12.781 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:12.782 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:12.783 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:12.783 DEBUG:boto:Method: GET 2022-09-12T20:06:12.784 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:12.784 DEBUG:boto:Data: 2022-09-12T20:06:12.785 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.785 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.786 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.786 DEBUG:boto:Params: {} 2022-09-12T20:06:12.787 DEBUG:boto:Token: None 2022-09-12T20:06:12.787 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:12.788 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2pjVpmXfm1qkt0Slh0+vjl30bnI= 2022-09-12T20:06:12.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2pjVpmXfm1qkt0Slh0+vjl30bnI=', 'Content-Length': '0'} 2022-09-12T20:06:12.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027b515eda4918b2d-00631f9134-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.792 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:12.792 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:12.793 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:12.793 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:12.794 DEBUG:boto:Method: GET 2022-09-12T20:06:12.794 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:12.795 DEBUG:boto:Data: 2022-09-12T20:06:12.795 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.796 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.797 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.797 DEBUG:boto:Params: {} 2022-09-12T20:06:12.798 DEBUG:boto:Token: None 2022-09-12T20:06:12.798 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:12.799 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2pjVpmXfm1qkt0Slh0+vjl30bnI= 2022-09-12T20:06:12.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2pjVpmXfm1qkt0Slh0+vjl30bnI=', 'Content-Length': '0'} 2022-09-12T20:06:12.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4149d5bfa7e4616-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.802 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:12.803 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:12.803 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:12.804 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:12.804 DEBUG:boto:Method: HEAD 2022-09-12T20:06:12.805 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:12.805 DEBUG:boto:Data: 2022-09-12T20:06:12.806 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.806 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.807 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.807 DEBUG:boto:Params: {} 2022-09-12T20:06:12.808 DEBUG:boto:Token: None 2022-09-12T20:06:12.808 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:12.809 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:59dmIqNEXtCwvr0Z7vYd6KLAfWs= 2022-09-12T20:06:12.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:59dmIqNEXtCwvr0Z7vYd6KLAfWs=', 'Content-Length': '0'} 2022-09-12T20:06:12.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b22f9feee09e4c0c-00631f9134-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.812 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:12.813 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:12.813 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:12.814 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:12.814 DEBUG:boto:Method: HEAD 2022-09-12T20:06:12.815 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:12.815 DEBUG:boto:Data: 2022-09-12T20:06:12.816 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.816 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.817 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.818 DEBUG:boto:Params: {} 2022-09-12T20:06:12.818 DEBUG:boto:Token: None 2022-09-12T20:06:12.819 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:12.819 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:59dmIqNEXtCwvr0Z7vYd6KLAfWs= 2022-09-12T20:06:12.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:59dmIqNEXtCwvr0Z7vYd6KLAfWs=', 'Content-Length': '0'} 2022-09-12T20:06:12.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000757e109378ee05e5-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:06:12.823 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:12.824 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:12.824 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:12.825 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:12.826 DEBUG:boto:Method: GET 2022-09-12T20:06:12.826 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:12.827 DEBUG:boto:Data: 2022-09-12T20:06:12.827 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.828 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.828 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.829 DEBUG:boto:Params: {} 2022-09-12T20:06:12.829 DEBUG:boto:Token: None 2022-09-12T20:06:12.830 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:12.831 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2pjVpmXfm1qkt0Slh0+vjl30bnI= 2022-09-12T20:06:12.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2pjVpmXfm1qkt0Slh0+vjl30bnI=', 'Content-Length': '0'} 2022-09-12T20:06:12.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4d7459d6edc7eb6-00631f9134-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.874 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:12.875 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:12.875 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:12.876 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:12.876 DEBUG:boto:Method: GET 2022-09-12T20:06:12.877 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:12.877 DEBUG:boto:Data: 2022-09-12T20:06:12.878 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.879 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.879 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.880 DEBUG:boto:Params: {} 2022-09-12T20:06:12.880 DEBUG:boto:Token: None 2022-09-12T20:06:12.881 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:12.882 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2pjVpmXfm1qkt0Slh0+vjl30bnI= 2022-09-12T20:06:12.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2pjVpmXfm1qkt0Slh0+vjl30bnI=', 'Content-Length': '0'} 2022-09-12T20:06:12.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4064df4a3ba6234-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.885 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:12.886 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:12.887 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:06:12.887 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:06:12.888 DEBUG:boto:Method: GET 2022-09-12T20:06:12.889 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:06:12.890 DEBUG:boto:Data: 2022-09-12T20:06:12.890 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.891 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.891 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.892 DEBUG:boto:Params: {} 2022-09-12T20:06:12.893 DEBUG:boto:Token: None 2022-09-12T20:06:12.893 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/ 2022-09-12T20:06:12.894 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0= 2022-09-12T20:06:12.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0=', 'Content-Length': '0'} 2022-09-12T20:06:12.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009ba119606c922fc-00631f9134-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.898 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:12.899 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:12.900 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:12.900 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:06:12.901 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:06:12.901 DEBUG:boto:Method: GET 2022-09-12T20:06:12.902 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:06:12.902 DEBUG:boto:Data: 2022-09-12T20:06:12.903 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.903 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.904 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.904 DEBUG:boto:Params: {} 2022-09-12T20:06:12.905 DEBUG:boto:Token: None 2022-09-12T20:06:12.906 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/ 2022-09-12T20:06:12.906 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0= 2022-09-12T20:06:12.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0=', 'Content-Length': '0'} 2022-09-12T20:06:12.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000860c661397d3d12d-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.911 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:12.911 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:12.912 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:12.913 DEBUG:boto:Method: HEAD 2022-09-12T20:06:12.913 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:06:12.914 DEBUG:boto:Data: 2022-09-12T20:06:12.914 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.915 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.915 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.916 DEBUG:boto:Params: {} 2022-09-12T20:06:12.916 DEBUG:boto:Token: None 2022-09-12T20:06:12.917 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj33 2022-09-12T20:06:12.917 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1DrfUj/MjQJqqsnCsaVUktEoIDw= 2022-09-12T20:06:12.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1DrfUj/MjQJqqsnCsaVUktEoIDw=', 'Content-Length': '0'} 2022-09-12T20:06:12.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c448a924eb91ebdf-00631f9134-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.921 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:12.921 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:12.922 DEBUG:boto:Method: HEAD 2022-09-12T20:06:12.922 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:06:12.923 DEBUG:boto:Data: 2022-09-12T20:06:12.923 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.924 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.924 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.925 DEBUG:boto:Params: {} 2022-09-12T20:06:12.925 DEBUG:boto:Token: None 2022-09-12T20:06:12.926 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj33 2022-09-12T20:06:12.927 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1DrfUj/MjQJqqsnCsaVUktEoIDw= 2022-09-12T20:06:12.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1DrfUj/MjQJqqsnCsaVUktEoIDw=', 'Content-Length': '0'} 2022-09-12T20:06:12.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f987d2c1a220042b-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:06:12.931 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:12.931 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:12.932 DEBUG:boto:Method: GET 2022-09-12T20:06:12.932 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:06:12.933 DEBUG:boto:Data: 2022-09-12T20:06:12.933 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.934 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.935 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.935 DEBUG:boto:Params: {} 2022-09-12T20:06:12.936 DEBUG:boto:Token: None 2022-09-12T20:06:12.936 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj33 2022-09-12T20:06:12.937 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qe50EUsx7aZIWIIpr968l7dK820= 2022-09-12T20:06:12.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qe50EUsx7aZIWIIpr968l7dK820=', 'Content-Length': '0'} 2022-09-12T20:06:12.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097c6bc0e91b02eea-00631f9134-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.940 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:12.941 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:12.941 DEBUG:boto:Method: GET 2022-09-12T20:06:12.942 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:06:12.942 DEBUG:boto:Data: 2022-09-12T20:06:12.943 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.944 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:12.944 DEBUG:boto:Port: 8000 2022-09-12T20:06:12.945 DEBUG:boto:Params: {} 2022-09-12T20:06:12.945 DEBUG:boto:Token: None 2022-09-12T20:06:12.946 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/obj33 2022-09-12T20:06:12.946 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qe50EUsx7aZIWIIpr968l7dK820= 2022-09-12T20:06:12.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qe50EUsx7aZIWIIpr968l7dK820=', 'Content-Length': '0'} 2022-09-12T20:06:12.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd62f477e1b6e223-00631f9134-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:12.990 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:12.991 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:12.991 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:06:12.992 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:06:12.992 DEBUG:boto:Method: GET 2022-09-12T20:06:12.993 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:06:12.993 DEBUG:boto:Data: 2022-09-12T20:06:12.994 DEBUG:boto:Headers: {} 2022-09-12T20:06:12.994 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:12.995 DEBUG:boto:Port: 8001 2022-09-12T20:06:12.995 DEBUG:boto:Params: {} 2022-09-12T20:06:12.996 DEBUG:boto:Token: None 2022-09-12T20:06:12.997 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:12 GMT /jdpiak-75/ 2022-09-12T20:06:12.997 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0= 2022-09-12T20:06:12.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sgiwYkfw+L7bFFGOGJYUQ7rPQr0=', 'Content-Length': '0'} 2022-09-12T20:06:13.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a155718fdbd1f4a7-00631f9134-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.002 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:13.003 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:13.003 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:13.004 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:06:13.004 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:06:13.005 DEBUG:boto:Method: GET 2022-09-12T20:06:13.005 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:06:13.006 DEBUG:boto:Data: 2022-09-12T20:06:13.006 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.007 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.007 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.008 DEBUG:boto:Params: {} 2022-09-12T20:06:13.008 DEBUG:boto:Token: None 2022-09-12T20:06:13.009 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/ 2022-09-12T20:06:13.010 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE= 2022-09-12T20:06:13.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE=', 'Content-Length': '0'} 2022-09-12T20:06:13.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0350413f06bd9a7-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.014 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:13.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:06:13.016 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:13.016 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:13.017 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:13.017 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:13.018 DEBUG:boto:Method: GET 2022-09-12T20:06:13.018 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:13.019 DEBUG:boto:Data: 2022-09-12T20:06:13.019 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.020 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.020 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.021 DEBUG:boto:Params: {} 2022-09-12T20:06:13.021 DEBUG:boto:Token: None 2022-09-12T20:06:13.022 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:13.023 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Edq//RmBOayosr0zlxVFvnjJTwk= 2022-09-12T20:06:13.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Edq//RmBOayosr0zlxVFvnjJTwk=', 'Content-Length': '0'} 2022-09-12T20:06:13.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c37ad7c45e05116-00631f9135-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.026 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:13.027 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:13.028 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:13.028 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:13.029 DEBUG:boto:Method: GET 2022-09-12T20:06:13.029 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:13.030 DEBUG:boto:Data: 2022-09-12T20:06:13.030 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.031 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.031 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.032 DEBUG:boto:Params: {} 2022-09-12T20:06:13.032 DEBUG:boto:Token: None 2022-09-12T20:06:13.033 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:13.034 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Edq//RmBOayosr0zlxVFvnjJTwk= 2022-09-12T20:06:13.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Edq//RmBOayosr0zlxVFvnjJTwk=', 'Content-Length': '0'} 2022-09-12T20:06:13.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000945f47ada53ae462-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.037 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:13.038 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:13.038 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:13.039 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:13.039 DEBUG:boto:Method: HEAD 2022-09-12T20:06:13.040 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:13.040 DEBUG:boto:Data: 2022-09-12T20:06:13.041 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.041 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.042 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.042 DEBUG:boto:Params: {} 2022-09-12T20:06:13.043 DEBUG:boto:Token: None 2022-09-12T20:06:13.043 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:13.044 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qyBxbUCuM7zj2RpvbOpeAJ1JHZ4= 2022-09-12T20:06:13.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qyBxbUCuM7zj2RpvbOpeAJ1JHZ4=', 'Content-Length': '0'} 2022-09-12T20:06:13.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000484725e0ccf42520-00631f9135-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.047 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:13.048 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:13.048 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:13.049 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:13.049 DEBUG:boto:Method: HEAD 2022-09-12T20:06:13.050 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:13.050 DEBUG:boto:Data: 2022-09-12T20:06:13.051 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.051 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.052 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.052 DEBUG:boto:Params: {} 2022-09-12T20:06:13.053 DEBUG:boto:Token: None 2022-09-12T20:06:13.053 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:13.054 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qyBxbUCuM7zj2RpvbOpeAJ1JHZ4= 2022-09-12T20:06:13.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qyBxbUCuM7zj2RpvbOpeAJ1JHZ4=', 'Content-Length': '0'} 2022-09-12T20:06:13.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ea3b702c1fae7f3-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:06:13.057 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:13.058 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:13.059 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:13.059 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:13.060 DEBUG:boto:Method: GET 2022-09-12T20:06:13.060 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:13.061 DEBUG:boto:Data: 2022-09-12T20:06:13.061 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.062 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.062 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.063 DEBUG:boto:Params: {} 2022-09-12T20:06:13.063 DEBUG:boto:Token: None 2022-09-12T20:06:13.064 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:13.065 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Edq//RmBOayosr0zlxVFvnjJTwk= 2022-09-12T20:06:13.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Edq//RmBOayosr0zlxVFvnjJTwk=', 'Content-Length': '0'} 2022-09-12T20:06:13.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf69d3f31c10027b-00631f9135-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.068 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:13.069 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:13.069 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:13.070 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:13.070 DEBUG:boto:Method: GET 2022-09-12T20:06:13.071 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:13.071 DEBUG:boto:Data: 2022-09-12T20:06:13.072 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.072 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.073 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.073 DEBUG:boto:Params: {} 2022-09-12T20:06:13.074 DEBUG:boto:Token: None 2022-09-12T20:06:13.075 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:13.075 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Edq//RmBOayosr0zlxVFvnjJTwk= 2022-09-12T20:06:13.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Edq//RmBOayosr0zlxVFvnjJTwk=', 'Content-Length': '0'} 2022-09-12T20:06:13.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000218989d093482d77-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.079 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:13.079 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:13.080 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:06:13.080 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:06:13.081 DEBUG:boto:Method: GET 2022-09-12T20:06:13.081 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:06:13.082 DEBUG:boto:Data: 2022-09-12T20:06:13.082 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.083 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.083 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.084 DEBUG:boto:Params: {} 2022-09-12T20:06:13.084 DEBUG:boto:Token: None 2022-09-12T20:06:13.085 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/ 2022-09-12T20:06:13.086 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE= 2022-09-12T20:06:13.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE=', 'Content-Length': '0'} 2022-09-12T20:06:13.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0fdc0b9c2fc4aa0-00631f9135-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.090 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:13.091 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:13.091 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:13.092 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:06:13.092 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:06:13.093 DEBUG:boto:Method: GET 2022-09-12T20:06:13.093 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:06:13.094 DEBUG:boto:Data: 2022-09-12T20:06:13.094 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.095 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.095 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.096 DEBUG:boto:Params: {} 2022-09-12T20:06:13.097 DEBUG:boto:Token: None 2022-09-12T20:06:13.097 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/ 2022-09-12T20:06:13.098 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE= 2022-09-12T20:06:13.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE=', 'Content-Length': '0'} 2022-09-12T20:06:13.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047623d91e7e8d205-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.102 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:13.103 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:13.103 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:13.104 DEBUG:boto:Method: HEAD 2022-09-12T20:06:13.104 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:06:13.105 DEBUG:boto:Data: 2022-09-12T20:06:13.105 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.106 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.106 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.107 DEBUG:boto:Params: {} 2022-09-12T20:06:13.107 DEBUG:boto:Token: None 2022-09-12T20:06:13.108 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj34 2022-09-12T20:06:13.109 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZiuUmYQOPbSlnL+DEO/WyZgeyPU= 2022-09-12T20:06:13.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZiuUmYQOPbSlnL+DEO/WyZgeyPU=', 'Content-Length': '0'} 2022-09-12T20:06:13.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbc090634f8f7509-00631f9135-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.112 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:13.112 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:13.113 DEBUG:boto:Method: HEAD 2022-09-12T20:06:13.113 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:06:13.114 DEBUG:boto:Data: 2022-09-12T20:06:13.114 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.115 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.115 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.116 DEBUG:boto:Params: {} 2022-09-12T20:06:13.116 DEBUG:boto:Token: None 2022-09-12T20:06:13.117 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj34 2022-09-12T20:06:13.117 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZiuUmYQOPbSlnL+DEO/WyZgeyPU= 2022-09-12T20:06:13.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZiuUmYQOPbSlnL+DEO/WyZgeyPU=', 'Content-Length': '0'} 2022-09-12T20:06:13.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d483fb42f7bd54d7-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:06:13.121 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:13.122 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:13.123 DEBUG:boto:Method: GET 2022-09-12T20:06:13.123 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:06:13.124 DEBUG:boto:Data: 2022-09-12T20:06:13.124 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.125 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.126 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.126 DEBUG:boto:Params: {} 2022-09-12T20:06:13.127 DEBUG:boto:Token: None 2022-09-12T20:06:13.128 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj34 2022-09-12T20:06:13.128 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DsE3icxtIoUivOfvGAA5IBzAQys= 2022-09-12T20:06:13.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DsE3icxtIoUivOfvGAA5IBzAQys=', 'Content-Length': '0'} 2022-09-12T20:06:13.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a2871ae21dd325d-00631f9135-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.132 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:13.133 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:13.134 DEBUG:boto:Method: GET 2022-09-12T20:06:13.134 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:06:13.135 DEBUG:boto:Data: 2022-09-12T20:06:13.135 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.136 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.137 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.137 DEBUG:boto:Params: {} 2022-09-12T20:06:13.138 DEBUG:boto:Token: None 2022-09-12T20:06:13.139 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj34 2022-09-12T20:06:13.140 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DsE3icxtIoUivOfvGAA5IBzAQys= 2022-09-12T20:06:13.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DsE3icxtIoUivOfvGAA5IBzAQys=', 'Content-Length': '0'} 2022-09-12T20:06:13.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d9488a20edc0d0f-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.143 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:13.144 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:13.145 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:06:13.145 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:06:13.146 DEBUG:boto:Method: GET 2022-09-12T20:06:13.146 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:06:13.147 DEBUG:boto:Data: 2022-09-12T20:06:13.148 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.149 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.149 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.150 DEBUG:boto:Params: {} 2022-09-12T20:06:13.151 DEBUG:boto:Token: None 2022-09-12T20:06:13.152 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/ 2022-09-12T20:06:13.153 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE= 2022-09-12T20:06:13.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE=', 'Content-Length': '0'} 2022-09-12T20:06:13.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dd1473a55f238b4-00631f9135-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.158 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:13.159 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:13.160 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:13.161 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:06:13.162 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:06:13.163 DEBUG:boto:Method: GET 2022-09-12T20:06:13.164 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:06:13.165 DEBUG:boto:Data: 2022-09-12T20:06:13.165 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.166 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.167 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.168 DEBUG:boto:Params: {} 2022-09-12T20:06:13.169 DEBUG:boto:Token: None 2022-09-12T20:06:13.170 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/ 2022-09-12T20:06:13.171 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE= 2022-09-12T20:06:13.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE=', 'Content-Length': '0'} 2022-09-12T20:06:13.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b436f968398757c0-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.218 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:13.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:06:13.220 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:13.221 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:13.221 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:13.222 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:13.223 DEBUG:boto:Method: GET 2022-09-12T20:06:13.224 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:13.224 DEBUG:boto:Data: 2022-09-12T20:06:13.225 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.226 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.226 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.227 DEBUG:boto:Params: {} 2022-09-12T20:06:13.228 DEBUG:boto:Token: None 2022-09-12T20:06:13.228 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:13.229 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xBM9uv1bL22CpKCr1Pz11OTiOVA= 2022-09-12T20:06:13.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xBM9uv1bL22CpKCr1Pz11OTiOVA=', 'Content-Length': '0'} 2022-09-12T20:06:13.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001760c5e5c73e401d-00631f9135-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.233 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:13.234 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:13.235 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:13.235 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:13.236 DEBUG:boto:Method: GET 2022-09-12T20:06:13.237 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:13.237 DEBUG:boto:Data: 2022-09-12T20:06:13.238 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.238 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.239 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.240 DEBUG:boto:Params: {} 2022-09-12T20:06:13.240 DEBUG:boto:Token: None 2022-09-12T20:06:13.241 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:13.242 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xBM9uv1bL22CpKCr1Pz11OTiOVA= 2022-09-12T20:06:13.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xBM9uv1bL22CpKCr1Pz11OTiOVA=', 'Content-Length': '0'} 2022-09-12T20:06:13.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072480904e571e7e9-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.246 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:13.246 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:13.247 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:13.247 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:13.248 DEBUG:boto:Method: HEAD 2022-09-12T20:06:13.249 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:13.250 DEBUG:boto:Data: 2022-09-12T20:06:13.250 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.251 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.252 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.253 DEBUG:boto:Params: {} 2022-09-12T20:06:13.253 DEBUG:boto:Token: None 2022-09-12T20:06:13.254 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:13.255 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZvYagEDNlDWDiSO+ksZEIzDv6k= 2022-09-12T20:06:13.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZvYagEDNlDWDiSO+ksZEIzDv6k=', 'Content-Length': '0'} 2022-09-12T20:06:13.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b796675c8860ffbd-00631f9135-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.259 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:13.259 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:13.260 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:13.260 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:13.261 DEBUG:boto:Method: HEAD 2022-09-12T20:06:13.261 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:13.262 DEBUG:boto:Data: 2022-09-12T20:06:13.262 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.263 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.263 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.264 DEBUG:boto:Params: {} 2022-09-12T20:06:13.264 DEBUG:boto:Token: None 2022-09-12T20:06:13.265 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:13.266 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZvYagEDNlDWDiSO+ksZEIzDv6k= 2022-09-12T20:06:13.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZvYagEDNlDWDiSO+ksZEIzDv6k=', 'Content-Length': '0'} 2022-09-12T20:06:13.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078df50a936d5d1b9-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:06:13.269 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:13.270 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:13.271 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:13.271 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:13.272 DEBUG:boto:Method: GET 2022-09-12T20:06:13.272 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:13.273 DEBUG:boto:Data: 2022-09-12T20:06:13.273 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.274 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.274 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.275 DEBUG:boto:Params: {} 2022-09-12T20:06:13.276 DEBUG:boto:Token: None 2022-09-12T20:06:13.276 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:13.277 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xBM9uv1bL22CpKCr1Pz11OTiOVA= 2022-09-12T20:06:13.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xBM9uv1bL22CpKCr1Pz11OTiOVA=', 'Content-Length': '0'} 2022-09-12T20:06:13.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096618ae1e246311f-00631f9135-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.322 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:13.323 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:13.323 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:13.324 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:13.325 DEBUG:boto:Method: GET 2022-09-12T20:06:13.325 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:13.326 DEBUG:boto:Data: 2022-09-12T20:06:13.326 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.327 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.327 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.328 DEBUG:boto:Params: {} 2022-09-12T20:06:13.329 DEBUG:boto:Token: None 2022-09-12T20:06:13.329 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:13.330 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xBM9uv1bL22CpKCr1Pz11OTiOVA= 2022-09-12T20:06:13.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xBM9uv1bL22CpKCr1Pz11OTiOVA=', 'Content-Length': '0'} 2022-09-12T20:06:13.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1dc070621357112-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.334 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:13.334 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:13.335 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:06:13.335 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:06:13.336 DEBUG:boto:Method: GET 2022-09-12T20:06:13.336 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:06:13.337 DEBUG:boto:Data: 2022-09-12T20:06:13.338 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.338 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.339 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.339 DEBUG:boto:Params: {} 2022-09-12T20:06:13.340 DEBUG:boto:Token: None 2022-09-12T20:06:13.340 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/ 2022-09-12T20:06:13.341 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE= 2022-09-12T20:06:13.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE=', 'Content-Length': '0'} 2022-09-12T20:06:13.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000170c41e6ef449b10-00631f9135-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.345 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:13.346 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:13.347 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:13.347 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:06:13.348 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:06:13.348 DEBUG:boto:Method: GET 2022-09-12T20:06:13.349 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:06:13.349 DEBUG:boto:Data: 2022-09-12T20:06:13.350 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.350 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.351 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.351 DEBUG:boto:Params: {} 2022-09-12T20:06:13.352 DEBUG:boto:Token: None 2022-09-12T20:06:13.353 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/ 2022-09-12T20:06:13.353 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE= 2022-09-12T20:06:13.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE=', 'Content-Length': '0'} 2022-09-12T20:06:13.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df352c153af13428-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.358 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:13.359 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:13.359 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:13.360 DEBUG:boto:Method: HEAD 2022-09-12T20:06:13.360 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:06:13.361 DEBUG:boto:Data: 2022-09-12T20:06:13.361 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.362 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.362 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.363 DEBUG:boto:Params: {} 2022-09-12T20:06:13.363 DEBUG:boto:Token: None 2022-09-12T20:06:13.364 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj44 2022-09-12T20:06:13.364 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cqSftAHV7zUnaCXhvk89uSCTL8Q= 2022-09-12T20:06:13.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cqSftAHV7zUnaCXhvk89uSCTL8Q=', 'Content-Length': '0'} 2022-09-12T20:06:13.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a9391c93244aad8-00631f9135-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.368 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:13.368 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:13.369 DEBUG:boto:Method: HEAD 2022-09-12T20:06:13.369 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:06:13.370 DEBUG:boto:Data: 2022-09-12T20:06:13.370 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.371 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.371 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.372 DEBUG:boto:Params: {} 2022-09-12T20:06:13.372 DEBUG:boto:Token: None 2022-09-12T20:06:13.373 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj44 2022-09-12T20:06:13.373 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cqSftAHV7zUnaCXhvk89uSCTL8Q= 2022-09-12T20:06:13.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cqSftAHV7zUnaCXhvk89uSCTL8Q=', 'Content-Length': '0'} 2022-09-12T20:06:13.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000194ea1890d642835-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:06:13.377 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:13.378 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:13.378 DEBUG:boto:Method: GET 2022-09-12T20:06:13.379 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:06:13.379 DEBUG:boto:Data: 2022-09-12T20:06:13.380 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.381 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.381 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.382 DEBUG:boto:Params: {} 2022-09-12T20:06:13.383 DEBUG:boto:Token: None 2022-09-12T20:06:13.383 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj44 2022-09-12T20:06:13.384 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NMpylNmNcWPrG9XKUeNC3dNmBXE= 2022-09-12T20:06:13.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NMpylNmNcWPrG9XKUeNC3dNmBXE=', 'Content-Length': '0'} 2022-09-12T20:06:13.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b08f76867321d8ac-00631f9135-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.387 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:13.388 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:13.389 DEBUG:boto:Method: GET 2022-09-12T20:06:13.389 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:06:13.390 DEBUG:boto:Data: 2022-09-12T20:06:13.390 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.391 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.391 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.392 DEBUG:boto:Params: {} 2022-09-12T20:06:13.392 DEBUG:boto:Token: None 2022-09-12T20:06:13.393 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj44 2022-09-12T20:06:13.394 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NMpylNmNcWPrG9XKUeNC3dNmBXE= 2022-09-12T20:06:13.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NMpylNmNcWPrG9XKUeNC3dNmBXE=', 'Content-Length': '0'} 2022-09-12T20:06:13.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ca68833dfd75040-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.397 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:13.398 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:13.399 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:06:13.399 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:06:13.400 DEBUG:boto:Method: GET 2022-09-12T20:06:13.400 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:06:13.401 DEBUG:boto:Data: 2022-09-12T20:06:13.401 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.402 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.402 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.403 DEBUG:boto:Params: {} 2022-09-12T20:06:13.403 DEBUG:boto:Token: None 2022-09-12T20:06:13.404 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/ 2022-09-12T20:06:13.405 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE= 2022-09-12T20:06:13.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE=', 'Content-Length': '0'} 2022-09-12T20:06:13.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ea0c5037f4c1b5a-00631f9135-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.409 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:13.409 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:13.410 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:13.411 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:06:13.411 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:06:13.412 DEBUG:boto:Method: GET 2022-09-12T20:06:13.412 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:06:13.413 DEBUG:boto:Data: 2022-09-12T20:06:13.413 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.414 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.414 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.415 DEBUG:boto:Params: {} 2022-09-12T20:06:13.415 DEBUG:boto:Token: None 2022-09-12T20:06:13.416 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/ 2022-09-12T20:06:13.417 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE= 2022-09-12T20:06:13.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE=', 'Content-Length': '0'} 2022-09-12T20:06:13.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2e3edc9d893df13-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.421 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:13.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:06:13.423 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:13.423 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:13.424 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:13.424 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:13.425 DEBUG:boto:Method: GET 2022-09-12T20:06:13.425 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:13.426 DEBUG:boto:Data: 2022-09-12T20:06:13.426 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.427 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.427 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.428 DEBUG:boto:Params: {} 2022-09-12T20:06:13.429 DEBUG:boto:Token: None 2022-09-12T20:06:13.429 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:13.430 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SeIZkwG9H6s6OscxBr6VSAg6tE8= 2022-09-12T20:06:13.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SeIZkwG9H6s6OscxBr6VSAg6tE8=', 'Content-Length': '0'} 2022-09-12T20:06:13.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002617fd3dc98656a5-00631f9135-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.434 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:13.434 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:13.435 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:13.435 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:13.436 DEBUG:boto:Method: GET 2022-09-12T20:06:13.436 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:13.437 DEBUG:boto:Data: 2022-09-12T20:06:13.437 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.438 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.439 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.439 DEBUG:boto:Params: {} 2022-09-12T20:06:13.440 DEBUG:boto:Token: None 2022-09-12T20:06:13.440 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:13.441 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SeIZkwG9H6s6OscxBr6VSAg6tE8= 2022-09-12T20:06:13.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SeIZkwG9H6s6OscxBr6VSAg6tE8=', 'Content-Length': '0'} 2022-09-12T20:06:13.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1e08c5c12bf0e5c-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.444 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:13.445 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:13.445 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:13.446 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:13.446 DEBUG:boto:Method: HEAD 2022-09-12T20:06:13.447 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:13.447 DEBUG:boto:Data: 2022-09-12T20:06:13.448 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.448 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.449 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.449 DEBUG:boto:Params: {} 2022-09-12T20:06:13.450 DEBUG:boto:Token: None 2022-09-12T20:06:13.451 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:13.451 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:srLX/GH04OWwbugDKbPGMboTHzo= 2022-09-12T20:06:13.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:srLX/GH04OWwbugDKbPGMboTHzo=', 'Content-Length': '0'} 2022-09-12T20:06:13.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000bd58479ae85d93-00631f9135-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.455 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:13.455 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:13.456 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:13.456 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:13.457 DEBUG:boto:Method: HEAD 2022-09-12T20:06:13.457 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:13.458 DEBUG:boto:Data: 2022-09-12T20:06:13.458 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.459 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.459 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.460 DEBUG:boto:Params: {} 2022-09-12T20:06:13.460 DEBUG:boto:Token: None 2022-09-12T20:06:13.461 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:13.462 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:srLX/GH04OWwbugDKbPGMboTHzo= 2022-09-12T20:06:13.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:srLX/GH04OWwbugDKbPGMboTHzo=', 'Content-Length': '0'} 2022-09-12T20:06:13.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3dbb181e95aa1d9-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:06:13.466 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:13.466 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:13.467 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:13.467 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:13.468 DEBUG:boto:Method: GET 2022-09-12T20:06:13.468 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:13.469 DEBUG:boto:Data: 2022-09-12T20:06:13.469 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.470 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.471 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.471 DEBUG:boto:Params: {} 2022-09-12T20:06:13.472 DEBUG:boto:Token: None 2022-09-12T20:06:13.472 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:13.473 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SeIZkwG9H6s6OscxBr6VSAg6tE8= 2022-09-12T20:06:13.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SeIZkwG9H6s6OscxBr6VSAg6tE8=', 'Content-Length': '0'} 2022-09-12T20:06:13.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003239707c692af04d-00631f9135-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.518 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:13.519 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:13.519 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:13.520 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:13.520 DEBUG:boto:Method: GET 2022-09-12T20:06:13.521 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:13.521 DEBUG:boto:Data: 2022-09-12T20:06:13.522 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.523 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.523 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.524 DEBUG:boto:Params: {} 2022-09-12T20:06:13.524 DEBUG:boto:Token: None 2022-09-12T20:06:13.525 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:13.525 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SeIZkwG9H6s6OscxBr6VSAg6tE8= 2022-09-12T20:06:13.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SeIZkwG9H6s6OscxBr6VSAg6tE8=', 'Content-Length': '0'} 2022-09-12T20:06:13.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1e17829cbae0462-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.529 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:13.530 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:13.530 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:06:13.531 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:06:13.531 DEBUG:boto:Method: GET 2022-09-12T20:06:13.532 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:06:13.532 DEBUG:boto:Data: 2022-09-12T20:06:13.533 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.533 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.534 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.534 DEBUG:boto:Params: {} 2022-09-12T20:06:13.535 DEBUG:boto:Token: None 2022-09-12T20:06:13.536 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/ 2022-09-12T20:06:13.536 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE= 2022-09-12T20:06:13.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE=', 'Content-Length': '0'} 2022-09-12T20:06:13.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e4585221339e6a5-00631f9135-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.540 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:13.541 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:13.542 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:13.542 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:06:13.543 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:06:13.543 DEBUG:boto:Method: GET 2022-09-12T20:06:13.544 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:06:13.544 DEBUG:boto:Data: 2022-09-12T20:06:13.545 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.545 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.546 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.546 DEBUG:boto:Params: {} 2022-09-12T20:06:13.547 DEBUG:boto:Token: None 2022-09-12T20:06:13.547 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/ 2022-09-12T20:06:13.548 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE= 2022-09-12T20:06:13.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE=', 'Content-Length': '0'} 2022-09-12T20:06:13.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000750770c322dc21a6-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.553 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:13.554 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:13.555 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:13.555 DEBUG:boto:Method: HEAD 2022-09-12T20:06:13.556 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:06:13.556 DEBUG:boto:Data: 2022-09-12T20:06:13.557 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.557 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.558 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.558 DEBUG:boto:Params: {} 2022-09-12T20:06:13.559 DEBUG:boto:Token: None 2022-09-12T20:06:13.559 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj51 2022-09-12T20:06:13.560 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cQMgMwrz0CpGPZFKjPh+yuO73aI= 2022-09-12T20:06:13.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cQMgMwrz0CpGPZFKjPh+yuO73aI=', 'Content-Length': '0'} 2022-09-12T20:06:13.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ff40e3b1fd4f220-00631f9135-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.563 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:13.564 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:13.564 DEBUG:boto:Method: HEAD 2022-09-12T20:06:13.565 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:06:13.565 DEBUG:boto:Data: 2022-09-12T20:06:13.566 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.566 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.567 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.567 DEBUG:boto:Params: {} 2022-09-12T20:06:13.568 DEBUG:boto:Token: None 2022-09-12T20:06:13.568 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj51 2022-09-12T20:06:13.569 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cQMgMwrz0CpGPZFKjPh+yuO73aI= 2022-09-12T20:06:13.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cQMgMwrz0CpGPZFKjPh+yuO73aI=', 'Content-Length': '0'} 2022-09-12T20:06:13.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed0abea084ab012c-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:06:13.573 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:13.573 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:13.574 DEBUG:boto:Method: GET 2022-09-12T20:06:13.574 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:06:13.575 DEBUG:boto:Data: 2022-09-12T20:06:13.576 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.576 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.577 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.577 DEBUG:boto:Params: {} 2022-09-12T20:06:13.578 DEBUG:boto:Token: None 2022-09-12T20:06:13.578 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj51 2022-09-12T20:06:13.579 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:I9NNDNiNz1icR+PN1R0rSaYUmKE= 2022-09-12T20:06:13.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:I9NNDNiNz1icR+PN1R0rSaYUmKE=', 'Content-Length': '0'} 2022-09-12T20:06:13.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088ebac17ad321d47-00631f9135-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.622 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:13.623 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:13.623 DEBUG:boto:Method: GET 2022-09-12T20:06:13.624 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:06:13.624 DEBUG:boto:Data: 2022-09-12T20:06:13.625 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.626 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.626 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.627 DEBUG:boto:Params: {} 2022-09-12T20:06:13.628 DEBUG:boto:Token: None 2022-09-12T20:06:13.628 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj51 2022-09-12T20:06:13.629 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:I9NNDNiNz1icR+PN1R0rSaYUmKE= 2022-09-12T20:06:13.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:I9NNDNiNz1icR+PN1R0rSaYUmKE=', 'Content-Length': '0'} 2022-09-12T20:06:13.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e5a88d39a327741-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.633 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:13.633 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:13.634 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:06:13.634 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:06:13.635 DEBUG:boto:Method: GET 2022-09-12T20:06:13.635 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:06:13.636 DEBUG:boto:Data: 2022-09-12T20:06:13.636 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.637 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.638 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.638 DEBUG:boto:Params: {} 2022-09-12T20:06:13.639 DEBUG:boto:Token: None 2022-09-12T20:06:13.640 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/ 2022-09-12T20:06:13.640 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE= 2022-09-12T20:06:13.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE=', 'Content-Length': '0'} 2022-09-12T20:06:13.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea6b22661595e612-00631f9135-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.645 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:13.646 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:13.647 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:13.648 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:06:13.649 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:06:13.649 DEBUG:boto:Method: GET 2022-09-12T20:06:13.650 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:06:13.651 DEBUG:boto:Data: 2022-09-12T20:06:13.651 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.652 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.652 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.653 DEBUG:boto:Params: {} 2022-09-12T20:06:13.654 DEBUG:boto:Token: None 2022-09-12T20:06:13.655 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/ 2022-09-12T20:06:13.655 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE= 2022-09-12T20:06:13.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE=', 'Content-Length': '0'} 2022-09-12T20:06:13.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b60564585feb12fa-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.660 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:13.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:06:13.661 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:13.662 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:13.663 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:13.663 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:13.664 DEBUG:boto:Method: GET 2022-09-12T20:06:13.664 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:13.665 DEBUG:boto:Data: 2022-09-12T20:06:13.665 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.666 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.667 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.667 DEBUG:boto:Params: {} 2022-09-12T20:06:13.668 DEBUG:boto:Token: None 2022-09-12T20:06:13.668 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:13.669 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0eIRvtpYiL7QIJz4E/vYlbjjldY= 2022-09-12T20:06:13.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0eIRvtpYiL7QIJz4E/vYlbjjldY=', 'Content-Length': '0'} 2022-09-12T20:06:13.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019edac541de36a2c-00631f9135-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.673 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:13.674 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:13.675 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:13.675 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:13.676 DEBUG:boto:Method: GET 2022-09-12T20:06:13.676 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:13.677 DEBUG:boto:Data: 2022-09-12T20:06:13.677 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.678 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.678 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.679 DEBUG:boto:Params: {} 2022-09-12T20:06:13.680 DEBUG:boto:Token: None 2022-09-12T20:06:13.681 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:13.681 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0eIRvtpYiL7QIJz4E/vYlbjjldY= 2022-09-12T20:06:13.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0eIRvtpYiL7QIJz4E/vYlbjjldY=', 'Content-Length': '0'} 2022-09-12T20:06:13.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063dce8ebe90d853f-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.726 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:13.727 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:13.727 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:13.728 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:13.728 DEBUG:boto:Method: HEAD 2022-09-12T20:06:13.729 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:13.729 DEBUG:boto:Data: 2022-09-12T20:06:13.730 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.730 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.731 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.732 DEBUG:boto:Params: {} 2022-09-12T20:06:13.732 DEBUG:boto:Token: None 2022-09-12T20:06:13.733 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:13.734 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PNSUqPbBJWwWLckulmmXQBoRXGk= 2022-09-12T20:06:13.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PNSUqPbBJWwWLckulmmXQBoRXGk=', 'Content-Length': '0'} 2022-09-12T20:06:13.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd588042853af0b9-00631f9135-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.737 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:13.738 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:13.738 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:13.739 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:13.739 DEBUG:boto:Method: HEAD 2022-09-12T20:06:13.740 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:13.740 DEBUG:boto:Data: 2022-09-12T20:06:13.741 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.741 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.742 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.743 DEBUG:boto:Params: {} 2022-09-12T20:06:13.743 DEBUG:boto:Token: None 2022-09-12T20:06:13.744 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:13.744 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PNSUqPbBJWwWLckulmmXQBoRXGk= 2022-09-12T20:06:13.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PNSUqPbBJWwWLckulmmXQBoRXGk=', 'Content-Length': '0'} 2022-09-12T20:06:13.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000212aca52fbe79a4f-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:06:13.748 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:13.749 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:13.749 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:13.750 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:13.750 DEBUG:boto:Method: GET 2022-09-12T20:06:13.751 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:13.752 DEBUG:boto:Data: 2022-09-12T20:06:13.752 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.753 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.753 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.754 DEBUG:boto:Params: {} 2022-09-12T20:06:13.755 DEBUG:boto:Token: None 2022-09-12T20:06:13.755 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:13.756 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0eIRvtpYiL7QIJz4E/vYlbjjldY= 2022-09-12T20:06:13.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0eIRvtpYiL7QIJz4E/vYlbjjldY=', 'Content-Length': '0'} 2022-09-12T20:06:13.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae3c63bc4480c840-00631f9135-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.759 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:13.760 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:13.761 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:13.761 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:13.762 DEBUG:boto:Method: GET 2022-09-12T20:06:13.762 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:13.763 DEBUG:boto:Data: 2022-09-12T20:06:13.764 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.764 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.765 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.765 DEBUG:boto:Params: {} 2022-09-12T20:06:13.766 DEBUG:boto:Token: None 2022-09-12T20:06:13.767 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:13.767 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0eIRvtpYiL7QIJz4E/vYlbjjldY= 2022-09-12T20:06:13.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0eIRvtpYiL7QIJz4E/vYlbjjldY=', 'Content-Length': '0'} 2022-09-12T20:06:13.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec55028050de3f32-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.771 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:13.771 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:13.772 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:06:13.772 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:06:13.773 DEBUG:boto:Method: GET 2022-09-12T20:06:13.774 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:06:13.774 DEBUG:boto:Data: 2022-09-12T20:06:13.775 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.775 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.776 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.776 DEBUG:boto:Params: {} 2022-09-12T20:06:13.777 DEBUG:boto:Token: None 2022-09-12T20:06:13.777 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/ 2022-09-12T20:06:13.778 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE= 2022-09-12T20:06:13.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE=', 'Content-Length': '0'} 2022-09-12T20:06:13.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ff0e999632d64df-00631f9135-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.782 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:13.783 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:13.784 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:13.785 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:06:13.785 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:06:13.786 DEBUG:boto:Method: GET 2022-09-12T20:06:13.786 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:06:13.787 DEBUG:boto:Data: 2022-09-12T20:06:13.787 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.788 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.789 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.789 DEBUG:boto:Params: {} 2022-09-12T20:06:13.790 DEBUG:boto:Token: None 2022-09-12T20:06:13.790 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/ 2022-09-12T20:06:13.791 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE= 2022-09-12T20:06:13.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE=', 'Content-Length': '0'} 2022-09-12T20:06:13.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ecbe4425cbfbc9d-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.797 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:13.798 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:13.799 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:13.799 DEBUG:boto:Method: HEAD 2022-09-12T20:06:13.800 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:06:13.800 DEBUG:boto:Data: 2022-09-12T20:06:13.801 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.802 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.802 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.803 DEBUG:boto:Params: {} 2022-09-12T20:06:13.803 DEBUG:boto:Token: None 2022-09-12T20:06:13.804 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj55 2022-09-12T20:06:13.805 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oqC77IgbaN9oPiJp3nSdvc27xFk= 2022-09-12T20:06:13.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oqC77IgbaN9oPiJp3nSdvc27xFk=', 'Content-Length': '0'} 2022-09-12T20:06:13.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1575f2758f62f8c-00631f9135-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.808 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:13.809 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:13.809 DEBUG:boto:Method: HEAD 2022-09-12T20:06:13.810 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:06:13.810 DEBUG:boto:Data: 2022-09-12T20:06:13.811 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.812 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.812 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.813 DEBUG:boto:Params: {} 2022-09-12T20:06:13.813 DEBUG:boto:Token: None 2022-09-12T20:06:13.814 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj55 2022-09-12T20:06:13.815 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oqC77IgbaN9oPiJp3nSdvc27xFk= 2022-09-12T20:06:13.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oqC77IgbaN9oPiJp3nSdvc27xFk=', 'Content-Length': '0'} 2022-09-12T20:06:13.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002089495328354fa9-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:06:13.819 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:13.819 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:13.820 DEBUG:boto:Method: GET 2022-09-12T20:06:13.821 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:06:13.821 DEBUG:boto:Data: 2022-09-12T20:06:13.822 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.822 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.823 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.824 DEBUG:boto:Params: {} 2022-09-12T20:06:13.825 DEBUG:boto:Token: None 2022-09-12T20:06:13.826 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj55 2022-09-12T20:06:13.827 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:veUbLdonpRn5ToLjJqEV/4JfWQA= 2022-09-12T20:06:13.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:veUbLdonpRn5ToLjJqEV/4JfWQA=', 'Content-Length': '0'} 2022-09-12T20:06:13.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b976880043de5d12-00631f9135-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.831 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:13.832 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:13.833 DEBUG:boto:Method: GET 2022-09-12T20:06:13.834 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:06:13.835 DEBUG:boto:Data: 2022-09-12T20:06:13.835 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.836 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.837 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.837 DEBUG:boto:Params: {} 2022-09-12T20:06:13.838 DEBUG:boto:Token: None 2022-09-12T20:06:13.839 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj55 2022-09-12T20:06:13.840 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:veUbLdonpRn5ToLjJqEV/4JfWQA= 2022-09-12T20:06:13.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:veUbLdonpRn5ToLjJqEV/4JfWQA=', 'Content-Length': '0'} 2022-09-12T20:06:13.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bffe8476bd81650-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.893 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:13.894 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:13.895 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:06:13.895 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:06:13.896 DEBUG:boto:Method: GET 2022-09-12T20:06:13.896 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:06:13.897 DEBUG:boto:Data: 2022-09-12T20:06:13.897 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.898 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.899 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.899 DEBUG:boto:Params: {} 2022-09-12T20:06:13.900 DEBUG:boto:Token: None 2022-09-12T20:06:13.900 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/ 2022-09-12T20:06:13.901 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE= 2022-09-12T20:06:13.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE=', 'Content-Length': '0'} 2022-09-12T20:06:13.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000157adc7c9457c92b-00631f9135-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.905 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:13.906 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:13.907 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:13.907 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:06:13.908 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:06:13.908 DEBUG:boto:Method: GET 2022-09-12T20:06:13.909 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:06:13.910 DEBUG:boto:Data: 2022-09-12T20:06:13.910 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.911 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.911 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.912 DEBUG:boto:Params: {} 2022-09-12T20:06:13.913 DEBUG:boto:Token: None 2022-09-12T20:06:13.913 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/ 2022-09-12T20:06:13.914 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE= 2022-09-12T20:06:13.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KyV+m9MrmQn3xsxOHDmxiEHQ2mE=', 'Content-Length': '0'} 2022-09-12T20:06:13.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd368e591ca0c83d-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.919 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:13.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:06:13.920 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:13.920 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:13.921 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:13.922 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:13.922 DEBUG:boto:Method: GET 2022-09-12T20:06:13.923 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:13.923 DEBUG:boto:Data: 2022-09-12T20:06:13.924 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.924 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.925 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.925 DEBUG:boto:Params: {} 2022-09-12T20:06:13.926 DEBUG:boto:Token: None 2022-09-12T20:06:13.927 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:13.927 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kgKjUuDs+8kTfTqDuutOwvuBEUc= 2022-09-12T20:06:13.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kgKjUuDs+8kTfTqDuutOwvuBEUc=', 'Content-Length': '0'} 2022-09-12T20:06:13.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025323cfff5bd4b73-00631f9135-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.931 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:13.932 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:13.932 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:13.933 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:13.933 DEBUG:boto:Method: GET 2022-09-12T20:06:13.934 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:13.935 DEBUG:boto:Data: 2022-09-12T20:06:13.935 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.936 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.936 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.937 DEBUG:boto:Params: {} 2022-09-12T20:06:13.937 DEBUG:boto:Token: None 2022-09-12T20:06:13.938 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:13.939 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kgKjUuDs+8kTfTqDuutOwvuBEUc= 2022-09-12T20:06:13.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kgKjUuDs+8kTfTqDuutOwvuBEUc=', 'Content-Length': '0'} 2022-09-12T20:06:13.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000418bde160514be30-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.942 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:13.943 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:13.943 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:13.944 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:13.944 DEBUG:boto:Method: HEAD 2022-09-12T20:06:13.945 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:13.945 DEBUG:boto:Data: 2022-09-12T20:06:13.946 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.946 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.947 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.947 DEBUG:boto:Params: {} 2022-09-12T20:06:13.948 DEBUG:boto:Token: None 2022-09-12T20:06:13.949 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:13.949 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ON/2gyyoWaxJeLl9fccHXA9K+ps= 2022-09-12T20:06:13.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ON/2gyyoWaxJeLl9fccHXA9K+ps=', 'Content-Length': '0'} 2022-09-12T20:06:13.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d056e862b222c64-00631f9135-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.953 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:13.953 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:13.954 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:13.954 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:13.955 DEBUG:boto:Method: HEAD 2022-09-12T20:06:13.955 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:13.956 DEBUG:boto:Data: 2022-09-12T20:06:13.956 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.957 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:13.957 DEBUG:boto:Port: 8000 2022-09-12T20:06:13.958 DEBUG:boto:Params: {} 2022-09-12T20:06:13.959 DEBUG:boto:Token: None 2022-09-12T20:06:13.959 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:13.960 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ON/2gyyoWaxJeLl9fccHXA9K+ps= 2022-09-12T20:06:13.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ON/2gyyoWaxJeLl9fccHXA9K+ps=', 'Content-Length': '0'} 2022-09-12T20:06:13.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d84ee0106378a4a2-00631f9135-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:13.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:06:13.964 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:13.964 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:13.965 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:13.965 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:13.966 DEBUG:boto:Method: GET 2022-09-12T20:06:13.967 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:13.967 DEBUG:boto:Data: 2022-09-12T20:06:13.968 DEBUG:boto:Headers: {} 2022-09-12T20:06:13.968 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:13.969 DEBUG:boto:Port: 8001 2022-09-12T20:06:13.969 DEBUG:boto:Params: {} 2022-09-12T20:06:13.970 DEBUG:boto:Token: None 2022-09-12T20:06:13.971 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:13 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:13.971 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kgKjUuDs+8kTfTqDuutOwvuBEUc= 2022-09-12T20:06:13.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kgKjUuDs+8kTfTqDuutOwvuBEUc=', 'Content-Length': '0'} 2022-09-12T20:06:13.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb2b93e4e74ac2f7-00631f9135-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.018 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:14.019 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:14.019 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:14.020 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:14.021 DEBUG:boto:Method: GET 2022-09-12T20:06:14.021 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:14.022 DEBUG:boto:Data: 2022-09-12T20:06:14.023 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.023 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:14.024 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.024 DEBUG:boto:Params: {} 2022-09-12T20:06:14.025 DEBUG:boto:Token: None 2022-09-12T20:06:14.026 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:14.026 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2sO8NYUTqRQpEJ93YOQKr7v0nPI= 2022-09-12T20:06:14.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2sO8NYUTqRQpEJ93YOQKr7v0nPI=', 'Content-Length': '0'} 2022-09-12T20:06:14.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6b66af5297a5aaa-00631f9136-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.030 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:14.031 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:14.031 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:06:14.032 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:06:14.033 DEBUG:boto:Method: GET 2022-09-12T20:06:14.033 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:06:14.034 DEBUG:boto:Data: 2022-09-12T20:06:14.034 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.035 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.035 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.036 DEBUG:boto:Params: {} 2022-09-12T20:06:14.037 DEBUG:boto:Token: None 2022-09-12T20:06:14.037 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/ 2022-09-12T20:06:14.038 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4= 2022-09-12T20:06:14.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4=', 'Content-Length': '0'} 2022-09-12T20:06:14.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f242a0aa1f940c7-00631f9136-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.042 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:14.043 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:14.044 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:14.044 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:06:14.045 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:06:14.045 DEBUG:boto:Method: GET 2022-09-12T20:06:14.046 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:06:14.046 DEBUG:boto:Data: 2022-09-12T20:06:14.047 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.047 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:14.048 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.048 DEBUG:boto:Params: {} 2022-09-12T20:06:14.049 DEBUG:boto:Token: None 2022-09-12T20:06:14.050 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/ 2022-09-12T20:06:14.051 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4= 2022-09-12T20:06:14.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4=', 'Content-Length': '0'} 2022-09-12T20:06:14.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009f5321e5986642e-00631f9136-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.056 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:14.057 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:14.057 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:14.058 DEBUG:boto:Method: HEAD 2022-09-12T20:06:14.058 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:06:14.059 DEBUG:boto:Data: 2022-09-12T20:06:14.059 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.060 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.061 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.062 DEBUG:boto:Params: {} 2022-09-12T20:06:14.063 DEBUG:boto:Token: None 2022-09-12T20:06:14.064 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj68 2022-09-12T20:06:14.064 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tasNOSzMeRuQ6wgV+jbNESMbrXY= 2022-09-12T20:06:14.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tasNOSzMeRuQ6wgV+jbNESMbrXY=', 'Content-Length': '0'} 2022-09-12T20:06:14.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce9586e4bb434ff0-00631f9136-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.068 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:14.068 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:14.069 DEBUG:boto:Method: HEAD 2022-09-12T20:06:14.069 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:06:14.070 DEBUG:boto:Data: 2022-09-12T20:06:14.070 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.071 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:14.071 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.072 DEBUG:boto:Params: {} 2022-09-12T20:06:14.073 DEBUG:boto:Token: None 2022-09-12T20:06:14.073 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj68 2022-09-12T20:06:14.074 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tasNOSzMeRuQ6wgV+jbNESMbrXY= 2022-09-12T20:06:14.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tasNOSzMeRuQ6wgV+jbNESMbrXY=', 'Content-Length': '0'} 2022-09-12T20:06:14.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e2882b6f299ce7b-00631f9136-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:06:14.080 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:14.081 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:14.082 DEBUG:boto:Method: GET 2022-09-12T20:06:14.083 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:06:14.084 DEBUG:boto:Data: 2022-09-12T20:06:14.085 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.086 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.087 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.088 DEBUG:boto:Params: {} 2022-09-12T20:06:14.089 DEBUG:boto:Token: None 2022-09-12T20:06:14.090 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj68 2022-09-12T20:06:14.092 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JQj69VnmcyYWTaAN8flL0/6W3LQ= 2022-09-12T20:06:14.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JQj69VnmcyYWTaAN8flL0/6W3LQ=', 'Content-Length': '0'} 2022-09-12T20:06:14.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e48c9571812003e0-00631f9136-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.097 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:14.098 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:14.099 DEBUG:boto:Method: GET 2022-09-12T20:06:14.099 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:06:14.100 DEBUG:boto:Data: 2022-09-12T20:06:14.101 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.102 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:14.103 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.104 DEBUG:boto:Params: {} 2022-09-12T20:06:14.105 DEBUG:boto:Token: None 2022-09-12T20:06:14.106 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj68 2022-09-12T20:06:14.107 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JQj69VnmcyYWTaAN8flL0/6W3LQ= 2022-09-12T20:06:14.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JQj69VnmcyYWTaAN8flL0/6W3LQ=', 'Content-Length': '0'} 2022-09-12T20:06:14.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ad23f06d7072a2f-00631f9136-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.150 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:14.151 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:14.151 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:06:14.152 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:06:14.152 DEBUG:boto:Method: GET 2022-09-12T20:06:14.153 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:06:14.153 DEBUG:boto:Data: 2022-09-12T20:06:14.154 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.154 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.155 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.155 DEBUG:boto:Params: {} 2022-09-12T20:06:14.156 DEBUG:boto:Token: None 2022-09-12T20:06:14.157 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/ 2022-09-12T20:06:14.157 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4= 2022-09-12T20:06:14.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4=', 'Content-Length': '0'} 2022-09-12T20:06:14.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001ee14a463622708-00631f9136-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.161 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:14.162 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:14.163 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:14.163 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:06:14.164 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:06:14.164 DEBUG:boto:Method: GET 2022-09-12T20:06:14.165 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:06:14.165 DEBUG:boto:Data: 2022-09-12T20:06:14.166 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.166 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:14.167 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.167 DEBUG:boto:Params: {} 2022-09-12T20:06:14.168 DEBUG:boto:Token: None 2022-09-12T20:06:14.168 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/ 2022-09-12T20:06:14.169 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4= 2022-09-12T20:06:14.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4=', 'Content-Length': '0'} 2022-09-12T20:06:14.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088a07eab1f38761d-00631f9136-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.173 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:14.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:06:14.175 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:14.175 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:14.176 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:14.176 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:14.177 DEBUG:boto:Method: GET 2022-09-12T20:06:14.177 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:14.178 DEBUG:boto:Data: 2022-09-12T20:06:14.178 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.179 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.179 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.180 DEBUG:boto:Params: {} 2022-09-12T20:06:14.180 DEBUG:boto:Token: None 2022-09-12T20:06:14.181 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:14.182 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9x/xkA7OS4PQ7dViJja/SFA0Svo= 2022-09-12T20:06:14.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9x/xkA7OS4PQ7dViJja/SFA0Svo=', 'Content-Length': '0'} 2022-09-12T20:06:14.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b3361a047d0ce73-00631f9136-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.185 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:14.186 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:14.186 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:14.187 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:14.187 DEBUG:boto:Method: GET 2022-09-12T20:06:14.188 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:14.188 DEBUG:boto:Data: 2022-09-12T20:06:14.189 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.189 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:14.190 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.190 DEBUG:boto:Params: {} 2022-09-12T20:06:14.191 DEBUG:boto:Token: None 2022-09-12T20:06:14.192 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:14.192 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9x/xkA7OS4PQ7dViJja/SFA0Svo= 2022-09-12T20:06:14.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9x/xkA7OS4PQ7dViJja/SFA0Svo=', 'Content-Length': '0'} 2022-09-12T20:06:14.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071bafebf906b5ec1-00631f9136-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.196 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:14.196 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:14.197 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:14.198 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:14.198 DEBUG:boto:Method: HEAD 2022-09-12T20:06:14.199 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:14.199 DEBUG:boto:Data: 2022-09-12T20:06:14.200 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.200 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.201 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.201 DEBUG:boto:Params: {} 2022-09-12T20:06:14.202 DEBUG:boto:Token: None 2022-09-12T20:06:14.203 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:14.203 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f27vff5mC4SPiw8KXdtnr0ING8w= 2022-09-12T20:06:14.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f27vff5mC4SPiw8KXdtnr0ING8w=', 'Content-Length': '0'} 2022-09-12T20:06:14.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfaab4cfe1e219a0-00631f9136-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.206 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:14.207 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:14.207 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:14.208 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:14.208 DEBUG:boto:Method: HEAD 2022-09-12T20:06:14.209 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:14.209 DEBUG:boto:Data: 2022-09-12T20:06:14.210 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.211 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:14.211 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.212 DEBUG:boto:Params: {} 2022-09-12T20:06:14.212 DEBUG:boto:Token: None 2022-09-12T20:06:14.213 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:14.213 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f27vff5mC4SPiw8KXdtnr0ING8w= 2022-09-12T20:06:14.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f27vff5mC4SPiw8KXdtnr0ING8w=', 'Content-Length': '0'} 2022-09-12T20:06:14.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e2723eb3eb34bbf-00631f9136-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:06:14.217 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:14.218 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:14.218 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:14.219 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:14.219 DEBUG:boto:Method: GET 2022-09-12T20:06:14.220 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:14.220 DEBUG:boto:Data: 2022-09-12T20:06:14.221 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.222 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.222 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.223 DEBUG:boto:Params: {} 2022-09-12T20:06:14.223 DEBUG:boto:Token: None 2022-09-12T20:06:14.224 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:14.225 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9x/xkA7OS4PQ7dViJja/SFA0Svo= 2022-09-12T20:06:14.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9x/xkA7OS4PQ7dViJja/SFA0Svo=', 'Content-Length': '0'} 2022-09-12T20:06:14.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000548c4b0c3af7396-00631f9136-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.228 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:14.229 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:14.229 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:14.230 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:14.230 DEBUG:boto:Method: GET 2022-09-12T20:06:14.231 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:14.231 DEBUG:boto:Data: 2022-09-12T20:06:14.232 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.232 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:14.233 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.233 DEBUG:boto:Params: {} 2022-09-12T20:06:14.234 DEBUG:boto:Token: None 2022-09-12T20:06:14.235 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:14.235 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9x/xkA7OS4PQ7dViJja/SFA0Svo= 2022-09-12T20:06:14.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9x/xkA7OS4PQ7dViJja/SFA0Svo=', 'Content-Length': '0'} 2022-09-12T20:06:14.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000317c8bbf898419e9-00631f9136-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.239 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:14.239 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:14.240 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:06:14.240 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:06:14.241 DEBUG:boto:Method: GET 2022-09-12T20:06:14.241 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:06:14.242 DEBUG:boto:Data: 2022-09-12T20:06:14.242 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.243 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.243 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.244 DEBUG:boto:Params: {} 2022-09-12T20:06:14.245 DEBUG:boto:Token: None 2022-09-12T20:06:14.245 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/ 2022-09-12T20:06:14.246 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4= 2022-09-12T20:06:14.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4=', 'Content-Length': '0'} 2022-09-12T20:06:14.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000068ca99bdf75ef0-00631f9136-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.250 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:14.251 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:14.251 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:14.252 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:06:14.252 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:06:14.253 DEBUG:boto:Method: GET 2022-09-12T20:06:14.254 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:06:14.254 DEBUG:boto:Data: 2022-09-12T20:06:14.255 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.255 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:14.256 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.256 DEBUG:boto:Params: {} 2022-09-12T20:06:14.257 DEBUG:boto:Token: None 2022-09-12T20:06:14.257 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/ 2022-09-12T20:06:14.258 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4= 2022-09-12T20:06:14.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4=', 'Content-Length': '0'} 2022-09-12T20:06:14.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2837b71e22fc940-00631f9136-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.306 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:14.307 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:14.308 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:14.308 DEBUG:boto:Method: HEAD 2022-09-12T20:06:14.309 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:06:14.309 DEBUG:boto:Data: 2022-09-12T20:06:14.310 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.310 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.311 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.311 DEBUG:boto:Params: {} 2022-09-12T20:06:14.312 DEBUG:boto:Token: None 2022-09-12T20:06:14.313 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj85 2022-09-12T20:06:14.313 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9LIJ1dSnOF7liU1IQIkxPcYzRDw= 2022-09-12T20:06:14.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9LIJ1dSnOF7liU1IQIkxPcYzRDw=', 'Content-Length': '0'} 2022-09-12T20:06:14.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000568e801370bc5f84-00631f9136-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.317 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:14.317 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:14.318 DEBUG:boto:Method: HEAD 2022-09-12T20:06:14.318 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:06:14.319 DEBUG:boto:Data: 2022-09-12T20:06:14.319 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.320 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:14.320 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.321 DEBUG:boto:Params: {} 2022-09-12T20:06:14.321 DEBUG:boto:Token: None 2022-09-12T20:06:14.322 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj85 2022-09-12T20:06:14.322 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9LIJ1dSnOF7liU1IQIkxPcYzRDw= 2022-09-12T20:06:14.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9LIJ1dSnOF7liU1IQIkxPcYzRDw=', 'Content-Length': '0'} 2022-09-12T20:06:14.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eef5d2905dd29684-00631f9136-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:06:14.326 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:14.327 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:14.328 DEBUG:boto:Method: GET 2022-09-12T20:06:14.328 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:06:14.329 DEBUG:boto:Data: 2022-09-12T20:06:14.329 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.330 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.330 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.331 DEBUG:boto:Params: {} 2022-09-12T20:06:14.331 DEBUG:boto:Token: None 2022-09-12T20:06:14.332 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj85 2022-09-12T20:06:14.333 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T2KS9AjRyJ0LrcKUStFPyvbBg5c= 2022-09-12T20:06:14.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T2KS9AjRyJ0LrcKUStFPyvbBg5c=', 'Content-Length': '0'} 2022-09-12T20:06:14.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db5a007cf6478c9e-00631f9136-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.336 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:14.337 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:14.337 DEBUG:boto:Method: GET 2022-09-12T20:06:14.338 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:06:14.339 DEBUG:boto:Data: 2022-09-12T20:06:14.339 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.340 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:14.340 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.341 DEBUG:boto:Params: {} 2022-09-12T20:06:14.341 DEBUG:boto:Token: None 2022-09-12T20:06:14.342 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj85 2022-09-12T20:06:14.343 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T2KS9AjRyJ0LrcKUStFPyvbBg5c= 2022-09-12T20:06:14.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T2KS9AjRyJ0LrcKUStFPyvbBg5c=', 'Content-Length': '0'} 2022-09-12T20:06:14.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cc9a71002c9e6fd-00631f9136-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.346 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:14.347 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:14.347 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:06:14.348 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:06:14.348 DEBUG:boto:Method: GET 2022-09-12T20:06:14.349 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:06:14.349 DEBUG:boto:Data: 2022-09-12T20:06:14.350 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.350 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.351 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.351 DEBUG:boto:Params: {} 2022-09-12T20:06:14.352 DEBUG:boto:Token: None 2022-09-12T20:06:14.353 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/ 2022-09-12T20:06:14.353 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4= 2022-09-12T20:06:14.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4=', 'Content-Length': '0'} 2022-09-12T20:06:14.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea0ce4e140dd0e3f-00631f9136-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.357 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:14.358 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:14.359 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:14.359 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:06:14.360 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:06:14.360 DEBUG:boto:Method: GET 2022-09-12T20:06:14.361 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:06:14.361 DEBUG:boto:Data: 2022-09-12T20:06:14.362 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.362 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:14.363 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.363 DEBUG:boto:Params: {} 2022-09-12T20:06:14.364 DEBUG:boto:Token: None 2022-09-12T20:06:14.365 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/ 2022-09-12T20:06:14.365 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4= 2022-09-12T20:06:14.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4=', 'Content-Length': '0'} 2022-09-12T20:06:14.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be2b029a7ac6a069-00631f9136-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.370 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:14.371 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-75 zones={test-zone3, test-zone1} 2022-09-12T20:06:14.372 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-75 zones={test-zone3, test-zone2} 2022-09-12T20:06:14.372 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:14.373 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:14.373 DEBUG:boto:Method: HEAD 2022-09-12T20:06:14.374 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:06:14.374 DEBUG:boto:Data: 2022-09-12T20:06:14.375 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.375 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.376 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.376 DEBUG:boto:Params: {} 2022-09-12T20:06:14.377 DEBUG:boto:Token: None 2022-09-12T20:06:14.378 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/ 2022-09-12T20:06:14.378 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cu7GEVfKE5bYvyARsuqlm76pwfM= 2022-09-12T20:06:14.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cu7GEVfKE5bYvyARsuqlm76pwfM=', 'Content-Length': '0'} 2022-09-12T20:06:14.381 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f2c31d30045b134-00631f9136-1826-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.382 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:14.383 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:14.383 DEBUG:boto:Method: HEAD 2022-09-12T20:06:14.384 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:06:14.384 DEBUG:boto:Data: 2022-09-12T20:06:14.385 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.385 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:14.386 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.386 DEBUG:boto:Params: {} 2022-09-12T20:06:14.387 DEBUG:boto:Token: None 2022-09-12T20:06:14.387 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/ 2022-09-12T20:06:14.388 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cu7GEVfKE5bYvyARsuqlm76pwfM= 2022-09-12T20:06:14.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cu7GEVfKE5bYvyARsuqlm76pwfM=', 'Content-Length': '0'} 2022-09-12T20:06:14.391 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005867d8f9b20e078b-00631f9136-189d-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.392 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T20:06:14.393 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:14.393 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:14.394 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:14.394 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:14.395 DEBUG:boto:Method: GET 2022-09-12T20:06:14.395 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:14.396 DEBUG:boto:Data: 2022-09-12T20:06:14.396 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.397 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.397 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.398 DEBUG:boto:Params: {} 2022-09-12T20:06:14.399 DEBUG:boto:Token: None 2022-09-12T20:06:14.399 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/?versions 2022-09-12T20:06:14.400 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OQFk6s9hRMEO5bCPVUpM5hlwUdE= 2022-09-12T20:06:14.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OQFk6s9hRMEO5bCPVUpM5hlwUdE=', 'Content-Length': '0'} 2022-09-12T20:06:14.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da5a48e451697003-00631f9136-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.404 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:14.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:06:14.406 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:06:14.406 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:06:14.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:06:14.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:06:14.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:06:14.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:06:14.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:06:14.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:06:14.410 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:06:14.410 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T20:06:14.411 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:14.411 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:14.412 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:14.412 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:14.413 DEBUG:boto:Method: GET 2022-09-12T20:06:14.413 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:14.414 DEBUG:boto:Data: 2022-09-12T20:06:14.414 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.415 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:14.415 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.416 DEBUG:boto:Params: {} 2022-09-12T20:06:14.417 DEBUG:boto:Token: None 2022-09-12T20:06:14.417 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/?versions 2022-09-12T20:06:14.418 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OQFk6s9hRMEO5bCPVUpM5hlwUdE= 2022-09-12T20:06:14.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OQFk6s9hRMEO5bCPVUpM5hlwUdE=', 'Content-Length': '0'} 2022-09-12T20:06:14.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024b6af9118a69bde-00631f9136-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.422 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:14.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:06:14.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:06:14.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:06:14.425 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:06:14.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:06:14.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:06:14.427 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:06:14.427 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:06:14.428 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:06:14.428 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:06:14.429 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:14.429 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:14.430 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:14.430 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:14.431 DEBUG:boto:Method: GET 2022-09-12T20:06:14.431 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:14.432 DEBUG:boto:Data: 2022-09-12T20:06:14.432 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.433 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.434 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.434 DEBUG:boto:Params: {} 2022-09-12T20:06:14.435 DEBUG:boto:Token: None 2022-09-12T20:06:14.435 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/?versions 2022-09-12T20:06:14.436 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OQFk6s9hRMEO5bCPVUpM5hlwUdE= 2022-09-12T20:06:14.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OQFk6s9hRMEO5bCPVUpM5hlwUdE=', 'Content-Length': '0'} 2022-09-12T20:06:14.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc7908e48a8ce1c6-00631f9136-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.440 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:14.442 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:14.442 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:14.443 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:14.443 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:14.444 DEBUG:boto:Method: GET 2022-09-12T20:06:14.444 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:06:14.445 DEBUG:boto:Data: 2022-09-12T20:06:14.446 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.446 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:14.447 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.447 DEBUG:boto:Params: {} 2022-09-12T20:06:14.448 DEBUG:boto:Token: None 2022-09-12T20:06:14.448 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/?versions 2022-09-12T20:06:14.449 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OQFk6s9hRMEO5bCPVUpM5hlwUdE= 2022-09-12T20:06:14.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OQFk6s9hRMEO5bCPVUpM5hlwUdE=', 'Content-Length': '0'} 2022-09-12T20:06:14.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2e6a2cddbfd904e-00631f9136-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.453 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:14.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:06:14.455 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:14.456 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:14.456 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:14.457 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:14.457 DEBUG:boto:Method: GET 2022-09-12T20:06:14.458 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:14.459 DEBUG:boto:Data: 2022-09-12T20:06:14.459 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.460 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.460 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.461 DEBUG:boto:Params: {} 2022-09-12T20:06:14.461 DEBUG:boto:Token: None 2022-09-12T20:06:14.462 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:14.463 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mPbwG7ZmQ0Gx5HSIiuzhRcbegwI= 2022-09-12T20:06:14.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mPbwG7ZmQ0Gx5HSIiuzhRcbegwI=', 'Content-Length': '0'} 2022-09-12T20:06:14.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b836e8f3591c1da-00631f9136-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.506 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:14.507 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:14.507 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:14.508 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:14.508 DEBUG:boto:Method: GET 2022-09-12T20:06:14.509 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:14.509 DEBUG:boto:Data: 2022-09-12T20:06:14.510 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.510 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:14.511 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.512 DEBUG:boto:Params: {} 2022-09-12T20:06:14.512 DEBUG:boto:Token: None 2022-09-12T20:06:14.513 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:14.513 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mPbwG7ZmQ0Gx5HSIiuzhRcbegwI= 2022-09-12T20:06:14.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mPbwG7ZmQ0Gx5HSIiuzhRcbegwI=', 'Content-Length': '0'} 2022-09-12T20:06:14.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011e6a567b3501a6d-00631f9136-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.524 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:14.525 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:14.525 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:14.526 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:14.526 DEBUG:boto:Method: HEAD 2022-09-12T20:06:14.527 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:14.527 DEBUG:boto:Data: 2022-09-12T20:06:14.528 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.528 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.529 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.529 DEBUG:boto:Params: {} 2022-09-12T20:06:14.530 DEBUG:boto:Token: None 2022-09-12T20:06:14.531 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:14.531 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LC4nX9w6LOOdDyaqEA8B3OpVSWE= 2022-09-12T20:06:14.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LC4nX9w6LOOdDyaqEA8B3OpVSWE=', 'Content-Length': '0'} 2022-09-12T20:06:14.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094cc3e5986f4368a-00631f9136-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.534 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:14.535 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:14.536 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:14.536 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:14.537 DEBUG:boto:Method: HEAD 2022-09-12T20:06:14.537 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:14.538 DEBUG:boto:Data: 2022-09-12T20:06:14.538 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.539 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:14.539 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.540 DEBUG:boto:Params: {} 2022-09-12T20:06:14.540 DEBUG:boto:Token: None 2022-09-12T20:06:14.541 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:14.541 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LC4nX9w6LOOdDyaqEA8B3OpVSWE= 2022-09-12T20:06:14.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LC4nX9w6LOOdDyaqEA8B3OpVSWE=', 'Content-Length': '0'} 2022-09-12T20:06:14.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f038d9e387f22ecd-00631f9136-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:06:14.545 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:14.546 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:14.547 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:14.547 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:14.548 DEBUG:boto:Method: GET 2022-09-12T20:06:14.548 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:14.549 DEBUG:boto:Data: 2022-09-12T20:06:14.549 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.550 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.550 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.551 DEBUG:boto:Params: {} 2022-09-12T20:06:14.552 DEBUG:boto:Token: None 2022-09-12T20:06:14.552 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:14.553 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mPbwG7ZmQ0Gx5HSIiuzhRcbegwI= 2022-09-12T20:06:14.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mPbwG7ZmQ0Gx5HSIiuzhRcbegwI=', 'Content-Length': '0'} 2022-09-12T20:06:14.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f771eb12d252cbd-00631f9136-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.556 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:14.557 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:14.558 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:14.558 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:06:14.559 DEBUG:boto:Method: GET 2022-09-12T20:06:14.559 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:14.560 DEBUG:boto:Data: 2022-09-12T20:06:14.560 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.561 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:14.561 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.562 DEBUG:boto:Params: {} 2022-09-12T20:06:14.562 DEBUG:boto:Token: None 2022-09-12T20:06:14.563 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:06:14.564 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mPbwG7ZmQ0Gx5HSIiuzhRcbegwI= 2022-09-12T20:06:14.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mPbwG7ZmQ0Gx5HSIiuzhRcbegwI=', 'Content-Length': '0'} 2022-09-12T20:06:14.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019f5d1ab4b59ba5a-00631f9136-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.567 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:14.568 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:14.568 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:06:14.569 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:06:14.569 DEBUG:boto:Method: GET 2022-09-12T20:06:14.570 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:06:14.570 DEBUG:boto:Data: 2022-09-12T20:06:14.571 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.571 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.572 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.572 DEBUG:boto:Params: {} 2022-09-12T20:06:14.573 DEBUG:boto:Token: None 2022-09-12T20:06:14.574 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/ 2022-09-12T20:06:14.574 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4= 2022-09-12T20:06:14.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4=', 'Content-Length': '0'} 2022-09-12T20:06:14.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fba43661c54fb295-00631f9136-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.578 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:14.579 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:14.580 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:14.580 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:06:14.581 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:06:14.581 DEBUG:boto:Method: GET 2022-09-12T20:06:14.582 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:06:14.582 DEBUG:boto:Data: 2022-09-12T20:06:14.583 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.583 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:14.584 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.584 DEBUG:boto:Params: {} 2022-09-12T20:06:14.585 DEBUG:boto:Token: None 2022-09-12T20:06:14.586 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/ 2022-09-12T20:06:14.586 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4= 2022-09-12T20:06:14.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4=', 'Content-Length': '0'} 2022-09-12T20:06:14.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082231e19969010e6-00631f9136-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.590 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:14.591 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:14.592 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:14.592 DEBUG:boto:Method: HEAD 2022-09-12T20:06:14.593 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:06:14.593 DEBUG:boto:Data: 2022-09-12T20:06:14.594 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.594 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.595 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.595 DEBUG:boto:Params: {} 2022-09-12T20:06:14.596 DEBUG:boto:Token: None 2022-09-12T20:06:14.596 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj11 2022-09-12T20:06:14.597 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:caTHpyJyldYqHfbpj6QT/5PVegM= 2022-09-12T20:06:14.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:caTHpyJyldYqHfbpj6QT/5PVegM=', 'Content-Length': '0'} 2022-09-12T20:06:14.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f60c8d238bf3d30-00631f9136-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.600 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:14.601 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:14.601 DEBUG:boto:Method: HEAD 2022-09-12T20:06:14.602 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:06:14.602 DEBUG:boto:Data: 2022-09-12T20:06:14.603 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.603 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:14.604 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.604 DEBUG:boto:Params: {} 2022-09-12T20:06:14.605 DEBUG:boto:Token: None 2022-09-12T20:06:14.605 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj11 2022-09-12T20:06:14.606 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:caTHpyJyldYqHfbpj6QT/5PVegM= 2022-09-12T20:06:14.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:caTHpyJyldYqHfbpj6QT/5PVegM=', 'Content-Length': '0'} 2022-09-12T20:06:14.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000773e510ddb7f5a9a-00631f9136-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:06:14.610 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:14.611 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:14.611 DEBUG:boto:Method: GET 2022-09-12T20:06:14.612 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:06:14.612 DEBUG:boto:Data: 2022-09-12T20:06:14.613 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.613 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.614 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.615 DEBUG:boto:Params: {} 2022-09-12T20:06:14.615 DEBUG:boto:Token: None 2022-09-12T20:06:14.616 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj11 2022-09-12T20:06:14.616 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A9KBoXXaU8gEEq2peNJffhSS1m4= 2022-09-12T20:06:14.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A9KBoXXaU8gEEq2peNJffhSS1m4=', 'Content-Length': '0'} 2022-09-12T20:06:14.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e727c3622a88e9b-00631f9136-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.620 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:06:14.620 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:06:14.621 DEBUG:boto:Method: GET 2022-09-12T20:06:14.621 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:06:14.622 DEBUG:boto:Data: 2022-09-12T20:06:14.622 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.623 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:14.623 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.624 DEBUG:boto:Params: {} 2022-09-12T20:06:14.625 DEBUG:boto:Token: None 2022-09-12T20:06:14.625 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj11 2022-09-12T20:06:14.626 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A9KBoXXaU8gEEq2peNJffhSS1m4= 2022-09-12T20:06:14.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A9KBoXXaU8gEEq2peNJffhSS1m4=', 'Content-Length': '0'} 2022-09-12T20:06:14.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008918b3884504d57a-00631f9136-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.630 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:14.630 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:14.631 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:06:14.631 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:06:14.632 DEBUG:boto:Method: GET 2022-09-12T20:06:14.632 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:06:14.633 DEBUG:boto:Data: 2022-09-12T20:06:14.633 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.634 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.635 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.635 DEBUG:boto:Params: {} 2022-09-12T20:06:14.636 DEBUG:boto:Token: None 2022-09-12T20:06:14.636 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/ 2022-09-12T20:06:14.637 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4= 2022-09-12T20:06:14.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4=', 'Content-Length': '0'} 2022-09-12T20:06:14.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1ff4381f3577b49-00631f9136-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.641 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:14.642 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:14.643 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:14.643 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:06:14.644 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:06:14.644 DEBUG:boto:Method: GET 2022-09-12T20:06:14.645 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:06:14.645 DEBUG:boto:Data: 2022-09-12T20:06:14.646 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.646 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:14.647 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.648 DEBUG:boto:Params: {} 2022-09-12T20:06:14.648 DEBUG:boto:Token: None 2022-09-12T20:06:14.649 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/ 2022-09-12T20:06:14.649 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4= 2022-09-12T20:06:14.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4=', 'Content-Length': '0'} 2022-09-12T20:06:14.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf3570bf3e27b93d-00631f9136-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.654 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:14.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:06:14.655 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:14.656 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:14.656 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:14.657 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:14.657 DEBUG:boto:Method: GET 2022-09-12T20:06:14.658 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:14.658 DEBUG:boto:Data: 2022-09-12T20:06:14.659 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.659 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.660 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.660 DEBUG:boto:Params: {} 2022-09-12T20:06:14.661 DEBUG:boto:Token: None 2022-09-12T20:06:14.662 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:14.663 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rpzgYe2pOjUx06qZO6ZIG3LXFw0= 2022-09-12T20:06:14.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rpzgYe2pOjUx06qZO6ZIG3LXFw0=', 'Content-Length': '0'} 2022-09-12T20:06:14.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a2142cbf2cbff02-00631f9136-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.667 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:14.668 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:14.668 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:14.669 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:14.669 DEBUG:boto:Method: GET 2022-09-12T20:06:14.670 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:14.670 DEBUG:boto:Data: 2022-09-12T20:06:14.671 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.672 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:14.672 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.673 DEBUG:boto:Params: {} 2022-09-12T20:06:14.673 DEBUG:boto:Token: None 2022-09-12T20:06:14.674 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:14.675 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rpzgYe2pOjUx06qZO6ZIG3LXFw0= 2022-09-12T20:06:14.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rpzgYe2pOjUx06qZO6ZIG3LXFw0=', 'Content-Length': '0'} 2022-09-12T20:06:14.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000907626b237f6a52c-00631f9136-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.678 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:14.679 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:14.679 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:14.680 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:14.680 DEBUG:boto:Method: HEAD 2022-09-12T20:06:14.681 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:14.682 DEBUG:boto:Data: 2022-09-12T20:06:14.682 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.683 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.683 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.684 DEBUG:boto:Params: {} 2022-09-12T20:06:14.684 DEBUG:boto:Token: None 2022-09-12T20:06:14.685 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:14.685 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HWocRSyzfm0mD21tjsEhKAx6kos= 2022-09-12T20:06:14.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HWocRSyzfm0mD21tjsEhKAx6kos=', 'Content-Length': '0'} 2022-09-12T20:06:14.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000529dcc5da13246dd-00631f9136-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.689 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:14.689 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:14.690 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:14.690 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:14.691 DEBUG:boto:Method: HEAD 2022-09-12T20:06:14.691 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:14.692 DEBUG:boto:Data: 2022-09-12T20:06:14.692 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.693 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:14.693 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.694 DEBUG:boto:Params: {} 2022-09-12T20:06:14.695 DEBUG:boto:Token: None 2022-09-12T20:06:14.695 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:14.696 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HWocRSyzfm0mD21tjsEhKAx6kos= 2022-09-12T20:06:14.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HWocRSyzfm0mD21tjsEhKAx6kos=', 'Content-Length': '0'} 2022-09-12T20:06:14.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e33378a503a3dc0-00631f9136-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:06:14.699 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:14.700 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:14.701 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:14.701 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:14.702 DEBUG:boto:Method: GET 2022-09-12T20:06:14.702 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:14.703 DEBUG:boto:Data: 2022-09-12T20:06:14.703 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.704 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.704 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.705 DEBUG:boto:Params: {} 2022-09-12T20:06:14.706 DEBUG:boto:Token: None 2022-09-12T20:06:14.706 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:14.707 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rpzgYe2pOjUx06qZO6ZIG3LXFw0= 2022-09-12T20:06:14.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rpzgYe2pOjUx06qZO6ZIG3LXFw0=', 'Content-Length': '0'} 2022-09-12T20:06:14.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079efd3f9f8cdf995-00631f9136-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.711 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:14.711 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:14.712 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:14.712 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:06:14.713 DEBUG:boto:Method: GET 2022-09-12T20:06:14.713 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:14.714 DEBUG:boto:Data: 2022-09-12T20:06:14.715 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.715 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:14.716 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.716 DEBUG:boto:Params: {} 2022-09-12T20:06:14.717 DEBUG:boto:Token: None 2022-09-12T20:06:14.717 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:06:14.718 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rpzgYe2pOjUx06qZO6ZIG3LXFw0= 2022-09-12T20:06:14.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rpzgYe2pOjUx06qZO6ZIG3LXFw0=', 'Content-Length': '0'} 2022-09-12T20:06:14.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059b2ee60fdb5eda4-00631f9136-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.762 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:14.763 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:14.763 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:06:14.764 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:06:14.764 DEBUG:boto:Method: GET 2022-09-12T20:06:14.765 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:06:14.766 DEBUG:boto:Data: 2022-09-12T20:06:14.766 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.767 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.767 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.768 DEBUG:boto:Params: {} 2022-09-12T20:06:14.768 DEBUG:boto:Token: None 2022-09-12T20:06:14.769 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/ 2022-09-12T20:06:14.770 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4= 2022-09-12T20:06:14.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4=', 'Content-Length': '0'} 2022-09-12T20:06:14.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002234852843529bf1-00631f9136-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.774 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:14.775 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:14.775 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:14.776 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:06:14.777 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:06:14.777 DEBUG:boto:Method: GET 2022-09-12T20:06:14.778 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:06:14.778 DEBUG:boto:Data: 2022-09-12T20:06:14.779 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.779 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:14.780 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.780 DEBUG:boto:Params: {} 2022-09-12T20:06:14.781 DEBUG:boto:Token: None 2022-09-12T20:06:14.782 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/ 2022-09-12T20:06:14.782 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4= 2022-09-12T20:06:14.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4=', 'Content-Length': '0'} 2022-09-12T20:06:14.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ca85a6daef12f50-00631f9136-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.786 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:14.787 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:14.787 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:14.788 DEBUG:boto:Method: HEAD 2022-09-12T20:06:14.789 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:06:14.789 DEBUG:boto:Data: 2022-09-12T20:06:14.790 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.790 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.791 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.791 DEBUG:boto:Params: {} 2022-09-12T20:06:14.792 DEBUG:boto:Token: None 2022-09-12T20:06:14.792 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj17 2022-09-12T20:06:14.793 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DEEAI7DhvwXkiXTK2XJue0vHidY= 2022-09-12T20:06:14.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DEEAI7DhvwXkiXTK2XJue0vHidY=', 'Content-Length': '0'} 2022-09-12T20:06:14.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000969cd7bbcbbdf660-00631f9136-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.796 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:14.797 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:14.797 DEBUG:boto:Method: HEAD 2022-09-12T20:06:14.798 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:06:14.799 DEBUG:boto:Data: 2022-09-12T20:06:14.799 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.800 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:14.800 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.801 DEBUG:boto:Params: {} 2022-09-12T20:06:14.801 DEBUG:boto:Token: None 2022-09-12T20:06:14.802 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj17 2022-09-12T20:06:14.802 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DEEAI7DhvwXkiXTK2XJue0vHidY= 2022-09-12T20:06:14.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DEEAI7DhvwXkiXTK2XJue0vHidY=', 'Content-Length': '0'} 2022-09-12T20:06:14.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076db60379da7f8b3-00631f9136-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:06:14.806 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:14.807 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:14.808 DEBUG:boto:Method: GET 2022-09-12T20:06:14.808 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:06:14.809 DEBUG:boto:Data: 2022-09-12T20:06:14.809 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.810 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.810 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.811 DEBUG:boto:Params: {} 2022-09-12T20:06:14.811 DEBUG:boto:Token: None 2022-09-12T20:06:14.812 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj17 2022-09-12T20:06:14.813 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bC45djBeNo7JywoVEHMmdPufMXs= 2022-09-12T20:06:14.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bC45djBeNo7JywoVEHMmdPufMXs=', 'Content-Length': '0'} 2022-09-12T20:06:14.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0ae5313c6eb08cb-00631f9136-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.816 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:06:14.817 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:06:14.818 DEBUG:boto:Method: GET 2022-09-12T20:06:14.818 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:06:14.819 DEBUG:boto:Data: 2022-09-12T20:06:14.819 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.820 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:14.820 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.821 DEBUG:boto:Params: {} 2022-09-12T20:06:14.821 DEBUG:boto:Token: None 2022-09-12T20:06:14.822 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj17 2022-09-12T20:06:14.823 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bC45djBeNo7JywoVEHMmdPufMXs= 2022-09-12T20:06:14.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bC45djBeNo7JywoVEHMmdPufMXs=', 'Content-Length': '0'} 2022-09-12T20:06:14.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009668d0479ce330fd-00631f9136-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.826 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:14.827 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:14.827 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:06:14.828 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:06:14.828 DEBUG:boto:Method: GET 2022-09-12T20:06:14.829 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:06:14.829 DEBUG:boto:Data: 2022-09-12T20:06:14.830 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.831 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.831 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.832 DEBUG:boto:Params: {} 2022-09-12T20:06:14.832 DEBUG:boto:Token: None 2022-09-12T20:06:14.833 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/ 2022-09-12T20:06:14.833 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4= 2022-09-12T20:06:14.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4=', 'Content-Length': '0'} 2022-09-12T20:06:14.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d1a3dffd75258b0-00631f9136-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.838 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:14.839 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:14.839 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:14.840 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:06:14.840 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:06:14.841 DEBUG:boto:Method: GET 2022-09-12T20:06:14.841 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:06:14.842 DEBUG:boto:Data: 2022-09-12T20:06:14.842 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.843 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:14.844 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.844 DEBUG:boto:Params: {} 2022-09-12T20:06:14.845 DEBUG:boto:Token: None 2022-09-12T20:06:14.845 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/ 2022-09-12T20:06:14.846 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4= 2022-09-12T20:06:14.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4=', 'Content-Length': '0'} 2022-09-12T20:06:14.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022e7a1709e91e31e-00631f9136-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.850 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:14.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:06:14.851 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:14.852 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:14.852 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:14.853 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:14.854 DEBUG:boto:Method: GET 2022-09-12T20:06:14.854 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:14.855 DEBUG:boto:Data: 2022-09-12T20:06:14.855 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.856 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.856 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.857 DEBUG:boto:Params: {} 2022-09-12T20:06:14.857 DEBUG:boto:Token: None 2022-09-12T20:06:14.858 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:14.859 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:05/t3gcVZTUD33YiLdDBl7zjdvY= 2022-09-12T20:06:14.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:05/t3gcVZTUD33YiLdDBl7zjdvY=', 'Content-Length': '0'} 2022-09-12T20:06:14.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df766ae530f1cc8b-00631f9136-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.862 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:14.863 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:14.863 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:14.864 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:14.864 DEBUG:boto:Method: GET 2022-09-12T20:06:14.865 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:14.865 DEBUG:boto:Data: 2022-09-12T20:06:14.866 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.867 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:14.867 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.868 DEBUG:boto:Params: {} 2022-09-12T20:06:14.868 DEBUG:boto:Token: None 2022-09-12T20:06:14.869 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:14.869 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:05/t3gcVZTUD33YiLdDBl7zjdvY= 2022-09-12T20:06:14.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:05/t3gcVZTUD33YiLdDBl7zjdvY=', 'Content-Length': '0'} 2022-09-12T20:06:14.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1d4240075c16d3d-00631f9136-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.873 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:14.874 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:14.874 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:14.875 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:14.875 DEBUG:boto:Method: HEAD 2022-09-12T20:06:14.876 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:14.876 DEBUG:boto:Data: 2022-09-12T20:06:14.877 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.877 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.878 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.878 DEBUG:boto:Params: {} 2022-09-12T20:06:14.879 DEBUG:boto:Token: None 2022-09-12T20:06:14.880 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:14.880 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xpsFJZTDlnFT3zscTiIDD0lkjBk= 2022-09-12T20:06:14.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xpsFJZTDlnFT3zscTiIDD0lkjBk=', 'Content-Length': '0'} 2022-09-12T20:06:14.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a9c31017e3b420f-00631f9136-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.883 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:14.884 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:14.884 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:14.885 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:14.885 DEBUG:boto:Method: HEAD 2022-09-12T20:06:14.886 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:14.886 DEBUG:boto:Data: 2022-09-12T20:06:14.887 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.888 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:14.888 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.889 DEBUG:boto:Params: {} 2022-09-12T20:06:14.889 DEBUG:boto:Token: None 2022-09-12T20:06:14.890 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:14.890 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xpsFJZTDlnFT3zscTiIDD0lkjBk= 2022-09-12T20:06:14.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xpsFJZTDlnFT3zscTiIDD0lkjBk=', 'Content-Length': '0'} 2022-09-12T20:06:14.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a67ff1bd99b29140-00631f9136-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:06:14.894 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:14.895 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:14.895 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:14.896 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:14.897 DEBUG:boto:Method: GET 2022-09-12T20:06:14.897 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:14.898 DEBUG:boto:Data: 2022-09-12T20:06:14.898 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.899 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.899 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.900 DEBUG:boto:Params: {} 2022-09-12T20:06:14.900 DEBUG:boto:Token: None 2022-09-12T20:06:14.901 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:14.902 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:05/t3gcVZTUD33YiLdDBl7zjdvY= 2022-09-12T20:06:14.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:05/t3gcVZTUD33YiLdDBl7zjdvY=', 'Content-Length': '0'} 2022-09-12T20:06:14.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074945ea99f55d951-00631f9136-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.946 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:14.947 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:14.947 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:14.948 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:06:14.948 DEBUG:boto:Method: GET 2022-09-12T20:06:14.949 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:14.949 DEBUG:boto:Data: 2022-09-12T20:06:14.950 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.951 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:14.951 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.952 DEBUG:boto:Params: {} 2022-09-12T20:06:14.952 DEBUG:boto:Token: None 2022-09-12T20:06:14.953 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:06:14.954 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:05/t3gcVZTUD33YiLdDBl7zjdvY= 2022-09-12T20:06:14.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:05/t3gcVZTUD33YiLdDBl7zjdvY=', 'Content-Length': '0'} 2022-09-12T20:06:14.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba32f4a065eba413-00631f9136-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.957 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:14.958 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:14.958 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:06:14.959 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:06:14.959 DEBUG:boto:Method: GET 2022-09-12T20:06:14.960 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:06:14.961 DEBUG:boto:Data: 2022-09-12T20:06:14.961 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.962 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.962 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.963 DEBUG:boto:Params: {} 2022-09-12T20:06:14.963 DEBUG:boto:Token: None 2022-09-12T20:06:14.964 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/ 2022-09-12T20:06:14.964 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4= 2022-09-12T20:06:14.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4=', 'Content-Length': '0'} 2022-09-12T20:06:14.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb2c8901e4ff745a-00631f9136-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.969 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:14.969 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:14.970 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:14.971 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:06:14.971 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:06:14.972 DEBUG:boto:Method: GET 2022-09-12T20:06:14.972 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:06:14.973 DEBUG:boto:Data: 2022-09-12T20:06:14.973 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.974 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:14.974 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.975 DEBUG:boto:Params: {} 2022-09-12T20:06:14.975 DEBUG:boto:Token: None 2022-09-12T20:06:14.976 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/ 2022-09-12T20:06:14.977 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4= 2022-09-12T20:06:14.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A0ymkQQzq/HfUNSq/vw9Dsdedz4=', 'Content-Length': '0'} 2022-09-12T20:06:14.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005813f7abad85034-00631f9136-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.981 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:14.981 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:14.982 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:14.983 DEBUG:boto:Method: HEAD 2022-09-12T20:06:14.983 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:06:14.984 DEBUG:boto:Data: 2022-09-12T20:06:14.984 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.985 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:14.985 DEBUG:boto:Port: 8001 2022-09-12T20:06:14.986 DEBUG:boto:Params: {} 2022-09-12T20:06:14.986 DEBUG:boto:Token: None 2022-09-12T20:06:14.987 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj22 2022-09-12T20:06:14.987 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8guhfLYMhYsS+Kn+GLhUY5jSK48= 2022-09-12T20:06:14.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8guhfLYMhYsS+Kn+GLhUY5jSK48=', 'Content-Length': '0'} 2022-09-12T20:06:14.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000061630491d1f97fe-00631f9136-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:14.991 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:14.991 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:14.992 DEBUG:boto:Method: HEAD 2022-09-12T20:06:14.992 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:06:14.993 DEBUG:boto:Data: 2022-09-12T20:06:14.994 DEBUG:boto:Headers: {} 2022-09-12T20:06:14.994 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:14.995 DEBUG:boto:Port: 8000 2022-09-12T20:06:14.995 DEBUG:boto:Params: {} 2022-09-12T20:06:14.996 DEBUG:boto:Token: None 2022-09-12T20:06:14.996 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:14 GMT /jdpiak-75/obj22 2022-09-12T20:06:14.997 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8guhfLYMhYsS+Kn+GLhUY5jSK48= 2022-09-12T20:06:14.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8guhfLYMhYsS+Kn+GLhUY5jSK48=', 'Content-Length': '0'} 2022-09-12T20:06:14.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0e1421785f277a0-00631f9136-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:06:15.001 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:15.001 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:15.002 DEBUG:boto:Method: GET 2022-09-12T20:06:15.002 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:06:15.003 DEBUG:boto:Data: 2022-09-12T20:06:15.003 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.004 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.004 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.005 DEBUG:boto:Params: {} 2022-09-12T20:06:15.006 DEBUG:boto:Token: None 2022-09-12T20:06:15.006 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj22 2022-09-12T20:06:15.007 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6tjGqalApdXeo44e2bE6YGGPOok= 2022-09-12T20:06:15.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6tjGqalApdXeo44e2bE6YGGPOok=', 'Content-Length': '0'} 2022-09-12T20:06:15.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af13fde156b48875-00631f9137-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.011 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:06:15.011 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:06:15.012 DEBUG:boto:Method: GET 2022-09-12T20:06:15.012 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:06:15.013 DEBUG:boto:Data: 2022-09-12T20:06:15.013 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.014 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.014 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.015 DEBUG:boto:Params: {} 2022-09-12T20:06:15.015 DEBUG:boto:Token: None 2022-09-12T20:06:15.016 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj22 2022-09-12T20:06:15.017 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6tjGqalApdXeo44e2bE6YGGPOok= 2022-09-12T20:06:15.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6tjGqalApdXeo44e2bE6YGGPOok=', 'Content-Length': '0'} 2022-09-12T20:06:15.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003212ba210ea96732-00631f9137-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.020 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:15.021 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:15.021 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:06:15.022 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:06:15.022 DEBUG:boto:Method: GET 2022-09-12T20:06:15.023 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:06:15.023 DEBUG:boto:Data: 2022-09-12T20:06:15.024 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.025 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.025 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.026 DEBUG:boto:Params: {} 2022-09-12T20:06:15.026 DEBUG:boto:Token: None 2022-09-12T20:06:15.027 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/ 2022-09-12T20:06:15.027 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8= 2022-09-12T20:06:15.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8=', 'Content-Length': '0'} 2022-09-12T20:06:15.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc3be1c70ddae6ca-00631f9137-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.074 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:15.075 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:15.075 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:15.076 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:06:15.077 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:06:15.077 DEBUG:boto:Method: GET 2022-09-12T20:06:15.078 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:06:15.078 DEBUG:boto:Data: 2022-09-12T20:06:15.079 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.079 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.080 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.080 DEBUG:boto:Params: {} 2022-09-12T20:06:15.081 DEBUG:boto:Token: None 2022-09-12T20:06:15.082 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/ 2022-09-12T20:06:15.082 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8= 2022-09-12T20:06:15.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8=', 'Content-Length': '0'} 2022-09-12T20:06:15.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f88c03d7fe747584-00631f9137-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.086 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:15.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:06:15.088 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:15.088 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:15.089 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:15.089 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:15.090 DEBUG:boto:Method: GET 2022-09-12T20:06:15.090 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:15.091 DEBUG:boto:Data: 2022-09-12T20:06:15.091 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.092 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.092 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.093 DEBUG:boto:Params: {} 2022-09-12T20:06:15.093 DEBUG:boto:Token: None 2022-09-12T20:06:15.094 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:15.095 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7ub9/0m6+xzRBGEsKaqOwcdWxHo= 2022-09-12T20:06:15.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7ub9/0m6+xzRBGEsKaqOwcdWxHo=', 'Content-Length': '0'} 2022-09-12T20:06:15.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008f3f25f6b8b38f0-00631f9137-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.098 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:15.099 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:15.099 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:15.100 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:15.100 DEBUG:boto:Method: GET 2022-09-12T20:06:15.101 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:15.101 DEBUG:boto:Data: 2022-09-12T20:06:15.102 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.102 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.103 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.103 DEBUG:boto:Params: {} 2022-09-12T20:06:15.104 DEBUG:boto:Token: None 2022-09-12T20:06:15.105 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:15.105 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7ub9/0m6+xzRBGEsKaqOwcdWxHo= 2022-09-12T20:06:15.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7ub9/0m6+xzRBGEsKaqOwcdWxHo=', 'Content-Length': '0'} 2022-09-12T20:06:15.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035f09793b51d4768-00631f9137-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.109 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:15.109 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:15.110 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:15.110 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:15.111 DEBUG:boto:Method: HEAD 2022-09-12T20:06:15.111 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:15.112 DEBUG:boto:Data: 2022-09-12T20:06:15.112 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.113 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.113 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.114 DEBUG:boto:Params: {} 2022-09-12T20:06:15.115 DEBUG:boto:Token: None 2022-09-12T20:06:15.115 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:15.116 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3/Zf4WG8Ki0Lu9nmnPXjy9iNq1A= 2022-09-12T20:06:15.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3/Zf4WG8Ki0Lu9nmnPXjy9iNq1A=', 'Content-Length': '0'} 2022-09-12T20:06:15.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c08460216384519-00631f9137-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.119 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:15.119 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:15.120 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:15.120 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:15.121 DEBUG:boto:Method: HEAD 2022-09-12T20:06:15.121 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:15.122 DEBUG:boto:Data: 2022-09-12T20:06:15.122 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.123 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.123 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.124 DEBUG:boto:Params: {} 2022-09-12T20:06:15.125 DEBUG:boto:Token: None 2022-09-12T20:06:15.125 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:15.126 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3/Zf4WG8Ki0Lu9nmnPXjy9iNq1A= 2022-09-12T20:06:15.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3/Zf4WG8Ki0Lu9nmnPXjy9iNq1A=', 'Content-Length': '0'} 2022-09-12T20:06:15.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f849714e5a39bd01-00631f9137-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:06:15.130 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:15.130 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:15.131 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:15.131 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:15.132 DEBUG:boto:Method: GET 2022-09-12T20:06:15.133 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:15.133 DEBUG:boto:Data: 2022-09-12T20:06:15.134 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.134 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.135 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.136 DEBUG:boto:Params: {} 2022-09-12T20:06:15.136 DEBUG:boto:Token: None 2022-09-12T20:06:15.137 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:15.138 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7ub9/0m6+xzRBGEsKaqOwcdWxHo= 2022-09-12T20:06:15.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7ub9/0m6+xzRBGEsKaqOwcdWxHo=', 'Content-Length': '0'} 2022-09-12T20:06:15.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3ff251697bf086a-00631f9137-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.141 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:15.142 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:15.142 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:15.143 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:06:15.143 DEBUG:boto:Method: GET 2022-09-12T20:06:15.144 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:15.145 DEBUG:boto:Data: 2022-09-12T20:06:15.145 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.146 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.146 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.147 DEBUG:boto:Params: {} 2022-09-12T20:06:15.148 DEBUG:boto:Token: None 2022-09-12T20:06:15.148 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:06:15.149 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7ub9/0m6+xzRBGEsKaqOwcdWxHo= 2022-09-12T20:06:15.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7ub9/0m6+xzRBGEsKaqOwcdWxHo=', 'Content-Length': '0'} 2022-09-12T20:06:15.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2f81812900c27cb-00631f9137-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.153 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:15.154 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:15.154 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:06:15.155 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:06:15.155 DEBUG:boto:Method: GET 2022-09-12T20:06:15.156 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:06:15.156 DEBUG:boto:Data: 2022-09-12T20:06:15.157 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.157 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.158 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.159 DEBUG:boto:Params: {} 2022-09-12T20:06:15.159 DEBUG:boto:Token: None 2022-09-12T20:06:15.160 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/ 2022-09-12T20:06:15.161 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8= 2022-09-12T20:06:15.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8=', 'Content-Length': '0'} 2022-09-12T20:06:15.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082fe716291947fe1-00631f9137-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.165 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:15.166 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:15.166 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:15.167 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:06:15.167 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:06:15.168 DEBUG:boto:Method: GET 2022-09-12T20:06:15.169 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:06:15.169 DEBUG:boto:Data: 2022-09-12T20:06:15.170 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.170 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.171 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.172 DEBUG:boto:Params: {} 2022-09-12T20:06:15.172 DEBUG:boto:Token: None 2022-09-12T20:06:15.173 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/ 2022-09-12T20:06:15.174 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8= 2022-09-12T20:06:15.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8=', 'Content-Length': '0'} 2022-09-12T20:06:15.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000068ebf2c711750bd-00631f9137-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.178 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:15.179 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:15.179 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:15.180 DEBUG:boto:Method: HEAD 2022-09-12T20:06:15.180 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:06:15.181 DEBUG:boto:Data: 2022-09-12T20:06:15.182 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.182 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.183 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.183 DEBUG:boto:Params: {} 2022-09-12T20:06:15.184 DEBUG:boto:Token: None 2022-09-12T20:06:15.184 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj33 2022-09-12T20:06:15.185 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PJxYfFXXcACceFZ21fq0fDtw+IU= 2022-09-12T20:06:15.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PJxYfFXXcACceFZ21fq0fDtw+IU=', 'Content-Length': '0'} 2022-09-12T20:06:15.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057624da09024f21f-00631f9137-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.188 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:15.189 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:15.189 DEBUG:boto:Method: HEAD 2022-09-12T20:06:15.190 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:06:15.191 DEBUG:boto:Data: 2022-09-12T20:06:15.191 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.192 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.192 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.193 DEBUG:boto:Params: {} 2022-09-12T20:06:15.194 DEBUG:boto:Token: None 2022-09-12T20:06:15.194 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj33 2022-09-12T20:06:15.195 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PJxYfFXXcACceFZ21fq0fDtw+IU= 2022-09-12T20:06:15.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PJxYfFXXcACceFZ21fq0fDtw+IU=', 'Content-Length': '0'} 2022-09-12T20:06:15.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dd9dcbafceed581-00631f9137-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:06:15.199 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:15.200 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:15.200 DEBUG:boto:Method: GET 2022-09-12T20:06:15.201 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:06:15.201 DEBUG:boto:Data: 2022-09-12T20:06:15.202 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.203 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.203 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.204 DEBUG:boto:Params: {} 2022-09-12T20:06:15.205 DEBUG:boto:Token: None 2022-09-12T20:06:15.206 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj33 2022-09-12T20:06:15.206 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ni0ZQXZs+fEeE4DVYi+A2MlKSlw= 2022-09-12T20:06:15.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ni0ZQXZs+fEeE4DVYi+A2MlKSlw=', 'Content-Length': '0'} 2022-09-12T20:06:15.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ca701c83e307c7f-00631f9137-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.210 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:06:15.211 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:06:15.212 DEBUG:boto:Method: GET 2022-09-12T20:06:15.213 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:06:15.213 DEBUG:boto:Data: 2022-09-12T20:06:15.214 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.215 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.216 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.216 DEBUG:boto:Params: {} 2022-09-12T20:06:15.217 DEBUG:boto:Token: None 2022-09-12T20:06:15.218 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj33 2022-09-12T20:06:15.219 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ni0ZQXZs+fEeE4DVYi+A2MlKSlw= 2022-09-12T20:06:15.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ni0ZQXZs+fEeE4DVYi+A2MlKSlw=', 'Content-Length': '0'} 2022-09-12T20:06:15.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000466563533b237dcb-00631f9137-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.223 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:15.223 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:15.224 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:06:15.225 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:06:15.226 DEBUG:boto:Method: GET 2022-09-12T20:06:15.226 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:06:15.227 DEBUG:boto:Data: 2022-09-12T20:06:15.228 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.228 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.229 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.229 DEBUG:boto:Params: {} 2022-09-12T20:06:15.230 DEBUG:boto:Token: None 2022-09-12T20:06:15.231 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/ 2022-09-12T20:06:15.231 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8= 2022-09-12T20:06:15.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8=', 'Content-Length': '0'} 2022-09-12T20:06:15.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4fc66bb998aff3e-00631f9137-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.236 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:15.237 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:15.238 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:15.239 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:06:15.239 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:06:15.240 DEBUG:boto:Method: GET 2022-09-12T20:06:15.241 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:06:15.241 DEBUG:boto:Data: 2022-09-12T20:06:15.242 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.242 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.243 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.244 DEBUG:boto:Params: {} 2022-09-12T20:06:15.244 DEBUG:boto:Token: None 2022-09-12T20:06:15.245 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/ 2022-09-12T20:06:15.246 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8= 2022-09-12T20:06:15.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8=', 'Content-Length': '0'} 2022-09-12T20:06:15.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa0acc74adc172c8-00631f9137-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.250 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:15.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:06:15.252 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:15.252 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:15.253 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:15.253 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:15.254 DEBUG:boto:Method: GET 2022-09-12T20:06:15.254 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:15.255 DEBUG:boto:Data: 2022-09-12T20:06:15.256 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.256 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.257 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.257 DEBUG:boto:Params: {} 2022-09-12T20:06:15.258 DEBUG:boto:Token: None 2022-09-12T20:06:15.259 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:15.259 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NulvlFS/nDLD9HrhdfJBz9fZu6k= 2022-09-12T20:06:15.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NulvlFS/nDLD9HrhdfJBz9fZu6k=', 'Content-Length': '0'} 2022-09-12T20:06:15.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039653e436621482e-00631f9137-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.306 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:15.307 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:15.308 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:15.308 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:15.309 DEBUG:boto:Method: GET 2022-09-12T20:06:15.310 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:15.310 DEBUG:boto:Data: 2022-09-12T20:06:15.311 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.312 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.312 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.313 DEBUG:boto:Params: {} 2022-09-12T20:06:15.314 DEBUG:boto:Token: None 2022-09-12T20:06:15.315 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:15.316 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NulvlFS/nDLD9HrhdfJBz9fZu6k= 2022-09-12T20:06:15.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NulvlFS/nDLD9HrhdfJBz9fZu6k=', 'Content-Length': '0'} 2022-09-12T20:06:15.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004055bce0f870c2fa-00631f9137-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.320 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:15.321 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:15.321 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:15.322 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:15.323 DEBUG:boto:Method: HEAD 2022-09-12T20:06:15.324 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:15.325 DEBUG:boto:Data: 2022-09-12T20:06:15.326 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.327 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.328 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.329 DEBUG:boto:Params: {} 2022-09-12T20:06:15.330 DEBUG:boto:Token: None 2022-09-12T20:06:15.331 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:15.332 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DYdDiRubGKs7yRlx9qzJbZKUzx4= 2022-09-12T20:06:15.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DYdDiRubGKs7yRlx9qzJbZKUzx4=', 'Content-Length': '0'} 2022-09-12T20:06:15.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050fae692cc853927-00631f9137-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.337 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:15.338 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:15.338 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:15.339 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:15.339 DEBUG:boto:Method: HEAD 2022-09-12T20:06:15.340 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:15.340 DEBUG:boto:Data: 2022-09-12T20:06:15.341 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.341 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.342 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.342 DEBUG:boto:Params: {} 2022-09-12T20:06:15.343 DEBUG:boto:Token: None 2022-09-12T20:06:15.343 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:15.344 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DYdDiRubGKs7yRlx9qzJbZKUzx4= 2022-09-12T20:06:15.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DYdDiRubGKs7yRlx9qzJbZKUzx4=', 'Content-Length': '0'} 2022-09-12T20:06:15.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b21de4a6c8a8512a-00631f9137-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:06:15.348 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:15.349 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:15.349 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:15.350 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:15.350 DEBUG:boto:Method: GET 2022-09-12T20:06:15.351 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:15.351 DEBUG:boto:Data: 2022-09-12T20:06:15.352 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.352 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.353 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.353 DEBUG:boto:Params: {} 2022-09-12T20:06:15.354 DEBUG:boto:Token: None 2022-09-12T20:06:15.355 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:15.355 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NulvlFS/nDLD9HrhdfJBz9fZu6k= 2022-09-12T20:06:15.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NulvlFS/nDLD9HrhdfJBz9fZu6k=', 'Content-Length': '0'} 2022-09-12T20:06:15.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e765b2b3edeb281-00631f9137-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.359 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:15.359 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:15.360 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:15.360 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:06:15.361 DEBUG:boto:Method: GET 2022-09-12T20:06:15.362 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:15.362 DEBUG:boto:Data: 2022-09-12T20:06:15.363 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.363 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.364 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.364 DEBUG:boto:Params: {} 2022-09-12T20:06:15.365 DEBUG:boto:Token: None 2022-09-12T20:06:15.366 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:06:15.366 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NulvlFS/nDLD9HrhdfJBz9fZu6k= 2022-09-12T20:06:15.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NulvlFS/nDLD9HrhdfJBz9fZu6k=', 'Content-Length': '0'} 2022-09-12T20:06:15.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004628400ce06a37ab-00631f9137-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.410 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:15.411 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:15.411 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:06:15.412 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:06:15.413 DEBUG:boto:Method: GET 2022-09-12T20:06:15.413 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:06:15.414 DEBUG:boto:Data: 2022-09-12T20:06:15.414 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.415 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.416 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.416 DEBUG:boto:Params: {} 2022-09-12T20:06:15.417 DEBUG:boto:Token: None 2022-09-12T20:06:15.418 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/ 2022-09-12T20:06:15.418 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8= 2022-09-12T20:06:15.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8=', 'Content-Length': '0'} 2022-09-12T20:06:15.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f41dde53d96f68d-00631f9137-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.423 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:15.424 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:15.424 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:15.425 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:06:15.425 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:06:15.426 DEBUG:boto:Method: GET 2022-09-12T20:06:15.426 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:06:15.427 DEBUG:boto:Data: 2022-09-12T20:06:15.428 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.428 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.429 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.430 DEBUG:boto:Params: {} 2022-09-12T20:06:15.431 DEBUG:boto:Token: None 2022-09-12T20:06:15.432 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/ 2022-09-12T20:06:15.433 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8= 2022-09-12T20:06:15.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8=', 'Content-Length': '0'} 2022-09-12T20:06:15.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fe1d38b26889fb6-00631f9137-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.438 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:15.439 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:15.440 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:15.440 DEBUG:boto:Method: HEAD 2022-09-12T20:06:15.441 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:06:15.441 DEBUG:boto:Data: 2022-09-12T20:06:15.442 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.442 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.443 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.443 DEBUG:boto:Params: {} 2022-09-12T20:06:15.444 DEBUG:boto:Token: None 2022-09-12T20:06:15.444 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj34 2022-09-12T20:06:15.445 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hU+rnFGdKoPGgOWQcfrucv5GnRE= 2022-09-12T20:06:15.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hU+rnFGdKoPGgOWQcfrucv5GnRE=', 'Content-Length': '0'} 2022-09-12T20:06:15.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e438b09afae6ab6-00631f9137-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.448 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:15.449 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:15.449 DEBUG:boto:Method: HEAD 2022-09-12T20:06:15.450 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:06:15.450 DEBUG:boto:Data: 2022-09-12T20:06:15.451 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.451 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.452 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.452 DEBUG:boto:Params: {} 2022-09-12T20:06:15.453 DEBUG:boto:Token: None 2022-09-12T20:06:15.453 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj34 2022-09-12T20:06:15.454 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hU+rnFGdKoPGgOWQcfrucv5GnRE= 2022-09-12T20:06:15.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hU+rnFGdKoPGgOWQcfrucv5GnRE=', 'Content-Length': '0'} 2022-09-12T20:06:15.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068e8150a95dc2862-00631f9137-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:06:15.458 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:15.458 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:15.459 DEBUG:boto:Method: GET 2022-09-12T20:06:15.460 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:06:15.460 DEBUG:boto:Data: 2022-09-12T20:06:15.461 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.461 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.462 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.462 DEBUG:boto:Params: {} 2022-09-12T20:06:15.463 DEBUG:boto:Token: None 2022-09-12T20:06:15.463 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj34 2022-09-12T20:06:15.464 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FGd8I/MPvpji4jRZdutA2UgJHRQ= 2022-09-12T20:06:15.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FGd8I/MPvpji4jRZdutA2UgJHRQ=', 'Content-Length': '0'} 2022-09-12T20:06:15.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e60e2553bd4428f-00631f9137-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.468 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:06:15.468 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:06:15.469 DEBUG:boto:Method: GET 2022-09-12T20:06:15.469 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:06:15.470 DEBUG:boto:Data: 2022-09-12T20:06:15.470 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.471 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.471 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.472 DEBUG:boto:Params: {} 2022-09-12T20:06:15.472 DEBUG:boto:Token: None 2022-09-12T20:06:15.473 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj34 2022-09-12T20:06:15.474 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FGd8I/MPvpji4jRZdutA2UgJHRQ= 2022-09-12T20:06:15.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FGd8I/MPvpji4jRZdutA2UgJHRQ=', 'Content-Length': '0'} 2022-09-12T20:06:15.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e20eeb927adfc9dc-00631f9137-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.477 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:15.478 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:15.478 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:06:15.479 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:06:15.479 DEBUG:boto:Method: GET 2022-09-12T20:06:15.480 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:06:15.481 DEBUG:boto:Data: 2022-09-12T20:06:15.481 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.482 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.482 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.483 DEBUG:boto:Params: {} 2022-09-12T20:06:15.483 DEBUG:boto:Token: None 2022-09-12T20:06:15.484 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/ 2022-09-12T20:06:15.484 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8= 2022-09-12T20:06:15.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8=', 'Content-Length': '0'} 2022-09-12T20:06:15.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050c1f04e1608b9fa-00631f9137-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.530 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:15.531 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:15.532 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:15.532 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:06:15.533 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:06:15.533 DEBUG:boto:Method: GET 2022-09-12T20:06:15.534 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:06:15.534 DEBUG:boto:Data: 2022-09-12T20:06:15.535 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.535 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.536 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.536 DEBUG:boto:Params: {} 2022-09-12T20:06:15.537 DEBUG:boto:Token: None 2022-09-12T20:06:15.538 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/ 2022-09-12T20:06:15.538 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8= 2022-09-12T20:06:15.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8=', 'Content-Length': '0'} 2022-09-12T20:06:15.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006af36ce7fb19e902-00631f9137-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.542 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:15.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:06:15.544 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:15.544 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:15.545 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:15.545 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:15.546 DEBUG:boto:Method: GET 2022-09-12T20:06:15.546 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:15.547 DEBUG:boto:Data: 2022-09-12T20:06:15.547 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.548 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.548 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.549 DEBUG:boto:Params: {} 2022-09-12T20:06:15.550 DEBUG:boto:Token: None 2022-09-12T20:06:15.550 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:15.551 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NaVyD9zH/Vfiln+NXRIWO+ozJYM= 2022-09-12T20:06:15.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NaVyD9zH/Vfiln+NXRIWO+ozJYM=', 'Content-Length': '0'} 2022-09-12T20:06:15.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000095f36ceee921019-00631f9137-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.554 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:15.555 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:15.555 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:15.556 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:15.556 DEBUG:boto:Method: GET 2022-09-12T20:06:15.557 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:15.557 DEBUG:boto:Data: 2022-09-12T20:06:15.558 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.559 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.559 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.560 DEBUG:boto:Params: {} 2022-09-12T20:06:15.560 DEBUG:boto:Token: None 2022-09-12T20:06:15.561 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:15.561 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NaVyD9zH/Vfiln+NXRIWO+ozJYM= 2022-09-12T20:06:15.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NaVyD9zH/Vfiln+NXRIWO+ozJYM=', 'Content-Length': '0'} 2022-09-12T20:06:15.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067d8b971eb8ca3b7-00631f9137-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.565 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:15.566 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:15.566 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:15.567 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:15.567 DEBUG:boto:Method: HEAD 2022-09-12T20:06:15.568 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:15.568 DEBUG:boto:Data: 2022-09-12T20:06:15.569 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.569 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.570 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.570 DEBUG:boto:Params: {} 2022-09-12T20:06:15.571 DEBUG:boto:Token: None 2022-09-12T20:06:15.571 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:15.572 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vD97hZ03E53O2ObPBVWGT1yNKjY= 2022-09-12T20:06:15.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vD97hZ03E53O2ObPBVWGT1yNKjY=', 'Content-Length': '0'} 2022-09-12T20:06:15.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c35f8910dcc46928-00631f9137-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.575 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:15.576 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:15.577 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:15.577 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:15.578 DEBUG:boto:Method: HEAD 2022-09-12T20:06:15.578 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:15.579 DEBUG:boto:Data: 2022-09-12T20:06:15.579 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.580 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.580 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.581 DEBUG:boto:Params: {} 2022-09-12T20:06:15.581 DEBUG:boto:Token: None 2022-09-12T20:06:15.582 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:15.582 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vD97hZ03E53O2ObPBVWGT1yNKjY= 2022-09-12T20:06:15.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vD97hZ03E53O2ObPBVWGT1yNKjY=', 'Content-Length': '0'} 2022-09-12T20:06:15.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004aee609da459d97b-00631f9137-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:06:15.586 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:15.587 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:15.587 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:15.588 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:15.588 DEBUG:boto:Method: GET 2022-09-12T20:06:15.589 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:15.590 DEBUG:boto:Data: 2022-09-12T20:06:15.590 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.591 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.591 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.592 DEBUG:boto:Params: {} 2022-09-12T20:06:15.592 DEBUG:boto:Token: None 2022-09-12T20:06:15.593 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:15.594 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NaVyD9zH/Vfiln+NXRIWO+ozJYM= 2022-09-12T20:06:15.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NaVyD9zH/Vfiln+NXRIWO+ozJYM=', 'Content-Length': '0'} 2022-09-12T20:06:15.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd5b86ac5262a6e8-00631f9137-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.597 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:15.598 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:15.598 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:15.599 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:06:15.599 DEBUG:boto:Method: GET 2022-09-12T20:06:15.600 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:15.600 DEBUG:boto:Data: 2022-09-12T20:06:15.601 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.601 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.602 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.603 DEBUG:boto:Params: {} 2022-09-12T20:06:15.603 DEBUG:boto:Token: None 2022-09-12T20:06:15.604 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:06:15.604 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NaVyD9zH/Vfiln+NXRIWO+ozJYM= 2022-09-12T20:06:15.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NaVyD9zH/Vfiln+NXRIWO+ozJYM=', 'Content-Length': '0'} 2022-09-12T20:06:15.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b6103286eaafdb7-00631f9137-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.608 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:15.609 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:15.609 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:06:15.610 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:06:15.610 DEBUG:boto:Method: GET 2022-09-12T20:06:15.611 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:06:15.611 DEBUG:boto:Data: 2022-09-12T20:06:15.612 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.612 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.613 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.613 DEBUG:boto:Params: {} 2022-09-12T20:06:15.614 DEBUG:boto:Token: None 2022-09-12T20:06:15.615 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/ 2022-09-12T20:06:15.615 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8= 2022-09-12T20:06:15.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8=', 'Content-Length': '0'} 2022-09-12T20:06:15.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075244fd7f15c5cd6-00631f9137-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.619 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:15.620 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:15.621 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:15.621 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:06:15.622 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:06:15.622 DEBUG:boto:Method: GET 2022-09-12T20:06:15.623 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:06:15.623 DEBUG:boto:Data: 2022-09-12T20:06:15.624 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.624 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.625 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.625 DEBUG:boto:Params: {} 2022-09-12T20:06:15.626 DEBUG:boto:Token: None 2022-09-12T20:06:15.627 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/ 2022-09-12T20:06:15.627 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8= 2022-09-12T20:06:15.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8=', 'Content-Length': '0'} 2022-09-12T20:06:15.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ece48dcb6ebf132-00631f9137-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.631 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:15.632 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:15.633 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:15.633 DEBUG:boto:Method: HEAD 2022-09-12T20:06:15.634 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:06:15.634 DEBUG:boto:Data: 2022-09-12T20:06:15.635 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.635 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.636 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.636 DEBUG:boto:Params: {} 2022-09-12T20:06:15.637 DEBUG:boto:Token: None 2022-09-12T20:06:15.637 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj44 2022-09-12T20:06:15.638 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TDNP6ONB4CMScQLO8Fsqrc9FMMY= 2022-09-12T20:06:15.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TDNP6ONB4CMScQLO8Fsqrc9FMMY=', 'Content-Length': '0'} 2022-09-12T20:06:15.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cacbc47be475a059-00631f9137-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.641 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:15.642 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:15.642 DEBUG:boto:Method: HEAD 2022-09-12T20:06:15.643 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:06:15.643 DEBUG:boto:Data: 2022-09-12T20:06:15.644 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.645 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.645 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.646 DEBUG:boto:Params: {} 2022-09-12T20:06:15.646 DEBUG:boto:Token: None 2022-09-12T20:06:15.647 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj44 2022-09-12T20:06:15.648 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TDNP6ONB4CMScQLO8Fsqrc9FMMY= 2022-09-12T20:06:15.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TDNP6ONB4CMScQLO8Fsqrc9FMMY=', 'Content-Length': '0'} 2022-09-12T20:06:15.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076a1c5a112e64577-00631f9137-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:06:15.651 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:15.652 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:15.653 DEBUG:boto:Method: GET 2022-09-12T20:06:15.653 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:06:15.654 DEBUG:boto:Data: 2022-09-12T20:06:15.654 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.655 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.655 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.656 DEBUG:boto:Params: {} 2022-09-12T20:06:15.656 DEBUG:boto:Token: None 2022-09-12T20:06:15.657 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj44 2022-09-12T20:06:15.658 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MWvwVJlsZOev8esrtsU+XRXAmZ8= 2022-09-12T20:06:15.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MWvwVJlsZOev8esrtsU+XRXAmZ8=', 'Content-Length': '0'} 2022-09-12T20:06:15.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8058fd24a37b86b-00631f9137-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.702 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:06:15.703 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:06:15.704 DEBUG:boto:Method: GET 2022-09-12T20:06:15.705 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:06:15.706 DEBUG:boto:Data: 2022-09-12T20:06:15.706 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.707 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.707 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.708 DEBUG:boto:Params: {} 2022-09-12T20:06:15.708 DEBUG:boto:Token: None 2022-09-12T20:06:15.709 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj44 2022-09-12T20:06:15.710 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MWvwVJlsZOev8esrtsU+XRXAmZ8= 2022-09-12T20:06:15.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MWvwVJlsZOev8esrtsU+XRXAmZ8=', 'Content-Length': '0'} 2022-09-12T20:06:15.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5c04d57f12d2b0e-00631f9137-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.714 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:15.714 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:15.715 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:06:15.715 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:06:15.716 DEBUG:boto:Method: GET 2022-09-12T20:06:15.716 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:06:15.717 DEBUG:boto:Data: 2022-09-12T20:06:15.718 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.719 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.720 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.721 DEBUG:boto:Params: {} 2022-09-12T20:06:15.722 DEBUG:boto:Token: None 2022-09-12T20:06:15.723 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/ 2022-09-12T20:06:15.724 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8= 2022-09-12T20:06:15.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8=', 'Content-Length': '0'} 2022-09-12T20:06:15.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a565cc7ee7148ee8-00631f9137-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.728 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:15.729 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:15.730 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:15.731 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:06:15.732 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:06:15.732 DEBUG:boto:Method: GET 2022-09-12T20:06:15.733 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:06:15.734 DEBUG:boto:Data: 2022-09-12T20:06:15.735 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.736 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.737 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.738 DEBUG:boto:Params: {} 2022-09-12T20:06:15.739 DEBUG:boto:Token: None 2022-09-12T20:06:15.740 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/ 2022-09-12T20:06:15.741 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8= 2022-09-12T20:06:15.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8=', 'Content-Length': '0'} 2022-09-12T20:06:15.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fb2dffe5255b5e2-00631f9137-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.746 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:15.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:06:15.747 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:15.748 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:15.748 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:15.749 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:15.749 DEBUG:boto:Method: GET 2022-09-12T20:06:15.750 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:15.750 DEBUG:boto:Data: 2022-09-12T20:06:15.751 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.752 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.753 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.753 DEBUG:boto:Params: {} 2022-09-12T20:06:15.754 DEBUG:boto:Token: None 2022-09-12T20:06:15.755 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:15.756 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LlytKcOv0Ns5p+t/WTXJ8DHYpas= 2022-09-12T20:06:15.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LlytKcOv0Ns5p+t/WTXJ8DHYpas=', 'Content-Length': '0'} 2022-09-12T20:06:15.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004810bc906555c3c9-00631f9137-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.761 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:15.762 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:15.763 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:15.764 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:15.764 DEBUG:boto:Method: GET 2022-09-12T20:06:15.765 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:15.765 DEBUG:boto:Data: 2022-09-12T20:06:15.766 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.766 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.767 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.767 DEBUG:boto:Params: {} 2022-09-12T20:06:15.768 DEBUG:boto:Token: None 2022-09-12T20:06:15.769 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:15.769 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LlytKcOv0Ns5p+t/WTXJ8DHYpas= 2022-09-12T20:06:15.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LlytKcOv0Ns5p+t/WTXJ8DHYpas=', 'Content-Length': '0'} 2022-09-12T20:06:15.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d48785b048996102-00631f9137-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.773 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:15.774 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:15.775 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:15.775 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:15.776 DEBUG:boto:Method: HEAD 2022-09-12T20:06:15.777 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:15.778 DEBUG:boto:Data: 2022-09-12T20:06:15.779 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.779 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.780 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.781 DEBUG:boto:Params: {} 2022-09-12T20:06:15.782 DEBUG:boto:Token: None 2022-09-12T20:06:15.783 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:15.784 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PlmsZZ2kuP0fRFEM5PToA4mtAk4= 2022-09-12T20:06:15.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PlmsZZ2kuP0fRFEM5PToA4mtAk4=', 'Content-Length': '0'} 2022-09-12T20:06:15.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009057eb89b6f3d1d8-00631f9137-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.789 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:15.789 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:15.790 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:15.791 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:15.792 DEBUG:boto:Method: HEAD 2022-09-12T20:06:15.793 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:15.794 DEBUG:boto:Data: 2022-09-12T20:06:15.794 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.795 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.796 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.797 DEBUG:boto:Params: {} 2022-09-12T20:06:15.798 DEBUG:boto:Token: None 2022-09-12T20:06:15.799 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:15.800 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PlmsZZ2kuP0fRFEM5PToA4mtAk4= 2022-09-12T20:06:15.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PlmsZZ2kuP0fRFEM5PToA4mtAk4=', 'Content-Length': '0'} 2022-09-12T20:06:15.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041e725d3552d322f-00631f9137-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:06:15.805 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:15.806 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:15.807 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:15.807 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:15.808 DEBUG:boto:Method: GET 2022-09-12T20:06:15.808 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:15.809 DEBUG:boto:Data: 2022-09-12T20:06:15.809 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.810 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.810 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.811 DEBUG:boto:Params: {} 2022-09-12T20:06:15.812 DEBUG:boto:Token: None 2022-09-12T20:06:15.813 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:15.814 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LlytKcOv0Ns5p+t/WTXJ8DHYpas= 2022-09-12T20:06:15.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LlytKcOv0Ns5p+t/WTXJ8DHYpas=', 'Content-Length': '0'} 2022-09-12T20:06:15.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000406fd66d5b0f0b32-00631f9137-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.819 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:15.820 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:15.820 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:15.821 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:06:15.821 DEBUG:boto:Method: GET 2022-09-12T20:06:15.822 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:15.822 DEBUG:boto:Data: 2022-09-12T20:06:15.823 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.823 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.824 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.824 DEBUG:boto:Params: {} 2022-09-12T20:06:15.825 DEBUG:boto:Token: None 2022-09-12T20:06:15.826 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:06:15.826 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LlytKcOv0Ns5p+t/WTXJ8DHYpas= 2022-09-12T20:06:15.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LlytKcOv0Ns5p+t/WTXJ8DHYpas=', 'Content-Length': '0'} 2022-09-12T20:06:15.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c53cabfc8f8f474a-00631f9137-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.830 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:15.831 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:15.831 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:06:15.832 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:06:15.832 DEBUG:boto:Method: GET 2022-09-12T20:06:15.833 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:06:15.833 DEBUG:boto:Data: 2022-09-12T20:06:15.834 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.834 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.835 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.836 DEBUG:boto:Params: {} 2022-09-12T20:06:15.836 DEBUG:boto:Token: None 2022-09-12T20:06:15.837 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/ 2022-09-12T20:06:15.837 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8= 2022-09-12T20:06:15.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8=', 'Content-Length': '0'} 2022-09-12T20:06:15.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047e217d1f98ca908-00631f9137-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.842 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:15.842 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:15.843 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:15.844 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:06:15.844 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:06:15.845 DEBUG:boto:Method: GET 2022-09-12T20:06:15.845 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:06:15.846 DEBUG:boto:Data: 2022-09-12T20:06:15.846 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.847 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.847 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.848 DEBUG:boto:Params: {} 2022-09-12T20:06:15.848 DEBUG:boto:Token: None 2022-09-12T20:06:15.849 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/ 2022-09-12T20:06:15.850 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8= 2022-09-12T20:06:15.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8=', 'Content-Length': '0'} 2022-09-12T20:06:15.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a13c2b7ee5a900a3-00631f9137-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.853 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:15.854 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:15.855 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:15.855 DEBUG:boto:Method: HEAD 2022-09-12T20:06:15.856 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:06:15.856 DEBUG:boto:Data: 2022-09-12T20:06:15.857 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.857 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.858 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.858 DEBUG:boto:Params: {} 2022-09-12T20:06:15.859 DEBUG:boto:Token: None 2022-09-12T20:06:15.859 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj51 2022-09-12T20:06:15.860 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bsCm2eEaX8+VyCVsud7plsdfj68= 2022-09-12T20:06:15.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bsCm2eEaX8+VyCVsud7plsdfj68=', 'Content-Length': '0'} 2022-09-12T20:06:15.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3ff8d868d4e2652-00631f9137-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.863 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:15.864 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:15.864 DEBUG:boto:Method: HEAD 2022-09-12T20:06:15.865 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:06:15.865 DEBUG:boto:Data: 2022-09-12T20:06:15.866 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.866 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.867 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.867 DEBUG:boto:Params: {} 2022-09-12T20:06:15.868 DEBUG:boto:Token: None 2022-09-12T20:06:15.868 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj51 2022-09-12T20:06:15.869 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bsCm2eEaX8+VyCVsud7plsdfj68= 2022-09-12T20:06:15.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bsCm2eEaX8+VyCVsud7plsdfj68=', 'Content-Length': '0'} 2022-09-12T20:06:15.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4c0fc2c40d71515-00631f9137-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:06:15.873 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:15.873 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:15.874 DEBUG:boto:Method: GET 2022-09-12T20:06:15.874 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:06:15.875 DEBUG:boto:Data: 2022-09-12T20:06:15.875 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.876 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.877 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.877 DEBUG:boto:Params: {} 2022-09-12T20:06:15.878 DEBUG:boto:Token: None 2022-09-12T20:06:15.878 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj51 2022-09-12T20:06:15.879 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:buy6uZZAEVLt24FkY08vh4JUpd4= 2022-09-12T20:06:15.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:buy6uZZAEVLt24FkY08vh4JUpd4=', 'Content-Length': '0'} 2022-09-12T20:06:15.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ada758ebd4e33788-00631f9137-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.922 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:06:15.923 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:06:15.923 DEBUG:boto:Method: GET 2022-09-12T20:06:15.924 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:06:15.924 DEBUG:boto:Data: 2022-09-12T20:06:15.925 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.925 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.926 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.926 DEBUG:boto:Params: {} 2022-09-12T20:06:15.927 DEBUG:boto:Token: None 2022-09-12T20:06:15.928 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj51 2022-09-12T20:06:15.928 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:buy6uZZAEVLt24FkY08vh4JUpd4= 2022-09-12T20:06:15.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:buy6uZZAEVLt24FkY08vh4JUpd4=', 'Content-Length': '0'} 2022-09-12T20:06:15.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9ec9d2cd2d641e8-00631f9137-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.932 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:15.932 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:15.933 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:06:15.933 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:06:15.934 DEBUG:boto:Method: GET 2022-09-12T20:06:15.934 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:06:15.935 DEBUG:boto:Data: 2022-09-12T20:06:15.935 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.936 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.936 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.937 DEBUG:boto:Params: {} 2022-09-12T20:06:15.938 DEBUG:boto:Token: None 2022-09-12T20:06:15.938 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/ 2022-09-12T20:06:15.939 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8= 2022-09-12T20:06:15.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8=', 'Content-Length': '0'} 2022-09-12T20:06:15.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cf5ec64d70fbc34-00631f9137-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.943 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:15.943 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:15.944 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:15.945 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:06:15.945 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:06:15.946 DEBUG:boto:Method: GET 2022-09-12T20:06:15.946 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:06:15.947 DEBUG:boto:Data: 2022-09-12T20:06:15.947 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.948 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.948 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.949 DEBUG:boto:Params: {} 2022-09-12T20:06:15.949 DEBUG:boto:Token: None 2022-09-12T20:06:15.950 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/ 2022-09-12T20:06:15.950 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8= 2022-09-12T20:06:15.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uraNcnDELKPebIWemAGRGXytnS8=', 'Content-Length': '0'} 2022-09-12T20:06:15.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8a3fd93ca2dd9f5-00631f9137-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.954 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:15.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:06:15.955 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:15.956 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:15.956 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:15.957 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:15.957 DEBUG:boto:Method: GET 2022-09-12T20:06:15.958 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:15.959 DEBUG:boto:Data: 2022-09-12T20:06:15.959 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.960 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.960 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.961 DEBUG:boto:Params: {} 2022-09-12T20:06:15.961 DEBUG:boto:Token: None 2022-09-12T20:06:15.962 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:15.962 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:a0zm0RjklRnP5dmLk7A5dEojtwM= 2022-09-12T20:06:15.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:a0zm0RjklRnP5dmLk7A5dEojtwM=', 'Content-Length': '0'} 2022-09-12T20:06:15.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fc1c93157faec38-00631f9137-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.966 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:15.966 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:15.967 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:15.967 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:15.968 DEBUG:boto:Method: GET 2022-09-12T20:06:15.968 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:15.969 DEBUG:boto:Data: 2022-09-12T20:06:15.969 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.970 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.970 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.971 DEBUG:boto:Params: {} 2022-09-12T20:06:15.971 DEBUG:boto:Token: None 2022-09-12T20:06:15.972 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:15.973 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:a0zm0RjklRnP5dmLk7A5dEojtwM= 2022-09-12T20:06:15.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:a0zm0RjklRnP5dmLk7A5dEojtwM=', 'Content-Length': '0'} 2022-09-12T20:06:15.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004771d0ade4b0ac2d-00631f9137-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.976 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:15.977 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:15.977 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:15.978 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:15.978 DEBUG:boto:Method: HEAD 2022-09-12T20:06:15.979 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:15.979 DEBUG:boto:Data: 2022-09-12T20:06:15.980 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.980 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:15.981 DEBUG:boto:Port: 8001 2022-09-12T20:06:15.981 DEBUG:boto:Params: {} 2022-09-12T20:06:15.982 DEBUG:boto:Token: None 2022-09-12T20:06:15.982 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:15.983 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q5yRhaxVJU+VNFQCxX+IoNteao0= 2022-09-12T20:06:15.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q5yRhaxVJU+VNFQCxX+IoNteao0=', 'Content-Length': '0'} 2022-09-12T20:06:15.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f1f0fe0c41d1863-00631f9137-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.986 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:15.987 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:15.987 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:15.988 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:15.988 DEBUG:boto:Method: HEAD 2022-09-12T20:06:15.989 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:15.989 DEBUG:boto:Data: 2022-09-12T20:06:15.990 DEBUG:boto:Headers: {} 2022-09-12T20:06:15.990 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:15.992 DEBUG:boto:Port: 8000 2022-09-12T20:06:15.992 DEBUG:boto:Params: {} 2022-09-12T20:06:15.993 DEBUG:boto:Token: None 2022-09-12T20:06:15.993 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:15 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:15.994 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q5yRhaxVJU+VNFQCxX+IoNteao0= 2022-09-12T20:06:15.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q5yRhaxVJU+VNFQCxX+IoNteao0=', 'Content-Length': '0'} 2022-09-12T20:06:15.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4110cc5a67a7da9-00631f9137-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:15.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:06:15.998 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:15.999 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:15.999 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:16.000 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:16.000 DEBUG:boto:Method: GET 2022-09-12T20:06:16.001 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:16.001 DEBUG:boto:Data: 2022-09-12T20:06:16.002 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.002 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:16.003 DEBUG:boto:Port: 8001 2022-09-12T20:06:16.003 DEBUG:boto:Params: {} 2022-09-12T20:06:16.004 DEBUG:boto:Token: None 2022-09-12T20:06:16.005 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:16.005 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xgHBZ6CK7AlXlLJqkDJwJXnpDm0= 2022-09-12T20:06:16.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xgHBZ6CK7AlXlLJqkDJwJXnpDm0=', 'Content-Length': '0'} 2022-09-12T20:06:16.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064369fdb6b98f556-00631f9138-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.009 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:16.009 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:16.010 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:16.010 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:06:16.011 DEBUG:boto:Method: GET 2022-09-12T20:06:16.012 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:16.012 DEBUG:boto:Data: 2022-09-12T20:06:16.013 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.013 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:16.014 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.014 DEBUG:boto:Params: {} 2022-09-12T20:06:16.015 DEBUG:boto:Token: None 2022-09-12T20:06:16.015 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:06:16.016 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xgHBZ6CK7AlXlLJqkDJwJXnpDm0= 2022-09-12T20:06:16.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xgHBZ6CK7AlXlLJqkDJwJXnpDm0=', 'Content-Length': '0'} 2022-09-12T20:06:16.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c32bbb34cb73535b-00631f9138-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.019 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:16.020 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:16.020 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:06:16.021 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:06:16.021 DEBUG:boto:Method: GET 2022-09-12T20:06:16.022 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:06:16.023 DEBUG:boto:Data: 2022-09-12T20:06:16.023 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.024 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:16.024 DEBUG:boto:Port: 8001 2022-09-12T20:06:16.025 DEBUG:boto:Params: {} 2022-09-12T20:06:16.025 DEBUG:boto:Token: None 2022-09-12T20:06:16.026 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/ 2022-09-12T20:06:16.026 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/OOgyDWi73sN6nJJkwnGodYA8e0= 2022-09-12T20:06:16.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/OOgyDWi73sN6nJJkwnGodYA8e0=', 'Content-Length': '0'} 2022-09-12T20:06:16.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a96259644a3a065-00631f9138-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.031 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:16.031 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:16.032 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:16.033 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:06:16.033 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:06:16.034 DEBUG:boto:Method: GET 2022-09-12T20:06:16.034 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:06:16.035 DEBUG:boto:Data: 2022-09-12T20:06:16.035 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.036 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:16.036 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.037 DEBUG:boto:Params: {} 2022-09-12T20:06:16.037 DEBUG:boto:Token: None 2022-09-12T20:06:16.038 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/ 2022-09-12T20:06:16.039 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/OOgyDWi73sN6nJJkwnGodYA8e0= 2022-09-12T20:06:16.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/OOgyDWi73sN6nJJkwnGodYA8e0=', 'Content-Length': '0'} 2022-09-12T20:06:16.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002537d5b554800eca-00631f9138-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.086 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:16.087 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:16.088 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:16.088 DEBUG:boto:Method: HEAD 2022-09-12T20:06:16.089 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:06:16.089 DEBUG:boto:Data: 2022-09-12T20:06:16.090 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.090 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:16.091 DEBUG:boto:Port: 8001 2022-09-12T20:06:16.091 DEBUG:boto:Params: {} 2022-09-12T20:06:16.092 DEBUG:boto:Token: None 2022-09-12T20:06:16.093 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj55 2022-09-12T20:06:16.093 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PtRYoysZu4SUvuCtO95Ymy570F8= 2022-09-12T20:06:16.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PtRYoysZu4SUvuCtO95Ymy570F8=', 'Content-Length': '0'} 2022-09-12T20:06:16.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000655c8e33a9872599-00631f9138-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.097 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:16.097 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:16.098 DEBUG:boto:Method: HEAD 2022-09-12T20:06:16.098 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:06:16.099 DEBUG:boto:Data: 2022-09-12T20:06:16.099 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.100 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:16.100 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.101 DEBUG:boto:Params: {} 2022-09-12T20:06:16.101 DEBUG:boto:Token: None 2022-09-12T20:06:16.102 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj55 2022-09-12T20:06:16.103 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PtRYoysZu4SUvuCtO95Ymy570F8= 2022-09-12T20:06:16.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PtRYoysZu4SUvuCtO95Ymy570F8=', 'Content-Length': '0'} 2022-09-12T20:06:16.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086cc02d3dc6d6171-00631f9138-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:06:16.107 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:16.107 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:16.108 DEBUG:boto:Method: GET 2022-09-12T20:06:16.108 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:06:16.109 DEBUG:boto:Data: 2022-09-12T20:06:16.109 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.110 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:16.110 DEBUG:boto:Port: 8001 2022-09-12T20:06:16.111 DEBUG:boto:Params: {} 2022-09-12T20:06:16.112 DEBUG:boto:Token: None 2022-09-12T20:06:16.112 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj55 2022-09-12T20:06:16.113 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8sliwXRk4keUobwza4xqXQka2YM= 2022-09-12T20:06:16.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8sliwXRk4keUobwza4xqXQka2YM=', 'Content-Length': '0'} 2022-09-12T20:06:16.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c41403b16baffdc6-00631f9138-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.116 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:06:16.117 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:06:16.117 DEBUG:boto:Method: GET 2022-09-12T20:06:16.118 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:06:16.119 DEBUG:boto:Data: 2022-09-12T20:06:16.119 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.120 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:16.120 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.121 DEBUG:boto:Params: {} 2022-09-12T20:06:16.122 DEBUG:boto:Token: None 2022-09-12T20:06:16.122 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj55 2022-09-12T20:06:16.123 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8sliwXRk4keUobwza4xqXQka2YM= 2022-09-12T20:06:16.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8sliwXRk4keUobwza4xqXQka2YM=', 'Content-Length': '0'} 2022-09-12T20:06:16.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec970aabae229ce3-00631f9138-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.126 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:16.127 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:16.127 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:06:16.128 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:06:16.128 DEBUG:boto:Method: GET 2022-09-12T20:06:16.129 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:06:16.129 DEBUG:boto:Data: 2022-09-12T20:06:16.130 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.130 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:16.131 DEBUG:boto:Port: 8001 2022-09-12T20:06:16.132 DEBUG:boto:Params: {} 2022-09-12T20:06:16.132 DEBUG:boto:Token: None 2022-09-12T20:06:16.133 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/ 2022-09-12T20:06:16.133 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/OOgyDWi73sN6nJJkwnGodYA8e0= 2022-09-12T20:06:16.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/OOgyDWi73sN6nJJkwnGodYA8e0=', 'Content-Length': '0'} 2022-09-12T20:06:16.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3e9e1b6f01f4dc1-00631f9138-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.137 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:16.138 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:16.139 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:16.139 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:06:16.140 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:06:16.140 DEBUG:boto:Method: GET 2022-09-12T20:06:16.141 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:06:16.141 DEBUG:boto:Data: 2022-09-12T20:06:16.142 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.142 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:16.143 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.143 DEBUG:boto:Params: {} 2022-09-12T20:06:16.144 DEBUG:boto:Token: None 2022-09-12T20:06:16.144 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/ 2022-09-12T20:06:16.145 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/OOgyDWi73sN6nJJkwnGodYA8e0= 2022-09-12T20:06:16.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/OOgyDWi73sN6nJJkwnGodYA8e0=', 'Content-Length': '0'} 2022-09-12T20:06:16.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4719ee42d294fc5-00631f9138-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.149 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:16.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:06:16.150 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:16.151 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:16.151 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:16.152 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:16.152 DEBUG:boto:Method: GET 2022-09-12T20:06:16.153 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:16.153 DEBUG:boto:Data: 2022-09-12T20:06:16.154 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.154 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:16.155 DEBUG:boto:Port: 8001 2022-09-12T20:06:16.156 DEBUG:boto:Params: {} 2022-09-12T20:06:16.156 DEBUG:boto:Token: None 2022-09-12T20:06:16.157 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:16.157 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sl9tXqeryty2FRP5HCb4d/D+WBE= 2022-09-12T20:06:16.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sl9tXqeryty2FRP5HCb4d/D+WBE=', 'Content-Length': '0'} 2022-09-12T20:06:16.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008a37fce29a9a60e-00631f9138-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.161 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:16.161 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:16.162 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:16.162 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:16.163 DEBUG:boto:Method: GET 2022-09-12T20:06:16.164 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:16.164 DEBUG:boto:Data: 2022-09-12T20:06:16.165 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.165 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:16.166 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.166 DEBUG:boto:Params: {} 2022-09-12T20:06:16.167 DEBUG:boto:Token: None 2022-09-12T20:06:16.167 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:16.168 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sl9tXqeryty2FRP5HCb4d/D+WBE= 2022-09-12T20:06:16.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sl9tXqeryty2FRP5HCb4d/D+WBE=', 'Content-Length': '0'} 2022-09-12T20:06:16.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b0bd1abfdb64f2b-00631f9138-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.172 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:16.172 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:16.173 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:16.173 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:16.173 DEBUG:boto:Method: HEAD 2022-09-12T20:06:16.174 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:16.174 DEBUG:boto:Data: 2022-09-12T20:06:16.175 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.175 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:16.176 DEBUG:boto:Port: 8001 2022-09-12T20:06:16.177 DEBUG:boto:Params: {} 2022-09-12T20:06:16.177 DEBUG:boto:Token: None 2022-09-12T20:06:16.178 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:16.178 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G56CEa0y+hNahPGrwjxGUKmz5J0= 2022-09-12T20:06:16.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G56CEa0y+hNahPGrwjxGUKmz5J0=', 'Content-Length': '0'} 2022-09-12T20:06:16.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fb68f5e7624252d-00631f9138-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.182 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:16.182 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:16.183 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:16.183 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:16.184 DEBUG:boto:Method: HEAD 2022-09-12T20:06:16.184 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:16.185 DEBUG:boto:Data: 2022-09-12T20:06:16.185 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.186 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:16.186 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.187 DEBUG:boto:Params: {} 2022-09-12T20:06:16.187 DEBUG:boto:Token: None 2022-09-12T20:06:16.188 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:16.189 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G56CEa0y+hNahPGrwjxGUKmz5J0= 2022-09-12T20:06:16.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G56CEa0y+hNahPGrwjxGUKmz5J0=', 'Content-Length': '0'} 2022-09-12T20:06:16.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f4833d4bd65bec7-00631f9138-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:06:16.195 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:16.196 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:16.197 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:16.198 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:16.199 DEBUG:boto:Method: GET 2022-09-12T20:06:16.200 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:16.200 DEBUG:boto:Data: 2022-09-12T20:06:16.201 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.201 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:16.202 DEBUG:boto:Port: 8001 2022-09-12T20:06:16.202 DEBUG:boto:Params: {} 2022-09-12T20:06:16.203 DEBUG:boto:Token: None 2022-09-12T20:06:16.204 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:16.204 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sl9tXqeryty2FRP5HCb4d/D+WBE= 2022-09-12T20:06:16.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sl9tXqeryty2FRP5HCb4d/D+WBE=', 'Content-Length': '0'} 2022-09-12T20:06:16.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abe5bd39403f9afc-00631f9138-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.208 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:16.208 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:16.209 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:16.209 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:06:16.210 DEBUG:boto:Method: GET 2022-09-12T20:06:16.210 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:16.211 DEBUG:boto:Data: 2022-09-12T20:06:16.211 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.212 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:16.212 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.213 DEBUG:boto:Params: {} 2022-09-12T20:06:16.214 DEBUG:boto:Token: None 2022-09-12T20:06:16.214 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:06:16.215 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sl9tXqeryty2FRP5HCb4d/D+WBE= 2022-09-12T20:06:16.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sl9tXqeryty2FRP5HCb4d/D+WBE=', 'Content-Length': '0'} 2022-09-12T20:06:16.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007aa58a8129bc13fb-00631f9138-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.218 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:16.219 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:16.219 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:06:16.220 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:06:16.220 DEBUG:boto:Method: GET 2022-09-12T20:06:16.221 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:06:16.222 DEBUG:boto:Data: 2022-09-12T20:06:16.222 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.223 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:16.223 DEBUG:boto:Port: 8001 2022-09-12T20:06:16.224 DEBUG:boto:Params: {} 2022-09-12T20:06:16.224 DEBUG:boto:Token: None 2022-09-12T20:06:16.225 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/ 2022-09-12T20:06:16.226 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/OOgyDWi73sN6nJJkwnGodYA8e0= 2022-09-12T20:06:16.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/OOgyDWi73sN6nJJkwnGodYA8e0=', 'Content-Length': '0'} 2022-09-12T20:06:16.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007542ebe7a41167f0-00631f9138-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.270 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:16.271 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:16.272 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:16.272 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:06:16.273 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:06:16.273 DEBUG:boto:Method: GET 2022-09-12T20:06:16.274 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:06:16.274 DEBUG:boto:Data: 2022-09-12T20:06:16.275 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.275 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:16.276 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.276 DEBUG:boto:Params: {} 2022-09-12T20:06:16.277 DEBUG:boto:Token: None 2022-09-12T20:06:16.278 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/ 2022-09-12T20:06:16.278 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/OOgyDWi73sN6nJJkwnGodYA8e0= 2022-09-12T20:06:16.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/OOgyDWi73sN6nJJkwnGodYA8e0=', 'Content-Length': '0'} 2022-09-12T20:06:16.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a04e6aa27756021-00631f9138-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.282 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:16.283 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:16.284 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:16.284 DEBUG:boto:Method: HEAD 2022-09-12T20:06:16.285 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:06:16.285 DEBUG:boto:Data: 2022-09-12T20:06:16.286 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.286 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:16.287 DEBUG:boto:Port: 8001 2022-09-12T20:06:16.287 DEBUG:boto:Params: {} 2022-09-12T20:06:16.288 DEBUG:boto:Token: None 2022-09-12T20:06:16.288 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj68 2022-09-12T20:06:16.289 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TWmN1C5MO+39pp6HMQuHVzbjiUM= 2022-09-12T20:06:16.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TWmN1C5MO+39pp6HMQuHVzbjiUM=', 'Content-Length': '0'} 2022-09-12T20:06:16.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054b7add87b0f6793-00631f9138-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.292 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:16.293 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:16.293 DEBUG:boto:Method: HEAD 2022-09-12T20:06:16.294 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:06:16.294 DEBUG:boto:Data: 2022-09-12T20:06:16.295 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.295 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:16.296 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.296 DEBUG:boto:Params: {} 2022-09-12T20:06:16.297 DEBUG:boto:Token: None 2022-09-12T20:06:16.297 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj68 2022-09-12T20:06:16.298 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TWmN1C5MO+39pp6HMQuHVzbjiUM= 2022-09-12T20:06:16.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TWmN1C5MO+39pp6HMQuHVzbjiUM=', 'Content-Length': '0'} 2022-09-12T20:06:16.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005beb3cbff2173461-00631f9138-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:06:16.302 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:16.302 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:16.303 DEBUG:boto:Method: GET 2022-09-12T20:06:16.303 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:06:16.304 DEBUG:boto:Data: 2022-09-12T20:06:16.304 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.305 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:16.305 DEBUG:boto:Port: 8001 2022-09-12T20:06:16.306 DEBUG:boto:Params: {} 2022-09-12T20:06:16.306 DEBUG:boto:Token: None 2022-09-12T20:06:16.307 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj68 2022-09-12T20:06:16.308 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R2A4s+av/f7Lrnv+EAoSZDkbh1Y= 2022-09-12T20:06:16.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R2A4s+av/f7Lrnv+EAoSZDkbh1Y=', 'Content-Length': '0'} 2022-09-12T20:06:16.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ac104a6212dffba-00631f9138-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.311 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:06:16.312 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:06:16.312 DEBUG:boto:Method: GET 2022-09-12T20:06:16.313 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:06:16.313 DEBUG:boto:Data: 2022-09-12T20:06:16.314 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.314 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:16.315 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.315 DEBUG:boto:Params: {} 2022-09-12T20:06:16.316 DEBUG:boto:Token: None 2022-09-12T20:06:16.317 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj68 2022-09-12T20:06:16.317 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R2A4s+av/f7Lrnv+EAoSZDkbh1Y= 2022-09-12T20:06:16.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R2A4s+av/f7Lrnv+EAoSZDkbh1Y=', 'Content-Length': '0'} 2022-09-12T20:06:16.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d57bc2470a33e8fd-00631f9138-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.321 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:16.321 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:16.322 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:06:16.322 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:06:16.323 DEBUG:boto:Method: GET 2022-09-12T20:06:16.323 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:06:16.324 DEBUG:boto:Data: 2022-09-12T20:06:16.324 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.325 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:16.325 DEBUG:boto:Port: 8001 2022-09-12T20:06:16.326 DEBUG:boto:Params: {} 2022-09-12T20:06:16.326 DEBUG:boto:Token: None 2022-09-12T20:06:16.327 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/ 2022-09-12T20:06:16.327 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/OOgyDWi73sN6nJJkwnGodYA8e0= 2022-09-12T20:06:16.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/OOgyDWi73sN6nJJkwnGodYA8e0=', 'Content-Length': '0'} 2022-09-12T20:06:16.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfb725666317b917-00631f9138-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.332 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:16.333 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:16.333 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:16.334 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:06:16.334 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:06:16.335 DEBUG:boto:Method: GET 2022-09-12T20:06:16.335 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:06:16.336 DEBUG:boto:Data: 2022-09-12T20:06:16.336 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.337 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:16.338 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.338 DEBUG:boto:Params: {} 2022-09-12T20:06:16.339 DEBUG:boto:Token: None 2022-09-12T20:06:16.339 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/ 2022-09-12T20:06:16.340 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/OOgyDWi73sN6nJJkwnGodYA8e0= 2022-09-12T20:06:16.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/OOgyDWi73sN6nJJkwnGodYA8e0=', 'Content-Length': '0'} 2022-09-12T20:06:16.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c9ddf94af0d5614-00631f9138-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.344 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:16.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:06:16.346 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:16.346 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:16.347 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:16.347 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:16.348 DEBUG:boto:Method: GET 2022-09-12T20:06:16.348 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:16.349 DEBUG:boto:Data: 2022-09-12T20:06:16.349 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.350 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:16.351 DEBUG:boto:Port: 8001 2022-09-12T20:06:16.351 DEBUG:boto:Params: {} 2022-09-12T20:06:16.352 DEBUG:boto:Token: None 2022-09-12T20:06:16.352 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:16.353 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5w5xpfWcXPemeLrxkWHoYB8vmHs= 2022-09-12T20:06:16.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5w5xpfWcXPemeLrxkWHoYB8vmHs=', 'Content-Length': '0'} 2022-09-12T20:06:16.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b215ce2922cf56b5-00631f9138-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.398 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:16.399 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:16.399 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:16.400 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:16.401 DEBUG:boto:Method: GET 2022-09-12T20:06:16.401 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:16.402 DEBUG:boto:Data: 2022-09-12T20:06:16.402 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.403 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:16.403 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.404 DEBUG:boto:Params: {} 2022-09-12T20:06:16.405 DEBUG:boto:Token: None 2022-09-12T20:06:16.405 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:16.406 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5w5xpfWcXPemeLrxkWHoYB8vmHs= 2022-09-12T20:06:16.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5w5xpfWcXPemeLrxkWHoYB8vmHs=', 'Content-Length': '0'} 2022-09-12T20:06:16.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cd12402d9e357ab-00631f9138-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.410 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:16.410 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:16.411 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:16.411 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:16.412 DEBUG:boto:Method: HEAD 2022-09-12T20:06:16.412 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:16.413 DEBUG:boto:Data: 2022-09-12T20:06:16.413 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.414 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:16.414 DEBUG:boto:Port: 8001 2022-09-12T20:06:16.415 DEBUG:boto:Params: {} 2022-09-12T20:06:16.415 DEBUG:boto:Token: None 2022-09-12T20:06:16.416 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:16.416 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KuSNeWl5zr0IUOMC2dSzbHJGO8A= 2022-09-12T20:06:16.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KuSNeWl5zr0IUOMC2dSzbHJGO8A=', 'Content-Length': '0'} 2022-09-12T20:06:16.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfb13c9fbb448a21-00631f9138-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.420 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:16.420 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:16.421 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:16.421 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:16.422 DEBUG:boto:Method: HEAD 2022-09-12T20:06:16.422 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:16.423 DEBUG:boto:Data: 2022-09-12T20:06:16.423 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.424 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:16.424 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.425 DEBUG:boto:Params: {} 2022-09-12T20:06:16.425 DEBUG:boto:Token: None 2022-09-12T20:06:16.426 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:16.427 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KuSNeWl5zr0IUOMC2dSzbHJGO8A= 2022-09-12T20:06:16.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KuSNeWl5zr0IUOMC2dSzbHJGO8A=', 'Content-Length': '0'} 2022-09-12T20:06:16.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1f922b66ecb3697-00631f9138-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:06:16.430 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:16.431 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:16.431 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:16.432 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:16.432 DEBUG:boto:Method: GET 2022-09-12T20:06:16.433 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:16.433 DEBUG:boto:Data: 2022-09-12T20:06:16.434 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.434 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:16.435 DEBUG:boto:Port: 8001 2022-09-12T20:06:16.436 DEBUG:boto:Params: {} 2022-09-12T20:06:16.436 DEBUG:boto:Token: None 2022-09-12T20:06:16.437 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:16.437 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5w5xpfWcXPemeLrxkWHoYB8vmHs= 2022-09-12T20:06:16.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5w5xpfWcXPemeLrxkWHoYB8vmHs=', 'Content-Length': '0'} 2022-09-12T20:06:16.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba4a11e44093e5d3-00631f9138-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.442 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:16.443 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:16.444 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:16.445 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:06:16.445 DEBUG:boto:Method: GET 2022-09-12T20:06:16.446 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:16.447 DEBUG:boto:Data: 2022-09-12T20:06:16.447 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.448 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:16.448 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.449 DEBUG:boto:Params: {} 2022-09-12T20:06:16.449 DEBUG:boto:Token: None 2022-09-12T20:06:16.450 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:06:16.450 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5w5xpfWcXPemeLrxkWHoYB8vmHs= 2022-09-12T20:06:16.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5w5xpfWcXPemeLrxkWHoYB8vmHs=', 'Content-Length': '0'} 2022-09-12T20:06:16.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cde91a2c4dc9ba10-00631f9138-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.454 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:16.455 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:16.455 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:06:16.456 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:06:16.456 DEBUG:boto:Method: GET 2022-09-12T20:06:16.457 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:06:16.458 DEBUG:boto:Data: 2022-09-12T20:06:16.458 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.459 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:16.459 DEBUG:boto:Port: 8001 2022-09-12T20:06:16.460 DEBUG:boto:Params: {} 2022-09-12T20:06:16.460 DEBUG:boto:Token: None 2022-09-12T20:06:16.461 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/ 2022-09-12T20:06:16.461 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/OOgyDWi73sN6nJJkwnGodYA8e0= 2022-09-12T20:06:16.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/OOgyDWi73sN6nJJkwnGodYA8e0=', 'Content-Length': '0'} 2022-09-12T20:06:16.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000738aa8615a425ca-00631f9138-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.465 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:16.466 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:16.467 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:16.467 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:06:16.468 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:06:16.468 DEBUG:boto:Method: GET 2022-09-12T20:06:16.469 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:06:16.469 DEBUG:boto:Data: 2022-09-12T20:06:16.470 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.470 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:16.471 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.471 DEBUG:boto:Params: {} 2022-09-12T20:06:16.472 DEBUG:boto:Token: None 2022-09-12T20:06:16.473 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/ 2022-09-12T20:06:16.473 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/OOgyDWi73sN6nJJkwnGodYA8e0= 2022-09-12T20:06:16.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/OOgyDWi73sN6nJJkwnGodYA8e0=', 'Content-Length': '0'} 2022-09-12T20:06:16.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075ed15ef68c3a6ef-00631f9138-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.477 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:16.478 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:16.478 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:16.479 DEBUG:boto:Method: HEAD 2022-09-12T20:06:16.479 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:06:16.480 DEBUG:boto:Data: 2022-09-12T20:06:16.480 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.481 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:16.481 DEBUG:boto:Port: 8001 2022-09-12T20:06:16.482 DEBUG:boto:Params: {} 2022-09-12T20:06:16.482 DEBUG:boto:Token: None 2022-09-12T20:06:16.483 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj85 2022-09-12T20:06:16.483 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uSihf+o3CpaiT8Rngk5cKGbRT5g= 2022-09-12T20:06:16.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uSihf+o3CpaiT8Rngk5cKGbRT5g=', 'Content-Length': '0'} 2022-09-12T20:06:16.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079b2385f605eef2d-00631f9138-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.487 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:16.488 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:16.488 DEBUG:boto:Method: HEAD 2022-09-12T20:06:16.489 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:06:16.489 DEBUG:boto:Data: 2022-09-12T20:06:16.490 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.490 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:16.491 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.491 DEBUG:boto:Params: {} 2022-09-12T20:06:16.492 DEBUG:boto:Token: None 2022-09-12T20:06:16.492 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj85 2022-09-12T20:06:16.493 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uSihf+o3CpaiT8Rngk5cKGbRT5g= 2022-09-12T20:06:16.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uSihf+o3CpaiT8Rngk5cKGbRT5g=', 'Content-Length': '0'} 2022-09-12T20:06:16.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013232721052f87d1-00631f9138-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:06:16.497 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:16.497 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:16.498 DEBUG:boto:Method: GET 2022-09-12T20:06:16.498 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:06:16.499 DEBUG:boto:Data: 2022-09-12T20:06:16.499 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.500 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:16.500 DEBUG:boto:Port: 8001 2022-09-12T20:06:16.501 DEBUG:boto:Params: {} 2022-09-12T20:06:16.501 DEBUG:boto:Token: None 2022-09-12T20:06:16.502 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj85 2022-09-12T20:06:16.503 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pbBhSWd4Th5HP3lJqEPFcDD8FKA= 2022-09-12T20:06:16.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pbBhSWd4Th5HP3lJqEPFcDD8FKA=', 'Content-Length': '0'} 2022-09-12T20:06:16.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029e122f64ff8ab5c-00631f9138-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.506 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:06:16.507 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:06:16.507 DEBUG:boto:Method: GET 2022-09-12T20:06:16.508 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:06:16.508 DEBUG:boto:Data: 2022-09-12T20:06:16.509 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.509 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:16.510 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.511 DEBUG:boto:Params: {} 2022-09-12T20:06:16.511 DEBUG:boto:Token: None 2022-09-12T20:06:16.512 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj85 2022-09-12T20:06:16.512 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pbBhSWd4Th5HP3lJqEPFcDD8FKA= 2022-09-12T20:06:16.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pbBhSWd4Th5HP3lJqEPFcDD8FKA=', 'Content-Length': '0'} 2022-09-12T20:06:16.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e0dbb1936b788f1-00631f9138-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.516 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:16.516 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:16.517 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:06:16.518 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:06:16.518 DEBUG:boto:Method: GET 2022-09-12T20:06:16.519 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:06:16.519 DEBUG:boto:Data: 2022-09-12T20:06:16.520 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.520 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:06:16.521 DEBUG:boto:Port: 8001 2022-09-12T20:06:16.522 DEBUG:boto:Params: {} 2022-09-12T20:06:16.522 DEBUG:boto:Token: None 2022-09-12T20:06:16.523 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/ 2022-09-12T20:06:16.523 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/OOgyDWi73sN6nJJkwnGodYA8e0= 2022-09-12T20:06:16.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/OOgyDWi73sN6nJJkwnGodYA8e0=', 'Content-Length': '0'} 2022-09-12T20:06:16.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b87abd3b2252f01e-00631f9138-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.528 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:16.529 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:16.529 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:16.530 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:06:16.530 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:06:16.531 DEBUG:boto:Method: GET 2022-09-12T20:06:16.531 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:06:16.532 DEBUG:boto:Data: 2022-09-12T20:06:16.532 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.533 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:06:16.533 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.534 DEBUG:boto:Params: {} 2022-09-12T20:06:16.534 DEBUG:boto:Token: None 2022-09-12T20:06:16.535 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/ 2022-09-12T20:06:16.536 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/OOgyDWi73sN6nJJkwnGodYA8e0= 2022-09-12T20:06:16.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/OOgyDWi73sN6nJJkwnGodYA8e0=', 'Content-Length': '0'} 2022-09-12T20:06:16.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d70cc3363198be54-00631f9138-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.582 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:06:16.584 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-75 zones={test-zone3, test-zone2} 2022-09-12T20:06:16.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:06:16.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:06:16.587 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T20:06:16.651 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:06:16.652 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:06:16.709 INFO:teuthology.orchestra.run.smithi035.stdout:tenant: 2022-09-12T20:06:16.709 INFO:teuthology.orchestra.run.smithi035.stdout:bucket name: jdpiak-75 2022-09-12T20:06:16.717 INFO:teuthology.orchestra.run.smithi035.stdout:total entries: 10 2022-09-12T20:06:16.780 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:16.780 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:16.781 DEBUG:boto:Method: HEAD 2022-09-12T20:06:16.781 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:06:16.782 DEBUG:boto:Data: 2022-09-12T20:06:16.783 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.783 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:16.783 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.784 DEBUG:boto:Params: {} 2022-09-12T20:06:16.784 DEBUG:boto:Token: None 2022-09-12T20:06:16.785 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/ 2022-09-12T20:06:16.786 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yaAltX4yCNUdqwaEh1uiJqnzXoc= 2022-09-12T20:06:16.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yaAltX4yCNUdqwaEh1uiJqnzXoc=', 'Content-Length': '0'} 2022-09-12T20:06:16.790 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da1db1173e37a783-00631f9138-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.791 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:06:16.791 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:06:16.792 DEBUG:boto:Method: PUT 2022-09-12T20:06:16.792 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:06:16.793 DEBUG:boto:Data: 2022-09-12T20:06:16.794 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:06:16.794 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:16.795 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.796 DEBUG:boto:Params: {} 2022-09-12T20:06:16.797 DEBUG:boto:Token: None 2022-09-12T20:06:16.797 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj19 2022-09-12T20:06:16.798 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xJq5UmHv13YTxKAGQ9CXDowMLYs= 2022-09-12T20:06:16.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xJq5UmHv13YTxKAGQ9CXDowMLYs='} 2022-09-12T20:06:16.805 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c4364c2c80b0241d-00631f9138-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.806 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:16.806 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:16.807 DEBUG:boto:Method: HEAD 2022-09-12T20:06:16.807 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:06:16.808 DEBUG:boto:Data: 2022-09-12T20:06:16.808 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.809 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:16.809 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.810 DEBUG:boto:Params: {} 2022-09-12T20:06:16.811 DEBUG:boto:Token: None 2022-09-12T20:06:16.811 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/ 2022-09-12T20:06:16.812 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yaAltX4yCNUdqwaEh1uiJqnzXoc= 2022-09-12T20:06:16.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yaAltX4yCNUdqwaEh1uiJqnzXoc=', 'Content-Length': '0'} 2022-09-12T20:06:16.815 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa7991217c6ad43c-00631f9138-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.816 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:06:16.816 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:06:16.817 DEBUG:boto:Method: PUT 2022-09-12T20:06:16.817 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:06:16.818 DEBUG:boto:Data: 2022-09-12T20:06:16.819 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:06:16.819 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:16.820 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.820 DEBUG:boto:Params: {} 2022-09-12T20:06:16.821 DEBUG:boto:Token: None 2022-09-12T20:06:16.821 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj38 2022-09-12T20:06:16.822 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vLGpnNSHvgyTXAP1MSlYlmkuAqE= 2022-09-12T20:06:16.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vLGpnNSHvgyTXAP1MSlYlmkuAqE='} 2022-09-12T20:06:16.866 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e6f67e00e5db254-00631f9138-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.867 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:16.868 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:16.868 DEBUG:boto:Method: HEAD 2022-09-12T20:06:16.869 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:06:16.869 DEBUG:boto:Data: 2022-09-12T20:06:16.870 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.870 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:16.871 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.871 DEBUG:boto:Params: {} 2022-09-12T20:06:16.872 DEBUG:boto:Token: None 2022-09-12T20:06:16.872 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/ 2022-09-12T20:06:16.873 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yaAltX4yCNUdqwaEh1uiJqnzXoc= 2022-09-12T20:06:16.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yaAltX4yCNUdqwaEh1uiJqnzXoc=', 'Content-Length': '0'} 2022-09-12T20:06:16.876 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc5b1a7e10e8390e-00631f9138-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.877 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:06:16.878 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:06:16.878 DEBUG:boto:Method: PUT 2022-09-12T20:06:16.879 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:06:16.879 DEBUG:boto:Data: 2022-09-12T20:06:16.880 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:06:16.880 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:16.881 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.881 DEBUG:boto:Params: {} 2022-09-12T20:06:16.882 DEBUG:boto:Token: None 2022-09-12T20:06:16.882 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj57 2022-09-12T20:06:16.883 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:004MsyqNnrLdHJ7T11NFOH65puM= 2022-09-12T20:06:16.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:004MsyqNnrLdHJ7T11NFOH65puM='} 2022-09-12T20:06:16.889 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000065f2d224b9bb1cdd-00631f9138-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.890 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:16.891 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:16.891 DEBUG:boto:Method: HEAD 2022-09-12T20:06:16.892 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:06:16.892 DEBUG:boto:Data: 2022-09-12T20:06:16.893 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.893 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:16.893 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.894 DEBUG:boto:Params: {} 2022-09-12T20:06:16.895 DEBUG:boto:Token: None 2022-09-12T20:06:16.895 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/ 2022-09-12T20:06:16.896 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yaAltX4yCNUdqwaEh1uiJqnzXoc= 2022-09-12T20:06:16.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yaAltX4yCNUdqwaEh1uiJqnzXoc=', 'Content-Length': '0'} 2022-09-12T20:06:16.899 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c96b4f9822c32907-00631f9138-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.900 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:06:16.900 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:06:16.901 DEBUG:boto:Method: PUT 2022-09-12T20:06:16.901 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:06:16.902 DEBUG:boto:Data: 2022-09-12T20:06:16.902 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:06:16.903 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:16.903 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.904 DEBUG:boto:Params: {} 2022-09-12T20:06:16.904 DEBUG:boto:Token: None 2022-09-12T20:06:16.905 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj76 2022-09-12T20:06:16.905 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YaTYA12M2RLQRvliG4Ks+sXkwoQ= 2022-09-12T20:06:16.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YaTYA12M2RLQRvliG4Ks+sXkwoQ='} 2022-09-12T20:06:16.950 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000015be7452be9470f2-00631f9138-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.951 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:06:16.951 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:06:16.952 DEBUG:boto:Method: HEAD 2022-09-12T20:06:16.952 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:06:16.953 DEBUG:boto:Data: 2022-09-12T20:06:16.953 DEBUG:boto:Headers: {} 2022-09-12T20:06:16.954 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:16.954 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.955 DEBUG:boto:Params: {} 2022-09-12T20:06:16.956 DEBUG:boto:Token: None 2022-09-12T20:06:16.956 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/ 2022-09-12T20:06:16.957 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yaAltX4yCNUdqwaEh1uiJqnzXoc= 2022-09-12T20:06:16.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yaAltX4yCNUdqwaEh1uiJqnzXoc=', 'Content-Length': '0'} 2022-09-12T20:06:16.960 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b099f735d61e7ec-00631f9138-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.961 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:06:16.961 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:06:16.962 DEBUG:boto:Method: PUT 2022-09-12T20:06:16.962 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:06:16.963 DEBUG:boto:Data: 2022-09-12T20:06:16.963 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:06:16.964 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:06:16.964 DEBUG:boto:Port: 8000 2022-09-12T20:06:16.965 DEBUG:boto:Params: {} 2022-09-12T20:06:16.965 DEBUG:boto:Token: None 2022-09-12T20:06:16.966 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:06:16 GMT /jdpiak-75/obj95 2022-09-12T20:06:16.967 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hbVS0hGW/3PYzwlrTE9frRUxlb0= 2022-09-12T20:06:16.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:06:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hbVS0hGW/3PYzwlrTE9frRUxlb0='} 2022-09-12T20:06:16.976 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a055e35ecc2fc677-00631f9138-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:06:16.976 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-75 2022-09-12T20:06:16.977 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:06:16.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:06:16.978 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T20:06:17.042 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:06:17.042 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:06:17.098 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:06:17.096+0000 7ff75363f600 1 waiting to reach latest gen 2, on 1.. 2022-09-12T20:06:22.099 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:06:22.097+0000 7ff75363f600 1 waiting to reach latest gen 2, on 1.. 2022-09-12T20:06:27.101 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:06:27.099+0000 7ff75363f600 1 waiting for incremental sync to catch up: 2022-09-12T20:06:27.102 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2022-09-12T20:06:27.103 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:06:32.103 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:06:32.102+0000 7ff75363f600 1 waiting for incremental sync to catch up: 2022-09-12T20:06:32.104 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2022-09-12T20:06:32.104 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:06:37.105 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:06:37.103+0000 7ff75363f600 1 waiting for incremental sync to catch up: 2022-09-12T20:06:37.105 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2022-09-12T20:06:37.106 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:06:42.117 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:06:42.104+0000 7ff75363f600 1 waiting for incremental sync to catch up: 2022-09-12T20:06:42.117 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2022-09-12T20:06:42.118 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:06:47.108 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:06:47.106+0000 7ff75363f600 1 bucket sync caught up with source: 2022-09-12T20:06:47.109 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:06:47.110 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:06:47.110 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:06:47.106+0000 7ff75363f600 0 bucket checkpoint complete 2022-09-12T20:06:47.127 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-75 2022-09-12T20:06:47.128 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:06:47.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:06:47.129 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:06:47.195 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:06:47.196 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:06:47.239 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:06:47.237+0000 7f3f9d9ce600 1 waiting for incremental sync to catch up: 2022-09-12T20:06:47.239 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:06:47.240 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:06:52.241 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:06:52.239+0000 7f3f9d9ce600 1 waiting for incremental sync to catch up: 2022-09-12T20:06:52.242 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:06:52.242 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:06:57.243 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:06:57.241+0000 7f3f9d9ce600 1 waiting for incremental sync to catch up: 2022-09-12T20:06:57.243 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:06:57.244 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:07:02.245 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:07:02.243+0000 7f3f9d9ce600 1 waiting for incremental sync to catch up: 2022-09-12T20:07:02.245 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:07:02.245 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:07:07.246 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:07:07.245+0000 7f3f9d9ce600 1 waiting for incremental sync to catch up: 2022-09-12T20:07:07.247 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:07:07.247 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:07:12.248 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:07:12.246+0000 7f3f9d9ce600 1 waiting for incremental sync to catch up: 2022-09-12T20:07:12.249 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:07:12.249 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:07:17.250 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:07:17.249+0000 7f3f9d9ce600 1 waiting for incremental sync to catch up: 2022-09-12T20:07:17.251 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:07:17.251 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:07:22.252 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:07:22.251+0000 7f3f9d9ce600 1 waiting for incremental sync to catch up: 2022-09-12T20:07:22.253 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:07:22.253 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:07:27.254 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:07:27.252+0000 7f3f9d9ce600 1 waiting for incremental sync to catch up: 2022-09-12T20:07:27.255 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:07:27.255 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:07:32.256 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:07:32.254+0000 7f3f9d9ce600 1 waiting for incremental sync to catch up: 2022-09-12T20:07:32.256 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:07:32.257 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:07:37.258 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:07:37.256+0000 7f3f9d9ce600 1 waiting for incremental sync to catch up: 2022-09-12T20:07:37.258 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:07:37.259 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:07:42.260 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:07:42.258+0000 7f3f9d9ce600 1 waiting for incremental sync to catch up: 2022-09-12T20:07:42.260 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:07:42.260 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:07:47.262 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:07:47.260+0000 7f3f9d9ce600 1 waiting for incremental sync to catch up: 2022-09-12T20:07:47.263 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:07:47.263 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:07:52.264 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:07:52.262+0000 7f3f9d9ce600 1 waiting for incremental sync to catch up: 2022-09-12T20:07:52.264 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:07:52.264 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:07:57.266 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:07:57.264+0000 7f3f9d9ce600 1 waiting for incremental sync to catch up: 2022-09-12T20:07:57.266 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:07:57.266 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:08:02.268 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:08:02.266+0000 7f3f9d9ce600 1 waiting for incremental sync to catch up: 2022-09-12T20:08:02.269 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:08:02.269 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:08:07.270 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:08:07.269+0000 7f3f9d9ce600 1 waiting for incremental sync to catch up: 2022-09-12T20:08:07.270 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:08:07.271 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:08:12.272 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:08:12.271+0000 7f3f9d9ce600 1 waiting for incremental sync to catch up: 2022-09-12T20:08:12.272 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:08:12.273 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:08:17.274 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:08:17.272+0000 7f3f9d9ce600 1 waiting for incremental sync to catch up: 2022-09-12T20:08:17.274 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:08:17.274 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:08:22.276 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:08:22.274+0000 7f3f9d9ce600 1 waiting for incremental sync to catch up: 2022-09-12T20:08:22.276 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:08:22.277 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:08:27.278 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:08:27.276+0000 7f3f9d9ce600 1 bucket sync caught up with source: 2022-09-12T20:08:27.278 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:08:27.279 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:08:27.279 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:08:27.276+0000 7f3f9d9ce600 0 bucket checkpoint complete 2022-09-12T20:08:27.298 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-75 2022-09-12T20:08:27.299 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:08:27.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:08:27.300 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T20:08:27.365 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:08:27.365 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:08:27.405 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:08:27.403+0000 7fd6d126d600 1 bucket sync caught up with source: 2022-09-12T20:08:27.406 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:08:27.406 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , 00000000002.76.5, 00000000004.100.5, , , , , , , , , , , ] 2022-09-12T20:08:27.406 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:08:27.403+0000 7fd6d126d600 0 bucket checkpoint complete 2022-09-12T20:08:27.421 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-75 2022-09-12T20:08:27.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:08:27.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:08:27.423 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:08:27.489 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:08:27.489 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:08:27.541 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:08:27.539+0000 7f54b33b6600 1 bucket sync caught up with source: 2022-09-12T20:08:27.541 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.639.5, 00000000001.831.5, 00000000001.634.5, 00000000001.342.5, 00000000001.911.5, , , 00000000002.550.5, 00000000004.919.5, 00000000003.574.5] 2022-09-12T20:08:27.541 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.639.5, 00000000001.831.5, 00000000001.634.5, 00000000001.342.5, 00000000001.911.5, , , 00000000002.550.5, 00000000004.919.5, 00000000003.574.5] 2022-09-12T20:08:27.542 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:08:27.539+0000 7f54b33b6600 0 bucket checkpoint complete 2022-09-12T20:08:27.555 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-75 2022-09-12T20:08:27.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:08:27.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:08:27.557 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:08:27.615 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:08:27.615 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:08:27.656 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:08:27.654+0000 7f81054de600 1 bucket sync caught up with source: 2022-09-12T20:08:27.656 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.639.5, 00000000001.831.5, 00000000001.634.5, 00000000001.342.5, 00000000001.911.5, , , 00000000002.550.5, 00000000004.919.5, 00000000003.574.5] 2022-09-12T20:08:27.656 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.639.5, 00000000001.831.5, 00000000001.634.5, 00000000001.342.5, 00000000001.911.5, , , 00000000002.550.5, 00000000004.919.5, 00000000003.574.5] 2022-09-12T20:08:27.657 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:08:27.654+0000 7f81054de600 0 bucket checkpoint complete 2022-09-12T20:08:27.672 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-75 2022-09-12T20:08:27.673 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-75 2022-09-12T20:08:27.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:08:27.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:08:27.675 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:08:27.732 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:08:27.732 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:08:27.767 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:08:27.766+0000 7efc8bc4b600 1 bucket sync caught up with source: 2022-09-12T20:08:27.768 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.296.5, 00000000001.331.5, 00000000001.279.5, 00000000001.340.5, 00000000001.527.5, , , 00000000002.320.5, 00000000004.535.5, 00000000003.488.5] 2022-09-12T20:08:27.768 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.296.5, 00000000001.331.5, 00000000001.279.5, 00000000001.340.5, 00000000001.527.5, , , 00000000002.320.5, 00000000004.535.5, 00000000003.488.5] 2022-09-12T20:08:27.768 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:08:27.766+0000 7efc8bc4b600 0 bucket checkpoint complete 2022-09-12T20:08:27.782 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-75 2022-09-12T20:08:27.782 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:08:27.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:08:27.783 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T20:08:27.841 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:08:27.842 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:08:27.886 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:08:27.884+0000 7eff34fd2600 1 bucket sync caught up with source: 2022-09-12T20:08:27.886 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.296.5, 00000000001.331.5, 00000000001.279.5, 00000000001.340.5, 00000000001.527.5, , , 00000000002.320.5, 00000000004.535.5, 00000000003.488.5] 2022-09-12T20:08:27.886 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000002.296.5, 00000000001.331.5, 00000000001.279.5, 00000000001.340.5, 00000000001.527.5, , , 00000000002.320.5, 00000000004.535.5, 00000000003.488.5] 2022-09-12T20:08:27.887 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:08:27.884+0000 7eff34fd2600 0 bucket checkpoint complete 2022-09-12T20:08:27.907 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-75 2022-09-12T20:08:27.907 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-75 zones={test-zone2, test-zone1} 2022-09-12T20:08:27.908 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:27.909 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:27.910 DEBUG:boto:Method: HEAD 2022-09-12T20:08:27.910 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:08:27.911 DEBUG:boto:Data: 2022-09-12T20:08:27.911 DEBUG:boto:Headers: {} 2022-09-12T20:08:27.912 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:27.912 DEBUG:boto:Port: 8000 2022-09-12T20:08:27.913 DEBUG:boto:Params: {} 2022-09-12T20:08:27.914 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T20:08:27.915 DEBUG:boto:Token: None 2022-09-12T20:08:27.915 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:27 GMT /jdpiak-75/ 2022-09-12T20:08:27.916 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Vvu59kYdfpXOX3u+Z0S/ls6Kikw= 2022-09-12T20:08:27.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Vvu59kYdfpXOX3u+Z0S/ls6Kikw=', 'Content-Length': '0'} 2022-09-12T20:08:27.921 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f4c0124b0900e83-00631f91bb-189d-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:08:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:27.922 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:27.922 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:27.923 DEBUG:boto:Method: HEAD 2022-09-12T20:08:27.923 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:08:27.924 DEBUG:boto:Data: 2022-09-12T20:08:27.924 DEBUG:boto:Headers: {} 2022-09-12T20:08:27.925 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:27.925 DEBUG:boto:Port: 8000 2022-09-12T20:08:27.926 DEBUG:boto:Params: {} 2022-09-12T20:08:27.926 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T20:08:27.927 DEBUG:boto:Token: None 2022-09-12T20:08:27.928 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:27 GMT /jdpiak-75/ 2022-09-12T20:08:27.928 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Vvu59kYdfpXOX3u+Z0S/ls6Kikw= 2022-09-12T20:08:27.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Vvu59kYdfpXOX3u+Z0S/ls6Kikw=', 'Content-Length': '0'} 2022-09-12T20:08:27.933 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c3388160588f6ef-00631f91bb-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:08:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:27.934 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T20:08:27.934 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:27.935 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:27.936 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:27.936 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:27.937 DEBUG:boto:Method: GET 2022-09-12T20:08:27.937 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:27.938 DEBUG:boto:Data: 2022-09-12T20:08:27.938 DEBUG:boto:Headers: {} 2022-09-12T20:08:27.939 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:27.939 DEBUG:boto:Port: 8000 2022-09-12T20:08:27.940 DEBUG:boto:Params: {} 2022-09-12T20:08:27.941 DEBUG:boto:Token: None 2022-09-12T20:08:27.941 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:27 GMT /jdpiak-75/?versions 2022-09-12T20:08:27.942 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zhiTTEagwREz2dKvRO+QR1/Pl1k= 2022-09-12T20:08:27.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zhiTTEagwREz2dKvRO+QR1/Pl1k=', 'Content-Length': '0'} 2022-09-12T20:08:27.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006831076134f47827-00631f91bb-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:27.947 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:27.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:08:27.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:08:27.950 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:08:27.950 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:08:27.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:08:27.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:08:27.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:08:27.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:08:27.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:08:27.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:08:27.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:08:27.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:08:27.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:08:27.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:08:27.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:08:27.956 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T20:08:27.957 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:27.957 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:27.958 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:27.958 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:27.959 DEBUG:boto:Method: GET 2022-09-12T20:08:27.959 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:27.960 DEBUG:boto:Data: 2022-09-12T20:08:27.960 DEBUG:boto:Headers: {} 2022-09-12T20:08:27.961 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:27.961 DEBUG:boto:Port: 8000 2022-09-12T20:08:27.962 DEBUG:boto:Params: {} 2022-09-12T20:08:27.963 DEBUG:boto:Token: None 2022-09-12T20:08:27.963 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:27 GMT /jdpiak-75/?versions 2022-09-12T20:08:27.964 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zhiTTEagwREz2dKvRO+QR1/Pl1k= 2022-09-12T20:08:27.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zhiTTEagwREz2dKvRO+QR1/Pl1k=', 'Content-Length': '0'} 2022-09-12T20:08:27.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000441e40c92be6e191-00631f91bb-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:27.968 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:27.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:08:27.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:08:27.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:08:27.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:08:27.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:08:27.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:08:27.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:08:27.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:08:27.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:08:27.974 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:08:27.974 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:08:27.975 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:08:27.975 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:08:27.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:08:27.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:08:27.977 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:27.977 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:27.978 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:27.978 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:27.979 DEBUG:boto:Method: GET 2022-09-12T20:08:27.979 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:27.980 DEBUG:boto:Data: 2022-09-12T20:08:27.980 DEBUG:boto:Headers: {} 2022-09-12T20:08:27.981 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:27.981 DEBUG:boto:Port: 8000 2022-09-12T20:08:27.982 DEBUG:boto:Params: {} 2022-09-12T20:08:27.983 DEBUG:boto:Token: None 2022-09-12T20:08:27.983 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:27 GMT /jdpiak-75/?versions 2022-09-12T20:08:27.984 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zhiTTEagwREz2dKvRO+QR1/Pl1k= 2022-09-12T20:08:27.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zhiTTEagwREz2dKvRO+QR1/Pl1k=', 'Content-Length': '0'} 2022-09-12T20:08:27.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000480691532e44cb77-00631f91bb-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:27.987 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:27.989 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:27.989 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:27.990 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:27.990 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:27.991 DEBUG:boto:Method: GET 2022-09-12T20:08:27.991 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:27.992 DEBUG:boto:Data: 2022-09-12T20:08:27.992 DEBUG:boto:Headers: {} 2022-09-12T20:08:27.993 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:27.993 DEBUG:boto:Port: 8000 2022-09-12T20:08:27.994 DEBUG:boto:Params: {} 2022-09-12T20:08:27.995 DEBUG:boto:Token: None 2022-09-12T20:08:27.995 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:27 GMT /jdpiak-75/?versions 2022-09-12T20:08:27.996 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zhiTTEagwREz2dKvRO+QR1/Pl1k= 2022-09-12T20:08:27.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zhiTTEagwREz2dKvRO+QR1/Pl1k=', 'Content-Length': '0'} 2022-09-12T20:08:27.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e31625007959aeb6-00631f91bb-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.000 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:28.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:08:28.002 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:28.003 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:28.003 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:28.004 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:28.004 DEBUG:boto:Method: GET 2022-09-12T20:08:28.005 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:28.005 DEBUG:boto:Data: 2022-09-12T20:08:28.006 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.007 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.007 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.008 DEBUG:boto:Params: {} 2022-09-12T20:08:28.009 DEBUG:boto:Token: None 2022-09-12T20:08:28.009 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:28.010 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fg2DrBAP91oIop4SkB5Gk05U8Ns= 2022-09-12T20:08:28.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fg2DrBAP91oIop4SkB5Gk05U8Ns=', 'Content-Length': '0'} 2022-09-12T20:08:28.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca072fc2138773cb-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.014 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:28.014 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:28.015 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:28.016 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:28.016 DEBUG:boto:Method: GET 2022-09-12T20:08:28.017 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:28.017 DEBUG:boto:Data: 2022-09-12T20:08:28.018 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.019 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.019 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.020 DEBUG:boto:Params: {} 2022-09-12T20:08:28.021 DEBUG:boto:Token: None 2022-09-12T20:08:28.021 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:28.022 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fg2DrBAP91oIop4SkB5Gk05U8Ns= 2022-09-12T20:08:28.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fg2DrBAP91oIop4SkB5Gk05U8Ns=', 'Content-Length': '0'} 2022-09-12T20:08:28.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f408be81041d302-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.026 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:28.026 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:28.027 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:28.027 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:28.028 DEBUG:boto:Method: HEAD 2022-09-12T20:08:28.028 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:28.029 DEBUG:boto:Data: 2022-09-12T20:08:28.030 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.030 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.031 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.031 DEBUG:boto:Params: {} 2022-09-12T20:08:28.032 DEBUG:boto:Token: None 2022-09-12T20:08:28.032 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:28.033 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NSRNVIWHyXbDp06IspkKHvANFo0= 2022-09-12T20:08:28.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NSRNVIWHyXbDp06IspkKHvANFo0=', 'Content-Length': '0'} 2022-09-12T20:08:28.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000003bb968a1245398-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.037 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:28.037 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:28.038 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:28.038 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:28.039 DEBUG:boto:Method: HEAD 2022-09-12T20:08:28.039 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:28.040 DEBUG:boto:Data: 2022-09-12T20:08:28.041 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.041 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.042 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.042 DEBUG:boto:Params: {} 2022-09-12T20:08:28.043 DEBUG:boto:Token: None 2022-09-12T20:08:28.043 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:28.044 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NSRNVIWHyXbDp06IspkKHvANFo0= 2022-09-12T20:08:28.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NSRNVIWHyXbDp06IspkKHvANFo0=', 'Content-Length': '0'} 2022-09-12T20:08:28.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b07226356a92839-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:08:28.048 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:28.049 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:28.049 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:28.050 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:28.051 DEBUG:boto:Method: GET 2022-09-12T20:08:28.052 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:28.052 DEBUG:boto:Data: 2022-09-12T20:08:28.053 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.054 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.055 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.055 DEBUG:boto:Params: {} 2022-09-12T20:08:28.056 DEBUG:boto:Token: None 2022-09-12T20:08:28.057 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:28.058 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fg2DrBAP91oIop4SkB5Gk05U8Ns= 2022-09-12T20:08:28.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fg2DrBAP91oIop4SkB5Gk05U8Ns=', 'Content-Length': '0'} 2022-09-12T20:08:28.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7ed168f5d17725d-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.061 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:28.062 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:28.063 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:28.064 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:28.064 DEBUG:boto:Method: GET 2022-09-12T20:08:28.065 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:28.066 DEBUG:boto:Data: 2022-09-12T20:08:28.067 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.067 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.068 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.069 DEBUG:boto:Params: {} 2022-09-12T20:08:28.070 DEBUG:boto:Token: None 2022-09-12T20:08:28.070 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:28.071 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fg2DrBAP91oIop4SkB5Gk05U8Ns= 2022-09-12T20:08:28.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fg2DrBAP91oIop4SkB5Gk05U8Ns=', 'Content-Length': '0'} 2022-09-12T20:08:28.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079bb2d76f860ce81-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.118 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:28.119 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:28.119 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:08:28.120 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:08:28.121 DEBUG:boto:Method: GET 2022-09-12T20:08:28.121 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:08:28.122 DEBUG:boto:Data: 2022-09-12T20:08:28.123 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.124 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.125 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.126 DEBUG:boto:Params: {} 2022-09-12T20:08:28.127 DEBUG:boto:Token: None 2022-09-12T20:08:28.128 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/ 2022-09-12T20:08:28.129 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk= 2022-09-12T20:08:28.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk=', 'Content-Length': '0'} 2022-09-12T20:08:28.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd2f0c93c30d349e-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.134 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:28.135 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:28.135 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:28.136 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:08:28.136 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:08:28.137 DEBUG:boto:Method: GET 2022-09-12T20:08:28.138 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:08:28.138 DEBUG:boto:Data: 2022-09-12T20:08:28.139 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.140 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.141 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.141 DEBUG:boto:Params: {} 2022-09-12T20:08:28.142 DEBUG:boto:Token: None 2022-09-12T20:08:28.143 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/ 2022-09-12T20:08:28.144 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk= 2022-09-12T20:08:28.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk=', 'Content-Length': '0'} 2022-09-12T20:08:28.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cfcb7c8f31ac38c-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.149 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:28.150 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:28.150 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:28.151 DEBUG:boto:Method: HEAD 2022-09-12T20:08:28.152 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:08:28.152 DEBUG:boto:Data: 2022-09-12T20:08:28.153 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.154 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.154 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.155 DEBUG:boto:Params: {} 2022-09-12T20:08:28.156 DEBUG:boto:Token: None 2022-09-12T20:08:28.157 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj11 2022-09-12T20:08:28.157 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sqGRO74EVHRCaxm8+Zf+/FGa6Ak= 2022-09-12T20:08:28.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sqGRO74EVHRCaxm8+Zf+/FGa6Ak=', 'Content-Length': '0'} 2022-09-12T20:08:28.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2ead763641df891-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.162 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:28.163 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:28.164 DEBUG:boto:Method: HEAD 2022-09-12T20:08:28.164 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:08:28.165 DEBUG:boto:Data: 2022-09-12T20:08:28.166 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.167 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.168 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.169 DEBUG:boto:Params: {} 2022-09-12T20:08:28.170 DEBUG:boto:Token: None 2022-09-12T20:08:28.171 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj11 2022-09-12T20:08:28.172 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sqGRO74EVHRCaxm8+Zf+/FGa6Ak= 2022-09-12T20:08:28.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sqGRO74EVHRCaxm8+Zf+/FGa6Ak=', 'Content-Length': '0'} 2022-09-12T20:08:28.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f11b5b323196bbe6-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:08:28.176 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:28.177 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:28.178 DEBUG:boto:Method: GET 2022-09-12T20:08:28.178 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:08:28.179 DEBUG:boto:Data: 2022-09-12T20:08:28.180 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.180 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.181 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.182 DEBUG:boto:Params: {} 2022-09-12T20:08:28.182 DEBUG:boto:Token: None 2022-09-12T20:08:28.183 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj11 2022-09-12T20:08:28.184 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V2590G9H0PdlfabAPjhi/d3IRDE= 2022-09-12T20:08:28.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V2590G9H0PdlfabAPjhi/d3IRDE=', 'Content-Length': '0'} 2022-09-12T20:08:28.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002eb51f15af43afeb-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.187 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:28.188 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:28.188 DEBUG:boto:Method: GET 2022-09-12T20:08:28.189 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:08:28.190 DEBUG:boto:Data: 2022-09-12T20:08:28.190 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.191 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.191 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.192 DEBUG:boto:Params: {} 2022-09-12T20:08:28.192 DEBUG:boto:Token: None 2022-09-12T20:08:28.193 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj11 2022-09-12T20:08:28.194 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V2590G9H0PdlfabAPjhi/d3IRDE= 2022-09-12T20:08:28.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V2590G9H0PdlfabAPjhi/d3IRDE=', 'Content-Length': '0'} 2022-09-12T20:08:28.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000213807694600988-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.238 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:28.239 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:28.239 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:08:28.240 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:08:28.240 DEBUG:boto:Method: GET 2022-09-12T20:08:28.241 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:08:28.241 DEBUG:boto:Data: 2022-09-12T20:08:28.242 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.242 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.243 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.243 DEBUG:boto:Params: {} 2022-09-12T20:08:28.244 DEBUG:boto:Token: None 2022-09-12T20:08:28.245 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/ 2022-09-12T20:08:28.245 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk= 2022-09-12T20:08:28.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk=', 'Content-Length': '0'} 2022-09-12T20:08:28.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f47e1bf934d45b5-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.249 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:28.250 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:28.251 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:28.252 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:08:28.252 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:08:28.253 DEBUG:boto:Method: GET 2022-09-12T20:08:28.253 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:08:28.254 DEBUG:boto:Data: 2022-09-12T20:08:28.254 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.255 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.255 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.256 DEBUG:boto:Params: {} 2022-09-12T20:08:28.257 DEBUG:boto:Token: None 2022-09-12T20:08:28.257 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/ 2022-09-12T20:08:28.258 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk= 2022-09-12T20:08:28.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk=', 'Content-Length': '0'} 2022-09-12T20:08:28.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e0cc376d7b717ee-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.262 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:28.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:08:28.263 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:28.264 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:28.264 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:28.265 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:28.266 DEBUG:boto:Method: GET 2022-09-12T20:08:28.266 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:28.267 DEBUG:boto:Data: 2022-09-12T20:08:28.267 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.268 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.268 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.269 DEBUG:boto:Params: {} 2022-09-12T20:08:28.269 DEBUG:boto:Token: None 2022-09-12T20:08:28.270 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:28.271 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+4GpNeyscXgdzyMqo90RzPjd1+4= 2022-09-12T20:08:28.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+4GpNeyscXgdzyMqo90RzPjd1+4=', 'Content-Length': '0'} 2022-09-12T20:08:28.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005430442c08d795ab-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.274 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:28.275 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:28.275 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:28.276 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:28.276 DEBUG:boto:Method: GET 2022-09-12T20:08:28.277 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:28.277 DEBUG:boto:Data: 2022-09-12T20:08:28.278 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.279 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.279 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.280 DEBUG:boto:Params: {} 2022-09-12T20:08:28.280 DEBUG:boto:Token: None 2022-09-12T20:08:28.281 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:28.281 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+4GpNeyscXgdzyMqo90RzPjd1+4= 2022-09-12T20:08:28.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+4GpNeyscXgdzyMqo90RzPjd1+4=', 'Content-Length': '0'} 2022-09-12T20:08:28.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8ae55b69843f4e0-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.285 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:28.285 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:28.286 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:28.287 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:28.287 DEBUG:boto:Method: HEAD 2022-09-12T20:08:28.288 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:28.288 DEBUG:boto:Data: 2022-09-12T20:08:28.289 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.289 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.290 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.290 DEBUG:boto:Params: {} 2022-09-12T20:08:28.291 DEBUG:boto:Token: None 2022-09-12T20:08:28.291 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:28.292 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4VPCqpfFGbjr/C9CRU87XO2M5IE= 2022-09-12T20:08:28.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4VPCqpfFGbjr/C9CRU87XO2M5IE=', 'Content-Length': '0'} 2022-09-12T20:08:28.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd79d8355edf53a1-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.295 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:28.296 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:28.297 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:28.297 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:28.298 DEBUG:boto:Method: HEAD 2022-09-12T20:08:28.298 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:28.299 DEBUG:boto:Data: 2022-09-12T20:08:28.299 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.300 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.300 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.301 DEBUG:boto:Params: {} 2022-09-12T20:08:28.301 DEBUG:boto:Token: None 2022-09-12T20:08:28.302 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:28.303 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4VPCqpfFGbjr/C9CRU87XO2M5IE= 2022-09-12T20:08:28.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4VPCqpfFGbjr/C9CRU87XO2M5IE=', 'Content-Length': '0'} 2022-09-12T20:08:28.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007b02fc961623648-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:08:28.307 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:28.307 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:28.308 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:28.309 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:28.309 DEBUG:boto:Method: GET 2022-09-12T20:08:28.310 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:28.310 DEBUG:boto:Data: 2022-09-12T20:08:28.311 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.311 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.312 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.312 DEBUG:boto:Params: {} 2022-09-12T20:08:28.313 DEBUG:boto:Token: None 2022-09-12T20:08:28.314 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:28.314 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+4GpNeyscXgdzyMqo90RzPjd1+4= 2022-09-12T20:08:28.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+4GpNeyscXgdzyMqo90RzPjd1+4=', 'Content-Length': '0'} 2022-09-12T20:08:28.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f36c7b56e07cf7f9-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.358 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:28.359 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:28.359 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:28.360 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:28.360 DEBUG:boto:Method: GET 2022-09-12T20:08:28.361 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:28.362 DEBUG:boto:Data: 2022-09-12T20:08:28.362 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.363 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.363 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.364 DEBUG:boto:Params: {} 2022-09-12T20:08:28.364 DEBUG:boto:Token: None 2022-09-12T20:08:28.365 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:28.366 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+4GpNeyscXgdzyMqo90RzPjd1+4= 2022-09-12T20:08:28.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+4GpNeyscXgdzyMqo90RzPjd1+4=', 'Content-Length': '0'} 2022-09-12T20:08:28.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdf37ff0a2de70c2-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.369 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:28.370 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:28.371 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:08:28.371 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:08:28.372 DEBUG:boto:Method: GET 2022-09-12T20:08:28.372 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:08:28.373 DEBUG:boto:Data: 2022-09-12T20:08:28.373 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.374 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.374 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.375 DEBUG:boto:Params: {} 2022-09-12T20:08:28.376 DEBUG:boto:Token: None 2022-09-12T20:08:28.376 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/ 2022-09-12T20:08:28.377 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk= 2022-09-12T20:08:28.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk=', 'Content-Length': '0'} 2022-09-12T20:08:28.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af1cc7955875f286-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.381 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:28.381 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:28.382 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:28.383 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:08:28.383 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:08:28.384 DEBUG:boto:Method: GET 2022-09-12T20:08:28.384 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:08:28.385 DEBUG:boto:Data: 2022-09-12T20:08:28.385 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.386 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.386 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.387 DEBUG:boto:Params: {} 2022-09-12T20:08:28.388 DEBUG:boto:Token: None 2022-09-12T20:08:28.388 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/ 2022-09-12T20:08:28.389 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk= 2022-09-12T20:08:28.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk=', 'Content-Length': '0'} 2022-09-12T20:08:28.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000116a86768b99d544-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.393 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:28.394 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:28.395 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:28.395 DEBUG:boto:Method: HEAD 2022-09-12T20:08:28.396 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:08:28.396 DEBUG:boto:Data: 2022-09-12T20:08:28.397 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.397 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.398 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.399 DEBUG:boto:Params: {} 2022-09-12T20:08:28.399 DEBUG:boto:Token: None 2022-09-12T20:08:28.400 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj17 2022-09-12T20:08:28.400 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VvaOJBsl1sYRuygmUs26gXFDkSE= 2022-09-12T20:08:28.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VvaOJBsl1sYRuygmUs26gXFDkSE=', 'Content-Length': '0'} 2022-09-12T20:08:28.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001eee079f3eb533fb-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.404 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:28.404 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:28.405 DEBUG:boto:Method: HEAD 2022-09-12T20:08:28.405 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:08:28.406 DEBUG:boto:Data: 2022-09-12T20:08:28.406 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.407 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.407 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.408 DEBUG:boto:Params: {} 2022-09-12T20:08:28.408 DEBUG:boto:Token: None 2022-09-12T20:08:28.409 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj17 2022-09-12T20:08:28.410 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VvaOJBsl1sYRuygmUs26gXFDkSE= 2022-09-12T20:08:28.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VvaOJBsl1sYRuygmUs26gXFDkSE=', 'Content-Length': '0'} 2022-09-12T20:08:28.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2673794034a24ad-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:08:28.414 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:28.414 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:28.415 DEBUG:boto:Method: GET 2022-09-12T20:08:28.415 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:08:28.416 DEBUG:boto:Data: 2022-09-12T20:08:28.416 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.417 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.417 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.418 DEBUG:boto:Params: {} 2022-09-12T20:08:28.419 DEBUG:boto:Token: None 2022-09-12T20:08:28.419 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj17 2022-09-12T20:08:28.420 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4j0eTLv/JBmrD0C/4WeH3GRi22k= 2022-09-12T20:08:28.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4j0eTLv/JBmrD0C/4WeH3GRi22k=', 'Content-Length': '0'} 2022-09-12T20:08:28.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd668391b4f99443-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.423 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:28.424 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:28.425 DEBUG:boto:Method: GET 2022-09-12T20:08:28.426 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:08:28.427 DEBUG:boto:Data: 2022-09-12T20:08:28.427 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.428 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.428 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.429 DEBUG:boto:Params: {} 2022-09-12T20:08:28.429 DEBUG:boto:Token: None 2022-09-12T20:08:28.430 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj17 2022-09-12T20:08:28.431 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4j0eTLv/JBmrD0C/4WeH3GRi22k= 2022-09-12T20:08:28.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4j0eTLv/JBmrD0C/4WeH3GRi22k=', 'Content-Length': '0'} 2022-09-12T20:08:28.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000319b21c4391fc4d5-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.434 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:28.434 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:28.435 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:08:28.435 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:08:28.436 DEBUG:boto:Method: GET 2022-09-12T20:08:28.437 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:08:28.437 DEBUG:boto:Data: 2022-09-12T20:08:28.438 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.438 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.439 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.439 DEBUG:boto:Params: {} 2022-09-12T20:08:28.440 DEBUG:boto:Token: None 2022-09-12T20:08:28.440 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/ 2022-09-12T20:08:28.441 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk= 2022-09-12T20:08:28.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk=', 'Content-Length': '0'} 2022-09-12T20:08:28.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e8ebd64402e4214-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.445 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:28.446 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:28.446 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:28.447 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:08:28.447 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:08:28.448 DEBUG:boto:Method: GET 2022-09-12T20:08:28.449 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:08:28.449 DEBUG:boto:Data: 2022-09-12T20:08:28.450 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.450 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.451 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.451 DEBUG:boto:Params: {} 2022-09-12T20:08:28.452 DEBUG:boto:Token: None 2022-09-12T20:08:28.453 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/ 2022-09-12T20:08:28.453 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk= 2022-09-12T20:08:28.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk=', 'Content-Length': '0'} 2022-09-12T20:08:28.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000693a71fd533b1e4f-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.458 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:28.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:08:28.459 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:28.460 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:28.461 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:28.461 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:28.462 DEBUG:boto:Method: GET 2022-09-12T20:08:28.462 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:28.463 DEBUG:boto:Data: 2022-09-12T20:08:28.463 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.464 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.464 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.465 DEBUG:boto:Params: {} 2022-09-12T20:08:28.466 DEBUG:boto:Token: None 2022-09-12T20:08:28.466 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:28.467 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yMWMBL8QMTUR9KctTPpHFdkhQn8= 2022-09-12T20:08:28.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yMWMBL8QMTUR9KctTPpHFdkhQn8=', 'Content-Length': '0'} 2022-09-12T20:08:28.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a66ce131d2b96cf-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.470 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:28.471 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:28.471 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:28.472 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:28.472 DEBUG:boto:Method: GET 2022-09-12T20:08:28.473 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:28.474 DEBUG:boto:Data: 2022-09-12T20:08:28.474 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.475 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.475 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.476 DEBUG:boto:Params: {} 2022-09-12T20:08:28.476 DEBUG:boto:Token: None 2022-09-12T20:08:28.477 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:28.478 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yMWMBL8QMTUR9KctTPpHFdkhQn8= 2022-09-12T20:08:28.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yMWMBL8QMTUR9KctTPpHFdkhQn8=', 'Content-Length': '0'} 2022-09-12T20:08:28.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0d2302d8b6cee65-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.481 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:28.482 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:28.482 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:28.483 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:28.483 DEBUG:boto:Method: HEAD 2022-09-12T20:08:28.484 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:28.485 DEBUG:boto:Data: 2022-09-12T20:08:28.485 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.486 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.486 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.487 DEBUG:boto:Params: {} 2022-09-12T20:08:28.487 DEBUG:boto:Token: None 2022-09-12T20:08:28.488 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:28.488 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3yLCfoHmv97TsL38SgZjaN+68Xk= 2022-09-12T20:08:28.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3yLCfoHmv97TsL38SgZjaN+68Xk=', 'Content-Length': '0'} 2022-09-12T20:08:28.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf931d75c89d4b84-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.492 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:28.492 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:28.493 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:28.493 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:28.494 DEBUG:boto:Method: HEAD 2022-09-12T20:08:28.494 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:28.495 DEBUG:boto:Data: 2022-09-12T20:08:28.495 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.496 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.496 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.497 DEBUG:boto:Params: {} 2022-09-12T20:08:28.497 DEBUG:boto:Token: None 2022-09-12T20:08:28.498 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:28.499 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3yLCfoHmv97TsL38SgZjaN+68Xk= 2022-09-12T20:08:28.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3yLCfoHmv97TsL38SgZjaN+68Xk=', 'Content-Length': '0'} 2022-09-12T20:08:28.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9b6f84c2ff5653d-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:08:28.503 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:28.503 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:28.504 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:28.504 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:28.505 DEBUG:boto:Method: GET 2022-09-12T20:08:28.505 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:28.506 DEBUG:boto:Data: 2022-09-12T20:08:28.506 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.507 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.508 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.508 DEBUG:boto:Params: {} 2022-09-12T20:08:28.509 DEBUG:boto:Token: None 2022-09-12T20:08:28.509 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:28.510 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yMWMBL8QMTUR9KctTPpHFdkhQn8= 2022-09-12T20:08:28.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yMWMBL8QMTUR9KctTPpHFdkhQn8=', 'Content-Length': '0'} 2022-09-12T20:08:28.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089d05990f4116fd1-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.554 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:28.555 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:28.556 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:28.556 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:28.557 DEBUG:boto:Method: GET 2022-09-12T20:08:28.557 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:28.558 DEBUG:boto:Data: 2022-09-12T20:08:28.558 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.559 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.559 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.560 DEBUG:boto:Params: {} 2022-09-12T20:08:28.561 DEBUG:boto:Token: None 2022-09-12T20:08:28.561 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:28.562 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yMWMBL8QMTUR9KctTPpHFdkhQn8= 2022-09-12T20:08:28.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yMWMBL8QMTUR9KctTPpHFdkhQn8=', 'Content-Length': '0'} 2022-09-12T20:08:28.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072b9bb5d34927dcd-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.566 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:28.566 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:28.567 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:08:28.567 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:08:28.568 DEBUG:boto:Method: GET 2022-09-12T20:08:28.569 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:08:28.569 DEBUG:boto:Data: 2022-09-12T20:08:28.570 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.570 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.571 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.571 DEBUG:boto:Params: {} 2022-09-12T20:08:28.572 DEBUG:boto:Token: None 2022-09-12T20:08:28.572 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/ 2022-09-12T20:08:28.573 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk= 2022-09-12T20:08:28.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk=', 'Content-Length': '0'} 2022-09-12T20:08:28.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e3f8c70233e45d3-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.577 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:28.578 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:28.578 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:28.579 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:08:28.579 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:08:28.580 DEBUG:boto:Method: GET 2022-09-12T20:08:28.580 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:08:28.581 DEBUG:boto:Data: 2022-09-12T20:08:28.581 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.582 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.582 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.583 DEBUG:boto:Params: {} 2022-09-12T20:08:28.584 DEBUG:boto:Token: None 2022-09-12T20:08:28.584 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/ 2022-09-12T20:08:28.585 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk= 2022-09-12T20:08:28.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk=', 'Content-Length': '0'} 2022-09-12T20:08:28.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d82e67663c697db-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.589 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:28.590 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:28.590 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:28.591 DEBUG:boto:Method: HEAD 2022-09-12T20:08:28.591 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:08:28.592 DEBUG:boto:Data: 2022-09-12T20:08:28.592 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.593 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.593 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.594 DEBUG:boto:Params: {} 2022-09-12T20:08:28.594 DEBUG:boto:Token: None 2022-09-12T20:08:28.595 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj19 2022-09-12T20:08:28.595 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EaiPEu+e8S8va83sYRI46ZImU1E= 2022-09-12T20:08:28.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EaiPEu+e8S8va83sYRI46ZImU1E=', 'Content-Length': '0'} 2022-09-12T20:08:28.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040aa0299b07a95cb-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.599 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:28.599 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:28.600 DEBUG:boto:Method: HEAD 2022-09-12T20:08:28.601 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:08:28.601 DEBUG:boto:Data: 2022-09-12T20:08:28.602 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.602 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.603 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.603 DEBUG:boto:Params: {} 2022-09-12T20:08:28.604 DEBUG:boto:Token: None 2022-09-12T20:08:28.604 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj19 2022-09-12T20:08:28.605 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EaiPEu+e8S8va83sYRI46ZImU1E= 2022-09-12T20:08:28.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EaiPEu+e8S8va83sYRI46ZImU1E=', 'Content-Length': '0'} 2022-09-12T20:08:28.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000b5f29fcfc002fb-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:08:28.609 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:28.609 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:28.610 DEBUG:boto:Method: GET 2022-09-12T20:08:28.610 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:08:28.611 DEBUG:boto:Data: 2022-09-12T20:08:28.611 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.612 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.612 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.613 DEBUG:boto:Params: {} 2022-09-12T20:08:28.614 DEBUG:boto:Token: None 2022-09-12T20:08:28.614 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj19 2022-09-12T20:08:28.615 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NZY4DcVrJQguq/mvPCoLrSPZROs= 2022-09-12T20:08:28.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NZY4DcVrJQguq/mvPCoLrSPZROs=', 'Content-Length': '0'} 2022-09-12T20:08:28.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000746a54240fcc7e0f-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.619 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:28.619 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:28.620 DEBUG:boto:Method: GET 2022-09-12T20:08:28.620 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:08:28.621 DEBUG:boto:Data: 2022-09-12T20:08:28.621 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.622 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.623 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.623 DEBUG:boto:Params: {} 2022-09-12T20:08:28.624 DEBUG:boto:Token: None 2022-09-12T20:08:28.625 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj19 2022-09-12T20:08:28.625 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NZY4DcVrJQguq/mvPCoLrSPZROs= 2022-09-12T20:08:28.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NZY4DcVrJQguq/mvPCoLrSPZROs=', 'Content-Length': '0'} 2022-09-12T20:08:28.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bda3b72b9826d0e-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.629 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:28.629 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:28.630 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:08:28.631 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:08:28.631 DEBUG:boto:Method: GET 2022-09-12T20:08:28.632 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:08:28.632 DEBUG:boto:Data: 2022-09-12T20:08:28.633 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.633 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.634 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.634 DEBUG:boto:Params: {} 2022-09-12T20:08:28.635 DEBUG:boto:Token: None 2022-09-12T20:08:28.635 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/ 2022-09-12T20:08:28.636 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk= 2022-09-12T20:08:28.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk=', 'Content-Length': '0'} 2022-09-12T20:08:28.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c618ca89f2d28593-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.683 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:28.684 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:28.684 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:28.685 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:08:28.686 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:08:28.686 DEBUG:boto:Method: GET 2022-09-12T20:08:28.687 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:08:28.687 DEBUG:boto:Data: 2022-09-12T20:08:28.688 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.688 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.689 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.689 DEBUG:boto:Params: {} 2022-09-12T20:08:28.690 DEBUG:boto:Token: None 2022-09-12T20:08:28.691 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/ 2022-09-12T20:08:28.691 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk= 2022-09-12T20:08:28.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk=', 'Content-Length': '0'} 2022-09-12T20:08:28.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085e9fc3371b3964a-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.696 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:28.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:08:28.697 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:28.698 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:28.698 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:28.699 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:28.699 DEBUG:boto:Method: GET 2022-09-12T20:08:28.700 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:28.700 DEBUG:boto:Data: 2022-09-12T20:08:28.701 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.701 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.702 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.702 DEBUG:boto:Params: {} 2022-09-12T20:08:28.703 DEBUG:boto:Token: None 2022-09-12T20:08:28.704 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:28.704 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uhC7L+vKiUcmAMEUIHlNiVaH/4E= 2022-09-12T20:08:28.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uhC7L+vKiUcmAMEUIHlNiVaH/4E=', 'Content-Length': '0'} 2022-09-12T20:08:28.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3084fa7812d8857-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.708 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:28.708 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:28.709 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:28.709 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:28.710 DEBUG:boto:Method: GET 2022-09-12T20:08:28.710 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:28.711 DEBUG:boto:Data: 2022-09-12T20:08:28.712 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.712 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.713 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.713 DEBUG:boto:Params: {} 2022-09-12T20:08:28.714 DEBUG:boto:Token: None 2022-09-12T20:08:28.714 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:28.715 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uhC7L+vKiUcmAMEUIHlNiVaH/4E= 2022-09-12T20:08:28.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uhC7L+vKiUcmAMEUIHlNiVaH/4E=', 'Content-Length': '0'} 2022-09-12T20:08:28.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c7d98e8818d8b94-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.719 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:28.719 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:28.720 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:28.720 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:28.721 DEBUG:boto:Method: HEAD 2022-09-12T20:08:28.721 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:28.722 DEBUG:boto:Data: 2022-09-12T20:08:28.722 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.723 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.723 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.724 DEBUG:boto:Params: {} 2022-09-12T20:08:28.724 DEBUG:boto:Token: None 2022-09-12T20:08:28.725 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:28.725 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:C8O1c6PiGNd6TKhZa0Sj/vHhmd0= 2022-09-12T20:08:28.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:C8O1c6PiGNd6TKhZa0Sj/vHhmd0=', 'Content-Length': '0'} 2022-09-12T20:08:28.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d109f7b786a7e873-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.728 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:28.729 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:28.730 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:28.730 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:28.731 DEBUG:boto:Method: HEAD 2022-09-12T20:08:28.731 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:28.732 DEBUG:boto:Data: 2022-09-12T20:08:28.732 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.733 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.733 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.734 DEBUG:boto:Params: {} 2022-09-12T20:08:28.734 DEBUG:boto:Token: None 2022-09-12T20:08:28.735 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:28.735 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:C8O1c6PiGNd6TKhZa0Sj/vHhmd0= 2022-09-12T20:08:28.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:C8O1c6PiGNd6TKhZa0Sj/vHhmd0=', 'Content-Length': '0'} 2022-09-12T20:08:28.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5bcd86bab59a16b-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:08:28.739 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:28.740 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:28.740 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:28.741 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:28.741 DEBUG:boto:Method: GET 2022-09-12T20:08:28.742 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:28.743 DEBUG:boto:Data: 2022-09-12T20:08:28.743 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.744 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.744 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.745 DEBUG:boto:Params: {} 2022-09-12T20:08:28.745 DEBUG:boto:Token: None 2022-09-12T20:08:28.746 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:28.747 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uhC7L+vKiUcmAMEUIHlNiVaH/4E= 2022-09-12T20:08:28.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uhC7L+vKiUcmAMEUIHlNiVaH/4E=', 'Content-Length': '0'} 2022-09-12T20:08:28.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009da5039e028f52ea-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.751 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:28.751 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:28.752 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:28.752 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:28.753 DEBUG:boto:Method: GET 2022-09-12T20:08:28.753 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:28.754 DEBUG:boto:Data: 2022-09-12T20:08:28.754 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.755 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.755 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.756 DEBUG:boto:Params: {} 2022-09-12T20:08:28.757 DEBUG:boto:Token: None 2022-09-12T20:08:28.757 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:28.758 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uhC7L+vKiUcmAMEUIHlNiVaH/4E= 2022-09-12T20:08:28.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uhC7L+vKiUcmAMEUIHlNiVaH/4E=', 'Content-Length': '0'} 2022-09-12T20:08:28.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061a8f52d304e926c-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.802 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:28.803 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:28.803 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:08:28.804 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:08:28.804 DEBUG:boto:Method: GET 2022-09-12T20:08:28.805 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:08:28.806 DEBUG:boto:Data: 2022-09-12T20:08:28.806 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.807 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.807 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.808 DEBUG:boto:Params: {} 2022-09-12T20:08:28.809 DEBUG:boto:Token: None 2022-09-12T20:08:28.809 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/ 2022-09-12T20:08:28.810 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk= 2022-09-12T20:08:28.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk=', 'Content-Length': '0'} 2022-09-12T20:08:28.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a54ca2cbff7829b6-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.814 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:28.815 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:28.815 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:28.816 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:08:28.816 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:08:28.817 DEBUG:boto:Method: GET 2022-09-12T20:08:28.817 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:08:28.818 DEBUG:boto:Data: 2022-09-12T20:08:28.818 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.819 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.820 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.820 DEBUG:boto:Params: {} 2022-09-12T20:08:28.821 DEBUG:boto:Token: None 2022-09-12T20:08:28.821 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/ 2022-09-12T20:08:28.822 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk= 2022-09-12T20:08:28.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk=', 'Content-Length': '0'} 2022-09-12T20:08:28.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af69043ef18a3853-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.826 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:28.827 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:28.828 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:28.828 DEBUG:boto:Method: HEAD 2022-09-12T20:08:28.829 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:08:28.829 DEBUG:boto:Data: 2022-09-12T20:08:28.830 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.830 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.831 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.831 DEBUG:boto:Params: {} 2022-09-12T20:08:28.832 DEBUG:boto:Token: None 2022-09-12T20:08:28.832 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj22 2022-09-12T20:08:28.833 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1AR3nL2Jw4z8gFgIhgrs1VIbwdE= 2022-09-12T20:08:28.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1AR3nL2Jw4z8gFgIhgrs1VIbwdE=', 'Content-Length': '0'} 2022-09-12T20:08:28.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000923d9378c845ed86-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.836 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:28.836 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:28.837 DEBUG:boto:Method: HEAD 2022-09-12T20:08:28.837 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:08:28.838 DEBUG:boto:Data: 2022-09-12T20:08:28.838 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.839 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.839 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.840 DEBUG:boto:Params: {} 2022-09-12T20:08:28.840 DEBUG:boto:Token: None 2022-09-12T20:08:28.841 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj22 2022-09-12T20:08:28.842 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1AR3nL2Jw4z8gFgIhgrs1VIbwdE= 2022-09-12T20:08:28.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1AR3nL2Jw4z8gFgIhgrs1VIbwdE=', 'Content-Length': '0'} 2022-09-12T20:08:28.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000341544e1f066480c-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:08:28.845 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:28.846 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:28.846 DEBUG:boto:Method: GET 2022-09-12T20:08:28.847 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:08:28.847 DEBUG:boto:Data: 2022-09-12T20:08:28.848 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.848 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.849 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.849 DEBUG:boto:Params: {} 2022-09-12T20:08:28.850 DEBUG:boto:Token: None 2022-09-12T20:08:28.851 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj22 2022-09-12T20:08:28.851 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eBodG5gT3RFp5A2v4/sNh2JQ1KE= 2022-09-12T20:08:28.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eBodG5gT3RFp5A2v4/sNh2JQ1KE=', 'Content-Length': '0'} 2022-09-12T20:08:28.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f57243386b8295d-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.854 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:28.855 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:28.856 DEBUG:boto:Method: GET 2022-09-12T20:08:28.856 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:08:28.857 DEBUG:boto:Data: 2022-09-12T20:08:28.857 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.858 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.858 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.859 DEBUG:boto:Params: {} 2022-09-12T20:08:28.859 DEBUG:boto:Token: None 2022-09-12T20:08:28.860 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj22 2022-09-12T20:08:28.861 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eBodG5gT3RFp5A2v4/sNh2JQ1KE= 2022-09-12T20:08:28.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eBodG5gT3RFp5A2v4/sNh2JQ1KE=', 'Content-Length': '0'} 2022-09-12T20:08:28.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026c4256b93d97ea2-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.864 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:28.865 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:28.865 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:08:28.866 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:08:28.866 DEBUG:boto:Method: GET 2022-09-12T20:08:28.867 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:08:28.867 DEBUG:boto:Data: 2022-09-12T20:08:28.868 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.869 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.869 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.870 DEBUG:boto:Params: {} 2022-09-12T20:08:28.870 DEBUG:boto:Token: None 2022-09-12T20:08:28.871 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/ 2022-09-12T20:08:28.871 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk= 2022-09-12T20:08:28.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk=', 'Content-Length': '0'} 2022-09-12T20:08:28.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa4a2c92ae4295b5-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.875 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:28.876 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:28.877 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:28.877 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:08:28.878 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:08:28.878 DEBUG:boto:Method: GET 2022-09-12T20:08:28.879 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:08:28.879 DEBUG:boto:Data: 2022-09-12T20:08:28.880 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.880 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.881 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.881 DEBUG:boto:Params: {} 2022-09-12T20:08:28.882 DEBUG:boto:Token: None 2022-09-12T20:08:28.883 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/ 2022-09-12T20:08:28.883 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk= 2022-09-12T20:08:28.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5D/J8DSRHMpgPIL7pDq9nqWsMFk=', 'Content-Length': '0'} 2022-09-12T20:08:28.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee1097a14e673f83-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.888 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:28.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:08:28.889 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:28.890 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:28.890 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:28.891 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:28.891 DEBUG:boto:Method: GET 2022-09-12T20:08:28.892 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:28.892 DEBUG:boto:Data: 2022-09-12T20:08:28.893 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.893 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.894 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.894 DEBUG:boto:Params: {} 2022-09-12T20:08:28.895 DEBUG:boto:Token: None 2022-09-12T20:08:28.896 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:28.896 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AVRu6OQ8o6X+RUKYxxvKHhsHEAM= 2022-09-12T20:08:28.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AVRu6OQ8o6X+RUKYxxvKHhsHEAM=', 'Content-Length': '0'} 2022-09-12T20:08:28.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e1633a98edd5ad7-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.900 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:28.900 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:28.901 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:28.901 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:28.902 DEBUG:boto:Method: GET 2022-09-12T20:08:28.902 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:28.903 DEBUG:boto:Data: 2022-09-12T20:08:28.903 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.904 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.904 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.905 DEBUG:boto:Params: {} 2022-09-12T20:08:28.905 DEBUG:boto:Token: None 2022-09-12T20:08:28.906 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:28.907 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AVRu6OQ8o6X+RUKYxxvKHhsHEAM= 2022-09-12T20:08:28.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AVRu6OQ8o6X+RUKYxxvKHhsHEAM=', 'Content-Length': '0'} 2022-09-12T20:08:28.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071cda0271e65fd3f-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.950 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:28.951 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:28.951 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:28.951 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:28.952 DEBUG:boto:Method: HEAD 2022-09-12T20:08:28.952 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:28.953 DEBUG:boto:Data: 2022-09-12T20:08:28.953 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.954 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.954 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.955 DEBUG:boto:Params: {} 2022-09-12T20:08:28.955 DEBUG:boto:Token: None 2022-09-12T20:08:28.956 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:28.957 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qJy70mJ2pQcmjyI+gnaVC+h0lag= 2022-09-12T20:08:28.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qJy70mJ2pQcmjyI+gnaVC+h0lag=', 'Content-Length': '0'} 2022-09-12T20:08:28.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020f4accd3b277f5b-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.960 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:28.961 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:28.961 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:28.962 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:28.962 DEBUG:boto:Method: HEAD 2022-09-12T20:08:28.963 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:28.963 DEBUG:boto:Data: 2022-09-12T20:08:28.964 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.964 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.965 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.965 DEBUG:boto:Params: {} 2022-09-12T20:08:28.966 DEBUG:boto:Token: None 2022-09-12T20:08:28.966 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:28.967 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qJy70mJ2pQcmjyI+gnaVC+h0lag= 2022-09-12T20:08:28.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qJy70mJ2pQcmjyI+gnaVC+h0lag=', 'Content-Length': '0'} 2022-09-12T20:08:28.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b41dabd53f8f35b-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:08:28.971 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:28.971 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:28.972 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:28.972 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:28.973 DEBUG:boto:Method: GET 2022-09-12T20:08:28.973 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:28.974 DEBUG:boto:Data: 2022-09-12T20:08:28.975 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.975 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:28.976 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.976 DEBUG:boto:Params: {} 2022-09-12T20:08:28.977 DEBUG:boto:Token: None 2022-09-12T20:08:28.977 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:28.978 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AVRu6OQ8o6X+RUKYxxvKHhsHEAM= 2022-09-12T20:08:28.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AVRu6OQ8o6X+RUKYxxvKHhsHEAM=', 'Content-Length': '0'} 2022-09-12T20:08:28.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000444c2045b286313d-00631f91bc-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:28.982 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:28.982 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:28.983 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:28.983 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:28.984 DEBUG:boto:Method: GET 2022-09-12T20:08:28.985 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:28.985 DEBUG:boto:Data: 2022-09-12T20:08:28.992 DEBUG:boto:Headers: {} 2022-09-12T20:08:28.993 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:28.994 DEBUG:boto:Port: 8000 2022-09-12T20:08:28.995 DEBUG:boto:Params: {} 2022-09-12T20:08:28.995 DEBUG:boto:Token: None 2022-09-12T20:08:28.996 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:28 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:28.996 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AVRu6OQ8o6X+RUKYxxvKHhsHEAM= 2022-09-12T20:08:28.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:28 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AVRu6OQ8o6X+RUKYxxvKHhsHEAM=', 'Content-Length': '0'} 2022-09-12T20:08:28.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004eecc27277f22426-00631f91bc-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.000 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:29.001 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:29.001 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:08:29.002 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:08:29.003 DEBUG:boto:Method: GET 2022-09-12T20:08:29.003 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:08:29.004 DEBUG:boto:Data: 2022-09-12T20:08:29.004 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.005 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.005 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.006 DEBUG:boto:Params: {} 2022-09-12T20:08:29.007 DEBUG:boto:Token: None 2022-09-12T20:08:29.007 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/ 2022-09-12T20:08:29.008 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs= 2022-09-12T20:08:29.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs=', 'Content-Length': '0'} 2022-09-12T20:08:29.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee9aba0225ec9175-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.012 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:29.013 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:29.014 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:29.014 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:08:29.015 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:08:29.015 DEBUG:boto:Method: GET 2022-09-12T20:08:29.016 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:08:29.016 DEBUG:boto:Data: 2022-09-12T20:08:29.017 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.017 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.018 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.018 DEBUG:boto:Params: {} 2022-09-12T20:08:29.019 DEBUG:boto:Token: None 2022-09-12T20:08:29.020 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/ 2022-09-12T20:08:29.020 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs= 2022-09-12T20:08:29.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs=', 'Content-Length': '0'} 2022-09-12T20:08:29.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006275c89abbb4b965-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.025 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:29.025 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:29.026 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:29.026 DEBUG:boto:Method: HEAD 2022-09-12T20:08:29.027 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:08:29.028 DEBUG:boto:Data: 2022-09-12T20:08:29.028 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.029 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.029 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.030 DEBUG:boto:Params: {} 2022-09-12T20:08:29.030 DEBUG:boto:Token: None 2022-09-12T20:08:29.031 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj33 2022-09-12T20:08:29.031 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/1rQDk8Goj/BmYQ8qf0ZxtQoxdQ= 2022-09-12T20:08:29.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/1rQDk8Goj/BmYQ8qf0ZxtQoxdQ=', 'Content-Length': '0'} 2022-09-12T20:08:29.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d67ef092f5f623f-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.035 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:29.035 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:29.036 DEBUG:boto:Method: HEAD 2022-09-12T20:08:29.036 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:08:29.037 DEBUG:boto:Data: 2022-09-12T20:08:29.038 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.038 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.039 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.039 DEBUG:boto:Params: {} 2022-09-12T20:08:29.040 DEBUG:boto:Token: None 2022-09-12T20:08:29.040 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj33 2022-09-12T20:08:29.041 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/1rQDk8Goj/BmYQ8qf0ZxtQoxdQ= 2022-09-12T20:08:29.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/1rQDk8Goj/BmYQ8qf0ZxtQoxdQ=', 'Content-Length': '0'} 2022-09-12T20:08:29.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079879eb5076c4438-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:08:29.045 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:29.045 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:29.046 DEBUG:boto:Method: GET 2022-09-12T20:08:29.047 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:08:29.047 DEBUG:boto:Data: 2022-09-12T20:08:29.048 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.048 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.049 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.049 DEBUG:boto:Params: {} 2022-09-12T20:08:29.050 DEBUG:boto:Token: None 2022-09-12T20:08:29.050 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj33 2022-09-12T20:08:29.051 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UO7jNkbcdUs2Wi6v/n7nCBVNkGY= 2022-09-12T20:08:29.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UO7jNkbcdUs2Wi6v/n7nCBVNkGY=', 'Content-Length': '0'} 2022-09-12T20:08:29.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035e8290e621f0448-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.094 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:29.095 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:29.096 DEBUG:boto:Method: GET 2022-09-12T20:08:29.096 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:08:29.097 DEBUG:boto:Data: 2022-09-12T20:08:29.097 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.098 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.099 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.099 DEBUG:boto:Params: {} 2022-09-12T20:08:29.100 DEBUG:boto:Token: None 2022-09-12T20:08:29.100 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj33 2022-09-12T20:08:29.101 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UO7jNkbcdUs2Wi6v/n7nCBVNkGY= 2022-09-12T20:08:29.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UO7jNkbcdUs2Wi6v/n7nCBVNkGY=', 'Content-Length': '0'} 2022-09-12T20:08:29.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006c8427b6fe8b2ce-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.105 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:29.105 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:29.106 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:08:29.106 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:08:29.107 DEBUG:boto:Method: GET 2022-09-12T20:08:29.107 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:08:29.108 DEBUG:boto:Data: 2022-09-12T20:08:29.109 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.109 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.110 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.110 DEBUG:boto:Params: {} 2022-09-12T20:08:29.111 DEBUG:boto:Token: None 2022-09-12T20:08:29.111 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/ 2022-09-12T20:08:29.112 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs= 2022-09-12T20:08:29.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs=', 'Content-Length': '0'} 2022-09-12T20:08:29.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000087073dd7bcc20d5-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.116 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:29.117 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:29.117 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:29.118 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:08:29.119 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:08:29.119 DEBUG:boto:Method: GET 2022-09-12T20:08:29.120 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:08:29.120 DEBUG:boto:Data: 2022-09-12T20:08:29.121 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.121 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.122 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.122 DEBUG:boto:Params: {} 2022-09-12T20:08:29.123 DEBUG:boto:Token: None 2022-09-12T20:08:29.123 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/ 2022-09-12T20:08:29.124 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs= 2022-09-12T20:08:29.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs=', 'Content-Length': '0'} 2022-09-12T20:08:29.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080cf28fc3ddd1c82-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.129 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:29.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:08:29.130 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:29.131 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:29.131 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:29.132 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:29.132 DEBUG:boto:Method: GET 2022-09-12T20:08:29.133 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:29.134 DEBUG:boto:Data: 2022-09-12T20:08:29.134 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.135 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.135 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.136 DEBUG:boto:Params: {} 2022-09-12T20:08:29.136 DEBUG:boto:Token: None 2022-09-12T20:08:29.137 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:29.138 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZWaRDIPNuXLWGHTltu8snVqkpho= 2022-09-12T20:08:29.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZWaRDIPNuXLWGHTltu8snVqkpho=', 'Content-Length': '0'} 2022-09-12T20:08:29.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e67e0bbf209985e-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.141 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:29.142 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:29.142 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:29.143 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:29.144 DEBUG:boto:Method: GET 2022-09-12T20:08:29.144 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:29.145 DEBUG:boto:Data: 2022-09-12T20:08:29.145 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.146 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.146 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.147 DEBUG:boto:Params: {} 2022-09-12T20:08:29.147 DEBUG:boto:Token: None 2022-09-12T20:08:29.148 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:29.148 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZWaRDIPNuXLWGHTltu8snVqkpho= 2022-09-12T20:08:29.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZWaRDIPNuXLWGHTltu8snVqkpho=', 'Content-Length': '0'} 2022-09-12T20:08:29.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb85beab453dffad-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.194 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:29.195 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:29.196 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:29.197 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:29.198 DEBUG:boto:Method: HEAD 2022-09-12T20:08:29.199 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:29.200 DEBUG:boto:Data: 2022-09-12T20:08:29.201 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.202 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.203 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.204 DEBUG:boto:Params: {} 2022-09-12T20:08:29.206 DEBUG:boto:Token: None 2022-09-12T20:08:29.207 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:29.208 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F6v6agJpE/I2tnD1dQfbtxZFcJo= 2022-09-12T20:08:29.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F6v6agJpE/I2tnD1dQfbtxZFcJo=', 'Content-Length': '0'} 2022-09-12T20:08:29.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034ae35865fc005b6-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.213 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:29.215 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:29.216 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:29.218 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:29.219 DEBUG:boto:Method: HEAD 2022-09-12T20:08:29.220 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:29.221 DEBUG:boto:Data: 2022-09-12T20:08:29.222 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.223 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.224 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.225 DEBUG:boto:Params: {} 2022-09-12T20:08:29.226 DEBUG:boto:Token: None 2022-09-12T20:08:29.227 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:29.229 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F6v6agJpE/I2tnD1dQfbtxZFcJo= 2022-09-12T20:08:29.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F6v6agJpE/I2tnD1dQfbtxZFcJo=', 'Content-Length': '0'} 2022-09-12T20:08:29.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5d7bc78b5d1d1d0-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:08:29.240 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:29.241 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:29.243 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:29.244 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:29.245 DEBUG:boto:Method: GET 2022-09-12T20:08:29.246 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:29.247 DEBUG:boto:Data: 2022-09-12T20:08:29.248 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.249 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.250 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.251 DEBUG:boto:Params: {} 2022-09-12T20:08:29.252 DEBUG:boto:Token: None 2022-09-12T20:08:29.253 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:29.253 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZWaRDIPNuXLWGHTltu8snVqkpho= 2022-09-12T20:08:29.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZWaRDIPNuXLWGHTltu8snVqkpho=', 'Content-Length': '0'} 2022-09-12T20:08:29.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022915eef619a64d6-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.259 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:29.259 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:29.260 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:29.260 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:29.261 DEBUG:boto:Method: GET 2022-09-12T20:08:29.262 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:29.262 DEBUG:boto:Data: 2022-09-12T20:08:29.263 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.263 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.264 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.264 DEBUG:boto:Params: {} 2022-09-12T20:08:29.265 DEBUG:boto:Token: None 2022-09-12T20:08:29.265 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:29.266 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZWaRDIPNuXLWGHTltu8snVqkpho= 2022-09-12T20:08:29.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZWaRDIPNuXLWGHTltu8snVqkpho=', 'Content-Length': '0'} 2022-09-12T20:08:29.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b46eb310cdc6991-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.270 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:29.270 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:29.271 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:08:29.271 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:08:29.272 DEBUG:boto:Method: GET 2022-09-12T20:08:29.272 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:08:29.273 DEBUG:boto:Data: 2022-09-12T20:08:29.274 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.274 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.275 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.275 DEBUG:boto:Params: {} 2022-09-12T20:08:29.276 DEBUG:boto:Token: None 2022-09-12T20:08:29.276 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/ 2022-09-12T20:08:29.277 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs= 2022-09-12T20:08:29.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs=', 'Content-Length': '0'} 2022-09-12T20:08:29.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007769e5a63522585c-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.322 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:29.324 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:29.324 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:29.325 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:08:29.326 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:08:29.326 DEBUG:boto:Method: GET 2022-09-12T20:08:29.327 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:08:29.327 DEBUG:boto:Data: 2022-09-12T20:08:29.328 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.328 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.329 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.330 DEBUG:boto:Params: {} 2022-09-12T20:08:29.330 DEBUG:boto:Token: None 2022-09-12T20:08:29.331 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/ 2022-09-12T20:08:29.331 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs= 2022-09-12T20:08:29.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs=', 'Content-Length': '0'} 2022-09-12T20:08:29.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000953a5df580f62e13-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.336 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:29.337 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:29.337 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:29.338 DEBUG:boto:Method: HEAD 2022-09-12T20:08:29.338 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:08:29.339 DEBUG:boto:Data: 2022-09-12T20:08:29.339 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.340 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.340 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.341 DEBUG:boto:Params: {} 2022-09-12T20:08:29.341 DEBUG:boto:Token: None 2022-09-12T20:08:29.342 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj34 2022-09-12T20:08:29.343 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fIzpCQrBK+qC/3V9rNFPAuFVOoA= 2022-09-12T20:08:29.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fIzpCQrBK+qC/3V9rNFPAuFVOoA=', 'Content-Length': '0'} 2022-09-12T20:08:29.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006738d43aeb093c2d-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.348 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:29.349 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:29.349 DEBUG:boto:Method: HEAD 2022-09-12T20:08:29.350 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:08:29.351 DEBUG:boto:Data: 2022-09-12T20:08:29.351 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.352 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.352 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.353 DEBUG:boto:Params: {} 2022-09-12T20:08:29.353 DEBUG:boto:Token: None 2022-09-12T20:08:29.354 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj34 2022-09-12T20:08:29.354 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fIzpCQrBK+qC/3V9rNFPAuFVOoA= 2022-09-12T20:08:29.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fIzpCQrBK+qC/3V9rNFPAuFVOoA=', 'Content-Length': '0'} 2022-09-12T20:08:29.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004ba791149daf869-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:08:29.358 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:29.359 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:29.359 DEBUG:boto:Method: GET 2022-09-12T20:08:29.360 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:08:29.360 DEBUG:boto:Data: 2022-09-12T20:08:29.361 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.361 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.362 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.362 DEBUG:boto:Params: {} 2022-09-12T20:08:29.363 DEBUG:boto:Token: None 2022-09-12T20:08:29.364 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj34 2022-09-12T20:08:29.364 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f3WcRS1IlhjDVGC9L+IqKN7dzuY= 2022-09-12T20:08:29.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f3WcRS1IlhjDVGC9L+IqKN7dzuY=', 'Content-Length': '0'} 2022-09-12T20:08:29.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022a509d277bc352e-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.368 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:29.369 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:29.369 DEBUG:boto:Method: GET 2022-09-12T20:08:29.370 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:08:29.370 DEBUG:boto:Data: 2022-09-12T20:08:29.371 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.371 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.372 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.372 DEBUG:boto:Params: {} 2022-09-12T20:08:29.373 DEBUG:boto:Token: None 2022-09-12T20:08:29.373 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj34 2022-09-12T20:08:29.374 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f3WcRS1IlhjDVGC9L+IqKN7dzuY= 2022-09-12T20:08:29.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f3WcRS1IlhjDVGC9L+IqKN7dzuY=', 'Content-Length': '0'} 2022-09-12T20:08:29.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a70df94e709ff4bc-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.377 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:29.378 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:29.379 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:08:29.379 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:08:29.380 DEBUG:boto:Method: GET 2022-09-12T20:08:29.380 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:08:29.381 DEBUG:boto:Data: 2022-09-12T20:08:29.381 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.382 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.382 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.383 DEBUG:boto:Params: {} 2022-09-12T20:08:29.383 DEBUG:boto:Token: None 2022-09-12T20:08:29.384 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/ 2022-09-12T20:08:29.384 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs= 2022-09-12T20:08:29.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs=', 'Content-Length': '0'} 2022-09-12T20:08:29.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a47b2df8943d8c6-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.388 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:29.389 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:29.390 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:29.390 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:08:29.391 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:08:29.391 DEBUG:boto:Method: GET 2022-09-12T20:08:29.392 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:08:29.393 DEBUG:boto:Data: 2022-09-12T20:08:29.393 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.394 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.394 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.395 DEBUG:boto:Params: {} 2022-09-12T20:08:29.395 DEBUG:boto:Token: None 2022-09-12T20:08:29.396 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/ 2022-09-12T20:08:29.397 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs= 2022-09-12T20:08:29.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs=', 'Content-Length': '0'} 2022-09-12T20:08:29.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099fa2943856fe17e-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.402 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:29.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:08:29.403 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:29.404 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:29.404 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:29.405 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:29.405 DEBUG:boto:Method: GET 2022-09-12T20:08:29.406 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:29.406 DEBUG:boto:Data: 2022-09-12T20:08:29.407 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.407 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.408 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.409 DEBUG:boto:Params: {} 2022-09-12T20:08:29.409 DEBUG:boto:Token: None 2022-09-12T20:08:29.410 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:29.410 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UAORKORLx5BoQUvU/MYJ+TfaKw8= 2022-09-12T20:08:29.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UAORKORLx5BoQUvU/MYJ+TfaKw8=', 'Content-Length': '0'} 2022-09-12T20:08:29.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7af4d2cad2dd425-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.414 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:29.415 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:29.415 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:29.416 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:29.416 DEBUG:boto:Method: GET 2022-09-12T20:08:29.417 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:29.417 DEBUG:boto:Data: 2022-09-12T20:08:29.418 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.419 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.419 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.420 DEBUG:boto:Params: {} 2022-09-12T20:08:29.420 DEBUG:boto:Token: None 2022-09-12T20:08:29.421 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:29.421 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UAORKORLx5BoQUvU/MYJ+TfaKw8= 2022-09-12T20:08:29.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UAORKORLx5BoQUvU/MYJ+TfaKw8=', 'Content-Length': '0'} 2022-09-12T20:08:29.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe4e15617786e9ab-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.425 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:29.426 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:29.426 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:29.427 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:29.427 DEBUG:boto:Method: HEAD 2022-09-12T20:08:29.428 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:29.428 DEBUG:boto:Data: 2022-09-12T20:08:29.429 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.429 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.430 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.430 DEBUG:boto:Params: {} 2022-09-12T20:08:29.431 DEBUG:boto:Token: None 2022-09-12T20:08:29.432 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:29.432 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ReTSg6lxT366BVJRRDElJkscwEE= 2022-09-12T20:08:29.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ReTSg6lxT366BVJRRDElJkscwEE=', 'Content-Length': '0'} 2022-09-12T20:08:29.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee78f1a8d2a31c99-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.436 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:29.437 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:29.437 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:29.438 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:29.439 DEBUG:boto:Method: HEAD 2022-09-12T20:08:29.439 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:29.440 DEBUG:boto:Data: 2022-09-12T20:08:29.440 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.441 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.442 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.443 DEBUG:boto:Params: {} 2022-09-12T20:08:29.444 DEBUG:boto:Token: None 2022-09-12T20:08:29.445 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:29.446 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ReTSg6lxT366BVJRRDElJkscwEE= 2022-09-12T20:08:29.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ReTSg6lxT366BVJRRDElJkscwEE=', 'Content-Length': '0'} 2022-09-12T20:08:29.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adcf3c9649eb467a-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:08:29.451 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:29.452 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:29.453 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:29.453 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:29.454 DEBUG:boto:Method: GET 2022-09-12T20:08:29.455 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:29.455 DEBUG:boto:Data: 2022-09-12T20:08:29.456 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.456 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.457 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.457 DEBUG:boto:Params: {} 2022-09-12T20:08:29.458 DEBUG:boto:Token: None 2022-09-12T20:08:29.459 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:29.459 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UAORKORLx5BoQUvU/MYJ+TfaKw8= 2022-09-12T20:08:29.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UAORKORLx5BoQUvU/MYJ+TfaKw8=', 'Content-Length': '0'} 2022-09-12T20:08:29.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8c6937d06e36a72-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.463 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:29.463 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:29.464 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:29.464 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:29.465 DEBUG:boto:Method: GET 2022-09-12T20:08:29.466 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:29.466 DEBUG:boto:Data: 2022-09-12T20:08:29.467 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.467 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.468 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.468 DEBUG:boto:Params: {} 2022-09-12T20:08:29.469 DEBUG:boto:Token: None 2022-09-12T20:08:29.470 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:29.470 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UAORKORLx5BoQUvU/MYJ+TfaKw8= 2022-09-12T20:08:29.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UAORKORLx5BoQUvU/MYJ+TfaKw8=', 'Content-Length': '0'} 2022-09-12T20:08:29.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdf00dd63e7b9385-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.474 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:29.474 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:29.475 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:08:29.475 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:08:29.476 DEBUG:boto:Method: GET 2022-09-12T20:08:29.477 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:08:29.477 DEBUG:boto:Data: 2022-09-12T20:08:29.478 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.478 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.479 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.479 DEBUG:boto:Params: {} 2022-09-12T20:08:29.480 DEBUG:boto:Token: None 2022-09-12T20:08:29.480 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/ 2022-09-12T20:08:29.481 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs= 2022-09-12T20:08:29.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs=', 'Content-Length': '0'} 2022-09-12T20:08:29.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032d1bf4885841be1-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.526 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:29.527 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:29.527 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:29.528 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:08:29.528 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:08:29.529 DEBUG:boto:Method: GET 2022-09-12T20:08:29.530 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:08:29.530 DEBUG:boto:Data: 2022-09-12T20:08:29.531 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.531 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.532 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.532 DEBUG:boto:Params: {} 2022-09-12T20:08:29.533 DEBUG:boto:Token: None 2022-09-12T20:08:29.533 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/ 2022-09-12T20:08:29.534 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs= 2022-09-12T20:08:29.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs=', 'Content-Length': '0'} 2022-09-12T20:08:29.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e0c775870af2654-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.539 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:29.540 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:29.540 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:29.541 DEBUG:boto:Method: HEAD 2022-09-12T20:08:29.541 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:08:29.542 DEBUG:boto:Data: 2022-09-12T20:08:29.542 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.543 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.543 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.544 DEBUG:boto:Params: {} 2022-09-12T20:08:29.544 DEBUG:boto:Token: None 2022-09-12T20:08:29.545 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj38 2022-09-12T20:08:29.545 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:umfZ5z9/LCK8oibAUTlOhigQxzI= 2022-09-12T20:08:29.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:umfZ5z9/LCK8oibAUTlOhigQxzI=', 'Content-Length': '0'} 2022-09-12T20:08:29.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb8fe12457cf8fa1-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.549 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:29.550 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:29.550 DEBUG:boto:Method: HEAD 2022-09-12T20:08:29.551 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:08:29.551 DEBUG:boto:Data: 2022-09-12T20:08:29.552 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.552 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.553 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.553 DEBUG:boto:Params: {} 2022-09-12T20:08:29.554 DEBUG:boto:Token: None 2022-09-12T20:08:29.554 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj38 2022-09-12T20:08:29.555 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:umfZ5z9/LCK8oibAUTlOhigQxzI= 2022-09-12T20:08:29.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:umfZ5z9/LCK8oibAUTlOhigQxzI=', 'Content-Length': '0'} 2022-09-12T20:08:29.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000428c1b546e7d2d22-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:08:29.559 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:29.559 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:29.560 DEBUG:boto:Method: GET 2022-09-12T20:08:29.561 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:08:29.561 DEBUG:boto:Data: 2022-09-12T20:08:29.562 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.562 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.563 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.563 DEBUG:boto:Params: {} 2022-09-12T20:08:29.564 DEBUG:boto:Token: None 2022-09-12T20:08:29.564 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj38 2022-09-12T20:08:29.565 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SuEjq2Bq7H+h0iL5s9kReo5xEIg= 2022-09-12T20:08:29.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SuEjq2Bq7H+h0iL5s9kReo5xEIg=', 'Content-Length': '0'} 2022-09-12T20:08:29.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b16594ae42750918-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.568 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:29.569 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:29.570 DEBUG:boto:Method: GET 2022-09-12T20:08:29.570 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:08:29.571 DEBUG:boto:Data: 2022-09-12T20:08:29.571 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.572 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.572 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.573 DEBUG:boto:Params: {} 2022-09-12T20:08:29.573 DEBUG:boto:Token: None 2022-09-12T20:08:29.574 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj38 2022-09-12T20:08:29.575 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SuEjq2Bq7H+h0iL5s9kReo5xEIg= 2022-09-12T20:08:29.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SuEjq2Bq7H+h0iL5s9kReo5xEIg=', 'Content-Length': '0'} 2022-09-12T20:08:29.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3a6c0bda2d76454-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.618 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:29.619 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:29.619 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:08:29.620 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:08:29.620 DEBUG:boto:Method: GET 2022-09-12T20:08:29.621 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:08:29.621 DEBUG:boto:Data: 2022-09-12T20:08:29.622 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.622 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.623 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.623 DEBUG:boto:Params: {} 2022-09-12T20:08:29.624 DEBUG:boto:Token: None 2022-09-12T20:08:29.624 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/ 2022-09-12T20:08:29.625 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs= 2022-09-12T20:08:29.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs=', 'Content-Length': '0'} 2022-09-12T20:08:29.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe0c8812d944e537-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.629 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:29.630 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:29.630 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:29.631 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:08:29.631 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:08:29.632 DEBUG:boto:Method: GET 2022-09-12T20:08:29.632 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:08:29.633 DEBUG:boto:Data: 2022-09-12T20:08:29.634 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.634 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.635 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.635 DEBUG:boto:Params: {} 2022-09-12T20:08:29.636 DEBUG:boto:Token: None 2022-09-12T20:08:29.636 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/ 2022-09-12T20:08:29.637 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs= 2022-09-12T20:08:29.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs=', 'Content-Length': '0'} 2022-09-12T20:08:29.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e00cbe0f48f45549-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.641 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:29.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:08:29.643 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:29.643 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:29.644 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:29.644 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:29.645 DEBUG:boto:Method: GET 2022-09-12T20:08:29.645 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:29.646 DEBUG:boto:Data: 2022-09-12T20:08:29.646 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.647 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.647 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.648 DEBUG:boto:Params: {} 2022-09-12T20:08:29.648 DEBUG:boto:Token: None 2022-09-12T20:08:29.649 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:29.650 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lxcZoenoDs9i3Ow5zVsyLXEjCgc= 2022-09-12T20:08:29.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lxcZoenoDs9i3Ow5zVsyLXEjCgc=', 'Content-Length': '0'} 2022-09-12T20:08:29.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d00e6787832f9ca-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.653 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:29.654 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:29.654 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:29.655 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:29.655 DEBUG:boto:Method: GET 2022-09-12T20:08:29.656 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:29.656 DEBUG:boto:Data: 2022-09-12T20:08:29.657 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.657 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.658 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.658 DEBUG:boto:Params: {} 2022-09-12T20:08:29.659 DEBUG:boto:Token: None 2022-09-12T20:08:29.659 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:29.660 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lxcZoenoDs9i3Ow5zVsyLXEjCgc= 2022-09-12T20:08:29.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lxcZoenoDs9i3Ow5zVsyLXEjCgc=', 'Content-Length': '0'} 2022-09-12T20:08:29.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fea0bf3ddc7afb1-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.663 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:29.664 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:29.664 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:29.665 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:29.665 DEBUG:boto:Method: HEAD 2022-09-12T20:08:29.666 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:29.666 DEBUG:boto:Data: 2022-09-12T20:08:29.667 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.667 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.668 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.668 DEBUG:boto:Params: {} 2022-09-12T20:08:29.669 DEBUG:boto:Token: None 2022-09-12T20:08:29.670 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:29.670 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4qDWdslq4d18GvM0EKN7/UoEt9M= 2022-09-12T20:08:29.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4qDWdslq4d18GvM0EKN7/UoEt9M=', 'Content-Length': '0'} 2022-09-12T20:08:29.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073a2723b8d6b368d-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.673 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:29.674 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:29.674 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:29.675 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:29.675 DEBUG:boto:Method: HEAD 2022-09-12T20:08:29.676 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:29.676 DEBUG:boto:Data: 2022-09-12T20:08:29.677 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.677 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.678 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.678 DEBUG:boto:Params: {} 2022-09-12T20:08:29.679 DEBUG:boto:Token: None 2022-09-12T20:08:29.679 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:29.680 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4qDWdslq4d18GvM0EKN7/UoEt9M= 2022-09-12T20:08:29.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4qDWdslq4d18GvM0EKN7/UoEt9M=', 'Content-Length': '0'} 2022-09-12T20:08:29.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000638ceeb2e4e40573-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:08:29.684 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:29.684 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:29.685 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:29.685 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:29.686 DEBUG:boto:Method: GET 2022-09-12T20:08:29.687 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:29.687 DEBUG:boto:Data: 2022-09-12T20:08:29.688 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.688 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.689 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.689 DEBUG:boto:Params: {} 2022-09-12T20:08:29.690 DEBUG:boto:Token: None 2022-09-12T20:08:29.690 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:29.691 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lxcZoenoDs9i3Ow5zVsyLXEjCgc= 2022-09-12T20:08:29.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lxcZoenoDs9i3Ow5zVsyLXEjCgc=', 'Content-Length': '0'} 2022-09-12T20:08:29.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2ae647443d31996-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.694 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:29.695 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:29.695 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:29.696 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:29.697 DEBUG:boto:Method: GET 2022-09-12T20:08:29.697 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:29.698 DEBUG:boto:Data: 2022-09-12T20:08:29.698 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.699 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.699 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.700 DEBUG:boto:Params: {} 2022-09-12T20:08:29.700 DEBUG:boto:Token: None 2022-09-12T20:08:29.701 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:29.701 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lxcZoenoDs9i3Ow5zVsyLXEjCgc= 2022-09-12T20:08:29.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lxcZoenoDs9i3Ow5zVsyLXEjCgc=', 'Content-Length': '0'} 2022-09-12T20:08:29.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000329e781cb946469a-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.746 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:29.747 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:29.748 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:08:29.748 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:08:29.749 DEBUG:boto:Method: GET 2022-09-12T20:08:29.750 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:08:29.750 DEBUG:boto:Data: 2022-09-12T20:08:29.751 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.752 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.752 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.753 DEBUG:boto:Params: {} 2022-09-12T20:08:29.753 DEBUG:boto:Token: None 2022-09-12T20:08:29.754 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/ 2022-09-12T20:08:29.755 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs= 2022-09-12T20:08:29.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs=', 'Content-Length': '0'} 2022-09-12T20:08:29.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000660c80f1659a4994-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.759 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:29.760 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:29.760 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:29.761 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:08:29.761 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:08:29.762 DEBUG:boto:Method: GET 2022-09-12T20:08:29.762 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:08:29.763 DEBUG:boto:Data: 2022-09-12T20:08:29.763 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.764 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.764 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.765 DEBUG:boto:Params: {} 2022-09-12T20:08:29.766 DEBUG:boto:Token: None 2022-09-12T20:08:29.766 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/ 2022-09-12T20:08:29.767 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs= 2022-09-12T20:08:29.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs=', 'Content-Length': '0'} 2022-09-12T20:08:29.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000688d2841f4691c90-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.771 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:29.772 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:29.773 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:29.773 DEBUG:boto:Method: HEAD 2022-09-12T20:08:29.774 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:08:29.775 DEBUG:boto:Data: 2022-09-12T20:08:29.775 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.776 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.776 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.777 DEBUG:boto:Params: {} 2022-09-12T20:08:29.777 DEBUG:boto:Token: None 2022-09-12T20:08:29.778 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj44 2022-09-12T20:08:29.778 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KYUqyEa7T97HelPgs4JE9JL8n58= 2022-09-12T20:08:29.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KYUqyEa7T97HelPgs4JE9JL8n58=', 'Content-Length': '0'} 2022-09-12T20:08:29.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031b701c184cd3cb9-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.782 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:29.782 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:29.783 DEBUG:boto:Method: HEAD 2022-09-12T20:08:29.783 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:08:29.784 DEBUG:boto:Data: 2022-09-12T20:08:29.784 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.785 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.785 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.786 DEBUG:boto:Params: {} 2022-09-12T20:08:29.786 DEBUG:boto:Token: None 2022-09-12T20:08:29.787 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj44 2022-09-12T20:08:29.787 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KYUqyEa7T97HelPgs4JE9JL8n58= 2022-09-12T20:08:29.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KYUqyEa7T97HelPgs4JE9JL8n58=', 'Content-Length': '0'} 2022-09-12T20:08:29.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e67851afb09829c4-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:08:29.791 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:29.792 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:29.792 DEBUG:boto:Method: GET 2022-09-12T20:08:29.793 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:08:29.793 DEBUG:boto:Data: 2022-09-12T20:08:29.794 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.794 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.795 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.795 DEBUG:boto:Params: {} 2022-09-12T20:08:29.796 DEBUG:boto:Token: None 2022-09-12T20:08:29.797 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj44 2022-09-12T20:08:29.797 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/qQGO0JjzaHwpMuQMRSNdJ2fZFM= 2022-09-12T20:08:29.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/qQGO0JjzaHwpMuQMRSNdJ2fZFM=', 'Content-Length': '0'} 2022-09-12T20:08:29.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e82c378fbd535fc-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.801 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:29.801 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:29.802 DEBUG:boto:Method: GET 2022-09-12T20:08:29.803 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:08:29.803 DEBUG:boto:Data: 2022-09-12T20:08:29.804 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.804 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.805 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.805 DEBUG:boto:Params: {} 2022-09-12T20:08:29.806 DEBUG:boto:Token: None 2022-09-12T20:08:29.806 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj44 2022-09-12T20:08:29.807 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/qQGO0JjzaHwpMuQMRSNdJ2fZFM= 2022-09-12T20:08:29.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/qQGO0JjzaHwpMuQMRSNdJ2fZFM=', 'Content-Length': '0'} 2022-09-12T20:08:29.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000313d3c04116a1af5-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.810 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:29.811 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:29.811 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:08:29.812 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:08:29.813 DEBUG:boto:Method: GET 2022-09-12T20:08:29.813 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:08:29.814 DEBUG:boto:Data: 2022-09-12T20:08:29.814 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.815 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.815 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.816 DEBUG:boto:Params: {} 2022-09-12T20:08:29.816 DEBUG:boto:Token: None 2022-09-12T20:08:29.817 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/ 2022-09-12T20:08:29.817 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs= 2022-09-12T20:08:29.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs=', 'Content-Length': '0'} 2022-09-12T20:08:29.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001137f12c9a95cfac-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.821 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:29.822 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:29.823 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:29.823 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:08:29.824 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:08:29.824 DEBUG:boto:Method: GET 2022-09-12T20:08:29.825 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:08:29.825 DEBUG:boto:Data: 2022-09-12T20:08:29.826 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.826 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.827 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.827 DEBUG:boto:Params: {} 2022-09-12T20:08:29.828 DEBUG:boto:Token: None 2022-09-12T20:08:29.829 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/ 2022-09-12T20:08:29.829 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs= 2022-09-12T20:08:29.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs=', 'Content-Length': '0'} 2022-09-12T20:08:29.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c89c159c5c30715a-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.834 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:29.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:08:29.835 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:29.836 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:29.836 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:29.837 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:29.837 DEBUG:boto:Method: GET 2022-09-12T20:08:29.838 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:29.838 DEBUG:boto:Data: 2022-09-12T20:08:29.839 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.840 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.840 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.841 DEBUG:boto:Params: {} 2022-09-12T20:08:29.841 DEBUG:boto:Token: None 2022-09-12T20:08:29.842 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:29.842 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s4XLf2tNfP6D07gXzkhiTgHUZNY= 2022-09-12T20:08:29.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s4XLf2tNfP6D07gXzkhiTgHUZNY=', 'Content-Length': '0'} 2022-09-12T20:08:29.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ec0c2aa0756ec93-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.846 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:29.847 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:29.847 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:29.848 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:29.848 DEBUG:boto:Method: GET 2022-09-12T20:08:29.849 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:29.849 DEBUG:boto:Data: 2022-09-12T20:08:29.850 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.851 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.851 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.852 DEBUG:boto:Params: {} 2022-09-12T20:08:29.852 DEBUG:boto:Token: None 2022-09-12T20:08:29.853 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:29.853 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s4XLf2tNfP6D07gXzkhiTgHUZNY= 2022-09-12T20:08:29.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s4XLf2tNfP6D07gXzkhiTgHUZNY=', 'Content-Length': '0'} 2022-09-12T20:08:29.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000311be58cf888376d-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.857 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:29.858 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:29.858 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:29.859 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:29.859 DEBUG:boto:Method: HEAD 2022-09-12T20:08:29.860 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:29.860 DEBUG:boto:Data: 2022-09-12T20:08:29.861 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.861 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.862 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.862 DEBUG:boto:Params: {} 2022-09-12T20:08:29.863 DEBUG:boto:Token: None 2022-09-12T20:08:29.863 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:29.864 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SnL/C9vF4iQ9unSl3HjI793Lmjc= 2022-09-12T20:08:29.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SnL/C9vF4iQ9unSl3HjI793Lmjc=', 'Content-Length': '0'} 2022-09-12T20:08:29.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f6861c3fe35fdb5-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.867 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:29.868 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:29.868 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:29.869 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:29.869 DEBUG:boto:Method: HEAD 2022-09-12T20:08:29.870 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:29.870 DEBUG:boto:Data: 2022-09-12T20:08:29.871 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.871 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.872 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.872 DEBUG:boto:Params: {} 2022-09-12T20:08:29.873 DEBUG:boto:Token: None 2022-09-12T20:08:29.873 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:29.874 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SnL/C9vF4iQ9unSl3HjI793Lmjc= 2022-09-12T20:08:29.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SnL/C9vF4iQ9unSl3HjI793Lmjc=', 'Content-Length': '0'} 2022-09-12T20:08:29.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b64406bd7f3999e-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:08:29.878 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:29.879 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:29.880 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:29.880 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:29.881 DEBUG:boto:Method: GET 2022-09-12T20:08:29.881 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:29.882 DEBUG:boto:Data: 2022-09-12T20:08:29.882 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.883 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.883 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.884 DEBUG:boto:Params: {} 2022-09-12T20:08:29.884 DEBUG:boto:Token: None 2022-09-12T20:08:29.885 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:29.886 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s4XLf2tNfP6D07gXzkhiTgHUZNY= 2022-09-12T20:08:29.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s4XLf2tNfP6D07gXzkhiTgHUZNY=', 'Content-Length': '0'} 2022-09-12T20:08:29.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e8574173cfd0d41-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.930 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:29.931 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:29.931 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:29.932 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:29.932 DEBUG:boto:Method: GET 2022-09-12T20:08:29.933 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:29.933 DEBUG:boto:Data: 2022-09-12T20:08:29.934 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.935 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.935 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.936 DEBUG:boto:Params: {} 2022-09-12T20:08:29.936 DEBUG:boto:Token: None 2022-09-12T20:08:29.937 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:29.937 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s4XLf2tNfP6D07gXzkhiTgHUZNY= 2022-09-12T20:08:29.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s4XLf2tNfP6D07gXzkhiTgHUZNY=', 'Content-Length': '0'} 2022-09-12T20:08:29.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000871ce5f2d808ec5-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.941 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:29.942 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:29.942 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:08:29.943 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:08:29.943 DEBUG:boto:Method: GET 2022-09-12T20:08:29.944 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:08:29.945 DEBUG:boto:Data: 2022-09-12T20:08:29.945 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.946 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.946 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.947 DEBUG:boto:Params: {} 2022-09-12T20:08:29.947 DEBUG:boto:Token: None 2022-09-12T20:08:29.948 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/ 2022-09-12T20:08:29.948 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs= 2022-09-12T20:08:29.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs=', 'Content-Length': '0'} 2022-09-12T20:08:29.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff9d08ecbef9b3d6-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.953 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:29.953 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:29.954 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:29.954 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:08:29.955 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:08:29.956 DEBUG:boto:Method: GET 2022-09-12T20:08:29.956 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:08:29.957 DEBUG:boto:Data: 2022-09-12T20:08:29.957 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.958 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.958 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.959 DEBUG:boto:Params: {} 2022-09-12T20:08:29.959 DEBUG:boto:Token: None 2022-09-12T20:08:29.960 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/ 2022-09-12T20:08:29.961 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs= 2022-09-12T20:08:29.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YQ6yLPm8i0F1QhyQpYIk0wc2qIs=', 'Content-Length': '0'} 2022-09-12T20:08:29.965 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ef36805a0989c23-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.965 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:29.966 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:29.967 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:29.967 DEBUG:boto:Method: HEAD 2022-09-12T20:08:29.968 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:08:29.968 DEBUG:boto:Data: 2022-09-12T20:08:29.969 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.969 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.970 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.970 DEBUG:boto:Params: {} 2022-09-12T20:08:29.971 DEBUG:boto:Token: None 2022-09-12T20:08:29.971 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj51 2022-09-12T20:08:29.972 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W9vvm/5961ZDKzL4tZgUsbcmvSs= 2022-09-12T20:08:29.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W9vvm/5961ZDKzL4tZgUsbcmvSs=', 'Content-Length': '0'} 2022-09-12T20:08:29.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d33b66ca74c8bf31-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.976 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:29.976 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:29.977 DEBUG:boto:Method: HEAD 2022-09-12T20:08:29.977 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:08:29.978 DEBUG:boto:Data: 2022-09-12T20:08:29.978 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.979 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:29.979 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.980 DEBUG:boto:Params: {} 2022-09-12T20:08:29.980 DEBUG:boto:Token: None 2022-09-12T20:08:29.981 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj51 2022-09-12T20:08:29.981 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W9vvm/5961ZDKzL4tZgUsbcmvSs= 2022-09-12T20:08:29.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W9vvm/5961ZDKzL4tZgUsbcmvSs=', 'Content-Length': '0'} 2022-09-12T20:08:29.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d02a6661dea9a7a-00631f91bd-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:08:29.985 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:29.986 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:29.986 DEBUG:boto:Method: GET 2022-09-12T20:08:29.987 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:08:29.988 DEBUG:boto:Data: 2022-09-12T20:08:29.988 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.989 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:29.989 DEBUG:boto:Port: 8000 2022-09-12T20:08:29.990 DEBUG:boto:Params: {} 2022-09-12T20:08:29.991 DEBUG:boto:Token: None 2022-09-12T20:08:29.991 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:29 GMT /jdpiak-75/obj51 2022-09-12T20:08:29.992 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VAT8aGCx69dWSrXXqd3TvWVJA2M= 2022-09-12T20:08:29.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:29 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VAT8aGCx69dWSrXXqd3TvWVJA2M=', 'Content-Length': '0'} 2022-09-12T20:08:29.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6534afed377f983-00631f91bd-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:29.996 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:29.996 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:29.997 DEBUG:boto:Method: GET 2022-09-12T20:08:29.997 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:08:29.998 DEBUG:boto:Data: 2022-09-12T20:08:29.999 DEBUG:boto:Headers: {} 2022-09-12T20:08:29.999 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.000 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.001 DEBUG:boto:Params: {} 2022-09-12T20:08:30.001 DEBUG:boto:Token: None 2022-09-12T20:08:30.002 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj51 2022-09-12T20:08:30.003 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0n/q+YilTAlLh973sEOcClMOLDU= 2022-09-12T20:08:30.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0n/q+YilTAlLh973sEOcClMOLDU=', 'Content-Length': '0'} 2022-09-12T20:08:30.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087a80c2fb4d22aeb-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.007 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:30.007 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:30.008 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:08:30.008 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:08:30.009 DEBUG:boto:Method: GET 2022-09-12T20:08:30.009 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:08:30.010 DEBUG:boto:Data: 2022-09-12T20:08:30.011 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.011 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.012 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.012 DEBUG:boto:Params: {} 2022-09-12T20:08:30.013 DEBUG:boto:Token: None 2022-09-12T20:08:30.014 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/ 2022-09-12T20:08:30.014 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI= 2022-09-12T20:08:30.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI=', 'Content-Length': '0'} 2022-09-12T20:08:30.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fff1873c940d000-00631f91be-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.058 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:30.059 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:30.060 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:30.060 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:08:30.061 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:08:30.061 DEBUG:boto:Method: GET 2022-09-12T20:08:30.062 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:08:30.063 DEBUG:boto:Data: 2022-09-12T20:08:30.063 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.064 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.065 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.065 DEBUG:boto:Params: {} 2022-09-12T20:08:30.066 DEBUG:boto:Token: None 2022-09-12T20:08:30.067 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/ 2022-09-12T20:08:30.067 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI= 2022-09-12T20:08:30.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI=', 'Content-Length': '0'} 2022-09-12T20:08:30.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9c4fde760e196bf-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.072 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:30.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:08:30.073 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:30.074 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:30.075 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:30.075 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:30.076 DEBUG:boto:Method: GET 2022-09-12T20:08:30.076 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:30.077 DEBUG:boto:Data: 2022-09-12T20:08:30.077 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.078 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.078 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.079 DEBUG:boto:Params: {} 2022-09-12T20:08:30.079 DEBUG:boto:Token: None 2022-09-12T20:08:30.080 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:30.080 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EW1dzPuhLv6kfDzcXxum15rlVcY= 2022-09-12T20:08:30.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EW1dzPuhLv6kfDzcXxum15rlVcY=', 'Content-Length': '0'} 2022-09-12T20:08:30.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f388357598be6066-00631f91be-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.084 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:30.085 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:30.085 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:30.086 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:30.086 DEBUG:boto:Method: GET 2022-09-12T20:08:30.087 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:30.087 DEBUG:boto:Data: 2022-09-12T20:08:30.088 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.089 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.089 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.090 DEBUG:boto:Params: {} 2022-09-12T20:08:30.090 DEBUG:boto:Token: None 2022-09-12T20:08:30.091 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:30.091 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EW1dzPuhLv6kfDzcXxum15rlVcY= 2022-09-12T20:08:30.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EW1dzPuhLv6kfDzcXxum15rlVcY=', 'Content-Length': '0'} 2022-09-12T20:08:30.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e27ff41af2e9d99f-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.138 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:30.138 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:30.139 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:30.139 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:30.140 DEBUG:boto:Method: HEAD 2022-09-12T20:08:30.140 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:30.141 DEBUG:boto:Data: 2022-09-12T20:08:30.141 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.142 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.142 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.143 DEBUG:boto:Params: {} 2022-09-12T20:08:30.144 DEBUG:boto:Token: None 2022-09-12T20:08:30.144 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:30.145 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0EZ9hwmjJYmYVDklBWKTEDXKdbk= 2022-09-12T20:08:30.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0EZ9hwmjJYmYVDklBWKTEDXKdbk=', 'Content-Length': '0'} 2022-09-12T20:08:30.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044aece934e3e3c89-00631f91be-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.148 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:30.149 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:30.149 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:30.149 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:30.150 DEBUG:boto:Method: HEAD 2022-09-12T20:08:30.151 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:30.151 DEBUG:boto:Data: 2022-09-12T20:08:30.152 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.152 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.152 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.153 DEBUG:boto:Params: {} 2022-09-12T20:08:30.154 DEBUG:boto:Token: None 2022-09-12T20:08:30.154 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:30.155 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0EZ9hwmjJYmYVDklBWKTEDXKdbk= 2022-09-12T20:08:30.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0EZ9hwmjJYmYVDklBWKTEDXKdbk=', 'Content-Length': '0'} 2022-09-12T20:08:30.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000be991c4771bfe2b-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:08:30.159 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:30.159 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:30.160 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:30.160 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:30.161 DEBUG:boto:Method: GET 2022-09-12T20:08:30.161 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:30.162 DEBUG:boto:Data: 2022-09-12T20:08:30.162 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.163 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.163 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.164 DEBUG:boto:Params: {} 2022-09-12T20:08:30.164 DEBUG:boto:Token: None 2022-09-12T20:08:30.165 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:30.166 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EW1dzPuhLv6kfDzcXxum15rlVcY= 2022-09-12T20:08:30.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EW1dzPuhLv6kfDzcXxum15rlVcY=', 'Content-Length': '0'} 2022-09-12T20:08:30.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ba0c0667eedd82e-00631f91be-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.169 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:30.169 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:30.170 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:30.171 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:30.171 DEBUG:boto:Method: GET 2022-09-12T20:08:30.172 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:30.172 DEBUG:boto:Data: 2022-09-12T20:08:30.173 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.173 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.174 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.174 DEBUG:boto:Params: {} 2022-09-12T20:08:30.175 DEBUG:boto:Token: None 2022-09-12T20:08:30.175 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:30.176 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EW1dzPuhLv6kfDzcXxum15rlVcY= 2022-09-12T20:08:30.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EW1dzPuhLv6kfDzcXxum15rlVcY=', 'Content-Length': '0'} 2022-09-12T20:08:30.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b25410b090f2453-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.179 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:30.180 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:30.180 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:08:30.181 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:08:30.181 DEBUG:boto:Method: GET 2022-09-12T20:08:30.182 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:08:30.182 DEBUG:boto:Data: 2022-09-12T20:08:30.183 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.184 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.184 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.185 DEBUG:boto:Params: {} 2022-09-12T20:08:30.185 DEBUG:boto:Token: None 2022-09-12T20:08:30.186 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/ 2022-09-12T20:08:30.186 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI= 2022-09-12T20:08:30.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI=', 'Content-Length': '0'} 2022-09-12T20:08:30.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075a2d83455b71d95-00631f91be-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.190 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:30.191 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:30.192 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:30.192 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:08:30.193 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:08:30.193 DEBUG:boto:Method: GET 2022-09-12T20:08:30.194 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:08:30.194 DEBUG:boto:Data: 2022-09-12T20:08:30.195 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.195 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.196 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.196 DEBUG:boto:Params: {} 2022-09-12T20:08:30.197 DEBUG:boto:Token: None 2022-09-12T20:08:30.197 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/ 2022-09-12T20:08:30.198 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI= 2022-09-12T20:08:30.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI=', 'Content-Length': '0'} 2022-09-12T20:08:30.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d037ff48d7e6fc4-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.203 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:30.203 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:30.204 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:30.204 DEBUG:boto:Method: HEAD 2022-09-12T20:08:30.205 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:08:30.205 DEBUG:boto:Data: 2022-09-12T20:08:30.206 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.206 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.207 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.207 DEBUG:boto:Params: {} 2022-09-12T20:08:30.208 DEBUG:boto:Token: None 2022-09-12T20:08:30.209 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj55 2022-09-12T20:08:30.209 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6vCRLV/YWZjO2AZVqzvnI7IeYbs= 2022-09-12T20:08:30.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6vCRLV/YWZjO2AZVqzvnI7IeYbs=', 'Content-Length': '0'} 2022-09-12T20:08:30.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000919891b928ae2b5c-00631f91be-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.213 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:30.213 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:30.214 DEBUG:boto:Method: HEAD 2022-09-12T20:08:30.214 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:08:30.215 DEBUG:boto:Data: 2022-09-12T20:08:30.215 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.216 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.216 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.216 DEBUG:boto:Params: {} 2022-09-12T20:08:30.217 DEBUG:boto:Token: None 2022-09-12T20:08:30.217 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj55 2022-09-12T20:08:30.218 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6vCRLV/YWZjO2AZVqzvnI7IeYbs= 2022-09-12T20:08:30.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6vCRLV/YWZjO2AZVqzvnI7IeYbs=', 'Content-Length': '0'} 2022-09-12T20:08:30.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000042bd75aa3dfd596-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:08:30.222 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:30.222 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:30.223 DEBUG:boto:Method: GET 2022-09-12T20:08:30.223 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:08:30.224 DEBUG:boto:Data: 2022-09-12T20:08:30.224 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.225 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.225 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.226 DEBUG:boto:Params: {} 2022-09-12T20:08:30.226 DEBUG:boto:Token: None 2022-09-12T20:08:30.227 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj55 2022-09-12T20:08:30.228 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WFDHYtU/273WSw+yk5I+XNSuvmo= 2022-09-12T20:08:30.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WFDHYtU/273WSw+yk5I+XNSuvmo=', 'Content-Length': '0'} 2022-09-12T20:08:30.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b661cabec310df1-00631f91be-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.231 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:30.232 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:30.232 DEBUG:boto:Method: GET 2022-09-12T20:08:30.233 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:08:30.233 DEBUG:boto:Data: 2022-09-12T20:08:30.234 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.234 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.235 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.236 DEBUG:boto:Params: {} 2022-09-12T20:08:30.236 DEBUG:boto:Token: None 2022-09-12T20:08:30.237 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj55 2022-09-12T20:08:30.237 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WFDHYtU/273WSw+yk5I+XNSuvmo= 2022-09-12T20:08:30.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WFDHYtU/273WSw+yk5I+XNSuvmo=', 'Content-Length': '0'} 2022-09-12T20:08:30.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d6fedd4a910d98b-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.282 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:30.283 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:30.283 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:08:30.284 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:08:30.284 DEBUG:boto:Method: GET 2022-09-12T20:08:30.285 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:08:30.286 DEBUG:boto:Data: 2022-09-12T20:08:30.286 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.287 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.288 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.288 DEBUG:boto:Params: {} 2022-09-12T20:08:30.289 DEBUG:boto:Token: None 2022-09-12T20:08:30.290 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/ 2022-09-12T20:08:30.290 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI= 2022-09-12T20:08:30.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI=', 'Content-Length': '0'} 2022-09-12T20:08:30.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000202b5f32fe410063-00631f91be-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.294 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:30.295 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:30.296 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:30.296 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:08:30.297 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:08:30.297 DEBUG:boto:Method: GET 2022-09-12T20:08:30.298 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:08:30.298 DEBUG:boto:Data: 2022-09-12T20:08:30.299 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.299 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.300 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.300 DEBUG:boto:Params: {} 2022-09-12T20:08:30.301 DEBUG:boto:Token: None 2022-09-12T20:08:30.301 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/ 2022-09-12T20:08:30.302 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI= 2022-09-12T20:08:30.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI=', 'Content-Length': '0'} 2022-09-12T20:08:30.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b79fc0b653cde1cb-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.307 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:30.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:08:30.308 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:30.309 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:30.309 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:30.310 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:30.310 DEBUG:boto:Method: GET 2022-09-12T20:08:30.311 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:30.311 DEBUG:boto:Data: 2022-09-12T20:08:30.312 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.312 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.313 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.313 DEBUG:boto:Params: {} 2022-09-12T20:08:30.314 DEBUG:boto:Token: None 2022-09-12T20:08:30.315 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:30.315 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GewHh5sWedVOAIVA/grQLZmrmLA= 2022-09-12T20:08:30.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GewHh5sWedVOAIVA/grQLZmrmLA=', 'Content-Length': '0'} 2022-09-12T20:08:30.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000187e5802bb8f54a4-00631f91be-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.319 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:30.319 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:30.320 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:30.320 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:30.321 DEBUG:boto:Method: GET 2022-09-12T20:08:30.321 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:30.322 DEBUG:boto:Data: 2022-09-12T20:08:30.322 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.323 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.323 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.324 DEBUG:boto:Params: {} 2022-09-12T20:08:30.325 DEBUG:boto:Token: None 2022-09-12T20:08:30.325 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:30.326 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GewHh5sWedVOAIVA/grQLZmrmLA= 2022-09-12T20:08:30.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GewHh5sWedVOAIVA/grQLZmrmLA=', 'Content-Length': '0'} 2022-09-12T20:08:30.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000539e36f121da9107-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.329 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:30.330 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:30.330 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:30.331 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:30.331 DEBUG:boto:Method: HEAD 2022-09-12T20:08:30.332 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:30.332 DEBUG:boto:Data: 2022-09-12T20:08:30.333 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.333 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.334 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.334 DEBUG:boto:Params: {} 2022-09-12T20:08:30.335 DEBUG:boto:Token: None 2022-09-12T20:08:30.335 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:30.336 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jvDPXio/2pjG1vLMDhgjk0ImVy8= 2022-09-12T20:08:30.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jvDPXio/2pjG1vLMDhgjk0ImVy8=', 'Content-Length': '0'} 2022-09-12T20:08:30.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f67c5a866248cd58-00631f91be-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.339 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:30.340 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:30.340 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:30.341 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:30.341 DEBUG:boto:Method: HEAD 2022-09-12T20:08:30.342 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:30.342 DEBUG:boto:Data: 2022-09-12T20:08:30.343 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.343 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.344 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.344 DEBUG:boto:Params: {} 2022-09-12T20:08:30.345 DEBUG:boto:Token: None 2022-09-12T20:08:30.345 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:30.346 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jvDPXio/2pjG1vLMDhgjk0ImVy8= 2022-09-12T20:08:30.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jvDPXio/2pjG1vLMDhgjk0ImVy8=', 'Content-Length': '0'} 2022-09-12T20:08:30.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5ef98cd32761bb6-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:08:30.350 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:30.351 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:30.351 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:30.352 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:30.352 DEBUG:boto:Method: GET 2022-09-12T20:08:30.353 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:30.353 DEBUG:boto:Data: 2022-09-12T20:08:30.354 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.354 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.355 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.355 DEBUG:boto:Params: {} 2022-09-12T20:08:30.356 DEBUG:boto:Token: None 2022-09-12T20:08:30.356 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:30.357 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GewHh5sWedVOAIVA/grQLZmrmLA= 2022-09-12T20:08:30.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GewHh5sWedVOAIVA/grQLZmrmLA=', 'Content-Length': '0'} 2022-09-12T20:08:30.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7b60f79e48f449b-00631f91be-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.402 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:30.403 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:30.403 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:30.404 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:30.404 DEBUG:boto:Method: GET 2022-09-12T20:08:30.405 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:30.406 DEBUG:boto:Data: 2022-09-12T20:08:30.406 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.407 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.407 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.408 DEBUG:boto:Params: {} 2022-09-12T20:08:30.408 DEBUG:boto:Token: None 2022-09-12T20:08:30.409 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:30.410 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GewHh5sWedVOAIVA/grQLZmrmLA= 2022-09-12T20:08:30.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GewHh5sWedVOAIVA/grQLZmrmLA=', 'Content-Length': '0'} 2022-09-12T20:08:30.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fee284181716902a-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.414 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:30.414 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:30.415 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:08:30.415 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:08:30.416 DEBUG:boto:Method: GET 2022-09-12T20:08:30.416 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:08:30.417 DEBUG:boto:Data: 2022-09-12T20:08:30.418 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.418 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.419 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.419 DEBUG:boto:Params: {} 2022-09-12T20:08:30.420 DEBUG:boto:Token: None 2022-09-12T20:08:30.420 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/ 2022-09-12T20:08:30.421 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI= 2022-09-12T20:08:30.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI=', 'Content-Length': '0'} 2022-09-12T20:08:30.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b63583800ddeefe-00631f91be-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.425 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:30.426 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:30.426 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:30.427 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:08:30.427 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:08:30.428 DEBUG:boto:Method: GET 2022-09-12T20:08:30.429 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:08:30.429 DEBUG:boto:Data: 2022-09-12T20:08:30.430 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.430 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.431 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.431 DEBUG:boto:Params: {} 2022-09-12T20:08:30.432 DEBUG:boto:Token: None 2022-09-12T20:08:30.432 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/ 2022-09-12T20:08:30.433 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI= 2022-09-12T20:08:30.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI=', 'Content-Length': '0'} 2022-09-12T20:08:30.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003342dbcf87cf705b-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.438 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:30.439 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:30.439 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:30.440 DEBUG:boto:Method: HEAD 2022-09-12T20:08:30.440 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:08:30.441 DEBUG:boto:Data: 2022-09-12T20:08:30.441 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.442 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.442 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.443 DEBUG:boto:Params: {} 2022-09-12T20:08:30.443 DEBUG:boto:Token: None 2022-09-12T20:08:30.444 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj57 2022-09-12T20:08:30.444 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/yct/AHurO44lDzBbAdU8WlQbpA= 2022-09-12T20:08:30.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/yct/AHurO44lDzBbAdU8WlQbpA=', 'Content-Length': '0'} 2022-09-12T20:08:30.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a1b691a72b643c5-00631f91be-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.448 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:30.449 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:30.449 DEBUG:boto:Method: HEAD 2022-09-12T20:08:30.450 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:08:30.450 DEBUG:boto:Data: 2022-09-12T20:08:30.451 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.451 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.452 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.452 DEBUG:boto:Params: {} 2022-09-12T20:08:30.453 DEBUG:boto:Token: None 2022-09-12T20:08:30.454 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj57 2022-09-12T20:08:30.454 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/yct/AHurO44lDzBbAdU8WlQbpA= 2022-09-12T20:08:30.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/yct/AHurO44lDzBbAdU8WlQbpA=', 'Content-Length': '0'} 2022-09-12T20:08:30.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d4439a0df387865-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:08:30.458 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:30.459 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:30.459 DEBUG:boto:Method: GET 2022-09-12T20:08:30.460 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:08:30.461 DEBUG:boto:Data: 2022-09-12T20:08:30.461 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.462 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.462 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.463 DEBUG:boto:Params: {} 2022-09-12T20:08:30.463 DEBUG:boto:Token: None 2022-09-12T20:08:30.464 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj57 2022-09-12T20:08:30.465 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3oVOA0xeYIiCJ7DAZ0aOo9BKPsA= 2022-09-12T20:08:30.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3oVOA0xeYIiCJ7DAZ0aOo9BKPsA=', 'Content-Length': '0'} 2022-09-12T20:08:30.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021c932a93915c5cd-00631f91be-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.468 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:30.469 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:30.469 DEBUG:boto:Method: GET 2022-09-12T20:08:30.470 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:08:30.470 DEBUG:boto:Data: 2022-09-12T20:08:30.471 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.471 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.472 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.472 DEBUG:boto:Params: {} 2022-09-12T20:08:30.473 DEBUG:boto:Token: None 2022-09-12T20:08:30.474 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj57 2022-09-12T20:08:30.474 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3oVOA0xeYIiCJ7DAZ0aOo9BKPsA= 2022-09-12T20:08:30.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3oVOA0xeYIiCJ7DAZ0aOo9BKPsA=', 'Content-Length': '0'} 2022-09-12T20:08:30.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da729957499b8a5a-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.478 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:30.479 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:30.479 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:08:30.480 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:08:30.480 DEBUG:boto:Method: GET 2022-09-12T20:08:30.481 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:08:30.481 DEBUG:boto:Data: 2022-09-12T20:08:30.482 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.482 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.483 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.483 DEBUG:boto:Params: {} 2022-09-12T20:08:30.484 DEBUG:boto:Token: None 2022-09-12T20:08:30.485 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/ 2022-09-12T20:08:30.485 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI= 2022-09-12T20:08:30.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI=', 'Content-Length': '0'} 2022-09-12T20:08:30.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afec2ae6b43b1bc4-00631f91be-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.489 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:30.490 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:30.491 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:30.491 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:08:30.492 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:08:30.492 DEBUG:boto:Method: GET 2022-09-12T20:08:30.493 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:08:30.493 DEBUG:boto:Data: 2022-09-12T20:08:30.494 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.494 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.495 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.495 DEBUG:boto:Params: {} 2022-09-12T20:08:30.496 DEBUG:boto:Token: None 2022-09-12T20:08:30.497 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/ 2022-09-12T20:08:30.497 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI= 2022-09-12T20:08:30.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI=', 'Content-Length': '0'} 2022-09-12T20:08:30.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000720a6204472ae35-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.502 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:30.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:08:30.503 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:30.504 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:30.505 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:30.505 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:30.506 DEBUG:boto:Method: GET 2022-09-12T20:08:30.506 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:30.507 DEBUG:boto:Data: 2022-09-12T20:08:30.507 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.508 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.508 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.509 DEBUG:boto:Params: {} 2022-09-12T20:08:30.509 DEBUG:boto:Token: None 2022-09-12T20:08:30.510 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:30.511 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XUED3E/pUz44sF106PzHyLGbwnY= 2022-09-12T20:08:30.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XUED3E/pUz44sF106PzHyLGbwnY=', 'Content-Length': '0'} 2022-09-12T20:08:30.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007222e88b5b185824-00631f91be-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.514 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:30.515 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:30.515 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:30.516 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:30.516 DEBUG:boto:Method: GET 2022-09-12T20:08:30.517 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:30.517 DEBUG:boto:Data: 2022-09-12T20:08:30.518 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.519 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.519 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.520 DEBUG:boto:Params: {} 2022-09-12T20:08:30.520 DEBUG:boto:Token: None 2022-09-12T20:08:30.521 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:30.521 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XUED3E/pUz44sF106PzHyLGbwnY= 2022-09-12T20:08:30.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XUED3E/pUz44sF106PzHyLGbwnY=', 'Content-Length': '0'} 2022-09-12T20:08:30.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d3856edc6c65103-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.525 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:30.526 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:30.526 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:30.527 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:30.527 DEBUG:boto:Method: HEAD 2022-09-12T20:08:30.528 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:30.528 DEBUG:boto:Data: 2022-09-12T20:08:30.529 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.529 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.530 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.530 DEBUG:boto:Params: {} 2022-09-12T20:08:30.531 DEBUG:boto:Token: None 2022-09-12T20:08:30.531 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:30.532 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WODGEEMe9Lp/ol2oiaKVevd9wnU= 2022-09-12T20:08:30.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WODGEEMe9Lp/ol2oiaKVevd9wnU=', 'Content-Length': '0'} 2022-09-12T20:08:30.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000596c2086ade7de58-00631f91be-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.535 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:30.536 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:30.536 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:30.537 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:30.537 DEBUG:boto:Method: HEAD 2022-09-12T20:08:30.538 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:30.538 DEBUG:boto:Data: 2022-09-12T20:08:30.539 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.539 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.540 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.540 DEBUG:boto:Params: {} 2022-09-12T20:08:30.541 DEBUG:boto:Token: None 2022-09-12T20:08:30.542 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:30.542 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WODGEEMe9Lp/ol2oiaKVevd9wnU= 2022-09-12T20:08:30.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WODGEEMe9Lp/ol2oiaKVevd9wnU=', 'Content-Length': '0'} 2022-09-12T20:08:30.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c2fabdecabb8c7e-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:08:30.547 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:30.547 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:30.548 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:30.548 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:30.549 DEBUG:boto:Method: GET 2022-09-12T20:08:30.549 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:30.550 DEBUG:boto:Data: 2022-09-12T20:08:30.550 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.551 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.551 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.552 DEBUG:boto:Params: {} 2022-09-12T20:08:30.553 DEBUG:boto:Token: None 2022-09-12T20:08:30.553 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:30.554 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XUED3E/pUz44sF106PzHyLGbwnY= 2022-09-12T20:08:30.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XUED3E/pUz44sF106PzHyLGbwnY=', 'Content-Length': '0'} 2022-09-12T20:08:30.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074d1912481db621e-00631f91be-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.557 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:30.558 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:30.558 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:30.559 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:30.560 DEBUG:boto:Method: GET 2022-09-12T20:08:30.560 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:30.561 DEBUG:boto:Data: 2022-09-12T20:08:30.561 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.562 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.562 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.563 DEBUG:boto:Params: {} 2022-09-12T20:08:30.563 DEBUG:boto:Token: None 2022-09-12T20:08:30.564 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:30.564 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XUED3E/pUz44sF106PzHyLGbwnY= 2022-09-12T20:08:30.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XUED3E/pUz44sF106PzHyLGbwnY=', 'Content-Length': '0'} 2022-09-12T20:08:30.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000024d2ad6273aecd-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.568 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:30.569 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:30.569 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:08:30.570 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:08:30.570 DEBUG:boto:Method: GET 2022-09-12T20:08:30.571 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:08:30.571 DEBUG:boto:Data: 2022-09-12T20:08:30.572 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.572 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.573 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.573 DEBUG:boto:Params: {} 2022-09-12T20:08:30.574 DEBUG:boto:Token: None 2022-09-12T20:08:30.575 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/ 2022-09-12T20:08:30.575 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI= 2022-09-12T20:08:30.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI=', 'Content-Length': '0'} 2022-09-12T20:08:30.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008429bd2c8fe0c778-00631f91be-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.579 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:30.580 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:30.581 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:30.581 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:08:30.582 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:08:30.582 DEBUG:boto:Method: GET 2022-09-12T20:08:30.583 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:08:30.583 DEBUG:boto:Data: 2022-09-12T20:08:30.584 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.584 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.585 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.585 DEBUG:boto:Params: {} 2022-09-12T20:08:30.586 DEBUG:boto:Token: None 2022-09-12T20:08:30.586 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/ 2022-09-12T20:08:30.587 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI= 2022-09-12T20:08:30.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI=', 'Content-Length': '0'} 2022-09-12T20:08:30.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e497cd13c1f64e7-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.592 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:30.593 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:30.593 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:30.594 DEBUG:boto:Method: HEAD 2022-09-12T20:08:30.594 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:08:30.595 DEBUG:boto:Data: 2022-09-12T20:08:30.595 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.596 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.596 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.597 DEBUG:boto:Params: {} 2022-09-12T20:08:30.597 DEBUG:boto:Token: None 2022-09-12T20:08:30.598 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj68 2022-09-12T20:08:30.598 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iqPokeuZiXOkqhpSTYwzWPvGeDM= 2022-09-12T20:08:30.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iqPokeuZiXOkqhpSTYwzWPvGeDM=', 'Content-Length': '0'} 2022-09-12T20:08:30.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051f66ae649f034cc-00631f91be-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.602 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:30.602 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:30.603 DEBUG:boto:Method: HEAD 2022-09-12T20:08:30.603 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:08:30.604 DEBUG:boto:Data: 2022-09-12T20:08:30.604 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.604 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.605 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.605 DEBUG:boto:Params: {} 2022-09-12T20:08:30.606 DEBUG:boto:Token: None 2022-09-12T20:08:30.607 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj68 2022-09-12T20:08:30.607 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iqPokeuZiXOkqhpSTYwzWPvGeDM= 2022-09-12T20:08:30.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iqPokeuZiXOkqhpSTYwzWPvGeDM=', 'Content-Length': '0'} 2022-09-12T20:08:30.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043af3533b8e00768-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:08:30.611 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:30.611 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:30.612 DEBUG:boto:Method: GET 2022-09-12T20:08:30.613 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:08:30.613 DEBUG:boto:Data: 2022-09-12T20:08:30.614 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.614 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.615 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.615 DEBUG:boto:Params: {} 2022-09-12T20:08:30.616 DEBUG:boto:Token: None 2022-09-12T20:08:30.616 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj68 2022-09-12T20:08:30.617 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0v1Zy0SQcuPhvEexTaUplzI+7CI= 2022-09-12T20:08:30.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0v1Zy0SQcuPhvEexTaUplzI+7CI=', 'Content-Length': '0'} 2022-09-12T20:08:30.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a10370a2cda468d1-00631f91be-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.662 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:30.663 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:30.663 DEBUG:boto:Method: GET 2022-09-12T20:08:30.664 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:08:30.664 DEBUG:boto:Data: 2022-09-12T20:08:30.665 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.665 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.666 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.666 DEBUG:boto:Params: {} 2022-09-12T20:08:30.667 DEBUG:boto:Token: None 2022-09-12T20:08:30.668 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj68 2022-09-12T20:08:30.669 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0v1Zy0SQcuPhvEexTaUplzI+7CI= 2022-09-12T20:08:30.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0v1Zy0SQcuPhvEexTaUplzI+7CI=', 'Content-Length': '0'} 2022-09-12T20:08:30.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047af90718fab256f-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.674 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:30.675 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:30.675 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:08:30.676 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:08:30.676 DEBUG:boto:Method: GET 2022-09-12T20:08:30.677 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:08:30.677 DEBUG:boto:Data: 2022-09-12T20:08:30.678 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.678 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.679 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.679 DEBUG:boto:Params: {} 2022-09-12T20:08:30.680 DEBUG:boto:Token: None 2022-09-12T20:08:30.680 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/ 2022-09-12T20:08:30.681 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI= 2022-09-12T20:08:30.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI=', 'Content-Length': '0'} 2022-09-12T20:08:30.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040dcb829320b7ffa-00631f91be-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.685 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:30.686 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:30.686 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:30.687 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:08:30.687 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:08:30.688 DEBUG:boto:Method: GET 2022-09-12T20:08:30.689 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:08:30.689 DEBUG:boto:Data: 2022-09-12T20:08:30.690 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.690 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.691 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.691 DEBUG:boto:Params: {} 2022-09-12T20:08:30.692 DEBUG:boto:Token: None 2022-09-12T20:08:30.692 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/ 2022-09-12T20:08:30.693 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI= 2022-09-12T20:08:30.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI=', 'Content-Length': '0'} 2022-09-12T20:08:30.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099f93cd78c304645-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.698 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:30.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:08:30.699 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:30.699 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:30.700 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:30.700 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:30.701 DEBUG:boto:Method: GET 2022-09-12T20:08:30.701 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:30.702 DEBUG:boto:Data: 2022-09-12T20:08:30.703 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.703 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.704 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.704 DEBUG:boto:Params: {} 2022-09-12T20:08:30.705 DEBUG:boto:Token: None 2022-09-12T20:08:30.705 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:30.706 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NTIdL1kegVJT2xJd4c8Zbh72Qyc= 2022-09-12T20:08:30.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NTIdL1kegVJT2xJd4c8Zbh72Qyc=', 'Content-Length': '0'} 2022-09-12T20:08:30.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000044dbc8a85f3a599-00631f91be-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.709 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:30.710 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:30.711 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:30.711 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:30.711 DEBUG:boto:Method: GET 2022-09-12T20:08:30.712 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:30.713 DEBUG:boto:Data: 2022-09-12T20:08:30.713 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.714 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.714 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.715 DEBUG:boto:Params: {} 2022-09-12T20:08:30.715 DEBUG:boto:Token: None 2022-09-12T20:08:30.716 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:30.716 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NTIdL1kegVJT2xJd4c8Zbh72Qyc= 2022-09-12T20:08:30.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NTIdL1kegVJT2xJd4c8Zbh72Qyc=', 'Content-Length': '0'} 2022-09-12T20:08:30.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037d1f9561549d169-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.720 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:30.720 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:30.721 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:30.721 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:30.722 DEBUG:boto:Method: HEAD 2022-09-12T20:08:30.722 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:30.723 DEBUG:boto:Data: 2022-09-12T20:08:30.723 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.724 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.724 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.725 DEBUG:boto:Params: {} 2022-09-12T20:08:30.725 DEBUG:boto:Token: None 2022-09-12T20:08:30.726 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:30.726 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8oJpVM4tMuLRDGpqNcBGUwVQLsE= 2022-09-12T20:08:30.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8oJpVM4tMuLRDGpqNcBGUwVQLsE=', 'Content-Length': '0'} 2022-09-12T20:08:30.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a591fba20ff959d9-00631f91be-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.730 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:30.730 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:30.731 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:30.731 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:30.732 DEBUG:boto:Method: HEAD 2022-09-12T20:08:30.732 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:30.733 DEBUG:boto:Data: 2022-09-12T20:08:30.733 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.734 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.734 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.735 DEBUG:boto:Params: {} 2022-09-12T20:08:30.735 DEBUG:boto:Token: None 2022-09-12T20:08:30.736 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:30.737 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8oJpVM4tMuLRDGpqNcBGUwVQLsE= 2022-09-12T20:08:30.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8oJpVM4tMuLRDGpqNcBGUwVQLsE=', 'Content-Length': '0'} 2022-09-12T20:08:30.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007131236bdd563d91-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:08:30.740 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:30.741 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:30.741 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:30.742 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:30.742 DEBUG:boto:Method: GET 2022-09-12T20:08:30.743 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:30.744 DEBUG:boto:Data: 2022-09-12T20:08:30.744 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.745 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.745 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.746 DEBUG:boto:Params: {} 2022-09-12T20:08:30.746 DEBUG:boto:Token: None 2022-09-12T20:08:30.747 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:30.748 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NTIdL1kegVJT2xJd4c8Zbh72Qyc= 2022-09-12T20:08:30.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NTIdL1kegVJT2xJd4c8Zbh72Qyc=', 'Content-Length': '0'} 2022-09-12T20:08:30.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002165e9b24e0c4901-00631f91be-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.751 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:30.751 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:30.752 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:30.752 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:30.753 DEBUG:boto:Method: GET 2022-09-12T20:08:30.753 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:30.754 DEBUG:boto:Data: 2022-09-12T20:08:30.755 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.755 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.756 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.756 DEBUG:boto:Params: {} 2022-09-12T20:08:30.757 DEBUG:boto:Token: None 2022-09-12T20:08:30.757 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:30.758 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NTIdL1kegVJT2xJd4c8Zbh72Qyc= 2022-09-12T20:08:30.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NTIdL1kegVJT2xJd4c8Zbh72Qyc=', 'Content-Length': '0'} 2022-09-12T20:08:30.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000668f777a8f9dea5a-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.762 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:30.762 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:30.763 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:08:30.763 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:08:30.764 DEBUG:boto:Method: GET 2022-09-12T20:08:30.764 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:08:30.765 DEBUG:boto:Data: 2022-09-12T20:08:30.765 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.766 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.766 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.767 DEBUG:boto:Params: {} 2022-09-12T20:08:30.767 DEBUG:boto:Token: None 2022-09-12T20:08:30.768 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/ 2022-09-12T20:08:30.769 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI= 2022-09-12T20:08:30.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI=', 'Content-Length': '0'} 2022-09-12T20:08:30.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083ba77ee7f48091a-00631f91be-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.772 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:30.773 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:30.774 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:30.774 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:08:30.775 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:08:30.775 DEBUG:boto:Method: GET 2022-09-12T20:08:30.776 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:08:30.776 DEBUG:boto:Data: 2022-09-12T20:08:30.777 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.777 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.778 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.778 DEBUG:boto:Params: {} 2022-09-12T20:08:30.779 DEBUG:boto:Token: None 2022-09-12T20:08:30.780 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/ 2022-09-12T20:08:30.780 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI= 2022-09-12T20:08:30.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI=', 'Content-Length': '0'} 2022-09-12T20:08:30.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5c07a8788598949-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.785 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:30.786 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:30.786 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:30.787 DEBUG:boto:Method: HEAD 2022-09-12T20:08:30.787 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:08:30.788 DEBUG:boto:Data: 2022-09-12T20:08:30.788 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.789 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.789 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.790 DEBUG:boto:Params: {} 2022-09-12T20:08:30.790 DEBUG:boto:Token: None 2022-09-12T20:08:30.791 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj76 2022-09-12T20:08:30.791 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vkVEtwiCylK2n0Wsxh8QesqjDUU= 2022-09-12T20:08:30.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vkVEtwiCylK2n0Wsxh8QesqjDUU=', 'Content-Length': '0'} 2022-09-12T20:08:30.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c15400d7debcadb-00631f91be-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.794 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:30.795 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:30.795 DEBUG:boto:Method: HEAD 2022-09-12T20:08:30.796 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:08:30.796 DEBUG:boto:Data: 2022-09-12T20:08:30.797 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.797 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.798 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.798 DEBUG:boto:Params: {} 2022-09-12T20:08:30.799 DEBUG:boto:Token: None 2022-09-12T20:08:30.800 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj76 2022-09-12T20:08:30.800 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vkVEtwiCylK2n0Wsxh8QesqjDUU= 2022-09-12T20:08:30.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vkVEtwiCylK2n0Wsxh8QesqjDUU=', 'Content-Length': '0'} 2022-09-12T20:08:30.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df9ce97b6ed9260e-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:08:30.804 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:30.804 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:30.805 DEBUG:boto:Method: GET 2022-09-12T20:08:30.805 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:08:30.806 DEBUG:boto:Data: 2022-09-12T20:08:30.806 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.807 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.807 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.808 DEBUG:boto:Params: {} 2022-09-12T20:08:30.809 DEBUG:boto:Token: None 2022-09-12T20:08:30.809 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj76 2022-09-12T20:08:30.810 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xU6ZDrVaD1+Qq5SPLSgInXOdHa4= 2022-09-12T20:08:30.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xU6ZDrVaD1+Qq5SPLSgInXOdHa4=', 'Content-Length': '0'} 2022-09-12T20:08:30.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb68956530b1f3db-00631f91be-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.813 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:30.814 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:30.814 DEBUG:boto:Method: GET 2022-09-12T20:08:30.815 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:08:30.815 DEBUG:boto:Data: 2022-09-12T20:08:30.816 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.816 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.817 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.817 DEBUG:boto:Params: {} 2022-09-12T20:08:30.818 DEBUG:boto:Token: None 2022-09-12T20:08:30.819 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj76 2022-09-12T20:08:30.819 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xU6ZDrVaD1+Qq5SPLSgInXOdHa4= 2022-09-12T20:08:30.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xU6ZDrVaD1+Qq5SPLSgInXOdHa4=', 'Content-Length': '0'} 2022-09-12T20:08:30.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ec4c238b443c119-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.866 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:30.867 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:30.867 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:08:30.868 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:08:30.869 DEBUG:boto:Method: GET 2022-09-12T20:08:30.869 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:08:30.870 DEBUG:boto:Data: 2022-09-12T20:08:30.870 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.871 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.871 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.872 DEBUG:boto:Params: {} 2022-09-12T20:08:30.872 DEBUG:boto:Token: None 2022-09-12T20:08:30.873 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/ 2022-09-12T20:08:30.874 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI= 2022-09-12T20:08:30.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI=', 'Content-Length': '0'} 2022-09-12T20:08:30.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3374af9d95d0368-00631f91be-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.878 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:30.880 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:30.880 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:30.881 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:08:30.882 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:08:30.883 DEBUG:boto:Method: GET 2022-09-12T20:08:30.884 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:08:30.884 DEBUG:boto:Data: 2022-09-12T20:08:30.885 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.886 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.887 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.888 DEBUG:boto:Params: {} 2022-09-12T20:08:30.889 DEBUG:boto:Token: None 2022-09-12T20:08:30.889 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/ 2022-09-12T20:08:30.890 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI= 2022-09-12T20:08:30.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI=', 'Content-Length': '0'} 2022-09-12T20:08:30.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d24afb53284b872f-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.896 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:30.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:08:30.898 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:30.899 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:30.900 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:30.900 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:30.901 DEBUG:boto:Method: GET 2022-09-12T20:08:30.902 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:30.903 DEBUG:boto:Data: 2022-09-12T20:08:30.904 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.905 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.905 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.906 DEBUG:boto:Params: {} 2022-09-12T20:08:30.907 DEBUG:boto:Token: None 2022-09-12T20:08:30.908 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:30.909 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YVwDZV33I7R2DJ9fK2XRTLrv8Rw= 2022-09-12T20:08:30.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YVwDZV33I7R2DJ9fK2XRTLrv8Rw=', 'Content-Length': '0'} 2022-09-12T20:08:30.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5a2dbaeb7808307-00631f91be-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.913 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:30.914 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:30.915 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:30.915 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:30.916 DEBUG:boto:Method: GET 2022-09-12T20:08:30.916 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:30.917 DEBUG:boto:Data: 2022-09-12T20:08:30.917 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.918 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.918 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.919 DEBUG:boto:Params: {} 2022-09-12T20:08:30.919 DEBUG:boto:Token: None 2022-09-12T20:08:30.920 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:30.921 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YVwDZV33I7R2DJ9fK2XRTLrv8Rw= 2022-09-12T20:08:30.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YVwDZV33I7R2DJ9fK2XRTLrv8Rw=', 'Content-Length': '0'} 2022-09-12T20:08:30.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000618a691d85ce2752-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.924 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:30.925 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:30.925 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:30.926 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:30.926 DEBUG:boto:Method: HEAD 2022-09-12T20:08:30.927 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:30.927 DEBUG:boto:Data: 2022-09-12T20:08:30.928 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.928 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.929 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.930 DEBUG:boto:Params: {} 2022-09-12T20:08:30.930 DEBUG:boto:Token: None 2022-09-12T20:08:30.931 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:30.931 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Wl/S3Nm1FgSu0LDIDyQ5pODnXP4= 2022-09-12T20:08:30.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Wl/S3Nm1FgSu0LDIDyQ5pODnXP4=', 'Content-Length': '0'} 2022-09-12T20:08:30.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000392b904ed170e1c6-00631f91be-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.935 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:30.935 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:30.936 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:30.936 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:30.937 DEBUG:boto:Method: HEAD 2022-09-12T20:08:30.937 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:30.938 DEBUG:boto:Data: 2022-09-12T20:08:30.938 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.939 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.939 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.940 DEBUG:boto:Params: {} 2022-09-12T20:08:30.940 DEBUG:boto:Token: None 2022-09-12T20:08:30.941 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:30.941 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Wl/S3Nm1FgSu0LDIDyQ5pODnXP4= 2022-09-12T20:08:30.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Wl/S3Nm1FgSu0LDIDyQ5pODnXP4=', 'Content-Length': '0'} 2022-09-12T20:08:30.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5c6d041c80dc751-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:08:30.945 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:30.946 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:30.946 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:30.947 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:30.948 DEBUG:boto:Method: GET 2022-09-12T20:08:30.948 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:30.949 DEBUG:boto:Data: 2022-09-12T20:08:30.949 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.950 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.950 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.951 DEBUG:boto:Params: {} 2022-09-12T20:08:30.951 DEBUG:boto:Token: None 2022-09-12T20:08:30.952 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:30.953 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YVwDZV33I7R2DJ9fK2XRTLrv8Rw= 2022-09-12T20:08:30.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YVwDZV33I7R2DJ9fK2XRTLrv8Rw=', 'Content-Length': '0'} 2022-09-12T20:08:30.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092a399761ad42010-00631f91be-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.956 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:30.957 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:30.957 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:30.958 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:30.958 DEBUG:boto:Method: GET 2022-09-12T20:08:30.959 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:30.959 DEBUG:boto:Data: 2022-09-12T20:08:30.960 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.960 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.961 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.961 DEBUG:boto:Params: {} 2022-09-12T20:08:30.962 DEBUG:boto:Token: None 2022-09-12T20:08:30.963 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:30.963 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YVwDZV33I7R2DJ9fK2XRTLrv8Rw= 2022-09-12T20:08:30.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YVwDZV33I7R2DJ9fK2XRTLrv8Rw=', 'Content-Length': '0'} 2022-09-12T20:08:30.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1c86c72b3715844-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.967 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:30.968 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:30.968 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:08:30.969 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:08:30.969 DEBUG:boto:Method: GET 2022-09-12T20:08:30.970 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:08:30.970 DEBUG:boto:Data: 2022-09-12T20:08:30.971 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.971 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:30.972 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.972 DEBUG:boto:Params: {} 2022-09-12T20:08:30.973 DEBUG:boto:Token: None 2022-09-12T20:08:30.973 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/ 2022-09-12T20:08:30.974 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI= 2022-09-12T20:08:30.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI=', 'Content-Length': '0'} 2022-09-12T20:08:30.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000709f63321dd8c9c-00631f91be-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:30.978 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:30.979 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:30.979 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:30.980 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:08:30.980 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:08:30.981 DEBUG:boto:Method: GET 2022-09-12T20:08:30.981 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:08:30.982 DEBUG:boto:Data: 2022-09-12T20:08:30.982 DEBUG:boto:Headers: {} 2022-09-12T20:08:30.983 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:30.983 DEBUG:boto:Port: 8000 2022-09-12T20:08:30.984 DEBUG:boto:Params: {} 2022-09-12T20:08:30.984 DEBUG:boto:Token: None 2022-09-12T20:08:30.985 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:30 GMT /jdpiak-75/ 2022-09-12T20:08:30.986 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI= 2022-09-12T20:08:30.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:30 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LFkkLVTw5C9lai951IeobldyuoI=', 'Content-Length': '0'} 2022-09-12T20:08:30.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061f7bbf016c9d01b-00631f91be-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.034 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:31.035 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:31.036 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:31.036 DEBUG:boto:Method: HEAD 2022-09-12T20:08:31.037 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:08:31.037 DEBUG:boto:Data: 2022-09-12T20:08:31.038 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.038 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:31.039 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.039 DEBUG:boto:Params: {} 2022-09-12T20:08:31.040 DEBUG:boto:Token: None 2022-09-12T20:08:31.041 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj85 2022-09-12T20:08:31.041 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Bw8wXSoeTgFNyet1FNjf56GjIsA= 2022-09-12T20:08:31.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Bw8wXSoeTgFNyet1FNjf56GjIsA=', 'Content-Length': '0'} 2022-09-12T20:08:31.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b7c06e589621ebd-00631f91bf-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.045 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:31.045 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:31.046 DEBUG:boto:Method: HEAD 2022-09-12T20:08:31.046 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:08:31.047 DEBUG:boto:Data: 2022-09-12T20:08:31.047 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.048 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.048 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.049 DEBUG:boto:Params: {} 2022-09-12T20:08:31.049 DEBUG:boto:Token: None 2022-09-12T20:08:31.050 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj85 2022-09-12T20:08:31.051 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Bw8wXSoeTgFNyet1FNjf56GjIsA= 2022-09-12T20:08:31.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Bw8wXSoeTgFNyet1FNjf56GjIsA=', 'Content-Length': '0'} 2022-09-12T20:08:31.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000106de71f7d0a401b-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:08:31.055 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:31.055 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:31.056 DEBUG:boto:Method: GET 2022-09-12T20:08:31.056 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:08:31.057 DEBUG:boto:Data: 2022-09-12T20:08:31.057 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.058 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:31.059 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.059 DEBUG:boto:Params: {} 2022-09-12T20:08:31.060 DEBUG:boto:Token: None 2022-09-12T20:08:31.060 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj85 2022-09-12T20:08:31.061 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bbFwJOsjubzN+hRq6WwlrGDfv0k= 2022-09-12T20:08:31.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bbFwJOsjubzN+hRq6WwlrGDfv0k=', 'Content-Length': '0'} 2022-09-12T20:08:31.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cb3a431b8f47223-00631f91bf-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.064 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:31.065 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:31.066 DEBUG:boto:Method: GET 2022-09-12T20:08:31.066 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:08:31.067 DEBUG:boto:Data: 2022-09-12T20:08:31.067 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.068 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.068 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.069 DEBUG:boto:Params: {} 2022-09-12T20:08:31.069 DEBUG:boto:Token: None 2022-09-12T20:08:31.070 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj85 2022-09-12T20:08:31.071 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bbFwJOsjubzN+hRq6WwlrGDfv0k= 2022-09-12T20:08:31.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bbFwJOsjubzN+hRq6WwlrGDfv0k=', 'Content-Length': '0'} 2022-09-12T20:08:31.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002da86d09627c1360-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.074 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:31.075 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:31.076 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:08:31.076 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:08:31.077 DEBUG:boto:Method: GET 2022-09-12T20:08:31.077 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:08:31.078 DEBUG:boto:Data: 2022-09-12T20:08:31.078 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.079 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:31.079 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.080 DEBUG:boto:Params: {} 2022-09-12T20:08:31.080 DEBUG:boto:Token: None 2022-09-12T20:08:31.081 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/ 2022-09-12T20:08:31.082 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU= 2022-09-12T20:08:31.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU=', 'Content-Length': '0'} 2022-09-12T20:08:31.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034197e9a9b2860e5-00631f91bf-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.085 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:31.086 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:31.087 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:31.087 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:08:31.088 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:08:31.088 DEBUG:boto:Method: GET 2022-09-12T20:08:31.089 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:08:31.090 DEBUG:boto:Data: 2022-09-12T20:08:31.090 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.091 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.091 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.092 DEBUG:boto:Params: {} 2022-09-12T20:08:31.092 DEBUG:boto:Token: None 2022-09-12T20:08:31.093 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/ 2022-09-12T20:08:31.093 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU= 2022-09-12T20:08:31.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU=', 'Content-Length': '0'} 2022-09-12T20:08:31.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d5b23a1bf94a300-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.098 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:31.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:08:31.100 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:31.100 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:31.101 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:31.101 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:31.102 DEBUG:boto:Method: GET 2022-09-12T20:08:31.102 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:31.103 DEBUG:boto:Data: 2022-09-12T20:08:31.103 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.104 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:31.104 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.105 DEBUG:boto:Params: {} 2022-09-12T20:08:31.105 DEBUG:boto:Token: None 2022-09-12T20:08:31.106 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:31.107 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AoGo74l1P3QiGUSAfR0CB1GJu+U= 2022-09-12T20:08:31.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AoGo74l1P3QiGUSAfR0CB1GJu+U=', 'Content-Length': '0'} 2022-09-12T20:08:31.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046522345dbcc1150-00631f91bf-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.110 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:31.111 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:31.111 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:31.112 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:31.112 DEBUG:boto:Method: GET 2022-09-12T20:08:31.113 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:31.114 DEBUG:boto:Data: 2022-09-12T20:08:31.114 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.115 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.115 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.116 DEBUG:boto:Params: {} 2022-09-12T20:08:31.116 DEBUG:boto:Token: None 2022-09-12T20:08:31.117 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:31.117 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AoGo74l1P3QiGUSAfR0CB1GJu+U= 2022-09-12T20:08:31.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AoGo74l1P3QiGUSAfR0CB1GJu+U=', 'Content-Length': '0'} 2022-09-12T20:08:31.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000feac8d3b6a2a1d10-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.121 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:31.122 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:31.122 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:31.123 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:31.123 DEBUG:boto:Method: HEAD 2022-09-12T20:08:31.124 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:31.124 DEBUG:boto:Data: 2022-09-12T20:08:31.125 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.125 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:31.126 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.126 DEBUG:boto:Params: {} 2022-09-12T20:08:31.127 DEBUG:boto:Token: None 2022-09-12T20:08:31.127 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:31.128 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t/dJCy8C2D3odprUpGfVFKTWywI= 2022-09-12T20:08:31.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t/dJCy8C2D3odprUpGfVFKTWywI=', 'Content-Length': '0'} 2022-09-12T20:08:31.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b804a24810648ac1-00631f91bf-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.131 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:31.132 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:31.132 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:31.133 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:31.133 DEBUG:boto:Method: HEAD 2022-09-12T20:08:31.134 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:31.134 DEBUG:boto:Data: 2022-09-12T20:08:31.135 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.135 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.136 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.136 DEBUG:boto:Params: {} 2022-09-12T20:08:31.137 DEBUG:boto:Token: None 2022-09-12T20:08:31.137 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:31.138 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t/dJCy8C2D3odprUpGfVFKTWywI= 2022-09-12T20:08:31.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t/dJCy8C2D3odprUpGfVFKTWywI=', 'Content-Length': '0'} 2022-09-12T20:08:31.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3b7d4ff252567f4-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:08:31.142 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:31.142 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:31.143 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:31.143 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:31.144 DEBUG:boto:Method: GET 2022-09-12T20:08:31.144 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:31.145 DEBUG:boto:Data: 2022-09-12T20:08:31.145 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.146 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:31.147 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.147 DEBUG:boto:Params: {} 2022-09-12T20:08:31.148 DEBUG:boto:Token: None 2022-09-12T20:08:31.148 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:31.149 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AoGo74l1P3QiGUSAfR0CB1GJu+U= 2022-09-12T20:08:31.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AoGo74l1P3QiGUSAfR0CB1GJu+U=', 'Content-Length': '0'} 2022-09-12T20:08:31.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003398167b3d786a6b-00631f91bf-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.194 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:31.195 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:31.195 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:31.196 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:31.196 DEBUG:boto:Method: GET 2022-09-12T20:08:31.197 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:31.197 DEBUG:boto:Data: 2022-09-12T20:08:31.198 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.199 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.199 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.199 DEBUG:boto:Params: {} 2022-09-12T20:08:31.200 DEBUG:boto:Token: None 2022-09-12T20:08:31.201 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:31.201 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AoGo74l1P3QiGUSAfR0CB1GJu+U= 2022-09-12T20:08:31.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AoGo74l1P3QiGUSAfR0CB1GJu+U=', 'Content-Length': '0'} 2022-09-12T20:08:31.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000856ae99746721f0a-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.205 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:31.206 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:31.206 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:08:31.220 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:08:31.221 DEBUG:boto:Method: GET 2022-09-12T20:08:31.221 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:08:31.222 DEBUG:boto:Data: 2022-09-12T20:08:31.222 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.223 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:31.223 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.224 DEBUG:boto:Params: {} 2022-09-12T20:08:31.224 DEBUG:boto:Token: None 2022-09-12T20:08:31.225 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/ 2022-09-12T20:08:31.225 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU= 2022-09-12T20:08:31.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU=', 'Content-Length': '0'} 2022-09-12T20:08:31.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e063c01be82752bf-00631f91bf-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.230 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:31.230 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:31.231 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:31.231 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:08:31.232 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:08:31.232 DEBUG:boto:Method: GET 2022-09-12T20:08:31.233 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:08:31.233 DEBUG:boto:Data: 2022-09-12T20:08:31.234 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.235 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.235 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.236 DEBUG:boto:Params: {} 2022-09-12T20:08:31.236 DEBUG:boto:Token: None 2022-09-12T20:08:31.237 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/ 2022-09-12T20:08:31.238 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU= 2022-09-12T20:08:31.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU=', 'Content-Length': '0'} 2022-09-12T20:08:31.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af8df76be4a88a7b-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.242 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:31.243 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:31.244 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:31.244 DEBUG:boto:Method: HEAD 2022-09-12T20:08:31.245 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:08:31.245 DEBUG:boto:Data: 2022-09-12T20:08:31.246 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.246 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:31.247 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.247 DEBUG:boto:Params: {} 2022-09-12T20:08:31.248 DEBUG:boto:Token: None 2022-09-12T20:08:31.249 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj95 2022-09-12T20:08:31.249 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ne3ODgpBs30Sirkpec/IbGs3l0w= 2022-09-12T20:08:31.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ne3ODgpBs30Sirkpec/IbGs3l0w=', 'Content-Length': '0'} 2022-09-12T20:08:31.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002286dcd1fdfbf16a-00631f91bf-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.253 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:31.253 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:31.254 DEBUG:boto:Method: HEAD 2022-09-12T20:08:31.254 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:08:31.255 DEBUG:boto:Data: 2022-09-12T20:08:31.255 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.256 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.256 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.257 DEBUG:boto:Params: {} 2022-09-12T20:08:31.257 DEBUG:boto:Token: None 2022-09-12T20:08:31.258 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj95 2022-09-12T20:08:31.259 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ne3ODgpBs30Sirkpec/IbGs3l0w= 2022-09-12T20:08:31.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ne3ODgpBs30Sirkpec/IbGs3l0w=', 'Content-Length': '0'} 2022-09-12T20:08:31.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000635c0f81548d2c72-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:08:31.264 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:31.265 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:31.266 DEBUG:boto:Method: GET 2022-09-12T20:08:31.267 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:08:31.268 DEBUG:boto:Data: 2022-09-12T20:08:31.268 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.269 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:31.270 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.271 DEBUG:boto:Params: {} 2022-09-12T20:08:31.272 DEBUG:boto:Token: None 2022-09-12T20:08:31.273 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj95 2022-09-12T20:08:31.274 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jEKZr7jBqAUgAGbnOLlAe76NtUs= 2022-09-12T20:08:31.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jEKZr7jBqAUgAGbnOLlAe76NtUs=', 'Content-Length': '0'} 2022-09-12T20:08:31.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7e921de35bc13c6-00631f91bf-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.279 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:31.280 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:31.280 DEBUG:boto:Method: GET 2022-09-12T20:08:31.281 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:08:31.282 DEBUG:boto:Data: 2022-09-12T20:08:31.283 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.284 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.284 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.285 DEBUG:boto:Params: {} 2022-09-12T20:08:31.286 DEBUG:boto:Token: None 2022-09-12T20:08:31.287 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj95 2022-09-12T20:08:31.288 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jEKZr7jBqAUgAGbnOLlAe76NtUs= 2022-09-12T20:08:31.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jEKZr7jBqAUgAGbnOLlAe76NtUs=', 'Content-Length': '0'} 2022-09-12T20:08:31.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003492ae25c1848235-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.293 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:31.294 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:31.295 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:08:31.295 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:08:31.296 DEBUG:boto:Method: GET 2022-09-12T20:08:31.297 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:08:31.298 DEBUG:boto:Data: 2022-09-12T20:08:31.299 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.300 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:31.301 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.301 DEBUG:boto:Params: {} 2022-09-12T20:08:31.302 DEBUG:boto:Token: None 2022-09-12T20:08:31.303 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/ 2022-09-12T20:08:31.304 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU= 2022-09-12T20:08:31.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU=', 'Content-Length': '0'} 2022-09-12T20:08:31.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071bb5e244157a0f4-00631f91bf-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.309 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:31.310 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:31.311 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:31.312 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:08:31.313 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:08:31.314 DEBUG:boto:Method: GET 2022-09-12T20:08:31.315 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:08:31.315 DEBUG:boto:Data: 2022-09-12T20:08:31.316 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.317 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.318 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.319 DEBUG:boto:Params: {} 2022-09-12T20:08:31.320 DEBUG:boto:Token: None 2022-09-12T20:08:31.321 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/ 2022-09-12T20:08:31.322 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU= 2022-09-12T20:08:31.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU=', 'Content-Length': '0'} 2022-09-12T20:08:31.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070617b545e4b9f71-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.327 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:31.328 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-75 zones={test-zone2, test-zone1} 2022-09-12T20:08:31.329 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-75 zones={test-zone3, test-zone1} 2022-09-12T20:08:31.330 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:31.330 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:31.331 DEBUG:boto:Method: HEAD 2022-09-12T20:08:31.331 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:08:31.332 DEBUG:boto:Data: 2022-09-12T20:08:31.332 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.333 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:31.333 DEBUG:boto:Port: 8001 2022-09-12T20:08:31.334 DEBUG:boto:Params: {} 2022-09-12T20:08:31.335 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T20:08:31.335 DEBUG:boto:Token: None 2022-09-12T20:08:31.336 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/ 2022-09-12T20:08:31.337 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V/Kz00zNFEheV1lhNDwz8L1qc3g= 2022-09-12T20:08:31.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V/Kz00zNFEheV1lhNDwz8L1qc3g=', 'Content-Length': '0'} 2022-09-12T20:08:31.341 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000070272553736c3feb-00631f91bf-1826-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.342 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:31.342 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:31.343 DEBUG:boto:Method: HEAD 2022-09-12T20:08:31.343 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:08:31.344 DEBUG:boto:Data: 2022-09-12T20:08:31.344 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.345 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.345 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.346 DEBUG:boto:Params: {} 2022-09-12T20:08:31.346 DEBUG:boto:Token: None 2022-09-12T20:08:31.347 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/ 2022-09-12T20:08:31.348 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V/Kz00zNFEheV1lhNDwz8L1qc3g= 2022-09-12T20:08:31.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V/Kz00zNFEheV1lhNDwz8L1qc3g=', 'Content-Length': '0'} 2022-09-12T20:08:31.352 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ff6458d22c9a0f7-00631f91bf-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.353 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T20:08:31.353 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:31.354 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:31.354 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:31.355 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:31.355 DEBUG:boto:Method: GET 2022-09-12T20:08:31.356 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:31.356 DEBUG:boto:Data: 2022-09-12T20:08:31.357 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.357 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:31.358 DEBUG:boto:Port: 8001 2022-09-12T20:08:31.358 DEBUG:boto:Params: {} 2022-09-12T20:08:31.359 DEBUG:boto:Token: None 2022-09-12T20:08:31.359 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/?versions 2022-09-12T20:08:31.360 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OhnbPzdqbVomN5dY6YRKqrT6kc4= 2022-09-12T20:08:31.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OhnbPzdqbVomN5dY6YRKqrT6kc4=', 'Content-Length': '0'} 2022-09-12T20:08:31.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee3f7e26dbec32ca-00631f91bf-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.364 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:31.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:08:31.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:08:31.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:08:31.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:08:31.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:08:31.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:08:31.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:08:31.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:08:31.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:08:31.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:08:31.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:08:31.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:08:31.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:08:31.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:08:31.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:08:31.373 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T20:08:31.373 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:31.374 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:31.374 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:31.375 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:31.375 DEBUG:boto:Method: GET 2022-09-12T20:08:31.376 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:31.376 DEBUG:boto:Data: 2022-09-12T20:08:31.377 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.377 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.378 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.378 DEBUG:boto:Params: {} 2022-09-12T20:08:31.379 DEBUG:boto:Token: None 2022-09-12T20:08:31.379 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/?versions 2022-09-12T20:08:31.380 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OhnbPzdqbVomN5dY6YRKqrT6kc4= 2022-09-12T20:08:31.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OhnbPzdqbVomN5dY6YRKqrT6kc4=', 'Content-Length': '0'} 2022-09-12T20:08:31.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad1250d8e6ad2871-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.385 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:31.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:08:31.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:08:31.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:08:31.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:08:31.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:08:31.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:08:31.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:08:31.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:08:31.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:08:31.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:08:31.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:08:31.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:08:31.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:08:31.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:08:31.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:08:31.394 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:31.394 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:31.395 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:31.395 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:31.396 DEBUG:boto:Method: GET 2022-09-12T20:08:31.396 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:31.397 DEBUG:boto:Data: 2022-09-12T20:08:31.397 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.398 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:31.398 DEBUG:boto:Port: 8001 2022-09-12T20:08:31.399 DEBUG:boto:Params: {} 2022-09-12T20:08:31.400 DEBUG:boto:Token: None 2022-09-12T20:08:31.400 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/?versions 2022-09-12T20:08:31.401 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OhnbPzdqbVomN5dY6YRKqrT6kc4= 2022-09-12T20:08:31.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OhnbPzdqbVomN5dY6YRKqrT6kc4=', 'Content-Length': '0'} 2022-09-12T20:08:31.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000431641a848093df8-00631f91bf-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.405 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:31.406 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:31.407 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:31.407 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:31.408 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:31.408 DEBUG:boto:Method: GET 2022-09-12T20:08:31.409 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:31.409 DEBUG:boto:Data: 2022-09-12T20:08:31.410 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.410 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.411 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.411 DEBUG:boto:Params: {} 2022-09-12T20:08:31.412 DEBUG:boto:Token: None 2022-09-12T20:08:31.413 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/?versions 2022-09-12T20:08:31.413 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OhnbPzdqbVomN5dY6YRKqrT6kc4= 2022-09-12T20:08:31.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OhnbPzdqbVomN5dY6YRKqrT6kc4=', 'Content-Length': '0'} 2022-09-12T20:08:31.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e179e2ee8c77fc1-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.418 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:31.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:08:31.420 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:31.421 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:31.421 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:31.422 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:31.422 DEBUG:boto:Method: GET 2022-09-12T20:08:31.423 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:31.424 DEBUG:boto:Data: 2022-09-12T20:08:31.424 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.425 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:31.425 DEBUG:boto:Port: 8001 2022-09-12T20:08:31.426 DEBUG:boto:Params: {} 2022-09-12T20:08:31.426 DEBUG:boto:Token: None 2022-09-12T20:08:31.427 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:31.427 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EEY/lie9V8ceMMXGA6PRebwpniM= 2022-09-12T20:08:31.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EEY/lie9V8ceMMXGA6PRebwpniM=', 'Content-Length': '0'} 2022-09-12T20:08:31.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000794a068ff0c35a47-00631f91bf-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.431 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:31.432 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:31.432 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:31.433 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:31.433 DEBUG:boto:Method: GET 2022-09-12T20:08:31.434 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:31.434 DEBUG:boto:Data: 2022-09-12T20:08:31.435 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.435 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.436 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.437 DEBUG:boto:Params: {} 2022-09-12T20:08:31.437 DEBUG:boto:Token: None 2022-09-12T20:08:31.438 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:31.438 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EEY/lie9V8ceMMXGA6PRebwpniM= 2022-09-12T20:08:31.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EEY/lie9V8ceMMXGA6PRebwpniM=', 'Content-Length': '0'} 2022-09-12T20:08:31.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054f455e21694d710-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.482 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:31.483 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:31.483 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:31.484 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:31.484 DEBUG:boto:Method: HEAD 2022-09-12T20:08:31.485 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:31.485 DEBUG:boto:Data: 2022-09-12T20:08:31.485 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.486 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:31.487 DEBUG:boto:Port: 8001 2022-09-12T20:08:31.487 DEBUG:boto:Params: {} 2022-09-12T20:08:31.488 DEBUG:boto:Token: None 2022-09-12T20:08:31.488 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:31.489 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A5TtcAEaMzL6xs00+wH07pXf/tw= 2022-09-12T20:08:31.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A5TtcAEaMzL6xs00+wH07pXf/tw=', 'Content-Length': '0'} 2022-09-12T20:08:31.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1cdbb58ac5934a6-00631f91bf-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.492 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:31.493 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:31.493 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:31.494 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:31.494 DEBUG:boto:Method: HEAD 2022-09-12T20:08:31.495 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:31.495 DEBUG:boto:Data: 2022-09-12T20:08:31.496 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.496 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.497 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.497 DEBUG:boto:Params: {} 2022-09-12T20:08:31.498 DEBUG:boto:Token: None 2022-09-12T20:08:31.498 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:31.499 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A5TtcAEaMzL6xs00+wH07pXf/tw= 2022-09-12T20:08:31.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A5TtcAEaMzL6xs00+wH07pXf/tw=', 'Content-Length': '0'} 2022-09-12T20:08:31.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001693bcbfa04ae6a9-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:08:31.503 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:31.504 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:31.504 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:31.505 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:31.505 DEBUG:boto:Method: GET 2022-09-12T20:08:31.506 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:31.506 DEBUG:boto:Data: 2022-09-12T20:08:31.507 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.507 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:31.508 DEBUG:boto:Port: 8001 2022-09-12T20:08:31.508 DEBUG:boto:Params: {} 2022-09-12T20:08:31.509 DEBUG:boto:Token: None 2022-09-12T20:08:31.509 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:31.510 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EEY/lie9V8ceMMXGA6PRebwpniM= 2022-09-12T20:08:31.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EEY/lie9V8ceMMXGA6PRebwpniM=', 'Content-Length': '0'} 2022-09-12T20:08:31.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa2bf788d0647b59-00631f91bf-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.513 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:31.514 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:31.515 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:31.515 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:31.516 DEBUG:boto:Method: GET 2022-09-12T20:08:31.516 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:31.517 DEBUG:boto:Data: 2022-09-12T20:08:31.517 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.518 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.518 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.519 DEBUG:boto:Params: {} 2022-09-12T20:08:31.519 DEBUG:boto:Token: None 2022-09-12T20:08:31.520 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:31.520 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EEY/lie9V8ceMMXGA6PRebwpniM= 2022-09-12T20:08:31.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EEY/lie9V8ceMMXGA6PRebwpniM=', 'Content-Length': '0'} 2022-09-12T20:08:31.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007be5672b341cdc67-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.524 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:31.525 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:31.526 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:08:31.526 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:08:31.527 DEBUG:boto:Method: GET 2022-09-12T20:08:31.527 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:08:31.528 DEBUG:boto:Data: 2022-09-12T20:08:31.528 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.529 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:31.529 DEBUG:boto:Port: 8001 2022-09-12T20:08:31.530 DEBUG:boto:Params: {} 2022-09-12T20:08:31.530 DEBUG:boto:Token: None 2022-09-12T20:08:31.531 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/ 2022-09-12T20:08:31.532 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU= 2022-09-12T20:08:31.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU=', 'Content-Length': '0'} 2022-09-12T20:08:31.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a29a45a3f25e946-00631f91bf-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.536 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:31.537 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:31.537 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:31.538 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:08:31.538 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:08:31.539 DEBUG:boto:Method: GET 2022-09-12T20:08:31.540 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:08:31.540 DEBUG:boto:Data: 2022-09-12T20:08:31.541 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.541 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.542 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.542 DEBUG:boto:Params: {} 2022-09-12T20:08:31.543 DEBUG:boto:Token: None 2022-09-12T20:08:31.543 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/ 2022-09-12T20:08:31.544 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU= 2022-09-12T20:08:31.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU=', 'Content-Length': '0'} 2022-09-12T20:08:31.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb3e9d8f2abdcdd4-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.548 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:31.549 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:31.550 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:31.550 DEBUG:boto:Method: HEAD 2022-09-12T20:08:31.551 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:08:31.551 DEBUG:boto:Data: 2022-09-12T20:08:31.552 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.553 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:31.553 DEBUG:boto:Port: 8001 2022-09-12T20:08:31.554 DEBUG:boto:Params: {} 2022-09-12T20:08:31.554 DEBUG:boto:Token: None 2022-09-12T20:08:31.555 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj11 2022-09-12T20:08:31.555 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H3tb7hglcScwxUd2dACpMrJSEoY= 2022-09-12T20:08:31.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H3tb7hglcScwxUd2dACpMrJSEoY=', 'Content-Length': '0'} 2022-09-12T20:08:31.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c3dc496c5747744-00631f91bf-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.559 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:31.559 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:31.560 DEBUG:boto:Method: HEAD 2022-09-12T20:08:31.560 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:08:31.561 DEBUG:boto:Data: 2022-09-12T20:08:31.561 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.562 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.562 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.563 DEBUG:boto:Params: {} 2022-09-12T20:08:31.563 DEBUG:boto:Token: None 2022-09-12T20:08:31.564 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj11 2022-09-12T20:08:31.564 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H3tb7hglcScwxUd2dACpMrJSEoY= 2022-09-12T20:08:31.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H3tb7hglcScwxUd2dACpMrJSEoY=', 'Content-Length': '0'} 2022-09-12T20:08:31.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008340adc509a05c1e-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:08:31.568 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:31.569 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:31.569 DEBUG:boto:Method: GET 2022-09-12T20:08:31.570 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:08:31.570 DEBUG:boto:Data: 2022-09-12T20:08:31.571 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.571 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:31.572 DEBUG:boto:Port: 8001 2022-09-12T20:08:31.572 DEBUG:boto:Params: {} 2022-09-12T20:08:31.573 DEBUG:boto:Token: None 2022-09-12T20:08:31.573 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj11 2022-09-12T20:08:31.574 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fpr4kumh+HCH98IWnCmFrz1XEMs= 2022-09-12T20:08:31.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fpr4kumh+HCH98IWnCmFrz1XEMs=', 'Content-Length': '0'} 2022-09-12T20:08:31.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031e0b9f96a212514-00631f91bf-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.578 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:31.578 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:31.579 DEBUG:boto:Method: GET 2022-09-12T20:08:31.579 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:08:31.580 DEBUG:boto:Data: 2022-09-12T20:08:31.581 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.581 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.582 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.582 DEBUG:boto:Params: {} 2022-09-12T20:08:31.583 DEBUG:boto:Token: None 2022-09-12T20:08:31.583 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj11 2022-09-12T20:08:31.584 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fpr4kumh+HCH98IWnCmFrz1XEMs= 2022-09-12T20:08:31.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fpr4kumh+HCH98IWnCmFrz1XEMs=', 'Content-Length': '0'} 2022-09-12T20:08:31.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1f6616565118983-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.630 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:31.631 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:31.631 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:08:31.632 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:08:31.632 DEBUG:boto:Method: GET 2022-09-12T20:08:31.633 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:08:31.633 DEBUG:boto:Data: 2022-09-12T20:08:31.634 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.634 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:31.635 DEBUG:boto:Port: 8001 2022-09-12T20:08:31.636 DEBUG:boto:Params: {} 2022-09-12T20:08:31.636 DEBUG:boto:Token: None 2022-09-12T20:08:31.637 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/ 2022-09-12T20:08:31.637 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU= 2022-09-12T20:08:31.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU=', 'Content-Length': '0'} 2022-09-12T20:08:31.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000250a60fa04a09f8e-00631f91bf-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.642 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:31.642 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:31.643 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:31.644 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:08:31.644 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:08:31.645 DEBUG:boto:Method: GET 2022-09-12T20:08:31.645 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:08:31.646 DEBUG:boto:Data: 2022-09-12T20:08:31.646 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.647 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.647 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.648 DEBUG:boto:Params: {} 2022-09-12T20:08:31.648 DEBUG:boto:Token: None 2022-09-12T20:08:31.649 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/ 2022-09-12T20:08:31.650 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU= 2022-09-12T20:08:31.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU=', 'Content-Length': '0'} 2022-09-12T20:08:31.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000926fb24dfc89d64b-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.654 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:31.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:08:31.656 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:31.656 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:31.657 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:31.657 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:31.658 DEBUG:boto:Method: GET 2022-09-12T20:08:31.659 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:31.659 DEBUG:boto:Data: 2022-09-12T20:08:31.660 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.660 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:31.661 DEBUG:boto:Port: 8001 2022-09-12T20:08:31.661 DEBUG:boto:Params: {} 2022-09-12T20:08:31.662 DEBUG:boto:Token: None 2022-09-12T20:08:31.662 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:31.663 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:47IptNd/UyETyh06YoMiFwQteFY= 2022-09-12T20:08:31.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:47IptNd/UyETyh06YoMiFwQteFY=', 'Content-Length': '0'} 2022-09-12T20:08:31.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df1a48daf0e495a3-00631f91bf-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.667 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:31.667 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:31.668 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:31.668 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:31.669 DEBUG:boto:Method: GET 2022-09-12T20:08:31.669 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:31.670 DEBUG:boto:Data: 2022-09-12T20:08:31.670 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.671 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.672 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.672 DEBUG:boto:Params: {} 2022-09-12T20:08:31.673 DEBUG:boto:Token: None 2022-09-12T20:08:31.673 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:31.674 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:47IptNd/UyETyh06YoMiFwQteFY= 2022-09-12T20:08:31.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:47IptNd/UyETyh06YoMiFwQteFY=', 'Content-Length': '0'} 2022-09-12T20:08:31.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031507c5e70e904b2-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.677 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:31.678 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:31.678 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:31.679 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:31.679 DEBUG:boto:Method: HEAD 2022-09-12T20:08:31.680 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:31.680 DEBUG:boto:Data: 2022-09-12T20:08:31.681 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.681 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:31.682 DEBUG:boto:Port: 8001 2022-09-12T20:08:31.682 DEBUG:boto:Params: {} 2022-09-12T20:08:31.683 DEBUG:boto:Token: None 2022-09-12T20:08:31.683 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:31.684 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R8Gr/HBE2i9WwLoyY0pJzHTwpxw= 2022-09-12T20:08:31.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R8Gr/HBE2i9WwLoyY0pJzHTwpxw=', 'Content-Length': '0'} 2022-09-12T20:08:31.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afdb51d79049b05d-00631f91bf-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.687 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:31.688 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:31.688 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:31.689 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:31.689 DEBUG:boto:Method: HEAD 2022-09-12T20:08:31.690 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:31.690 DEBUG:boto:Data: 2022-09-12T20:08:31.691 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.691 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.692 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.692 DEBUG:boto:Params: {} 2022-09-12T20:08:31.693 DEBUG:boto:Token: None 2022-09-12T20:08:31.694 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:31.694 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R8Gr/HBE2i9WwLoyY0pJzHTwpxw= 2022-09-12T20:08:31.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R8Gr/HBE2i9WwLoyY0pJzHTwpxw=', 'Content-Length': '0'} 2022-09-12T20:08:31.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be696d6ffd7cc2e0-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:08:31.698 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:31.699 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:31.699 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:31.700 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:31.700 DEBUG:boto:Method: GET 2022-09-12T20:08:31.701 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:31.701 DEBUG:boto:Data: 2022-09-12T20:08:31.702 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.702 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:31.703 DEBUG:boto:Port: 8001 2022-09-12T20:08:31.703 DEBUG:boto:Params: {} 2022-09-12T20:08:31.704 DEBUG:boto:Token: None 2022-09-12T20:08:31.704 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:31.705 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:47IptNd/UyETyh06YoMiFwQteFY= 2022-09-12T20:08:31.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:47IptNd/UyETyh06YoMiFwQteFY=', 'Content-Length': '0'} 2022-09-12T20:08:31.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f89cf733afe2e22e-00631f91bf-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.750 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:31.751 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:31.751 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:31.752 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:31.753 DEBUG:boto:Method: GET 2022-09-12T20:08:31.753 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:31.754 DEBUG:boto:Data: 2022-09-12T20:08:31.754 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.755 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.755 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.756 DEBUG:boto:Params: {} 2022-09-12T20:08:31.756 DEBUG:boto:Token: None 2022-09-12T20:08:31.757 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:31.758 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:47IptNd/UyETyh06YoMiFwQteFY= 2022-09-12T20:08:31.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:47IptNd/UyETyh06YoMiFwQteFY=', 'Content-Length': '0'} 2022-09-12T20:08:31.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000913a3ba2c75ccb9-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.761 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:31.762 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:31.762 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:08:31.763 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:08:31.763 DEBUG:boto:Method: GET 2022-09-12T20:08:31.764 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:08:31.764 DEBUG:boto:Data: 2022-09-12T20:08:31.765 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.765 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:31.766 DEBUG:boto:Port: 8001 2022-09-12T20:08:31.767 DEBUG:boto:Params: {} 2022-09-12T20:08:31.767 DEBUG:boto:Token: None 2022-09-12T20:08:31.768 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/ 2022-09-12T20:08:31.768 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU= 2022-09-12T20:08:31.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU=', 'Content-Length': '0'} 2022-09-12T20:08:31.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a435d515839c4e57-00631f91bf-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.773 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:31.774 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:31.774 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:31.775 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:08:31.775 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:08:31.776 DEBUG:boto:Method: GET 2022-09-12T20:08:31.776 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:08:31.777 DEBUG:boto:Data: 2022-09-12T20:08:31.777 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.778 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.778 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.779 DEBUG:boto:Params: {} 2022-09-12T20:08:31.780 DEBUG:boto:Token: None 2022-09-12T20:08:31.780 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/ 2022-09-12T20:08:31.781 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU= 2022-09-12T20:08:31.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU=', 'Content-Length': '0'} 2022-09-12T20:08:31.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006501df8b27f06f07-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.786 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:31.787 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:31.787 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:31.788 DEBUG:boto:Method: HEAD 2022-09-12T20:08:31.788 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:08:31.789 DEBUG:boto:Data: 2022-09-12T20:08:31.789 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.790 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:31.790 DEBUG:boto:Port: 8001 2022-09-12T20:08:31.791 DEBUG:boto:Params: {} 2022-09-12T20:08:31.791 DEBUG:boto:Token: None 2022-09-12T20:08:31.792 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj17 2022-09-12T20:08:31.792 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F7N9AGnJ+87u+mFTqOfzOLIhJm8= 2022-09-12T20:08:31.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F7N9AGnJ+87u+mFTqOfzOLIhJm8=', 'Content-Length': '0'} 2022-09-12T20:08:31.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d45aaf89ca0dc633-00631f91bf-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.796 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:31.797 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:31.797 DEBUG:boto:Method: HEAD 2022-09-12T20:08:31.798 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:08:31.799 DEBUG:boto:Data: 2022-09-12T20:08:31.799 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.800 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.800 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.801 DEBUG:boto:Params: {} 2022-09-12T20:08:31.801 DEBUG:boto:Token: None 2022-09-12T20:08:31.802 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj17 2022-09-12T20:08:31.802 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F7N9AGnJ+87u+mFTqOfzOLIhJm8= 2022-09-12T20:08:31.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F7N9AGnJ+87u+mFTqOfzOLIhJm8=', 'Content-Length': '0'} 2022-09-12T20:08:31.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1663e4ff9736438-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:08:31.806 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:31.807 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:31.807 DEBUG:boto:Method: GET 2022-09-12T20:08:31.808 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:08:31.808 DEBUG:boto:Data: 2022-09-12T20:08:31.809 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.810 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:31.810 DEBUG:boto:Port: 8001 2022-09-12T20:08:31.811 DEBUG:boto:Params: {} 2022-09-12T20:08:31.811 DEBUG:boto:Token: None 2022-09-12T20:08:31.812 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj17 2022-09-12T20:08:31.812 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BOj9emz2ADUcRj3eoPmVr3wO6Qk= 2022-09-12T20:08:31.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BOj9emz2ADUcRj3eoPmVr3wO6Qk=', 'Content-Length': '0'} 2022-09-12T20:08:31.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090ec4e5b5069d925-00631f91bf-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.816 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:31.817 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:31.817 DEBUG:boto:Method: GET 2022-09-12T20:08:31.818 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:08:31.818 DEBUG:boto:Data: 2022-09-12T20:08:31.819 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.820 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.820 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.821 DEBUG:boto:Params: {} 2022-09-12T20:08:31.821 DEBUG:boto:Token: None 2022-09-12T20:08:31.822 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj17 2022-09-12T20:08:31.822 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BOj9emz2ADUcRj3eoPmVr3wO6Qk= 2022-09-12T20:08:31.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BOj9emz2ADUcRj3eoPmVr3wO6Qk=', 'Content-Length': '0'} 2022-09-12T20:08:31.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000290aaf5454d0e19-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.826 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:31.827 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:31.827 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:08:31.828 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:08:31.828 DEBUG:boto:Method: GET 2022-09-12T20:08:31.829 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:08:31.829 DEBUG:boto:Data: 2022-09-12T20:08:31.830 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.830 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:31.831 DEBUG:boto:Port: 8001 2022-09-12T20:08:31.831 DEBUG:boto:Params: {} 2022-09-12T20:08:31.832 DEBUG:boto:Token: None 2022-09-12T20:08:31.832 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/ 2022-09-12T20:08:31.833 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU= 2022-09-12T20:08:31.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU=', 'Content-Length': '0'} 2022-09-12T20:08:31.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9b6d9354004fc3e-00631f91bf-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.837 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:31.838 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:31.839 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:31.839 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:08:31.840 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:08:31.840 DEBUG:boto:Method: GET 2022-09-12T20:08:31.841 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:08:31.841 DEBUG:boto:Data: 2022-09-12T20:08:31.842 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.843 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.843 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.844 DEBUG:boto:Params: {} 2022-09-12T20:08:31.844 DEBUG:boto:Token: None 2022-09-12T20:08:31.845 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/ 2022-09-12T20:08:31.845 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU= 2022-09-12T20:08:31.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU=', 'Content-Length': '0'} 2022-09-12T20:08:31.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000866864ac45b37320-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.850 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:31.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:08:31.851 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:31.852 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:31.852 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:31.853 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:31.853 DEBUG:boto:Method: GET 2022-09-12T20:08:31.854 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:31.854 DEBUG:boto:Data: 2022-09-12T20:08:31.855 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.855 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:31.856 DEBUG:boto:Port: 8001 2022-09-12T20:08:31.856 DEBUG:boto:Params: {} 2022-09-12T20:08:31.857 DEBUG:boto:Token: None 2022-09-12T20:08:31.858 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:31.858 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z3ubqBse6EFsqac0rWxj51PRJx0= 2022-09-12T20:08:31.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z3ubqBse6EFsqac0rWxj51PRJx0=', 'Content-Length': '0'} 2022-09-12T20:08:31.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010b02f2e7b12c1b5-00631f91bf-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.862 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:31.863 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:31.863 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:31.864 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:31.864 DEBUG:boto:Method: GET 2022-09-12T20:08:31.865 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:31.865 DEBUG:boto:Data: 2022-09-12T20:08:31.866 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.866 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.867 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.867 DEBUG:boto:Params: {} 2022-09-12T20:08:31.868 DEBUG:boto:Token: None 2022-09-12T20:08:31.869 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:31.869 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z3ubqBse6EFsqac0rWxj51PRJx0= 2022-09-12T20:08:31.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z3ubqBse6EFsqac0rWxj51PRJx0=', 'Content-Length': '0'} 2022-09-12T20:08:31.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006396d2cd7779c64c-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.873 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:31.873 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:31.874 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:31.874 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:31.875 DEBUG:boto:Method: HEAD 2022-09-12T20:08:31.875 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:31.876 DEBUG:boto:Data: 2022-09-12T20:08:31.877 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.877 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:31.877 DEBUG:boto:Port: 8001 2022-09-12T20:08:31.878 DEBUG:boto:Params: {} 2022-09-12T20:08:31.879 DEBUG:boto:Token: None 2022-09-12T20:08:31.879 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:31.880 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N5iWvdsLIJbdBg8xkbbYt7fpOlI= 2022-09-12T20:08:31.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N5iWvdsLIJbdBg8xkbbYt7fpOlI=', 'Content-Length': '0'} 2022-09-12T20:08:31.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f4d9d6ca4006565-00631f91bf-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.883 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:31.884 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:31.884 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:31.885 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:31.885 DEBUG:boto:Method: HEAD 2022-09-12T20:08:31.886 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:31.886 DEBUG:boto:Data: 2022-09-12T20:08:31.887 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.887 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.888 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.888 DEBUG:boto:Params: {} 2022-09-12T20:08:31.889 DEBUG:boto:Token: None 2022-09-12T20:08:31.889 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:31.890 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N5iWvdsLIJbdBg8xkbbYt7fpOlI= 2022-09-12T20:08:31.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N5iWvdsLIJbdBg8xkbbYt7fpOlI=', 'Content-Length': '0'} 2022-09-12T20:08:31.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013f81563b8a8b10a-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:08:31.894 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:31.894 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:31.895 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:31.895 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:31.896 DEBUG:boto:Method: GET 2022-09-12T20:08:31.896 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:31.897 DEBUG:boto:Data: 2022-09-12T20:08:31.898 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.898 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:31.899 DEBUG:boto:Port: 8001 2022-09-12T20:08:31.899 DEBUG:boto:Params: {} 2022-09-12T20:08:31.900 DEBUG:boto:Token: None 2022-09-12T20:08:31.900 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:31.901 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z3ubqBse6EFsqac0rWxj51PRJx0= 2022-09-12T20:08:31.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z3ubqBse6EFsqac0rWxj51PRJx0=', 'Content-Length': '0'} 2022-09-12T20:08:31.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009610c469755d2877-00631f91bf-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.904 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:31.905 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:31.905 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:31.906 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:31.907 DEBUG:boto:Method: GET 2022-09-12T20:08:31.907 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:31.908 DEBUG:boto:Data: 2022-09-12T20:08:31.908 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.909 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.909 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.910 DEBUG:boto:Params: {} 2022-09-12T20:08:31.910 DEBUG:boto:Token: None 2022-09-12T20:08:31.911 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:31.912 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z3ubqBse6EFsqac0rWxj51PRJx0= 2022-09-12T20:08:31.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z3ubqBse6EFsqac0rWxj51PRJx0=', 'Content-Length': '0'} 2022-09-12T20:08:31.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088397e174c268eeb-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.915 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:31.916 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:31.916 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:08:31.917 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:08:31.917 DEBUG:boto:Method: GET 2022-09-12T20:08:31.918 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:08:31.918 DEBUG:boto:Data: 2022-09-12T20:08:31.919 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.919 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:31.920 DEBUG:boto:Port: 8001 2022-09-12T20:08:31.920 DEBUG:boto:Params: {} 2022-09-12T20:08:31.921 DEBUG:boto:Token: None 2022-09-12T20:08:31.922 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/ 2022-09-12T20:08:31.922 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU= 2022-09-12T20:08:31.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU=', 'Content-Length': '0'} 2022-09-12T20:08:31.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2e2c123be650bae-00631f91bf-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.926 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:31.927 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:31.928 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:31.928 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:08:31.929 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:08:31.929 DEBUG:boto:Method: GET 2022-09-12T20:08:31.930 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:08:31.930 DEBUG:boto:Data: 2022-09-12T20:08:31.931 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.931 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.932 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.932 DEBUG:boto:Params: {} 2022-09-12T20:08:31.933 DEBUG:boto:Token: None 2022-09-12T20:08:31.933 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/ 2022-09-12T20:08:31.934 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU= 2022-09-12T20:08:31.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BRhbrrGm1VUB+hnxZqmY1XtvmrU=', 'Content-Length': '0'} 2022-09-12T20:08:31.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f1e1f7068cc0996-00631f91bf-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.982 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:31.983 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:31.984 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:31.984 DEBUG:boto:Method: HEAD 2022-09-12T20:08:31.985 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:08:31.985 DEBUG:boto:Data: 2022-09-12T20:08:31.986 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.986 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:31.987 DEBUG:boto:Port: 8001 2022-09-12T20:08:31.988 DEBUG:boto:Params: {} 2022-09-12T20:08:31.988 DEBUG:boto:Token: None 2022-09-12T20:08:31.989 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj19 2022-09-12T20:08:31.989 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lxMAArmNPghutGf8/7MXEVgPUkM= 2022-09-12T20:08:31.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lxMAArmNPghutGf8/7MXEVgPUkM=', 'Content-Length': '0'} 2022-09-12T20:08:31.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e86f829c401490b-00631f91bf-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:31.993 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:31.994 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:31.994 DEBUG:boto:Method: HEAD 2022-09-12T20:08:31.995 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:08:31.995 DEBUG:boto:Data: 2022-09-12T20:08:31.996 DEBUG:boto:Headers: {} 2022-09-12T20:08:31.996 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:31.997 DEBUG:boto:Port: 8000 2022-09-12T20:08:31.997 DEBUG:boto:Params: {} 2022-09-12T20:08:31.998 DEBUG:boto:Token: None 2022-09-12T20:08:31.999 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:31 GMT /jdpiak-75/obj19 2022-09-12T20:08:31.999 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lxMAArmNPghutGf8/7MXEVgPUkM= 2022-09-12T20:08:32.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:31 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lxMAArmNPghutGf8/7MXEVgPUkM=', 'Content-Length': '0'} 2022-09-12T20:08:32.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d07b2f394ce5f06-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:08:32.004 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:32.004 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:32.005 DEBUG:boto:Method: GET 2022-09-12T20:08:32.006 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:08:32.007 DEBUG:boto:Data: 2022-09-12T20:08:32.007 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.008 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.009 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.009 DEBUG:boto:Params: {} 2022-09-12T20:08:32.010 DEBUG:boto:Token: None 2022-09-12T20:08:32.011 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj19 2022-09-12T20:08:32.012 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y6PkzJJ2yHa0zwVY4yWTUkDEiyY= 2022-09-12T20:08:32.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y6PkzJJ2yHa0zwVY4yWTUkDEiyY=', 'Content-Length': '0'} 2022-09-12T20:08:32.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082a6f1cdb2005f75-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.016 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:32.017 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:32.017 DEBUG:boto:Method: GET 2022-09-12T20:08:32.018 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:08:32.019 DEBUG:boto:Data: 2022-09-12T20:08:32.019 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.020 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.021 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.021 DEBUG:boto:Params: {} 2022-09-12T20:08:32.022 DEBUG:boto:Token: None 2022-09-12T20:08:32.023 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj19 2022-09-12T20:08:32.024 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y6PkzJJ2yHa0zwVY4yWTUkDEiyY= 2022-09-12T20:08:32.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y6PkzJJ2yHa0zwVY4yWTUkDEiyY=', 'Content-Length': '0'} 2022-09-12T20:08:32.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000406904acdbc0e806-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.028 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:32.029 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:32.030 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:08:32.030 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:08:32.031 DEBUG:boto:Method: GET 2022-09-12T20:08:32.032 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:08:32.032 DEBUG:boto:Data: 2022-09-12T20:08:32.033 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.034 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.035 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.036 DEBUG:boto:Params: {} 2022-09-12T20:08:32.036 DEBUG:boto:Token: None 2022-09-12T20:08:32.037 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/ 2022-09-12T20:08:32.038 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088= 2022-09-12T20:08:32.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088=', 'Content-Length': '0'} 2022-09-12T20:08:32.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b708b7665e64123-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.086 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:32.087 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:32.087 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:32.088 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:08:32.088 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:08:32.089 DEBUG:boto:Method: GET 2022-09-12T20:08:32.089 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:08:32.090 DEBUG:boto:Data: 2022-09-12T20:08:32.091 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.091 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.092 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.092 DEBUG:boto:Params: {} 2022-09-12T20:08:32.093 DEBUG:boto:Token: None 2022-09-12T20:08:32.093 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/ 2022-09-12T20:08:32.094 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088= 2022-09-12T20:08:32.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088=', 'Content-Length': '0'} 2022-09-12T20:08:32.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073105a5220f3cff8-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.101 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:32.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:08:32.103 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:32.103 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:32.104 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:32.104 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:32.105 DEBUG:boto:Method: GET 2022-09-12T20:08:32.105 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:32.106 DEBUG:boto:Data: 2022-09-12T20:08:32.107 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.107 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.108 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.108 DEBUG:boto:Params: {} 2022-09-12T20:08:32.109 DEBUG:boto:Token: None 2022-09-12T20:08:32.109 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:32.110 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yFLBQWlnpxn1fldH/tkrLlX1tAk= 2022-09-12T20:08:32.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yFLBQWlnpxn1fldH/tkrLlX1tAk=', 'Content-Length': '0'} 2022-09-12T20:08:32.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003ec30957f22d163-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.114 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:32.114 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:32.115 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:32.115 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:32.116 DEBUG:boto:Method: GET 2022-09-12T20:08:32.116 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:32.117 DEBUG:boto:Data: 2022-09-12T20:08:32.117 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.118 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.118 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.119 DEBUG:boto:Params: {} 2022-09-12T20:08:32.120 DEBUG:boto:Token: None 2022-09-12T20:08:32.120 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:32.121 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yFLBQWlnpxn1fldH/tkrLlX1tAk= 2022-09-12T20:08:32.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yFLBQWlnpxn1fldH/tkrLlX1tAk=', 'Content-Length': '0'} 2022-09-12T20:08:32.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e8eb2f35ad4c311-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.166 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:32.166 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:32.167 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:32.167 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:32.168 DEBUG:boto:Method: HEAD 2022-09-12T20:08:32.169 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:32.169 DEBUG:boto:Data: 2022-09-12T20:08:32.170 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.170 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.171 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.171 DEBUG:boto:Params: {} 2022-09-12T20:08:32.172 DEBUG:boto:Token: None 2022-09-12T20:08:32.172 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:32.173 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AzmxhT3qloPyIhmuY4kGj30xxiM= 2022-09-12T20:08:32.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AzmxhT3qloPyIhmuY4kGj30xxiM=', 'Content-Length': '0'} 2022-09-12T20:08:32.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef6ea86c562ffc98-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.176 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:32.177 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:32.177 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:32.178 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:32.178 DEBUG:boto:Method: HEAD 2022-09-12T20:08:32.179 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:32.179 DEBUG:boto:Data: 2022-09-12T20:08:32.180 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.180 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.181 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.181 DEBUG:boto:Params: {} 2022-09-12T20:08:32.182 DEBUG:boto:Token: None 2022-09-12T20:08:32.182 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:32.183 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AzmxhT3qloPyIhmuY4kGj30xxiM= 2022-09-12T20:08:32.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AzmxhT3qloPyIhmuY4kGj30xxiM=', 'Content-Length': '0'} 2022-09-12T20:08:32.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000676157e5321f497a-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:08:32.187 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:32.188 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:32.188 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:32.189 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:32.189 DEBUG:boto:Method: GET 2022-09-12T20:08:32.190 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:32.190 DEBUG:boto:Data: 2022-09-12T20:08:32.191 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.191 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.192 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.192 DEBUG:boto:Params: {} 2022-09-12T20:08:32.193 DEBUG:boto:Token: None 2022-09-12T20:08:32.194 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:32.195 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yFLBQWlnpxn1fldH/tkrLlX1tAk= 2022-09-12T20:08:32.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yFLBQWlnpxn1fldH/tkrLlX1tAk=', 'Content-Length': '0'} 2022-09-12T20:08:32.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce1b90982ef8f7d6-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.199 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:32.200 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:32.200 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:32.201 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:32.201 DEBUG:boto:Method: GET 2022-09-12T20:08:32.202 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:32.202 DEBUG:boto:Data: 2022-09-12T20:08:32.203 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.203 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.204 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.204 DEBUG:boto:Params: {} 2022-09-12T20:08:32.205 DEBUG:boto:Token: None 2022-09-12T20:08:32.206 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:32.206 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yFLBQWlnpxn1fldH/tkrLlX1tAk= 2022-09-12T20:08:32.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yFLBQWlnpxn1fldH/tkrLlX1tAk=', 'Content-Length': '0'} 2022-09-12T20:08:32.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b228cdde614ac6e-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.210 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:32.211 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:32.211 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:08:32.212 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:08:32.212 DEBUG:boto:Method: GET 2022-09-12T20:08:32.213 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:08:32.213 DEBUG:boto:Data: 2022-09-12T20:08:32.214 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.214 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.215 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.215 DEBUG:boto:Params: {} 2022-09-12T20:08:32.216 DEBUG:boto:Token: None 2022-09-12T20:08:32.217 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/ 2022-09-12T20:08:32.217 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088= 2022-09-12T20:08:32.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088=', 'Content-Length': '0'} 2022-09-12T20:08:32.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abc856ee2c67a830-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.221 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:32.222 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:32.223 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:32.224 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:08:32.224 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:08:32.225 DEBUG:boto:Method: GET 2022-09-12T20:08:32.225 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:08:32.226 DEBUG:boto:Data: 2022-09-12T20:08:32.226 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.227 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.228 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.228 DEBUG:boto:Params: {} 2022-09-12T20:08:32.229 DEBUG:boto:Token: None 2022-09-12T20:08:32.229 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/ 2022-09-12T20:08:32.230 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088= 2022-09-12T20:08:32.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088=', 'Content-Length': '0'} 2022-09-12T20:08:32.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de351163e340a752-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.235 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:32.235 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:32.236 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:32.237 DEBUG:boto:Method: HEAD 2022-09-12T20:08:32.237 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:08:32.238 DEBUG:boto:Data: 2022-09-12T20:08:32.238 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.239 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.239 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.240 DEBUG:boto:Params: {} 2022-09-12T20:08:32.241 DEBUG:boto:Token: None 2022-09-12T20:08:32.241 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj22 2022-09-12T20:08:32.242 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JKzWrHHMrOcj1oPqKHyJGP/Sl3o= 2022-09-12T20:08:32.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JKzWrHHMrOcj1oPqKHyJGP/Sl3o=', 'Content-Length': '0'} 2022-09-12T20:08:32.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df090fb41bab5cf1-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.245 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:32.246 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:32.246 DEBUG:boto:Method: HEAD 2022-09-12T20:08:32.247 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:08:32.247 DEBUG:boto:Data: 2022-09-12T20:08:32.248 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.248 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.249 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.249 DEBUG:boto:Params: {} 2022-09-12T20:08:32.250 DEBUG:boto:Token: None 2022-09-12T20:08:32.251 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj22 2022-09-12T20:08:32.251 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JKzWrHHMrOcj1oPqKHyJGP/Sl3o= 2022-09-12T20:08:32.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JKzWrHHMrOcj1oPqKHyJGP/Sl3o=', 'Content-Length': '0'} 2022-09-12T20:08:32.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000067d358e7299aeff-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:08:32.256 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:32.256 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:32.257 DEBUG:boto:Method: GET 2022-09-12T20:08:32.257 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:08:32.258 DEBUG:boto:Data: 2022-09-12T20:08:32.258 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.259 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.260 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.260 DEBUG:boto:Params: {} 2022-09-12T20:08:32.261 DEBUG:boto:Token: None 2022-09-12T20:08:32.261 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj22 2022-09-12T20:08:32.262 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:schxdjxU8JqCXhEZwbDLRNRprS4= 2022-09-12T20:08:32.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:schxdjxU8JqCXhEZwbDLRNRprS4=', 'Content-Length': '0'} 2022-09-12T20:08:32.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e099a3b78b33f20f-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.266 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:32.266 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:32.267 DEBUG:boto:Method: GET 2022-09-12T20:08:32.267 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:08:32.268 DEBUG:boto:Data: 2022-09-12T20:08:32.268 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.269 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.269 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.270 DEBUG:boto:Params: {} 2022-09-12T20:08:32.270 DEBUG:boto:Token: None 2022-09-12T20:08:32.271 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj22 2022-09-12T20:08:32.272 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:schxdjxU8JqCXhEZwbDLRNRprS4= 2022-09-12T20:08:32.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:schxdjxU8JqCXhEZwbDLRNRprS4=', 'Content-Length': '0'} 2022-09-12T20:08:32.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000918c3e38566346ab-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.318 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:32.319 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:32.319 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:08:32.320 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:08:32.321 DEBUG:boto:Method: GET 2022-09-12T20:08:32.321 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:08:32.322 DEBUG:boto:Data: 2022-09-12T20:08:32.322 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.323 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.323 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.324 DEBUG:boto:Params: {} 2022-09-12T20:08:32.324 DEBUG:boto:Token: None 2022-09-12T20:08:32.325 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/ 2022-09-12T20:08:32.326 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088= 2022-09-12T20:08:32.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088=', 'Content-Length': '0'} 2022-09-12T20:08:32.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b554a23e397fa8c1-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.330 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:32.331 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:32.331 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:32.332 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:08:32.333 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:08:32.333 DEBUG:boto:Method: GET 2022-09-12T20:08:32.334 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:08:32.334 DEBUG:boto:Data: 2022-09-12T20:08:32.335 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.335 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.336 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.336 DEBUG:boto:Params: {} 2022-09-12T20:08:32.337 DEBUG:boto:Token: None 2022-09-12T20:08:32.338 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/ 2022-09-12T20:08:32.338 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088= 2022-09-12T20:08:32.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088=', 'Content-Length': '0'} 2022-09-12T20:08:32.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c532040798885b6d-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.344 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:32.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:08:32.345 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:32.346 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:32.346 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:32.347 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:32.347 DEBUG:boto:Method: GET 2022-09-12T20:08:32.348 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:32.349 DEBUG:boto:Data: 2022-09-12T20:08:32.349 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.350 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.350 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.351 DEBUG:boto:Params: {} 2022-09-12T20:08:32.352 DEBUG:boto:Token: None 2022-09-12T20:08:32.352 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:32.353 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:atze6NOwij+35Oh1OKAGL/FGlqE= 2022-09-12T20:08:32.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:atze6NOwij+35Oh1OKAGL/FGlqE=', 'Content-Length': '0'} 2022-09-12T20:08:32.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057e72d1034e3d908-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.357 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:32.357 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:32.358 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:32.358 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:32.359 DEBUG:boto:Method: GET 2022-09-12T20:08:32.360 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:32.360 DEBUG:boto:Data: 2022-09-12T20:08:32.361 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.361 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.362 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.362 DEBUG:boto:Params: {} 2022-09-12T20:08:32.363 DEBUG:boto:Token: None 2022-09-12T20:08:32.363 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:32.364 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:atze6NOwij+35Oh1OKAGL/FGlqE= 2022-09-12T20:08:32.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:atze6NOwij+35Oh1OKAGL/FGlqE=', 'Content-Length': '0'} 2022-09-12T20:08:32.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ff388bb9b5e7f3d-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.368 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:32.368 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:32.369 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:32.369 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:32.370 DEBUG:boto:Method: HEAD 2022-09-12T20:08:32.371 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:32.371 DEBUG:boto:Data: 2022-09-12T20:08:32.372 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.372 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.373 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.373 DEBUG:boto:Params: {} 2022-09-12T20:08:32.374 DEBUG:boto:Token: None 2022-09-12T20:08:32.374 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:32.375 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Rw/ZD6CwNupst5/M5H701E6FtN0= 2022-09-12T20:08:32.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Rw/ZD6CwNupst5/M5H701E6FtN0=', 'Content-Length': '0'} 2022-09-12T20:08:32.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c05d63d3651d515-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.378 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:32.379 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:32.379 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:32.380 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:32.381 DEBUG:boto:Method: HEAD 2022-09-12T20:08:32.381 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:32.382 DEBUG:boto:Data: 2022-09-12T20:08:32.382 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.383 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.383 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.384 DEBUG:boto:Params: {} 2022-09-12T20:08:32.384 DEBUG:boto:Token: None 2022-09-12T20:08:32.385 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:32.385 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Rw/ZD6CwNupst5/M5H701E6FtN0= 2022-09-12T20:08:32.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Rw/ZD6CwNupst5/M5H701E6FtN0=', 'Content-Length': '0'} 2022-09-12T20:08:32.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed3c2d405e90db9f-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:08:32.390 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:32.390 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:32.391 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:32.391 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:32.392 DEBUG:boto:Method: GET 2022-09-12T20:08:32.392 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:32.393 DEBUG:boto:Data: 2022-09-12T20:08:32.393 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.394 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.395 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.395 DEBUG:boto:Params: {} 2022-09-12T20:08:32.396 DEBUG:boto:Token: None 2022-09-12T20:08:32.396 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:32.397 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:atze6NOwij+35Oh1OKAGL/FGlqE= 2022-09-12T20:08:32.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:atze6NOwij+35Oh1OKAGL/FGlqE=', 'Content-Length': '0'} 2022-09-12T20:08:32.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007993c96944caf9f2-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.400 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:32.401 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:32.401 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:32.402 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:32.402 DEBUG:boto:Method: GET 2022-09-12T20:08:32.403 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:32.403 DEBUG:boto:Data: 2022-09-12T20:08:32.404 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.404 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.405 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.405 DEBUG:boto:Params: {} 2022-09-12T20:08:32.406 DEBUG:boto:Token: None 2022-09-12T20:08:32.407 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:32.408 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:atze6NOwij+35Oh1OKAGL/FGlqE= 2022-09-12T20:08:32.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:atze6NOwij+35Oh1OKAGL/FGlqE=', 'Content-Length': '0'} 2022-09-12T20:08:32.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2e1ba1f77815467-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.411 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:32.412 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:32.412 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:08:32.413 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:08:32.413 DEBUG:boto:Method: GET 2022-09-12T20:08:32.414 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:08:32.414 DEBUG:boto:Data: 2022-09-12T20:08:32.415 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.415 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.416 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.416 DEBUG:boto:Params: {} 2022-09-12T20:08:32.417 DEBUG:boto:Token: None 2022-09-12T20:08:32.418 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/ 2022-09-12T20:08:32.418 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088= 2022-09-12T20:08:32.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088=', 'Content-Length': '0'} 2022-09-12T20:08:32.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b3a9800bab87ecf-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.422 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:32.423 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:32.424 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:32.424 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:08:32.425 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:08:32.425 DEBUG:boto:Method: GET 2022-09-12T20:08:32.426 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:08:32.426 DEBUG:boto:Data: 2022-09-12T20:08:32.427 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.427 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.428 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.428 DEBUG:boto:Params: {} 2022-09-12T20:08:32.429 DEBUG:boto:Token: None 2022-09-12T20:08:32.430 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/ 2022-09-12T20:08:32.430 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088= 2022-09-12T20:08:32.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088=', 'Content-Length': '0'} 2022-09-12T20:08:32.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041c29138928f2de1-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.478 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:32.479 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:32.480 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:32.480 DEBUG:boto:Method: HEAD 2022-09-12T20:08:32.481 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:08:32.481 DEBUG:boto:Data: 2022-09-12T20:08:32.482 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.482 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.483 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.483 DEBUG:boto:Params: {} 2022-09-12T20:08:32.484 DEBUG:boto:Token: None 2022-09-12T20:08:32.484 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj33 2022-09-12T20:08:32.485 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8fZ/7B23NPQNALHcBAT/7iJhYfY= 2022-09-12T20:08:32.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8fZ/7B23NPQNALHcBAT/7iJhYfY=', 'Content-Length': '0'} 2022-09-12T20:08:32.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c7e315f6fbfdba2-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.489 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:32.489 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:32.490 DEBUG:boto:Method: HEAD 2022-09-12T20:08:32.490 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:08:32.491 DEBUG:boto:Data: 2022-09-12T20:08:32.491 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.492 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.492 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.493 DEBUG:boto:Params: {} 2022-09-12T20:08:32.493 DEBUG:boto:Token: None 2022-09-12T20:08:32.494 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj33 2022-09-12T20:08:32.494 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8fZ/7B23NPQNALHcBAT/7iJhYfY= 2022-09-12T20:08:32.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8fZ/7B23NPQNALHcBAT/7iJhYfY=', 'Content-Length': '0'} 2022-09-12T20:08:32.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5a58d32b91fb7d9-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:08:32.498 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:32.499 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:32.499 DEBUG:boto:Method: GET 2022-09-12T20:08:32.500 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:08:32.500 DEBUG:boto:Data: 2022-09-12T20:08:32.501 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.502 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.502 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.503 DEBUG:boto:Params: {} 2022-09-12T20:08:32.503 DEBUG:boto:Token: None 2022-09-12T20:08:32.504 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj33 2022-09-12T20:08:32.504 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Tfa0qnWX5drszb97jQbWylLwDLU= 2022-09-12T20:08:32.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Tfa0qnWX5drszb97jQbWylLwDLU=', 'Content-Length': '0'} 2022-09-12T20:08:32.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa7766f6e036e471-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.508 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:32.508 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:32.509 DEBUG:boto:Method: GET 2022-09-12T20:08:32.509 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:08:32.510 DEBUG:boto:Data: 2022-09-12T20:08:32.510 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.511 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.512 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.512 DEBUG:boto:Params: {} 2022-09-12T20:08:32.513 DEBUG:boto:Token: None 2022-09-12T20:08:32.513 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj33 2022-09-12T20:08:32.514 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Tfa0qnWX5drszb97jQbWylLwDLU= 2022-09-12T20:08:32.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Tfa0qnWX5drszb97jQbWylLwDLU=', 'Content-Length': '0'} 2022-09-12T20:08:32.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3b4147518888e56-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.517 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:32.518 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:32.518 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:08:32.519 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:08:32.519 DEBUG:boto:Method: GET 2022-09-12T20:08:32.520 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:08:32.520 DEBUG:boto:Data: 2022-09-12T20:08:32.521 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.522 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.522 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.523 DEBUG:boto:Params: {} 2022-09-12T20:08:32.523 DEBUG:boto:Token: None 2022-09-12T20:08:32.524 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/ 2022-09-12T20:08:32.524 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088= 2022-09-12T20:08:32.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088=', 'Content-Length': '0'} 2022-09-12T20:08:32.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f994f5e95f0fdb6-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.528 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:32.529 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:32.530 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:32.530 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:08:32.531 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:08:32.531 DEBUG:boto:Method: GET 2022-09-12T20:08:32.532 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:08:32.532 DEBUG:boto:Data: 2022-09-12T20:08:32.533 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.533 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.534 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.534 DEBUG:boto:Params: {} 2022-09-12T20:08:32.535 DEBUG:boto:Token: None 2022-09-12T20:08:32.536 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/ 2022-09-12T20:08:32.536 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088= 2022-09-12T20:08:32.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088=', 'Content-Length': '0'} 2022-09-12T20:08:32.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000033b3ae13e406d00-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.541 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:32.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:08:32.542 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:32.543 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:32.543 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:32.544 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:32.544 DEBUG:boto:Method: GET 2022-09-12T20:08:32.545 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:32.545 DEBUG:boto:Data: 2022-09-12T20:08:32.546 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.546 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.547 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.547 DEBUG:boto:Params: {} 2022-09-12T20:08:32.548 DEBUG:boto:Token: None 2022-09-12T20:08:32.548 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:32.549 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YpTuTomNa7qtByWa2ElxJ8DzagY= 2022-09-12T20:08:32.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YpTuTomNa7qtByWa2ElxJ8DzagY=', 'Content-Length': '0'} 2022-09-12T20:08:32.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f19417351cb29e40-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.594 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:32.595 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:32.596 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:32.596 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:32.597 DEBUG:boto:Method: GET 2022-09-12T20:08:32.597 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:32.598 DEBUG:boto:Data: 2022-09-12T20:08:32.598 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.599 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.600 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.600 DEBUG:boto:Params: {} 2022-09-12T20:08:32.601 DEBUG:boto:Token: None 2022-09-12T20:08:32.601 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:32.602 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YpTuTomNa7qtByWa2ElxJ8DzagY= 2022-09-12T20:08:32.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YpTuTomNa7qtByWa2ElxJ8DzagY=', 'Content-Length': '0'} 2022-09-12T20:08:32.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006173805dc6149fba-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.606 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:32.606 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:32.607 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:32.607 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:32.608 DEBUG:boto:Method: HEAD 2022-09-12T20:08:32.608 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:32.609 DEBUG:boto:Data: 2022-09-12T20:08:32.609 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.610 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.610 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.611 DEBUG:boto:Params: {} 2022-09-12T20:08:32.611 DEBUG:boto:Token: None 2022-09-12T20:08:32.612 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:32.613 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FsZQYQIobT6Z/h3FHUNQYtYm7hY= 2022-09-12T20:08:32.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FsZQYQIobT6Z/h3FHUNQYtYm7hY=', 'Content-Length': '0'} 2022-09-12T20:08:32.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009059c20ea841e34c-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.616 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:32.617 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:32.617 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:32.618 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:32.618 DEBUG:boto:Method: HEAD 2022-09-12T20:08:32.619 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:32.619 DEBUG:boto:Data: 2022-09-12T20:08:32.620 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.620 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.621 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.621 DEBUG:boto:Params: {} 2022-09-12T20:08:32.622 DEBUG:boto:Token: None 2022-09-12T20:08:32.623 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:32.623 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FsZQYQIobT6Z/h3FHUNQYtYm7hY= 2022-09-12T20:08:32.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FsZQYQIobT6Z/h3FHUNQYtYm7hY=', 'Content-Length': '0'} 2022-09-12T20:08:32.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6cf96aad21e7b42-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:08:32.627 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:32.628 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:32.629 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:32.629 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:32.630 DEBUG:boto:Method: GET 2022-09-12T20:08:32.630 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:32.631 DEBUG:boto:Data: 2022-09-12T20:08:32.631 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.632 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.632 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.633 DEBUG:boto:Params: {} 2022-09-12T20:08:32.634 DEBUG:boto:Token: None 2022-09-12T20:08:32.634 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:32.635 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YpTuTomNa7qtByWa2ElxJ8DzagY= 2022-09-12T20:08:32.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YpTuTomNa7qtByWa2ElxJ8DzagY=', 'Content-Length': '0'} 2022-09-12T20:08:32.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000639515a3132c5881-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.639 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:32.639 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:32.640 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:32.641 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:32.641 DEBUG:boto:Method: GET 2022-09-12T20:08:32.642 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:32.642 DEBUG:boto:Data: 2022-09-12T20:08:32.643 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.643 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.644 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.644 DEBUG:boto:Params: {} 2022-09-12T20:08:32.645 DEBUG:boto:Token: None 2022-09-12T20:08:32.646 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:32.646 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YpTuTomNa7qtByWa2ElxJ8DzagY= 2022-09-12T20:08:32.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YpTuTomNa7qtByWa2ElxJ8DzagY=', 'Content-Length': '0'} 2022-09-12T20:08:32.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038fecdfedd11c678-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.650 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:32.651 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:32.651 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:08:32.652 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:08:32.652 DEBUG:boto:Method: GET 2022-09-12T20:08:32.653 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:08:32.653 DEBUG:boto:Data: 2022-09-12T20:08:32.654 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.654 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.655 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.655 DEBUG:boto:Params: {} 2022-09-12T20:08:32.656 DEBUG:boto:Token: None 2022-09-12T20:08:32.657 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/ 2022-09-12T20:08:32.657 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088= 2022-09-12T20:08:32.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088=', 'Content-Length': '0'} 2022-09-12T20:08:32.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002802a624ebc9a208-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.662 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:32.663 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:32.663 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:32.664 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:08:32.664 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:08:32.665 DEBUG:boto:Method: GET 2022-09-12T20:08:32.665 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:08:32.666 DEBUG:boto:Data: 2022-09-12T20:08:32.666 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.667 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.667 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.668 DEBUG:boto:Params: {} 2022-09-12T20:08:32.669 DEBUG:boto:Token: None 2022-09-12T20:08:32.669 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/ 2022-09-12T20:08:32.670 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088= 2022-09-12T20:08:32.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088=', 'Content-Length': '0'} 2022-09-12T20:08:32.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdf050c6e0cffe8a-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.675 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:32.675 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:32.676 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:32.676 DEBUG:boto:Method: HEAD 2022-09-12T20:08:32.677 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:08:32.678 DEBUG:boto:Data: 2022-09-12T20:08:32.678 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.679 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.679 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.680 DEBUG:boto:Params: {} 2022-09-12T20:08:32.680 DEBUG:boto:Token: None 2022-09-12T20:08:32.681 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj34 2022-09-12T20:08:32.681 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yExzXpdBmv9ltAlQX6n0Jp/eHW4= 2022-09-12T20:08:32.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yExzXpdBmv9ltAlQX6n0Jp/eHW4=', 'Content-Length': '0'} 2022-09-12T20:08:32.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004db03b9aa372a071-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.685 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:32.685 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:32.686 DEBUG:boto:Method: HEAD 2022-09-12T20:08:32.686 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:08:32.687 DEBUG:boto:Data: 2022-09-12T20:08:32.688 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.688 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.689 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.689 DEBUG:boto:Params: {} 2022-09-12T20:08:32.690 DEBUG:boto:Token: None 2022-09-12T20:08:32.690 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj34 2022-09-12T20:08:32.691 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yExzXpdBmv9ltAlQX6n0Jp/eHW4= 2022-09-12T20:08:32.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yExzXpdBmv9ltAlQX6n0Jp/eHW4=', 'Content-Length': '0'} 2022-09-12T20:08:32.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5ad0895ef6cfb15-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:08:32.695 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:32.695 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:32.696 DEBUG:boto:Method: GET 2022-09-12T20:08:32.697 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:08:32.697 DEBUG:boto:Data: 2022-09-12T20:08:32.698 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.698 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.699 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.699 DEBUG:boto:Params: {} 2022-09-12T20:08:32.700 DEBUG:boto:Token: None 2022-09-12T20:08:32.701 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj34 2022-09-12T20:08:32.701 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NXzj6FcjkOTagzQewnySR/DuG+w= 2022-09-12T20:08:32.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NXzj6FcjkOTagzQewnySR/DuG+w=', 'Content-Length': '0'} 2022-09-12T20:08:32.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2c2bc7092bb72b1-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.705 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:32.705 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:32.706 DEBUG:boto:Method: GET 2022-09-12T20:08:32.706 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:08:32.707 DEBUG:boto:Data: 2022-09-12T20:08:32.708 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.708 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.709 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.709 DEBUG:boto:Params: {} 2022-09-12T20:08:32.710 DEBUG:boto:Token: None 2022-09-12T20:08:32.710 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj34 2022-09-12T20:08:32.711 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NXzj6FcjkOTagzQewnySR/DuG+w= 2022-09-12T20:08:32.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NXzj6FcjkOTagzQewnySR/DuG+w=', 'Content-Length': '0'} 2022-09-12T20:08:32.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4350e1e5d92c86a-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.754 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:32.755 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:32.755 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:08:32.756 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:08:32.756 DEBUG:boto:Method: GET 2022-09-12T20:08:32.757 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:08:32.757 DEBUG:boto:Data: 2022-09-12T20:08:32.758 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.758 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.759 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.759 DEBUG:boto:Params: {} 2022-09-12T20:08:32.760 DEBUG:boto:Token: None 2022-09-12T20:08:32.761 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/ 2022-09-12T20:08:32.761 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088= 2022-09-12T20:08:32.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088=', 'Content-Length': '0'} 2022-09-12T20:08:32.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f899001ea74db718-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.765 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:32.766 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:32.767 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:32.768 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:08:32.769 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:08:32.769 DEBUG:boto:Method: GET 2022-09-12T20:08:32.770 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:08:32.770 DEBUG:boto:Data: 2022-09-12T20:08:32.771 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.771 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.772 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.772 DEBUG:boto:Params: {} 2022-09-12T20:08:32.773 DEBUG:boto:Token: None 2022-09-12T20:08:32.773 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/ 2022-09-12T20:08:32.774 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088= 2022-09-12T20:08:32.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088=', 'Content-Length': '0'} 2022-09-12T20:08:32.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1ddd1844c57cf60-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.779 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:32.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:08:32.780 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:32.781 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:32.781 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:32.782 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:32.782 DEBUG:boto:Method: GET 2022-09-12T20:08:32.783 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:32.783 DEBUG:boto:Data: 2022-09-12T20:08:32.784 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.784 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.785 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.786 DEBUG:boto:Params: {} 2022-09-12T20:08:32.786 DEBUG:boto:Token: None 2022-09-12T20:08:32.787 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:32.787 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4MLn8Qs9xuIOitmcclzDslZDjbo= 2022-09-12T20:08:32.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4MLn8Qs9xuIOitmcclzDslZDjbo=', 'Content-Length': '0'} 2022-09-12T20:08:32.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc83a6b75c60e619-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.791 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:32.792 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:32.792 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:32.793 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:32.794 DEBUG:boto:Method: GET 2022-09-12T20:08:32.794 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:32.795 DEBUG:boto:Data: 2022-09-12T20:08:32.795 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.796 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.796 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.797 DEBUG:boto:Params: {} 2022-09-12T20:08:32.797 DEBUG:boto:Token: None 2022-09-12T20:08:32.798 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:32.799 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4MLn8Qs9xuIOitmcclzDslZDjbo= 2022-09-12T20:08:32.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4MLn8Qs9xuIOitmcclzDslZDjbo=', 'Content-Length': '0'} 2022-09-12T20:08:32.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064ec265603e2c69f-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.803 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:32.803 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:32.804 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:32.804 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:32.805 DEBUG:boto:Method: HEAD 2022-09-12T20:08:32.805 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:32.806 DEBUG:boto:Data: 2022-09-12T20:08:32.807 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.807 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.808 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.808 DEBUG:boto:Params: {} 2022-09-12T20:08:32.809 DEBUG:boto:Token: None 2022-09-12T20:08:32.809 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:32.810 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GSir7gkDAc1syTu4wvnZUeO+8fQ= 2022-09-12T20:08:32.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GSir7gkDAc1syTu4wvnZUeO+8fQ=', 'Content-Length': '0'} 2022-09-12T20:08:32.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9b332d2c0d1239a-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.813 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:32.814 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:32.814 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:32.815 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:32.815 DEBUG:boto:Method: HEAD 2022-09-12T20:08:32.816 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:32.816 DEBUG:boto:Data: 2022-09-12T20:08:32.817 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.817 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.818 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.818 DEBUG:boto:Params: {} 2022-09-12T20:08:32.819 DEBUG:boto:Token: None 2022-09-12T20:08:32.820 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:32.820 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GSir7gkDAc1syTu4wvnZUeO+8fQ= 2022-09-12T20:08:32.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GSir7gkDAc1syTu4wvnZUeO+8fQ=', 'Content-Length': '0'} 2022-09-12T20:08:32.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022f2465e940b59f5-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:08:32.824 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:32.825 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:32.825 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:32.826 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:32.827 DEBUG:boto:Method: GET 2022-09-12T20:08:32.827 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:32.828 DEBUG:boto:Data: 2022-09-12T20:08:32.828 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.829 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.829 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.830 DEBUG:boto:Params: {} 2022-09-12T20:08:32.831 DEBUG:boto:Token: None 2022-09-12T20:08:32.831 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:32.832 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4MLn8Qs9xuIOitmcclzDslZDjbo= 2022-09-12T20:08:32.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4MLn8Qs9xuIOitmcclzDslZDjbo=', 'Content-Length': '0'} 2022-09-12T20:08:32.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3744b19c20f7d50-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.878 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:32.879 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:32.879 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:32.880 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:32.880 DEBUG:boto:Method: GET 2022-09-12T20:08:32.881 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:32.882 DEBUG:boto:Data: 2022-09-12T20:08:32.882 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.883 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.883 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.884 DEBUG:boto:Params: {} 2022-09-12T20:08:32.884 DEBUG:boto:Token: None 2022-09-12T20:08:32.885 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:32.885 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4MLn8Qs9xuIOitmcclzDslZDjbo= 2022-09-12T20:08:32.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4MLn8Qs9xuIOitmcclzDslZDjbo=', 'Content-Length': '0'} 2022-09-12T20:08:32.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b432431dcee286b2-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.889 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:32.890 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:32.890 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:08:32.891 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:08:32.891 DEBUG:boto:Method: GET 2022-09-12T20:08:32.892 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:08:32.892 DEBUG:boto:Data: 2022-09-12T20:08:32.893 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.893 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.894 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.895 DEBUG:boto:Params: {} 2022-09-12T20:08:32.895 DEBUG:boto:Token: None 2022-09-12T20:08:32.896 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/ 2022-09-12T20:08:32.896 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088= 2022-09-12T20:08:32.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088=', 'Content-Length': '0'} 2022-09-12T20:08:32.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6c642f3a995b3bc-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.901 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:32.901 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:32.902 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:32.903 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:08:32.903 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:08:32.904 DEBUG:boto:Method: GET 2022-09-12T20:08:32.904 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:08:32.905 DEBUG:boto:Data: 2022-09-12T20:08:32.905 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.906 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.906 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.907 DEBUG:boto:Params: {} 2022-09-12T20:08:32.907 DEBUG:boto:Token: None 2022-09-12T20:08:32.908 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/ 2022-09-12T20:08:32.909 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088= 2022-09-12T20:08:32.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088=', 'Content-Length': '0'} 2022-09-12T20:08:32.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f51d2cecc872e33-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.913 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:32.914 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:32.915 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:32.915 DEBUG:boto:Method: HEAD 2022-09-12T20:08:32.916 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:08:32.916 DEBUG:boto:Data: 2022-09-12T20:08:32.917 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.917 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.918 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.918 DEBUG:boto:Params: {} 2022-09-12T20:08:32.919 DEBUG:boto:Token: None 2022-09-12T20:08:32.919 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj38 2022-09-12T20:08:32.920 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e9vcWYNZhohHNNLIVAh+SN2we+w= 2022-09-12T20:08:32.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e9vcWYNZhohHNNLIVAh+SN2we+w=', 'Content-Length': '0'} 2022-09-12T20:08:32.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095855ade760e84ba-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.923 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:32.924 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:32.924 DEBUG:boto:Method: HEAD 2022-09-12T20:08:32.925 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:08:32.925 DEBUG:boto:Data: 2022-09-12T20:08:32.926 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.926 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.927 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.927 DEBUG:boto:Params: {} 2022-09-12T20:08:32.928 DEBUG:boto:Token: None 2022-09-12T20:08:32.928 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj38 2022-09-12T20:08:32.929 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e9vcWYNZhohHNNLIVAh+SN2we+w= 2022-09-12T20:08:32.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e9vcWYNZhohHNNLIVAh+SN2we+w=', 'Content-Length': '0'} 2022-09-12T20:08:32.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000918338cbb600ee1f-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:08:32.932 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:32.933 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:32.933 DEBUG:boto:Method: GET 2022-09-12T20:08:32.934 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:08:32.934 DEBUG:boto:Data: 2022-09-12T20:08:32.935 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.936 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.936 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.937 DEBUG:boto:Params: {} 2022-09-12T20:08:32.937 DEBUG:boto:Token: None 2022-09-12T20:08:32.938 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj38 2022-09-12T20:08:32.938 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dosowxMX/tWudJr9cJBEvYPXlho= 2022-09-12T20:08:32.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dosowxMX/tWudJr9cJBEvYPXlho=', 'Content-Length': '0'} 2022-09-12T20:08:32.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe6bc399b0422f9f-00631f91c0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.982 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:32.983 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:32.984 DEBUG:boto:Method: GET 2022-09-12T20:08:32.984 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:08:32.985 DEBUG:boto:Data: 2022-09-12T20:08:32.985 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.986 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:32.987 DEBUG:boto:Port: 8000 2022-09-12T20:08:32.987 DEBUG:boto:Params: {} 2022-09-12T20:08:32.988 DEBUG:boto:Token: None 2022-09-12T20:08:32.988 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/obj38 2022-09-12T20:08:32.989 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dosowxMX/tWudJr9cJBEvYPXlho= 2022-09-12T20:08:32.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dosowxMX/tWudJr9cJBEvYPXlho=', 'Content-Length': '0'} 2022-09-12T20:08:32.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee5c29ab8ffa94b4-00631f91c0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:32.993 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:32.993 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:32.994 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:08:32.994 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:08:32.995 DEBUG:boto:Method: GET 2022-09-12T20:08:32.995 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:08:32.996 DEBUG:boto:Data: 2022-09-12T20:08:32.997 DEBUG:boto:Headers: {} 2022-09-12T20:08:32.997 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:32.998 DEBUG:boto:Port: 8001 2022-09-12T20:08:32.998 DEBUG:boto:Params: {} 2022-09-12T20:08:32.999 DEBUG:boto:Token: None 2022-09-12T20:08:32.999 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:32 GMT /jdpiak-75/ 2022-09-12T20:08:33.000 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088= 2022-09-12T20:08:33.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:32 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KxTHR5fGWvNZ+NzYEiIQ91CC088=', 'Content-Length': '0'} 2022-09-12T20:08:33.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004439edfbdd87c9c6-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.004 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:33.005 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:33.005 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:33.006 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:08:33.007 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:08:33.007 DEBUG:boto:Method: GET 2022-09-12T20:08:33.008 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:08:33.008 DEBUG:boto:Data: 2022-09-12T20:08:33.009 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.009 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.010 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.011 DEBUG:boto:Params: {} 2022-09-12T20:08:33.011 DEBUG:boto:Token: None 2022-09-12T20:08:33.012 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/ 2022-09-12T20:08:33.013 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM= 2022-09-12T20:08:33.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM=', 'Content-Length': '0'} 2022-09-12T20:08:33.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a7db1e21a5dc6cd-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.017 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:33.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:08:33.019 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:33.019 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:33.020 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:33.020 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:33.029 DEBUG:boto:Method: GET 2022-09-12T20:08:33.030 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:33.031 DEBUG:boto:Data: 2022-09-12T20:08:33.031 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.032 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.032 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.033 DEBUG:boto:Params: {} 2022-09-12T20:08:33.034 DEBUG:boto:Token: None 2022-09-12T20:08:33.034 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:33.035 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zj0PgTAP6k7cGbMU/TKZxzFxQmo= 2022-09-12T20:08:33.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zj0PgTAP6k7cGbMU/TKZxzFxQmo=', 'Content-Length': '0'} 2022-09-12T20:08:33.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000915fb8e08148b30d-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.039 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:33.040 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:33.040 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:33.041 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:33.041 DEBUG:boto:Method: GET 2022-09-12T20:08:33.042 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:33.042 DEBUG:boto:Data: 2022-09-12T20:08:33.043 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.044 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.044 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.045 DEBUG:boto:Params: {} 2022-09-12T20:08:33.045 DEBUG:boto:Token: None 2022-09-12T20:08:33.046 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:33.048 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zj0PgTAP6k7cGbMU/TKZxzFxQmo= 2022-09-12T20:08:33.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zj0PgTAP6k7cGbMU/TKZxzFxQmo=', 'Content-Length': '0'} 2022-09-12T20:08:33.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b130e01f3ad368e6-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.052 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:33.052 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:33.053 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:33.053 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:33.054 DEBUG:boto:Method: HEAD 2022-09-12T20:08:33.054 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:33.055 DEBUG:boto:Data: 2022-09-12T20:08:33.055 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.056 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.056 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.057 DEBUG:boto:Params: {} 2022-09-12T20:08:33.058 DEBUG:boto:Token: None 2022-09-12T20:08:33.058 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:33.059 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hQAml187UFTW45H2uEkJCQNiUrw= 2022-09-12T20:08:33.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hQAml187UFTW45H2uEkJCQNiUrw=', 'Content-Length': '0'} 2022-09-12T20:08:33.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d319115ed92e403-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.062 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:33.063 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:33.063 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:33.064 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:33.065 DEBUG:boto:Method: HEAD 2022-09-12T20:08:33.065 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:33.066 DEBUG:boto:Data: 2022-09-12T20:08:33.066 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.067 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.067 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.068 DEBUG:boto:Params: {} 2022-09-12T20:08:33.068 DEBUG:boto:Token: None 2022-09-12T20:08:33.069 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:33.069 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hQAml187UFTW45H2uEkJCQNiUrw= 2022-09-12T20:08:33.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hQAml187UFTW45H2uEkJCQNiUrw=', 'Content-Length': '0'} 2022-09-12T20:08:33.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067b233ceaa0757cc-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:08:33.073 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:33.074 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:33.075 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:33.075 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:33.076 DEBUG:boto:Method: GET 2022-09-12T20:08:33.076 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:33.077 DEBUG:boto:Data: 2022-09-12T20:08:33.077 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.078 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.079 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.079 DEBUG:boto:Params: {} 2022-09-12T20:08:33.080 DEBUG:boto:Token: None 2022-09-12T20:08:33.080 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:33.081 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zj0PgTAP6k7cGbMU/TKZxzFxQmo= 2022-09-12T20:08:33.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zj0PgTAP6k7cGbMU/TKZxzFxQmo=', 'Content-Length': '0'} 2022-09-12T20:08:33.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068d71fa9e364d8ec-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.126 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:33.127 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:33.127 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:33.128 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:33.128 DEBUG:boto:Method: GET 2022-09-12T20:08:33.129 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:33.129 DEBUG:boto:Data: 2022-09-12T20:08:33.130 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.130 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.131 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.132 DEBUG:boto:Params: {} 2022-09-12T20:08:33.132 DEBUG:boto:Token: None 2022-09-12T20:08:33.133 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:33.133 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zj0PgTAP6k7cGbMU/TKZxzFxQmo= 2022-09-12T20:08:33.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zj0PgTAP6k7cGbMU/TKZxzFxQmo=', 'Content-Length': '0'} 2022-09-12T20:08:33.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d4e2c44d942776e-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.137 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:33.138 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:33.138 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:08:33.139 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:08:33.139 DEBUG:boto:Method: GET 2022-09-12T20:08:33.140 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:08:33.140 DEBUG:boto:Data: 2022-09-12T20:08:33.141 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.141 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.142 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.142 DEBUG:boto:Params: {} 2022-09-12T20:08:33.143 DEBUG:boto:Token: None 2022-09-12T20:08:33.144 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/ 2022-09-12T20:08:33.144 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM= 2022-09-12T20:08:33.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM=', 'Content-Length': '0'} 2022-09-12T20:08:33.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4e06918dea99712-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.148 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:33.149 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:33.150 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:33.150 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:08:33.151 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:08:33.151 DEBUG:boto:Method: GET 2022-09-12T20:08:33.152 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:08:33.152 DEBUG:boto:Data: 2022-09-12T20:08:33.153 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.153 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.154 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.154 DEBUG:boto:Params: {} 2022-09-12T20:08:33.155 DEBUG:boto:Token: None 2022-09-12T20:08:33.156 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/ 2022-09-12T20:08:33.156 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM= 2022-09-12T20:08:33.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM=', 'Content-Length': '0'} 2022-09-12T20:08:33.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd493e32d20376cd-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.162 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:33.162 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:33.163 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:33.164 DEBUG:boto:Method: HEAD 2022-09-12T20:08:33.164 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:08:33.165 DEBUG:boto:Data: 2022-09-12T20:08:33.165 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.166 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.166 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.167 DEBUG:boto:Params: {} 2022-09-12T20:08:33.167 DEBUG:boto:Token: None 2022-09-12T20:08:33.168 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj44 2022-09-12T20:08:33.168 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h9Xw7JSFV9TbNs4UM1V331X5Ejw= 2022-09-12T20:08:33.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h9Xw7JSFV9TbNs4UM1V331X5Ejw=', 'Content-Length': '0'} 2022-09-12T20:08:33.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002323cf317b60e026-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.171 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:33.172 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:33.172 DEBUG:boto:Method: HEAD 2022-09-12T20:08:33.173 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:08:33.173 DEBUG:boto:Data: 2022-09-12T20:08:33.174 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.174 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.175 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.175 DEBUG:boto:Params: {} 2022-09-12T20:08:33.176 DEBUG:boto:Token: None 2022-09-12T20:08:33.176 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj44 2022-09-12T20:08:33.177 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h9Xw7JSFV9TbNs4UM1V331X5Ejw= 2022-09-12T20:08:33.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h9Xw7JSFV9TbNs4UM1V331X5Ejw=', 'Content-Length': '0'} 2022-09-12T20:08:33.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a365503d935b5f11-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:08:33.181 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:33.181 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:33.182 DEBUG:boto:Method: GET 2022-09-12T20:08:33.183 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:08:33.183 DEBUG:boto:Data: 2022-09-12T20:08:33.184 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.184 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.185 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.185 DEBUG:boto:Params: {} 2022-09-12T20:08:33.186 DEBUG:boto:Token: None 2022-09-12T20:08:33.186 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj44 2022-09-12T20:08:33.187 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nKivHnA+3Z0COedU9Hc5vz1qLOo= 2022-09-12T20:08:33.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nKivHnA+3Z0COedU9Hc5vz1qLOo=', 'Content-Length': '0'} 2022-09-12T20:08:33.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a22b5fd712b08e3-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.230 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:33.231 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:33.231 DEBUG:boto:Method: GET 2022-09-12T20:08:33.232 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:08:33.232 DEBUG:boto:Data: 2022-09-12T20:08:33.233 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.233 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.234 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.234 DEBUG:boto:Params: {} 2022-09-12T20:08:33.235 DEBUG:boto:Token: None 2022-09-12T20:08:33.236 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj44 2022-09-12T20:08:33.236 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nKivHnA+3Z0COedU9Hc5vz1qLOo= 2022-09-12T20:08:33.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nKivHnA+3Z0COedU9Hc5vz1qLOo=', 'Content-Length': '0'} 2022-09-12T20:08:33.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000592cca07b255e061-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.240 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:33.240 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:33.241 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:08:33.241 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:08:33.242 DEBUG:boto:Method: GET 2022-09-12T20:08:33.243 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:08:33.243 DEBUG:boto:Data: 2022-09-12T20:08:33.244 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.244 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.245 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.245 DEBUG:boto:Params: {} 2022-09-12T20:08:33.246 DEBUG:boto:Token: None 2022-09-12T20:08:33.246 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/ 2022-09-12T20:08:33.247 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM= 2022-09-12T20:08:33.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM=', 'Content-Length': '0'} 2022-09-12T20:08:33.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ec9acc2f9b40a2f-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.251 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:33.252 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:33.253 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:33.253 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:08:33.254 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:08:33.254 DEBUG:boto:Method: GET 2022-09-12T20:08:33.255 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:08:33.255 DEBUG:boto:Data: 2022-09-12T20:08:33.256 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.256 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.257 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.257 DEBUG:boto:Params: {} 2022-09-12T20:08:33.258 DEBUG:boto:Token: None 2022-09-12T20:08:33.259 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/ 2022-09-12T20:08:33.259 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM= 2022-09-12T20:08:33.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM=', 'Content-Length': '0'} 2022-09-12T20:08:33.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bed32ff93aa2cb5-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.264 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:33.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:08:33.266 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:33.266 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:33.267 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:33.267 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:33.268 DEBUG:boto:Method: GET 2022-09-12T20:08:33.268 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:33.269 DEBUG:boto:Data: 2022-09-12T20:08:33.269 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.270 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.270 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.271 DEBUG:boto:Params: {} 2022-09-12T20:08:33.272 DEBUG:boto:Token: None 2022-09-12T20:08:33.272 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:33.273 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X5WgjIG85XWsZzsV2lJMC8ZOWsY= 2022-09-12T20:08:33.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X5WgjIG85XWsZzsV2lJMC8ZOWsY=', 'Content-Length': '0'} 2022-09-12T20:08:33.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5b6af77629b5490-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.276 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:33.277 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:33.277 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:33.278 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:33.278 DEBUG:boto:Method: GET 2022-09-12T20:08:33.279 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:33.279 DEBUG:boto:Data: 2022-09-12T20:08:33.280 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.281 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.281 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.282 DEBUG:boto:Params: {} 2022-09-12T20:08:33.282 DEBUG:boto:Token: None 2022-09-12T20:08:33.283 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:33.283 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X5WgjIG85XWsZzsV2lJMC8ZOWsY= 2022-09-12T20:08:33.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X5WgjIG85XWsZzsV2lJMC8ZOWsY=', 'Content-Length': '0'} 2022-09-12T20:08:33.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a69451ba0cbbb3f-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.330 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:33.331 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:33.331 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:33.332 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:33.332 DEBUG:boto:Method: HEAD 2022-09-12T20:08:33.333 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:33.333 DEBUG:boto:Data: 2022-09-12T20:08:33.334 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.334 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.335 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.335 DEBUG:boto:Params: {} 2022-09-12T20:08:33.336 DEBUG:boto:Token: None 2022-09-12T20:08:33.336 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:33.337 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:U4YricG9lRnKIp1JyEvimVDJpTY= 2022-09-12T20:08:33.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:U4YricG9lRnKIp1JyEvimVDJpTY=', 'Content-Length': '0'} 2022-09-12T20:08:33.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000700b232d963a2ed2-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.340 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:33.341 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:33.341 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:33.342 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:33.343 DEBUG:boto:Method: HEAD 2022-09-12T20:08:33.343 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:33.344 DEBUG:boto:Data: 2022-09-12T20:08:33.344 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.345 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.345 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.346 DEBUG:boto:Params: {} 2022-09-12T20:08:33.346 DEBUG:boto:Token: None 2022-09-12T20:08:33.347 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:33.347 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:U4YricG9lRnKIp1JyEvimVDJpTY= 2022-09-12T20:08:33.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:U4YricG9lRnKIp1JyEvimVDJpTY=', 'Content-Length': '0'} 2022-09-12T20:08:33.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001eb0c81a557fc84a-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:08:33.351 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:33.352 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:33.353 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:33.353 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:33.354 DEBUG:boto:Method: GET 2022-09-12T20:08:33.354 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:33.355 DEBUG:boto:Data: 2022-09-12T20:08:33.355 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.356 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.357 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.357 DEBUG:boto:Params: {} 2022-09-12T20:08:33.358 DEBUG:boto:Token: None 2022-09-12T20:08:33.359 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:33.359 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X5WgjIG85XWsZzsV2lJMC8ZOWsY= 2022-09-12T20:08:33.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X5WgjIG85XWsZzsV2lJMC8ZOWsY=', 'Content-Length': '0'} 2022-09-12T20:08:33.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c50dd7890b54eac-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.363 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:33.363 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:33.364 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:33.365 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:33.365 DEBUG:boto:Method: GET 2022-09-12T20:08:33.366 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:33.366 DEBUG:boto:Data: 2022-09-12T20:08:33.367 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.367 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.368 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.368 DEBUG:boto:Params: {} 2022-09-12T20:08:33.369 DEBUG:boto:Token: None 2022-09-12T20:08:33.370 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:33.370 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X5WgjIG85XWsZzsV2lJMC8ZOWsY= 2022-09-12T20:08:33.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X5WgjIG85XWsZzsV2lJMC8ZOWsY=', 'Content-Length': '0'} 2022-09-12T20:08:33.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060599dd0b5aafad7-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.374 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:33.375 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:33.375 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:08:33.376 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:08:33.376 DEBUG:boto:Method: GET 2022-09-12T20:08:33.377 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:08:33.377 DEBUG:boto:Data: 2022-09-12T20:08:33.378 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.379 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.379 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.380 DEBUG:boto:Params: {} 2022-09-12T20:08:33.380 DEBUG:boto:Token: None 2022-09-12T20:08:33.381 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/ 2022-09-12T20:08:33.381 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM= 2022-09-12T20:08:33.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM=', 'Content-Length': '0'} 2022-09-12T20:08:33.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d3284e22a9c6bb3-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.426 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:33.427 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:33.428 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:33.428 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:08:33.429 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:08:33.429 DEBUG:boto:Method: GET 2022-09-12T20:08:33.430 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:08:33.430 DEBUG:boto:Data: 2022-09-12T20:08:33.431 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.432 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.432 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.433 DEBUG:boto:Params: {} 2022-09-12T20:08:33.434 DEBUG:boto:Token: None 2022-09-12T20:08:33.435 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/ 2022-09-12T20:08:33.435 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM= 2022-09-12T20:08:33.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM=', 'Content-Length': '0'} 2022-09-12T20:08:33.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd0375ac683a186e-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.440 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:33.441 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:33.441 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:33.442 DEBUG:boto:Method: HEAD 2022-09-12T20:08:33.442 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:08:33.443 DEBUG:boto:Data: 2022-09-12T20:08:33.444 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.444 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.445 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.446 DEBUG:boto:Params: {} 2022-09-12T20:08:33.447 DEBUG:boto:Token: None 2022-09-12T20:08:33.448 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj51 2022-09-12T20:08:33.448 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IIsqDJGunhjtjQoYETN3ez+nU+4= 2022-09-12T20:08:33.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IIsqDJGunhjtjQoYETN3ez+nU+4=', 'Content-Length': '0'} 2022-09-12T20:08:33.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cca63540b54ec88c-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.452 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:33.452 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:33.453 DEBUG:boto:Method: HEAD 2022-09-12T20:08:33.453 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:08:33.454 DEBUG:boto:Data: 2022-09-12T20:08:33.455 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.455 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.456 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.456 DEBUG:boto:Params: {} 2022-09-12T20:08:33.457 DEBUG:boto:Token: None 2022-09-12T20:08:33.457 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj51 2022-09-12T20:08:33.458 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IIsqDJGunhjtjQoYETN3ez+nU+4= 2022-09-12T20:08:33.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IIsqDJGunhjtjQoYETN3ez+nU+4=', 'Content-Length': '0'} 2022-09-12T20:08:33.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b41061036e639b66-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:08:33.462 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:33.463 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:33.463 DEBUG:boto:Method: GET 2022-09-12T20:08:33.464 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:08:33.464 DEBUG:boto:Data: 2022-09-12T20:08:33.465 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.466 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.466 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.467 DEBUG:boto:Params: {} 2022-09-12T20:08:33.468 DEBUG:boto:Token: None 2022-09-12T20:08:33.468 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj51 2022-09-12T20:08:33.469 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KrGydhqP63JoIp4iuQbBwedJ+WY= 2022-09-12T20:08:33.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KrGydhqP63JoIp4iuQbBwedJ+WY=', 'Content-Length': '0'} 2022-09-12T20:08:33.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e241153eb4cc7e9-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.472 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:33.473 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:33.473 DEBUG:boto:Method: GET 2022-09-12T20:08:33.474 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:08:33.475 DEBUG:boto:Data: 2022-09-12T20:08:33.475 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.476 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.476 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.477 DEBUG:boto:Params: {} 2022-09-12T20:08:33.478 DEBUG:boto:Token: None 2022-09-12T20:08:33.478 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj51 2022-09-12T20:08:33.479 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KrGydhqP63JoIp4iuQbBwedJ+WY= 2022-09-12T20:08:33.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KrGydhqP63JoIp4iuQbBwedJ+WY=', 'Content-Length': '0'} 2022-09-12T20:08:33.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000928d3d61a33f048c-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.522 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:33.523 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:33.523 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:08:33.524 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:08:33.524 DEBUG:boto:Method: GET 2022-09-12T20:08:33.525 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:08:33.525 DEBUG:boto:Data: 2022-09-12T20:08:33.526 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.527 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.527 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.528 DEBUG:boto:Params: {} 2022-09-12T20:08:33.528 DEBUG:boto:Token: None 2022-09-12T20:08:33.529 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/ 2022-09-12T20:08:33.529 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM= 2022-09-12T20:08:33.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM=', 'Content-Length': '0'} 2022-09-12T20:08:33.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f85056bde3a01dc3-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.534 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:33.535 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:33.535 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:33.536 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:08:33.536 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:08:33.537 DEBUG:boto:Method: GET 2022-09-12T20:08:33.537 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:08:33.538 DEBUG:boto:Data: 2022-09-12T20:08:33.538 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.539 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.539 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.540 DEBUG:boto:Params: {} 2022-09-12T20:08:33.541 DEBUG:boto:Token: None 2022-09-12T20:08:33.542 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/ 2022-09-12T20:08:33.542 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM= 2022-09-12T20:08:33.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM=', 'Content-Length': '0'} 2022-09-12T20:08:33.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000377b657eae9c0feb-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.547 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:33.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:08:33.548 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:33.549 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:33.549 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:33.550 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:33.550 DEBUG:boto:Method: GET 2022-09-12T20:08:33.551 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:33.551 DEBUG:boto:Data: 2022-09-12T20:08:33.552 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.552 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.553 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.553 DEBUG:boto:Params: {} 2022-09-12T20:08:33.554 DEBUG:boto:Token: None 2022-09-12T20:08:33.555 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:33.555 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RNps7HPwbsrY/ZNIWbz5Kaixp6s= 2022-09-12T20:08:33.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RNps7HPwbsrY/ZNIWbz5Kaixp6s=', 'Content-Length': '0'} 2022-09-12T20:08:33.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081349ac4c4699a5f-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.559 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:33.559 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:33.560 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:33.560 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:33.561 DEBUG:boto:Method: GET 2022-09-12T20:08:33.561 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:33.562 DEBUG:boto:Data: 2022-09-12T20:08:33.562 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.563 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.563 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.564 DEBUG:boto:Params: {} 2022-09-12T20:08:33.564 DEBUG:boto:Token: None 2022-09-12T20:08:33.565 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:33.566 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RNps7HPwbsrY/ZNIWbz5Kaixp6s= 2022-09-12T20:08:33.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RNps7HPwbsrY/ZNIWbz5Kaixp6s=', 'Content-Length': '0'} 2022-09-12T20:08:33.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033500deacc43fedc-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.569 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:33.569 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:33.570 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:33.570 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:33.571 DEBUG:boto:Method: HEAD 2022-09-12T20:08:33.571 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:33.572 DEBUG:boto:Data: 2022-09-12T20:08:33.572 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.573 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.573 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.574 DEBUG:boto:Params: {} 2022-09-12T20:08:33.574 DEBUG:boto:Token: None 2022-09-12T20:08:33.575 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:33.575 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KJBNEV2uSLdfMrn0srqLDFztRlA= 2022-09-12T20:08:33.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KJBNEV2uSLdfMrn0srqLDFztRlA=', 'Content-Length': '0'} 2022-09-12T20:08:33.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000466a6057ca0dee9d-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.579 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:33.579 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:33.580 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:33.580 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:33.581 DEBUG:boto:Method: HEAD 2022-09-12T20:08:33.581 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:33.581 DEBUG:boto:Data: 2022-09-12T20:08:33.582 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.582 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.583 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.583 DEBUG:boto:Params: {} 2022-09-12T20:08:33.584 DEBUG:boto:Token: None 2022-09-12T20:08:33.585 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:33.585 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KJBNEV2uSLdfMrn0srqLDFztRlA= 2022-09-12T20:08:33.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KJBNEV2uSLdfMrn0srqLDFztRlA=', 'Content-Length': '0'} 2022-09-12T20:08:33.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6e579e3ebb84469-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:08:33.589 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:33.589 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:33.590 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:33.591 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:33.591 DEBUG:boto:Method: GET 2022-09-12T20:08:33.592 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:33.592 DEBUG:boto:Data: 2022-09-12T20:08:33.593 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.593 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.594 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.594 DEBUG:boto:Params: {} 2022-09-12T20:08:33.595 DEBUG:boto:Token: None 2022-09-12T20:08:33.595 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:33.596 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RNps7HPwbsrY/ZNIWbz5Kaixp6s= 2022-09-12T20:08:33.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RNps7HPwbsrY/ZNIWbz5Kaixp6s=', 'Content-Length': '0'} 2022-09-12T20:08:33.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000671e3777c51fe10e-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.642 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:33.643 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:33.643 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:33.644 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:33.644 DEBUG:boto:Method: GET 2022-09-12T20:08:33.645 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:33.645 DEBUG:boto:Data: 2022-09-12T20:08:33.646 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.646 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.647 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.647 DEBUG:boto:Params: {} 2022-09-12T20:08:33.648 DEBUG:boto:Token: None 2022-09-12T20:08:33.649 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:33.649 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RNps7HPwbsrY/ZNIWbz5Kaixp6s= 2022-09-12T20:08:33.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RNps7HPwbsrY/ZNIWbz5Kaixp6s=', 'Content-Length': '0'} 2022-09-12T20:08:33.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca153e3a48ff0a89-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.653 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:33.653 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:33.654 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:08:33.654 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:08:33.655 DEBUG:boto:Method: GET 2022-09-12T20:08:33.655 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:08:33.656 DEBUG:boto:Data: 2022-09-12T20:08:33.656 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.657 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.658 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.658 DEBUG:boto:Params: {} 2022-09-12T20:08:33.659 DEBUG:boto:Token: None 2022-09-12T20:08:33.659 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/ 2022-09-12T20:08:33.660 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM= 2022-09-12T20:08:33.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM=', 'Content-Length': '0'} 2022-09-12T20:08:33.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af930f31814b7a17-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.664 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:33.665 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:33.665 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:33.666 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:08:33.666 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:08:33.667 DEBUG:boto:Method: GET 2022-09-12T20:08:33.667 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:08:33.668 DEBUG:boto:Data: 2022-09-12T20:08:33.668 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.669 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.669 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.670 DEBUG:boto:Params: {} 2022-09-12T20:08:33.671 DEBUG:boto:Token: None 2022-09-12T20:08:33.671 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/ 2022-09-12T20:08:33.672 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM= 2022-09-12T20:08:33.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM=', 'Content-Length': '0'} 2022-09-12T20:08:33.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d852bddedf3112a-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.677 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:33.677 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:33.678 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:33.678 DEBUG:boto:Method: HEAD 2022-09-12T20:08:33.679 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:08:33.679 DEBUG:boto:Data: 2022-09-12T20:08:33.680 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.680 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.681 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.681 DEBUG:boto:Params: {} 2022-09-12T20:08:33.682 DEBUG:boto:Token: None 2022-09-12T20:08:33.682 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj55 2022-09-12T20:08:33.683 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FA9YUFtrbCKi/1OI2sYp1lysOKg= 2022-09-12T20:08:33.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FA9YUFtrbCKi/1OI2sYp1lysOKg=', 'Content-Length': '0'} 2022-09-12T20:08:33.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025d1bacfc138d891-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.686 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:33.687 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:33.687 DEBUG:boto:Method: HEAD 2022-09-12T20:08:33.688 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:08:33.688 DEBUG:boto:Data: 2022-09-12T20:08:33.689 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.689 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.690 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.690 DEBUG:boto:Params: {} 2022-09-12T20:08:33.691 DEBUG:boto:Token: None 2022-09-12T20:08:33.691 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj55 2022-09-12T20:08:33.692 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FA9YUFtrbCKi/1OI2sYp1lysOKg= 2022-09-12T20:08:33.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FA9YUFtrbCKi/1OI2sYp1lysOKg=', 'Content-Length': '0'} 2022-09-12T20:08:33.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060204036ca301e18-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:08:33.695 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:33.696 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:33.697 DEBUG:boto:Method: GET 2022-09-12T20:08:33.697 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:08:33.698 DEBUG:boto:Data: 2022-09-12T20:08:33.698 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.699 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.699 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.700 DEBUG:boto:Params: {} 2022-09-12T20:08:33.700 DEBUG:boto:Token: None 2022-09-12T20:08:33.701 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj55 2022-09-12T20:08:33.701 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zzY6AIEqHV2eTKjjgAMyaStL3CQ= 2022-09-12T20:08:33.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zzY6AIEqHV2eTKjjgAMyaStL3CQ=', 'Content-Length': '0'} 2022-09-12T20:08:33.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001c84a0201c4ae0e-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.705 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:33.705 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:33.706 DEBUG:boto:Method: GET 2022-09-12T20:08:33.706 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:08:33.707 DEBUG:boto:Data: 2022-09-12T20:08:33.708 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.708 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.709 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.709 DEBUG:boto:Params: {} 2022-09-12T20:08:33.710 DEBUG:boto:Token: None 2022-09-12T20:08:33.710 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj55 2022-09-12T20:08:33.711 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zzY6AIEqHV2eTKjjgAMyaStL3CQ= 2022-09-12T20:08:33.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zzY6AIEqHV2eTKjjgAMyaStL3CQ=', 'Content-Length': '0'} 2022-09-12T20:08:33.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000468af05b7d299be7-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.754 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:33.755 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:33.755 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:08:33.756 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:08:33.756 DEBUG:boto:Method: GET 2022-09-12T20:08:33.757 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:08:33.758 DEBUG:boto:Data: 2022-09-12T20:08:33.758 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.759 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.759 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.760 DEBUG:boto:Params: {} 2022-09-12T20:08:33.760 DEBUG:boto:Token: None 2022-09-12T20:08:33.761 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/ 2022-09-12T20:08:33.762 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM= 2022-09-12T20:08:33.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM=', 'Content-Length': '0'} 2022-09-12T20:08:33.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000016b74dee5dde6eb-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.766 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:33.766 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:33.767 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:33.767 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:08:33.768 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:08:33.769 DEBUG:boto:Method: GET 2022-09-12T20:08:33.769 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:08:33.770 DEBUG:boto:Data: 2022-09-12T20:08:33.770 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.771 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.771 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.772 DEBUG:boto:Params: {} 2022-09-12T20:08:33.772 DEBUG:boto:Token: None 2022-09-12T20:08:33.773 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/ 2022-09-12T20:08:33.773 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM= 2022-09-12T20:08:33.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM=', 'Content-Length': '0'} 2022-09-12T20:08:33.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025c84d80eb901457-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.778 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:33.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:08:33.780 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:33.780 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:33.781 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:33.781 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:33.782 DEBUG:boto:Method: GET 2022-09-12T20:08:33.782 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:33.783 DEBUG:boto:Data: 2022-09-12T20:08:33.783 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.784 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.784 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.785 DEBUG:boto:Params: {} 2022-09-12T20:08:33.785 DEBUG:boto:Token: None 2022-09-12T20:08:33.786 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:33.787 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pXABsSDuPJmDGSTO/+3m+2VJHRk= 2022-09-12T20:08:33.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pXABsSDuPJmDGSTO/+3m+2VJHRk=', 'Content-Length': '0'} 2022-09-12T20:08:33.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031529cf60833f6ff-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.790 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:33.791 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:33.791 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:33.792 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:33.792 DEBUG:boto:Method: GET 2022-09-12T20:08:33.793 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:33.793 DEBUG:boto:Data: 2022-09-12T20:08:33.794 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.795 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.795 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.796 DEBUG:boto:Params: {} 2022-09-12T20:08:33.796 DEBUG:boto:Token: None 2022-09-12T20:08:33.797 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:33.797 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pXABsSDuPJmDGSTO/+3m+2VJHRk= 2022-09-12T20:08:33.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pXABsSDuPJmDGSTO/+3m+2VJHRk=', 'Content-Length': '0'} 2022-09-12T20:08:33.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e448fe7957553645-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.801 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:33.802 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:33.802 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:33.803 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:33.803 DEBUG:boto:Method: HEAD 2022-09-12T20:08:33.804 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:33.804 DEBUG:boto:Data: 2022-09-12T20:08:33.805 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.805 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.806 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.806 DEBUG:boto:Params: {} 2022-09-12T20:08:33.807 DEBUG:boto:Token: None 2022-09-12T20:08:33.807 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:33.808 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KsYTmWDcTiuO1GX/CnvKMAz+Txc= 2022-09-12T20:08:33.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KsYTmWDcTiuO1GX/CnvKMAz+Txc=', 'Content-Length': '0'} 2022-09-12T20:08:33.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4c23eefc65e1a0a-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.812 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:33.812 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:33.813 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:33.813 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:33.814 DEBUG:boto:Method: HEAD 2022-09-12T20:08:33.814 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:33.815 DEBUG:boto:Data: 2022-09-12T20:08:33.815 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.816 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.816 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.817 DEBUG:boto:Params: {} 2022-09-12T20:08:33.817 DEBUG:boto:Token: None 2022-09-12T20:08:33.818 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:33.818 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KsYTmWDcTiuO1GX/CnvKMAz+Txc= 2022-09-12T20:08:33.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KsYTmWDcTiuO1GX/CnvKMAz+Txc=', 'Content-Length': '0'} 2022-09-12T20:08:33.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b3efb9a1af734f2-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:08:33.822 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:33.823 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:33.824 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:33.824 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:33.825 DEBUG:boto:Method: GET 2022-09-12T20:08:33.825 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:33.826 DEBUG:boto:Data: 2022-09-12T20:08:33.826 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.827 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.827 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.828 DEBUG:boto:Params: {} 2022-09-12T20:08:33.828 DEBUG:boto:Token: None 2022-09-12T20:08:33.829 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:33.830 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pXABsSDuPJmDGSTO/+3m+2VJHRk= 2022-09-12T20:08:33.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pXABsSDuPJmDGSTO/+3m+2VJHRk=', 'Content-Length': '0'} 2022-09-12T20:08:33.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c51371c3eafef9f-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.874 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:33.875 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:33.875 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:33.876 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:33.877 DEBUG:boto:Method: GET 2022-09-12T20:08:33.877 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:33.878 DEBUG:boto:Data: 2022-09-12T20:08:33.878 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.879 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.879 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.880 DEBUG:boto:Params: {} 2022-09-12T20:08:33.880 DEBUG:boto:Token: None 2022-09-12T20:08:33.881 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:33.882 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pXABsSDuPJmDGSTO/+3m+2VJHRk= 2022-09-12T20:08:33.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pXABsSDuPJmDGSTO/+3m+2VJHRk=', 'Content-Length': '0'} 2022-09-12T20:08:33.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd86769c5793d274-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.885 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:33.886 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:33.887 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:08:33.887 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:08:33.888 DEBUG:boto:Method: GET 2022-09-12T20:08:33.888 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:08:33.889 DEBUG:boto:Data: 2022-09-12T20:08:33.889 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.890 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.890 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.891 DEBUG:boto:Params: {} 2022-09-12T20:08:33.891 DEBUG:boto:Token: None 2022-09-12T20:08:33.892 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/ 2022-09-12T20:08:33.893 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM= 2022-09-12T20:08:33.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM=', 'Content-Length': '0'} 2022-09-12T20:08:33.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8192357145c862c-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.897 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:33.898 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:33.898 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:33.899 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:08:33.899 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:08:33.900 DEBUG:boto:Method: GET 2022-09-12T20:08:33.900 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:08:33.901 DEBUG:boto:Data: 2022-09-12T20:08:33.902 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.902 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.903 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.903 DEBUG:boto:Params: {} 2022-09-12T20:08:33.904 DEBUG:boto:Token: None 2022-09-12T20:08:33.904 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/ 2022-09-12T20:08:33.905 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM= 2022-09-12T20:08:33.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM=', 'Content-Length': '0'} 2022-09-12T20:08:33.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5c2bfe2b27f683f-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.909 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:33.910 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:33.911 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:33.911 DEBUG:boto:Method: HEAD 2022-09-12T20:08:33.912 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:08:33.912 DEBUG:boto:Data: 2022-09-12T20:08:33.913 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.913 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.914 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.914 DEBUG:boto:Params: {} 2022-09-12T20:08:33.915 DEBUG:boto:Token: None 2022-09-12T20:08:33.915 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj57 2022-09-12T20:08:33.916 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0uomwlp0zPo3w6bnY9fJY9uWvGU= 2022-09-12T20:08:33.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0uomwlp0zPo3w6bnY9fJY9uWvGU=', 'Content-Length': '0'} 2022-09-12T20:08:33.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb18ee27d7161caa-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.920 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:33.920 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:33.921 DEBUG:boto:Method: HEAD 2022-09-12T20:08:33.921 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:08:33.922 DEBUG:boto:Data: 2022-09-12T20:08:33.922 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.923 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.923 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.924 DEBUG:boto:Params: {} 2022-09-12T20:08:33.924 DEBUG:boto:Token: None 2022-09-12T20:08:33.925 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj57 2022-09-12T20:08:33.925 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0uomwlp0zPo3w6bnY9fJY9uWvGU= 2022-09-12T20:08:33.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0uomwlp0zPo3w6bnY9fJY9uWvGU=', 'Content-Length': '0'} 2022-09-12T20:08:33.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dd86b6a3f869a7d-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:08:33.929 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:33.930 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:33.931 DEBUG:boto:Method: GET 2022-09-12T20:08:33.931 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:08:33.932 DEBUG:boto:Data: 2022-09-12T20:08:33.932 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.933 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.933 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.934 DEBUG:boto:Params: {} 2022-09-12T20:08:33.934 DEBUG:boto:Token: None 2022-09-12T20:08:33.935 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj57 2022-09-12T20:08:33.936 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dR3vFpLo6O4UWQEXPrehk9Dqh5E= 2022-09-12T20:08:33.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dR3vFpLo6O4UWQEXPrehk9Dqh5E=', 'Content-Length': '0'} 2022-09-12T20:08:33.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e91b07dc36d99ad5-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.939 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:33.940 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:33.940 DEBUG:boto:Method: GET 2022-09-12T20:08:33.941 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:08:33.941 DEBUG:boto:Data: 2022-09-12T20:08:33.942 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.942 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:33.943 DEBUG:boto:Port: 8000 2022-09-12T20:08:33.943 DEBUG:boto:Params: {} 2022-09-12T20:08:33.944 DEBUG:boto:Token: None 2022-09-12T20:08:33.945 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/obj57 2022-09-12T20:08:33.945 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dR3vFpLo6O4UWQEXPrehk9Dqh5E= 2022-09-12T20:08:33.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dR3vFpLo6O4UWQEXPrehk9Dqh5E=', 'Content-Length': '0'} 2022-09-12T20:08:33.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024bb4f4af7b8f5a7-00631f91c1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:33.949 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:33.949 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:33.950 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:08:33.950 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:08:33.951 DEBUG:boto:Method: GET 2022-09-12T20:08:33.951 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:08:33.952 DEBUG:boto:Data: 2022-09-12T20:08:33.953 DEBUG:boto:Headers: {} 2022-09-12T20:08:33.953 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:33.954 DEBUG:boto:Port: 8001 2022-09-12T20:08:33.954 DEBUG:boto:Params: {} 2022-09-12T20:08:33.955 DEBUG:boto:Token: None 2022-09-12T20:08:33.955 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:33 GMT /jdpiak-75/ 2022-09-12T20:08:33.956 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM= 2022-09-12T20:08:33.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:33 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8c3pcEJ48arSPD5nUS7lLmIJOLM=', 'Content-Length': '0'} 2022-09-12T20:08:33.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8ddb523e3ca9338-00631f91c1-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.002 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:34.003 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:34.004 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:34.004 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:08:34.005 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:08:34.005 DEBUG:boto:Method: GET 2022-09-12T20:08:34.006 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:08:34.007 DEBUG:boto:Data: 2022-09-12T20:08:34.007 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.008 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.008 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.009 DEBUG:boto:Params: {} 2022-09-12T20:08:34.009 DEBUG:boto:Token: None 2022-09-12T20:08:34.010 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/ 2022-09-12T20:08:34.011 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U= 2022-09-12T20:08:34.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U=', 'Content-Length': '0'} 2022-09-12T20:08:34.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021499ebacb81f6d9-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.016 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:34.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:08:34.017 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:34.018 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:34.018 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:34.019 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:34.020 DEBUG:boto:Method: GET 2022-09-12T20:08:34.020 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:34.021 DEBUG:boto:Data: 2022-09-12T20:08:34.021 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.022 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.023 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.023 DEBUG:boto:Params: {} 2022-09-12T20:08:34.024 DEBUG:boto:Token: None 2022-09-12T20:08:34.025 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:34.025 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oaSx4CM73Yn+q/Z91FGaIMWdbfQ= 2022-09-12T20:08:34.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oaSx4CM73Yn+q/Z91FGaIMWdbfQ=', 'Content-Length': '0'} 2022-09-12T20:08:34.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2ee48c2881f4409-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.029 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:34.029 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:34.030 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:34.030 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:34.031 DEBUG:boto:Method: GET 2022-09-12T20:08:34.032 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:34.032 DEBUG:boto:Data: 2022-09-12T20:08:34.033 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.033 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.034 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.034 DEBUG:boto:Params: {} 2022-09-12T20:08:34.035 DEBUG:boto:Token: None 2022-09-12T20:08:34.036 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:34.036 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oaSx4CM73Yn+q/Z91FGaIMWdbfQ= 2022-09-12T20:08:34.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oaSx4CM73Yn+q/Z91FGaIMWdbfQ=', 'Content-Length': '0'} 2022-09-12T20:08:34.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8b233d0140a3b5b-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.082 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:34.083 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:34.083 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:34.084 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:34.084 DEBUG:boto:Method: HEAD 2022-09-12T20:08:34.085 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:34.085 DEBUG:boto:Data: 2022-09-12T20:08:34.086 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.087 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.087 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.088 DEBUG:boto:Params: {} 2022-09-12T20:08:34.088 DEBUG:boto:Token: None 2022-09-12T20:08:34.089 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:34.089 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NiHsKEIFYTRAiG16IKcD5gdaEIQ= 2022-09-12T20:08:34.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NiHsKEIFYTRAiG16IKcD5gdaEIQ=', 'Content-Length': '0'} 2022-09-12T20:08:34.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9efd4cd793c7e68-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.093 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:34.093 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:34.094 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:34.094 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:34.095 DEBUG:boto:Method: HEAD 2022-09-12T20:08:34.095 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:34.096 DEBUG:boto:Data: 2022-09-12T20:08:34.096 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.097 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.097 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.098 DEBUG:boto:Params: {} 2022-09-12T20:08:34.098 DEBUG:boto:Token: None 2022-09-12T20:08:34.099 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:34.100 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NiHsKEIFYTRAiG16IKcD5gdaEIQ= 2022-09-12T20:08:34.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NiHsKEIFYTRAiG16IKcD5gdaEIQ=', 'Content-Length': '0'} 2022-09-12T20:08:34.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009443126cadd33c7b-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:08:34.103 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:34.104 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:34.104 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:34.105 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:34.105 DEBUG:boto:Method: GET 2022-09-12T20:08:34.106 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:34.106 DEBUG:boto:Data: 2022-09-12T20:08:34.107 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.107 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.108 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.108 DEBUG:boto:Params: {} 2022-09-12T20:08:34.109 DEBUG:boto:Token: None 2022-09-12T20:08:34.109 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:34.110 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oaSx4CM73Yn+q/Z91FGaIMWdbfQ= 2022-09-12T20:08:34.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oaSx4CM73Yn+q/Z91FGaIMWdbfQ=', 'Content-Length': '0'} 2022-09-12T20:08:34.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003437b469196c9a35-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.113 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:34.114 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:34.115 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:34.115 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:34.116 DEBUG:boto:Method: GET 2022-09-12T20:08:34.116 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:34.117 DEBUG:boto:Data: 2022-09-12T20:08:34.117 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.118 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.118 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.119 DEBUG:boto:Params: {} 2022-09-12T20:08:34.119 DEBUG:boto:Token: None 2022-09-12T20:08:34.120 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:34.121 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oaSx4CM73Yn+q/Z91FGaIMWdbfQ= 2022-09-12T20:08:34.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oaSx4CM73Yn+q/Z91FGaIMWdbfQ=', 'Content-Length': '0'} 2022-09-12T20:08:34.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac1c5b4bc2ecc40d-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.124 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:34.125 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:34.125 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:08:34.126 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:08:34.126 DEBUG:boto:Method: GET 2022-09-12T20:08:34.127 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:08:34.127 DEBUG:boto:Data: 2022-09-12T20:08:34.128 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.128 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.129 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.129 DEBUG:boto:Params: {} 2022-09-12T20:08:34.130 DEBUG:boto:Token: None 2022-09-12T20:08:34.130 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/ 2022-09-12T20:08:34.131 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U= 2022-09-12T20:08:34.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U=', 'Content-Length': '0'} 2022-09-12T20:08:34.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007479d118013778a7-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.135 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:34.136 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:34.136 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:34.137 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:08:34.138 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:08:34.138 DEBUG:boto:Method: GET 2022-09-12T20:08:34.139 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:08:34.139 DEBUG:boto:Data: 2022-09-12T20:08:34.140 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.140 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.141 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.141 DEBUG:boto:Params: {} 2022-09-12T20:08:34.142 DEBUG:boto:Token: None 2022-09-12T20:08:34.143 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/ 2022-09-12T20:08:34.143 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U= 2022-09-12T20:08:34.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U=', 'Content-Length': '0'} 2022-09-12T20:08:34.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc24b3304a50bdc2-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.148 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:34.149 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:34.149 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:34.150 DEBUG:boto:Method: HEAD 2022-09-12T20:08:34.150 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:08:34.151 DEBUG:boto:Data: 2022-09-12T20:08:34.151 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.152 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.153 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.153 DEBUG:boto:Params: {} 2022-09-12T20:08:34.154 DEBUG:boto:Token: None 2022-09-12T20:08:34.154 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj68 2022-09-12T20:08:34.155 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/FNqq+ykjydEKguH5rmT2A2KxuQ= 2022-09-12T20:08:34.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/FNqq+ykjydEKguH5rmT2A2KxuQ=', 'Content-Length': '0'} 2022-09-12T20:08:34.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000789664c6510c78d0-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.159 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:34.159 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:34.160 DEBUG:boto:Method: HEAD 2022-09-12T20:08:34.160 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:08:34.161 DEBUG:boto:Data: 2022-09-12T20:08:34.161 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.162 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.162 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.163 DEBUG:boto:Params: {} 2022-09-12T20:08:34.163 DEBUG:boto:Token: None 2022-09-12T20:08:34.164 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj68 2022-09-12T20:08:34.164 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/FNqq+ykjydEKguH5rmT2A2KxuQ= 2022-09-12T20:08:34.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/FNqq+ykjydEKguH5rmT2A2KxuQ=', 'Content-Length': '0'} 2022-09-12T20:08:34.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e87b32b43e96e0a-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:08:34.168 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:34.169 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:34.169 DEBUG:boto:Method: GET 2022-09-12T20:08:34.170 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:08:34.171 DEBUG:boto:Data: 2022-09-12T20:08:34.171 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.172 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.172 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.173 DEBUG:boto:Params: {} 2022-09-12T20:08:34.174 DEBUG:boto:Token: None 2022-09-12T20:08:34.175 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj68 2022-09-12T20:08:34.175 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:59ONw1Tw78Q2wBDpVsr6hFmMueo= 2022-09-12T20:08:34.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:59ONw1Tw78Q2wBDpVsr6hFmMueo=', 'Content-Length': '0'} 2022-09-12T20:08:34.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c729148197618eb5-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.179 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:34.179 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:34.180 DEBUG:boto:Method: GET 2022-09-12T20:08:34.180 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:08:34.181 DEBUG:boto:Data: 2022-09-12T20:08:34.182 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.182 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.183 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.183 DEBUG:boto:Params: {} 2022-09-12T20:08:34.184 DEBUG:boto:Token: None 2022-09-12T20:08:34.185 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj68 2022-09-12T20:08:34.185 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:59ONw1Tw78Q2wBDpVsr6hFmMueo= 2022-09-12T20:08:34.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:59ONw1Tw78Q2wBDpVsr6hFmMueo=', 'Content-Length': '0'} 2022-09-12T20:08:34.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7d65399f0bf55fc-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.190 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:34.190 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:34.191 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:08:34.191 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:08:34.192 DEBUG:boto:Method: GET 2022-09-12T20:08:34.192 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:08:34.193 DEBUG:boto:Data: 2022-09-12T20:08:34.194 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.194 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.195 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.195 DEBUG:boto:Params: {} 2022-09-12T20:08:34.196 DEBUG:boto:Token: None 2022-09-12T20:08:34.197 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/ 2022-09-12T20:08:34.197 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U= 2022-09-12T20:08:34.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U=', 'Content-Length': '0'} 2022-09-12T20:08:34.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070a5b04a591cd0d3-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.242 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:34.243 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:34.244 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:34.244 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:08:34.245 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:08:34.245 DEBUG:boto:Method: GET 2022-09-12T20:08:34.246 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:08:34.247 DEBUG:boto:Data: 2022-09-12T20:08:34.248 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.249 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.249 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.250 DEBUG:boto:Params: {} 2022-09-12T20:08:34.251 DEBUG:boto:Token: None 2022-09-12T20:08:34.252 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/ 2022-09-12T20:08:34.253 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U= 2022-09-12T20:08:34.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U=', 'Content-Length': '0'} 2022-09-12T20:08:34.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4cbe9a0fb81a1c4-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.258 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:34.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:08:34.260 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:34.261 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:34.261 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:34.262 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:34.263 DEBUG:boto:Method: GET 2022-09-12T20:08:34.264 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:34.265 DEBUG:boto:Data: 2022-09-12T20:08:34.265 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.266 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.267 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.267 DEBUG:boto:Params: {} 2022-09-12T20:08:34.268 DEBUG:boto:Token: None 2022-09-12T20:08:34.269 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:34.270 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ufzzw6jqhu2Lgj1vddyYhEarw+k= 2022-09-12T20:08:34.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ufzzw6jqhu2Lgj1vddyYhEarw+k=', 'Content-Length': '0'} 2022-09-12T20:08:34.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000880fca5ae91f6669-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.273 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:34.274 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:34.274 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:34.275 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:34.276 DEBUG:boto:Method: GET 2022-09-12T20:08:34.276 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:34.277 DEBUG:boto:Data: 2022-09-12T20:08:34.277 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.278 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.278 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.279 DEBUG:boto:Params: {} 2022-09-12T20:08:34.280 DEBUG:boto:Token: None 2022-09-12T20:08:34.280 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:34.281 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ufzzw6jqhu2Lgj1vddyYhEarw+k= 2022-09-12T20:08:34.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ufzzw6jqhu2Lgj1vddyYhEarw+k=', 'Content-Length': '0'} 2022-09-12T20:08:34.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007184e4d7d1893d4-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.285 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:34.286 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:34.287 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:34.287 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:34.288 DEBUG:boto:Method: HEAD 2022-09-12T20:08:34.289 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:34.290 DEBUG:boto:Data: 2022-09-12T20:08:34.291 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.292 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.293 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.293 DEBUG:boto:Params: {} 2022-09-12T20:08:34.294 DEBUG:boto:Token: None 2022-09-12T20:08:34.295 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:34.296 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9+dIZh4+jkHhAwD9h9tC+3HMlBE= 2022-09-12T20:08:34.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9+dIZh4+jkHhAwD9h9tC+3HMlBE=', 'Content-Length': '0'} 2022-09-12T20:08:34.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fb16ec574fa7e57-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.300 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:34.301 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:34.302 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:34.303 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:34.304 DEBUG:boto:Method: HEAD 2022-09-12T20:08:34.305 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:34.305 DEBUG:boto:Data: 2022-09-12T20:08:34.306 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.307 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.308 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.309 DEBUG:boto:Params: {} 2022-09-12T20:08:34.310 DEBUG:boto:Token: None 2022-09-12T20:08:34.311 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:34.312 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9+dIZh4+jkHhAwD9h9tC+3HMlBE= 2022-09-12T20:08:34.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9+dIZh4+jkHhAwD9h9tC+3HMlBE=', 'Content-Length': '0'} 2022-09-12T20:08:34.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005590d625597475b4-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:08:34.317 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:34.317 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:34.318 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:34.318 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:34.319 DEBUG:boto:Method: GET 2022-09-12T20:08:34.320 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:34.320 DEBUG:boto:Data: 2022-09-12T20:08:34.321 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.321 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.322 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.322 DEBUG:boto:Params: {} 2022-09-12T20:08:34.323 DEBUG:boto:Token: None 2022-09-12T20:08:34.323 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:34.324 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ufzzw6jqhu2Lgj1vddyYhEarw+k= 2022-09-12T20:08:34.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ufzzw6jqhu2Lgj1vddyYhEarw+k=', 'Content-Length': '0'} 2022-09-12T20:08:34.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002877c122fa7369ec-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.327 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:34.328 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:34.329 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:34.329 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:34.330 DEBUG:boto:Method: GET 2022-09-12T20:08:34.331 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:34.331 DEBUG:boto:Data: 2022-09-12T20:08:34.332 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.333 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.333 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.334 DEBUG:boto:Params: {} 2022-09-12T20:08:34.334 DEBUG:boto:Token: None 2022-09-12T20:08:34.335 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:34.336 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ufzzw6jqhu2Lgj1vddyYhEarw+k= 2022-09-12T20:08:34.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ufzzw6jqhu2Lgj1vddyYhEarw+k=', 'Content-Length': '0'} 2022-09-12T20:08:34.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e35a1e4cd5ab47c-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.339 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:34.340 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:34.340 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:08:34.341 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:08:34.341 DEBUG:boto:Method: GET 2022-09-12T20:08:34.342 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:08:34.342 DEBUG:boto:Data: 2022-09-12T20:08:34.343 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.343 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.344 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.344 DEBUG:boto:Params: {} 2022-09-12T20:08:34.345 DEBUG:boto:Token: None 2022-09-12T20:08:34.346 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/ 2022-09-12T20:08:34.346 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U= 2022-09-12T20:08:34.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U=', 'Content-Length': '0'} 2022-09-12T20:08:34.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075a4fcd3ef135232-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.350 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:34.351 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:34.351 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:34.352 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:08:34.352 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:08:34.353 DEBUG:boto:Method: GET 2022-09-12T20:08:34.353 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:08:34.354 DEBUG:boto:Data: 2022-09-12T20:08:34.355 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.355 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.356 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.356 DEBUG:boto:Params: {} 2022-09-12T20:08:34.357 DEBUG:boto:Token: None 2022-09-12T20:08:34.357 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/ 2022-09-12T20:08:34.358 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U= 2022-09-12T20:08:34.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U=', 'Content-Length': '0'} 2022-09-12T20:08:34.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000163551f3b459e4f6-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.402 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:34.403 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:34.404 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:34.405 DEBUG:boto:Method: HEAD 2022-09-12T20:08:34.405 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:08:34.406 DEBUG:boto:Data: 2022-09-12T20:08:34.406 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.407 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.408 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.408 DEBUG:boto:Params: {} 2022-09-12T20:08:34.409 DEBUG:boto:Token: None 2022-09-12T20:08:34.409 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj76 2022-09-12T20:08:34.410 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HQu8gph+VvmoPgsYOtWGTaOHRi8= 2022-09-12T20:08:34.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HQu8gph+VvmoPgsYOtWGTaOHRi8=', 'Content-Length': '0'} 2022-09-12T20:08:34.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1393739e1d01c46-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.413 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:34.414 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:34.415 DEBUG:boto:Method: HEAD 2022-09-12T20:08:34.415 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:08:34.416 DEBUG:boto:Data: 2022-09-12T20:08:34.416 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.417 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.417 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.418 DEBUG:boto:Params: {} 2022-09-12T20:08:34.418 DEBUG:boto:Token: None 2022-09-12T20:08:34.419 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj76 2022-09-12T20:08:34.419 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HQu8gph+VvmoPgsYOtWGTaOHRi8= 2022-09-12T20:08:34.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HQu8gph+VvmoPgsYOtWGTaOHRi8=', 'Content-Length': '0'} 2022-09-12T20:08:34.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8e08b04b80e3736-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:08:34.423 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:34.424 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:34.424 DEBUG:boto:Method: GET 2022-09-12T20:08:34.425 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:08:34.425 DEBUG:boto:Data: 2022-09-12T20:08:34.426 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.426 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.427 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.427 DEBUG:boto:Params: {} 2022-09-12T20:08:34.428 DEBUG:boto:Token: None 2022-09-12T20:08:34.429 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj76 2022-09-12T20:08:34.429 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DIvgEuX2TMBVsWWjH/8gKLIFxlA= 2022-09-12T20:08:34.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DIvgEuX2TMBVsWWjH/8gKLIFxlA=', 'Content-Length': '0'} 2022-09-12T20:08:34.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cb0878ae06410ba-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.433 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:34.433 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:34.434 DEBUG:boto:Method: GET 2022-09-12T20:08:34.434 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:08:34.435 DEBUG:boto:Data: 2022-09-12T20:08:34.435 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.436 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.436 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.437 DEBUG:boto:Params: {} 2022-09-12T20:08:34.437 DEBUG:boto:Token: None 2022-09-12T20:08:34.438 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj76 2022-09-12T20:08:34.439 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DIvgEuX2TMBVsWWjH/8gKLIFxlA= 2022-09-12T20:08:34.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DIvgEuX2TMBVsWWjH/8gKLIFxlA=', 'Content-Length': '0'} 2022-09-12T20:08:34.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022dfe59d27385d73-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.442 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:34.443 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:34.443 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:08:34.444 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:08:34.444 DEBUG:boto:Method: GET 2022-09-12T20:08:34.445 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:08:34.445 DEBUG:boto:Data: 2022-09-12T20:08:34.446 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.446 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.447 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.448 DEBUG:boto:Params: {} 2022-09-12T20:08:34.448 DEBUG:boto:Token: None 2022-09-12T20:08:34.449 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/ 2022-09-12T20:08:34.449 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U= 2022-09-12T20:08:34.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U=', 'Content-Length': '0'} 2022-09-12T20:08:34.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000972527b1b40ff608-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.453 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:34.454 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:34.455 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:34.455 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:08:34.456 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:08:34.456 DEBUG:boto:Method: GET 2022-09-12T20:08:34.457 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:08:34.457 DEBUG:boto:Data: 2022-09-12T20:08:34.458 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.459 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.459 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.460 DEBUG:boto:Params: {} 2022-09-12T20:08:34.460 DEBUG:boto:Token: None 2022-09-12T20:08:34.461 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/ 2022-09-12T20:08:34.461 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U= 2022-09-12T20:08:34.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U=', 'Content-Length': '0'} 2022-09-12T20:08:34.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e583dda3d28c7951-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.466 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:34.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:08:34.467 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:34.467 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:34.468 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:34.469 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:34.469 DEBUG:boto:Method: GET 2022-09-12T20:08:34.470 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:34.470 DEBUG:boto:Data: 2022-09-12T20:08:34.471 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.471 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.472 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.472 DEBUG:boto:Params: {} 2022-09-12T20:08:34.473 DEBUG:boto:Token: None 2022-09-12T20:08:34.473 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:34.474 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rxKx/VplrvyWEiJ8im/JQNw92RM= 2022-09-12T20:08:34.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rxKx/VplrvyWEiJ8im/JQNw92RM=', 'Content-Length': '0'} 2022-09-12T20:08:34.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d7227e96d6e7309-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.477 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:34.478 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:34.479 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:34.479 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:34.480 DEBUG:boto:Method: GET 2022-09-12T20:08:34.480 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:34.481 DEBUG:boto:Data: 2022-09-12T20:08:34.481 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.482 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.482 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.483 DEBUG:boto:Params: {} 2022-09-12T20:08:34.483 DEBUG:boto:Token: None 2022-09-12T20:08:34.484 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:34.485 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rxKx/VplrvyWEiJ8im/JQNw92RM= 2022-09-12T20:08:34.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rxKx/VplrvyWEiJ8im/JQNw92RM=', 'Content-Length': '0'} 2022-09-12T20:08:34.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d25f116d75697a6d-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.488 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:34.489 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:34.489 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:34.490 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:34.491 DEBUG:boto:Method: HEAD 2022-09-12T20:08:34.491 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:34.492 DEBUG:boto:Data: 2022-09-12T20:08:34.492 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.493 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.493 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.493 DEBUG:boto:Params: {} 2022-09-12T20:08:34.494 DEBUG:boto:Token: None 2022-09-12T20:08:34.495 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:34.495 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zR4Xo2bzFtdCWGsWM5iaYaAZgEw= 2022-09-12T20:08:34.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zR4Xo2bzFtdCWGsWM5iaYaAZgEw=', 'Content-Length': '0'} 2022-09-12T20:08:34.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000681dc7dad5e247b2-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.498 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:34.499 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:34.500 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:34.500 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:34.501 DEBUG:boto:Method: HEAD 2022-09-12T20:08:34.501 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:34.502 DEBUG:boto:Data: 2022-09-12T20:08:34.502 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.503 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.503 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.504 DEBUG:boto:Params: {} 2022-09-12T20:08:34.504 DEBUG:boto:Token: None 2022-09-12T20:08:34.505 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:34.505 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zR4Xo2bzFtdCWGsWM5iaYaAZgEw= 2022-09-12T20:08:34.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zR4Xo2bzFtdCWGsWM5iaYaAZgEw=', 'Content-Length': '0'} 2022-09-12T20:08:34.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000530e8df474da81f7-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:08:34.509 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:34.510 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:34.510 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:34.511 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:34.511 DEBUG:boto:Method: GET 2022-09-12T20:08:34.512 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:34.512 DEBUG:boto:Data: 2022-09-12T20:08:34.513 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.513 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.514 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.514 DEBUG:boto:Params: {} 2022-09-12T20:08:34.515 DEBUG:boto:Token: None 2022-09-12T20:08:34.516 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:34.516 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rxKx/VplrvyWEiJ8im/JQNw92RM= 2022-09-12T20:08:34.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rxKx/VplrvyWEiJ8im/JQNw92RM=', 'Content-Length': '0'} 2022-09-12T20:08:34.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000072e41a6bef776a4-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.520 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:34.520 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:34.521 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:34.521 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:34.522 DEBUG:boto:Method: GET 2022-09-12T20:08:34.522 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:34.523 DEBUG:boto:Data: 2022-09-12T20:08:34.524 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.524 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.525 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.525 DEBUG:boto:Params: {} 2022-09-12T20:08:34.526 DEBUG:boto:Token: None 2022-09-12T20:08:34.526 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:34.527 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rxKx/VplrvyWEiJ8im/JQNw92RM= 2022-09-12T20:08:34.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rxKx/VplrvyWEiJ8im/JQNw92RM=', 'Content-Length': '0'} 2022-09-12T20:08:34.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7f4a3787c6147ec-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.530 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:34.531 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:34.531 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:08:34.532 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:08:34.533 DEBUG:boto:Method: GET 2022-09-12T20:08:34.533 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:08:34.534 DEBUG:boto:Data: 2022-09-12T20:08:34.534 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.535 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.535 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.536 DEBUG:boto:Params: {} 2022-09-12T20:08:34.536 DEBUG:boto:Token: None 2022-09-12T20:08:34.537 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/ 2022-09-12T20:08:34.538 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U= 2022-09-12T20:08:34.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U=', 'Content-Length': '0'} 2022-09-12T20:08:34.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fddcdb54e617bc1a-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.542 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:34.542 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:34.543 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:34.544 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:08:34.544 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:08:34.545 DEBUG:boto:Method: GET 2022-09-12T20:08:34.545 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:08:34.546 DEBUG:boto:Data: 2022-09-12T20:08:34.546 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.547 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.547 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.548 DEBUG:boto:Params: {} 2022-09-12T20:08:34.548 DEBUG:boto:Token: None 2022-09-12T20:08:34.549 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/ 2022-09-12T20:08:34.550 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U= 2022-09-12T20:08:34.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U=', 'Content-Length': '0'} 2022-09-12T20:08:34.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccaf24034a11544e-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.554 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:34.555 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:34.555 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:34.556 DEBUG:boto:Method: HEAD 2022-09-12T20:08:34.556 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:08:34.557 DEBUG:boto:Data: 2022-09-12T20:08:34.557 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.558 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.558 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.559 DEBUG:boto:Params: {} 2022-09-12T20:08:34.560 DEBUG:boto:Token: None 2022-09-12T20:08:34.560 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj85 2022-09-12T20:08:34.561 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RbkT9Bhi5mBeJzBbwZnmQSFu6UQ= 2022-09-12T20:08:34.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RbkT9Bhi5mBeJzBbwZnmQSFu6UQ=', 'Content-Length': '0'} 2022-09-12T20:08:34.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000416cea60256c2974-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.564 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:34.565 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:34.565 DEBUG:boto:Method: HEAD 2022-09-12T20:08:34.566 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:08:34.566 DEBUG:boto:Data: 2022-09-12T20:08:34.567 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.567 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.568 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.568 DEBUG:boto:Params: {} 2022-09-12T20:08:34.569 DEBUG:boto:Token: None 2022-09-12T20:08:34.569 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj85 2022-09-12T20:08:34.570 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RbkT9Bhi5mBeJzBbwZnmQSFu6UQ= 2022-09-12T20:08:34.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RbkT9Bhi5mBeJzBbwZnmQSFu6UQ=', 'Content-Length': '0'} 2022-09-12T20:08:34.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ba6bfffb2759924-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:08:34.574 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:34.574 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:34.575 DEBUG:boto:Method: GET 2022-09-12T20:08:34.575 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:08:34.576 DEBUG:boto:Data: 2022-09-12T20:08:34.577 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.577 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.578 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.578 DEBUG:boto:Params: {} 2022-09-12T20:08:34.579 DEBUG:boto:Token: None 2022-09-12T20:08:34.579 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj85 2022-09-12T20:08:34.580 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wSC9pxplSDtBTQslmmkDxazLL6o= 2022-09-12T20:08:34.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wSC9pxplSDtBTQslmmkDxazLL6o=', 'Content-Length': '0'} 2022-09-12T20:08:34.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085a115080e5580ec-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.583 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:34.584 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:34.584 DEBUG:boto:Method: GET 2022-09-12T20:08:34.585 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:08:34.585 DEBUG:boto:Data: 2022-09-12T20:08:34.586 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.586 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.587 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.588 DEBUG:boto:Params: {} 2022-09-12T20:08:34.588 DEBUG:boto:Token: None 2022-09-12T20:08:34.589 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj85 2022-09-12T20:08:34.589 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wSC9pxplSDtBTQslmmkDxazLL6o= 2022-09-12T20:08:34.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wSC9pxplSDtBTQslmmkDxazLL6o=', 'Content-Length': '0'} 2022-09-12T20:08:34.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7cc59b112fb4010-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.593 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:34.593 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:34.594 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:08:34.595 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:08:34.595 DEBUG:boto:Method: GET 2022-09-12T20:08:34.596 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:08:34.596 DEBUG:boto:Data: 2022-09-12T20:08:34.597 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.597 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.598 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.598 DEBUG:boto:Params: {} 2022-09-12T20:08:34.599 DEBUG:boto:Token: None 2022-09-12T20:08:34.599 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/ 2022-09-12T20:08:34.600 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U= 2022-09-12T20:08:34.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U=', 'Content-Length': '0'} 2022-09-12T20:08:34.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003524818e8ada187f-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.646 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:34.647 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:34.648 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:34.648 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:08:34.649 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:08:34.649 DEBUG:boto:Method: GET 2022-09-12T20:08:34.650 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:08:34.650 DEBUG:boto:Data: 2022-09-12T20:08:34.651 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.651 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.652 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.652 DEBUG:boto:Params: {} 2022-09-12T20:08:34.653 DEBUG:boto:Token: None 2022-09-12T20:08:34.654 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/ 2022-09-12T20:08:34.654 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U= 2022-09-12T20:08:34.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U=', 'Content-Length': '0'} 2022-09-12T20:08:34.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025c84c346ae58eba-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.659 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:34.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:08:34.660 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:34.660 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:34.661 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:34.661 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:34.662 DEBUG:boto:Method: GET 2022-09-12T20:08:34.663 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:34.663 DEBUG:boto:Data: 2022-09-12T20:08:34.664 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.664 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.665 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.665 DEBUG:boto:Params: {} 2022-09-12T20:08:34.666 DEBUG:boto:Token: None 2022-09-12T20:08:34.667 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:34.667 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2K83yajW18cCwbdBwvuj9Pg2S4Q= 2022-09-12T20:08:34.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2K83yajW18cCwbdBwvuj9Pg2S4Q=', 'Content-Length': '0'} 2022-09-12T20:08:34.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bff10c85d51902ce-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.671 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:34.671 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:34.672 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:34.672 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:34.673 DEBUG:boto:Method: GET 2022-09-12T20:08:34.673 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:34.674 DEBUG:boto:Data: 2022-09-12T20:08:34.675 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.675 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.676 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.676 DEBUG:boto:Params: {} 2022-09-12T20:08:34.677 DEBUG:boto:Token: None 2022-09-12T20:08:34.677 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:34.678 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2K83yajW18cCwbdBwvuj9Pg2S4Q= 2022-09-12T20:08:34.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2K83yajW18cCwbdBwvuj9Pg2S4Q=', 'Content-Length': '0'} 2022-09-12T20:08:34.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000332ef594533eb239-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.726 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:34.727 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:34.727 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:34.728 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:34.728 DEBUG:boto:Method: HEAD 2022-09-12T20:08:34.729 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:34.729 DEBUG:boto:Data: 2022-09-12T20:08:34.730 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.730 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.731 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.731 DEBUG:boto:Params: {} 2022-09-12T20:08:34.732 DEBUG:boto:Token: None 2022-09-12T20:08:34.732 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:34.733 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QUmrQrQRSOUiwgwBBj+C2Mo/Ov0= 2022-09-12T20:08:34.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QUmrQrQRSOUiwgwBBj+C2Mo/Ov0=', 'Content-Length': '0'} 2022-09-12T20:08:34.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9d99ada13271cd6-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.737 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:34.737 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:34.738 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:34.738 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:34.739 DEBUG:boto:Method: HEAD 2022-09-12T20:08:34.739 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:34.740 DEBUG:boto:Data: 2022-09-12T20:08:34.740 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.741 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.741 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.742 DEBUG:boto:Params: {} 2022-09-12T20:08:34.742 DEBUG:boto:Token: None 2022-09-12T20:08:34.743 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:34.743 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QUmrQrQRSOUiwgwBBj+C2Mo/Ov0= 2022-09-12T20:08:34.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QUmrQrQRSOUiwgwBBj+C2Mo/Ov0=', 'Content-Length': '0'} 2022-09-12T20:08:34.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001507e6699f30ee37-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:08:34.747 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:34.748 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:34.748 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:34.749 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:34.749 DEBUG:boto:Method: GET 2022-09-12T20:08:34.750 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:34.750 DEBUG:boto:Data: 2022-09-12T20:08:34.751 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.752 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.752 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.753 DEBUG:boto:Params: {} 2022-09-12T20:08:34.753 DEBUG:boto:Token: None 2022-09-12T20:08:34.754 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:34.754 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2K83yajW18cCwbdBwvuj9Pg2S4Q= 2022-09-12T20:08:34.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2K83yajW18cCwbdBwvuj9Pg2S4Q=', 'Content-Length': '0'} 2022-09-12T20:08:34.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4654382d6d507a1-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.758 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:34.758 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:34.759 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:34.759 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:34.760 DEBUG:boto:Method: GET 2022-09-12T20:08:34.760 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:34.761 DEBUG:boto:Data: 2022-09-12T20:08:34.761 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.762 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.763 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.763 DEBUG:boto:Params: {} 2022-09-12T20:08:34.764 DEBUG:boto:Token: None 2022-09-12T20:08:34.764 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:34.765 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2K83yajW18cCwbdBwvuj9Pg2S4Q= 2022-09-12T20:08:34.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2K83yajW18cCwbdBwvuj9Pg2S4Q=', 'Content-Length': '0'} 2022-09-12T20:08:34.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066416da70a07d867-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.769 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:34.769 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:34.770 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:08:34.770 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:08:34.771 DEBUG:boto:Method: GET 2022-09-12T20:08:34.771 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:08:34.772 DEBUG:boto:Data: 2022-09-12T20:08:34.772 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.773 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.773 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.774 DEBUG:boto:Params: {} 2022-09-12T20:08:34.775 DEBUG:boto:Token: None 2022-09-12T20:08:34.775 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/ 2022-09-12T20:08:34.776 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U= 2022-09-12T20:08:34.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U=', 'Content-Length': '0'} 2022-09-12T20:08:34.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024e01920d5a38e03-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.780 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:34.781 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:34.781 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:34.782 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:08:34.782 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:08:34.783 DEBUG:boto:Method: GET 2022-09-12T20:08:34.783 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:08:34.784 DEBUG:boto:Data: 2022-09-12T20:08:34.784 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.785 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.785 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.786 DEBUG:boto:Params: {} 2022-09-12T20:08:34.787 DEBUG:boto:Token: None 2022-09-12T20:08:34.787 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/ 2022-09-12T20:08:34.788 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U= 2022-09-12T20:08:34.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U=', 'Content-Length': '0'} 2022-09-12T20:08:34.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008126ed03c08b9d87-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.792 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:34.793 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:34.794 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:34.794 DEBUG:boto:Method: HEAD 2022-09-12T20:08:34.795 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:08:34.795 DEBUG:boto:Data: 2022-09-12T20:08:34.796 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.796 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.797 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.797 DEBUG:boto:Params: {} 2022-09-12T20:08:34.798 DEBUG:boto:Token: None 2022-09-12T20:08:34.799 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj95 2022-09-12T20:08:34.799 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5A0t5cpfxJkOTJQm8zlx71HTo/A= 2022-09-12T20:08:34.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5A0t5cpfxJkOTJQm8zlx71HTo/A=', 'Content-Length': '0'} 2022-09-12T20:08:34.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be25098cea9ab9fc-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.802 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:34.803 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:34.803 DEBUG:boto:Method: HEAD 2022-09-12T20:08:34.804 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:08:34.804 DEBUG:boto:Data: 2022-09-12T20:08:34.805 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.805 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.806 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.806 DEBUG:boto:Params: {} 2022-09-12T20:08:34.807 DEBUG:boto:Token: None 2022-09-12T20:08:34.808 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj95 2022-09-12T20:08:34.808 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5A0t5cpfxJkOTJQm8zlx71HTo/A= 2022-09-12T20:08:34.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5A0t5cpfxJkOTJQm8zlx71HTo/A=', 'Content-Length': '0'} 2022-09-12T20:08:34.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b16639c4171ec1cf-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:08:34.812 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:34.812 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:34.813 DEBUG:boto:Method: GET 2022-09-12T20:08:34.813 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:08:34.814 DEBUG:boto:Data: 2022-09-12T20:08:34.815 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.815 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.816 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.816 DEBUG:boto:Params: {} 2022-09-12T20:08:34.817 DEBUG:boto:Token: None 2022-09-12T20:08:34.817 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj95 2022-09-12T20:08:34.818 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DXtGAin0S8bPweLrtXqlsNmH51o= 2022-09-12T20:08:34.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DXtGAin0S8bPweLrtXqlsNmH51o=', 'Content-Length': '0'} 2022-09-12T20:08:34.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000337be590c84a4705-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.821 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:34.822 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:34.822 DEBUG:boto:Method: GET 2022-09-12T20:08:34.823 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:08:34.823 DEBUG:boto:Data: 2022-09-12T20:08:34.824 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.824 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.825 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.825 DEBUG:boto:Params: {} 2022-09-12T20:08:34.826 DEBUG:boto:Token: None 2022-09-12T20:08:34.827 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj95 2022-09-12T20:08:34.827 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DXtGAin0S8bPweLrtXqlsNmH51o= 2022-09-12T20:08:34.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DXtGAin0S8bPweLrtXqlsNmH51o=', 'Content-Length': '0'} 2022-09-12T20:08:34.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fde891d167dc31a3-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.870 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:34.871 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:34.871 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:08:34.872 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:08:34.872 DEBUG:boto:Method: GET 2022-09-12T20:08:34.873 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:08:34.874 DEBUG:boto:Data: 2022-09-12T20:08:34.874 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.875 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.875 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.876 DEBUG:boto:Params: {} 2022-09-12T20:08:34.876 DEBUG:boto:Token: None 2022-09-12T20:08:34.877 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/ 2022-09-12T20:08:34.878 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U= 2022-09-12T20:08:34.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U=', 'Content-Length': '0'} 2022-09-12T20:08:34.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000571dc727eb866984-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.881 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:34.882 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:34.883 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:34.883 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:08:34.884 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:08:34.884 DEBUG:boto:Method: GET 2022-09-12T20:08:34.885 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:08:34.885 DEBUG:boto:Data: 2022-09-12T20:08:34.886 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.886 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:34.887 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.888 DEBUG:boto:Params: {} 2022-09-12T20:08:34.888 DEBUG:boto:Token: None 2022-09-12T20:08:34.889 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/ 2022-09-12T20:08:34.889 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U= 2022-09-12T20:08:34.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B8xVzsO/Y+jdjrWSO/UCJvnjM+U=', 'Content-Length': '0'} 2022-09-12T20:08:34.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d67d4f2f9fc19eba-00631f91c2-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.894 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:34.895 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-75 zones={test-zone3, test-zone1} 2022-09-12T20:08:34.896 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-75 zones={test-zone3, test-zone2} 2022-09-12T20:08:34.896 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:34.897 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:34.897 DEBUG:boto:Method: HEAD 2022-09-12T20:08:34.898 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:08:34.898 DEBUG:boto:Data: 2022-09-12T20:08:34.899 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.899 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.900 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.900 DEBUG:boto:Params: {} 2022-09-12T20:08:34.901 DEBUG:boto:Token: None 2022-09-12T20:08:34.902 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/ 2022-09-12T20:08:34.903 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DyvRX1MW3nTmKgMLYUBs6ZqkcGo= 2022-09-12T20:08:34.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DyvRX1MW3nTmKgMLYUBs6ZqkcGo=', 'Content-Length': '0'} 2022-09-12T20:08:34.906 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3c3cd60a8b484f9-00631f91c2-1826-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.906 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:34.907 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:34.908 DEBUG:boto:Method: HEAD 2022-09-12T20:08:34.909 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:08:34.909 DEBUG:boto:Data: 2022-09-12T20:08:34.910 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.910 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:34.911 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.911 DEBUG:boto:Params: {} 2022-09-12T20:08:34.912 DEBUG:boto:Token: None 2022-09-12T20:08:34.913 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/ 2022-09-12T20:08:34.913 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DyvRX1MW3nTmKgMLYUBs6ZqkcGo= 2022-09-12T20:08:34.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DyvRX1MW3nTmKgMLYUBs6ZqkcGo=', 'Content-Length': '0'} 2022-09-12T20:08:34.917 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e913c70c7e96b053-00631f91c2-189d-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.918 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T20:08:34.919 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:34.919 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:34.920 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:34.920 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:34.921 DEBUG:boto:Method: GET 2022-09-12T20:08:34.921 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:34.922 DEBUG:boto:Data: 2022-09-12T20:08:34.922 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.923 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.923 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.924 DEBUG:boto:Params: {} 2022-09-12T20:08:34.924 DEBUG:boto:Token: None 2022-09-12T20:08:34.925 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/?versions 2022-09-12T20:08:34.926 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TJHejOoPLEaXibfXWYe3237ABSE= 2022-09-12T20:08:34.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TJHejOoPLEaXibfXWYe3237ABSE=', 'Content-Length': '0'} 2022-09-12T20:08:34.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023f3c59622938788-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.930 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:34.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:08:34.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:08:34.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:08:34.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:08:34.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:08:34.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:08:34.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:08:34.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:08:34.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:08:34.936 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:08:34.936 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:08:34.937 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:08:34.937 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:08:34.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:08:34.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:08:34.939 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T20:08:34.939 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:34.940 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:34.940 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:34.941 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:34.941 DEBUG:boto:Method: GET 2022-09-12T20:08:34.942 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:34.943 DEBUG:boto:Data: 2022-09-12T20:08:34.943 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.944 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:34.944 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.945 DEBUG:boto:Params: {} 2022-09-12T20:08:34.945 DEBUG:boto:Token: None 2022-09-12T20:08:34.946 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/?versions 2022-09-12T20:08:34.947 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TJHejOoPLEaXibfXWYe3237ABSE= 2022-09-12T20:08:34.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TJHejOoPLEaXibfXWYe3237ABSE=', 'Content-Length': '0'} 2022-09-12T20:08:34.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d590cc76d4a24b2-00631f91c2-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.951 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:34.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:08:34.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:08:34.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:08:34.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:08:34.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:08:34.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:08:34.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:08:34.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:08:34.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:08:34.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:08:34.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:08:34.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:08:34.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:08:34.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:08:34.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:08:34.960 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:34.960 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:34.961 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:34.961 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:34.962 DEBUG:boto:Method: GET 2022-09-12T20:08:34.963 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:34.963 DEBUG:boto:Data: 2022-09-12T20:08:34.964 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.964 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.965 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.965 DEBUG:boto:Params: {} 2022-09-12T20:08:34.966 DEBUG:boto:Token: None 2022-09-12T20:08:34.966 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/?versions 2022-09-12T20:08:34.967 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TJHejOoPLEaXibfXWYe3237ABSE= 2022-09-12T20:08:34.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TJHejOoPLEaXibfXWYe3237ABSE=', 'Content-Length': '0'} 2022-09-12T20:08:34.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0e519879e4d1e6d-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.971 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:34.972 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:34.973 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:34.973 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:34.974 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:34.974 DEBUG:boto:Method: GET 2022-09-12T20:08:34.975 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:08:34.975 DEBUG:boto:Data: 2022-09-12T20:08:34.976 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.976 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:34.977 DEBUG:boto:Port: 8000 2022-09-12T20:08:34.977 DEBUG:boto:Params: {} 2022-09-12T20:08:34.978 DEBUG:boto:Token: None 2022-09-12T20:08:34.978 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/?versions 2022-09-12T20:08:34.979 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TJHejOoPLEaXibfXWYe3237ABSE= 2022-09-12T20:08:34.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TJHejOoPLEaXibfXWYe3237ABSE=', 'Content-Length': '0'} 2022-09-12T20:08:34.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000479aafbb5aa4cd18-00631f91c2-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:34.983 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:34.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:08:34.985 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:34.985 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:34.986 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:34.986 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:34.987 DEBUG:boto:Method: GET 2022-09-12T20:08:34.987 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:34.988 DEBUG:boto:Data: 2022-09-12T20:08:34.989 DEBUG:boto:Headers: {} 2022-09-12T20:08:34.989 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:34.990 DEBUG:boto:Port: 8001 2022-09-12T20:08:34.990 DEBUG:boto:Params: {} 2022-09-12T20:08:34.991 DEBUG:boto:Token: None 2022-09-12T20:08:34.991 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:34 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:34.992 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xBA28tL9kKkFac9VLw+8qWtLzW4= 2022-09-12T20:08:34.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:34 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xBA28tL9kKkFac9VLw+8qWtLzW4=', 'Content-Length': '0'} 2022-09-12T20:08:34.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d74d85f2e7f77e93-00631f91c2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.041 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:35.042 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:35.042 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:35.043 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:35.043 DEBUG:boto:Method: GET 2022-09-12T20:08:35.044 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:35.044 DEBUG:boto:Data: 2022-09-12T20:08:35.045 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.045 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.046 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.047 DEBUG:boto:Params: {} 2022-09-12T20:08:35.047 DEBUG:boto:Token: None 2022-09-12T20:08:35.048 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:35.048 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fQGLT1rdlO6l13F7pxDq/oBJUwA= 2022-09-12T20:08:35.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fQGLT1rdlO6l13F7pxDq/oBJUwA=', 'Content-Length': '0'} 2022-09-12T20:08:35.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1b009c351e44e67-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.052 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:35.053 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:35.053 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:35.054 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:35.054 DEBUG:boto:Method: HEAD 2022-09-12T20:08:35.055 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:35.055 DEBUG:boto:Data: 2022-09-12T20:08:35.056 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.056 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.057 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.057 DEBUG:boto:Params: {} 2022-09-12T20:08:35.058 DEBUG:boto:Token: None 2022-09-12T20:08:35.058 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:35.059 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:o8vM3rfc3iczSTQCy6QzYOBP1lA= 2022-09-12T20:08:35.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:o8vM3rfc3iczSTQCy6QzYOBP1lA=', 'Content-Length': '0'} 2022-09-12T20:08:35.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011f350403266468f-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.062 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:35.063 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:35.063 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:35.064 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:35.064 DEBUG:boto:Method: HEAD 2022-09-12T20:08:35.065 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:35.065 DEBUG:boto:Data: 2022-09-12T20:08:35.066 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.066 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.067 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.067 DEBUG:boto:Params: {} 2022-09-12T20:08:35.068 DEBUG:boto:Token: None 2022-09-12T20:08:35.068 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:35.069 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:o8vM3rfc3iczSTQCy6QzYOBP1lA= 2022-09-12T20:08:35.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:o8vM3rfc3iczSTQCy6QzYOBP1lA=', 'Content-Length': '0'} 2022-09-12T20:08:35.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004181b5984f6fa38-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:08:35.073 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:35.073 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:35.074 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:35.074 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:35.075 DEBUG:boto:Method: GET 2022-09-12T20:08:35.075 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:35.076 DEBUG:boto:Data: 2022-09-12T20:08:35.077 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.077 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.078 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.078 DEBUG:boto:Params: {} 2022-09-12T20:08:35.079 DEBUG:boto:Token: None 2022-09-12T20:08:35.079 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:35.080 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fQGLT1rdlO6l13F7pxDq/oBJUwA= 2022-09-12T20:08:35.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fQGLT1rdlO6l13F7pxDq/oBJUwA=', 'Content-Length': '0'} 2022-09-12T20:08:35.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea067478f37b5829-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.083 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:35.084 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:35.084 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:35.085 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:08:35.085 DEBUG:boto:Method: GET 2022-09-12T20:08:35.086 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:35.086 DEBUG:boto:Data: 2022-09-12T20:08:35.087 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.087 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.088 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.089 DEBUG:boto:Params: {} 2022-09-12T20:08:35.089 DEBUG:boto:Token: None 2022-09-12T20:08:35.090 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:08:35.090 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fQGLT1rdlO6l13F7pxDq/oBJUwA= 2022-09-12T20:08:35.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fQGLT1rdlO6l13F7pxDq/oBJUwA=', 'Content-Length': '0'} 2022-09-12T20:08:35.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed50cf6873823344-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.094 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:35.095 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:35.095 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:08:35.096 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:08:35.096 DEBUG:boto:Method: GET 2022-09-12T20:08:35.097 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:08:35.097 DEBUG:boto:Data: 2022-09-12T20:08:35.098 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.098 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.099 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.099 DEBUG:boto:Params: {} 2022-09-12T20:08:35.100 DEBUG:boto:Token: None 2022-09-12T20:08:35.101 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/ 2022-09-12T20:08:35.101 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM= 2022-09-12T20:08:35.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM=', 'Content-Length': '0'} 2022-09-12T20:08:35.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2c1940abab930ba-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.105 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:35.106 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:35.106 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:35.107 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:08:35.108 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:08:35.108 DEBUG:boto:Method: GET 2022-09-12T20:08:35.109 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:08:35.109 DEBUG:boto:Data: 2022-09-12T20:08:35.110 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.110 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.111 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.111 DEBUG:boto:Params: {} 2022-09-12T20:08:35.112 DEBUG:boto:Token: None 2022-09-12T20:08:35.113 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/ 2022-09-12T20:08:35.113 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM= 2022-09-12T20:08:35.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM=', 'Content-Length': '0'} 2022-09-12T20:08:35.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e7f00d6ee24f623-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.117 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:35.118 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:35.119 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:35.119 DEBUG:boto:Method: HEAD 2022-09-12T20:08:35.120 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:08:35.120 DEBUG:boto:Data: 2022-09-12T20:08:35.121 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.121 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.122 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.122 DEBUG:boto:Params: {} 2022-09-12T20:08:35.123 DEBUG:boto:Token: None 2022-09-12T20:08:35.124 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj11 2022-09-12T20:08:35.124 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jNh75XWHG2I0hAOThwiHwLCS4w0= 2022-09-12T20:08:35.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jNh75XWHG2I0hAOThwiHwLCS4w0=', 'Content-Length': '0'} 2022-09-12T20:08:35.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5e98c785867b03d-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.127 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:35.128 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:35.128 DEBUG:boto:Method: HEAD 2022-09-12T20:08:35.129 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:08:35.129 DEBUG:boto:Data: 2022-09-12T20:08:35.130 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.130 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.131 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.131 DEBUG:boto:Params: {} 2022-09-12T20:08:35.132 DEBUG:boto:Token: None 2022-09-12T20:08:35.132 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj11 2022-09-12T20:08:35.133 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jNh75XWHG2I0hAOThwiHwLCS4w0= 2022-09-12T20:08:35.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jNh75XWHG2I0hAOThwiHwLCS4w0=', 'Content-Length': '0'} 2022-09-12T20:08:35.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000712c11e243e7593-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:08:35.137 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:35.137 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:35.138 DEBUG:boto:Method: GET 2022-09-12T20:08:35.139 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:08:35.139 DEBUG:boto:Data: 2022-09-12T20:08:35.140 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.140 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.141 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.141 DEBUG:boto:Params: {} 2022-09-12T20:08:35.142 DEBUG:boto:Token: None 2022-09-12T20:08:35.142 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj11 2022-09-12T20:08:35.143 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k+Q1C7JikIDZGKY4lpYQyvtZd20= 2022-09-12T20:08:35.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k+Q1C7JikIDZGKY4lpYQyvtZd20=', 'Content-Length': '0'} 2022-09-12T20:08:35.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f37ea64e31b9c871-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.186 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:08:35.187 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:08:35.187 DEBUG:boto:Method: GET 2022-09-12T20:08:35.188 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:08:35.188 DEBUG:boto:Data: 2022-09-12T20:08:35.189 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.189 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.190 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.190 DEBUG:boto:Params: {} 2022-09-12T20:08:35.191 DEBUG:boto:Token: None 2022-09-12T20:08:35.192 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj11 2022-09-12T20:08:35.192 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k+Q1C7JikIDZGKY4lpYQyvtZd20= 2022-09-12T20:08:35.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k+Q1C7JikIDZGKY4lpYQyvtZd20=', 'Content-Length': '0'} 2022-09-12T20:08:35.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000877434181e41ad6b-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.196 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:35.197 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:35.197 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:08:35.198 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:08:35.198 DEBUG:boto:Method: GET 2022-09-12T20:08:35.199 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:08:35.199 DEBUG:boto:Data: 2022-09-12T20:08:35.200 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.200 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.201 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.201 DEBUG:boto:Params: {} 2022-09-12T20:08:35.202 DEBUG:boto:Token: None 2022-09-12T20:08:35.203 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/ 2022-09-12T20:08:35.203 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM= 2022-09-12T20:08:35.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM=', 'Content-Length': '0'} 2022-09-12T20:08:35.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb34ada7f21aedf5-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.207 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:35.208 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:35.208 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:35.209 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:08:35.209 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:08:35.210 DEBUG:boto:Method: GET 2022-09-12T20:08:35.210 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:08:35.211 DEBUG:boto:Data: 2022-09-12T20:08:35.212 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.212 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.213 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.213 DEBUG:boto:Params: {} 2022-09-12T20:08:35.214 DEBUG:boto:Token: None 2022-09-12T20:08:35.214 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/ 2022-09-12T20:08:35.215 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM= 2022-09-12T20:08:35.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM=', 'Content-Length': '0'} 2022-09-12T20:08:35.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ef4b889ed2dc05b-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.219 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:35.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:08:35.220 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:35.221 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:35.221 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:35.222 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:35.222 DEBUG:boto:Method: GET 2022-09-12T20:08:35.223 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:35.223 DEBUG:boto:Data: 2022-09-12T20:08:35.224 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.224 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.225 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.225 DEBUG:boto:Params: {} 2022-09-12T20:08:35.226 DEBUG:boto:Token: None 2022-09-12T20:08:35.227 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:35.227 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mLsIEQydoBigcZ/b+5TPWA5WRR8= 2022-09-12T20:08:35.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mLsIEQydoBigcZ/b+5TPWA5WRR8=', 'Content-Length': '0'} 2022-09-12T20:08:35.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007024a64b2bb22df3-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.231 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:35.231 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:35.232 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:35.232 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:35.233 DEBUG:boto:Method: GET 2022-09-12T20:08:35.234 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:35.234 DEBUG:boto:Data: 2022-09-12T20:08:35.235 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.235 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.236 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.236 DEBUG:boto:Params: {} 2022-09-12T20:08:35.237 DEBUG:boto:Token: None 2022-09-12T20:08:35.237 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:35.238 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mLsIEQydoBigcZ/b+5TPWA5WRR8= 2022-09-12T20:08:35.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mLsIEQydoBigcZ/b+5TPWA5WRR8=', 'Content-Length': '0'} 2022-09-12T20:08:35.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002705d046c142f333-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.282 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:35.283 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:35.283 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:35.284 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:35.285 DEBUG:boto:Method: HEAD 2022-09-12T20:08:35.285 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:35.286 DEBUG:boto:Data: 2022-09-12T20:08:35.286 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.287 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.287 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.288 DEBUG:boto:Params: {} 2022-09-12T20:08:35.288 DEBUG:boto:Token: None 2022-09-12T20:08:35.289 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:35.290 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QCGFwzoe8QfwkKaKoVh2VrjtHUM= 2022-09-12T20:08:35.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QCGFwzoe8QfwkKaKoVh2VrjtHUM=', 'Content-Length': '0'} 2022-09-12T20:08:35.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000726ecfc5c85c1edb-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.293 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:35.293 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:35.294 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:35.294 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:35.295 DEBUG:boto:Method: HEAD 2022-09-12T20:08:35.295 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:35.296 DEBUG:boto:Data: 2022-09-12T20:08:35.296 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.297 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.297 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.298 DEBUG:boto:Params: {} 2022-09-12T20:08:35.299 DEBUG:boto:Token: None 2022-09-12T20:08:35.299 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:35.300 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QCGFwzoe8QfwkKaKoVh2VrjtHUM= 2022-09-12T20:08:35.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QCGFwzoe8QfwkKaKoVh2VrjtHUM=', 'Content-Length': '0'} 2022-09-12T20:08:35.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081a739340c9e7590-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:08:35.303 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:35.304 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:35.304 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:35.305 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:35.305 DEBUG:boto:Method: GET 2022-09-12T20:08:35.306 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:35.306 DEBUG:boto:Data: 2022-09-12T20:08:35.307 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.308 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.308 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.309 DEBUG:boto:Params: {} 2022-09-12T20:08:35.309 DEBUG:boto:Token: None 2022-09-12T20:08:35.310 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:35.310 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mLsIEQydoBigcZ/b+5TPWA5WRR8= 2022-09-12T20:08:35.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mLsIEQydoBigcZ/b+5TPWA5WRR8=', 'Content-Length': '0'} 2022-09-12T20:08:35.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a2d0dfe1f31b0c1-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.314 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:35.314 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:35.315 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:35.315 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:08:35.316 DEBUG:boto:Method: GET 2022-09-12T20:08:35.316 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:35.317 DEBUG:boto:Data: 2022-09-12T20:08:35.317 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.318 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.319 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.319 DEBUG:boto:Params: {} 2022-09-12T20:08:35.320 DEBUG:boto:Token: None 2022-09-12T20:08:35.321 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:08:35.321 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mLsIEQydoBigcZ/b+5TPWA5WRR8= 2022-09-12T20:08:35.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mLsIEQydoBigcZ/b+5TPWA5WRR8=', 'Content-Length': '0'} 2022-09-12T20:08:35.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b58a28642c9de1f4-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.325 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:35.326 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:35.326 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:08:35.327 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:08:35.328 DEBUG:boto:Method: GET 2022-09-12T20:08:35.328 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:08:35.329 DEBUG:boto:Data: 2022-09-12T20:08:35.329 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.330 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.330 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.331 DEBUG:boto:Params: {} 2022-09-12T20:08:35.332 DEBUG:boto:Token: None 2022-09-12T20:08:35.332 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/ 2022-09-12T20:08:35.333 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM= 2022-09-12T20:08:35.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM=', 'Content-Length': '0'} 2022-09-12T20:08:35.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000531786378e7c8e5e-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.337 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:35.337 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:35.338 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:35.339 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:08:35.339 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:08:35.340 DEBUG:boto:Method: GET 2022-09-12T20:08:35.340 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:08:35.341 DEBUG:boto:Data: 2022-09-12T20:08:35.341 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.342 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.342 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.343 DEBUG:boto:Params: {} 2022-09-12T20:08:35.343 DEBUG:boto:Token: None 2022-09-12T20:08:35.344 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/ 2022-09-12T20:08:35.345 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM= 2022-09-12T20:08:35.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM=', 'Content-Length': '0'} 2022-09-12T20:08:35.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094ec6623849c150f-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.349 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:35.349 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:35.350 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:35.350 DEBUG:boto:Method: HEAD 2022-09-12T20:08:35.351 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:08:35.351 DEBUG:boto:Data: 2022-09-12T20:08:35.352 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.352 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.353 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.353 DEBUG:boto:Params: {} 2022-09-12T20:08:35.354 DEBUG:boto:Token: None 2022-09-12T20:08:35.354 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj17 2022-09-12T20:08:35.355 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9PgCU7YSkGz8jsIxVhnOuHuUv6s= 2022-09-12T20:08:35.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9PgCU7YSkGz8jsIxVhnOuHuUv6s=', 'Content-Length': '0'} 2022-09-12T20:08:35.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0113aaf160534f6-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.358 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:35.359 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:35.359 DEBUG:boto:Method: HEAD 2022-09-12T20:08:35.360 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:08:35.360 DEBUG:boto:Data: 2022-09-12T20:08:35.361 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.361 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.362 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.362 DEBUG:boto:Params: {} 2022-09-12T20:08:35.363 DEBUG:boto:Token: None 2022-09-12T20:08:35.363 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj17 2022-09-12T20:08:35.364 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9PgCU7YSkGz8jsIxVhnOuHuUv6s= 2022-09-12T20:08:35.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9PgCU7YSkGz8jsIxVhnOuHuUv6s=', 'Content-Length': '0'} 2022-09-12T20:08:35.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8958cf962613725-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:08:35.368 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:35.368 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:35.369 DEBUG:boto:Method: GET 2022-09-12T20:08:35.369 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:08:35.370 DEBUG:boto:Data: 2022-09-12T20:08:35.370 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.371 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.371 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.372 DEBUG:boto:Params: {} 2022-09-12T20:08:35.372 DEBUG:boto:Token: None 2022-09-12T20:08:35.373 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj17 2022-09-12T20:08:35.374 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1C36K4wF7p/S2OncOQpeVFBLLJI= 2022-09-12T20:08:35.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1C36K4wF7p/S2OncOQpeVFBLLJI=', 'Content-Length': '0'} 2022-09-12T20:08:35.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000250cdabfc9a7e327-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.377 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:08:35.377 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:08:35.378 DEBUG:boto:Method: GET 2022-09-12T20:08:35.379 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:08:35.379 DEBUG:boto:Data: 2022-09-12T20:08:35.380 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.380 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.381 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.381 DEBUG:boto:Params: {} 2022-09-12T20:08:35.382 DEBUG:boto:Token: None 2022-09-12T20:08:35.382 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj17 2022-09-12T20:08:35.383 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1C36K4wF7p/S2OncOQpeVFBLLJI= 2022-09-12T20:08:35.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1C36K4wF7p/S2OncOQpeVFBLLJI=', 'Content-Length': '0'} 2022-09-12T20:08:35.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073a9ae84c35c4054-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.426 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:35.427 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:35.427 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:08:35.428 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:08:35.428 DEBUG:boto:Method: GET 2022-09-12T20:08:35.429 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:08:35.429 DEBUG:boto:Data: 2022-09-12T20:08:35.430 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.431 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.431 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.432 DEBUG:boto:Params: {} 2022-09-12T20:08:35.432 DEBUG:boto:Token: None 2022-09-12T20:08:35.433 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/ 2022-09-12T20:08:35.434 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM= 2022-09-12T20:08:35.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM=', 'Content-Length': '0'} 2022-09-12T20:08:35.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008076574fb0e53aca-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.438 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:35.438 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:35.439 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:35.440 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:08:35.440 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:08:35.441 DEBUG:boto:Method: GET 2022-09-12T20:08:35.441 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:08:35.442 DEBUG:boto:Data: 2022-09-12T20:08:35.442 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.443 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.443 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.444 DEBUG:boto:Params: {} 2022-09-12T20:08:35.444 DEBUG:boto:Token: None 2022-09-12T20:08:35.445 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/ 2022-09-12T20:08:35.446 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM= 2022-09-12T20:08:35.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM=', 'Content-Length': '0'} 2022-09-12T20:08:35.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004870a4e894f3f59b-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.449 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:35.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:08:35.451 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:35.451 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:35.452 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:35.452 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:35.453 DEBUG:boto:Method: GET 2022-09-12T20:08:35.453 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:35.454 DEBUG:boto:Data: 2022-09-12T20:08:35.455 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.455 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.456 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.456 DEBUG:boto:Params: {} 2022-09-12T20:08:35.457 DEBUG:boto:Token: None 2022-09-12T20:08:35.457 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:35.458 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IRb+x+6+fR4Krol836DU1v81ZwI= 2022-09-12T20:08:35.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IRb+x+6+fR4Krol836DU1v81ZwI=', 'Content-Length': '0'} 2022-09-12T20:08:35.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ae066ab7968c379-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.461 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:35.462 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:35.463 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:35.463 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:35.464 DEBUG:boto:Method: GET 2022-09-12T20:08:35.464 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:35.465 DEBUG:boto:Data: 2022-09-12T20:08:35.465 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.466 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.466 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.467 DEBUG:boto:Params: {} 2022-09-12T20:08:35.467 DEBUG:boto:Token: None 2022-09-12T20:08:35.468 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:35.469 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IRb+x+6+fR4Krol836DU1v81ZwI= 2022-09-12T20:08:35.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IRb+x+6+fR4Krol836DU1v81ZwI=', 'Content-Length': '0'} 2022-09-12T20:08:35.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a74171d5d62de0f6-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.472 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:35.473 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:35.473 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:35.474 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:35.474 DEBUG:boto:Method: HEAD 2022-09-12T20:08:35.475 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:35.475 DEBUG:boto:Data: 2022-09-12T20:08:35.476 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.476 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.477 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.477 DEBUG:boto:Params: {} 2022-09-12T20:08:35.478 DEBUG:boto:Token: None 2022-09-12T20:08:35.478 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:35.479 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XQx2A/9XooJV3VAebXvBMUl8qOA= 2022-09-12T20:08:35.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XQx2A/9XooJV3VAebXvBMUl8qOA=', 'Content-Length': '0'} 2022-09-12T20:08:35.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a38a7577a024cffb-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.482 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:35.483 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:35.483 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:35.484 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:35.484 DEBUG:boto:Method: HEAD 2022-09-12T20:08:35.485 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:35.485 DEBUG:boto:Data: 2022-09-12T20:08:35.486 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.486 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.487 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.487 DEBUG:boto:Params: {} 2022-09-12T20:08:35.488 DEBUG:boto:Token: None 2022-09-12T20:08:35.488 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:35.489 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XQx2A/9XooJV3VAebXvBMUl8qOA= 2022-09-12T20:08:35.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XQx2A/9XooJV3VAebXvBMUl8qOA=', 'Content-Length': '0'} 2022-09-12T20:08:35.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9e9fd6e3a563d74-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:08:35.493 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:35.493 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:35.494 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:35.494 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:35.495 DEBUG:boto:Method: GET 2022-09-12T20:08:35.496 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:35.496 DEBUG:boto:Data: 2022-09-12T20:08:35.497 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.497 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.498 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.498 DEBUG:boto:Params: {} 2022-09-12T20:08:35.499 DEBUG:boto:Token: None 2022-09-12T20:08:35.499 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:35.500 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IRb+x+6+fR4Krol836DU1v81ZwI= 2022-09-12T20:08:35.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IRb+x+6+fR4Krol836DU1v81ZwI=', 'Content-Length': '0'} 2022-09-12T20:08:35.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000788394e2ab3af67a-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.503 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:35.504 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:35.504 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:35.505 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:08:35.505 DEBUG:boto:Method: GET 2022-09-12T20:08:35.506 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:35.506 DEBUG:boto:Data: 2022-09-12T20:08:35.507 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.507 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.508 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.509 DEBUG:boto:Params: {} 2022-09-12T20:08:35.509 DEBUG:boto:Token: None 2022-09-12T20:08:35.510 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:08:35.510 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IRb+x+6+fR4Krol836DU1v81ZwI= 2022-09-12T20:08:35.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IRb+x+6+fR4Krol836DU1v81ZwI=', 'Content-Length': '0'} 2022-09-12T20:08:35.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d43d99c4df81ac23-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.514 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:35.514 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:35.515 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:08:35.516 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:08:35.516 DEBUG:boto:Method: GET 2022-09-12T20:08:35.517 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:08:35.517 DEBUG:boto:Data: 2022-09-12T20:08:35.518 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.518 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.519 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.519 DEBUG:boto:Params: {} 2022-09-12T20:08:35.520 DEBUG:boto:Token: None 2022-09-12T20:08:35.521 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/ 2022-09-12T20:08:35.521 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM= 2022-09-12T20:08:35.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM=', 'Content-Length': '0'} 2022-09-12T20:08:35.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c818052621aa46b-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.566 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:35.567 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:35.568 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:35.569 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:08:35.569 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:08:35.570 DEBUG:boto:Method: GET 2022-09-12T20:08:35.570 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:08:35.571 DEBUG:boto:Data: 2022-09-12T20:08:35.572 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.572 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.573 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.573 DEBUG:boto:Params: {} 2022-09-12T20:08:35.574 DEBUG:boto:Token: None 2022-09-12T20:08:35.575 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/ 2022-09-12T20:08:35.575 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM= 2022-09-12T20:08:35.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM=', 'Content-Length': '0'} 2022-09-12T20:08:35.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3d6c237417a04bc-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.579 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:35.580 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:35.580 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:35.581 DEBUG:boto:Method: HEAD 2022-09-12T20:08:35.582 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:08:35.582 DEBUG:boto:Data: 2022-09-12T20:08:35.583 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.583 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.584 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.584 DEBUG:boto:Params: {} 2022-09-12T20:08:35.585 DEBUG:boto:Token: None 2022-09-12T20:08:35.585 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj19 2022-09-12T20:08:35.586 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Xkbb0/oiSw1FNnB5OtL6besjG88= 2022-09-12T20:08:35.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Xkbb0/oiSw1FNnB5OtL6besjG88=', 'Content-Length': '0'} 2022-09-12T20:08:35.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d1754e0592ed07a-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.590 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:35.591 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:35.591 DEBUG:boto:Method: HEAD 2022-09-12T20:08:35.592 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:08:35.592 DEBUG:boto:Data: 2022-09-12T20:08:35.593 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.593 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.594 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.594 DEBUG:boto:Params: {} 2022-09-12T20:08:35.595 DEBUG:boto:Token: None 2022-09-12T20:08:35.596 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj19 2022-09-12T20:08:35.596 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Xkbb0/oiSw1FNnB5OtL6besjG88= 2022-09-12T20:08:35.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Xkbb0/oiSw1FNnB5OtL6besjG88=', 'Content-Length': '0'} 2022-09-12T20:08:35.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007149eca7137999b8-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:08:35.601 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:35.601 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:35.602 DEBUG:boto:Method: GET 2022-09-12T20:08:35.602 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:08:35.603 DEBUG:boto:Data: 2022-09-12T20:08:35.603 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.604 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.605 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.605 DEBUG:boto:Params: {} 2022-09-12T20:08:35.606 DEBUG:boto:Token: None 2022-09-12T20:08:35.606 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj19 2022-09-12T20:08:35.607 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ldIVaGds7XNWhccCqPsJ+x55YL0= 2022-09-12T20:08:35.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ldIVaGds7XNWhccCqPsJ+x55YL0=', 'Content-Length': '0'} 2022-09-12T20:08:35.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bef8b0d976653c72-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.611 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:08:35.611 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:08:35.612 DEBUG:boto:Method: GET 2022-09-12T20:08:35.612 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:08:35.613 DEBUG:boto:Data: 2022-09-12T20:08:35.614 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.614 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.615 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.615 DEBUG:boto:Params: {} 2022-09-12T20:08:35.616 DEBUG:boto:Token: None 2022-09-12T20:08:35.616 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj19 2022-09-12T20:08:35.617 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ldIVaGds7XNWhccCqPsJ+x55YL0= 2022-09-12T20:08:35.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ldIVaGds7XNWhccCqPsJ+x55YL0=', 'Content-Length': '0'} 2022-09-12T20:08:35.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f4d2a3228e782dc-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.621 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:35.621 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:35.622 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:08:35.622 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:08:35.623 DEBUG:boto:Method: GET 2022-09-12T20:08:35.623 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:08:35.624 DEBUG:boto:Data: 2022-09-12T20:08:35.625 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.625 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.626 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.626 DEBUG:boto:Params: {} 2022-09-12T20:08:35.627 DEBUG:boto:Token: None 2022-09-12T20:08:35.628 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/ 2022-09-12T20:08:35.628 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM= 2022-09-12T20:08:35.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM=', 'Content-Length': '0'} 2022-09-12T20:08:35.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bbb7636b1af1177-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.632 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:35.633 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:35.633 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:35.634 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:08:35.635 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:08:35.635 DEBUG:boto:Method: GET 2022-09-12T20:08:35.636 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:08:35.636 DEBUG:boto:Data: 2022-09-12T20:08:35.637 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.638 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.638 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.639 DEBUG:boto:Params: {} 2022-09-12T20:08:35.639 DEBUG:boto:Token: None 2022-09-12T20:08:35.640 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/ 2022-09-12T20:08:35.641 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM= 2022-09-12T20:08:35.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM=', 'Content-Length': '0'} 2022-09-12T20:08:35.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea9fce8598e3c5e3-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.645 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:35.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:08:35.646 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:35.647 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:35.648 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:35.648 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:35.649 DEBUG:boto:Method: GET 2022-09-12T20:08:35.649 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:35.650 DEBUG:boto:Data: 2022-09-12T20:08:35.650 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.651 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.652 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.652 DEBUG:boto:Params: {} 2022-09-12T20:08:35.653 DEBUG:boto:Token: None 2022-09-12T20:08:35.653 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:35.654 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X7JCHyKiTOJBnTtvJa72Bt4yI8Q= 2022-09-12T20:08:35.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X7JCHyKiTOJBnTtvJa72Bt4yI8Q=', 'Content-Length': '0'} 2022-09-12T20:08:35.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075777cc2196316cf-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.698 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:35.699 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:35.699 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:35.700 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:35.701 DEBUG:boto:Method: GET 2022-09-12T20:08:35.701 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:35.702 DEBUG:boto:Data: 2022-09-12T20:08:35.702 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.703 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.703 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.704 DEBUG:boto:Params: {} 2022-09-12T20:08:35.705 DEBUG:boto:Token: None 2022-09-12T20:08:35.705 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:35.706 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X7JCHyKiTOJBnTtvJa72Bt4yI8Q= 2022-09-12T20:08:35.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X7JCHyKiTOJBnTtvJa72Bt4yI8Q=', 'Content-Length': '0'} 2022-09-12T20:08:35.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b9e351cd0eb4a7d-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.710 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:35.710 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:35.711 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:35.711 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:35.712 DEBUG:boto:Method: HEAD 2022-09-12T20:08:35.712 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:35.713 DEBUG:boto:Data: 2022-09-12T20:08:35.713 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.714 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.714 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.715 DEBUG:boto:Params: {} 2022-09-12T20:08:35.716 DEBUG:boto:Token: None 2022-09-12T20:08:35.716 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:35.717 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ncyX4XANbYyVmpOTOneYdoReEQs= 2022-09-12T20:08:35.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ncyX4XANbYyVmpOTOneYdoReEQs=', 'Content-Length': '0'} 2022-09-12T20:08:35.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbdae53df0985f83-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.721 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:35.722 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:35.722 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:35.723 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:35.723 DEBUG:boto:Method: HEAD 2022-09-12T20:08:35.724 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:35.724 DEBUG:boto:Data: 2022-09-12T20:08:35.725 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.725 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.726 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.727 DEBUG:boto:Params: {} 2022-09-12T20:08:35.727 DEBUG:boto:Token: None 2022-09-12T20:08:35.728 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:35.728 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ncyX4XANbYyVmpOTOneYdoReEQs= 2022-09-12T20:08:35.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ncyX4XANbYyVmpOTOneYdoReEQs=', 'Content-Length': '0'} 2022-09-12T20:08:35.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af4037fd8539a8ba-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:08:35.732 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:35.733 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:35.733 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:35.734 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:35.735 DEBUG:boto:Method: GET 2022-09-12T20:08:35.735 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:35.736 DEBUG:boto:Data: 2022-09-12T20:08:35.736 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.737 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.737 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.738 DEBUG:boto:Params: {} 2022-09-12T20:08:35.738 DEBUG:boto:Token: None 2022-09-12T20:08:35.739 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:35.740 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X7JCHyKiTOJBnTtvJa72Bt4yI8Q= 2022-09-12T20:08:35.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X7JCHyKiTOJBnTtvJa72Bt4yI8Q=', 'Content-Length': '0'} 2022-09-12T20:08:35.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097ccbb0e24e8e4f1-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.743 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:35.744 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:35.745 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:35.745 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:08:35.746 DEBUG:boto:Method: GET 2022-09-12T20:08:35.746 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:35.747 DEBUG:boto:Data: 2022-09-12T20:08:35.747 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.748 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.748 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.749 DEBUG:boto:Params: {} 2022-09-12T20:08:35.750 DEBUG:boto:Token: None 2022-09-12T20:08:35.750 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:08:35.751 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X7JCHyKiTOJBnTtvJa72Bt4yI8Q= 2022-09-12T20:08:35.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X7JCHyKiTOJBnTtvJa72Bt4yI8Q=', 'Content-Length': '0'} 2022-09-12T20:08:35.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b3c6b6a2a70a500-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.754 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:35.755 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:35.756 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:08:35.756 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:08:35.757 DEBUG:boto:Method: GET 2022-09-12T20:08:35.757 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:08:35.758 DEBUG:boto:Data: 2022-09-12T20:08:35.758 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.759 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.760 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.760 DEBUG:boto:Params: {} 2022-09-12T20:08:35.761 DEBUG:boto:Token: None 2022-09-12T20:08:35.761 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/ 2022-09-12T20:08:35.762 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM= 2022-09-12T20:08:35.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM=', 'Content-Length': '0'} 2022-09-12T20:08:35.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bbad9e8018502be-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.766 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:35.767 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:35.768 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:35.768 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:08:35.769 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:08:35.769 DEBUG:boto:Method: GET 2022-09-12T20:08:35.770 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:08:35.770 DEBUG:boto:Data: 2022-09-12T20:08:35.771 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.771 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.772 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.773 DEBUG:boto:Params: {} 2022-09-12T20:08:35.773 DEBUG:boto:Token: None 2022-09-12T20:08:35.774 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/ 2022-09-12T20:08:35.774 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM= 2022-09-12T20:08:35.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM=', 'Content-Length': '0'} 2022-09-12T20:08:35.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000193da16b556b071d-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.779 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:35.779 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:35.780 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:35.780 DEBUG:boto:Method: HEAD 2022-09-12T20:08:35.781 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:08:35.781 DEBUG:boto:Data: 2022-09-12T20:08:35.782 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.782 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.783 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.783 DEBUG:boto:Params: {} 2022-09-12T20:08:35.784 DEBUG:boto:Token: None 2022-09-12T20:08:35.785 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj22 2022-09-12T20:08:35.785 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jLkDVciQ/HXa4eAcptAzhAw1fbc= 2022-09-12T20:08:35.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jLkDVciQ/HXa4eAcptAzhAw1fbc=', 'Content-Length': '0'} 2022-09-12T20:08:35.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042c108b693433352-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.789 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:35.789 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:35.790 DEBUG:boto:Method: HEAD 2022-09-12T20:08:35.790 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:08:35.791 DEBUG:boto:Data: 2022-09-12T20:08:35.791 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.792 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.792 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.793 DEBUG:boto:Params: {} 2022-09-12T20:08:35.794 DEBUG:boto:Token: None 2022-09-12T20:08:35.795 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj22 2022-09-12T20:08:35.795 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jLkDVciQ/HXa4eAcptAzhAw1fbc= 2022-09-12T20:08:35.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jLkDVciQ/HXa4eAcptAzhAw1fbc=', 'Content-Length': '0'} 2022-09-12T20:08:35.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9a1d8acdaabe432-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:08:35.799 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:35.800 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:35.800 DEBUG:boto:Method: GET 2022-09-12T20:08:35.801 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:08:35.802 DEBUG:boto:Data: 2022-09-12T20:08:35.802 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.803 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.803 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.804 DEBUG:boto:Params: {} 2022-09-12T20:08:35.804 DEBUG:boto:Token: None 2022-09-12T20:08:35.805 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj22 2022-09-12T20:08:35.806 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TWQzCz2q12FHPj5b154+OediKiY= 2022-09-12T20:08:35.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TWQzCz2q12FHPj5b154+OediKiY=', 'Content-Length': '0'} 2022-09-12T20:08:35.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd54a2ba09bf0728-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.809 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:08:35.810 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:08:35.811 DEBUG:boto:Method: GET 2022-09-12T20:08:35.811 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:08:35.812 DEBUG:boto:Data: 2022-09-12T20:08:35.812 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.813 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.813 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.814 DEBUG:boto:Params: {} 2022-09-12T20:08:35.814 DEBUG:boto:Token: None 2022-09-12T20:08:35.815 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj22 2022-09-12T20:08:35.816 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TWQzCz2q12FHPj5b154+OediKiY= 2022-09-12T20:08:35.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TWQzCz2q12FHPj5b154+OediKiY=', 'Content-Length': '0'} 2022-09-12T20:08:35.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007acd62da98f66a83-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.862 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:35.863 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:35.863 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:08:35.864 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:08:35.865 DEBUG:boto:Method: GET 2022-09-12T20:08:35.865 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:08:35.866 DEBUG:boto:Data: 2022-09-12T20:08:35.866 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.867 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.867 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.868 DEBUG:boto:Params: {} 2022-09-12T20:08:35.869 DEBUG:boto:Token: None 2022-09-12T20:08:35.869 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/ 2022-09-12T20:08:35.870 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM= 2022-09-12T20:08:35.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM=', 'Content-Length': '0'} 2022-09-12T20:08:35.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c41928d18f4df2a-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.874 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:35.875 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:35.876 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:35.876 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:08:35.877 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:08:35.877 DEBUG:boto:Method: GET 2022-09-12T20:08:35.878 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:08:35.879 DEBUG:boto:Data: 2022-09-12T20:08:35.879 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.880 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.881 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.881 DEBUG:boto:Params: {} 2022-09-12T20:08:35.882 DEBUG:boto:Token: None 2022-09-12T20:08:35.883 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/ 2022-09-12T20:08:35.883 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM= 2022-09-12T20:08:35.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM=', 'Content-Length': '0'} 2022-09-12T20:08:35.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a960e5a88b8a5722-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.887 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:35.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:08:35.889 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:35.890 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:35.890 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:35.891 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:35.891 DEBUG:boto:Method: GET 2022-09-12T20:08:35.892 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:35.892 DEBUG:boto:Data: 2022-09-12T20:08:35.893 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.893 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.894 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.895 DEBUG:boto:Params: {} 2022-09-12T20:08:35.895 DEBUG:boto:Token: None 2022-09-12T20:08:35.896 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:35.897 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YIiHQaDdwRUiHixp5DwjwEh4ze0= 2022-09-12T20:08:35.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YIiHQaDdwRUiHixp5DwjwEh4ze0=', 'Content-Length': '0'} 2022-09-12T20:08:35.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030196bb16975aa71-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.900 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:35.901 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:35.902 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:35.902 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:35.903 DEBUG:boto:Method: GET 2022-09-12T20:08:35.903 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:35.904 DEBUG:boto:Data: 2022-09-12T20:08:35.904 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.905 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.906 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.907 DEBUG:boto:Params: {} 2022-09-12T20:08:35.907 DEBUG:boto:Token: None 2022-09-12T20:08:35.908 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:35.908 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YIiHQaDdwRUiHixp5DwjwEh4ze0= 2022-09-12T20:08:35.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YIiHQaDdwRUiHixp5DwjwEh4ze0=', 'Content-Length': '0'} 2022-09-12T20:08:35.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000301ca469906a37a4-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.912 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:35.913 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:35.913 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:35.914 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:35.914 DEBUG:boto:Method: HEAD 2022-09-12T20:08:35.915 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:35.915 DEBUG:boto:Data: 2022-09-12T20:08:35.916 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.916 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.917 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.917 DEBUG:boto:Params: {} 2022-09-12T20:08:35.918 DEBUG:boto:Token: None 2022-09-12T20:08:35.918 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:35.919 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ajtgNNGXuNyEwC9aN66cfvHqiCA= 2022-09-12T20:08:35.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ajtgNNGXuNyEwC9aN66cfvHqiCA=', 'Content-Length': '0'} 2022-09-12T20:08:35.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a3a811604f7eee9-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.923 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:35.923 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:35.924 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:35.924 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:35.925 DEBUG:boto:Method: HEAD 2022-09-12T20:08:35.925 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:35.926 DEBUG:boto:Data: 2022-09-12T20:08:35.926 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.927 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.927 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.928 DEBUG:boto:Params: {} 2022-09-12T20:08:35.928 DEBUG:boto:Token: None 2022-09-12T20:08:35.929 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:35.930 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ajtgNNGXuNyEwC9aN66cfvHqiCA= 2022-09-12T20:08:35.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ajtgNNGXuNyEwC9aN66cfvHqiCA=', 'Content-Length': '0'} 2022-09-12T20:08:35.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060a7379a01cb8be6-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:08:35.933 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:35.934 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:35.935 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:35.935 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:35.936 DEBUG:boto:Method: GET 2022-09-12T20:08:35.936 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:35.937 DEBUG:boto:Data: 2022-09-12T20:08:35.937 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.938 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.939 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.939 DEBUG:boto:Params: {} 2022-09-12T20:08:35.940 DEBUG:boto:Token: None 2022-09-12T20:08:35.940 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:35.941 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YIiHQaDdwRUiHixp5DwjwEh4ze0= 2022-09-12T20:08:35.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YIiHQaDdwRUiHixp5DwjwEh4ze0=', 'Content-Length': '0'} 2022-09-12T20:08:35.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceba201b003e7d57-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.945 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:35.945 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:35.946 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:35.946 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:08:35.947 DEBUG:boto:Method: GET 2022-09-12T20:08:35.947 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:35.948 DEBUG:boto:Data: 2022-09-12T20:08:35.949 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.949 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.950 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.950 DEBUG:boto:Params: {} 2022-09-12T20:08:35.951 DEBUG:boto:Token: None 2022-09-12T20:08:35.951 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:08:35.952 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YIiHQaDdwRUiHixp5DwjwEh4ze0= 2022-09-12T20:08:35.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YIiHQaDdwRUiHixp5DwjwEh4ze0=', 'Content-Length': '0'} 2022-09-12T20:08:35.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b441433f0049734b-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.955 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:35.956 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:35.957 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:08:35.957 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:08:35.958 DEBUG:boto:Method: GET 2022-09-12T20:08:35.958 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:08:35.959 DEBUG:boto:Data: 2022-09-12T20:08:35.960 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.960 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:35.961 DEBUG:boto:Port: 8001 2022-09-12T20:08:35.961 DEBUG:boto:Params: {} 2022-09-12T20:08:35.962 DEBUG:boto:Token: None 2022-09-12T20:08:35.962 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/ 2022-09-12T20:08:35.963 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM= 2022-09-12T20:08:35.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM=', 'Content-Length': '0'} 2022-09-12T20:08:35.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9fbf67d91fa3453-00631f91c3-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:35.967 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:35.968 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:35.969 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:35.969 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:08:35.970 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:08:35.970 DEBUG:boto:Method: GET 2022-09-12T20:08:35.971 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:08:35.971 DEBUG:boto:Data: 2022-09-12T20:08:35.972 DEBUG:boto:Headers: {} 2022-09-12T20:08:35.972 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:35.973 DEBUG:boto:Port: 8000 2022-09-12T20:08:35.973 DEBUG:boto:Params: {} 2022-09-12T20:08:35.974 DEBUG:boto:Token: None 2022-09-12T20:08:35.974 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:35 GMT /jdpiak-75/ 2022-09-12T20:08:35.975 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM= 2022-09-12T20:08:35.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:35 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fZdqOfsfGAqUv/4W+l4z9u40DmM=', 'Content-Length': '0'} 2022-09-12T20:08:35.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae82c11d5aa0ab38-00631f91c3-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.022 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:36.023 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:36.023 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:36.024 DEBUG:boto:Method: HEAD 2022-09-12T20:08:36.024 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:08:36.025 DEBUG:boto:Data: 2022-09-12T20:08:36.025 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.026 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.026 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.027 DEBUG:boto:Params: {} 2022-09-12T20:08:36.028 DEBUG:boto:Token: None 2022-09-12T20:08:36.028 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj33 2022-09-12T20:08:36.029 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nEH8HiuyQDrglHLhamRbtbs0zL0= 2022-09-12T20:08:36.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nEH8HiuyQDrglHLhamRbtbs0zL0=', 'Content-Length': '0'} 2022-09-12T20:08:36.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd7c86e09f0741a8-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.032 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:36.033 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:36.033 DEBUG:boto:Method: HEAD 2022-09-12T20:08:36.034 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:08:36.034 DEBUG:boto:Data: 2022-09-12T20:08:36.035 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.035 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.036 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.036 DEBUG:boto:Params: {} 2022-09-12T20:08:36.037 DEBUG:boto:Token: None 2022-09-12T20:08:36.038 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj33 2022-09-12T20:08:36.038 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nEH8HiuyQDrglHLhamRbtbs0zL0= 2022-09-12T20:08:36.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nEH8HiuyQDrglHLhamRbtbs0zL0=', 'Content-Length': '0'} 2022-09-12T20:08:36.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ae5f527d2ed6756-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:08:36.042 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:36.043 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:36.043 DEBUG:boto:Method: GET 2022-09-12T20:08:36.044 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:08:36.044 DEBUG:boto:Data: 2022-09-12T20:08:36.045 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.045 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.046 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.046 DEBUG:boto:Params: {} 2022-09-12T20:08:36.047 DEBUG:boto:Token: None 2022-09-12T20:08:36.048 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj33 2022-09-12T20:08:36.048 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7QB7P4a5DPU5ryEanxK5AGgU/yY= 2022-09-12T20:08:36.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7QB7P4a5DPU5ryEanxK5AGgU/yY=', 'Content-Length': '0'} 2022-09-12T20:08:36.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc59fc129e03b6f2-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.052 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:08:36.052 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:08:36.053 DEBUG:boto:Method: GET 2022-09-12T20:08:36.054 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:08:36.054 DEBUG:boto:Data: 2022-09-12T20:08:36.055 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.055 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.056 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.056 DEBUG:boto:Params: {} 2022-09-12T20:08:36.057 DEBUG:boto:Token: None 2022-09-12T20:08:36.057 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj33 2022-09-12T20:08:36.058 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7QB7P4a5DPU5ryEanxK5AGgU/yY= 2022-09-12T20:08:36.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7QB7P4a5DPU5ryEanxK5AGgU/yY=', 'Content-Length': '0'} 2022-09-12T20:08:36.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a270db69b4e91cce-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.062 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:36.062 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:36.063 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:08:36.063 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:08:36.064 DEBUG:boto:Method: GET 2022-09-12T20:08:36.064 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:08:36.065 DEBUG:boto:Data: 2022-09-12T20:08:36.066 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.066 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.067 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.067 DEBUG:boto:Params: {} 2022-09-12T20:08:36.068 DEBUG:boto:Token: None 2022-09-12T20:08:36.069 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/ 2022-09-12T20:08:36.069 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4= 2022-09-12T20:08:36.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4=', 'Content-Length': '0'} 2022-09-12T20:08:36.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7fe7ec2ca71ebd2-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.073 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:36.074 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:36.075 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:36.075 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:08:36.076 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:08:36.077 DEBUG:boto:Method: GET 2022-09-12T20:08:36.077 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:08:36.078 DEBUG:boto:Data: 2022-09-12T20:08:36.078 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.079 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.079 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.080 DEBUG:boto:Params: {} 2022-09-12T20:08:36.080 DEBUG:boto:Token: None 2022-09-12T20:08:36.081 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/ 2022-09-12T20:08:36.082 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4= 2022-09-12T20:08:36.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4=', 'Content-Length': '0'} 2022-09-12T20:08:36.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004892f4732d92d347-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.086 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:36.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:08:36.087 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:36.088 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:36.088 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:36.089 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:36.089 DEBUG:boto:Method: GET 2022-09-12T20:08:36.090 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:36.090 DEBUG:boto:Data: 2022-09-12T20:08:36.091 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.091 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.092 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.093 DEBUG:boto:Params: {} 2022-09-12T20:08:36.093 DEBUG:boto:Token: None 2022-09-12T20:08:36.094 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:36.094 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Kj045u95LVCQTf+DTLX9Owrqw3E= 2022-09-12T20:08:36.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Kj045u95LVCQTf+DTLX9Owrqw3E=', 'Content-Length': '0'} 2022-09-12T20:08:36.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4c4e74ce34abe67-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.098 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:36.099 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:36.099 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:36.100 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:36.100 DEBUG:boto:Method: GET 2022-09-12T20:08:36.101 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:36.101 DEBUG:boto:Data: 2022-09-12T20:08:36.102 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.102 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.103 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.104 DEBUG:boto:Params: {} 2022-09-12T20:08:36.104 DEBUG:boto:Token: None 2022-09-12T20:08:36.105 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:36.105 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Kj045u95LVCQTf+DTLX9Owrqw3E= 2022-09-12T20:08:36.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Kj045u95LVCQTf+DTLX9Owrqw3E=', 'Content-Length': '0'} 2022-09-12T20:08:36.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e1531552a607aba-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.109 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:36.110 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:36.110 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:36.111 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:36.111 DEBUG:boto:Method: HEAD 2022-09-12T20:08:36.112 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:36.112 DEBUG:boto:Data: 2022-09-12T20:08:36.113 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.113 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.114 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.114 DEBUG:boto:Params: {} 2022-09-12T20:08:36.115 DEBUG:boto:Token: None 2022-09-12T20:08:36.116 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:36.116 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/9Uswd+qENbqPExHx64hptuPdxE= 2022-09-12T20:08:36.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/9Uswd+qENbqPExHx64hptuPdxE=', 'Content-Length': '0'} 2022-09-12T20:08:36.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000600de4b0e00c1749-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.120 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:36.120 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:36.121 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:36.121 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:36.122 DEBUG:boto:Method: HEAD 2022-09-12T20:08:36.122 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:36.123 DEBUG:boto:Data: 2022-09-12T20:08:36.123 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.124 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.125 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.125 DEBUG:boto:Params: {} 2022-09-12T20:08:36.126 DEBUG:boto:Token: None 2022-09-12T20:08:36.126 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:36.127 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/9Uswd+qENbqPExHx64hptuPdxE= 2022-09-12T20:08:36.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/9Uswd+qENbqPExHx64hptuPdxE=', 'Content-Length': '0'} 2022-09-12T20:08:36.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0e943f001353c9f-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:08:36.131 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:36.131 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:36.132 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:36.132 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:36.133 DEBUG:boto:Method: GET 2022-09-12T20:08:36.134 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:36.134 DEBUG:boto:Data: 2022-09-12T20:08:36.135 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.136 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.136 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.137 DEBUG:boto:Params: {} 2022-09-12T20:08:36.138 DEBUG:boto:Token: None 2022-09-12T20:08:36.139 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:36.139 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Kj045u95LVCQTf+DTLX9Owrqw3E= 2022-09-12T20:08:36.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Kj045u95LVCQTf+DTLX9Owrqw3E=', 'Content-Length': '0'} 2022-09-12T20:08:36.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e62c3274269bafc3-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.186 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:36.187 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:36.187 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:36.188 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:08:36.189 DEBUG:boto:Method: GET 2022-09-12T20:08:36.189 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:36.190 DEBUG:boto:Data: 2022-09-12T20:08:36.191 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.191 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.192 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.192 DEBUG:boto:Params: {} 2022-09-12T20:08:36.193 DEBUG:boto:Token: None 2022-09-12T20:08:36.194 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:08:36.195 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Kj045u95LVCQTf+DTLX9Owrqw3E= 2022-09-12T20:08:36.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Kj045u95LVCQTf+DTLX9Owrqw3E=', 'Content-Length': '0'} 2022-09-12T20:08:36.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082ba49059d393db9-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.199 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:36.199 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:36.200 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:08:36.200 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:08:36.201 DEBUG:boto:Method: GET 2022-09-12T20:08:36.202 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:08:36.203 DEBUG:boto:Data: 2022-09-12T20:08:36.203 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.204 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.205 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.206 DEBUG:boto:Params: {} 2022-09-12T20:08:36.207 DEBUG:boto:Token: None 2022-09-12T20:08:36.207 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/ 2022-09-12T20:08:36.208 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4= 2022-09-12T20:08:36.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4=', 'Content-Length': '0'} 2022-09-12T20:08:36.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a3d3f2a6d0864f1-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.213 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:36.214 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:36.215 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:36.215 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:08:36.216 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:08:36.216 DEBUG:boto:Method: GET 2022-09-12T20:08:36.217 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:08:36.218 DEBUG:boto:Data: 2022-09-12T20:08:36.219 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.220 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.221 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.221 DEBUG:boto:Params: {} 2022-09-12T20:08:36.222 DEBUG:boto:Token: None 2022-09-12T20:08:36.223 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/ 2022-09-12T20:08:36.224 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4= 2022-09-12T20:08:36.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4=', 'Content-Length': '0'} 2022-09-12T20:08:36.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b5f4aa12cbea219-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.229 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:36.230 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:36.230 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:36.231 DEBUG:boto:Method: HEAD 2022-09-12T20:08:36.231 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:08:36.232 DEBUG:boto:Data: 2022-09-12T20:08:36.233 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.233 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.234 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.234 DEBUG:boto:Params: {} 2022-09-12T20:08:36.235 DEBUG:boto:Token: None 2022-09-12T20:08:36.236 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj34 2022-09-12T20:08:36.236 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qE/uo73rcdlRRIk89ct3ianmUeg= 2022-09-12T20:08:36.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qE/uo73rcdlRRIk89ct3ianmUeg=', 'Content-Length': '0'} 2022-09-12T20:08:36.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e87e2a42c41bff55-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.240 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:36.241 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:36.241 DEBUG:boto:Method: HEAD 2022-09-12T20:08:36.242 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:08:36.243 DEBUG:boto:Data: 2022-09-12T20:08:36.243 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.244 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.244 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.245 DEBUG:boto:Params: {} 2022-09-12T20:08:36.246 DEBUG:boto:Token: None 2022-09-12T20:08:36.246 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj34 2022-09-12T20:08:36.247 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qE/uo73rcdlRRIk89ct3ianmUeg= 2022-09-12T20:08:36.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qE/uo73rcdlRRIk89ct3ianmUeg=', 'Content-Length': '0'} 2022-09-12T20:08:36.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffc3d16522c55053-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:08:36.251 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:36.252 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:36.253 DEBUG:boto:Method: GET 2022-09-12T20:08:36.254 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:08:36.255 DEBUG:boto:Data: 2022-09-12T20:08:36.256 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.257 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.257 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.258 DEBUG:boto:Params: {} 2022-09-12T20:08:36.259 DEBUG:boto:Token: None 2022-09-12T20:08:36.260 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj34 2022-09-12T20:08:36.261 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FT0xKDHunmotGG1X+dtFFyEtW+Q= 2022-09-12T20:08:36.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FT0xKDHunmotGG1X+dtFFyEtW+Q=', 'Content-Length': '0'} 2022-09-12T20:08:36.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016014e20875b8f1d-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.265 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:08:36.266 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:08:36.267 DEBUG:boto:Method: GET 2022-09-12T20:08:36.268 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:08:36.269 DEBUG:boto:Data: 2022-09-12T20:08:36.270 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.271 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.272 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.272 DEBUG:boto:Params: {} 2022-09-12T20:08:36.273 DEBUG:boto:Token: None 2022-09-12T20:08:36.274 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj34 2022-09-12T20:08:36.275 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FT0xKDHunmotGG1X+dtFFyEtW+Q= 2022-09-12T20:08:36.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FT0xKDHunmotGG1X+dtFFyEtW+Q=', 'Content-Length': '0'} 2022-09-12T20:08:36.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000efd171657f98107-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.280 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:36.281 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:36.281 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:08:36.282 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:08:36.283 DEBUG:boto:Method: GET 2022-09-12T20:08:36.284 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:08:36.285 DEBUG:boto:Data: 2022-09-12T20:08:36.286 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.287 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.287 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.288 DEBUG:boto:Params: {} 2022-09-12T20:08:36.289 DEBUG:boto:Token: None 2022-09-12T20:08:36.290 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/ 2022-09-12T20:08:36.291 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4= 2022-09-12T20:08:36.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4=', 'Content-Length': '0'} 2022-09-12T20:08:36.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075c0c01e2c08e558-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.296 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:36.297 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:36.298 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:36.299 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:08:36.299 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:08:36.300 DEBUG:boto:Method: GET 2022-09-12T20:08:36.301 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:08:36.302 DEBUG:boto:Data: 2022-09-12T20:08:36.303 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.304 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.305 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.306 DEBUG:boto:Params: {} 2022-09-12T20:08:36.307 DEBUG:boto:Token: None 2022-09-12T20:08:36.308 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/ 2022-09-12T20:08:36.309 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4= 2022-09-12T20:08:36.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4=', 'Content-Length': '0'} 2022-09-12T20:08:36.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019e696c9fbd1509b-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.354 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:36.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:08:36.356 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:36.357 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:36.358 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:36.358 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:36.359 DEBUG:boto:Method: GET 2022-09-12T20:08:36.360 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:36.361 DEBUG:boto:Data: 2022-09-12T20:08:36.362 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.363 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.363 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.364 DEBUG:boto:Params: {} 2022-09-12T20:08:36.365 DEBUG:boto:Token: None 2022-09-12T20:08:36.365 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:36.366 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fee4bkyyVwbRxsNzxBu+WAPewpY= 2022-09-12T20:08:36.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fee4bkyyVwbRxsNzxBu+WAPewpY=', 'Content-Length': '0'} 2022-09-12T20:08:36.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1f9e1ceb0cbab1e-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.370 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:36.371 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:36.372 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:36.372 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:36.373 DEBUG:boto:Method: GET 2022-09-12T20:08:36.374 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:36.374 DEBUG:boto:Data: 2022-09-12T20:08:36.375 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.376 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.376 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.377 DEBUG:boto:Params: {} 2022-09-12T20:08:36.378 DEBUG:boto:Token: None 2022-09-12T20:08:36.379 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:36.380 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fee4bkyyVwbRxsNzxBu+WAPewpY= 2022-09-12T20:08:36.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fee4bkyyVwbRxsNzxBu+WAPewpY=', 'Content-Length': '0'} 2022-09-12T20:08:36.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be1d31d8e4f543ec-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.384 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:36.384 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:36.385 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:36.385 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:36.386 DEBUG:boto:Method: HEAD 2022-09-12T20:08:36.386 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:36.387 DEBUG:boto:Data: 2022-09-12T20:08:36.388 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.388 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.389 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.389 DEBUG:boto:Params: {} 2022-09-12T20:08:36.390 DEBUG:boto:Token: None 2022-09-12T20:08:36.390 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:36.391 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3O22X7pTyi+h/qI7L3XLhm8inRw= 2022-09-12T20:08:36.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3O22X7pTyi+h/qI7L3XLhm8inRw=', 'Content-Length': '0'} 2022-09-12T20:08:36.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf9d3e6b5a67df4f-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.395 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:36.395 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:36.396 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:36.396 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:36.397 DEBUG:boto:Method: HEAD 2022-09-12T20:08:36.398 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:36.398 DEBUG:boto:Data: 2022-09-12T20:08:36.399 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.399 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.400 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.400 DEBUG:boto:Params: {} 2022-09-12T20:08:36.401 DEBUG:boto:Token: None 2022-09-12T20:08:36.401 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:36.402 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3O22X7pTyi+h/qI7L3XLhm8inRw= 2022-09-12T20:08:36.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3O22X7pTyi+h/qI7L3XLhm8inRw=', 'Content-Length': '0'} 2022-09-12T20:08:36.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d84dbaa04cfdd11-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:08:36.406 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:36.406 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:36.407 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:36.408 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:36.408 DEBUG:boto:Method: GET 2022-09-12T20:08:36.409 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:36.409 DEBUG:boto:Data: 2022-09-12T20:08:36.410 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.410 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.411 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.412 DEBUG:boto:Params: {} 2022-09-12T20:08:36.413 DEBUG:boto:Token: None 2022-09-12T20:08:36.414 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:36.415 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fee4bkyyVwbRxsNzxBu+WAPewpY= 2022-09-12T20:08:36.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fee4bkyyVwbRxsNzxBu+WAPewpY=', 'Content-Length': '0'} 2022-09-12T20:08:36.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000663799d9ffceb81b-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.419 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:36.419 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:36.420 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:36.420 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:08:36.421 DEBUG:boto:Method: GET 2022-09-12T20:08:36.422 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:36.422 DEBUG:boto:Data: 2022-09-12T20:08:36.423 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.423 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.424 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.424 DEBUG:boto:Params: {} 2022-09-12T20:08:36.425 DEBUG:boto:Token: None 2022-09-12T20:08:36.426 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:08:36.426 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fee4bkyyVwbRxsNzxBu+WAPewpY= 2022-09-12T20:08:36.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fee4bkyyVwbRxsNzxBu+WAPewpY=', 'Content-Length': '0'} 2022-09-12T20:08:36.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb0f3f0bbfaedf11-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.430 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:36.430 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:36.431 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:08:36.432 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:08:36.432 DEBUG:boto:Method: GET 2022-09-12T20:08:36.433 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:08:36.433 DEBUG:boto:Data: 2022-09-12T20:08:36.434 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.434 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.435 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.435 DEBUG:boto:Params: {} 2022-09-12T20:08:36.436 DEBUG:boto:Token: None 2022-09-12T20:08:36.437 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/ 2022-09-12T20:08:36.437 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4= 2022-09-12T20:08:36.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4=', 'Content-Length': '0'} 2022-09-12T20:08:36.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca84e7800ce4cb38-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.442 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:36.442 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:36.443 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:36.444 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:08:36.444 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:08:36.445 DEBUG:boto:Method: GET 2022-09-12T20:08:36.445 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:08:36.446 DEBUG:boto:Data: 2022-09-12T20:08:36.446 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.447 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.447 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.448 DEBUG:boto:Params: {} 2022-09-12T20:08:36.449 DEBUG:boto:Token: None 2022-09-12T20:08:36.449 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/ 2022-09-12T20:08:36.450 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4= 2022-09-12T20:08:36.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4=', 'Content-Length': '0'} 2022-09-12T20:08:36.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009899dca8d866abec-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.454 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:36.454 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:36.455 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:36.456 DEBUG:boto:Method: HEAD 2022-09-12T20:08:36.456 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:08:36.457 DEBUG:boto:Data: 2022-09-12T20:08:36.457 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.458 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.458 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.459 DEBUG:boto:Params: {} 2022-09-12T20:08:36.459 DEBUG:boto:Token: None 2022-09-12T20:08:36.460 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj38 2022-09-12T20:08:36.461 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BHzGpXs9fp0cLt512LmKyD11b7A= 2022-09-12T20:08:36.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BHzGpXs9fp0cLt512LmKyD11b7A=', 'Content-Length': '0'} 2022-09-12T20:08:36.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050a422e0cd4c915a-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.464 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:36.464 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:36.465 DEBUG:boto:Method: HEAD 2022-09-12T20:08:36.465 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:08:36.466 DEBUG:boto:Data: 2022-09-12T20:08:36.467 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.467 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.468 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.468 DEBUG:boto:Params: {} 2022-09-12T20:08:36.469 DEBUG:boto:Token: None 2022-09-12T20:08:36.469 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj38 2022-09-12T20:08:36.470 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BHzGpXs9fp0cLt512LmKyD11b7A= 2022-09-12T20:08:36.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BHzGpXs9fp0cLt512LmKyD11b7A=', 'Content-Length': '0'} 2022-09-12T20:08:36.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a192e49edf79bcf-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:08:36.474 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:36.474 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:36.475 DEBUG:boto:Method: GET 2022-09-12T20:08:36.476 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:08:36.476 DEBUG:boto:Data: 2022-09-12T20:08:36.477 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.477 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.478 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.478 DEBUG:boto:Params: {} 2022-09-12T20:08:36.479 DEBUG:boto:Token: None 2022-09-12T20:08:36.480 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj38 2022-09-12T20:08:36.480 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ASrJZXTtdiOlwGeFdmpneLYCKsA= 2022-09-12T20:08:36.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ASrJZXTtdiOlwGeFdmpneLYCKsA=', 'Content-Length': '0'} 2022-09-12T20:08:36.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003922b4ed469f580-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.484 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:08:36.484 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:08:36.485 DEBUG:boto:Method: GET 2022-09-12T20:08:36.486 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:08:36.486 DEBUG:boto:Data: 2022-09-12T20:08:36.487 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.487 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.488 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.488 DEBUG:boto:Params: {} 2022-09-12T20:08:36.489 DEBUG:boto:Token: None 2022-09-12T20:08:36.490 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj38 2022-09-12T20:08:36.490 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ASrJZXTtdiOlwGeFdmpneLYCKsA= 2022-09-12T20:08:36.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ASrJZXTtdiOlwGeFdmpneLYCKsA=', 'Content-Length': '0'} 2022-09-12T20:08:36.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000855496f4d3b5147a-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.494 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:36.495 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:36.496 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:08:36.496 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:08:36.497 DEBUG:boto:Method: GET 2022-09-12T20:08:36.497 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:08:36.498 DEBUG:boto:Data: 2022-09-12T20:08:36.498 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.499 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.499 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.500 DEBUG:boto:Params: {} 2022-09-12T20:08:36.500 DEBUG:boto:Token: None 2022-09-12T20:08:36.501 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/ 2022-09-12T20:08:36.502 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4= 2022-09-12T20:08:36.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4=', 'Content-Length': '0'} 2022-09-12T20:08:36.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe4a94e9abc6a09d-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.506 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:36.507 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:36.507 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:36.508 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:08:36.508 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:08:36.509 DEBUG:boto:Method: GET 2022-09-12T20:08:36.509 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:08:36.510 DEBUG:boto:Data: 2022-09-12T20:08:36.510 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.511 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.512 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.512 DEBUG:boto:Params: {} 2022-09-12T20:08:36.513 DEBUG:boto:Token: None 2022-09-12T20:08:36.513 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/ 2022-09-12T20:08:36.514 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4= 2022-09-12T20:08:36.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4=', 'Content-Length': '0'} 2022-09-12T20:08:36.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee79ac25b246ba16-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.518 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:36.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:08:36.519 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:36.520 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:36.520 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:36.521 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:36.521 DEBUG:boto:Method: GET 2022-09-12T20:08:36.522 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:36.522 DEBUG:boto:Data: 2022-09-12T20:08:36.523 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.524 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.524 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.525 DEBUG:boto:Params: {} 2022-09-12T20:08:36.525 DEBUG:boto:Token: None 2022-09-12T20:08:36.526 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:36.527 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rooFt1rV0F3S4kv5ZXFuw0AlfIo= 2022-09-12T20:08:36.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rooFt1rV0F3S4kv5ZXFuw0AlfIo=', 'Content-Length': '0'} 2022-09-12T20:08:36.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c5e5f367e02d530-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.530 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:36.531 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:36.531 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:36.532 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:36.532 DEBUG:boto:Method: GET 2022-09-12T20:08:36.533 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:36.534 DEBUG:boto:Data: 2022-09-12T20:08:36.534 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.535 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.535 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.536 DEBUG:boto:Params: {} 2022-09-12T20:08:36.536 DEBUG:boto:Token: None 2022-09-12T20:08:36.537 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:36.538 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rooFt1rV0F3S4kv5ZXFuw0AlfIo= 2022-09-12T20:08:36.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rooFt1rV0F3S4kv5ZXFuw0AlfIo=', 'Content-Length': '0'} 2022-09-12T20:08:36.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e27a55ea11574e15-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.542 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:36.542 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:36.543 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:36.543 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:36.544 DEBUG:boto:Method: HEAD 2022-09-12T20:08:36.544 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:36.545 DEBUG:boto:Data: 2022-09-12T20:08:36.545 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.546 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.546 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.547 DEBUG:boto:Params: {} 2022-09-12T20:08:36.548 DEBUG:boto:Token: None 2022-09-12T20:08:36.548 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:36.549 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RFJUO77msuX5MgEbqP0l/bdYges= 2022-09-12T20:08:36.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RFJUO77msuX5MgEbqP0l/bdYges=', 'Content-Length': '0'} 2022-09-12T20:08:36.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcffe5d3d4975e75-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.552 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:36.553 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:36.553 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:36.554 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:36.554 DEBUG:boto:Method: HEAD 2022-09-12T20:08:36.555 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:36.555 DEBUG:boto:Data: 2022-09-12T20:08:36.556 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.556 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.557 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.557 DEBUG:boto:Params: {} 2022-09-12T20:08:36.558 DEBUG:boto:Token: None 2022-09-12T20:08:36.559 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:36.559 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RFJUO77msuX5MgEbqP0l/bdYges= 2022-09-12T20:08:36.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RFJUO77msuX5MgEbqP0l/bdYges=', 'Content-Length': '0'} 2022-09-12T20:08:36.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008988c7263f745b99-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:08:36.564 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:36.564 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:36.565 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:36.565 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:36.566 DEBUG:boto:Method: GET 2022-09-12T20:08:36.566 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:36.567 DEBUG:boto:Data: 2022-09-12T20:08:36.568 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.568 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.569 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.569 DEBUG:boto:Params: {} 2022-09-12T20:08:36.570 DEBUG:boto:Token: None 2022-09-12T20:08:36.570 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:36.571 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rooFt1rV0F3S4kv5ZXFuw0AlfIo= 2022-09-12T20:08:36.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rooFt1rV0F3S4kv5ZXFuw0AlfIo=', 'Content-Length': '0'} 2022-09-12T20:08:36.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a78fe55911fa204-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.574 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:36.575 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:36.576 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:36.576 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:08:36.577 DEBUG:boto:Method: GET 2022-09-12T20:08:36.577 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:36.578 DEBUG:boto:Data: 2022-09-12T20:08:36.578 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.579 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.579 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.580 DEBUG:boto:Params: {} 2022-09-12T20:08:36.581 DEBUG:boto:Token: None 2022-09-12T20:08:36.581 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:08:36.582 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rooFt1rV0F3S4kv5ZXFuw0AlfIo= 2022-09-12T20:08:36.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rooFt1rV0F3S4kv5ZXFuw0AlfIo=', 'Content-Length': '0'} 2022-09-12T20:08:36.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5cafed80d198dcf-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.626 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:36.627 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:36.627 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:08:36.628 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:08:36.628 DEBUG:boto:Method: GET 2022-09-12T20:08:36.629 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:08:36.629 DEBUG:boto:Data: 2022-09-12T20:08:36.630 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.630 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.631 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.632 DEBUG:boto:Params: {} 2022-09-12T20:08:36.632 DEBUG:boto:Token: None 2022-09-12T20:08:36.633 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/ 2022-09-12T20:08:36.633 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4= 2022-09-12T20:08:36.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4=', 'Content-Length': '0'} 2022-09-12T20:08:36.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcb5067d6ea98ba7-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.637 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:36.638 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:36.639 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:36.640 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:08:36.640 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:08:36.641 DEBUG:boto:Method: GET 2022-09-12T20:08:36.641 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:08:36.642 DEBUG:boto:Data: 2022-09-12T20:08:36.642 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.643 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.643 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.644 DEBUG:boto:Params: {} 2022-09-12T20:08:36.644 DEBUG:boto:Token: None 2022-09-12T20:08:36.645 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/ 2022-09-12T20:08:36.646 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4= 2022-09-12T20:08:36.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4=', 'Content-Length': '0'} 2022-09-12T20:08:36.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076812d3fdb80964c-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.650 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:36.651 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:36.651 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:36.652 DEBUG:boto:Method: HEAD 2022-09-12T20:08:36.652 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:08:36.653 DEBUG:boto:Data: 2022-09-12T20:08:36.653 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.654 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.654 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.655 DEBUG:boto:Params: {} 2022-09-12T20:08:36.655 DEBUG:boto:Token: None 2022-09-12T20:08:36.656 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj44 2022-09-12T20:08:36.657 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uC4q37zd+9gOq4NdaIi9zwltqLo= 2022-09-12T20:08:36.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uC4q37zd+9gOq4NdaIi9zwltqLo=', 'Content-Length': '0'} 2022-09-12T20:08:36.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e4e162257e7c6dc-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.660 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:36.660 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:36.661 DEBUG:boto:Method: HEAD 2022-09-12T20:08:36.662 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:08:36.662 DEBUG:boto:Data: 2022-09-12T20:08:36.663 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.663 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.664 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.664 DEBUG:boto:Params: {} 2022-09-12T20:08:36.665 DEBUG:boto:Token: None 2022-09-12T20:08:36.665 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj44 2022-09-12T20:08:36.666 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uC4q37zd+9gOq4NdaIi9zwltqLo= 2022-09-12T20:08:36.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uC4q37zd+9gOq4NdaIi9zwltqLo=', 'Content-Length': '0'} 2022-09-12T20:08:36.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fda10fe0bbcf074-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:08:36.670 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:36.670 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:36.671 DEBUG:boto:Method: GET 2022-09-12T20:08:36.672 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:08:36.672 DEBUG:boto:Data: 2022-09-12T20:08:36.673 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.673 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.674 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.675 DEBUG:boto:Params: {} 2022-09-12T20:08:36.675 DEBUG:boto:Token: None 2022-09-12T20:08:36.676 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj44 2022-09-12T20:08:36.676 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DyAwW2CcA1hIWwrFuHt8AY8N2cc= 2022-09-12T20:08:36.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DyAwW2CcA1hIWwrFuHt8AY8N2cc=', 'Content-Length': '0'} 2022-09-12T20:08:36.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b89af3ee27a058fd-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.680 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:08:36.681 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:08:36.681 DEBUG:boto:Method: GET 2022-09-12T20:08:36.682 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:08:36.682 DEBUG:boto:Data: 2022-09-12T20:08:36.683 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.684 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.684 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.685 DEBUG:boto:Params: {} 2022-09-12T20:08:36.685 DEBUG:boto:Token: None 2022-09-12T20:08:36.686 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj44 2022-09-12T20:08:36.687 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DyAwW2CcA1hIWwrFuHt8AY8N2cc= 2022-09-12T20:08:36.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DyAwW2CcA1hIWwrFuHt8AY8N2cc=', 'Content-Length': '0'} 2022-09-12T20:08:36.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007930a4d6ec6b6fb3-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.690 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:36.691 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:36.691 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:08:36.692 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:08:36.692 DEBUG:boto:Method: GET 2022-09-12T20:08:36.693 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:08:36.693 DEBUG:boto:Data: 2022-09-12T20:08:36.694 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.695 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.695 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.696 DEBUG:boto:Params: {} 2022-09-12T20:08:36.696 DEBUG:boto:Token: None 2022-09-12T20:08:36.697 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/ 2022-09-12T20:08:36.697 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4= 2022-09-12T20:08:36.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4=', 'Content-Length': '0'} 2022-09-12T20:08:36.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fade531848cd620b-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.702 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:36.703 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:36.703 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:36.704 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:08:36.704 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:08:36.705 DEBUG:boto:Method: GET 2022-09-12T20:08:36.705 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:08:36.706 DEBUG:boto:Data: 2022-09-12T20:08:36.707 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.707 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.708 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.708 DEBUG:boto:Params: {} 2022-09-12T20:08:36.709 DEBUG:boto:Token: None 2022-09-12T20:08:36.709 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/ 2022-09-12T20:08:36.710 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4= 2022-09-12T20:08:36.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4=', 'Content-Length': '0'} 2022-09-12T20:08:36.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc4fc825627e6d66-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.714 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:36.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:08:36.715 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:36.716 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:36.717 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:36.717 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:36.718 DEBUG:boto:Method: GET 2022-09-12T20:08:36.718 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:36.719 DEBUG:boto:Data: 2022-09-12T20:08:36.719 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.720 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.721 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.721 DEBUG:boto:Params: {} 2022-09-12T20:08:36.722 DEBUG:boto:Token: None 2022-09-12T20:08:36.722 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:36.723 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2ZcoQtVfy7ESVZ6p6YZndtvxxaU= 2022-09-12T20:08:36.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2ZcoQtVfy7ESVZ6p6YZndtvxxaU=', 'Content-Length': '0'} 2022-09-12T20:08:36.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006da2b05ef1495129-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.727 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:36.727 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:36.728 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:36.728 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:36.729 DEBUG:boto:Method: GET 2022-09-12T20:08:36.730 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:36.730 DEBUG:boto:Data: 2022-09-12T20:08:36.731 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.731 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.732 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.732 DEBUG:boto:Params: {} 2022-09-12T20:08:36.733 DEBUG:boto:Token: None 2022-09-12T20:08:36.734 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:36.734 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2ZcoQtVfy7ESVZ6p6YZndtvxxaU= 2022-09-12T20:08:36.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2ZcoQtVfy7ESVZ6p6YZndtvxxaU=', 'Content-Length': '0'} 2022-09-12T20:08:36.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da77ceaaf566af02-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.738 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:36.739 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:36.739 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:36.740 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:36.740 DEBUG:boto:Method: HEAD 2022-09-12T20:08:36.741 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:36.741 DEBUG:boto:Data: 2022-09-12T20:08:36.742 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.743 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.743 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.744 DEBUG:boto:Params: {} 2022-09-12T20:08:36.744 DEBUG:boto:Token: None 2022-09-12T20:08:36.745 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:36.745 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l8IsFSlxyo5ElX5VZb9COVRWPLQ= 2022-09-12T20:08:36.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l8IsFSlxyo5ElX5VZb9COVRWPLQ=', 'Content-Length': '0'} 2022-09-12T20:08:36.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbea8c57d7a16df9-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.749 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:36.749 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:36.750 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:36.750 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:36.751 DEBUG:boto:Method: HEAD 2022-09-12T20:08:36.752 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:36.752 DEBUG:boto:Data: 2022-09-12T20:08:36.753 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.753 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.754 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.754 DEBUG:boto:Params: {} 2022-09-12T20:08:36.755 DEBUG:boto:Token: None 2022-09-12T20:08:36.756 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:36.756 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l8IsFSlxyo5ElX5VZb9COVRWPLQ= 2022-09-12T20:08:36.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l8IsFSlxyo5ElX5VZb9COVRWPLQ=', 'Content-Length': '0'} 2022-09-12T20:08:36.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024fd3dc4c7d01189-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:08:36.760 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:36.761 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:36.761 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:36.762 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:36.762 DEBUG:boto:Method: GET 2022-09-12T20:08:36.763 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:36.763 DEBUG:boto:Data: 2022-09-12T20:08:36.764 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.765 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.765 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.766 DEBUG:boto:Params: {} 2022-09-12T20:08:36.766 DEBUG:boto:Token: None 2022-09-12T20:08:36.767 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:36.768 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2ZcoQtVfy7ESVZ6p6YZndtvxxaU= 2022-09-12T20:08:36.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2ZcoQtVfy7ESVZ6p6YZndtvxxaU=', 'Content-Length': '0'} 2022-09-12T20:08:36.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f2732cd5055a292-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.814 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:36.815 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:36.815 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:36.816 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:08:36.817 DEBUG:boto:Method: GET 2022-09-12T20:08:36.817 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:36.818 DEBUG:boto:Data: 2022-09-12T20:08:36.818 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.819 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.819 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.820 DEBUG:boto:Params: {} 2022-09-12T20:08:36.821 DEBUG:boto:Token: None 2022-09-12T20:08:36.821 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:08:36.822 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2ZcoQtVfy7ESVZ6p6YZndtvxxaU= 2022-09-12T20:08:36.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2ZcoQtVfy7ESVZ6p6YZndtvxxaU=', 'Content-Length': '0'} 2022-09-12T20:08:36.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c739a53a963a0f40-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.826 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:36.826 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:36.827 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:08:36.827 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:08:36.828 DEBUG:boto:Method: GET 2022-09-12T20:08:36.828 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:08:36.829 DEBUG:boto:Data: 2022-09-12T20:08:36.829 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.830 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.831 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.831 DEBUG:boto:Params: {} 2022-09-12T20:08:36.832 DEBUG:boto:Token: None 2022-09-12T20:08:36.832 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/ 2022-09-12T20:08:36.833 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4= 2022-09-12T20:08:36.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4=', 'Content-Length': '0'} 2022-09-12T20:08:36.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a8d0c59fa184ef5-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.837 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:36.838 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:36.839 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:36.839 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:08:36.840 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:08:36.841 DEBUG:boto:Method: GET 2022-09-12T20:08:36.841 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:08:36.842 DEBUG:boto:Data: 2022-09-12T20:08:36.842 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.843 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.843 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.844 DEBUG:boto:Params: {} 2022-09-12T20:08:36.845 DEBUG:boto:Token: None 2022-09-12T20:08:36.845 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/ 2022-09-12T20:08:36.846 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4= 2022-09-12T20:08:36.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4=', 'Content-Length': '0'} 2022-09-12T20:08:36.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d5208a51423a07a-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.850 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:36.851 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:36.851 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:36.852 DEBUG:boto:Method: HEAD 2022-09-12T20:08:36.852 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:08:36.853 DEBUG:boto:Data: 2022-09-12T20:08:36.853 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.854 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.855 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.855 DEBUG:boto:Params: {} 2022-09-12T20:08:36.856 DEBUG:boto:Token: None 2022-09-12T20:08:36.856 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj51 2022-09-12T20:08:36.857 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s9bBSbvSn5zauQgHFPLIMvI0orI= 2022-09-12T20:08:36.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s9bBSbvSn5zauQgHFPLIMvI0orI=', 'Content-Length': '0'} 2022-09-12T20:08:36.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f4bd97bdc37b398-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.860 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:36.861 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:36.862 DEBUG:boto:Method: HEAD 2022-09-12T20:08:36.862 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:08:36.863 DEBUG:boto:Data: 2022-09-12T20:08:36.864 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.864 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.865 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.865 DEBUG:boto:Params: {} 2022-09-12T20:08:36.866 DEBUG:boto:Token: None 2022-09-12T20:08:36.867 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj51 2022-09-12T20:08:36.867 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s9bBSbvSn5zauQgHFPLIMvI0orI= 2022-09-12T20:08:36.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s9bBSbvSn5zauQgHFPLIMvI0orI=', 'Content-Length': '0'} 2022-09-12T20:08:36.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035c4941b229d5b40-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:08:36.871 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:36.872 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:36.872 DEBUG:boto:Method: GET 2022-09-12T20:08:36.873 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:08:36.873 DEBUG:boto:Data: 2022-09-12T20:08:36.874 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.874 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.875 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.876 DEBUG:boto:Params: {} 2022-09-12T20:08:36.876 DEBUG:boto:Token: None 2022-09-12T20:08:36.877 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj51 2022-09-12T20:08:36.878 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bnvhoCFVOyx+OhRBBEHKRUZxr08= 2022-09-12T20:08:36.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bnvhoCFVOyx+OhRBBEHKRUZxr08=', 'Content-Length': '0'} 2022-09-12T20:08:36.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083032539faf7c2b1-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.881 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:08:36.882 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:08:36.882 DEBUG:boto:Method: GET 2022-09-12T20:08:36.883 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:08:36.883 DEBUG:boto:Data: 2022-09-12T20:08:36.884 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.884 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.885 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.886 DEBUG:boto:Params: {} 2022-09-12T20:08:36.886 DEBUG:boto:Token: None 2022-09-12T20:08:36.887 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj51 2022-09-12T20:08:36.888 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bnvhoCFVOyx+OhRBBEHKRUZxr08= 2022-09-12T20:08:36.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bnvhoCFVOyx+OhRBBEHKRUZxr08=', 'Content-Length': '0'} 2022-09-12T20:08:36.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045de79f08ab69f48-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.891 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:36.892 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:36.892 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:08:36.893 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:08:36.893 DEBUG:boto:Method: GET 2022-09-12T20:08:36.894 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:08:36.895 DEBUG:boto:Data: 2022-09-12T20:08:36.895 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.896 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.896 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.897 DEBUG:boto:Params: {} 2022-09-12T20:08:36.897 DEBUG:boto:Token: None 2022-09-12T20:08:36.898 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/ 2022-09-12T20:08:36.899 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4= 2022-09-12T20:08:36.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4=', 'Content-Length': '0'} 2022-09-12T20:08:36.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000353de4c68ff66ec9-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.903 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:36.904 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:36.905 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:36.905 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:08:36.906 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:08:36.906 DEBUG:boto:Method: GET 2022-09-12T20:08:36.907 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:08:36.908 DEBUG:boto:Data: 2022-09-12T20:08:36.908 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.909 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.909 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.910 DEBUG:boto:Params: {} 2022-09-12T20:08:36.911 DEBUG:boto:Token: None 2022-09-12T20:08:36.911 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/ 2022-09-12T20:08:36.912 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4= 2022-09-12T20:08:36.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OmhjRKkgSCtz/Ith29gFtgky0n4=', 'Content-Length': '0'} 2022-09-12T20:08:36.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004860d66d79b954fd-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.958 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:36.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:08:36.959 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:36.960 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:36.960 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:36.961 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:36.962 DEBUG:boto:Method: GET 2022-09-12T20:08:36.962 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:36.963 DEBUG:boto:Data: 2022-09-12T20:08:36.963 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.964 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.965 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.965 DEBUG:boto:Params: {} 2022-09-12T20:08:36.966 DEBUG:boto:Token: None 2022-09-12T20:08:36.966 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:36.967 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:d7ZDeIsBsojjONRwzY1E6sUoGrI= 2022-09-12T20:08:36.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:d7ZDeIsBsojjONRwzY1E6sUoGrI=', 'Content-Length': '0'} 2022-09-12T20:08:36.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a9c2d625c604bfb-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.971 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:36.971 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:36.972 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:36.972 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:36.973 DEBUG:boto:Method: GET 2022-09-12T20:08:36.974 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:36.974 DEBUG:boto:Data: 2022-09-12T20:08:36.975 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.975 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.976 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.976 DEBUG:boto:Params: {} 2022-09-12T20:08:36.977 DEBUG:boto:Token: None 2022-09-12T20:08:36.977 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:36.978 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:d7ZDeIsBsojjONRwzY1E6sUoGrI= 2022-09-12T20:08:36.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:d7ZDeIsBsojjONRwzY1E6sUoGrI=', 'Content-Length': '0'} 2022-09-12T20:08:36.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a3e55ea7bfd3872-00631f91c4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.982 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:36.983 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:36.983 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:36.984 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:36.984 DEBUG:boto:Method: HEAD 2022-09-12T20:08:36.985 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:36.985 DEBUG:boto:Data: 2022-09-12T20:08:36.986 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.986 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:36.987 DEBUG:boto:Port: 8001 2022-09-12T20:08:36.987 DEBUG:boto:Params: {} 2022-09-12T20:08:36.988 DEBUG:boto:Token: None 2022-09-12T20:08:36.988 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:36.989 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WolFXf7GmJ11QvTld2T072B2/Ys= 2022-09-12T20:08:36.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WolFXf7GmJ11QvTld2T072B2/Ys=', 'Content-Length': '0'} 2022-09-12T20:08:36.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c63c14ca0c04bc37-00631f91c4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:36.992 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:36.993 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:36.994 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:36.994 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:36.995 DEBUG:boto:Method: HEAD 2022-09-12T20:08:36.995 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:36.996 DEBUG:boto:Data: 2022-09-12T20:08:36.996 DEBUG:boto:Headers: {} 2022-09-12T20:08:36.997 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:36.997 DEBUG:boto:Port: 8000 2022-09-12T20:08:36.998 DEBUG:boto:Params: {} 2022-09-12T20:08:36.998 DEBUG:boto:Token: None 2022-09-12T20:08:36.999 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:36 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:36.999 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WolFXf7GmJ11QvTld2T072B2/Ys= 2022-09-12T20:08:37.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WolFXf7GmJ11QvTld2T072B2/Ys=', 'Content-Length': '0'} 2022-09-12T20:08:37.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041a3e34d56b1652e-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:08:37.003 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:37.004 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:37.005 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:37.005 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:37.006 DEBUG:boto:Method: GET 2022-09-12T20:08:37.006 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:37.007 DEBUG:boto:Data: 2022-09-12T20:08:37.007 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.008 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.009 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.009 DEBUG:boto:Params: {} 2022-09-12T20:08:37.010 DEBUG:boto:Token: None 2022-09-12T20:08:37.010 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:37.011 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jLSZaxe/y3aFRve+wz0HGj3reBo= 2022-09-12T20:08:37.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jLSZaxe/y3aFRve+wz0HGj3reBo=', 'Content-Length': '0'} 2022-09-12T20:08:37.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5e31181a59c11df-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.015 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:37.015 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:37.016 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:37.016 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:08:37.017 DEBUG:boto:Method: GET 2022-09-12T20:08:37.018 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:37.018 DEBUG:boto:Data: 2022-09-12T20:08:37.019 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.019 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.020 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.020 DEBUG:boto:Params: {} 2022-09-12T20:08:37.021 DEBUG:boto:Token: None 2022-09-12T20:08:37.022 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:08:37.022 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jLSZaxe/y3aFRve+wz0HGj3reBo= 2022-09-12T20:08:37.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jLSZaxe/y3aFRve+wz0HGj3reBo=', 'Content-Length': '0'} 2022-09-12T20:08:37.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0f2cda4c56e57ec-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.026 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:37.027 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:37.027 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:08:37.028 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:08:37.028 DEBUG:boto:Method: GET 2022-09-12T20:08:37.029 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:08:37.029 DEBUG:boto:Data: 2022-09-12T20:08:37.030 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.030 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.031 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.031 DEBUG:boto:Params: {} 2022-09-12T20:08:37.032 DEBUG:boto:Token: None 2022-09-12T20:08:37.033 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/ 2022-09-12T20:08:37.033 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y= 2022-09-12T20:08:37.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y=', 'Content-Length': '0'} 2022-09-12T20:08:37.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ab6ffc01dc73543-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.078 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:37.079 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:37.079 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:37.080 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:08:37.081 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:08:37.081 DEBUG:boto:Method: GET 2022-09-12T20:08:37.082 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:08:37.082 DEBUG:boto:Data: 2022-09-12T20:08:37.083 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.083 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.084 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.084 DEBUG:boto:Params: {} 2022-09-12T20:08:37.085 DEBUG:boto:Token: None 2022-09-12T20:08:37.086 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/ 2022-09-12T20:08:37.086 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y= 2022-09-12T20:08:37.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y=', 'Content-Length': '0'} 2022-09-12T20:08:37.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026f65d88d5148e37-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.090 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:37.091 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:37.092 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:37.092 DEBUG:boto:Method: HEAD 2022-09-12T20:08:37.093 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:08:37.093 DEBUG:boto:Data: 2022-09-12T20:08:37.094 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.094 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.095 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.095 DEBUG:boto:Params: {} 2022-09-12T20:08:37.096 DEBUG:boto:Token: None 2022-09-12T20:08:37.096 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj55 2022-09-12T20:08:37.097 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6EzECLwaJzz4nRchmKRuHyr1/cY= 2022-09-12T20:08:37.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6EzECLwaJzz4nRchmKRuHyr1/cY=', 'Content-Length': '0'} 2022-09-12T20:08:37.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022f29f5441aab16c-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.101 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:37.101 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:37.102 DEBUG:boto:Method: HEAD 2022-09-12T20:08:37.102 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:08:37.103 DEBUG:boto:Data: 2022-09-12T20:08:37.103 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.104 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.104 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.105 DEBUG:boto:Params: {} 2022-09-12T20:08:37.106 DEBUG:boto:Token: None 2022-09-12T20:08:37.106 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj55 2022-09-12T20:08:37.107 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6EzECLwaJzz4nRchmKRuHyr1/cY= 2022-09-12T20:08:37.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6EzECLwaJzz4nRchmKRuHyr1/cY=', 'Content-Length': '0'} 2022-09-12T20:08:37.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eccbfde655e81d28-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:08:37.111 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:37.112 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:37.112 DEBUG:boto:Method: GET 2022-09-12T20:08:37.113 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:08:37.113 DEBUG:boto:Data: 2022-09-12T20:08:37.114 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.115 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.115 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.116 DEBUG:boto:Params: {} 2022-09-12T20:08:37.116 DEBUG:boto:Token: None 2022-09-12T20:08:37.117 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj55 2022-09-12T20:08:37.118 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NEO6Ay/aN0bjOS15jCGW+GMsQNc= 2022-09-12T20:08:37.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NEO6Ay/aN0bjOS15jCGW+GMsQNc=', 'Content-Length': '0'} 2022-09-12T20:08:37.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000724bf8b9ea3ed13f-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.122 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:08:37.122 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:08:37.123 DEBUG:boto:Method: GET 2022-09-12T20:08:37.123 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:08:37.124 DEBUG:boto:Data: 2022-09-12T20:08:37.125 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.125 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.126 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.126 DEBUG:boto:Params: {} 2022-09-12T20:08:37.127 DEBUG:boto:Token: None 2022-09-12T20:08:37.128 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj55 2022-09-12T20:08:37.128 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NEO6Ay/aN0bjOS15jCGW+GMsQNc= 2022-09-12T20:08:37.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NEO6Ay/aN0bjOS15jCGW+GMsQNc=', 'Content-Length': '0'} 2022-09-12T20:08:37.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe382bee3abb9557-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.132 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:37.132 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:37.133 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:08:37.134 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:08:37.134 DEBUG:boto:Method: GET 2022-09-12T20:08:37.135 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:08:37.135 DEBUG:boto:Data: 2022-09-12T20:08:37.136 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.137 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.137 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.138 DEBUG:boto:Params: {} 2022-09-12T20:08:37.139 DEBUG:boto:Token: None 2022-09-12T20:08:37.140 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/ 2022-09-12T20:08:37.140 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y= 2022-09-12T20:08:37.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y=', 'Content-Length': '0'} 2022-09-12T20:08:37.144 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d2f05e73f9ded79-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.145 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:37.146 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:37.147 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:37.147 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:08:37.148 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:08:37.149 DEBUG:boto:Method: GET 2022-09-12T20:08:37.149 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:08:37.150 DEBUG:boto:Data: 2022-09-12T20:08:37.150 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.151 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.151 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.152 DEBUG:boto:Params: {} 2022-09-12T20:08:37.153 DEBUG:boto:Token: None 2022-09-12T20:08:37.153 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/ 2022-09-12T20:08:37.154 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y= 2022-09-12T20:08:37.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y=', 'Content-Length': '0'} 2022-09-12T20:08:37.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e3134e8a84e5009-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.198 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:37.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:08:37.199 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:37.200 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:37.201 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:37.201 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:37.202 DEBUG:boto:Method: GET 2022-09-12T20:08:37.202 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:37.203 DEBUG:boto:Data: 2022-09-12T20:08:37.203 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.204 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.205 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.205 DEBUG:boto:Params: {} 2022-09-12T20:08:37.206 DEBUG:boto:Token: None 2022-09-12T20:08:37.206 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:37.207 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jlu2r+O4EgogfdNqbM/vZxzPN8o= 2022-09-12T20:08:37.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jlu2r+O4EgogfdNqbM/vZxzPN8o=', 'Content-Length': '0'} 2022-09-12T20:08:37.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012e8fe9b31a33ed9-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.211 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:37.212 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:37.212 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:37.213 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:37.214 DEBUG:boto:Method: GET 2022-09-12T20:08:37.214 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:37.215 DEBUG:boto:Data: 2022-09-12T20:08:37.215 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.216 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.217 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.217 DEBUG:boto:Params: {} 2022-09-12T20:08:37.218 DEBUG:boto:Token: None 2022-09-12T20:08:37.218 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:37.219 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jlu2r+O4EgogfdNqbM/vZxzPN8o= 2022-09-12T20:08:37.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jlu2r+O4EgogfdNqbM/vZxzPN8o=', 'Content-Length': '0'} 2022-09-12T20:08:37.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5616319c3e0c159-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.223 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:37.223 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:37.224 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:37.224 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:37.225 DEBUG:boto:Method: HEAD 2022-09-12T20:08:37.225 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:37.226 DEBUG:boto:Data: 2022-09-12T20:08:37.226 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.227 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.227 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.228 DEBUG:boto:Params: {} 2022-09-12T20:08:37.229 DEBUG:boto:Token: None 2022-09-12T20:08:37.229 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:37.230 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6CNLonA9Ech0YZNtl6IRD07ykjI= 2022-09-12T20:08:37.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6CNLonA9Ech0YZNtl6IRD07ykjI=', 'Content-Length': '0'} 2022-09-12T20:08:37.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcbfd50ff97e398b-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.233 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:37.234 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:37.234 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:37.235 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:37.235 DEBUG:boto:Method: HEAD 2022-09-12T20:08:37.236 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:37.236 DEBUG:boto:Data: 2022-09-12T20:08:37.237 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.237 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.238 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.238 DEBUG:boto:Params: {} 2022-09-12T20:08:37.239 DEBUG:boto:Token: None 2022-09-12T20:08:37.240 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:37.240 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6CNLonA9Ech0YZNtl6IRD07ykjI= 2022-09-12T20:08:37.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6CNLonA9Ech0YZNtl6IRD07ykjI=', 'Content-Length': '0'} 2022-09-12T20:08:37.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e23d1489272dc4cc-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:08:37.244 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:37.245 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:37.246 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:37.246 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:37.247 DEBUG:boto:Method: GET 2022-09-12T20:08:37.247 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:37.248 DEBUG:boto:Data: 2022-09-12T20:08:37.248 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.249 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.249 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.250 DEBUG:boto:Params: {} 2022-09-12T20:08:37.251 DEBUG:boto:Token: None 2022-09-12T20:08:37.252 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:37.253 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jlu2r+O4EgogfdNqbM/vZxzPN8o= 2022-09-12T20:08:37.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jlu2r+O4EgogfdNqbM/vZxzPN8o=', 'Content-Length': '0'} 2022-09-12T20:08:37.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000055426bf1a273753-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.258 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:37.259 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:37.260 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:37.261 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:08:37.262 DEBUG:boto:Method: GET 2022-09-12T20:08:37.263 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:37.264 DEBUG:boto:Data: 2022-09-12T20:08:37.266 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.267 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.268 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.269 DEBUG:boto:Params: {} 2022-09-12T20:08:37.270 DEBUG:boto:Token: None 2022-09-12T20:08:37.271 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:08:37.271 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jlu2r+O4EgogfdNqbM/vZxzPN8o= 2022-09-12T20:08:37.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jlu2r+O4EgogfdNqbM/vZxzPN8o=', 'Content-Length': '0'} 2022-09-12T20:08:37.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f446f02ee96d4749-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.318 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:37.319 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:37.319 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:08:37.320 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:08:37.321 DEBUG:boto:Method: GET 2022-09-12T20:08:37.321 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:08:37.322 DEBUG:boto:Data: 2022-09-12T20:08:37.322 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.323 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.324 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.324 DEBUG:boto:Params: {} 2022-09-12T20:08:37.325 DEBUG:boto:Token: None 2022-09-12T20:08:37.326 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/ 2022-09-12T20:08:37.326 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y= 2022-09-12T20:08:37.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y=', 'Content-Length': '0'} 2022-09-12T20:08:37.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043118cc75e1dd1e9-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.331 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:37.332 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:37.332 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:37.333 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:08:37.333 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:08:37.334 DEBUG:boto:Method: GET 2022-09-12T20:08:37.335 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:08:37.335 DEBUG:boto:Data: 2022-09-12T20:08:37.336 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.337 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.337 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.338 DEBUG:boto:Params: {} 2022-09-12T20:08:37.339 DEBUG:boto:Token: None 2022-09-12T20:08:37.339 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/ 2022-09-12T20:08:37.340 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y= 2022-09-12T20:08:37.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y=', 'Content-Length': '0'} 2022-09-12T20:08:37.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c710f9e13af0b54-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.345 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:37.345 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:37.346 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:37.347 DEBUG:boto:Method: HEAD 2022-09-12T20:08:37.347 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:08:37.348 DEBUG:boto:Data: 2022-09-12T20:08:37.348 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.349 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.350 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.350 DEBUG:boto:Params: {} 2022-09-12T20:08:37.351 DEBUG:boto:Token: None 2022-09-12T20:08:37.352 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj57 2022-09-12T20:08:37.352 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lWEPLHpmyf2VnHS3hLvCWwAKNsA= 2022-09-12T20:08:37.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lWEPLHpmyf2VnHS3hLvCWwAKNsA=', 'Content-Length': '0'} 2022-09-12T20:08:37.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066185e3afb451f05-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.356 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:37.356 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:37.357 DEBUG:boto:Method: HEAD 2022-09-12T20:08:37.358 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:08:37.358 DEBUG:boto:Data: 2022-09-12T20:08:37.359 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.359 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.360 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.360 DEBUG:boto:Params: {} 2022-09-12T20:08:37.361 DEBUG:boto:Token: None 2022-09-12T20:08:37.362 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj57 2022-09-12T20:08:37.362 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lWEPLHpmyf2VnHS3hLvCWwAKNsA= 2022-09-12T20:08:37.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lWEPLHpmyf2VnHS3hLvCWwAKNsA=', 'Content-Length': '0'} 2022-09-12T20:08:37.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f35d251c02edd9db-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:08:37.367 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:37.367 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:37.368 DEBUG:boto:Method: GET 2022-09-12T20:08:37.369 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:08:37.369 DEBUG:boto:Data: 2022-09-12T20:08:37.370 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.409 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.410 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.411 DEBUG:boto:Params: {} 2022-09-12T20:08:37.411 DEBUG:boto:Token: None 2022-09-12T20:08:37.412 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj57 2022-09-12T20:08:37.413 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UyfoUYACOFdsA6Zy7cLI+N2Mat0= 2022-09-12T20:08:37.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UyfoUYACOFdsA6Zy7cLI+N2Mat0=', 'Content-Length': '0'} 2022-09-12T20:08:37.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024efadf6534dd44a-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.416 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:08:37.417 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:08:37.418 DEBUG:boto:Method: GET 2022-09-12T20:08:37.418 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:08:37.419 DEBUG:boto:Data: 2022-09-12T20:08:37.419 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.420 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.420 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.421 DEBUG:boto:Params: {} 2022-09-12T20:08:37.422 DEBUG:boto:Token: None 2022-09-12T20:08:37.422 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj57 2022-09-12T20:08:37.423 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UyfoUYACOFdsA6Zy7cLI+N2Mat0= 2022-09-12T20:08:37.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UyfoUYACOFdsA6Zy7cLI+N2Mat0=', 'Content-Length': '0'} 2022-09-12T20:08:37.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f187dc3056fc68f-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.426 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:37.427 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:37.427 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:08:37.428 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:08:37.429 DEBUG:boto:Method: GET 2022-09-12T20:08:37.429 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:08:37.430 DEBUG:boto:Data: 2022-09-12T20:08:37.430 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.431 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.431 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.432 DEBUG:boto:Params: {} 2022-09-12T20:08:37.432 DEBUG:boto:Token: None 2022-09-12T20:08:37.433 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/ 2022-09-12T20:08:37.433 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y= 2022-09-12T20:08:37.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y=', 'Content-Length': '0'} 2022-09-12T20:08:37.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1f8cf1523e754fe-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.438 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:37.439 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:37.440 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:37.440 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:08:37.441 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:08:37.441 DEBUG:boto:Method: GET 2022-09-12T20:08:37.442 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:08:37.443 DEBUG:boto:Data: 2022-09-12T20:08:37.443 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.444 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.444 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.445 DEBUG:boto:Params: {} 2022-09-12T20:08:37.445 DEBUG:boto:Token: None 2022-09-12T20:08:37.446 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/ 2022-09-12T20:08:37.447 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y= 2022-09-12T20:08:37.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y=', 'Content-Length': '0'} 2022-09-12T20:08:37.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ada650154d10e606-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.450 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:37.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:08:37.452 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:37.452 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:37.453 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:37.453 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:37.454 DEBUG:boto:Method: GET 2022-09-12T20:08:37.454 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:37.455 DEBUG:boto:Data: 2022-09-12T20:08:37.455 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.456 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.457 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.457 DEBUG:boto:Params: {} 2022-09-12T20:08:37.458 DEBUG:boto:Token: None 2022-09-12T20:08:37.458 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:37.459 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DyJczFjaZE4TqgaLN9O7oieK/bg= 2022-09-12T20:08:37.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DyJczFjaZE4TqgaLN9O7oieK/bg=', 'Content-Length': '0'} 2022-09-12T20:08:37.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a1fdbd79ab6d9f3-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.462 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:37.463 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:37.464 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:37.464 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:37.465 DEBUG:boto:Method: GET 2022-09-12T20:08:37.465 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:37.466 DEBUG:boto:Data: 2022-09-12T20:08:37.466 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.467 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.467 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.468 DEBUG:boto:Params: {} 2022-09-12T20:08:37.468 DEBUG:boto:Token: None 2022-09-12T20:08:37.469 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:37.470 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DyJczFjaZE4TqgaLN9O7oieK/bg= 2022-09-12T20:08:37.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DyJczFjaZE4TqgaLN9O7oieK/bg=', 'Content-Length': '0'} 2022-09-12T20:08:37.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8bde678d3d7f8c9-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.514 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:37.515 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:37.515 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:37.516 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:37.516 DEBUG:boto:Method: HEAD 2022-09-12T20:08:37.517 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:37.517 DEBUG:boto:Data: 2022-09-12T20:08:37.518 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.518 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.519 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.519 DEBUG:boto:Params: {} 2022-09-12T20:08:37.520 DEBUG:boto:Token: None 2022-09-12T20:08:37.520 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:37.521 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9vn5NP4H//9yZE+lW59UIvcV58o= 2022-09-12T20:08:37.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9vn5NP4H//9yZE+lW59UIvcV58o=', 'Content-Length': '0'} 2022-09-12T20:08:37.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cf7b1c11f91d2fc-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.525 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:37.525 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:37.526 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:37.526 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:37.527 DEBUG:boto:Method: HEAD 2022-09-12T20:08:37.527 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:37.528 DEBUG:boto:Data: 2022-09-12T20:08:37.528 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.529 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.529 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.530 DEBUG:boto:Params: {} 2022-09-12T20:08:37.530 DEBUG:boto:Token: None 2022-09-12T20:08:37.531 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:37.531 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9vn5NP4H//9yZE+lW59UIvcV58o= 2022-09-12T20:08:37.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9vn5NP4H//9yZE+lW59UIvcV58o=', 'Content-Length': '0'} 2022-09-12T20:08:37.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fde8b8417ba8abc-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:08:37.535 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:37.536 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:37.536 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:37.537 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:37.537 DEBUG:boto:Method: GET 2022-09-12T20:08:37.538 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:37.539 DEBUG:boto:Data: 2022-09-12T20:08:37.539 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.540 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.540 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.541 DEBUG:boto:Params: {} 2022-09-12T20:08:37.541 DEBUG:boto:Token: None 2022-09-12T20:08:37.542 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:37.542 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DyJczFjaZE4TqgaLN9O7oieK/bg= 2022-09-12T20:08:37.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DyJczFjaZE4TqgaLN9O7oieK/bg=', 'Content-Length': '0'} 2022-09-12T20:08:37.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000df38c352925f647-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.546 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:37.547 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:37.547 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:37.548 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:08:37.548 DEBUG:boto:Method: GET 2022-09-12T20:08:37.549 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:37.549 DEBUG:boto:Data: 2022-09-12T20:08:37.550 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.550 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.551 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.551 DEBUG:boto:Params: {} 2022-09-12T20:08:37.552 DEBUG:boto:Token: None 2022-09-12T20:08:37.553 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:08:37.553 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DyJczFjaZE4TqgaLN9O7oieK/bg= 2022-09-12T20:08:37.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DyJczFjaZE4TqgaLN9O7oieK/bg=', 'Content-Length': '0'} 2022-09-12T20:08:37.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2537ee8861710ef-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.557 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:37.557 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:37.558 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:08:37.558 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:08:37.559 DEBUG:boto:Method: GET 2022-09-12T20:08:37.559 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:08:37.560 DEBUG:boto:Data: 2022-09-12T20:08:37.561 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.561 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.562 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.562 DEBUG:boto:Params: {} 2022-09-12T20:08:37.563 DEBUG:boto:Token: None 2022-09-12T20:08:37.563 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/ 2022-09-12T20:08:37.564 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y= 2022-09-12T20:08:37.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y=', 'Content-Length': '0'} 2022-09-12T20:08:37.567 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfa27447067904ed-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.568 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:37.569 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:37.569 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:37.570 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:08:37.571 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:08:37.571 DEBUG:boto:Method: GET 2022-09-12T20:08:37.572 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:08:37.572 DEBUG:boto:Data: 2022-09-12T20:08:37.573 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.573 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.574 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.574 DEBUG:boto:Params: {} 2022-09-12T20:08:37.575 DEBUG:boto:Token: None 2022-09-12T20:08:37.575 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/ 2022-09-12T20:08:37.576 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y= 2022-09-12T20:08:37.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y=', 'Content-Length': '0'} 2022-09-12T20:08:37.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ee1ac5a686e82fc-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.580 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:37.581 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:37.581 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:37.582 DEBUG:boto:Method: HEAD 2022-09-12T20:08:37.582 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:08:37.583 DEBUG:boto:Data: 2022-09-12T20:08:37.583 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.584 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.584 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.585 DEBUG:boto:Params: {} 2022-09-12T20:08:37.585 DEBUG:boto:Token: None 2022-09-12T20:08:37.586 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj68 2022-09-12T20:08:37.586 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SxmdTqo0S1wr2BeEv76J8T8TQUw= 2022-09-12T20:08:37.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SxmdTqo0S1wr2BeEv76J8T8TQUw=', 'Content-Length': '0'} 2022-09-12T20:08:37.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2c87b8a252f0ea5-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.590 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:37.590 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:37.591 DEBUG:boto:Method: HEAD 2022-09-12T20:08:37.591 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:08:37.591 DEBUG:boto:Data: 2022-09-12T20:08:37.592 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.592 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.593 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.593 DEBUG:boto:Params: {} 2022-09-12T20:08:37.594 DEBUG:boto:Token: None 2022-09-12T20:08:37.595 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj68 2022-09-12T20:08:37.595 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SxmdTqo0S1wr2BeEv76J8T8TQUw= 2022-09-12T20:08:37.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SxmdTqo0S1wr2BeEv76J8T8TQUw=', 'Content-Length': '0'} 2022-09-12T20:08:37.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a5cbfc614f32484-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:08:37.599 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:37.600 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:37.600 DEBUG:boto:Method: GET 2022-09-12T20:08:37.601 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:08:37.601 DEBUG:boto:Data: 2022-09-12T20:08:37.602 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.602 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.603 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.603 DEBUG:boto:Params: {} 2022-09-12T20:08:37.604 DEBUG:boto:Token: None 2022-09-12T20:08:37.605 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj68 2022-09-12T20:08:37.605 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Bk5O19ETl3db99QwOIteyutIDI4= 2022-09-12T20:08:37.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Bk5O19ETl3db99QwOIteyutIDI4=', 'Content-Length': '0'} 2022-09-12T20:08:37.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080d6a102927121eb-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.609 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:08:37.609 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:08:37.610 DEBUG:boto:Method: GET 2022-09-12T20:08:37.611 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:08:37.611 DEBUG:boto:Data: 2022-09-12T20:08:37.612 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.612 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.613 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.613 DEBUG:boto:Params: {} 2022-09-12T20:08:37.614 DEBUG:boto:Token: None 2022-09-12T20:08:37.614 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj68 2022-09-12T20:08:37.615 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Bk5O19ETl3db99QwOIteyutIDI4= 2022-09-12T20:08:37.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Bk5O19ETl3db99QwOIteyutIDI4=', 'Content-Length': '0'} 2022-09-12T20:08:37.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9a4b361d16a8079-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.662 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:37.663 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:37.663 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:08:37.664 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:08:37.664 DEBUG:boto:Method: GET 2022-09-12T20:08:37.665 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:08:37.665 DEBUG:boto:Data: 2022-09-12T20:08:37.666 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.666 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.667 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.667 DEBUG:boto:Params: {} 2022-09-12T20:08:37.668 DEBUG:boto:Token: None 2022-09-12T20:08:37.669 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/ 2022-09-12T20:08:37.669 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y= 2022-09-12T20:08:37.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y=', 'Content-Length': '0'} 2022-09-12T20:08:37.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffdfa140662c76bd-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.673 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:37.674 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:37.675 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:37.675 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:08:37.676 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:08:37.676 DEBUG:boto:Method: GET 2022-09-12T20:08:37.677 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:08:37.677 DEBUG:boto:Data: 2022-09-12T20:08:37.678 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.678 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.679 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.679 DEBUG:boto:Params: {} 2022-09-12T20:08:37.680 DEBUG:boto:Token: None 2022-09-12T20:08:37.680 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/ 2022-09-12T20:08:37.681 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y= 2022-09-12T20:08:37.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y=', 'Content-Length': '0'} 2022-09-12T20:08:37.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6a8ec893de184cc-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.685 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:37.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:08:37.687 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:37.687 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:37.688 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:37.688 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:37.689 DEBUG:boto:Method: GET 2022-09-12T20:08:37.689 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:37.690 DEBUG:boto:Data: 2022-09-12T20:08:37.690 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.691 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.691 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.692 DEBUG:boto:Params: {} 2022-09-12T20:08:37.692 DEBUG:boto:Token: None 2022-09-12T20:08:37.693 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:37.693 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LKrX0uwCShiwU3m9pEBGu/dtmzE= 2022-09-12T20:08:37.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LKrX0uwCShiwU3m9pEBGu/dtmzE=', 'Content-Length': '0'} 2022-09-12T20:08:37.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095aaa15b93e6dc02-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.697 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:37.698 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:37.698 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:37.699 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:37.699 DEBUG:boto:Method: GET 2022-09-12T20:08:37.700 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:37.700 DEBUG:boto:Data: 2022-09-12T20:08:37.701 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.701 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.702 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.702 DEBUG:boto:Params: {} 2022-09-12T20:08:37.703 DEBUG:boto:Token: None 2022-09-12T20:08:37.704 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:37.704 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LKrX0uwCShiwU3m9pEBGu/dtmzE= 2022-09-12T20:08:37.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LKrX0uwCShiwU3m9pEBGu/dtmzE=', 'Content-Length': '0'} 2022-09-12T20:08:37.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002a34f43d4c4b85d-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.708 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:37.708 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:37.709 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:37.709 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:37.710 DEBUG:boto:Method: HEAD 2022-09-12T20:08:37.711 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:37.711 DEBUG:boto:Data: 2022-09-12T20:08:37.712 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.712 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.713 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.713 DEBUG:boto:Params: {} 2022-09-12T20:08:37.714 DEBUG:boto:Token: None 2022-09-12T20:08:37.714 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:37.715 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ARhdG3PRTiHpGRg9p+PC5oSt8Ak= 2022-09-12T20:08:37.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ARhdG3PRTiHpGRg9p+PC5oSt8Ak=', 'Content-Length': '0'} 2022-09-12T20:08:37.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0265ad6eb94b6f6-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.718 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:37.719 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:37.719 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:37.720 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:37.720 DEBUG:boto:Method: HEAD 2022-09-12T20:08:37.721 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:37.721 DEBUG:boto:Data: 2022-09-12T20:08:37.722 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.722 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.723 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.723 DEBUG:boto:Params: {} 2022-09-12T20:08:37.724 DEBUG:boto:Token: None 2022-09-12T20:08:37.724 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:37.725 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ARhdG3PRTiHpGRg9p+PC5oSt8Ak= 2022-09-12T20:08:37.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ARhdG3PRTiHpGRg9p+PC5oSt8Ak=', 'Content-Length': '0'} 2022-09-12T20:08:37.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000461a9696debfd981-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:08:37.729 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:37.729 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:37.730 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:37.730 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:37.731 DEBUG:boto:Method: GET 2022-09-12T20:08:37.731 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:37.732 DEBUG:boto:Data: 2022-09-12T20:08:37.733 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.733 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.734 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.734 DEBUG:boto:Params: {} 2022-09-12T20:08:37.735 DEBUG:boto:Token: None 2022-09-12T20:08:37.735 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:37.736 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LKrX0uwCShiwU3m9pEBGu/dtmzE= 2022-09-12T20:08:37.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LKrX0uwCShiwU3m9pEBGu/dtmzE=', 'Content-Length': '0'} 2022-09-12T20:08:37.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2d713bea148e4c9-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.739 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:37.740 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:37.741 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:37.741 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:08:37.742 DEBUG:boto:Method: GET 2022-09-12T20:08:37.742 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:37.743 DEBUG:boto:Data: 2022-09-12T20:08:37.743 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.744 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.744 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.745 DEBUG:boto:Params: {} 2022-09-12T20:08:37.746 DEBUG:boto:Token: None 2022-09-12T20:08:37.746 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:08:37.747 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LKrX0uwCShiwU3m9pEBGu/dtmzE= 2022-09-12T20:08:37.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LKrX0uwCShiwU3m9pEBGu/dtmzE=', 'Content-Length': '0'} 2022-09-12T20:08:37.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096ce89a7c82728cb-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.750 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:37.751 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:37.751 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:08:37.752 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:08:37.752 DEBUG:boto:Method: GET 2022-09-12T20:08:37.753 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:08:37.753 DEBUG:boto:Data: 2022-09-12T20:08:37.754 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.754 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.755 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.755 DEBUG:boto:Params: {} 2022-09-12T20:08:37.756 DEBUG:boto:Token: None 2022-09-12T20:08:37.757 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/ 2022-09-12T20:08:37.757 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y= 2022-09-12T20:08:37.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y=', 'Content-Length': '0'} 2022-09-12T20:08:37.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7fcec34bbfe824c-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.761 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:37.762 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:37.763 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:37.763 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:08:37.764 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:08:37.764 DEBUG:boto:Method: GET 2022-09-12T20:08:37.765 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:08:37.765 DEBUG:boto:Data: 2022-09-12T20:08:37.766 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.766 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.767 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.768 DEBUG:boto:Params: {} 2022-09-12T20:08:37.768 DEBUG:boto:Token: None 2022-09-12T20:08:37.769 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/ 2022-09-12T20:08:37.769 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y= 2022-09-12T20:08:37.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y=', 'Content-Length': '0'} 2022-09-12T20:08:37.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd0e5d5cfff16996-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.773 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:37.774 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:37.775 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:37.775 DEBUG:boto:Method: HEAD 2022-09-12T20:08:37.776 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:08:37.776 DEBUG:boto:Data: 2022-09-12T20:08:37.777 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.777 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.778 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.778 DEBUG:boto:Params: {} 2022-09-12T20:08:37.779 DEBUG:boto:Token: None 2022-09-12T20:08:37.779 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj76 2022-09-12T20:08:37.780 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B7viSS+RubnZkAv7X/yg8VSaI0Y= 2022-09-12T20:08:37.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B7viSS+RubnZkAv7X/yg8VSaI0Y=', 'Content-Length': '0'} 2022-09-12T20:08:37.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d57c332c441936e-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.783 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:37.784 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:37.784 DEBUG:boto:Method: HEAD 2022-09-12T20:08:37.785 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:08:37.785 DEBUG:boto:Data: 2022-09-12T20:08:37.786 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.786 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.787 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.788 DEBUG:boto:Params: {} 2022-09-12T20:08:37.788 DEBUG:boto:Token: None 2022-09-12T20:08:37.789 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj76 2022-09-12T20:08:37.789 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B7viSS+RubnZkAv7X/yg8VSaI0Y= 2022-09-12T20:08:37.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B7viSS+RubnZkAv7X/yg8VSaI0Y=', 'Content-Length': '0'} 2022-09-12T20:08:37.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efa2de641ce53cd8-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:08:37.793 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:37.794 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:37.795 DEBUG:boto:Method: GET 2022-09-12T20:08:37.795 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:08:37.796 DEBUG:boto:Data: 2022-09-12T20:08:37.796 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.797 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.797 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.798 DEBUG:boto:Params: {} 2022-09-12T20:08:37.798 DEBUG:boto:Token: None 2022-09-12T20:08:37.799 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj76 2022-09-12T20:08:37.800 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1JosicBi0Ff+TR201fyNTLjGbfA= 2022-09-12T20:08:37.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1JosicBi0Ff+TR201fyNTLjGbfA=', 'Content-Length': '0'} 2022-09-12T20:08:37.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7ab6d78509517eb-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.846 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:08:37.847 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:08:37.847 DEBUG:boto:Method: GET 2022-09-12T20:08:37.848 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:08:37.848 DEBUG:boto:Data: 2022-09-12T20:08:37.849 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.849 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.850 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.851 DEBUG:boto:Params: {} 2022-09-12T20:08:37.851 DEBUG:boto:Token: None 2022-09-12T20:08:37.852 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj76 2022-09-12T20:08:37.852 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1JosicBi0Ff+TR201fyNTLjGbfA= 2022-09-12T20:08:37.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1JosicBi0Ff+TR201fyNTLjGbfA=', 'Content-Length': '0'} 2022-09-12T20:08:37.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fb57324e46196a5-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.856 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:37.857 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:37.857 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:08:37.858 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:08:37.858 DEBUG:boto:Method: GET 2022-09-12T20:08:37.859 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:08:37.859 DEBUG:boto:Data: 2022-09-12T20:08:37.860 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.860 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.861 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.861 DEBUG:boto:Params: {} 2022-09-12T20:08:37.862 DEBUG:boto:Token: None 2022-09-12T20:08:37.863 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/ 2022-09-12T20:08:37.863 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y= 2022-09-12T20:08:37.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y=', 'Content-Length': '0'} 2022-09-12T20:08:37.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000547fe905e138cb01-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.867 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:37.868 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:37.869 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:37.869 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:08:37.870 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:08:37.870 DEBUG:boto:Method: GET 2022-09-12T20:08:37.871 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:08:37.871 DEBUG:boto:Data: 2022-09-12T20:08:37.872 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.872 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.873 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.873 DEBUG:boto:Params: {} 2022-09-12T20:08:37.874 DEBUG:boto:Token: None 2022-09-12T20:08:37.875 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/ 2022-09-12T20:08:37.875 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y= 2022-09-12T20:08:37.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y=', 'Content-Length': '0'} 2022-09-12T20:08:37.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d2ce9614a696ecb-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.879 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:37.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:08:37.880 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:37.881 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:37.881 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:37.882 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:37.883 DEBUG:boto:Method: GET 2022-09-12T20:08:37.883 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:37.884 DEBUG:boto:Data: 2022-09-12T20:08:37.884 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.885 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.885 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.886 DEBUG:boto:Params: {} 2022-09-12T20:08:37.886 DEBUG:boto:Token: None 2022-09-12T20:08:37.887 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:37.888 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u58h4ayu/U5PpqRgDcoJwt/carc= 2022-09-12T20:08:37.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u58h4ayu/U5PpqRgDcoJwt/carc=', 'Content-Length': '0'} 2022-09-12T20:08:37.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f79d66891f12e4b-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.891 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:37.892 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:37.892 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:37.893 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:37.893 DEBUG:boto:Method: GET 2022-09-12T20:08:37.894 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:37.894 DEBUG:boto:Data: 2022-09-12T20:08:37.895 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.895 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.896 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.896 DEBUG:boto:Params: {} 2022-09-12T20:08:37.897 DEBUG:boto:Token: None 2022-09-12T20:08:37.898 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:37.898 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u58h4ayu/U5PpqRgDcoJwt/carc= 2022-09-12T20:08:37.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u58h4ayu/U5PpqRgDcoJwt/carc=', 'Content-Length': '0'} 2022-09-12T20:08:37.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a8ae8a7a1354361-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.942 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:37.943 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:37.944 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:37.944 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:37.945 DEBUG:boto:Method: HEAD 2022-09-12T20:08:37.945 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:37.946 DEBUG:boto:Data: 2022-09-12T20:08:37.946 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.947 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.947 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.948 DEBUG:boto:Params: {} 2022-09-12T20:08:37.948 DEBUG:boto:Token: None 2022-09-12T20:08:37.949 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:37.950 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JoNmXRy2jsi7qgdD7TR2Smyhh0E= 2022-09-12T20:08:37.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JoNmXRy2jsi7qgdD7TR2Smyhh0E=', 'Content-Length': '0'} 2022-09-12T20:08:37.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e93b2eb53405e0b0-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.953 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:37.954 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:37.955 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:37.955 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:37.956 DEBUG:boto:Method: HEAD 2022-09-12T20:08:37.956 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:37.957 DEBUG:boto:Data: 2022-09-12T20:08:37.957 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.958 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.958 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.959 DEBUG:boto:Params: {} 2022-09-12T20:08:37.959 DEBUG:boto:Token: None 2022-09-12T20:08:37.960 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:37.960 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JoNmXRy2jsi7qgdD7TR2Smyhh0E= 2022-09-12T20:08:37.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JoNmXRy2jsi7qgdD7TR2Smyhh0E=', 'Content-Length': '0'} 2022-09-12T20:08:37.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a50f9de9345496d9-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:08:37.964 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:37.965 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:37.965 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:37.966 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:37.967 DEBUG:boto:Method: GET 2022-09-12T20:08:37.967 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:37.968 DEBUG:boto:Data: 2022-09-12T20:08:37.968 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.969 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.969 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.970 DEBUG:boto:Params: {} 2022-09-12T20:08:37.970 DEBUG:boto:Token: None 2022-09-12T20:08:37.971 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:37.972 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u58h4ayu/U5PpqRgDcoJwt/carc= 2022-09-12T20:08:37.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u58h4ayu/U5PpqRgDcoJwt/carc=', 'Content-Length': '0'} 2022-09-12T20:08:37.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018ffc5199bcb6129-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.975 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:37.976 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:37.976 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:37.977 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:08:37.977 DEBUG:boto:Method: GET 2022-09-12T20:08:37.978 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:37.978 DEBUG:boto:Data: 2022-09-12T20:08:37.979 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.980 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:37.980 DEBUG:boto:Port: 8000 2022-09-12T20:08:37.981 DEBUG:boto:Params: {} 2022-09-12T20:08:37.981 DEBUG:boto:Token: None 2022-09-12T20:08:37.982 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:08:37.982 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u58h4ayu/U5PpqRgDcoJwt/carc= 2022-09-12T20:08:37.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u58h4ayu/U5PpqRgDcoJwt/carc=', 'Content-Length': '0'} 2022-09-12T20:08:37.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007591679fe4109516-00631f91c5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:37.986 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:37.987 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:37.987 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:08:37.988 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:08:37.988 DEBUG:boto:Method: GET 2022-09-12T20:08:37.989 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:08:37.990 DEBUG:boto:Data: 2022-09-12T20:08:37.990 DEBUG:boto:Headers: {} 2022-09-12T20:08:37.991 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:37.991 DEBUG:boto:Port: 8001 2022-09-12T20:08:37.992 DEBUG:boto:Params: {} 2022-09-12T20:08:37.992 DEBUG:boto:Token: None 2022-09-12T20:08:37.993 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:37 GMT /jdpiak-75/ 2022-09-12T20:08:37.993 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y= 2022-09-12T20:08:37.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:37 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1IwSldHhyUBQ8awaElsGHhLzg2Y=', 'Content-Length': '0'} 2022-09-12T20:08:37.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a47c9df450dc3a70-00631f91c5-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.038 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:38.039 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:38.040 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:38.040 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:08:38.041 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:08:38.041 DEBUG:boto:Method: GET 2022-09-12T20:08:38.042 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:08:38.043 DEBUG:boto:Data: 2022-09-12T20:08:38.044 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.044 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:38.045 DEBUG:boto:Port: 8000 2022-09-12T20:08:38.046 DEBUG:boto:Params: {} 2022-09-12T20:08:38.046 DEBUG:boto:Token: None 2022-09-12T20:08:38.047 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/ 2022-09-12T20:08:38.048 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WZP+/xPDVPbUoQ//wqt11NdJchs= 2022-09-12T20:08:38.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WZP+/xPDVPbUoQ//wqt11NdJchs=', 'Content-Length': '0'} 2022-09-12T20:08:38.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077c7fc4c75ebd6b8-00631f91c6-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.052 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:38.053 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:38.053 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:38.054 DEBUG:boto:Method: HEAD 2022-09-12T20:08:38.054 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:08:38.055 DEBUG:boto:Data: 2022-09-12T20:08:38.056 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.056 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:38.057 DEBUG:boto:Port: 8001 2022-09-12T20:08:38.057 DEBUG:boto:Params: {} 2022-09-12T20:08:38.058 DEBUG:boto:Token: None 2022-09-12T20:08:38.058 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/obj85 2022-09-12T20:08:38.059 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cYPL4pJ7YtxX2VGlbfFW7ru+bvI= 2022-09-12T20:08:38.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cYPL4pJ7YtxX2VGlbfFW7ru+bvI=', 'Content-Length': '0'} 2022-09-12T20:08:38.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000986d089fdaf0c624-00631f91c6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.063 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:38.063 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:38.064 DEBUG:boto:Method: HEAD 2022-09-12T20:08:38.064 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:08:38.065 DEBUG:boto:Data: 2022-09-12T20:08:38.065 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.066 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:38.066 DEBUG:boto:Port: 8000 2022-09-12T20:08:38.067 DEBUG:boto:Params: {} 2022-09-12T20:08:38.067 DEBUG:boto:Token: None 2022-09-12T20:08:38.068 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/obj85 2022-09-12T20:08:38.068 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cYPL4pJ7YtxX2VGlbfFW7ru+bvI= 2022-09-12T20:08:38.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cYPL4pJ7YtxX2VGlbfFW7ru+bvI=', 'Content-Length': '0'} 2022-09-12T20:08:38.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088e4f136399eb391-00631f91c6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:08:38.072 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:38.073 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:38.073 DEBUG:boto:Method: GET 2022-09-12T20:08:38.074 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:08:38.075 DEBUG:boto:Data: 2022-09-12T20:08:38.075 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.076 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:38.076 DEBUG:boto:Port: 8001 2022-09-12T20:08:38.077 DEBUG:boto:Params: {} 2022-09-12T20:08:38.077 DEBUG:boto:Token: None 2022-09-12T20:08:38.078 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/obj85 2022-09-12T20:08:38.078 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PDxgeVwkLITPvMusgWEOpw9irBQ= 2022-09-12T20:08:38.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PDxgeVwkLITPvMusgWEOpw9irBQ=', 'Content-Length': '0'} 2022-09-12T20:08:38.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce6df521f03ca60f-00631f91c6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.082 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:08:38.082 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:08:38.083 DEBUG:boto:Method: GET 2022-09-12T20:08:38.084 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:08:38.084 DEBUG:boto:Data: 2022-09-12T20:08:38.085 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.085 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:38.086 DEBUG:boto:Port: 8000 2022-09-12T20:08:38.086 DEBUG:boto:Params: {} 2022-09-12T20:08:38.087 DEBUG:boto:Token: None 2022-09-12T20:08:38.087 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/obj85 2022-09-12T20:08:38.088 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PDxgeVwkLITPvMusgWEOpw9irBQ= 2022-09-12T20:08:38.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PDxgeVwkLITPvMusgWEOpw9irBQ=', 'Content-Length': '0'} 2022-09-12T20:08:38.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d06c4371a5d1f995-00631f91c6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.091 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:38.092 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:38.092 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:08:38.093 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:08:38.093 DEBUG:boto:Method: GET 2022-09-12T20:08:38.094 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:08:38.094 DEBUG:boto:Data: 2022-09-12T20:08:38.095 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.095 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:38.096 DEBUG:boto:Port: 8001 2022-09-12T20:08:38.096 DEBUG:boto:Params: {} 2022-09-12T20:08:38.097 DEBUG:boto:Token: None 2022-09-12T20:08:38.097 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/ 2022-09-12T20:08:38.098 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WZP+/xPDVPbUoQ//wqt11NdJchs= 2022-09-12T20:08:38.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WZP+/xPDVPbUoQ//wqt11NdJchs=', 'Content-Length': '0'} 2022-09-12T20:08:38.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047c5f8f60da12953-00631f91c6-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.102 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:38.103 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:38.104 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:38.104 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:08:38.104 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:08:38.105 DEBUG:boto:Method: GET 2022-09-12T20:08:38.106 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:08:38.106 DEBUG:boto:Data: 2022-09-12T20:08:38.107 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.107 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:38.108 DEBUG:boto:Port: 8000 2022-09-12T20:08:38.108 DEBUG:boto:Params: {} 2022-09-12T20:08:38.109 DEBUG:boto:Token: None 2022-09-12T20:08:38.109 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/ 2022-09-12T20:08:38.110 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WZP+/xPDVPbUoQ//wqt11NdJchs= 2022-09-12T20:08:38.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WZP+/xPDVPbUoQ//wqt11NdJchs=', 'Content-Length': '0'} 2022-09-12T20:08:38.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006dcdc07006f77b6-00631f91c6-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.154 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:38.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:08:38.155 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:38.156 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:38.156 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:38.157 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:38.158 DEBUG:boto:Method: GET 2022-09-12T20:08:38.158 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:38.159 DEBUG:boto:Data: 2022-09-12T20:08:38.159 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.160 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:38.160 DEBUG:boto:Port: 8001 2022-09-12T20:08:38.161 DEBUG:boto:Params: {} 2022-09-12T20:08:38.162 DEBUG:boto:Token: None 2022-09-12T20:08:38.162 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:38.163 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OwPXXo95DlP2zAInRt5NCQXkv4A= 2022-09-12T20:08:38.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OwPXXo95DlP2zAInRt5NCQXkv4A=', 'Content-Length': '0'} 2022-09-12T20:08:38.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040f51cd6b49d62ba-00631f91c6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.167 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:38.167 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:38.168 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:38.168 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:38.169 DEBUG:boto:Method: GET 2022-09-12T20:08:38.169 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:38.170 DEBUG:boto:Data: 2022-09-12T20:08:38.170 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.171 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:38.172 DEBUG:boto:Port: 8000 2022-09-12T20:08:38.172 DEBUG:boto:Params: {} 2022-09-12T20:08:38.173 DEBUG:boto:Token: None 2022-09-12T20:08:38.173 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:38.174 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OwPXXo95DlP2zAInRt5NCQXkv4A= 2022-09-12T20:08:38.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OwPXXo95DlP2zAInRt5NCQXkv4A=', 'Content-Length': '0'} 2022-09-12T20:08:38.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006200289b68d4ba91-00631f91c6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.178 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:38.178 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:38.179 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:38.179 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:38.180 DEBUG:boto:Method: HEAD 2022-09-12T20:08:38.180 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:38.181 DEBUG:boto:Data: 2022-09-12T20:08:38.182 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.182 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:38.183 DEBUG:boto:Port: 8001 2022-09-12T20:08:38.183 DEBUG:boto:Params: {} 2022-09-12T20:08:38.184 DEBUG:boto:Token: None 2022-09-12T20:08:38.184 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:38.185 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sd9cM1KEgc0jR6nuq6gLYaNJU0I= 2022-09-12T20:08:38.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sd9cM1KEgc0jR6nuq6gLYaNJU0I=', 'Content-Length': '0'} 2022-09-12T20:08:38.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9ee52aad3348466-00631f91c6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.189 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:38.189 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:38.190 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:38.191 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:38.191 DEBUG:boto:Method: HEAD 2022-09-12T20:08:38.192 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:38.192 DEBUG:boto:Data: 2022-09-12T20:08:38.193 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.193 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:38.194 DEBUG:boto:Port: 8000 2022-09-12T20:08:38.195 DEBUG:boto:Params: {} 2022-09-12T20:08:38.195 DEBUG:boto:Token: None 2022-09-12T20:08:38.196 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:38.196 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sd9cM1KEgc0jR6nuq6gLYaNJU0I= 2022-09-12T20:08:38.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sd9cM1KEgc0jR6nuq6gLYaNJU0I=', 'Content-Length': '0'} 2022-09-12T20:08:38.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054b35010bdcf3564-00631f91c6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:08:38.200 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:38.201 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:38.202 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:38.202 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:38.203 DEBUG:boto:Method: GET 2022-09-12T20:08:38.204 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:38.204 DEBUG:boto:Data: 2022-09-12T20:08:38.205 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.206 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:38.206 DEBUG:boto:Port: 8001 2022-09-12T20:08:38.207 DEBUG:boto:Params: {} 2022-09-12T20:08:38.208 DEBUG:boto:Token: None 2022-09-12T20:08:38.209 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:38.209 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OwPXXo95DlP2zAInRt5NCQXkv4A= 2022-09-12T20:08:38.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OwPXXo95DlP2zAInRt5NCQXkv4A=', 'Content-Length': '0'} 2022-09-12T20:08:38.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000206852f2409a747d-00631f91c6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.214 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:38.214 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:38.215 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:38.216 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:08:38.216 DEBUG:boto:Method: GET 2022-09-12T20:08:38.217 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:38.218 DEBUG:boto:Data: 2022-09-12T20:08:38.218 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.219 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:38.220 DEBUG:boto:Port: 8000 2022-09-12T20:08:38.220 DEBUG:boto:Params: {} 2022-09-12T20:08:38.221 DEBUG:boto:Token: None 2022-09-12T20:08:38.222 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:08:38.222 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OwPXXo95DlP2zAInRt5NCQXkv4A= 2022-09-12T20:08:38.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OwPXXo95DlP2zAInRt5NCQXkv4A=', 'Content-Length': '0'} 2022-09-12T20:08:38.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ca2bcfb6dbfa1cf-00631f91c6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.226 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:38.227 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:38.228 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:08:38.228 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:08:38.229 DEBUG:boto:Method: GET 2022-09-12T20:08:38.229 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:08:38.230 DEBUG:boto:Data: 2022-09-12T20:08:38.231 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.231 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:38.232 DEBUG:boto:Port: 8001 2022-09-12T20:08:38.233 DEBUG:boto:Params: {} 2022-09-12T20:08:38.233 DEBUG:boto:Token: None 2022-09-12T20:08:38.234 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/ 2022-09-12T20:08:38.235 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WZP+/xPDVPbUoQ//wqt11NdJchs= 2022-09-12T20:08:38.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WZP+/xPDVPbUoQ//wqt11NdJchs=', 'Content-Length': '0'} 2022-09-12T20:08:38.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000491b169f7e697a30-00631f91c6-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.239 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:38.240 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:38.241 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:38.241 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:08:38.242 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:08:38.243 DEBUG:boto:Method: GET 2022-09-12T20:08:38.243 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:08:38.244 DEBUG:boto:Data: 2022-09-12T20:08:38.244 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.245 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:38.246 DEBUG:boto:Port: 8000 2022-09-12T20:08:38.246 DEBUG:boto:Params: {} 2022-09-12T20:08:38.247 DEBUG:boto:Token: None 2022-09-12T20:08:38.248 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/ 2022-09-12T20:08:38.248 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WZP+/xPDVPbUoQ//wqt11NdJchs= 2022-09-12T20:08:38.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WZP+/xPDVPbUoQ//wqt11NdJchs=', 'Content-Length': '0'} 2022-09-12T20:08:38.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f228b758d59c1a3-00631f91c6-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.253 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:38.254 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:38.254 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:38.255 DEBUG:boto:Method: HEAD 2022-09-12T20:08:38.255 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:08:38.256 DEBUG:boto:Data: 2022-09-12T20:08:38.256 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.257 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:38.258 DEBUG:boto:Port: 8001 2022-09-12T20:08:38.258 DEBUG:boto:Params: {} 2022-09-12T20:08:38.259 DEBUG:boto:Token: None 2022-09-12T20:08:38.259 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/obj95 2022-09-12T20:08:38.260 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0SAtNoRGsInFaAuwikKGbdbBXZM= 2022-09-12T20:08:38.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0SAtNoRGsInFaAuwikKGbdbBXZM=', 'Content-Length': '0'} 2022-09-12T20:08:38.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c0349c1a7b3b197-00631f91c6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.264 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:38.264 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:38.265 DEBUG:boto:Method: HEAD 2022-09-12T20:08:38.265 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:08:38.266 DEBUG:boto:Data: 2022-09-12T20:08:38.266 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.267 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:38.268 DEBUG:boto:Port: 8000 2022-09-12T20:08:38.268 DEBUG:boto:Params: {} 2022-09-12T20:08:38.269 DEBUG:boto:Token: None 2022-09-12T20:08:38.270 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/obj95 2022-09-12T20:08:38.270 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0SAtNoRGsInFaAuwikKGbdbBXZM= 2022-09-12T20:08:38.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0SAtNoRGsInFaAuwikKGbdbBXZM=', 'Content-Length': '0'} 2022-09-12T20:08:38.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f94d5d49a71da73f-00631f91c6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:08:38.275 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:38.276 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:38.276 DEBUG:boto:Method: GET 2022-09-12T20:08:38.277 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:08:38.278 DEBUG:boto:Data: 2022-09-12T20:08:38.279 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.279 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:38.280 DEBUG:boto:Port: 8001 2022-09-12T20:08:38.281 DEBUG:boto:Params: {} 2022-09-12T20:08:38.282 DEBUG:boto:Token: None 2022-09-12T20:08:38.283 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/obj95 2022-09-12T20:08:38.284 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MOnr8sKp8EqD2luMWrODFf/i5JY= 2022-09-12T20:08:38.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MOnr8sKp8EqD2luMWrODFf/i5JY=', 'Content-Length': '0'} 2022-09-12T20:08:38.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c61f27ef367c688-00631f91c6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.330 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:08:38.331 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:08:38.331 DEBUG:boto:Method: GET 2022-09-12T20:08:38.332 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:08:38.332 DEBUG:boto:Data: 2022-09-12T20:08:38.333 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.333 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:38.334 DEBUG:boto:Port: 8000 2022-09-12T20:08:38.335 DEBUG:boto:Params: {} 2022-09-12T20:08:38.335 DEBUG:boto:Token: None 2022-09-12T20:08:38.336 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/obj95 2022-09-12T20:08:38.337 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MOnr8sKp8EqD2luMWrODFf/i5JY= 2022-09-12T20:08:38.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MOnr8sKp8EqD2luMWrODFf/i5JY=', 'Content-Length': '0'} 2022-09-12T20:08:38.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082fb7dfc8e2cf262-00631f91c6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.340 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:38.341 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:38.341 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:08:38.342 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:08:38.342 DEBUG:boto:Method: GET 2022-09-12T20:08:38.343 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:08:38.344 DEBUG:boto:Data: 2022-09-12T20:08:38.344 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.345 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:08:38.345 DEBUG:boto:Port: 8001 2022-09-12T20:08:38.346 DEBUG:boto:Params: {} 2022-09-12T20:08:38.346 DEBUG:boto:Token: None 2022-09-12T20:08:38.347 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/ 2022-09-12T20:08:38.348 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WZP+/xPDVPbUoQ//wqt11NdJchs= 2022-09-12T20:08:38.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WZP+/xPDVPbUoQ//wqt11NdJchs=', 'Content-Length': '0'} 2022-09-12T20:08:38.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adad8c2202e2a72a-00631f91c6-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.352 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:38.353 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:38.353 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:38.354 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:08:38.354 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:08:38.355 DEBUG:boto:Method: GET 2022-09-12T20:08:38.355 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:08:38.356 DEBUG:boto:Data: 2022-09-12T20:08:38.356 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.357 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:08:38.357 DEBUG:boto:Port: 8000 2022-09-12T20:08:38.358 DEBUG:boto:Params: {} 2022-09-12T20:08:38.358 DEBUG:boto:Token: None 2022-09-12T20:08:38.359 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/ 2022-09-12T20:08:38.360 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WZP+/xPDVPbUoQ//wqt11NdJchs= 2022-09-12T20:08:38.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WZP+/xPDVPbUoQ//wqt11NdJchs=', 'Content-Length': '0'} 2022-09-12T20:08:38.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073bdd6ae19bcd742-00631f91c6-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.364 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:08:38.364 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-75 zones={test-zone3, test-zone2} 2022-09-12T20:08:38.365 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:08:38.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:08:38.366 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T20:08:38.429 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:08:38.429 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:08:38.481 INFO:teuthology.orchestra.run.smithi035.stdout:tenant: 2022-09-12T20:08:38.481 INFO:teuthology.orchestra.run.smithi035.stdout:bucket name: jdpiak-75 2022-09-12T20:08:38.490 INFO:teuthology.orchestra.run.smithi035.stdout:total entries: 15 2022-09-12T20:08:38.541 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:38.541 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:38.542 DEBUG:boto:Method: HEAD 2022-09-12T20:08:38.543 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:08:38.543 DEBUG:boto:Data: 2022-09-12T20:08:38.544 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.545 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:38.545 DEBUG:boto:Port: 8000 2022-09-12T20:08:38.546 DEBUG:boto:Params: {} 2022-09-12T20:08:38.547 DEBUG:boto:Token: None 2022-09-12T20:08:38.547 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/ 2022-09-12T20:08:38.548 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:adPyFnLt/QBJ1077vc8JHjCeitE= 2022-09-12T20:08:38.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:adPyFnLt/QBJ1077vc8JHjCeitE=', 'Content-Length': '0'} 2022-09-12T20:08:38.553 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e86f6d46352428c9-00631f91c6-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.554 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:08:38.555 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:08:38.556 DEBUG:boto:Method: PUT 2022-09-12T20:08:38.557 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:08:38.558 DEBUG:boto:Data: 2022-09-12T20:08:38.559 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:08:38.559 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:38.560 DEBUG:boto:Port: 8000 2022-09-12T20:08:38.561 DEBUG:boto:Params: {} 2022-09-12T20:08:38.562 DEBUG:boto:Token: None 2022-09-12T20:08:38.563 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/obj23 2022-09-12T20:08:38.564 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NhFMkKTFxTJ1GCaItAIJZA2p0pM= 2022-09-12T20:08:38.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NhFMkKTFxTJ1GCaItAIJZA2p0pM='} 2022-09-12T20:08:38.571 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000068050cf32717ca2e-00631f91c6-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.572 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:38.573 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:38.574 DEBUG:boto:Method: HEAD 2022-09-12T20:08:38.574 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:08:38.575 DEBUG:boto:Data: 2022-09-12T20:08:38.576 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.577 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:38.577 DEBUG:boto:Port: 8000 2022-09-12T20:08:38.578 DEBUG:boto:Params: {} 2022-09-12T20:08:38.579 DEBUG:boto:Token: None 2022-09-12T20:08:38.580 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/ 2022-09-12T20:08:38.580 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:adPyFnLt/QBJ1077vc8JHjCeitE= 2022-09-12T20:08:38.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:adPyFnLt/QBJ1077vc8JHjCeitE=', 'Content-Length': '0'} 2022-09-12T20:08:38.584 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000739829ccc761bb12-00631f91c6-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.586 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:08:38.586 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:08:38.587 DEBUG:boto:Method: PUT 2022-09-12T20:08:38.587 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:08:38.588 DEBUG:boto:Data: 2022-09-12T20:08:38.588 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:08:38.589 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:38.589 DEBUG:boto:Port: 8000 2022-09-12T20:08:38.590 DEBUG:boto:Params: {} 2022-09-12T20:08:38.590 DEBUG:boto:Token: None 2022-09-12T20:08:38.591 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/obj46 2022-09-12T20:08:38.592 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q9/+9DKtOznjRM4CDrsIG3mlqMw= 2022-09-12T20:08:38.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q9/+9DKtOznjRM4CDrsIG3mlqMw='} 2022-09-12T20:08:38.598 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ea09fcbc433b048a-00631f91c6-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.599 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:38.600 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:38.601 DEBUG:boto:Method: HEAD 2022-09-12T20:08:38.602 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:08:38.603 DEBUG:boto:Data: 2022-09-12T20:08:38.604 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.605 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:38.606 DEBUG:boto:Port: 8000 2022-09-12T20:08:38.607 DEBUG:boto:Params: {} 2022-09-12T20:08:38.608 DEBUG:boto:Token: None 2022-09-12T20:08:38.609 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/ 2022-09-12T20:08:38.610 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:adPyFnLt/QBJ1077vc8JHjCeitE= 2022-09-12T20:08:38.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:adPyFnLt/QBJ1077vc8JHjCeitE=', 'Content-Length': '0'} 2022-09-12T20:08:38.614 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fca3f6f86eca629c-00631f91c6-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.616 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:08:38.616 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:08:38.617 DEBUG:boto:Method: PUT 2022-09-12T20:08:38.617 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:08:38.618 DEBUG:boto:Data: 2022-09-12T20:08:38.618 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:08:38.619 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:38.619 DEBUG:boto:Port: 8000 2022-09-12T20:08:38.620 DEBUG:boto:Params: {} 2022-09-12T20:08:38.620 DEBUG:boto:Token: None 2022-09-12T20:08:38.621 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/obj69 2022-09-12T20:08:38.622 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UrGME9A0Mxj+vfGBXB1OnbdPVK0= 2022-09-12T20:08:38.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UrGME9A0Mxj+vfGBXB1OnbdPVK0='} 2022-09-12T20:08:38.666 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000523d734437c0ba41-00631f91c6-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.667 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:38.667 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:38.668 DEBUG:boto:Method: HEAD 2022-09-12T20:08:38.668 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:08:38.669 DEBUG:boto:Data: 2022-09-12T20:08:38.669 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.670 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:38.670 DEBUG:boto:Port: 8000 2022-09-12T20:08:38.671 DEBUG:boto:Params: {} 2022-09-12T20:08:38.671 DEBUG:boto:Token: None 2022-09-12T20:08:38.672 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/ 2022-09-12T20:08:38.673 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:adPyFnLt/QBJ1077vc8JHjCeitE= 2022-09-12T20:08:38.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:adPyFnLt/QBJ1077vc8JHjCeitE=', 'Content-Length': '0'} 2022-09-12T20:08:38.676 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007359d364cf2588c2-00631f91c6-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.677 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:08:38.677 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:08:38.678 DEBUG:boto:Method: PUT 2022-09-12T20:08:38.678 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:08:38.679 DEBUG:boto:Data: 2022-09-12T20:08:38.679 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:08:38.680 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:38.680 DEBUG:boto:Port: 8000 2022-09-12T20:08:38.681 DEBUG:boto:Params: {} 2022-09-12T20:08:38.681 DEBUG:boto:Token: None 2022-09-12T20:08:38.682 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/obj92 2022-09-12T20:08:38.683 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:faMkyZlM6ofHlPjRo2NAcTdNxDM= 2022-09-12T20:08:38.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:faMkyZlM6ofHlPjRo2NAcTdNxDM='} 2022-09-12T20:08:38.694 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e13bfc56cd7aedab-00631f91c6-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.695 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:08:38.695 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:08:38.696 DEBUG:boto:Method: HEAD 2022-09-12T20:08:38.696 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:08:38.697 DEBUG:boto:Data: 2022-09-12T20:08:38.697 DEBUG:boto:Headers: {} 2022-09-12T20:08:38.698 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:38.698 DEBUG:boto:Port: 8000 2022-09-12T20:08:38.699 DEBUG:boto:Params: {} 2022-09-12T20:08:38.699 DEBUG:boto:Token: None 2022-09-12T20:08:38.700 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/ 2022-09-12T20:08:38.700 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:adPyFnLt/QBJ1077vc8JHjCeitE= 2022-09-12T20:08:38.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:adPyFnLt/QBJ1077vc8JHjCeitE=', 'Content-Length': '0'} 2022-09-12T20:08:38.704 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000058b5e5bd3925c2de-00631f91c6-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.705 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:08:38.705 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:08:38.706 DEBUG:boto:Method: PUT 2022-09-12T20:08:38.706 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:08:38.707 DEBUG:boto:Data: 2022-09-12T20:08:38.707 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:08:38.708 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:08:38.708 DEBUG:boto:Port: 8000 2022-09-12T20:08:38.709 DEBUG:boto:Params: {} 2022-09-12T20:08:38.709 DEBUG:boto:Token: None 2022-09-12T20:08:38.710 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:08:38 GMT /jdpiak-75/obj115 2022-09-12T20:08:38.710 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PAAhoSLCOxRskFHqi3A97KJG0HU= 2022-09-12T20:08:38.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:08:38 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PAAhoSLCOxRskFHqi3A97KJG0HU='} 2022-09-12T20:08:38.754 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000650d6e8695727646-00631f91c6-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:08:38.755 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-75 2022-09-12T20:08:38.755 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:08:38.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:08:38.757 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T20:08:38.822 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:08:38.823 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:08:38.863 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:08:38.861+0000 7f0deaa51600 1 waiting to reach latest gen 3, on 2.. 2022-09-12T20:08:43.866 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:08:43.863+0000 7f0deaa51600 1 waiting for incremental sync to catch up: 2022-09-12T20:08:43.866 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:08:43.866 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000001.64.5, , , 00000000002.49.5, , 00000000002.60.5, , , , , , 00000000001.107.5, , 00000000001.84.5, , , , ] 2022-09-12T20:08:48.868 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:08:48.866+0000 7f0deaa51600 1 waiting for incremental sync to catch up: 2022-09-12T20:08:48.868 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:08:48.868 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000001.64.5, , , 00000000002.49.5, , 00000000002.60.5, , , , , , 00000000001.107.5, , 00000000001.84.5, , , , ] 2022-09-12T20:08:53.870 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:08:53.869+0000 7f0deaa51600 1 waiting for incremental sync to catch up: 2022-09-12T20:08:53.870 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:08:53.871 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000001.64.5, , , 00000000002.49.5, , 00000000002.60.5, , , , , , 00000000001.107.5, , 00000000001.84.5, , , , ] 2022-09-12T20:08:58.872 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:08:58.870+0000 7f0deaa51600 1 waiting for incremental sync to catch up: 2022-09-12T20:08:58.872 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:08:58.873 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000001.64.5, , , 00000000002.49.5, , 00000000002.60.5, , , , , , 00000000001.107.5, , 00000000001.84.5, , , , ] 2022-09-12T20:09:03.874 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:09:03.872+0000 7f0deaa51600 1 bucket sync caught up with source: 2022-09-12T20:09:03.875 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000001.64.5, , , 00000000002.49.5, , 00000000002.60.5, , , , , , 00000000001.107.5, , 00000000001.84.5, , , , ] 2022-09-12T20:09:03.875 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000001.64.5, , , 00000000002.49.5, , 00000000002.60.5, , , , , , 00000000001.107.5, , 00000000001.84.5, , , , ] 2022-09-12T20:09:03.875 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:09:03.872+0000 7f0deaa51600 0 bucket checkpoint complete 2022-09-12T20:09:03.893 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-75 2022-09-12T20:09:03.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:09:03.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:09:03.895 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:09:03.962 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:09:03.962 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:09:04.013 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:09:04.013+0000 7fdac8419600 1 waiting for incremental sync to catch up: 2022-09-12T20:09:04.014 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:09:04.014 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000001.64.5, , , 00000000002.49.5, , 00000000002.60.5, , , , , , 00000000001.107.5, , 00000000001.84.5, , , , ] 2022-09-12T20:09:09.016 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:09:09.014+0000 7fdac8419600 1 waiting for incremental sync to catch up: 2022-09-12T20:09:09.016 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:09:09.017 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000001.64.5, , , 00000000002.49.5, , 00000000002.60.5, , , , , , 00000000001.107.5, , 00000000001.84.5, , , , ] 2022-09-12T20:09:14.018 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:09:14.016+0000 7fdac8419600 1 waiting for incremental sync to catch up: 2022-09-12T20:09:14.018 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:09:14.019 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000001.64.5, , , 00000000002.49.5, , 00000000002.60.5, , , , , , 00000000001.107.5, , 00000000001.84.5, , , , ] 2022-09-12T20:09:19.020 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:09:19.018+0000 7fdac8419600 1 waiting for incremental sync to catch up: 2022-09-12T20:09:19.021 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:09:19.021 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000001.64.5, , , 00000000002.49.5, , 00000000002.60.5, , , , , , 00000000001.107.5, , 00000000001.84.5, , , , ] 2022-09-12T20:09:24.022 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:09:24.020+0000 7fdac8419600 1 waiting for incremental sync to catch up: 2022-09-12T20:09:24.023 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:09:24.023 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000001.64.5, , , 00000000002.49.5, , 00000000002.60.5, , , , , , 00000000001.107.5, , 00000000001.84.5, , , , ] 2022-09-12T20:09:29.025 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:09:29.023+0000 7fdac8419600 1 waiting for incremental sync to catch up: 2022-09-12T20:09:29.025 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:09:29.025 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000001.64.5, , , 00000000002.49.5, , 00000000002.60.5, , , , , , 00000000001.107.5, , 00000000001.84.5, , , , ] 2022-09-12T20:09:34.026 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:09:34.025+0000 7fdac8419600 1 waiting for incremental sync to catch up: 2022-09-12T20:09:34.027 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:09:34.027 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000001.64.5, , , 00000000002.49.5, , 00000000002.60.5, , , , , , 00000000001.107.5, , 00000000001.84.5, , , , ] 2022-09-12T20:09:39.029 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:09:39.027+0000 7fdac8419600 1 waiting for incremental sync to catch up: 2022-09-12T20:09:39.029 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:09:39.029 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000001.64.5, , , 00000000002.49.5, , 00000000002.60.5, , , , , , 00000000001.107.5, , 00000000001.84.5, , , , ] 2022-09-12T20:09:44.030 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:09:44.029+0000 7fdac8419600 1 waiting for incremental sync to catch up: 2022-09-12T20:09:44.031 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:09:44.031 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000001.64.5, , , 00000000002.49.5, , 00000000002.60.5, , , , , , 00000000001.107.5, , 00000000001.84.5, , , , ] 2022-09-12T20:09:49.033 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:09:49.032+0000 7fdac8419600 1 bucket sync caught up with source: 2022-09-12T20:09:49.033 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , 00000000001.64.5, , , 00000000002.49.5, , 00000000002.60.5, , , , , , 00000000001.107.5, , 00000000001.84.5, , , , ] 2022-09-12T20:09:49.034 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , 00000000001.64.5, , , 00000000002.49.5, , 00000000002.60.5, , , , , , 00000000001.107.5, , 00000000001.84.5, , , , ] 2022-09-12T20:09:49.034 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:09:49.032+0000 7fdac8419600 0 bucket checkpoint complete 2022-09-12T20:09:49.052 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-75 2022-09-12T20:09:49.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:09:49.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:09:49.054 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T20:09:49.117 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:09:49.118 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:09:49.158 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:09:49.156+0000 7f68d0530600 1 waiting for incremental sync to catch up: 2022-09-12T20:09:49.159 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000002.49.5, , 00000000002.60.5, , , , , , 00000000001.107.5, , 00000000001.84.5, , , , ] 2022-09-12T20:09:49.159 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000001.64.5, , , 00000000002.49.5, , 00000000002.60.5, , , , , , 00000000001.107.5, , 00000000001.84.5, , , , ] 2022-09-12T20:09:54.161 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:09:54.159+0000 7f68d0530600 1 bucket sync caught up with source: 2022-09-12T20:09:54.162 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , 00000000001.64.5, , , 00000000002.49.5, , 00000000002.60.5, , , , , , 00000000001.107.5, , 00000000001.84.5, , , , ] 2022-09-12T20:09:54.162 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , 00000000001.64.5, , , 00000000002.49.5, , 00000000002.60.5, , , , , , 00000000001.107.5, , 00000000001.84.5, , , , ] 2022-09-12T20:09:54.163 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:09:54.159+0000 7f68d0530600 0 bucket checkpoint complete 2022-09-12T20:09:54.178 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-75 2022-09-12T20:09:54.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:09:54.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:09:54.181 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:09:54.247 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:09:54.247 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:09:54.291 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:09:54.290+0000 7f6399bd5600 1 bucket sync caught up with source: 2022-09-12T20:09:54.291 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.639.5, 00000000002.837.5, 00000000001.634.5, 00000000002.346.5, 00000000002.927.5, , , 00000000002.550.5, 00000000005.925.5, 00000000004.580.5] 2022-09-12T20:09:54.291 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.639.5, 00000000002.837.5, 00000000001.634.5, 00000000002.346.5, 00000000002.927.5, , , 00000000002.550.5, 00000000005.925.5, 00000000004.580.5] 2022-09-12T20:09:54.292 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:09:54.290+0000 7f6399bd5600 0 bucket checkpoint complete 2022-09-12T20:09:54.306 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-75 2022-09-12T20:09:54.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:09:54.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:09:54.307 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:09:54.366 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:09:54.367 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:09:54.404 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:09:54.403+0000 7fa004b5b600 1 bucket sync caught up with source: 2022-09-12T20:09:54.404 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.639.5, 00000000002.837.5, 00000000001.634.5, 00000000002.346.5, 00000000002.927.5, , , 00000000002.550.5, 00000000005.925.5, 00000000004.580.5] 2022-09-12T20:09:54.404 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.639.5, 00000000002.837.5, 00000000001.634.5, 00000000002.346.5, 00000000002.927.5, , , 00000000002.550.5, 00000000005.925.5, 00000000004.580.5] 2022-09-12T20:09:54.405 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:09:54.403+0000 7fa004b5b600 0 bucket checkpoint complete 2022-09-12T20:09:54.418 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-75 2022-09-12T20:09:54.419 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-75 2022-09-12T20:09:54.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:09:54.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:09:54.420 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:09:54.478 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:09:54.478 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:09:54.528 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:09:54.527+0000 7f54862c8600 1 bucket sync caught up with source: 2022-09-12T20:09:54.528 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.296.5, 00000000002.337.5, 00000000001.279.5, 00000000002.344.5, 00000000002.543.5, , , 00000000002.320.5, 00000000005.541.5, 00000000004.494.5] 2022-09-12T20:09:54.529 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.296.5, 00000000002.337.5, 00000000001.279.5, 00000000002.344.5, 00000000002.543.5, , , 00000000002.320.5, 00000000005.541.5, 00000000004.494.5] 2022-09-12T20:09:54.529 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:09:54.527+0000 7f54862c8600 0 bucket checkpoint complete 2022-09-12T20:09:54.543 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-75 2022-09-12T20:09:54.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:09:54.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:09:54.546 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T20:09:54.612 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:09:54.613 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:09:54.654 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:09:54.652+0000 7fb913a76600 1 bucket sync caught up with source: 2022-09-12T20:09:54.655 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.296.5, 00000000002.337.5, 00000000001.279.5, 00000000002.344.5, 00000000002.543.5, , , 00000000002.320.5, 00000000005.541.5, 00000000004.494.5] 2022-09-12T20:09:54.655 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000002.296.5, 00000000002.337.5, 00000000001.279.5, 00000000002.344.5, 00000000002.543.5, , , 00000000002.320.5, 00000000005.541.5, 00000000004.494.5] 2022-09-12T20:09:54.656 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:09:54.652+0000 7fb913a76600 0 bucket checkpoint complete 2022-09-12T20:09:54.677 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-75 2022-09-12T20:09:54.678 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-75 zones={test-zone2, test-zone1} 2022-09-12T20:09:54.679 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:54.680 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:54.681 DEBUG:boto:Method: HEAD 2022-09-12T20:09:54.681 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:09:54.682 DEBUG:boto:Data: 2022-09-12T20:09:54.683 DEBUG:boto:Headers: {} 2022-09-12T20:09:54.684 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:54.685 DEBUG:boto:Port: 8000 2022-09-12T20:09:54.686 DEBUG:boto:Params: {} 2022-09-12T20:09:54.687 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T20:09:54.688 DEBUG:boto:Token: None 2022-09-12T20:09:54.689 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:54 GMT /jdpiak-75/ 2022-09-12T20:09:54.690 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uyHCBfyY67Jtii/QiaPQoXsBirI= 2022-09-12T20:09:54.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uyHCBfyY67Jtii/QiaPQoXsBirI=', 'Content-Length': '0'} 2022-09-12T20:09:54.696 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000abd46d0f5f39326c-00631f9212-189d-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:09:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:54.697 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:54.698 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:54.699 DEBUG:boto:Method: HEAD 2022-09-12T20:09:54.699 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:09:54.700 DEBUG:boto:Data: 2022-09-12T20:09:54.701 DEBUG:boto:Headers: {} 2022-09-12T20:09:54.702 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:54.703 DEBUG:boto:Port: 8000 2022-09-12T20:09:54.704 DEBUG:boto:Params: {} 2022-09-12T20:09:54.705 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T20:09:54.706 DEBUG:boto:Token: None 2022-09-12T20:09:54.707 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:54 GMT /jdpiak-75/ 2022-09-12T20:09:54.708 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uyHCBfyY67Jtii/QiaPQoXsBirI= 2022-09-12T20:09:54.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uyHCBfyY67Jtii/QiaPQoXsBirI=', 'Content-Length': '0'} 2022-09-12T20:09:54.713 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e33b60b511aa4e89-00631f9212-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:09:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:54.714 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T20:09:54.714 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:54.715 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:54.716 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:09:54.716 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:09:54.717 DEBUG:boto:Method: GET 2022-09-12T20:09:54.717 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:09:54.718 DEBUG:boto:Data: 2022-09-12T20:09:54.718 DEBUG:boto:Headers: {} 2022-09-12T20:09:54.719 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:54.719 DEBUG:boto:Port: 8000 2022-09-12T20:09:54.720 DEBUG:boto:Params: {} 2022-09-12T20:09:54.720 DEBUG:boto:Token: None 2022-09-12T20:09:54.721 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:54 GMT /jdpiak-75/?versions 2022-09-12T20:09:54.721 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:twOdioSJQLyjFMcEuIB0gMoxfIw= 2022-09-12T20:09:54.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:twOdioSJQLyjFMcEuIB0gMoxfIw=', 'Content-Length': '0'} 2022-09-12T20:09:54.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000141047fc218b1912-00631f9212-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:54.725 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:54.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:09:54.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-12T20:09:54.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:09:54.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:09:54.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:09:54.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-12T20:09:54.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:09:54.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:09:54.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:09:54.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:09:54.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-12T20:09:54.733 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:09:54.733 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:09:54.734 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:09:54.734 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:09:54.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-12T20:09:54.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:09:54.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:09:54.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-12T20:09:54.737 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:09:54.737 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T20:09:54.738 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:54.738 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:54.739 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:09:54.739 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:09:54.740 DEBUG:boto:Method: GET 2022-09-12T20:09:54.740 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:09:54.741 DEBUG:boto:Data: 2022-09-12T20:09:54.741 DEBUG:boto:Headers: {} 2022-09-12T20:09:54.742 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:54.742 DEBUG:boto:Port: 8000 2022-09-12T20:09:54.743 DEBUG:boto:Params: {} 2022-09-12T20:09:54.744 DEBUG:boto:Token: None 2022-09-12T20:09:54.744 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:54 GMT /jdpiak-75/?versions 2022-09-12T20:09:54.745 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:twOdioSJQLyjFMcEuIB0gMoxfIw= 2022-09-12T20:09:54.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:twOdioSJQLyjFMcEuIB0gMoxfIw=', 'Content-Length': '0'} 2022-09-12T20:09:54.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024a66721bee09d12-00631f9212-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:54.749 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:54.751 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:09:54.751 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-12T20:09:54.752 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:09:54.752 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:09:54.753 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:09:54.753 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-12T20:09:54.754 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:09:54.754 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:09:54.755 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:09:54.755 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:09:54.756 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-12T20:09:54.756 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:09:54.756 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:09:54.757 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:09:54.757 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:09:54.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-12T20:09:54.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:09:54.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:09:54.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-12T20:09:54.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:09:54.760 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:54.761 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:54.761 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:09:54.762 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:09:54.762 DEBUG:boto:Method: GET 2022-09-12T20:09:54.763 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:09:54.763 DEBUG:boto:Data: 2022-09-12T20:09:54.764 DEBUG:boto:Headers: {} 2022-09-12T20:09:54.764 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:54.765 DEBUG:boto:Port: 8000 2022-09-12T20:09:54.765 DEBUG:boto:Params: {} 2022-09-12T20:09:54.766 DEBUG:boto:Token: None 2022-09-12T20:09:54.766 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:54 GMT /jdpiak-75/?versions 2022-09-12T20:09:54.767 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:twOdioSJQLyjFMcEuIB0gMoxfIw= 2022-09-12T20:09:54.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:twOdioSJQLyjFMcEuIB0gMoxfIw=', 'Content-Length': '0'} 2022-09-12T20:09:54.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067003f5ae67f9333-00631f9212-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:54.771 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:54.773 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:54.773 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:54.774 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:09:54.775 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:09:54.775 DEBUG:boto:Method: GET 2022-09-12T20:09:54.776 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:09:54.776 DEBUG:boto:Data: 2022-09-12T20:09:54.777 DEBUG:boto:Headers: {} 2022-09-12T20:09:54.777 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:54.778 DEBUG:boto:Port: 8000 2022-09-12T20:09:54.778 DEBUG:boto:Params: {} 2022-09-12T20:09:54.779 DEBUG:boto:Token: None 2022-09-12T20:09:54.779 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:54 GMT /jdpiak-75/?versions 2022-09-12T20:09:54.780 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:twOdioSJQLyjFMcEuIB0gMoxfIw= 2022-09-12T20:09:54.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:twOdioSJQLyjFMcEuIB0gMoxfIw=', 'Content-Length': '0'} 2022-09-12T20:09:54.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000791acf68b9e473d1-00631f9212-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:54.785 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:54.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:09:54.787 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:09:54.788 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:09:54.788 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:09:54.789 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:09:54.789 DEBUG:boto:Method: GET 2022-09-12T20:09:54.790 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:09:54.790 DEBUG:boto:Data: 2022-09-12T20:09:54.791 DEBUG:boto:Headers: {} 2022-09-12T20:09:54.792 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:54.792 DEBUG:boto:Port: 8000 2022-09-12T20:09:54.793 DEBUG:boto:Params: {} 2022-09-12T20:09:54.794 DEBUG:boto:Token: None 2022-09-12T20:09:54.794 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:54 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:09:54.795 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Jm2t4V9gfYwviZtKfkTqMHe89ME= 2022-09-12T20:09:54.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Jm2t4V9gfYwviZtKfkTqMHe89ME=', 'Content-Length': '0'} 2022-09-12T20:09:54.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d357532846cf2c7a-00631f9212-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:54.798 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:09:54.799 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:09:54.799 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:09:54.800 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:09:54.800 DEBUG:boto:Method: GET 2022-09-12T20:09:54.801 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:09:54.801 DEBUG:boto:Data: 2022-09-12T20:09:54.802 DEBUG:boto:Headers: {} 2022-09-12T20:09:54.802 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:54.803 DEBUG:boto:Port: 8000 2022-09-12T20:09:54.803 DEBUG:boto:Params: {} 2022-09-12T20:09:54.804 DEBUG:boto:Token: None 2022-09-12T20:09:54.805 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:54 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:09:54.805 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Jm2t4V9gfYwviZtKfkTqMHe89ME= 2022-09-12T20:09:54.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Jm2t4V9gfYwviZtKfkTqMHe89ME=', 'Content-Length': '0'} 2022-09-12T20:09:54.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000425c28abd774cebc-00631f9212-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:54.809 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:09:54.809 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:09:54.810 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:09:54.810 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:09:54.811 DEBUG:boto:Method: HEAD 2022-09-12T20:09:54.811 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:09:54.812 DEBUG:boto:Data: 2022-09-12T20:09:54.812 DEBUG:boto:Headers: {} 2022-09-12T20:09:54.813 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:54.813 DEBUG:boto:Port: 8000 2022-09-12T20:09:54.814 DEBUG:boto:Params: {} 2022-09-12T20:09:54.814 DEBUG:boto:Token: None 2022-09-12T20:09:54.815 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:54 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:09:54.815 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5G1eBWqcv8RDq3lWT1MG1bvEsgk= 2022-09-12T20:09:54.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5G1eBWqcv8RDq3lWT1MG1bvEsgk=', 'Content-Length': '0'} 2022-09-12T20:09:54.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1220390bbab4fdc-00631f9212-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:54.818 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:09:54.819 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:09:54.819 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:09:54.820 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:09:54.820 DEBUG:boto:Method: HEAD 2022-09-12T20:09:54.821 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:09:54.821 DEBUG:boto:Data: 2022-09-12T20:09:54.822 DEBUG:boto:Headers: {} 2022-09-12T20:09:54.822 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:54.823 DEBUG:boto:Port: 8000 2022-09-12T20:09:54.823 DEBUG:boto:Params: {} 2022-09-12T20:09:54.824 DEBUG:boto:Token: None 2022-09-12T20:09:54.824 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:54 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:09:54.825 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5G1eBWqcv8RDq3lWT1MG1bvEsgk= 2022-09-12T20:09:54.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5G1eBWqcv8RDq3lWT1MG1bvEsgk=', 'Content-Length': '0'} 2022-09-12T20:09:54.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a4bdcbecbe5a3f6-00631f9212-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:54.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:09:54.829 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:09:54.829 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:09:54.830 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:09:54.830 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:09:54.831 DEBUG:boto:Method: GET 2022-09-12T20:09:54.831 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:09:54.832 DEBUG:boto:Data: 2022-09-12T20:09:54.832 DEBUG:boto:Headers: {} 2022-09-12T20:09:54.833 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:54.833 DEBUG:boto:Port: 8000 2022-09-12T20:09:54.834 DEBUG:boto:Params: {} 2022-09-12T20:09:54.834 DEBUG:boto:Token: None 2022-09-12T20:09:54.835 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:54 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:09:54.835 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Jm2t4V9gfYwviZtKfkTqMHe89ME= 2022-09-12T20:09:54.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Jm2t4V9gfYwviZtKfkTqMHe89ME=', 'Content-Length': '0'} 2022-09-12T20:09:54.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000051aff274a5efd28-00631f9212-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:54.839 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:09:54.839 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:09:54.840 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:09:54.840 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:09:54.841 DEBUG:boto:Method: GET 2022-09-12T20:09:54.841 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:09:54.842 DEBUG:boto:Data: 2022-09-12T20:09:54.842 DEBUG:boto:Headers: {} 2022-09-12T20:09:54.843 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:54.843 DEBUG:boto:Port: 8000 2022-09-12T20:09:54.844 DEBUG:boto:Params: {} 2022-09-12T20:09:54.845 DEBUG:boto:Token: None 2022-09-12T20:09:54.845 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:54 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:09:54.846 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Jm2t4V9gfYwviZtKfkTqMHe89ME= 2022-09-12T20:09:54.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Jm2t4V9gfYwviZtKfkTqMHe89ME=', 'Content-Length': '0'} 2022-09-12T20:09:54.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3f75008f37472fa-00631f9212-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:54.849 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:54.850 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:54.850 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:09:54.851 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:09:54.851 DEBUG:boto:Method: GET 2022-09-12T20:09:54.852 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:09:54.852 DEBUG:boto:Data: 2022-09-12T20:09:54.853 DEBUG:boto:Headers: {} 2022-09-12T20:09:54.853 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:54.854 DEBUG:boto:Port: 8000 2022-09-12T20:09:54.854 DEBUG:boto:Params: {} 2022-09-12T20:09:54.855 DEBUG:boto:Token: None 2022-09-12T20:09:54.855 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:54 GMT /jdpiak-75/ 2022-09-12T20:09:54.856 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8ZV3Er/AjReHNDtwen1cmIxdmgI= 2022-09-12T20:09:54.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8ZV3Er/AjReHNDtwen1cmIxdmgI=', 'Content-Length': '0'} 2022-09-12T20:09:54.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da146b587cd8c143-00631f9212-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:54.859 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:54.860 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:54.861 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:54.861 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:09:54.862 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:09:54.862 DEBUG:boto:Method: GET 2022-09-12T20:09:54.863 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:09:54.863 DEBUG:boto:Data: 2022-09-12T20:09:54.864 DEBUG:boto:Headers: {} 2022-09-12T20:09:54.864 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:54.865 DEBUG:boto:Port: 8000 2022-09-12T20:09:54.865 DEBUG:boto:Params: {} 2022-09-12T20:09:54.866 DEBUG:boto:Token: None 2022-09-12T20:09:54.867 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:54 GMT /jdpiak-75/ 2022-09-12T20:09:54.867 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8ZV3Er/AjReHNDtwen1cmIxdmgI= 2022-09-12T20:09:54.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8ZV3Er/AjReHNDtwen1cmIxdmgI=', 'Content-Length': '0'} 2022-09-12T20:09:54.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000089f7d939b5d11a1-00631f9212-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:54.872 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:54.872 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:09:54.873 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:09:54.873 DEBUG:boto:Method: HEAD 2022-09-12T20:09:54.874 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:09:54.875 DEBUG:boto:Data: 2022-09-12T20:09:54.875 DEBUG:boto:Headers: {} 2022-09-12T20:09:54.876 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:54.876 DEBUG:boto:Port: 8000 2022-09-12T20:09:54.877 DEBUG:boto:Params: {} 2022-09-12T20:09:54.877 DEBUG:boto:Token: None 2022-09-12T20:09:54.878 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:54 GMT /jdpiak-75/obj11 2022-09-12T20:09:54.878 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G5NonGfne7cFBnrH9L/RvDpEttg= 2022-09-12T20:09:54.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G5NonGfne7cFBnrH9L/RvDpEttg=', 'Content-Length': '0'} 2022-09-12T20:09:54.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005868da88b624a7e2-00631f9212-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:54.881 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:09:54.882 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:09:54.882 DEBUG:boto:Method: HEAD 2022-09-12T20:09:54.883 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:09:54.883 DEBUG:boto:Data: 2022-09-12T20:09:54.884 DEBUG:boto:Headers: {} 2022-09-12T20:09:54.884 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:54.885 DEBUG:boto:Port: 8000 2022-09-12T20:09:54.885 DEBUG:boto:Params: {} 2022-09-12T20:09:54.886 DEBUG:boto:Token: None 2022-09-12T20:09:54.886 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:54 GMT /jdpiak-75/obj11 2022-09-12T20:09:54.887 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G5NonGfne7cFBnrH9L/RvDpEttg= 2022-09-12T20:09:54.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G5NonGfne7cFBnrH9L/RvDpEttg=', 'Content-Length': '0'} 2022-09-12T20:09:54.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097a8005d4ec25aa1-00631f9212-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:54.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:09:54.891 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:09:54.891 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:09:54.892 DEBUG:boto:Method: GET 2022-09-12T20:09:54.892 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:09:54.893 DEBUG:boto:Data: 2022-09-12T20:09:54.893 DEBUG:boto:Headers: {} 2022-09-12T20:09:54.894 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:54.894 DEBUG:boto:Port: 8000 2022-09-12T20:09:54.895 DEBUG:boto:Params: {} 2022-09-12T20:09:54.895 DEBUG:boto:Token: None 2022-09-12T20:09:54.896 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:54 GMT /jdpiak-75/obj11 2022-09-12T20:09:54.897 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HzjPifu4xWIn+iueZtlKRzo8IRw= 2022-09-12T20:09:54.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HzjPifu4xWIn+iueZtlKRzo8IRw=', 'Content-Length': '0'} 2022-09-12T20:09:54.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016f47ae13d8d42e1-00631f9212-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:54.942 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:09:54.943 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:09:54.943 DEBUG:boto:Method: GET 2022-09-12T20:09:54.944 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:09:54.944 DEBUG:boto:Data: 2022-09-12T20:09:54.945 DEBUG:boto:Headers: {} 2022-09-12T20:09:54.945 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:54.946 DEBUG:boto:Port: 8000 2022-09-12T20:09:54.946 DEBUG:boto:Params: {} 2022-09-12T20:09:54.947 DEBUG:boto:Token: None 2022-09-12T20:09:54.948 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:54 GMT /jdpiak-75/obj11 2022-09-12T20:09:54.948 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HzjPifu4xWIn+iueZtlKRzo8IRw= 2022-09-12T20:09:54.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HzjPifu4xWIn+iueZtlKRzo8IRw=', 'Content-Length': '0'} 2022-09-12T20:09:54.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093200b53ed11a4bc-00631f9212-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:54.952 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:54.952 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:54.953 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:09:54.953 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:09:54.954 DEBUG:boto:Method: GET 2022-09-12T20:09:54.954 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:09:54.955 DEBUG:boto:Data: 2022-09-12T20:09:54.955 DEBUG:boto:Headers: {} 2022-09-12T20:09:54.956 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:54.956 DEBUG:boto:Port: 8000 2022-09-12T20:09:54.957 DEBUG:boto:Params: {} 2022-09-12T20:09:54.957 DEBUG:boto:Token: None 2022-09-12T20:09:54.958 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:54 GMT /jdpiak-75/ 2022-09-12T20:09:54.959 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8ZV3Er/AjReHNDtwen1cmIxdmgI= 2022-09-12T20:09:54.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8ZV3Er/AjReHNDtwen1cmIxdmgI=', 'Content-Length': '0'} 2022-09-12T20:09:54.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008943f6c0f4ec6f21-00631f9212-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:54.963 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:54.964 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:54.964 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:54.965 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:09:54.966 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:09:54.966 DEBUG:boto:Method: GET 2022-09-12T20:09:54.967 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:09:54.968 DEBUG:boto:Data: 2022-09-12T20:09:54.968 DEBUG:boto:Headers: {} 2022-09-12T20:09:54.969 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:54.970 DEBUG:boto:Port: 8000 2022-09-12T20:09:54.970 DEBUG:boto:Params: {} 2022-09-12T20:09:54.971 DEBUG:boto:Token: None 2022-09-12T20:09:54.972 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:54 GMT /jdpiak-75/ 2022-09-12T20:09:54.972 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8ZV3Er/AjReHNDtwen1cmIxdmgI= 2022-09-12T20:09:54.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8ZV3Er/AjReHNDtwen1cmIxdmgI=', 'Content-Length': '0'} 2022-09-12T20:09:54.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c022677724f05e4-00631f9212-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:54.977 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:54.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:09:54.979 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:09:54.980 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:09:54.980 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:09:54.981 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:09:54.982 DEBUG:boto:Method: GET 2022-09-12T20:09:54.982 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:09:54.983 DEBUG:boto:Data: 2022-09-12T20:09:54.983 DEBUG:boto:Headers: {} 2022-09-12T20:09:54.984 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:54.984 DEBUG:boto:Port: 8000 2022-09-12T20:09:54.985 DEBUG:boto:Params: {} 2022-09-12T20:09:54.986 DEBUG:boto:Token: None 2022-09-12T20:09:54.986 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:54 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:09:54.987 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PoE63pgBvyjqATf6UTjBYTnem3k= 2022-09-12T20:09:54.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PoE63pgBvyjqATf6UTjBYTnem3k=', 'Content-Length': '0'} 2022-09-12T20:09:54.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c04f652a37c0b62-00631f9212-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:54.991 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:09:54.991 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:09:54.992 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:09:54.992 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:09:54.993 DEBUG:boto:Method: GET 2022-09-12T20:09:54.993 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:09:54.994 DEBUG:boto:Data: 2022-09-12T20:09:54.994 DEBUG:boto:Headers: {} 2022-09-12T20:09:54.995 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:54.995 DEBUG:boto:Port: 8000 2022-09-12T20:09:54.996 DEBUG:boto:Params: {} 2022-09-12T20:09:54.997 DEBUG:boto:Token: None 2022-09-12T20:09:54.997 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:54 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:09:54.998 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PoE63pgBvyjqATf6UTjBYTnem3k= 2022-09-12T20:09:54.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:54 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PoE63pgBvyjqATf6UTjBYTnem3k=', 'Content-Length': '0'} 2022-09-12T20:09:55.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3ffd225507e631a-00631f9212-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.001 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:09:55.002 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:09:55.002 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:09:55.003 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:09:55.003 DEBUG:boto:Method: HEAD 2022-09-12T20:09:55.004 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:09:55.005 DEBUG:boto:Data: 2022-09-12T20:09:55.005 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.006 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.006 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.007 DEBUG:boto:Params: {} 2022-09-12T20:09:55.007 DEBUG:boto:Token: None 2022-09-12T20:09:55.008 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:09:55.008 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VCTGlbinLcKaCKUw7ONknnYCbNc= 2022-09-12T20:09:55.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VCTGlbinLcKaCKUw7ONknnYCbNc=', 'Content-Length': '0'} 2022-09-12T20:09:55.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b4b0679e8020390-00631f9213-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.012 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:09:55.013 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:09:55.013 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:09:55.014 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:09:55.014 DEBUG:boto:Method: HEAD 2022-09-12T20:09:55.015 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:09:55.016 DEBUG:boto:Data: 2022-09-12T20:09:55.016 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.017 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.017 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.018 DEBUG:boto:Params: {} 2022-09-12T20:09:55.019 DEBUG:boto:Token: None 2022-09-12T20:09:55.019 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:09:55.020 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VCTGlbinLcKaCKUw7ONknnYCbNc= 2022-09-12T20:09:55.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VCTGlbinLcKaCKUw7ONknnYCbNc=', 'Content-Length': '0'} 2022-09-12T20:09:55.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ee7b78141e41a59-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:09:55.024 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:09:55.025 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:09:55.025 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:09:55.026 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:09:55.026 DEBUG:boto:Method: GET 2022-09-12T20:09:55.027 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:09:55.027 DEBUG:boto:Data: 2022-09-12T20:09:55.028 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.028 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.029 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.029 DEBUG:boto:Params: {} 2022-09-12T20:09:55.030 DEBUG:boto:Token: None 2022-09-12T20:09:55.031 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:09:55.031 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1uVWL/IRWfKlr1B9cFEsxloKEfo= 2022-09-12T20:09:55.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1uVWL/IRWfKlr1B9cFEsxloKEfo=', 'Content-Length': '0'} 2022-09-12T20:09:55.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbfc45cf2ea8ce9e-00631f9213-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.034 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:09:55.035 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:09:55.036 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:09:55.036 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:09:55.037 DEBUG:boto:Method: GET 2022-09-12T20:09:55.037 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:09:55.038 DEBUG:boto:Data: 2022-09-12T20:09:55.038 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.039 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.039 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.040 DEBUG:boto:Params: {} 2022-09-12T20:09:55.040 DEBUG:boto:Token: None 2022-09-12T20:09:55.041 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:09:55.042 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1uVWL/IRWfKlr1B9cFEsxloKEfo= 2022-09-12T20:09:55.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1uVWL/IRWfKlr1B9cFEsxloKEfo=', 'Content-Length': '0'} 2022-09-12T20:09:55.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbc7fe571efa2031-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.086 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:55.087 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:55.087 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:09:55.088 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:09:55.088 DEBUG:boto:Method: GET 2022-09-12T20:09:55.089 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:09:55.089 DEBUG:boto:Data: 2022-09-12T20:09:55.090 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.090 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.091 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.092 DEBUG:boto:Params: {} 2022-09-12T20:09:55.092 DEBUG:boto:Token: None 2022-09-12T20:09:55.093 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/ 2022-09-12T20:09:55.093 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms= 2022-09-12T20:09:55.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms=', 'Content-Length': '0'} 2022-09-12T20:09:55.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006de94fc92946a415-00631f9213-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.097 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:55.098 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:55.098 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:55.099 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:09:55.099 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:09:55.100 DEBUG:boto:Method: GET 2022-09-12T20:09:55.101 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:09:55.101 DEBUG:boto:Data: 2022-09-12T20:09:55.102 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.102 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.103 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.103 DEBUG:boto:Params: {} 2022-09-12T20:09:55.104 DEBUG:boto:Token: None 2022-09-12T20:09:55.104 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/ 2022-09-12T20:09:55.105 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms= 2022-09-12T20:09:55.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms=', 'Content-Length': '0'} 2022-09-12T20:09:55.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bf684dd9b400e72-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.109 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:55.110 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:09:55.111 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:09:55.111 DEBUG:boto:Method: HEAD 2022-09-12T20:09:55.112 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:09:55.112 DEBUG:boto:Data: 2022-09-12T20:09:55.113 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.114 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.114 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.115 DEBUG:boto:Params: {} 2022-09-12T20:09:55.115 DEBUG:boto:Token: None 2022-09-12T20:09:55.116 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj115 2022-09-12T20:09:55.117 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vCbC30lDWAI+sjJ7p9/QCNNs+bk= 2022-09-12T20:09:55.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vCbC30lDWAI+sjJ7p9/QCNNs+bk=', 'Content-Length': '0'} 2022-09-12T20:09:55.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002da8eea424f39989-00631f9213-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.121 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:09:55.121 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:09:55.122 DEBUG:boto:Method: HEAD 2022-09-12T20:09:55.122 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:09:55.123 DEBUG:boto:Data: 2022-09-12T20:09:55.124 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.124 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.125 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.125 DEBUG:boto:Params: {} 2022-09-12T20:09:55.126 DEBUG:boto:Token: None 2022-09-12T20:09:55.127 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj115 2022-09-12T20:09:55.127 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vCbC30lDWAI+sjJ7p9/QCNNs+bk= 2022-09-12T20:09:55.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vCbC30lDWAI+sjJ7p9/QCNNs+bk=', 'Content-Length': '0'} 2022-09-12T20:09:55.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000618066b963b008ba-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:09:55.131 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:09:55.132 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:09:55.132 DEBUG:boto:Method: GET 2022-09-12T20:09:55.133 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:09:55.134 DEBUG:boto:Data: 2022-09-12T20:09:55.134 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.135 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.135 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.136 DEBUG:boto:Params: {} 2022-09-12T20:09:55.137 DEBUG:boto:Token: None 2022-09-12T20:09:55.137 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj115 2022-09-12T20:09:55.138 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:77RO5bASOiKzwXlEMNacY39VkYk= 2022-09-12T20:09:55.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:77RO5bASOiKzwXlEMNacY39VkYk=', 'Content-Length': '0'} 2022-09-12T20:09:55.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfcfe88616198071-00631f9213-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.141 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:09:55.141 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:09:55.142 DEBUG:boto:Method: GET 2022-09-12T20:09:55.143 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:09:55.143 DEBUG:boto:Data: 2022-09-12T20:09:55.144 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.144 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.145 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.145 DEBUG:boto:Params: {} 2022-09-12T20:09:55.146 DEBUG:boto:Token: None 2022-09-12T20:09:55.146 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj115 2022-09-12T20:09:55.147 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:77RO5bASOiKzwXlEMNacY39VkYk= 2022-09-12T20:09:55.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:77RO5bASOiKzwXlEMNacY39VkYk=', 'Content-Length': '0'} 2022-09-12T20:09:55.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010b440ec1153db00-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.150 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:55.151 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:55.151 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:09:55.152 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:09:55.152 DEBUG:boto:Method: GET 2022-09-12T20:09:55.153 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:09:55.153 DEBUG:boto:Data: 2022-09-12T20:09:55.154 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.154 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.155 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.156 DEBUG:boto:Params: {} 2022-09-12T20:09:55.156 DEBUG:boto:Token: None 2022-09-12T20:09:55.157 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/ 2022-09-12T20:09:55.157 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms= 2022-09-12T20:09:55.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms=', 'Content-Length': '0'} 2022-09-12T20:09:55.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc14bc22fa481fd2-00631f9213-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.161 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:55.162 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:55.162 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:55.163 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:09:55.164 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:09:55.164 DEBUG:boto:Method: GET 2022-09-12T20:09:55.165 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:09:55.166 DEBUG:boto:Data: 2022-09-12T20:09:55.166 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.167 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.167 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.168 DEBUG:boto:Params: {} 2022-09-12T20:09:55.168 DEBUG:boto:Token: None 2022-09-12T20:09:55.169 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/ 2022-09-12T20:09:55.170 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms= 2022-09-12T20:09:55.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms=', 'Content-Length': '0'} 2022-09-12T20:09:55.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000790f9d8dfc24f977-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.175 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:55.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:09:55.176 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:09:55.177 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:09:55.178 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:09:55.178 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:09:55.179 DEBUG:boto:Method: GET 2022-09-12T20:09:55.180 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:09:55.181 DEBUG:boto:Data: 2022-09-12T20:09:55.181 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.182 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.183 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.183 DEBUG:boto:Params: {} 2022-09-12T20:09:55.184 DEBUG:boto:Token: None 2022-09-12T20:09:55.184 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:09:55.185 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TXxHXwZoMeRZAtQ9oPI3FHArVSI= 2022-09-12T20:09:55.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TXxHXwZoMeRZAtQ9oPI3FHArVSI=', 'Content-Length': '0'} 2022-09-12T20:09:55.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ac4d123f3ec4ab3-00631f9213-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.188 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:09:55.189 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:09:55.189 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:09:55.190 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:09:55.190 DEBUG:boto:Method: GET 2022-09-12T20:09:55.191 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:09:55.191 DEBUG:boto:Data: 2022-09-12T20:09:55.192 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.193 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.193 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.194 DEBUG:boto:Params: {} 2022-09-12T20:09:55.194 DEBUG:boto:Token: None 2022-09-12T20:09:55.195 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:09:55.196 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TXxHXwZoMeRZAtQ9oPI3FHArVSI= 2022-09-12T20:09:55.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TXxHXwZoMeRZAtQ9oPI3FHArVSI=', 'Content-Length': '0'} 2022-09-12T20:09:55.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004eb707ede9f86c22-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.242 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:09:55.243 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:09:55.243 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:09:55.244 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:09:55.244 DEBUG:boto:Method: HEAD 2022-09-12T20:09:55.245 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:09:55.245 DEBUG:boto:Data: 2022-09-12T20:09:55.246 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.246 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.247 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.247 DEBUG:boto:Params: {} 2022-09-12T20:09:55.248 DEBUG:boto:Token: None 2022-09-12T20:09:55.248 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:09:55.249 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D8HOVmgDidnmNczpzqM8nj+gDO4= 2022-09-12T20:09:55.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D8HOVmgDidnmNczpzqM8nj+gDO4=', 'Content-Length': '0'} 2022-09-12T20:09:55.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fce88b9b9aad04a6-00631f9213-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.252 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:09:55.253 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:09:55.253 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:09:55.254 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:09:55.254 DEBUG:boto:Method: HEAD 2022-09-12T20:09:55.255 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:09:55.255 DEBUG:boto:Data: 2022-09-12T20:09:55.256 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.256 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.257 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.257 DEBUG:boto:Params: {} 2022-09-12T20:09:55.258 DEBUG:boto:Token: None 2022-09-12T20:09:55.258 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:09:55.259 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D8HOVmgDidnmNczpzqM8nj+gDO4= 2022-09-12T20:09:55.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D8HOVmgDidnmNczpzqM8nj+gDO4=', 'Content-Length': '0'} 2022-09-12T20:09:55.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6ea25c2f105204d-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:09:55.262 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:09:55.263 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:09:55.264 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:09:55.264 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:09:55.265 DEBUG:boto:Method: GET 2022-09-12T20:09:55.265 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:09:55.266 DEBUG:boto:Data: 2022-09-12T20:09:55.266 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.267 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.267 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.268 DEBUG:boto:Params: {} 2022-09-12T20:09:55.268 DEBUG:boto:Token: None 2022-09-12T20:09:55.269 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:09:55.270 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TXxHXwZoMeRZAtQ9oPI3FHArVSI= 2022-09-12T20:09:55.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TXxHXwZoMeRZAtQ9oPI3FHArVSI=', 'Content-Length': '0'} 2022-09-12T20:09:55.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022624c12bd581b69-00631f9213-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.273 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:09:55.273 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:09:55.274 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:09:55.274 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:09:55.275 DEBUG:boto:Method: GET 2022-09-12T20:09:55.275 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:09:55.276 DEBUG:boto:Data: 2022-09-12T20:09:55.276 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.277 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.278 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.278 DEBUG:boto:Params: {} 2022-09-12T20:09:55.279 DEBUG:boto:Token: None 2022-09-12T20:09:55.279 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:09:55.280 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TXxHXwZoMeRZAtQ9oPI3FHArVSI= 2022-09-12T20:09:55.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TXxHXwZoMeRZAtQ9oPI3FHArVSI=', 'Content-Length': '0'} 2022-09-12T20:09:55.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081b487bbdfa67f48-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.283 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:55.284 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:55.284 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:09:55.285 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:09:55.286 DEBUG:boto:Method: GET 2022-09-12T20:09:55.286 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:09:55.287 DEBUG:boto:Data: 2022-09-12T20:09:55.287 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.288 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.288 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.289 DEBUG:boto:Params: {} 2022-09-12T20:09:55.289 DEBUG:boto:Token: None 2022-09-12T20:09:55.290 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/ 2022-09-12T20:09:55.290 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms= 2022-09-12T20:09:55.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms=', 'Content-Length': '0'} 2022-09-12T20:09:55.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011fd7283e4dcfa42-00631f9213-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.294 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:55.295 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:55.296 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:55.296 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:09:55.297 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:09:55.297 DEBUG:boto:Method: GET 2022-09-12T20:09:55.298 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:09:55.298 DEBUG:boto:Data: 2022-09-12T20:09:55.299 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.299 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.300 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.300 DEBUG:boto:Params: {} 2022-09-12T20:09:55.301 DEBUG:boto:Token: None 2022-09-12T20:09:55.301 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/ 2022-09-12T20:09:55.302 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms= 2022-09-12T20:09:55.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms=', 'Content-Length': '0'} 2022-09-12T20:09:55.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003023b3aaaf42e375-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.306 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:55.307 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:09:55.308 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:09:55.308 DEBUG:boto:Method: HEAD 2022-09-12T20:09:55.309 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:09:55.309 DEBUG:boto:Data: 2022-09-12T20:09:55.310 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.310 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.311 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.311 DEBUG:boto:Params: {} 2022-09-12T20:09:55.312 DEBUG:boto:Token: None 2022-09-12T20:09:55.312 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj17 2022-09-12T20:09:55.313 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cp6IbCAwQ7Gh1wEFLPmoeMxFao8= 2022-09-12T20:09:55.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cp6IbCAwQ7Gh1wEFLPmoeMxFao8=', 'Content-Length': '0'} 2022-09-12T20:09:55.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a8b2726c14357af-00631f9213-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.316 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:09:55.317 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:09:55.317 DEBUG:boto:Method: HEAD 2022-09-12T20:09:55.318 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:09:55.318 DEBUG:boto:Data: 2022-09-12T20:09:55.319 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.319 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.320 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.320 DEBUG:boto:Params: {} 2022-09-12T20:09:55.321 DEBUG:boto:Token: None 2022-09-12T20:09:55.321 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj17 2022-09-12T20:09:55.322 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cp6IbCAwQ7Gh1wEFLPmoeMxFao8= 2022-09-12T20:09:55.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cp6IbCAwQ7Gh1wEFLPmoeMxFao8=', 'Content-Length': '0'} 2022-09-12T20:09:55.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ba7713a0d57b65e-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:09:55.326 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:09:55.326 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:09:55.327 DEBUG:boto:Method: GET 2022-09-12T20:09:55.327 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:09:55.328 DEBUG:boto:Data: 2022-09-12T20:09:55.328 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.329 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.329 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.330 DEBUG:boto:Params: {} 2022-09-12T20:09:55.331 DEBUG:boto:Token: None 2022-09-12T20:09:55.331 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj17 2022-09-12T20:09:55.332 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:K5T7LKxmeOwUa+DigmG/DF+GaIk= 2022-09-12T20:09:55.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:K5T7LKxmeOwUa+DigmG/DF+GaIk=', 'Content-Length': '0'} 2022-09-12T20:09:55.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9eab2dd77626c21-00631f9213-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.335 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:09:55.335 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:09:55.336 DEBUG:boto:Method: GET 2022-09-12T20:09:55.336 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:09:55.337 DEBUG:boto:Data: 2022-09-12T20:09:55.338 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.338 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.339 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.339 DEBUG:boto:Params: {} 2022-09-12T20:09:55.340 DEBUG:boto:Token: None 2022-09-12T20:09:55.340 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj17 2022-09-12T20:09:55.341 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:K5T7LKxmeOwUa+DigmG/DF+GaIk= 2022-09-12T20:09:55.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:K5T7LKxmeOwUa+DigmG/DF+GaIk=', 'Content-Length': '0'} 2022-09-12T20:09:55.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000464734a6c087ba4f-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.345 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:55.345 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:55.346 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:09:55.346 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:09:55.347 DEBUG:boto:Method: GET 2022-09-12T20:09:55.347 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:09:55.348 DEBUG:boto:Data: 2022-09-12T20:09:55.348 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.349 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.349 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.350 DEBUG:boto:Params: {} 2022-09-12T20:09:55.351 DEBUG:boto:Token: None 2022-09-12T20:09:55.351 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/ 2022-09-12T20:09:55.352 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms= 2022-09-12T20:09:55.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms=', 'Content-Length': '0'} 2022-09-12T20:09:55.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000712d4316e06be5fb-00631f9213-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.398 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:55.399 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:55.400 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:55.400 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:09:55.401 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:09:55.401 DEBUG:boto:Method: GET 2022-09-12T20:09:55.402 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:09:55.402 DEBUG:boto:Data: 2022-09-12T20:09:55.403 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.404 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.404 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.405 DEBUG:boto:Params: {} 2022-09-12T20:09:55.405 DEBUG:boto:Token: None 2022-09-12T20:09:55.406 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/ 2022-09-12T20:09:55.407 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms= 2022-09-12T20:09:55.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms=', 'Content-Length': '0'} 2022-09-12T20:09:55.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa842a106bbd565d-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.412 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:55.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:09:55.413 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:09:55.413 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:09:55.414 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:09:55.414 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:09:55.415 DEBUG:boto:Method: GET 2022-09-12T20:09:55.416 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:09:55.416 DEBUG:boto:Data: 2022-09-12T20:09:55.417 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.417 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.418 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.418 DEBUG:boto:Params: {} 2022-09-12T20:09:55.419 DEBUG:boto:Token: None 2022-09-12T20:09:55.419 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:09:55.420 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GPSZhwBsL01VI5mKw6odYUXtCSI= 2022-09-12T20:09:55.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GPSZhwBsL01VI5mKw6odYUXtCSI=', 'Content-Length': '0'} 2022-09-12T20:09:55.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c88bb067f3a99442-00631f9213-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.423 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:09:55.424 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:09:55.424 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:09:55.425 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:09:55.426 DEBUG:boto:Method: GET 2022-09-12T20:09:55.426 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:09:55.427 DEBUG:boto:Data: 2022-09-12T20:09:55.427 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.428 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.428 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.429 DEBUG:boto:Params: {} 2022-09-12T20:09:55.430 DEBUG:boto:Token: None 2022-09-12T20:09:55.430 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:09:55.431 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GPSZhwBsL01VI5mKw6odYUXtCSI= 2022-09-12T20:09:55.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GPSZhwBsL01VI5mKw6odYUXtCSI=', 'Content-Length': '0'} 2022-09-12T20:09:55.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f721fe89d51e6ab-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.474 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:09:55.475 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:09:55.475 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:09:55.476 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:09:55.476 DEBUG:boto:Method: HEAD 2022-09-12T20:09:55.477 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:09:55.477 DEBUG:boto:Data: 2022-09-12T20:09:55.478 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.478 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.479 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.479 DEBUG:boto:Params: {} 2022-09-12T20:09:55.480 DEBUG:boto:Token: None 2022-09-12T20:09:55.480 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:09:55.481 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3wGJJzP3Oe9AgzBiirmuX9dFnww= 2022-09-12T20:09:55.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3wGJJzP3Oe9AgzBiirmuX9dFnww=', 'Content-Length': '0'} 2022-09-12T20:09:55.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0910bbf02f388a2-00631f9213-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.485 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:09:55.485 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:09:55.486 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:09:55.486 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:09:55.487 DEBUG:boto:Method: HEAD 2022-09-12T20:09:55.487 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:09:55.488 DEBUG:boto:Data: 2022-09-12T20:09:55.488 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.489 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.489 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.490 DEBUG:boto:Params: {} 2022-09-12T20:09:55.490 DEBUG:boto:Token: None 2022-09-12T20:09:55.491 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:09:55.492 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3wGJJzP3Oe9AgzBiirmuX9dFnww= 2022-09-12T20:09:55.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3wGJJzP3Oe9AgzBiirmuX9dFnww=', 'Content-Length': '0'} 2022-09-12T20:09:55.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061e7055e048853d6-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:09:55.495 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:09:55.496 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:09:55.496 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:09:55.497 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:09:55.497 DEBUG:boto:Method: GET 2022-09-12T20:09:55.498 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:09:55.498 DEBUG:boto:Data: 2022-09-12T20:09:55.499 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.499 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.500 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.501 DEBUG:boto:Params: {} 2022-09-12T20:09:55.501 DEBUG:boto:Token: None 2022-09-12T20:09:55.502 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:09:55.502 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GPSZhwBsL01VI5mKw6odYUXtCSI= 2022-09-12T20:09:55.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GPSZhwBsL01VI5mKw6odYUXtCSI=', 'Content-Length': '0'} 2022-09-12T20:09:55.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070197b607181c69f-00631f9213-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.506 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:09:55.506 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:09:55.507 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:09:55.507 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:09:55.508 DEBUG:boto:Method: GET 2022-09-12T20:09:55.508 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:09:55.509 DEBUG:boto:Data: 2022-09-12T20:09:55.509 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.510 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.510 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.511 DEBUG:boto:Params: {} 2022-09-12T20:09:55.512 DEBUG:boto:Token: None 2022-09-12T20:09:55.512 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:09:55.513 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GPSZhwBsL01VI5mKw6odYUXtCSI= 2022-09-12T20:09:55.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GPSZhwBsL01VI5mKw6odYUXtCSI=', 'Content-Length': '0'} 2022-09-12T20:09:55.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5bfaac71dcdfa23-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.516 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:55.517 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:55.517 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:09:55.518 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:09:55.518 DEBUG:boto:Method: GET 2022-09-12T20:09:55.519 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:09:55.519 DEBUG:boto:Data: 2022-09-12T20:09:55.520 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.521 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.521 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.522 DEBUG:boto:Params: {} 2022-09-12T20:09:55.522 DEBUG:boto:Token: None 2022-09-12T20:09:55.523 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/ 2022-09-12T20:09:55.523 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms= 2022-09-12T20:09:55.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms=', 'Content-Length': '0'} 2022-09-12T20:09:55.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3c43026fb5d3991-00631f9213-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.527 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:55.528 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:55.529 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:55.529 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:09:55.530 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:09:55.530 DEBUG:boto:Method: GET 2022-09-12T20:09:55.531 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:09:55.531 DEBUG:boto:Data: 2022-09-12T20:09:55.532 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.532 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.533 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.533 DEBUG:boto:Params: {} 2022-09-12T20:09:55.534 DEBUG:boto:Token: None 2022-09-12T20:09:55.534 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/ 2022-09-12T20:09:55.535 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms= 2022-09-12T20:09:55.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms=', 'Content-Length': '0'} 2022-09-12T20:09:55.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000490bfc98bfba5f1a-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.540 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:55.540 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:09:55.541 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:09:55.541 DEBUG:boto:Method: HEAD 2022-09-12T20:09:55.542 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:09:55.542 DEBUG:boto:Data: 2022-09-12T20:09:55.543 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.543 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.544 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.544 DEBUG:boto:Params: {} 2022-09-12T20:09:55.545 DEBUG:boto:Token: None 2022-09-12T20:09:55.546 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj19 2022-09-12T20:09:55.546 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EaQKBjyq00LZkq8u7MtFZ7TgyS0= 2022-09-12T20:09:55.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EaQKBjyq00LZkq8u7MtFZ7TgyS0=', 'Content-Length': '0'} 2022-09-12T20:09:55.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cdcaf12a5636ee0-00631f9213-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.549 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:09:55.550 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:09:55.550 DEBUG:boto:Method: HEAD 2022-09-12T20:09:55.551 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:09:55.551 DEBUG:boto:Data: 2022-09-12T20:09:55.552 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.552 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.553 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.553 DEBUG:boto:Params: {} 2022-09-12T20:09:55.554 DEBUG:boto:Token: None 2022-09-12T20:09:55.555 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj19 2022-09-12T20:09:55.555 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EaQKBjyq00LZkq8u7MtFZ7TgyS0= 2022-09-12T20:09:55.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EaQKBjyq00LZkq8u7MtFZ7TgyS0=', 'Content-Length': '0'} 2022-09-12T20:09:55.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dd90c3df2030d40-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:09:55.559 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:09:55.559 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:09:55.560 DEBUG:boto:Method: GET 2022-09-12T20:09:55.560 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:09:55.561 DEBUG:boto:Data: 2022-09-12T20:09:55.562 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.562 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.563 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.563 DEBUG:boto:Params: {} 2022-09-12T20:09:55.564 DEBUG:boto:Token: None 2022-09-12T20:09:55.564 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj19 2022-09-12T20:09:55.565 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DXUGaA9NCAUnwdRIo0KTZbt1Xug= 2022-09-12T20:09:55.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DXUGaA9NCAUnwdRIo0KTZbt1Xug=', 'Content-Length': '0'} 2022-09-12T20:09:55.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8f1eb78617018f0-00631f9213-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.568 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:09:55.569 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:09:55.569 DEBUG:boto:Method: GET 2022-09-12T20:09:55.570 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:09:55.570 DEBUG:boto:Data: 2022-09-12T20:09:55.571 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.572 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.572 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.572 DEBUG:boto:Params: {} 2022-09-12T20:09:55.573 DEBUG:boto:Token: None 2022-09-12T20:09:55.574 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj19 2022-09-12T20:09:55.574 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DXUGaA9NCAUnwdRIo0KTZbt1Xug= 2022-09-12T20:09:55.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DXUGaA9NCAUnwdRIo0KTZbt1Xug=', 'Content-Length': '0'} 2022-09-12T20:09:55.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036599dab9197111e-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.618 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:55.619 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:55.619 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:09:55.620 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:09:55.620 DEBUG:boto:Method: GET 2022-09-12T20:09:55.621 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:09:55.621 DEBUG:boto:Data: 2022-09-12T20:09:55.622 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.622 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.623 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.624 DEBUG:boto:Params: {} 2022-09-12T20:09:55.624 DEBUG:boto:Token: None 2022-09-12T20:09:55.625 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/ 2022-09-12T20:09:55.625 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms= 2022-09-12T20:09:55.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms=', 'Content-Length': '0'} 2022-09-12T20:09:55.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a994200fd13bee1c-00631f9213-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.629 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:55.630 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:55.631 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:55.631 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:09:55.632 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:09:55.632 DEBUG:boto:Method: GET 2022-09-12T20:09:55.633 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:09:55.633 DEBUG:boto:Data: 2022-09-12T20:09:55.634 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.635 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.635 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.636 DEBUG:boto:Params: {} 2022-09-12T20:09:55.636 DEBUG:boto:Token: None 2022-09-12T20:09:55.637 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/ 2022-09-12T20:09:55.637 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms= 2022-09-12T20:09:55.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms=', 'Content-Length': '0'} 2022-09-12T20:09:55.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df139cf384ffafd7-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.642 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:55.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:09:55.643 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:09:55.644 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:09:55.644 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:09:55.645 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:09:55.646 DEBUG:boto:Method: GET 2022-09-12T20:09:55.646 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:09:55.647 DEBUG:boto:Data: 2022-09-12T20:09:55.647 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.648 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.648 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.649 DEBUG:boto:Params: {} 2022-09-12T20:09:55.649 DEBUG:boto:Token: None 2022-09-12T20:09:55.650 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:09:55.651 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CS6ZgLPYiFe1zwxuI60BoIG6P3k= 2022-09-12T20:09:55.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CS6ZgLPYiFe1zwxuI60BoIG6P3k=', 'Content-Length': '0'} 2022-09-12T20:09:55.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015029c14689ff2a0-00631f9213-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.655 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:09:55.656 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:09:55.656 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:09:55.657 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:09:55.657 DEBUG:boto:Method: GET 2022-09-12T20:09:55.658 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:09:55.658 DEBUG:boto:Data: 2022-09-12T20:09:55.659 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.659 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.660 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.660 DEBUG:boto:Params: {} 2022-09-12T20:09:55.661 DEBUG:boto:Token: None 2022-09-12T20:09:55.661 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:09:55.662 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CS6ZgLPYiFe1zwxuI60BoIG6P3k= 2022-09-12T20:09:55.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CS6ZgLPYiFe1zwxuI60BoIG6P3k=', 'Content-Length': '0'} 2022-09-12T20:09:55.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4995ec9afd98ad4-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.666 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:09:55.666 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:09:55.667 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:09:55.667 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:09:55.668 DEBUG:boto:Method: HEAD 2022-09-12T20:09:55.668 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:09:55.669 DEBUG:boto:Data: 2022-09-12T20:09:55.669 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.670 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.670 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.671 DEBUG:boto:Params: {} 2022-09-12T20:09:55.671 DEBUG:boto:Token: None 2022-09-12T20:09:55.672 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:09:55.672 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8vYNY6fIX6mUbQRHbZGnogbV5FY= 2022-09-12T20:09:55.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8vYNY6fIX6mUbQRHbZGnogbV5FY=', 'Content-Length': '0'} 2022-09-12T20:09:55.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b3d9bc82688681d-00631f9213-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.681 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:09:55.682 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:09:55.682 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:09:55.683 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:09:55.683 DEBUG:boto:Method: HEAD 2022-09-12T20:09:55.684 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:09:55.684 DEBUG:boto:Data: 2022-09-12T20:09:55.685 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.685 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.686 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.686 DEBUG:boto:Params: {} 2022-09-12T20:09:55.687 DEBUG:boto:Token: None 2022-09-12T20:09:55.688 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:09:55.688 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8vYNY6fIX6mUbQRHbZGnogbV5FY= 2022-09-12T20:09:55.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8vYNY6fIX6mUbQRHbZGnogbV5FY=', 'Content-Length': '0'} 2022-09-12T20:09:55.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000535dce7919c9869-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:09:55.692 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:09:55.692 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:09:55.693 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:09:55.693 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:09:55.694 DEBUG:boto:Method: GET 2022-09-12T20:09:55.695 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:09:55.695 DEBUG:boto:Data: 2022-09-12T20:09:55.696 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.696 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.697 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.698 DEBUG:boto:Params: {} 2022-09-12T20:09:55.698 DEBUG:boto:Token: None 2022-09-12T20:09:55.699 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:09:55.699 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CS6ZgLPYiFe1zwxuI60BoIG6P3k= 2022-09-12T20:09:55.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CS6ZgLPYiFe1zwxuI60BoIG6P3k=', 'Content-Length': '0'} 2022-09-12T20:09:55.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021a947e8dabb6887-00631f9213-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.742 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:09:55.743 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:09:55.743 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:09:55.744 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:09:55.745 DEBUG:boto:Method: GET 2022-09-12T20:09:55.745 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:09:55.746 DEBUG:boto:Data: 2022-09-12T20:09:55.746 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.747 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.747 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.748 DEBUG:boto:Params: {} 2022-09-12T20:09:55.749 DEBUG:boto:Token: None 2022-09-12T20:09:55.749 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:09:55.750 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CS6ZgLPYiFe1zwxuI60BoIG6P3k= 2022-09-12T20:09:55.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CS6ZgLPYiFe1zwxuI60BoIG6P3k=', 'Content-Length': '0'} 2022-09-12T20:09:55.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012d724c28a007cb2-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.754 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:55.754 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:55.755 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:09:55.756 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:09:55.756 DEBUG:boto:Method: GET 2022-09-12T20:09:55.757 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:09:55.757 DEBUG:boto:Data: 2022-09-12T20:09:55.758 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.758 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.759 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.759 DEBUG:boto:Params: {} 2022-09-12T20:09:55.760 DEBUG:boto:Token: None 2022-09-12T20:09:55.761 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/ 2022-09-12T20:09:55.761 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms= 2022-09-12T20:09:55.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms=', 'Content-Length': '0'} 2022-09-12T20:09:55.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aca21e625c6bee3d-00631f9213-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.765 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:55.766 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:55.767 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:55.767 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:09:55.768 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:09:55.768 DEBUG:boto:Method: GET 2022-09-12T20:09:55.769 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:09:55.769 DEBUG:boto:Data: 2022-09-12T20:09:55.770 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.770 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.771 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.772 DEBUG:boto:Params: {} 2022-09-12T20:09:55.772 DEBUG:boto:Token: None 2022-09-12T20:09:55.773 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/ 2022-09-12T20:09:55.773 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms= 2022-09-12T20:09:55.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms=', 'Content-Length': '0'} 2022-09-12T20:09:55.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4ba0398ad9fbfeb-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.778 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:55.779 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:09:55.779 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:09:55.780 DEBUG:boto:Method: HEAD 2022-09-12T20:09:55.780 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:09:55.781 DEBUG:boto:Data: 2022-09-12T20:09:55.782 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.782 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.783 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.783 DEBUG:boto:Params: {} 2022-09-12T20:09:55.784 DEBUG:boto:Token: None 2022-09-12T20:09:55.785 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj22 2022-09-12T20:09:55.785 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:joqK4LAmCc7/MZn2mLpzcbKNyVU= 2022-09-12T20:09:55.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:joqK4LAmCc7/MZn2mLpzcbKNyVU=', 'Content-Length': '0'} 2022-09-12T20:09:55.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b037f77f93d8da2-00631f9213-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.789 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:09:55.790 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:09:55.790 DEBUG:boto:Method: HEAD 2022-09-12T20:09:55.791 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:09:55.791 DEBUG:boto:Data: 2022-09-12T20:09:55.792 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.792 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.793 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.793 DEBUG:boto:Params: {} 2022-09-12T20:09:55.794 DEBUG:boto:Token: None 2022-09-12T20:09:55.794 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj22 2022-09-12T20:09:55.795 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:joqK4LAmCc7/MZn2mLpzcbKNyVU= 2022-09-12T20:09:55.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:joqK4LAmCc7/MZn2mLpzcbKNyVU=', 'Content-Length': '0'} 2022-09-12T20:09:55.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa010c0d2d7ade53-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:09:55.799 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:09:55.799 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:09:55.800 DEBUG:boto:Method: GET 2022-09-12T20:09:55.800 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:09:55.801 DEBUG:boto:Data: 2022-09-12T20:09:55.801 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.802 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.802 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.803 DEBUG:boto:Params: {} 2022-09-12T20:09:55.803 DEBUG:boto:Token: None 2022-09-12T20:09:55.804 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj22 2022-09-12T20:09:55.805 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Y0udf7Th9WwHK9BcYZBV1i8uTk= 2022-09-12T20:09:55.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Y0udf7Th9WwHK9BcYZBV1i8uTk=', 'Content-Length': '0'} 2022-09-12T20:09:55.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e231aeab4ec1b894-00631f9213-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.808 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:09:55.809 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:09:55.809 DEBUG:boto:Method: GET 2022-09-12T20:09:55.810 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:09:55.810 DEBUG:boto:Data: 2022-09-12T20:09:55.811 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.811 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.812 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.812 DEBUG:boto:Params: {} 2022-09-12T20:09:55.813 DEBUG:boto:Token: None 2022-09-12T20:09:55.814 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj22 2022-09-12T20:09:55.814 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Y0udf7Th9WwHK9BcYZBV1i8uTk= 2022-09-12T20:09:55.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Y0udf7Th9WwHK9BcYZBV1i8uTk=', 'Content-Length': '0'} 2022-09-12T20:09:55.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045bc6d0694432581-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.817 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:55.818 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:55.819 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:09:55.819 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:09:55.820 DEBUG:boto:Method: GET 2022-09-12T20:09:55.820 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:09:55.821 DEBUG:boto:Data: 2022-09-12T20:09:55.821 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.822 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.822 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.823 DEBUG:boto:Params: {} 2022-09-12T20:09:55.823 DEBUG:boto:Token: None 2022-09-12T20:09:55.824 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/ 2022-09-12T20:09:55.825 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms= 2022-09-12T20:09:55.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms=', 'Content-Length': '0'} 2022-09-12T20:09:55.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002360aae5b1657e6c-00631f9213-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.828 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:55.829 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:55.830 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:55.830 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:09:55.831 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:09:55.831 DEBUG:boto:Method: GET 2022-09-12T20:09:55.832 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:09:55.832 DEBUG:boto:Data: 2022-09-12T20:09:55.833 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.833 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.834 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.834 DEBUG:boto:Params: {} 2022-09-12T20:09:55.835 DEBUG:boto:Token: None 2022-09-12T20:09:55.836 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/ 2022-09-12T20:09:55.836 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms= 2022-09-12T20:09:55.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms=', 'Content-Length': '0'} 2022-09-12T20:09:55.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c79517f4fc732ce-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.841 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:55.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:09:55.842 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:09:55.843 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:09:55.843 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:09:55.844 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:09:55.844 DEBUG:boto:Method: GET 2022-09-12T20:09:55.845 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:09:55.845 DEBUG:boto:Data: 2022-09-12T20:09:55.846 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.847 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.847 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.848 DEBUG:boto:Params: {} 2022-09-12T20:09:55.848 DEBUG:boto:Token: None 2022-09-12T20:09:55.849 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:09:55.849 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KorNNDVpNnF9kKEW2nym4ZybCVU= 2022-09-12T20:09:55.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KorNNDVpNnF9kKEW2nym4ZybCVU=', 'Content-Length': '0'} 2022-09-12T20:09:55.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026a2f0838c2d7b70-00631f9213-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.853 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:09:55.853 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:09:55.854 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:09:55.854 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:09:55.855 DEBUG:boto:Method: GET 2022-09-12T20:09:55.856 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:09:55.856 DEBUG:boto:Data: 2022-09-12T20:09:55.857 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.857 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.858 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.858 DEBUG:boto:Params: {} 2022-09-12T20:09:55.859 DEBUG:boto:Token: None 2022-09-12T20:09:55.860 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:09:55.860 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KorNNDVpNnF9kKEW2nym4ZybCVU= 2022-09-12T20:09:55.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KorNNDVpNnF9kKEW2nym4ZybCVU=', 'Content-Length': '0'} 2022-09-12T20:09:55.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000239077f33524ad04-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.864 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:09:55.864 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:09:55.865 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:09:55.865 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:09:55.866 DEBUG:boto:Method: HEAD 2022-09-12T20:09:55.866 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:09:55.867 DEBUG:boto:Data: 2022-09-12T20:09:55.867 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.868 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.868 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.869 DEBUG:boto:Params: {} 2022-09-12T20:09:55.869 DEBUG:boto:Token: None 2022-09-12T20:09:55.870 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:09:55.870 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5C7qUIDA1Bqia5Ixe8ltJvgdIEM= 2022-09-12T20:09:55.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5C7qUIDA1Bqia5Ixe8ltJvgdIEM=', 'Content-Length': '0'} 2022-09-12T20:09:55.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000983fc5ba249fff1f-00631f9213-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.874 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:09:55.874 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:09:55.875 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:09:55.875 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:09:55.876 DEBUG:boto:Method: HEAD 2022-09-12T20:09:55.877 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:09:55.877 DEBUG:boto:Data: 2022-09-12T20:09:55.878 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.878 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.879 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.879 DEBUG:boto:Params: {} 2022-09-12T20:09:55.880 DEBUG:boto:Token: None 2022-09-12T20:09:55.880 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:09:55.881 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5C7qUIDA1Bqia5Ixe8ltJvgdIEM= 2022-09-12T20:09:55.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5C7qUIDA1Bqia5Ixe8ltJvgdIEM=', 'Content-Length': '0'} 2022-09-12T20:09:55.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000785e4d538cd5d858-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:09:55.885 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:09:55.886 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:09:55.886 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:09:55.887 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:09:55.887 DEBUG:boto:Method: GET 2022-09-12T20:09:55.888 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:09:55.888 DEBUG:boto:Data: 2022-09-12T20:09:55.889 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.889 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.890 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.891 DEBUG:boto:Params: {} 2022-09-12T20:09:55.891 DEBUG:boto:Token: None 2022-09-12T20:09:55.892 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:09:55.892 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KorNNDVpNnF9kKEW2nym4ZybCVU= 2022-09-12T20:09:55.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KorNNDVpNnF9kKEW2nym4ZybCVU=', 'Content-Length': '0'} 2022-09-12T20:09:55.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000de2764ae5fee292-00631f9213-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.896 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:09:55.897 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:09:55.897 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:09:55.898 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:09:55.898 DEBUG:boto:Method: GET 2022-09-12T20:09:55.899 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:09:55.899 DEBUG:boto:Data: 2022-09-12T20:09:55.900 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.900 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.901 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.901 DEBUG:boto:Params: {} 2022-09-12T20:09:55.902 DEBUG:boto:Token: None 2022-09-12T20:09:55.903 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:09:55.903 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KorNNDVpNnF9kKEW2nym4ZybCVU= 2022-09-12T20:09:55.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KorNNDVpNnF9kKEW2nym4ZybCVU=', 'Content-Length': '0'} 2022-09-12T20:09:55.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4c1be20444f1207-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.950 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:55.951 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:55.951 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:09:55.952 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:09:55.952 DEBUG:boto:Method: GET 2022-09-12T20:09:55.953 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:09:55.953 DEBUG:boto:Data: 2022-09-12T20:09:55.954 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.954 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.955 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.955 DEBUG:boto:Params: {} 2022-09-12T20:09:55.956 DEBUG:boto:Token: None 2022-09-12T20:09:55.956 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/ 2022-09-12T20:09:55.957 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms= 2022-09-12T20:09:55.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms=', 'Content-Length': '0'} 2022-09-12T20:09:55.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c19c05d2ce38d21-00631f9213-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.961 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:55.962 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:55.963 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:55.963 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:09:55.964 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:09:55.964 DEBUG:boto:Method: GET 2022-09-12T20:09:55.965 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:09:55.965 DEBUG:boto:Data: 2022-09-12T20:09:55.966 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.966 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.967 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.967 DEBUG:boto:Params: {} 2022-09-12T20:09:55.968 DEBUG:boto:Token: None 2022-09-12T20:09:55.968 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/ 2022-09-12T20:09:55.969 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms= 2022-09-12T20:09:55.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RlUb/Q+TX0FZtL9erS4W9CaGzms=', 'Content-Length': '0'} 2022-09-12T20:09:55.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c50a95465f7552d5-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.974 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:55.974 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:09:55.975 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:09:55.975 DEBUG:boto:Method: HEAD 2022-09-12T20:09:55.976 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:09:55.976 DEBUG:boto:Data: 2022-09-12T20:09:55.977 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.977 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.978 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.978 DEBUG:boto:Params: {} 2022-09-12T20:09:55.979 DEBUG:boto:Token: None 2022-09-12T20:09:55.980 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj23 2022-09-12T20:09:55.980 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TvpjZPPRrjCBZbU1QPC0ERo3Ph8= 2022-09-12T20:09:55.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TvpjZPPRrjCBZbU1QPC0ERo3Ph8=', 'Content-Length': '0'} 2022-09-12T20:09:55.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df22488b0b0bd031-00631f9213-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.984 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:09:55.984 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:09:55.985 DEBUG:boto:Method: HEAD 2022-09-12T20:09:55.985 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:09:55.986 DEBUG:boto:Data: 2022-09-12T20:09:55.986 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.987 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:55.987 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.988 DEBUG:boto:Params: {} 2022-09-12T20:09:55.989 DEBUG:boto:Token: None 2022-09-12T20:09:55.989 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj23 2022-09-12T20:09:55.990 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TvpjZPPRrjCBZbU1QPC0ERo3Ph8= 2022-09-12T20:09:55.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TvpjZPPRrjCBZbU1QPC0ERo3Ph8=', 'Content-Length': '0'} 2022-09-12T20:09:55.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bbe0a2b6424e9d5-00631f9213-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:55.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:09:55.994 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:09:55.994 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:09:55.995 DEBUG:boto:Method: GET 2022-09-12T20:09:55.995 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:09:55.996 DEBUG:boto:Data: 2022-09-12T20:09:55.996 DEBUG:boto:Headers: {} 2022-09-12T20:09:55.997 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:55.997 DEBUG:boto:Port: 8000 2022-09-12T20:09:55.998 DEBUG:boto:Params: {} 2022-09-12T20:09:55.999 DEBUG:boto:Token: None 2022-09-12T20:09:55.999 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:55 GMT /jdpiak-75/obj23 2022-09-12T20:09:56.000 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tsdObPx2lE8XjKsSTCanrz5XMl4= 2022-09-12T20:09:56.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:55 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tsdObPx2lE8XjKsSTCanrz5XMl4=', 'Content-Length': '0'} 2022-09-12T20:09:56.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000274fc9e9316343f8-00631f9214-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.003 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:09:56.004 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:09:56.004 DEBUG:boto:Method: GET 2022-09-12T20:09:56.005 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:09:56.006 DEBUG:boto:Data: 2022-09-12T20:09:56.006 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.007 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.007 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.008 DEBUG:boto:Params: {} 2022-09-12T20:09:56.008 DEBUG:boto:Token: None 2022-09-12T20:09:56.009 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj23 2022-09-12T20:09:56.009 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CEBoWeaFkcUQ+Vv54Af3J2DDLa0= 2022-09-12T20:09:56.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CEBoWeaFkcUQ+Vv54Af3J2DDLa0=', 'Content-Length': '0'} 2022-09-12T20:09:56.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc2c66549d35e1c7-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.013 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:56.014 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:56.014 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:09:56.015 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:09:56.015 DEBUG:boto:Method: GET 2022-09-12T20:09:56.016 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:09:56.016 DEBUG:boto:Data: 2022-09-12T20:09:56.017 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.017 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.018 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.018 DEBUG:boto:Params: {} 2022-09-12T20:09:56.019 DEBUG:boto:Token: None 2022-09-12T20:09:56.020 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/ 2022-09-12T20:09:56.020 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE= 2022-09-12T20:09:56.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE=', 'Content-Length': '0'} 2022-09-12T20:09:56.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2a38b5ab3984bac-00631f9214-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.066 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:56.067 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:56.067 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:56.068 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:09:56.068 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:09:56.069 DEBUG:boto:Method: GET 2022-09-12T20:09:56.070 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:09:56.070 DEBUG:boto:Data: 2022-09-12T20:09:56.071 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.071 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.072 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.072 DEBUG:boto:Params: {} 2022-09-12T20:09:56.073 DEBUG:boto:Token: None 2022-09-12T20:09:56.073 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/ 2022-09-12T20:09:56.074 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE= 2022-09-12T20:09:56.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE=', 'Content-Length': '0'} 2022-09-12T20:09:56.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084a69d501aabbcbb-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.079 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:56.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:09:56.080 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:09:56.081 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:09:56.081 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:09:56.082 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:09:56.083 DEBUG:boto:Method: GET 2022-09-12T20:09:56.083 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:09:56.084 DEBUG:boto:Data: 2022-09-12T20:09:56.084 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.085 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.085 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.086 DEBUG:boto:Params: {} 2022-09-12T20:09:56.087 DEBUG:boto:Token: None 2022-09-12T20:09:56.087 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:09:56.088 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z2Ua9mSYHdA2HYYXGog+Rs33ssA= 2022-09-12T20:09:56.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z2Ua9mSYHdA2HYYXGog+Rs33ssA=', 'Content-Length': '0'} 2022-09-12T20:09:56.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e36588a2b04e896-00631f9214-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.091 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:09:56.092 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:09:56.093 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:09:56.093 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:09:56.094 DEBUG:boto:Method: GET 2022-09-12T20:09:56.094 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:09:56.095 DEBUG:boto:Data: 2022-09-12T20:09:56.095 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.096 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.096 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.097 DEBUG:boto:Params: {} 2022-09-12T20:09:56.098 DEBUG:boto:Token: None 2022-09-12T20:09:56.098 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:09:56.099 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z2Ua9mSYHdA2HYYXGog+Rs33ssA= 2022-09-12T20:09:56.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z2Ua9mSYHdA2HYYXGog+Rs33ssA=', 'Content-Length': '0'} 2022-09-12T20:09:56.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c27bc0970b5648a0-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.102 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:09:56.103 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:09:56.103 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:09:56.104 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:09:56.104 DEBUG:boto:Method: HEAD 2022-09-12T20:09:56.105 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:09:56.105 DEBUG:boto:Data: 2022-09-12T20:09:56.106 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.106 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.107 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.107 DEBUG:boto:Params: {} 2022-09-12T20:09:56.108 DEBUG:boto:Token: None 2022-09-12T20:09:56.109 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:09:56.109 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:us1zqziGoY2MoPmJCy2zop0PeQc= 2022-09-12T20:09:56.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:us1zqziGoY2MoPmJCy2zop0PeQc=', 'Content-Length': '0'} 2022-09-12T20:09:56.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c031319f3aa00a96-00631f9214-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.113 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:09:56.113 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:09:56.114 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:09:56.114 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:09:56.115 DEBUG:boto:Method: HEAD 2022-09-12T20:09:56.116 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:09:56.116 DEBUG:boto:Data: 2022-09-12T20:09:56.117 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.117 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.118 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.119 DEBUG:boto:Params: {} 2022-09-12T20:09:56.119 DEBUG:boto:Token: None 2022-09-12T20:09:56.120 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:09:56.121 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:us1zqziGoY2MoPmJCy2zop0PeQc= 2022-09-12T20:09:56.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:us1zqziGoY2MoPmJCy2zop0PeQc=', 'Content-Length': '0'} 2022-09-12T20:09:56.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f9e708df765f799-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:09:56.125 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:09:56.126 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:09:56.127 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:09:56.127 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:09:56.128 DEBUG:boto:Method: GET 2022-09-12T20:09:56.129 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:09:56.129 DEBUG:boto:Data: 2022-09-12T20:09:56.130 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.131 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.131 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.132 DEBUG:boto:Params: {} 2022-09-12T20:09:56.133 DEBUG:boto:Token: None 2022-09-12T20:09:56.134 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:09:56.134 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z2Ua9mSYHdA2HYYXGog+Rs33ssA= 2022-09-12T20:09:56.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z2Ua9mSYHdA2HYYXGog+Rs33ssA=', 'Content-Length': '0'} 2022-09-12T20:09:56.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a8dab836bb9a476-00631f9214-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.138 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:09:56.139 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:09:56.140 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:09:56.140 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:09:56.141 DEBUG:boto:Method: GET 2022-09-12T20:09:56.142 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:09:56.142 DEBUG:boto:Data: 2022-09-12T20:09:56.143 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.143 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.144 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.145 DEBUG:boto:Params: {} 2022-09-12T20:09:56.145 DEBUG:boto:Token: None 2022-09-12T20:09:56.146 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:09:56.147 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z2Ua9mSYHdA2HYYXGog+Rs33ssA= 2022-09-12T20:09:56.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z2Ua9mSYHdA2HYYXGog+Rs33ssA=', 'Content-Length': '0'} 2022-09-12T20:09:56.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063c710fc353d8b6e-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.150 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:56.151 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:56.151 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:09:56.152 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:09:56.153 DEBUG:boto:Method: GET 2022-09-12T20:09:56.153 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:09:56.154 DEBUG:boto:Data: 2022-09-12T20:09:56.154 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.155 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.155 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.156 DEBUG:boto:Params: {} 2022-09-12T20:09:56.157 DEBUG:boto:Token: None 2022-09-12T20:09:56.157 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/ 2022-09-12T20:09:56.158 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE= 2022-09-12T20:09:56.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE=', 'Content-Length': '0'} 2022-09-12T20:09:56.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077137828621f3797-00631f9214-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.162 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:56.163 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:56.164 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:56.164 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:09:56.165 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:09:56.165 DEBUG:boto:Method: GET 2022-09-12T20:09:56.166 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:09:56.166 DEBUG:boto:Data: 2022-09-12T20:09:56.167 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.168 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.168 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.169 DEBUG:boto:Params: {} 2022-09-12T20:09:56.169 DEBUG:boto:Token: None 2022-09-12T20:09:56.170 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/ 2022-09-12T20:09:56.171 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE= 2022-09-12T20:09:56.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE=', 'Content-Length': '0'} 2022-09-12T20:09:56.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009670ea9d44b4cfe2-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.176 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:56.177 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:09:56.178 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:09:56.178 DEBUG:boto:Method: HEAD 2022-09-12T20:09:56.179 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:09:56.180 DEBUG:boto:Data: 2022-09-12T20:09:56.181 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.182 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.182 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.184 DEBUG:boto:Params: {} 2022-09-12T20:09:56.184 DEBUG:boto:Token: None 2022-09-12T20:09:56.185 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj33 2022-09-12T20:09:56.186 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:y1PIR6aF7Av1kEzXAXYjP+ZUxlk= 2022-09-12T20:09:56.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:y1PIR6aF7Av1kEzXAXYjP+ZUxlk=', 'Content-Length': '0'} 2022-09-12T20:09:56.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091406d02f46a6c66-00631f9214-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.190 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:09:56.191 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:09:56.191 DEBUG:boto:Method: HEAD 2022-09-12T20:09:56.192 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:09:56.192 DEBUG:boto:Data: 2022-09-12T20:09:56.193 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.193 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.194 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.194 DEBUG:boto:Params: {} 2022-09-12T20:09:56.195 DEBUG:boto:Token: None 2022-09-12T20:09:56.195 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj33 2022-09-12T20:09:56.196 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:y1PIR6aF7Av1kEzXAXYjP+ZUxlk= 2022-09-12T20:09:56.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:y1PIR6aF7Av1kEzXAXYjP+ZUxlk=', 'Content-Length': '0'} 2022-09-12T20:09:56.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091dbe2cfaea21fa2-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:09:56.200 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:09:56.200 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:09:56.201 DEBUG:boto:Method: GET 2022-09-12T20:09:56.201 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:09:56.202 DEBUG:boto:Data: 2022-09-12T20:09:56.202 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.203 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.203 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.204 DEBUG:boto:Params: {} 2022-09-12T20:09:56.204 DEBUG:boto:Token: None 2022-09-12T20:09:56.205 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj33 2022-09-12T20:09:56.206 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VWiIa7apiqHOVosenW0Hm4Q1sQA= 2022-09-12T20:09:56.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VWiIa7apiqHOVosenW0Hm4Q1sQA=', 'Content-Length': '0'} 2022-09-12T20:09:56.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccbfe62d62453429-00631f9214-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.209 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:09:56.210 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:09:56.210 DEBUG:boto:Method: GET 2022-09-12T20:09:56.211 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:09:56.211 DEBUG:boto:Data: 2022-09-12T20:09:56.212 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.212 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.213 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.213 DEBUG:boto:Params: {} 2022-09-12T20:09:56.214 DEBUG:boto:Token: None 2022-09-12T20:09:56.214 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj33 2022-09-12T20:09:56.215 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VWiIa7apiqHOVosenW0Hm4Q1sQA= 2022-09-12T20:09:56.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VWiIa7apiqHOVosenW0Hm4Q1sQA=', 'Content-Length': '0'} 2022-09-12T20:09:56.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001f883a63d3443a7-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.262 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:56.263 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:56.263 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:09:56.264 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:09:56.264 DEBUG:boto:Method: GET 2022-09-12T20:09:56.265 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:09:56.265 DEBUG:boto:Data: 2022-09-12T20:09:56.266 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.266 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.267 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.267 DEBUG:boto:Params: {} 2022-09-12T20:09:56.268 DEBUG:boto:Token: None 2022-09-12T20:09:56.269 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/ 2022-09-12T20:09:56.269 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE= 2022-09-12T20:09:56.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE=', 'Content-Length': '0'} 2022-09-12T20:09:56.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8405c4211d32ad9-00631f9214-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.273 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:56.274 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:56.275 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:56.275 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:09:56.275 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:09:56.276 DEBUG:boto:Method: GET 2022-09-12T20:09:56.276 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:09:56.277 DEBUG:boto:Data: 2022-09-12T20:09:56.277 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.278 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.278 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.279 DEBUG:boto:Params: {} 2022-09-12T20:09:56.279 DEBUG:boto:Token: None 2022-09-12T20:09:56.280 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/ 2022-09-12T20:09:56.281 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE= 2022-09-12T20:09:56.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE=', 'Content-Length': '0'} 2022-09-12T20:09:56.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc28b2c1b740c6dd-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.285 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:56.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:09:56.287 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:09:56.287 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:09:56.288 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:09:56.288 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:09:56.289 DEBUG:boto:Method: GET 2022-09-12T20:09:56.289 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:09:56.290 DEBUG:boto:Data: 2022-09-12T20:09:56.290 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.291 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.291 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.292 DEBUG:boto:Params: {} 2022-09-12T20:09:56.292 DEBUG:boto:Token: None 2022-09-12T20:09:56.293 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:09:56.294 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cvKZi4oLN5DgxXhD3w/m27lVhl4= 2022-09-12T20:09:56.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cvKZi4oLN5DgxXhD3w/m27lVhl4=', 'Content-Length': '0'} 2022-09-12T20:09:56.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a06522c21e0803ca-00631f9214-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.297 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:09:56.298 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:09:56.298 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:09:56.299 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:09:56.299 DEBUG:boto:Method: GET 2022-09-12T20:09:56.300 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:09:56.300 DEBUG:boto:Data: 2022-09-12T20:09:56.301 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.301 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.302 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.302 DEBUG:boto:Params: {} 2022-09-12T20:09:56.303 DEBUG:boto:Token: None 2022-09-12T20:09:56.303 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:09:56.304 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cvKZi4oLN5DgxXhD3w/m27lVhl4= 2022-09-12T20:09:56.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cvKZi4oLN5DgxXhD3w/m27lVhl4=', 'Content-Length': '0'} 2022-09-12T20:09:56.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b643bb6958a00c6-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.307 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:09:56.308 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:09:56.308 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:09:56.309 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:09:56.309 DEBUG:boto:Method: HEAD 2022-09-12T20:09:56.310 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:09:56.310 DEBUG:boto:Data: 2022-09-12T20:09:56.311 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.311 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.312 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.312 DEBUG:boto:Params: {} 2022-09-12T20:09:56.313 DEBUG:boto:Token: None 2022-09-12T20:09:56.313 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:09:56.314 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X+agfwCTPjHuz+lxdQlIdd1LotQ= 2022-09-12T20:09:56.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X+agfwCTPjHuz+lxdQlIdd1LotQ=', 'Content-Length': '0'} 2022-09-12T20:09:56.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e2596e53315d1e2-00631f9214-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.317 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:09:56.318 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:09:56.323 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:09:56.324 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:09:56.324 DEBUG:boto:Method: HEAD 2022-09-12T20:09:56.325 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:09:56.326 DEBUG:boto:Data: 2022-09-12T20:09:56.326 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.327 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.327 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.328 DEBUG:boto:Params: {} 2022-09-12T20:09:56.328 DEBUG:boto:Token: None 2022-09-12T20:09:56.329 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:09:56.329 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X+agfwCTPjHuz+lxdQlIdd1LotQ= 2022-09-12T20:09:56.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X+agfwCTPjHuz+lxdQlIdd1LotQ=', 'Content-Length': '0'} 2022-09-12T20:09:56.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000182130a1757b0d95-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:09:56.333 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:09:56.334 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:09:56.334 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:09:56.335 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:09:56.335 DEBUG:boto:Method: GET 2022-09-12T20:09:56.336 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:09:56.336 DEBUG:boto:Data: 2022-09-12T20:09:56.337 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.337 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.338 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.338 DEBUG:boto:Params: {} 2022-09-12T20:09:56.339 DEBUG:boto:Token: None 2022-09-12T20:09:56.340 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:09:56.340 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cvKZi4oLN5DgxXhD3w/m27lVhl4= 2022-09-12T20:09:56.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cvKZi4oLN5DgxXhD3w/m27lVhl4=', 'Content-Length': '0'} 2022-09-12T20:09:56.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0291690d2702f55-00631f9214-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.344 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:09:56.344 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:09:56.345 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:09:56.345 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:09:56.346 DEBUG:boto:Method: GET 2022-09-12T20:09:56.346 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:09:56.347 DEBUG:boto:Data: 2022-09-12T20:09:56.347 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.348 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.349 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.349 DEBUG:boto:Params: {} 2022-09-12T20:09:56.350 DEBUG:boto:Token: None 2022-09-12T20:09:56.350 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:09:56.351 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cvKZi4oLN5DgxXhD3w/m27lVhl4= 2022-09-12T20:09:56.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cvKZi4oLN5DgxXhD3w/m27lVhl4=', 'Content-Length': '0'} 2022-09-12T20:09:56.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c62a5cfb75d6eef4-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.354 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:56.355 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:56.355 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:09:56.356 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:09:56.356 DEBUG:boto:Method: GET 2022-09-12T20:09:56.357 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:09:56.357 DEBUG:boto:Data: 2022-09-12T20:09:56.358 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.358 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.359 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.359 DEBUG:boto:Params: {} 2022-09-12T20:09:56.360 DEBUG:boto:Token: None 2022-09-12T20:09:56.361 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/ 2022-09-12T20:09:56.361 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE= 2022-09-12T20:09:56.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE=', 'Content-Length': '0'} 2022-09-12T20:09:56.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba254df465af724d-00631f9214-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.406 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:56.407 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:56.408 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:56.408 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:09:56.409 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:09:56.409 DEBUG:boto:Method: GET 2022-09-12T20:09:56.410 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:09:56.410 DEBUG:boto:Data: 2022-09-12T20:09:56.411 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.411 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.412 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.412 DEBUG:boto:Params: {} 2022-09-12T20:09:56.413 DEBUG:boto:Token: None 2022-09-12T20:09:56.413 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/ 2022-09-12T20:09:56.414 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE= 2022-09-12T20:09:56.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE=', 'Content-Length': '0'} 2022-09-12T20:09:56.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000487f760cae41d612-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.419 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:56.420 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:09:56.420 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:09:56.421 DEBUG:boto:Method: HEAD 2022-09-12T20:09:56.421 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:09:56.422 DEBUG:boto:Data: 2022-09-12T20:09:56.422 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.423 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.423 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.424 DEBUG:boto:Params: {} 2022-09-12T20:09:56.424 DEBUG:boto:Token: None 2022-09-12T20:09:56.425 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj34 2022-09-12T20:09:56.425 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:57zeTa9NcOHIrQHFNwqCbmpq0Ow= 2022-09-12T20:09:56.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:57zeTa9NcOHIrQHFNwqCbmpq0Ow=', 'Content-Length': '0'} 2022-09-12T20:09:56.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d59d81e5d903aa16-00631f9214-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.429 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:09:56.429 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:09:56.430 DEBUG:boto:Method: HEAD 2022-09-12T20:09:56.430 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:09:56.431 DEBUG:boto:Data: 2022-09-12T20:09:56.431 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.432 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.432 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.433 DEBUG:boto:Params: {} 2022-09-12T20:09:56.434 DEBUG:boto:Token: None 2022-09-12T20:09:56.434 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj34 2022-09-12T20:09:56.435 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:57zeTa9NcOHIrQHFNwqCbmpq0Ow= 2022-09-12T20:09:56.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:57zeTa9NcOHIrQHFNwqCbmpq0Ow=', 'Content-Length': '0'} 2022-09-12T20:09:56.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc273c8477532094-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:09:56.438 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:09:56.439 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:09:56.439 DEBUG:boto:Method: GET 2022-09-12T20:09:56.440 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:09:56.440 DEBUG:boto:Data: 2022-09-12T20:09:56.441 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.442 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.442 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.443 DEBUG:boto:Params: {} 2022-09-12T20:09:56.443 DEBUG:boto:Token: None 2022-09-12T20:09:56.444 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj34 2022-09-12T20:09:56.444 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PdTMj7ls8Ys6TL76cSMsj+t/bv8= 2022-09-12T20:09:56.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PdTMj7ls8Ys6TL76cSMsj+t/bv8=', 'Content-Length': '0'} 2022-09-12T20:09:56.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c2374a8a6f44d7e-00631f9214-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.448 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:09:56.448 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:09:56.449 DEBUG:boto:Method: GET 2022-09-12T20:09:56.449 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:09:56.450 DEBUG:boto:Data: 2022-09-12T20:09:56.450 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.451 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.451 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.452 DEBUG:boto:Params: {} 2022-09-12T20:09:56.453 DEBUG:boto:Token: None 2022-09-12T20:09:56.453 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj34 2022-09-12T20:09:56.454 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PdTMj7ls8Ys6TL76cSMsj+t/bv8= 2022-09-12T20:09:56.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PdTMj7ls8Ys6TL76cSMsj+t/bv8=', 'Content-Length': '0'} 2022-09-12T20:09:56.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000679eaab9a53bf8a6-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.498 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:56.499 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:56.499 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:09:56.500 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:09:56.500 DEBUG:boto:Method: GET 2022-09-12T20:09:56.501 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:09:56.501 DEBUG:boto:Data: 2022-09-12T20:09:56.502 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.502 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.503 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.503 DEBUG:boto:Params: {} 2022-09-12T20:09:56.504 DEBUG:boto:Token: None 2022-09-12T20:09:56.505 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/ 2022-09-12T20:09:56.505 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE= 2022-09-12T20:09:56.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE=', 'Content-Length': '0'} 2022-09-12T20:09:56.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efa6615869ec3509-00631f9214-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.509 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:56.510 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:56.511 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:56.511 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:09:56.512 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:09:56.512 DEBUG:boto:Method: GET 2022-09-12T20:09:56.513 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:09:56.513 DEBUG:boto:Data: 2022-09-12T20:09:56.514 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.514 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.515 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.515 DEBUG:boto:Params: {} 2022-09-12T20:09:56.516 DEBUG:boto:Token: None 2022-09-12T20:09:56.516 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/ 2022-09-12T20:09:56.517 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE= 2022-09-12T20:09:56.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE=', 'Content-Length': '0'} 2022-09-12T20:09:56.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000974c618851ff71b1-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.522 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:56.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:09:56.523 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:09:56.524 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:09:56.524 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:09:56.525 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:09:56.525 DEBUG:boto:Method: GET 2022-09-12T20:09:56.526 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:09:56.527 DEBUG:boto:Data: 2022-09-12T20:09:56.527 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.528 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.528 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.529 DEBUG:boto:Params: {} 2022-09-12T20:09:56.529 DEBUG:boto:Token: None 2022-09-12T20:09:56.530 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:09:56.530 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2VneCiW7IpwN6rUDBP5m2iPmVLo= 2022-09-12T20:09:56.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2VneCiW7IpwN6rUDBP5m2iPmVLo=', 'Content-Length': '0'} 2022-09-12T20:09:56.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000972984a6f0e433de-00631f9214-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.534 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:09:56.534 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:09:56.535 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:09:56.535 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:09:56.536 DEBUG:boto:Method: GET 2022-09-12T20:09:56.536 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:09:56.537 DEBUG:boto:Data: 2022-09-12T20:09:56.537 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.538 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.538 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.539 DEBUG:boto:Params: {} 2022-09-12T20:09:56.539 DEBUG:boto:Token: None 2022-09-12T20:09:56.540 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:09:56.541 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2VneCiW7IpwN6rUDBP5m2iPmVLo= 2022-09-12T20:09:56.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2VneCiW7IpwN6rUDBP5m2iPmVLo=', 'Content-Length': '0'} 2022-09-12T20:09:56.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046170123883ccd40-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.544 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:09:56.545 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:09:56.545 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:09:56.546 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:09:56.546 DEBUG:boto:Method: HEAD 2022-09-12T20:09:56.547 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:09:56.547 DEBUG:boto:Data: 2022-09-12T20:09:56.548 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.548 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.549 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.549 DEBUG:boto:Params: {} 2022-09-12T20:09:56.550 DEBUG:boto:Token: None 2022-09-12T20:09:56.550 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:09:56.551 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7HbrE895Xxd1b2NEym7KlgSqpBg= 2022-09-12T20:09:56.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7HbrE895Xxd1b2NEym7KlgSqpBg=', 'Content-Length': '0'} 2022-09-12T20:09:56.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f05e08ab99acd1f-00631f9214-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.554 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:09:56.555 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:09:56.555 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:09:56.556 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:09:56.556 DEBUG:boto:Method: HEAD 2022-09-12T20:09:56.557 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:09:56.557 DEBUG:boto:Data: 2022-09-12T20:09:56.558 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.558 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.559 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.559 DEBUG:boto:Params: {} 2022-09-12T20:09:56.560 DEBUG:boto:Token: None 2022-09-12T20:09:56.560 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:09:56.561 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7HbrE895Xxd1b2NEym7KlgSqpBg= 2022-09-12T20:09:56.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7HbrE895Xxd1b2NEym7KlgSqpBg=', 'Content-Length': '0'} 2022-09-12T20:09:56.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e33965278d5e2f20-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:09:56.565 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:09:56.565 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:09:56.566 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:09:56.566 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:09:56.567 DEBUG:boto:Method: GET 2022-09-12T20:09:56.567 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:09:56.568 DEBUG:boto:Data: 2022-09-12T20:09:56.568 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.569 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.569 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.570 DEBUG:boto:Params: {} 2022-09-12T20:09:56.571 DEBUG:boto:Token: None 2022-09-12T20:09:56.571 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:09:56.572 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2VneCiW7IpwN6rUDBP5m2iPmVLo= 2022-09-12T20:09:56.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2VneCiW7IpwN6rUDBP5m2iPmVLo=', 'Content-Length': '0'} 2022-09-12T20:09:56.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4851ffecddd8311-00631f9214-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.575 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:09:56.576 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:09:56.576 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:09:56.577 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:09:56.577 DEBUG:boto:Method: GET 2022-09-12T20:09:56.578 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:09:56.578 DEBUG:boto:Data: 2022-09-12T20:09:56.579 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.579 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.580 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.580 DEBUG:boto:Params: {} 2022-09-12T20:09:56.581 DEBUG:boto:Token: None 2022-09-12T20:09:56.582 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:09:56.582 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2VneCiW7IpwN6rUDBP5m2iPmVLo= 2022-09-12T20:09:56.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2VneCiW7IpwN6rUDBP5m2iPmVLo=', 'Content-Length': '0'} 2022-09-12T20:09:56.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000135c52fc572787ff-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.586 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:56.586 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:56.587 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:09:56.587 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:09:56.588 DEBUG:boto:Method: GET 2022-09-12T20:09:56.588 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:09:56.589 DEBUG:boto:Data: 2022-09-12T20:09:56.589 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.590 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.590 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.591 DEBUG:boto:Params: {} 2022-09-12T20:09:56.591 DEBUG:boto:Token: None 2022-09-12T20:09:56.592 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/ 2022-09-12T20:09:56.593 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE= 2022-09-12T20:09:56.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE=', 'Content-Length': '0'} 2022-09-12T20:09:56.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d4dda936f5c617f-00631f9214-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.597 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:56.597 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:56.598 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:56.598 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:09:56.599 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:09:56.599 DEBUG:boto:Method: GET 2022-09-12T20:09:56.600 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:09:56.600 DEBUG:boto:Data: 2022-09-12T20:09:56.601 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.602 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.602 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.603 DEBUG:boto:Params: {} 2022-09-12T20:09:56.603 DEBUG:boto:Token: None 2022-09-12T20:09:56.604 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/ 2022-09-12T20:09:56.604 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE= 2022-09-12T20:09:56.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE=', 'Content-Length': '0'} 2022-09-12T20:09:56.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb7b6cb77a52b4c0-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.609 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:56.610 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:09:56.610 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:09:56.611 DEBUG:boto:Method: HEAD 2022-09-12T20:09:56.611 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:09:56.612 DEBUG:boto:Data: 2022-09-12T20:09:56.612 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.613 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.613 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.614 DEBUG:boto:Params: {} 2022-09-12T20:09:56.615 DEBUG:boto:Token: None 2022-09-12T20:09:56.615 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj38 2022-09-12T20:09:56.616 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DxvtTMklsJuKsc507PnA3uW5A24= 2022-09-12T20:09:56.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DxvtTMklsJuKsc507PnA3uW5A24=', 'Content-Length': '0'} 2022-09-12T20:09:56.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ef89d476c59d10e-00631f9214-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.619 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:09:56.619 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:09:56.620 DEBUG:boto:Method: HEAD 2022-09-12T20:09:56.620 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:09:56.621 DEBUG:boto:Data: 2022-09-12T20:09:56.621 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.622 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.622 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.623 DEBUG:boto:Params: {} 2022-09-12T20:09:56.623 DEBUG:boto:Token: None 2022-09-12T20:09:56.624 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj38 2022-09-12T20:09:56.624 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DxvtTMklsJuKsc507PnA3uW5A24= 2022-09-12T20:09:56.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DxvtTMklsJuKsc507PnA3uW5A24=', 'Content-Length': '0'} 2022-09-12T20:09:56.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1fa111fe9101d97-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:09:56.629 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:09:56.630 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:09:56.630 DEBUG:boto:Method: GET 2022-09-12T20:09:56.631 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:09:56.631 DEBUG:boto:Data: 2022-09-12T20:09:56.632 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.632 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.633 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.633 DEBUG:boto:Params: {} 2022-09-12T20:09:56.634 DEBUG:boto:Token: None 2022-09-12T20:09:56.635 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj38 2022-09-12T20:09:56.635 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KVChUD8xsoDWmCNxzaV2fOs47KE= 2022-09-12T20:09:56.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KVChUD8xsoDWmCNxzaV2fOs47KE=', 'Content-Length': '0'} 2022-09-12T20:09:56.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091b6d8372f8105e7-00631f9214-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.682 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:09:56.683 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:09:56.683 DEBUG:boto:Method: GET 2022-09-12T20:09:56.684 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:09:56.684 DEBUG:boto:Data: 2022-09-12T20:09:56.685 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.685 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.686 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.686 DEBUG:boto:Params: {} 2022-09-12T20:09:56.687 DEBUG:boto:Token: None 2022-09-12T20:09:56.688 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj38 2022-09-12T20:09:56.688 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KVChUD8xsoDWmCNxzaV2fOs47KE= 2022-09-12T20:09:56.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KVChUD8xsoDWmCNxzaV2fOs47KE=', 'Content-Length': '0'} 2022-09-12T20:09:56.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e27e0cca78ff0c7e-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.692 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:56.692 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:56.693 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:09:56.693 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:09:56.694 DEBUG:boto:Method: GET 2022-09-12T20:09:56.694 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:09:56.695 DEBUG:boto:Data: 2022-09-12T20:09:56.696 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.696 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.697 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.697 DEBUG:boto:Params: {} 2022-09-12T20:09:56.698 DEBUG:boto:Token: None 2022-09-12T20:09:56.698 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/ 2022-09-12T20:09:56.699 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE= 2022-09-12T20:09:56.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE=', 'Content-Length': '0'} 2022-09-12T20:09:56.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c4f7021d5582e84-00631f9214-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.704 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:56.705 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:56.705 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:56.706 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:09:56.706 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:09:56.707 DEBUG:boto:Method: GET 2022-09-12T20:09:56.707 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:09:56.708 DEBUG:boto:Data: 2022-09-12T20:09:56.708 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.709 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.709 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.710 DEBUG:boto:Params: {} 2022-09-12T20:09:56.711 DEBUG:boto:Token: None 2022-09-12T20:09:56.711 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/ 2022-09-12T20:09:56.712 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE= 2022-09-12T20:09:56.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE=', 'Content-Length': '0'} 2022-09-12T20:09:56.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ef20304498194c9-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.758 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:56.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:09:56.759 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:09:56.760 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:09:56.760 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:09:56.761 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:09:56.761 DEBUG:boto:Method: GET 2022-09-12T20:09:56.762 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:09:56.762 DEBUG:boto:Data: 2022-09-12T20:09:56.763 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.763 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.764 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.765 DEBUG:boto:Params: {} 2022-09-12T20:09:56.765 DEBUG:boto:Token: None 2022-09-12T20:09:56.766 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:09:56.766 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CgV8tmGOby2zyQKTapcbx3wVTzw= 2022-09-12T20:09:56.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CgV8tmGOby2zyQKTapcbx3wVTzw=', 'Content-Length': '0'} 2022-09-12T20:09:56.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faad7568b93c7e9e-00631f9214-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.770 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:09:56.771 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:09:56.771 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:09:56.772 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:09:56.772 DEBUG:boto:Method: GET 2022-09-12T20:09:56.773 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:09:56.773 DEBUG:boto:Data: 2022-09-12T20:09:56.774 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.775 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.775 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.776 DEBUG:boto:Params: {} 2022-09-12T20:09:56.776 DEBUG:boto:Token: None 2022-09-12T20:09:56.777 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:09:56.778 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CgV8tmGOby2zyQKTapcbx3wVTzw= 2022-09-12T20:09:56.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CgV8tmGOby2zyQKTapcbx3wVTzw=', 'Content-Length': '0'} 2022-09-12T20:09:56.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e7cfacb4477023f-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.781 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:09:56.782 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:09:56.783 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:09:56.783 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:09:56.784 DEBUG:boto:Method: HEAD 2022-09-12T20:09:56.784 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:09:56.785 DEBUG:boto:Data: 2022-09-12T20:09:56.785 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.786 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.786 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.787 DEBUG:boto:Params: {} 2022-09-12T20:09:56.788 DEBUG:boto:Token: None 2022-09-12T20:09:56.788 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:09:56.789 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oxzwaUF9LM3Ws7jguYOkOwIRgxk= 2022-09-12T20:09:56.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oxzwaUF9LM3Ws7jguYOkOwIRgxk=', 'Content-Length': '0'} 2022-09-12T20:09:56.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f0f2adffd53c8ba-00631f9214-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.792 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:09:56.793 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:09:56.793 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:09:56.794 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:09:56.795 DEBUG:boto:Method: HEAD 2022-09-12T20:09:56.795 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:09:56.796 DEBUG:boto:Data: 2022-09-12T20:09:56.796 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.797 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.797 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.798 DEBUG:boto:Params: {} 2022-09-12T20:09:56.798 DEBUG:boto:Token: None 2022-09-12T20:09:56.799 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:09:56.799 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oxzwaUF9LM3Ws7jguYOkOwIRgxk= 2022-09-12T20:09:56.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oxzwaUF9LM3Ws7jguYOkOwIRgxk=', 'Content-Length': '0'} 2022-09-12T20:09:56.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000213f3c7d783c1887-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:09:56.803 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:09:56.804 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:09:56.805 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:09:56.805 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:09:56.806 DEBUG:boto:Method: GET 2022-09-12T20:09:56.806 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:09:56.807 DEBUG:boto:Data: 2022-09-12T20:09:56.808 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.808 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.809 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.809 DEBUG:boto:Params: {} 2022-09-12T20:09:56.810 DEBUG:boto:Token: None 2022-09-12T20:09:56.811 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:09:56.811 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CgV8tmGOby2zyQKTapcbx3wVTzw= 2022-09-12T20:09:56.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CgV8tmGOby2zyQKTapcbx3wVTzw=', 'Content-Length': '0'} 2022-09-12T20:09:56.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac2370cbe4f9bcba-00631f9214-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.815 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:09:56.815 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:09:56.816 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:09:56.816 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:09:56.817 DEBUG:boto:Method: GET 2022-09-12T20:09:56.817 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:09:56.818 DEBUG:boto:Data: 2022-09-12T20:09:56.819 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.819 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.820 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.820 DEBUG:boto:Params: {} 2022-09-12T20:09:56.821 DEBUG:boto:Token: None 2022-09-12T20:09:56.822 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:09:56.822 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CgV8tmGOby2zyQKTapcbx3wVTzw= 2022-09-12T20:09:56.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CgV8tmGOby2zyQKTapcbx3wVTzw=', 'Content-Length': '0'} 2022-09-12T20:09:56.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041a80d7b797b6224-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.826 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:56.826 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:56.827 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:09:56.827 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:09:56.828 DEBUG:boto:Method: GET 2022-09-12T20:09:56.828 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:09:56.829 DEBUG:boto:Data: 2022-09-12T20:09:56.829 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.830 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.830 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.831 DEBUG:boto:Params: {} 2022-09-12T20:09:56.832 DEBUG:boto:Token: None 2022-09-12T20:09:56.832 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/ 2022-09-12T20:09:56.833 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE= 2022-09-12T20:09:56.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE=', 'Content-Length': '0'} 2022-09-12T20:09:56.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3167ed3a769981c-00631f9214-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.837 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:56.838 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:56.838 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:56.839 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:09:56.839 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:09:56.840 DEBUG:boto:Method: GET 2022-09-12T20:09:56.840 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:09:56.841 DEBUG:boto:Data: 2022-09-12T20:09:56.842 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.842 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.843 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.843 DEBUG:boto:Params: {} 2022-09-12T20:09:56.844 DEBUG:boto:Token: None 2022-09-12T20:09:56.845 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/ 2022-09-12T20:09:56.845 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE= 2022-09-12T20:09:56.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE=', 'Content-Length': '0'} 2022-09-12T20:09:56.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000965cbf576cbb7b91-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.850 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:56.851 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:09:56.852 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:09:56.852 DEBUG:boto:Method: HEAD 2022-09-12T20:09:56.853 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:09:56.853 DEBUG:boto:Data: 2022-09-12T20:09:56.854 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.854 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.855 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.855 DEBUG:boto:Params: {} 2022-09-12T20:09:56.856 DEBUG:boto:Token: None 2022-09-12T20:09:56.857 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj44 2022-09-12T20:09:56.857 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CJ9APEyv7OFOmo3lKxA+ZOZpQSc= 2022-09-12T20:09:56.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CJ9APEyv7OFOmo3lKxA+ZOZpQSc=', 'Content-Length': '0'} 2022-09-12T20:09:56.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb88094b3217d468-00631f9214-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.861 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:09:56.861 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:09:56.862 DEBUG:boto:Method: HEAD 2022-09-12T20:09:56.862 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:09:56.863 DEBUG:boto:Data: 2022-09-12T20:09:56.864 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.864 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.865 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.865 DEBUG:boto:Params: {} 2022-09-12T20:09:56.866 DEBUG:boto:Token: None 2022-09-12T20:09:56.866 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj44 2022-09-12T20:09:56.867 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CJ9APEyv7OFOmo3lKxA+ZOZpQSc= 2022-09-12T20:09:56.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CJ9APEyv7OFOmo3lKxA+ZOZpQSc=', 'Content-Length': '0'} 2022-09-12T20:09:56.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ab11765362ac567-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:09:56.871 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:09:56.872 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:09:56.872 DEBUG:boto:Method: GET 2022-09-12T20:09:56.873 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:09:56.874 DEBUG:boto:Data: 2022-09-12T20:09:56.874 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.875 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.875 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.876 DEBUG:boto:Params: {} 2022-09-12T20:09:56.876 DEBUG:boto:Token: None 2022-09-12T20:09:56.877 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj44 2022-09-12T20:09:56.878 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SzQBv9lfXKbQIvk9r9GF7RPBXII= 2022-09-12T20:09:56.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SzQBv9lfXKbQIvk9r9GF7RPBXII=', 'Content-Length': '0'} 2022-09-12T20:09:56.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbd3ec295d1ee20f-00631f9214-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.881 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:09:56.882 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:09:56.882 DEBUG:boto:Method: GET 2022-09-12T20:09:56.883 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:09:56.883 DEBUG:boto:Data: 2022-09-12T20:09:56.884 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.885 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.885 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.886 DEBUG:boto:Params: {} 2022-09-12T20:09:56.886 DEBUG:boto:Token: None 2022-09-12T20:09:56.887 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj44 2022-09-12T20:09:56.888 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SzQBv9lfXKbQIvk9r9GF7RPBXII= 2022-09-12T20:09:56.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SzQBv9lfXKbQIvk9r9GF7RPBXII=', 'Content-Length': '0'} 2022-09-12T20:09:56.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023aa9272183f52d5-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.934 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:56.935 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:56.935 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:09:56.936 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:09:56.936 DEBUG:boto:Method: GET 2022-09-12T20:09:56.937 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:09:56.937 DEBUG:boto:Data: 2022-09-12T20:09:56.938 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.938 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.939 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.940 DEBUG:boto:Params: {} 2022-09-12T20:09:56.940 DEBUG:boto:Token: None 2022-09-12T20:09:56.941 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/ 2022-09-12T20:09:56.941 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE= 2022-09-12T20:09:56.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE=', 'Content-Length': '0'} 2022-09-12T20:09:56.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000588abcc916b6ce84-00631f9214-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.946 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:56.947 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:56.947 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:56.948 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:09:56.948 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:09:56.949 DEBUG:boto:Method: GET 2022-09-12T20:09:56.949 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:09:56.950 DEBUG:boto:Data: 2022-09-12T20:09:56.950 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.951 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.952 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.952 DEBUG:boto:Params: {} 2022-09-12T20:09:56.953 DEBUG:boto:Token: None 2022-09-12T20:09:56.953 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/ 2022-09-12T20:09:56.954 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE= 2022-09-12T20:09:56.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QgR/zVwx24fcLzWvapsqkmWGcSE=', 'Content-Length': '0'} 2022-09-12T20:09:56.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002203cfc5486c8421-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.959 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:56.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:09:56.960 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:09:56.961 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:09:56.961 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:09:56.962 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:09:56.962 DEBUG:boto:Method: GET 2022-09-12T20:09:56.963 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:09:56.963 DEBUG:boto:Data: 2022-09-12T20:09:56.964 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.964 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.965 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.965 DEBUG:boto:Params: {} 2022-09-12T20:09:56.966 DEBUG:boto:Token: None 2022-09-12T20:09:56.967 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:09:56.967 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h8J3NLlwO0wu2CD2LJ22ZwEmPHk= 2022-09-12T20:09:56.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h8J3NLlwO0wu2CD2LJ22ZwEmPHk=', 'Content-Length': '0'} 2022-09-12T20:09:56.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090f9eeac4d63a39d-00631f9214-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.971 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:09:56.971 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:09:56.972 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:09:56.972 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:09:56.973 DEBUG:boto:Method: GET 2022-09-12T20:09:56.974 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:09:56.974 DEBUG:boto:Data: 2022-09-12T20:09:56.975 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.975 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.976 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.976 DEBUG:boto:Params: {} 2022-09-12T20:09:56.977 DEBUG:boto:Token: None 2022-09-12T20:09:56.977 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:09:56.978 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:h8J3NLlwO0wu2CD2LJ22ZwEmPHk= 2022-09-12T20:09:56.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:h8J3NLlwO0wu2CD2LJ22ZwEmPHk=', 'Content-Length': '0'} 2022-09-12T20:09:56.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f776aa114168da5-00631f9214-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.982 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:09:56.982 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:09:56.983 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:09:56.983 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:09:56.984 DEBUG:boto:Method: HEAD 2022-09-12T20:09:56.984 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:09:56.985 DEBUG:boto:Data: 2022-09-12T20:09:56.985 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.986 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:56.986 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.987 DEBUG:boto:Params: {} 2022-09-12T20:09:56.987 DEBUG:boto:Token: None 2022-09-12T20:09:56.988 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:09:56.988 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YiKEXPj7Mfo3n8CfFt0YHGzNEeo= 2022-09-12T20:09:56.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YiKEXPj7Mfo3n8CfFt0YHGzNEeo=', 'Content-Length': '0'} 2022-09-12T20:09:56.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089f72d435d9ef912-00631f9214-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:56.992 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:09:56.992 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:09:56.993 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:09:56.993 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:09:56.994 DEBUG:boto:Method: HEAD 2022-09-12T20:09:56.994 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:09:56.995 DEBUG:boto:Data: 2022-09-12T20:09:56.995 DEBUG:boto:Headers: {} 2022-09-12T20:09:56.996 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:56.996 DEBUG:boto:Port: 8000 2022-09-12T20:09:56.997 DEBUG:boto:Params: {} 2022-09-12T20:09:56.997 DEBUG:boto:Token: None 2022-09-12T20:09:56.998 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:56 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:09:56.999 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YiKEXPj7Mfo3n8CfFt0YHGzNEeo= 2022-09-12T20:09:56.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:56 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YiKEXPj7Mfo3n8CfFt0YHGzNEeo=', 'Content-Length': '0'} 2022-09-12T20:09:57.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2ad7aa26be95ef0-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:09:57.003 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:09:57.003 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:09:57.004 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:09:57.004 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:09:57.005 DEBUG:boto:Method: GET 2022-09-12T20:09:57.005 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:09:57.006 DEBUG:boto:Data: 2022-09-12T20:09:57.006 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.007 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.007 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.008 DEBUG:boto:Params: {} 2022-09-12T20:09:57.008 DEBUG:boto:Token: None 2022-09-12T20:09:57.009 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:09:57.010 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WuSjB+Wfr/FSePXhBxLc732QOe0= 2022-09-12T20:09:57.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WuSjB+Wfr/FSePXhBxLc732QOe0=', 'Content-Length': '0'} 2022-09-12T20:09:57.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000420b3e43581d92b4-00631f9215-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.054 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:09:57.054 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:09:57.055 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:09:57.056 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:09:57.056 DEBUG:boto:Method: GET 2022-09-12T20:09:57.057 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:09:57.057 DEBUG:boto:Data: 2022-09-12T20:09:57.058 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.058 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.059 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.059 DEBUG:boto:Params: {} 2022-09-12T20:09:57.060 DEBUG:boto:Token: None 2022-09-12T20:09:57.061 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:09:57.061 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WuSjB+Wfr/FSePXhBxLc732QOe0= 2022-09-12T20:09:57.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WuSjB+Wfr/FSePXhBxLc732QOe0=', 'Content-Length': '0'} 2022-09-12T20:09:57.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072843c2e4506eea6-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.065 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:57.065 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:57.066 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:09:57.066 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:09:57.067 DEBUG:boto:Method: GET 2022-09-12T20:09:57.068 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:09:57.068 DEBUG:boto:Data: 2022-09-12T20:09:57.069 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.069 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.070 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.070 DEBUG:boto:Params: {} 2022-09-12T20:09:57.071 DEBUG:boto:Token: None 2022-09-12T20:09:57.071 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/ 2022-09-12T20:09:57.072 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw= 2022-09-12T20:09:57.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw=', 'Content-Length': '0'} 2022-09-12T20:09:57.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028b7a12cbc8f096c-00631f9215-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.076 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:57.077 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:57.077 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:57.078 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:09:57.078 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:09:57.079 DEBUG:boto:Method: GET 2022-09-12T20:09:57.079 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:09:57.080 DEBUG:boto:Data: 2022-09-12T20:09:57.081 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.081 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.082 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.082 DEBUG:boto:Params: {} 2022-09-12T20:09:57.083 DEBUG:boto:Token: None 2022-09-12T20:09:57.084 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/ 2022-09-12T20:09:57.084 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw= 2022-09-12T20:09:57.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw=', 'Content-Length': '0'} 2022-09-12T20:09:57.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b4269af3d4766c8-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.089 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:57.090 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:09:57.091 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:09:57.091 DEBUG:boto:Method: HEAD 2022-09-12T20:09:57.092 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:09:57.092 DEBUG:boto:Data: 2022-09-12T20:09:57.093 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.093 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.094 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.094 DEBUG:boto:Params: {} 2022-09-12T20:09:57.095 DEBUG:boto:Token: None 2022-09-12T20:09:57.095 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj46 2022-09-12T20:09:57.096 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3oUItcnX2WolFeCJDNPwxW5FLFM= 2022-09-12T20:09:57.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3oUItcnX2WolFeCJDNPwxW5FLFM=', 'Content-Length': '0'} 2022-09-12T20:09:57.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f33b2b15962cd0d-00631f9215-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.100 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:09:57.100 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:09:57.101 DEBUG:boto:Method: HEAD 2022-09-12T20:09:57.101 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:09:57.102 DEBUG:boto:Data: 2022-09-12T20:09:57.102 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.103 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.103 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.104 DEBUG:boto:Params: {} 2022-09-12T20:09:57.104 DEBUG:boto:Token: None 2022-09-12T20:09:57.105 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj46 2022-09-12T20:09:57.105 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3oUItcnX2WolFeCJDNPwxW5FLFM= 2022-09-12T20:09:57.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3oUItcnX2WolFeCJDNPwxW5FLFM=', 'Content-Length': '0'} 2022-09-12T20:09:57.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbd2ac97593adc43-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:09:57.109 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:09:57.110 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:09:57.110 DEBUG:boto:Method: GET 2022-09-12T20:09:57.111 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:09:57.111 DEBUG:boto:Data: 2022-09-12T20:09:57.112 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.112 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.113 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.113 DEBUG:boto:Params: {} 2022-09-12T20:09:57.114 DEBUG:boto:Token: None 2022-09-12T20:09:57.115 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj46 2022-09-12T20:09:57.115 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m2G+/MALq9GfSNh5+huQMgivLd4= 2022-09-12T20:09:57.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m2G+/MALq9GfSNh5+huQMgivLd4=', 'Content-Length': '0'} 2022-09-12T20:09:57.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dec1d3579330946b-00631f9215-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.162 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:09:57.163 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:09:57.163 DEBUG:boto:Method: GET 2022-09-12T20:09:57.164 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:09:57.164 DEBUG:boto:Data: 2022-09-12T20:09:57.165 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.165 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.166 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.166 DEBUG:boto:Params: {} 2022-09-12T20:09:57.167 DEBUG:boto:Token: None 2022-09-12T20:09:57.168 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj46 2022-09-12T20:09:57.168 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m2G+/MALq9GfSNh5+huQMgivLd4= 2022-09-12T20:09:57.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m2G+/MALq9GfSNh5+huQMgivLd4=', 'Content-Length': '0'} 2022-09-12T20:09:57.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036c3da260d3f3c74-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.172 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:57.173 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:57.173 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:09:57.174 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:09:57.174 DEBUG:boto:Method: GET 2022-09-12T20:09:57.175 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:09:57.175 DEBUG:boto:Data: 2022-09-12T20:09:57.176 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.176 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.177 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.177 DEBUG:boto:Params: {} 2022-09-12T20:09:57.178 DEBUG:boto:Token: None 2022-09-12T20:09:57.178 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/ 2022-09-12T20:09:57.179 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw= 2022-09-12T20:09:57.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw=', 'Content-Length': '0'} 2022-09-12T20:09:57.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000233870940fecee34-00631f9215-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.183 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:57.184 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:57.184 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:57.185 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:09:57.185 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:09:57.186 DEBUG:boto:Method: GET 2022-09-12T20:09:57.187 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:09:57.187 DEBUG:boto:Data: 2022-09-12T20:09:57.188 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.188 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.189 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.189 DEBUG:boto:Params: {} 2022-09-12T20:09:57.190 DEBUG:boto:Token: None 2022-09-12T20:09:57.190 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/ 2022-09-12T20:09:57.191 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw= 2022-09-12T20:09:57.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw=', 'Content-Length': '0'} 2022-09-12T20:09:57.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e8a127efb61bd47-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.196 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:57.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:09:57.197 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:09:57.198 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:09:57.199 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:09:57.199 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:09:57.200 DEBUG:boto:Method: GET 2022-09-12T20:09:57.200 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:09:57.201 DEBUG:boto:Data: 2022-09-12T20:09:57.201 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.202 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.202 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.203 DEBUG:boto:Params: {} 2022-09-12T20:09:57.203 DEBUG:boto:Token: None 2022-09-12T20:09:57.204 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:09:57.205 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1oXKyi2bWD7LtBwnIQnGRnQM5sM= 2022-09-12T20:09:57.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1oXKyi2bWD7LtBwnIQnGRnQM5sM=', 'Content-Length': '0'} 2022-09-12T20:09:57.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e46d0099cedb202-00631f9215-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.208 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:09:57.208 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:09:57.209 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:09:57.210 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:09:57.210 DEBUG:boto:Method: GET 2022-09-12T20:09:57.211 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:09:57.211 DEBUG:boto:Data: 2022-09-12T20:09:57.212 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.212 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.213 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.213 DEBUG:boto:Params: {} 2022-09-12T20:09:57.214 DEBUG:boto:Token: None 2022-09-12T20:09:57.215 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:09:57.215 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1oXKyi2bWD7LtBwnIQnGRnQM5sM= 2022-09-12T20:09:57.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1oXKyi2bWD7LtBwnIQnGRnQM5sM=', 'Content-Length': '0'} 2022-09-12T20:09:57.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002291d57a1e65ca05-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.219 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:09:57.219 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:09:57.220 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:09:57.220 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:09:57.221 DEBUG:boto:Method: HEAD 2022-09-12T20:09:57.221 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:09:57.222 DEBUG:boto:Data: 2022-09-12T20:09:57.222 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.223 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.223 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.224 DEBUG:boto:Params: {} 2022-09-12T20:09:57.224 DEBUG:boto:Token: None 2022-09-12T20:09:57.225 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:09:57.225 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cZUx6r/r75zh+ppLheXgFY7Bad4= 2022-09-12T20:09:57.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cZUx6r/r75zh+ppLheXgFY7Bad4=', 'Content-Length': '0'} 2022-09-12T20:09:57.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e27a7fcba6d1cdc4-00631f9215-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.229 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:09:57.229 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:09:57.230 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:09:57.230 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:09:57.231 DEBUG:boto:Method: HEAD 2022-09-12T20:09:57.231 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:09:57.232 DEBUG:boto:Data: 2022-09-12T20:09:57.232 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.233 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.233 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.234 DEBUG:boto:Params: {} 2022-09-12T20:09:57.235 DEBUG:boto:Token: None 2022-09-12T20:09:57.235 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:09:57.236 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cZUx6r/r75zh+ppLheXgFY7Bad4= 2022-09-12T20:09:57.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cZUx6r/r75zh+ppLheXgFY7Bad4=', 'Content-Length': '0'} 2022-09-12T20:09:57.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9d604581c1a0b6d-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:09:57.240 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:09:57.240 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:09:57.241 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:09:57.241 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:09:57.242 DEBUG:boto:Method: GET 2022-09-12T20:09:57.242 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:09:57.243 DEBUG:boto:Data: 2022-09-12T20:09:57.243 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.244 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.244 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.245 DEBUG:boto:Params: {} 2022-09-12T20:09:57.245 DEBUG:boto:Token: None 2022-09-12T20:09:57.246 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:09:57.247 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1oXKyi2bWD7LtBwnIQnGRnQM5sM= 2022-09-12T20:09:57.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1oXKyi2bWD7LtBwnIQnGRnQM5sM=', 'Content-Length': '0'} 2022-09-12T20:09:57.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006839877b90f70f59-00631f9215-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.250 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:09:57.251 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:09:57.251 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:09:57.252 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:09:57.252 DEBUG:boto:Method: GET 2022-09-12T20:09:57.253 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:09:57.253 DEBUG:boto:Data: 2022-09-12T20:09:57.254 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.254 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.255 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.255 DEBUG:boto:Params: {} 2022-09-12T20:09:57.256 DEBUG:boto:Token: None 2022-09-12T20:09:57.257 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:09:57.257 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1oXKyi2bWD7LtBwnIQnGRnQM5sM= 2022-09-12T20:09:57.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1oXKyi2bWD7LtBwnIQnGRnQM5sM=', 'Content-Length': '0'} 2022-09-12T20:09:57.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001f5d0865ac31c64-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.302 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:57.303 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:57.303 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:09:57.304 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:09:57.304 DEBUG:boto:Method: GET 2022-09-12T20:09:57.305 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:09:57.305 DEBUG:boto:Data: 2022-09-12T20:09:57.306 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.306 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.307 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.307 DEBUG:boto:Params: {} 2022-09-12T20:09:57.308 DEBUG:boto:Token: None 2022-09-12T20:09:57.309 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/ 2022-09-12T20:09:57.309 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw= 2022-09-12T20:09:57.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw=', 'Content-Length': '0'} 2022-09-12T20:09:57.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000137b5d4b4266eff5-00631f9215-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.313 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:57.314 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:57.315 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:57.315 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:09:57.316 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:09:57.316 DEBUG:boto:Method: GET 2022-09-12T20:09:57.317 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:09:57.317 DEBUG:boto:Data: 2022-09-12T20:09:57.318 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.318 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.319 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.319 DEBUG:boto:Params: {} 2022-09-12T20:09:57.320 DEBUG:boto:Token: None 2022-09-12T20:09:57.320 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/ 2022-09-12T20:09:57.321 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw= 2022-09-12T20:09:57.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw=', 'Content-Length': '0'} 2022-09-12T20:09:57.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f9a6287030be1ad-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.326 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:57.327 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:09:57.327 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:09:57.328 DEBUG:boto:Method: HEAD 2022-09-12T20:09:57.328 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:09:57.329 DEBUG:boto:Data: 2022-09-12T20:09:57.329 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.330 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.330 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.331 DEBUG:boto:Params: {} 2022-09-12T20:09:57.332 DEBUG:boto:Token: None 2022-09-12T20:09:57.332 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj51 2022-09-12T20:09:57.333 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PLAy0JvCUKia1TmYN/6TsCg2fcs= 2022-09-12T20:09:57.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PLAy0JvCUKia1TmYN/6TsCg2fcs=', 'Content-Length': '0'} 2022-09-12T20:09:57.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071e27c945139bd7c-00631f9215-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.336 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:09:57.337 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:09:57.337 DEBUG:boto:Method: HEAD 2022-09-12T20:09:57.338 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:09:57.338 DEBUG:boto:Data: 2022-09-12T20:09:57.339 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.339 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.340 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.340 DEBUG:boto:Params: {} 2022-09-12T20:09:57.341 DEBUG:boto:Token: None 2022-09-12T20:09:57.341 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj51 2022-09-12T20:09:57.342 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PLAy0JvCUKia1TmYN/6TsCg2fcs= 2022-09-12T20:09:57.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PLAy0JvCUKia1TmYN/6TsCg2fcs=', 'Content-Length': '0'} 2022-09-12T20:09:57.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee3f9f476ce967af-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:09:57.346 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:09:57.346 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:09:57.347 DEBUG:boto:Method: GET 2022-09-12T20:09:57.348 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:09:57.348 DEBUG:boto:Data: 2022-09-12T20:09:57.349 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.349 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.350 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.350 DEBUG:boto:Params: {} 2022-09-12T20:09:57.351 DEBUG:boto:Token: None 2022-09-12T20:09:57.351 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj51 2022-09-12T20:09:57.352 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XHIBmyTwbX7YGjk1V2zUcHqvY70= 2022-09-12T20:09:57.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XHIBmyTwbX7YGjk1V2zUcHqvY70=', 'Content-Length': '0'} 2022-09-12T20:09:57.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c180b534cbf195a5-00631f9215-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.355 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:09:57.356 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:09:57.356 DEBUG:boto:Method: GET 2022-09-12T20:09:57.357 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:09:57.357 DEBUG:boto:Data: 2022-09-12T20:09:57.358 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.358 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.359 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.359 DEBUG:boto:Params: {} 2022-09-12T20:09:57.360 DEBUG:boto:Token: None 2022-09-12T20:09:57.361 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj51 2022-09-12T20:09:57.361 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XHIBmyTwbX7YGjk1V2zUcHqvY70= 2022-09-12T20:09:57.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XHIBmyTwbX7YGjk1V2zUcHqvY70=', 'Content-Length': '0'} 2022-09-12T20:09:57.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092492cb5bb0a7034-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.365 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:57.365 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:57.366 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:09:57.366 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:09:57.367 DEBUG:boto:Method: GET 2022-09-12T20:09:57.367 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:09:57.368 DEBUG:boto:Data: 2022-09-12T20:09:57.368 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.369 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.369 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.370 DEBUG:boto:Params: {} 2022-09-12T20:09:57.371 DEBUG:boto:Token: None 2022-09-12T20:09:57.371 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/ 2022-09-12T20:09:57.372 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw= 2022-09-12T20:09:57.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw=', 'Content-Length': '0'} 2022-09-12T20:09:57.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089e375b5c0fe3ff1-00631f9215-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.418 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:57.419 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:57.420 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:57.420 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:09:57.421 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:09:57.421 DEBUG:boto:Method: GET 2022-09-12T20:09:57.422 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:09:57.422 DEBUG:boto:Data: 2022-09-12T20:09:57.423 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.424 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.424 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.425 DEBUG:boto:Params: {} 2022-09-12T20:09:57.425 DEBUG:boto:Token: None 2022-09-12T20:09:57.426 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/ 2022-09-12T20:09:57.426 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw= 2022-09-12T20:09:57.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw=', 'Content-Length': '0'} 2022-09-12T20:09:57.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aef1b12e3f8a81ee-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.432 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:57.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:09:57.433 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:09:57.434 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:09:57.435 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:09:57.435 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:09:57.436 DEBUG:boto:Method: GET 2022-09-12T20:09:57.436 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:09:57.437 DEBUG:boto:Data: 2022-09-12T20:09:57.437 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.438 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.438 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.439 DEBUG:boto:Params: {} 2022-09-12T20:09:57.439 DEBUG:boto:Token: None 2022-09-12T20:09:57.440 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:09:57.441 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qN8NNx8p5bX8YHo29S94NXhZv5g= 2022-09-12T20:09:57.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qN8NNx8p5bX8YHo29S94NXhZv5g=', 'Content-Length': '0'} 2022-09-12T20:09:57.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1a25d8bced01fce-00631f9215-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.444 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:09:57.445 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:09:57.445 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:09:57.446 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:09:57.447 DEBUG:boto:Method: GET 2022-09-12T20:09:57.447 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:09:57.448 DEBUG:boto:Data: 2022-09-12T20:09:57.448 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.449 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.449 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.450 DEBUG:boto:Params: {} 2022-09-12T20:09:57.450 DEBUG:boto:Token: None 2022-09-12T20:09:57.451 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:09:57.452 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qN8NNx8p5bX8YHo29S94NXhZv5g= 2022-09-12T20:09:57.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qN8NNx8p5bX8YHo29S94NXhZv5g=', 'Content-Length': '0'} 2022-09-12T20:09:57.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000038f0a5239a13700-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.455 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:09:57.456 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:09:57.456 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:09:57.457 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:09:57.457 DEBUG:boto:Method: HEAD 2022-09-12T20:09:57.458 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:09:57.459 DEBUG:boto:Data: 2022-09-12T20:09:57.459 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.460 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.460 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.461 DEBUG:boto:Params: {} 2022-09-12T20:09:57.461 DEBUG:boto:Token: None 2022-09-12T20:09:57.462 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:09:57.462 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lSM5sxfAJOkrbvAYOm1hM16PpTM= 2022-09-12T20:09:57.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lSM5sxfAJOkrbvAYOm1hM16PpTM=', 'Content-Length': '0'} 2022-09-12T20:09:57.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf1666feb33113e8-00631f9215-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.466 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:09:57.466 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:09:57.467 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:09:57.467 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:09:57.468 DEBUG:boto:Method: HEAD 2022-09-12T20:09:57.468 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:09:57.469 DEBUG:boto:Data: 2022-09-12T20:09:57.469 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.470 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.470 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.471 DEBUG:boto:Params: {} 2022-09-12T20:09:57.471 DEBUG:boto:Token: None 2022-09-12T20:09:57.472 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:09:57.472 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lSM5sxfAJOkrbvAYOm1hM16PpTM= 2022-09-12T20:09:57.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lSM5sxfAJOkrbvAYOm1hM16PpTM=', 'Content-Length': '0'} 2022-09-12T20:09:57.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0ff58905425549b-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:09:57.476 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:09:57.477 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:09:57.477 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:09:57.478 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:09:57.478 DEBUG:boto:Method: GET 2022-09-12T20:09:57.479 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:09:57.479 DEBUG:boto:Data: 2022-09-12T20:09:57.480 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.480 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.481 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.482 DEBUG:boto:Params: {} 2022-09-12T20:09:57.482 DEBUG:boto:Token: None 2022-09-12T20:09:57.483 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:09:57.483 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qN8NNx8p5bX8YHo29S94NXhZv5g= 2022-09-12T20:09:57.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qN8NNx8p5bX8YHo29S94NXhZv5g=', 'Content-Length': '0'} 2022-09-12T20:09:57.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4ce60e75f4246a9-00631f9215-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.487 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:09:57.487 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:09:57.488 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:09:57.488 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:09:57.489 DEBUG:boto:Method: GET 2022-09-12T20:09:57.489 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:09:57.490 DEBUG:boto:Data: 2022-09-12T20:09:57.491 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.491 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.492 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.492 DEBUG:boto:Params: {} 2022-09-12T20:09:57.493 DEBUG:boto:Token: None 2022-09-12T20:09:57.493 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:09:57.494 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qN8NNx8p5bX8YHo29S94NXhZv5g= 2022-09-12T20:09:57.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qN8NNx8p5bX8YHo29S94NXhZv5g=', 'Content-Length': '0'} 2022-09-12T20:09:57.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c07eadf277aa182-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.538 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:57.539 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:57.539 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:09:57.540 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:09:57.540 DEBUG:boto:Method: GET 2022-09-12T20:09:57.541 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:09:57.541 DEBUG:boto:Data: 2022-09-12T20:09:57.542 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.542 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.543 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.543 DEBUG:boto:Params: {} 2022-09-12T20:09:57.544 DEBUG:boto:Token: None 2022-09-12T20:09:57.545 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/ 2022-09-12T20:09:57.545 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw= 2022-09-12T20:09:57.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw=', 'Content-Length': '0'} 2022-09-12T20:09:57.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080c00d0a7e327424-00631f9215-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.549 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:57.550 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:57.551 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:57.551 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:09:57.552 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:09:57.552 DEBUG:boto:Method: GET 2022-09-12T20:09:57.553 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:09:57.553 DEBUG:boto:Data: 2022-09-12T20:09:57.554 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.555 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.555 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.556 DEBUG:boto:Params: {} 2022-09-12T20:09:57.556 DEBUG:boto:Token: None 2022-09-12T20:09:57.557 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/ 2022-09-12T20:09:57.557 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw= 2022-09-12T20:09:57.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw=', 'Content-Length': '0'} 2022-09-12T20:09:57.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000070d7c9e70ab6a33-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.563 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:57.563 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:09:57.564 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:09:57.564 DEBUG:boto:Method: HEAD 2022-09-12T20:09:57.565 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:09:57.565 DEBUG:boto:Data: 2022-09-12T20:09:57.566 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.567 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.567 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.568 DEBUG:boto:Params: {} 2022-09-12T20:09:57.568 DEBUG:boto:Token: None 2022-09-12T20:09:57.569 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj55 2022-09-12T20:09:57.569 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xDui9K+0Tr6k08NBvYNTSD6N6rE= 2022-09-12T20:09:57.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xDui9K+0Tr6k08NBvYNTSD6N6rE=', 'Content-Length': '0'} 2022-09-12T20:09:57.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038dd73f9c25d31e4-00631f9215-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.573 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:09:57.573 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:09:57.574 DEBUG:boto:Method: HEAD 2022-09-12T20:09:57.574 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:09:57.575 DEBUG:boto:Data: 2022-09-12T20:09:57.575 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.576 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.576 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.577 DEBUG:boto:Params: {} 2022-09-12T20:09:57.577 DEBUG:boto:Token: None 2022-09-12T20:09:57.578 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj55 2022-09-12T20:09:57.579 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xDui9K+0Tr6k08NBvYNTSD6N6rE= 2022-09-12T20:09:57.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xDui9K+0Tr6k08NBvYNTSD6N6rE=', 'Content-Length': '0'} 2022-09-12T20:09:57.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067cf1878fb64b0b3-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:09:57.583 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:09:57.583 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:09:57.584 DEBUG:boto:Method: GET 2022-09-12T20:09:57.584 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:09:57.585 DEBUG:boto:Data: 2022-09-12T20:09:57.585 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.586 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.586 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.587 DEBUG:boto:Params: {} 2022-09-12T20:09:57.587 DEBUG:boto:Token: None 2022-09-12T20:09:57.588 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj55 2022-09-12T20:09:57.589 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:axwYlFJDZ+ejYS0qECag4YzCCV0= 2022-09-12T20:09:57.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:axwYlFJDZ+ejYS0qECag4YzCCV0=', 'Content-Length': '0'} 2022-09-12T20:09:57.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d6ca04c776fd838-00631f9215-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.592 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:09:57.593 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:09:57.593 DEBUG:boto:Method: GET 2022-09-12T20:09:57.594 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:09:57.594 DEBUG:boto:Data: 2022-09-12T20:09:57.595 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.595 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.596 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.596 DEBUG:boto:Params: {} 2022-09-12T20:09:57.597 DEBUG:boto:Token: None 2022-09-12T20:09:57.598 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj55 2022-09-12T20:09:57.598 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:axwYlFJDZ+ejYS0qECag4YzCCV0= 2022-09-12T20:09:57.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:axwYlFJDZ+ejYS0qECag4YzCCV0=', 'Content-Length': '0'} 2022-09-12T20:09:57.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4d39c256e150999-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.602 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:57.602 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:57.603 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:09:57.603 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:09:57.604 DEBUG:boto:Method: GET 2022-09-12T20:09:57.605 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:09:57.605 DEBUG:boto:Data: 2022-09-12T20:09:57.606 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.606 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.607 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.607 DEBUG:boto:Params: {} 2022-09-12T20:09:57.608 DEBUG:boto:Token: None 2022-09-12T20:09:57.608 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/ 2022-09-12T20:09:57.609 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw= 2022-09-12T20:09:57.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw=', 'Content-Length': '0'} 2022-09-12T20:09:57.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9b48b361170b1e0-00631f9215-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.613 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:57.614 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:57.614 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:57.615 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:09:57.615 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:09:57.616 DEBUG:boto:Method: GET 2022-09-12T20:09:57.616 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:09:57.617 DEBUG:boto:Data: 2022-09-12T20:09:57.617 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.618 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.619 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.619 DEBUG:boto:Params: {} 2022-09-12T20:09:57.620 DEBUG:boto:Token: None 2022-09-12T20:09:57.620 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/ 2022-09-12T20:09:57.621 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw= 2022-09-12T20:09:57.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw=', 'Content-Length': '0'} 2022-09-12T20:09:57.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063b37efaabde926b-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.626 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:57.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:09:57.627 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:09:57.628 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:09:57.628 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:09:57.629 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:09:57.629 DEBUG:boto:Method: GET 2022-09-12T20:09:57.630 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:09:57.630 DEBUG:boto:Data: 2022-09-12T20:09:57.631 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.632 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.632 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.633 DEBUG:boto:Params: {} 2022-09-12T20:09:57.633 DEBUG:boto:Token: None 2022-09-12T20:09:57.634 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:09:57.634 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OrAsUl4aHZ5JIlV9xpFwBk3Bx4c= 2022-09-12T20:09:57.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OrAsUl4aHZ5JIlV9xpFwBk3Bx4c=', 'Content-Length': '0'} 2022-09-12T20:09:57.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044f4b775a89f75ca-00631f9215-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.638 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:09:57.639 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:09:57.639 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:09:57.640 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:09:57.640 DEBUG:boto:Method: GET 2022-09-12T20:09:57.641 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:09:57.641 DEBUG:boto:Data: 2022-09-12T20:09:57.642 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.642 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.643 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.643 DEBUG:boto:Params: {} 2022-09-12T20:09:57.644 DEBUG:boto:Token: None 2022-09-12T20:09:57.645 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:09:57.645 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OrAsUl4aHZ5JIlV9xpFwBk3Bx4c= 2022-09-12T20:09:57.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OrAsUl4aHZ5JIlV9xpFwBk3Bx4c=', 'Content-Length': '0'} 2022-09-12T20:09:57.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003699dc99045cca87-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.649 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:09:57.650 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:09:57.650 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:09:57.651 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:09:57.651 DEBUG:boto:Method: HEAD 2022-09-12T20:09:57.652 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:09:57.652 DEBUG:boto:Data: 2022-09-12T20:09:57.653 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.653 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.654 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.654 DEBUG:boto:Params: {} 2022-09-12T20:09:57.655 DEBUG:boto:Token: None 2022-09-12T20:09:57.655 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:09:57.656 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dYG545aHIzUdOPNixYcbOxUZYlM= 2022-09-12T20:09:57.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dYG545aHIzUdOPNixYcbOxUZYlM=', 'Content-Length': '0'} 2022-09-12T20:09:57.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000631296fb26a3c36b-00631f9215-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.659 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:09:57.660 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:09:57.660 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:09:57.661 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:09:57.661 DEBUG:boto:Method: HEAD 2022-09-12T20:09:57.662 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:09:57.662 DEBUG:boto:Data: 2022-09-12T20:09:57.663 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.663 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.664 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.664 DEBUG:boto:Params: {} 2022-09-12T20:09:57.665 DEBUG:boto:Token: None 2022-09-12T20:09:57.665 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:09:57.666 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dYG545aHIzUdOPNixYcbOxUZYlM= 2022-09-12T20:09:57.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dYG545aHIzUdOPNixYcbOxUZYlM=', 'Content-Length': '0'} 2022-09-12T20:09:57.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003778a17b43819c65-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:09:57.670 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:09:57.670 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:09:57.671 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:09:57.671 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:09:57.672 DEBUG:boto:Method: GET 2022-09-12T20:09:57.672 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:09:57.673 DEBUG:boto:Data: 2022-09-12T20:09:57.673 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.674 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.675 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.675 DEBUG:boto:Params: {} 2022-09-12T20:09:57.676 DEBUG:boto:Token: None 2022-09-12T20:09:57.676 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:09:57.677 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OrAsUl4aHZ5JIlV9xpFwBk3Bx4c= 2022-09-12T20:09:57.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OrAsUl4aHZ5JIlV9xpFwBk3Bx4c=', 'Content-Length': '0'} 2022-09-12T20:09:57.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d859ebfe8dc836e-00631f9215-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.680 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:09:57.681 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:09:57.681 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:09:57.682 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:09:57.683 DEBUG:boto:Method: GET 2022-09-12T20:09:57.683 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:09:57.684 DEBUG:boto:Data: 2022-09-12T20:09:57.684 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.685 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.685 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.686 DEBUG:boto:Params: {} 2022-09-12T20:09:57.686 DEBUG:boto:Token: None 2022-09-12T20:09:57.687 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:09:57.688 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OrAsUl4aHZ5JIlV9xpFwBk3Bx4c= 2022-09-12T20:09:57.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OrAsUl4aHZ5JIlV9xpFwBk3Bx4c=', 'Content-Length': '0'} 2022-09-12T20:09:57.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097a2ee9d3535f91c-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.691 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:57.692 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:57.692 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:09:57.693 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:09:57.693 DEBUG:boto:Method: GET 2022-09-12T20:09:57.694 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:09:57.694 DEBUG:boto:Data: 2022-09-12T20:09:57.695 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.695 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.696 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.696 DEBUG:boto:Params: {} 2022-09-12T20:09:57.697 DEBUG:boto:Token: None 2022-09-12T20:09:57.698 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/ 2022-09-12T20:09:57.698 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw= 2022-09-12T20:09:57.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw=', 'Content-Length': '0'} 2022-09-12T20:09:57.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026d32b63c83f2061-00631f9215-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.702 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:57.703 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:57.703 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:57.704 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:09:57.704 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:09:57.705 DEBUG:boto:Method: GET 2022-09-12T20:09:57.706 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:09:57.706 DEBUG:boto:Data: 2022-09-12T20:09:57.707 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.707 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.708 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.708 DEBUG:boto:Params: {} 2022-09-12T20:09:57.709 DEBUG:boto:Token: None 2022-09-12T20:09:57.709 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/ 2022-09-12T20:09:57.710 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw= 2022-09-12T20:09:57.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw=', 'Content-Length': '0'} 2022-09-12T20:09:57.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045f936bac10c8fc7-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.758 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:57.759 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:09:57.760 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:09:57.760 DEBUG:boto:Method: HEAD 2022-09-12T20:09:57.761 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:09:57.761 DEBUG:boto:Data: 2022-09-12T20:09:57.762 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.762 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.763 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.763 DEBUG:boto:Params: {} 2022-09-12T20:09:57.764 DEBUG:boto:Token: None 2022-09-12T20:09:57.764 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj57 2022-09-12T20:09:57.765 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q5t7jiJHzetmHcLLpyuwVaFvF+Q= 2022-09-12T20:09:57.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q5t7jiJHzetmHcLLpyuwVaFvF+Q=', 'Content-Length': '0'} 2022-09-12T20:09:57.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce833c3659c2af92-00631f9215-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.768 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:09:57.769 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:09:57.769 DEBUG:boto:Method: HEAD 2022-09-12T20:09:57.770 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:09:57.770 DEBUG:boto:Data: 2022-09-12T20:09:57.771 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.771 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.772 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.772 DEBUG:boto:Params: {} 2022-09-12T20:09:57.773 DEBUG:boto:Token: None 2022-09-12T20:09:57.773 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj57 2022-09-12T20:09:57.774 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q5t7jiJHzetmHcLLpyuwVaFvF+Q= 2022-09-12T20:09:57.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q5t7jiJHzetmHcLLpyuwVaFvF+Q=', 'Content-Length': '0'} 2022-09-12T20:09:57.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038ddc68d6cb68b07-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:09:57.778 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:09:57.779 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:09:57.779 DEBUG:boto:Method: GET 2022-09-12T20:09:57.780 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:09:57.780 DEBUG:boto:Data: 2022-09-12T20:09:57.781 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.781 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.782 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.782 DEBUG:boto:Params: {} 2022-09-12T20:09:57.783 DEBUG:boto:Token: None 2022-09-12T20:09:57.783 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj57 2022-09-12T20:09:57.784 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2nEL2bfbGEiKk8RMQyIuspVn8jU= 2022-09-12T20:09:57.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2nEL2bfbGEiKk8RMQyIuspVn8jU=', 'Content-Length': '0'} 2022-09-12T20:09:57.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e58f9cc2d8cefed-00631f9215-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.787 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:09:57.788 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:09:57.789 DEBUG:boto:Method: GET 2022-09-12T20:09:57.789 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:09:57.790 DEBUG:boto:Data: 2022-09-12T20:09:57.790 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.791 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.791 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.792 DEBUG:boto:Params: {} 2022-09-12T20:09:57.793 DEBUG:boto:Token: None 2022-09-12T20:09:57.793 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj57 2022-09-12T20:09:57.794 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2nEL2bfbGEiKk8RMQyIuspVn8jU= 2022-09-12T20:09:57.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2nEL2bfbGEiKk8RMQyIuspVn8jU=', 'Content-Length': '0'} 2022-09-12T20:09:57.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074f13ae4bedff564-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.797 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:57.798 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:57.799 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:09:57.799 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:09:57.800 DEBUG:boto:Method: GET 2022-09-12T20:09:57.800 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:09:57.801 DEBUG:boto:Data: 2022-09-12T20:09:57.801 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.802 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.802 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.803 DEBUG:boto:Params: {} 2022-09-12T20:09:57.803 DEBUG:boto:Token: None 2022-09-12T20:09:57.804 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/ 2022-09-12T20:09:57.805 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw= 2022-09-12T20:09:57.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw=', 'Content-Length': '0'} 2022-09-12T20:09:57.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c62713a7465d7f03-00631f9215-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.850 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:57.851 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:57.852 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:57.852 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:09:57.853 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:09:57.853 DEBUG:boto:Method: GET 2022-09-12T20:09:57.854 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:09:57.854 DEBUG:boto:Data: 2022-09-12T20:09:57.855 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.856 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.856 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.857 DEBUG:boto:Params: {} 2022-09-12T20:09:57.857 DEBUG:boto:Token: None 2022-09-12T20:09:57.858 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/ 2022-09-12T20:09:57.858 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw= 2022-09-12T20:09:57.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw=', 'Content-Length': '0'} 2022-09-12T20:09:57.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026ffeeb93d2457c6-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.864 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:57.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:09:57.865 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:09:57.866 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:09:57.866 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:09:57.867 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:09:57.867 DEBUG:boto:Method: GET 2022-09-12T20:09:57.868 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:09:57.869 DEBUG:boto:Data: 2022-09-12T20:09:57.869 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.870 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.870 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.871 DEBUG:boto:Params: {} 2022-09-12T20:09:57.871 DEBUG:boto:Token: None 2022-09-12T20:09:57.872 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:09:57.872 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UkoboeRY8KzQ17bs4AG7i9UyZvo= 2022-09-12T20:09:57.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UkoboeRY8KzQ17bs4AG7i9UyZvo=', 'Content-Length': '0'} 2022-09-12T20:09:57.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e93b4ce0c38609e-00631f9215-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.876 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:09:57.876 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:09:57.877 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:09:57.877 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:09:57.878 DEBUG:boto:Method: GET 2022-09-12T20:09:57.879 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:09:57.879 DEBUG:boto:Data: 2022-09-12T20:09:57.880 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.880 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.881 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.881 DEBUG:boto:Params: {} 2022-09-12T20:09:57.882 DEBUG:boto:Token: None 2022-09-12T20:09:57.882 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:09:57.883 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UkoboeRY8KzQ17bs4AG7i9UyZvo= 2022-09-12T20:09:57.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UkoboeRY8KzQ17bs4AG7i9UyZvo=', 'Content-Length': '0'} 2022-09-12T20:09:57.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000236a67752fb36e71-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.887 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:09:57.887 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:09:57.888 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:09:57.888 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:09:57.888 DEBUG:boto:Method: HEAD 2022-09-12T20:09:57.889 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:09:57.889 DEBUG:boto:Data: 2022-09-12T20:09:57.890 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.890 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.891 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.891 DEBUG:boto:Params: {} 2022-09-12T20:09:57.892 DEBUG:boto:Token: None 2022-09-12T20:09:57.893 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:09:57.893 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oEkDjwkl352wr5eopMFMnppsnbo= 2022-09-12T20:09:57.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oEkDjwkl352wr5eopMFMnppsnbo=', 'Content-Length': '0'} 2022-09-12T20:09:57.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ee9b75ae863e8d8-00631f9215-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.896 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:09:57.897 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:09:57.897 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:09:57.898 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:09:57.898 DEBUG:boto:Method: HEAD 2022-09-12T20:09:57.899 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:09:57.899 DEBUG:boto:Data: 2022-09-12T20:09:57.900 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.900 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.901 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.901 DEBUG:boto:Params: {} 2022-09-12T20:09:57.902 DEBUG:boto:Token: None 2022-09-12T20:09:57.902 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:09:57.903 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oEkDjwkl352wr5eopMFMnppsnbo= 2022-09-12T20:09:57.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oEkDjwkl352wr5eopMFMnppsnbo=', 'Content-Length': '0'} 2022-09-12T20:09:57.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080dcc4d4388bb100-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:09:57.906 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:09:57.907 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:09:57.907 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:09:57.908 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:09:57.908 DEBUG:boto:Method: GET 2022-09-12T20:09:57.909 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:09:57.910 DEBUG:boto:Data: 2022-09-12T20:09:57.910 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.911 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.911 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.912 DEBUG:boto:Params: {} 2022-09-12T20:09:57.912 DEBUG:boto:Token: None 2022-09-12T20:09:57.913 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:09:57.913 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UkoboeRY8KzQ17bs4AG7i9UyZvo= 2022-09-12T20:09:57.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UkoboeRY8KzQ17bs4AG7i9UyZvo=', 'Content-Length': '0'} 2022-09-12T20:09:57.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005625974323466153-00631f9215-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.917 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:09:57.917 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:09:57.918 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:09:57.918 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:09:57.919 DEBUG:boto:Method: GET 2022-09-12T20:09:57.919 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:09:57.920 DEBUG:boto:Data: 2022-09-12T20:09:57.921 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.921 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.922 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.922 DEBUG:boto:Params: {} 2022-09-12T20:09:57.923 DEBUG:boto:Token: None 2022-09-12T20:09:57.923 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:09:57.924 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UkoboeRY8KzQ17bs4AG7i9UyZvo= 2022-09-12T20:09:57.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UkoboeRY8KzQ17bs4AG7i9UyZvo=', 'Content-Length': '0'} 2022-09-12T20:09:57.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5d887669de402f7-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.970 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:57.971 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:57.971 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:09:57.972 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:09:57.972 DEBUG:boto:Method: GET 2022-09-12T20:09:57.973 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:09:57.974 DEBUG:boto:Data: 2022-09-12T20:09:57.974 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.975 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.975 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.976 DEBUG:boto:Params: {} 2022-09-12T20:09:57.976 DEBUG:boto:Token: None 2022-09-12T20:09:57.977 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/ 2022-09-12T20:09:57.978 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw= 2022-09-12T20:09:57.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw=', 'Content-Length': '0'} 2022-09-12T20:09:57.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000485f9ec459e3817c-00631f9215-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.982 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:57.983 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:57.983 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:57.984 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:09:57.984 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:09:57.985 DEBUG:boto:Method: GET 2022-09-12T20:09:57.985 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:09:57.986 DEBUG:boto:Data: 2022-09-12T20:09:57.987 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.987 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:57.988 DEBUG:boto:Port: 8000 2022-09-12T20:09:57.988 DEBUG:boto:Params: {} 2022-09-12T20:09:57.989 DEBUG:boto:Token: None 2022-09-12T20:09:57.989 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:57 GMT /jdpiak-75/ 2022-09-12T20:09:57.990 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw= 2022-09-12T20:09:57.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:57 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LK1rGybOl2CgZPBjabx7oTCZ9vw=', 'Content-Length': '0'} 2022-09-12T20:09:57.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc46e38d32f3044f-00631f9215-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:57.995 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:57.996 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:09:57.996 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:09:57.997 DEBUG:boto:Method: HEAD 2022-09-12T20:09:57.997 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:09:57.998 DEBUG:boto:Data: 2022-09-12T20:09:57.998 DEBUG:boto:Headers: {} 2022-09-12T20:09:57.999 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:57.999 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.000 DEBUG:boto:Params: {} 2022-09-12T20:09:58.000 DEBUG:boto:Token: None 2022-09-12T20:09:58.001 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj68 2022-09-12T20:09:58.002 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:C1AD2ZJxJ2iAhvGwBGEtqJnYqGQ= 2022-09-12T20:09:58.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:C1AD2ZJxJ2iAhvGwBGEtqJnYqGQ=', 'Content-Length': '0'} 2022-09-12T20:09:58.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce7b66fcd562a06b-00631f9216-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.005 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:09:58.005 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:09:58.006 DEBUG:boto:Method: HEAD 2022-09-12T20:09:58.006 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:09:58.007 DEBUG:boto:Data: 2022-09-12T20:09:58.007 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.008 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.009 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.009 DEBUG:boto:Params: {} 2022-09-12T20:09:58.010 DEBUG:boto:Token: None 2022-09-12T20:09:58.011 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj68 2022-09-12T20:09:58.011 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:C1AD2ZJxJ2iAhvGwBGEtqJnYqGQ= 2022-09-12T20:09:58.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:C1AD2ZJxJ2iAhvGwBGEtqJnYqGQ=', 'Content-Length': '0'} 2022-09-12T20:09:58.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a28436c0a23f0908-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:09:58.016 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:09:58.016 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:09:58.017 DEBUG:boto:Method: GET 2022-09-12T20:09:58.017 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:09:58.018 DEBUG:boto:Data: 2022-09-12T20:09:58.018 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.019 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.019 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.020 DEBUG:boto:Params: {} 2022-09-12T20:09:58.021 DEBUG:boto:Token: None 2022-09-12T20:09:58.021 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj68 2022-09-12T20:09:58.022 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tysfcwffgcFM3TERWC09e0GvP8Y= 2022-09-12T20:09:58.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tysfcwffgcFM3TERWC09e0GvP8Y=', 'Content-Length': '0'} 2022-09-12T20:09:58.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcba5d2b6b900671-00631f9216-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.025 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:09:58.026 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:09:58.027 DEBUG:boto:Method: GET 2022-09-12T20:09:58.027 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:09:58.028 DEBUG:boto:Data: 2022-09-12T20:09:58.028 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.029 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.029 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.030 DEBUG:boto:Params: {} 2022-09-12T20:09:58.030 DEBUG:boto:Token: None 2022-09-12T20:09:58.031 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj68 2022-09-12T20:09:58.032 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tysfcwffgcFM3TERWC09e0GvP8Y= 2022-09-12T20:09:58.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tysfcwffgcFM3TERWC09e0GvP8Y=', 'Content-Length': '0'} 2022-09-12T20:09:58.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009449a0b70a887d8b-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.078 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:58.079 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:58.079 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:09:58.080 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:09:58.080 DEBUG:boto:Method: GET 2022-09-12T20:09:58.081 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:09:58.081 DEBUG:boto:Data: 2022-09-12T20:09:58.082 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.082 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.083 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.083 DEBUG:boto:Params: {} 2022-09-12T20:09:58.084 DEBUG:boto:Token: None 2022-09-12T20:09:58.085 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/ 2022-09-12T20:09:58.085 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY= 2022-09-12T20:09:58.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY=', 'Content-Length': '0'} 2022-09-12T20:09:58.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cab109f22f471659-00631f9216-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.089 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:58.090 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:58.090 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:58.091 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:09:58.091 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:09:58.092 DEBUG:boto:Method: GET 2022-09-12T20:09:58.093 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:09:58.093 DEBUG:boto:Data: 2022-09-12T20:09:58.094 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.094 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.095 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.095 DEBUG:boto:Params: {} 2022-09-12T20:09:58.096 DEBUG:boto:Token: None 2022-09-12T20:09:58.096 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/ 2022-09-12T20:09:58.097 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY= 2022-09-12T20:09:58.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY=', 'Content-Length': '0'} 2022-09-12T20:09:58.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfbe9d4e379c1a2f-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.102 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:58.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:09:58.103 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:09:58.104 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:09:58.105 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:09:58.105 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:09:58.106 DEBUG:boto:Method: GET 2022-09-12T20:09:58.106 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:09:58.107 DEBUG:boto:Data: 2022-09-12T20:09:58.108 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.108 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.109 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.110 DEBUG:boto:Params: {} 2022-09-12T20:09:58.110 DEBUG:boto:Token: None 2022-09-12T20:09:58.111 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:09:58.112 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BSIY5B0XwP0oaisfiNiG2JJgrBQ= 2022-09-12T20:09:58.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BSIY5B0XwP0oaisfiNiG2JJgrBQ=', 'Content-Length': '0'} 2022-09-12T20:09:58.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003e6a1a125606130-00631f9216-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.116 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:09:58.117 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:09:58.118 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:09:58.118 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:09:58.119 DEBUG:boto:Method: GET 2022-09-12T20:09:58.120 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:09:58.120 DEBUG:boto:Data: 2022-09-12T20:09:58.121 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.122 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.123 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.123 DEBUG:boto:Params: {} 2022-09-12T20:09:58.124 DEBUG:boto:Token: None 2022-09-12T20:09:58.125 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:09:58.126 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BSIY5B0XwP0oaisfiNiG2JJgrBQ= 2022-09-12T20:09:58.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BSIY5B0XwP0oaisfiNiG2JJgrBQ=', 'Content-Length': '0'} 2022-09-12T20:09:58.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000141c4c79cb50df8b-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.130 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:09:58.130 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:09:58.131 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:09:58.132 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:09:58.132 DEBUG:boto:Method: HEAD 2022-09-12T20:09:58.133 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:09:58.133 DEBUG:boto:Data: 2022-09-12T20:09:58.134 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.134 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.135 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.136 DEBUG:boto:Params: {} 2022-09-12T20:09:58.136 DEBUG:boto:Token: None 2022-09-12T20:09:58.137 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:09:58.138 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:APjLJRlc2qIy7Md+EDWBZyzVtB8= 2022-09-12T20:09:58.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:APjLJRlc2qIy7Md+EDWBZyzVtB8=', 'Content-Length': '0'} 2022-09-12T20:09:58.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080e608a2f8ebf63a-00631f9216-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.141 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:09:58.142 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:09:58.142 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:09:58.143 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:09:58.144 DEBUG:boto:Method: HEAD 2022-09-12T20:09:58.144 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:09:58.145 DEBUG:boto:Data: 2022-09-12T20:09:58.145 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.146 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.147 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.147 DEBUG:boto:Params: {} 2022-09-12T20:09:58.148 DEBUG:boto:Token: None 2022-09-12T20:09:58.148 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:09:58.149 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:APjLJRlc2qIy7Md+EDWBZyzVtB8= 2022-09-12T20:09:58.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:APjLJRlc2qIy7Md+EDWBZyzVtB8=', 'Content-Length': '0'} 2022-09-12T20:09:58.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3e6f5a4507946d6-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:09:58.153 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:09:58.154 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:09:58.155 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:09:58.155 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:09:58.156 DEBUG:boto:Method: GET 2022-09-12T20:09:58.157 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:09:58.157 DEBUG:boto:Data: 2022-09-12T20:09:58.158 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.159 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.159 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.160 DEBUG:boto:Params: {} 2022-09-12T20:09:58.161 DEBUG:boto:Token: None 2022-09-12T20:09:58.162 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:09:58.162 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BSIY5B0XwP0oaisfiNiG2JJgrBQ= 2022-09-12T20:09:58.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BSIY5B0XwP0oaisfiNiG2JJgrBQ=', 'Content-Length': '0'} 2022-09-12T20:09:58.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9ac4df8f7b85454-00631f9216-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.166 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:09:58.167 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:09:58.168 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:09:58.168 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:09:58.169 DEBUG:boto:Method: GET 2022-09-12T20:09:58.169 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:09:58.170 DEBUG:boto:Data: 2022-09-12T20:09:58.171 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.171 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.172 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.172 DEBUG:boto:Params: {} 2022-09-12T20:09:58.173 DEBUG:boto:Token: None 2022-09-12T20:09:58.174 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:09:58.174 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BSIY5B0XwP0oaisfiNiG2JJgrBQ= 2022-09-12T20:09:58.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BSIY5B0XwP0oaisfiNiG2JJgrBQ=', 'Content-Length': '0'} 2022-09-12T20:09:58.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038dfd69afb861770-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.178 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:58.179 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:58.180 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:09:58.181 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:09:58.181 DEBUG:boto:Method: GET 2022-09-12T20:09:58.182 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:09:58.183 DEBUG:boto:Data: 2022-09-12T20:09:58.183 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.184 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.184 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.185 DEBUG:boto:Params: {} 2022-09-12T20:09:58.185 DEBUG:boto:Token: None 2022-09-12T20:09:58.186 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/ 2022-09-12T20:09:58.187 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY= 2022-09-12T20:09:58.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY=', 'Content-Length': '0'} 2022-09-12T20:09:58.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059e69151f9045a83-00631f9216-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.191 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:58.192 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:58.193 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:58.193 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:09:58.194 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:09:58.195 DEBUG:boto:Method: GET 2022-09-12T20:09:58.196 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:09:58.196 DEBUG:boto:Data: 2022-09-12T20:09:58.197 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.197 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.198 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.198 DEBUG:boto:Params: {} 2022-09-12T20:09:58.199 DEBUG:boto:Token: None 2022-09-12T20:09:58.200 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/ 2022-09-12T20:09:58.200 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY= 2022-09-12T20:09:58.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY=', 'Content-Length': '0'} 2022-09-12T20:09:58.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cba066a1f0ec9d0-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.206 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:58.207 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:09:58.207 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:09:58.208 DEBUG:boto:Method: HEAD 2022-09-12T20:09:58.209 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:09:58.210 DEBUG:boto:Data: 2022-09-12T20:09:58.211 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.211 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.212 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.213 DEBUG:boto:Params: {} 2022-09-12T20:09:58.214 DEBUG:boto:Token: None 2022-09-12T20:09:58.215 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj69 2022-09-12T20:09:58.216 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QFWGFYUlBAhAKRez92OjzmzW/U0= 2022-09-12T20:09:58.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QFWGFYUlBAhAKRez92OjzmzW/U0=', 'Content-Length': '0'} 2022-09-12T20:09:58.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6eee51f004fd061-00631f9216-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.219 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:09:58.220 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:09:58.220 DEBUG:boto:Method: HEAD 2022-09-12T20:09:58.221 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:09:58.221 DEBUG:boto:Data: 2022-09-12T20:09:58.222 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.222 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.223 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.223 DEBUG:boto:Params: {} 2022-09-12T20:09:58.224 DEBUG:boto:Token: None 2022-09-12T20:09:58.224 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj69 2022-09-12T20:09:58.225 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QFWGFYUlBAhAKRez92OjzmzW/U0= 2022-09-12T20:09:58.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QFWGFYUlBAhAKRez92OjzmzW/U0=', 'Content-Length': '0'} 2022-09-12T20:09:58.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6902ef70db9c771-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:09:58.229 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:09:58.230 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:09:58.230 DEBUG:boto:Method: GET 2022-09-12T20:09:58.231 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:09:58.231 DEBUG:boto:Data: 2022-09-12T20:09:58.232 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.232 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.233 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.233 DEBUG:boto:Params: {} 2022-09-12T20:09:58.234 DEBUG:boto:Token: None 2022-09-12T20:09:58.235 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj69 2022-09-12T20:09:58.235 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P55uSssgrajV5OBD4qmtDHWdy0Q= 2022-09-12T20:09:58.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P55uSssgrajV5OBD4qmtDHWdy0Q=', 'Content-Length': '0'} 2022-09-12T20:09:58.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002299570df7dffad5-00631f9216-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.282 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:09:58.283 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:09:58.283 DEBUG:boto:Method: GET 2022-09-12T20:09:58.284 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:09:58.284 DEBUG:boto:Data: 2022-09-12T20:09:58.285 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.285 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.286 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.286 DEBUG:boto:Params: {} 2022-09-12T20:09:58.287 DEBUG:boto:Token: None 2022-09-12T20:09:58.288 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj69 2022-09-12T20:09:58.288 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P55uSssgrajV5OBD4qmtDHWdy0Q= 2022-09-12T20:09:58.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P55uSssgrajV5OBD4qmtDHWdy0Q=', 'Content-Length': '0'} 2022-09-12T20:09:58.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003337ce915cd6e264-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.292 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:58.292 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:58.293 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:09:58.293 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:09:58.294 DEBUG:boto:Method: GET 2022-09-12T20:09:58.294 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:09:58.295 DEBUG:boto:Data: 2022-09-12T20:09:58.296 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.296 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.297 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.297 DEBUG:boto:Params: {} 2022-09-12T20:09:58.298 DEBUG:boto:Token: None 2022-09-12T20:09:58.298 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/ 2022-09-12T20:09:58.299 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY= 2022-09-12T20:09:58.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY=', 'Content-Length': '0'} 2022-09-12T20:09:58.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cbaef22d6695e6b-00631f9216-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.303 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:58.303 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:58.304 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:58.305 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:09:58.305 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:09:58.306 DEBUG:boto:Method: GET 2022-09-12T20:09:58.306 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:09:58.307 DEBUG:boto:Data: 2022-09-12T20:09:58.307 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.308 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.308 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.309 DEBUG:boto:Params: {} 2022-09-12T20:09:58.309 DEBUG:boto:Token: None 2022-09-12T20:09:58.310 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/ 2022-09-12T20:09:58.311 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY= 2022-09-12T20:09:58.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY=', 'Content-Length': '0'} 2022-09-12T20:09:58.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003089e3ee977fa386-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.316 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:58.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:09:58.317 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:09:58.317 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:09:58.318 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:09:58.319 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:09:58.319 DEBUG:boto:Method: GET 2022-09-12T20:09:58.320 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:09:58.320 DEBUG:boto:Data: 2022-09-12T20:09:58.321 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.321 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.322 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.322 DEBUG:boto:Params: {} 2022-09-12T20:09:58.323 DEBUG:boto:Token: None 2022-09-12T20:09:58.323 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:09:58.324 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MQ7yQochtyp6GuARjqKoR1iTaOw= 2022-09-12T20:09:58.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MQ7yQochtyp6GuARjqKoR1iTaOw=', 'Content-Length': '0'} 2022-09-12T20:09:58.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025bdca760777921e-00631f9216-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.328 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:09:58.328 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:09:58.329 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:09:58.329 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:09:58.330 DEBUG:boto:Method: GET 2022-09-12T20:09:58.330 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:09:58.331 DEBUG:boto:Data: 2022-09-12T20:09:58.331 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.332 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.332 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.333 DEBUG:boto:Params: {} 2022-09-12T20:09:58.334 DEBUG:boto:Token: None 2022-09-12T20:09:58.334 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:09:58.335 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MQ7yQochtyp6GuARjqKoR1iTaOw= 2022-09-12T20:09:58.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MQ7yQochtyp6GuARjqKoR1iTaOw=', 'Content-Length': '0'} 2022-09-12T20:09:58.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b870841fe4255ba0-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.378 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:09:58.378 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:09:58.379 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:09:58.379 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:09:58.380 DEBUG:boto:Method: HEAD 2022-09-12T20:09:58.380 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:09:58.381 DEBUG:boto:Data: 2022-09-12T20:09:58.382 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.382 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.383 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.383 DEBUG:boto:Params: {} 2022-09-12T20:09:58.384 DEBUG:boto:Token: None 2022-09-12T20:09:58.384 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:09:58.385 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ai/Ttz7FjHt/a/26iaeWgRQ5axE= 2022-09-12T20:09:58.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ai/Ttz7FjHt/a/26iaeWgRQ5axE=', 'Content-Length': '0'} 2022-09-12T20:09:58.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000821f8578be6b8f58-00631f9216-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.389 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:09:58.389 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:09:58.390 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:09:58.390 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:09:58.391 DEBUG:boto:Method: HEAD 2022-09-12T20:09:58.391 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:09:58.392 DEBUG:boto:Data: 2022-09-12T20:09:58.392 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.393 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.393 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.394 DEBUG:boto:Params: {} 2022-09-12T20:09:58.394 DEBUG:boto:Token: None 2022-09-12T20:09:58.395 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:09:58.396 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ai/Ttz7FjHt/a/26iaeWgRQ5axE= 2022-09-12T20:09:58.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ai/Ttz7FjHt/a/26iaeWgRQ5axE=', 'Content-Length': '0'} 2022-09-12T20:09:58.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bccc026cf83de70-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:09:58.400 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:09:58.400 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:09:58.401 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:09:58.401 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:09:58.402 DEBUG:boto:Method: GET 2022-09-12T20:09:58.402 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:09:58.403 DEBUG:boto:Data: 2022-09-12T20:09:58.404 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.404 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.405 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.405 DEBUG:boto:Params: {} 2022-09-12T20:09:58.406 DEBUG:boto:Token: None 2022-09-12T20:09:58.406 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:09:58.407 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MQ7yQochtyp6GuARjqKoR1iTaOw= 2022-09-12T20:09:58.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MQ7yQochtyp6GuARjqKoR1iTaOw=', 'Content-Length': '0'} 2022-09-12T20:09:58.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8d811cbdf934359-00631f9216-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.410 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:09:58.411 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:09:58.412 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:09:58.412 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:09:58.413 DEBUG:boto:Method: GET 2022-09-12T20:09:58.413 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:09:58.414 DEBUG:boto:Data: 2022-09-12T20:09:58.414 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.415 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.415 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.416 DEBUG:boto:Params: {} 2022-09-12T20:09:58.416 DEBUG:boto:Token: None 2022-09-12T20:09:58.417 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:09:58.418 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MQ7yQochtyp6GuARjqKoR1iTaOw= 2022-09-12T20:09:58.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MQ7yQochtyp6GuARjqKoR1iTaOw=', 'Content-Length': '0'} 2022-09-12T20:09:58.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094198e77634cb87f-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.422 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:58.422 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:58.423 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:09:58.423 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:09:58.424 DEBUG:boto:Method: GET 2022-09-12T20:09:58.424 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:09:58.425 DEBUG:boto:Data: 2022-09-12T20:09:58.425 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.426 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.427 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.427 DEBUG:boto:Params: {} 2022-09-12T20:09:58.428 DEBUG:boto:Token: None 2022-09-12T20:09:58.428 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/ 2022-09-12T20:09:58.429 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY= 2022-09-12T20:09:58.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY=', 'Content-Length': '0'} 2022-09-12T20:09:58.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8aa699af54f9200-00631f9216-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.433 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:58.434 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:58.434 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:58.435 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:09:58.436 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:09:58.436 DEBUG:boto:Method: GET 2022-09-12T20:09:58.437 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:09:58.437 DEBUG:boto:Data: 2022-09-12T20:09:58.438 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.438 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.439 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.439 DEBUG:boto:Params: {} 2022-09-12T20:09:58.440 DEBUG:boto:Token: None 2022-09-12T20:09:58.441 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/ 2022-09-12T20:09:58.441 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY= 2022-09-12T20:09:58.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY=', 'Content-Length': '0'} 2022-09-12T20:09:58.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000582c8d384928ef0-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.446 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:58.447 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:09:58.447 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:09:58.448 DEBUG:boto:Method: HEAD 2022-09-12T20:09:58.448 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:09:58.449 DEBUG:boto:Data: 2022-09-12T20:09:58.449 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.450 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.451 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.451 DEBUG:boto:Params: {} 2022-09-12T20:09:58.452 DEBUG:boto:Token: None 2022-09-12T20:09:58.452 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj76 2022-09-12T20:09:58.453 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AbdumxOPMgeJD2WCjdBoy8ciV68= 2022-09-12T20:09:58.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AbdumxOPMgeJD2WCjdBoy8ciV68=', 'Content-Length': '0'} 2022-09-12T20:09:58.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ba6a53024b5aaa9-00631f9216-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.456 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:09:58.457 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:09:58.457 DEBUG:boto:Method: HEAD 2022-09-12T20:09:58.458 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:09:58.458 DEBUG:boto:Data: 2022-09-12T20:09:58.459 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.459 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.460 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.460 DEBUG:boto:Params: {} 2022-09-12T20:09:58.461 DEBUG:boto:Token: None 2022-09-12T20:09:58.461 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj76 2022-09-12T20:09:58.462 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AbdumxOPMgeJD2WCjdBoy8ciV68= 2022-09-12T20:09:58.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AbdumxOPMgeJD2WCjdBoy8ciV68=', 'Content-Length': '0'} 2022-09-12T20:09:58.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008951748de032b371-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:09:58.466 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:09:58.467 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:09:58.467 DEBUG:boto:Method: GET 2022-09-12T20:09:58.468 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:09:58.468 DEBUG:boto:Data: 2022-09-12T20:09:58.469 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.469 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.470 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.470 DEBUG:boto:Params: {} 2022-09-12T20:09:58.471 DEBUG:boto:Token: None 2022-09-12T20:09:58.471 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj76 2022-09-12T20:09:58.472 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7cmQQcp2XS6/4PymHvuZrkHyJ40= 2022-09-12T20:09:58.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7cmQQcp2XS6/4PymHvuZrkHyJ40=', 'Content-Length': '0'} 2022-09-12T20:09:58.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecd059dd923a1792-00631f9216-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.476 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:09:58.476 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:09:58.477 DEBUG:boto:Method: GET 2022-09-12T20:09:58.477 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:09:58.478 DEBUG:boto:Data: 2022-09-12T20:09:58.478 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.479 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.479 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.480 DEBUG:boto:Params: {} 2022-09-12T20:09:58.480 DEBUG:boto:Token: None 2022-09-12T20:09:58.481 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj76 2022-09-12T20:09:58.482 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7cmQQcp2XS6/4PymHvuZrkHyJ40= 2022-09-12T20:09:58.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7cmQQcp2XS6/4PymHvuZrkHyJ40=', 'Content-Length': '0'} 2022-09-12T20:09:58.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000900cc8b1ae67371b-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.485 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:58.486 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:58.486 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:09:58.487 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:09:58.487 DEBUG:boto:Method: GET 2022-09-12T20:09:58.488 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:09:58.488 DEBUG:boto:Data: 2022-09-12T20:09:58.489 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.489 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.490 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.491 DEBUG:boto:Params: {} 2022-09-12T20:09:58.491 DEBUG:boto:Token: None 2022-09-12T20:09:58.492 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/ 2022-09-12T20:09:58.492 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY= 2022-09-12T20:09:58.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY=', 'Content-Length': '0'} 2022-09-12T20:09:58.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d1c279d926b163f-00631f9216-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.538 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:58.539 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:58.539 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:58.540 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:09:58.540 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:09:58.541 DEBUG:boto:Method: GET 2022-09-12T20:09:58.542 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:09:58.542 DEBUG:boto:Data: 2022-09-12T20:09:58.543 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.543 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.544 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.544 DEBUG:boto:Params: {} 2022-09-12T20:09:58.545 DEBUG:boto:Token: None 2022-09-12T20:09:58.545 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/ 2022-09-12T20:09:58.546 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY= 2022-09-12T20:09:58.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY=', 'Content-Length': '0'} 2022-09-12T20:09:58.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009767fa0c790513f5-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.551 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:58.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:09:58.552 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:09:58.553 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:09:58.553 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:09:58.554 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:09:58.555 DEBUG:boto:Method: GET 2022-09-12T20:09:58.555 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:09:58.556 DEBUG:boto:Data: 2022-09-12T20:09:58.556 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.557 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.557 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.558 DEBUG:boto:Params: {} 2022-09-12T20:09:58.558 DEBUG:boto:Token: None 2022-09-12T20:09:58.559 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:09:58.560 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oUtpityIEEkod0/yefTmvNSTqbI= 2022-09-12T20:09:58.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oUtpityIEEkod0/yefTmvNSTqbI=', 'Content-Length': '0'} 2022-09-12T20:09:58.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038035017e2f22bb0-00631f9216-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.563 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:09:58.564 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:09:58.564 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:09:58.565 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:09:58.565 DEBUG:boto:Method: GET 2022-09-12T20:09:58.566 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:09:58.567 DEBUG:boto:Data: 2022-09-12T20:09:58.567 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.568 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.568 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.569 DEBUG:boto:Params: {} 2022-09-12T20:09:58.569 DEBUG:boto:Token: None 2022-09-12T20:09:58.570 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:09:58.570 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oUtpityIEEkod0/yefTmvNSTqbI= 2022-09-12T20:09:58.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oUtpityIEEkod0/yefTmvNSTqbI=', 'Content-Length': '0'} 2022-09-12T20:09:58.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000646a086f48fc41c-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.614 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:09:58.615 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:09:58.615 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:09:58.616 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:09:58.616 DEBUG:boto:Method: HEAD 2022-09-12T20:09:58.617 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:09:58.617 DEBUG:boto:Data: 2022-09-12T20:09:58.618 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.618 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.619 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.619 DEBUG:boto:Params: {} 2022-09-12T20:09:58.620 DEBUG:boto:Token: None 2022-09-12T20:09:58.620 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:09:58.621 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TZVoq70Pdg4BEJq1pjyWd9nZPWE= 2022-09-12T20:09:58.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TZVoq70Pdg4BEJq1pjyWd9nZPWE=', 'Content-Length': '0'} 2022-09-12T20:09:58.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000645db927f8d8acc1-00631f9216-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.624 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:09:58.625 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:09:58.625 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:09:58.626 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:09:58.627 DEBUG:boto:Method: HEAD 2022-09-12T20:09:58.627 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:09:58.628 DEBUG:boto:Data: 2022-09-12T20:09:58.628 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.629 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.629 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.630 DEBUG:boto:Params: {} 2022-09-12T20:09:58.630 DEBUG:boto:Token: None 2022-09-12T20:09:58.631 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:09:58.631 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TZVoq70Pdg4BEJq1pjyWd9nZPWE= 2022-09-12T20:09:58.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TZVoq70Pdg4BEJq1pjyWd9nZPWE=', 'Content-Length': '0'} 2022-09-12T20:09:58.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000574740fb30dadb52-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:09:58.635 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:09:58.636 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:09:58.636 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:09:58.637 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:09:58.637 DEBUG:boto:Method: GET 2022-09-12T20:09:58.638 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:09:58.638 DEBUG:boto:Data: 2022-09-12T20:09:58.639 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.640 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.640 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.641 DEBUG:boto:Params: {} 2022-09-12T20:09:58.641 DEBUG:boto:Token: None 2022-09-12T20:09:58.642 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:09:58.642 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oUtpityIEEkod0/yefTmvNSTqbI= 2022-09-12T20:09:58.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oUtpityIEEkod0/yefTmvNSTqbI=', 'Content-Length': '0'} 2022-09-12T20:09:58.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4c1eb1cfb619a93-00631f9216-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.646 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:09:58.646 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:09:58.647 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:09:58.647 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:09:58.648 DEBUG:boto:Method: GET 2022-09-12T20:09:58.648 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:09:58.649 DEBUG:boto:Data: 2022-09-12T20:09:58.650 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.650 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.651 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.651 DEBUG:boto:Params: {} 2022-09-12T20:09:58.652 DEBUG:boto:Token: None 2022-09-12T20:09:58.652 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:09:58.653 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oUtpityIEEkod0/yefTmvNSTqbI= 2022-09-12T20:09:58.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oUtpityIEEkod0/yefTmvNSTqbI=', 'Content-Length': '0'} 2022-09-12T20:09:58.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d5f0443fca75d9d-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.657 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:58.658 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:58.658 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:09:58.659 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:09:58.659 DEBUG:boto:Method: GET 2022-09-12T20:09:58.660 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:09:58.660 DEBUG:boto:Data: 2022-09-12T20:09:58.661 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.661 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.662 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.662 DEBUG:boto:Params: {} 2022-09-12T20:09:58.663 DEBUG:boto:Token: None 2022-09-12T20:09:58.663 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/ 2022-09-12T20:09:58.664 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY= 2022-09-12T20:09:58.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY=', 'Content-Length': '0'} 2022-09-12T20:09:58.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000279ca7571cea9013-00631f9216-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.668 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:58.669 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:58.670 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:58.670 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:09:58.671 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:09:58.671 DEBUG:boto:Method: GET 2022-09-12T20:09:58.672 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:09:58.672 DEBUG:boto:Data: 2022-09-12T20:09:58.673 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.674 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.674 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.675 DEBUG:boto:Params: {} 2022-09-12T20:09:58.675 DEBUG:boto:Token: None 2022-09-12T20:09:58.676 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/ 2022-09-12T20:09:58.676 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY= 2022-09-12T20:09:58.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY=', 'Content-Length': '0'} 2022-09-12T20:09:58.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002977cd9dcad9b5db-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.681 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:58.682 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:09:58.683 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:09:58.683 DEBUG:boto:Method: HEAD 2022-09-12T20:09:58.684 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:09:58.684 DEBUG:boto:Data: 2022-09-12T20:09:58.685 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.685 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.686 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.686 DEBUG:boto:Params: {} 2022-09-12T20:09:58.687 DEBUG:boto:Token: None 2022-09-12T20:09:58.688 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj85 2022-09-12T20:09:58.688 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qcokiH3YIjnomAOb1OJQkseuPxg= 2022-09-12T20:09:58.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qcokiH3YIjnomAOb1OJQkseuPxg=', 'Content-Length': '0'} 2022-09-12T20:09:58.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000422f39594c006935-00631f9216-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.691 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:09:58.692 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:09:58.693 DEBUG:boto:Method: HEAD 2022-09-12T20:09:58.693 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:09:58.694 DEBUG:boto:Data: 2022-09-12T20:09:58.694 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.695 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.695 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.696 DEBUG:boto:Params: {} 2022-09-12T20:09:58.697 DEBUG:boto:Token: None 2022-09-12T20:09:58.697 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj85 2022-09-12T20:09:58.698 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qcokiH3YIjnomAOb1OJQkseuPxg= 2022-09-12T20:09:58.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qcokiH3YIjnomAOb1OJQkseuPxg=', 'Content-Length': '0'} 2022-09-12T20:09:58.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041c5224fe38be389-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:09:58.702 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:09:58.703 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:09:58.703 DEBUG:boto:Method: GET 2022-09-12T20:09:58.704 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:09:58.704 DEBUG:boto:Data: 2022-09-12T20:09:58.705 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.705 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.706 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.706 DEBUG:boto:Params: {} 2022-09-12T20:09:58.707 DEBUG:boto:Token: None 2022-09-12T20:09:58.708 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj85 2022-09-12T20:09:58.708 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JGHO8F+5Wvys567r5yqGRMpDWiE= 2022-09-12T20:09:58.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JGHO8F+5Wvys567r5yqGRMpDWiE=', 'Content-Length': '0'} 2022-09-12T20:09:58.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8822a6f5159cb1e-00631f9216-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.754 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:09:58.755 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:09:58.755 DEBUG:boto:Method: GET 2022-09-12T20:09:58.756 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:09:58.756 DEBUG:boto:Data: 2022-09-12T20:09:58.757 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.757 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.758 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.758 DEBUG:boto:Params: {} 2022-09-12T20:09:58.759 DEBUG:boto:Token: None 2022-09-12T20:09:58.759 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj85 2022-09-12T20:09:58.760 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JGHO8F+5Wvys567r5yqGRMpDWiE= 2022-09-12T20:09:58.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JGHO8F+5Wvys567r5yqGRMpDWiE=', 'Content-Length': '0'} 2022-09-12T20:09:58.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbe712fdcf86e979-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.764 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:58.764 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:58.765 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:09:58.765 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:09:58.766 DEBUG:boto:Method: GET 2022-09-12T20:09:58.766 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:09:58.767 DEBUG:boto:Data: 2022-09-12T20:09:58.767 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.768 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.768 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.769 DEBUG:boto:Params: {} 2022-09-12T20:09:58.770 DEBUG:boto:Token: None 2022-09-12T20:09:58.770 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/ 2022-09-12T20:09:58.771 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY= 2022-09-12T20:09:58.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY=', 'Content-Length': '0'} 2022-09-12T20:09:58.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009cf3acb60fd6d09-00631f9216-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.774 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:58.775 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:58.776 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:58.776 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:09:58.777 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:09:58.777 DEBUG:boto:Method: GET 2022-09-12T20:09:58.778 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:09:58.779 DEBUG:boto:Data: 2022-09-12T20:09:58.779 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.780 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.780 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.781 DEBUG:boto:Params: {} 2022-09-12T20:09:58.781 DEBUG:boto:Token: None 2022-09-12T20:09:58.782 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/ 2022-09-12T20:09:58.783 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY= 2022-09-12T20:09:58.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY=', 'Content-Length': '0'} 2022-09-12T20:09:58.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e681a054f6a1ffe-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.830 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:58.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:09:58.831 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:09:58.832 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:09:58.832 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:09:58.833 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:09:58.833 DEBUG:boto:Method: GET 2022-09-12T20:09:58.834 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:09:58.834 DEBUG:boto:Data: 2022-09-12T20:09:58.835 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.836 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.836 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.837 DEBUG:boto:Params: {} 2022-09-12T20:09:58.837 DEBUG:boto:Token: None 2022-09-12T20:09:58.838 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:09:58.838 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Sjw4AhLMqbF8n9arKXohiTldGlA= 2022-09-12T20:09:58.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Sjw4AhLMqbF8n9arKXohiTldGlA=', 'Content-Length': '0'} 2022-09-12T20:09:58.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012a52c97e99100b6-00631f9216-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.842 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:09:58.843 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:09:58.843 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:09:58.844 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:09:58.844 DEBUG:boto:Method: GET 2022-09-12T20:09:58.845 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:09:58.845 DEBUG:boto:Data: 2022-09-12T20:09:58.846 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.846 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.847 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.848 DEBUG:boto:Params: {} 2022-09-12T20:09:58.848 DEBUG:boto:Token: None 2022-09-12T20:09:58.849 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:09:58.849 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Sjw4AhLMqbF8n9arKXohiTldGlA= 2022-09-12T20:09:58.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Sjw4AhLMqbF8n9arKXohiTldGlA=', 'Content-Length': '0'} 2022-09-12T20:09:58.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a709bcae4479f1a-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.853 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:09:58.854 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:09:58.854 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:09:58.855 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:09:58.855 DEBUG:boto:Method: HEAD 2022-09-12T20:09:58.856 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:09:58.857 DEBUG:boto:Data: 2022-09-12T20:09:58.857 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.858 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.858 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.859 DEBUG:boto:Params: {} 2022-09-12T20:09:58.859 DEBUG:boto:Token: None 2022-09-12T20:09:58.860 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:09:58.860 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m4u63Ve46xM/dICP3jfNg+Hng/M= 2022-09-12T20:09:58.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m4u63Ve46xM/dICP3jfNg+Hng/M=', 'Content-Length': '0'} 2022-09-12T20:09:58.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d30521381f7eab20-00631f9216-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.864 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:09:58.864 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:09:58.865 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:09:58.865 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:09:58.866 DEBUG:boto:Method: HEAD 2022-09-12T20:09:58.866 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:09:58.867 DEBUG:boto:Data: 2022-09-12T20:09:58.867 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.868 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.868 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.869 DEBUG:boto:Params: {} 2022-09-12T20:09:58.869 DEBUG:boto:Token: None 2022-09-12T20:09:58.870 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:09:58.871 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m4u63Ve46xM/dICP3jfNg+Hng/M= 2022-09-12T20:09:58.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m4u63Ve46xM/dICP3jfNg+Hng/M=', 'Content-Length': '0'} 2022-09-12T20:09:58.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a89f3c3db9dd9c6d-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:09:58.875 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:09:58.875 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:09:58.876 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:09:58.877 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:09:58.877 DEBUG:boto:Method: GET 2022-09-12T20:09:58.878 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:09:58.878 DEBUG:boto:Data: 2022-09-12T20:09:58.879 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.879 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.880 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.880 DEBUG:boto:Params: {} 2022-09-12T20:09:58.881 DEBUG:boto:Token: None 2022-09-12T20:09:58.882 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:09:58.882 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Sjw4AhLMqbF8n9arKXohiTldGlA= 2022-09-12T20:09:58.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Sjw4AhLMqbF8n9arKXohiTldGlA=', 'Content-Length': '0'} 2022-09-12T20:09:58.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ee92e4f621ac4e5-00631f9216-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.886 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:09:58.886 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:09:58.887 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:09:58.887 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:09:58.888 DEBUG:boto:Method: GET 2022-09-12T20:09:58.889 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:09:58.889 DEBUG:boto:Data: 2022-09-12T20:09:58.890 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.890 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.891 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.891 DEBUG:boto:Params: {} 2022-09-12T20:09:58.892 DEBUG:boto:Token: None 2022-09-12T20:09:58.893 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:09:58.893 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Sjw4AhLMqbF8n9arKXohiTldGlA= 2022-09-12T20:09:58.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Sjw4AhLMqbF8n9arKXohiTldGlA=', 'Content-Length': '0'} 2022-09-12T20:09:58.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be48eb63d18b0747-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.897 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:58.898 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:58.898 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:09:58.899 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:09:58.900 DEBUG:boto:Method: GET 2022-09-12T20:09:58.900 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:09:58.901 DEBUG:boto:Data: 2022-09-12T20:09:58.901 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.902 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.902 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.903 DEBUG:boto:Params: {} 2022-09-12T20:09:58.904 DEBUG:boto:Token: None 2022-09-12T20:09:58.904 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/ 2022-09-12T20:09:58.905 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY= 2022-09-12T20:09:58.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY=', 'Content-Length': '0'} 2022-09-12T20:09:58.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e9bb112249e184e-00631f9216-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.909 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:58.909 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:58.910 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:58.911 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:09:58.911 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:09:58.912 DEBUG:boto:Method: GET 2022-09-12T20:09:58.912 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:09:58.913 DEBUG:boto:Data: 2022-09-12T20:09:58.913 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.914 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.914 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.915 DEBUG:boto:Params: {} 2022-09-12T20:09:58.915 DEBUG:boto:Token: None 2022-09-12T20:09:58.916 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/ 2022-09-12T20:09:58.916 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY= 2022-09-12T20:09:58.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY=', 'Content-Length': '0'} 2022-09-12T20:09:58.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003eb73c8a64f32329-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.922 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:58.922 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:09:58.923 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:09:58.923 DEBUG:boto:Method: HEAD 2022-09-12T20:09:58.924 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:09:58.924 DEBUG:boto:Data: 2022-09-12T20:09:58.925 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.925 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.926 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.926 DEBUG:boto:Params: {} 2022-09-12T20:09:58.927 DEBUG:boto:Token: None 2022-09-12T20:09:58.928 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj92 2022-09-12T20:09:58.928 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ajg6B9Iplb1txCzTzzupyF2T4Q4= 2022-09-12T20:09:58.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ajg6B9Iplb1txCzTzzupyF2T4Q4=', 'Content-Length': '0'} 2022-09-12T20:09:58.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faf06f2d92683f2f-00631f9216-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.931 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:09:58.932 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:09:58.932 DEBUG:boto:Method: HEAD 2022-09-12T20:09:58.933 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:09:58.933 DEBUG:boto:Data: 2022-09-12T20:09:58.934 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.934 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.935 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.935 DEBUG:boto:Params: {} 2022-09-12T20:09:58.936 DEBUG:boto:Token: None 2022-09-12T20:09:58.936 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj92 2022-09-12T20:09:58.937 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ajg6B9Iplb1txCzTzzupyF2T4Q4= 2022-09-12T20:09:58.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ajg6B9Iplb1txCzTzzupyF2T4Q4=', 'Content-Length': '0'} 2022-09-12T20:09:58.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000752c943faec42d1-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:09:58.941 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:09:58.942 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:09:58.942 DEBUG:boto:Method: GET 2022-09-12T20:09:58.943 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:09:58.943 DEBUG:boto:Data: 2022-09-12T20:09:58.944 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.944 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.945 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.945 DEBUG:boto:Params: {} 2022-09-12T20:09:58.946 DEBUG:boto:Token: None 2022-09-12T20:09:58.947 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj92 2022-09-12T20:09:58.947 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N5hJCtZ4ht4Bzv5anEYYy4sE2hs= 2022-09-12T20:09:58.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N5hJCtZ4ht4Bzv5anEYYy4sE2hs=', 'Content-Length': '0'} 2022-09-12T20:09:58.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006c51f71d41273e6-00631f9216-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.951 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:09:58.951 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:09:58.952 DEBUG:boto:Method: GET 2022-09-12T20:09:58.952 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:09:58.953 DEBUG:boto:Data: 2022-09-12T20:09:58.954 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.955 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:58.955 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.956 DEBUG:boto:Params: {} 2022-09-12T20:09:58.956 DEBUG:boto:Token: None 2022-09-12T20:09:58.957 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/obj92 2022-09-12T20:09:58.958 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N5hJCtZ4ht4Bzv5anEYYy4sE2hs= 2022-09-12T20:09:58.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N5hJCtZ4ht4Bzv5anEYYy4sE2hs=', 'Content-Length': '0'} 2022-09-12T20:09:58.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc83e301fb4d79ba-00631f9216-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:58.961 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:58.962 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:58.962 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:09:58.963 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:09:58.963 DEBUG:boto:Method: GET 2022-09-12T20:09:58.964 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:09:58.964 DEBUG:boto:Data: 2022-09-12T20:09:58.965 DEBUG:boto:Headers: {} 2022-09-12T20:09:58.965 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:58.966 DEBUG:boto:Port: 8000 2022-09-12T20:09:58.966 DEBUG:boto:Params: {} 2022-09-12T20:09:58.967 DEBUG:boto:Token: None 2022-09-12T20:09:58.968 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:58 GMT /jdpiak-75/ 2022-09-12T20:09:58.968 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY= 2022-09-12T20:09:58.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T5IjcUCKQXaUvTvZWRDa7a3R1uY=', 'Content-Length': '0'} 2022-09-12T20:09:58.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099ab463c10756a82-00631f9216-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.014 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:59.015 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:59.016 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:59.016 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:09:59.017 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:09:59.017 DEBUG:boto:Method: GET 2022-09-12T20:09:59.018 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:09:59.019 DEBUG:boto:Data: 2022-09-12T20:09:59.019 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.020 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.020 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.021 DEBUG:boto:Params: {} 2022-09-12T20:09:59.021 DEBUG:boto:Token: None 2022-09-12T20:09:59.022 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/ 2022-09-12T20:09:59.023 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0= 2022-09-12T20:09:59.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0=', 'Content-Length': '0'} 2022-09-12T20:09:59.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058235d0492d391cd-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.028 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:59.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:09:59.029 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:09:59.030 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:09:59.031 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:09:59.031 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:09:59.032 DEBUG:boto:Method: GET 2022-09-12T20:09:59.032 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:09:59.033 DEBUG:boto:Data: 2022-09-12T20:09:59.033 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.034 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:59.034 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.035 DEBUG:boto:Params: {} 2022-09-12T20:09:59.036 DEBUG:boto:Token: None 2022-09-12T20:09:59.036 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:09:59.037 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J/Yb/6LVrYrMbnvzh4AygzeINLw= 2022-09-12T20:09:59.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J/Yb/6LVrYrMbnvzh4AygzeINLw=', 'Content-Length': '0'} 2022-09-12T20:09:59.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edca09911f4fbcf5-00631f9217-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.040 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:09:59.041 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:09:59.042 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:09:59.042 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:09:59.043 DEBUG:boto:Method: GET 2022-09-12T20:09:59.043 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:09:59.044 DEBUG:boto:Data: 2022-09-12T20:09:59.044 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.045 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.046 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.046 DEBUG:boto:Params: {} 2022-09-12T20:09:59.047 DEBUG:boto:Token: None 2022-09-12T20:09:59.047 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:09:59.048 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J/Yb/6LVrYrMbnvzh4AygzeINLw= 2022-09-12T20:09:59.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J/Yb/6LVrYrMbnvzh4AygzeINLw=', 'Content-Length': '0'} 2022-09-12T20:09:59.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db38abcade4dfb08-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.052 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:09:59.052 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:09:59.053 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:09:59.053 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:09:59.054 DEBUG:boto:Method: HEAD 2022-09-12T20:09:59.054 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:09:59.055 DEBUG:boto:Data: 2022-09-12T20:09:59.055 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.056 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:59.057 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.057 DEBUG:boto:Params: {} 2022-09-12T20:09:59.058 DEBUG:boto:Token: None 2022-09-12T20:09:59.058 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:09:59.059 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e8tNWlnjJoGjG0oHYqEfTkFd3b4= 2022-09-12T20:09:59.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e8tNWlnjJoGjG0oHYqEfTkFd3b4=', 'Content-Length': '0'} 2022-09-12T20:09:59.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005de20d9d5050eeec-00631f9217-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.062 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:09:59.063 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:09:59.063 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:09:59.064 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:09:59.064 DEBUG:boto:Method: HEAD 2022-09-12T20:09:59.065 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:09:59.066 DEBUG:boto:Data: 2022-09-12T20:09:59.066 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.067 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.067 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.068 DEBUG:boto:Params: {} 2022-09-12T20:09:59.068 DEBUG:boto:Token: None 2022-09-12T20:09:59.069 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:09:59.069 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e8tNWlnjJoGjG0oHYqEfTkFd3b4= 2022-09-12T20:09:59.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e8tNWlnjJoGjG0oHYqEfTkFd3b4=', 'Content-Length': '0'} 2022-09-12T20:09:59.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fd2c1336bc98544-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:09:59.074 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:09:59.074 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:09:59.075 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:09:59.075 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:09:59.076 DEBUG:boto:Method: GET 2022-09-12T20:09:59.076 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:09:59.077 DEBUG:boto:Data: 2022-09-12T20:09:59.077 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.078 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:59.078 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.079 DEBUG:boto:Params: {} 2022-09-12T20:09:59.080 DEBUG:boto:Token: None 2022-09-12T20:09:59.080 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:09:59.081 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J/Yb/6LVrYrMbnvzh4AygzeINLw= 2022-09-12T20:09:59.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J/Yb/6LVrYrMbnvzh4AygzeINLw=', 'Content-Length': '0'} 2022-09-12T20:09:59.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f419e61dd05b91a-00631f9217-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.084 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:09:59.085 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:09:59.085 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:09:59.086 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:09:59.086 DEBUG:boto:Method: GET 2022-09-12T20:09:59.087 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:09:59.087 DEBUG:boto:Data: 2022-09-12T20:09:59.088 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.088 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.089 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.090 DEBUG:boto:Params: {} 2022-09-12T20:09:59.090 DEBUG:boto:Token: None 2022-09-12T20:09:59.091 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:09:59.091 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J/Yb/6LVrYrMbnvzh4AygzeINLw= 2022-09-12T20:09:59.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J/Yb/6LVrYrMbnvzh4AygzeINLw=', 'Content-Length': '0'} 2022-09-12T20:09:59.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003566660ad5d3e77-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.138 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:59.139 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:59.139 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:09:59.140 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:09:59.140 DEBUG:boto:Method: GET 2022-09-12T20:09:59.141 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:09:59.141 DEBUG:boto:Data: 2022-09-12T20:09:59.142 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.142 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:59.143 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.143 DEBUG:boto:Params: {} 2022-09-12T20:09:59.144 DEBUG:boto:Token: None 2022-09-12T20:09:59.145 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/ 2022-09-12T20:09:59.145 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0= 2022-09-12T20:09:59.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0=', 'Content-Length': '0'} 2022-09-12T20:09:59.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c3de1df347ac2da-00631f9217-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.149 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:59.150 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:59.151 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:59.151 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:09:59.152 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:09:59.152 DEBUG:boto:Method: GET 2022-09-12T20:09:59.153 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:09:59.153 DEBUG:boto:Data: 2022-09-12T20:09:59.154 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.154 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.155 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.155 DEBUG:boto:Params: {} 2022-09-12T20:09:59.156 DEBUG:boto:Token: None 2022-09-12T20:09:59.157 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/ 2022-09-12T20:09:59.157 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0= 2022-09-12T20:09:59.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0=', 'Content-Length': '0'} 2022-09-12T20:09:59.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003080a44a59425215-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.162 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:59.163 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:09:59.163 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:09:59.164 DEBUG:boto:Method: HEAD 2022-09-12T20:09:59.164 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:09:59.165 DEBUG:boto:Data: 2022-09-12T20:09:59.165 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.166 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:59.166 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.167 DEBUG:boto:Params: {} 2022-09-12T20:09:59.168 DEBUG:boto:Token: None 2022-09-12T20:09:59.168 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj95 2022-09-12T20:09:59.169 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wPi7nEsgEufO7VnkF91a+lJi9jw= 2022-09-12T20:09:59.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wPi7nEsgEufO7VnkF91a+lJi9jw=', 'Content-Length': '0'} 2022-09-12T20:09:59.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1976a3311203337-00631f9217-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.172 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:09:59.173 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:09:59.173 DEBUG:boto:Method: HEAD 2022-09-12T20:09:59.174 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:09:59.174 DEBUG:boto:Data: 2022-09-12T20:09:59.175 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.175 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.176 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.176 DEBUG:boto:Params: {} 2022-09-12T20:09:59.177 DEBUG:boto:Token: None 2022-09-12T20:09:59.177 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj95 2022-09-12T20:09:59.178 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wPi7nEsgEufO7VnkF91a+lJi9jw= 2022-09-12T20:09:59.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wPi7nEsgEufO7VnkF91a+lJi9jw=', 'Content-Length': '0'} 2022-09-12T20:09:59.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061b872db2218f0c8-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:09:59.182 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:09:59.182 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:09:59.183 DEBUG:boto:Method: GET 2022-09-12T20:09:59.184 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:09:59.184 DEBUG:boto:Data: 2022-09-12T20:09:59.185 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.185 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:59.186 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.186 DEBUG:boto:Params: {} 2022-09-12T20:09:59.187 DEBUG:boto:Token: None 2022-09-12T20:09:59.187 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj95 2022-09-12T20:09:59.188 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YPksbT83q36kOnxtKHod4C8H1Zw= 2022-09-12T20:09:59.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YPksbT83q36kOnxtKHod4C8H1Zw=', 'Content-Length': '0'} 2022-09-12T20:09:59.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000826db82b58f78ec1-00631f9217-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.192 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:09:59.192 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:09:59.193 DEBUG:boto:Method: GET 2022-09-12T20:09:59.193 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:09:59.194 DEBUG:boto:Data: 2022-09-12T20:09:59.194 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.195 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.195 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.196 DEBUG:boto:Params: {} 2022-09-12T20:09:59.197 DEBUG:boto:Token: None 2022-09-12T20:09:59.197 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj95 2022-09-12T20:09:59.198 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YPksbT83q36kOnxtKHod4C8H1Zw= 2022-09-12T20:09:59.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YPksbT83q36kOnxtKHod4C8H1Zw=', 'Content-Length': '0'} 2022-09-12T20:09:59.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000809e9081d1ca5967-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.201 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:59.202 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:59.202 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:09:59.203 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:09:59.203 DEBUG:boto:Method: GET 2022-09-12T20:09:59.204 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:09:59.204 DEBUG:boto:Data: 2022-09-12T20:09:59.205 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.206 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:09:59.206 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.207 DEBUG:boto:Params: {} 2022-09-12T20:09:59.207 DEBUG:boto:Token: None 2022-09-12T20:09:59.208 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/ 2022-09-12T20:09:59.208 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0= 2022-09-12T20:09:59.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0=', 'Content-Length': '0'} 2022-09-12T20:09:59.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004dd118b87e1bb76-00631f9217-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.212 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:59.213 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:59.214 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:59.214 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:09:59.215 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:09:59.215 DEBUG:boto:Method: GET 2022-09-12T20:09:59.216 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:09:59.216 DEBUG:boto:Data: 2022-09-12T20:09:59.217 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.217 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.218 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.219 DEBUG:boto:Params: {} 2022-09-12T20:09:59.219 DEBUG:boto:Token: None 2022-09-12T20:09:59.220 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/ 2022-09-12T20:09:59.220 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0= 2022-09-12T20:09:59.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0=', 'Content-Length': '0'} 2022-09-12T20:09:59.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d2bb3b82b9a70f0-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.225 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:59.226 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-75 zones={test-zone2, test-zone1} 2022-09-12T20:09:59.227 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-75 zones={test-zone3, test-zone1} 2022-09-12T20:09:59.227 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:59.228 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:59.228 DEBUG:boto:Method: HEAD 2022-09-12T20:09:59.229 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:09:59.229 DEBUG:boto:Data: 2022-09-12T20:09:59.230 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.230 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:09:59.231 DEBUG:boto:Port: 8001 2022-09-12T20:09:59.231 DEBUG:boto:Params: {} 2022-09-12T20:09:59.232 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T20:09:59.232 DEBUG:boto:Token: None 2022-09-12T20:09:59.233 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/ 2022-09-12T20:09:59.234 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zg6LjLwEXuqVlWFaAN/wqA/RdJI= 2022-09-12T20:09:59.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zg6LjLwEXuqVlWFaAN/wqA/RdJI=', 'Content-Length': '0'} 2022-09-12T20:09:59.238 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037c4fedced5d7f62-00631f9217-1826-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.238 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:59.239 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:59.240 DEBUG:boto:Method: HEAD 2022-09-12T20:09:59.240 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:09:59.241 DEBUG:boto:Data: 2022-09-12T20:09:59.241 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.242 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.242 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.243 DEBUG:boto:Params: {} 2022-09-12T20:09:59.243 DEBUG:boto:Token: None 2022-09-12T20:09:59.244 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/ 2022-09-12T20:09:59.244 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zg6LjLwEXuqVlWFaAN/wqA/RdJI= 2022-09-12T20:09:59.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zg6LjLwEXuqVlWFaAN/wqA/RdJI=', 'Content-Length': '0'} 2022-09-12T20:09:59.248 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a895e09d6d464024-00631f9217-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.249 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T20:09:59.249 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:59.250 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:59.250 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:09:59.251 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:09:59.251 DEBUG:boto:Method: GET 2022-09-12T20:09:59.252 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:09:59.252 DEBUG:boto:Data: 2022-09-12T20:09:59.253 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.253 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:09:59.254 DEBUG:boto:Port: 8001 2022-09-12T20:09:59.254 DEBUG:boto:Params: {} 2022-09-12T20:09:59.255 DEBUG:boto:Token: None 2022-09-12T20:09:59.255 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/?versions 2022-09-12T20:09:59.256 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2nYRUwEQk4e88JEGGv/NZhRGr8k= 2022-09-12T20:09:59.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2nYRUwEQk4e88JEGGv/NZhRGr8k=', 'Content-Length': '0'} 2022-09-12T20:09:59.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a802f19412806b4-00631f9217-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.260 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:59.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:09:59.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-12T20:09:59.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:09:59.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:09:59.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:09:59.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-12T20:09:59.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:09:59.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:09:59.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:09:59.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:09:59.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-12T20:09:59.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:09:59.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:09:59.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:09:59.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:09:59.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-12T20:09:59.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:09:59.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:09:59.271 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-12T20:09:59.271 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:09:59.272 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T20:09:59.272 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:59.273 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:59.273 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:09:59.274 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:09:59.274 DEBUG:boto:Method: GET 2022-09-12T20:09:59.275 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:09:59.275 DEBUG:boto:Data: 2022-09-12T20:09:59.276 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.276 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.277 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.277 DEBUG:boto:Params: {} 2022-09-12T20:09:59.278 DEBUG:boto:Token: None 2022-09-12T20:09:59.279 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/?versions 2022-09-12T20:09:59.279 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2nYRUwEQk4e88JEGGv/NZhRGr8k= 2022-09-12T20:09:59.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2nYRUwEQk4e88JEGGv/NZhRGr8k=', 'Content-Length': '0'} 2022-09-12T20:09:59.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b05f1cafa747bb3-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.284 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:59.286 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:09:59.286 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-12T20:09:59.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:09:59.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:09:59.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:09:59.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-12T20:09:59.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:09:59.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:09:59.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:09:59.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:09:59.291 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-12T20:09:59.291 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:09:59.292 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:09:59.292 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:09:59.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:09:59.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-12T20:09:59.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:09:59.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:09:59.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-12T20:09:59.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:09:59.296 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:59.297 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:59.297 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:09:59.298 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:09:59.298 DEBUG:boto:Method: GET 2022-09-12T20:09:59.299 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:09:59.299 DEBUG:boto:Data: 2022-09-12T20:09:59.300 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.300 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:09:59.301 DEBUG:boto:Port: 8001 2022-09-12T20:09:59.301 DEBUG:boto:Params: {} 2022-09-12T20:09:59.302 DEBUG:boto:Token: None 2022-09-12T20:09:59.302 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/?versions 2022-09-12T20:09:59.303 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2nYRUwEQk4e88JEGGv/NZhRGr8k= 2022-09-12T20:09:59.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2nYRUwEQk4e88JEGGv/NZhRGr8k=', 'Content-Length': '0'} 2022-09-12T20:09:59.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000084e7b498fc689ea-00631f9217-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.307 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:59.309 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:59.310 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:59.310 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:09:59.311 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:09:59.311 DEBUG:boto:Method: GET 2022-09-12T20:09:59.312 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:09:59.313 DEBUG:boto:Data: 2022-09-12T20:09:59.313 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.314 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.314 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.315 DEBUG:boto:Params: {} 2022-09-12T20:09:59.316 DEBUG:boto:Token: None 2022-09-12T20:09:59.316 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/?versions 2022-09-12T20:09:59.317 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2nYRUwEQk4e88JEGGv/NZhRGr8k= 2022-09-12T20:09:59.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2nYRUwEQk4e88JEGGv/NZhRGr8k=', 'Content-Length': '0'} 2022-09-12T20:09:59.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d9f23b05f45ff54-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.322 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:59.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:09:59.325 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:09:59.325 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:09:59.326 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:09:59.327 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:09:59.327 DEBUG:boto:Method: GET 2022-09-12T20:09:59.328 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:09:59.328 DEBUG:boto:Data: 2022-09-12T20:09:59.329 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.329 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:09:59.330 DEBUG:boto:Port: 8001 2022-09-12T20:09:59.330 DEBUG:boto:Params: {} 2022-09-12T20:09:59.331 DEBUG:boto:Token: None 2022-09-12T20:09:59.332 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:09:59.332 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5ESwldxO8LbjAIzNNQ7scy8YkDU= 2022-09-12T20:09:59.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5ESwldxO8LbjAIzNNQ7scy8YkDU=', 'Content-Length': '0'} 2022-09-12T20:09:59.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bd870fece65921c-00631f9217-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.336 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:09:59.337 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:09:59.337 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:09:59.338 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:09:59.338 DEBUG:boto:Method: GET 2022-09-12T20:09:59.339 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:09:59.339 DEBUG:boto:Data: 2022-09-12T20:09:59.340 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.341 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.341 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.342 DEBUG:boto:Params: {} 2022-09-12T20:09:59.342 DEBUG:boto:Token: None 2022-09-12T20:09:59.343 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:09:59.344 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5ESwldxO8LbjAIzNNQ7scy8YkDU= 2022-09-12T20:09:59.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5ESwldxO8LbjAIzNNQ7scy8YkDU=', 'Content-Length': '0'} 2022-09-12T20:09:59.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005981d7d7510062a-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.394 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:09:59.395 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:09:59.395 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:09:59.396 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:09:59.396 DEBUG:boto:Method: HEAD 2022-09-12T20:09:59.397 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:09:59.397 DEBUG:boto:Data: 2022-09-12T20:09:59.398 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.398 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:09:59.399 DEBUG:boto:Port: 8001 2022-09-12T20:09:59.399 DEBUG:boto:Params: {} 2022-09-12T20:09:59.400 DEBUG:boto:Token: None 2022-09-12T20:09:59.401 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:09:59.401 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sVnSnEzCtx/Vt0R9vaw8OZsN1D8= 2022-09-12T20:09:59.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sVnSnEzCtx/Vt0R9vaw8OZsN1D8=', 'Content-Length': '0'} 2022-09-12T20:09:59.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6b2d71613c5db12-00631f9217-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.405 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:09:59.405 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:09:59.406 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:09:59.406 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:09:59.407 DEBUG:boto:Method: HEAD 2022-09-12T20:09:59.407 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:09:59.408 DEBUG:boto:Data: 2022-09-12T20:09:59.408 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.409 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.410 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.410 DEBUG:boto:Params: {} 2022-09-12T20:09:59.411 DEBUG:boto:Token: None 2022-09-12T20:09:59.411 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:09:59.412 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sVnSnEzCtx/Vt0R9vaw8OZsN1D8= 2022-09-12T20:09:59.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sVnSnEzCtx/Vt0R9vaw8OZsN1D8=', 'Content-Length': '0'} 2022-09-12T20:09:59.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079a15301361d87f0-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:09:59.416 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:09:59.417 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:09:59.417 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:09:59.418 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:09:59.418 DEBUG:boto:Method: GET 2022-09-12T20:09:59.419 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:09:59.419 DEBUG:boto:Data: 2022-09-12T20:09:59.420 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.420 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:09:59.421 DEBUG:boto:Port: 8001 2022-09-12T20:09:59.421 DEBUG:boto:Params: {} 2022-09-12T20:09:59.422 DEBUG:boto:Token: None 2022-09-12T20:09:59.423 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:09:59.423 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5ESwldxO8LbjAIzNNQ7scy8YkDU= 2022-09-12T20:09:59.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5ESwldxO8LbjAIzNNQ7scy8YkDU=', 'Content-Length': '0'} 2022-09-12T20:09:59.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f0450a36c84c44a-00631f9217-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.427 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:09:59.427 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:09:59.428 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:09:59.428 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:09:59.429 DEBUG:boto:Method: GET 2022-09-12T20:09:59.429 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:09:59.430 DEBUG:boto:Data: 2022-09-12T20:09:59.430 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.431 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.432 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.432 DEBUG:boto:Params: {} 2022-09-12T20:09:59.433 DEBUG:boto:Token: None 2022-09-12T20:09:59.433 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:09:59.434 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5ESwldxO8LbjAIzNNQ7scy8YkDU= 2022-09-12T20:09:59.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5ESwldxO8LbjAIzNNQ7scy8YkDU=', 'Content-Length': '0'} 2022-09-12T20:09:59.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022e38450827a17da-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.437 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:59.438 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:59.438 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:09:59.439 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:09:59.439 DEBUG:boto:Method: GET 2022-09-12T20:09:59.440 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:09:59.440 DEBUG:boto:Data: 2022-09-12T20:09:59.441 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.441 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:09:59.442 DEBUG:boto:Port: 8001 2022-09-12T20:09:59.442 DEBUG:boto:Params: {} 2022-09-12T20:09:59.443 DEBUG:boto:Token: None 2022-09-12T20:09:59.444 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/ 2022-09-12T20:09:59.444 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0= 2022-09-12T20:09:59.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0=', 'Content-Length': '0'} 2022-09-12T20:09:59.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ca4be76551779d9-00631f9217-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.490 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:59.491 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:59.492 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:59.492 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:09:59.493 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:09:59.493 DEBUG:boto:Method: GET 2022-09-12T20:09:59.494 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:09:59.494 DEBUG:boto:Data: 2022-09-12T20:09:59.495 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.495 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.496 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.496 DEBUG:boto:Params: {} 2022-09-12T20:09:59.497 DEBUG:boto:Token: None 2022-09-12T20:09:59.497 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/ 2022-09-12T20:09:59.498 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0= 2022-09-12T20:09:59.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0=', 'Content-Length': '0'} 2022-09-12T20:09:59.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f1c4af6f88c7f25-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.503 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:59.504 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:09:59.504 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:09:59.505 DEBUG:boto:Method: HEAD 2022-09-12T20:09:59.505 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:09:59.506 DEBUG:boto:Data: 2022-09-12T20:09:59.506 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.507 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:09:59.507 DEBUG:boto:Port: 8001 2022-09-12T20:09:59.508 DEBUG:boto:Params: {} 2022-09-12T20:09:59.508 DEBUG:boto:Token: None 2022-09-12T20:09:59.509 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj11 2022-09-12T20:09:59.510 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8hair/lR8ynkLi5jRDEXOUrNEwQ= 2022-09-12T20:09:59.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8hair/lR8ynkLi5jRDEXOUrNEwQ=', 'Content-Length': '0'} 2022-09-12T20:09:59.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aab1870d2f6f6d08-00631f9217-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.513 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:09:59.514 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:09:59.514 DEBUG:boto:Method: HEAD 2022-09-12T20:09:59.515 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:09:59.515 DEBUG:boto:Data: 2022-09-12T20:09:59.516 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.516 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.517 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.517 DEBUG:boto:Params: {} 2022-09-12T20:09:59.518 DEBUG:boto:Token: None 2022-09-12T20:09:59.518 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj11 2022-09-12T20:09:59.519 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8hair/lR8ynkLi5jRDEXOUrNEwQ= 2022-09-12T20:09:59.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8hair/lR8ynkLi5jRDEXOUrNEwQ=', 'Content-Length': '0'} 2022-09-12T20:09:59.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cab731985758c001-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:09:59.523 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:09:59.523 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:09:59.524 DEBUG:boto:Method: GET 2022-09-12T20:09:59.524 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:09:59.525 DEBUG:boto:Data: 2022-09-12T20:09:59.525 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.526 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:09:59.526 DEBUG:boto:Port: 8001 2022-09-12T20:09:59.527 DEBUG:boto:Params: {} 2022-09-12T20:09:59.528 DEBUG:boto:Token: None 2022-09-12T20:09:59.528 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj11 2022-09-12T20:09:59.529 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4pLYA/5qVK2i0d82Jp3gWsjpx2g= 2022-09-12T20:09:59.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4pLYA/5qVK2i0d82Jp3gWsjpx2g=', 'Content-Length': '0'} 2022-09-12T20:09:59.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e4b32fb9cd3bf82-00631f9217-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.532 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:09:59.533 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:09:59.533 DEBUG:boto:Method: GET 2022-09-12T20:09:59.534 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:09:59.535 DEBUG:boto:Data: 2022-09-12T20:09:59.535 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.536 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.536 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.537 DEBUG:boto:Params: {} 2022-09-12T20:09:59.537 DEBUG:boto:Token: None 2022-09-12T20:09:59.538 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj11 2022-09-12T20:09:59.538 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4pLYA/5qVK2i0d82Jp3gWsjpx2g= 2022-09-12T20:09:59.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4pLYA/5qVK2i0d82Jp3gWsjpx2g=', 'Content-Length': '0'} 2022-09-12T20:09:59.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d67ff0637906b31-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.582 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:59.583 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:59.583 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:09:59.584 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:09:59.584 DEBUG:boto:Method: GET 2022-09-12T20:09:59.585 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:09:59.585 DEBUG:boto:Data: 2022-09-12T20:09:59.586 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.586 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:09:59.587 DEBUG:boto:Port: 8001 2022-09-12T20:09:59.587 DEBUG:boto:Params: {} 2022-09-12T20:09:59.588 DEBUG:boto:Token: None 2022-09-12T20:09:59.589 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/ 2022-09-12T20:09:59.589 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0= 2022-09-12T20:09:59.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0=', 'Content-Length': '0'} 2022-09-12T20:09:59.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a0fb6b5d30bb80e-00631f9217-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.593 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:59.594 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:59.595 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:59.595 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:09:59.596 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:09:59.596 DEBUG:boto:Method: GET 2022-09-12T20:09:59.597 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:09:59.597 DEBUG:boto:Data: 2022-09-12T20:09:59.598 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.598 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.599 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.599 DEBUG:boto:Params: {} 2022-09-12T20:09:59.600 DEBUG:boto:Token: None 2022-09-12T20:09:59.600 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/ 2022-09-12T20:09:59.601 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0= 2022-09-12T20:09:59.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0=', 'Content-Length': '0'} 2022-09-12T20:09:59.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da9961c5ad6205f3-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.606 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:59.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:09:59.607 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:09:59.608 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:09:59.608 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:09:59.609 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:09:59.609 DEBUG:boto:Method: GET 2022-09-12T20:09:59.610 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:09:59.610 DEBUG:boto:Data: 2022-09-12T20:09:59.611 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.612 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:09:59.612 DEBUG:boto:Port: 8001 2022-09-12T20:09:59.613 DEBUG:boto:Params: {} 2022-09-12T20:09:59.614 DEBUG:boto:Token: None 2022-09-12T20:09:59.614 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:09:59.615 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9By3Z/V/sHZhuCW7yk1wAKdVOXU= 2022-09-12T20:09:59.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9By3Z/V/sHZhuCW7yk1wAKdVOXU=', 'Content-Length': '0'} 2022-09-12T20:09:59.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007973151f7f14fe6c-00631f9217-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.619 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:09:59.619 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:09:59.620 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:09:59.620 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:09:59.621 DEBUG:boto:Method: GET 2022-09-12T20:09:59.621 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:09:59.622 DEBUG:boto:Data: 2022-09-12T20:09:59.622 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.623 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.623 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.624 DEBUG:boto:Params: {} 2022-09-12T20:09:59.624 DEBUG:boto:Token: None 2022-09-12T20:09:59.625 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:09:59.625 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9By3Z/V/sHZhuCW7yk1wAKdVOXU= 2022-09-12T20:09:59.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9By3Z/V/sHZhuCW7yk1wAKdVOXU=', 'Content-Length': '0'} 2022-09-12T20:09:59.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afc40e2021e936ba-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.629 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:09:59.630 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:09:59.630 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:09:59.631 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:09:59.631 DEBUG:boto:Method: HEAD 2022-09-12T20:09:59.632 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:09:59.632 DEBUG:boto:Data: 2022-09-12T20:09:59.633 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.633 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:09:59.634 DEBUG:boto:Port: 8001 2022-09-12T20:09:59.634 DEBUG:boto:Params: {} 2022-09-12T20:09:59.635 DEBUG:boto:Token: None 2022-09-12T20:09:59.635 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:09:59.636 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dZEMj9gm5rmXFHZo4Lizr3GHozo= 2022-09-12T20:09:59.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dZEMj9gm5rmXFHZo4Lizr3GHozo=', 'Content-Length': '0'} 2022-09-12T20:09:59.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5807782944d0a2c-00631f9217-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.639 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:09:59.640 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:09:59.640 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:09:59.641 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:09:59.641 DEBUG:boto:Method: HEAD 2022-09-12T20:09:59.642 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:09:59.642 DEBUG:boto:Data: 2022-09-12T20:09:59.643 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.643 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.644 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.644 DEBUG:boto:Params: {} 2022-09-12T20:09:59.645 DEBUG:boto:Token: None 2022-09-12T20:09:59.645 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:09:59.646 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dZEMj9gm5rmXFHZo4Lizr3GHozo= 2022-09-12T20:09:59.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dZEMj9gm5rmXFHZo4Lizr3GHozo=', 'Content-Length': '0'} 2022-09-12T20:09:59.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a842bae4c8f48ca-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:09:59.650 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:09:59.650 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:09:59.651 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:09:59.652 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:09:59.652 DEBUG:boto:Method: GET 2022-09-12T20:09:59.653 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:09:59.653 DEBUG:boto:Data: 2022-09-12T20:09:59.654 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.654 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:09:59.655 DEBUG:boto:Port: 8001 2022-09-12T20:09:59.655 DEBUG:boto:Params: {} 2022-09-12T20:09:59.656 DEBUG:boto:Token: None 2022-09-12T20:09:59.656 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:09:59.657 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9By3Z/V/sHZhuCW7yk1wAKdVOXU= 2022-09-12T20:09:59.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9By3Z/V/sHZhuCW7yk1wAKdVOXU=', 'Content-Length': '0'} 2022-09-12T20:09:59.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e9d0c1316959451-00631f9217-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.702 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:09:59.702 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:09:59.703 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:09:59.703 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:09:59.704 DEBUG:boto:Method: GET 2022-09-12T20:09:59.705 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:09:59.705 DEBUG:boto:Data: 2022-09-12T20:09:59.706 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.706 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.707 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.707 DEBUG:boto:Params: {} 2022-09-12T20:09:59.708 DEBUG:boto:Token: None 2022-09-12T20:09:59.708 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:09:59.709 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9By3Z/V/sHZhuCW7yk1wAKdVOXU= 2022-09-12T20:09:59.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9By3Z/V/sHZhuCW7yk1wAKdVOXU=', 'Content-Length': '0'} 2022-09-12T20:09:59.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f68098c50f8172a-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.712 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:59.713 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:59.713 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:09:59.714 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:09:59.714 DEBUG:boto:Method: GET 2022-09-12T20:09:59.715 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:09:59.716 DEBUG:boto:Data: 2022-09-12T20:09:59.716 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.717 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:09:59.717 DEBUG:boto:Port: 8001 2022-09-12T20:09:59.718 DEBUG:boto:Params: {} 2022-09-12T20:09:59.718 DEBUG:boto:Token: None 2022-09-12T20:09:59.719 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/ 2022-09-12T20:09:59.719 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0= 2022-09-12T20:09:59.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0=', 'Content-Length': '0'} 2022-09-12T20:09:59.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1e7c2ede5c09469-00631f9217-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.724 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:59.724 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:59.725 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:59.726 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:09:59.726 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:09:59.727 DEBUG:boto:Method: GET 2022-09-12T20:09:59.727 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:09:59.728 DEBUG:boto:Data: 2022-09-12T20:09:59.728 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.729 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.729 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.730 DEBUG:boto:Params: {} 2022-09-12T20:09:59.730 DEBUG:boto:Token: None 2022-09-12T20:09:59.731 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/ 2022-09-12T20:09:59.731 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0= 2022-09-12T20:09:59.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0=', 'Content-Length': '0'} 2022-09-12T20:09:59.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d2de090092e51db-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.736 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:59.737 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:09:59.738 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:09:59.738 DEBUG:boto:Method: HEAD 2022-09-12T20:09:59.739 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:09:59.739 DEBUG:boto:Data: 2022-09-12T20:09:59.740 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.740 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:09:59.741 DEBUG:boto:Port: 8001 2022-09-12T20:09:59.741 DEBUG:boto:Params: {} 2022-09-12T20:09:59.742 DEBUG:boto:Token: None 2022-09-12T20:09:59.742 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj115 2022-09-12T20:09:59.743 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dBlBwrPjaQC3Tss35x0C3wBa9/I= 2022-09-12T20:09:59.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dBlBwrPjaQC3Tss35x0C3wBa9/I=', 'Content-Length': '0'} 2022-09-12T20:09:59.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c77e37ef411ed533-00631f9217-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.746 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:09:59.747 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:09:59.747 DEBUG:boto:Method: HEAD 2022-09-12T20:09:59.748 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:09:59.748 DEBUG:boto:Data: 2022-09-12T20:09:59.749 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.749 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.750 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.750 DEBUG:boto:Params: {} 2022-09-12T20:09:59.751 DEBUG:boto:Token: None 2022-09-12T20:09:59.751 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj115 2022-09-12T20:09:59.752 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dBlBwrPjaQC3Tss35x0C3wBa9/I= 2022-09-12T20:09:59.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dBlBwrPjaQC3Tss35x0C3wBa9/I=', 'Content-Length': '0'} 2022-09-12T20:09:59.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000816b8bf071b493aa-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:09:59.756 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:09:59.757 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:09:59.757 DEBUG:boto:Method: GET 2022-09-12T20:09:59.758 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:09:59.758 DEBUG:boto:Data: 2022-09-12T20:09:59.759 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.759 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:09:59.760 DEBUG:boto:Port: 8001 2022-09-12T20:09:59.760 DEBUG:boto:Params: {} 2022-09-12T20:09:59.761 DEBUG:boto:Token: None 2022-09-12T20:09:59.761 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj115 2022-09-12T20:09:59.762 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mC+04q+8eFuVvh5ohORXMSmIam4= 2022-09-12T20:09:59.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mC+04q+8eFuVvh5ohORXMSmIam4=', 'Content-Length': '0'} 2022-09-12T20:09:59.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7eccb9bd6a77f89-00631f9217-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.766 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:09:59.767 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:09:59.767 DEBUG:boto:Method: GET 2022-09-12T20:09:59.768 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:09:59.768 DEBUG:boto:Data: 2022-09-12T20:09:59.769 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.769 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.770 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.770 DEBUG:boto:Params: {} 2022-09-12T20:09:59.771 DEBUG:boto:Token: None 2022-09-12T20:09:59.771 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj115 2022-09-12T20:09:59.772 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mC+04q+8eFuVvh5ohORXMSmIam4= 2022-09-12T20:09:59.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mC+04q+8eFuVvh5ohORXMSmIam4=', 'Content-Length': '0'} 2022-09-12T20:09:59.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006aa2542a63ffc968-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.818 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:59.819 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:59.819 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:09:59.820 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:09:59.821 DEBUG:boto:Method: GET 2022-09-12T20:09:59.821 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:09:59.822 DEBUG:boto:Data: 2022-09-12T20:09:59.822 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.823 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:09:59.823 DEBUG:boto:Port: 8001 2022-09-12T20:09:59.824 DEBUG:boto:Params: {} 2022-09-12T20:09:59.825 DEBUG:boto:Token: None 2022-09-12T20:09:59.825 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/ 2022-09-12T20:09:59.826 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0= 2022-09-12T20:09:59.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0=', 'Content-Length': '0'} 2022-09-12T20:09:59.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd5aceb2f408cc48-00631f9217-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.830 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:59.831 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:59.832 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:59.832 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:09:59.833 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:09:59.833 DEBUG:boto:Method: GET 2022-09-12T20:09:59.834 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:09:59.835 DEBUG:boto:Data: 2022-09-12T20:09:59.835 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.836 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.836 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.837 DEBUG:boto:Params: {} 2022-09-12T20:09:59.837 DEBUG:boto:Token: None 2022-09-12T20:09:59.838 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/ 2022-09-12T20:09:59.838 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0= 2022-09-12T20:09:59.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0=', 'Content-Length': '0'} 2022-09-12T20:09:59.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063c031522ecce643-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.843 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:59.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:09:59.845 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:09:59.845 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:09:59.846 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:09:59.846 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:09:59.847 DEBUG:boto:Method: GET 2022-09-12T20:09:59.847 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:09:59.848 DEBUG:boto:Data: 2022-09-12T20:09:59.848 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.849 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:09:59.849 DEBUG:boto:Port: 8001 2022-09-12T20:09:59.850 DEBUG:boto:Params: {} 2022-09-12T20:09:59.851 DEBUG:boto:Token: None 2022-09-12T20:09:59.851 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:09:59.852 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WO0dnCQDXpGBPlSSfavpMc1HWHM= 2022-09-12T20:09:59.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WO0dnCQDXpGBPlSSfavpMc1HWHM=', 'Content-Length': '0'} 2022-09-12T20:09:59.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065147b81b320a7db-00631f9217-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.856 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:09:59.856 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:09:59.857 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:09:59.857 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:09:59.858 DEBUG:boto:Method: GET 2022-09-12T20:09:59.858 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:09:59.859 DEBUG:boto:Data: 2022-09-12T20:09:59.860 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.860 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.861 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.861 DEBUG:boto:Params: {} 2022-09-12T20:09:59.862 DEBUG:boto:Token: None 2022-09-12T20:09:59.862 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:09:59.863 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WO0dnCQDXpGBPlSSfavpMc1HWHM= 2022-09-12T20:09:59.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WO0dnCQDXpGBPlSSfavpMc1HWHM=', 'Content-Length': '0'} 2022-09-12T20:09:59.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023a9df54bc6373cd-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.867 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:09:59.867 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:09:59.868 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:09:59.868 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:09:59.869 DEBUG:boto:Method: HEAD 2022-09-12T20:09:59.869 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:09:59.870 DEBUG:boto:Data: 2022-09-12T20:09:59.870 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.871 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:09:59.871 DEBUG:boto:Port: 8001 2022-09-12T20:09:59.872 DEBUG:boto:Params: {} 2022-09-12T20:09:59.872 DEBUG:boto:Token: None 2022-09-12T20:09:59.873 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:09:59.873 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZkNqQl1WMmKA90uvdqF0CSoR1JY= 2022-09-12T20:09:59.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZkNqQl1WMmKA90uvdqF0CSoR1JY=', 'Content-Length': '0'} 2022-09-12T20:09:59.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008264dbda4daa6c4e-00631f9217-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.877 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:09:59.877 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:09:59.878 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:09:59.878 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:09:59.879 DEBUG:boto:Method: HEAD 2022-09-12T20:09:59.880 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:09:59.880 DEBUG:boto:Data: 2022-09-12T20:09:59.881 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.881 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.882 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.882 DEBUG:boto:Params: {} 2022-09-12T20:09:59.883 DEBUG:boto:Token: None 2022-09-12T20:09:59.883 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:09:59.884 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZkNqQl1WMmKA90uvdqF0CSoR1JY= 2022-09-12T20:09:59.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZkNqQl1WMmKA90uvdqF0CSoR1JY=', 'Content-Length': '0'} 2022-09-12T20:09:59.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bf28e2eb2e08b7f-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:09:59.888 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:09:59.888 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:09:59.889 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:09:59.889 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:09:59.890 DEBUG:boto:Method: GET 2022-09-12T20:09:59.890 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:09:59.891 DEBUG:boto:Data: 2022-09-12T20:09:59.891 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.892 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:09:59.892 DEBUG:boto:Port: 8001 2022-09-12T20:09:59.893 DEBUG:boto:Params: {} 2022-09-12T20:09:59.893 DEBUG:boto:Token: None 2022-09-12T20:09:59.894 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:09:59.895 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WO0dnCQDXpGBPlSSfavpMc1HWHM= 2022-09-12T20:09:59.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WO0dnCQDXpGBPlSSfavpMc1HWHM=', 'Content-Length': '0'} 2022-09-12T20:09:59.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec1db47e73f6ac65-00631f9217-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.938 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:09:59.938 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:09:59.939 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:09:59.940 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:09:59.940 DEBUG:boto:Method: GET 2022-09-12T20:09:59.941 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:09:59.941 DEBUG:boto:Data: 2022-09-12T20:09:59.942 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.942 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.943 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.943 DEBUG:boto:Params: {} 2022-09-12T20:09:59.944 DEBUG:boto:Token: None 2022-09-12T20:09:59.945 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:09:59.945 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WO0dnCQDXpGBPlSSfavpMc1HWHM= 2022-09-12T20:09:59.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WO0dnCQDXpGBPlSSfavpMc1HWHM=', 'Content-Length': '0'} 2022-09-12T20:09:59.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2e75fd6fcf07407-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.949 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:59.949 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:59.950 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:09:59.950 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:09:59.951 DEBUG:boto:Method: GET 2022-09-12T20:09:59.952 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:09:59.952 DEBUG:boto:Data: 2022-09-12T20:09:59.953 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.953 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:09:59.954 DEBUG:boto:Port: 8001 2022-09-12T20:09:59.954 DEBUG:boto:Params: {} 2022-09-12T20:09:59.955 DEBUG:boto:Token: None 2022-09-12T20:09:59.956 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/ 2022-09-12T20:09:59.956 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0= 2022-09-12T20:09:59.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0=', 'Content-Length': '0'} 2022-09-12T20:09:59.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089f99fa49edbac1e-00631f9217-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.960 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:59.961 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:09:59.962 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:09:59.962 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:09:59.963 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:09:59.963 DEBUG:boto:Method: GET 2022-09-12T20:09:59.964 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:09:59.964 DEBUG:boto:Data: 2022-09-12T20:09:59.965 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.965 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.966 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.967 DEBUG:boto:Params: {} 2022-09-12T20:09:59.967 DEBUG:boto:Token: None 2022-09-12T20:09:59.968 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/ 2022-09-12T20:09:59.969 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0= 2022-09-12T20:09:59.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yyWFaKIjAbhQe9X5OcwZ33LFBT0=', 'Content-Length': '0'} 2022-09-12T20:09:59.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b124c60d0612b76-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.973 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:09:59.974 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:09:59.975 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:09:59.975 DEBUG:boto:Method: HEAD 2022-09-12T20:09:59.976 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:09:59.976 DEBUG:boto:Data: 2022-09-12T20:09:59.977 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.977 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:09:59.978 DEBUG:boto:Port: 8001 2022-09-12T20:09:59.978 DEBUG:boto:Params: {} 2022-09-12T20:09:59.979 DEBUG:boto:Token: None 2022-09-12T20:09:59.980 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj17 2022-09-12T20:09:59.980 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kC4NxBESolYDNs8V7jVeUqdr05M= 2022-09-12T20:09:59.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kC4NxBESolYDNs8V7jVeUqdr05M=', 'Content-Length': '0'} 2022-09-12T20:09:59.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066506177b51e043b-00631f9217-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.983 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:09:59.984 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:09:59.984 DEBUG:boto:Method: HEAD 2022-09-12T20:09:59.985 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:09:59.986 DEBUG:boto:Data: 2022-09-12T20:09:59.986 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.987 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:09:59.987 DEBUG:boto:Port: 8000 2022-09-12T20:09:59.988 DEBUG:boto:Params: {} 2022-09-12T20:09:59.988 DEBUG:boto:Token: None 2022-09-12T20:09:59.989 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj17 2022-09-12T20:09:59.989 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kC4NxBESolYDNs8V7jVeUqdr05M= 2022-09-12T20:09:59.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kC4NxBESolYDNs8V7jVeUqdr05M=', 'Content-Length': '0'} 2022-09-12T20:09:59.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000370088b72c1e2742-00631f9217-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:09:59.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:09:59.993 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:09:59.994 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:09:59.994 DEBUG:boto:Method: GET 2022-09-12T20:09:59.995 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:09:59.995 DEBUG:boto:Data: 2022-09-12T20:09:59.996 DEBUG:boto:Headers: {} 2022-09-12T20:09:59.996 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:09:59.997 DEBUG:boto:Port: 8001 2022-09-12T20:09:59.998 DEBUG:boto:Params: {} 2022-09-12T20:09:59.998 DEBUG:boto:Token: None 2022-09-12T20:09:59.999 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:09:59 GMT /jdpiak-75/obj17 2022-09-12T20:09:59.999 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lQInnoPAyddlP4UC3m7Gd8IxV/8= 2022-09-12T20:10:00.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:09:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lQInnoPAyddlP4UC3m7Gd8IxV/8=', 'Content-Length': '0'} 2022-09-12T20:10:00.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e001fc25c4b6ecc-00631f9218-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.003 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:10:00.003 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:10:00.004 DEBUG:boto:Method: GET 2022-09-12T20:10:00.004 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:10:00.005 DEBUG:boto:Data: 2022-09-12T20:10:00.005 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.006 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.007 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.007 DEBUG:boto:Params: {} 2022-09-12T20:10:00.008 DEBUG:boto:Token: None 2022-09-12T20:10:00.008 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj17 2022-09-12T20:10:00.009 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ad0TDkbZ9PPYolW/C7KzZ6JO+28= 2022-09-12T20:10:00.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ad0TDkbZ9PPYolW/C7KzZ6JO+28=', 'Content-Length': '0'} 2022-09-12T20:10:00.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbf1e0a219e1eb0a-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.012 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:00.013 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:00.014 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:10:00.014 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:10:00.015 DEBUG:boto:Method: GET 2022-09-12T20:10:00.015 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:10:00.016 DEBUG:boto:Data: 2022-09-12T20:10:00.016 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.017 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.017 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.018 DEBUG:boto:Params: {} 2022-09-12T20:10:00.019 DEBUG:boto:Token: None 2022-09-12T20:10:00.019 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/ 2022-09-12T20:10:00.020 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA= 2022-09-12T20:10:00.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA=', 'Content-Length': '0'} 2022-09-12T20:10:00.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091e7365efdaab5a9-00631f9218-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.066 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:00.067 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:00.067 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:00.068 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:10:00.068 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:10:00.069 DEBUG:boto:Method: GET 2022-09-12T20:10:00.070 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:10:00.070 DEBUG:boto:Data: 2022-09-12T20:10:00.071 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.071 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.072 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.072 DEBUG:boto:Params: {} 2022-09-12T20:10:00.073 DEBUG:boto:Token: None 2022-09-12T20:10:00.073 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/ 2022-09-12T20:10:00.074 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA= 2022-09-12T20:10:00.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA=', 'Content-Length': '0'} 2022-09-12T20:10:00.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f8f268bc01ce59e-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.079 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:00.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:10:00.080 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:10:00.081 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:10:00.081 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:10:00.082 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:10:00.083 DEBUG:boto:Method: GET 2022-09-12T20:10:00.083 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:10:00.084 DEBUG:boto:Data: 2022-09-12T20:10:00.084 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.085 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.085 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.086 DEBUG:boto:Params: {} 2022-09-12T20:10:00.086 DEBUG:boto:Token: None 2022-09-12T20:10:00.087 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:10:00.088 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E6iL2Hv1TEx2hIArK8WLKMTCWjs= 2022-09-12T20:10:00.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E6iL2Hv1TEx2hIArK8WLKMTCWjs=', 'Content-Length': '0'} 2022-09-12T20:10:00.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009785bfe4bdde0e29-00631f9218-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.091 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:10:00.092 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:10:00.092 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:10:00.093 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:10:00.093 DEBUG:boto:Method: GET 2022-09-12T20:10:00.094 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:10:00.094 DEBUG:boto:Data: 2022-09-12T20:10:00.095 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.095 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.096 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.096 DEBUG:boto:Params: {} 2022-09-12T20:10:00.097 DEBUG:boto:Token: None 2022-09-12T20:10:00.098 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:10:00.098 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E6iL2Hv1TEx2hIArK8WLKMTCWjs= 2022-09-12T20:10:00.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E6iL2Hv1TEx2hIArK8WLKMTCWjs=', 'Content-Length': '0'} 2022-09-12T20:10:00.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5cf76329777bff5-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.142 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:10:00.143 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:10:00.143 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:10:00.144 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:10:00.144 DEBUG:boto:Method: HEAD 2022-09-12T20:10:00.145 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:10:00.145 DEBUG:boto:Data: 2022-09-12T20:10:00.146 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.146 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.147 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.147 DEBUG:boto:Params: {} 2022-09-12T20:10:00.148 DEBUG:boto:Token: None 2022-09-12T20:10:00.148 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:10:00.149 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sw8GJNPlNmzvqdzNqAQnHDpEUg0= 2022-09-12T20:10:00.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sw8GJNPlNmzvqdzNqAQnHDpEUg0=', 'Content-Length': '0'} 2022-09-12T20:10:00.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b55724ca032dc22-00631f9218-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.152 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:10:00.153 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:10:00.154 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:10:00.154 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:10:00.155 DEBUG:boto:Method: HEAD 2022-09-12T20:10:00.156 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:10:00.156 DEBUG:boto:Data: 2022-09-12T20:10:00.157 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.158 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.159 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.159 DEBUG:boto:Params: {} 2022-09-12T20:10:00.160 DEBUG:boto:Token: None 2022-09-12T20:10:00.160 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:10:00.161 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sw8GJNPlNmzvqdzNqAQnHDpEUg0= 2022-09-12T20:10:00.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sw8GJNPlNmzvqdzNqAQnHDpEUg0=', 'Content-Length': '0'} 2022-09-12T20:10:00.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b49152a38f4e542a-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:10:00.165 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:10:00.165 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:10:00.166 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:10:00.166 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:10:00.167 DEBUG:boto:Method: GET 2022-09-12T20:10:00.168 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:10:00.168 DEBUG:boto:Data: 2022-09-12T20:10:00.169 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.170 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.170 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.171 DEBUG:boto:Params: {} 2022-09-12T20:10:00.172 DEBUG:boto:Token: None 2022-09-12T20:10:00.172 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:10:00.173 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E6iL2Hv1TEx2hIArK8WLKMTCWjs= 2022-09-12T20:10:00.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E6iL2Hv1TEx2hIArK8WLKMTCWjs=', 'Content-Length': '0'} 2022-09-12T20:10:00.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b2369fcf77b84ce-00631f9218-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.176 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:10:00.177 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:10:00.177 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:10:00.178 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:10:00.178 DEBUG:boto:Method: GET 2022-09-12T20:10:00.179 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:10:00.180 DEBUG:boto:Data: 2022-09-12T20:10:00.180 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.181 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.181 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.182 DEBUG:boto:Params: {} 2022-09-12T20:10:00.182 DEBUG:boto:Token: None 2022-09-12T20:10:00.183 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:10:00.184 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E6iL2Hv1TEx2hIArK8WLKMTCWjs= 2022-09-12T20:10:00.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E6iL2Hv1TEx2hIArK8WLKMTCWjs=', 'Content-Length': '0'} 2022-09-12T20:10:00.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb2fe5261e1ec46b-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.187 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:00.188 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:00.188 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:10:00.189 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:10:00.189 DEBUG:boto:Method: GET 2022-09-12T20:10:00.190 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:10:00.190 DEBUG:boto:Data: 2022-09-12T20:10:00.191 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.191 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.192 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.192 DEBUG:boto:Params: {} 2022-09-12T20:10:00.193 DEBUG:boto:Token: None 2022-09-12T20:10:00.194 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/ 2022-09-12T20:10:00.194 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA= 2022-09-12T20:10:00.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA=', 'Content-Length': '0'} 2022-09-12T20:10:00.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016167734617eaafb-00631f9218-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.198 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:00.199 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:00.200 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:00.200 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:10:00.201 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:10:00.201 DEBUG:boto:Method: GET 2022-09-12T20:10:00.202 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:10:00.202 DEBUG:boto:Data: 2022-09-12T20:10:00.203 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.203 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.204 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.204 DEBUG:boto:Params: {} 2022-09-12T20:10:00.205 DEBUG:boto:Token: None 2022-09-12T20:10:00.206 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/ 2022-09-12T20:10:00.206 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA= 2022-09-12T20:10:00.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA=', 'Content-Length': '0'} 2022-09-12T20:10:00.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e7bf6f93f37718a-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.211 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:00.212 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:10:00.212 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:10:00.213 DEBUG:boto:Method: HEAD 2022-09-12T20:10:00.213 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:10:00.214 DEBUG:boto:Data: 2022-09-12T20:10:00.215 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.215 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.216 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.216 DEBUG:boto:Params: {} 2022-09-12T20:10:00.217 DEBUG:boto:Token: None 2022-09-12T20:10:00.217 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj19 2022-09-12T20:10:00.218 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9N5KDwL5csIY7KN2cJOsfvNeISI= 2022-09-12T20:10:00.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9N5KDwL5csIY7KN2cJOsfvNeISI=', 'Content-Length': '0'} 2022-09-12T20:10:00.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006618d56138600201-00631f9218-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.221 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:10:00.222 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:10:00.222 DEBUG:boto:Method: HEAD 2022-09-12T20:10:00.223 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:10:00.223 DEBUG:boto:Data: 2022-09-12T20:10:00.224 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.225 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.225 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.226 DEBUG:boto:Params: {} 2022-09-12T20:10:00.226 DEBUG:boto:Token: None 2022-09-12T20:10:00.227 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj19 2022-09-12T20:10:00.227 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9N5KDwL5csIY7KN2cJOsfvNeISI= 2022-09-12T20:10:00.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9N5KDwL5csIY7KN2cJOsfvNeISI=', 'Content-Length': '0'} 2022-09-12T20:10:00.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e3c1c1e7fc41cfa-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:10:00.231 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:10:00.232 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:10:00.232 DEBUG:boto:Method: GET 2022-09-12T20:10:00.233 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:10:00.233 DEBUG:boto:Data: 2022-09-12T20:10:00.234 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.235 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.235 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.236 DEBUG:boto:Params: {} 2022-09-12T20:10:00.237 DEBUG:boto:Token: None 2022-09-12T20:10:00.237 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj19 2022-09-12T20:10:00.238 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VJ3uUlV5mgD5yMDyZZ68HWwAJB0= 2022-09-12T20:10:00.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VJ3uUlV5mgD5yMDyZZ68HWwAJB0=', 'Content-Length': '0'} 2022-09-12T20:10:00.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5a74c691e7898de-00631f9218-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.242 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:10:00.243 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:10:00.243 DEBUG:boto:Method: GET 2022-09-12T20:10:00.244 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:10:00.244 DEBUG:boto:Data: 2022-09-12T20:10:00.245 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.246 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.246 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.247 DEBUG:boto:Params: {} 2022-09-12T20:10:00.247 DEBUG:boto:Token: None 2022-09-12T20:10:00.248 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj19 2022-09-12T20:10:00.249 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VJ3uUlV5mgD5yMDyZZ68HWwAJB0= 2022-09-12T20:10:00.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VJ3uUlV5mgD5yMDyZZ68HWwAJB0=', 'Content-Length': '0'} 2022-09-12T20:10:00.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008dc0060a3c9c821-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.294 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:00.295 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:00.295 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:10:00.296 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:10:00.297 DEBUG:boto:Method: GET 2022-09-12T20:10:00.297 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:10:00.298 DEBUG:boto:Data: 2022-09-12T20:10:00.299 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.299 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.300 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.301 DEBUG:boto:Params: {} 2022-09-12T20:10:00.301 DEBUG:boto:Token: None 2022-09-12T20:10:00.302 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/ 2022-09-12T20:10:00.303 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA= 2022-09-12T20:10:00.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA=', 'Content-Length': '0'} 2022-09-12T20:10:00.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042da9ab9305eafa5-00631f9218-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.307 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:00.308 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:00.309 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:00.309 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:10:00.310 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:10:00.310 DEBUG:boto:Method: GET 2022-09-12T20:10:00.311 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:10:00.311 DEBUG:boto:Data: 2022-09-12T20:10:00.312 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.312 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.313 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.313 DEBUG:boto:Params: {} 2022-09-12T20:10:00.314 DEBUG:boto:Token: None 2022-09-12T20:10:00.315 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/ 2022-09-12T20:10:00.315 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA= 2022-09-12T20:10:00.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA=', 'Content-Length': '0'} 2022-09-12T20:10:00.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce8c83421bcc1b6e-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.320 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:00.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:10:00.322 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:10:00.322 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:10:00.323 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:10:00.323 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:10:00.324 DEBUG:boto:Method: GET 2022-09-12T20:10:00.324 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:10:00.325 DEBUG:boto:Data: 2022-09-12T20:10:00.325 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.326 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.327 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.327 DEBUG:boto:Params: {} 2022-09-12T20:10:00.328 DEBUG:boto:Token: None 2022-09-12T20:10:00.328 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:10:00.329 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ASmkcXuJui5cqYZtNsBYCicrPdA= 2022-09-12T20:10:00.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ASmkcXuJui5cqYZtNsBYCicrPdA=', 'Content-Length': '0'} 2022-09-12T20:10:00.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000562628cbede98dee-00631f9218-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.334 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:10:00.334 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:10:00.335 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:10:00.335 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:10:00.336 DEBUG:boto:Method: GET 2022-09-12T20:10:00.336 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:10:00.337 DEBUG:boto:Data: 2022-09-12T20:10:00.338 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.338 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.339 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.339 DEBUG:boto:Params: {} 2022-09-12T20:10:00.340 DEBUG:boto:Token: None 2022-09-12T20:10:00.340 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:10:00.341 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ASmkcXuJui5cqYZtNsBYCicrPdA= 2022-09-12T20:10:00.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ASmkcXuJui5cqYZtNsBYCicrPdA=', 'Content-Length': '0'} 2022-09-12T20:10:00.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f083a627641806c2-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.345 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:10:00.346 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:10:00.346 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:10:00.347 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:10:00.347 DEBUG:boto:Method: HEAD 2022-09-12T20:10:00.348 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:10:00.348 DEBUG:boto:Data: 2022-09-12T20:10:00.348 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.349 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.349 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.350 DEBUG:boto:Params: {} 2022-09-12T20:10:00.350 DEBUG:boto:Token: None 2022-09-12T20:10:00.351 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:10:00.352 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2SrUIUL1zOF84+ScMk+KO/QqoRo= 2022-09-12T20:10:00.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2SrUIUL1zOF84+ScMk+KO/QqoRo=', 'Content-Length': '0'} 2022-09-12T20:10:00.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b674ac41f1e2596b-00631f9218-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.355 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:10:00.356 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:10:00.356 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:10:00.357 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:10:00.357 DEBUG:boto:Method: HEAD 2022-09-12T20:10:00.358 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:10:00.359 DEBUG:boto:Data: 2022-09-12T20:10:00.360 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.361 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.361 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.362 DEBUG:boto:Params: {} 2022-09-12T20:10:00.362 DEBUG:boto:Token: None 2022-09-12T20:10:00.363 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:10:00.364 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2SrUIUL1zOF84+ScMk+KO/QqoRo= 2022-09-12T20:10:00.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2SrUIUL1zOF84+ScMk+KO/QqoRo=', 'Content-Length': '0'} 2022-09-12T20:10:00.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000844f7e1fec86e6dc-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:10:00.367 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:10:00.368 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:10:00.369 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:10:00.369 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:10:00.370 DEBUG:boto:Method: GET 2022-09-12T20:10:00.370 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:10:00.371 DEBUG:boto:Data: 2022-09-12T20:10:00.371 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.372 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.372 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.373 DEBUG:boto:Params: {} 2022-09-12T20:10:00.373 DEBUG:boto:Token: None 2022-09-12T20:10:00.374 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:10:00.375 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ASmkcXuJui5cqYZtNsBYCicrPdA= 2022-09-12T20:10:00.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ASmkcXuJui5cqYZtNsBYCicrPdA=', 'Content-Length': '0'} 2022-09-12T20:10:00.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ebb1359fcfaf6c6-00631f9218-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.418 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:10:00.418 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:10:00.419 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:10:00.419 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:10:00.420 DEBUG:boto:Method: GET 2022-09-12T20:10:00.421 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:10:00.421 DEBUG:boto:Data: 2022-09-12T20:10:00.422 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.422 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.423 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.423 DEBUG:boto:Params: {} 2022-09-12T20:10:00.424 DEBUG:boto:Token: None 2022-09-12T20:10:00.424 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:10:00.425 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ASmkcXuJui5cqYZtNsBYCicrPdA= 2022-09-12T20:10:00.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ASmkcXuJui5cqYZtNsBYCicrPdA=', 'Content-Length': '0'} 2022-09-12T20:10:00.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000138a8b430b4fe00f-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.429 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:00.429 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:00.430 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:10:00.431 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:10:00.431 DEBUG:boto:Method: GET 2022-09-12T20:10:00.432 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:10:00.432 DEBUG:boto:Data: 2022-09-12T20:10:00.433 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.433 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.434 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.434 DEBUG:boto:Params: {} 2022-09-12T20:10:00.435 DEBUG:boto:Token: None 2022-09-12T20:10:00.435 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/ 2022-09-12T20:10:00.436 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA= 2022-09-12T20:10:00.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA=', 'Content-Length': '0'} 2022-09-12T20:10:00.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cee1db1b608115f-00631f9218-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.440 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:00.441 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:00.441 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:00.442 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:10:00.443 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:10:00.443 DEBUG:boto:Method: GET 2022-09-12T20:10:00.444 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:10:00.444 DEBUG:boto:Data: 2022-09-12T20:10:00.445 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.445 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.446 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.446 DEBUG:boto:Params: {} 2022-09-12T20:10:00.447 DEBUG:boto:Token: None 2022-09-12T20:10:00.448 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/ 2022-09-12T20:10:00.448 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA= 2022-09-12T20:10:00.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA=', 'Content-Length': '0'} 2022-09-12T20:10:00.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008db1fa0f086a5238-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.453 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:00.454 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:10:00.454 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:10:00.455 DEBUG:boto:Method: HEAD 2022-09-12T20:10:00.456 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:10:00.456 DEBUG:boto:Data: 2022-09-12T20:10:00.457 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.457 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.458 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.458 DEBUG:boto:Params: {} 2022-09-12T20:10:00.459 DEBUG:boto:Token: None 2022-09-12T20:10:00.459 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj22 2022-09-12T20:10:00.460 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yraifgw7DxR4byzPRXWybdfZQbo= 2022-09-12T20:10:00.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yraifgw7DxR4byzPRXWybdfZQbo=', 'Content-Length': '0'} 2022-09-12T20:10:00.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8b9c763d6be4784-00631f9218-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.463 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:10:00.464 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:10:00.464 DEBUG:boto:Method: HEAD 2022-09-12T20:10:00.465 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:10:00.465 DEBUG:boto:Data: 2022-09-12T20:10:00.466 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.466 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.467 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.467 DEBUG:boto:Params: {} 2022-09-12T20:10:00.468 DEBUG:boto:Token: None 2022-09-12T20:10:00.469 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj22 2022-09-12T20:10:00.469 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yraifgw7DxR4byzPRXWybdfZQbo= 2022-09-12T20:10:00.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yraifgw7DxR4byzPRXWybdfZQbo=', 'Content-Length': '0'} 2022-09-12T20:10:00.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004339a0d3ebf6032f-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:10:00.473 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:10:00.474 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:10:00.474 DEBUG:boto:Method: GET 2022-09-12T20:10:00.475 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:10:00.475 DEBUG:boto:Data: 2022-09-12T20:10:00.476 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.476 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.477 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.478 DEBUG:boto:Params: {} 2022-09-12T20:10:00.478 DEBUG:boto:Token: None 2022-09-12T20:10:00.479 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj22 2022-09-12T20:10:00.479 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kzXfnMni0nGRIxv1c3r0UYJBE0c= 2022-09-12T20:10:00.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kzXfnMni0nGRIxv1c3r0UYJBE0c=', 'Content-Length': '0'} 2022-09-12T20:10:00.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081b53e473d44a61c-00631f9218-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.526 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:10:00.527 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:10:00.527 DEBUG:boto:Method: GET 2022-09-12T20:10:00.528 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:10:00.528 DEBUG:boto:Data: 2022-09-12T20:10:00.529 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.529 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.530 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.530 DEBUG:boto:Params: {} 2022-09-12T20:10:00.531 DEBUG:boto:Token: None 2022-09-12T20:10:00.532 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj22 2022-09-12T20:10:00.532 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kzXfnMni0nGRIxv1c3r0UYJBE0c= 2022-09-12T20:10:00.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kzXfnMni0nGRIxv1c3r0UYJBE0c=', 'Content-Length': '0'} 2022-09-12T20:10:00.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001059005c44e9c2a8-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.536 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:00.536 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:00.537 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:10:00.537 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:10:00.538 DEBUG:boto:Method: GET 2022-09-12T20:10:00.538 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:10:00.539 DEBUG:boto:Data: 2022-09-12T20:10:00.539 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.540 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.540 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.541 DEBUG:boto:Params: {} 2022-09-12T20:10:00.542 DEBUG:boto:Token: None 2022-09-12T20:10:00.542 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/ 2022-09-12T20:10:00.543 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA= 2022-09-12T20:10:00.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA=', 'Content-Length': '0'} 2022-09-12T20:10:00.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e3193091894cb0e-00631f9218-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.547 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:00.547 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:00.548 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:00.549 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:10:00.549 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:10:00.550 DEBUG:boto:Method: GET 2022-09-12T20:10:00.550 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:10:00.551 DEBUG:boto:Data: 2022-09-12T20:10:00.551 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.552 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.552 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.553 DEBUG:boto:Params: {} 2022-09-12T20:10:00.553 DEBUG:boto:Token: None 2022-09-12T20:10:00.554 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/ 2022-09-12T20:10:00.554 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA= 2022-09-12T20:10:00.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA=', 'Content-Length': '0'} 2022-09-12T20:10:00.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005917abcaddc1eba9-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.559 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:00.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:10:00.561 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:10:00.561 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:10:00.562 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:10:00.562 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:10:00.563 DEBUG:boto:Method: GET 2022-09-12T20:10:00.563 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:10:00.564 DEBUG:boto:Data: 2022-09-12T20:10:00.564 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.565 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.565 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.566 DEBUG:boto:Params: {} 2022-09-12T20:10:00.566 DEBUG:boto:Token: None 2022-09-12T20:10:00.567 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:10:00.568 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xm53Xk9s/rs6AA6RUcqr4jpHQf0= 2022-09-12T20:10:00.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xm53Xk9s/rs6AA6RUcqr4jpHQf0=', 'Content-Length': '0'} 2022-09-12T20:10:00.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f39446a34926317-00631f9218-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.571 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:10:00.572 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:10:00.572 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:10:00.573 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:10:00.573 DEBUG:boto:Method: GET 2022-09-12T20:10:00.574 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:10:00.575 DEBUG:boto:Data: 2022-09-12T20:10:00.575 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.576 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.576 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.577 DEBUG:boto:Params: {} 2022-09-12T20:10:00.577 DEBUG:boto:Token: None 2022-09-12T20:10:00.578 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:10:00.579 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xm53Xk9s/rs6AA6RUcqr4jpHQf0= 2022-09-12T20:10:00.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xm53Xk9s/rs6AA6RUcqr4jpHQf0=', 'Content-Length': '0'} 2022-09-12T20:10:00.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6e04cac44158a33-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.582 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:10:00.583 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:10:00.583 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:10:00.584 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:10:00.584 DEBUG:boto:Method: HEAD 2022-09-12T20:10:00.585 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:10:00.585 DEBUG:boto:Data: 2022-09-12T20:10:00.586 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.586 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.587 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.587 DEBUG:boto:Params: {} 2022-09-12T20:10:00.588 DEBUG:boto:Token: None 2022-09-12T20:10:00.588 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:10:00.589 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/yq8ADzhJcC3SZJ6m/8NMkTTH6g= 2022-09-12T20:10:00.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/yq8ADzhJcC3SZJ6m/8NMkTTH6g=', 'Content-Length': '0'} 2022-09-12T20:10:00.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ae9cc5e8b2b24b9-00631f9218-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.592 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:10:00.593 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:10:00.593 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:10:00.594 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:10:00.594 DEBUG:boto:Method: HEAD 2022-09-12T20:10:00.595 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:10:00.595 DEBUG:boto:Data: 2022-09-12T20:10:00.596 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.596 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.597 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.597 DEBUG:boto:Params: {} 2022-09-12T20:10:00.598 DEBUG:boto:Token: None 2022-09-12T20:10:00.598 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:10:00.599 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/yq8ADzhJcC3SZJ6m/8NMkTTH6g= 2022-09-12T20:10:00.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/yq8ADzhJcC3SZJ6m/8NMkTTH6g=', 'Content-Length': '0'} 2022-09-12T20:10:00.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee34dfc269ccfc32-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:10:00.603 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:10:00.603 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:10:00.604 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:10:00.604 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:10:00.605 DEBUG:boto:Method: GET 2022-09-12T20:10:00.605 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:10:00.606 DEBUG:boto:Data: 2022-09-12T20:10:00.606 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.607 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.607 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.608 DEBUG:boto:Params: {} 2022-09-12T20:10:00.608 DEBUG:boto:Token: None 2022-09-12T20:10:00.609 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:10:00.610 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xm53Xk9s/rs6AA6RUcqr4jpHQf0= 2022-09-12T20:10:00.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xm53Xk9s/rs6AA6RUcqr4jpHQf0=', 'Content-Length': '0'} 2022-09-12T20:10:00.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e530e9a847b83fc9-00631f9218-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.654 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:10:00.655 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:10:00.655 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:10:00.656 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:10:00.656 DEBUG:boto:Method: GET 2022-09-12T20:10:00.657 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:10:00.657 DEBUG:boto:Data: 2022-09-12T20:10:00.658 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.658 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.659 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.659 DEBUG:boto:Params: {} 2022-09-12T20:10:00.660 DEBUG:boto:Token: None 2022-09-12T20:10:00.660 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:10:00.661 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xm53Xk9s/rs6AA6RUcqr4jpHQf0= 2022-09-12T20:10:00.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xm53Xk9s/rs6AA6RUcqr4jpHQf0=', 'Content-Length': '0'} 2022-09-12T20:10:00.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000848f1a918bf87612-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.665 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:00.665 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:00.666 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:10:00.666 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:10:00.667 DEBUG:boto:Method: GET 2022-09-12T20:10:00.667 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:10:00.668 DEBUG:boto:Data: 2022-09-12T20:10:00.668 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.669 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.669 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.670 DEBUG:boto:Params: {} 2022-09-12T20:10:00.671 DEBUG:boto:Token: None 2022-09-12T20:10:00.671 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/ 2022-09-12T20:10:00.672 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA= 2022-09-12T20:10:00.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA=', 'Content-Length': '0'} 2022-09-12T20:10:00.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000499a22e5a66ad8ee-00631f9218-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.676 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:00.676 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:00.677 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:00.677 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:10:00.678 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:10:00.679 DEBUG:boto:Method: GET 2022-09-12T20:10:00.679 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:10:00.680 DEBUG:boto:Data: 2022-09-12T20:10:00.680 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.681 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.681 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.682 DEBUG:boto:Params: {} 2022-09-12T20:10:00.682 DEBUG:boto:Token: None 2022-09-12T20:10:00.683 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/ 2022-09-12T20:10:00.683 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA= 2022-09-12T20:10:00.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA=', 'Content-Length': '0'} 2022-09-12T20:10:00.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000283649f0e7885f35-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.689 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:00.689 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:10:00.690 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:10:00.690 DEBUG:boto:Method: HEAD 2022-09-12T20:10:00.691 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:10:00.692 DEBUG:boto:Data: 2022-09-12T20:10:00.692 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.693 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.693 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.694 DEBUG:boto:Params: {} 2022-09-12T20:10:00.694 DEBUG:boto:Token: None 2022-09-12T20:10:00.695 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj23 2022-09-12T20:10:00.695 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tIAye0deJdNhjUbJm+0c3IkVAT4= 2022-09-12T20:10:00.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tIAye0deJdNhjUbJm+0c3IkVAT4=', 'Content-Length': '0'} 2022-09-12T20:10:00.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001369c53cbc819032-00631f9218-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.699 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:10:00.699 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:10:00.700 DEBUG:boto:Method: HEAD 2022-09-12T20:10:00.700 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:10:00.701 DEBUG:boto:Data: 2022-09-12T20:10:00.701 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.702 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.702 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.703 DEBUG:boto:Params: {} 2022-09-12T20:10:00.703 DEBUG:boto:Token: None 2022-09-12T20:10:00.704 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj23 2022-09-12T20:10:00.704 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tIAye0deJdNhjUbJm+0c3IkVAT4= 2022-09-12T20:10:00.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tIAye0deJdNhjUbJm+0c3IkVAT4=', 'Content-Length': '0'} 2022-09-12T20:10:00.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032d9b9e91d12afce-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:10:00.708 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:10:00.708 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:10:00.709 DEBUG:boto:Method: GET 2022-09-12T20:10:00.709 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:10:00.710 DEBUG:boto:Data: 2022-09-12T20:10:00.711 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.711 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.712 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.712 DEBUG:boto:Params: {} 2022-09-12T20:10:00.713 DEBUG:boto:Token: None 2022-09-12T20:10:00.713 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj23 2022-09-12T20:10:00.714 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A7UB6WreuQdUYJVkaos+niuQ9P0= 2022-09-12T20:10:00.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A7UB6WreuQdUYJVkaos+niuQ9P0=', 'Content-Length': '0'} 2022-09-12T20:10:00.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d3546a914903da4-00631f9218-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.758 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:10:00.759 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:10:00.760 DEBUG:boto:Method: GET 2022-09-12T20:10:00.760 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:10:00.761 DEBUG:boto:Data: 2022-09-12T20:10:00.761 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.762 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.763 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.763 DEBUG:boto:Params: {} 2022-09-12T20:10:00.764 DEBUG:boto:Token: None 2022-09-12T20:10:00.764 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj23 2022-09-12T20:10:00.765 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A7UB6WreuQdUYJVkaos+niuQ9P0= 2022-09-12T20:10:00.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A7UB6WreuQdUYJVkaos+niuQ9P0=', 'Content-Length': '0'} 2022-09-12T20:10:00.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fc21a7d61ab392f-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.769 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:00.770 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:00.770 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:10:00.771 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:10:00.771 DEBUG:boto:Method: GET 2022-09-12T20:10:00.772 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:10:00.772 DEBUG:boto:Data: 2022-09-12T20:10:00.773 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.773 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.774 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.775 DEBUG:boto:Params: {} 2022-09-12T20:10:00.775 DEBUG:boto:Token: None 2022-09-12T20:10:00.776 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/ 2022-09-12T20:10:00.777 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA= 2022-09-12T20:10:00.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA=', 'Content-Length': '0'} 2022-09-12T20:10:00.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d45b53149decebbc-00631f9218-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.781 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:00.782 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:00.782 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:00.783 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:10:00.783 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:10:00.784 DEBUG:boto:Method: GET 2022-09-12T20:10:00.785 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:10:00.785 DEBUG:boto:Data: 2022-09-12T20:10:00.786 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.786 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.787 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.787 DEBUG:boto:Params: {} 2022-09-12T20:10:00.788 DEBUG:boto:Token: None 2022-09-12T20:10:00.788 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/ 2022-09-12T20:10:00.789 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA= 2022-09-12T20:10:00.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA=', 'Content-Length': '0'} 2022-09-12T20:10:00.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7ee983742df3066-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.794 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:00.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:10:00.795 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:10:00.796 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:10:00.797 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:10:00.797 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:10:00.798 DEBUG:boto:Method: GET 2022-09-12T20:10:00.798 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:10:00.799 DEBUG:boto:Data: 2022-09-12T20:10:00.799 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.800 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.800 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.801 DEBUG:boto:Params: {} 2022-09-12T20:10:00.802 DEBUG:boto:Token: None 2022-09-12T20:10:00.802 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:10:00.803 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qR87Fv7ShlGLciOhRFKVvbX9ZKo= 2022-09-12T20:10:00.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qR87Fv7ShlGLciOhRFKVvbX9ZKo=', 'Content-Length': '0'} 2022-09-12T20:10:00.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001abcb31e4a3542b-00631f9218-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.806 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:10:00.807 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:10:00.808 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:10:00.808 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:10:00.809 DEBUG:boto:Method: GET 2022-09-12T20:10:00.809 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:10:00.810 DEBUG:boto:Data: 2022-09-12T20:10:00.810 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.811 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.812 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.812 DEBUG:boto:Params: {} 2022-09-12T20:10:00.813 DEBUG:boto:Token: None 2022-09-12T20:10:00.813 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:10:00.814 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qR87Fv7ShlGLciOhRFKVvbX9ZKo= 2022-09-12T20:10:00.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qR87Fv7ShlGLciOhRFKVvbX9ZKo=', 'Content-Length': '0'} 2022-09-12T20:10:00.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009648e0ef3c2df96d-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.818 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:10:00.818 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:10:00.819 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:10:00.819 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:10:00.820 DEBUG:boto:Method: HEAD 2022-09-12T20:10:00.820 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:10:00.821 DEBUG:boto:Data: 2022-09-12T20:10:00.821 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.822 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.822 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.823 DEBUG:boto:Params: {} 2022-09-12T20:10:00.823 DEBUG:boto:Token: None 2022-09-12T20:10:00.824 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:10:00.825 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eAyXnRPFcUcw5myHR0fYocC8SaY= 2022-09-12T20:10:00.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eAyXnRPFcUcw5myHR0fYocC8SaY=', 'Content-Length': '0'} 2022-09-12T20:10:00.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000818a8a01d6acf959-00631f9218-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.828 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:10:00.829 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:10:00.829 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:10:00.830 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:10:00.831 DEBUG:boto:Method: HEAD 2022-09-12T20:10:00.831 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:10:00.832 DEBUG:boto:Data: 2022-09-12T20:10:00.832 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.833 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.833 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.834 DEBUG:boto:Params: {} 2022-09-12T20:10:00.835 DEBUG:boto:Token: None 2022-09-12T20:10:00.835 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:10:00.836 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eAyXnRPFcUcw5myHR0fYocC8SaY= 2022-09-12T20:10:00.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eAyXnRPFcUcw5myHR0fYocC8SaY=', 'Content-Length': '0'} 2022-09-12T20:10:00.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b19e2ccf48a9e5dc-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:10:00.840 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:10:00.840 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:10:00.841 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:10:00.841 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:10:00.842 DEBUG:boto:Method: GET 2022-09-12T20:10:00.843 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:10:00.843 DEBUG:boto:Data: 2022-09-12T20:10:00.844 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.844 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.845 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.845 DEBUG:boto:Params: {} 2022-09-12T20:10:00.846 DEBUG:boto:Token: None 2022-09-12T20:10:00.847 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:10:00.847 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qR87Fv7ShlGLciOhRFKVvbX9ZKo= 2022-09-12T20:10:00.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qR87Fv7ShlGLciOhRFKVvbX9ZKo=', 'Content-Length': '0'} 2022-09-12T20:10:00.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e10b6fe2e792f7f1-00631f9218-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.851 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:10:00.851 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:10:00.852 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:10:00.852 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:10:00.853 DEBUG:boto:Method: GET 2022-09-12T20:10:00.854 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:10:00.854 DEBUG:boto:Data: 2022-09-12T20:10:00.855 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.855 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.856 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.857 DEBUG:boto:Params: {} 2022-09-12T20:10:00.857 DEBUG:boto:Token: None 2022-09-12T20:10:00.858 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:10:00.859 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qR87Fv7ShlGLciOhRFKVvbX9ZKo= 2022-09-12T20:10:00.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qR87Fv7ShlGLciOhRFKVvbX9ZKo=', 'Content-Length': '0'} 2022-09-12T20:10:00.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003267fa58472a5e15-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.902 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:00.902 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:00.903 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:10:00.904 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:10:00.904 DEBUG:boto:Method: GET 2022-09-12T20:10:00.905 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:10:00.905 DEBUG:boto:Data: 2022-09-12T20:10:00.906 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.906 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.907 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.907 DEBUG:boto:Params: {} 2022-09-12T20:10:00.908 DEBUG:boto:Token: None 2022-09-12T20:10:00.909 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/ 2022-09-12T20:10:00.909 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA= 2022-09-12T20:10:00.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA=', 'Content-Length': '0'} 2022-09-12T20:10:00.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009121ae272fdcacae-00631f9218-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.914 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:00.915 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:00.915 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:00.916 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:10:00.916 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:10:00.917 DEBUG:boto:Method: GET 2022-09-12T20:10:00.917 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:10:00.918 DEBUG:boto:Data: 2022-09-12T20:10:00.918 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.919 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.919 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.920 DEBUG:boto:Params: {} 2022-09-12T20:10:00.921 DEBUG:boto:Token: None 2022-09-12T20:10:00.921 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/ 2022-09-12T20:10:00.922 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA= 2022-09-12T20:10:00.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t10jhn5HnVMg+dFnE2qxlOai+dA=', 'Content-Length': '0'} 2022-09-12T20:10:00.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fda034a44b57bf3-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.927 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:00.928 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:10:00.928 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:10:00.929 DEBUG:boto:Method: HEAD 2022-09-12T20:10:00.929 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:10:00.930 DEBUG:boto:Data: 2022-09-12T20:10:00.931 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.931 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.932 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.932 DEBUG:boto:Params: {} 2022-09-12T20:10:00.933 DEBUG:boto:Token: None 2022-09-12T20:10:00.933 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj33 2022-09-12T20:10:00.934 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:i5CXG0MIdvRBwFh7+Vdd8EY4pTw= 2022-09-12T20:10:00.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:i5CXG0MIdvRBwFh7+Vdd8EY4pTw=', 'Content-Length': '0'} 2022-09-12T20:10:00.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fddc30c30ea2973-00631f9218-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.938 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:10:00.938 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:10:00.939 DEBUG:boto:Method: HEAD 2022-09-12T20:10:00.939 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:10:00.940 DEBUG:boto:Data: 2022-09-12T20:10:00.940 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.941 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.941 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.942 DEBUG:boto:Params: {} 2022-09-12T20:10:00.943 DEBUG:boto:Token: None 2022-09-12T20:10:00.943 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj33 2022-09-12T20:10:00.944 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:i5CXG0MIdvRBwFh7+Vdd8EY4pTw= 2022-09-12T20:10:00.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:i5CXG0MIdvRBwFh7+Vdd8EY4pTw=', 'Content-Length': '0'} 2022-09-12T20:10:00.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018800e2a1c43f0eb-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:10:00.948 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:10:00.949 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:10:00.949 DEBUG:boto:Method: GET 2022-09-12T20:10:00.950 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:10:00.950 DEBUG:boto:Data: 2022-09-12T20:10:00.951 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.951 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:00.952 DEBUG:boto:Port: 8001 2022-09-12T20:10:00.952 DEBUG:boto:Params: {} 2022-09-12T20:10:00.953 DEBUG:boto:Token: None 2022-09-12T20:10:00.954 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj33 2022-09-12T20:10:00.954 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TcxGnbSWlWW8hzimtXLAuPDU3AE= 2022-09-12T20:10:00.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TcxGnbSWlWW8hzimtXLAuPDU3AE=', 'Content-Length': '0'} 2022-09-12T20:10:00.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000567e7f443bb2369e-00631f9218-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:00.958 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:10:00.959 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:10:00.960 DEBUG:boto:Method: GET 2022-09-12T20:10:00.960 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:10:00.961 DEBUG:boto:Data: 2022-09-12T20:10:00.962 DEBUG:boto:Headers: {} 2022-09-12T20:10:00.962 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:00.963 DEBUG:boto:Port: 8000 2022-09-12T20:10:00.963 DEBUG:boto:Params: {} 2022-09-12T20:10:00.964 DEBUG:boto:Token: None 2022-09-12T20:10:00.965 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:00 GMT /jdpiak-75/obj33 2022-09-12T20:10:00.965 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TcxGnbSWlWW8hzimtXLAuPDU3AE= 2022-09-12T20:10:00.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TcxGnbSWlWW8hzimtXLAuPDU3AE=', 'Content-Length': '0'} 2022-09-12T20:10:00.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0611eb1aeca75ea-00631f9218-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.010 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:01.011 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:01.011 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:10:01.012 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:10:01.012 DEBUG:boto:Method: GET 2022-09-12T20:10:01.013 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:10:01.013 DEBUG:boto:Data: 2022-09-12T20:10:01.014 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.015 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.015 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.016 DEBUG:boto:Params: {} 2022-09-12T20:10:01.016 DEBUG:boto:Token: None 2022-09-12T20:10:01.017 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/ 2022-09-12T20:10:01.017 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0= 2022-09-12T20:10:01.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0=', 'Content-Length': '0'} 2022-09-12T20:10:01.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8cb3e554ba9447d-00631f9219-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.022 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:01.023 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:01.023 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:01.024 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:10:01.024 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:10:01.025 DEBUG:boto:Method: GET 2022-09-12T20:10:01.025 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:10:01.026 DEBUG:boto:Data: 2022-09-12T20:10:01.027 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.027 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.028 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.028 DEBUG:boto:Params: {} 2022-09-12T20:10:01.029 DEBUG:boto:Token: None 2022-09-12T20:10:01.030 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/ 2022-09-12T20:10:01.030 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0= 2022-09-12T20:10:01.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0=', 'Content-Length': '0'} 2022-09-12T20:10:01.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da236accade73680-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.035 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:01.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:10:01.037 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:10:01.037 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:10:01.038 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:10:01.038 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:10:01.039 DEBUG:boto:Method: GET 2022-09-12T20:10:01.039 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:10:01.040 DEBUG:boto:Data: 2022-09-12T20:10:01.041 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.041 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.042 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.042 DEBUG:boto:Params: {} 2022-09-12T20:10:01.043 DEBUG:boto:Token: None 2022-09-12T20:10:01.044 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:10:01.044 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bungudekEmxbk4ZU+PZBztWKKnM= 2022-09-12T20:10:01.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bungudekEmxbk4ZU+PZBztWKKnM=', 'Content-Length': '0'} 2022-09-12T20:10:01.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceded9f4d5f6c481-00631f9219-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.048 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:10:01.049 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:10:01.049 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:10:01.050 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:10:01.050 DEBUG:boto:Method: GET 2022-09-12T20:10:01.051 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:10:01.052 DEBUG:boto:Data: 2022-09-12T20:10:01.052 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.053 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.053 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.054 DEBUG:boto:Params: {} 2022-09-12T20:10:01.054 DEBUG:boto:Token: None 2022-09-12T20:10:01.055 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:10:01.056 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bungudekEmxbk4ZU+PZBztWKKnM= 2022-09-12T20:10:01.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bungudekEmxbk4ZU+PZBztWKKnM=', 'Content-Length': '0'} 2022-09-12T20:10:01.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000beaf2452be39c9c6-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.060 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:10:01.060 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:10:01.061 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:10:01.061 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:10:01.062 DEBUG:boto:Method: HEAD 2022-09-12T20:10:01.062 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:10:01.063 DEBUG:boto:Data: 2022-09-12T20:10:01.063 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.064 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.064 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.065 DEBUG:boto:Params: {} 2022-09-12T20:10:01.066 DEBUG:boto:Token: None 2022-09-12T20:10:01.066 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:10:01.067 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gdZNKjqobu41f+NJVSP8D5x7njU= 2022-09-12T20:10:01.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gdZNKjqobu41f+NJVSP8D5x7njU=', 'Content-Length': '0'} 2022-09-12T20:10:01.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a3ae931da5db3a9-00631f9219-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.070 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:10:01.071 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:10:01.072 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:10:01.072 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:10:01.073 DEBUG:boto:Method: HEAD 2022-09-12T20:10:01.073 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:10:01.074 DEBUG:boto:Data: 2022-09-12T20:10:01.074 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.075 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.075 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.076 DEBUG:boto:Params: {} 2022-09-12T20:10:01.076 DEBUG:boto:Token: None 2022-09-12T20:10:01.077 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:10:01.078 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gdZNKjqobu41f+NJVSP8D5x7njU= 2022-09-12T20:10:01.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gdZNKjqobu41f+NJVSP8D5x7njU=', 'Content-Length': '0'} 2022-09-12T20:10:01.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b11466caa098684-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:10:01.081 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:10:01.082 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:10:01.083 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:10:01.083 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:10:01.084 DEBUG:boto:Method: GET 2022-09-12T20:10:01.084 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:10:01.085 DEBUG:boto:Data: 2022-09-12T20:10:01.085 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.086 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.086 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.087 DEBUG:boto:Params: {} 2022-09-12T20:10:01.088 DEBUG:boto:Token: None 2022-09-12T20:10:01.088 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:10:01.089 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bungudekEmxbk4ZU+PZBztWKKnM= 2022-09-12T20:10:01.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bungudekEmxbk4ZU+PZBztWKKnM=', 'Content-Length': '0'} 2022-09-12T20:10:01.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063590b3d487f24c7-00631f9219-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.093 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:10:01.093 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:10:01.094 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:10:01.094 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:10:01.095 DEBUG:boto:Method: GET 2022-09-12T20:10:01.095 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:10:01.096 DEBUG:boto:Data: 2022-09-12T20:10:01.097 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.097 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.098 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.098 DEBUG:boto:Params: {} 2022-09-12T20:10:01.099 DEBUG:boto:Token: None 2022-09-12T20:10:01.099 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:10:01.100 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bungudekEmxbk4ZU+PZBztWKKnM= 2022-09-12T20:10:01.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bungudekEmxbk4ZU+PZBztWKKnM=', 'Content-Length': '0'} 2022-09-12T20:10:01.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca7d19e68c6dd051-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.146 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:01.147 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:01.147 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:10:01.148 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:10:01.148 DEBUG:boto:Method: GET 2022-09-12T20:10:01.149 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:10:01.149 DEBUG:boto:Data: 2022-09-12T20:10:01.150 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.150 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.151 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.151 DEBUG:boto:Params: {} 2022-09-12T20:10:01.152 DEBUG:boto:Token: None 2022-09-12T20:10:01.153 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/ 2022-09-12T20:10:01.153 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0= 2022-09-12T20:10:01.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0=', 'Content-Length': '0'} 2022-09-12T20:10:01.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000625d43d928af54dd-00631f9219-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.157 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:01.158 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:01.159 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:01.159 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:10:01.160 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:10:01.161 DEBUG:boto:Method: GET 2022-09-12T20:10:01.161 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:10:01.162 DEBUG:boto:Data: 2022-09-12T20:10:01.162 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.163 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.163 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.164 DEBUG:boto:Params: {} 2022-09-12T20:10:01.165 DEBUG:boto:Token: None 2022-09-12T20:10:01.165 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/ 2022-09-12T20:10:01.166 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0= 2022-09-12T20:10:01.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0=', 'Content-Length': '0'} 2022-09-12T20:10:01.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088e25f76684b93b4-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.171 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:01.172 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:10:01.172 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:10:01.173 DEBUG:boto:Method: HEAD 2022-09-12T20:10:01.173 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:10:01.174 DEBUG:boto:Data: 2022-09-12T20:10:01.174 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.175 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.175 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.176 DEBUG:boto:Params: {} 2022-09-12T20:10:01.177 DEBUG:boto:Token: None 2022-09-12T20:10:01.177 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj34 2022-09-12T20:10:01.178 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gaG4VzJpzuZPhFGju/todOfYL7o= 2022-09-12T20:10:01.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gaG4VzJpzuZPhFGju/todOfYL7o=', 'Content-Length': '0'} 2022-09-12T20:10:01.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef9c71d3d17c2b32-00631f9219-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.181 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:10:01.182 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:10:01.182 DEBUG:boto:Method: HEAD 2022-09-12T20:10:01.183 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:10:01.183 DEBUG:boto:Data: 2022-09-12T20:10:01.184 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.184 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.185 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.186 DEBUG:boto:Params: {} 2022-09-12T20:10:01.186 DEBUG:boto:Token: None 2022-09-12T20:10:01.187 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj34 2022-09-12T20:10:01.187 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gaG4VzJpzuZPhFGju/todOfYL7o= 2022-09-12T20:10:01.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gaG4VzJpzuZPhFGju/todOfYL7o=', 'Content-Length': '0'} 2022-09-12T20:10:01.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000877aa6b06cc2d707-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:10:01.191 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:10:01.192 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:10:01.192 DEBUG:boto:Method: GET 2022-09-12T20:10:01.193 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:10:01.194 DEBUG:boto:Data: 2022-09-12T20:10:01.194 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.195 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.195 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.196 DEBUG:boto:Params: {} 2022-09-12T20:10:01.197 DEBUG:boto:Token: None 2022-09-12T20:10:01.197 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj34 2022-09-12T20:10:01.198 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qn7NPRAOBlbLFtTLPEk7MEHdj+M= 2022-09-12T20:10:01.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qn7NPRAOBlbLFtTLPEk7MEHdj+M=', 'Content-Length': '0'} 2022-09-12T20:10:01.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082e148999d47daa9-00631f9219-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.202 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:10:01.202 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:10:01.203 DEBUG:boto:Method: GET 2022-09-12T20:10:01.203 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:10:01.204 DEBUG:boto:Data: 2022-09-12T20:10:01.204 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.205 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.205 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.206 DEBUG:boto:Params: {} 2022-09-12T20:10:01.207 DEBUG:boto:Token: None 2022-09-12T20:10:01.208 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj34 2022-09-12T20:10:01.208 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qn7NPRAOBlbLFtTLPEk7MEHdj+M= 2022-09-12T20:10:01.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qn7NPRAOBlbLFtTLPEk7MEHdj+M=', 'Content-Length': '0'} 2022-09-12T20:10:01.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa22b6c89195422a-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.254 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:01.255 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:01.256 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:10:01.256 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:10:01.257 DEBUG:boto:Method: GET 2022-09-12T20:10:01.257 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:10:01.258 DEBUG:boto:Data: 2022-09-12T20:10:01.258 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.259 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.260 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.260 DEBUG:boto:Params: {} 2022-09-12T20:10:01.261 DEBUG:boto:Token: None 2022-09-12T20:10:01.261 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/ 2022-09-12T20:10:01.262 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0= 2022-09-12T20:10:01.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0=', 'Content-Length': '0'} 2022-09-12T20:10:01.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eaa26d47f0e3319f-00631f9219-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.267 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:01.268 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:01.268 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:01.269 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:10:01.269 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:10:01.270 DEBUG:boto:Method: GET 2022-09-12T20:10:01.270 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:10:01.271 DEBUG:boto:Data: 2022-09-12T20:10:01.271 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.272 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.272 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.273 DEBUG:boto:Params: {} 2022-09-12T20:10:01.274 DEBUG:boto:Token: None 2022-09-12T20:10:01.274 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/ 2022-09-12T20:10:01.275 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0= 2022-09-12T20:10:01.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0=', 'Content-Length': '0'} 2022-09-12T20:10:01.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009126647e9a88287e-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.280 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:01.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:10:01.281 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:10:01.282 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:10:01.282 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:10:01.283 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:10:01.284 DEBUG:boto:Method: GET 2022-09-12T20:10:01.284 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:10:01.285 DEBUG:boto:Data: 2022-09-12T20:10:01.285 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.286 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.286 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.287 DEBUG:boto:Params: {} 2022-09-12T20:10:01.287 DEBUG:boto:Token: None 2022-09-12T20:10:01.288 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:10:01.289 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Z7AcLt7uyWe4CALrdnnLSoJKsHI= 2022-09-12T20:10:01.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Z7AcLt7uyWe4CALrdnnLSoJKsHI=', 'Content-Length': '0'} 2022-09-12T20:10:01.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e095e6c0a06ceda5-00631f9219-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.334 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:10:01.335 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:10:01.336 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:10:01.336 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:10:01.337 DEBUG:boto:Method: GET 2022-09-12T20:10:01.337 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:10:01.338 DEBUG:boto:Data: 2022-09-12T20:10:01.339 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.339 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.340 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.340 DEBUG:boto:Params: {} 2022-09-12T20:10:01.341 DEBUG:boto:Token: None 2022-09-12T20:10:01.342 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:10:01.342 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Z7AcLt7uyWe4CALrdnnLSoJKsHI= 2022-09-12T20:10:01.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Z7AcLt7uyWe4CALrdnnLSoJKsHI=', 'Content-Length': '0'} 2022-09-12T20:10:01.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038af4ca896e71fa5-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.346 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:10:01.347 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:10:01.347 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:10:01.348 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:10:01.348 DEBUG:boto:Method: HEAD 2022-09-12T20:10:01.349 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:10:01.349 DEBUG:boto:Data: 2022-09-12T20:10:01.350 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.351 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.351 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.352 DEBUG:boto:Params: {} 2022-09-12T20:10:01.353 DEBUG:boto:Token: None 2022-09-12T20:10:01.353 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:10:01.354 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UqxR7tzeTbP3eFRi+fGblz3Z/ro= 2022-09-12T20:10:01.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UqxR7tzeTbP3eFRi+fGblz3Z/ro=', 'Content-Length': '0'} 2022-09-12T20:10:01.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eff23bbfd09c4dc2-00631f9219-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.358 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:10:01.359 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:10:01.359 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:10:01.360 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:10:01.360 DEBUG:boto:Method: HEAD 2022-09-12T20:10:01.361 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:10:01.361 DEBUG:boto:Data: 2022-09-12T20:10:01.362 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.363 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.363 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.364 DEBUG:boto:Params: {} 2022-09-12T20:10:01.364 DEBUG:boto:Token: None 2022-09-12T20:10:01.365 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:10:01.366 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UqxR7tzeTbP3eFRi+fGblz3Z/ro= 2022-09-12T20:10:01.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UqxR7tzeTbP3eFRi+fGblz3Z/ro=', 'Content-Length': '0'} 2022-09-12T20:10:01.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090e55955bf053507-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:10:01.370 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:10:01.370 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:10:01.371 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:10:01.371 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:10:01.372 DEBUG:boto:Method: GET 2022-09-12T20:10:01.373 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:10:01.373 DEBUG:boto:Data: 2022-09-12T20:10:01.374 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.374 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.375 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.376 DEBUG:boto:Params: {} 2022-09-12T20:10:01.376 DEBUG:boto:Token: None 2022-09-12T20:10:01.377 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:10:01.378 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Z7AcLt7uyWe4CALrdnnLSoJKsHI= 2022-09-12T20:10:01.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Z7AcLt7uyWe4CALrdnnLSoJKsHI=', 'Content-Length': '0'} 2022-09-12T20:10:01.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf311151000ed96d-00631f9219-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.381 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:10:01.382 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:10:01.382 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:10:01.383 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:10:01.384 DEBUG:boto:Method: GET 2022-09-12T20:10:01.384 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:10:01.385 DEBUG:boto:Data: 2022-09-12T20:10:01.385 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.386 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.386 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.387 DEBUG:boto:Params: {} 2022-09-12T20:10:01.388 DEBUG:boto:Token: None 2022-09-12T20:10:01.388 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:10:01.389 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Z7AcLt7uyWe4CALrdnnLSoJKsHI= 2022-09-12T20:10:01.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Z7AcLt7uyWe4CALrdnnLSoJKsHI=', 'Content-Length': '0'} 2022-09-12T20:10:01.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000411fde96221e62c1-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.393 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:01.393 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:01.394 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:10:01.394 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:10:01.395 DEBUG:boto:Method: GET 2022-09-12T20:10:01.395 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:10:01.396 DEBUG:boto:Data: 2022-09-12T20:10:01.396 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.397 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.398 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.398 DEBUG:boto:Params: {} 2022-09-12T20:10:01.399 DEBUG:boto:Token: None 2022-09-12T20:10:01.399 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/ 2022-09-12T20:10:01.400 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0= 2022-09-12T20:10:01.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0=', 'Content-Length': '0'} 2022-09-12T20:10:01.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a955ef59ec6b89f-00631f9219-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.404 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:01.405 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:01.405 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:01.406 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:10:01.407 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:10:01.407 DEBUG:boto:Method: GET 2022-09-12T20:10:01.408 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:10:01.408 DEBUG:boto:Data: 2022-09-12T20:10:01.409 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.409 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.410 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.410 DEBUG:boto:Params: {} 2022-09-12T20:10:01.411 DEBUG:boto:Token: None 2022-09-12T20:10:01.412 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/ 2022-09-12T20:10:01.412 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0= 2022-09-12T20:10:01.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0=', 'Content-Length': '0'} 2022-09-12T20:10:01.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054f41f337a89bd1f-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.417 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:01.418 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:10:01.419 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:10:01.419 DEBUG:boto:Method: HEAD 2022-09-12T20:10:01.420 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:10:01.420 DEBUG:boto:Data: 2022-09-12T20:10:01.421 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.421 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.422 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.422 DEBUG:boto:Params: {} 2022-09-12T20:10:01.423 DEBUG:boto:Token: None 2022-09-12T20:10:01.424 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj38 2022-09-12T20:10:01.424 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FoUGh285wUxtf7nIRvtiqfxxryk= 2022-09-12T20:10:01.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FoUGh285wUxtf7nIRvtiqfxxryk=', 'Content-Length': '0'} 2022-09-12T20:10:01.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d977ef1e3c0b29df-00631f9219-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.427 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:10:01.428 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:10:01.428 DEBUG:boto:Method: HEAD 2022-09-12T20:10:01.429 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:10:01.429 DEBUG:boto:Data: 2022-09-12T20:10:01.430 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.431 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.431 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.432 DEBUG:boto:Params: {} 2022-09-12T20:10:01.432 DEBUG:boto:Token: None 2022-09-12T20:10:01.433 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj38 2022-09-12T20:10:01.433 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FoUGh285wUxtf7nIRvtiqfxxryk= 2022-09-12T20:10:01.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FoUGh285wUxtf7nIRvtiqfxxryk=', 'Content-Length': '0'} 2022-09-12T20:10:01.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8fbb42dc5f86a3b-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:10:01.437 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:10:01.438 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:10:01.439 DEBUG:boto:Method: GET 2022-09-12T20:10:01.439 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:10:01.440 DEBUG:boto:Data: 2022-09-12T20:10:01.440 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.441 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.441 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.442 DEBUG:boto:Params: {} 2022-09-12T20:10:01.442 DEBUG:boto:Token: None 2022-09-12T20:10:01.443 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj38 2022-09-12T20:10:01.444 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c8m9KiG/GAkN/rT8RubXbHHQjAs= 2022-09-12T20:10:01.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c8m9KiG/GAkN/rT8RubXbHHQjAs=', 'Content-Length': '0'} 2022-09-12T20:10:01.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084ab43abfd24e758-00631f9219-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.490 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:10:01.491 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:10:01.491 DEBUG:boto:Method: GET 2022-09-12T20:10:01.492 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:10:01.493 DEBUG:boto:Data: 2022-09-12T20:10:01.493 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.494 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.494 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.495 DEBUG:boto:Params: {} 2022-09-12T20:10:01.495 DEBUG:boto:Token: None 2022-09-12T20:10:01.496 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj38 2022-09-12T20:10:01.497 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c8m9KiG/GAkN/rT8RubXbHHQjAs= 2022-09-12T20:10:01.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c8m9KiG/GAkN/rT8RubXbHHQjAs=', 'Content-Length': '0'} 2022-09-12T20:10:01.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f3919a43ea1ff52-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.500 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:01.501 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:01.501 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:10:01.502 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:10:01.503 DEBUG:boto:Method: GET 2022-09-12T20:10:01.503 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:10:01.504 DEBUG:boto:Data: 2022-09-12T20:10:01.504 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.505 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.505 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.506 DEBUG:boto:Params: {} 2022-09-12T20:10:01.506 DEBUG:boto:Token: None 2022-09-12T20:10:01.507 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/ 2022-09-12T20:10:01.508 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0= 2022-09-12T20:10:01.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0=', 'Content-Length': '0'} 2022-09-12T20:10:01.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039affa0ab9d2a6a6-00631f9219-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.512 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:01.512 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:01.513 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:01.514 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:10:01.514 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:10:01.515 DEBUG:boto:Method: GET 2022-09-12T20:10:01.515 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:10:01.516 DEBUG:boto:Data: 2022-09-12T20:10:01.516 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.517 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.517 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.518 DEBUG:boto:Params: {} 2022-09-12T20:10:01.519 DEBUG:boto:Token: None 2022-09-12T20:10:01.519 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/ 2022-09-12T20:10:01.520 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0= 2022-09-12T20:10:01.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0=', 'Content-Length': '0'} 2022-09-12T20:10:01.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b21e0e22ddeec302-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.525 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:01.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:10:01.526 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:10:01.527 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:10:01.527 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:10:01.528 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:10:01.528 DEBUG:boto:Method: GET 2022-09-12T20:10:01.529 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:10:01.530 DEBUG:boto:Data: 2022-09-12T20:10:01.530 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.531 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.531 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.532 DEBUG:boto:Params: {} 2022-09-12T20:10:01.532 DEBUG:boto:Token: None 2022-09-12T20:10:01.533 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:10:01.534 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lPpKG7IscDuZEFmlNf1squsRVzg= 2022-09-12T20:10:01.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lPpKG7IscDuZEFmlNf1squsRVzg=', 'Content-Length': '0'} 2022-09-12T20:10:01.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dbdb9a82ed9445d-00631f9219-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.538 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:10:01.538 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:10:01.539 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:10:01.539 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:10:01.540 DEBUG:boto:Method: GET 2022-09-12T20:10:01.540 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:10:01.541 DEBUG:boto:Data: 2022-09-12T20:10:01.542 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.542 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.543 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.543 DEBUG:boto:Params: {} 2022-09-12T20:10:01.544 DEBUG:boto:Token: None 2022-09-12T20:10:01.544 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:10:01.545 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lPpKG7IscDuZEFmlNf1squsRVzg= 2022-09-12T20:10:01.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lPpKG7IscDuZEFmlNf1squsRVzg=', 'Content-Length': '0'} 2022-09-12T20:10:01.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbda8928e15f12da-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.590 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:10:01.591 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:10:01.591 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:10:01.592 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:10:01.592 DEBUG:boto:Method: HEAD 2022-09-12T20:10:01.593 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:10:01.593 DEBUG:boto:Data: 2022-09-12T20:10:01.594 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.595 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.595 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.596 DEBUG:boto:Params: {} 2022-09-12T20:10:01.596 DEBUG:boto:Token: None 2022-09-12T20:10:01.597 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:10:01.597 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SUVF5PFcmalhTvHuxVrYglvDcrE= 2022-09-12T20:10:01.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SUVF5PFcmalhTvHuxVrYglvDcrE=', 'Content-Length': '0'} 2022-09-12T20:10:01.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7365a67a5fbd11f-00631f9219-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.601 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:10:01.602 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:10:01.602 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:10:01.603 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:10:01.603 DEBUG:boto:Method: HEAD 2022-09-12T20:10:01.604 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:10:01.605 DEBUG:boto:Data: 2022-09-12T20:10:01.606 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.607 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.607 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.608 DEBUG:boto:Params: {} 2022-09-12T20:10:01.608 DEBUG:boto:Token: None 2022-09-12T20:10:01.609 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:10:01.610 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SUVF5PFcmalhTvHuxVrYglvDcrE= 2022-09-12T20:10:01.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SUVF5PFcmalhTvHuxVrYglvDcrE=', 'Content-Length': '0'} 2022-09-12T20:10:01.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a216094a2138d1e-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:10:01.614 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:10:01.615 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:10:01.617 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:10:01.618 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:10:01.619 DEBUG:boto:Method: GET 2022-09-12T20:10:01.620 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:10:01.620 DEBUG:boto:Data: 2022-09-12T20:10:01.621 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.621 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.622 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.622 DEBUG:boto:Params: {} 2022-09-12T20:10:01.623 DEBUG:boto:Token: None 2022-09-12T20:10:01.624 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:10:01.624 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lPpKG7IscDuZEFmlNf1squsRVzg= 2022-09-12T20:10:01.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lPpKG7IscDuZEFmlNf1squsRVzg=', 'Content-Length': '0'} 2022-09-12T20:10:01.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3d40f63df2f127e-00631f9219-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.628 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:10:01.629 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:10:01.629 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:10:01.630 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:10:01.631 DEBUG:boto:Method: GET 2022-09-12T20:10:01.631 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:10:01.632 DEBUG:boto:Data: 2022-09-12T20:10:01.632 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.633 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.633 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.634 DEBUG:boto:Params: {} 2022-09-12T20:10:01.634 DEBUG:boto:Token: None 2022-09-12T20:10:01.635 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:10:01.636 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lPpKG7IscDuZEFmlNf1squsRVzg= 2022-09-12T20:10:01.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lPpKG7IscDuZEFmlNf1squsRVzg=', 'Content-Length': '0'} 2022-09-12T20:10:01.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0f0d099605ea226-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.640 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:01.640 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:01.641 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:10:01.641 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:10:01.642 DEBUG:boto:Method: GET 2022-09-12T20:10:01.642 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:10:01.643 DEBUG:boto:Data: 2022-09-12T20:10:01.643 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.644 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.644 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.645 DEBUG:boto:Params: {} 2022-09-12T20:10:01.646 DEBUG:boto:Token: None 2022-09-12T20:10:01.646 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/ 2022-09-12T20:10:01.647 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0= 2022-09-12T20:10:01.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0=', 'Content-Length': '0'} 2022-09-12T20:10:01.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2132c0cc918269b-00631f9219-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.651 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:01.652 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:01.653 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:01.653 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:10:01.654 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:10:01.654 DEBUG:boto:Method: GET 2022-09-12T20:10:01.655 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:10:01.656 DEBUG:boto:Data: 2022-09-12T20:10:01.656 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.657 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.657 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.658 DEBUG:boto:Params: {} 2022-09-12T20:10:01.658 DEBUG:boto:Token: None 2022-09-12T20:10:01.659 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/ 2022-09-12T20:10:01.660 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0= 2022-09-12T20:10:01.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0=', 'Content-Length': '0'} 2022-09-12T20:10:01.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbf87af77cc406a2-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.665 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:01.666 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:10:01.666 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:10:01.667 DEBUG:boto:Method: HEAD 2022-09-12T20:10:01.667 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:10:01.668 DEBUG:boto:Data: 2022-09-12T20:10:01.669 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.669 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.670 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.670 DEBUG:boto:Params: {} 2022-09-12T20:10:01.671 DEBUG:boto:Token: None 2022-09-12T20:10:01.671 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj44 2022-09-12T20:10:01.672 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iBmLrl7o7KzcEP+krjudH6VChxI= 2022-09-12T20:10:01.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iBmLrl7o7KzcEP+krjudH6VChxI=', 'Content-Length': '0'} 2022-09-12T20:10:01.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000342f9579ef3f5c17-00631f9219-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.675 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:10:01.676 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:10:01.676 DEBUG:boto:Method: HEAD 2022-09-12T20:10:01.677 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:10:01.678 DEBUG:boto:Data: 2022-09-12T20:10:01.678 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.679 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.679 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.680 DEBUG:boto:Params: {} 2022-09-12T20:10:01.680 DEBUG:boto:Token: None 2022-09-12T20:10:01.681 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj44 2022-09-12T20:10:01.682 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iBmLrl7o7KzcEP+krjudH6VChxI= 2022-09-12T20:10:01.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iBmLrl7o7KzcEP+krjudH6VChxI=', 'Content-Length': '0'} 2022-09-12T20:10:01.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e01f8949ec6203c5-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:10:01.686 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:10:01.686 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:10:01.687 DEBUG:boto:Method: GET 2022-09-12T20:10:01.688 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:10:01.688 DEBUG:boto:Data: 2022-09-12T20:10:01.689 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.689 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.690 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.690 DEBUG:boto:Params: {} 2022-09-12T20:10:01.691 DEBUG:boto:Token: None 2022-09-12T20:10:01.692 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj44 2022-09-12T20:10:01.692 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5I8RcXF5LoMqLeTUXTYpTSutHyE= 2022-09-12T20:10:01.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5I8RcXF5LoMqLeTUXTYpTSutHyE=', 'Content-Length': '0'} 2022-09-12T20:10:01.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003de5702d4cba7870-00631f9219-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.738 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:10:01.739 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:10:01.740 DEBUG:boto:Method: GET 2022-09-12T20:10:01.740 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:10:01.741 DEBUG:boto:Data: 2022-09-12T20:10:01.741 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.742 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.743 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.743 DEBUG:boto:Params: {} 2022-09-12T20:10:01.744 DEBUG:boto:Token: None 2022-09-12T20:10:01.745 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj44 2022-09-12T20:10:01.745 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5I8RcXF5LoMqLeTUXTYpTSutHyE= 2022-09-12T20:10:01.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5I8RcXF5LoMqLeTUXTYpTSutHyE=', 'Content-Length': '0'} 2022-09-12T20:10:01.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060e43390c85b84e5-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.749 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:01.750 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:01.751 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:10:01.751 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:10:01.752 DEBUG:boto:Method: GET 2022-09-12T20:10:01.752 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:10:01.753 DEBUG:boto:Data: 2022-09-12T20:10:01.754 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.754 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.755 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.755 DEBUG:boto:Params: {} 2022-09-12T20:10:01.756 DEBUG:boto:Token: None 2022-09-12T20:10:01.757 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/ 2022-09-12T20:10:01.757 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0= 2022-09-12T20:10:01.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0=', 'Content-Length': '0'} 2022-09-12T20:10:01.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddc37b6c22ada6d3-00631f9219-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.762 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:01.763 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:01.763 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:01.764 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:10:01.764 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:10:01.765 DEBUG:boto:Method: GET 2022-09-12T20:10:01.766 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:10:01.766 DEBUG:boto:Data: 2022-09-12T20:10:01.767 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.767 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.768 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.768 DEBUG:boto:Params: {} 2022-09-12T20:10:01.769 DEBUG:boto:Token: None 2022-09-12T20:10:01.770 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/ 2022-09-12T20:10:01.770 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0= 2022-09-12T20:10:01.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0=', 'Content-Length': '0'} 2022-09-12T20:10:01.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000977e74af8045f02a-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.775 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:01.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:10:01.777 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:10:01.777 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:10:01.778 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:10:01.779 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:10:01.779 DEBUG:boto:Method: GET 2022-09-12T20:10:01.780 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:10:01.780 DEBUG:boto:Data: 2022-09-12T20:10:01.781 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.782 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.782 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.783 DEBUG:boto:Params: {} 2022-09-12T20:10:01.783 DEBUG:boto:Token: None 2022-09-12T20:10:01.784 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:10:01.785 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UzjoBGqzOGqIuS/ro6kOgYHiWtg= 2022-09-12T20:10:01.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UzjoBGqzOGqIuS/ro6kOgYHiWtg=', 'Content-Length': '0'} 2022-09-12T20:10:01.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8ba5fbea4e0e2d3-00631f9219-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.789 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:10:01.789 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:10:01.790 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:10:01.790 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:10:01.791 DEBUG:boto:Method: GET 2022-09-12T20:10:01.792 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:10:01.792 DEBUG:boto:Data: 2022-09-12T20:10:01.793 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.793 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.794 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.795 DEBUG:boto:Params: {} 2022-09-12T20:10:01.795 DEBUG:boto:Token: None 2022-09-12T20:10:01.796 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:10:01.796 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UzjoBGqzOGqIuS/ro6kOgYHiWtg= 2022-09-12T20:10:01.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UzjoBGqzOGqIuS/ro6kOgYHiWtg=', 'Content-Length': '0'} 2022-09-12T20:10:01.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e40f887bdf63277-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.800 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:10:01.801 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:10:01.801 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:10:01.802 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:10:01.802 DEBUG:boto:Method: HEAD 2022-09-12T20:10:01.803 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:10:01.803 DEBUG:boto:Data: 2022-09-12T20:10:01.804 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.804 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.805 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.805 DEBUG:boto:Params: {} 2022-09-12T20:10:01.806 DEBUG:boto:Token: None 2022-09-12T20:10:01.807 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:10:01.807 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jwn/feunnFt4XHB+gLRiigvMe0w= 2022-09-12T20:10:01.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jwn/feunnFt4XHB+gLRiigvMe0w=', 'Content-Length': '0'} 2022-09-12T20:10:01.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ae951f42c633206-00631f9219-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.811 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:10:01.811 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:10:01.812 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:10:01.812 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:10:01.813 DEBUG:boto:Method: HEAD 2022-09-12T20:10:01.813 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:10:01.814 DEBUG:boto:Data: 2022-09-12T20:10:01.814 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.815 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.815 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.816 DEBUG:boto:Params: {} 2022-09-12T20:10:01.816 DEBUG:boto:Token: None 2022-09-12T20:10:01.817 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:10:01.818 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jwn/feunnFt4XHB+gLRiigvMe0w= 2022-09-12T20:10:01.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jwn/feunnFt4XHB+gLRiigvMe0w=', 'Content-Length': '0'} 2022-09-12T20:10:01.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000037521aa7015fe55-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:10:01.822 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:10:01.822 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:10:01.823 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:10:01.823 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:10:01.824 DEBUG:boto:Method: GET 2022-09-12T20:10:01.825 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:10:01.825 DEBUG:boto:Data: 2022-09-12T20:10:01.826 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.826 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.827 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.827 DEBUG:boto:Params: {} 2022-09-12T20:10:01.828 DEBUG:boto:Token: None 2022-09-12T20:10:01.829 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:10:01.829 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UzjoBGqzOGqIuS/ro6kOgYHiWtg= 2022-09-12T20:10:01.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UzjoBGqzOGqIuS/ro6kOgYHiWtg=', 'Content-Length': '0'} 2022-09-12T20:10:01.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0a7763541e954f3-00631f9219-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.833 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:10:01.834 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:10:01.834 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:10:01.835 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:10:01.835 DEBUG:boto:Method: GET 2022-09-12T20:10:01.836 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:10:01.836 DEBUG:boto:Data: 2022-09-12T20:10:01.837 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.837 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.838 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.839 DEBUG:boto:Params: {} 2022-09-12T20:10:01.839 DEBUG:boto:Token: None 2022-09-12T20:10:01.840 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:10:01.840 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UzjoBGqzOGqIuS/ro6kOgYHiWtg= 2022-09-12T20:10:01.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UzjoBGqzOGqIuS/ro6kOgYHiWtg=', 'Content-Length': '0'} 2022-09-12T20:10:01.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ec634fd126ad40f-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.844 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:01.845 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:01.845 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:10:01.846 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:10:01.846 DEBUG:boto:Method: GET 2022-09-12T20:10:01.847 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:10:01.847 DEBUG:boto:Data: 2022-09-12T20:10:01.848 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.849 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.849 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.850 DEBUG:boto:Params: {} 2022-09-12T20:10:01.850 DEBUG:boto:Token: None 2022-09-12T20:10:01.851 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/ 2022-09-12T20:10:01.852 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0= 2022-09-12T20:10:01.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0=', 'Content-Length': '0'} 2022-09-12T20:10:01.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0c1e46f2ff3872b-00631f9219-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.856 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:01.857 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:01.857 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:01.858 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:10:01.859 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:10:01.859 DEBUG:boto:Method: GET 2022-09-12T20:10:01.860 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:10:01.860 DEBUG:boto:Data: 2022-09-12T20:10:01.861 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.861 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.862 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.863 DEBUG:boto:Params: {} 2022-09-12T20:10:01.863 DEBUG:boto:Token: None 2022-09-12T20:10:01.864 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/ 2022-09-12T20:10:01.865 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0= 2022-09-12T20:10:01.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0=', 'Content-Length': '0'} 2022-09-12T20:10:01.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003cb48259eae50e1-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.870 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:01.871 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:10:01.871 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:10:01.872 DEBUG:boto:Method: HEAD 2022-09-12T20:10:01.872 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:10:01.873 DEBUG:boto:Data: 2022-09-12T20:10:01.873 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.874 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.875 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.875 DEBUG:boto:Params: {} 2022-09-12T20:10:01.876 DEBUG:boto:Token: None 2022-09-12T20:10:01.876 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj46 2022-09-12T20:10:01.877 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J+Tt+QmvM1+Ifp6Q5PfbotCLGrY= 2022-09-12T20:10:01.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J+Tt+QmvM1+Ifp6Q5PfbotCLGrY=', 'Content-Length': '0'} 2022-09-12T20:10:01.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035e2cedb236a96df-00631f9219-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.881 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:10:01.881 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:10:01.882 DEBUG:boto:Method: HEAD 2022-09-12T20:10:01.882 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:10:01.883 DEBUG:boto:Data: 2022-09-12T20:10:01.883 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.884 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.885 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.885 DEBUG:boto:Params: {} 2022-09-12T20:10:01.886 DEBUG:boto:Token: None 2022-09-12T20:10:01.886 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj46 2022-09-12T20:10:01.887 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J+Tt+QmvM1+Ifp6Q5PfbotCLGrY= 2022-09-12T20:10:01.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J+Tt+QmvM1+Ifp6Q5PfbotCLGrY=', 'Content-Length': '0'} 2022-09-12T20:10:01.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e677e18af71511b7-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:10:01.892 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:10:01.892 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:10:01.893 DEBUG:boto:Method: GET 2022-09-12T20:10:01.894 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:10:01.894 DEBUG:boto:Data: 2022-09-12T20:10:01.895 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.895 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.896 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.896 DEBUG:boto:Params: {} 2022-09-12T20:10:01.897 DEBUG:boto:Token: None 2022-09-12T20:10:01.898 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj46 2022-09-12T20:10:01.898 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E0TXmhyS2gbTIAQOaYR1CLj2Zzg= 2022-09-12T20:10:01.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E0TXmhyS2gbTIAQOaYR1CLj2Zzg=', 'Content-Length': '0'} 2022-09-12T20:10:01.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ca715be486f30cb-00631f9219-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.902 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:10:01.902 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:10:01.903 DEBUG:boto:Method: GET 2022-09-12T20:10:01.904 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:10:01.904 DEBUG:boto:Data: 2022-09-12T20:10:01.905 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.905 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.906 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.906 DEBUG:boto:Params: {} 2022-09-12T20:10:01.907 DEBUG:boto:Token: None 2022-09-12T20:10:01.908 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj46 2022-09-12T20:10:01.908 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E0TXmhyS2gbTIAQOaYR1CLj2Zzg= 2022-09-12T20:10:01.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E0TXmhyS2gbTIAQOaYR1CLj2Zzg=', 'Content-Length': '0'} 2022-09-12T20:10:01.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d36c6ed29a1129b3-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.912 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:01.913 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:01.913 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:10:01.914 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:10:01.914 DEBUG:boto:Method: GET 2022-09-12T20:10:01.915 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:10:01.916 DEBUG:boto:Data: 2022-09-12T20:10:01.916 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.917 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.917 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.918 DEBUG:boto:Params: {} 2022-09-12T20:10:01.918 DEBUG:boto:Token: None 2022-09-12T20:10:01.919 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/ 2022-09-12T20:10:01.920 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0= 2022-09-12T20:10:01.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0=', 'Content-Length': '0'} 2022-09-12T20:10:01.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a58f17839467cd85-00631f9219-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.924 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:01.925 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:01.926 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:01.926 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:10:01.927 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:10:01.927 DEBUG:boto:Method: GET 2022-09-12T20:10:01.928 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:10:01.928 DEBUG:boto:Data: 2022-09-12T20:10:01.929 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.929 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.930 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.931 DEBUG:boto:Params: {} 2022-09-12T20:10:01.931 DEBUG:boto:Token: None 2022-09-12T20:10:01.932 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/ 2022-09-12T20:10:01.932 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0= 2022-09-12T20:10:01.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZPNw3gc/JYLtWgRLYIYeaJwnah0=', 'Content-Length': '0'} 2022-09-12T20:10:01.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000412c4d0302381550-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.937 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:01.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:10:01.939 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:10:01.939 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:10:01.940 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:10:01.940 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:10:01.941 DEBUG:boto:Method: GET 2022-09-12T20:10:01.942 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:10:01.942 DEBUG:boto:Data: 2022-09-12T20:10:01.943 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.943 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:01.944 DEBUG:boto:Port: 8001 2022-09-12T20:10:01.944 DEBUG:boto:Params: {} 2022-09-12T20:10:01.945 DEBUG:boto:Token: None 2022-09-12T20:10:01.946 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:10:01.946 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tpkcILJ4LF0KBSY59w19cmHzbdE= 2022-09-12T20:10:01.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tpkcILJ4LF0KBSY59w19cmHzbdE=', 'Content-Length': '0'} 2022-09-12T20:10:01.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff9a9948ff24826b-00631f9219-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:01.990 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:10:01.991 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:10:01.991 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:10:01.992 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:10:01.993 DEBUG:boto:Method: GET 2022-09-12T20:10:01.993 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:10:01.994 DEBUG:boto:Data: 2022-09-12T20:10:01.994 DEBUG:boto:Headers: {} 2022-09-12T20:10:01.995 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:01.995 DEBUG:boto:Port: 8000 2022-09-12T20:10:01.996 DEBUG:boto:Params: {} 2022-09-12T20:10:01.997 DEBUG:boto:Token: None 2022-09-12T20:10:01.997 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:01 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:10:01.998 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tpkcILJ4LF0KBSY59w19cmHzbdE= 2022-09-12T20:10:01.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tpkcILJ4LF0KBSY59w19cmHzbdE=', 'Content-Length': '0'} 2022-09-12T20:10:02.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bec9806142f3d192-00631f9219-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.002 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:10:02.003 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:10:02.003 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:10:02.004 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:10:02.004 DEBUG:boto:Method: HEAD 2022-09-12T20:10:02.005 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:10:02.005 DEBUG:boto:Data: 2022-09-12T20:10:02.006 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.006 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.007 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.007 DEBUG:boto:Params: {} 2022-09-12T20:10:02.008 DEBUG:boto:Token: None 2022-09-12T20:10:02.009 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:10:02.009 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cefns+OhthVWgor9dsm21tCqyQA= 2022-09-12T20:10:02.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cefns+OhthVWgor9dsm21tCqyQA=', 'Content-Length': '0'} 2022-09-12T20:10:02.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d91ba64968191cee-00631f921a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.013 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:10:02.013 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:10:02.014 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:10:02.014 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:10:02.015 DEBUG:boto:Method: HEAD 2022-09-12T20:10:02.016 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:10:02.016 DEBUG:boto:Data: 2022-09-12T20:10:02.017 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.017 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.018 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.018 DEBUG:boto:Params: {} 2022-09-12T20:10:02.019 DEBUG:boto:Token: None 2022-09-12T20:10:02.019 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:10:02.020 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cefns+OhthVWgor9dsm21tCqyQA= 2022-09-12T20:10:02.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cefns+OhthVWgor9dsm21tCqyQA=', 'Content-Length': '0'} 2022-09-12T20:10:02.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad3430ddc61a3760-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:10:02.024 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:10:02.024 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:10:02.025 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:10:02.026 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:10:02.026 DEBUG:boto:Method: GET 2022-09-12T20:10:02.027 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:10:02.027 DEBUG:boto:Data: 2022-09-12T20:10:02.028 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.028 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.029 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.029 DEBUG:boto:Params: {} 2022-09-12T20:10:02.030 DEBUG:boto:Token: None 2022-09-12T20:10:02.031 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:10:02.032 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YwmzS0qn+Wt61XAQm/ttODLTZP4= 2022-09-12T20:10:02.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YwmzS0qn+Wt61XAQm/ttODLTZP4=', 'Content-Length': '0'} 2022-09-12T20:10:02.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2a307a3900caac7-00631f921a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.035 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:10:02.036 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:10:02.037 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:10:02.037 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:10:02.038 DEBUG:boto:Method: GET 2022-09-12T20:10:02.039 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:10:02.039 DEBUG:boto:Data: 2022-09-12T20:10:02.040 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.040 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.041 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.042 DEBUG:boto:Params: {} 2022-09-12T20:10:02.042 DEBUG:boto:Token: None 2022-09-12T20:10:02.043 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:10:02.043 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YwmzS0qn+Wt61XAQm/ttODLTZP4= 2022-09-12T20:10:02.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YwmzS0qn+Wt61XAQm/ttODLTZP4=', 'Content-Length': '0'} 2022-09-12T20:10:02.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005110509cc7f8fe21-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.090 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:02.091 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:02.091 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:10:02.092 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:10:02.093 DEBUG:boto:Method: GET 2022-09-12T20:10:02.093 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:10:02.094 DEBUG:boto:Data: 2022-09-12T20:10:02.094 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.095 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.095 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.096 DEBUG:boto:Params: {} 2022-09-12T20:10:02.097 DEBUG:boto:Token: None 2022-09-12T20:10:02.097 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/ 2022-09-12T20:10:02.098 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ= 2022-09-12T20:10:02.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ=', 'Content-Length': '0'} 2022-09-12T20:10:02.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000144c9d02927e00b1-00631f921a-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.102 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:02.103 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:02.104 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:02.104 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:10:02.105 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:10:02.105 DEBUG:boto:Method: GET 2022-09-12T20:10:02.106 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:10:02.106 DEBUG:boto:Data: 2022-09-12T20:10:02.107 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.107 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.108 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.108 DEBUG:boto:Params: {} 2022-09-12T20:10:02.109 DEBUG:boto:Token: None 2022-09-12T20:10:02.110 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/ 2022-09-12T20:10:02.110 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ= 2022-09-12T20:10:02.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ=', 'Content-Length': '0'} 2022-09-12T20:10:02.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f009bba65df2d83-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.115 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:02.116 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:10:02.117 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:10:02.117 DEBUG:boto:Method: HEAD 2022-09-12T20:10:02.118 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:10:02.119 DEBUG:boto:Data: 2022-09-12T20:10:02.119 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.120 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.120 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.121 DEBUG:boto:Params: {} 2022-09-12T20:10:02.121 DEBUG:boto:Token: None 2022-09-12T20:10:02.122 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj51 2022-09-12T20:10:02.122 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ph8Fr9FEwCvFBx9ID5GgtkomPMQ= 2022-09-12T20:10:02.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ph8Fr9FEwCvFBx9ID5GgtkomPMQ=', 'Content-Length': '0'} 2022-09-12T20:10:02.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8ee0dbca5db2f0c-00631f921a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.126 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:10:02.126 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:10:02.127 DEBUG:boto:Method: HEAD 2022-09-12T20:10:02.128 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:10:02.128 DEBUG:boto:Data: 2022-09-12T20:10:02.129 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.129 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.130 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.130 DEBUG:boto:Params: {} 2022-09-12T20:10:02.131 DEBUG:boto:Token: None 2022-09-12T20:10:02.132 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj51 2022-09-12T20:10:02.132 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ph8Fr9FEwCvFBx9ID5GgtkomPMQ= 2022-09-12T20:10:02.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ph8Fr9FEwCvFBx9ID5GgtkomPMQ=', 'Content-Length': '0'} 2022-09-12T20:10:02.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8b33f8a1d580ec6-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:10:02.136 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:10:02.137 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:10:02.138 DEBUG:boto:Method: GET 2022-09-12T20:10:02.138 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:10:02.139 DEBUG:boto:Data: 2022-09-12T20:10:02.140 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.140 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.141 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.141 DEBUG:boto:Params: {} 2022-09-12T20:10:02.142 DEBUG:boto:Token: None 2022-09-12T20:10:02.143 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj51 2022-09-12T20:10:02.144 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N1zPQCCQZIf4UnSKxBhj52tlDrs= 2022-09-12T20:10:02.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N1zPQCCQZIf4UnSKxBhj52tlDrs=', 'Content-Length': '0'} 2022-09-12T20:10:02.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000330cee5fdfb8bf69-00631f921a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.147 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:10:02.148 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:10:02.148 DEBUG:boto:Method: GET 2022-09-12T20:10:02.149 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:10:02.150 DEBUG:boto:Data: 2022-09-12T20:10:02.150 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.151 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.151 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.152 DEBUG:boto:Params: {} 2022-09-12T20:10:02.152 DEBUG:boto:Token: None 2022-09-12T20:10:02.153 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj51 2022-09-12T20:10:02.154 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N1zPQCCQZIf4UnSKxBhj52tlDrs= 2022-09-12T20:10:02.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N1zPQCCQZIf4UnSKxBhj52tlDrs=', 'Content-Length': '0'} 2022-09-12T20:10:02.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b06ec9efb1778fc-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.198 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:02.199 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:02.199 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:10:02.200 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:10:02.200 DEBUG:boto:Method: GET 2022-09-12T20:10:02.201 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:10:02.201 DEBUG:boto:Data: 2022-09-12T20:10:02.202 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.203 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.203 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.204 DEBUG:boto:Params: {} 2022-09-12T20:10:02.204 DEBUG:boto:Token: None 2022-09-12T20:10:02.205 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/ 2022-09-12T20:10:02.205 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ= 2022-09-12T20:10:02.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ=', 'Content-Length': '0'} 2022-09-12T20:10:02.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0164dcfd046afef-00631f921a-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.210 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:02.211 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:02.211 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:02.212 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:10:02.212 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:10:02.213 DEBUG:boto:Method: GET 2022-09-12T20:10:02.213 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:10:02.214 DEBUG:boto:Data: 2022-09-12T20:10:02.215 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.215 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.216 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.216 DEBUG:boto:Params: {} 2022-09-12T20:10:02.217 DEBUG:boto:Token: None 2022-09-12T20:10:02.218 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/ 2022-09-12T20:10:02.218 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ= 2022-09-12T20:10:02.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ=', 'Content-Length': '0'} 2022-09-12T20:10:02.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052996ec5e40ab28e-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.223 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:02.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:10:02.225 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:10:02.225 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:10:02.226 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:10:02.226 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:10:02.227 DEBUG:boto:Method: GET 2022-09-12T20:10:02.227 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:10:02.228 DEBUG:boto:Data: 2022-09-12T20:10:02.229 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.229 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.230 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.230 DEBUG:boto:Params: {} 2022-09-12T20:10:02.231 DEBUG:boto:Token: None 2022-09-12T20:10:02.232 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:10:02.232 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UnkQp3GsGKZVnfu4bNhQ3PC4cMU= 2022-09-12T20:10:02.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UnkQp3GsGKZVnfu4bNhQ3PC4cMU=', 'Content-Length': '0'} 2022-09-12T20:10:02.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b44a7ccebb68c6f5-00631f921a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.236 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:10:02.237 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:10:02.238 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:10:02.238 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:10:02.239 DEBUG:boto:Method: GET 2022-09-12T20:10:02.239 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:10:02.240 DEBUG:boto:Data: 2022-09-12T20:10:02.240 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.241 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.242 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.242 DEBUG:boto:Params: {} 2022-09-12T20:10:02.243 DEBUG:boto:Token: None 2022-09-12T20:10:02.244 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:10:02.244 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UnkQp3GsGKZVnfu4bNhQ3PC4cMU= 2022-09-12T20:10:02.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UnkQp3GsGKZVnfu4bNhQ3PC4cMU=', 'Content-Length': '0'} 2022-09-12T20:10:02.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058e0f2a2cd30ede1-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.248 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:10:02.249 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:10:02.249 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:10:02.250 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:10:02.250 DEBUG:boto:Method: HEAD 2022-09-12T20:10:02.251 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:10:02.251 DEBUG:boto:Data: 2022-09-12T20:10:02.252 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.252 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.253 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.253 DEBUG:boto:Params: {} 2022-09-12T20:10:02.254 DEBUG:boto:Token: None 2022-09-12T20:10:02.255 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:10:02.255 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5JOoi6mNFfXvnKQ32jwaXGqmt5M= 2022-09-12T20:10:02.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5JOoi6mNFfXvnKQ32jwaXGqmt5M=', 'Content-Length': '0'} 2022-09-12T20:10:02.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000242e6f4cb860fa05-00631f921a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.259 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:10:02.259 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:10:02.260 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:10:02.260 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:10:02.261 DEBUG:boto:Method: HEAD 2022-09-12T20:10:02.261 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:10:02.262 DEBUG:boto:Data: 2022-09-12T20:10:02.262 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.263 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.263 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.264 DEBUG:boto:Params: {} 2022-09-12T20:10:02.265 DEBUG:boto:Token: None 2022-09-12T20:10:02.265 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:10:02.266 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5JOoi6mNFfXvnKQ32jwaXGqmt5M= 2022-09-12T20:10:02.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5JOoi6mNFfXvnKQ32jwaXGqmt5M=', 'Content-Length': '0'} 2022-09-12T20:10:02.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a41dd7dcb62e399-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:10:02.270 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:10:02.271 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:10:02.271 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:10:02.272 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:10:02.272 DEBUG:boto:Method: GET 2022-09-12T20:10:02.273 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:10:02.274 DEBUG:boto:Data: 2022-09-12T20:10:02.274 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.275 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.275 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.276 DEBUG:boto:Params: {} 2022-09-12T20:10:02.277 DEBUG:boto:Token: None 2022-09-12T20:10:02.277 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:10:02.278 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UnkQp3GsGKZVnfu4bNhQ3PC4cMU= 2022-09-12T20:10:02.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UnkQp3GsGKZVnfu4bNhQ3PC4cMU=', 'Content-Length': '0'} 2022-09-12T20:10:02.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bc7fb64fd7510fe-00631f921a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.282 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:10:02.283 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:10:02.284 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:10:02.285 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:10:02.286 DEBUG:boto:Method: GET 2022-09-12T20:10:02.286 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:10:02.287 DEBUG:boto:Data: 2022-09-12T20:10:02.288 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.288 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.289 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.289 DEBUG:boto:Params: {} 2022-09-12T20:10:02.290 DEBUG:boto:Token: None 2022-09-12T20:10:02.291 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:10:02.291 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UnkQp3GsGKZVnfu4bNhQ3PC4cMU= 2022-09-12T20:10:02.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UnkQp3GsGKZVnfu4bNhQ3PC4cMU=', 'Content-Length': '0'} 2022-09-12T20:10:02.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aafb5bdf8caea9ad-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.295 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:02.296 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:02.296 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:10:02.297 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:10:02.297 DEBUG:boto:Method: GET 2022-09-12T20:10:02.298 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:10:02.299 DEBUG:boto:Data: 2022-09-12T20:10:02.299 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.300 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.300 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.301 DEBUG:boto:Params: {} 2022-09-12T20:10:02.301 DEBUG:boto:Token: None 2022-09-12T20:10:02.302 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/ 2022-09-12T20:10:02.303 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ= 2022-09-12T20:10:02.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ=', 'Content-Length': '0'} 2022-09-12T20:10:02.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033dc0e0f98ceb778-00631f921a-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.307 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:02.308 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:02.308 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:02.309 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:10:02.309 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:10:02.310 DEBUG:boto:Method: GET 2022-09-12T20:10:02.311 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:10:02.311 DEBUG:boto:Data: 2022-09-12T20:10:02.312 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.312 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.313 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.313 DEBUG:boto:Params: {} 2022-09-12T20:10:02.314 DEBUG:boto:Token: None 2022-09-12T20:10:02.315 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/ 2022-09-12T20:10:02.315 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ= 2022-09-12T20:10:02.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ=', 'Content-Length': '0'} 2022-09-12T20:10:02.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bc33a970d6ae4d5-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.362 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:02.363 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:10:02.364 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:10:02.364 DEBUG:boto:Method: HEAD 2022-09-12T20:10:02.365 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:10:02.366 DEBUG:boto:Data: 2022-09-12T20:10:02.366 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.367 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.368 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.368 DEBUG:boto:Params: {} 2022-09-12T20:10:02.369 DEBUG:boto:Token: None 2022-09-12T20:10:02.370 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj55 2022-09-12T20:10:02.370 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ng1nL4JMHeHA/9oTxhiXtRHQwiY= 2022-09-12T20:10:02.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ng1nL4JMHeHA/9oTxhiXtRHQwiY=', 'Content-Length': '0'} 2022-09-12T20:10:02.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c0bb0cb96d9dd63-00631f921a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.374 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:10:02.375 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:10:02.375 DEBUG:boto:Method: HEAD 2022-09-12T20:10:02.376 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:10:02.376 DEBUG:boto:Data: 2022-09-12T20:10:02.377 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.377 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.378 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.379 DEBUG:boto:Params: {} 2022-09-12T20:10:02.379 DEBUG:boto:Token: None 2022-09-12T20:10:02.380 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj55 2022-09-12T20:10:02.380 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ng1nL4JMHeHA/9oTxhiXtRHQwiY= 2022-09-12T20:10:02.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ng1nL4JMHeHA/9oTxhiXtRHQwiY=', 'Content-Length': '0'} 2022-09-12T20:10:02.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037e7b334b9aea8f6-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:10:02.384 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:10:02.385 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:10:02.386 DEBUG:boto:Method: GET 2022-09-12T20:10:02.386 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:10:02.387 DEBUG:boto:Data: 2022-09-12T20:10:02.387 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.388 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.389 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.389 DEBUG:boto:Params: {} 2022-09-12T20:10:02.390 DEBUG:boto:Token: None 2022-09-12T20:10:02.391 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj55 2022-09-12T20:10:02.392 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ljSxcuyNkdAFXuV+5ocK0HVcw9U= 2022-09-12T20:10:02.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ljSxcuyNkdAFXuV+5ocK0HVcw9U=', 'Content-Length': '0'} 2022-09-12T20:10:02.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b30f2be885f5cc2-00631f921a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.397 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:10:02.397 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:10:02.398 DEBUG:boto:Method: GET 2022-09-12T20:10:02.399 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:10:02.399 DEBUG:boto:Data: 2022-09-12T20:10:02.400 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.401 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.401 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.402 DEBUG:boto:Params: {} 2022-09-12T20:10:02.403 DEBUG:boto:Token: None 2022-09-12T20:10:02.404 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj55 2022-09-12T20:10:02.404 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ljSxcuyNkdAFXuV+5ocK0HVcw9U= 2022-09-12T20:10:02.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ljSxcuyNkdAFXuV+5ocK0HVcw9U=', 'Content-Length': '0'} 2022-09-12T20:10:02.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000413064151b34f688-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.408 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:02.409 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:02.410 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:10:02.410 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:10:02.411 DEBUG:boto:Method: GET 2022-09-12T20:10:02.412 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:10:02.413 DEBUG:boto:Data: 2022-09-12T20:10:02.413 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.414 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.415 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.416 DEBUG:boto:Params: {} 2022-09-12T20:10:02.416 DEBUG:boto:Token: None 2022-09-12T20:10:02.417 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/ 2022-09-12T20:10:02.418 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ= 2022-09-12T20:10:02.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ=', 'Content-Length': '0'} 2022-09-12T20:10:02.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a93c07c12354bb91-00631f921a-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.423 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:02.424 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:02.424 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:02.425 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:10:02.425 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:10:02.426 DEBUG:boto:Method: GET 2022-09-12T20:10:02.426 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:10:02.427 DEBUG:boto:Data: 2022-09-12T20:10:02.427 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.428 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.429 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.429 DEBUG:boto:Params: {} 2022-09-12T20:10:02.430 DEBUG:boto:Token: None 2022-09-12T20:10:02.430 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/ 2022-09-12T20:10:02.431 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ= 2022-09-12T20:10:02.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ=', 'Content-Length': '0'} 2022-09-12T20:10:02.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bd672d1a8c0c009-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.436 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:02.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:10:02.438 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:10:02.438 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:10:02.439 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:10:02.440 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:10:02.440 DEBUG:boto:Method: GET 2022-09-12T20:10:02.441 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:10:02.441 DEBUG:boto:Data: 2022-09-12T20:10:02.442 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.442 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.443 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.444 DEBUG:boto:Params: {} 2022-09-12T20:10:02.444 DEBUG:boto:Token: None 2022-09-12T20:10:02.445 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:10:02.445 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OiY/wKXhGdioKXJiYwACqY/Ie3s= 2022-09-12T20:10:02.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OiY/wKXhGdioKXJiYwACqY/Ie3s=', 'Content-Length': '0'} 2022-09-12T20:10:02.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054f3d77cd5c29ede-00631f921a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.449 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:10:02.450 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:10:02.450 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:10:02.451 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:10:02.452 DEBUG:boto:Method: GET 2022-09-12T20:10:02.452 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:10:02.453 DEBUG:boto:Data: 2022-09-12T20:10:02.453 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.454 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.454 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.455 DEBUG:boto:Params: {} 2022-09-12T20:10:02.456 DEBUG:boto:Token: None 2022-09-12T20:10:02.456 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:10:02.457 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OiY/wKXhGdioKXJiYwACqY/Ie3s= 2022-09-12T20:10:02.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OiY/wKXhGdioKXJiYwACqY/Ie3s=', 'Content-Length': '0'} 2022-09-12T20:10:02.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd4fe09dd90b3b2b-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.502 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:10:02.503 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:10:02.503 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:10:02.504 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:10:02.504 DEBUG:boto:Method: HEAD 2022-09-12T20:10:02.505 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:10:02.505 DEBUG:boto:Data: 2022-09-12T20:10:02.506 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.507 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.507 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.508 DEBUG:boto:Params: {} 2022-09-12T20:10:02.508 DEBUG:boto:Token: None 2022-09-12T20:10:02.509 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:10:02.509 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WcKp9029FVFo9b7lJ3wlOeHsWuY= 2022-09-12T20:10:02.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WcKp9029FVFo9b7lJ3wlOeHsWuY=', 'Content-Length': '0'} 2022-09-12T20:10:02.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095ed1885aa180e10-00631f921a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.513 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:10:02.513 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:10:02.514 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:10:02.514 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:10:02.515 DEBUG:boto:Method: HEAD 2022-09-12T20:10:02.515 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:10:02.516 DEBUG:boto:Data: 2022-09-12T20:10:02.516 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.517 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.518 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.518 DEBUG:boto:Params: {} 2022-09-12T20:10:02.519 DEBUG:boto:Token: None 2022-09-12T20:10:02.519 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:10:02.520 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WcKp9029FVFo9b7lJ3wlOeHsWuY= 2022-09-12T20:10:02.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WcKp9029FVFo9b7lJ3wlOeHsWuY=', 'Content-Length': '0'} 2022-09-12T20:10:02.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1e2bffcf3cdc061-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:10:02.524 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:10:02.524 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:10:02.525 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:10:02.525 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:10:02.526 DEBUG:boto:Method: GET 2022-09-12T20:10:02.527 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:10:02.527 DEBUG:boto:Data: 2022-09-12T20:10:02.528 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.529 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.529 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.530 DEBUG:boto:Params: {} 2022-09-12T20:10:02.531 DEBUG:boto:Token: None 2022-09-12T20:10:02.532 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:10:02.532 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OiY/wKXhGdioKXJiYwACqY/Ie3s= 2022-09-12T20:10:02.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OiY/wKXhGdioKXJiYwACqY/Ie3s=', 'Content-Length': '0'} 2022-09-12T20:10:02.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f09309ddc8913d13-00631f921a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.536 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:10:02.536 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:10:02.537 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:10:02.537 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:10:02.538 DEBUG:boto:Method: GET 2022-09-12T20:10:02.538 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:10:02.539 DEBUG:boto:Data: 2022-09-12T20:10:02.539 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.540 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.541 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.541 DEBUG:boto:Params: {} 2022-09-12T20:10:02.542 DEBUG:boto:Token: None 2022-09-12T20:10:02.542 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:10:02.543 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OiY/wKXhGdioKXJiYwACqY/Ie3s= 2022-09-12T20:10:02.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OiY/wKXhGdioKXJiYwACqY/Ie3s=', 'Content-Length': '0'} 2022-09-12T20:10:02.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006822501d28ebb270-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.547 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:02.547 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:02.548 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:10:02.548 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:10:02.549 DEBUG:boto:Method: GET 2022-09-12T20:10:02.549 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:10:02.550 DEBUG:boto:Data: 2022-09-12T20:10:02.551 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.551 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.552 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.552 DEBUG:boto:Params: {} 2022-09-12T20:10:02.553 DEBUG:boto:Token: None 2022-09-12T20:10:02.553 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/ 2022-09-12T20:10:02.554 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ= 2022-09-12T20:10:02.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ=', 'Content-Length': '0'} 2022-09-12T20:10:02.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098b026c2c6c36587-00631f921a-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.598 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:02.599 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:02.599 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:02.600 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:10:02.601 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:10:02.601 DEBUG:boto:Method: GET 2022-09-12T20:10:02.602 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:10:02.603 DEBUG:boto:Data: 2022-09-12T20:10:02.603 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.604 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.604 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.605 DEBUG:boto:Params: {} 2022-09-12T20:10:02.605 DEBUG:boto:Token: None 2022-09-12T20:10:02.606 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/ 2022-09-12T20:10:02.607 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ= 2022-09-12T20:10:02.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ=', 'Content-Length': '0'} 2022-09-12T20:10:02.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e471d02f4671f70b-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.612 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:02.613 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:10:02.613 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:10:02.614 DEBUG:boto:Method: HEAD 2022-09-12T20:10:02.614 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:10:02.615 DEBUG:boto:Data: 2022-09-12T20:10:02.616 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.616 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.617 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.617 DEBUG:boto:Params: {} 2022-09-12T20:10:02.618 DEBUG:boto:Token: None 2022-09-12T20:10:02.618 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj57 2022-09-12T20:10:02.619 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HdIRXXiTLK1TPD3T/XC/CRRXq/I= 2022-09-12T20:10:02.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HdIRXXiTLK1TPD3T/XC/CRRXq/I=', 'Content-Length': '0'} 2022-09-12T20:10:02.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034ecfd62122b1de9-00631f921a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.622 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:10:02.623 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:10:02.623 DEBUG:boto:Method: HEAD 2022-09-12T20:10:02.624 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:10:02.624 DEBUG:boto:Data: 2022-09-12T20:10:02.625 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.625 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.626 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.626 DEBUG:boto:Params: {} 2022-09-12T20:10:02.627 DEBUG:boto:Token: None 2022-09-12T20:10:02.628 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj57 2022-09-12T20:10:02.628 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HdIRXXiTLK1TPD3T/XC/CRRXq/I= 2022-09-12T20:10:02.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HdIRXXiTLK1TPD3T/XC/CRRXq/I=', 'Content-Length': '0'} 2022-09-12T20:10:02.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045bed7125139af56-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:10:02.632 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:10:02.632 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:10:02.633 DEBUG:boto:Method: GET 2022-09-12T20:10:02.634 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:10:02.634 DEBUG:boto:Data: 2022-09-12T20:10:02.635 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.635 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.636 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.636 DEBUG:boto:Params: {} 2022-09-12T20:10:02.637 DEBUG:boto:Token: None 2022-09-12T20:10:02.638 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj57 2022-09-12T20:10:02.638 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:awX7HXfBQj11++rKXsOom7ctcKQ= 2022-09-12T20:10:02.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:awX7HXfBQj11++rKXsOom7ctcKQ=', 'Content-Length': '0'} 2022-09-12T20:10:02.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e279495187aeb4d-00631f921a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.642 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:10:02.643 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:10:02.644 DEBUG:boto:Method: GET 2022-09-12T20:10:02.645 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:10:02.646 DEBUG:boto:Data: 2022-09-12T20:10:02.647 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.647 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.648 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.648 DEBUG:boto:Params: {} 2022-09-12T20:10:02.649 DEBUG:boto:Token: None 2022-09-12T20:10:02.650 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj57 2022-09-12T20:10:02.650 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:awX7HXfBQj11++rKXsOom7ctcKQ= 2022-09-12T20:10:02.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:awX7HXfBQj11++rKXsOom7ctcKQ=', 'Content-Length': '0'} 2022-09-12T20:10:02.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049de9374bfcc8f8a-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.694 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:02.695 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:02.695 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:10:02.696 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:10:02.696 DEBUG:boto:Method: GET 2022-09-12T20:10:02.697 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:10:02.697 DEBUG:boto:Data: 2022-09-12T20:10:02.698 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.698 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.699 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.699 DEBUG:boto:Params: {} 2022-09-12T20:10:02.700 DEBUG:boto:Token: None 2022-09-12T20:10:02.701 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/ 2022-09-12T20:10:02.701 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ= 2022-09-12T20:10:02.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ=', 'Content-Length': '0'} 2022-09-12T20:10:02.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bcdda020010bace-00631f921a-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.705 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:02.706 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:02.707 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:02.707 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:10:02.708 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:10:02.708 DEBUG:boto:Method: GET 2022-09-12T20:10:02.709 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:10:02.709 DEBUG:boto:Data: 2022-09-12T20:10:02.710 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.710 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.711 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.712 DEBUG:boto:Params: {} 2022-09-12T20:10:02.712 DEBUG:boto:Token: None 2022-09-12T20:10:02.713 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/ 2022-09-12T20:10:02.713 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ= 2022-09-12T20:10:02.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ=', 'Content-Length': '0'} 2022-09-12T20:10:02.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000173b6ec520dd2d6c-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.719 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:02.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:10:02.720 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:10:02.721 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:10:02.721 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:10:02.722 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:10:02.722 DEBUG:boto:Method: GET 2022-09-12T20:10:02.723 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:10:02.723 DEBUG:boto:Data: 2022-09-12T20:10:02.724 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.724 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.725 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.726 DEBUG:boto:Params: {} 2022-09-12T20:10:02.726 DEBUG:boto:Token: None 2022-09-12T20:10:02.727 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:10:02.727 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R5Rc4Z/VzralCyt3oO+QtjOcK/A= 2022-09-12T20:10:02.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R5Rc4Z/VzralCyt3oO+QtjOcK/A=', 'Content-Length': '0'} 2022-09-12T20:10:02.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa7529ccba4476a5-00631f921a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.731 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:10:02.732 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:10:02.732 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:10:02.733 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:10:02.733 DEBUG:boto:Method: GET 2022-09-12T20:10:02.734 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:10:02.734 DEBUG:boto:Data: 2022-09-12T20:10:02.735 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.736 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.737 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.738 DEBUG:boto:Params: {} 2022-09-12T20:10:02.739 DEBUG:boto:Token: None 2022-09-12T20:10:02.739 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:10:02.740 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R5Rc4Z/VzralCyt3oO+QtjOcK/A= 2022-09-12T20:10:02.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R5Rc4Z/VzralCyt3oO+QtjOcK/A=', 'Content-Length': '0'} 2022-09-12T20:10:02.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064d3de4b900093fd-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.743 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:10:02.744 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:10:02.745 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:10:02.745 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:10:02.746 DEBUG:boto:Method: HEAD 2022-09-12T20:10:02.746 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:10:02.747 DEBUG:boto:Data: 2022-09-12T20:10:02.748 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.748 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.749 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.749 DEBUG:boto:Params: {} 2022-09-12T20:10:02.750 DEBUG:boto:Token: None 2022-09-12T20:10:02.751 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:10:02.751 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DAs0TkKUTwwjv30QO47Kg8ekvTs= 2022-09-12T20:10:02.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DAs0TkKUTwwjv30QO47Kg8ekvTs=', 'Content-Length': '0'} 2022-09-12T20:10:02.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023681a68019b81ad-00631f921a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.755 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:10:02.756 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:10:02.756 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:10:02.757 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:10:02.757 DEBUG:boto:Method: HEAD 2022-09-12T20:10:02.758 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:10:02.759 DEBUG:boto:Data: 2022-09-12T20:10:02.759 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.760 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.760 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.761 DEBUG:boto:Params: {} 2022-09-12T20:10:02.762 DEBUG:boto:Token: None 2022-09-12T20:10:02.762 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:10:02.763 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DAs0TkKUTwwjv30QO47Kg8ekvTs= 2022-09-12T20:10:02.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DAs0TkKUTwwjv30QO47Kg8ekvTs=', 'Content-Length': '0'} 2022-09-12T20:10:02.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000814827f8cb52077b-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:10:02.767 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:10:02.768 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:10:02.768 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:10:02.769 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:10:02.769 DEBUG:boto:Method: GET 2022-09-12T20:10:02.770 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:10:02.770 DEBUG:boto:Data: 2022-09-12T20:10:02.771 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.771 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.772 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.773 DEBUG:boto:Params: {} 2022-09-12T20:10:02.773 DEBUG:boto:Token: None 2022-09-12T20:10:02.774 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:10:02.775 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R5Rc4Z/VzralCyt3oO+QtjOcK/A= 2022-09-12T20:10:02.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R5Rc4Z/VzralCyt3oO+QtjOcK/A=', 'Content-Length': '0'} 2022-09-12T20:10:02.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007005d2635c597669-00631f921a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.818 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:10:02.819 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:10:02.819 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:10:02.820 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:10:02.821 DEBUG:boto:Method: GET 2022-09-12T20:10:02.821 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:10:02.822 DEBUG:boto:Data: 2022-09-12T20:10:02.822 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.823 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.823 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.824 DEBUG:boto:Params: {} 2022-09-12T20:10:02.825 DEBUG:boto:Token: None 2022-09-12T20:10:02.826 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:10:02.826 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R5Rc4Z/VzralCyt3oO+QtjOcK/A= 2022-09-12T20:10:02.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R5Rc4Z/VzralCyt3oO+QtjOcK/A=', 'Content-Length': '0'} 2022-09-12T20:10:02.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000340f4e05e15a586e-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.831 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:02.831 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:02.832 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:10:02.832 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:10:02.833 DEBUG:boto:Method: GET 2022-09-12T20:10:02.833 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:10:02.834 DEBUG:boto:Data: 2022-09-12T20:10:02.834 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.835 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.835 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.836 DEBUG:boto:Params: {} 2022-09-12T20:10:02.837 DEBUG:boto:Token: None 2022-09-12T20:10:02.837 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/ 2022-09-12T20:10:02.838 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ= 2022-09-12T20:10:02.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ=', 'Content-Length': '0'} 2022-09-12T20:10:02.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002db1e8593f5366e4-00631f921a-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.842 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:02.843 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:02.844 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:02.844 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:10:02.845 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:10:02.845 DEBUG:boto:Method: GET 2022-09-12T20:10:02.846 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:10:02.846 DEBUG:boto:Data: 2022-09-12T20:10:02.847 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.847 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.848 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.849 DEBUG:boto:Params: {} 2022-09-12T20:10:02.849 DEBUG:boto:Token: None 2022-09-12T20:10:02.850 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/ 2022-09-12T20:10:02.850 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ= 2022-09-12T20:10:02.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ=', 'Content-Length': '0'} 2022-09-12T20:10:02.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dfbdbd40c4db685-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.856 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:02.857 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:10:02.857 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:10:02.858 DEBUG:boto:Method: HEAD 2022-09-12T20:10:02.858 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:10:02.859 DEBUG:boto:Data: 2022-09-12T20:10:02.859 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.860 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.860 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.861 DEBUG:boto:Params: {} 2022-09-12T20:10:02.861 DEBUG:boto:Token: None 2022-09-12T20:10:02.862 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj68 2022-09-12T20:10:02.862 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DTG/CEu5cuSMbWinbsi2S+X2jrQ= 2022-09-12T20:10:02.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DTG/CEu5cuSMbWinbsi2S+X2jrQ=', 'Content-Length': '0'} 2022-09-12T20:10:02.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046c4a8db97427122-00631f921a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.866 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:10:02.866 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:10:02.867 DEBUG:boto:Method: HEAD 2022-09-12T20:10:02.868 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:10:02.868 DEBUG:boto:Data: 2022-09-12T20:10:02.869 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.869 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.870 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.870 DEBUG:boto:Params: {} 2022-09-12T20:10:02.871 DEBUG:boto:Token: None 2022-09-12T20:10:02.871 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj68 2022-09-12T20:10:02.872 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DTG/CEu5cuSMbWinbsi2S+X2jrQ= 2022-09-12T20:10:02.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DTG/CEu5cuSMbWinbsi2S+X2jrQ=', 'Content-Length': '0'} 2022-09-12T20:10:02.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f9c92aac2514f19-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:10:02.876 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:10:02.876 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:10:02.877 DEBUG:boto:Method: GET 2022-09-12T20:10:02.877 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:10:02.878 DEBUG:boto:Data: 2022-09-12T20:10:02.879 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.879 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.880 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.880 DEBUG:boto:Params: {} 2022-09-12T20:10:02.881 DEBUG:boto:Token: None 2022-09-12T20:10:02.881 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj68 2022-09-12T20:10:02.882 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:i4AJIKuva673jIj+NJAHyha0d2M= 2022-09-12T20:10:02.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:i4AJIKuva673jIj+NJAHyha0d2M=', 'Content-Length': '0'} 2022-09-12T20:10:02.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7164b3934f3a8a5-00631f921a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.886 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:10:02.886 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:10:02.887 DEBUG:boto:Method: GET 2022-09-12T20:10:02.888 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:10:02.888 DEBUG:boto:Data: 2022-09-12T20:10:02.889 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.889 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.890 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.890 DEBUG:boto:Params: {} 2022-09-12T20:10:02.891 DEBUG:boto:Token: None 2022-09-12T20:10:02.891 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj68 2022-09-12T20:10:02.892 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:i4AJIKuva673jIj+NJAHyha0d2M= 2022-09-12T20:10:02.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:i4AJIKuva673jIj+NJAHyha0d2M=', 'Content-Length': '0'} 2022-09-12T20:10:02.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000969f0a8eec62a060-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.896 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:02.896 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:02.897 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:10:02.897 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:10:02.898 DEBUG:boto:Method: GET 2022-09-12T20:10:02.898 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:10:02.899 DEBUG:boto:Data: 2022-09-12T20:10:02.900 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.900 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.901 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.901 DEBUG:boto:Params: {} 2022-09-12T20:10:02.902 DEBUG:boto:Token: None 2022-09-12T20:10:02.902 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/ 2022-09-12T20:10:02.903 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ= 2022-09-12T20:10:02.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ=', 'Content-Length': '0'} 2022-09-12T20:10:02.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029609c0d65c8b224-00631f921a-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.907 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:02.908 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:02.908 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:02.909 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:10:02.909 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:10:02.910 DEBUG:boto:Method: GET 2022-09-12T20:10:02.911 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:10:02.911 DEBUG:boto:Data: 2022-09-12T20:10:02.912 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.912 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.913 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.913 DEBUG:boto:Params: {} 2022-09-12T20:10:02.914 DEBUG:boto:Token: None 2022-09-12T20:10:02.914 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/ 2022-09-12T20:10:02.915 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ= 2022-09-12T20:10:02.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nYlhOPLki/JwZeq2PxtRubCjiDQ=', 'Content-Length': '0'} 2022-09-12T20:10:02.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002755e57dd3443a6b-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.962 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:02.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:10:02.964 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:10:02.964 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:10:02.965 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:10:02.965 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:10:02.966 DEBUG:boto:Method: GET 2022-09-12T20:10:02.966 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:10:02.967 DEBUG:boto:Data: 2022-09-12T20:10:02.968 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.968 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.969 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.969 DEBUG:boto:Params: {} 2022-09-12T20:10:02.970 DEBUG:boto:Token: None 2022-09-12T20:10:02.970 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:10:02.971 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s69uVOCmBOIjtLOK3TGO6Vr0N+o= 2022-09-12T20:10:02.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s69uVOCmBOIjtLOK3TGO6Vr0N+o=', 'Content-Length': '0'} 2022-09-12T20:10:02.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0176eb5e46fe46c-00631f921a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.975 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:10:02.976 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:10:02.976 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:10:02.977 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:10:02.977 DEBUG:boto:Method: GET 2022-09-12T20:10:02.978 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:10:02.978 DEBUG:boto:Data: 2022-09-12T20:10:02.979 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.979 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:02.980 DEBUG:boto:Port: 8000 2022-09-12T20:10:02.981 DEBUG:boto:Params: {} 2022-09-12T20:10:02.981 DEBUG:boto:Token: None 2022-09-12T20:10:02.982 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:10:02.982 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s69uVOCmBOIjtLOK3TGO6Vr0N+o= 2022-09-12T20:10:02.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s69uVOCmBOIjtLOK3TGO6Vr0N+o=', 'Content-Length': '0'} 2022-09-12T20:10:02.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000442a9a9a816f0cad-00631f921a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.986 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:10:02.987 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:10:02.987 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:10:02.988 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:10:02.988 DEBUG:boto:Method: HEAD 2022-09-12T20:10:02.989 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:10:02.989 DEBUG:boto:Data: 2022-09-12T20:10:02.990 DEBUG:boto:Headers: {} 2022-09-12T20:10:02.990 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:02.991 DEBUG:boto:Port: 8001 2022-09-12T20:10:02.991 DEBUG:boto:Params: {} 2022-09-12T20:10:02.992 DEBUG:boto:Token: None 2022-09-12T20:10:02.993 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:02 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:10:02.993 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MvJbqYFssxo6H1ml44sD/YD7nlE= 2022-09-12T20:10:02.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MvJbqYFssxo6H1ml44sD/YD7nlE=', 'Content-Length': '0'} 2022-09-12T20:10:02.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b77e667738fb8e73-00631f921a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:02.997 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:10:02.997 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:10:02.998 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:10:02.998 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:10:02.999 DEBUG:boto:Method: HEAD 2022-09-12T20:10:02.999 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:10:03.000 DEBUG:boto:Data: 2022-09-12T20:10:03.000 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.001 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.001 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.002 DEBUG:boto:Params: {} 2022-09-12T20:10:03.002 DEBUG:boto:Token: None 2022-09-12T20:10:03.003 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:10:03.003 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:va+Q75KRK3c+2s17roGeYSthscc= 2022-09-12T20:10:03.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:va+Q75KRK3c+2s17roGeYSthscc=', 'Content-Length': '0'} 2022-09-12T20:10:03.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016a9cd567128f766-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:10:03.007 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:10:03.008 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:10:03.009 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:10:03.009 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:10:03.010 DEBUG:boto:Method: GET 2022-09-12T20:10:03.010 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:10:03.011 DEBUG:boto:Data: 2022-09-12T20:10:03.011 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.012 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.012 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.013 DEBUG:boto:Params: {} 2022-09-12T20:10:03.013 DEBUG:boto:Token: None 2022-09-12T20:10:03.014 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:10:03.015 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T9+EkXC6aqff5auSEvNxQEJ+EzM= 2022-09-12T20:10:03.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T9+EkXC6aqff5auSEvNxQEJ+EzM=', 'Content-Length': '0'} 2022-09-12T20:10:03.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034fe45dfef428c67-00631f921b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.018 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:10:03.019 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:10:03.020 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:10:03.020 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:10:03.021 DEBUG:boto:Method: GET 2022-09-12T20:10:03.021 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:10:03.022 DEBUG:boto:Data: 2022-09-12T20:10:03.022 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.023 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.023 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.024 DEBUG:boto:Params: {} 2022-09-12T20:10:03.025 DEBUG:boto:Token: None 2022-09-12T20:10:03.025 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:10:03.026 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T9+EkXC6aqff5auSEvNxQEJ+EzM= 2022-09-12T20:10:03.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T9+EkXC6aqff5auSEvNxQEJ+EzM=', 'Content-Length': '0'} 2022-09-12T20:10:03.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001552c4b2049448f9-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.029 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:03.030 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:03.031 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:10:03.031 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:10:03.032 DEBUG:boto:Method: GET 2022-09-12T20:10:03.032 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:10:03.033 DEBUG:boto:Data: 2022-09-12T20:10:03.033 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.034 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.034 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.035 DEBUG:boto:Params: {} 2022-09-12T20:10:03.035 DEBUG:boto:Token: None 2022-09-12T20:10:03.036 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/ 2022-09-12T20:10:03.037 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8= 2022-09-12T20:10:03.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8=', 'Content-Length': '0'} 2022-09-12T20:10:03.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003de7c6927f9d20a4-00631f921b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.041 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:03.042 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:03.042 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:03.043 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:10:03.043 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:10:03.044 DEBUG:boto:Method: GET 2022-09-12T20:10:03.044 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:10:03.045 DEBUG:boto:Data: 2022-09-12T20:10:03.045 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.046 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.046 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.047 DEBUG:boto:Params: {} 2022-09-12T20:10:03.048 DEBUG:boto:Token: None 2022-09-12T20:10:03.048 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/ 2022-09-12T20:10:03.049 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8= 2022-09-12T20:10:03.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8=', 'Content-Length': '0'} 2022-09-12T20:10:03.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000669ebf0c65a8509e-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.054 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:03.054 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:10:03.055 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:10:03.055 DEBUG:boto:Method: HEAD 2022-09-12T20:10:03.056 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:10:03.057 DEBUG:boto:Data: 2022-09-12T20:10:03.057 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.058 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.058 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.059 DEBUG:boto:Params: {} 2022-09-12T20:10:03.059 DEBUG:boto:Token: None 2022-09-12T20:10:03.060 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj69 2022-09-12T20:10:03.060 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wVJJDRhdornZO/bshAsV4dQHrqk= 2022-09-12T20:10:03.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wVJJDRhdornZO/bshAsV4dQHrqk=', 'Content-Length': '0'} 2022-09-12T20:10:03.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e9976731ffa1b65-00631f921b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.064 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:10:03.064 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:10:03.065 DEBUG:boto:Method: HEAD 2022-09-12T20:10:03.066 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:10:03.066 DEBUG:boto:Data: 2022-09-12T20:10:03.067 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.067 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.068 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.068 DEBUG:boto:Params: {} 2022-09-12T20:10:03.069 DEBUG:boto:Token: None 2022-09-12T20:10:03.069 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj69 2022-09-12T20:10:03.070 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wVJJDRhdornZO/bshAsV4dQHrqk= 2022-09-12T20:10:03.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wVJJDRhdornZO/bshAsV4dQHrqk=', 'Content-Length': '0'} 2022-09-12T20:10:03.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000833bb1d9fb46492b-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:10:03.074 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:10:03.074 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:10:03.075 DEBUG:boto:Method: GET 2022-09-12T20:10:03.076 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:10:03.076 DEBUG:boto:Data: 2022-09-12T20:10:03.077 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.077 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.078 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.078 DEBUG:boto:Params: {} 2022-09-12T20:10:03.079 DEBUG:boto:Token: None 2022-09-12T20:10:03.080 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj69 2022-09-12T20:10:03.080 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G7VzLVeFoQRJNPIOOlace0nMos4= 2022-09-12T20:10:03.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G7VzLVeFoQRJNPIOOlace0nMos4=', 'Content-Length': '0'} 2022-09-12T20:10:03.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5368e738cadb313-00631f921b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.084 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:10:03.084 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:10:03.085 DEBUG:boto:Method: GET 2022-09-12T20:10:03.085 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:10:03.086 DEBUG:boto:Data: 2022-09-12T20:10:03.087 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.087 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.088 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.088 DEBUG:boto:Params: {} 2022-09-12T20:10:03.089 DEBUG:boto:Token: None 2022-09-12T20:10:03.089 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj69 2022-09-12T20:10:03.090 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G7VzLVeFoQRJNPIOOlace0nMos4= 2022-09-12T20:10:03.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G7VzLVeFoQRJNPIOOlace0nMos4=', 'Content-Length': '0'} 2022-09-12T20:10:03.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abb0e9e528d6887e-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.093 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:03.094 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:03.094 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:10:03.095 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:10:03.096 DEBUG:boto:Method: GET 2022-09-12T20:10:03.096 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:10:03.097 DEBUG:boto:Data: 2022-09-12T20:10:03.097 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.098 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.098 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.099 DEBUG:boto:Params: {} 2022-09-12T20:10:03.099 DEBUG:boto:Token: None 2022-09-12T20:10:03.100 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/ 2022-09-12T20:10:03.101 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8= 2022-09-12T20:10:03.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8=', 'Content-Length': '0'} 2022-09-12T20:10:03.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f131d056361ee44-00631f921b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.146 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:03.147 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:03.147 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:03.148 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:10:03.149 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:10:03.149 DEBUG:boto:Method: GET 2022-09-12T20:10:03.150 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:10:03.150 DEBUG:boto:Data: 2022-09-12T20:10:03.151 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.151 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.152 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.152 DEBUG:boto:Params: {} 2022-09-12T20:10:03.153 DEBUG:boto:Token: None 2022-09-12T20:10:03.154 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/ 2022-09-12T20:10:03.154 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8= 2022-09-12T20:10:03.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8=', 'Content-Length': '0'} 2022-09-12T20:10:03.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eefad5336f9ec2a7-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.159 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:03.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:10:03.160 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:10:03.161 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:10:03.162 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:10:03.162 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:10:03.163 DEBUG:boto:Method: GET 2022-09-12T20:10:03.163 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:10:03.164 DEBUG:boto:Data: 2022-09-12T20:10:03.164 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.165 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.166 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.166 DEBUG:boto:Params: {} 2022-09-12T20:10:03.167 DEBUG:boto:Token: None 2022-09-12T20:10:03.167 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:10:03.168 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dLinLFDVGSHGj9tT3jCRErya5f4= 2022-09-12T20:10:03.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dLinLFDVGSHGj9tT3jCRErya5f4=', 'Content-Length': '0'} 2022-09-12T20:10:03.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be6b860590aca03f-00631f921b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.171 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:10:03.172 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:10:03.172 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:10:03.173 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:10:03.174 DEBUG:boto:Method: GET 2022-09-12T20:10:03.174 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:10:03.175 DEBUG:boto:Data: 2022-09-12T20:10:03.175 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.176 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.176 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.177 DEBUG:boto:Params: {} 2022-09-12T20:10:03.177 DEBUG:boto:Token: None 2022-09-12T20:10:03.178 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:10:03.179 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dLinLFDVGSHGj9tT3jCRErya5f4= 2022-09-12T20:10:03.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dLinLFDVGSHGj9tT3jCRErya5f4=', 'Content-Length': '0'} 2022-09-12T20:10:03.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d4fa0d798541ab0-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.222 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:10:03.223 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:10:03.223 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:10:03.224 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:10:03.224 DEBUG:boto:Method: HEAD 2022-09-12T20:10:03.225 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:10:03.225 DEBUG:boto:Data: 2022-09-12T20:10:03.226 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.226 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.227 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.227 DEBUG:boto:Params: {} 2022-09-12T20:10:03.228 DEBUG:boto:Token: None 2022-09-12T20:10:03.229 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:10:03.229 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZRp1MYLewoqGmKFfLM/xVRFBnpc= 2022-09-12T20:10:03.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZRp1MYLewoqGmKFfLM/xVRFBnpc=', 'Content-Length': '0'} 2022-09-12T20:10:03.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088e31f72cac1068c-00631f921b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.233 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:10:03.233 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:10:03.234 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:10:03.234 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:10:03.235 DEBUG:boto:Method: HEAD 2022-09-12T20:10:03.235 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:10:03.236 DEBUG:boto:Data: 2022-09-12T20:10:03.236 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.237 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.237 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.238 DEBUG:boto:Params: {} 2022-09-12T20:10:03.239 DEBUG:boto:Token: None 2022-09-12T20:10:03.239 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:10:03.240 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZRp1MYLewoqGmKFfLM/xVRFBnpc= 2022-09-12T20:10:03.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZRp1MYLewoqGmKFfLM/xVRFBnpc=', 'Content-Length': '0'} 2022-09-12T20:10:03.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d2ff45139e41cb6-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:10:03.244 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:10:03.245 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:10:03.245 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:10:03.246 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:10:03.246 DEBUG:boto:Method: GET 2022-09-12T20:10:03.247 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:10:03.247 DEBUG:boto:Data: 2022-09-12T20:10:03.248 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.248 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.249 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.249 DEBUG:boto:Params: {} 2022-09-12T20:10:03.250 DEBUG:boto:Token: None 2022-09-12T20:10:03.251 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:10:03.251 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dLinLFDVGSHGj9tT3jCRErya5f4= 2022-09-12T20:10:03.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dLinLFDVGSHGj9tT3jCRErya5f4=', 'Content-Length': '0'} 2022-09-12T20:10:03.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c0cbf9a01bd8bb6-00631f921b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.255 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:10:03.256 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:10:03.256 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:10:03.257 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:10:03.257 DEBUG:boto:Method: GET 2022-09-12T20:10:03.258 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:10:03.258 DEBUG:boto:Data: 2022-09-12T20:10:03.259 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.259 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.260 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.260 DEBUG:boto:Params: {} 2022-09-12T20:10:03.261 DEBUG:boto:Token: None 2022-09-12T20:10:03.262 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:10:03.262 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dLinLFDVGSHGj9tT3jCRErya5f4= 2022-09-12T20:10:03.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dLinLFDVGSHGj9tT3jCRErya5f4=', 'Content-Length': '0'} 2022-09-12T20:10:03.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003097a28ba40313f6-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.266 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:03.267 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:03.267 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:10:03.268 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:10:03.268 DEBUG:boto:Method: GET 2022-09-12T20:10:03.269 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:10:03.269 DEBUG:boto:Data: 2022-09-12T20:10:03.270 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.270 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.271 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.272 DEBUG:boto:Params: {} 2022-09-12T20:10:03.272 DEBUG:boto:Token: None 2022-09-12T20:10:03.273 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/ 2022-09-12T20:10:03.273 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8= 2022-09-12T20:10:03.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8=', 'Content-Length': '0'} 2022-09-12T20:10:03.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037d1de0c874b51aa-00631f921b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.278 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:03.279 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:03.279 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:03.280 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:10:03.281 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:10:03.281 DEBUG:boto:Method: GET 2022-09-12T20:10:03.282 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:10:03.282 DEBUG:boto:Data: 2022-09-12T20:10:03.283 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.283 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.284 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.284 DEBUG:boto:Params: {} 2022-09-12T20:10:03.285 DEBUG:boto:Token: None 2022-09-12T20:10:03.286 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/ 2022-09-12T20:10:03.286 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8= 2022-09-12T20:10:03.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8=', 'Content-Length': '0'} 2022-09-12T20:10:03.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f352ec29c4e6390-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.291 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:03.292 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:10:03.293 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:10:03.293 DEBUG:boto:Method: HEAD 2022-09-12T20:10:03.294 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:10:03.294 DEBUG:boto:Data: 2022-09-12T20:10:03.295 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.295 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.296 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.296 DEBUG:boto:Params: {} 2022-09-12T20:10:03.297 DEBUG:boto:Token: None 2022-09-12T20:10:03.297 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj76 2022-09-12T20:10:03.298 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AttVVG9RkKO7r+O7yQO5imwqQtc= 2022-09-12T20:10:03.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AttVVG9RkKO7r+O7yQO5imwqQtc=', 'Content-Length': '0'} 2022-09-12T20:10:03.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f865126d85cd972-00631f921b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.301 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:10:03.302 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:10:03.303 DEBUG:boto:Method: HEAD 2022-09-12T20:10:03.303 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:10:03.304 DEBUG:boto:Data: 2022-09-12T20:10:03.304 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.305 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.305 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.306 DEBUG:boto:Params: {} 2022-09-12T20:10:03.306 DEBUG:boto:Token: None 2022-09-12T20:10:03.307 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj76 2022-09-12T20:10:03.307 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AttVVG9RkKO7r+O7yQO5imwqQtc= 2022-09-12T20:10:03.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AttVVG9RkKO7r+O7yQO5imwqQtc=', 'Content-Length': '0'} 2022-09-12T20:10:03.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7092f6dfbf0beea-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:10:03.311 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:10:03.312 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:10:03.312 DEBUG:boto:Method: GET 2022-09-12T20:10:03.313 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:10:03.314 DEBUG:boto:Data: 2022-09-12T20:10:03.314 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.315 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.315 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.316 DEBUG:boto:Params: {} 2022-09-12T20:10:03.316 DEBUG:boto:Token: None 2022-09-12T20:10:03.317 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj76 2022-09-12T20:10:03.318 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1Di6H9kkHK5G5MNuN7fVL3igokQ= 2022-09-12T20:10:03.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1Di6H9kkHK5G5MNuN7fVL3igokQ=', 'Content-Length': '0'} 2022-09-12T20:10:03.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b8676f3c3bfa99a-00631f921b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.322 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:10:03.322 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:10:03.323 DEBUG:boto:Method: GET 2022-09-12T20:10:03.323 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:10:03.324 DEBUG:boto:Data: 2022-09-12T20:10:03.324 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.325 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.325 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.326 DEBUG:boto:Params: {} 2022-09-12T20:10:03.327 DEBUG:boto:Token: None 2022-09-12T20:10:03.327 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj76 2022-09-12T20:10:03.328 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1Di6H9kkHK5G5MNuN7fVL3igokQ= 2022-09-12T20:10:03.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1Di6H9kkHK5G5MNuN7fVL3igokQ=', 'Content-Length': '0'} 2022-09-12T20:10:03.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9ec1cfd7119dbdb-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.374 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:03.375 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:03.375 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:10:03.376 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:10:03.376 DEBUG:boto:Method: GET 2022-09-12T20:10:03.377 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:10:03.377 DEBUG:boto:Data: 2022-09-12T20:10:03.378 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.379 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.379 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.380 DEBUG:boto:Params: {} 2022-09-12T20:10:03.380 DEBUG:boto:Token: None 2022-09-12T20:10:03.381 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/ 2022-09-12T20:10:03.381 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8= 2022-09-12T20:10:03.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8=', 'Content-Length': '0'} 2022-09-12T20:10:03.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8665bdb2e9bb0e1-00631f921b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.385 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:03.386 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:03.387 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:03.387 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:10:03.388 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:10:03.388 DEBUG:boto:Method: GET 2022-09-12T20:10:03.389 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:10:03.390 DEBUG:boto:Data: 2022-09-12T20:10:03.390 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.391 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.391 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.392 DEBUG:boto:Params: {} 2022-09-12T20:10:03.392 DEBUG:boto:Token: None 2022-09-12T20:10:03.393 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/ 2022-09-12T20:10:03.393 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8= 2022-09-12T20:10:03.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8=', 'Content-Length': '0'} 2022-09-12T20:10:03.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001aaf886089872a6a-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.399 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:03.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:10:03.400 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:10:03.401 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:10:03.401 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:10:03.402 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:10:03.402 DEBUG:boto:Method: GET 2022-09-12T20:10:03.403 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:10:03.403 DEBUG:boto:Data: 2022-09-12T20:10:03.404 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.405 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.405 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.406 DEBUG:boto:Params: {} 2022-09-12T20:10:03.406 DEBUG:boto:Token: None 2022-09-12T20:10:03.407 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:10:03.407 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FhJ3Gk8I3ZSGJTS5TZdE84zVKvA= 2022-09-12T20:10:03.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FhJ3Gk8I3ZSGJTS5TZdE84zVKvA=', 'Content-Length': '0'} 2022-09-12T20:10:03.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b97cb84c7075867-00631f921b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.411 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:10:03.411 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:10:03.412 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:10:03.412 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:10:03.413 DEBUG:boto:Method: GET 2022-09-12T20:10:03.413 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:10:03.414 DEBUG:boto:Data: 2022-09-12T20:10:03.414 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.415 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.415 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.416 DEBUG:boto:Params: {} 2022-09-12T20:10:03.417 DEBUG:boto:Token: None 2022-09-12T20:10:03.417 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:10:03.418 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FhJ3Gk8I3ZSGJTS5TZdE84zVKvA= 2022-09-12T20:10:03.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FhJ3Gk8I3ZSGJTS5TZdE84zVKvA=', 'Content-Length': '0'} 2022-09-12T20:10:03.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7532172e82943a8-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.421 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:10:03.422 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:10:03.422 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:10:03.423 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:10:03.423 DEBUG:boto:Method: HEAD 2022-09-12T20:10:03.424 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:10:03.424 DEBUG:boto:Data: 2022-09-12T20:10:03.425 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.425 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.426 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.426 DEBUG:boto:Params: {} 2022-09-12T20:10:03.427 DEBUG:boto:Token: None 2022-09-12T20:10:03.427 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:10:03.428 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Lv52ZlYAf05yCHvo/ez+uQJLUnw= 2022-09-12T20:10:03.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Lv52ZlYAf05yCHvo/ez+uQJLUnw=', 'Content-Length': '0'} 2022-09-12T20:10:03.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc8c61b4569770dc-00631f921b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.431 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:10:03.432 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:10:03.432 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:10:03.433 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:10:03.433 DEBUG:boto:Method: HEAD 2022-09-12T20:10:03.434 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:10:03.434 DEBUG:boto:Data: 2022-09-12T20:10:03.435 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.436 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.436 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.436 DEBUG:boto:Params: {} 2022-09-12T20:10:03.437 DEBUG:boto:Token: None 2022-09-12T20:10:03.438 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:10:03.438 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Lv52ZlYAf05yCHvo/ez+uQJLUnw= 2022-09-12T20:10:03.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Lv52ZlYAf05yCHvo/ez+uQJLUnw=', 'Content-Length': '0'} 2022-09-12T20:10:03.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcb65352595b22b3-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:10:03.442 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:10:03.443 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:10:03.443 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:10:03.444 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:10:03.444 DEBUG:boto:Method: GET 2022-09-12T20:10:03.445 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:10:03.445 DEBUG:boto:Data: 2022-09-12T20:10:03.446 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.447 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.447 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.448 DEBUG:boto:Params: {} 2022-09-12T20:10:03.448 DEBUG:boto:Token: None 2022-09-12T20:10:03.449 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:10:03.449 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FhJ3Gk8I3ZSGJTS5TZdE84zVKvA= 2022-09-12T20:10:03.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FhJ3Gk8I3ZSGJTS5TZdE84zVKvA=', 'Content-Length': '0'} 2022-09-12T20:10:03.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d15f3e1e425e748-00631f921b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.494 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:10:03.495 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:10:03.495 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:10:03.496 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:10:03.496 DEBUG:boto:Method: GET 2022-09-12T20:10:03.497 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:10:03.497 DEBUG:boto:Data: 2022-09-12T20:10:03.498 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.499 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.499 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.500 DEBUG:boto:Params: {} 2022-09-12T20:10:03.500 DEBUG:boto:Token: None 2022-09-12T20:10:03.501 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:10:03.502 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FhJ3Gk8I3ZSGJTS5TZdE84zVKvA= 2022-09-12T20:10:03.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FhJ3Gk8I3ZSGJTS5TZdE84zVKvA=', 'Content-Length': '0'} 2022-09-12T20:10:03.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ea950f4615ed4dd-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.505 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:03.506 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:03.506 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:10:03.507 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:10:03.507 DEBUG:boto:Method: GET 2022-09-12T20:10:03.508 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:10:03.509 DEBUG:boto:Data: 2022-09-12T20:10:03.509 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.510 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.510 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.511 DEBUG:boto:Params: {} 2022-09-12T20:10:03.511 DEBUG:boto:Token: None 2022-09-12T20:10:03.512 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/ 2022-09-12T20:10:03.513 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8= 2022-09-12T20:10:03.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8=', 'Content-Length': '0'} 2022-09-12T20:10:03.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be1d47216fdf8461-00631f921b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.517 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:03.517 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:03.518 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:03.519 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:10:03.519 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:10:03.520 DEBUG:boto:Method: GET 2022-09-12T20:10:03.520 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:10:03.521 DEBUG:boto:Data: 2022-09-12T20:10:03.521 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.522 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.522 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.523 DEBUG:boto:Params: {} 2022-09-12T20:10:03.523 DEBUG:boto:Token: None 2022-09-12T20:10:03.524 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/ 2022-09-12T20:10:03.524 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8= 2022-09-12T20:10:03.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8=', 'Content-Length': '0'} 2022-09-12T20:10:03.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059ec7468fdecc42f-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.529 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:03.530 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:10:03.531 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:10:03.531 DEBUG:boto:Method: HEAD 2022-09-12T20:10:03.532 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:10:03.532 DEBUG:boto:Data: 2022-09-12T20:10:03.533 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.533 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.534 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.534 DEBUG:boto:Params: {} 2022-09-12T20:10:03.535 DEBUG:boto:Token: None 2022-09-12T20:10:03.535 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj85 2022-09-12T20:10:03.536 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k+D4vxXHg6BXzvKuyGDjkiwT+jk= 2022-09-12T20:10:03.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k+D4vxXHg6BXzvKuyGDjkiwT+jk=', 'Content-Length': '0'} 2022-09-12T20:10:03.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1ed83b57911bd41-00631f921b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.539 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:10:03.539 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:10:03.540 DEBUG:boto:Method: HEAD 2022-09-12T20:10:03.541 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:10:03.541 DEBUG:boto:Data: 2022-09-12T20:10:03.542 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.542 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.543 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.543 DEBUG:boto:Params: {} 2022-09-12T20:10:03.544 DEBUG:boto:Token: None 2022-09-12T20:10:03.544 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj85 2022-09-12T20:10:03.545 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k+D4vxXHg6BXzvKuyGDjkiwT+jk= 2022-09-12T20:10:03.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k+D4vxXHg6BXzvKuyGDjkiwT+jk=', 'Content-Length': '0'} 2022-09-12T20:10:03.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000282c5ccb504f602e-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:10:03.549 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:10:03.549 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:10:03.550 DEBUG:boto:Method: GET 2022-09-12T20:10:03.550 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:10:03.551 DEBUG:boto:Data: 2022-09-12T20:10:03.552 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.552 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.553 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.553 DEBUG:boto:Params: {} 2022-09-12T20:10:03.554 DEBUG:boto:Token: None 2022-09-12T20:10:03.554 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj85 2022-09-12T20:10:03.555 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BijDYMfAP0SKbMmMG17RUOj1rWE= 2022-09-12T20:10:03.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BijDYMfAP0SKbMmMG17RUOj1rWE=', 'Content-Length': '0'} 2022-09-12T20:10:03.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da9af15f7a5114d3-00631f921b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.558 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:10:03.559 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:10:03.560 DEBUG:boto:Method: GET 2022-09-12T20:10:03.560 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:10:03.561 DEBUG:boto:Data: 2022-09-12T20:10:03.561 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.562 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.562 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.563 DEBUG:boto:Params: {} 2022-09-12T20:10:03.563 DEBUG:boto:Token: None 2022-09-12T20:10:03.564 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj85 2022-09-12T20:10:03.565 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BijDYMfAP0SKbMmMG17RUOj1rWE= 2022-09-12T20:10:03.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BijDYMfAP0SKbMmMG17RUOj1rWE=', 'Content-Length': '0'} 2022-09-12T20:10:03.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7fc5c8e5252769a-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.568 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:03.569 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:03.569 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:10:03.570 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:10:03.570 DEBUG:boto:Method: GET 2022-09-12T20:10:03.571 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:10:03.571 DEBUG:boto:Data: 2022-09-12T20:10:03.572 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.572 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.573 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.573 DEBUG:boto:Params: {} 2022-09-12T20:10:03.574 DEBUG:boto:Token: None 2022-09-12T20:10:03.574 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/ 2022-09-12T20:10:03.575 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8= 2022-09-12T20:10:03.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8=', 'Content-Length': '0'} 2022-09-12T20:10:03.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023feabd8398ddca7-00631f921b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.579 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:03.580 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:03.581 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:03.581 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:10:03.582 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:10:03.582 DEBUG:boto:Method: GET 2022-09-12T20:10:03.583 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:10:03.583 DEBUG:boto:Data: 2022-09-12T20:10:03.584 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.584 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.585 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.585 DEBUG:boto:Params: {} 2022-09-12T20:10:03.586 DEBUG:boto:Token: None 2022-09-12T20:10:03.586 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/ 2022-09-12T20:10:03.587 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8= 2022-09-12T20:10:03.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8=', 'Content-Length': '0'} 2022-09-12T20:10:03.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1a51ff8191a8d37-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.592 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:03.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:10:03.593 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:10:03.594 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:10:03.594 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:10:03.595 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:10:03.595 DEBUG:boto:Method: GET 2022-09-12T20:10:03.596 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:10:03.596 DEBUG:boto:Data: 2022-09-12T20:10:03.597 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.597 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.598 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.599 DEBUG:boto:Params: {} 2022-09-12T20:10:03.599 DEBUG:boto:Token: None 2022-09-12T20:10:03.600 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:10:03.600 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vr0Xp3GfHy25Qu0n0xCAEQE+oVg= 2022-09-12T20:10:03.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vr0Xp3GfHy25Qu0n0xCAEQE+oVg=', 'Content-Length': '0'} 2022-09-12T20:10:03.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ae7bcf0bcad0272-00631f921b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.646 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:10:03.647 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:10:03.647 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:10:03.648 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:10:03.648 DEBUG:boto:Method: GET 2022-09-12T20:10:03.649 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:10:03.649 DEBUG:boto:Data: 2022-09-12T20:10:03.650 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.651 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.651 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.652 DEBUG:boto:Params: {} 2022-09-12T20:10:03.652 DEBUG:boto:Token: None 2022-09-12T20:10:03.653 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:10:03.653 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vr0Xp3GfHy25Qu0n0xCAEQE+oVg= 2022-09-12T20:10:03.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vr0Xp3GfHy25Qu0n0xCAEQE+oVg=', 'Content-Length': '0'} 2022-09-12T20:10:03.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011a01d038721813d-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.657 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:10:03.658 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:10:03.658 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:10:03.659 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:10:03.659 DEBUG:boto:Method: HEAD 2022-09-12T20:10:03.660 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:10:03.660 DEBUG:boto:Data: 2022-09-12T20:10:03.661 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.661 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.662 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.662 DEBUG:boto:Params: {} 2022-09-12T20:10:03.663 DEBUG:boto:Token: None 2022-09-12T20:10:03.664 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:10:03.664 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CF4GxVNT0sqUOBVTVjyJZREOGz8= 2022-09-12T20:10:03.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CF4GxVNT0sqUOBVTVjyJZREOGz8=', 'Content-Length': '0'} 2022-09-12T20:10:03.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcf1cf9edb28368d-00631f921b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.668 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:10:03.669 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:10:03.669 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:10:03.670 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:10:03.670 DEBUG:boto:Method: HEAD 2022-09-12T20:10:03.671 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:10:03.671 DEBUG:boto:Data: 2022-09-12T20:10:03.672 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.672 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.673 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.673 DEBUG:boto:Params: {} 2022-09-12T20:10:03.674 DEBUG:boto:Token: None 2022-09-12T20:10:03.675 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:10:03.675 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CF4GxVNT0sqUOBVTVjyJZREOGz8= 2022-09-12T20:10:03.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CF4GxVNT0sqUOBVTVjyJZREOGz8=', 'Content-Length': '0'} 2022-09-12T20:10:03.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c57d09861207adcb-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:10:03.680 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:10:03.680 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:10:03.681 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:10:03.681 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:10:03.682 DEBUG:boto:Method: GET 2022-09-12T20:10:03.682 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:10:03.683 DEBUG:boto:Data: 2022-09-12T20:10:03.683 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.684 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.685 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.685 DEBUG:boto:Params: {} 2022-09-12T20:10:03.686 DEBUG:boto:Token: None 2022-09-12T20:10:03.686 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:10:03.687 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vr0Xp3GfHy25Qu0n0xCAEQE+oVg= 2022-09-12T20:10:03.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vr0Xp3GfHy25Qu0n0xCAEQE+oVg=', 'Content-Length': '0'} 2022-09-12T20:10:03.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005873ad9b434822c6-00631f921b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.691 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:10:03.691 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:10:03.692 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:10:03.692 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:10:03.693 DEBUG:boto:Method: GET 2022-09-12T20:10:03.693 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:10:03.694 DEBUG:boto:Data: 2022-09-12T20:10:03.695 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.695 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.696 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.696 DEBUG:boto:Params: {} 2022-09-12T20:10:03.697 DEBUG:boto:Token: None 2022-09-12T20:10:03.697 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:10:03.698 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vr0Xp3GfHy25Qu0n0xCAEQE+oVg= 2022-09-12T20:10:03.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vr0Xp3GfHy25Qu0n0xCAEQE+oVg=', 'Content-Length': '0'} 2022-09-12T20:10:03.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f096d05eb83c5fbb-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.742 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:03.743 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:03.743 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:10:03.744 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:10:03.745 DEBUG:boto:Method: GET 2022-09-12T20:10:03.745 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:10:03.746 DEBUG:boto:Data: 2022-09-12T20:10:03.746 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.747 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.747 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.748 DEBUG:boto:Params: {} 2022-09-12T20:10:03.748 DEBUG:boto:Token: None 2022-09-12T20:10:03.749 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/ 2022-09-12T20:10:03.750 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8= 2022-09-12T20:10:03.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8=', 'Content-Length': '0'} 2022-09-12T20:10:03.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d823bc6043bd4396-00631f921b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.755 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:03.756 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:03.756 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:03.757 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:10:03.757 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:10:03.758 DEBUG:boto:Method: GET 2022-09-12T20:10:03.758 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:10:03.759 DEBUG:boto:Data: 2022-09-12T20:10:03.759 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.760 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.760 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.761 DEBUG:boto:Params: {} 2022-09-12T20:10:03.761 DEBUG:boto:Token: None 2022-09-12T20:10:03.762 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/ 2022-09-12T20:10:03.763 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8= 2022-09-12T20:10:03.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8=', 'Content-Length': '0'} 2022-09-12T20:10:03.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000317ccd56ab27548f-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.768 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:03.768 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:10:03.769 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:10:03.769 DEBUG:boto:Method: HEAD 2022-09-12T20:10:03.770 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:10:03.770 DEBUG:boto:Data: 2022-09-12T20:10:03.771 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.771 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.772 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.772 DEBUG:boto:Params: {} 2022-09-12T20:10:03.773 DEBUG:boto:Token: None 2022-09-12T20:10:03.774 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj92 2022-09-12T20:10:03.774 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UYyH+WYktCVS1Eug6VlwQDSw3WU= 2022-09-12T20:10:03.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UYyH+WYktCVS1Eug6VlwQDSw3WU=', 'Content-Length': '0'} 2022-09-12T20:10:03.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e9b02308aedc426-00631f921b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.778 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:10:03.778 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:10:03.779 DEBUG:boto:Method: HEAD 2022-09-12T20:10:03.779 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:10:03.780 DEBUG:boto:Data: 2022-09-12T20:10:03.780 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.781 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.781 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.782 DEBUG:boto:Params: {} 2022-09-12T20:10:03.782 DEBUG:boto:Token: None 2022-09-12T20:10:03.783 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj92 2022-09-12T20:10:03.783 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UYyH+WYktCVS1Eug6VlwQDSw3WU= 2022-09-12T20:10:03.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UYyH+WYktCVS1Eug6VlwQDSw3WU=', 'Content-Length': '0'} 2022-09-12T20:10:03.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059414bd336ae87c9-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:10:03.787 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:10:03.788 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:10:03.788 DEBUG:boto:Method: GET 2022-09-12T20:10:03.789 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:10:03.789 DEBUG:boto:Data: 2022-09-12T20:10:03.790 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.791 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.791 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.792 DEBUG:boto:Params: {} 2022-09-12T20:10:03.792 DEBUG:boto:Token: None 2022-09-12T20:10:03.793 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj92 2022-09-12T20:10:03.794 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bbBTkf4w0azazOsO9hCzmXv8mNo= 2022-09-12T20:10:03.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bbBTkf4w0azazOsO9hCzmXv8mNo=', 'Content-Length': '0'} 2022-09-12T20:10:03.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c841249bb8938558-00631f921b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.838 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:10:03.839 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:10:03.839 DEBUG:boto:Method: GET 2022-09-12T20:10:03.840 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:10:03.840 DEBUG:boto:Data: 2022-09-12T20:10:03.841 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.841 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.842 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.842 DEBUG:boto:Params: {} 2022-09-12T20:10:03.843 DEBUG:boto:Token: None 2022-09-12T20:10:03.844 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj92 2022-09-12T20:10:03.844 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bbBTkf4w0azazOsO9hCzmXv8mNo= 2022-09-12T20:10:03.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bbBTkf4w0azazOsO9hCzmXv8mNo=', 'Content-Length': '0'} 2022-09-12T20:10:03.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000975227cc481f8d5f-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.848 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:03.848 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:03.849 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:10:03.850 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:10:03.850 DEBUG:boto:Method: GET 2022-09-12T20:10:03.851 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:10:03.851 DEBUG:boto:Data: 2022-09-12T20:10:03.852 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.852 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.853 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.853 DEBUG:boto:Params: {} 2022-09-12T20:10:03.854 DEBUG:boto:Token: None 2022-09-12T20:10:03.855 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/ 2022-09-12T20:10:03.855 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8= 2022-09-12T20:10:03.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8=', 'Content-Length': '0'} 2022-09-12T20:10:03.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004244b40ccfffbbae-00631f921b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.859 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:03.860 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:03.861 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:03.861 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:10:03.862 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:10:03.863 DEBUG:boto:Method: GET 2022-09-12T20:10:03.863 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:10:03.864 DEBUG:boto:Data: 2022-09-12T20:10:03.864 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.865 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.865 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.866 DEBUG:boto:Params: {} 2022-09-12T20:10:03.866 DEBUG:boto:Token: None 2022-09-12T20:10:03.867 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/ 2022-09-12T20:10:03.867 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8= 2022-09-12T20:10:03.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8=', 'Content-Length': '0'} 2022-09-12T20:10:03.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080549f20e5e5f011-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.914 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:03.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:10:03.916 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:10:03.916 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:10:03.917 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:10:03.917 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:10:03.918 DEBUG:boto:Method: GET 2022-09-12T20:10:03.918 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:10:03.919 DEBUG:boto:Data: 2022-09-12T20:10:03.919 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.920 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.920 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.921 DEBUG:boto:Params: {} 2022-09-12T20:10:03.922 DEBUG:boto:Token: None 2022-09-12T20:10:03.922 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:10:03.923 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:evmhaOblTqqo+dcizlMNBAoV8vM= 2022-09-12T20:10:03.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:evmhaOblTqqo+dcizlMNBAoV8vM=', 'Content-Length': '0'} 2022-09-12T20:10:03.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fda6950f90746bc-00631f921b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.927 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:10:03.927 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:10:03.928 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:10:03.928 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:10:03.929 DEBUG:boto:Method: GET 2022-09-12T20:10:03.929 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:10:03.930 DEBUG:boto:Data: 2022-09-12T20:10:03.931 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.931 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.932 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.932 DEBUG:boto:Params: {} 2022-09-12T20:10:03.933 DEBUG:boto:Token: None 2022-09-12T20:10:03.933 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:10:03.934 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:evmhaOblTqqo+dcizlMNBAoV8vM= 2022-09-12T20:10:03.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:evmhaOblTqqo+dcizlMNBAoV8vM=', 'Content-Length': '0'} 2022-09-12T20:10:03.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000781ebe75745af1eb-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.938 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:10:03.938 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:10:03.939 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:10:03.939 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:10:03.940 DEBUG:boto:Method: HEAD 2022-09-12T20:10:03.940 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:10:03.941 DEBUG:boto:Data: 2022-09-12T20:10:03.941 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.942 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.942 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.943 DEBUG:boto:Params: {} 2022-09-12T20:10:03.943 DEBUG:boto:Token: None 2022-09-12T20:10:03.944 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:10:03.944 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:a5fj9rNhDUtV7giLYDznSDiMOHY= 2022-09-12T20:10:03.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:a5fj9rNhDUtV7giLYDznSDiMOHY=', 'Content-Length': '0'} 2022-09-12T20:10:03.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003faaa02537463ff1-00631f921b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.948 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:10:03.948 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:10:03.949 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:10:03.949 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:10:03.950 DEBUG:boto:Method: HEAD 2022-09-12T20:10:03.950 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:10:03.951 DEBUG:boto:Data: 2022-09-12T20:10:03.952 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.952 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.953 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.953 DEBUG:boto:Params: {} 2022-09-12T20:10:03.954 DEBUG:boto:Token: None 2022-09-12T20:10:03.954 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:10:03.955 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:a5fj9rNhDUtV7giLYDznSDiMOHY= 2022-09-12T20:10:03.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:a5fj9rNhDUtV7giLYDznSDiMOHY=', 'Content-Length': '0'} 2022-09-12T20:10:03.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000134c522aa21bd53a-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:10:03.959 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:10:03.959 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:10:03.960 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:10:03.960 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:10:03.961 DEBUG:boto:Method: GET 2022-09-12T20:10:03.962 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:10:03.962 DEBUG:boto:Data: 2022-09-12T20:10:03.963 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.963 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.964 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.964 DEBUG:boto:Params: {} 2022-09-12T20:10:03.965 DEBUG:boto:Token: None 2022-09-12T20:10:03.965 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:10:03.966 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:evmhaOblTqqo+dcizlMNBAoV8vM= 2022-09-12T20:10:03.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:evmhaOblTqqo+dcizlMNBAoV8vM=', 'Content-Length': '0'} 2022-09-12T20:10:03.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e9aa617dbb4a8c1-00631f921b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.970 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:10:03.971 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:10:03.971 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:10:03.972 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:10:03.972 DEBUG:boto:Method: GET 2022-09-12T20:10:03.973 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:10:03.973 DEBUG:boto:Data: 2022-09-12T20:10:03.974 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.974 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.975 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.975 DEBUG:boto:Params: {} 2022-09-12T20:10:03.976 DEBUG:boto:Token: None 2022-09-12T20:10:03.977 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:10:03.977 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:evmhaOblTqqo+dcizlMNBAoV8vM= 2022-09-12T20:10:03.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:evmhaOblTqqo+dcizlMNBAoV8vM=', 'Content-Length': '0'} 2022-09-12T20:10:03.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f29465d60f1fa34-00631f921b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.981 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:03.982 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:03.982 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:10:03.983 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:10:03.983 DEBUG:boto:Method: GET 2022-09-12T20:10:03.984 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:10:03.984 DEBUG:boto:Data: 2022-09-12T20:10:03.985 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.985 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:03.986 DEBUG:boto:Port: 8001 2022-09-12T20:10:03.986 DEBUG:boto:Params: {} 2022-09-12T20:10:03.987 DEBUG:boto:Token: None 2022-09-12T20:10:03.988 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/ 2022-09-12T20:10:03.988 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8= 2022-09-12T20:10:03.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8=', 'Content-Length': '0'} 2022-09-12T20:10:03.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aacaba519bf44c6f-00631f921b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:03.992 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:03.993 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:03.993 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:03.994 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:10:03.994 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:10:03.995 DEBUG:boto:Method: GET 2022-09-12T20:10:03.996 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:10:03.996 DEBUG:boto:Data: 2022-09-12T20:10:03.997 DEBUG:boto:Headers: {} 2022-09-12T20:10:03.997 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:03.998 DEBUG:boto:Port: 8000 2022-09-12T20:10:03.998 DEBUG:boto:Params: {} 2022-09-12T20:10:03.999 DEBUG:boto:Token: None 2022-09-12T20:10:03.999 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:03 GMT /jdpiak-75/ 2022-09-12T20:10:04.000 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8= 2022-09-12T20:10:04.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9en7SUlonTCQekXIHkcVjbmpVI8=', 'Content-Length': '0'} 2022-09-12T20:10:04.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e81ffe3578589c33-00631f921c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.005 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:04.006 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:10:04.006 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:10:04.007 DEBUG:boto:Method: HEAD 2022-09-12T20:10:04.007 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:10:04.008 DEBUG:boto:Data: 2022-09-12T20:10:04.008 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.009 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.010 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.010 DEBUG:boto:Params: {} 2022-09-12T20:10:04.011 DEBUG:boto:Token: None 2022-09-12T20:10:04.011 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj95 2022-09-12T20:10:04.012 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jaeXp/chuKlcx0tU+rwehqfcOpE= 2022-09-12T20:10:04.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jaeXp/chuKlcx0tU+rwehqfcOpE=', 'Content-Length': '0'} 2022-09-12T20:10:04.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000699f0d42ebc4d2f3-00631f921c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.016 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:10:04.016 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:10:04.017 DEBUG:boto:Method: HEAD 2022-09-12T20:10:04.017 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:10:04.018 DEBUG:boto:Data: 2022-09-12T20:10:04.018 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.019 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:04.019 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.020 DEBUG:boto:Params: {} 2022-09-12T20:10:04.021 DEBUG:boto:Token: None 2022-09-12T20:10:04.021 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj95 2022-09-12T20:10:04.022 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jaeXp/chuKlcx0tU+rwehqfcOpE= 2022-09-12T20:10:04.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jaeXp/chuKlcx0tU+rwehqfcOpE=', 'Content-Length': '0'} 2022-09-12T20:10:04.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b5162148ab94248-00631f921c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:10:04.026 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:10:04.027 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:10:04.028 DEBUG:boto:Method: GET 2022-09-12T20:10:04.028 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:10:04.029 DEBUG:boto:Data: 2022-09-12T20:10:04.029 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.030 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.030 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.031 DEBUG:boto:Params: {} 2022-09-12T20:10:04.031 DEBUG:boto:Token: None 2022-09-12T20:10:04.032 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj95 2022-09-12T20:10:04.033 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4YkLl6YsAiLw3VBXuNouSlGYQps= 2022-09-12T20:10:04.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4YkLl6YsAiLw3VBXuNouSlGYQps=', 'Content-Length': '0'} 2022-09-12T20:10:04.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008112808dc723e156-00631f921c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.078 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:10:04.079 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:10:04.079 DEBUG:boto:Method: GET 2022-09-12T20:10:04.080 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:10:04.081 DEBUG:boto:Data: 2022-09-12T20:10:04.081 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.082 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:04.082 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.083 DEBUG:boto:Params: {} 2022-09-12T20:10:04.083 DEBUG:boto:Token: None 2022-09-12T20:10:04.084 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj95 2022-09-12T20:10:04.085 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4YkLl6YsAiLw3VBXuNouSlGYQps= 2022-09-12T20:10:04.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4YkLl6YsAiLw3VBXuNouSlGYQps=', 'Content-Length': '0'} 2022-09-12T20:10:04.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e4807dadcd92014-00631f921c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.088 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:04.089 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:04.089 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:10:04.090 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:10:04.090 DEBUG:boto:Method: GET 2022-09-12T20:10:04.091 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:10:04.092 DEBUG:boto:Data: 2022-09-12T20:10:04.092 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.093 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.093 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.094 DEBUG:boto:Params: {} 2022-09-12T20:10:04.094 DEBUG:boto:Token: None 2022-09-12T20:10:04.095 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/ 2022-09-12T20:10:04.096 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E= 2022-09-12T20:10:04.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E=', 'Content-Length': '0'} 2022-09-12T20:10:04.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030315e44b1830207-00631f921c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.100 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:04.101 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:04.101 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:04.102 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:10:04.102 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:10:04.103 DEBUG:boto:Method: GET 2022-09-12T20:10:04.103 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:10:04.104 DEBUG:boto:Data: 2022-09-12T20:10:04.104 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.105 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:04.106 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.106 DEBUG:boto:Params: {} 2022-09-12T20:10:04.107 DEBUG:boto:Token: None 2022-09-12T20:10:04.107 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/ 2022-09-12T20:10:04.108 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E= 2022-09-12T20:10:04.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E=', 'Content-Length': '0'} 2022-09-12T20:10:04.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045eb1418013d6540-00631f921c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.113 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:04.114 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-75 zones={test-zone3, test-zone1} 2022-09-12T20:10:04.114 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-75 zones={test-zone3, test-zone2} 2022-09-12T20:10:04.115 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:04.115 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:04.116 DEBUG:boto:Method: HEAD 2022-09-12T20:10:04.116 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:10:04.117 DEBUG:boto:Data: 2022-09-12T20:10:04.117 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.118 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.118 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.119 DEBUG:boto:Params: {} 2022-09-12T20:10:04.119 DEBUG:boto:Token: None 2022-09-12T20:10:04.120 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/ 2022-09-12T20:10:04.121 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RXMEXbOzYFkQr5HQ/Q/tL3ij8ow= 2022-09-12T20:10:04.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RXMEXbOzYFkQr5HQ/Q/tL3ij8ow=', 'Content-Length': '0'} 2022-09-12T20:10:04.124 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000956b12107b340f13-00631f921c-1826-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.125 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:04.125 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:04.126 DEBUG:boto:Method: HEAD 2022-09-12T20:10:04.126 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:10:04.127 DEBUG:boto:Data: 2022-09-12T20:10:04.127 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.128 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.128 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.129 DEBUG:boto:Params: {} 2022-09-12T20:10:04.129 DEBUG:boto:Token: None 2022-09-12T20:10:04.130 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/ 2022-09-12T20:10:04.130 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RXMEXbOzYFkQr5HQ/Q/tL3ij8ow= 2022-09-12T20:10:04.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RXMEXbOzYFkQr5HQ/Q/tL3ij8ow=', 'Content-Length': '0'} 2022-09-12T20:10:04.134 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ed6fb20cca7d201-00631f921c-189d-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.134 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T20:10:04.135 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:04.135 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:04.136 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:10:04.136 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:10:04.137 DEBUG:boto:Method: GET 2022-09-12T20:10:04.137 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:10:04.138 DEBUG:boto:Data: 2022-09-12T20:10:04.138 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.139 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.139 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.140 DEBUG:boto:Params: {} 2022-09-12T20:10:04.141 DEBUG:boto:Token: None 2022-09-12T20:10:04.141 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/?versions 2022-09-12T20:10:04.142 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rb1Z2JWcVOt+8j01YMn72qZcvDA= 2022-09-12T20:10:04.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rb1Z2JWcVOt+8j01YMn72qZcvDA=', 'Content-Length': '0'} 2022-09-12T20:10:04.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009da5da4bf0a86b37-00631f921c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.146 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:04.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:10:04.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-12T20:10:04.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:10:04.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:10:04.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:10:04.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-12T20:10:04.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:10:04.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:10:04.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:10:04.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:10:04.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-12T20:10:04.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:10:04.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:10:04.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:10:04.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:10:04.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-12T20:10:04.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:10:04.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:10:04.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-12T20:10:04.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:10:04.158 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T20:10:04.158 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:04.159 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:04.159 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:10:04.160 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:10:04.160 DEBUG:boto:Method: GET 2022-09-12T20:10:04.161 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:10:04.161 DEBUG:boto:Data: 2022-09-12T20:10:04.162 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.162 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.163 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.163 DEBUG:boto:Params: {} 2022-09-12T20:10:04.164 DEBUG:boto:Token: None 2022-09-12T20:10:04.165 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/?versions 2022-09-12T20:10:04.165 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rb1Z2JWcVOt+8j01YMn72qZcvDA= 2022-09-12T20:10:04.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rb1Z2JWcVOt+8j01YMn72qZcvDA=', 'Content-Length': '0'} 2022-09-12T20:10:04.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3261de9ea567aa7-00631f921c-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.170 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:04.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:10:04.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-12T20:10:04.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:10:04.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:10:04.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:10:04.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-12T20:10:04.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:10:04.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:10:04.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:10:04.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:10:04.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-12T20:10:04.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:10:04.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:10:04.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:10:04.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:10:04.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-12T20:10:04.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:10:04.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:10:04.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-12T20:10:04.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:10:04.182 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:04.183 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:04.183 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:10:04.184 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:10:04.184 DEBUG:boto:Method: GET 2022-09-12T20:10:04.185 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:10:04.185 DEBUG:boto:Data: 2022-09-12T20:10:04.186 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.187 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.188 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.189 DEBUG:boto:Params: {} 2022-09-12T20:10:04.190 DEBUG:boto:Token: None 2022-09-12T20:10:04.191 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/?versions 2022-09-12T20:10:04.192 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rb1Z2JWcVOt+8j01YMn72qZcvDA= 2022-09-12T20:10:04.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rb1Z2JWcVOt+8j01YMn72qZcvDA=', 'Content-Length': '0'} 2022-09-12T20:10:04.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8c22d28c540da26-00631f921c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.196 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:04.198 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:04.199 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:04.199 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:10:04.200 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:10:04.200 DEBUG:boto:Method: GET 2022-09-12T20:10:04.201 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:10:04.201 DEBUG:boto:Data: 2022-09-12T20:10:04.202 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.202 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.203 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.203 DEBUG:boto:Params: {} 2022-09-12T20:10:04.204 DEBUG:boto:Token: None 2022-09-12T20:10:04.204 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/?versions 2022-09-12T20:10:04.205 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rb1Z2JWcVOt+8j01YMn72qZcvDA= 2022-09-12T20:10:04.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rb1Z2JWcVOt+8j01YMn72qZcvDA=', 'Content-Length': '0'} 2022-09-12T20:10:04.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074494c189f22481a-00631f921c-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.209 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:04.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:10:04.211 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:10:04.212 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:10:04.212 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:10:04.213 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:10:04.214 DEBUG:boto:Method: GET 2022-09-12T20:10:04.214 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:10:04.215 DEBUG:boto:Data: 2022-09-12T20:10:04.215 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.216 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.216 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.217 DEBUG:boto:Params: {} 2022-09-12T20:10:04.218 DEBUG:boto:Token: None 2022-09-12T20:10:04.218 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:10:04.219 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V2bMkFTUQKRG2ffFP9+pEQubiZA= 2022-09-12T20:10:04.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V2bMkFTUQKRG2ffFP9+pEQubiZA=', 'Content-Length': '0'} 2022-09-12T20:10:04.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059dafdd90934ce7d-00631f921c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.262 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:10:04.263 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:10:04.263 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:10:04.264 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:10:04.264 DEBUG:boto:Method: GET 2022-09-12T20:10:04.265 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:10:04.265 DEBUG:boto:Data: 2022-09-12T20:10:04.266 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.267 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.267 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.268 DEBUG:boto:Params: {} 2022-09-12T20:10:04.268 DEBUG:boto:Token: None 2022-09-12T20:10:04.269 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:10:04.270 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V2bMkFTUQKRG2ffFP9+pEQubiZA= 2022-09-12T20:10:04.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V2bMkFTUQKRG2ffFP9+pEQubiZA=', 'Content-Length': '0'} 2022-09-12T20:10:04.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c4a2503de6186e3-00631f921c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.273 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:10:04.274 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:10:04.275 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:10:04.275 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:10:04.276 DEBUG:boto:Method: HEAD 2022-09-12T20:10:04.276 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:10:04.277 DEBUG:boto:Data: 2022-09-12T20:10:04.277 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.278 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.278 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.279 DEBUG:boto:Params: {} 2022-09-12T20:10:04.280 DEBUG:boto:Token: None 2022-09-12T20:10:04.280 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:10:04.281 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b7u7ovQEsbRTrMstf4IUg/oLubE= 2022-09-12T20:10:04.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b7u7ovQEsbRTrMstf4IUg/oLubE=', 'Content-Length': '0'} 2022-09-12T20:10:04.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000400b9505709beabb-00631f921c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.284 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:10:04.285 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:10:04.285 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:10:04.286 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:10:04.286 DEBUG:boto:Method: HEAD 2022-09-12T20:10:04.287 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:10:04.287 DEBUG:boto:Data: 2022-09-12T20:10:04.288 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.288 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.289 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.289 DEBUG:boto:Params: {} 2022-09-12T20:10:04.290 DEBUG:boto:Token: None 2022-09-12T20:10:04.290 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:10:04.291 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b7u7ovQEsbRTrMstf4IUg/oLubE= 2022-09-12T20:10:04.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b7u7ovQEsbRTrMstf4IUg/oLubE=', 'Content-Length': '0'} 2022-09-12T20:10:04.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000716e714a4de79ba1-00631f921c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:10:04.295 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:10:04.295 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:10:04.296 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:10:04.297 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:10:04.297 DEBUG:boto:Method: GET 2022-09-12T20:10:04.298 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:10:04.298 DEBUG:boto:Data: 2022-09-12T20:10:04.299 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.299 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.300 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.300 DEBUG:boto:Params: {} 2022-09-12T20:10:04.301 DEBUG:boto:Token: None 2022-09-12T20:10:04.301 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:10:04.302 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V2bMkFTUQKRG2ffFP9+pEQubiZA= 2022-09-12T20:10:04.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V2bMkFTUQKRG2ffFP9+pEQubiZA=', 'Content-Length': '0'} 2022-09-12T20:10:04.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000459dbb37d666d28b-00631f921c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.306 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:10:04.306 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:10:04.307 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:10:04.307 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:10:04.308 DEBUG:boto:Method: GET 2022-09-12T20:10:04.308 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:10:04.309 DEBUG:boto:Data: 2022-09-12T20:10:04.309 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.310 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.311 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.311 DEBUG:boto:Params: {} 2022-09-12T20:10:04.312 DEBUG:boto:Token: None 2022-09-12T20:10:04.312 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:10:04.313 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V2bMkFTUQKRG2ffFP9+pEQubiZA= 2022-09-12T20:10:04.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V2bMkFTUQKRG2ffFP9+pEQubiZA=', 'Content-Length': '0'} 2022-09-12T20:10:04.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000436ada41aa93e72e-00631f921c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.316 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:04.317 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:04.317 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:10:04.318 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:10:04.318 DEBUG:boto:Method: GET 2022-09-12T20:10:04.319 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:10:04.320 DEBUG:boto:Data: 2022-09-12T20:10:04.320 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.321 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.321 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.322 DEBUG:boto:Params: {} 2022-09-12T20:10:04.322 DEBUG:boto:Token: None 2022-09-12T20:10:04.323 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/ 2022-09-12T20:10:04.323 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E= 2022-09-12T20:10:04.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E=', 'Content-Length': '0'} 2022-09-12T20:10:04.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dbe4ba239b6c71a-00631f921c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.327 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:04.328 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:04.329 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:04.329 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:10:04.330 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:10:04.330 DEBUG:boto:Method: GET 2022-09-12T20:10:04.331 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:10:04.331 DEBUG:boto:Data: 2022-09-12T20:10:04.332 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.332 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.333 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.333 DEBUG:boto:Params: {} 2022-09-12T20:10:04.334 DEBUG:boto:Token: None 2022-09-12T20:10:04.335 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/ 2022-09-12T20:10:04.335 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E= 2022-09-12T20:10:04.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E=', 'Content-Length': '0'} 2022-09-12T20:10:04.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d0da041bef22678-00631f921c-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.339 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:04.340 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:10:04.340 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:10:04.341 DEBUG:boto:Method: HEAD 2022-09-12T20:10:04.342 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:10:04.342 DEBUG:boto:Data: 2022-09-12T20:10:04.343 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.343 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.344 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.344 DEBUG:boto:Params: {} 2022-09-12T20:10:04.345 DEBUG:boto:Token: None 2022-09-12T20:10:04.345 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj11 2022-09-12T20:10:04.346 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PVAXt2sz18/bvg4kF2kwp3pWw10= 2022-09-12T20:10:04.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PVAXt2sz18/bvg4kF2kwp3pWw10=', 'Content-Length': '0'} 2022-09-12T20:10:04.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df870e89397e0604-00631f921c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.349 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:10:04.350 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:10:04.350 DEBUG:boto:Method: HEAD 2022-09-12T20:10:04.351 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:10:04.352 DEBUG:boto:Data: 2022-09-12T20:10:04.352 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.353 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.353 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.354 DEBUG:boto:Params: {} 2022-09-12T20:10:04.355 DEBUG:boto:Token: None 2022-09-12T20:10:04.355 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj11 2022-09-12T20:10:04.356 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PVAXt2sz18/bvg4kF2kwp3pWw10= 2022-09-12T20:10:04.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PVAXt2sz18/bvg4kF2kwp3pWw10=', 'Content-Length': '0'} 2022-09-12T20:10:04.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf6c104721941cb5-00631f921c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:10:04.360 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:10:04.360 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:10:04.361 DEBUG:boto:Method: GET 2022-09-12T20:10:04.361 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:10:04.362 DEBUG:boto:Data: 2022-09-12T20:10:04.363 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.363 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.364 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.364 DEBUG:boto:Params: {} 2022-09-12T20:10:04.365 DEBUG:boto:Token: None 2022-09-12T20:10:04.365 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj11 2022-09-12T20:10:04.366 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nl9ZhYUGYWgRgXpD3WBsYley+s4= 2022-09-12T20:10:04.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nl9ZhYUGYWgRgXpD3WBsYley+s4=', 'Content-Length': '0'} 2022-09-12T20:10:04.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007052a0ccc02d1db7-00631f921c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.369 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:10:04.370 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:10:04.371 DEBUG:boto:Method: GET 2022-09-12T20:10:04.371 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:10:04.372 DEBUG:boto:Data: 2022-09-12T20:10:04.372 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.373 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.373 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.374 DEBUG:boto:Params: {} 2022-09-12T20:10:04.374 DEBUG:boto:Token: None 2022-09-12T20:10:04.375 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj11 2022-09-12T20:10:04.375 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nl9ZhYUGYWgRgXpD3WBsYley+s4= 2022-09-12T20:10:04.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nl9ZhYUGYWgRgXpD3WBsYley+s4=', 'Content-Length': '0'} 2022-09-12T20:10:04.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c870ba85a00f71e-00631f921c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.422 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:04.423 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:04.423 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:10:04.424 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:10:04.424 DEBUG:boto:Method: GET 2022-09-12T20:10:04.425 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:10:04.426 DEBUG:boto:Data: 2022-09-12T20:10:04.426 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.427 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.427 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.428 DEBUG:boto:Params: {} 2022-09-12T20:10:04.428 DEBUG:boto:Token: None 2022-09-12T20:10:04.429 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/ 2022-09-12T20:10:04.430 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E= 2022-09-12T20:10:04.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E=', 'Content-Length': '0'} 2022-09-12T20:10:04.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7a3bee8a0099662-00631f921c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.434 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:04.435 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:04.435 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:04.436 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:10:04.436 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:10:04.437 DEBUG:boto:Method: GET 2022-09-12T20:10:04.437 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:10:04.438 DEBUG:boto:Data: 2022-09-12T20:10:04.438 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.439 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.439 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.440 DEBUG:boto:Params: {} 2022-09-12T20:10:04.441 DEBUG:boto:Token: None 2022-09-12T20:10:04.441 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/ 2022-09-12T20:10:04.442 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E= 2022-09-12T20:10:04.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E=', 'Content-Length': '0'} 2022-09-12T20:10:04.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b7c08aa6a4a1ea0-00631f921c-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.446 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:04.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:10:04.447 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:10:04.448 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:10:04.448 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:10:04.449 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:10:04.449 DEBUG:boto:Method: GET 2022-09-12T20:10:04.450 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:10:04.450 DEBUG:boto:Data: 2022-09-12T20:10:04.451 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.451 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.452 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.452 DEBUG:boto:Params: {} 2022-09-12T20:10:04.453 DEBUG:boto:Token: None 2022-09-12T20:10:04.454 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:10:04.454 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:U5JipcePLi/Jg62bgDhJ8gKUbdU= 2022-09-12T20:10:04.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:U5JipcePLi/Jg62bgDhJ8gKUbdU=', 'Content-Length': '0'} 2022-09-12T20:10:04.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e5e2737b549c175-00631f921c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.458 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:10:04.458 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:10:04.459 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:10:04.459 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:10:04.460 DEBUG:boto:Method: GET 2022-09-12T20:10:04.460 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:10:04.461 DEBUG:boto:Data: 2022-09-12T20:10:04.461 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.462 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.463 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.463 DEBUG:boto:Params: {} 2022-09-12T20:10:04.464 DEBUG:boto:Token: None 2022-09-12T20:10:04.464 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:10:04.465 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:U5JipcePLi/Jg62bgDhJ8gKUbdU= 2022-09-12T20:10:04.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:U5JipcePLi/Jg62bgDhJ8gKUbdU=', 'Content-Length': '0'} 2022-09-12T20:10:04.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d6ed6bff9f01c67-00631f921c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.469 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:10:04.469 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:10:04.470 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:10:04.470 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:10:04.471 DEBUG:boto:Method: HEAD 2022-09-12T20:10:04.471 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:10:04.472 DEBUG:boto:Data: 2022-09-12T20:10:04.472 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.473 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.473 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.474 DEBUG:boto:Params: {} 2022-09-12T20:10:04.474 DEBUG:boto:Token: None 2022-09-12T20:10:04.475 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:10:04.475 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S7u0h+902MQHhtlbczX1d/2/Jwg= 2022-09-12T20:10:04.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S7u0h+902MQHhtlbczX1d/2/Jwg=', 'Content-Length': '0'} 2022-09-12T20:10:04.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ba577ca2a22a4a3-00631f921c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.479 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:10:04.479 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:10:04.480 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:10:04.480 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:10:04.481 DEBUG:boto:Method: HEAD 2022-09-12T20:10:04.481 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:10:04.482 DEBUG:boto:Data: 2022-09-12T20:10:04.482 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.483 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.483 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.484 DEBUG:boto:Params: {} 2022-09-12T20:10:04.484 DEBUG:boto:Token: None 2022-09-12T20:10:04.485 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:10:04.486 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S7u0h+902MQHhtlbczX1d/2/Jwg= 2022-09-12T20:10:04.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S7u0h+902MQHhtlbczX1d/2/Jwg=', 'Content-Length': '0'} 2022-09-12T20:10:04.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a61e8ba1a64c010-00631f921c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:10:04.489 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:10:04.490 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:10:04.491 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:10:04.491 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:10:04.492 DEBUG:boto:Method: GET 2022-09-12T20:10:04.492 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:10:04.493 DEBUG:boto:Data: 2022-09-12T20:10:04.493 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.494 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.494 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.495 DEBUG:boto:Params: {} 2022-09-12T20:10:04.495 DEBUG:boto:Token: None 2022-09-12T20:10:04.496 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:10:04.497 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:U5JipcePLi/Jg62bgDhJ8gKUbdU= 2022-09-12T20:10:04.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:U5JipcePLi/Jg62bgDhJ8gKUbdU=', 'Content-Length': '0'} 2022-09-12T20:10:04.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a8e5f7e4806156d-00631f921c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.542 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:10:04.543 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:10:04.543 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:10:04.544 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:10:04.545 DEBUG:boto:Method: GET 2022-09-12T20:10:04.545 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:10:04.546 DEBUG:boto:Data: 2022-09-12T20:10:04.546 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.547 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.547 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.548 DEBUG:boto:Params: {} 2022-09-12T20:10:04.548 DEBUG:boto:Token: None 2022-09-12T20:10:04.549 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:10:04.550 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:U5JipcePLi/Jg62bgDhJ8gKUbdU= 2022-09-12T20:10:04.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:U5JipcePLi/Jg62bgDhJ8gKUbdU=', 'Content-Length': '0'} 2022-09-12T20:10:04.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca2ab07b48a95469-00631f921c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.553 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:04.554 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:04.554 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:10:04.555 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:10:04.555 DEBUG:boto:Method: GET 2022-09-12T20:10:04.556 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:10:04.556 DEBUG:boto:Data: 2022-09-12T20:10:04.557 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.558 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.558 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.559 DEBUG:boto:Params: {} 2022-09-12T20:10:04.559 DEBUG:boto:Token: None 2022-09-12T20:10:04.560 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/ 2022-09-12T20:10:04.560 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E= 2022-09-12T20:10:04.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E=', 'Content-Length': '0'} 2022-09-12T20:10:04.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d3c83234a992057-00631f921c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.565 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:04.566 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:04.566 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:04.567 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:10:04.568 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:10:04.568 DEBUG:boto:Method: GET 2022-09-12T20:10:04.569 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:10:04.569 DEBUG:boto:Data: 2022-09-12T20:10:04.570 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.570 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.571 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.572 DEBUG:boto:Params: {} 2022-09-12T20:10:04.572 DEBUG:boto:Token: None 2022-09-12T20:10:04.573 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/ 2022-09-12T20:10:04.574 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E= 2022-09-12T20:10:04.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E=', 'Content-Length': '0'} 2022-09-12T20:10:04.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c6b7ffef9933dfb-00631f921c-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.579 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:04.580 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:10:04.580 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:10:04.581 DEBUG:boto:Method: HEAD 2022-09-12T20:10:04.581 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:10:04.582 DEBUG:boto:Data: 2022-09-12T20:10:04.582 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.583 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.583 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.584 DEBUG:boto:Params: {} 2022-09-12T20:10:04.584 DEBUG:boto:Token: None 2022-09-12T20:10:04.585 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj115 2022-09-12T20:10:04.585 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HUR3LQkl4EWxfUudoJc5ZNLbUrk= 2022-09-12T20:10:04.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HUR3LQkl4EWxfUudoJc5ZNLbUrk=', 'Content-Length': '0'} 2022-09-12T20:10:04.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f725b2933b4ae8c8-00631f921c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.589 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:10:04.590 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:10:04.590 DEBUG:boto:Method: HEAD 2022-09-12T20:10:04.591 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:10:04.591 DEBUG:boto:Data: 2022-09-12T20:10:04.592 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.592 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.593 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.594 DEBUG:boto:Params: {} 2022-09-12T20:10:04.595 DEBUG:boto:Token: None 2022-09-12T20:10:04.596 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj115 2022-09-12T20:10:04.596 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HUR3LQkl4EWxfUudoJc5ZNLbUrk= 2022-09-12T20:10:04.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HUR3LQkl4EWxfUudoJc5ZNLbUrk=', 'Content-Length': '0'} 2022-09-12T20:10:04.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d09021280aa7047-00631f921c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:10:04.601 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:10:04.601 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:10:04.602 DEBUG:boto:Method: GET 2022-09-12T20:10:04.603 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:10:04.604 DEBUG:boto:Data: 2022-09-12T20:10:04.605 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.606 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.606 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.607 DEBUG:boto:Params: {} 2022-09-12T20:10:04.608 DEBUG:boto:Token: None 2022-09-12T20:10:04.609 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj115 2022-09-12T20:10:04.610 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZDQiJTf1JYfMFFt+Oq7JwIjK6Q= 2022-09-12T20:10:04.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZDQiJTf1JYfMFFt+Oq7JwIjK6Q=', 'Content-Length': '0'} 2022-09-12T20:10:04.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d2426958c325950-00631f921c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.614 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:10:04.615 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:10:04.615 DEBUG:boto:Method: GET 2022-09-12T20:10:04.616 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:10:04.617 DEBUG:boto:Data: 2022-09-12T20:10:04.618 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.618 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.619 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.620 DEBUG:boto:Params: {} 2022-09-12T20:10:04.621 DEBUG:boto:Token: None 2022-09-12T20:10:04.622 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj115 2022-09-12T20:10:04.622 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ZDQiJTf1JYfMFFt+Oq7JwIjK6Q= 2022-09-12T20:10:04.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ZDQiJTf1JYfMFFt+Oq7JwIjK6Q=', 'Content-Length': '0'} 2022-09-12T20:10:04.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a59a5bd5dba2d8b-00631f921c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.666 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:04.667 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:04.668 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:10:04.668 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:10:04.669 DEBUG:boto:Method: GET 2022-09-12T20:10:04.670 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:10:04.671 DEBUG:boto:Data: 2022-09-12T20:10:04.671 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.672 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.673 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.674 DEBUG:boto:Params: {} 2022-09-12T20:10:04.675 DEBUG:boto:Token: None 2022-09-12T20:10:04.675 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/ 2022-09-12T20:10:04.676 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E= 2022-09-12T20:10:04.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E=', 'Content-Length': '0'} 2022-09-12T20:10:04.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f45ce1a7aa55bdb4-00631f921c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.681 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:04.682 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:04.683 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:04.684 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:10:04.684 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:10:04.685 DEBUG:boto:Method: GET 2022-09-12T20:10:04.686 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:10:04.686 DEBUG:boto:Data: 2022-09-12T20:10:04.687 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.688 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.689 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.689 DEBUG:boto:Params: {} 2022-09-12T20:10:04.690 DEBUG:boto:Token: None 2022-09-12T20:10:04.690 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/ 2022-09-12T20:10:04.691 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E= 2022-09-12T20:10:04.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E=', 'Content-Length': '0'} 2022-09-12T20:10:04.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ac7d5ad801df6f5-00631f921c-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.695 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:04.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:10:04.697 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:10:04.698 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:10:04.698 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:10:04.699 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:10:04.700 DEBUG:boto:Method: GET 2022-09-12T20:10:04.701 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:10:04.701 DEBUG:boto:Data: 2022-09-12T20:10:04.702 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.702 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.703 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.704 DEBUG:boto:Params: {} 2022-09-12T20:10:04.704 DEBUG:boto:Token: None 2022-09-12T20:10:04.705 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:10:04.706 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/xq+rwgkPdEAUxwuB2hS+O4v5R4= 2022-09-12T20:10:04.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/xq+rwgkPdEAUxwuB2hS+O4v5R4=', 'Content-Length': '0'} 2022-09-12T20:10:04.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000726678508131b65b-00631f921c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.710 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:10:04.710 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:10:04.711 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:10:04.712 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:10:04.712 DEBUG:boto:Method: GET 2022-09-12T20:10:04.713 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:10:04.713 DEBUG:boto:Data: 2022-09-12T20:10:04.714 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.714 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.715 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.716 DEBUG:boto:Params: {} 2022-09-12T20:10:04.716 DEBUG:boto:Token: None 2022-09-12T20:10:04.717 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:10:04.717 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/xq+rwgkPdEAUxwuB2hS+O4v5R4= 2022-09-12T20:10:04.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/xq+rwgkPdEAUxwuB2hS+O4v5R4=', 'Content-Length': '0'} 2022-09-12T20:10:04.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c4ec57b7dd3a6e2-00631f921c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.721 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:10:04.722 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:10:04.723 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:10:04.724 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:10:04.725 DEBUG:boto:Method: HEAD 2022-09-12T20:10:04.725 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:10:04.726 DEBUG:boto:Data: 2022-09-12T20:10:04.726 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.727 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.728 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.728 DEBUG:boto:Params: {} 2022-09-12T20:10:04.729 DEBUG:boto:Token: None 2022-09-12T20:10:04.729 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:10:04.730 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XU4VO9Tre39hdwrGKK4YpczrjJU= 2022-09-12T20:10:04.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XU4VO9Tre39hdwrGKK4YpczrjJU=', 'Content-Length': '0'} 2022-09-12T20:10:04.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d901ea28b56419ef-00631f921c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.733 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:10:04.733 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:10:04.734 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:10:04.734 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:10:04.735 DEBUG:boto:Method: HEAD 2022-09-12T20:10:04.735 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:10:04.736 DEBUG:boto:Data: 2022-09-12T20:10:04.736 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.737 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.737 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.738 DEBUG:boto:Params: {} 2022-09-12T20:10:04.738 DEBUG:boto:Token: None 2022-09-12T20:10:04.739 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:10:04.740 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XU4VO9Tre39hdwrGKK4YpczrjJU= 2022-09-12T20:10:04.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XU4VO9Tre39hdwrGKK4YpczrjJU=', 'Content-Length': '0'} 2022-09-12T20:10:04.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042a3553fd7d76dcd-00631f921c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:10:04.743 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:10:04.744 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:10:04.744 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:10:04.745 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:10:04.746 DEBUG:boto:Method: GET 2022-09-12T20:10:04.746 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:10:04.747 DEBUG:boto:Data: 2022-09-12T20:10:04.747 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.748 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.748 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.749 DEBUG:boto:Params: {} 2022-09-12T20:10:04.749 DEBUG:boto:Token: None 2022-09-12T20:10:04.750 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:10:04.751 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/xq+rwgkPdEAUxwuB2hS+O4v5R4= 2022-09-12T20:10:04.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/xq+rwgkPdEAUxwuB2hS+O4v5R4=', 'Content-Length': '0'} 2022-09-12T20:10:04.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074dafa76d5790b81-00631f921c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.754 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:10:04.755 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:10:04.755 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:10:04.756 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:10:04.756 DEBUG:boto:Method: GET 2022-09-12T20:10:04.757 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:10:04.757 DEBUG:boto:Data: 2022-09-12T20:10:04.758 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.759 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.759 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.760 DEBUG:boto:Params: {} 2022-09-12T20:10:04.760 DEBUG:boto:Token: None 2022-09-12T20:10:04.761 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:10:04.761 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/xq+rwgkPdEAUxwuB2hS+O4v5R4= 2022-09-12T20:10:04.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/xq+rwgkPdEAUxwuB2hS+O4v5R4=', 'Content-Length': '0'} 2022-09-12T20:10:04.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f3ee77738a27756-00631f921c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.806 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:04.807 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:04.807 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:10:04.808 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:10:04.808 DEBUG:boto:Method: GET 2022-09-12T20:10:04.809 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:10:04.809 DEBUG:boto:Data: 2022-09-12T20:10:04.810 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.810 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.811 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.812 DEBUG:boto:Params: {} 2022-09-12T20:10:04.812 DEBUG:boto:Token: None 2022-09-12T20:10:04.813 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/ 2022-09-12T20:10:04.813 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E= 2022-09-12T20:10:04.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E=', 'Content-Length': '0'} 2022-09-12T20:10:04.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e204d959d0a3dda-00631f921c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.817 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:04.818 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:04.819 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:04.819 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:10:04.820 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:10:04.820 DEBUG:boto:Method: GET 2022-09-12T20:10:04.821 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:10:04.821 DEBUG:boto:Data: 2022-09-12T20:10:04.822 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.823 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.823 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.824 DEBUG:boto:Params: {} 2022-09-12T20:10:04.824 DEBUG:boto:Token: None 2022-09-12T20:10:04.825 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/ 2022-09-12T20:10:04.825 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E= 2022-09-12T20:10:04.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E=', 'Content-Length': '0'} 2022-09-12T20:10:04.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000326f481986109ae6-00631f921c-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.829 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:04.830 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:10:04.831 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:10:04.831 DEBUG:boto:Method: HEAD 2022-09-12T20:10:04.832 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:10:04.832 DEBUG:boto:Data: 2022-09-12T20:10:04.833 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.833 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.834 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.834 DEBUG:boto:Params: {} 2022-09-12T20:10:04.835 DEBUG:boto:Token: None 2022-09-12T20:10:04.836 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj17 2022-09-12T20:10:04.836 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VEKNnA8q8oUGC/sc7h5E7TuNCiY= 2022-09-12T20:10:04.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VEKNnA8q8oUGC/sc7h5E7TuNCiY=', 'Content-Length': '0'} 2022-09-12T20:10:04.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000654a0b93ab558346-00631f921c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.840 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:10:04.840 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:10:04.841 DEBUG:boto:Method: HEAD 2022-09-12T20:10:04.841 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:10:04.842 DEBUG:boto:Data: 2022-09-12T20:10:04.842 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.843 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.843 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.844 DEBUG:boto:Params: {} 2022-09-12T20:10:04.844 DEBUG:boto:Token: None 2022-09-12T20:10:04.845 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj17 2022-09-12T20:10:04.845 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VEKNnA8q8oUGC/sc7h5E7TuNCiY= 2022-09-12T20:10:04.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VEKNnA8q8oUGC/sc7h5E7TuNCiY=', 'Content-Length': '0'} 2022-09-12T20:10:04.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d35ca7dd82c5837-00631f921c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:10:04.849 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:10:04.850 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:10:04.850 DEBUG:boto:Method: GET 2022-09-12T20:10:04.851 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:10:04.851 DEBUG:boto:Data: 2022-09-12T20:10:04.852 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.853 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.853 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.854 DEBUG:boto:Params: {} 2022-09-12T20:10:04.854 DEBUG:boto:Token: None 2022-09-12T20:10:04.855 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj17 2022-09-12T20:10:04.855 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s0nDVTgC9rFLRXk3dUZ/VSpQW/I= 2022-09-12T20:10:04.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s0nDVTgC9rFLRXk3dUZ/VSpQW/I=', 'Content-Length': '0'} 2022-09-12T20:10:04.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d328e57cf7593a1-00631f921c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.859 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:10:04.859 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:10:04.860 DEBUG:boto:Method: GET 2022-09-12T20:10:04.861 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:10:04.861 DEBUG:boto:Data: 2022-09-12T20:10:04.862 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.862 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.863 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.863 DEBUG:boto:Params: {} 2022-09-12T20:10:04.864 DEBUG:boto:Token: None 2022-09-12T20:10:04.865 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj17 2022-09-12T20:10:04.865 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s0nDVTgC9rFLRXk3dUZ/VSpQW/I= 2022-09-12T20:10:04.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s0nDVTgC9rFLRXk3dUZ/VSpQW/I=', 'Content-Length': '0'} 2022-09-12T20:10:04.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050d5aa9d341301e4-00631f921c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.869 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:04.870 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:04.870 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:10:04.871 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:10:04.871 DEBUG:boto:Method: GET 2022-09-12T20:10:04.872 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:10:04.872 DEBUG:boto:Data: 2022-09-12T20:10:04.873 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.874 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.874 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.875 DEBUG:boto:Params: {} 2022-09-12T20:10:04.875 DEBUG:boto:Token: None 2022-09-12T20:10:04.876 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/ 2022-09-12T20:10:04.877 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E= 2022-09-12T20:10:04.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E=', 'Content-Length': '0'} 2022-09-12T20:10:04.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000583896a6757420b7-00631f921c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.922 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:04.923 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:04.924 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:04.924 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:10:04.925 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:10:04.925 DEBUG:boto:Method: GET 2022-09-12T20:10:04.926 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:10:04.926 DEBUG:boto:Data: 2022-09-12T20:10:04.927 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.927 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.928 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.928 DEBUG:boto:Params: {} 2022-09-12T20:10:04.929 DEBUG:boto:Token: None 2022-09-12T20:10:04.929 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/ 2022-09-12T20:10:04.930 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E= 2022-09-12T20:10:04.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AHv5Vxbi0KcddRzbrGau57QxF/E=', 'Content-Length': '0'} 2022-09-12T20:10:04.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcd946b9c6037542-00631f921c-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.934 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:04.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:10:04.935 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:10:04.936 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:10:04.936 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:10:04.937 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:10:04.938 DEBUG:boto:Method: GET 2022-09-12T20:10:04.938 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:10:04.939 DEBUG:boto:Data: 2022-09-12T20:10:04.939 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.940 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.940 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.941 DEBUG:boto:Params: {} 2022-09-12T20:10:04.941 DEBUG:boto:Token: None 2022-09-12T20:10:04.942 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:10:04.943 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1DhdWV8ZYVDJB7/Jp7eIfW5j4As= 2022-09-12T20:10:04.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1DhdWV8ZYVDJB7/Jp7eIfW5j4As=', 'Content-Length': '0'} 2022-09-12T20:10:04.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a34bbf3d76c474f7-00631f921c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.946 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:10:04.947 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:10:04.947 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:10:04.948 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:10:04.948 DEBUG:boto:Method: GET 2022-09-12T20:10:04.949 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:10:04.949 DEBUG:boto:Data: 2022-09-12T20:10:04.950 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.951 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.951 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.952 DEBUG:boto:Params: {} 2022-09-12T20:10:04.952 DEBUG:boto:Token: None 2022-09-12T20:10:04.953 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:10:04.953 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1DhdWV8ZYVDJB7/Jp7eIfW5j4As= 2022-09-12T20:10:04.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1DhdWV8ZYVDJB7/Jp7eIfW5j4As=', 'Content-Length': '0'} 2022-09-12T20:10:04.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fa93108bb013052-00631f921c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.957 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:10:04.957 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:10:04.958 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:10:04.958 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:10:04.959 DEBUG:boto:Method: HEAD 2022-09-12T20:10:04.959 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:10:04.960 DEBUG:boto:Data: 2022-09-12T20:10:04.960 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.961 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.961 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.962 DEBUG:boto:Params: {} 2022-09-12T20:10:04.963 DEBUG:boto:Token: None 2022-09-12T20:10:04.963 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:10:04.964 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jOchfQY7yOOEla/KDbU8YwZo9jc= 2022-09-12T20:10:04.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jOchfQY7yOOEla/KDbU8YwZo9jc=', 'Content-Length': '0'} 2022-09-12T20:10:04.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000150efc3c4168363d-00631f921c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.967 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:10:04.968 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:10:04.968 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:10:04.969 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:10:04.969 DEBUG:boto:Method: HEAD 2022-09-12T20:10:04.970 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:10:04.971 DEBUG:boto:Data: 2022-09-12T20:10:04.971 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.972 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.972 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.973 DEBUG:boto:Params: {} 2022-09-12T20:10:04.973 DEBUG:boto:Token: None 2022-09-12T20:10:04.974 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:10:04.974 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jOchfQY7yOOEla/KDbU8YwZo9jc= 2022-09-12T20:10:04.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jOchfQY7yOOEla/KDbU8YwZo9jc=', 'Content-Length': '0'} 2022-09-12T20:10:04.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a3cd7f7981a6031-00631f921c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:10:04.978 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:10:04.979 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:10:04.979 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:10:04.980 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:10:04.980 DEBUG:boto:Method: GET 2022-09-12T20:10:04.981 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:10:04.981 DEBUG:boto:Data: 2022-09-12T20:10:04.982 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.982 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:04.983 DEBUG:boto:Port: 8001 2022-09-12T20:10:04.984 DEBUG:boto:Params: {} 2022-09-12T20:10:04.984 DEBUG:boto:Token: None 2022-09-12T20:10:04.985 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:10:04.985 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1DhdWV8ZYVDJB7/Jp7eIfW5j4As= 2022-09-12T20:10:04.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1DhdWV8ZYVDJB7/Jp7eIfW5j4As=', 'Content-Length': '0'} 2022-09-12T20:10:04.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da346176e10781b5-00631f921c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.989 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:10:04.989 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:10:04.990 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:10:04.990 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:10:04.991 DEBUG:boto:Method: GET 2022-09-12T20:10:04.991 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:10:04.992 DEBUG:boto:Data: 2022-09-12T20:10:04.992 DEBUG:boto:Headers: {} 2022-09-12T20:10:04.993 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:04.994 DEBUG:boto:Port: 8000 2022-09-12T20:10:04.994 DEBUG:boto:Params: {} 2022-09-12T20:10:04.995 DEBUG:boto:Token: None 2022-09-12T20:10:04.995 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:04 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:10:04.996 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1DhdWV8ZYVDJB7/Jp7eIfW5j4As= 2022-09-12T20:10:04.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1DhdWV8ZYVDJB7/Jp7eIfW5j4As=', 'Content-Length': '0'} 2022-09-12T20:10:04.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8759f18626d4729-00631f921c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:04.999 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:05.000 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:05.000 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:10:05.001 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:10:05.001 DEBUG:boto:Method: GET 2022-09-12T20:10:05.002 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:10:05.002 DEBUG:boto:Data: 2022-09-12T20:10:05.003 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.004 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.004 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.005 DEBUG:boto:Params: {} 2022-09-12T20:10:05.005 DEBUG:boto:Token: None 2022-09-12T20:10:05.006 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/ 2022-09-12T20:10:05.007 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc= 2022-09-12T20:10:05.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc=', 'Content-Length': '0'} 2022-09-12T20:10:05.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a957e8bc6f8476e1-00631f921d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.011 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:05.011 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:05.012 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:05.013 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:10:05.014 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:10:05.014 DEBUG:boto:Method: GET 2022-09-12T20:10:05.015 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:10:05.015 DEBUG:boto:Data: 2022-09-12T20:10:05.016 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.016 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.017 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.017 DEBUG:boto:Params: {} 2022-09-12T20:10:05.018 DEBUG:boto:Token: None 2022-09-12T20:10:05.019 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/ 2022-09-12T20:10:05.019 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc= 2022-09-12T20:10:05.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc=', 'Content-Length': '0'} 2022-09-12T20:10:05.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003960abab0dcade94-00631f921d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.023 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:05.024 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:10:05.025 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:10:05.025 DEBUG:boto:Method: HEAD 2022-09-12T20:10:05.026 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:10:05.026 DEBUG:boto:Data: 2022-09-12T20:10:05.027 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.027 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.028 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.028 DEBUG:boto:Params: {} 2022-09-12T20:10:05.029 DEBUG:boto:Token: None 2022-09-12T20:10:05.029 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj19 2022-09-12T20:10:05.030 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0MyVXLgE/yjZGB1ZVC7G5YStcbA= 2022-09-12T20:10:05.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0MyVXLgE/yjZGB1ZVC7G5YStcbA=', 'Content-Length': '0'} 2022-09-12T20:10:05.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000776b7b568534445-00631f921d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.033 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:10:05.034 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:10:05.034 DEBUG:boto:Method: HEAD 2022-09-12T20:10:05.035 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:10:05.035 DEBUG:boto:Data: 2022-09-12T20:10:05.036 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.036 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.037 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.037 DEBUG:boto:Params: {} 2022-09-12T20:10:05.038 DEBUG:boto:Token: None 2022-09-12T20:10:05.038 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj19 2022-09-12T20:10:05.039 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0MyVXLgE/yjZGB1ZVC7G5YStcbA= 2022-09-12T20:10:05.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0MyVXLgE/yjZGB1ZVC7G5YStcbA=', 'Content-Length': '0'} 2022-09-12T20:10:05.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c3709087188a69b-00631f921d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:10:05.043 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:10:05.043 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:10:05.044 DEBUG:boto:Method: GET 2022-09-12T20:10:05.044 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:10:05.045 DEBUG:boto:Data: 2022-09-12T20:10:05.045 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.046 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.047 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.047 DEBUG:boto:Params: {} 2022-09-12T20:10:05.048 DEBUG:boto:Token: None 2022-09-12T20:10:05.048 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj19 2022-09-12T20:10:05.049 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1nnPGxQHwOHjUjY9+qMWW7SrL/g= 2022-09-12T20:10:05.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1nnPGxQHwOHjUjY9+qMWW7SrL/g=', 'Content-Length': '0'} 2022-09-12T20:10:05.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005659d512ee01afa9-00631f921d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.094 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:10:05.095 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:10:05.095 DEBUG:boto:Method: GET 2022-09-12T20:10:05.096 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:10:05.096 DEBUG:boto:Data: 2022-09-12T20:10:05.097 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.098 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.098 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.099 DEBUG:boto:Params: {} 2022-09-12T20:10:05.099 DEBUG:boto:Token: None 2022-09-12T20:10:05.100 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj19 2022-09-12T20:10:05.101 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1nnPGxQHwOHjUjY9+qMWW7SrL/g= 2022-09-12T20:10:05.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1nnPGxQHwOHjUjY9+qMWW7SrL/g=', 'Content-Length': '0'} 2022-09-12T20:10:05.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8466aa30cefaa72-00631f921d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.104 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:05.105 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:05.105 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:10:05.106 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:10:05.106 DEBUG:boto:Method: GET 2022-09-12T20:10:05.107 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:10:05.108 DEBUG:boto:Data: 2022-09-12T20:10:05.108 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.109 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.109 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.110 DEBUG:boto:Params: {} 2022-09-12T20:10:05.110 DEBUG:boto:Token: None 2022-09-12T20:10:05.111 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/ 2022-09-12T20:10:05.112 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc= 2022-09-12T20:10:05.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc=', 'Content-Length': '0'} 2022-09-12T20:10:05.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006486f7131f73d3d9-00631f921d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.116 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:05.116 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:05.117 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:05.117 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:10:05.118 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:10:05.119 DEBUG:boto:Method: GET 2022-09-12T20:10:05.119 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:10:05.120 DEBUG:boto:Data: 2022-09-12T20:10:05.120 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.121 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.121 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.122 DEBUG:boto:Params: {} 2022-09-12T20:10:05.122 DEBUG:boto:Token: None 2022-09-12T20:10:05.123 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/ 2022-09-12T20:10:05.123 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc= 2022-09-12T20:10:05.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc=', 'Content-Length': '0'} 2022-09-12T20:10:05.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bc2d01576cbe74f-00631f921d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.127 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:05.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:10:05.129 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:10:05.129 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:10:05.130 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:10:05.130 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:10:05.131 DEBUG:boto:Method: GET 2022-09-12T20:10:05.131 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:10:05.132 DEBUG:boto:Data: 2022-09-12T20:10:05.132 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.133 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.133 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.134 DEBUG:boto:Params: {} 2022-09-12T20:10:05.135 DEBUG:boto:Token: None 2022-09-12T20:10:05.135 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:10:05.136 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p3ridCagjcZ+VOFJv6TEQXlKE4E= 2022-09-12T20:10:05.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p3ridCagjcZ+VOFJv6TEQXlKE4E=', 'Content-Length': '0'} 2022-09-12T20:10:05.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d032234f56934986-00631f921d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.139 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:10:05.140 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:10:05.140 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:10:05.141 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:10:05.141 DEBUG:boto:Method: GET 2022-09-12T20:10:05.142 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:10:05.143 DEBUG:boto:Data: 2022-09-12T20:10:05.143 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.144 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.144 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.145 DEBUG:boto:Params: {} 2022-09-12T20:10:05.145 DEBUG:boto:Token: None 2022-09-12T20:10:05.146 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:10:05.146 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p3ridCagjcZ+VOFJv6TEQXlKE4E= 2022-09-12T20:10:05.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p3ridCagjcZ+VOFJv6TEQXlKE4E=', 'Content-Length': '0'} 2022-09-12T20:10:05.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000998951e5a6a01cf-00631f921d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.150 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:10:05.150 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:10:05.151 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:10:05.151 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:10:05.152 DEBUG:boto:Method: HEAD 2022-09-12T20:10:05.152 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:10:05.153 DEBUG:boto:Data: 2022-09-12T20:10:05.153 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.154 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.154 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.155 DEBUG:boto:Params: {} 2022-09-12T20:10:05.155 DEBUG:boto:Token: None 2022-09-12T20:10:05.156 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:10:05.156 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yrJmzByvbqRylEnlb0hriSqeWto= 2022-09-12T20:10:05.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yrJmzByvbqRylEnlb0hriSqeWto=', 'Content-Length': '0'} 2022-09-12T20:10:05.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002196b0b5d1fc9ab6-00631f921d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.160 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:10:05.160 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:10:05.161 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:10:05.161 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:10:05.162 DEBUG:boto:Method: HEAD 2022-09-12T20:10:05.162 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:10:05.163 DEBUG:boto:Data: 2022-09-12T20:10:05.163 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.164 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.164 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.165 DEBUG:boto:Params: {} 2022-09-12T20:10:05.165 DEBUG:boto:Token: None 2022-09-12T20:10:05.166 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:10:05.166 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yrJmzByvbqRylEnlb0hriSqeWto= 2022-09-12T20:10:05.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yrJmzByvbqRylEnlb0hriSqeWto=', 'Content-Length': '0'} 2022-09-12T20:10:05.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d5e5407c245c8ae-00631f921d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:10:05.170 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:10:05.171 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:10:05.171 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:10:05.172 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:10:05.172 DEBUG:boto:Method: GET 2022-09-12T20:10:05.173 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:10:05.173 DEBUG:boto:Data: 2022-09-12T20:10:05.174 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.174 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.175 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.175 DEBUG:boto:Params: {} 2022-09-12T20:10:05.176 DEBUG:boto:Token: None 2022-09-12T20:10:05.176 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:10:05.177 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p3ridCagjcZ+VOFJv6TEQXlKE4E= 2022-09-12T20:10:05.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p3ridCagjcZ+VOFJv6TEQXlKE4E=', 'Content-Length': '0'} 2022-09-12T20:10:05.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033e02c58b57f153c-00631f921d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.222 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:10:05.223 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:10:05.223 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:10:05.224 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:10:05.224 DEBUG:boto:Method: GET 2022-09-12T20:10:05.225 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:10:05.225 DEBUG:boto:Data: 2022-09-12T20:10:05.226 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.227 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.227 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.228 DEBUG:boto:Params: {} 2022-09-12T20:10:05.228 DEBUG:boto:Token: None 2022-09-12T20:10:05.229 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:10:05.229 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p3ridCagjcZ+VOFJv6TEQXlKE4E= 2022-09-12T20:10:05.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p3ridCagjcZ+VOFJv6TEQXlKE4E=', 'Content-Length': '0'} 2022-09-12T20:10:05.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1bd97c4e57b780b-00631f921d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.233 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:05.234 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:05.234 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:10:05.235 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:10:05.235 DEBUG:boto:Method: GET 2022-09-12T20:10:05.236 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:10:05.236 DEBUG:boto:Data: 2022-09-12T20:10:05.237 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.237 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.238 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.238 DEBUG:boto:Params: {} 2022-09-12T20:10:05.239 DEBUG:boto:Token: None 2022-09-12T20:10:05.239 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/ 2022-09-12T20:10:05.240 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc= 2022-09-12T20:10:05.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc=', 'Content-Length': '0'} 2022-09-12T20:10:05.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034eff24016538018-00631f921d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.244 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:05.245 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:05.245 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:05.246 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:10:05.246 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:10:05.247 DEBUG:boto:Method: GET 2022-09-12T20:10:05.247 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:10:05.248 DEBUG:boto:Data: 2022-09-12T20:10:05.248 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.249 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.249 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.250 DEBUG:boto:Params: {} 2022-09-12T20:10:05.251 DEBUG:boto:Token: None 2022-09-12T20:10:05.251 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/ 2022-09-12T20:10:05.252 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc= 2022-09-12T20:10:05.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc=', 'Content-Length': '0'} 2022-09-12T20:10:05.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008491e2f3f950ca0e-00631f921d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.256 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:05.256 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:10:05.257 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:10:05.257 DEBUG:boto:Method: HEAD 2022-09-12T20:10:05.258 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:10:05.258 DEBUG:boto:Data: 2022-09-12T20:10:05.259 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.259 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.260 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.260 DEBUG:boto:Params: {} 2022-09-12T20:10:05.261 DEBUG:boto:Token: None 2022-09-12T20:10:05.262 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj22 2022-09-12T20:10:05.262 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VHn4Bcatl+lkxNqGqYk5ErjEhv4= 2022-09-12T20:10:05.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VHn4Bcatl+lkxNqGqYk5ErjEhv4=', 'Content-Length': '0'} 2022-09-12T20:10:05.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058d25ad35149dd57-00631f921d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.265 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:10:05.266 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:10:05.267 DEBUG:boto:Method: HEAD 2022-09-12T20:10:05.267 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:10:05.268 DEBUG:boto:Data: 2022-09-12T20:10:05.268 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.269 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.269 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.270 DEBUG:boto:Params: {} 2022-09-12T20:10:05.270 DEBUG:boto:Token: None 2022-09-12T20:10:05.271 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj22 2022-09-12T20:10:05.271 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VHn4Bcatl+lkxNqGqYk5ErjEhv4= 2022-09-12T20:10:05.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VHn4Bcatl+lkxNqGqYk5ErjEhv4=', 'Content-Length': '0'} 2022-09-12T20:10:05.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dc40bb16ca795fd-00631f921d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:10:05.275 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:10:05.275 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:10:05.276 DEBUG:boto:Method: GET 2022-09-12T20:10:05.276 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:10:05.277 DEBUG:boto:Data: 2022-09-12T20:10:05.278 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.278 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.279 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.279 DEBUG:boto:Params: {} 2022-09-12T20:10:05.280 DEBUG:boto:Token: None 2022-09-12T20:10:05.280 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj22 2022-09-12T20:10:05.281 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nU7VzbzCIlHRoEbjt/3/yuX4g7E= 2022-09-12T20:10:05.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nU7VzbzCIlHRoEbjt/3/yuX4g7E=', 'Content-Length': '0'} 2022-09-12T20:10:05.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000388cf8637bdb7490-00631f921d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.284 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:10:05.285 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:10:05.285 DEBUG:boto:Method: GET 2022-09-12T20:10:05.286 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:10:05.286 DEBUG:boto:Data: 2022-09-12T20:10:05.287 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.287 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.288 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.288 DEBUG:boto:Params: {} 2022-09-12T20:10:05.289 DEBUG:boto:Token: None 2022-09-12T20:10:05.289 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj22 2022-09-12T20:10:05.290 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nU7VzbzCIlHRoEbjt/3/yuX4g7E= 2022-09-12T20:10:05.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nU7VzbzCIlHRoEbjt/3/yuX4g7E=', 'Content-Length': '0'} 2022-09-12T20:10:05.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4a5f72ea579056d-00631f921d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.294 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:05.294 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:05.295 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:10:05.295 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:10:05.296 DEBUG:boto:Method: GET 2022-09-12T20:10:05.296 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:10:05.297 DEBUG:boto:Data: 2022-09-12T20:10:05.297 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.298 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.298 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.299 DEBUG:boto:Params: {} 2022-09-12T20:10:05.299 DEBUG:boto:Token: None 2022-09-12T20:10:05.300 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/ 2022-09-12T20:10:05.300 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc= 2022-09-12T20:10:05.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc=', 'Content-Length': '0'} 2022-09-12T20:10:05.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5f2db8c5e902a87-00631f921d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.304 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:05.305 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:05.306 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:05.306 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:10:05.307 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:10:05.307 DEBUG:boto:Method: GET 2022-09-12T20:10:05.308 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:10:05.308 DEBUG:boto:Data: 2022-09-12T20:10:05.309 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.309 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.310 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.310 DEBUG:boto:Params: {} 2022-09-12T20:10:05.311 DEBUG:boto:Token: None 2022-09-12T20:10:05.311 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/ 2022-09-12T20:10:05.312 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc= 2022-09-12T20:10:05.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc=', 'Content-Length': '0'} 2022-09-12T20:10:05.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adccbbb2013d16ea-00631f921d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.316 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:05.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:10:05.317 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:10:05.318 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:10:05.318 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:10:05.319 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:10:05.319 DEBUG:boto:Method: GET 2022-09-12T20:10:05.320 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:10:05.320 DEBUG:boto:Data: 2022-09-12T20:10:05.321 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.321 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.322 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.322 DEBUG:boto:Params: {} 2022-09-12T20:10:05.323 DEBUG:boto:Token: None 2022-09-12T20:10:05.323 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:10:05.324 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GfJBFU/FSEoBfH9ZHH3TkDQd/PQ= 2022-09-12T20:10:05.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GfJBFU/FSEoBfH9ZHH3TkDQd/PQ=', 'Content-Length': '0'} 2022-09-12T20:10:05.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c21a65899d9a0dd5-00631f921d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.327 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:10:05.328 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:10:05.328 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:10:05.329 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:10:05.329 DEBUG:boto:Method: GET 2022-09-12T20:10:05.330 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:10:05.330 DEBUG:boto:Data: 2022-09-12T20:10:05.331 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.331 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.332 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.333 DEBUG:boto:Params: {} 2022-09-12T20:10:05.333 DEBUG:boto:Token: None 2022-09-12T20:10:05.334 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:10:05.334 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GfJBFU/FSEoBfH9ZHH3TkDQd/PQ= 2022-09-12T20:10:05.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GfJBFU/FSEoBfH9ZHH3TkDQd/PQ=', 'Content-Length': '0'} 2022-09-12T20:10:05.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055931444ffe48951-00631f921d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.378 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:10:05.379 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:10:05.379 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:10:05.380 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:10:05.380 DEBUG:boto:Method: HEAD 2022-09-12T20:10:05.381 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:10:05.381 DEBUG:boto:Data: 2022-09-12T20:10:05.382 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.382 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.383 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.383 DEBUG:boto:Params: {} 2022-09-12T20:10:05.384 DEBUG:boto:Token: None 2022-09-12T20:10:05.384 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:10:05.385 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GxCyHlRxB3PAdsViUcXoerMxspE= 2022-09-12T20:10:05.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GxCyHlRxB3PAdsViUcXoerMxspE=', 'Content-Length': '0'} 2022-09-12T20:10:05.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e09a4623de85cd8-00631f921d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.388 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:10:05.389 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:10:05.389 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:10:05.390 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:10:05.390 DEBUG:boto:Method: HEAD 2022-09-12T20:10:05.391 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:10:05.391 DEBUG:boto:Data: 2022-09-12T20:10:05.392 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.392 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.393 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.393 DEBUG:boto:Params: {} 2022-09-12T20:10:05.394 DEBUG:boto:Token: None 2022-09-12T20:10:05.394 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:10:05.395 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GxCyHlRxB3PAdsViUcXoerMxspE= 2022-09-12T20:10:05.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GxCyHlRxB3PAdsViUcXoerMxspE=', 'Content-Length': '0'} 2022-09-12T20:10:05.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b108a420187e9407-00631f921d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:10:05.399 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:10:05.399 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:10:05.400 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:10:05.400 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:10:05.401 DEBUG:boto:Method: GET 2022-09-12T20:10:05.401 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:10:05.402 DEBUG:boto:Data: 2022-09-12T20:10:05.402 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.403 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.403 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.404 DEBUG:boto:Params: {} 2022-09-12T20:10:05.404 DEBUG:boto:Token: None 2022-09-12T20:10:05.405 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:10:05.406 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GfJBFU/FSEoBfH9ZHH3TkDQd/PQ= 2022-09-12T20:10:05.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GfJBFU/FSEoBfH9ZHH3TkDQd/PQ=', 'Content-Length': '0'} 2022-09-12T20:10:05.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9473aabe1a0bb23-00631f921d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.409 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:10:05.410 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:10:05.410 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:10:05.411 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:10:05.411 DEBUG:boto:Method: GET 2022-09-12T20:10:05.412 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:10:05.412 DEBUG:boto:Data: 2022-09-12T20:10:05.413 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.413 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.414 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.414 DEBUG:boto:Params: {} 2022-09-12T20:10:05.415 DEBUG:boto:Token: None 2022-09-12T20:10:05.415 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:10:05.416 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GfJBFU/FSEoBfH9ZHH3TkDQd/PQ= 2022-09-12T20:10:05.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GfJBFU/FSEoBfH9ZHH3TkDQd/PQ=', 'Content-Length': '0'} 2022-09-12T20:10:05.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff0a1b9d65116bc4-00631f921d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.420 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:05.420 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:05.421 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:10:05.421 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:10:05.422 DEBUG:boto:Method: GET 2022-09-12T20:10:05.422 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:10:05.423 DEBUG:boto:Data: 2022-09-12T20:10:05.423 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.424 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.424 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.425 DEBUG:boto:Params: {} 2022-09-12T20:10:05.425 DEBUG:boto:Token: None 2022-09-12T20:10:05.426 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/ 2022-09-12T20:10:05.426 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc= 2022-09-12T20:10:05.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc=', 'Content-Length': '0'} 2022-09-12T20:10:05.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0f84f1eb8463d07-00631f921d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.431 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:05.431 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:05.432 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:05.432 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:10:05.433 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:10:05.433 DEBUG:boto:Method: GET 2022-09-12T20:10:05.434 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:10:05.435 DEBUG:boto:Data: 2022-09-12T20:10:05.435 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.436 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.436 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.437 DEBUG:boto:Params: {} 2022-09-12T20:10:05.437 DEBUG:boto:Token: None 2022-09-12T20:10:05.438 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/ 2022-09-12T20:10:05.438 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc= 2022-09-12T20:10:05.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc=', 'Content-Length': '0'} 2022-09-12T20:10:05.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004719ce40c201efab-00631f921d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.442 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:05.451 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:10:05.451 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:10:05.452 DEBUG:boto:Method: HEAD 2022-09-12T20:10:05.452 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:10:05.453 DEBUG:boto:Data: 2022-09-12T20:10:05.453 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.454 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.454 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.455 DEBUG:boto:Params: {} 2022-09-12T20:10:05.455 DEBUG:boto:Token: None 2022-09-12T20:10:05.456 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj23 2022-09-12T20:10:05.457 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tchKwxXHRV9VNfZVBr2D6lVhNg4= 2022-09-12T20:10:05.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tchKwxXHRV9VNfZVBr2D6lVhNg4=', 'Content-Length': '0'} 2022-09-12T20:10:05.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c76fe713699b2332-00631f921d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.460 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:10:05.461 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:10:05.461 DEBUG:boto:Method: HEAD 2022-09-12T20:10:05.462 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:10:05.462 DEBUG:boto:Data: 2022-09-12T20:10:05.463 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.463 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.464 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.464 DEBUG:boto:Params: {} 2022-09-12T20:10:05.465 DEBUG:boto:Token: None 2022-09-12T20:10:05.465 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj23 2022-09-12T20:10:05.466 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tchKwxXHRV9VNfZVBr2D6lVhNg4= 2022-09-12T20:10:05.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tchKwxXHRV9VNfZVBr2D6lVhNg4=', 'Content-Length': '0'} 2022-09-12T20:10:05.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afd33a2869d44abf-00631f921d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:10:05.469 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:10:05.470 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:10:05.470 DEBUG:boto:Method: GET 2022-09-12T20:10:05.471 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:10:05.471 DEBUG:boto:Data: 2022-09-12T20:10:05.472 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.472 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.473 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.473 DEBUG:boto:Params: {} 2022-09-12T20:10:05.474 DEBUG:boto:Token: None 2022-09-12T20:10:05.475 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj23 2022-09-12T20:10:05.475 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J0pGHDmAGXmZ09edJVChYF6yHfY= 2022-09-12T20:10:05.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J0pGHDmAGXmZ09edJVChYF6yHfY=', 'Content-Length': '0'} 2022-09-12T20:10:05.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b24f4d1e1f6ea41-00631f921d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.522 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:10:05.523 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:10:05.523 DEBUG:boto:Method: GET 2022-09-12T20:10:05.524 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:10:05.524 DEBUG:boto:Data: 2022-09-12T20:10:05.525 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.525 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.526 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.526 DEBUG:boto:Params: {} 2022-09-12T20:10:05.527 DEBUG:boto:Token: None 2022-09-12T20:10:05.528 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj23 2022-09-12T20:10:05.528 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J0pGHDmAGXmZ09edJVChYF6yHfY= 2022-09-12T20:10:05.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J0pGHDmAGXmZ09edJVChYF6yHfY=', 'Content-Length': '0'} 2022-09-12T20:10:05.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000941d9b9a77316da1-00631f921d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.532 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:05.532 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:05.533 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:10:05.533 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:10:05.534 DEBUG:boto:Method: GET 2022-09-12T20:10:05.535 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:10:05.535 DEBUG:boto:Data: 2022-09-12T20:10:05.536 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.536 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.537 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.537 DEBUG:boto:Params: {} 2022-09-12T20:10:05.538 DEBUG:boto:Token: None 2022-09-12T20:10:05.538 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/ 2022-09-12T20:10:05.539 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc= 2022-09-12T20:10:05.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc=', 'Content-Length': '0'} 2022-09-12T20:10:05.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e4c2b43fc2910da-00631f921d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.543 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:05.544 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:05.544 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:05.545 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:10:05.545 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:10:05.546 DEBUG:boto:Method: GET 2022-09-12T20:10:05.546 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:10:05.547 DEBUG:boto:Data: 2022-09-12T20:10:05.547 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.548 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.549 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.549 DEBUG:boto:Params: {} 2022-09-12T20:10:05.550 DEBUG:boto:Token: None 2022-09-12T20:10:05.550 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/ 2022-09-12T20:10:05.551 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc= 2022-09-12T20:10:05.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc=', 'Content-Length': '0'} 2022-09-12T20:10:05.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d37f4cff473669c9-00631f921d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.598 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:05.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:10:05.599 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:10:05.600 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:10:05.600 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:10:05.601 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:10:05.601 DEBUG:boto:Method: GET 2022-09-12T20:10:05.602 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:10:05.602 DEBUG:boto:Data: 2022-09-12T20:10:05.603 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.604 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.604 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.605 DEBUG:boto:Params: {} 2022-09-12T20:10:05.605 DEBUG:boto:Token: None 2022-09-12T20:10:05.606 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:10:05.607 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uI8V/Cvv+rMRtiXcBM9/8b2WGlE= 2022-09-12T20:10:05.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uI8V/Cvv+rMRtiXcBM9/8b2WGlE=', 'Content-Length': '0'} 2022-09-12T20:10:05.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049360f7fd3900b0b-00631f921d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.611 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:10:05.612 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:10:05.612 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:10:05.613 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:10:05.613 DEBUG:boto:Method: GET 2022-09-12T20:10:05.614 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:10:05.615 DEBUG:boto:Data: 2022-09-12T20:10:05.615 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.616 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.616 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.617 DEBUG:boto:Params: {} 2022-09-12T20:10:05.617 DEBUG:boto:Token: None 2022-09-12T20:10:05.618 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:10:05.619 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uI8V/Cvv+rMRtiXcBM9/8b2WGlE= 2022-09-12T20:10:05.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uI8V/Cvv+rMRtiXcBM9/8b2WGlE=', 'Content-Length': '0'} 2022-09-12T20:10:05.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070f5f3e482e829a9-00631f921d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.622 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:10:05.623 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:10:05.623 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:10:05.624 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:10:05.625 DEBUG:boto:Method: HEAD 2022-09-12T20:10:05.625 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:10:05.626 DEBUG:boto:Data: 2022-09-12T20:10:05.626 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.627 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.627 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.628 DEBUG:boto:Params: {} 2022-09-12T20:10:05.629 DEBUG:boto:Token: None 2022-09-12T20:10:05.629 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:10:05.630 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f6zyq8om5Bmnu3pWtQW4kZvCwCI= 2022-09-12T20:10:05.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f6zyq8om5Bmnu3pWtQW4kZvCwCI=', 'Content-Length': '0'} 2022-09-12T20:10:05.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0f070a40f272dad-00631f921d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.633 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:10:05.634 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:10:05.634 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:10:05.635 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:10:05.636 DEBUG:boto:Method: HEAD 2022-09-12T20:10:05.636 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:10:05.637 DEBUG:boto:Data: 2022-09-12T20:10:05.637 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.638 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.639 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.639 DEBUG:boto:Params: {} 2022-09-12T20:10:05.640 DEBUG:boto:Token: None 2022-09-12T20:10:05.641 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:10:05.641 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f6zyq8om5Bmnu3pWtQW4kZvCwCI= 2022-09-12T20:10:05.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f6zyq8om5Bmnu3pWtQW4kZvCwCI=', 'Content-Length': '0'} 2022-09-12T20:10:05.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ece56996df5c8fc7-00631f921d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:10:05.645 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:10:05.646 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:10:05.647 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:10:05.647 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:10:05.648 DEBUG:boto:Method: GET 2022-09-12T20:10:05.648 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:10:05.649 DEBUG:boto:Data: 2022-09-12T20:10:05.649 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.650 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.650 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.651 DEBUG:boto:Params: {} 2022-09-12T20:10:05.651 DEBUG:boto:Token: None 2022-09-12T20:10:05.652 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:10:05.653 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uI8V/Cvv+rMRtiXcBM9/8b2WGlE= 2022-09-12T20:10:05.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uI8V/Cvv+rMRtiXcBM9/8b2WGlE=', 'Content-Length': '0'} 2022-09-12T20:10:05.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db3496d1c664800a-00631f921d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.656 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:10:05.657 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:10:05.657 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:10:05.658 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:10:05.659 DEBUG:boto:Method: GET 2022-09-12T20:10:05.659 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:10:05.660 DEBUG:boto:Data: 2022-09-12T20:10:05.660 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.661 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.661 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.662 DEBUG:boto:Params: {} 2022-09-12T20:10:05.663 DEBUG:boto:Token: None 2022-09-12T20:10:05.663 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:10:05.664 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uI8V/Cvv+rMRtiXcBM9/8b2WGlE= 2022-09-12T20:10:05.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uI8V/Cvv+rMRtiXcBM9/8b2WGlE=', 'Content-Length': '0'} 2022-09-12T20:10:05.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b094e2c43b169af8-00631f921d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.668 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:05.668 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:05.669 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:10:05.669 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:10:05.670 DEBUG:boto:Method: GET 2022-09-12T20:10:05.671 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:10:05.671 DEBUG:boto:Data: 2022-09-12T20:10:05.672 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.672 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.673 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.673 DEBUG:boto:Params: {} 2022-09-12T20:10:05.674 DEBUG:boto:Token: None 2022-09-12T20:10:05.675 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/ 2022-09-12T20:10:05.675 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc= 2022-09-12T20:10:05.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc=', 'Content-Length': '0'} 2022-09-12T20:10:05.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6a218e13d995e35-00631f921d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.680 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:05.681 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:05.681 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:05.682 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:10:05.682 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:10:05.683 DEBUG:boto:Method: GET 2022-09-12T20:10:05.683 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:10:05.684 DEBUG:boto:Data: 2022-09-12T20:10:05.684 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.685 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.686 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.686 DEBUG:boto:Params: {} 2022-09-12T20:10:05.687 DEBUG:boto:Token: None 2022-09-12T20:10:05.687 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/ 2022-09-12T20:10:05.688 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc= 2022-09-12T20:10:05.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc=', 'Content-Length': '0'} 2022-09-12T20:10:05.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008810947350c41959-00631f921d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.692 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:05.693 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:10:05.693 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:10:05.694 DEBUG:boto:Method: HEAD 2022-09-12T20:10:05.694 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:10:05.695 DEBUG:boto:Data: 2022-09-12T20:10:05.695 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.696 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.696 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.697 DEBUG:boto:Params: {} 2022-09-12T20:10:05.697 DEBUG:boto:Token: None 2022-09-12T20:10:05.698 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj33 2022-09-12T20:10:05.699 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:w+hG13h1k1d9EScNn4yW50R7AD4= 2022-09-12T20:10:05.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:w+hG13h1k1d9EScNn4yW50R7AD4=', 'Content-Length': '0'} 2022-09-12T20:10:05.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e47cf407fd15172-00631f921d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.702 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:10:05.702 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:10:05.703 DEBUG:boto:Method: HEAD 2022-09-12T20:10:05.703 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:10:05.704 DEBUG:boto:Data: 2022-09-12T20:10:05.704 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.705 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.705 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.706 DEBUG:boto:Params: {} 2022-09-12T20:10:05.707 DEBUG:boto:Token: None 2022-09-12T20:10:05.707 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj33 2022-09-12T20:10:05.708 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:w+hG13h1k1d9EScNn4yW50R7AD4= 2022-09-12T20:10:05.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:w+hG13h1k1d9EScNn4yW50R7AD4=', 'Content-Length': '0'} 2022-09-12T20:10:05.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068558eb5f0c415df-00631f921d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:10:05.711 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:10:05.712 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:10:05.712 DEBUG:boto:Method: GET 2022-09-12T20:10:05.713 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:10:05.714 DEBUG:boto:Data: 2022-09-12T20:10:05.714 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.715 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.715 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.716 DEBUG:boto:Params: {} 2022-09-12T20:10:05.716 DEBUG:boto:Token: None 2022-09-12T20:10:05.717 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj33 2022-09-12T20:10:05.718 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BAVV8iBxtpHKu17N9E7iYj5Jce0= 2022-09-12T20:10:05.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BAVV8iBxtpHKu17N9E7iYj5Jce0=', 'Content-Length': '0'} 2022-09-12T20:10:05.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012ad8d6db58a809b-00631f921d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.721 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:10:05.722 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:10:05.722 DEBUG:boto:Method: GET 2022-09-12T20:10:05.723 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:10:05.723 DEBUG:boto:Data: 2022-09-12T20:10:05.724 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.725 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.725 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.726 DEBUG:boto:Params: {} 2022-09-12T20:10:05.726 DEBUG:boto:Token: None 2022-09-12T20:10:05.727 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj33 2022-09-12T20:10:05.728 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BAVV8iBxtpHKu17N9E7iYj5Jce0= 2022-09-12T20:10:05.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BAVV8iBxtpHKu17N9E7iYj5Jce0=', 'Content-Length': '0'} 2022-09-12T20:10:05.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffe124ff2d0748f5-00631f921d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.774 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:05.775 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:05.775 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:10:05.776 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:10:05.776 DEBUG:boto:Method: GET 2022-09-12T20:10:05.777 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:10:05.778 DEBUG:boto:Data: 2022-09-12T20:10:05.778 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.779 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.779 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.780 DEBUG:boto:Params: {} 2022-09-12T20:10:05.780 DEBUG:boto:Token: None 2022-09-12T20:10:05.781 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/ 2022-09-12T20:10:05.781 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc= 2022-09-12T20:10:05.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc=', 'Content-Length': '0'} 2022-09-12T20:10:05.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dad3ec47b32e95c3-00631f921d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.786 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:05.787 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:05.787 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:05.788 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:10:05.788 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:10:05.789 DEBUG:boto:Method: GET 2022-09-12T20:10:05.789 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:10:05.790 DEBUG:boto:Data: 2022-09-12T20:10:05.790 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.791 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.791 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.792 DEBUG:boto:Params: {} 2022-09-12T20:10:05.792 DEBUG:boto:Token: None 2022-09-12T20:10:05.793 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/ 2022-09-12T20:10:05.794 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc= 2022-09-12T20:10:05.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc=', 'Content-Length': '0'} 2022-09-12T20:10:05.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fb1cabe242685ea-00631f921d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.798 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:05.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:10:05.799 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:10:05.799 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:10:05.800 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:10:05.800 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:10:05.801 DEBUG:boto:Method: GET 2022-09-12T20:10:05.801 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:10:05.802 DEBUG:boto:Data: 2022-09-12T20:10:05.802 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.803 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.804 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.804 DEBUG:boto:Params: {} 2022-09-12T20:10:05.805 DEBUG:boto:Token: None 2022-09-12T20:10:05.805 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:10:05.806 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pIp/ow5n7xTkBBUD5ofYuYB1A9M= 2022-09-12T20:10:05.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pIp/ow5n7xTkBBUD5ofYuYB1A9M=', 'Content-Length': '0'} 2022-09-12T20:10:05.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000538ae0d537339a81-00631f921d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.809 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:10:05.810 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:10:05.810 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:10:05.811 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:10:05.811 DEBUG:boto:Method: GET 2022-09-12T20:10:05.812 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:10:05.813 DEBUG:boto:Data: 2022-09-12T20:10:05.813 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.814 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.814 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.815 DEBUG:boto:Params: {} 2022-09-12T20:10:05.815 DEBUG:boto:Token: None 2022-09-12T20:10:05.816 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:10:05.816 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pIp/ow5n7xTkBBUD5ofYuYB1A9M= 2022-09-12T20:10:05.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pIp/ow5n7xTkBBUD5ofYuYB1A9M=', 'Content-Length': '0'} 2022-09-12T20:10:05.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdf42f547cd14fab-00631f921d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.820 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:10:05.820 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:10:05.821 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:10:05.821 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:10:05.822 DEBUG:boto:Method: HEAD 2022-09-12T20:10:05.822 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:10:05.823 DEBUG:boto:Data: 2022-09-12T20:10:05.823 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.824 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.824 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.825 DEBUG:boto:Params: {} 2022-09-12T20:10:05.825 DEBUG:boto:Token: None 2022-09-12T20:10:05.826 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:10:05.826 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RZmus5tzhZ5NBc8JOPRMCibcp8g= 2022-09-12T20:10:05.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RZmus5tzhZ5NBc8JOPRMCibcp8g=', 'Content-Length': '0'} 2022-09-12T20:10:05.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae4615fbbd84cd25-00631f921d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.830 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:10:05.830 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:10:05.831 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:10:05.831 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:10:05.832 DEBUG:boto:Method: HEAD 2022-09-12T20:10:05.832 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:10:05.833 DEBUG:boto:Data: 2022-09-12T20:10:05.833 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.834 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.834 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.835 DEBUG:boto:Params: {} 2022-09-12T20:10:05.835 DEBUG:boto:Token: None 2022-09-12T20:10:05.836 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:10:05.836 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RZmus5tzhZ5NBc8JOPRMCibcp8g= 2022-09-12T20:10:05.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RZmus5tzhZ5NBc8JOPRMCibcp8g=', 'Content-Length': '0'} 2022-09-12T20:10:05.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b969beb2b529841e-00631f921d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:10:05.840 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:10:05.841 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:10:05.841 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:10:05.842 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:10:05.842 DEBUG:boto:Method: GET 2022-09-12T20:10:05.843 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:10:05.843 DEBUG:boto:Data: 2022-09-12T20:10:05.844 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.844 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.845 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.845 DEBUG:boto:Params: {} 2022-09-12T20:10:05.846 DEBUG:boto:Token: None 2022-09-12T20:10:05.847 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:10:05.847 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pIp/ow5n7xTkBBUD5ofYuYB1A9M= 2022-09-12T20:10:05.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pIp/ow5n7xTkBBUD5ofYuYB1A9M=', 'Content-Length': '0'} 2022-09-12T20:10:05.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d82b4f932640e95c-00631f921d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.851 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:10:05.851 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:10:05.852 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:10:05.852 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:10:05.853 DEBUG:boto:Method: GET 2022-09-12T20:10:05.853 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:10:05.854 DEBUG:boto:Data: 2022-09-12T20:10:05.854 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.855 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.855 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.856 DEBUG:boto:Params: {} 2022-09-12T20:10:05.856 DEBUG:boto:Token: None 2022-09-12T20:10:05.857 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:10:05.858 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pIp/ow5n7xTkBBUD5ofYuYB1A9M= 2022-09-12T20:10:05.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pIp/ow5n7xTkBBUD5ofYuYB1A9M=', 'Content-Length': '0'} 2022-09-12T20:10:05.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea79ffc4b5293ef2-00631f921d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.861 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:05.861 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:05.862 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:10:05.862 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:10:05.863 DEBUG:boto:Method: GET 2022-09-12T20:10:05.864 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:10:05.864 DEBUG:boto:Data: 2022-09-12T20:10:05.865 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.865 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.866 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.866 DEBUG:boto:Params: {} 2022-09-12T20:10:05.867 DEBUG:boto:Token: None 2022-09-12T20:10:05.867 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/ 2022-09-12T20:10:05.868 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc= 2022-09-12T20:10:05.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc=', 'Content-Length': '0'} 2022-09-12T20:10:05.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d599c86d701ba5f-00631f921d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.914 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:05.915 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:05.915 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:05.916 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:10:05.916 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:10:05.917 DEBUG:boto:Method: GET 2022-09-12T20:10:05.917 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:10:05.918 DEBUG:boto:Data: 2022-09-12T20:10:05.918 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.919 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.919 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.920 DEBUG:boto:Params: {} 2022-09-12T20:10:05.920 DEBUG:boto:Token: None 2022-09-12T20:10:05.921 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/ 2022-09-12T20:10:05.922 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc= 2022-09-12T20:10:05.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc=', 'Content-Length': '0'} 2022-09-12T20:10:05.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b4288218ce63472-00631f921d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.925 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:05.926 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:10:05.927 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:10:05.927 DEBUG:boto:Method: HEAD 2022-09-12T20:10:05.928 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:10:05.928 DEBUG:boto:Data: 2022-09-12T20:10:05.929 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.929 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.930 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.930 DEBUG:boto:Params: {} 2022-09-12T20:10:05.931 DEBUG:boto:Token: None 2022-09-12T20:10:05.931 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj34 2022-09-12T20:10:05.932 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l3Q866trSox0MSb+0cK9DlB8HEI= 2022-09-12T20:10:05.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l3Q866trSox0MSb+0cK9DlB8HEI=', 'Content-Length': '0'} 2022-09-12T20:10:05.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008915cb4ac7563b30-00631f921d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.935 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:10:05.936 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:10:05.936 DEBUG:boto:Method: HEAD 2022-09-12T20:10:05.937 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:10:05.937 DEBUG:boto:Data: 2022-09-12T20:10:05.938 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.938 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.939 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.939 DEBUG:boto:Params: {} 2022-09-12T20:10:05.940 DEBUG:boto:Token: None 2022-09-12T20:10:05.941 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj34 2022-09-12T20:10:05.941 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l3Q866trSox0MSb+0cK9DlB8HEI= 2022-09-12T20:10:05.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l3Q866trSox0MSb+0cK9DlB8HEI=', 'Content-Length': '0'} 2022-09-12T20:10:05.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2d576048d1bc323-00631f921d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:10:05.945 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:10:05.945 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:10:05.946 DEBUG:boto:Method: GET 2022-09-12T20:10:05.947 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:10:05.947 DEBUG:boto:Data: 2022-09-12T20:10:05.948 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.948 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.949 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.949 DEBUG:boto:Params: {} 2022-09-12T20:10:05.950 DEBUG:boto:Token: None 2022-09-12T20:10:05.950 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj34 2022-09-12T20:10:05.951 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ox4m0hZDYF3iE2x8hYpfWEjkWFs= 2022-09-12T20:10:05.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ox4m0hZDYF3iE2x8hYpfWEjkWFs=', 'Content-Length': '0'} 2022-09-12T20:10:05.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072a69f4567ac586f-00631f921d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.955 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:10:05.955 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:10:05.956 DEBUG:boto:Method: GET 2022-09-12T20:10:05.956 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:10:05.957 DEBUG:boto:Data: 2022-09-12T20:10:05.957 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.958 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:05.958 DEBUG:boto:Port: 8000 2022-09-12T20:10:05.959 DEBUG:boto:Params: {} 2022-09-12T20:10:05.959 DEBUG:boto:Token: None 2022-09-12T20:10:05.960 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/obj34 2022-09-12T20:10:05.961 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ox4m0hZDYF3iE2x8hYpfWEjkWFs= 2022-09-12T20:10:05.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ox4m0hZDYF3iE2x8hYpfWEjkWFs=', 'Content-Length': '0'} 2022-09-12T20:10:05.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013efec4a8944c322-00631f921d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:05.964 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:05.965 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:05.965 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:10:05.966 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:10:05.966 DEBUG:boto:Method: GET 2022-09-12T20:10:05.967 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:10:05.968 DEBUG:boto:Data: 2022-09-12T20:10:05.968 DEBUG:boto:Headers: {} 2022-09-12T20:10:05.969 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:05.969 DEBUG:boto:Port: 8001 2022-09-12T20:10:05.970 DEBUG:boto:Params: {} 2022-09-12T20:10:05.970 DEBUG:boto:Token: None 2022-09-12T20:10:05.971 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:05 GMT /jdpiak-75/ 2022-09-12T20:10:05.971 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc= 2022-09-12T20:10:05.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2dTMkrRwPKTZQxauCHwn6wa8vfc=', 'Content-Length': '0'} 2022-09-12T20:10:05.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f908a788ea3f5a2e-00631f921d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.018 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:06.019 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:06.019 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:06.020 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:10:06.020 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:10:06.021 DEBUG:boto:Method: GET 2022-09-12T20:10:06.021 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:10:06.022 DEBUG:boto:Data: 2022-09-12T20:10:06.023 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.023 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.024 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.024 DEBUG:boto:Params: {} 2022-09-12T20:10:06.025 DEBUG:boto:Token: None 2022-09-12T20:10:06.025 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/ 2022-09-12T20:10:06.026 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc= 2022-09-12T20:10:06.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc=', 'Content-Length': '0'} 2022-09-12T20:10:06.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000787d4a02f118b1b3-00631f921e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.030 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:06.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:10:06.031 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:10:06.032 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:10:06.032 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:10:06.033 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:10:06.033 DEBUG:boto:Method: GET 2022-09-12T20:10:06.034 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:10:06.034 DEBUG:boto:Data: 2022-09-12T20:10:06.035 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.035 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.036 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.036 DEBUG:boto:Params: {} 2022-09-12T20:10:06.037 DEBUG:boto:Token: None 2022-09-12T20:10:06.038 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:10:06.038 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5FKO+gf0glOX5UQPTPkNk+vZnl8= 2022-09-12T20:10:06.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5FKO+gf0glOX5UQPTPkNk+vZnl8=', 'Content-Length': '0'} 2022-09-12T20:10:06.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea40ef30f12ef995-00631f921e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.042 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:10:06.043 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:10:06.043 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:10:06.044 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:10:06.044 DEBUG:boto:Method: GET 2022-09-12T20:10:06.045 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:10:06.045 DEBUG:boto:Data: 2022-09-12T20:10:06.046 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.046 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.047 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.048 DEBUG:boto:Params: {} 2022-09-12T20:10:06.048 DEBUG:boto:Token: None 2022-09-12T20:10:06.049 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:10:06.049 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5FKO+gf0glOX5UQPTPkNk+vZnl8= 2022-09-12T20:10:06.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5FKO+gf0glOX5UQPTPkNk+vZnl8=', 'Content-Length': '0'} 2022-09-12T20:10:06.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000632a323941e75d6f-00631f921e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.053 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:10:06.053 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:10:06.054 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:10:06.054 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:10:06.055 DEBUG:boto:Method: HEAD 2022-09-12T20:10:06.055 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:10:06.056 DEBUG:boto:Data: 2022-09-12T20:10:06.056 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.057 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.057 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.058 DEBUG:boto:Params: {} 2022-09-12T20:10:06.058 DEBUG:boto:Token: None 2022-09-12T20:10:06.059 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:10:06.060 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z7F5h4jefjPuk3nrnZPDru88jBQ= 2022-09-12T20:10:06.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z7F5h4jefjPuk3nrnZPDru88jBQ=', 'Content-Length': '0'} 2022-09-12T20:10:06.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038c7e3d3b35e14d2-00631f921e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.063 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:10:06.064 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:10:06.064 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:10:06.065 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:10:06.065 DEBUG:boto:Method: HEAD 2022-09-12T20:10:06.066 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:10:06.066 DEBUG:boto:Data: 2022-09-12T20:10:06.067 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.067 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.068 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.068 DEBUG:boto:Params: {} 2022-09-12T20:10:06.069 DEBUG:boto:Token: None 2022-09-12T20:10:06.069 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:10:06.070 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z7F5h4jefjPuk3nrnZPDru88jBQ= 2022-09-12T20:10:06.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z7F5h4jefjPuk3nrnZPDru88jBQ=', 'Content-Length': '0'} 2022-09-12T20:10:06.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd83a416e508a75a-00631f921e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:10:06.074 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:10:06.074 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:10:06.075 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:10:06.075 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:10:06.076 DEBUG:boto:Method: GET 2022-09-12T20:10:06.077 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:10:06.077 DEBUG:boto:Data: 2022-09-12T20:10:06.078 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.078 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.079 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.079 DEBUG:boto:Params: {} 2022-09-12T20:10:06.080 DEBUG:boto:Token: None 2022-09-12T20:10:06.080 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:10:06.081 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5FKO+gf0glOX5UQPTPkNk+vZnl8= 2022-09-12T20:10:06.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5FKO+gf0glOX5UQPTPkNk+vZnl8=', 'Content-Length': '0'} 2022-09-12T20:10:06.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007aa3362877e6273e-00631f921e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.084 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:10:06.085 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:10:06.086 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:10:06.086 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:10:06.087 DEBUG:boto:Method: GET 2022-09-12T20:10:06.087 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:10:06.088 DEBUG:boto:Data: 2022-09-12T20:10:06.088 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.089 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.089 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.090 DEBUG:boto:Params: {} 2022-09-12T20:10:06.091 DEBUG:boto:Token: None 2022-09-12T20:10:06.091 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:10:06.092 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5FKO+gf0glOX5UQPTPkNk+vZnl8= 2022-09-12T20:10:06.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5FKO+gf0glOX5UQPTPkNk+vZnl8=', 'Content-Length': '0'} 2022-09-12T20:10:06.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d08ea045c5418ca6-00631f921e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.095 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:06.096 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:06.096 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:10:06.097 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:10:06.098 DEBUG:boto:Method: GET 2022-09-12T20:10:06.098 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:10:06.099 DEBUG:boto:Data: 2022-09-12T20:10:06.099 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.100 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.100 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.101 DEBUG:boto:Params: {} 2022-09-12T20:10:06.101 DEBUG:boto:Token: None 2022-09-12T20:10:06.102 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/ 2022-09-12T20:10:06.102 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc= 2022-09-12T20:10:06.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc=', 'Content-Length': '0'} 2022-09-12T20:10:06.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fb9755352a76fe4-00631f921e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.106 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:06.107 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:06.108 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:06.108 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:10:06.109 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:10:06.109 DEBUG:boto:Method: GET 2022-09-12T20:10:06.110 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:10:06.111 DEBUG:boto:Data: 2022-09-12T20:10:06.111 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.112 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.112 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.113 DEBUG:boto:Params: {} 2022-09-12T20:10:06.113 DEBUG:boto:Token: None 2022-09-12T20:10:06.114 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/ 2022-09-12T20:10:06.115 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc= 2022-09-12T20:10:06.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc=', 'Content-Length': '0'} 2022-09-12T20:10:06.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000708ee095263ae5c4-00631f921e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.158 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:06.159 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:10:06.159 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:10:06.160 DEBUG:boto:Method: HEAD 2022-09-12T20:10:06.161 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:10:06.161 DEBUG:boto:Data: 2022-09-12T20:10:06.162 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.162 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.163 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.163 DEBUG:boto:Params: {} 2022-09-12T20:10:06.164 DEBUG:boto:Token: None 2022-09-12T20:10:06.164 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj38 2022-09-12T20:10:06.165 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DXpgWMtP0ufAJotNX9sLmoh9fVo= 2022-09-12T20:10:06.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DXpgWMtP0ufAJotNX9sLmoh9fVo=', 'Content-Length': '0'} 2022-09-12T20:10:06.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7624e7590943be1-00631f921e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.168 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:10:06.169 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:10:06.169 DEBUG:boto:Method: HEAD 2022-09-12T20:10:06.170 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:10:06.171 DEBUG:boto:Data: 2022-09-12T20:10:06.171 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.172 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.172 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.173 DEBUG:boto:Params: {} 2022-09-12T20:10:06.173 DEBUG:boto:Token: None 2022-09-12T20:10:06.174 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj38 2022-09-12T20:10:06.174 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DXpgWMtP0ufAJotNX9sLmoh9fVo= 2022-09-12T20:10:06.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DXpgWMtP0ufAJotNX9sLmoh9fVo=', 'Content-Length': '0'} 2022-09-12T20:10:06.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007df16ecdf37f3af9-00631f921e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:10:06.178 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:10:06.179 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:10:06.179 DEBUG:boto:Method: GET 2022-09-12T20:10:06.180 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:10:06.180 DEBUG:boto:Data: 2022-09-12T20:10:06.181 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.181 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.182 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.183 DEBUG:boto:Params: {} 2022-09-12T20:10:06.183 DEBUG:boto:Token: None 2022-09-12T20:10:06.184 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj38 2022-09-12T20:10:06.184 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:efH9nuarzff/lUp6yEMJO+p87yQ= 2022-09-12T20:10:06.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:efH9nuarzff/lUp6yEMJO+p87yQ=', 'Content-Length': '0'} 2022-09-12T20:10:06.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ecd206c1ee4af2c-00631f921e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.188 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:10:06.189 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:10:06.189 DEBUG:boto:Method: GET 2022-09-12T20:10:06.190 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:10:06.190 DEBUG:boto:Data: 2022-09-12T20:10:06.191 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.191 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.192 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.192 DEBUG:boto:Params: {} 2022-09-12T20:10:06.193 DEBUG:boto:Token: None 2022-09-12T20:10:06.194 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj38 2022-09-12T20:10:06.194 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:efH9nuarzff/lUp6yEMJO+p87yQ= 2022-09-12T20:10:06.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:efH9nuarzff/lUp6yEMJO+p87yQ=', 'Content-Length': '0'} 2022-09-12T20:10:06.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b59aa3e3a09391ba-00631f921e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.198 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:06.198 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:06.199 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:10:06.199 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:10:06.200 DEBUG:boto:Method: GET 2022-09-12T20:10:06.201 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:10:06.201 DEBUG:boto:Data: 2022-09-12T20:10:06.202 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.202 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.203 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.203 DEBUG:boto:Params: {} 2022-09-12T20:10:06.204 DEBUG:boto:Token: None 2022-09-12T20:10:06.204 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/ 2022-09-12T20:10:06.205 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc= 2022-09-12T20:10:06.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc=', 'Content-Length': '0'} 2022-09-12T20:10:06.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bc56007c733a183-00631f921e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.209 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:06.210 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:06.211 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:06.211 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:10:06.212 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:10:06.212 DEBUG:boto:Method: GET 2022-09-12T20:10:06.213 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:10:06.213 DEBUG:boto:Data: 2022-09-12T20:10:06.214 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.214 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.215 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.216 DEBUG:boto:Params: {} 2022-09-12T20:10:06.216 DEBUG:boto:Token: None 2022-09-12T20:10:06.217 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/ 2022-09-12T20:10:06.217 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc= 2022-09-12T20:10:06.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc=', 'Content-Length': '0'} 2022-09-12T20:10:06.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000148a3c4e2cd1af43-00631f921e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.221 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:06.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:10:06.222 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:10:06.223 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:10:06.223 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:10:06.224 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:10:06.225 DEBUG:boto:Method: GET 2022-09-12T20:10:06.225 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:10:06.226 DEBUG:boto:Data: 2022-09-12T20:10:06.226 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.227 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.227 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.228 DEBUG:boto:Params: {} 2022-09-12T20:10:06.228 DEBUG:boto:Token: None 2022-09-12T20:10:06.229 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:10:06.229 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Oesfo8qeB3bmQ5XdVLQlMX5nK+8= 2022-09-12T20:10:06.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Oesfo8qeB3bmQ5XdVLQlMX5nK+8=', 'Content-Length': '0'} 2022-09-12T20:10:06.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dec1e68e1d0b05ac-00631f921e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.233 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:10:06.234 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:10:06.234 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:10:06.235 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:10:06.235 DEBUG:boto:Method: GET 2022-09-12T20:10:06.236 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:10:06.236 DEBUG:boto:Data: 2022-09-12T20:10:06.237 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.237 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.238 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.238 DEBUG:boto:Params: {} 2022-09-12T20:10:06.239 DEBUG:boto:Token: None 2022-09-12T20:10:06.240 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:10:06.240 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Oesfo8qeB3bmQ5XdVLQlMX5nK+8= 2022-09-12T20:10:06.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Oesfo8qeB3bmQ5XdVLQlMX5nK+8=', 'Content-Length': '0'} 2022-09-12T20:10:06.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098a3ed8981f26e5b-00631f921e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.244 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:10:06.245 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:10:06.245 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:10:06.246 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:10:06.246 DEBUG:boto:Method: HEAD 2022-09-12T20:10:06.247 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:10:06.247 DEBUG:boto:Data: 2022-09-12T20:10:06.248 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.248 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.249 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.249 DEBUG:boto:Params: {} 2022-09-12T20:10:06.250 DEBUG:boto:Token: None 2022-09-12T20:10:06.250 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:10:06.251 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:907qIIP+h4lPZghPQf296uAzoJI= 2022-09-12T20:10:06.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:907qIIP+h4lPZghPQf296uAzoJI=', 'Content-Length': '0'} 2022-09-12T20:10:06.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003973af5e4e2ecb0b-00631f921e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.254 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:10:06.255 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:10:06.255 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:10:06.256 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:10:06.256 DEBUG:boto:Method: HEAD 2022-09-12T20:10:06.257 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:10:06.257 DEBUG:boto:Data: 2022-09-12T20:10:06.258 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.258 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.259 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.259 DEBUG:boto:Params: {} 2022-09-12T20:10:06.260 DEBUG:boto:Token: None 2022-09-12T20:10:06.260 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:10:06.261 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:907qIIP+h4lPZghPQf296uAzoJI= 2022-09-12T20:10:06.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:907qIIP+h4lPZghPQf296uAzoJI=', 'Content-Length': '0'} 2022-09-12T20:10:06.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b700b99445551da-00631f921e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:10:06.265 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:10:06.266 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:10:06.266 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:10:06.267 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:10:06.267 DEBUG:boto:Method: GET 2022-09-12T20:10:06.268 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:10:06.268 DEBUG:boto:Data: 2022-09-12T20:10:06.269 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.269 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.270 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.271 DEBUG:boto:Params: {} 2022-09-12T20:10:06.271 DEBUG:boto:Token: None 2022-09-12T20:10:06.272 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:10:06.272 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Oesfo8qeB3bmQ5XdVLQlMX5nK+8= 2022-09-12T20:10:06.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Oesfo8qeB3bmQ5XdVLQlMX5nK+8=', 'Content-Length': '0'} 2022-09-12T20:10:06.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e38ec9dc5895a924-00631f921e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.276 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:10:06.276 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:10:06.277 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:10:06.277 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:10:06.278 DEBUG:boto:Method: GET 2022-09-12T20:10:06.278 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:10:06.279 DEBUG:boto:Data: 2022-09-12T20:10:06.280 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.280 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.281 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.281 DEBUG:boto:Params: {} 2022-09-12T20:10:06.282 DEBUG:boto:Token: None 2022-09-12T20:10:06.282 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:10:06.283 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Oesfo8qeB3bmQ5XdVLQlMX5nK+8= 2022-09-12T20:10:06.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Oesfo8qeB3bmQ5XdVLQlMX5nK+8=', 'Content-Length': '0'} 2022-09-12T20:10:06.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017160a9e73ce86a5-00631f921e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.286 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:06.287 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:06.288 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:10:06.288 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:10:06.289 DEBUG:boto:Method: GET 2022-09-12T20:10:06.289 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:10:06.290 DEBUG:boto:Data: 2022-09-12T20:10:06.290 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.291 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.291 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.292 DEBUG:boto:Params: {} 2022-09-12T20:10:06.292 DEBUG:boto:Token: None 2022-09-12T20:10:06.293 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/ 2022-09-12T20:10:06.294 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc= 2022-09-12T20:10:06.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc=', 'Content-Length': '0'} 2022-09-12T20:10:06.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000161d3c1bda360c6d-00631f921e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.298 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:06.298 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:06.299 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:06.299 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:10:06.300 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:10:06.300 DEBUG:boto:Method: GET 2022-09-12T20:10:06.301 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:10:06.301 DEBUG:boto:Data: 2022-09-12T20:10:06.302 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.302 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.303 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.303 DEBUG:boto:Params: {} 2022-09-12T20:10:06.304 DEBUG:boto:Token: None 2022-09-12T20:10:06.305 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/ 2022-09-12T20:10:06.305 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc= 2022-09-12T20:10:06.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc=', 'Content-Length': '0'} 2022-09-12T20:10:06.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001a18713a0d04975-00631f921e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.309 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:06.310 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:10:06.311 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:10:06.311 DEBUG:boto:Method: HEAD 2022-09-12T20:10:06.312 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:10:06.312 DEBUG:boto:Data: 2022-09-12T20:10:06.313 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.313 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.314 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.314 DEBUG:boto:Params: {} 2022-09-12T20:10:06.315 DEBUG:boto:Token: None 2022-09-12T20:10:06.315 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj44 2022-09-12T20:10:06.316 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G7N0FYGs1zkY9+YA++J1sZw4ycY= 2022-09-12T20:10:06.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G7N0FYGs1zkY9+YA++J1sZw4ycY=', 'Content-Length': '0'} 2022-09-12T20:10:06.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd8c9d333f1aa199-00631f921e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.319 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:10:06.320 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:10:06.320 DEBUG:boto:Method: HEAD 2022-09-12T20:10:06.321 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:10:06.321 DEBUG:boto:Data: 2022-09-12T20:10:06.322 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.322 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.323 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.323 DEBUG:boto:Params: {} 2022-09-12T20:10:06.324 DEBUG:boto:Token: None 2022-09-12T20:10:06.324 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj44 2022-09-12T20:10:06.325 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G7N0FYGs1zkY9+YA++J1sZw4ycY= 2022-09-12T20:10:06.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G7N0FYGs1zkY9+YA++J1sZw4ycY=', 'Content-Length': '0'} 2022-09-12T20:10:06.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc6f0ffbcd4b6151-00631f921e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:10:06.329 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:10:06.329 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:10:06.330 DEBUG:boto:Method: GET 2022-09-12T20:10:06.330 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:10:06.331 DEBUG:boto:Data: 2022-09-12T20:10:06.331 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.332 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.342 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.342 DEBUG:boto:Params: {} 2022-09-12T20:10:06.343 DEBUG:boto:Token: None 2022-09-12T20:10:06.343 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj44 2022-09-12T20:10:06.344 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7Qwcwqkj2nbgnOqexMKjtMxIkUg= 2022-09-12T20:10:06.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7Qwcwqkj2nbgnOqexMKjtMxIkUg=', 'Content-Length': '0'} 2022-09-12T20:10:06.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093b09df291e322a1-00631f921e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.390 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:10:06.390 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:10:06.391 DEBUG:boto:Method: GET 2022-09-12T20:10:06.392 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:10:06.392 DEBUG:boto:Data: 2022-09-12T20:10:06.393 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.393 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.394 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.394 DEBUG:boto:Params: {} 2022-09-12T20:10:06.395 DEBUG:boto:Token: None 2022-09-12T20:10:06.395 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj44 2022-09-12T20:10:06.396 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7Qwcwqkj2nbgnOqexMKjtMxIkUg= 2022-09-12T20:10:06.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7Qwcwqkj2nbgnOqexMKjtMxIkUg=', 'Content-Length': '0'} 2022-09-12T20:10:06.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e98586e4d4c89a9-00631f921e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.399 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:06.400 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:06.400 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:10:06.401 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:10:06.402 DEBUG:boto:Method: GET 2022-09-12T20:10:06.402 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:10:06.403 DEBUG:boto:Data: 2022-09-12T20:10:06.403 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.404 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.404 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.405 DEBUG:boto:Params: {} 2022-09-12T20:10:06.405 DEBUG:boto:Token: None 2022-09-12T20:10:06.406 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/ 2022-09-12T20:10:06.406 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc= 2022-09-12T20:10:06.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc=', 'Content-Length': '0'} 2022-09-12T20:10:06.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aab140ba0b5ea00a-00631f921e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.411 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:06.411 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:06.412 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:06.412 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:10:06.413 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:10:06.413 DEBUG:boto:Method: GET 2022-09-12T20:10:06.414 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:10:06.414 DEBUG:boto:Data: 2022-09-12T20:10:06.415 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.415 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.416 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.416 DEBUG:boto:Params: {} 2022-09-12T20:10:06.417 DEBUG:boto:Token: None 2022-09-12T20:10:06.418 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/ 2022-09-12T20:10:06.418 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc= 2022-09-12T20:10:06.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc=', 'Content-Length': '0'} 2022-09-12T20:10:06.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba832028035e9010-00631f921e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.462 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:06.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:10:06.463 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:10:06.464 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:10:06.464 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:10:06.465 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:10:06.465 DEBUG:boto:Method: GET 2022-09-12T20:10:06.466 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:10:06.466 DEBUG:boto:Data: 2022-09-12T20:10:06.467 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.467 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.468 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.469 DEBUG:boto:Params: {} 2022-09-12T20:10:06.469 DEBUG:boto:Token: None 2022-09-12T20:10:06.470 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:10:06.470 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VkOzu1tAdQItMprElXHakwBkjxY= 2022-09-12T20:10:06.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VkOzu1tAdQItMprElXHakwBkjxY=', 'Content-Length': '0'} 2022-09-12T20:10:06.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000644b2cf775bf5529-00631f921e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.474 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:10:06.474 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:10:06.475 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:10:06.475 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:10:06.476 DEBUG:boto:Method: GET 2022-09-12T20:10:06.476 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:10:06.477 DEBUG:boto:Data: 2022-09-12T20:10:06.477 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.478 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.479 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.479 DEBUG:boto:Params: {} 2022-09-12T20:10:06.480 DEBUG:boto:Token: None 2022-09-12T20:10:06.480 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:10:06.481 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VkOzu1tAdQItMprElXHakwBkjxY= 2022-09-12T20:10:06.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VkOzu1tAdQItMprElXHakwBkjxY=', 'Content-Length': '0'} 2022-09-12T20:10:06.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffdcd6a977b2a214-00631f921e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.484 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:10:06.484 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:10:06.485 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:10:06.485 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:10:06.486 DEBUG:boto:Method: HEAD 2022-09-12T20:10:06.486 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:10:06.487 DEBUG:boto:Data: 2022-09-12T20:10:06.487 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.488 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.488 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.489 DEBUG:boto:Params: {} 2022-09-12T20:10:06.489 DEBUG:boto:Token: None 2022-09-12T20:10:06.490 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:10:06.491 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YFVicEzXx5KnzL/1GbcDXF9N2Vg= 2022-09-12T20:10:06.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YFVicEzXx5KnzL/1GbcDXF9N2Vg=', 'Content-Length': '0'} 2022-09-12T20:10:06.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcf461a1ac9cedd6-00631f921e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.494 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:10:06.494 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:10:06.495 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:10:06.495 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:10:06.496 DEBUG:boto:Method: HEAD 2022-09-12T20:10:06.496 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:10:06.497 DEBUG:boto:Data: 2022-09-12T20:10:06.497 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.498 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.498 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.499 DEBUG:boto:Params: {} 2022-09-12T20:10:06.499 DEBUG:boto:Token: None 2022-09-12T20:10:06.500 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:10:06.500 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YFVicEzXx5KnzL/1GbcDXF9N2Vg= 2022-09-12T20:10:06.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YFVicEzXx5KnzL/1GbcDXF9N2Vg=', 'Content-Length': '0'} 2022-09-12T20:10:06.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afd0e1cc35ef6a7f-00631f921e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:10:06.504 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:10:06.505 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:10:06.505 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:10:06.506 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:10:06.506 DEBUG:boto:Method: GET 2022-09-12T20:10:06.507 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:10:06.507 DEBUG:boto:Data: 2022-09-12T20:10:06.508 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.508 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.509 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.509 DEBUG:boto:Params: {} 2022-09-12T20:10:06.510 DEBUG:boto:Token: None 2022-09-12T20:10:06.510 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:10:06.511 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VkOzu1tAdQItMprElXHakwBkjxY= 2022-09-12T20:10:06.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VkOzu1tAdQItMprElXHakwBkjxY=', 'Content-Length': '0'} 2022-09-12T20:10:06.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051b722df55c9cd85-00631f921e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.515 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:10:06.515 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:10:06.516 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:10:06.516 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:10:06.517 DEBUG:boto:Method: GET 2022-09-12T20:10:06.517 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:10:06.518 DEBUG:boto:Data: 2022-09-12T20:10:06.519 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.520 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.521 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.522 DEBUG:boto:Params: {} 2022-09-12T20:10:06.522 DEBUG:boto:Token: None 2022-09-12T20:10:06.523 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:10:06.524 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VkOzu1tAdQItMprElXHakwBkjxY= 2022-09-12T20:10:06.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VkOzu1tAdQItMprElXHakwBkjxY=', 'Content-Length': '0'} 2022-09-12T20:10:06.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a84cc5ca6f6a85db-00631f921e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.527 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:06.528 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:06.528 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:10:06.529 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:10:06.529 DEBUG:boto:Method: GET 2022-09-12T20:10:06.530 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:10:06.530 DEBUG:boto:Data: 2022-09-12T20:10:06.531 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.532 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.532 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.533 DEBUG:boto:Params: {} 2022-09-12T20:10:06.534 DEBUG:boto:Token: None 2022-09-12T20:10:06.534 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/ 2022-09-12T20:10:06.535 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc= 2022-09-12T20:10:06.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc=', 'Content-Length': '0'} 2022-09-12T20:10:06.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d35b45e14d2800f8-00631f921e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.539 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:06.540 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:06.540 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:06.541 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:10:06.541 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:10:06.542 DEBUG:boto:Method: GET 2022-09-12T20:10:06.543 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:10:06.543 DEBUG:boto:Data: 2022-09-12T20:10:06.544 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.544 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.545 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.546 DEBUG:boto:Params: {} 2022-09-12T20:10:06.546 DEBUG:boto:Token: None 2022-09-12T20:10:06.547 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/ 2022-09-12T20:10:06.547 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc= 2022-09-12T20:10:06.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc=', 'Content-Length': '0'} 2022-09-12T20:10:06.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b61e1e4a0e4eff0-00631f921e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.594 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:06.595 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:10:06.595 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:10:06.596 DEBUG:boto:Method: HEAD 2022-09-12T20:10:06.596 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:10:06.597 DEBUG:boto:Data: 2022-09-12T20:10:06.597 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.598 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.598 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.599 DEBUG:boto:Params: {} 2022-09-12T20:10:06.599 DEBUG:boto:Token: None 2022-09-12T20:10:06.600 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj46 2022-09-12T20:10:06.600 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Mij30TdLwZFemnfWFqLIbRS7pVM= 2022-09-12T20:10:06.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Mij30TdLwZFemnfWFqLIbRS7pVM=', 'Content-Length': '0'} 2022-09-12T20:10:06.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db68e72c1aa209e9-00631f921e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.605 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:10:06.606 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:10:06.607 DEBUG:boto:Method: HEAD 2022-09-12T20:10:06.608 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:10:06.609 DEBUG:boto:Data: 2022-09-12T20:10:06.610 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.611 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.611 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.612 DEBUG:boto:Params: {} 2022-09-12T20:10:06.613 DEBUG:boto:Token: None 2022-09-12T20:10:06.614 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj46 2022-09-12T20:10:06.615 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Mij30TdLwZFemnfWFqLIbRS7pVM= 2022-09-12T20:10:06.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Mij30TdLwZFemnfWFqLIbRS7pVM=', 'Content-Length': '0'} 2022-09-12T20:10:06.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000699f366780de6500-00631f921e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:10:06.621 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:10:06.622 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:10:06.623 DEBUG:boto:Method: GET 2022-09-12T20:10:06.624 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:10:06.625 DEBUG:boto:Data: 2022-09-12T20:10:06.626 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.627 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.627 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.628 DEBUG:boto:Params: {} 2022-09-12T20:10:06.628 DEBUG:boto:Token: None 2022-09-12T20:10:06.629 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj46 2022-09-12T20:10:06.630 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9dAE9biSZFm7dI2Ej9xvHVC2K1o= 2022-09-12T20:10:06.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9dAE9biSZFm7dI2Ej9xvHVC2K1o=', 'Content-Length': '0'} 2022-09-12T20:10:06.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e17214a68ba15bb-00631f921e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.633 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:10:06.633 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:10:06.634 DEBUG:boto:Method: GET 2022-09-12T20:10:06.634 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:10:06.635 DEBUG:boto:Data: 2022-09-12T20:10:06.636 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.636 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.637 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.637 DEBUG:boto:Params: {} 2022-09-12T20:10:06.638 DEBUG:boto:Token: None 2022-09-12T20:10:06.638 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj46 2022-09-12T20:10:06.639 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9dAE9biSZFm7dI2Ej9xvHVC2K1o= 2022-09-12T20:10:06.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9dAE9biSZFm7dI2Ej9xvHVC2K1o=', 'Content-Length': '0'} 2022-09-12T20:10:06.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f04207d13d40b6e-00631f921e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.642 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:06.643 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:06.643 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:10:06.644 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:10:06.644 DEBUG:boto:Method: GET 2022-09-12T20:10:06.645 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:10:06.645 DEBUG:boto:Data: 2022-09-12T20:10:06.646 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.646 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.647 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.647 DEBUG:boto:Params: {} 2022-09-12T20:10:06.648 DEBUG:boto:Token: None 2022-09-12T20:10:06.649 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/ 2022-09-12T20:10:06.649 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc= 2022-09-12T20:10:06.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc=', 'Content-Length': '0'} 2022-09-12T20:10:06.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c1f0fce06115a0a-00631f921e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.653 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:06.654 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:06.654 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:06.655 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:10:06.656 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:10:06.656 DEBUG:boto:Method: GET 2022-09-12T20:10:06.657 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:10:06.657 DEBUG:boto:Data: 2022-09-12T20:10:06.658 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.658 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.659 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.659 DEBUG:boto:Params: {} 2022-09-12T20:10:06.660 DEBUG:boto:Token: None 2022-09-12T20:10:06.660 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/ 2022-09-12T20:10:06.661 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc= 2022-09-12T20:10:06.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc=', 'Content-Length': '0'} 2022-09-12T20:10:06.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049e15cc3ad9e2d63-00631f921e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.665 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:06.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:10:06.666 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:10:06.667 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:10:06.668 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:10:06.668 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:10:06.669 DEBUG:boto:Method: GET 2022-09-12T20:10:06.669 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:10:06.670 DEBUG:boto:Data: 2022-09-12T20:10:06.670 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.671 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.672 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.672 DEBUG:boto:Params: {} 2022-09-12T20:10:06.673 DEBUG:boto:Token: None 2022-09-12T20:10:06.673 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:10:06.674 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ix7iP+IDpG2vFO1UAdYvphMdviw= 2022-09-12T20:10:06.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ix7iP+IDpG2vFO1UAdYvphMdviw=', 'Content-Length': '0'} 2022-09-12T20:10:06.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa68246e2cc2e66e-00631f921e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.678 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:10:06.678 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:10:06.679 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:10:06.679 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:10:06.680 DEBUG:boto:Method: GET 2022-09-12T20:10:06.680 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:10:06.681 DEBUG:boto:Data: 2022-09-12T20:10:06.682 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.682 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.683 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.683 DEBUG:boto:Params: {} 2022-09-12T20:10:06.684 DEBUG:boto:Token: None 2022-09-12T20:10:06.684 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:10:06.685 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ix7iP+IDpG2vFO1UAdYvphMdviw= 2022-09-12T20:10:06.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ix7iP+IDpG2vFO1UAdYvphMdviw=', 'Content-Length': '0'} 2022-09-12T20:10:06.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045cc0cd0178b7250-00631f921e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.689 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:10:06.689 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:10:06.690 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:10:06.690 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:10:06.691 DEBUG:boto:Method: HEAD 2022-09-12T20:10:06.691 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:10:06.692 DEBUG:boto:Data: 2022-09-12T20:10:06.692 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.693 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.693 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.694 DEBUG:boto:Params: {} 2022-09-12T20:10:06.694 DEBUG:boto:Token: None 2022-09-12T20:10:06.695 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:10:06.696 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YHaxOxmrYGJJxcyRJHTXDtFYbg0= 2022-09-12T20:10:06.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YHaxOxmrYGJJxcyRJHTXDtFYbg0=', 'Content-Length': '0'} 2022-09-12T20:10:06.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079471630eee941ce-00631f921e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.699 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:10:06.699 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:10:06.700 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:10:06.700 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:10:06.701 DEBUG:boto:Method: HEAD 2022-09-12T20:10:06.701 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:10:06.702 DEBUG:boto:Data: 2022-09-12T20:10:06.702 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.703 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.703 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.704 DEBUG:boto:Params: {} 2022-09-12T20:10:06.704 DEBUG:boto:Token: None 2022-09-12T20:10:06.705 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:10:06.706 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YHaxOxmrYGJJxcyRJHTXDtFYbg0= 2022-09-12T20:10:06.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YHaxOxmrYGJJxcyRJHTXDtFYbg0=', 'Content-Length': '0'} 2022-09-12T20:10:06.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f41c4df80c183e37-00631f921e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:10:06.710 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:10:06.710 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:10:06.711 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:10:06.711 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:10:06.712 DEBUG:boto:Method: GET 2022-09-12T20:10:06.712 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:10:06.713 DEBUG:boto:Data: 2022-09-12T20:10:06.713 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.714 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.715 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.715 DEBUG:boto:Params: {} 2022-09-12T20:10:06.716 DEBUG:boto:Token: None 2022-09-12T20:10:06.716 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:10:06.717 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ix7iP+IDpG2vFO1UAdYvphMdviw= 2022-09-12T20:10:06.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ix7iP+IDpG2vFO1UAdYvphMdviw=', 'Content-Length': '0'} 2022-09-12T20:10:06.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eda67d04641d36b4-00631f921e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.762 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:10:06.762 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:10:06.763 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:10:06.763 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:10:06.764 DEBUG:boto:Method: GET 2022-09-12T20:10:06.765 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:10:06.765 DEBUG:boto:Data: 2022-09-12T20:10:06.766 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.766 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.767 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.767 DEBUG:boto:Params: {} 2022-09-12T20:10:06.768 DEBUG:boto:Token: None 2022-09-12T20:10:06.768 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:10:06.769 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ix7iP+IDpG2vFO1UAdYvphMdviw= 2022-09-12T20:10:06.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ix7iP+IDpG2vFO1UAdYvphMdviw=', 'Content-Length': '0'} 2022-09-12T20:10:06.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc509d3f2fc7c135-00631f921e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.773 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:06.773 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:06.774 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:10:06.774 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:10:06.775 DEBUG:boto:Method: GET 2022-09-12T20:10:06.775 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:10:06.776 DEBUG:boto:Data: 2022-09-12T20:10:06.776 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.777 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.777 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.778 DEBUG:boto:Params: {} 2022-09-12T20:10:06.778 DEBUG:boto:Token: None 2022-09-12T20:10:06.779 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/ 2022-09-12T20:10:06.780 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc= 2022-09-12T20:10:06.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc=', 'Content-Length': '0'} 2022-09-12T20:10:06.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fedd7a8981bd9fce-00631f921e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.783 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:06.784 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:06.785 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:06.785 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:10:06.786 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:10:06.787 DEBUG:boto:Method: GET 2022-09-12T20:10:06.787 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:10:06.788 DEBUG:boto:Data: 2022-09-12T20:10:06.788 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.789 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.789 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.790 DEBUG:boto:Params: {} 2022-09-12T20:10:06.791 DEBUG:boto:Token: None 2022-09-12T20:10:06.791 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/ 2022-09-12T20:10:06.792 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc= 2022-09-12T20:10:06.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc=', 'Content-Length': '0'} 2022-09-12T20:10:06.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000007e65a8528e9cb7-00631f921e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.796 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:06.797 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:10:06.798 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:10:06.798 DEBUG:boto:Method: HEAD 2022-09-12T20:10:06.799 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:10:06.800 DEBUG:boto:Data: 2022-09-12T20:10:06.800 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.801 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.801 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.802 DEBUG:boto:Params: {} 2022-09-12T20:10:06.802 DEBUG:boto:Token: None 2022-09-12T20:10:06.803 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj51 2022-09-12T20:10:06.803 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YAeLim1XksYyOV5lJzXNbd2d97c= 2022-09-12T20:10:06.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YAeLim1XksYyOV5lJzXNbd2d97c=', 'Content-Length': '0'} 2022-09-12T20:10:06.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008217d7fbce6c722e-00631f921e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.807 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:10:06.807 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:10:06.808 DEBUG:boto:Method: HEAD 2022-09-12T20:10:06.809 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:10:06.809 DEBUG:boto:Data: 2022-09-12T20:10:06.810 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.810 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.811 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.811 DEBUG:boto:Params: {} 2022-09-12T20:10:06.812 DEBUG:boto:Token: None 2022-09-12T20:10:06.812 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj51 2022-09-12T20:10:06.813 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YAeLim1XksYyOV5lJzXNbd2d97c= 2022-09-12T20:10:06.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YAeLim1XksYyOV5lJzXNbd2d97c=', 'Content-Length': '0'} 2022-09-12T20:10:06.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e49bf06be11e3e33-00631f921e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:10:06.817 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:10:06.817 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:10:06.818 DEBUG:boto:Method: GET 2022-09-12T20:10:06.819 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:10:06.819 DEBUG:boto:Data: 2022-09-12T20:10:06.820 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.820 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.821 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.821 DEBUG:boto:Params: {} 2022-09-12T20:10:06.822 DEBUG:boto:Token: None 2022-09-12T20:10:06.823 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj51 2022-09-12T20:10:06.823 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NhLiz1cxpZM/IpXng+NBokrC7QU= 2022-09-12T20:10:06.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NhLiz1cxpZM/IpXng+NBokrC7QU=', 'Content-Length': '0'} 2022-09-12T20:10:06.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000959748c1a7f8716a-00631f921e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.870 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:10:06.870 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:10:06.871 DEBUG:boto:Method: GET 2022-09-12T20:10:06.872 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:10:06.872 DEBUG:boto:Data: 2022-09-12T20:10:06.873 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.873 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.874 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.874 DEBUG:boto:Params: {} 2022-09-12T20:10:06.875 DEBUG:boto:Token: None 2022-09-12T20:10:06.875 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj51 2022-09-12T20:10:06.876 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NhLiz1cxpZM/IpXng+NBokrC7QU= 2022-09-12T20:10:06.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NhLiz1cxpZM/IpXng+NBokrC7QU=', 'Content-Length': '0'} 2022-09-12T20:10:06.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adecc2116dc1cd26-00631f921e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.879 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:06.880 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:06.880 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:10:06.881 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:10:06.881 DEBUG:boto:Method: GET 2022-09-12T20:10:06.882 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:10:06.882 DEBUG:boto:Data: 2022-09-12T20:10:06.883 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.884 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.884 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.885 DEBUG:boto:Params: {} 2022-09-12T20:10:06.885 DEBUG:boto:Token: None 2022-09-12T20:10:06.886 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/ 2022-09-12T20:10:06.886 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc= 2022-09-12T20:10:06.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc=', 'Content-Length': '0'} 2022-09-12T20:10:06.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020070cbc5d8e867d-00631f921e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.890 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:06.891 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:06.892 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:06.892 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:10:06.893 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:10:06.893 DEBUG:boto:Method: GET 2022-09-12T20:10:06.894 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:10:06.895 DEBUG:boto:Data: 2022-09-12T20:10:06.895 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.896 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.896 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.897 DEBUG:boto:Params: {} 2022-09-12T20:10:06.897 DEBUG:boto:Token: None 2022-09-12T20:10:06.898 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/ 2022-09-12T20:10:06.898 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc= 2022-09-12T20:10:06.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y5xncw2jkFtVWuluLT4chcYPzrc=', 'Content-Length': '0'} 2022-09-12T20:10:06.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020a7475389dafd6d-00631f921e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.902 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:06.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:10:06.904 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:10:06.904 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:10:06.905 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:10:06.905 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:10:06.906 DEBUG:boto:Method: GET 2022-09-12T20:10:06.906 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:10:06.907 DEBUG:boto:Data: 2022-09-12T20:10:06.907 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.908 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.908 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.909 DEBUG:boto:Params: {} 2022-09-12T20:10:06.910 DEBUG:boto:Token: None 2022-09-12T20:10:06.910 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:10:06.911 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6l5zitwN55pPcFY9Y8fiU/bG61o= 2022-09-12T20:10:06.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6l5zitwN55pPcFY9Y8fiU/bG61o=', 'Content-Length': '0'} 2022-09-12T20:10:06.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076cdd028a904e8f5-00631f921e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.915 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:10:06.915 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:10:06.916 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:10:06.916 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:10:06.917 DEBUG:boto:Method: GET 2022-09-12T20:10:06.917 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:10:06.918 DEBUG:boto:Data: 2022-09-12T20:10:06.918 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.919 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.920 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.920 DEBUG:boto:Params: {} 2022-09-12T20:10:06.921 DEBUG:boto:Token: None 2022-09-12T20:10:06.921 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:10:06.922 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6l5zitwN55pPcFY9Y8fiU/bG61o= 2022-09-12T20:10:06.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6l5zitwN55pPcFY9Y8fiU/bG61o=', 'Content-Length': '0'} 2022-09-12T20:10:06.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060d60c50862e6c57-00631f921e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.925 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:10:06.926 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:10:06.926 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:10:06.927 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:10:06.927 DEBUG:boto:Method: HEAD 2022-09-12T20:10:06.928 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:10:06.929 DEBUG:boto:Data: 2022-09-12T20:10:06.929 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.930 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.930 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.931 DEBUG:boto:Params: {} 2022-09-12T20:10:06.931 DEBUG:boto:Token: None 2022-09-12T20:10:06.932 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:10:06.932 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:d6yCao6ROAYDUZm6f4TIDxgqNzI= 2022-09-12T20:10:06.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:d6yCao6ROAYDUZm6f4TIDxgqNzI=', 'Content-Length': '0'} 2022-09-12T20:10:06.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000612400d300d5e1e0-00631f921e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.936 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:10:06.936 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:10:06.937 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:10:06.937 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:10:06.938 DEBUG:boto:Method: HEAD 2022-09-12T20:10:06.938 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:10:06.939 DEBUG:boto:Data: 2022-09-12T20:10:06.939 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.940 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:06.940 DEBUG:boto:Port: 8000 2022-09-12T20:10:06.941 DEBUG:boto:Params: {} 2022-09-12T20:10:06.941 DEBUG:boto:Token: None 2022-09-12T20:10:06.942 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:10:06.943 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:d6yCao6ROAYDUZm6f4TIDxgqNzI= 2022-09-12T20:10:06.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:d6yCao6ROAYDUZm6f4TIDxgqNzI=', 'Content-Length': '0'} 2022-09-12T20:10:06.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7967ecbb4662689-00631f921e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:10:06.947 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:10:06.947 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:10:06.948 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:10:06.948 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:10:06.949 DEBUG:boto:Method: GET 2022-09-12T20:10:06.949 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:10:06.950 DEBUG:boto:Data: 2022-09-12T20:10:06.950 DEBUG:boto:Headers: {} 2022-09-12T20:10:06.951 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:06.952 DEBUG:boto:Port: 8001 2022-09-12T20:10:06.952 DEBUG:boto:Params: {} 2022-09-12T20:10:06.953 DEBUG:boto:Token: None 2022-09-12T20:10:06.953 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:06 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:10:06.954 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6l5zitwN55pPcFY9Y8fiU/bG61o= 2022-09-12T20:10:06.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6l5zitwN55pPcFY9Y8fiU/bG61o=', 'Content-Length': '0'} 2022-09-12T20:10:06.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094c4a46620fa1b9f-00631f921e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:06.998 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:10:06.998 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:10:06.999 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:10:06.999 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:10:07.000 DEBUG:boto:Method: GET 2022-09-12T20:10:07.001 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:10:07.001 DEBUG:boto:Data: 2022-09-12T20:10:07.002 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.002 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.003 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.004 DEBUG:boto:Params: {} 2022-09-12T20:10:07.004 DEBUG:boto:Token: None 2022-09-12T20:10:07.005 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:10:07.005 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GD8INO5nqA0WUMvFuRoYXw3V0dI= 2022-09-12T20:10:07.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GD8INO5nqA0WUMvFuRoYXw3V0dI=', 'Content-Length': '0'} 2022-09-12T20:10:07.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f43e25aa6761c27-00631f921f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.009 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:07.010 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:07.010 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:10:07.011 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:10:07.011 DEBUG:boto:Method: GET 2022-09-12T20:10:07.012 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:10:07.013 DEBUG:boto:Data: 2022-09-12T20:10:07.013 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.014 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.014 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.015 DEBUG:boto:Params: {} 2022-09-12T20:10:07.015 DEBUG:boto:Token: None 2022-09-12T20:10:07.016 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/ 2022-09-12T20:10:07.016 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE= 2022-09-12T20:10:07.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE=', 'Content-Length': '0'} 2022-09-12T20:10:07.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005feaea676fdc2166-00631f921f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.021 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:07.021 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:07.022 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:07.022 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:10:07.023 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:10:07.023 DEBUG:boto:Method: GET 2022-09-12T20:10:07.024 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:10:07.024 DEBUG:boto:Data: 2022-09-12T20:10:07.025 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.025 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.026 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.026 DEBUG:boto:Params: {} 2022-09-12T20:10:07.027 DEBUG:boto:Token: None 2022-09-12T20:10:07.028 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/ 2022-09-12T20:10:07.028 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE= 2022-09-12T20:10:07.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE=', 'Content-Length': '0'} 2022-09-12T20:10:07.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031e1820f50be3d71-00631f921f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.032 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:07.033 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:10:07.033 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:10:07.034 DEBUG:boto:Method: HEAD 2022-09-12T20:10:07.035 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:10:07.035 DEBUG:boto:Data: 2022-09-12T20:10:07.036 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.036 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.036 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.037 DEBUG:boto:Params: {} 2022-09-12T20:10:07.038 DEBUG:boto:Token: None 2022-09-12T20:10:07.038 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj55 2022-09-12T20:10:07.039 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZkUcLTNn7uhWI636/VpBq+vama8= 2022-09-12T20:10:07.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZkUcLTNn7uhWI636/VpBq+vama8=', 'Content-Length': '0'} 2022-09-12T20:10:07.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2eaff2d3885747d-00631f921f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.042 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:10:07.043 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:10:07.043 DEBUG:boto:Method: HEAD 2022-09-12T20:10:07.044 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:10:07.044 DEBUG:boto:Data: 2022-09-12T20:10:07.045 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.045 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.046 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.046 DEBUG:boto:Params: {} 2022-09-12T20:10:07.047 DEBUG:boto:Token: None 2022-09-12T20:10:07.048 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj55 2022-09-12T20:10:07.048 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZkUcLTNn7uhWI636/VpBq+vama8= 2022-09-12T20:10:07.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZkUcLTNn7uhWI636/VpBq+vama8=', 'Content-Length': '0'} 2022-09-12T20:10:07.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000668dc2d42fae7eb7-00631f921f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:10:07.052 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:10:07.052 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:10:07.053 DEBUG:boto:Method: GET 2022-09-12T20:10:07.053 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:10:07.054 DEBUG:boto:Data: 2022-09-12T20:10:07.055 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.055 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.056 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.056 DEBUG:boto:Params: {} 2022-09-12T20:10:07.057 DEBUG:boto:Token: None 2022-09-12T20:10:07.057 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj55 2022-09-12T20:10:07.058 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tGYu9kmKlreaWEwUmR3fSoWy9Gk= 2022-09-12T20:10:07.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tGYu9kmKlreaWEwUmR3fSoWy9Gk=', 'Content-Length': '0'} 2022-09-12T20:10:07.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036946fb13c10d917-00631f921f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.061 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:10:07.062 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:10:07.063 DEBUG:boto:Method: GET 2022-09-12T20:10:07.063 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:10:07.064 DEBUG:boto:Data: 2022-09-12T20:10:07.065 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.065 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.066 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.066 DEBUG:boto:Params: {} 2022-09-12T20:10:07.067 DEBUG:boto:Token: None 2022-09-12T20:10:07.067 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj55 2022-09-12T20:10:07.068 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tGYu9kmKlreaWEwUmR3fSoWy9Gk= 2022-09-12T20:10:07.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tGYu9kmKlreaWEwUmR3fSoWy9Gk=', 'Content-Length': '0'} 2022-09-12T20:10:07.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df83cff58f8874de-00631f921f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.114 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:07.115 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:07.115 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:10:07.116 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:10:07.116 DEBUG:boto:Method: GET 2022-09-12T20:10:07.117 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:10:07.117 DEBUG:boto:Data: 2022-09-12T20:10:07.118 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.118 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.119 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.119 DEBUG:boto:Params: {} 2022-09-12T20:10:07.120 DEBUG:boto:Token: None 2022-09-12T20:10:07.120 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/ 2022-09-12T20:10:07.121 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE= 2022-09-12T20:10:07.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE=', 'Content-Length': '0'} 2022-09-12T20:10:07.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019bc8a30eb376136-00631f921f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.125 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:07.126 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:07.126 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:07.127 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:10:07.127 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:10:07.128 DEBUG:boto:Method: GET 2022-09-12T20:10:07.128 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:10:07.129 DEBUG:boto:Data: 2022-09-12T20:10:07.129 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.130 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.130 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.131 DEBUG:boto:Params: {} 2022-09-12T20:10:07.132 DEBUG:boto:Token: None 2022-09-12T20:10:07.132 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/ 2022-09-12T20:10:07.133 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE= 2022-09-12T20:10:07.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE=', 'Content-Length': '0'} 2022-09-12T20:10:07.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe7d7238c1ba06a7-00631f921f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.137 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:07.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:10:07.138 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:10:07.139 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:10:07.139 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:10:07.140 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:10:07.141 DEBUG:boto:Method: GET 2022-09-12T20:10:07.141 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:10:07.142 DEBUG:boto:Data: 2022-09-12T20:10:07.142 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.143 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.143 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.144 DEBUG:boto:Params: {} 2022-09-12T20:10:07.144 DEBUG:boto:Token: None 2022-09-12T20:10:07.145 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:10:07.146 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BncVuppKHlwR84nHo1OG9aOREsQ= 2022-09-12T20:10:07.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BncVuppKHlwR84nHo1OG9aOREsQ=', 'Content-Length': '0'} 2022-09-12T20:10:07.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033707987ecb4715f-00631f921f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.149 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:10:07.150 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:10:07.151 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:10:07.151 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:10:07.152 DEBUG:boto:Method: GET 2022-09-12T20:10:07.152 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:10:07.153 DEBUG:boto:Data: 2022-09-12T20:10:07.153 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.154 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.154 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.155 DEBUG:boto:Params: {} 2022-09-12T20:10:07.156 DEBUG:boto:Token: None 2022-09-12T20:10:07.156 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:10:07.157 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BncVuppKHlwR84nHo1OG9aOREsQ= 2022-09-12T20:10:07.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BncVuppKHlwR84nHo1OG9aOREsQ=', 'Content-Length': '0'} 2022-09-12T20:10:07.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cba28af70a1dfef-00631f921f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.160 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:10:07.161 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:10:07.161 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:10:07.162 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:10:07.162 DEBUG:boto:Method: HEAD 2022-09-12T20:10:07.163 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:10:07.163 DEBUG:boto:Data: 2022-09-12T20:10:07.164 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.164 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.165 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.165 DEBUG:boto:Params: {} 2022-09-12T20:10:07.166 DEBUG:boto:Token: None 2022-09-12T20:10:07.167 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:10:07.167 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hschh5wKOZiyweUaClwzZWsv8FE= 2022-09-12T20:10:07.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hschh5wKOZiyweUaClwzZWsv8FE=', 'Content-Length': '0'} 2022-09-12T20:10:07.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bd258336df09a0b-00631f921f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.170 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:10:07.171 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:10:07.172 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:10:07.172 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:10:07.173 DEBUG:boto:Method: HEAD 2022-09-12T20:10:07.173 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:10:07.174 DEBUG:boto:Data: 2022-09-12T20:10:07.174 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.175 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.175 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.176 DEBUG:boto:Params: {} 2022-09-12T20:10:07.176 DEBUG:boto:Token: None 2022-09-12T20:10:07.177 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:10:07.177 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hschh5wKOZiyweUaClwzZWsv8FE= 2022-09-12T20:10:07.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hschh5wKOZiyweUaClwzZWsv8FE=', 'Content-Length': '0'} 2022-09-12T20:10:07.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6a81e5eb426e18b-00631f921f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:10:07.181 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:10:07.182 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:10:07.182 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:10:07.183 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:10:07.183 DEBUG:boto:Method: GET 2022-09-12T20:10:07.184 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:10:07.185 DEBUG:boto:Data: 2022-09-12T20:10:07.185 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.186 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.186 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.187 DEBUG:boto:Params: {} 2022-09-12T20:10:07.187 DEBUG:boto:Token: None 2022-09-12T20:10:07.188 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:10:07.189 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BncVuppKHlwR84nHo1OG9aOREsQ= 2022-09-12T20:10:07.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BncVuppKHlwR84nHo1OG9aOREsQ=', 'Content-Length': '0'} 2022-09-12T20:10:07.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000786705cd05c15800-00631f921f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.192 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:10:07.193 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:10:07.194 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:10:07.194 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:10:07.195 DEBUG:boto:Method: GET 2022-09-12T20:10:07.195 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:10:07.196 DEBUG:boto:Data: 2022-09-12T20:10:07.196 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.197 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.197 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.198 DEBUG:boto:Params: {} 2022-09-12T20:10:07.198 DEBUG:boto:Token: None 2022-09-12T20:10:07.199 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:10:07.199 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BncVuppKHlwR84nHo1OG9aOREsQ= 2022-09-12T20:10:07.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BncVuppKHlwR84nHo1OG9aOREsQ=', 'Content-Length': '0'} 2022-09-12T20:10:07.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acfaa634438365a6-00631f921f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.203 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:07.204 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:07.204 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:10:07.205 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:10:07.205 DEBUG:boto:Method: GET 2022-09-12T20:10:07.206 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:10:07.206 DEBUG:boto:Data: 2022-09-12T20:10:07.207 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.207 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.208 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.208 DEBUG:boto:Params: {} 2022-09-12T20:10:07.209 DEBUG:boto:Token: None 2022-09-12T20:10:07.210 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/ 2022-09-12T20:10:07.210 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE= 2022-09-12T20:10:07.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE=', 'Content-Length': '0'} 2022-09-12T20:10:07.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a444c154aaec45f-00631f921f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.214 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:07.215 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:07.216 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:07.216 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:10:07.217 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:10:07.217 DEBUG:boto:Method: GET 2022-09-12T20:10:07.218 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:10:07.218 DEBUG:boto:Data: 2022-09-12T20:10:07.219 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.220 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.220 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.221 DEBUG:boto:Params: {} 2022-09-12T20:10:07.221 DEBUG:boto:Token: None 2022-09-12T20:10:07.222 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/ 2022-09-12T20:10:07.222 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE= 2022-09-12T20:10:07.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE=', 'Content-Length': '0'} 2022-09-12T20:10:07.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8f83fbd60642661-00631f921f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.226 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:07.227 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:10:07.228 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:10:07.228 DEBUG:boto:Method: HEAD 2022-09-12T20:10:07.229 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:10:07.229 DEBUG:boto:Data: 2022-09-12T20:10:07.230 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.230 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.231 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.231 DEBUG:boto:Params: {} 2022-09-12T20:10:07.232 DEBUG:boto:Token: None 2022-09-12T20:10:07.232 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj57 2022-09-12T20:10:07.233 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VeXAY3tHOhV2t/x4qAHPL/zOaJ0= 2022-09-12T20:10:07.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VeXAY3tHOhV2t/x4qAHPL/zOaJ0=', 'Content-Length': '0'} 2022-09-12T20:10:07.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071b69e2dd1bd002e-00631f921f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.236 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:10:07.237 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:10:07.237 DEBUG:boto:Method: HEAD 2022-09-12T20:10:07.238 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:10:07.238 DEBUG:boto:Data: 2022-09-12T20:10:07.239 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.239 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.240 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.240 DEBUG:boto:Params: {} 2022-09-12T20:10:07.241 DEBUG:boto:Token: None 2022-09-12T20:10:07.241 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj57 2022-09-12T20:10:07.242 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VeXAY3tHOhV2t/x4qAHPL/zOaJ0= 2022-09-12T20:10:07.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VeXAY3tHOhV2t/x4qAHPL/zOaJ0=', 'Content-Length': '0'} 2022-09-12T20:10:07.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000526217c202847f2d-00631f921f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:10:07.246 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:10:07.246 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:10:07.247 DEBUG:boto:Method: GET 2022-09-12T20:10:07.247 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:10:07.248 DEBUG:boto:Data: 2022-09-12T20:10:07.248 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.249 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.249 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.250 DEBUG:boto:Params: {} 2022-09-12T20:10:07.250 DEBUG:boto:Token: None 2022-09-12T20:10:07.251 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj57 2022-09-12T20:10:07.252 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fAGG7pjsINpDkivJrgrEeqqPF0k= 2022-09-12T20:10:07.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fAGG7pjsINpDkivJrgrEeqqPF0k=', 'Content-Length': '0'} 2022-09-12T20:10:07.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a5f6aa83329cd61-00631f921f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.255 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:10:07.256 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:10:07.256 DEBUG:boto:Method: GET 2022-09-12T20:10:07.257 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:10:07.257 DEBUG:boto:Data: 2022-09-12T20:10:07.258 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.259 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.259 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.260 DEBUG:boto:Params: {} 2022-09-12T20:10:07.260 DEBUG:boto:Token: None 2022-09-12T20:10:07.261 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj57 2022-09-12T20:10:07.261 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fAGG7pjsINpDkivJrgrEeqqPF0k= 2022-09-12T20:10:07.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fAGG7pjsINpDkivJrgrEeqqPF0k=', 'Content-Length': '0'} 2022-09-12T20:10:07.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b47b656550abf33-00631f921f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.265 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:07.265 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:07.266 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:10:07.266 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:10:07.267 DEBUG:boto:Method: GET 2022-09-12T20:10:07.267 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:10:07.268 DEBUG:boto:Data: 2022-09-12T20:10:07.268 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.269 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.269 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.270 DEBUG:boto:Params: {} 2022-09-12T20:10:07.271 DEBUG:boto:Token: None 2022-09-12T20:10:07.271 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/ 2022-09-12T20:10:07.272 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE= 2022-09-12T20:10:07.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE=', 'Content-Length': '0'} 2022-09-12T20:10:07.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000003311e730eca3a0-00631f921f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.318 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:07.319 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:07.319 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:07.320 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:10:07.320 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:10:07.321 DEBUG:boto:Method: GET 2022-09-12T20:10:07.321 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:10:07.322 DEBUG:boto:Data: 2022-09-12T20:10:07.322 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.323 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.323 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.324 DEBUG:boto:Params: {} 2022-09-12T20:10:07.324 DEBUG:boto:Token: None 2022-09-12T20:10:07.325 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/ 2022-09-12T20:10:07.325 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE= 2022-09-12T20:10:07.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE=', 'Content-Length': '0'} 2022-09-12T20:10:07.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a1e4fabaf887892-00631f921f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.330 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:07.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:10:07.331 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:10:07.331 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:10:07.332 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:10:07.332 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:10:07.333 DEBUG:boto:Method: GET 2022-09-12T20:10:07.333 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:10:07.334 DEBUG:boto:Data: 2022-09-12T20:10:07.335 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.335 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.336 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.336 DEBUG:boto:Params: {} 2022-09-12T20:10:07.337 DEBUG:boto:Token: None 2022-09-12T20:10:07.337 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:10:07.338 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dE0uJsGgRmcZLkDdjpgRZmS7204= 2022-09-12T20:10:07.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dE0uJsGgRmcZLkDdjpgRZmS7204=', 'Content-Length': '0'} 2022-09-12T20:10:07.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068f5eaa82e2b8077-00631f921f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.341 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:10:07.342 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:10:07.343 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:10:07.343 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:10:07.344 DEBUG:boto:Method: GET 2022-09-12T20:10:07.344 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:10:07.345 DEBUG:boto:Data: 2022-09-12T20:10:07.345 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.346 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.346 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.347 DEBUG:boto:Params: {} 2022-09-12T20:10:07.348 DEBUG:boto:Token: None 2022-09-12T20:10:07.348 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:10:07.349 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dE0uJsGgRmcZLkDdjpgRZmS7204= 2022-09-12T20:10:07.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dE0uJsGgRmcZLkDdjpgRZmS7204=', 'Content-Length': '0'} 2022-09-12T20:10:07.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000146682bbda8854eb-00631f921f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.352 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:10:07.353 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:10:07.353 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:10:07.354 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:10:07.354 DEBUG:boto:Method: HEAD 2022-09-12T20:10:07.355 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:10:07.355 DEBUG:boto:Data: 2022-09-12T20:10:07.356 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.356 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.357 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.357 DEBUG:boto:Params: {} 2022-09-12T20:10:07.358 DEBUG:boto:Token: None 2022-09-12T20:10:07.359 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:10:07.359 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1slEjSET0tLEcw0BaKbTen/AVw4= 2022-09-12T20:10:07.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1slEjSET0tLEcw0BaKbTen/AVw4=', 'Content-Length': '0'} 2022-09-12T20:10:07.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000228c1fee30d8ab87-00631f921f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.362 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:10:07.363 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:10:07.363 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:10:07.364 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:10:07.364 DEBUG:boto:Method: HEAD 2022-09-12T20:10:07.365 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:10:07.365 DEBUG:boto:Data: 2022-09-12T20:10:07.366 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.366 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.367 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.367 DEBUG:boto:Params: {} 2022-09-12T20:10:07.368 DEBUG:boto:Token: None 2022-09-12T20:10:07.369 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:10:07.370 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1slEjSET0tLEcw0BaKbTen/AVw4= 2022-09-12T20:10:07.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1slEjSET0tLEcw0BaKbTen/AVw4=', 'Content-Length': '0'} 2022-09-12T20:10:07.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddb16bc1df70dff7-00631f921f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:10:07.374 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:10:07.374 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:10:07.375 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:10:07.375 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:10:07.376 DEBUG:boto:Method: GET 2022-09-12T20:10:07.376 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:10:07.377 DEBUG:boto:Data: 2022-09-12T20:10:07.377 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.378 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.378 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.379 DEBUG:boto:Params: {} 2022-09-12T20:10:07.379 DEBUG:boto:Token: None 2022-09-12T20:10:07.380 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:10:07.381 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dE0uJsGgRmcZLkDdjpgRZmS7204= 2022-09-12T20:10:07.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dE0uJsGgRmcZLkDdjpgRZmS7204=', 'Content-Length': '0'} 2022-09-12T20:10:07.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ab0e93bbe7e82f7-00631f921f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.384 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:10:07.385 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:10:07.385 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:10:07.386 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:10:07.386 DEBUG:boto:Method: GET 2022-09-12T20:10:07.387 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:10:07.387 DEBUG:boto:Data: 2022-09-12T20:10:07.388 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.388 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.389 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.389 DEBUG:boto:Params: {} 2022-09-12T20:10:07.390 DEBUG:boto:Token: None 2022-09-12T20:10:07.391 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:10:07.391 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dE0uJsGgRmcZLkDdjpgRZmS7204= 2022-09-12T20:10:07.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dE0uJsGgRmcZLkDdjpgRZmS7204=', 'Content-Length': '0'} 2022-09-12T20:10:07.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fec931fe9ada24c-00631f921f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.395 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:07.395 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:07.396 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:10:07.396 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:10:07.397 DEBUG:boto:Method: GET 2022-09-12T20:10:07.397 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:10:07.398 DEBUG:boto:Data: 2022-09-12T20:10:07.398 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.399 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.399 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.400 DEBUG:boto:Params: {} 2022-09-12T20:10:07.400 DEBUG:boto:Token: None 2022-09-12T20:10:07.401 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/ 2022-09-12T20:10:07.401 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE= 2022-09-12T20:10:07.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE=', 'Content-Length': '0'} 2022-09-12T20:10:07.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000173bcc97d0fb456b-00631f921f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.406 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:07.406 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:07.407 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:07.408 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:10:07.408 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:10:07.408 DEBUG:boto:Method: GET 2022-09-12T20:10:07.409 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:10:07.410 DEBUG:boto:Data: 2022-09-12T20:10:07.410 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.411 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.411 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.412 DEBUG:boto:Params: {} 2022-09-12T20:10:07.412 DEBUG:boto:Token: None 2022-09-12T20:10:07.413 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/ 2022-09-12T20:10:07.413 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE= 2022-09-12T20:10:07.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE=', 'Content-Length': '0'} 2022-09-12T20:10:07.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007378bfe4ab1b243c-00631f921f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.458 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:07.459 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:10:07.459 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:10:07.460 DEBUG:boto:Method: HEAD 2022-09-12T20:10:07.460 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:10:07.461 DEBUG:boto:Data: 2022-09-12T20:10:07.461 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.462 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.462 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.463 DEBUG:boto:Params: {} 2022-09-12T20:10:07.463 DEBUG:boto:Token: None 2022-09-12T20:10:07.464 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj68 2022-09-12T20:10:07.465 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ScAxL35UxUArSKk9rrnAw/EcKN4= 2022-09-12T20:10:07.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ScAxL35UxUArSKk9rrnAw/EcKN4=', 'Content-Length': '0'} 2022-09-12T20:10:07.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038401b86909e855f-00631f921f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.468 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:10:07.469 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:10:07.469 DEBUG:boto:Method: HEAD 2022-09-12T20:10:07.470 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:10:07.470 DEBUG:boto:Data: 2022-09-12T20:10:07.471 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.471 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.472 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.472 DEBUG:boto:Params: {} 2022-09-12T20:10:07.473 DEBUG:boto:Token: None 2022-09-12T20:10:07.473 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj68 2022-09-12T20:10:07.474 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ScAxL35UxUArSKk9rrnAw/EcKN4= 2022-09-12T20:10:07.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ScAxL35UxUArSKk9rrnAw/EcKN4=', 'Content-Length': '0'} 2022-09-12T20:10:07.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5727e0fe99e1c77-00631f921f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:10:07.478 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:10:07.479 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:10:07.479 DEBUG:boto:Method: GET 2022-09-12T20:10:07.480 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:10:07.480 DEBUG:boto:Data: 2022-09-12T20:10:07.481 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.481 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.482 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.482 DEBUG:boto:Params: {} 2022-09-12T20:10:07.483 DEBUG:boto:Token: None 2022-09-12T20:10:07.484 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj68 2022-09-12T20:10:07.484 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ka3ULz1OrjnG/c/b4+Q17Nrk3Vk= 2022-09-12T20:10:07.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ka3ULz1OrjnG/c/b4+Q17Nrk3Vk=', 'Content-Length': '0'} 2022-09-12T20:10:07.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021d768ec02f5be3a-00631f921f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.488 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:10:07.488 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:10:07.489 DEBUG:boto:Method: GET 2022-09-12T20:10:07.489 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:10:07.490 DEBUG:boto:Data: 2022-09-12T20:10:07.490 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.491 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.491 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.492 DEBUG:boto:Params: {} 2022-09-12T20:10:07.493 DEBUG:boto:Token: None 2022-09-12T20:10:07.493 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj68 2022-09-12T20:10:07.494 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ka3ULz1OrjnG/c/b4+Q17Nrk3Vk= 2022-09-12T20:10:07.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ka3ULz1OrjnG/c/b4+Q17Nrk3Vk=', 'Content-Length': '0'} 2022-09-12T20:10:07.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d3051ee89e61036-00631f921f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.497 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:07.498 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:07.498 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:10:07.499 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:10:07.499 DEBUG:boto:Method: GET 2022-09-12T20:10:07.500 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:10:07.500 DEBUG:boto:Data: 2022-09-12T20:10:07.501 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.501 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.502 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.502 DEBUG:boto:Params: {} 2022-09-12T20:10:07.503 DEBUG:boto:Token: None 2022-09-12T20:10:07.503 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/ 2022-09-12T20:10:07.504 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE= 2022-09-12T20:10:07.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE=', 'Content-Length': '0'} 2022-09-12T20:10:07.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025f5d116c30d9e3c-00631f921f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.508 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:07.509 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:07.510 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:07.510 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:10:07.511 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:10:07.511 DEBUG:boto:Method: GET 2022-09-12T20:10:07.512 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:10:07.512 DEBUG:boto:Data: 2022-09-12T20:10:07.513 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.513 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.514 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.514 DEBUG:boto:Params: {} 2022-09-12T20:10:07.515 DEBUG:boto:Token: None 2022-09-12T20:10:07.516 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/ 2022-09-12T20:10:07.516 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE= 2022-09-12T20:10:07.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE=', 'Content-Length': '0'} 2022-09-12T20:10:07.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009107bae4e0e19580-00631f921f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.520 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:07.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:10:07.521 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:10:07.522 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:10:07.522 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:10:07.523 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:10:07.523 DEBUG:boto:Method: GET 2022-09-12T20:10:07.524 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:10:07.524 DEBUG:boto:Data: 2022-09-12T20:10:07.525 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.525 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.526 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.527 DEBUG:boto:Params: {} 2022-09-12T20:10:07.527 DEBUG:boto:Token: None 2022-09-12T20:10:07.528 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:10:07.528 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D8vgEOmHPR02BQvrkEeHV4aI5gM= 2022-09-12T20:10:07.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D8vgEOmHPR02BQvrkEeHV4aI5gM=', 'Content-Length': '0'} 2022-09-12T20:10:07.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d70e69cd8a522afb-00631f921f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.574 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:10:07.575 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:10:07.575 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:10:07.576 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:10:07.576 DEBUG:boto:Method: GET 2022-09-12T20:10:07.577 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:10:07.577 DEBUG:boto:Data: 2022-09-12T20:10:07.578 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.578 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.579 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.579 DEBUG:boto:Params: {} 2022-09-12T20:10:07.580 DEBUG:boto:Token: None 2022-09-12T20:10:07.581 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:10:07.581 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D8vgEOmHPR02BQvrkEeHV4aI5gM= 2022-09-12T20:10:07.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D8vgEOmHPR02BQvrkEeHV4aI5gM=', 'Content-Length': '0'} 2022-09-12T20:10:07.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c74ae380ee142333-00631f921f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.585 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:10:07.586 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:10:07.586 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:10:07.587 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:10:07.587 DEBUG:boto:Method: HEAD 2022-09-12T20:10:07.588 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:10:07.588 DEBUG:boto:Data: 2022-09-12T20:10:07.589 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.589 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.590 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.590 DEBUG:boto:Params: {} 2022-09-12T20:10:07.591 DEBUG:boto:Token: None 2022-09-12T20:10:07.591 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:10:07.592 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WVAbWP8dTrJhVyTj4RzDWBzVbUw= 2022-09-12T20:10:07.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WVAbWP8dTrJhVyTj4RzDWBzVbUw=', 'Content-Length': '0'} 2022-09-12T20:10:07.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006df1e4568fc5fd85-00631f921f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.595 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:10:07.596 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:10:07.597 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:10:07.597 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:10:07.598 DEBUG:boto:Method: HEAD 2022-09-12T20:10:07.598 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:10:07.599 DEBUG:boto:Data: 2022-09-12T20:10:07.599 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.600 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.600 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.601 DEBUG:boto:Params: {} 2022-09-12T20:10:07.601 DEBUG:boto:Token: None 2022-09-12T20:10:07.602 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:10:07.602 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WVAbWP8dTrJhVyTj4RzDWBzVbUw= 2022-09-12T20:10:07.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WVAbWP8dTrJhVyTj4RzDWBzVbUw=', 'Content-Length': '0'} 2022-09-12T20:10:07.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f34a077ab53de63e-00631f921f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:10:07.606 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:10:07.607 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:10:07.607 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:10:07.608 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:10:07.608 DEBUG:boto:Method: GET 2022-09-12T20:10:07.609 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:10:07.609 DEBUG:boto:Data: 2022-09-12T20:10:07.610 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.610 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.611 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.612 DEBUG:boto:Params: {} 2022-09-12T20:10:07.612 DEBUG:boto:Token: None 2022-09-12T20:10:07.613 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:10:07.613 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D8vgEOmHPR02BQvrkEeHV4aI5gM= 2022-09-12T20:10:07.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D8vgEOmHPR02BQvrkEeHV4aI5gM=', 'Content-Length': '0'} 2022-09-12T20:10:07.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c550ff2b7c053f4-00631f921f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.617 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:10:07.617 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:10:07.618 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:10:07.618 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:10:07.619 DEBUG:boto:Method: GET 2022-09-12T20:10:07.620 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:10:07.620 DEBUG:boto:Data: 2022-09-12T20:10:07.621 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.621 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.622 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.622 DEBUG:boto:Params: {} 2022-09-12T20:10:07.623 DEBUG:boto:Token: None 2022-09-12T20:10:07.623 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:10:07.624 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D8vgEOmHPR02BQvrkEeHV4aI5gM= 2022-09-12T20:10:07.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D8vgEOmHPR02BQvrkEeHV4aI5gM=', 'Content-Length': '0'} 2022-09-12T20:10:07.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5a020e7d122b07e-00631f921f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.627 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:07.628 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:07.629 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:10:07.629 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:10:07.630 DEBUG:boto:Method: GET 2022-09-12T20:10:07.630 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:10:07.631 DEBUG:boto:Data: 2022-09-12T20:10:07.631 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.632 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.632 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.633 DEBUG:boto:Params: {} 2022-09-12T20:10:07.633 DEBUG:boto:Token: None 2022-09-12T20:10:07.634 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/ 2022-09-12T20:10:07.635 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE= 2022-09-12T20:10:07.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE=', 'Content-Length': '0'} 2022-09-12T20:10:07.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007af9c868ddc36fbc-00631f921f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.639 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:07.640 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:07.640 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:07.641 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:10:07.641 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:10:07.642 DEBUG:boto:Method: GET 2022-09-12T20:10:07.642 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:10:07.643 DEBUG:boto:Data: 2022-09-12T20:10:07.643 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.644 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.644 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.645 DEBUG:boto:Params: {} 2022-09-12T20:10:07.646 DEBUG:boto:Token: None 2022-09-12T20:10:07.646 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/ 2022-09-12T20:10:07.647 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE= 2022-09-12T20:10:07.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE=', 'Content-Length': '0'} 2022-09-12T20:10:07.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059e98c7980662a71-00631f921f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.651 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:07.652 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:10:07.652 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:10:07.653 DEBUG:boto:Method: HEAD 2022-09-12T20:10:07.653 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:10:07.654 DEBUG:boto:Data: 2022-09-12T20:10:07.654 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.655 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.655 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.656 DEBUG:boto:Params: {} 2022-09-12T20:10:07.656 DEBUG:boto:Token: None 2022-09-12T20:10:07.657 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj69 2022-09-12T20:10:07.657 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:94547Q2sKLGiIWFSRVeJcrbpMaQ= 2022-09-12T20:10:07.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:94547Q2sKLGiIWFSRVeJcrbpMaQ=', 'Content-Length': '0'} 2022-09-12T20:10:07.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de14c848548272be-00631f921f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.661 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:10:07.661 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:10:07.662 DEBUG:boto:Method: HEAD 2022-09-12T20:10:07.662 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:10:07.663 DEBUG:boto:Data: 2022-09-12T20:10:07.663 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.664 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.665 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.665 DEBUG:boto:Params: {} 2022-09-12T20:10:07.666 DEBUG:boto:Token: None 2022-09-12T20:10:07.666 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj69 2022-09-12T20:10:07.667 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:94547Q2sKLGiIWFSRVeJcrbpMaQ= 2022-09-12T20:10:07.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:94547Q2sKLGiIWFSRVeJcrbpMaQ=', 'Content-Length': '0'} 2022-09-12T20:10:07.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013eabc09e013cef8-00631f921f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:10:07.671 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:10:07.671 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:10:07.672 DEBUG:boto:Method: GET 2022-09-12T20:10:07.672 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:10:07.673 DEBUG:boto:Data: 2022-09-12T20:10:07.673 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.674 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.675 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.675 DEBUG:boto:Params: {} 2022-09-12T20:10:07.676 DEBUG:boto:Token: None 2022-09-12T20:10:07.676 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj69 2022-09-12T20:10:07.677 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mHXGk1IhCAC5yxhyPawU+Vkael4= 2022-09-12T20:10:07.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mHXGk1IhCAC5yxhyPawU+Vkael4=', 'Content-Length': '0'} 2022-09-12T20:10:07.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae690b93ab4c9339-00631f921f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.722 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:10:07.723 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:10:07.723 DEBUG:boto:Method: GET 2022-09-12T20:10:07.724 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:10:07.724 DEBUG:boto:Data: 2022-09-12T20:10:07.725 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.725 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.726 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.726 DEBUG:boto:Params: {} 2022-09-12T20:10:07.727 DEBUG:boto:Token: None 2022-09-12T20:10:07.727 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj69 2022-09-12T20:10:07.728 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mHXGk1IhCAC5yxhyPawU+Vkael4= 2022-09-12T20:10:07.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mHXGk1IhCAC5yxhyPawU+Vkael4=', 'Content-Length': '0'} 2022-09-12T20:10:07.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008781614de0331f6f-00631f921f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.731 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:07.732 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:07.732 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:10:07.733 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:10:07.733 DEBUG:boto:Method: GET 2022-09-12T20:10:07.734 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:10:07.735 DEBUG:boto:Data: 2022-09-12T20:10:07.735 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.736 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.736 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.737 DEBUG:boto:Params: {} 2022-09-12T20:10:07.737 DEBUG:boto:Token: None 2022-09-12T20:10:07.738 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/ 2022-09-12T20:10:07.739 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE= 2022-09-12T20:10:07.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE=', 'Content-Length': '0'} 2022-09-12T20:10:07.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e54e8152f4e29856-00631f921f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.743 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:07.743 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:07.744 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:07.744 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:10:07.745 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:10:07.746 DEBUG:boto:Method: GET 2022-09-12T20:10:07.746 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:10:07.747 DEBUG:boto:Data: 2022-09-12T20:10:07.747 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.748 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.748 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.749 DEBUG:boto:Params: {} 2022-09-12T20:10:07.749 DEBUG:boto:Token: None 2022-09-12T20:10:07.750 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/ 2022-09-12T20:10:07.751 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE= 2022-09-12T20:10:07.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE=', 'Content-Length': '0'} 2022-09-12T20:10:07.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c49fee002d08262c-00631f921f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.754 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:07.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:10:07.756 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:10:07.756 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:10:07.757 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:10:07.757 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:10:07.758 DEBUG:boto:Method: GET 2022-09-12T20:10:07.758 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:10:07.759 DEBUG:boto:Data: 2022-09-12T20:10:07.760 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.760 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.761 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.761 DEBUG:boto:Params: {} 2022-09-12T20:10:07.762 DEBUG:boto:Token: None 2022-09-12T20:10:07.762 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:10:07.763 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KfLVen2wkDcmOG2bzT8nvbOOPC0= 2022-09-12T20:10:07.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KfLVen2wkDcmOG2bzT8nvbOOPC0=', 'Content-Length': '0'} 2022-09-12T20:10:07.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e30e5db07c1c218e-00631f921f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.767 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:10:07.767 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:10:07.768 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:10:07.768 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:10:07.769 DEBUG:boto:Method: GET 2022-09-12T20:10:07.769 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:10:07.770 DEBUG:boto:Data: 2022-09-12T20:10:07.776 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.777 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.777 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.778 DEBUG:boto:Params: {} 2022-09-12T20:10:07.778 DEBUG:boto:Token: None 2022-09-12T20:10:07.779 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:10:07.780 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KfLVen2wkDcmOG2bzT8nvbOOPC0= 2022-09-12T20:10:07.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KfLVen2wkDcmOG2bzT8nvbOOPC0=', 'Content-Length': '0'} 2022-09-12T20:10:07.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c76879361a51093-00631f921f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.783 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:10:07.784 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:10:07.784 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:10:07.785 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:10:07.785 DEBUG:boto:Method: HEAD 2022-09-12T20:10:07.786 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:10:07.786 DEBUG:boto:Data: 2022-09-12T20:10:07.787 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.787 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.788 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.789 DEBUG:boto:Params: {} 2022-09-12T20:10:07.789 DEBUG:boto:Token: None 2022-09-12T20:10:07.790 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:10:07.790 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ep50Gn7zabMxps1e5jDucxEyyr0= 2022-09-12T20:10:07.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ep50Gn7zabMxps1e5jDucxEyyr0=', 'Content-Length': '0'} 2022-09-12T20:10:07.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002639accd101984c3-00631f921f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.793 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:10:07.794 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:10:07.794 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:10:07.795 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:10:07.795 DEBUG:boto:Method: HEAD 2022-09-12T20:10:07.796 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:10:07.796 DEBUG:boto:Data: 2022-09-12T20:10:07.797 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.797 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.798 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.798 DEBUG:boto:Params: {} 2022-09-12T20:10:07.799 DEBUG:boto:Token: None 2022-09-12T20:10:07.800 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:10:07.800 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ep50Gn7zabMxps1e5jDucxEyyr0= 2022-09-12T20:10:07.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ep50Gn7zabMxps1e5jDucxEyyr0=', 'Content-Length': '0'} 2022-09-12T20:10:07.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d208f426cb89f034-00631f921f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:10:07.804 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:10:07.805 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:10:07.806 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:10:07.807 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:10:07.807 DEBUG:boto:Method: GET 2022-09-12T20:10:07.808 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:10:07.808 DEBUG:boto:Data: 2022-09-12T20:10:07.809 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.809 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.810 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.810 DEBUG:boto:Params: {} 2022-09-12T20:10:07.811 DEBUG:boto:Token: None 2022-09-12T20:10:07.812 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:10:07.812 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KfLVen2wkDcmOG2bzT8nvbOOPC0= 2022-09-12T20:10:07.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KfLVen2wkDcmOG2bzT8nvbOOPC0=', 'Content-Length': '0'} 2022-09-12T20:10:07.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000190f5fa59c63f357-00631f921f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.816 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:10:07.816 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:10:07.817 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:10:07.817 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:10:07.818 DEBUG:boto:Method: GET 2022-09-12T20:10:07.818 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:10:07.819 DEBUG:boto:Data: 2022-09-12T20:10:07.819 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.820 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.820 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.821 DEBUG:boto:Params: {} 2022-09-12T20:10:07.821 DEBUG:boto:Token: None 2022-09-12T20:10:07.822 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:10:07.823 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KfLVen2wkDcmOG2bzT8nvbOOPC0= 2022-09-12T20:10:07.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KfLVen2wkDcmOG2bzT8nvbOOPC0=', 'Content-Length': '0'} 2022-09-12T20:10:07.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c71ebcfbc99aafee-00631f921f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.870 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:07.871 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:07.871 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:10:07.872 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:10:07.872 DEBUG:boto:Method: GET 2022-09-12T20:10:07.873 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:10:07.873 DEBUG:boto:Data: 2022-09-12T20:10:07.874 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.874 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.875 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.876 DEBUG:boto:Params: {} 2022-09-12T20:10:07.876 DEBUG:boto:Token: None 2022-09-12T20:10:07.877 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/ 2022-09-12T20:10:07.877 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE= 2022-09-12T20:10:07.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE=', 'Content-Length': '0'} 2022-09-12T20:10:07.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003701723e23e17ba3-00631f921f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.882 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:07.883 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:07.883 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:07.884 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:10:07.884 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:10:07.885 DEBUG:boto:Method: GET 2022-09-12T20:10:07.885 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:10:07.886 DEBUG:boto:Data: 2022-09-12T20:10:07.886 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.887 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.887 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.888 DEBUG:boto:Params: {} 2022-09-12T20:10:07.889 DEBUG:boto:Token: None 2022-09-12T20:10:07.889 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/ 2022-09-12T20:10:07.890 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE= 2022-09-12T20:10:07.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE=', 'Content-Length': '0'} 2022-09-12T20:10:07.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b66501a2691c973-00631f921f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.894 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:07.895 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:10:07.895 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:10:07.896 DEBUG:boto:Method: HEAD 2022-09-12T20:10:07.896 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:10:07.897 DEBUG:boto:Data: 2022-09-12T20:10:07.897 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.898 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.898 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.899 DEBUG:boto:Params: {} 2022-09-12T20:10:07.900 DEBUG:boto:Token: None 2022-09-12T20:10:07.900 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj76 2022-09-12T20:10:07.901 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gr/M6tHCR08oa2anEl0GAzj6Q08= 2022-09-12T20:10:07.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gr/M6tHCR08oa2anEl0GAzj6Q08=', 'Content-Length': '0'} 2022-09-12T20:10:07.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c67c7cb613d08ab1-00631f921f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.904 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:10:07.905 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:10:07.905 DEBUG:boto:Method: HEAD 2022-09-12T20:10:07.906 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:10:07.906 DEBUG:boto:Data: 2022-09-12T20:10:07.907 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.907 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.908 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.908 DEBUG:boto:Params: {} 2022-09-12T20:10:07.909 DEBUG:boto:Token: None 2022-09-12T20:10:07.909 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj76 2022-09-12T20:10:07.910 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gr/M6tHCR08oa2anEl0GAzj6Q08= 2022-09-12T20:10:07.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gr/M6tHCR08oa2anEl0GAzj6Q08=', 'Content-Length': '0'} 2022-09-12T20:10:07.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ae6f08e33dc3b00-00631f921f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:10:07.914 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:10:07.914 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:10:07.915 DEBUG:boto:Method: GET 2022-09-12T20:10:07.915 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:10:07.916 DEBUG:boto:Data: 2022-09-12T20:10:07.916 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.917 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.917 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.918 DEBUG:boto:Params: {} 2022-09-12T20:10:07.919 DEBUG:boto:Token: None 2022-09-12T20:10:07.919 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj76 2022-09-12T20:10:07.920 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Emcu+Ot9AZq6R1JXHLOyon7NIuM= 2022-09-12T20:10:07.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Emcu+Ot9AZq6R1JXHLOyon7NIuM=', 'Content-Length': '0'} 2022-09-12T20:10:07.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006bf7da770a7517d-00631f921f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.924 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:10:07.924 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:10:07.925 DEBUG:boto:Method: GET 2022-09-12T20:10:07.925 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:10:07.926 DEBUG:boto:Data: 2022-09-12T20:10:07.926 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.927 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.927 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.928 DEBUG:boto:Params: {} 2022-09-12T20:10:07.929 DEBUG:boto:Token: None 2022-09-12T20:10:07.929 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/obj76 2022-09-12T20:10:07.930 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Emcu+Ot9AZq6R1JXHLOyon7NIuM= 2022-09-12T20:10:07.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Emcu+Ot9AZq6R1JXHLOyon7NIuM=', 'Content-Length': '0'} 2022-09-12T20:10:07.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092dab4785d943682-00631f921f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.933 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:07.934 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:07.934 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:10:07.935 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:10:07.936 DEBUG:boto:Method: GET 2022-09-12T20:10:07.936 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:10:07.937 DEBUG:boto:Data: 2022-09-12T20:10:07.937 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.938 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:07.938 DEBUG:boto:Port: 8001 2022-09-12T20:10:07.939 DEBUG:boto:Params: {} 2022-09-12T20:10:07.939 DEBUG:boto:Token: None 2022-09-12T20:10:07.940 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/ 2022-09-12T20:10:07.941 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE= 2022-09-12T20:10:07.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE=', 'Content-Length': '0'} 2022-09-12T20:10:07.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000314de73b1dd8bdec-00631f921f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.986 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:07.987 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:07.987 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:07.988 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:10:07.988 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:10:07.989 DEBUG:boto:Method: GET 2022-09-12T20:10:07.989 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:10:07.990 DEBUG:boto:Data: 2022-09-12T20:10:07.991 DEBUG:boto:Headers: {} 2022-09-12T20:10:07.991 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:07.992 DEBUG:boto:Port: 8000 2022-09-12T20:10:07.992 DEBUG:boto:Params: {} 2022-09-12T20:10:07.993 DEBUG:boto:Token: None 2022-09-12T20:10:07.993 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:07 GMT /jdpiak-75/ 2022-09-12T20:10:07.994 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE= 2022-09-12T20:10:07.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MneMRomRj4SSyOxpyQPIfbTY4IE=', 'Content-Length': '0'} 2022-09-12T20:10:07.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000782294416bde22e7-00631f921f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:07.998 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:07.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:10:07.999 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:10:08.000 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:10:08.000 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:10:08.001 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:10:08.001 DEBUG:boto:Method: GET 2022-09-12T20:10:08.002 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:10:08.003 DEBUG:boto:Data: 2022-09-12T20:10:08.003 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.004 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:08.004 DEBUG:boto:Port: 8001 2022-09-12T20:10:08.005 DEBUG:boto:Params: {} 2022-09-12T20:10:08.005 DEBUG:boto:Token: None 2022-09-12T20:10:08.006 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:10:08.006 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0OR80D4c//mAdYh+7GErBusnGWQ= 2022-09-12T20:10:08.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0OR80D4c//mAdYh+7GErBusnGWQ=', 'Content-Length': '0'} 2022-09-12T20:10:08.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071a71e855eeedfe9-00631f9220-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.011 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:10:08.011 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:10:08.012 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:10:08.012 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:10:08.013 DEBUG:boto:Method: GET 2022-09-12T20:10:08.013 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:10:08.014 DEBUG:boto:Data: 2022-09-12T20:10:08.015 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.015 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:08.016 DEBUG:boto:Port: 8000 2022-09-12T20:10:08.016 DEBUG:boto:Params: {} 2022-09-12T20:10:08.017 DEBUG:boto:Token: None 2022-09-12T20:10:08.017 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:10:08.018 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0OR80D4c//mAdYh+7GErBusnGWQ= 2022-09-12T20:10:08.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0OR80D4c//mAdYh+7GErBusnGWQ=', 'Content-Length': '0'} 2022-09-12T20:10:08.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc4c32b36bfb50b8-00631f9220-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.021 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:10:08.022 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:10:08.022 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:10:08.023 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:10:08.023 DEBUG:boto:Method: HEAD 2022-09-12T20:10:08.024 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:10:08.025 DEBUG:boto:Data: 2022-09-12T20:10:08.025 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.026 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:08.026 DEBUG:boto:Port: 8001 2022-09-12T20:10:08.027 DEBUG:boto:Params: {} 2022-09-12T20:10:08.027 DEBUG:boto:Token: None 2022-09-12T20:10:08.028 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:10:08.028 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wf0uxKQIpWZ/yr49Cj74TWXv/bM= 2022-09-12T20:10:08.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wf0uxKQIpWZ/yr49Cj74TWXv/bM=', 'Content-Length': '0'} 2022-09-12T20:10:08.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009904ca0eaeca7dd6-00631f9220-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.032 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:10:08.032 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:10:08.033 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:10:08.033 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:10:08.034 DEBUG:boto:Method: HEAD 2022-09-12T20:10:08.034 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:10:08.035 DEBUG:boto:Data: 2022-09-12T20:10:08.035 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.036 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:08.036 DEBUG:boto:Port: 8000 2022-09-12T20:10:08.037 DEBUG:boto:Params: {} 2022-09-12T20:10:08.037 DEBUG:boto:Token: None 2022-09-12T20:10:08.038 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:10:08.039 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wf0uxKQIpWZ/yr49Cj74TWXv/bM= 2022-09-12T20:10:08.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wf0uxKQIpWZ/yr49Cj74TWXv/bM=', 'Content-Length': '0'} 2022-09-12T20:10:08.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000430d71ca431e8dc3-00631f9220-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:10:08.042 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:10:08.043 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:10:08.043 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:10:08.044 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:10:08.044 DEBUG:boto:Method: GET 2022-09-12T20:10:08.045 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:10:08.045 DEBUG:boto:Data: 2022-09-12T20:10:08.046 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.047 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:08.047 DEBUG:boto:Port: 8001 2022-09-12T20:10:08.048 DEBUG:boto:Params: {} 2022-09-12T20:10:08.048 DEBUG:boto:Token: None 2022-09-12T20:10:08.049 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:10:08.049 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0OR80D4c//mAdYh+7GErBusnGWQ= 2022-09-12T20:10:08.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0OR80D4c//mAdYh+7GErBusnGWQ=', 'Content-Length': '0'} 2022-09-12T20:10:08.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025e66efc5f67b303-00631f9220-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.053 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:10:08.054 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:10:08.054 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:10:08.055 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:10:08.055 DEBUG:boto:Method: GET 2022-09-12T20:10:08.056 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:10:08.056 DEBUG:boto:Data: 2022-09-12T20:10:08.057 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.057 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:08.058 DEBUG:boto:Port: 8000 2022-09-12T20:10:08.058 DEBUG:boto:Params: {} 2022-09-12T20:10:08.059 DEBUG:boto:Token: None 2022-09-12T20:10:08.059 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:10:08.060 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0OR80D4c//mAdYh+7GErBusnGWQ= 2022-09-12T20:10:08.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0OR80D4c//mAdYh+7GErBusnGWQ=', 'Content-Length': '0'} 2022-09-12T20:10:08.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000264da0124b5a25c9-00631f9220-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.064 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:08.064 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:08.065 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:10:08.065 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:10:08.066 DEBUG:boto:Method: GET 2022-09-12T20:10:08.066 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:10:08.067 DEBUG:boto:Data: 2022-09-12T20:10:08.067 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.068 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:08.068 DEBUG:boto:Port: 8001 2022-09-12T20:10:08.069 DEBUG:boto:Params: {} 2022-09-12T20:10:08.070 DEBUG:boto:Token: None 2022-09-12T20:10:08.070 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/ 2022-09-12T20:10:08.071 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KYuRrmPm9o687t5IkchvnOEWHnM= 2022-09-12T20:10:08.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KYuRrmPm9o687t5IkchvnOEWHnM=', 'Content-Length': '0'} 2022-09-12T20:10:08.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0bddd381f1952d9-00631f9220-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.075 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:08.076 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:08.076 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:08.077 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:10:08.077 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:10:08.078 DEBUG:boto:Method: GET 2022-09-12T20:10:08.078 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:10:08.079 DEBUG:boto:Data: 2022-09-12T20:10:08.080 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.080 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:08.081 DEBUG:boto:Port: 8000 2022-09-12T20:10:08.081 DEBUG:boto:Params: {} 2022-09-12T20:10:08.082 DEBUG:boto:Token: None 2022-09-12T20:10:08.082 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/ 2022-09-12T20:10:08.083 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KYuRrmPm9o687t5IkchvnOEWHnM= 2022-09-12T20:10:08.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KYuRrmPm9o687t5IkchvnOEWHnM=', 'Content-Length': '0'} 2022-09-12T20:10:08.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002713660938d2c4fa-00631f9220-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.087 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:08.088 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:10:08.088 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:10:08.089 DEBUG:boto:Method: HEAD 2022-09-12T20:10:08.089 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:10:08.090 DEBUG:boto:Data: 2022-09-12T20:10:08.090 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.091 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:08.091 DEBUG:boto:Port: 8001 2022-09-12T20:10:08.092 DEBUG:boto:Params: {} 2022-09-12T20:10:08.092 DEBUG:boto:Token: None 2022-09-12T20:10:08.093 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj85 2022-09-12T20:10:08.093 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u+2AP4pdcGyhCln55JtFQDh50EI= 2022-09-12T20:10:08.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u+2AP4pdcGyhCln55JtFQDh50EI=', 'Content-Length': '0'} 2022-09-12T20:10:08.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b25aa214ef265d34-00631f9220-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.097 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:10:08.097 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:10:08.098 DEBUG:boto:Method: HEAD 2022-09-12T20:10:08.099 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:10:08.099 DEBUG:boto:Data: 2022-09-12T20:10:08.100 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.100 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:08.101 DEBUG:boto:Port: 8000 2022-09-12T20:10:08.101 DEBUG:boto:Params: {} 2022-09-12T20:10:08.102 DEBUG:boto:Token: None 2022-09-12T20:10:08.102 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj85 2022-09-12T20:10:08.103 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u+2AP4pdcGyhCln55JtFQDh50EI= 2022-09-12T20:10:08.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u+2AP4pdcGyhCln55JtFQDh50EI=', 'Content-Length': '0'} 2022-09-12T20:10:08.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e2cf2f8595417fe-00631f9220-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:10:08.107 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:10:08.107 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:10:08.108 DEBUG:boto:Method: GET 2022-09-12T20:10:08.108 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:10:08.109 DEBUG:boto:Data: 2022-09-12T20:10:08.109 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.110 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:08.110 DEBUG:boto:Port: 8001 2022-09-12T20:10:08.111 DEBUG:boto:Params: {} 2022-09-12T20:10:08.111 DEBUG:boto:Token: None 2022-09-12T20:10:08.112 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj85 2022-09-12T20:10:08.113 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qWS1y8rjRVLX5MN5dcTycibZ2OM= 2022-09-12T20:10:08.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qWS1y8rjRVLX5MN5dcTycibZ2OM=', 'Content-Length': '0'} 2022-09-12T20:10:08.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039df9911a5f733dd-00631f9220-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.158 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:10:08.158 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:10:08.159 DEBUG:boto:Method: GET 2022-09-12T20:10:08.160 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:10:08.160 DEBUG:boto:Data: 2022-09-12T20:10:08.161 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.161 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:08.162 DEBUG:boto:Port: 8000 2022-09-12T20:10:08.162 DEBUG:boto:Params: {} 2022-09-12T20:10:08.163 DEBUG:boto:Token: None 2022-09-12T20:10:08.163 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj85 2022-09-12T20:10:08.164 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qWS1y8rjRVLX5MN5dcTycibZ2OM= 2022-09-12T20:10:08.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qWS1y8rjRVLX5MN5dcTycibZ2OM=', 'Content-Length': '0'} 2022-09-12T20:10:08.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df1eae6c32236637-00631f9220-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.167 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:08.168 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:08.168 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:10:08.169 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:10:08.169 DEBUG:boto:Method: GET 2022-09-12T20:10:08.170 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:10:08.171 DEBUG:boto:Data: 2022-09-12T20:10:08.171 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.172 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:08.172 DEBUG:boto:Port: 8001 2022-09-12T20:10:08.173 DEBUG:boto:Params: {} 2022-09-12T20:10:08.173 DEBUG:boto:Token: None 2022-09-12T20:10:08.174 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/ 2022-09-12T20:10:08.175 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KYuRrmPm9o687t5IkchvnOEWHnM= 2022-09-12T20:10:08.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KYuRrmPm9o687t5IkchvnOEWHnM=', 'Content-Length': '0'} 2022-09-12T20:10:08.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0fad87ae54cfbd0-00631f9220-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.179 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:08.179 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:08.180 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:08.181 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:10:08.181 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:10:08.182 DEBUG:boto:Method: GET 2022-09-12T20:10:08.182 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:10:08.183 DEBUG:boto:Data: 2022-09-12T20:10:08.183 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.184 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:08.185 DEBUG:boto:Port: 8000 2022-09-12T20:10:08.186 DEBUG:boto:Params: {} 2022-09-12T20:10:08.186 DEBUG:boto:Token: None 2022-09-12T20:10:08.187 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/ 2022-09-12T20:10:08.188 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KYuRrmPm9o687t5IkchvnOEWHnM= 2022-09-12T20:10:08.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KYuRrmPm9o687t5IkchvnOEWHnM=', 'Content-Length': '0'} 2022-09-12T20:10:08.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034c693205eb0e253-00631f9220-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.191 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:08.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:10:08.193 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:10:08.193 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:10:08.194 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:10:08.195 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:10:08.195 DEBUG:boto:Method: GET 2022-09-12T20:10:08.196 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:10:08.196 DEBUG:boto:Data: 2022-09-12T20:10:08.197 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.197 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:08.198 DEBUG:boto:Port: 8001 2022-09-12T20:10:08.198 DEBUG:boto:Params: {} 2022-09-12T20:10:08.199 DEBUG:boto:Token: None 2022-09-12T20:10:08.199 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:10:08.200 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l4yRJfO3DHm6eVAM1TkQ0w94F0A= 2022-09-12T20:10:08.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l4yRJfO3DHm6eVAM1TkQ0w94F0A=', 'Content-Length': '0'} 2022-09-12T20:10:08.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c0ef8450560da70-00631f9220-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.203 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:10:08.204 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:10:08.204 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:10:08.205 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:10:08.205 DEBUG:boto:Method: GET 2022-09-12T20:10:08.206 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:10:08.207 DEBUG:boto:Data: 2022-09-12T20:10:08.207 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.208 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:08.208 DEBUG:boto:Port: 8000 2022-09-12T20:10:08.209 DEBUG:boto:Params: {} 2022-09-12T20:10:08.209 DEBUG:boto:Token: None 2022-09-12T20:10:08.210 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:10:08.211 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l4yRJfO3DHm6eVAM1TkQ0w94F0A= 2022-09-12T20:10:08.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l4yRJfO3DHm6eVAM1TkQ0w94F0A=', 'Content-Length': '0'} 2022-09-12T20:10:08.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e76ccaba11338748-00631f9220-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.214 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:10:08.215 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:10:08.215 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:10:08.216 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:10:08.216 DEBUG:boto:Method: HEAD 2022-09-12T20:10:08.217 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:10:08.217 DEBUG:boto:Data: 2022-09-12T20:10:08.218 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.219 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:08.219 DEBUG:boto:Port: 8001 2022-09-12T20:10:08.220 DEBUG:boto:Params: {} 2022-09-12T20:10:08.220 DEBUG:boto:Token: None 2022-09-12T20:10:08.221 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:10:08.221 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ItQMRYU4zWUi1jL9UmRc4z7cw5k= 2022-09-12T20:10:08.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ItQMRYU4zWUi1jL9UmRc4z7cw5k=', 'Content-Length': '0'} 2022-09-12T20:10:08.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009aa043b0b4f98e50-00631f9220-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.225 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:10:08.225 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:10:08.226 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:10:08.226 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:10:08.227 DEBUG:boto:Method: HEAD 2022-09-12T20:10:08.227 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:10:08.228 DEBUG:boto:Data: 2022-09-12T20:10:08.228 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.229 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:08.229 DEBUG:boto:Port: 8000 2022-09-12T20:10:08.230 DEBUG:boto:Params: {} 2022-09-12T20:10:08.230 DEBUG:boto:Token: None 2022-09-12T20:10:08.231 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:10:08.231 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ItQMRYU4zWUi1jL9UmRc4z7cw5k= 2022-09-12T20:10:08.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ItQMRYU4zWUi1jL9UmRc4z7cw5k=', 'Content-Length': '0'} 2022-09-12T20:10:08.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5fef524b327e5a1-00631f9220-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:10:08.235 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:10:08.236 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:10:08.236 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:10:08.237 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:10:08.237 DEBUG:boto:Method: GET 2022-09-12T20:10:08.238 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:10:08.238 DEBUG:boto:Data: 2022-09-12T20:10:08.239 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.240 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:08.240 DEBUG:boto:Port: 8001 2022-09-12T20:10:08.241 DEBUG:boto:Params: {} 2022-09-12T20:10:08.241 DEBUG:boto:Token: None 2022-09-12T20:10:08.242 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:10:08.243 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l4yRJfO3DHm6eVAM1TkQ0w94F0A= 2022-09-12T20:10:08.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l4yRJfO3DHm6eVAM1TkQ0w94F0A=', 'Content-Length': '0'} 2022-09-12T20:10:08.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c10b7fe4afde082-00631f9220-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.286 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:10:08.286 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:10:08.287 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:10:08.288 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:10:08.288 DEBUG:boto:Method: GET 2022-09-12T20:10:08.289 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:10:08.289 DEBUG:boto:Data: 2022-09-12T20:10:08.290 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.290 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:08.291 DEBUG:boto:Port: 8000 2022-09-12T20:10:08.291 DEBUG:boto:Params: {} 2022-09-12T20:10:08.292 DEBUG:boto:Token: None 2022-09-12T20:10:08.293 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:10:08.293 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l4yRJfO3DHm6eVAM1TkQ0w94F0A= 2022-09-12T20:10:08.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l4yRJfO3DHm6eVAM1TkQ0w94F0A=', 'Content-Length': '0'} 2022-09-12T20:10:08.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099416ba94703443d-00631f9220-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.297 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:08.297 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:08.298 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:10:08.298 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:10:08.299 DEBUG:boto:Method: GET 2022-09-12T20:10:08.299 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:10:08.300 DEBUG:boto:Data: 2022-09-12T20:10:08.300 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.301 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:08.301 DEBUG:boto:Port: 8001 2022-09-12T20:10:08.302 DEBUG:boto:Params: {} 2022-09-12T20:10:08.302 DEBUG:boto:Token: None 2022-09-12T20:10:08.303 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/ 2022-09-12T20:10:08.304 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KYuRrmPm9o687t5IkchvnOEWHnM= 2022-09-12T20:10:08.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KYuRrmPm9o687t5IkchvnOEWHnM=', 'Content-Length': '0'} 2022-09-12T20:10:08.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c96bbc455a7f5baf-00631f9220-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.307 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:08.308 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:08.309 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:08.309 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:10:08.309 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:10:08.310 DEBUG:boto:Method: GET 2022-09-12T20:10:08.311 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:10:08.311 DEBUG:boto:Data: 2022-09-12T20:10:08.312 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.312 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:08.313 DEBUG:boto:Port: 8000 2022-09-12T20:10:08.313 DEBUG:boto:Params: {} 2022-09-12T20:10:08.314 DEBUG:boto:Token: None 2022-09-12T20:10:08.314 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/ 2022-09-12T20:10:08.315 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KYuRrmPm9o687t5IkchvnOEWHnM= 2022-09-12T20:10:08.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KYuRrmPm9o687t5IkchvnOEWHnM=', 'Content-Length': '0'} 2022-09-12T20:10:08.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002489aa5201f3da7a-00631f9220-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.319 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:08.320 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:10:08.320 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:10:08.321 DEBUG:boto:Method: HEAD 2022-09-12T20:10:08.321 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:10:08.322 DEBUG:boto:Data: 2022-09-12T20:10:08.322 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.323 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:08.323 DEBUG:boto:Port: 8001 2022-09-12T20:10:08.324 DEBUG:boto:Params: {} 2022-09-12T20:10:08.324 DEBUG:boto:Token: None 2022-09-12T20:10:08.325 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj92 2022-09-12T20:10:08.325 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BZ5fbrMI7nihpPNzHJcbxSKEH9o= 2022-09-12T20:10:08.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BZ5fbrMI7nihpPNzHJcbxSKEH9o=', 'Content-Length': '0'} 2022-09-12T20:10:08.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3ea81c962371b04-00631f9220-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.328 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:10:08.329 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:10:08.329 DEBUG:boto:Method: HEAD 2022-09-12T20:10:08.330 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:10:08.330 DEBUG:boto:Data: 2022-09-12T20:10:08.331 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.331 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:08.332 DEBUG:boto:Port: 8000 2022-09-12T20:10:08.332 DEBUG:boto:Params: {} 2022-09-12T20:10:08.333 DEBUG:boto:Token: None 2022-09-12T20:10:08.333 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj92 2022-09-12T20:10:08.334 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BZ5fbrMI7nihpPNzHJcbxSKEH9o= 2022-09-12T20:10:08.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BZ5fbrMI7nihpPNzHJcbxSKEH9o=', 'Content-Length': '0'} 2022-09-12T20:10:08.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5828c38a5812f62-00631f9220-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:10:08.338 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:10:08.338 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:10:08.339 DEBUG:boto:Method: GET 2022-09-12T20:10:08.339 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:10:08.340 DEBUG:boto:Data: 2022-09-12T20:10:08.340 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.341 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:08.341 DEBUG:boto:Port: 8001 2022-09-12T20:10:08.342 DEBUG:boto:Params: {} 2022-09-12T20:10:08.342 DEBUG:boto:Token: None 2022-09-12T20:10:08.343 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj92 2022-09-12T20:10:08.344 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:08UX+Ttm+/7/BY8w37y7bimnEbA= 2022-09-12T20:10:08.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:08UX+Ttm+/7/BY8w37y7bimnEbA=', 'Content-Length': '0'} 2022-09-12T20:10:08.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002689fa203ee15813-00631f9220-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.347 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:10:08.348 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:10:08.348 DEBUG:boto:Method: GET 2022-09-12T20:10:08.349 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:10:08.349 DEBUG:boto:Data: 2022-09-12T20:10:08.350 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.350 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:08.351 DEBUG:boto:Port: 8000 2022-09-12T20:10:08.351 DEBUG:boto:Params: {} 2022-09-12T20:10:08.352 DEBUG:boto:Token: None 2022-09-12T20:10:08.352 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj92 2022-09-12T20:10:08.353 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:08UX+Ttm+/7/BY8w37y7bimnEbA= 2022-09-12T20:10:08.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:08UX+Ttm+/7/BY8w37y7bimnEbA=', 'Content-Length': '0'} 2022-09-12T20:10:08.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003659b6881e57165b-00631f9220-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.398 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:08.399 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:08.399 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:10:08.400 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:10:08.400 DEBUG:boto:Method: GET 2022-09-12T20:10:08.401 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:10:08.401 DEBUG:boto:Data: 2022-09-12T20:10:08.402 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.402 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:08.403 DEBUG:boto:Port: 8001 2022-09-12T20:10:08.403 DEBUG:boto:Params: {} 2022-09-12T20:10:08.404 DEBUG:boto:Token: None 2022-09-12T20:10:08.405 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/ 2022-09-12T20:10:08.405 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KYuRrmPm9o687t5IkchvnOEWHnM= 2022-09-12T20:10:08.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KYuRrmPm9o687t5IkchvnOEWHnM=', 'Content-Length': '0'} 2022-09-12T20:10:08.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cab909c5c9804404-00631f9220-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.409 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:08.410 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:08.411 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:08.411 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:10:08.412 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:10:08.412 DEBUG:boto:Method: GET 2022-09-12T20:10:08.413 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:10:08.413 DEBUG:boto:Data: 2022-09-12T20:10:08.414 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.415 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:08.415 DEBUG:boto:Port: 8000 2022-09-12T20:10:08.416 DEBUG:boto:Params: {} 2022-09-12T20:10:08.416 DEBUG:boto:Token: None 2022-09-12T20:10:08.417 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/ 2022-09-12T20:10:08.417 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KYuRrmPm9o687t5IkchvnOEWHnM= 2022-09-12T20:10:08.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KYuRrmPm9o687t5IkchvnOEWHnM=', 'Content-Length': '0'} 2022-09-12T20:10:08.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c207a16c7dd28660-00631f9220-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.421 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:08.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:10:08.423 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:10:08.423 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:10:08.424 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:10:08.424 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:10:08.425 DEBUG:boto:Method: GET 2022-09-12T20:10:08.425 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:10:08.426 DEBUG:boto:Data: 2022-09-12T20:10:08.426 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.427 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:08.427 DEBUG:boto:Port: 8001 2022-09-12T20:10:08.428 DEBUG:boto:Params: {} 2022-09-12T20:10:08.428 DEBUG:boto:Token: None 2022-09-12T20:10:08.429 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:10:08.429 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CVoNvtMGTv/GR7RplW/7Scu5QmI= 2022-09-12T20:10:08.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CVoNvtMGTv/GR7RplW/7Scu5QmI=', 'Content-Length': '0'} 2022-09-12T20:10:08.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb7dbdd70a81b93f-00631f9220-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.433 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:10:08.434 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:10:08.434 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:10:08.435 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:10:08.435 DEBUG:boto:Method: GET 2022-09-12T20:10:08.436 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:10:08.436 DEBUG:boto:Data: 2022-09-12T20:10:08.437 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.437 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:08.438 DEBUG:boto:Port: 8000 2022-09-12T20:10:08.438 DEBUG:boto:Params: {} 2022-09-12T20:10:08.439 DEBUG:boto:Token: None 2022-09-12T20:10:08.440 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:10:08.440 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CVoNvtMGTv/GR7RplW/7Scu5QmI= 2022-09-12T20:10:08.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CVoNvtMGTv/GR7RplW/7Scu5QmI=', 'Content-Length': '0'} 2022-09-12T20:10:08.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003443c6c0ca4cefb2-00631f9220-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.443 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:10:08.444 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:10:08.444 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:10:08.445 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:10:08.445 DEBUG:boto:Method: HEAD 2022-09-12T20:10:08.446 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:10:08.446 DEBUG:boto:Data: 2022-09-12T20:10:08.447 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.447 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:08.448 DEBUG:boto:Port: 8001 2022-09-12T20:10:08.448 DEBUG:boto:Params: {} 2022-09-12T20:10:08.449 DEBUG:boto:Token: None 2022-09-12T20:10:08.449 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:10:08.450 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tCE0CTxrPLl9qECYb3W4LS9JtWs= 2022-09-12T20:10:08.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tCE0CTxrPLl9qECYb3W4LS9JtWs=', 'Content-Length': '0'} 2022-09-12T20:10:08.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa3feb8cec05f994-00631f9220-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.454 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:10:08.454 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:10:08.455 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:10:08.455 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:10:08.455 DEBUG:boto:Method: HEAD 2022-09-12T20:10:08.456 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:10:08.456 DEBUG:boto:Data: 2022-09-12T20:10:08.457 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.457 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:08.458 DEBUG:boto:Port: 8000 2022-09-12T20:10:08.458 DEBUG:boto:Params: {} 2022-09-12T20:10:08.459 DEBUG:boto:Token: None 2022-09-12T20:10:08.459 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:10:08.460 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tCE0CTxrPLl9qECYb3W4LS9JtWs= 2022-09-12T20:10:08.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tCE0CTxrPLl9qECYb3W4LS9JtWs=', 'Content-Length': '0'} 2022-09-12T20:10:08.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004246ea7af27dc17c-00631f9220-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:10:08.464 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:10:08.464 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:10:08.465 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:10:08.465 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:10:08.466 DEBUG:boto:Method: GET 2022-09-12T20:10:08.466 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:10:08.467 DEBUG:boto:Data: 2022-09-12T20:10:08.467 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.468 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:08.468 DEBUG:boto:Port: 8001 2022-09-12T20:10:08.469 DEBUG:boto:Params: {} 2022-09-12T20:10:08.469 DEBUG:boto:Token: None 2022-09-12T20:10:08.470 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:10:08.470 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CVoNvtMGTv/GR7RplW/7Scu5QmI= 2022-09-12T20:10:08.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CVoNvtMGTv/GR7RplW/7Scu5QmI=', 'Content-Length': '0'} 2022-09-12T20:10:08.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a920a7fa99687894-00631f9220-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.474 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:10:08.474 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:10:08.475 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:10:08.475 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:10:08.476 DEBUG:boto:Method: GET 2022-09-12T20:10:08.476 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:10:08.477 DEBUG:boto:Data: 2022-09-12T20:10:08.477 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.478 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:08.478 DEBUG:boto:Port: 8000 2022-09-12T20:10:08.479 DEBUG:boto:Params: {} 2022-09-12T20:10:08.480 DEBUG:boto:Token: None 2022-09-12T20:10:08.481 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:10:08.481 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CVoNvtMGTv/GR7RplW/7Scu5QmI= 2022-09-12T20:10:08.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CVoNvtMGTv/GR7RplW/7Scu5QmI=', 'Content-Length': '0'} 2022-09-12T20:10:08.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a37305349aec1385-00631f9220-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.526 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:08.527 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:08.527 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:10:08.528 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:10:08.528 DEBUG:boto:Method: GET 2022-09-12T20:10:08.529 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:10:08.529 DEBUG:boto:Data: 2022-09-12T20:10:08.530 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.530 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:08.531 DEBUG:boto:Port: 8001 2022-09-12T20:10:08.532 DEBUG:boto:Params: {} 2022-09-12T20:10:08.532 DEBUG:boto:Token: None 2022-09-12T20:10:08.533 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/ 2022-09-12T20:10:08.534 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KYuRrmPm9o687t5IkchvnOEWHnM= 2022-09-12T20:10:08.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KYuRrmPm9o687t5IkchvnOEWHnM=', 'Content-Length': '0'} 2022-09-12T20:10:08.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001009e0b130d5d843-00631f9220-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.538 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:08.539 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:08.540 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:08.540 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:10:08.541 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:10:08.541 DEBUG:boto:Method: GET 2022-09-12T20:10:08.542 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:10:08.542 DEBUG:boto:Data: 2022-09-12T20:10:08.543 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.544 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:08.544 DEBUG:boto:Port: 8000 2022-09-12T20:10:08.545 DEBUG:boto:Params: {} 2022-09-12T20:10:08.545 DEBUG:boto:Token: None 2022-09-12T20:10:08.546 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/ 2022-09-12T20:10:08.547 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KYuRrmPm9o687t5IkchvnOEWHnM= 2022-09-12T20:10:08.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KYuRrmPm9o687t5IkchvnOEWHnM=', 'Content-Length': '0'} 2022-09-12T20:10:08.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025f5eb39d075155a-00631f9220-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.577 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:08.578 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:10:08.578 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:10:08.579 DEBUG:boto:Method: HEAD 2022-09-12T20:10:08.579 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:10:08.580 DEBUG:boto:Data: 2022-09-12T20:10:08.581 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.581 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:08.582 DEBUG:boto:Port: 8001 2022-09-12T20:10:08.582 DEBUG:boto:Params: {} 2022-09-12T20:10:08.583 DEBUG:boto:Token: None 2022-09-12T20:10:08.583 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj95 2022-09-12T20:10:08.584 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bE1x0SDy92UcDAy+GN8IolacU70= 2022-09-12T20:10:08.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bE1x0SDy92UcDAy+GN8IolacU70=', 'Content-Length': '0'} 2022-09-12T20:10:08.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e499eb86d82b8bc-00631f9220-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.587 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:10:08.588 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:10:08.588 DEBUG:boto:Method: HEAD 2022-09-12T20:10:08.589 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:10:08.590 DEBUG:boto:Data: 2022-09-12T20:10:08.590 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.591 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:08.592 DEBUG:boto:Port: 8000 2022-09-12T20:10:08.592 DEBUG:boto:Params: {} 2022-09-12T20:10:08.593 DEBUG:boto:Token: None 2022-09-12T20:10:08.593 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj95 2022-09-12T20:10:08.594 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bE1x0SDy92UcDAy+GN8IolacU70= 2022-09-12T20:10:08.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bE1x0SDy92UcDAy+GN8IolacU70=', 'Content-Length': '0'} 2022-09-12T20:10:08.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000311bbe35e45b5347-00631f9220-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:10:08.598 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:10:08.599 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:10:08.600 DEBUG:boto:Method: GET 2022-09-12T20:10:08.600 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:10:08.601 DEBUG:boto:Data: 2022-09-12T20:10:08.602 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.602 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:08.603 DEBUG:boto:Port: 8001 2022-09-12T20:10:08.604 DEBUG:boto:Params: {} 2022-09-12T20:10:08.605 DEBUG:boto:Token: None 2022-09-12T20:10:08.605 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj95 2022-09-12T20:10:08.606 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uzW3RatuVd9qx/uuYh++57Ikh94= 2022-09-12T20:10:08.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uzW3RatuVd9qx/uuYh++57Ikh94=', 'Content-Length': '0'} 2022-09-12T20:10:08.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e46c29e660b88aeb-00631f9220-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.650 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:10:08.651 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:10:08.652 DEBUG:boto:Method: GET 2022-09-12T20:10:08.653 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:10:08.653 DEBUG:boto:Data: 2022-09-12T20:10:08.654 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.655 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:08.656 DEBUG:boto:Port: 8000 2022-09-12T20:10:08.656 DEBUG:boto:Params: {} 2022-09-12T20:10:08.657 DEBUG:boto:Token: None 2022-09-12T20:10:08.658 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj95 2022-09-12T20:10:08.659 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uzW3RatuVd9qx/uuYh++57Ikh94= 2022-09-12T20:10:08.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uzW3RatuVd9qx/uuYh++57Ikh94=', 'Content-Length': '0'} 2022-09-12T20:10:08.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc3a5186e09707ff-00631f9220-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.663 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:08.664 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:08.664 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:10:08.665 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:10:08.665 DEBUG:boto:Method: GET 2022-09-12T20:10:08.666 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:10:08.667 DEBUG:boto:Data: 2022-09-12T20:10:08.668 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.668 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:10:08.669 DEBUG:boto:Port: 8001 2022-09-12T20:10:08.670 DEBUG:boto:Params: {} 2022-09-12T20:10:08.671 DEBUG:boto:Token: None 2022-09-12T20:10:08.672 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/ 2022-09-12T20:10:08.672 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KYuRrmPm9o687t5IkchvnOEWHnM= 2022-09-12T20:10:08.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KYuRrmPm9o687t5IkchvnOEWHnM=', 'Content-Length': '0'} 2022-09-12T20:10:08.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042a0fb49a3208aaf-00631f9220-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.677 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:08.678 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:08.679 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:08.679 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:10:08.680 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:10:08.680 DEBUG:boto:Method: GET 2022-09-12T20:10:08.681 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:10:08.682 DEBUG:boto:Data: 2022-09-12T20:10:08.682 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.683 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:10:08.684 DEBUG:boto:Port: 8000 2022-09-12T20:10:08.685 DEBUG:boto:Params: {} 2022-09-12T20:10:08.686 DEBUG:boto:Token: None 2022-09-12T20:10:08.686 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/ 2022-09-12T20:10:08.687 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KYuRrmPm9o687t5IkchvnOEWHnM= 2022-09-12T20:10:08.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KYuRrmPm9o687t5IkchvnOEWHnM=', 'Content-Length': '0'} 2022-09-12T20:10:08.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000341fdff3e0d1514b-00631f9220-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.734 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:10:08.735 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-75 zones={test-zone3, test-zone2} 2022-09-12T20:10:08.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:10:08.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:10:08.737 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T20:10:08.799 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:10:08.800 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:10:08.856 INFO:teuthology.orchestra.run.smithi035.stdout:tenant: 2022-09-12T20:10:08.857 INFO:teuthology.orchestra.run.smithi035.stdout:bucket name: jdpiak-75 2022-09-12T20:10:08.868 INFO:teuthology.orchestra.run.smithi035.stdout:total entries: 20 2022-09-12T20:10:08.928 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:08.929 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:08.930 DEBUG:boto:Method: HEAD 2022-09-12T20:10:08.931 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:10:08.931 DEBUG:boto:Data: 2022-09-12T20:10:08.932 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.933 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:08.933 DEBUG:boto:Port: 8000 2022-09-12T20:10:08.934 DEBUG:boto:Params: {} 2022-09-12T20:10:08.935 DEBUG:boto:Token: None 2022-09-12T20:10:08.936 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/ 2022-09-12T20:10:08.936 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bjLBKh3m4IzzmseaN0ejaviBdTA= 2022-09-12T20:10:08.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bjLBKh3m4IzzmseaN0ejaviBdTA=', 'Content-Length': '0'} 2022-09-12T20:10:08.941 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011812a34a7ef6447-00631f9220-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.942 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:10:08.942 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:10:08.943 DEBUG:boto:Method: PUT 2022-09-12T20:10:08.943 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:10:08.944 DEBUG:boto:Data: 2022-09-12T20:10:08.944 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:10:08.945 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:08.945 DEBUG:boto:Port: 8000 2022-09-12T20:10:08.946 DEBUG:boto:Params: {} 2022-09-12T20:10:08.946 DEBUG:boto:Token: None 2022-09-12T20:10:08.947 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj29 2022-09-12T20:10:08.948 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BAz39tDkxbINO8PgY2FFjCMurz0= 2022-09-12T20:10:08.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BAz39tDkxbINO8PgY2FFjCMurz0='} 2022-09-12T20:10:08.954 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6bb566aef6328ad-00631f9220-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.955 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:08.956 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:08.956 DEBUG:boto:Method: HEAD 2022-09-12T20:10:08.957 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:10:08.958 DEBUG:boto:Data: 2022-09-12T20:10:08.958 DEBUG:boto:Headers: {} 2022-09-12T20:10:08.959 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:08.960 DEBUG:boto:Port: 8000 2022-09-12T20:10:08.960 DEBUG:boto:Params: {} 2022-09-12T20:10:08.961 DEBUG:boto:Token: None 2022-09-12T20:10:08.962 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/ 2022-09-12T20:10:08.963 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bjLBKh3m4IzzmseaN0ejaviBdTA= 2022-09-12T20:10:08.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bjLBKh3m4IzzmseaN0ejaviBdTA=', 'Content-Length': '0'} 2022-09-12T20:10:08.967 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd68e0b26b4f1a92-00631f9220-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:08.968 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:10:08.969 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:10:08.969 DEBUG:boto:Method: PUT 2022-09-12T20:10:08.970 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:10:08.970 DEBUG:boto:Data: 2022-09-12T20:10:08.971 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:10:08.971 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:08.972 DEBUG:boto:Port: 8000 2022-09-12T20:10:08.972 DEBUG:boto:Params: {} 2022-09-12T20:10:08.973 DEBUG:boto:Token: None 2022-09-12T20:10:08.973 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:10:08 GMT /jdpiak-75/obj58 2022-09-12T20:10:08.974 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MXaVKWTAjEUgn8ZC2A9nvSP6xZg= 2022-09-12T20:10:08.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:10:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MXaVKWTAjEUgn8ZC2A9nvSP6xZg='} 2022-09-12T20:10:09.018 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bafb2c4714463027-00631f9220-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:09.019 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:09.020 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:09.020 DEBUG:boto:Method: HEAD 2022-09-12T20:10:09.021 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:10:09.022 DEBUG:boto:Data: 2022-09-12T20:10:09.023 DEBUG:boto:Headers: {} 2022-09-12T20:10:09.023 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:09.024 DEBUG:boto:Port: 8000 2022-09-12T20:10:09.025 DEBUG:boto:Params: {} 2022-09-12T20:10:09.026 DEBUG:boto:Token: None 2022-09-12T20:10:09.027 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:09 GMT /jdpiak-75/ 2022-09-12T20:10:09.028 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YdrQY6JB5gNRVHhjf/d66vlsTSE= 2022-09-12T20:10:09.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YdrQY6JB5gNRVHhjf/d66vlsTSE=', 'Content-Length': '0'} 2022-09-12T20:10:09.032 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007dc7ecb1a9eb4ab7-00631f9221-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:09.033 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:10:09.033 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:10:09.034 DEBUG:boto:Method: PUT 2022-09-12T20:10:09.034 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:10:09.035 DEBUG:boto:Data: 2022-09-12T20:10:09.035 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:10:09.036 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:09.037 DEBUG:boto:Port: 8000 2022-09-12T20:10:09.037 DEBUG:boto:Params: {} 2022-09-12T20:10:09.038 DEBUG:boto:Token: None 2022-09-12T20:10:09.038 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:10:09 GMT /jdpiak-75/obj87 2022-09-12T20:10:09.039 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YWRVkOU4q26EB6TGAakU45//alU= 2022-09-12T20:10:09.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:10:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YWRVkOU4q26EB6TGAakU45//alU='} 2022-09-12T20:10:09.047 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e187df9db8b7f57f-00631f9221-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:09.047 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:09.048 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:09.048 DEBUG:boto:Method: HEAD 2022-09-12T20:10:09.049 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:10:09.049 DEBUG:boto:Data: 2022-09-12T20:10:09.050 DEBUG:boto:Headers: {} 2022-09-12T20:10:09.050 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:09.051 DEBUG:boto:Port: 8000 2022-09-12T20:10:09.051 DEBUG:boto:Params: {} 2022-09-12T20:10:09.052 DEBUG:boto:Token: None 2022-09-12T20:10:09.053 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:09 GMT /jdpiak-75/ 2022-09-12T20:10:09.053 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YdrQY6JB5gNRVHhjf/d66vlsTSE= 2022-09-12T20:10:09.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YdrQY6JB5gNRVHhjf/d66vlsTSE=', 'Content-Length': '0'} 2022-09-12T20:10:09.057 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c15e13f3096a7413-00631f9221-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:09.057 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:10:09.058 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:10:09.058 DEBUG:boto:Method: PUT 2022-09-12T20:10:09.059 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:10:09.059 DEBUG:boto:Data: 2022-09-12T20:10:09.060 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:10:09.060 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:09.061 DEBUG:boto:Port: 8000 2022-09-12T20:10:09.061 DEBUG:boto:Params: {} 2022-09-12T20:10:09.062 DEBUG:boto:Token: None 2022-09-12T20:10:09.063 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:10:09 GMT /jdpiak-75/obj116 2022-09-12T20:10:09.063 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IJb/Bh2lugdj8fUGXK/ArK8ahuc= 2022-09-12T20:10:09.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:10:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IJb/Bh2lugdj8fUGXK/ArK8ahuc='} 2022-09-12T20:10:09.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000064471375db826f90-00631f9221-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:09.107 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:10:09.107 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:10:09.108 DEBUG:boto:Method: HEAD 2022-09-12T20:10:09.108 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:10:09.109 DEBUG:boto:Data: 2022-09-12T20:10:09.109 DEBUG:boto:Headers: {} 2022-09-12T20:10:09.110 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:09.110 DEBUG:boto:Port: 8000 2022-09-12T20:10:09.111 DEBUG:boto:Params: {} 2022-09-12T20:10:09.111 DEBUG:boto:Token: None 2022-09-12T20:10:09.112 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:10:09 GMT /jdpiak-75/ 2022-09-12T20:10:09.112 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YdrQY6JB5gNRVHhjf/d66vlsTSE= 2022-09-12T20:10:09.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:10:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YdrQY6JB5gNRVHhjf/d66vlsTSE=', 'Content-Length': '0'} 2022-09-12T20:10:09.117 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dab323da63249d7a-00631f9221-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:09.117 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:10:09.118 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:10:09.118 DEBUG:boto:Method: PUT 2022-09-12T20:10:09.119 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:10:09.119 DEBUG:boto:Data: 2022-09-12T20:10:09.120 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:10:09.120 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:10:09.121 DEBUG:boto:Port: 8000 2022-09-12T20:10:09.121 DEBUG:boto:Params: {} 2022-09-12T20:10:09.122 DEBUG:boto:Token: None 2022-09-12T20:10:09.123 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:10:09 GMT /jdpiak-75/obj145 2022-09-12T20:10:09.123 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jT6nt3FQHXPPaH3S5n/dv3jLxgw= 2022-09-12T20:10:09.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:10:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jT6nt3FQHXPPaH3S5n/dv3jLxgw='} 2022-09-12T20:10:09.130 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce76a33863d47c19-00631f9221-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:10:09.131 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-75 2022-09-12T20:10:09.131 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:10:09.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:10:09.133 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T20:10:09.196 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:10:09.197 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:10:10.239 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:10:10.237+0000 7fd6be669600 1 waiting to reach latest gen 4, on 3.. 2022-09-12T20:10:15.240 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:10:15.238+0000 7fd6be669600 1 waiting to reach latest gen 4, on 3.. 2022-09-12T20:10:20.241 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:10:20.240+0000 7fd6be669600 1 waiting to reach latest gen 4, on 3.. 2022-09-12T20:10:25.244 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:10:25.242+0000 7fd6be669600 1 waiting for incremental sync to catch up: 2022-09-12T20:10:25.244 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:10:25.244 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000002.48.5, 00000000001.76.5, , , , , 00000000001.75.5, , , , , , , , , , , 00000000002.52.5, 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:10:30.246 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:10:30.244+0000 7fd6be669600 1 waiting for incremental sync to catch up: 2022-09-12T20:10:30.247 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:10:30.247 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000002.48.5, 00000000001.76.5, , , , , 00000000001.75.5, , , , , , , , , , , 00000000002.52.5, 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:10:35.248 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:10:35.246+0000 7fd6be669600 1 waiting for incremental sync to catch up: 2022-09-12T20:10:35.249 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:10:35.249 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000002.48.5, 00000000001.76.5, , , , , 00000000001.75.5, , , , , , , , , , , 00000000002.52.5, 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:10:40.251 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:10:40.248+0000 7fd6be669600 1 waiting for incremental sync to catch up: 2022-09-12T20:10:40.251 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:10:40.251 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000002.48.5, 00000000001.76.5, , , , , 00000000001.75.5, , , , , , , , , , , 00000000002.52.5, 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:10:45.253 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:10:45.251+0000 7fd6be669600 1 waiting for incremental sync to catch up: 2022-09-12T20:10:45.253 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:10:45.254 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000002.48.5, 00000000001.76.5, , , , , 00000000001.75.5, , , , , , , , , , , 00000000002.52.5, 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:10:50.255 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:10:50.253+0000 7fd6be669600 1 waiting for incremental sync to catch up: 2022-09-12T20:10:50.256 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:10:50.256 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000002.48.5, 00000000001.76.5, , , , , 00000000001.75.5, , , , , , , , , , , 00000000002.52.5, 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:10:55.257 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:10:55.255+0000 7fd6be669600 1 waiting for incremental sync to catch up: 2022-09-12T20:10:55.258 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:10:55.258 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000002.48.5, 00000000001.76.5, , , , , 00000000001.75.5, , , , , , , , , , , 00000000002.52.5, 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:11:00.260 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:11:00.258+0000 7fd6be669600 1 waiting for incremental sync to catch up: 2022-09-12T20:11:00.260 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:11:00.261 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000002.48.5, 00000000001.76.5, , , , , 00000000001.75.5, , , , , , , , , , , 00000000002.52.5, 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:11:05.262 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:11:05.261+0000 7fd6be669600 1 waiting for incremental sync to catch up: 2022-09-12T20:11:05.263 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:11:05.263 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000002.48.5, 00000000001.76.5, , , , , 00000000001.75.5, , , , , , , , , , , 00000000002.52.5, 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:11:10.264 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:11:10.263+0000 7fd6be669600 1 waiting for incremental sync to catch up: 2022-09-12T20:11:10.265 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:11:10.265 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000002.48.5, 00000000001.76.5, , , , , 00000000001.75.5, , , , , , , , , , , 00000000002.52.5, 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:11:15.267 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:11:15.265+0000 7fd6be669600 1 waiting for incremental sync to catch up: 2022-09-12T20:11:15.267 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:11:15.268 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000002.48.5, 00000000001.76.5, , , , , 00000000001.75.5, , , , , , , , , , , 00000000002.52.5, 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:11:20.269 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:11:20.267+0000 7fd6be669600 1 waiting for incremental sync to catch up: 2022-09-12T20:11:20.269 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:11:20.270 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000002.48.5, 00000000001.76.5, , , , , 00000000001.75.5, , , , , , , , , , , 00000000002.52.5, 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:11:25.271 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:11:25.269+0000 7fd6be669600 1 waiting for incremental sync to catch up: 2022-09-12T20:11:25.272 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000002.48.5, , , , , , 00000000001.75.5, , , , , , , , , , , , 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:11:25.272 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000002.48.5, 00000000001.76.5, , , , , 00000000001.75.5, , , , , , , , , , , 00000000002.52.5, 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:11:30.274 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:11:30.272+0000 7fd6be669600 1 waiting for incremental sync to catch up: 2022-09-12T20:11:30.274 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000002.48.5, , , , , , 00000000001.75.5, , , , , , , , , , , , 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:11:30.275 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000002.48.5, 00000000001.76.5, , , , , 00000000001.75.5, , , , , , , , , , , 00000000002.52.5, 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:11:35.276 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:11:35.274+0000 7fd6be669600 1 waiting for incremental sync to catch up: 2022-09-12T20:11:35.277 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000002.48.5, , , , , , 00000000001.75.5, , , , , , , , , , , , 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:11:35.277 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000002.48.5, 00000000001.76.5, , , , , 00000000001.75.5, , , , , , , , , , , 00000000002.52.5, 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:11:40.279 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:11:40.276+0000 7fd6be669600 1 waiting for incremental sync to catch up: 2022-09-12T20:11:40.279 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000002.48.5, , , , , , 00000000001.75.5, , , , , , , , , , , , 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:11:40.280 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000002.48.5, 00000000001.76.5, , , , , 00000000001.75.5, , , , , , , , , , , 00000000002.52.5, 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:11:45.281 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:11:45.279+0000 7fd6be669600 1 bucket sync caught up with source: 2022-09-12T20:11:45.281 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000002.48.5, 00000000001.76.5, , , , , 00000000001.75.5, , , , , , , , , , , 00000000002.52.5, 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:11:45.282 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000002.48.5, 00000000001.76.5, , , , , 00000000001.75.5, , , , , , , , , , , 00000000002.52.5, 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:11:45.282 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:11:45.279+0000 7fd6be669600 0 bucket checkpoint complete 2022-09-12T20:11:45.306 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-75 2022-09-12T20:11:45.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:11:45.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:11:45.310 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:11:45.376 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:11:45.377 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:11:45.434 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:11:45.432+0000 7fc56bd1f600 1 bucket sync caught up with source: 2022-09-12T20:11:45.434 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [00000000002.48.5, 00000000001.76.5, , , , , 00000000001.75.5, , , , , , , , , , , 00000000002.52.5, 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:11:45.434 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [00000000002.48.5, 00000000001.76.5, , , , , 00000000001.75.5, , , , , , , , , , , 00000000002.52.5, 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:11:45.434 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:11:45.432+0000 7fc56bd1f600 0 bucket checkpoint complete 2022-09-12T20:11:45.456 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-75 2022-09-12T20:11:45.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:11:45.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:11:45.458 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T20:11:45.519 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:11:45.520 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:11:45.571 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:11:45.569+0000 7f4b76e16600 1 waiting for incremental sync to catch up: 2022-09-12T20:11:45.571 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000002.48.5, 00000000001.76.5, , , , , , , , , , , , , , , , , 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:11:45.572 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000002.48.5, 00000000001.76.5, , , , , 00000000001.75.5, , , , , , , , , , , 00000000002.52.5, 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:11:50.574 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:11:50.573+0000 7f4b76e16600 1 waiting for incremental sync to catch up: 2022-09-12T20:11:50.575 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000002.48.5, 00000000001.76.5, , , , , , , , , , , , , , , , , 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:11:50.575 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000002.48.5, 00000000001.76.5, , , , , 00000000001.75.5, , , , , , , , , , , 00000000002.52.5, 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:11:55.577 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:11:55.575+0000 7f4b76e16600 1 bucket sync caught up with source: 2022-09-12T20:11:55.577 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [00000000002.48.5, 00000000001.76.5, , , , , 00000000001.75.5, , , , , , , , , , , 00000000002.52.5, 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:11:55.577 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [00000000002.48.5, 00000000001.76.5, , , , , 00000000001.75.5, , , , , , , , , , , 00000000002.52.5, 00000000001.59.5, , , , , , , , , , ] 2022-09-12T20:11:55.578 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:11:55.575+0000 7f4b76e16600 0 bucket checkpoint complete 2022-09-12T20:11:55.595 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-75 2022-09-12T20:11:55.596 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:11:55.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:11:55.597 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:11:55.662 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:11:55.663 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:11:55.706 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:11:55.704+0000 7fc7ffa1a600 1 bucket sync caught up with source: 2022-09-12T20:11:55.706 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.639.5, 00000000002.837.5, 00000000001.634.5, 00000000003.350.5, 00000000004.935.5, 00000000001.434.5, 00000000001.1155.5, 00000000002.550.5, 00000000005.925.5, 00000000004.580.5] 2022-09-12T20:11:55.706 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.639.5, 00000000002.837.5, 00000000001.634.5, 00000000003.350.5, 00000000004.935.5, 00000000001.434.5, 00000000001.1155.5, 00000000002.550.5, 00000000005.925.5, 00000000004.580.5] 2022-09-12T20:11:55.706 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:11:55.704+0000 7fc7ffa1a600 0 bucket checkpoint complete 2022-09-12T20:11:55.721 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-75 2022-09-12T20:11:55.722 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:11:55.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:11:55.723 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:11:55.782 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:11:55.783 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:11:55.819 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:11:55.818+0000 7faa04492600 1 bucket sync caught up with source: 2022-09-12T20:11:55.820 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.639.5, 00000000002.837.5, 00000000001.634.5, 00000000003.350.5, 00000000004.935.5, 00000000001.434.5, 00000000001.1155.5, 00000000002.550.5, 00000000005.925.5, 00000000004.580.5] 2022-09-12T20:11:55.820 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.639.5, 00000000002.837.5, 00000000001.634.5, 00000000003.350.5, 00000000004.935.5, 00000000001.434.5, 00000000001.1155.5, 00000000002.550.5, 00000000005.925.5, 00000000004.580.5] 2022-09-12T20:11:55.820 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:11:55.818+0000 7faa04492600 0 bucket checkpoint complete 2022-09-12T20:11:55.833 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-75 2022-09-12T20:11:55.833 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-75 2022-09-12T20:11:55.834 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:11:55.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:11:55.835 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:11:55.892 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:11:55.892 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:11:55.939 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:11:55.938+0000 7fac99bc7600 1 waiting for incremental sync to catch up: 2022-09-12T20:11:55.939 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.296.5, 00000000002.337.5, 00000000001.279.5, 00000000003.348.5, 00000000004.551.5, 00000000001.338.5, , 00000000002.320.5, 00000000005.541.5, 00000000004.494.5] 2022-09-12T20:11:55.940 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.296.5, 00000000002.337.5, 00000000001.279.5, 00000000003.348.5, 00000000004.551.5, 00000000001.338.5, 00000000001.466.5, 00000000002.320.5, 00000000005.541.5, 00000000004.494.5] 2022-09-12T20:12:00.941 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:12:00.939+0000 7fac99bc7600 1 waiting for incremental sync to catch up: 2022-09-12T20:12:00.941 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.296.5, 00000000002.337.5, 00000000001.279.5, 00000000003.348.5, 00000000004.551.5, 00000000001.338.5, , 00000000002.320.5, 00000000005.541.5, 00000000004.494.5] 2022-09-12T20:12:00.942 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.296.5, 00000000002.337.5, 00000000001.279.5, 00000000003.348.5, 00000000004.551.5, 00000000001.338.5, 00000000001.466.5, 00000000002.320.5, 00000000005.541.5, 00000000004.494.5] 2022-09-12T20:12:05.942 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:12:05.941+0000 7fac99bc7600 1 bucket sync caught up with source: 2022-09-12T20:12:05.943 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000002.296.5, 00000000002.337.5, 00000000001.279.5, 00000000003.348.5, 00000000004.551.5, 00000000001.338.5, 00000000001.466.5, 00000000002.320.5, 00000000005.541.5, 00000000004.494.5] 2022-09-12T20:12:05.943 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000002.296.5, 00000000002.337.5, 00000000001.279.5, 00000000003.348.5, 00000000004.551.5, 00000000001.338.5, 00000000001.466.5, 00000000002.320.5, 00000000005.541.5, 00000000004.494.5] 2022-09-12T20:12:05.943 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:12:05.941+0000 7fac99bc7600 0 bucket checkpoint complete 2022-09-12T20:12:05.962 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-75 2022-09-12T20:12:05.962 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:12:05.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:12:05.964 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T20:12:06.027 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:12:06.027 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:12:06.067 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:12:06.065+0000 7f2374282600 1 bucket sync caught up with source: 2022-09-12T20:12:06.067 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000002.296.5, 00000000002.337.5, 00000000001.279.5, 00000000003.348.5, 00000000004.551.5, 00000000001.338.5, 00000000001.466.5, 00000000002.320.5, 00000000005.541.5, 00000000004.494.5] 2022-09-12T20:12:06.068 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000002.296.5, 00000000002.337.5, 00000000001.279.5, 00000000003.348.5, 00000000004.551.5, 00000000001.338.5, 00000000001.466.5, 00000000002.320.5, 00000000005.541.5, 00000000004.494.5] 2022-09-12T20:12:06.068 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:12:06.065+0000 7f2374282600 0 bucket checkpoint complete 2022-09-12T20:12:06.087 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-75 2022-09-12T20:12:06.087 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-75 zones={test-zone2, test-zone1} 2022-09-12T20:12:06.088 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:06.089 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:06.089 DEBUG:boto:Method: HEAD 2022-09-12T20:12:06.090 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:12:06.090 DEBUG:boto:Data: 2022-09-12T20:12:06.091 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.091 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.092 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.092 DEBUG:boto:Params: {} 2022-09-12T20:12:06.093 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T20:12:06.094 DEBUG:boto:Token: None 2022-09-12T20:12:06.095 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/ 2022-09-12T20:12:06.096 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vsKjwFBkwouspTsjOr0xCeXk0mw= 2022-09-12T20:12:06.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vsKjwFBkwouspTsjOr0xCeXk0mw=', 'Content-Length': '0'} 2022-09-12T20:12:06.101 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008566d95430f322a8-00631f9296-189d-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.101 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:06.102 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:06.103 DEBUG:boto:Method: HEAD 2022-09-12T20:12:06.103 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:12:06.104 DEBUG:boto:Data: 2022-09-12T20:12:06.104 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.105 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.105 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.106 DEBUG:boto:Params: {} 2022-09-12T20:12:06.106 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T20:12:06.107 DEBUG:boto:Token: None 2022-09-12T20:12:06.108 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/ 2022-09-12T20:12:06.108 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vsKjwFBkwouspTsjOr0xCeXk0mw= 2022-09-12T20:12:06.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vsKjwFBkwouspTsjOr0xCeXk0mw=', 'Content-Length': '0'} 2022-09-12T20:12:06.115 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8734eddd2b3308f-00631f9296-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.116 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T20:12:06.116 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:06.117 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:06.117 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:06.118 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:06.119 DEBUG:boto:Method: GET 2022-09-12T20:12:06.119 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:06.120 DEBUG:boto:Data: 2022-09-12T20:12:06.120 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.121 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.121 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.122 DEBUG:boto:Params: {} 2022-09-12T20:12:06.122 DEBUG:boto:Token: None 2022-09-12T20:12:06.123 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/?versions 2022-09-12T20:12:06.124 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qQHyY13B6NZ9zD0ARXW+Y2NXn/c= 2022-09-12T20:12:06.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qQHyY13B6NZ9zD0ARXW+Y2NXn/c=', 'Content-Length': '0'} 2022-09-12T20:12:06.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b8cf1bc9672fc23-00631f9296-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.128 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:06.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:12:06.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-12T20:12:06.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-12T20:12:06.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-12T20:12:06.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:12:06.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:12:06.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:12:06.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-12T20:12:06.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-12T20:12:06.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:12:06.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:12:06.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:12:06.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:12:06.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-12T20:12:06.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:12:06.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:12:06.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:12:06.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-12T20:12:06.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:12:06.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-12T20:12:06.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:12:06.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:12:06.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-12T20:12:06.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-12T20:12:06.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:12:06.142 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T20:12:06.143 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:06.143 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:06.144 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:06.144 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:06.145 DEBUG:boto:Method: GET 2022-09-12T20:12:06.145 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:06.146 DEBUG:boto:Data: 2022-09-12T20:12:06.146 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.147 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.148 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.148 DEBUG:boto:Params: {} 2022-09-12T20:12:06.149 DEBUG:boto:Token: None 2022-09-12T20:12:06.149 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/?versions 2022-09-12T20:12:06.150 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qQHyY13B6NZ9zD0ARXW+Y2NXn/c= 2022-09-12T20:12:06.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qQHyY13B6NZ9zD0ARXW+Y2NXn/c=', 'Content-Length': '0'} 2022-09-12T20:12:06.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c2234a342d45746-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.156 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:06.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:12:06.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-12T20:12:06.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-12T20:12:06.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-12T20:12:06.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:12:06.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:12:06.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:12:06.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-12T20:12:06.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-12T20:12:06.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:12:06.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:12:06.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:12:06.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:12:06.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-12T20:12:06.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:12:06.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:12:06.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:12:06.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-12T20:12:06.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:12:06.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-12T20:12:06.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:12:06.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:12:06.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-12T20:12:06.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-12T20:12:06.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:12:06.170 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:06.170 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:06.171 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:06.171 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:06.172 DEBUG:boto:Method: GET 2022-09-12T20:12:06.172 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:06.173 DEBUG:boto:Data: 2022-09-12T20:12:06.173 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.174 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.174 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.175 DEBUG:boto:Params: {} 2022-09-12T20:12:06.175 DEBUG:boto:Token: None 2022-09-12T20:12:06.176 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/?versions 2022-09-12T20:12:06.177 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qQHyY13B6NZ9zD0ARXW+Y2NXn/c= 2022-09-12T20:12:06.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qQHyY13B6NZ9zD0ARXW+Y2NXn/c=', 'Content-Length': '0'} 2022-09-12T20:12:06.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024bba3c9230ddc13-00631f9296-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.180 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:06.182 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:06.183 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:06.183 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:06.184 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:06.184 DEBUG:boto:Method: GET 2022-09-12T20:12:06.185 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:06.185 DEBUG:boto:Data: 2022-09-12T20:12:06.186 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.186 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.187 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.187 DEBUG:boto:Params: {} 2022-09-12T20:12:06.188 DEBUG:boto:Token: None 2022-09-12T20:12:06.188 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/?versions 2022-09-12T20:12:06.189 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qQHyY13B6NZ9zD0ARXW+Y2NXn/c= 2022-09-12T20:12:06.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qQHyY13B6NZ9zD0ARXW+Y2NXn/c=', 'Content-Length': '0'} 2022-09-12T20:12:06.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000523983afb3043a2a-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.195 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:06.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:12:06.197 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:06.198 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:06.198 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:06.199 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:06.199 DEBUG:boto:Method: GET 2022-09-12T20:12:06.200 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:06.201 DEBUG:boto:Data: 2022-09-12T20:12:06.201 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.202 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.202 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.203 DEBUG:boto:Params: {} 2022-09-12T20:12:06.203 DEBUG:boto:Token: None 2022-09-12T20:12:06.204 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:06.204 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DlLgJ/IPVgfL8dF/T0i09GDIYJc= 2022-09-12T20:12:06.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DlLgJ/IPVgfL8dF/T0i09GDIYJc=', 'Content-Length': '0'} 2022-09-12T20:12:06.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000511699eb014d18b3-00631f9296-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.208 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:06.208 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:06.209 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:06.209 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:06.210 DEBUG:boto:Method: GET 2022-09-12T20:12:06.210 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:06.211 DEBUG:boto:Data: 2022-09-12T20:12:06.212 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.212 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.213 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.213 DEBUG:boto:Params: {} 2022-09-12T20:12:06.214 DEBUG:boto:Token: None 2022-09-12T20:12:06.215 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:06.215 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DlLgJ/IPVgfL8dF/T0i09GDIYJc= 2022-09-12T20:12:06.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DlLgJ/IPVgfL8dF/T0i09GDIYJc=', 'Content-Length': '0'} 2022-09-12T20:12:06.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015453f69157766d1-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.262 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:06.263 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:06.263 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:06.264 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:06.264 DEBUG:boto:Method: HEAD 2022-09-12T20:12:06.265 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:06.265 DEBUG:boto:Data: 2022-09-12T20:12:06.266 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.266 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.267 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.267 DEBUG:boto:Params: {} 2022-09-12T20:12:06.268 DEBUG:boto:Token: None 2022-09-12T20:12:06.269 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:06.269 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0q/TIqX78b47SC+/H4tLQEFPoRc= 2022-09-12T20:12:06.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0q/TIqX78b47SC+/H4tLQEFPoRc=', 'Content-Length': '0'} 2022-09-12T20:12:06.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095b1579c851acdd3-00631f9296-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.272 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:06.273 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:06.273 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:06.274 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:06.275 DEBUG:boto:Method: HEAD 2022-09-12T20:12:06.275 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:06.276 DEBUG:boto:Data: 2022-09-12T20:12:06.276 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.277 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.277 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.278 DEBUG:boto:Params: {} 2022-09-12T20:12:06.278 DEBUG:boto:Token: None 2022-09-12T20:12:06.279 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:06.279 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0q/TIqX78b47SC+/H4tLQEFPoRc= 2022-09-12T20:12:06.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0q/TIqX78b47SC+/H4tLQEFPoRc=', 'Content-Length': '0'} 2022-09-12T20:12:06.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087284a7dafc616f4-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:12:06.283 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:06.284 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:06.284 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:06.285 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:06.285 DEBUG:boto:Method: GET 2022-09-12T20:12:06.286 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:06.286 DEBUG:boto:Data: 2022-09-12T20:12:06.287 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.287 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.288 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.288 DEBUG:boto:Params: {} 2022-09-12T20:12:06.289 DEBUG:boto:Token: None 2022-09-12T20:12:06.289 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:06.290 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DlLgJ/IPVgfL8dF/T0i09GDIYJc= 2022-09-12T20:12:06.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DlLgJ/IPVgfL8dF/T0i09GDIYJc=', 'Content-Length': '0'} 2022-09-12T20:12:06.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ade7e5f1efb60d86-00631f9296-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.293 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:06.294 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:06.295 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:06.295 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:06.296 DEBUG:boto:Method: GET 2022-09-12T20:12:06.296 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:06.297 DEBUG:boto:Data: 2022-09-12T20:12:06.297 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.298 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.298 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.299 DEBUG:boto:Params: {} 2022-09-12T20:12:06.299 DEBUG:boto:Token: None 2022-09-12T20:12:06.300 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:06.301 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DlLgJ/IPVgfL8dF/T0i09GDIYJc= 2022-09-12T20:12:06.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DlLgJ/IPVgfL8dF/T0i09GDIYJc=', 'Content-Length': '0'} 2022-09-12T20:12:06.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000192fb7a632c5a51d-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.304 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:06.305 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:06.305 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:12:06.306 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:12:06.306 DEBUG:boto:Method: GET 2022-09-12T20:12:06.307 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:12:06.307 DEBUG:boto:Data: 2022-09-12T20:12:06.308 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.308 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.309 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.309 DEBUG:boto:Params: {} 2022-09-12T20:12:06.310 DEBUG:boto:Token: None 2022-09-12T20:12:06.311 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/ 2022-09-12T20:12:06.311 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ= 2022-09-12T20:12:06.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ=', 'Content-Length': '0'} 2022-09-12T20:12:06.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002de946c3a4a6fcf-00631f9296-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.315 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:06.316 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:06.317 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:06.317 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:12:06.318 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:12:06.318 DEBUG:boto:Method: GET 2022-09-12T20:12:06.319 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:12:06.319 DEBUG:boto:Data: 2022-09-12T20:12:06.320 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.321 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.321 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.322 DEBUG:boto:Params: {} 2022-09-12T20:12:06.322 DEBUG:boto:Token: None 2022-09-12T20:12:06.323 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/ 2022-09-12T20:12:06.323 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ= 2022-09-12T20:12:06.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ=', 'Content-Length': '0'} 2022-09-12T20:12:06.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000950d47516157b67d-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.329 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:06.330 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:06.331 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:06.331 DEBUG:boto:Method: HEAD 2022-09-12T20:12:06.332 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:12:06.332 DEBUG:boto:Data: 2022-09-12T20:12:06.333 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.333 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.334 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.334 DEBUG:boto:Params: {} 2022-09-12T20:12:06.335 DEBUG:boto:Token: None 2022-09-12T20:12:06.335 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj11 2022-09-12T20:12:06.336 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:C/1H0JHyvpIHmB/XOybdjxazMYc= 2022-09-12T20:12:06.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:C/1H0JHyvpIHmB/XOybdjxazMYc=', 'Content-Length': '0'} 2022-09-12T20:12:06.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d664420e46cbf017-00631f9296-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.339 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:06.340 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:06.341 DEBUG:boto:Method: HEAD 2022-09-12T20:12:06.341 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:12:06.342 DEBUG:boto:Data: 2022-09-12T20:12:06.342 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.343 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.343 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.344 DEBUG:boto:Params: {} 2022-09-12T20:12:06.344 DEBUG:boto:Token: None 2022-09-12T20:12:06.345 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj11 2022-09-12T20:12:06.345 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:C/1H0JHyvpIHmB/XOybdjxazMYc= 2022-09-12T20:12:06.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:C/1H0JHyvpIHmB/XOybdjxazMYc=', 'Content-Length': '0'} 2022-09-12T20:12:06.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000703f171fbb1388fb-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:12:06.349 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:06.350 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:06.351 DEBUG:boto:Method: GET 2022-09-12T20:12:06.351 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:12:06.352 DEBUG:boto:Data: 2022-09-12T20:12:06.352 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.353 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.353 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.354 DEBUG:boto:Params: {} 2022-09-12T20:12:06.354 DEBUG:boto:Token: None 2022-09-12T20:12:06.355 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj11 2022-09-12T20:12:06.356 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X1xy5l2l4eDqSt75FeS0kEdDBHM= 2022-09-12T20:12:06.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X1xy5l2l4eDqSt75FeS0kEdDBHM=', 'Content-Length': '0'} 2022-09-12T20:12:06.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015d190f00bbf0d67-00631f9296-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.359 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:06.360 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:06.360 DEBUG:boto:Method: GET 2022-09-12T20:12:06.361 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:12:06.362 DEBUG:boto:Data: 2022-09-12T20:12:06.362 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.363 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.363 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.364 DEBUG:boto:Params: {} 2022-09-12T20:12:06.364 DEBUG:boto:Token: None 2022-09-12T20:12:06.365 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj11 2022-09-12T20:12:06.365 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X1xy5l2l4eDqSt75FeS0kEdDBHM= 2022-09-12T20:12:06.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X1xy5l2l4eDqSt75FeS0kEdDBHM=', 'Content-Length': '0'} 2022-09-12T20:12:06.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d0719d986492b63-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.369 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:06.370 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:06.370 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:12:06.371 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:12:06.371 DEBUG:boto:Method: GET 2022-09-12T20:12:06.372 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:12:06.372 DEBUG:boto:Data: 2022-09-12T20:12:06.373 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.373 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.374 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.374 DEBUG:boto:Params: {} 2022-09-12T20:12:06.375 DEBUG:boto:Token: None 2022-09-12T20:12:06.376 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/ 2022-09-12T20:12:06.376 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ= 2022-09-12T20:12:06.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ=', 'Content-Length': '0'} 2022-09-12T20:12:06.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c430d4bbb3183ac-00631f9296-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.422 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:06.423 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:06.424 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:06.424 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:12:06.425 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:12:06.425 DEBUG:boto:Method: GET 2022-09-12T20:12:06.426 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:12:06.426 DEBUG:boto:Data: 2022-09-12T20:12:06.427 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.428 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.428 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.429 DEBUG:boto:Params: {} 2022-09-12T20:12:06.429 DEBUG:boto:Token: None 2022-09-12T20:12:06.430 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/ 2022-09-12T20:12:06.431 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ= 2022-09-12T20:12:06.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ=', 'Content-Length': '0'} 2022-09-12T20:12:06.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2c9fcd32ac66648-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.436 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:06.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:12:06.437 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:06.438 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:06.439 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:06.439 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:06.440 DEBUG:boto:Method: GET 2022-09-12T20:12:06.440 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:06.441 DEBUG:boto:Data: 2022-09-12T20:12:06.441 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.442 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.442 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.443 DEBUG:boto:Params: {} 2022-09-12T20:12:06.443 DEBUG:boto:Token: None 2022-09-12T20:12:06.444 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:06.445 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zj+LuEH/37Yo2EZu3EhYLpI9pQw= 2022-09-12T20:12:06.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zj+LuEH/37Yo2EZu3EhYLpI9pQw=', 'Content-Length': '0'} 2022-09-12T20:12:06.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d2942eceb2ceea8-00631f9296-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.448 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:06.449 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:06.449 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:06.450 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:06.450 DEBUG:boto:Method: GET 2022-09-12T20:12:06.451 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:06.451 DEBUG:boto:Data: 2022-09-12T20:12:06.452 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.452 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.453 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.453 DEBUG:boto:Params: {} 2022-09-12T20:12:06.454 DEBUG:boto:Token: None 2022-09-12T20:12:06.455 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:06.455 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zj+LuEH/37Yo2EZu3EhYLpI9pQw= 2022-09-12T20:12:06.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zj+LuEH/37Yo2EZu3EhYLpI9pQw=', 'Content-Length': '0'} 2022-09-12T20:12:06.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd1d9374cd044dfe-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.502 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:06.503 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:06.503 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:06.504 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:06.504 DEBUG:boto:Method: HEAD 2022-09-12T20:12:06.505 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:06.505 DEBUG:boto:Data: 2022-09-12T20:12:06.506 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.506 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.507 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.507 DEBUG:boto:Params: {} 2022-09-12T20:12:06.508 DEBUG:boto:Token: None 2022-09-12T20:12:06.509 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:06.509 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cpd1eCm3rq6xiBrjNkiXpF+HRQ8= 2022-09-12T20:12:06.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cpd1eCm3rq6xiBrjNkiXpF+HRQ8=', 'Content-Length': '0'} 2022-09-12T20:12:06.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008910060bd1bbf9f4-00631f9296-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.513 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:06.513 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:06.514 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:06.514 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:06.515 DEBUG:boto:Method: HEAD 2022-09-12T20:12:06.515 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:06.516 DEBUG:boto:Data: 2022-09-12T20:12:06.516 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.517 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.517 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.518 DEBUG:boto:Params: {} 2022-09-12T20:12:06.518 DEBUG:boto:Token: None 2022-09-12T20:12:06.519 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:06.520 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cpd1eCm3rq6xiBrjNkiXpF+HRQ8= 2022-09-12T20:12:06.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cpd1eCm3rq6xiBrjNkiXpF+HRQ8=', 'Content-Length': '0'} 2022-09-12T20:12:06.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061d198100767d134-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:12:06.524 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:06.524 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:06.525 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:06.525 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:06.526 DEBUG:boto:Method: GET 2022-09-12T20:12:06.527 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:06.527 DEBUG:boto:Data: 2022-09-12T20:12:06.528 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.528 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.529 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.529 DEBUG:boto:Params: {} 2022-09-12T20:12:06.530 DEBUG:boto:Token: None 2022-09-12T20:12:06.530 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:06.531 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zj+LuEH/37Yo2EZu3EhYLpI9pQw= 2022-09-12T20:12:06.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zj+LuEH/37Yo2EZu3EhYLpI9pQw=', 'Content-Length': '0'} 2022-09-12T20:12:06.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008950d90d9072fecb-00631f9296-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.534 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:06.535 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:06.535 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:06.536 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:06.536 DEBUG:boto:Method: GET 2022-09-12T20:12:06.537 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:06.537 DEBUG:boto:Data: 2022-09-12T20:12:06.538 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.538 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.539 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.540 DEBUG:boto:Params: {} 2022-09-12T20:12:06.540 DEBUG:boto:Token: None 2022-09-12T20:12:06.541 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:06.541 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zj+LuEH/37Yo2EZu3EhYLpI9pQw= 2022-09-12T20:12:06.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zj+LuEH/37Yo2EZu3EhYLpI9pQw=', 'Content-Length': '0'} 2022-09-12T20:12:06.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f0b4835ff384c2a-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.545 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:06.545 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:06.546 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:12:06.546 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:12:06.547 DEBUG:boto:Method: GET 2022-09-12T20:12:06.548 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:12:06.548 DEBUG:boto:Data: 2022-09-12T20:12:06.549 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.549 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.550 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.550 DEBUG:boto:Params: {} 2022-09-12T20:12:06.551 DEBUG:boto:Token: None 2022-09-12T20:12:06.551 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/ 2022-09-12T20:12:06.552 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ= 2022-09-12T20:12:06.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ=', 'Content-Length': '0'} 2022-09-12T20:12:06.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2f784f0a496e7e7-00631f9296-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.556 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:06.557 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:06.557 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:06.558 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:12:06.558 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:12:06.559 DEBUG:boto:Method: GET 2022-09-12T20:12:06.559 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:12:06.560 DEBUG:boto:Data: 2022-09-12T20:12:06.560 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.561 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.561 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.562 DEBUG:boto:Params: {} 2022-09-12T20:12:06.563 DEBUG:boto:Token: None 2022-09-12T20:12:06.563 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/ 2022-09-12T20:12:06.564 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ= 2022-09-12T20:12:06.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ=', 'Content-Length': '0'} 2022-09-12T20:12:06.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000864a0d13b61b176c-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.569 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:06.570 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:06.571 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:06.571 DEBUG:boto:Method: HEAD 2022-09-12T20:12:06.572 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:12:06.572 DEBUG:boto:Data: 2022-09-12T20:12:06.573 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.573 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.574 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.574 DEBUG:boto:Params: {} 2022-09-12T20:12:06.575 DEBUG:boto:Token: None 2022-09-12T20:12:06.575 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj115 2022-09-12T20:12:06.576 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kdkugAyjIkdsTfk0Dh/nyiMINME= 2022-09-12T20:12:06.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kdkugAyjIkdsTfk0Dh/nyiMINME=', 'Content-Length': '0'} 2022-09-12T20:12:06.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b1a9a1252880c3d-00631f9296-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.579 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:06.580 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:06.580 DEBUG:boto:Method: HEAD 2022-09-12T20:12:06.581 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:12:06.581 DEBUG:boto:Data: 2022-09-12T20:12:06.582 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.582 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.583 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.583 DEBUG:boto:Params: {} 2022-09-12T20:12:06.584 DEBUG:boto:Token: None 2022-09-12T20:12:06.584 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj115 2022-09-12T20:12:06.585 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kdkugAyjIkdsTfk0Dh/nyiMINME= 2022-09-12T20:12:06.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kdkugAyjIkdsTfk0Dh/nyiMINME=', 'Content-Length': '0'} 2022-09-12T20:12:06.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac514ef06ae8a0f6-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:12:06.589 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:06.590 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:06.590 DEBUG:boto:Method: GET 2022-09-12T20:12:06.591 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:12:06.591 DEBUG:boto:Data: 2022-09-12T20:12:06.592 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.592 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.593 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.593 DEBUG:boto:Params: {} 2022-09-12T20:12:06.594 DEBUG:boto:Token: None 2022-09-12T20:12:06.595 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj115 2022-09-12T20:12:06.595 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nkPFYT98rHemWHPw4Hih0TZfaqQ= 2022-09-12T20:12:06.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nkPFYT98rHemWHPw4Hih0TZfaqQ=', 'Content-Length': '0'} 2022-09-12T20:12:06.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3978a6135a14e36-00631f9296-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.642 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:06.643 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:06.643 DEBUG:boto:Method: GET 2022-09-12T20:12:06.644 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:12:06.644 DEBUG:boto:Data: 2022-09-12T20:12:06.645 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.645 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.646 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.646 DEBUG:boto:Params: {} 2022-09-12T20:12:06.647 DEBUG:boto:Token: None 2022-09-12T20:12:06.648 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj115 2022-09-12T20:12:06.648 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nkPFYT98rHemWHPw4Hih0TZfaqQ= 2022-09-12T20:12:06.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nkPFYT98rHemWHPw4Hih0TZfaqQ=', 'Content-Length': '0'} 2022-09-12T20:12:06.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e60294c94ee59b2a-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.652 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:06.652 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:06.653 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:12:06.654 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:12:06.654 DEBUG:boto:Method: GET 2022-09-12T20:12:06.655 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:12:06.655 DEBUG:boto:Data: 2022-09-12T20:12:06.656 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.656 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.657 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.657 DEBUG:boto:Params: {} 2022-09-12T20:12:06.658 DEBUG:boto:Token: None 2022-09-12T20:12:06.659 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/ 2022-09-12T20:12:06.659 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ= 2022-09-12T20:12:06.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ=', 'Content-Length': '0'} 2022-09-12T20:12:06.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1b522c2cf293cf6-00631f9296-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.663 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:06.664 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:06.664 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:06.665 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:12:06.665 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:12:06.666 DEBUG:boto:Method: GET 2022-09-12T20:12:06.666 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:12:06.667 DEBUG:boto:Data: 2022-09-12T20:12:06.667 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.668 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.669 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.669 DEBUG:boto:Params: {} 2022-09-12T20:12:06.670 DEBUG:boto:Token: None 2022-09-12T20:12:06.670 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/ 2022-09-12T20:12:06.671 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ= 2022-09-12T20:12:06.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ=', 'Content-Length': '0'} 2022-09-12T20:12:06.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d48216de7f51e248-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.676 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:06.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:12:06.678 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:06.678 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:06.679 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:06.679 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:06.680 DEBUG:boto:Method: GET 2022-09-12T20:12:06.680 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:06.681 DEBUG:boto:Data: 2022-09-12T20:12:06.681 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.682 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.682 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.683 DEBUG:boto:Params: {} 2022-09-12T20:12:06.683 DEBUG:boto:Token: None 2022-09-12T20:12:06.684 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:06.685 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qm0kCITs7Q/1uw24ZxkY71WKcY0= 2022-09-12T20:12:06.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qm0kCITs7Q/1uw24ZxkY71WKcY0=', 'Content-Length': '0'} 2022-09-12T20:12:06.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ded97d7a361769fc-00631f9296-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.688 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:06.689 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:06.690 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:06.690 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:06.691 DEBUG:boto:Method: GET 2022-09-12T20:12:06.691 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:06.692 DEBUG:boto:Data: 2022-09-12T20:12:06.692 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.693 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.693 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.694 DEBUG:boto:Params: {} 2022-09-12T20:12:06.694 DEBUG:boto:Token: None 2022-09-12T20:12:06.695 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:06.696 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qm0kCITs7Q/1uw24ZxkY71WKcY0= 2022-09-12T20:12:06.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qm0kCITs7Q/1uw24ZxkY71WKcY0=', 'Content-Length': '0'} 2022-09-12T20:12:06.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004865e6b012820fbb-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.699 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:06.700 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:06.700 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:06.701 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:06.701 DEBUG:boto:Method: HEAD 2022-09-12T20:12:06.702 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:06.702 DEBUG:boto:Data: 2022-09-12T20:12:06.703 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.703 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.704 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.704 DEBUG:boto:Params: {} 2022-09-12T20:12:06.705 DEBUG:boto:Token: None 2022-09-12T20:12:06.705 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:06.706 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BN903DnAzrklaj2ud1PCwt0zlM8= 2022-09-12T20:12:06.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BN903DnAzrklaj2ud1PCwt0zlM8=', 'Content-Length': '0'} 2022-09-12T20:12:06.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f8395cdf1b3201d-00631f9296-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.709 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:06.710 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:06.710 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:06.711 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:06.711 DEBUG:boto:Method: HEAD 2022-09-12T20:12:06.712 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:06.712 DEBUG:boto:Data: 2022-09-12T20:12:06.713 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.713 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.714 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.722 DEBUG:boto:Params: {} 2022-09-12T20:12:06.723 DEBUG:boto:Token: None 2022-09-12T20:12:06.723 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:06.724 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BN903DnAzrklaj2ud1PCwt0zlM8= 2022-09-12T20:12:06.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BN903DnAzrklaj2ud1PCwt0zlM8=', 'Content-Length': '0'} 2022-09-12T20:12:06.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000060a3b2afb3c6346-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:12:06.728 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:06.728 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:06.729 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:06.729 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:06.730 DEBUG:boto:Method: GET 2022-09-12T20:12:06.730 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:06.731 DEBUG:boto:Data: 2022-09-12T20:12:06.731 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.732 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.732 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.733 DEBUG:boto:Params: {} 2022-09-12T20:12:06.734 DEBUG:boto:Token: None 2022-09-12T20:12:06.734 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:06.735 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qm0kCITs7Q/1uw24ZxkY71WKcY0= 2022-09-12T20:12:06.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qm0kCITs7Q/1uw24ZxkY71WKcY0=', 'Content-Length': '0'} 2022-09-12T20:12:06.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014cb54838f060f7d-00631f9296-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.738 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:06.739 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:06.739 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:06.740 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:06.740 DEBUG:boto:Method: GET 2022-09-12T20:12:06.741 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:06.741 DEBUG:boto:Data: 2022-09-12T20:12:06.742 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.743 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.743 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.744 DEBUG:boto:Params: {} 2022-09-12T20:12:06.744 DEBUG:boto:Token: None 2022-09-12T20:12:06.745 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:06.745 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qm0kCITs7Q/1uw24ZxkY71WKcY0= 2022-09-12T20:12:06.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qm0kCITs7Q/1uw24ZxkY71WKcY0=', 'Content-Length': '0'} 2022-09-12T20:12:06.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a6e4c05f4c95815-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.790 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:06.791 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:06.791 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:12:06.792 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:12:06.792 DEBUG:boto:Method: GET 2022-09-12T20:12:06.793 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:12:06.793 DEBUG:boto:Data: 2022-09-12T20:12:06.794 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.794 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.795 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.796 DEBUG:boto:Params: {} 2022-09-12T20:12:06.796 DEBUG:boto:Token: None 2022-09-12T20:12:06.797 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/ 2022-09-12T20:12:06.797 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ= 2022-09-12T20:12:06.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ=', 'Content-Length': '0'} 2022-09-12T20:12:06.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002adfbe1be2f5f75e-00631f9296-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.801 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:06.802 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:06.803 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:06.803 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:12:06.804 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:12:06.804 DEBUG:boto:Method: GET 2022-09-12T20:12:06.805 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:12:06.806 DEBUG:boto:Data: 2022-09-12T20:12:06.806 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.807 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.807 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.808 DEBUG:boto:Params: {} 2022-09-12T20:12:06.808 DEBUG:boto:Token: None 2022-09-12T20:12:06.809 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/ 2022-09-12T20:12:06.810 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ= 2022-09-12T20:12:06.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ=', 'Content-Length': '0'} 2022-09-12T20:12:06.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007593cf349fc406d-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.815 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:06.816 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:06.817 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:06.817 DEBUG:boto:Method: HEAD 2022-09-12T20:12:06.818 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:12:06.818 DEBUG:boto:Data: 2022-09-12T20:12:06.819 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.819 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.820 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.820 DEBUG:boto:Params: {} 2022-09-12T20:12:06.821 DEBUG:boto:Token: None 2022-09-12T20:12:06.821 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj116 2022-09-12T20:12:06.822 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:U/lywhVbreZhdD488gUyO626Lws= 2022-09-12T20:12:06.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:U/lywhVbreZhdD488gUyO626Lws=', 'Content-Length': '0'} 2022-09-12T20:12:06.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009122158a64e279f2-00631f9296-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.825 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:06.826 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:06.826 DEBUG:boto:Method: HEAD 2022-09-12T20:12:06.827 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:12:06.827 DEBUG:boto:Data: 2022-09-12T20:12:06.828 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.828 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.829 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.829 DEBUG:boto:Params: {} 2022-09-12T20:12:06.830 DEBUG:boto:Token: None 2022-09-12T20:12:06.831 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj116 2022-09-12T20:12:06.831 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:U/lywhVbreZhdD488gUyO626Lws= 2022-09-12T20:12:06.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:U/lywhVbreZhdD488gUyO626Lws=', 'Content-Length': '0'} 2022-09-12T20:12:06.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000712ac2f99823c4f1-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:12:06.835 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:06.835 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:06.836 DEBUG:boto:Method: GET 2022-09-12T20:12:06.837 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:12:06.837 DEBUG:boto:Data: 2022-09-12T20:12:06.838 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.838 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.839 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.839 DEBUG:boto:Params: {} 2022-09-12T20:12:06.840 DEBUG:boto:Token: None 2022-09-12T20:12:06.841 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj116 2022-09-12T20:12:06.841 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:r7GLrT1TVcSOHCynGYpQ6qWWgQU= 2022-09-12T20:12:06.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:r7GLrT1TVcSOHCynGYpQ6qWWgQU=', 'Content-Length': '0'} 2022-09-12T20:12:06.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bcff886764b088e-00631f9296-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.845 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:06.845 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:06.846 DEBUG:boto:Method: GET 2022-09-12T20:12:06.846 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:12:06.847 DEBUG:boto:Data: 2022-09-12T20:12:06.847 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.848 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.848 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.849 DEBUG:boto:Params: {} 2022-09-12T20:12:06.849 DEBUG:boto:Token: None 2022-09-12T20:12:06.850 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj116 2022-09-12T20:12:06.851 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:r7GLrT1TVcSOHCynGYpQ6qWWgQU= 2022-09-12T20:12:06.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:r7GLrT1TVcSOHCynGYpQ6qWWgQU=', 'Content-Length': '0'} 2022-09-12T20:12:06.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a13b17dfd872198-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.854 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:06.855 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:06.855 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:12:06.856 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:12:06.856 DEBUG:boto:Method: GET 2022-09-12T20:12:06.857 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:12:06.857 DEBUG:boto:Data: 2022-09-12T20:12:06.858 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.859 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.859 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.860 DEBUG:boto:Params: {} 2022-09-12T20:12:06.860 DEBUG:boto:Token: None 2022-09-12T20:12:06.861 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/ 2022-09-12T20:12:06.861 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ= 2022-09-12T20:12:06.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ=', 'Content-Length': '0'} 2022-09-12T20:12:06.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009abb56da979b9690-00631f9296-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.865 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:06.866 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:06.866 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:06.867 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:12:06.868 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:12:06.868 DEBUG:boto:Method: GET 2022-09-12T20:12:06.869 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:12:06.869 DEBUG:boto:Data: 2022-09-12T20:12:06.870 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.870 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.871 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.871 DEBUG:boto:Params: {} 2022-09-12T20:12:06.872 DEBUG:boto:Token: None 2022-09-12T20:12:06.873 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/ 2022-09-12T20:12:06.873 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ= 2022-09-12T20:12:06.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ=', 'Content-Length': '0'} 2022-09-12T20:12:06.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000022e5b665a2ca65-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.878 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:06.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:12:06.880 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:06.880 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:06.881 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:06.881 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:06.882 DEBUG:boto:Method: GET 2022-09-12T20:12:06.882 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:06.883 DEBUG:boto:Data: 2022-09-12T20:12:06.883 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.884 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.885 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.885 DEBUG:boto:Params: {} 2022-09-12T20:12:06.886 DEBUG:boto:Token: None 2022-09-12T20:12:06.886 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:06.887 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:coQyOxkUWVvz30jVzgRgYyC40UU= 2022-09-12T20:12:06.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:coQyOxkUWVvz30jVzgRgYyC40UU=', 'Content-Length': '0'} 2022-09-12T20:12:06.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096e4489c6f27790e-00631f9296-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.890 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:06.891 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:06.891 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:06.892 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:06.892 DEBUG:boto:Method: GET 2022-09-12T20:12:06.893 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:06.894 DEBUG:boto:Data: 2022-09-12T20:12:06.894 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.895 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.895 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.896 DEBUG:boto:Params: {} 2022-09-12T20:12:06.896 DEBUG:boto:Token: None 2022-09-12T20:12:06.897 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:06.898 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:coQyOxkUWVvz30jVzgRgYyC40UU= 2022-09-12T20:12:06.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:coQyOxkUWVvz30jVzgRgYyC40UU=', 'Content-Length': '0'} 2022-09-12T20:12:06.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b14211100bd6e1a2-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.901 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:06.902 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:06.903 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:06.903 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:06.904 DEBUG:boto:Method: HEAD 2022-09-12T20:12:06.904 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:06.905 DEBUG:boto:Data: 2022-09-12T20:12:06.905 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.906 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.906 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.907 DEBUG:boto:Params: {} 2022-09-12T20:12:06.907 DEBUG:boto:Token: None 2022-09-12T20:12:06.908 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:06.908 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wiPxzEJ5cDTWuHc0pYHHIjPlQYY= 2022-09-12T20:12:06.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wiPxzEJ5cDTWuHc0pYHHIjPlQYY=', 'Content-Length': '0'} 2022-09-12T20:12:06.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035fa198195595e98-00631f9296-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.911 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:06.912 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:06.913 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:06.913 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:06.914 DEBUG:boto:Method: HEAD 2022-09-12T20:12:06.914 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:06.915 DEBUG:boto:Data: 2022-09-12T20:12:06.915 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.916 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.916 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.917 DEBUG:boto:Params: {} 2022-09-12T20:12:06.917 DEBUG:boto:Token: None 2022-09-12T20:12:06.918 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:06.918 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wiPxzEJ5cDTWuHc0pYHHIjPlQYY= 2022-09-12T20:12:06.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wiPxzEJ5cDTWuHc0pYHHIjPlQYY=', 'Content-Length': '0'} 2022-09-12T20:12:06.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003c80de876f2443c-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:12:06.923 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:06.923 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:06.924 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:06.924 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:06.925 DEBUG:boto:Method: GET 2022-09-12T20:12:06.926 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:06.926 DEBUG:boto:Data: 2022-09-12T20:12:06.927 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.927 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.928 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.928 DEBUG:boto:Params: {} 2022-09-12T20:12:06.929 DEBUG:boto:Token: None 2022-09-12T20:12:06.929 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:06.930 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:coQyOxkUWVvz30jVzgRgYyC40UU= 2022-09-12T20:12:06.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:coQyOxkUWVvz30jVzgRgYyC40UU=', 'Content-Length': '0'} 2022-09-12T20:12:06.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000727bb4ff6562ce07-00631f9296-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.974 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:06.975 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:06.975 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:06.976 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:06.976 DEBUG:boto:Method: GET 2022-09-12T20:12:06.977 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:06.978 DEBUG:boto:Data: 2022-09-12T20:12:06.978 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.979 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:06.979 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.980 DEBUG:boto:Params: {} 2022-09-12T20:12:06.980 DEBUG:boto:Token: None 2022-09-12T20:12:06.981 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:06.982 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:coQyOxkUWVvz30jVzgRgYyC40UU= 2022-09-12T20:12:06.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:coQyOxkUWVvz30jVzgRgYyC40UU=', 'Content-Length': '0'} 2022-09-12T20:12:06.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3fecfdfcc0a2a82-00631f9296-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.986 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:06.986 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:06.987 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:12:06.987 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:12:06.988 DEBUG:boto:Method: GET 2022-09-12T20:12:06.988 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:12:06.989 DEBUG:boto:Data: 2022-09-12T20:12:06.989 DEBUG:boto:Headers: {} 2022-09-12T20:12:06.990 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:06.991 DEBUG:boto:Port: 8000 2022-09-12T20:12:06.991 DEBUG:boto:Params: {} 2022-09-12T20:12:06.992 DEBUG:boto:Token: None 2022-09-12T20:12:06.992 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:06 GMT /jdpiak-75/ 2022-09-12T20:12:06.993 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ= 2022-09-12T20:12:06.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qDwKaMsKV+sB6+l1+riwh1rACzQ=', 'Content-Length': '0'} 2022-09-12T20:12:06.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016a2a3c9c4d40f42-00631f9296-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:06.997 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:06.998 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:06.999 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:06.999 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:12:07.000 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:12:07.000 DEBUG:boto:Method: GET 2022-09-12T20:12:07.001 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:12:07.001 DEBUG:boto:Data: 2022-09-12T20:12:07.002 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.002 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.003 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.003 DEBUG:boto:Params: {} 2022-09-12T20:12:07.004 DEBUG:boto:Token: None 2022-09-12T20:12:07.005 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/ 2022-09-12T20:12:07.005 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk= 2022-09-12T20:12:07.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk=', 'Content-Length': '0'} 2022-09-12T20:12:07.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3c1347653b49353-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.011 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:07.012 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:07.012 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:07.013 DEBUG:boto:Method: HEAD 2022-09-12T20:12:07.013 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:12:07.014 DEBUG:boto:Data: 2022-09-12T20:12:07.015 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.015 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.016 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.016 DEBUG:boto:Params: {} 2022-09-12T20:12:07.017 DEBUG:boto:Token: None 2022-09-12T20:12:07.017 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj145 2022-09-12T20:12:07.018 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0SCkvdKM0OZ4iAzqZW7NoMlo8Ek= 2022-09-12T20:12:07.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0SCkvdKM0OZ4iAzqZW7NoMlo8Ek=', 'Content-Length': '0'} 2022-09-12T20:12:07.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7d964d1e3716479-00631f9297-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.022 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:07.022 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:07.023 DEBUG:boto:Method: HEAD 2022-09-12T20:12:07.023 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:12:07.024 DEBUG:boto:Data: 2022-09-12T20:12:07.025 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.025 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.026 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.026 DEBUG:boto:Params: {} 2022-09-12T20:12:07.027 DEBUG:boto:Token: None 2022-09-12T20:12:07.028 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj145 2022-09-12T20:12:07.028 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0SCkvdKM0OZ4iAzqZW7NoMlo8Ek= 2022-09-12T20:12:07.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0SCkvdKM0OZ4iAzqZW7NoMlo8Ek=', 'Content-Length': '0'} 2022-09-12T20:12:07.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8cdef9ead29203f-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:12:07.032 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:07.033 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:07.034 DEBUG:boto:Method: GET 2022-09-12T20:12:07.034 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:12:07.035 DEBUG:boto:Data: 2022-09-12T20:12:07.035 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.036 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.036 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.037 DEBUG:boto:Params: {} 2022-09-12T20:12:07.038 DEBUG:boto:Token: None 2022-09-12T20:12:07.038 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj145 2022-09-12T20:12:07.039 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EUqX3UNGDzIN11Pq9EMFfuoefHI= 2022-09-12T20:12:07.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EUqX3UNGDzIN11Pq9EMFfuoefHI=', 'Content-Length': '0'} 2022-09-12T20:12:07.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009329a6f8da7e1178-00631f9297-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.042 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:07.043 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:07.043 DEBUG:boto:Method: GET 2022-09-12T20:12:07.044 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:12:07.044 DEBUG:boto:Data: 2022-09-12T20:12:07.045 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.046 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.046 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.047 DEBUG:boto:Params: {} 2022-09-12T20:12:07.047 DEBUG:boto:Token: None 2022-09-12T20:12:07.048 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj145 2022-09-12T20:12:07.048 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EUqX3UNGDzIN11Pq9EMFfuoefHI= 2022-09-12T20:12:07.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EUqX3UNGDzIN11Pq9EMFfuoefHI=', 'Content-Length': '0'} 2022-09-12T20:12:07.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ebbc69f98ecc26e-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.094 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:07.094 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:07.095 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:12:07.096 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:12:07.096 DEBUG:boto:Method: GET 2022-09-12T20:12:07.097 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:12:07.097 DEBUG:boto:Data: 2022-09-12T20:12:07.098 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.099 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.099 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.100 DEBUG:boto:Params: {} 2022-09-12T20:12:07.100 DEBUG:boto:Token: None 2022-09-12T20:12:07.101 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/ 2022-09-12T20:12:07.101 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk= 2022-09-12T20:12:07.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk=', 'Content-Length': '0'} 2022-09-12T20:12:07.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da3b63b4f933e1a8-00631f9297-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.105 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:07.106 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:07.107 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:07.107 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:12:07.108 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:12:07.108 DEBUG:boto:Method: GET 2022-09-12T20:12:07.109 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:12:07.109 DEBUG:boto:Data: 2022-09-12T20:12:07.110 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.111 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.111 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.112 DEBUG:boto:Params: {} 2022-09-12T20:12:07.112 DEBUG:boto:Token: None 2022-09-12T20:12:07.113 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/ 2022-09-12T20:12:07.113 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk= 2022-09-12T20:12:07.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk=', 'Content-Length': '0'} 2022-09-12T20:12:07.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ba9e26c6dfa5d7b-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.119 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:07.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:12:07.120 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:07.121 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:07.121 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:07.122 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:07.123 DEBUG:boto:Method: GET 2022-09-12T20:12:07.123 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:07.124 DEBUG:boto:Data: 2022-09-12T20:12:07.124 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.125 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.125 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.126 DEBUG:boto:Params: {} 2022-09-12T20:12:07.127 DEBUG:boto:Token: None 2022-09-12T20:12:07.127 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:07.128 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9znyNpq+2KDcTKlNsMdoXrrDgqA= 2022-09-12T20:12:07.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9znyNpq+2KDcTKlNsMdoXrrDgqA=', 'Content-Length': '0'} 2022-09-12T20:12:07.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057eebef896bdbe1f-00631f9297-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.132 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:07.132 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:07.133 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:07.133 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:07.134 DEBUG:boto:Method: GET 2022-09-12T20:12:07.134 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:07.135 DEBUG:boto:Data: 2022-09-12T20:12:07.135 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.136 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.137 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.137 DEBUG:boto:Params: {} 2022-09-12T20:12:07.138 DEBUG:boto:Token: None 2022-09-12T20:12:07.138 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:07.139 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9znyNpq+2KDcTKlNsMdoXrrDgqA= 2022-09-12T20:12:07.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9znyNpq+2KDcTKlNsMdoXrrDgqA=', 'Content-Length': '0'} 2022-09-12T20:12:07.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003077952d33e7db04-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.143 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:07.143 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:07.144 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:07.144 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:07.145 DEBUG:boto:Method: HEAD 2022-09-12T20:12:07.145 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:07.146 DEBUG:boto:Data: 2022-09-12T20:12:07.147 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.147 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.148 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.148 DEBUG:boto:Params: {} 2022-09-12T20:12:07.149 DEBUG:boto:Token: None 2022-09-12T20:12:07.149 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:07.150 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:phfUGXJK7C1yHDFRrrYTf/TGjXM= 2022-09-12T20:12:07.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:phfUGXJK7C1yHDFRrrYTf/TGjXM=', 'Content-Length': '0'} 2022-09-12T20:12:07.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078a36115f9ea1658-00631f9297-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.153 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:07.154 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:07.154 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:07.155 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:07.155 DEBUG:boto:Method: HEAD 2022-09-12T20:12:07.156 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:07.156 DEBUG:boto:Data: 2022-09-12T20:12:07.157 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.157 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.158 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.158 DEBUG:boto:Params: {} 2022-09-12T20:12:07.159 DEBUG:boto:Token: None 2022-09-12T20:12:07.160 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:07.160 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:phfUGXJK7C1yHDFRrrYTf/TGjXM= 2022-09-12T20:12:07.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:phfUGXJK7C1yHDFRrrYTf/TGjXM=', 'Content-Length': '0'} 2022-09-12T20:12:07.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c091844820cd22d-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:12:07.164 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:07.165 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:07.165 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:07.166 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:07.166 DEBUG:boto:Method: GET 2022-09-12T20:12:07.167 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:07.168 DEBUG:boto:Data: 2022-09-12T20:12:07.168 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.169 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.169 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.170 DEBUG:boto:Params: {} 2022-09-12T20:12:07.170 DEBUG:boto:Token: None 2022-09-12T20:12:07.171 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:07.171 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9znyNpq+2KDcTKlNsMdoXrrDgqA= 2022-09-12T20:12:07.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9znyNpq+2KDcTKlNsMdoXrrDgqA=', 'Content-Length': '0'} 2022-09-12T20:12:07.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3c0b8fc746163a5-00631f9297-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.175 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:07.175 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:07.176 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:07.177 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:07.177 DEBUG:boto:Method: GET 2022-09-12T20:12:07.178 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:07.178 DEBUG:boto:Data: 2022-09-12T20:12:07.179 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.179 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.180 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.180 DEBUG:boto:Params: {} 2022-09-12T20:12:07.181 DEBUG:boto:Token: None 2022-09-12T20:12:07.182 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:07.182 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9znyNpq+2KDcTKlNsMdoXrrDgqA= 2022-09-12T20:12:07.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9znyNpq+2KDcTKlNsMdoXrrDgqA=', 'Content-Length': '0'} 2022-09-12T20:12:07.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af8cdd17d3ddf232-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.226 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:07.226 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:07.227 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:12:07.228 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:12:07.228 DEBUG:boto:Method: GET 2022-09-12T20:12:07.229 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:12:07.229 DEBUG:boto:Data: 2022-09-12T20:12:07.230 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.230 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.231 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.231 DEBUG:boto:Params: {} 2022-09-12T20:12:07.232 DEBUG:boto:Token: None 2022-09-12T20:12:07.233 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/ 2022-09-12T20:12:07.233 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk= 2022-09-12T20:12:07.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk=', 'Content-Length': '0'} 2022-09-12T20:12:07.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009649d6dd3a4dda3a-00631f9297-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.237 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:07.238 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:07.239 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:07.239 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:12:07.240 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:12:07.240 DEBUG:boto:Method: GET 2022-09-12T20:12:07.241 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:12:07.241 DEBUG:boto:Data: 2022-09-12T20:12:07.242 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.243 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.243 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.244 DEBUG:boto:Params: {} 2022-09-12T20:12:07.244 DEBUG:boto:Token: None 2022-09-12T20:12:07.245 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/ 2022-09-12T20:12:07.246 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk= 2022-09-12T20:12:07.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk=', 'Content-Length': '0'} 2022-09-12T20:12:07.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb429ea938766d5c-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.251 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:07.252 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:07.252 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:07.253 DEBUG:boto:Method: HEAD 2022-09-12T20:12:07.254 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:12:07.254 DEBUG:boto:Data: 2022-09-12T20:12:07.255 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.255 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.256 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.256 DEBUG:boto:Params: {} 2022-09-12T20:12:07.257 DEBUG:boto:Token: None 2022-09-12T20:12:07.257 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj17 2022-09-12T20:12:07.258 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6tAh3RpOAyAb/bC0smOSCZaUjV0= 2022-09-12T20:12:07.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6tAh3RpOAyAb/bC0smOSCZaUjV0=', 'Content-Length': '0'} 2022-09-12T20:12:07.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1eebd4239ded27d-00631f9297-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.261 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:07.262 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:07.263 DEBUG:boto:Method: HEAD 2022-09-12T20:12:07.263 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:12:07.264 DEBUG:boto:Data: 2022-09-12T20:12:07.264 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.265 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.265 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.266 DEBUG:boto:Params: {} 2022-09-12T20:12:07.267 DEBUG:boto:Token: None 2022-09-12T20:12:07.267 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj17 2022-09-12T20:12:07.268 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6tAh3RpOAyAb/bC0smOSCZaUjV0= 2022-09-12T20:12:07.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6tAh3RpOAyAb/bC0smOSCZaUjV0=', 'Content-Length': '0'} 2022-09-12T20:12:07.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e3349d6199c8c8b-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:12:07.272 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:07.272 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:07.273 DEBUG:boto:Method: GET 2022-09-12T20:12:07.273 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:12:07.274 DEBUG:boto:Data: 2022-09-12T20:12:07.275 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.275 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.276 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.276 DEBUG:boto:Params: {} 2022-09-12T20:12:07.277 DEBUG:boto:Token: None 2022-09-12T20:12:07.277 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj17 2022-09-12T20:12:07.278 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nsahdtYlkRe9fShLuaATlW939aA= 2022-09-12T20:12:07.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nsahdtYlkRe9fShLuaATlW939aA=', 'Content-Length': '0'} 2022-09-12T20:12:07.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007581296c0f89fdfd-00631f9297-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.281 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:07.282 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:07.283 DEBUG:boto:Method: GET 2022-09-12T20:12:07.283 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:12:07.284 DEBUG:boto:Data: 2022-09-12T20:12:07.284 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.285 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.286 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.286 DEBUG:boto:Params: {} 2022-09-12T20:12:07.287 DEBUG:boto:Token: None 2022-09-12T20:12:07.287 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj17 2022-09-12T20:12:07.288 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nsahdtYlkRe9fShLuaATlW939aA= 2022-09-12T20:12:07.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nsahdtYlkRe9fShLuaATlW939aA=', 'Content-Length': '0'} 2022-09-12T20:12:07.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050a74dcaed2cb701-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.334 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:07.335 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:07.335 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:12:07.336 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:12:07.336 DEBUG:boto:Method: GET 2022-09-12T20:12:07.337 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:12:07.337 DEBUG:boto:Data: 2022-09-12T20:12:07.338 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.338 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.339 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.340 DEBUG:boto:Params: {} 2022-09-12T20:12:07.340 DEBUG:boto:Token: None 2022-09-12T20:12:07.341 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/ 2022-09-12T20:12:07.342 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk= 2022-09-12T20:12:07.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk=', 'Content-Length': '0'} 2022-09-12T20:12:07.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dc9e6166d1b6ccd-00631f9297-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.345 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:07.346 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:07.347 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:07.347 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:12:07.348 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:12:07.349 DEBUG:boto:Method: GET 2022-09-12T20:12:07.349 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:12:07.350 DEBUG:boto:Data: 2022-09-12T20:12:07.350 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.351 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.351 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.352 DEBUG:boto:Params: {} 2022-09-12T20:12:07.352 DEBUG:boto:Token: None 2022-09-12T20:12:07.353 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/ 2022-09-12T20:12:07.353 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk= 2022-09-12T20:12:07.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk=', 'Content-Length': '0'} 2022-09-12T20:12:07.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fa15fcc18ab66a6-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.359 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:07.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:12:07.360 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:07.361 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:07.362 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:07.362 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:07.363 DEBUG:boto:Method: GET 2022-09-12T20:12:07.363 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:07.364 DEBUG:boto:Data: 2022-09-12T20:12:07.364 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.365 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.365 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.366 DEBUG:boto:Params: {} 2022-09-12T20:12:07.367 DEBUG:boto:Token: None 2022-09-12T20:12:07.367 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:07.368 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MqKUyjktdw46POABIZ7I+ZzmhMk= 2022-09-12T20:12:07.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MqKUyjktdw46POABIZ7I+ZzmhMk=', 'Content-Length': '0'} 2022-09-12T20:12:07.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ca017eef2880029-00631f9297-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.372 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:07.372 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:07.373 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:07.373 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:07.374 DEBUG:boto:Method: GET 2022-09-12T20:12:07.374 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:07.375 DEBUG:boto:Data: 2022-09-12T20:12:07.376 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.376 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.377 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.377 DEBUG:boto:Params: {} 2022-09-12T20:12:07.378 DEBUG:boto:Token: None 2022-09-12T20:12:07.378 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:07.379 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MqKUyjktdw46POABIZ7I+ZzmhMk= 2022-09-12T20:12:07.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MqKUyjktdw46POABIZ7I+ZzmhMk=', 'Content-Length': '0'} 2022-09-12T20:12:07.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c482551b7f3d0681-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.383 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:07.383 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:07.384 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:07.384 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:07.385 DEBUG:boto:Method: HEAD 2022-09-12T20:12:07.385 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:07.386 DEBUG:boto:Data: 2022-09-12T20:12:07.387 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.387 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.388 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.388 DEBUG:boto:Params: {} 2022-09-12T20:12:07.389 DEBUG:boto:Token: None 2022-09-12T20:12:07.389 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:07.390 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lHrwUMyvRdIKbY7O3+LmqX2mEFw= 2022-09-12T20:12:07.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lHrwUMyvRdIKbY7O3+LmqX2mEFw=', 'Content-Length': '0'} 2022-09-12T20:12:07.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd1e0574077abeb4-00631f9297-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.393 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:07.394 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:07.394 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:07.395 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:07.395 DEBUG:boto:Method: HEAD 2022-09-12T20:12:07.396 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:07.396 DEBUG:boto:Data: 2022-09-12T20:12:07.397 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.397 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.398 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.398 DEBUG:boto:Params: {} 2022-09-12T20:12:07.399 DEBUG:boto:Token: None 2022-09-12T20:12:07.400 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:07.400 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lHrwUMyvRdIKbY7O3+LmqX2mEFw= 2022-09-12T20:12:07.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lHrwUMyvRdIKbY7O3+LmqX2mEFw=', 'Content-Length': '0'} 2022-09-12T20:12:07.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003070e3934516f81f-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:12:07.404 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:07.405 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:07.405 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:07.406 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:07.406 DEBUG:boto:Method: GET 2022-09-12T20:12:07.407 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:07.407 DEBUG:boto:Data: 2022-09-12T20:12:07.408 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.409 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.409 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.410 DEBUG:boto:Params: {} 2022-09-12T20:12:07.410 DEBUG:boto:Token: None 2022-09-12T20:12:07.411 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:07.411 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MqKUyjktdw46POABIZ7I+ZzmhMk= 2022-09-12T20:12:07.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MqKUyjktdw46POABIZ7I+ZzmhMk=', 'Content-Length': '0'} 2022-09-12T20:12:07.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057c4e1141e17ac67-00631f9297-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.458 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:07.459 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:07.459 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:07.460 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:07.461 DEBUG:boto:Method: GET 2022-09-12T20:12:07.461 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:07.462 DEBUG:boto:Data: 2022-09-12T20:12:07.462 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.463 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.463 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.464 DEBUG:boto:Params: {} 2022-09-12T20:12:07.464 DEBUG:boto:Token: None 2022-09-12T20:12:07.465 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:07.466 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MqKUyjktdw46POABIZ7I+ZzmhMk= 2022-09-12T20:12:07.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MqKUyjktdw46POABIZ7I+ZzmhMk=', 'Content-Length': '0'} 2022-09-12T20:12:07.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010b49c64403cf556-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.470 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:07.470 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:07.471 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:12:07.471 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:12:07.472 DEBUG:boto:Method: GET 2022-09-12T20:12:07.472 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:12:07.473 DEBUG:boto:Data: 2022-09-12T20:12:07.473 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.474 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.474 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.475 DEBUG:boto:Params: {} 2022-09-12T20:12:07.476 DEBUG:boto:Token: None 2022-09-12T20:12:07.476 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/ 2022-09-12T20:12:07.477 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk= 2022-09-12T20:12:07.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk=', 'Content-Length': '0'} 2022-09-12T20:12:07.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041f8c2bf5d915279-00631f9297-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.481 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:07.482 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:07.482 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:07.483 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:12:07.483 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:12:07.484 DEBUG:boto:Method: GET 2022-09-12T20:12:07.484 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:12:07.485 DEBUG:boto:Data: 2022-09-12T20:12:07.485 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.486 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.487 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.487 DEBUG:boto:Params: {} 2022-09-12T20:12:07.488 DEBUG:boto:Token: None 2022-09-12T20:12:07.488 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/ 2022-09-12T20:12:07.489 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk= 2022-09-12T20:12:07.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk=', 'Content-Length': '0'} 2022-09-12T20:12:07.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be6b1551d6dff036-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.495 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:07.496 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:07.496 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:07.497 DEBUG:boto:Method: HEAD 2022-09-12T20:12:07.497 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:12:07.498 DEBUG:boto:Data: 2022-09-12T20:12:07.498 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.499 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.499 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.500 DEBUG:boto:Params: {} 2022-09-12T20:12:07.500 DEBUG:boto:Token: None 2022-09-12T20:12:07.501 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj19 2022-09-12T20:12:07.502 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QVX0Qb8bDTRVb4joJMpO3OdAqbg= 2022-09-12T20:12:07.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QVX0Qb8bDTRVb4joJMpO3OdAqbg=', 'Content-Length': '0'} 2022-09-12T20:12:07.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a07c961924d4e008-00631f9297-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.505 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:07.505 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:07.506 DEBUG:boto:Method: HEAD 2022-09-12T20:12:07.507 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:12:07.507 DEBUG:boto:Data: 2022-09-12T20:12:07.508 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.508 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.509 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.509 DEBUG:boto:Params: {} 2022-09-12T20:12:07.510 DEBUG:boto:Token: None 2022-09-12T20:12:07.511 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj19 2022-09-12T20:12:07.512 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QVX0Qb8bDTRVb4joJMpO3OdAqbg= 2022-09-12T20:12:07.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QVX0Qb8bDTRVb4joJMpO3OdAqbg=', 'Content-Length': '0'} 2022-09-12T20:12:07.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005473d0862d3035f4-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:12:07.516 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:07.516 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:07.517 DEBUG:boto:Method: GET 2022-09-12T20:12:07.517 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:12:07.518 DEBUG:boto:Data: 2022-09-12T20:12:07.519 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.519 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.520 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.520 DEBUG:boto:Params: {} 2022-09-12T20:12:07.521 DEBUG:boto:Token: None 2022-09-12T20:12:07.522 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj19 2022-09-12T20:12:07.522 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lWmMpq3ci+F3QBb5FAl+XDep1kI= 2022-09-12T20:12:07.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lWmMpq3ci+F3QBb5FAl+XDep1kI=', 'Content-Length': '0'} 2022-09-12T20:12:07.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a02971d9814c779-00631f9297-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.526 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:07.526 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:07.527 DEBUG:boto:Method: GET 2022-09-12T20:12:07.527 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:12:07.528 DEBUG:boto:Data: 2022-09-12T20:12:07.529 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.529 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.530 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.530 DEBUG:boto:Params: {} 2022-09-12T20:12:07.531 DEBUG:boto:Token: None 2022-09-12T20:12:07.532 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj19 2022-09-12T20:12:07.532 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lWmMpq3ci+F3QBb5FAl+XDep1kI= 2022-09-12T20:12:07.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lWmMpq3ci+F3QBb5FAl+XDep1kI=', 'Content-Length': '0'} 2022-09-12T20:12:07.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d6f3b756d96bd7e-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.536 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:07.537 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:07.537 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:12:07.538 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:12:07.538 DEBUG:boto:Method: GET 2022-09-12T20:12:07.539 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:12:07.540 DEBUG:boto:Data: 2022-09-12T20:12:07.540 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.541 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.541 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.542 DEBUG:boto:Params: {} 2022-09-12T20:12:07.543 DEBUG:boto:Token: None 2022-09-12T20:12:07.543 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/ 2022-09-12T20:12:07.544 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk= 2022-09-12T20:12:07.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk=', 'Content-Length': '0'} 2022-09-12T20:12:07.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb0f3fc9800fed85-00631f9297-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.548 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:07.549 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:07.549 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:07.550 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:12:07.550 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:12:07.551 DEBUG:boto:Method: GET 2022-09-12T20:12:07.551 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:12:07.552 DEBUG:boto:Data: 2022-09-12T20:12:07.553 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.553 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.554 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.554 DEBUG:boto:Params: {} 2022-09-12T20:12:07.555 DEBUG:boto:Token: None 2022-09-12T20:12:07.555 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/ 2022-09-12T20:12:07.556 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk= 2022-09-12T20:12:07.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk=', 'Content-Length': '0'} 2022-09-12T20:12:07.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d266d61d7158659-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.562 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:07.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:12:07.563 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:07.563 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:07.564 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:07.565 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:07.565 DEBUG:boto:Method: GET 2022-09-12T20:12:07.566 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:07.566 DEBUG:boto:Data: 2022-09-12T20:12:07.567 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.567 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.568 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.568 DEBUG:boto:Params: {} 2022-09-12T20:12:07.569 DEBUG:boto:Token: None 2022-09-12T20:12:07.570 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:07.571 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VyNstd0qppc9S8iggTn3GjsxXAA= 2022-09-12T20:12:07.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VyNstd0qppc9S8iggTn3GjsxXAA=', 'Content-Length': '0'} 2022-09-12T20:12:07.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000192519e440f9c4fe-00631f9297-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.574 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:07.575 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:07.575 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:07.576 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:07.576 DEBUG:boto:Method: GET 2022-09-12T20:12:07.577 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:07.577 DEBUG:boto:Data: 2022-09-12T20:12:07.578 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.578 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.579 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.579 DEBUG:boto:Params: {} 2022-09-12T20:12:07.580 DEBUG:boto:Token: None 2022-09-12T20:12:07.581 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:07.581 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VyNstd0qppc9S8iggTn3GjsxXAA= 2022-09-12T20:12:07.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VyNstd0qppc9S8iggTn3GjsxXAA=', 'Content-Length': '0'} 2022-09-12T20:12:07.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d44c71527c196399-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.626 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:07.627 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:07.627 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:07.628 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:07.628 DEBUG:boto:Method: HEAD 2022-09-12T20:12:07.629 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:07.629 DEBUG:boto:Data: 2022-09-12T20:12:07.630 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.630 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.631 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.632 DEBUG:boto:Params: {} 2022-09-12T20:12:07.632 DEBUG:boto:Token: None 2022-09-12T20:12:07.633 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:07.633 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CWe0YkWU3qJICTBTYagKYB7H2Yg= 2022-09-12T20:12:07.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CWe0YkWU3qJICTBTYagKYB7H2Yg=', 'Content-Length': '0'} 2022-09-12T20:12:07.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea566c477667a5ac-00631f9297-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.637 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:07.637 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:07.638 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:07.638 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:07.639 DEBUG:boto:Method: HEAD 2022-09-12T20:12:07.639 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:07.640 DEBUG:boto:Data: 2022-09-12T20:12:07.640 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.641 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.641 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.642 DEBUG:boto:Params: {} 2022-09-12T20:12:07.643 DEBUG:boto:Token: None 2022-09-12T20:12:07.643 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:07.644 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CWe0YkWU3qJICTBTYagKYB7H2Yg= 2022-09-12T20:12:07.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CWe0YkWU3qJICTBTYagKYB7H2Yg=', 'Content-Length': '0'} 2022-09-12T20:12:07.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1aadea4eb55e030-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:12:07.648 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:07.649 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:07.649 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:07.650 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:07.651 DEBUG:boto:Method: GET 2022-09-12T20:12:07.651 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:07.652 DEBUG:boto:Data: 2022-09-12T20:12:07.652 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.653 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.653 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.654 DEBUG:boto:Params: {} 2022-09-12T20:12:07.655 DEBUG:boto:Token: None 2022-09-12T20:12:07.655 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:07.656 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VyNstd0qppc9S8iggTn3GjsxXAA= 2022-09-12T20:12:07.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VyNstd0qppc9S8iggTn3GjsxXAA=', 'Content-Length': '0'} 2022-09-12T20:12:07.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee3d841fd86857c2-00631f9297-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.660 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:07.660 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:07.661 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:07.661 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:07.662 DEBUG:boto:Method: GET 2022-09-12T20:12:07.662 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:07.663 DEBUG:boto:Data: 2022-09-12T20:12:07.664 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.664 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.665 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.665 DEBUG:boto:Params: {} 2022-09-12T20:12:07.666 DEBUG:boto:Token: None 2022-09-12T20:12:07.666 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:07.667 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VyNstd0qppc9S8iggTn3GjsxXAA= 2022-09-12T20:12:07.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VyNstd0qppc9S8iggTn3GjsxXAA=', 'Content-Length': '0'} 2022-09-12T20:12:07.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000277cd66ccbd02ca9-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.671 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:07.671 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:07.672 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:12:07.672 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:12:07.673 DEBUG:boto:Method: GET 2022-09-12T20:12:07.673 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:12:07.674 DEBUG:boto:Data: 2022-09-12T20:12:07.675 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.675 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.676 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.676 DEBUG:boto:Params: {} 2022-09-12T20:12:07.677 DEBUG:boto:Token: None 2022-09-12T20:12:07.678 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/ 2022-09-12T20:12:07.678 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk= 2022-09-12T20:12:07.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk=', 'Content-Length': '0'} 2022-09-12T20:12:07.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9adb963121911db-00631f9297-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.722 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:07.723 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:07.724 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:07.724 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:12:07.725 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:12:07.725 DEBUG:boto:Method: GET 2022-09-12T20:12:07.726 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:12:07.726 DEBUG:boto:Data: 2022-09-12T20:12:07.727 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.727 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.728 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.728 DEBUG:boto:Params: {} 2022-09-12T20:12:07.729 DEBUG:boto:Token: None 2022-09-12T20:12:07.730 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/ 2022-09-12T20:12:07.730 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk= 2022-09-12T20:12:07.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk=', 'Content-Length': '0'} 2022-09-12T20:12:07.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a237c7e16f6c278-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.736 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:07.737 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:07.738 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:07.738 DEBUG:boto:Method: HEAD 2022-09-12T20:12:07.739 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:12:07.739 DEBUG:boto:Data: 2022-09-12T20:12:07.740 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.740 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.741 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.742 DEBUG:boto:Params: {} 2022-09-12T20:12:07.742 DEBUG:boto:Token: None 2022-09-12T20:12:07.743 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj22 2022-09-12T20:12:07.743 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MpAC0ilbVvUTfgBgyVH8ljUIxzA= 2022-09-12T20:12:07.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MpAC0ilbVvUTfgBgyVH8ljUIxzA=', 'Content-Length': '0'} 2022-09-12T20:12:07.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009aeb0e2490312e7f-00631f9297-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.747 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:07.748 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:07.748 DEBUG:boto:Method: HEAD 2022-09-12T20:12:07.749 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:12:07.749 DEBUG:boto:Data: 2022-09-12T20:12:07.750 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.751 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.751 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.752 DEBUG:boto:Params: {} 2022-09-12T20:12:07.752 DEBUG:boto:Token: None 2022-09-12T20:12:07.753 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj22 2022-09-12T20:12:07.754 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MpAC0ilbVvUTfgBgyVH8ljUIxzA= 2022-09-12T20:12:07.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MpAC0ilbVvUTfgBgyVH8ljUIxzA=', 'Content-Length': '0'} 2022-09-12T20:12:07.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df3d886f049df435-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:12:07.758 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:07.759 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:07.759 DEBUG:boto:Method: GET 2022-09-12T20:12:07.760 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:12:07.761 DEBUG:boto:Data: 2022-09-12T20:12:07.762 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.763 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.764 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.765 DEBUG:boto:Params: {} 2022-09-12T20:12:07.766 DEBUG:boto:Token: None 2022-09-12T20:12:07.767 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj22 2022-09-12T20:12:07.768 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ulBAkVNq2hju5fHrM7b4SqKXXLc= 2022-09-12T20:12:07.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ulBAkVNq2hju5fHrM7b4SqKXXLc=', 'Content-Length': '0'} 2022-09-12T20:12:07.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e592d08a7d2ae527-00631f9297-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.772 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:07.773 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:07.774 DEBUG:boto:Method: GET 2022-09-12T20:12:07.775 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:12:07.776 DEBUG:boto:Data: 2022-09-12T20:12:07.777 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.778 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.779 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.780 DEBUG:boto:Params: {} 2022-09-12T20:12:07.781 DEBUG:boto:Token: None 2022-09-12T20:12:07.782 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj22 2022-09-12T20:12:07.783 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ulBAkVNq2hju5fHrM7b4SqKXXLc= 2022-09-12T20:12:07.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ulBAkVNq2hju5fHrM7b4SqKXXLc=', 'Content-Length': '0'} 2022-09-12T20:12:07.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066b35d6978a7ae95-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.830 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:07.831 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:07.832 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:12:07.833 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:12:07.835 DEBUG:boto:Method: GET 2022-09-12T20:12:07.836 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:12:07.837 DEBUG:boto:Data: 2022-09-12T20:12:07.838 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.839 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.840 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.841 DEBUG:boto:Params: {} 2022-09-12T20:12:07.842 DEBUG:boto:Token: None 2022-09-12T20:12:07.843 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/ 2022-09-12T20:12:07.844 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk= 2022-09-12T20:12:07.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk=', 'Content-Length': '0'} 2022-09-12T20:12:07.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000502d074437bd3e7d-00631f9297-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.850 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:07.851 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:07.852 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:07.853 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:12:07.853 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:12:07.854 DEBUG:boto:Method: GET 2022-09-12T20:12:07.855 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:12:07.856 DEBUG:boto:Data: 2022-09-12T20:12:07.857 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.858 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.859 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.860 DEBUG:boto:Params: {} 2022-09-12T20:12:07.861 DEBUG:boto:Token: None 2022-09-12T20:12:07.862 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/ 2022-09-12T20:12:07.863 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk= 2022-09-12T20:12:07.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk=', 'Content-Length': '0'} 2022-09-12T20:12:07.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034ba485e10e1be8d-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.870 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:07.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:12:07.871 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:07.872 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:07.873 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:07.874 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:07.875 DEBUG:boto:Method: GET 2022-09-12T20:12:07.875 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:07.876 DEBUG:boto:Data: 2022-09-12T20:12:07.877 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.877 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.878 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.879 DEBUG:boto:Params: {} 2022-09-12T20:12:07.879 DEBUG:boto:Token: None 2022-09-12T20:12:07.880 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:07.881 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KKKOh5LUxriYD/TwibDekPrDB+I= 2022-09-12T20:12:07.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KKKOh5LUxriYD/TwibDekPrDB+I=', 'Content-Length': '0'} 2022-09-12T20:12:07.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004faa32239edc0112-00631f9297-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.885 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:07.886 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:07.887 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:07.887 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:07.888 DEBUG:boto:Method: GET 2022-09-12T20:12:07.889 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:07.890 DEBUG:boto:Data: 2022-09-12T20:12:07.890 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.891 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.892 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.892 DEBUG:boto:Params: {} 2022-09-12T20:12:07.893 DEBUG:boto:Token: None 2022-09-12T20:12:07.894 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:07.895 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KKKOh5LUxriYD/TwibDekPrDB+I= 2022-09-12T20:12:07.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KKKOh5LUxriYD/TwibDekPrDB+I=', 'Content-Length': '0'} 2022-09-12T20:12:07.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ec2a1f1fd230dbf-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.899 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:07.899 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:07.900 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:07.900 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:07.901 DEBUG:boto:Method: HEAD 2022-09-12T20:12:07.901 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:07.902 DEBUG:boto:Data: 2022-09-12T20:12:07.902 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.903 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.903 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.904 DEBUG:boto:Params: {} 2022-09-12T20:12:07.904 DEBUG:boto:Token: None 2022-09-12T20:12:07.905 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:07.905 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Yb5Po6GtQOpO4MnH4ZYp0pjGObo= 2022-09-12T20:12:07.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Yb5Po6GtQOpO4MnH4ZYp0pjGObo=', 'Content-Length': '0'} 2022-09-12T20:12:07.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c58ebaf524f0c4f3-00631f9297-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.908 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:07.909 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:07.910 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:07.910 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:07.911 DEBUG:boto:Method: HEAD 2022-09-12T20:12:07.911 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:07.912 DEBUG:boto:Data: 2022-09-12T20:12:07.912 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.913 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.913 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.914 DEBUG:boto:Params: {} 2022-09-12T20:12:07.914 DEBUG:boto:Token: None 2022-09-12T20:12:07.915 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:07.915 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Yb5Po6GtQOpO4MnH4ZYp0pjGObo= 2022-09-12T20:12:07.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Yb5Po6GtQOpO4MnH4ZYp0pjGObo=', 'Content-Length': '0'} 2022-09-12T20:12:07.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084c30986ae42e819-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:12:07.919 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:07.920 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:07.920 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:07.921 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:07.921 DEBUG:boto:Method: GET 2022-09-12T20:12:07.922 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:07.922 DEBUG:boto:Data: 2022-09-12T20:12:07.923 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.923 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.924 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.924 DEBUG:boto:Params: {} 2022-09-12T20:12:07.925 DEBUG:boto:Token: None 2022-09-12T20:12:07.925 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:07.926 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KKKOh5LUxriYD/TwibDekPrDB+I= 2022-09-12T20:12:07.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KKKOh5LUxriYD/TwibDekPrDB+I=', 'Content-Length': '0'} 2022-09-12T20:12:07.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd6ccabe0d35381f-00631f9297-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.929 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:07.930 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:07.931 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:07.931 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:07.932 DEBUG:boto:Method: GET 2022-09-12T20:12:07.932 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:07.933 DEBUG:boto:Data: 2022-09-12T20:12:07.933 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.934 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.934 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.935 DEBUG:boto:Params: {} 2022-09-12T20:12:07.935 DEBUG:boto:Token: None 2022-09-12T20:12:07.936 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:07.937 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KKKOh5LUxriYD/TwibDekPrDB+I= 2022-09-12T20:12:07.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KKKOh5LUxriYD/TwibDekPrDB+I=', 'Content-Length': '0'} 2022-09-12T20:12:07.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e507c7752f5443e1-00631f9297-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.982 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:07.983 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:07.983 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:12:07.984 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:12:07.984 DEBUG:boto:Method: GET 2022-09-12T20:12:07.985 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:12:07.985 DEBUG:boto:Data: 2022-09-12T20:12:07.986 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.986 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:07.987 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.987 DEBUG:boto:Params: {} 2022-09-12T20:12:07.988 DEBUG:boto:Token: None 2022-09-12T20:12:07.989 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:07 GMT /jdpiak-75/ 2022-09-12T20:12:07.989 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk= 2022-09-12T20:12:07.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lFOrsXCX/QFJNe93zFk812rgFFk=', 'Content-Length': '0'} 2022-09-12T20:12:07.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050faa584ab27ff75-00631f9297-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:07.993 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:07.994 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:07.994 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:07.995 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:12:07.995 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:12:07.996 DEBUG:boto:Method: GET 2022-09-12T20:12:07.997 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:12:07.997 DEBUG:boto:Data: 2022-09-12T20:12:07.998 DEBUG:boto:Headers: {} 2022-09-12T20:12:07.998 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:07.999 DEBUG:boto:Port: 8000 2022-09-12T20:12:07.999 DEBUG:boto:Params: {} 2022-09-12T20:12:08.000 DEBUG:boto:Token: None 2022-09-12T20:12:08.000 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/ 2022-09-12T20:12:08.001 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI= 2022-09-12T20:12:08.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI=', 'Content-Length': '0'} 2022-09-12T20:12:08.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f592da2fa21d2e42-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.006 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:08.007 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:08.007 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:08.008 DEBUG:boto:Method: HEAD 2022-09-12T20:12:08.008 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:12:08.009 DEBUG:boto:Data: 2022-09-12T20:12:08.009 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.010 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.010 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.011 DEBUG:boto:Params: {} 2022-09-12T20:12:08.011 DEBUG:boto:Token: None 2022-09-12T20:12:08.012 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj23 2022-09-12T20:12:08.013 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ttw537JxrOtLbxCERvG+kGKSARc= 2022-09-12T20:12:08.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ttw537JxrOtLbxCERvG+kGKSARc=', 'Content-Length': '0'} 2022-09-12T20:12:08.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099c191f37fa63424-00631f9298-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.016 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:08.016 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:08.017 DEBUG:boto:Method: HEAD 2022-09-12T20:12:08.017 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:12:08.018 DEBUG:boto:Data: 2022-09-12T20:12:08.018 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.019 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.019 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.020 DEBUG:boto:Params: {} 2022-09-12T20:12:08.020 DEBUG:boto:Token: None 2022-09-12T20:12:08.021 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj23 2022-09-12T20:12:08.022 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ttw537JxrOtLbxCERvG+kGKSARc= 2022-09-12T20:12:08.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ttw537JxrOtLbxCERvG+kGKSARc=', 'Content-Length': '0'} 2022-09-12T20:12:08.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f25bd93f0ebbd3f1-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:12:08.026 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:08.026 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:08.027 DEBUG:boto:Method: GET 2022-09-12T20:12:08.027 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:12:08.028 DEBUG:boto:Data: 2022-09-12T20:12:08.028 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.029 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.029 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.030 DEBUG:boto:Params: {} 2022-09-12T20:12:08.031 DEBUG:boto:Token: None 2022-09-12T20:12:08.031 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj23 2022-09-12T20:12:08.032 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bIfRWiWPS4pzUV8hG18PZN4Z+Xs= 2022-09-12T20:12:08.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bIfRWiWPS4pzUV8hG18PZN4Z+Xs=', 'Content-Length': '0'} 2022-09-12T20:12:08.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006790af35797cb870-00631f9298-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.035 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:08.035 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:08.036 DEBUG:boto:Method: GET 2022-09-12T20:12:08.036 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:12:08.037 DEBUG:boto:Data: 2022-09-12T20:12:08.037 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.038 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.039 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.039 DEBUG:boto:Params: {} 2022-09-12T20:12:08.040 DEBUG:boto:Token: None 2022-09-12T20:12:08.040 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj23 2022-09-12T20:12:08.041 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bIfRWiWPS4pzUV8hG18PZN4Z+Xs= 2022-09-12T20:12:08.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bIfRWiWPS4pzUV8hG18PZN4Z+Xs=', 'Content-Length': '0'} 2022-09-12T20:12:08.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007324f5c53274984b-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.044 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:08.045 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:08.045 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:12:08.046 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:12:08.047 DEBUG:boto:Method: GET 2022-09-12T20:12:08.047 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:12:08.048 DEBUG:boto:Data: 2022-09-12T20:12:08.048 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.049 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.049 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.050 DEBUG:boto:Params: {} 2022-09-12T20:12:08.050 DEBUG:boto:Token: None 2022-09-12T20:12:08.051 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/ 2022-09-12T20:12:08.051 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI= 2022-09-12T20:12:08.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI=', 'Content-Length': '0'} 2022-09-12T20:12:08.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011dad402a3bee42f-00631f9298-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.098 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:08.099 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:08.099 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:08.100 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:12:08.100 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:12:08.101 DEBUG:boto:Method: GET 2022-09-12T20:12:08.101 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:12:08.102 DEBUG:boto:Data: 2022-09-12T20:12:08.102 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.103 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.104 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.104 DEBUG:boto:Params: {} 2022-09-12T20:12:08.105 DEBUG:boto:Token: None 2022-09-12T20:12:08.105 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/ 2022-09-12T20:12:08.106 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI= 2022-09-12T20:12:08.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI=', 'Content-Length': '0'} 2022-09-12T20:12:08.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000aa64b2b2c890800-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.112 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:08.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:12:08.113 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:08.113 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:08.114 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:08.115 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:08.115 DEBUG:boto:Method: GET 2022-09-12T20:12:08.116 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:08.116 DEBUG:boto:Data: 2022-09-12T20:12:08.117 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.117 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.118 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.118 DEBUG:boto:Params: {} 2022-09-12T20:12:08.119 DEBUG:boto:Token: None 2022-09-12T20:12:08.119 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:08.120 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b0IPXd9ndhL+HrkIgBMzU5nSSvE= 2022-09-12T20:12:08.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b0IPXd9ndhL+HrkIgBMzU5nSSvE=', 'Content-Length': '0'} 2022-09-12T20:12:08.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000181e1538dd8ea7dd-00631f9298-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.123 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:08.124 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:08.124 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:08.125 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:08.125 DEBUG:boto:Method: GET 2022-09-12T20:12:08.126 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:08.126 DEBUG:boto:Data: 2022-09-12T20:12:08.127 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.127 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.128 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.128 DEBUG:boto:Params: {} 2022-09-12T20:12:08.129 DEBUG:boto:Token: None 2022-09-12T20:12:08.129 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:08.130 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b0IPXd9ndhL+HrkIgBMzU5nSSvE= 2022-09-12T20:12:08.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b0IPXd9ndhL+HrkIgBMzU5nSSvE=', 'Content-Length': '0'} 2022-09-12T20:12:08.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000264063b081a0cd0d-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.134 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:08.134 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:08.135 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:08.135 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:08.136 DEBUG:boto:Method: HEAD 2022-09-12T20:12:08.136 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:08.137 DEBUG:boto:Data: 2022-09-12T20:12:08.137 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.138 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.138 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.139 DEBUG:boto:Params: {} 2022-09-12T20:12:08.139 DEBUG:boto:Token: None 2022-09-12T20:12:08.140 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:08.140 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FAEnsY6JFwD+dgxj2lrcfEhknvU= 2022-09-12T20:12:08.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FAEnsY6JFwD+dgxj2lrcfEhknvU=', 'Content-Length': '0'} 2022-09-12T20:12:08.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bbe498e44a64155-00631f9298-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.143 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:08.144 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:08.144 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:08.145 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:08.145 DEBUG:boto:Method: HEAD 2022-09-12T20:12:08.146 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:08.146 DEBUG:boto:Data: 2022-09-12T20:12:08.147 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.147 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.148 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.148 DEBUG:boto:Params: {} 2022-09-12T20:12:08.149 DEBUG:boto:Token: None 2022-09-12T20:12:08.150 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:08.150 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FAEnsY6JFwD+dgxj2lrcfEhknvU= 2022-09-12T20:12:08.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FAEnsY6JFwD+dgxj2lrcfEhknvU=', 'Content-Length': '0'} 2022-09-12T20:12:08.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1a8efe9ad9c8901-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:12:08.154 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:08.154 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:08.155 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:08.155 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:08.156 DEBUG:boto:Method: GET 2022-09-12T20:12:08.156 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:08.157 DEBUG:boto:Data: 2022-09-12T20:12:08.157 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.158 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.158 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.159 DEBUG:boto:Params: {} 2022-09-12T20:12:08.159 DEBUG:boto:Token: None 2022-09-12T20:12:08.160 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:08.161 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b0IPXd9ndhL+HrkIgBMzU5nSSvE= 2022-09-12T20:12:08.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b0IPXd9ndhL+HrkIgBMzU5nSSvE=', 'Content-Length': '0'} 2022-09-12T20:12:08.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c0b8494802caed7-00631f9298-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.206 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:08.207 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:08.207 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:08.208 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:08.208 DEBUG:boto:Method: GET 2022-09-12T20:12:08.209 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:08.209 DEBUG:boto:Data: 2022-09-12T20:12:08.210 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.210 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.211 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.212 DEBUG:boto:Params: {} 2022-09-12T20:12:08.212 DEBUG:boto:Token: None 2022-09-12T20:12:08.213 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:08.213 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b0IPXd9ndhL+HrkIgBMzU5nSSvE= 2022-09-12T20:12:08.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b0IPXd9ndhL+HrkIgBMzU5nSSvE=', 'Content-Length': '0'} 2022-09-12T20:12:08.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002be1635284fa9da7-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.217 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:08.218 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:08.218 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:12:08.219 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:12:08.219 DEBUG:boto:Method: GET 2022-09-12T20:12:08.220 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:12:08.220 DEBUG:boto:Data: 2022-09-12T20:12:08.221 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.221 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.222 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.223 DEBUG:boto:Params: {} 2022-09-12T20:12:08.223 DEBUG:boto:Token: None 2022-09-12T20:12:08.224 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/ 2022-09-12T20:12:08.224 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI= 2022-09-12T20:12:08.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI=', 'Content-Length': '0'} 2022-09-12T20:12:08.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002ed188e2c1a07b2-00631f9298-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.228 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:08.229 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:08.230 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:08.230 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:12:08.231 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:12:08.231 DEBUG:boto:Method: GET 2022-09-12T20:12:08.232 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:12:08.232 DEBUG:boto:Data: 2022-09-12T20:12:08.233 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.233 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.234 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.235 DEBUG:boto:Params: {} 2022-09-12T20:12:08.235 DEBUG:boto:Token: None 2022-09-12T20:12:08.236 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/ 2022-09-12T20:12:08.236 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI= 2022-09-12T20:12:08.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI=', 'Content-Length': '0'} 2022-09-12T20:12:08.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c039ad5794633774-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.242 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:08.242 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:08.243 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:08.244 DEBUG:boto:Method: HEAD 2022-09-12T20:12:08.244 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:12:08.245 DEBUG:boto:Data: 2022-09-12T20:12:08.245 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.245 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.246 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.247 DEBUG:boto:Params: {} 2022-09-12T20:12:08.247 DEBUG:boto:Token: None 2022-09-12T20:12:08.248 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj29 2022-09-12T20:12:08.248 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qteolvvX7OMVyfOsNCzQmhnIWmg= 2022-09-12T20:12:08.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qteolvvX7OMVyfOsNCzQmhnIWmg=', 'Content-Length': '0'} 2022-09-12T20:12:08.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9dfa2d74afc76b4-00631f9298-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.252 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:08.252 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:08.253 DEBUG:boto:Method: HEAD 2022-09-12T20:12:08.253 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:12:08.254 DEBUG:boto:Data: 2022-09-12T20:12:08.254 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.255 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.255 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.256 DEBUG:boto:Params: {} 2022-09-12T20:12:08.256 DEBUG:boto:Token: None 2022-09-12T20:12:08.257 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj29 2022-09-12T20:12:08.257 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qteolvvX7OMVyfOsNCzQmhnIWmg= 2022-09-12T20:12:08.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qteolvvX7OMVyfOsNCzQmhnIWmg=', 'Content-Length': '0'} 2022-09-12T20:12:08.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004526b30a4acf2c6a-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:12:08.262 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:08.263 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:08.263 DEBUG:boto:Method: GET 2022-09-12T20:12:08.264 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:12:08.264 DEBUG:boto:Data: 2022-09-12T20:12:08.265 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.265 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.266 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.266 DEBUG:boto:Params: {} 2022-09-12T20:12:08.267 DEBUG:boto:Token: None 2022-09-12T20:12:08.267 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj29 2022-09-12T20:12:08.268 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7bVWVHQnmMbMr4UvsHBunHmyo4E= 2022-09-12T20:12:08.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7bVWVHQnmMbMr4UvsHBunHmyo4E=', 'Content-Length': '0'} 2022-09-12T20:12:08.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008591c172b221d725-00631f9298-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.271 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:08.272 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:08.273 DEBUG:boto:Method: GET 2022-09-12T20:12:08.273 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:12:08.274 DEBUG:boto:Data: 2022-09-12T20:12:08.274 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.275 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.275 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.276 DEBUG:boto:Params: {} 2022-09-12T20:12:08.277 DEBUG:boto:Token: None 2022-09-12T20:12:08.277 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj29 2022-09-12T20:12:08.278 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7bVWVHQnmMbMr4UvsHBunHmyo4E= 2022-09-12T20:12:08.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7bVWVHQnmMbMr4UvsHBunHmyo4E=', 'Content-Length': '0'} 2022-09-12T20:12:08.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a7ba1ad0b79f879-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.281 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:08.282 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:08.282 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:12:08.283 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:12:08.283 DEBUG:boto:Method: GET 2022-09-12T20:12:08.284 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:12:08.284 DEBUG:boto:Data: 2022-09-12T20:12:08.285 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.285 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.286 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.287 DEBUG:boto:Params: {} 2022-09-12T20:12:08.287 DEBUG:boto:Token: None 2022-09-12T20:12:08.288 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/ 2022-09-12T20:12:08.288 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI= 2022-09-12T20:12:08.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI=', 'Content-Length': '0'} 2022-09-12T20:12:08.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000815f5c55b1d2c175-00631f9298-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.334 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:08.335 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:08.335 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:08.336 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:12:08.337 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:12:08.337 DEBUG:boto:Method: GET 2022-09-12T20:12:08.338 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:12:08.338 DEBUG:boto:Data: 2022-09-12T20:12:08.339 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.339 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.340 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.340 DEBUG:boto:Params: {} 2022-09-12T20:12:08.341 DEBUG:boto:Token: None 2022-09-12T20:12:08.341 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/ 2022-09-12T20:12:08.342 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI= 2022-09-12T20:12:08.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI=', 'Content-Length': '0'} 2022-09-12T20:12:08.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000baffd436f78e49e6-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.348 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:08.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:12:08.349 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:08.349 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:08.350 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:08.350 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:08.351 DEBUG:boto:Method: GET 2022-09-12T20:12:08.352 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:08.353 DEBUG:boto:Data: 2022-09-12T20:12:08.353 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.354 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.354 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.355 DEBUG:boto:Params: {} 2022-09-12T20:12:08.355 DEBUG:boto:Token: None 2022-09-12T20:12:08.356 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:08.356 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fGKPZo8Hf8zulVrFr1z4MzRGKP4= 2022-09-12T20:12:08.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fGKPZo8Hf8zulVrFr1z4MzRGKP4=', 'Content-Length': '0'} 2022-09-12T20:12:08.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000711f09e3950b092-00631f9298-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.360 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:08.360 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:08.361 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:08.362 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:08.362 DEBUG:boto:Method: GET 2022-09-12T20:12:08.363 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:08.363 DEBUG:boto:Data: 2022-09-12T20:12:08.364 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.364 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.365 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.365 DEBUG:boto:Params: {} 2022-09-12T20:12:08.366 DEBUG:boto:Token: None 2022-09-12T20:12:08.367 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:08.368 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fGKPZo8Hf8zulVrFr1z4MzRGKP4= 2022-09-12T20:12:08.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fGKPZo8Hf8zulVrFr1z4MzRGKP4=', 'Content-Length': '0'} 2022-09-12T20:12:08.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057363d9cf736866f-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.414 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:08.415 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:08.415 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:08.416 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:08.416 DEBUG:boto:Method: HEAD 2022-09-12T20:12:08.417 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:08.417 DEBUG:boto:Data: 2022-09-12T20:12:08.418 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.418 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.419 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.419 DEBUG:boto:Params: {} 2022-09-12T20:12:08.420 DEBUG:boto:Token: None 2022-09-12T20:12:08.420 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:08.421 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uidcLHp416uK3U6qnlIUTtPyeIY= 2022-09-12T20:12:08.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uidcLHp416uK3U6qnlIUTtPyeIY=', 'Content-Length': '0'} 2022-09-12T20:12:08.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000616f27978b65a47a-00631f9298-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.424 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:08.425 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:08.425 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:08.426 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:08.426 DEBUG:boto:Method: HEAD 2022-09-12T20:12:08.427 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:08.427 DEBUG:boto:Data: 2022-09-12T20:12:08.428 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.428 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.429 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.429 DEBUG:boto:Params: {} 2022-09-12T20:12:08.430 DEBUG:boto:Token: None 2022-09-12T20:12:08.430 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:08.431 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uidcLHp416uK3U6qnlIUTtPyeIY= 2022-09-12T20:12:08.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uidcLHp416uK3U6qnlIUTtPyeIY=', 'Content-Length': '0'} 2022-09-12T20:12:08.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2cde9740dffa2b8-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:12:08.435 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:08.436 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:08.436 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:08.437 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:08.437 DEBUG:boto:Method: GET 2022-09-12T20:12:08.438 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:08.438 DEBUG:boto:Data: 2022-09-12T20:12:08.439 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.439 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.440 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.440 DEBUG:boto:Params: {} 2022-09-12T20:12:08.441 DEBUG:boto:Token: None 2022-09-12T20:12:08.442 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:08.442 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fGKPZo8Hf8zulVrFr1z4MzRGKP4= 2022-09-12T20:12:08.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fGKPZo8Hf8zulVrFr1z4MzRGKP4=', 'Content-Length': '0'} 2022-09-12T20:12:08.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1336865c74bc68f-00631f9298-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.445 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:08.446 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:08.446 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:08.447 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:08.448 DEBUG:boto:Method: GET 2022-09-12T20:12:08.448 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:08.449 DEBUG:boto:Data: 2022-09-12T20:12:08.449 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.450 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.450 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.451 DEBUG:boto:Params: {} 2022-09-12T20:12:08.451 DEBUG:boto:Token: None 2022-09-12T20:12:08.452 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:08.453 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fGKPZo8Hf8zulVrFr1z4MzRGKP4= 2022-09-12T20:12:08.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fGKPZo8Hf8zulVrFr1z4MzRGKP4=', 'Content-Length': '0'} 2022-09-12T20:12:08.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf3d022a1fc18333-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.456 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:08.457 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:08.458 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:12:08.459 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:12:08.459 DEBUG:boto:Method: GET 2022-09-12T20:12:08.460 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:12:08.460 DEBUG:boto:Data: 2022-09-12T20:12:08.461 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.461 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.462 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.462 DEBUG:boto:Params: {} 2022-09-12T20:12:08.463 DEBUG:boto:Token: None 2022-09-12T20:12:08.464 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/ 2022-09-12T20:12:08.464 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI= 2022-09-12T20:12:08.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI=', 'Content-Length': '0'} 2022-09-12T20:12:08.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007226b68a4e9b60de-00631f9298-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.468 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:08.469 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:08.470 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:08.470 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:12:08.471 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:12:08.471 DEBUG:boto:Method: GET 2022-09-12T20:12:08.472 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:12:08.472 DEBUG:boto:Data: 2022-09-12T20:12:08.473 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.473 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.474 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.474 DEBUG:boto:Params: {} 2022-09-12T20:12:08.475 DEBUG:boto:Token: None 2022-09-12T20:12:08.476 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/ 2022-09-12T20:12:08.476 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI= 2022-09-12T20:12:08.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI=', 'Content-Length': '0'} 2022-09-12T20:12:08.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006898d28defed3a94-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.481 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:08.482 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:08.483 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:08.483 DEBUG:boto:Method: HEAD 2022-09-12T20:12:08.484 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:12:08.484 DEBUG:boto:Data: 2022-09-12T20:12:08.485 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.485 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.486 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.486 DEBUG:boto:Params: {} 2022-09-12T20:12:08.487 DEBUG:boto:Token: None 2022-09-12T20:12:08.488 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj33 2022-09-12T20:12:08.488 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pphWpBdP4sq2VxUdTOg88XgFUvM= 2022-09-12T20:12:08.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pphWpBdP4sq2VxUdTOg88XgFUvM=', 'Content-Length': '0'} 2022-09-12T20:12:08.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061db4314fcb6cfb0-00631f9298-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.492 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:08.492 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:08.493 DEBUG:boto:Method: HEAD 2022-09-12T20:12:08.493 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:12:08.494 DEBUG:boto:Data: 2022-09-12T20:12:08.494 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.495 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.495 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.496 DEBUG:boto:Params: {} 2022-09-12T20:12:08.496 DEBUG:boto:Token: None 2022-09-12T20:12:08.497 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj33 2022-09-12T20:12:08.497 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pphWpBdP4sq2VxUdTOg88XgFUvM= 2022-09-12T20:12:08.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pphWpBdP4sq2VxUdTOg88XgFUvM=', 'Content-Length': '0'} 2022-09-12T20:12:08.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007256ce0b78892f59-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:12:08.501 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:08.502 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:08.503 DEBUG:boto:Method: GET 2022-09-12T20:12:08.503 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:12:08.504 DEBUG:boto:Data: 2022-09-12T20:12:08.504 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.505 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.505 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.506 DEBUG:boto:Params: {} 2022-09-12T20:12:08.506 DEBUG:boto:Token: None 2022-09-12T20:12:08.507 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj33 2022-09-12T20:12:08.508 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B2N3czidEDH+pR9cXu7D2Dilhu8= 2022-09-12T20:12:08.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B2N3czidEDH+pR9cXu7D2Dilhu8=', 'Content-Length': '0'} 2022-09-12T20:12:08.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000428b803d9ef3d521-00631f9298-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.554 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:08.555 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:08.555 DEBUG:boto:Method: GET 2022-09-12T20:12:08.556 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:12:08.556 DEBUG:boto:Data: 2022-09-12T20:12:08.557 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.557 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.558 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.558 DEBUG:boto:Params: {} 2022-09-12T20:12:08.559 DEBUG:boto:Token: None 2022-09-12T20:12:08.559 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj33 2022-09-12T20:12:08.560 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B2N3czidEDH+pR9cXu7D2Dilhu8= 2022-09-12T20:12:08.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B2N3czidEDH+pR9cXu7D2Dilhu8=', 'Content-Length': '0'} 2022-09-12T20:12:08.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1ecf6da90d3ddf1-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.564 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:08.564 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:08.565 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:12:08.565 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:12:08.566 DEBUG:boto:Method: GET 2022-09-12T20:12:08.566 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:12:08.567 DEBUG:boto:Data: 2022-09-12T20:12:08.567 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.568 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.568 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.569 DEBUG:boto:Params: {} 2022-09-12T20:12:08.570 DEBUG:boto:Token: None 2022-09-12T20:12:08.570 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/ 2022-09-12T20:12:08.571 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI= 2022-09-12T20:12:08.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI=', 'Content-Length': '0'} 2022-09-12T20:12:08.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a7b116d417e7f60-00631f9298-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.575 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:08.575 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:08.576 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:08.577 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:12:08.577 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:12:08.578 DEBUG:boto:Method: GET 2022-09-12T20:12:08.578 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:12:08.579 DEBUG:boto:Data: 2022-09-12T20:12:08.579 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.580 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.580 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.581 DEBUG:boto:Params: {} 2022-09-12T20:12:08.581 DEBUG:boto:Token: None 2022-09-12T20:12:08.582 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/ 2022-09-12T20:12:08.582 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI= 2022-09-12T20:12:08.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI=', 'Content-Length': '0'} 2022-09-12T20:12:08.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c03a7b8bfe236e7c-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.588 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:08.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:12:08.589 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:08.589 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:08.590 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:08.590 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:08.591 DEBUG:boto:Method: GET 2022-09-12T20:12:08.591 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:08.592 DEBUG:boto:Data: 2022-09-12T20:12:08.593 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.593 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.594 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.594 DEBUG:boto:Params: {} 2022-09-12T20:12:08.595 DEBUG:boto:Token: None 2022-09-12T20:12:08.595 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:08.596 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TMKG5VNZHBsL1KvUEj7JRgPhXTs= 2022-09-12T20:12:08.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TMKG5VNZHBsL1KvUEj7JRgPhXTs=', 'Content-Length': '0'} 2022-09-12T20:12:08.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e500c49996d7a13-00631f9298-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.599 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:08.600 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:08.601 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:08.601 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:08.602 DEBUG:boto:Method: GET 2022-09-12T20:12:08.602 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:08.603 DEBUG:boto:Data: 2022-09-12T20:12:08.603 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.604 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.604 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.605 DEBUG:boto:Params: {} 2022-09-12T20:12:08.605 DEBUG:boto:Token: None 2022-09-12T20:12:08.606 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:08.606 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TMKG5VNZHBsL1KvUEj7JRgPhXTs= 2022-09-12T20:12:08.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TMKG5VNZHBsL1KvUEj7JRgPhXTs=', 'Content-Length': '0'} 2022-09-12T20:12:08.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f64a8d8f582385a-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.610 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:08.611 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:08.611 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:08.612 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:08.612 DEBUG:boto:Method: HEAD 2022-09-12T20:12:08.613 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:08.613 DEBUG:boto:Data: 2022-09-12T20:12:08.614 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.615 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.616 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.617 DEBUG:boto:Params: {} 2022-09-12T20:12:08.618 DEBUG:boto:Token: None 2022-09-12T20:12:08.619 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:08.619 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:65+wDYzIni3VqbzEKJMzM+vzCKA= 2022-09-12T20:12:08.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:65+wDYzIni3VqbzEKJMzM+vzCKA=', 'Content-Length': '0'} 2022-09-12T20:12:08.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de38b45c40b2ecab-00631f9298-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.623 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:08.623 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:08.624 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:08.624 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:08.625 DEBUG:boto:Method: HEAD 2022-09-12T20:12:08.625 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:08.626 DEBUG:boto:Data: 2022-09-12T20:12:08.626 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.627 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.627 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.628 DEBUG:boto:Params: {} 2022-09-12T20:12:08.628 DEBUG:boto:Token: None 2022-09-12T20:12:08.629 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:08.629 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:65+wDYzIni3VqbzEKJMzM+vzCKA= 2022-09-12T20:12:08.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:65+wDYzIni3VqbzEKJMzM+vzCKA=', 'Content-Length': '0'} 2022-09-12T20:12:08.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031b6d68aeea3db78-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:12:08.633 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:08.634 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:08.634 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:08.635 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:08.635 DEBUG:boto:Method: GET 2022-09-12T20:12:08.636 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:08.636 DEBUG:boto:Data: 2022-09-12T20:12:08.637 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.637 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.638 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.638 DEBUG:boto:Params: {} 2022-09-12T20:12:08.639 DEBUG:boto:Token: None 2022-09-12T20:12:08.640 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:08.640 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TMKG5VNZHBsL1KvUEj7JRgPhXTs= 2022-09-12T20:12:08.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TMKG5VNZHBsL1KvUEj7JRgPhXTs=', 'Content-Length': '0'} 2022-09-12T20:12:08.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b929e77ff7a9b36-00631f9298-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.686 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:08.687 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:08.687 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:08.688 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:08.688 DEBUG:boto:Method: GET 2022-09-12T20:12:08.689 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:08.690 DEBUG:boto:Data: 2022-09-12T20:12:08.690 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.691 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.691 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.692 DEBUG:boto:Params: {} 2022-09-12T20:12:08.692 DEBUG:boto:Token: None 2022-09-12T20:12:08.693 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:08.693 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TMKG5VNZHBsL1KvUEj7JRgPhXTs= 2022-09-12T20:12:08.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TMKG5VNZHBsL1KvUEj7JRgPhXTs=', 'Content-Length': '0'} 2022-09-12T20:12:08.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096900905ba4313b7-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.697 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:08.698 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:08.698 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:12:08.699 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:12:08.699 DEBUG:boto:Method: GET 2022-09-12T20:12:08.700 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:12:08.700 DEBUG:boto:Data: 2022-09-12T20:12:08.701 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.701 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.702 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.703 DEBUG:boto:Params: {} 2022-09-12T20:12:08.703 DEBUG:boto:Token: None 2022-09-12T20:12:08.704 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/ 2022-09-12T20:12:08.705 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI= 2022-09-12T20:12:08.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI=', 'Content-Length': '0'} 2022-09-12T20:12:08.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab1ff30ed09f6a68-00631f9298-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.709 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:08.710 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:08.710 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:08.711 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:12:08.711 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:12:08.712 DEBUG:boto:Method: GET 2022-09-12T20:12:08.712 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:12:08.713 DEBUG:boto:Data: 2022-09-12T20:12:08.713 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.714 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.714 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.715 DEBUG:boto:Params: {} 2022-09-12T20:12:08.715 DEBUG:boto:Token: None 2022-09-12T20:12:08.716 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/ 2022-09-12T20:12:08.717 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI= 2022-09-12T20:12:08.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI=', 'Content-Length': '0'} 2022-09-12T20:12:08.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bdd761dc1758ad8-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.722 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:08.723 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:08.723 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:08.724 DEBUG:boto:Method: HEAD 2022-09-12T20:12:08.724 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:12:08.725 DEBUG:boto:Data: 2022-09-12T20:12:08.725 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.726 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.726 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.727 DEBUG:boto:Params: {} 2022-09-12T20:12:08.727 DEBUG:boto:Token: None 2022-09-12T20:12:08.728 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj34 2022-09-12T20:12:08.728 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WoQQbVfgNuqiti6tAcK60ih+OR8= 2022-09-12T20:12:08.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WoQQbVfgNuqiti6tAcK60ih+OR8=', 'Content-Length': '0'} 2022-09-12T20:12:08.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046eb1e17505dc6fc-00631f9298-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.741 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:08.742 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:08.742 DEBUG:boto:Method: HEAD 2022-09-12T20:12:08.743 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:12:08.743 DEBUG:boto:Data: 2022-09-12T20:12:08.744 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.744 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.745 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.745 DEBUG:boto:Params: {} 2022-09-12T20:12:08.746 DEBUG:boto:Token: None 2022-09-12T20:12:08.746 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj34 2022-09-12T20:12:08.747 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WoQQbVfgNuqiti6tAcK60ih+OR8= 2022-09-12T20:12:08.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WoQQbVfgNuqiti6tAcK60ih+OR8=', 'Content-Length': '0'} 2022-09-12T20:12:08.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d32dd62f7fdda155-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:12:08.751 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:08.751 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:08.752 DEBUG:boto:Method: GET 2022-09-12T20:12:08.752 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:12:08.753 DEBUG:boto:Data: 2022-09-12T20:12:08.753 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.754 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.754 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.755 DEBUG:boto:Params: {} 2022-09-12T20:12:08.756 DEBUG:boto:Token: None 2022-09-12T20:12:08.756 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj34 2022-09-12T20:12:08.757 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gfG+H3bB2UI9aDHpgST4bjhptTo= 2022-09-12T20:12:08.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gfG+H3bB2UI9aDHpgST4bjhptTo=', 'Content-Length': '0'} 2022-09-12T20:12:08.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000289626cb1141c49b-00631f9298-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.760 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:08.761 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:08.761 DEBUG:boto:Method: GET 2022-09-12T20:12:08.762 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:12:08.763 DEBUG:boto:Data: 2022-09-12T20:12:08.763 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.764 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.764 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.765 DEBUG:boto:Params: {} 2022-09-12T20:12:08.765 DEBUG:boto:Token: None 2022-09-12T20:12:08.766 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj34 2022-09-12T20:12:08.766 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gfG+H3bB2UI9aDHpgST4bjhptTo= 2022-09-12T20:12:08.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gfG+H3bB2UI9aDHpgST4bjhptTo=', 'Content-Length': '0'} 2022-09-12T20:12:08.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de0c5fc68499b721-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.770 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:08.771 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:08.771 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:12:08.772 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:12:08.772 DEBUG:boto:Method: GET 2022-09-12T20:12:08.773 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:12:08.773 DEBUG:boto:Data: 2022-09-12T20:12:08.774 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.774 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.775 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.775 DEBUG:boto:Params: {} 2022-09-12T20:12:08.776 DEBUG:boto:Token: None 2022-09-12T20:12:08.777 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/ 2022-09-12T20:12:08.777 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI= 2022-09-12T20:12:08.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI=', 'Content-Length': '0'} 2022-09-12T20:12:08.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e80d95ae07890e75-00631f9298-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.781 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:08.782 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:08.782 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:08.783 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:12:08.783 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:12:08.784 DEBUG:boto:Method: GET 2022-09-12T20:12:08.785 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:12:08.785 DEBUG:boto:Data: 2022-09-12T20:12:08.786 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.786 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.787 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.787 DEBUG:boto:Params: {} 2022-09-12T20:12:08.788 DEBUG:boto:Token: None 2022-09-12T20:12:08.788 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/ 2022-09-12T20:12:08.789 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI= 2022-09-12T20:12:08.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI=', 'Content-Length': '0'} 2022-09-12T20:12:08.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a182067038fc7d7-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.794 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:08.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:12:08.796 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:08.796 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:08.797 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:08.797 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:08.798 DEBUG:boto:Method: GET 2022-09-12T20:12:08.798 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:08.799 DEBUG:boto:Data: 2022-09-12T20:12:08.799 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.800 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.801 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.801 DEBUG:boto:Params: {} 2022-09-12T20:12:08.802 DEBUG:boto:Token: None 2022-09-12T20:12:08.802 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:08.803 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jxQCLPLwLtR484wObnTFUxoL02o= 2022-09-12T20:12:08.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jxQCLPLwLtR484wObnTFUxoL02o=', 'Content-Length': '0'} 2022-09-12T20:12:08.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002894f7e6f5eb8ab2-00631f9298-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.806 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:08.807 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:08.808 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:08.808 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:08.809 DEBUG:boto:Method: GET 2022-09-12T20:12:08.810 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:08.811 DEBUG:boto:Data: 2022-09-12T20:12:08.812 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.813 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.814 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.814 DEBUG:boto:Params: {} 2022-09-12T20:12:08.815 DEBUG:boto:Token: None 2022-09-12T20:12:08.816 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:08.817 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jxQCLPLwLtR484wObnTFUxoL02o= 2022-09-12T20:12:08.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jxQCLPLwLtR484wObnTFUxoL02o=', 'Content-Length': '0'} 2022-09-12T20:12:08.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b1e3025e95e0c82-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.822 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:08.823 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:08.824 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:08.824 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:08.825 DEBUG:boto:Method: HEAD 2022-09-12T20:12:08.826 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:08.827 DEBUG:boto:Data: 2022-09-12T20:12:08.828 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.828 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.829 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.830 DEBUG:boto:Params: {} 2022-09-12T20:12:08.831 DEBUG:boto:Token: None 2022-09-12T20:12:08.832 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:08.833 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Pw9fk+5SRMQCemit+KyVNaZOfLc= 2022-09-12T20:12:08.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Pw9fk+5SRMQCemit+KyVNaZOfLc=', 'Content-Length': '0'} 2022-09-12T20:12:08.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098f7a8e9f308fa16-00631f9298-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.837 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:08.837 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:08.838 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:08.839 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:08.840 DEBUG:boto:Method: HEAD 2022-09-12T20:12:08.841 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:08.841 DEBUG:boto:Data: 2022-09-12T20:12:08.842 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.843 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.844 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.844 DEBUG:boto:Params: {} 2022-09-12T20:12:08.845 DEBUG:boto:Token: None 2022-09-12T20:12:08.846 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:08.847 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Pw9fk+5SRMQCemit+KyVNaZOfLc= 2022-09-12T20:12:08.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Pw9fk+5SRMQCemit+KyVNaZOfLc=', 'Content-Length': '0'} 2022-09-12T20:12:08.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a65d2bf6c4709096-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:12:08.852 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:08.853 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:08.854 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:08.855 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:08.856 DEBUG:boto:Method: GET 2022-09-12T20:12:08.857 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:08.858 DEBUG:boto:Data: 2022-09-12T20:12:08.858 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.859 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.859 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.860 DEBUG:boto:Params: {} 2022-09-12T20:12:08.860 DEBUG:boto:Token: None 2022-09-12T20:12:08.861 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:08.862 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jxQCLPLwLtR484wObnTFUxoL02o= 2022-09-12T20:12:08.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jxQCLPLwLtR484wObnTFUxoL02o=', 'Content-Length': '0'} 2022-09-12T20:12:08.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020b205638c381e18-00631f9298-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.865 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:08.866 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:08.866 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:08.867 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:08.867 DEBUG:boto:Method: GET 2022-09-12T20:12:08.868 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:08.868 DEBUG:boto:Data: 2022-09-12T20:12:08.869 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.869 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.870 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.870 DEBUG:boto:Params: {} 2022-09-12T20:12:08.871 DEBUG:boto:Token: None 2022-09-12T20:12:08.872 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:08.872 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jxQCLPLwLtR484wObnTFUxoL02o= 2022-09-12T20:12:08.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jxQCLPLwLtR484wObnTFUxoL02o=', 'Content-Length': '0'} 2022-09-12T20:12:08.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b34f70f27bc582b6-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.918 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:08.919 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:08.919 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:12:08.920 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:12:08.920 DEBUG:boto:Method: GET 2022-09-12T20:12:08.921 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:12:08.921 DEBUG:boto:Data: 2022-09-12T20:12:08.922 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.922 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.923 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.923 DEBUG:boto:Params: {} 2022-09-12T20:12:08.924 DEBUG:boto:Token: None 2022-09-12T20:12:08.925 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/ 2022-09-12T20:12:08.925 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI= 2022-09-12T20:12:08.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI=', 'Content-Length': '0'} 2022-09-12T20:12:08.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dce0ba0bd82d5e85-00631f9298-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.929 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:08.930 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:08.931 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:08.931 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:12:08.932 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:12:08.932 DEBUG:boto:Method: GET 2022-09-12T20:12:08.933 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:12:08.933 DEBUG:boto:Data: 2022-09-12T20:12:08.934 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.934 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.935 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.935 DEBUG:boto:Params: {} 2022-09-12T20:12:08.936 DEBUG:boto:Token: None 2022-09-12T20:12:08.936 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/ 2022-09-12T20:12:08.937 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI= 2022-09-12T20:12:08.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI=', 'Content-Length': '0'} 2022-09-12T20:12:08.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ba91d3de8d97b88-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.942 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:08.943 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:08.943 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:08.944 DEBUG:boto:Method: HEAD 2022-09-12T20:12:08.944 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:12:08.945 DEBUG:boto:Data: 2022-09-12T20:12:08.945 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.946 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.946 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.947 DEBUG:boto:Params: {} 2022-09-12T20:12:08.947 DEBUG:boto:Token: None 2022-09-12T20:12:08.948 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj38 2022-09-12T20:12:08.948 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jXO2Ynf5i//ZJ4u6JDQwLHYzW1Y= 2022-09-12T20:12:08.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jXO2Ynf5i//ZJ4u6JDQwLHYzW1Y=', 'Content-Length': '0'} 2022-09-12T20:12:08.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cad47c52dd0dc988-00631f9298-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.952 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:08.952 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:08.953 DEBUG:boto:Method: HEAD 2022-09-12T20:12:08.953 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:12:08.954 DEBUG:boto:Data: 2022-09-12T20:12:08.954 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.955 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.955 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.956 DEBUG:boto:Params: {} 2022-09-12T20:12:08.956 DEBUG:boto:Token: None 2022-09-12T20:12:08.957 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj38 2022-09-12T20:12:08.957 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jXO2Ynf5i//ZJ4u6JDQwLHYzW1Y= 2022-09-12T20:12:08.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jXO2Ynf5i//ZJ4u6JDQwLHYzW1Y=', 'Content-Length': '0'} 2022-09-12T20:12:08.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d182c14078cc256-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:12:08.961 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:08.962 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:08.962 DEBUG:boto:Method: GET 2022-09-12T20:12:08.963 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:12:08.963 DEBUG:boto:Data: 2022-09-12T20:12:08.964 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.964 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.965 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.965 DEBUG:boto:Params: {} 2022-09-12T20:12:08.966 DEBUG:boto:Token: None 2022-09-12T20:12:08.966 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj38 2022-09-12T20:12:08.967 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y+MEkXPL+y7sTf3KP8RyLSmBK9w= 2022-09-12T20:12:08.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y+MEkXPL+y7sTf3KP8RyLSmBK9w=', 'Content-Length': '0'} 2022-09-12T20:12:08.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b9fafe5f32d5188-00631f9298-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.970 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:08.971 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:08.972 DEBUG:boto:Method: GET 2022-09-12T20:12:08.972 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:12:08.973 DEBUG:boto:Data: 2022-09-12T20:12:08.973 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.974 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:08.974 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.975 DEBUG:boto:Params: {} 2022-09-12T20:12:08.975 DEBUG:boto:Token: None 2022-09-12T20:12:08.976 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/obj38 2022-09-12T20:12:08.976 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y+MEkXPL+y7sTf3KP8RyLSmBK9w= 2022-09-12T20:12:08.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y+MEkXPL+y7sTf3KP8RyLSmBK9w=', 'Content-Length': '0'} 2022-09-12T20:12:08.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a73e027493310be9-00631f9298-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:08.980 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:08.981 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:08.981 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:12:08.982 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:12:08.982 DEBUG:boto:Method: GET 2022-09-12T20:12:08.983 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:12:08.983 DEBUG:boto:Data: 2022-09-12T20:12:08.984 DEBUG:boto:Headers: {} 2022-09-12T20:12:08.984 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:08.985 DEBUG:boto:Port: 8000 2022-09-12T20:12:08.985 DEBUG:boto:Params: {} 2022-09-12T20:12:08.986 DEBUG:boto:Token: None 2022-09-12T20:12:08.986 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:08 GMT /jdpiak-75/ 2022-09-12T20:12:08.987 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI= 2022-09-12T20:12:08.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Soys8aIfBCkBL49hKwdvudLzEeI=', 'Content-Length': '0'} 2022-09-12T20:12:08.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eec7df634d1a4453-00631f9298-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.034 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:09.035 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:09.035 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:09.036 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:12:09.036 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:12:09.037 DEBUG:boto:Method: GET 2022-09-12T20:12:09.037 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:12:09.038 DEBUG:boto:Data: 2022-09-12T20:12:09.038 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.039 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.039 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.040 DEBUG:boto:Params: {} 2022-09-12T20:12:09.040 DEBUG:boto:Token: None 2022-09-12T20:12:09.041 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/ 2022-09-12T20:12:09.041 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ= 2022-09-12T20:12:09.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ=', 'Content-Length': '0'} 2022-09-12T20:12:09.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000def56b7888f3712a-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.047 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:09.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:12:09.048 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:09.049 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:09.049 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:09.050 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:09.051 DEBUG:boto:Method: GET 2022-09-12T20:12:09.051 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:09.052 DEBUG:boto:Data: 2022-09-12T20:12:09.052 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.053 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.053 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.054 DEBUG:boto:Params: {} 2022-09-12T20:12:09.054 DEBUG:boto:Token: None 2022-09-12T20:12:09.055 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:09.056 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ssdZ8nmPUN9aq0qSy4FLmxr07n8= 2022-09-12T20:12:09.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ssdZ8nmPUN9aq0qSy4FLmxr07n8=', 'Content-Length': '0'} 2022-09-12T20:12:09.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7b43dd4b5c05622-00631f9299-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.059 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:09.060 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:09.060 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:09.061 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:09.061 DEBUG:boto:Method: GET 2022-09-12T20:12:09.062 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:09.062 DEBUG:boto:Data: 2022-09-12T20:12:09.063 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.063 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.064 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.064 DEBUG:boto:Params: {} 2022-09-12T20:12:09.065 DEBUG:boto:Token: None 2022-09-12T20:12:09.065 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:09.066 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ssdZ8nmPUN9aq0qSy4FLmxr07n8= 2022-09-12T20:12:09.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ssdZ8nmPUN9aq0qSy4FLmxr07n8=', 'Content-Length': '0'} 2022-09-12T20:12:09.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdd8391b05bd2951-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.070 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:09.070 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:09.071 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:09.071 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:09.072 DEBUG:boto:Method: HEAD 2022-09-12T20:12:09.072 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:09.073 DEBUG:boto:Data: 2022-09-12T20:12:09.073 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.074 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.074 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.075 DEBUG:boto:Params: {} 2022-09-12T20:12:09.075 DEBUG:boto:Token: None 2022-09-12T20:12:09.076 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:09.076 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SbHVPVmyIxIXwGS4GzRQRJduXwc= 2022-09-12T20:12:09.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SbHVPVmyIxIXwGS4GzRQRJduXwc=', 'Content-Length': '0'} 2022-09-12T20:12:09.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f06b76e47cee281-00631f9299-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.080 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:09.080 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:09.081 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:09.081 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:09.082 DEBUG:boto:Method: HEAD 2022-09-12T20:12:09.082 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:09.083 DEBUG:boto:Data: 2022-09-12T20:12:09.083 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.084 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.084 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.085 DEBUG:boto:Params: {} 2022-09-12T20:12:09.085 DEBUG:boto:Token: None 2022-09-12T20:12:09.086 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:09.086 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SbHVPVmyIxIXwGS4GzRQRJduXwc= 2022-09-12T20:12:09.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SbHVPVmyIxIXwGS4GzRQRJduXwc=', 'Content-Length': '0'} 2022-09-12T20:12:09.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad6f29468443c0f7-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:12:09.090 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:09.091 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:09.091 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:09.092 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:09.092 DEBUG:boto:Method: GET 2022-09-12T20:12:09.093 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:09.094 DEBUG:boto:Data: 2022-09-12T20:12:09.094 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.095 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.095 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.096 DEBUG:boto:Params: {} 2022-09-12T20:12:09.096 DEBUG:boto:Token: None 2022-09-12T20:12:09.097 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:09.097 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ssdZ8nmPUN9aq0qSy4FLmxr07n8= 2022-09-12T20:12:09.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ssdZ8nmPUN9aq0qSy4FLmxr07n8=', 'Content-Length': '0'} 2022-09-12T20:12:09.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056292e737aade6dc-00631f9299-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.142 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:09.143 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:09.143 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:09.144 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:09.144 DEBUG:boto:Method: GET 2022-09-12T20:12:09.145 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:09.145 DEBUG:boto:Data: 2022-09-12T20:12:09.146 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.146 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.147 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.147 DEBUG:boto:Params: {} 2022-09-12T20:12:09.148 DEBUG:boto:Token: None 2022-09-12T20:12:09.148 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:09.149 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ssdZ8nmPUN9aq0qSy4FLmxr07n8= 2022-09-12T20:12:09.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ssdZ8nmPUN9aq0qSy4FLmxr07n8=', 'Content-Length': '0'} 2022-09-12T20:12:09.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea303bf860a541af-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.153 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:09.153 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:09.154 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:12:09.154 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:12:09.155 DEBUG:boto:Method: GET 2022-09-12T20:12:09.155 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:12:09.156 DEBUG:boto:Data: 2022-09-12T20:12:09.156 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.157 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.157 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.158 DEBUG:boto:Params: {} 2022-09-12T20:12:09.158 DEBUG:boto:Token: None 2022-09-12T20:12:09.159 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/ 2022-09-12T20:12:09.159 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ= 2022-09-12T20:12:09.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ=', 'Content-Length': '0'} 2022-09-12T20:12:09.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073af334517e2dfd9-00631f9299-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.163 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:09.164 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:09.165 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:09.165 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:12:09.166 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:12:09.166 DEBUG:boto:Method: GET 2022-09-12T20:12:09.167 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:12:09.167 DEBUG:boto:Data: 2022-09-12T20:12:09.168 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.168 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.169 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.169 DEBUG:boto:Params: {} 2022-09-12T20:12:09.170 DEBUG:boto:Token: None 2022-09-12T20:12:09.170 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/ 2022-09-12T20:12:09.171 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ= 2022-09-12T20:12:09.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ=', 'Content-Length': '0'} 2022-09-12T20:12:09.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ac004869dd3d81e-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.177 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:09.178 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:09.178 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:09.179 DEBUG:boto:Method: HEAD 2022-09-12T20:12:09.179 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:12:09.180 DEBUG:boto:Data: 2022-09-12T20:12:09.180 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.181 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.181 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.182 DEBUG:boto:Params: {} 2022-09-12T20:12:09.182 DEBUG:boto:Token: None 2022-09-12T20:12:09.183 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj44 2022-09-12T20:12:09.183 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0gQH43C1kpiNBDjMI9g/bSEDPYE= 2022-09-12T20:12:09.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0gQH43C1kpiNBDjMI9g/bSEDPYE=', 'Content-Length': '0'} 2022-09-12T20:12:09.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa3a6ac5198867be-00631f9299-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.187 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:09.187 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:09.188 DEBUG:boto:Method: HEAD 2022-09-12T20:12:09.188 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:12:09.189 DEBUG:boto:Data: 2022-09-12T20:12:09.189 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.190 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.190 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.191 DEBUG:boto:Params: {} 2022-09-12T20:12:09.191 DEBUG:boto:Token: None 2022-09-12T20:12:09.192 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj44 2022-09-12T20:12:09.192 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0gQH43C1kpiNBDjMI9g/bSEDPYE= 2022-09-12T20:12:09.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0gQH43C1kpiNBDjMI9g/bSEDPYE=', 'Content-Length': '0'} 2022-09-12T20:12:09.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000161e17ce4ae1e587-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:12:09.196 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:09.197 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:09.197 DEBUG:boto:Method: GET 2022-09-12T20:12:09.198 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:12:09.198 DEBUG:boto:Data: 2022-09-12T20:12:09.199 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.199 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.200 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.200 DEBUG:boto:Params: {} 2022-09-12T20:12:09.201 DEBUG:boto:Token: None 2022-09-12T20:12:09.201 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj44 2022-09-12T20:12:09.202 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QmzrC1vpxOfnKz/Hlw8gkVMmUbU= 2022-09-12T20:12:09.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QmzrC1vpxOfnKz/Hlw8gkVMmUbU=', 'Content-Length': '0'} 2022-09-12T20:12:09.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fa5c806db48bf51-00631f9299-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.205 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:09.206 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:09.207 DEBUG:boto:Method: GET 2022-09-12T20:12:09.207 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:12:09.208 DEBUG:boto:Data: 2022-09-12T20:12:09.208 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.209 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.209 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.210 DEBUG:boto:Params: {} 2022-09-12T20:12:09.210 DEBUG:boto:Token: None 2022-09-12T20:12:09.211 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj44 2022-09-12T20:12:09.211 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QmzrC1vpxOfnKz/Hlw8gkVMmUbU= 2022-09-12T20:12:09.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QmzrC1vpxOfnKz/Hlw8gkVMmUbU=', 'Content-Length': '0'} 2022-09-12T20:12:09.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059a8618b9497f762-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.258 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:09.259 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:09.259 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:12:09.260 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:12:09.260 DEBUG:boto:Method: GET 2022-09-12T20:12:09.261 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:12:09.261 DEBUG:boto:Data: 2022-09-12T20:12:09.262 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.262 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.263 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.263 DEBUG:boto:Params: {} 2022-09-12T20:12:09.264 DEBUG:boto:Token: None 2022-09-12T20:12:09.265 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/ 2022-09-12T20:12:09.265 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ= 2022-09-12T20:12:09.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ=', 'Content-Length': '0'} 2022-09-12T20:12:09.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6e723ef46370c98-00631f9299-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.269 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:09.270 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:09.271 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:09.271 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:12:09.272 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:12:09.272 DEBUG:boto:Method: GET 2022-09-12T20:12:09.273 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:12:09.273 DEBUG:boto:Data: 2022-09-12T20:12:09.274 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.274 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.275 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.275 DEBUG:boto:Params: {} 2022-09-12T20:12:09.276 DEBUG:boto:Token: None 2022-09-12T20:12:09.276 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/ 2022-09-12T20:12:09.277 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ= 2022-09-12T20:12:09.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ=', 'Content-Length': '0'} 2022-09-12T20:12:09.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cecc5f745c0a86d-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.282 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:09.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:12:09.283 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:09.284 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:09.285 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:09.285 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:09.286 DEBUG:boto:Method: GET 2022-09-12T20:12:09.286 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:09.287 DEBUG:boto:Data: 2022-09-12T20:12:09.287 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.288 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.288 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.289 DEBUG:boto:Params: {} 2022-09-12T20:12:09.289 DEBUG:boto:Token: None 2022-09-12T20:12:09.290 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:09.290 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Qwa5sTUHwCqX5nHxpFBWhUfgcY0= 2022-09-12T20:12:09.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Qwa5sTUHwCqX5nHxpFBWhUfgcY0=', 'Content-Length': '0'} 2022-09-12T20:12:09.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003defc5291933fd53-00631f9299-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.294 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:09.295 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:09.295 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:09.296 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:09.296 DEBUG:boto:Method: GET 2022-09-12T20:12:09.297 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:09.297 DEBUG:boto:Data: 2022-09-12T20:12:09.298 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.298 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.299 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.299 DEBUG:boto:Params: {} 2022-09-12T20:12:09.300 DEBUG:boto:Token: None 2022-09-12T20:12:09.300 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:09.301 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Qwa5sTUHwCqX5nHxpFBWhUfgcY0= 2022-09-12T20:12:09.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Qwa5sTUHwCqX5nHxpFBWhUfgcY0=', 'Content-Length': '0'} 2022-09-12T20:12:09.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062a1ac043dc37291-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.305 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:09.305 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:09.306 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:09.306 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:09.307 DEBUG:boto:Method: HEAD 2022-09-12T20:12:09.307 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:09.308 DEBUG:boto:Data: 2022-09-12T20:12:09.308 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.309 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.309 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.310 DEBUG:boto:Params: {} 2022-09-12T20:12:09.310 DEBUG:boto:Token: None 2022-09-12T20:12:09.311 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:09.311 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:I7RKoaFs7A+eqciTXHN/jGUXW3Q= 2022-09-12T20:12:09.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:I7RKoaFs7A+eqciTXHN/jGUXW3Q=', 'Content-Length': '0'} 2022-09-12T20:12:09.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2d24f8e2f8463d7-00631f9299-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.315 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:09.315 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:09.316 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:09.316 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:09.317 DEBUG:boto:Method: HEAD 2022-09-12T20:12:09.317 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:09.318 DEBUG:boto:Data: 2022-09-12T20:12:09.318 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.319 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.319 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.320 DEBUG:boto:Params: {} 2022-09-12T20:12:09.320 DEBUG:boto:Token: None 2022-09-12T20:12:09.321 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:09.321 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:I7RKoaFs7A+eqciTXHN/jGUXW3Q= 2022-09-12T20:12:09.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:I7RKoaFs7A+eqciTXHN/jGUXW3Q=', 'Content-Length': '0'} 2022-09-12T20:12:09.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d29f17d84b4befe-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:12:09.325 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:09.326 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:09.326 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:09.327 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:09.327 DEBUG:boto:Method: GET 2022-09-12T20:12:09.328 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:09.328 DEBUG:boto:Data: 2022-09-12T20:12:09.329 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.329 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.330 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.331 DEBUG:boto:Params: {} 2022-09-12T20:12:09.331 DEBUG:boto:Token: None 2022-09-12T20:12:09.332 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:09.332 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Qwa5sTUHwCqX5nHxpFBWhUfgcY0= 2022-09-12T20:12:09.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Qwa5sTUHwCqX5nHxpFBWhUfgcY0=', 'Content-Length': '0'} 2022-09-12T20:12:09.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1c0b68a1f20b91e-00631f9299-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.378 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:09.379 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:09.379 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:09.380 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:09.380 DEBUG:boto:Method: GET 2022-09-12T20:12:09.381 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:09.381 DEBUG:boto:Data: 2022-09-12T20:12:09.382 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.382 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.383 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.383 DEBUG:boto:Params: {} 2022-09-12T20:12:09.384 DEBUG:boto:Token: None 2022-09-12T20:12:09.385 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:09.385 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Qwa5sTUHwCqX5nHxpFBWhUfgcY0= 2022-09-12T20:12:09.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Qwa5sTUHwCqX5nHxpFBWhUfgcY0=', 'Content-Length': '0'} 2022-09-12T20:12:09.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc93d5dec9301025-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.389 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:09.389 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:09.390 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:12:09.390 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:12:09.391 DEBUG:boto:Method: GET 2022-09-12T20:12:09.391 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:12:09.392 DEBUG:boto:Data: 2022-09-12T20:12:09.392 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.393 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.394 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.394 DEBUG:boto:Params: {} 2022-09-12T20:12:09.395 DEBUG:boto:Token: None 2022-09-12T20:12:09.395 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/ 2022-09-12T20:12:09.396 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ= 2022-09-12T20:12:09.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ=', 'Content-Length': '0'} 2022-09-12T20:12:09.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000823b41b9f7c57ee4-00631f9299-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.400 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:09.401 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:09.401 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:09.402 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:12:09.402 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:12:09.403 DEBUG:boto:Method: GET 2022-09-12T20:12:09.404 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:12:09.404 DEBUG:boto:Data: 2022-09-12T20:12:09.405 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.405 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.406 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.406 DEBUG:boto:Params: {} 2022-09-12T20:12:09.407 DEBUG:boto:Token: None 2022-09-12T20:12:09.407 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/ 2022-09-12T20:12:09.408 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ= 2022-09-12T20:12:09.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ=', 'Content-Length': '0'} 2022-09-12T20:12:09.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d34bfc6af6252127-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.413 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:09.414 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:09.414 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:09.415 DEBUG:boto:Method: HEAD 2022-09-12T20:12:09.415 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:12:09.416 DEBUG:boto:Data: 2022-09-12T20:12:09.416 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.417 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.417 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.418 DEBUG:boto:Params: {} 2022-09-12T20:12:09.418 DEBUG:boto:Token: None 2022-09-12T20:12:09.419 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj46 2022-09-12T20:12:09.420 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rMgJUjN0gxyQXYGJ6ziVaJaGVXs= 2022-09-12T20:12:09.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rMgJUjN0gxyQXYGJ6ziVaJaGVXs=', 'Content-Length': '0'} 2022-09-12T20:12:09.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048935ec5e41a0846-00631f9299-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.423 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:09.423 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:09.424 DEBUG:boto:Method: HEAD 2022-09-12T20:12:09.425 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:12:09.425 DEBUG:boto:Data: 2022-09-12T20:12:09.426 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.426 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.426 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.427 DEBUG:boto:Params: {} 2022-09-12T20:12:09.428 DEBUG:boto:Token: None 2022-09-12T20:12:09.428 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj46 2022-09-12T20:12:09.429 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rMgJUjN0gxyQXYGJ6ziVaJaGVXs= 2022-09-12T20:12:09.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rMgJUjN0gxyQXYGJ6ziVaJaGVXs=', 'Content-Length': '0'} 2022-09-12T20:12:09.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a59eefb434b3ae2-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:12:09.433 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:09.433 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:09.434 DEBUG:boto:Method: GET 2022-09-12T20:12:09.434 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:12:09.435 DEBUG:boto:Data: 2022-09-12T20:12:09.435 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.436 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.437 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.437 DEBUG:boto:Params: {} 2022-09-12T20:12:09.438 DEBUG:boto:Token: None 2022-09-12T20:12:09.438 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj46 2022-09-12T20:12:09.439 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rSYdV9pwmnChg0vyQROA+QiAkn0= 2022-09-12T20:12:09.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rSYdV9pwmnChg0vyQROA+QiAkn0=', 'Content-Length': '0'} 2022-09-12T20:12:09.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e65533ac2e2b7e3-00631f9299-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.482 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:09.483 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:09.483 DEBUG:boto:Method: GET 2022-09-12T20:12:09.484 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:12:09.484 DEBUG:boto:Data: 2022-09-12T20:12:09.485 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.485 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.486 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.486 DEBUG:boto:Params: {} 2022-09-12T20:12:09.487 DEBUG:boto:Token: None 2022-09-12T20:12:09.487 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj46 2022-09-12T20:12:09.488 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rSYdV9pwmnChg0vyQROA+QiAkn0= 2022-09-12T20:12:09.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rSYdV9pwmnChg0vyQROA+QiAkn0=', 'Content-Length': '0'} 2022-09-12T20:12:09.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab7921830d2c75f1-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.492 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:09.492 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:09.493 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:12:09.493 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:12:09.494 DEBUG:boto:Method: GET 2022-09-12T20:12:09.494 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:12:09.495 DEBUG:boto:Data: 2022-09-12T20:12:09.495 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.496 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.497 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.497 DEBUG:boto:Params: {} 2022-09-12T20:12:09.498 DEBUG:boto:Token: None 2022-09-12T20:12:09.498 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/ 2022-09-12T20:12:09.499 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ= 2022-09-12T20:12:09.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ=', 'Content-Length': '0'} 2022-09-12T20:12:09.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000105864e7752ff3d5-00631f9299-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.503 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:09.504 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:09.504 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:09.505 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:12:09.505 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:12:09.506 DEBUG:boto:Method: GET 2022-09-12T20:12:09.506 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:12:09.507 DEBUG:boto:Data: 2022-09-12T20:12:09.507 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.508 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.508 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.509 DEBUG:boto:Params: {} 2022-09-12T20:12:09.509 DEBUG:boto:Token: None 2022-09-12T20:12:09.510 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/ 2022-09-12T20:12:09.511 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ= 2022-09-12T20:12:09.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ=', 'Content-Length': '0'} 2022-09-12T20:12:09.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df425575faeb8b9a-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.558 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:09.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:12:09.560 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:09.560 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:09.561 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:09.561 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:09.562 DEBUG:boto:Method: GET 2022-09-12T20:12:09.562 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:09.563 DEBUG:boto:Data: 2022-09-12T20:12:09.563 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.564 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.564 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.565 DEBUG:boto:Params: {} 2022-09-12T20:12:09.565 DEBUG:boto:Token: None 2022-09-12T20:12:09.566 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:09.567 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YYbMqDBLb56ePT8H7RBhNgtP4pk= 2022-09-12T20:12:09.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YYbMqDBLb56ePT8H7RBhNgtP4pk=', 'Content-Length': '0'} 2022-09-12T20:12:09.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b4ae1e9d541c1f0-00631f9299-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.570 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:09.571 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:09.571 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:09.572 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:09.572 DEBUG:boto:Method: GET 2022-09-12T20:12:09.573 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:09.573 DEBUG:boto:Data: 2022-09-12T20:12:09.574 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.575 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.575 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.576 DEBUG:boto:Params: {} 2022-09-12T20:12:09.576 DEBUG:boto:Token: None 2022-09-12T20:12:09.577 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:09.577 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YYbMqDBLb56ePT8H7RBhNgtP4pk= 2022-09-12T20:12:09.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YYbMqDBLb56ePT8H7RBhNgtP4pk=', 'Content-Length': '0'} 2022-09-12T20:12:09.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084017626e7e42d92-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.581 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:09.582 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:09.582 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:09.583 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:09.583 DEBUG:boto:Method: HEAD 2022-09-12T20:12:09.584 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:09.584 DEBUG:boto:Data: 2022-09-12T20:12:09.585 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.585 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.586 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.586 DEBUG:boto:Params: {} 2022-09-12T20:12:09.587 DEBUG:boto:Token: None 2022-09-12T20:12:09.587 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:09.588 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VayGiTpwAVtr3LxsY0dhgU0iE4Q= 2022-09-12T20:12:09.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VayGiTpwAVtr3LxsY0dhgU0iE4Q=', 'Content-Length': '0'} 2022-09-12T20:12:09.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d21bb97eebfc3f0-00631f9299-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.591 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:09.592 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:09.592 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:09.593 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:09.593 DEBUG:boto:Method: HEAD 2022-09-12T20:12:09.594 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:09.594 DEBUG:boto:Data: 2022-09-12T20:12:09.595 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.595 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.596 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.596 DEBUG:boto:Params: {} 2022-09-12T20:12:09.597 DEBUG:boto:Token: None 2022-09-12T20:12:09.597 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:09.598 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VayGiTpwAVtr3LxsY0dhgU0iE4Q= 2022-09-12T20:12:09.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VayGiTpwAVtr3LxsY0dhgU0iE4Q=', 'Content-Length': '0'} 2022-09-12T20:12:09.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000933e60d4fb975f25-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:12:09.601 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:09.602 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:09.603 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:09.603 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:09.604 DEBUG:boto:Method: GET 2022-09-12T20:12:09.604 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:09.605 DEBUG:boto:Data: 2022-09-12T20:12:09.605 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.606 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.606 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.607 DEBUG:boto:Params: {} 2022-09-12T20:12:09.607 DEBUG:boto:Token: None 2022-09-12T20:12:09.608 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:09.609 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YYbMqDBLb56ePT8H7RBhNgtP4pk= 2022-09-12T20:12:09.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YYbMqDBLb56ePT8H7RBhNgtP4pk=', 'Content-Length': '0'} 2022-09-12T20:12:09.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b5819065c47e9da-00631f9299-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.612 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:09.612 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:09.613 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:09.613 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:09.614 DEBUG:boto:Method: GET 2022-09-12T20:12:09.614 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:09.615 DEBUG:boto:Data: 2022-09-12T20:12:09.615 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.616 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.616 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.617 DEBUG:boto:Params: {} 2022-09-12T20:12:09.618 DEBUG:boto:Token: None 2022-09-12T20:12:09.618 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:09.619 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YYbMqDBLb56ePT8H7RBhNgtP4pk= 2022-09-12T20:12:09.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YYbMqDBLb56ePT8H7RBhNgtP4pk=', 'Content-Length': '0'} 2022-09-12T20:12:09.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b7eb6f76a43408c-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.622 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:09.623 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:09.624 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:12:09.624 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:12:09.625 DEBUG:boto:Method: GET 2022-09-12T20:12:09.625 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:12:09.626 DEBUG:boto:Data: 2022-09-12T20:12:09.626 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.627 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.627 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.628 DEBUG:boto:Params: {} 2022-09-12T20:12:09.628 DEBUG:boto:Token: None 2022-09-12T20:12:09.629 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/ 2022-09-12T20:12:09.630 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ= 2022-09-12T20:12:09.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ=', 'Content-Length': '0'} 2022-09-12T20:12:09.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c025cd82e32b2ff-00631f9299-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.634 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:09.635 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:09.635 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:09.636 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:12:09.636 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:12:09.637 DEBUG:boto:Method: GET 2022-09-12T20:12:09.637 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:12:09.638 DEBUG:boto:Data: 2022-09-12T20:12:09.638 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.639 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.639 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.640 DEBUG:boto:Params: {} 2022-09-12T20:12:09.640 DEBUG:boto:Token: None 2022-09-12T20:12:09.641 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/ 2022-09-12T20:12:09.641 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ= 2022-09-12T20:12:09.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ=', 'Content-Length': '0'} 2022-09-12T20:12:09.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da2236dd0ffa631e-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.647 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:09.648 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:09.648 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:09.649 DEBUG:boto:Method: HEAD 2022-09-12T20:12:09.650 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:12:09.650 DEBUG:boto:Data: 2022-09-12T20:12:09.651 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.651 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.652 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.652 DEBUG:boto:Params: {} 2022-09-12T20:12:09.653 DEBUG:boto:Token: None 2022-09-12T20:12:09.653 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj51 2022-09-12T20:12:09.654 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Jfk4qHv8aMlOITKWrFG+vtg+iPA= 2022-09-12T20:12:09.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Jfk4qHv8aMlOITKWrFG+vtg+iPA=', 'Content-Length': '0'} 2022-09-12T20:12:09.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dd539383bead415-00631f9299-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.657 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:09.657 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:09.658 DEBUG:boto:Method: HEAD 2022-09-12T20:12:09.658 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:12:09.659 DEBUG:boto:Data: 2022-09-12T20:12:09.659 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.660 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.660 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.661 DEBUG:boto:Params: {} 2022-09-12T20:12:09.661 DEBUG:boto:Token: None 2022-09-12T20:12:09.662 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj51 2022-09-12T20:12:09.663 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Jfk4qHv8aMlOITKWrFG+vtg+iPA= 2022-09-12T20:12:09.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Jfk4qHv8aMlOITKWrFG+vtg+iPA=', 'Content-Length': '0'} 2022-09-12T20:12:09.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cd7a75730f88507-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:12:09.667 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:09.667 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:09.668 DEBUG:boto:Method: GET 2022-09-12T20:12:09.668 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:12:09.669 DEBUG:boto:Data: 2022-09-12T20:12:09.669 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.670 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.670 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.671 DEBUG:boto:Params: {} 2022-09-12T20:12:09.671 DEBUG:boto:Token: None 2022-09-12T20:12:09.672 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj51 2022-09-12T20:12:09.673 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AIgZ4EXXtkH6I8HKUco5PkF0nNA= 2022-09-12T20:12:09.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AIgZ4EXXtkH6I8HKUco5PkF0nNA=', 'Content-Length': '0'} 2022-09-12T20:12:09.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b796840917181f9-00631f9299-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.676 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:09.677 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:09.677 DEBUG:boto:Method: GET 2022-09-12T20:12:09.678 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:12:09.678 DEBUG:boto:Data: 2022-09-12T20:12:09.679 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.679 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.680 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.680 DEBUG:boto:Params: {} 2022-09-12T20:12:09.681 DEBUG:boto:Token: None 2022-09-12T20:12:09.681 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj51 2022-09-12T20:12:09.682 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AIgZ4EXXtkH6I8HKUco5PkF0nNA= 2022-09-12T20:12:09.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AIgZ4EXXtkH6I8HKUco5PkF0nNA=', 'Content-Length': '0'} 2022-09-12T20:12:09.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096927392777bc5d6-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.686 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:09.686 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:09.687 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:12:09.687 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:12:09.688 DEBUG:boto:Method: GET 2022-09-12T20:12:09.688 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:12:09.689 DEBUG:boto:Data: 2022-09-12T20:12:09.689 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.690 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.691 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.691 DEBUG:boto:Params: {} 2022-09-12T20:12:09.692 DEBUG:boto:Token: None 2022-09-12T20:12:09.692 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/ 2022-09-12T20:12:09.693 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ= 2022-09-12T20:12:09.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ=', 'Content-Length': '0'} 2022-09-12T20:12:09.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc5c8c714ea2824d-00631f9299-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.697 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:09.698 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:09.698 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:09.699 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:12:09.699 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:12:09.700 DEBUG:boto:Method: GET 2022-09-12T20:12:09.700 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:12:09.701 DEBUG:boto:Data: 2022-09-12T20:12:09.701 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.702 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.702 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.703 DEBUG:boto:Params: {} 2022-09-12T20:12:09.703 DEBUG:boto:Token: None 2022-09-12T20:12:09.704 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/ 2022-09-12T20:12:09.704 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ= 2022-09-12T20:12:09.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ=', 'Content-Length': '0'} 2022-09-12T20:12:09.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000764d89bf5fe89172-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.710 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:09.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:12:09.711 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:09.712 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:09.712 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:09.713 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:09.713 DEBUG:boto:Method: GET 2022-09-12T20:12:09.714 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:09.715 DEBUG:boto:Data: 2022-09-12T20:12:09.715 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.716 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.716 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.717 DEBUG:boto:Params: {} 2022-09-12T20:12:09.717 DEBUG:boto:Token: None 2022-09-12T20:12:09.718 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:09.718 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:noA9A7lQdYw9SOSjVvGRQZkjOjQ= 2022-09-12T20:12:09.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:noA9A7lQdYw9SOSjVvGRQZkjOjQ=', 'Content-Length': '0'} 2022-09-12T20:12:09.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cb4790d661da7f7-00631f9299-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.722 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:09.723 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:09.723 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:09.724 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:09.724 DEBUG:boto:Method: GET 2022-09-12T20:12:09.725 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:09.725 DEBUG:boto:Data: 2022-09-12T20:12:09.726 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.726 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.727 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.727 DEBUG:boto:Params: {} 2022-09-12T20:12:09.728 DEBUG:boto:Token: None 2022-09-12T20:12:09.729 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:09.729 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:noA9A7lQdYw9SOSjVvGRQZkjOjQ= 2022-09-12T20:12:09.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:noA9A7lQdYw9SOSjVvGRQZkjOjQ=', 'Content-Length': '0'} 2022-09-12T20:12:09.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af4712eb893cbb8d-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.733 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:09.734 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:09.734 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:09.735 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:09.735 DEBUG:boto:Method: HEAD 2022-09-12T20:12:09.736 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:09.736 DEBUG:boto:Data: 2022-09-12T20:12:09.737 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.737 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.738 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.738 DEBUG:boto:Params: {} 2022-09-12T20:12:09.739 DEBUG:boto:Token: None 2022-09-12T20:12:09.739 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:09.740 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lMYwxCsqHpczGUOHUgmw2AhDkg0= 2022-09-12T20:12:09.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lMYwxCsqHpczGUOHUgmw2AhDkg0=', 'Content-Length': '0'} 2022-09-12T20:12:09.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f9f06fba48b6b57-00631f9299-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.743 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:09.744 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:09.744 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:09.745 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:09.745 DEBUG:boto:Method: HEAD 2022-09-12T20:12:09.746 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:09.747 DEBUG:boto:Data: 2022-09-12T20:12:09.747 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.748 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.748 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.749 DEBUG:boto:Params: {} 2022-09-12T20:12:09.749 DEBUG:boto:Token: None 2022-09-12T20:12:09.750 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:09.751 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lMYwxCsqHpczGUOHUgmw2AhDkg0= 2022-09-12T20:12:09.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lMYwxCsqHpczGUOHUgmw2AhDkg0=', 'Content-Length': '0'} 2022-09-12T20:12:09.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002cdcf08206f34d6-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:12:09.755 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:09.755 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:09.756 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:09.756 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:09.757 DEBUG:boto:Method: GET 2022-09-12T20:12:09.757 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:09.758 DEBUG:boto:Data: 2022-09-12T20:12:09.759 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.759 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.760 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.761 DEBUG:boto:Params: {} 2022-09-12T20:12:09.761 DEBUG:boto:Token: None 2022-09-12T20:12:09.762 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:09.763 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:noA9A7lQdYw9SOSjVvGRQZkjOjQ= 2022-09-12T20:12:09.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:noA9A7lQdYw9SOSjVvGRQZkjOjQ=', 'Content-Length': '0'} 2022-09-12T20:12:09.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7ddd34929418021-00631f9299-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.810 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:09.811 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:09.812 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:09.813 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:09.814 DEBUG:boto:Method: GET 2022-09-12T20:12:09.815 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:09.816 DEBUG:boto:Data: 2022-09-12T20:12:09.817 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.819 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.820 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.821 DEBUG:boto:Params: {} 2022-09-12T20:12:09.821 DEBUG:boto:Token: None 2022-09-12T20:12:09.822 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:09.823 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:noA9A7lQdYw9SOSjVvGRQZkjOjQ= 2022-09-12T20:12:09.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:noA9A7lQdYw9SOSjVvGRQZkjOjQ=', 'Content-Length': '0'} 2022-09-12T20:12:09.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003552755a1be3b401-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.827 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:09.828 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:09.828 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:12:09.829 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:12:09.830 DEBUG:boto:Method: GET 2022-09-12T20:12:09.830 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:12:09.831 DEBUG:boto:Data: 2022-09-12T20:12:09.832 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.832 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.833 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.834 DEBUG:boto:Params: {} 2022-09-12T20:12:09.835 DEBUG:boto:Token: None 2022-09-12T20:12:09.836 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/ 2022-09-12T20:12:09.837 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ= 2022-09-12T20:12:09.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ=', 'Content-Length': '0'} 2022-09-12T20:12:09.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f3cbe6fbd02ce7a-00631f9299-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.841 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:09.842 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:09.843 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:09.844 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:12:09.844 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:12:09.845 DEBUG:boto:Method: GET 2022-09-12T20:12:09.845 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:12:09.846 DEBUG:boto:Data: 2022-09-12T20:12:09.847 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.848 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.848 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.849 DEBUG:boto:Params: {} 2022-09-12T20:12:09.850 DEBUG:boto:Token: None 2022-09-12T20:12:09.851 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/ 2022-09-12T20:12:09.851 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ= 2022-09-12T20:12:09.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ=', 'Content-Length': '0'} 2022-09-12T20:12:09.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ff7ad651b548949-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.898 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:09.899 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:09.900 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:09.901 DEBUG:boto:Method: HEAD 2022-09-12T20:12:09.901 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:12:09.902 DEBUG:boto:Data: 2022-09-12T20:12:09.903 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.904 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.905 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.905 DEBUG:boto:Params: {} 2022-09-12T20:12:09.906 DEBUG:boto:Token: None 2022-09-12T20:12:09.907 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj55 2022-09-12T20:12:09.908 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1iKxGND7+mXIbliYTB0X4IRIPBI= 2022-09-12T20:12:09.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1iKxGND7+mXIbliYTB0X4IRIPBI=', 'Content-Length': '0'} 2022-09-12T20:12:09.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000292ede5ae35584d8-00631f9299-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.913 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:09.913 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:09.914 DEBUG:boto:Method: HEAD 2022-09-12T20:12:09.915 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:12:09.916 DEBUG:boto:Data: 2022-09-12T20:12:09.917 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.918 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.919 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.920 DEBUG:boto:Params: {} 2022-09-12T20:12:09.921 DEBUG:boto:Token: None 2022-09-12T20:12:09.922 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj55 2022-09-12T20:12:09.922 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1iKxGND7+mXIbliYTB0X4IRIPBI= 2022-09-12T20:12:09.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1iKxGND7+mXIbliYTB0X4IRIPBI=', 'Content-Length': '0'} 2022-09-12T20:12:09.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5de82ac4f32810b-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:12:09.928 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:09.928 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:09.929 DEBUG:boto:Method: GET 2022-09-12T20:12:09.929 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:12:09.930 DEBUG:boto:Data: 2022-09-12T20:12:09.930 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.931 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.931 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.932 DEBUG:boto:Params: {} 2022-09-12T20:12:09.933 DEBUG:boto:Token: None 2022-09-12T20:12:09.933 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj55 2022-09-12T20:12:09.934 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s0PVby+Qd72O8Jd6LpRaWR2uYQ8= 2022-09-12T20:12:09.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s0PVby+Qd72O8Jd6LpRaWR2uYQ8=', 'Content-Length': '0'} 2022-09-12T20:12:09.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d34fddb1f38b94b1-00631f9299-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.937 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:09.938 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:09.939 DEBUG:boto:Method: GET 2022-09-12T20:12:09.939 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:12:09.940 DEBUG:boto:Data: 2022-09-12T20:12:09.940 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.941 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.941 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.942 DEBUG:boto:Params: {} 2022-09-12T20:12:09.942 DEBUG:boto:Token: None 2022-09-12T20:12:09.943 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/obj55 2022-09-12T20:12:09.943 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s0PVby+Qd72O8Jd6LpRaWR2uYQ8= 2022-09-12T20:12:09.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s0PVby+Qd72O8Jd6LpRaWR2uYQ8=', 'Content-Length': '0'} 2022-09-12T20:12:09.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009da27bee36654bad-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.947 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:09.948 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:09.948 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:12:09.949 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:12:09.949 DEBUG:boto:Method: GET 2022-09-12T20:12:09.950 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:12:09.950 DEBUG:boto:Data: 2022-09-12T20:12:09.951 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.951 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:09.952 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.952 DEBUG:boto:Params: {} 2022-09-12T20:12:09.953 DEBUG:boto:Token: None 2022-09-12T20:12:09.954 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/ 2022-09-12T20:12:09.954 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ= 2022-09-12T20:12:09.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ=', 'Content-Length': '0'} 2022-09-12T20:12:09.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f0d7cc5a0c02735-00631f9299-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:09.958 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:09.959 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:09.960 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:09.960 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:12:09.961 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:12:09.961 DEBUG:boto:Method: GET 2022-09-12T20:12:09.962 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:12:09.962 DEBUG:boto:Data: 2022-09-12T20:12:09.963 DEBUG:boto:Headers: {} 2022-09-12T20:12:09.963 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:09.964 DEBUG:boto:Port: 8000 2022-09-12T20:12:09.964 DEBUG:boto:Params: {} 2022-09-12T20:12:09.965 DEBUG:boto:Token: None 2022-09-12T20:12:09.965 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:09 GMT /jdpiak-75/ 2022-09-12T20:12:09.966 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ= 2022-09-12T20:12:09.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t5KItXrDMyvfhKuE6eLsLbM75OQ=', 'Content-Length': '0'} 2022-09-12T20:12:09.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fe70dccf4da46a1-00631f9299-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.014 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:10.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:12:10.015 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:10.016 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:10.016 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:10.017 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:10.018 DEBUG:boto:Method: GET 2022-09-12T20:12:10.018 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:10.019 DEBUG:boto:Data: 2022-09-12T20:12:10.019 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.020 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.020 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.021 DEBUG:boto:Params: {} 2022-09-12T20:12:10.021 DEBUG:boto:Token: None 2022-09-12T20:12:10.022 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:10.022 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HBExvWq6h9zcz0bUlwkMuAsEeVU= 2022-09-12T20:12:10.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HBExvWq6h9zcz0bUlwkMuAsEeVU=', 'Content-Length': '0'} 2022-09-12T20:12:10.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6bc9d5bb70f0dc2-00631f929a-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.026 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:10.027 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:10.027 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:10.028 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:10.028 DEBUG:boto:Method: GET 2022-09-12T20:12:10.029 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:10.029 DEBUG:boto:Data: 2022-09-12T20:12:10.030 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.030 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.031 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.031 DEBUG:boto:Params: {} 2022-09-12T20:12:10.032 DEBUG:boto:Token: None 2022-09-12T20:12:10.032 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:10.033 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HBExvWq6h9zcz0bUlwkMuAsEeVU= 2022-09-12T20:12:10.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HBExvWq6h9zcz0bUlwkMuAsEeVU=', 'Content-Length': '0'} 2022-09-12T20:12:10.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092a3ed5845d9470a-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.037 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:10.037 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:10.038 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:10.038 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:10.039 DEBUG:boto:Method: HEAD 2022-09-12T20:12:10.039 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:10.040 DEBUG:boto:Data: 2022-09-12T20:12:10.040 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.041 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.041 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.042 DEBUG:boto:Params: {} 2022-09-12T20:12:10.042 DEBUG:boto:Token: None 2022-09-12T20:12:10.043 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:10.043 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IyEcOZuEjpuntyg0N7D2N1d7C2Y= 2022-09-12T20:12:10.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IyEcOZuEjpuntyg0N7D2N1d7C2Y=', 'Content-Length': '0'} 2022-09-12T20:12:10.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041e9a0f39ae66b17-00631f929a-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.046 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:10.047 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:10.047 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:10.048 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:10.048 DEBUG:boto:Method: HEAD 2022-09-12T20:12:10.049 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:10.049 DEBUG:boto:Data: 2022-09-12T20:12:10.050 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.050 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.051 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.051 DEBUG:boto:Params: {} 2022-09-12T20:12:10.052 DEBUG:boto:Token: None 2022-09-12T20:12:10.052 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:10.053 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IyEcOZuEjpuntyg0N7D2N1d7C2Y= 2022-09-12T20:12:10.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IyEcOZuEjpuntyg0N7D2N1d7C2Y=', 'Content-Length': '0'} 2022-09-12T20:12:10.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000345e908c874e6720-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:12:10.057 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:10.058 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:10.058 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:10.059 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:10.059 DEBUG:boto:Method: GET 2022-09-12T20:12:10.060 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:10.060 DEBUG:boto:Data: 2022-09-12T20:12:10.061 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.062 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.062 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.063 DEBUG:boto:Params: {} 2022-09-12T20:12:10.063 DEBUG:boto:Token: None 2022-09-12T20:12:10.064 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:10.064 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HBExvWq6h9zcz0bUlwkMuAsEeVU= 2022-09-12T20:12:10.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HBExvWq6h9zcz0bUlwkMuAsEeVU=', 'Content-Length': '0'} 2022-09-12T20:12:10.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e871418ca54b98ac-00631f929a-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.068 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:10.068 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:10.069 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:10.069 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:10.070 DEBUG:boto:Method: GET 2022-09-12T20:12:10.070 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:10.071 DEBUG:boto:Data: 2022-09-12T20:12:10.071 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.072 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.072 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.073 DEBUG:boto:Params: {} 2022-09-12T20:12:10.073 DEBUG:boto:Token: None 2022-09-12T20:12:10.074 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:10.075 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HBExvWq6h9zcz0bUlwkMuAsEeVU= 2022-09-12T20:12:10.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HBExvWq6h9zcz0bUlwkMuAsEeVU=', 'Content-Length': '0'} 2022-09-12T20:12:10.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c75bbe1f12d22ef0-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.078 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:10.079 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:10.079 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:12:10.080 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:12:10.080 DEBUG:boto:Method: GET 2022-09-12T20:12:10.081 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:12:10.081 DEBUG:boto:Data: 2022-09-12T20:12:10.082 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.082 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.083 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.083 DEBUG:boto:Params: {} 2022-09-12T20:12:10.084 DEBUG:boto:Token: None 2022-09-12T20:12:10.084 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/ 2022-09-12T20:12:10.085 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA= 2022-09-12T20:12:10.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA=', 'Content-Length': '0'} 2022-09-12T20:12:10.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4b5ff393e38a79d-00631f929a-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.089 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:10.090 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:10.090 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:10.091 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:12:10.091 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:12:10.092 DEBUG:boto:Method: GET 2022-09-12T20:12:10.092 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:12:10.093 DEBUG:boto:Data: 2022-09-12T20:12:10.093 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.094 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.094 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.095 DEBUG:boto:Params: {} 2022-09-12T20:12:10.095 DEBUG:boto:Token: None 2022-09-12T20:12:10.096 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/ 2022-09-12T20:12:10.096 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA= 2022-09-12T20:12:10.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA=', 'Content-Length': '0'} 2022-09-12T20:12:10.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097f6c9f1b6af9d5f-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.102 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:10.103 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:10.103 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:10.104 DEBUG:boto:Method: HEAD 2022-09-12T20:12:10.104 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:12:10.105 DEBUG:boto:Data: 2022-09-12T20:12:10.105 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.106 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.106 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.107 DEBUG:boto:Params: {} 2022-09-12T20:12:10.107 DEBUG:boto:Token: None 2022-09-12T20:12:10.108 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj57 2022-09-12T20:12:10.108 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4og4/i3fRdl3mNFjJvFHkNxrrpQ= 2022-09-12T20:12:10.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4og4/i3fRdl3mNFjJvFHkNxrrpQ=', 'Content-Length': '0'} 2022-09-12T20:12:10.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000537d838d5c9acba3-00631f929a-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.112 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:10.112 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:10.113 DEBUG:boto:Method: HEAD 2022-09-12T20:12:10.113 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:12:10.114 DEBUG:boto:Data: 2022-09-12T20:12:10.114 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.115 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.115 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.116 DEBUG:boto:Params: {} 2022-09-12T20:12:10.116 DEBUG:boto:Token: None 2022-09-12T20:12:10.117 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj57 2022-09-12T20:12:10.117 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4og4/i3fRdl3mNFjJvFHkNxrrpQ= 2022-09-12T20:12:10.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4og4/i3fRdl3mNFjJvFHkNxrrpQ=', 'Content-Length': '0'} 2022-09-12T20:12:10.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002890c71dcfbe53ee-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:12:10.121 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:10.121 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:10.122 DEBUG:boto:Method: GET 2022-09-12T20:12:10.122 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:12:10.123 DEBUG:boto:Data: 2022-09-12T20:12:10.123 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.124 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.124 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.125 DEBUG:boto:Params: {} 2022-09-12T20:12:10.126 DEBUG:boto:Token: None 2022-09-12T20:12:10.126 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj57 2022-09-12T20:12:10.127 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uhz4J3XkR0NHmGc9gjcdT1GEi88= 2022-09-12T20:12:10.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uhz4J3XkR0NHmGc9gjcdT1GEi88=', 'Content-Length': '0'} 2022-09-12T20:12:10.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050b3aaebc4678598-00631f929a-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.170 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:10.171 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:10.171 DEBUG:boto:Method: GET 2022-09-12T20:12:10.172 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:12:10.173 DEBUG:boto:Data: 2022-09-12T20:12:10.173 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.174 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.174 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.175 DEBUG:boto:Params: {} 2022-09-12T20:12:10.175 DEBUG:boto:Token: None 2022-09-12T20:12:10.176 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj57 2022-09-12T20:12:10.176 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uhz4J3XkR0NHmGc9gjcdT1GEi88= 2022-09-12T20:12:10.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uhz4J3XkR0NHmGc9gjcdT1GEi88=', 'Content-Length': '0'} 2022-09-12T20:12:10.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091795d6d27e62944-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.180 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:10.181 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:10.181 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:12:10.182 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:12:10.182 DEBUG:boto:Method: GET 2022-09-12T20:12:10.183 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:12:10.183 DEBUG:boto:Data: 2022-09-12T20:12:10.184 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.184 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.185 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.186 DEBUG:boto:Params: {} 2022-09-12T20:12:10.186 DEBUG:boto:Token: None 2022-09-12T20:12:10.187 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/ 2022-09-12T20:12:10.187 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA= 2022-09-12T20:12:10.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA=', 'Content-Length': '0'} 2022-09-12T20:12:10.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c481f316ccf8220e-00631f929a-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.191 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:10.192 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:10.192 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:10.193 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:12:10.193 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:12:10.194 DEBUG:boto:Method: GET 2022-09-12T20:12:10.195 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:12:10.195 DEBUG:boto:Data: 2022-09-12T20:12:10.196 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.196 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.197 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.197 DEBUG:boto:Params: {} 2022-09-12T20:12:10.198 DEBUG:boto:Token: None 2022-09-12T20:12:10.198 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/ 2022-09-12T20:12:10.199 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA= 2022-09-12T20:12:10.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA=', 'Content-Length': '0'} 2022-09-12T20:12:10.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005524870d4dac5055-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.204 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:10.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:12:10.206 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:10.206 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:10.207 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:10.207 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:10.208 DEBUG:boto:Method: GET 2022-09-12T20:12:10.208 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:10.209 DEBUG:boto:Data: 2022-09-12T20:12:10.209 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.210 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.210 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.211 DEBUG:boto:Params: {} 2022-09-12T20:12:10.212 DEBUG:boto:Token: None 2022-09-12T20:12:10.212 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:10.213 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YUEeDFje9dxqCkAnGTCwmCp8xw0= 2022-09-12T20:12:10.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YUEeDFje9dxqCkAnGTCwmCp8xw0=', 'Content-Length': '0'} 2022-09-12T20:12:10.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009b97861b836fc4e-00631f929a-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.216 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:10.217 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:10.217 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:10.218 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:10.219 DEBUG:boto:Method: GET 2022-09-12T20:12:10.219 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:10.220 DEBUG:boto:Data: 2022-09-12T20:12:10.220 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.221 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.221 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.222 DEBUG:boto:Params: {} 2022-09-12T20:12:10.222 DEBUG:boto:Token: None 2022-09-12T20:12:10.223 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:10.224 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YUEeDFje9dxqCkAnGTCwmCp8xw0= 2022-09-12T20:12:10.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YUEeDFje9dxqCkAnGTCwmCp8xw0=', 'Content-Length': '0'} 2022-09-12T20:12:10.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb03ba68d77b608d-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.227 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:10.228 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:10.228 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:10.229 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:10.229 DEBUG:boto:Method: HEAD 2022-09-12T20:12:10.230 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:10.230 DEBUG:boto:Data: 2022-09-12T20:12:10.231 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.232 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.232 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.233 DEBUG:boto:Params: {} 2022-09-12T20:12:10.233 DEBUG:boto:Token: None 2022-09-12T20:12:10.234 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:10.235 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GdoC9SGuvm4KibQF4jYXtmwubvk= 2022-09-12T20:12:10.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GdoC9SGuvm4KibQF4jYXtmwubvk=', 'Content-Length': '0'} 2022-09-12T20:12:10.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2b765f38231d764-00631f929a-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.238 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:10.239 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:10.239 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:10.240 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:10.240 DEBUG:boto:Method: HEAD 2022-09-12T20:12:10.241 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:10.241 DEBUG:boto:Data: 2022-09-12T20:12:10.242 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.242 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.243 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.243 DEBUG:boto:Params: {} 2022-09-12T20:12:10.244 DEBUG:boto:Token: None 2022-09-12T20:12:10.244 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:10.245 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GdoC9SGuvm4KibQF4jYXtmwubvk= 2022-09-12T20:12:10.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GdoC9SGuvm4KibQF4jYXtmwubvk=', 'Content-Length': '0'} 2022-09-12T20:12:10.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016a082b12d46f4ab-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:12:10.249 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:10.250 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:10.250 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:10.251 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:10.251 DEBUG:boto:Method: GET 2022-09-12T20:12:10.252 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:10.252 DEBUG:boto:Data: 2022-09-12T20:12:10.253 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.253 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.254 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.254 DEBUG:boto:Params: {} 2022-09-12T20:12:10.255 DEBUG:boto:Token: None 2022-09-12T20:12:10.255 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:10.256 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YUEeDFje9dxqCkAnGTCwmCp8xw0= 2022-09-12T20:12:10.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YUEeDFje9dxqCkAnGTCwmCp8xw0=', 'Content-Length': '0'} 2022-09-12T20:12:10.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b068b49a92c855b-00631f929a-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.260 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:10.260 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:10.261 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:10.261 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:10.262 DEBUG:boto:Method: GET 2022-09-12T20:12:10.262 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:10.263 DEBUG:boto:Data: 2022-09-12T20:12:10.263 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.264 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.264 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.265 DEBUG:boto:Params: {} 2022-09-12T20:12:10.265 DEBUG:boto:Token: None 2022-09-12T20:12:10.266 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:10.267 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YUEeDFje9dxqCkAnGTCwmCp8xw0= 2022-09-12T20:12:10.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YUEeDFje9dxqCkAnGTCwmCp8xw0=', 'Content-Length': '0'} 2022-09-12T20:12:10.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbcb3c8ec8bf7c27-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.310 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:10.311 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:10.312 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:12:10.313 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:12:10.313 DEBUG:boto:Method: GET 2022-09-12T20:12:10.314 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:12:10.314 DEBUG:boto:Data: 2022-09-12T20:12:10.315 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.315 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.316 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.316 DEBUG:boto:Params: {} 2022-09-12T20:12:10.317 DEBUG:boto:Token: None 2022-09-12T20:12:10.318 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/ 2022-09-12T20:12:10.318 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA= 2022-09-12T20:12:10.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA=', 'Content-Length': '0'} 2022-09-12T20:12:10.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038ec7dbe78b59240-00631f929a-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.322 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:10.323 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:10.324 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:10.324 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:12:10.325 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:12:10.325 DEBUG:boto:Method: GET 2022-09-12T20:12:10.326 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:12:10.327 DEBUG:boto:Data: 2022-09-12T20:12:10.327 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.328 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.328 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.329 DEBUG:boto:Params: {} 2022-09-12T20:12:10.329 DEBUG:boto:Token: None 2022-09-12T20:12:10.330 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/ 2022-09-12T20:12:10.330 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA= 2022-09-12T20:12:10.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA=', 'Content-Length': '0'} 2022-09-12T20:12:10.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5ecc1dc9da79d37-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.336 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:10.337 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:10.337 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:10.338 DEBUG:boto:Method: HEAD 2022-09-12T20:12:10.338 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:12:10.339 DEBUG:boto:Data: 2022-09-12T20:12:10.340 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.340 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.340 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.341 DEBUG:boto:Params: {} 2022-09-12T20:12:10.342 DEBUG:boto:Token: None 2022-09-12T20:12:10.342 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj58 2022-09-12T20:12:10.343 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:o68U7nySdVDxTuBwZ8LB14yXelM= 2022-09-12T20:12:10.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:o68U7nySdVDxTuBwZ8LB14yXelM=', 'Content-Length': '0'} 2022-09-12T20:12:10.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3d6cd78473f8886-00631f929a-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.346 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:10.346 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:10.347 DEBUG:boto:Method: HEAD 2022-09-12T20:12:10.347 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:12:10.348 DEBUG:boto:Data: 2022-09-12T20:12:10.348 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.349 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.349 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.350 DEBUG:boto:Params: {} 2022-09-12T20:12:10.351 DEBUG:boto:Token: None 2022-09-12T20:12:10.351 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj58 2022-09-12T20:12:10.352 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:o68U7nySdVDxTuBwZ8LB14yXelM= 2022-09-12T20:12:10.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:o68U7nySdVDxTuBwZ8LB14yXelM=', 'Content-Length': '0'} 2022-09-12T20:12:10.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d58ebd1c460bab48-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:12:10.356 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:10.356 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:10.357 DEBUG:boto:Method: GET 2022-09-12T20:12:10.357 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:12:10.358 DEBUG:boto:Data: 2022-09-12T20:12:10.359 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.359 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.360 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.360 DEBUG:boto:Params: {} 2022-09-12T20:12:10.361 DEBUG:boto:Token: None 2022-09-12T20:12:10.362 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj58 2022-09-12T20:12:10.362 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y5FPj7hQWcW0mF+S25sfvnXZPpc= 2022-09-12T20:12:10.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y5FPj7hQWcW0mF+S25sfvnXZPpc=', 'Content-Length': '0'} 2022-09-12T20:12:10.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eef18146a6fa391b-00631f929a-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.366 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:10.366 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:10.367 DEBUG:boto:Method: GET 2022-09-12T20:12:10.367 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:12:10.368 DEBUG:boto:Data: 2022-09-12T20:12:10.368 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.369 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.369 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.370 DEBUG:boto:Params: {} 2022-09-12T20:12:10.371 DEBUG:boto:Token: None 2022-09-12T20:12:10.371 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj58 2022-09-12T20:12:10.372 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y5FPj7hQWcW0mF+S25sfvnXZPpc= 2022-09-12T20:12:10.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y5FPj7hQWcW0mF+S25sfvnXZPpc=', 'Content-Length': '0'} 2022-09-12T20:12:10.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003eb6ab639a827a88-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.418 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:10.419 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:10.419 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:12:10.420 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:12:10.420 DEBUG:boto:Method: GET 2022-09-12T20:12:10.421 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:12:10.422 DEBUG:boto:Data: 2022-09-12T20:12:10.422 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.423 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.423 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.424 DEBUG:boto:Params: {} 2022-09-12T20:12:10.424 DEBUG:boto:Token: None 2022-09-12T20:12:10.425 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/ 2022-09-12T20:12:10.426 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA= 2022-09-12T20:12:10.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA=', 'Content-Length': '0'} 2022-09-12T20:12:10.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c66227c392bbe4fd-00631f929a-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.430 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:10.431 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:10.431 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:10.432 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:12:10.432 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:12:10.433 DEBUG:boto:Method: GET 2022-09-12T20:12:10.433 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:12:10.434 DEBUG:boto:Data: 2022-09-12T20:12:10.434 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.435 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.435 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.436 DEBUG:boto:Params: {} 2022-09-12T20:12:10.437 DEBUG:boto:Token: None 2022-09-12T20:12:10.437 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/ 2022-09-12T20:12:10.438 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA= 2022-09-12T20:12:10.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA=', 'Content-Length': '0'} 2022-09-12T20:12:10.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6f1ee9e8f0ccdbf-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.443 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:10.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:12:10.445 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:10.445 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:10.446 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:10.447 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:10.447 DEBUG:boto:Method: GET 2022-09-12T20:12:10.448 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:10.448 DEBUG:boto:Data: 2022-09-12T20:12:10.449 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.449 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.450 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.450 DEBUG:boto:Params: {} 2022-09-12T20:12:10.451 DEBUG:boto:Token: None 2022-09-12T20:12:10.451 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:10.452 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BzSPKpyXhT26vnaEwL2o8DhkBF4= 2022-09-12T20:12:10.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BzSPKpyXhT26vnaEwL2o8DhkBF4=', 'Content-Length': '0'} 2022-09-12T20:12:10.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007903269717aa3104-00631f929a-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.456 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:10.457 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:10.458 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:10.459 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:10.460 DEBUG:boto:Method: GET 2022-09-12T20:12:10.461 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:10.462 DEBUG:boto:Data: 2022-09-12T20:12:10.463 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.464 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.465 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.466 DEBUG:boto:Params: {} 2022-09-12T20:12:10.467 DEBUG:boto:Token: None 2022-09-12T20:12:10.468 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:10.469 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BzSPKpyXhT26vnaEwL2o8DhkBF4= 2022-09-12T20:12:10.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BzSPKpyXhT26vnaEwL2o8DhkBF4=', 'Content-Length': '0'} 2022-09-12T20:12:10.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000829e050d1e456e47-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.474 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:10.475 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:10.475 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:10.476 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:10.476 DEBUG:boto:Method: HEAD 2022-09-12T20:12:10.477 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:10.477 DEBUG:boto:Data: 2022-09-12T20:12:10.478 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.478 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.479 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.479 DEBUG:boto:Params: {} 2022-09-12T20:12:10.480 DEBUG:boto:Token: None 2022-09-12T20:12:10.480 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:10.481 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DF1W8fanHbzudblVyjNz5lnDcuI= 2022-09-12T20:12:10.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DF1W8fanHbzudblVyjNz5lnDcuI=', 'Content-Length': '0'} 2022-09-12T20:12:10.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c661f0af767f8f5-00631f929a-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.484 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:10.485 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:10.485 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:10.486 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:10.486 DEBUG:boto:Method: HEAD 2022-09-12T20:12:10.487 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:10.487 DEBUG:boto:Data: 2022-09-12T20:12:10.488 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.488 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.489 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.489 DEBUG:boto:Params: {} 2022-09-12T20:12:10.490 DEBUG:boto:Token: None 2022-09-12T20:12:10.491 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:10.491 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DF1W8fanHbzudblVyjNz5lnDcuI= 2022-09-12T20:12:10.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DF1W8fanHbzudblVyjNz5lnDcuI=', 'Content-Length': '0'} 2022-09-12T20:12:10.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000745d7bc7795e9dc5-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:12:10.495 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:10.496 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:10.496 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:10.497 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:10.497 DEBUG:boto:Method: GET 2022-09-12T20:12:10.498 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:10.498 DEBUG:boto:Data: 2022-09-12T20:12:10.499 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.500 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.500 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.501 DEBUG:boto:Params: {} 2022-09-12T20:12:10.501 DEBUG:boto:Token: None 2022-09-12T20:12:10.502 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:10.502 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BzSPKpyXhT26vnaEwL2o8DhkBF4= 2022-09-12T20:12:10.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BzSPKpyXhT26vnaEwL2o8DhkBF4=', 'Content-Length': '0'} 2022-09-12T20:12:10.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000595596189e9020f-00631f929a-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.546 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:10.547 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:10.548 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:10.549 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:10.549 DEBUG:boto:Method: GET 2022-09-12T20:12:10.550 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:10.550 DEBUG:boto:Data: 2022-09-12T20:12:10.551 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.552 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.552 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.553 DEBUG:boto:Params: {} 2022-09-12T20:12:10.553 DEBUG:boto:Token: None 2022-09-12T20:12:10.554 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:10.556 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BzSPKpyXhT26vnaEwL2o8DhkBF4= 2022-09-12T20:12:10.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BzSPKpyXhT26vnaEwL2o8DhkBF4=', 'Content-Length': '0'} 2022-09-12T20:12:10.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009637093315a0275d-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.560 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:10.561 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:10.561 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:12:10.562 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:12:10.563 DEBUG:boto:Method: GET 2022-09-12T20:12:10.563 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:12:10.564 DEBUG:boto:Data: 2022-09-12T20:12:10.564 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.565 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.565 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.566 DEBUG:boto:Params: {} 2022-09-12T20:12:10.566 DEBUG:boto:Token: None 2022-09-12T20:12:10.567 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/ 2022-09-12T20:12:10.567 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA= 2022-09-12T20:12:10.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA=', 'Content-Length': '0'} 2022-09-12T20:12:10.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000add2f18530939eee-00631f929a-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.572 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:10.572 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:10.573 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:10.573 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:12:10.574 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:12:10.575 DEBUG:boto:Method: GET 2022-09-12T20:12:10.575 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:12:10.576 DEBUG:boto:Data: 2022-09-12T20:12:10.576 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.577 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.577 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.578 DEBUG:boto:Params: {} 2022-09-12T20:12:10.578 DEBUG:boto:Token: None 2022-09-12T20:12:10.579 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/ 2022-09-12T20:12:10.579 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA= 2022-09-12T20:12:10.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA=', 'Content-Length': '0'} 2022-09-12T20:12:10.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059b79f9ce5c2dd01-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.585 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:10.586 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:10.586 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:10.587 DEBUG:boto:Method: HEAD 2022-09-12T20:12:10.587 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:12:10.588 DEBUG:boto:Data: 2022-09-12T20:12:10.588 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.589 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.589 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.590 DEBUG:boto:Params: {} 2022-09-12T20:12:10.590 DEBUG:boto:Token: None 2022-09-12T20:12:10.591 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj68 2022-09-12T20:12:10.592 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LmrmrL/2e2cWUFzK//oxQUfiCNc= 2022-09-12T20:12:10.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LmrmrL/2e2cWUFzK//oxQUfiCNc=', 'Content-Length': '0'} 2022-09-12T20:12:10.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000832c2aee84bec688-00631f929a-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.595 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:10.595 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:10.596 DEBUG:boto:Method: HEAD 2022-09-12T20:12:10.596 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:12:10.597 DEBUG:boto:Data: 2022-09-12T20:12:10.597 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.598 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.598 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.599 DEBUG:boto:Params: {} 2022-09-12T20:12:10.600 DEBUG:boto:Token: None 2022-09-12T20:12:10.600 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj68 2022-09-12T20:12:10.601 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LmrmrL/2e2cWUFzK//oxQUfiCNc= 2022-09-12T20:12:10.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LmrmrL/2e2cWUFzK//oxQUfiCNc=', 'Content-Length': '0'} 2022-09-12T20:12:10.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036d6826b1ebf2f79-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:12:10.605 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:10.605 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:10.606 DEBUG:boto:Method: GET 2022-09-12T20:12:10.606 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:12:10.607 DEBUG:boto:Data: 2022-09-12T20:12:10.608 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.608 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.609 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.609 DEBUG:boto:Params: {} 2022-09-12T20:12:10.610 DEBUG:boto:Token: None 2022-09-12T20:12:10.610 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj68 2022-09-12T20:12:10.611 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m2prObfXGjC35QnWPK+CTd8WTLE= 2022-09-12T20:12:10.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m2prObfXGjC35QnWPK+CTd8WTLE=', 'Content-Length': '0'} 2022-09-12T20:12:10.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ed6db8a9c745aef-00631f929a-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.654 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:10.655 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:10.655 DEBUG:boto:Method: GET 2022-09-12T20:12:10.656 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:12:10.656 DEBUG:boto:Data: 2022-09-12T20:12:10.657 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.657 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.658 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.659 DEBUG:boto:Params: {} 2022-09-12T20:12:10.659 DEBUG:boto:Token: None 2022-09-12T20:12:10.660 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj68 2022-09-12T20:12:10.660 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m2prObfXGjC35QnWPK+CTd8WTLE= 2022-09-12T20:12:10.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m2prObfXGjC35QnWPK+CTd8WTLE=', 'Content-Length': '0'} 2022-09-12T20:12:10.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae1cd5fa95d7c9b1-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.664 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:10.664 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:10.665 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:12:10.665 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:12:10.666 DEBUG:boto:Method: GET 2022-09-12T20:12:10.667 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:12:10.667 DEBUG:boto:Data: 2022-09-12T20:12:10.668 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.668 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.669 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.669 DEBUG:boto:Params: {} 2022-09-12T20:12:10.670 DEBUG:boto:Token: None 2022-09-12T20:12:10.670 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/ 2022-09-12T20:12:10.671 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA= 2022-09-12T20:12:10.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA=', 'Content-Length': '0'} 2022-09-12T20:12:10.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bbdcebc05a84409-00631f929a-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.675 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:10.676 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:10.676 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:10.677 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:12:10.677 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:12:10.678 DEBUG:boto:Method: GET 2022-09-12T20:12:10.678 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:12:10.679 DEBUG:boto:Data: 2022-09-12T20:12:10.679 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.680 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.680 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.681 DEBUG:boto:Params: {} 2022-09-12T20:12:10.681 DEBUG:boto:Token: None 2022-09-12T20:12:10.682 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/ 2022-09-12T20:12:10.683 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA= 2022-09-12T20:12:10.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA=', 'Content-Length': '0'} 2022-09-12T20:12:10.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b5b4116bab003c5-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.689 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:10.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:12:10.690 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:10.691 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:10.691 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:10.692 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:10.692 DEBUG:boto:Method: GET 2022-09-12T20:12:10.693 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:10.693 DEBUG:boto:Data: 2022-09-12T20:12:10.694 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.694 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.695 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.696 DEBUG:boto:Params: {} 2022-09-12T20:12:10.696 DEBUG:boto:Token: None 2022-09-12T20:12:10.697 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:10.697 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:paa4PQXF3/w81mKDgmL7//jo3zc= 2022-09-12T20:12:10.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:paa4PQXF3/w81mKDgmL7//jo3zc=', 'Content-Length': '0'} 2022-09-12T20:12:10.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ee51ff02d93ae5c-00631f929a-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.701 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:10.701 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:10.702 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:10.702 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:10.703 DEBUG:boto:Method: GET 2022-09-12T20:12:10.703 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:10.704 DEBUG:boto:Data: 2022-09-12T20:12:10.704 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.705 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.705 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.706 DEBUG:boto:Params: {} 2022-09-12T20:12:10.706 DEBUG:boto:Token: None 2022-09-12T20:12:10.707 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:10.708 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:paa4PQXF3/w81mKDgmL7//jo3zc= 2022-09-12T20:12:10.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:paa4PQXF3/w81mKDgmL7//jo3zc=', 'Content-Length': '0'} 2022-09-12T20:12:10.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4cd912b34b8a3d8-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.711 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:10.712 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:10.712 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:10.713 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:10.713 DEBUG:boto:Method: HEAD 2022-09-12T20:12:10.714 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:10.714 DEBUG:boto:Data: 2022-09-12T20:12:10.715 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.715 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.716 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.716 DEBUG:boto:Params: {} 2022-09-12T20:12:10.717 DEBUG:boto:Token: None 2022-09-12T20:12:10.717 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:10.718 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rDflSErUkcMPpqmBcAejtH7hIWs= 2022-09-12T20:12:10.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rDflSErUkcMPpqmBcAejtH7hIWs=', 'Content-Length': '0'} 2022-09-12T20:12:10.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003beb4666b709cf88-00631f929a-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.721 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:10.722 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:10.722 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:10.723 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:10.723 DEBUG:boto:Method: HEAD 2022-09-12T20:12:10.724 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:10.724 DEBUG:boto:Data: 2022-09-12T20:12:10.725 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.725 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.726 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.726 DEBUG:boto:Params: {} 2022-09-12T20:12:10.727 DEBUG:boto:Token: None 2022-09-12T20:12:10.727 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:10.728 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rDflSErUkcMPpqmBcAejtH7hIWs= 2022-09-12T20:12:10.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rDflSErUkcMPpqmBcAejtH7hIWs=', 'Content-Length': '0'} 2022-09-12T20:12:10.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7d45a3d0d92ba81-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:12:10.732 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:10.732 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:10.733 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:10.733 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:10.734 DEBUG:boto:Method: GET 2022-09-12T20:12:10.734 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:10.735 DEBUG:boto:Data: 2022-09-12T20:12:10.735 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.736 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.737 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.737 DEBUG:boto:Params: {} 2022-09-12T20:12:10.738 DEBUG:boto:Token: None 2022-09-12T20:12:10.738 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:10.739 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:paa4PQXF3/w81mKDgmL7//jo3zc= 2022-09-12T20:12:10.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:paa4PQXF3/w81mKDgmL7//jo3zc=', 'Content-Length': '0'} 2022-09-12T20:12:10.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088f1a8f4bf524c2f-00631f929a-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.742 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:10.743 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:10.743 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:10.744 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:10.744 DEBUG:boto:Method: GET 2022-09-12T20:12:10.745 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:10.745 DEBUG:boto:Data: 2022-09-12T20:12:10.746 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.746 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.747 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.747 DEBUG:boto:Params: {} 2022-09-12T20:12:10.748 DEBUG:boto:Token: None 2022-09-12T20:12:10.749 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:10.749 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:paa4PQXF3/w81mKDgmL7//jo3zc= 2022-09-12T20:12:10.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:paa4PQXF3/w81mKDgmL7//jo3zc=', 'Content-Length': '0'} 2022-09-12T20:12:10.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000abb0abfd0916682-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.798 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:10.799 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:10.799 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:12:10.800 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:12:10.800 DEBUG:boto:Method: GET 2022-09-12T20:12:10.801 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:12:10.801 DEBUG:boto:Data: 2022-09-12T20:12:10.802 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.802 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.803 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.803 DEBUG:boto:Params: {} 2022-09-12T20:12:10.804 DEBUG:boto:Token: None 2022-09-12T20:12:10.805 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/ 2022-09-12T20:12:10.805 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA= 2022-09-12T20:12:10.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA=', 'Content-Length': '0'} 2022-09-12T20:12:10.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3332d8b9d1ce589-00631f929a-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.810 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:10.811 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:10.811 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:10.812 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:12:10.812 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:12:10.813 DEBUG:boto:Method: GET 2022-09-12T20:12:10.813 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:12:10.814 DEBUG:boto:Data: 2022-09-12T20:12:10.814 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.815 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.815 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.816 DEBUG:boto:Params: {} 2022-09-12T20:12:10.816 DEBUG:boto:Token: None 2022-09-12T20:12:10.817 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/ 2022-09-12T20:12:10.818 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA= 2022-09-12T20:12:10.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA=', 'Content-Length': '0'} 2022-09-12T20:12:10.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001139160930c74cea-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.823 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:10.824 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:10.824 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:10.825 DEBUG:boto:Method: HEAD 2022-09-12T20:12:10.825 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:12:10.826 DEBUG:boto:Data: 2022-09-12T20:12:10.826 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.827 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.827 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.828 DEBUG:boto:Params: {} 2022-09-12T20:12:10.828 DEBUG:boto:Token: None 2022-09-12T20:12:10.829 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj69 2022-09-12T20:12:10.829 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+uTk1jAaLYAyCqwQnnr4EW3A1KI= 2022-09-12T20:12:10.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+uTk1jAaLYAyCqwQnnr4EW3A1KI=', 'Content-Length': '0'} 2022-09-12T20:12:10.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a416ed732a3c461-00631f929a-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.833 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:10.833 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:10.834 DEBUG:boto:Method: HEAD 2022-09-12T20:12:10.834 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:12:10.835 DEBUG:boto:Data: 2022-09-12T20:12:10.835 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.836 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.836 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.837 DEBUG:boto:Params: {} 2022-09-12T20:12:10.837 DEBUG:boto:Token: None 2022-09-12T20:12:10.838 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj69 2022-09-12T20:12:10.838 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+uTk1jAaLYAyCqwQnnr4EW3A1KI= 2022-09-12T20:12:10.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+uTk1jAaLYAyCqwQnnr4EW3A1KI=', 'Content-Length': '0'} 2022-09-12T20:12:10.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000958a0793dbc951b6-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:12:10.842 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:10.843 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:10.844 DEBUG:boto:Method: GET 2022-09-12T20:12:10.844 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:12:10.845 DEBUG:boto:Data: 2022-09-12T20:12:10.845 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.846 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.846 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.847 DEBUG:boto:Params: {} 2022-09-12T20:12:10.847 DEBUG:boto:Token: None 2022-09-12T20:12:10.848 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj69 2022-09-12T20:12:10.849 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A8sbFFDlffHjDn9Mkq4j2f2nUWA= 2022-09-12T20:12:10.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A8sbFFDlffHjDn9Mkq4j2f2nUWA=', 'Content-Length': '0'} 2022-09-12T20:12:10.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cefd52c3769d41b-00631f929a-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.852 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:10.853 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:10.853 DEBUG:boto:Method: GET 2022-09-12T20:12:10.854 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:12:10.854 DEBUG:boto:Data: 2022-09-12T20:12:10.855 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.855 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.856 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.856 DEBUG:boto:Params: {} 2022-09-12T20:12:10.857 DEBUG:boto:Token: None 2022-09-12T20:12:10.858 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj69 2022-09-12T20:12:10.858 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A8sbFFDlffHjDn9Mkq4j2f2nUWA= 2022-09-12T20:12:10.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A8sbFFDlffHjDn9Mkq4j2f2nUWA=', 'Content-Length': '0'} 2022-09-12T20:12:10.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000194b8985f735ae54-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.861 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:10.862 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:10.863 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:12:10.863 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:12:10.864 DEBUG:boto:Method: GET 2022-09-12T20:12:10.864 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:12:10.865 DEBUG:boto:Data: 2022-09-12T20:12:10.865 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.866 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.866 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.867 DEBUG:boto:Params: {} 2022-09-12T20:12:10.867 DEBUG:boto:Token: None 2022-09-12T20:12:10.868 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/ 2022-09-12T20:12:10.869 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA= 2022-09-12T20:12:10.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA=', 'Content-Length': '0'} 2022-09-12T20:12:10.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ec6d34723970a63-00631f929a-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.872 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:10.873 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:10.874 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:10.874 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:12:10.875 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:12:10.875 DEBUG:boto:Method: GET 2022-09-12T20:12:10.876 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:12:10.876 DEBUG:boto:Data: 2022-09-12T20:12:10.877 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.877 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.878 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.878 DEBUG:boto:Params: {} 2022-09-12T20:12:10.879 DEBUG:boto:Token: None 2022-09-12T20:12:10.879 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/ 2022-09-12T20:12:10.880 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA= 2022-09-12T20:12:10.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6FaEY3g/wB97IRVyvbFD9RhStOA=', 'Content-Length': '0'} 2022-09-12T20:12:10.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca45f6f1459a14a1-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.885 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:10.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:12:10.886 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:10.887 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:10.888 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:10.888 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:10.889 DEBUG:boto:Method: GET 2022-09-12T20:12:10.889 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:10.890 DEBUG:boto:Data: 2022-09-12T20:12:10.890 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.891 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.891 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.892 DEBUG:boto:Params: {} 2022-09-12T20:12:10.892 DEBUG:boto:Token: None 2022-09-12T20:12:10.893 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:10.894 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1OwR31/oqyc/lmTXzE20fNHGW90= 2022-09-12T20:12:10.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1OwR31/oqyc/lmTXzE20fNHGW90=', 'Content-Length': '0'} 2022-09-12T20:12:10.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000337359976631245d-00631f929a-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.897 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:10.898 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:10.898 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:10.899 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:10.899 DEBUG:boto:Method: GET 2022-09-12T20:12:10.900 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:10.901 DEBUG:boto:Data: 2022-09-12T20:12:10.901 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.902 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.902 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.903 DEBUG:boto:Params: {} 2022-09-12T20:12:10.903 DEBUG:boto:Token: None 2022-09-12T20:12:10.904 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:10.904 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1OwR31/oqyc/lmTXzE20fNHGW90= 2022-09-12T20:12:10.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1OwR31/oqyc/lmTXzE20fNHGW90=', 'Content-Length': '0'} 2022-09-12T20:12:10.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022bb5eef6bbe6501-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.950 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:10.951 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:10.951 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:10.952 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:10.952 DEBUG:boto:Method: HEAD 2022-09-12T20:12:10.953 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:10.953 DEBUG:boto:Data: 2022-09-12T20:12:10.954 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.954 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.955 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.956 DEBUG:boto:Params: {} 2022-09-12T20:12:10.956 DEBUG:boto:Token: None 2022-09-12T20:12:10.957 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:10.957 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:d/V6rkIDGPoejSKMl7MTi5r2MsY= 2022-09-12T20:12:10.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:d/V6rkIDGPoejSKMl7MTi5r2MsY=', 'Content-Length': '0'} 2022-09-12T20:12:10.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbe6619a348d1416-00631f929a-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.961 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:10.962 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:10.962 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:10.963 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:10.963 DEBUG:boto:Method: HEAD 2022-09-12T20:12:10.964 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:10.964 DEBUG:boto:Data: 2022-09-12T20:12:10.965 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.965 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.966 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.966 DEBUG:boto:Params: {} 2022-09-12T20:12:10.967 DEBUG:boto:Token: None 2022-09-12T20:12:10.967 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:10.968 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:d/V6rkIDGPoejSKMl7MTi5r2MsY= 2022-09-12T20:12:10.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:d/V6rkIDGPoejSKMl7MTi5r2MsY=', 'Content-Length': '0'} 2022-09-12T20:12:10.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1e1bdcd4f647e16-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:12:10.972 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:10.973 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:10.973 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:10.974 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:10.974 DEBUG:boto:Method: GET 2022-09-12T20:12:10.975 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:10.975 DEBUG:boto:Data: 2022-09-12T20:12:10.976 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.976 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.977 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.977 DEBUG:boto:Params: {} 2022-09-12T20:12:10.978 DEBUG:boto:Token: None 2022-09-12T20:12:10.979 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:10.979 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1OwR31/oqyc/lmTXzE20fNHGW90= 2022-09-12T20:12:10.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1OwR31/oqyc/lmTXzE20fNHGW90=', 'Content-Length': '0'} 2022-09-12T20:12:10.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f3c73ef16cd83d5-00631f929a-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.983 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:10.983 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:10.984 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:10.984 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:10.985 DEBUG:boto:Method: GET 2022-09-12T20:12:10.985 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:10.986 DEBUG:boto:Data: 2022-09-12T20:12:10.987 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.987 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:10.988 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.988 DEBUG:boto:Params: {} 2022-09-12T20:12:10.989 DEBUG:boto:Token: None 2022-09-12T20:12:10.989 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:10 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:10.990 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1OwR31/oqyc/lmTXzE20fNHGW90= 2022-09-12T20:12:10.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1OwR31/oqyc/lmTXzE20fNHGW90=', 'Content-Length': '0'} 2022-09-12T20:12:10.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2abdb2656d57536-00631f929a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:10.994 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:10.994 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:10.995 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:12:10.995 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:12:10.996 DEBUG:boto:Method: GET 2022-09-12T20:12:10.996 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:12:10.997 DEBUG:boto:Data: 2022-09-12T20:12:10.997 DEBUG:boto:Headers: {} 2022-09-12T20:12:10.998 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:10.998 DEBUG:boto:Port: 8000 2022-09-12T20:12:10.999 DEBUG:boto:Params: {} 2022-09-12T20:12:10.999 DEBUG:boto:Token: None 2022-09-12T20:12:11.000 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/ 2022-09-12T20:12:11.001 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU= 2022-09-12T20:12:11.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU=', 'Content-Length': '0'} 2022-09-12T20:12:11.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c55556e0362317a6-00631f929b-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.046 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:11.047 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:11.048 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:11.048 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:12:11.049 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:12:11.049 DEBUG:boto:Method: GET 2022-09-12T20:12:11.050 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:12:11.050 DEBUG:boto:Data: 2022-09-12T20:12:11.051 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.051 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.052 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.052 DEBUG:boto:Params: {} 2022-09-12T20:12:11.053 DEBUG:boto:Token: None 2022-09-12T20:12:11.054 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/ 2022-09-12T20:12:11.054 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU= 2022-09-12T20:12:11.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU=', 'Content-Length': '0'} 2022-09-12T20:12:11.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077df4925e8fb5a2f-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.060 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:11.060 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:11.061 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:11.061 DEBUG:boto:Method: HEAD 2022-09-12T20:12:11.062 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:12:11.062 DEBUG:boto:Data: 2022-09-12T20:12:11.063 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.063 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.064 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.064 DEBUG:boto:Params: {} 2022-09-12T20:12:11.065 DEBUG:boto:Token: None 2022-09-12T20:12:11.065 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj76 2022-09-12T20:12:11.066 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jmrTqZRNWZhkl2EqNTzL0pWmY9k= 2022-09-12T20:12:11.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jmrTqZRNWZhkl2EqNTzL0pWmY9k=', 'Content-Length': '0'} 2022-09-12T20:12:11.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d683dc98fc037bc0-00631f929b-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.069 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:11.070 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:11.070 DEBUG:boto:Method: HEAD 2022-09-12T20:12:11.071 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:12:11.071 DEBUG:boto:Data: 2022-09-12T20:12:11.072 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.072 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.073 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.073 DEBUG:boto:Params: {} 2022-09-12T20:12:11.074 DEBUG:boto:Token: None 2022-09-12T20:12:11.074 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj76 2022-09-12T20:12:11.075 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jmrTqZRNWZhkl2EqNTzL0pWmY9k= 2022-09-12T20:12:11.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jmrTqZRNWZhkl2EqNTzL0pWmY9k=', 'Content-Length': '0'} 2022-09-12T20:12:11.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d8fdf91cb000ed4-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:12:11.079 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:11.079 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:11.080 DEBUG:boto:Method: GET 2022-09-12T20:12:11.080 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:12:11.081 DEBUG:boto:Data: 2022-09-12T20:12:11.081 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.082 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.083 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.083 DEBUG:boto:Params: {} 2022-09-12T20:12:11.084 DEBUG:boto:Token: None 2022-09-12T20:12:11.084 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj76 2022-09-12T20:12:11.085 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fsTx5J3soJ0eRgrWuLuNWQIR/Mw= 2022-09-12T20:12:11.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fsTx5J3soJ0eRgrWuLuNWQIR/Mw=', 'Content-Length': '0'} 2022-09-12T20:12:11.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d1456d9b054c39f-00631f929b-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.088 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:11.089 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:11.089 DEBUG:boto:Method: GET 2022-09-12T20:12:11.090 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:12:11.091 DEBUG:boto:Data: 2022-09-12T20:12:11.091 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.092 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.092 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.093 DEBUG:boto:Params: {} 2022-09-12T20:12:11.093 DEBUG:boto:Token: None 2022-09-12T20:12:11.094 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj76 2022-09-12T20:12:11.094 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fsTx5J3soJ0eRgrWuLuNWQIR/Mw= 2022-09-12T20:12:11.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fsTx5J3soJ0eRgrWuLuNWQIR/Mw=', 'Content-Length': '0'} 2022-09-12T20:12:11.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014056f281173efbf-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.138 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:11.139 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:11.139 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:12:11.140 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:12:11.140 DEBUG:boto:Method: GET 2022-09-12T20:12:11.141 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:12:11.141 DEBUG:boto:Data: 2022-09-12T20:12:11.142 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.142 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.143 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.143 DEBUG:boto:Params: {} 2022-09-12T20:12:11.144 DEBUG:boto:Token: None 2022-09-12T20:12:11.145 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/ 2022-09-12T20:12:11.145 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU= 2022-09-12T20:12:11.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU=', 'Content-Length': '0'} 2022-09-12T20:12:11.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae4dd77637c46a16-00631f929b-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.149 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:11.150 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:11.151 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:11.151 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:12:11.152 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:12:11.152 DEBUG:boto:Method: GET 2022-09-12T20:12:11.153 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:12:11.153 DEBUG:boto:Data: 2022-09-12T20:12:11.154 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.155 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.155 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.156 DEBUG:boto:Params: {} 2022-09-12T20:12:11.156 DEBUG:boto:Token: None 2022-09-12T20:12:11.157 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/ 2022-09-12T20:12:11.157 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU= 2022-09-12T20:12:11.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU=', 'Content-Length': '0'} 2022-09-12T20:12:11.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ba745bd970ac0f0-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.163 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:11.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:12:11.164 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:11.165 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:11.165 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:11.166 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:11.166 DEBUG:boto:Method: GET 2022-09-12T20:12:11.167 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:11.168 DEBUG:boto:Data: 2022-09-12T20:12:11.168 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.169 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.169 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.170 DEBUG:boto:Params: {} 2022-09-12T20:12:11.170 DEBUG:boto:Token: None 2022-09-12T20:12:11.171 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:11.172 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cd8uVWhDgnWbvtCkI0itcg43KZM= 2022-09-12T20:12:11.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cd8uVWhDgnWbvtCkI0itcg43KZM=', 'Content-Length': '0'} 2022-09-12T20:12:11.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032c91f2e99188654-00631f929b-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.175 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:11.176 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:11.176 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:11.177 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:11.177 DEBUG:boto:Method: GET 2022-09-12T20:12:11.178 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:11.179 DEBUG:boto:Data: 2022-09-12T20:12:11.179 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.180 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.180 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.181 DEBUG:boto:Params: {} 2022-09-12T20:12:11.181 DEBUG:boto:Token: None 2022-09-12T20:12:11.182 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:11.182 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cd8uVWhDgnWbvtCkI0itcg43KZM= 2022-09-12T20:12:11.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cd8uVWhDgnWbvtCkI0itcg43KZM=', 'Content-Length': '0'} 2022-09-12T20:12:11.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061a03fa6b8e64602-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.186 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:11.187 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:11.187 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:11.188 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:11.188 DEBUG:boto:Method: HEAD 2022-09-12T20:12:11.189 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:11.189 DEBUG:boto:Data: 2022-09-12T20:12:11.190 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.190 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.191 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.191 DEBUG:boto:Params: {} 2022-09-12T20:12:11.192 DEBUG:boto:Token: None 2022-09-12T20:12:11.192 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:11.193 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fZvh93hyZSoxIX+xxw7fTACsLaA= 2022-09-12T20:12:11.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fZvh93hyZSoxIX+xxw7fTACsLaA=', 'Content-Length': '0'} 2022-09-12T20:12:11.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3c3dd9971070493-00631f929b-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.196 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:11.197 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:11.197 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:11.198 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:11.198 DEBUG:boto:Method: HEAD 2022-09-12T20:12:11.199 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:11.200 DEBUG:boto:Data: 2022-09-12T20:12:11.200 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.201 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.201 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.202 DEBUG:boto:Params: {} 2022-09-12T20:12:11.202 DEBUG:boto:Token: None 2022-09-12T20:12:11.203 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:11.203 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fZvh93hyZSoxIX+xxw7fTACsLaA= 2022-09-12T20:12:11.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fZvh93hyZSoxIX+xxw7fTACsLaA=', 'Content-Length': '0'} 2022-09-12T20:12:11.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d71e30a8a1ac2db-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:12:11.207 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:11.208 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:11.208 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:11.209 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:11.209 DEBUG:boto:Method: GET 2022-09-12T20:12:11.210 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:11.211 DEBUG:boto:Data: 2022-09-12T20:12:11.211 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.212 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.212 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.213 DEBUG:boto:Params: {} 2022-09-12T20:12:11.213 DEBUG:boto:Token: None 2022-09-12T20:12:11.214 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:11.215 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cd8uVWhDgnWbvtCkI0itcg43KZM= 2022-09-12T20:12:11.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cd8uVWhDgnWbvtCkI0itcg43KZM=', 'Content-Length': '0'} 2022-09-12T20:12:11.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000553b6f768d3939c6-00631f929b-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.218 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:11.219 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:11.219 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:11.220 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:11.220 DEBUG:boto:Method: GET 2022-09-12T20:12:11.221 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:11.221 DEBUG:boto:Data: 2022-09-12T20:12:11.222 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.222 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.223 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.223 DEBUG:boto:Params: {} 2022-09-12T20:12:11.224 DEBUG:boto:Token: None 2022-09-12T20:12:11.225 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:11.225 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cd8uVWhDgnWbvtCkI0itcg43KZM= 2022-09-12T20:12:11.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cd8uVWhDgnWbvtCkI0itcg43KZM=', 'Content-Length': '0'} 2022-09-12T20:12:11.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f7397ba7e101662-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.270 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:11.271 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:11.272 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:12:11.272 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:12:11.273 DEBUG:boto:Method: GET 2022-09-12T20:12:11.273 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:12:11.274 DEBUG:boto:Data: 2022-09-12T20:12:11.274 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.275 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.276 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.276 DEBUG:boto:Params: {} 2022-09-12T20:12:11.277 DEBUG:boto:Token: None 2022-09-12T20:12:11.277 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/ 2022-09-12T20:12:11.278 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU= 2022-09-12T20:12:11.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU=', 'Content-Length': '0'} 2022-09-12T20:12:11.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb1284ca8d50cf1c-00631f929b-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.282 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:11.283 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:11.284 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:11.284 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:12:11.285 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:12:11.286 DEBUG:boto:Method: GET 2022-09-12T20:12:11.286 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:12:11.287 DEBUG:boto:Data: 2022-09-12T20:12:11.287 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.288 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.288 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.289 DEBUG:boto:Params: {} 2022-09-12T20:12:11.289 DEBUG:boto:Token: None 2022-09-12T20:12:11.290 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/ 2022-09-12T20:12:11.291 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU= 2022-09-12T20:12:11.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU=', 'Content-Length': '0'} 2022-09-12T20:12:11.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003522bf5423f40414-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.296 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:11.297 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:11.297 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:11.298 DEBUG:boto:Method: HEAD 2022-09-12T20:12:11.298 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:12:11.299 DEBUG:boto:Data: 2022-09-12T20:12:11.299 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.300 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.300 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.301 DEBUG:boto:Params: {} 2022-09-12T20:12:11.302 DEBUG:boto:Token: None 2022-09-12T20:12:11.302 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj85 2022-09-12T20:12:11.303 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x8vrAz91OgpE7lGOLCeB+91QrXE= 2022-09-12T20:12:11.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x8vrAz91OgpE7lGOLCeB+91QrXE=', 'Content-Length': '0'} 2022-09-12T20:12:11.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed34409c4e2e882f-00631f929b-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.306 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:11.307 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:11.307 DEBUG:boto:Method: HEAD 2022-09-12T20:12:11.308 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:12:11.308 DEBUG:boto:Data: 2022-09-12T20:12:11.309 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.309 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.310 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.310 DEBUG:boto:Params: {} 2022-09-12T20:12:11.311 DEBUG:boto:Token: None 2022-09-12T20:12:11.311 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj85 2022-09-12T20:12:11.312 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x8vrAz91OgpE7lGOLCeB+91QrXE= 2022-09-12T20:12:11.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x8vrAz91OgpE7lGOLCeB+91QrXE=', 'Content-Length': '0'} 2022-09-12T20:12:11.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005352be9ced76cc72-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:12:11.316 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:11.316 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:11.317 DEBUG:boto:Method: GET 2022-09-12T20:12:11.317 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:12:11.318 DEBUG:boto:Data: 2022-09-12T20:12:11.318 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.319 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.319 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.320 DEBUG:boto:Params: {} 2022-09-12T20:12:11.321 DEBUG:boto:Token: None 2022-09-12T20:12:11.321 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj85 2022-09-12T20:12:11.322 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gmgYbvXUI2/S/0f1GPpX6h7agEI= 2022-09-12T20:12:11.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gmgYbvXUI2/S/0f1GPpX6h7agEI=', 'Content-Length': '0'} 2022-09-12T20:12:11.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054c4ff30cf8934b6-00631f929b-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.366 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:11.367 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:11.367 DEBUG:boto:Method: GET 2022-09-12T20:12:11.368 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:12:11.369 DEBUG:boto:Data: 2022-09-12T20:12:11.370 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.371 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.372 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.372 DEBUG:boto:Params: {} 2022-09-12T20:12:11.373 DEBUG:boto:Token: None 2022-09-12T20:12:11.374 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj85 2022-09-12T20:12:11.375 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gmgYbvXUI2/S/0f1GPpX6h7agEI= 2022-09-12T20:12:11.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gmgYbvXUI2/S/0f1GPpX6h7agEI=', 'Content-Length': '0'} 2022-09-12T20:12:11.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c03d577df583d873-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.379 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:11.380 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:11.380 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:12:11.381 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:12:11.381 DEBUG:boto:Method: GET 2022-09-12T20:12:11.382 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:12:11.382 DEBUG:boto:Data: 2022-09-12T20:12:11.383 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.383 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.384 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.384 DEBUG:boto:Params: {} 2022-09-12T20:12:11.385 DEBUG:boto:Token: None 2022-09-12T20:12:11.386 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/ 2022-09-12T20:12:11.386 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU= 2022-09-12T20:12:11.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU=', 'Content-Length': '0'} 2022-09-12T20:12:11.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6164b370af1aadc-00631f929b-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.390 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:11.391 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:11.392 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:11.392 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:12:11.393 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:12:11.393 DEBUG:boto:Method: GET 2022-09-12T20:12:11.394 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:12:11.394 DEBUG:boto:Data: 2022-09-12T20:12:11.395 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.395 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.396 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.396 DEBUG:boto:Params: {} 2022-09-12T20:12:11.397 DEBUG:boto:Token: None 2022-09-12T20:12:11.398 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/ 2022-09-12T20:12:11.398 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU= 2022-09-12T20:12:11.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU=', 'Content-Length': '0'} 2022-09-12T20:12:11.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010ceaa8a9d027e6c-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.446 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:11.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:12:11.447 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:11.448 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:11.449 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:11.449 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:11.450 DEBUG:boto:Method: GET 2022-09-12T20:12:11.450 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:11.451 DEBUG:boto:Data: 2022-09-12T20:12:11.451 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.452 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.452 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.453 DEBUG:boto:Params: {} 2022-09-12T20:12:11.454 DEBUG:boto:Token: None 2022-09-12T20:12:11.454 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:11.455 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WO3ABgSVtWjnVTRPIC25wgZ4r+4= 2022-09-12T20:12:11.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WO3ABgSVtWjnVTRPIC25wgZ4r+4=', 'Content-Length': '0'} 2022-09-12T20:12:11.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b04a94429f6fa4bd-00631f929b-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.459 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:11.459 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:11.460 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:11.461 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:11.461 DEBUG:boto:Method: GET 2022-09-12T20:12:11.462 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:11.462 DEBUG:boto:Data: 2022-09-12T20:12:11.463 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.464 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.464 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.465 DEBUG:boto:Params: {} 2022-09-12T20:12:11.465 DEBUG:boto:Token: None 2022-09-12T20:12:11.466 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:11.467 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WO3ABgSVtWjnVTRPIC25wgZ4r+4= 2022-09-12T20:12:11.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WO3ABgSVtWjnVTRPIC25wgZ4r+4=', 'Content-Length': '0'} 2022-09-12T20:12:11.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000472278483384ec5f-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.470 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:11.471 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:11.472 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:11.473 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:11.474 DEBUG:boto:Method: HEAD 2022-09-12T20:12:11.474 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:11.475 DEBUG:boto:Data: 2022-09-12T20:12:11.475 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.476 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.476 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.477 DEBUG:boto:Params: {} 2022-09-12T20:12:11.477 DEBUG:boto:Token: None 2022-09-12T20:12:11.478 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:11.479 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tgNNCHCDHFBMf6tNA6LZXh3CZUo= 2022-09-12T20:12:11.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tgNNCHCDHFBMf6tNA6LZXh3CZUo=', 'Content-Length': '0'} 2022-09-12T20:12:11.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da84ab6a8e147cea-00631f929b-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.482 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:11.483 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:11.483 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:11.484 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:11.484 DEBUG:boto:Method: HEAD 2022-09-12T20:12:11.485 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:11.485 DEBUG:boto:Data: 2022-09-12T20:12:11.486 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.487 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.487 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.488 DEBUG:boto:Params: {} 2022-09-12T20:12:11.488 DEBUG:boto:Token: None 2022-09-12T20:12:11.489 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:11.490 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tgNNCHCDHFBMf6tNA6LZXh3CZUo= 2022-09-12T20:12:11.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tgNNCHCDHFBMf6tNA6LZXh3CZUo=', 'Content-Length': '0'} 2022-09-12T20:12:11.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096b9e3593114ea85-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:12:11.494 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:11.494 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:11.495 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:11.496 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:11.496 DEBUG:boto:Method: GET 2022-09-12T20:12:11.497 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:11.497 DEBUG:boto:Data: 2022-09-12T20:12:11.498 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.499 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.499 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.500 DEBUG:boto:Params: {} 2022-09-12T20:12:11.500 DEBUG:boto:Token: None 2022-09-12T20:12:11.501 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:11.502 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WO3ABgSVtWjnVTRPIC25wgZ4r+4= 2022-09-12T20:12:11.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WO3ABgSVtWjnVTRPIC25wgZ4r+4=', 'Content-Length': '0'} 2022-09-12T20:12:11.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6cf40d9261b7833-00631f929b-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.506 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:11.507 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:11.508 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:11.509 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:11.510 DEBUG:boto:Method: GET 2022-09-12T20:12:11.511 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:11.511 DEBUG:boto:Data: 2022-09-12T20:12:11.512 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.513 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.513 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.514 DEBUG:boto:Params: {} 2022-09-12T20:12:11.514 DEBUG:boto:Token: None 2022-09-12T20:12:11.515 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:11.516 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WO3ABgSVtWjnVTRPIC25wgZ4r+4= 2022-09-12T20:12:11.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WO3ABgSVtWjnVTRPIC25wgZ4r+4=', 'Content-Length': '0'} 2022-09-12T20:12:11.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ba803dd188ceb8d-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.519 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:11.520 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:11.520 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:12:11.521 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:12:11.521 DEBUG:boto:Method: GET 2022-09-12T20:12:11.522 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:12:11.523 DEBUG:boto:Data: 2022-09-12T20:12:11.523 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.524 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.524 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.525 DEBUG:boto:Params: {} 2022-09-12T20:12:11.526 DEBUG:boto:Token: None 2022-09-12T20:12:11.526 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/ 2022-09-12T20:12:11.527 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU= 2022-09-12T20:12:11.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU=', 'Content-Length': '0'} 2022-09-12T20:12:11.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6acd57c3f37340e-00631f929b-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.531 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:11.532 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:11.533 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:11.533 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:12:11.534 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:12:11.534 DEBUG:boto:Method: GET 2022-09-12T20:12:11.535 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:12:11.535 DEBUG:boto:Data: 2022-09-12T20:12:11.536 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.537 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.537 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.538 DEBUG:boto:Params: {} 2022-09-12T20:12:11.538 DEBUG:boto:Token: None 2022-09-12T20:12:11.539 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/ 2022-09-12T20:12:11.540 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU= 2022-09-12T20:12:11.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU=', 'Content-Length': '0'} 2022-09-12T20:12:11.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b1bb842854da82d-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.545 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:11.546 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:11.546 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:11.547 DEBUG:boto:Method: HEAD 2022-09-12T20:12:11.547 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:12:11.548 DEBUG:boto:Data: 2022-09-12T20:12:11.548 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.549 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.549 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.550 DEBUG:boto:Params: {} 2022-09-12T20:12:11.551 DEBUG:boto:Token: None 2022-09-12T20:12:11.551 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj87 2022-09-12T20:12:11.552 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+cx8fhR3kXxdV6ycN/+julwelkQ= 2022-09-12T20:12:11.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+cx8fhR3kXxdV6ycN/+julwelkQ=', 'Content-Length': '0'} 2022-09-12T20:12:11.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001b01abe1a97384d-00631f929b-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.555 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:11.556 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:11.556 DEBUG:boto:Method: HEAD 2022-09-12T20:12:11.557 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:12:11.557 DEBUG:boto:Data: 2022-09-12T20:12:11.558 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.559 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.559 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.560 DEBUG:boto:Params: {} 2022-09-12T20:12:11.560 DEBUG:boto:Token: None 2022-09-12T20:12:11.561 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj87 2022-09-12T20:12:11.562 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+cx8fhR3kXxdV6ycN/+julwelkQ= 2022-09-12T20:12:11.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+cx8fhR3kXxdV6ycN/+julwelkQ=', 'Content-Length': '0'} 2022-09-12T20:12:11.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa002c3edab4d9c8-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:12:11.567 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:11.568 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:11.568 DEBUG:boto:Method: GET 2022-09-12T20:12:11.569 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:12:11.569 DEBUG:boto:Data: 2022-09-12T20:12:11.570 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.570 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.571 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.572 DEBUG:boto:Params: {} 2022-09-12T20:12:11.572 DEBUG:boto:Token: None 2022-09-12T20:12:11.573 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj87 2022-09-12T20:12:11.574 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B5Qoi5OhDSFVTRD227gvsh+NAKo= 2022-09-12T20:12:11.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B5Qoi5OhDSFVTRD227gvsh+NAKo=', 'Content-Length': '0'} 2022-09-12T20:12:11.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edd3ce27a1bb9400-00631f929b-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.622 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:11.623 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:11.623 DEBUG:boto:Method: GET 2022-09-12T20:12:11.624 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:12:11.625 DEBUG:boto:Data: 2022-09-12T20:12:11.625 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.626 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.626 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.627 DEBUG:boto:Params: {} 2022-09-12T20:12:11.627 DEBUG:boto:Token: None 2022-09-12T20:12:11.628 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj87 2022-09-12T20:12:11.629 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B5Qoi5OhDSFVTRD227gvsh+NAKo= 2022-09-12T20:12:11.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B5Qoi5OhDSFVTRD227gvsh+NAKo=', 'Content-Length': '0'} 2022-09-12T20:12:11.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bddd152403bc5858-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.633 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:11.633 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:11.634 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:12:11.635 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:12:11.636 DEBUG:boto:Method: GET 2022-09-12T20:12:11.637 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:12:11.637 DEBUG:boto:Data: 2022-09-12T20:12:11.638 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.638 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.639 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.639 DEBUG:boto:Params: {} 2022-09-12T20:12:11.640 DEBUG:boto:Token: None 2022-09-12T20:12:11.641 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/ 2022-09-12T20:12:11.641 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU= 2022-09-12T20:12:11.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU=', 'Content-Length': '0'} 2022-09-12T20:12:11.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5c5411d5b59609b-00631f929b-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.646 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:11.647 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:11.648 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:11.649 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:12:11.649 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:12:11.650 DEBUG:boto:Method: GET 2022-09-12T20:12:11.650 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:12:11.651 DEBUG:boto:Data: 2022-09-12T20:12:11.651 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.652 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.652 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.653 DEBUG:boto:Params: {} 2022-09-12T20:12:11.653 DEBUG:boto:Token: None 2022-09-12T20:12:11.654 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/ 2022-09-12T20:12:11.655 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU= 2022-09-12T20:12:11.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU=', 'Content-Length': '0'} 2022-09-12T20:12:11.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7d82d9a83d6ac0c-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.660 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:11.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:12:11.662 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:11.662 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:11.663 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:11.663 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:11.664 DEBUG:boto:Method: GET 2022-09-12T20:12:11.664 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:11.665 DEBUG:boto:Data: 2022-09-12T20:12:11.666 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.666 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.667 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.667 DEBUG:boto:Params: {} 2022-09-12T20:12:11.668 DEBUG:boto:Token: None 2022-09-12T20:12:11.668 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:11.669 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B77a6tykvWob2UqIFEANKVr9SnQ= 2022-09-12T20:12:11.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B77a6tykvWob2UqIFEANKVr9SnQ=', 'Content-Length': '0'} 2022-09-12T20:12:11.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce45be76d0a35778-00631f929b-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.673 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:11.673 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:11.674 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:11.675 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:11.675 DEBUG:boto:Method: GET 2022-09-12T20:12:11.676 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:11.676 DEBUG:boto:Data: 2022-09-12T20:12:11.677 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.677 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.678 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.678 DEBUG:boto:Params: {} 2022-09-12T20:12:11.679 DEBUG:boto:Token: None 2022-09-12T20:12:11.680 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:11.680 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B77a6tykvWob2UqIFEANKVr9SnQ= 2022-09-12T20:12:11.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B77a6tykvWob2UqIFEANKVr9SnQ=', 'Content-Length': '0'} 2022-09-12T20:12:11.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e75788eaaa6c9b3-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.684 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:11.685 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:11.685 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:11.686 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:11.687 DEBUG:boto:Method: HEAD 2022-09-12T20:12:11.687 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:11.688 DEBUG:boto:Data: 2022-09-12T20:12:11.688 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.689 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.689 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.690 DEBUG:boto:Params: {} 2022-09-12T20:12:11.691 DEBUG:boto:Token: None 2022-09-12T20:12:11.691 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:11.692 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RFqD8IFbHTerQYdCseRmFOJt+xA= 2022-09-12T20:12:11.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RFqD8IFbHTerQYdCseRmFOJt+xA=', 'Content-Length': '0'} 2022-09-12T20:12:11.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bedd8eea0c302041-00631f929b-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.696 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:11.696 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:11.697 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:11.697 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:11.698 DEBUG:boto:Method: HEAD 2022-09-12T20:12:11.698 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:11.699 DEBUG:boto:Data: 2022-09-12T20:12:11.699 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.700 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.700 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.701 DEBUG:boto:Params: {} 2022-09-12T20:12:11.701 DEBUG:boto:Token: None 2022-09-12T20:12:11.702 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:11.703 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RFqD8IFbHTerQYdCseRmFOJt+xA= 2022-09-12T20:12:11.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RFqD8IFbHTerQYdCseRmFOJt+xA=', 'Content-Length': '0'} 2022-09-12T20:12:11.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f2149884841e59a-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:12:11.707 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:11.707 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:11.708 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:11.708 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:11.709 DEBUG:boto:Method: GET 2022-09-12T20:12:11.709 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:11.710 DEBUG:boto:Data: 2022-09-12T20:12:11.711 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.711 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.712 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.712 DEBUG:boto:Params: {} 2022-09-12T20:12:11.713 DEBUG:boto:Token: None 2022-09-12T20:12:11.714 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:11.714 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B77a6tykvWob2UqIFEANKVr9SnQ= 2022-09-12T20:12:11.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B77a6tykvWob2UqIFEANKVr9SnQ=', 'Content-Length': '0'} 2022-09-12T20:12:11.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022459ff108f9351d-00631f929b-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.718 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:11.719 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:11.719 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:11.720 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:11.720 DEBUG:boto:Method: GET 2022-09-12T20:12:11.721 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:11.721 DEBUG:boto:Data: 2022-09-12T20:12:11.722 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.723 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.723 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.724 DEBUG:boto:Params: {} 2022-09-12T20:12:11.724 DEBUG:boto:Token: None 2022-09-12T20:12:11.725 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:11.726 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B77a6tykvWob2UqIFEANKVr9SnQ= 2022-09-12T20:12:11.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B77a6tykvWob2UqIFEANKVr9SnQ=', 'Content-Length': '0'} 2022-09-12T20:12:11.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000905c46b6f2de7584-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.729 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:11.730 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:11.730 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:12:11.731 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:12:11.731 DEBUG:boto:Method: GET 2022-09-12T20:12:11.732 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:12:11.732 DEBUG:boto:Data: 2022-09-12T20:12:11.733 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.733 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.734 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.735 DEBUG:boto:Params: {} 2022-09-12T20:12:11.735 DEBUG:boto:Token: None 2022-09-12T20:12:11.736 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/ 2022-09-12T20:12:11.736 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU= 2022-09-12T20:12:11.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU=', 'Content-Length': '0'} 2022-09-12T20:12:11.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000911ff9aa5002cb30-00631f929b-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.741 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:11.742 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:11.742 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:11.743 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:12:11.743 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:12:11.744 DEBUG:boto:Method: GET 2022-09-12T20:12:11.745 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:12:11.745 DEBUG:boto:Data: 2022-09-12T20:12:11.746 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.746 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.747 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.747 DEBUG:boto:Params: {} 2022-09-12T20:12:11.748 DEBUG:boto:Token: None 2022-09-12T20:12:11.749 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/ 2022-09-12T20:12:11.749 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU= 2022-09-12T20:12:11.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU=', 'Content-Length': '0'} 2022-09-12T20:12:11.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000531ef89f602ed876-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.755 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:11.756 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:11.757 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:11.757 DEBUG:boto:Method: HEAD 2022-09-12T20:12:11.758 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:12:11.758 DEBUG:boto:Data: 2022-09-12T20:12:11.759 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.759 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.760 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.760 DEBUG:boto:Params: {} 2022-09-12T20:12:11.761 DEBUG:boto:Token: None 2022-09-12T20:12:11.762 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj92 2022-09-12T20:12:11.762 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6JdJ9tSZ1SXGnnxjPARz8SFOjJk= 2022-09-12T20:12:11.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6JdJ9tSZ1SXGnnxjPARz8SFOjJk=', 'Content-Length': '0'} 2022-09-12T20:12:11.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe8d4653dfd72785-00631f929b-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.766 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:11.766 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:11.767 DEBUG:boto:Method: HEAD 2022-09-12T20:12:11.767 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:12:11.768 DEBUG:boto:Data: 2022-09-12T20:12:11.768 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.769 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.770 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.770 DEBUG:boto:Params: {} 2022-09-12T20:12:11.771 DEBUG:boto:Token: None 2022-09-12T20:12:11.771 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj92 2022-09-12T20:12:11.772 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6JdJ9tSZ1SXGnnxjPARz8SFOjJk= 2022-09-12T20:12:11.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6JdJ9tSZ1SXGnnxjPARz8SFOjJk=', 'Content-Length': '0'} 2022-09-12T20:12:11.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000461b45dc38a31419-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:12:11.776 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:11.777 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:11.777 DEBUG:boto:Method: GET 2022-09-12T20:12:11.778 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:12:11.778 DEBUG:boto:Data: 2022-09-12T20:12:11.779 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.779 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.780 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.781 DEBUG:boto:Params: {} 2022-09-12T20:12:11.781 DEBUG:boto:Token: None 2022-09-12T20:12:11.782 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj92 2022-09-12T20:12:11.782 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vUTUFQMf8cgSx2X3k6JbB0LF+LU= 2022-09-12T20:12:11.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vUTUFQMf8cgSx2X3k6JbB0LF+LU=', 'Content-Length': '0'} 2022-09-12T20:12:11.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003dd2b5b531e529ad-00631f929b-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.786 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:11.786 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:11.787 DEBUG:boto:Method: GET 2022-09-12T20:12:11.788 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:12:11.788 DEBUG:boto:Data: 2022-09-12T20:12:11.789 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.790 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.790 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.791 DEBUG:boto:Params: {} 2022-09-12T20:12:11.791 DEBUG:boto:Token: None 2022-09-12T20:12:11.792 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj92 2022-09-12T20:12:11.793 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vUTUFQMf8cgSx2X3k6JbB0LF+LU= 2022-09-12T20:12:11.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vUTUFQMf8cgSx2X3k6JbB0LF+LU=', 'Content-Length': '0'} 2022-09-12T20:12:11.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df6a40f1f3f6119b-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.838 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:11.839 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:11.839 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:12:11.840 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:12:11.841 DEBUG:boto:Method: GET 2022-09-12T20:12:11.842 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:12:11.842 DEBUG:boto:Data: 2022-09-12T20:12:11.843 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.843 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.844 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.844 DEBUG:boto:Params: {} 2022-09-12T20:12:11.845 DEBUG:boto:Token: None 2022-09-12T20:12:11.846 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/ 2022-09-12T20:12:11.846 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU= 2022-09-12T20:12:11.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU=', 'Content-Length': '0'} 2022-09-12T20:12:11.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000584e8023324495f0-00631f929b-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.850 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:11.851 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:11.852 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:11.852 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:12:11.853 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:12:11.853 DEBUG:boto:Method: GET 2022-09-12T20:12:11.854 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:12:11.854 DEBUG:boto:Data: 2022-09-12T20:12:11.855 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.856 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.856 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.857 DEBUG:boto:Params: {} 2022-09-12T20:12:11.857 DEBUG:boto:Token: None 2022-09-12T20:12:11.858 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/ 2022-09-12T20:12:11.859 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU= 2022-09-12T20:12:11.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU=', 'Content-Length': '0'} 2022-09-12T20:12:11.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000465ab3243fe4832a-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.865 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:11.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:12:11.866 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:11.867 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:11.868 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:11.868 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:11.869 DEBUG:boto:Method: GET 2022-09-12T20:12:11.869 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:11.870 DEBUG:boto:Data: 2022-09-12T20:12:11.870 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.871 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.871 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.872 DEBUG:boto:Params: {} 2022-09-12T20:12:11.873 DEBUG:boto:Token: None 2022-09-12T20:12:11.873 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:11.874 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Tfz8Ht5KkMLB9eAGK10HaUw4zOM= 2022-09-12T20:12:11.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Tfz8Ht5KkMLB9eAGK10HaUw4zOM=', 'Content-Length': '0'} 2022-09-12T20:12:11.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b15f6afd995768e7-00631f929b-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.878 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:11.878 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:11.879 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:11.879 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:11.880 DEBUG:boto:Method: GET 2022-09-12T20:12:11.880 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:11.881 DEBUG:boto:Data: 2022-09-12T20:12:11.881 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.882 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.882 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.883 DEBUG:boto:Params: {} 2022-09-12T20:12:11.883 DEBUG:boto:Token: None 2022-09-12T20:12:11.884 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:11.885 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Tfz8Ht5KkMLB9eAGK10HaUw4zOM= 2022-09-12T20:12:11.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Tfz8Ht5KkMLB9eAGK10HaUw4zOM=', 'Content-Length': '0'} 2022-09-12T20:12:11.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000764fde9111fc4a83-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.888 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:11.889 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:11.889 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:11.890 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:11.890 DEBUG:boto:Method: HEAD 2022-09-12T20:12:11.891 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:11.891 DEBUG:boto:Data: 2022-09-12T20:12:11.892 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.892 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.893 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.893 DEBUG:boto:Params: {} 2022-09-12T20:12:11.894 DEBUG:boto:Token: None 2022-09-12T20:12:11.894 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:11.895 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hRjkDbgPjIXjAe+/8vse2IC8RNE= 2022-09-12T20:12:11.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hRjkDbgPjIXjAe+/8vse2IC8RNE=', 'Content-Length': '0'} 2022-09-12T20:12:11.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fae5b1f19077d7cc-00631f929b-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.899 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:11.899 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:11.900 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:11.900 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:11.901 DEBUG:boto:Method: HEAD 2022-09-12T20:12:11.901 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:11.902 DEBUG:boto:Data: 2022-09-12T20:12:11.902 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.903 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.903 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.904 DEBUG:boto:Params: {} 2022-09-12T20:12:11.904 DEBUG:boto:Token: None 2022-09-12T20:12:11.905 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:11.905 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hRjkDbgPjIXjAe+/8vse2IC8RNE= 2022-09-12T20:12:11.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hRjkDbgPjIXjAe+/8vse2IC8RNE=', 'Content-Length': '0'} 2022-09-12T20:12:11.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028fd54c57b9c3eee-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:12:11.909 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:11.909 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:11.910 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:11.911 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:11.911 DEBUG:boto:Method: GET 2022-09-12T20:12:11.912 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:11.912 DEBUG:boto:Data: 2022-09-12T20:12:11.913 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.914 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.914 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.915 DEBUG:boto:Params: {} 2022-09-12T20:12:11.916 DEBUG:boto:Token: None 2022-09-12T20:12:11.916 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:11.917 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Tfz8Ht5KkMLB9eAGK10HaUw4zOM= 2022-09-12T20:12:11.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Tfz8Ht5KkMLB9eAGK10HaUw4zOM=', 'Content-Length': '0'} 2022-09-12T20:12:11.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003058463a2574afa6-00631f929b-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.921 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:11.921 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:11.922 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:11.923 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:11.923 DEBUG:boto:Method: GET 2022-09-12T20:12:11.924 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:11.924 DEBUG:boto:Data: 2022-09-12T20:12:11.925 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.926 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.926 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.927 DEBUG:boto:Params: {} 2022-09-12T20:12:11.927 DEBUG:boto:Token: None 2022-09-12T20:12:11.928 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:11.929 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Tfz8Ht5KkMLB9eAGK10HaUw4zOM= 2022-09-12T20:12:11.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Tfz8Ht5KkMLB9eAGK10HaUw4zOM=', 'Content-Length': '0'} 2022-09-12T20:12:11.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006428e0305e8b0cda-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.933 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:11.933 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:11.934 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:12:11.934 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:12:11.935 DEBUG:boto:Method: GET 2022-09-12T20:12:11.935 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:12:11.936 DEBUG:boto:Data: 2022-09-12T20:12:11.936 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.937 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.938 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.938 DEBUG:boto:Params: {} 2022-09-12T20:12:11.939 DEBUG:boto:Token: None 2022-09-12T20:12:11.939 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/ 2022-09-12T20:12:11.940 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU= 2022-09-12T20:12:11.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU=', 'Content-Length': '0'} 2022-09-12T20:12:11.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c89e8dd3207bbfc-00631f929b-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.944 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:11.945 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:11.946 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:11.946 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:12:11.947 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:12:11.948 DEBUG:boto:Method: GET 2022-09-12T20:12:11.948 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:12:11.949 DEBUG:boto:Data: 2022-09-12T20:12:11.949 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.950 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.951 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.951 DEBUG:boto:Params: {} 2022-09-12T20:12:11.952 DEBUG:boto:Token: None 2022-09-12T20:12:11.953 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/ 2022-09-12T20:12:11.953 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU= 2022-09-12T20:12:11.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YkA/3es0ZJY4SJJgV6KevTmmiXU=', 'Content-Length': '0'} 2022-09-12T20:12:11.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc95f0c5dd904d93-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.959 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:11.959 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:11.960 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:11.961 DEBUG:boto:Method: HEAD 2022-09-12T20:12:11.961 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:12:11.962 DEBUG:boto:Data: 2022-09-12T20:12:11.962 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.963 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.963 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.964 DEBUG:boto:Params: {} 2022-09-12T20:12:11.964 DEBUG:boto:Token: None 2022-09-12T20:12:11.965 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj95 2022-09-12T20:12:11.966 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gKpr01L97ztMDT4uLoHNPdFAdME= 2022-09-12T20:12:11.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gKpr01L97ztMDT4uLoHNPdFAdME=', 'Content-Length': '0'} 2022-09-12T20:12:11.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005098b8cc72b769e1-00631f929b-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.969 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:11.970 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:11.970 DEBUG:boto:Method: HEAD 2022-09-12T20:12:11.971 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:12:11.971 DEBUG:boto:Data: 2022-09-12T20:12:11.972 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.972 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.973 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.973 DEBUG:boto:Params: {} 2022-09-12T20:12:11.974 DEBUG:boto:Token: None 2022-09-12T20:12:11.975 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj95 2022-09-12T20:12:11.975 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gKpr01L97ztMDT4uLoHNPdFAdME= 2022-09-12T20:12:11.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gKpr01L97ztMDT4uLoHNPdFAdME=', 'Content-Length': '0'} 2022-09-12T20:12:11.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de989fc2f1e26c2e-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:12:11.979 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:11.980 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:11.981 DEBUG:boto:Method: GET 2022-09-12T20:12:11.981 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:12:11.982 DEBUG:boto:Data: 2022-09-12T20:12:11.982 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.983 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:11.983 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.984 DEBUG:boto:Params: {} 2022-09-12T20:12:11.985 DEBUG:boto:Token: None 2022-09-12T20:12:11.985 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj95 2022-09-12T20:12:11.986 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:g/tGuYlAg4gXezkfjKmxLVUWQo0= 2022-09-12T20:12:11.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:g/tGuYlAg4gXezkfjKmxLVUWQo0=', 'Content-Length': '0'} 2022-09-12T20:12:11.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009228390054ce1d1b-00631f929b-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:11.989 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:11.990 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:11.990 DEBUG:boto:Method: GET 2022-09-12T20:12:11.991 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:12:11.992 DEBUG:boto:Data: 2022-09-12T20:12:11.992 DEBUG:boto:Headers: {} 2022-09-12T20:12:11.993 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:11.993 DEBUG:boto:Port: 8000 2022-09-12T20:12:11.994 DEBUG:boto:Params: {} 2022-09-12T20:12:11.994 DEBUG:boto:Token: None 2022-09-12T20:12:11.995 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:11 GMT /jdpiak-75/obj95 2022-09-12T20:12:11.995 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:g/tGuYlAg4gXezkfjKmxLVUWQo0= 2022-09-12T20:12:11.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:g/tGuYlAg4gXezkfjKmxLVUWQo0=', 'Content-Length': '0'} 2022-09-12T20:12:11.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e92d67f9a3587bef-00631f929b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.042 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:12.043 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:12.043 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:12:12.044 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:12:12.044 DEBUG:boto:Method: GET 2022-09-12T20:12:12.045 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:12:12.045 DEBUG:boto:Data: 2022-09-12T20:12:12.046 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.046 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:12.047 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.047 DEBUG:boto:Params: {} 2022-09-12T20:12:12.048 DEBUG:boto:Token: None 2022-09-12T20:12:12.049 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/ 2022-09-12T20:12:12.049 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA= 2022-09-12T20:12:12.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA=', 'Content-Length': '0'} 2022-09-12T20:12:12.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073c5cff5309a3ad0-00631f929c-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.053 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:12.054 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:12.055 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:12.055 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:12:12.056 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:12:12.056 DEBUG:boto:Method: GET 2022-09-12T20:12:12.057 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:12:12.057 DEBUG:boto:Data: 2022-09-12T20:12:12.058 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.058 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.059 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.060 DEBUG:boto:Params: {} 2022-09-12T20:12:12.060 DEBUG:boto:Token: None 2022-09-12T20:12:12.061 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/ 2022-09-12T20:12:12.061 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA= 2022-09-12T20:12:12.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA=', 'Content-Length': '0'} 2022-09-12T20:12:12.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cd88ed0e7b003b7-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.067 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:12.068 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-75 zones={test-zone2, test-zone1} 2022-09-12T20:12:12.069 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-75 zones={test-zone3, test-zone1} 2022-09-12T20:12:12.069 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:12.070 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:12.070 DEBUG:boto:Method: HEAD 2022-09-12T20:12:12.071 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:12:12.071 DEBUG:boto:Data: 2022-09-12T20:12:12.072 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.072 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.073 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.073 DEBUG:boto:Params: {} 2022-09-12T20:12:12.074 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T20:12:12.075 DEBUG:boto:Token: None 2022-09-12T20:12:12.075 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/ 2022-09-12T20:12:12.076 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8F7JAOlse9/Ev+3HmoeiViqyidQ= 2022-09-12T20:12:12.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8F7JAOlse9/Ev+3HmoeiViqyidQ=', 'Content-Length': '0'} 2022-09-12T20:12:12.081 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd669f81034c91e0-00631f929c-1826-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.082 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:12.082 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:12.083 DEBUG:boto:Method: HEAD 2022-09-12T20:12:12.083 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:12:12.084 DEBUG:boto:Data: 2022-09-12T20:12:12.084 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.085 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.085 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.086 DEBUG:boto:Params: {} 2022-09-12T20:12:12.086 DEBUG:boto:Token: None 2022-09-12T20:12:12.087 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/ 2022-09-12T20:12:12.088 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8F7JAOlse9/Ev+3HmoeiViqyidQ= 2022-09-12T20:12:12.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8F7JAOlse9/Ev+3HmoeiViqyidQ=', 'Content-Length': '0'} 2022-09-12T20:12:12.093 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000788029b56a65b2cf-00631f929c-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.093 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T20:12:12.094 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:12.094 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:12.095 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:12.095 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:12.096 DEBUG:boto:Method: GET 2022-09-12T20:12:12.096 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:12.097 DEBUG:boto:Data: 2022-09-12T20:12:12.098 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.098 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.099 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.099 DEBUG:boto:Params: {} 2022-09-12T20:12:12.100 DEBUG:boto:Token: None 2022-09-12T20:12:12.100 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/?versions 2022-09-12T20:12:12.101 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q9RoH18NSNeRgtj1xB+V1Q1beZY= 2022-09-12T20:12:12.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q9RoH18NSNeRgtj1xB+V1Q1beZY=', 'Content-Length': '0'} 2022-09-12T20:12:12.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014eb1ec5dcb68e57-00631f929c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.106 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:12.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:12:12.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-12T20:12:12.109 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-12T20:12:12.109 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-12T20:12:12.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:12:12.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:12:12.111 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:12:12.111 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-12T20:12:12.112 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-12T20:12:12.112 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:12:12.113 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:12:12.113 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:12:12.114 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:12:12.114 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-12T20:12:12.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:12:12.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:12:12.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:12:12.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-12T20:12:12.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:12:12.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-12T20:12:12.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:12:12.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:12:12.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-12T20:12:12.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-12T20:12:12.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:12:12.120 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T20:12:12.120 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:12.121 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:12.122 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:12.122 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:12.123 DEBUG:boto:Method: GET 2022-09-12T20:12:12.123 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:12.124 DEBUG:boto:Data: 2022-09-12T20:12:12.124 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.125 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.125 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.126 DEBUG:boto:Params: {} 2022-09-12T20:12:12.126 DEBUG:boto:Token: None 2022-09-12T20:12:12.127 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/?versions 2022-09-12T20:12:12.128 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q9RoH18NSNeRgtj1xB+V1Q1beZY= 2022-09-12T20:12:12.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q9RoH18NSNeRgtj1xB+V1Q1beZY=', 'Content-Length': '0'} 2022-09-12T20:12:12.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8b42f4b984c18f8-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.134 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:12.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:12:12.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-12T20:12:12.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-12T20:12:12.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-12T20:12:12.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:12:12.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:12:12.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:12:12.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-12T20:12:12.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-12T20:12:12.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:12:12.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:12:12.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:12:12.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:12:12.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-12T20:12:12.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:12:12.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:12:12.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:12:12.144 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-12T20:12:12.144 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:12:12.145 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-12T20:12:12.145 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:12:12.145 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:12:12.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-12T20:12:12.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-12T20:12:12.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:12:12.148 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:12.148 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:12.149 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:12.149 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:12.150 DEBUG:boto:Method: GET 2022-09-12T20:12:12.151 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:12.151 DEBUG:boto:Data: 2022-09-12T20:12:12.152 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.152 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.153 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.154 DEBUG:boto:Params: {} 2022-09-12T20:12:12.155 DEBUG:boto:Token: None 2022-09-12T20:12:12.155 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/?versions 2022-09-12T20:12:12.156 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q9RoH18NSNeRgtj1xB+V1Q1beZY= 2022-09-12T20:12:12.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q9RoH18NSNeRgtj1xB+V1Q1beZY=', 'Content-Length': '0'} 2022-09-12T20:12:12.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008591867093dec291-00631f929c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.160 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:12.162 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:12.163 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:12.163 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:12.164 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:12.165 DEBUG:boto:Method: GET 2022-09-12T20:12:12.165 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:12.166 DEBUG:boto:Data: 2022-09-12T20:12:12.167 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.167 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.168 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.168 DEBUG:boto:Params: {} 2022-09-12T20:12:12.169 DEBUG:boto:Token: None 2022-09-12T20:12:12.169 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/?versions 2022-09-12T20:12:12.170 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q9RoH18NSNeRgtj1xB+V1Q1beZY= 2022-09-12T20:12:12.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q9RoH18NSNeRgtj1xB+V1Q1beZY=', 'Content-Length': '0'} 2022-09-12T20:12:12.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e70ace7250c7c213-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.176 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:12.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:12:12.178 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:12.179 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:12.179 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:12.180 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:12.180 DEBUG:boto:Method: GET 2022-09-12T20:12:12.181 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:12.182 DEBUG:boto:Data: 2022-09-12T20:12:12.182 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.183 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.183 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.184 DEBUG:boto:Params: {} 2022-09-12T20:12:12.184 DEBUG:boto:Token: None 2022-09-12T20:12:12.185 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:12.186 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8sOVu7y9wpY7wB+FoU1abvJxyKo= 2022-09-12T20:12:12.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8sOVu7y9wpY7wB+FoU1abvJxyKo=', 'Content-Length': '0'} 2022-09-12T20:12:12.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f6e3ebb5ff8662d-00631f929c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.189 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:12.190 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:12.190 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:12.191 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:12.191 DEBUG:boto:Method: GET 2022-09-12T20:12:12.192 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:12.192 DEBUG:boto:Data: 2022-09-12T20:12:12.193 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.193 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.194 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.195 DEBUG:boto:Params: {} 2022-09-12T20:12:12.195 DEBUG:boto:Token: None 2022-09-12T20:12:12.196 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:12.196 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8sOVu7y9wpY7wB+FoU1abvJxyKo= 2022-09-12T20:12:12.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8sOVu7y9wpY7wB+FoU1abvJxyKo=', 'Content-Length': '0'} 2022-09-12T20:12:12.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088a51ebbfe776e0c-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.242 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:12.243 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:12.243 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:12.244 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:12.244 DEBUG:boto:Method: HEAD 2022-09-12T20:12:12.245 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:12.245 DEBUG:boto:Data: 2022-09-12T20:12:12.246 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.246 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.247 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.247 DEBUG:boto:Params: {} 2022-09-12T20:12:12.248 DEBUG:boto:Token: None 2022-09-12T20:12:12.248 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:12.249 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GNExwm3Wb9LspyKJiaa5m4MpLWg= 2022-09-12T20:12:12.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GNExwm3Wb9LspyKJiaa5m4MpLWg=', 'Content-Length': '0'} 2022-09-12T20:12:12.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f377661679d84f2-00631f929c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.252 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:12.253 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:12.253 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:12.254 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:12.254 DEBUG:boto:Method: HEAD 2022-09-12T20:12:12.255 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:12.255 DEBUG:boto:Data: 2022-09-12T20:12:12.256 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.256 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.257 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.258 DEBUG:boto:Params: {} 2022-09-12T20:12:12.258 DEBUG:boto:Token: None 2022-09-12T20:12:12.259 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:12.259 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GNExwm3Wb9LspyKJiaa5m4MpLWg= 2022-09-12T20:12:12.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GNExwm3Wb9LspyKJiaa5m4MpLWg=', 'Content-Length': '0'} 2022-09-12T20:12:12.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006e956266688e35e-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:12:12.263 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:12.264 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:12.264 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:12.265 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:12.265 DEBUG:boto:Method: GET 2022-09-12T20:12:12.266 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:12.267 DEBUG:boto:Data: 2022-09-12T20:12:12.267 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.268 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.268 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.269 DEBUG:boto:Params: {} 2022-09-12T20:12:12.269 DEBUG:boto:Token: None 2022-09-12T20:12:12.270 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:12.271 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8sOVu7y9wpY7wB+FoU1abvJxyKo= 2022-09-12T20:12:12.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8sOVu7y9wpY7wB+FoU1abvJxyKo=', 'Content-Length': '0'} 2022-09-12T20:12:12.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013c4c654685ee7da-00631f929c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.274 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:12.275 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:12.275 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:12.276 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:12.276 DEBUG:boto:Method: GET 2022-09-12T20:12:12.277 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:12.277 DEBUG:boto:Data: 2022-09-12T20:12:12.278 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.278 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.279 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.279 DEBUG:boto:Params: {} 2022-09-12T20:12:12.280 DEBUG:boto:Token: None 2022-09-12T20:12:12.281 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:12.281 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8sOVu7y9wpY7wB+FoU1abvJxyKo= 2022-09-12T20:12:12.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8sOVu7y9wpY7wB+FoU1abvJxyKo=', 'Content-Length': '0'} 2022-09-12T20:12:12.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027f9a82e4f3ad398-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.285 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:12.285 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:12.286 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:12:12.287 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:12:12.287 DEBUG:boto:Method: GET 2022-09-12T20:12:12.288 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:12:12.288 DEBUG:boto:Data: 2022-09-12T20:12:12.289 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.289 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.290 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.291 DEBUG:boto:Params: {} 2022-09-12T20:12:12.291 DEBUG:boto:Token: None 2022-09-12T20:12:12.292 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/ 2022-09-12T20:12:12.292 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA= 2022-09-12T20:12:12.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA=', 'Content-Length': '0'} 2022-09-12T20:12:12.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9a24abeb52c23d0-00631f929c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.297 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:12.298 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:12.298 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:12.299 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:12:12.300 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:12:12.300 DEBUG:boto:Method: GET 2022-09-12T20:12:12.301 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:12:12.301 DEBUG:boto:Data: 2022-09-12T20:12:12.302 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.302 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.303 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.303 DEBUG:boto:Params: {} 2022-09-12T20:12:12.304 DEBUG:boto:Token: None 2022-09-12T20:12:12.304 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/ 2022-09-12T20:12:12.305 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA= 2022-09-12T20:12:12.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA=', 'Content-Length': '0'} 2022-09-12T20:12:12.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000055f92ac75aef286-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.310 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:12.311 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:12.312 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:12.312 DEBUG:boto:Method: HEAD 2022-09-12T20:12:12.313 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:12:12.313 DEBUG:boto:Data: 2022-09-12T20:12:12.314 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.314 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.315 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.316 DEBUG:boto:Params: {} 2022-09-12T20:12:12.316 DEBUG:boto:Token: None 2022-09-12T20:12:12.317 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj11 2022-09-12T20:12:12.317 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yvOmXgrxSmwhz8elJfgdS1oGxFU= 2022-09-12T20:12:12.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yvOmXgrxSmwhz8elJfgdS1oGxFU=', 'Content-Length': '0'} 2022-09-12T20:12:12.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d952f6cc4659109-00631f929c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.321 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:12.321 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:12.322 DEBUG:boto:Method: HEAD 2022-09-12T20:12:12.322 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:12:12.323 DEBUG:boto:Data: 2022-09-12T20:12:12.323 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.324 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.324 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.325 DEBUG:boto:Params: {} 2022-09-12T20:12:12.325 DEBUG:boto:Token: None 2022-09-12T20:12:12.326 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj11 2022-09-12T20:12:12.327 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yvOmXgrxSmwhz8elJfgdS1oGxFU= 2022-09-12T20:12:12.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yvOmXgrxSmwhz8elJfgdS1oGxFU=', 'Content-Length': '0'} 2022-09-12T20:12:12.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d4d55902b241d63-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:12:12.331 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:12.331 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:12.332 DEBUG:boto:Method: GET 2022-09-12T20:12:12.332 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:12:12.333 DEBUG:boto:Data: 2022-09-12T20:12:12.333 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.334 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.334 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.335 DEBUG:boto:Params: {} 2022-09-12T20:12:12.335 DEBUG:boto:Token: None 2022-09-12T20:12:12.336 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj11 2022-09-12T20:12:12.337 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q7L4MunJjOeVaClLZMLzv0Ja7JY= 2022-09-12T20:12:12.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q7L4MunJjOeVaClLZMLzv0Ja7JY=', 'Content-Length': '0'} 2022-09-12T20:12:12.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e177a52d6638e29-00631f929c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.340 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:12.341 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:12.341 DEBUG:boto:Method: GET 2022-09-12T20:12:12.342 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:12:12.343 DEBUG:boto:Data: 2022-09-12T20:12:12.343 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.344 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.344 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.345 DEBUG:boto:Params: {} 2022-09-12T20:12:12.345 DEBUG:boto:Token: None 2022-09-12T20:12:12.346 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj11 2022-09-12T20:12:12.347 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q7L4MunJjOeVaClLZMLzv0Ja7JY= 2022-09-12T20:12:12.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q7L4MunJjOeVaClLZMLzv0Ja7JY=', 'Content-Length': '0'} 2022-09-12T20:12:12.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099b83299b125d7bc-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.390 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:12.391 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:12.391 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:12:12.392 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:12:12.393 DEBUG:boto:Method: GET 2022-09-12T20:12:12.393 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:12:12.394 DEBUG:boto:Data: 2022-09-12T20:12:12.394 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.395 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.395 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.396 DEBUG:boto:Params: {} 2022-09-12T20:12:12.397 DEBUG:boto:Token: None 2022-09-12T20:12:12.397 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/ 2022-09-12T20:12:12.398 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA= 2022-09-12T20:12:12.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA=', 'Content-Length': '0'} 2022-09-12T20:12:12.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f03c7d8b65be7561-00631f929c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.402 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:12.403 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:12.404 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:12.404 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:12:12.405 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:12:12.405 DEBUG:boto:Method: GET 2022-09-12T20:12:12.406 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:12:12.406 DEBUG:boto:Data: 2022-09-12T20:12:12.407 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.407 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.408 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.408 DEBUG:boto:Params: {} 2022-09-12T20:12:12.409 DEBUG:boto:Token: None 2022-09-12T20:12:12.409 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/ 2022-09-12T20:12:12.410 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA= 2022-09-12T20:12:12.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA=', 'Content-Length': '0'} 2022-09-12T20:12:12.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b8cfd573c65701d-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.461 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:12.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:12:12.463 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:12.464 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:12.465 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:12.465 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:12.466 DEBUG:boto:Method: GET 2022-09-12T20:12:12.466 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:12.467 DEBUG:boto:Data: 2022-09-12T20:12:12.467 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.468 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.469 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.469 DEBUG:boto:Params: {} 2022-09-12T20:12:12.470 DEBUG:boto:Token: None 2022-09-12T20:12:12.471 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:12.471 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9vWupDTrTqP9B/y9pXVvY8XMABM= 2022-09-12T20:12:12.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9vWupDTrTqP9B/y9pXVvY8XMABM=', 'Content-Length': '0'} 2022-09-12T20:12:12.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041b1280424b0e722-00631f929c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.475 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:12.476 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:12.476 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:12.477 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:12.477 DEBUG:boto:Method: GET 2022-09-12T20:12:12.478 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:12.479 DEBUG:boto:Data: 2022-09-12T20:12:12.479 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.480 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.481 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.482 DEBUG:boto:Params: {} 2022-09-12T20:12:12.483 DEBUG:boto:Token: None 2022-09-12T20:12:12.484 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:12.485 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9vWupDTrTqP9B/y9pXVvY8XMABM= 2022-09-12T20:12:12.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9vWupDTrTqP9B/y9pXVvY8XMABM=', 'Content-Length': '0'} 2022-09-12T20:12:12.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e26d95fc0771093c-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.488 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:12.489 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:12.490 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:12.491 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:12.492 DEBUG:boto:Method: HEAD 2022-09-12T20:12:12.493 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:12.494 DEBUG:boto:Data: 2022-09-12T20:12:12.495 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.495 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.496 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.497 DEBUG:boto:Params: {} 2022-09-12T20:12:12.497 DEBUG:boto:Token: None 2022-09-12T20:12:12.498 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:12.499 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:C4erf/JrgOQvbLMGk91f37vp584= 2022-09-12T20:12:12.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:C4erf/JrgOQvbLMGk91f37vp584=', 'Content-Length': '0'} 2022-09-12T20:12:12.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000316d409e31a63daa-00631f929c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.503 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:12.504 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:12.505 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:12.505 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:12.506 DEBUG:boto:Method: HEAD 2022-09-12T20:12:12.506 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:12.507 DEBUG:boto:Data: 2022-09-12T20:12:12.507 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.508 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.508 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.509 DEBUG:boto:Params: {} 2022-09-12T20:12:12.510 DEBUG:boto:Token: None 2022-09-12T20:12:12.511 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:12.511 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:C4erf/JrgOQvbLMGk91f37vp584= 2022-09-12T20:12:12.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:C4erf/JrgOQvbLMGk91f37vp584=', 'Content-Length': '0'} 2022-09-12T20:12:12.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040b60d75dd844d6f-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:12:12.516 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:12.516 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:12.517 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:12.517 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:12.518 DEBUG:boto:Method: GET 2022-09-12T20:12:12.519 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:12.519 DEBUG:boto:Data: 2022-09-12T20:12:12.520 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.520 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.521 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.522 DEBUG:boto:Params: {} 2022-09-12T20:12:12.523 DEBUG:boto:Token: None 2022-09-12T20:12:12.524 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:12.525 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9vWupDTrTqP9B/y9pXVvY8XMABM= 2022-09-12T20:12:12.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9vWupDTrTqP9B/y9pXVvY8XMABM=', 'Content-Length': '0'} 2022-09-12T20:12:12.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4ab874fffd6da3a-00631f929c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.529 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:12.530 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:12.530 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:12.531 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:12.531 DEBUG:boto:Method: GET 2022-09-12T20:12:12.532 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:12.533 DEBUG:boto:Data: 2022-09-12T20:12:12.533 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.534 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.535 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.535 DEBUG:boto:Params: {} 2022-09-12T20:12:12.536 DEBUG:boto:Token: None 2022-09-12T20:12:12.536 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:12.537 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9vWupDTrTqP9B/y9pXVvY8XMABM= 2022-09-12T20:12:12.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9vWupDTrTqP9B/y9pXVvY8XMABM=', 'Content-Length': '0'} 2022-09-12T20:12:12.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bb5b1d01c838641-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.541 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:12.541 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:12.542 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:12:12.543 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:12:12.543 DEBUG:boto:Method: GET 2022-09-12T20:12:12.544 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:12:12.545 DEBUG:boto:Data: 2022-09-12T20:12:12.545 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.546 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.546 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.547 DEBUG:boto:Params: {} 2022-09-12T20:12:12.547 DEBUG:boto:Token: None 2022-09-12T20:12:12.548 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/ 2022-09-12T20:12:12.549 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA= 2022-09-12T20:12:12.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA=', 'Content-Length': '0'} 2022-09-12T20:12:12.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000322cf92e848389b4-00631f929c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.553 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:12.554 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:12.554 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:12.555 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:12:12.555 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:12:12.556 DEBUG:boto:Method: GET 2022-09-12T20:12:12.557 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:12:12.558 DEBUG:boto:Data: 2022-09-12T20:12:12.559 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.559 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.560 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.561 DEBUG:boto:Params: {} 2022-09-12T20:12:12.562 DEBUG:boto:Token: None 2022-09-12T20:12:12.562 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/ 2022-09-12T20:12:12.563 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA= 2022-09-12T20:12:12.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA=', 'Content-Length': '0'} 2022-09-12T20:12:12.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007297b637421ad054-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.569 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:12.571 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:12.572 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:12.573 DEBUG:boto:Method: HEAD 2022-09-12T20:12:12.573 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:12:12.574 DEBUG:boto:Data: 2022-09-12T20:12:12.574 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.575 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.575 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.576 DEBUG:boto:Params: {} 2022-09-12T20:12:12.576 DEBUG:boto:Token: None 2022-09-12T20:12:12.577 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj115 2022-09-12T20:12:12.578 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nCg4GW/qcL6QNs5A5ohTpCXtdS4= 2022-09-12T20:12:12.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nCg4GW/qcL6QNs5A5ohTpCXtdS4=', 'Content-Length': '0'} 2022-09-12T20:12:12.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051fb864b686ecf03-00631f929c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.581 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:12.582 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:12.582 DEBUG:boto:Method: HEAD 2022-09-12T20:12:12.583 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:12:12.583 DEBUG:boto:Data: 2022-09-12T20:12:12.584 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.584 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.585 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.585 DEBUG:boto:Params: {} 2022-09-12T20:12:12.586 DEBUG:boto:Token: None 2022-09-12T20:12:12.587 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj115 2022-09-12T20:12:12.587 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nCg4GW/qcL6QNs5A5ohTpCXtdS4= 2022-09-12T20:12:12.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nCg4GW/qcL6QNs5A5ohTpCXtdS4=', 'Content-Length': '0'} 2022-09-12T20:12:12.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003575b4642f564caf-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:12:12.593 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:12.594 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:12.595 DEBUG:boto:Method: GET 2022-09-12T20:12:12.596 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:12:12.596 DEBUG:boto:Data: 2022-09-12T20:12:12.597 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.598 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.598 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.599 DEBUG:boto:Params: {} 2022-09-12T20:12:12.599 DEBUG:boto:Token: None 2022-09-12T20:12:12.600 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj115 2022-09-12T20:12:12.601 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yPXChcKZ+ZOE8FRQ8RPrWqHTrLk= 2022-09-12T20:12:12.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yPXChcKZ+ZOE8FRQ8RPrWqHTrLk=', 'Content-Length': '0'} 2022-09-12T20:12:12.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075165453c59d1ca0-00631f929c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.604 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:12.605 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:12.605 DEBUG:boto:Method: GET 2022-09-12T20:12:12.606 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:12:12.606 DEBUG:boto:Data: 2022-09-12T20:12:12.607 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.608 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.608 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.609 DEBUG:boto:Params: {} 2022-09-12T20:12:12.609 DEBUG:boto:Token: None 2022-09-12T20:12:12.610 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj115 2022-09-12T20:12:12.611 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yPXChcKZ+ZOE8FRQ8RPrWqHTrLk= 2022-09-12T20:12:12.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yPXChcKZ+ZOE8FRQ8RPrWqHTrLk=', 'Content-Length': '0'} 2022-09-12T20:12:12.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000842fba50a2255fe6-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.654 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:12.655 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:12.655 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:12:12.656 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:12:12.656 DEBUG:boto:Method: GET 2022-09-12T20:12:12.657 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:12:12.657 DEBUG:boto:Data: 2022-09-12T20:12:12.658 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.658 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.659 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.659 DEBUG:boto:Params: {} 2022-09-12T20:12:12.660 DEBUG:boto:Token: None 2022-09-12T20:12:12.661 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/ 2022-09-12T20:12:12.661 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA= 2022-09-12T20:12:12.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA=', 'Content-Length': '0'} 2022-09-12T20:12:12.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b88216916729c501-00631f929c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.665 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:12.666 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:12.667 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:12.667 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:12:12.668 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:12:12.668 DEBUG:boto:Method: GET 2022-09-12T20:12:12.669 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:12:12.669 DEBUG:boto:Data: 2022-09-12T20:12:12.670 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.671 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.671 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.672 DEBUG:boto:Params: {} 2022-09-12T20:12:12.672 DEBUG:boto:Token: None 2022-09-12T20:12:12.673 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/ 2022-09-12T20:12:12.673 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA= 2022-09-12T20:12:12.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA=', 'Content-Length': '0'} 2022-09-12T20:12:12.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f8a7bd35b65573a-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.679 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:12.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:12:12.680 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:12.681 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:12.682 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:12.682 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:12.683 DEBUG:boto:Method: GET 2022-09-12T20:12:12.683 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:12.684 DEBUG:boto:Data: 2022-09-12T20:12:12.684 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.685 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.685 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.686 DEBUG:boto:Params: {} 2022-09-12T20:12:12.686 DEBUG:boto:Token: None 2022-09-12T20:12:12.687 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:12.687 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aAcYZwaPY9Izy5dWheANcDdYP6U= 2022-09-12T20:12:12.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aAcYZwaPY9Izy5dWheANcDdYP6U=', 'Content-Length': '0'} 2022-09-12T20:12:12.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5e1b572731bc888-00631f929c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.691 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:12.692 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:12.692 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:12.693 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:12.693 DEBUG:boto:Method: GET 2022-09-12T20:12:12.694 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:12.694 DEBUG:boto:Data: 2022-09-12T20:12:12.695 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.695 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.696 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.696 DEBUG:boto:Params: {} 2022-09-12T20:12:12.697 DEBUG:boto:Token: None 2022-09-12T20:12:12.698 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:12.698 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aAcYZwaPY9Izy5dWheANcDdYP6U= 2022-09-12T20:12:12.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aAcYZwaPY9Izy5dWheANcDdYP6U=', 'Content-Length': '0'} 2022-09-12T20:12:12.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd1bf84b1a27898e-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.702 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:12.702 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:12.703 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:12.703 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:12.704 DEBUG:boto:Method: HEAD 2022-09-12T20:12:12.704 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:12.705 DEBUG:boto:Data: 2022-09-12T20:12:12.705 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.706 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.706 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.707 DEBUG:boto:Params: {} 2022-09-12T20:12:12.707 DEBUG:boto:Token: None 2022-09-12T20:12:12.708 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:12.709 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G6O5gBC/XKYZFFt3k6eTiByy9KM= 2022-09-12T20:12:12.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G6O5gBC/XKYZFFt3k6eTiByy9KM=', 'Content-Length': '0'} 2022-09-12T20:12:12.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a023a7cb092e1321-00631f929c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.712 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:12.712 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:12.713 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:12.713 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:12.714 DEBUG:boto:Method: HEAD 2022-09-12T20:12:12.714 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:12.715 DEBUG:boto:Data: 2022-09-12T20:12:12.715 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.716 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.716 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.717 DEBUG:boto:Params: {} 2022-09-12T20:12:12.717 DEBUG:boto:Token: None 2022-09-12T20:12:12.718 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:12.719 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G6O5gBC/XKYZFFt3k6eTiByy9KM= 2022-09-12T20:12:12.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G6O5gBC/XKYZFFt3k6eTiByy9KM=', 'Content-Length': '0'} 2022-09-12T20:12:12.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a99970b68b0ab368-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:12:12.722 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:12.723 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:12.723 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:12.724 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:12.724 DEBUG:boto:Method: GET 2022-09-12T20:12:12.725 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:12.725 DEBUG:boto:Data: 2022-09-12T20:12:12.726 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.726 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.727 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.728 DEBUG:boto:Params: {} 2022-09-12T20:12:12.728 DEBUG:boto:Token: None 2022-09-12T20:12:12.729 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:12.729 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aAcYZwaPY9Izy5dWheANcDdYP6U= 2022-09-12T20:12:12.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aAcYZwaPY9Izy5dWheANcDdYP6U=', 'Content-Length': '0'} 2022-09-12T20:12:12.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e364e1485a77010-00631f929c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.774 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:12.775 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:12.776 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:12.776 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:12.777 DEBUG:boto:Method: GET 2022-09-12T20:12:12.777 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:12.778 DEBUG:boto:Data: 2022-09-12T20:12:12.779 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.779 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.780 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.780 DEBUG:boto:Params: {} 2022-09-12T20:12:12.781 DEBUG:boto:Token: None 2022-09-12T20:12:12.781 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:12.782 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aAcYZwaPY9Izy5dWheANcDdYP6U= 2022-09-12T20:12:12.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aAcYZwaPY9Izy5dWheANcDdYP6U=', 'Content-Length': '0'} 2022-09-12T20:12:12.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fe02d01edd56aec-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.786 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:12.787 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:12.787 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:12:12.788 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:12:12.788 DEBUG:boto:Method: GET 2022-09-12T20:12:12.789 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:12:12.789 DEBUG:boto:Data: 2022-09-12T20:12:12.790 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.790 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.791 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.791 DEBUG:boto:Params: {} 2022-09-12T20:12:12.792 DEBUG:boto:Token: None 2022-09-12T20:12:12.793 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/ 2022-09-12T20:12:12.793 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA= 2022-09-12T20:12:12.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA=', 'Content-Length': '0'} 2022-09-12T20:12:12.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1686bd7f7621bb9-00631f929c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.798 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:12.799 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:12.799 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:12.800 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:12:12.800 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:12:12.801 DEBUG:boto:Method: GET 2022-09-12T20:12:12.801 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:12:12.802 DEBUG:boto:Data: 2022-09-12T20:12:12.802 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.803 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.803 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.804 DEBUG:boto:Params: {} 2022-09-12T20:12:12.804 DEBUG:boto:Token: None 2022-09-12T20:12:12.805 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/ 2022-09-12T20:12:12.806 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA= 2022-09-12T20:12:12.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA=', 'Content-Length': '0'} 2022-09-12T20:12:12.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000172ec8ef2458fd29-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.811 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:12.812 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:12.812 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:12.813 DEBUG:boto:Method: HEAD 2022-09-12T20:12:12.813 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:12:12.814 DEBUG:boto:Data: 2022-09-12T20:12:12.814 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.815 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.815 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.816 DEBUG:boto:Params: {} 2022-09-12T20:12:12.816 DEBUG:boto:Token: None 2022-09-12T20:12:12.817 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj116 2022-09-12T20:12:12.817 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kZPdkZYW4hXc8kCHRgteQB6y0fc= 2022-09-12T20:12:12.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kZPdkZYW4hXc8kCHRgteQB6y0fc=', 'Content-Length': '0'} 2022-09-12T20:12:12.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000479cf072026b343-00631f929c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.821 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:12.821 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:12.822 DEBUG:boto:Method: HEAD 2022-09-12T20:12:12.822 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:12:12.823 DEBUG:boto:Data: 2022-09-12T20:12:12.823 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.824 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.824 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.825 DEBUG:boto:Params: {} 2022-09-12T20:12:12.825 DEBUG:boto:Token: None 2022-09-12T20:12:12.826 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj116 2022-09-12T20:12:12.827 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kZPdkZYW4hXc8kCHRgteQB6y0fc= 2022-09-12T20:12:12.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kZPdkZYW4hXc8kCHRgteQB6y0fc=', 'Content-Length': '0'} 2022-09-12T20:12:12.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fbcb1753b5bd543-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:12:12.831 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:12.831 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:12.832 DEBUG:boto:Method: GET 2022-09-12T20:12:12.832 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:12:12.833 DEBUG:boto:Data: 2022-09-12T20:12:12.833 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.834 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.835 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.835 DEBUG:boto:Params: {} 2022-09-12T20:12:12.836 DEBUG:boto:Token: None 2022-09-12T20:12:12.837 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj116 2022-09-12T20:12:12.837 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u1BOy57Ld3bhXqyZm4LG6r1s4tY= 2022-09-12T20:12:12.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u1BOy57Ld3bhXqyZm4LG6r1s4tY=', 'Content-Length': '0'} 2022-09-12T20:12:12.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad919a48ae0c71ae-00631f929c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.841 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:12.841 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:12.842 DEBUG:boto:Method: GET 2022-09-12T20:12:12.842 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:12:12.843 DEBUG:boto:Data: 2022-09-12T20:12:12.843 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.844 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.845 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.845 DEBUG:boto:Params: {} 2022-09-12T20:12:12.846 DEBUG:boto:Token: None 2022-09-12T20:12:12.846 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj116 2022-09-12T20:12:12.847 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u1BOy57Ld3bhXqyZm4LG6r1s4tY= 2022-09-12T20:12:12.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u1BOy57Ld3bhXqyZm4LG6r1s4tY=', 'Content-Length': '0'} 2022-09-12T20:12:12.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000692f66a4293b4b8a-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.894 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:12.895 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:12.895 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:12:12.896 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:12:12.896 DEBUG:boto:Method: GET 2022-09-12T20:12:12.897 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:12:12.898 DEBUG:boto:Data: 2022-09-12T20:12:12.898 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.899 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.899 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.900 DEBUG:boto:Params: {} 2022-09-12T20:12:12.900 DEBUG:boto:Token: None 2022-09-12T20:12:12.901 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/ 2022-09-12T20:12:12.902 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA= 2022-09-12T20:12:12.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA=', 'Content-Length': '0'} 2022-09-12T20:12:12.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8d4f050648a7a24-00631f929c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.906 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:12.907 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:12.908 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:12.908 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:12:12.909 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:12:12.909 DEBUG:boto:Method: GET 2022-09-12T20:12:12.910 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:12:12.910 DEBUG:boto:Data: 2022-09-12T20:12:12.911 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.911 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.912 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.912 DEBUG:boto:Params: {} 2022-09-12T20:12:12.913 DEBUG:boto:Token: None 2022-09-12T20:12:12.914 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/ 2022-09-12T20:12:12.914 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA= 2022-09-12T20:12:12.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6cqa7kRAinLsgn8a4OlSy8bkDJA=', 'Content-Length': '0'} 2022-09-12T20:12:12.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be633ee151e06229-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.920 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:12.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:12:12.921 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:12.922 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:12.922 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:12.923 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:12.923 DEBUG:boto:Method: GET 2022-09-12T20:12:12.924 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:12.925 DEBUG:boto:Data: 2022-09-12T20:12:12.925 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.926 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.926 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.927 DEBUG:boto:Params: {} 2022-09-12T20:12:12.927 DEBUG:boto:Token: None 2022-09-12T20:12:12.928 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:12.929 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DGih9Z1fdzoFTFYM2ORDNn8W9EU= 2022-09-12T20:12:12.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DGih9Z1fdzoFTFYM2ORDNn8W9EU=', 'Content-Length': '0'} 2022-09-12T20:12:12.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000469a44fdd9e6f8ca-00631f929c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.932 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:12.933 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:12.933 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:12.934 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:12.935 DEBUG:boto:Method: GET 2022-09-12T20:12:12.935 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:12.936 DEBUG:boto:Data: 2022-09-12T20:12:12.936 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.937 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.937 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.938 DEBUG:boto:Params: {} 2022-09-12T20:12:12.938 DEBUG:boto:Token: None 2022-09-12T20:12:12.939 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:12.940 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DGih9Z1fdzoFTFYM2ORDNn8W9EU= 2022-09-12T20:12:12.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DGih9Z1fdzoFTFYM2ORDNn8W9EU=', 'Content-Length': '0'} 2022-09-12T20:12:12.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000031a004db3ac03bb-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.943 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:12.944 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:12.944 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:12.945 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:12.945 DEBUG:boto:Method: HEAD 2022-09-12T20:12:12.946 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:12.946 DEBUG:boto:Data: 2022-09-12T20:12:12.947 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.948 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.948 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.949 DEBUG:boto:Params: {} 2022-09-12T20:12:12.949 DEBUG:boto:Token: None 2022-09-12T20:12:12.950 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:12.950 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bt7UuJi6cKW/sJyi7hYu5ZhsGOE= 2022-09-12T20:12:12.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bt7UuJi6cKW/sJyi7hYu5ZhsGOE=', 'Content-Length': '0'} 2022-09-12T20:12:12.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070b5c1fd16d045fc-00631f929c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.954 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:12.954 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:12.955 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:12.955 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:12.956 DEBUG:boto:Method: HEAD 2022-09-12T20:12:12.956 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:12.957 DEBUG:boto:Data: 2022-09-12T20:12:12.957 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.958 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.958 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.959 DEBUG:boto:Params: {} 2022-09-12T20:12:12.959 DEBUG:boto:Token: None 2022-09-12T20:12:12.960 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:12.960 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bt7UuJi6cKW/sJyi7hYu5ZhsGOE= 2022-09-12T20:12:12.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bt7UuJi6cKW/sJyi7hYu5ZhsGOE=', 'Content-Length': '0'} 2022-09-12T20:12:12.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f764600f602f82e0-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:12:12.965 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:12.965 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:12.966 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:12.966 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:12.967 DEBUG:boto:Method: GET 2022-09-12T20:12:12.967 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:12.968 DEBUG:boto:Data: 2022-09-12T20:12:12.968 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.969 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:12.969 DEBUG:boto:Port: 8001 2022-09-12T20:12:12.970 DEBUG:boto:Params: {} 2022-09-12T20:12:12.970 DEBUG:boto:Token: None 2022-09-12T20:12:12.971 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:12.972 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DGih9Z1fdzoFTFYM2ORDNn8W9EU= 2022-09-12T20:12:12.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DGih9Z1fdzoFTFYM2ORDNn8W9EU=', 'Content-Length': '0'} 2022-09-12T20:12:12.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050ccd1732a63bdb7-00631f929c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:12.975 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:12.976 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:12.976 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:12.977 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:12.977 DEBUG:boto:Method: GET 2022-09-12T20:12:12.978 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:12.978 DEBUG:boto:Data: 2022-09-12T20:12:12.979 DEBUG:boto:Headers: {} 2022-09-12T20:12:12.979 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:12.980 DEBUG:boto:Port: 8000 2022-09-12T20:12:12.980 DEBUG:boto:Params: {} 2022-09-12T20:12:12.981 DEBUG:boto:Token: None 2022-09-12T20:12:12.982 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:12 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:12.982 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DGih9Z1fdzoFTFYM2ORDNn8W9EU= 2022-09-12T20:12:12.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DGih9Z1fdzoFTFYM2ORDNn8W9EU=', 'Content-Length': '0'} 2022-09-12T20:12:12.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e9be87d010b88f4-00631f929c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.026 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:13.027 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:13.027 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:12:13.028 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:12:13.028 DEBUG:boto:Method: GET 2022-09-12T20:12:13.029 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:12:13.029 DEBUG:boto:Data: 2022-09-12T20:12:13.030 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.030 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.031 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.031 DEBUG:boto:Params: {} 2022-09-12T20:12:13.032 DEBUG:boto:Token: None 2022-09-12T20:12:13.033 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/ 2022-09-12T20:12:13.033 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048= 2022-09-12T20:12:13.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048=', 'Content-Length': '0'} 2022-09-12T20:12:13.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ec073c41edfc7f7-00631f929d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.037 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:13.038 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:13.039 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:13.039 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:12:13.040 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:12:13.040 DEBUG:boto:Method: GET 2022-09-12T20:12:13.041 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:12:13.041 DEBUG:boto:Data: 2022-09-12T20:12:13.042 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.043 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.043 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.044 DEBUG:boto:Params: {} 2022-09-12T20:12:13.044 DEBUG:boto:Token: None 2022-09-12T20:12:13.045 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/ 2022-09-12T20:12:13.045 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048= 2022-09-12T20:12:13.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048=', 'Content-Length': '0'} 2022-09-12T20:12:13.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0517a0fc9068cbc-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.051 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:13.052 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:13.052 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:13.053 DEBUG:boto:Method: HEAD 2022-09-12T20:12:13.053 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:12:13.054 DEBUG:boto:Data: 2022-09-12T20:12:13.054 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.055 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.055 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.056 DEBUG:boto:Params: {} 2022-09-12T20:12:13.056 DEBUG:boto:Token: None 2022-09-12T20:12:13.057 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj145 2022-09-12T20:12:13.058 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rvhg+JeuUa0P0+cIXHsc9xisozA= 2022-09-12T20:12:13.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rvhg+JeuUa0P0+cIXHsc9xisozA=', 'Content-Length': '0'} 2022-09-12T20:12:13.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060310ba367237106-00631f929d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.061 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:13.061 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:13.062 DEBUG:boto:Method: HEAD 2022-09-12T20:12:13.062 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:12:13.063 DEBUG:boto:Data: 2022-09-12T20:12:13.063 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.064 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.064 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.065 DEBUG:boto:Params: {} 2022-09-12T20:12:13.065 DEBUG:boto:Token: None 2022-09-12T20:12:13.066 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj145 2022-09-12T20:12:13.067 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rvhg+JeuUa0P0+cIXHsc9xisozA= 2022-09-12T20:12:13.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rvhg+JeuUa0P0+cIXHsc9xisozA=', 'Content-Length': '0'} 2022-09-12T20:12:13.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003415f37ec9f3186f-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:12:13.071 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:13.071 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:13.072 DEBUG:boto:Method: GET 2022-09-12T20:12:13.072 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:12:13.073 DEBUG:boto:Data: 2022-09-12T20:12:13.073 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.074 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.074 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.075 DEBUG:boto:Params: {} 2022-09-12T20:12:13.075 DEBUG:boto:Token: None 2022-09-12T20:12:13.076 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj145 2022-09-12T20:12:13.077 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AElNKPzVntYmVyo/sBqSST7Z+Ag= 2022-09-12T20:12:13.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AElNKPzVntYmVyo/sBqSST7Z+Ag=', 'Content-Length': '0'} 2022-09-12T20:12:13.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a6903085ee99d4a-00631f929d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.080 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:13.081 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:13.081 DEBUG:boto:Method: GET 2022-09-12T20:12:13.082 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:12:13.082 DEBUG:boto:Data: 2022-09-12T20:12:13.083 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.083 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.084 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.084 DEBUG:boto:Params: {} 2022-09-12T20:12:13.085 DEBUG:boto:Token: None 2022-09-12T20:12:13.085 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj145 2022-09-12T20:12:13.086 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AElNKPzVntYmVyo/sBqSST7Z+Ag= 2022-09-12T20:12:13.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AElNKPzVntYmVyo/sBqSST7Z+Ag=', 'Content-Length': '0'} 2022-09-12T20:12:13.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006ace5396746f79e-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.090 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:13.090 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:13.091 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:12:13.091 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:12:13.092 DEBUG:boto:Method: GET 2022-09-12T20:12:13.092 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:12:13.093 DEBUG:boto:Data: 2022-09-12T20:12:13.093 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.094 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.094 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.095 DEBUG:boto:Params: {} 2022-09-12T20:12:13.095 DEBUG:boto:Token: None 2022-09-12T20:12:13.096 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/ 2022-09-12T20:12:13.097 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048= 2022-09-12T20:12:13.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048=', 'Content-Length': '0'} 2022-09-12T20:12:13.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b36ff3ee2855f8d-00631f929d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.101 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:13.102 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:13.102 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:13.103 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:12:13.103 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:12:13.104 DEBUG:boto:Method: GET 2022-09-12T20:12:13.104 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:12:13.105 DEBUG:boto:Data: 2022-09-12T20:12:13.105 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.106 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.106 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.107 DEBUG:boto:Params: {} 2022-09-12T20:12:13.107 DEBUG:boto:Token: None 2022-09-12T20:12:13.108 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/ 2022-09-12T20:12:13.109 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048= 2022-09-12T20:12:13.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048=', 'Content-Length': '0'} 2022-09-12T20:12:13.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c13d6cf34a852df1-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.114 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:13.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:12:13.116 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:13.116 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:13.117 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:13.117 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:13.118 DEBUG:boto:Method: GET 2022-09-12T20:12:13.118 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:13.119 DEBUG:boto:Data: 2022-09-12T20:12:13.119 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.120 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.120 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.121 DEBUG:boto:Params: {} 2022-09-12T20:12:13.121 DEBUG:boto:Token: None 2022-09-12T20:12:13.122 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:13.123 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:98cWd2ubcBQzLypAkaSFnAiAbts= 2022-09-12T20:12:13.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:98cWd2ubcBQzLypAkaSFnAiAbts=', 'Content-Length': '0'} 2022-09-12T20:12:13.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a46adf3b37226de4-00631f929d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.126 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:13.127 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:13.127 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:13.128 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:13.128 DEBUG:boto:Method: GET 2022-09-12T20:12:13.129 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:13.129 DEBUG:boto:Data: 2022-09-12T20:12:13.130 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.131 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.131 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.132 DEBUG:boto:Params: {} 2022-09-12T20:12:13.133 DEBUG:boto:Token: None 2022-09-12T20:12:13.133 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:13.134 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:98cWd2ubcBQzLypAkaSFnAiAbts= 2022-09-12T20:12:13.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:98cWd2ubcBQzLypAkaSFnAiAbts=', 'Content-Length': '0'} 2022-09-12T20:12:13.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000850232f192b6a248-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.137 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:13.138 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:13.138 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:13.139 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:13.139 DEBUG:boto:Method: HEAD 2022-09-12T20:12:13.140 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:13.140 DEBUG:boto:Data: 2022-09-12T20:12:13.141 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.141 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.142 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.142 DEBUG:boto:Params: {} 2022-09-12T20:12:13.143 DEBUG:boto:Token: None 2022-09-12T20:12:13.143 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:13.144 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nynT493XIhv0gHjJBlOWhLKojH0= 2022-09-12T20:12:13.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nynT493XIhv0gHjJBlOWhLKojH0=', 'Content-Length': '0'} 2022-09-12T20:12:13.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007c46245bd0e4234-00631f929d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.148 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:13.148 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:13.149 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:13.149 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:13.150 DEBUG:boto:Method: HEAD 2022-09-12T20:12:13.150 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:13.151 DEBUG:boto:Data: 2022-09-12T20:12:13.151 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.152 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.152 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.153 DEBUG:boto:Params: {} 2022-09-12T20:12:13.153 DEBUG:boto:Token: None 2022-09-12T20:12:13.154 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:13.154 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nynT493XIhv0gHjJBlOWhLKojH0= 2022-09-12T20:12:13.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nynT493XIhv0gHjJBlOWhLKojH0=', 'Content-Length': '0'} 2022-09-12T20:12:13.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000302a12baf3bfcfc1-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:12:13.158 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:13.159 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:13.159 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:13.160 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:13.161 DEBUG:boto:Method: GET 2022-09-12T20:12:13.161 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:13.162 DEBUG:boto:Data: 2022-09-12T20:12:13.162 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.163 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.163 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.164 DEBUG:boto:Params: {} 2022-09-12T20:12:13.164 DEBUG:boto:Token: None 2022-09-12T20:12:13.165 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:13.166 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:98cWd2ubcBQzLypAkaSFnAiAbts= 2022-09-12T20:12:13.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:98cWd2ubcBQzLypAkaSFnAiAbts=', 'Content-Length': '0'} 2022-09-12T20:12:13.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe9d9795be236aaa-00631f929d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.169 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:13.170 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:13.170 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:13.171 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:13.171 DEBUG:boto:Method: GET 2022-09-12T20:12:13.172 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:13.172 DEBUG:boto:Data: 2022-09-12T20:12:13.173 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.173 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.174 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.174 DEBUG:boto:Params: {} 2022-09-12T20:12:13.175 DEBUG:boto:Token: None 2022-09-12T20:12:13.176 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:13.176 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:98cWd2ubcBQzLypAkaSFnAiAbts= 2022-09-12T20:12:13.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:98cWd2ubcBQzLypAkaSFnAiAbts=', 'Content-Length': '0'} 2022-09-12T20:12:13.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074dd54a88a4e7252-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.180 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:13.180 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:13.181 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:12:13.182 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:12:13.182 DEBUG:boto:Method: GET 2022-09-12T20:12:13.183 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:12:13.183 DEBUG:boto:Data: 2022-09-12T20:12:13.184 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.184 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.185 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.185 DEBUG:boto:Params: {} 2022-09-12T20:12:13.186 DEBUG:boto:Token: None 2022-09-12T20:12:13.186 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/ 2022-09-12T20:12:13.187 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048= 2022-09-12T20:12:13.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048=', 'Content-Length': '0'} 2022-09-12T20:12:13.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d8373984c28e0b1-00631f929d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.193 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:13.194 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:13.194 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:13.195 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:12:13.195 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:12:13.196 DEBUG:boto:Method: GET 2022-09-12T20:12:13.196 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:12:13.197 DEBUG:boto:Data: 2022-09-12T20:12:13.197 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.198 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.199 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.199 DEBUG:boto:Params: {} 2022-09-12T20:12:13.200 DEBUG:boto:Token: None 2022-09-12T20:12:13.200 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/ 2022-09-12T20:12:13.201 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048= 2022-09-12T20:12:13.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048=', 'Content-Length': '0'} 2022-09-12T20:12:13.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fdb1a8c33edaf91-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.246 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:13.247 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:13.247 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:13.248 DEBUG:boto:Method: HEAD 2022-09-12T20:12:13.248 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:12:13.249 DEBUG:boto:Data: 2022-09-12T20:12:13.249 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.250 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.250 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.251 DEBUG:boto:Params: {} 2022-09-12T20:12:13.251 DEBUG:boto:Token: None 2022-09-12T20:12:13.252 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj17 2022-09-12T20:12:13.253 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5cSU+tXu0N3MOKqrTKNZXrOuB9Y= 2022-09-12T20:12:13.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5cSU+tXu0N3MOKqrTKNZXrOuB9Y=', 'Content-Length': '0'} 2022-09-12T20:12:13.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d495713bec27d9e3-00631f929d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.256 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:13.257 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:13.257 DEBUG:boto:Method: HEAD 2022-09-12T20:12:13.258 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:12:13.258 DEBUG:boto:Data: 2022-09-12T20:12:13.259 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.259 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.260 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.260 DEBUG:boto:Params: {} 2022-09-12T20:12:13.261 DEBUG:boto:Token: None 2022-09-12T20:12:13.261 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj17 2022-09-12T20:12:13.262 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5cSU+tXu0N3MOKqrTKNZXrOuB9Y= 2022-09-12T20:12:13.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5cSU+tXu0N3MOKqrTKNZXrOuB9Y=', 'Content-Length': '0'} 2022-09-12T20:12:13.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eff5afbe95f710d6-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:12:13.266 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:13.267 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:13.268 DEBUG:boto:Method: GET 2022-09-12T20:12:13.268 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:12:13.269 DEBUG:boto:Data: 2022-09-12T20:12:13.269 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.270 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.270 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.271 DEBUG:boto:Params: {} 2022-09-12T20:12:13.271 DEBUG:boto:Token: None 2022-09-12T20:12:13.272 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj17 2022-09-12T20:12:13.273 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YuzJiOZab+gZDdXpWjM7II/elI4= 2022-09-12T20:12:13.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YuzJiOZab+gZDdXpWjM7II/elI4=', 'Content-Length': '0'} 2022-09-12T20:12:13.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009468cbdb069e6213-00631f929d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.276 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:13.277 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:13.277 DEBUG:boto:Method: GET 2022-09-12T20:12:13.278 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:12:13.278 DEBUG:boto:Data: 2022-09-12T20:12:13.279 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.279 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.280 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.280 DEBUG:boto:Params: {} 2022-09-12T20:12:13.281 DEBUG:boto:Token: None 2022-09-12T20:12:13.281 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj17 2022-09-12T20:12:13.283 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YuzJiOZab+gZDdXpWjM7II/elI4= 2022-09-12T20:12:13.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YuzJiOZab+gZDdXpWjM7II/elI4=', 'Content-Length': '0'} 2022-09-12T20:12:13.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbf969e3974b67f9-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.286 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:13.287 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:13.287 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:12:13.288 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:12:13.289 DEBUG:boto:Method: GET 2022-09-12T20:12:13.289 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:12:13.290 DEBUG:boto:Data: 2022-09-12T20:12:13.290 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.291 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.291 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.292 DEBUG:boto:Params: {} 2022-09-12T20:12:13.292 DEBUG:boto:Token: None 2022-09-12T20:12:13.293 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/ 2022-09-12T20:12:13.293 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048= 2022-09-12T20:12:13.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048=', 'Content-Length': '0'} 2022-09-12T20:12:13.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000251996ebd1a0cbec-00631f929d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.298 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:13.299 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:13.299 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:13.300 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:12:13.300 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:12:13.301 DEBUG:boto:Method: GET 2022-09-12T20:12:13.301 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:12:13.302 DEBUG:boto:Data: 2022-09-12T20:12:13.303 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.304 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.304 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.305 DEBUG:boto:Params: {} 2022-09-12T20:12:13.305 DEBUG:boto:Token: None 2022-09-12T20:12:13.306 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/ 2022-09-12T20:12:13.307 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048= 2022-09-12T20:12:13.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048=', 'Content-Length': '0'} 2022-09-12T20:12:13.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008300bd0dbef6e051-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.312 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:13.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:12:13.314 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:13.314 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:13.315 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:13.315 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:13.316 DEBUG:boto:Method: GET 2022-09-12T20:12:13.316 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:13.317 DEBUG:boto:Data: 2022-09-12T20:12:13.317 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.318 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.318 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.319 DEBUG:boto:Params: {} 2022-09-12T20:12:13.319 DEBUG:boto:Token: None 2022-09-12T20:12:13.320 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:13.321 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KkVnoaqMcjoS07lVhJs0IvD0Zok= 2022-09-12T20:12:13.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KkVnoaqMcjoS07lVhJs0IvD0Zok=', 'Content-Length': '0'} 2022-09-12T20:12:13.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066934fbf8865bd50-00631f929d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.324 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:13.325 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:13.325 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:13.326 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:13.326 DEBUG:boto:Method: GET 2022-09-12T20:12:13.327 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:13.327 DEBUG:boto:Data: 2022-09-12T20:12:13.328 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.328 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.329 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.329 DEBUG:boto:Params: {} 2022-09-12T20:12:13.330 DEBUG:boto:Token: None 2022-09-12T20:12:13.331 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:13.331 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KkVnoaqMcjoS07lVhJs0IvD0Zok= 2022-09-12T20:12:13.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KkVnoaqMcjoS07lVhJs0IvD0Zok=', 'Content-Length': '0'} 2022-09-12T20:12:13.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5dbfd1f6421b575-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.335 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:13.336 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:13.336 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:13.337 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:13.337 DEBUG:boto:Method: HEAD 2022-09-12T20:12:13.338 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:13.338 DEBUG:boto:Data: 2022-09-12T20:12:13.339 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.339 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.340 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.340 DEBUG:boto:Params: {} 2022-09-12T20:12:13.341 DEBUG:boto:Token: None 2022-09-12T20:12:13.341 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:13.342 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5IOiQdbncmoLqjdcivGw3PPHmlw= 2022-09-12T20:12:13.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5IOiQdbncmoLqjdcivGw3PPHmlw=', 'Content-Length': '0'} 2022-09-12T20:12:13.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a4c53fde90c0084-00631f929d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.345 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:13.346 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:13.346 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:13.347 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:13.347 DEBUG:boto:Method: HEAD 2022-09-12T20:12:13.348 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:13.348 DEBUG:boto:Data: 2022-09-12T20:12:13.349 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.349 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.350 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.351 DEBUG:boto:Params: {} 2022-09-12T20:12:13.352 DEBUG:boto:Token: None 2022-09-12T20:12:13.352 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:13.353 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5IOiQdbncmoLqjdcivGw3PPHmlw= 2022-09-12T20:12:13.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5IOiQdbncmoLqjdcivGw3PPHmlw=', 'Content-Length': '0'} 2022-09-12T20:12:13.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000389523002c061cfb-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:12:13.357 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:13.358 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:13.358 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:13.359 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:13.359 DEBUG:boto:Method: GET 2022-09-12T20:12:13.360 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:13.360 DEBUG:boto:Data: 2022-09-12T20:12:13.361 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.361 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.362 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.362 DEBUG:boto:Params: {} 2022-09-12T20:12:13.363 DEBUG:boto:Token: None 2022-09-12T20:12:13.364 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:13.364 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KkVnoaqMcjoS07lVhJs0IvD0Zok= 2022-09-12T20:12:13.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KkVnoaqMcjoS07lVhJs0IvD0Zok=', 'Content-Length': '0'} 2022-09-12T20:12:13.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076b69c0e42a649b4-00631f929d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.368 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:13.368 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:13.369 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:13.369 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:13.370 DEBUG:boto:Method: GET 2022-09-12T20:12:13.370 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:13.371 DEBUG:boto:Data: 2022-09-12T20:12:13.372 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.372 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.373 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.373 DEBUG:boto:Params: {} 2022-09-12T20:12:13.374 DEBUG:boto:Token: None 2022-09-12T20:12:13.374 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:13.375 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KkVnoaqMcjoS07lVhJs0IvD0Zok= 2022-09-12T20:12:13.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KkVnoaqMcjoS07lVhJs0IvD0Zok=', 'Content-Length': '0'} 2022-09-12T20:12:13.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000292301ecce1b415a-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.422 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:13.423 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:13.423 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:12:13.424 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:12:13.424 DEBUG:boto:Method: GET 2022-09-12T20:12:13.425 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:12:13.425 DEBUG:boto:Data: 2022-09-12T20:12:13.426 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.426 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.427 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.427 DEBUG:boto:Params: {} 2022-09-12T20:12:13.428 DEBUG:boto:Token: None 2022-09-12T20:12:13.429 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/ 2022-09-12T20:12:13.429 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048= 2022-09-12T20:12:13.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048=', 'Content-Length': '0'} 2022-09-12T20:12:13.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf88013f15f6fe92-00631f929d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.433 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:13.434 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:13.435 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:13.436 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:12:13.436 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:12:13.437 DEBUG:boto:Method: GET 2022-09-12T20:12:13.437 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:12:13.438 DEBUG:boto:Data: 2022-09-12T20:12:13.438 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.439 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.439 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.440 DEBUG:boto:Params: {} 2022-09-12T20:12:13.440 DEBUG:boto:Token: None 2022-09-12T20:12:13.441 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/ 2022-09-12T20:12:13.441 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048= 2022-09-12T20:12:13.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048=', 'Content-Length': '0'} 2022-09-12T20:12:13.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3268fff64d7d13f-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.447 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:13.447 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:13.448 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:13.448 DEBUG:boto:Method: HEAD 2022-09-12T20:12:13.449 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:12:13.449 DEBUG:boto:Data: 2022-09-12T20:12:13.450 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.450 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.451 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.451 DEBUG:boto:Params: {} 2022-09-12T20:12:13.452 DEBUG:boto:Token: None 2022-09-12T20:12:13.453 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj19 2022-09-12T20:12:13.453 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Wvq2/hx/TcJr38+gqDiZL+Y0UgA= 2022-09-12T20:12:13.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Wvq2/hx/TcJr38+gqDiZL+Y0UgA=', 'Content-Length': '0'} 2022-09-12T20:12:13.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007af5458d8d7f49c4-00631f929d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.456 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:13.457 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:13.457 DEBUG:boto:Method: HEAD 2022-09-12T20:12:13.458 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:12:13.458 DEBUG:boto:Data: 2022-09-12T20:12:13.459 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.459 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.460 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.460 DEBUG:boto:Params: {} 2022-09-12T20:12:13.461 DEBUG:boto:Token: None 2022-09-12T20:12:13.461 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj19 2022-09-12T20:12:13.462 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Wvq2/hx/TcJr38+gqDiZL+Y0UgA= 2022-09-12T20:12:13.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Wvq2/hx/TcJr38+gqDiZL+Y0UgA=', 'Content-Length': '0'} 2022-09-12T20:12:13.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000892c05592d786ef8-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:12:13.466 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:13.466 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:13.467 DEBUG:boto:Method: GET 2022-09-12T20:12:13.468 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:12:13.468 DEBUG:boto:Data: 2022-09-12T20:12:13.469 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.469 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.470 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.471 DEBUG:boto:Params: {} 2022-09-12T20:12:13.471 DEBUG:boto:Token: None 2022-09-12T20:12:13.472 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj19 2022-09-12T20:12:13.472 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A/d0V/ciT+Xt0eMUXWP1OaChUg0= 2022-09-12T20:12:13.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A/d0V/ciT+Xt0eMUXWP1OaChUg0=', 'Content-Length': '0'} 2022-09-12T20:12:13.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5aad7a66fc057c1-00631f929d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.476 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:13.477 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:13.477 DEBUG:boto:Method: GET 2022-09-12T20:12:13.478 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:12:13.478 DEBUG:boto:Data: 2022-09-12T20:12:13.479 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.479 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.480 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.480 DEBUG:boto:Params: {} 2022-09-12T20:12:13.481 DEBUG:boto:Token: None 2022-09-12T20:12:13.482 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj19 2022-09-12T20:12:13.482 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A/d0V/ciT+Xt0eMUXWP1OaChUg0= 2022-09-12T20:12:13.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A/d0V/ciT+Xt0eMUXWP1OaChUg0=', 'Content-Length': '0'} 2022-09-12T20:12:13.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008494874ad42c8e91-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.486 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:13.486 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:13.487 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:12:13.488 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:12:13.488 DEBUG:boto:Method: GET 2022-09-12T20:12:13.489 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:12:13.490 DEBUG:boto:Data: 2022-09-12T20:12:13.490 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.491 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.491 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.492 DEBUG:boto:Params: {} 2022-09-12T20:12:13.492 DEBUG:boto:Token: None 2022-09-12T20:12:13.493 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/ 2022-09-12T20:12:13.493 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048= 2022-09-12T20:12:13.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048=', 'Content-Length': '0'} 2022-09-12T20:12:13.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086fae918e9d18239-00631f929d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.497 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:13.498 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:13.499 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:13.499 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:12:13.500 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:12:13.500 DEBUG:boto:Method: GET 2022-09-12T20:12:13.501 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:12:13.501 DEBUG:boto:Data: 2022-09-12T20:12:13.502 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.503 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.503 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.504 DEBUG:boto:Params: {} 2022-09-12T20:12:13.504 DEBUG:boto:Token: None 2022-09-12T20:12:13.505 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/ 2022-09-12T20:12:13.505 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048= 2022-09-12T20:12:13.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048=', 'Content-Length': '0'} 2022-09-12T20:12:13.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000464dc1a19c90ebe4-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.511 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:13.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:12:13.512 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:13.513 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:13.513 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:13.514 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:13.514 DEBUG:boto:Method: GET 2022-09-12T20:12:13.515 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:13.515 DEBUG:boto:Data: 2022-09-12T20:12:13.516 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.516 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.517 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.517 DEBUG:boto:Params: {} 2022-09-12T20:12:13.518 DEBUG:boto:Token: None 2022-09-12T20:12:13.519 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:13.519 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:djV3Vphqx3QRRB/KS8JfpK9f4Qs= 2022-09-12T20:12:13.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:djV3Vphqx3QRRB/KS8JfpK9f4Qs=', 'Content-Length': '0'} 2022-09-12T20:12:13.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000008f40c4f78d8885-00631f929d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.524 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:13.525 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:13.527 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:13.528 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:13.529 DEBUG:boto:Method: GET 2022-09-12T20:12:13.530 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:13.530 DEBUG:boto:Data: 2022-09-12T20:12:13.531 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.532 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.533 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.533 DEBUG:boto:Params: {} 2022-09-12T20:12:13.534 DEBUG:boto:Token: None 2022-09-12T20:12:13.535 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:13.536 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:djV3Vphqx3QRRB/KS8JfpK9f4Qs= 2022-09-12T20:12:13.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:djV3Vphqx3QRRB/KS8JfpK9f4Qs=', 'Content-Length': '0'} 2022-09-12T20:12:13.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000991afc6690f7d1b7-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.582 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:13.583 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:13.583 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:13.584 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:13.584 DEBUG:boto:Method: HEAD 2022-09-12T20:12:13.585 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:13.585 DEBUG:boto:Data: 2022-09-12T20:12:13.586 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.586 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.587 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.587 DEBUG:boto:Params: {} 2022-09-12T20:12:13.588 DEBUG:boto:Token: None 2022-09-12T20:12:13.589 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:13.589 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZNkCPq4Az8eL4Ho9JHcB17USV8k= 2022-09-12T20:12:13.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZNkCPq4Az8eL4Ho9JHcB17USV8k=', 'Content-Length': '0'} 2022-09-12T20:12:13.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fea15260a7cd4a5c-00631f929d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.593 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:13.593 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:13.594 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:13.594 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:13.595 DEBUG:boto:Method: HEAD 2022-09-12T20:12:13.595 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:13.596 DEBUG:boto:Data: 2022-09-12T20:12:13.596 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.597 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.597 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.598 DEBUG:boto:Params: {} 2022-09-12T20:12:13.599 DEBUG:boto:Token: None 2022-09-12T20:12:13.599 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:13.600 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZNkCPq4Az8eL4Ho9JHcB17USV8k= 2022-09-12T20:12:13.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZNkCPq4Az8eL4Ho9JHcB17USV8k=', 'Content-Length': '0'} 2022-09-12T20:12:13.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042c43d273805ed5a-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:12:13.604 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:13.604 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:13.605 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:13.605 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:13.606 DEBUG:boto:Method: GET 2022-09-12T20:12:13.606 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:13.607 DEBUG:boto:Data: 2022-09-12T20:12:13.607 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.608 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.609 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.610 DEBUG:boto:Params: {} 2022-09-12T20:12:13.611 DEBUG:boto:Token: None 2022-09-12T20:12:13.613 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:13.614 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:djV3Vphqx3QRRB/KS8JfpK9f4Qs= 2022-09-12T20:12:13.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:djV3Vphqx3QRRB/KS8JfpK9f4Qs=', 'Content-Length': '0'} 2022-09-12T20:12:13.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e074468661ba028-00631f929d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.619 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:13.620 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:13.620 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:13.621 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:13.621 DEBUG:boto:Method: GET 2022-09-12T20:12:13.622 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:13.623 DEBUG:boto:Data: 2022-09-12T20:12:13.623 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.624 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.624 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.625 DEBUG:boto:Params: {} 2022-09-12T20:12:13.625 DEBUG:boto:Token: None 2022-09-12T20:12:13.626 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:13.627 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:djV3Vphqx3QRRB/KS8JfpK9f4Qs= 2022-09-12T20:12:13.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:djV3Vphqx3QRRB/KS8JfpK9f4Qs=', 'Content-Length': '0'} 2022-09-12T20:12:13.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016a024777cf409e6-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.630 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:13.631 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:13.632 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:12:13.632 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:12:13.633 DEBUG:boto:Method: GET 2022-09-12T20:12:13.633 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:12:13.634 DEBUG:boto:Data: 2022-09-12T20:12:13.634 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.635 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.635 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.636 DEBUG:boto:Params: {} 2022-09-12T20:12:13.637 DEBUG:boto:Token: None 2022-09-12T20:12:13.637 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/ 2022-09-12T20:12:13.638 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048= 2022-09-12T20:12:13.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048=', 'Content-Length': '0'} 2022-09-12T20:12:13.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fae76228ee212e59-00631f929d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.682 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:13.683 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:13.684 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:13.684 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:12:13.685 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:12:13.685 DEBUG:boto:Method: GET 2022-09-12T20:12:13.686 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:12:13.686 DEBUG:boto:Data: 2022-09-12T20:12:13.687 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.687 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.688 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.689 DEBUG:boto:Params: {} 2022-09-12T20:12:13.689 DEBUG:boto:Token: None 2022-09-12T20:12:13.690 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/ 2022-09-12T20:12:13.690 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048= 2022-09-12T20:12:13.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048=', 'Content-Length': '0'} 2022-09-12T20:12:13.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f73472df4bf6ecb-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.696 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:13.697 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:13.697 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:13.698 DEBUG:boto:Method: HEAD 2022-09-12T20:12:13.699 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:12:13.699 DEBUG:boto:Data: 2022-09-12T20:12:13.700 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.700 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.701 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.702 DEBUG:boto:Params: {} 2022-09-12T20:12:13.703 DEBUG:boto:Token: None 2022-09-12T20:12:13.705 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj22 2022-09-12T20:12:13.705 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gKTdJ2pbcoVFTe4sEzELZADBNw4= 2022-09-12T20:12:13.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gKTdJ2pbcoVFTe4sEzELZADBNw4=', 'Content-Length': '0'} 2022-09-12T20:12:13.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc44a3400b63becd-00631f929d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.709 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:13.709 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:13.710 DEBUG:boto:Method: HEAD 2022-09-12T20:12:13.710 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:12:13.711 DEBUG:boto:Data: 2022-09-12T20:12:13.711 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.712 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.712 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.713 DEBUG:boto:Params: {} 2022-09-12T20:12:13.713 DEBUG:boto:Token: None 2022-09-12T20:12:13.714 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj22 2022-09-12T20:12:13.715 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gKTdJ2pbcoVFTe4sEzELZADBNw4= 2022-09-12T20:12:13.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gKTdJ2pbcoVFTe4sEzELZADBNw4=', 'Content-Length': '0'} 2022-09-12T20:12:13.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b03101b0820a906f-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:12:13.719 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:13.719 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:13.720 DEBUG:boto:Method: GET 2022-09-12T20:12:13.720 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:12:13.721 DEBUG:boto:Data: 2022-09-12T20:12:13.721 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.722 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.722 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.723 DEBUG:boto:Params: {} 2022-09-12T20:12:13.723 DEBUG:boto:Token: None 2022-09-12T20:12:13.724 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj22 2022-09-12T20:12:13.725 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F1SuletBVZO7LFTk/bghANll9NI= 2022-09-12T20:12:13.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F1SuletBVZO7LFTk/bghANll9NI=', 'Content-Length': '0'} 2022-09-12T20:12:13.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df08cf187a88dbd8-00631f929d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.728 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:13.729 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:13.730 DEBUG:boto:Method: GET 2022-09-12T20:12:13.730 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:12:13.731 DEBUG:boto:Data: 2022-09-12T20:12:13.731 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.732 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.732 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.733 DEBUG:boto:Params: {} 2022-09-12T20:12:13.734 DEBUG:boto:Token: None 2022-09-12T20:12:13.734 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj22 2022-09-12T20:12:13.735 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F1SuletBVZO7LFTk/bghANll9NI= 2022-09-12T20:12:13.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F1SuletBVZO7LFTk/bghANll9NI=', 'Content-Length': '0'} 2022-09-12T20:12:13.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078a1f77f40798d56-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.786 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:13.787 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:13.787 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:12:13.788 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:12:13.788 DEBUG:boto:Method: GET 2022-09-12T20:12:13.789 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:12:13.790 DEBUG:boto:Data: 2022-09-12T20:12:13.791 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.791 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.792 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.792 DEBUG:boto:Params: {} 2022-09-12T20:12:13.793 DEBUG:boto:Token: None 2022-09-12T20:12:13.793 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/ 2022-09-12T20:12:13.794 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048= 2022-09-12T20:12:13.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048=', 'Content-Length': '0'} 2022-09-12T20:12:13.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f717db1ea212fd3-00631f929d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.798 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:13.800 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:13.800 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:13.801 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:12:13.801 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:12:13.802 DEBUG:boto:Method: GET 2022-09-12T20:12:13.802 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:12:13.803 DEBUG:boto:Data: 2022-09-12T20:12:13.803 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.804 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.804 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.805 DEBUG:boto:Params: {} 2022-09-12T20:12:13.805 DEBUG:boto:Token: None 2022-09-12T20:12:13.806 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/ 2022-09-12T20:12:13.807 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048= 2022-09-12T20:12:13.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048=', 'Content-Length': '0'} 2022-09-12T20:12:13.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1d3f4117d2cd8dd-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.813 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:13.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:12:13.815 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:13.815 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:13.816 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:13.817 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:13.817 DEBUG:boto:Method: GET 2022-09-12T20:12:13.818 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:13.818 DEBUG:boto:Data: 2022-09-12T20:12:13.819 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.819 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.820 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.820 DEBUG:boto:Params: {} 2022-09-12T20:12:13.821 DEBUG:boto:Token: None 2022-09-12T20:12:13.822 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:13.822 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1xSHL+CNW6b4sgKZPQSMp5JZ1go= 2022-09-12T20:12:13.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1xSHL+CNW6b4sgKZPQSMp5JZ1go=', 'Content-Length': '0'} 2022-09-12T20:12:13.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000649c7166a341ea92-00631f929d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.826 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:13.827 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:13.827 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:13.828 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:13.828 DEBUG:boto:Method: GET 2022-09-12T20:12:13.829 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:13.829 DEBUG:boto:Data: 2022-09-12T20:12:13.830 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.830 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.831 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.831 DEBUG:boto:Params: {} 2022-09-12T20:12:13.832 DEBUG:boto:Token: None 2022-09-12T20:12:13.833 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:13.833 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1xSHL+CNW6b4sgKZPQSMp5JZ1go= 2022-09-12T20:12:13.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1xSHL+CNW6b4sgKZPQSMp5JZ1go=', 'Content-Length': '0'} 2022-09-12T20:12:13.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d200176d4007360c-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.837 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:13.837 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:13.838 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:13.838 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:13.839 DEBUG:boto:Method: HEAD 2022-09-12T20:12:13.839 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:13.840 DEBUG:boto:Data: 2022-09-12T20:12:13.840 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.841 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.841 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.842 DEBUG:boto:Params: {} 2022-09-12T20:12:13.843 DEBUG:boto:Token: None 2022-09-12T20:12:13.843 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:13.844 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CBsK1EaFUzme+VN1kxLmWaSqHCY= 2022-09-12T20:12:13.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CBsK1EaFUzme+VN1kxLmWaSqHCY=', 'Content-Length': '0'} 2022-09-12T20:12:13.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2567df50f78300f-00631f929d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.847 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:13.848 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:13.848 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:13.849 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:13.850 DEBUG:boto:Method: HEAD 2022-09-12T20:12:13.850 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:13.851 DEBUG:boto:Data: 2022-09-12T20:12:13.851 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.852 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.852 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.853 DEBUG:boto:Params: {} 2022-09-12T20:12:13.853 DEBUG:boto:Token: None 2022-09-12T20:12:13.854 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:13.854 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CBsK1EaFUzme+VN1kxLmWaSqHCY= 2022-09-12T20:12:13.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CBsK1EaFUzme+VN1kxLmWaSqHCY=', 'Content-Length': '0'} 2022-09-12T20:12:13.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ac3ebf7de626a61-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:12:13.858 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:13.859 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:13.859 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:13.860 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:13.860 DEBUG:boto:Method: GET 2022-09-12T20:12:13.861 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:13.862 DEBUG:boto:Data: 2022-09-12T20:12:13.862 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.863 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.863 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.864 DEBUG:boto:Params: {} 2022-09-12T20:12:13.864 DEBUG:boto:Token: None 2022-09-12T20:12:13.865 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:13.865 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1xSHL+CNW6b4sgKZPQSMp5JZ1go= 2022-09-12T20:12:13.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1xSHL+CNW6b4sgKZPQSMp5JZ1go=', 'Content-Length': '0'} 2022-09-12T20:12:13.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afcdc0d81b2b07f4-00631f929d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.869 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:13.870 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:13.870 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:13.871 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:13.871 DEBUG:boto:Method: GET 2022-09-12T20:12:13.872 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:13.872 DEBUG:boto:Data: 2022-09-12T20:12:13.873 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.873 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.874 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.874 DEBUG:boto:Params: {} 2022-09-12T20:12:13.875 DEBUG:boto:Token: None 2022-09-12T20:12:13.876 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:13.876 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1xSHL+CNW6b4sgKZPQSMp5JZ1go= 2022-09-12T20:12:13.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1xSHL+CNW6b4sgKZPQSMp5JZ1go=', 'Content-Length': '0'} 2022-09-12T20:12:13.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c81bd86a17f4a1a-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.880 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:13.880 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:13.881 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:12:13.881 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:12:13.882 DEBUG:boto:Method: GET 2022-09-12T20:12:13.883 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:12:13.883 DEBUG:boto:Data: 2022-09-12T20:12:13.884 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.884 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.885 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.885 DEBUG:boto:Params: {} 2022-09-12T20:12:13.886 DEBUG:boto:Token: None 2022-09-12T20:12:13.886 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/ 2022-09-12T20:12:13.887 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048= 2022-09-12T20:12:13.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048=', 'Content-Length': '0'} 2022-09-12T20:12:13.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000809203a2e21383e4-00631f929d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.891 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:13.892 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:13.892 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:13.893 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:12:13.893 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:12:13.894 DEBUG:boto:Method: GET 2022-09-12T20:12:13.895 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:12:13.895 DEBUG:boto:Data: 2022-09-12T20:12:13.896 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.896 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.897 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.897 DEBUG:boto:Params: {} 2022-09-12T20:12:13.898 DEBUG:boto:Token: None 2022-09-12T20:12:13.898 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/ 2022-09-12T20:12:13.899 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048= 2022-09-12T20:12:13.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048=', 'Content-Length': '0'} 2022-09-12T20:12:13.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d082db103befe8fd-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.946 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:13.947 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:13.948 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:13.948 DEBUG:boto:Method: HEAD 2022-09-12T20:12:13.949 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:12:13.950 DEBUG:boto:Data: 2022-09-12T20:12:13.950 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.951 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.951 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.952 DEBUG:boto:Params: {} 2022-09-12T20:12:13.952 DEBUG:boto:Token: None 2022-09-12T20:12:13.953 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj23 2022-09-12T20:12:13.954 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JhMp5SS0CZKZt+s10e+l4HK5h6E= 2022-09-12T20:12:13.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JhMp5SS0CZKZt+s10e+l4HK5h6E=', 'Content-Length': '0'} 2022-09-12T20:12:13.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042ce5cb56918b670-00631f929d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.957 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:13.958 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:13.958 DEBUG:boto:Method: HEAD 2022-09-12T20:12:13.959 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:12:13.959 DEBUG:boto:Data: 2022-09-12T20:12:13.960 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.960 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.961 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.961 DEBUG:boto:Params: {} 2022-09-12T20:12:13.962 DEBUG:boto:Token: None 2022-09-12T20:12:13.963 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj23 2022-09-12T20:12:13.963 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JhMp5SS0CZKZt+s10e+l4HK5h6E= 2022-09-12T20:12:13.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JhMp5SS0CZKZt+s10e+l4HK5h6E=', 'Content-Length': '0'} 2022-09-12T20:12:13.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccaf9a0886984266-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:12:13.967 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:13.968 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:13.968 DEBUG:boto:Method: GET 2022-09-12T20:12:13.969 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:12:13.969 DEBUG:boto:Data: 2022-09-12T20:12:13.970 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.971 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.971 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.972 DEBUG:boto:Params: {} 2022-09-12T20:12:13.972 DEBUG:boto:Token: None 2022-09-12T20:12:13.973 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj23 2022-09-12T20:12:13.973 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:i/j8gg9w8SWCpb8nqCEkXzVOysE= 2022-09-12T20:12:13.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:i/j8gg9w8SWCpb8nqCEkXzVOysE=', 'Content-Length': '0'} 2022-09-12T20:12:13.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005eec1a5ec1bfa46c-00631f929d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.977 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:13.978 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:13.978 DEBUG:boto:Method: GET 2022-09-12T20:12:13.979 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:12:13.979 DEBUG:boto:Data: 2022-09-12T20:12:13.980 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.980 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:13.981 DEBUG:boto:Port: 8000 2022-09-12T20:12:13.981 DEBUG:boto:Params: {} 2022-09-12T20:12:13.982 DEBUG:boto:Token: None 2022-09-12T20:12:13.983 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/obj23 2022-09-12T20:12:13.983 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:i/j8gg9w8SWCpb8nqCEkXzVOysE= 2022-09-12T20:12:13.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:i/j8gg9w8SWCpb8nqCEkXzVOysE=', 'Content-Length': '0'} 2022-09-12T20:12:13.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d0b0d642d0cbc67-00631f929d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.987 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:13.987 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:13.988 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:12:13.988 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:12:13.989 DEBUG:boto:Method: GET 2022-09-12T20:12:13.989 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:12:13.990 DEBUG:boto:Data: 2022-09-12T20:12:13.990 DEBUG:boto:Headers: {} 2022-09-12T20:12:13.991 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:13.991 DEBUG:boto:Port: 8001 2022-09-12T20:12:13.992 DEBUG:boto:Params: {} 2022-09-12T20:12:13.993 DEBUG:boto:Token: None 2022-09-12T20:12:13.993 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:13 GMT /jdpiak-75/ 2022-09-12T20:12:13.994 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048= 2022-09-12T20:12:13.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+Gmk/vcdYR/LsV+AS3fgdKRV048=', 'Content-Length': '0'} 2022-09-12T20:12:13.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6f784f54c997c74-00631f929d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:13.998 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:13.999 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:13.999 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:14.000 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:12:14.000 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:12:14.001 DEBUG:boto:Method: GET 2022-09-12T20:12:14.002 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:12:14.002 DEBUG:boto:Data: 2022-09-12T20:12:14.003 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.003 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.004 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.004 DEBUG:boto:Params: {} 2022-09-12T20:12:14.005 DEBUG:boto:Token: None 2022-09-12T20:12:14.005 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/ 2022-09-12T20:12:14.006 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk= 2022-09-12T20:12:14.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk=', 'Content-Length': '0'} 2022-09-12T20:12:14.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4af7487635b7a22-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.011 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:14.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:12:14.012 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:14.013 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:14.014 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:14.014 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:14.015 DEBUG:boto:Method: GET 2022-09-12T20:12:14.015 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:14.016 DEBUG:boto:Data: 2022-09-12T20:12:14.016 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.017 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.017 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.018 DEBUG:boto:Params: {} 2022-09-12T20:12:14.019 DEBUG:boto:Token: None 2022-09-12T20:12:14.019 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:14.020 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5TE1wNTVFAU3Pee1WK2TTwZ9HO8= 2022-09-12T20:12:14.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5TE1wNTVFAU3Pee1WK2TTwZ9HO8=', 'Content-Length': '0'} 2022-09-12T20:12:14.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002edfe83a209212a-00631f929e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.023 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:14.024 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:14.025 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:14.025 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:14.026 DEBUG:boto:Method: GET 2022-09-12T20:12:14.026 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:14.027 DEBUG:boto:Data: 2022-09-12T20:12:14.027 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.028 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.028 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.029 DEBUG:boto:Params: {} 2022-09-12T20:12:14.030 DEBUG:boto:Token: None 2022-09-12T20:12:14.030 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:14.031 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5TE1wNTVFAU3Pee1WK2TTwZ9HO8= 2022-09-12T20:12:14.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5TE1wNTVFAU3Pee1WK2TTwZ9HO8=', 'Content-Length': '0'} 2022-09-12T20:12:14.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b57871ba1705d518-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.074 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:14.075 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:14.075 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:14.076 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:14.076 DEBUG:boto:Method: HEAD 2022-09-12T20:12:14.077 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:14.077 DEBUG:boto:Data: 2022-09-12T20:12:14.078 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.078 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.079 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.080 DEBUG:boto:Params: {} 2022-09-12T20:12:14.080 DEBUG:boto:Token: None 2022-09-12T20:12:14.081 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:14.082 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m7aT7K9iQy50y9WAoOtN/mDA2R8= 2022-09-12T20:12:14.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m7aT7K9iQy50y9WAoOtN/mDA2R8=', 'Content-Length': '0'} 2022-09-12T20:12:14.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a95bc52eea89e90-00631f929e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.085 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:14.086 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:14.087 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:14.087 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:14.088 DEBUG:boto:Method: HEAD 2022-09-12T20:12:14.088 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:14.089 DEBUG:boto:Data: 2022-09-12T20:12:14.089 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.090 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.090 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.091 DEBUG:boto:Params: {} 2022-09-12T20:12:14.091 DEBUG:boto:Token: None 2022-09-12T20:12:14.092 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:14.093 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m7aT7K9iQy50y9WAoOtN/mDA2R8= 2022-09-12T20:12:14.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m7aT7K9iQy50y9WAoOtN/mDA2R8=', 'Content-Length': '0'} 2022-09-12T20:12:14.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097581ac0552aa3ed-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:12:14.097 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:14.097 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:14.098 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:14.098 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:14.099 DEBUG:boto:Method: GET 2022-09-12T20:12:14.100 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:14.100 DEBUG:boto:Data: 2022-09-12T20:12:14.101 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.101 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.102 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.103 DEBUG:boto:Params: {} 2022-09-12T20:12:14.103 DEBUG:boto:Token: None 2022-09-12T20:12:14.104 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:14.104 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5TE1wNTVFAU3Pee1WK2TTwZ9HO8= 2022-09-12T20:12:14.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5TE1wNTVFAU3Pee1WK2TTwZ9HO8=', 'Content-Length': '0'} 2022-09-12T20:12:14.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001479459af9acc674-00631f929e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.108 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:14.109 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:14.109 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:14.110 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:14.111 DEBUG:boto:Method: GET 2022-09-12T20:12:14.111 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:14.112 DEBUG:boto:Data: 2022-09-12T20:12:14.113 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.113 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.114 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.115 DEBUG:boto:Params: {} 2022-09-12T20:12:14.116 DEBUG:boto:Token: None 2022-09-12T20:12:14.116 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:14.117 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5TE1wNTVFAU3Pee1WK2TTwZ9HO8= 2022-09-12T20:12:14.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5TE1wNTVFAU3Pee1WK2TTwZ9HO8=', 'Content-Length': '0'} 2022-09-12T20:12:14.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002881d29e244eebd-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.121 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:14.122 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:14.122 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:12:14.123 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:12:14.124 DEBUG:boto:Method: GET 2022-09-12T20:12:14.124 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:12:14.125 DEBUG:boto:Data: 2022-09-12T20:12:14.126 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.127 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.128 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.129 DEBUG:boto:Params: {} 2022-09-12T20:12:14.130 DEBUG:boto:Token: None 2022-09-12T20:12:14.131 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/ 2022-09-12T20:12:14.132 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk= 2022-09-12T20:12:14.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk=', 'Content-Length': '0'} 2022-09-12T20:12:14.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090f4735ffdb5d9fa-00631f929e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.137 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:14.138 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:14.138 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:14.139 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:12:14.140 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:12:14.140 DEBUG:boto:Method: GET 2022-09-12T20:12:14.141 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:12:14.142 DEBUG:boto:Data: 2022-09-12T20:12:14.143 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.143 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.144 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.145 DEBUG:boto:Params: {} 2022-09-12T20:12:14.146 DEBUG:boto:Token: None 2022-09-12T20:12:14.146 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/ 2022-09-12T20:12:14.147 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk= 2022-09-12T20:12:14.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk=', 'Content-Length': '0'} 2022-09-12T20:12:14.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031945eef4178b5e2-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.154 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:14.155 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:14.155 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:14.156 DEBUG:boto:Method: HEAD 2022-09-12T20:12:14.156 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:12:14.157 DEBUG:boto:Data: 2022-09-12T20:12:14.158 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.158 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.159 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.159 DEBUG:boto:Params: {} 2022-09-12T20:12:14.160 DEBUG:boto:Token: None 2022-09-12T20:12:14.161 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj29 2022-09-12T20:12:14.161 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:I67WJn16CAZrXksyBRtFXk+c95E= 2022-09-12T20:12:14.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:I67WJn16CAZrXksyBRtFXk+c95E=', 'Content-Length': '0'} 2022-09-12T20:12:14.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d627c7b3c2b44230-00631f929e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.165 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:14.166 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:14.167 DEBUG:boto:Method: HEAD 2022-09-12T20:12:14.167 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:12:14.168 DEBUG:boto:Data: 2022-09-12T20:12:14.169 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.169 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.170 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.171 DEBUG:boto:Params: {} 2022-09-12T20:12:14.171 DEBUG:boto:Token: None 2022-09-12T20:12:14.172 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj29 2022-09-12T20:12:14.173 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:I67WJn16CAZrXksyBRtFXk+c95E= 2022-09-12T20:12:14.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:I67WJn16CAZrXksyBRtFXk+c95E=', 'Content-Length': '0'} 2022-09-12T20:12:14.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038f0e91cbcedf716-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:12:14.177 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:14.178 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:14.179 DEBUG:boto:Method: GET 2022-09-12T20:12:14.180 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:12:14.181 DEBUG:boto:Data: 2022-09-12T20:12:14.181 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.182 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.183 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.183 DEBUG:boto:Params: {} 2022-09-12T20:12:14.184 DEBUG:boto:Token: None 2022-09-12T20:12:14.185 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj29 2022-09-12T20:12:14.186 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bOkxzZ5t68XmOk0xvoySaX5TAiU= 2022-09-12T20:12:14.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bOkxzZ5t68XmOk0xvoySaX5TAiU=', 'Content-Length': '0'} 2022-09-12T20:12:14.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4a99b7c439b121d-00631f929e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.234 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:14.235 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:14.236 DEBUG:boto:Method: GET 2022-09-12T20:12:14.236 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:12:14.237 DEBUG:boto:Data: 2022-09-12T20:12:14.237 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.238 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.239 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.239 DEBUG:boto:Params: {} 2022-09-12T20:12:14.240 DEBUG:boto:Token: None 2022-09-12T20:12:14.241 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj29 2022-09-12T20:12:14.242 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bOkxzZ5t68XmOk0xvoySaX5TAiU= 2022-09-12T20:12:14.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bOkxzZ5t68XmOk0xvoySaX5TAiU=', 'Content-Length': '0'} 2022-09-12T20:12:14.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000298f78f26e5dbb86-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.246 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:14.247 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:14.247 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:12:14.248 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:12:14.248 DEBUG:boto:Method: GET 2022-09-12T20:12:14.249 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:12:14.250 DEBUG:boto:Data: 2022-09-12T20:12:14.250 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.251 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.252 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.252 DEBUG:boto:Params: {} 2022-09-12T20:12:14.253 DEBUG:boto:Token: None 2022-09-12T20:12:14.254 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/ 2022-09-12T20:12:14.254 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk= 2022-09-12T20:12:14.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk=', 'Content-Length': '0'} 2022-09-12T20:12:14.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009ea1ec8ab6dcb46-00631f929e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.259 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:14.260 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:14.260 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:14.261 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:12:14.262 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:12:14.263 DEBUG:boto:Method: GET 2022-09-12T20:12:14.264 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:12:14.265 DEBUG:boto:Data: 2022-09-12T20:12:14.265 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.266 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.267 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.267 DEBUG:boto:Params: {} 2022-09-12T20:12:14.268 DEBUG:boto:Token: None 2022-09-12T20:12:14.269 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/ 2022-09-12T20:12:14.269 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk= 2022-09-12T20:12:14.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk=', 'Content-Length': '0'} 2022-09-12T20:12:14.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5898228575d209b-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.275 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:14.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:12:14.277 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:14.278 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:14.278 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:14.279 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:14.280 DEBUG:boto:Method: GET 2022-09-12T20:12:14.280 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:14.281 DEBUG:boto:Data: 2022-09-12T20:12:14.282 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.282 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.283 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.283 DEBUG:boto:Params: {} 2022-09-12T20:12:14.284 DEBUG:boto:Token: None 2022-09-12T20:12:14.285 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:14.285 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KXQR3nJ7u+SIBrgFtt+QT6MoOpw= 2022-09-12T20:12:14.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KXQR3nJ7u+SIBrgFtt+QT6MoOpw=', 'Content-Length': '0'} 2022-09-12T20:12:14.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e922ef3a44879609-00631f929e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.290 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:14.290 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:14.291 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:14.291 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:14.292 DEBUG:boto:Method: GET 2022-09-12T20:12:14.292 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:14.293 DEBUG:boto:Data: 2022-09-12T20:12:14.293 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.294 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.295 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.295 DEBUG:boto:Params: {} 2022-09-12T20:12:14.296 DEBUG:boto:Token: None 2022-09-12T20:12:14.296 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:14.297 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KXQR3nJ7u+SIBrgFtt+QT6MoOpw= 2022-09-12T20:12:14.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KXQR3nJ7u+SIBrgFtt+QT6MoOpw=', 'Content-Length': '0'} 2022-09-12T20:12:14.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbff615da6a85f43-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.342 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:14.343 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:14.344 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:14.346 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:14.347 DEBUG:boto:Method: HEAD 2022-09-12T20:12:14.348 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:14.349 DEBUG:boto:Data: 2022-09-12T20:12:14.349 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.350 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.351 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.352 DEBUG:boto:Params: {} 2022-09-12T20:12:14.353 DEBUG:boto:Token: None 2022-09-12T20:12:14.355 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:14.356 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Pf6GZgXVu13xkbCg1omwseVfKYY= 2022-09-12T20:12:14.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Pf6GZgXVu13xkbCg1omwseVfKYY=', 'Content-Length': '0'} 2022-09-12T20:12:14.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f78e9e528a41cb69-00631f929e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.361 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:14.362 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:14.363 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:14.364 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:14.365 DEBUG:boto:Method: HEAD 2022-09-12T20:12:14.365 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:14.366 DEBUG:boto:Data: 2022-09-12T20:12:14.367 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.368 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.368 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.369 DEBUG:boto:Params: {} 2022-09-12T20:12:14.369 DEBUG:boto:Token: None 2022-09-12T20:12:14.370 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:14.371 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Pf6GZgXVu13xkbCg1omwseVfKYY= 2022-09-12T20:12:14.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Pf6GZgXVu13xkbCg1omwseVfKYY=', 'Content-Length': '0'} 2022-09-12T20:12:14.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073c29b6c0bcaf033-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:12:14.375 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:14.375 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:14.376 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:14.376 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:14.377 DEBUG:boto:Method: GET 2022-09-12T20:12:14.377 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:14.378 DEBUG:boto:Data: 2022-09-12T20:12:14.378 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.379 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.380 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.380 DEBUG:boto:Params: {} 2022-09-12T20:12:14.381 DEBUG:boto:Token: None 2022-09-12T20:12:14.381 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:14.382 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KXQR3nJ7u+SIBrgFtt+QT6MoOpw= 2022-09-12T20:12:14.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KXQR3nJ7u+SIBrgFtt+QT6MoOpw=', 'Content-Length': '0'} 2022-09-12T20:12:14.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000353005acf2c9e4dc-00631f929e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.385 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:14.386 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:14.387 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:14.387 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:14.388 DEBUG:boto:Method: GET 2022-09-12T20:12:14.388 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:14.389 DEBUG:boto:Data: 2022-09-12T20:12:14.389 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.390 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.390 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.391 DEBUG:boto:Params: {} 2022-09-12T20:12:14.391 DEBUG:boto:Token: None 2022-09-12T20:12:14.392 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:14.393 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KXQR3nJ7u+SIBrgFtt+QT6MoOpw= 2022-09-12T20:12:14.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KXQR3nJ7u+SIBrgFtt+QT6MoOpw=', 'Content-Length': '0'} 2022-09-12T20:12:14.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e109bd5308cb34c-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.396 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:14.397 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:14.397 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:12:14.398 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:12:14.399 DEBUG:boto:Method: GET 2022-09-12T20:12:14.399 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:12:14.400 DEBUG:boto:Data: 2022-09-12T20:12:14.400 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.401 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.401 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.402 DEBUG:boto:Params: {} 2022-09-12T20:12:14.402 DEBUG:boto:Token: None 2022-09-12T20:12:14.403 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/ 2022-09-12T20:12:14.404 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk= 2022-09-12T20:12:14.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk=', 'Content-Length': '0'} 2022-09-12T20:12:14.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000491ae3ea646832de-00631f929e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.408 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:14.409 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:14.409 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:14.410 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:12:14.410 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:12:14.411 DEBUG:boto:Method: GET 2022-09-12T20:12:14.412 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:12:14.412 DEBUG:boto:Data: 2022-09-12T20:12:14.413 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.413 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.414 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.414 DEBUG:boto:Params: {} 2022-09-12T20:12:14.415 DEBUG:boto:Token: None 2022-09-12T20:12:14.415 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/ 2022-09-12T20:12:14.416 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk= 2022-09-12T20:12:14.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk=', 'Content-Length': '0'} 2022-09-12T20:12:14.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084170cc8fa36e022-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.421 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:14.422 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:14.422 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:14.423 DEBUG:boto:Method: HEAD 2022-09-12T20:12:14.424 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:12:14.424 DEBUG:boto:Data: 2022-09-12T20:12:14.425 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.425 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.426 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.426 DEBUG:boto:Params: {} 2022-09-12T20:12:14.427 DEBUG:boto:Token: None 2022-09-12T20:12:14.427 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj33 2022-09-12T20:12:14.428 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mJr20ZFGBIWRQxa9FOKrQrYMp4s= 2022-09-12T20:12:14.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mJr20ZFGBIWRQxa9FOKrQrYMp4s=', 'Content-Length': '0'} 2022-09-12T20:12:14.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcab3ca2298a83c8-00631f929e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.431 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:14.432 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:14.432 DEBUG:boto:Method: HEAD 2022-09-12T20:12:14.433 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:12:14.433 DEBUG:boto:Data: 2022-09-12T20:12:14.434 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.434 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.435 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.435 DEBUG:boto:Params: {} 2022-09-12T20:12:14.436 DEBUG:boto:Token: None 2022-09-12T20:12:14.436 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj33 2022-09-12T20:12:14.437 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mJr20ZFGBIWRQxa9FOKrQrYMp4s= 2022-09-12T20:12:14.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mJr20ZFGBIWRQxa9FOKrQrYMp4s=', 'Content-Length': '0'} 2022-09-12T20:12:14.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eabd509fbafeaf85-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:12:14.441 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:14.441 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:14.442 DEBUG:boto:Method: GET 2022-09-12T20:12:14.442 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:12:14.443 DEBUG:boto:Data: 2022-09-12T20:12:14.443 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.444 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.444 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.445 DEBUG:boto:Params: {} 2022-09-12T20:12:14.445 DEBUG:boto:Token: None 2022-09-12T20:12:14.446 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj33 2022-09-12T20:12:14.447 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TaVaFqSd5gHyXtU+n+o7qkXY1Mc= 2022-09-12T20:12:14.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TaVaFqSd5gHyXtU+n+o7qkXY1Mc=', 'Content-Length': '0'} 2022-09-12T20:12:14.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a4080c7788ca2ed-00631f929e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.490 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:14.491 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:14.491 DEBUG:boto:Method: GET 2022-09-12T20:12:14.492 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:12:14.492 DEBUG:boto:Data: 2022-09-12T20:12:14.493 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.493 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.494 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.495 DEBUG:boto:Params: {} 2022-09-12T20:12:14.495 DEBUG:boto:Token: None 2022-09-12T20:12:14.496 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj33 2022-09-12T20:12:14.497 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TaVaFqSd5gHyXtU+n+o7qkXY1Mc= 2022-09-12T20:12:14.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TaVaFqSd5gHyXtU+n+o7qkXY1Mc=', 'Content-Length': '0'} 2022-09-12T20:12:14.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a96c2223cebcc3bb-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.500 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:14.501 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:14.502 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:12:14.502 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:12:14.503 DEBUG:boto:Method: GET 2022-09-12T20:12:14.503 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:12:14.504 DEBUG:boto:Data: 2022-09-12T20:12:14.504 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.505 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.505 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.506 DEBUG:boto:Params: {} 2022-09-12T20:12:14.506 DEBUG:boto:Token: None 2022-09-12T20:12:14.507 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/ 2022-09-12T20:12:14.508 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk= 2022-09-12T20:12:14.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk=', 'Content-Length': '0'} 2022-09-12T20:12:14.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e22408c7cef5124-00631f929e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.512 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:14.512 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:14.513 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:14.514 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:12:14.514 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:12:14.515 DEBUG:boto:Method: GET 2022-09-12T20:12:14.515 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:12:14.516 DEBUG:boto:Data: 2022-09-12T20:12:14.516 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.517 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.517 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.518 DEBUG:boto:Params: {} 2022-09-12T20:12:14.519 DEBUG:boto:Token: None 2022-09-12T20:12:14.519 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/ 2022-09-12T20:12:14.520 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk= 2022-09-12T20:12:14.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk=', 'Content-Length': '0'} 2022-09-12T20:12:14.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000022d6ec5e0d15978-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.526 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:14.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:12:14.527 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:14.528 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:14.528 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:14.529 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:14.529 DEBUG:boto:Method: GET 2022-09-12T20:12:14.530 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:14.530 DEBUG:boto:Data: 2022-09-12T20:12:14.531 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.531 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.532 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.532 DEBUG:boto:Params: {} 2022-09-12T20:12:14.533 DEBUG:boto:Token: None 2022-09-12T20:12:14.534 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:14.534 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9SV6Yes2RCl1OQ5DraknfdcyCR0= 2022-09-12T20:12:14.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9SV6Yes2RCl1OQ5DraknfdcyCR0=', 'Content-Length': '0'} 2022-09-12T20:12:14.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028cfbbbc191e2580-00631f929e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.538 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:14.539 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:14.539 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:14.540 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:14.540 DEBUG:boto:Method: GET 2022-09-12T20:12:14.541 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:14.541 DEBUG:boto:Data: 2022-09-12T20:12:14.542 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.542 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.543 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.544 DEBUG:boto:Params: {} 2022-09-12T20:12:14.544 DEBUG:boto:Token: None 2022-09-12T20:12:14.545 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:14.545 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9SV6Yes2RCl1OQ5DraknfdcyCR0= 2022-09-12T20:12:14.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9SV6Yes2RCl1OQ5DraknfdcyCR0=', 'Content-Length': '0'} 2022-09-12T20:12:14.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a876d7f202ddfbf1-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.590 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:14.591 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:14.591 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:14.592 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:14.592 DEBUG:boto:Method: HEAD 2022-09-12T20:12:14.593 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:14.593 DEBUG:boto:Data: 2022-09-12T20:12:14.594 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.594 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.595 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.595 DEBUG:boto:Params: {} 2022-09-12T20:12:14.596 DEBUG:boto:Token: None 2022-09-12T20:12:14.597 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:14.597 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TkMuth3dmACxeokVTMNfwbTBBt8= 2022-09-12T20:12:14.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TkMuth3dmACxeokVTMNfwbTBBt8=', 'Content-Length': '0'} 2022-09-12T20:12:14.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078dfce5a774bb9e3-00631f929e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.601 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:14.601 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:14.602 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:14.602 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:14.603 DEBUG:boto:Method: HEAD 2022-09-12T20:12:14.603 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:14.604 DEBUG:boto:Data: 2022-09-12T20:12:14.604 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.605 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.606 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.606 DEBUG:boto:Params: {} 2022-09-12T20:12:14.607 DEBUG:boto:Token: None 2022-09-12T20:12:14.607 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:14.608 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TkMuth3dmACxeokVTMNfwbTBBt8= 2022-09-12T20:12:14.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TkMuth3dmACxeokVTMNfwbTBBt8=', 'Content-Length': '0'} 2022-09-12T20:12:14.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d51b5ef09b8a647-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:12:14.613 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:14.614 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:14.614 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:14.615 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:14.616 DEBUG:boto:Method: GET 2022-09-12T20:12:14.616 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:14.617 DEBUG:boto:Data: 2022-09-12T20:12:14.618 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.618 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.619 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.619 DEBUG:boto:Params: {} 2022-09-12T20:12:14.620 DEBUG:boto:Token: None 2022-09-12T20:12:14.620 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:14.621 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9SV6Yes2RCl1OQ5DraknfdcyCR0= 2022-09-12T20:12:14.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9SV6Yes2RCl1OQ5DraknfdcyCR0=', 'Content-Length': '0'} 2022-09-12T20:12:14.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d26a0e0a97dad6da-00631f929e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.625 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:14.625 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:14.626 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:14.626 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:14.627 DEBUG:boto:Method: GET 2022-09-12T20:12:14.628 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:14.628 DEBUG:boto:Data: 2022-09-12T20:12:14.629 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.629 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.630 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.631 DEBUG:boto:Params: {} 2022-09-12T20:12:14.631 DEBUG:boto:Token: None 2022-09-12T20:12:14.632 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:14.632 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9SV6Yes2RCl1OQ5DraknfdcyCR0= 2022-09-12T20:12:14.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9SV6Yes2RCl1OQ5DraknfdcyCR0=', 'Content-Length': '0'} 2022-09-12T20:12:14.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036f40430c0af04d4-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.636 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:14.636 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:14.637 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:12:14.637 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:12:14.638 DEBUG:boto:Method: GET 2022-09-12T20:12:14.639 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:12:14.639 DEBUG:boto:Data: 2022-09-12T20:12:14.640 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.640 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.641 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.641 DEBUG:boto:Params: {} 2022-09-12T20:12:14.642 DEBUG:boto:Token: None 2022-09-12T20:12:14.643 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/ 2022-09-12T20:12:14.643 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk= 2022-09-12T20:12:14.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk=', 'Content-Length': '0'} 2022-09-12T20:12:14.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000596726e320b90a90-00631f929e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.648 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:14.648 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:14.649 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:14.649 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:12:14.650 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:12:14.651 DEBUG:boto:Method: GET 2022-09-12T20:12:14.651 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:12:14.652 DEBUG:boto:Data: 2022-09-12T20:12:14.652 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.653 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.653 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.654 DEBUG:boto:Params: {} 2022-09-12T20:12:14.654 DEBUG:boto:Token: None 2022-09-12T20:12:14.655 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/ 2022-09-12T20:12:14.655 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk= 2022-09-12T20:12:14.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk=', 'Content-Length': '0'} 2022-09-12T20:12:14.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcdec1d715981db7-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.661 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:14.662 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:14.663 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:14.663 DEBUG:boto:Method: HEAD 2022-09-12T20:12:14.664 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:12:14.664 DEBUG:boto:Data: 2022-09-12T20:12:14.665 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.665 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.666 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.666 DEBUG:boto:Params: {} 2022-09-12T20:12:14.667 DEBUG:boto:Token: None 2022-09-12T20:12:14.667 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj34 2022-09-12T20:12:14.668 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6X3MAkouxaqfHDCGvCND6tNKWJE= 2022-09-12T20:12:14.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6X3MAkouxaqfHDCGvCND6tNKWJE=', 'Content-Length': '0'} 2022-09-12T20:12:14.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0511db619aa0e5d-00631f929e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.671 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:14.672 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:14.673 DEBUG:boto:Method: HEAD 2022-09-12T20:12:14.673 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:12:14.674 DEBUG:boto:Data: 2022-09-12T20:12:14.674 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.675 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.675 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.676 DEBUG:boto:Params: {} 2022-09-12T20:12:14.676 DEBUG:boto:Token: None 2022-09-12T20:12:14.677 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj34 2022-09-12T20:12:14.678 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6X3MAkouxaqfHDCGvCND6tNKWJE= 2022-09-12T20:12:14.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6X3MAkouxaqfHDCGvCND6tNKWJE=', 'Content-Length': '0'} 2022-09-12T20:12:14.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d160d3ad92525fe8-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:12:14.682 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:14.682 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:14.683 DEBUG:boto:Method: GET 2022-09-12T20:12:14.683 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:12:14.684 DEBUG:boto:Data: 2022-09-12T20:12:14.684 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.685 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.685 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.686 DEBUG:boto:Params: {} 2022-09-12T20:12:14.687 DEBUG:boto:Token: None 2022-09-12T20:12:14.687 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj34 2022-09-12T20:12:14.688 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cxi8EZDgTo9Jzn1iYi91h09Vjvw= 2022-09-12T20:12:14.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cxi8EZDgTo9Jzn1iYi91h09Vjvw=', 'Content-Length': '0'} 2022-09-12T20:12:14.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca00d9b2d3a8a4c9-00631f929e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.734 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:14.735 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:14.735 DEBUG:boto:Method: GET 2022-09-12T20:12:14.736 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:12:14.736 DEBUG:boto:Data: 2022-09-12T20:12:14.737 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.737 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.738 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.738 DEBUG:boto:Params: {} 2022-09-12T20:12:14.739 DEBUG:boto:Token: None 2022-09-12T20:12:14.740 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj34 2022-09-12T20:12:14.740 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cxi8EZDgTo9Jzn1iYi91h09Vjvw= 2022-09-12T20:12:14.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cxi8EZDgTo9Jzn1iYi91h09Vjvw=', 'Content-Length': '0'} 2022-09-12T20:12:14.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc3d79141ebdefd3-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.744 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:14.745 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:14.745 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:12:14.746 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:12:14.746 DEBUG:boto:Method: GET 2022-09-12T20:12:14.747 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:12:14.747 DEBUG:boto:Data: 2022-09-12T20:12:14.748 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.748 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.749 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.749 DEBUG:boto:Params: {} 2022-09-12T20:12:14.750 DEBUG:boto:Token: None 2022-09-12T20:12:14.751 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/ 2022-09-12T20:12:14.751 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk= 2022-09-12T20:12:14.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk=', 'Content-Length': '0'} 2022-09-12T20:12:14.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066793d69297a2cb4-00631f929e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.756 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:14.756 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:14.757 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:14.757 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:12:14.758 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:12:14.759 DEBUG:boto:Method: GET 2022-09-12T20:12:14.759 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:12:14.760 DEBUG:boto:Data: 2022-09-12T20:12:14.760 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.761 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.761 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.762 DEBUG:boto:Params: {} 2022-09-12T20:12:14.762 DEBUG:boto:Token: None 2022-09-12T20:12:14.763 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/ 2022-09-12T20:12:14.764 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk= 2022-09-12T20:12:14.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk=', 'Content-Length': '0'} 2022-09-12T20:12:14.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c908b5fb91ac8b3-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.769 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:14.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:12:14.770 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:14.771 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:14.771 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:14.772 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:14.772 DEBUG:boto:Method: GET 2022-09-12T20:12:14.773 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:14.773 DEBUG:boto:Data: 2022-09-12T20:12:14.774 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.775 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.775 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.776 DEBUG:boto:Params: {} 2022-09-12T20:12:14.776 DEBUG:boto:Token: None 2022-09-12T20:12:14.777 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:14.777 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ehOWB8hJqZ+1N0GHsKptZkzDGps= 2022-09-12T20:12:14.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ehOWB8hJqZ+1N0GHsKptZkzDGps=', 'Content-Length': '0'} 2022-09-12T20:12:14.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a299e5afc107e734-00631f929e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.782 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:14.783 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:14.783 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:14.784 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:14.785 DEBUG:boto:Method: GET 2022-09-12T20:12:14.785 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:14.786 DEBUG:boto:Data: 2022-09-12T20:12:14.786 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.787 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.787 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.788 DEBUG:boto:Params: {} 2022-09-12T20:12:14.788 DEBUG:boto:Token: None 2022-09-12T20:12:14.789 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:14.790 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ehOWB8hJqZ+1N0GHsKptZkzDGps= 2022-09-12T20:12:14.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ehOWB8hJqZ+1N0GHsKptZkzDGps=', 'Content-Length': '0'} 2022-09-12T20:12:14.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000817f893e321ab64b-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.793 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:14.793 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:14.794 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:14.795 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:14.795 DEBUG:boto:Method: HEAD 2022-09-12T20:12:14.796 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:14.796 DEBUG:boto:Data: 2022-09-12T20:12:14.797 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.797 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.798 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.798 DEBUG:boto:Params: {} 2022-09-12T20:12:14.799 DEBUG:boto:Token: None 2022-09-12T20:12:14.799 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:14.800 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RHKFxtJs/8it8TEZO0copGzix9s= 2022-09-12T20:12:14.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RHKFxtJs/8it8TEZO0copGzix9s=', 'Content-Length': '0'} 2022-09-12T20:12:14.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e39d142a2cbd0dbf-00631f929e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.803 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:14.804 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:14.804 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:14.805 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:14.805 DEBUG:boto:Method: HEAD 2022-09-12T20:12:14.806 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:14.806 DEBUG:boto:Data: 2022-09-12T20:12:14.807 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.807 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.808 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.808 DEBUG:boto:Params: {} 2022-09-12T20:12:14.809 DEBUG:boto:Token: None 2022-09-12T20:12:14.809 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:14.810 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RHKFxtJs/8it8TEZO0copGzix9s= 2022-09-12T20:12:14.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RHKFxtJs/8it8TEZO0copGzix9s=', 'Content-Length': '0'} 2022-09-12T20:12:14.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e0621ad5103f249-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:12:14.814 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:14.814 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:14.815 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:14.816 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:14.816 DEBUG:boto:Method: GET 2022-09-12T20:12:14.817 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:14.817 DEBUG:boto:Data: 2022-09-12T20:12:14.818 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.818 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.819 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.819 DEBUG:boto:Params: {} 2022-09-12T20:12:14.820 DEBUG:boto:Token: None 2022-09-12T20:12:14.820 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:14.821 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ehOWB8hJqZ+1N0GHsKptZkzDGps= 2022-09-12T20:12:14.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ehOWB8hJqZ+1N0GHsKptZkzDGps=', 'Content-Length': '0'} 2022-09-12T20:12:14.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000735d5cf19f8e30e2-00631f929e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.866 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:14.866 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:14.867 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:14.868 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:14.868 DEBUG:boto:Method: GET 2022-09-12T20:12:14.869 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:14.869 DEBUG:boto:Data: 2022-09-12T20:12:14.870 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.870 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.871 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.871 DEBUG:boto:Params: {} 2022-09-12T20:12:14.872 DEBUG:boto:Token: None 2022-09-12T20:12:14.873 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:14.873 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ehOWB8hJqZ+1N0GHsKptZkzDGps= 2022-09-12T20:12:14.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ehOWB8hJqZ+1N0GHsKptZkzDGps=', 'Content-Length': '0'} 2022-09-12T20:12:14.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087bcbdd7d1b67d90-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.877 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:14.878 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:14.878 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:12:14.879 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:12:14.879 DEBUG:boto:Method: GET 2022-09-12T20:12:14.880 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:12:14.880 DEBUG:boto:Data: 2022-09-12T20:12:14.881 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.881 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.882 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.882 DEBUG:boto:Params: {} 2022-09-12T20:12:14.883 DEBUG:boto:Token: None 2022-09-12T20:12:14.884 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/ 2022-09-12T20:12:14.884 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk= 2022-09-12T20:12:14.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk=', 'Content-Length': '0'} 2022-09-12T20:12:14.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d36c3a86ff3f6960-00631f929e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.888 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:14.889 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:14.890 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:14.891 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:12:14.891 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:12:14.892 DEBUG:boto:Method: GET 2022-09-12T20:12:14.892 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:12:14.893 DEBUG:boto:Data: 2022-09-12T20:12:14.893 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.894 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.894 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.895 DEBUG:boto:Params: {} 2022-09-12T20:12:14.895 DEBUG:boto:Token: None 2022-09-12T20:12:14.896 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/ 2022-09-12T20:12:14.897 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk= 2022-09-12T20:12:14.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk=', 'Content-Length': '0'} 2022-09-12T20:12:14.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b0d3c9897d62ca0-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.903 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:14.903 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:14.904 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:14.905 DEBUG:boto:Method: HEAD 2022-09-12T20:12:14.905 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:12:14.906 DEBUG:boto:Data: 2022-09-12T20:12:14.906 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.907 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.907 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.908 DEBUG:boto:Params: {} 2022-09-12T20:12:14.908 DEBUG:boto:Token: None 2022-09-12T20:12:14.909 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj38 2022-09-12T20:12:14.909 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9W4tqlbgskyBhtw1WXycnBtQHqE= 2022-09-12T20:12:14.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9W4tqlbgskyBhtw1WXycnBtQHqE=', 'Content-Length': '0'} 2022-09-12T20:12:14.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000025ebfe62b966e90-00631f929e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.913 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:14.913 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:14.914 DEBUG:boto:Method: HEAD 2022-09-12T20:12:14.914 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:12:14.915 DEBUG:boto:Data: 2022-09-12T20:12:14.915 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.916 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.916 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.917 DEBUG:boto:Params: {} 2022-09-12T20:12:14.918 DEBUG:boto:Token: None 2022-09-12T20:12:14.919 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj38 2022-09-12T20:12:14.919 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9W4tqlbgskyBhtw1WXycnBtQHqE= 2022-09-12T20:12:14.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9W4tqlbgskyBhtw1WXycnBtQHqE=', 'Content-Length': '0'} 2022-09-12T20:12:14.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b951d71272db061-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:12:14.923 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:14.924 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:14.924 DEBUG:boto:Method: GET 2022-09-12T20:12:14.925 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:12:14.926 DEBUG:boto:Data: 2022-09-12T20:12:14.926 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.927 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.927 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.928 DEBUG:boto:Params: {} 2022-09-12T20:12:14.929 DEBUG:boto:Token: None 2022-09-12T20:12:14.929 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj38 2022-09-12T20:12:14.930 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5DHyCNx0P3pqrRDEh2dhoxkAhzw= 2022-09-12T20:12:14.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5DHyCNx0P3pqrRDEh2dhoxkAhzw=', 'Content-Length': '0'} 2022-09-12T20:12:14.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ea755e1bd53437c-00631f929e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.933 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:14.934 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:14.934 DEBUG:boto:Method: GET 2022-09-12T20:12:14.935 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:12:14.935 DEBUG:boto:Data: 2022-09-12T20:12:14.936 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.937 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:14.937 DEBUG:boto:Port: 8000 2022-09-12T20:12:14.938 DEBUG:boto:Params: {} 2022-09-12T20:12:14.938 DEBUG:boto:Token: None 2022-09-12T20:12:14.939 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/obj38 2022-09-12T20:12:14.940 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5DHyCNx0P3pqrRDEh2dhoxkAhzw= 2022-09-12T20:12:14.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5DHyCNx0P3pqrRDEh2dhoxkAhzw=', 'Content-Length': '0'} 2022-09-12T20:12:14.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048e44c0164e40e4c-00631f929e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:14.986 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:14.988 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:14.989 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:12:14.990 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:12:14.990 DEBUG:boto:Method: GET 2022-09-12T20:12:14.991 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:12:14.991 DEBUG:boto:Data: 2022-09-12T20:12:14.992 DEBUG:boto:Headers: {} 2022-09-12T20:12:14.993 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:14.993 DEBUG:boto:Port: 8001 2022-09-12T20:12:14.994 DEBUG:boto:Params: {} 2022-09-12T20:12:14.995 DEBUG:boto:Token: None 2022-09-12T20:12:14.996 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:14 GMT /jdpiak-75/ 2022-09-12T20:12:14.996 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk= 2022-09-12T20:12:14.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TWGN4kLWJiS4f8nTkUNX8L8VEBk=', 'Content-Length': '0'} 2022-09-12T20:12:15.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a786dfba719d6c6-00631f929e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.001 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:15.002 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:15.002 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:15.003 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:12:15.003 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:12:15.004 DEBUG:boto:Method: GET 2022-09-12T20:12:15.004 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:12:15.005 DEBUG:boto:Data: 2022-09-12T20:12:15.005 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.006 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.006 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.007 DEBUG:boto:Params: {} 2022-09-12T20:12:15.008 DEBUG:boto:Token: None 2022-09-12T20:12:15.008 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/ 2022-09-12T20:12:15.009 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs= 2022-09-12T20:12:15.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs=', 'Content-Length': '0'} 2022-09-12T20:12:15.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c316056aa931f30-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.014 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:15.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:12:15.016 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:15.016 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:15.017 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:15.017 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:15.018 DEBUG:boto:Method: GET 2022-09-12T20:12:15.019 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:15.019 DEBUG:boto:Data: 2022-09-12T20:12:15.020 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.021 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.021 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.022 DEBUG:boto:Params: {} 2022-09-12T20:12:15.023 DEBUG:boto:Token: None 2022-09-12T20:12:15.023 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:15.024 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q/fRDKiYtp5hLfRwAN1j9KLYZxk= 2022-09-12T20:12:15.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q/fRDKiYtp5hLfRwAN1j9KLYZxk=', 'Content-Length': '0'} 2022-09-12T20:12:15.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067507b080c3a5a87-00631f929f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.028 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:15.029 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:15.029 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:15.030 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:15.030 DEBUG:boto:Method: GET 2022-09-12T20:12:15.031 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:15.032 DEBUG:boto:Data: 2022-09-12T20:12:15.033 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.034 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.035 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.036 DEBUG:boto:Params: {} 2022-09-12T20:12:15.037 DEBUG:boto:Token: None 2022-09-12T20:12:15.038 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:15.038 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q/fRDKiYtp5hLfRwAN1j9KLYZxk= 2022-09-12T20:12:15.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q/fRDKiYtp5hLfRwAN1j9KLYZxk=', 'Content-Length': '0'} 2022-09-12T20:12:15.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001981a1f4355949fd-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.042 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:15.043 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:15.043 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:15.044 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:15.044 DEBUG:boto:Method: HEAD 2022-09-12T20:12:15.045 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:15.046 DEBUG:boto:Data: 2022-09-12T20:12:15.046 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.047 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.048 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.050 DEBUG:boto:Params: {} 2022-09-12T20:12:15.051 DEBUG:boto:Token: None 2022-09-12T20:12:15.051 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:15.052 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l3lJZC5ON9FBDqyqn15xyKDgASE= 2022-09-12T20:12:15.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l3lJZC5ON9FBDqyqn15xyKDgASE=', 'Content-Length': '0'} 2022-09-12T20:12:15.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f8e9045f30ade6f-00631f929f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.055 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:15.056 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:15.056 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:15.057 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:15.057 DEBUG:boto:Method: HEAD 2022-09-12T20:12:15.058 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:15.059 DEBUG:boto:Data: 2022-09-12T20:12:15.059 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.060 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.061 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.061 DEBUG:boto:Params: {} 2022-09-12T20:12:15.062 DEBUG:boto:Token: None 2022-09-12T20:12:15.063 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:15.063 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l3lJZC5ON9FBDqyqn15xyKDgASE= 2022-09-12T20:12:15.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l3lJZC5ON9FBDqyqn15xyKDgASE=', 'Content-Length': '0'} 2022-09-12T20:12:15.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005855a3095b305e73-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:12:15.068 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:15.068 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:15.069 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:15.069 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:15.070 DEBUG:boto:Method: GET 2022-09-12T20:12:15.071 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:15.071 DEBUG:boto:Data: 2022-09-12T20:12:15.072 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.072 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.073 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.074 DEBUG:boto:Params: {} 2022-09-12T20:12:15.074 DEBUG:boto:Token: None 2022-09-12T20:12:15.075 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:15.076 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q/fRDKiYtp5hLfRwAN1j9KLYZxk= 2022-09-12T20:12:15.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q/fRDKiYtp5hLfRwAN1j9KLYZxk=', 'Content-Length': '0'} 2022-09-12T20:12:15.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3314f1267bacc60-00631f929f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.122 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:15.123 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:15.123 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:15.124 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:15.124 DEBUG:boto:Method: GET 2022-09-12T20:12:15.125 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:15.125 DEBUG:boto:Data: 2022-09-12T20:12:15.126 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.127 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.127 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.128 DEBUG:boto:Params: {} 2022-09-12T20:12:15.128 DEBUG:boto:Token: None 2022-09-12T20:12:15.129 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:15.129 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q/fRDKiYtp5hLfRwAN1j9KLYZxk= 2022-09-12T20:12:15.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q/fRDKiYtp5hLfRwAN1j9KLYZxk=', 'Content-Length': '0'} 2022-09-12T20:12:15.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005960d9d5e578dc22-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.133 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:15.134 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:15.134 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:12:15.135 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:12:15.136 DEBUG:boto:Method: GET 2022-09-12T20:12:15.136 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:12:15.137 DEBUG:boto:Data: 2022-09-12T20:12:15.137 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.138 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.138 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.139 DEBUG:boto:Params: {} 2022-09-12T20:12:15.139 DEBUG:boto:Token: None 2022-09-12T20:12:15.140 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/ 2022-09-12T20:12:15.141 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs= 2022-09-12T20:12:15.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs=', 'Content-Length': '0'} 2022-09-12T20:12:15.144 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a9e21ef5d1523bf-00631f929f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.145 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:15.145 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:15.146 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:15.147 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:12:15.147 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:12:15.148 DEBUG:boto:Method: GET 2022-09-12T20:12:15.148 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:12:15.149 DEBUG:boto:Data: 2022-09-12T20:12:15.149 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.150 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.150 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.151 DEBUG:boto:Params: {} 2022-09-12T20:12:15.151 DEBUG:boto:Token: None 2022-09-12T20:12:15.152 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/ 2022-09-12T20:12:15.153 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs= 2022-09-12T20:12:15.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs=', 'Content-Length': '0'} 2022-09-12T20:12:15.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc223e2ce7b9ce2c-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.158 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:15.159 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:15.160 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:15.160 DEBUG:boto:Method: HEAD 2022-09-12T20:12:15.161 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:12:15.161 DEBUG:boto:Data: 2022-09-12T20:12:15.162 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.162 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.163 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.163 DEBUG:boto:Params: {} 2022-09-12T20:12:15.164 DEBUG:boto:Token: None 2022-09-12T20:12:15.164 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj44 2022-09-12T20:12:15.165 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UoxitBykWtI7MMInMuQrMHF7WBs= 2022-09-12T20:12:15.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UoxitBykWtI7MMInMuQrMHF7WBs=', 'Content-Length': '0'} 2022-09-12T20:12:15.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000debf09bba5377950-00631f929f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.168 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:15.169 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:15.169 DEBUG:boto:Method: HEAD 2022-09-12T20:12:15.170 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:12:15.170 DEBUG:boto:Data: 2022-09-12T20:12:15.171 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.171 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.172 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.172 DEBUG:boto:Params: {} 2022-09-12T20:12:15.173 DEBUG:boto:Token: None 2022-09-12T20:12:15.173 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj44 2022-09-12T20:12:15.174 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UoxitBykWtI7MMInMuQrMHF7WBs= 2022-09-12T20:12:15.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UoxitBykWtI7MMInMuQrMHF7WBs=', 'Content-Length': '0'} 2022-09-12T20:12:15.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b974e637ed9bff9b-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:12:15.178 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:15.178 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:15.179 DEBUG:boto:Method: GET 2022-09-12T20:12:15.179 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:12:15.180 DEBUG:boto:Data: 2022-09-12T20:12:15.180 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.181 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.182 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.182 DEBUG:boto:Params: {} 2022-09-12T20:12:15.183 DEBUG:boto:Token: None 2022-09-12T20:12:15.183 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj44 2022-09-12T20:12:15.184 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yKee4ma4lsfbRokWY3nSX1aPMDI= 2022-09-12T20:12:15.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yKee4ma4lsfbRokWY3nSX1aPMDI=', 'Content-Length': '0'} 2022-09-12T20:12:15.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024190758bdff1167-00631f929f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.187 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:15.188 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:15.188 DEBUG:boto:Method: GET 2022-09-12T20:12:15.189 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:12:15.189 DEBUG:boto:Data: 2022-09-12T20:12:15.190 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.190 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.191 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.192 DEBUG:boto:Params: {} 2022-09-12T20:12:15.192 DEBUG:boto:Token: None 2022-09-12T20:12:15.193 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj44 2022-09-12T20:12:15.193 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yKee4ma4lsfbRokWY3nSX1aPMDI= 2022-09-12T20:12:15.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yKee4ma4lsfbRokWY3nSX1aPMDI=', 'Content-Length': '0'} 2022-09-12T20:12:15.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000635bb7582756b598-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.197 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:15.198 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:15.198 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:12:15.199 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:12:15.199 DEBUG:boto:Method: GET 2022-09-12T20:12:15.200 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:12:15.200 DEBUG:boto:Data: 2022-09-12T20:12:15.201 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.201 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.202 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.202 DEBUG:boto:Params: {} 2022-09-12T20:12:15.203 DEBUG:boto:Token: None 2022-09-12T20:12:15.204 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/ 2022-09-12T20:12:15.204 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs= 2022-09-12T20:12:15.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs=', 'Content-Length': '0'} 2022-09-12T20:12:15.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c88a77485313b100-00631f929f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.209 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:15.209 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:15.210 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:15.211 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:12:15.211 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:12:15.212 DEBUG:boto:Method: GET 2022-09-12T20:12:15.212 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:12:15.213 DEBUG:boto:Data: 2022-09-12T20:12:15.213 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.214 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.214 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.215 DEBUG:boto:Params: {} 2022-09-12T20:12:15.215 DEBUG:boto:Token: None 2022-09-12T20:12:15.216 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/ 2022-09-12T20:12:15.217 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs= 2022-09-12T20:12:15.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs=', 'Content-Length': '0'} 2022-09-12T20:12:15.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076ccee83b54c4042-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.223 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:15.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:12:15.224 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:15.224 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:15.225 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:15.226 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:15.226 DEBUG:boto:Method: GET 2022-09-12T20:12:15.227 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:15.227 DEBUG:boto:Data: 2022-09-12T20:12:15.228 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.228 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.229 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.229 DEBUG:boto:Params: {} 2022-09-12T20:12:15.230 DEBUG:boto:Token: None 2022-09-12T20:12:15.231 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:15.231 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EXWzHowcupHRiS/iBD+q3VrG6QM= 2022-09-12T20:12:15.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EXWzHowcupHRiS/iBD+q3VrG6QM=', 'Content-Length': '0'} 2022-09-12T20:12:15.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002de1263ef9a7fd91-00631f929f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.278 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:15.279 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:15.280 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:15.280 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:15.281 DEBUG:boto:Method: GET 2022-09-12T20:12:15.281 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:15.282 DEBUG:boto:Data: 2022-09-12T20:12:15.282 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.283 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.283 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.284 DEBUG:boto:Params: {} 2022-09-12T20:12:15.284 DEBUG:boto:Token: None 2022-09-12T20:12:15.285 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:15.286 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EXWzHowcupHRiS/iBD+q3VrG6QM= 2022-09-12T20:12:15.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EXWzHowcupHRiS/iBD+q3VrG6QM=', 'Content-Length': '0'} 2022-09-12T20:12:15.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c4c6e19aa713b52-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.289 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:15.290 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:15.290 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:15.291 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:15.291 DEBUG:boto:Method: HEAD 2022-09-12T20:12:15.292 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:15.292 DEBUG:boto:Data: 2022-09-12T20:12:15.293 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.293 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.294 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.294 DEBUG:boto:Params: {} 2022-09-12T20:12:15.295 DEBUG:boto:Token: None 2022-09-12T20:12:15.296 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:15.296 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JOt44GOvCN33/4k2dA79Egvkv7E= 2022-09-12T20:12:15.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JOt44GOvCN33/4k2dA79Egvkv7E=', 'Content-Length': '0'} 2022-09-12T20:12:15.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d12f8dfacd69e29d-00631f929f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.300 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:15.300 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:15.301 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:15.301 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:15.302 DEBUG:boto:Method: HEAD 2022-09-12T20:12:15.302 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:15.303 DEBUG:boto:Data: 2022-09-12T20:12:15.303 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.304 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.304 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.305 DEBUG:boto:Params: {} 2022-09-12T20:12:15.305 DEBUG:boto:Token: None 2022-09-12T20:12:15.306 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:15.307 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JOt44GOvCN33/4k2dA79Egvkv7E= 2022-09-12T20:12:15.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JOt44GOvCN33/4k2dA79Egvkv7E=', 'Content-Length': '0'} 2022-09-12T20:12:15.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000116c2ee6e546174b-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:12:15.311 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:15.311 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:15.312 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:15.312 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:15.313 DEBUG:boto:Method: GET 2022-09-12T20:12:15.313 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:15.314 DEBUG:boto:Data: 2022-09-12T20:12:15.314 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.315 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.315 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.316 DEBUG:boto:Params: {} 2022-09-12T20:12:15.317 DEBUG:boto:Token: None 2022-09-12T20:12:15.317 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:15.318 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EXWzHowcupHRiS/iBD+q3VrG6QM= 2022-09-12T20:12:15.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EXWzHowcupHRiS/iBD+q3VrG6QM=', 'Content-Length': '0'} 2022-09-12T20:12:15.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9a4eee50ff00f66-00631f929f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.321 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:15.322 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:15.323 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:15.323 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:15.324 DEBUG:boto:Method: GET 2022-09-12T20:12:15.324 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:15.325 DEBUG:boto:Data: 2022-09-12T20:12:15.325 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.326 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.327 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.327 DEBUG:boto:Params: {} 2022-09-12T20:12:15.328 DEBUG:boto:Token: None 2022-09-12T20:12:15.328 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:15.329 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EXWzHowcupHRiS/iBD+q3VrG6QM= 2022-09-12T20:12:15.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EXWzHowcupHRiS/iBD+q3VrG6QM=', 'Content-Length': '0'} 2022-09-12T20:12:15.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7ed4c9598cc7428-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.333 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:15.333 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:15.334 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:12:15.335 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:12:15.335 DEBUG:boto:Method: GET 2022-09-12T20:12:15.336 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:12:15.336 DEBUG:boto:Data: 2022-09-12T20:12:15.337 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.338 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.338 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.339 DEBUG:boto:Params: {} 2022-09-12T20:12:15.339 DEBUG:boto:Token: None 2022-09-12T20:12:15.340 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/ 2022-09-12T20:12:15.341 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs= 2022-09-12T20:12:15.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs=', 'Content-Length': '0'} 2022-09-12T20:12:15.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c25f49e5725ee30e-00631f929f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.345 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:15.346 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:15.346 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:15.347 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:12:15.348 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:12:15.348 DEBUG:boto:Method: GET 2022-09-12T20:12:15.349 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:12:15.349 DEBUG:boto:Data: 2022-09-12T20:12:15.350 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.350 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.351 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.351 DEBUG:boto:Params: {} 2022-09-12T20:12:15.352 DEBUG:boto:Token: None 2022-09-12T20:12:15.352 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/ 2022-09-12T20:12:15.353 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs= 2022-09-12T20:12:15.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs=', 'Content-Length': '0'} 2022-09-12T20:12:15.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b14a77985ac451b-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.358 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:15.359 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:15.359 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:15.360 DEBUG:boto:Method: HEAD 2022-09-12T20:12:15.360 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:12:15.361 DEBUG:boto:Data: 2022-09-12T20:12:15.361 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.362 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.362 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.363 DEBUG:boto:Params: {} 2022-09-12T20:12:15.363 DEBUG:boto:Token: None 2022-09-12T20:12:15.364 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj46 2022-09-12T20:12:15.365 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D6s6Tcesf4ogTEhvHdtWgc4wcic= 2022-09-12T20:12:15.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D6s6Tcesf4ogTEhvHdtWgc4wcic=', 'Content-Length': '0'} 2022-09-12T20:12:15.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cc02ccf467c98a0-00631f929f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.368 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:15.368 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:15.369 DEBUG:boto:Method: HEAD 2022-09-12T20:12:15.369 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:12:15.370 DEBUG:boto:Data: 2022-09-12T20:12:15.370 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.371 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.371 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.372 DEBUG:boto:Params: {} 2022-09-12T20:12:15.372 DEBUG:boto:Token: None 2022-09-12T20:12:15.373 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj46 2022-09-12T20:12:15.374 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D6s6Tcesf4ogTEhvHdtWgc4wcic= 2022-09-12T20:12:15.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D6s6Tcesf4ogTEhvHdtWgc4wcic=', 'Content-Length': '0'} 2022-09-12T20:12:15.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfc0b8be4c425891-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:12:15.377 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:15.378 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:15.379 DEBUG:boto:Method: GET 2022-09-12T20:12:15.379 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:12:15.380 DEBUG:boto:Data: 2022-09-12T20:12:15.380 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.381 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.381 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.382 DEBUG:boto:Params: {} 2022-09-12T20:12:15.382 DEBUG:boto:Token: None 2022-09-12T20:12:15.383 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj46 2022-09-12T20:12:15.384 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cp/IruQXjNRvipMnZfWK0O4Rk5Q= 2022-09-12T20:12:15.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cp/IruQXjNRvipMnZfWK0O4Rk5Q=', 'Content-Length': '0'} 2022-09-12T20:12:15.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062e36294712de874-00631f929f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.387 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:15.388 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:15.388 DEBUG:boto:Method: GET 2022-09-12T20:12:15.389 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:12:15.389 DEBUG:boto:Data: 2022-09-12T20:12:15.390 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.390 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.391 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.391 DEBUG:boto:Params: {} 2022-09-12T20:12:15.392 DEBUG:boto:Token: None 2022-09-12T20:12:15.392 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj46 2022-09-12T20:12:15.393 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cp/IruQXjNRvipMnZfWK0O4Rk5Q= 2022-09-12T20:12:15.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cp/IruQXjNRvipMnZfWK0O4Rk5Q=', 'Content-Length': '0'} 2022-09-12T20:12:15.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d96d6c55d81d7a6-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.397 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:15.397 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:15.398 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:12:15.398 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:12:15.399 DEBUG:boto:Method: GET 2022-09-12T20:12:15.399 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:12:15.400 DEBUG:boto:Data: 2022-09-12T20:12:15.400 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.401 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.401 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.402 DEBUG:boto:Params: {} 2022-09-12T20:12:15.403 DEBUG:boto:Token: None 2022-09-12T20:12:15.403 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/ 2022-09-12T20:12:15.404 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs= 2022-09-12T20:12:15.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs=', 'Content-Length': '0'} 2022-09-12T20:12:15.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7a170c81163aaf2-00631f929f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.450 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:15.451 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:15.452 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:15.452 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:12:15.453 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:12:15.453 DEBUG:boto:Method: GET 2022-09-12T20:12:15.454 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:12:15.454 DEBUG:boto:Data: 2022-09-12T20:12:15.455 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.456 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.456 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.457 DEBUG:boto:Params: {} 2022-09-12T20:12:15.457 DEBUG:boto:Token: None 2022-09-12T20:12:15.458 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/ 2022-09-12T20:12:15.458 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs= 2022-09-12T20:12:15.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs=', 'Content-Length': '0'} 2022-09-12T20:12:15.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052c4c4801d6fa87e-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.464 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:15.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:12:15.465 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:15.466 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:15.466 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:15.467 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:15.468 DEBUG:boto:Method: GET 2022-09-12T20:12:15.468 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:15.469 DEBUG:boto:Data: 2022-09-12T20:12:15.469 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.470 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.471 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.471 DEBUG:boto:Params: {} 2022-09-12T20:12:15.472 DEBUG:boto:Token: None 2022-09-12T20:12:15.472 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:15.473 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IeUXiTgKV0nFs/rEZwU1M+s0UVg= 2022-09-12T20:12:15.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IeUXiTgKV0nFs/rEZwU1M+s0UVg=', 'Content-Length': '0'} 2022-09-12T20:12:15.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091a1fb8f7c8f86e7-00631f929f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.477 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:15.477 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:15.478 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:15.478 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:15.479 DEBUG:boto:Method: GET 2022-09-12T20:12:15.479 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:15.480 DEBUG:boto:Data: 2022-09-12T20:12:15.480 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.481 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.481 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.482 DEBUG:boto:Params: {} 2022-09-12T20:12:15.483 DEBUG:boto:Token: None 2022-09-12T20:12:15.483 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:15.484 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IeUXiTgKV0nFs/rEZwU1M+s0UVg= 2022-09-12T20:12:15.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IeUXiTgKV0nFs/rEZwU1M+s0UVg=', 'Content-Length': '0'} 2022-09-12T20:12:15.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009548370f9b0f3da5-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.530 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:15.531 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:15.531 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:15.532 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:15.532 DEBUG:boto:Method: HEAD 2022-09-12T20:12:15.533 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:15.533 DEBUG:boto:Data: 2022-09-12T20:12:15.534 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.534 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.535 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.535 DEBUG:boto:Params: {} 2022-09-12T20:12:15.536 DEBUG:boto:Token: None 2022-09-12T20:12:15.537 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:15.537 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SiyktFS0sHF2rsr/ZcsL5q2BWZI= 2022-09-12T20:12:15.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SiyktFS0sHF2rsr/ZcsL5q2BWZI=', 'Content-Length': '0'} 2022-09-12T20:12:15.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9cfb4665f5e5b18-00631f929f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.541 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:15.541 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:15.542 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:15.542 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:15.543 DEBUG:boto:Method: HEAD 2022-09-12T20:12:15.543 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:15.544 DEBUG:boto:Data: 2022-09-12T20:12:15.544 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.545 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.545 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.546 DEBUG:boto:Params: {} 2022-09-12T20:12:15.546 DEBUG:boto:Token: None 2022-09-12T20:12:15.547 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:15.548 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SiyktFS0sHF2rsr/ZcsL5q2BWZI= 2022-09-12T20:12:15.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SiyktFS0sHF2rsr/ZcsL5q2BWZI=', 'Content-Length': '0'} 2022-09-12T20:12:15.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000449ea62e84e721ec-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:12:15.552 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:15.552 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:15.553 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:15.554 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:15.554 DEBUG:boto:Method: GET 2022-09-12T20:12:15.555 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:15.555 DEBUG:boto:Data: 2022-09-12T20:12:15.556 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.557 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.557 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.558 DEBUG:boto:Params: {} 2022-09-12T20:12:15.558 DEBUG:boto:Token: None 2022-09-12T20:12:15.559 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:15.559 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IeUXiTgKV0nFs/rEZwU1M+s0UVg= 2022-09-12T20:12:15.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IeUXiTgKV0nFs/rEZwU1M+s0UVg=', 'Content-Length': '0'} 2022-09-12T20:12:15.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062bfc5242d5215f0-00631f929f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.563 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:15.564 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:15.564 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:15.565 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:15.565 DEBUG:boto:Method: GET 2022-09-12T20:12:15.566 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:15.567 DEBUG:boto:Data: 2022-09-12T20:12:15.567 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.568 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.568 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.569 DEBUG:boto:Params: {} 2022-09-12T20:12:15.569 DEBUG:boto:Token: None 2022-09-12T20:12:15.570 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:15.570 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IeUXiTgKV0nFs/rEZwU1M+s0UVg= 2022-09-12T20:12:15.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IeUXiTgKV0nFs/rEZwU1M+s0UVg=', 'Content-Length': '0'} 2022-09-12T20:12:15.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093c0faec7631f0f6-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.574 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:15.575 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:15.575 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:12:15.576 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:12:15.576 DEBUG:boto:Method: GET 2022-09-12T20:12:15.577 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:12:15.578 DEBUG:boto:Data: 2022-09-12T20:12:15.578 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.579 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.579 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.580 DEBUG:boto:Params: {} 2022-09-12T20:12:15.580 DEBUG:boto:Token: None 2022-09-12T20:12:15.581 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/ 2022-09-12T20:12:15.581 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs= 2022-09-12T20:12:15.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs=', 'Content-Length': '0'} 2022-09-12T20:12:15.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015c89318ef40a838-00631f929f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.586 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:15.586 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:15.587 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:15.588 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:12:15.588 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:12:15.589 DEBUG:boto:Method: GET 2022-09-12T20:12:15.589 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:12:15.590 DEBUG:boto:Data: 2022-09-12T20:12:15.590 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.591 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.591 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.592 DEBUG:boto:Params: {} 2022-09-12T20:12:15.592 DEBUG:boto:Token: None 2022-09-12T20:12:15.593 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/ 2022-09-12T20:12:15.593 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs= 2022-09-12T20:12:15.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs=', 'Content-Length': '0'} 2022-09-12T20:12:15.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000960eeceb038f2264-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.599 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:15.600 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:15.600 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:15.601 DEBUG:boto:Method: HEAD 2022-09-12T20:12:15.601 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:12:15.602 DEBUG:boto:Data: 2022-09-12T20:12:15.602 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.603 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.603 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.604 DEBUG:boto:Params: {} 2022-09-12T20:12:15.604 DEBUG:boto:Token: None 2022-09-12T20:12:15.605 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj51 2022-09-12T20:12:15.606 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A34PoAu5lLn9rmSyZshWMWlpano= 2022-09-12T20:12:15.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A34PoAu5lLn9rmSyZshWMWlpano=', 'Content-Length': '0'} 2022-09-12T20:12:15.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060cece35cf4ea781-00631f929f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.609 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:15.609 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:15.610 DEBUG:boto:Method: HEAD 2022-09-12T20:12:15.610 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:12:15.611 DEBUG:boto:Data: 2022-09-12T20:12:15.612 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.612 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.613 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.613 DEBUG:boto:Params: {} 2022-09-12T20:12:15.614 DEBUG:boto:Token: None 2022-09-12T20:12:15.614 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj51 2022-09-12T20:12:15.615 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A34PoAu5lLn9rmSyZshWMWlpano= 2022-09-12T20:12:15.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A34PoAu5lLn9rmSyZshWMWlpano=', 'Content-Length': '0'} 2022-09-12T20:12:15.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000746936267d211a47-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:12:15.619 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:15.619 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:15.620 DEBUG:boto:Method: GET 2022-09-12T20:12:15.620 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:12:15.621 DEBUG:boto:Data: 2022-09-12T20:12:15.622 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.622 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.623 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.623 DEBUG:boto:Params: {} 2022-09-12T20:12:15.624 DEBUG:boto:Token: None 2022-09-12T20:12:15.624 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj51 2022-09-12T20:12:15.625 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qxPw0emiHqDHf0lGgAMS61Xs6Tk= 2022-09-12T20:12:15.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qxPw0emiHqDHf0lGgAMS61Xs6Tk=', 'Content-Length': '0'} 2022-09-12T20:12:15.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d55020f72a0b57b0-00631f929f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.629 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:15.629 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:15.630 DEBUG:boto:Method: GET 2022-09-12T20:12:15.630 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:12:15.631 DEBUG:boto:Data: 2022-09-12T20:12:15.631 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.632 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.633 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.633 DEBUG:boto:Params: {} 2022-09-12T20:12:15.634 DEBUG:boto:Token: None 2022-09-12T20:12:15.634 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj51 2022-09-12T20:12:15.635 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qxPw0emiHqDHf0lGgAMS61Xs6Tk= 2022-09-12T20:12:15.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qxPw0emiHqDHf0lGgAMS61Xs6Tk=', 'Content-Length': '0'} 2022-09-12T20:12:15.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b0a74154a6cfc83-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.678 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:15.679 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:15.679 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:12:15.680 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:12:15.680 DEBUG:boto:Method: GET 2022-09-12T20:12:15.681 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:12:15.681 DEBUG:boto:Data: 2022-09-12T20:12:15.682 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.683 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.684 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.684 DEBUG:boto:Params: {} 2022-09-12T20:12:15.685 DEBUG:boto:Token: None 2022-09-12T20:12:15.685 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/ 2022-09-12T20:12:15.686 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs= 2022-09-12T20:12:15.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs=', 'Content-Length': '0'} 2022-09-12T20:12:15.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6be1b279adbd88d-00631f929f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.690 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:15.691 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:15.692 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:15.692 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:12:15.693 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:12:15.693 DEBUG:boto:Method: GET 2022-09-12T20:12:15.694 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:12:15.694 DEBUG:boto:Data: 2022-09-12T20:12:15.695 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.695 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.696 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.697 DEBUG:boto:Params: {} 2022-09-12T20:12:15.697 DEBUG:boto:Token: None 2022-09-12T20:12:15.698 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/ 2022-09-12T20:12:15.698 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs= 2022-09-12T20:12:15.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs=', 'Content-Length': '0'} 2022-09-12T20:12:15.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa21423b52856a67-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.704 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:15.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:12:15.705 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:15.706 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:15.706 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:15.707 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:15.707 DEBUG:boto:Method: GET 2022-09-12T20:12:15.708 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:15.709 DEBUG:boto:Data: 2022-09-12T20:12:15.710 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.710 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.711 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.711 DEBUG:boto:Params: {} 2022-09-12T20:12:15.712 DEBUG:boto:Token: None 2022-09-12T20:12:15.712 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:15.713 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X1Tb1sok5znlUjpI9qY35tWgOtk= 2022-09-12T20:12:15.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X1Tb1sok5znlUjpI9qY35tWgOtk=', 'Content-Length': '0'} 2022-09-12T20:12:15.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005664811b671fc3a9-00631f929f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.717 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:15.717 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:15.718 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:15.718 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:15.719 DEBUG:boto:Method: GET 2022-09-12T20:12:15.719 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:15.720 DEBUG:boto:Data: 2022-09-12T20:12:15.720 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.721 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.721 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.722 DEBUG:boto:Params: {} 2022-09-12T20:12:15.723 DEBUG:boto:Token: None 2022-09-12T20:12:15.723 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:15.724 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X1Tb1sok5znlUjpI9qY35tWgOtk= 2022-09-12T20:12:15.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X1Tb1sok5znlUjpI9qY35tWgOtk=', 'Content-Length': '0'} 2022-09-12T20:12:15.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cff1859d33335c2a-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.728 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:15.728 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:15.729 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:15.729 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:15.730 DEBUG:boto:Method: HEAD 2022-09-12T20:12:15.730 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:15.731 DEBUG:boto:Data: 2022-09-12T20:12:15.731 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.732 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.732 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.733 DEBUG:boto:Params: {} 2022-09-12T20:12:15.734 DEBUG:boto:Token: None 2022-09-12T20:12:15.734 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:15.735 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UlegJbLzI+JcUUkb3wewjGnpR80= 2022-09-12T20:12:15.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UlegJbLzI+JcUUkb3wewjGnpR80=', 'Content-Length': '0'} 2022-09-12T20:12:15.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0f2a2840f4fecf8-00631f929f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.738 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:15.739 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:15.739 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:15.740 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:15.740 DEBUG:boto:Method: HEAD 2022-09-12T20:12:15.741 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:15.741 DEBUG:boto:Data: 2022-09-12T20:12:15.742 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.742 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.743 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.743 DEBUG:boto:Params: {} 2022-09-12T20:12:15.744 DEBUG:boto:Token: None 2022-09-12T20:12:15.744 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:15.745 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UlegJbLzI+JcUUkb3wewjGnpR80= 2022-09-12T20:12:15.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UlegJbLzI+JcUUkb3wewjGnpR80=', 'Content-Length': '0'} 2022-09-12T20:12:15.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020eb040e4702fb1b-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:12:15.749 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:15.749 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:15.750 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:15.750 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:15.751 DEBUG:boto:Method: GET 2022-09-12T20:12:15.752 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:15.752 DEBUG:boto:Data: 2022-09-12T20:12:15.753 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.753 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.754 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.754 DEBUG:boto:Params: {} 2022-09-12T20:12:15.755 DEBUG:boto:Token: None 2022-09-12T20:12:15.755 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:15.756 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X1Tb1sok5znlUjpI9qY35tWgOtk= 2022-09-12T20:12:15.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X1Tb1sok5znlUjpI9qY35tWgOtk=', 'Content-Length': '0'} 2022-09-12T20:12:15.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003699601544818204-00631f929f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.760 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:15.760 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:15.761 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:15.761 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:15.762 DEBUG:boto:Method: GET 2022-09-12T20:12:15.762 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:15.763 DEBUG:boto:Data: 2022-09-12T20:12:15.764 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.764 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.765 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.765 DEBUG:boto:Params: {} 2022-09-12T20:12:15.766 DEBUG:boto:Token: None 2022-09-12T20:12:15.766 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:15.767 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X1Tb1sok5znlUjpI9qY35tWgOtk= 2022-09-12T20:12:15.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X1Tb1sok5znlUjpI9qY35tWgOtk=', 'Content-Length': '0'} 2022-09-12T20:12:15.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c90ed51ab55987e1-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.810 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:15.811 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:15.811 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:12:15.812 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:12:15.813 DEBUG:boto:Method: GET 2022-09-12T20:12:15.813 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:12:15.814 DEBUG:boto:Data: 2022-09-12T20:12:15.814 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.815 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.815 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.816 DEBUG:boto:Params: {} 2022-09-12T20:12:15.817 DEBUG:boto:Token: None 2022-09-12T20:12:15.817 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/ 2022-09-12T20:12:15.818 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs= 2022-09-12T20:12:15.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs=', 'Content-Length': '0'} 2022-09-12T20:12:15.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000214f5a433c2105d4-00631f929f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.822 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:15.824 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:15.825 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:15.825 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:12:15.826 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:12:15.826 DEBUG:boto:Method: GET 2022-09-12T20:12:15.827 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:12:15.827 DEBUG:boto:Data: 2022-09-12T20:12:15.828 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.828 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.829 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.829 DEBUG:boto:Params: {} 2022-09-12T20:12:15.830 DEBUG:boto:Token: None 2022-09-12T20:12:15.831 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/ 2022-09-12T20:12:15.832 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs= 2022-09-12T20:12:15.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs=', 'Content-Length': '0'} 2022-09-12T20:12:15.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002adbaedbcdfb8d34-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.838 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:15.839 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:15.839 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:15.840 DEBUG:boto:Method: HEAD 2022-09-12T20:12:15.841 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:12:15.841 DEBUG:boto:Data: 2022-09-12T20:12:15.842 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.842 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.843 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.843 DEBUG:boto:Params: {} 2022-09-12T20:12:15.844 DEBUG:boto:Token: None 2022-09-12T20:12:15.844 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj55 2022-09-12T20:12:15.845 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qD8xDQuDzIQxdgkSD+l3DnAZmWE= 2022-09-12T20:12:15.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qD8xDQuDzIQxdgkSD+l3DnAZmWE=', 'Content-Length': '0'} 2022-09-12T20:12:15.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f07ef9307e41c9e-00631f929f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.848 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:15.849 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:15.849 DEBUG:boto:Method: HEAD 2022-09-12T20:12:15.850 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:12:15.850 DEBUG:boto:Data: 2022-09-12T20:12:15.851 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.851 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.852 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.852 DEBUG:boto:Params: {} 2022-09-12T20:12:15.853 DEBUG:boto:Token: None 2022-09-12T20:12:15.853 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj55 2022-09-12T20:12:15.854 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qD8xDQuDzIQxdgkSD+l3DnAZmWE= 2022-09-12T20:12:15.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qD8xDQuDzIQxdgkSD+l3DnAZmWE=', 'Content-Length': '0'} 2022-09-12T20:12:15.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c966b755c7851e49-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:12:15.858 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:15.859 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:15.859 DEBUG:boto:Method: GET 2022-09-12T20:12:15.860 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:12:15.860 DEBUG:boto:Data: 2022-09-12T20:12:15.861 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.861 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.862 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.863 DEBUG:boto:Params: {} 2022-09-12T20:12:15.863 DEBUG:boto:Token: None 2022-09-12T20:12:15.864 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj55 2022-09-12T20:12:15.864 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0XTlrPNyC2jbgDjaYKcl/xObl/U= 2022-09-12T20:12:15.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0XTlrPNyC2jbgDjaYKcl/xObl/U=', 'Content-Length': '0'} 2022-09-12T20:12:15.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000605560c98936c1cd-00631f929f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.868 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:15.868 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:15.869 DEBUG:boto:Method: GET 2022-09-12T20:12:15.869 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:12:15.870 DEBUG:boto:Data: 2022-09-12T20:12:15.871 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.871 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.872 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.872 DEBUG:boto:Params: {} 2022-09-12T20:12:15.873 DEBUG:boto:Token: None 2022-09-12T20:12:15.873 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj55 2022-09-12T20:12:15.874 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0XTlrPNyC2jbgDjaYKcl/xObl/U= 2022-09-12T20:12:15.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0XTlrPNyC2jbgDjaYKcl/xObl/U=', 'Content-Length': '0'} 2022-09-12T20:12:15.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016d895d3ef47245d-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.877 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:15.878 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:15.879 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:12:15.879 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:12:15.880 DEBUG:boto:Method: GET 2022-09-12T20:12:15.880 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:12:15.881 DEBUG:boto:Data: 2022-09-12T20:12:15.881 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.882 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.882 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.883 DEBUG:boto:Params: {} 2022-09-12T20:12:15.884 DEBUG:boto:Token: None 2022-09-12T20:12:15.884 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/ 2022-09-12T20:12:15.885 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs= 2022-09-12T20:12:15.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs=', 'Content-Length': '0'} 2022-09-12T20:12:15.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecb7e0ae852402e3-00631f929f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.889 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:15.890 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:15.891 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:15.891 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:12:15.892 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:12:15.892 DEBUG:boto:Method: GET 2022-09-12T20:12:15.893 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:12:15.893 DEBUG:boto:Data: 2022-09-12T20:12:15.894 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.895 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.895 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.895 DEBUG:boto:Params: {} 2022-09-12T20:12:15.896 DEBUG:boto:Token: None 2022-09-12T20:12:15.897 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/ 2022-09-12T20:12:15.897 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs= 2022-09-12T20:12:15.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DPv1LwcMLKkobn7VzXrk7wuMYJs=', 'Content-Length': '0'} 2022-09-12T20:12:15.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4a4aec61d0d316d-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.903 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:15.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:12:15.904 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:15.904 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:15.905 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:15.905 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:15.906 DEBUG:boto:Method: GET 2022-09-12T20:12:15.907 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:15.907 DEBUG:boto:Data: 2022-09-12T20:12:15.908 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.908 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.909 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.909 DEBUG:boto:Params: {} 2022-09-12T20:12:15.910 DEBUG:boto:Token: None 2022-09-12T20:12:15.910 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:15.911 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:boldbB420+qYs3RQpJZIE/OPXck= 2022-09-12T20:12:15.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:boldbB420+qYs3RQpJZIE/OPXck=', 'Content-Length': '0'} 2022-09-12T20:12:15.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002778bba130ea4b52-00631f929f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.915 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:15.915 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:15.916 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:15.916 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:15.917 DEBUG:boto:Method: GET 2022-09-12T20:12:15.917 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:15.918 DEBUG:boto:Data: 2022-09-12T20:12:15.918 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.919 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.919 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.920 DEBUG:boto:Params: {} 2022-09-12T20:12:15.920 DEBUG:boto:Token: None 2022-09-12T20:12:15.921 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:15.921 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:boldbB420+qYs3RQpJZIE/OPXck= 2022-09-12T20:12:15.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:boldbB420+qYs3RQpJZIE/OPXck=', 'Content-Length': '0'} 2022-09-12T20:12:15.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfa9b278dd64e779-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.966 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:15.966 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:15.967 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:15.967 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:15.968 DEBUG:boto:Method: HEAD 2022-09-12T20:12:15.968 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:15.969 DEBUG:boto:Data: 2022-09-12T20:12:15.969 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.970 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.970 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.971 DEBUG:boto:Params: {} 2022-09-12T20:12:15.971 DEBUG:boto:Token: None 2022-09-12T20:12:15.972 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:15.972 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vNOivEp0iLO7tU6ezFxKijsLEnE= 2022-09-12T20:12:15.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vNOivEp0iLO7tU6ezFxKijsLEnE=', 'Content-Length': '0'} 2022-09-12T20:12:15.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec93c93282b0c4e6-00631f929f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.976 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:15.976 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:15.977 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:15.977 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:15.978 DEBUG:boto:Method: HEAD 2022-09-12T20:12:15.978 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:15.979 DEBUG:boto:Data: 2022-09-12T20:12:15.979 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.980 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:15.980 DEBUG:boto:Port: 8000 2022-09-12T20:12:15.981 DEBUG:boto:Params: {} 2022-09-12T20:12:15.981 DEBUG:boto:Token: None 2022-09-12T20:12:15.982 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:15.982 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vNOivEp0iLO7tU6ezFxKijsLEnE= 2022-09-12T20:12:15.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vNOivEp0iLO7tU6ezFxKijsLEnE=', 'Content-Length': '0'} 2022-09-12T20:12:15.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055a72fec4fdb3e51-00631f929f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:15.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:12:15.986 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:15.986 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:15.987 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:15.987 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:15.988 DEBUG:boto:Method: GET 2022-09-12T20:12:15.988 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:15.989 DEBUG:boto:Data: 2022-09-12T20:12:15.989 DEBUG:boto:Headers: {} 2022-09-12T20:12:15.990 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:15.990 DEBUG:boto:Port: 8001 2022-09-12T20:12:15.991 DEBUG:boto:Params: {} 2022-09-12T20:12:15.991 DEBUG:boto:Token: None 2022-09-12T20:12:15.992 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:15 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:15.993 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:boldbB420+qYs3RQpJZIE/OPXck= 2022-09-12T20:12:15.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:boldbB420+qYs3RQpJZIE/OPXck=', 'Content-Length': '0'} 2022-09-12T20:12:15.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d31f133dedeaacf-00631f929f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.038 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:16.038 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:16.039 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:16.040 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:16.040 DEBUG:boto:Method: GET 2022-09-12T20:12:16.041 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:16.041 DEBUG:boto:Data: 2022-09-12T20:12:16.042 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.042 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.043 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.043 DEBUG:boto:Params: {} 2022-09-12T20:12:16.044 DEBUG:boto:Token: None 2022-09-12T20:12:16.044 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:16.045 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hQOPlr++GEr+TTFFnqfCKa0uDQU= 2022-09-12T20:12:16.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hQOPlr++GEr+TTFFnqfCKa0uDQU=', 'Content-Length': '0'} 2022-09-12T20:12:16.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fd92f0f57ddf1e4-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.049 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:16.050 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:16.050 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:12:16.051 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:12:16.051 DEBUG:boto:Method: GET 2022-09-12T20:12:16.052 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:12:16.052 DEBUG:boto:Data: 2022-09-12T20:12:16.053 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.054 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.054 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.055 DEBUG:boto:Params: {} 2022-09-12T20:12:16.055 DEBUG:boto:Token: None 2022-09-12T20:12:16.056 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/ 2022-09-12T20:12:16.056 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw= 2022-09-12T20:12:16.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw=', 'Content-Length': '0'} 2022-09-12T20:12:16.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a1427753a0492ce-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.060 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:16.061 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:16.062 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:16.062 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:12:16.063 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:12:16.063 DEBUG:boto:Method: GET 2022-09-12T20:12:16.064 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:12:16.064 DEBUG:boto:Data: 2022-09-12T20:12:16.065 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.065 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.066 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.066 DEBUG:boto:Params: {} 2022-09-12T20:12:16.067 DEBUG:boto:Token: None 2022-09-12T20:12:16.068 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/ 2022-09-12T20:12:16.068 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw= 2022-09-12T20:12:16.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw=', 'Content-Length': '0'} 2022-09-12T20:12:16.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082a7f19ab53037b8-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.074 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:16.075 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:16.075 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:16.076 DEBUG:boto:Method: HEAD 2022-09-12T20:12:16.076 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:12:16.077 DEBUG:boto:Data: 2022-09-12T20:12:16.077 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.078 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.078 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.079 DEBUG:boto:Params: {} 2022-09-12T20:12:16.079 DEBUG:boto:Token: None 2022-09-12T20:12:16.080 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj57 2022-09-12T20:12:16.080 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P/OFpmui0IFfqx085cK7dV4Wsb0= 2022-09-12T20:12:16.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P/OFpmui0IFfqx085cK7dV4Wsb0=', 'Content-Length': '0'} 2022-09-12T20:12:16.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000447f24cc5c9653d2-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.084 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:16.084 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:16.085 DEBUG:boto:Method: HEAD 2022-09-12T20:12:16.085 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:12:16.086 DEBUG:boto:Data: 2022-09-12T20:12:16.086 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.087 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.087 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.088 DEBUG:boto:Params: {} 2022-09-12T20:12:16.088 DEBUG:boto:Token: None 2022-09-12T20:12:16.089 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj57 2022-09-12T20:12:16.090 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P/OFpmui0IFfqx085cK7dV4Wsb0= 2022-09-12T20:12:16.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P/OFpmui0IFfqx085cK7dV4Wsb0=', 'Content-Length': '0'} 2022-09-12T20:12:16.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8e63d82c40ba24f-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:12:16.093 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:16.094 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:16.095 DEBUG:boto:Method: GET 2022-09-12T20:12:16.095 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:12:16.096 DEBUG:boto:Data: 2022-09-12T20:12:16.096 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.097 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.097 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.098 DEBUG:boto:Params: {} 2022-09-12T20:12:16.098 DEBUG:boto:Token: None 2022-09-12T20:12:16.099 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj57 2022-09-12T20:12:16.099 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kKAv3D1uGEb92zTZujTIlvBFbhQ= 2022-09-12T20:12:16.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kKAv3D1uGEb92zTZujTIlvBFbhQ=', 'Content-Length': '0'} 2022-09-12T20:12:16.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de2f718f00d674ba-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.103 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:16.103 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:16.104 DEBUG:boto:Method: GET 2022-09-12T20:12:16.104 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:12:16.105 DEBUG:boto:Data: 2022-09-12T20:12:16.106 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.106 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.107 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.107 DEBUG:boto:Params: {} 2022-09-12T20:12:16.108 DEBUG:boto:Token: None 2022-09-12T20:12:16.108 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj57 2022-09-12T20:12:16.109 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kKAv3D1uGEb92zTZujTIlvBFbhQ= 2022-09-12T20:12:16.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kKAv3D1uGEb92zTZujTIlvBFbhQ=', 'Content-Length': '0'} 2022-09-12T20:12:16.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea07fc68bc7c31e5-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.112 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:16.113 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:16.113 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:12:16.114 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:12:16.114 DEBUG:boto:Method: GET 2022-09-12T20:12:16.115 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:12:16.115 DEBUG:boto:Data: 2022-09-12T20:12:16.116 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.116 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.117 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.117 DEBUG:boto:Params: {} 2022-09-12T20:12:16.118 DEBUG:boto:Token: None 2022-09-12T20:12:16.119 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/ 2022-09-12T20:12:16.119 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw= 2022-09-12T20:12:16.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw=', 'Content-Length': '0'} 2022-09-12T20:12:16.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffcb292266ca63fc-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.166 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:16.167 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:16.167 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:16.168 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:12:16.168 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:12:16.169 DEBUG:boto:Method: GET 2022-09-12T20:12:16.169 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:12:16.170 DEBUG:boto:Data: 2022-09-12T20:12:16.170 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.171 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.171 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.172 DEBUG:boto:Params: {} 2022-09-12T20:12:16.172 DEBUG:boto:Token: None 2022-09-12T20:12:16.173 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/ 2022-09-12T20:12:16.173 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw= 2022-09-12T20:12:16.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw=', 'Content-Length': '0'} 2022-09-12T20:12:16.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe3db0ee0f63075d-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.179 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:16.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:12:16.180 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:16.181 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:16.181 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:16.182 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:16.182 DEBUG:boto:Method: GET 2022-09-12T20:12:16.183 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:16.183 DEBUG:boto:Data: 2022-09-12T20:12:16.184 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.184 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.185 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.185 DEBUG:boto:Params: {} 2022-09-12T20:12:16.186 DEBUG:boto:Token: None 2022-09-12T20:12:16.187 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:16.187 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:I2FFCHtEXI9rpmVM5yN2iIDq2WU= 2022-09-12T20:12:16.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:I2FFCHtEXI9rpmVM5yN2iIDq2WU=', 'Content-Length': '0'} 2022-09-12T20:12:16.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005989e11df00ecf1c-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.191 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:16.192 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:16.193 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:16.193 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:16.194 DEBUG:boto:Method: GET 2022-09-12T20:12:16.194 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:16.195 DEBUG:boto:Data: 2022-09-12T20:12:16.195 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.196 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.196 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.197 DEBUG:boto:Params: {} 2022-09-12T20:12:16.198 DEBUG:boto:Token: None 2022-09-12T20:12:16.198 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:16.199 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:I2FFCHtEXI9rpmVM5yN2iIDq2WU= 2022-09-12T20:12:16.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:I2FFCHtEXI9rpmVM5yN2iIDq2WU=', 'Content-Length': '0'} 2022-09-12T20:12:16.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000924253c7b80df161-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.242 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:16.243 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:16.243 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:16.244 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:16.244 DEBUG:boto:Method: HEAD 2022-09-12T20:12:16.245 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:16.245 DEBUG:boto:Data: 2022-09-12T20:12:16.246 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.246 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.247 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.247 DEBUG:boto:Params: {} 2022-09-12T20:12:16.248 DEBUG:boto:Token: None 2022-09-12T20:12:16.249 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:16.249 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RZsdZ+tIDmktkggQXOsZj9jdzr4= 2022-09-12T20:12:16.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RZsdZ+tIDmktkggQXOsZj9jdzr4=', 'Content-Length': '0'} 2022-09-12T20:12:16.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065ae8684334a1eb2-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.253 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:16.254 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:16.254 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:16.255 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:16.255 DEBUG:boto:Method: HEAD 2022-09-12T20:12:16.256 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:16.256 DEBUG:boto:Data: 2022-09-12T20:12:16.257 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.257 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.258 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.259 DEBUG:boto:Params: {} 2022-09-12T20:12:16.259 DEBUG:boto:Token: None 2022-09-12T20:12:16.260 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:16.260 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RZsdZ+tIDmktkggQXOsZj9jdzr4= 2022-09-12T20:12:16.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RZsdZ+tIDmktkggQXOsZj9jdzr4=', 'Content-Length': '0'} 2022-09-12T20:12:16.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eac33a204330f030-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:12:16.265 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:16.265 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:16.266 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:16.267 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:16.267 DEBUG:boto:Method: GET 2022-09-12T20:12:16.268 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:16.268 DEBUG:boto:Data: 2022-09-12T20:12:16.269 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.270 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.270 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.271 DEBUG:boto:Params: {} 2022-09-12T20:12:16.272 DEBUG:boto:Token: None 2022-09-12T20:12:16.273 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:16.274 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:I2FFCHtEXI9rpmVM5yN2iIDq2WU= 2022-09-12T20:12:16.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:I2FFCHtEXI9rpmVM5yN2iIDq2WU=', 'Content-Length': '0'} 2022-09-12T20:12:16.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d934e2845936b716-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.278 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:16.278 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:16.279 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:16.279 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:16.280 DEBUG:boto:Method: GET 2022-09-12T20:12:16.281 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:16.281 DEBUG:boto:Data: 2022-09-12T20:12:16.282 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.282 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.283 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.284 DEBUG:boto:Params: {} 2022-09-12T20:12:16.284 DEBUG:boto:Token: None 2022-09-12T20:12:16.285 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:16.286 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:I2FFCHtEXI9rpmVM5yN2iIDq2WU= 2022-09-12T20:12:16.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:I2FFCHtEXI9rpmVM5yN2iIDq2WU=', 'Content-Length': '0'} 2022-09-12T20:12:16.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073b249f32d627828-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.290 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:16.291 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:16.291 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:12:16.292 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:12:16.293 DEBUG:boto:Method: GET 2022-09-12T20:12:16.293 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:12:16.294 DEBUG:boto:Data: 2022-09-12T20:12:16.295 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.296 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.296 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.297 DEBUG:boto:Params: {} 2022-09-12T20:12:16.298 DEBUG:boto:Token: None 2022-09-12T20:12:16.299 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/ 2022-09-12T20:12:16.299 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw= 2022-09-12T20:12:16.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw=', 'Content-Length': '0'} 2022-09-12T20:12:16.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000920811905754669a-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.304 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:16.305 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:16.306 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:16.306 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:12:16.307 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:12:16.307 DEBUG:boto:Method: GET 2022-09-12T20:12:16.308 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:12:16.309 DEBUG:boto:Data: 2022-09-12T20:12:16.310 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.310 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.311 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.313 DEBUG:boto:Params: {} 2022-09-12T20:12:16.314 DEBUG:boto:Token: None 2022-09-12T20:12:16.316 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/ 2022-09-12T20:12:16.318 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw= 2022-09-12T20:12:16.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw=', 'Content-Length': '0'} 2022-09-12T20:12:16.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6e56de1e58fc5d5-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.326 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:16.328 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:16.328 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:16.329 DEBUG:boto:Method: HEAD 2022-09-12T20:12:16.330 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:12:16.330 DEBUG:boto:Data: 2022-09-12T20:12:16.331 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.331 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.332 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.333 DEBUG:boto:Params: {} 2022-09-12T20:12:16.334 DEBUG:boto:Token: None 2022-09-12T20:12:16.334 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj58 2022-09-12T20:12:16.335 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vWtwqN274qmYlS5T1wG2Ff1a0r4= 2022-09-12T20:12:16.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vWtwqN274qmYlS5T1wG2Ff1a0r4=', 'Content-Length': '0'} 2022-09-12T20:12:16.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f220aaf8b4f25fd3-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.340 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:16.341 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:16.341 DEBUG:boto:Method: HEAD 2022-09-12T20:12:16.342 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:12:16.343 DEBUG:boto:Data: 2022-09-12T20:12:16.343 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.344 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.345 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.345 DEBUG:boto:Params: {} 2022-09-12T20:12:16.346 DEBUG:boto:Token: None 2022-09-12T20:12:16.347 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj58 2022-09-12T20:12:16.347 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vWtwqN274qmYlS5T1wG2Ff1a0r4= 2022-09-12T20:12:16.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vWtwqN274qmYlS5T1wG2Ff1a0r4=', 'Content-Length': '0'} 2022-09-12T20:12:16.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6b00b9384d32f6d-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:12:16.353 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:16.354 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:16.355 DEBUG:boto:Method: GET 2022-09-12T20:12:16.355 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:12:16.356 DEBUG:boto:Data: 2022-09-12T20:12:16.357 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.357 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.358 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.359 DEBUG:boto:Params: {} 2022-09-12T20:12:16.359 DEBUG:boto:Token: None 2022-09-12T20:12:16.360 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj58 2022-09-12T20:12:16.361 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tdKJjQtEiefIuF7Jj3hWiY4AC8E= 2022-09-12T20:12:16.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tdKJjQtEiefIuF7Jj3hWiY4AC8E=', 'Content-Length': '0'} 2022-09-12T20:12:16.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000574fc3e48e4dedb4-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.406 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:16.407 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:16.408 DEBUG:boto:Method: GET 2022-09-12T20:12:16.409 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:12:16.410 DEBUG:boto:Data: 2022-09-12T20:12:16.410 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.411 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.412 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.412 DEBUG:boto:Params: {} 2022-09-12T20:12:16.413 DEBUG:boto:Token: None 2022-09-12T20:12:16.414 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj58 2022-09-12T20:12:16.415 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tdKJjQtEiefIuF7Jj3hWiY4AC8E= 2022-09-12T20:12:16.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tdKJjQtEiefIuF7Jj3hWiY4AC8E=', 'Content-Length': '0'} 2022-09-12T20:12:16.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b9c73438a37472b-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.419 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:16.420 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:16.420 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:12:16.421 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:12:16.422 DEBUG:boto:Method: GET 2022-09-12T20:12:16.423 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:12:16.423 DEBUG:boto:Data: 2022-09-12T20:12:16.424 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.424 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.425 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.426 DEBUG:boto:Params: {} 2022-09-12T20:12:16.426 DEBUG:boto:Token: None 2022-09-12T20:12:16.427 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/ 2022-09-12T20:12:16.428 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw= 2022-09-12T20:12:16.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw=', 'Content-Length': '0'} 2022-09-12T20:12:16.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000130ff1ceab5969b-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.432 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:16.433 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:16.434 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:16.435 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:12:16.436 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:12:16.436 DEBUG:boto:Method: GET 2022-09-12T20:12:16.437 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:12:16.437 DEBUG:boto:Data: 2022-09-12T20:12:16.438 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.439 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.439 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.440 DEBUG:boto:Params: {} 2022-09-12T20:12:16.441 DEBUG:boto:Token: None 2022-09-12T20:12:16.441 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/ 2022-09-12T20:12:16.442 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw= 2022-09-12T20:12:16.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw=', 'Content-Length': '0'} 2022-09-12T20:12:16.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b40d920a7daa5c69-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.490 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:16.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:12:16.492 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:16.493 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:16.493 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:16.494 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:16.494 DEBUG:boto:Method: GET 2022-09-12T20:12:16.495 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:16.495 DEBUG:boto:Data: 2022-09-12T20:12:16.496 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.497 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.497 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.498 DEBUG:boto:Params: {} 2022-09-12T20:12:16.499 DEBUG:boto:Token: None 2022-09-12T20:12:16.499 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:16.500 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EJsVqRYN9yCeRvGqDoBSip0M27w= 2022-09-12T20:12:16.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EJsVqRYN9yCeRvGqDoBSip0M27w=', 'Content-Length': '0'} 2022-09-12T20:12:16.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063a6f3553cc88407-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.504 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:16.505 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:16.516 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:16.517 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:16.517 DEBUG:boto:Method: GET 2022-09-12T20:12:16.518 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:16.519 DEBUG:boto:Data: 2022-09-12T20:12:16.519 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.520 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.521 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.521 DEBUG:boto:Params: {} 2022-09-12T20:12:16.522 DEBUG:boto:Token: None 2022-09-12T20:12:16.522 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:16.523 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EJsVqRYN9yCeRvGqDoBSip0M27w= 2022-09-12T20:12:16.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EJsVqRYN9yCeRvGqDoBSip0M27w=', 'Content-Length': '0'} 2022-09-12T20:12:16.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009a629b651607c27-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.527 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:16.527 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:16.528 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:16.528 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:16.529 DEBUG:boto:Method: HEAD 2022-09-12T20:12:16.530 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:16.530 DEBUG:boto:Data: 2022-09-12T20:12:16.531 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.531 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.532 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.532 DEBUG:boto:Params: {} 2022-09-12T20:12:16.533 DEBUG:boto:Token: None 2022-09-12T20:12:16.533 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:16.534 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qmPfvx5LP12yz1evH9ZTiDbXJhQ= 2022-09-12T20:12:16.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qmPfvx5LP12yz1evH9ZTiDbXJhQ=', 'Content-Length': '0'} 2022-09-12T20:12:16.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d32fc2527c41cc75-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.538 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:16.538 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:16.539 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:16.539 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:16.540 DEBUG:boto:Method: HEAD 2022-09-12T20:12:16.540 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:16.541 DEBUG:boto:Data: 2022-09-12T20:12:16.541 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.542 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.542 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.543 DEBUG:boto:Params: {} 2022-09-12T20:12:16.543 DEBUG:boto:Token: None 2022-09-12T20:12:16.544 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:16.545 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qmPfvx5LP12yz1evH9ZTiDbXJhQ= 2022-09-12T20:12:16.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qmPfvx5LP12yz1evH9ZTiDbXJhQ=', 'Content-Length': '0'} 2022-09-12T20:12:16.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f874f5342b4c8faa-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:12:16.549 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:16.549 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:16.550 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:16.550 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:16.551 DEBUG:boto:Method: GET 2022-09-12T20:12:16.551 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:16.552 DEBUG:boto:Data: 2022-09-12T20:12:16.552 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.553 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.554 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.554 DEBUG:boto:Params: {} 2022-09-12T20:12:16.555 DEBUG:boto:Token: None 2022-09-12T20:12:16.555 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:16.556 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EJsVqRYN9yCeRvGqDoBSip0M27w= 2022-09-12T20:12:16.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EJsVqRYN9yCeRvGqDoBSip0M27w=', 'Content-Length': '0'} 2022-09-12T20:12:16.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000acf7ef8cdf42a11-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.560 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:16.560 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:16.561 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:16.561 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:16.562 DEBUG:boto:Method: GET 2022-09-12T20:12:16.562 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:16.563 DEBUG:boto:Data: 2022-09-12T20:12:16.563 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.564 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.564 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.565 DEBUG:boto:Params: {} 2022-09-12T20:12:16.565 DEBUG:boto:Token: None 2022-09-12T20:12:16.566 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:16.567 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EJsVqRYN9yCeRvGqDoBSip0M27w= 2022-09-12T20:12:16.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EJsVqRYN9yCeRvGqDoBSip0M27w=', 'Content-Length': '0'} 2022-09-12T20:12:16.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fef932b2b8fe186a-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.570 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:16.571 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:16.571 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:12:16.572 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:12:16.572 DEBUG:boto:Method: GET 2022-09-12T20:12:16.573 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:12:16.573 DEBUG:boto:Data: 2022-09-12T20:12:16.574 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.575 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.575 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.576 DEBUG:boto:Params: {} 2022-09-12T20:12:16.576 DEBUG:boto:Token: None 2022-09-12T20:12:16.577 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/ 2022-09-12T20:12:16.577 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw= 2022-09-12T20:12:16.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw=', 'Content-Length': '0'} 2022-09-12T20:12:16.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a04a16e73f258560-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.622 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:16.623 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:16.623 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:16.624 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:12:16.624 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:12:16.625 DEBUG:boto:Method: GET 2022-09-12T20:12:16.626 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:12:16.626 DEBUG:boto:Data: 2022-09-12T20:12:16.627 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.627 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.628 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.628 DEBUG:boto:Params: {} 2022-09-12T20:12:16.629 DEBUG:boto:Token: None 2022-09-12T20:12:16.629 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/ 2022-09-12T20:12:16.630 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw= 2022-09-12T20:12:16.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw=', 'Content-Length': '0'} 2022-09-12T20:12:16.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e87990cee841896e-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.636 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:16.636 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:16.637 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:16.637 DEBUG:boto:Method: HEAD 2022-09-12T20:12:16.638 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:12:16.638 DEBUG:boto:Data: 2022-09-12T20:12:16.639 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.640 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.640 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.640 DEBUG:boto:Params: {} 2022-09-12T20:12:16.641 DEBUG:boto:Token: None 2022-09-12T20:12:16.642 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj68 2022-09-12T20:12:16.642 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5o6oQGB/z0A6RCoz1mC8dlszkHE= 2022-09-12T20:12:16.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5o6oQGB/z0A6RCoz1mC8dlszkHE=', 'Content-Length': '0'} 2022-09-12T20:12:16.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f98135f33e03ca2-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.646 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:16.646 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:16.647 DEBUG:boto:Method: HEAD 2022-09-12T20:12:16.647 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:12:16.648 DEBUG:boto:Data: 2022-09-12T20:12:16.648 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.649 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.649 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.650 DEBUG:boto:Params: {} 2022-09-12T20:12:16.650 DEBUG:boto:Token: None 2022-09-12T20:12:16.651 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj68 2022-09-12T20:12:16.651 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5o6oQGB/z0A6RCoz1mC8dlszkHE= 2022-09-12T20:12:16.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5o6oQGB/z0A6RCoz1mC8dlszkHE=', 'Content-Length': '0'} 2022-09-12T20:12:16.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000740fda35a9403341-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:12:16.655 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:16.656 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:16.656 DEBUG:boto:Method: GET 2022-09-12T20:12:16.657 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:12:16.657 DEBUG:boto:Data: 2022-09-12T20:12:16.658 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.659 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.659 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.660 DEBUG:boto:Params: {} 2022-09-12T20:12:16.660 DEBUG:boto:Token: None 2022-09-12T20:12:16.661 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj68 2022-09-12T20:12:16.662 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KX4jlNBcrTycr6u2IYdnjIzddaQ= 2022-09-12T20:12:16.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KX4jlNBcrTycr6u2IYdnjIzddaQ=', 'Content-Length': '0'} 2022-09-12T20:12:16.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dda214ce81832304-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.665 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:16.666 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:16.667 DEBUG:boto:Method: GET 2022-09-12T20:12:16.667 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:12:16.668 DEBUG:boto:Data: 2022-09-12T20:12:16.668 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.669 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.669 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.670 DEBUG:boto:Params: {} 2022-09-12T20:12:16.670 DEBUG:boto:Token: None 2022-09-12T20:12:16.671 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj68 2022-09-12T20:12:16.672 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KX4jlNBcrTycr6u2IYdnjIzddaQ= 2022-09-12T20:12:16.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KX4jlNBcrTycr6u2IYdnjIzddaQ=', 'Content-Length': '0'} 2022-09-12T20:12:16.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035053b8800fa86fe-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.718 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:16.719 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:16.720 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:12:16.720 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:12:16.721 DEBUG:boto:Method: GET 2022-09-12T20:12:16.721 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:12:16.722 DEBUG:boto:Data: 2022-09-12T20:12:16.723 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.723 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.724 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.724 DEBUG:boto:Params: {} 2022-09-12T20:12:16.725 DEBUG:boto:Token: None 2022-09-12T20:12:16.726 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/ 2022-09-12T20:12:16.726 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw= 2022-09-12T20:12:16.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw=', 'Content-Length': '0'} 2022-09-12T20:12:16.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b7d1bef7775ff93-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.731 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:16.732 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:16.732 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:16.733 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:12:16.733 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:12:16.734 DEBUG:boto:Method: GET 2022-09-12T20:12:16.734 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:12:16.735 DEBUG:boto:Data: 2022-09-12T20:12:16.736 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.736 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.737 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.737 DEBUG:boto:Params: {} 2022-09-12T20:12:16.738 DEBUG:boto:Token: None 2022-09-12T20:12:16.739 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/ 2022-09-12T20:12:16.739 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw= 2022-09-12T20:12:16.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw=', 'Content-Length': '0'} 2022-09-12T20:12:16.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000864c2914cb0bb2b8-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.745 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:16.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:12:16.746 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:16.747 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:16.747 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:16.748 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:16.749 DEBUG:boto:Method: GET 2022-09-12T20:12:16.749 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:16.750 DEBUG:boto:Data: 2022-09-12T20:12:16.751 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.751 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.752 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.753 DEBUG:boto:Params: {} 2022-09-12T20:12:16.753 DEBUG:boto:Token: None 2022-09-12T20:12:16.754 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:16.755 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F9XikTPMrHCk9xxoBKuUmrpaCTE= 2022-09-12T20:12:16.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F9XikTPMrHCk9xxoBKuUmrpaCTE=', 'Content-Length': '0'} 2022-09-12T20:12:16.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2d6d814e2b09dd3-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.759 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:16.759 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:16.760 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:16.761 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:16.761 DEBUG:boto:Method: GET 2022-09-12T20:12:16.762 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:16.763 DEBUG:boto:Data: 2022-09-12T20:12:16.763 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.764 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.764 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.765 DEBUG:boto:Params: {} 2022-09-12T20:12:16.766 DEBUG:boto:Token: None 2022-09-12T20:12:16.766 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:16.767 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F9XikTPMrHCk9xxoBKuUmrpaCTE= 2022-09-12T20:12:16.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F9XikTPMrHCk9xxoBKuUmrpaCTE=', 'Content-Length': '0'} 2022-09-12T20:12:16.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004aeb16d8f4a6f92b-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.771 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:16.772 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:16.773 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:16.773 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:16.774 DEBUG:boto:Method: HEAD 2022-09-12T20:12:16.774 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:16.775 DEBUG:boto:Data: 2022-09-12T20:12:16.775 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.776 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.776 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.777 DEBUG:boto:Params: {} 2022-09-12T20:12:16.778 DEBUG:boto:Token: None 2022-09-12T20:12:16.778 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:16.779 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8QPlfosheWZqwao7eUI5XRJsdCA= 2022-09-12T20:12:16.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8QPlfosheWZqwao7eUI5XRJsdCA=', 'Content-Length': '0'} 2022-09-12T20:12:16.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8b42eb7d80117f5-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.782 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:16.783 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:16.783 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:16.784 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:16.784 DEBUG:boto:Method: HEAD 2022-09-12T20:12:16.785 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:16.786 DEBUG:boto:Data: 2022-09-12T20:12:16.786 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.787 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.787 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.788 DEBUG:boto:Params: {} 2022-09-12T20:12:16.788 DEBUG:boto:Token: None 2022-09-12T20:12:16.789 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:16.790 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8QPlfosheWZqwao7eUI5XRJsdCA= 2022-09-12T20:12:16.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8QPlfosheWZqwao7eUI5XRJsdCA=', 'Content-Length': '0'} 2022-09-12T20:12:16.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b8c23f28ce8cfaf-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:12:16.794 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:16.801 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:16.801 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:16.802 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:16.802 DEBUG:boto:Method: GET 2022-09-12T20:12:16.803 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:16.803 DEBUG:boto:Data: 2022-09-12T20:12:16.804 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.804 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.805 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.805 DEBUG:boto:Params: {} 2022-09-12T20:12:16.806 DEBUG:boto:Token: None 2022-09-12T20:12:16.807 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:16.807 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F9XikTPMrHCk9xxoBKuUmrpaCTE= 2022-09-12T20:12:16.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F9XikTPMrHCk9xxoBKuUmrpaCTE=', 'Content-Length': '0'} 2022-09-12T20:12:16.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bc2d9d1dbcb237b-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.854 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:16.855 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:16.855 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:16.856 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:16.856 DEBUG:boto:Method: GET 2022-09-12T20:12:16.857 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:16.857 DEBUG:boto:Data: 2022-09-12T20:12:16.858 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.859 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.859 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.860 DEBUG:boto:Params: {} 2022-09-12T20:12:16.861 DEBUG:boto:Token: None 2022-09-12T20:12:16.861 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:16.862 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F9XikTPMrHCk9xxoBKuUmrpaCTE= 2022-09-12T20:12:16.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F9XikTPMrHCk9xxoBKuUmrpaCTE=', 'Content-Length': '0'} 2022-09-12T20:12:16.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec7a016fca043d6d-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.866 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:16.866 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:16.867 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:12:16.867 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:12:16.868 DEBUG:boto:Method: GET 2022-09-12T20:12:16.869 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:12:16.869 DEBUG:boto:Data: 2022-09-12T20:12:16.870 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.870 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.871 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.871 DEBUG:boto:Params: {} 2022-09-12T20:12:16.872 DEBUG:boto:Token: None 2022-09-12T20:12:16.873 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/ 2022-09-12T20:12:16.873 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw= 2022-09-12T20:12:16.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw=', 'Content-Length': '0'} 2022-09-12T20:12:16.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d031b19c1c439122-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.878 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:16.879 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:16.879 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:16.880 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:12:16.880 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:12:16.881 DEBUG:boto:Method: GET 2022-09-12T20:12:16.881 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:12:16.882 DEBUG:boto:Data: 2022-09-12T20:12:16.883 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.883 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.884 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.884 DEBUG:boto:Params: {} 2022-09-12T20:12:16.885 DEBUG:boto:Token: None 2022-09-12T20:12:16.886 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/ 2022-09-12T20:12:16.886 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw= 2022-09-12T20:12:16.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw=', 'Content-Length': '0'} 2022-09-12T20:12:16.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9ec58e4efd25137-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.892 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:16.892 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:16.893 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:16.893 DEBUG:boto:Method: HEAD 2022-09-12T20:12:16.894 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:12:16.895 DEBUG:boto:Data: 2022-09-12T20:12:16.895 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.896 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.896 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.897 DEBUG:boto:Params: {} 2022-09-12T20:12:16.897 DEBUG:boto:Token: None 2022-09-12T20:12:16.898 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj69 2022-09-12T20:12:16.899 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/jeWPrhiHixdObeEoKmgvWKb8Vk= 2022-09-12T20:12:16.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/jeWPrhiHixdObeEoKmgvWKb8Vk=', 'Content-Length': '0'} 2022-09-12T20:12:16.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb8c3bd47929d67d-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.902 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:16.903 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:16.903 DEBUG:boto:Method: HEAD 2022-09-12T20:12:16.904 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:12:16.904 DEBUG:boto:Data: 2022-09-12T20:12:16.905 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.905 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.906 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.906 DEBUG:boto:Params: {} 2022-09-12T20:12:16.907 DEBUG:boto:Token: None 2022-09-12T20:12:16.908 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj69 2022-09-12T20:12:16.908 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/jeWPrhiHixdObeEoKmgvWKb8Vk= 2022-09-12T20:12:16.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/jeWPrhiHixdObeEoKmgvWKb8Vk=', 'Content-Length': '0'} 2022-09-12T20:12:16.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0f390e1298db652-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:12:16.912 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:16.913 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:16.913 DEBUG:boto:Method: GET 2022-09-12T20:12:16.914 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:12:16.914 DEBUG:boto:Data: 2022-09-12T20:12:16.915 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.916 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.916 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.917 DEBUG:boto:Params: {} 2022-09-12T20:12:16.917 DEBUG:boto:Token: None 2022-09-12T20:12:16.918 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj69 2022-09-12T20:12:16.919 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+7FFPeIAJLl2WMgW2YPS90kcP4= 2022-09-12T20:12:16.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+7FFPeIAJLl2WMgW2YPS90kcP4=', 'Content-Length': '0'} 2022-09-12T20:12:16.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cdfd5ce631ddf0d-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.923 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:16.923 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:16.924 DEBUG:boto:Method: GET 2022-09-12T20:12:16.924 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:12:16.925 DEBUG:boto:Data: 2022-09-12T20:12:16.925 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.926 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.926 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.927 DEBUG:boto:Params: {} 2022-09-12T20:12:16.927 DEBUG:boto:Token: None 2022-09-12T20:12:16.928 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj69 2022-09-12T20:12:16.929 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+7FFPeIAJLl2WMgW2YPS90kcP4= 2022-09-12T20:12:16.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+7FFPeIAJLl2WMgW2YPS90kcP4=', 'Content-Length': '0'} 2022-09-12T20:12:16.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbeb6e54cb863184-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.933 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:16.933 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:16.934 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:12:16.935 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:12:16.935 DEBUG:boto:Method: GET 2022-09-12T20:12:16.936 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:12:16.936 DEBUG:boto:Data: 2022-09-12T20:12:16.937 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.937 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.938 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.939 DEBUG:boto:Params: {} 2022-09-12T20:12:16.939 DEBUG:boto:Token: None 2022-09-12T20:12:16.940 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/ 2022-09-12T20:12:16.940 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw= 2022-09-12T20:12:16.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw=', 'Content-Length': '0'} 2022-09-12T20:12:16.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3f71ec25a2fe60b-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.944 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:16.945 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:16.946 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:16.946 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:12:16.947 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:12:16.948 DEBUG:boto:Method: GET 2022-09-12T20:12:16.948 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:12:16.949 DEBUG:boto:Data: 2022-09-12T20:12:16.949 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.950 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.951 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.951 DEBUG:boto:Params: {} 2022-09-12T20:12:16.952 DEBUG:boto:Token: None 2022-09-12T20:12:16.952 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/ 2022-09-12T20:12:16.953 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw= 2022-09-12T20:12:16.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wUgTvxWf7OqTBFhngRClbBTQTUw=', 'Content-Length': '0'} 2022-09-12T20:12:16.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005121c171b9c0591f-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.958 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:16.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:12:16.960 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:16.960 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:16.961 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:16.961 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:16.962 DEBUG:boto:Method: GET 2022-09-12T20:12:16.962 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:16.963 DEBUG:boto:Data: 2022-09-12T20:12:16.964 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.964 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.965 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.965 DEBUG:boto:Params: {} 2022-09-12T20:12:16.966 DEBUG:boto:Token: None 2022-09-12T20:12:16.966 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:16.967 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CI292s6J5sVaSUx0XwH5zYEurOA= 2022-09-12T20:12:16.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CI292s6J5sVaSUx0XwH5zYEurOA=', 'Content-Length': '0'} 2022-09-12T20:12:16.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014ba3ae6c9f992af-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.971 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:16.971 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:16.972 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:16.973 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:16.973 DEBUG:boto:Method: GET 2022-09-12T20:12:16.974 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:16.974 DEBUG:boto:Data: 2022-09-12T20:12:16.975 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.975 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.976 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.977 DEBUG:boto:Params: {} 2022-09-12T20:12:16.977 DEBUG:boto:Token: None 2022-09-12T20:12:16.978 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:16.978 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CI292s6J5sVaSUx0XwH5zYEurOA= 2022-09-12T20:12:16.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CI292s6J5sVaSUx0XwH5zYEurOA=', 'Content-Length': '0'} 2022-09-12T20:12:16.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000687f0c57b80c0b38-00631f92a0-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.982 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:16.983 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:16.983 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:16.984 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:16.984 DEBUG:boto:Method: HEAD 2022-09-12T20:12:16.985 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:16.985 DEBUG:boto:Data: 2022-09-12T20:12:16.986 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.987 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:16.987 DEBUG:boto:Port: 8001 2022-09-12T20:12:16.988 DEBUG:boto:Params: {} 2022-09-12T20:12:16.988 DEBUG:boto:Token: None 2022-09-12T20:12:16.989 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:16 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:16.989 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6/WsYz6ltnMzewHxbNjMfpsGpI0= 2022-09-12T20:12:16.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6/WsYz6ltnMzewHxbNjMfpsGpI0=', 'Content-Length': '0'} 2022-09-12T20:12:16.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a5d72d2b0c5e928-00631f92a0-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:16.994 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:16.995 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:16.995 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:16.996 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:16.996 DEBUG:boto:Method: HEAD 2022-09-12T20:12:16.997 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:16.997 DEBUG:boto:Data: 2022-09-12T20:12:16.998 DEBUG:boto:Headers: {} 2022-09-12T20:12:16.998 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:16.999 DEBUG:boto:Port: 8000 2022-09-12T20:12:16.999 DEBUG:boto:Params: {} 2022-09-12T20:12:17.000 DEBUG:boto:Token: None 2022-09-12T20:12:17.001 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:17.001 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VWzj2LQ9mfncnSoo/XdaajJASZw= 2022-09-12T20:12:17.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VWzj2LQ9mfncnSoo/XdaajJASZw=', 'Content-Length': '0'} 2022-09-12T20:12:17.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f485b7711b8f8c8-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:12:17.006 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:17.006 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:17.007 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:17.007 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:17.008 DEBUG:boto:Method: GET 2022-09-12T20:12:17.008 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:17.009 DEBUG:boto:Data: 2022-09-12T20:12:17.009 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.010 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.011 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.011 DEBUG:boto:Params: {} 2022-09-12T20:12:17.012 DEBUG:boto:Token: None 2022-09-12T20:12:17.012 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:17.013 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f2RvEUMuPKoL9uEWEG2z6AbcEi4= 2022-09-12T20:12:17.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f2RvEUMuPKoL9uEWEG2z6AbcEi4=', 'Content-Length': '0'} 2022-09-12T20:12:17.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000039099a9066e2b28-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.017 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:17.018 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:17.018 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:17.019 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:17.019 DEBUG:boto:Method: GET 2022-09-12T20:12:17.020 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:17.021 DEBUG:boto:Data: 2022-09-12T20:12:17.021 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.022 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.022 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.023 DEBUG:boto:Params: {} 2022-09-12T20:12:17.023 DEBUG:boto:Token: None 2022-09-12T20:12:17.024 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:17.025 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f2RvEUMuPKoL9uEWEG2z6AbcEi4= 2022-09-12T20:12:17.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f2RvEUMuPKoL9uEWEG2z6AbcEi4=', 'Content-Length': '0'} 2022-09-12T20:12:17.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000078ee1c3ccd49669-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.028 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:17.029 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:17.030 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:12:17.030 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:12:17.031 DEBUG:boto:Method: GET 2022-09-12T20:12:17.031 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:12:17.032 DEBUG:boto:Data: 2022-09-12T20:12:17.033 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.033 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.034 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.034 DEBUG:boto:Params: {} 2022-09-12T20:12:17.035 DEBUG:boto:Token: None 2022-09-12T20:12:17.036 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/ 2022-09-12T20:12:17.036 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34= 2022-09-12T20:12:17.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34=', 'Content-Length': '0'} 2022-09-12T20:12:17.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004779399d31874e4a-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.041 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:17.042 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:17.042 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:17.043 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:12:17.043 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:12:17.044 DEBUG:boto:Method: GET 2022-09-12T20:12:17.044 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:12:17.045 DEBUG:boto:Data: 2022-09-12T20:12:17.046 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.046 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.047 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.047 DEBUG:boto:Params: {} 2022-09-12T20:12:17.048 DEBUG:boto:Token: None 2022-09-12T20:12:17.048 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/ 2022-09-12T20:12:17.049 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34= 2022-09-12T20:12:17.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34=', 'Content-Length': '0'} 2022-09-12T20:12:17.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000735215f656f93b49-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.055 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:17.055 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:17.056 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:17.057 DEBUG:boto:Method: HEAD 2022-09-12T20:12:17.057 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:12:17.058 DEBUG:boto:Data: 2022-09-12T20:12:17.058 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.059 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.059 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.060 DEBUG:boto:Params: {} 2022-09-12T20:12:17.060 DEBUG:boto:Token: None 2022-09-12T20:12:17.061 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj76 2022-09-12T20:12:17.062 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LQATWSRTxR0ghaHFWj2pKm4WJzs= 2022-09-12T20:12:17.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LQATWSRTxR0ghaHFWj2pKm4WJzs=', 'Content-Length': '0'} 2022-09-12T20:12:17.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042757c0fd47315be-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.065 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:17.066 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:17.066 DEBUG:boto:Method: HEAD 2022-09-12T20:12:17.067 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:12:17.067 DEBUG:boto:Data: 2022-09-12T20:12:17.068 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.068 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.069 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.070 DEBUG:boto:Params: {} 2022-09-12T20:12:17.070 DEBUG:boto:Token: None 2022-09-12T20:12:17.071 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj76 2022-09-12T20:12:17.071 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LQATWSRTxR0ghaHFWj2pKm4WJzs= 2022-09-12T20:12:17.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LQATWSRTxR0ghaHFWj2pKm4WJzs=', 'Content-Length': '0'} 2022-09-12T20:12:17.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd53bf3715117bef-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:12:17.075 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:17.076 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:17.077 DEBUG:boto:Method: GET 2022-09-12T20:12:17.077 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:12:17.078 DEBUG:boto:Data: 2022-09-12T20:12:17.078 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.079 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.079 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.080 DEBUG:boto:Params: {} 2022-09-12T20:12:17.081 DEBUG:boto:Token: None 2022-09-12T20:12:17.081 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj76 2022-09-12T20:12:17.082 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ICjC7i4gReUmWCikWPHLzSniX7o= 2022-09-12T20:12:17.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ICjC7i4gReUmWCikWPHLzSniX7o=', 'Content-Length': '0'} 2022-09-12T20:12:17.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daeab238610ddedf-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.085 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:17.086 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:17.086 DEBUG:boto:Method: GET 2022-09-12T20:12:17.087 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:12:17.088 DEBUG:boto:Data: 2022-09-12T20:12:17.088 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.089 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.089 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.090 DEBUG:boto:Params: {} 2022-09-12T20:12:17.090 DEBUG:boto:Token: None 2022-09-12T20:12:17.091 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj76 2022-09-12T20:12:17.092 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ICjC7i4gReUmWCikWPHLzSniX7o= 2022-09-12T20:12:17.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ICjC7i4gReUmWCikWPHLzSniX7o=', 'Content-Length': '0'} 2022-09-12T20:12:17.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b041fe08b82ce68-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.138 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:17.139 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:17.139 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:12:17.140 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:12:17.141 DEBUG:boto:Method: GET 2022-09-12T20:12:17.141 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:12:17.142 DEBUG:boto:Data: 2022-09-12T20:12:17.142 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.143 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.143 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.144 DEBUG:boto:Params: {} 2022-09-12T20:12:17.144 DEBUG:boto:Token: None 2022-09-12T20:12:17.145 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/ 2022-09-12T20:12:17.146 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34= 2022-09-12T20:12:17.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34=', 'Content-Length': '0'} 2022-09-12T20:12:17.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c40a8a560fe2107-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.150 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:17.151 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:17.151 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:17.152 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:12:17.153 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:12:17.154 DEBUG:boto:Method: GET 2022-09-12T20:12:17.154 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:12:17.155 DEBUG:boto:Data: 2022-09-12T20:12:17.155 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.156 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.156 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.157 DEBUG:boto:Params: {} 2022-09-12T20:12:17.157 DEBUG:boto:Token: None 2022-09-12T20:12:17.158 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/ 2022-09-12T20:12:17.159 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34= 2022-09-12T20:12:17.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34=', 'Content-Length': '0'} 2022-09-12T20:12:17.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe98d3374c464bd4-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.164 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:17.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:12:17.165 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:17.166 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:17.167 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:17.167 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:17.168 DEBUG:boto:Method: GET 2022-09-12T20:12:17.168 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:17.169 DEBUG:boto:Data: 2022-09-12T20:12:17.169 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.170 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.171 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.171 DEBUG:boto:Params: {} 2022-09-12T20:12:17.172 DEBUG:boto:Token: None 2022-09-12T20:12:17.172 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:17.173 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AWQmz8fttuSWBCmBsXS9fkIoIVk= 2022-09-12T20:12:17.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AWQmz8fttuSWBCmBsXS9fkIoIVk=', 'Content-Length': '0'} 2022-09-12T20:12:17.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae34d825e5e4acdd-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.177 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:17.178 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:17.178 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:17.179 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:17.179 DEBUG:boto:Method: GET 2022-09-12T20:12:17.180 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:17.180 DEBUG:boto:Data: 2022-09-12T20:12:17.181 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.181 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.182 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.182 DEBUG:boto:Params: {} 2022-09-12T20:12:17.183 DEBUG:boto:Token: None 2022-09-12T20:12:17.184 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:17.184 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AWQmz8fttuSWBCmBsXS9fkIoIVk= 2022-09-12T20:12:17.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AWQmz8fttuSWBCmBsXS9fkIoIVk=', 'Content-Length': '0'} 2022-09-12T20:12:17.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df163901db9bf5e5-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.188 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:17.189 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:17.189 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:17.190 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:17.190 DEBUG:boto:Method: HEAD 2022-09-12T20:12:17.191 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:17.191 DEBUG:boto:Data: 2022-09-12T20:12:17.192 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.192 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.193 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.193 DEBUG:boto:Params: {} 2022-09-12T20:12:17.194 DEBUG:boto:Token: None 2022-09-12T20:12:17.195 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:17.195 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4Y2goFDcKrA/GnmPBDkwtVO3p7M= 2022-09-12T20:12:17.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4Y2goFDcKrA/GnmPBDkwtVO3p7M=', 'Content-Length': '0'} 2022-09-12T20:12:17.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dab0adf6a01b1f35-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.199 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:17.199 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:17.200 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:17.200 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:17.201 DEBUG:boto:Method: HEAD 2022-09-12T20:12:17.201 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:17.202 DEBUG:boto:Data: 2022-09-12T20:12:17.202 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.203 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.203 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.204 DEBUG:boto:Params: {} 2022-09-12T20:12:17.205 DEBUG:boto:Token: None 2022-09-12T20:12:17.205 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:17.206 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4Y2goFDcKrA/GnmPBDkwtVO3p7M= 2022-09-12T20:12:17.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4Y2goFDcKrA/GnmPBDkwtVO3p7M=', 'Content-Length': '0'} 2022-09-12T20:12:17.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c857fd8cf031cfbf-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:12:17.210 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:17.210 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:17.211 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:17.211 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:17.212 DEBUG:boto:Method: GET 2022-09-12T20:12:17.212 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:17.213 DEBUG:boto:Data: 2022-09-12T20:12:17.213 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.214 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.215 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.215 DEBUG:boto:Params: {} 2022-09-12T20:12:17.216 DEBUG:boto:Token: None 2022-09-12T20:12:17.216 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:17.217 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AWQmz8fttuSWBCmBsXS9fkIoIVk= 2022-09-12T20:12:17.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AWQmz8fttuSWBCmBsXS9fkIoIVk=', 'Content-Length': '0'} 2022-09-12T20:12:17.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bca602277b3cc67-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.262 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:17.263 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:17.263 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:17.264 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:17.264 DEBUG:boto:Method: GET 2022-09-12T20:12:17.265 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:17.265 DEBUG:boto:Data: 2022-09-12T20:12:17.266 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.266 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.267 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.267 DEBUG:boto:Params: {} 2022-09-12T20:12:17.268 DEBUG:boto:Token: None 2022-09-12T20:12:17.269 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:17.269 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AWQmz8fttuSWBCmBsXS9fkIoIVk= 2022-09-12T20:12:17.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AWQmz8fttuSWBCmBsXS9fkIoIVk=', 'Content-Length': '0'} 2022-09-12T20:12:17.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bd5088f0ef60488-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.273 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:17.273 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:17.274 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:12:17.274 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:12:17.275 DEBUG:boto:Method: GET 2022-09-12T20:12:17.275 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:12:17.276 DEBUG:boto:Data: 2022-09-12T20:12:17.276 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.277 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.277 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.278 DEBUG:boto:Params: {} 2022-09-12T20:12:17.278 DEBUG:boto:Token: None 2022-09-12T20:12:17.279 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/ 2022-09-12T20:12:17.280 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34= 2022-09-12T20:12:17.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34=', 'Content-Length': '0'} 2022-09-12T20:12:17.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046d45a17237d1ee5-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.284 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:17.284 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:17.285 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:17.286 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:12:17.286 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:12:17.287 DEBUG:boto:Method: GET 2022-09-12T20:12:17.288 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:12:17.288 DEBUG:boto:Data: 2022-09-12T20:12:17.289 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.289 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.290 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.290 DEBUG:boto:Params: {} 2022-09-12T20:12:17.291 DEBUG:boto:Token: None 2022-09-12T20:12:17.292 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/ 2022-09-12T20:12:17.292 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34= 2022-09-12T20:12:17.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34=', 'Content-Length': '0'} 2022-09-12T20:12:17.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c821bcc09d5a6eea-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.298 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:17.299 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:17.299 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:17.300 DEBUG:boto:Method: HEAD 2022-09-12T20:12:17.300 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:12:17.301 DEBUG:boto:Data: 2022-09-12T20:12:17.301 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.302 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.302 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.303 DEBUG:boto:Params: {} 2022-09-12T20:12:17.303 DEBUG:boto:Token: None 2022-09-12T20:12:17.304 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj85 2022-09-12T20:12:17.304 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:44YoHp5uyNsSO/puVqjSiqL/pc4= 2022-09-12T20:12:17.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:44YoHp5uyNsSO/puVqjSiqL/pc4=', 'Content-Length': '0'} 2022-09-12T20:12:17.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5087280270c6679-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.307 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:17.308 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:17.308 DEBUG:boto:Method: HEAD 2022-09-12T20:12:17.309 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:12:17.309 DEBUG:boto:Data: 2022-09-12T20:12:17.310 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.310 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.311 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.311 DEBUG:boto:Params: {} 2022-09-12T20:12:17.312 DEBUG:boto:Token: None 2022-09-12T20:12:17.312 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj85 2022-09-12T20:12:17.313 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:44YoHp5uyNsSO/puVqjSiqL/pc4= 2022-09-12T20:12:17.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:44YoHp5uyNsSO/puVqjSiqL/pc4=', 'Content-Length': '0'} 2022-09-12T20:12:17.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002bffe6d8ab1d89f-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:12:17.317 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:17.317 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:17.318 DEBUG:boto:Method: GET 2022-09-12T20:12:17.318 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:12:17.319 DEBUG:boto:Data: 2022-09-12T20:12:17.319 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.320 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.320 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.321 DEBUG:boto:Params: {} 2022-09-12T20:12:17.321 DEBUG:boto:Token: None 2022-09-12T20:12:17.322 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj85 2022-09-12T20:12:17.323 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cQYigBDcaE0QP98obkw7q/WDpH4= 2022-09-12T20:12:17.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cQYigBDcaE0QP98obkw7q/WDpH4=', 'Content-Length': '0'} 2022-09-12T20:12:17.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b1e9cbc810e2174-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.366 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:17.366 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:17.367 DEBUG:boto:Method: GET 2022-09-12T20:12:17.368 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:12:17.368 DEBUG:boto:Data: 2022-09-12T20:12:17.369 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.369 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.370 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.370 DEBUG:boto:Params: {} 2022-09-12T20:12:17.371 DEBUG:boto:Token: None 2022-09-12T20:12:17.371 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj85 2022-09-12T20:12:17.372 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cQYigBDcaE0QP98obkw7q/WDpH4= 2022-09-12T20:12:17.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cQYigBDcaE0QP98obkw7q/WDpH4=', 'Content-Length': '0'} 2022-09-12T20:12:17.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000586488797ee700e2-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.375 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:17.376 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:17.376 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:12:17.377 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:12:17.377 DEBUG:boto:Method: GET 2022-09-12T20:12:17.378 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:12:17.378 DEBUG:boto:Data: 2022-09-12T20:12:17.379 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.379 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.380 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.380 DEBUG:boto:Params: {} 2022-09-12T20:12:17.381 DEBUG:boto:Token: None 2022-09-12T20:12:17.382 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/ 2022-09-12T20:12:17.382 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34= 2022-09-12T20:12:17.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34=', 'Content-Length': '0'} 2022-09-12T20:12:17.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000205a3dbe07aeb767-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.387 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:17.388 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:17.388 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:17.389 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:12:17.389 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:12:17.390 DEBUG:boto:Method: GET 2022-09-12T20:12:17.390 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:12:17.391 DEBUG:boto:Data: 2022-09-12T20:12:17.391 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.392 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.392 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.393 DEBUG:boto:Params: {} 2022-09-12T20:12:17.393 DEBUG:boto:Token: None 2022-09-12T20:12:17.394 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/ 2022-09-12T20:12:17.394 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34= 2022-09-12T20:12:17.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34=', 'Content-Length': '0'} 2022-09-12T20:12:17.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cefe34afea970c9-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.400 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:17.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:12:17.401 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:17.402 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:17.402 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:17.403 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:17.403 DEBUG:boto:Method: GET 2022-09-12T20:12:17.404 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:17.404 DEBUG:boto:Data: 2022-09-12T20:12:17.405 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.405 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.406 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.406 DEBUG:boto:Params: {} 2022-09-12T20:12:17.407 DEBUG:boto:Token: None 2022-09-12T20:12:17.408 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:17.408 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cA8t6up05b6HCzTUCFHXfxdA8Vc= 2022-09-12T20:12:17.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cA8t6up05b6HCzTUCFHXfxdA8Vc=', 'Content-Length': '0'} 2022-09-12T20:12:17.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b604e27b1d77a7b-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.412 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:17.412 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:17.413 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:17.413 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:17.414 DEBUG:boto:Method: GET 2022-09-12T20:12:17.414 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:17.415 DEBUG:boto:Data: 2022-09-12T20:12:17.415 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.416 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.417 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.417 DEBUG:boto:Params: {} 2022-09-12T20:12:17.418 DEBUG:boto:Token: None 2022-09-12T20:12:17.418 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:17.419 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cA8t6up05b6HCzTUCFHXfxdA8Vc= 2022-09-12T20:12:17.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cA8t6up05b6HCzTUCFHXfxdA8Vc=', 'Content-Length': '0'} 2022-09-12T20:12:17.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059df6da34fb1be8b-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.422 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:17.423 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:17.423 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:17.424 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:17.424 DEBUG:boto:Method: HEAD 2022-09-12T20:12:17.425 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:17.425 DEBUG:boto:Data: 2022-09-12T20:12:17.426 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.426 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.427 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.427 DEBUG:boto:Params: {} 2022-09-12T20:12:17.428 DEBUG:boto:Token: None 2022-09-12T20:12:17.428 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:17.429 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Mti8wVRjmELlsmpm9+HZB0w9e/k= 2022-09-12T20:12:17.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Mti8wVRjmELlsmpm9+HZB0w9e/k=', 'Content-Length': '0'} 2022-09-12T20:12:17.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d89019c456dfd18-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.432 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:17.433 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:17.433 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:17.434 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:17.434 DEBUG:boto:Method: HEAD 2022-09-12T20:12:17.435 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:17.435 DEBUG:boto:Data: 2022-09-12T20:12:17.436 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.436 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.437 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.437 DEBUG:boto:Params: {} 2022-09-12T20:12:17.438 DEBUG:boto:Token: None 2022-09-12T20:12:17.438 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:17.439 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Mti8wVRjmELlsmpm9+HZB0w9e/k= 2022-09-12T20:12:17.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Mti8wVRjmELlsmpm9+HZB0w9e/k=', 'Content-Length': '0'} 2022-09-12T20:12:17.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007db521fe86c2d840-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:12:17.443 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:17.443 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:17.444 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:17.444 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:17.445 DEBUG:boto:Method: GET 2022-09-12T20:12:17.446 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:17.446 DEBUG:boto:Data: 2022-09-12T20:12:17.447 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.447 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.448 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.448 DEBUG:boto:Params: {} 2022-09-12T20:12:17.449 DEBUG:boto:Token: None 2022-09-12T20:12:17.449 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:17.450 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cA8t6up05b6HCzTUCFHXfxdA8Vc= 2022-09-12T20:12:17.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cA8t6up05b6HCzTUCFHXfxdA8Vc=', 'Content-Length': '0'} 2022-09-12T20:12:17.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceb03a526ccc77d8-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.494 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:17.495 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:17.495 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:17.496 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:17.496 DEBUG:boto:Method: GET 2022-09-12T20:12:17.497 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:17.497 DEBUG:boto:Data: 2022-09-12T20:12:17.498 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.498 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.499 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.499 DEBUG:boto:Params: {} 2022-09-12T20:12:17.500 DEBUG:boto:Token: None 2022-09-12T20:12:17.501 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:17.501 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cA8t6up05b6HCzTUCFHXfxdA8Vc= 2022-09-12T20:12:17.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cA8t6up05b6HCzTUCFHXfxdA8Vc=', 'Content-Length': '0'} 2022-09-12T20:12:17.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003dc1c4a401a3860c-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.505 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:17.506 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:17.506 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:12:17.507 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:12:17.508 DEBUG:boto:Method: GET 2022-09-12T20:12:17.508 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:12:17.509 DEBUG:boto:Data: 2022-09-12T20:12:17.509 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.510 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.510 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.511 DEBUG:boto:Params: {} 2022-09-12T20:12:17.511 DEBUG:boto:Token: None 2022-09-12T20:12:17.512 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/ 2022-09-12T20:12:17.512 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34= 2022-09-12T20:12:17.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34=', 'Content-Length': '0'} 2022-09-12T20:12:17.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df6c9f1b2ad4861d-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.516 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:17.517 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:17.518 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:17.518 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:12:17.519 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:12:17.519 DEBUG:boto:Method: GET 2022-09-12T20:12:17.520 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:12:17.520 DEBUG:boto:Data: 2022-09-12T20:12:17.521 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.521 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.522 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.522 DEBUG:boto:Params: {} 2022-09-12T20:12:17.523 DEBUG:boto:Token: None 2022-09-12T20:12:17.524 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/ 2022-09-12T20:12:17.524 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34= 2022-09-12T20:12:17.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34=', 'Content-Length': '0'} 2022-09-12T20:12:17.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059c1eb19ad5da62e-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.530 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:17.531 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:17.531 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:17.532 DEBUG:boto:Method: HEAD 2022-09-12T20:12:17.532 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:12:17.533 DEBUG:boto:Data: 2022-09-12T20:12:17.533 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.534 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.535 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.535 DEBUG:boto:Params: {} 2022-09-12T20:12:17.536 DEBUG:boto:Token: None 2022-09-12T20:12:17.536 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj87 2022-09-12T20:12:17.537 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/vL8hmloMa3VWKNVc4cV/SA7FZ0= 2022-09-12T20:12:17.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/vL8hmloMa3VWKNVc4cV/SA7FZ0=', 'Content-Length': '0'} 2022-09-12T20:12:17.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae0d04ef794cbb73-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.541 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:17.541 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:17.542 DEBUG:boto:Method: HEAD 2022-09-12T20:12:17.542 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:12:17.543 DEBUG:boto:Data: 2022-09-12T20:12:17.543 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.544 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.544 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.545 DEBUG:boto:Params: {} 2022-09-12T20:12:17.545 DEBUG:boto:Token: None 2022-09-12T20:12:17.546 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj87 2022-09-12T20:12:17.547 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/vL8hmloMa3VWKNVc4cV/SA7FZ0= 2022-09-12T20:12:17.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/vL8hmloMa3VWKNVc4cV/SA7FZ0=', 'Content-Length': '0'} 2022-09-12T20:12:17.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae70f38bb94988f1-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:12:17.552 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:17.553 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:17.553 DEBUG:boto:Method: GET 2022-09-12T20:12:17.554 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:12:17.555 DEBUG:boto:Data: 2022-09-12T20:12:17.555 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.556 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.557 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.557 DEBUG:boto:Params: {} 2022-09-12T20:12:17.558 DEBUG:boto:Token: None 2022-09-12T20:12:17.559 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj87 2022-09-12T20:12:17.560 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+dgW0PkJS1LFxLK4DnFHbbhcxgQ= 2022-09-12T20:12:17.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+dgW0PkJS1LFxLK4DnFHbbhcxgQ=', 'Content-Length': '0'} 2022-09-12T20:12:17.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000305a4ee4b07a7071-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.564 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:17.564 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:17.565 DEBUG:boto:Method: GET 2022-09-12T20:12:17.565 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:12:17.566 DEBUG:boto:Data: 2022-09-12T20:12:17.566 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.567 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.567 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.568 DEBUG:boto:Params: {} 2022-09-12T20:12:17.568 DEBUG:boto:Token: None 2022-09-12T20:12:17.569 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj87 2022-09-12T20:12:17.570 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+dgW0PkJS1LFxLK4DnFHbbhcxgQ= 2022-09-12T20:12:17.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+dgW0PkJS1LFxLK4DnFHbbhcxgQ=', 'Content-Length': '0'} 2022-09-12T20:12:17.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008df591f132acdfd1-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.614 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:17.615 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:17.616 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:12:17.616 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:12:17.617 DEBUG:boto:Method: GET 2022-09-12T20:12:17.617 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:12:17.618 DEBUG:boto:Data: 2022-09-12T20:12:17.618 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.619 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.619 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.620 DEBUG:boto:Params: {} 2022-09-12T20:12:17.621 DEBUG:boto:Token: None 2022-09-12T20:12:17.621 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/ 2022-09-12T20:12:17.622 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34= 2022-09-12T20:12:17.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34=', 'Content-Length': '0'} 2022-09-12T20:12:17.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8222b2535f77367-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.628 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:17.629 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:17.630 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:17.631 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:12:17.632 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:12:17.633 DEBUG:boto:Method: GET 2022-09-12T20:12:17.634 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:12:17.635 DEBUG:boto:Data: 2022-09-12T20:12:17.636 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.636 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.637 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.637 DEBUG:boto:Params: {} 2022-09-12T20:12:17.638 DEBUG:boto:Token: None 2022-09-12T20:12:17.639 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/ 2022-09-12T20:12:17.639 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34= 2022-09-12T20:12:17.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34=', 'Content-Length': '0'} 2022-09-12T20:12:17.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3ff1060ad1f76ec-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.645 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:17.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:12:17.646 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:17.647 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:17.647 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:17.648 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:17.649 DEBUG:boto:Method: GET 2022-09-12T20:12:17.649 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:17.650 DEBUG:boto:Data: 2022-09-12T20:12:17.650 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.651 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.652 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.652 DEBUG:boto:Params: {} 2022-09-12T20:12:17.653 DEBUG:boto:Token: None 2022-09-12T20:12:17.653 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:17.654 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NnhQU3PXp5Fk352OvdhmMBeSfeQ= 2022-09-12T20:12:17.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NnhQU3PXp5Fk352OvdhmMBeSfeQ=', 'Content-Length': '0'} 2022-09-12T20:12:17.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000452dfc56076c406a-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.658 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:17.659 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:17.659 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:17.660 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:17.660 DEBUG:boto:Method: GET 2022-09-12T20:12:17.661 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:17.661 DEBUG:boto:Data: 2022-09-12T20:12:17.662 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.662 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.663 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.663 DEBUG:boto:Params: {} 2022-09-12T20:12:17.664 DEBUG:boto:Token: None 2022-09-12T20:12:17.665 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:17.665 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NnhQU3PXp5Fk352OvdhmMBeSfeQ= 2022-09-12T20:12:17.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NnhQU3PXp5Fk352OvdhmMBeSfeQ=', 'Content-Length': '0'} 2022-09-12T20:12:17.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000761998473c545c8d-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.669 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:17.669 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:17.670 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:17.670 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:17.671 DEBUG:boto:Method: HEAD 2022-09-12T20:12:17.671 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:17.672 DEBUG:boto:Data: 2022-09-12T20:12:17.672 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.673 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.673 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.674 DEBUG:boto:Params: {} 2022-09-12T20:12:17.674 DEBUG:boto:Token: None 2022-09-12T20:12:17.675 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:17.675 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0d9boPiZlDnhZ0hIvIJtoE4K/C4= 2022-09-12T20:12:17.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0d9boPiZlDnhZ0hIvIJtoE4K/C4=', 'Content-Length': '0'} 2022-09-12T20:12:17.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061b2a9972a6ef71a-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.679 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:17.679 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:17.680 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:17.680 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:17.681 DEBUG:boto:Method: HEAD 2022-09-12T20:12:17.682 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:17.682 DEBUG:boto:Data: 2022-09-12T20:12:17.683 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.684 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.684 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.685 DEBUG:boto:Params: {} 2022-09-12T20:12:17.685 DEBUG:boto:Token: None 2022-09-12T20:12:17.686 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:17.687 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0d9boPiZlDnhZ0hIvIJtoE4K/C4= 2022-09-12T20:12:17.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0d9boPiZlDnhZ0hIvIJtoE4K/C4=', 'Content-Length': '0'} 2022-09-12T20:12:17.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008856d03bd5a97e6-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:12:17.691 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:17.691 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:17.692 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:17.692 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:17.693 DEBUG:boto:Method: GET 2022-09-12T20:12:17.694 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:17.694 DEBUG:boto:Data: 2022-09-12T20:12:17.695 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.695 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.696 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.696 DEBUG:boto:Params: {} 2022-09-12T20:12:17.697 DEBUG:boto:Token: None 2022-09-12T20:12:17.698 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:17.698 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NnhQU3PXp5Fk352OvdhmMBeSfeQ= 2022-09-12T20:12:17.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NnhQU3PXp5Fk352OvdhmMBeSfeQ=', 'Content-Length': '0'} 2022-09-12T20:12:17.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ab0e2001ac8a735-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.746 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:17.747 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:17.747 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:17.748 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:17.748 DEBUG:boto:Method: GET 2022-09-12T20:12:17.749 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:17.750 DEBUG:boto:Data: 2022-09-12T20:12:17.750 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.751 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.751 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.752 DEBUG:boto:Params: {} 2022-09-12T20:12:17.752 DEBUG:boto:Token: None 2022-09-12T20:12:17.753 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:17.753 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NnhQU3PXp5Fk352OvdhmMBeSfeQ= 2022-09-12T20:12:17.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NnhQU3PXp5Fk352OvdhmMBeSfeQ=', 'Content-Length': '0'} 2022-09-12T20:12:17.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095a2e4c89c2b207c-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.757 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:17.758 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:17.758 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:12:17.759 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:12:17.760 DEBUG:boto:Method: GET 2022-09-12T20:12:17.760 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:12:17.761 DEBUG:boto:Data: 2022-09-12T20:12:17.762 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.762 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.763 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.763 DEBUG:boto:Params: {} 2022-09-12T20:12:17.764 DEBUG:boto:Token: None 2022-09-12T20:12:17.765 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/ 2022-09-12T20:12:17.765 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34= 2022-09-12T20:12:17.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34=', 'Content-Length': '0'} 2022-09-12T20:12:17.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000215bd03c20198adb-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.769 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:17.770 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:17.771 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:17.771 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:12:17.772 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:12:17.772 DEBUG:boto:Method: GET 2022-09-12T20:12:17.773 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:12:17.774 DEBUG:boto:Data: 2022-09-12T20:12:17.774 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.775 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.775 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.776 DEBUG:boto:Params: {} 2022-09-12T20:12:17.776 DEBUG:boto:Token: None 2022-09-12T20:12:17.777 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/ 2022-09-12T20:12:17.777 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34= 2022-09-12T20:12:17.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34=', 'Content-Length': '0'} 2022-09-12T20:12:17.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000441cb04d5e83a9b7-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.783 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:17.784 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:17.784 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:17.785 DEBUG:boto:Method: HEAD 2022-09-12T20:12:17.785 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:12:17.786 DEBUG:boto:Data: 2022-09-12T20:12:17.786 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.787 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.787 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.788 DEBUG:boto:Params: {} 2022-09-12T20:12:17.788 DEBUG:boto:Token: None 2022-09-12T20:12:17.789 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj92 2022-09-12T20:12:17.789 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5hCgot48ckYgPiXpDRgiPu4jemM= 2022-09-12T20:12:17.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5hCgot48ckYgPiXpDRgiPu4jemM=', 'Content-Length': '0'} 2022-09-12T20:12:17.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091e0a0631d0366ca-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.793 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:17.793 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:17.794 DEBUG:boto:Method: HEAD 2022-09-12T20:12:17.795 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:12:17.795 DEBUG:boto:Data: 2022-09-12T20:12:17.796 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.796 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.797 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.797 DEBUG:boto:Params: {} 2022-09-12T20:12:17.798 DEBUG:boto:Token: None 2022-09-12T20:12:17.798 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj92 2022-09-12T20:12:17.799 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5hCgot48ckYgPiXpDRgiPu4jemM= 2022-09-12T20:12:17.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5hCgot48ckYgPiXpDRgiPu4jemM=', 'Content-Length': '0'} 2022-09-12T20:12:17.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d02e9a8206a8cae-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:12:17.803 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:17.803 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:17.804 DEBUG:boto:Method: GET 2022-09-12T20:12:17.805 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:12:17.805 DEBUG:boto:Data: 2022-09-12T20:12:17.806 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.806 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.807 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.807 DEBUG:boto:Params: {} 2022-09-12T20:12:17.808 DEBUG:boto:Token: None 2022-09-12T20:12:17.808 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj92 2022-09-12T20:12:17.809 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eMHhi9Ne8lLKvmV+Xi5GMkm5C6A= 2022-09-12T20:12:17.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eMHhi9Ne8lLKvmV+Xi5GMkm5C6A=', 'Content-Length': '0'} 2022-09-12T20:12:17.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8a2816fe738748a-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.813 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:17.813 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:17.814 DEBUG:boto:Method: GET 2022-09-12T20:12:17.814 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:12:17.815 DEBUG:boto:Data: 2022-09-12T20:12:17.815 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.816 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.816 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.817 DEBUG:boto:Params: {} 2022-09-12T20:12:17.817 DEBUG:boto:Token: None 2022-09-12T20:12:17.818 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj92 2022-09-12T20:12:17.819 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eMHhi9Ne8lLKvmV+Xi5GMkm5C6A= 2022-09-12T20:12:17.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eMHhi9Ne8lLKvmV+Xi5GMkm5C6A=', 'Content-Length': '0'} 2022-09-12T20:12:17.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c1df240139f770c-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.862 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:17.863 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:17.863 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:12:17.864 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:12:17.865 DEBUG:boto:Method: GET 2022-09-12T20:12:17.865 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:12:17.866 DEBUG:boto:Data: 2022-09-12T20:12:17.866 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.867 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.867 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.868 DEBUG:boto:Params: {} 2022-09-12T20:12:17.868 DEBUG:boto:Token: None 2022-09-12T20:12:17.869 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/ 2022-09-12T20:12:17.870 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34= 2022-09-12T20:12:17.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34=', 'Content-Length': '0'} 2022-09-12T20:12:17.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086d9916bd050616f-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.874 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:17.875 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:17.875 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:17.876 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:12:17.876 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:12:17.877 DEBUG:boto:Method: GET 2022-09-12T20:12:17.877 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:12:17.878 DEBUG:boto:Data: 2022-09-12T20:12:17.878 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.879 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.879 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.880 DEBUG:boto:Params: {} 2022-09-12T20:12:17.881 DEBUG:boto:Token: None 2022-09-12T20:12:17.881 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/ 2022-09-12T20:12:17.882 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34= 2022-09-12T20:12:17.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34=', 'Content-Length': '0'} 2022-09-12T20:12:17.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072805ad2c795f7af-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.887 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:17.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:12:17.889 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:17.889 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:17.890 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:17.890 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:17.891 DEBUG:boto:Method: GET 2022-09-12T20:12:17.891 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:17.892 DEBUG:boto:Data: 2022-09-12T20:12:17.892 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.893 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.893 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.894 DEBUG:boto:Params: {} 2022-09-12T20:12:17.894 DEBUG:boto:Token: None 2022-09-12T20:12:17.895 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:17.896 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:krfZzDhMu+obkQDwOAU//fy7Phw= 2022-09-12T20:12:17.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:krfZzDhMu+obkQDwOAU//fy7Phw=', 'Content-Length': '0'} 2022-09-12T20:12:17.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021e792f2cd2337ab-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.899 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:17.900 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:17.900 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:17.901 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:17.901 DEBUG:boto:Method: GET 2022-09-12T20:12:17.902 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:17.902 DEBUG:boto:Data: 2022-09-12T20:12:17.903 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.903 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.904 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.904 DEBUG:boto:Params: {} 2022-09-12T20:12:17.905 DEBUG:boto:Token: None 2022-09-12T20:12:17.905 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:17.906 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:krfZzDhMu+obkQDwOAU//fy7Phw= 2022-09-12T20:12:17.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:krfZzDhMu+obkQDwOAU//fy7Phw=', 'Content-Length': '0'} 2022-09-12T20:12:17.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3d3fd57f5ee0915-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.910 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:17.910 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:17.911 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:17.911 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:17.912 DEBUG:boto:Method: HEAD 2022-09-12T20:12:17.912 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:17.913 DEBUG:boto:Data: 2022-09-12T20:12:17.913 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.914 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.914 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.915 DEBUG:boto:Params: {} 2022-09-12T20:12:17.915 DEBUG:boto:Token: None 2022-09-12T20:12:17.916 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:17.916 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1wY3JRA77mbqrFnQw19/hHcCGNI= 2022-09-12T20:12:17.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1wY3JRA77mbqrFnQw19/hHcCGNI=', 'Content-Length': '0'} 2022-09-12T20:12:17.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001154438ea0edc61d-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.920 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:17.920 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:17.921 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:17.921 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:17.922 DEBUG:boto:Method: HEAD 2022-09-12T20:12:17.922 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:17.923 DEBUG:boto:Data: 2022-09-12T20:12:17.923 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.924 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.924 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.925 DEBUG:boto:Params: {} 2022-09-12T20:12:17.925 DEBUG:boto:Token: None 2022-09-12T20:12:17.926 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:17.926 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1wY3JRA77mbqrFnQw19/hHcCGNI= 2022-09-12T20:12:17.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1wY3JRA77mbqrFnQw19/hHcCGNI=', 'Content-Length': '0'} 2022-09-12T20:12:17.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084bbd7d17dbeeac0-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:12:17.930 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:17.931 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:17.932 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:17.932 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:17.933 DEBUG:boto:Method: GET 2022-09-12T20:12:17.933 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:17.934 DEBUG:boto:Data: 2022-09-12T20:12:17.934 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.935 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.935 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.936 DEBUG:boto:Params: {} 2022-09-12T20:12:17.936 DEBUG:boto:Token: None 2022-09-12T20:12:17.937 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:17.937 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:krfZzDhMu+obkQDwOAU//fy7Phw= 2022-09-12T20:12:17.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:krfZzDhMu+obkQDwOAU//fy7Phw=', 'Content-Length': '0'} 2022-09-12T20:12:17.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d34960489f7e57ab-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.941 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:17.942 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:17.942 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:17.943 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:17.943 DEBUG:boto:Method: GET 2022-09-12T20:12:17.944 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:17.944 DEBUG:boto:Data: 2022-09-12T20:12:17.945 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.945 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.946 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.946 DEBUG:boto:Params: {} 2022-09-12T20:12:17.947 DEBUG:boto:Token: None 2022-09-12T20:12:17.947 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:17.948 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:krfZzDhMu+obkQDwOAU//fy7Phw= 2022-09-12T20:12:17.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:krfZzDhMu+obkQDwOAU//fy7Phw=', 'Content-Length': '0'} 2022-09-12T20:12:17.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec4445a7bb649d66-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.952 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:17.952 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:17.953 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:12:17.953 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:12:17.954 DEBUG:boto:Method: GET 2022-09-12T20:12:17.954 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:12:17.955 DEBUG:boto:Data: 2022-09-12T20:12:17.955 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.956 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:17.956 DEBUG:boto:Port: 8001 2022-09-12T20:12:17.957 DEBUG:boto:Params: {} 2022-09-12T20:12:17.957 DEBUG:boto:Token: None 2022-09-12T20:12:17.958 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/ 2022-09-12T20:12:17.959 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34= 2022-09-12T20:12:17.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34=', 'Content-Length': '0'} 2022-09-12T20:12:17.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031a26c9376839358-00631f92a1-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:17.963 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:17.963 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:17.964 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:17.964 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:12:17.965 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:12:17.965 DEBUG:boto:Method: GET 2022-09-12T20:12:17.966 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:12:17.967 DEBUG:boto:Data: 2022-09-12T20:12:17.967 DEBUG:boto:Headers: {} 2022-09-12T20:12:17.968 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:17.968 DEBUG:boto:Port: 8000 2022-09-12T20:12:17.969 DEBUG:boto:Params: {} 2022-09-12T20:12:17.969 DEBUG:boto:Token: None 2022-09-12T20:12:17.970 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:17 GMT /jdpiak-75/ 2022-09-12T20:12:17.970 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34= 2022-09-12T20:12:17.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:La5OJDqfoZzO1YYW8VijsEhrk34=', 'Content-Length': '0'} 2022-09-12T20:12:17.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa73c07780f58512-00631f92a1-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.018 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:18.019 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:18.019 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:18.020 DEBUG:boto:Method: HEAD 2022-09-12T20:12:18.020 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:12:18.021 DEBUG:boto:Data: 2022-09-12T20:12:18.021 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.022 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.022 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.023 DEBUG:boto:Params: {} 2022-09-12T20:12:18.023 DEBUG:boto:Token: None 2022-09-12T20:12:18.024 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj95 2022-09-12T20:12:18.024 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:31hd2PkALCiBT561vD7s/mTqUWE= 2022-09-12T20:12:18.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:31hd2PkALCiBT561vD7s/mTqUWE=', 'Content-Length': '0'} 2022-09-12T20:12:18.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026eef076061e8f57-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.028 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:18.028 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:18.029 DEBUG:boto:Method: HEAD 2022-09-12T20:12:18.029 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:12:18.030 DEBUG:boto:Data: 2022-09-12T20:12:18.030 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.031 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:18.031 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.032 DEBUG:boto:Params: {} 2022-09-12T20:12:18.032 DEBUG:boto:Token: None 2022-09-12T20:12:18.033 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj95 2022-09-12T20:12:18.034 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:31hd2PkALCiBT561vD7s/mTqUWE= 2022-09-12T20:12:18.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:31hd2PkALCiBT561vD7s/mTqUWE=', 'Content-Length': '0'} 2022-09-12T20:12:18.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfbb0c7763763705-00631f92a2-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:12:18.038 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:18.038 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:18.039 DEBUG:boto:Method: GET 2022-09-12T20:12:18.039 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:12:18.040 DEBUG:boto:Data: 2022-09-12T20:12:18.040 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.041 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.041 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.042 DEBUG:boto:Params: {} 2022-09-12T20:12:18.043 DEBUG:boto:Token: None 2022-09-12T20:12:18.043 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj95 2022-09-12T20:12:18.044 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ItXK1gj4CDyUO6fW4PZXuQhzVUM= 2022-09-12T20:12:18.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ItXK1gj4CDyUO6fW4PZXuQhzVUM=', 'Content-Length': '0'} 2022-09-12T20:12:18.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091ff5cb960461b6b-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.090 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:18.091 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:18.091 DEBUG:boto:Method: GET 2022-09-12T20:12:18.092 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:12:18.092 DEBUG:boto:Data: 2022-09-12T20:12:18.093 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.093 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:18.094 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.095 DEBUG:boto:Params: {} 2022-09-12T20:12:18.095 DEBUG:boto:Token: None 2022-09-12T20:12:18.096 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj95 2022-09-12T20:12:18.096 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ItXK1gj4CDyUO6fW4PZXuQhzVUM= 2022-09-12T20:12:18.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ItXK1gj4CDyUO6fW4PZXuQhzVUM=', 'Content-Length': '0'} 2022-09-12T20:12:18.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee40062fa1afdebf-00631f92a2-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.100 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:18.101 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:18.101 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:12:18.102 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:12:18.102 DEBUG:boto:Method: GET 2022-09-12T20:12:18.103 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:12:18.103 DEBUG:boto:Data: 2022-09-12T20:12:18.104 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.104 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.105 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.105 DEBUG:boto:Params: {} 2022-09-12T20:12:18.106 DEBUG:boto:Token: None 2022-09-12T20:12:18.107 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/ 2022-09-12T20:12:18.107 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o= 2022-09-12T20:12:18.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o=', 'Content-Length': '0'} 2022-09-12T20:12:18.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072dbc4eba94d2ca1-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.111 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:18.112 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:18.113 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:18.113 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:12:18.114 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:12:18.114 DEBUG:boto:Method: GET 2022-09-12T20:12:18.115 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:12:18.115 DEBUG:boto:Data: 2022-09-12T20:12:18.116 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.116 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:18.117 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.117 DEBUG:boto:Params: {} 2022-09-12T20:12:18.118 DEBUG:boto:Token: None 2022-09-12T20:12:18.118 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/ 2022-09-12T20:12:18.119 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o= 2022-09-12T20:12:18.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o=', 'Content-Length': '0'} 2022-09-12T20:12:18.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8efdf7e94b81722-00631f92a2-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.124 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:18.125 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-75 zones={test-zone3, test-zone1} 2022-09-12T20:12:18.126 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-75 zones={test-zone3, test-zone2} 2022-09-12T20:12:18.126 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:18.127 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:18.127 DEBUG:boto:Method: HEAD 2022-09-12T20:12:18.128 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:12:18.128 DEBUG:boto:Data: 2022-09-12T20:12:18.129 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.129 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.130 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.130 DEBUG:boto:Params: {} 2022-09-12T20:12:18.131 DEBUG:boto:Token: None 2022-09-12T20:12:18.131 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/ 2022-09-12T20:12:18.132 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nud7SC3YosmqN97Z+GERypDgh8M= 2022-09-12T20:12:18.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nud7SC3YosmqN97Z+GERypDgh8M=', 'Content-Length': '0'} 2022-09-12T20:12:18.135 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa88008363962c9f-00631f92a2-1826-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.136 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:18.137 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:18.137 DEBUG:boto:Method: HEAD 2022-09-12T20:12:18.138 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:12:18.138 DEBUG:boto:Data: 2022-09-12T20:12:18.139 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.139 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:18.140 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.140 DEBUG:boto:Params: {} 2022-09-12T20:12:18.141 DEBUG:boto:Token: None 2022-09-12T20:12:18.141 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/ 2022-09-12T20:12:18.142 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nud7SC3YosmqN97Z+GERypDgh8M= 2022-09-12T20:12:18.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nud7SC3YosmqN97Z+GERypDgh8M=', 'Content-Length': '0'} 2022-09-12T20:12:18.145 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038e2573b44116758-00631f92a2-189d-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.146 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T20:12:18.146 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:18.147 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:18.147 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:18.148 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:18.148 DEBUG:boto:Method: GET 2022-09-12T20:12:18.149 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:18.149 DEBUG:boto:Data: 2022-09-12T20:12:18.150 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.150 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.151 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.151 DEBUG:boto:Params: {} 2022-09-12T20:12:18.152 DEBUG:boto:Token: None 2022-09-12T20:12:18.152 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/?versions 2022-09-12T20:12:18.153 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z4YAFvcziox4Ust2ynwe71rhzDs= 2022-09-12T20:12:18.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z4YAFvcziox4Ust2ynwe71rhzDs=', 'Content-Length': '0'} 2022-09-12T20:12:18.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca75ab16bc935829-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.157 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:18.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:12:18.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-12T20:12:18.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-12T20:12:18.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-12T20:12:18.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:12:18.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:12:18.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:12:18.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-12T20:12:18.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-12T20:12:18.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:12:18.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:12:18.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:12:18.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:12:18.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-12T20:12:18.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:12:18.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:12:18.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:12:18.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-12T20:12:18.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:12:18.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-12T20:12:18.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:12:18.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:12:18.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-12T20:12:18.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-12T20:12:18.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:12:18.171 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T20:12:18.171 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:18.172 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:18.172 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:18.173 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:18.173 DEBUG:boto:Method: GET 2022-09-12T20:12:18.174 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:18.174 DEBUG:boto:Data: 2022-09-12T20:12:18.175 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.175 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:18.176 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.176 DEBUG:boto:Params: {} 2022-09-12T20:12:18.177 DEBUG:boto:Token: None 2022-09-12T20:12:18.177 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/?versions 2022-09-12T20:12:18.178 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z4YAFvcziox4Ust2ynwe71rhzDs= 2022-09-12T20:12:18.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z4YAFvcziox4Ust2ynwe71rhzDs=', 'Content-Length': '0'} 2022-09-12T20:12:18.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029d78922e37b3b55-00631f92a2-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.182 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:18.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:12:18.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-12T20:12:18.185 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-12T20:12:18.185 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-12T20:12:18.186 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:12:18.186 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:12:18.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:12:18.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-12T20:12:18.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-12T20:12:18.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:12:18.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:12:18.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:12:18.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:12:18.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-12T20:12:18.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:12:18.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:12:18.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:12:18.192 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-12T20:12:18.192 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:12:18.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-12T20:12:18.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:12:18.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:12:18.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-12T20:12:18.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-12T20:12:18.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:12:18.196 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:18.196 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:18.197 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:18.197 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:18.198 DEBUG:boto:Method: GET 2022-09-12T20:12:18.198 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:18.199 DEBUG:boto:Data: 2022-09-12T20:12:18.199 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.200 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.200 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.201 DEBUG:boto:Params: {} 2022-09-12T20:12:18.201 DEBUG:boto:Token: None 2022-09-12T20:12:18.202 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/?versions 2022-09-12T20:12:18.203 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z4YAFvcziox4Ust2ynwe71rhzDs= 2022-09-12T20:12:18.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z4YAFvcziox4Ust2ynwe71rhzDs=', 'Content-Length': '0'} 2022-09-12T20:12:18.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b54337224aa75e97-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.207 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:18.209 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:18.210 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:18.210 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:18.211 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:18.211 DEBUG:boto:Method: GET 2022-09-12T20:12:18.212 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:12:18.212 DEBUG:boto:Data: 2022-09-12T20:12:18.213 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.213 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:18.214 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.214 DEBUG:boto:Params: {} 2022-09-12T20:12:18.215 DEBUG:boto:Token: None 2022-09-12T20:12:18.216 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/?versions 2022-09-12T20:12:18.216 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z4YAFvcziox4Ust2ynwe71rhzDs= 2022-09-12T20:12:18.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z4YAFvcziox4Ust2ynwe71rhzDs=', 'Content-Length': '0'} 2022-09-12T20:12:18.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004efc5e03f91090c5-00631f92a2-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.220 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:18.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:12:18.223 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:18.223 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:18.224 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:18.224 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:18.225 DEBUG:boto:Method: GET 2022-09-12T20:12:18.226 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:18.227 DEBUG:boto:Data: 2022-09-12T20:12:18.227 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.228 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.228 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.229 DEBUG:boto:Params: {} 2022-09-12T20:12:18.229 DEBUG:boto:Token: None 2022-09-12T20:12:18.230 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:18.231 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QgpufJ4jBFTBWRKi1erJFvZ6ikk= 2022-09-12T20:12:18.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QgpufJ4jBFTBWRKi1erJFvZ6ikk=', 'Content-Length': '0'} 2022-09-12T20:12:18.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8e24e481f7d39d3-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.234 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:18.235 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:18.235 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:18.236 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:18.236 DEBUG:boto:Method: GET 2022-09-12T20:12:18.237 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:18.238 DEBUG:boto:Data: 2022-09-12T20:12:18.238 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.239 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:18.239 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.240 DEBUG:boto:Params: {} 2022-09-12T20:12:18.241 DEBUG:boto:Token: None 2022-09-12T20:12:18.242 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:18.242 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QgpufJ4jBFTBWRKi1erJFvZ6ikk= 2022-09-12T20:12:18.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QgpufJ4jBFTBWRKi1erJFvZ6ikk=', 'Content-Length': '0'} 2022-09-12T20:12:18.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc699806ff555c11-00631f92a2-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.286 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:18.287 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:18.287 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:18.288 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:18.288 DEBUG:boto:Method: HEAD 2022-09-12T20:12:18.289 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:18.289 DEBUG:boto:Data: 2022-09-12T20:12:18.290 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.290 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.291 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.291 DEBUG:boto:Params: {} 2022-09-12T20:12:18.292 DEBUG:boto:Token: None 2022-09-12T20:12:18.292 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:18.293 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vzZ+HOuSSqmUXmUchJ0iDXFt6uE= 2022-09-12T20:12:18.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vzZ+HOuSSqmUXmUchJ0iDXFt6uE=', 'Content-Length': '0'} 2022-09-12T20:12:18.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047ea4b65ae500c33-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.296 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:18.297 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:18.297 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:18.298 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:18.298 DEBUG:boto:Method: HEAD 2022-09-12T20:12:18.299 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:18.299 DEBUG:boto:Data: 2022-09-12T20:12:18.300 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.300 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:18.301 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.301 DEBUG:boto:Params: {} 2022-09-12T20:12:18.302 DEBUG:boto:Token: None 2022-09-12T20:12:18.303 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:18.303 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vzZ+HOuSSqmUXmUchJ0iDXFt6uE= 2022-09-12T20:12:18.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vzZ+HOuSSqmUXmUchJ0iDXFt6uE=', 'Content-Length': '0'} 2022-09-12T20:12:18.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064e8cdffccb6fa1a-00631f92a2-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:12:18.307 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:18.308 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:18.308 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:18.309 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:18.309 DEBUG:boto:Method: GET 2022-09-12T20:12:18.310 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:18.311 DEBUG:boto:Data: 2022-09-12T20:12:18.311 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.312 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.312 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.313 DEBUG:boto:Params: {} 2022-09-12T20:12:18.313 DEBUG:boto:Token: None 2022-09-12T20:12:18.314 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:18.314 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QgpufJ4jBFTBWRKi1erJFvZ6ikk= 2022-09-12T20:12:18.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QgpufJ4jBFTBWRKi1erJFvZ6ikk=', 'Content-Length': '0'} 2022-09-12T20:12:18.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006927355d75a5f9a1-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.318 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:18.319 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:18.319 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:18.320 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:12:18.320 DEBUG:boto:Method: GET 2022-09-12T20:12:18.321 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:18.322 DEBUG:boto:Data: 2022-09-12T20:12:18.322 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.323 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:18.323 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.324 DEBUG:boto:Params: {} 2022-09-12T20:12:18.324 DEBUG:boto:Token: None 2022-09-12T20:12:18.325 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:12:18.326 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QgpufJ4jBFTBWRKi1erJFvZ6ikk= 2022-09-12T20:12:18.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QgpufJ4jBFTBWRKi1erJFvZ6ikk=', 'Content-Length': '0'} 2022-09-12T20:12:18.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001107d37bed7db48-00631f92a2-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.329 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:18.330 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:18.330 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:12:18.331 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:12:18.331 DEBUG:boto:Method: GET 2022-09-12T20:12:18.332 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:12:18.332 DEBUG:boto:Data: 2022-09-12T20:12:18.333 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.334 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.334 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.335 DEBUG:boto:Params: {} 2022-09-12T20:12:18.335 DEBUG:boto:Token: None 2022-09-12T20:12:18.336 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/ 2022-09-12T20:12:18.337 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o= 2022-09-12T20:12:18.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o=', 'Content-Length': '0'} 2022-09-12T20:12:18.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cfbb37ccd9c912a-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.341 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:18.342 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:18.342 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:18.343 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:12:18.343 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:12:18.344 DEBUG:boto:Method: GET 2022-09-12T20:12:18.344 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:12:18.345 DEBUG:boto:Data: 2022-09-12T20:12:18.345 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.346 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:18.346 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.347 DEBUG:boto:Params: {} 2022-09-12T20:12:18.347 DEBUG:boto:Token: None 2022-09-12T20:12:18.348 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/ 2022-09-12T20:12:18.349 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o= 2022-09-12T20:12:18.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o=', 'Content-Length': '0'} 2022-09-12T20:12:18.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000267977c5476718b4-00631f92a2-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.353 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:18.354 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:18.354 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:18.355 DEBUG:boto:Method: HEAD 2022-09-12T20:12:18.355 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:12:18.356 DEBUG:boto:Data: 2022-09-12T20:12:18.356 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.357 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.357 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.358 DEBUG:boto:Params: {} 2022-09-12T20:12:18.358 DEBUG:boto:Token: None 2022-09-12T20:12:18.359 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj11 2022-09-12T20:12:18.359 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1Cb8qy8bI8EZLgHZ06rhaMm9tRM= 2022-09-12T20:12:18.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1Cb8qy8bI8EZLgHZ06rhaMm9tRM=', 'Content-Length': '0'} 2022-09-12T20:12:18.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6686ee16af0de19-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.363 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:18.363 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:18.364 DEBUG:boto:Method: HEAD 2022-09-12T20:12:18.364 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:12:18.365 DEBUG:boto:Data: 2022-09-12T20:12:18.365 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.366 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:18.366 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.367 DEBUG:boto:Params: {} 2022-09-12T20:12:18.367 DEBUG:boto:Token: None 2022-09-12T20:12:18.368 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj11 2022-09-12T20:12:18.368 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1Cb8qy8bI8EZLgHZ06rhaMm9tRM= 2022-09-12T20:12:18.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1Cb8qy8bI8EZLgHZ06rhaMm9tRM=', 'Content-Length': '0'} 2022-09-12T20:12:18.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4cad561cdd63f16-00631f92a2-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:12:18.372 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:18.373 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:18.373 DEBUG:boto:Method: GET 2022-09-12T20:12:18.374 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:12:18.374 DEBUG:boto:Data: 2022-09-12T20:12:18.375 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.375 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.376 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.376 DEBUG:boto:Params: {} 2022-09-12T20:12:18.377 DEBUG:boto:Token: None 2022-09-12T20:12:18.378 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj11 2022-09-12T20:12:18.378 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:v3S6SngEfn0AE7+xy+8tV0kKdRw= 2022-09-12T20:12:18.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:v3S6SngEfn0AE7+xy+8tV0kKdRw=', 'Content-Length': '0'} 2022-09-12T20:12:18.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1c79726460dcc4c-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.382 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:12:18.382 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:12:18.383 DEBUG:boto:Method: GET 2022-09-12T20:12:18.383 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:12:18.384 DEBUG:boto:Data: 2022-09-12T20:12:18.384 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.385 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:18.385 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.386 DEBUG:boto:Params: {} 2022-09-12T20:12:18.386 DEBUG:boto:Token: None 2022-09-12T20:12:18.387 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj11 2022-09-12T20:12:18.388 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:v3S6SngEfn0AE7+xy+8tV0kKdRw= 2022-09-12T20:12:18.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:v3S6SngEfn0AE7+xy+8tV0kKdRw=', 'Content-Length': '0'} 2022-09-12T20:12:18.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009eaf615a9d86239e-00631f92a2-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.391 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:18.392 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:18.392 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:12:18.393 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:12:18.393 DEBUG:boto:Method: GET 2022-09-12T20:12:18.394 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:12:18.395 DEBUG:boto:Data: 2022-09-12T20:12:18.395 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.396 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.396 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.397 DEBUG:boto:Params: {} 2022-09-12T20:12:18.397 DEBUG:boto:Token: None 2022-09-12T20:12:18.398 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/ 2022-09-12T20:12:18.398 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o= 2022-09-12T20:12:18.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o=', 'Content-Length': '0'} 2022-09-12T20:12:18.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3df1339255585c2-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.403 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:18.403 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:18.404 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:18.405 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:12:18.405 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:12:18.406 DEBUG:boto:Method: GET 2022-09-12T20:12:18.406 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:12:18.407 DEBUG:boto:Data: 2022-09-12T20:12:18.407 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.408 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:18.408 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.409 DEBUG:boto:Params: {} 2022-09-12T20:12:18.409 DEBUG:boto:Token: None 2022-09-12T20:12:18.410 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/ 2022-09-12T20:12:18.410 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o= 2022-09-12T20:12:18.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o=', 'Content-Length': '0'} 2022-09-12T20:12:18.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071bb35312c573010-00631f92a2-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.458 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:18.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:12:18.460 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:18.460 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:18.461 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:18.461 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:18.462 DEBUG:boto:Method: GET 2022-09-12T20:12:18.462 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:18.463 DEBUG:boto:Data: 2022-09-12T20:12:18.463 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.464 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.465 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.465 DEBUG:boto:Params: {} 2022-09-12T20:12:18.466 DEBUG:boto:Token: None 2022-09-12T20:12:18.466 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:18.467 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pUOE5j+nH7eaQQqETuxxBqZmw/E= 2022-09-12T20:12:18.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pUOE5j+nH7eaQQqETuxxBqZmw/E=', 'Content-Length': '0'} 2022-09-12T20:12:18.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c089b791a0c1da8a-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.471 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:18.471 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:18.472 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:18.472 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:18.473 DEBUG:boto:Method: GET 2022-09-12T20:12:18.473 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:18.474 DEBUG:boto:Data: 2022-09-12T20:12:18.474 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.475 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:18.475 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.476 DEBUG:boto:Params: {} 2022-09-12T20:12:18.476 DEBUG:boto:Token: None 2022-09-12T20:12:18.477 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:18.477 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pUOE5j+nH7eaQQqETuxxBqZmw/E= 2022-09-12T20:12:18.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pUOE5j+nH7eaQQqETuxxBqZmw/E=', 'Content-Length': '0'} 2022-09-12T20:12:18.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010e4e1bd81ad8c41-00631f92a2-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.481 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:18.482 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:18.482 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:18.483 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:18.483 DEBUG:boto:Method: HEAD 2022-09-12T20:12:18.484 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:18.484 DEBUG:boto:Data: 2022-09-12T20:12:18.485 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.485 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.486 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.486 DEBUG:boto:Params: {} 2022-09-12T20:12:18.487 DEBUG:boto:Token: None 2022-09-12T20:12:18.487 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:18.488 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gu/FfvhxRarz5PA506wv2WLx7Mc= 2022-09-12T20:12:18.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gu/FfvhxRarz5PA506wv2WLx7Mc=', 'Content-Length': '0'} 2022-09-12T20:12:18.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e927dca1cb702c79-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.491 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:18.491 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:18.492 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:18.492 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:18.493 DEBUG:boto:Method: HEAD 2022-09-12T20:12:18.493 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:18.494 DEBUG:boto:Data: 2022-09-12T20:12:18.494 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.495 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:18.495 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.496 DEBUG:boto:Params: {} 2022-09-12T20:12:18.496 DEBUG:boto:Token: None 2022-09-12T20:12:18.497 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:18.498 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gu/FfvhxRarz5PA506wv2WLx7Mc= 2022-09-12T20:12:18.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gu/FfvhxRarz5PA506wv2WLx7Mc=', 'Content-Length': '0'} 2022-09-12T20:12:18.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041fb14638e041989-00631f92a2-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:12:18.502 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:18.502 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:18.503 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:18.503 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:18.504 DEBUG:boto:Method: GET 2022-09-12T20:12:18.504 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:18.505 DEBUG:boto:Data: 2022-09-12T20:12:18.505 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.506 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.506 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.507 DEBUG:boto:Params: {} 2022-09-12T20:12:18.508 DEBUG:boto:Token: None 2022-09-12T20:12:18.508 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:18.509 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pUOE5j+nH7eaQQqETuxxBqZmw/E= 2022-09-12T20:12:18.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pUOE5j+nH7eaQQqETuxxBqZmw/E=', 'Content-Length': '0'} 2022-09-12T20:12:18.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2a0eefa7cd30eeb-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.554 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:18.555 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:18.556 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:18.556 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:12:18.557 DEBUG:boto:Method: GET 2022-09-12T20:12:18.557 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:18.558 DEBUG:boto:Data: 2022-09-12T20:12:18.558 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.559 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:18.559 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.560 DEBUG:boto:Params: {} 2022-09-12T20:12:18.561 DEBUG:boto:Token: None 2022-09-12T20:12:18.561 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:12:18.562 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pUOE5j+nH7eaQQqETuxxBqZmw/E= 2022-09-12T20:12:18.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pUOE5j+nH7eaQQqETuxxBqZmw/E=', 'Content-Length': '0'} 2022-09-12T20:12:18.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff8682a2b1bf045b-00631f92a2-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.566 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:18.567 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:18.567 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:12:18.568 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:12:18.568 DEBUG:boto:Method: GET 2022-09-12T20:12:18.569 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:12:18.569 DEBUG:boto:Data: 2022-09-12T20:12:18.570 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.571 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.571 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.572 DEBUG:boto:Params: {} 2022-09-12T20:12:18.573 DEBUG:boto:Token: None 2022-09-12T20:12:18.573 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/ 2022-09-12T20:12:18.574 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o= 2022-09-12T20:12:18.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o=', 'Content-Length': '0'} 2022-09-12T20:12:18.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d08b69811f4ce8b0-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.579 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:18.580 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:18.580 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:18.581 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:12:18.581 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:12:18.582 DEBUG:boto:Method: GET 2022-09-12T20:12:18.583 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:12:18.583 DEBUG:boto:Data: 2022-09-12T20:12:18.584 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.584 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:18.585 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.586 DEBUG:boto:Params: {} 2022-09-12T20:12:18.586 DEBUG:boto:Token: None 2022-09-12T20:12:18.587 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/ 2022-09-12T20:12:18.588 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o= 2022-09-12T20:12:18.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o=', 'Content-Length': '0'} 2022-09-12T20:12:18.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005998fef9b0b504c4-00631f92a2-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.593 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:18.593 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:18.594 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:18.595 DEBUG:boto:Method: HEAD 2022-09-12T20:12:18.595 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:12:18.596 DEBUG:boto:Data: 2022-09-12T20:12:18.596 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.597 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.597 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.598 DEBUG:boto:Params: {} 2022-09-12T20:12:18.598 DEBUG:boto:Token: None 2022-09-12T20:12:18.599 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj115 2022-09-12T20:12:18.599 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:v7qJWJg40vv20ez4B9KOSAz9sbo= 2022-09-12T20:12:18.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:v7qJWJg40vv20ez4B9KOSAz9sbo=', 'Content-Length': '0'} 2022-09-12T20:12:18.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007193417caf15603a-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.603 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:18.604 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:18.605 DEBUG:boto:Method: HEAD 2022-09-12T20:12:18.605 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:12:18.606 DEBUG:boto:Data: 2022-09-12T20:12:18.607 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.607 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:18.608 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.609 DEBUG:boto:Params: {} 2022-09-12T20:12:18.609 DEBUG:boto:Token: None 2022-09-12T20:12:18.610 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj115 2022-09-12T20:12:18.611 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:v7qJWJg40vv20ez4B9KOSAz9sbo= 2022-09-12T20:12:18.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:v7qJWJg40vv20ez4B9KOSAz9sbo=', 'Content-Length': '0'} 2022-09-12T20:12:18.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001746413dadb1ef72-00631f92a2-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:12:18.615 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:18.616 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:18.617 DEBUG:boto:Method: GET 2022-09-12T20:12:18.618 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:12:18.619 DEBUG:boto:Data: 2022-09-12T20:12:18.619 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.620 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.621 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.622 DEBUG:boto:Params: {} 2022-09-12T20:12:18.623 DEBUG:boto:Token: None 2022-09-12T20:12:18.624 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj115 2022-09-12T20:12:18.625 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:y91J6hyKTIhuHjClDBFRUOFJh+Q= 2022-09-12T20:12:18.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:y91J6hyKTIhuHjClDBFRUOFJh+Q=', 'Content-Length': '0'} 2022-09-12T20:12:18.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccd61af0c5b03d9d-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.670 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:12:18.671 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:12:18.672 DEBUG:boto:Method: GET 2022-09-12T20:12:18.673 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:12:18.674 DEBUG:boto:Data: 2022-09-12T20:12:18.674 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.675 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:18.676 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.677 DEBUG:boto:Params: {} 2022-09-12T20:12:18.678 DEBUG:boto:Token: None 2022-09-12T20:12:18.679 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj115 2022-09-12T20:12:18.680 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:y91J6hyKTIhuHjClDBFRUOFJh+Q= 2022-09-12T20:12:18.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:y91J6hyKTIhuHjClDBFRUOFJh+Q=', 'Content-Length': '0'} 2022-09-12T20:12:18.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002745f501269900de-00631f92a2-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.684 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:18.685 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:18.685 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:12:18.686 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:12:18.687 DEBUG:boto:Method: GET 2022-09-12T20:12:18.688 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:12:18.688 DEBUG:boto:Data: 2022-09-12T20:12:18.689 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.689 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.690 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.691 DEBUG:boto:Params: {} 2022-09-12T20:12:18.691 DEBUG:boto:Token: None 2022-09-12T20:12:18.692 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/ 2022-09-12T20:12:18.693 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o= 2022-09-12T20:12:18.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o=', 'Content-Length': '0'} 2022-09-12T20:12:18.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d2bfccc31158140-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.697 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:18.698 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:18.699 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:18.700 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:12:18.701 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:12:18.702 DEBUG:boto:Method: GET 2022-09-12T20:12:18.702 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:12:18.703 DEBUG:boto:Data: 2022-09-12T20:12:18.703 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.704 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:18.705 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.705 DEBUG:boto:Params: {} 2022-09-12T20:12:18.706 DEBUG:boto:Token: None 2022-09-12T20:12:18.707 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/ 2022-09-12T20:12:18.707 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o= 2022-09-12T20:12:18.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o=', 'Content-Length': '0'} 2022-09-12T20:12:18.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006610424bd2a17d41-00631f92a2-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.712 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:18.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:12:18.714 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:18.715 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:18.716 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:18.716 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:18.717 DEBUG:boto:Method: GET 2022-09-12T20:12:18.718 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:18.718 DEBUG:boto:Data: 2022-09-12T20:12:18.719 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.720 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.720 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.721 DEBUG:boto:Params: {} 2022-09-12T20:12:18.721 DEBUG:boto:Token: None 2022-09-12T20:12:18.722 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:18.723 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9vE2BMiLRwRfd0qWnoyXpBjlUpE= 2022-09-12T20:12:18.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9vE2BMiLRwRfd0qWnoyXpBjlUpE=', 'Content-Length': '0'} 2022-09-12T20:12:18.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d538a52c1ee8adea-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.726 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:18.727 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:18.728 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:18.728 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:18.729 DEBUG:boto:Method: GET 2022-09-12T20:12:18.729 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:18.730 DEBUG:boto:Data: 2022-09-12T20:12:18.730 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.731 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:18.731 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.732 DEBUG:boto:Params: {} 2022-09-12T20:12:18.733 DEBUG:boto:Token: None 2022-09-12T20:12:18.733 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:18.734 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9vE2BMiLRwRfd0qWnoyXpBjlUpE= 2022-09-12T20:12:18.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9vE2BMiLRwRfd0qWnoyXpBjlUpE=', 'Content-Length': '0'} 2022-09-12T20:12:18.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0f5a6557b22e9fb-00631f92a2-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.782 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:18.783 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:18.783 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:18.784 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:18.784 DEBUG:boto:Method: HEAD 2022-09-12T20:12:18.785 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:18.785 DEBUG:boto:Data: 2022-09-12T20:12:18.786 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.786 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.787 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.787 DEBUG:boto:Params: {} 2022-09-12T20:12:18.788 DEBUG:boto:Token: None 2022-09-12T20:12:18.789 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:18.789 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PU8f7oH0E03D4kSJFnCuSs//YfA= 2022-09-12T20:12:18.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PU8f7oH0E03D4kSJFnCuSs//YfA=', 'Content-Length': '0'} 2022-09-12T20:12:18.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abaab6bbfaddc39e-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.793 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:18.794 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:18.794 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:18.795 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:18.795 DEBUG:boto:Method: HEAD 2022-09-12T20:12:18.796 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:18.796 DEBUG:boto:Data: 2022-09-12T20:12:18.797 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.797 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:18.798 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.798 DEBUG:boto:Params: {} 2022-09-12T20:12:18.799 DEBUG:boto:Token: None 2022-09-12T20:12:18.799 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:18.800 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PU8f7oH0E03D4kSJFnCuSs//YfA= 2022-09-12T20:12:18.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PU8f7oH0E03D4kSJFnCuSs//YfA=', 'Content-Length': '0'} 2022-09-12T20:12:18.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040d0ce0515c96586-00631f92a2-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:12:18.804 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:18.805 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:18.805 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:18.806 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:18.806 DEBUG:boto:Method: GET 2022-09-12T20:12:18.807 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:18.807 DEBUG:boto:Data: 2022-09-12T20:12:18.808 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.808 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.809 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.809 DEBUG:boto:Params: {} 2022-09-12T20:12:18.816 DEBUG:boto:Token: None 2022-09-12T20:12:18.817 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:18.817 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9vE2BMiLRwRfd0qWnoyXpBjlUpE= 2022-09-12T20:12:18.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9vE2BMiLRwRfd0qWnoyXpBjlUpE=', 'Content-Length': '0'} 2022-09-12T20:12:18.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000719bf1e0a3db892b-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.821 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:18.822 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:18.822 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:18.823 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:12:18.823 DEBUG:boto:Method: GET 2022-09-12T20:12:18.824 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:18.824 DEBUG:boto:Data: 2022-09-12T20:12:18.825 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.825 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:18.826 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.827 DEBUG:boto:Params: {} 2022-09-12T20:12:18.827 DEBUG:boto:Token: None 2022-09-12T20:12:18.828 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:12:18.828 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9vE2BMiLRwRfd0qWnoyXpBjlUpE= 2022-09-12T20:12:18.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9vE2BMiLRwRfd0qWnoyXpBjlUpE=', 'Content-Length': '0'} 2022-09-12T20:12:18.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020938f51ecfec6c4-00631f92a2-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.832 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:18.833 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:18.833 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:12:18.834 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:12:18.835 DEBUG:boto:Method: GET 2022-09-12T20:12:18.836 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:12:18.836 DEBUG:boto:Data: 2022-09-12T20:12:18.837 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.837 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.838 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.838 DEBUG:boto:Params: {} 2022-09-12T20:12:18.839 DEBUG:boto:Token: None 2022-09-12T20:12:18.840 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/ 2022-09-12T20:12:18.840 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o= 2022-09-12T20:12:18.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o=', 'Content-Length': '0'} 2022-09-12T20:12:18.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000018a031d117d009c-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.844 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:18.845 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:18.846 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:18.846 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:12:18.847 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:12:18.847 DEBUG:boto:Method: GET 2022-09-12T20:12:18.848 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:12:18.848 DEBUG:boto:Data: 2022-09-12T20:12:18.849 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.850 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:18.850 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.851 DEBUG:boto:Params: {} 2022-09-12T20:12:18.851 DEBUG:boto:Token: None 2022-09-12T20:12:18.852 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/ 2022-09-12T20:12:18.852 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o= 2022-09-12T20:12:18.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o=', 'Content-Length': '0'} 2022-09-12T20:12:18.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000780108cf37d07dd1-00631f92a2-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.856 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:18.857 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:18.858 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:18.858 DEBUG:boto:Method: HEAD 2022-09-12T20:12:18.859 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:12:18.859 DEBUG:boto:Data: 2022-09-12T20:12:18.860 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.860 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.861 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.861 DEBUG:boto:Params: {} 2022-09-12T20:12:18.862 DEBUG:boto:Token: None 2022-09-12T20:12:18.862 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj116 2022-09-12T20:12:18.863 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ji3T6TTGWi7KmUymAjGwcIt7Lrg= 2022-09-12T20:12:18.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ji3T6TTGWi7KmUymAjGwcIt7Lrg=', 'Content-Length': '0'} 2022-09-12T20:12:18.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bf39adadca984af-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.867 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:18.867 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:18.868 DEBUG:boto:Method: HEAD 2022-09-12T20:12:18.868 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:12:18.869 DEBUG:boto:Data: 2022-09-12T20:12:18.869 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.870 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:18.870 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.871 DEBUG:boto:Params: {} 2022-09-12T20:12:18.871 DEBUG:boto:Token: None 2022-09-12T20:12:18.872 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj116 2022-09-12T20:12:18.872 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ji3T6TTGWi7KmUymAjGwcIt7Lrg= 2022-09-12T20:12:18.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ji3T6TTGWi7KmUymAjGwcIt7Lrg=', 'Content-Length': '0'} 2022-09-12T20:12:18.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a35e14775dad8428-00631f92a2-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:12:18.876 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:18.877 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:18.877 DEBUG:boto:Method: GET 2022-09-12T20:12:18.878 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:12:18.878 DEBUG:boto:Data: 2022-09-12T20:12:18.879 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.879 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.880 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.880 DEBUG:boto:Params: {} 2022-09-12T20:12:18.881 DEBUG:boto:Token: None 2022-09-12T20:12:18.882 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj116 2022-09-12T20:12:18.882 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:knMhcmpUYJWbVUBIHaJBYe5+p70= 2022-09-12T20:12:18.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:knMhcmpUYJWbVUBIHaJBYe5+p70=', 'Content-Length': '0'} 2022-09-12T20:12:18.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4aab5f334774b75-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.926 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:12:18.927 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:12:18.927 DEBUG:boto:Method: GET 2022-09-12T20:12:18.928 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:12:18.928 DEBUG:boto:Data: 2022-09-12T20:12:18.929 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.930 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:18.930 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.931 DEBUG:boto:Params: {} 2022-09-12T20:12:18.931 DEBUG:boto:Token: None 2022-09-12T20:12:18.932 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj116 2022-09-12T20:12:18.932 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:knMhcmpUYJWbVUBIHaJBYe5+p70= 2022-09-12T20:12:18.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:knMhcmpUYJWbVUBIHaJBYe5+p70=', 'Content-Length': '0'} 2022-09-12T20:12:18.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013bbcfed4fb0f90b-00631f92a2-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.936 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:18.937 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:18.937 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:12:18.938 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:12:18.938 DEBUG:boto:Method: GET 2022-09-12T20:12:18.939 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:12:18.939 DEBUG:boto:Data: 2022-09-12T20:12:18.940 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.940 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.941 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.941 DEBUG:boto:Params: {} 2022-09-12T20:12:18.942 DEBUG:boto:Token: None 2022-09-12T20:12:18.943 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/ 2022-09-12T20:12:18.943 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o= 2022-09-12T20:12:18.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o=', 'Content-Length': '0'} 2022-09-12T20:12:18.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fea81c19a30df00-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.947 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:18.948 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:18.949 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:18.949 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:12:18.950 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:12:18.950 DEBUG:boto:Method: GET 2022-09-12T20:12:18.951 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:12:18.951 DEBUG:boto:Data: 2022-09-12T20:12:18.952 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.952 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:18.953 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.953 DEBUG:boto:Params: {} 2022-09-12T20:12:18.954 DEBUG:boto:Token: None 2022-09-12T20:12:18.955 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/ 2022-09-12T20:12:18.955 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o= 2022-09-12T20:12:18.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k2XiHwQaSFxbrhy5HwCLK1e78/o=', 'Content-Length': '0'} 2022-09-12T20:12:18.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000900f7e84123a7d77-00631f92a2-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.959 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:18.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:12:18.960 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:18.961 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:18.962 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:18.962 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:18.963 DEBUG:boto:Method: GET 2022-09-12T20:12:18.963 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:18.964 DEBUG:boto:Data: 2022-09-12T20:12:18.964 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.965 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:18.965 DEBUG:boto:Port: 8001 2022-09-12T20:12:18.966 DEBUG:boto:Params: {} 2022-09-12T20:12:18.966 DEBUG:boto:Token: None 2022-09-12T20:12:18.967 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:18.968 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B9kOT/mOoCkLJHJsENHUbV0HadM= 2022-09-12T20:12:18.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B9kOT/mOoCkLJHJsENHUbV0HadM=', 'Content-Length': '0'} 2022-09-12T20:12:18.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000877465ef37d11a91-00631f92a2-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:18.971 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:18.972 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:18.972 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:18.973 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:18.973 DEBUG:boto:Method: GET 2022-09-12T20:12:18.974 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:18.975 DEBUG:boto:Data: 2022-09-12T20:12:18.975 DEBUG:boto:Headers: {} 2022-09-12T20:12:18.976 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:18.976 DEBUG:boto:Port: 8000 2022-09-12T20:12:18.977 DEBUG:boto:Params: {} 2022-09-12T20:12:18.977 DEBUG:boto:Token: None 2022-09-12T20:12:18.978 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:18 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:18.979 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B9kOT/mOoCkLJHJsENHUbV0HadM= 2022-09-12T20:12:18.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B9kOT/mOoCkLJHJsENHUbV0HadM=', 'Content-Length': '0'} 2022-09-12T20:12:18.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1fc7a01056c7aa2-00631f92a2-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.022 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:19.023 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:19.023 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:19.024 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:19.024 DEBUG:boto:Method: HEAD 2022-09-12T20:12:19.025 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:19.025 DEBUG:boto:Data: 2022-09-12T20:12:19.026 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.026 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.027 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.027 DEBUG:boto:Params: {} 2022-09-12T20:12:19.028 DEBUG:boto:Token: None 2022-09-12T20:12:19.028 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:19.029 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9+yLVmxprKOWUHBH3B8oLr7bf8Y= 2022-09-12T20:12:19.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9+yLVmxprKOWUHBH3B8oLr7bf8Y=', 'Content-Length': '0'} 2022-09-12T20:12:19.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8c858fd4f76f10a-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.033 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:19.033 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:19.034 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:19.034 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:19.035 DEBUG:boto:Method: HEAD 2022-09-12T20:12:19.035 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:19.036 DEBUG:boto:Data: 2022-09-12T20:12:19.036 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.037 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.037 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.038 DEBUG:boto:Params: {} 2022-09-12T20:12:19.039 DEBUG:boto:Token: None 2022-09-12T20:12:19.039 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:19.040 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9+yLVmxprKOWUHBH3B8oLr7bf8Y= 2022-09-12T20:12:19.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9+yLVmxprKOWUHBH3B8oLr7bf8Y=', 'Content-Length': '0'} 2022-09-12T20:12:19.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059d4c11b56524730-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:12:19.044 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:19.044 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:19.045 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:19.045 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:19.046 DEBUG:boto:Method: GET 2022-09-12T20:12:19.046 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:19.047 DEBUG:boto:Data: 2022-09-12T20:12:19.047 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.048 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.049 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.049 DEBUG:boto:Params: {} 2022-09-12T20:12:19.050 DEBUG:boto:Token: None 2022-09-12T20:12:19.050 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:19.051 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lozxeXVpaNQdFF3KVIcalxbx5sw= 2022-09-12T20:12:19.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lozxeXVpaNQdFF3KVIcalxbx5sw=', 'Content-Length': '0'} 2022-09-12T20:12:19.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5da36e21b2c3aa3-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.055 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:19.055 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:19.056 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:19.056 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:12:19.057 DEBUG:boto:Method: GET 2022-09-12T20:12:19.057 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:19.058 DEBUG:boto:Data: 2022-09-12T20:12:19.058 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.059 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.059 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.060 DEBUG:boto:Params: {} 2022-09-12T20:12:19.061 DEBUG:boto:Token: None 2022-09-12T20:12:19.061 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:12:19.062 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lozxeXVpaNQdFF3KVIcalxbx5sw= 2022-09-12T20:12:19.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lozxeXVpaNQdFF3KVIcalxbx5sw=', 'Content-Length': '0'} 2022-09-12T20:12:19.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c0062e92aba204f-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.065 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:19.066 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:19.066 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:12:19.067 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:12:19.068 DEBUG:boto:Method: GET 2022-09-12T20:12:19.068 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:12:19.069 DEBUG:boto:Data: 2022-09-12T20:12:19.069 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.070 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.070 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.071 DEBUG:boto:Params: {} 2022-09-12T20:12:19.071 DEBUG:boto:Token: None 2022-09-12T20:12:19.072 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/ 2022-09-12T20:12:19.073 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg= 2022-09-12T20:12:19.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg=', 'Content-Length': '0'} 2022-09-12T20:12:19.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8661e5e6f11dcbd-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.118 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:19.119 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:19.120 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:19.120 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:12:19.121 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:12:19.121 DEBUG:boto:Method: GET 2022-09-12T20:12:19.122 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:12:19.123 DEBUG:boto:Data: 2022-09-12T20:12:19.123 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.124 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.124 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.125 DEBUG:boto:Params: {} 2022-09-12T20:12:19.125 DEBUG:boto:Token: None 2022-09-12T20:12:19.126 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/ 2022-09-12T20:12:19.127 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg= 2022-09-12T20:12:19.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg=', 'Content-Length': '0'} 2022-09-12T20:12:19.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce74773446adb6ef-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.131 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:19.132 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:19.132 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:19.133 DEBUG:boto:Method: HEAD 2022-09-12T20:12:19.133 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:12:19.134 DEBUG:boto:Data: 2022-09-12T20:12:19.135 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.136 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.136 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.137 DEBUG:boto:Params: {} 2022-09-12T20:12:19.137 DEBUG:boto:Token: None 2022-09-12T20:12:19.138 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj145 2022-09-12T20:12:19.139 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BYVj8BNHkWoUOKkobnWVrBRUz3E= 2022-09-12T20:12:19.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BYVj8BNHkWoUOKkobnWVrBRUz3E=', 'Content-Length': '0'} 2022-09-12T20:12:19.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a15f54bc6e13590-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.142 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:19.143 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:19.143 DEBUG:boto:Method: HEAD 2022-09-12T20:12:19.144 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:12:19.144 DEBUG:boto:Data: 2022-09-12T20:12:19.145 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.145 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.146 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.146 DEBUG:boto:Params: {} 2022-09-12T20:12:19.147 DEBUG:boto:Token: None 2022-09-12T20:12:19.148 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj145 2022-09-12T20:12:19.148 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BYVj8BNHkWoUOKkobnWVrBRUz3E= 2022-09-12T20:12:19.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BYVj8BNHkWoUOKkobnWVrBRUz3E=', 'Content-Length': '0'} 2022-09-12T20:12:19.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010e09b245050d903-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:12:19.152 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:19.153 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:19.154 DEBUG:boto:Method: GET 2022-09-12T20:12:19.154 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:12:19.155 DEBUG:boto:Data: 2022-09-12T20:12:19.155 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.156 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.156 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.157 DEBUG:boto:Params: {} 2022-09-12T20:12:19.158 DEBUG:boto:Token: None 2022-09-12T20:12:19.158 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj145 2022-09-12T20:12:19.159 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p7hgltiHk8atFInCAbJtGkSf3SI= 2022-09-12T20:12:19.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p7hgltiHk8atFInCAbJtGkSf3SI=', 'Content-Length': '0'} 2022-09-12T20:12:19.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cefff616d0f787c-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.162 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:12:19.163 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:12:19.163 DEBUG:boto:Method: GET 2022-09-12T20:12:19.164 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:12:19.164 DEBUG:boto:Data: 2022-09-12T20:12:19.165 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.166 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.166 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.167 DEBUG:boto:Params: {} 2022-09-12T20:12:19.167 DEBUG:boto:Token: None 2022-09-12T20:12:19.168 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj145 2022-09-12T20:12:19.169 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p7hgltiHk8atFInCAbJtGkSf3SI= 2022-09-12T20:12:19.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p7hgltiHk8atFInCAbJtGkSf3SI=', 'Content-Length': '0'} 2022-09-12T20:12:19.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c49cc08e8bdbeeda-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.214 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:19.215 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:19.215 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:12:19.216 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:12:19.216 DEBUG:boto:Method: GET 2022-09-12T20:12:19.217 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:12:19.218 DEBUG:boto:Data: 2022-09-12T20:12:19.218 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.219 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.219 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.220 DEBUG:boto:Params: {} 2022-09-12T20:12:19.220 DEBUG:boto:Token: None 2022-09-12T20:12:19.221 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/ 2022-09-12T20:12:19.222 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg= 2022-09-12T20:12:19.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg=', 'Content-Length': '0'} 2022-09-12T20:12:19.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b70a4ae0ad4d90fb-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.226 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:19.227 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:19.227 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:19.228 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:12:19.228 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:12:19.229 DEBUG:boto:Method: GET 2022-09-12T20:12:19.230 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:12:19.230 DEBUG:boto:Data: 2022-09-12T20:12:19.231 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.231 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.232 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.232 DEBUG:boto:Params: {} 2022-09-12T20:12:19.233 DEBUG:boto:Token: None 2022-09-12T20:12:19.233 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/ 2022-09-12T20:12:19.234 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg= 2022-09-12T20:12:19.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg=', 'Content-Length': '0'} 2022-09-12T20:12:19.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035cba94b44814ccd-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.238 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:19.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:12:19.240 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:19.240 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:19.241 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:19.241 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:19.242 DEBUG:boto:Method: GET 2022-09-12T20:12:19.242 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:19.243 DEBUG:boto:Data: 2022-09-12T20:12:19.244 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.244 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.245 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.245 DEBUG:boto:Params: {} 2022-09-12T20:12:19.246 DEBUG:boto:Token: None 2022-09-12T20:12:19.246 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:19.247 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RPHHHWDD6Q5pofJ0D+hKOAHZVYk= 2022-09-12T20:12:19.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RPHHHWDD6Q5pofJ0D+hKOAHZVYk=', 'Content-Length': '0'} 2022-09-12T20:12:19.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003abf47455cc3ea25-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.251 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:19.251 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:19.252 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:19.252 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:19.253 DEBUG:boto:Method: GET 2022-09-12T20:12:19.253 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:19.254 DEBUG:boto:Data: 2022-09-12T20:12:19.255 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.255 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.256 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.256 DEBUG:boto:Params: {} 2022-09-12T20:12:19.257 DEBUG:boto:Token: None 2022-09-12T20:12:19.257 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:19.258 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RPHHHWDD6Q5pofJ0D+hKOAHZVYk= 2022-09-12T20:12:19.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RPHHHWDD6Q5pofJ0D+hKOAHZVYk=', 'Content-Length': '0'} 2022-09-12T20:12:19.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002368d5e861c434ad-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.262 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:19.262 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:19.263 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:19.263 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:19.264 DEBUG:boto:Method: HEAD 2022-09-12T20:12:19.264 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:19.265 DEBUG:boto:Data: 2022-09-12T20:12:19.265 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.266 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.266 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.267 DEBUG:boto:Params: {} 2022-09-12T20:12:19.268 DEBUG:boto:Token: None 2022-09-12T20:12:19.268 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:19.269 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Z9v7kKn/dxUr4T+lol1BtNskJe0= 2022-09-12T20:12:19.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Z9v7kKn/dxUr4T+lol1BtNskJe0=', 'Content-Length': '0'} 2022-09-12T20:12:19.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000144613cc3863b50f-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.272 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:19.273 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:19.273 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:19.274 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:19.274 DEBUG:boto:Method: HEAD 2022-09-12T20:12:19.275 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:19.275 DEBUG:boto:Data: 2022-09-12T20:12:19.276 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.276 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.277 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.277 DEBUG:boto:Params: {} 2022-09-12T20:12:19.278 DEBUG:boto:Token: None 2022-09-12T20:12:19.278 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:19.279 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Z9v7kKn/dxUr4T+lol1BtNskJe0= 2022-09-12T20:12:19.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Z9v7kKn/dxUr4T+lol1BtNskJe0=', 'Content-Length': '0'} 2022-09-12T20:12:19.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000717ea00b9a81e27b-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:12:19.283 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:19.284 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:19.284 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:19.285 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:19.285 DEBUG:boto:Method: GET 2022-09-12T20:12:19.286 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:19.286 DEBUG:boto:Data: 2022-09-12T20:12:19.287 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.287 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.288 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.289 DEBUG:boto:Params: {} 2022-09-12T20:12:19.289 DEBUG:boto:Token: None 2022-09-12T20:12:19.290 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:19.290 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RPHHHWDD6Q5pofJ0D+hKOAHZVYk= 2022-09-12T20:12:19.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RPHHHWDD6Q5pofJ0D+hKOAHZVYk=', 'Content-Length': '0'} 2022-09-12T20:12:19.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015842940595045dd-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.294 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:19.294 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:19.295 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:19.295 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:12:19.296 DEBUG:boto:Method: GET 2022-09-12T20:12:19.297 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:19.297 DEBUG:boto:Data: 2022-09-12T20:12:19.298 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.298 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.299 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.299 DEBUG:boto:Params: {} 2022-09-12T20:12:19.300 DEBUG:boto:Token: None 2022-09-12T20:12:19.300 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:12:19.301 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RPHHHWDD6Q5pofJ0D+hKOAHZVYk= 2022-09-12T20:12:19.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RPHHHWDD6Q5pofJ0D+hKOAHZVYk=', 'Content-Length': '0'} 2022-09-12T20:12:19.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fda21c73458f79e-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.346 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:19.347 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:19.347 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:12:19.348 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:12:19.348 DEBUG:boto:Method: GET 2022-09-12T20:12:19.349 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:12:19.350 DEBUG:boto:Data: 2022-09-12T20:12:19.350 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.351 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.351 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.352 DEBUG:boto:Params: {} 2022-09-12T20:12:19.352 DEBUG:boto:Token: None 2022-09-12T20:12:19.353 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/ 2022-09-12T20:12:19.354 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg= 2022-09-12T20:12:19.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg=', 'Content-Length': '0'} 2022-09-12T20:12:19.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0a77aed24548814-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.358 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:19.359 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:19.359 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:19.360 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:12:19.361 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:12:19.361 DEBUG:boto:Method: GET 2022-09-12T20:12:19.362 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:12:19.362 DEBUG:boto:Data: 2022-09-12T20:12:19.363 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.363 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.364 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.364 DEBUG:boto:Params: {} 2022-09-12T20:12:19.365 DEBUG:boto:Token: None 2022-09-12T20:12:19.365 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/ 2022-09-12T20:12:19.366 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg= 2022-09-12T20:12:19.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg=', 'Content-Length': '0'} 2022-09-12T20:12:19.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b4fd63c3fb6473f-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.370 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:19.371 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:19.372 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:19.372 DEBUG:boto:Method: HEAD 2022-09-12T20:12:19.373 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:12:19.373 DEBUG:boto:Data: 2022-09-12T20:12:19.374 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.374 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.375 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.375 DEBUG:boto:Params: {} 2022-09-12T20:12:19.376 DEBUG:boto:Token: None 2022-09-12T20:12:19.376 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj17 2022-09-12T20:12:19.377 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RdZxQ9eqNlipOjtUxNtH+GLMbUw= 2022-09-12T20:12:19.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RdZxQ9eqNlipOjtUxNtH+GLMbUw=', 'Content-Length': '0'} 2022-09-12T20:12:19.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d2a7c1cabcb16f3-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.380 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:19.381 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:19.381 DEBUG:boto:Method: HEAD 2022-09-12T20:12:19.382 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:12:19.382 DEBUG:boto:Data: 2022-09-12T20:12:19.383 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.383 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.384 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.384 DEBUG:boto:Params: {} 2022-09-12T20:12:19.385 DEBUG:boto:Token: None 2022-09-12T20:12:19.385 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj17 2022-09-12T20:12:19.386 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RdZxQ9eqNlipOjtUxNtH+GLMbUw= 2022-09-12T20:12:19.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RdZxQ9eqNlipOjtUxNtH+GLMbUw=', 'Content-Length': '0'} 2022-09-12T20:12:19.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c71a2d03f2ad521-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:12:19.390 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:19.390 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:19.391 DEBUG:boto:Method: GET 2022-09-12T20:12:19.391 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:12:19.392 DEBUG:boto:Data: 2022-09-12T20:12:19.392 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.393 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.393 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.394 DEBUG:boto:Params: {} 2022-09-12T20:12:19.395 DEBUG:boto:Token: None 2022-09-12T20:12:19.395 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj17 2022-09-12T20:12:19.396 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OvDQ4yHdMyMS9ItUvN2cn7M6Eno= 2022-09-12T20:12:19.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OvDQ4yHdMyMS9ItUvN2cn7M6Eno=', 'Content-Length': '0'} 2022-09-12T20:12:19.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d18f2dd9d7dd3675-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.399 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:12:19.400 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:12:19.400 DEBUG:boto:Method: GET 2022-09-12T20:12:19.401 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:12:19.401 DEBUG:boto:Data: 2022-09-12T20:12:19.402 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.402 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.403 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.404 DEBUG:boto:Params: {} 2022-09-12T20:12:19.404 DEBUG:boto:Token: None 2022-09-12T20:12:19.405 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj17 2022-09-12T20:12:19.405 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OvDQ4yHdMyMS9ItUvN2cn7M6Eno= 2022-09-12T20:12:19.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OvDQ4yHdMyMS9ItUvN2cn7M6Eno=', 'Content-Length': '0'} 2022-09-12T20:12:19.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbc888006593660b-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.409 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:19.410 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:19.410 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:12:19.411 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:12:19.411 DEBUG:boto:Method: GET 2022-09-12T20:12:19.412 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:12:19.412 DEBUG:boto:Data: 2022-09-12T20:12:19.413 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.413 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.414 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.414 DEBUG:boto:Params: {} 2022-09-12T20:12:19.415 DEBUG:boto:Token: None 2022-09-12T20:12:19.416 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/ 2022-09-12T20:12:19.416 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg= 2022-09-12T20:12:19.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg=', 'Content-Length': '0'} 2022-09-12T20:12:19.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b8c3a2e1850b5fa-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.462 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:19.463 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:19.464 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:19.464 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:12:19.465 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:12:19.465 DEBUG:boto:Method: GET 2022-09-12T20:12:19.466 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:12:19.466 DEBUG:boto:Data: 2022-09-12T20:12:19.467 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.467 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.468 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.469 DEBUG:boto:Params: {} 2022-09-12T20:12:19.469 DEBUG:boto:Token: None 2022-09-12T20:12:19.470 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/ 2022-09-12T20:12:19.470 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg= 2022-09-12T20:12:19.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg=', 'Content-Length': '0'} 2022-09-12T20:12:19.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003dace9d14fa55fc-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.474 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:19.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:12:19.476 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:19.476 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:19.477 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:19.477 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:19.478 DEBUG:boto:Method: GET 2022-09-12T20:12:19.479 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:19.479 DEBUG:boto:Data: 2022-09-12T20:12:19.480 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.480 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.481 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.481 DEBUG:boto:Params: {} 2022-09-12T20:12:19.482 DEBUG:boto:Token: None 2022-09-12T20:12:19.482 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:19.483 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Sj/AEDa9/5wSnOQ9KT8IPnjYxa4= 2022-09-12T20:12:19.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Sj/AEDa9/5wSnOQ9KT8IPnjYxa4=', 'Content-Length': '0'} 2022-09-12T20:12:19.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004bd9857e3c3164d-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.486 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:19.487 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:19.488 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:19.488 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:19.489 DEBUG:boto:Method: GET 2022-09-12T20:12:19.489 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:19.490 DEBUG:boto:Data: 2022-09-12T20:12:19.490 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.491 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.491 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.492 DEBUG:boto:Params: {} 2022-09-12T20:12:19.492 DEBUG:boto:Token: None 2022-09-12T20:12:19.493 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:19.494 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Sj/AEDa9/5wSnOQ9KT8IPnjYxa4= 2022-09-12T20:12:19.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Sj/AEDa9/5wSnOQ9KT8IPnjYxa4=', 'Content-Length': '0'} 2022-09-12T20:12:19.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af75aa516e2db139-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.497 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:19.498 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:19.498 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:19.499 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:19.499 DEBUG:boto:Method: HEAD 2022-09-12T20:12:19.500 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:19.500 DEBUG:boto:Data: 2022-09-12T20:12:19.501 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.501 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.502 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.502 DEBUG:boto:Params: {} 2022-09-12T20:12:19.503 DEBUG:boto:Token: None 2022-09-12T20:12:19.504 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:19.504 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dyhq6MS1jwSP+S4ZgybEhVZTIzo= 2022-09-12T20:12:19.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dyhq6MS1jwSP+S4ZgybEhVZTIzo=', 'Content-Length': '0'} 2022-09-12T20:12:19.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa11887421e7892a-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.507 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:19.508 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:19.508 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:19.509 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:19.509 DEBUG:boto:Method: HEAD 2022-09-12T20:12:19.510 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:19.510 DEBUG:boto:Data: 2022-09-12T20:12:19.511 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.511 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.512 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.512 DEBUG:boto:Params: {} 2022-09-12T20:12:19.513 DEBUG:boto:Token: None 2022-09-12T20:12:19.514 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:19.514 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dyhq6MS1jwSP+S4ZgybEhVZTIzo= 2022-09-12T20:12:19.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dyhq6MS1jwSP+S4ZgybEhVZTIzo=', 'Content-Length': '0'} 2022-09-12T20:12:19.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000feb74ff2eeea4ca6-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:12:19.518 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:19.518 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:19.519 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:19.519 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:19.520 DEBUG:boto:Method: GET 2022-09-12T20:12:19.520 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:19.521 DEBUG:boto:Data: 2022-09-12T20:12:19.522 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.522 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.523 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.523 DEBUG:boto:Params: {} 2022-09-12T20:12:19.524 DEBUG:boto:Token: None 2022-09-12T20:12:19.524 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:19.525 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Sj/AEDa9/5wSnOQ9KT8IPnjYxa4= 2022-09-12T20:12:19.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Sj/AEDa9/5wSnOQ9KT8IPnjYxa4=', 'Content-Length': '0'} 2022-09-12T20:12:19.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091a697bb59cf2fa4-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.570 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:19.571 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:19.571 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:19.572 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:12:19.573 DEBUG:boto:Method: GET 2022-09-12T20:12:19.573 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:19.574 DEBUG:boto:Data: 2022-09-12T20:12:19.574 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.575 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.575 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.576 DEBUG:boto:Params: {} 2022-09-12T20:12:19.576 DEBUG:boto:Token: None 2022-09-12T20:12:19.577 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:12:19.578 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Sj/AEDa9/5wSnOQ9KT8IPnjYxa4= 2022-09-12T20:12:19.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Sj/AEDa9/5wSnOQ9KT8IPnjYxa4=', 'Content-Length': '0'} 2022-09-12T20:12:19.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000856fffd4e24c0556-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.583 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:19.584 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:19.584 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:12:19.585 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:12:19.585 DEBUG:boto:Method: GET 2022-09-12T20:12:19.586 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:12:19.586 DEBUG:boto:Data: 2022-09-12T20:12:19.587 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.587 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.588 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.588 DEBUG:boto:Params: {} 2022-09-12T20:12:19.589 DEBUG:boto:Token: None 2022-09-12T20:12:19.589 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/ 2022-09-12T20:12:19.590 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg= 2022-09-12T20:12:19.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg=', 'Content-Length': '0'} 2022-09-12T20:12:19.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fd27f2ca9ed8ec5-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.594 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:19.595 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:19.596 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:19.596 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:12:19.597 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:12:19.597 DEBUG:boto:Method: GET 2022-09-12T20:12:19.598 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:12:19.598 DEBUG:boto:Data: 2022-09-12T20:12:19.599 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.599 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.600 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.601 DEBUG:boto:Params: {} 2022-09-12T20:12:19.601 DEBUG:boto:Token: None 2022-09-12T20:12:19.602 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/ 2022-09-12T20:12:19.602 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg= 2022-09-12T20:12:19.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg=', 'Content-Length': '0'} 2022-09-12T20:12:19.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005c111d2f5ee678b-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.606 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:19.607 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:19.608 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:19.608 DEBUG:boto:Method: HEAD 2022-09-12T20:12:19.609 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:12:19.610 DEBUG:boto:Data: 2022-09-12T20:12:19.610 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.611 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.611 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.612 DEBUG:boto:Params: {} 2022-09-12T20:12:19.612 DEBUG:boto:Token: None 2022-09-12T20:12:19.613 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj19 2022-09-12T20:12:19.614 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ggp7dbXN+dKTZxK4L6PTIa+PCPY= 2022-09-12T20:12:19.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ggp7dbXN+dKTZxK4L6PTIa+PCPY=', 'Content-Length': '0'} 2022-09-12T20:12:19.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050dccdc15d1ed7f9-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.617 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:19.618 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:19.618 DEBUG:boto:Method: HEAD 2022-09-12T20:12:19.619 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:12:19.619 DEBUG:boto:Data: 2022-09-12T20:12:19.620 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.620 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.621 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.621 DEBUG:boto:Params: {} 2022-09-12T20:12:19.622 DEBUG:boto:Token: None 2022-09-12T20:12:19.622 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj19 2022-09-12T20:12:19.623 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ggp7dbXN+dKTZxK4L6PTIa+PCPY= 2022-09-12T20:12:19.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ggp7dbXN+dKTZxK4L6PTIa+PCPY=', 'Content-Length': '0'} 2022-09-12T20:12:19.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000297a42b58d7b9bc-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:12:19.627 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:19.627 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:19.628 DEBUG:boto:Method: GET 2022-09-12T20:12:19.628 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:12:19.629 DEBUG:boto:Data: 2022-09-12T20:12:19.629 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.630 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.630 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.631 DEBUG:boto:Params: {} 2022-09-12T20:12:19.631 DEBUG:boto:Token: None 2022-09-12T20:12:19.632 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj19 2022-09-12T20:12:19.632 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uhqhzCVzZgL/osWhgG+TfvbN+/0= 2022-09-12T20:12:19.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uhqhzCVzZgL/osWhgG+TfvbN+/0=', 'Content-Length': '0'} 2022-09-12T20:12:19.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f31289623face970-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.636 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:12:19.637 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:12:19.637 DEBUG:boto:Method: GET 2022-09-12T20:12:19.638 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:12:19.639 DEBUG:boto:Data: 2022-09-12T20:12:19.639 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.640 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.640 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.641 DEBUG:boto:Params: {} 2022-09-12T20:12:19.641 DEBUG:boto:Token: None 2022-09-12T20:12:19.642 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj19 2022-09-12T20:12:19.642 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uhqhzCVzZgL/osWhgG+TfvbN+/0= 2022-09-12T20:12:19.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uhqhzCVzZgL/osWhgG+TfvbN+/0=', 'Content-Length': '0'} 2022-09-12T20:12:19.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086f5c983c92ce07c-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.686 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:19.687 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:19.687 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:12:19.688 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:12:19.689 DEBUG:boto:Method: GET 2022-09-12T20:12:19.689 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:12:19.690 DEBUG:boto:Data: 2022-09-12T20:12:19.690 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.691 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.691 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.692 DEBUG:boto:Params: {} 2022-09-12T20:12:19.692 DEBUG:boto:Token: None 2022-09-12T20:12:19.693 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/ 2022-09-12T20:12:19.694 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg= 2022-09-12T20:12:19.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg=', 'Content-Length': '0'} 2022-09-12T20:12:19.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b304146d55a4b4f7-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.698 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:19.699 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:19.699 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:19.700 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:12:19.700 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:12:19.701 DEBUG:boto:Method: GET 2022-09-12T20:12:19.701 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:12:19.702 DEBUG:boto:Data: 2022-09-12T20:12:19.702 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.703 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.703 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.704 DEBUG:boto:Params: {} 2022-09-12T20:12:19.705 DEBUG:boto:Token: None 2022-09-12T20:12:19.705 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/ 2022-09-12T20:12:19.706 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg= 2022-09-12T20:12:19.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg=', 'Content-Length': '0'} 2022-09-12T20:12:19.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005bccf35469c4bda-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.710 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:19.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:12:19.711 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:19.712 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:19.712 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:19.713 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:19.713 DEBUG:boto:Method: GET 2022-09-12T20:12:19.714 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:19.714 DEBUG:boto:Data: 2022-09-12T20:12:19.715 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.715 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.716 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.716 DEBUG:boto:Params: {} 2022-09-12T20:12:19.717 DEBUG:boto:Token: None 2022-09-12T20:12:19.717 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:19.718 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m7UyCPUXtp/QfshPXw5whVLdZwU= 2022-09-12T20:12:19.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m7UyCPUXtp/QfshPXw5whVLdZwU=', 'Content-Length': '0'} 2022-09-12T20:12:19.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000141f80466fbf7e2b-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.722 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:19.723 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:19.723 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:19.724 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:19.724 DEBUG:boto:Method: GET 2022-09-12T20:12:19.725 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:19.725 DEBUG:boto:Data: 2022-09-12T20:12:19.726 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.726 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.727 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.727 DEBUG:boto:Params: {} 2022-09-12T20:12:19.728 DEBUG:boto:Token: None 2022-09-12T20:12:19.728 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:19.729 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m7UyCPUXtp/QfshPXw5whVLdZwU= 2022-09-12T20:12:19.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m7UyCPUXtp/QfshPXw5whVLdZwU=', 'Content-Length': '0'} 2022-09-12T20:12:19.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080317a5a5febbb0c-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.733 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:19.733 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:19.734 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:19.734 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:19.735 DEBUG:boto:Method: HEAD 2022-09-12T20:12:19.735 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:19.736 DEBUG:boto:Data: 2022-09-12T20:12:19.736 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.737 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.737 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.738 DEBUG:boto:Params: {} 2022-09-12T20:12:19.738 DEBUG:boto:Token: None 2022-09-12T20:12:19.739 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:19.740 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vO58Xy7E90HdB//2qGoI9fUhCNo= 2022-09-12T20:12:19.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vO58Xy7E90HdB//2qGoI9fUhCNo=', 'Content-Length': '0'} 2022-09-12T20:12:19.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a6ef986cdc64dea-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.743 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:19.744 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:19.744 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:19.745 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:19.745 DEBUG:boto:Method: HEAD 2022-09-12T20:12:19.746 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:19.746 DEBUG:boto:Data: 2022-09-12T20:12:19.747 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.747 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.748 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.748 DEBUG:boto:Params: {} 2022-09-12T20:12:19.749 DEBUG:boto:Token: None 2022-09-12T20:12:19.749 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:19.750 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vO58Xy7E90HdB//2qGoI9fUhCNo= 2022-09-12T20:12:19.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vO58Xy7E90HdB//2qGoI9fUhCNo=', 'Content-Length': '0'} 2022-09-12T20:12:19.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007366599bbb7279b5-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:12:19.754 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:19.754 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:19.755 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:19.755 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:19.756 DEBUG:boto:Method: GET 2022-09-12T20:12:19.756 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:19.757 DEBUG:boto:Data: 2022-09-12T20:12:19.757 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.758 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.758 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.759 DEBUG:boto:Params: {} 2022-09-12T20:12:19.760 DEBUG:boto:Token: None 2022-09-12T20:12:19.760 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:19.761 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m7UyCPUXtp/QfshPXw5whVLdZwU= 2022-09-12T20:12:19.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m7UyCPUXtp/QfshPXw5whVLdZwU=', 'Content-Length': '0'} 2022-09-12T20:12:19.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000003519853b68b14e-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.764 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:19.765 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:19.766 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:19.766 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:12:19.767 DEBUG:boto:Method: GET 2022-09-12T20:12:19.767 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:19.768 DEBUG:boto:Data: 2022-09-12T20:12:19.768 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.769 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.769 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.770 DEBUG:boto:Params: {} 2022-09-12T20:12:19.770 DEBUG:boto:Token: None 2022-09-12T20:12:19.771 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:12:19.771 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m7UyCPUXtp/QfshPXw5whVLdZwU= 2022-09-12T20:12:19.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m7UyCPUXtp/QfshPXw5whVLdZwU=', 'Content-Length': '0'} 2022-09-12T20:12:19.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8f7b3ded8a83871-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.775 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:19.775 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:19.776 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:12:19.776 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:12:19.777 DEBUG:boto:Method: GET 2022-09-12T20:12:19.777 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:12:19.778 DEBUG:boto:Data: 2022-09-12T20:12:19.779 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.779 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.780 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.780 DEBUG:boto:Params: {} 2022-09-12T20:12:19.781 DEBUG:boto:Token: None 2022-09-12T20:12:19.781 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/ 2022-09-12T20:12:19.782 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg= 2022-09-12T20:12:19.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg=', 'Content-Length': '0'} 2022-09-12T20:12:19.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088f07ce92571ae5b-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.786 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:19.787 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:19.787 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:19.788 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:12:19.788 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:12:19.789 DEBUG:boto:Method: GET 2022-09-12T20:12:19.790 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:12:19.790 DEBUG:boto:Data: 2022-09-12T20:12:19.791 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.791 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.792 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.792 DEBUG:boto:Params: {} 2022-09-12T20:12:19.793 DEBUG:boto:Token: None 2022-09-12T20:12:19.793 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/ 2022-09-12T20:12:19.794 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg= 2022-09-12T20:12:19.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg=', 'Content-Length': '0'} 2022-09-12T20:12:19.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c102e4ee45ff4d43-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.798 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:19.799 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:19.799 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:19.800 DEBUG:boto:Method: HEAD 2022-09-12T20:12:19.800 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:12:19.801 DEBUG:boto:Data: 2022-09-12T20:12:19.801 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.802 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.802 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.803 DEBUG:boto:Params: {} 2022-09-12T20:12:19.803 DEBUG:boto:Token: None 2022-09-12T20:12:19.804 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj22 2022-09-12T20:12:19.805 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WHazf1bV0FmoZYP+UDRAb4CkPaU= 2022-09-12T20:12:19.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WHazf1bV0FmoZYP+UDRAb4CkPaU=', 'Content-Length': '0'} 2022-09-12T20:12:19.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a657ca4001e1ee50-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.808 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:19.808 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:19.809 DEBUG:boto:Method: HEAD 2022-09-12T20:12:19.809 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:12:19.810 DEBUG:boto:Data: 2022-09-12T20:12:19.810 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.811 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.811 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.812 DEBUG:boto:Params: {} 2022-09-12T20:12:19.812 DEBUG:boto:Token: None 2022-09-12T20:12:19.813 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj22 2022-09-12T20:12:19.813 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WHazf1bV0FmoZYP+UDRAb4CkPaU= 2022-09-12T20:12:19.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WHazf1bV0FmoZYP+UDRAb4CkPaU=', 'Content-Length': '0'} 2022-09-12T20:12:19.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e4d2c8860196bbf-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:12:19.817 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:19.818 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:19.818 DEBUG:boto:Method: GET 2022-09-12T20:12:19.819 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:12:19.819 DEBUG:boto:Data: 2022-09-12T20:12:19.820 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.820 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.821 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.822 DEBUG:boto:Params: {} 2022-09-12T20:12:19.822 DEBUG:boto:Token: None 2022-09-12T20:12:19.823 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj22 2022-09-12T20:12:19.823 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1ESq9KO74U/840cxaIHiwAq35wM= 2022-09-12T20:12:19.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1ESq9KO74U/840cxaIHiwAq35wM=', 'Content-Length': '0'} 2022-09-12T20:12:19.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b6753878b9d3be5-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.827 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:12:19.827 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:12:19.828 DEBUG:boto:Method: GET 2022-09-12T20:12:19.828 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:12:19.829 DEBUG:boto:Data: 2022-09-12T20:12:19.830 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.830 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.831 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.831 DEBUG:boto:Params: {} 2022-09-12T20:12:19.832 DEBUG:boto:Token: None 2022-09-12T20:12:19.832 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj22 2022-09-12T20:12:19.833 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1ESq9KO74U/840cxaIHiwAq35wM= 2022-09-12T20:12:19.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1ESq9KO74U/840cxaIHiwAq35wM=', 'Content-Length': '0'} 2022-09-12T20:12:19.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c9b757a720f5c8e-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.836 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:19.837 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:19.837 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:12:19.838 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:12:19.839 DEBUG:boto:Method: GET 2022-09-12T20:12:19.839 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:12:19.840 DEBUG:boto:Data: 2022-09-12T20:12:19.840 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.841 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.841 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.842 DEBUG:boto:Params: {} 2022-09-12T20:12:19.842 DEBUG:boto:Token: None 2022-09-12T20:12:19.843 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/ 2022-09-12T20:12:19.844 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg= 2022-09-12T20:12:19.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg=', 'Content-Length': '0'} 2022-09-12T20:12:19.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1d93ff9f48bddd2-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.890 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:19.891 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:19.891 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:19.892 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:12:19.892 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:12:19.893 DEBUG:boto:Method: GET 2022-09-12T20:12:19.893 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:12:19.894 DEBUG:boto:Data: 2022-09-12T20:12:19.895 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.895 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.896 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.896 DEBUG:boto:Params: {} 2022-09-12T20:12:19.897 DEBUG:boto:Token: None 2022-09-12T20:12:19.897 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/ 2022-09-12T20:12:19.898 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg= 2022-09-12T20:12:19.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg=', 'Content-Length': '0'} 2022-09-12T20:12:19.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000808e1558de00c47c-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.902 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:19.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:12:19.904 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:19.904 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:19.905 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:19.905 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:19.906 DEBUG:boto:Method: GET 2022-09-12T20:12:19.906 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:19.907 DEBUG:boto:Data: 2022-09-12T20:12:19.907 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.908 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.909 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.909 DEBUG:boto:Params: {} 2022-09-12T20:12:19.910 DEBUG:boto:Token: None 2022-09-12T20:12:19.910 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:19.911 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sssdwj9yacuIKc/mrr1DIQXGXDI= 2022-09-12T20:12:19.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sssdwj9yacuIKc/mrr1DIQXGXDI=', 'Content-Length': '0'} 2022-09-12T20:12:19.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d96df5ba1e9b921-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.914 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:19.915 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:19.916 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:19.916 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:19.917 DEBUG:boto:Method: GET 2022-09-12T20:12:19.917 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:19.918 DEBUG:boto:Data: 2022-09-12T20:12:19.918 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.919 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.919 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.920 DEBUG:boto:Params: {} 2022-09-12T20:12:19.920 DEBUG:boto:Token: None 2022-09-12T20:12:19.921 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:19.922 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sssdwj9yacuIKc/mrr1DIQXGXDI= 2022-09-12T20:12:19.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sssdwj9yacuIKc/mrr1DIQXGXDI=', 'Content-Length': '0'} 2022-09-12T20:12:19.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ad1dd33e4dd5341-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.925 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:19.926 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:19.927 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:19.927 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:19.928 DEBUG:boto:Method: HEAD 2022-09-12T20:12:19.928 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:19.929 DEBUG:boto:Data: 2022-09-12T20:12:19.929 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.930 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.930 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.931 DEBUG:boto:Params: {} 2022-09-12T20:12:19.931 DEBUG:boto:Token: None 2022-09-12T20:12:19.932 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:19.932 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q0SXtx7MQhjCmF5YbgiLBb1XTNg= 2022-09-12T20:12:19.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q0SXtx7MQhjCmF5YbgiLBb1XTNg=', 'Content-Length': '0'} 2022-09-12T20:12:19.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083249a8723aa1215-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.935 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:19.936 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:19.936 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:19.937 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:19.938 DEBUG:boto:Method: HEAD 2022-09-12T20:12:19.938 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:19.939 DEBUG:boto:Data: 2022-09-12T20:12:19.939 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.940 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.940 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.941 DEBUG:boto:Params: {} 2022-09-12T20:12:19.941 DEBUG:boto:Token: None 2022-09-12T20:12:19.942 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:19.942 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q0SXtx7MQhjCmF5YbgiLBb1XTNg= 2022-09-12T20:12:19.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q0SXtx7MQhjCmF5YbgiLBb1XTNg=', 'Content-Length': '0'} 2022-09-12T20:12:19.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfa54a913af37873-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:12:19.946 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:19.947 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:19.948 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:19.948 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:19.949 DEBUG:boto:Method: GET 2022-09-12T20:12:19.949 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:19.950 DEBUG:boto:Data: 2022-09-12T20:12:19.950 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.951 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.951 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.952 DEBUG:boto:Params: {} 2022-09-12T20:12:19.952 DEBUG:boto:Token: None 2022-09-12T20:12:19.953 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:19.954 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sssdwj9yacuIKc/mrr1DIQXGXDI= 2022-09-12T20:12:19.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sssdwj9yacuIKc/mrr1DIQXGXDI=', 'Content-Length': '0'} 2022-09-12T20:12:19.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014aae813ee29a8a1-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.957 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:19.958 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:19.958 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:19.959 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:12:19.959 DEBUG:boto:Method: GET 2022-09-12T20:12:19.960 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:19.960 DEBUG:boto:Data: 2022-09-12T20:12:19.961 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.962 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.962 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.963 DEBUG:boto:Params: {} 2022-09-12T20:12:19.963 DEBUG:boto:Token: None 2022-09-12T20:12:19.964 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:12:19.964 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sssdwj9yacuIKc/mrr1DIQXGXDI= 2022-09-12T20:12:19.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sssdwj9yacuIKc/mrr1DIQXGXDI=', 'Content-Length': '0'} 2022-09-12T20:12:19.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d821acec353c700c-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.968 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:19.969 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:19.969 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:12:19.970 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:12:19.970 DEBUG:boto:Method: GET 2022-09-12T20:12:19.971 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:12:19.971 DEBUG:boto:Data: 2022-09-12T20:12:19.972 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.972 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.973 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.973 DEBUG:boto:Params: {} 2022-09-12T20:12:19.974 DEBUG:boto:Token: None 2022-09-12T20:12:19.975 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/ 2022-09-12T20:12:19.975 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg= 2022-09-12T20:12:19.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg=', 'Content-Length': '0'} 2022-09-12T20:12:19.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb41f98863c6ced1-00631f92a3-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.979 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:19.980 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:19.981 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:19.981 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:12:19.982 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:12:19.982 DEBUG:boto:Method: GET 2022-09-12T20:12:19.983 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:12:19.983 DEBUG:boto:Data: 2022-09-12T20:12:19.984 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.984 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:19.985 DEBUG:boto:Port: 8000 2022-09-12T20:12:19.986 DEBUG:boto:Params: {} 2022-09-12T20:12:19.986 DEBUG:boto:Token: None 2022-09-12T20:12:19.987 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/ 2022-09-12T20:12:19.987 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg= 2022-09-12T20:12:19.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TX9z+mdhWZtBbcH072d7Sk3Djwg=', 'Content-Length': '0'} 2022-09-12T20:12:19.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc36c0490c3c520f-00631f92a3-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:19.991 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:19.992 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:19.993 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:19.993 DEBUG:boto:Method: HEAD 2022-09-12T20:12:19.994 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:12:19.994 DEBUG:boto:Data: 2022-09-12T20:12:19.995 DEBUG:boto:Headers: {} 2022-09-12T20:12:19.995 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:19.996 DEBUG:boto:Port: 8001 2022-09-12T20:12:19.996 DEBUG:boto:Params: {} 2022-09-12T20:12:19.997 DEBUG:boto:Token: None 2022-09-12T20:12:19.998 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:19 GMT /jdpiak-75/obj23 2022-09-12T20:12:19.998 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XwohsRhIml1J72Y5W93f/HB56h8= 2022-09-12T20:12:19.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XwohsRhIml1J72Y5W93f/HB56h8=', 'Content-Length': '0'} 2022-09-12T20:12:20.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d02fe03cad98681-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.001 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:20.002 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:20.002 DEBUG:boto:Method: HEAD 2022-09-12T20:12:20.003 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:12:20.003 DEBUG:boto:Data: 2022-09-12T20:12:20.004 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.004 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.005 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.005 DEBUG:boto:Params: {} 2022-09-12T20:12:20.006 DEBUG:boto:Token: None 2022-09-12T20:12:20.007 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj23 2022-09-12T20:12:20.007 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:anTPJhJIZEsYp6opSInX/rNoZx4= 2022-09-12T20:12:20.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:anTPJhJIZEsYp6opSInX/rNoZx4=', 'Content-Length': '0'} 2022-09-12T20:12:20.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac64a595d5d65d61-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:12:20.011 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:20.012 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:20.012 DEBUG:boto:Method: GET 2022-09-12T20:12:20.013 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:12:20.013 DEBUG:boto:Data: 2022-09-12T20:12:20.014 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.014 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.015 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.015 DEBUG:boto:Params: {} 2022-09-12T20:12:20.016 DEBUG:boto:Token: None 2022-09-12T20:12:20.017 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj23 2022-09-12T20:12:20.017 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x2PNS6VbxcHsO4ctpjdplUuX/vM= 2022-09-12T20:12:20.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x2PNS6VbxcHsO4ctpjdplUuX/vM=', 'Content-Length': '0'} 2022-09-12T20:12:20.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ce55e31d9828865-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.062 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:12:20.062 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:12:20.063 DEBUG:boto:Method: GET 2022-09-12T20:12:20.064 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:12:20.064 DEBUG:boto:Data: 2022-09-12T20:12:20.065 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.065 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.066 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.066 DEBUG:boto:Params: {} 2022-09-12T20:12:20.067 DEBUG:boto:Token: None 2022-09-12T20:12:20.067 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj23 2022-09-12T20:12:20.068 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x2PNS6VbxcHsO4ctpjdplUuX/vM= 2022-09-12T20:12:20.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x2PNS6VbxcHsO4ctpjdplUuX/vM=', 'Content-Length': '0'} 2022-09-12T20:12:20.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed8bf59b49271599-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.072 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:20.072 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:20.073 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:12:20.073 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:12:20.074 DEBUG:boto:Method: GET 2022-09-12T20:12:20.074 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:12:20.075 DEBUG:boto:Data: 2022-09-12T20:12:20.075 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.076 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.076 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.077 DEBUG:boto:Params: {} 2022-09-12T20:12:20.077 DEBUG:boto:Token: None 2022-09-12T20:12:20.078 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/ 2022-09-12T20:12:20.079 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs= 2022-09-12T20:12:20.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs=', 'Content-Length': '0'} 2022-09-12T20:12:20.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa26cf20ab399544-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.083 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:20.083 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:20.084 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:20.085 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:12:20.085 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:12:20.086 DEBUG:boto:Method: GET 2022-09-12T20:12:20.086 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:12:20.087 DEBUG:boto:Data: 2022-09-12T20:12:20.087 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.088 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.088 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.089 DEBUG:boto:Params: {} 2022-09-12T20:12:20.089 DEBUG:boto:Token: None 2022-09-12T20:12:20.090 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/ 2022-09-12T20:12:20.091 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs= 2022-09-12T20:12:20.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs=', 'Content-Length': '0'} 2022-09-12T20:12:20.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5f1f95eea9180df-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.094 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:20.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:12:20.096 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:20.096 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:20.097 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:20.097 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:20.098 DEBUG:boto:Method: GET 2022-09-12T20:12:20.098 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:20.099 DEBUG:boto:Data: 2022-09-12T20:12:20.099 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.100 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.101 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.101 DEBUG:boto:Params: {} 2022-09-12T20:12:20.102 DEBUG:boto:Token: None 2022-09-12T20:12:20.102 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:20.103 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Xc5vG3NDcID9uIVdLOfyr4RpOAA= 2022-09-12T20:12:20.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Xc5vG3NDcID9uIVdLOfyr4RpOAA=', 'Content-Length': '0'} 2022-09-12T20:12:20.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d6a891b64f503dc-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.106 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:20.107 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:20.108 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:20.108 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:20.109 DEBUG:boto:Method: GET 2022-09-12T20:12:20.109 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:20.110 DEBUG:boto:Data: 2022-09-12T20:12:20.110 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.111 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.111 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.112 DEBUG:boto:Params: {} 2022-09-12T20:12:20.112 DEBUG:boto:Token: None 2022-09-12T20:12:20.113 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:20.114 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Xc5vG3NDcID9uIVdLOfyr4RpOAA= 2022-09-12T20:12:20.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Xc5vG3NDcID9uIVdLOfyr4RpOAA=', 'Content-Length': '0'} 2022-09-12T20:12:20.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e6808dc669dc2da-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.117 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:20.118 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:20.118 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:20.119 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:20.119 DEBUG:boto:Method: HEAD 2022-09-12T20:12:20.120 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:20.120 DEBUG:boto:Data: 2022-09-12T20:12:20.121 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.121 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.122 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.122 DEBUG:boto:Params: {} 2022-09-12T20:12:20.123 DEBUG:boto:Token: None 2022-09-12T20:12:20.123 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:20.124 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+7htUf4QvfNNYtZ+XtC6/n3dnWQ= 2022-09-12T20:12:20.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+7htUf4QvfNNYtZ+XtC6/n3dnWQ=', 'Content-Length': '0'} 2022-09-12T20:12:20.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2eac2bf1987710b-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.127 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:20.128 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:20.128 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:20.129 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:20.129 DEBUG:boto:Method: HEAD 2022-09-12T20:12:20.130 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:20.130 DEBUG:boto:Data: 2022-09-12T20:12:20.131 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.131 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.132 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.132 DEBUG:boto:Params: {} 2022-09-12T20:12:20.133 DEBUG:boto:Token: None 2022-09-12T20:12:20.133 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:20.134 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+7htUf4QvfNNYtZ+XtC6/n3dnWQ= 2022-09-12T20:12:20.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+7htUf4QvfNNYtZ+XtC6/n3dnWQ=', 'Content-Length': '0'} 2022-09-12T20:12:20.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000850a74424e85b515-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:12:20.138 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:20.138 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:20.139 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:20.139 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:20.140 DEBUG:boto:Method: GET 2022-09-12T20:12:20.140 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:20.141 DEBUG:boto:Data: 2022-09-12T20:12:20.141 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.142 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.142 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.143 DEBUG:boto:Params: {} 2022-09-12T20:12:20.143 DEBUG:boto:Token: None 2022-09-12T20:12:20.144 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:20.145 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Xc5vG3NDcID9uIVdLOfyr4RpOAA= 2022-09-12T20:12:20.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Xc5vG3NDcID9uIVdLOfyr4RpOAA=', 'Content-Length': '0'} 2022-09-12T20:12:20.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000605a2db2ff7579d1-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.190 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:20.191 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:20.191 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:20.192 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:12:20.192 DEBUG:boto:Method: GET 2022-09-12T20:12:20.193 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:20.193 DEBUG:boto:Data: 2022-09-12T20:12:20.194 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.194 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.195 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.195 DEBUG:boto:Params: {} 2022-09-12T20:12:20.196 DEBUG:boto:Token: None 2022-09-12T20:12:20.196 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:12:20.197 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Xc5vG3NDcID9uIVdLOfyr4RpOAA= 2022-09-12T20:12:20.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Xc5vG3NDcID9uIVdLOfyr4RpOAA=', 'Content-Length': '0'} 2022-09-12T20:12:20.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000d18da3f80dde3f-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.201 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:20.201 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:20.202 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:12:20.202 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:12:20.203 DEBUG:boto:Method: GET 2022-09-12T20:12:20.203 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:12:20.204 DEBUG:boto:Data: 2022-09-12T20:12:20.204 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.205 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.205 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.206 DEBUG:boto:Params: {} 2022-09-12T20:12:20.206 DEBUG:boto:Token: None 2022-09-12T20:12:20.207 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/ 2022-09-12T20:12:20.207 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs= 2022-09-12T20:12:20.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs=', 'Content-Length': '0'} 2022-09-12T20:12:20.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e1c7ab007055079-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.212 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:20.212 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:20.213 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:20.214 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:12:20.214 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:12:20.215 DEBUG:boto:Method: GET 2022-09-12T20:12:20.215 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:12:20.216 DEBUG:boto:Data: 2022-09-12T20:12:20.216 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.217 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.217 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.218 DEBUG:boto:Params: {} 2022-09-12T20:12:20.218 DEBUG:boto:Token: None 2022-09-12T20:12:20.219 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/ 2022-09-12T20:12:20.220 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs= 2022-09-12T20:12:20.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs=', 'Content-Length': '0'} 2022-09-12T20:12:20.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001deff85ff3774c1b-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.223 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:20.224 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:20.225 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:20.225 DEBUG:boto:Method: HEAD 2022-09-12T20:12:20.226 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:12:20.226 DEBUG:boto:Data: 2022-09-12T20:12:20.227 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.227 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.228 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.228 DEBUG:boto:Params: {} 2022-09-12T20:12:20.229 DEBUG:boto:Token: None 2022-09-12T20:12:20.229 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj29 2022-09-12T20:12:20.230 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4sSgAj8cFoIbHmuXC9PryJlMEtI= 2022-09-12T20:12:20.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4sSgAj8cFoIbHmuXC9PryJlMEtI=', 'Content-Length': '0'} 2022-09-12T20:12:20.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006409306b9e70b06d-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.233 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:20.234 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:20.234 DEBUG:boto:Method: HEAD 2022-09-12T20:12:20.235 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:12:20.235 DEBUG:boto:Data: 2022-09-12T20:12:20.236 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.236 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.237 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.237 DEBUG:boto:Params: {} 2022-09-12T20:12:20.238 DEBUG:boto:Token: None 2022-09-12T20:12:20.239 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj29 2022-09-12T20:12:20.239 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4sSgAj8cFoIbHmuXC9PryJlMEtI= 2022-09-12T20:12:20.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4sSgAj8cFoIbHmuXC9PryJlMEtI=', 'Content-Length': '0'} 2022-09-12T20:12:20.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a33542e5eea37b6f-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:12:20.243 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:20.243 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:20.244 DEBUG:boto:Method: GET 2022-09-12T20:12:20.245 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:12:20.245 DEBUG:boto:Data: 2022-09-12T20:12:20.246 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.246 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.247 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.247 DEBUG:boto:Params: {} 2022-09-12T20:12:20.248 DEBUG:boto:Token: None 2022-09-12T20:12:20.248 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj29 2022-09-12T20:12:20.249 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ECzh2nOqVpw994kFxvWXuY0FK0= 2022-09-12T20:12:20.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ECzh2nOqVpw994kFxvWXuY0FK0=', 'Content-Length': '0'} 2022-09-12T20:12:20.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009308a0cdeaca47e8-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.253 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:12:20.253 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:12:20.254 DEBUG:boto:Method: GET 2022-09-12T20:12:20.254 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:12:20.255 DEBUG:boto:Data: 2022-09-12T20:12:20.255 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.256 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.257 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.257 DEBUG:boto:Params: {} 2022-09-12T20:12:20.258 DEBUG:boto:Token: None 2022-09-12T20:12:20.258 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj29 2022-09-12T20:12:20.259 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ECzh2nOqVpw994kFxvWXuY0FK0= 2022-09-12T20:12:20.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ECzh2nOqVpw994kFxvWXuY0FK0=', 'Content-Length': '0'} 2022-09-12T20:12:20.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e70c9862bba2331a-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.302 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:20.303 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:20.303 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:12:20.304 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:12:20.304 DEBUG:boto:Method: GET 2022-09-12T20:12:20.305 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:12:20.306 DEBUG:boto:Data: 2022-09-12T20:12:20.306 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.307 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.307 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.308 DEBUG:boto:Params: {} 2022-09-12T20:12:20.308 DEBUG:boto:Token: None 2022-09-12T20:12:20.309 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/ 2022-09-12T20:12:20.309 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs= 2022-09-12T20:12:20.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs=', 'Content-Length': '0'} 2022-09-12T20:12:20.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c6acdd29e343fdb-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.313 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:20.314 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:20.315 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:20.315 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:12:20.316 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:12:20.317 DEBUG:boto:Method: GET 2022-09-12T20:12:20.317 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:12:20.318 DEBUG:boto:Data: 2022-09-12T20:12:20.318 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.319 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.319 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.320 DEBUG:boto:Params: {} 2022-09-12T20:12:20.321 DEBUG:boto:Token: None 2022-09-12T20:12:20.321 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/ 2022-09-12T20:12:20.322 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs= 2022-09-12T20:12:20.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs=', 'Content-Length': '0'} 2022-09-12T20:12:20.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000740d8519f70686cc-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.326 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:20.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:12:20.327 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:20.328 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:20.328 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:20.329 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:20.329 DEBUG:boto:Method: GET 2022-09-12T20:12:20.330 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:20.330 DEBUG:boto:Data: 2022-09-12T20:12:20.331 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.331 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.332 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.332 DEBUG:boto:Params: {} 2022-09-12T20:12:20.333 DEBUG:boto:Token: None 2022-09-12T20:12:20.334 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:20.334 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9y6TyEkQS0jgCwqg7dms0Da//Jc= 2022-09-12T20:12:20.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9y6TyEkQS0jgCwqg7dms0Da//Jc=', 'Content-Length': '0'} 2022-09-12T20:12:20.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001aebd1bd747ef68a-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.338 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:20.339 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:20.339 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:20.340 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:20.341 DEBUG:boto:Method: GET 2022-09-12T20:12:20.341 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:20.342 DEBUG:boto:Data: 2022-09-12T20:12:20.342 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.343 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.343 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.344 DEBUG:boto:Params: {} 2022-09-12T20:12:20.345 DEBUG:boto:Token: None 2022-09-12T20:12:20.345 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:20.346 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9y6TyEkQS0jgCwqg7dms0Da//Jc= 2022-09-12T20:12:20.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9y6TyEkQS0jgCwqg7dms0Da//Jc=', 'Content-Length': '0'} 2022-09-12T20:12:20.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001aaa0cef39c6a8bb-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.349 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:20.350 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:20.351 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:20.351 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:20.352 DEBUG:boto:Method: HEAD 2022-09-12T20:12:20.353 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:20.353 DEBUG:boto:Data: 2022-09-12T20:12:20.354 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.354 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.355 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.355 DEBUG:boto:Params: {} 2022-09-12T20:12:20.356 DEBUG:boto:Token: None 2022-09-12T20:12:20.357 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:20.357 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WtUqla9TgkW3DVb9AT537ecDrxk= 2022-09-12T20:12:20.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WtUqla9TgkW3DVb9AT537ecDrxk=', 'Content-Length': '0'} 2022-09-12T20:12:20.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b51669aad0967e3-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.360 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:20.361 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:20.361 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:20.362 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:20.363 DEBUG:boto:Method: HEAD 2022-09-12T20:12:20.363 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:20.364 DEBUG:boto:Data: 2022-09-12T20:12:20.364 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.365 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.365 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.366 DEBUG:boto:Params: {} 2022-09-12T20:12:20.366 DEBUG:boto:Token: None 2022-09-12T20:12:20.367 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:20.368 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WtUqla9TgkW3DVb9AT537ecDrxk= 2022-09-12T20:12:20.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WtUqla9TgkW3DVb9AT537ecDrxk=', 'Content-Length': '0'} 2022-09-12T20:12:20.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b155a1c09489063b-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:12:20.372 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:20.372 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:20.373 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:20.373 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:20.374 DEBUG:boto:Method: GET 2022-09-12T20:12:20.374 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:20.375 DEBUG:boto:Data: 2022-09-12T20:12:20.375 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.376 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.377 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.377 DEBUG:boto:Params: {} 2022-09-12T20:12:20.378 DEBUG:boto:Token: None 2022-09-12T20:12:20.379 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:20.379 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9y6TyEkQS0jgCwqg7dms0Da//Jc= 2022-09-12T20:12:20.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9y6TyEkQS0jgCwqg7dms0Da//Jc=', 'Content-Length': '0'} 2022-09-12T20:12:20.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008812c683710503a1-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.383 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:20.384 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:20.384 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:20.385 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:12:20.386 DEBUG:boto:Method: GET 2022-09-12T20:12:20.386 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:20.387 DEBUG:boto:Data: 2022-09-12T20:12:20.388 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.388 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.389 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.390 DEBUG:boto:Params: {} 2022-09-12T20:12:20.391 DEBUG:boto:Token: None 2022-09-12T20:12:20.392 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:12:20.392 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9y6TyEkQS0jgCwqg7dms0Da//Jc= 2022-09-12T20:12:20.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9y6TyEkQS0jgCwqg7dms0Da//Jc=', 'Content-Length': '0'} 2022-09-12T20:12:20.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd8c520d58901424-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.397 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:20.397 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:20.398 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:12:20.399 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:12:20.400 DEBUG:boto:Method: GET 2022-09-12T20:12:20.400 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:12:20.401 DEBUG:boto:Data: 2022-09-12T20:12:20.402 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.402 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.403 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.404 DEBUG:boto:Params: {} 2022-09-12T20:12:20.404 DEBUG:boto:Token: None 2022-09-12T20:12:20.405 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/ 2022-09-12T20:12:20.406 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs= 2022-09-12T20:12:20.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs=', 'Content-Length': '0'} 2022-09-12T20:12:20.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007af3804e25109fa-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.450 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:20.451 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:20.452 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:20.452 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:12:20.453 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:12:20.453 DEBUG:boto:Method: GET 2022-09-12T20:12:20.454 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:12:20.454 DEBUG:boto:Data: 2022-09-12T20:12:20.455 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.455 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.456 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.457 DEBUG:boto:Params: {} 2022-09-12T20:12:20.457 DEBUG:boto:Token: None 2022-09-12T20:12:20.458 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/ 2022-09-12T20:12:20.458 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs= 2022-09-12T20:12:20.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs=', 'Content-Length': '0'} 2022-09-12T20:12:20.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000753471103705e0bc-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.462 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:20.463 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:20.464 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:20.464 DEBUG:boto:Method: HEAD 2022-09-12T20:12:20.465 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:12:20.465 DEBUG:boto:Data: 2022-09-12T20:12:20.466 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.466 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.467 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.467 DEBUG:boto:Params: {} 2022-09-12T20:12:20.468 DEBUG:boto:Token: None 2022-09-12T20:12:20.469 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj33 2022-09-12T20:12:20.469 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LQpMB/Rlv4MeHpGzyz9QHPGFZLg= 2022-09-12T20:12:20.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LQpMB/Rlv4MeHpGzyz9QHPGFZLg=', 'Content-Length': '0'} 2022-09-12T20:12:20.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df4f1d2b08cf269f-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.473 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:20.473 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:20.474 DEBUG:boto:Method: HEAD 2022-09-12T20:12:20.474 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:12:20.475 DEBUG:boto:Data: 2022-09-12T20:12:20.475 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.476 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.477 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.477 DEBUG:boto:Params: {} 2022-09-12T20:12:20.478 DEBUG:boto:Token: None 2022-09-12T20:12:20.478 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj33 2022-09-12T20:12:20.479 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LQpMB/Rlv4MeHpGzyz9QHPGFZLg= 2022-09-12T20:12:20.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LQpMB/Rlv4MeHpGzyz9QHPGFZLg=', 'Content-Length': '0'} 2022-09-12T20:12:20.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b9719002b7a94b9-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:12:20.483 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:20.483 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:20.484 DEBUG:boto:Method: GET 2022-09-12T20:12:20.485 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:12:20.485 DEBUG:boto:Data: 2022-09-12T20:12:20.486 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.486 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.487 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.487 DEBUG:boto:Params: {} 2022-09-12T20:12:20.488 DEBUG:boto:Token: None 2022-09-12T20:12:20.489 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj33 2022-09-12T20:12:20.489 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ndvkrwCU2oiZ07bP/hXRLvNUetE= 2022-09-12T20:12:20.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ndvkrwCU2oiZ07bP/hXRLvNUetE=', 'Content-Length': '0'} 2022-09-12T20:12:20.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff3504ab66baf3fc-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.493 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:12:20.494 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:12:20.494 DEBUG:boto:Method: GET 2022-09-12T20:12:20.495 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:12:20.495 DEBUG:boto:Data: 2022-09-12T20:12:20.496 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.496 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.497 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.497 DEBUG:boto:Params: {} 2022-09-12T20:12:20.498 DEBUG:boto:Token: None 2022-09-12T20:12:20.499 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj33 2022-09-12T20:12:20.499 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ndvkrwCU2oiZ07bP/hXRLvNUetE= 2022-09-12T20:12:20.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ndvkrwCU2oiZ07bP/hXRLvNUetE=', 'Content-Length': '0'} 2022-09-12T20:12:20.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000973fd39861bc9a44-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.546 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:20.546 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:20.547 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:12:20.548 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:12:20.548 DEBUG:boto:Method: GET 2022-09-12T20:12:20.549 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:12:20.549 DEBUG:boto:Data: 2022-09-12T20:12:20.550 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.550 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.551 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.551 DEBUG:boto:Params: {} 2022-09-12T20:12:20.552 DEBUG:boto:Token: None 2022-09-12T20:12:20.552 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/ 2022-09-12T20:12:20.553 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs= 2022-09-12T20:12:20.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs=', 'Content-Length': '0'} 2022-09-12T20:12:20.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cc2f262f2640ef3-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.557 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:20.558 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:20.559 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:20.559 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:12:20.560 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:12:20.560 DEBUG:boto:Method: GET 2022-09-12T20:12:20.561 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:12:20.561 DEBUG:boto:Data: 2022-09-12T20:12:20.562 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.562 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.563 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.563 DEBUG:boto:Params: {} 2022-09-12T20:12:20.564 DEBUG:boto:Token: None 2022-09-12T20:12:20.564 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/ 2022-09-12T20:12:20.565 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs= 2022-09-12T20:12:20.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs=', 'Content-Length': '0'} 2022-09-12T20:12:20.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007735aa0aeb888ab3-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.569 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:20.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:12:20.570 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:20.571 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:20.571 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:20.572 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:20.572 DEBUG:boto:Method: GET 2022-09-12T20:12:20.573 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:20.573 DEBUG:boto:Data: 2022-09-12T20:12:20.574 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.575 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.575 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.576 DEBUG:boto:Params: {} 2022-09-12T20:12:20.576 DEBUG:boto:Token: None 2022-09-12T20:12:20.577 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:20.577 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ox3rns3ZNG1RAgFfWIUpl8c6P14= 2022-09-12T20:12:20.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ox3rns3ZNG1RAgFfWIUpl8c6P14=', 'Content-Length': '0'} 2022-09-12T20:12:20.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dde8721e979a4e12-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.581 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:20.582 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:20.583 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:20.584 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:20.584 DEBUG:boto:Method: GET 2022-09-12T20:12:20.585 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:20.586 DEBUG:boto:Data: 2022-09-12T20:12:20.586 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.587 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.588 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.589 DEBUG:boto:Params: {} 2022-09-12T20:12:20.589 DEBUG:boto:Token: None 2022-09-12T20:12:20.590 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:20.591 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ox3rns3ZNG1RAgFfWIUpl8c6P14= 2022-09-12T20:12:20.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ox3rns3ZNG1RAgFfWIUpl8c6P14=', 'Content-Length': '0'} 2022-09-12T20:12:20.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c42a10abbab3edc6-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.595 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:20.596 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:20.596 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:20.597 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:20.597 DEBUG:boto:Method: HEAD 2022-09-12T20:12:20.598 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:20.599 DEBUG:boto:Data: 2022-09-12T20:12:20.599 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.600 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.600 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.601 DEBUG:boto:Params: {} 2022-09-12T20:12:20.602 DEBUG:boto:Token: None 2022-09-12T20:12:20.603 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:20.604 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XwBjyZO55g4ZfipnFhb0acr2uAE= 2022-09-12T20:12:20.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XwBjyZO55g4ZfipnFhb0acr2uAE=', 'Content-Length': '0'} 2022-09-12T20:12:20.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008331b9495e323c36-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.608 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:20.608 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:20.609 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:20.610 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:20.610 DEBUG:boto:Method: HEAD 2022-09-12T20:12:20.611 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:20.612 DEBUG:boto:Data: 2022-09-12T20:12:20.612 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.613 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.614 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.615 DEBUG:boto:Params: {} 2022-09-12T20:12:20.615 DEBUG:boto:Token: None 2022-09-12T20:12:20.616 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:20.617 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XwBjyZO55g4ZfipnFhb0acr2uAE= 2022-09-12T20:12:20.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XwBjyZO55g4ZfipnFhb0acr2uAE=', 'Content-Length': '0'} 2022-09-12T20:12:20.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcc4bcd73d84b234-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:12:20.621 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:20.622 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:20.623 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:20.623 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:20.624 DEBUG:boto:Method: GET 2022-09-12T20:12:20.625 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:20.626 DEBUG:boto:Data: 2022-09-12T20:12:20.626 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.627 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.628 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.629 DEBUG:boto:Params: {} 2022-09-12T20:12:20.630 DEBUG:boto:Token: None 2022-09-12T20:12:20.631 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:20.631 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ox3rns3ZNG1RAgFfWIUpl8c6P14= 2022-09-12T20:12:20.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ox3rns3ZNG1RAgFfWIUpl8c6P14=', 'Content-Length': '0'} 2022-09-12T20:12:20.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d02ec8901e32c11-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.636 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:20.636 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:20.637 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:20.638 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:12:20.638 DEBUG:boto:Method: GET 2022-09-12T20:12:20.639 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:20.640 DEBUG:boto:Data: 2022-09-12T20:12:20.641 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.641 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.642 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.643 DEBUG:boto:Params: {} 2022-09-12T20:12:20.643 DEBUG:boto:Token: None 2022-09-12T20:12:20.644 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:12:20.645 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ox3rns3ZNG1RAgFfWIUpl8c6P14= 2022-09-12T20:12:20.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ox3rns3ZNG1RAgFfWIUpl8c6P14=', 'Content-Length': '0'} 2022-09-12T20:12:20.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046b60de819bf0c30-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.649 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:20.650 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:20.650 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:12:20.651 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:12:20.651 DEBUG:boto:Method: GET 2022-09-12T20:12:20.652 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:12:20.653 DEBUG:boto:Data: 2022-09-12T20:12:20.653 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.654 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.655 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.656 DEBUG:boto:Params: {} 2022-09-12T20:12:20.656 DEBUG:boto:Token: None 2022-09-12T20:12:20.657 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/ 2022-09-12T20:12:20.658 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs= 2022-09-12T20:12:20.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs=', 'Content-Length': '0'} 2022-09-12T20:12:20.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091180f36c0bfeb45-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.702 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:20.703 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:20.704 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:20.704 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:12:20.705 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:12:20.705 DEBUG:boto:Method: GET 2022-09-12T20:12:20.706 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:12:20.707 DEBUG:boto:Data: 2022-09-12T20:12:20.708 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.708 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.709 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.710 DEBUG:boto:Params: {} 2022-09-12T20:12:20.711 DEBUG:boto:Token: None 2022-09-12T20:12:20.711 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/ 2022-09-12T20:12:20.712 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs= 2022-09-12T20:12:20.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs=', 'Content-Length': '0'} 2022-09-12T20:12:20.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000566e2d445d04828d-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.717 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:20.718 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:20.719 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:20.719 DEBUG:boto:Method: HEAD 2022-09-12T20:12:20.720 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:12:20.721 DEBUG:boto:Data: 2022-09-12T20:12:20.721 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.722 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.722 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.723 DEBUG:boto:Params: {} 2022-09-12T20:12:20.724 DEBUG:boto:Token: None 2022-09-12T20:12:20.724 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj34 2022-09-12T20:12:20.725 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2j+DlGqggGifvwBjrN7htEMD4To= 2022-09-12T20:12:20.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2j+DlGqggGifvwBjrN7htEMD4To=', 'Content-Length': '0'} 2022-09-12T20:12:20.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fccc5ab5a5a451e-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.729 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:20.729 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:20.730 DEBUG:boto:Method: HEAD 2022-09-12T20:12:20.731 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:12:20.731 DEBUG:boto:Data: 2022-09-12T20:12:20.732 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.732 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.733 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.734 DEBUG:boto:Params: {} 2022-09-12T20:12:20.734 DEBUG:boto:Token: None 2022-09-12T20:12:20.735 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj34 2022-09-12T20:12:20.736 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2j+DlGqggGifvwBjrN7htEMD4To= 2022-09-12T20:12:20.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2j+DlGqggGifvwBjrN7htEMD4To=', 'Content-Length': '0'} 2022-09-12T20:12:20.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f411d23aabbe80e5-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:12:20.740 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:20.741 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:20.742 DEBUG:boto:Method: GET 2022-09-12T20:12:20.743 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:12:20.744 DEBUG:boto:Data: 2022-09-12T20:12:20.744 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.745 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.746 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.747 DEBUG:boto:Params: {} 2022-09-12T20:12:20.748 DEBUG:boto:Token: None 2022-09-12T20:12:20.748 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj34 2022-09-12T20:12:20.749 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cWQO0Gatw9tnTB0yqu9ILgUwL7Y= 2022-09-12T20:12:20.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cWQO0Gatw9tnTB0yqu9ILgUwL7Y=', 'Content-Length': '0'} 2022-09-12T20:12:20.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000120914f45c598053-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.753 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:12:20.754 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:12:20.754 DEBUG:boto:Method: GET 2022-09-12T20:12:20.755 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:12:20.756 DEBUG:boto:Data: 2022-09-12T20:12:20.757 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.758 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.758 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.759 DEBUG:boto:Params: {} 2022-09-12T20:12:20.760 DEBUG:boto:Token: None 2022-09-12T20:12:20.761 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj34 2022-09-12T20:12:20.762 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cWQO0Gatw9tnTB0yqu9ILgUwL7Y= 2022-09-12T20:12:20.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cWQO0Gatw9tnTB0yqu9ILgUwL7Y=', 'Content-Length': '0'} 2022-09-12T20:12:20.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af9fc84987b04314-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.806 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:20.807 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:20.808 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:12:20.808 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:12:20.809 DEBUG:boto:Method: GET 2022-09-12T20:12:20.810 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:12:20.810 DEBUG:boto:Data: 2022-09-12T20:12:20.811 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.812 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.812 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.813 DEBUG:boto:Params: {} 2022-09-12T20:12:20.813 DEBUG:boto:Token: None 2022-09-12T20:12:20.814 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/ 2022-09-12T20:12:20.815 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs= 2022-09-12T20:12:20.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs=', 'Content-Length': '0'} 2022-09-12T20:12:20.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004275bda576158b75-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.819 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:20.820 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:20.821 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:20.822 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:12:20.823 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:12:20.823 DEBUG:boto:Method: GET 2022-09-12T20:12:20.824 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:12:20.824 DEBUG:boto:Data: 2022-09-12T20:12:20.825 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.826 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.832 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.833 DEBUG:boto:Params: {} 2022-09-12T20:12:20.833 DEBUG:boto:Token: None 2022-09-12T20:12:20.834 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/ 2022-09-12T20:12:20.835 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs= 2022-09-12T20:12:20.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs=', 'Content-Length': '0'} 2022-09-12T20:12:20.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006533153b64a30419-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.839 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:20.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:12:20.841 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:20.841 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:20.842 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:20.843 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:20.843 DEBUG:boto:Method: GET 2022-09-12T20:12:20.844 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:20.845 DEBUG:boto:Data: 2022-09-12T20:12:20.845 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.846 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.847 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.847 DEBUG:boto:Params: {} 2022-09-12T20:12:20.848 DEBUG:boto:Token: None 2022-09-12T20:12:20.849 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:20.850 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:twcc4BRiHXDK2HM4myQeb2wdVgs= 2022-09-12T20:12:20.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:twcc4BRiHXDK2HM4myQeb2wdVgs=', 'Content-Length': '0'} 2022-09-12T20:12:20.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bc12fa4e13ba000-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.854 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:20.854 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:20.855 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:20.855 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:20.856 DEBUG:boto:Method: GET 2022-09-12T20:12:20.857 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:20.857 DEBUG:boto:Data: 2022-09-12T20:12:20.858 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.859 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.859 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.860 DEBUG:boto:Params: {} 2022-09-12T20:12:20.861 DEBUG:boto:Token: None 2022-09-12T20:12:20.861 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:20.862 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:twcc4BRiHXDK2HM4myQeb2wdVgs= 2022-09-12T20:12:20.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:twcc4BRiHXDK2HM4myQeb2wdVgs=', 'Content-Length': '0'} 2022-09-12T20:12:20.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dbaf9156582b2eb-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.866 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:20.866 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:20.867 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:20.867 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:20.868 DEBUG:boto:Method: HEAD 2022-09-12T20:12:20.868 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:20.869 DEBUG:boto:Data: 2022-09-12T20:12:20.869 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.870 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.870 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.871 DEBUG:boto:Params: {} 2022-09-12T20:12:20.871 DEBUG:boto:Token: None 2022-09-12T20:12:20.872 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:20.872 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nPWILpETDEX1ljA+Ml/0CJKI22c= 2022-09-12T20:12:20.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nPWILpETDEX1ljA+Ml/0CJKI22c=', 'Content-Length': '0'} 2022-09-12T20:12:20.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053cc3c893c0538cc-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.876 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:20.876 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:20.877 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:20.877 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:20.877 DEBUG:boto:Method: HEAD 2022-09-12T20:12:20.878 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:20.878 DEBUG:boto:Data: 2022-09-12T20:12:20.879 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.879 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.880 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.880 DEBUG:boto:Params: {} 2022-09-12T20:12:20.881 DEBUG:boto:Token: None 2022-09-12T20:12:20.881 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:20.882 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nPWILpETDEX1ljA+Ml/0CJKI22c= 2022-09-12T20:12:20.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nPWILpETDEX1ljA+Ml/0CJKI22c=', 'Content-Length': '0'} 2022-09-12T20:12:20.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfb6dd5e9e26be5d-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:12:20.886 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:20.886 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:20.887 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:20.887 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:20.888 DEBUG:boto:Method: GET 2022-09-12T20:12:20.888 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:20.889 DEBUG:boto:Data: 2022-09-12T20:12:20.890 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.890 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.891 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.891 DEBUG:boto:Params: {} 2022-09-12T20:12:20.892 DEBUG:boto:Token: None 2022-09-12T20:12:20.892 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:20.893 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:twcc4BRiHXDK2HM4myQeb2wdVgs= 2022-09-12T20:12:20.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:twcc4BRiHXDK2HM4myQeb2wdVgs=', 'Content-Length': '0'} 2022-09-12T20:12:20.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f267d56234389f1b-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.897 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:20.897 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:20.898 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:20.898 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:12:20.899 DEBUG:boto:Method: GET 2022-09-12T20:12:20.899 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:20.900 DEBUG:boto:Data: 2022-09-12T20:12:20.900 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.901 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.901 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.902 DEBUG:boto:Params: {} 2022-09-12T20:12:20.903 DEBUG:boto:Token: None 2022-09-12T20:12:20.903 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:12:20.904 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:twcc4BRiHXDK2HM4myQeb2wdVgs= 2022-09-12T20:12:20.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:twcc4BRiHXDK2HM4myQeb2wdVgs=', 'Content-Length': '0'} 2022-09-12T20:12:20.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ddee4b5f178015e-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.907 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:20.908 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:20.908 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:12:20.909 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:12:20.910 DEBUG:boto:Method: GET 2022-09-12T20:12:20.910 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:12:20.911 DEBUG:boto:Data: 2022-09-12T20:12:20.911 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.912 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.912 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.913 DEBUG:boto:Params: {} 2022-09-12T20:12:20.913 DEBUG:boto:Token: None 2022-09-12T20:12:20.914 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/ 2022-09-12T20:12:20.914 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs= 2022-09-12T20:12:20.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs=', 'Content-Length': '0'} 2022-09-12T20:12:20.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5d28c0dac8a1bae-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.918 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:20.919 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:20.920 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:20.920 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:12:20.921 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:12:20.921 DEBUG:boto:Method: GET 2022-09-12T20:12:20.922 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:12:20.922 DEBUG:boto:Data: 2022-09-12T20:12:20.923 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.923 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.924 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.924 DEBUG:boto:Params: {} 2022-09-12T20:12:20.925 DEBUG:boto:Token: None 2022-09-12T20:12:20.925 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/ 2022-09-12T20:12:20.926 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs= 2022-09-12T20:12:20.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DNQsRYbOUAp+yBmJaPfekDAh0Vs=', 'Content-Length': '0'} 2022-09-12T20:12:20.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087bb7ee9ae2682c3-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.970 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:20.971 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:20.972 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:20.972 DEBUG:boto:Method: HEAD 2022-09-12T20:12:20.973 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:12:20.973 DEBUG:boto:Data: 2022-09-12T20:12:20.974 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.974 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.975 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.975 DEBUG:boto:Params: {} 2022-09-12T20:12:20.976 DEBUG:boto:Token: None 2022-09-12T20:12:20.976 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj38 2022-09-12T20:12:20.977 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:U5uly+T2ry1KR4+xRkViPMpLU70= 2022-09-12T20:12:20.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:U5uly+T2ry1KR4+xRkViPMpLU70=', 'Content-Length': '0'} 2022-09-12T20:12:20.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000138ebb1d31024317-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.980 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:20.981 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:20.981 DEBUG:boto:Method: HEAD 2022-09-12T20:12:20.982 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:12:20.982 DEBUG:boto:Data: 2022-09-12T20:12:20.983 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.983 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:20.984 DEBUG:boto:Port: 8000 2022-09-12T20:12:20.984 DEBUG:boto:Params: {} 2022-09-12T20:12:20.985 DEBUG:boto:Token: None 2022-09-12T20:12:20.985 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj38 2022-09-12T20:12:20.986 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:U5uly+T2ry1KR4+xRkViPMpLU70= 2022-09-12T20:12:20.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:U5uly+T2ry1KR4+xRkViPMpLU70=', 'Content-Length': '0'} 2022-09-12T20:12:20.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f64827991d228bfb-00631f92a4-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:12:20.990 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:20.990 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:20.991 DEBUG:boto:Method: GET 2022-09-12T20:12:20.991 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:12:20.992 DEBUG:boto:Data: 2022-09-12T20:12:20.992 DEBUG:boto:Headers: {} 2022-09-12T20:12:20.993 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:20.993 DEBUG:boto:Port: 8001 2022-09-12T20:12:20.994 DEBUG:boto:Params: {} 2022-09-12T20:12:20.994 DEBUG:boto:Token: None 2022-09-12T20:12:20.995 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:20 GMT /jdpiak-75/obj38 2022-09-12T20:12:20.996 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KvoU09hjomjc5oKL3ARQTCr/LUo= 2022-09-12T20:12:20.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KvoU09hjomjc5oKL3ARQTCr/LUo=', 'Content-Length': '0'} 2022-09-12T20:12:20.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000042d7bbcfb77c191-00631f92a4-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:20.999 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:12:21.000 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:12:21.000 DEBUG:boto:Method: GET 2022-09-12T20:12:21.001 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:12:21.001 DEBUG:boto:Data: 2022-09-12T20:12:21.002 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.002 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.003 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.003 DEBUG:boto:Params: {} 2022-09-12T20:12:21.004 DEBUG:boto:Token: None 2022-09-12T20:12:21.005 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj38 2022-09-12T20:12:21.005 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6l6yzmq+WKrrSVrXyOl3QZhurvU= 2022-09-12T20:12:21.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6l6yzmq+WKrrSVrXyOl3QZhurvU=', 'Content-Length': '0'} 2022-09-12T20:12:21.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b64e910ce70d8ba-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.008 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:21.009 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:21.010 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:12:21.010 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:12:21.011 DEBUG:boto:Method: GET 2022-09-12T20:12:21.011 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:12:21.012 DEBUG:boto:Data: 2022-09-12T20:12:21.012 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.013 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.013 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.014 DEBUG:boto:Params: {} 2022-09-12T20:12:21.014 DEBUG:boto:Token: None 2022-09-12T20:12:21.015 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/ 2022-09-12T20:12:21.016 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA= 2022-09-12T20:12:21.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA=', 'Content-Length': '0'} 2022-09-12T20:12:21.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1dad3d183a8c664-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.062 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:21.063 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:21.063 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:21.064 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:12:21.064 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:12:21.065 DEBUG:boto:Method: GET 2022-09-12T20:12:21.066 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:12:21.066 DEBUG:boto:Data: 2022-09-12T20:12:21.067 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.067 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.068 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.068 DEBUG:boto:Params: {} 2022-09-12T20:12:21.069 DEBUG:boto:Token: None 2022-09-12T20:12:21.069 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/ 2022-09-12T20:12:21.070 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA= 2022-09-12T20:12:21.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA=', 'Content-Length': '0'} 2022-09-12T20:12:21.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f06e7dc30a4eec1f-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.074 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:21.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:12:21.075 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:21.076 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:21.076 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:21.077 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:21.077 DEBUG:boto:Method: GET 2022-09-12T20:12:21.078 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:21.079 DEBUG:boto:Data: 2022-09-12T20:12:21.079 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.080 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.080 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.081 DEBUG:boto:Params: {} 2022-09-12T20:12:21.081 DEBUG:boto:Token: None 2022-09-12T20:12:21.082 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:21.082 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0mZAOPqW8KRqHay8IKXqxTAyPmE= 2022-09-12T20:12:21.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0mZAOPqW8KRqHay8IKXqxTAyPmE=', 'Content-Length': '0'} 2022-09-12T20:12:21.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000225061790d1dbe78-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.086 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:21.086 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:21.087 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:21.087 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:21.088 DEBUG:boto:Method: GET 2022-09-12T20:12:21.088 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:21.089 DEBUG:boto:Data: 2022-09-12T20:12:21.089 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.090 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.091 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.091 DEBUG:boto:Params: {} 2022-09-12T20:12:21.092 DEBUG:boto:Token: None 2022-09-12T20:12:21.092 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:21.093 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0mZAOPqW8KRqHay8IKXqxTAyPmE= 2022-09-12T20:12:21.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0mZAOPqW8KRqHay8IKXqxTAyPmE=', 'Content-Length': '0'} 2022-09-12T20:12:21.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dab2ea0045795676-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.097 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:21.097 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:21.098 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:21.098 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:21.099 DEBUG:boto:Method: HEAD 2022-09-12T20:12:21.099 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:21.100 DEBUG:boto:Data: 2022-09-12T20:12:21.100 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.100 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.101 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.101 DEBUG:boto:Params: {} 2022-09-12T20:12:21.102 DEBUG:boto:Token: None 2022-09-12T20:12:21.103 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:21.103 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1B3Hbf+74WzIqsVXGx5OmU0gNo0= 2022-09-12T20:12:21.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1B3Hbf+74WzIqsVXGx5OmU0gNo0=', 'Content-Length': '0'} 2022-09-12T20:12:21.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000691471ec5636663e-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.106 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:21.107 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:21.107 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:21.108 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:21.108 DEBUG:boto:Method: HEAD 2022-09-12T20:12:21.109 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:21.109 DEBUG:boto:Data: 2022-09-12T20:12:21.110 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.110 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.111 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.111 DEBUG:boto:Params: {} 2022-09-12T20:12:21.112 DEBUG:boto:Token: None 2022-09-12T20:12:21.113 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:21.113 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1B3Hbf+74WzIqsVXGx5OmU0gNo0= 2022-09-12T20:12:21.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1B3Hbf+74WzIqsVXGx5OmU0gNo0=', 'Content-Length': '0'} 2022-09-12T20:12:21.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c632ad43b952fbcc-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:12:21.117 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:21.117 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:21.118 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:21.118 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:21.119 DEBUG:boto:Method: GET 2022-09-12T20:12:21.120 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:21.120 DEBUG:boto:Data: 2022-09-12T20:12:21.121 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.121 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.122 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.122 DEBUG:boto:Params: {} 2022-09-12T20:12:21.123 DEBUG:boto:Token: None 2022-09-12T20:12:21.123 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:21.124 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0mZAOPqW8KRqHay8IKXqxTAyPmE= 2022-09-12T20:12:21.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0mZAOPqW8KRqHay8IKXqxTAyPmE=', 'Content-Length': '0'} 2022-09-12T20:12:21.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062a62d138d1baf91-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.127 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:21.128 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:21.128 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:21.129 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:12:21.129 DEBUG:boto:Method: GET 2022-09-12T20:12:21.130 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:21.131 DEBUG:boto:Data: 2022-09-12T20:12:21.131 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.132 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.132 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.133 DEBUG:boto:Params: {} 2022-09-12T20:12:21.133 DEBUG:boto:Token: None 2022-09-12T20:12:21.134 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:12:21.134 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0mZAOPqW8KRqHay8IKXqxTAyPmE= 2022-09-12T20:12:21.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0mZAOPqW8KRqHay8IKXqxTAyPmE=', 'Content-Length': '0'} 2022-09-12T20:12:21.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fceca223b3631ffa-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.138 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:21.138 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:21.139 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:12:21.139 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:12:21.140 DEBUG:boto:Method: GET 2022-09-12T20:12:21.140 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:12:21.141 DEBUG:boto:Data: 2022-09-12T20:12:21.141 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.142 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.142 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.143 DEBUG:boto:Params: {} 2022-09-12T20:12:21.143 DEBUG:boto:Token: None 2022-09-12T20:12:21.144 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/ 2022-09-12T20:12:21.145 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA= 2022-09-12T20:12:21.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA=', 'Content-Length': '0'} 2022-09-12T20:12:21.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7e83ecc61478608-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.149 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:21.150 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:21.150 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:21.151 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:12:21.151 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:12:21.152 DEBUG:boto:Method: GET 2022-09-12T20:12:21.152 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:12:21.153 DEBUG:boto:Data: 2022-09-12T20:12:21.153 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.154 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.155 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.155 DEBUG:boto:Params: {} 2022-09-12T20:12:21.156 DEBUG:boto:Token: None 2022-09-12T20:12:21.156 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/ 2022-09-12T20:12:21.157 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA= 2022-09-12T20:12:21.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA=', 'Content-Length': '0'} 2022-09-12T20:12:21.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000189b7096a78c6d3-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.161 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:21.162 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:21.162 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:21.163 DEBUG:boto:Method: HEAD 2022-09-12T20:12:21.163 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:12:21.164 DEBUG:boto:Data: 2022-09-12T20:12:21.164 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.165 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.165 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.166 DEBUG:boto:Params: {} 2022-09-12T20:12:21.166 DEBUG:boto:Token: None 2022-09-12T20:12:21.167 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj44 2022-09-12T20:12:21.167 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zErrQzqx3cWUAPirYFN+gHPrlSk= 2022-09-12T20:12:21.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zErrQzqx3cWUAPirYFN+gHPrlSk=', 'Content-Length': '0'} 2022-09-12T20:12:21.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c42f8dd88b5823ce-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.171 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:21.171 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:21.172 DEBUG:boto:Method: HEAD 2022-09-12T20:12:21.172 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:12:21.173 DEBUG:boto:Data: 2022-09-12T20:12:21.173 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.174 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.174 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.175 DEBUG:boto:Params: {} 2022-09-12T20:12:21.175 DEBUG:boto:Token: None 2022-09-12T20:12:21.176 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj44 2022-09-12T20:12:21.176 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zErrQzqx3cWUAPirYFN+gHPrlSk= 2022-09-12T20:12:21.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zErrQzqx3cWUAPirYFN+gHPrlSk=', 'Content-Length': '0'} 2022-09-12T20:12:21.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8799e6d8f3da528-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:12:21.180 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:21.181 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:21.181 DEBUG:boto:Method: GET 2022-09-12T20:12:21.182 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:12:21.182 DEBUG:boto:Data: 2022-09-12T20:12:21.183 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.183 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.184 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.184 DEBUG:boto:Params: {} 2022-09-12T20:12:21.185 DEBUG:boto:Token: None 2022-09-12T20:12:21.186 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj44 2022-09-12T20:12:21.186 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:COM7rPLjj7/zwRNQSiE6akZbRG8= 2022-09-12T20:12:21.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:COM7rPLjj7/zwRNQSiE6akZbRG8=', 'Content-Length': '0'} 2022-09-12T20:12:21.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c3618e836633fd4-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.230 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:12:21.230 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:12:21.231 DEBUG:boto:Method: GET 2022-09-12T20:12:21.231 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:12:21.232 DEBUG:boto:Data: 2022-09-12T20:12:21.233 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.233 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.234 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.234 DEBUG:boto:Params: {} 2022-09-12T20:12:21.235 DEBUG:boto:Token: None 2022-09-12T20:12:21.235 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj44 2022-09-12T20:12:21.236 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:COM7rPLjj7/zwRNQSiE6akZbRG8= 2022-09-12T20:12:21.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:COM7rPLjj7/zwRNQSiE6akZbRG8=', 'Content-Length': '0'} 2022-09-12T20:12:21.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d28c3e90a0916fb9-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.239 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:21.240 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:21.240 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:12:21.241 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:12:21.241 DEBUG:boto:Method: GET 2022-09-12T20:12:21.242 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:12:21.242 DEBUG:boto:Data: 2022-09-12T20:12:21.243 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.243 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.244 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.244 DEBUG:boto:Params: {} 2022-09-12T20:12:21.245 DEBUG:boto:Token: None 2022-09-12T20:12:21.246 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/ 2022-09-12T20:12:21.246 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA= 2022-09-12T20:12:21.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA=', 'Content-Length': '0'} 2022-09-12T20:12:21.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4cb869d68bb4f42-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.250 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:21.251 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:21.252 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:21.252 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:12:21.253 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:12:21.253 DEBUG:boto:Method: GET 2022-09-12T20:12:21.254 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:12:21.254 DEBUG:boto:Data: 2022-09-12T20:12:21.255 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.255 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.256 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.256 DEBUG:boto:Params: {} 2022-09-12T20:12:21.257 DEBUG:boto:Token: None 2022-09-12T20:12:21.258 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/ 2022-09-12T20:12:21.258 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA= 2022-09-12T20:12:21.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA=', 'Content-Length': '0'} 2022-09-12T20:12:21.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ffbe9cba7755b6d-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.262 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:21.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:12:21.263 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:21.264 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:21.265 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:21.265 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:21.266 DEBUG:boto:Method: GET 2022-09-12T20:12:21.266 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:21.267 DEBUG:boto:Data: 2022-09-12T20:12:21.267 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.268 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.268 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.269 DEBUG:boto:Params: {} 2022-09-12T20:12:21.269 DEBUG:boto:Token: None 2022-09-12T20:12:21.270 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:21.270 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IIX3a0xaUIwVPueiDQXDTyiCtLk= 2022-09-12T20:12:21.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IIX3a0xaUIwVPueiDQXDTyiCtLk=', 'Content-Length': '0'} 2022-09-12T20:12:21.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ad8f09945d4760f-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.274 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:21.274 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:21.275 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:21.275 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:21.276 DEBUG:boto:Method: GET 2022-09-12T20:12:21.277 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:21.277 DEBUG:boto:Data: 2022-09-12T20:12:21.278 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.278 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.279 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.279 DEBUG:boto:Params: {} 2022-09-12T20:12:21.280 DEBUG:boto:Token: None 2022-09-12T20:12:21.280 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:21.281 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IIX3a0xaUIwVPueiDQXDTyiCtLk= 2022-09-12T20:12:21.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IIX3a0xaUIwVPueiDQXDTyiCtLk=', 'Content-Length': '0'} 2022-09-12T20:12:21.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041b37a1d555f6d5e-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.284 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:21.285 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:21.285 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:21.286 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:21.286 DEBUG:boto:Method: HEAD 2022-09-12T20:12:21.287 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:21.287 DEBUG:boto:Data: 2022-09-12T20:12:21.288 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.288 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.289 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.289 DEBUG:boto:Params: {} 2022-09-12T20:12:21.290 DEBUG:boto:Token: None 2022-09-12T20:12:21.290 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:21.291 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NILQVZp+b8feCMJq0yEhXjcQiIA= 2022-09-12T20:12:21.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NILQVZp+b8feCMJq0yEhXjcQiIA=', 'Content-Length': '0'} 2022-09-12T20:12:21.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de32514cfa295ba9-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.294 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:21.295 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:21.295 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:21.296 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:21.296 DEBUG:boto:Method: HEAD 2022-09-12T20:12:21.297 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:21.297 DEBUG:boto:Data: 2022-09-12T20:12:21.298 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.298 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.299 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.299 DEBUG:boto:Params: {} 2022-09-12T20:12:21.300 DEBUG:boto:Token: None 2022-09-12T20:12:21.301 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:21.301 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NILQVZp+b8feCMJq0yEhXjcQiIA= 2022-09-12T20:12:21.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NILQVZp+b8feCMJq0yEhXjcQiIA=', 'Content-Length': '0'} 2022-09-12T20:12:21.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb3859a4742ceadf-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:12:21.305 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:21.305 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:21.306 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:21.306 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:21.307 DEBUG:boto:Method: GET 2022-09-12T20:12:21.307 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:21.308 DEBUG:boto:Data: 2022-09-12T20:12:21.308 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.309 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.309 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.310 DEBUG:boto:Params: {} 2022-09-12T20:12:21.310 DEBUG:boto:Token: None 2022-09-12T20:12:21.311 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:21.312 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IIX3a0xaUIwVPueiDQXDTyiCtLk= 2022-09-12T20:12:21.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IIX3a0xaUIwVPueiDQXDTyiCtLk=', 'Content-Length': '0'} 2022-09-12T20:12:21.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7ce90c755cfc07d-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.315 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:21.315 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:21.316 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:21.317 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:12:21.317 DEBUG:boto:Method: GET 2022-09-12T20:12:21.318 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:21.318 DEBUG:boto:Data: 2022-09-12T20:12:21.319 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.319 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.320 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.320 DEBUG:boto:Params: {} 2022-09-12T20:12:21.321 DEBUG:boto:Token: None 2022-09-12T20:12:21.321 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:12:21.322 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IIX3a0xaUIwVPueiDQXDTyiCtLk= 2022-09-12T20:12:21.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IIX3a0xaUIwVPueiDQXDTyiCtLk=', 'Content-Length': '0'} 2022-09-12T20:12:21.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028d3787783c6a637-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.325 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:21.326 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:21.326 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:12:21.327 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:12:21.327 DEBUG:boto:Method: GET 2022-09-12T20:12:21.328 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:12:21.328 DEBUG:boto:Data: 2022-09-12T20:12:21.329 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.330 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.330 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.331 DEBUG:boto:Params: {} 2022-09-12T20:12:21.331 DEBUG:boto:Token: None 2022-09-12T20:12:21.332 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/ 2022-09-12T20:12:21.332 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA= 2022-09-12T20:12:21.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA=', 'Content-Length': '0'} 2022-09-12T20:12:21.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bffde65c6c53c53-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.378 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:21.379 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:21.379 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:21.380 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:12:21.380 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:12:21.381 DEBUG:boto:Method: GET 2022-09-12T20:12:21.381 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:12:21.382 DEBUG:boto:Data: 2022-09-12T20:12:21.382 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.383 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.384 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.384 DEBUG:boto:Params: {} 2022-09-12T20:12:21.385 DEBUG:boto:Token: None 2022-09-12T20:12:21.385 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/ 2022-09-12T20:12:21.386 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA= 2022-09-12T20:12:21.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA=', 'Content-Length': '0'} 2022-09-12T20:12:21.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003110171aea7a346d-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.390 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:21.390 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:21.391 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:21.391 DEBUG:boto:Method: HEAD 2022-09-12T20:12:21.392 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:12:21.392 DEBUG:boto:Data: 2022-09-12T20:12:21.393 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.393 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.394 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.394 DEBUG:boto:Params: {} 2022-09-12T20:12:21.395 DEBUG:boto:Token: None 2022-09-12T20:12:21.395 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj46 2022-09-12T20:12:21.396 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:o/KIkMCEdWarFYJI26/CzaLKVBk= 2022-09-12T20:12:21.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:o/KIkMCEdWarFYJI26/CzaLKVBk=', 'Content-Length': '0'} 2022-09-12T20:12:21.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f9fd4ccce18bbef-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.399 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:21.400 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:21.400 DEBUG:boto:Method: HEAD 2022-09-12T20:12:21.401 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:12:21.401 DEBUG:boto:Data: 2022-09-12T20:12:21.402 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.402 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.403 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.403 DEBUG:boto:Params: {} 2022-09-12T20:12:21.404 DEBUG:boto:Token: None 2022-09-12T20:12:21.404 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj46 2022-09-12T20:12:21.405 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:o/KIkMCEdWarFYJI26/CzaLKVBk= 2022-09-12T20:12:21.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:o/KIkMCEdWarFYJI26/CzaLKVBk=', 'Content-Length': '0'} 2022-09-12T20:12:21.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5a7570823ec0e6c-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:12:21.408 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:21.409 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:21.409 DEBUG:boto:Method: GET 2022-09-12T20:12:21.410 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:12:21.411 DEBUG:boto:Data: 2022-09-12T20:12:21.411 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.412 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.412 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.413 DEBUG:boto:Params: {} 2022-09-12T20:12:21.413 DEBUG:boto:Token: None 2022-09-12T20:12:21.414 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj46 2022-09-12T20:12:21.414 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eNoO2u8QXwdrN8mTX9bn+KDhW7Q= 2022-09-12T20:12:21.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eNoO2u8QXwdrN8mTX9bn+KDhW7Q=', 'Content-Length': '0'} 2022-09-12T20:12:21.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000596d263eb420ad89-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.418 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:12:21.418 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:12:21.419 DEBUG:boto:Method: GET 2022-09-12T20:12:21.419 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:12:21.420 DEBUG:boto:Data: 2022-09-12T20:12:21.420 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.421 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.421 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.422 DEBUG:boto:Params: {} 2022-09-12T20:12:21.422 DEBUG:boto:Token: None 2022-09-12T20:12:21.423 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj46 2022-09-12T20:12:21.424 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eNoO2u8QXwdrN8mTX9bn+KDhW7Q= 2022-09-12T20:12:21.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eNoO2u8QXwdrN8mTX9bn+KDhW7Q=', 'Content-Length': '0'} 2022-09-12T20:12:21.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cde2a1aa7c405e75-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.427 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:21.427 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:21.428 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:12:21.428 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:12:21.429 DEBUG:boto:Method: GET 2022-09-12T20:12:21.429 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:12:21.430 DEBUG:boto:Data: 2022-09-12T20:12:21.430 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.431 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.431 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.432 DEBUG:boto:Params: {} 2022-09-12T20:12:21.433 DEBUG:boto:Token: None 2022-09-12T20:12:21.433 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/ 2022-09-12T20:12:21.434 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA= 2022-09-12T20:12:21.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA=', 'Content-Length': '0'} 2022-09-12T20:12:21.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b732fd5b0c659668-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.438 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:21.439 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:21.439 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:21.440 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:12:21.440 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:12:21.441 DEBUG:boto:Method: GET 2022-09-12T20:12:21.441 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:12:21.442 DEBUG:boto:Data: 2022-09-12T20:12:21.442 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.443 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.444 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.444 DEBUG:boto:Params: {} 2022-09-12T20:12:21.445 DEBUG:boto:Token: None 2022-09-12T20:12:21.445 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/ 2022-09-12T20:12:21.446 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA= 2022-09-12T20:12:21.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA=', 'Content-Length': '0'} 2022-09-12T20:12:21.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092ec9d888d04c534-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.490 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:21.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:12:21.491 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:21.492 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:21.492 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:21.493 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:21.493 DEBUG:boto:Method: GET 2022-09-12T20:12:21.494 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:21.494 DEBUG:boto:Data: 2022-09-12T20:12:21.495 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.495 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.496 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.496 DEBUG:boto:Params: {} 2022-09-12T20:12:21.497 DEBUG:boto:Token: None 2022-09-12T20:12:21.497 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:21.498 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SYpwIyX9BLOqSNvE0sSeOQkV0xg= 2022-09-12T20:12:21.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SYpwIyX9BLOqSNvE0sSeOQkV0xg=', 'Content-Length': '0'} 2022-09-12T20:12:21.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008040c92b39ad7686-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.501 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:21.502 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:21.502 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:21.503 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:21.504 DEBUG:boto:Method: GET 2022-09-12T20:12:21.504 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:21.505 DEBUG:boto:Data: 2022-09-12T20:12:21.505 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.506 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.506 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.507 DEBUG:boto:Params: {} 2022-09-12T20:12:21.507 DEBUG:boto:Token: None 2022-09-12T20:12:21.508 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:21.508 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SYpwIyX9BLOqSNvE0sSeOQkV0xg= 2022-09-12T20:12:21.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SYpwIyX9BLOqSNvE0sSeOQkV0xg=', 'Content-Length': '0'} 2022-09-12T20:12:21.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a94242a5d548049-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.512 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:21.513 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:21.513 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:21.514 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:21.514 DEBUG:boto:Method: HEAD 2022-09-12T20:12:21.515 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:21.515 DEBUG:boto:Data: 2022-09-12T20:12:21.521 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.522 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.522 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.523 DEBUG:boto:Params: {} 2022-09-12T20:12:21.524 DEBUG:boto:Token: None 2022-09-12T20:12:21.524 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:21.525 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bqBRLZ1rzw2p6fWzCluRbGOQSY8= 2022-09-12T20:12:21.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bqBRLZ1rzw2p6fWzCluRbGOQSY8=', 'Content-Length': '0'} 2022-09-12T20:12:21.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067c2de0ab5f54135-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.528 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:21.529 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:21.529 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:21.530 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:21.530 DEBUG:boto:Method: HEAD 2022-09-12T20:12:21.531 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:21.531 DEBUG:boto:Data: 2022-09-12T20:12:21.532 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.532 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.533 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.533 DEBUG:boto:Params: {} 2022-09-12T20:12:21.534 DEBUG:boto:Token: None 2022-09-12T20:12:21.534 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:21.535 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bqBRLZ1rzw2p6fWzCluRbGOQSY8= 2022-09-12T20:12:21.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bqBRLZ1rzw2p6fWzCluRbGOQSY8=', 'Content-Length': '0'} 2022-09-12T20:12:21.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4d2adedf3eacda5-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:12:21.539 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:21.539 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:21.540 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:21.540 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:21.541 DEBUG:boto:Method: GET 2022-09-12T20:12:21.541 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:21.542 DEBUG:boto:Data: 2022-09-12T20:12:21.542 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.543 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.543 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.544 DEBUG:boto:Params: {} 2022-09-12T20:12:21.544 DEBUG:boto:Token: None 2022-09-12T20:12:21.545 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:21.546 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SYpwIyX9BLOqSNvE0sSeOQkV0xg= 2022-09-12T20:12:21.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SYpwIyX9BLOqSNvE0sSeOQkV0xg=', 'Content-Length': '0'} 2022-09-12T20:12:21.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043b00dc527f115f8-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.549 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:21.550 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:21.550 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:21.551 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:12:21.551 DEBUG:boto:Method: GET 2022-09-12T20:12:21.552 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:21.552 DEBUG:boto:Data: 2022-09-12T20:12:21.553 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.553 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.554 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.554 DEBUG:boto:Params: {} 2022-09-12T20:12:21.555 DEBUG:boto:Token: None 2022-09-12T20:12:21.555 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:12:21.556 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SYpwIyX9BLOqSNvE0sSeOQkV0xg= 2022-09-12T20:12:21.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SYpwIyX9BLOqSNvE0sSeOQkV0xg=', 'Content-Length': '0'} 2022-09-12T20:12:21.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b131606b7ca60db-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.560 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:21.560 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:21.561 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:12:21.561 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:12:21.562 DEBUG:boto:Method: GET 2022-09-12T20:12:21.562 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:12:21.563 DEBUG:boto:Data: 2022-09-12T20:12:21.563 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.564 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.564 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.565 DEBUG:boto:Params: {} 2022-09-12T20:12:21.565 DEBUG:boto:Token: None 2022-09-12T20:12:21.566 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/ 2022-09-12T20:12:21.566 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA= 2022-09-12T20:12:21.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA=', 'Content-Length': '0'} 2022-09-12T20:12:21.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bdbd9170da5f10f-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.571 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:21.571 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:21.572 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:21.572 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:12:21.573 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:12:21.573 DEBUG:boto:Method: GET 2022-09-12T20:12:21.574 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:12:21.575 DEBUG:boto:Data: 2022-09-12T20:12:21.575 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.576 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.576 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.577 DEBUG:boto:Params: {} 2022-09-12T20:12:21.577 DEBUG:boto:Token: None 2022-09-12T20:12:21.578 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/ 2022-09-12T20:12:21.578 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA= 2022-09-12T20:12:21.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA=', 'Content-Length': '0'} 2022-09-12T20:12:21.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a543dd7f7ff8fc10-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.582 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:21.583 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:21.584 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:21.584 DEBUG:boto:Method: HEAD 2022-09-12T20:12:21.585 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:12:21.585 DEBUG:boto:Data: 2022-09-12T20:12:21.586 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.586 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.587 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.587 DEBUG:boto:Params: {} 2022-09-12T20:12:21.588 DEBUG:boto:Token: None 2022-09-12T20:12:21.588 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj51 2022-09-12T20:12:21.589 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ehK0wHVGwj1UJbJD/X+16DZteSU= 2022-09-12T20:12:21.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ehK0wHVGwj1UJbJD/X+16DZteSU=', 'Content-Length': '0'} 2022-09-12T20:12:21.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9958d28ac7a9aff-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.592 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:21.593 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:21.593 DEBUG:boto:Method: HEAD 2022-09-12T20:12:21.594 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:12:21.594 DEBUG:boto:Data: 2022-09-12T20:12:21.595 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.595 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.596 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.596 DEBUG:boto:Params: {} 2022-09-12T20:12:21.597 DEBUG:boto:Token: None 2022-09-12T20:12:21.597 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj51 2022-09-12T20:12:21.598 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ehK0wHVGwj1UJbJD/X+16DZteSU= 2022-09-12T20:12:21.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ehK0wHVGwj1UJbJD/X+16DZteSU=', 'Content-Length': '0'} 2022-09-12T20:12:21.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008544e963495b73db-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:12:21.602 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:21.602 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:21.603 DEBUG:boto:Method: GET 2022-09-12T20:12:21.603 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:12:21.604 DEBUG:boto:Data: 2022-09-12T20:12:21.604 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.605 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.605 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.606 DEBUG:boto:Params: {} 2022-09-12T20:12:21.606 DEBUG:boto:Token: None 2022-09-12T20:12:21.607 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj51 2022-09-12T20:12:21.608 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qqzltBXtEAXyzg7f6j+WYNOaDwc= 2022-09-12T20:12:21.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qqzltBXtEAXyzg7f6j+WYNOaDwc=', 'Content-Length': '0'} 2022-09-12T20:12:21.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e03a8488c2e6dba-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.654 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:12:21.655 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:12:21.655 DEBUG:boto:Method: GET 2022-09-12T20:12:21.656 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:12:21.656 DEBUG:boto:Data: 2022-09-12T20:12:21.657 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.657 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.658 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.658 DEBUG:boto:Params: {} 2022-09-12T20:12:21.659 DEBUG:boto:Token: None 2022-09-12T20:12:21.660 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj51 2022-09-12T20:12:21.660 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qqzltBXtEAXyzg7f6j+WYNOaDwc= 2022-09-12T20:12:21.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qqzltBXtEAXyzg7f6j+WYNOaDwc=', 'Content-Length': '0'} 2022-09-12T20:12:21.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d055042ab9b859f7-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.664 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:21.664 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:21.665 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:12:21.665 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:12:21.666 DEBUG:boto:Method: GET 2022-09-12T20:12:21.666 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:12:21.667 DEBUG:boto:Data: 2022-09-12T20:12:21.667 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.668 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.668 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.669 DEBUG:boto:Params: {} 2022-09-12T20:12:21.669 DEBUG:boto:Token: None 2022-09-12T20:12:21.670 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/ 2022-09-12T20:12:21.670 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA= 2022-09-12T20:12:21.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA=', 'Content-Length': '0'} 2022-09-12T20:12:21.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ed3e41bd6fbabf5-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.675 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:21.676 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:21.676 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:21.677 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:12:21.677 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:12:21.678 DEBUG:boto:Method: GET 2022-09-12T20:12:21.678 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:12:21.679 DEBUG:boto:Data: 2022-09-12T20:12:21.679 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.680 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.680 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.681 DEBUG:boto:Params: {} 2022-09-12T20:12:21.681 DEBUG:boto:Token: None 2022-09-12T20:12:21.682 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/ 2022-09-12T20:12:21.682 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA= 2022-09-12T20:12:21.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA=', 'Content-Length': '0'} 2022-09-12T20:12:21.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058ac1eadb57549d9-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.686 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:21.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:12:21.687 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:21.688 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:21.689 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:21.689 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:21.690 DEBUG:boto:Method: GET 2022-09-12T20:12:21.690 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:21.691 DEBUG:boto:Data: 2022-09-12T20:12:21.691 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.692 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.692 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.693 DEBUG:boto:Params: {} 2022-09-12T20:12:21.693 DEBUG:boto:Token: None 2022-09-12T20:12:21.694 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:21.694 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GChLTu1QsHn8MgbYEFKAdKUfmaQ= 2022-09-12T20:12:21.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GChLTu1QsHn8MgbYEFKAdKUfmaQ=', 'Content-Length': '0'} 2022-09-12T20:12:21.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebce3e8933a1f29c-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.698 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:21.698 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:21.699 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:21.699 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:21.700 DEBUG:boto:Method: GET 2022-09-12T20:12:21.700 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:21.701 DEBUG:boto:Data: 2022-09-12T20:12:21.701 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.702 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.702 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.703 DEBUG:boto:Params: {} 2022-09-12T20:12:21.703 DEBUG:boto:Token: None 2022-09-12T20:12:21.704 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:21.705 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GChLTu1QsHn8MgbYEFKAdKUfmaQ= 2022-09-12T20:12:21.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GChLTu1QsHn8MgbYEFKAdKUfmaQ=', 'Content-Length': '0'} 2022-09-12T20:12:21.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4fed9d4fb266189-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.708 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:21.709 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:21.709 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:21.710 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:21.710 DEBUG:boto:Method: HEAD 2022-09-12T20:12:21.711 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:21.711 DEBUG:boto:Data: 2022-09-12T20:12:21.712 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.712 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.712 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.713 DEBUG:boto:Params: {} 2022-09-12T20:12:21.713 DEBUG:boto:Token: None 2022-09-12T20:12:21.714 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:21.715 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xw0TUy084LMyqdLWxHnQ8Ssi4hA= 2022-09-12T20:12:21.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xw0TUy084LMyqdLWxHnQ8Ssi4hA=', 'Content-Length': '0'} 2022-09-12T20:12:21.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000beff75486eded18f-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.718 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:21.718 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:21.719 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:21.719 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:21.720 DEBUG:boto:Method: HEAD 2022-09-12T20:12:21.720 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:21.721 DEBUG:boto:Data: 2022-09-12T20:12:21.721 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.722 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.722 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.723 DEBUG:boto:Params: {} 2022-09-12T20:12:21.723 DEBUG:boto:Token: None 2022-09-12T20:12:21.724 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:21.724 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xw0TUy084LMyqdLWxHnQ8Ssi4hA= 2022-09-12T20:12:21.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xw0TUy084LMyqdLWxHnQ8Ssi4hA=', 'Content-Length': '0'} 2022-09-12T20:12:21.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef153a02f8d6153d-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:12:21.728 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:21.729 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:21.729 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:21.730 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:21.730 DEBUG:boto:Method: GET 2022-09-12T20:12:21.731 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:21.731 DEBUG:boto:Data: 2022-09-12T20:12:21.732 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.732 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.733 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.733 DEBUG:boto:Params: {} 2022-09-12T20:12:21.734 DEBUG:boto:Token: None 2022-09-12T20:12:21.735 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:21.735 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GChLTu1QsHn8MgbYEFKAdKUfmaQ= 2022-09-12T20:12:21.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GChLTu1QsHn8MgbYEFKAdKUfmaQ=', 'Content-Length': '0'} 2022-09-12T20:12:21.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018be584c0e34c4ff-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.739 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:21.739 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:21.740 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:21.740 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:12:21.741 DEBUG:boto:Method: GET 2022-09-12T20:12:21.741 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:21.742 DEBUG:boto:Data: 2022-09-12T20:12:21.742 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.743 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.743 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.744 DEBUG:boto:Params: {} 2022-09-12T20:12:21.744 DEBUG:boto:Token: None 2022-09-12T20:12:21.745 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:12:21.746 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GChLTu1QsHn8MgbYEFKAdKUfmaQ= 2022-09-12T20:12:21.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GChLTu1QsHn8MgbYEFKAdKUfmaQ=', 'Content-Length': '0'} 2022-09-12T20:12:21.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009da8046d8cd7ff0e-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.749 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:21.749 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:21.750 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:12:21.750 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:12:21.751 DEBUG:boto:Method: GET 2022-09-12T20:12:21.752 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:12:21.752 DEBUG:boto:Data: 2022-09-12T20:12:21.753 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.753 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.754 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.754 DEBUG:boto:Params: {} 2022-09-12T20:12:21.755 DEBUG:boto:Token: None 2022-09-12T20:12:21.755 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/ 2022-09-12T20:12:21.756 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA= 2022-09-12T20:12:21.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA=', 'Content-Length': '0'} 2022-09-12T20:12:21.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000408782efff3105f-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.802 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:21.803 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:21.803 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:21.804 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:12:21.804 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:12:21.805 DEBUG:boto:Method: GET 2022-09-12T20:12:21.806 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:12:21.806 DEBUG:boto:Data: 2022-09-12T20:12:21.807 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.807 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.808 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.808 DEBUG:boto:Params: {} 2022-09-12T20:12:21.809 DEBUG:boto:Token: None 2022-09-12T20:12:21.809 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/ 2022-09-12T20:12:21.810 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA= 2022-09-12T20:12:21.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA=', 'Content-Length': '0'} 2022-09-12T20:12:21.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6c756e8571be7f9-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.814 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:21.815 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:21.815 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:21.816 DEBUG:boto:Method: HEAD 2022-09-12T20:12:21.816 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:12:21.817 DEBUG:boto:Data: 2022-09-12T20:12:21.818 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.818 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.819 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.819 DEBUG:boto:Params: {} 2022-09-12T20:12:21.820 DEBUG:boto:Token: None 2022-09-12T20:12:21.820 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj55 2022-09-12T20:12:21.821 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ozXrxLVKxIkahnb5l55OjmCpoaI= 2022-09-12T20:12:21.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ozXrxLVKxIkahnb5l55OjmCpoaI=', 'Content-Length': '0'} 2022-09-12T20:12:21.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7025351975192f7-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.824 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:21.825 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:21.825 DEBUG:boto:Method: HEAD 2022-09-12T20:12:21.826 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:12:21.826 DEBUG:boto:Data: 2022-09-12T20:12:21.827 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.827 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.828 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.828 DEBUG:boto:Params: {} 2022-09-12T20:12:21.829 DEBUG:boto:Token: None 2022-09-12T20:12:21.829 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj55 2022-09-12T20:12:21.830 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ozXrxLVKxIkahnb5l55OjmCpoaI= 2022-09-12T20:12:21.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ozXrxLVKxIkahnb5l55OjmCpoaI=', 'Content-Length': '0'} 2022-09-12T20:12:21.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c65c2cf401a9367c-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:12:21.834 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:21.834 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:21.835 DEBUG:boto:Method: GET 2022-09-12T20:12:21.835 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:12:21.836 DEBUG:boto:Data: 2022-09-12T20:12:21.836 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.837 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.837 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.838 DEBUG:boto:Params: {} 2022-09-12T20:12:21.839 DEBUG:boto:Token: None 2022-09-12T20:12:21.839 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj55 2022-09-12T20:12:21.840 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CQBiUgk04iHQFRVrQQ7B3N1Y3Dc= 2022-09-12T20:12:21.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CQBiUgk04iHQFRVrQQ7B3N1Y3Dc=', 'Content-Length': '0'} 2022-09-12T20:12:21.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e4d57637613e00d-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.843 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:12:21.844 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:12:21.844 DEBUG:boto:Method: GET 2022-09-12T20:12:21.845 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:12:21.845 DEBUG:boto:Data: 2022-09-12T20:12:21.846 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.846 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.847 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.847 DEBUG:boto:Params: {} 2022-09-12T20:12:21.848 DEBUG:boto:Token: None 2022-09-12T20:12:21.848 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj55 2022-09-12T20:12:21.849 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CQBiUgk04iHQFRVrQQ7B3N1Y3Dc= 2022-09-12T20:12:21.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CQBiUgk04iHQFRVrQQ7B3N1Y3Dc=', 'Content-Length': '0'} 2022-09-12T20:12:21.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dce4828ac8de503f-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.852 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:21.853 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:21.853 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:12:21.854 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:12:21.854 DEBUG:boto:Method: GET 2022-09-12T20:12:21.855 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:12:21.855 DEBUG:boto:Data: 2022-09-12T20:12:21.856 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.856 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.857 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.857 DEBUG:boto:Params: {} 2022-09-12T20:12:21.858 DEBUG:boto:Token: None 2022-09-12T20:12:21.859 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/ 2022-09-12T20:12:21.859 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA= 2022-09-12T20:12:21.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA=', 'Content-Length': '0'} 2022-09-12T20:12:21.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fa09b5393e9a6d8-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.863 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:21.864 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:21.865 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:21.865 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:12:21.866 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:12:21.866 DEBUG:boto:Method: GET 2022-09-12T20:12:21.867 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:12:21.867 DEBUG:boto:Data: 2022-09-12T20:12:21.868 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.868 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.869 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.869 DEBUG:boto:Params: {} 2022-09-12T20:12:21.870 DEBUG:boto:Token: None 2022-09-12T20:12:21.870 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/ 2022-09-12T20:12:21.871 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA= 2022-09-12T20:12:21.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA=', 'Content-Length': '0'} 2022-09-12T20:12:21.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e84fb07a6e5b7356-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.875 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:21.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:12:21.876 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:21.877 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:21.877 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:21.878 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:21.878 DEBUG:boto:Method: GET 2022-09-12T20:12:21.879 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:21.879 DEBUG:boto:Data: 2022-09-12T20:12:21.880 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.880 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.881 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.881 DEBUG:boto:Params: {} 2022-09-12T20:12:21.882 DEBUG:boto:Token: None 2022-09-12T20:12:21.882 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:21.883 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D44HOQZ3XyszWIY4GA0L/Kto7lI= 2022-09-12T20:12:21.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D44HOQZ3XyszWIY4GA0L/Kto7lI=', 'Content-Length': '0'} 2022-09-12T20:12:21.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf02c939a3c9a9e2-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.926 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:21.927 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:21.927 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:21.928 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:21.928 DEBUG:boto:Method: GET 2022-09-12T20:12:21.929 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:21.929 DEBUG:boto:Data: 2022-09-12T20:12:21.930 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.930 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.931 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.931 DEBUG:boto:Params: {} 2022-09-12T20:12:21.932 DEBUG:boto:Token: None 2022-09-12T20:12:21.932 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:21.933 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D44HOQZ3XyszWIY4GA0L/Kto7lI= 2022-09-12T20:12:21.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D44HOQZ3XyszWIY4GA0L/Kto7lI=', 'Content-Length': '0'} 2022-09-12T20:12:21.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003063a169786659ed-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.937 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:21.937 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:21.938 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:21.938 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:21.939 DEBUG:boto:Method: HEAD 2022-09-12T20:12:21.939 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:21.940 DEBUG:boto:Data: 2022-09-12T20:12:21.940 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.941 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.941 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.942 DEBUG:boto:Params: {} 2022-09-12T20:12:21.942 DEBUG:boto:Token: None 2022-09-12T20:12:21.943 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:21.943 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pldoUTDADiW+wrQjXQLO9ay2JYs= 2022-09-12T20:12:21.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pldoUTDADiW+wrQjXQLO9ay2JYs=', 'Content-Length': '0'} 2022-09-12T20:12:21.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7315003b036fcd9-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.947 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:21.947 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:21.948 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:21.948 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:21.949 DEBUG:boto:Method: HEAD 2022-09-12T20:12:21.949 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:21.950 DEBUG:boto:Data: 2022-09-12T20:12:21.950 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.951 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.951 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.952 DEBUG:boto:Params: {} 2022-09-12T20:12:21.952 DEBUG:boto:Token: None 2022-09-12T20:12:21.953 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:21.953 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pldoUTDADiW+wrQjXQLO9ay2JYs= 2022-09-12T20:12:21.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pldoUTDADiW+wrQjXQLO9ay2JYs=', 'Content-Length': '0'} 2022-09-12T20:12:21.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fdffcbb7513b51a-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:12:21.957 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:21.958 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:21.958 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:21.959 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:21.959 DEBUG:boto:Method: GET 2022-09-12T20:12:21.960 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:21.960 DEBUG:boto:Data: 2022-09-12T20:12:21.961 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.961 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.962 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.962 DEBUG:boto:Params: {} 2022-09-12T20:12:21.963 DEBUG:boto:Token: None 2022-09-12T20:12:21.964 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:21.964 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D44HOQZ3XyszWIY4GA0L/Kto7lI= 2022-09-12T20:12:21.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D44HOQZ3XyszWIY4GA0L/Kto7lI=', 'Content-Length': '0'} 2022-09-12T20:12:21.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a343eb4af31d062-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.968 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:21.968 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:21.969 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:21.969 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:12:21.970 DEBUG:boto:Method: GET 2022-09-12T20:12:21.970 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:21.971 DEBUG:boto:Data: 2022-09-12T20:12:21.971 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.972 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:21.972 DEBUG:boto:Port: 8000 2022-09-12T20:12:21.973 DEBUG:boto:Params: {} 2022-09-12T20:12:21.973 DEBUG:boto:Token: None 2022-09-12T20:12:21.974 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:12:21.975 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D44HOQZ3XyszWIY4GA0L/Kto7lI= 2022-09-12T20:12:21.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D44HOQZ3XyszWIY4GA0L/Kto7lI=', 'Content-Length': '0'} 2022-09-12T20:12:21.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bf255d695fb3a0f-00631f92a5-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:21.978 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:21.978 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:21.979 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:12:21.979 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:12:21.980 DEBUG:boto:Method: GET 2022-09-12T20:12:21.980 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:12:21.981 DEBUG:boto:Data: 2022-09-12T20:12:21.981 DEBUG:boto:Headers: {} 2022-09-12T20:12:21.982 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:21.982 DEBUG:boto:Port: 8001 2022-09-12T20:12:21.983 DEBUG:boto:Params: {} 2022-09-12T20:12:21.984 DEBUG:boto:Token: None 2022-09-12T20:12:21.984 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:21 GMT /jdpiak-75/ 2022-09-12T20:12:21.985 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA= 2022-09-12T20:12:21.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9pROTZk9X6iEZhM7cemN3sFrZJA=', 'Content-Length': '0'} 2022-09-12T20:12:21.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080f1db8809425d1e-00631f92a5-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.030 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:22.031 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:22.031 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:22.032 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:12:22.032 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:12:22.033 DEBUG:boto:Method: GET 2022-09-12T20:12:22.033 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:12:22.034 DEBUG:boto:Data: 2022-09-12T20:12:22.035 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.035 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.036 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.036 DEBUG:boto:Params: {} 2022-09-12T20:12:22.037 DEBUG:boto:Token: None 2022-09-12T20:12:22.037 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/ 2022-09-12T20:12:22.038 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao= 2022-09-12T20:12:22.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao=', 'Content-Length': '0'} 2022-09-12T20:12:22.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f61950e397b99ad4-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.042 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:22.043 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:22.043 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:22.044 DEBUG:boto:Method: HEAD 2022-09-12T20:12:22.044 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:12:22.045 DEBUG:boto:Data: 2022-09-12T20:12:22.045 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.046 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.046 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.047 DEBUG:boto:Params: {} 2022-09-12T20:12:22.047 DEBUG:boto:Token: None 2022-09-12T20:12:22.048 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj57 2022-09-12T20:12:22.048 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m35edrbCznZf7cSfGTMpm4/6Iec= 2022-09-12T20:12:22.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m35edrbCznZf7cSfGTMpm4/6Iec=', 'Content-Length': '0'} 2022-09-12T20:12:22.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d7cb1edd51898b1-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.052 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:22.052 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:22.053 DEBUG:boto:Method: HEAD 2022-09-12T20:12:22.053 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:12:22.054 DEBUG:boto:Data: 2022-09-12T20:12:22.054 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.055 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.055 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.056 DEBUG:boto:Params: {} 2022-09-12T20:12:22.056 DEBUG:boto:Token: None 2022-09-12T20:12:22.057 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj57 2022-09-12T20:12:22.057 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m35edrbCznZf7cSfGTMpm4/6Iec= 2022-09-12T20:12:22.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m35edrbCznZf7cSfGTMpm4/6Iec=', 'Content-Length': '0'} 2022-09-12T20:12:22.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d79c5a035555650-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:12:22.061 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:22.061 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:22.062 DEBUG:boto:Method: GET 2022-09-12T20:12:22.063 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:12:22.063 DEBUG:boto:Data: 2022-09-12T20:12:22.064 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.064 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.065 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.065 DEBUG:boto:Params: {} 2022-09-12T20:12:22.066 DEBUG:boto:Token: None 2022-09-12T20:12:22.066 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj57 2022-09-12T20:12:22.067 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uhPqrxFPosi2eHq2X7rI4uM/Juc= 2022-09-12T20:12:22.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uhPqrxFPosi2eHq2X7rI4uM/Juc=', 'Content-Length': '0'} 2022-09-12T20:12:22.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000403e7e138c95b4a1-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.071 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:12:22.071 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:12:22.072 DEBUG:boto:Method: GET 2022-09-12T20:12:22.072 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:12:22.073 DEBUG:boto:Data: 2022-09-12T20:12:22.073 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.074 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.074 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.075 DEBUG:boto:Params: {} 2022-09-12T20:12:22.075 DEBUG:boto:Token: None 2022-09-12T20:12:22.076 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj57 2022-09-12T20:12:22.077 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uhPqrxFPosi2eHq2X7rI4uM/Juc= 2022-09-12T20:12:22.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uhPqrxFPosi2eHq2X7rI4uM/Juc=', 'Content-Length': '0'} 2022-09-12T20:12:22.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003091c737a315b1a8-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.080 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:22.081 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:22.081 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:12:22.082 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:12:22.082 DEBUG:boto:Method: GET 2022-09-12T20:12:22.083 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:12:22.083 DEBUG:boto:Data: 2022-09-12T20:12:22.084 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.084 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.085 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.085 DEBUG:boto:Params: {} 2022-09-12T20:12:22.086 DEBUG:boto:Token: None 2022-09-12T20:12:22.087 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/ 2022-09-12T20:12:22.087 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao= 2022-09-12T20:12:22.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao=', 'Content-Length': '0'} 2022-09-12T20:12:22.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f654f65d2ea4de56-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.134 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:22.135 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:22.136 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:22.136 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:12:22.137 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:12:22.137 DEBUG:boto:Method: GET 2022-09-12T20:12:22.138 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:12:22.139 DEBUG:boto:Data: 2022-09-12T20:12:22.139 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.140 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.140 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.141 DEBUG:boto:Params: {} 2022-09-12T20:12:22.141 DEBUG:boto:Token: None 2022-09-12T20:12:22.142 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/ 2022-09-12T20:12:22.142 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao= 2022-09-12T20:12:22.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao=', 'Content-Length': '0'} 2022-09-12T20:12:22.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005283f6ef61834af7-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.147 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:22.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:12:22.148 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:22.149 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:22.149 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:22.150 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:22.150 DEBUG:boto:Method: GET 2022-09-12T20:12:22.151 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:22.152 DEBUG:boto:Data: 2022-09-12T20:12:22.152 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.153 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.153 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.154 DEBUG:boto:Params: {} 2022-09-12T20:12:22.154 DEBUG:boto:Token: None 2022-09-12T20:12:22.155 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:22.156 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ef2RjK2ozWMyqra2c+F01K31+fM= 2022-09-12T20:12:22.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ef2RjK2ozWMyqra2c+F01K31+fM=', 'Content-Length': '0'} 2022-09-12T20:12:22.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdb06dd19d350477-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.159 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:22.160 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:22.160 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:22.161 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:22.161 DEBUG:boto:Method: GET 2022-09-12T20:12:22.162 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:22.162 DEBUG:boto:Data: 2022-09-12T20:12:22.163 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.163 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.164 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.164 DEBUG:boto:Params: {} 2022-09-12T20:12:22.165 DEBUG:boto:Token: None 2022-09-12T20:12:22.166 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:22.166 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ef2RjK2ozWMyqra2c+F01K31+fM= 2022-09-12T20:12:22.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ef2RjK2ozWMyqra2c+F01K31+fM=', 'Content-Length': '0'} 2022-09-12T20:12:22.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4d0861978e99d1f-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.171 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:22.171 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:22.172 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:22.173 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:22.174 DEBUG:boto:Method: HEAD 2022-09-12T20:12:22.174 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:22.175 DEBUG:boto:Data: 2022-09-12T20:12:22.176 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.177 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.177 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.178 DEBUG:boto:Params: {} 2022-09-12T20:12:22.179 DEBUG:boto:Token: None 2022-09-12T20:12:22.180 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:22.181 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NqT4bvQ9eAG6TRbEYan8PTVINTI= 2022-09-12T20:12:22.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NqT4bvQ9eAG6TRbEYan8PTVINTI=', 'Content-Length': '0'} 2022-09-12T20:12:22.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9088eef5da1b14f-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.185 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:22.186 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:22.187 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:22.187 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:22.188 DEBUG:boto:Method: HEAD 2022-09-12T20:12:22.188 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:22.189 DEBUG:boto:Data: 2022-09-12T20:12:22.189 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.190 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.190 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.191 DEBUG:boto:Params: {} 2022-09-12T20:12:22.191 DEBUG:boto:Token: None 2022-09-12T20:12:22.192 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:22.193 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NqT4bvQ9eAG6TRbEYan8PTVINTI= 2022-09-12T20:12:22.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NqT4bvQ9eAG6TRbEYan8PTVINTI=', 'Content-Length': '0'} 2022-09-12T20:12:22.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077cf31ac28516480-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:12:22.197 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:22.197 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:22.198 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:22.198 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:22.199 DEBUG:boto:Method: GET 2022-09-12T20:12:22.199 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:22.200 DEBUG:boto:Data: 2022-09-12T20:12:22.200 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.201 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.202 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.202 DEBUG:boto:Params: {} 2022-09-12T20:12:22.203 DEBUG:boto:Token: None 2022-09-12T20:12:22.203 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:22.204 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ef2RjK2ozWMyqra2c+F01K31+fM= 2022-09-12T20:12:22.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ef2RjK2ozWMyqra2c+F01K31+fM=', 'Content-Length': '0'} 2022-09-12T20:12:22.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c645cbd1077c3947-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.208 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:22.208 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:22.209 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:22.209 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:12:22.210 DEBUG:boto:Method: GET 2022-09-12T20:12:22.210 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:22.211 DEBUG:boto:Data: 2022-09-12T20:12:22.212 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.212 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.213 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.213 DEBUG:boto:Params: {} 2022-09-12T20:12:22.214 DEBUG:boto:Token: None 2022-09-12T20:12:22.215 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:12:22.215 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ef2RjK2ozWMyqra2c+F01K31+fM= 2022-09-12T20:12:22.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ef2RjK2ozWMyqra2c+F01K31+fM=', 'Content-Length': '0'} 2022-09-12T20:12:22.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093165770f3367c14-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.219 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:22.220 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:22.220 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:12:22.221 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:12:22.221 DEBUG:boto:Method: GET 2022-09-12T20:12:22.222 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:12:22.222 DEBUG:boto:Data: 2022-09-12T20:12:22.223 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.224 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.224 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.225 DEBUG:boto:Params: {} 2022-09-12T20:12:22.225 DEBUG:boto:Token: None 2022-09-12T20:12:22.226 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/ 2022-09-12T20:12:22.226 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao= 2022-09-12T20:12:22.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao=', 'Content-Length': '0'} 2022-09-12T20:12:22.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025edfe78c003e288-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.270 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:22.271 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:22.271 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:22.272 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:12:22.273 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:12:22.273 DEBUG:boto:Method: GET 2022-09-12T20:12:22.274 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:12:22.274 DEBUG:boto:Data: 2022-09-12T20:12:22.275 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.275 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.276 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.277 DEBUG:boto:Params: {} 2022-09-12T20:12:22.277 DEBUG:boto:Token: None 2022-09-12T20:12:22.278 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/ 2022-09-12T20:12:22.279 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao= 2022-09-12T20:12:22.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao=', 'Content-Length': '0'} 2022-09-12T20:12:22.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047c2f09e7f7bf985-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.282 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:22.283 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:22.284 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:22.284 DEBUG:boto:Method: HEAD 2022-09-12T20:12:22.285 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:12:22.285 DEBUG:boto:Data: 2022-09-12T20:12:22.286 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.286 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.287 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.287 DEBUG:boto:Params: {} 2022-09-12T20:12:22.288 DEBUG:boto:Token: None 2022-09-12T20:12:22.289 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj58 2022-09-12T20:12:22.289 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AECOt4gx4kssXAnMxu+RzhODlhY= 2022-09-12T20:12:22.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AECOt4gx4kssXAnMxu+RzhODlhY=', 'Content-Length': '0'} 2022-09-12T20:12:22.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d22990e4054203b-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.292 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:22.293 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:22.294 DEBUG:boto:Method: HEAD 2022-09-12T20:12:22.294 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:12:22.295 DEBUG:boto:Data: 2022-09-12T20:12:22.295 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.296 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.296 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.297 DEBUG:boto:Params: {} 2022-09-12T20:12:22.297 DEBUG:boto:Token: None 2022-09-12T20:12:22.298 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj58 2022-09-12T20:12:22.298 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AECOt4gx4kssXAnMxu+RzhODlhY= 2022-09-12T20:12:22.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AECOt4gx4kssXAnMxu+RzhODlhY=', 'Content-Length': '0'} 2022-09-12T20:12:22.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ab5ff369a36b103-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:12:22.302 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:22.303 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:22.304 DEBUG:boto:Method: GET 2022-09-12T20:12:22.305 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:12:22.305 DEBUG:boto:Data: 2022-09-12T20:12:22.306 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.306 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.307 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.308 DEBUG:boto:Params: {} 2022-09-12T20:12:22.308 DEBUG:boto:Token: None 2022-09-12T20:12:22.309 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj58 2022-09-12T20:12:22.309 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6ANIoT/CTFs+B366RPXyEs5L9Zo= 2022-09-12T20:12:22.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6ANIoT/CTFs+B366RPXyEs5L9Zo=', 'Content-Length': '0'} 2022-09-12T20:12:22.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000282b96e7ead30390-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.313 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:12:22.314 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:12:22.314 DEBUG:boto:Method: GET 2022-09-12T20:12:22.315 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:12:22.315 DEBUG:boto:Data: 2022-09-12T20:12:22.316 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.317 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.317 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.318 DEBUG:boto:Params: {} 2022-09-12T20:12:22.318 DEBUG:boto:Token: None 2022-09-12T20:12:22.319 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj58 2022-09-12T20:12:22.320 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6ANIoT/CTFs+B366RPXyEs5L9Zo= 2022-09-12T20:12:22.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6ANIoT/CTFs+B366RPXyEs5L9Zo=', 'Content-Length': '0'} 2022-09-12T20:12:22.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073aa91b5b7dfff7f-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.366 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:22.367 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:22.367 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:12:22.368 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:12:22.368 DEBUG:boto:Method: GET 2022-09-12T20:12:22.369 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:12:22.369 DEBUG:boto:Data: 2022-09-12T20:12:22.370 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.371 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.371 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.372 DEBUG:boto:Params: {} 2022-09-12T20:12:22.372 DEBUG:boto:Token: None 2022-09-12T20:12:22.373 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/ 2022-09-12T20:12:22.373 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao= 2022-09-12T20:12:22.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao=', 'Content-Length': '0'} 2022-09-12T20:12:22.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000334b2428c73a26c3-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.377 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:22.378 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:22.379 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:22.380 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:12:22.380 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:12:22.381 DEBUG:boto:Method: GET 2022-09-12T20:12:22.381 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:12:22.382 DEBUG:boto:Data: 2022-09-12T20:12:22.382 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.383 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.383 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.384 DEBUG:boto:Params: {} 2022-09-12T20:12:22.384 DEBUG:boto:Token: None 2022-09-12T20:12:22.385 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/ 2022-09-12T20:12:22.385 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao= 2022-09-12T20:12:22.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao=', 'Content-Length': '0'} 2022-09-12T20:12:22.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ab889440795c64d-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.389 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:22.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:12:22.390 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:22.391 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:22.392 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:22.392 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:22.393 DEBUG:boto:Method: GET 2022-09-12T20:12:22.393 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:22.394 DEBUG:boto:Data: 2022-09-12T20:12:22.394 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.395 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.395 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.396 DEBUG:boto:Params: {} 2022-09-12T20:12:22.396 DEBUG:boto:Token: None 2022-09-12T20:12:22.397 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:22.398 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Lmu70EjTKNR3PQ/aHKjkYDhTHTA= 2022-09-12T20:12:22.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Lmu70EjTKNR3PQ/aHKjkYDhTHTA=', 'Content-Length': '0'} 2022-09-12T20:12:22.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6cc0a3b2039de55-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.401 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:22.402 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:22.402 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:22.403 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:22.403 DEBUG:boto:Method: GET 2022-09-12T20:12:22.404 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:22.404 DEBUG:boto:Data: 2022-09-12T20:12:22.405 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.405 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.406 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.406 DEBUG:boto:Params: {} 2022-09-12T20:12:22.407 DEBUG:boto:Token: None 2022-09-12T20:12:22.408 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:22.408 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Lmu70EjTKNR3PQ/aHKjkYDhTHTA= 2022-09-12T20:12:22.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Lmu70EjTKNR3PQ/aHKjkYDhTHTA=', 'Content-Length': '0'} 2022-09-12T20:12:22.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000063d5a8e33131f18-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.412 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:22.412 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:22.413 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:22.414 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:22.414 DEBUG:boto:Method: HEAD 2022-09-12T20:12:22.415 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:22.415 DEBUG:boto:Data: 2022-09-12T20:12:22.416 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.416 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.417 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.417 DEBUG:boto:Params: {} 2022-09-12T20:12:22.418 DEBUG:boto:Token: None 2022-09-12T20:12:22.418 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:22.419 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5Q8BRDH1tjVV3OilURHeqPYw+iE= 2022-09-12T20:12:22.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5Q8BRDH1tjVV3OilURHeqPYw+iE=', 'Content-Length': '0'} 2022-09-12T20:12:22.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b050ce2bd0dedec3-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.422 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:22.423 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:22.423 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:22.424 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:22.424 DEBUG:boto:Method: HEAD 2022-09-12T20:12:22.425 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:22.425 DEBUG:boto:Data: 2022-09-12T20:12:22.426 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.426 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.427 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.427 DEBUG:boto:Params: {} 2022-09-12T20:12:22.428 DEBUG:boto:Token: None 2022-09-12T20:12:22.429 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:22.429 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5Q8BRDH1tjVV3OilURHeqPYw+iE= 2022-09-12T20:12:22.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5Q8BRDH1tjVV3OilURHeqPYw+iE=', 'Content-Length': '0'} 2022-09-12T20:12:22.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019ce9ec9b0fba083-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:12:22.433 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:22.433 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:22.434 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:22.434 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:22.435 DEBUG:boto:Method: GET 2022-09-12T20:12:22.436 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:22.436 DEBUG:boto:Data: 2022-09-12T20:12:22.437 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.437 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.438 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.438 DEBUG:boto:Params: {} 2022-09-12T20:12:22.439 DEBUG:boto:Token: None 2022-09-12T20:12:22.440 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:22.440 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Lmu70EjTKNR3PQ/aHKjkYDhTHTA= 2022-09-12T20:12:22.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Lmu70EjTKNR3PQ/aHKjkYDhTHTA=', 'Content-Length': '0'} 2022-09-12T20:12:22.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fd62ba02c3bce24-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.444 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:22.444 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:22.445 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:22.445 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:12:22.446 DEBUG:boto:Method: GET 2022-09-12T20:12:22.446 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:22.447 DEBUG:boto:Data: 2022-09-12T20:12:22.447 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.448 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.448 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.449 DEBUG:boto:Params: {} 2022-09-12T20:12:22.450 DEBUG:boto:Token: None 2022-09-12T20:12:22.450 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:12:22.451 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Lmu70EjTKNR3PQ/aHKjkYDhTHTA= 2022-09-12T20:12:22.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Lmu70EjTKNR3PQ/aHKjkYDhTHTA=', 'Content-Length': '0'} 2022-09-12T20:12:22.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001234331079388897-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.455 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:22.455 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:22.456 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:12:22.456 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:12:22.457 DEBUG:boto:Method: GET 2022-09-12T20:12:22.457 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:12:22.458 DEBUG:boto:Data: 2022-09-12T20:12:22.458 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.459 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.459 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.460 DEBUG:boto:Params: {} 2022-09-12T20:12:22.461 DEBUG:boto:Token: None 2022-09-12T20:12:22.461 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/ 2022-09-12T20:12:22.462 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao= 2022-09-12T20:12:22.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao=', 'Content-Length': '0'} 2022-09-12T20:12:22.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000118e424fdf0e6ec8-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.466 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:22.467 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:22.467 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:22.468 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:12:22.468 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:12:22.469 DEBUG:boto:Method: GET 2022-09-12T20:12:22.470 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:12:22.470 DEBUG:boto:Data: 2022-09-12T20:12:22.471 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.471 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.472 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.472 DEBUG:boto:Params: {} 2022-09-12T20:12:22.473 DEBUG:boto:Token: None 2022-09-12T20:12:22.474 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/ 2022-09-12T20:12:22.474 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao= 2022-09-12T20:12:22.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao=', 'Content-Length': '0'} 2022-09-12T20:12:22.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2e907c7d65b110d-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.479 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:22.480 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:22.480 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:22.481 DEBUG:boto:Method: HEAD 2022-09-12T20:12:22.481 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:12:22.482 DEBUG:boto:Data: 2022-09-12T20:12:22.482 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.483 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.483 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.484 DEBUG:boto:Params: {} 2022-09-12T20:12:22.484 DEBUG:boto:Token: None 2022-09-12T20:12:22.485 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj68 2022-09-12T20:12:22.485 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:udr2loS0NMzqLb+mMswpengBxnA= 2022-09-12T20:12:22.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:udr2loS0NMzqLb+mMswpengBxnA=', 'Content-Length': '0'} 2022-09-12T20:12:22.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014c9f0ab5887ab0c-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.489 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:22.489 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:22.490 DEBUG:boto:Method: HEAD 2022-09-12T20:12:22.490 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:12:22.491 DEBUG:boto:Data: 2022-09-12T20:12:22.491 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.492 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.492 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.493 DEBUG:boto:Params: {} 2022-09-12T20:12:22.493 DEBUG:boto:Token: None 2022-09-12T20:12:22.494 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj68 2022-09-12T20:12:22.494 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:udr2loS0NMzqLb+mMswpengBxnA= 2022-09-12T20:12:22.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:udr2loS0NMzqLb+mMswpengBxnA=', 'Content-Length': '0'} 2022-09-12T20:12:22.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c40569276e752a8-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:12:22.498 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:22.499 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:22.500 DEBUG:boto:Method: GET 2022-09-12T20:12:22.500 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:12:22.501 DEBUG:boto:Data: 2022-09-12T20:12:22.501 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.502 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.502 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.503 DEBUG:boto:Params: {} 2022-09-12T20:12:22.504 DEBUG:boto:Token: None 2022-09-12T20:12:22.504 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj68 2022-09-12T20:12:22.505 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pQwrjElmQmExbCRPPbkWfB/7wjQ= 2022-09-12T20:12:22.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pQwrjElmQmExbCRPPbkWfB/7wjQ=', 'Content-Length': '0'} 2022-09-12T20:12:22.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005306958b5e6ef895-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.509 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:12:22.509 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:12:22.510 DEBUG:boto:Method: GET 2022-09-12T20:12:22.510 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:12:22.511 DEBUG:boto:Data: 2022-09-12T20:12:22.511 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.512 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.512 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.513 DEBUG:boto:Params: {} 2022-09-12T20:12:22.513 DEBUG:boto:Token: None 2022-09-12T20:12:22.514 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj68 2022-09-12T20:12:22.515 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pQwrjElmQmExbCRPPbkWfB/7wjQ= 2022-09-12T20:12:22.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pQwrjElmQmExbCRPPbkWfB/7wjQ=', 'Content-Length': '0'} 2022-09-12T20:12:22.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc909f028330136c-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.562 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:22.563 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:22.564 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:12:22.564 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:12:22.565 DEBUG:boto:Method: GET 2022-09-12T20:12:22.565 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:12:22.566 DEBUG:boto:Data: 2022-09-12T20:12:22.567 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.567 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.568 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.568 DEBUG:boto:Params: {} 2022-09-12T20:12:22.569 DEBUG:boto:Token: None 2022-09-12T20:12:22.570 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/ 2022-09-12T20:12:22.571 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao= 2022-09-12T20:12:22.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao=', 'Content-Length': '0'} 2022-09-12T20:12:22.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b480da001ccfca56-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.575 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:22.576 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:22.576 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:22.577 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:12:22.577 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:12:22.578 DEBUG:boto:Method: GET 2022-09-12T20:12:22.578 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:12:22.579 DEBUG:boto:Data: 2022-09-12T20:12:22.579 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.580 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.580 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.581 DEBUG:boto:Params: {} 2022-09-12T20:12:22.581 DEBUG:boto:Token: None 2022-09-12T20:12:22.582 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/ 2022-09-12T20:12:22.583 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao= 2022-09-12T20:12:22.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao=', 'Content-Length': '0'} 2022-09-12T20:12:22.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041e3a2b6081174ca-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.587 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:22.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:12:22.588 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:22.588 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:22.589 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:22.589 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:22.590 DEBUG:boto:Method: GET 2022-09-12T20:12:22.590 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:22.591 DEBUG:boto:Data: 2022-09-12T20:12:22.591 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.592 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.593 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.593 DEBUG:boto:Params: {} 2022-09-12T20:12:22.594 DEBUG:boto:Token: None 2022-09-12T20:12:22.594 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:22.595 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9p1GMVb2zprkPjVVKVMpAbVPQx4= 2022-09-12T20:12:22.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9p1GMVb2zprkPjVVKVMpAbVPQx4=', 'Content-Length': '0'} 2022-09-12T20:12:22.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000448810db6ff67ed7-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.638 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:22.639 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:22.639 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:22.640 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:22.640 DEBUG:boto:Method: GET 2022-09-12T20:12:22.641 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:22.641 DEBUG:boto:Data: 2022-09-12T20:12:22.642 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.642 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.643 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.644 DEBUG:boto:Params: {} 2022-09-12T20:12:22.644 DEBUG:boto:Token: None 2022-09-12T20:12:22.645 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:22.645 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9p1GMVb2zprkPjVVKVMpAbVPQx4= 2022-09-12T20:12:22.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9p1GMVb2zprkPjVVKVMpAbVPQx4=', 'Content-Length': '0'} 2022-09-12T20:12:22.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009602517f8c6232c4-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.649 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:22.649 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:22.650 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:22.650 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:22.651 DEBUG:boto:Method: HEAD 2022-09-12T20:12:22.651 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:22.652 DEBUG:boto:Data: 2022-09-12T20:12:22.653 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.653 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.654 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.654 DEBUG:boto:Params: {} 2022-09-12T20:12:22.655 DEBUG:boto:Token: None 2022-09-12T20:12:22.655 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:22.656 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f9f8cb7a0lTXfTlH3wqypGsq3R4= 2022-09-12T20:12:22.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f9f8cb7a0lTXfTlH3wqypGsq3R4=', 'Content-Length': '0'} 2022-09-12T20:12:22.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3f86841c32b00ef-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.659 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:22.660 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:22.660 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:22.661 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:22.661 DEBUG:boto:Method: HEAD 2022-09-12T20:12:22.662 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:22.662 DEBUG:boto:Data: 2022-09-12T20:12:22.663 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.664 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.664 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.665 DEBUG:boto:Params: {} 2022-09-12T20:12:22.665 DEBUG:boto:Token: None 2022-09-12T20:12:22.666 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:22.666 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f9f8cb7a0lTXfTlH3wqypGsq3R4= 2022-09-12T20:12:22.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f9f8cb7a0lTXfTlH3wqypGsq3R4=', 'Content-Length': '0'} 2022-09-12T20:12:22.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004619e25f82539c9b-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:12:22.670 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:22.671 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:22.672 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:22.673 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:22.673 DEBUG:boto:Method: GET 2022-09-12T20:12:22.674 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:22.675 DEBUG:boto:Data: 2022-09-12T20:12:22.676 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.676 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.677 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.678 DEBUG:boto:Params: {} 2022-09-12T20:12:22.679 DEBUG:boto:Token: None 2022-09-12T20:12:22.680 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:22.680 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9p1GMVb2zprkPjVVKVMpAbVPQx4= 2022-09-12T20:12:22.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9p1GMVb2zprkPjVVKVMpAbVPQx4=', 'Content-Length': '0'} 2022-09-12T20:12:22.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c643a87beed448e-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.684 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:22.685 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:22.685 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:22.686 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:12:22.687 DEBUG:boto:Method: GET 2022-09-12T20:12:22.687 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:22.688 DEBUG:boto:Data: 2022-09-12T20:12:22.688 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.689 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.690 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.690 DEBUG:boto:Params: {} 2022-09-12T20:12:22.691 DEBUG:boto:Token: None 2022-09-12T20:12:22.692 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:12:22.693 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9p1GMVb2zprkPjVVKVMpAbVPQx4= 2022-09-12T20:12:22.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9p1GMVb2zprkPjVVKVMpAbVPQx4=', 'Content-Length': '0'} 2022-09-12T20:12:22.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032aa8beceae770c6-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.696 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:22.697 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:22.698 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:12:22.699 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:12:22.699 DEBUG:boto:Method: GET 2022-09-12T20:12:22.700 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:12:22.701 DEBUG:boto:Data: 2022-09-12T20:12:22.701 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.702 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.702 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.703 DEBUG:boto:Params: {} 2022-09-12T20:12:22.704 DEBUG:boto:Token: None 2022-09-12T20:12:22.704 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/ 2022-09-12T20:12:22.705 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao= 2022-09-12T20:12:22.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao=', 'Content-Length': '0'} 2022-09-12T20:12:22.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f62de35d24a01e88-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.709 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:22.710 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:22.711 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:22.712 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:12:22.713 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:12:22.714 DEBUG:boto:Method: GET 2022-09-12T20:12:22.714 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:12:22.715 DEBUG:boto:Data: 2022-09-12T20:12:22.715 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.716 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.717 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.717 DEBUG:boto:Params: {} 2022-09-12T20:12:22.718 DEBUG:boto:Token: None 2022-09-12T20:12:22.718 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/ 2022-09-12T20:12:22.719 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao= 2022-09-12T20:12:22.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao=', 'Content-Length': '0'} 2022-09-12T20:12:22.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055274d8dba0d72fd-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.723 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:22.724 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:22.725 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:22.726 DEBUG:boto:Method: HEAD 2022-09-12T20:12:22.726 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:12:22.727 DEBUG:boto:Data: 2022-09-12T20:12:22.728 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.728 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.729 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.729 DEBUG:boto:Params: {} 2022-09-12T20:12:22.730 DEBUG:boto:Token: None 2022-09-12T20:12:22.731 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj69 2022-09-12T20:12:22.732 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/hng6dKAwn+wllG+z0oMT+WObVA= 2022-09-12T20:12:22.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/hng6dKAwn+wllG+z0oMT+WObVA=', 'Content-Length': '0'} 2022-09-12T20:12:22.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f5ab1f5f94ac21b-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.735 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:22.736 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:22.736 DEBUG:boto:Method: HEAD 2022-09-12T20:12:22.737 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:12:22.737 DEBUG:boto:Data: 2022-09-12T20:12:22.738 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.738 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.739 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.739 DEBUG:boto:Params: {} 2022-09-12T20:12:22.740 DEBUG:boto:Token: None 2022-09-12T20:12:22.740 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj69 2022-09-12T20:12:22.741 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/hng6dKAwn+wllG+z0oMT+WObVA= 2022-09-12T20:12:22.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/hng6dKAwn+wllG+z0oMT+WObVA=', 'Content-Length': '0'} 2022-09-12T20:12:22.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003daa874a1686fd96-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:12:22.745 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:22.745 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:22.746 DEBUG:boto:Method: GET 2022-09-12T20:12:22.746 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:12:22.747 DEBUG:boto:Data: 2022-09-12T20:12:22.747 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.748 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.748 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.749 DEBUG:boto:Params: {} 2022-09-12T20:12:22.750 DEBUG:boto:Token: None 2022-09-12T20:12:22.750 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj69 2022-09-12T20:12:22.751 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mYCl/7lbo9ggiI6eYNaPCgv59Lw= 2022-09-12T20:12:22.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mYCl/7lbo9ggiI6eYNaPCgv59Lw=', 'Content-Length': '0'} 2022-09-12T20:12:22.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000587576bf2db9b330-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.794 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:12:22.794 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:12:22.795 DEBUG:boto:Method: GET 2022-09-12T20:12:22.796 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:12:22.796 DEBUG:boto:Data: 2022-09-12T20:12:22.797 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.797 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.798 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.798 DEBUG:boto:Params: {} 2022-09-12T20:12:22.799 DEBUG:boto:Token: None 2022-09-12T20:12:22.799 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj69 2022-09-12T20:12:22.800 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mYCl/7lbo9ggiI6eYNaPCgv59Lw= 2022-09-12T20:12:22.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mYCl/7lbo9ggiI6eYNaPCgv59Lw=', 'Content-Length': '0'} 2022-09-12T20:12:22.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000238da73fc3887e9e-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.804 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:22.804 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:22.805 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:12:22.805 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:12:22.806 DEBUG:boto:Method: GET 2022-09-12T20:12:22.806 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:12:22.807 DEBUG:boto:Data: 2022-09-12T20:12:22.807 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.808 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.809 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.809 DEBUG:boto:Params: {} 2022-09-12T20:12:22.810 DEBUG:boto:Token: None 2022-09-12T20:12:22.810 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/ 2022-09-12T20:12:22.811 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao= 2022-09-12T20:12:22.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao=', 'Content-Length': '0'} 2022-09-12T20:12:22.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a594a3426c84efc1-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.815 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:22.816 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:22.817 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:22.817 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:12:22.818 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:12:22.818 DEBUG:boto:Method: GET 2022-09-12T20:12:22.819 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:12:22.820 DEBUG:boto:Data: 2022-09-12T20:12:22.820 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.821 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.821 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.822 DEBUG:boto:Params: {} 2022-09-12T20:12:22.822 DEBUG:boto:Token: None 2022-09-12T20:12:22.823 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/ 2022-09-12T20:12:22.823 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao= 2022-09-12T20:12:22.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao=', 'Content-Length': '0'} 2022-09-12T20:12:22.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f45c983a04ce903-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.870 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:22.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:12:22.872 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:22.872 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:22.873 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:22.873 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:22.874 DEBUG:boto:Method: GET 2022-09-12T20:12:22.874 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:22.875 DEBUG:boto:Data: 2022-09-12T20:12:22.875 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.876 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.876 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.877 DEBUG:boto:Params: {} 2022-09-12T20:12:22.878 DEBUG:boto:Token: None 2022-09-12T20:12:22.878 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:22.879 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ng2lhu8jQFhY3IiHJZcxpttguok= 2022-09-12T20:12:22.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ng2lhu8jQFhY3IiHJZcxpttguok=', 'Content-Length': '0'} 2022-09-12T20:12:22.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f24a09c96c6cbb68-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.883 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:22.883 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:22.884 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:22.884 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:22.885 DEBUG:boto:Method: GET 2022-09-12T20:12:22.885 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:22.886 DEBUG:boto:Data: 2022-09-12T20:12:22.886 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.887 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.887 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.888 DEBUG:boto:Params: {} 2022-09-12T20:12:22.888 DEBUG:boto:Token: None 2022-09-12T20:12:22.889 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:22.890 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ng2lhu8jQFhY3IiHJZcxpttguok= 2022-09-12T20:12:22.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ng2lhu8jQFhY3IiHJZcxpttguok=', 'Content-Length': '0'} 2022-09-12T20:12:22.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea85689b0f1b507f-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.893 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:22.894 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:22.895 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:22.895 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:22.895 DEBUG:boto:Method: HEAD 2022-09-12T20:12:22.896 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:22.896 DEBUG:boto:Data: 2022-09-12T20:12:22.897 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.897 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.898 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.899 DEBUG:boto:Params: {} 2022-09-12T20:12:22.899 DEBUG:boto:Token: None 2022-09-12T20:12:22.900 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:22.900 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IaMcFSsrSG4bKlIkQSJyOpRzxsA= 2022-09-12T20:12:22.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IaMcFSsrSG4bKlIkQSJyOpRzxsA=', 'Content-Length': '0'} 2022-09-12T20:12:22.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030a73f0725998239-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.904 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:22.904 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:22.905 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:22.905 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:22.906 DEBUG:boto:Method: HEAD 2022-09-12T20:12:22.906 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:22.907 DEBUG:boto:Data: 2022-09-12T20:12:22.907 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.908 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.908 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.909 DEBUG:boto:Params: {} 2022-09-12T20:12:22.909 DEBUG:boto:Token: None 2022-09-12T20:12:22.910 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:22.910 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IaMcFSsrSG4bKlIkQSJyOpRzxsA= 2022-09-12T20:12:22.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IaMcFSsrSG4bKlIkQSJyOpRzxsA=', 'Content-Length': '0'} 2022-09-12T20:12:22.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035105910b44e2ba4-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:12:22.914 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:22.914 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:22.915 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:22.916 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:22.916 DEBUG:boto:Method: GET 2022-09-12T20:12:22.917 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:22.917 DEBUG:boto:Data: 2022-09-12T20:12:22.918 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.918 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.919 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.919 DEBUG:boto:Params: {} 2022-09-12T20:12:22.920 DEBUG:boto:Token: None 2022-09-12T20:12:22.921 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:22.921 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ng2lhu8jQFhY3IiHJZcxpttguok= 2022-09-12T20:12:22.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ng2lhu8jQFhY3IiHJZcxpttguok=', 'Content-Length': '0'} 2022-09-12T20:12:22.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb5f201276ec1df1-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.925 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:22.925 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:22.926 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:22.927 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:12:22.927 DEBUG:boto:Method: GET 2022-09-12T20:12:22.928 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:22.928 DEBUG:boto:Data: 2022-09-12T20:12:22.929 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.929 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.930 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.930 DEBUG:boto:Params: {} 2022-09-12T20:12:22.931 DEBUG:boto:Token: None 2022-09-12T20:12:22.932 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:12:22.932 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ng2lhu8jQFhY3IiHJZcxpttguok= 2022-09-12T20:12:22.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ng2lhu8jQFhY3IiHJZcxpttguok=', 'Content-Length': '0'} 2022-09-12T20:12:22.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caeea885493c2616-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.936 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:22.936 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:22.937 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:12:22.937 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:12:22.938 DEBUG:boto:Method: GET 2022-09-12T20:12:22.938 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:12:22.939 DEBUG:boto:Data: 2022-09-12T20:12:22.939 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.940 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.940 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.941 DEBUG:boto:Params: {} 2022-09-12T20:12:22.942 DEBUG:boto:Token: None 2022-09-12T20:12:22.942 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/ 2022-09-12T20:12:22.943 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao= 2022-09-12T20:12:22.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao=', 'Content-Length': '0'} 2022-09-12T20:12:22.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007330eb1b5475cd7-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.947 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:22.948 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:22.948 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:22.949 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:12:22.949 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:12:22.950 DEBUG:boto:Method: GET 2022-09-12T20:12:22.950 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:12:22.951 DEBUG:boto:Data: 2022-09-12T20:12:22.951 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.952 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.952 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.953 DEBUG:boto:Params: {} 2022-09-12T20:12:22.954 DEBUG:boto:Token: None 2022-09-12T20:12:22.954 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/ 2022-09-12T20:12:22.955 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao= 2022-09-12T20:12:22.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gY0pOvI86non6wkP05tWBURs4Ao=', 'Content-Length': '0'} 2022-09-12T20:12:22.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000105686f50012b43a-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.959 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:22.960 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:22.960 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:22.961 DEBUG:boto:Method: HEAD 2022-09-12T20:12:22.961 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:12:22.962 DEBUG:boto:Data: 2022-09-12T20:12:22.962 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.963 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.964 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.964 DEBUG:boto:Params: {} 2022-09-12T20:12:22.965 DEBUG:boto:Token: None 2022-09-12T20:12:22.965 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj76 2022-09-12T20:12:22.966 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Yl90+vt0SIxfDp0hODldrJxH/qI= 2022-09-12T20:12:22.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Yl90+vt0SIxfDp0hODldrJxH/qI=', 'Content-Length': '0'} 2022-09-12T20:12:22.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040f08a69dc5d4833-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.969 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:22.970 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:22.970 DEBUG:boto:Method: HEAD 2022-09-12T20:12:22.971 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:12:22.971 DEBUG:boto:Data: 2022-09-12T20:12:22.972 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.972 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:22.973 DEBUG:boto:Port: 8000 2022-09-12T20:12:22.973 DEBUG:boto:Params: {} 2022-09-12T20:12:22.974 DEBUG:boto:Token: None 2022-09-12T20:12:22.975 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj76 2022-09-12T20:12:22.975 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Yl90+vt0SIxfDp0hODldrJxH/qI= 2022-09-12T20:12:22.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Yl90+vt0SIxfDp0hODldrJxH/qI=', 'Content-Length': '0'} 2022-09-12T20:12:22.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000338bb786e54dda31-00631f92a6-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:22.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:12:22.979 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:22.980 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:22.980 DEBUG:boto:Method: GET 2022-09-12T20:12:22.981 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:12:22.981 DEBUG:boto:Data: 2022-09-12T20:12:22.982 DEBUG:boto:Headers: {} 2022-09-12T20:12:22.982 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:22.983 DEBUG:boto:Port: 8001 2022-09-12T20:12:22.984 DEBUG:boto:Params: {} 2022-09-12T20:12:22.984 DEBUG:boto:Token: None 2022-09-12T20:12:22.985 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:22 GMT /jdpiak-75/obj76 2022-09-12T20:12:22.985 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5LUdfdOuV0LNKeTV4dlI/hJbFyY= 2022-09-12T20:12:22.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5LUdfdOuV0LNKeTV4dlI/hJbFyY=', 'Content-Length': '0'} 2022-09-12T20:12:22.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2e7d7701a45ff32-00631f92a6-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.030 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:12:23.031 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:12:23.031 DEBUG:boto:Method: GET 2022-09-12T20:12:23.032 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:12:23.032 DEBUG:boto:Data: 2022-09-12T20:12:23.033 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.033 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.034 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.034 DEBUG:boto:Params: {} 2022-09-12T20:12:23.035 DEBUG:boto:Token: None 2022-09-12T20:12:23.036 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj76 2022-09-12T20:12:23.036 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LOYtE5fBM1CXSsoLON9ARfNX2n0= 2022-09-12T20:12:23.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LOYtE5fBM1CXSsoLON9ARfNX2n0=', 'Content-Length': '0'} 2022-09-12T20:12:23.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000350bde46e82d4f3b-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.040 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:23.040 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:23.041 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:12:23.042 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:12:23.042 DEBUG:boto:Method: GET 2022-09-12T20:12:23.043 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:12:23.043 DEBUG:boto:Data: 2022-09-12T20:12:23.044 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.044 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.045 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.045 DEBUG:boto:Params: {} 2022-09-12T20:12:23.046 DEBUG:boto:Token: None 2022-09-12T20:12:23.046 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/ 2022-09-12T20:12:23.047 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc= 2022-09-12T20:12:23.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc=', 'Content-Length': '0'} 2022-09-12T20:12:23.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d52ac46080f6706-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.051 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:23.052 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:23.053 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:23.053 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:12:23.054 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:12:23.055 DEBUG:boto:Method: GET 2022-09-12T20:12:23.055 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:12:23.056 DEBUG:boto:Data: 2022-09-12T20:12:23.056 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.057 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.057 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.058 DEBUG:boto:Params: {} 2022-09-12T20:12:23.058 DEBUG:boto:Token: None 2022-09-12T20:12:23.059 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/ 2022-09-12T20:12:23.060 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc= 2022-09-12T20:12:23.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc=', 'Content-Length': '0'} 2022-09-12T20:12:23.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ab90969a94eb62c-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.064 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:23.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:12:23.065 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:23.066 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:23.066 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:23.067 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:23.067 DEBUG:boto:Method: GET 2022-09-12T20:12:23.068 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:23.068 DEBUG:boto:Data: 2022-09-12T20:12:23.069 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.069 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.070 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.070 DEBUG:boto:Params: {} 2022-09-12T20:12:23.071 DEBUG:boto:Token: None 2022-09-12T20:12:23.072 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:23.072 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LtqCqN9kxyP4BsmlaM7uBw0MgGY= 2022-09-12T20:12:23.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LtqCqN9kxyP4BsmlaM7uBw0MgGY=', 'Content-Length': '0'} 2022-09-12T20:12:23.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b02ac9abd194119c-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.076 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:23.077 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:23.077 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:23.078 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:23.078 DEBUG:boto:Method: GET 2022-09-12T20:12:23.079 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:23.079 DEBUG:boto:Data: 2022-09-12T20:12:23.080 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.081 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.081 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.082 DEBUG:boto:Params: {} 2022-09-12T20:12:23.082 DEBUG:boto:Token: None 2022-09-12T20:12:23.083 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:23.083 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LtqCqN9kxyP4BsmlaM7uBw0MgGY= 2022-09-12T20:12:23.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LtqCqN9kxyP4BsmlaM7uBw0MgGY=', 'Content-Length': '0'} 2022-09-12T20:12:23.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d62727a7709f9912-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.087 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:23.087 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:23.088 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:23.088 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:23.089 DEBUG:boto:Method: HEAD 2022-09-12T20:12:23.089 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:23.090 DEBUG:boto:Data: 2022-09-12T20:12:23.090 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.091 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.091 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.092 DEBUG:boto:Params: {} 2022-09-12T20:12:23.092 DEBUG:boto:Token: None 2022-09-12T20:12:23.093 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:23.093 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zLhpAInWbDMfb5tzMp3uOYZlTko= 2022-09-12T20:12:23.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zLhpAInWbDMfb5tzMp3uOYZlTko=', 'Content-Length': '0'} 2022-09-12T20:12:23.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006eee63a10de5e991-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.097 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:23.097 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:23.098 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:23.098 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:23.099 DEBUG:boto:Method: HEAD 2022-09-12T20:12:23.099 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:23.100 DEBUG:boto:Data: 2022-09-12T20:12:23.100 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.101 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.101 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.102 DEBUG:boto:Params: {} 2022-09-12T20:12:23.102 DEBUG:boto:Token: None 2022-09-12T20:12:23.103 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:23.103 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zLhpAInWbDMfb5tzMp3uOYZlTko= 2022-09-12T20:12:23.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zLhpAInWbDMfb5tzMp3uOYZlTko=', 'Content-Length': '0'} 2022-09-12T20:12:23.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5ad845ecee7a801-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:12:23.107 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:23.108 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:23.108 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:23.109 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:23.109 DEBUG:boto:Method: GET 2022-09-12T20:12:23.110 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:23.110 DEBUG:boto:Data: 2022-09-12T20:12:23.111 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.112 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.112 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.113 DEBUG:boto:Params: {} 2022-09-12T20:12:23.113 DEBUG:boto:Token: None 2022-09-12T20:12:23.114 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:23.114 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LtqCqN9kxyP4BsmlaM7uBw0MgGY= 2022-09-12T20:12:23.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LtqCqN9kxyP4BsmlaM7uBw0MgGY=', 'Content-Length': '0'} 2022-09-12T20:12:23.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000938a7f3985259600-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.118 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:23.118 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:23.119 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:23.119 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:12:23.120 DEBUG:boto:Method: GET 2022-09-12T20:12:23.121 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:23.121 DEBUG:boto:Data: 2022-09-12T20:12:23.122 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.122 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.123 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.123 DEBUG:boto:Params: {} 2022-09-12T20:12:23.124 DEBUG:boto:Token: None 2022-09-12T20:12:23.124 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:12:23.125 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LtqCqN9kxyP4BsmlaM7uBw0MgGY= 2022-09-12T20:12:23.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LtqCqN9kxyP4BsmlaM7uBw0MgGY=', 'Content-Length': '0'} 2022-09-12T20:12:23.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a15864965f08880b-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.170 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:23.171 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:23.171 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:12:23.172 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:12:23.172 DEBUG:boto:Method: GET 2022-09-12T20:12:23.173 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:12:23.173 DEBUG:boto:Data: 2022-09-12T20:12:23.174 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.174 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.175 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.176 DEBUG:boto:Params: {} 2022-09-12T20:12:23.176 DEBUG:boto:Token: None 2022-09-12T20:12:23.177 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/ 2022-09-12T20:12:23.177 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc= 2022-09-12T20:12:23.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc=', 'Content-Length': '0'} 2022-09-12T20:12:23.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec95629f54a9e56a-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.182 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:23.182 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:23.183 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:23.184 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:12:23.184 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:12:23.185 DEBUG:boto:Method: GET 2022-09-12T20:12:23.185 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:12:23.186 DEBUG:boto:Data: 2022-09-12T20:12:23.186 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.187 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.187 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.188 DEBUG:boto:Params: {} 2022-09-12T20:12:23.189 DEBUG:boto:Token: None 2022-09-12T20:12:23.189 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/ 2022-09-12T20:12:23.190 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc= 2022-09-12T20:12:23.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc=', 'Content-Length': '0'} 2022-09-12T20:12:23.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085f62a4930cde90d-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.194 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:23.194 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:23.195 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:23.195 DEBUG:boto:Method: HEAD 2022-09-12T20:12:23.196 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:12:23.197 DEBUG:boto:Data: 2022-09-12T20:12:23.197 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.198 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.198 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.199 DEBUG:boto:Params: {} 2022-09-12T20:12:23.199 DEBUG:boto:Token: None 2022-09-12T20:12:23.200 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj85 2022-09-12T20:12:23.200 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Rzz4pFy7Ih54l7ijbHPPKx8c5hk= 2022-09-12T20:12:23.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Rzz4pFy7Ih54l7ijbHPPKx8c5hk=', 'Content-Length': '0'} 2022-09-12T20:12:23.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a35afef2c8c67e58-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.204 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:23.204 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:23.205 DEBUG:boto:Method: HEAD 2022-09-12T20:12:23.205 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:12:23.206 DEBUG:boto:Data: 2022-09-12T20:12:23.206 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.207 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.207 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.208 DEBUG:boto:Params: {} 2022-09-12T20:12:23.208 DEBUG:boto:Token: None 2022-09-12T20:12:23.209 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj85 2022-09-12T20:12:23.209 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Rzz4pFy7Ih54l7ijbHPPKx8c5hk= 2022-09-12T20:12:23.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Rzz4pFy7Ih54l7ijbHPPKx8c5hk=', 'Content-Length': '0'} 2022-09-12T20:12:23.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016325974ae55133d-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:12:23.213 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:23.214 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:23.214 DEBUG:boto:Method: GET 2022-09-12T20:12:23.215 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:12:23.215 DEBUG:boto:Data: 2022-09-12T20:12:23.216 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.217 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.217 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.218 DEBUG:boto:Params: {} 2022-09-12T20:12:23.218 DEBUG:boto:Token: None 2022-09-12T20:12:23.219 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj85 2022-09-12T20:12:23.219 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pQxAJxuOUiYCdEaq0DZ5yATIXXY= 2022-09-12T20:12:23.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pQxAJxuOUiYCdEaq0DZ5yATIXXY=', 'Content-Length': '0'} 2022-09-12T20:12:23.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be335b7042a5d04e-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.266 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:12:23.267 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:12:23.267 DEBUG:boto:Method: GET 2022-09-12T20:12:23.268 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:12:23.268 DEBUG:boto:Data: 2022-09-12T20:12:23.269 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.269 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.270 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.270 DEBUG:boto:Params: {} 2022-09-12T20:12:23.271 DEBUG:boto:Token: None 2022-09-12T20:12:23.272 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj85 2022-09-12T20:12:23.272 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pQxAJxuOUiYCdEaq0DZ5yATIXXY= 2022-09-12T20:12:23.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pQxAJxuOUiYCdEaq0DZ5yATIXXY=', 'Content-Length': '0'} 2022-09-12T20:12:23.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fe0658d8ad77495-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.276 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:23.276 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:23.277 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:12:23.278 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:12:23.278 DEBUG:boto:Method: GET 2022-09-12T20:12:23.279 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:12:23.279 DEBUG:boto:Data: 2022-09-12T20:12:23.280 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.281 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.281 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.282 DEBUG:boto:Params: {} 2022-09-12T20:12:23.282 DEBUG:boto:Token: None 2022-09-12T20:12:23.283 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/ 2022-09-12T20:12:23.284 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc= 2022-09-12T20:12:23.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc=', 'Content-Length': '0'} 2022-09-12T20:12:23.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a5f1b2dca67c897-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.288 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:23.289 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:23.289 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:23.290 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:12:23.290 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:12:23.291 DEBUG:boto:Method: GET 2022-09-12T20:12:23.291 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:12:23.292 DEBUG:boto:Data: 2022-09-12T20:12:23.293 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.293 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.294 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.294 DEBUG:boto:Params: {} 2022-09-12T20:12:23.295 DEBUG:boto:Token: None 2022-09-12T20:12:23.295 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/ 2022-09-12T20:12:23.296 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc= 2022-09-12T20:12:23.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc=', 'Content-Length': '0'} 2022-09-12T20:12:23.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033cb4c9d80f97673-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.300 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:23.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:12:23.301 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:23.302 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:23.302 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:23.303 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:23.303 DEBUG:boto:Method: GET 2022-09-12T20:12:23.304 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:23.305 DEBUG:boto:Data: 2022-09-12T20:12:23.305 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.306 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.306 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.307 DEBUG:boto:Params: {} 2022-09-12T20:12:23.307 DEBUG:boto:Token: None 2022-09-12T20:12:23.308 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:23.309 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m9w0jWRH5rS/EIp6uq8uxOTh/A8= 2022-09-12T20:12:23.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m9w0jWRH5rS/EIp6uq8uxOTh/A8=', 'Content-Length': '0'} 2022-09-12T20:12:23.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b123bc89fdd1e0c7-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.312 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:23.313 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:23.313 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:23.314 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:23.314 DEBUG:boto:Method: GET 2022-09-12T20:12:23.315 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:23.315 DEBUG:boto:Data: 2022-09-12T20:12:23.316 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.317 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.317 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.318 DEBUG:boto:Params: {} 2022-09-12T20:12:23.318 DEBUG:boto:Token: None 2022-09-12T20:12:23.319 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:23.319 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m9w0jWRH5rS/EIp6uq8uxOTh/A8= 2022-09-12T20:12:23.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m9w0jWRH5rS/EIp6uq8uxOTh/A8=', 'Content-Length': '0'} 2022-09-12T20:12:23.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000697bab8d468a7232-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.323 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:23.323 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:23.324 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:23.324 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:23.325 DEBUG:boto:Method: HEAD 2022-09-12T20:12:23.326 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:23.326 DEBUG:boto:Data: 2022-09-12T20:12:23.327 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.327 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.328 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.328 DEBUG:boto:Params: {} 2022-09-12T20:12:23.329 DEBUG:boto:Token: None 2022-09-12T20:12:23.329 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:23.330 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4MbBWaCJGb5cYgWAMY9MO6/wKuI= 2022-09-12T20:12:23.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4MbBWaCJGb5cYgWAMY9MO6/wKuI=', 'Content-Length': '0'} 2022-09-12T20:12:23.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000669f53e9b2006f80-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.333 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:23.334 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:23.334 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:23.335 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:23.335 DEBUG:boto:Method: HEAD 2022-09-12T20:12:23.336 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:23.336 DEBUG:boto:Data: 2022-09-12T20:12:23.337 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.337 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.338 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.339 DEBUG:boto:Params: {} 2022-09-12T20:12:23.339 DEBUG:boto:Token: None 2022-09-12T20:12:23.340 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:23.340 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4MbBWaCJGb5cYgWAMY9MO6/wKuI= 2022-09-12T20:12:23.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4MbBWaCJGb5cYgWAMY9MO6/wKuI=', 'Content-Length': '0'} 2022-09-12T20:12:23.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fad4b9355c47bd63-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:12:23.344 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:23.345 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:23.345 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:23.346 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:23.347 DEBUG:boto:Method: GET 2022-09-12T20:12:23.347 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:23.348 DEBUG:boto:Data: 2022-09-12T20:12:23.348 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.349 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.349 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.350 DEBUG:boto:Params: {} 2022-09-12T20:12:23.350 DEBUG:boto:Token: None 2022-09-12T20:12:23.351 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:23.352 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m9w0jWRH5rS/EIp6uq8uxOTh/A8= 2022-09-12T20:12:23.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m9w0jWRH5rS/EIp6uq8uxOTh/A8=', 'Content-Length': '0'} 2022-09-12T20:12:23.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086d59e9e49ed0984-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.356 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:23.356 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:23.357 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:23.357 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:12:23.358 DEBUG:boto:Method: GET 2022-09-12T20:12:23.358 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:23.359 DEBUG:boto:Data: 2022-09-12T20:12:23.359 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.360 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.360 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.361 DEBUG:boto:Params: {} 2022-09-12T20:12:23.362 DEBUG:boto:Token: None 2022-09-12T20:12:23.362 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:12:23.363 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m9w0jWRH5rS/EIp6uq8uxOTh/A8= 2022-09-12T20:12:23.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m9w0jWRH5rS/EIp6uq8uxOTh/A8=', 'Content-Length': '0'} 2022-09-12T20:12:23.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dadd5692874b6ed5-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.406 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:23.407 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:23.407 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:12:23.408 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:12:23.408 DEBUG:boto:Method: GET 2022-09-12T20:12:23.409 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:12:23.409 DEBUG:boto:Data: 2022-09-12T20:12:23.410 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.410 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.411 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.411 DEBUG:boto:Params: {} 2022-09-12T20:12:23.412 DEBUG:boto:Token: None 2022-09-12T20:12:23.413 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/ 2022-09-12T20:12:23.413 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc= 2022-09-12T20:12:23.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc=', 'Content-Length': '0'} 2022-09-12T20:12:23.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004903b6fd81b7ef44-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.417 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:23.418 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:23.419 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:23.419 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:12:23.420 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:12:23.420 DEBUG:boto:Method: GET 2022-09-12T20:12:23.421 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:12:23.421 DEBUG:boto:Data: 2022-09-12T20:12:23.422 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.423 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.423 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.424 DEBUG:boto:Params: {} 2022-09-12T20:12:23.424 DEBUG:boto:Token: None 2022-09-12T20:12:23.425 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/ 2022-09-12T20:12:23.425 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc= 2022-09-12T20:12:23.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc=', 'Content-Length': '0'} 2022-09-12T20:12:23.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb9b68f7c1a60d50-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.430 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:23.430 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:23.431 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:23.431 DEBUG:boto:Method: HEAD 2022-09-12T20:12:23.432 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:12:23.432 DEBUG:boto:Data: 2022-09-12T20:12:23.433 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.433 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.434 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.434 DEBUG:boto:Params: {} 2022-09-12T20:12:23.435 DEBUG:boto:Token: None 2022-09-12T20:12:23.436 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj87 2022-09-12T20:12:23.436 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ys2ENTM/H0d90W2QoN+GG5WWPpY= 2022-09-12T20:12:23.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ys2ENTM/H0d90W2QoN+GG5WWPpY=', 'Content-Length': '0'} 2022-09-12T20:12:23.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088c5f871d75e3d24-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.440 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:23.440 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:23.441 DEBUG:boto:Method: HEAD 2022-09-12T20:12:23.441 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:12:23.442 DEBUG:boto:Data: 2022-09-12T20:12:23.442 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.443 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.443 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.444 DEBUG:boto:Params: {} 2022-09-12T20:12:23.444 DEBUG:boto:Token: None 2022-09-12T20:12:23.445 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj87 2022-09-12T20:12:23.445 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ys2ENTM/H0d90W2QoN+GG5WWPpY= 2022-09-12T20:12:23.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ys2ENTM/H0d90W2QoN+GG5WWPpY=', 'Content-Length': '0'} 2022-09-12T20:12:23.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065c45c00e5d88a45-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:12:23.449 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:23.449 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:23.450 DEBUG:boto:Method: GET 2022-09-12T20:12:23.451 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:12:23.451 DEBUG:boto:Data: 2022-09-12T20:12:23.452 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.452 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.453 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.454 DEBUG:boto:Params: {} 2022-09-12T20:12:23.454 DEBUG:boto:Token: None 2022-09-12T20:12:23.455 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj87 2022-09-12T20:12:23.455 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pnS0vbo0T4Vx3Fw4f21bWTBhemc= 2022-09-12T20:12:23.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pnS0vbo0T4Vx3Fw4f21bWTBhemc=', 'Content-Length': '0'} 2022-09-12T20:12:23.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c19a8210073b1fb3-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.502 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:12:23.503 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:12:23.503 DEBUG:boto:Method: GET 2022-09-12T20:12:23.504 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:12:23.505 DEBUG:boto:Data: 2022-09-12T20:12:23.505 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.506 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.506 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.507 DEBUG:boto:Params: {} 2022-09-12T20:12:23.507 DEBUG:boto:Token: None 2022-09-12T20:12:23.508 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj87 2022-09-12T20:12:23.509 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pnS0vbo0T4Vx3Fw4f21bWTBhemc= 2022-09-12T20:12:23.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pnS0vbo0T4Vx3Fw4f21bWTBhemc=', 'Content-Length': '0'} 2022-09-12T20:12:23.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1000e74b16c8deb-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.512 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:23.513 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:23.513 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:12:23.514 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:12:23.514 DEBUG:boto:Method: GET 2022-09-12T20:12:23.515 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:12:23.515 DEBUG:boto:Data: 2022-09-12T20:12:23.516 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.516 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.517 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.518 DEBUG:boto:Params: {} 2022-09-12T20:12:23.518 DEBUG:boto:Token: None 2022-09-12T20:12:23.519 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/ 2022-09-12T20:12:23.519 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc= 2022-09-12T20:12:23.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc=', 'Content-Length': '0'} 2022-09-12T20:12:23.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd4ff58c63c7539c-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.524 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:23.525 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:23.525 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:23.526 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:12:23.526 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:12:23.527 DEBUG:boto:Method: GET 2022-09-12T20:12:23.527 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:12:23.528 DEBUG:boto:Data: 2022-09-12T20:12:23.528 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.529 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.529 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.530 DEBUG:boto:Params: {} 2022-09-12T20:12:23.531 DEBUG:boto:Token: None 2022-09-12T20:12:23.531 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/ 2022-09-12T20:12:23.532 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc= 2022-09-12T20:12:23.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc=', 'Content-Length': '0'} 2022-09-12T20:12:23.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1bed40be7aee783-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.536 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:23.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:12:23.537 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:23.538 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:23.538 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:23.539 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:23.539 DEBUG:boto:Method: GET 2022-09-12T20:12:23.540 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:23.540 DEBUG:boto:Data: 2022-09-12T20:12:23.541 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.541 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.542 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.542 DEBUG:boto:Params: {} 2022-09-12T20:12:23.543 DEBUG:boto:Token: None 2022-09-12T20:12:23.544 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:23.544 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZQRy3kLL/9LaKeTpWPwGP6jwIYY= 2022-09-12T20:12:23.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZQRy3kLL/9LaKeTpWPwGP6jwIYY=', 'Content-Length': '0'} 2022-09-12T20:12:23.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042a1abacfd9671f3-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.548 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:23.548 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:23.549 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:23.549 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:23.550 DEBUG:boto:Method: GET 2022-09-12T20:12:23.551 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:23.551 DEBUG:boto:Data: 2022-09-12T20:12:23.552 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.552 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.553 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.553 DEBUG:boto:Params: {} 2022-09-12T20:12:23.554 DEBUG:boto:Token: None 2022-09-12T20:12:23.554 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:23.555 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZQRy3kLL/9LaKeTpWPwGP6jwIYY= 2022-09-12T20:12:23.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZQRy3kLL/9LaKeTpWPwGP6jwIYY=', 'Content-Length': '0'} 2022-09-12T20:12:23.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c63ce186750cd69-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.559 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:23.559 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:23.560 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:23.560 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:23.561 DEBUG:boto:Method: HEAD 2022-09-12T20:12:23.561 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:23.562 DEBUG:boto:Data: 2022-09-12T20:12:23.562 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.563 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.563 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.564 DEBUG:boto:Params: {} 2022-09-12T20:12:23.565 DEBUG:boto:Token: None 2022-09-12T20:12:23.565 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:23.566 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kwSPxFSS8XVpQbXxWkG2jm+wizY= 2022-09-12T20:12:23.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kwSPxFSS8XVpQbXxWkG2jm+wizY=', 'Content-Length': '0'} 2022-09-12T20:12:23.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bd423d579aaf72f-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.569 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:23.570 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:23.570 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:23.571 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:23.571 DEBUG:boto:Method: HEAD 2022-09-12T20:12:23.572 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:23.572 DEBUG:boto:Data: 2022-09-12T20:12:23.573 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.573 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.574 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.574 DEBUG:boto:Params: {} 2022-09-12T20:12:23.575 DEBUG:boto:Token: None 2022-09-12T20:12:23.575 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:23.576 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kwSPxFSS8XVpQbXxWkG2jm+wizY= 2022-09-12T20:12:23.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kwSPxFSS8XVpQbXxWkG2jm+wizY=', 'Content-Length': '0'} 2022-09-12T20:12:23.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c52603eb7101ff49-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:12:23.580 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:23.580 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:23.581 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:23.581 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:23.582 DEBUG:boto:Method: GET 2022-09-12T20:12:23.583 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:23.583 DEBUG:boto:Data: 2022-09-12T20:12:23.584 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.584 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.585 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.585 DEBUG:boto:Params: {} 2022-09-12T20:12:23.586 DEBUG:boto:Token: None 2022-09-12T20:12:23.586 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:23.587 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZQRy3kLL/9LaKeTpWPwGP6jwIYY= 2022-09-12T20:12:23.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZQRy3kLL/9LaKeTpWPwGP6jwIYY=', 'Content-Length': '0'} 2022-09-12T20:12:23.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c360ba55f95337c1-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.634 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:23.635 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:23.635 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:23.636 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:12:23.637 DEBUG:boto:Method: GET 2022-09-12T20:12:23.637 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:23.638 DEBUG:boto:Data: 2022-09-12T20:12:23.638 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.639 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.639 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.640 DEBUG:boto:Params: {} 2022-09-12T20:12:23.640 DEBUG:boto:Token: None 2022-09-12T20:12:23.641 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:12:23.642 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZQRy3kLL/9LaKeTpWPwGP6jwIYY= 2022-09-12T20:12:23.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZQRy3kLL/9LaKeTpWPwGP6jwIYY=', 'Content-Length': '0'} 2022-09-12T20:12:23.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e14176dd9cd667b-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.645 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:23.646 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:23.646 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:12:23.647 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:12:23.647 DEBUG:boto:Method: GET 2022-09-12T20:12:23.648 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:12:23.648 DEBUG:boto:Data: 2022-09-12T20:12:23.649 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.649 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.650 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.650 DEBUG:boto:Params: {} 2022-09-12T20:12:23.651 DEBUG:boto:Token: None 2022-09-12T20:12:23.652 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/ 2022-09-12T20:12:23.652 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc= 2022-09-12T20:12:23.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc=', 'Content-Length': '0'} 2022-09-12T20:12:23.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d83e899e82a6737-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.656 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:23.657 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:23.658 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:23.659 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:12:23.659 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:12:23.660 DEBUG:boto:Method: GET 2022-09-12T20:12:23.660 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:12:23.661 DEBUG:boto:Data: 2022-09-12T20:12:23.661 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.662 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.662 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.663 DEBUG:boto:Params: {} 2022-09-12T20:12:23.663 DEBUG:boto:Token: None 2022-09-12T20:12:23.664 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/ 2022-09-12T20:12:23.664 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc= 2022-09-12T20:12:23.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc=', 'Content-Length': '0'} 2022-09-12T20:12:23.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090be8e98da0fe33d-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.669 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:23.670 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:23.670 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:23.671 DEBUG:boto:Method: HEAD 2022-09-12T20:12:23.671 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:12:23.672 DEBUG:boto:Data: 2022-09-12T20:12:23.672 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.673 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.673 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.674 DEBUG:boto:Params: {} 2022-09-12T20:12:23.674 DEBUG:boto:Token: None 2022-09-12T20:12:23.675 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj92 2022-09-12T20:12:23.675 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:v7Zm1rQqD3GT0uoqNed7r8dM1Xk= 2022-09-12T20:12:23.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:v7Zm1rQqD3GT0uoqNed7r8dM1Xk=', 'Content-Length': '0'} 2022-09-12T20:12:23.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e418223c4886b17-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.679 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:23.679 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:23.680 DEBUG:boto:Method: HEAD 2022-09-12T20:12:23.680 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:12:23.681 DEBUG:boto:Data: 2022-09-12T20:12:23.681 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.682 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.682 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.683 DEBUG:boto:Params: {} 2022-09-12T20:12:23.683 DEBUG:boto:Token: None 2022-09-12T20:12:23.684 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj92 2022-09-12T20:12:23.685 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:v7Zm1rQqD3GT0uoqNed7r8dM1Xk= 2022-09-12T20:12:23.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:v7Zm1rQqD3GT0uoqNed7r8dM1Xk=', 'Content-Length': '0'} 2022-09-12T20:12:23.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fb2a996766d5ac1-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:12:23.688 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:23.689 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:23.690 DEBUG:boto:Method: GET 2022-09-12T20:12:23.690 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:12:23.691 DEBUG:boto:Data: 2022-09-12T20:12:23.691 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.692 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.692 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.693 DEBUG:boto:Params: {} 2022-09-12T20:12:23.694 DEBUG:boto:Token: None 2022-09-12T20:12:23.694 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj92 2022-09-12T20:12:23.695 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IIzUF40B5+rdqVS5CYGq+V8fH/4= 2022-09-12T20:12:23.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IIzUF40B5+rdqVS5CYGq+V8fH/4=', 'Content-Length': '0'} 2022-09-12T20:12:23.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfc2d2079fcda033-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.738 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:12:23.739 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:12:23.739 DEBUG:boto:Method: GET 2022-09-12T20:12:23.740 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:12:23.740 DEBUG:boto:Data: 2022-09-12T20:12:23.741 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.742 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.742 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.743 DEBUG:boto:Params: {} 2022-09-12T20:12:23.743 DEBUG:boto:Token: None 2022-09-12T20:12:23.744 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj92 2022-09-12T20:12:23.745 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IIzUF40B5+rdqVS5CYGq+V8fH/4= 2022-09-12T20:12:23.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IIzUF40B5+rdqVS5CYGq+V8fH/4=', 'Content-Length': '0'} 2022-09-12T20:12:23.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0c275fe1a99d7eb-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.748 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:23.749 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:23.749 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:12:23.750 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:12:23.750 DEBUG:boto:Method: GET 2022-09-12T20:12:23.751 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:12:23.751 DEBUG:boto:Data: 2022-09-12T20:12:23.752 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.752 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.753 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.753 DEBUG:boto:Params: {} 2022-09-12T20:12:23.754 DEBUG:boto:Token: None 2022-09-12T20:12:23.755 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/ 2022-09-12T20:12:23.755 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc= 2022-09-12T20:12:23.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc=', 'Content-Length': '0'} 2022-09-12T20:12:23.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015d2a2e0183c0cc8-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.759 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:23.760 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:23.761 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:23.761 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:12:23.762 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:12:23.762 DEBUG:boto:Method: GET 2022-09-12T20:12:23.763 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:12:23.764 DEBUG:boto:Data: 2022-09-12T20:12:23.764 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.765 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.765 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.766 DEBUG:boto:Params: {} 2022-09-12T20:12:23.766 DEBUG:boto:Token: None 2022-09-12T20:12:23.767 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/ 2022-09-12T20:12:23.768 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc= 2022-09-12T20:12:23.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc=', 'Content-Length': '0'} 2022-09-12T20:12:23.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011645f46e3c53303-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.772 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:23.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:12:23.773 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:23.774 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:23.774 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:23.775 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:23.775 DEBUG:boto:Method: GET 2022-09-12T20:12:23.776 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:23.776 DEBUG:boto:Data: 2022-09-12T20:12:23.777 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.777 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.778 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.778 DEBUG:boto:Params: {} 2022-09-12T20:12:23.779 DEBUG:boto:Token: None 2022-09-12T20:12:23.780 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:23.780 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FiFVcm4QL6dHFEMN7XksztjgBVs= 2022-09-12T20:12:23.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FiFVcm4QL6dHFEMN7XksztjgBVs=', 'Content-Length': '0'} 2022-09-12T20:12:23.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013567597d52b1b6c-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.784 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:23.784 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:23.785 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:23.786 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:23.786 DEBUG:boto:Method: GET 2022-09-12T20:12:23.787 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:23.787 DEBUG:boto:Data: 2022-09-12T20:12:23.788 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.788 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.789 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.789 DEBUG:boto:Params: {} 2022-09-12T20:12:23.790 DEBUG:boto:Token: None 2022-09-12T20:12:23.791 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:23.791 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FiFVcm4QL6dHFEMN7XksztjgBVs= 2022-09-12T20:12:23.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FiFVcm4QL6dHFEMN7XksztjgBVs=', 'Content-Length': '0'} 2022-09-12T20:12:23.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009883e6ee954f346c-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.795 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:23.795 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:23.796 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:23.796 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:23.797 DEBUG:boto:Method: HEAD 2022-09-12T20:12:23.797 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:23.798 DEBUG:boto:Data: 2022-09-12T20:12:23.798 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.799 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.799 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.800 DEBUG:boto:Params: {} 2022-09-12T20:12:23.800 DEBUG:boto:Token: None 2022-09-12T20:12:23.801 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:23.802 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ivCSM0dqbX1qigBjUjCMQ2iEnZc= 2022-09-12T20:12:23.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ivCSM0dqbX1qigBjUjCMQ2iEnZc=', 'Content-Length': '0'} 2022-09-12T20:12:23.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000942cb6c936785647-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.805 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:23.805 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:23.806 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:23.807 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:23.807 DEBUG:boto:Method: HEAD 2022-09-12T20:12:23.808 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:23.808 DEBUG:boto:Data: 2022-09-12T20:12:23.809 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.809 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.810 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.810 DEBUG:boto:Params: {} 2022-09-12T20:12:23.811 DEBUG:boto:Token: None 2022-09-12T20:12:23.811 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:23.812 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ivCSM0dqbX1qigBjUjCMQ2iEnZc= 2022-09-12T20:12:23.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ivCSM0dqbX1qigBjUjCMQ2iEnZc=', 'Content-Length': '0'} 2022-09-12T20:12:23.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076d230bee7b59655-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:12:23.816 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:23.816 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:23.817 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:23.817 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:23.818 DEBUG:boto:Method: GET 2022-09-12T20:12:23.818 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:23.819 DEBUG:boto:Data: 2022-09-12T20:12:23.819 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.820 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.820 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.821 DEBUG:boto:Params: {} 2022-09-12T20:12:23.822 DEBUG:boto:Token: None 2022-09-12T20:12:23.822 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:23.823 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FiFVcm4QL6dHFEMN7XksztjgBVs= 2022-09-12T20:12:23.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FiFVcm4QL6dHFEMN7XksztjgBVs=', 'Content-Length': '0'} 2022-09-12T20:12:23.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074a142e7bbdbcef0-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.826 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:23.827 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:23.828 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:23.828 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:12:23.829 DEBUG:boto:Method: GET 2022-09-12T20:12:23.829 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:23.830 DEBUG:boto:Data: 2022-09-12T20:12:23.830 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.831 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.831 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.832 DEBUG:boto:Params: {} 2022-09-12T20:12:23.832 DEBUG:boto:Token: None 2022-09-12T20:12:23.833 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:12:23.834 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FiFVcm4QL6dHFEMN7XksztjgBVs= 2022-09-12T20:12:23.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FiFVcm4QL6dHFEMN7XksztjgBVs=', 'Content-Length': '0'} 2022-09-12T20:12:23.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd1f357da368c632-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.878 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:23.879 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:23.879 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:12:23.880 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:12:23.880 DEBUG:boto:Method: GET 2022-09-12T20:12:23.881 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:12:23.881 DEBUG:boto:Data: 2022-09-12T20:12:23.882 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.882 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.883 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.883 DEBUG:boto:Params: {} 2022-09-12T20:12:23.884 DEBUG:boto:Token: None 2022-09-12T20:12:23.885 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/ 2022-09-12T20:12:23.885 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc= 2022-09-12T20:12:23.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc=', 'Content-Length': '0'} 2022-09-12T20:12:23.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8a7c42db9e509e5-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.889 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:23.890 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:23.891 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:23.891 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:12:23.892 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:12:23.892 DEBUG:boto:Method: GET 2022-09-12T20:12:23.893 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:12:23.893 DEBUG:boto:Data: 2022-09-12T20:12:23.894 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.895 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.895 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.896 DEBUG:boto:Params: {} 2022-09-12T20:12:23.896 DEBUG:boto:Token: None 2022-09-12T20:12:23.897 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/ 2022-09-12T20:12:23.897 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc= 2022-09-12T20:12:23.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc=', 'Content-Length': '0'} 2022-09-12T20:12:23.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007029ab89bed4ad89-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.902 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:23.902 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:23.903 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:23.903 DEBUG:boto:Method: HEAD 2022-09-12T20:12:23.904 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:12:23.904 DEBUG:boto:Data: 2022-09-12T20:12:23.905 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.906 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.906 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.906 DEBUG:boto:Params: {} 2022-09-12T20:12:23.907 DEBUG:boto:Token: None 2022-09-12T20:12:23.908 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj95 2022-09-12T20:12:23.908 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R3QLaiT+xt/xKZK+MGOVR68ZOSY= 2022-09-12T20:12:23.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R3QLaiT+xt/xKZK+MGOVR68ZOSY=', 'Content-Length': '0'} 2022-09-12T20:12:23.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e18c6d988eb07112-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.912 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:23.912 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:23.913 DEBUG:boto:Method: HEAD 2022-09-12T20:12:23.913 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:12:23.914 DEBUG:boto:Data: 2022-09-12T20:12:23.914 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.915 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.915 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.916 DEBUG:boto:Params: {} 2022-09-12T20:12:23.916 DEBUG:boto:Token: None 2022-09-12T20:12:23.917 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj95 2022-09-12T20:12:23.918 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R3QLaiT+xt/xKZK+MGOVR68ZOSY= 2022-09-12T20:12:23.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R3QLaiT+xt/xKZK+MGOVR68ZOSY=', 'Content-Length': '0'} 2022-09-12T20:12:23.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079f3044f1b75b2d0-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:12:23.922 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:23.922 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:23.923 DEBUG:boto:Method: GET 2022-09-12T20:12:23.923 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:12:23.924 DEBUG:boto:Data: 2022-09-12T20:12:23.924 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.925 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.926 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.926 DEBUG:boto:Params: {} 2022-09-12T20:12:23.927 DEBUG:boto:Token: None 2022-09-12T20:12:23.927 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj95 2022-09-12T20:12:23.928 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PLThNVS+cKV58OSF6Tdbihf7r/k= 2022-09-12T20:12:23.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PLThNVS+cKV58OSF6Tdbihf7r/k=', 'Content-Length': '0'} 2022-09-12T20:12:23.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000994596d2ea4dfac0-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.974 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:12:23.975 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:12:23.975 DEBUG:boto:Method: GET 2022-09-12T20:12:23.976 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:12:23.976 DEBUG:boto:Data: 2022-09-12T20:12:23.977 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.977 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:23.978 DEBUG:boto:Port: 8000 2022-09-12T20:12:23.978 DEBUG:boto:Params: {} 2022-09-12T20:12:23.979 DEBUG:boto:Token: None 2022-09-12T20:12:23.980 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/obj95 2022-09-12T20:12:23.980 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PLThNVS+cKV58OSF6Tdbihf7r/k= 2022-09-12T20:12:23.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PLThNVS+cKV58OSF6Tdbihf7r/k=', 'Content-Length': '0'} 2022-09-12T20:12:23.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f407e12992831ef1-00631f92a7-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.984 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:23.984 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:23.985 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:12:23.985 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:12:23.986 DEBUG:boto:Method: GET 2022-09-12T20:12:23.987 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:12:23.987 DEBUG:boto:Data: 2022-09-12T20:12:23.988 DEBUG:boto:Headers: {} 2022-09-12T20:12:23.988 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:12:23.989 DEBUG:boto:Port: 8001 2022-09-12T20:12:23.989 DEBUG:boto:Params: {} 2022-09-12T20:12:23.990 DEBUG:boto:Token: None 2022-09-12T20:12:23.990 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:23 GMT /jdpiak-75/ 2022-09-12T20:12:23.991 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc= 2022-09-12T20:12:23.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YyA3eG6y2NGsK3Enr71ddjjwOMc=', 'Content-Length': '0'} 2022-09-12T20:12:23.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc2f507f1be2582b-00631f92a7-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:23.995 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:23.996 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:23.997 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:23.997 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:12:23.998 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:12:23.998 DEBUG:boto:Method: GET 2022-09-12T20:12:23.999 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:12:23.999 DEBUG:boto:Data: 2022-09-12T20:12:24.000 DEBUG:boto:Headers: {} 2022-09-12T20:12:24.000 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:12:24.001 DEBUG:boto:Port: 8000 2022-09-12T20:12:24.001 DEBUG:boto:Params: {} 2022-09-12T20:12:24.002 DEBUG:boto:Token: None 2022-09-12T20:12:24.003 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:12:24 GMT /jdpiak-75/ 2022-09-12T20:12:24.003 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:K99ypiyv3UWOPdqB41oA5aWiKWo= 2022-09-12T20:12:24.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:K99ypiyv3UWOPdqB41oA5aWiKWo=', 'Content-Length': '0'} 2022-09-12T20:12:24.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001503784ad98f7de2-00631f92a8-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:24.007 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:12:24.008 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-75 zones={test-zone3, test-zone2} 2022-09-12T20:12:24.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:12:24.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:12:24.010 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T20:12:24.074 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:12:24.075 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:12:24.152 INFO:teuthology.orchestra.run.smithi035.stdout:tenant: 2022-09-12T20:12:24.153 INFO:teuthology.orchestra.run.smithi035.stdout:bucket name: jdpiak-75 2022-09-12T20:12:24.166 INFO:teuthology.orchestra.run.smithi035.stdout:total entries: 25 2022-09-12T20:12:24.246 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:24.247 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:24.247 DEBUG:boto:Method: HEAD 2022-09-12T20:12:24.248 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:12:24.248 DEBUG:boto:Data: 2022-09-12T20:12:24.249 DEBUG:boto:Headers: {} 2022-09-12T20:12:24.249 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:24.250 DEBUG:boto:Port: 8000 2022-09-12T20:12:24.250 DEBUG:boto:Params: {} 2022-09-12T20:12:24.251 DEBUG:boto:Token: None 2022-09-12T20:12:24.251 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:24 GMT /jdpiak-75/ 2022-09-12T20:12:24.252 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4fGo94yHm2zGLHQYSYlnNA/n0aQ= 2022-09-12T20:12:24.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4fGo94yHm2zGLHQYSYlnNA/n0aQ=', 'Content-Length': '0'} 2022-09-12T20:12:24.257 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000813d2c57a964786-00631f92a8-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:24.257 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:12:24.258 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:12:24.259 DEBUG:boto:Method: PUT 2022-09-12T20:12:24.259 DEBUG:boto:Path: /jdpiak-75/obj31 2022-09-12T20:12:24.260 DEBUG:boto:Data: 2022-09-12T20:12:24.260 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:12:24.261 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:24.261 DEBUG:boto:Port: 8000 2022-09-12T20:12:24.262 DEBUG:boto:Params: {} 2022-09-12T20:12:24.262 DEBUG:boto:Token: None 2022-09-12T20:12:24.263 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:12:24 GMT /jdpiak-75/obj31 2022-09-12T20:12:24.263 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R2G/VwTeKQbaUcD6Z0ljaWDp8b0= 2022-09-12T20:12:24.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:12:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R2G/VwTeKQbaUcD6Z0ljaWDp8b0='} 2022-09-12T20:12:24.306 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000009935ab6ac03e7b9-00631f92a8-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:24.307 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:24.307 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:24.308 DEBUG:boto:Method: HEAD 2022-09-12T20:12:24.308 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:12:24.309 DEBUG:boto:Data: 2022-09-12T20:12:24.309 DEBUG:boto:Headers: {} 2022-09-12T20:12:24.310 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:24.310 DEBUG:boto:Port: 8000 2022-09-12T20:12:24.311 DEBUG:boto:Params: {} 2022-09-12T20:12:24.311 DEBUG:boto:Token: None 2022-09-12T20:12:24.312 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:24 GMT /jdpiak-75/ 2022-09-12T20:12:24.313 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4fGo94yHm2zGLHQYSYlnNA/n0aQ= 2022-09-12T20:12:24.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4fGo94yHm2zGLHQYSYlnNA/n0aQ=', 'Content-Length': '0'} 2022-09-12T20:12:24.318 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005fb36e547b554fa-00631f92a8-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:24.319 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:12:24.319 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:12:24.320 DEBUG:boto:Method: PUT 2022-09-12T20:12:24.321 DEBUG:boto:Path: /jdpiak-75/obj62 2022-09-12T20:12:24.322 DEBUG:boto:Data: 2022-09-12T20:12:24.322 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:12:24.323 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:24.323 DEBUG:boto:Port: 8000 2022-09-12T20:12:24.324 DEBUG:boto:Params: {} 2022-09-12T20:12:24.324 DEBUG:boto:Token: None 2022-09-12T20:12:24.325 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:12:24 GMT /jdpiak-75/obj62 2022-09-12T20:12:24.326 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uEyFgVDASQds03eZp0gk8KPt4yU= 2022-09-12T20:12:24.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:12:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uEyFgVDASQds03eZp0gk8KPt4yU='} 2022-09-12T20:12:24.332 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005311de5ca7f4f80a-00631f92a8-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:24.333 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:24.333 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:24.334 DEBUG:boto:Method: HEAD 2022-09-12T20:12:24.334 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:12:24.335 DEBUG:boto:Data: 2022-09-12T20:12:24.335 DEBUG:boto:Headers: {} 2022-09-12T20:12:24.336 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:24.336 DEBUG:boto:Port: 8000 2022-09-12T20:12:24.337 DEBUG:boto:Params: {} 2022-09-12T20:12:24.338 DEBUG:boto:Token: None 2022-09-12T20:12:24.338 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:24 GMT /jdpiak-75/ 2022-09-12T20:12:24.339 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4fGo94yHm2zGLHQYSYlnNA/n0aQ= 2022-09-12T20:12:24.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4fGo94yHm2zGLHQYSYlnNA/n0aQ=', 'Content-Length': '0'} 2022-09-12T20:12:24.343 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4cb392440ffff48-00631f92a8-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:24.344 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:12:24.344 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:12:24.345 DEBUG:boto:Method: PUT 2022-09-12T20:12:24.345 DEBUG:boto:Path: /jdpiak-75/obj93 2022-09-12T20:12:24.346 DEBUG:boto:Data: 2022-09-12T20:12:24.346 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:12:24.347 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:24.347 DEBUG:boto:Port: 8000 2022-09-12T20:12:24.348 DEBUG:boto:Params: {} 2022-09-12T20:12:24.349 DEBUG:boto:Token: None 2022-09-12T20:12:24.349 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:12:24 GMT /jdpiak-75/obj93 2022-09-12T20:12:24.350 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Peq1UpGzEf6AwtycsdL9IDhZ2ZA= 2022-09-12T20:12:24.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:12:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Peq1UpGzEf6AwtycsdL9IDhZ2ZA='} 2022-09-12T20:12:24.394 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000272197e3006fbc0d-00631f92a8-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:24.395 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:24.395 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:24.396 DEBUG:boto:Method: HEAD 2022-09-12T20:12:24.396 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:12:24.397 DEBUG:boto:Data: 2022-09-12T20:12:24.397 DEBUG:boto:Headers: {} 2022-09-12T20:12:24.398 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:24.398 DEBUG:boto:Port: 8000 2022-09-12T20:12:24.399 DEBUG:boto:Params: {} 2022-09-12T20:12:24.400 DEBUG:boto:Token: None 2022-09-12T20:12:24.400 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:24 GMT /jdpiak-75/ 2022-09-12T20:12:24.401 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4fGo94yHm2zGLHQYSYlnNA/n0aQ= 2022-09-12T20:12:24.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4fGo94yHm2zGLHQYSYlnNA/n0aQ=', 'Content-Length': '0'} 2022-09-12T20:12:24.405 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042633a1577691568-00631f92a8-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:24.406 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:12:24.407 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:12:24.407 DEBUG:boto:Method: PUT 2022-09-12T20:12:24.408 DEBUG:boto:Path: /jdpiak-75/obj124 2022-09-12T20:12:24.408 DEBUG:boto:Data: 2022-09-12T20:12:24.409 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:12:24.409 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:24.410 DEBUG:boto:Port: 8000 2022-09-12T20:12:24.410 DEBUG:boto:Params: {} 2022-09-12T20:12:24.411 DEBUG:boto:Token: None 2022-09-12T20:12:24.412 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:12:24 GMT /jdpiak-75/obj124 2022-09-12T20:12:24.412 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CxfN4ZriiaBxk3QrNHDSLXek+UQ= 2022-09-12T20:12:24.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:12:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CxfN4ZriiaBxk3QrNHDSLXek+UQ='} 2022-09-12T20:12:24.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000952605eb92079d1a-00631f92a8-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:24.422 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:12:24.423 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:12:24.423 DEBUG:boto:Method: HEAD 2022-09-12T20:12:24.424 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:12:24.424 DEBUG:boto:Data: 2022-09-12T20:12:24.425 DEBUG:boto:Headers: {} 2022-09-12T20:12:24.425 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:24.426 DEBUG:boto:Port: 8000 2022-09-12T20:12:24.426 DEBUG:boto:Params: {} 2022-09-12T20:12:24.427 DEBUG:boto:Token: None 2022-09-12T20:12:24.428 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:12:24 GMT /jdpiak-75/ 2022-09-12T20:12:24.428 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4fGo94yHm2zGLHQYSYlnNA/n0aQ= 2022-09-12T20:12:24.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:12:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4fGo94yHm2zGLHQYSYlnNA/n0aQ=', 'Content-Length': '0'} 2022-09-12T20:12:24.432 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000245b3203acf4e819-00631f92a8-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:24.433 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:12:24.434 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:12:24.434 DEBUG:boto:Method: PUT 2022-09-12T20:12:24.435 DEBUG:boto:Path: /jdpiak-75/obj155 2022-09-12T20:12:24.435 DEBUG:boto:Data: 2022-09-12T20:12:24.436 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:12:24.436 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:12:24.437 DEBUG:boto:Port: 8000 2022-09-12T20:12:24.438 DEBUG:boto:Params: {} 2022-09-12T20:12:24.438 DEBUG:boto:Token: None 2022-09-12T20:12:24.439 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:12:24 GMT /jdpiak-75/obj155 2022-09-12T20:12:24.439 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UU7ZmHe+XfsjTnGpZASLhuUXY7I= 2022-09-12T20:12:24.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:12:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UU7ZmHe+XfsjTnGpZASLhuUXY7I='} 2022-09-12T20:12:24.482 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fbf4af000a4b6a93-00631f92a8-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:12:24.483 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-75 2022-09-12T20:12:24.483 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:12:24.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:12:24.485 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T20:12:24.550 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:12:24.550 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:12:24.594 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:12:24.592+0000 7f2e91dd9600 1 waiting to reach latest gen 5, on 4.. 2022-09-12T20:12:29.595 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:12:29.593+0000 7f2e91dd9600 1 waiting to reach latest gen 5, on 4.. 2022-09-12T20:12:34.596 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:12:34.594+0000 7f2e91dd9600 1 waiting to reach latest gen 5, on 4.. 2022-09-12T20:12:39.596 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:12:39.595+0000 7f2e91dd9600 1 waiting to reach latest gen 5, on 4.. 2022-09-12T20:12:44.599 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:12:44.597+0000 7f2e91dd9600 1 waiting for incremental sync to catch up: 2022-09-12T20:12:44.599 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:12:44.600 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , 00000000002.59.5, , 00000000001.55.5, , 00000000001.78.5, , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , 00000000001.59.5] 2022-09-12T20:12:49.601 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:12:49.599+0000 7f2e91dd9600 1 waiting for incremental sync to catch up: 2022-09-12T20:12:49.602 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:12:49.602 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , 00000000002.59.5, , 00000000001.55.5, , 00000000001.78.5, , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , 00000000001.59.5] 2022-09-12T20:12:54.603 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:12:54.601+0000 7f2e91dd9600 1 waiting for incremental sync to catch up: 2022-09-12T20:12:54.604 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:12:54.604 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , 00000000002.59.5, , 00000000001.55.5, , 00000000001.78.5, , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , 00000000001.59.5] 2022-09-12T20:12:59.606 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:12:59.604+0000 7f2e91dd9600 1 waiting for incremental sync to catch up: 2022-09-12T20:12:59.607 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:12:59.607 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , 00000000002.59.5, , 00000000001.55.5, , 00000000001.78.5, , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , 00000000001.59.5] 2022-09-12T20:13:04.609 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:13:04.606+0000 7f2e91dd9600 1 waiting for incremental sync to catch up: 2022-09-12T20:13:04.609 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:13:04.609 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , 00000000002.59.5, , 00000000001.55.5, , 00000000001.78.5, , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , 00000000001.59.5] 2022-09-12T20:13:09.611 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:13:09.609+0000 7f2e91dd9600 1 waiting for incremental sync to catch up: 2022-09-12T20:13:09.611 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:13:09.612 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , 00000000002.59.5, , 00000000001.55.5, , 00000000001.78.5, , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , 00000000001.59.5] 2022-09-12T20:13:14.613 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:13:14.611+0000 7f2e91dd9600 1 waiting for incremental sync to catch up: 2022-09-12T20:13:14.614 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:13:14.614 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , 00000000002.59.5, , 00000000001.55.5, , 00000000001.78.5, , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , 00000000001.59.5] 2022-09-12T20:13:19.616 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:13:19.613+0000 7f2e91dd9600 1 waiting for incremental sync to catch up: 2022-09-12T20:13:19.616 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:13:19.616 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , 00000000002.59.5, , 00000000001.55.5, , 00000000001.78.5, , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , 00000000001.59.5] 2022-09-12T20:13:24.618 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:13:24.617+0000 7f2e91dd9600 1 waiting for incremental sync to catch up: 2022-09-12T20:13:24.618 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , 00000000001.55.5, , , , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , ] 2022-09-12T20:13:24.619 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , 00000000002.59.5, , 00000000001.55.5, , 00000000001.78.5, , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , 00000000001.59.5] 2022-09-12T20:13:29.620 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:13:29.618+0000 7f2e91dd9600 1 waiting for incremental sync to catch up: 2022-09-12T20:13:29.621 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , 00000000001.55.5, , , , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , ] 2022-09-12T20:13:29.621 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , 00000000002.59.5, , 00000000001.55.5, , 00000000001.78.5, , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , 00000000001.59.5] 2022-09-12T20:13:34.623 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:13:34.620+0000 7f2e91dd9600 1 waiting for incremental sync to catch up: 2022-09-12T20:13:34.623 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , 00000000001.55.5, , , , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , ] 2022-09-12T20:13:34.623 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , 00000000002.59.5, , 00000000001.55.5, , 00000000001.78.5, , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , 00000000001.59.5] 2022-09-12T20:13:39.625 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:13:39.623+0000 7f2e91dd9600 1 waiting for incremental sync to catch up: 2022-09-12T20:13:39.626 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , 00000000001.55.5, , , , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , ] 2022-09-12T20:13:39.627 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , 00000000002.59.5, , 00000000001.55.5, , 00000000001.78.5, , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , 00000000001.59.5] 2022-09-12T20:13:44.628 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:13:44.625+0000 7f2e91dd9600 1 waiting for incremental sync to catch up: 2022-09-12T20:13:44.628 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , 00000000001.55.5, , , , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , ] 2022-09-12T20:13:44.628 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , 00000000002.59.5, , 00000000001.55.5, , 00000000001.78.5, , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , 00000000001.59.5] 2022-09-12T20:13:49.630 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:13:49.628+0000 7f2e91dd9600 1 waiting for incremental sync to catch up: 2022-09-12T20:13:49.631 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , 00000000001.55.5, , , , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , ] 2022-09-12T20:13:49.631 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , 00000000002.59.5, , 00000000001.55.5, , 00000000001.78.5, , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , 00000000001.59.5] 2022-09-12T20:13:54.633 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:13:54.630+0000 7f2e91dd9600 1 waiting for incremental sync to catch up: 2022-09-12T20:13:54.633 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , 00000000001.55.5, , , , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , ] 2022-09-12T20:13:54.633 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , 00000000002.59.5, , 00000000001.55.5, , 00000000001.78.5, , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , 00000000001.59.5] 2022-09-12T20:13:59.635 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:13:59.633+0000 7f2e91dd9600 1 waiting for incremental sync to catch up: 2022-09-12T20:13:59.636 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , 00000000001.55.5, , , , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , ] 2022-09-12T20:13:59.636 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , 00000000002.59.5, , 00000000001.55.5, , 00000000001.78.5, , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , 00000000001.59.5] 2022-09-12T20:14:04.638 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:14:04.635+0000 7f2e91dd9600 1 bucket sync caught up with source: 2022-09-12T20:14:04.638 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , 00000000002.59.5, , 00000000001.55.5, , 00000000001.78.5, , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , 00000000001.59.5] 2022-09-12T20:14:04.638 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , 00000000002.59.5, , 00000000001.55.5, , 00000000001.78.5, , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , 00000000001.59.5] 2022-09-12T20:14:04.639 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:14:04.635+0000 7f2e91dd9600 0 bucket checkpoint complete 2022-09-12T20:14:04.658 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-75 2022-09-12T20:14:04.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:14:04.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:14:04.661 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:14:04.727 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:14:04.728 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:14:04.771 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:14:04.770+0000 7f3c17aa6600 1 bucket sync caught up with source: 2022-09-12T20:14:04.771 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , 00000000002.59.5, , 00000000001.55.5, , 00000000001.78.5, , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , 00000000001.59.5] 2022-09-12T20:14:04.771 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , 00000000002.59.5, , 00000000001.55.5, , 00000000001.78.5, , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , 00000000001.59.5] 2022-09-12T20:14:04.772 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:14:04.770+0000 7f3c17aa6600 0 bucket checkpoint complete 2022-09-12T20:14:04.786 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-75 2022-09-12T20:14:04.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:14:04.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:14:04.788 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T20:14:04.851 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:14:04.851 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:14:04.907 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:14:04.905+0000 7fc8d6638600 1 bucket sync caught up with source: 2022-09-12T20:14:04.907 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , 00000000002.59.5, , 00000000001.55.5, , 00000000001.78.5, , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , 00000000001.59.5] 2022-09-12T20:14:04.908 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , 00000000002.59.5, , 00000000001.55.5, , 00000000001.78.5, , , , , , , , , , 00000000001.51.5, , , , , , , , , , , , , , 00000000001.59.5] 2022-09-12T20:14:04.908 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:14:04.905+0000 7fc8d6638600 0 bucket checkpoint complete 2022-09-12T20:14:04.920 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-75 2022-09-12T20:14:04.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:14:04.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:14:04.922 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:14:04.983 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:14:04.983 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:14:05.029 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:14:05.029+0000 7f1c58a50600 1 bucket sync caught up with source: 2022-09-12T20:14:05.030 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.648.5, 00000000002.837.5, 00000000002.643.5, 00000000004.355.5, 00000000004.935.5, 00000000001.434.5, 00000000001.1155.5, 00000000003.559.5, 00000000006.943.5, 00000000004.580.5] 2022-09-12T20:14:05.030 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.648.5, 00000000002.837.5, 00000000002.643.5, 00000000004.355.5, 00000000004.935.5, 00000000001.434.5, 00000000001.1155.5, 00000000003.559.5, 00000000006.943.5, 00000000004.580.5] 2022-09-12T20:14:05.031 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:14:05.029+0000 7f1c58a50600 0 bucket checkpoint complete 2022-09-12T20:14:05.049 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-75 2022-09-12T20:14:05.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:14:05.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:14:05.090 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:14:05.148 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:14:05.148 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:14:05.188 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:14:05.187+0000 7f6d82139600 1 bucket sync caught up with source: 2022-09-12T20:14:05.189 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.648.5, 00000000002.837.5, 00000000002.643.5, 00000000004.355.5, 00000000004.935.5, 00000000001.434.5, 00000000001.1155.5, 00000000003.559.5, 00000000006.943.5, 00000000004.580.5] 2022-09-12T20:14:05.189 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.648.5, 00000000002.837.5, 00000000002.643.5, 00000000004.355.5, 00000000004.935.5, 00000000001.434.5, 00000000001.1155.5, 00000000003.559.5, 00000000006.943.5, 00000000004.580.5] 2022-09-12T20:14:05.190 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:14:05.187+0000 7f6d82139600 0 bucket checkpoint complete 2022-09-12T20:14:05.205 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-75 2022-09-12T20:14:05.206 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-75 2022-09-12T20:14:05.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:14:05.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:14:05.208 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:14:05.265 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:14:05.266 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:14:05.304 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:14:05.304+0000 7fa44a478600 1 bucket sync caught up with source: 2022-09-12T20:14:05.305 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.304.5, 00000000002.337.5, 00000000002.287.5, 00000000004.352.5, 00000000004.551.5, 00000000001.338.5, 00000000001.466.5, 00000000003.328.5, 00000000006.557.5, 00000000004.494.5] 2022-09-12T20:14:05.306 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.304.5, 00000000002.337.5, 00000000002.287.5, 00000000004.352.5, 00000000004.551.5, 00000000001.338.5, 00000000001.466.5, 00000000003.328.5, 00000000006.557.5, 00000000004.494.5] 2022-09-12T20:14:05.306 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:14:05.304+0000 7fa44a478600 0 bucket checkpoint complete 2022-09-12T20:14:05.318 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-75 2022-09-12T20:14:05.319 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:14:05.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:14:05.320 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T20:14:05.378 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:14:05.379 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:14:05.426 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:14:05.424+0000 7f6577b38600 1 bucket sync caught up with source: 2022-09-12T20:14:05.426 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000003.304.5, 00000000002.337.5, 00000000002.287.5, 00000000004.352.5, 00000000004.551.5, 00000000001.338.5, 00000000001.466.5, 00000000003.328.5, 00000000006.557.5, 00000000004.494.5] 2022-09-12T20:14:05.426 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000003.304.5, 00000000002.337.5, 00000000002.287.5, 00000000004.352.5, 00000000004.551.5, 00000000001.338.5, 00000000001.466.5, 00000000003.328.5, 00000000006.557.5, 00000000004.494.5] 2022-09-12T20:14:05.427 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:14:05.424+0000 7f6577b38600 0 bucket checkpoint complete 2022-09-12T20:14:05.443 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-75 2022-09-12T20:14:05.444 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-75 zones={test-zone2, test-zone1} 2022-09-12T20:14:05.445 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:05.445 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:05.446 DEBUG:boto:Method: HEAD 2022-09-12T20:14:05.446 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:14:05.447 DEBUG:boto:Data: 2022-09-12T20:14:05.447 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.448 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:05.449 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.449 DEBUG:boto:Params: {} 2022-09-12T20:14:05.451 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T20:14:05.451 DEBUG:boto:Token: None 2022-09-12T20:14:05.452 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/ 2022-09-12T20:14:05.453 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PJJx3bI0RiMa1Y81wS5S7L9sZwI= 2022-09-12T20:14:05.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PJJx3bI0RiMa1Y81wS5S7L9sZwI=', 'Content-Length': '0'} 2022-09-12T20:14:05.458 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f15175cc70d77c47-00631f930d-189d-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.458 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:05.459 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:05.460 DEBUG:boto:Method: HEAD 2022-09-12T20:14:05.460 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:14:05.461 DEBUG:boto:Data: 2022-09-12T20:14:05.461 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.462 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:05.463 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.463 DEBUG:boto:Params: {} 2022-09-12T20:14:05.464 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T20:14:05.465 DEBUG:boto:Token: None 2022-09-12T20:14:05.465 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/ 2022-09-12T20:14:05.466 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PJJx3bI0RiMa1Y81wS5S7L9sZwI= 2022-09-12T20:14:05.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PJJx3bI0RiMa1Y81wS5S7L9sZwI=', 'Content-Length': '0'} 2022-09-12T20:14:05.471 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000345ea45611743bdf-00631f930d-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.472 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T20:14:05.472 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:05.473 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:05.474 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:05.474 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:05.475 DEBUG:boto:Method: GET 2022-09-12T20:14:05.475 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:05.476 DEBUG:boto:Data: 2022-09-12T20:14:05.477 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.477 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:05.478 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.478 DEBUG:boto:Params: {} 2022-09-12T20:14:05.479 DEBUG:boto:Token: None 2022-09-12T20:14:05.480 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/?versions 2022-09-12T20:14:05.481 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:g55MLaANmPhx3ZNh4MCgsojjDpM= 2022-09-12T20:14:05.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:g55MLaANmPhx3ZNh4MCgsojjDpM=', 'Content-Length': '0'} 2022-09-12T20:14:05.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000493b940427f826de-00631f930d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.485 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:05.489 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:14:05.489 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-12T20:14:05.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-12T20:14:05.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2022-09-12T20:14:05.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-12T20:14:05.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2022-09-12T20:14:05.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:14:05.493 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:14:05.493 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:14:05.494 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-12T20:14:05.494 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-12T20:14:05.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2022-09-12T20:14:05.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:14:05.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:14:05.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:14:05.497 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:14:05.497 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-12T20:14:05.498 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:14:05.498 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:14:05.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:14:05.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-12T20:14:05.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2022-09-12T20:14:05.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:14:05.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-12T20:14:05.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:14:05.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:14:05.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-12T20:14:05.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-12T20:14:05.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2022-09-12T20:14:05.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:14:05.504 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T20:14:05.505 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:05.506 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:05.506 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:05.507 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:05.507 DEBUG:boto:Method: GET 2022-09-12T20:14:05.508 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:05.508 DEBUG:boto:Data: 2022-09-12T20:14:05.509 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.509 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:05.510 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.511 DEBUG:boto:Params: {} 2022-09-12T20:14:05.511 DEBUG:boto:Token: None 2022-09-12T20:14:05.512 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/?versions 2022-09-12T20:14:05.513 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:g55MLaANmPhx3ZNh4MCgsojjDpM= 2022-09-12T20:14:05.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:g55MLaANmPhx3ZNh4MCgsojjDpM=', 'Content-Length': '0'} 2022-09-12T20:14:05.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f29e9d444c574e4d-00631f930d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.518 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:05.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:14:05.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-12T20:14:05.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-12T20:14:05.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2022-09-12T20:14:05.523 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-12T20:14:05.523 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2022-09-12T20:14:05.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:14:05.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:14:05.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:14:05.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-12T20:14:05.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-12T20:14:05.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2022-09-12T20:14:05.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:14:05.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:14:05.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:14:05.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:14:05.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-12T20:14:05.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:14:05.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:14:05.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:14:05.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-12T20:14:05.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2022-09-12T20:14:05.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:14:05.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-12T20:14:05.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:14:05.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:14:05.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-12T20:14:05.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-12T20:14:05.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2022-09-12T20:14:05.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:14:05.536 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:05.537 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:05.537 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:05.538 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:05.539 DEBUG:boto:Method: GET 2022-09-12T20:14:05.539 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:05.540 DEBUG:boto:Data: 2022-09-12T20:14:05.540 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.541 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:05.541 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.542 DEBUG:boto:Params: {} 2022-09-12T20:14:05.543 DEBUG:boto:Token: None 2022-09-12T20:14:05.543 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/?versions 2022-09-12T20:14:05.544 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:g55MLaANmPhx3ZNh4MCgsojjDpM= 2022-09-12T20:14:05.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:g55MLaANmPhx3ZNh4MCgsojjDpM=', 'Content-Length': '0'} 2022-09-12T20:14:05.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000064efe47bef45f49-00631f930d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.548 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:05.550 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:05.551 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:05.551 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:05.552 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:05.553 DEBUG:boto:Method: GET 2022-09-12T20:14:05.553 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:05.554 DEBUG:boto:Data: 2022-09-12T20:14:05.554 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.555 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:05.555 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.556 DEBUG:boto:Params: {} 2022-09-12T20:14:05.557 DEBUG:boto:Token: None 2022-09-12T20:14:05.557 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/?versions 2022-09-12T20:14:05.558 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:g55MLaANmPhx3ZNh4MCgsojjDpM= 2022-09-12T20:14:05.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:g55MLaANmPhx3ZNh4MCgsojjDpM=', 'Content-Length': '0'} 2022-09-12T20:14:05.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093fae14ba7d125a7-00631f930d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.564 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:05.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:14:05.567 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:05.567 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:05.568 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:05.569 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:05.569 DEBUG:boto:Method: GET 2022-09-12T20:14:05.570 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:05.570 DEBUG:boto:Data: 2022-09-12T20:14:05.571 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.571 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:05.572 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.573 DEBUG:boto:Params: {} 2022-09-12T20:14:05.574 DEBUG:boto:Token: None 2022-09-12T20:14:05.575 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:05.576 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8Ivj936aZIvfF/8xQBo1bdujGd0= 2022-09-12T20:14:05.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8Ivj936aZIvfF/8xQBo1bdujGd0=', 'Content-Length': '0'} 2022-09-12T20:14:05.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000668a08f4467d23f3-00631f930d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.580 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:05.580 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:05.581 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:05.581 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:05.582 DEBUG:boto:Method: GET 2022-09-12T20:14:05.582 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:05.583 DEBUG:boto:Data: 2022-09-12T20:14:05.584 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.584 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:05.585 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.585 DEBUG:boto:Params: {} 2022-09-12T20:14:05.586 DEBUG:boto:Token: None 2022-09-12T20:14:05.587 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:05.587 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8Ivj936aZIvfF/8xQBo1bdujGd0= 2022-09-12T20:14:05.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8Ivj936aZIvfF/8xQBo1bdujGd0=', 'Content-Length': '0'} 2022-09-12T20:14:05.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b74f2f4802184c07-00631f930d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.591 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:05.591 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:05.592 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:05.592 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:05.593 DEBUG:boto:Method: HEAD 2022-09-12T20:14:05.593 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:05.594 DEBUG:boto:Data: 2022-09-12T20:14:05.595 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.595 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:05.596 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.596 DEBUG:boto:Params: {} 2022-09-12T20:14:05.597 DEBUG:boto:Token: None 2022-09-12T20:14:05.597 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:05.598 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:soxBlxknjBgtaLAuAAiRtxWsz40= 2022-09-12T20:14:05.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:soxBlxknjBgtaLAuAAiRtxWsz40=', 'Content-Length': '0'} 2022-09-12T20:14:05.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8e4e350727ea133-00631f930d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.602 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:05.603 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:05.604 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:05.605 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:05.606 DEBUG:boto:Method: HEAD 2022-09-12T20:14:05.607 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:05.608 DEBUG:boto:Data: 2022-09-12T20:14:05.609 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.610 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:05.611 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.612 DEBUG:boto:Params: {} 2022-09-12T20:14:05.613 DEBUG:boto:Token: None 2022-09-12T20:14:05.614 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:05.615 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:soxBlxknjBgtaLAuAAiRtxWsz40= 2022-09-12T20:14:05.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:soxBlxknjBgtaLAuAAiRtxWsz40=', 'Content-Length': '0'} 2022-09-12T20:14:05.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000122fa60ba3c0435-00631f930d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:14:05.621 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:05.622 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:05.623 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:05.624 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:05.625 DEBUG:boto:Method: GET 2022-09-12T20:14:05.627 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:05.627 DEBUG:boto:Data: 2022-09-12T20:14:05.628 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.629 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:05.630 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.631 DEBUG:boto:Params: {} 2022-09-12T20:14:05.633 DEBUG:boto:Token: None 2022-09-12T20:14:05.634 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:05.635 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8Ivj936aZIvfF/8xQBo1bdujGd0= 2022-09-12T20:14:05.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8Ivj936aZIvfF/8xQBo1bdujGd0=', 'Content-Length': '0'} 2022-09-12T20:14:05.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d204091e45d8a63a-00631f930d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.639 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:05.640 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:05.640 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:05.641 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:05.642 DEBUG:boto:Method: GET 2022-09-12T20:14:05.642 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:05.643 DEBUG:boto:Data: 2022-09-12T20:14:05.644 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.644 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:05.645 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.645 DEBUG:boto:Params: {} 2022-09-12T20:14:05.646 DEBUG:boto:Token: None 2022-09-12T20:14:05.647 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:05.647 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8Ivj936aZIvfF/8xQBo1bdujGd0= 2022-09-12T20:14:05.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8Ivj936aZIvfF/8xQBo1bdujGd0=', 'Content-Length': '0'} 2022-09-12T20:14:05.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b4172d3beaebf6c-00631f930d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.694 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:05.695 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:05.696 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:14:05.696 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:14:05.697 DEBUG:boto:Method: GET 2022-09-12T20:14:05.697 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:14:05.698 DEBUG:boto:Data: 2022-09-12T20:14:05.699 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.699 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:05.700 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.700 DEBUG:boto:Params: {} 2022-09-12T20:14:05.701 DEBUG:boto:Token: None 2022-09-12T20:14:05.702 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/ 2022-09-12T20:14:05.702 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yJcr7xsctBwoHUM/Ti0R5dPmTlI= 2022-09-12T20:14:05.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yJcr7xsctBwoHUM/Ti0R5dPmTlI=', 'Content-Length': '0'} 2022-09-12T20:14:05.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ec8b269b95148df-00631f930d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.706 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:05.707 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:05.708 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:05.709 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:14:05.709 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:14:05.710 DEBUG:boto:Method: GET 2022-09-12T20:14:05.710 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:14:05.711 DEBUG:boto:Data: 2022-09-12T20:14:05.711 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.712 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:05.713 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.713 DEBUG:boto:Params: {} 2022-09-12T20:14:05.714 DEBUG:boto:Token: None 2022-09-12T20:14:05.715 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/ 2022-09-12T20:14:05.715 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yJcr7xsctBwoHUM/Ti0R5dPmTlI= 2022-09-12T20:14:05.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yJcr7xsctBwoHUM/Ti0R5dPmTlI=', 'Content-Length': '0'} 2022-09-12T20:14:05.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d0492c07a6baf07-00631f930d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.721 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:05.722 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:05.722 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:05.723 DEBUG:boto:Method: HEAD 2022-09-12T20:14:05.724 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:14:05.724 DEBUG:boto:Data: 2022-09-12T20:14:05.725 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.725 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:05.726 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.726 DEBUG:boto:Params: {} 2022-09-12T20:14:05.727 DEBUG:boto:Token: None 2022-09-12T20:14:05.728 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/obj11 2022-09-12T20:14:05.728 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HeoU7gklsQovGzCazdXlPXB5PTs= 2022-09-12T20:14:05.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HeoU7gklsQovGzCazdXlPXB5PTs=', 'Content-Length': '0'} 2022-09-12T20:14:05.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef2939503a424893-00631f930d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.732 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:05.732 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:05.733 DEBUG:boto:Method: HEAD 2022-09-12T20:14:05.734 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:14:05.734 DEBUG:boto:Data: 2022-09-12T20:14:05.735 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.735 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:05.736 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.736 DEBUG:boto:Params: {} 2022-09-12T20:14:05.737 DEBUG:boto:Token: None 2022-09-12T20:14:05.738 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/obj11 2022-09-12T20:14:05.738 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HeoU7gklsQovGzCazdXlPXB5PTs= 2022-09-12T20:14:05.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HeoU7gklsQovGzCazdXlPXB5PTs=', 'Content-Length': '0'} 2022-09-12T20:14:05.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034db0f56602c6c04-00631f930d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:14:05.742 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:05.743 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:05.744 DEBUG:boto:Method: GET 2022-09-12T20:14:05.744 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:14:05.745 DEBUG:boto:Data: 2022-09-12T20:14:05.746 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.746 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:05.747 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.748 DEBUG:boto:Params: {} 2022-09-12T20:14:05.749 DEBUG:boto:Token: None 2022-09-12T20:14:05.749 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/obj11 2022-09-12T20:14:05.750 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aaeXjFEn0QlS3n1K2KhaWw3l0ws= 2022-09-12T20:14:05.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aaeXjFEn0QlS3n1K2KhaWw3l0ws=', 'Content-Length': '0'} 2022-09-12T20:14:05.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e640fee9513ea8aa-00631f930d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.754 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:05.755 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:05.755 DEBUG:boto:Method: GET 2022-09-12T20:14:05.756 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:14:05.757 DEBUG:boto:Data: 2022-09-12T20:14:05.757 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.758 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:05.759 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.760 DEBUG:boto:Params: {} 2022-09-12T20:14:05.760 DEBUG:boto:Token: None 2022-09-12T20:14:05.761 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/obj11 2022-09-12T20:14:05.762 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aaeXjFEn0QlS3n1K2KhaWw3l0ws= 2022-09-12T20:14:05.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aaeXjFEn0QlS3n1K2KhaWw3l0ws=', 'Content-Length': '0'} 2022-09-12T20:14:05.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086cac8b8bb56c4a2-00631f930d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.766 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:05.767 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:05.767 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:14:05.768 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:14:05.769 DEBUG:boto:Method: GET 2022-09-12T20:14:05.770 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:14:05.770 DEBUG:boto:Data: 2022-09-12T20:14:05.771 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.772 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:05.773 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.774 DEBUG:boto:Params: {} 2022-09-12T20:14:05.775 DEBUG:boto:Token: None 2022-09-12T20:14:05.775 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/ 2022-09-12T20:14:05.776 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yJcr7xsctBwoHUM/Ti0R5dPmTlI= 2022-09-12T20:14:05.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yJcr7xsctBwoHUM/Ti0R5dPmTlI=', 'Content-Length': '0'} 2022-09-12T20:14:05.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf35c92fcb828932-00631f930d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.781 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:05.782 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:05.783 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:05.784 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:14:05.785 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:14:05.785 DEBUG:boto:Method: GET 2022-09-12T20:14:05.786 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:14:05.787 DEBUG:boto:Data: 2022-09-12T20:14:05.788 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.789 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:05.790 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.791 DEBUG:boto:Params: {} 2022-09-12T20:14:05.791 DEBUG:boto:Token: None 2022-09-12T20:14:05.792 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/ 2022-09-12T20:14:05.793 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yJcr7xsctBwoHUM/Ti0R5dPmTlI= 2022-09-12T20:14:05.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yJcr7xsctBwoHUM/Ti0R5dPmTlI=', 'Content-Length': '0'} 2022-09-12T20:14:05.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b71f350be732637-00631f930d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.799 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:05.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:14:05.801 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:05.802 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:05.802 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:05.803 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:05.804 DEBUG:boto:Method: GET 2022-09-12T20:14:05.805 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:05.805 DEBUG:boto:Data: 2022-09-12T20:14:05.806 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.807 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:05.808 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.808 DEBUG:boto:Params: {} 2022-09-12T20:14:05.809 DEBUG:boto:Token: None 2022-09-12T20:14:05.810 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:05.811 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XVz98qWRrC6elzXXYGjmscuP8eA= 2022-09-12T20:14:05.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XVz98qWRrC6elzXXYGjmscuP8eA=', 'Content-Length': '0'} 2022-09-12T20:14:05.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086632f929a060d3c-00631f930d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.815 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:05.816 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:05.817 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:05.818 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:05.819 DEBUG:boto:Method: GET 2022-09-12T20:14:05.820 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:05.821 DEBUG:boto:Data: 2022-09-12T20:14:05.822 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.823 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:05.824 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.825 DEBUG:boto:Params: {} 2022-09-12T20:14:05.826 DEBUG:boto:Token: None 2022-09-12T20:14:05.827 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:05.828 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XVz98qWRrC6elzXXYGjmscuP8eA= 2022-09-12T20:14:05.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XVz98qWRrC6elzXXYGjmscuP8eA=', 'Content-Length': '0'} 2022-09-12T20:14:05.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008ecf5d2d64a1126-00631f930d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.874 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:05.875 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:05.876 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:05.876 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:05.877 DEBUG:boto:Method: HEAD 2022-09-12T20:14:05.877 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:05.878 DEBUG:boto:Data: 2022-09-12T20:14:05.879 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.879 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:05.880 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.881 DEBUG:boto:Params: {} 2022-09-12T20:14:05.881 DEBUG:boto:Token: None 2022-09-12T20:14:05.882 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:05.883 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XuNjbbZJRles4YWbMgIBSQkQEvU= 2022-09-12T20:14:05.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XuNjbbZJRles4YWbMgIBSQkQEvU=', 'Content-Length': '0'} 2022-09-12T20:14:05.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e32ed00b20017f58-00631f930d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.887 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:05.888 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:05.888 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:05.889 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:05.890 DEBUG:boto:Method: HEAD 2022-09-12T20:14:05.891 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:05.891 DEBUG:boto:Data: 2022-09-12T20:14:05.892 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.893 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:05.894 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.894 DEBUG:boto:Params: {} 2022-09-12T20:14:05.895 DEBUG:boto:Token: None 2022-09-12T20:14:05.896 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:05.897 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XuNjbbZJRles4YWbMgIBSQkQEvU= 2022-09-12T20:14:05.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XuNjbbZJRles4YWbMgIBSQkQEvU=', 'Content-Length': '0'} 2022-09-12T20:14:05.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df3a8571f9375e6a-00631f930d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:14:05.901 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:05.902 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:05.903 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:05.903 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:05.904 DEBUG:boto:Method: GET 2022-09-12T20:14:05.905 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:05.906 DEBUG:boto:Data: 2022-09-12T20:14:05.907 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.907 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:05.908 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.909 DEBUG:boto:Params: {} 2022-09-12T20:14:05.909 DEBUG:boto:Token: None 2022-09-12T20:14:05.910 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:05.911 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XVz98qWRrC6elzXXYGjmscuP8eA= 2022-09-12T20:14:05.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XVz98qWRrC6elzXXYGjmscuP8eA=', 'Content-Length': '0'} 2022-09-12T20:14:05.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000de25f2eda24e07a-00631f930d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.954 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:05.955 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:05.955 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:05.956 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:05.957 DEBUG:boto:Method: GET 2022-09-12T20:14:05.957 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:05.958 DEBUG:boto:Data: 2022-09-12T20:14:05.958 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.959 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:05.960 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.960 DEBUG:boto:Params: {} 2022-09-12T20:14:05.961 DEBUG:boto:Token: None 2022-09-12T20:14:05.962 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:05.962 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XVz98qWRrC6elzXXYGjmscuP8eA= 2022-09-12T20:14:05.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XVz98qWRrC6elzXXYGjmscuP8eA=', 'Content-Length': '0'} 2022-09-12T20:14:05.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1e1f45eab60562d-00631f930d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.966 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:05.967 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:05.968 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:14:05.968 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:14:05.969 DEBUG:boto:Method: GET 2022-09-12T20:14:05.969 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:14:05.970 DEBUG:boto:Data: 2022-09-12T20:14:05.971 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.971 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:05.972 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.972 DEBUG:boto:Params: {} 2022-09-12T20:14:05.973 DEBUG:boto:Token: None 2022-09-12T20:14:05.974 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/ 2022-09-12T20:14:05.974 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yJcr7xsctBwoHUM/Ti0R5dPmTlI= 2022-09-12T20:14:05.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yJcr7xsctBwoHUM/Ti0R5dPmTlI=', 'Content-Length': '0'} 2022-09-12T20:14:05.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000273a597432a4e310-00631f930d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.978 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:05.979 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:05.979 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:05.980 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:14:05.980 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:14:05.981 DEBUG:boto:Method: GET 2022-09-12T20:14:05.982 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:14:05.982 DEBUG:boto:Data: 2022-09-12T20:14:05.983 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.983 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:05.984 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.984 DEBUG:boto:Params: {} 2022-09-12T20:14:05.985 DEBUG:boto:Token: None 2022-09-12T20:14:05.985 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/ 2022-09-12T20:14:05.986 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yJcr7xsctBwoHUM/Ti0R5dPmTlI= 2022-09-12T20:14:05.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yJcr7xsctBwoHUM/Ti0R5dPmTlI=', 'Content-Length': '0'} 2022-09-12T20:14:05.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad141f9dd401bc65-00631f930d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:05.991 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:05.992 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:05.992 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:05.993 DEBUG:boto:Method: HEAD 2022-09-12T20:14:05.993 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:14:05.994 DEBUG:boto:Data: 2022-09-12T20:14:05.994 DEBUG:boto:Headers: {} 2022-09-12T20:14:05.995 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:05.995 DEBUG:boto:Port: 8000 2022-09-12T20:14:05.996 DEBUG:boto:Params: {} 2022-09-12T20:14:05.996 DEBUG:boto:Token: None 2022-09-12T20:14:05.997 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:05 GMT /jdpiak-75/obj115 2022-09-12T20:14:05.997 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/sHDfUx6eXg2M3JT3ntpJJUoix4= 2022-09-12T20:14:05.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/sHDfUx6eXg2M3JT3ntpJJUoix4=', 'Content-Length': '0'} 2022-09-12T20:14:06.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8e1742c0fa187ab-00631f930d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.000 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:06.001 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:06.001 DEBUG:boto:Method: HEAD 2022-09-12T20:14:06.002 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:14:06.002 DEBUG:boto:Data: 2022-09-12T20:14:06.003 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.003 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.004 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.004 DEBUG:boto:Params: {} 2022-09-12T20:14:06.005 DEBUG:boto:Token: None 2022-09-12T20:14:06.005 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj115 2022-09-12T20:14:06.006 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Biw0FCcd/Ius1YqMmgEVDX42Lr8= 2022-09-12T20:14:06.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Biw0FCcd/Ius1YqMmgEVDX42Lr8=', 'Content-Length': '0'} 2022-09-12T20:14:06.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb60782707628441-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:14:06.010 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:06.011 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:06.011 DEBUG:boto:Method: GET 2022-09-12T20:14:06.012 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:14:06.012 DEBUG:boto:Data: 2022-09-12T20:14:06.013 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.013 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.014 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.014 DEBUG:boto:Params: {} 2022-09-12T20:14:06.015 DEBUG:boto:Token: None 2022-09-12T20:14:06.015 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj115 2022-09-12T20:14:06.016 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OjDrhk61RncOerHuEXJj8NMZK94= 2022-09-12T20:14:06.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OjDrhk61RncOerHuEXJj8NMZK94=', 'Content-Length': '0'} 2022-09-12T20:14:06.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000163d9f8ef676a4cc-00631f930e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.019 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:06.020 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:06.020 DEBUG:boto:Method: GET 2022-09-12T20:14:06.021 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:14:06.021 DEBUG:boto:Data: 2022-09-12T20:14:06.022 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.022 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.023 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.023 DEBUG:boto:Params: {} 2022-09-12T20:14:06.024 DEBUG:boto:Token: None 2022-09-12T20:14:06.025 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj115 2022-09-12T20:14:06.025 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OjDrhk61RncOerHuEXJj8NMZK94= 2022-09-12T20:14:06.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OjDrhk61RncOerHuEXJj8NMZK94=', 'Content-Length': '0'} 2022-09-12T20:14:06.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049b2973eb10605c1-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.029 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:06.029 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:06.030 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:14:06.030 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:14:06.031 DEBUG:boto:Method: GET 2022-09-12T20:14:06.031 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:14:06.032 DEBUG:boto:Data: 2022-09-12T20:14:06.032 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.033 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.033 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.034 DEBUG:boto:Params: {} 2022-09-12T20:14:06.034 DEBUG:boto:Token: None 2022-09-12T20:14:06.035 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/ 2022-09-12T20:14:06.036 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw= 2022-09-12T20:14:06.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw=', 'Content-Length': '0'} 2022-09-12T20:14:06.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a7987597765a1d4-00631f930e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.039 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:06.040 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:06.041 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:06.041 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:14:06.042 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:14:06.042 DEBUG:boto:Method: GET 2022-09-12T20:14:06.043 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:14:06.043 DEBUG:boto:Data: 2022-09-12T20:14:06.044 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.044 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.045 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.045 DEBUG:boto:Params: {} 2022-09-12T20:14:06.046 DEBUG:boto:Token: None 2022-09-12T20:14:06.046 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/ 2022-09-12T20:14:06.047 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw= 2022-09-12T20:14:06.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw=', 'Content-Length': '0'} 2022-09-12T20:14:06.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7ee2bd78b4bc33d-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.053 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:06.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:14:06.054 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:06.054 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:06.055 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:06.055 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:06.056 DEBUG:boto:Method: GET 2022-09-12T20:14:06.056 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:06.057 DEBUG:boto:Data: 2022-09-12T20:14:06.058 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.058 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.059 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.059 DEBUG:boto:Params: {} 2022-09-12T20:14:06.060 DEBUG:boto:Token: None 2022-09-12T20:14:06.060 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:06.061 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VBBkWtDuBQIa/1iR9xwjTu9XRAU= 2022-09-12T20:14:06.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VBBkWtDuBQIa/1iR9xwjTu9XRAU=', 'Content-Length': '0'} 2022-09-12T20:14:06.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3a93343e42817aa-00631f930e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.106 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:06.107 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:06.107 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:06.108 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:06.108 DEBUG:boto:Method: GET 2022-09-12T20:14:06.109 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:06.109 DEBUG:boto:Data: 2022-09-12T20:14:06.110 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.111 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.111 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.112 DEBUG:boto:Params: {} 2022-09-12T20:14:06.112 DEBUG:boto:Token: None 2022-09-12T20:14:06.113 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:06.114 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VBBkWtDuBQIa/1iR9xwjTu9XRAU= 2022-09-12T20:14:06.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VBBkWtDuBQIa/1iR9xwjTu9XRAU=', 'Content-Length': '0'} 2022-09-12T20:14:06.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000146b8e6270f3e691-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.117 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:06.118 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:06.118 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:06.119 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:06.119 DEBUG:boto:Method: HEAD 2022-09-12T20:14:06.120 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:06.120 DEBUG:boto:Data: 2022-09-12T20:14:06.121 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.121 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.122 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.122 DEBUG:boto:Params: {} 2022-09-12T20:14:06.123 DEBUG:boto:Token: None 2022-09-12T20:14:06.124 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:06.124 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sB6lHF8YIAk+8hr5SseG8MXk7vU= 2022-09-12T20:14:06.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sB6lHF8YIAk+8hr5SseG8MXk7vU=', 'Content-Length': '0'} 2022-09-12T20:14:06.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d65cac616303250d-00631f930e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.127 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:06.128 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:06.129 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:06.129 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:06.130 DEBUG:boto:Method: HEAD 2022-09-12T20:14:06.130 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:06.131 DEBUG:boto:Data: 2022-09-12T20:14:06.131 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.132 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.132 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.133 DEBUG:boto:Params: {} 2022-09-12T20:14:06.133 DEBUG:boto:Token: None 2022-09-12T20:14:06.134 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:06.135 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sB6lHF8YIAk+8hr5SseG8MXk7vU= 2022-09-12T20:14:06.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sB6lHF8YIAk+8hr5SseG8MXk7vU=', 'Content-Length': '0'} 2022-09-12T20:14:06.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035c0d0e5b4a2ceeb-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:14:06.138 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:06.139 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:06.139 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:06.140 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:06.140 DEBUG:boto:Method: GET 2022-09-12T20:14:06.141 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:06.142 DEBUG:boto:Data: 2022-09-12T20:14:06.142 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.143 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.143 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.144 DEBUG:boto:Params: {} 2022-09-12T20:14:06.144 DEBUG:boto:Token: None 2022-09-12T20:14:06.145 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:06.146 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VBBkWtDuBQIa/1iR9xwjTu9XRAU= 2022-09-12T20:14:06.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VBBkWtDuBQIa/1iR9xwjTu9XRAU=', 'Content-Length': '0'} 2022-09-12T20:14:06.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000073d5a9f7d1c19ae-00631f930e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.149 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:06.150 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:06.150 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:06.151 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:06.151 DEBUG:boto:Method: GET 2022-09-12T20:14:06.152 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:06.152 DEBUG:boto:Data: 2022-09-12T20:14:06.153 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.154 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.154 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.155 DEBUG:boto:Params: {} 2022-09-12T20:14:06.155 DEBUG:boto:Token: None 2022-09-12T20:14:06.156 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:06.156 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VBBkWtDuBQIa/1iR9xwjTu9XRAU= 2022-09-12T20:14:06.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VBBkWtDuBQIa/1iR9xwjTu9XRAU=', 'Content-Length': '0'} 2022-09-12T20:14:06.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016f7c9ad6fd46ddc-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.160 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:06.160 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:06.161 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:14:06.162 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:14:06.162 DEBUG:boto:Method: GET 2022-09-12T20:14:06.163 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:14:06.163 DEBUG:boto:Data: 2022-09-12T20:14:06.164 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.164 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.165 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.165 DEBUG:boto:Params: {} 2022-09-12T20:14:06.166 DEBUG:boto:Token: None 2022-09-12T20:14:06.167 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/ 2022-09-12T20:14:06.167 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw= 2022-09-12T20:14:06.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw=', 'Content-Length': '0'} 2022-09-12T20:14:06.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ee10f6a97f29741-00631f930e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.171 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:06.172 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:06.172 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:06.173 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:14:06.173 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:14:06.174 DEBUG:boto:Method: GET 2022-09-12T20:14:06.174 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:14:06.175 DEBUG:boto:Data: 2022-09-12T20:14:06.176 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.176 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.177 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.177 DEBUG:boto:Params: {} 2022-09-12T20:14:06.178 DEBUG:boto:Token: None 2022-09-12T20:14:06.178 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/ 2022-09-12T20:14:06.179 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw= 2022-09-12T20:14:06.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw=', 'Content-Length': '0'} 2022-09-12T20:14:06.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067157198c555918f-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.185 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:06.186 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:06.186 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:06.187 DEBUG:boto:Method: HEAD 2022-09-12T20:14:06.187 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:14:06.188 DEBUG:boto:Data: 2022-09-12T20:14:06.188 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.189 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.189 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.190 DEBUG:boto:Params: {} 2022-09-12T20:14:06.190 DEBUG:boto:Token: None 2022-09-12T20:14:06.191 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj116 2022-09-12T20:14:06.192 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f4T8Ywp687NPIIG5AA7W9fw8DfY= 2022-09-12T20:14:06.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f4T8Ywp687NPIIG5AA7W9fw8DfY=', 'Content-Length': '0'} 2022-09-12T20:14:06.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe8b600be673beb3-00631f930e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.195 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:06.196 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:06.196 DEBUG:boto:Method: HEAD 2022-09-12T20:14:06.197 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:14:06.197 DEBUG:boto:Data: 2022-09-12T20:14:06.198 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.198 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.199 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.199 DEBUG:boto:Params: {} 2022-09-12T20:14:06.200 DEBUG:boto:Token: None 2022-09-12T20:14:06.201 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj116 2022-09-12T20:14:06.201 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f4T8Ywp687NPIIG5AA7W9fw8DfY= 2022-09-12T20:14:06.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f4T8Ywp687NPIIG5AA7W9fw8DfY=', 'Content-Length': '0'} 2022-09-12T20:14:06.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3e2a6f9faf9fe90-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:14:06.205 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:06.206 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:06.206 DEBUG:boto:Method: GET 2022-09-12T20:14:06.207 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:14:06.207 DEBUG:boto:Data: 2022-09-12T20:14:06.208 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.209 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.209 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.210 DEBUG:boto:Params: {} 2022-09-12T20:14:06.210 DEBUG:boto:Token: None 2022-09-12T20:14:06.211 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj116 2022-09-12T20:14:06.212 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ApCkRznmxz3D+wbOW12hzhxvdUY= 2022-09-12T20:14:06.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ApCkRznmxz3D+wbOW12hzhxvdUY=', 'Content-Length': '0'} 2022-09-12T20:14:06.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df75ccd35fe0e556-00631f930e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.258 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:06.259 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:06.259 DEBUG:boto:Method: GET 2022-09-12T20:14:06.260 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:14:06.260 DEBUG:boto:Data: 2022-09-12T20:14:06.261 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.262 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.262 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.263 DEBUG:boto:Params: {} 2022-09-12T20:14:06.263 DEBUG:boto:Token: None 2022-09-12T20:14:06.264 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj116 2022-09-12T20:14:06.265 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ApCkRznmxz3D+wbOW12hzhxvdUY= 2022-09-12T20:14:06.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ApCkRznmxz3D+wbOW12hzhxvdUY=', 'Content-Length': '0'} 2022-09-12T20:14:06.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fbc2bdf36d8d1c7-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.268 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:06.269 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:06.269 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:14:06.270 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:14:06.271 DEBUG:boto:Method: GET 2022-09-12T20:14:06.271 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:14:06.272 DEBUG:boto:Data: 2022-09-12T20:14:06.272 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.273 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.273 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.274 DEBUG:boto:Params: {} 2022-09-12T20:14:06.275 DEBUG:boto:Token: None 2022-09-12T20:14:06.275 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/ 2022-09-12T20:14:06.276 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw= 2022-09-12T20:14:06.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw=', 'Content-Length': '0'} 2022-09-12T20:14:06.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdadd982ea924c54-00631f930e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.280 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:06.281 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:06.282 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:06.282 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:14:06.283 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:14:06.283 DEBUG:boto:Method: GET 2022-09-12T20:14:06.284 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:14:06.284 DEBUG:boto:Data: 2022-09-12T20:14:06.285 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.286 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.286 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.287 DEBUG:boto:Params: {} 2022-09-12T20:14:06.287 DEBUG:boto:Token: None 2022-09-12T20:14:06.288 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/ 2022-09-12T20:14:06.289 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw= 2022-09-12T20:14:06.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw=', 'Content-Length': '0'} 2022-09-12T20:14:06.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c05d64f4e5aba1fa-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.294 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:06.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-12T20:14:06.296 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:06.296 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:06.297 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:06.297 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:06.298 DEBUG:boto:Method: GET 2022-09-12T20:14:06.299 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:06.299 DEBUG:boto:Data: 2022-09-12T20:14:06.300 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.300 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.301 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.301 DEBUG:boto:Params: {} 2022-09-12T20:14:06.302 DEBUG:boto:Token: None 2022-09-12T20:14:06.303 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:06.303 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dk7M/Keu6pIHqEtG3VoPmDPUeXA= 2022-09-12T20:14:06.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dk7M/Keu6pIHqEtG3VoPmDPUeXA=', 'Content-Length': '0'} 2022-09-12T20:14:06.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5dc226a3742409c-00631f930e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.307 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:06.307 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:06.308 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:06.308 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:06.309 DEBUG:boto:Method: GET 2022-09-12T20:14:06.310 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:06.310 DEBUG:boto:Data: 2022-09-12T20:14:06.311 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.311 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.312 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.312 DEBUG:boto:Params: {} 2022-09-12T20:14:06.313 DEBUG:boto:Token: None 2022-09-12T20:14:06.314 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:06.314 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dk7M/Keu6pIHqEtG3VoPmDPUeXA= 2022-09-12T20:14:06.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dk7M/Keu6pIHqEtG3VoPmDPUeXA=', 'Content-Length': '0'} 2022-09-12T20:14:06.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000035c152797719903-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.318 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:06.318 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:06.319 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:06.319 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:06.320 DEBUG:boto:Method: HEAD 2022-09-12T20:14:06.320 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:06.321 DEBUG:boto:Data: 2022-09-12T20:14:06.321 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.322 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.322 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.323 DEBUG:boto:Params: {} 2022-09-12T20:14:06.324 DEBUG:boto:Token: None 2022-09-12T20:14:06.324 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:06.325 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xhWL83GRhvj0PaByi6EDZo9QpIc= 2022-09-12T20:14:06.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xhWL83GRhvj0PaByi6EDZo9QpIc=', 'Content-Length': '0'} 2022-09-12T20:14:06.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de4e2279be3dd9c8-00631f930e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.328 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:06.329 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:06.329 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:06.330 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:06.330 DEBUG:boto:Method: HEAD 2022-09-12T20:14:06.331 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:06.331 DEBUG:boto:Data: 2022-09-12T20:14:06.332 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.332 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.333 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.333 DEBUG:boto:Params: {} 2022-09-12T20:14:06.334 DEBUG:boto:Token: None 2022-09-12T20:14:06.335 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:06.335 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xhWL83GRhvj0PaByi6EDZo9QpIc= 2022-09-12T20:14:06.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xhWL83GRhvj0PaByi6EDZo9QpIc=', 'Content-Length': '0'} 2022-09-12T20:14:06.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa2c5ae06355e223-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-12T20:14:06.339 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:06.340 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:06.340 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:06.341 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:06.341 DEBUG:boto:Method: GET 2022-09-12T20:14:06.342 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:06.342 DEBUG:boto:Data: 2022-09-12T20:14:06.343 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.344 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.344 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.345 DEBUG:boto:Params: {} 2022-09-12T20:14:06.345 DEBUG:boto:Token: None 2022-09-12T20:14:06.346 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:06.347 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dk7M/Keu6pIHqEtG3VoPmDPUeXA= 2022-09-12T20:14:06.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dk7M/Keu6pIHqEtG3VoPmDPUeXA=', 'Content-Length': '0'} 2022-09-12T20:14:06.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be59df3e033a125c-00631f930e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.390 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:06.391 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:06.391 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:06.392 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:06.392 DEBUG:boto:Method: GET 2022-09-12T20:14:06.393 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:06.393 DEBUG:boto:Data: 2022-09-12T20:14:06.394 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.394 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.395 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.395 DEBUG:boto:Params: {} 2022-09-12T20:14:06.396 DEBUG:boto:Token: None 2022-09-12T20:14:06.397 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:06.397 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dk7M/Keu6pIHqEtG3VoPmDPUeXA= 2022-09-12T20:14:06.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dk7M/Keu6pIHqEtG3VoPmDPUeXA=', 'Content-Length': '0'} 2022-09-12T20:14:06.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efd411ddd8758c4f-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.401 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:06.402 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:06.402 DEBUG:boto:path=/jdpiak-75/?prefix=obj124 2022-09-12T20:14:06.403 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj124 2022-09-12T20:14:06.403 DEBUG:boto:Method: GET 2022-09-12T20:14:06.404 DEBUG:boto:Path: /jdpiak-75/?prefix=obj124 2022-09-12T20:14:06.404 DEBUG:boto:Data: 2022-09-12T20:14:06.405 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.405 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.406 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.407 DEBUG:boto:Params: {} 2022-09-12T20:14:06.407 DEBUG:boto:Token: None 2022-09-12T20:14:06.408 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/ 2022-09-12T20:14:06.408 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw= 2022-09-12T20:14:06.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw=', 'Content-Length': '0'} 2022-09-12T20:14:06.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098033592155999ab-00631f930e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.412 DEBUG:boto:b'jdpiak-75obj1241000falseobj1242022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:06.413 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:06.413 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:06.414 DEBUG:boto:path=/jdpiak-75/?prefix=obj124 2022-09-12T20:14:06.414 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj124 2022-09-12T20:14:06.415 DEBUG:boto:Method: GET 2022-09-12T20:14:06.415 DEBUG:boto:Path: /jdpiak-75/?prefix=obj124 2022-09-12T20:14:06.416 DEBUG:boto:Data: 2022-09-12T20:14:06.417 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.417 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.418 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.418 DEBUG:boto:Params: {} 2022-09-12T20:14:06.419 DEBUG:boto:Token: None 2022-09-12T20:14:06.419 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/ 2022-09-12T20:14:06.420 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw= 2022-09-12T20:14:06.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw=', 'Content-Length': '0'} 2022-09-12T20:14:06.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adb44eb07a65cfe2-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.425 DEBUG:boto:b'jdpiak-75obj1241000falseobj1242022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:06.426 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:06.427 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:06.427 DEBUG:boto:Method: HEAD 2022-09-12T20:14:06.428 DEBUG:boto:Path: /jdpiak-75/obj124 2022-09-12T20:14:06.428 DEBUG:boto:Data: 2022-09-12T20:14:06.428 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.429 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.429 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.430 DEBUG:boto:Params: {} 2022-09-12T20:14:06.431 DEBUG:boto:Token: None 2022-09-12T20:14:06.431 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj124 2022-09-12T20:14:06.432 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3WqK3NsdUwIDdk9RAAoM4v2LUoU= 2022-09-12T20:14:06.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3WqK3NsdUwIDdk9RAAoM4v2LUoU=', 'Content-Length': '0'} 2022-09-12T20:14:06.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000680a1fc441302d9d-00631f930e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.435 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:06.435 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:06.436 DEBUG:boto:Method: HEAD 2022-09-12T20:14:06.436 DEBUG:boto:Path: /jdpiak-75/obj124 2022-09-12T20:14:06.437 DEBUG:boto:Data: 2022-09-12T20:14:06.437 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.438 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.438 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.439 DEBUG:boto:Params: {} 2022-09-12T20:14:06.439 DEBUG:boto:Token: None 2022-09-12T20:14:06.440 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj124 2022-09-12T20:14:06.440 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3WqK3NsdUwIDdk9RAAoM4v2LUoU= 2022-09-12T20:14:06.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3WqK3NsdUwIDdk9RAAoM4v2LUoU=', 'Content-Length': '0'} 2022-09-12T20:14:06.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027d052891c74a2bc-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-12T20:14:06.444 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:06.445 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:06.445 DEBUG:boto:Method: GET 2022-09-12T20:14:06.446 DEBUG:boto:Path: /jdpiak-75/obj124 2022-09-12T20:14:06.446 DEBUG:boto:Data: 2022-09-12T20:14:06.447 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.447 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.448 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.448 DEBUG:boto:Params: {} 2022-09-12T20:14:06.449 DEBUG:boto:Token: None 2022-09-12T20:14:06.450 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj124 2022-09-12T20:14:06.450 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tuhiZggpxf0yLt3ATKqBF5DlzFI= 2022-09-12T20:14:06.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tuhiZggpxf0yLt3ATKqBF5DlzFI=', 'Content-Length': '0'} 2022-09-12T20:14:06.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000928bf3d97a3f2b3b-00631f930e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.494 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:06.495 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:06.495 DEBUG:boto:Method: GET 2022-09-12T20:14:06.496 DEBUG:boto:Path: /jdpiak-75/obj124 2022-09-12T20:14:06.497 DEBUG:boto:Data: 2022-09-12T20:14:06.497 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.498 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.498 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.499 DEBUG:boto:Params: {} 2022-09-12T20:14:06.499 DEBUG:boto:Token: None 2022-09-12T20:14:06.500 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj124 2022-09-12T20:14:06.501 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tuhiZggpxf0yLt3ATKqBF5DlzFI= 2022-09-12T20:14:06.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tuhiZggpxf0yLt3ATKqBF5DlzFI=', 'Content-Length': '0'} 2022-09-12T20:14:06.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3246badaba451b7-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.504 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:06.505 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:06.505 DEBUG:boto:path=/jdpiak-75/?prefix=obj124 2022-09-12T20:14:06.506 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj124 2022-09-12T20:14:06.506 DEBUG:boto:Method: GET 2022-09-12T20:14:06.507 DEBUG:boto:Path: /jdpiak-75/?prefix=obj124 2022-09-12T20:14:06.507 DEBUG:boto:Data: 2022-09-12T20:14:06.508 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.509 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.509 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.510 DEBUG:boto:Params: {} 2022-09-12T20:14:06.510 DEBUG:boto:Token: None 2022-09-12T20:14:06.511 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/ 2022-09-12T20:14:06.512 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw= 2022-09-12T20:14:06.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw=', 'Content-Length': '0'} 2022-09-12T20:14:06.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000845e91e5a800f028-00631f930e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.515 DEBUG:boto:b'jdpiak-75obj1241000falseobj1242022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:06.516 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:06.517 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:06.517 DEBUG:boto:path=/jdpiak-75/?prefix=obj124 2022-09-12T20:14:06.518 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj124 2022-09-12T20:14:06.518 DEBUG:boto:Method: GET 2022-09-12T20:14:06.519 DEBUG:boto:Path: /jdpiak-75/?prefix=obj124 2022-09-12T20:14:06.519 DEBUG:boto:Data: 2022-09-12T20:14:06.520 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.520 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.521 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.522 DEBUG:boto:Params: {} 2022-09-12T20:14:06.522 DEBUG:boto:Token: None 2022-09-12T20:14:06.523 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/ 2022-09-12T20:14:06.523 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw= 2022-09-12T20:14:06.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw=', 'Content-Length': '0'} 2022-09-12T20:14:06.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8bbd08a22044447-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.529 DEBUG:boto:b'jdpiak-75obj1241000falseobj1242022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:06.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:14:06.531 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:06.531 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:06.532 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:06.532 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:06.533 DEBUG:boto:Method: GET 2022-09-12T20:14:06.533 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:06.534 DEBUG:boto:Data: 2022-09-12T20:14:06.534 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.535 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.536 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.536 DEBUG:boto:Params: {} 2022-09-12T20:14:06.537 DEBUG:boto:Token: None 2022-09-12T20:14:06.537 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:06.538 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ot16iwgEViFoGna0vh63XU6EZ24= 2022-09-12T20:14:06.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ot16iwgEViFoGna0vh63XU6EZ24=', 'Content-Length': '0'} 2022-09-12T20:14:06.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7c6e2b625845a13-00631f930e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.541 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:06.542 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:06.543 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:06.543 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:06.544 DEBUG:boto:Method: GET 2022-09-12T20:14:06.544 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:06.545 DEBUG:boto:Data: 2022-09-12T20:14:06.545 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.546 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.546 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.547 DEBUG:boto:Params: {} 2022-09-12T20:14:06.548 DEBUG:boto:Token: None 2022-09-12T20:14:06.548 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:06.549 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ot16iwgEViFoGna0vh63XU6EZ24= 2022-09-12T20:14:06.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ot16iwgEViFoGna0vh63XU6EZ24=', 'Content-Length': '0'} 2022-09-12T20:14:06.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dd148dab6c98dba-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.552 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:06.553 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:06.553 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:06.554 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:06.554 DEBUG:boto:Method: HEAD 2022-09-12T20:14:06.555 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:06.555 DEBUG:boto:Data: 2022-09-12T20:14:06.556 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.557 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.557 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.558 DEBUG:boto:Params: {} 2022-09-12T20:14:06.558 DEBUG:boto:Token: None 2022-09-12T20:14:06.559 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:06.559 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7+CUkd/PQx+VCrWTKol749ErKi8= 2022-09-12T20:14:06.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7+CUkd/PQx+VCrWTKol749ErKi8=', 'Content-Length': '0'} 2022-09-12T20:14:06.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8c0d372b3f41501-00631f930e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.562 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:06.563 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:06.563 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:06.564 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:06.565 DEBUG:boto:Method: HEAD 2022-09-12T20:14:06.565 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:06.566 DEBUG:boto:Data: 2022-09-12T20:14:06.566 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.567 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.567 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.568 DEBUG:boto:Params: {} 2022-09-12T20:14:06.568 DEBUG:boto:Token: None 2022-09-12T20:14:06.569 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:06.569 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7+CUkd/PQx+VCrWTKol749ErKi8= 2022-09-12T20:14:06.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7+CUkd/PQx+VCrWTKol749ErKi8=', 'Content-Length': '0'} 2022-09-12T20:14:06.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006501c74b9b80ee2c-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:14:06.573 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:06.574 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:06.575 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:06.575 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:06.576 DEBUG:boto:Method: GET 2022-09-12T20:14:06.576 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:06.577 DEBUG:boto:Data: 2022-09-12T20:14:06.577 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.578 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.578 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.579 DEBUG:boto:Params: {} 2022-09-12T20:14:06.579 DEBUG:boto:Token: None 2022-09-12T20:14:06.580 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:06.581 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ot16iwgEViFoGna0vh63XU6EZ24= 2022-09-12T20:14:06.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ot16iwgEViFoGna0vh63XU6EZ24=', 'Content-Length': '0'} 2022-09-12T20:14:06.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000428b5089867cef19-00631f930e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.584 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:06.585 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:06.585 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:06.586 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:06.586 DEBUG:boto:Method: GET 2022-09-12T20:14:06.587 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:06.587 DEBUG:boto:Data: 2022-09-12T20:14:06.588 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.588 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.589 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.589 DEBUG:boto:Params: {} 2022-09-12T20:14:06.590 DEBUG:boto:Token: None 2022-09-12T20:14:06.591 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:06.591 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ot16iwgEViFoGna0vh63XU6EZ24= 2022-09-12T20:14:06.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ot16iwgEViFoGna0vh63XU6EZ24=', 'Content-Length': '0'} 2022-09-12T20:14:06.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bafad8d5e97afcd7-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.595 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:06.596 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:06.596 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:14:06.597 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:14:06.597 DEBUG:boto:Method: GET 2022-09-12T20:14:06.598 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:14:06.598 DEBUG:boto:Data: 2022-09-12T20:14:06.599 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.599 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.600 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.600 DEBUG:boto:Params: {} 2022-09-12T20:14:06.601 DEBUG:boto:Token: None 2022-09-12T20:14:06.602 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/ 2022-09-12T20:14:06.602 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw= 2022-09-12T20:14:06.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw=', 'Content-Length': '0'} 2022-09-12T20:14:06.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ead06363831c6841-00631f930e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.606 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:06.607 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:06.607 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:06.608 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:14:06.609 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:14:06.609 DEBUG:boto:Method: GET 2022-09-12T20:14:06.610 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:14:06.610 DEBUG:boto:Data: 2022-09-12T20:14:06.611 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.611 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.612 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.612 DEBUG:boto:Params: {} 2022-09-12T20:14:06.613 DEBUG:boto:Token: None 2022-09-12T20:14:06.614 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/ 2022-09-12T20:14:06.614 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw= 2022-09-12T20:14:06.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw=', 'Content-Length': '0'} 2022-09-12T20:14:06.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ec79cfb6b096a9a-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.662 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:06.663 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:06.664 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:06.664 DEBUG:boto:Method: HEAD 2022-09-12T20:14:06.665 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:14:06.665 DEBUG:boto:Data: 2022-09-12T20:14:06.666 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.666 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.667 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.667 DEBUG:boto:Params: {} 2022-09-12T20:14:06.668 DEBUG:boto:Token: None 2022-09-12T20:14:06.668 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj145 2022-09-12T20:14:06.669 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hoQy4vA3zUmN8QXU34zcCAvVyxA= 2022-09-12T20:14:06.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hoQy4vA3zUmN8QXU34zcCAvVyxA=', 'Content-Length': '0'} 2022-09-12T20:14:06.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000829cf38a12005568-00631f930e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.672 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:06.673 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:06.673 DEBUG:boto:Method: HEAD 2022-09-12T20:14:06.674 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:14:06.674 DEBUG:boto:Data: 2022-09-12T20:14:06.675 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.675 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.676 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.676 DEBUG:boto:Params: {} 2022-09-12T20:14:06.677 DEBUG:boto:Token: None 2022-09-12T20:14:06.677 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj145 2022-09-12T20:14:06.678 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hoQy4vA3zUmN8QXU34zcCAvVyxA= 2022-09-12T20:14:06.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hoQy4vA3zUmN8QXU34zcCAvVyxA=', 'Content-Length': '0'} 2022-09-12T20:14:06.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055b09a4ea40424ae-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:14:06.682 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:06.683 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:06.684 DEBUG:boto:Method: GET 2022-09-12T20:14:06.684 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:14:06.685 DEBUG:boto:Data: 2022-09-12T20:14:06.685 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.686 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.686 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.687 DEBUG:boto:Params: {} 2022-09-12T20:14:06.687 DEBUG:boto:Token: None 2022-09-12T20:14:06.688 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj145 2022-09-12T20:14:06.689 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1WQuWaiNmEDUR3tlaelVSbIwHbI= 2022-09-12T20:14:06.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1WQuWaiNmEDUR3tlaelVSbIwHbI=', 'Content-Length': '0'} 2022-09-12T20:14:06.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000203424457da26f45-00631f930e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.692 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:06.693 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:06.693 DEBUG:boto:Method: GET 2022-09-12T20:14:06.694 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:14:06.694 DEBUG:boto:Data: 2022-09-12T20:14:06.695 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.695 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.696 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.697 DEBUG:boto:Params: {} 2022-09-12T20:14:06.698 DEBUG:boto:Token: None 2022-09-12T20:14:06.699 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj145 2022-09-12T20:14:06.700 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1WQuWaiNmEDUR3tlaelVSbIwHbI= 2022-09-12T20:14:06.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1WQuWaiNmEDUR3tlaelVSbIwHbI=', 'Content-Length': '0'} 2022-09-12T20:14:06.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bddc347590825c0c-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.704 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:06.705 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:06.706 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:14:06.706 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:14:06.707 DEBUG:boto:Method: GET 2022-09-12T20:14:06.707 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:14:06.708 DEBUG:boto:Data: 2022-09-12T20:14:06.708 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.709 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.709 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.710 DEBUG:boto:Params: {} 2022-09-12T20:14:06.711 DEBUG:boto:Token: None 2022-09-12T20:14:06.711 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/ 2022-09-12T20:14:06.712 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw= 2022-09-12T20:14:06.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw=', 'Content-Length': '0'} 2022-09-12T20:14:06.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca8e52cb61857195-00631f930e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.716 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:06.717 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:06.717 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:06.718 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:14:06.718 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:14:06.719 DEBUG:boto:Method: GET 2022-09-12T20:14:06.719 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:14:06.720 DEBUG:boto:Data: 2022-09-12T20:14:06.720 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.721 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.722 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.722 DEBUG:boto:Params: {} 2022-09-12T20:14:06.723 DEBUG:boto:Token: None 2022-09-12T20:14:06.723 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/ 2022-09-12T20:14:06.724 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw= 2022-09-12T20:14:06.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw=', 'Content-Length': '0'} 2022-09-12T20:14:06.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051660200ba847161-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.730 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:06.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-12T20:14:06.732 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:06.732 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:06.733 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:06.733 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:06.734 DEBUG:boto:Method: GET 2022-09-12T20:14:06.734 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:06.735 DEBUG:boto:Data: 2022-09-12T20:14:06.735 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.736 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.737 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.737 DEBUG:boto:Params: {} 2022-09-12T20:14:06.738 DEBUG:boto:Token: None 2022-09-12T20:14:06.738 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:06.739 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5uLJ9ey+rWZUVeHaL+D+YtRCHaE= 2022-09-12T20:14:06.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5uLJ9ey+rWZUVeHaL+D+YtRCHaE=', 'Content-Length': '0'} 2022-09-12T20:14:06.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba2d18c864ba8f9d-00631f930e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.786 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:06.787 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:06.787 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:06.788 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:06.788 DEBUG:boto:Method: GET 2022-09-12T20:14:06.789 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:06.789 DEBUG:boto:Data: 2022-09-12T20:14:06.790 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.791 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.791 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.792 DEBUG:boto:Params: {} 2022-09-12T20:14:06.792 DEBUG:boto:Token: None 2022-09-12T20:14:06.793 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:06.794 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5uLJ9ey+rWZUVeHaL+D+YtRCHaE= 2022-09-12T20:14:06.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5uLJ9ey+rWZUVeHaL+D+YtRCHaE=', 'Content-Length': '0'} 2022-09-12T20:14:06.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e180360fcad0426-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.798 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:06.798 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:06.799 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:06.799 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:06.800 DEBUG:boto:Method: HEAD 2022-09-12T20:14:06.800 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:06.801 DEBUG:boto:Data: 2022-09-12T20:14:06.801 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.802 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.802 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.803 DEBUG:boto:Params: {} 2022-09-12T20:14:06.803 DEBUG:boto:Token: None 2022-09-12T20:14:06.804 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:06.804 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ng5m9z6I5Q6FW+cdW1yYKglaaCo= 2022-09-12T20:14:06.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ng5m9z6I5Q6FW+cdW1yYKglaaCo=', 'Content-Length': '0'} 2022-09-12T20:14:06.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f62316cc9ad30b38-00631f930e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.808 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:06.808 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:06.809 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:06.809 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:06.810 DEBUG:boto:Method: HEAD 2022-09-12T20:14:06.810 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:06.811 DEBUG:boto:Data: 2022-09-12T20:14:06.811 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.812 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.812 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.813 DEBUG:boto:Params: {} 2022-09-12T20:14:06.813 DEBUG:boto:Token: None 2022-09-12T20:14:06.814 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:06.815 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ng5m9z6I5Q6FW+cdW1yYKglaaCo= 2022-09-12T20:14:06.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ng5m9z6I5Q6FW+cdW1yYKglaaCo=', 'Content-Length': '0'} 2022-09-12T20:14:06.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000102f4309dcf7d124-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-12T20:14:06.819 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:06.819 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:06.820 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:06.820 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:06.821 DEBUG:boto:Method: GET 2022-09-12T20:14:06.822 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:06.822 DEBUG:boto:Data: 2022-09-12T20:14:06.823 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.823 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.824 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.824 DEBUG:boto:Params: {} 2022-09-12T20:14:06.825 DEBUG:boto:Token: None 2022-09-12T20:14:06.825 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:06.826 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5uLJ9ey+rWZUVeHaL+D+YtRCHaE= 2022-09-12T20:14:06.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5uLJ9ey+rWZUVeHaL+D+YtRCHaE=', 'Content-Length': '0'} 2022-09-12T20:14:06.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cab22b3843a18696-00631f930e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.829 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:06.830 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:06.831 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:06.831 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:06.832 DEBUG:boto:Method: GET 2022-09-12T20:14:06.832 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:06.833 DEBUG:boto:Data: 2022-09-12T20:14:06.833 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.834 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.834 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.835 DEBUG:boto:Params: {} 2022-09-12T20:14:06.836 DEBUG:boto:Token: None 2022-09-12T20:14:06.836 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:06.837 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5uLJ9ey+rWZUVeHaL+D+YtRCHaE= 2022-09-12T20:14:06.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5uLJ9ey+rWZUVeHaL+D+YtRCHaE=', 'Content-Length': '0'} 2022-09-12T20:14:06.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b498317815517b1e-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.840 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:06.841 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:06.842 DEBUG:boto:path=/jdpiak-75/?prefix=obj155 2022-09-12T20:14:06.842 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj155 2022-09-12T20:14:06.843 DEBUG:boto:Method: GET 2022-09-12T20:14:06.843 DEBUG:boto:Path: /jdpiak-75/?prefix=obj155 2022-09-12T20:14:06.844 DEBUG:boto:Data: 2022-09-12T20:14:06.844 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.845 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.846 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.846 DEBUG:boto:Params: {} 2022-09-12T20:14:06.847 DEBUG:boto:Token: None 2022-09-12T20:14:06.847 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/ 2022-09-12T20:14:06.848 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw= 2022-09-12T20:14:06.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw=', 'Content-Length': '0'} 2022-09-12T20:14:06.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5dfab5287558517-00631f930e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.852 DEBUG:boto:b'jdpiak-75obj1551000falseobj1552022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:06.852 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:06.853 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:06.854 DEBUG:boto:path=/jdpiak-75/?prefix=obj155 2022-09-12T20:14:06.854 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj155 2022-09-12T20:14:06.855 DEBUG:boto:Method: GET 2022-09-12T20:14:06.855 DEBUG:boto:Path: /jdpiak-75/?prefix=obj155 2022-09-12T20:14:06.856 DEBUG:boto:Data: 2022-09-12T20:14:06.856 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.857 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.857 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.858 DEBUG:boto:Params: {} 2022-09-12T20:14:06.859 DEBUG:boto:Token: None 2022-09-12T20:14:06.859 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/ 2022-09-12T20:14:06.860 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw= 2022-09-12T20:14:06.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw=', 'Content-Length': '0'} 2022-09-12T20:14:06.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5777f571ab71b79-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.865 DEBUG:boto:b'jdpiak-75obj1551000falseobj1552022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:06.866 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:06.867 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:06.867 DEBUG:boto:Method: HEAD 2022-09-12T20:14:06.868 DEBUG:boto:Path: /jdpiak-75/obj155 2022-09-12T20:14:06.868 DEBUG:boto:Data: 2022-09-12T20:14:06.869 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.869 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.870 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.870 DEBUG:boto:Params: {} 2022-09-12T20:14:06.871 DEBUG:boto:Token: None 2022-09-12T20:14:06.872 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj155 2022-09-12T20:14:06.872 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hhWxcYu9TOOA1ip1CO6CEtYPtdA= 2022-09-12T20:14:06.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hhWxcYu9TOOA1ip1CO6CEtYPtdA=', 'Content-Length': '0'} 2022-09-12T20:14:06.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000408da383239e95ad-00631f930e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.875 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:06.876 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:06.877 DEBUG:boto:Method: HEAD 2022-09-12T20:14:06.877 DEBUG:boto:Path: /jdpiak-75/obj155 2022-09-12T20:14:06.878 DEBUG:boto:Data: 2022-09-12T20:14:06.878 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.879 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.879 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.880 DEBUG:boto:Params: {} 2022-09-12T20:14:06.880 DEBUG:boto:Token: None 2022-09-12T20:14:06.881 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj155 2022-09-12T20:14:06.881 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hhWxcYu9TOOA1ip1CO6CEtYPtdA= 2022-09-12T20:14:06.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hhWxcYu9TOOA1ip1CO6CEtYPtdA=', 'Content-Length': '0'} 2022-09-12T20:14:06.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8925e39808a17c1-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-12T20:14:06.886 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:06.886 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:06.887 DEBUG:boto:Method: GET 2022-09-12T20:14:06.887 DEBUG:boto:Path: /jdpiak-75/obj155 2022-09-12T20:14:06.888 DEBUG:boto:Data: 2022-09-12T20:14:06.888 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.889 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.889 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.890 DEBUG:boto:Params: {} 2022-09-12T20:14:06.891 DEBUG:boto:Token: None 2022-09-12T20:14:06.891 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj155 2022-09-12T20:14:06.892 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L+aPx2KN64W/Qit+wU80BjNJ+7Q= 2022-09-12T20:14:06.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L+aPx2KN64W/Qit+wU80BjNJ+7Q=', 'Content-Length': '0'} 2022-09-12T20:14:06.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000406a07cf60a67780-00631f930e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.938 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:06.939 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:06.939 DEBUG:boto:Method: GET 2022-09-12T20:14:06.940 DEBUG:boto:Path: /jdpiak-75/obj155 2022-09-12T20:14:06.941 DEBUG:boto:Data: 2022-09-12T20:14:06.941 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.942 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.942 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.943 DEBUG:boto:Params: {} 2022-09-12T20:14:06.943 DEBUG:boto:Token: None 2022-09-12T20:14:06.944 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj155 2022-09-12T20:14:06.945 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L+aPx2KN64W/Qit+wU80BjNJ+7Q= 2022-09-12T20:14:06.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L+aPx2KN64W/Qit+wU80BjNJ+7Q=', 'Content-Length': '0'} 2022-09-12T20:14:06.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fef4c600f41796f1-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.948 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:06.949 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:06.949 DEBUG:boto:path=/jdpiak-75/?prefix=obj155 2022-09-12T20:14:06.950 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj155 2022-09-12T20:14:06.950 DEBUG:boto:Method: GET 2022-09-12T20:14:06.951 DEBUG:boto:Path: /jdpiak-75/?prefix=obj155 2022-09-12T20:14:06.952 DEBUG:boto:Data: 2022-09-12T20:14:06.952 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.953 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.953 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.954 DEBUG:boto:Params: {} 2022-09-12T20:14:06.954 DEBUG:boto:Token: None 2022-09-12T20:14:06.955 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/ 2022-09-12T20:14:06.956 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw= 2022-09-12T20:14:06.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw=', 'Content-Length': '0'} 2022-09-12T20:14:06.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040c460f2b66166a0-00631f930e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.959 DEBUG:boto:b'jdpiak-75obj1551000falseobj1552022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:06.960 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:06.961 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:06.961 DEBUG:boto:path=/jdpiak-75/?prefix=obj155 2022-09-12T20:14:06.962 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj155 2022-09-12T20:14:06.963 DEBUG:boto:Method: GET 2022-09-12T20:14:06.963 DEBUG:boto:Path: /jdpiak-75/?prefix=obj155 2022-09-12T20:14:06.964 DEBUG:boto:Data: 2022-09-12T20:14:06.964 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.965 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.965 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.966 DEBUG:boto:Params: {} 2022-09-12T20:14:06.966 DEBUG:boto:Token: None 2022-09-12T20:14:06.967 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/ 2022-09-12T20:14:06.967 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw= 2022-09-12T20:14:06.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yuBQ2ToMCInPjc8tw4s2ZcgB2zw=', 'Content-Length': '0'} 2022-09-12T20:14:06.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092f1be192e2a207e-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.973 DEBUG:boto:b'jdpiak-75obj1551000falseobj1552022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:06.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:14:06.974 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:06.975 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:06.976 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:06.976 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:06.977 DEBUG:boto:Method: GET 2022-09-12T20:14:06.977 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:06.978 DEBUG:boto:Data: 2022-09-12T20:14:06.978 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.979 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:06.979 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.980 DEBUG:boto:Params: {} 2022-09-12T20:14:06.981 DEBUG:boto:Token: None 2022-09-12T20:14:06.981 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:06.982 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8fvdm2C3bpauF5UUhXZkLH0Mb4Q= 2022-09-12T20:14:06.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8fvdm2C3bpauF5UUhXZkLH0Mb4Q=', 'Content-Length': '0'} 2022-09-12T20:14:06.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3d78119e35eb73d-00631f930e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:06.985 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:06.986 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:06.986 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:06.987 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:06.987 DEBUG:boto:Method: GET 2022-09-12T20:14:06.988 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:06.988 DEBUG:boto:Data: 2022-09-12T20:14:06.989 DEBUG:boto:Headers: {} 2022-09-12T20:14:06.990 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:06.990 DEBUG:boto:Port: 8000 2022-09-12T20:14:06.991 DEBUG:boto:Params: {} 2022-09-12T20:14:06.991 DEBUG:boto:Token: None 2022-09-12T20:14:06.992 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:06 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:06.992 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8fvdm2C3bpauF5UUhXZkLH0Mb4Q= 2022-09-12T20:14:06.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8fvdm2C3bpauF5UUhXZkLH0Mb4Q=', 'Content-Length': '0'} 2022-09-12T20:14:06.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c8be78557ea9c05-00631f930e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.038 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:07.039 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:07.039 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:07.040 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:07.040 DEBUG:boto:Method: HEAD 2022-09-12T20:14:07.041 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:07.041 DEBUG:boto:Data: 2022-09-12T20:14:07.042 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.042 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.043 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.044 DEBUG:boto:Params: {} 2022-09-12T20:14:07.045 DEBUG:boto:Token: None 2022-09-12T20:14:07.045 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:07.046 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:47l3NvNqC4HIF97yBUIaTTNFH+M= 2022-09-12T20:14:07.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:47l3NvNqC4HIF97yBUIaTTNFH+M=', 'Content-Length': '0'} 2022-09-12T20:14:07.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ec9e6e997889a00-00631f930f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.049 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:07.050 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:07.050 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:07.051 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:07.051 DEBUG:boto:Method: HEAD 2022-09-12T20:14:07.052 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:07.052 DEBUG:boto:Data: 2022-09-12T20:14:07.053 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.053 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.054 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.054 DEBUG:boto:Params: {} 2022-09-12T20:14:07.055 DEBUG:boto:Token: None 2022-09-12T20:14:07.055 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:07.056 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:47l3NvNqC4HIF97yBUIaTTNFH+M= 2022-09-12T20:14:07.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:47l3NvNqC4HIF97yBUIaTTNFH+M=', 'Content-Length': '0'} 2022-09-12T20:14:07.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ae3b3274a8063be-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:14:07.060 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:07.060 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:07.061 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:07.062 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:07.062 DEBUG:boto:Method: GET 2022-09-12T20:14:07.063 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:07.063 DEBUG:boto:Data: 2022-09-12T20:14:07.064 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.064 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.065 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.065 DEBUG:boto:Params: {} 2022-09-12T20:14:07.066 DEBUG:boto:Token: None 2022-09-12T20:14:07.066 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:07.067 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bBbMiG0FqDVFcpWtrULVTN4rwbw= 2022-09-12T20:14:07.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bBbMiG0FqDVFcpWtrULVTN4rwbw=', 'Content-Length': '0'} 2022-09-12T20:14:07.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e044b8b3fe64734-00631f930f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.070 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:07.071 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:07.071 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:07.072 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:07.072 DEBUG:boto:Method: GET 2022-09-12T20:14:07.073 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:07.074 DEBUG:boto:Data: 2022-09-12T20:14:07.074 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.075 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.075 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.076 DEBUG:boto:Params: {} 2022-09-12T20:14:07.076 DEBUG:boto:Token: None 2022-09-12T20:14:07.077 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:07.077 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bBbMiG0FqDVFcpWtrULVTN4rwbw= 2022-09-12T20:14:07.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bBbMiG0FqDVFcpWtrULVTN4rwbw=', 'Content-Length': '0'} 2022-09-12T20:14:07.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5261f32defee4be-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.081 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:07.081 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:07.082 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:14:07.082 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:14:07.083 DEBUG:boto:Method: GET 2022-09-12T20:14:07.083 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:14:07.084 DEBUG:boto:Data: 2022-09-12T20:14:07.084 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.085 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.085 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.086 DEBUG:boto:Params: {} 2022-09-12T20:14:07.087 DEBUG:boto:Token: None 2022-09-12T20:14:07.087 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/ 2022-09-12T20:14:07.088 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8= 2022-09-12T20:14:07.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8=', 'Content-Length': '0'} 2022-09-12T20:14:07.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000514fd125d91a40fc-00631f930f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.091 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:07.092 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:07.093 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:07.093 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:14:07.094 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:14:07.094 DEBUG:boto:Method: GET 2022-09-12T20:14:07.095 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:14:07.095 DEBUG:boto:Data: 2022-09-12T20:14:07.096 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.096 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.097 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.097 DEBUG:boto:Params: {} 2022-09-12T20:14:07.098 DEBUG:boto:Token: None 2022-09-12T20:14:07.099 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/ 2022-09-12T20:14:07.099 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8= 2022-09-12T20:14:07.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8=', 'Content-Length': '0'} 2022-09-12T20:14:07.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8fa696ebc596414-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.105 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:07.105 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:07.106 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:07.106 DEBUG:boto:Method: HEAD 2022-09-12T20:14:07.107 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:14:07.108 DEBUG:boto:Data: 2022-09-12T20:14:07.108 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.109 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.109 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.110 DEBUG:boto:Params: {} 2022-09-12T20:14:07.110 DEBUG:boto:Token: None 2022-09-12T20:14:07.111 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj17 2022-09-12T20:14:07.111 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qDmjd9DYHvJfh6RmLQdwrICmjBM= 2022-09-12T20:14:07.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qDmjd9DYHvJfh6RmLQdwrICmjBM=', 'Content-Length': '0'} 2022-09-12T20:14:07.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000836d5f49f562bbbe-00631f930f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.114 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:07.115 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:07.115 DEBUG:boto:Method: HEAD 2022-09-12T20:14:07.116 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:14:07.116 DEBUG:boto:Data: 2022-09-12T20:14:07.117 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.117 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.118 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.118 DEBUG:boto:Params: {} 2022-09-12T20:14:07.119 DEBUG:boto:Token: None 2022-09-12T20:14:07.120 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj17 2022-09-12T20:14:07.120 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qDmjd9DYHvJfh6RmLQdwrICmjBM= 2022-09-12T20:14:07.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qDmjd9DYHvJfh6RmLQdwrICmjBM=', 'Content-Length': '0'} 2022-09-12T20:14:07.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c702b0e6675072c0-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:14:07.124 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:07.125 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:07.125 DEBUG:boto:Method: GET 2022-09-12T20:14:07.126 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:14:07.126 DEBUG:boto:Data: 2022-09-12T20:14:07.127 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.127 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.128 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.128 DEBUG:boto:Params: {} 2022-09-12T20:14:07.129 DEBUG:boto:Token: None 2022-09-12T20:14:07.129 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj17 2022-09-12T20:14:07.130 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qDGdLeRVfXX9LFbMOfUvH/0mGes= 2022-09-12T20:14:07.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qDGdLeRVfXX9LFbMOfUvH/0mGes=', 'Content-Length': '0'} 2022-09-12T20:14:07.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebfb005227bbd1e4-00631f930f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.174 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:07.175 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:07.175 DEBUG:boto:Method: GET 2022-09-12T20:14:07.176 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:14:07.176 DEBUG:boto:Data: 2022-09-12T20:14:07.177 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.178 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.178 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.179 DEBUG:boto:Params: {} 2022-09-12T20:14:07.179 DEBUG:boto:Token: None 2022-09-12T20:14:07.180 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj17 2022-09-12T20:14:07.181 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qDGdLeRVfXX9LFbMOfUvH/0mGes= 2022-09-12T20:14:07.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qDGdLeRVfXX9LFbMOfUvH/0mGes=', 'Content-Length': '0'} 2022-09-12T20:14:07.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a2b1ac8ff2cf25e-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.184 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:07.185 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:07.186 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:14:07.186 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:14:07.187 DEBUG:boto:Method: GET 2022-09-12T20:14:07.187 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:14:07.188 DEBUG:boto:Data: 2022-09-12T20:14:07.188 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.189 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.189 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.190 DEBUG:boto:Params: {} 2022-09-12T20:14:07.190 DEBUG:boto:Token: None 2022-09-12T20:14:07.191 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/ 2022-09-12T20:14:07.192 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8= 2022-09-12T20:14:07.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8=', 'Content-Length': '0'} 2022-09-12T20:14:07.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094b0f3b887e040e4-00631f930f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.195 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:07.196 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:07.197 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:07.197 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:14:07.198 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:14:07.198 DEBUG:boto:Method: GET 2022-09-12T20:14:07.199 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:14:07.199 DEBUG:boto:Data: 2022-09-12T20:14:07.200 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.201 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.201 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.202 DEBUG:boto:Params: {} 2022-09-12T20:14:07.202 DEBUG:boto:Token: None 2022-09-12T20:14:07.203 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/ 2022-09-12T20:14:07.204 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8= 2022-09-12T20:14:07.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8=', 'Content-Length': '0'} 2022-09-12T20:14:07.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064ad2e6e38d3654e-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.209 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:07.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:14:07.211 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:07.211 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:07.212 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:07.212 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:07.213 DEBUG:boto:Method: GET 2022-09-12T20:14:07.213 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:07.214 DEBUG:boto:Data: 2022-09-12T20:14:07.214 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.215 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.216 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.216 DEBUG:boto:Params: {} 2022-09-12T20:14:07.217 DEBUG:boto:Token: None 2022-09-12T20:14:07.217 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:07.218 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UfqJ4ZthAPvuiJ5OnoU7n1zL19o= 2022-09-12T20:14:07.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UfqJ4ZthAPvuiJ5OnoU7n1zL19o=', 'Content-Length': '0'} 2022-09-12T20:14:07.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dacc25b9b65b318-00631f930f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.222 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:07.222 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:07.223 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:07.223 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:07.224 DEBUG:boto:Method: GET 2022-09-12T20:14:07.224 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:07.225 DEBUG:boto:Data: 2022-09-12T20:14:07.225 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.226 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.226 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.227 DEBUG:boto:Params: {} 2022-09-12T20:14:07.228 DEBUG:boto:Token: None 2022-09-12T20:14:07.228 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:07.229 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UfqJ4ZthAPvuiJ5OnoU7n1zL19o= 2022-09-12T20:14:07.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UfqJ4ZthAPvuiJ5OnoU7n1zL19o=', 'Content-Length': '0'} 2022-09-12T20:14:07.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cb6befd308efad2-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.233 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:07.233 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:07.234 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:07.234 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:07.235 DEBUG:boto:Method: HEAD 2022-09-12T20:14:07.235 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:07.236 DEBUG:boto:Data: 2022-09-12T20:14:07.236 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.237 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.237 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.238 DEBUG:boto:Params: {} 2022-09-12T20:14:07.239 DEBUG:boto:Token: None 2022-09-12T20:14:07.239 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:07.240 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GPDY+JVY0FqKnTnnh0jL6AcpkV0= 2022-09-12T20:14:07.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GPDY+JVY0FqKnTnnh0jL6AcpkV0=', 'Content-Length': '0'} 2022-09-12T20:14:07.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e73d374b2b176b6-00631f930f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.243 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:07.244 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:07.244 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:07.245 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:07.245 DEBUG:boto:Method: HEAD 2022-09-12T20:14:07.246 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:07.246 DEBUG:boto:Data: 2022-09-12T20:14:07.247 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.247 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.248 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.248 DEBUG:boto:Params: {} 2022-09-12T20:14:07.249 DEBUG:boto:Token: None 2022-09-12T20:14:07.250 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:07.250 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GPDY+JVY0FqKnTnnh0jL6AcpkV0= 2022-09-12T20:14:07.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GPDY+JVY0FqKnTnnh0jL6AcpkV0=', 'Content-Length': '0'} 2022-09-12T20:14:07.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7d54835279dbe41-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:14:07.255 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:07.255 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:07.256 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:07.256 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:07.257 DEBUG:boto:Method: GET 2022-09-12T20:14:07.258 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:07.258 DEBUG:boto:Data: 2022-09-12T20:14:07.259 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.259 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.260 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.260 DEBUG:boto:Params: {} 2022-09-12T20:14:07.261 DEBUG:boto:Token: None 2022-09-12T20:14:07.262 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:07.262 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UfqJ4ZthAPvuiJ5OnoU7n1zL19o= 2022-09-12T20:14:07.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UfqJ4ZthAPvuiJ5OnoU7n1zL19o=', 'Content-Length': '0'} 2022-09-12T20:14:07.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002108ea9a1ca143b2-00631f930f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.306 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:07.307 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:07.307 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:07.308 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:07.308 DEBUG:boto:Method: GET 2022-09-12T20:14:07.309 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:07.309 DEBUG:boto:Data: 2022-09-12T20:14:07.310 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.310 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.311 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.311 DEBUG:boto:Params: {} 2022-09-12T20:14:07.312 DEBUG:boto:Token: None 2022-09-12T20:14:07.313 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:07.313 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UfqJ4ZthAPvuiJ5OnoU7n1zL19o= 2022-09-12T20:14:07.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UfqJ4ZthAPvuiJ5OnoU7n1zL19o=', 'Content-Length': '0'} 2022-09-12T20:14:07.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036c425f9b422a3bd-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.317 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:07.317 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:07.318 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:14:07.318 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:14:07.319 DEBUG:boto:Method: GET 2022-09-12T20:14:07.319 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:14:07.320 DEBUG:boto:Data: 2022-09-12T20:14:07.320 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.321 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.321 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.322 DEBUG:boto:Params: {} 2022-09-12T20:14:07.323 DEBUG:boto:Token: None 2022-09-12T20:14:07.323 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/ 2022-09-12T20:14:07.324 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8= 2022-09-12T20:14:07.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8=', 'Content-Length': '0'} 2022-09-12T20:14:07.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053307229566d986c-00631f930f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.327 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:07.328 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:07.329 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:07.329 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:14:07.330 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:14:07.330 DEBUG:boto:Method: GET 2022-09-12T20:14:07.331 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:14:07.331 DEBUG:boto:Data: 2022-09-12T20:14:07.332 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.332 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.333 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.333 DEBUG:boto:Params: {} 2022-09-12T20:14:07.334 DEBUG:boto:Token: None 2022-09-12T20:14:07.335 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/ 2022-09-12T20:14:07.335 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8= 2022-09-12T20:14:07.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8=', 'Content-Length': '0'} 2022-09-12T20:14:07.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dfe492fcaf5184a-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.341 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:07.342 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:07.342 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:07.343 DEBUG:boto:Method: HEAD 2022-09-12T20:14:07.343 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:14:07.344 DEBUG:boto:Data: 2022-09-12T20:14:07.344 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.345 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.345 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.345 DEBUG:boto:Params: {} 2022-09-12T20:14:07.346 DEBUG:boto:Token: None 2022-09-12T20:14:07.347 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj19 2022-09-12T20:14:07.347 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EtCt5OAZLY79U9iE7mVbvXLaRF4= 2022-09-12T20:14:07.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EtCt5OAZLY79U9iE7mVbvXLaRF4=', 'Content-Length': '0'} 2022-09-12T20:14:07.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2d8fd333c4d5007-00631f930f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.350 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:07.351 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:07.351 DEBUG:boto:Method: HEAD 2022-09-12T20:14:07.352 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:14:07.352 DEBUG:boto:Data: 2022-09-12T20:14:07.353 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.353 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.354 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.354 DEBUG:boto:Params: {} 2022-09-12T20:14:07.355 DEBUG:boto:Token: None 2022-09-12T20:14:07.355 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj19 2022-09-12T20:14:07.356 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EtCt5OAZLY79U9iE7mVbvXLaRF4= 2022-09-12T20:14:07.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EtCt5OAZLY79U9iE7mVbvXLaRF4=', 'Content-Length': '0'} 2022-09-12T20:14:07.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000948bec6f8a71caa4-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:14:07.360 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:07.360 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:07.361 DEBUG:boto:Method: GET 2022-09-12T20:14:07.361 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:14:07.362 DEBUG:boto:Data: 2022-09-12T20:14:07.362 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.363 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.364 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.364 DEBUG:boto:Params: {} 2022-09-12T20:14:07.365 DEBUG:boto:Token: None 2022-09-12T20:14:07.365 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj19 2022-09-12T20:14:07.366 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/sTHiPDE+IZZZszgR5kAGDpr9Lw= 2022-09-12T20:14:07.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/sTHiPDE+IZZZszgR5kAGDpr9Lw=', 'Content-Length': '0'} 2022-09-12T20:14:07.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003812c719515056af-00631f930f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.369 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:07.370 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:07.370 DEBUG:boto:Method: GET 2022-09-12T20:14:07.371 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:14:07.371 DEBUG:boto:Data: 2022-09-12T20:14:07.372 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.372 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.373 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.373 DEBUG:boto:Params: {} 2022-09-12T20:14:07.374 DEBUG:boto:Token: None 2022-09-12T20:14:07.375 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj19 2022-09-12T20:14:07.375 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/sTHiPDE+IZZZszgR5kAGDpr9Lw= 2022-09-12T20:14:07.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/sTHiPDE+IZZZszgR5kAGDpr9Lw=', 'Content-Length': '0'} 2022-09-12T20:14:07.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d1ae72fae16d091-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.422 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:07.423 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:07.423 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:14:07.424 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:14:07.424 DEBUG:boto:Method: GET 2022-09-12T20:14:07.425 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:14:07.425 DEBUG:boto:Data: 2022-09-12T20:14:07.426 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.427 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.427 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.428 DEBUG:boto:Params: {} 2022-09-12T20:14:07.428 DEBUG:boto:Token: None 2022-09-12T20:14:07.429 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/ 2022-09-12T20:14:07.430 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8= 2022-09-12T20:14:07.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8=', 'Content-Length': '0'} 2022-09-12T20:14:07.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9526a8f5e5d9af9-00631f930f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.434 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:07.435 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:07.435 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:07.436 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:14:07.436 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:14:07.437 DEBUG:boto:Method: GET 2022-09-12T20:14:07.437 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:14:07.438 DEBUG:boto:Data: 2022-09-12T20:14:07.438 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.439 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.439 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.440 DEBUG:boto:Params: {} 2022-09-12T20:14:07.440 DEBUG:boto:Token: None 2022-09-12T20:14:07.441 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/ 2022-09-12T20:14:07.442 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8= 2022-09-12T20:14:07.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8=', 'Content-Length': '0'} 2022-09-12T20:14:07.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000213aaae42068b749-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.447 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:07.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:14:07.448 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:07.449 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:07.449 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:07.450 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:07.450 DEBUG:boto:Method: GET 2022-09-12T20:14:07.451 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:07.452 DEBUG:boto:Data: 2022-09-12T20:14:07.452 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.453 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.453 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.454 DEBUG:boto:Params: {} 2022-09-12T20:14:07.454 DEBUG:boto:Token: None 2022-09-12T20:14:07.455 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:07.455 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tAotppnJNuME1kwjGjKKeFtIs18= 2022-09-12T20:14:07.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tAotppnJNuME1kwjGjKKeFtIs18=', 'Content-Length': '0'} 2022-09-12T20:14:07.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbefb46d1ca1502e-00631f930f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.459 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:07.459 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:07.460 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:07.460 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:07.461 DEBUG:boto:Method: GET 2022-09-12T20:14:07.461 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:07.462 DEBUG:boto:Data: 2022-09-12T20:14:07.462 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.463 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.463 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.464 DEBUG:boto:Params: {} 2022-09-12T20:14:07.465 DEBUG:boto:Token: None 2022-09-12T20:14:07.465 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:07.466 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tAotppnJNuME1kwjGjKKeFtIs18= 2022-09-12T20:14:07.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tAotppnJNuME1kwjGjKKeFtIs18=', 'Content-Length': '0'} 2022-09-12T20:14:07.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d9df0a9a1d0583c-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.469 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:07.470 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:07.470 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:07.471 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:07.471 DEBUG:boto:Method: HEAD 2022-09-12T20:14:07.472 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:07.472 DEBUG:boto:Data: 2022-09-12T20:14:07.473 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.473 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.474 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.474 DEBUG:boto:Params: {} 2022-09-12T20:14:07.475 DEBUG:boto:Token: None 2022-09-12T20:14:07.475 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:07.476 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mE4c4L0WV5hgZnmjHmuMdpHJxsM= 2022-09-12T20:14:07.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mE4c4L0WV5hgZnmjHmuMdpHJxsM=', 'Content-Length': '0'} 2022-09-12T20:14:07.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d04d9803eb3ae00-00631f930f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.479 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:07.480 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:07.480 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:07.481 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:07.481 DEBUG:boto:Method: HEAD 2022-09-12T20:14:07.482 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:07.482 DEBUG:boto:Data: 2022-09-12T20:14:07.483 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.483 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.484 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.484 DEBUG:boto:Params: {} 2022-09-12T20:14:07.485 DEBUG:boto:Token: None 2022-09-12T20:14:07.485 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:07.486 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mE4c4L0WV5hgZnmjHmuMdpHJxsM= 2022-09-12T20:14:07.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mE4c4L0WV5hgZnmjHmuMdpHJxsM=', 'Content-Length': '0'} 2022-09-12T20:14:07.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091ac97d96d9a9256-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:14:07.490 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:07.490 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:07.491 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:07.491 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:07.492 DEBUG:boto:Method: GET 2022-09-12T20:14:07.492 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:07.493 DEBUG:boto:Data: 2022-09-12T20:14:07.493 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.494 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.495 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.495 DEBUG:boto:Params: {} 2022-09-12T20:14:07.496 DEBUG:boto:Token: None 2022-09-12T20:14:07.496 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:07.497 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tAotppnJNuME1kwjGjKKeFtIs18= 2022-09-12T20:14:07.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tAotppnJNuME1kwjGjKKeFtIs18=', 'Content-Length': '0'} 2022-09-12T20:14:07.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024d77da8eb98354a-00631f930f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.542 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:07.543 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:07.543 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:07.544 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:07.544 DEBUG:boto:Method: GET 2022-09-12T20:14:07.545 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:07.545 DEBUG:boto:Data: 2022-09-12T20:14:07.546 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.546 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.547 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.547 DEBUG:boto:Params: {} 2022-09-12T20:14:07.548 DEBUG:boto:Token: None 2022-09-12T20:14:07.549 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:07.549 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tAotppnJNuME1kwjGjKKeFtIs18= 2022-09-12T20:14:07.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tAotppnJNuME1kwjGjKKeFtIs18=', 'Content-Length': '0'} 2022-09-12T20:14:07.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dae51c8383d4f987-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.553 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:07.553 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:07.554 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:14:07.555 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:14:07.555 DEBUG:boto:Method: GET 2022-09-12T20:14:07.556 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:14:07.556 DEBUG:boto:Data: 2022-09-12T20:14:07.557 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.557 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.558 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.558 DEBUG:boto:Params: {} 2022-09-12T20:14:07.559 DEBUG:boto:Token: None 2022-09-12T20:14:07.559 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/ 2022-09-12T20:14:07.560 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8= 2022-09-12T20:14:07.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8=', 'Content-Length': '0'} 2022-09-12T20:14:07.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab69c896ae3bbc04-00631f930f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.564 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:07.565 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:07.565 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:07.566 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:14:07.566 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:14:07.567 DEBUG:boto:Method: GET 2022-09-12T20:14:07.567 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:14:07.568 DEBUG:boto:Data: 2022-09-12T20:14:07.568 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.569 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.569 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.570 DEBUG:boto:Params: {} 2022-09-12T20:14:07.571 DEBUG:boto:Token: None 2022-09-12T20:14:07.571 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/ 2022-09-12T20:14:07.572 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8= 2022-09-12T20:14:07.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8=', 'Content-Length': '0'} 2022-09-12T20:14:07.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039048f37509b8a53-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.578 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:07.578 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:07.579 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:07.579 DEBUG:boto:Method: HEAD 2022-09-12T20:14:07.580 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:14:07.580 DEBUG:boto:Data: 2022-09-12T20:14:07.581 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.581 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.582 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.582 DEBUG:boto:Params: {} 2022-09-12T20:14:07.583 DEBUG:boto:Token: None 2022-09-12T20:14:07.583 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj22 2022-09-12T20:14:07.584 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PWkRe6h/L+tYUe0cyG06veeIbyU= 2022-09-12T20:14:07.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PWkRe6h/L+tYUe0cyG06veeIbyU=', 'Content-Length': '0'} 2022-09-12T20:14:07.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002371b773efbfc989-00631f930f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.587 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:07.588 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:07.588 DEBUG:boto:Method: HEAD 2022-09-12T20:14:07.589 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:14:07.589 DEBUG:boto:Data: 2022-09-12T20:14:07.590 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.590 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.591 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.591 DEBUG:boto:Params: {} 2022-09-12T20:14:07.592 DEBUG:boto:Token: None 2022-09-12T20:14:07.592 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj22 2022-09-12T20:14:07.593 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PWkRe6h/L+tYUe0cyG06veeIbyU= 2022-09-12T20:14:07.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PWkRe6h/L+tYUe0cyG06veeIbyU=', 'Content-Length': '0'} 2022-09-12T20:14:07.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000939b01b77cb1c87d-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:14:07.597 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:07.597 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:07.598 DEBUG:boto:Method: GET 2022-09-12T20:14:07.598 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:14:07.599 DEBUG:boto:Data: 2022-09-12T20:14:07.599 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.600 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.600 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.601 DEBUG:boto:Params: {} 2022-09-12T20:14:07.602 DEBUG:boto:Token: None 2022-09-12T20:14:07.602 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj22 2022-09-12T20:14:07.603 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aTMrrYMscbfJca9QGfW5acZkX0g= 2022-09-12T20:14:07.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aTMrrYMscbfJca9QGfW5acZkX0g=', 'Content-Length': '0'} 2022-09-12T20:14:07.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f31c5c7cc98a7ea3-00631f930f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.646 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:07.647 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:07.647 DEBUG:boto:Method: GET 2022-09-12T20:14:07.648 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:14:07.648 DEBUG:boto:Data: 2022-09-12T20:14:07.649 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.649 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.650 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.650 DEBUG:boto:Params: {} 2022-09-12T20:14:07.651 DEBUG:boto:Token: None 2022-09-12T20:14:07.651 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj22 2022-09-12T20:14:07.652 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aTMrrYMscbfJca9QGfW5acZkX0g= 2022-09-12T20:14:07.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aTMrrYMscbfJca9QGfW5acZkX0g=', 'Content-Length': '0'} 2022-09-12T20:14:07.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000224122dcbbab8fab-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.656 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:07.656 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:07.657 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:14:07.657 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:14:07.658 DEBUG:boto:Method: GET 2022-09-12T20:14:07.658 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:14:07.659 DEBUG:boto:Data: 2022-09-12T20:14:07.659 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.660 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.660 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.661 DEBUG:boto:Params: {} 2022-09-12T20:14:07.661 DEBUG:boto:Token: None 2022-09-12T20:14:07.662 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/ 2022-09-12T20:14:07.663 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8= 2022-09-12T20:14:07.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8=', 'Content-Length': '0'} 2022-09-12T20:14:07.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007693f1e0fb8bcfe9-00631f930f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.667 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:07.667 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:07.668 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:07.668 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:14:07.669 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:14:07.669 DEBUG:boto:Method: GET 2022-09-12T20:14:07.670 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:14:07.671 DEBUG:boto:Data: 2022-09-12T20:14:07.671 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.672 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.672 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.673 DEBUG:boto:Params: {} 2022-09-12T20:14:07.673 DEBUG:boto:Token: None 2022-09-12T20:14:07.674 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/ 2022-09-12T20:14:07.683 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8= 2022-09-12T20:14:07.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8=', 'Content-Length': '0'} 2022-09-12T20:14:07.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006136d6eea9d7d866-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.689 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:07.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:14:07.690 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:07.691 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:07.691 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:07.692 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:07.692 DEBUG:boto:Method: GET 2022-09-12T20:14:07.693 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:07.693 DEBUG:boto:Data: 2022-09-12T20:14:07.694 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.695 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.695 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.696 DEBUG:boto:Params: {} 2022-09-12T20:14:07.696 DEBUG:boto:Token: None 2022-09-12T20:14:07.697 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:07.698 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tHFPCMl9mgUaOQd7QKdql0GbKss= 2022-09-12T20:14:07.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tHFPCMl9mgUaOQd7QKdql0GbKss=', 'Content-Length': '0'} 2022-09-12T20:14:07.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e44e4fae916d1042-00631f930f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.701 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:07.702 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:07.702 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:07.703 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:07.703 DEBUG:boto:Method: GET 2022-09-12T20:14:07.704 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:07.704 DEBUG:boto:Data: 2022-09-12T20:14:07.705 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.705 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.706 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.706 DEBUG:boto:Params: {} 2022-09-12T20:14:07.707 DEBUG:boto:Token: None 2022-09-12T20:14:07.708 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:07.708 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tHFPCMl9mgUaOQd7QKdql0GbKss= 2022-09-12T20:14:07.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tHFPCMl9mgUaOQd7QKdql0GbKss=', 'Content-Length': '0'} 2022-09-12T20:14:07.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dab3cca4b9b50784-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.712 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:07.712 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:07.713 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:07.713 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:07.714 DEBUG:boto:Method: HEAD 2022-09-12T20:14:07.714 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:07.715 DEBUG:boto:Data: 2022-09-12T20:14:07.715 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.716 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.716 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.717 DEBUG:boto:Params: {} 2022-09-12T20:14:07.717 DEBUG:boto:Token: None 2022-09-12T20:14:07.718 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:07.718 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nwg+SlCBtcsYVTUJ0UPVt+VIeXA= 2022-09-12T20:14:07.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nwg+SlCBtcsYVTUJ0UPVt+VIeXA=', 'Content-Length': '0'} 2022-09-12T20:14:07.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000389e6ea4f06b2811-00631f930f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.722 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:07.722 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:07.723 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:07.723 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:07.724 DEBUG:boto:Method: HEAD 2022-09-12T20:14:07.724 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:07.725 DEBUG:boto:Data: 2022-09-12T20:14:07.725 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.726 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.726 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.727 DEBUG:boto:Params: {} 2022-09-12T20:14:07.727 DEBUG:boto:Token: None 2022-09-12T20:14:07.728 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:07.728 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nwg+SlCBtcsYVTUJ0UPVt+VIeXA= 2022-09-12T20:14:07.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nwg+SlCBtcsYVTUJ0UPVt+VIeXA=', 'Content-Length': '0'} 2022-09-12T20:14:07.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b7c76a422b0d6e5-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:14:07.732 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:07.733 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:07.733 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:07.734 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:07.734 DEBUG:boto:Method: GET 2022-09-12T20:14:07.735 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:07.736 DEBUG:boto:Data: 2022-09-12T20:14:07.736 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.737 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.737 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.738 DEBUG:boto:Params: {} 2022-09-12T20:14:07.738 DEBUG:boto:Token: None 2022-09-12T20:14:07.739 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:07.740 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tHFPCMl9mgUaOQd7QKdql0GbKss= 2022-09-12T20:14:07.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tHFPCMl9mgUaOQd7QKdql0GbKss=', 'Content-Length': '0'} 2022-09-12T20:14:07.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001429cdbc385d4452-00631f930f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.743 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:07.744 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:07.745 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:07.745 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:07.746 DEBUG:boto:Method: GET 2022-09-12T20:14:07.746 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:07.747 DEBUG:boto:Data: 2022-09-12T20:14:07.747 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.748 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.748 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.749 DEBUG:boto:Params: {} 2022-09-12T20:14:07.750 DEBUG:boto:Token: None 2022-09-12T20:14:07.750 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:07.751 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tHFPCMl9mgUaOQd7QKdql0GbKss= 2022-09-12T20:14:07.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tHFPCMl9mgUaOQd7QKdql0GbKss=', 'Content-Length': '0'} 2022-09-12T20:14:07.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bccd4eb2a2a12798-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.754 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:07.755 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:07.755 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:14:07.756 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:14:07.757 DEBUG:boto:Method: GET 2022-09-12T20:14:07.757 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:14:07.758 DEBUG:boto:Data: 2022-09-12T20:14:07.758 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.759 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.759 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.760 DEBUG:boto:Params: {} 2022-09-12T20:14:07.760 DEBUG:boto:Token: None 2022-09-12T20:14:07.761 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/ 2022-09-12T20:14:07.762 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8= 2022-09-12T20:14:07.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8=', 'Content-Length': '0'} 2022-09-12T20:14:07.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000faaf5900b13f1090-00631f930f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.806 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:07.807 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:07.808 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:07.808 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:14:07.809 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:14:07.809 DEBUG:boto:Method: GET 2022-09-12T20:14:07.810 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:14:07.810 DEBUG:boto:Data: 2022-09-12T20:14:07.811 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.811 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.812 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.812 DEBUG:boto:Params: {} 2022-09-12T20:14:07.813 DEBUG:boto:Token: None 2022-09-12T20:14:07.814 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/ 2022-09-12T20:14:07.814 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8= 2022-09-12T20:14:07.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8=', 'Content-Length': '0'} 2022-09-12T20:14:07.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e98a5b424eb4e8c3-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.820 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:07.821 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:07.821 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:07.822 DEBUG:boto:Method: HEAD 2022-09-12T20:14:07.822 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:14:07.823 DEBUG:boto:Data: 2022-09-12T20:14:07.823 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.824 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.825 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.825 DEBUG:boto:Params: {} 2022-09-12T20:14:07.826 DEBUG:boto:Token: None 2022-09-12T20:14:07.826 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj23 2022-09-12T20:14:07.827 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:v7hm7j/cNvb5vy+STAxCs6xpFe4= 2022-09-12T20:14:07.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:v7hm7j/cNvb5vy+STAxCs6xpFe4=', 'Content-Length': '0'} 2022-09-12T20:14:07.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b75bf4050e75e1b6-00631f930f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.830 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:07.831 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:07.831 DEBUG:boto:Method: HEAD 2022-09-12T20:14:07.832 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:14:07.833 DEBUG:boto:Data: 2022-09-12T20:14:07.833 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.834 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.834 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.835 DEBUG:boto:Params: {} 2022-09-12T20:14:07.835 DEBUG:boto:Token: None 2022-09-12T20:14:07.836 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj23 2022-09-12T20:14:07.836 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:v7hm7j/cNvb5vy+STAxCs6xpFe4= 2022-09-12T20:14:07.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:v7hm7j/cNvb5vy+STAxCs6xpFe4=', 'Content-Length': '0'} 2022-09-12T20:14:07.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab57305ca4f60d2d-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:14:07.841 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:07.841 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:07.842 DEBUG:boto:Method: GET 2022-09-12T20:14:07.843 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:14:07.843 DEBUG:boto:Data: 2022-09-12T20:14:07.844 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.844 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.845 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.846 DEBUG:boto:Params: {} 2022-09-12T20:14:07.846 DEBUG:boto:Token: None 2022-09-12T20:14:07.847 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj23 2022-09-12T20:14:07.848 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1rVpqMNpJFasbnusS7e7C/suKGM= 2022-09-12T20:14:07.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1rVpqMNpJFasbnusS7e7C/suKGM=', 'Content-Length': '0'} 2022-09-12T20:14:07.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005389d7b52f3b559a-00631f930f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.852 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:07.852 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:07.853 DEBUG:boto:Method: GET 2022-09-12T20:14:07.853 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:14:07.855 DEBUG:boto:Data: 2022-09-12T20:14:07.855 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.856 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.857 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.858 DEBUG:boto:Params: {} 2022-09-12T20:14:07.858 DEBUG:boto:Token: None 2022-09-12T20:14:07.859 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj23 2022-09-12T20:14:07.860 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1rVpqMNpJFasbnusS7e7C/suKGM= 2022-09-12T20:14:07.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1rVpqMNpJFasbnusS7e7C/suKGM=', 'Content-Length': '0'} 2022-09-12T20:14:07.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004899cf17316ff2b6-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.906 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:07.907 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:07.907 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:14:07.908 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:14:07.909 DEBUG:boto:Method: GET 2022-09-12T20:14:07.910 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:14:07.911 DEBUG:boto:Data: 2022-09-12T20:14:07.911 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.912 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.913 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.914 DEBUG:boto:Params: {} 2022-09-12T20:14:07.915 DEBUG:boto:Token: None 2022-09-12T20:14:07.915 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/ 2022-09-12T20:14:07.916 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8= 2022-09-12T20:14:07.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8=', 'Content-Length': '0'} 2022-09-12T20:14:07.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a02b2b69e3b43a79-00631f930f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.920 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:07.921 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:07.922 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:07.923 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:14:07.923 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:14:07.924 DEBUG:boto:Method: GET 2022-09-12T20:14:07.924 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:14:07.925 DEBUG:boto:Data: 2022-09-12T20:14:07.926 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.926 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.927 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.927 DEBUG:boto:Params: {} 2022-09-12T20:14:07.928 DEBUG:boto:Token: None 2022-09-12T20:14:07.929 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/ 2022-09-12T20:14:07.929 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8= 2022-09-12T20:14:07.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MNTjRLMqnvmZI03shCju/Igujz8=', 'Content-Length': '0'} 2022-09-12T20:14:07.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c920d966f2c1eb91-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.935 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:07.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:14:07.937 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:07.937 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:07.938 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:07.939 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:07.939 DEBUG:boto:Method: GET 2022-09-12T20:14:07.941 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:07.941 DEBUG:boto:Data: 2022-09-12T20:14:07.942 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.943 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.943 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.944 DEBUG:boto:Params: {} 2022-09-12T20:14:07.945 DEBUG:boto:Token: None 2022-09-12T20:14:07.945 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:07.946 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UrexPLkQlw5n7cl/6TkBZkqNfL8= 2022-09-12T20:14:07.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UrexPLkQlw5n7cl/6TkBZkqNfL8=', 'Content-Length': '0'} 2022-09-12T20:14:07.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cecded7b281622b-00631f930f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.950 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:07.951 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:07.952 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:07.952 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:07.953 DEBUG:boto:Method: GET 2022-09-12T20:14:07.954 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:07.955 DEBUG:boto:Data: 2022-09-12T20:14:07.956 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.957 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.957 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.958 DEBUG:boto:Params: {} 2022-09-12T20:14:07.959 DEBUG:boto:Token: None 2022-09-12T20:14:07.960 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:07.961 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UrexPLkQlw5n7cl/6TkBZkqNfL8= 2022-09-12T20:14:07.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UrexPLkQlw5n7cl/6TkBZkqNfL8=', 'Content-Length': '0'} 2022-09-12T20:14:07.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ea47747849578d7-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.966 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:07.966 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:07.967 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:07.968 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:07.968 DEBUG:boto:Method: HEAD 2022-09-12T20:14:07.969 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:07.970 DEBUG:boto:Data: 2022-09-12T20:14:07.970 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.971 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.971 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.972 DEBUG:boto:Params: {} 2022-09-12T20:14:07.973 DEBUG:boto:Token: None 2022-09-12T20:14:07.973 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:07.974 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1KNrdLpXvtxdJ+pAGa9qKuXNdvk= 2022-09-12T20:14:07.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1KNrdLpXvtxdJ+pAGa9qKuXNdvk=', 'Content-Length': '0'} 2022-09-12T20:14:07.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078f4a0cd25016837-00631f930f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.978 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:07.978 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:07.979 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:07.980 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:07.980 DEBUG:boto:Method: HEAD 2022-09-12T20:14:07.981 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:07.981 DEBUG:boto:Data: 2022-09-12T20:14:07.982 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.983 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:07.983 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.984 DEBUG:boto:Params: {} 2022-09-12T20:14:07.985 DEBUG:boto:Token: None 2022-09-12T20:14:07.985 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:07.986 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1KNrdLpXvtxdJ+pAGa9qKuXNdvk= 2022-09-12T20:14:07.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1KNrdLpXvtxdJ+pAGa9qKuXNdvk=', 'Content-Length': '0'} 2022-09-12T20:14:07.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005003a5d9feedd7af-00631f930f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:07.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:14:07.991 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:07.991 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:07.992 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:07.993 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:07.993 DEBUG:boto:Method: GET 2022-09-12T20:14:07.994 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:07.995 DEBUG:boto:Data: 2022-09-12T20:14:07.995 DEBUG:boto:Headers: {} 2022-09-12T20:14:07.996 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:07.997 DEBUG:boto:Port: 8000 2022-09-12T20:14:07.998 DEBUG:boto:Params: {} 2022-09-12T20:14:07.998 DEBUG:boto:Token: None 2022-09-12T20:14:07.999 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:07 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:08.000 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UrexPLkQlw5n7cl/6TkBZkqNfL8= 2022-09-12T20:14:08.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UrexPLkQlw5n7cl/6TkBZkqNfL8=', 'Content-Length': '0'} 2022-09-12T20:14:08.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002a3029d2d755cbc-00631f9310-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.004 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:08.005 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:08.005 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:08.006 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:08.007 DEBUG:boto:Method: GET 2022-09-12T20:14:08.007 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:08.008 DEBUG:boto:Data: 2022-09-12T20:14:08.009 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.009 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.010 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.011 DEBUG:boto:Params: {} 2022-09-12T20:14:08.012 DEBUG:boto:Token: None 2022-09-12T20:14:08.012 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:08.013 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+yHuDQQKOyfTfDrUiwGa44/CD64= 2022-09-12T20:14:08.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+yHuDQQKOyfTfDrUiwGa44/CD64=', 'Content-Length': '0'} 2022-09-12T20:14:08.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064c479c5ab1cf331-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.017 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:08.018 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:08.018 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:14:08.019 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:14:08.019 DEBUG:boto:Method: GET 2022-09-12T20:14:08.020 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:14:08.020 DEBUG:boto:Data: 2022-09-12T20:14:08.021 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.021 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.022 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.022 DEBUG:boto:Params: {} 2022-09-12T20:14:08.023 DEBUG:boto:Token: None 2022-09-12T20:14:08.023 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/ 2022-09-12T20:14:08.024 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4= 2022-09-12T20:14:08.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4=', 'Content-Length': '0'} 2022-09-12T20:14:08.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3ad06d53a86a3d4-00631f9310-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.028 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:08.029 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:08.029 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:08.030 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:14:08.030 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:14:08.031 DEBUG:boto:Method: GET 2022-09-12T20:14:08.031 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:14:08.032 DEBUG:boto:Data: 2022-09-12T20:14:08.032 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.033 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.033 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.034 DEBUG:boto:Params: {} 2022-09-12T20:14:08.035 DEBUG:boto:Token: None 2022-09-12T20:14:08.035 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/ 2022-09-12T20:14:08.036 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4= 2022-09-12T20:14:08.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4=', 'Content-Length': '0'} 2022-09-12T20:14:08.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055c60e78a4a936e8-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.041 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:08.042 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:08.042 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:08.043 DEBUG:boto:Method: HEAD 2022-09-12T20:14:08.043 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:14:08.044 DEBUG:boto:Data: 2022-09-12T20:14:08.044 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.045 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.045 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.046 DEBUG:boto:Params: {} 2022-09-12T20:14:08.047 DEBUG:boto:Token: None 2022-09-12T20:14:08.047 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj29 2022-09-12T20:14:08.048 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KCXzCibgHYffRIqfdf6wIobUiY4= 2022-09-12T20:14:08.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KCXzCibgHYffRIqfdf6wIobUiY4=', 'Content-Length': '0'} 2022-09-12T20:14:08.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004137591aac999aa6-00631f9310-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.051 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:08.051 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:08.052 DEBUG:boto:Method: HEAD 2022-09-12T20:14:08.053 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:14:08.053 DEBUG:boto:Data: 2022-09-12T20:14:08.054 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.054 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.055 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.055 DEBUG:boto:Params: {} 2022-09-12T20:14:08.056 DEBUG:boto:Token: None 2022-09-12T20:14:08.056 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj29 2022-09-12T20:14:08.057 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KCXzCibgHYffRIqfdf6wIobUiY4= 2022-09-12T20:14:08.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KCXzCibgHYffRIqfdf6wIobUiY4=', 'Content-Length': '0'} 2022-09-12T20:14:08.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4b9766a05372ecb-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:14:08.061 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:08.061 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:08.062 DEBUG:boto:Method: GET 2022-09-12T20:14:08.062 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:14:08.063 DEBUG:boto:Data: 2022-09-12T20:14:08.063 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.064 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.064 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.065 DEBUG:boto:Params: {} 2022-09-12T20:14:08.066 DEBUG:boto:Token: None 2022-09-12T20:14:08.066 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj29 2022-09-12T20:14:08.067 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5F8OaaDzEpSds2pYN5U+QEplGrQ= 2022-09-12T20:14:08.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5F8OaaDzEpSds2pYN5U+QEplGrQ=', 'Content-Length': '0'} 2022-09-12T20:14:08.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fc619dfee8379c4-00631f9310-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.110 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:08.111 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:08.111 DEBUG:boto:Method: GET 2022-09-12T20:14:08.112 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:14:08.112 DEBUG:boto:Data: 2022-09-12T20:14:08.113 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.113 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.114 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.115 DEBUG:boto:Params: {} 2022-09-12T20:14:08.115 DEBUG:boto:Token: None 2022-09-12T20:14:08.116 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj29 2022-09-12T20:14:08.116 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5F8OaaDzEpSds2pYN5U+QEplGrQ= 2022-09-12T20:14:08.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5F8OaaDzEpSds2pYN5U+QEplGrQ=', 'Content-Length': '0'} 2022-09-12T20:14:08.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e07cf4d0d57a9e4-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.120 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:08.121 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:08.121 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:14:08.122 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:14:08.123 DEBUG:boto:Method: GET 2022-09-12T20:14:08.123 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:14:08.124 DEBUG:boto:Data: 2022-09-12T20:14:08.124 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.125 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.125 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.126 DEBUG:boto:Params: {} 2022-09-12T20:14:08.127 DEBUG:boto:Token: None 2022-09-12T20:14:08.127 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/ 2022-09-12T20:14:08.128 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4= 2022-09-12T20:14:08.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4=', 'Content-Length': '0'} 2022-09-12T20:14:08.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019f99271b815073f-00631f9310-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.132 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:08.133 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:08.133 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:08.134 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:14:08.134 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:14:08.135 DEBUG:boto:Method: GET 2022-09-12T20:14:08.135 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:14:08.136 DEBUG:boto:Data: 2022-09-12T20:14:08.136 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.137 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.137 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.138 DEBUG:boto:Params: {} 2022-09-12T20:14:08.139 DEBUG:boto:Token: None 2022-09-12T20:14:08.139 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/ 2022-09-12T20:14:08.140 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4= 2022-09-12T20:14:08.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4=', 'Content-Length': '0'} 2022-09-12T20:14:08.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000342d5782c8e40daf-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.146 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:08.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-12T20:14:08.147 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:08.148 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:08.148 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:08.149 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:08.149 DEBUG:boto:Method: GET 2022-09-12T20:14:08.150 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:08.151 DEBUG:boto:Data: 2022-09-12T20:14:08.151 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.152 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.152 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.153 DEBUG:boto:Params: {} 2022-09-12T20:14:08.153 DEBUG:boto:Token: None 2022-09-12T20:14:08.154 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:08.154 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tkyIW0dhSCbsvy1U7vBnFIBQl9g= 2022-09-12T20:14:08.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tkyIW0dhSCbsvy1U7vBnFIBQl9g=', 'Content-Length': '0'} 2022-09-12T20:14:08.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8a902d71d5245df-00631f9310-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.158 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:08.159 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:08.160 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:08.160 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:08.161 DEBUG:boto:Method: GET 2022-09-12T20:14:08.161 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:08.162 DEBUG:boto:Data: 2022-09-12T20:14:08.162 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.163 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.163 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.164 DEBUG:boto:Params: {} 2022-09-12T20:14:08.165 DEBUG:boto:Token: None 2022-09-12T20:14:08.165 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:08.166 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tkyIW0dhSCbsvy1U7vBnFIBQl9g= 2022-09-12T20:14:08.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tkyIW0dhSCbsvy1U7vBnFIBQl9g=', 'Content-Length': '0'} 2022-09-12T20:14:08.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2e8d34d3b702f8c-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.169 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:08.170 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:08.170 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:08.171 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:08.171 DEBUG:boto:Method: HEAD 2022-09-12T20:14:08.172 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:08.173 DEBUG:boto:Data: 2022-09-12T20:14:08.173 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.174 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.174 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.175 DEBUG:boto:Params: {} 2022-09-12T20:14:08.175 DEBUG:boto:Token: None 2022-09-12T20:14:08.176 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:08.176 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XXEym3RrpKMnvnABnMtwQw3R4pQ= 2022-09-12T20:14:08.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XXEym3RrpKMnvnABnMtwQw3R4pQ=', 'Content-Length': '0'} 2022-09-12T20:14:08.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014bc34484d288d60-00631f9310-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.180 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:08.180 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:08.181 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:08.181 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:08.182 DEBUG:boto:Method: HEAD 2022-09-12T20:14:08.182 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:08.183 DEBUG:boto:Data: 2022-09-12T20:14:08.183 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.184 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.184 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.185 DEBUG:boto:Params: {} 2022-09-12T20:14:08.185 DEBUG:boto:Token: None 2022-09-12T20:14:08.186 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:08.187 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XXEym3RrpKMnvnABnMtwQw3R4pQ= 2022-09-12T20:14:08.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XXEym3RrpKMnvnABnMtwQw3R4pQ=', 'Content-Length': '0'} 2022-09-12T20:14:08.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002815abc92d2c2f83-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-12T20:14:08.191 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:08.191 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:08.192 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:08.192 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:08.193 DEBUG:boto:Method: GET 2022-09-12T20:14:08.193 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:08.194 DEBUG:boto:Data: 2022-09-12T20:14:08.194 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.195 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.196 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.196 DEBUG:boto:Params: {} 2022-09-12T20:14:08.197 DEBUG:boto:Token: None 2022-09-12T20:14:08.197 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:08.198 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tkyIW0dhSCbsvy1U7vBnFIBQl9g= 2022-09-12T20:14:08.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tkyIW0dhSCbsvy1U7vBnFIBQl9g=', 'Content-Length': '0'} 2022-09-12T20:14:08.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2f6e2d0f171fcb5-00631f9310-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.242 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:08.243 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:08.243 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:08.244 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:08.244 DEBUG:boto:Method: GET 2022-09-12T20:14:08.245 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:08.245 DEBUG:boto:Data: 2022-09-12T20:14:08.246 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.246 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.247 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.248 DEBUG:boto:Params: {} 2022-09-12T20:14:08.248 DEBUG:boto:Token: None 2022-09-12T20:14:08.249 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:08.249 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tkyIW0dhSCbsvy1U7vBnFIBQl9g= 2022-09-12T20:14:08.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tkyIW0dhSCbsvy1U7vBnFIBQl9g=', 'Content-Length': '0'} 2022-09-12T20:14:08.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dcfc5a47034bc77-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.253 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:08.254 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:08.254 DEBUG:boto:path=/jdpiak-75/?prefix=obj31 2022-09-12T20:14:08.255 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj31 2022-09-12T20:14:08.255 DEBUG:boto:Method: GET 2022-09-12T20:14:08.256 DEBUG:boto:Path: /jdpiak-75/?prefix=obj31 2022-09-12T20:14:08.257 DEBUG:boto:Data: 2022-09-12T20:14:08.257 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.258 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.258 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.259 DEBUG:boto:Params: {} 2022-09-12T20:14:08.259 DEBUG:boto:Token: None 2022-09-12T20:14:08.260 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/ 2022-09-12T20:14:08.261 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4= 2022-09-12T20:14:08.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4=', 'Content-Length': '0'} 2022-09-12T20:14:08.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dd4394cb8ad64b1-00631f9310-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.265 DEBUG:boto:b'jdpiak-75obj311000falseobj312022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:08.265 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:08.266 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:08.267 DEBUG:boto:path=/jdpiak-75/?prefix=obj31 2022-09-12T20:14:08.267 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj31 2022-09-12T20:14:08.268 DEBUG:boto:Method: GET 2022-09-12T20:14:08.268 DEBUG:boto:Path: /jdpiak-75/?prefix=obj31 2022-09-12T20:14:08.269 DEBUG:boto:Data: 2022-09-12T20:14:08.269 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.270 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.270 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.271 DEBUG:boto:Params: {} 2022-09-12T20:14:08.271 DEBUG:boto:Token: None 2022-09-12T20:14:08.272 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/ 2022-09-12T20:14:08.273 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4= 2022-09-12T20:14:08.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4=', 'Content-Length': '0'} 2022-09-12T20:14:08.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bd6964ffaf419fb-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.278 DEBUG:boto:b'jdpiak-75obj311000falseobj312022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:08.279 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:08.279 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:08.280 DEBUG:boto:Method: HEAD 2022-09-12T20:14:08.280 DEBUG:boto:Path: /jdpiak-75/obj31 2022-09-12T20:14:08.281 DEBUG:boto:Data: 2022-09-12T20:14:08.281 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.282 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.283 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.283 DEBUG:boto:Params: {} 2022-09-12T20:14:08.284 DEBUG:boto:Token: None 2022-09-12T20:14:08.284 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj31 2022-09-12T20:14:08.285 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:afRziNd727xOPaSytDivouiLWGY= 2022-09-12T20:14:08.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:afRziNd727xOPaSytDivouiLWGY=', 'Content-Length': '0'} 2022-09-12T20:14:08.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043e74fdd7ebcbac6-00631f9310-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.288 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:08.289 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:08.289 DEBUG:boto:Method: HEAD 2022-09-12T20:14:08.290 DEBUG:boto:Path: /jdpiak-75/obj31 2022-09-12T20:14:08.291 DEBUG:boto:Data: 2022-09-12T20:14:08.291 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.292 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.292 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.293 DEBUG:boto:Params: {} 2022-09-12T20:14:08.293 DEBUG:boto:Token: None 2022-09-12T20:14:08.294 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj31 2022-09-12T20:14:08.294 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:afRziNd727xOPaSytDivouiLWGY= 2022-09-12T20:14:08.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:afRziNd727xOPaSytDivouiLWGY=', 'Content-Length': '0'} 2022-09-12T20:14:08.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9b42af7201481d7-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-12T20:14:08.298 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:08.299 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:08.299 DEBUG:boto:Method: GET 2022-09-12T20:14:08.300 DEBUG:boto:Path: /jdpiak-75/obj31 2022-09-12T20:14:08.300 DEBUG:boto:Data: 2022-09-12T20:14:08.301 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.302 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.302 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.303 DEBUG:boto:Params: {} 2022-09-12T20:14:08.303 DEBUG:boto:Token: None 2022-09-12T20:14:08.304 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj31 2022-09-12T20:14:08.305 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uae6xyypONnqG7Kvszsf+JkAMfI= 2022-09-12T20:14:08.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uae6xyypONnqG7Kvszsf+JkAMfI=', 'Content-Length': '0'} 2022-09-12T20:14:08.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b38eed220ab2e55-00631f9310-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.308 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:08.309 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:08.309 DEBUG:boto:Method: GET 2022-09-12T20:14:08.310 DEBUG:boto:Path: /jdpiak-75/obj31 2022-09-12T20:14:08.310 DEBUG:boto:Data: 2022-09-12T20:14:08.311 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.311 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.312 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.312 DEBUG:boto:Params: {} 2022-09-12T20:14:08.313 DEBUG:boto:Token: None 2022-09-12T20:14:08.313 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj31 2022-09-12T20:14:08.314 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uae6xyypONnqG7Kvszsf+JkAMfI= 2022-09-12T20:14:08.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uae6xyypONnqG7Kvszsf+JkAMfI=', 'Content-Length': '0'} 2022-09-12T20:14:08.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da09f5b89fa2229e-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.318 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:08.318 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:08.319 DEBUG:boto:path=/jdpiak-75/?prefix=obj31 2022-09-12T20:14:08.319 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj31 2022-09-12T20:14:08.320 DEBUG:boto:Method: GET 2022-09-12T20:14:08.320 DEBUG:boto:Path: /jdpiak-75/?prefix=obj31 2022-09-12T20:14:08.321 DEBUG:boto:Data: 2022-09-12T20:14:08.321 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.322 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.323 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.323 DEBUG:boto:Params: {} 2022-09-12T20:14:08.324 DEBUG:boto:Token: None 2022-09-12T20:14:08.325 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/ 2022-09-12T20:14:08.325 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4= 2022-09-12T20:14:08.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4=', 'Content-Length': '0'} 2022-09-12T20:14:08.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014c982a7a01bd1e3-00631f9310-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.329 DEBUG:boto:b'jdpiak-75obj311000falseobj312022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:08.330 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:08.331 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:08.331 DEBUG:boto:path=/jdpiak-75/?prefix=obj31 2022-09-12T20:14:08.332 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj31 2022-09-12T20:14:08.332 DEBUG:boto:Method: GET 2022-09-12T20:14:08.333 DEBUG:boto:Path: /jdpiak-75/?prefix=obj31 2022-09-12T20:14:08.333 DEBUG:boto:Data: 2022-09-12T20:14:08.334 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.335 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.335 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.336 DEBUG:boto:Params: {} 2022-09-12T20:14:08.336 DEBUG:boto:Token: None 2022-09-12T20:14:08.337 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/ 2022-09-12T20:14:08.338 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4= 2022-09-12T20:14:08.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4=', 'Content-Length': '0'} 2022-09-12T20:14:08.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1d70d9454206291-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.343 DEBUG:boto:b'jdpiak-75obj311000falseobj312022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:08.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:14:08.344 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:08.345 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:08.346 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:08.346 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:08.347 DEBUG:boto:Method: GET 2022-09-12T20:14:08.347 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:08.348 DEBUG:boto:Data: 2022-09-12T20:14:08.348 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.349 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.349 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.350 DEBUG:boto:Params: {} 2022-09-12T20:14:08.350 DEBUG:boto:Token: None 2022-09-12T20:14:08.351 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:08.352 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H3mh1E0P3WfNmvnLY8YW2fYhR2M= 2022-09-12T20:14:08.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H3mh1E0P3WfNmvnLY8YW2fYhR2M=', 'Content-Length': '0'} 2022-09-12T20:14:08.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000876c316f02946d89-00631f9310-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.355 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:08.356 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:08.356 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:08.357 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:08.357 DEBUG:boto:Method: GET 2022-09-12T20:14:08.358 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:08.358 DEBUG:boto:Data: 2022-09-12T20:14:08.359 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.360 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.360 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.361 DEBUG:boto:Params: {} 2022-09-12T20:14:08.361 DEBUG:boto:Token: None 2022-09-12T20:14:08.362 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:08.362 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H3mh1E0P3WfNmvnLY8YW2fYhR2M= 2022-09-12T20:14:08.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H3mh1E0P3WfNmvnLY8YW2fYhR2M=', 'Content-Length': '0'} 2022-09-12T20:14:08.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d702bbd38ceaf2ab-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.366 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:08.366 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:08.367 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:08.367 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:08.368 DEBUG:boto:Method: HEAD 2022-09-12T20:14:08.368 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:08.369 DEBUG:boto:Data: 2022-09-12T20:14:08.369 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.370 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.371 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.371 DEBUG:boto:Params: {} 2022-09-12T20:14:08.372 DEBUG:boto:Token: None 2022-09-12T20:14:08.372 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:08.373 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Feu4Ldhw9pxorZQkfAlY6FxbNeY= 2022-09-12T20:14:08.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Feu4Ldhw9pxorZQkfAlY6FxbNeY=', 'Content-Length': '0'} 2022-09-12T20:14:08.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d03908225fa9678b-00631f9310-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.376 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:08.377 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:08.378 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:08.379 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:08.379 DEBUG:boto:Method: HEAD 2022-09-12T20:14:08.380 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:08.380 DEBUG:boto:Data: 2022-09-12T20:14:08.381 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.381 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.382 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.383 DEBUG:boto:Params: {} 2022-09-12T20:14:08.384 DEBUG:boto:Token: None 2022-09-12T20:14:08.385 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:08.386 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Feu4Ldhw9pxorZQkfAlY6FxbNeY= 2022-09-12T20:14:08.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Feu4Ldhw9pxorZQkfAlY6FxbNeY=', 'Content-Length': '0'} 2022-09-12T20:14:08.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc842bfc40ba0d1c-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:14:08.390 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:08.391 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:08.392 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:08.393 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:08.394 DEBUG:boto:Method: GET 2022-09-12T20:14:08.395 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:08.396 DEBUG:boto:Data: 2022-09-12T20:14:08.397 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.398 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.399 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.400 DEBUG:boto:Params: {} 2022-09-12T20:14:08.401 DEBUG:boto:Token: None 2022-09-12T20:14:08.402 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:08.403 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H3mh1E0P3WfNmvnLY8YW2fYhR2M= 2022-09-12T20:14:08.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H3mh1E0P3WfNmvnLY8YW2fYhR2M=', 'Content-Length': '0'} 2022-09-12T20:14:08.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008eb3f6f5d767a307-00631f9310-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.408 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:08.409 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:08.410 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:08.411 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:08.412 DEBUG:boto:Method: GET 2022-09-12T20:14:08.412 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:08.413 DEBUG:boto:Data: 2022-09-12T20:14:08.414 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.415 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.416 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.417 DEBUG:boto:Params: {} 2022-09-12T20:14:08.418 DEBUG:boto:Token: None 2022-09-12T20:14:08.419 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:08.420 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H3mh1E0P3WfNmvnLY8YW2fYhR2M= 2022-09-12T20:14:08.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H3mh1E0P3WfNmvnLY8YW2fYhR2M=', 'Content-Length': '0'} 2022-09-12T20:14:08.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fe424ce56c9a7bc-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.425 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:08.425 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:08.426 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:14:08.426 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:14:08.427 DEBUG:boto:Method: GET 2022-09-12T20:14:08.428 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:14:08.429 DEBUG:boto:Data: 2022-09-12T20:14:08.430 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.431 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.432 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.433 DEBUG:boto:Params: {} 2022-09-12T20:14:08.434 DEBUG:boto:Token: None 2022-09-12T20:14:08.435 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/ 2022-09-12T20:14:08.436 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4= 2022-09-12T20:14:08.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4=', 'Content-Length': '0'} 2022-09-12T20:14:08.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c41765cda7deef87-00631f9310-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.482 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:08.483 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:08.483 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:08.484 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:14:08.485 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:14:08.485 DEBUG:boto:Method: GET 2022-09-12T20:14:08.486 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:14:08.486 DEBUG:boto:Data: 2022-09-12T20:14:08.487 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.487 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.488 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.488 DEBUG:boto:Params: {} 2022-09-12T20:14:08.489 DEBUG:boto:Token: None 2022-09-12T20:14:08.489 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/ 2022-09-12T20:14:08.490 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4= 2022-09-12T20:14:08.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4=', 'Content-Length': '0'} 2022-09-12T20:14:08.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000064ec1b09fe196a3-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.495 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:08.496 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:08.497 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:08.497 DEBUG:boto:Method: HEAD 2022-09-12T20:14:08.498 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:14:08.498 DEBUG:boto:Data: 2022-09-12T20:14:08.499 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.499 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.500 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.500 DEBUG:boto:Params: {} 2022-09-12T20:14:08.501 DEBUG:boto:Token: None 2022-09-12T20:14:08.501 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj33 2022-09-12T20:14:08.502 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8kwi5FjyS2NbixbVw2wuOZaibRI= 2022-09-12T20:14:08.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8kwi5FjyS2NbixbVw2wuOZaibRI=', 'Content-Length': '0'} 2022-09-12T20:14:08.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000002e791f556a21cb-00631f9310-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.505 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:08.506 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:08.506 DEBUG:boto:Method: HEAD 2022-09-12T20:14:08.507 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:14:08.507 DEBUG:boto:Data: 2022-09-12T20:14:08.508 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.508 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.509 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.509 DEBUG:boto:Params: {} 2022-09-12T20:14:08.510 DEBUG:boto:Token: None 2022-09-12T20:14:08.510 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj33 2022-09-12T20:14:08.511 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8kwi5FjyS2NbixbVw2wuOZaibRI= 2022-09-12T20:14:08.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8kwi5FjyS2NbixbVw2wuOZaibRI=', 'Content-Length': '0'} 2022-09-12T20:14:08.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076f5337ddc3c1eaf-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:14:08.515 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:08.515 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:08.516 DEBUG:boto:Method: GET 2022-09-12T20:14:08.516 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:14:08.517 DEBUG:boto:Data: 2022-09-12T20:14:08.517 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.518 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.518 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.519 DEBUG:boto:Params: {} 2022-09-12T20:14:08.519 DEBUG:boto:Token: None 2022-09-12T20:14:08.520 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj33 2022-09-12T20:14:08.521 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:13NUrfvFBotAeOxsw4nhelXB3kg= 2022-09-12T20:14:08.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:13NUrfvFBotAeOxsw4nhelXB3kg=', 'Content-Length': '0'} 2022-09-12T20:14:08.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041f9ea11f505e7d3-00631f9310-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.524 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:08.524 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:08.525 DEBUG:boto:Method: GET 2022-09-12T20:14:08.525 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:14:08.526 DEBUG:boto:Data: 2022-09-12T20:14:08.526 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.527 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.527 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.528 DEBUG:boto:Params: {} 2022-09-12T20:14:08.529 DEBUG:boto:Token: None 2022-09-12T20:14:08.529 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj33 2022-09-12T20:14:08.530 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:13NUrfvFBotAeOxsw4nhelXB3kg= 2022-09-12T20:14:08.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:13NUrfvFBotAeOxsw4nhelXB3kg=', 'Content-Length': '0'} 2022-09-12T20:14:08.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086536d88052c96a3-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.574 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:08.575 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:08.575 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:14:08.576 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:14:08.576 DEBUG:boto:Method: GET 2022-09-12T20:14:08.577 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:14:08.577 DEBUG:boto:Data: 2022-09-12T20:14:08.578 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.578 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.579 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.579 DEBUG:boto:Params: {} 2022-09-12T20:14:08.580 DEBUG:boto:Token: None 2022-09-12T20:14:08.580 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/ 2022-09-12T20:14:08.581 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4= 2022-09-12T20:14:08.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4=', 'Content-Length': '0'} 2022-09-12T20:14:08.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e96add9da84f6996-00631f9310-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.585 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:08.585 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:08.586 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:08.587 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:14:08.587 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:14:08.588 DEBUG:boto:Method: GET 2022-09-12T20:14:08.588 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:14:08.589 DEBUG:boto:Data: 2022-09-12T20:14:08.589 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.590 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.590 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.591 DEBUG:boto:Params: {} 2022-09-12T20:14:08.591 DEBUG:boto:Token: None 2022-09-12T20:14:08.592 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/ 2022-09-12T20:14:08.592 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4= 2022-09-12T20:14:08.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4=', 'Content-Length': '0'} 2022-09-12T20:14:08.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c66e3039bea7301-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.598 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:08.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:14:08.599 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:08.600 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:08.601 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:08.601 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:08.602 DEBUG:boto:Method: GET 2022-09-12T20:14:08.602 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:08.603 DEBUG:boto:Data: 2022-09-12T20:14:08.603 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.604 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.604 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.605 DEBUG:boto:Params: {} 2022-09-12T20:14:08.605 DEBUG:boto:Token: None 2022-09-12T20:14:08.606 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:08.606 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/03KaCjuu0Gt4yBP3heSLRbWrnY= 2022-09-12T20:14:08.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/03KaCjuu0Gt4yBP3heSLRbWrnY=', 'Content-Length': '0'} 2022-09-12T20:14:08.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e67575fab957086f-00631f9310-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.610 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:08.610 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:08.611 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:08.612 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:08.612 DEBUG:boto:Method: GET 2022-09-12T20:14:08.613 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:08.613 DEBUG:boto:Data: 2022-09-12T20:14:08.614 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.614 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.615 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.615 DEBUG:boto:Params: {} 2022-09-12T20:14:08.616 DEBUG:boto:Token: None 2022-09-12T20:14:08.616 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:08.617 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/03KaCjuu0Gt4yBP3heSLRbWrnY= 2022-09-12T20:14:08.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/03KaCjuu0Gt4yBP3heSLRbWrnY=', 'Content-Length': '0'} 2022-09-12T20:14:08.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052c7ea86bace3b88-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.620 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:08.621 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:08.621 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:08.622 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:08.622 DEBUG:boto:Method: HEAD 2022-09-12T20:14:08.623 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:08.623 DEBUG:boto:Data: 2022-09-12T20:14:08.624 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.624 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.625 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.625 DEBUG:boto:Params: {} 2022-09-12T20:14:08.626 DEBUG:boto:Token: None 2022-09-12T20:14:08.627 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:08.627 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YSpVf1phvRoogYGPMwsWoC8RnHg= 2022-09-12T20:14:08.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YSpVf1phvRoogYGPMwsWoC8RnHg=', 'Content-Length': '0'} 2022-09-12T20:14:08.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f5afe3ba06799c0-00631f9310-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.630 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:08.631 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:08.631 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:08.632 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:08.632 DEBUG:boto:Method: HEAD 2022-09-12T20:14:08.633 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:08.633 DEBUG:boto:Data: 2022-09-12T20:14:08.634 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.634 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.635 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.635 DEBUG:boto:Params: {} 2022-09-12T20:14:08.636 DEBUG:boto:Token: None 2022-09-12T20:14:08.636 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:08.637 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YSpVf1phvRoogYGPMwsWoC8RnHg= 2022-09-12T20:14:08.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YSpVf1phvRoogYGPMwsWoC8RnHg=', 'Content-Length': '0'} 2022-09-12T20:14:08.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dede48e02b8f677d-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:14:08.642 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:08.642 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:08.643 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:08.644 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:08.644 DEBUG:boto:Method: GET 2022-09-12T20:14:08.645 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:08.645 DEBUG:boto:Data: 2022-09-12T20:14:08.646 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.646 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.647 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.647 DEBUG:boto:Params: {} 2022-09-12T20:14:08.648 DEBUG:boto:Token: None 2022-09-12T20:14:08.648 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:08.649 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/03KaCjuu0Gt4yBP3heSLRbWrnY= 2022-09-12T20:14:08.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/03KaCjuu0Gt4yBP3heSLRbWrnY=', 'Content-Length': '0'} 2022-09-12T20:14:08.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e33d7e8827925ce-00631f9310-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.652 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:08.653 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:08.654 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:08.654 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:08.655 DEBUG:boto:Method: GET 2022-09-12T20:14:08.655 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:08.656 DEBUG:boto:Data: 2022-09-12T20:14:08.656 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.657 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.657 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.658 DEBUG:boto:Params: {} 2022-09-12T20:14:08.658 DEBUG:boto:Token: None 2022-09-12T20:14:08.659 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:08.659 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/03KaCjuu0Gt4yBP3heSLRbWrnY= 2022-09-12T20:14:08.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/03KaCjuu0Gt4yBP3heSLRbWrnY=', 'Content-Length': '0'} 2022-09-12T20:14:08.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065fdb1f64d6803f7-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.663 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:08.663 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:08.664 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:14:08.664 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:14:08.665 DEBUG:boto:Method: GET 2022-09-12T20:14:08.665 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:14:08.666 DEBUG:boto:Data: 2022-09-12T20:14:08.666 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.667 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.667 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.668 DEBUG:boto:Params: {} 2022-09-12T20:14:08.668 DEBUG:boto:Token: None 2022-09-12T20:14:08.669 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/ 2022-09-12T20:14:08.670 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4= 2022-09-12T20:14:08.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4=', 'Content-Length': '0'} 2022-09-12T20:14:08.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2418b9b71666604-00631f9310-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.673 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:08.674 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:08.675 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:08.675 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:14:08.676 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:14:08.676 DEBUG:boto:Method: GET 2022-09-12T20:14:08.677 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:14:08.677 DEBUG:boto:Data: 2022-09-12T20:14:08.678 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.678 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.679 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.679 DEBUG:boto:Params: {} 2022-09-12T20:14:08.680 DEBUG:boto:Token: None 2022-09-12T20:14:08.681 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/ 2022-09-12T20:14:08.681 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4= 2022-09-12T20:14:08.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4=', 'Content-Length': '0'} 2022-09-12T20:14:08.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddd2a0946711dc72-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.687 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:08.688 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:08.689 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:08.689 DEBUG:boto:Method: HEAD 2022-09-12T20:14:08.690 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:14:08.690 DEBUG:boto:Data: 2022-09-12T20:14:08.691 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.691 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.692 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.692 DEBUG:boto:Params: {} 2022-09-12T20:14:08.693 DEBUG:boto:Token: None 2022-09-12T20:14:08.693 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj34 2022-09-12T20:14:08.694 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3ovLVVvtMzVakbTobBIIG8n7PXI= 2022-09-12T20:14:08.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3ovLVVvtMzVakbTobBIIG8n7PXI=', 'Content-Length': '0'} 2022-09-12T20:14:08.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000750468c594e519dc-00631f9310-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.697 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:08.698 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:08.698 DEBUG:boto:Method: HEAD 2022-09-12T20:14:08.699 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:14:08.699 DEBUG:boto:Data: 2022-09-12T20:14:08.700 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.700 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.701 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.701 DEBUG:boto:Params: {} 2022-09-12T20:14:08.702 DEBUG:boto:Token: None 2022-09-12T20:14:08.702 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj34 2022-09-12T20:14:08.703 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3ovLVVvtMzVakbTobBIIG8n7PXI= 2022-09-12T20:14:08.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3ovLVVvtMzVakbTobBIIG8n7PXI=', 'Content-Length': '0'} 2022-09-12T20:14:08.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026309ec5eb4de1b4-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:14:08.707 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:08.707 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:08.708 DEBUG:boto:Method: GET 2022-09-12T20:14:08.708 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:14:08.709 DEBUG:boto:Data: 2022-09-12T20:14:08.710 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.710 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.711 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.711 DEBUG:boto:Params: {} 2022-09-12T20:14:08.712 DEBUG:boto:Token: None 2022-09-12T20:14:08.712 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj34 2022-09-12T20:14:08.713 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MqIxnRWip+DzdrBx9E8Jyu6FhoE= 2022-09-12T20:14:08.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MqIxnRWip+DzdrBx9E8Jyu6FhoE=', 'Content-Length': '0'} 2022-09-12T20:14:08.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006fe5dc02f80a252-00631f9310-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.717 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:08.718 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:08.719 DEBUG:boto:Method: GET 2022-09-12T20:14:08.720 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:14:08.720 DEBUG:boto:Data: 2022-09-12T20:14:08.721 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.721 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.722 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.723 DEBUG:boto:Params: {} 2022-09-12T20:14:08.723 DEBUG:boto:Token: None 2022-09-12T20:14:08.724 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj34 2022-09-12T20:14:08.724 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MqIxnRWip+DzdrBx9E8Jyu6FhoE= 2022-09-12T20:14:08.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MqIxnRWip+DzdrBx9E8Jyu6FhoE=', 'Content-Length': '0'} 2022-09-12T20:14:08.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb06d5eeaf76432e-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.770 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:08.771 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:08.771 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:14:08.772 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:14:08.772 DEBUG:boto:Method: GET 2022-09-12T20:14:08.773 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:14:08.773 DEBUG:boto:Data: 2022-09-12T20:14:08.774 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.775 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.775 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.776 DEBUG:boto:Params: {} 2022-09-12T20:14:08.776 DEBUG:boto:Token: None 2022-09-12T20:14:08.777 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/ 2022-09-12T20:14:08.777 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4= 2022-09-12T20:14:08.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4=', 'Content-Length': '0'} 2022-09-12T20:14:08.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8afc067b2454ea2-00631f9310-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.782 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:08.783 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:08.783 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:08.784 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:14:08.784 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:14:08.785 DEBUG:boto:Method: GET 2022-09-12T20:14:08.785 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:14:08.786 DEBUG:boto:Data: 2022-09-12T20:14:08.787 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.787 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.788 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.788 DEBUG:boto:Params: {} 2022-09-12T20:14:08.789 DEBUG:boto:Token: None 2022-09-12T20:14:08.790 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/ 2022-09-12T20:14:08.790 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4= 2022-09-12T20:14:08.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4=', 'Content-Length': '0'} 2022-09-12T20:14:08.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ef40ab8b409e101-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.796 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:08.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:14:08.798 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:08.798 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:08.799 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:08.800 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:08.800 DEBUG:boto:Method: GET 2022-09-12T20:14:08.801 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:08.801 DEBUG:boto:Data: 2022-09-12T20:14:08.802 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.802 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.803 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.803 DEBUG:boto:Params: {} 2022-09-12T20:14:08.804 DEBUG:boto:Token: None 2022-09-12T20:14:08.805 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:08.805 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DUhHZ/s7kKWnmaCMM4LQt9GL0OA= 2022-09-12T20:14:08.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DUhHZ/s7kKWnmaCMM4LQt9GL0OA=', 'Content-Length': '0'} 2022-09-12T20:14:08.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cb5a8d241e8f0a2-00631f9310-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.809 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:08.810 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:08.810 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:08.811 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:08.811 DEBUG:boto:Method: GET 2022-09-12T20:14:08.812 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:08.813 DEBUG:boto:Data: 2022-09-12T20:14:08.813 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.814 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.814 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.815 DEBUG:boto:Params: {} 2022-09-12T20:14:08.815 DEBUG:boto:Token: None 2022-09-12T20:14:08.816 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:08.816 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DUhHZ/s7kKWnmaCMM4LQt9GL0OA= 2022-09-12T20:14:08.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DUhHZ/s7kKWnmaCMM4LQt9GL0OA=', 'Content-Length': '0'} 2022-09-12T20:14:08.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002279a495b662ba5f-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.820 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:08.821 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:08.821 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:08.822 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:08.822 DEBUG:boto:Method: HEAD 2022-09-12T20:14:08.823 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:08.823 DEBUG:boto:Data: 2022-09-12T20:14:08.824 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.824 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.825 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.826 DEBUG:boto:Params: {} 2022-09-12T20:14:08.826 DEBUG:boto:Token: None 2022-09-12T20:14:08.827 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:08.827 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KPNsKKrERz6vtWOd2fiZMVX6oxI= 2022-09-12T20:14:08.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KPNsKKrERz6vtWOd2fiZMVX6oxI=', 'Content-Length': '0'} 2022-09-12T20:14:08.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002113929d93af9f40-00631f9310-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.831 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:08.831 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:08.832 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:08.832 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:08.833 DEBUG:boto:Method: HEAD 2022-09-12T20:14:08.834 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:08.834 DEBUG:boto:Data: 2022-09-12T20:14:08.835 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.835 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.836 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.836 DEBUG:boto:Params: {} 2022-09-12T20:14:08.837 DEBUG:boto:Token: None 2022-09-12T20:14:08.837 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:08.838 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KPNsKKrERz6vtWOd2fiZMVX6oxI= 2022-09-12T20:14:08.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KPNsKKrERz6vtWOd2fiZMVX6oxI=', 'Content-Length': '0'} 2022-09-12T20:14:08.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e478c52dd7e704b-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:14:08.842 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:08.843 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:08.844 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:08.844 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:08.845 DEBUG:boto:Method: GET 2022-09-12T20:14:08.845 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:08.846 DEBUG:boto:Data: 2022-09-12T20:14:08.846 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.847 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.847 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.848 DEBUG:boto:Params: {} 2022-09-12T20:14:08.849 DEBUG:boto:Token: None 2022-09-12T20:14:08.849 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:08.850 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DUhHZ/s7kKWnmaCMM4LQt9GL0OA= 2022-09-12T20:14:08.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DUhHZ/s7kKWnmaCMM4LQt9GL0OA=', 'Content-Length': '0'} 2022-09-12T20:14:08.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033cf9bdb36c51864-00631f9310-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.853 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:08.854 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:08.854 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:08.855 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:08.856 DEBUG:boto:Method: GET 2022-09-12T20:14:08.856 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:08.857 DEBUG:boto:Data: 2022-09-12T20:14:08.857 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.858 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.858 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.859 DEBUG:boto:Params: {} 2022-09-12T20:14:08.859 DEBUG:boto:Token: None 2022-09-12T20:14:08.860 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:08.860 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DUhHZ/s7kKWnmaCMM4LQt9GL0OA= 2022-09-12T20:14:08.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DUhHZ/s7kKWnmaCMM4LQt9GL0OA=', 'Content-Length': '0'} 2022-09-12T20:14:08.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be982510f2f53a9a-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.864 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:08.865 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:08.865 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:14:08.866 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:14:08.866 DEBUG:boto:Method: GET 2022-09-12T20:14:08.867 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:14:08.867 DEBUG:boto:Data: 2022-09-12T20:14:08.868 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.869 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.869 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.870 DEBUG:boto:Params: {} 2022-09-12T20:14:08.870 DEBUG:boto:Token: None 2022-09-12T20:14:08.871 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/ 2022-09-12T20:14:08.871 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4= 2022-09-12T20:14:08.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4=', 'Content-Length': '0'} 2022-09-12T20:14:08.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069b0132926b53980-00631f9310-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.876 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:08.878 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:08.879 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:08.880 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:14:08.880 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:14:08.881 DEBUG:boto:Method: GET 2022-09-12T20:14:08.881 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:14:08.882 DEBUG:boto:Data: 2022-09-12T20:14:08.882 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.883 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.883 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.884 DEBUG:boto:Params: {} 2022-09-12T20:14:08.884 DEBUG:boto:Token: None 2022-09-12T20:14:08.885 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/ 2022-09-12T20:14:08.886 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4= 2022-09-12T20:14:08.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4=', 'Content-Length': '0'} 2022-09-12T20:14:08.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032a8948e68c093f3-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.892 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:08.893 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:08.893 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:08.894 DEBUG:boto:Method: HEAD 2022-09-12T20:14:08.894 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:14:08.895 DEBUG:boto:Data: 2022-09-12T20:14:08.895 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.896 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.896 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.897 DEBUG:boto:Params: {} 2022-09-12T20:14:08.897 DEBUG:boto:Token: None 2022-09-12T20:14:08.898 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj38 2022-09-12T20:14:08.898 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q6S7Y0FUg//LUKqTiDQvlrwApGw= 2022-09-12T20:14:08.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q6S7Y0FUg//LUKqTiDQvlrwApGw=', 'Content-Length': '0'} 2022-09-12T20:14:08.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5d341b989b5a367-00631f9310-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.902 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:08.902 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:08.903 DEBUG:boto:Method: HEAD 2022-09-12T20:14:08.903 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:14:08.904 DEBUG:boto:Data: 2022-09-12T20:14:08.904 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.905 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.905 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.906 DEBUG:boto:Params: {} 2022-09-12T20:14:08.906 DEBUG:boto:Token: None 2022-09-12T20:14:08.907 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj38 2022-09-12T20:14:08.907 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q6S7Y0FUg//LUKqTiDQvlrwApGw= 2022-09-12T20:14:08.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q6S7Y0FUg//LUKqTiDQvlrwApGw=', 'Content-Length': '0'} 2022-09-12T20:14:08.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008808f952ea2fe212-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:14:08.911 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:08.912 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:08.913 DEBUG:boto:Method: GET 2022-09-12T20:14:08.913 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:14:08.914 DEBUG:boto:Data: 2022-09-12T20:14:08.914 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.915 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.915 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.916 DEBUG:boto:Params: {} 2022-09-12T20:14:08.916 DEBUG:boto:Token: None 2022-09-12T20:14:08.917 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj38 2022-09-12T20:14:08.918 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IhthVAQUQK1bns0CpmasvxnroNc= 2022-09-12T20:14:08.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IhthVAQUQK1bns0CpmasvxnroNc=', 'Content-Length': '0'} 2022-09-12T20:14:08.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000febe517986b8d2b4-00631f9310-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.962 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:08.963 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:08.964 DEBUG:boto:Method: GET 2022-09-12T20:14:08.964 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:14:08.965 DEBUG:boto:Data: 2022-09-12T20:14:08.965 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.966 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.967 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.967 DEBUG:boto:Params: {} 2022-09-12T20:14:08.968 DEBUG:boto:Token: None 2022-09-12T20:14:08.969 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/obj38 2022-09-12T20:14:08.969 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IhthVAQUQK1bns0CpmasvxnroNc= 2022-09-12T20:14:08.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IhthVAQUQK1bns0CpmasvxnroNc=', 'Content-Length': '0'} 2022-09-12T20:14:08.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000707d555b26e9f8ee-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.973 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:08.974 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:08.974 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:14:08.975 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:14:08.975 DEBUG:boto:Method: GET 2022-09-12T20:14:08.976 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:14:08.976 DEBUG:boto:Data: 2022-09-12T20:14:08.977 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.977 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:08.978 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.978 DEBUG:boto:Params: {} 2022-09-12T20:14:08.979 DEBUG:boto:Token: None 2022-09-12T20:14:08.980 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/ 2022-09-12T20:14:08.980 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4= 2022-09-12T20:14:08.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4=', 'Content-Length': '0'} 2022-09-12T20:14:08.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a432d625fa4a0e99-00631f9310-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:08.984 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:08.985 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:08.986 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:08.986 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:14:08.987 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:14:08.988 DEBUG:boto:Method: GET 2022-09-12T20:14:08.988 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:14:08.989 DEBUG:boto:Data: 2022-09-12T20:14:08.989 DEBUG:boto:Headers: {} 2022-09-12T20:14:08.990 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:08.991 DEBUG:boto:Port: 8000 2022-09-12T20:14:08.991 DEBUG:boto:Params: {} 2022-09-12T20:14:08.992 DEBUG:boto:Token: None 2022-09-12T20:14:08.992 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:08 GMT /jdpiak-75/ 2022-09-12T20:14:08.993 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4= 2022-09-12T20:14:08.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YnW+bk5jsR30BNy+POsCTVY8uV4=', 'Content-Length': '0'} 2022-09-12T20:14:09.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c7d4d7c5ae1878c-00631f9310-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.002 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:09.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:14:09.003 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:09.004 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:09.004 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:09.005 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:09.005 DEBUG:boto:Method: GET 2022-09-12T20:14:09.006 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:09.006 DEBUG:boto:Data: 2022-09-12T20:14:09.007 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.007 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.008 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.008 DEBUG:boto:Params: {} 2022-09-12T20:14:09.009 DEBUG:boto:Token: None 2022-09-12T20:14:09.010 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:09.010 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7vpIxjsL2ZtJZuSRMBbDoZlJJ30= 2022-09-12T20:14:09.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7vpIxjsL2ZtJZuSRMBbDoZlJJ30=', 'Content-Length': '0'} 2022-09-12T20:14:09.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008302f80c63cc2964-00631f9311-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.014 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:09.014 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:09.015 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:09.015 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:09.016 DEBUG:boto:Method: GET 2022-09-12T20:14:09.017 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:09.017 DEBUG:boto:Data: 2022-09-12T20:14:09.018 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.018 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.019 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.019 DEBUG:boto:Params: {} 2022-09-12T20:14:09.020 DEBUG:boto:Token: None 2022-09-12T20:14:09.020 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:09.021 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7vpIxjsL2ZtJZuSRMBbDoZlJJ30= 2022-09-12T20:14:09.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7vpIxjsL2ZtJZuSRMBbDoZlJJ30=', 'Content-Length': '0'} 2022-09-12T20:14:09.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006550b412157af35e-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.025 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:09.025 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:09.026 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:09.026 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:09.027 DEBUG:boto:Method: HEAD 2022-09-12T20:14:09.027 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:09.028 DEBUG:boto:Data: 2022-09-12T20:14:09.028 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.029 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.030 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.030 DEBUG:boto:Params: {} 2022-09-12T20:14:09.031 DEBUG:boto:Token: None 2022-09-12T20:14:09.031 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:09.032 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Up7TD4Mq3FsPIocLK+22qaE/zc= 2022-09-12T20:14:09.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Up7TD4Mq3FsPIocLK+22qaE/zc=', 'Content-Length': '0'} 2022-09-12T20:14:09.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9b5465331667e70-00631f9311-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.035 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:09.036 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:09.037 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:09.037 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:09.037 DEBUG:boto:Method: HEAD 2022-09-12T20:14:09.038 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:09.039 DEBUG:boto:Data: 2022-09-12T20:14:09.039 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.040 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.041 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.041 DEBUG:boto:Params: {} 2022-09-12T20:14:09.042 DEBUG:boto:Token: None 2022-09-12T20:14:09.043 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:09.043 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2Up7TD4Mq3FsPIocLK+22qaE/zc= 2022-09-12T20:14:09.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2Up7TD4Mq3FsPIocLK+22qaE/zc=', 'Content-Length': '0'} 2022-09-12T20:14:09.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007df908c2d97b631-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:14:09.047 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:09.048 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:09.048 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:09.049 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:09.049 DEBUG:boto:Method: GET 2022-09-12T20:14:09.050 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:09.051 DEBUG:boto:Data: 2022-09-12T20:14:09.051 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.052 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.052 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.053 DEBUG:boto:Params: {} 2022-09-12T20:14:09.053 DEBUG:boto:Token: None 2022-09-12T20:14:09.054 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:09.055 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7vpIxjsL2ZtJZuSRMBbDoZlJJ30= 2022-09-12T20:14:09.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7vpIxjsL2ZtJZuSRMBbDoZlJJ30=', 'Content-Length': '0'} 2022-09-12T20:14:09.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031d34c708732606c-00631f9311-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.058 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:09.058 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:09.059 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:09.060 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:09.060 DEBUG:boto:Method: GET 2022-09-12T20:14:09.061 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:09.061 DEBUG:boto:Data: 2022-09-12T20:14:09.062 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.062 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.063 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.063 DEBUG:boto:Params: {} 2022-09-12T20:14:09.064 DEBUG:boto:Token: None 2022-09-12T20:14:09.065 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:09.065 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7vpIxjsL2ZtJZuSRMBbDoZlJJ30= 2022-09-12T20:14:09.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7vpIxjsL2ZtJZuSRMBbDoZlJJ30=', 'Content-Length': '0'} 2022-09-12T20:14:09.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b092e0b63388563-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.069 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:09.069 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:09.070 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:14:09.071 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:14:09.071 DEBUG:boto:Method: GET 2022-09-12T20:14:09.072 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:14:09.072 DEBUG:boto:Data: 2022-09-12T20:14:09.073 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.073 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.074 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.074 DEBUG:boto:Params: {} 2022-09-12T20:14:09.075 DEBUG:boto:Token: None 2022-09-12T20:14:09.075 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/ 2022-09-12T20:14:09.076 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A= 2022-09-12T20:14:09.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A=', 'Content-Length': '0'} 2022-09-12T20:14:09.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2cf6bc6f41f1330-00631f9311-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.080 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:09.081 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:09.081 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:09.082 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:14:09.082 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:14:09.083 DEBUG:boto:Method: GET 2022-09-12T20:14:09.084 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:14:09.084 DEBUG:boto:Data: 2022-09-12T20:14:09.085 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.085 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.086 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.086 DEBUG:boto:Params: {} 2022-09-12T20:14:09.087 DEBUG:boto:Token: None 2022-09-12T20:14:09.088 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/ 2022-09-12T20:14:09.088 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A= 2022-09-12T20:14:09.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A=', 'Content-Length': '0'} 2022-09-12T20:14:09.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000145ebd805e5bdf30-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.094 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:09.095 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:09.095 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:09.096 DEBUG:boto:Method: HEAD 2022-09-12T20:14:09.096 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:14:09.097 DEBUG:boto:Data: 2022-09-12T20:14:09.097 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.098 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.098 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.099 DEBUG:boto:Params: {} 2022-09-12T20:14:09.099 DEBUG:boto:Token: None 2022-09-12T20:14:09.100 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj44 2022-09-12T20:14:09.101 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tylKjLF/CC3lYOwlzdrI5Gm6s2A= 2022-09-12T20:14:09.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tylKjLF/CC3lYOwlzdrI5Gm6s2A=', 'Content-Length': '0'} 2022-09-12T20:14:09.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d12bbc7dd81c5d77-00631f9311-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.104 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:09.104 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:09.105 DEBUG:boto:Method: HEAD 2022-09-12T20:14:09.106 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:14:09.106 DEBUG:boto:Data: 2022-09-12T20:14:09.107 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.107 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.108 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.108 DEBUG:boto:Params: {} 2022-09-12T20:14:09.109 DEBUG:boto:Token: None 2022-09-12T20:14:09.109 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj44 2022-09-12T20:14:09.110 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tylKjLF/CC3lYOwlzdrI5Gm6s2A= 2022-09-12T20:14:09.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tylKjLF/CC3lYOwlzdrI5Gm6s2A=', 'Content-Length': '0'} 2022-09-12T20:14:09.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d44a42a2dd94165c-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:14:09.114 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:09.114 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:09.115 DEBUG:boto:Method: GET 2022-09-12T20:14:09.115 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:14:09.116 DEBUG:boto:Data: 2022-09-12T20:14:09.116 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.117 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.118 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.118 DEBUG:boto:Params: {} 2022-09-12T20:14:09.119 DEBUG:boto:Token: None 2022-09-12T20:14:09.119 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj44 2022-09-12T20:14:09.120 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F9Bi8bTGAtXCp02eg3GcH9ooIRw= 2022-09-12T20:14:09.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F9Bi8bTGAtXCp02eg3GcH9ooIRw=', 'Content-Length': '0'} 2022-09-12T20:14:09.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4653f6094b60d55-00631f9311-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.166 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:09.167 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:09.167 DEBUG:boto:Method: GET 2022-09-12T20:14:09.168 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:14:09.168 DEBUG:boto:Data: 2022-09-12T20:14:09.169 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.170 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.170 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.171 DEBUG:boto:Params: {} 2022-09-12T20:14:09.171 DEBUG:boto:Token: None 2022-09-12T20:14:09.172 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj44 2022-09-12T20:14:09.173 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F9Bi8bTGAtXCp02eg3GcH9ooIRw= 2022-09-12T20:14:09.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F9Bi8bTGAtXCp02eg3GcH9ooIRw=', 'Content-Length': '0'} 2022-09-12T20:14:09.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002244ea63abdd51b5-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.176 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:09.177 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:09.178 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:14:09.178 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:14:09.179 DEBUG:boto:Method: GET 2022-09-12T20:14:09.179 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:14:09.180 DEBUG:boto:Data: 2022-09-12T20:14:09.180 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.181 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.181 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.182 DEBUG:boto:Params: {} 2022-09-12T20:14:09.183 DEBUG:boto:Token: None 2022-09-12T20:14:09.183 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/ 2022-09-12T20:14:09.184 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A= 2022-09-12T20:14:09.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A=', 'Content-Length': '0'} 2022-09-12T20:14:09.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078d9895e402ed505-00631f9311-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.188 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:09.188 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:09.189 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:09.190 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:14:09.190 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:14:09.191 DEBUG:boto:Method: GET 2022-09-12T20:14:09.191 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:14:09.192 DEBUG:boto:Data: 2022-09-12T20:14:09.192 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.193 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.194 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.194 DEBUG:boto:Params: {} 2022-09-12T20:14:09.195 DEBUG:boto:Token: None 2022-09-12T20:14:09.195 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/ 2022-09-12T20:14:09.196 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A= 2022-09-12T20:14:09.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A=', 'Content-Length': '0'} 2022-09-12T20:14:09.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c2ecd3bb88e7ce8-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.202 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:09.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:14:09.203 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:09.204 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:09.204 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:09.205 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:09.205 DEBUG:boto:Method: GET 2022-09-12T20:14:09.206 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:09.206 DEBUG:boto:Data: 2022-09-12T20:14:09.207 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.207 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.208 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.209 DEBUG:boto:Params: {} 2022-09-12T20:14:09.209 DEBUG:boto:Token: None 2022-09-12T20:14:09.210 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:09.210 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:db4VcpuNWk/8NU//GL0c0f8SH0M= 2022-09-12T20:14:09.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:db4VcpuNWk/8NU//GL0c0f8SH0M=', 'Content-Length': '0'} 2022-09-12T20:14:09.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b454041c9e668be5-00631f9311-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.214 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:09.214 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:09.215 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:09.215 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:09.216 DEBUG:boto:Method: GET 2022-09-12T20:14:09.217 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:09.217 DEBUG:boto:Data: 2022-09-12T20:14:09.218 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.218 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.219 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.219 DEBUG:boto:Params: {} 2022-09-12T20:14:09.220 DEBUG:boto:Token: None 2022-09-12T20:14:09.220 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:09.221 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:db4VcpuNWk/8NU//GL0c0f8SH0M= 2022-09-12T20:14:09.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:db4VcpuNWk/8NU//GL0c0f8SH0M=', 'Content-Length': '0'} 2022-09-12T20:14:09.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097db9a17c5573a3f-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.225 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:09.225 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:09.226 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:09.226 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:09.227 DEBUG:boto:Method: HEAD 2022-09-12T20:14:09.227 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:09.228 DEBUG:boto:Data: 2022-09-12T20:14:09.228 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.229 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.229 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.230 DEBUG:boto:Params: {} 2022-09-12T20:14:09.231 DEBUG:boto:Token: None 2022-09-12T20:14:09.231 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:09.232 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BDsMtjpoR4wfuSlt4k8APD54Me4= 2022-09-12T20:14:09.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BDsMtjpoR4wfuSlt4k8APD54Me4=', 'Content-Length': '0'} 2022-09-12T20:14:09.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c344ed44b759372-00631f9311-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.235 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:09.236 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:09.236 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:09.237 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:09.237 DEBUG:boto:Method: HEAD 2022-09-12T20:14:09.238 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:09.238 DEBUG:boto:Data: 2022-09-12T20:14:09.239 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.239 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.240 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.240 DEBUG:boto:Params: {} 2022-09-12T20:14:09.241 DEBUG:boto:Token: None 2022-09-12T20:14:09.242 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:09.242 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BDsMtjpoR4wfuSlt4k8APD54Me4= 2022-09-12T20:14:09.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BDsMtjpoR4wfuSlt4k8APD54Me4=', 'Content-Length': '0'} 2022-09-12T20:14:09.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b697870d51ebbae2-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:14:09.246 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:09.247 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:09.247 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:09.248 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:09.248 DEBUG:boto:Method: GET 2022-09-12T20:14:09.249 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:09.249 DEBUG:boto:Data: 2022-09-12T20:14:09.250 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.250 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.251 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.251 DEBUG:boto:Params: {} 2022-09-12T20:14:09.252 DEBUG:boto:Token: None 2022-09-12T20:14:09.253 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:09.253 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:db4VcpuNWk/8NU//GL0c0f8SH0M= 2022-09-12T20:14:09.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:db4VcpuNWk/8NU//GL0c0f8SH0M=', 'Content-Length': '0'} 2022-09-12T20:14:09.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec1ff75d1cd5dc38-00631f9311-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.298 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:09.299 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:09.299 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:09.300 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:09.301 DEBUG:boto:Method: GET 2022-09-12T20:14:09.301 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:09.302 DEBUG:boto:Data: 2022-09-12T20:14:09.302 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.303 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.303 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.304 DEBUG:boto:Params: {} 2022-09-12T20:14:09.305 DEBUG:boto:Token: None 2022-09-12T20:14:09.305 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:09.306 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:db4VcpuNWk/8NU//GL0c0f8SH0M= 2022-09-12T20:14:09.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:db4VcpuNWk/8NU//GL0c0f8SH0M=', 'Content-Length': '0'} 2022-09-12T20:14:09.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023e440482ce014ec-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.309 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:09.310 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:09.311 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:14:09.311 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:14:09.312 DEBUG:boto:Method: GET 2022-09-12T20:14:09.312 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:14:09.313 DEBUG:boto:Data: 2022-09-12T20:14:09.313 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.314 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.314 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.315 DEBUG:boto:Params: {} 2022-09-12T20:14:09.316 DEBUG:boto:Token: None 2022-09-12T20:14:09.316 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/ 2022-09-12T20:14:09.317 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A= 2022-09-12T20:14:09.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A=', 'Content-Length': '0'} 2022-09-12T20:14:09.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002848e030efcda19-00631f9311-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.320 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:09.321 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:09.322 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:09.322 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:14:09.323 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:14:09.324 DEBUG:boto:Method: GET 2022-09-12T20:14:09.324 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:14:09.325 DEBUG:boto:Data: 2022-09-12T20:14:09.325 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.326 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.327 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.327 DEBUG:boto:Params: {} 2022-09-12T20:14:09.328 DEBUG:boto:Token: None 2022-09-12T20:14:09.329 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/ 2022-09-12T20:14:09.329 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A= 2022-09-12T20:14:09.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A=', 'Content-Length': '0'} 2022-09-12T20:14:09.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054bfb97f9b593ae1-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.336 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:09.336 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:09.337 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:09.338 DEBUG:boto:Method: HEAD 2022-09-12T20:14:09.338 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:14:09.339 DEBUG:boto:Data: 2022-09-12T20:14:09.339 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.340 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.340 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.341 DEBUG:boto:Params: {} 2022-09-12T20:14:09.341 DEBUG:boto:Token: None 2022-09-12T20:14:09.342 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj46 2022-09-12T20:14:09.343 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DjaWoKgOVxjg4iAaHMQRRZbEjoA= 2022-09-12T20:14:09.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DjaWoKgOVxjg4iAaHMQRRZbEjoA=', 'Content-Length': '0'} 2022-09-12T20:14:09.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d43452921d6fca2-00631f9311-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.346 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:09.347 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:09.347 DEBUG:boto:Method: HEAD 2022-09-12T20:14:09.348 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:14:09.348 DEBUG:boto:Data: 2022-09-12T20:14:09.349 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.349 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.350 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.350 DEBUG:boto:Params: {} 2022-09-12T20:14:09.351 DEBUG:boto:Token: None 2022-09-12T20:14:09.351 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj46 2022-09-12T20:14:09.352 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DjaWoKgOVxjg4iAaHMQRRZbEjoA= 2022-09-12T20:14:09.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DjaWoKgOVxjg4iAaHMQRRZbEjoA=', 'Content-Length': '0'} 2022-09-12T20:14:09.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005801230985435e1c-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:14:09.356 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:09.357 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:09.357 DEBUG:boto:Method: GET 2022-09-12T20:14:09.358 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:14:09.358 DEBUG:boto:Data: 2022-09-12T20:14:09.359 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.359 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.360 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.360 DEBUG:boto:Params: {} 2022-09-12T20:14:09.361 DEBUG:boto:Token: None 2022-09-12T20:14:09.362 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj46 2022-09-12T20:14:09.362 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+Y4QDiE0+Kv4hhodIg0tL3wgOWg= 2022-09-12T20:14:09.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+Y4QDiE0+Kv4hhodIg0tL3wgOWg=', 'Content-Length': '0'} 2022-09-12T20:14:09.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad564b1ec365114d-00631f9311-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.366 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:09.366 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:09.367 DEBUG:boto:Method: GET 2022-09-12T20:14:09.368 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:14:09.368 DEBUG:boto:Data: 2022-09-12T20:14:09.369 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.369 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.370 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.370 DEBUG:boto:Params: {} 2022-09-12T20:14:09.371 DEBUG:boto:Token: None 2022-09-12T20:14:09.372 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj46 2022-09-12T20:14:09.372 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+Y4QDiE0+Kv4hhodIg0tL3wgOWg= 2022-09-12T20:14:09.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+Y4QDiE0+Kv4hhodIg0tL3wgOWg=', 'Content-Length': '0'} 2022-09-12T20:14:09.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074f40d7d3174defc-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.418 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:09.419 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:09.419 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:14:09.420 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:14:09.420 DEBUG:boto:Method: GET 2022-09-12T20:14:09.421 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:14:09.421 DEBUG:boto:Data: 2022-09-12T20:14:09.422 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.422 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.423 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.423 DEBUG:boto:Params: {} 2022-09-12T20:14:09.424 DEBUG:boto:Token: None 2022-09-12T20:14:09.424 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/ 2022-09-12T20:14:09.425 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A= 2022-09-12T20:14:09.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A=', 'Content-Length': '0'} 2022-09-12T20:14:09.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a3912aa290db5de-00631f9311-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.429 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:09.430 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:09.430 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:09.431 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:14:09.431 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:14:09.432 DEBUG:boto:Method: GET 2022-09-12T20:14:09.432 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:14:09.433 DEBUG:boto:Data: 2022-09-12T20:14:09.434 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.434 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.435 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.435 DEBUG:boto:Params: {} 2022-09-12T20:14:09.436 DEBUG:boto:Token: None 2022-09-12T20:14:09.436 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/ 2022-09-12T20:14:09.437 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A= 2022-09-12T20:14:09.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A=', 'Content-Length': '0'} 2022-09-12T20:14:09.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d8469f3ea80351c-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.442 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:09.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:14:09.444 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:09.444 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:09.445 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:09.445 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:09.446 DEBUG:boto:Method: GET 2022-09-12T20:14:09.446 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:09.447 DEBUG:boto:Data: 2022-09-12T20:14:09.447 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.448 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.448 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.449 DEBUG:boto:Params: {} 2022-09-12T20:14:09.449 DEBUG:boto:Token: None 2022-09-12T20:14:09.450 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:09.451 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tXG3RzMPk65iuJyTBJkLXNHyV4g= 2022-09-12T20:14:09.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tXG3RzMPk65iuJyTBJkLXNHyV4g=', 'Content-Length': '0'} 2022-09-12T20:14:09.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2886696e7f0b78f-00631f9311-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.454 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:09.454 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:09.455 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:09.455 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:09.456 DEBUG:boto:Method: GET 2022-09-12T20:14:09.457 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:09.457 DEBUG:boto:Data: 2022-09-12T20:14:09.458 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.458 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.459 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.459 DEBUG:boto:Params: {} 2022-09-12T20:14:09.460 DEBUG:boto:Token: None 2022-09-12T20:14:09.460 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:09.461 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tXG3RzMPk65iuJyTBJkLXNHyV4g= 2022-09-12T20:14:09.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tXG3RzMPk65iuJyTBJkLXNHyV4g=', 'Content-Length': '0'} 2022-09-12T20:14:09.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024b6356e5a0364bf-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.465 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:09.465 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:09.466 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:09.466 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:09.467 DEBUG:boto:Method: HEAD 2022-09-12T20:14:09.467 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:09.468 DEBUG:boto:Data: 2022-09-12T20:14:09.468 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.469 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.469 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.470 DEBUG:boto:Params: {} 2022-09-12T20:14:09.470 DEBUG:boto:Token: None 2022-09-12T20:14:09.471 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:09.472 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cb8UkKOEw3PHoVOV9Bb/yVUce6w= 2022-09-12T20:14:09.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cb8UkKOEw3PHoVOV9Bb/yVUce6w=', 'Content-Length': '0'} 2022-09-12T20:14:09.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f283940d6ecf7f4b-00631f9311-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.475 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:09.475 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:09.476 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:09.476 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:09.477 DEBUG:boto:Method: HEAD 2022-09-12T20:14:09.477 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:09.478 DEBUG:boto:Data: 2022-09-12T20:14:09.478 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.479 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.479 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.480 DEBUG:boto:Params: {} 2022-09-12T20:14:09.480 DEBUG:boto:Token: None 2022-09-12T20:14:09.481 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:09.482 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cb8UkKOEw3PHoVOV9Bb/yVUce6w= 2022-09-12T20:14:09.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cb8UkKOEw3PHoVOV9Bb/yVUce6w=', 'Content-Length': '0'} 2022-09-12T20:14:09.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022e33d7e5e27e780-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:14:09.486 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:09.486 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:09.487 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:09.487 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:09.488 DEBUG:boto:Method: GET 2022-09-12T20:14:09.488 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:09.489 DEBUG:boto:Data: 2022-09-12T20:14:09.489 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.490 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.490 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.491 DEBUG:boto:Params: {} 2022-09-12T20:14:09.492 DEBUG:boto:Token: None 2022-09-12T20:14:09.492 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:09.493 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tXG3RzMPk65iuJyTBJkLXNHyV4g= 2022-09-12T20:14:09.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tXG3RzMPk65iuJyTBJkLXNHyV4g=', 'Content-Length': '0'} 2022-09-12T20:14:09.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e089d0faedca505c-00631f9311-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.496 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:09.497 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:09.497 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:09.498 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:09.498 DEBUG:boto:Method: GET 2022-09-12T20:14:09.499 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:09.500 DEBUG:boto:Data: 2022-09-12T20:14:09.500 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.501 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.501 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.502 DEBUG:boto:Params: {} 2022-09-12T20:14:09.502 DEBUG:boto:Token: None 2022-09-12T20:14:09.503 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:09.503 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tXG3RzMPk65iuJyTBJkLXNHyV4g= 2022-09-12T20:14:09.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tXG3RzMPk65iuJyTBJkLXNHyV4g=', 'Content-Length': '0'} 2022-09-12T20:14:09.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000402afe268c7c730b-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.507 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:09.507 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:09.508 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:14:09.508 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:14:09.509 DEBUG:boto:Method: GET 2022-09-12T20:14:09.509 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:14:09.510 DEBUG:boto:Data: 2022-09-12T20:14:09.511 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.511 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.512 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.512 DEBUG:boto:Params: {} 2022-09-12T20:14:09.513 DEBUG:boto:Token: None 2022-09-12T20:14:09.513 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/ 2022-09-12T20:14:09.514 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A= 2022-09-12T20:14:09.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A=', 'Content-Length': '0'} 2022-09-12T20:14:09.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b451c407f6f58876-00631f9311-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.518 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:09.518 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:09.519 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:09.520 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:14:09.520 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:14:09.521 DEBUG:boto:Method: GET 2022-09-12T20:14:09.521 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:14:09.522 DEBUG:boto:Data: 2022-09-12T20:14:09.522 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.523 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.523 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.524 DEBUG:boto:Params: {} 2022-09-12T20:14:09.524 DEBUG:boto:Token: None 2022-09-12T20:14:09.525 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/ 2022-09-12T20:14:09.526 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A= 2022-09-12T20:14:09.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A=', 'Content-Length': '0'} 2022-09-12T20:14:09.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003056a1fa4deeb6ca-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.531 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:09.532 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:09.532 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:09.533 DEBUG:boto:Method: HEAD 2022-09-12T20:14:09.533 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:14:09.534 DEBUG:boto:Data: 2022-09-12T20:14:09.535 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.536 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.536 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.537 DEBUG:boto:Params: {} 2022-09-12T20:14:09.537 DEBUG:boto:Token: None 2022-09-12T20:14:09.538 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj51 2022-09-12T20:14:09.538 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yP3QlzlHyRH0MeAzML70foXO2lk= 2022-09-12T20:14:09.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yP3QlzlHyRH0MeAzML70foXO2lk=', 'Content-Length': '0'} 2022-09-12T20:14:09.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000664b0118f33fcd65-00631f9311-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.541 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:09.542 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:09.542 DEBUG:boto:Method: HEAD 2022-09-12T20:14:09.543 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:14:09.543 DEBUG:boto:Data: 2022-09-12T20:14:09.544 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.544 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.545 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.545 DEBUG:boto:Params: {} 2022-09-12T20:14:09.546 DEBUG:boto:Token: None 2022-09-12T20:14:09.547 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj51 2022-09-12T20:14:09.547 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yP3QlzlHyRH0MeAzML70foXO2lk= 2022-09-12T20:14:09.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yP3QlzlHyRH0MeAzML70foXO2lk=', 'Content-Length': '0'} 2022-09-12T20:14:09.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0ac5f14a67dab6c-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:14:09.551 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:09.552 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:09.552 DEBUG:boto:Method: GET 2022-09-12T20:14:09.553 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:14:09.553 DEBUG:boto:Data: 2022-09-12T20:14:09.554 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.554 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.555 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.555 DEBUG:boto:Params: {} 2022-09-12T20:14:09.556 DEBUG:boto:Token: None 2022-09-12T20:14:09.557 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj51 2022-09-12T20:14:09.557 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hfinzICzlcJc3wq4WsV1S7y2j74= 2022-09-12T20:14:09.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hfinzICzlcJc3wq4WsV1S7y2j74=', 'Content-Length': '0'} 2022-09-12T20:14:09.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b05dbb9df21cd7c4-00631f9311-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.560 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:09.561 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:09.562 DEBUG:boto:Method: GET 2022-09-12T20:14:09.562 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:14:09.563 DEBUG:boto:Data: 2022-09-12T20:14:09.563 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.564 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.564 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.565 DEBUG:boto:Params: {} 2022-09-12T20:14:09.565 DEBUG:boto:Token: None 2022-09-12T20:14:09.566 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj51 2022-09-12T20:14:09.566 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hfinzICzlcJc3wq4WsV1S7y2j74= 2022-09-12T20:14:09.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hfinzICzlcJc3wq4WsV1S7y2j74=', 'Content-Length': '0'} 2022-09-12T20:14:09.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009551faa41cf6c41c-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.610 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:09.611 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:09.611 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:14:09.612 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:14:09.612 DEBUG:boto:Method: GET 2022-09-12T20:14:09.613 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:14:09.613 DEBUG:boto:Data: 2022-09-12T20:14:09.614 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.614 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.615 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.615 DEBUG:boto:Params: {} 2022-09-12T20:14:09.616 DEBUG:boto:Token: None 2022-09-12T20:14:09.617 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/ 2022-09-12T20:14:09.617 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A= 2022-09-12T20:14:09.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A=', 'Content-Length': '0'} 2022-09-12T20:14:09.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c62be95a411dddc6-00631f9311-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.621 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:09.622 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:09.623 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:09.623 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:14:09.624 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:14:09.624 DEBUG:boto:Method: GET 2022-09-12T20:14:09.625 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:14:09.625 DEBUG:boto:Data: 2022-09-12T20:14:09.626 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.626 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.627 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.627 DEBUG:boto:Params: {} 2022-09-12T20:14:09.628 DEBUG:boto:Token: None 2022-09-12T20:14:09.629 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/ 2022-09-12T20:14:09.629 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A= 2022-09-12T20:14:09.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A=', 'Content-Length': '0'} 2022-09-12T20:14:09.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043947b6c04153d22-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.635 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:09.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:14:09.636 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:09.636 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:09.637 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:09.637 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:09.638 DEBUG:boto:Method: GET 2022-09-12T20:14:09.638 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:09.639 DEBUG:boto:Data: 2022-09-12T20:14:09.639 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.640 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.640 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.641 DEBUG:boto:Params: {} 2022-09-12T20:14:09.642 DEBUG:boto:Token: None 2022-09-12T20:14:09.642 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:09.643 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5YE+o/1/TTnjZzHg7EytOZNklrY= 2022-09-12T20:14:09.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5YE+o/1/TTnjZzHg7EytOZNklrY=', 'Content-Length': '0'} 2022-09-12T20:14:09.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b001bdadc497ba6-00631f9311-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.646 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:09.647 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:09.647 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:09.648 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:09.649 DEBUG:boto:Method: GET 2022-09-12T20:14:09.649 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:09.650 DEBUG:boto:Data: 2022-09-12T20:14:09.650 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.651 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.651 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.652 DEBUG:boto:Params: {} 2022-09-12T20:14:09.652 DEBUG:boto:Token: None 2022-09-12T20:14:09.653 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:09.654 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5YE+o/1/TTnjZzHg7EytOZNklrY= 2022-09-12T20:14:09.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5YE+o/1/TTnjZzHg7EytOZNklrY=', 'Content-Length': '0'} 2022-09-12T20:14:09.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065991bd4b7817961-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.657 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:09.658 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:09.658 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:09.659 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:09.659 DEBUG:boto:Method: HEAD 2022-09-12T20:14:09.660 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:09.661 DEBUG:boto:Data: 2022-09-12T20:14:09.661 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.662 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.662 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.663 DEBUG:boto:Params: {} 2022-09-12T20:14:09.663 DEBUG:boto:Token: None 2022-09-12T20:14:09.664 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:09.664 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5f3mDaJ/veO6b6H7kjPqZCXtI8g= 2022-09-12T20:14:09.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5f3mDaJ/veO6b6H7kjPqZCXtI8g=', 'Content-Length': '0'} 2022-09-12T20:14:09.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2b7c02650745533-00631f9311-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.668 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:09.668 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:09.669 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:09.670 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:09.670 DEBUG:boto:Method: HEAD 2022-09-12T20:14:09.671 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:09.671 DEBUG:boto:Data: 2022-09-12T20:14:09.672 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.673 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.673 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.674 DEBUG:boto:Params: {} 2022-09-12T20:14:09.675 DEBUG:boto:Token: None 2022-09-12T20:14:09.675 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:09.676 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5f3mDaJ/veO6b6H7kjPqZCXtI8g= 2022-09-12T20:14:09.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5f3mDaJ/veO6b6H7kjPqZCXtI8g=', 'Content-Length': '0'} 2022-09-12T20:14:09.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0ed04936dde9666-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:14:09.680 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:09.681 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:09.681 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:09.682 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:09.683 DEBUG:boto:Method: GET 2022-09-12T20:14:09.684 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:09.684 DEBUG:boto:Data: 2022-09-12T20:14:09.685 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.686 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.686 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.687 DEBUG:boto:Params: {} 2022-09-12T20:14:09.688 DEBUG:boto:Token: None 2022-09-12T20:14:09.688 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:09.689 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5YE+o/1/TTnjZzHg7EytOZNklrY= 2022-09-12T20:14:09.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5YE+o/1/TTnjZzHg7EytOZNklrY=', 'Content-Length': '0'} 2022-09-12T20:14:09.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f152a5360c722d47-00631f9311-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.699 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:09.700 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:09.700 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:09.701 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:09.702 DEBUG:boto:Method: GET 2022-09-12T20:14:09.702 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:09.703 DEBUG:boto:Data: 2022-09-12T20:14:09.704 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.705 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.706 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.706 DEBUG:boto:Params: {} 2022-09-12T20:14:09.707 DEBUG:boto:Token: None 2022-09-12T20:14:09.708 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:09.708 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5YE+o/1/TTnjZzHg7EytOZNklrY= 2022-09-12T20:14:09.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5YE+o/1/TTnjZzHg7EytOZNklrY=', 'Content-Length': '0'} 2022-09-12T20:14:09.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cca89e8271ddfba3-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.754 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:09.755 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:09.755 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:14:09.756 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:14:09.757 DEBUG:boto:Method: GET 2022-09-12T20:14:09.757 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:14:09.758 DEBUG:boto:Data: 2022-09-12T20:14:09.758 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.759 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.760 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.760 DEBUG:boto:Params: {} 2022-09-12T20:14:09.761 DEBUG:boto:Token: None 2022-09-12T20:14:09.762 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/ 2022-09-12T20:14:09.762 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A= 2022-09-12T20:14:09.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A=', 'Content-Length': '0'} 2022-09-12T20:14:09.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd35edb60a3e4c00-00631f9311-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.767 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:09.768 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:09.768 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:09.769 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:14:09.769 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:14:09.770 DEBUG:boto:Method: GET 2022-09-12T20:14:09.771 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:14:09.772 DEBUG:boto:Data: 2022-09-12T20:14:09.772 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.773 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.774 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.774 DEBUG:boto:Params: {} 2022-09-12T20:14:09.775 DEBUG:boto:Token: None 2022-09-12T20:14:09.776 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/ 2022-09-12T20:14:09.777 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A= 2022-09-12T20:14:09.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A=', 'Content-Length': '0'} 2022-09-12T20:14:09.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f071d87fc2d7f90-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.783 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:09.784 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:09.784 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:09.785 DEBUG:boto:Method: HEAD 2022-09-12T20:14:09.786 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:14:09.786 DEBUG:boto:Data: 2022-09-12T20:14:09.787 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.787 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.788 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.789 DEBUG:boto:Params: {} 2022-09-12T20:14:09.789 DEBUG:boto:Token: None 2022-09-12T20:14:09.790 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj55 2022-09-12T20:14:09.791 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AYDVN4e98o7KtBajqiuseruCcBA= 2022-09-12T20:14:09.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AYDVN4e98o7KtBajqiuseruCcBA=', 'Content-Length': '0'} 2022-09-12T20:14:09.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4f8d5c9455701a1-00631f9311-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.795 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:09.795 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:09.796 DEBUG:boto:Method: HEAD 2022-09-12T20:14:09.797 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:14:09.798 DEBUG:boto:Data: 2022-09-12T20:14:09.798 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.799 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.800 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.800 DEBUG:boto:Params: {} 2022-09-12T20:14:09.801 DEBUG:boto:Token: None 2022-09-12T20:14:09.802 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj55 2022-09-12T20:14:09.803 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AYDVN4e98o7KtBajqiuseruCcBA= 2022-09-12T20:14:09.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AYDVN4e98o7KtBajqiuseruCcBA=', 'Content-Length': '0'} 2022-09-12T20:14:09.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d2750f8abcd42f1-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:14:09.807 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:09.808 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:09.809 DEBUG:boto:Method: GET 2022-09-12T20:14:09.809 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:14:09.810 DEBUG:boto:Data: 2022-09-12T20:14:09.811 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.812 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.812 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.813 DEBUG:boto:Params: {} 2022-09-12T20:14:09.814 DEBUG:boto:Token: None 2022-09-12T20:14:09.815 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj55 2022-09-12T20:14:09.816 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FS/oSA7hnCClNYzBzUEzboG/GcI= 2022-09-12T20:14:09.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FS/oSA7hnCClNYzBzUEzboG/GcI=', 'Content-Length': '0'} 2022-09-12T20:14:09.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f55a60ea2bbbd117-00631f9311-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.862 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:09.863 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:09.864 DEBUG:boto:Method: GET 2022-09-12T20:14:09.864 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:14:09.865 DEBUG:boto:Data: 2022-09-12T20:14:09.866 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.866 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.867 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.868 DEBUG:boto:Params: {} 2022-09-12T20:14:09.868 DEBUG:boto:Token: None 2022-09-12T20:14:09.869 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj55 2022-09-12T20:14:09.870 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FS/oSA7hnCClNYzBzUEzboG/GcI= 2022-09-12T20:14:09.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FS/oSA7hnCClNYzBzUEzboG/GcI=', 'Content-Length': '0'} 2022-09-12T20:14:09.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007db2984a398fee36-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.874 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:09.875 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:09.876 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:14:09.876 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:14:09.877 DEBUG:boto:Method: GET 2022-09-12T20:14:09.878 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:14:09.878 DEBUG:boto:Data: 2022-09-12T20:14:09.879 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.879 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.880 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.881 DEBUG:boto:Params: {} 2022-09-12T20:14:09.881 DEBUG:boto:Token: None 2022-09-12T20:14:09.882 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/ 2022-09-12T20:14:09.883 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A= 2022-09-12T20:14:09.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A=', 'Content-Length': '0'} 2022-09-12T20:14:09.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a6a700a1a1c2c78-00631f9311-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.887 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:09.888 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:09.888 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:09.889 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:14:09.890 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:14:09.890 DEBUG:boto:Method: GET 2022-09-12T20:14:09.891 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:14:09.892 DEBUG:boto:Data: 2022-09-12T20:14:09.892 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.893 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.893 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.894 DEBUG:boto:Params: {} 2022-09-12T20:14:09.896 DEBUG:boto:Token: None 2022-09-12T20:14:09.897 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/ 2022-09-12T20:14:09.898 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A= 2022-09-12T20:14:09.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WZVSMci4Px5tBQ2tARH8XtHR05A=', 'Content-Length': '0'} 2022-09-12T20:14:09.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a683b6378867897-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.906 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:09.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:14:09.908 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:09.909 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:09.910 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:09.912 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:09.913 DEBUG:boto:Method: GET 2022-09-12T20:14:09.914 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:09.915 DEBUG:boto:Data: 2022-09-12T20:14:09.916 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.917 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.918 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.920 DEBUG:boto:Params: {} 2022-09-12T20:14:09.921 DEBUG:boto:Token: None 2022-09-12T20:14:09.921 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:09.922 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:j4XwZ/ScBzsUBHoCkPv8TbYig3o= 2022-09-12T20:14:09.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:j4XwZ/ScBzsUBHoCkPv8TbYig3o=', 'Content-Length': '0'} 2022-09-12T20:14:09.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed6ce9585ce782c1-00631f9311-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.926 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:09.927 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:09.928 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:09.929 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:09.930 DEBUG:boto:Method: GET 2022-09-12T20:14:09.931 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:09.932 DEBUG:boto:Data: 2022-09-12T20:14:09.933 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.933 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:09.934 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.935 DEBUG:boto:Params: {} 2022-09-12T20:14:09.936 DEBUG:boto:Token: None 2022-09-12T20:14:09.937 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:09.938 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:j4XwZ/ScBzsUBHoCkPv8TbYig3o= 2022-09-12T20:14:09.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:j4XwZ/ScBzsUBHoCkPv8TbYig3o=', 'Content-Length': '0'} 2022-09-12T20:14:09.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ef2e1295b6f0c8b-00631f9311-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.982 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:09.983 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:09.984 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:09.984 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:09.985 DEBUG:boto:Method: HEAD 2022-09-12T20:14:09.986 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:09.987 DEBUG:boto:Data: 2022-09-12T20:14:09.987 DEBUG:boto:Headers: {} 2022-09-12T20:14:09.988 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:09.988 DEBUG:boto:Port: 8000 2022-09-12T20:14:09.989 DEBUG:boto:Params: {} 2022-09-12T20:14:09.990 DEBUG:boto:Token: None 2022-09-12T20:14:09.991 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:09 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:09.991 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PB+OpRzQtnHwUMQFaGhECMUjO4Y= 2022-09-12T20:14:09.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PB+OpRzQtnHwUMQFaGhECMUjO4Y=', 'Content-Length': '0'} 2022-09-12T20:14:09.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047522e2e2b64330c-00631f9311-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:09.995 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:09.996 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:09.997 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:09.997 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:09.998 DEBUG:boto:Method: HEAD 2022-09-12T20:14:09.999 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:09.999 DEBUG:boto:Data: 2022-09-12T20:14:10.000 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.001 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:10.001 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.002 DEBUG:boto:Params: {} 2022-09-12T20:14:10.003 DEBUG:boto:Token: None 2022-09-12T20:14:10.003 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:10.004 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GVGaF48qm5Xu19Ho/P75djh1WPc= 2022-09-12T20:14:10.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GVGaF48qm5Xu19Ho/P75djh1WPc=', 'Content-Length': '0'} 2022-09-12T20:14:10.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058c64ad2a19cfba6-00631f9312-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:14:10.009 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:10.010 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:10.010 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:10.011 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:10.012 DEBUG:boto:Method: GET 2022-09-12T20:14:10.012 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:10.013 DEBUG:boto:Data: 2022-09-12T20:14:10.014 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.014 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:10.015 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.015 DEBUG:boto:Params: {} 2022-09-12T20:14:10.016 DEBUG:boto:Token: None 2022-09-12T20:14:10.017 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:10.017 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HnjZuFM23O6GUDoK4bDZWDReX3o= 2022-09-12T20:14:10.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HnjZuFM23O6GUDoK4bDZWDReX3o=', 'Content-Length': '0'} 2022-09-12T20:14:10.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015eea2be7442deb6-00631f9312-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.062 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:10.063 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:10.063 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:10.064 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:10.065 DEBUG:boto:Method: GET 2022-09-12T20:14:10.065 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:10.066 DEBUG:boto:Data: 2022-09-12T20:14:10.067 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.067 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:10.068 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.069 DEBUG:boto:Params: {} 2022-09-12T20:14:10.069 DEBUG:boto:Token: None 2022-09-12T20:14:10.070 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:10.071 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HnjZuFM23O6GUDoK4bDZWDReX3o= 2022-09-12T20:14:10.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HnjZuFM23O6GUDoK4bDZWDReX3o=', 'Content-Length': '0'} 2022-09-12T20:14:10.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000493fd487d6361765-00631f9312-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.074 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:10.075 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:10.076 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:14:10.076 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:14:10.077 DEBUG:boto:Method: GET 2022-09-12T20:14:10.077 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:14:10.078 DEBUG:boto:Data: 2022-09-12T20:14:10.078 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.079 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:10.079 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.080 DEBUG:boto:Params: {} 2022-09-12T20:14:10.080 DEBUG:boto:Token: None 2022-09-12T20:14:10.081 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/ 2022-09-12T20:14:10.081 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ= 2022-09-12T20:14:10.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ=', 'Content-Length': '0'} 2022-09-12T20:14:10.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c7258ffdc9a717b-00631f9312-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.085 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:10.086 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:10.087 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:10.087 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:14:10.088 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:14:10.088 DEBUG:boto:Method: GET 2022-09-12T20:14:10.089 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:14:10.089 DEBUG:boto:Data: 2022-09-12T20:14:10.090 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.090 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:10.091 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.091 DEBUG:boto:Params: {} 2022-09-12T20:14:10.092 DEBUG:boto:Token: None 2022-09-12T20:14:10.093 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/ 2022-09-12T20:14:10.093 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ= 2022-09-12T20:14:10.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ=', 'Content-Length': '0'} 2022-09-12T20:14:10.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a64240ea76617b03-00631f9312-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.099 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:10.099 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:10.100 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:10.101 DEBUG:boto:Method: HEAD 2022-09-12T20:14:10.101 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:14:10.102 DEBUG:boto:Data: 2022-09-12T20:14:10.102 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.103 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:10.103 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.104 DEBUG:boto:Params: {} 2022-09-12T20:14:10.104 DEBUG:boto:Token: None 2022-09-12T20:14:10.105 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj57 2022-09-12T20:14:10.105 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+Pxkmf9rVB0xZiwHrqv0H8iUMdw= 2022-09-12T20:14:10.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+Pxkmf9rVB0xZiwHrqv0H8iUMdw=', 'Content-Length': '0'} 2022-09-12T20:14:10.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c49f6ff0ec61c730-00631f9312-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.108 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:10.109 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:10.110 DEBUG:boto:Method: HEAD 2022-09-12T20:14:10.110 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:14:10.111 DEBUG:boto:Data: 2022-09-12T20:14:10.111 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.111 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:10.112 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.112 DEBUG:boto:Params: {} 2022-09-12T20:14:10.113 DEBUG:boto:Token: None 2022-09-12T20:14:10.114 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj57 2022-09-12T20:14:10.114 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+Pxkmf9rVB0xZiwHrqv0H8iUMdw= 2022-09-12T20:14:10.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+Pxkmf9rVB0xZiwHrqv0H8iUMdw=', 'Content-Length': '0'} 2022-09-12T20:14:10.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008385939e5f28d81e-00631f9312-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:14:10.118 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:10.118 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:10.119 DEBUG:boto:Method: GET 2022-09-12T20:14:10.119 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:14:10.120 DEBUG:boto:Data: 2022-09-12T20:14:10.120 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.121 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:10.121 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.122 DEBUG:boto:Params: {} 2022-09-12T20:14:10.123 DEBUG:boto:Token: None 2022-09-12T20:14:10.123 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj57 2022-09-12T20:14:10.124 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c/cu6TDkLDzQAH1JAllxUXltjvg= 2022-09-12T20:14:10.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c/cu6TDkLDzQAH1JAllxUXltjvg=', 'Content-Length': '0'} 2022-09-12T20:14:10.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000459c5bd1a2300d0e-00631f9312-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.127 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:10.128 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:10.128 DEBUG:boto:Method: GET 2022-09-12T20:14:10.129 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:14:10.129 DEBUG:boto:Data: 2022-09-12T20:14:10.130 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.130 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:10.131 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.132 DEBUG:boto:Params: {} 2022-09-12T20:14:10.132 DEBUG:boto:Token: None 2022-09-12T20:14:10.133 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj57 2022-09-12T20:14:10.133 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c/cu6TDkLDzQAH1JAllxUXltjvg= 2022-09-12T20:14:10.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c/cu6TDkLDzQAH1JAllxUXltjvg=', 'Content-Length': '0'} 2022-09-12T20:14:10.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069201c90d1e7e757-00631f9312-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.137 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:10.138 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:10.138 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:14:10.139 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:14:10.139 DEBUG:boto:Method: GET 2022-09-12T20:14:10.140 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:14:10.140 DEBUG:boto:Data: 2022-09-12T20:14:10.141 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.141 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:10.142 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.142 DEBUG:boto:Params: {} 2022-09-12T20:14:10.143 DEBUG:boto:Token: None 2022-09-12T20:14:10.143 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/ 2022-09-12T20:14:10.144 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ= 2022-09-12T20:14:10.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ=', 'Content-Length': '0'} 2022-09-12T20:14:10.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000086e2f775223e7b1-00631f9312-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.148 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:10.149 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:10.149 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:10.150 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:14:10.150 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:14:10.151 DEBUG:boto:Method: GET 2022-09-12T20:14:10.151 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:14:10.152 DEBUG:boto:Data: 2022-09-12T20:14:10.152 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.153 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:10.153 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.154 DEBUG:boto:Params: {} 2022-09-12T20:14:10.155 DEBUG:boto:Token: None 2022-09-12T20:14:10.155 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/ 2022-09-12T20:14:10.156 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ= 2022-09-12T20:14:10.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ=', 'Content-Length': '0'} 2022-09-12T20:14:10.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057db01f4f7505088-00631f9312-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.161 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:10.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:14:10.162 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:10.163 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:10.164 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:10.164 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:10.165 DEBUG:boto:Method: GET 2022-09-12T20:14:10.165 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:10.166 DEBUG:boto:Data: 2022-09-12T20:14:10.166 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.167 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:10.167 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.168 DEBUG:boto:Params: {} 2022-09-12T20:14:10.168 DEBUG:boto:Token: None 2022-09-12T20:14:10.169 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:10.169 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q3eo2s0Tgc4GnkCyTwtjPfBsF6Q= 2022-09-12T20:14:10.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q3eo2s0Tgc4GnkCyTwtjPfBsF6Q=', 'Content-Length': '0'} 2022-09-12T20:14:10.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df0f57c2d45b9ca8-00631f9312-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.214 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:10.215 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:10.216 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:10.217 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:10.218 DEBUG:boto:Method: GET 2022-09-12T20:14:10.219 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:10.219 DEBUG:boto:Data: 2022-09-12T20:14:10.220 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.221 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:10.221 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.222 DEBUG:boto:Params: {} 2022-09-12T20:14:10.223 DEBUG:boto:Token: None 2022-09-12T20:14:10.224 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:10.225 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q3eo2s0Tgc4GnkCyTwtjPfBsF6Q= 2022-09-12T20:14:10.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q3eo2s0Tgc4GnkCyTwtjPfBsF6Q=', 'Content-Length': '0'} 2022-09-12T20:14:10.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5e0b1ea6e022b37-00631f9312-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.231 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:10.232 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:10.232 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:10.233 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:10.233 DEBUG:boto:Method: HEAD 2022-09-12T20:14:10.234 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:10.235 DEBUG:boto:Data: 2022-09-12T20:14:10.236 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.237 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:10.238 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.238 DEBUG:boto:Params: {} 2022-09-12T20:14:10.239 DEBUG:boto:Token: None 2022-09-12T20:14:10.240 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:10.241 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5AToiMZjMkMokI03TJ4ymTd0fAA= 2022-09-12T20:14:10.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5AToiMZjMkMokI03TJ4ymTd0fAA=', 'Content-Length': '0'} 2022-09-12T20:14:10.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d45bd69f616312eb-00631f9312-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.246 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:10.246 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:10.247 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:10.248 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:10.249 DEBUG:boto:Method: HEAD 2022-09-12T20:14:10.249 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:10.250 DEBUG:boto:Data: 2022-09-12T20:14:10.251 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.252 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:10.252 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.253 DEBUG:boto:Params: {} 2022-09-12T20:14:10.254 DEBUG:boto:Token: None 2022-09-12T20:14:10.255 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:10.256 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5AToiMZjMkMokI03TJ4ymTd0fAA= 2022-09-12T20:14:10.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5AToiMZjMkMokI03TJ4ymTd0fAA=', 'Content-Length': '0'} 2022-09-12T20:14:10.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcc3dbc80a1852bf-00631f9312-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:14:10.262 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:10.263 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:10.263 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:10.264 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:10.265 DEBUG:boto:Method: GET 2022-09-12T20:14:10.265 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:10.266 DEBUG:boto:Data: 2022-09-12T20:14:10.266 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.267 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:10.267 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.268 DEBUG:boto:Params: {} 2022-09-12T20:14:10.268 DEBUG:boto:Token: None 2022-09-12T20:14:10.269 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:10.270 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q3eo2s0Tgc4GnkCyTwtjPfBsF6Q= 2022-09-12T20:14:10.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q3eo2s0Tgc4GnkCyTwtjPfBsF6Q=', 'Content-Length': '0'} 2022-09-12T20:14:10.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c50eca1007aff33e-00631f9312-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.274 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:10.275 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:10.276 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:10.277 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:10.278 DEBUG:boto:Method: GET 2022-09-12T20:14:10.279 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:10.280 DEBUG:boto:Data: 2022-09-12T20:14:10.281 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.282 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:10.283 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.284 DEBUG:boto:Params: {} 2022-09-12T20:14:10.285 DEBUG:boto:Token: None 2022-09-12T20:14:10.287 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:10.288 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q3eo2s0Tgc4GnkCyTwtjPfBsF6Q= 2022-09-12T20:14:10.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q3eo2s0Tgc4GnkCyTwtjPfBsF6Q=', 'Content-Length': '0'} 2022-09-12T20:14:10.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b03ebca30cce086-00631f9312-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.293 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:10.293 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:10.294 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:14:10.294 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:14:10.295 DEBUG:boto:Method: GET 2022-09-12T20:14:10.295 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:14:10.296 DEBUG:boto:Data: 2022-09-12T20:14:10.296 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.297 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:10.298 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.298 DEBUG:boto:Params: {} 2022-09-12T20:14:10.299 DEBUG:boto:Token: None 2022-09-12T20:14:10.299 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/ 2022-09-12T20:14:10.300 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ= 2022-09-12T20:14:10.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ=', 'Content-Length': '0'} 2022-09-12T20:14:10.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fdd3fd9ada1c7d2-00631f9312-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.307 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:10.308 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:10.309 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:10.309 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:14:10.310 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:14:10.311 DEBUG:boto:Method: GET 2022-09-12T20:14:10.311 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:14:10.312 DEBUG:boto:Data: 2022-09-12T20:14:10.312 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.313 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:10.313 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.314 DEBUG:boto:Params: {} 2022-09-12T20:14:10.315 DEBUG:boto:Token: None 2022-09-12T20:14:10.315 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/ 2022-09-12T20:14:10.316 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ= 2022-09-12T20:14:10.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ=', 'Content-Length': '0'} 2022-09-12T20:14:10.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000893e05b2e8da970f-00631f9312-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.330 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:10.331 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:10.332 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:10.332 DEBUG:boto:Method: HEAD 2022-09-12T20:14:10.333 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:14:10.333 DEBUG:boto:Data: 2022-09-12T20:14:10.334 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.334 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:10.335 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.335 DEBUG:boto:Params: {} 2022-09-12T20:14:10.336 DEBUG:boto:Token: None 2022-09-12T20:14:10.336 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj58 2022-09-12T20:14:10.337 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E8saRQ/AYp+CVAO2nXnJ/POLRKE= 2022-09-12T20:14:10.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E8saRQ/AYp+CVAO2nXnJ/POLRKE=', 'Content-Length': '0'} 2022-09-12T20:14:10.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b4844fe3e333f3c-00631f9312-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.341 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:10.341 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:10.342 DEBUG:boto:Method: HEAD 2022-09-12T20:14:10.342 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:14:10.343 DEBUG:boto:Data: 2022-09-12T20:14:10.343 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.344 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:10.344 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.345 DEBUG:boto:Params: {} 2022-09-12T20:14:10.345 DEBUG:boto:Token: None 2022-09-12T20:14:10.346 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj58 2022-09-12T20:14:10.347 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E8saRQ/AYp+CVAO2nXnJ/POLRKE= 2022-09-12T20:14:10.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E8saRQ/AYp+CVAO2nXnJ/POLRKE=', 'Content-Length': '0'} 2022-09-12T20:14:10.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029efd3ad58a5a6f7-00631f9312-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:14:10.351 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:10.351 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:10.352 DEBUG:boto:Method: GET 2022-09-12T20:14:10.352 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:14:10.353 DEBUG:boto:Data: 2022-09-12T20:14:10.353 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.354 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:10.354 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.355 DEBUG:boto:Params: {} 2022-09-12T20:14:10.356 DEBUG:boto:Token: None 2022-09-12T20:14:10.356 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj58 2022-09-12T20:14:10.357 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RMwKgJkJSS44Sm23K/CW7KYqW1Y= 2022-09-12T20:14:10.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RMwKgJkJSS44Sm23K/CW7KYqW1Y=', 'Content-Length': '0'} 2022-09-12T20:14:10.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4dc9f082d2a18aa-00631f9312-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.363 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:10.364 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:10.365 DEBUG:boto:Method: GET 2022-09-12T20:14:10.365 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:14:10.366 DEBUG:boto:Data: 2022-09-12T20:14:10.366 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.367 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:10.367 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.368 DEBUG:boto:Params: {} 2022-09-12T20:14:10.368 DEBUG:boto:Token: None 2022-09-12T20:14:10.369 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj58 2022-09-12T20:14:10.370 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RMwKgJkJSS44Sm23K/CW7KYqW1Y= 2022-09-12T20:14:10.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RMwKgJkJSS44Sm23K/CW7KYqW1Y=', 'Content-Length': '0'} 2022-09-12T20:14:10.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002553d579b0798b34-00631f9312-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.415 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:10.416 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:10.417 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:14:10.417 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:14:10.418 DEBUG:boto:Method: GET 2022-09-12T20:14:10.418 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:14:10.419 DEBUG:boto:Data: 2022-09-12T20:14:10.419 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.420 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:10.421 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.421 DEBUG:boto:Params: {} 2022-09-12T20:14:10.422 DEBUG:boto:Token: None 2022-09-12T20:14:10.422 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/ 2022-09-12T20:14:10.423 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ= 2022-09-12T20:14:10.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ=', 'Content-Length': '0'} 2022-09-12T20:14:10.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccf9133e34dbaca8-00631f9312-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.427 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:10.428 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:10.429 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:10.429 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:14:10.430 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:14:10.430 DEBUG:boto:Method: GET 2022-09-12T20:14:10.431 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:14:10.431 DEBUG:boto:Data: 2022-09-12T20:14:10.432 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.432 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:10.433 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.434 DEBUG:boto:Params: {} 2022-09-12T20:14:10.434 DEBUG:boto:Token: None 2022-09-12T20:14:10.435 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/ 2022-09-12T20:14:10.436 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ= 2022-09-12T20:14:10.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ=', 'Content-Length': '0'} 2022-09-12T20:14:10.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4ef02e29b9383c0-00631f9312-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.444 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:10.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-12T20:14:10.446 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:10.446 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:10.447 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:10.448 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:10.449 DEBUG:boto:Method: GET 2022-09-12T20:14:10.450 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:10.450 DEBUG:boto:Data: 2022-09-12T20:14:10.451 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.452 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:10.453 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.453 DEBUG:boto:Params: {} 2022-09-12T20:14:10.454 DEBUG:boto:Token: None 2022-09-12T20:14:10.455 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:10.456 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7W0wAbJnsoTJ+BgdhrDUgXmHQ3M= 2022-09-12T20:14:10.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7W0wAbJnsoTJ+BgdhrDUgXmHQ3M=', 'Content-Length': '0'} 2022-09-12T20:14:10.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000851dba8ab10f1f9e-00631f9312-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.506 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:10.507 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:10.508 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:10.508 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:10.509 DEBUG:boto:Method: GET 2022-09-12T20:14:10.509 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:10.510 DEBUG:boto:Data: 2022-09-12T20:14:10.511 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.511 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:10.512 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.512 DEBUG:boto:Params: {} 2022-09-12T20:14:10.513 DEBUG:boto:Token: None 2022-09-12T20:14:10.513 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:10.514 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7W0wAbJnsoTJ+BgdhrDUgXmHQ3M= 2022-09-12T20:14:10.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7W0wAbJnsoTJ+BgdhrDUgXmHQ3M=', 'Content-Length': '0'} 2022-09-12T20:14:10.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6ea54b16916ac55-00631f9312-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.519 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:10.520 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:10.520 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:10.521 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:10.521 DEBUG:boto:Method: HEAD 2022-09-12T20:14:10.522 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:10.522 DEBUG:boto:Data: 2022-09-12T20:14:10.523 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.523 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:10.524 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.524 DEBUG:boto:Params: {} 2022-09-12T20:14:10.525 DEBUG:boto:Token: None 2022-09-12T20:14:10.526 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:10.526 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yhccEc+oesEu5HlRqarGmXNvoNQ= 2022-09-12T20:14:10.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yhccEc+oesEu5HlRqarGmXNvoNQ=', 'Content-Length': '0'} 2022-09-12T20:14:10.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000855072ebb4120edc-00631f9312-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.534 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:10.535 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:10.536 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:10.536 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:10.537 DEBUG:boto:Method: HEAD 2022-09-12T20:14:10.537 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:10.538 DEBUG:boto:Data: 2022-09-12T20:14:10.538 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.539 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:10.539 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.540 DEBUG:boto:Params: {} 2022-09-12T20:14:10.540 DEBUG:boto:Token: None 2022-09-12T20:14:10.541 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:10.542 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yhccEc+oesEu5HlRqarGmXNvoNQ= 2022-09-12T20:14:10.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yhccEc+oesEu5HlRqarGmXNvoNQ=', 'Content-Length': '0'} 2022-09-12T20:14:10.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058ddef9841c3ad55-00631f9312-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-12T20:14:10.549 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:10.550 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:10.550 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:10.551 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:10.551 DEBUG:boto:Method: GET 2022-09-12T20:14:10.552 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:10.553 DEBUG:boto:Data: 2022-09-12T20:14:10.553 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.554 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:10.554 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.555 DEBUG:boto:Params: {} 2022-09-12T20:14:10.555 DEBUG:boto:Token: None 2022-09-12T20:14:10.556 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:10.557 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7W0wAbJnsoTJ+BgdhrDUgXmHQ3M= 2022-09-12T20:14:10.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7W0wAbJnsoTJ+BgdhrDUgXmHQ3M=', 'Content-Length': '0'} 2022-09-12T20:14:10.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f88ce39f71383bb0-00631f9312-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.564 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:10.565 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:10.565 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:10.566 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:10.567 DEBUG:boto:Method: GET 2022-09-12T20:14:10.568 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:10.569 DEBUG:boto:Data: 2022-09-12T20:14:10.569 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.570 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:10.570 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.571 DEBUG:boto:Params: {} 2022-09-12T20:14:10.571 DEBUG:boto:Token: None 2022-09-12T20:14:10.572 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:10.573 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7W0wAbJnsoTJ+BgdhrDUgXmHQ3M= 2022-09-12T20:14:10.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7W0wAbJnsoTJ+BgdhrDUgXmHQ3M=', 'Content-Length': '0'} 2022-09-12T20:14:10.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000830068a293abb863-00631f9312-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.619 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:10.620 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:10.620 DEBUG:boto:path=/jdpiak-75/?prefix=obj62 2022-09-12T20:14:10.621 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj62 2022-09-12T20:14:10.621 DEBUG:boto:Method: GET 2022-09-12T20:14:10.622 DEBUG:boto:Path: /jdpiak-75/?prefix=obj62 2022-09-12T20:14:10.622 DEBUG:boto:Data: 2022-09-12T20:14:10.623 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.623 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:10.624 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.624 DEBUG:boto:Params: {} 2022-09-12T20:14:10.625 DEBUG:boto:Token: None 2022-09-12T20:14:10.626 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/ 2022-09-12T20:14:10.626 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ= 2022-09-12T20:14:10.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ=', 'Content-Length': '0'} 2022-09-12T20:14:10.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ea746945579bbb3-00631f9312-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.634 DEBUG:boto:b'jdpiak-75obj621000falseobj622022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:10.635 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:10.635 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:10.636 DEBUG:boto:path=/jdpiak-75/?prefix=obj62 2022-09-12T20:14:10.636 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj62 2022-09-12T20:14:10.637 DEBUG:boto:Method: GET 2022-09-12T20:14:10.638 DEBUG:boto:Path: /jdpiak-75/?prefix=obj62 2022-09-12T20:14:10.638 DEBUG:boto:Data: 2022-09-12T20:14:10.639 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.639 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:10.640 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.640 DEBUG:boto:Params: {} 2022-09-12T20:14:10.641 DEBUG:boto:Token: None 2022-09-12T20:14:10.641 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/ 2022-09-12T20:14:10.642 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ= 2022-09-12T20:14:10.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ=', 'Content-Length': '0'} 2022-09-12T20:14:10.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dea244eb643aea4-00631f9312-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.649 DEBUG:boto:b'jdpiak-75obj621000falseobj622022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:10.650 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:10.651 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:10.651 DEBUG:boto:Method: HEAD 2022-09-12T20:14:10.652 DEBUG:boto:Path: /jdpiak-75/obj62 2022-09-12T20:14:10.653 DEBUG:boto:Data: 2022-09-12T20:14:10.654 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.655 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:10.656 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.656 DEBUG:boto:Params: {} 2022-09-12T20:14:10.657 DEBUG:boto:Token: None 2022-09-12T20:14:10.658 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj62 2022-09-12T20:14:10.659 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4eCqQjyQ7BoOxWCZxVg5W3+iGh0= 2022-09-12T20:14:10.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4eCqQjyQ7BoOxWCZxVg5W3+iGh0=', 'Content-Length': '0'} 2022-09-12T20:14:10.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a56c2117a6334e29-00631f9312-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.663 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:10.664 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:10.665 DEBUG:boto:Method: HEAD 2022-09-12T20:14:10.665 DEBUG:boto:Path: /jdpiak-75/obj62 2022-09-12T20:14:10.666 DEBUG:boto:Data: 2022-09-12T20:14:10.667 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.667 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:10.668 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.668 DEBUG:boto:Params: {} 2022-09-12T20:14:10.669 DEBUG:boto:Token: None 2022-09-12T20:14:10.670 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj62 2022-09-12T20:14:10.671 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4eCqQjyQ7BoOxWCZxVg5W3+iGh0= 2022-09-12T20:14:10.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4eCqQjyQ7BoOxWCZxVg5W3+iGh0=', 'Content-Length': '0'} 2022-09-12T20:14:10.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a90c574e2e40e122-00631f9312-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-12T20:14:10.677 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:10.678 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:10.679 DEBUG:boto:Method: GET 2022-09-12T20:14:10.680 DEBUG:boto:Path: /jdpiak-75/obj62 2022-09-12T20:14:10.680 DEBUG:boto:Data: 2022-09-12T20:14:10.681 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.682 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:10.683 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.684 DEBUG:boto:Params: {} 2022-09-12T20:14:10.684 DEBUG:boto:Token: None 2022-09-12T20:14:10.685 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj62 2022-09-12T20:14:10.686 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Jqj03WkUEojTtq/dH9AyOPdBlVk= 2022-09-12T20:14:10.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Jqj03WkUEojTtq/dH9AyOPdBlVk=', 'Content-Length': '0'} 2022-09-12T20:14:10.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbfa3726e090badf-00631f9312-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.735 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:10.736 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:10.737 DEBUG:boto:Method: GET 2022-09-12T20:14:10.737 DEBUG:boto:Path: /jdpiak-75/obj62 2022-09-12T20:14:10.738 DEBUG:boto:Data: 2022-09-12T20:14:10.738 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.739 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:10.739 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.740 DEBUG:boto:Params: {} 2022-09-12T20:14:10.741 DEBUG:boto:Token: None 2022-09-12T20:14:10.742 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj62 2022-09-12T20:14:10.742 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Jqj03WkUEojTtq/dH9AyOPdBlVk= 2022-09-12T20:14:10.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Jqj03WkUEojTtq/dH9AyOPdBlVk=', 'Content-Length': '0'} 2022-09-12T20:14:10.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec0265bcf2c4e3da-00631f9312-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.747 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:10.748 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:10.748 DEBUG:boto:path=/jdpiak-75/?prefix=obj62 2022-09-12T20:14:10.749 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj62 2022-09-12T20:14:10.749 DEBUG:boto:Method: GET 2022-09-12T20:14:10.750 DEBUG:boto:Path: /jdpiak-75/?prefix=obj62 2022-09-12T20:14:10.750 DEBUG:boto:Data: 2022-09-12T20:14:10.751 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.751 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:10.752 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.753 DEBUG:boto:Params: {} 2022-09-12T20:14:10.753 DEBUG:boto:Token: None 2022-09-12T20:14:10.754 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/ 2022-09-12T20:14:10.755 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ= 2022-09-12T20:14:10.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ=', 'Content-Length': '0'} 2022-09-12T20:14:10.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012a9a246b31350ab-00631f9312-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.760 DEBUG:boto:b'jdpiak-75obj621000falseobj622022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:10.761 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:10.762 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:10.762 DEBUG:boto:path=/jdpiak-75/?prefix=obj62 2022-09-12T20:14:10.763 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj62 2022-09-12T20:14:10.764 DEBUG:boto:Method: GET 2022-09-12T20:14:10.764 DEBUG:boto:Path: /jdpiak-75/?prefix=obj62 2022-09-12T20:14:10.765 DEBUG:boto:Data: 2022-09-12T20:14:10.766 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.766 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:10.767 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.767 DEBUG:boto:Params: {} 2022-09-12T20:14:10.768 DEBUG:boto:Token: None 2022-09-12T20:14:10.769 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/ 2022-09-12T20:14:10.769 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ= 2022-09-12T20:14:10.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ=', 'Content-Length': '0'} 2022-09-12T20:14:10.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba8b06b810a2e5b8-00631f9312-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.818 DEBUG:boto:b'jdpiak-75obj621000falseobj622022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:10.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:14:10.821 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:10.822 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:10.823 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:10.823 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:10.824 DEBUG:boto:Method: GET 2022-09-12T20:14:10.825 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:10.826 DEBUG:boto:Data: 2022-09-12T20:14:10.827 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.827 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:10.828 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.829 DEBUG:boto:Params: {} 2022-09-12T20:14:10.830 DEBUG:boto:Token: None 2022-09-12T20:14:10.831 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:10.832 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k6d1JlUkszJbjr4HLsd5+xEzk2s= 2022-09-12T20:14:10.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k6d1JlUkszJbjr4HLsd5+xEzk2s=', 'Content-Length': '0'} 2022-09-12T20:14:10.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d778c15613da525d-00631f9312-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.837 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:10.838 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:10.839 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:10.839 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:10.840 DEBUG:boto:Method: GET 2022-09-12T20:14:10.841 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:10.841 DEBUG:boto:Data: 2022-09-12T20:14:10.842 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.843 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:10.844 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.845 DEBUG:boto:Params: {} 2022-09-12T20:14:10.845 DEBUG:boto:Token: None 2022-09-12T20:14:10.846 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:10.847 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k6d1JlUkszJbjr4HLsd5+xEzk2s= 2022-09-12T20:14:10.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k6d1JlUkszJbjr4HLsd5+xEzk2s=', 'Content-Length': '0'} 2022-09-12T20:14:10.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3314026201355b0-00631f9312-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.852 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:10.853 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:10.854 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:10.854 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:10.855 DEBUG:boto:Method: HEAD 2022-09-12T20:14:10.855 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:10.856 DEBUG:boto:Data: 2022-09-12T20:14:10.856 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.857 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:10.857 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.858 DEBUG:boto:Params: {} 2022-09-12T20:14:10.858 DEBUG:boto:Token: None 2022-09-12T20:14:10.859 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:10.860 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G6yfsiw6s9KmRmd+S27ZpZnCoqc= 2022-09-12T20:14:10.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G6yfsiw6s9KmRmd+S27ZpZnCoqc=', 'Content-Length': '0'} 2022-09-12T20:14:10.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000915724ce9b08afd0-00631f9312-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.863 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:10.864 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:10.864 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:10.865 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:10.865 DEBUG:boto:Method: HEAD 2022-09-12T20:14:10.866 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:10.866 DEBUG:boto:Data: 2022-09-12T20:14:10.867 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.867 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:10.868 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.868 DEBUG:boto:Params: {} 2022-09-12T20:14:10.869 DEBUG:boto:Token: None 2022-09-12T20:14:10.870 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:10.870 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G6yfsiw6s9KmRmd+S27ZpZnCoqc= 2022-09-12T20:14:10.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G6yfsiw6s9KmRmd+S27ZpZnCoqc=', 'Content-Length': '0'} 2022-09-12T20:14:10.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000feccaac2628ebb37-00631f9312-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:14:10.875 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:10.875 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:10.876 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:10.876 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:10.877 DEBUG:boto:Method: GET 2022-09-12T20:14:10.877 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:10.878 DEBUG:boto:Data: 2022-09-12T20:14:10.879 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.879 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:10.880 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.880 DEBUG:boto:Params: {} 2022-09-12T20:14:10.881 DEBUG:boto:Token: None 2022-09-12T20:14:10.882 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:10.882 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k6d1JlUkszJbjr4HLsd5+xEzk2s= 2022-09-12T20:14:10.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k6d1JlUkszJbjr4HLsd5+xEzk2s=', 'Content-Length': '0'} 2022-09-12T20:14:10.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3bc56c254da67de-00631f9312-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.887 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:10.888 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:10.889 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:10.889 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:10.890 DEBUG:boto:Method: GET 2022-09-12T20:14:10.891 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:10.891 DEBUG:boto:Data: 2022-09-12T20:14:10.892 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.892 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:10.893 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.894 DEBUG:boto:Params: {} 2022-09-12T20:14:10.894 DEBUG:boto:Token: None 2022-09-12T20:14:10.895 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:10.896 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k6d1JlUkszJbjr4HLsd5+xEzk2s= 2022-09-12T20:14:10.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k6d1JlUkszJbjr4HLsd5+xEzk2s=', 'Content-Length': '0'} 2022-09-12T20:14:10.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000124bfed7ad604535-00631f9312-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.899 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:10.900 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:10.900 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:14:10.901 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:14:10.901 DEBUG:boto:Method: GET 2022-09-12T20:14:10.902 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:14:10.903 DEBUG:boto:Data: 2022-09-12T20:14:10.903 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.904 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:10.904 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.905 DEBUG:boto:Params: {} 2022-09-12T20:14:10.905 DEBUG:boto:Token: None 2022-09-12T20:14:10.906 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/ 2022-09-12T20:14:10.907 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ= 2022-09-12T20:14:10.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ=', 'Content-Length': '0'} 2022-09-12T20:14:10.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032109700f611dff0-00631f9312-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.954 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:10.955 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:10.956 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:10.956 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:14:10.957 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:14:10.957 DEBUG:boto:Method: GET 2022-09-12T20:14:10.958 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:14:10.958 DEBUG:boto:Data: 2022-09-12T20:14:10.959 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.959 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:10.960 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.961 DEBUG:boto:Params: {} 2022-09-12T20:14:10.961 DEBUG:boto:Token: None 2022-09-12T20:14:10.962 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/ 2022-09-12T20:14:10.962 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ= 2022-09-12T20:14:10.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SV7ryw0XJlOkYGIGr+U5w+d5+rQ=', 'Content-Length': '0'} 2022-09-12T20:14:10.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e73ecc417ab04ec4-00631f9312-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.968 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:10.969 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:10.969 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:10.970 DEBUG:boto:Method: HEAD 2022-09-12T20:14:10.971 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:14:10.971 DEBUG:boto:Data: 2022-09-12T20:14:10.972 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.972 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:10.973 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.973 DEBUG:boto:Params: {} 2022-09-12T20:14:10.974 DEBUG:boto:Token: None 2022-09-12T20:14:10.974 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj68 2022-09-12T20:14:10.975 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iWmal5qaiaxzK8Tyl4BH90ar0ZQ= 2022-09-12T20:14:10.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iWmal5qaiaxzK8Tyl4BH90ar0ZQ=', 'Content-Length': '0'} 2022-09-12T20:14:10.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c78d141be0153f9f-00631f9312-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.978 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:10.979 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:10.980 DEBUG:boto:Method: HEAD 2022-09-12T20:14:10.980 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:14:10.981 DEBUG:boto:Data: 2022-09-12T20:14:10.981 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.982 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:10.982 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.983 DEBUG:boto:Params: {} 2022-09-12T20:14:10.983 DEBUG:boto:Token: None 2022-09-12T20:14:10.984 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj68 2022-09-12T20:14:10.984 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iWmal5qaiaxzK8Tyl4BH90ar0ZQ= 2022-09-12T20:14:10.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iWmal5qaiaxzK8Tyl4BH90ar0ZQ=', 'Content-Length': '0'} 2022-09-12T20:14:10.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abe8ba0448788ead-00631f9312-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:14:10.988 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:10.989 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:10.990 DEBUG:boto:Method: GET 2022-09-12T20:14:10.990 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:14:10.991 DEBUG:boto:Data: 2022-09-12T20:14:10.991 DEBUG:boto:Headers: {} 2022-09-12T20:14:10.992 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:10.992 DEBUG:boto:Port: 8000 2022-09-12T20:14:10.993 DEBUG:boto:Params: {} 2022-09-12T20:14:10.994 DEBUG:boto:Token: None 2022-09-12T20:14:10.994 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:10 GMT /jdpiak-75/obj68 2022-09-12T20:14:10.995 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JVr1XhLWQwPfGOJdMvuHJjl1pvA= 2022-09-12T20:14:10.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JVr1XhLWQwPfGOJdMvuHJjl1pvA=', 'Content-Length': '0'} 2022-09-12T20:14:10.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a964ee089cfde2e6-00631f9312-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:10.998 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:10.999 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:10.999 DEBUG:boto:Method: GET 2022-09-12T20:14:11.000 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:14:11.001 DEBUG:boto:Data: 2022-09-12T20:14:11.001 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.002 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.002 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.003 DEBUG:boto:Params: {} 2022-09-12T20:14:11.004 DEBUG:boto:Token: None 2022-09-12T20:14:11.004 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj68 2022-09-12T20:14:11.005 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yVobp11kDgN9E2raXlfGCb1yyLM= 2022-09-12T20:14:11.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yVobp11kDgN9E2raXlfGCb1yyLM=', 'Content-Length': '0'} 2022-09-12T20:14:11.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018fffa61371e9ffa-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.050 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:11.051 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:11.051 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:14:11.052 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:14:11.052 DEBUG:boto:Method: GET 2022-09-12T20:14:11.053 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:14:11.053 DEBUG:boto:Data: 2022-09-12T20:14:11.054 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.055 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.055 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.056 DEBUG:boto:Params: {} 2022-09-12T20:14:11.056 DEBUG:boto:Token: None 2022-09-12T20:14:11.057 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/ 2022-09-12T20:14:11.057 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc= 2022-09-12T20:14:11.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc=', 'Content-Length': '0'} 2022-09-12T20:14:11.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df839077eef8a071-00631f9313-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.061 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:11.062 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:11.063 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:11.064 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:14:11.064 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:14:11.065 DEBUG:boto:Method: GET 2022-09-12T20:14:11.065 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:14:11.066 DEBUG:boto:Data: 2022-09-12T20:14:11.066 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.067 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.067 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.068 DEBUG:boto:Params: {} 2022-09-12T20:14:11.069 DEBUG:boto:Token: None 2022-09-12T20:14:11.069 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/ 2022-09-12T20:14:11.070 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc= 2022-09-12T20:14:11.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc=', 'Content-Length': '0'} 2022-09-12T20:14:11.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005340f5224e38df77-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.076 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:11.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:14:11.077 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:11.078 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:11.078 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:11.079 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:11.080 DEBUG:boto:Method: GET 2022-09-12T20:14:11.080 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:11.081 DEBUG:boto:Data: 2022-09-12T20:14:11.081 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.082 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.082 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.083 DEBUG:boto:Params: {} 2022-09-12T20:14:11.083 DEBUG:boto:Token: None 2022-09-12T20:14:11.084 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:11.084 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ObD8G0LySsU5XIdwHFpte8P/EKc= 2022-09-12T20:14:11.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ObD8G0LySsU5XIdwHFpte8P/EKc=', 'Content-Length': '0'} 2022-09-12T20:14:11.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ead639a601705f00-00631f9313-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.088 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:11.089 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:11.089 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:11.090 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:11.090 DEBUG:boto:Method: GET 2022-09-12T20:14:11.091 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:11.092 DEBUG:boto:Data: 2022-09-12T20:14:11.092 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.093 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.093 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.094 DEBUG:boto:Params: {} 2022-09-12T20:14:11.094 DEBUG:boto:Token: None 2022-09-12T20:14:11.095 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:11.096 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ObD8G0LySsU5XIdwHFpte8P/EKc= 2022-09-12T20:14:11.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ObD8G0LySsU5XIdwHFpte8P/EKc=', 'Content-Length': '0'} 2022-09-12T20:14:11.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2cb6c0967192de5-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.099 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:11.100 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:11.100 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:11.101 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:11.102 DEBUG:boto:Method: HEAD 2022-09-12T20:14:11.102 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:11.103 DEBUG:boto:Data: 2022-09-12T20:14:11.103 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.104 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.104 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.105 DEBUG:boto:Params: {} 2022-09-12T20:14:11.105 DEBUG:boto:Token: None 2022-09-12T20:14:11.106 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:11.106 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QnsmW7eeEffxYFtScOPJgRhIffc= 2022-09-12T20:14:11.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QnsmW7eeEffxYFtScOPJgRhIffc=', 'Content-Length': '0'} 2022-09-12T20:14:11.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a373dee58aa5593-00631f9313-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.110 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:11.110 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:11.111 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:11.111 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:11.112 DEBUG:boto:Method: HEAD 2022-09-12T20:14:11.112 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:11.113 DEBUG:boto:Data: 2022-09-12T20:14:11.114 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.114 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.115 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.115 DEBUG:boto:Params: {} 2022-09-12T20:14:11.116 DEBUG:boto:Token: None 2022-09-12T20:14:11.116 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:11.117 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QnsmW7eeEffxYFtScOPJgRhIffc= 2022-09-12T20:14:11.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QnsmW7eeEffxYFtScOPJgRhIffc=', 'Content-Length': '0'} 2022-09-12T20:14:11.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc3a11779fbd498b-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:14:11.121 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:11.121 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:11.122 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:11.122 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:11.123 DEBUG:boto:Method: GET 2022-09-12T20:14:11.123 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:11.124 DEBUG:boto:Data: 2022-09-12T20:14:11.124 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.125 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.126 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.126 DEBUG:boto:Params: {} 2022-09-12T20:14:11.127 DEBUG:boto:Token: None 2022-09-12T20:14:11.128 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:11.128 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ObD8G0LySsU5XIdwHFpte8P/EKc= 2022-09-12T20:14:11.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ObD8G0LySsU5XIdwHFpte8P/EKc=', 'Content-Length': '0'} 2022-09-12T20:14:11.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008aadff6e20037996-00631f9313-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.132 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:11.132 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:11.133 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:11.133 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:11.134 DEBUG:boto:Method: GET 2022-09-12T20:14:11.134 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:11.135 DEBUG:boto:Data: 2022-09-12T20:14:11.135 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.136 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.137 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.137 DEBUG:boto:Params: {} 2022-09-12T20:14:11.138 DEBUG:boto:Token: None 2022-09-12T20:14:11.139 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:11.139 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ObD8G0LySsU5XIdwHFpte8P/EKc= 2022-09-12T20:14:11.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ObD8G0LySsU5XIdwHFpte8P/EKc=', 'Content-Length': '0'} 2022-09-12T20:14:11.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c21107e6cffa93ad-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.186 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:11.187 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:11.187 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:14:11.188 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:14:11.188 DEBUG:boto:Method: GET 2022-09-12T20:14:11.189 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:14:11.189 DEBUG:boto:Data: 2022-09-12T20:14:11.190 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.190 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.191 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.191 DEBUG:boto:Params: {} 2022-09-12T20:14:11.192 DEBUG:boto:Token: None 2022-09-12T20:14:11.192 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/ 2022-09-12T20:14:11.193 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc= 2022-09-12T20:14:11.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc=', 'Content-Length': '0'} 2022-09-12T20:14:11.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f02a0bc67031c35c-00631f9313-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.197 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:11.198 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:11.198 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:11.199 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:14:11.199 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:14:11.200 DEBUG:boto:Method: GET 2022-09-12T20:14:11.200 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:14:11.201 DEBUG:boto:Data: 2022-09-12T20:14:11.201 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.202 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.202 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.203 DEBUG:boto:Params: {} 2022-09-12T20:14:11.204 DEBUG:boto:Token: None 2022-09-12T20:14:11.204 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/ 2022-09-12T20:14:11.205 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc= 2022-09-12T20:14:11.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc=', 'Content-Length': '0'} 2022-09-12T20:14:11.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061e0c3a64f182c9f-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.211 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:11.211 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:11.212 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:11.212 DEBUG:boto:Method: HEAD 2022-09-12T20:14:11.213 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:14:11.213 DEBUG:boto:Data: 2022-09-12T20:14:11.214 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.214 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.215 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.215 DEBUG:boto:Params: {} 2022-09-12T20:14:11.216 DEBUG:boto:Token: None 2022-09-12T20:14:11.216 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj69 2022-09-12T20:14:11.217 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9nXgr6KCrVos+eRZCvU5aaScN0U= 2022-09-12T20:14:11.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9nXgr6KCrVos+eRZCvU5aaScN0U=', 'Content-Length': '0'} 2022-09-12T20:14:11.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001937f9f76f18438f-00631f9313-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.220 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:11.221 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:11.221 DEBUG:boto:Method: HEAD 2022-09-12T20:14:11.222 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:14:11.222 DEBUG:boto:Data: 2022-09-12T20:14:11.223 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.223 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.224 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.224 DEBUG:boto:Params: {} 2022-09-12T20:14:11.225 DEBUG:boto:Token: None 2022-09-12T20:14:11.225 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj69 2022-09-12T20:14:11.226 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9nXgr6KCrVos+eRZCvU5aaScN0U= 2022-09-12T20:14:11.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9nXgr6KCrVos+eRZCvU5aaScN0U=', 'Content-Length': '0'} 2022-09-12T20:14:11.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009972ba8a10a0f2d4-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:14:11.230 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:11.230 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:11.231 DEBUG:boto:Method: GET 2022-09-12T20:14:11.231 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:14:11.232 DEBUG:boto:Data: 2022-09-12T20:14:11.232 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.233 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.233 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.234 DEBUG:boto:Params: {} 2022-09-12T20:14:11.234 DEBUG:boto:Token: None 2022-09-12T20:14:11.235 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj69 2022-09-12T20:14:11.235 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bQ8CK+CmpdP8VS4LMrOGxdWfJHU= 2022-09-12T20:14:11.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bQ8CK+CmpdP8VS4LMrOGxdWfJHU=', 'Content-Length': '0'} 2022-09-12T20:14:11.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000785c4fcb8785f706-00631f9313-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.239 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:11.239 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:11.240 DEBUG:boto:Method: GET 2022-09-12T20:14:11.240 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:14:11.241 DEBUG:boto:Data: 2022-09-12T20:14:11.242 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.242 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.243 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.243 DEBUG:boto:Params: {} 2022-09-12T20:14:11.244 DEBUG:boto:Token: None 2022-09-12T20:14:11.244 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj69 2022-09-12T20:14:11.245 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bQ8CK+CmpdP8VS4LMrOGxdWfJHU= 2022-09-12T20:14:11.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bQ8CK+CmpdP8VS4LMrOGxdWfJHU=', 'Content-Length': '0'} 2022-09-12T20:14:11.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019a9144cf043a13a-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.249 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:11.249 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:11.250 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:14:11.250 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:14:11.251 DEBUG:boto:Method: GET 2022-09-12T20:14:11.251 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:14:11.252 DEBUG:boto:Data: 2022-09-12T20:14:11.252 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.253 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.253 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.254 DEBUG:boto:Params: {} 2022-09-12T20:14:11.254 DEBUG:boto:Token: None 2022-09-12T20:14:11.255 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/ 2022-09-12T20:14:11.255 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc= 2022-09-12T20:14:11.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc=', 'Content-Length': '0'} 2022-09-12T20:14:11.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c5d7ebd8ae3009d-00631f9313-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.260 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:11.260 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:11.261 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:11.261 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:14:11.262 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:14:11.262 DEBUG:boto:Method: GET 2022-09-12T20:14:11.263 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:14:11.263 DEBUG:boto:Data: 2022-09-12T20:14:11.264 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.264 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.265 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.265 DEBUG:boto:Params: {} 2022-09-12T20:14:11.266 DEBUG:boto:Token: None 2022-09-12T20:14:11.267 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/ 2022-09-12T20:14:11.267 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc= 2022-09-12T20:14:11.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc=', 'Content-Length': '0'} 2022-09-12T20:14:11.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000680bc81e9ad12748-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.273 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:11.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:14:11.274 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:11.274 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:11.275 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:11.275 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:11.276 DEBUG:boto:Method: GET 2022-09-12T20:14:11.276 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:11.277 DEBUG:boto:Data: 2022-09-12T20:14:11.277 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.278 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.279 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.279 DEBUG:boto:Params: {} 2022-09-12T20:14:11.280 DEBUG:boto:Token: None 2022-09-12T20:14:11.280 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:11.281 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1QyElXmA8duo2dG9K3XKmgDF+w0= 2022-09-12T20:14:11.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1QyElXmA8duo2dG9K3XKmgDF+w0=', 'Content-Length': '0'} 2022-09-12T20:14:11.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da22430096340fe1-00631f9313-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.326 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:11.327 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:11.327 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:11.328 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:11.328 DEBUG:boto:Method: GET 2022-09-12T20:14:11.329 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:11.329 DEBUG:boto:Data: 2022-09-12T20:14:11.330 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.330 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.331 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.332 DEBUG:boto:Params: {} 2022-09-12T20:14:11.332 DEBUG:boto:Token: None 2022-09-12T20:14:11.333 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:11.333 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1QyElXmA8duo2dG9K3XKmgDF+w0= 2022-09-12T20:14:11.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1QyElXmA8duo2dG9K3XKmgDF+w0=', 'Content-Length': '0'} 2022-09-12T20:14:11.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bef7698e4e8b3fc3-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.337 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:11.337 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:11.338 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:11.338 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:11.339 DEBUG:boto:Method: HEAD 2022-09-12T20:14:11.339 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:11.340 DEBUG:boto:Data: 2022-09-12T20:14:11.340 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.341 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.341 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.342 DEBUG:boto:Params: {} 2022-09-12T20:14:11.342 DEBUG:boto:Token: None 2022-09-12T20:14:11.343 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:11.343 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ltuKKTkvo3ouD2pLcIbJZ+lCOMg= 2022-09-12T20:14:11.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ltuKKTkvo3ouD2pLcIbJZ+lCOMg=', 'Content-Length': '0'} 2022-09-12T20:14:11.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000707536fb7d0652cb-00631f9313-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.347 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:11.347 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:11.348 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:11.348 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:11.349 DEBUG:boto:Method: HEAD 2022-09-12T20:14:11.349 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:11.350 DEBUG:boto:Data: 2022-09-12T20:14:11.350 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.351 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.351 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.352 DEBUG:boto:Params: {} 2022-09-12T20:14:11.352 DEBUG:boto:Token: None 2022-09-12T20:14:11.353 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:11.353 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ltuKKTkvo3ouD2pLcIbJZ+lCOMg= 2022-09-12T20:14:11.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ltuKKTkvo3ouD2pLcIbJZ+lCOMg=', 'Content-Length': '0'} 2022-09-12T20:14:11.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e041b3c7720084af-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:14:11.357 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:11.358 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:11.358 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:11.359 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:11.360 DEBUG:boto:Method: GET 2022-09-12T20:14:11.360 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:11.361 DEBUG:boto:Data: 2022-09-12T20:14:11.361 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.362 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.362 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.363 DEBUG:boto:Params: {} 2022-09-12T20:14:11.363 DEBUG:boto:Token: None 2022-09-12T20:14:11.364 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:11.364 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1QyElXmA8duo2dG9K3XKmgDF+w0= 2022-09-12T20:14:11.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1QyElXmA8duo2dG9K3XKmgDF+w0=', 'Content-Length': '0'} 2022-09-12T20:14:11.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000002ca41664def22f-00631f9313-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.368 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:11.368 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:11.369 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:11.369 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:11.370 DEBUG:boto:Method: GET 2022-09-12T20:14:11.370 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:11.371 DEBUG:boto:Data: 2022-09-12T20:14:11.371 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.372 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.372 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.373 DEBUG:boto:Params: {} 2022-09-12T20:14:11.373 DEBUG:boto:Token: None 2022-09-12T20:14:11.374 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:11.375 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1QyElXmA8duo2dG9K3XKmgDF+w0= 2022-09-12T20:14:11.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1QyElXmA8duo2dG9K3XKmgDF+w0=', 'Content-Length': '0'} 2022-09-12T20:14:11.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000777f6ad6be82a9f7-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.378 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:11.379 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:11.379 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:14:11.380 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:14:11.380 DEBUG:boto:Method: GET 2022-09-12T20:14:11.381 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:14:11.381 DEBUG:boto:Data: 2022-09-12T20:14:11.382 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.382 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.383 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.383 DEBUG:boto:Params: {} 2022-09-12T20:14:11.384 DEBUG:boto:Token: None 2022-09-12T20:14:11.384 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/ 2022-09-12T20:14:11.385 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc= 2022-09-12T20:14:11.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc=', 'Content-Length': '0'} 2022-09-12T20:14:11.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007a2e83411ea0483-00631f9313-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.389 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:11.390 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:11.390 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:11.391 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:14:11.392 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:14:11.392 DEBUG:boto:Method: GET 2022-09-12T20:14:11.393 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:14:11.393 DEBUG:boto:Data: 2022-09-12T20:14:11.394 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.394 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.395 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.395 DEBUG:boto:Params: {} 2022-09-12T20:14:11.396 DEBUG:boto:Token: None 2022-09-12T20:14:11.396 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/ 2022-09-12T20:14:11.397 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc= 2022-09-12T20:14:11.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc=', 'Content-Length': '0'} 2022-09-12T20:14:11.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000356c122bbe8a7ed0-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.403 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:11.403 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:11.404 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:11.404 DEBUG:boto:Method: HEAD 2022-09-12T20:14:11.405 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:14:11.405 DEBUG:boto:Data: 2022-09-12T20:14:11.406 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.406 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.407 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.407 DEBUG:boto:Params: {} 2022-09-12T20:14:11.408 DEBUG:boto:Token: None 2022-09-12T20:14:11.408 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj76 2022-09-12T20:14:11.409 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PlRtdW0kEURstU7QsFi9Rr1yoXQ= 2022-09-12T20:14:11.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PlRtdW0kEURstU7QsFi9Rr1yoXQ=', 'Content-Length': '0'} 2022-09-12T20:14:11.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002428e4d5e0ec0e99-00631f9313-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.412 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:11.413 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:11.413 DEBUG:boto:Method: HEAD 2022-09-12T20:14:11.414 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:14:11.414 DEBUG:boto:Data: 2022-09-12T20:14:11.415 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.415 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.416 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.416 DEBUG:boto:Params: {} 2022-09-12T20:14:11.417 DEBUG:boto:Token: None 2022-09-12T20:14:11.417 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj76 2022-09-12T20:14:11.418 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PlRtdW0kEURstU7QsFi9Rr1yoXQ= 2022-09-12T20:14:11.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PlRtdW0kEURstU7QsFi9Rr1yoXQ=', 'Content-Length': '0'} 2022-09-12T20:14:11.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000176a2d5819e57d1a-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:14:11.422 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:11.422 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:11.423 DEBUG:boto:Method: GET 2022-09-12T20:14:11.423 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:14:11.424 DEBUG:boto:Data: 2022-09-12T20:14:11.424 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.425 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.425 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.426 DEBUG:boto:Params: {} 2022-09-12T20:14:11.427 DEBUG:boto:Token: None 2022-09-12T20:14:11.427 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj76 2022-09-12T20:14:11.428 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HoBssIRyg/+kz3/LBnGa8zUovII= 2022-09-12T20:14:11.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HoBssIRyg/+kz3/LBnGa8zUovII=', 'Content-Length': '0'} 2022-09-12T20:14:11.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000073d9aa61df1ef0f-00631f9313-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.431 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:11.432 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:11.432 DEBUG:boto:Method: GET 2022-09-12T20:14:11.433 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:14:11.433 DEBUG:boto:Data: 2022-09-12T20:14:11.434 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.434 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.435 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.435 DEBUG:boto:Params: {} 2022-09-12T20:14:11.436 DEBUG:boto:Token: None 2022-09-12T20:14:11.436 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj76 2022-09-12T20:14:11.437 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HoBssIRyg/+kz3/LBnGa8zUovII= 2022-09-12T20:14:11.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HoBssIRyg/+kz3/LBnGa8zUovII=', 'Content-Length': '0'} 2022-09-12T20:14:11.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c70837c43cecc74c-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.482 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:11.483 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:11.483 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:14:11.484 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:14:11.484 DEBUG:boto:Method: GET 2022-09-12T20:14:11.485 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:14:11.485 DEBUG:boto:Data: 2022-09-12T20:14:11.486 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.486 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.487 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.487 DEBUG:boto:Params: {} 2022-09-12T20:14:11.488 DEBUG:boto:Token: None 2022-09-12T20:14:11.489 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/ 2022-09-12T20:14:11.489 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc= 2022-09-12T20:14:11.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc=', 'Content-Length': '0'} 2022-09-12T20:14:11.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039f79957750ad88c-00631f9313-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.494 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:11.494 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:11.495 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:11.495 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:14:11.496 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:14:11.496 DEBUG:boto:Method: GET 2022-09-12T20:14:11.497 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:14:11.498 DEBUG:boto:Data: 2022-09-12T20:14:11.498 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.499 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.499 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.500 DEBUG:boto:Params: {} 2022-09-12T20:14:11.500 DEBUG:boto:Token: None 2022-09-12T20:14:11.501 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/ 2022-09-12T20:14:11.501 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc= 2022-09-12T20:14:11.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc=', 'Content-Length': '0'} 2022-09-12T20:14:11.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025b478d23fff0cac-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.507 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:11.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:14:11.508 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:11.509 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:11.510 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:11.511 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:11.511 DEBUG:boto:Method: GET 2022-09-12T20:14:11.512 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:11.512 DEBUG:boto:Data: 2022-09-12T20:14:11.513 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.513 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.514 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.514 DEBUG:boto:Params: {} 2022-09-12T20:14:11.515 DEBUG:boto:Token: None 2022-09-12T20:14:11.515 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:11.516 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jpC+1cwuvWPhTOSjIe5Wqv4AsLI= 2022-09-12T20:14:11.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jpC+1cwuvWPhTOSjIe5Wqv4AsLI=', 'Content-Length': '0'} 2022-09-12T20:14:11.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000331a4c7d3abf0c7e-00631f9313-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.520 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:11.520 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:11.521 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:11.521 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:11.522 DEBUG:boto:Method: GET 2022-09-12T20:14:11.522 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:11.523 DEBUG:boto:Data: 2022-09-12T20:14:11.523 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.524 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.524 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.525 DEBUG:boto:Params: {} 2022-09-12T20:14:11.525 DEBUG:boto:Token: None 2022-09-12T20:14:11.526 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:11.526 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jpC+1cwuvWPhTOSjIe5Wqv4AsLI= 2022-09-12T20:14:11.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jpC+1cwuvWPhTOSjIe5Wqv4AsLI=', 'Content-Length': '0'} 2022-09-12T20:14:11.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c7a9078b35d75e7-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.530 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:11.531 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:11.531 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:11.532 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:11.532 DEBUG:boto:Method: HEAD 2022-09-12T20:14:11.533 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:11.533 DEBUG:boto:Data: 2022-09-12T20:14:11.534 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.534 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.535 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.535 DEBUG:boto:Params: {} 2022-09-12T20:14:11.536 DEBUG:boto:Token: None 2022-09-12T20:14:11.536 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:11.537 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DYmzJeuiNVbKkdhTqTuYbutR+Cc= 2022-09-12T20:14:11.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DYmzJeuiNVbKkdhTqTuYbutR+Cc=', 'Content-Length': '0'} 2022-09-12T20:14:11.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c45d9d40aa923fd-00631f9313-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.540 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:11.540 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:11.541 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:11.541 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:11.542 DEBUG:boto:Method: HEAD 2022-09-12T20:14:11.543 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:11.543 DEBUG:boto:Data: 2022-09-12T20:14:11.544 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.544 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.545 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.545 DEBUG:boto:Params: {} 2022-09-12T20:14:11.546 DEBUG:boto:Token: None 2022-09-12T20:14:11.546 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:11.547 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DYmzJeuiNVbKkdhTqTuYbutR+Cc= 2022-09-12T20:14:11.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DYmzJeuiNVbKkdhTqTuYbutR+Cc=', 'Content-Length': '0'} 2022-09-12T20:14:11.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb01d00a9bfee341-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:14:11.551 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:11.551 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:11.552 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:11.552 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:11.553 DEBUG:boto:Method: GET 2022-09-12T20:14:11.553 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:11.554 DEBUG:boto:Data: 2022-09-12T20:14:11.554 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.555 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.556 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.556 DEBUG:boto:Params: {} 2022-09-12T20:14:11.557 DEBUG:boto:Token: None 2022-09-12T20:14:11.557 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:11.558 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jpC+1cwuvWPhTOSjIe5Wqv4AsLI= 2022-09-12T20:14:11.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jpC+1cwuvWPhTOSjIe5Wqv4AsLI=', 'Content-Length': '0'} 2022-09-12T20:14:11.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003df72a21e40256af-00631f9313-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.602 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:11.603 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:11.603 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:11.604 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:11.604 DEBUG:boto:Method: GET 2022-09-12T20:14:11.605 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:11.605 DEBUG:boto:Data: 2022-09-12T20:14:11.606 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.606 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.607 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.607 DEBUG:boto:Params: {} 2022-09-12T20:14:11.608 DEBUG:boto:Token: None 2022-09-12T20:14:11.609 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:11.609 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jpC+1cwuvWPhTOSjIe5Wqv4AsLI= 2022-09-12T20:14:11.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jpC+1cwuvWPhTOSjIe5Wqv4AsLI=', 'Content-Length': '0'} 2022-09-12T20:14:11.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000245549a9a559b00f-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.613 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:11.613 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:11.614 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:14:11.614 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:14:11.615 DEBUG:boto:Method: GET 2022-09-12T20:14:11.615 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:14:11.616 DEBUG:boto:Data: 2022-09-12T20:14:11.616 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.617 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.617 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.618 DEBUG:boto:Params: {} 2022-09-12T20:14:11.619 DEBUG:boto:Token: None 2022-09-12T20:14:11.619 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/ 2022-09-12T20:14:11.620 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc= 2022-09-12T20:14:11.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc=', 'Content-Length': '0'} 2022-09-12T20:14:11.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e556a609a20d437-00631f9313-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.624 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:11.625 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:11.625 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:11.626 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:14:11.626 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:14:11.627 DEBUG:boto:Method: GET 2022-09-12T20:14:11.627 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:14:11.628 DEBUG:boto:Data: 2022-09-12T20:14:11.628 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.629 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.629 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.630 DEBUG:boto:Params: {} 2022-09-12T20:14:11.630 DEBUG:boto:Token: None 2022-09-12T20:14:11.631 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/ 2022-09-12T20:14:11.632 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc= 2022-09-12T20:14:11.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc=', 'Content-Length': '0'} 2022-09-12T20:14:11.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005511cccf32f4c056-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.637 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:11.638 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:11.638 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:11.639 DEBUG:boto:Method: HEAD 2022-09-12T20:14:11.639 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:14:11.640 DEBUG:boto:Data: 2022-09-12T20:14:11.641 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.641 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.641 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.642 DEBUG:boto:Params: {} 2022-09-12T20:14:11.643 DEBUG:boto:Token: None 2022-09-12T20:14:11.643 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj85 2022-09-12T20:14:11.644 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Oer9+DJe57LoUZmLsvQTVOFU6Kk= 2022-09-12T20:14:11.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Oer9+DJe57LoUZmLsvQTVOFU6Kk=', 'Content-Length': '0'} 2022-09-12T20:14:11.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e919bbb38c805a5-00631f9313-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.647 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:11.647 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:11.648 DEBUG:boto:Method: HEAD 2022-09-12T20:14:11.648 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:14:11.654 DEBUG:boto:Data: 2022-09-12T20:14:11.655 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.655 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.656 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.656 DEBUG:boto:Params: {} 2022-09-12T20:14:11.657 DEBUG:boto:Token: None 2022-09-12T20:14:11.657 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj85 2022-09-12T20:14:11.658 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Oer9+DJe57LoUZmLsvQTVOFU6Kk= 2022-09-12T20:14:11.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Oer9+DJe57LoUZmLsvQTVOFU6Kk=', 'Content-Length': '0'} 2022-09-12T20:14:11.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061cac65e08211db9-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:14:11.662 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:11.662 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:11.663 DEBUG:boto:Method: GET 2022-09-12T20:14:11.663 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:14:11.664 DEBUG:boto:Data: 2022-09-12T20:14:11.664 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.665 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.665 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.666 DEBUG:boto:Params: {} 2022-09-12T20:14:11.666 DEBUG:boto:Token: None 2022-09-12T20:14:11.667 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj85 2022-09-12T20:14:11.668 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6Ht742jLbuk+Q0EA20XBZnjjKsA= 2022-09-12T20:14:11.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6Ht742jLbuk+Q0EA20XBZnjjKsA=', 'Content-Length': '0'} 2022-09-12T20:14:11.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8b52a643912cb44-00631f9313-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.714 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:11.715 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:11.715 DEBUG:boto:Method: GET 2022-09-12T20:14:11.716 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:14:11.716 DEBUG:boto:Data: 2022-09-12T20:14:11.717 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.717 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.718 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.719 DEBUG:boto:Params: {} 2022-09-12T20:14:11.719 DEBUG:boto:Token: None 2022-09-12T20:14:11.720 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj85 2022-09-12T20:14:11.720 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6Ht742jLbuk+Q0EA20XBZnjjKsA= 2022-09-12T20:14:11.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6Ht742jLbuk+Q0EA20XBZnjjKsA=', 'Content-Length': '0'} 2022-09-12T20:14:11.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb63fbd6b4c95316-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.724 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:11.725 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:11.725 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:14:11.726 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:14:11.726 DEBUG:boto:Method: GET 2022-09-12T20:14:11.727 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:14:11.727 DEBUG:boto:Data: 2022-09-12T20:14:11.728 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.728 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.729 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.729 DEBUG:boto:Params: {} 2022-09-12T20:14:11.730 DEBUG:boto:Token: None 2022-09-12T20:14:11.731 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/ 2022-09-12T20:14:11.732 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc= 2022-09-12T20:14:11.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc=', 'Content-Length': '0'} 2022-09-12T20:14:11.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcfe5e05cba81560-00631f9313-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.737 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:11.738 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:11.738 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:11.739 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:14:11.739 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:14:11.740 DEBUG:boto:Method: GET 2022-09-12T20:14:11.740 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:14:11.741 DEBUG:boto:Data: 2022-09-12T20:14:11.741 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.742 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.743 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.743 DEBUG:boto:Params: {} 2022-09-12T20:14:11.744 DEBUG:boto:Token: None 2022-09-12T20:14:11.745 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/ 2022-09-12T20:14:11.746 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc= 2022-09-12T20:14:11.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc=', 'Content-Length': '0'} 2022-09-12T20:14:11.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a1a36feb5b9219b-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.752 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:11.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:14:11.753 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:11.754 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:11.755 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:11.755 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:11.756 DEBUG:boto:Method: GET 2022-09-12T20:14:11.756 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:11.757 DEBUG:boto:Data: 2022-09-12T20:14:11.758 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.758 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.759 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.759 DEBUG:boto:Params: {} 2022-09-12T20:14:11.760 DEBUG:boto:Token: None 2022-09-12T20:14:11.760 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:11.761 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m3ifdx/4mM3gHA1YQQ9TzpG7Q6A= 2022-09-12T20:14:11.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m3ifdx/4mM3gHA1YQQ9TzpG7Q6A=', 'Content-Length': '0'} 2022-09-12T20:14:11.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1a5668f1d4e2974-00631f9313-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.765 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:11.765 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:11.766 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:11.766 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:11.767 DEBUG:boto:Method: GET 2022-09-12T20:14:11.767 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:11.768 DEBUG:boto:Data: 2022-09-12T20:14:11.768 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.769 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.769 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.770 DEBUG:boto:Params: {} 2022-09-12T20:14:11.770 DEBUG:boto:Token: None 2022-09-12T20:14:11.771 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:11.772 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m3ifdx/4mM3gHA1YQQ9TzpG7Q6A= 2022-09-12T20:14:11.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m3ifdx/4mM3gHA1YQQ9TzpG7Q6A=', 'Content-Length': '0'} 2022-09-12T20:14:11.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a62113f972cb2a37-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.775 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:11.776 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:11.776 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:11.777 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:11.777 DEBUG:boto:Method: HEAD 2022-09-12T20:14:11.778 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:11.778 DEBUG:boto:Data: 2022-09-12T20:14:11.779 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.779 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.780 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.780 DEBUG:boto:Params: {} 2022-09-12T20:14:11.781 DEBUG:boto:Token: None 2022-09-12T20:14:11.781 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:11.782 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4VJzW+99fMaV653ZW36Xwo8FBiQ= 2022-09-12T20:14:11.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4VJzW+99fMaV653ZW36Xwo8FBiQ=', 'Content-Length': '0'} 2022-09-12T20:14:11.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebe72073766e3fff-00631f9313-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.785 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:11.786 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:11.786 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:11.787 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:11.787 DEBUG:boto:Method: HEAD 2022-09-12T20:14:11.788 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:11.788 DEBUG:boto:Data: 2022-09-12T20:14:11.789 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.789 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.790 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.790 DEBUG:boto:Params: {} 2022-09-12T20:14:11.791 DEBUG:boto:Token: None 2022-09-12T20:14:11.792 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:11.792 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4VJzW+99fMaV653ZW36Xwo8FBiQ= 2022-09-12T20:14:11.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4VJzW+99fMaV653ZW36Xwo8FBiQ=', 'Content-Length': '0'} 2022-09-12T20:14:11.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9d5c103b991f507-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:14:11.796 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:11.797 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:11.797 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:11.798 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:11.798 DEBUG:boto:Method: GET 2022-09-12T20:14:11.799 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:11.800 DEBUG:boto:Data: 2022-09-12T20:14:11.800 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.801 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.801 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.802 DEBUG:boto:Params: {} 2022-09-12T20:14:11.802 DEBUG:boto:Token: None 2022-09-12T20:14:11.803 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:11.804 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m3ifdx/4mM3gHA1YQQ9TzpG7Q6A= 2022-09-12T20:14:11.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m3ifdx/4mM3gHA1YQQ9TzpG7Q6A=', 'Content-Length': '0'} 2022-09-12T20:14:11.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099519bee1464459a-00631f9313-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.807 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:11.808 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:11.808 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:11.809 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:11.809 DEBUG:boto:Method: GET 2022-09-12T20:14:11.810 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:11.810 DEBUG:boto:Data: 2022-09-12T20:14:11.811 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.811 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.812 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.812 DEBUG:boto:Params: {} 2022-09-12T20:14:11.813 DEBUG:boto:Token: None 2022-09-12T20:14:11.814 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:11.814 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m3ifdx/4mM3gHA1YQQ9TzpG7Q6A= 2022-09-12T20:14:11.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m3ifdx/4mM3gHA1YQQ9TzpG7Q6A=', 'Content-Length': '0'} 2022-09-12T20:14:11.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cad2abcd6ed3326f-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.858 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:11.859 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:11.859 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:14:11.860 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:14:11.860 DEBUG:boto:Method: GET 2022-09-12T20:14:11.861 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:14:11.861 DEBUG:boto:Data: 2022-09-12T20:14:11.862 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.862 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.863 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.863 DEBUG:boto:Params: {} 2022-09-12T20:14:11.864 DEBUG:boto:Token: None 2022-09-12T20:14:11.865 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/ 2022-09-12T20:14:11.865 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc= 2022-09-12T20:14:11.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc=', 'Content-Length': '0'} 2022-09-12T20:14:11.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b94804b6693d8b7e-00631f9313-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.869 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:11.870 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:11.870 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:11.871 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:14:11.871 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:14:11.872 DEBUG:boto:Method: GET 2022-09-12T20:14:11.872 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:14:11.873 DEBUG:boto:Data: 2022-09-12T20:14:11.873 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.874 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.875 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.875 DEBUG:boto:Params: {} 2022-09-12T20:14:11.876 DEBUG:boto:Token: None 2022-09-12T20:14:11.876 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/ 2022-09-12T20:14:11.877 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc= 2022-09-12T20:14:11.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc=', 'Content-Length': '0'} 2022-09-12T20:14:11.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1f817fee6013073-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.883 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:11.884 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:11.884 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:11.885 DEBUG:boto:Method: HEAD 2022-09-12T20:14:11.886 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:14:11.886 DEBUG:boto:Data: 2022-09-12T20:14:11.887 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.888 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.888 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.889 DEBUG:boto:Params: {} 2022-09-12T20:14:11.889 DEBUG:boto:Token: None 2022-09-12T20:14:11.890 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj87 2022-09-12T20:14:11.890 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:C8sy7wn5RmqYOmDYUJ+dblL/VXQ= 2022-09-12T20:14:11.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:C8sy7wn5RmqYOmDYUJ+dblL/VXQ=', 'Content-Length': '0'} 2022-09-12T20:14:11.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d2ec3f1ddcbe6b9-00631f9313-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.894 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:11.894 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:11.895 DEBUG:boto:Method: HEAD 2022-09-12T20:14:11.895 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:14:11.896 DEBUG:boto:Data: 2022-09-12T20:14:11.896 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.897 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.897 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.898 DEBUG:boto:Params: {} 2022-09-12T20:14:11.898 DEBUG:boto:Token: None 2022-09-12T20:14:11.899 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj87 2022-09-12T20:14:11.899 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:C8sy7wn5RmqYOmDYUJ+dblL/VXQ= 2022-09-12T20:14:11.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:C8sy7wn5RmqYOmDYUJ+dblL/VXQ=', 'Content-Length': '0'} 2022-09-12T20:14:11.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7008ed34f429e9a-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:14:11.903 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:11.904 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:11.904 DEBUG:boto:Method: GET 2022-09-12T20:14:11.905 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:14:11.905 DEBUG:boto:Data: 2022-09-12T20:14:11.906 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.906 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.907 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.908 DEBUG:boto:Params: {} 2022-09-12T20:14:11.908 DEBUG:boto:Token: None 2022-09-12T20:14:11.909 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj87 2022-09-12T20:14:11.909 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J5l+VNtYw14+xcbFcmFRUcWOnLw= 2022-09-12T20:14:11.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J5l+VNtYw14+xcbFcmFRUcWOnLw=', 'Content-Length': '0'} 2022-09-12T20:14:11.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023c6fe6eabb8bf6f-00631f9313-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.913 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:11.913 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:11.914 DEBUG:boto:Method: GET 2022-09-12T20:14:11.914 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:14:11.915 DEBUG:boto:Data: 2022-09-12T20:14:11.915 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.916 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.916 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.917 DEBUG:boto:Params: {} 2022-09-12T20:14:11.917 DEBUG:boto:Token: None 2022-09-12T20:14:11.918 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj87 2022-09-12T20:14:11.919 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J5l+VNtYw14+xcbFcmFRUcWOnLw= 2022-09-12T20:14:11.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J5l+VNtYw14+xcbFcmFRUcWOnLw=', 'Content-Length': '0'} 2022-09-12T20:14:11.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011bbf5964f0154d8-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.962 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:11.963 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:11.963 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:14:11.964 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:14:11.965 DEBUG:boto:Method: GET 2022-09-12T20:14:11.965 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:14:11.966 DEBUG:boto:Data: 2022-09-12T20:14:11.967 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.967 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.968 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.969 DEBUG:boto:Params: {} 2022-09-12T20:14:11.969 DEBUG:boto:Token: None 2022-09-12T20:14:11.970 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/ 2022-09-12T20:14:11.970 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc= 2022-09-12T20:14:11.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc=', 'Content-Length': '0'} 2022-09-12T20:14:11.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acd973ed3d3087fe-00631f9313-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.974 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:11.975 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:11.976 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:11.976 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:14:11.977 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:14:11.977 DEBUG:boto:Method: GET 2022-09-12T20:14:11.978 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:14:11.978 DEBUG:boto:Data: 2022-09-12T20:14:11.979 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.979 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:11.980 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.981 DEBUG:boto:Params: {} 2022-09-12T20:14:11.981 DEBUG:boto:Token: None 2022-09-12T20:14:11.982 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/ 2022-09-12T20:14:11.982 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc= 2022-09-12T20:14:11.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qSvTT8osMzRRnKtj4W0eX3wXqVc=', 'Content-Length': '0'} 2022-09-12T20:14:11.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004baab2a9b8c5b5b4-00631f9313-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:11.988 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:11.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:14:11.989 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:11.990 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:11.990 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:11.991 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:11.992 DEBUG:boto:Method: GET 2022-09-12T20:14:11.992 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:11.993 DEBUG:boto:Data: 2022-09-12T20:14:11.993 DEBUG:boto:Headers: {} 2022-09-12T20:14:11.994 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:11.994 DEBUG:boto:Port: 8000 2022-09-12T20:14:11.995 DEBUG:boto:Params: {} 2022-09-12T20:14:11.995 DEBUG:boto:Token: None 2022-09-12T20:14:11.996 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:11 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:11.997 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LAVwXZ7LcHMzeUtPXZ8huh3sa/8= 2022-09-12T20:14:11.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LAVwXZ7LcHMzeUtPXZ8huh3sa/8=', 'Content-Length': '0'} 2022-09-12T20:14:12.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001203a4f01274d180-00631f9313-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.001 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:12.001 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:12.002 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:12.002 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:12.003 DEBUG:boto:Method: GET 2022-09-12T20:14:12.003 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:12.004 DEBUG:boto:Data: 2022-09-12T20:14:12.004 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.005 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.005 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.006 DEBUG:boto:Params: {} 2022-09-12T20:14:12.007 DEBUG:boto:Token: None 2022-09-12T20:14:12.007 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:12.008 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L0cKRbsCR1Qsb3VICGH7X09tWPA= 2022-09-12T20:14:12.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L0cKRbsCR1Qsb3VICGH7X09tWPA=', 'Content-Length': '0'} 2022-09-12T20:14:12.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cff139290bfde6a2-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.011 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:12.011 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:12.012 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:12.012 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:12.013 DEBUG:boto:Method: HEAD 2022-09-12T20:14:12.013 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:12.014 DEBUG:boto:Data: 2022-09-12T20:14:12.014 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.015 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:12.015 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.016 DEBUG:boto:Params: {} 2022-09-12T20:14:12.016 DEBUG:boto:Token: None 2022-09-12T20:14:12.017 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:12.018 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ptWVIruiR9ti40r4xuktOXFoKx4= 2022-09-12T20:14:12.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ptWVIruiR9ti40r4xuktOXFoKx4=', 'Content-Length': '0'} 2022-09-12T20:14:12.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000deae9601353d95d6-00631f9314-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.021 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:12.022 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:12.022 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:12.023 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:12.023 DEBUG:boto:Method: HEAD 2022-09-12T20:14:12.024 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:12.024 DEBUG:boto:Data: 2022-09-12T20:14:12.024 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.025 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.025 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.026 DEBUG:boto:Params: {} 2022-09-12T20:14:12.027 DEBUG:boto:Token: None 2022-09-12T20:14:12.027 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:12.028 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ptWVIruiR9ti40r4xuktOXFoKx4= 2022-09-12T20:14:12.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ptWVIruiR9ti40r4xuktOXFoKx4=', 'Content-Length': '0'} 2022-09-12T20:14:12.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000891e924647b2726c-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:14:12.032 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:12.032 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:12.033 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:12.034 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:12.034 DEBUG:boto:Method: GET 2022-09-12T20:14:12.035 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:12.035 DEBUG:boto:Data: 2022-09-12T20:14:12.036 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.037 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:12.038 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.039 DEBUG:boto:Params: {} 2022-09-12T20:14:12.040 DEBUG:boto:Token: None 2022-09-12T20:14:12.041 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:12.042 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L0cKRbsCR1Qsb3VICGH7X09tWPA= 2022-09-12T20:14:12.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L0cKRbsCR1Qsb3VICGH7X09tWPA=', 'Content-Length': '0'} 2022-09-12T20:14:12.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005e3565508e4bb42-00631f9314-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.047 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:12.047 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:12.048 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:12.049 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:12.049 DEBUG:boto:Method: GET 2022-09-12T20:14:12.050 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:12.050 DEBUG:boto:Data: 2022-09-12T20:14:12.051 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.052 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.052 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.053 DEBUG:boto:Params: {} 2022-09-12T20:14:12.053 DEBUG:boto:Token: None 2022-09-12T20:14:12.054 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:12.055 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L0cKRbsCR1Qsb3VICGH7X09tWPA= 2022-09-12T20:14:12.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L0cKRbsCR1Qsb3VICGH7X09tWPA=', 'Content-Length': '0'} 2022-09-12T20:14:12.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072cf1e26ac4e28a2-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.102 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:12.103 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:12.103 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:14:12.104 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:14:12.104 DEBUG:boto:Method: GET 2022-09-12T20:14:12.105 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:14:12.106 DEBUG:boto:Data: 2022-09-12T20:14:12.106 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.107 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:12.107 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.108 DEBUG:boto:Params: {} 2022-09-12T20:14:12.108 DEBUG:boto:Token: None 2022-09-12T20:14:12.109 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/ 2022-09-12T20:14:12.110 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk= 2022-09-12T20:14:12.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk=', 'Content-Length': '0'} 2022-09-12T20:14:12.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000573e213182f34ce4-00631f9314-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.114 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:12.115 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:12.115 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:12.116 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:14:12.116 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:14:12.117 DEBUG:boto:Method: GET 2022-09-12T20:14:12.118 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:14:12.118 DEBUG:boto:Data: 2022-09-12T20:14:12.119 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.119 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.120 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.120 DEBUG:boto:Params: {} 2022-09-12T20:14:12.121 DEBUG:boto:Token: None 2022-09-12T20:14:12.122 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/ 2022-09-12T20:14:12.122 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk= 2022-09-12T20:14:12.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk=', 'Content-Length': '0'} 2022-09-12T20:14:12.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c1d0bd85d768ac0-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.130 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:12.131 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:12.132 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:12.132 DEBUG:boto:Method: HEAD 2022-09-12T20:14:12.133 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:14:12.133 DEBUG:boto:Data: 2022-09-12T20:14:12.134 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.134 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:12.135 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.135 DEBUG:boto:Params: {} 2022-09-12T20:14:12.136 DEBUG:boto:Token: None 2022-09-12T20:14:12.137 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj92 2022-09-12T20:14:12.138 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:r7YwItM0TOk+eOYH+zhyXZt0lig= 2022-09-12T20:14:12.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:r7YwItM0TOk+eOYH+zhyXZt0lig=', 'Content-Length': '0'} 2022-09-12T20:14:12.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ef9073242aafb75-00631f9314-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.141 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:12.142 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:12.142 DEBUG:boto:Method: HEAD 2022-09-12T20:14:12.143 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:14:12.144 DEBUG:boto:Data: 2022-09-12T20:14:12.144 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.145 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.146 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.146 DEBUG:boto:Params: {} 2022-09-12T20:14:12.147 DEBUG:boto:Token: None 2022-09-12T20:14:12.147 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj92 2022-09-12T20:14:12.148 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:r7YwItM0TOk+eOYH+zhyXZt0lig= 2022-09-12T20:14:12.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:r7YwItM0TOk+eOYH+zhyXZt0lig=', 'Content-Length': '0'} 2022-09-12T20:14:12.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d91f8793e596632-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:14:12.152 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:12.153 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:12.154 DEBUG:boto:Method: GET 2022-09-12T20:14:12.155 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:14:12.155 DEBUG:boto:Data: 2022-09-12T20:14:12.156 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.157 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:12.158 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.158 DEBUG:boto:Params: {} 2022-09-12T20:14:12.159 DEBUG:boto:Token: None 2022-09-12T20:14:12.160 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj92 2022-09-12T20:14:12.161 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7+czDFKh7ol5O+TLCNfSWO5N7IM= 2022-09-12T20:14:12.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7+czDFKh7ol5O+TLCNfSWO5N7IM=', 'Content-Length': '0'} 2022-09-12T20:14:12.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dee0fbcf8dfaa93-00631f9314-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.165 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:12.166 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:12.166 DEBUG:boto:Method: GET 2022-09-12T20:14:12.167 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:14:12.168 DEBUG:boto:Data: 2022-09-12T20:14:12.168 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.169 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.170 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.170 DEBUG:boto:Params: {} 2022-09-12T20:14:12.171 DEBUG:boto:Token: None 2022-09-12T20:14:12.171 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj92 2022-09-12T20:14:12.172 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7+czDFKh7ol5O+TLCNfSWO5N7IM= 2022-09-12T20:14:12.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7+czDFKh7ol5O+TLCNfSWO5N7IM=', 'Content-Length': '0'} 2022-09-12T20:14:12.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032fa978bf7fd2a0e-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.218 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:12.219 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:12.220 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:14:12.220 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:14:12.221 DEBUG:boto:Method: GET 2022-09-12T20:14:12.221 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:14:12.222 DEBUG:boto:Data: 2022-09-12T20:14:12.223 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.223 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:12.224 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.224 DEBUG:boto:Params: {} 2022-09-12T20:14:12.225 DEBUG:boto:Token: None 2022-09-12T20:14:12.226 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/ 2022-09-12T20:14:12.226 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk= 2022-09-12T20:14:12.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk=', 'Content-Length': '0'} 2022-09-12T20:14:12.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c00f0e0f38b1f30-00631f9314-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.230 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:12.231 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:12.232 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:12.233 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:14:12.233 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:14:12.234 DEBUG:boto:Method: GET 2022-09-12T20:14:12.234 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:14:12.235 DEBUG:boto:Data: 2022-09-12T20:14:12.236 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.236 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.237 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.237 DEBUG:boto:Params: {} 2022-09-12T20:14:12.238 DEBUG:boto:Token: None 2022-09-12T20:14:12.239 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/ 2022-09-12T20:14:12.239 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk= 2022-09-12T20:14:12.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk=', 'Content-Length': '0'} 2022-09-12T20:14:12.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a2ea5206ffcbffe-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.245 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:12.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-12T20:14:12.247 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:12.247 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:12.248 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:12.248 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:12.249 DEBUG:boto:Method: GET 2022-09-12T20:14:12.250 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:12.250 DEBUG:boto:Data: 2022-09-12T20:14:12.251 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.251 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:12.252 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.253 DEBUG:boto:Params: {} 2022-09-12T20:14:12.253 DEBUG:boto:Token: None 2022-09-12T20:14:12.254 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:12.255 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DHXcyjIiFQadUJG0UEiImflO53g= 2022-09-12T20:14:12.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DHXcyjIiFQadUJG0UEiImflO53g=', 'Content-Length': '0'} 2022-09-12T20:14:12.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013a6e2c8144a3b9e-00631f9314-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.259 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:12.260 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:12.260 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:12.261 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:12.262 DEBUG:boto:Method: GET 2022-09-12T20:14:12.263 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:12.264 DEBUG:boto:Data: 2022-09-12T20:14:12.265 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.265 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.266 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.267 DEBUG:boto:Params: {} 2022-09-12T20:14:12.268 DEBUG:boto:Token: None 2022-09-12T20:14:12.269 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:12.270 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DHXcyjIiFQadUJG0UEiImflO53g= 2022-09-12T20:14:12.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DHXcyjIiFQadUJG0UEiImflO53g=', 'Content-Length': '0'} 2022-09-12T20:14:12.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002528104f3dedcd3e-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.274 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:12.275 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:12.276 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:12.276 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:12.277 DEBUG:boto:Method: HEAD 2022-09-12T20:14:12.278 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:12.279 DEBUG:boto:Data: 2022-09-12T20:14:12.279 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.280 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:12.281 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.282 DEBUG:boto:Params: {} 2022-09-12T20:14:12.282 DEBUG:boto:Token: None 2022-09-12T20:14:12.283 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:12.284 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P4PHJJ5WnzPq25vx0UszDXcarJk= 2022-09-12T20:14:12.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P4PHJJ5WnzPq25vx0UszDXcarJk=', 'Content-Length': '0'} 2022-09-12T20:14:12.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000461748fa2d1661f-00631f9314-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.288 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:12.289 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:12.290 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:12.290 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:12.291 DEBUG:boto:Method: HEAD 2022-09-12T20:14:12.292 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:12.292 DEBUG:boto:Data: 2022-09-12T20:14:12.293 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.294 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.295 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.295 DEBUG:boto:Params: {} 2022-09-12T20:14:12.296 DEBUG:boto:Token: None 2022-09-12T20:14:12.297 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:12.298 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P4PHJJ5WnzPq25vx0UszDXcarJk= 2022-09-12T20:14:12.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P4PHJJ5WnzPq25vx0UszDXcarJk=', 'Content-Length': '0'} 2022-09-12T20:14:12.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000454d578a9c63b8-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-12T20:14:12.303 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:12.303 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:12.304 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:12.305 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:12.306 DEBUG:boto:Method: GET 2022-09-12T20:14:12.307 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:12.307 DEBUG:boto:Data: 2022-09-12T20:14:12.308 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.309 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:12.310 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.311 DEBUG:boto:Params: {} 2022-09-12T20:14:12.312 DEBUG:boto:Token: None 2022-09-12T20:14:12.313 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:12.313 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DHXcyjIiFQadUJG0UEiImflO53g= 2022-09-12T20:14:12.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DHXcyjIiFQadUJG0UEiImflO53g=', 'Content-Length': '0'} 2022-09-12T20:14:12.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ee57e343ded7525-00631f9314-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.358 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:12.359 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:12.360 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:12.360 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:12.361 DEBUG:boto:Method: GET 2022-09-12T20:14:12.362 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:12.363 DEBUG:boto:Data: 2022-09-12T20:14:12.364 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.364 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.365 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.366 DEBUG:boto:Params: {} 2022-09-12T20:14:12.367 DEBUG:boto:Token: None 2022-09-12T20:14:12.368 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:12.369 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DHXcyjIiFQadUJG0UEiImflO53g= 2022-09-12T20:14:12.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DHXcyjIiFQadUJG0UEiImflO53g=', 'Content-Length': '0'} 2022-09-12T20:14:12.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065b6f9118859a422-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.373 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:12.374 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:12.374 DEBUG:boto:path=/jdpiak-75/?prefix=obj93 2022-09-12T20:14:12.375 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj93 2022-09-12T20:14:12.376 DEBUG:boto:Method: GET 2022-09-12T20:14:12.377 DEBUG:boto:Path: /jdpiak-75/?prefix=obj93 2022-09-12T20:14:12.377 DEBUG:boto:Data: 2022-09-12T20:14:12.378 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.379 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:12.380 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.381 DEBUG:boto:Params: {} 2022-09-12T20:14:12.382 DEBUG:boto:Token: None 2022-09-12T20:14:12.382 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/ 2022-09-12T20:14:12.383 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk= 2022-09-12T20:14:12.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk=', 'Content-Length': '0'} 2022-09-12T20:14:12.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000572e85f401fff48d-00631f9314-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.388 DEBUG:boto:b'jdpiak-75obj931000falseobj932022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:12.389 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:12.390 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:12.390 DEBUG:boto:path=/jdpiak-75/?prefix=obj93 2022-09-12T20:14:12.391 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj93 2022-09-12T20:14:12.392 DEBUG:boto:Method: GET 2022-09-12T20:14:12.393 DEBUG:boto:Path: /jdpiak-75/?prefix=obj93 2022-09-12T20:14:12.393 DEBUG:boto:Data: 2022-09-12T20:14:12.394 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.395 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.396 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.396 DEBUG:boto:Params: {} 2022-09-12T20:14:12.397 DEBUG:boto:Token: None 2022-09-12T20:14:12.398 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/ 2022-09-12T20:14:12.399 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk= 2022-09-12T20:14:12.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk=', 'Content-Length': '0'} 2022-09-12T20:14:12.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ccdbd9a3b80545c-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.405 DEBUG:boto:b'jdpiak-75obj931000falseobj932022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:12.406 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:12.406 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:12.407 DEBUG:boto:Method: HEAD 2022-09-12T20:14:12.408 DEBUG:boto:Path: /jdpiak-75/obj93 2022-09-12T20:14:12.409 DEBUG:boto:Data: 2022-09-12T20:14:12.409 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.410 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:12.411 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.411 DEBUG:boto:Params: {} 2022-09-12T20:14:12.412 DEBUG:boto:Token: None 2022-09-12T20:14:12.413 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj93 2022-09-12T20:14:12.414 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PVonjkHp5PkIk4YdjfEs8v9kosM= 2022-09-12T20:14:12.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PVonjkHp5PkIk4YdjfEs8v9kosM=', 'Content-Length': '0'} 2022-09-12T20:14:12.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea2db8d116ce21c1-00631f9314-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.417 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:12.418 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:12.419 DEBUG:boto:Method: HEAD 2022-09-12T20:14:12.419 DEBUG:boto:Path: /jdpiak-75/obj93 2022-09-12T20:14:12.420 DEBUG:boto:Data: 2022-09-12T20:14:12.420 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.421 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.421 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.422 DEBUG:boto:Params: {} 2022-09-12T20:14:12.422 DEBUG:boto:Token: None 2022-09-12T20:14:12.423 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj93 2022-09-12T20:14:12.423 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PVonjkHp5PkIk4YdjfEs8v9kosM= 2022-09-12T20:14:12.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PVonjkHp5PkIk4YdjfEs8v9kosM=', 'Content-Length': '0'} 2022-09-12T20:14:12.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070ef37e50ec21347-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-12T20:14:12.428 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:12.428 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:12.429 DEBUG:boto:Method: GET 2022-09-12T20:14:12.429 DEBUG:boto:Path: /jdpiak-75/obj93 2022-09-12T20:14:12.430 DEBUG:boto:Data: 2022-09-12T20:14:12.430 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.431 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:12.431 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.432 DEBUG:boto:Params: {} 2022-09-12T20:14:12.432 DEBUG:boto:Token: None 2022-09-12T20:14:12.433 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj93 2022-09-12T20:14:12.434 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k9akPWEcjyAKSW79Y4bPak7nHn0= 2022-09-12T20:14:12.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k9akPWEcjyAKSW79Y4bPak7nHn0=', 'Content-Length': '0'} 2022-09-12T20:14:12.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000725894a7e1aeae18-00631f9314-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.437 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:12.438 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:12.439 DEBUG:boto:Method: GET 2022-09-12T20:14:12.439 DEBUG:boto:Path: /jdpiak-75/obj93 2022-09-12T20:14:12.440 DEBUG:boto:Data: 2022-09-12T20:14:12.440 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.441 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.441 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.442 DEBUG:boto:Params: {} 2022-09-12T20:14:12.442 DEBUG:boto:Token: None 2022-09-12T20:14:12.443 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj93 2022-09-12T20:14:12.444 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k9akPWEcjyAKSW79Y4bPak7nHn0= 2022-09-12T20:14:12.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k9akPWEcjyAKSW79Y4bPak7nHn0=', 'Content-Length': '0'} 2022-09-12T20:14:12.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014f3f36864132cd9-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.490 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:12.491 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:12.491 DEBUG:boto:path=/jdpiak-75/?prefix=obj93 2022-09-12T20:14:12.492 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj93 2022-09-12T20:14:12.492 DEBUG:boto:Method: GET 2022-09-12T20:14:12.493 DEBUG:boto:Path: /jdpiak-75/?prefix=obj93 2022-09-12T20:14:12.493 DEBUG:boto:Data: 2022-09-12T20:14:12.494 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.495 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:12.495 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.496 DEBUG:boto:Params: {} 2022-09-12T20:14:12.496 DEBUG:boto:Token: None 2022-09-12T20:14:12.497 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/ 2022-09-12T20:14:12.497 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk= 2022-09-12T20:14:12.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk=', 'Content-Length': '0'} 2022-09-12T20:14:12.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb2b11dac499598b-00631f9314-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.501 DEBUG:boto:b'jdpiak-75obj931000falseobj932022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:12.502 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:12.503 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:12.503 DEBUG:boto:path=/jdpiak-75/?prefix=obj93 2022-09-12T20:14:12.504 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj93 2022-09-12T20:14:12.504 DEBUG:boto:Method: GET 2022-09-12T20:14:12.505 DEBUG:boto:Path: /jdpiak-75/?prefix=obj93 2022-09-12T20:14:12.505 DEBUG:boto:Data: 2022-09-12T20:14:12.506 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.507 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.507 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.508 DEBUG:boto:Params: {} 2022-09-12T20:14:12.508 DEBUG:boto:Token: None 2022-09-12T20:14:12.509 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/ 2022-09-12T20:14:12.510 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk= 2022-09-12T20:14:12.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk=', 'Content-Length': '0'} 2022-09-12T20:14:12.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000022bc21604f7a03c-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.516 DEBUG:boto:b'jdpiak-75obj931000falseobj932022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:12.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:14:12.517 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:12.518 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:12.518 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:12.519 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:12.519 DEBUG:boto:Method: GET 2022-09-12T20:14:12.520 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:12.520 DEBUG:boto:Data: 2022-09-12T20:14:12.521 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.522 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:12.522 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.523 DEBUG:boto:Params: {} 2022-09-12T20:14:12.523 DEBUG:boto:Token: None 2022-09-12T20:14:12.524 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:12.524 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HbwliGgSavkaehVVjLSQ3IYi4eE= 2022-09-12T20:14:12.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HbwliGgSavkaehVVjLSQ3IYi4eE=', 'Content-Length': '0'} 2022-09-12T20:14:12.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e56bcd3b7b328329-00631f9314-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.528 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:12.529 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:12.529 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:12.530 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:12.530 DEBUG:boto:Method: GET 2022-09-12T20:14:12.531 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:12.531 DEBUG:boto:Data: 2022-09-12T20:14:12.532 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.532 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.533 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.533 DEBUG:boto:Params: {} 2022-09-12T20:14:12.534 DEBUG:boto:Token: None 2022-09-12T20:14:12.535 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:12.535 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HbwliGgSavkaehVVjLSQ3IYi4eE= 2022-09-12T20:14:12.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HbwliGgSavkaehVVjLSQ3IYi4eE=', 'Content-Length': '0'} 2022-09-12T20:14:12.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0ef5244bc4a2bb0-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.539 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:12.539 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:12.540 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:12.540 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:12.541 DEBUG:boto:Method: HEAD 2022-09-12T20:14:12.541 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:12.542 DEBUG:boto:Data: 2022-09-12T20:14:12.543 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.543 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:12.544 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.544 DEBUG:boto:Params: {} 2022-09-12T20:14:12.545 DEBUG:boto:Token: None 2022-09-12T20:14:12.545 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:12.546 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7+pbAp8h/bd7subrI9G8rQSESnc= 2022-09-12T20:14:12.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7+pbAp8h/bd7subrI9G8rQSESnc=', 'Content-Length': '0'} 2022-09-12T20:14:12.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f01aa7ff0bb7c83-00631f9314-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.549 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:12.550 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:12.550 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:12.551 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:12.551 DEBUG:boto:Method: HEAD 2022-09-12T20:14:12.552 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:12.552 DEBUG:boto:Data: 2022-09-12T20:14:12.553 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.553 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.554 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.554 DEBUG:boto:Params: {} 2022-09-12T20:14:12.555 DEBUG:boto:Token: None 2022-09-12T20:14:12.555 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:12.556 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7+pbAp8h/bd7subrI9G8rQSESnc= 2022-09-12T20:14:12.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7+pbAp8h/bd7subrI9G8rQSESnc=', 'Content-Length': '0'} 2022-09-12T20:14:12.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f43c5b77fc44a558-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:14:12.560 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:12.561 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:12.561 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:12.562 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:12.563 DEBUG:boto:Method: GET 2022-09-12T20:14:12.563 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:12.564 DEBUG:boto:Data: 2022-09-12T20:14:12.564 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.565 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:12.565 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.566 DEBUG:boto:Params: {} 2022-09-12T20:14:12.567 DEBUG:boto:Token: None 2022-09-12T20:14:12.568 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:12.568 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HbwliGgSavkaehVVjLSQ3IYi4eE= 2022-09-12T20:14:12.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HbwliGgSavkaehVVjLSQ3IYi4eE=', 'Content-Length': '0'} 2022-09-12T20:14:12.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000257fffd5d6eaddf9-00631f9314-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.572 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:12.573 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:12.573 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:12.574 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:12.574 DEBUG:boto:Method: GET 2022-09-12T20:14:12.575 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:12.575 DEBUG:boto:Data: 2022-09-12T20:14:12.576 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.577 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.577 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.578 DEBUG:boto:Params: {} 2022-09-12T20:14:12.578 DEBUG:boto:Token: None 2022-09-12T20:14:12.579 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:12.579 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HbwliGgSavkaehVVjLSQ3IYi4eE= 2022-09-12T20:14:12.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HbwliGgSavkaehVVjLSQ3IYi4eE=', 'Content-Length': '0'} 2022-09-12T20:14:12.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf60fb84f2c094f0-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.626 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:12.627 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:12.627 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:14:12.628 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:14:12.628 DEBUG:boto:Method: GET 2022-09-12T20:14:12.629 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:14:12.629 DEBUG:boto:Data: 2022-09-12T20:14:12.630 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.631 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:12.631 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.632 DEBUG:boto:Params: {} 2022-09-12T20:14:12.632 DEBUG:boto:Token: None 2022-09-12T20:14:12.633 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/ 2022-09-12T20:14:12.633 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk= 2022-09-12T20:14:12.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk=', 'Content-Length': '0'} 2022-09-12T20:14:12.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4be836b3c54c629-00631f9314-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.638 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:12.638 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:12.639 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:12.639 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:14:12.640 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:14:12.641 DEBUG:boto:Method: GET 2022-09-12T20:14:12.641 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:14:12.642 DEBUG:boto:Data: 2022-09-12T20:14:12.642 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.643 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.643 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.644 DEBUG:boto:Params: {} 2022-09-12T20:14:12.644 DEBUG:boto:Token: None 2022-09-12T20:14:12.645 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/ 2022-09-12T20:14:12.645 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk= 2022-09-12T20:14:12.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk=', 'Content-Length': '0'} 2022-09-12T20:14:12.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7e3868cb5a227a2-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.651 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:12.652 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:12.652 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:12.653 DEBUG:boto:Method: HEAD 2022-09-12T20:14:12.653 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:14:12.654 DEBUG:boto:Data: 2022-09-12T20:14:12.654 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.655 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:12.655 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.656 DEBUG:boto:Params: {} 2022-09-12T20:14:12.657 DEBUG:boto:Token: None 2022-09-12T20:14:12.657 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj95 2022-09-12T20:14:12.658 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q+i86UJxeEIEp8++8R/zARcJFvI= 2022-09-12T20:14:12.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q+i86UJxeEIEp8++8R/zARcJFvI=', 'Content-Length': '0'} 2022-09-12T20:14:12.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000242f03b8047ee399-00631f9314-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.661 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:12.662 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:12.662 DEBUG:boto:Method: HEAD 2022-09-12T20:14:12.663 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:14:12.663 DEBUG:boto:Data: 2022-09-12T20:14:12.664 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.664 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.665 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.665 DEBUG:boto:Params: {} 2022-09-12T20:14:12.666 DEBUG:boto:Token: None 2022-09-12T20:14:12.667 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj95 2022-09-12T20:14:12.667 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Q+i86UJxeEIEp8++8R/zARcJFvI= 2022-09-12T20:14:12.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Q+i86UJxeEIEp8++8R/zARcJFvI=', 'Content-Length': '0'} 2022-09-12T20:14:12.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082e626f69c79b890-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:14:12.671 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:12.672 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:12.672 DEBUG:boto:Method: GET 2022-09-12T20:14:12.673 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:14:12.673 DEBUG:boto:Data: 2022-09-12T20:14:12.674 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.674 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:12.675 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.675 DEBUG:boto:Params: {} 2022-09-12T20:14:12.676 DEBUG:boto:Token: None 2022-09-12T20:14:12.677 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj95 2022-09-12T20:14:12.677 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LL9A4ZZyo/vVmpYQg3R6PpfaB3k= 2022-09-12T20:14:12.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LL9A4ZZyo/vVmpYQg3R6PpfaB3k=', 'Content-Length': '0'} 2022-09-12T20:14:12.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000266c963fac469790-00631f9314-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.681 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:12.681 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:12.682 DEBUG:boto:Method: GET 2022-09-12T20:14:12.683 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:14:12.683 DEBUG:boto:Data: 2022-09-12T20:14:12.684 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.684 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.685 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.685 DEBUG:boto:Params: {} 2022-09-12T20:14:12.686 DEBUG:boto:Token: None 2022-09-12T20:14:12.687 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj95 2022-09-12T20:14:12.687 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LL9A4ZZyo/vVmpYQg3R6PpfaB3k= 2022-09-12T20:14:12.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LL9A4ZZyo/vVmpYQg3R6PpfaB3k=', 'Content-Length': '0'} 2022-09-12T20:14:12.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0fdb3a83c488f11-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.691 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:12.691 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:12.692 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:14:12.693 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:14:12.693 DEBUG:boto:Method: GET 2022-09-12T20:14:12.694 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:14:12.694 DEBUG:boto:Data: 2022-09-12T20:14:12.695 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.696 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:12.696 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.697 DEBUG:boto:Params: {} 2022-09-12T20:14:12.697 DEBUG:boto:Token: None 2022-09-12T20:14:12.698 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/ 2022-09-12T20:14:12.699 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk= 2022-09-12T20:14:12.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk=', 'Content-Length': '0'} 2022-09-12T20:14:12.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007132b46daafdec81-00631f9314-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.703 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:12.703 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:12.704 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:12.705 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:14:12.705 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:14:12.706 DEBUG:boto:Method: GET 2022-09-12T20:14:12.706 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:14:12.707 DEBUG:boto:Data: 2022-09-12T20:14:12.707 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.708 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.709 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.709 DEBUG:boto:Params: {} 2022-09-12T20:14:12.710 DEBUG:boto:Token: None 2022-09-12T20:14:12.710 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/ 2022-09-12T20:14:12.711 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk= 2022-09-12T20:14:12.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk=', 'Content-Length': '0'} 2022-09-12T20:14:12.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033e8b0b113ae5eb4-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.758 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:12.759 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-75 zones={test-zone2, test-zone1} 2022-09-12T20:14:12.760 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-75 zones={test-zone3, test-zone1} 2022-09-12T20:14:12.760 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:12.761 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:12.761 DEBUG:boto:Method: HEAD 2022-09-12T20:14:12.762 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:14:12.762 DEBUG:boto:Data: 2022-09-12T20:14:12.763 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.763 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:12.764 DEBUG:boto:Port: 8001 2022-09-12T20:14:12.764 DEBUG:boto:Params: {} 2022-09-12T20:14:12.765 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T20:14:12.766 DEBUG:boto:Token: None 2022-09-12T20:14:12.766 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/ 2022-09-12T20:14:12.767 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9XDswG9IKAF6Npr5DK4teQt9ZvQ= 2022-09-12T20:14:12.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9XDswG9IKAF6Npr5DK4teQt9ZvQ=', 'Content-Length': '0'} 2022-09-12T20:14:12.771 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a39780dd50bf0bdf-00631f9314-1826-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.772 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:12.772 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:12.773 DEBUG:boto:Method: HEAD 2022-09-12T20:14:12.773 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:14:12.774 DEBUG:boto:Data: 2022-09-12T20:14:12.774 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.775 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.775 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.776 DEBUG:boto:Params: {} 2022-09-12T20:14:12.776 DEBUG:boto:Token: None 2022-09-12T20:14:12.777 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/ 2022-09-12T20:14:12.778 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9XDswG9IKAF6Npr5DK4teQt9ZvQ= 2022-09-12T20:14:12.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9XDswG9IKAF6Npr5DK4teQt9ZvQ=', 'Content-Length': '0'} 2022-09-12T20:14:12.783 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000966bb604994950a-00631f9314-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.784 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T20:14:12.784 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:12.785 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:12.785 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:12.786 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:12.786 DEBUG:boto:Method: GET 2022-09-12T20:14:12.787 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:12.787 DEBUG:boto:Data: 2022-09-12T20:14:12.788 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.788 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:12.789 DEBUG:boto:Port: 8001 2022-09-12T20:14:12.789 DEBUG:boto:Params: {} 2022-09-12T20:14:12.790 DEBUG:boto:Token: None 2022-09-12T20:14:12.791 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/?versions 2022-09-12T20:14:12.791 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KT78WRAvfq4J2O7zU3I+ZLb49t4= 2022-09-12T20:14:12.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KT78WRAvfq4J2O7zU3I+ZLb49t4=', 'Content-Length': '0'} 2022-09-12T20:14:12.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b110043e60e3340-00631f9314-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.795 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:12.798 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:14:12.798 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-12T20:14:12.799 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-12T20:14:12.799 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2022-09-12T20:14:12.799 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-12T20:14:12.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2022-09-12T20:14:12.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:14:12.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:14:12.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:14:12.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-12T20:14:12.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-12T20:14:12.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2022-09-12T20:14:12.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:14:12.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:14:12.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:14:12.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:14:12.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-12T20:14:12.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:14:12.806 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:14:12.806 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:14:12.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-12T20:14:12.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2022-09-12T20:14:12.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:14:12.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-12T20:14:12.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:14:12.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:14:12.810 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-12T20:14:12.810 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-12T20:14:12.810 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2022-09-12T20:14:12.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:14:12.811 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T20:14:12.812 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:12.812 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:12.813 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:12.813 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:12.814 DEBUG:boto:Method: GET 2022-09-12T20:14:12.815 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:12.815 DEBUG:boto:Data: 2022-09-12T20:14:12.816 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.816 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.817 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.817 DEBUG:boto:Params: {} 2022-09-12T20:14:12.818 DEBUG:boto:Token: None 2022-09-12T20:14:12.818 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/?versions 2022-09-12T20:14:12.819 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KT78WRAvfq4J2O7zU3I+ZLb49t4= 2022-09-12T20:14:12.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KT78WRAvfq4J2O7zU3I+ZLb49t4=', 'Content-Length': '0'} 2022-09-12T20:14:12.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ed749ce9500a41f-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.825 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:12.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:14:12.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-12T20:14:12.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-12T20:14:12.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2022-09-12T20:14:12.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-12T20:14:12.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2022-09-12T20:14:12.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:14:12.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:14:12.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:14:12.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-12T20:14:12.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-12T20:14:12.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2022-09-12T20:14:12.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:14:12.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:14:12.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:14:12.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:14:12.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-12T20:14:12.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:14:12.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:14:12.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:14:12.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-12T20:14:12.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2022-09-12T20:14:12.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:14:12.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-12T20:14:12.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:14:12.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:14:12.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-12T20:14:12.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-12T20:14:12.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2022-09-12T20:14:12.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:14:12.841 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:12.842 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:12.842 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:12.843 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:12.843 DEBUG:boto:Method: GET 2022-09-12T20:14:12.844 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:12.844 DEBUG:boto:Data: 2022-09-12T20:14:12.845 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.845 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:12.846 DEBUG:boto:Port: 8001 2022-09-12T20:14:12.846 DEBUG:boto:Params: {} 2022-09-12T20:14:12.847 DEBUG:boto:Token: None 2022-09-12T20:14:12.848 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/?versions 2022-09-12T20:14:12.848 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KT78WRAvfq4J2O7zU3I+ZLb49t4= 2022-09-12T20:14:12.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KT78WRAvfq4J2O7zU3I+ZLb49t4=', 'Content-Length': '0'} 2022-09-12T20:14:12.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc6a58c0622af799-00631f9314-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.853 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:12.855 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:12.855 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:12.856 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:12.856 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:12.857 DEBUG:boto:Method: GET 2022-09-12T20:14:12.857 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:12.858 DEBUG:boto:Data: 2022-09-12T20:14:12.858 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.859 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.859 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.860 DEBUG:boto:Params: {} 2022-09-12T20:14:12.860 DEBUG:boto:Token: None 2022-09-12T20:14:12.861 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/?versions 2022-09-12T20:14:12.861 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KT78WRAvfq4J2O7zU3I+ZLb49t4= 2022-09-12T20:14:12.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KT78WRAvfq4J2O7zU3I+ZLb49t4=', 'Content-Length': '0'} 2022-09-12T20:14:12.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011d65693d64316ba-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.867 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:12.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:14:12.870 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:12.870 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:12.871 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:12.871 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:12.872 DEBUG:boto:Method: GET 2022-09-12T20:14:12.872 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:12.873 DEBUG:boto:Data: 2022-09-12T20:14:12.874 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.874 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:12.875 DEBUG:boto:Port: 8001 2022-09-12T20:14:12.875 DEBUG:boto:Params: {} 2022-09-12T20:14:12.876 DEBUG:boto:Token: None 2022-09-12T20:14:12.876 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:12.877 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P4bUxsW3EyyrZ4KASJ0UtnXSsqw= 2022-09-12T20:14:12.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P4bUxsW3EyyrZ4KASJ0UtnXSsqw=', 'Content-Length': '0'} 2022-09-12T20:14:12.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084029c397eb4e812-00631f9314-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.880 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:12.881 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:12.882 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:12.882 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:12.883 DEBUG:boto:Method: GET 2022-09-12T20:14:12.883 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:12.884 DEBUG:boto:Data: 2022-09-12T20:14:12.884 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.885 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.885 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.886 DEBUG:boto:Params: {} 2022-09-12T20:14:12.886 DEBUG:boto:Token: None 2022-09-12T20:14:12.887 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:12.887 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P4bUxsW3EyyrZ4KASJ0UtnXSsqw= 2022-09-12T20:14:12.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P4bUxsW3EyyrZ4KASJ0UtnXSsqw=', 'Content-Length': '0'} 2022-09-12T20:14:12.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f8a10b46476ec54-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.934 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:12.935 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:12.935 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:12.936 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:12.936 DEBUG:boto:Method: HEAD 2022-09-12T20:14:12.937 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:12.937 DEBUG:boto:Data: 2022-09-12T20:14:12.938 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.938 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:12.939 DEBUG:boto:Port: 8001 2022-09-12T20:14:12.939 DEBUG:boto:Params: {} 2022-09-12T20:14:12.940 DEBUG:boto:Token: None 2022-09-12T20:14:12.940 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:12.941 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:APpTkOaoDf5DnXi5FM0W3c4TmtU= 2022-09-12T20:14:12.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:APpTkOaoDf5DnXi5FM0W3c4TmtU=', 'Content-Length': '0'} 2022-09-12T20:14:12.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b932bb7a7e9cac10-00631f9314-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.944 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:12.945 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:12.945 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:12.946 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:12.946 DEBUG:boto:Method: HEAD 2022-09-12T20:14:12.947 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:12.947 DEBUG:boto:Data: 2022-09-12T20:14:12.948 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.948 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.949 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.949 DEBUG:boto:Params: {} 2022-09-12T20:14:12.950 DEBUG:boto:Token: None 2022-09-12T20:14:12.951 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:12.951 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:APpTkOaoDf5DnXi5FM0W3c4TmtU= 2022-09-12T20:14:12.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:APpTkOaoDf5DnXi5FM0W3c4TmtU=', 'Content-Length': '0'} 2022-09-12T20:14:12.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014f8caeedfc34d3f-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:14:12.955 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:12.956 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:12.956 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:12.957 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:12.957 DEBUG:boto:Method: GET 2022-09-12T20:14:12.958 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:12.958 DEBUG:boto:Data: 2022-09-12T20:14:12.959 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.959 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:12.960 DEBUG:boto:Port: 8001 2022-09-12T20:14:12.960 DEBUG:boto:Params: {} 2022-09-12T20:14:12.961 DEBUG:boto:Token: None 2022-09-12T20:14:12.962 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:12.962 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P4bUxsW3EyyrZ4KASJ0UtnXSsqw= 2022-09-12T20:14:12.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P4bUxsW3EyyrZ4KASJ0UtnXSsqw=', 'Content-Length': '0'} 2022-09-12T20:14:12.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e516e0aaed982f8d-00631f9314-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.966 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:12.966 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:12.967 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:12.967 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:12.968 DEBUG:boto:Method: GET 2022-09-12T20:14:12.968 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:12.969 DEBUG:boto:Data: 2022-09-12T20:14:12.969 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.970 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.971 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.971 DEBUG:boto:Params: {} 2022-09-12T20:14:12.972 DEBUG:boto:Token: None 2022-09-12T20:14:12.972 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:12.973 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:P4bUxsW3EyyrZ4KASJ0UtnXSsqw= 2022-09-12T20:14:12.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:P4bUxsW3EyyrZ4KASJ0UtnXSsqw=', 'Content-Length': '0'} 2022-09-12T20:14:12.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b954929bd98b69e-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.977 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:12.977 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:12.978 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:14:12.979 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:14:12.979 DEBUG:boto:Method: GET 2022-09-12T20:14:12.980 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:14:12.980 DEBUG:boto:Data: 2022-09-12T20:14:12.981 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.981 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:12.982 DEBUG:boto:Port: 8001 2022-09-12T20:14:12.983 DEBUG:boto:Params: {} 2022-09-12T20:14:12.983 DEBUG:boto:Token: None 2022-09-12T20:14:12.984 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/ 2022-09-12T20:14:12.984 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk= 2022-09-12T20:14:12.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk=', 'Content-Length': '0'} 2022-09-12T20:14:12.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000aa52b95990c6648-00631f9314-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:12.989 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:12.990 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:12.991 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:12.991 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:14:12.992 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:14:12.992 DEBUG:boto:Method: GET 2022-09-12T20:14:12.993 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:14:12.993 DEBUG:boto:Data: 2022-09-12T20:14:12.994 DEBUG:boto:Headers: {} 2022-09-12T20:14:12.995 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:12.995 DEBUG:boto:Port: 8000 2022-09-12T20:14:12.996 DEBUG:boto:Params: {} 2022-09-12T20:14:12.996 DEBUG:boto:Token: None 2022-09-12T20:14:12.997 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:12 GMT /jdpiak-75/ 2022-09-12T20:14:12.997 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk= 2022-09-12T20:14:12.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fA0AWrWmXvj4Uy/93EuO6s8ECYk=', 'Content-Length': '0'} 2022-09-12T20:14:13.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005101025a6e805758-00631f9314-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.003 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:13.004 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:13.004 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:13.005 DEBUG:boto:Method: HEAD 2022-09-12T20:14:13.006 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:14:13.006 DEBUG:boto:Data: 2022-09-12T20:14:13.007 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.007 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.008 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.008 DEBUG:boto:Params: {} 2022-09-12T20:14:13.009 DEBUG:boto:Token: None 2022-09-12T20:14:13.009 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj11 2022-09-12T20:14:13.010 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wojT5XOEcAAoL+RbPUKKmt2GL5E= 2022-09-12T20:14:13.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wojT5XOEcAAoL+RbPUKKmt2GL5E=', 'Content-Length': '0'} 2022-09-12T20:14:13.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000479464be2073a8f0-00631f9315-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.014 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:13.014 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:13.015 DEBUG:boto:Method: HEAD 2022-09-12T20:14:13.015 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:14:13.016 DEBUG:boto:Data: 2022-09-12T20:14:13.016 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.017 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.017 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.018 DEBUG:boto:Params: {} 2022-09-12T20:14:13.019 DEBUG:boto:Token: None 2022-09-12T20:14:13.019 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj11 2022-09-12T20:14:13.020 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wojT5XOEcAAoL+RbPUKKmt2GL5E= 2022-09-12T20:14:13.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wojT5XOEcAAoL+RbPUKKmt2GL5E=', 'Content-Length': '0'} 2022-09-12T20:14:13.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e38793d825edb1b-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:14:13.024 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:13.024 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:13.025 DEBUG:boto:Method: GET 2022-09-12T20:14:13.025 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:14:13.026 DEBUG:boto:Data: 2022-09-12T20:14:13.027 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.027 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.028 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.028 DEBUG:boto:Params: {} 2022-09-12T20:14:13.029 DEBUG:boto:Token: None 2022-09-12T20:14:13.029 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj11 2022-09-12T20:14:13.030 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6c6NBLhvh0iz/l6kTJ7pMrYpeZQ= 2022-09-12T20:14:13.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6c6NBLhvh0iz/l6kTJ7pMrYpeZQ=', 'Content-Length': '0'} 2022-09-12T20:14:13.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c378a5e13ddc001-00631f9315-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.034 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:13.034 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:13.035 DEBUG:boto:Method: GET 2022-09-12T20:14:13.035 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:14:13.036 DEBUG:boto:Data: 2022-09-12T20:14:13.037 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.037 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.038 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.038 DEBUG:boto:Params: {} 2022-09-12T20:14:13.039 DEBUG:boto:Token: None 2022-09-12T20:14:13.039 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj11 2022-09-12T20:14:13.040 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6c6NBLhvh0iz/l6kTJ7pMrYpeZQ= 2022-09-12T20:14:13.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6c6NBLhvh0iz/l6kTJ7pMrYpeZQ=', 'Content-Length': '0'} 2022-09-12T20:14:13.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fcf317c5838be6b-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.086 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:13.087 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:13.087 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:14:13.088 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:14:13.089 DEBUG:boto:Method: GET 2022-09-12T20:14:13.089 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:14:13.090 DEBUG:boto:Data: 2022-09-12T20:14:13.090 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.091 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.091 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.092 DEBUG:boto:Params: {} 2022-09-12T20:14:13.093 DEBUG:boto:Token: None 2022-09-12T20:14:13.093 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/ 2022-09-12T20:14:13.094 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw= 2022-09-12T20:14:13.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw=', 'Content-Length': '0'} 2022-09-12T20:14:13.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb16fd15382aaa58-00631f9315-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.098 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:13.099 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:13.100 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:13.100 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:14:13.101 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:14:13.101 DEBUG:boto:Method: GET 2022-09-12T20:14:13.102 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:14:13.102 DEBUG:boto:Data: 2022-09-12T20:14:13.103 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.104 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.104 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.105 DEBUG:boto:Params: {} 2022-09-12T20:14:13.105 DEBUG:boto:Token: None 2022-09-12T20:14:13.106 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/ 2022-09-12T20:14:13.107 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw= 2022-09-12T20:14:13.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw=', 'Content-Length': '0'} 2022-09-12T20:14:13.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c591dc75bffd359b-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.113 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:13.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:14:13.115 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:13.116 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:13.117 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:13.117 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:13.118 DEBUG:boto:Method: GET 2022-09-12T20:14:13.119 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:13.119 DEBUG:boto:Data: 2022-09-12T20:14:13.120 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.121 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.121 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.122 DEBUG:boto:Params: {} 2022-09-12T20:14:13.123 DEBUG:boto:Token: None 2022-09-12T20:14:13.124 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:13.125 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NvTV86YfAC7cuOFMLOHQtjDFaj8= 2022-09-12T20:14:13.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NvTV86YfAC7cuOFMLOHQtjDFaj8=', 'Content-Length': '0'} 2022-09-12T20:14:13.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b47b4e10c49a0a3b-00631f9315-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.128 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:13.129 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:13.129 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:13.130 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:13.130 DEBUG:boto:Method: GET 2022-09-12T20:14:13.131 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:13.132 DEBUG:boto:Data: 2022-09-12T20:14:13.132 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.133 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.133 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.134 DEBUG:boto:Params: {} 2022-09-12T20:14:13.134 DEBUG:boto:Token: None 2022-09-12T20:14:13.135 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:13.136 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NvTV86YfAC7cuOFMLOHQtjDFaj8= 2022-09-12T20:14:13.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NvTV86YfAC7cuOFMLOHQtjDFaj8=', 'Content-Length': '0'} 2022-09-12T20:14:13.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8d992f7851c90f6-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.139 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:13.140 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:13.140 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:13.141 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:13.141 DEBUG:boto:Method: HEAD 2022-09-12T20:14:13.142 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:13.142 DEBUG:boto:Data: 2022-09-12T20:14:13.143 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.143 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.144 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.145 DEBUG:boto:Params: {} 2022-09-12T20:14:13.145 DEBUG:boto:Token: None 2022-09-12T20:14:13.146 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:13.146 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9mD2hh4gh1t8BdrdepmIjNs8IBc= 2022-09-12T20:14:13.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9mD2hh4gh1t8BdrdepmIjNs8IBc=', 'Content-Length': '0'} 2022-09-12T20:14:13.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bc9b2b89ec3672a-00631f9315-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.150 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:13.150 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:13.151 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:13.151 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:13.152 DEBUG:boto:Method: HEAD 2022-09-12T20:14:13.152 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:13.153 DEBUG:boto:Data: 2022-09-12T20:14:13.153 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.154 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.154 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.155 DEBUG:boto:Params: {} 2022-09-12T20:14:13.155 DEBUG:boto:Token: None 2022-09-12T20:14:13.156 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:13.156 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9mD2hh4gh1t8BdrdepmIjNs8IBc= 2022-09-12T20:14:13.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9mD2hh4gh1t8BdrdepmIjNs8IBc=', 'Content-Length': '0'} 2022-09-12T20:14:13.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2002640e7a1ca18-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:14:13.160 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:13.161 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:13.162 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:13.162 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:13.163 DEBUG:boto:Method: GET 2022-09-12T20:14:13.163 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:13.164 DEBUG:boto:Data: 2022-09-12T20:14:13.164 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.165 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.165 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.166 DEBUG:boto:Params: {} 2022-09-12T20:14:13.167 DEBUG:boto:Token: None 2022-09-12T20:14:13.167 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:13.168 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NvTV86YfAC7cuOFMLOHQtjDFaj8= 2022-09-12T20:14:13.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NvTV86YfAC7cuOFMLOHQtjDFaj8=', 'Content-Length': '0'} 2022-09-12T20:14:13.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a52bdcd530c1716-00631f9315-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.171 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:13.172 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:13.172 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:13.173 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:13.173 DEBUG:boto:Method: GET 2022-09-12T20:14:13.174 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:13.175 DEBUG:boto:Data: 2022-09-12T20:14:13.175 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.176 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.176 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.177 DEBUG:boto:Params: {} 2022-09-12T20:14:13.177 DEBUG:boto:Token: None 2022-09-12T20:14:13.178 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:13.179 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NvTV86YfAC7cuOFMLOHQtjDFaj8= 2022-09-12T20:14:13.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NvTV86YfAC7cuOFMLOHQtjDFaj8=', 'Content-Length': '0'} 2022-09-12T20:14:13.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b59221bfc385c8ef-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.222 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:13.223 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:13.223 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:14:13.224 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:14:13.224 DEBUG:boto:Method: GET 2022-09-12T20:14:13.225 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:14:13.225 DEBUG:boto:Data: 2022-09-12T20:14:13.226 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.226 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.227 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.227 DEBUG:boto:Params: {} 2022-09-12T20:14:13.228 DEBUG:boto:Token: None 2022-09-12T20:14:13.229 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/ 2022-09-12T20:14:13.229 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw= 2022-09-12T20:14:13.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw=', 'Content-Length': '0'} 2022-09-12T20:14:13.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082f7f846bfbc29c2-00631f9315-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.234 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:13.235 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:13.235 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:13.236 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:14:13.236 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:14:13.237 DEBUG:boto:Method: GET 2022-09-12T20:14:13.237 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:14:13.238 DEBUG:boto:Data: 2022-09-12T20:14:13.238 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.239 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.239 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.240 DEBUG:boto:Params: {} 2022-09-12T20:14:13.241 DEBUG:boto:Token: None 2022-09-12T20:14:13.241 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/ 2022-09-12T20:14:13.242 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw= 2022-09-12T20:14:13.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw=', 'Content-Length': '0'} 2022-09-12T20:14:13.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e8a8e97db06b991-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.248 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:13.249 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:13.249 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:13.250 DEBUG:boto:Method: HEAD 2022-09-12T20:14:13.250 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:14:13.251 DEBUG:boto:Data: 2022-09-12T20:14:13.251 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.252 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.252 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.253 DEBUG:boto:Params: {} 2022-09-12T20:14:13.253 DEBUG:boto:Token: None 2022-09-12T20:14:13.254 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj115 2022-09-12T20:14:13.254 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XuHBShmQIkJNIJeKYBrQ31RLPHU= 2022-09-12T20:14:13.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XuHBShmQIkJNIJeKYBrQ31RLPHU=', 'Content-Length': '0'} 2022-09-12T20:14:13.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ddab8236d74bb37-00631f9315-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.258 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:13.259 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:13.259 DEBUG:boto:Method: HEAD 2022-09-12T20:14:13.260 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:14:13.260 DEBUG:boto:Data: 2022-09-12T20:14:13.261 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.261 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.262 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.262 DEBUG:boto:Params: {} 2022-09-12T20:14:13.263 DEBUG:boto:Token: None 2022-09-12T20:14:13.263 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj115 2022-09-12T20:14:13.264 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XuHBShmQIkJNIJeKYBrQ31RLPHU= 2022-09-12T20:14:13.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XuHBShmQIkJNIJeKYBrQ31RLPHU=', 'Content-Length': '0'} 2022-09-12T20:14:13.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e489b1600595224e-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:14:13.268 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:13.269 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:13.269 DEBUG:boto:Method: GET 2022-09-12T20:14:13.270 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:14:13.270 DEBUG:boto:Data: 2022-09-12T20:14:13.271 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.271 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.272 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.272 DEBUG:boto:Params: {} 2022-09-12T20:14:13.273 DEBUG:boto:Token: None 2022-09-12T20:14:13.274 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj115 2022-09-12T20:14:13.274 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BAwFStnYmZHZ8Qg+/NBNHz0pjaM= 2022-09-12T20:14:13.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BAwFStnYmZHZ8Qg+/NBNHz0pjaM=', 'Content-Length': '0'} 2022-09-12T20:14:13.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005282b10dba62b347-00631f9315-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.278 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:13.278 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:13.279 DEBUG:boto:Method: GET 2022-09-12T20:14:13.279 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:14:13.280 DEBUG:boto:Data: 2022-09-12T20:14:13.281 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.281 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.282 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.282 DEBUG:boto:Params: {} 2022-09-12T20:14:13.283 DEBUG:boto:Token: None 2022-09-12T20:14:13.283 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj115 2022-09-12T20:14:13.284 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BAwFStnYmZHZ8Qg+/NBNHz0pjaM= 2022-09-12T20:14:13.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BAwFStnYmZHZ8Qg+/NBNHz0pjaM=', 'Content-Length': '0'} 2022-09-12T20:14:13.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de0c9859a19d8c45-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.330 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:13.331 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:13.331 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:14:13.332 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:14:13.332 DEBUG:boto:Method: GET 2022-09-12T20:14:13.333 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:14:13.333 DEBUG:boto:Data: 2022-09-12T20:14:13.334 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.334 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.335 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.335 DEBUG:boto:Params: {} 2022-09-12T20:14:13.336 DEBUG:boto:Token: None 2022-09-12T20:14:13.337 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/ 2022-09-12T20:14:13.337 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw= 2022-09-12T20:14:13.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw=', 'Content-Length': '0'} 2022-09-12T20:14:13.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000216a7ce1a6b02341-00631f9315-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.342 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:13.342 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:13.343 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:13.344 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:14:13.344 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:14:13.345 DEBUG:boto:Method: GET 2022-09-12T20:14:13.345 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:14:13.346 DEBUG:boto:Data: 2022-09-12T20:14:13.346 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.347 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.347 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.348 DEBUG:boto:Params: {} 2022-09-12T20:14:13.348 DEBUG:boto:Token: None 2022-09-12T20:14:13.349 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/ 2022-09-12T20:14:13.349 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw= 2022-09-12T20:14:13.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw=', 'Content-Length': '0'} 2022-09-12T20:14:13.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000270fe81d6e60dc64-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.355 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:13.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:14:13.356 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:13.357 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:13.357 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:13.358 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:13.358 DEBUG:boto:Method: GET 2022-09-12T20:14:13.359 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:13.360 DEBUG:boto:Data: 2022-09-12T20:14:13.360 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.361 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.361 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.362 DEBUG:boto:Params: {} 2022-09-12T20:14:13.362 DEBUG:boto:Token: None 2022-09-12T20:14:13.363 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:13.363 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QCZK5BvThWo889ekO8Up3koss0s= 2022-09-12T20:14:13.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QCZK5BvThWo889ekO8Up3koss0s=', 'Content-Length': '0'} 2022-09-12T20:14:13.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b03a0c53a5190c96-00631f9315-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.367 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:13.368 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:13.368 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:13.369 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:13.369 DEBUG:boto:Method: GET 2022-09-12T20:14:13.370 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:13.370 DEBUG:boto:Data: 2022-09-12T20:14:13.371 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.371 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.372 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.372 DEBUG:boto:Params: {} 2022-09-12T20:14:13.373 DEBUG:boto:Token: None 2022-09-12T20:14:13.374 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:13.374 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QCZK5BvThWo889ekO8Up3koss0s= 2022-09-12T20:14:13.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QCZK5BvThWo889ekO8Up3koss0s=', 'Content-Length': '0'} 2022-09-12T20:14:13.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000668efe58d17e8085-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.378 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:13.379 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:13.379 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:13.380 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:13.380 DEBUG:boto:Method: HEAD 2022-09-12T20:14:13.381 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:13.381 DEBUG:boto:Data: 2022-09-12T20:14:13.382 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.382 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.383 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.383 DEBUG:boto:Params: {} 2022-09-12T20:14:13.384 DEBUG:boto:Token: None 2022-09-12T20:14:13.384 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:13.385 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Vp1XUxrjdr3/o3dcEyTUw4owc5U= 2022-09-12T20:14:13.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Vp1XUxrjdr3/o3dcEyTUw4owc5U=', 'Content-Length': '0'} 2022-09-12T20:14:13.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dba44fec927c509-00631f9315-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.388 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:13.389 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:13.389 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:13.390 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:13.390 DEBUG:boto:Method: HEAD 2022-09-12T20:14:13.391 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:13.391 DEBUG:boto:Data: 2022-09-12T20:14:13.392 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.392 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.393 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.393 DEBUG:boto:Params: {} 2022-09-12T20:14:13.394 DEBUG:boto:Token: None 2022-09-12T20:14:13.394 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:13.395 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Vp1XUxrjdr3/o3dcEyTUw4owc5U= 2022-09-12T20:14:13.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Vp1XUxrjdr3/o3dcEyTUw4owc5U=', 'Content-Length': '0'} 2022-09-12T20:14:13.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d52f18aeb955980-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:14:13.399 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:13.399 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:13.400 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:13.400 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:13.401 DEBUG:boto:Method: GET 2022-09-12T20:14:13.401 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:13.402 DEBUG:boto:Data: 2022-09-12T20:14:13.402 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.403 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.404 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.404 DEBUG:boto:Params: {} 2022-09-12T20:14:13.405 DEBUG:boto:Token: None 2022-09-12T20:14:13.405 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:13.406 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QCZK5BvThWo889ekO8Up3koss0s= 2022-09-12T20:14:13.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QCZK5BvThWo889ekO8Up3koss0s=', 'Content-Length': '0'} 2022-09-12T20:14:13.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d71001515b508325-00631f9315-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.450 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:13.451 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:13.451 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:13.452 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:13.452 DEBUG:boto:Method: GET 2022-09-12T20:14:13.453 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:13.453 DEBUG:boto:Data: 2022-09-12T20:14:13.454 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.454 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.455 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.456 DEBUG:boto:Params: {} 2022-09-12T20:14:13.456 DEBUG:boto:Token: None 2022-09-12T20:14:13.457 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:13.457 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QCZK5BvThWo889ekO8Up3koss0s= 2022-09-12T20:14:13.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QCZK5BvThWo889ekO8Up3koss0s=', 'Content-Length': '0'} 2022-09-12T20:14:13.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c63662181fedfebd-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.461 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:13.462 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:13.462 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:14:13.463 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:14:13.463 DEBUG:boto:Method: GET 2022-09-12T20:14:13.464 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:14:13.465 DEBUG:boto:Data: 2022-09-12T20:14:13.465 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.466 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.466 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.467 DEBUG:boto:Params: {} 2022-09-12T20:14:13.467 DEBUG:boto:Token: None 2022-09-12T20:14:13.468 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/ 2022-09-12T20:14:13.469 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw= 2022-09-12T20:14:13.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw=', 'Content-Length': '0'} 2022-09-12T20:14:13.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae7c9947327663b0-00631f9315-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.473 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:13.473 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:13.474 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:13.475 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:14:13.475 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:14:13.476 DEBUG:boto:Method: GET 2022-09-12T20:14:13.476 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:14:13.477 DEBUG:boto:Data: 2022-09-12T20:14:13.477 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.478 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.478 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.479 DEBUG:boto:Params: {} 2022-09-12T20:14:13.479 DEBUG:boto:Token: None 2022-09-12T20:14:13.480 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/ 2022-09-12T20:14:13.481 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw= 2022-09-12T20:14:13.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw=', 'Content-Length': '0'} 2022-09-12T20:14:13.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f977c71be786b41e-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.487 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:13.487 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:13.488 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:13.488 DEBUG:boto:Method: HEAD 2022-09-12T20:14:13.489 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:14:13.489 DEBUG:boto:Data: 2022-09-12T20:14:13.490 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.491 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.491 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.492 DEBUG:boto:Params: {} 2022-09-12T20:14:13.492 DEBUG:boto:Token: None 2022-09-12T20:14:13.493 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj116 2022-09-12T20:14:13.493 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R0TrbDRaeim5O9eTuCjchI5Qrnk= 2022-09-12T20:14:13.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R0TrbDRaeim5O9eTuCjchI5Qrnk=', 'Content-Length': '0'} 2022-09-12T20:14:13.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df2e0801116a8a59-00631f9315-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.497 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:13.497 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:13.498 DEBUG:boto:Method: HEAD 2022-09-12T20:14:13.498 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:14:13.499 DEBUG:boto:Data: 2022-09-12T20:14:13.499 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.500 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.500 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.501 DEBUG:boto:Params: {} 2022-09-12T20:14:13.501 DEBUG:boto:Token: None 2022-09-12T20:14:13.502 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj116 2022-09-12T20:14:13.502 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:R0TrbDRaeim5O9eTuCjchI5Qrnk= 2022-09-12T20:14:13.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:R0TrbDRaeim5O9eTuCjchI5Qrnk=', 'Content-Length': '0'} 2022-09-12T20:14:13.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfaf8a305c78290a-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:14:13.506 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:13.507 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:13.507 DEBUG:boto:Method: GET 2022-09-12T20:14:13.508 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:14:13.508 DEBUG:boto:Data: 2022-09-12T20:14:13.509 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.510 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.510 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.511 DEBUG:boto:Params: {} 2022-09-12T20:14:13.511 DEBUG:boto:Token: None 2022-09-12T20:14:13.512 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj116 2022-09-12T20:14:13.512 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wmoMo/aC98MQ5N9Xqy4DoZSfdfU= 2022-09-12T20:14:13.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wmoMo/aC98MQ5N9Xqy4DoZSfdfU=', 'Content-Length': '0'} 2022-09-12T20:14:13.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c8b10b749c2d599-00631f9315-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.516 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:13.516 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:13.517 DEBUG:boto:Method: GET 2022-09-12T20:14:13.517 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:14:13.518 DEBUG:boto:Data: 2022-09-12T20:14:13.519 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.519 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.520 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.520 DEBUG:boto:Params: {} 2022-09-12T20:14:13.521 DEBUG:boto:Token: None 2022-09-12T20:14:13.521 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj116 2022-09-12T20:14:13.522 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wmoMo/aC98MQ5N9Xqy4DoZSfdfU= 2022-09-12T20:14:13.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wmoMo/aC98MQ5N9Xqy4DoZSfdfU=', 'Content-Length': '0'} 2022-09-12T20:14:13.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa7664d37bd532ae-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.566 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:13.567 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:13.567 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:14:13.568 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:14:13.568 DEBUG:boto:Method: GET 2022-09-12T20:14:13.569 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:14:13.569 DEBUG:boto:Data: 2022-09-12T20:14:13.570 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.570 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.571 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.572 DEBUG:boto:Params: {} 2022-09-12T20:14:13.572 DEBUG:boto:Token: None 2022-09-12T20:14:13.573 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/ 2022-09-12T20:14:13.573 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw= 2022-09-12T20:14:13.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw=', 'Content-Length': '0'} 2022-09-12T20:14:13.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002553f952c2dbf2ee-00631f9315-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.578 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:13.578 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:13.579 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:13.580 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:14:13.580 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:14:13.581 DEBUG:boto:Method: GET 2022-09-12T20:14:13.581 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:14:13.582 DEBUG:boto:Data: 2022-09-12T20:14:13.582 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.583 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.583 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.584 DEBUG:boto:Params: {} 2022-09-12T20:14:13.584 DEBUG:boto:Token: None 2022-09-12T20:14:13.585 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/ 2022-09-12T20:14:13.586 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw= 2022-09-12T20:14:13.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw=', 'Content-Length': '0'} 2022-09-12T20:14:13.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d23b4f849ed931ee-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.591 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:13.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-12T20:14:13.592 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:13.593 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:13.593 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:13.594 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:13.594 DEBUG:boto:Method: GET 2022-09-12T20:14:13.595 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:13.595 DEBUG:boto:Data: 2022-09-12T20:14:13.596 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.596 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.597 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.598 DEBUG:boto:Params: {} 2022-09-12T20:14:13.598 DEBUG:boto:Token: None 2022-09-12T20:14:13.599 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:13.599 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0N+8tyerGDWAQcoz9Sss2/S7R8Q= 2022-09-12T20:14:13.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0N+8tyerGDWAQcoz9Sss2/S7R8Q=', 'Content-Length': '0'} 2022-09-12T20:14:13.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0f195a7c56cfd40-00631f9315-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.603 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:13.603 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:13.604 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:13.605 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:13.605 DEBUG:boto:Method: GET 2022-09-12T20:14:13.606 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:13.606 DEBUG:boto:Data: 2022-09-12T20:14:13.607 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.607 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.608 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.608 DEBUG:boto:Params: {} 2022-09-12T20:14:13.609 DEBUG:boto:Token: None 2022-09-12T20:14:13.609 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:13.610 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0N+8tyerGDWAQcoz9Sss2/S7R8Q= 2022-09-12T20:14:13.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0N+8tyerGDWAQcoz9Sss2/S7R8Q=', 'Content-Length': '0'} 2022-09-12T20:14:13.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000908c31d206073c83-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.613 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:13.614 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:13.614 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:13.615 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:13.615 DEBUG:boto:Method: HEAD 2022-09-12T20:14:13.616 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:13.616 DEBUG:boto:Data: 2022-09-12T20:14:13.617 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.617 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.618 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.618 DEBUG:boto:Params: {} 2022-09-12T20:14:13.619 DEBUG:boto:Token: None 2022-09-12T20:14:13.620 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:13.620 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jUmcZHPrMgubphm7T1H8LsCFNzs= 2022-09-12T20:14:13.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jUmcZHPrMgubphm7T1H8LsCFNzs=', 'Content-Length': '0'} 2022-09-12T20:14:13.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a23478993ce3dacb-00631f9315-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.623 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:13.624 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:13.624 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:13.625 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:13.625 DEBUG:boto:Method: HEAD 2022-09-12T20:14:13.626 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:13.626 DEBUG:boto:Data: 2022-09-12T20:14:13.627 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.627 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.628 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.628 DEBUG:boto:Params: {} 2022-09-12T20:14:13.629 DEBUG:boto:Token: None 2022-09-12T20:14:13.630 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:13.630 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jUmcZHPrMgubphm7T1H8LsCFNzs= 2022-09-12T20:14:13.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jUmcZHPrMgubphm7T1H8LsCFNzs=', 'Content-Length': '0'} 2022-09-12T20:14:13.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d2ca757088aa631-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-12T20:14:13.634 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:13.635 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:13.635 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:13.636 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:13.636 DEBUG:boto:Method: GET 2022-09-12T20:14:13.637 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:13.637 DEBUG:boto:Data: 2022-09-12T20:14:13.638 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.639 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.639 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.640 DEBUG:boto:Params: {} 2022-09-12T20:14:13.640 DEBUG:boto:Token: None 2022-09-12T20:14:13.641 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:13.641 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0N+8tyerGDWAQcoz9Sss2/S7R8Q= 2022-09-12T20:14:13.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0N+8tyerGDWAQcoz9Sss2/S7R8Q=', 'Content-Length': '0'} 2022-09-12T20:14:13.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfbc2fa138c83b0b-00631f9315-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.686 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:13.687 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:13.687 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:13.688 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:13.689 DEBUG:boto:Method: GET 2022-09-12T20:14:13.689 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:13.690 DEBUG:boto:Data: 2022-09-12T20:14:13.690 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.691 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.692 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.692 DEBUG:boto:Params: {} 2022-09-12T20:14:13.693 DEBUG:boto:Token: None 2022-09-12T20:14:13.693 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:13.694 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0N+8tyerGDWAQcoz9Sss2/S7R8Q= 2022-09-12T20:14:13.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0N+8tyerGDWAQcoz9Sss2/S7R8Q=', 'Content-Length': '0'} 2022-09-12T20:14:13.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea727513d702d666-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.698 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:13.698 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:13.699 DEBUG:boto:path=/jdpiak-75/?prefix=obj124 2022-09-12T20:14:13.699 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj124 2022-09-12T20:14:13.700 DEBUG:boto:Method: GET 2022-09-12T20:14:13.701 DEBUG:boto:Path: /jdpiak-75/?prefix=obj124 2022-09-12T20:14:13.701 DEBUG:boto:Data: 2022-09-12T20:14:13.702 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.702 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.703 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.703 DEBUG:boto:Params: {} 2022-09-12T20:14:13.704 DEBUG:boto:Token: None 2022-09-12T20:14:13.705 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/ 2022-09-12T20:14:13.705 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw= 2022-09-12T20:14:13.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw=', 'Content-Length': '0'} 2022-09-12T20:14:13.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c045372cff414fb7-00631f9315-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.709 DEBUG:boto:b'jdpiak-75obj1241000falseobj1242022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:13.710 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:13.711 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:13.711 DEBUG:boto:path=/jdpiak-75/?prefix=obj124 2022-09-12T20:14:13.712 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj124 2022-09-12T20:14:13.712 DEBUG:boto:Method: GET 2022-09-12T20:14:13.713 DEBUG:boto:Path: /jdpiak-75/?prefix=obj124 2022-09-12T20:14:13.713 DEBUG:boto:Data: 2022-09-12T20:14:13.714 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.714 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.715 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.715 DEBUG:boto:Params: {} 2022-09-12T20:14:13.716 DEBUG:boto:Token: None 2022-09-12T20:14:13.717 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/ 2022-09-12T20:14:13.717 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw= 2022-09-12T20:14:13.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw=', 'Content-Length': '0'} 2022-09-12T20:14:13.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005708454863d98053-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.729 DEBUG:boto:b'jdpiak-75obj1241000falseobj1242022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:13.730 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:13.730 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:13.731 DEBUG:boto:Method: HEAD 2022-09-12T20:14:13.731 DEBUG:boto:Path: /jdpiak-75/obj124 2022-09-12T20:14:13.732 DEBUG:boto:Data: 2022-09-12T20:14:13.732 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.733 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.733 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.734 DEBUG:boto:Params: {} 2022-09-12T20:14:13.734 DEBUG:boto:Token: None 2022-09-12T20:14:13.735 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj124 2022-09-12T20:14:13.736 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Mq5Jc1rGZf9hDcBL8iiW5MfO6TE= 2022-09-12T20:14:13.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Mq5Jc1rGZf9hDcBL8iiW5MfO6TE=', 'Content-Length': '0'} 2022-09-12T20:14:13.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ad3c99ba1658573-00631f9315-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.739 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:13.740 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:13.740 DEBUG:boto:Method: HEAD 2022-09-12T20:14:13.741 DEBUG:boto:Path: /jdpiak-75/obj124 2022-09-12T20:14:13.741 DEBUG:boto:Data: 2022-09-12T20:14:13.742 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.743 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.743 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.744 DEBUG:boto:Params: {} 2022-09-12T20:14:13.744 DEBUG:boto:Token: None 2022-09-12T20:14:13.745 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj124 2022-09-12T20:14:13.745 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Mq5Jc1rGZf9hDcBL8iiW5MfO6TE= 2022-09-12T20:14:13.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Mq5Jc1rGZf9hDcBL8iiW5MfO6TE=', 'Content-Length': '0'} 2022-09-12T20:14:13.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005135f31614fde04-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-12T20:14:13.750 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:13.750 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:13.751 DEBUG:boto:Method: GET 2022-09-12T20:14:13.751 DEBUG:boto:Path: /jdpiak-75/obj124 2022-09-12T20:14:13.752 DEBUG:boto:Data: 2022-09-12T20:14:13.752 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.753 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.754 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.754 DEBUG:boto:Params: {} 2022-09-12T20:14:13.755 DEBUG:boto:Token: None 2022-09-12T20:14:13.756 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj124 2022-09-12T20:14:13.757 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bBLPRiPtx2uIgvtY9djLZZutDQU= 2022-09-12T20:14:13.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bBLPRiPtx2uIgvtY9djLZZutDQU=', 'Content-Length': '0'} 2022-09-12T20:14:13.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd8dab5249a29390-00631f9315-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.760 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:13.761 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:13.762 DEBUG:boto:Method: GET 2022-09-12T20:14:13.762 DEBUG:boto:Path: /jdpiak-75/obj124 2022-09-12T20:14:13.763 DEBUG:boto:Data: 2022-09-12T20:14:13.763 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.764 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.764 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.765 DEBUG:boto:Params: {} 2022-09-12T20:14:13.765 DEBUG:boto:Token: None 2022-09-12T20:14:13.766 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj124 2022-09-12T20:14:13.767 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bBLPRiPtx2uIgvtY9djLZZutDQU= 2022-09-12T20:14:13.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bBLPRiPtx2uIgvtY9djLZZutDQU=', 'Content-Length': '0'} 2022-09-12T20:14:13.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073073a13f2d0c403-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.810 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:13.811 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:13.811 DEBUG:boto:path=/jdpiak-75/?prefix=obj124 2022-09-12T20:14:13.812 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj124 2022-09-12T20:14:13.812 DEBUG:boto:Method: GET 2022-09-12T20:14:13.813 DEBUG:boto:Path: /jdpiak-75/?prefix=obj124 2022-09-12T20:14:13.813 DEBUG:boto:Data: 2022-09-12T20:14:13.814 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.814 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.815 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.815 DEBUG:boto:Params: {} 2022-09-12T20:14:13.816 DEBUG:boto:Token: None 2022-09-12T20:14:13.817 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/ 2022-09-12T20:14:13.817 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw= 2022-09-12T20:14:13.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw=', 'Content-Length': '0'} 2022-09-12T20:14:13.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000618277ac72f547fa-00631f9315-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.822 DEBUG:boto:b'jdpiak-75obj1241000falseobj1242022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:13.823 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:13.823 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:13.824 DEBUG:boto:path=/jdpiak-75/?prefix=obj124 2022-09-12T20:14:13.824 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj124 2022-09-12T20:14:13.825 DEBUG:boto:Method: GET 2022-09-12T20:14:13.825 DEBUG:boto:Path: /jdpiak-75/?prefix=obj124 2022-09-12T20:14:13.826 DEBUG:boto:Data: 2022-09-12T20:14:13.827 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.827 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.828 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.828 DEBUG:boto:Params: {} 2022-09-12T20:14:13.829 DEBUG:boto:Token: None 2022-09-12T20:14:13.829 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/ 2022-09-12T20:14:13.830 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw= 2022-09-12T20:14:13.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw=', 'Content-Length': '0'} 2022-09-12T20:14:13.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000308d4b49565f14d8-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.836 DEBUG:boto:b'jdpiak-75obj1241000falseobj1242022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:13.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:14:13.837 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:13.838 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:13.839 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:13.839 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:13.840 DEBUG:boto:Method: GET 2022-09-12T20:14:13.840 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:13.841 DEBUG:boto:Data: 2022-09-12T20:14:13.841 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.842 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.842 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.843 DEBUG:boto:Params: {} 2022-09-12T20:14:13.844 DEBUG:boto:Token: None 2022-09-12T20:14:13.844 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:13.845 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:INBA7n9fTIn0QweJsefAlZ4DV00= 2022-09-12T20:14:13.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:INBA7n9fTIn0QweJsefAlZ4DV00=', 'Content-Length': '0'} 2022-09-12T20:14:13.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023198b37b7a04bd7-00631f9315-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.848 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:13.849 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:13.850 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:13.850 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:13.851 DEBUG:boto:Method: GET 2022-09-12T20:14:13.851 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:13.852 DEBUG:boto:Data: 2022-09-12T20:14:13.852 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.853 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.853 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.854 DEBUG:boto:Params: {} 2022-09-12T20:14:13.855 DEBUG:boto:Token: None 2022-09-12T20:14:13.855 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:13.856 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:INBA7n9fTIn0QweJsefAlZ4DV00= 2022-09-12T20:14:13.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:INBA7n9fTIn0QweJsefAlZ4DV00=', 'Content-Length': '0'} 2022-09-12T20:14:13.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c3551006fde10d2-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.859 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:13.860 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:13.860 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:13.861 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:13.861 DEBUG:boto:Method: HEAD 2022-09-12T20:14:13.862 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:13.863 DEBUG:boto:Data: 2022-09-12T20:14:13.863 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.864 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.864 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.865 DEBUG:boto:Params: {} 2022-09-12T20:14:13.865 DEBUG:boto:Token: None 2022-09-12T20:14:13.866 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:13.866 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l8YIOMJNu8ndy3sbDbsFm09V3m4= 2022-09-12T20:14:13.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l8YIOMJNu8ndy3sbDbsFm09V3m4=', 'Content-Length': '0'} 2022-09-12T20:14:13.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002789db1478f57d47-00631f9315-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.870 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:13.870 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:13.871 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:13.871 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:13.872 DEBUG:boto:Method: HEAD 2022-09-12T20:14:13.872 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:13.873 DEBUG:boto:Data: 2022-09-12T20:14:13.873 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.874 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.874 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.875 DEBUG:boto:Params: {} 2022-09-12T20:14:13.876 DEBUG:boto:Token: None 2022-09-12T20:14:13.876 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:13.877 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l8YIOMJNu8ndy3sbDbsFm09V3m4= 2022-09-12T20:14:13.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l8YIOMJNu8ndy3sbDbsFm09V3m4=', 'Content-Length': '0'} 2022-09-12T20:14:13.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000778389605b53da3a-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:14:13.881 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:13.882 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:13.882 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:13.883 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:13.883 DEBUG:boto:Method: GET 2022-09-12T20:14:13.884 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:13.885 DEBUG:boto:Data: 2022-09-12T20:14:13.886 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.886 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.887 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.888 DEBUG:boto:Params: {} 2022-09-12T20:14:13.888 DEBUG:boto:Token: None 2022-09-12T20:14:13.889 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:13.890 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:INBA7n9fTIn0QweJsefAlZ4DV00= 2022-09-12T20:14:13.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:INBA7n9fTIn0QweJsefAlZ4DV00=', 'Content-Length': '0'} 2022-09-12T20:14:13.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028932d370aaee3d0-00631f9315-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.894 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:13.894 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:13.895 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:13.895 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:13.896 DEBUG:boto:Method: GET 2022-09-12T20:14:13.896 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:13.897 DEBUG:boto:Data: 2022-09-12T20:14:13.897 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.898 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.898 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.899 DEBUG:boto:Params: {} 2022-09-12T20:14:13.900 DEBUG:boto:Token: None 2022-09-12T20:14:13.900 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:13.901 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:INBA7n9fTIn0QweJsefAlZ4DV00= 2022-09-12T20:14:13.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:INBA7n9fTIn0QweJsefAlZ4DV00=', 'Content-Length': '0'} 2022-09-12T20:14:13.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001960886fbbb91c0e-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.946 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:13.947 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:13.947 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:14:13.948 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:14:13.948 DEBUG:boto:Method: GET 2022-09-12T20:14:13.949 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:14:13.950 DEBUG:boto:Data: 2022-09-12T20:14:13.950 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.951 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.951 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.952 DEBUG:boto:Params: {} 2022-09-12T20:14:13.952 DEBUG:boto:Token: None 2022-09-12T20:14:13.953 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/ 2022-09-12T20:14:13.954 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw= 2022-09-12T20:14:13.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw=', 'Content-Length': '0'} 2022-09-12T20:14:13.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d711a54632b39c6-00631f9315-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.958 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:13.959 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:13.959 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:13.960 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:14:13.960 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:14:13.961 DEBUG:boto:Method: GET 2022-09-12T20:14:13.961 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:14:13.962 DEBUG:boto:Data: 2022-09-12T20:14:13.962 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.963 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.964 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.964 DEBUG:boto:Params: {} 2022-09-12T20:14:13.965 DEBUG:boto:Token: None 2022-09-12T20:14:13.966 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/ 2022-09-12T20:14:13.966 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw= 2022-09-12T20:14:13.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p1y/kXLjaZc2SwqYmPJ/qEQx/Iw=', 'Content-Length': '0'} 2022-09-12T20:14:13.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082578e34fb168b8f-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.972 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:13.972 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:13.973 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:13.973 DEBUG:boto:Method: HEAD 2022-09-12T20:14:13.974 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:14:13.974 DEBUG:boto:Data: 2022-09-12T20:14:13.975 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.975 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.976 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.977 DEBUG:boto:Params: {} 2022-09-12T20:14:13.977 DEBUG:boto:Token: None 2022-09-12T20:14:13.978 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj145 2022-09-12T20:14:13.978 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:56RfcJNo39yUu9cIzkSJRwH9pt8= 2022-09-12T20:14:13.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:56RfcJNo39yUu9cIzkSJRwH9pt8=', 'Content-Length': '0'} 2022-09-12T20:14:13.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007979aa245f3da0e5-00631f9315-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.982 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:13.982 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:13.983 DEBUG:boto:Method: HEAD 2022-09-12T20:14:13.983 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:14:13.984 DEBUG:boto:Data: 2022-09-12T20:14:13.984 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.985 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:13.985 DEBUG:boto:Port: 8000 2022-09-12T20:14:13.986 DEBUG:boto:Params: {} 2022-09-12T20:14:13.986 DEBUG:boto:Token: None 2022-09-12T20:14:13.987 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj145 2022-09-12T20:14:13.987 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:56RfcJNo39yUu9cIzkSJRwH9pt8= 2022-09-12T20:14:13.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:56RfcJNo39yUu9cIzkSJRwH9pt8=', 'Content-Length': '0'} 2022-09-12T20:14:13.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d640e86ea5b727b2-00631f9315-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:13.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:14:13.991 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:13.992 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:13.992 DEBUG:boto:Method: GET 2022-09-12T20:14:13.993 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:14:13.993 DEBUG:boto:Data: 2022-09-12T20:14:13.994 DEBUG:boto:Headers: {} 2022-09-12T20:14:13.995 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:13.995 DEBUG:boto:Port: 8001 2022-09-12T20:14:13.996 DEBUG:boto:Params: {} 2022-09-12T20:14:13.996 DEBUG:boto:Token: None 2022-09-12T20:14:13.997 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:13 GMT /jdpiak-75/obj145 2022-09-12T20:14:13.997 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:e0IoZ7vUbGMvZdoxi2mF51KyZhk= 2022-09-12T20:14:13.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:e0IoZ7vUbGMvZdoxi2mF51KyZhk=', 'Content-Length': '0'} 2022-09-12T20:14:14.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000515e5b62ce0b6dad-00631f9315-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.001 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:14.001 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:14.002 DEBUG:boto:Method: GET 2022-09-12T20:14:14.002 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:14:14.003 DEBUG:boto:Data: 2022-09-12T20:14:14.004 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.004 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.005 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.005 DEBUG:boto:Params: {} 2022-09-12T20:14:14.006 DEBUG:boto:Token: None 2022-09-12T20:14:14.007 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj145 2022-09-12T20:14:14.007 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DJfydE8t3yL/2ec9txToztbJEF0= 2022-09-12T20:14:14.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DJfydE8t3yL/2ec9txToztbJEF0=', 'Content-Length': '0'} 2022-09-12T20:14:14.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c33fe16da0d58aea-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.054 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:14.055 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:14.055 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:14:14.056 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:14:14.056 DEBUG:boto:Method: GET 2022-09-12T20:14:14.057 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:14:14.058 DEBUG:boto:Data: 2022-09-12T20:14:14.058 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.059 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.059 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.060 DEBUG:boto:Params: {} 2022-09-12T20:14:14.061 DEBUG:boto:Token: None 2022-09-12T20:14:14.061 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/ 2022-09-12T20:14:14.062 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0= 2022-09-12T20:14:14.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0=', 'Content-Length': '0'} 2022-09-12T20:14:14.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016ada0e0e783fcf3-00631f9316-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.066 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:14.067 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:14.068 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:14.068 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:14:14.069 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:14:14.069 DEBUG:boto:Method: GET 2022-09-12T20:14:14.070 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:14:14.070 DEBUG:boto:Data: 2022-09-12T20:14:14.071 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.071 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.072 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.072 DEBUG:boto:Params: {} 2022-09-12T20:14:14.073 DEBUG:boto:Token: None 2022-09-12T20:14:14.074 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/ 2022-09-12T20:14:14.074 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0= 2022-09-12T20:14:14.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0=', 'Content-Length': '0'} 2022-09-12T20:14:14.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f5c64c198452a5e-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.080 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:14.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-12T20:14:14.081 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:14.082 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:14.082 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:14.083 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:14.083 DEBUG:boto:Method: GET 2022-09-12T20:14:14.084 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:14.084 DEBUG:boto:Data: 2022-09-12T20:14:14.085 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.085 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.086 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.086 DEBUG:boto:Params: {} 2022-09-12T20:14:14.087 DEBUG:boto:Token: None 2022-09-12T20:14:14.088 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:14.088 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eA7bvBuQXXUQeYcaXpKPVKXPI2g= 2022-09-12T20:14:14.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eA7bvBuQXXUQeYcaXpKPVKXPI2g=', 'Content-Length': '0'} 2022-09-12T20:14:14.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031fa1eb9700684ae-00631f9316-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.092 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:14.093 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:14.093 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:14.094 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:14.095 DEBUG:boto:Method: GET 2022-09-12T20:14:14.095 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:14.096 DEBUG:boto:Data: 2022-09-12T20:14:14.096 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.097 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.098 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.098 DEBUG:boto:Params: {} 2022-09-12T20:14:14.099 DEBUG:boto:Token: None 2022-09-12T20:14:14.099 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:14.100 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eA7bvBuQXXUQeYcaXpKPVKXPI2g= 2022-09-12T20:14:14.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eA7bvBuQXXUQeYcaXpKPVKXPI2g=', 'Content-Length': '0'} 2022-09-12T20:14:14.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e1ac8c4e87c3f11-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.103 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:14.104 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:14.105 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:14.105 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:14.106 DEBUG:boto:Method: HEAD 2022-09-12T20:14:14.106 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:14.107 DEBUG:boto:Data: 2022-09-12T20:14:14.107 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.108 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.108 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.109 DEBUG:boto:Params: {} 2022-09-12T20:14:14.109 DEBUG:boto:Token: None 2022-09-12T20:14:14.110 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:14.110 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Hx0nQJCj6YQvp3axFt70TxZMDCg= 2022-09-12T20:14:14.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Hx0nQJCj6YQvp3axFt70TxZMDCg=', 'Content-Length': '0'} 2022-09-12T20:14:14.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0356af0252c6ae5-00631f9316-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.114 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:14.114 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:14.115 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:14.116 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:14.116 DEBUG:boto:Method: HEAD 2022-09-12T20:14:14.117 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:14.117 DEBUG:boto:Data: 2022-09-12T20:14:14.118 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.118 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.119 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.119 DEBUG:boto:Params: {} 2022-09-12T20:14:14.120 DEBUG:boto:Token: None 2022-09-12T20:14:14.120 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:14.121 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Hx0nQJCj6YQvp3axFt70TxZMDCg= 2022-09-12T20:14:14.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Hx0nQJCj6YQvp3axFt70TxZMDCg=', 'Content-Length': '0'} 2022-09-12T20:14:14.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f537977254952c43-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-12T20:14:14.125 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:14.126 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:14.126 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:14.127 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:14.127 DEBUG:boto:Method: GET 2022-09-12T20:14:14.128 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:14.128 DEBUG:boto:Data: 2022-09-12T20:14:14.129 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.130 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.130 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.131 DEBUG:boto:Params: {} 2022-09-12T20:14:14.131 DEBUG:boto:Token: None 2022-09-12T20:14:14.132 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:14.132 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eA7bvBuQXXUQeYcaXpKPVKXPI2g= 2022-09-12T20:14:14.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eA7bvBuQXXUQeYcaXpKPVKXPI2g=', 'Content-Length': '0'} 2022-09-12T20:14:14.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac4d0887b9a48b97-00631f9316-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.136 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:14.137 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:14.137 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:14.138 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:14.138 DEBUG:boto:Method: GET 2022-09-12T20:14:14.139 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:14.139 DEBUG:boto:Data: 2022-09-12T20:14:14.140 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.141 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.141 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.142 DEBUG:boto:Params: {} 2022-09-12T20:14:14.142 DEBUG:boto:Token: None 2022-09-12T20:14:14.143 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:14.144 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eA7bvBuQXXUQeYcaXpKPVKXPI2g= 2022-09-12T20:14:14.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eA7bvBuQXXUQeYcaXpKPVKXPI2g=', 'Content-Length': '0'} 2022-09-12T20:14:14.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e54d4168d0d11f0b-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.190 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:14.191 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:14.192 DEBUG:boto:path=/jdpiak-75/?prefix=obj155 2022-09-12T20:14:14.192 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj155 2022-09-12T20:14:14.193 DEBUG:boto:Method: GET 2022-09-12T20:14:14.193 DEBUG:boto:Path: /jdpiak-75/?prefix=obj155 2022-09-12T20:14:14.194 DEBUG:boto:Data: 2022-09-12T20:14:14.194 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.195 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.195 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.196 DEBUG:boto:Params: {} 2022-09-12T20:14:14.196 DEBUG:boto:Token: None 2022-09-12T20:14:14.197 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/ 2022-09-12T20:14:14.198 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0= 2022-09-12T20:14:14.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0=', 'Content-Length': '0'} 2022-09-12T20:14:14.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081dca068efde2873-00631f9316-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.202 DEBUG:boto:b'jdpiak-75obj1551000falseobj1552022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:14.203 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:14.203 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:14.204 DEBUG:boto:path=/jdpiak-75/?prefix=obj155 2022-09-12T20:14:14.204 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj155 2022-09-12T20:14:14.205 DEBUG:boto:Method: GET 2022-09-12T20:14:14.206 DEBUG:boto:Path: /jdpiak-75/?prefix=obj155 2022-09-12T20:14:14.206 DEBUG:boto:Data: 2022-09-12T20:14:14.207 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.207 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.208 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.208 DEBUG:boto:Params: {} 2022-09-12T20:14:14.209 DEBUG:boto:Token: None 2022-09-12T20:14:14.209 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/ 2022-09-12T20:14:14.210 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0= 2022-09-12T20:14:14.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0=', 'Content-Length': '0'} 2022-09-12T20:14:14.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e65669a8cd315c3-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.216 DEBUG:boto:b'jdpiak-75obj1551000falseobj1552022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:14.217 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:14.217 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:14.218 DEBUG:boto:Method: HEAD 2022-09-12T20:14:14.218 DEBUG:boto:Path: /jdpiak-75/obj155 2022-09-12T20:14:14.219 DEBUG:boto:Data: 2022-09-12T20:14:14.219 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.220 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.220 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.221 DEBUG:boto:Params: {} 2022-09-12T20:14:14.221 DEBUG:boto:Token: None 2022-09-12T20:14:14.222 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj155 2022-09-12T20:14:14.223 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H7Wax1CyXUbWUfHWgcNUd7opoNw= 2022-09-12T20:14:14.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H7Wax1CyXUbWUfHWgcNUd7opoNw=', 'Content-Length': '0'} 2022-09-12T20:14:14.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075ce29340caf461c-00631f9316-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.227 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:14.227 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:14.228 DEBUG:boto:Method: HEAD 2022-09-12T20:14:14.228 DEBUG:boto:Path: /jdpiak-75/obj155 2022-09-12T20:14:14.229 DEBUG:boto:Data: 2022-09-12T20:14:14.229 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.230 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.230 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.231 DEBUG:boto:Params: {} 2022-09-12T20:14:14.231 DEBUG:boto:Token: None 2022-09-12T20:14:14.232 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj155 2022-09-12T20:14:14.232 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H7Wax1CyXUbWUfHWgcNUd7opoNw= 2022-09-12T20:14:14.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H7Wax1CyXUbWUfHWgcNUd7opoNw=', 'Content-Length': '0'} 2022-09-12T20:14:14.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c42c0ba33ed3f2d-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-12T20:14:14.237 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:14.237 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:14.238 DEBUG:boto:Method: GET 2022-09-12T20:14:14.239 DEBUG:boto:Path: /jdpiak-75/obj155 2022-09-12T20:14:14.239 DEBUG:boto:Data: 2022-09-12T20:14:14.240 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.240 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.241 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.241 DEBUG:boto:Params: {} 2022-09-12T20:14:14.242 DEBUG:boto:Token: None 2022-09-12T20:14:14.243 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj155 2022-09-12T20:14:14.243 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:42/owo46LxbHcJcYDJ1Rt1/27uw= 2022-09-12T20:14:14.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:42/owo46LxbHcJcYDJ1Rt1/27uw=', 'Content-Length': '0'} 2022-09-12T20:14:14.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c07e750d5a76d2b1-00631f9316-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.247 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:14.247 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:14.248 DEBUG:boto:Method: GET 2022-09-12T20:14:14.248 DEBUG:boto:Path: /jdpiak-75/obj155 2022-09-12T20:14:14.249 DEBUG:boto:Data: 2022-09-12T20:14:14.249 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.250 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.251 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.251 DEBUG:boto:Params: {} 2022-09-12T20:14:14.252 DEBUG:boto:Token: None 2022-09-12T20:14:14.252 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj155 2022-09-12T20:14:14.253 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:42/owo46LxbHcJcYDJ1Rt1/27uw= 2022-09-12T20:14:14.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:42/owo46LxbHcJcYDJ1Rt1/27uw=', 'Content-Length': '0'} 2022-09-12T20:14:14.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bb57dd681dcda3a-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.257 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:14.257 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:14.258 DEBUG:boto:path=/jdpiak-75/?prefix=obj155 2022-09-12T20:14:14.258 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj155 2022-09-12T20:14:14.259 DEBUG:boto:Method: GET 2022-09-12T20:14:14.259 DEBUG:boto:Path: /jdpiak-75/?prefix=obj155 2022-09-12T20:14:14.260 DEBUG:boto:Data: 2022-09-12T20:14:14.260 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.261 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.261 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.262 DEBUG:boto:Params: {} 2022-09-12T20:14:14.262 DEBUG:boto:Token: None 2022-09-12T20:14:14.263 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/ 2022-09-12T20:14:14.264 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0= 2022-09-12T20:14:14.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0=', 'Content-Length': '0'} 2022-09-12T20:14:14.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ad9b096e84b104f-00631f9316-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.310 DEBUG:boto:b'jdpiak-75obj1551000falseobj1552022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:14.311 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:14.311 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:14.312 DEBUG:boto:path=/jdpiak-75/?prefix=obj155 2022-09-12T20:14:14.313 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj155 2022-09-12T20:14:14.313 DEBUG:boto:Method: GET 2022-09-12T20:14:14.314 DEBUG:boto:Path: /jdpiak-75/?prefix=obj155 2022-09-12T20:14:14.314 DEBUG:boto:Data: 2022-09-12T20:14:14.315 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.315 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.316 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.317 DEBUG:boto:Params: {} 2022-09-12T20:14:14.317 DEBUG:boto:Token: None 2022-09-12T20:14:14.318 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/ 2022-09-12T20:14:14.318 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0= 2022-09-12T20:14:14.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0=', 'Content-Length': '0'} 2022-09-12T20:14:14.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5ec68786fcf9a4f-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.324 DEBUG:boto:b'jdpiak-75obj1551000falseobj1552022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:14.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:14:14.326 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:14.326 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:14.327 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:14.327 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:14.328 DEBUG:boto:Method: GET 2022-09-12T20:14:14.329 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:14.329 DEBUG:boto:Data: 2022-09-12T20:14:14.330 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.330 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.331 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.331 DEBUG:boto:Params: {} 2022-09-12T20:14:14.332 DEBUG:boto:Token: None 2022-09-12T20:14:14.333 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:14.333 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cq76YRx+QJr5v0GUcjsx9QFObbY= 2022-09-12T20:14:14.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cq76YRx+QJr5v0GUcjsx9QFObbY=', 'Content-Length': '0'} 2022-09-12T20:14:14.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098c31efe8145994d-00631f9316-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.337 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:14.337 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:14.338 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:14.339 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:14.339 DEBUG:boto:Method: GET 2022-09-12T20:14:14.340 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:14.340 DEBUG:boto:Data: 2022-09-12T20:14:14.341 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.342 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.342 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.343 DEBUG:boto:Params: {} 2022-09-12T20:14:14.343 DEBUG:boto:Token: None 2022-09-12T20:14:14.344 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:14.345 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cq76YRx+QJr5v0GUcjsx9QFObbY= 2022-09-12T20:14:14.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cq76YRx+QJr5v0GUcjsx9QFObbY=', 'Content-Length': '0'} 2022-09-12T20:14:14.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064af1619b2ed1f63-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.390 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:14.391 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:14.391 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:14.392 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:14.392 DEBUG:boto:Method: HEAD 2022-09-12T20:14:14.393 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:14.393 DEBUG:boto:Data: 2022-09-12T20:14:14.394 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.394 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.395 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.395 DEBUG:boto:Params: {} 2022-09-12T20:14:14.396 DEBUG:boto:Token: None 2022-09-12T20:14:14.397 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:14.397 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Xwpp/NVqNP5DRc4YZUJJrgQ9L/s= 2022-09-12T20:14:14.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Xwpp/NVqNP5DRc4YZUJJrgQ9L/s=', 'Content-Length': '0'} 2022-09-12T20:14:14.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001265c61365c790a8-00631f9316-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.401 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:14.401 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:14.402 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:14.402 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:14.403 DEBUG:boto:Method: HEAD 2022-09-12T20:14:14.403 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:14.404 DEBUG:boto:Data: 2022-09-12T20:14:14.404 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.405 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.406 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.406 DEBUG:boto:Params: {} 2022-09-12T20:14:14.407 DEBUG:boto:Token: None 2022-09-12T20:14:14.407 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:14.408 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Xwpp/NVqNP5DRc4YZUJJrgQ9L/s= 2022-09-12T20:14:14.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Xwpp/NVqNP5DRc4YZUJJrgQ9L/s=', 'Content-Length': '0'} 2022-09-12T20:14:14.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020a95c959b55e883-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:14:14.412 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:14.412 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:14.413 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:14.413 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:14.414 DEBUG:boto:Method: GET 2022-09-12T20:14:14.414 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:14.415 DEBUG:boto:Data: 2022-09-12T20:14:14.416 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.416 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.417 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.417 DEBUG:boto:Params: {} 2022-09-12T20:14:14.418 DEBUG:boto:Token: None 2022-09-12T20:14:14.418 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:14.419 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cq76YRx+QJr5v0GUcjsx9QFObbY= 2022-09-12T20:14:14.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cq76YRx+QJr5v0GUcjsx9QFObbY=', 'Content-Length': '0'} 2022-09-12T20:14:14.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a15ba8398b96bfd2-00631f9316-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.423 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:14.423 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:14.424 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:14.424 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:14.425 DEBUG:boto:Method: GET 2022-09-12T20:14:14.426 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:14.426 DEBUG:boto:Data: 2022-09-12T20:14:14.427 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.427 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.428 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.428 DEBUG:boto:Params: {} 2022-09-12T20:14:14.429 DEBUG:boto:Token: None 2022-09-12T20:14:14.430 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:14.430 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cq76YRx+QJr5v0GUcjsx9QFObbY= 2022-09-12T20:14:14.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cq76YRx+QJr5v0GUcjsx9QFObbY=', 'Content-Length': '0'} 2022-09-12T20:14:14.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0cdec253b60985c-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.434 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:14.435 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:14.435 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:14:14.436 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:14:14.436 DEBUG:boto:Method: GET 2022-09-12T20:14:14.437 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:14:14.437 DEBUG:boto:Data: 2022-09-12T20:14:14.438 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.439 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.439 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.440 DEBUG:boto:Params: {} 2022-09-12T20:14:14.440 DEBUG:boto:Token: None 2022-09-12T20:14:14.441 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/ 2022-09-12T20:14:14.441 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0= 2022-09-12T20:14:14.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0=', 'Content-Length': '0'} 2022-09-12T20:14:14.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f194c244f470130-00631f9316-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.446 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:14.446 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:14.447 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:14.448 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:14:14.448 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:14:14.449 DEBUG:boto:Method: GET 2022-09-12T20:14:14.449 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:14:14.450 DEBUG:boto:Data: 2022-09-12T20:14:14.451 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.451 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.452 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.452 DEBUG:boto:Params: {} 2022-09-12T20:14:14.453 DEBUG:boto:Token: None 2022-09-12T20:14:14.454 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/ 2022-09-12T20:14:14.454 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0= 2022-09-12T20:14:14.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0=', 'Content-Length': '0'} 2022-09-12T20:14:14.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b141b993deff4b5-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.460 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:14.461 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:14.462 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:14.462 DEBUG:boto:Method: HEAD 2022-09-12T20:14:14.463 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:14:14.463 DEBUG:boto:Data: 2022-09-12T20:14:14.464 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.464 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.465 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.465 DEBUG:boto:Params: {} 2022-09-12T20:14:14.466 DEBUG:boto:Token: None 2022-09-12T20:14:14.467 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj17 2022-09-12T20:14:14.467 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:loWO7ADWQqXD0YLfDkJgDDNiBXk= 2022-09-12T20:14:14.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:loWO7ADWQqXD0YLfDkJgDDNiBXk=', 'Content-Length': '0'} 2022-09-12T20:14:14.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038f16ecc817d6a51-00631f9316-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.471 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:14.471 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:14.472 DEBUG:boto:Method: HEAD 2022-09-12T20:14:14.473 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:14:14.473 DEBUG:boto:Data: 2022-09-12T20:14:14.474 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.475 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.475 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.476 DEBUG:boto:Params: {} 2022-09-12T20:14:14.477 DEBUG:boto:Token: None 2022-09-12T20:14:14.478 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj17 2022-09-12T20:14:14.478 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:loWO7ADWQqXD0YLfDkJgDDNiBXk= 2022-09-12T20:14:14.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:loWO7ADWQqXD0YLfDkJgDDNiBXk=', 'Content-Length': '0'} 2022-09-12T20:14:14.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002237ccc9f1ecf523-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:14:14.483 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:14.483 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:14.484 DEBUG:boto:Method: GET 2022-09-12T20:14:14.485 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:14:14.485 DEBUG:boto:Data: 2022-09-12T20:14:14.486 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.486 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.487 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.488 DEBUG:boto:Params: {} 2022-09-12T20:14:14.488 DEBUG:boto:Token: None 2022-09-12T20:14:14.489 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj17 2022-09-12T20:14:14.490 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2ooohRrfiZ2gHq4YaD7V5bdMEeM= 2022-09-12T20:14:14.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2ooohRrfiZ2gHq4YaD7V5bdMEeM=', 'Content-Length': '0'} 2022-09-12T20:14:14.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006246cc2ba2f2f41f-00631f9316-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.534 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:14.535 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:14.536 DEBUG:boto:Method: GET 2022-09-12T20:14:14.536 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:14:14.537 DEBUG:boto:Data: 2022-09-12T20:14:14.538 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.538 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.539 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.539 DEBUG:boto:Params: {} 2022-09-12T20:14:14.540 DEBUG:boto:Token: None 2022-09-12T20:14:14.541 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj17 2022-09-12T20:14:14.541 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2ooohRrfiZ2gHq4YaD7V5bdMEeM= 2022-09-12T20:14:14.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2ooohRrfiZ2gHq4YaD7V5bdMEeM=', 'Content-Length': '0'} 2022-09-12T20:14:14.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032a5e37087570744-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.546 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:14.546 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:14.547 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:14:14.548 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:14:14.548 DEBUG:boto:Method: GET 2022-09-12T20:14:14.549 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:14:14.549 DEBUG:boto:Data: 2022-09-12T20:14:14.550 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.551 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.551 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.552 DEBUG:boto:Params: {} 2022-09-12T20:14:14.552 DEBUG:boto:Token: None 2022-09-12T20:14:14.553 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/ 2022-09-12T20:14:14.554 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0= 2022-09-12T20:14:14.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0=', 'Content-Length': '0'} 2022-09-12T20:14:14.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028e4ebbef6cf8f74-00631f9316-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.559 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:14.559 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:14.560 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:14.561 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:14:14.561 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:14:14.562 DEBUG:boto:Method: GET 2022-09-12T20:14:14.562 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:14:14.563 DEBUG:boto:Data: 2022-09-12T20:14:14.564 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.564 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.565 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.565 DEBUG:boto:Params: {} 2022-09-12T20:14:14.566 DEBUG:boto:Token: None 2022-09-12T20:14:14.567 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/ 2022-09-12T20:14:14.567 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0= 2022-09-12T20:14:14.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0=', 'Content-Length': '0'} 2022-09-12T20:14:14.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d30396a1d5277e0-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.573 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:14.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:14:14.574 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:14.575 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:14.575 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:14.576 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:14.577 DEBUG:boto:Method: GET 2022-09-12T20:14:14.577 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:14.578 DEBUG:boto:Data: 2022-09-12T20:14:14.578 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.579 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.580 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.580 DEBUG:boto:Params: {} 2022-09-12T20:14:14.581 DEBUG:boto:Token: None 2022-09-12T20:14:14.582 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:14.582 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y3B24apqrIJZHzU4C/wQi1IhqNI= 2022-09-12T20:14:14.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y3B24apqrIJZHzU4C/wQi1IhqNI=', 'Content-Length': '0'} 2022-09-12T20:14:14.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062c06126fd54b7cf-00631f9316-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.586 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:14.587 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:14.587 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:14.588 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:14.588 DEBUG:boto:Method: GET 2022-09-12T20:14:14.589 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:14.589 DEBUG:boto:Data: 2022-09-12T20:14:14.590 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.591 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.591 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.592 DEBUG:boto:Params: {} 2022-09-12T20:14:14.593 DEBUG:boto:Token: None 2022-09-12T20:14:14.594 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:14.595 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y3B24apqrIJZHzU4C/wQi1IhqNI= 2022-09-12T20:14:14.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y3B24apqrIJZHzU4C/wQi1IhqNI=', 'Content-Length': '0'} 2022-09-12T20:14:14.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021bbdfb2e3c7a6fe-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.600 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:14.600 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:14.601 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:14.602 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:14.602 DEBUG:boto:Method: HEAD 2022-09-12T20:14:14.603 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:14.604 DEBUG:boto:Data: 2022-09-12T20:14:14.604 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.605 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.606 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.607 DEBUG:boto:Params: {} 2022-09-12T20:14:14.607 DEBUG:boto:Token: None 2022-09-12T20:14:14.608 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:14.609 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gE8U3cWRICSCFcmBG68fSM6+jj8= 2022-09-12T20:14:14.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gE8U3cWRICSCFcmBG68fSM6+jj8=', 'Content-Length': '0'} 2022-09-12T20:14:14.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f8fe1595fe8a474-00631f9316-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.612 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:14.613 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:14.613 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:14.614 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:14.615 DEBUG:boto:Method: HEAD 2022-09-12T20:14:14.615 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:14.616 DEBUG:boto:Data: 2022-09-12T20:14:14.616 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.617 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.618 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.618 DEBUG:boto:Params: {} 2022-09-12T20:14:14.619 DEBUG:boto:Token: None 2022-09-12T20:14:14.620 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:14.620 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gE8U3cWRICSCFcmBG68fSM6+jj8= 2022-09-12T20:14:14.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gE8U3cWRICSCFcmBG68fSM6+jj8=', 'Content-Length': '0'} 2022-09-12T20:14:14.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d2e27190d5bc6a4-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:14:14.624 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:14.625 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:14.625 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:14.626 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:14.627 DEBUG:boto:Method: GET 2022-09-12T20:14:14.627 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:14.628 DEBUG:boto:Data: 2022-09-12T20:14:14.628 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.629 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.629 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.630 DEBUG:boto:Params: {} 2022-09-12T20:14:14.630 DEBUG:boto:Token: None 2022-09-12T20:14:14.631 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:14.631 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y3B24apqrIJZHzU4C/wQi1IhqNI= 2022-09-12T20:14:14.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y3B24apqrIJZHzU4C/wQi1IhqNI=', 'Content-Length': '0'} 2022-09-12T20:14:14.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b7447adf04d2d4b-00631f9316-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.635 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:14.635 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:14.636 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:14.636 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:14.637 DEBUG:boto:Method: GET 2022-09-12T20:14:14.637 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:14.638 DEBUG:boto:Data: 2022-09-12T20:14:14.639 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.639 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.640 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.640 DEBUG:boto:Params: {} 2022-09-12T20:14:14.641 DEBUG:boto:Token: None 2022-09-12T20:14:14.641 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:14.642 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y3B24apqrIJZHzU4C/wQi1IhqNI= 2022-09-12T20:14:14.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y3B24apqrIJZHzU4C/wQi1IhqNI=', 'Content-Length': '0'} 2022-09-12T20:14:14.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000284c018a8ac54d0-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.646 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:14.646 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:14.647 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:14:14.647 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:14:14.648 DEBUG:boto:Method: GET 2022-09-12T20:14:14.648 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:14:14.649 DEBUG:boto:Data: 2022-09-12T20:14:14.649 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.650 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.650 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.651 DEBUG:boto:Params: {} 2022-09-12T20:14:14.652 DEBUG:boto:Token: None 2022-09-12T20:14:14.652 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/ 2022-09-12T20:14:14.653 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0= 2022-09-12T20:14:14.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0=', 'Content-Length': '0'} 2022-09-12T20:14:14.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab747a832a5f2fec-00631f9316-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.698 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:14.699 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:14.700 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:14.700 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:14:14.701 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:14:14.701 DEBUG:boto:Method: GET 2022-09-12T20:14:14.702 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:14:14.702 DEBUG:boto:Data: 2022-09-12T20:14:14.703 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.703 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.704 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.704 DEBUG:boto:Params: {} 2022-09-12T20:14:14.705 DEBUG:boto:Token: None 2022-09-12T20:14:14.706 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/ 2022-09-12T20:14:14.706 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0= 2022-09-12T20:14:14.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0=', 'Content-Length': '0'} 2022-09-12T20:14:14.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093ffc1bd28141afb-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.712 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:14.713 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:14.714 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:14.714 DEBUG:boto:Method: HEAD 2022-09-12T20:14:14.715 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:14:14.715 DEBUG:boto:Data: 2022-09-12T20:14:14.716 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.716 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.717 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.717 DEBUG:boto:Params: {} 2022-09-12T20:14:14.718 DEBUG:boto:Token: None 2022-09-12T20:14:14.719 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj19 2022-09-12T20:14:14.719 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ip4LUIceu3ii41RDHSLqUALf8Dw= 2022-09-12T20:14:14.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ip4LUIceu3ii41RDHSLqUALf8Dw=', 'Content-Length': '0'} 2022-09-12T20:14:14.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c27690dde26a907b-00631f9316-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.723 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:14.723 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:14.724 DEBUG:boto:Method: HEAD 2022-09-12T20:14:14.724 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:14:14.725 DEBUG:boto:Data: 2022-09-12T20:14:14.725 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.726 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.726 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.727 DEBUG:boto:Params: {} 2022-09-12T20:14:14.727 DEBUG:boto:Token: None 2022-09-12T20:14:14.728 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj19 2022-09-12T20:14:14.729 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ip4LUIceu3ii41RDHSLqUALf8Dw= 2022-09-12T20:14:14.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ip4LUIceu3ii41RDHSLqUALf8Dw=', 'Content-Length': '0'} 2022-09-12T20:14:14.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3493090c7050201-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:14:14.732 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:14.733 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:14.733 DEBUG:boto:Method: GET 2022-09-12T20:14:14.734 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:14:14.735 DEBUG:boto:Data: 2022-09-12T20:14:14.735 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.736 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.736 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.737 DEBUG:boto:Params: {} 2022-09-12T20:14:14.737 DEBUG:boto:Token: None 2022-09-12T20:14:14.738 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj19 2022-09-12T20:14:14.738 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RVjWAE9AvX1PyRfejexnxMRmlOM= 2022-09-12T20:14:14.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RVjWAE9AvX1PyRfejexnxMRmlOM=', 'Content-Length': '0'} 2022-09-12T20:14:14.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd5cf0680be66b42-00631f9316-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.742 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:14.743 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:14.743 DEBUG:boto:Method: GET 2022-09-12T20:14:14.744 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:14:14.744 DEBUG:boto:Data: 2022-09-12T20:14:14.745 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.745 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.746 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.746 DEBUG:boto:Params: {} 2022-09-12T20:14:14.747 DEBUG:boto:Token: None 2022-09-12T20:14:14.748 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj19 2022-09-12T20:14:14.748 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RVjWAE9AvX1PyRfejexnxMRmlOM= 2022-09-12T20:14:14.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RVjWAE9AvX1PyRfejexnxMRmlOM=', 'Content-Length': '0'} 2022-09-12T20:14:14.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afbc959f4d16a00e-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.794 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:14.795 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:14.796 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:14:14.796 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:14:14.797 DEBUG:boto:Method: GET 2022-09-12T20:14:14.798 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:14:14.798 DEBUG:boto:Data: 2022-09-12T20:14:14.799 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.800 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.800 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.801 DEBUG:boto:Params: {} 2022-09-12T20:14:14.802 DEBUG:boto:Token: None 2022-09-12T20:14:14.803 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/ 2022-09-12T20:14:14.804 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0= 2022-09-12T20:14:14.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0=', 'Content-Length': '0'} 2022-09-12T20:14:14.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4ddef7a79b64355-00631f9316-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.809 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:14.810 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:14.810 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:14.811 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:14:14.812 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:14:14.812 DEBUG:boto:Method: GET 2022-09-12T20:14:14.813 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:14:14.814 DEBUG:boto:Data: 2022-09-12T20:14:14.814 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.815 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.816 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.816 DEBUG:boto:Params: {} 2022-09-12T20:14:14.817 DEBUG:boto:Token: None 2022-09-12T20:14:14.818 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/ 2022-09-12T20:14:14.819 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0= 2022-09-12T20:14:14.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0=', 'Content-Length': '0'} 2022-09-12T20:14:14.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059306ff3114b1752-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.825 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:14.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:14:14.826 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:14.827 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:14.827 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:14.828 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:14.828 DEBUG:boto:Method: GET 2022-09-12T20:14:14.829 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:14.830 DEBUG:boto:Data: 2022-09-12T20:14:14.830 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.831 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.832 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.833 DEBUG:boto:Params: {} 2022-09-12T20:14:14.833 DEBUG:boto:Token: None 2022-09-12T20:14:14.834 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:14.835 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z8vpVyyWeXdGGau5MBtjhPlHjxM= 2022-09-12T20:14:14.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z8vpVyyWeXdGGau5MBtjhPlHjxM=', 'Content-Length': '0'} 2022-09-12T20:14:14.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acfc2b568b148921-00631f9316-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.839 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:14.839 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:14.840 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:14.841 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:14.841 DEBUG:boto:Method: GET 2022-09-12T20:14:14.842 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:14.842 DEBUG:boto:Data: 2022-09-12T20:14:14.843 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.844 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.844 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.845 DEBUG:boto:Params: {} 2022-09-12T20:14:14.845 DEBUG:boto:Token: None 2022-09-12T20:14:14.846 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:14.847 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z8vpVyyWeXdGGau5MBtjhPlHjxM= 2022-09-12T20:14:14.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z8vpVyyWeXdGGau5MBtjhPlHjxM=', 'Content-Length': '0'} 2022-09-12T20:14:14.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ada43d533cfa279-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.851 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:14.852 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:14.852 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:14.853 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:14.853 DEBUG:boto:Method: HEAD 2022-09-12T20:14:14.854 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:14.854 DEBUG:boto:Data: 2022-09-12T20:14:14.855 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.855 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.856 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.857 DEBUG:boto:Params: {} 2022-09-12T20:14:14.857 DEBUG:boto:Token: None 2022-09-12T20:14:14.858 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:14.858 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F9N5WfV0bAOw0kTiWfBSty+1rz4= 2022-09-12T20:14:14.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F9N5WfV0bAOw0kTiWfBSty+1rz4=', 'Content-Length': '0'} 2022-09-12T20:14:14.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cdc84556f77c988-00631f9316-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.862 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:14.862 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:14.863 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:14.863 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:14.864 DEBUG:boto:Method: HEAD 2022-09-12T20:14:14.865 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:14.865 DEBUG:boto:Data: 2022-09-12T20:14:14.866 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.866 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.867 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.868 DEBUG:boto:Params: {} 2022-09-12T20:14:14.868 DEBUG:boto:Token: None 2022-09-12T20:14:14.869 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:14.869 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F9N5WfV0bAOw0kTiWfBSty+1rz4= 2022-09-12T20:14:14.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F9N5WfV0bAOw0kTiWfBSty+1rz4=', 'Content-Length': '0'} 2022-09-12T20:14:14.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e0de10b4a2d56e5-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:14:14.874 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:14.875 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:14.875 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:14.876 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:14.876 DEBUG:boto:Method: GET 2022-09-12T20:14:14.877 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:14.877 DEBUG:boto:Data: 2022-09-12T20:14:14.878 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.879 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.879 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.880 DEBUG:boto:Params: {} 2022-09-12T20:14:14.881 DEBUG:boto:Token: None 2022-09-12T20:14:14.881 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:14.882 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z8vpVyyWeXdGGau5MBtjhPlHjxM= 2022-09-12T20:14:14.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z8vpVyyWeXdGGau5MBtjhPlHjxM=', 'Content-Length': '0'} 2022-09-12T20:14:14.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ae9d7b830fd8b27-00631f9316-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.886 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:14.886 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:14.887 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:14.888 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:14.888 DEBUG:boto:Method: GET 2022-09-12T20:14:14.889 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:14.889 DEBUG:boto:Data: 2022-09-12T20:14:14.890 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.891 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.892 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.892 DEBUG:boto:Params: {} 2022-09-12T20:14:14.893 DEBUG:boto:Token: None 2022-09-12T20:14:14.893 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:14.894 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z8vpVyyWeXdGGau5MBtjhPlHjxM= 2022-09-12T20:14:14.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z8vpVyyWeXdGGau5MBtjhPlHjxM=', 'Content-Length': '0'} 2022-09-12T20:14:14.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000046999344fec2b34-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.938 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:14.939 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:14.940 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:14:14.940 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:14:14.941 DEBUG:boto:Method: GET 2022-09-12T20:14:14.942 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:14:14.942 DEBUG:boto:Data: 2022-09-12T20:14:14.943 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.943 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.944 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.944 DEBUG:boto:Params: {} 2022-09-12T20:14:14.945 DEBUG:boto:Token: None 2022-09-12T20:14:14.946 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/ 2022-09-12T20:14:14.947 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0= 2022-09-12T20:14:14.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0=', 'Content-Length': '0'} 2022-09-12T20:14:14.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5b9f539f040fc7b-00631f9316-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.951 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:14.952 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:14.952 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:14.953 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:14:14.953 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:14:14.954 DEBUG:boto:Method: GET 2022-09-12T20:14:14.954 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:14:14.955 DEBUG:boto:Data: 2022-09-12T20:14:14.955 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.956 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.957 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.957 DEBUG:boto:Params: {} 2022-09-12T20:14:14.958 DEBUG:boto:Token: None 2022-09-12T20:14:14.958 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/ 2022-09-12T20:14:14.959 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0= 2022-09-12T20:14:14.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VKBDMoeB237X7rJHF9Nkw+wtRK0=', 'Content-Length': '0'} 2022-09-12T20:14:14.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f8a4256bdebcf47-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.964 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:14.965 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:14.966 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:14.966 DEBUG:boto:Method: HEAD 2022-09-12T20:14:14.967 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:14:14.967 DEBUG:boto:Data: 2022-09-12T20:14:14.968 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.968 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.969 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.969 DEBUG:boto:Params: {} 2022-09-12T20:14:14.970 DEBUG:boto:Token: None 2022-09-12T20:14:14.970 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj22 2022-09-12T20:14:14.971 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ym3bSDirF9gm855lA3+tAJLiICQ= 2022-09-12T20:14:14.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ym3bSDirF9gm855lA3+tAJLiICQ=', 'Content-Length': '0'} 2022-09-12T20:14:14.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ada397afe7eba82c-00631f9316-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.974 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:14.975 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:14.975 DEBUG:boto:Method: HEAD 2022-09-12T20:14:14.976 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:14:14.976 DEBUG:boto:Data: 2022-09-12T20:14:14.977 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.978 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:14.978 DEBUG:boto:Port: 8000 2022-09-12T20:14:14.979 DEBUG:boto:Params: {} 2022-09-12T20:14:14.979 DEBUG:boto:Token: None 2022-09-12T20:14:14.980 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj22 2022-09-12T20:14:14.980 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ym3bSDirF9gm855lA3+tAJLiICQ= 2022-09-12T20:14:14.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ym3bSDirF9gm855lA3+tAJLiICQ=', 'Content-Length': '0'} 2022-09-12T20:14:14.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b37d3d34f888125-00631f9316-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:14.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:14:14.985 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:14.986 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:14.986 DEBUG:boto:Method: GET 2022-09-12T20:14:14.987 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:14:14.987 DEBUG:boto:Data: 2022-09-12T20:14:14.988 DEBUG:boto:Headers: {} 2022-09-12T20:14:14.988 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:14.989 DEBUG:boto:Port: 8001 2022-09-12T20:14:14.989 DEBUG:boto:Params: {} 2022-09-12T20:14:14.990 DEBUG:boto:Token: None 2022-09-12T20:14:14.991 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:14 GMT /jdpiak-75/obj22 2022-09-12T20:14:14.991 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kyRkgzGCk74r9umU1F70V2WsuKo= 2022-09-12T20:14:14.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kyRkgzGCk74r9umU1F70V2WsuKo=', 'Content-Length': '0'} 2022-09-12T20:14:14.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe0466adadd48efd-00631f9316-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.034 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:15.035 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:15.035 DEBUG:boto:Method: GET 2022-09-12T20:14:15.036 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:14:15.036 DEBUG:boto:Data: 2022-09-12T20:14:15.037 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.037 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.038 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.039 DEBUG:boto:Params: {} 2022-09-12T20:14:15.039 DEBUG:boto:Token: None 2022-09-12T20:14:15.040 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj22 2022-09-12T20:14:15.041 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZgmQXgNNAfxjyjCeEDb7wr6n0+g= 2022-09-12T20:14:15.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZgmQXgNNAfxjyjCeEDb7wr6n0+g=', 'Content-Length': '0'} 2022-09-12T20:14:15.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b418e784cdae3e1-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.045 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:15.045 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:15.046 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:14:15.046 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:14:15.047 DEBUG:boto:Method: GET 2022-09-12T20:14:15.047 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:14:15.048 DEBUG:boto:Data: 2022-09-12T20:14:15.048 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.049 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.049 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.050 DEBUG:boto:Params: {} 2022-09-12T20:14:15.050 DEBUG:boto:Token: None 2022-09-12T20:14:15.051 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/ 2022-09-12T20:14:15.052 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0= 2022-09-12T20:14:15.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0=', 'Content-Length': '0'} 2022-09-12T20:14:15.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba04490e76a255af-00631f9317-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.056 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:15.057 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:15.057 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:15.058 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:14:15.058 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:14:15.059 DEBUG:boto:Method: GET 2022-09-12T20:14:15.060 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:14:15.060 DEBUG:boto:Data: 2022-09-12T20:14:15.061 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.061 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.062 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.063 DEBUG:boto:Params: {} 2022-09-12T20:14:15.063 DEBUG:boto:Token: None 2022-09-12T20:14:15.064 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/ 2022-09-12T20:14:15.064 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0= 2022-09-12T20:14:15.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0=', 'Content-Length': '0'} 2022-09-12T20:14:15.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba47c83b94391ff6-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.070 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:15.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:14:15.071 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:15.072 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:15.072 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:15.073 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:15.073 DEBUG:boto:Method: GET 2022-09-12T20:14:15.074 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:15.074 DEBUG:boto:Data: 2022-09-12T20:14:15.075 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.075 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.076 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.076 DEBUG:boto:Params: {} 2022-09-12T20:14:15.077 DEBUG:boto:Token: None 2022-09-12T20:14:15.078 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:15.078 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TcGDrXslX66V91TEAaTnsnqwHNk= 2022-09-12T20:14:15.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TcGDrXslX66V91TEAaTnsnqwHNk=', 'Content-Length': '0'} 2022-09-12T20:14:15.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047bf57dda7470eb7-00631f9317-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.082 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:15.082 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:15.083 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:15.083 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:15.084 DEBUG:boto:Method: GET 2022-09-12T20:14:15.084 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:15.085 DEBUG:boto:Data: 2022-09-12T20:14:15.086 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.086 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.087 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.087 DEBUG:boto:Params: {} 2022-09-12T20:14:15.088 DEBUG:boto:Token: None 2022-09-12T20:14:15.088 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:15.089 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TcGDrXslX66V91TEAaTnsnqwHNk= 2022-09-12T20:14:15.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TcGDrXslX66V91TEAaTnsnqwHNk=', 'Content-Length': '0'} 2022-09-12T20:14:15.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb445535195dcf3e-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.093 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:15.093 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:15.094 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:15.094 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:15.095 DEBUG:boto:Method: HEAD 2022-09-12T20:14:15.095 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:15.096 DEBUG:boto:Data: 2022-09-12T20:14:15.096 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.097 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.097 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.098 DEBUG:boto:Params: {} 2022-09-12T20:14:15.098 DEBUG:boto:Token: None 2022-09-12T20:14:15.099 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:15.099 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uY0bSygFdKfgJUC9vpz1YG8qwcA= 2022-09-12T20:14:15.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uY0bSygFdKfgJUC9vpz1YG8qwcA=', 'Content-Length': '0'} 2022-09-12T20:14:15.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c77845953462b929-00631f9317-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.103 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:15.103 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:15.104 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:15.104 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:15.105 DEBUG:boto:Method: HEAD 2022-09-12T20:14:15.105 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:15.106 DEBUG:boto:Data: 2022-09-12T20:14:15.106 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.107 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.107 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.108 DEBUG:boto:Params: {} 2022-09-12T20:14:15.108 DEBUG:boto:Token: None 2022-09-12T20:14:15.109 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:15.109 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uY0bSygFdKfgJUC9vpz1YG8qwcA= 2022-09-12T20:14:15.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uY0bSygFdKfgJUC9vpz1YG8qwcA=', 'Content-Length': '0'} 2022-09-12T20:14:15.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000facc31cefc88f1c-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:14:15.113 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:15.114 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:15.114 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:15.115 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:15.115 DEBUG:boto:Method: GET 2022-09-12T20:14:15.116 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:15.116 DEBUG:boto:Data: 2022-09-12T20:14:15.117 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.117 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.118 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.119 DEBUG:boto:Params: {} 2022-09-12T20:14:15.119 DEBUG:boto:Token: None 2022-09-12T20:14:15.120 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:15.120 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TcGDrXslX66V91TEAaTnsnqwHNk= 2022-09-12T20:14:15.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TcGDrXslX66V91TEAaTnsnqwHNk=', 'Content-Length': '0'} 2022-09-12T20:14:15.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000703448b7b054a6e0-00631f9317-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.166 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:15.167 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:15.167 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:15.168 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:15.168 DEBUG:boto:Method: GET 2022-09-12T20:14:15.169 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:15.169 DEBUG:boto:Data: 2022-09-12T20:14:15.170 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.171 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.171 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.172 DEBUG:boto:Params: {} 2022-09-12T20:14:15.172 DEBUG:boto:Token: None 2022-09-12T20:14:15.173 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:15.173 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TcGDrXslX66V91TEAaTnsnqwHNk= 2022-09-12T20:14:15.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TcGDrXslX66V91TEAaTnsnqwHNk=', 'Content-Length': '0'} 2022-09-12T20:14:15.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005880558e40e20811-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.177 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:15.178 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:15.178 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:14:15.179 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:14:15.180 DEBUG:boto:Method: GET 2022-09-12T20:14:15.180 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:14:15.181 DEBUG:boto:Data: 2022-09-12T20:14:15.181 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.182 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.182 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.183 DEBUG:boto:Params: {} 2022-09-12T20:14:15.183 DEBUG:boto:Token: None 2022-09-12T20:14:15.184 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/ 2022-09-12T20:14:15.185 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0= 2022-09-12T20:14:15.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0=', 'Content-Length': '0'} 2022-09-12T20:14:15.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e3aab9342c273cb-00631f9317-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.188 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:15.189 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:15.190 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:15.190 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:14:15.191 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:14:15.191 DEBUG:boto:Method: GET 2022-09-12T20:14:15.192 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:14:15.192 DEBUG:boto:Data: 2022-09-12T20:14:15.193 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.193 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.194 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.194 DEBUG:boto:Params: {} 2022-09-12T20:14:15.195 DEBUG:boto:Token: None 2022-09-12T20:14:15.196 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/ 2022-09-12T20:14:15.196 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0= 2022-09-12T20:14:15.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0=', 'Content-Length': '0'} 2022-09-12T20:14:15.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fa8faa19f1bc824-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.202 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:15.203 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:15.203 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:15.204 DEBUG:boto:Method: HEAD 2022-09-12T20:14:15.204 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:14:15.205 DEBUG:boto:Data: 2022-09-12T20:14:15.205 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.206 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.206 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.207 DEBUG:boto:Params: {} 2022-09-12T20:14:15.207 DEBUG:boto:Token: None 2022-09-12T20:14:15.208 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj23 2022-09-12T20:14:15.208 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b7VhFTEjNzE8jS8NZdGmT0wkn9E= 2022-09-12T20:14:15.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b7VhFTEjNzE8jS8NZdGmT0wkn9E=', 'Content-Length': '0'} 2022-09-12T20:14:15.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068bd652e31877cf9-00631f9317-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.212 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:15.213 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:15.213 DEBUG:boto:Method: HEAD 2022-09-12T20:14:15.214 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:14:15.214 DEBUG:boto:Data: 2022-09-12T20:14:15.215 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.215 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.216 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.216 DEBUG:boto:Params: {} 2022-09-12T20:14:15.217 DEBUG:boto:Token: None 2022-09-12T20:14:15.217 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj23 2022-09-12T20:14:15.218 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:b7VhFTEjNzE8jS8NZdGmT0wkn9E= 2022-09-12T20:14:15.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:b7VhFTEjNzE8jS8NZdGmT0wkn9E=', 'Content-Length': '0'} 2022-09-12T20:14:15.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc21a258d7cfeb03-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:14:15.222 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:15.223 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:15.223 DEBUG:boto:Method: GET 2022-09-12T20:14:15.224 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:14:15.224 DEBUG:boto:Data: 2022-09-12T20:14:15.225 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.225 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.226 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.226 DEBUG:boto:Params: {} 2022-09-12T20:14:15.227 DEBUG:boto:Token: None 2022-09-12T20:14:15.227 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj23 2022-09-12T20:14:15.228 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zbdRSe8K9v42Gf21eJEWCdBr/lY= 2022-09-12T20:14:15.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zbdRSe8K9v42Gf21eJEWCdBr/lY=', 'Content-Length': '0'} 2022-09-12T20:14:15.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7ebe307df6088c0-00631f9317-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.231 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:15.232 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:15.232 DEBUG:boto:Method: GET 2022-09-12T20:14:15.233 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:14:15.233 DEBUG:boto:Data: 2022-09-12T20:14:15.234 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.234 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.235 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.235 DEBUG:boto:Params: {} 2022-09-12T20:14:15.236 DEBUG:boto:Token: None 2022-09-12T20:14:15.237 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj23 2022-09-12T20:14:15.237 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zbdRSe8K9v42Gf21eJEWCdBr/lY= 2022-09-12T20:14:15.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zbdRSe8K9v42Gf21eJEWCdBr/lY=', 'Content-Length': '0'} 2022-09-12T20:14:15.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039693292ed4a95ad-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.282 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:15.283 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:15.284 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:14:15.284 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:14:15.285 DEBUG:boto:Method: GET 2022-09-12T20:14:15.285 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:14:15.286 DEBUG:boto:Data: 2022-09-12T20:14:15.286 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.287 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.287 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.288 DEBUG:boto:Params: {} 2022-09-12T20:14:15.288 DEBUG:boto:Token: None 2022-09-12T20:14:15.289 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/ 2022-09-12T20:14:15.290 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0= 2022-09-12T20:14:15.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0=', 'Content-Length': '0'} 2022-09-12T20:14:15.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009df0ce19e1154f10-00631f9317-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.294 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:15.295 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:15.296 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:15.296 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:14:15.297 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:14:15.297 DEBUG:boto:Method: GET 2022-09-12T20:14:15.298 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:14:15.299 DEBUG:boto:Data: 2022-09-12T20:14:15.300 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.301 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.302 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.302 DEBUG:boto:Params: {} 2022-09-12T20:14:15.303 DEBUG:boto:Token: None 2022-09-12T20:14:15.303 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/ 2022-09-12T20:14:15.304 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0= 2022-09-12T20:14:15.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0=', 'Content-Length': '0'} 2022-09-12T20:14:15.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e3bb33d49faa8f0-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.309 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:15.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:14:15.311 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:15.311 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:15.312 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:15.312 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:15.313 DEBUG:boto:Method: GET 2022-09-12T20:14:15.313 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:15.314 DEBUG:boto:Data: 2022-09-12T20:14:15.314 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.315 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.316 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.316 DEBUG:boto:Params: {} 2022-09-12T20:14:15.317 DEBUG:boto:Token: None 2022-09-12T20:14:15.317 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:15.318 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ckHPYPDRBAJqGKIIkzbmKaIvfi8= 2022-09-12T20:14:15.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ckHPYPDRBAJqGKIIkzbmKaIvfi8=', 'Content-Length': '0'} 2022-09-12T20:14:15.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000083d9815e6a18276-00631f9317-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.321 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:15.322 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:15.323 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:15.323 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:15.324 DEBUG:boto:Method: GET 2022-09-12T20:14:15.325 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:15.325 DEBUG:boto:Data: 2022-09-12T20:14:15.326 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.326 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.327 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.327 DEBUG:boto:Params: {} 2022-09-12T20:14:15.328 DEBUG:boto:Token: None 2022-09-12T20:14:15.328 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:15.329 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ckHPYPDRBAJqGKIIkzbmKaIvfi8= 2022-09-12T20:14:15.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ckHPYPDRBAJqGKIIkzbmKaIvfi8=', 'Content-Length': '0'} 2022-09-12T20:14:15.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004ea41c44b5e9558-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.333 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:15.333 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:15.334 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:15.334 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:15.335 DEBUG:boto:Method: HEAD 2022-09-12T20:14:15.335 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:15.336 DEBUG:boto:Data: 2022-09-12T20:14:15.336 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.337 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.337 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.338 DEBUG:boto:Params: {} 2022-09-12T20:14:15.338 DEBUG:boto:Token: None 2022-09-12T20:14:15.339 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:15.340 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ztWCa3nW0rNx3VQq84/f6+ybXBI= 2022-09-12T20:14:15.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ztWCa3nW0rNx3VQq84/f6+ybXBI=', 'Content-Length': '0'} 2022-09-12T20:14:15.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007362d8d06f871c5e-00631f9317-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.343 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:15.343 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:15.344 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:15.344 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:15.345 DEBUG:boto:Method: HEAD 2022-09-12T20:14:15.345 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:15.346 DEBUG:boto:Data: 2022-09-12T20:14:15.346 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.347 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.347 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.348 DEBUG:boto:Params: {} 2022-09-12T20:14:15.348 DEBUG:boto:Token: None 2022-09-12T20:14:15.349 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:15.349 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ztWCa3nW0rNx3VQq84/f6+ybXBI= 2022-09-12T20:14:15.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ztWCa3nW0rNx3VQq84/f6+ybXBI=', 'Content-Length': '0'} 2022-09-12T20:14:15.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbe7275699f6b8ac-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:14:15.353 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:15.354 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:15.355 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:15.355 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:15.356 DEBUG:boto:Method: GET 2022-09-12T20:14:15.356 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:15.357 DEBUG:boto:Data: 2022-09-12T20:14:15.357 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.358 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.358 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.359 DEBUG:boto:Params: {} 2022-09-12T20:14:15.359 DEBUG:boto:Token: None 2022-09-12T20:14:15.360 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:15.360 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ckHPYPDRBAJqGKIIkzbmKaIvfi8= 2022-09-12T20:14:15.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ckHPYPDRBAJqGKIIkzbmKaIvfi8=', 'Content-Length': '0'} 2022-09-12T20:14:15.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d548c4dde3c9ed6f-00631f9317-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.364 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:15.365 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:15.365 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:15.366 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:15.366 DEBUG:boto:Method: GET 2022-09-12T20:14:15.367 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:15.367 DEBUG:boto:Data: 2022-09-12T20:14:15.368 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.368 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.369 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.369 DEBUG:boto:Params: {} 2022-09-12T20:14:15.370 DEBUG:boto:Token: None 2022-09-12T20:14:15.371 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:15.371 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ckHPYPDRBAJqGKIIkzbmKaIvfi8= 2022-09-12T20:14:15.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ckHPYPDRBAJqGKIIkzbmKaIvfi8=', 'Content-Length': '0'} 2022-09-12T20:14:15.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efd3b865aa9305f9-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.375 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:15.375 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:15.376 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:14:15.376 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:14:15.377 DEBUG:boto:Method: GET 2022-09-12T20:14:15.377 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:14:15.378 DEBUG:boto:Data: 2022-09-12T20:14:15.378 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.379 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.379 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.380 DEBUG:boto:Params: {} 2022-09-12T20:14:15.380 DEBUG:boto:Token: None 2022-09-12T20:14:15.381 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/ 2022-09-12T20:14:15.381 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0= 2022-09-12T20:14:15.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0=', 'Content-Length': '0'} 2022-09-12T20:14:15.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073c9f05bce604d9b-00631f9317-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.386 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:15.387 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:15.387 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:15.388 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:14:15.388 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:14:15.389 DEBUG:boto:Method: GET 2022-09-12T20:14:15.389 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:14:15.390 DEBUG:boto:Data: 2022-09-12T20:14:15.390 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.391 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.391 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.392 DEBUG:boto:Params: {} 2022-09-12T20:14:15.392 DEBUG:boto:Token: None 2022-09-12T20:14:15.393 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/ 2022-09-12T20:14:15.394 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0= 2022-09-12T20:14:15.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0=', 'Content-Length': '0'} 2022-09-12T20:14:15.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6548f98f9ef9570-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.442 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:15.443 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:15.444 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:15.445 DEBUG:boto:Method: HEAD 2022-09-12T20:14:15.445 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:14:15.446 DEBUG:boto:Data: 2022-09-12T20:14:15.447 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.447 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.448 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.448 DEBUG:boto:Params: {} 2022-09-12T20:14:15.449 DEBUG:boto:Token: None 2022-09-12T20:14:15.450 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj29 2022-09-12T20:14:15.451 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t781TkjSDeW2oPIrzYASVOEguRY= 2022-09-12T20:14:15.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t781TkjSDeW2oPIrzYASVOEguRY=', 'Content-Length': '0'} 2022-09-12T20:14:15.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000160459bf172f0671-00631f9317-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.455 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:15.456 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:15.456 DEBUG:boto:Method: HEAD 2022-09-12T20:14:15.457 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:14:15.458 DEBUG:boto:Data: 2022-09-12T20:14:15.458 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.459 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.459 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.460 DEBUG:boto:Params: {} 2022-09-12T20:14:15.460 DEBUG:boto:Token: None 2022-09-12T20:14:15.461 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj29 2022-09-12T20:14:15.461 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t781TkjSDeW2oPIrzYASVOEguRY= 2022-09-12T20:14:15.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t781TkjSDeW2oPIrzYASVOEguRY=', 'Content-Length': '0'} 2022-09-12T20:14:15.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002049a4840479be92-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:14:15.465 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:15.466 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:15.467 DEBUG:boto:Method: GET 2022-09-12T20:14:15.467 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:14:15.468 DEBUG:boto:Data: 2022-09-12T20:14:15.468 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.469 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.469 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.470 DEBUG:boto:Params: {} 2022-09-12T20:14:15.470 DEBUG:boto:Token: None 2022-09-12T20:14:15.471 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj29 2022-09-12T20:14:15.471 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WChyH7Eb56pCOrDktOJW8Q4SmJk= 2022-09-12T20:14:15.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WChyH7Eb56pCOrDktOJW8Q4SmJk=', 'Content-Length': '0'} 2022-09-12T20:14:15.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098c5ef4972ac2ffe-00631f9317-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.518 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:15.519 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:15.519 DEBUG:boto:Method: GET 2022-09-12T20:14:15.520 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:14:15.520 DEBUG:boto:Data: 2022-09-12T20:14:15.521 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.522 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.522 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.523 DEBUG:boto:Params: {} 2022-09-12T20:14:15.523 DEBUG:boto:Token: None 2022-09-12T20:14:15.524 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj29 2022-09-12T20:14:15.525 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WChyH7Eb56pCOrDktOJW8Q4SmJk= 2022-09-12T20:14:15.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WChyH7Eb56pCOrDktOJW8Q4SmJk=', 'Content-Length': '0'} 2022-09-12T20:14:15.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000466e8bf96d967176-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.529 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:15.529 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:15.530 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:14:15.530 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:14:15.531 DEBUG:boto:Method: GET 2022-09-12T20:14:15.531 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:14:15.532 DEBUG:boto:Data: 2022-09-12T20:14:15.533 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.533 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.534 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.534 DEBUG:boto:Params: {} 2022-09-12T20:14:15.535 DEBUG:boto:Token: None 2022-09-12T20:14:15.535 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/ 2022-09-12T20:14:15.536 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0= 2022-09-12T20:14:15.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0=', 'Content-Length': '0'} 2022-09-12T20:14:15.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e2f4bcecd772709-00631f9317-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.540 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:15.541 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:15.542 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:15.542 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:14:15.543 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:14:15.543 DEBUG:boto:Method: GET 2022-09-12T20:14:15.544 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:14:15.544 DEBUG:boto:Data: 2022-09-12T20:14:15.545 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.545 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.546 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.547 DEBUG:boto:Params: {} 2022-09-12T20:14:15.547 DEBUG:boto:Token: None 2022-09-12T20:14:15.548 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/ 2022-09-12T20:14:15.548 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0= 2022-09-12T20:14:15.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0=', 'Content-Length': '0'} 2022-09-12T20:14:15.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062c4011459373954-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.554 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:15.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-12T20:14:15.555 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:15.556 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:15.556 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:15.557 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:15.558 DEBUG:boto:Method: GET 2022-09-12T20:14:15.558 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:15.559 DEBUG:boto:Data: 2022-09-12T20:14:15.559 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.560 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.561 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.561 DEBUG:boto:Params: {} 2022-09-12T20:14:15.562 DEBUG:boto:Token: None 2022-09-12T20:14:15.563 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:15.563 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7vzvkr3BSQln+xON67jpqXbBedQ= 2022-09-12T20:14:15.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7vzvkr3BSQln+xON67jpqXbBedQ=', 'Content-Length': '0'} 2022-09-12T20:14:15.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054ca0178612673e6-00631f9317-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.567 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:15.568 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:15.568 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:15.569 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:15.569 DEBUG:boto:Method: GET 2022-09-12T20:14:15.570 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:15.570 DEBUG:boto:Data: 2022-09-12T20:14:15.571 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.572 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.572 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.573 DEBUG:boto:Params: {} 2022-09-12T20:14:15.573 DEBUG:boto:Token: None 2022-09-12T20:14:15.574 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:15.575 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7vzvkr3BSQln+xON67jpqXbBedQ= 2022-09-12T20:14:15.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7vzvkr3BSQln+xON67jpqXbBedQ=', 'Content-Length': '0'} 2022-09-12T20:14:15.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c3a16c8e273cfb4-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.579 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:15.580 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:15.580 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:15.581 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:15.581 DEBUG:boto:Method: HEAD 2022-09-12T20:14:15.582 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:15.582 DEBUG:boto:Data: 2022-09-12T20:14:15.583 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.583 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.584 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.584 DEBUG:boto:Params: {} 2022-09-12T20:14:15.585 DEBUG:boto:Token: None 2022-09-12T20:14:15.585 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:15.586 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DuDhAhWH2eNs/IYfVSWXo13AwNI= 2022-09-12T20:14:15.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DuDhAhWH2eNs/IYfVSWXo13AwNI=', 'Content-Length': '0'} 2022-09-12T20:14:15.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015823f405e04a1db-00631f9317-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.589 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:15.590 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:15.590 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:15.591 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:15.591 DEBUG:boto:Method: HEAD 2022-09-12T20:14:15.592 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:15.592 DEBUG:boto:Data: 2022-09-12T20:14:15.593 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.593 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.594 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.594 DEBUG:boto:Params: {} 2022-09-12T20:14:15.595 DEBUG:boto:Token: None 2022-09-12T20:14:15.596 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:15.596 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DuDhAhWH2eNs/IYfVSWXo13AwNI= 2022-09-12T20:14:15.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DuDhAhWH2eNs/IYfVSWXo13AwNI=', 'Content-Length': '0'} 2022-09-12T20:14:15.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bc6e8f8f0b7732d-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-12T20:14:15.601 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:15.601 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:15.602 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:15.602 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:15.603 DEBUG:boto:Method: GET 2022-09-12T20:14:15.604 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:15.604 DEBUG:boto:Data: 2022-09-12T20:14:15.605 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.606 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.607 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.607 DEBUG:boto:Params: {} 2022-09-12T20:14:15.608 DEBUG:boto:Token: None 2022-09-12T20:14:15.609 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:15.610 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7vzvkr3BSQln+xON67jpqXbBedQ= 2022-09-12T20:14:15.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7vzvkr3BSQln+xON67jpqXbBedQ=', 'Content-Length': '0'} 2022-09-12T20:14:15.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca8910b52f2ed38e-00631f9317-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.654 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:15.655 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:15.655 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:15.656 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:15.656 DEBUG:boto:Method: GET 2022-09-12T20:14:15.657 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:15.657 DEBUG:boto:Data: 2022-09-12T20:14:15.658 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.659 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.659 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.660 DEBUG:boto:Params: {} 2022-09-12T20:14:15.660 DEBUG:boto:Token: None 2022-09-12T20:14:15.661 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:15.662 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7vzvkr3BSQln+xON67jpqXbBedQ= 2022-09-12T20:14:15.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7vzvkr3BSQln+xON67jpqXbBedQ=', 'Content-Length': '0'} 2022-09-12T20:14:15.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbf174572257d9e3-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.666 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:15.666 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:15.667 DEBUG:boto:path=/jdpiak-75/?prefix=obj31 2022-09-12T20:14:15.667 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj31 2022-09-12T20:14:15.668 DEBUG:boto:Method: GET 2022-09-12T20:14:15.668 DEBUG:boto:Path: /jdpiak-75/?prefix=obj31 2022-09-12T20:14:15.669 DEBUG:boto:Data: 2022-09-12T20:14:15.669 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.670 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.670 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.671 DEBUG:boto:Params: {} 2022-09-12T20:14:15.672 DEBUG:boto:Token: None 2022-09-12T20:14:15.672 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/ 2022-09-12T20:14:15.673 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0= 2022-09-12T20:14:15.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0=', 'Content-Length': '0'} 2022-09-12T20:14:15.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d41dfea129fb679a-00631f9317-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.677 DEBUG:boto:b'jdpiak-75obj311000falseobj312022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:15.678 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:15.679 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:15.679 DEBUG:boto:path=/jdpiak-75/?prefix=obj31 2022-09-12T20:14:15.680 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj31 2022-09-12T20:14:15.681 DEBUG:boto:Method: GET 2022-09-12T20:14:15.681 DEBUG:boto:Path: /jdpiak-75/?prefix=obj31 2022-09-12T20:14:15.682 DEBUG:boto:Data: 2022-09-12T20:14:15.682 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.683 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.683 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.684 DEBUG:boto:Params: {} 2022-09-12T20:14:15.684 DEBUG:boto:Token: None 2022-09-12T20:14:15.685 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/ 2022-09-12T20:14:15.686 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0= 2022-09-12T20:14:15.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0=', 'Content-Length': '0'} 2022-09-12T20:14:15.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006804bfcc7f86c98d-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.692 DEBUG:boto:b'jdpiak-75obj311000falseobj312022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:15.693 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:15.693 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:15.694 DEBUG:boto:Method: HEAD 2022-09-12T20:14:15.694 DEBUG:boto:Path: /jdpiak-75/obj31 2022-09-12T20:14:15.695 DEBUG:boto:Data: 2022-09-12T20:14:15.695 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.696 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.696 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.697 DEBUG:boto:Params: {} 2022-09-12T20:14:15.698 DEBUG:boto:Token: None 2022-09-12T20:14:15.698 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj31 2022-09-12T20:14:15.699 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tYmdXvvx+1p5jV/pKXXeDHsFUAI= 2022-09-12T20:14:15.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tYmdXvvx+1p5jV/pKXXeDHsFUAI=', 'Content-Length': '0'} 2022-09-12T20:14:15.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2ee57984fbb725f-00631f9317-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.702 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:15.703 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:15.704 DEBUG:boto:Method: HEAD 2022-09-12T20:14:15.704 DEBUG:boto:Path: /jdpiak-75/obj31 2022-09-12T20:14:15.705 DEBUG:boto:Data: 2022-09-12T20:14:15.705 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.706 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.706 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.707 DEBUG:boto:Params: {} 2022-09-12T20:14:15.708 DEBUG:boto:Token: None 2022-09-12T20:14:15.708 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj31 2022-09-12T20:14:15.709 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tYmdXvvx+1p5jV/pKXXeDHsFUAI= 2022-09-12T20:14:15.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tYmdXvvx+1p5jV/pKXXeDHsFUAI=', 'Content-Length': '0'} 2022-09-12T20:14:15.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001aa7f369a61bf7a2-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-12T20:14:15.713 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:15.713 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:15.714 DEBUG:boto:Method: GET 2022-09-12T20:14:15.714 DEBUG:boto:Path: /jdpiak-75/obj31 2022-09-12T20:14:15.715 DEBUG:boto:Data: 2022-09-12T20:14:15.715 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.716 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.716 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.717 DEBUG:boto:Params: {} 2022-09-12T20:14:15.717 DEBUG:boto:Token: None 2022-09-12T20:14:15.718 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj31 2022-09-12T20:14:15.719 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B5EpWLQ+NZAo3DhVHfYOullijBA= 2022-09-12T20:14:15.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B5EpWLQ+NZAo3DhVHfYOullijBA=', 'Content-Length': '0'} 2022-09-12T20:14:15.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc5128e21764d964-00631f9317-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.722 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:15.723 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:15.723 DEBUG:boto:Method: GET 2022-09-12T20:14:15.724 DEBUG:boto:Path: /jdpiak-75/obj31 2022-09-12T20:14:15.724 DEBUG:boto:Data: 2022-09-12T20:14:15.725 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.725 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.726 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.726 DEBUG:boto:Params: {} 2022-09-12T20:14:15.727 DEBUG:boto:Token: None 2022-09-12T20:14:15.728 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj31 2022-09-12T20:14:15.728 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B5EpWLQ+NZAo3DhVHfYOullijBA= 2022-09-12T20:14:15.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B5EpWLQ+NZAo3DhVHfYOullijBA=', 'Content-Length': '0'} 2022-09-12T20:14:15.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b42092dabc98ee8-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.732 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:15.732 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:15.733 DEBUG:boto:path=/jdpiak-75/?prefix=obj31 2022-09-12T20:14:15.733 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj31 2022-09-12T20:14:15.734 DEBUG:boto:Method: GET 2022-09-12T20:14:15.735 DEBUG:boto:Path: /jdpiak-75/?prefix=obj31 2022-09-12T20:14:15.735 DEBUG:boto:Data: 2022-09-12T20:14:15.736 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.736 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.737 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.737 DEBUG:boto:Params: {} 2022-09-12T20:14:15.738 DEBUG:boto:Token: None 2022-09-12T20:14:15.745 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/ 2022-09-12T20:14:15.745 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0= 2022-09-12T20:14:15.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0=', 'Content-Length': '0'} 2022-09-12T20:14:15.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de9d8dbb79155f88-00631f9317-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.749 DEBUG:boto:b'jdpiak-75obj311000falseobj312022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:15.750 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:15.751 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:15.751 DEBUG:boto:path=/jdpiak-75/?prefix=obj31 2022-09-12T20:14:15.752 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj31 2022-09-12T20:14:15.752 DEBUG:boto:Method: GET 2022-09-12T20:14:15.753 DEBUG:boto:Path: /jdpiak-75/?prefix=obj31 2022-09-12T20:14:15.753 DEBUG:boto:Data: 2022-09-12T20:14:15.754 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.754 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.755 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.755 DEBUG:boto:Params: {} 2022-09-12T20:14:15.756 DEBUG:boto:Token: None 2022-09-12T20:14:15.756 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/ 2022-09-12T20:14:15.757 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0= 2022-09-12T20:14:15.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0=', 'Content-Length': '0'} 2022-09-12T20:14:15.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006efa985850670d29-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.763 DEBUG:boto:b'jdpiak-75obj311000falseobj312022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:15.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:14:15.764 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:15.765 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:15.765 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:15.766 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:15.766 DEBUG:boto:Method: GET 2022-09-12T20:14:15.767 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:15.767 DEBUG:boto:Data: 2022-09-12T20:14:15.768 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.768 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.769 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.769 DEBUG:boto:Params: {} 2022-09-12T20:14:15.770 DEBUG:boto:Token: None 2022-09-12T20:14:15.771 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:15.771 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wbfzieqCgGAbmg8tj4JAbjnDAb4= 2022-09-12T20:14:15.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wbfzieqCgGAbmg8tj4JAbjnDAb4=', 'Content-Length': '0'} 2022-09-12T20:14:15.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f96fcbb44fa58ef-00631f9317-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.775 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:15.775 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:15.776 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:15.776 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:15.777 DEBUG:boto:Method: GET 2022-09-12T20:14:15.777 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:15.778 DEBUG:boto:Data: 2022-09-12T20:14:15.778 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.779 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.779 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.780 DEBUG:boto:Params: {} 2022-09-12T20:14:15.781 DEBUG:boto:Token: None 2022-09-12T20:14:15.781 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:15.782 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wbfzieqCgGAbmg8tj4JAbjnDAb4= 2022-09-12T20:14:15.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wbfzieqCgGAbmg8tj4JAbjnDAb4=', 'Content-Length': '0'} 2022-09-12T20:14:15.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5e92c38670b8035-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.826 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:15.827 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:15.827 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:15.828 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:15.828 DEBUG:boto:Method: HEAD 2022-09-12T20:14:15.829 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:15.830 DEBUG:boto:Data: 2022-09-12T20:14:15.830 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.831 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.831 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.832 DEBUG:boto:Params: {} 2022-09-12T20:14:15.833 DEBUG:boto:Token: None 2022-09-12T20:14:15.833 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:15.834 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8Mg/EaxBRzaGRoulUtpHhFT3Y/s= 2022-09-12T20:14:15.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8Mg/EaxBRzaGRoulUtpHhFT3Y/s=', 'Content-Length': '0'} 2022-09-12T20:14:15.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df205ca08ec49ee6-00631f9317-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.837 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:15.838 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:15.838 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:15.839 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:15.839 DEBUG:boto:Method: HEAD 2022-09-12T20:14:15.840 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:15.841 DEBUG:boto:Data: 2022-09-12T20:14:15.842 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.843 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.844 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.845 DEBUG:boto:Params: {} 2022-09-12T20:14:15.846 DEBUG:boto:Token: None 2022-09-12T20:14:15.847 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:15.848 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8Mg/EaxBRzaGRoulUtpHhFT3Y/s= 2022-09-12T20:14:15.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8Mg/EaxBRzaGRoulUtpHhFT3Y/s=', 'Content-Length': '0'} 2022-09-12T20:14:15.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072913e0670baff23-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:14:15.852 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:15.852 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:15.853 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:15.853 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:15.854 DEBUG:boto:Method: GET 2022-09-12T20:14:15.855 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:15.855 DEBUG:boto:Data: 2022-09-12T20:14:15.856 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.856 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.857 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.857 DEBUG:boto:Params: {} 2022-09-12T20:14:15.858 DEBUG:boto:Token: None 2022-09-12T20:14:15.858 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:15.859 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wbfzieqCgGAbmg8tj4JAbjnDAb4= 2022-09-12T20:14:15.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wbfzieqCgGAbmg8tj4JAbjnDAb4=', 'Content-Length': '0'} 2022-09-12T20:14:15.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001efb8a152c5f4c25-00631f9317-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.863 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:15.863 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:15.864 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:15.864 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:15.865 DEBUG:boto:Method: GET 2022-09-12T20:14:15.865 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:15.866 DEBUG:boto:Data: 2022-09-12T20:14:15.866 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.867 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.867 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.868 DEBUG:boto:Params: {} 2022-09-12T20:14:15.868 DEBUG:boto:Token: None 2022-09-12T20:14:15.869 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:15.870 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wbfzieqCgGAbmg8tj4JAbjnDAb4= 2022-09-12T20:14:15.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wbfzieqCgGAbmg8tj4JAbjnDAb4=', 'Content-Length': '0'} 2022-09-12T20:14:15.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a4f6c2c1f522221-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.873 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:15.874 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:15.874 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:14:15.875 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:14:15.875 DEBUG:boto:Method: GET 2022-09-12T20:14:15.876 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:14:15.876 DEBUG:boto:Data: 2022-09-12T20:14:15.877 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.877 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.878 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.878 DEBUG:boto:Params: {} 2022-09-12T20:14:15.879 DEBUG:boto:Token: None 2022-09-12T20:14:15.880 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/ 2022-09-12T20:14:15.880 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0= 2022-09-12T20:14:15.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0=', 'Content-Length': '0'} 2022-09-12T20:14:15.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e6ab7eee6f8d8ef-00631f9317-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.885 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:15.885 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:15.886 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:15.887 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:14:15.887 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:14:15.888 DEBUG:boto:Method: GET 2022-09-12T20:14:15.888 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:14:15.889 DEBUG:boto:Data: 2022-09-12T20:14:15.889 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.890 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.890 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.891 DEBUG:boto:Params: {} 2022-09-12T20:14:15.891 DEBUG:boto:Token: None 2022-09-12T20:14:15.892 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/ 2022-09-12T20:14:15.892 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0= 2022-09-12T20:14:15.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0=', 'Content-Length': '0'} 2022-09-12T20:14:15.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050f2ac361446c33c-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.898 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:15.899 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:15.900 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:15.900 DEBUG:boto:Method: HEAD 2022-09-12T20:14:15.901 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:14:15.902 DEBUG:boto:Data: 2022-09-12T20:14:15.902 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.903 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.904 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.904 DEBUG:boto:Params: {} 2022-09-12T20:14:15.905 DEBUG:boto:Token: None 2022-09-12T20:14:15.906 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj33 2022-09-12T20:14:15.907 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1mc/XeS+U/AKxaec0vfYVJJL7Fc= 2022-09-12T20:14:15.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1mc/XeS+U/AKxaec0vfYVJJL7Fc=', 'Content-Length': '0'} 2022-09-12T20:14:15.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000026e2834126a7f2c-00631f9317-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.911 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:15.912 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:15.913 DEBUG:boto:Method: HEAD 2022-09-12T20:14:15.913 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:14:15.914 DEBUG:boto:Data: 2022-09-12T20:14:15.915 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.916 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.917 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.918 DEBUG:boto:Params: {} 2022-09-12T20:14:15.919 DEBUG:boto:Token: None 2022-09-12T20:14:15.920 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj33 2022-09-12T20:14:15.921 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1mc/XeS+U/AKxaec0vfYVJJL7Fc= 2022-09-12T20:14:15.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1mc/XeS+U/AKxaec0vfYVJJL7Fc=', 'Content-Length': '0'} 2022-09-12T20:14:15.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bff8ab7c3da51997-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:14:15.925 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:15.926 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:15.926 DEBUG:boto:Method: GET 2022-09-12T20:14:15.927 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:14:15.928 DEBUG:boto:Data: 2022-09-12T20:14:15.928 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.929 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.929 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.930 DEBUG:boto:Params: {} 2022-09-12T20:14:15.931 DEBUG:boto:Token: None 2022-09-12T20:14:15.931 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj33 2022-09-12T20:14:15.932 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LkyzfAhYT/V2auHAN6Hs0GPcVac= 2022-09-12T20:14:15.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LkyzfAhYT/V2auHAN6Hs0GPcVac=', 'Content-Length': '0'} 2022-09-12T20:14:15.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085e98207b30bc191-00631f9317-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.978 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:15.979 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:15.979 DEBUG:boto:Method: GET 2022-09-12T20:14:15.980 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:14:15.980 DEBUG:boto:Data: 2022-09-12T20:14:15.981 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.981 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:15.982 DEBUG:boto:Port: 8000 2022-09-12T20:14:15.983 DEBUG:boto:Params: {} 2022-09-12T20:14:15.983 DEBUG:boto:Token: None 2022-09-12T20:14:15.984 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/obj33 2022-09-12T20:14:15.984 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LkyzfAhYT/V2auHAN6Hs0GPcVac= 2022-09-12T20:14:15.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LkyzfAhYT/V2auHAN6Hs0GPcVac=', 'Content-Length': '0'} 2022-09-12T20:14:15.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f64522c110c89d88-00631f9317-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:15.988 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:15.989 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:15.989 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:14:15.990 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:14:15.990 DEBUG:boto:Method: GET 2022-09-12T20:14:15.991 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:14:15.991 DEBUG:boto:Data: 2022-09-12T20:14:15.992 DEBUG:boto:Headers: {} 2022-09-12T20:14:15.992 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:15.993 DEBUG:boto:Port: 8001 2022-09-12T20:14:15.993 DEBUG:boto:Params: {} 2022-09-12T20:14:15.994 DEBUG:boto:Token: None 2022-09-12T20:14:15.995 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:15 GMT /jdpiak-75/ 2022-09-12T20:14:15.995 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0= 2022-09-12T20:14:15.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fwbZHZKDdenD900vM+RhyEkuTi0=', 'Content-Length': '0'} 2022-09-12T20:14:15.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbe818b1d5bdab82-00631f9317-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.000 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:16.000 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:16.001 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:16.001 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:14:16.002 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:14:16.003 DEBUG:boto:Method: GET 2022-09-12T20:14:16.003 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:14:16.004 DEBUG:boto:Data: 2022-09-12T20:14:16.004 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.005 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.005 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.006 DEBUG:boto:Params: {} 2022-09-12T20:14:16.006 DEBUG:boto:Token: None 2022-09-12T20:14:16.007 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/ 2022-09-12T20:14:16.008 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY= 2022-09-12T20:14:16.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY=', 'Content-Length': '0'} 2022-09-12T20:14:16.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003af756259c9518c4-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.013 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:16.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:14:16.014 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:16.015 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:16.016 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:16.016 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:16.017 DEBUG:boto:Method: GET 2022-09-12T20:14:16.017 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:16.018 DEBUG:boto:Data: 2022-09-12T20:14:16.018 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.019 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.019 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.020 DEBUG:boto:Params: {} 2022-09-12T20:14:16.020 DEBUG:boto:Token: None 2022-09-12T20:14:16.021 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:16.021 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8ME5IHpKbhVjKpophPsG8p8IQIc= 2022-09-12T20:14:16.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8ME5IHpKbhVjKpophPsG8p8IQIc=', 'Content-Length': '0'} 2022-09-12T20:14:16.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095f9c872abc92f9f-00631f9318-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.025 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:16.025 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:16.026 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:16.026 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:16.027 DEBUG:boto:Method: GET 2022-09-12T20:14:16.027 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:16.028 DEBUG:boto:Data: 2022-09-12T20:14:16.029 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.029 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.030 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.030 DEBUG:boto:Params: {} 2022-09-12T20:14:16.031 DEBUG:boto:Token: None 2022-09-12T20:14:16.031 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:16.032 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8ME5IHpKbhVjKpophPsG8p8IQIc= 2022-09-12T20:14:16.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8ME5IHpKbhVjKpophPsG8p8IQIc=', 'Content-Length': '0'} 2022-09-12T20:14:16.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efef809a6a0e0468-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.078 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:16.079 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:16.079 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:16.080 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:16.080 DEBUG:boto:Method: HEAD 2022-09-12T20:14:16.081 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:16.081 DEBUG:boto:Data: 2022-09-12T20:14:16.082 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.082 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.083 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.084 DEBUG:boto:Params: {} 2022-09-12T20:14:16.084 DEBUG:boto:Token: None 2022-09-12T20:14:16.085 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:16.085 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2UAx1rROnq6ChFgF2aVLRxlPwUw= 2022-09-12T20:14:16.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2UAx1rROnq6ChFgF2aVLRxlPwUw=', 'Content-Length': '0'} 2022-09-12T20:14:16.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0a27eb7b5d349e9-00631f9318-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.089 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:16.089 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:16.090 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:16.090 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:16.091 DEBUG:boto:Method: HEAD 2022-09-12T20:14:16.092 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:16.092 DEBUG:boto:Data: 2022-09-12T20:14:16.093 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.093 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.094 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.094 DEBUG:boto:Params: {} 2022-09-12T20:14:16.095 DEBUG:boto:Token: None 2022-09-12T20:14:16.096 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:16.096 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2UAx1rROnq6ChFgF2aVLRxlPwUw= 2022-09-12T20:14:16.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2UAx1rROnq6ChFgF2aVLRxlPwUw=', 'Content-Length': '0'} 2022-09-12T20:14:16.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5ff9d1ac94ab09e-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:14:16.100 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:16.101 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:16.101 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:16.102 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:16.102 DEBUG:boto:Method: GET 2022-09-12T20:14:16.103 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:16.104 DEBUG:boto:Data: 2022-09-12T20:14:16.104 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.105 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.105 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.106 DEBUG:boto:Params: {} 2022-09-12T20:14:16.107 DEBUG:boto:Token: None 2022-09-12T20:14:16.108 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:16.109 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8ME5IHpKbhVjKpophPsG8p8IQIc= 2022-09-12T20:14:16.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8ME5IHpKbhVjKpophPsG8p8IQIc=', 'Content-Length': '0'} 2022-09-12T20:14:16.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbec2288a7f0e48d-00631f9318-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.113 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:16.114 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:16.114 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:16.115 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:16.115 DEBUG:boto:Method: GET 2022-09-12T20:14:16.116 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:16.117 DEBUG:boto:Data: 2022-09-12T20:14:16.117 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.118 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.118 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.119 DEBUG:boto:Params: {} 2022-09-12T20:14:16.120 DEBUG:boto:Token: None 2022-09-12T20:14:16.120 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:16.121 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8ME5IHpKbhVjKpophPsG8p8IQIc= 2022-09-12T20:14:16.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8ME5IHpKbhVjKpophPsG8p8IQIc=', 'Content-Length': '0'} 2022-09-12T20:14:16.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef3117b8c3584df6-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.125 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:16.125 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:16.126 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:14:16.126 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:14:16.127 DEBUG:boto:Method: GET 2022-09-12T20:14:16.127 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:14:16.128 DEBUG:boto:Data: 2022-09-12T20:14:16.128 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.129 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.130 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.131 DEBUG:boto:Params: {} 2022-09-12T20:14:16.131 DEBUG:boto:Token: None 2022-09-12T20:14:16.132 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/ 2022-09-12T20:14:16.133 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY= 2022-09-12T20:14:16.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY=', 'Content-Length': '0'} 2022-09-12T20:14:16.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080942ffa7f655d57-00631f9318-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.137 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:16.138 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:16.138 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:16.139 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:14:16.139 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:14:16.140 DEBUG:boto:Method: GET 2022-09-12T20:14:16.140 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:14:16.141 DEBUG:boto:Data: 2022-09-12T20:14:16.141 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.142 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.142 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.143 DEBUG:boto:Params: {} 2022-09-12T20:14:16.144 DEBUG:boto:Token: None 2022-09-12T20:14:16.144 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/ 2022-09-12T20:14:16.145 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY= 2022-09-12T20:14:16.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY=', 'Content-Length': '0'} 2022-09-12T20:14:16.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf3ab942d3c9a7b5-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.150 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:16.151 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:16.152 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:16.152 DEBUG:boto:Method: HEAD 2022-09-12T20:14:16.153 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:14:16.153 DEBUG:boto:Data: 2022-09-12T20:14:16.154 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.154 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.155 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.155 DEBUG:boto:Params: {} 2022-09-12T20:14:16.156 DEBUG:boto:Token: None 2022-09-12T20:14:16.156 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj34 2022-09-12T20:14:16.157 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FpXJwsfjUW4wXpAXArl/7W2FpEU= 2022-09-12T20:14:16.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FpXJwsfjUW4wXpAXArl/7W2FpEU=', 'Content-Length': '0'} 2022-09-12T20:14:16.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009eb1faf8f2e2654c-00631f9318-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.160 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:16.161 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:16.161 DEBUG:boto:Method: HEAD 2022-09-12T20:14:16.162 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:14:16.162 DEBUG:boto:Data: 2022-09-12T20:14:16.162 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.163 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.163 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.164 DEBUG:boto:Params: {} 2022-09-12T20:14:16.164 DEBUG:boto:Token: None 2022-09-12T20:14:16.165 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj34 2022-09-12T20:14:16.166 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FpXJwsfjUW4wXpAXArl/7W2FpEU= 2022-09-12T20:14:16.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FpXJwsfjUW4wXpAXArl/7W2FpEU=', 'Content-Length': '0'} 2022-09-12T20:14:16.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a9ff1920ad9e729-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:14:16.169 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:16.170 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:16.170 DEBUG:boto:Method: GET 2022-09-12T20:14:16.171 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:14:16.171 DEBUG:boto:Data: 2022-09-12T20:14:16.172 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.172 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.173 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.174 DEBUG:boto:Params: {} 2022-09-12T20:14:16.174 DEBUG:boto:Token: None 2022-09-12T20:14:16.175 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj34 2022-09-12T20:14:16.175 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZGIeiHLwK+3uWb87gm+LQSD/SiQ= 2022-09-12T20:14:16.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZGIeiHLwK+3uWb87gm+LQSD/SiQ=', 'Content-Length': '0'} 2022-09-12T20:14:16.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c844ae8f0931003f-00631f9318-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.179 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:16.179 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:16.180 DEBUG:boto:Method: GET 2022-09-12T20:14:16.180 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:14:16.181 DEBUG:boto:Data: 2022-09-12T20:14:16.181 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.182 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.182 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.183 DEBUG:boto:Params: {} 2022-09-12T20:14:16.183 DEBUG:boto:Token: None 2022-09-12T20:14:16.184 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj34 2022-09-12T20:14:16.185 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZGIeiHLwK+3uWb87gm+LQSD/SiQ= 2022-09-12T20:14:16.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZGIeiHLwK+3uWb87gm+LQSD/SiQ=', 'Content-Length': '0'} 2022-09-12T20:14:16.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a6842fc1c6c063b-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.230 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:16.231 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:16.231 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:14:16.232 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:14:16.232 DEBUG:boto:Method: GET 2022-09-12T20:14:16.233 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:14:16.233 DEBUG:boto:Data: 2022-09-12T20:14:16.234 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.234 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.235 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.235 DEBUG:boto:Params: {} 2022-09-12T20:14:16.236 DEBUG:boto:Token: None 2022-09-12T20:14:16.237 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/ 2022-09-12T20:14:16.237 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY= 2022-09-12T20:14:16.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY=', 'Content-Length': '0'} 2022-09-12T20:14:16.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004aa070302bca5104-00631f9318-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.241 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:16.242 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:16.243 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:16.244 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:14:16.244 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:14:16.245 DEBUG:boto:Method: GET 2022-09-12T20:14:16.245 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:14:16.246 DEBUG:boto:Data: 2022-09-12T20:14:16.246 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.247 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.247 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.248 DEBUG:boto:Params: {} 2022-09-12T20:14:16.249 DEBUG:boto:Token: None 2022-09-12T20:14:16.249 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/ 2022-09-12T20:14:16.250 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY= 2022-09-12T20:14:16.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY=', 'Content-Length': '0'} 2022-09-12T20:14:16.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f35cf793eb94188b-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.256 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:16.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:14:16.257 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:16.258 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:16.258 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:16.259 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:16.259 DEBUG:boto:Method: GET 2022-09-12T20:14:16.260 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:16.260 DEBUG:boto:Data: 2022-09-12T20:14:16.261 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.262 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.262 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.263 DEBUG:boto:Params: {} 2022-09-12T20:14:16.263 DEBUG:boto:Token: None 2022-09-12T20:14:16.264 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:16.265 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MH/yxd3voBZL9wsr7o6l5xBav08= 2022-09-12T20:14:16.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MH/yxd3voBZL9wsr7o6l5xBav08=', 'Content-Length': '0'} 2022-09-12T20:14:16.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b14cf649a251009-00631f9318-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.269 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:16.269 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:16.270 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:16.271 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:16.271 DEBUG:boto:Method: GET 2022-09-12T20:14:16.272 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:16.272 DEBUG:boto:Data: 2022-09-12T20:14:16.273 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.273 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.274 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.275 DEBUG:boto:Params: {} 2022-09-12T20:14:16.275 DEBUG:boto:Token: None 2022-09-12T20:14:16.276 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:16.277 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MH/yxd3voBZL9wsr7o6l5xBav08= 2022-09-12T20:14:16.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MH/yxd3voBZL9wsr7o6l5xBav08=', 'Content-Length': '0'} 2022-09-12T20:14:16.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0e171927e4b8e78-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.280 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:16.281 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:16.282 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:16.282 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:16.283 DEBUG:boto:Method: HEAD 2022-09-12T20:14:16.283 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:16.284 DEBUG:boto:Data: 2022-09-12T20:14:16.284 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.285 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.285 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.286 DEBUG:boto:Params: {} 2022-09-12T20:14:16.287 DEBUG:boto:Token: None 2022-09-12T20:14:16.287 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:16.288 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ic2Lv2/lCvtcqoB/t8l4WQwpTyo= 2022-09-12T20:14:16.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ic2Lv2/lCvtcqoB/t8l4WQwpTyo=', 'Content-Length': '0'} 2022-09-12T20:14:16.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa6510c278cd1bb1-00631f9318-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.291 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:16.292 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:16.292 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:16.293 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:16.293 DEBUG:boto:Method: HEAD 2022-09-12T20:14:16.294 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:16.294 DEBUG:boto:Data: 2022-09-12T20:14:16.295 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.295 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.296 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.296 DEBUG:boto:Params: {} 2022-09-12T20:14:16.297 DEBUG:boto:Token: None 2022-09-12T20:14:16.298 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:16.298 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ic2Lv2/lCvtcqoB/t8l4WQwpTyo= 2022-09-12T20:14:16.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ic2Lv2/lCvtcqoB/t8l4WQwpTyo=', 'Content-Length': '0'} 2022-09-12T20:14:16.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7e89c4311373461-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:14:16.302 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:16.303 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:16.304 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:16.305 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:16.306 DEBUG:boto:Method: GET 2022-09-12T20:14:16.306 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:16.307 DEBUG:boto:Data: 2022-09-12T20:14:16.307 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.308 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.308 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.309 DEBUG:boto:Params: {} 2022-09-12T20:14:16.309 DEBUG:boto:Token: None 2022-09-12T20:14:16.310 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:16.310 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MH/yxd3voBZL9wsr7o6l5xBav08= 2022-09-12T20:14:16.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MH/yxd3voBZL9wsr7o6l5xBav08=', 'Content-Length': '0'} 2022-09-12T20:14:16.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c67a8232101d815-00631f9318-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.354 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:16.355 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:16.355 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:16.356 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:16.356 DEBUG:boto:Method: GET 2022-09-12T20:14:16.357 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:16.358 DEBUG:boto:Data: 2022-09-12T20:14:16.358 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.359 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.359 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.360 DEBUG:boto:Params: {} 2022-09-12T20:14:16.360 DEBUG:boto:Token: None 2022-09-12T20:14:16.361 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:16.362 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MH/yxd3voBZL9wsr7o6l5xBav08= 2022-09-12T20:14:16.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MH/yxd3voBZL9wsr7o6l5xBav08=', 'Content-Length': '0'} 2022-09-12T20:14:16.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e44877d01245ce6b-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.365 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:16.366 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:16.366 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:14:16.367 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:14:16.367 DEBUG:boto:Method: GET 2022-09-12T20:14:16.368 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:14:16.369 DEBUG:boto:Data: 2022-09-12T20:14:16.369 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.370 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.370 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.371 DEBUG:boto:Params: {} 2022-09-12T20:14:16.371 DEBUG:boto:Token: None 2022-09-12T20:14:16.372 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/ 2022-09-12T20:14:16.373 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY= 2022-09-12T20:14:16.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY=', 'Content-Length': '0'} 2022-09-12T20:14:16.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bef686c95a454d0-00631f9318-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.377 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:16.378 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:16.378 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:16.379 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:14:16.379 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:14:16.380 DEBUG:boto:Method: GET 2022-09-12T20:14:16.380 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:14:16.381 DEBUG:boto:Data: 2022-09-12T20:14:16.381 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.382 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.383 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.383 DEBUG:boto:Params: {} 2022-09-12T20:14:16.384 DEBUG:boto:Token: None 2022-09-12T20:14:16.384 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/ 2022-09-12T20:14:16.386 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY= 2022-09-12T20:14:16.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY=', 'Content-Length': '0'} 2022-09-12T20:14:16.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dc80883f82f68f2-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.392 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:16.393 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:16.393 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:16.394 DEBUG:boto:Method: HEAD 2022-09-12T20:14:16.394 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:14:16.395 DEBUG:boto:Data: 2022-09-12T20:14:16.395 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.396 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.396 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.397 DEBUG:boto:Params: {} 2022-09-12T20:14:16.397 DEBUG:boto:Token: None 2022-09-12T20:14:16.398 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj38 2022-09-12T20:14:16.398 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:elRw1x/0wNrrW1/q2kFLbTxnObc= 2022-09-12T20:14:16.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:elRw1x/0wNrrW1/q2kFLbTxnObc=', 'Content-Length': '0'} 2022-09-12T20:14:16.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e5dc25ea190db33-00631f9318-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.402 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:16.402 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:16.403 DEBUG:boto:Method: HEAD 2022-09-12T20:14:16.403 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:14:16.404 DEBUG:boto:Data: 2022-09-12T20:14:16.405 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.405 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.406 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.406 DEBUG:boto:Params: {} 2022-09-12T20:14:16.407 DEBUG:boto:Token: None 2022-09-12T20:14:16.407 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj38 2022-09-12T20:14:16.408 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:elRw1x/0wNrrW1/q2kFLbTxnObc= 2022-09-12T20:14:16.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:elRw1x/0wNrrW1/q2kFLbTxnObc=', 'Content-Length': '0'} 2022-09-12T20:14:16.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c4468f78d7f5976-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:14:16.412 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:16.412 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:16.413 DEBUG:boto:Method: GET 2022-09-12T20:14:16.413 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:14:16.414 DEBUG:boto:Data: 2022-09-12T20:14:16.415 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.415 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.416 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.416 DEBUG:boto:Params: {} 2022-09-12T20:14:16.417 DEBUG:boto:Token: None 2022-09-12T20:14:16.417 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj38 2022-09-12T20:14:16.418 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oLnoGdiVp+3YPf9yzL3yYqz4MB0= 2022-09-12T20:14:16.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oLnoGdiVp+3YPf9yzL3yYqz4MB0=', 'Content-Length': '0'} 2022-09-12T20:14:16.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d03beaed5a8cd73c-00631f9318-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.421 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:16.422 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:16.422 DEBUG:boto:Method: GET 2022-09-12T20:14:16.423 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:14:16.423 DEBUG:boto:Data: 2022-09-12T20:14:16.424 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.425 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.425 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.426 DEBUG:boto:Params: {} 2022-09-12T20:14:16.426 DEBUG:boto:Token: None 2022-09-12T20:14:16.427 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj38 2022-09-12T20:14:16.427 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oLnoGdiVp+3YPf9yzL3yYqz4MB0= 2022-09-12T20:14:16.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oLnoGdiVp+3YPf9yzL3yYqz4MB0=', 'Content-Length': '0'} 2022-09-12T20:14:16.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000385b670710251a20-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.474 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:16.475 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:16.475 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:14:16.476 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:14:16.476 DEBUG:boto:Method: GET 2022-09-12T20:14:16.477 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:14:16.478 DEBUG:boto:Data: 2022-09-12T20:14:16.478 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.479 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.479 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.480 DEBUG:boto:Params: {} 2022-09-12T20:14:16.480 DEBUG:boto:Token: None 2022-09-12T20:14:16.481 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/ 2022-09-12T20:14:16.481 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY= 2022-09-12T20:14:16.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY=', 'Content-Length': '0'} 2022-09-12T20:14:16.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8139aa39dd01df2-00631f9318-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.486 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:16.487 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:16.487 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:16.488 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:14:16.488 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:14:16.489 DEBUG:boto:Method: GET 2022-09-12T20:14:16.489 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:14:16.490 DEBUG:boto:Data: 2022-09-12T20:14:16.490 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.491 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.491 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.492 DEBUG:boto:Params: {} 2022-09-12T20:14:16.492 DEBUG:boto:Token: None 2022-09-12T20:14:16.493 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/ 2022-09-12T20:14:16.494 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY= 2022-09-12T20:14:16.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY=', 'Content-Length': '0'} 2022-09-12T20:14:16.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2e5ae877eb7327b-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.499 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:16.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:14:16.500 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:16.501 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:16.502 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:16.502 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:16.503 DEBUG:boto:Method: GET 2022-09-12T20:14:16.503 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:16.504 DEBUG:boto:Data: 2022-09-12T20:14:16.504 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.505 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.505 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.506 DEBUG:boto:Params: {} 2022-09-12T20:14:16.507 DEBUG:boto:Token: None 2022-09-12T20:14:16.507 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:16.508 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sJiCaQfcEpDI24+ZFIVwyZ7dCqs= 2022-09-12T20:14:16.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sJiCaQfcEpDI24+ZFIVwyZ7dCqs=', 'Content-Length': '0'} 2022-09-12T20:14:16.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ec265b8be3f219e-00631f9318-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.511 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:16.512 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:16.512 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:16.513 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:16.514 DEBUG:boto:Method: GET 2022-09-12T20:14:16.514 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:16.515 DEBUG:boto:Data: 2022-09-12T20:14:16.515 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.516 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.516 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.517 DEBUG:boto:Params: {} 2022-09-12T20:14:16.517 DEBUG:boto:Token: None 2022-09-12T20:14:16.518 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:16.518 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sJiCaQfcEpDI24+ZFIVwyZ7dCqs= 2022-09-12T20:14:16.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sJiCaQfcEpDI24+ZFIVwyZ7dCqs=', 'Content-Length': '0'} 2022-09-12T20:14:16.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e8109527913a2f0-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.522 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:16.522 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:16.523 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:16.523 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:16.524 DEBUG:boto:Method: HEAD 2022-09-12T20:14:16.524 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:16.525 DEBUG:boto:Data: 2022-09-12T20:14:16.525 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.526 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.526 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.527 DEBUG:boto:Params: {} 2022-09-12T20:14:16.527 DEBUG:boto:Token: None 2022-09-12T20:14:16.528 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:16.528 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ecTQU+wv18AJvzTpFISMBHstgAE= 2022-09-12T20:14:16.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ecTQU+wv18AJvzTpFISMBHstgAE=', 'Content-Length': '0'} 2022-09-12T20:14:16.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c047fe5eebfd3888-00631f9318-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.532 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:16.533 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:16.533 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:16.534 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:16.534 DEBUG:boto:Method: HEAD 2022-09-12T20:14:16.535 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:16.535 DEBUG:boto:Data: 2022-09-12T20:14:16.536 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.536 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.537 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.537 DEBUG:boto:Params: {} 2022-09-12T20:14:16.538 DEBUG:boto:Token: None 2022-09-12T20:14:16.539 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:16.539 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ecTQU+wv18AJvzTpFISMBHstgAE= 2022-09-12T20:14:16.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ecTQU+wv18AJvzTpFISMBHstgAE=', 'Content-Length': '0'} 2022-09-12T20:14:16.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045d6c435167d248d-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:14:16.543 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:16.543 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:16.544 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:16.545 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:16.545 DEBUG:boto:Method: GET 2022-09-12T20:14:16.546 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:16.546 DEBUG:boto:Data: 2022-09-12T20:14:16.547 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.548 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.548 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.549 DEBUG:boto:Params: {} 2022-09-12T20:14:16.549 DEBUG:boto:Token: None 2022-09-12T20:14:16.550 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:16.551 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sJiCaQfcEpDI24+ZFIVwyZ7dCqs= 2022-09-12T20:14:16.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sJiCaQfcEpDI24+ZFIVwyZ7dCqs=', 'Content-Length': '0'} 2022-09-12T20:14:16.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000df6ad54c400a7e5-00631f9318-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.554 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:16.555 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:16.556 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:16.556 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:16.557 DEBUG:boto:Method: GET 2022-09-12T20:14:16.557 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:16.558 DEBUG:boto:Data: 2022-09-12T20:14:16.558 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.559 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.559 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.560 DEBUG:boto:Params: {} 2022-09-12T20:14:16.561 DEBUG:boto:Token: None 2022-09-12T20:14:16.561 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:16.562 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sJiCaQfcEpDI24+ZFIVwyZ7dCqs= 2022-09-12T20:14:16.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sJiCaQfcEpDI24+ZFIVwyZ7dCqs=', 'Content-Length': '0'} 2022-09-12T20:14:16.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c9e0f9587ab56e6-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.565 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:16.566 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:16.566 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:14:16.567 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:14:16.567 DEBUG:boto:Method: GET 2022-09-12T20:14:16.568 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:14:16.568 DEBUG:boto:Data: 2022-09-12T20:14:16.569 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.570 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.570 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.571 DEBUG:boto:Params: {} 2022-09-12T20:14:16.571 DEBUG:boto:Token: None 2022-09-12T20:14:16.572 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/ 2022-09-12T20:14:16.573 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY= 2022-09-12T20:14:16.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY=', 'Content-Length': '0'} 2022-09-12T20:14:16.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000691ffc49c15c4b0-00631f9318-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.577 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:16.577 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:16.578 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:16.578 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:14:16.579 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:14:16.580 DEBUG:boto:Method: GET 2022-09-12T20:14:16.580 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:14:16.581 DEBUG:boto:Data: 2022-09-12T20:14:16.581 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.582 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.582 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.583 DEBUG:boto:Params: {} 2022-09-12T20:14:16.584 DEBUG:boto:Token: None 2022-09-12T20:14:16.584 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/ 2022-09-12T20:14:16.585 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY= 2022-09-12T20:14:16.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY=', 'Content-Length': '0'} 2022-09-12T20:14:16.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec57303beee669e7-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.591 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:16.592 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:16.592 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:16.593 DEBUG:boto:Method: HEAD 2022-09-12T20:14:16.593 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:14:16.594 DEBUG:boto:Data: 2022-09-12T20:14:16.594 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.595 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.595 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.596 DEBUG:boto:Params: {} 2022-09-12T20:14:16.597 DEBUG:boto:Token: None 2022-09-12T20:14:16.597 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj44 2022-09-12T20:14:16.598 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aQ6A7ko93D1gP+g2WehM5Dx+L8w= 2022-09-12T20:14:16.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aQ6A7ko93D1gP+g2WehM5Dx+L8w=', 'Content-Length': '0'} 2022-09-12T20:14:16.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000617b42d4c0d5bb28-00631f9318-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.602 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:16.602 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:16.603 DEBUG:boto:Method: HEAD 2022-09-12T20:14:16.604 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:14:16.604 DEBUG:boto:Data: 2022-09-12T20:14:16.605 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.605 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.606 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.606 DEBUG:boto:Params: {} 2022-09-12T20:14:16.607 DEBUG:boto:Token: None 2022-09-12T20:14:16.608 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj44 2022-09-12T20:14:16.608 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aQ6A7ko93D1gP+g2WehM5Dx+L8w= 2022-09-12T20:14:16.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aQ6A7ko93D1gP+g2WehM5Dx+L8w=', 'Content-Length': '0'} 2022-09-12T20:14:16.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e92756468688c52b-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:14:16.613 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:16.613 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:16.614 DEBUG:boto:Method: GET 2022-09-12T20:14:16.615 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:14:16.615 DEBUG:boto:Data: 2022-09-12T20:14:16.616 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.616 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.617 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.618 DEBUG:boto:Params: {} 2022-09-12T20:14:16.618 DEBUG:boto:Token: None 2022-09-12T20:14:16.619 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj44 2022-09-12T20:14:16.619 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:khOdfsYzDZYea2RrCCwdvO1O36w= 2022-09-12T20:14:16.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:khOdfsYzDZYea2RrCCwdvO1O36w=', 'Content-Length': '0'} 2022-09-12T20:14:16.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067a76728202efc20-00631f9318-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.624 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:16.624 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:16.625 DEBUG:boto:Method: GET 2022-09-12T20:14:16.625 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:14:16.626 DEBUG:boto:Data: 2022-09-12T20:14:16.627 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.627 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.628 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.628 DEBUG:boto:Params: {} 2022-09-12T20:14:16.629 DEBUG:boto:Token: None 2022-09-12T20:14:16.630 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj44 2022-09-12T20:14:16.631 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:khOdfsYzDZYea2RrCCwdvO1O36w= 2022-09-12T20:14:16.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:khOdfsYzDZYea2RrCCwdvO1O36w=', 'Content-Length': '0'} 2022-09-12T20:14:16.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a82096f2c6ac733d-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.674 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:16.675 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:16.676 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:14:16.677 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:14:16.677 DEBUG:boto:Method: GET 2022-09-12T20:14:16.678 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:14:16.679 DEBUG:boto:Data: 2022-09-12T20:14:16.680 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.681 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.681 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.682 DEBUG:boto:Params: {} 2022-09-12T20:14:16.683 DEBUG:boto:Token: None 2022-09-12T20:14:16.684 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/ 2022-09-12T20:14:16.685 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY= 2022-09-12T20:14:16.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY=', 'Content-Length': '0'} 2022-09-12T20:14:16.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c60cb826c1ec4a43-00631f9318-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.690 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:16.691 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:16.692 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:16.692 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:14:16.693 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:14:16.694 DEBUG:boto:Method: GET 2022-09-12T20:14:16.695 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:14:16.696 DEBUG:boto:Data: 2022-09-12T20:14:16.697 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.697 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.698 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.699 DEBUG:boto:Params: {} 2022-09-12T20:14:16.700 DEBUG:boto:Token: None 2022-09-12T20:14:16.701 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/ 2022-09-12T20:14:16.702 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY= 2022-09-12T20:14:16.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY=', 'Content-Length': '0'} 2022-09-12T20:14:16.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3ae2dd191ce388d-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.708 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:16.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:14:16.710 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:16.711 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:16.712 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:16.712 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:16.713 DEBUG:boto:Method: GET 2022-09-12T20:14:16.713 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:16.714 DEBUG:boto:Data: 2022-09-12T20:14:16.715 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.715 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.716 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.717 DEBUG:boto:Params: {} 2022-09-12T20:14:16.717 DEBUG:boto:Token: None 2022-09-12T20:14:16.718 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:16.719 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5UDbpHCRV+mwh8sk79XwIGT5z7Q= 2022-09-12T20:14:16.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5UDbpHCRV+mwh8sk79XwIGT5z7Q=', 'Content-Length': '0'} 2022-09-12T20:14:16.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000659c8f68fca5dfd2-00631f9318-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.723 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:16.723 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:16.724 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:16.725 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:16.725 DEBUG:boto:Method: GET 2022-09-12T20:14:16.726 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:16.727 DEBUG:boto:Data: 2022-09-12T20:14:16.727 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.728 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.729 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.729 DEBUG:boto:Params: {} 2022-09-12T20:14:16.730 DEBUG:boto:Token: None 2022-09-12T20:14:16.731 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:16.731 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5UDbpHCRV+mwh8sk79XwIGT5z7Q= 2022-09-12T20:14:16.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5UDbpHCRV+mwh8sk79XwIGT5z7Q=', 'Content-Length': '0'} 2022-09-12T20:14:16.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fa41f6581835c20-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.735 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:16.736 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:16.736 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:16.737 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:16.737 DEBUG:boto:Method: HEAD 2022-09-12T20:14:16.738 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:16.738 DEBUG:boto:Data: 2022-09-12T20:14:16.739 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.739 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.740 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.740 DEBUG:boto:Params: {} 2022-09-12T20:14:16.741 DEBUG:boto:Token: None 2022-09-12T20:14:16.741 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:16.742 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2JtsXI/z/uiCafA9d2vK1ly7J00= 2022-09-12T20:14:16.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2JtsXI/z/uiCafA9d2vK1ly7J00=', 'Content-Length': '0'} 2022-09-12T20:14:16.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003657a4c10ea51dd3-00631f9318-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.745 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:16.746 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:16.746 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:16.747 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:16.747 DEBUG:boto:Method: HEAD 2022-09-12T20:14:16.748 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:16.748 DEBUG:boto:Data: 2022-09-12T20:14:16.749 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.749 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.750 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.751 DEBUG:boto:Params: {} 2022-09-12T20:14:16.751 DEBUG:boto:Token: None 2022-09-12T20:14:16.752 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:16.752 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2JtsXI/z/uiCafA9d2vK1ly7J00= 2022-09-12T20:14:16.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2JtsXI/z/uiCafA9d2vK1ly7J00=', 'Content-Length': '0'} 2022-09-12T20:14:16.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2b161f6cc6b2f6a-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:14:16.758 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:16.759 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:16.760 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:16.761 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:16.761 DEBUG:boto:Method: GET 2022-09-12T20:14:16.762 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:16.762 DEBUG:boto:Data: 2022-09-12T20:14:16.763 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.763 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.764 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.764 DEBUG:boto:Params: {} 2022-09-12T20:14:16.765 DEBUG:boto:Token: None 2022-09-12T20:14:16.766 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:16.766 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5UDbpHCRV+mwh8sk79XwIGT5z7Q= 2022-09-12T20:14:16.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5UDbpHCRV+mwh8sk79XwIGT5z7Q=', 'Content-Length': '0'} 2022-09-12T20:14:16.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076977d394ce06102-00631f9318-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.810 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:16.811 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:16.811 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:16.812 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:16.812 DEBUG:boto:Method: GET 2022-09-12T20:14:16.813 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:16.813 DEBUG:boto:Data: 2022-09-12T20:14:16.814 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.815 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.815 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.816 DEBUG:boto:Params: {} 2022-09-12T20:14:16.816 DEBUG:boto:Token: None 2022-09-12T20:14:16.817 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:16.817 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5UDbpHCRV+mwh8sk79XwIGT5z7Q= 2022-09-12T20:14:16.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5UDbpHCRV+mwh8sk79XwIGT5z7Q=', 'Content-Length': '0'} 2022-09-12T20:14:16.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa178df3b8755e8b-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.821 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:16.822 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:16.823 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:14:16.823 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:14:16.824 DEBUG:boto:Method: GET 2022-09-12T20:14:16.824 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:14:16.825 DEBUG:boto:Data: 2022-09-12T20:14:16.825 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.826 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.826 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.827 DEBUG:boto:Params: {} 2022-09-12T20:14:16.827 DEBUG:boto:Token: None 2022-09-12T20:14:16.828 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/ 2022-09-12T20:14:16.829 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY= 2022-09-12T20:14:16.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY=', 'Content-Length': '0'} 2022-09-12T20:14:16.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013cc1908a9ceebe9-00631f9318-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.833 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:16.834 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:16.835 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:16.836 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:14:16.837 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:14:16.838 DEBUG:boto:Method: GET 2022-09-12T20:14:16.839 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:14:16.840 DEBUG:boto:Data: 2022-09-12T20:14:16.841 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.841 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.842 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.843 DEBUG:boto:Params: {} 2022-09-12T20:14:16.843 DEBUG:boto:Token: None 2022-09-12T20:14:16.844 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/ 2022-09-12T20:14:16.844 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY= 2022-09-12T20:14:16.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY=', 'Content-Length': '0'} 2022-09-12T20:14:16.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d75422ac4c13b8b9-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.851 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:16.852 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:16.853 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:16.854 DEBUG:boto:Method: HEAD 2022-09-12T20:14:16.855 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:14:16.855 DEBUG:boto:Data: 2022-09-12T20:14:16.856 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.856 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.857 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.858 DEBUG:boto:Params: {} 2022-09-12T20:14:16.859 DEBUG:boto:Token: None 2022-09-12T20:14:16.860 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj46 2022-09-12T20:14:16.861 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lzmIx3HfxYqjDudLF+NNmHn3cEM= 2022-09-12T20:14:16.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lzmIx3HfxYqjDudLF+NNmHn3cEM=', 'Content-Length': '0'} 2022-09-12T20:14:16.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e05d8570b6664c6-00631f9318-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.866 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:16.866 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:16.867 DEBUG:boto:Method: HEAD 2022-09-12T20:14:16.867 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:14:16.868 DEBUG:boto:Data: 2022-09-12T20:14:16.868 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.869 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.869 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.870 DEBUG:boto:Params: {} 2022-09-12T20:14:16.870 DEBUG:boto:Token: None 2022-09-12T20:14:16.871 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj46 2022-09-12T20:14:16.872 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lzmIx3HfxYqjDudLF+NNmHn3cEM= 2022-09-12T20:14:16.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lzmIx3HfxYqjDudLF+NNmHn3cEM=', 'Content-Length': '0'} 2022-09-12T20:14:16.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000288109c93d4f5bf1-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:14:16.875 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:16.876 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:16.876 DEBUG:boto:Method: GET 2022-09-12T20:14:16.877 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:14:16.878 DEBUG:boto:Data: 2022-09-12T20:14:16.878 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.879 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.880 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.881 DEBUG:boto:Params: {} 2022-09-12T20:14:16.882 DEBUG:boto:Token: None 2022-09-12T20:14:16.883 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj46 2022-09-12T20:14:16.884 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:C1Ww9oaZ/ASex7//f21eH0vwMs4= 2022-09-12T20:14:16.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:C1Ww9oaZ/ASex7//f21eH0vwMs4=', 'Content-Length': '0'} 2022-09-12T20:14:16.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001c7316071d8a938-00631f9318-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.889 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:16.890 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:16.891 DEBUG:boto:Method: GET 2022-09-12T20:14:16.891 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:14:16.892 DEBUG:boto:Data: 2022-09-12T20:14:16.893 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.894 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.895 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.896 DEBUG:boto:Params: {} 2022-09-12T20:14:16.897 DEBUG:boto:Token: None 2022-09-12T20:14:16.898 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj46 2022-09-12T20:14:16.899 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:C1Ww9oaZ/ASex7//f21eH0vwMs4= 2022-09-12T20:14:16.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:C1Ww9oaZ/ASex7//f21eH0vwMs4=', 'Content-Length': '0'} 2022-09-12T20:14:16.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac3fad2fe00a5dbf-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.904 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:16.904 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:16.905 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:14:16.905 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:14:16.906 DEBUG:boto:Method: GET 2022-09-12T20:14:16.906 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:14:16.907 DEBUG:boto:Data: 2022-09-12T20:14:16.907 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.908 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.908 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.909 DEBUG:boto:Params: {} 2022-09-12T20:14:16.909 DEBUG:boto:Token: None 2022-09-12T20:14:16.910 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/ 2022-09-12T20:14:16.910 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY= 2022-09-12T20:14:16.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY=', 'Content-Length': '0'} 2022-09-12T20:14:16.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ea12248ef20fdd2-00631f9318-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.914 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:16.915 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:16.916 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:16.916 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:14:16.917 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:14:16.917 DEBUG:boto:Method: GET 2022-09-12T20:14:16.918 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:14:16.918 DEBUG:boto:Data: 2022-09-12T20:14:16.919 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.919 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.920 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.920 DEBUG:boto:Params: {} 2022-09-12T20:14:16.921 DEBUG:boto:Token: None 2022-09-12T20:14:16.922 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/ 2022-09-12T20:14:16.923 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY= 2022-09-12T20:14:16.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PGGv4cMeA/M7sfrSz0qR8lvwTnY=', 'Content-Length': '0'} 2022-09-12T20:14:16.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006841984e33e7a3ba-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.929 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:16.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:14:16.930 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:16.931 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:16.932 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:16.933 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:16.934 DEBUG:boto:Method: GET 2022-09-12T20:14:16.935 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:16.936 DEBUG:boto:Data: 2022-09-12T20:14:16.937 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.938 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:16.939 DEBUG:boto:Port: 8001 2022-09-12T20:14:16.939 DEBUG:boto:Params: {} 2022-09-12T20:14:16.940 DEBUG:boto:Token: None 2022-09-12T20:14:16.941 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:16.943 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AMXhDeSoDtu7SZj4IuTJAVC8Q/8= 2022-09-12T20:14:16.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AMXhDeSoDtu7SZj4IuTJAVC8Q/8=', 'Content-Length': '0'} 2022-09-12T20:14:16.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d66c4497627afbc-00631f9318-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:16.948 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:16.948 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:16.949 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:16.950 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:16.951 DEBUG:boto:Method: GET 2022-09-12T20:14:16.951 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:16.952 DEBUG:boto:Data: 2022-09-12T20:14:16.952 DEBUG:boto:Headers: {} 2022-09-12T20:14:16.953 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:16.953 DEBUG:boto:Port: 8000 2022-09-12T20:14:16.954 DEBUG:boto:Params: {} 2022-09-12T20:14:16.955 DEBUG:boto:Token: None 2022-09-12T20:14:16.955 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:16 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:16.956 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AMXhDeSoDtu7SZj4IuTJAVC8Q/8= 2022-09-12T20:14:16.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AMXhDeSoDtu7SZj4IuTJAVC8Q/8=', 'Content-Length': '0'} 2022-09-12T20:14:16.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcf0e231616385cd-00631f9318-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.002 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:17.003 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:17.004 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:17.005 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:17.005 DEBUG:boto:Method: HEAD 2022-09-12T20:14:17.006 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:17.007 DEBUG:boto:Data: 2022-09-12T20:14:17.008 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.008 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.009 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.010 DEBUG:boto:Params: {} 2022-09-12T20:14:17.011 DEBUG:boto:Token: None 2022-09-12T20:14:17.012 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:17.013 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cdxsOw0NcqZnLM59K7WLD5xLd3s= 2022-09-12T20:14:17.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cdxsOw0NcqZnLM59K7WLD5xLd3s=', 'Content-Length': '0'} 2022-09-12T20:14:17.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000214fd119f9132355-00631f9319-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.018 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:17.019 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:17.019 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:17.020 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:17.021 DEBUG:boto:Method: HEAD 2022-09-12T20:14:17.022 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:17.023 DEBUG:boto:Data: 2022-09-12T20:14:17.024 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.024 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.025 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.026 DEBUG:boto:Params: {} 2022-09-12T20:14:17.027 DEBUG:boto:Token: None 2022-09-12T20:14:17.028 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:17.029 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cdxsOw0NcqZnLM59K7WLD5xLd3s= 2022-09-12T20:14:17.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cdxsOw0NcqZnLM59K7WLD5xLd3s=', 'Content-Length': '0'} 2022-09-12T20:14:17.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2e04b263b4dd56c-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:14:17.035 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:17.036 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:17.036 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:17.037 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:17.038 DEBUG:boto:Method: GET 2022-09-12T20:14:17.039 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:17.040 DEBUG:boto:Data: 2022-09-12T20:14:17.041 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.041 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.042 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.043 DEBUG:boto:Params: {} 2022-09-12T20:14:17.043 DEBUG:boto:Token: None 2022-09-12T20:14:17.044 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:17.044 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z+OtQtyYzNB+So72An+my/HjRgM= 2022-09-12T20:14:17.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z+OtQtyYzNB+So72An+my/HjRgM=', 'Content-Length': '0'} 2022-09-12T20:14:17.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009015ea645d498de3-00631f9319-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.048 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:17.048 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:17.049 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:17.049 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:17.050 DEBUG:boto:Method: GET 2022-09-12T20:14:17.050 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:17.051 DEBUG:boto:Data: 2022-09-12T20:14:17.052 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.052 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.053 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.053 DEBUG:boto:Params: {} 2022-09-12T20:14:17.054 DEBUG:boto:Token: None 2022-09-12T20:14:17.054 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:17.055 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z+OtQtyYzNB+So72An+my/HjRgM= 2022-09-12T20:14:17.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z+OtQtyYzNB+So72An+my/HjRgM=', 'Content-Length': '0'} 2022-09-12T20:14:17.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb59fddb94ee777e-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.058 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:17.059 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:17.059 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:14:17.060 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:14:17.060 DEBUG:boto:Method: GET 2022-09-12T20:14:17.061 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:14:17.061 DEBUG:boto:Data: 2022-09-12T20:14:17.062 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.063 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.063 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.064 DEBUG:boto:Params: {} 2022-09-12T20:14:17.064 DEBUG:boto:Token: None 2022-09-12T20:14:17.065 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/ 2022-09-12T20:14:17.065 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E= 2022-09-12T20:14:17.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E=', 'Content-Length': '0'} 2022-09-12T20:14:17.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbd4f94cc725451d-00631f9319-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.069 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:17.070 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:17.071 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:17.071 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:14:17.072 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:14:17.072 DEBUG:boto:Method: GET 2022-09-12T20:14:17.073 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:14:17.073 DEBUG:boto:Data: 2022-09-12T20:14:17.074 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.074 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.075 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.075 DEBUG:boto:Params: {} 2022-09-12T20:14:17.076 DEBUG:boto:Token: None 2022-09-12T20:14:17.077 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/ 2022-09-12T20:14:17.077 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E= 2022-09-12T20:14:17.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E=', 'Content-Length': '0'} 2022-09-12T20:14:17.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000303e013bac8e827a-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.083 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:17.084 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:17.084 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:17.085 DEBUG:boto:Method: HEAD 2022-09-12T20:14:17.085 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:14:17.086 DEBUG:boto:Data: 2022-09-12T20:14:17.086 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.087 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.087 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.088 DEBUG:boto:Params: {} 2022-09-12T20:14:17.088 DEBUG:boto:Token: None 2022-09-12T20:14:17.089 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj51 2022-09-12T20:14:17.089 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L1pKTUGS2DguNLVkpkz1efebzKI= 2022-09-12T20:14:17.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L1pKTUGS2DguNLVkpkz1efebzKI=', 'Content-Length': '0'} 2022-09-12T20:14:17.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000600a4105823e46d8-00631f9319-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.092 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:17.093 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:17.093 DEBUG:boto:Method: HEAD 2022-09-12T20:14:17.094 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:14:17.094 DEBUG:boto:Data: 2022-09-12T20:14:17.095 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.095 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.096 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.096 DEBUG:boto:Params: {} 2022-09-12T20:14:17.097 DEBUG:boto:Token: None 2022-09-12T20:14:17.097 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj51 2022-09-12T20:14:17.098 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L1pKTUGS2DguNLVkpkz1efebzKI= 2022-09-12T20:14:17.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L1pKTUGS2DguNLVkpkz1efebzKI=', 'Content-Length': '0'} 2022-09-12T20:14:17.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2ec04c11eab7b37-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:14:17.102 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:17.102 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:17.103 DEBUG:boto:Method: GET 2022-09-12T20:14:17.103 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:14:17.104 DEBUG:boto:Data: 2022-09-12T20:14:17.105 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.105 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.106 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.106 DEBUG:boto:Params: {} 2022-09-12T20:14:17.107 DEBUG:boto:Token: None 2022-09-12T20:14:17.107 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj51 2022-09-12T20:14:17.108 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BZQtIhZ9xO1FUm5F4UfYYyMAChg= 2022-09-12T20:14:17.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BZQtIhZ9xO1FUm5F4UfYYyMAChg=', 'Content-Length': '0'} 2022-09-12T20:14:17.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a6b303d5cadc3cd-00631f9319-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.111 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:17.112 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:17.113 DEBUG:boto:Method: GET 2022-09-12T20:14:17.113 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:14:17.114 DEBUG:boto:Data: 2022-09-12T20:14:17.114 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.115 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.115 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.116 DEBUG:boto:Params: {} 2022-09-12T20:14:17.116 DEBUG:boto:Token: None 2022-09-12T20:14:17.117 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj51 2022-09-12T20:14:17.117 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BZQtIhZ9xO1FUm5F4UfYYyMAChg= 2022-09-12T20:14:17.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BZQtIhZ9xO1FUm5F4UfYYyMAChg=', 'Content-Length': '0'} 2022-09-12T20:14:17.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002efd95e3d901b367-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.121 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:17.121 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:17.122 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:14:17.122 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:14:17.123 DEBUG:boto:Method: GET 2022-09-12T20:14:17.123 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:14:17.124 DEBUG:boto:Data: 2022-09-12T20:14:17.124 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.125 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.125 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.126 DEBUG:boto:Params: {} 2022-09-12T20:14:17.126 DEBUG:boto:Token: None 2022-09-12T20:14:17.127 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/ 2022-09-12T20:14:17.127 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E= 2022-09-12T20:14:17.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E=', 'Content-Length': '0'} 2022-09-12T20:14:17.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd26fb9f200a29d1-00631f9319-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.174 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:17.175 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:17.175 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:17.176 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:14:17.176 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:14:17.177 DEBUG:boto:Method: GET 2022-09-12T20:14:17.178 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:14:17.178 DEBUG:boto:Data: 2022-09-12T20:14:17.179 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.179 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.180 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.180 DEBUG:boto:Params: {} 2022-09-12T20:14:17.181 DEBUG:boto:Token: None 2022-09-12T20:14:17.181 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/ 2022-09-12T20:14:17.182 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E= 2022-09-12T20:14:17.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E=', 'Content-Length': '0'} 2022-09-12T20:14:17.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae21944084949bf2-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.188 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:17.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:14:17.189 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:17.190 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:17.190 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:17.191 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:17.191 DEBUG:boto:Method: GET 2022-09-12T20:14:17.192 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:17.192 DEBUG:boto:Data: 2022-09-12T20:14:17.193 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.193 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.194 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.194 DEBUG:boto:Params: {} 2022-09-12T20:14:17.195 DEBUG:boto:Token: None 2022-09-12T20:14:17.196 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:17.196 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WtDKrH/XxX0udUJYry9w+09BfNo= 2022-09-12T20:14:17.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WtDKrH/XxX0udUJYry9w+09BfNo=', 'Content-Length': '0'} 2022-09-12T20:14:17.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073fbf49cf502cb98-00631f9319-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.200 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:17.201 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:17.201 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:17.202 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:17.202 DEBUG:boto:Method: GET 2022-09-12T20:14:17.203 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:17.203 DEBUG:boto:Data: 2022-09-12T20:14:17.204 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.204 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.205 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.205 DEBUG:boto:Params: {} 2022-09-12T20:14:17.206 DEBUG:boto:Token: None 2022-09-12T20:14:17.207 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:17.207 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WtDKrH/XxX0udUJYry9w+09BfNo= 2022-09-12T20:14:17.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WtDKrH/XxX0udUJYry9w+09BfNo=', 'Content-Length': '0'} 2022-09-12T20:14:17.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d67367930b8068f-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.254 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:17.255 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:17.256 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:17.257 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:17.258 DEBUG:boto:Method: HEAD 2022-09-12T20:14:17.258 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:17.259 DEBUG:boto:Data: 2022-09-12T20:14:17.259 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.260 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.260 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.261 DEBUG:boto:Params: {} 2022-09-12T20:14:17.261 DEBUG:boto:Token: None 2022-09-12T20:14:17.262 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:17.263 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DUA8nrDwjvkAkGNlehRIGShKizo= 2022-09-12T20:14:17.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DUA8nrDwjvkAkGNlehRIGShKizo=', 'Content-Length': '0'} 2022-09-12T20:14:17.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c08e29b268717973-00631f9319-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.266 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:17.267 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:17.267 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:17.268 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:17.268 DEBUG:boto:Method: HEAD 2022-09-12T20:14:17.269 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:17.269 DEBUG:boto:Data: 2022-09-12T20:14:17.270 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.270 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.271 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.271 DEBUG:boto:Params: {} 2022-09-12T20:14:17.272 DEBUG:boto:Token: None 2022-09-12T20:14:17.272 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:17.273 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DUA8nrDwjvkAkGNlehRIGShKizo= 2022-09-12T20:14:17.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DUA8nrDwjvkAkGNlehRIGShKizo=', 'Content-Length': '0'} 2022-09-12T20:14:17.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002960375a3223bbc0-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:14:17.277 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:17.277 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:17.278 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:17.279 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:17.279 DEBUG:boto:Method: GET 2022-09-12T20:14:17.280 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:17.280 DEBUG:boto:Data: 2022-09-12T20:14:17.281 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.281 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.282 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.282 DEBUG:boto:Params: {} 2022-09-12T20:14:17.283 DEBUG:boto:Token: None 2022-09-12T20:14:17.283 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:17.284 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WtDKrH/XxX0udUJYry9w+09BfNo= 2022-09-12T20:14:17.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WtDKrH/XxX0udUJYry9w+09BfNo=', 'Content-Length': '0'} 2022-09-12T20:14:17.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4643fb231962b72-00631f9319-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.288 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:17.288 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:17.289 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:17.289 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:17.290 DEBUG:boto:Method: GET 2022-09-12T20:14:17.290 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:17.291 DEBUG:boto:Data: 2022-09-12T20:14:17.291 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.292 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.293 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.293 DEBUG:boto:Params: {} 2022-09-12T20:14:17.294 DEBUG:boto:Token: None 2022-09-12T20:14:17.294 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:17.295 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WtDKrH/XxX0udUJYry9w+09BfNo= 2022-09-12T20:14:17.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WtDKrH/XxX0udUJYry9w+09BfNo=', 'Content-Length': '0'} 2022-09-12T20:14:17.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000c0694d2d8119ad-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.298 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:17.299 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:17.299 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:14:17.300 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:14:17.300 DEBUG:boto:Method: GET 2022-09-12T20:14:17.301 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:14:17.301 DEBUG:boto:Data: 2022-09-12T20:14:17.302 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.302 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.303 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.304 DEBUG:boto:Params: {} 2022-09-12T20:14:17.304 DEBUG:boto:Token: None 2022-09-12T20:14:17.305 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/ 2022-09-12T20:14:17.305 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E= 2022-09-12T20:14:17.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E=', 'Content-Length': '0'} 2022-09-12T20:14:17.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009db7c375a0264f6e-00631f9319-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.310 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:17.310 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:17.311 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:17.311 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:14:17.312 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:14:17.312 DEBUG:boto:Method: GET 2022-09-12T20:14:17.313 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:14:17.313 DEBUG:boto:Data: 2022-09-12T20:14:17.314 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.314 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.315 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.316 DEBUG:boto:Params: {} 2022-09-12T20:14:17.316 DEBUG:boto:Token: None 2022-09-12T20:14:17.317 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/ 2022-09-12T20:14:17.317 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E= 2022-09-12T20:14:17.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E=', 'Content-Length': '0'} 2022-09-12T20:14:17.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6a4790a357ec931-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.323 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:17.324 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:17.324 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:17.325 DEBUG:boto:Method: HEAD 2022-09-12T20:14:17.325 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:14:17.326 DEBUG:boto:Data: 2022-09-12T20:14:17.326 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.327 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.327 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.328 DEBUG:boto:Params: {} 2022-09-12T20:14:17.328 DEBUG:boto:Token: None 2022-09-12T20:14:17.329 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj55 2022-09-12T20:14:17.329 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HNl43wHOt3TtkZCRzopDTsid3OA= 2022-09-12T20:14:17.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HNl43wHOt3TtkZCRzopDTsid3OA=', 'Content-Length': '0'} 2022-09-12T20:14:17.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e853477fc5bfa3c-00631f9319-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.332 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:17.333 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:17.333 DEBUG:boto:Method: HEAD 2022-09-12T20:14:17.334 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:14:17.334 DEBUG:boto:Data: 2022-09-12T20:14:17.335 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.335 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.336 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.336 DEBUG:boto:Params: {} 2022-09-12T20:14:17.337 DEBUG:boto:Token: None 2022-09-12T20:14:17.337 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj55 2022-09-12T20:14:17.338 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HNl43wHOt3TtkZCRzopDTsid3OA= 2022-09-12T20:14:17.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HNl43wHOt3TtkZCRzopDTsid3OA=', 'Content-Length': '0'} 2022-09-12T20:14:17.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009feffc6c098cec76-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:14:17.342 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:17.342 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:17.343 DEBUG:boto:Method: GET 2022-09-12T20:14:17.343 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:14:17.344 DEBUG:boto:Data: 2022-09-12T20:14:17.344 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.345 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.345 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.346 DEBUG:boto:Params: {} 2022-09-12T20:14:17.346 DEBUG:boto:Token: None 2022-09-12T20:14:17.347 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj55 2022-09-12T20:14:17.348 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UD+mPC5hwwj5P+W9SIE4ii3a4Do= 2022-09-12T20:14:17.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UD+mPC5hwwj5P+W9SIE4ii3a4Do=', 'Content-Length': '0'} 2022-09-12T20:14:17.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d74258b3ac952d52-00631f9319-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.351 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:17.351 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:17.352 DEBUG:boto:Method: GET 2022-09-12T20:14:17.353 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:14:17.353 DEBUG:boto:Data: 2022-09-12T20:14:17.354 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.354 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.355 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.355 DEBUG:boto:Params: {} 2022-09-12T20:14:17.356 DEBUG:boto:Token: None 2022-09-12T20:14:17.356 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj55 2022-09-12T20:14:17.357 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UD+mPC5hwwj5P+W9SIE4ii3a4Do= 2022-09-12T20:14:17.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UD+mPC5hwwj5P+W9SIE4ii3a4Do=', 'Content-Length': '0'} 2022-09-12T20:14:17.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e070f4d8adf5486f-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.360 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:17.361 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:17.361 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:14:17.362 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:14:17.362 DEBUG:boto:Method: GET 2022-09-12T20:14:17.363 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:14:17.363 DEBUG:boto:Data: 2022-09-12T20:14:17.364 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.364 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.365 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.365 DEBUG:boto:Params: {} 2022-09-12T20:14:17.366 DEBUG:boto:Token: None 2022-09-12T20:14:17.367 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/ 2022-09-12T20:14:17.367 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E= 2022-09-12T20:14:17.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E=', 'Content-Length': '0'} 2022-09-12T20:14:17.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000752bfe5161dcf87d-00631f9319-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.414 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:17.415 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:17.416 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:17.416 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:14:17.417 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:14:17.417 DEBUG:boto:Method: GET 2022-09-12T20:14:17.418 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:14:17.418 DEBUG:boto:Data: 2022-09-12T20:14:17.419 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.419 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.420 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.421 DEBUG:boto:Params: {} 2022-09-12T20:14:17.421 DEBUG:boto:Token: None 2022-09-12T20:14:17.422 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/ 2022-09-12T20:14:17.422 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E= 2022-09-12T20:14:17.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E=', 'Content-Length': '0'} 2022-09-12T20:14:17.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe2763819f522069-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.428 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:17.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:14:17.429 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:17.430 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:17.430 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:17.431 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:17.431 DEBUG:boto:Method: GET 2022-09-12T20:14:17.432 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:17.432 DEBUG:boto:Data: 2022-09-12T20:14:17.433 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.434 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.434 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.435 DEBUG:boto:Params: {} 2022-09-12T20:14:17.435 DEBUG:boto:Token: None 2022-09-12T20:14:17.436 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:17.436 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9WYmftV4cvpbIOs3JuzKVxP5if8= 2022-09-12T20:14:17.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9WYmftV4cvpbIOs3JuzKVxP5if8=', 'Content-Length': '0'} 2022-09-12T20:14:17.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000532d3f07325628ef-00631f9319-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.440 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:17.440 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:17.441 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:17.441 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:17.442 DEBUG:boto:Method: GET 2022-09-12T20:14:17.443 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:17.443 DEBUG:boto:Data: 2022-09-12T20:14:17.444 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.444 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.445 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.445 DEBUG:boto:Params: {} 2022-09-12T20:14:17.446 DEBUG:boto:Token: None 2022-09-12T20:14:17.446 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:17.447 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9WYmftV4cvpbIOs3JuzKVxP5if8= 2022-09-12T20:14:17.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9WYmftV4cvpbIOs3JuzKVxP5if8=', 'Content-Length': '0'} 2022-09-12T20:14:17.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0a1e035248954cf-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.494 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:17.495 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:17.495 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:17.496 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:17.496 DEBUG:boto:Method: HEAD 2022-09-12T20:14:17.497 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:17.497 DEBUG:boto:Data: 2022-09-12T20:14:17.498 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.498 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.499 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.500 DEBUG:boto:Params: {} 2022-09-12T20:14:17.500 DEBUG:boto:Token: None 2022-09-12T20:14:17.501 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:17.502 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SkablA++WGHzbd7MRNxMtqqN3VY= 2022-09-12T20:14:17.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SkablA++WGHzbd7MRNxMtqqN3VY=', 'Content-Length': '0'} 2022-09-12T20:14:17.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a191b73844fbef5-00631f9319-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.505 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:17.506 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:17.506 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:17.507 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:17.507 DEBUG:boto:Method: HEAD 2022-09-12T20:14:17.508 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:17.508 DEBUG:boto:Data: 2022-09-12T20:14:17.509 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.510 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.510 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.511 DEBUG:boto:Params: {} 2022-09-12T20:14:17.511 DEBUG:boto:Token: None 2022-09-12T20:14:17.512 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:17.513 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SkablA++WGHzbd7MRNxMtqqN3VY= 2022-09-12T20:14:17.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SkablA++WGHzbd7MRNxMtqqN3VY=', 'Content-Length': '0'} 2022-09-12T20:14:17.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052271ef8c429b9d7-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:14:17.516 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:17.517 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:17.517 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:17.518 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:17.519 DEBUG:boto:Method: GET 2022-09-12T20:14:17.519 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:17.520 DEBUG:boto:Data: 2022-09-12T20:14:17.520 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.521 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.521 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.522 DEBUG:boto:Params: {} 2022-09-12T20:14:17.522 DEBUG:boto:Token: None 2022-09-12T20:14:17.523 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:17.524 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9WYmftV4cvpbIOs3JuzKVxP5if8= 2022-09-12T20:14:17.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9WYmftV4cvpbIOs3JuzKVxP5if8=', 'Content-Length': '0'} 2022-09-12T20:14:17.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f424898744f674b-00631f9319-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.527 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:17.527 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:17.528 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:17.529 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:17.529 DEBUG:boto:Method: GET 2022-09-12T20:14:17.530 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:17.530 DEBUG:boto:Data: 2022-09-12T20:14:17.531 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.531 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.532 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.532 DEBUG:boto:Params: {} 2022-09-12T20:14:17.533 DEBUG:boto:Token: None 2022-09-12T20:14:17.534 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:17.535 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9WYmftV4cvpbIOs3JuzKVxP5if8= 2022-09-12T20:14:17.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9WYmftV4cvpbIOs3JuzKVxP5if8=', 'Content-Length': '0'} 2022-09-12T20:14:17.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5b063f44825c9bb-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.538 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:17.539 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:17.539 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:14:17.540 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:14:17.540 DEBUG:boto:Method: GET 2022-09-12T20:14:17.541 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:14:17.541 DEBUG:boto:Data: 2022-09-12T20:14:17.542 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.542 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.543 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.543 DEBUG:boto:Params: {} 2022-09-12T20:14:17.544 DEBUG:boto:Token: None 2022-09-12T20:14:17.544 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/ 2022-09-12T20:14:17.545 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E= 2022-09-12T20:14:17.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E=', 'Content-Length': '0'} 2022-09-12T20:14:17.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033aff5ed7bb65777-00631f9319-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.549 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:17.550 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:17.550 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:17.551 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:14:17.551 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:14:17.552 DEBUG:boto:Method: GET 2022-09-12T20:14:17.552 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:14:17.553 DEBUG:boto:Data: 2022-09-12T20:14:17.553 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.554 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.555 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.555 DEBUG:boto:Params: {} 2022-09-12T20:14:17.556 DEBUG:boto:Token: None 2022-09-12T20:14:17.556 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/ 2022-09-12T20:14:17.557 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E= 2022-09-12T20:14:17.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E=', 'Content-Length': '0'} 2022-09-12T20:14:17.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009844802a1164f1de-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.562 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:17.563 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:17.564 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:17.564 DEBUG:boto:Method: HEAD 2022-09-12T20:14:17.565 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:14:17.565 DEBUG:boto:Data: 2022-09-12T20:14:17.566 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.566 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.567 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.567 DEBUG:boto:Params: {} 2022-09-12T20:14:17.568 DEBUG:boto:Token: None 2022-09-12T20:14:17.568 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj57 2022-09-12T20:14:17.569 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t9Ae12IdwH/9WHqEpc3Q4HFxmdE= 2022-09-12T20:14:17.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t9Ae12IdwH/9WHqEpc3Q4HFxmdE=', 'Content-Length': '0'} 2022-09-12T20:14:17.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004387ea1f92191778-00631f9319-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.572 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:17.573 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:17.573 DEBUG:boto:Method: HEAD 2022-09-12T20:14:17.574 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:14:17.574 DEBUG:boto:Data: 2022-09-12T20:14:17.575 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.575 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.576 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.576 DEBUG:boto:Params: {} 2022-09-12T20:14:17.577 DEBUG:boto:Token: None 2022-09-12T20:14:17.577 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj57 2022-09-12T20:14:17.578 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t9Ae12IdwH/9WHqEpc3Q4HFxmdE= 2022-09-12T20:14:17.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t9Ae12IdwH/9WHqEpc3Q4HFxmdE=', 'Content-Length': '0'} 2022-09-12T20:14:17.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011d32844b4a5053b-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:14:17.582 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:17.582 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:17.583 DEBUG:boto:Method: GET 2022-09-12T20:14:17.583 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:14:17.584 DEBUG:boto:Data: 2022-09-12T20:14:17.584 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.585 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.585 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.586 DEBUG:boto:Params: {} 2022-09-12T20:14:17.586 DEBUG:boto:Token: None 2022-09-12T20:14:17.587 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj57 2022-09-12T20:14:17.588 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6NBm4IFl/xPpG+5Ds15iGCipix4= 2022-09-12T20:14:17.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6NBm4IFl/xPpG+5Ds15iGCipix4=', 'Content-Length': '0'} 2022-09-12T20:14:17.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f47c404a40f554f-00631f9319-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.592 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:17.593 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:17.594 DEBUG:boto:Method: GET 2022-09-12T20:14:17.595 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:14:17.595 DEBUG:boto:Data: 2022-09-12T20:14:17.596 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.597 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.598 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.599 DEBUG:boto:Params: {} 2022-09-12T20:14:17.600 DEBUG:boto:Token: None 2022-09-12T20:14:17.601 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj57 2022-09-12T20:14:17.602 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6NBm4IFl/xPpG+5Ds15iGCipix4= 2022-09-12T20:14:17.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6NBm4IFl/xPpG+5Ds15iGCipix4=', 'Content-Length': '0'} 2022-09-12T20:14:17.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e6b73fd276a4e92-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.607 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:17.607 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:17.608 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:14:17.608 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:14:17.609 DEBUG:boto:Method: GET 2022-09-12T20:14:17.609 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:14:17.610 DEBUG:boto:Data: 2022-09-12T20:14:17.611 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.611 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.612 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.612 DEBUG:boto:Params: {} 2022-09-12T20:14:17.613 DEBUG:boto:Token: None 2022-09-12T20:14:17.613 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/ 2022-09-12T20:14:17.614 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E= 2022-09-12T20:14:17.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E=', 'Content-Length': '0'} 2022-09-12T20:14:17.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d58c43810119f1d2-00631f9319-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.658 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:17.659 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:17.659 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:17.660 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:14:17.660 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:14:17.661 DEBUG:boto:Method: GET 2022-09-12T20:14:17.661 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:14:17.662 DEBUG:boto:Data: 2022-09-12T20:14:17.662 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.663 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.664 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.664 DEBUG:boto:Params: {} 2022-09-12T20:14:17.665 DEBUG:boto:Token: None 2022-09-12T20:14:17.665 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/ 2022-09-12T20:14:17.666 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E= 2022-09-12T20:14:17.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E=', 'Content-Length': '0'} 2022-09-12T20:14:17.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008eeecae802ddbd37-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.671 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:17.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:14:17.673 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:17.673 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:17.674 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:17.674 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:17.675 DEBUG:boto:Method: GET 2022-09-12T20:14:17.675 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:17.676 DEBUG:boto:Data: 2022-09-12T20:14:17.677 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.677 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.678 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.678 DEBUG:boto:Params: {} 2022-09-12T20:14:17.679 DEBUG:boto:Token: None 2022-09-12T20:14:17.679 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:17.680 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iSsUhsjzVY3jPjtUGP3lQ6LN2Xc= 2022-09-12T20:14:17.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iSsUhsjzVY3jPjtUGP3lQ6LN2Xc=', 'Content-Length': '0'} 2022-09-12T20:14:17.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b8b89f2bdee4bae-00631f9319-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.683 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:17.684 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:17.684 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:17.685 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:17.685 DEBUG:boto:Method: GET 2022-09-12T20:14:17.686 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:17.686 DEBUG:boto:Data: 2022-09-12T20:14:17.687 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.688 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.688 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.689 DEBUG:boto:Params: {} 2022-09-12T20:14:17.689 DEBUG:boto:Token: None 2022-09-12T20:14:17.690 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:17.690 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iSsUhsjzVY3jPjtUGP3lQ6LN2Xc= 2022-09-12T20:14:17.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iSsUhsjzVY3jPjtUGP3lQ6LN2Xc=', 'Content-Length': '0'} 2022-09-12T20:14:17.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ca9682a79463ad5-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.694 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:17.694 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:17.695 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:17.695 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:17.696 DEBUG:boto:Method: HEAD 2022-09-12T20:14:17.696 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:17.697 DEBUG:boto:Data: 2022-09-12T20:14:17.697 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.698 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.698 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.699 DEBUG:boto:Params: {} 2022-09-12T20:14:17.699 DEBUG:boto:Token: None 2022-09-12T20:14:17.700 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:17.700 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AAEwaNxa1KeMtndDequwZ2W7obg= 2022-09-12T20:14:17.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AAEwaNxa1KeMtndDequwZ2W7obg=', 'Content-Length': '0'} 2022-09-12T20:14:17.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f2326ce5d07418e-00631f9319-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.704 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:17.704 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:17.705 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:17.705 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:17.706 DEBUG:boto:Method: HEAD 2022-09-12T20:14:17.706 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:17.707 DEBUG:boto:Data: 2022-09-12T20:14:17.707 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.708 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.708 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.709 DEBUG:boto:Params: {} 2022-09-12T20:14:17.709 DEBUG:boto:Token: None 2022-09-12T20:14:17.710 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:17.710 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AAEwaNxa1KeMtndDequwZ2W7obg= 2022-09-12T20:14:17.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AAEwaNxa1KeMtndDequwZ2W7obg=', 'Content-Length': '0'} 2022-09-12T20:14:17.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbb3a6066fe6c339-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:14:17.714 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:17.715 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:17.715 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:17.716 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:17.716 DEBUG:boto:Method: GET 2022-09-12T20:14:17.717 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:17.717 DEBUG:boto:Data: 2022-09-12T20:14:17.718 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.719 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.719 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.720 DEBUG:boto:Params: {} 2022-09-12T20:14:17.720 DEBUG:boto:Token: None 2022-09-12T20:14:17.721 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:17.721 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iSsUhsjzVY3jPjtUGP3lQ6LN2Xc= 2022-09-12T20:14:17.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iSsUhsjzVY3jPjtUGP3lQ6LN2Xc=', 'Content-Length': '0'} 2022-09-12T20:14:17.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1004b8b30c0a983-00631f9319-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.725 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:17.725 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:17.726 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:17.726 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:17.727 DEBUG:boto:Method: GET 2022-09-12T20:14:17.728 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:17.728 DEBUG:boto:Data: 2022-09-12T20:14:17.729 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.729 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.730 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.730 DEBUG:boto:Params: {} 2022-09-12T20:14:17.731 DEBUG:boto:Token: None 2022-09-12T20:14:17.731 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:17.732 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iSsUhsjzVY3jPjtUGP3lQ6LN2Xc= 2022-09-12T20:14:17.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iSsUhsjzVY3jPjtUGP3lQ6LN2Xc=', 'Content-Length': '0'} 2022-09-12T20:14:17.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ae5cd27be68d0be-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.736 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:17.737 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:17.737 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:14:17.738 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:14:17.738 DEBUG:boto:Method: GET 2022-09-12T20:14:17.739 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:14:17.739 DEBUG:boto:Data: 2022-09-12T20:14:17.740 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.740 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.741 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.741 DEBUG:boto:Params: {} 2022-09-12T20:14:17.742 DEBUG:boto:Token: None 2022-09-12T20:14:17.742 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/ 2022-09-12T20:14:17.743 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E= 2022-09-12T20:14:17.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E=', 'Content-Length': '0'} 2022-09-12T20:14:17.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd5582a4bc95a637-00631f9319-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.747 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:17.748 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:17.749 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:17.749 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:14:17.750 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:14:17.750 DEBUG:boto:Method: GET 2022-09-12T20:14:17.751 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:14:17.751 DEBUG:boto:Data: 2022-09-12T20:14:17.752 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.752 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.753 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.753 DEBUG:boto:Params: {} 2022-09-12T20:14:17.754 DEBUG:boto:Token: None 2022-09-12T20:14:17.754 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/ 2022-09-12T20:14:17.761 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E= 2022-09-12T20:14:17.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E=', 'Content-Length': '0'} 2022-09-12T20:14:17.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006df20f51a5d0db13-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.810 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:17.811 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:17.812 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:17.812 DEBUG:boto:Method: HEAD 2022-09-12T20:14:17.813 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:14:17.813 DEBUG:boto:Data: 2022-09-12T20:14:17.814 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.814 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.815 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.815 DEBUG:boto:Params: {} 2022-09-12T20:14:17.816 DEBUG:boto:Token: None 2022-09-12T20:14:17.816 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj58 2022-09-12T20:14:17.817 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VyZxWF+m4M6yO5re+F0MxDNbR5A= 2022-09-12T20:14:17.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VyZxWF+m4M6yO5re+F0MxDNbR5A=', 'Content-Length': '0'} 2022-09-12T20:14:17.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e0e029c1b6399b0-00631f9319-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.820 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:17.821 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:17.821 DEBUG:boto:Method: HEAD 2022-09-12T20:14:17.822 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:14:17.822 DEBUG:boto:Data: 2022-09-12T20:14:17.823 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.823 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.824 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.824 DEBUG:boto:Params: {} 2022-09-12T20:14:17.825 DEBUG:boto:Token: None 2022-09-12T20:14:17.825 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj58 2022-09-12T20:14:17.826 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VyZxWF+m4M6yO5re+F0MxDNbR5A= 2022-09-12T20:14:17.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VyZxWF+m4M6yO5re+F0MxDNbR5A=', 'Content-Length': '0'} 2022-09-12T20:14:17.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099c1fde2b700f5e0-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:14:17.830 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:17.831 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:17.831 DEBUG:boto:Method: GET 2022-09-12T20:14:17.832 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:14:17.832 DEBUG:boto:Data: 2022-09-12T20:14:17.833 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.833 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.834 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.834 DEBUG:boto:Params: {} 2022-09-12T20:14:17.835 DEBUG:boto:Token: None 2022-09-12T20:14:17.835 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj58 2022-09-12T20:14:17.836 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Sqr/q7nsOzPhG5NFE1YyxCp/CFA= 2022-09-12T20:14:17.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Sqr/q7nsOzPhG5NFE1YyxCp/CFA=', 'Content-Length': '0'} 2022-09-12T20:14:17.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5af10809a0f4e31-00631f9319-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.840 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:17.840 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:17.841 DEBUG:boto:Method: GET 2022-09-12T20:14:17.841 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:14:17.842 DEBUG:boto:Data: 2022-09-12T20:14:17.842 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.843 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.844 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.844 DEBUG:boto:Params: {} 2022-09-12T20:14:17.845 DEBUG:boto:Token: None 2022-09-12T20:14:17.845 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj58 2022-09-12T20:14:17.846 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Sqr/q7nsOzPhG5NFE1YyxCp/CFA= 2022-09-12T20:14:17.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Sqr/q7nsOzPhG5NFE1YyxCp/CFA=', 'Content-Length': '0'} 2022-09-12T20:14:17.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047c1bb6ce0ea030f-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.849 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:17.850 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:17.850 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:14:17.851 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:14:17.852 DEBUG:boto:Method: GET 2022-09-12T20:14:17.852 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:14:17.853 DEBUG:boto:Data: 2022-09-12T20:14:17.853 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.854 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.854 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.855 DEBUG:boto:Params: {} 2022-09-12T20:14:17.855 DEBUG:boto:Token: None 2022-09-12T20:14:17.856 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/ 2022-09-12T20:14:17.856 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E= 2022-09-12T20:14:17.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E=', 'Content-Length': '0'} 2022-09-12T20:14:17.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8ee6dee37304172-00631f9319-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.902 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:17.903 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:17.904 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:17.905 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:14:17.905 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:14:17.906 DEBUG:boto:Method: GET 2022-09-12T20:14:17.906 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:14:17.907 DEBUG:boto:Data: 2022-09-12T20:14:17.907 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.908 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.908 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.909 DEBUG:boto:Params: {} 2022-09-12T20:14:17.909 DEBUG:boto:Token: None 2022-09-12T20:14:17.910 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/ 2022-09-12T20:14:17.911 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E= 2022-09-12T20:14:17.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jlHh5O/zkXcvcP9ueSz342R8t3E=', 'Content-Length': '0'} 2022-09-12T20:14:17.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a684dd901fcb739e-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.916 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:17.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-12T20:14:17.918 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:17.918 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:17.919 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:17.919 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:17.920 DEBUG:boto:Method: GET 2022-09-12T20:14:17.920 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:17.921 DEBUG:boto:Data: 2022-09-12T20:14:17.921 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.922 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.922 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.923 DEBUG:boto:Params: {} 2022-09-12T20:14:17.924 DEBUG:boto:Token: None 2022-09-12T20:14:17.924 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:17.925 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nhGDxDg1t6IwehH+odMMyOy4CjU= 2022-09-12T20:14:17.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nhGDxDg1t6IwehH+odMMyOy4CjU=', 'Content-Length': '0'} 2022-09-12T20:14:17.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6b97f479dc41ff7-00631f9319-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.928 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:17.929 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:17.929 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:17.930 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:17.930 DEBUG:boto:Method: GET 2022-09-12T20:14:17.931 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:17.932 DEBUG:boto:Data: 2022-09-12T20:14:17.932 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.933 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.933 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.934 DEBUG:boto:Params: {} 2022-09-12T20:14:17.934 DEBUG:boto:Token: None 2022-09-12T20:14:17.935 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:17.936 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nhGDxDg1t6IwehH+odMMyOy4CjU= 2022-09-12T20:14:17.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nhGDxDg1t6IwehH+odMMyOy4CjU=', 'Content-Length': '0'} 2022-09-12T20:14:17.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b00c048a89cef14d-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.939 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:17.940 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:17.940 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:17.941 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:17.941 DEBUG:boto:Method: HEAD 2022-09-12T20:14:17.942 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:17.942 DEBUG:boto:Data: 2022-09-12T20:14:17.943 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.943 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.944 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.944 DEBUG:boto:Params: {} 2022-09-12T20:14:17.945 DEBUG:boto:Token: None 2022-09-12T20:14:17.946 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:17.946 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9C+NOT6Nt9HGjqXfik24mBA1J4U= 2022-09-12T20:14:17.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9C+NOT6Nt9HGjqXfik24mBA1J4U=', 'Content-Length': '0'} 2022-09-12T20:14:17.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ac53dd12ba87d5b-00631f9319-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.950 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:17.950 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:17.951 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:17.951 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:17.952 DEBUG:boto:Method: HEAD 2022-09-12T20:14:17.952 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:17.953 DEBUG:boto:Data: 2022-09-12T20:14:17.954 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.954 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.955 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.956 DEBUG:boto:Params: {} 2022-09-12T20:14:17.957 DEBUG:boto:Token: None 2022-09-12T20:14:17.958 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:17.959 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9C+NOT6Nt9HGjqXfik24mBA1J4U= 2022-09-12T20:14:17.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9C+NOT6Nt9HGjqXfik24mBA1J4U=', 'Content-Length': '0'} 2022-09-12T20:14:17.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c0cd18eaaf32bf2-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-12T20:14:17.966 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:17.967 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:17.968 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:17.969 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:17.970 DEBUG:boto:Method: GET 2022-09-12T20:14:17.971 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:17.972 DEBUG:boto:Data: 2022-09-12T20:14:17.973 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.974 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:17.975 DEBUG:boto:Port: 8001 2022-09-12T20:14:17.976 DEBUG:boto:Params: {} 2022-09-12T20:14:17.977 DEBUG:boto:Token: None 2022-09-12T20:14:17.978 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:17.978 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nhGDxDg1t6IwehH+odMMyOy4CjU= 2022-09-12T20:14:17.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nhGDxDg1t6IwehH+odMMyOy4CjU=', 'Content-Length': '0'} 2022-09-12T20:14:17.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000611229c65aa2110b-00631f9319-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:17.982 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:17.983 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:17.983 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:17.984 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:17.984 DEBUG:boto:Method: GET 2022-09-12T20:14:17.985 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:17.985 DEBUG:boto:Data: 2022-09-12T20:14:17.986 DEBUG:boto:Headers: {} 2022-09-12T20:14:17.986 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:17.987 DEBUG:boto:Port: 8000 2022-09-12T20:14:17.988 DEBUG:boto:Params: {} 2022-09-12T20:14:17.988 DEBUG:boto:Token: None 2022-09-12T20:14:17.989 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:17 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:17.990 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nhGDxDg1t6IwehH+odMMyOy4CjU= 2022-09-12T20:14:17.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nhGDxDg1t6IwehH+odMMyOy4CjU=', 'Content-Length': '0'} 2022-09-12T20:14:17.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081f5dd17e1374023-00631f9319-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.034 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:18.035 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:18.035 DEBUG:boto:path=/jdpiak-75/?prefix=obj62 2022-09-12T20:14:18.036 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj62 2022-09-12T20:14:18.036 DEBUG:boto:Method: GET 2022-09-12T20:14:18.037 DEBUG:boto:Path: /jdpiak-75/?prefix=obj62 2022-09-12T20:14:18.038 DEBUG:boto:Data: 2022-09-12T20:14:18.038 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.039 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.039 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.040 DEBUG:boto:Params: {} 2022-09-12T20:14:18.040 DEBUG:boto:Token: None 2022-09-12T20:14:18.041 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/ 2022-09-12T20:14:18.042 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s= 2022-09-12T20:14:18.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s=', 'Content-Length': '0'} 2022-09-12T20:14:18.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f813096b736b935-00631f931a-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.046 DEBUG:boto:b'jdpiak-75obj621000falseobj622022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:18.047 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:18.048 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:18.048 DEBUG:boto:path=/jdpiak-75/?prefix=obj62 2022-09-12T20:14:18.049 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj62 2022-09-12T20:14:18.049 DEBUG:boto:Method: GET 2022-09-12T20:14:18.050 DEBUG:boto:Path: /jdpiak-75/?prefix=obj62 2022-09-12T20:14:18.050 DEBUG:boto:Data: 2022-09-12T20:14:18.051 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.052 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.052 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.053 DEBUG:boto:Params: {} 2022-09-12T20:14:18.053 DEBUG:boto:Token: None 2022-09-12T20:14:18.054 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/ 2022-09-12T20:14:18.055 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s= 2022-09-12T20:14:18.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s=', 'Content-Length': '0'} 2022-09-12T20:14:18.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000196562ceb8ded889-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.062 DEBUG:boto:b'jdpiak-75obj621000falseobj622022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:18.063 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:18.064 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:18.064 DEBUG:boto:Method: HEAD 2022-09-12T20:14:18.065 DEBUG:boto:Path: /jdpiak-75/obj62 2022-09-12T20:14:18.065 DEBUG:boto:Data: 2022-09-12T20:14:18.066 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.066 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.067 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.067 DEBUG:boto:Params: {} 2022-09-12T20:14:18.068 DEBUG:boto:Token: None 2022-09-12T20:14:18.069 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj62 2022-09-12T20:14:18.069 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V0H1tyFEZ+UutEBRwyl4lGWWWUc= 2022-09-12T20:14:18.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V0H1tyFEZ+UutEBRwyl4lGWWWUc=', 'Content-Length': '0'} 2022-09-12T20:14:18.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000b6439cf948e31d-00631f931a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.073 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:18.073 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:18.074 DEBUG:boto:Method: HEAD 2022-09-12T20:14:18.075 DEBUG:boto:Path: /jdpiak-75/obj62 2022-09-12T20:14:18.075 DEBUG:boto:Data: 2022-09-12T20:14:18.077 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.077 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.078 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.078 DEBUG:boto:Params: {} 2022-09-12T20:14:18.079 DEBUG:boto:Token: None 2022-09-12T20:14:18.080 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj62 2022-09-12T20:14:18.080 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V0H1tyFEZ+UutEBRwyl4lGWWWUc= 2022-09-12T20:14:18.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V0H1tyFEZ+UutEBRwyl4lGWWWUc=', 'Content-Length': '0'} 2022-09-12T20:14:18.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c00121cf94a449ea-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-12T20:14:18.084 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:18.085 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:18.085 DEBUG:boto:Method: GET 2022-09-12T20:14:18.086 DEBUG:boto:Path: /jdpiak-75/obj62 2022-09-12T20:14:18.087 DEBUG:boto:Data: 2022-09-12T20:14:18.087 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.088 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.089 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.089 DEBUG:boto:Params: {} 2022-09-12T20:14:18.090 DEBUG:boto:Token: None 2022-09-12T20:14:18.090 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj62 2022-09-12T20:14:18.091 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x9YRGs+csOw/SCfosMflvPDUXfU= 2022-09-12T20:14:18.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x9YRGs+csOw/SCfosMflvPDUXfU=', 'Content-Length': '0'} 2022-09-12T20:14:18.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5e3cdb76621c96d-00631f931a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.094 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:18.095 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:18.095 DEBUG:boto:Method: GET 2022-09-12T20:14:18.096 DEBUG:boto:Path: /jdpiak-75/obj62 2022-09-12T20:14:18.096 DEBUG:boto:Data: 2022-09-12T20:14:18.097 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.098 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.098 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.099 DEBUG:boto:Params: {} 2022-09-12T20:14:18.100 DEBUG:boto:Token: None 2022-09-12T20:14:18.100 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj62 2022-09-12T20:14:18.101 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x9YRGs+csOw/SCfosMflvPDUXfU= 2022-09-12T20:14:18.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x9YRGs+csOw/SCfosMflvPDUXfU=', 'Content-Length': '0'} 2022-09-12T20:14:18.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abc1bf27cad53548-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.146 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:18.147 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:18.147 DEBUG:boto:path=/jdpiak-75/?prefix=obj62 2022-09-12T20:14:18.148 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj62 2022-09-12T20:14:18.149 DEBUG:boto:Method: GET 2022-09-12T20:14:18.149 DEBUG:boto:Path: /jdpiak-75/?prefix=obj62 2022-09-12T20:14:18.150 DEBUG:boto:Data: 2022-09-12T20:14:18.151 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.151 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.152 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.152 DEBUG:boto:Params: {} 2022-09-12T20:14:18.153 DEBUG:boto:Token: None 2022-09-12T20:14:18.153 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/ 2022-09-12T20:14:18.154 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s= 2022-09-12T20:14:18.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s=', 'Content-Length': '0'} 2022-09-12T20:14:18.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c62d2723641794e1-00631f931a-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.158 DEBUG:boto:b'jdpiak-75obj621000falseobj622022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:18.159 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:18.160 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:18.160 DEBUG:boto:path=/jdpiak-75/?prefix=obj62 2022-09-12T20:14:18.161 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj62 2022-09-12T20:14:18.161 DEBUG:boto:Method: GET 2022-09-12T20:14:18.162 DEBUG:boto:Path: /jdpiak-75/?prefix=obj62 2022-09-12T20:14:18.163 DEBUG:boto:Data: 2022-09-12T20:14:18.163 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.164 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.164 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.165 DEBUG:boto:Params: {} 2022-09-12T20:14:18.165 DEBUG:boto:Token: None 2022-09-12T20:14:18.166 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/ 2022-09-12T20:14:18.167 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s= 2022-09-12T20:14:18.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s=', 'Content-Length': '0'} 2022-09-12T20:14:18.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d528c0f478884f4f-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.172 DEBUG:boto:b'jdpiak-75obj621000falseobj622022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:18.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:14:18.173 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:18.174 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:18.174 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:18.175 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:18.176 DEBUG:boto:Method: GET 2022-09-12T20:14:18.176 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:18.177 DEBUG:boto:Data: 2022-09-12T20:14:18.177 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.178 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.178 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.179 DEBUG:boto:Params: {} 2022-09-12T20:14:18.180 DEBUG:boto:Token: None 2022-09-12T20:14:18.180 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:18.181 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4M8iq2RwBh9IBo+f61aP4Ck+kSE= 2022-09-12T20:14:18.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4M8iq2RwBh9IBo+f61aP4Ck+kSE=', 'Content-Length': '0'} 2022-09-12T20:14:18.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007417cecebdb77429-00631f931a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.184 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:18.185 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:18.185 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:18.186 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:18.186 DEBUG:boto:Method: GET 2022-09-12T20:14:18.187 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:18.188 DEBUG:boto:Data: 2022-09-12T20:14:18.188 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.189 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.189 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.190 DEBUG:boto:Params: {} 2022-09-12T20:14:18.190 DEBUG:boto:Token: None 2022-09-12T20:14:18.191 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:18.191 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4M8iq2RwBh9IBo+f61aP4Ck+kSE= 2022-09-12T20:14:18.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4M8iq2RwBh9IBo+f61aP4Ck+kSE=', 'Content-Length': '0'} 2022-09-12T20:14:18.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6f7911d52d57804-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.195 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:18.195 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:18.196 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:18.196 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:18.197 DEBUG:boto:Method: HEAD 2022-09-12T20:14:18.198 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:18.198 DEBUG:boto:Data: 2022-09-12T20:14:18.199 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.199 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.200 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.200 DEBUG:boto:Params: {} 2022-09-12T20:14:18.201 DEBUG:boto:Token: None 2022-09-12T20:14:18.201 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:18.202 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E1yt9Lm1ayk0p02Q38InobMoMJk= 2022-09-12T20:14:18.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E1yt9Lm1ayk0p02Q38InobMoMJk=', 'Content-Length': '0'} 2022-09-12T20:14:18.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9342a5e5ac5b80a-00631f931a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.205 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:18.206 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:18.206 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:18.207 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:18.207 DEBUG:boto:Method: HEAD 2022-09-12T20:14:18.208 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:18.208 DEBUG:boto:Data: 2022-09-12T20:14:18.209 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.209 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.210 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.210 DEBUG:boto:Params: {} 2022-09-12T20:14:18.211 DEBUG:boto:Token: None 2022-09-12T20:14:18.212 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:18.212 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E1yt9Lm1ayk0p02Q38InobMoMJk= 2022-09-12T20:14:18.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E1yt9Lm1ayk0p02Q38InobMoMJk=', 'Content-Length': '0'} 2022-09-12T20:14:18.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000094f201cae4e28ef-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:14:18.216 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:18.217 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:18.217 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:18.218 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:18.219 DEBUG:boto:Method: GET 2022-09-12T20:14:18.219 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:18.220 DEBUG:boto:Data: 2022-09-12T20:14:18.220 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.221 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.221 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.222 DEBUG:boto:Params: {} 2022-09-12T20:14:18.223 DEBUG:boto:Token: None 2022-09-12T20:14:18.223 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:18.224 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4M8iq2RwBh9IBo+f61aP4Ck+kSE= 2022-09-12T20:14:18.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4M8iq2RwBh9IBo+f61aP4Ck+kSE=', 'Content-Length': '0'} 2022-09-12T20:14:18.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000958c9ea4afd6afa3-00631f931a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.227 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:18.228 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:18.228 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:18.229 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:18.229 DEBUG:boto:Method: GET 2022-09-12T20:14:18.230 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:18.230 DEBUG:boto:Data: 2022-09-12T20:14:18.231 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.231 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.232 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.232 DEBUG:boto:Params: {} 2022-09-12T20:14:18.233 DEBUG:boto:Token: None 2022-09-12T20:14:18.234 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:18.234 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4M8iq2RwBh9IBo+f61aP4Ck+kSE= 2022-09-12T20:14:18.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4M8iq2RwBh9IBo+f61aP4Ck+kSE=', 'Content-Length': '0'} 2022-09-12T20:14:18.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a32c3d85ce47cda6-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.278 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:18.279 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:18.279 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:14:18.280 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:14:18.280 DEBUG:boto:Method: GET 2022-09-12T20:14:18.281 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:14:18.281 DEBUG:boto:Data: 2022-09-12T20:14:18.282 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.282 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.283 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.283 DEBUG:boto:Params: {} 2022-09-12T20:14:18.284 DEBUG:boto:Token: None 2022-09-12T20:14:18.285 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/ 2022-09-12T20:14:18.285 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s= 2022-09-12T20:14:18.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s=', 'Content-Length': '0'} 2022-09-12T20:14:18.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4641766bceaecf0-00631f931a-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.289 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:18.290 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:18.291 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:18.291 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:14:18.292 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:14:18.292 DEBUG:boto:Method: GET 2022-09-12T20:14:18.293 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:14:18.293 DEBUG:boto:Data: 2022-09-12T20:14:18.294 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.294 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.295 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.295 DEBUG:boto:Params: {} 2022-09-12T20:14:18.296 DEBUG:boto:Token: None 2022-09-12T20:14:18.297 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/ 2022-09-12T20:14:18.297 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s= 2022-09-12T20:14:18.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s=', 'Content-Length': '0'} 2022-09-12T20:14:18.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008fe94aa5faa83a9-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.303 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:18.304 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:18.305 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:18.305 DEBUG:boto:Method: HEAD 2022-09-12T20:14:18.306 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:14:18.306 DEBUG:boto:Data: 2022-09-12T20:14:18.307 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.307 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.308 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.308 DEBUG:boto:Params: {} 2022-09-12T20:14:18.309 DEBUG:boto:Token: None 2022-09-12T20:14:18.309 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj68 2022-09-12T20:14:18.310 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4Zk76BITXZxrxuFh89WyUIG43cE= 2022-09-12T20:14:18.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4Zk76BITXZxrxuFh89WyUIG43cE=', 'Content-Length': '0'} 2022-09-12T20:14:18.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb84e70636a6ef82-00631f931a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.313 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:18.314 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:18.315 DEBUG:boto:Method: HEAD 2022-09-12T20:14:18.315 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:14:18.316 DEBUG:boto:Data: 2022-09-12T20:14:18.316 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.317 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.317 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.318 DEBUG:boto:Params: {} 2022-09-12T20:14:18.318 DEBUG:boto:Token: None 2022-09-12T20:14:18.319 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj68 2022-09-12T20:14:18.319 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4Zk76BITXZxrxuFh89WyUIG43cE= 2022-09-12T20:14:18.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4Zk76BITXZxrxuFh89WyUIG43cE=', 'Content-Length': '0'} 2022-09-12T20:14:18.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000119bd4a154da6ee5-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:14:18.323 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:18.324 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:18.325 DEBUG:boto:Method: GET 2022-09-12T20:14:18.325 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:14:18.326 DEBUG:boto:Data: 2022-09-12T20:14:18.326 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.327 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.327 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.328 DEBUG:boto:Params: {} 2022-09-12T20:14:18.328 DEBUG:boto:Token: None 2022-09-12T20:14:18.329 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj68 2022-09-12T20:14:18.330 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0wxvXO6MEniNTvn7GpVNgJ+c1cM= 2022-09-12T20:14:18.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0wxvXO6MEniNTvn7GpVNgJ+c1cM=', 'Content-Length': '0'} 2022-09-12T20:14:18.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eada7465acb958df-00631f931a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.333 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:18.334 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:18.334 DEBUG:boto:Method: GET 2022-09-12T20:14:18.335 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:14:18.335 DEBUG:boto:Data: 2022-09-12T20:14:18.336 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.336 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.337 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.337 DEBUG:boto:Params: {} 2022-09-12T20:14:18.338 DEBUG:boto:Token: None 2022-09-12T20:14:18.339 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj68 2022-09-12T20:14:18.339 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0wxvXO6MEniNTvn7GpVNgJ+c1cM= 2022-09-12T20:14:18.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0wxvXO6MEniNTvn7GpVNgJ+c1cM=', 'Content-Length': '0'} 2022-09-12T20:14:18.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7f71f4e227808ad-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.382 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:18.383 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:18.384 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:14:18.384 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:14:18.385 DEBUG:boto:Method: GET 2022-09-12T20:14:18.385 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:14:18.386 DEBUG:boto:Data: 2022-09-12T20:14:18.387 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.387 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.388 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.388 DEBUG:boto:Params: {} 2022-09-12T20:14:18.389 DEBUG:boto:Token: None 2022-09-12T20:14:18.389 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/ 2022-09-12T20:14:18.390 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s= 2022-09-12T20:14:18.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s=', 'Content-Length': '0'} 2022-09-12T20:14:18.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb830bf243025fb8-00631f931a-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.394 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:18.395 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:18.396 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:18.397 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:14:18.397 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:14:18.398 DEBUG:boto:Method: GET 2022-09-12T20:14:18.398 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:14:18.399 DEBUG:boto:Data: 2022-09-12T20:14:18.400 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.400 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.401 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.401 DEBUG:boto:Params: {} 2022-09-12T20:14:18.402 DEBUG:boto:Token: None 2022-09-12T20:14:18.402 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/ 2022-09-12T20:14:18.403 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s= 2022-09-12T20:14:18.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s=', 'Content-Length': '0'} 2022-09-12T20:14:18.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a122a6c14f74cac-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.408 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:18.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:14:18.410 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:18.410 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:18.411 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:18.411 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:18.412 DEBUG:boto:Method: GET 2022-09-12T20:14:18.412 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:18.413 DEBUG:boto:Data: 2022-09-12T20:14:18.413 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.414 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.415 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.415 DEBUG:boto:Params: {} 2022-09-12T20:14:18.416 DEBUG:boto:Token: None 2022-09-12T20:14:18.416 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:18.417 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:d8rjYUGlSZsPsHn1LxkQoimvt+Q= 2022-09-12T20:14:18.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:d8rjYUGlSZsPsHn1LxkQoimvt+Q=', 'Content-Length': '0'} 2022-09-12T20:14:18.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2bab6ae15768c67-00631f931a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.421 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:18.421 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:18.422 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:18.422 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:18.423 DEBUG:boto:Method: GET 2022-09-12T20:14:18.424 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:18.424 DEBUG:boto:Data: 2022-09-12T20:14:18.425 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.425 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.426 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.426 DEBUG:boto:Params: {} 2022-09-12T20:14:18.427 DEBUG:boto:Token: None 2022-09-12T20:14:18.427 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:18.428 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:d8rjYUGlSZsPsHn1LxkQoimvt+Q= 2022-09-12T20:14:18.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:d8rjYUGlSZsPsHn1LxkQoimvt+Q=', 'Content-Length': '0'} 2022-09-12T20:14:18.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e6494a06122d15f-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.432 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:18.433 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:18.433 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:18.434 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:18.434 DEBUG:boto:Method: HEAD 2022-09-12T20:14:18.435 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:18.435 DEBUG:boto:Data: 2022-09-12T20:14:18.436 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.436 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.437 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.437 DEBUG:boto:Params: {} 2022-09-12T20:14:18.438 DEBUG:boto:Token: None 2022-09-12T20:14:18.438 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:18.439 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ucF0VQxB9O3MAxIUJsEWwTD58WI= 2022-09-12T20:14:18.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ucF0VQxB9O3MAxIUJsEWwTD58WI=', 'Content-Length': '0'} 2022-09-12T20:14:18.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029636208d1e55db4-00631f931a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.442 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:18.443 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:18.443 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:18.444 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:18.444 DEBUG:boto:Method: HEAD 2022-09-12T20:14:18.445 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:18.445 DEBUG:boto:Data: 2022-09-12T20:14:18.446 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.447 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.447 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.448 DEBUG:boto:Params: {} 2022-09-12T20:14:18.448 DEBUG:boto:Token: None 2022-09-12T20:14:18.449 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:18.449 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ucF0VQxB9O3MAxIUJsEWwTD58WI= 2022-09-12T20:14:18.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ucF0VQxB9O3MAxIUJsEWwTD58WI=', 'Content-Length': '0'} 2022-09-12T20:14:18.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086217d25e8baf41c-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:14:18.454 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:18.454 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:18.455 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:18.455 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:18.456 DEBUG:boto:Method: GET 2022-09-12T20:14:18.456 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:18.457 DEBUG:boto:Data: 2022-09-12T20:14:18.458 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.458 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.459 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.460 DEBUG:boto:Params: {} 2022-09-12T20:14:18.460 DEBUG:boto:Token: None 2022-09-12T20:14:18.461 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:18.461 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:d8rjYUGlSZsPsHn1LxkQoimvt+Q= 2022-09-12T20:14:18.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:d8rjYUGlSZsPsHn1LxkQoimvt+Q=', 'Content-Length': '0'} 2022-09-12T20:14:18.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1636d8d27d09b7e-00631f931a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.465 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:18.466 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:18.466 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:18.467 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:18.467 DEBUG:boto:Method: GET 2022-09-12T20:14:18.468 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:18.468 DEBUG:boto:Data: 2022-09-12T20:14:18.469 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.470 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.470 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.471 DEBUG:boto:Params: {} 2022-09-12T20:14:18.472 DEBUG:boto:Token: None 2022-09-12T20:14:18.472 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:18.473 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:d8rjYUGlSZsPsHn1LxkQoimvt+Q= 2022-09-12T20:14:18.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:d8rjYUGlSZsPsHn1LxkQoimvt+Q=', 'Content-Length': '0'} 2022-09-12T20:14:18.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1a01d474746d08e-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.518 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:18.519 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:18.519 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:14:18.520 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:14:18.520 DEBUG:boto:Method: GET 2022-09-12T20:14:18.521 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:14:18.521 DEBUG:boto:Data: 2022-09-12T20:14:18.522 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.523 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.523 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.524 DEBUG:boto:Params: {} 2022-09-12T20:14:18.524 DEBUG:boto:Token: None 2022-09-12T20:14:18.525 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/ 2022-09-12T20:14:18.525 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s= 2022-09-12T20:14:18.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s=', 'Content-Length': '0'} 2022-09-12T20:14:18.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091dfb5f624c9dc47-00631f931a-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.529 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:18.530 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:18.531 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:18.531 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:14:18.532 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:14:18.532 DEBUG:boto:Method: GET 2022-09-12T20:14:18.533 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:14:18.533 DEBUG:boto:Data: 2022-09-12T20:14:18.534 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.534 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.535 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.535 DEBUG:boto:Params: {} 2022-09-12T20:14:18.536 DEBUG:boto:Token: None 2022-09-12T20:14:18.536 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/ 2022-09-12T20:14:18.537 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s= 2022-09-12T20:14:18.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s=', 'Content-Length': '0'} 2022-09-12T20:14:18.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d3ff5fdaf5e7513-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.543 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:18.543 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:18.544 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:18.544 DEBUG:boto:Method: HEAD 2022-09-12T20:14:18.545 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:14:18.545 DEBUG:boto:Data: 2022-09-12T20:14:18.546 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.546 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.547 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.547 DEBUG:boto:Params: {} 2022-09-12T20:14:18.548 DEBUG:boto:Token: None 2022-09-12T20:14:18.548 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj69 2022-09-12T20:14:18.549 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lL64nbMIC/ACOqrw5yuGhnqbm4s= 2022-09-12T20:14:18.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lL64nbMIC/ACOqrw5yuGhnqbm4s=', 'Content-Length': '0'} 2022-09-12T20:14:18.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b6355f4ffb4fea0-00631f931a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.552 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:18.553 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:18.553 DEBUG:boto:Method: HEAD 2022-09-12T20:14:18.554 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:14:18.554 DEBUG:boto:Data: 2022-09-12T20:14:18.555 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.555 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.556 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.556 DEBUG:boto:Params: {} 2022-09-12T20:14:18.557 DEBUG:boto:Token: None 2022-09-12T20:14:18.558 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj69 2022-09-12T20:14:18.558 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lL64nbMIC/ACOqrw5yuGhnqbm4s= 2022-09-12T20:14:18.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lL64nbMIC/ACOqrw5yuGhnqbm4s=', 'Content-Length': '0'} 2022-09-12T20:14:18.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f88f727d4119d9e-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:14:18.562 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:18.562 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:18.563 DEBUG:boto:Method: GET 2022-09-12T20:14:18.563 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:14:18.564 DEBUG:boto:Data: 2022-09-12T20:14:18.564 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.565 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.565 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.566 DEBUG:boto:Params: {} 2022-09-12T20:14:18.567 DEBUG:boto:Token: None 2022-09-12T20:14:18.567 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj69 2022-09-12T20:14:18.568 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gMvfu62q4saVE6FFyryQCl+bqSY= 2022-09-12T20:14:18.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gMvfu62q4saVE6FFyryQCl+bqSY=', 'Content-Length': '0'} 2022-09-12T20:14:18.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fbe91a971a1a2c5-00631f931a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.571 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:18.572 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:18.572 DEBUG:boto:Method: GET 2022-09-12T20:14:18.573 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:14:18.573 DEBUG:boto:Data: 2022-09-12T20:14:18.574 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.574 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.575 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.575 DEBUG:boto:Params: {} 2022-09-12T20:14:18.576 DEBUG:boto:Token: None 2022-09-12T20:14:18.577 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj69 2022-09-12T20:14:18.577 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gMvfu62q4saVE6FFyryQCl+bqSY= 2022-09-12T20:14:18.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gMvfu62q4saVE6FFyryQCl+bqSY=', 'Content-Length': '0'} 2022-09-12T20:14:18.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005709f802c2cc99be-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.581 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:18.581 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:18.582 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:14:18.582 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:14:18.583 DEBUG:boto:Method: GET 2022-09-12T20:14:18.584 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:14:18.585 DEBUG:boto:Data: 2022-09-12T20:14:18.585 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.586 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.587 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.588 DEBUG:boto:Params: {} 2022-09-12T20:14:18.589 DEBUG:boto:Token: None 2022-09-12T20:14:18.590 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/ 2022-09-12T20:14:18.591 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s= 2022-09-12T20:14:18.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s=', 'Content-Length': '0'} 2022-09-12T20:14:18.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a47a9b6648f8d362-00631f931a-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.596 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:18.597 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:18.598 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:18.599 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:14:18.599 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:14:18.600 DEBUG:boto:Method: GET 2022-09-12T20:14:18.601 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:14:18.602 DEBUG:boto:Data: 2022-09-12T20:14:18.603 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.604 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.604 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.605 DEBUG:boto:Params: {} 2022-09-12T20:14:18.606 DEBUG:boto:Token: None 2022-09-12T20:14:18.607 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/ 2022-09-12T20:14:18.608 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s= 2022-09-12T20:14:18.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s=', 'Content-Length': '0'} 2022-09-12T20:14:18.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5de8051edfdd571-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.615 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:18.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:14:18.617 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:18.618 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:18.619 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:18.620 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:18.620 DEBUG:boto:Method: GET 2022-09-12T20:14:18.621 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:18.622 DEBUG:boto:Data: 2022-09-12T20:14:18.623 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.624 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.625 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.626 DEBUG:boto:Params: {} 2022-09-12T20:14:18.627 DEBUG:boto:Token: None 2022-09-12T20:14:18.627 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:18.628 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Gmua9XIgP4AIod6dL3zAQrAsY4M= 2022-09-12T20:14:18.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Gmua9XIgP4AIod6dL3zAQrAsY4M=', 'Content-Length': '0'} 2022-09-12T20:14:18.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000674a5a8a67d4bdba-00631f931a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.674 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:18.675 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:18.676 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:18.677 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:18.678 DEBUG:boto:Method: GET 2022-09-12T20:14:18.679 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:18.680 DEBUG:boto:Data: 2022-09-12T20:14:18.681 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.682 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.683 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.684 DEBUG:boto:Params: {} 2022-09-12T20:14:18.685 DEBUG:boto:Token: None 2022-09-12T20:14:18.686 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:18.687 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Gmua9XIgP4AIod6dL3zAQrAsY4M= 2022-09-12T20:14:18.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Gmua9XIgP4AIod6dL3zAQrAsY4M=', 'Content-Length': '0'} 2022-09-12T20:14:18.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dce21cf70e32791a-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.692 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:18.693 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:18.694 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:18.695 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:18.696 DEBUG:boto:Method: HEAD 2022-09-12T20:14:18.697 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:18.697 DEBUG:boto:Data: 2022-09-12T20:14:18.698 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.699 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.700 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.700 DEBUG:boto:Params: {} 2022-09-12T20:14:18.701 DEBUG:boto:Token: None 2022-09-12T20:14:18.702 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:18.703 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y0ic8vI3uiV7Ddxu5ihbNcvRP68= 2022-09-12T20:14:18.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y0ic8vI3uiV7Ddxu5ihbNcvRP68=', 'Content-Length': '0'} 2022-09-12T20:14:18.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5de04c22e9faf76-00631f931a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.708 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:18.709 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:18.710 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:18.710 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:18.711 DEBUG:boto:Method: HEAD 2022-09-12T20:14:18.712 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:18.713 DEBUG:boto:Data: 2022-09-12T20:14:18.714 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.714 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.715 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.716 DEBUG:boto:Params: {} 2022-09-12T20:14:18.717 DEBUG:boto:Token: None 2022-09-12T20:14:18.718 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:18.719 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y0ic8vI3uiV7Ddxu5ihbNcvRP68= 2022-09-12T20:14:18.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y0ic8vI3uiV7Ddxu5ihbNcvRP68=', 'Content-Length': '0'} 2022-09-12T20:14:18.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000482549f0ef9f0343-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:14:18.724 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:18.725 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:18.726 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:18.727 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:18.728 DEBUG:boto:Method: GET 2022-09-12T20:14:18.729 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:18.729 DEBUG:boto:Data: 2022-09-12T20:14:18.730 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.731 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.732 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.733 DEBUG:boto:Params: {} 2022-09-12T20:14:18.734 DEBUG:boto:Token: None 2022-09-12T20:14:18.735 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:18.736 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Gmua9XIgP4AIod6dL3zAQrAsY4M= 2022-09-12T20:14:18.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Gmua9XIgP4AIod6dL3zAQrAsY4M=', 'Content-Length': '0'} 2022-09-12T20:14:18.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d913007e78c24f37-00631f931a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.739 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:18.740 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:18.740 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:18.741 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:18.742 DEBUG:boto:Method: GET 2022-09-12T20:14:18.742 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:18.743 DEBUG:boto:Data: 2022-09-12T20:14:18.743 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.744 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.745 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.745 DEBUG:boto:Params: {} 2022-09-12T20:14:18.746 DEBUG:boto:Token: None 2022-09-12T20:14:18.746 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:18.747 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Gmua9XIgP4AIod6dL3zAQrAsY4M= 2022-09-12T20:14:18.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Gmua9XIgP4AIod6dL3zAQrAsY4M=', 'Content-Length': '0'} 2022-09-12T20:14:18.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2c3286cca1cdf2f-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.794 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:18.795 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:18.795 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:14:18.796 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:14:18.796 DEBUG:boto:Method: GET 2022-09-12T20:14:18.797 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:14:18.798 DEBUG:boto:Data: 2022-09-12T20:14:18.798 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.799 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.799 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.800 DEBUG:boto:Params: {} 2022-09-12T20:14:18.800 DEBUG:boto:Token: None 2022-09-12T20:14:18.801 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/ 2022-09-12T20:14:18.802 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s= 2022-09-12T20:14:18.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s=', 'Content-Length': '0'} 2022-09-12T20:14:18.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dccc6ea36fb918bb-00631f931a-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.806 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:18.807 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:18.808 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:18.808 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:14:18.809 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:14:18.810 DEBUG:boto:Method: GET 2022-09-12T20:14:18.810 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:14:18.811 DEBUG:boto:Data: 2022-09-12T20:14:18.812 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.812 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.813 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.814 DEBUG:boto:Params: {} 2022-09-12T20:14:18.814 DEBUG:boto:Token: None 2022-09-12T20:14:18.815 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/ 2022-09-12T20:14:18.816 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s= 2022-09-12T20:14:18.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s=', 'Content-Length': '0'} 2022-09-12T20:14:18.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e539c531529ed06c-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.822 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:18.823 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:18.824 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:18.824 DEBUG:boto:Method: HEAD 2022-09-12T20:14:18.825 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:14:18.825 DEBUG:boto:Data: 2022-09-12T20:14:18.826 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.826 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.827 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.828 DEBUG:boto:Params: {} 2022-09-12T20:14:18.828 DEBUG:boto:Token: None 2022-09-12T20:14:18.829 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj76 2022-09-12T20:14:18.830 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zelH9Xq/7mdW+HJn3n7KinYDX8I= 2022-09-12T20:14:18.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zelH9Xq/7mdW+HJn3n7KinYDX8I=', 'Content-Length': '0'} 2022-09-12T20:14:18.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c79ed631fb8e81f-00631f931a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.833 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:18.834 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:18.835 DEBUG:boto:Method: HEAD 2022-09-12T20:14:18.835 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:14:18.836 DEBUG:boto:Data: 2022-09-12T20:14:18.837 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.838 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.838 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.839 DEBUG:boto:Params: {} 2022-09-12T20:14:18.840 DEBUG:boto:Token: None 2022-09-12T20:14:18.841 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj76 2022-09-12T20:14:18.841 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zelH9Xq/7mdW+HJn3n7KinYDX8I= 2022-09-12T20:14:18.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zelH9Xq/7mdW+HJn3n7KinYDX8I=', 'Content-Length': '0'} 2022-09-12T20:14:18.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad646f4003f8a78b-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:14:18.846 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:18.847 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:18.847 DEBUG:boto:Method: GET 2022-09-12T20:14:18.848 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:14:18.849 DEBUG:boto:Data: 2022-09-12T20:14:18.849 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.850 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.851 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.851 DEBUG:boto:Params: {} 2022-09-12T20:14:18.852 DEBUG:boto:Token: None 2022-09-12T20:14:18.852 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj76 2022-09-12T20:14:18.853 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZVljvokbNynOsZz60yKtNrP2PoE= 2022-09-12T20:14:18.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZVljvokbNynOsZz60yKtNrP2PoE=', 'Content-Length': '0'} 2022-09-12T20:14:18.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000883b39db2fbc2195-00631f931a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.857 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:18.858 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:18.858 DEBUG:boto:Method: GET 2022-09-12T20:14:18.859 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:14:18.859 DEBUG:boto:Data: 2022-09-12T20:14:18.860 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.861 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.861 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.862 DEBUG:boto:Params: {} 2022-09-12T20:14:18.862 DEBUG:boto:Token: None 2022-09-12T20:14:18.863 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj76 2022-09-12T20:14:18.864 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZVljvokbNynOsZz60yKtNrP2PoE= 2022-09-12T20:14:18.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZVljvokbNynOsZz60yKtNrP2PoE=', 'Content-Length': '0'} 2022-09-12T20:14:18.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1a92f904e4122a2-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.868 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:18.868 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:18.869 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:14:18.870 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:14:18.870 DEBUG:boto:Method: GET 2022-09-12T20:14:18.871 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:14:18.872 DEBUG:boto:Data: 2022-09-12T20:14:18.872 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.873 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.874 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.874 DEBUG:boto:Params: {} 2022-09-12T20:14:18.875 DEBUG:boto:Token: None 2022-09-12T20:14:18.875 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/ 2022-09-12T20:14:18.876 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s= 2022-09-12T20:14:18.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s=', 'Content-Length': '0'} 2022-09-12T20:14:18.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004838da4ef826d883-00631f931a-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.880 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:18.881 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:18.882 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:18.882 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:14:18.883 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:14:18.883 DEBUG:boto:Method: GET 2022-09-12T20:14:18.884 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:14:18.885 DEBUG:boto:Data: 2022-09-12T20:14:18.885 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.886 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.886 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.887 DEBUG:boto:Params: {} 2022-09-12T20:14:18.888 DEBUG:boto:Token: None 2022-09-12T20:14:18.888 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/ 2022-09-12T20:14:18.889 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s= 2022-09-12T20:14:18.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A6tOTpPRsufo2l3+VOH6OMm8Y1s=', 'Content-Length': '0'} 2022-09-12T20:14:18.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f015f2bda192ebad-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.896 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:18.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:14:18.898 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:18.898 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:18.899 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:18.900 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:18.901 DEBUG:boto:Method: GET 2022-09-12T20:14:18.902 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:18.902 DEBUG:boto:Data: 2022-09-12T20:14:18.903 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.904 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.905 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.906 DEBUG:boto:Params: {} 2022-09-12T20:14:18.907 DEBUG:boto:Token: None 2022-09-12T20:14:18.907 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:18.908 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eAt73EX1h0vOZSD8w4rh1A13OR0= 2022-09-12T20:14:18.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eAt73EX1h0vOZSD8w4rh1A13OR0=', 'Content-Length': '0'} 2022-09-12T20:14:18.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb20f845d697598d-00631f931a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.954 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:18.955 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:18.956 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:18.957 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:18.957 DEBUG:boto:Method: GET 2022-09-12T20:14:18.958 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:18.959 DEBUG:boto:Data: 2022-09-12T20:14:18.960 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.961 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.962 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.962 DEBUG:boto:Params: {} 2022-09-12T20:14:18.963 DEBUG:boto:Token: None 2022-09-12T20:14:18.964 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:18.965 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eAt73EX1h0vOZSD8w4rh1A13OR0= 2022-09-12T20:14:18.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eAt73EX1h0vOZSD8w4rh1A13OR0=', 'Content-Length': '0'} 2022-09-12T20:14:18.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000213f1ec268fccf79-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.971 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:18.971 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:18.972 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:18.973 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:18.973 DEBUG:boto:Method: HEAD 2022-09-12T20:14:18.974 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:18.975 DEBUG:boto:Data: 2022-09-12T20:14:18.975 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.976 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:18.976 DEBUG:boto:Port: 8001 2022-09-12T20:14:18.977 DEBUG:boto:Params: {} 2022-09-12T20:14:18.978 DEBUG:boto:Token: None 2022-09-12T20:14:18.978 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:18.979 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wgRFMnsWLUv7zaIi7eJWHnEuBJw= 2022-09-12T20:14:18.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wgRFMnsWLUv7zaIi7eJWHnEuBJw=', 'Content-Length': '0'} 2022-09-12T20:14:18.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb9b9a545af7ae25-00631f931a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.983 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:18.983 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:18.984 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:18.985 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:18.985 DEBUG:boto:Method: HEAD 2022-09-12T20:14:18.986 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:18.987 DEBUG:boto:Data: 2022-09-12T20:14:18.987 DEBUG:boto:Headers: {} 2022-09-12T20:14:18.988 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:18.989 DEBUG:boto:Port: 8000 2022-09-12T20:14:18.989 DEBUG:boto:Params: {} 2022-09-12T20:14:18.990 DEBUG:boto:Token: None 2022-09-12T20:14:18.991 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:18 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:18.991 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wgRFMnsWLUv7zaIi7eJWHnEuBJw= 2022-09-12T20:14:18.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wgRFMnsWLUv7zaIi7eJWHnEuBJw=', 'Content-Length': '0'} 2022-09-12T20:14:18.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000352e99f1b8400fcd-00631f931a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:18.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:14:18.995 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:18.996 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:18.996 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:18.997 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:18.998 DEBUG:boto:Method: GET 2022-09-12T20:14:18.998 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:18.999 DEBUG:boto:Data: 2022-09-12T20:14:18.999 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.000 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.000 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.001 DEBUG:boto:Params: {} 2022-09-12T20:14:19.001 DEBUG:boto:Token: None 2022-09-12T20:14:19.002 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:19.003 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iDZ/CmtjFGbvizSyVPUo0gzd7XM= 2022-09-12T20:14:19.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iDZ/CmtjFGbvizSyVPUo0gzd7XM=', 'Content-Length': '0'} 2022-09-12T20:14:19.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1b15cbcffbc38ad-00631f931b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.006 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:19.007 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:19.007 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:19.008 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:19.008 DEBUG:boto:Method: GET 2022-09-12T20:14:19.009 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:19.009 DEBUG:boto:Data: 2022-09-12T20:14:19.010 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.010 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.011 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.011 DEBUG:boto:Params: {} 2022-09-12T20:14:19.012 DEBUG:boto:Token: None 2022-09-12T20:14:19.013 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:19.013 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iDZ/CmtjFGbvizSyVPUo0gzd7XM= 2022-09-12T20:14:19.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iDZ/CmtjFGbvizSyVPUo0gzd7XM=', 'Content-Length': '0'} 2022-09-12T20:14:19.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002321abcd62a63796-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.058 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:19.059 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:19.059 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:14:19.060 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:14:19.060 DEBUG:boto:Method: GET 2022-09-12T20:14:19.061 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:14:19.061 DEBUG:boto:Data: 2022-09-12T20:14:19.062 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.062 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.063 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.063 DEBUG:boto:Params: {} 2022-09-12T20:14:19.064 DEBUG:boto:Token: None 2022-09-12T20:14:19.065 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/ 2022-09-12T20:14:19.065 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk= 2022-09-12T20:14:19.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk=', 'Content-Length': '0'} 2022-09-12T20:14:19.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cb910d0011bb86c-00631f931b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.069 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:19.070 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:19.071 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:19.071 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:14:19.072 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:14:19.072 DEBUG:boto:Method: GET 2022-09-12T20:14:19.073 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:14:19.073 DEBUG:boto:Data: 2022-09-12T20:14:19.074 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.074 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.075 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.075 DEBUG:boto:Params: {} 2022-09-12T20:14:19.076 DEBUG:boto:Token: None 2022-09-12T20:14:19.077 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/ 2022-09-12T20:14:19.077 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk= 2022-09-12T20:14:19.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk=', 'Content-Length': '0'} 2022-09-12T20:14:19.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000accaaf83aa69c4f0-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.083 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:19.084 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:19.084 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:19.085 DEBUG:boto:Method: HEAD 2022-09-12T20:14:19.085 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:14:19.086 DEBUG:boto:Data: 2022-09-12T20:14:19.086 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.087 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.087 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.088 DEBUG:boto:Params: {} 2022-09-12T20:14:19.088 DEBUG:boto:Token: None 2022-09-12T20:14:19.089 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj85 2022-09-12T20:14:19.090 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DrwE92Xp747EEyKQiIHp5aSlPQc= 2022-09-12T20:14:19.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DrwE92Xp747EEyKQiIHp5aSlPQc=', 'Content-Length': '0'} 2022-09-12T20:14:19.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b3453fcc41b2c19-00631f931b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.093 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:19.094 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:19.094 DEBUG:boto:Method: HEAD 2022-09-12T20:14:19.095 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:14:19.095 DEBUG:boto:Data: 2022-09-12T20:14:19.096 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.096 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.097 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.097 DEBUG:boto:Params: {} 2022-09-12T20:14:19.098 DEBUG:boto:Token: None 2022-09-12T20:14:19.098 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj85 2022-09-12T20:14:19.099 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DrwE92Xp747EEyKQiIHp5aSlPQc= 2022-09-12T20:14:19.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DrwE92Xp747EEyKQiIHp5aSlPQc=', 'Content-Length': '0'} 2022-09-12T20:14:19.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008db07b86443ba39c-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:14:19.103 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:19.103 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:19.104 DEBUG:boto:Method: GET 2022-09-12T20:14:19.104 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:14:19.105 DEBUG:boto:Data: 2022-09-12T20:14:19.105 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.106 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.107 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.107 DEBUG:boto:Params: {} 2022-09-12T20:14:19.108 DEBUG:boto:Token: None 2022-09-12T20:14:19.108 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj85 2022-09-12T20:14:19.109 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aLdtyoYWUHibwqE8MQknvm13HEc= 2022-09-12T20:14:19.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aLdtyoYWUHibwqE8MQknvm13HEc=', 'Content-Length': '0'} 2022-09-12T20:14:19.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4dec2dc061e526e-00631f931b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.112 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:19.113 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:19.113 DEBUG:boto:Method: GET 2022-09-12T20:14:19.114 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:14:19.114 DEBUG:boto:Data: 2022-09-12T20:14:19.115 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.116 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.116 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.117 DEBUG:boto:Params: {} 2022-09-12T20:14:19.117 DEBUG:boto:Token: None 2022-09-12T20:14:19.118 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj85 2022-09-12T20:14:19.118 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aLdtyoYWUHibwqE8MQknvm13HEc= 2022-09-12T20:14:19.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aLdtyoYWUHibwqE8MQknvm13HEc=', 'Content-Length': '0'} 2022-09-12T20:14:19.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f78e764775dbfab-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.122 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:19.122 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:19.123 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:14:19.123 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:14:19.124 DEBUG:boto:Method: GET 2022-09-12T20:14:19.124 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:14:19.125 DEBUG:boto:Data: 2022-09-12T20:14:19.125 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.126 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.126 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.127 DEBUG:boto:Params: {} 2022-09-12T20:14:19.127 DEBUG:boto:Token: None 2022-09-12T20:14:19.128 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/ 2022-09-12T20:14:19.129 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk= 2022-09-12T20:14:19.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk=', 'Content-Length': '0'} 2022-09-12T20:14:19.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010b9570fad206bc8-00631f931b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.133 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:19.133 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:19.134 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:19.135 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:14:19.135 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:14:19.136 DEBUG:boto:Method: GET 2022-09-12T20:14:19.136 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:14:19.137 DEBUG:boto:Data: 2022-09-12T20:14:19.137 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.138 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.138 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.139 DEBUG:boto:Params: {} 2022-09-12T20:14:19.140 DEBUG:boto:Token: None 2022-09-12T20:14:19.140 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/ 2022-09-12T20:14:19.141 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk= 2022-09-12T20:14:19.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk=', 'Content-Length': '0'} 2022-09-12T20:14:19.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0c8421553b7815c-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.146 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:19.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:14:19.147 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:19.148 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:19.149 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:19.149 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:19.150 DEBUG:boto:Method: GET 2022-09-12T20:14:19.150 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:19.151 DEBUG:boto:Data: 2022-09-12T20:14:19.151 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.152 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.152 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.153 DEBUG:boto:Params: {} 2022-09-12T20:14:19.153 DEBUG:boto:Token: None 2022-09-12T20:14:19.154 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:19.154 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A5UDPh2lXxZrWbk0Vzy1Gqeikkg= 2022-09-12T20:14:19.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A5UDPh2lXxZrWbk0Vzy1Gqeikkg=', 'Content-Length': '0'} 2022-09-12T20:14:19.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f3b039a39e88270-00631f931b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.158 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:19.159 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:19.159 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:19.160 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:19.160 DEBUG:boto:Method: GET 2022-09-12T20:14:19.161 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:19.161 DEBUG:boto:Data: 2022-09-12T20:14:19.162 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.162 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.163 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.163 DEBUG:boto:Params: {} 2022-09-12T20:14:19.164 DEBUG:boto:Token: None 2022-09-12T20:14:19.164 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:19.165 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A5UDPh2lXxZrWbk0Vzy1Gqeikkg= 2022-09-12T20:14:19.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A5UDPh2lXxZrWbk0Vzy1Gqeikkg=', 'Content-Length': '0'} 2022-09-12T20:14:19.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039a28eced811f340-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.169 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:19.169 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:19.170 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:19.170 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:19.171 DEBUG:boto:Method: HEAD 2022-09-12T20:14:19.171 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:19.172 DEBUG:boto:Data: 2022-09-12T20:14:19.172 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.173 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.173 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.174 DEBUG:boto:Params: {} 2022-09-12T20:14:19.174 DEBUG:boto:Token: None 2022-09-12T20:14:19.175 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:19.175 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:78Wl/Y2p9oE22z/FE7W18uO6nvA= 2022-09-12T20:14:19.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:78Wl/Y2p9oE22z/FE7W18uO6nvA=', 'Content-Length': '0'} 2022-09-12T20:14:19.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb6f724dd5b855b5-00631f931b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.178 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:19.179 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:19.179 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:19.180 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:19.180 DEBUG:boto:Method: HEAD 2022-09-12T20:14:19.181 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:19.181 DEBUG:boto:Data: 2022-09-12T20:14:19.182 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.182 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.183 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.184 DEBUG:boto:Params: {} 2022-09-12T20:14:19.185 DEBUG:boto:Token: None 2022-09-12T20:14:19.185 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:19.186 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:78Wl/Y2p9oE22z/FE7W18uO6nvA= 2022-09-12T20:14:19.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:78Wl/Y2p9oE22z/FE7W18uO6nvA=', 'Content-Length': '0'} 2022-09-12T20:14:19.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9ebef77f9c06c95-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:14:19.190 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:19.190 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:19.191 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:19.191 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:19.192 DEBUG:boto:Method: GET 2022-09-12T20:14:19.192 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:19.193 DEBUG:boto:Data: 2022-09-12T20:14:19.193 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.194 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.194 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.195 DEBUG:boto:Params: {} 2022-09-12T20:14:19.196 DEBUG:boto:Token: None 2022-09-12T20:14:19.196 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:19.197 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A5UDPh2lXxZrWbk0Vzy1Gqeikkg= 2022-09-12T20:14:19.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A5UDPh2lXxZrWbk0Vzy1Gqeikkg=', 'Content-Length': '0'} 2022-09-12T20:14:19.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e672790d3126cd0-00631f931b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.200 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:19.201 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:19.201 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:19.202 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:19.202 DEBUG:boto:Method: GET 2022-09-12T20:14:19.203 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:19.203 DEBUG:boto:Data: 2022-09-12T20:14:19.204 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.204 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.205 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.205 DEBUG:boto:Params: {} 2022-09-12T20:14:19.206 DEBUG:boto:Token: None 2022-09-12T20:14:19.206 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:19.207 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A5UDPh2lXxZrWbk0Vzy1Gqeikkg= 2022-09-12T20:14:19.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A5UDPh2lXxZrWbk0Vzy1Gqeikkg=', 'Content-Length': '0'} 2022-09-12T20:14:19.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066ed2f4b9fc50024-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.210 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:19.211 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:19.212 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:14:19.212 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:14:19.213 DEBUG:boto:Method: GET 2022-09-12T20:14:19.213 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:14:19.214 DEBUG:boto:Data: 2022-09-12T20:14:19.214 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.215 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.215 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.216 DEBUG:boto:Params: {} 2022-09-12T20:14:19.216 DEBUG:boto:Token: None 2022-09-12T20:14:19.217 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/ 2022-09-12T20:14:19.217 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk= 2022-09-12T20:14:19.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk=', 'Content-Length': '0'} 2022-09-12T20:14:19.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e13c4bbaf6320c9-00631f931b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.221 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:19.222 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:19.223 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:19.223 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:14:19.224 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:14:19.224 DEBUG:boto:Method: GET 2022-09-12T20:14:19.225 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:14:19.225 DEBUG:boto:Data: 2022-09-12T20:14:19.226 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.226 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.227 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.227 DEBUG:boto:Params: {} 2022-09-12T20:14:19.228 DEBUG:boto:Token: None 2022-09-12T20:14:19.229 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/ 2022-09-12T20:14:19.229 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk= 2022-09-12T20:14:19.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk=', 'Content-Length': '0'} 2022-09-12T20:14:19.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000800cf6c62608f61f-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.278 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:19.279 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:19.280 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:19.280 DEBUG:boto:Method: HEAD 2022-09-12T20:14:19.281 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:14:19.281 DEBUG:boto:Data: 2022-09-12T20:14:19.282 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.282 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.283 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.283 DEBUG:boto:Params: {} 2022-09-12T20:14:19.284 DEBUG:boto:Token: None 2022-09-12T20:14:19.284 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj87 2022-09-12T20:14:19.285 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7xbCuVTZweK0UeAWjTxlUqB82cU= 2022-09-12T20:14:19.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7xbCuVTZweK0UeAWjTxlUqB82cU=', 'Content-Length': '0'} 2022-09-12T20:14:19.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065379b223676ede4-00631f931b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.288 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:19.289 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:19.289 DEBUG:boto:Method: HEAD 2022-09-12T20:14:19.290 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:14:19.290 DEBUG:boto:Data: 2022-09-12T20:14:19.291 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.291 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.292 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.292 DEBUG:boto:Params: {} 2022-09-12T20:14:19.293 DEBUG:boto:Token: None 2022-09-12T20:14:19.293 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj87 2022-09-12T20:14:19.294 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7xbCuVTZweK0UeAWjTxlUqB82cU= 2022-09-12T20:14:19.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7xbCuVTZweK0UeAWjTxlUqB82cU=', 'Content-Length': '0'} 2022-09-12T20:14:19.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055714fb12e8a3c0c-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:14:19.298 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:19.298 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:19.299 DEBUG:boto:Method: GET 2022-09-12T20:14:19.300 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:14:19.300 DEBUG:boto:Data: 2022-09-12T20:14:19.301 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.301 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.302 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.302 DEBUG:boto:Params: {} 2022-09-12T20:14:19.303 DEBUG:boto:Token: None 2022-09-12T20:14:19.303 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj87 2022-09-12T20:14:19.304 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t5Z79K9f3M7sNjlFE5jUs96Yew4= 2022-09-12T20:14:19.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t5Z79K9f3M7sNjlFE5jUs96Yew4=', 'Content-Length': '0'} 2022-09-12T20:14:19.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045c74202a261aa64-00631f931b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.307 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:19.308 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:19.308 DEBUG:boto:Method: GET 2022-09-12T20:14:19.309 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:14:19.309 DEBUG:boto:Data: 2022-09-12T20:14:19.310 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.310 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.311 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.311 DEBUG:boto:Params: {} 2022-09-12T20:14:19.312 DEBUG:boto:Token: None 2022-09-12T20:14:19.313 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj87 2022-09-12T20:14:19.313 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t5Z79K9f3M7sNjlFE5jUs96Yew4= 2022-09-12T20:14:19.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t5Z79K9f3M7sNjlFE5jUs96Yew4=', 'Content-Length': '0'} 2022-09-12T20:14:19.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061b9701823f85b6b-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.317 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:19.317 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:19.318 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:14:19.318 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:14:19.319 DEBUG:boto:Method: GET 2022-09-12T20:14:19.319 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:14:19.320 DEBUG:boto:Data: 2022-09-12T20:14:19.320 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.321 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.321 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.322 DEBUG:boto:Params: {} 2022-09-12T20:14:19.323 DEBUG:boto:Token: None 2022-09-12T20:14:19.323 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/ 2022-09-12T20:14:19.324 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk= 2022-09-12T20:14:19.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk=', 'Content-Length': '0'} 2022-09-12T20:14:19.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a89cedd30b38e96-00631f931b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.370 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:19.371 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:19.371 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:19.372 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:14:19.372 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:14:19.373 DEBUG:boto:Method: GET 2022-09-12T20:14:19.374 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:14:19.374 DEBUG:boto:Data: 2022-09-12T20:14:19.375 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.375 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.376 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.376 DEBUG:boto:Params: {} 2022-09-12T20:14:19.377 DEBUG:boto:Token: None 2022-09-12T20:14:19.377 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/ 2022-09-12T20:14:19.378 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk= 2022-09-12T20:14:19.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk=', 'Content-Length': '0'} 2022-09-12T20:14:19.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0811a2915b853ae-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.384 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:19.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:14:19.385 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:19.386 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:19.386 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:19.387 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:19.387 DEBUG:boto:Method: GET 2022-09-12T20:14:19.388 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:19.388 DEBUG:boto:Data: 2022-09-12T20:14:19.389 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.389 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.390 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.390 DEBUG:boto:Params: {} 2022-09-12T20:14:19.391 DEBUG:boto:Token: None 2022-09-12T20:14:19.392 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:19.392 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9niGmuvZ93d/J7Lgu0zX56AvFhU= 2022-09-12T20:14:19.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9niGmuvZ93d/J7Lgu0zX56AvFhU=', 'Content-Length': '0'} 2022-09-12T20:14:19.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003ac62b7c658b4f5-00631f931b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.396 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:19.396 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:19.397 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:19.397 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:19.398 DEBUG:boto:Method: GET 2022-09-12T20:14:19.399 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:19.399 DEBUG:boto:Data: 2022-09-12T20:14:19.400 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.400 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.401 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.401 DEBUG:boto:Params: {} 2022-09-12T20:14:19.402 DEBUG:boto:Token: None 2022-09-12T20:14:19.402 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:19.403 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9niGmuvZ93d/J7Lgu0zX56AvFhU= 2022-09-12T20:14:19.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9niGmuvZ93d/J7Lgu0zX56AvFhU=', 'Content-Length': '0'} 2022-09-12T20:14:19.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e539174079709ef-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.407 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:19.407 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:19.408 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:19.408 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:19.409 DEBUG:boto:Method: HEAD 2022-09-12T20:14:19.409 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:19.410 DEBUG:boto:Data: 2022-09-12T20:14:19.410 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.411 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.411 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.412 DEBUG:boto:Params: {} 2022-09-12T20:14:19.412 DEBUG:boto:Token: None 2022-09-12T20:14:19.413 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:19.414 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:62x5dT7mY9SyrVUL//xjtPOen24= 2022-09-12T20:14:19.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:62x5dT7mY9SyrVUL//xjtPOen24=', 'Content-Length': '0'} 2022-09-12T20:14:19.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000342fac32fae69891-00631f931b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.417 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:19.417 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:19.418 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:19.418 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:19.419 DEBUG:boto:Method: HEAD 2022-09-12T20:14:19.419 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:19.420 DEBUG:boto:Data: 2022-09-12T20:14:19.420 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.421 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.421 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.422 DEBUG:boto:Params: {} 2022-09-12T20:14:19.422 DEBUG:boto:Token: None 2022-09-12T20:14:19.423 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:19.424 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:62x5dT7mY9SyrVUL//xjtPOen24= 2022-09-12T20:14:19.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:62x5dT7mY9SyrVUL//xjtPOen24=', 'Content-Length': '0'} 2022-09-12T20:14:19.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045898655797fe323-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:14:19.428 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:19.428 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:19.429 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:19.429 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:19.430 DEBUG:boto:Method: GET 2022-09-12T20:14:19.430 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:19.431 DEBUG:boto:Data: 2022-09-12T20:14:19.431 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.432 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.433 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.433 DEBUG:boto:Params: {} 2022-09-12T20:14:19.434 DEBUG:boto:Token: None 2022-09-12T20:14:19.434 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:19.435 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9niGmuvZ93d/J7Lgu0zX56AvFhU= 2022-09-12T20:14:19.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9niGmuvZ93d/J7Lgu0zX56AvFhU=', 'Content-Length': '0'} 2022-09-12T20:14:19.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7705a0f24931265-00631f931b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.439 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:19.439 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:19.440 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:19.440 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:19.441 DEBUG:boto:Method: GET 2022-09-12T20:14:19.441 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:19.442 DEBUG:boto:Data: 2022-09-12T20:14:19.442 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.443 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.443 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.444 DEBUG:boto:Params: {} 2022-09-12T20:14:19.445 DEBUG:boto:Token: None 2022-09-12T20:14:19.445 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:19.446 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9niGmuvZ93d/J7Lgu0zX56AvFhU= 2022-09-12T20:14:19.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9niGmuvZ93d/J7Lgu0zX56AvFhU=', 'Content-Length': '0'} 2022-09-12T20:14:19.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023b82ab91119ac69-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.490 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:19.491 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:19.491 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:14:19.492 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:14:19.492 DEBUG:boto:Method: GET 2022-09-12T20:14:19.493 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:14:19.493 DEBUG:boto:Data: 2022-09-12T20:14:19.494 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.494 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.495 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.495 DEBUG:boto:Params: {} 2022-09-12T20:14:19.496 DEBUG:boto:Token: None 2022-09-12T20:14:19.497 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/ 2022-09-12T20:14:19.497 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk= 2022-09-12T20:14:19.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk=', 'Content-Length': '0'} 2022-09-12T20:14:19.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f62702fd9dd26466-00631f931b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.501 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:19.502 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:19.503 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:19.503 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:14:19.504 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:14:19.505 DEBUG:boto:Method: GET 2022-09-12T20:14:19.505 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:14:19.506 DEBUG:boto:Data: 2022-09-12T20:14:19.506 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.507 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.507 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.508 DEBUG:boto:Params: {} 2022-09-12T20:14:19.508 DEBUG:boto:Token: None 2022-09-12T20:14:19.509 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/ 2022-09-12T20:14:19.510 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk= 2022-09-12T20:14:19.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk=', 'Content-Length': '0'} 2022-09-12T20:14:19.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6b4093db0c8d2a4-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.515 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:19.516 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:19.516 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:19.517 DEBUG:boto:Method: HEAD 2022-09-12T20:14:19.517 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:14:19.518 DEBUG:boto:Data: 2022-09-12T20:14:19.519 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.519 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.520 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.520 DEBUG:boto:Params: {} 2022-09-12T20:14:19.521 DEBUG:boto:Token: None 2022-09-12T20:14:19.521 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj92 2022-09-12T20:14:19.522 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UneiOrFivGzVWTSwfdgwjMjFaeQ= 2022-09-12T20:14:19.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UneiOrFivGzVWTSwfdgwjMjFaeQ=', 'Content-Length': '0'} 2022-09-12T20:14:19.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b02d2f160c44451-00631f931b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.526 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:19.526 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:19.527 DEBUG:boto:Method: HEAD 2022-09-12T20:14:19.527 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:14:19.528 DEBUG:boto:Data: 2022-09-12T20:14:19.528 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.529 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.529 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.530 DEBUG:boto:Params: {} 2022-09-12T20:14:19.530 DEBUG:boto:Token: None 2022-09-12T20:14:19.531 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj92 2022-09-12T20:14:19.532 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UneiOrFivGzVWTSwfdgwjMjFaeQ= 2022-09-12T20:14:19.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UneiOrFivGzVWTSwfdgwjMjFaeQ=', 'Content-Length': '0'} 2022-09-12T20:14:19.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054bf9f5f8df34da1-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:14:19.536 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:19.536 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:19.537 DEBUG:boto:Method: GET 2022-09-12T20:14:19.537 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:14:19.538 DEBUG:boto:Data: 2022-09-12T20:14:19.538 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.539 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.539 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.540 DEBUG:boto:Params: {} 2022-09-12T20:14:19.541 DEBUG:boto:Token: None 2022-09-12T20:14:19.541 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj92 2022-09-12T20:14:19.542 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZLd6xs7+psS0r0Oy+lTC6x7muLc= 2022-09-12T20:14:19.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZLd6xs7+psS0r0Oy+lTC6x7muLc=', 'Content-Length': '0'} 2022-09-12T20:14:19.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008780d3cf97a2bf03-00631f931b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.546 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:19.546 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:19.547 DEBUG:boto:Method: GET 2022-09-12T20:14:19.548 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:14:19.548 DEBUG:boto:Data: 2022-09-12T20:14:19.549 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.549 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.550 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.550 DEBUG:boto:Params: {} 2022-09-12T20:14:19.551 DEBUG:boto:Token: None 2022-09-12T20:14:19.552 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj92 2022-09-12T20:14:19.552 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZLd6xs7+psS0r0Oy+lTC6x7muLc= 2022-09-12T20:14:19.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZLd6xs7+psS0r0Oy+lTC6x7muLc=', 'Content-Length': '0'} 2022-09-12T20:14:19.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b513e648869a7162-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.598 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:19.599 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:19.599 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:14:19.600 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:14:19.600 DEBUG:boto:Method: GET 2022-09-12T20:14:19.601 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:14:19.602 DEBUG:boto:Data: 2022-09-12T20:14:19.602 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.603 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.603 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.604 DEBUG:boto:Params: {} 2022-09-12T20:14:19.604 DEBUG:boto:Token: None 2022-09-12T20:14:19.605 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/ 2022-09-12T20:14:19.606 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk= 2022-09-12T20:14:19.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk=', 'Content-Length': '0'} 2022-09-12T20:14:19.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004325d88246bc670c-00631f931b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.610 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:19.611 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:19.611 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:19.612 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:14:19.612 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:14:19.613 DEBUG:boto:Method: GET 2022-09-12T20:14:19.613 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:14:19.614 DEBUG:boto:Data: 2022-09-12T20:14:19.615 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.615 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.616 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.616 DEBUG:boto:Params: {} 2022-09-12T20:14:19.617 DEBUG:boto:Token: None 2022-09-12T20:14:19.617 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/ 2022-09-12T20:14:19.618 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk= 2022-09-12T20:14:19.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk=', 'Content-Length': '0'} 2022-09-12T20:14:19.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a00dbbbccb0c6b59-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.624 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:19.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-12T20:14:19.625 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:19.626 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:19.627 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:19.627 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:19.628 DEBUG:boto:Method: GET 2022-09-12T20:14:19.628 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:19.629 DEBUG:boto:Data: 2022-09-12T20:14:19.629 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.630 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.630 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.631 DEBUG:boto:Params: {} 2022-09-12T20:14:19.631 DEBUG:boto:Token: None 2022-09-12T20:14:19.632 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:19.633 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PJ6DXpzBp7ERTgb0G2X47qvbXCA= 2022-09-12T20:14:19.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PJ6DXpzBp7ERTgb0G2X47qvbXCA=', 'Content-Length': '0'} 2022-09-12T20:14:19.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000055654a0b842df04-00631f931b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.636 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:19.637 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:19.638 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:19.638 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:19.639 DEBUG:boto:Method: GET 2022-09-12T20:14:19.639 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:19.640 DEBUG:boto:Data: 2022-09-12T20:14:19.640 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.641 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.641 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.642 DEBUG:boto:Params: {} 2022-09-12T20:14:19.643 DEBUG:boto:Token: None 2022-09-12T20:14:19.643 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:19.644 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PJ6DXpzBp7ERTgb0G2X47qvbXCA= 2022-09-12T20:14:19.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PJ6DXpzBp7ERTgb0G2X47qvbXCA=', 'Content-Length': '0'} 2022-09-12T20:14:19.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b4901093ea38d7a-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.647 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:19.648 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:19.648 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:19.649 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:19.649 DEBUG:boto:Method: HEAD 2022-09-12T20:14:19.650 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:19.650 DEBUG:boto:Data: 2022-09-12T20:14:19.651 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.651 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.652 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.652 DEBUG:boto:Params: {} 2022-09-12T20:14:19.653 DEBUG:boto:Token: None 2022-09-12T20:14:19.653 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:19.654 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6QzS/vlNWVX0IePWNQ6ZcPVjZrI= 2022-09-12T20:14:19.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6QzS/vlNWVX0IePWNQ6ZcPVjZrI=', 'Content-Length': '0'} 2022-09-12T20:14:19.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd875d002dbb454e-00631f931b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.657 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:19.658 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:19.659 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:19.659 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:19.660 DEBUG:boto:Method: HEAD 2022-09-12T20:14:19.660 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:19.661 DEBUG:boto:Data: 2022-09-12T20:14:19.661 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.662 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.662 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.663 DEBUG:boto:Params: {} 2022-09-12T20:14:19.663 DEBUG:boto:Token: None 2022-09-12T20:14:19.664 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:19.664 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6QzS/vlNWVX0IePWNQ6ZcPVjZrI= 2022-09-12T20:14:19.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6QzS/vlNWVX0IePWNQ6ZcPVjZrI=', 'Content-Length': '0'} 2022-09-12T20:14:19.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa7672b362f1be6d-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-12T20:14:19.668 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:19.669 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:19.669 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:19.670 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:19.671 DEBUG:boto:Method: GET 2022-09-12T20:14:19.671 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:19.672 DEBUG:boto:Data: 2022-09-12T20:14:19.672 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.673 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.673 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.674 DEBUG:boto:Params: {} 2022-09-12T20:14:19.674 DEBUG:boto:Token: None 2022-09-12T20:14:19.675 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:19.676 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PJ6DXpzBp7ERTgb0G2X47qvbXCA= 2022-09-12T20:14:19.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PJ6DXpzBp7ERTgb0G2X47qvbXCA=', 'Content-Length': '0'} 2022-09-12T20:14:19.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cf2f09def5c3014-00631f931b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.679 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:19.680 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:19.680 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:19.681 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:19.681 DEBUG:boto:Method: GET 2022-09-12T20:14:19.682 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:19.682 DEBUG:boto:Data: 2022-09-12T20:14:19.683 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.684 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.684 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.685 DEBUG:boto:Params: {} 2022-09-12T20:14:19.685 DEBUG:boto:Token: None 2022-09-12T20:14:19.686 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:19.686 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PJ6DXpzBp7ERTgb0G2X47qvbXCA= 2022-09-12T20:14:19.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PJ6DXpzBp7ERTgb0G2X47qvbXCA=', 'Content-Length': '0'} 2022-09-12T20:14:19.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000745dbb929a15ef73-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.730 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:19.731 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:19.731 DEBUG:boto:path=/jdpiak-75/?prefix=obj93 2022-09-12T20:14:19.732 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj93 2022-09-12T20:14:19.732 DEBUG:boto:Method: GET 2022-09-12T20:14:19.733 DEBUG:boto:Path: /jdpiak-75/?prefix=obj93 2022-09-12T20:14:19.733 DEBUG:boto:Data: 2022-09-12T20:14:19.734 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.735 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.735 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.736 DEBUG:boto:Params: {} 2022-09-12T20:14:19.736 DEBUG:boto:Token: None 2022-09-12T20:14:19.737 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/ 2022-09-12T20:14:19.737 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk= 2022-09-12T20:14:19.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk=', 'Content-Length': '0'} 2022-09-12T20:14:19.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000daab11acbf8213cb-00631f931b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.742 DEBUG:boto:b'jdpiak-75obj931000falseobj932022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:19.743 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:19.743 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:19.744 DEBUG:boto:path=/jdpiak-75/?prefix=obj93 2022-09-12T20:14:19.744 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj93 2022-09-12T20:14:19.745 DEBUG:boto:Method: GET 2022-09-12T20:14:19.745 DEBUG:boto:Path: /jdpiak-75/?prefix=obj93 2022-09-12T20:14:19.746 DEBUG:boto:Data: 2022-09-12T20:14:19.747 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.747 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.748 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.748 DEBUG:boto:Params: {} 2022-09-12T20:14:19.749 DEBUG:boto:Token: None 2022-09-12T20:14:19.749 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/ 2022-09-12T20:14:19.750 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk= 2022-09-12T20:14:19.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk=', 'Content-Length': '0'} 2022-09-12T20:14:19.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a96a3f88573bb046-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.756 DEBUG:boto:b'jdpiak-75obj931000falseobj932022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:19.757 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:19.757 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:19.758 DEBUG:boto:Method: HEAD 2022-09-12T20:14:19.758 DEBUG:boto:Path: /jdpiak-75/obj93 2022-09-12T20:14:19.759 DEBUG:boto:Data: 2022-09-12T20:14:19.759 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.760 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.760 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.761 DEBUG:boto:Params: {} 2022-09-12T20:14:19.761 DEBUG:boto:Token: None 2022-09-12T20:14:19.762 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj93 2022-09-12T20:14:19.763 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iELsaXTsS5cxRQuMkEtYB7Lq0f8= 2022-09-12T20:14:19.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iELsaXTsS5cxRQuMkEtYB7Lq0f8=', 'Content-Length': '0'} 2022-09-12T20:14:19.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007db130f781489736-00631f931b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.766 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:19.767 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:19.767 DEBUG:boto:Method: HEAD 2022-09-12T20:14:19.768 DEBUG:boto:Path: /jdpiak-75/obj93 2022-09-12T20:14:19.768 DEBUG:boto:Data: 2022-09-12T20:14:19.769 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.769 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.770 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.779 DEBUG:boto:Params: {} 2022-09-12T20:14:19.780 DEBUG:boto:Token: None 2022-09-12T20:14:19.781 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj93 2022-09-12T20:14:19.781 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iELsaXTsS5cxRQuMkEtYB7Lq0f8= 2022-09-12T20:14:19.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iELsaXTsS5cxRQuMkEtYB7Lq0f8=', 'Content-Length': '0'} 2022-09-12T20:14:19.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab926189003c09c7-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-12T20:14:19.786 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:19.786 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:19.787 DEBUG:boto:Method: GET 2022-09-12T20:14:19.787 DEBUG:boto:Path: /jdpiak-75/obj93 2022-09-12T20:14:19.788 DEBUG:boto:Data: 2022-09-12T20:14:19.789 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.789 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.790 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.790 DEBUG:boto:Params: {} 2022-09-12T20:14:19.791 DEBUG:boto:Token: None 2022-09-12T20:14:19.791 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj93 2022-09-12T20:14:19.792 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cFdJbv2lCyL3N12wNwpJpBth6G0= 2022-09-12T20:14:19.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cFdJbv2lCyL3N12wNwpJpBth6G0=', 'Content-Length': '0'} 2022-09-12T20:14:19.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa190c389c828626-00631f931b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.796 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:19.796 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:19.797 DEBUG:boto:Method: GET 2022-09-12T20:14:19.797 DEBUG:boto:Path: /jdpiak-75/obj93 2022-09-12T20:14:19.798 DEBUG:boto:Data: 2022-09-12T20:14:19.798 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.799 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.799 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.800 DEBUG:boto:Params: {} 2022-09-12T20:14:19.801 DEBUG:boto:Token: None 2022-09-12T20:14:19.801 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj93 2022-09-12T20:14:19.802 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cFdJbv2lCyL3N12wNwpJpBth6G0= 2022-09-12T20:14:19.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cFdJbv2lCyL3N12wNwpJpBth6G0=', 'Content-Length': '0'} 2022-09-12T20:14:19.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000478277e2ef112090-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.805 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:19.806 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:19.807 DEBUG:boto:path=/jdpiak-75/?prefix=obj93 2022-09-12T20:14:19.807 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj93 2022-09-12T20:14:19.808 DEBUG:boto:Method: GET 2022-09-12T20:14:19.808 DEBUG:boto:Path: /jdpiak-75/?prefix=obj93 2022-09-12T20:14:19.809 DEBUG:boto:Data: 2022-09-12T20:14:19.809 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.810 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.811 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.811 DEBUG:boto:Params: {} 2022-09-12T20:14:19.812 DEBUG:boto:Token: None 2022-09-12T20:14:19.812 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/ 2022-09-12T20:14:19.813 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk= 2022-09-12T20:14:19.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk=', 'Content-Length': '0'} 2022-09-12T20:14:19.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004621b41984e86a67-00631f931b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.858 DEBUG:boto:b'jdpiak-75obj931000falseobj932022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:19.859 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:19.860 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:19.860 DEBUG:boto:path=/jdpiak-75/?prefix=obj93 2022-09-12T20:14:19.861 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj93 2022-09-12T20:14:19.861 DEBUG:boto:Method: GET 2022-09-12T20:14:19.862 DEBUG:boto:Path: /jdpiak-75/?prefix=obj93 2022-09-12T20:14:19.862 DEBUG:boto:Data: 2022-09-12T20:14:19.863 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.863 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.864 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.865 DEBUG:boto:Params: {} 2022-09-12T20:14:19.865 DEBUG:boto:Token: None 2022-09-12T20:14:19.866 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/ 2022-09-12T20:14:19.866 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk= 2022-09-12T20:14:19.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk=', 'Content-Length': '0'} 2022-09-12T20:14:19.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a6e195bbacf2e6b-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.872 DEBUG:boto:b'jdpiak-75obj931000falseobj932022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:19.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:14:19.874 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:19.874 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:19.875 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:19.875 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:19.876 DEBUG:boto:Method: GET 2022-09-12T20:14:19.876 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:19.877 DEBUG:boto:Data: 2022-09-12T20:14:19.877 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.878 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.878 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.879 DEBUG:boto:Params: {} 2022-09-12T20:14:19.879 DEBUG:boto:Token: None 2022-09-12T20:14:19.880 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:19.881 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:obsrXjONG+P5pitD0L9UasnzeQU= 2022-09-12T20:14:19.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:obsrXjONG+P5pitD0L9UasnzeQU=', 'Content-Length': '0'} 2022-09-12T20:14:19.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba16014f8d0adb8b-00631f931b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.884 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:19.885 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:19.885 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:19.886 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:19.886 DEBUG:boto:Method: GET 2022-09-12T20:14:19.887 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:19.887 DEBUG:boto:Data: 2022-09-12T20:14:19.888 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.888 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.889 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.889 DEBUG:boto:Params: {} 2022-09-12T20:14:19.890 DEBUG:boto:Token: None 2022-09-12T20:14:19.891 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:19.891 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:obsrXjONG+P5pitD0L9UasnzeQU= 2022-09-12T20:14:19.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:obsrXjONG+P5pitD0L9UasnzeQU=', 'Content-Length': '0'} 2022-09-12T20:14:19.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebb360f0ec791c91-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.938 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:19.939 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:19.939 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:19.940 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:19.940 DEBUG:boto:Method: HEAD 2022-09-12T20:14:19.941 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:19.941 DEBUG:boto:Data: 2022-09-12T20:14:19.942 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.942 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.943 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.943 DEBUG:boto:Params: {} 2022-09-12T20:14:19.944 DEBUG:boto:Token: None 2022-09-12T20:14:19.945 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:19.945 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TOjcf8uF0WAaYFCdi6Ox3A64Ask= 2022-09-12T20:14:19.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TOjcf8uF0WAaYFCdi6Ox3A64Ask=', 'Content-Length': '0'} 2022-09-12T20:14:19.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001ebddedc0bef2c5-00631f931b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.948 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:19.949 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:19.950 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:19.950 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:19.951 DEBUG:boto:Method: HEAD 2022-09-12T20:14:19.951 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:19.952 DEBUG:boto:Data: 2022-09-12T20:14:19.952 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.953 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.953 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.954 DEBUG:boto:Params: {} 2022-09-12T20:14:19.954 DEBUG:boto:Token: None 2022-09-12T20:14:19.955 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:19.955 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TOjcf8uF0WAaYFCdi6Ox3A64Ask= 2022-09-12T20:14:19.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TOjcf8uF0WAaYFCdi6Ox3A64Ask=', 'Content-Length': '0'} 2022-09-12T20:14:19.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fc32bbe6dcd7262-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:14:19.959 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:19.960 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:19.960 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:19.961 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:19.961 DEBUG:boto:Method: GET 2022-09-12T20:14:19.962 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:19.963 DEBUG:boto:Data: 2022-09-12T20:14:19.963 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.964 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.964 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.965 DEBUG:boto:Params: {} 2022-09-12T20:14:19.965 DEBUG:boto:Token: None 2022-09-12T20:14:19.966 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:19.967 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:obsrXjONG+P5pitD0L9UasnzeQU= 2022-09-12T20:14:19.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:obsrXjONG+P5pitD0L9UasnzeQU=', 'Content-Length': '0'} 2022-09-12T20:14:19.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072e2064c5e90b5b2-00631f931b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.970 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:19.971 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:19.971 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:19.972 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:19.972 DEBUG:boto:Method: GET 2022-09-12T20:14:19.973 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:19.973 DEBUG:boto:Data: 2022-09-12T20:14:19.974 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.974 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.975 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.975 DEBUG:boto:Params: {} 2022-09-12T20:14:19.976 DEBUG:boto:Token: None 2022-09-12T20:14:19.977 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:19.977 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:obsrXjONG+P5pitD0L9UasnzeQU= 2022-09-12T20:14:19.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:obsrXjONG+P5pitD0L9UasnzeQU=', 'Content-Length': '0'} 2022-09-12T20:14:19.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000750da729d27e9073-00631f931b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.980 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:19.981 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:19.981 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:14:19.982 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:14:19.983 DEBUG:boto:Method: GET 2022-09-12T20:14:19.983 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:14:19.984 DEBUG:boto:Data: 2022-09-12T20:14:19.984 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.985 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:19.985 DEBUG:boto:Port: 8001 2022-09-12T20:14:19.986 DEBUG:boto:Params: {} 2022-09-12T20:14:19.986 DEBUG:boto:Token: None 2022-09-12T20:14:19.987 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/ 2022-09-12T20:14:19.987 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk= 2022-09-12T20:14:19.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk=', 'Content-Length': '0'} 2022-09-12T20:14:19.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034b186bb13435c66-00631f931b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:19.991 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:19.992 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:19.993 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:19.993 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:14:19.994 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:14:19.994 DEBUG:boto:Method: GET 2022-09-12T20:14:19.995 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:14:19.995 DEBUG:boto:Data: 2022-09-12T20:14:19.996 DEBUG:boto:Headers: {} 2022-09-12T20:14:19.996 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:19.997 DEBUG:boto:Port: 8000 2022-09-12T20:14:19.998 DEBUG:boto:Params: {} 2022-09-12T20:14:19.998 DEBUG:boto:Token: None 2022-09-12T20:14:19.999 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:19 GMT /jdpiak-75/ 2022-09-12T20:14:19.999 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk= 2022-09-12T20:14:20.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yo5KYpBSx7Nhl5MlsJxZ9axu4Nk=', 'Content-Length': '0'} 2022-09-12T20:14:20.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d6b51ee8bf36c53-00631f931c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.005 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:20.006 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:20.007 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:20.007 DEBUG:boto:Method: HEAD 2022-09-12T20:14:20.008 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:14:20.008 DEBUG:boto:Data: 2022-09-12T20:14:20.009 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.009 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.010 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.010 DEBUG:boto:Params: {} 2022-09-12T20:14:20.011 DEBUG:boto:Token: None 2022-09-12T20:14:20.011 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj95 2022-09-12T20:14:20.012 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rVb6SpavM0ryc0tO+fhX0J1G0GE= 2022-09-12T20:14:20.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rVb6SpavM0ryc0tO+fhX0J1G0GE=', 'Content-Length': '0'} 2022-09-12T20:14:20.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004083b71e0cc89171-00631f931c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.015 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:20.016 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:20.016 DEBUG:boto:Method: HEAD 2022-09-12T20:14:20.017 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:14:20.017 DEBUG:boto:Data: 2022-09-12T20:14:20.018 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.018 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:20.019 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.019 DEBUG:boto:Params: {} 2022-09-12T20:14:20.020 DEBUG:boto:Token: None 2022-09-12T20:14:20.021 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj95 2022-09-12T20:14:20.021 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rVb6SpavM0ryc0tO+fhX0J1G0GE= 2022-09-12T20:14:20.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rVb6SpavM0ryc0tO+fhX0J1G0GE=', 'Content-Length': '0'} 2022-09-12T20:14:20.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a15269bdce06017-00631f931c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:14:20.025 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:20.026 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:20.026 DEBUG:boto:Method: GET 2022-09-12T20:14:20.027 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:14:20.027 DEBUG:boto:Data: 2022-09-12T20:14:20.028 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.028 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.029 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.029 DEBUG:boto:Params: {} 2022-09-12T20:14:20.030 DEBUG:boto:Token: None 2022-09-12T20:14:20.030 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj95 2022-09-12T20:14:20.031 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1QbB6EFKRl7QMO7gbaK7OINibfc= 2022-09-12T20:14:20.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1QbB6EFKRl7QMO7gbaK7OINibfc=', 'Content-Length': '0'} 2022-09-12T20:14:20.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000552c075fa6815e10-00631f931c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.074 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:20.075 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:20.075 DEBUG:boto:Method: GET 2022-09-12T20:14:20.076 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:14:20.076 DEBUG:boto:Data: 2022-09-12T20:14:20.077 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.077 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:20.078 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.078 DEBUG:boto:Params: {} 2022-09-12T20:14:20.079 DEBUG:boto:Token: None 2022-09-12T20:14:20.080 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj95 2022-09-12T20:14:20.080 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1QbB6EFKRl7QMO7gbaK7OINibfc= 2022-09-12T20:14:20.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1QbB6EFKRl7QMO7gbaK7OINibfc=', 'Content-Length': '0'} 2022-09-12T20:14:20.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d685006611a9bc03-00631f931c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.084 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:20.085 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:20.085 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:14:20.086 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:14:20.086 DEBUG:boto:Method: GET 2022-09-12T20:14:20.087 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:14:20.087 DEBUG:boto:Data: 2022-09-12T20:14:20.088 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.088 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.089 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.089 DEBUG:boto:Params: {} 2022-09-12T20:14:20.090 DEBUG:boto:Token: None 2022-09-12T20:14:20.091 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/ 2022-09-12T20:14:20.091 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo= 2022-09-12T20:14:20.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo=', 'Content-Length': '0'} 2022-09-12T20:14:20.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049ea8016dd561e10-00631f931c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.095 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:20.096 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:20.097 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:20.097 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:14:20.098 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:14:20.098 DEBUG:boto:Method: GET 2022-09-12T20:14:20.099 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:14:20.099 DEBUG:boto:Data: 2022-09-12T20:14:20.100 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.100 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:20.101 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.101 DEBUG:boto:Params: {} 2022-09-12T20:14:20.102 DEBUG:boto:Token: None 2022-09-12T20:14:20.103 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/ 2022-09-12T20:14:20.103 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo= 2022-09-12T20:14:20.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo=', 'Content-Length': '0'} 2022-09-12T20:14:20.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ed776736be7223c-00631f931c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.109 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:20.110 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-75 zones={test-zone3, test-zone1} 2022-09-12T20:14:20.111 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-75 zones={test-zone3, test-zone2} 2022-09-12T20:14:20.111 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:20.112 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:20.112 DEBUG:boto:Method: HEAD 2022-09-12T20:14:20.113 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:14:20.113 DEBUG:boto:Data: 2022-09-12T20:14:20.114 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.114 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.115 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.115 DEBUG:boto:Params: {} 2022-09-12T20:14:20.116 DEBUG:boto:Token: None 2022-09-12T20:14:20.116 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/ 2022-09-12T20:14:20.117 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SP3wNoVzuOJQTwjm5eMpYaAemXY= 2022-09-12T20:14:20.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SP3wNoVzuOJQTwjm5eMpYaAemXY=', 'Content-Length': '0'} 2022-09-12T20:14:20.120 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006139d8e134b58002-00631f931c-1826-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.121 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:20.121 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:20.122 DEBUG:boto:Method: HEAD 2022-09-12T20:14:20.122 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:14:20.123 DEBUG:boto:Data: 2022-09-12T20:14:20.123 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.124 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.124 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.125 DEBUG:boto:Params: {} 2022-09-12T20:14:20.126 DEBUG:boto:Token: None 2022-09-12T20:14:20.126 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/ 2022-09-12T20:14:20.127 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SP3wNoVzuOJQTwjm5eMpYaAemXY= 2022-09-12T20:14:20.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SP3wNoVzuOJQTwjm5eMpYaAemXY=', 'Content-Length': '0'} 2022-09-12T20:14:20.130 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ddb00ca8ab6197bc-00631f931c-189d-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.131 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T20:14:20.131 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:20.132 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:20.132 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:20.133 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:20.133 DEBUG:boto:Method: GET 2022-09-12T20:14:20.134 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:20.135 DEBUG:boto:Data: 2022-09-12T20:14:20.135 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.136 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.136 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.137 DEBUG:boto:Params: {} 2022-09-12T20:14:20.137 DEBUG:boto:Token: None 2022-09-12T20:14:20.138 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/?versions 2022-09-12T20:14:20.138 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cyvp1t+0M2a1TQV8eC0nvHxWEFc= 2022-09-12T20:14:20.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cyvp1t+0M2a1TQV8eC0nvHxWEFc=', 'Content-Length': '0'} 2022-09-12T20:14:20.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cb48e531105ebac-00631f931c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.143 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:20.145 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:14:20.145 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-12T20:14:20.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-12T20:14:20.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2022-09-12T20:14:20.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-12T20:14:20.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2022-09-12T20:14:20.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:14:20.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:14:20.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:14:20.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-12T20:14:20.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-12T20:14:20.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2022-09-12T20:14:20.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:14:20.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:14:20.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:14:20.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:14:20.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-12T20:14:20.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:14:20.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:14:20.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:14:20.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-12T20:14:20.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2022-09-12T20:14:20.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:14:20.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-12T20:14:20.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:14:20.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:14:20.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-12T20:14:20.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-12T20:14:20.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2022-09-12T20:14:20.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:14:20.161 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T20:14:20.162 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:20.162 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:20.163 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:20.163 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:20.164 DEBUG:boto:Method: GET 2022-09-12T20:14:20.165 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:20.165 DEBUG:boto:Data: 2022-09-12T20:14:20.166 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.166 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.167 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.167 DEBUG:boto:Params: {} 2022-09-12T20:14:20.168 DEBUG:boto:Token: None 2022-09-12T20:14:20.169 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/?versions 2022-09-12T20:14:20.169 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cyvp1t+0M2a1TQV8eC0nvHxWEFc= 2022-09-12T20:14:20.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cyvp1t+0M2a1TQV8eC0nvHxWEFc=', 'Content-Length': '0'} 2022-09-12T20:14:20.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fde7b9f87929c1e8-00631f931c-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.174 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:20.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:14:20.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-12T20:14:20.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-12T20:14:20.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2022-09-12T20:14:20.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-12T20:14:20.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2022-09-12T20:14:20.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:14:20.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:14:20.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:14:20.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-12T20:14:20.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-12T20:14:20.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2022-09-12T20:14:20.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:14:20.185 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:14:20.185 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:14:20.186 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:14:20.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-12T20:14:20.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:14:20.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:14:20.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:14:20.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-12T20:14:20.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2022-09-12T20:14:20.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:14:20.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-12T20:14:20.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:14:20.192 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:14:20.192 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-12T20:14:20.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-12T20:14:20.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2022-09-12T20:14:20.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:14:20.195 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:20.195 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:20.196 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:20.196 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:20.197 DEBUG:boto:Method: GET 2022-09-12T20:14:20.198 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:20.198 DEBUG:boto:Data: 2022-09-12T20:14:20.199 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.200 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.200 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.201 DEBUG:boto:Params: {} 2022-09-12T20:14:20.201 DEBUG:boto:Token: None 2022-09-12T20:14:20.202 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/?versions 2022-09-12T20:14:20.203 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cyvp1t+0M2a1TQV8eC0nvHxWEFc= 2022-09-12T20:14:20.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cyvp1t+0M2a1TQV8eC0nvHxWEFc=', 'Content-Length': '0'} 2022-09-12T20:14:20.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e7c449b0707d8c8-00631f931c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.208 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:20.211 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:20.211 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:20.212 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:20.212 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:20.213 DEBUG:boto:Method: GET 2022-09-12T20:14:20.213 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:14:20.214 DEBUG:boto:Data: 2022-09-12T20:14:20.215 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.215 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.216 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.216 DEBUG:boto:Params: {} 2022-09-12T20:14:20.217 DEBUG:boto:Token: None 2022-09-12T20:14:20.218 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/?versions 2022-09-12T20:14:20.219 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cyvp1t+0M2a1TQV8eC0nvHxWEFc= 2022-09-12T20:14:20.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cyvp1t+0M2a1TQV8eC0nvHxWEFc=', 'Content-Length': '0'} 2022-09-12T20:14:20.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004eff76b40f97c3a8-00631f931c-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.223 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:20.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:14:20.226 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:20.227 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:20.227 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:20.228 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:20.229 DEBUG:boto:Method: GET 2022-09-12T20:14:20.229 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:20.230 DEBUG:boto:Data: 2022-09-12T20:14:20.231 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.232 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.232 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.233 DEBUG:boto:Params: {} 2022-09-12T20:14:20.234 DEBUG:boto:Token: None 2022-09-12T20:14:20.234 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:20.235 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ScADHCxzCPKOE/adLIKcEYHcbpQ= 2022-09-12T20:14:20.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ScADHCxzCPKOE/adLIKcEYHcbpQ=', 'Content-Length': '0'} 2022-09-12T20:14:20.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e230cd8a5b4cc209-00631f931c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.282 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:20.283 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:20.283 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:20.284 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:20.284 DEBUG:boto:Method: GET 2022-09-12T20:14:20.285 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:20.285 DEBUG:boto:Data: 2022-09-12T20:14:20.286 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.286 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.287 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.287 DEBUG:boto:Params: {} 2022-09-12T20:14:20.288 DEBUG:boto:Token: None 2022-09-12T20:14:20.289 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:20.289 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ScADHCxzCPKOE/adLIKcEYHcbpQ= 2022-09-12T20:14:20.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ScADHCxzCPKOE/adLIKcEYHcbpQ=', 'Content-Length': '0'} 2022-09-12T20:14:20.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6834623aeda1eee-00631f931c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.293 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:20.294 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:20.294 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:20.295 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:20.295 DEBUG:boto:Method: HEAD 2022-09-12T20:14:20.296 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:20.296 DEBUG:boto:Data: 2022-09-12T20:14:20.297 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.297 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.298 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.298 DEBUG:boto:Params: {} 2022-09-12T20:14:20.299 DEBUG:boto:Token: None 2022-09-12T20:14:20.299 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:20.300 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q0yN9Wggf8zVmRUUUOqlqYbsi/4= 2022-09-12T20:14:20.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q0yN9Wggf8zVmRUUUOqlqYbsi/4=', 'Content-Length': '0'} 2022-09-12T20:14:20.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000955ae083ad2ad6e6-00631f931c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.303 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:20.304 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:20.304 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:20.305 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:20.305 DEBUG:boto:Method: HEAD 2022-09-12T20:14:20.306 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:20.306 DEBUG:boto:Data: 2022-09-12T20:14:20.307 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.307 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.308 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.308 DEBUG:boto:Params: {} 2022-09-12T20:14:20.309 DEBUG:boto:Token: None 2022-09-12T20:14:20.309 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:20.310 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q0yN9Wggf8zVmRUUUOqlqYbsi/4= 2022-09-12T20:14:20.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q0yN9Wggf8zVmRUUUOqlqYbsi/4=', 'Content-Length': '0'} 2022-09-12T20:14:20.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfd681e2d87c1a4b-00631f931c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:14:20.314 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:20.314 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:20.315 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:20.315 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:20.316 DEBUG:boto:Method: GET 2022-09-12T20:14:20.316 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:20.317 DEBUG:boto:Data: 2022-09-12T20:14:20.317 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.318 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.318 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.319 DEBUG:boto:Params: {} 2022-09-12T20:14:20.320 DEBUG:boto:Token: None 2022-09-12T20:14:20.320 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:20.321 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ScADHCxzCPKOE/adLIKcEYHcbpQ= 2022-09-12T20:14:20.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ScADHCxzCPKOE/adLIKcEYHcbpQ=', 'Content-Length': '0'} 2022-09-12T20:14:20.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006da28e44dfcc27a5-00631f931c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.324 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:20.325 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:20.325 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:20.326 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:14:20.326 DEBUG:boto:Method: GET 2022-09-12T20:14:20.327 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:20.327 DEBUG:boto:Data: 2022-09-12T20:14:20.328 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.328 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.329 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.330 DEBUG:boto:Params: {} 2022-09-12T20:14:20.330 DEBUG:boto:Token: None 2022-09-12T20:14:20.331 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:14:20.331 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ScADHCxzCPKOE/adLIKcEYHcbpQ= 2022-09-12T20:14:20.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ScADHCxzCPKOE/adLIKcEYHcbpQ=', 'Content-Length': '0'} 2022-09-12T20:14:20.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065e4828e62de17cd-00631f931c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.335 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:20.335 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:20.336 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:14:20.336 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:14:20.337 DEBUG:boto:Method: GET 2022-09-12T20:14:20.337 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:14:20.338 DEBUG:boto:Data: 2022-09-12T20:14:20.339 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.339 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.340 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.340 DEBUG:boto:Params: {} 2022-09-12T20:14:20.341 DEBUG:boto:Token: None 2022-09-12T20:14:20.341 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/ 2022-09-12T20:14:20.342 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo= 2022-09-12T20:14:20.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo=', 'Content-Length': '0'} 2022-09-12T20:14:20.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037b866d877ef4088-00631f931c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.346 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:20.347 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:20.347 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:20.348 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:14:20.348 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:14:20.349 DEBUG:boto:Method: GET 2022-09-12T20:14:20.349 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:14:20.350 DEBUG:boto:Data: 2022-09-12T20:14:20.351 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.351 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.352 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.352 DEBUG:boto:Params: {} 2022-09-12T20:14:20.353 DEBUG:boto:Token: None 2022-09-12T20:14:20.353 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/ 2022-09-12T20:14:20.354 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo= 2022-09-12T20:14:20.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo=', 'Content-Length': '0'} 2022-09-12T20:14:20.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb5e197c5041e776-00631f931c-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.398 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:20.399 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:20.400 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:20.400 DEBUG:boto:Method: HEAD 2022-09-12T20:14:20.401 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:14:20.401 DEBUG:boto:Data: 2022-09-12T20:14:20.402 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.402 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.403 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.403 DEBUG:boto:Params: {} 2022-09-12T20:14:20.404 DEBUG:boto:Token: None 2022-09-12T20:14:20.404 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj11 2022-09-12T20:14:20.405 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SEZ7uQaKRG/lMHwGPadronvYPEw= 2022-09-12T20:14:20.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SEZ7uQaKRG/lMHwGPadronvYPEw=', 'Content-Length': '0'} 2022-09-12T20:14:20.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046c9480ff61053d4-00631f931c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.408 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:20.409 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:20.409 DEBUG:boto:Method: HEAD 2022-09-12T20:14:20.410 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:14:20.411 DEBUG:boto:Data: 2022-09-12T20:14:20.411 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.412 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.412 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.412 DEBUG:boto:Params: {} 2022-09-12T20:14:20.413 DEBUG:boto:Token: None 2022-09-12T20:14:20.414 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj11 2022-09-12T20:14:20.414 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SEZ7uQaKRG/lMHwGPadronvYPEw= 2022-09-12T20:14:20.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SEZ7uQaKRG/lMHwGPadronvYPEw=', 'Content-Length': '0'} 2022-09-12T20:14:20.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005af1189029446f26-00631f931c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:14:20.418 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:20.418 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:20.419 DEBUG:boto:Method: GET 2022-09-12T20:14:20.419 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:14:20.420 DEBUG:boto:Data: 2022-09-12T20:14:20.420 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.421 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.422 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.422 DEBUG:boto:Params: {} 2022-09-12T20:14:20.423 DEBUG:boto:Token: None 2022-09-12T20:14:20.423 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj11 2022-09-12T20:14:20.424 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DdEfFNyjFEDGYkCK7vvaxb5ddEU= 2022-09-12T20:14:20.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DdEfFNyjFEDGYkCK7vvaxb5ddEU=', 'Content-Length': '0'} 2022-09-12T20:14:20.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000303621f4a306a2ec-00631f931c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.427 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:14:20.428 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:14:20.428 DEBUG:boto:Method: GET 2022-09-12T20:14:20.429 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:14:20.430 DEBUG:boto:Data: 2022-09-12T20:14:20.430 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.431 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.431 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.432 DEBUG:boto:Params: {} 2022-09-12T20:14:20.432 DEBUG:boto:Token: None 2022-09-12T20:14:20.433 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj11 2022-09-12T20:14:20.433 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DdEfFNyjFEDGYkCK7vvaxb5ddEU= 2022-09-12T20:14:20.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DdEfFNyjFEDGYkCK7vvaxb5ddEU=', 'Content-Length': '0'} 2022-09-12T20:14:20.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f5ed757399c7808-00631f931c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.437 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:20.438 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:20.438 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:14:20.439 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:14:20.440 DEBUG:boto:Method: GET 2022-09-12T20:14:20.440 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:14:20.441 DEBUG:boto:Data: 2022-09-12T20:14:20.441 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.442 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.442 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.443 DEBUG:boto:Params: {} 2022-09-12T20:14:20.443 DEBUG:boto:Token: None 2022-09-12T20:14:20.444 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/ 2022-09-12T20:14:20.444 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo= 2022-09-12T20:14:20.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo=', 'Content-Length': '0'} 2022-09-12T20:14:20.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d9ba9a436ce9712-00631f931c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.490 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:20.491 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:20.492 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:20.492 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:14:20.493 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:14:20.493 DEBUG:boto:Method: GET 2022-09-12T20:14:20.494 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:14:20.494 DEBUG:boto:Data: 2022-09-12T20:14:20.495 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.495 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.496 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.496 DEBUG:boto:Params: {} 2022-09-12T20:14:20.497 DEBUG:boto:Token: None 2022-09-12T20:14:20.497 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/ 2022-09-12T20:14:20.498 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo= 2022-09-12T20:14:20.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo=', 'Content-Length': '0'} 2022-09-12T20:14:20.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053bfe5d1e8144134-00631f931c-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.502 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:20.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:14:20.503 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:20.504 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:20.505 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:20.505 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:20.506 DEBUG:boto:Method: GET 2022-09-12T20:14:20.506 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:20.507 DEBUG:boto:Data: 2022-09-12T20:14:20.507 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.508 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.508 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.509 DEBUG:boto:Params: {} 2022-09-12T20:14:20.509 DEBUG:boto:Token: None 2022-09-12T20:14:20.510 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:20.510 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7SkELyN+33tF+VHwUBZOcLCzD+Y= 2022-09-12T20:14:20.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7SkELyN+33tF+VHwUBZOcLCzD+Y=', 'Content-Length': '0'} 2022-09-12T20:14:20.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6778eaaac942ae8-00631f931c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.514 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:20.514 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:20.515 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:20.515 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:20.516 DEBUG:boto:Method: GET 2022-09-12T20:14:20.516 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:20.517 DEBUG:boto:Data: 2022-09-12T20:14:20.517 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.518 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.518 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.519 DEBUG:boto:Params: {} 2022-09-12T20:14:20.520 DEBUG:boto:Token: None 2022-09-12T20:14:20.520 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:20.521 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7SkELyN+33tF+VHwUBZOcLCzD+Y= 2022-09-12T20:14:20.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7SkELyN+33tF+VHwUBZOcLCzD+Y=', 'Content-Length': '0'} 2022-09-12T20:14:20.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b20cf1bb177354e0-00631f931c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.524 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:20.525 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:20.525 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:20.526 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:20.526 DEBUG:boto:Method: HEAD 2022-09-12T20:14:20.527 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:20.527 DEBUG:boto:Data: 2022-09-12T20:14:20.528 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.528 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.529 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.530 DEBUG:boto:Params: {} 2022-09-12T20:14:20.531 DEBUG:boto:Token: None 2022-09-12T20:14:20.531 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:20.532 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XFBZqljogLjMWbOuE636z+UbqJ0= 2022-09-12T20:14:20.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XFBZqljogLjMWbOuE636z+UbqJ0=', 'Content-Length': '0'} 2022-09-12T20:14:20.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002050e5623cc418c5-00631f931c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.536 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:20.536 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:20.537 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:20.537 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:20.538 DEBUG:boto:Method: HEAD 2022-09-12T20:14:20.538 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:20.539 DEBUG:boto:Data: 2022-09-12T20:14:20.539 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.540 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.540 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.541 DEBUG:boto:Params: {} 2022-09-12T20:14:20.542 DEBUG:boto:Token: None 2022-09-12T20:14:20.542 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:20.543 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XFBZqljogLjMWbOuE636z+UbqJ0= 2022-09-12T20:14:20.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XFBZqljogLjMWbOuE636z+UbqJ0=', 'Content-Length': '0'} 2022-09-12T20:14:20.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096c6d407d3fbc883-00631f931c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:14:20.547 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:20.547 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:20.548 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:20.548 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:20.549 DEBUG:boto:Method: GET 2022-09-12T20:14:20.550 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:20.550 DEBUG:boto:Data: 2022-09-12T20:14:20.551 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.551 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.552 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.552 DEBUG:boto:Params: {} 2022-09-12T20:14:20.553 DEBUG:boto:Token: None 2022-09-12T20:14:20.554 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:20.554 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7SkELyN+33tF+VHwUBZOcLCzD+Y= 2022-09-12T20:14:20.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7SkELyN+33tF+VHwUBZOcLCzD+Y=', 'Content-Length': '0'} 2022-09-12T20:14:20.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3998f54debe60ec-00631f931c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.558 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:20.559 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:20.559 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:20.560 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:14:20.560 DEBUG:boto:Method: GET 2022-09-12T20:14:20.561 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:20.562 DEBUG:boto:Data: 2022-09-12T20:14:20.562 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.563 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.563 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.564 DEBUG:boto:Params: {} 2022-09-12T20:14:20.564 DEBUG:boto:Token: None 2022-09-12T20:14:20.565 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:14:20.566 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7SkELyN+33tF+VHwUBZOcLCzD+Y= 2022-09-12T20:14:20.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7SkELyN+33tF+VHwUBZOcLCzD+Y=', 'Content-Length': '0'} 2022-09-12T20:14:20.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee42812e6757e978-00631f931c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.569 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:20.570 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:20.571 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:14:20.571 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:14:20.572 DEBUG:boto:Method: GET 2022-09-12T20:14:20.572 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:14:20.573 DEBUG:boto:Data: 2022-09-12T20:14:20.573 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.574 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.575 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.575 DEBUG:boto:Params: {} 2022-09-12T20:14:20.576 DEBUG:boto:Token: None 2022-09-12T20:14:20.576 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/ 2022-09-12T20:14:20.577 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo= 2022-09-12T20:14:20.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo=', 'Content-Length': '0'} 2022-09-12T20:14:20.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edceda6c97946eed-00631f931c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.581 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:20.582 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:20.583 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:20.584 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:14:20.584 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:14:20.585 DEBUG:boto:Method: GET 2022-09-12T20:14:20.585 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:14:20.586 DEBUG:boto:Data: 2022-09-12T20:14:20.586 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.587 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.587 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.588 DEBUG:boto:Params: {} 2022-09-12T20:14:20.589 DEBUG:boto:Token: None 2022-09-12T20:14:20.589 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/ 2022-09-12T20:14:20.590 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo= 2022-09-12T20:14:20.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo=', 'Content-Length': '0'} 2022-09-12T20:14:20.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b047850a8dc3968a-00631f931c-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.594 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:20.595 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:20.595 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:20.596 DEBUG:boto:Method: HEAD 2022-09-12T20:14:20.596 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:14:20.597 DEBUG:boto:Data: 2022-09-12T20:14:20.597 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.598 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.598 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.599 DEBUG:boto:Params: {} 2022-09-12T20:14:20.599 DEBUG:boto:Token: None 2022-09-12T20:14:20.600 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj115 2022-09-12T20:14:20.601 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cnmy5kkqXADdJfGOKjPM3bsSk5c= 2022-09-12T20:14:20.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cnmy5kkqXADdJfGOKjPM3bsSk5c=', 'Content-Length': '0'} 2022-09-12T20:14:20.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000374b133be68bff5b-00631f931c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.604 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:20.605 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:20.605 DEBUG:boto:Method: HEAD 2022-09-12T20:14:20.606 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:14:20.606 DEBUG:boto:Data: 2022-09-12T20:14:20.607 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.608 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.608 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.609 DEBUG:boto:Params: {} 2022-09-12T20:14:20.609 DEBUG:boto:Token: None 2022-09-12T20:14:20.610 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj115 2022-09-12T20:14:20.610 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cnmy5kkqXADdJfGOKjPM3bsSk5c= 2022-09-12T20:14:20.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cnmy5kkqXADdJfGOKjPM3bsSk5c=', 'Content-Length': '0'} 2022-09-12T20:14:20.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ee34e7cbedf0877-00631f931c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:14:20.614 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:20.615 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:20.615 DEBUG:boto:Method: GET 2022-09-12T20:14:20.616 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:14:20.617 DEBUG:boto:Data: 2022-09-12T20:14:20.617 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.618 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.618 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.619 DEBUG:boto:Params: {} 2022-09-12T20:14:20.619 DEBUG:boto:Token: None 2022-09-12T20:14:20.620 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj115 2022-09-12T20:14:20.621 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:InR36yj6uMclMOfpn28Hgo46csI= 2022-09-12T20:14:20.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:InR36yj6uMclMOfpn28Hgo46csI=', 'Content-Length': '0'} 2022-09-12T20:14:20.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002229833c9e288b53-00631f931c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.624 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:14:20.625 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:14:20.626 DEBUG:boto:Method: GET 2022-09-12T20:14:20.626 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:14:20.627 DEBUG:boto:Data: 2022-09-12T20:14:20.627 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.628 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.628 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.629 DEBUG:boto:Params: {} 2022-09-12T20:14:20.630 DEBUG:boto:Token: None 2022-09-12T20:14:20.630 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj115 2022-09-12T20:14:20.631 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:InR36yj6uMclMOfpn28Hgo46csI= 2022-09-12T20:14:20.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:InR36yj6uMclMOfpn28Hgo46csI=', 'Content-Length': '0'} 2022-09-12T20:14:20.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096419cd9023cc764-00631f931c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.634 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:20.635 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:20.635 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:14:20.636 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:14:20.636 DEBUG:boto:Method: GET 2022-09-12T20:14:20.637 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:14:20.637 DEBUG:boto:Data: 2022-09-12T20:14:20.638 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.639 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.639 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.640 DEBUG:boto:Params: {} 2022-09-12T20:14:20.640 DEBUG:boto:Token: None 2022-09-12T20:14:20.641 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/ 2022-09-12T20:14:20.642 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo= 2022-09-12T20:14:20.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo=', 'Content-Length': '0'} 2022-09-12T20:14:20.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce0b0c68368e31e5-00631f931c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.646 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:20.647 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:20.648 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:20.648 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:14:20.649 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:14:20.649 DEBUG:boto:Method: GET 2022-09-12T20:14:20.650 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:14:20.650 DEBUG:boto:Data: 2022-09-12T20:14:20.651 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.651 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.652 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.652 DEBUG:boto:Params: {} 2022-09-12T20:14:20.653 DEBUG:boto:Token: None 2022-09-12T20:14:20.654 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/ 2022-09-12T20:14:20.654 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo= 2022-09-12T20:14:20.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo=', 'Content-Length': '0'} 2022-09-12T20:14:20.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8d25add91516de3-00631f931c-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.658 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:20.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:14:20.660 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:20.660 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:20.661 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:20.661 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:20.662 DEBUG:boto:Method: GET 2022-09-12T20:14:20.663 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:20.663 DEBUG:boto:Data: 2022-09-12T20:14:20.664 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.664 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.665 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.665 DEBUG:boto:Params: {} 2022-09-12T20:14:20.666 DEBUG:boto:Token: None 2022-09-12T20:14:20.667 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:20.667 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/Qh2jimGCTNRjPqAxoFJQ/aFVoI= 2022-09-12T20:14:20.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/Qh2jimGCTNRjPqAxoFJQ/aFVoI=', 'Content-Length': '0'} 2022-09-12T20:14:20.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002058e25455830cc1-00631f931c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.671 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:20.672 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:20.672 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:20.673 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:20.673 DEBUG:boto:Method: GET 2022-09-12T20:14:20.674 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:20.674 DEBUG:boto:Data: 2022-09-12T20:14:20.675 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.675 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.676 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.677 DEBUG:boto:Params: {} 2022-09-12T20:14:20.677 DEBUG:boto:Token: None 2022-09-12T20:14:20.678 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:20.678 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/Qh2jimGCTNRjPqAxoFJQ/aFVoI= 2022-09-12T20:14:20.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/Qh2jimGCTNRjPqAxoFJQ/aFVoI=', 'Content-Length': '0'} 2022-09-12T20:14:20.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000036313e4add1f2ac-00631f931c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.682 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:20.683 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:20.683 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:20.684 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:20.684 DEBUG:boto:Method: HEAD 2022-09-12T20:14:20.685 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:20.685 DEBUG:boto:Data: 2022-09-12T20:14:20.686 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.686 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.687 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.687 DEBUG:boto:Params: {} 2022-09-12T20:14:20.688 DEBUG:boto:Token: None 2022-09-12T20:14:20.688 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:20.689 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TxUPuohoJb1yQOAmOB2dGq9ZIac= 2022-09-12T20:14:20.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TxUPuohoJb1yQOAmOB2dGq9ZIac=', 'Content-Length': '0'} 2022-09-12T20:14:20.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2755e83a54d60c0-00631f931c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.692 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:20.693 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:20.693 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:20.694 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:20.695 DEBUG:boto:Method: HEAD 2022-09-12T20:14:20.695 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:20.696 DEBUG:boto:Data: 2022-09-12T20:14:20.696 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.697 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.697 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.698 DEBUG:boto:Params: {} 2022-09-12T20:14:20.698 DEBUG:boto:Token: None 2022-09-12T20:14:20.699 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:20.700 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TxUPuohoJb1yQOAmOB2dGq9ZIac= 2022-09-12T20:14:20.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TxUPuohoJb1yQOAmOB2dGq9ZIac=', 'Content-Length': '0'} 2022-09-12T20:14:20.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d523e14cf8c79dc-00631f931c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:14:20.704 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:20.704 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:20.705 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:20.705 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:20.706 DEBUG:boto:Method: GET 2022-09-12T20:14:20.706 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:20.707 DEBUG:boto:Data: 2022-09-12T20:14:20.708 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.708 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.709 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.709 DEBUG:boto:Params: {} 2022-09-12T20:14:20.710 DEBUG:boto:Token: None 2022-09-12T20:14:20.710 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:20.711 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/Qh2jimGCTNRjPqAxoFJQ/aFVoI= 2022-09-12T20:14:20.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/Qh2jimGCTNRjPqAxoFJQ/aFVoI=', 'Content-Length': '0'} 2022-09-12T20:14:20.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037b1defc1e862544-00631f931c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.754 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:20.754 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:20.755 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:20.756 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:14:20.756 DEBUG:boto:Method: GET 2022-09-12T20:14:20.757 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:20.757 DEBUG:boto:Data: 2022-09-12T20:14:20.758 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.758 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.759 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.760 DEBUG:boto:Params: {} 2022-09-12T20:14:20.760 DEBUG:boto:Token: None 2022-09-12T20:14:20.761 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:14:20.761 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/Qh2jimGCTNRjPqAxoFJQ/aFVoI= 2022-09-12T20:14:20.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/Qh2jimGCTNRjPqAxoFJQ/aFVoI=', 'Content-Length': '0'} 2022-09-12T20:14:20.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fef1a95d5b89a52b-00631f931c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.765 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:20.765 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:20.766 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:14:20.766 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:14:20.767 DEBUG:boto:Method: GET 2022-09-12T20:14:20.767 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:14:20.768 DEBUG:boto:Data: 2022-09-12T20:14:20.769 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.769 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.770 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.770 DEBUG:boto:Params: {} 2022-09-12T20:14:20.771 DEBUG:boto:Token: None 2022-09-12T20:14:20.771 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/ 2022-09-12T20:14:20.772 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo= 2022-09-12T20:14:20.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo=', 'Content-Length': '0'} 2022-09-12T20:14:20.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006649503f1543205f-00631f931c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.776 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:20.777 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:20.778 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:20.778 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:14:20.779 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:14:20.779 DEBUG:boto:Method: GET 2022-09-12T20:14:20.780 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:14:20.780 DEBUG:boto:Data: 2022-09-12T20:14:20.781 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.782 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.782 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.783 DEBUG:boto:Params: {} 2022-09-12T20:14:20.783 DEBUG:boto:Token: None 2022-09-12T20:14:20.784 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/ 2022-09-12T20:14:20.784 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo= 2022-09-12T20:14:20.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo=', 'Content-Length': '0'} 2022-09-12T20:14:20.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5e0d1d31e5f6e55-00631f931c-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.788 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:20.789 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:20.790 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:20.790 DEBUG:boto:Method: HEAD 2022-09-12T20:14:20.791 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:14:20.791 DEBUG:boto:Data: 2022-09-12T20:14:20.792 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.792 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.793 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.793 DEBUG:boto:Params: {} 2022-09-12T20:14:20.794 DEBUG:boto:Token: None 2022-09-12T20:14:20.794 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj116 2022-09-12T20:14:20.795 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ud0w4QpIERhMvMbAKtofeUqJy0g= 2022-09-12T20:14:20.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ud0w4QpIERhMvMbAKtofeUqJy0g=', 'Content-Length': '0'} 2022-09-12T20:14:20.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085542e221df31021-00631f931c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.798 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:20.799 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:20.799 DEBUG:boto:Method: HEAD 2022-09-12T20:14:20.800 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:14:20.800 DEBUG:boto:Data: 2022-09-12T20:14:20.801 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.801 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.802 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.802 DEBUG:boto:Params: {} 2022-09-12T20:14:20.803 DEBUG:boto:Token: None 2022-09-12T20:14:20.803 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj116 2022-09-12T20:14:20.804 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ud0w4QpIERhMvMbAKtofeUqJy0g= 2022-09-12T20:14:20.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ud0w4QpIERhMvMbAKtofeUqJy0g=', 'Content-Length': '0'} 2022-09-12T20:14:20.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000997efeeb5b2a9b90-00631f931c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:14:20.808 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:20.808 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:20.809 DEBUG:boto:Method: GET 2022-09-12T20:14:20.810 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:14:20.810 DEBUG:boto:Data: 2022-09-12T20:14:20.811 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.811 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.812 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.812 DEBUG:boto:Params: {} 2022-09-12T20:14:20.813 DEBUG:boto:Token: None 2022-09-12T20:14:20.814 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj116 2022-09-12T20:14:20.814 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dlRnCBd5WpLGelHeb1jxn9gtZYo= 2022-09-12T20:14:20.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dlRnCBd5WpLGelHeb1jxn9gtZYo=', 'Content-Length': '0'} 2022-09-12T20:14:20.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000158cb9133d62d1e4-00631f931c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.818 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:14:20.818 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:14:20.819 DEBUG:boto:Method: GET 2022-09-12T20:14:20.819 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:14:20.820 DEBUG:boto:Data: 2022-09-12T20:14:20.821 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.821 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.822 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.822 DEBUG:boto:Params: {} 2022-09-12T20:14:20.823 DEBUG:boto:Token: None 2022-09-12T20:14:20.823 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj116 2022-09-12T20:14:20.824 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dlRnCBd5WpLGelHeb1jxn9gtZYo= 2022-09-12T20:14:20.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dlRnCBd5WpLGelHeb1jxn9gtZYo=', 'Content-Length': '0'} 2022-09-12T20:14:20.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e7f1d943ce3a67c-00631f931c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.827 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:20.828 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:20.829 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:14:20.829 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:14:20.830 DEBUG:boto:Method: GET 2022-09-12T20:14:20.830 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:14:20.831 DEBUG:boto:Data: 2022-09-12T20:14:20.831 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.832 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.832 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.833 DEBUG:boto:Params: {} 2022-09-12T20:14:20.834 DEBUG:boto:Token: None 2022-09-12T20:14:20.834 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/ 2022-09-12T20:14:20.835 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo= 2022-09-12T20:14:20.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo=', 'Content-Length': '0'} 2022-09-12T20:14:20.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c9ee59b156e570d-00631f931c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.839 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:20.840 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:20.840 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:20.841 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:14:20.841 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:14:20.842 DEBUG:boto:Method: GET 2022-09-12T20:14:20.843 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:14:20.843 DEBUG:boto:Data: 2022-09-12T20:14:20.844 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.844 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.845 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.845 DEBUG:boto:Params: {} 2022-09-12T20:14:20.846 DEBUG:boto:Token: None 2022-09-12T20:14:20.846 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/ 2022-09-12T20:14:20.847 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo= 2022-09-12T20:14:20.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SLzGkx7R19lJON8zaKzVVhyZuKo=', 'Content-Length': '0'} 2022-09-12T20:14:20.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f40d912f7d54343-00631f931c-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.894 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:20.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-12T20:14:20.895 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:20.896 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:20.896 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:20.897 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:20.897 DEBUG:boto:Method: GET 2022-09-12T20:14:20.898 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:20.899 DEBUG:boto:Data: 2022-09-12T20:14:20.899 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.900 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.900 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.901 DEBUG:boto:Params: {} 2022-09-12T20:14:20.901 DEBUG:boto:Token: None 2022-09-12T20:14:20.902 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:20.903 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Mil6UBdNA5D2iRIZBLLsXeIqze8= 2022-09-12T20:14:20.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Mil6UBdNA5D2iRIZBLLsXeIqze8=', 'Content-Length': '0'} 2022-09-12T20:14:20.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d6e43ab2e811462-00631f931c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.906 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:20.907 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:20.908 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:20.908 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:20.909 DEBUG:boto:Method: GET 2022-09-12T20:14:20.909 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:20.910 DEBUG:boto:Data: 2022-09-12T20:14:20.911 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.911 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.912 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.913 DEBUG:boto:Params: {} 2022-09-12T20:14:20.913 DEBUG:boto:Token: None 2022-09-12T20:14:20.914 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:20.915 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Mil6UBdNA5D2iRIZBLLsXeIqze8= 2022-09-12T20:14:20.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Mil6UBdNA5D2iRIZBLLsXeIqze8=', 'Content-Length': '0'} 2022-09-12T20:14:20.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000996bea0e91f97c9a-00631f931c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.919 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:20.919 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:20.920 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:20.920 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:20.921 DEBUG:boto:Method: HEAD 2022-09-12T20:14:20.922 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:20.922 DEBUG:boto:Data: 2022-09-12T20:14:20.923 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.923 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.924 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.925 DEBUG:boto:Params: {} 2022-09-12T20:14:20.925 DEBUG:boto:Token: None 2022-09-12T20:14:20.926 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:20.927 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nP9LSNaYeI1jXMkci8ZdpCv3/m0= 2022-09-12T20:14:20.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nP9LSNaYeI1jXMkci8ZdpCv3/m0=', 'Content-Length': '0'} 2022-09-12T20:14:20.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bdecf0532591d55-00631f931c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.931 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:20.931 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:20.932 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:20.932 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:20.933 DEBUG:boto:Method: HEAD 2022-09-12T20:14:20.934 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:20.934 DEBUG:boto:Data: 2022-09-12T20:14:20.935 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.936 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.936 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.937 DEBUG:boto:Params: {} 2022-09-12T20:14:20.938 DEBUG:boto:Token: None 2022-09-12T20:14:20.938 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:20.939 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nP9LSNaYeI1jXMkci8ZdpCv3/m0= 2022-09-12T20:14:20.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nP9LSNaYeI1jXMkci8ZdpCv3/m0=', 'Content-Length': '0'} 2022-09-12T20:14:20.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ddfcfb169faa092-00631f931c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-12T20:14:20.943 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:20.944 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:20.945 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:20.945 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:20.946 DEBUG:boto:Method: GET 2022-09-12T20:14:20.947 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:20.947 DEBUG:boto:Data: 2022-09-12T20:14:20.948 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.949 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:20.949 DEBUG:boto:Port: 8001 2022-09-12T20:14:20.950 DEBUG:boto:Params: {} 2022-09-12T20:14:20.951 DEBUG:boto:Token: None 2022-09-12T20:14:20.951 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:20.952 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Mil6UBdNA5D2iRIZBLLsXeIqze8= 2022-09-12T20:14:20.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Mil6UBdNA5D2iRIZBLLsXeIqze8=', 'Content-Length': '0'} 2022-09-12T20:14:20.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ad6b3d500174dee-00631f931c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:20.956 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:20.956 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:20.957 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:20.958 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:14:20.958 DEBUG:boto:Method: GET 2022-09-12T20:14:20.959 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:20.960 DEBUG:boto:Data: 2022-09-12T20:14:20.960 DEBUG:boto:Headers: {} 2022-09-12T20:14:20.961 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:20.962 DEBUG:boto:Port: 8000 2022-09-12T20:14:20.962 DEBUG:boto:Params: {} 2022-09-12T20:14:20.963 DEBUG:boto:Token: None 2022-09-12T20:14:20.964 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:20 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:14:20.964 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Mil6UBdNA5D2iRIZBLLsXeIqze8= 2022-09-12T20:14:20.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Mil6UBdNA5D2iRIZBLLsXeIqze8=', 'Content-Length': '0'} 2022-09-12T20:14:20.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019db710f4449f927-00631f931c-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.010 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:21.011 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:21.011 DEBUG:boto:path=/jdpiak-75/?prefix=obj124 2022-09-12T20:14:21.012 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj124 2022-09-12T20:14:21.013 DEBUG:boto:Method: GET 2022-09-12T20:14:21.014 DEBUG:boto:Path: /jdpiak-75/?prefix=obj124 2022-09-12T20:14:21.014 DEBUG:boto:Data: 2022-09-12T20:14:21.015 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.015 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.016 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.017 DEBUG:boto:Params: {} 2022-09-12T20:14:21.017 DEBUG:boto:Token: None 2022-09-12T20:14:21.018 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/ 2022-09-12T20:14:21.019 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg= 2022-09-12T20:14:21.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg=', 'Content-Length': '0'} 2022-09-12T20:14:21.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4abe2d90ec753d3-00631f931d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.023 DEBUG:boto:b'jdpiak-75obj1241000falseobj1242022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:21.024 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:21.025 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:21.026 DEBUG:boto:path=/jdpiak-75/?prefix=obj124 2022-09-12T20:14:21.026 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj124 2022-09-12T20:14:21.027 DEBUG:boto:Method: GET 2022-09-12T20:14:21.028 DEBUG:boto:Path: /jdpiak-75/?prefix=obj124 2022-09-12T20:14:21.028 DEBUG:boto:Data: 2022-09-12T20:14:21.029 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.030 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.030 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.031 DEBUG:boto:Params: {} 2022-09-12T20:14:21.032 DEBUG:boto:Token: None 2022-09-12T20:14:21.032 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/ 2022-09-12T20:14:21.033 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg= 2022-09-12T20:14:21.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg=', 'Content-Length': '0'} 2022-09-12T20:14:21.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039e201e6e24e8071-00631f931d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.037 DEBUG:boto:b'jdpiak-75obj1241000falseobj1242022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:21.038 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:21.039 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:21.040 DEBUG:boto:Method: HEAD 2022-09-12T20:14:21.040 DEBUG:boto:Path: /jdpiak-75/obj124 2022-09-12T20:14:21.041 DEBUG:boto:Data: 2022-09-12T20:14:21.041 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.042 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.043 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.043 DEBUG:boto:Params: {} 2022-09-12T20:14:21.044 DEBUG:boto:Token: None 2022-09-12T20:14:21.045 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj124 2022-09-12T20:14:21.045 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OzIG93xaO5pqAvNYjMm+YAGUicM= 2022-09-12T20:14:21.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OzIG93xaO5pqAvNYjMm+YAGUicM=', 'Content-Length': '0'} 2022-09-12T20:14:21.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e11c1744f20a8056-00631f931d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.049 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:21.050 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:21.050 DEBUG:boto:Method: HEAD 2022-09-12T20:14:21.051 DEBUG:boto:Path: /jdpiak-75/obj124 2022-09-12T20:14:21.052 DEBUG:boto:Data: 2022-09-12T20:14:21.052 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.053 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.053 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.054 DEBUG:boto:Params: {} 2022-09-12T20:14:21.055 DEBUG:boto:Token: None 2022-09-12T20:14:21.055 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj124 2022-09-12T20:14:21.056 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OzIG93xaO5pqAvNYjMm+YAGUicM= 2022-09-12T20:14:21.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OzIG93xaO5pqAvNYjMm+YAGUicM=', 'Content-Length': '0'} 2022-09-12T20:14:21.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f715b0ac85b126f1-00631f931d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-12T20:14:21.061 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:21.061 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:21.062 DEBUG:boto:Method: GET 2022-09-12T20:14:21.063 DEBUG:boto:Path: /jdpiak-75/obj124 2022-09-12T20:14:21.064 DEBUG:boto:Data: 2022-09-12T20:14:21.065 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.065 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.066 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.067 DEBUG:boto:Params: {} 2022-09-12T20:14:21.068 DEBUG:boto:Token: None 2022-09-12T20:14:21.069 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj124 2022-09-12T20:14:21.070 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ihpAjYSMVLRLZrMilpb/DRL1rXQ= 2022-09-12T20:14:21.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ihpAjYSMVLRLZrMilpb/DRL1rXQ=', 'Content-Length': '0'} 2022-09-12T20:14:21.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6a7a237747a02ad-00631f931d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.114 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:14:21.115 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:14:21.116 DEBUG:boto:Method: GET 2022-09-12T20:14:21.116 DEBUG:boto:Path: /jdpiak-75/obj124 2022-09-12T20:14:21.117 DEBUG:boto:Data: 2022-09-12T20:14:21.118 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.119 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.120 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.121 DEBUG:boto:Params: {} 2022-09-12T20:14:21.121 DEBUG:boto:Token: None 2022-09-12T20:14:21.122 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj124 2022-09-12T20:14:21.123 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ihpAjYSMVLRLZrMilpb/DRL1rXQ= 2022-09-12T20:14:21.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ihpAjYSMVLRLZrMilpb/DRL1rXQ=', 'Content-Length': '0'} 2022-09-12T20:14:21.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b3ebc677a8cd09b-00631f931d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.127 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:21.127 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:21.128 DEBUG:boto:path=/jdpiak-75/?prefix=obj124 2022-09-12T20:14:21.129 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj124 2022-09-12T20:14:21.130 DEBUG:boto:Method: GET 2022-09-12T20:14:21.130 DEBUG:boto:Path: /jdpiak-75/?prefix=obj124 2022-09-12T20:14:21.131 DEBUG:boto:Data: 2022-09-12T20:14:21.132 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.133 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.134 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.135 DEBUG:boto:Params: {} 2022-09-12T20:14:21.136 DEBUG:boto:Token: None 2022-09-12T20:14:21.137 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/ 2022-09-12T20:14:21.137 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg= 2022-09-12T20:14:21.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg=', 'Content-Length': '0'} 2022-09-12T20:14:21.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043fc171526afb766-00631f931d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.142 DEBUG:boto:b'jdpiak-75obj1241000falseobj1242022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:21.143 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:21.144 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:21.144 DEBUG:boto:path=/jdpiak-75/?prefix=obj124 2022-09-12T20:14:21.145 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj124 2022-09-12T20:14:21.145 DEBUG:boto:Method: GET 2022-09-12T20:14:21.146 DEBUG:boto:Path: /jdpiak-75/?prefix=obj124 2022-09-12T20:14:21.147 DEBUG:boto:Data: 2022-09-12T20:14:21.148 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.148 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.149 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.150 DEBUG:boto:Params: {} 2022-09-12T20:14:21.150 DEBUG:boto:Token: None 2022-09-12T20:14:21.151 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/ 2022-09-12T20:14:21.152 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg= 2022-09-12T20:14:21.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg=', 'Content-Length': '0'} 2022-09-12T20:14:21.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e7970d1f3d6739c-00631f931d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.156 DEBUG:boto:b'jdpiak-75obj1241000falseobj1242022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:21.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:14:21.158 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:21.158 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:21.159 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:21.160 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:21.160 DEBUG:boto:Method: GET 2022-09-12T20:14:21.161 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:21.162 DEBUG:boto:Data: 2022-09-12T20:14:21.162 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.163 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.164 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.164 DEBUG:boto:Params: {} 2022-09-12T20:14:21.165 DEBUG:boto:Token: None 2022-09-12T20:14:21.166 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:21.166 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2KaZr0z/XK4I4m3WyUaf3mlFhCs= 2022-09-12T20:14:21.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2KaZr0z/XK4I4m3WyUaf3mlFhCs=', 'Content-Length': '0'} 2022-09-12T20:14:21.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4af7454c3428a38-00631f931d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.171 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:21.171 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:21.172 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:21.172 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:21.173 DEBUG:boto:Method: GET 2022-09-12T20:14:21.174 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:21.174 DEBUG:boto:Data: 2022-09-12T20:14:21.175 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.175 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.176 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.177 DEBUG:boto:Params: {} 2022-09-12T20:14:21.177 DEBUG:boto:Token: None 2022-09-12T20:14:21.178 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:21.179 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2KaZr0z/XK4I4m3WyUaf3mlFhCs= 2022-09-12T20:14:21.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2KaZr0z/XK4I4m3WyUaf3mlFhCs=', 'Content-Length': '0'} 2022-09-12T20:14:21.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009044594fad77df00-00631f931d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.222 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:21.223 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:21.223 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:21.224 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:21.225 DEBUG:boto:Method: HEAD 2022-09-12T20:14:21.225 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:21.226 DEBUG:boto:Data: 2022-09-12T20:14:21.227 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.228 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.229 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.230 DEBUG:boto:Params: {} 2022-09-12T20:14:21.231 DEBUG:boto:Token: None 2022-09-12T20:14:21.232 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:21.233 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LeV8/ZWXapEHE6yvmItt0Yp3vKA= 2022-09-12T20:14:21.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LeV8/ZWXapEHE6yvmItt0Yp3vKA=', 'Content-Length': '0'} 2022-09-12T20:14:21.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004374c79a41e6cc4d-00631f931d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.239 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:21.240 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:21.241 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:21.242 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:21.243 DEBUG:boto:Method: HEAD 2022-09-12T20:14:21.244 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:21.245 DEBUG:boto:Data: 2022-09-12T20:14:21.246 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.247 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.247 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.248 DEBUG:boto:Params: {} 2022-09-12T20:14:21.249 DEBUG:boto:Token: None 2022-09-12T20:14:21.250 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:21.252 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LeV8/ZWXapEHE6yvmItt0Yp3vKA= 2022-09-12T20:14:21.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LeV8/ZWXapEHE6yvmItt0Yp3vKA=', 'Content-Length': '0'} 2022-09-12T20:14:21.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003769a81f959373bd-00631f931d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:14:21.258 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:21.259 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:21.259 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:21.260 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:21.261 DEBUG:boto:Method: GET 2022-09-12T20:14:21.262 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:21.263 DEBUG:boto:Data: 2022-09-12T20:14:21.264 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.265 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.266 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.267 DEBUG:boto:Params: {} 2022-09-12T20:14:21.268 DEBUG:boto:Token: None 2022-09-12T20:14:21.269 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:21.270 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2KaZr0z/XK4I4m3WyUaf3mlFhCs= 2022-09-12T20:14:21.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2KaZr0z/XK4I4m3WyUaf3mlFhCs=', 'Content-Length': '0'} 2022-09-12T20:14:21.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e3b0bf5f432c4f4-00631f931d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.274 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:21.275 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:21.276 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:21.277 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:14:21.278 DEBUG:boto:Method: GET 2022-09-12T20:14:21.279 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:21.280 DEBUG:boto:Data: 2022-09-12T20:14:21.281 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.282 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.282 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.283 DEBUG:boto:Params: {} 2022-09-12T20:14:21.284 DEBUG:boto:Token: None 2022-09-12T20:14:21.285 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:14:21.286 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2KaZr0z/XK4I4m3WyUaf3mlFhCs= 2022-09-12T20:14:21.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2KaZr0z/XK4I4m3WyUaf3mlFhCs=', 'Content-Length': '0'} 2022-09-12T20:14:21.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ea1bee8df74347c-00631f931d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.291 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:21.292 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:21.293 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:14:21.294 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:14:21.295 DEBUG:boto:Method: GET 2022-09-12T20:14:21.296 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:14:21.296 DEBUG:boto:Data: 2022-09-12T20:14:21.297 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.297 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.298 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.298 DEBUG:boto:Params: {} 2022-09-12T20:14:21.299 DEBUG:boto:Token: None 2022-09-12T20:14:21.300 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/ 2022-09-12T20:14:21.300 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg= 2022-09-12T20:14:21.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg=', 'Content-Length': '0'} 2022-09-12T20:14:21.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6b1edc83c1fd294-00631f931d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.304 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:21.305 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:21.306 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:21.306 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:14:21.307 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:14:21.307 DEBUG:boto:Method: GET 2022-09-12T20:14:21.308 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:14:21.308 DEBUG:boto:Data: 2022-09-12T20:14:21.309 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.309 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.310 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.310 DEBUG:boto:Params: {} 2022-09-12T20:14:21.311 DEBUG:boto:Token: None 2022-09-12T20:14:21.311 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/ 2022-09-12T20:14:21.312 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg= 2022-09-12T20:14:21.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg=', 'Content-Length': '0'} 2022-09-12T20:14:21.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006feab5f2f4f5ac48-00631f931d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.316 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:21.317 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:21.317 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:21.318 DEBUG:boto:Method: HEAD 2022-09-12T20:14:21.318 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:14:21.319 DEBUG:boto:Data: 2022-09-12T20:14:21.319 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.320 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.320 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.321 DEBUG:boto:Params: {} 2022-09-12T20:14:21.321 DEBUG:boto:Token: None 2022-09-12T20:14:21.322 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj145 2022-09-12T20:14:21.323 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WPLth5JlOwCnLJ6+z2+O0B+Hl0A= 2022-09-12T20:14:21.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WPLth5JlOwCnLJ6+z2+O0B+Hl0A=', 'Content-Length': '0'} 2022-09-12T20:14:21.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee555aff9240d2f5-00631f931d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.326 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:21.326 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:21.327 DEBUG:boto:Method: HEAD 2022-09-12T20:14:21.327 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:14:21.328 DEBUG:boto:Data: 2022-09-12T20:14:21.328 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.329 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.329 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.330 DEBUG:boto:Params: {} 2022-09-12T20:14:21.330 DEBUG:boto:Token: None 2022-09-12T20:14:21.331 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj145 2022-09-12T20:14:21.331 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WPLth5JlOwCnLJ6+z2+O0B+Hl0A= 2022-09-12T20:14:21.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WPLth5JlOwCnLJ6+z2+O0B+Hl0A=', 'Content-Length': '0'} 2022-09-12T20:14:21.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057ae3ec4027dd130-00631f931d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:14:21.335 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:21.336 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:21.336 DEBUG:boto:Method: GET 2022-09-12T20:14:21.337 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:14:21.337 DEBUG:boto:Data: 2022-09-12T20:14:21.338 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.338 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.339 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.339 DEBUG:boto:Params: {} 2022-09-12T20:14:21.340 DEBUG:boto:Token: None 2022-09-12T20:14:21.341 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj145 2022-09-12T20:14:21.341 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sVxuLxxIsFNEXKbvLLsZi+KJAWQ= 2022-09-12T20:14:21.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sVxuLxxIsFNEXKbvLLsZi+KJAWQ=', 'Content-Length': '0'} 2022-09-12T20:14:21.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000963d8b886c33949b-00631f931d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.344 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:14:21.345 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:14:21.345 DEBUG:boto:Method: GET 2022-09-12T20:14:21.346 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:14:21.347 DEBUG:boto:Data: 2022-09-12T20:14:21.347 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.348 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.348 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.349 DEBUG:boto:Params: {} 2022-09-12T20:14:21.349 DEBUG:boto:Token: None 2022-09-12T20:14:21.350 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj145 2022-09-12T20:14:21.350 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sVxuLxxIsFNEXKbvLLsZi+KJAWQ= 2022-09-12T20:14:21.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sVxuLxxIsFNEXKbvLLsZi+KJAWQ=', 'Content-Length': '0'} 2022-09-12T20:14:21.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e7175a39c00b759-00631f931d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.394 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:21.395 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:21.395 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:14:21.396 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:14:21.396 DEBUG:boto:Method: GET 2022-09-12T20:14:21.397 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:14:21.397 DEBUG:boto:Data: 2022-09-12T20:14:21.398 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.398 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.399 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.399 DEBUG:boto:Params: {} 2022-09-12T20:14:21.400 DEBUG:boto:Token: None 2022-09-12T20:14:21.401 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/ 2022-09-12T20:14:21.401 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg= 2022-09-12T20:14:21.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg=', 'Content-Length': '0'} 2022-09-12T20:14:21.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5eb4e6cdf28e83d-00631f931d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.405 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:21.406 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:21.407 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:21.407 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:14:21.408 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:14:21.408 DEBUG:boto:Method: GET 2022-09-12T20:14:21.409 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:14:21.409 DEBUG:boto:Data: 2022-09-12T20:14:21.410 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.410 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.411 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.411 DEBUG:boto:Params: {} 2022-09-12T20:14:21.412 DEBUG:boto:Token: None 2022-09-12T20:14:21.412 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/ 2022-09-12T20:14:21.413 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg= 2022-09-12T20:14:21.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg=', 'Content-Length': '0'} 2022-09-12T20:14:21.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045b141f259fd3408-00631f931d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.417 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:21.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-12T20:14:21.418 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:21.419 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:21.419 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:21.420 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:21.420 DEBUG:boto:Method: GET 2022-09-12T20:14:21.421 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:21.421 DEBUG:boto:Data: 2022-09-12T20:14:21.422 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.422 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.423 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.423 DEBUG:boto:Params: {} 2022-09-12T20:14:21.424 DEBUG:boto:Token: None 2022-09-12T20:14:21.425 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:21.425 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D6dOczgekXFtMZK5aRlfyPJQfmk= 2022-09-12T20:14:21.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D6dOczgekXFtMZK5aRlfyPJQfmk=', 'Content-Length': '0'} 2022-09-12T20:14:21.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9c9bdaff064b0a2-00631f931d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.429 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:21.429 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:21.430 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:21.430 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:21.431 DEBUG:boto:Method: GET 2022-09-12T20:14:21.432 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:21.432 DEBUG:boto:Data: 2022-09-12T20:14:21.433 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.433 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.434 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.434 DEBUG:boto:Params: {} 2022-09-12T20:14:21.435 DEBUG:boto:Token: None 2022-09-12T20:14:21.435 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:21.436 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D6dOczgekXFtMZK5aRlfyPJQfmk= 2022-09-12T20:14:21.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D6dOczgekXFtMZK5aRlfyPJQfmk=', 'Content-Length': '0'} 2022-09-12T20:14:21.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca1cd7982eac79bd-00631f931d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.440 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:21.440 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:21.441 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:21.441 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:21.442 DEBUG:boto:Method: HEAD 2022-09-12T20:14:21.442 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:21.443 DEBUG:boto:Data: 2022-09-12T20:14:21.443 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.444 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.444 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.445 DEBUG:boto:Params: {} 2022-09-12T20:14:21.445 DEBUG:boto:Token: None 2022-09-12T20:14:21.446 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:21.446 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:toz7yEtFJ4MmV6+RoEz+k86rTOs= 2022-09-12T20:14:21.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:toz7yEtFJ4MmV6+RoEz+k86rTOs=', 'Content-Length': '0'} 2022-09-12T20:14:21.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dac903041aa99db5-00631f931d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.450 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:21.450 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:21.451 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:21.451 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:21.452 DEBUG:boto:Method: HEAD 2022-09-12T20:14:21.452 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:21.453 DEBUG:boto:Data: 2022-09-12T20:14:21.453 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.454 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.454 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.455 DEBUG:boto:Params: {} 2022-09-12T20:14:21.455 DEBUG:boto:Token: None 2022-09-12T20:14:21.456 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:21.457 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:toz7yEtFJ4MmV6+RoEz+k86rTOs= 2022-09-12T20:14:21.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:toz7yEtFJ4MmV6+RoEz+k86rTOs=', 'Content-Length': '0'} 2022-09-12T20:14:21.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074d7e2cf3f797e93-00631f931d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-12T20:14:21.460 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:21.461 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:21.461 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:21.462 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:21.462 DEBUG:boto:Method: GET 2022-09-12T20:14:21.463 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:21.464 DEBUG:boto:Data: 2022-09-12T20:14:21.464 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.465 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.465 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.466 DEBUG:boto:Params: {} 2022-09-12T20:14:21.466 DEBUG:boto:Token: None 2022-09-12T20:14:21.467 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:21.468 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D6dOczgekXFtMZK5aRlfyPJQfmk= 2022-09-12T20:14:21.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D6dOczgekXFtMZK5aRlfyPJQfmk=', 'Content-Length': '0'} 2022-09-12T20:14:21.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004426b01b7f4e46f3-00631f931d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.471 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:21.472 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:21.472 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:21.473 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:14:21.473 DEBUG:boto:Method: GET 2022-09-12T20:14:21.474 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:21.474 DEBUG:boto:Data: 2022-09-12T20:14:21.475 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.475 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.476 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.476 DEBUG:boto:Params: {} 2022-09-12T20:14:21.477 DEBUG:boto:Token: None 2022-09-12T20:14:21.477 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:14:21.478 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D6dOczgekXFtMZK5aRlfyPJQfmk= 2022-09-12T20:14:21.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D6dOczgekXFtMZK5aRlfyPJQfmk=', 'Content-Length': '0'} 2022-09-12T20:14:21.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8926d9bc86d4d9b-00631f931d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.482 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:21.482 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:21.483 DEBUG:boto:path=/jdpiak-75/?prefix=obj155 2022-09-12T20:14:21.483 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj155 2022-09-12T20:14:21.484 DEBUG:boto:Method: GET 2022-09-12T20:14:21.484 DEBUG:boto:Path: /jdpiak-75/?prefix=obj155 2022-09-12T20:14:21.485 DEBUG:boto:Data: 2022-09-12T20:14:21.485 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.486 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.487 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.487 DEBUG:boto:Params: {} 2022-09-12T20:14:21.488 DEBUG:boto:Token: None 2022-09-12T20:14:21.488 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/ 2022-09-12T20:14:21.489 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg= 2022-09-12T20:14:21.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg=', 'Content-Length': '0'} 2022-09-12T20:14:21.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b407cc6807e548a-00631f931d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.534 DEBUG:boto:b'jdpiak-75obj1551000falseobj1552022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:21.535 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:21.536 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:21.536 DEBUG:boto:path=/jdpiak-75/?prefix=obj155 2022-09-12T20:14:21.537 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj155 2022-09-12T20:14:21.537 DEBUG:boto:Method: GET 2022-09-12T20:14:21.538 DEBUG:boto:Path: /jdpiak-75/?prefix=obj155 2022-09-12T20:14:21.538 DEBUG:boto:Data: 2022-09-12T20:14:21.539 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.539 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.540 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.541 DEBUG:boto:Params: {} 2022-09-12T20:14:21.541 DEBUG:boto:Token: None 2022-09-12T20:14:21.542 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/ 2022-09-12T20:14:21.542 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg= 2022-09-12T20:14:21.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg=', 'Content-Length': '0'} 2022-09-12T20:14:21.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008962783d1a000f7-00631f931d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.547 DEBUG:boto:b'jdpiak-75obj1551000falseobj1552022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:21.547 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:21.548 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:21.549 DEBUG:boto:Method: HEAD 2022-09-12T20:14:21.549 DEBUG:boto:Path: /jdpiak-75/obj155 2022-09-12T20:14:21.550 DEBUG:boto:Data: 2022-09-12T20:14:21.550 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.551 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.551 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.552 DEBUG:boto:Params: {} 2022-09-12T20:14:21.552 DEBUG:boto:Token: None 2022-09-12T20:14:21.553 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj155 2022-09-12T20:14:21.553 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0jk32gNHXxX9rWw4hA1w/3l+nJ8= 2022-09-12T20:14:21.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0jk32gNHXxX9rWw4hA1w/3l+nJ8=', 'Content-Length': '0'} 2022-09-12T20:14:21.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a7f3be5e46fb7f7-00631f931d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.557 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:21.557 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:21.558 DEBUG:boto:Method: HEAD 2022-09-12T20:14:21.558 DEBUG:boto:Path: /jdpiak-75/obj155 2022-09-12T20:14:21.559 DEBUG:boto:Data: 2022-09-12T20:14:21.559 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.560 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.560 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.561 DEBUG:boto:Params: {} 2022-09-12T20:14:21.561 DEBUG:boto:Token: None 2022-09-12T20:14:21.562 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj155 2022-09-12T20:14:21.563 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0jk32gNHXxX9rWw4hA1w/3l+nJ8= 2022-09-12T20:14:21.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0jk32gNHXxX9rWw4hA1w/3l+nJ8=', 'Content-Length': '0'} 2022-09-12T20:14:21.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6f0a2a864cefd3a-00631f931d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-12T20:14:21.566 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:21.567 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:21.568 DEBUG:boto:Method: GET 2022-09-12T20:14:21.568 DEBUG:boto:Path: /jdpiak-75/obj155 2022-09-12T20:14:21.569 DEBUG:boto:Data: 2022-09-12T20:14:21.569 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.570 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.570 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.571 DEBUG:boto:Params: {} 2022-09-12T20:14:21.571 DEBUG:boto:Token: None 2022-09-12T20:14:21.572 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj155 2022-09-12T20:14:21.573 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H1wX7q3YolzS3zNSarmiDCd+cMk= 2022-09-12T20:14:21.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H1wX7q3YolzS3zNSarmiDCd+cMk=', 'Content-Length': '0'} 2022-09-12T20:14:21.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a80904572fd67fb2-00631f931d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.576 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:14:21.577 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:14:21.577 DEBUG:boto:Method: GET 2022-09-12T20:14:21.578 DEBUG:boto:Path: /jdpiak-75/obj155 2022-09-12T20:14:21.578 DEBUG:boto:Data: 2022-09-12T20:14:21.579 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.579 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.580 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.580 DEBUG:boto:Params: {} 2022-09-12T20:14:21.581 DEBUG:boto:Token: None 2022-09-12T20:14:21.581 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj155 2022-09-12T20:14:21.582 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H1wX7q3YolzS3zNSarmiDCd+cMk= 2022-09-12T20:14:21.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H1wX7q3YolzS3zNSarmiDCd+cMk=', 'Content-Length': '0'} 2022-09-12T20:14:21.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001143d20a3216283f-00631f931d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.626 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:21.627 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:21.628 DEBUG:boto:path=/jdpiak-75/?prefix=obj155 2022-09-12T20:14:21.629 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj155 2022-09-12T20:14:21.630 DEBUG:boto:Method: GET 2022-09-12T20:14:21.631 DEBUG:boto:Path: /jdpiak-75/?prefix=obj155 2022-09-12T20:14:21.632 DEBUG:boto:Data: 2022-09-12T20:14:21.633 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.633 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.634 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.635 DEBUG:boto:Params: {} 2022-09-12T20:14:21.637 DEBUG:boto:Token: None 2022-09-12T20:14:21.638 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/ 2022-09-12T20:14:21.639 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg= 2022-09-12T20:14:21.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg=', 'Content-Length': '0'} 2022-09-12T20:14:21.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3873e3079ee870a-00631f931d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.645 DEBUG:boto:b'jdpiak-75obj1551000falseobj1552022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:21.646 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:21.647 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:21.648 DEBUG:boto:path=/jdpiak-75/?prefix=obj155 2022-09-12T20:14:21.649 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj155 2022-09-12T20:14:21.649 DEBUG:boto:Method: GET 2022-09-12T20:14:21.650 DEBUG:boto:Path: /jdpiak-75/?prefix=obj155 2022-09-12T20:14:21.651 DEBUG:boto:Data: 2022-09-12T20:14:21.651 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.652 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.653 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.653 DEBUG:boto:Params: {} 2022-09-12T20:14:21.654 DEBUG:boto:Token: None 2022-09-12T20:14:21.654 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/ 2022-09-12T20:14:21.655 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg= 2022-09-12T20:14:21.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg=', 'Content-Length': '0'} 2022-09-12T20:14:21.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d43a578ab3acd95c-00631f931d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.659 DEBUG:boto:b'jdpiak-75obj1551000falseobj1552022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:21.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:14:21.660 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:21.667 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:21.668 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:21.668 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:21.669 DEBUG:boto:Method: GET 2022-09-12T20:14:21.670 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:21.670 DEBUG:boto:Data: 2022-09-12T20:14:21.671 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.671 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.672 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.672 DEBUG:boto:Params: {} 2022-09-12T20:14:21.673 DEBUG:boto:Token: None 2022-09-12T20:14:21.674 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:21.674 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7UVZvsOFweA1SrPqmf9/8QHRKu0= 2022-09-12T20:14:21.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7UVZvsOFweA1SrPqmf9/8QHRKu0=', 'Content-Length': '0'} 2022-09-12T20:14:21.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f202ad61d3830eb4-00631f931d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.678 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:21.679 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:21.680 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:21.680 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:21.681 DEBUG:boto:Method: GET 2022-09-12T20:14:21.681 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:21.682 DEBUG:boto:Data: 2022-09-12T20:14:21.682 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.683 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.683 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.684 DEBUG:boto:Params: {} 2022-09-12T20:14:21.684 DEBUG:boto:Token: None 2022-09-12T20:14:21.685 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:21.686 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7UVZvsOFweA1SrPqmf9/8QHRKu0= 2022-09-12T20:14:21.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7UVZvsOFweA1SrPqmf9/8QHRKu0=', 'Content-Length': '0'} 2022-09-12T20:14:21.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7cb957eae5cb1f6-00631f931d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.690 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:21.690 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:21.691 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:21.691 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:21.692 DEBUG:boto:Method: HEAD 2022-09-12T20:14:21.692 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:21.693 DEBUG:boto:Data: 2022-09-12T20:14:21.693 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.694 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.694 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.695 DEBUG:boto:Params: {} 2022-09-12T20:14:21.695 DEBUG:boto:Token: None 2022-09-12T20:14:21.696 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:21.697 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E+Nh0pmhkMbb00DlauwAQ81rbZU= 2022-09-12T20:14:21.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E+Nh0pmhkMbb00DlauwAQ81rbZU=', 'Content-Length': '0'} 2022-09-12T20:14:21.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f905252f9dab434-00631f931d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.700 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:21.701 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:21.702 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:21.702 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:21.703 DEBUG:boto:Method: HEAD 2022-09-12T20:14:21.703 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:21.704 DEBUG:boto:Data: 2022-09-12T20:14:21.705 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.706 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.707 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.707 DEBUG:boto:Params: {} 2022-09-12T20:14:21.708 DEBUG:boto:Token: None 2022-09-12T20:14:21.709 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:21.709 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E+Nh0pmhkMbb00DlauwAQ81rbZU= 2022-09-12T20:14:21.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E+Nh0pmhkMbb00DlauwAQ81rbZU=', 'Content-Length': '0'} 2022-09-12T20:14:21.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb567e5389a89fb2-00631f931d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:14:21.714 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:21.715 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:21.715 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:21.716 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:21.716 DEBUG:boto:Method: GET 2022-09-12T20:14:21.717 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:21.717 DEBUG:boto:Data: 2022-09-12T20:14:21.718 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.719 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.719 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.720 DEBUG:boto:Params: {} 2022-09-12T20:14:21.720 DEBUG:boto:Token: None 2022-09-12T20:14:21.721 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:21.722 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7UVZvsOFweA1SrPqmf9/8QHRKu0= 2022-09-12T20:14:21.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7UVZvsOFweA1SrPqmf9/8QHRKu0=', 'Content-Length': '0'} 2022-09-12T20:14:21.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fcf72c7b9b39608-00631f931d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.726 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:21.726 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:21.727 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:21.727 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:14:21.728 DEBUG:boto:Method: GET 2022-09-12T20:14:21.728 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:21.729 DEBUG:boto:Data: 2022-09-12T20:14:21.729 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.730 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.730 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.731 DEBUG:boto:Params: {} 2022-09-12T20:14:21.732 DEBUG:boto:Token: None 2022-09-12T20:14:21.732 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:14:21.733 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7UVZvsOFweA1SrPqmf9/8QHRKu0= 2022-09-12T20:14:21.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7UVZvsOFweA1SrPqmf9/8QHRKu0=', 'Content-Length': '0'} 2022-09-12T20:14:21.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044f9ba22ba70c2d1-00631f931d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.737 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:21.737 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:21.738 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:14:21.738 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:14:21.739 DEBUG:boto:Method: GET 2022-09-12T20:14:21.739 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:14:21.740 DEBUG:boto:Data: 2022-09-12T20:14:21.740 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.741 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.741 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.742 DEBUG:boto:Params: {} 2022-09-12T20:14:21.743 DEBUG:boto:Token: None 2022-09-12T20:14:21.743 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/ 2022-09-12T20:14:21.744 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg= 2022-09-12T20:14:21.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg=', 'Content-Length': '0'} 2022-09-12T20:14:21.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093ee7407c8500598-00631f931d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.748 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:21.749 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:21.750 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:21.750 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:14:21.751 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:14:21.751 DEBUG:boto:Method: GET 2022-09-12T20:14:21.752 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:14:21.752 DEBUG:boto:Data: 2022-09-12T20:14:21.753 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.753 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.754 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.754 DEBUG:boto:Params: {} 2022-09-12T20:14:21.755 DEBUG:boto:Token: None 2022-09-12T20:14:21.755 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/ 2022-09-12T20:14:21.756 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg= 2022-09-12T20:14:21.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg=', 'Content-Length': '0'} 2022-09-12T20:14:21.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ba4bbaba9a74faa-00631f931d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.760 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:21.761 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:21.761 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:21.762 DEBUG:boto:Method: HEAD 2022-09-12T20:14:21.762 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:14:21.763 DEBUG:boto:Data: 2022-09-12T20:14:21.764 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.764 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.764 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.765 DEBUG:boto:Params: {} 2022-09-12T20:14:21.766 DEBUG:boto:Token: None 2022-09-12T20:14:21.766 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj17 2022-09-12T20:14:21.767 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+bkcSEo0jznnJKTPNEzj6jSNhhk= 2022-09-12T20:14:21.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+bkcSEo0jznnJKTPNEzj6jSNhhk=', 'Content-Length': '0'} 2022-09-12T20:14:21.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014e34701d708e3a7-00631f931d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.770 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:21.771 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:21.771 DEBUG:boto:Method: HEAD 2022-09-12T20:14:21.772 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:14:21.772 DEBUG:boto:Data: 2022-09-12T20:14:21.773 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.773 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.774 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.774 DEBUG:boto:Params: {} 2022-09-12T20:14:21.775 DEBUG:boto:Token: None 2022-09-12T20:14:21.775 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj17 2022-09-12T20:14:21.776 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+bkcSEo0jznnJKTPNEzj6jSNhhk= 2022-09-12T20:14:21.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+bkcSEo0jznnJKTPNEzj6jSNhhk=', 'Content-Length': '0'} 2022-09-12T20:14:21.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020de5115c3c215b9-00631f931d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:14:21.781 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:21.781 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:21.782 DEBUG:boto:Method: GET 2022-09-12T20:14:21.783 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:14:21.783 DEBUG:boto:Data: 2022-09-12T20:14:21.784 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.785 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.786 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.793 DEBUG:boto:Params: {} 2022-09-12T20:14:21.794 DEBUG:boto:Token: None 2022-09-12T20:14:21.795 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj17 2022-09-12T20:14:21.795 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SG/2NpazwxtWjmypFDaX3jXrNUg= 2022-09-12T20:14:21.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SG/2NpazwxtWjmypFDaX3jXrNUg=', 'Content-Length': '0'} 2022-09-12T20:14:21.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee650a1c1d6506c2-00631f931d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.842 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:14:21.843 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:14:21.843 DEBUG:boto:Method: GET 2022-09-12T20:14:21.844 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:14:21.845 DEBUG:boto:Data: 2022-09-12T20:14:21.845 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.846 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.846 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.847 DEBUG:boto:Params: {} 2022-09-12T20:14:21.848 DEBUG:boto:Token: None 2022-09-12T20:14:21.848 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj17 2022-09-12T20:14:21.849 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SG/2NpazwxtWjmypFDaX3jXrNUg= 2022-09-12T20:14:21.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SG/2NpazwxtWjmypFDaX3jXrNUg=', 'Content-Length': '0'} 2022-09-12T20:14:21.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016c02c366a2d043d-00631f931d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.853 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:21.853 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:21.854 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:14:21.854 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:14:21.855 DEBUG:boto:Method: GET 2022-09-12T20:14:21.855 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:14:21.856 DEBUG:boto:Data: 2022-09-12T20:14:21.856 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.857 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.857 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.858 DEBUG:boto:Params: {} 2022-09-12T20:14:21.859 DEBUG:boto:Token: None 2022-09-12T20:14:21.859 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/ 2022-09-12T20:14:21.860 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg= 2022-09-12T20:14:21.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg=', 'Content-Length': '0'} 2022-09-12T20:14:21.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009948c3d90b43107f-00631f931d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.864 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:21.865 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:21.865 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:21.866 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:14:21.866 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:14:21.867 DEBUG:boto:Method: GET 2022-09-12T20:14:21.867 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:14:21.868 DEBUG:boto:Data: 2022-09-12T20:14:21.868 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.869 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.869 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.870 DEBUG:boto:Params: {} 2022-09-12T20:14:21.871 DEBUG:boto:Token: None 2022-09-12T20:14:21.871 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/ 2022-09-12T20:14:21.872 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg= 2022-09-12T20:14:21.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg=', 'Content-Length': '0'} 2022-09-12T20:14:21.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079677b118be54c54-00631f931d-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.876 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:21.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:14:21.877 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:21.878 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:21.878 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:21.879 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:21.879 DEBUG:boto:Method: GET 2022-09-12T20:14:21.880 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:21.880 DEBUG:boto:Data: 2022-09-12T20:14:21.881 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.881 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.882 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.883 DEBUG:boto:Params: {} 2022-09-12T20:14:21.883 DEBUG:boto:Token: None 2022-09-12T20:14:21.884 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:21.884 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vNgBNMPiCJM9PRGFZh4Mghh+NWE= 2022-09-12T20:14:21.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vNgBNMPiCJM9PRGFZh4Mghh+NWE=', 'Content-Length': '0'} 2022-09-12T20:14:21.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee949740005d3f5b-00631f931d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.888 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:21.888 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:21.889 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:21.889 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:21.890 DEBUG:boto:Method: GET 2022-09-12T20:14:21.890 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:21.891 DEBUG:boto:Data: 2022-09-12T20:14:21.892 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.892 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.893 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.893 DEBUG:boto:Params: {} 2022-09-12T20:14:21.894 DEBUG:boto:Token: None 2022-09-12T20:14:21.894 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:21.895 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vNgBNMPiCJM9PRGFZh4Mghh+NWE= 2022-09-12T20:14:21.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vNgBNMPiCJM9PRGFZh4Mghh+NWE=', 'Content-Length': '0'} 2022-09-12T20:14:21.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e63eaa249f319a5b-00631f931d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.899 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:21.899 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:21.900 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:21.900 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:21.901 DEBUG:boto:Method: HEAD 2022-09-12T20:14:21.901 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:21.902 DEBUG:boto:Data: 2022-09-12T20:14:21.902 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.903 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.903 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.904 DEBUG:boto:Params: {} 2022-09-12T20:14:21.904 DEBUG:boto:Token: None 2022-09-12T20:14:21.905 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:21.906 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Tv2ouqEqP970H3AviRanlyBRbTg= 2022-09-12T20:14:21.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Tv2ouqEqP970H3AviRanlyBRbTg=', 'Content-Length': '0'} 2022-09-12T20:14:21.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e5661c3b90051e5-00631f931d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.909 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:21.910 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:21.910 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:21.911 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:21.911 DEBUG:boto:Method: HEAD 2022-09-12T20:14:21.912 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:21.912 DEBUG:boto:Data: 2022-09-12T20:14:21.913 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.913 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.914 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.914 DEBUG:boto:Params: {} 2022-09-12T20:14:21.915 DEBUG:boto:Token: None 2022-09-12T20:14:21.915 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:21.916 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Tv2ouqEqP970H3AviRanlyBRbTg= 2022-09-12T20:14:21.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Tv2ouqEqP970H3AviRanlyBRbTg=', 'Content-Length': '0'} 2022-09-12T20:14:21.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a7a55c06140009b-00631f931d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:14:21.920 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:21.920 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:21.921 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:21.921 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:21.922 DEBUG:boto:Method: GET 2022-09-12T20:14:21.922 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:21.923 DEBUG:boto:Data: 2022-09-12T20:14:21.924 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.924 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.925 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.925 DEBUG:boto:Params: {} 2022-09-12T20:14:21.926 DEBUG:boto:Token: None 2022-09-12T20:14:21.927 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:21.927 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vNgBNMPiCJM9PRGFZh4Mghh+NWE= 2022-09-12T20:14:21.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vNgBNMPiCJM9PRGFZh4Mghh+NWE=', 'Content-Length': '0'} 2022-09-12T20:14:21.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc22d725762dd3a1-00631f931d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.931 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:21.931 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:21.932 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:21.932 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:14:21.933 DEBUG:boto:Method: GET 2022-09-12T20:14:21.933 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:21.934 DEBUG:boto:Data: 2022-09-12T20:14:21.934 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.935 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:21.936 DEBUG:boto:Port: 8000 2022-09-12T20:14:21.936 DEBUG:boto:Params: {} 2022-09-12T20:14:21.937 DEBUG:boto:Token: None 2022-09-12T20:14:21.937 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:14:21.938 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vNgBNMPiCJM9PRGFZh4Mghh+NWE= 2022-09-12T20:14:21.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vNgBNMPiCJM9PRGFZh4Mghh+NWE=', 'Content-Length': '0'} 2022-09-12T20:14:21.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e612fda6394f50f-00631f931d-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.982 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:21.983 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:21.984 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:14:21.984 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:14:21.985 DEBUG:boto:Method: GET 2022-09-12T20:14:21.986 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:14:21.987 DEBUG:boto:Data: 2022-09-12T20:14:21.988 DEBUG:boto:Headers: {} 2022-09-12T20:14:21.988 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:21.989 DEBUG:boto:Port: 8001 2022-09-12T20:14:21.989 DEBUG:boto:Params: {} 2022-09-12T20:14:21.990 DEBUG:boto:Token: None 2022-09-12T20:14:21.990 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:21 GMT /jdpiak-75/ 2022-09-12T20:14:21.991 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg= 2022-09-12T20:14:21.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6MXkwIFKSfyIVgB1GnromE73zmg=', 'Content-Length': '0'} 2022-09-12T20:14:21.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dda7edd9015bd73-00631f931d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:21.996 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:21.997 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:21.998 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:21.999 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:14:22.000 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:14:22.000 DEBUG:boto:Method: GET 2022-09-12T20:14:22.001 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:14:22.001 DEBUG:boto:Data: 2022-09-12T20:14:22.002 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.003 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.003 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.004 DEBUG:boto:Params: {} 2022-09-12T20:14:22.005 DEBUG:boto:Token: None 2022-09-12T20:14:22.005 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/ 2022-09-12T20:14:22.006 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q= 2022-09-12T20:14:22.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q=', 'Content-Length': '0'} 2022-09-12T20:14:22.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3c6c4a4ae17caa4-00631f931e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.010 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:22.011 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:22.012 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:22.012 DEBUG:boto:Method: HEAD 2022-09-12T20:14:22.013 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:14:22.013 DEBUG:boto:Data: 2022-09-12T20:14:22.014 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.014 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.015 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.015 DEBUG:boto:Params: {} 2022-09-12T20:14:22.016 DEBUG:boto:Token: None 2022-09-12T20:14:22.016 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj19 2022-09-12T20:14:22.017 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QuRAINIDCOO30AaC4MdVyvbts+4= 2022-09-12T20:14:22.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QuRAINIDCOO30AaC4MdVyvbts+4=', 'Content-Length': '0'} 2022-09-12T20:14:22.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000585dc27c85a14eb9-00631f931e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.021 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:22.021 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:22.022 DEBUG:boto:Method: HEAD 2022-09-12T20:14:22.022 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:14:22.023 DEBUG:boto:Data: 2022-09-12T20:14:22.023 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.024 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.024 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.025 DEBUG:boto:Params: {} 2022-09-12T20:14:22.025 DEBUG:boto:Token: None 2022-09-12T20:14:22.026 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj19 2022-09-12T20:14:22.026 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QuRAINIDCOO30AaC4MdVyvbts+4= 2022-09-12T20:14:22.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QuRAINIDCOO30AaC4MdVyvbts+4=', 'Content-Length': '0'} 2022-09-12T20:14:22.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e41078445a514bb9-00631f931e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:14:22.030 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:22.031 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:22.032 DEBUG:boto:Method: GET 2022-09-12T20:14:22.032 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:14:22.033 DEBUG:boto:Data: 2022-09-12T20:14:22.033 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.034 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.034 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.035 DEBUG:boto:Params: {} 2022-09-12T20:14:22.036 DEBUG:boto:Token: None 2022-09-12T20:14:22.036 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj19 2022-09-12T20:14:22.037 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rscxY7FCFcjNOAS1A7DPE9FJHWg= 2022-09-12T20:14:22.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rscxY7FCFcjNOAS1A7DPE9FJHWg=', 'Content-Length': '0'} 2022-09-12T20:14:22.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5eaa55fe87759a9-00631f931e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.082 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:14:22.083 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:14:22.083 DEBUG:boto:Method: GET 2022-09-12T20:14:22.084 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:14:22.084 DEBUG:boto:Data: 2022-09-12T20:14:22.085 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.086 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.086 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.087 DEBUG:boto:Params: {} 2022-09-12T20:14:22.087 DEBUG:boto:Token: None 2022-09-12T20:14:22.088 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj19 2022-09-12T20:14:22.089 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rscxY7FCFcjNOAS1A7DPE9FJHWg= 2022-09-12T20:14:22.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rscxY7FCFcjNOAS1A7DPE9FJHWg=', 'Content-Length': '0'} 2022-09-12T20:14:22.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004a9865098bc4b33-00631f931e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.092 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:22.093 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:22.094 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:14:22.095 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:14:22.096 DEBUG:boto:Method: GET 2022-09-12T20:14:22.097 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:14:22.098 DEBUG:boto:Data: 2022-09-12T20:14:22.098 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.099 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.099 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.100 DEBUG:boto:Params: {} 2022-09-12T20:14:22.100 DEBUG:boto:Token: None 2022-09-12T20:14:22.101 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/ 2022-09-12T20:14:22.102 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q= 2022-09-12T20:14:22.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q=', 'Content-Length': '0'} 2022-09-12T20:14:22.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d579c38f84486170-00631f931e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.106 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:22.107 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:22.108 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:22.109 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:14:22.109 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:14:22.110 DEBUG:boto:Method: GET 2022-09-12T20:14:22.110 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:14:22.111 DEBUG:boto:Data: 2022-09-12T20:14:22.111 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.112 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.112 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.113 DEBUG:boto:Params: {} 2022-09-12T20:14:22.114 DEBUG:boto:Token: None 2022-09-12T20:14:22.115 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/ 2022-09-12T20:14:22.115 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q= 2022-09-12T20:14:22.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q=', 'Content-Length': '0'} 2022-09-12T20:14:22.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c199890387a76ee-00631f931e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.120 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:22.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:14:22.121 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:22.122 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:22.122 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:22.123 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:22.123 DEBUG:boto:Method: GET 2022-09-12T20:14:22.124 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:22.124 DEBUG:boto:Data: 2022-09-12T20:14:22.125 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.125 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.126 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.126 DEBUG:boto:Params: {} 2022-09-12T20:14:22.127 DEBUG:boto:Token: None 2022-09-12T20:14:22.128 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:22.128 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EGEe0FpjzF0y04bUfvyTLJmIxzs= 2022-09-12T20:14:22.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EGEe0FpjzF0y04bUfvyTLJmIxzs=', 'Content-Length': '0'} 2022-09-12T20:14:22.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f7deb3e69c853b3-00631f931e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.132 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:22.132 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:22.133 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:22.133 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:22.134 DEBUG:boto:Method: GET 2022-09-12T20:14:22.135 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:22.135 DEBUG:boto:Data: 2022-09-12T20:14:22.136 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.136 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.137 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.137 DEBUG:boto:Params: {} 2022-09-12T20:14:22.138 DEBUG:boto:Token: None 2022-09-12T20:14:22.139 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:22.139 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EGEe0FpjzF0y04bUfvyTLJmIxzs= 2022-09-12T20:14:22.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EGEe0FpjzF0y04bUfvyTLJmIxzs=', 'Content-Length': '0'} 2022-09-12T20:14:22.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4f14a5206cadc9d-00631f931e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.143 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:22.144 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:22.144 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:22.145 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:22.146 DEBUG:boto:Method: HEAD 2022-09-12T20:14:22.146 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:22.147 DEBUG:boto:Data: 2022-09-12T20:14:22.147 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.148 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.149 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.149 DEBUG:boto:Params: {} 2022-09-12T20:14:22.150 DEBUG:boto:Token: None 2022-09-12T20:14:22.151 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:22.152 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OPv50m9TTc5VmdFOlENPeJ9MYKo= 2022-09-12T20:14:22.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OPv50m9TTc5VmdFOlENPeJ9MYKo=', 'Content-Length': '0'} 2022-09-12T20:14:22.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000206581864e9fa742-00631f931e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.155 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:22.156 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:22.157 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:22.157 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:22.158 DEBUG:boto:Method: HEAD 2022-09-12T20:14:22.159 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:22.159 DEBUG:boto:Data: 2022-09-12T20:14:22.160 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.160 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.160 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.161 DEBUG:boto:Params: {} 2022-09-12T20:14:22.162 DEBUG:boto:Token: None 2022-09-12T20:14:22.162 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:22.163 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OPv50m9TTc5VmdFOlENPeJ9MYKo= 2022-09-12T20:14:22.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OPv50m9TTc5VmdFOlENPeJ9MYKo=', 'Content-Length': '0'} 2022-09-12T20:14:22.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083985844ff372578-00631f931e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:14:22.168 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:22.169 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:22.170 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:22.170 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:22.171 DEBUG:boto:Method: GET 2022-09-12T20:14:22.171 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:22.172 DEBUG:boto:Data: 2022-09-12T20:14:22.172 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.173 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.173 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.174 DEBUG:boto:Params: {} 2022-09-12T20:14:22.174 DEBUG:boto:Token: None 2022-09-12T20:14:22.175 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:22.176 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EGEe0FpjzF0y04bUfvyTLJmIxzs= 2022-09-12T20:14:22.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EGEe0FpjzF0y04bUfvyTLJmIxzs=', 'Content-Length': '0'} 2022-09-12T20:14:22.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f3febec9e348927-00631f931e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.222 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:22.223 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:22.224 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:22.225 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:14:22.226 DEBUG:boto:Method: GET 2022-09-12T20:14:22.227 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:22.227 DEBUG:boto:Data: 2022-09-12T20:14:22.228 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.229 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.230 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.231 DEBUG:boto:Params: {} 2022-09-12T20:14:22.232 DEBUG:boto:Token: None 2022-09-12T20:14:22.233 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:14:22.234 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EGEe0FpjzF0y04bUfvyTLJmIxzs= 2022-09-12T20:14:22.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EGEe0FpjzF0y04bUfvyTLJmIxzs=', 'Content-Length': '0'} 2022-09-12T20:14:22.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f97f4d624374e08-00631f931e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.238 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:22.239 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:22.240 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:14:22.241 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:14:22.242 DEBUG:boto:Method: GET 2022-09-12T20:14:22.243 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:14:22.244 DEBUG:boto:Data: 2022-09-12T20:14:22.245 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.246 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.246 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.247 DEBUG:boto:Params: {} 2022-09-12T20:14:22.248 DEBUG:boto:Token: None 2022-09-12T20:14:22.248 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/ 2022-09-12T20:14:22.249 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q= 2022-09-12T20:14:22.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q=', 'Content-Length': '0'} 2022-09-12T20:14:22.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6318119df631c1f-00631f931e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.253 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:22.255 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:22.255 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:22.256 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:14:22.256 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:14:22.257 DEBUG:boto:Method: GET 2022-09-12T20:14:22.257 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:14:22.258 DEBUG:boto:Data: 2022-09-12T20:14:22.258 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.259 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.260 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.261 DEBUG:boto:Params: {} 2022-09-12T20:14:22.262 DEBUG:boto:Token: None 2022-09-12T20:14:22.263 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/ 2022-09-12T20:14:22.264 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q= 2022-09-12T20:14:22.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q=', 'Content-Length': '0'} 2022-09-12T20:14:22.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a7727eebfe1454b-00631f931e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.268 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:22.268 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:22.269 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:22.269 DEBUG:boto:Method: HEAD 2022-09-12T20:14:22.270 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:14:22.270 DEBUG:boto:Data: 2022-09-12T20:14:22.271 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.271 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.272 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.272 DEBUG:boto:Params: {} 2022-09-12T20:14:22.273 DEBUG:boto:Token: None 2022-09-12T20:14:22.274 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj22 2022-09-12T20:14:22.274 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qf7MPkrWDQ3QGJEJDMIwBnX7BxA= 2022-09-12T20:14:22.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qf7MPkrWDQ3QGJEJDMIwBnX7BxA=', 'Content-Length': '0'} 2022-09-12T20:14:22.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046ce4eb389e9e99c-00631f931e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.277 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:22.278 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:22.279 DEBUG:boto:Method: HEAD 2022-09-12T20:14:22.279 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:14:22.280 DEBUG:boto:Data: 2022-09-12T20:14:22.280 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.281 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.281 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.281 DEBUG:boto:Params: {} 2022-09-12T20:14:22.282 DEBUG:boto:Token: None 2022-09-12T20:14:22.283 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj22 2022-09-12T20:14:22.283 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qf7MPkrWDQ3QGJEJDMIwBnX7BxA= 2022-09-12T20:14:22.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qf7MPkrWDQ3QGJEJDMIwBnX7BxA=', 'Content-Length': '0'} 2022-09-12T20:14:22.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6f8e904e3340561-00631f931e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:14:22.287 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:22.288 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:22.288 DEBUG:boto:Method: GET 2022-09-12T20:14:22.289 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:14:22.289 DEBUG:boto:Data: 2022-09-12T20:14:22.290 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.290 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.291 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.291 DEBUG:boto:Params: {} 2022-09-12T20:14:22.292 DEBUG:boto:Token: None 2022-09-12T20:14:22.292 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj22 2022-09-12T20:14:22.293 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:n4+Ce/nU9MLioJhaKj3rxVmWogM= 2022-09-12T20:14:22.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:n4+Ce/nU9MLioJhaKj3rxVmWogM=', 'Content-Length': '0'} 2022-09-12T20:14:22.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000942439065ba932e0-00631f931e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.296 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:14:22.297 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:14:22.298 DEBUG:boto:Method: GET 2022-09-12T20:14:22.298 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:14:22.299 DEBUG:boto:Data: 2022-09-12T20:14:22.299 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.300 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.300 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.301 DEBUG:boto:Params: {} 2022-09-12T20:14:22.301 DEBUG:boto:Token: None 2022-09-12T20:14:22.302 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj22 2022-09-12T20:14:22.302 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:n4+Ce/nU9MLioJhaKj3rxVmWogM= 2022-09-12T20:14:22.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:n4+Ce/nU9MLioJhaKj3rxVmWogM=', 'Content-Length': '0'} 2022-09-12T20:14:22.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047532e722ed112b5-00631f931e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.306 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:22.306 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:22.307 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:14:22.307 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:14:22.308 DEBUG:boto:Method: GET 2022-09-12T20:14:22.308 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:14:22.309 DEBUG:boto:Data: 2022-09-12T20:14:22.309 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.310 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.310 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.311 DEBUG:boto:Params: {} 2022-09-12T20:14:22.311 DEBUG:boto:Token: None 2022-09-12T20:14:22.312 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/ 2022-09-12T20:14:22.313 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q= 2022-09-12T20:14:22.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q=', 'Content-Length': '0'} 2022-09-12T20:14:22.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2041750ad204644-00631f931e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.317 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:22.317 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:22.318 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:22.319 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:14:22.319 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:14:22.320 DEBUG:boto:Method: GET 2022-09-12T20:14:22.320 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:14:22.321 DEBUG:boto:Data: 2022-09-12T20:14:22.321 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.322 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.322 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.323 DEBUG:boto:Params: {} 2022-09-12T20:14:22.323 DEBUG:boto:Token: None 2022-09-12T20:14:22.324 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/ 2022-09-12T20:14:22.324 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q= 2022-09-12T20:14:22.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q=', 'Content-Length': '0'} 2022-09-12T20:14:22.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000faf7c6a8d64bbb6a-00631f931e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.328 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:22.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:14:22.329 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:22.330 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:22.330 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:22.331 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:22.331 DEBUG:boto:Method: GET 2022-09-12T20:14:22.332 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:22.332 DEBUG:boto:Data: 2022-09-12T20:14:22.333 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.334 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.334 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.335 DEBUG:boto:Params: {} 2022-09-12T20:14:22.335 DEBUG:boto:Token: None 2022-09-12T20:14:22.336 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:22.336 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jpnjfR6kW302dsAvvHnJEydkbWU= 2022-09-12T20:14:22.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jpnjfR6kW302dsAvvHnJEydkbWU=', 'Content-Length': '0'} 2022-09-12T20:14:22.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d5a1615d5596d2e-00631f931e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.340 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:22.340 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:22.341 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:22.341 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:22.342 DEBUG:boto:Method: GET 2022-09-12T20:14:22.343 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:22.343 DEBUG:boto:Data: 2022-09-12T20:14:22.344 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.344 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.345 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.345 DEBUG:boto:Params: {} 2022-09-12T20:14:22.346 DEBUG:boto:Token: None 2022-09-12T20:14:22.346 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:22.347 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jpnjfR6kW302dsAvvHnJEydkbWU= 2022-09-12T20:14:22.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jpnjfR6kW302dsAvvHnJEydkbWU=', 'Content-Length': '0'} 2022-09-12T20:14:22.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1dc2f0fd5be3260-00631f931e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.350 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:22.351 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:22.351 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:22.352 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:22.352 DEBUG:boto:Method: HEAD 2022-09-12T20:14:22.353 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:22.353 DEBUG:boto:Data: 2022-09-12T20:14:22.354 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.354 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.355 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.355 DEBUG:boto:Params: {} 2022-09-12T20:14:22.356 DEBUG:boto:Token: None 2022-09-12T20:14:22.356 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:22.357 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:18yAr6A6GxldHHtmuBnTF9llDtw= 2022-09-12T20:14:22.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:18yAr6A6GxldHHtmuBnTF9llDtw=', 'Content-Length': '0'} 2022-09-12T20:14:22.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007985f7e0ae938eef-00631f931e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.360 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:22.360 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:22.361 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:22.361 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:22.362 DEBUG:boto:Method: HEAD 2022-09-12T20:14:22.362 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:22.363 DEBUG:boto:Data: 2022-09-12T20:14:22.363 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.364 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.364 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.365 DEBUG:boto:Params: {} 2022-09-12T20:14:22.365 DEBUG:boto:Token: None 2022-09-12T20:14:22.366 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:22.366 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:18yAr6A6GxldHHtmuBnTF9llDtw= 2022-09-12T20:14:22.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:18yAr6A6GxldHHtmuBnTF9llDtw=', 'Content-Length': '0'} 2022-09-12T20:14:22.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043943ee9e50cf7e8-00631f931e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:14:22.370 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:22.371 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:22.371 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:22.372 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:22.372 DEBUG:boto:Method: GET 2022-09-12T20:14:22.373 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:22.373 DEBUG:boto:Data: 2022-09-12T20:14:22.374 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.374 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.375 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.375 DEBUG:boto:Params: {} 2022-09-12T20:14:22.376 DEBUG:boto:Token: None 2022-09-12T20:14:22.376 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:22.377 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jpnjfR6kW302dsAvvHnJEydkbWU= 2022-09-12T20:14:22.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jpnjfR6kW302dsAvvHnJEydkbWU=', 'Content-Length': '0'} 2022-09-12T20:14:22.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bea89c2364bcf03-00631f931e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.381 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:22.381 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:22.382 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:22.382 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:14:22.383 DEBUG:boto:Method: GET 2022-09-12T20:14:22.383 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:22.384 DEBUG:boto:Data: 2022-09-12T20:14:22.384 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.385 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.385 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.386 DEBUG:boto:Params: {} 2022-09-12T20:14:22.386 DEBUG:boto:Token: None 2022-09-12T20:14:22.387 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:14:22.388 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jpnjfR6kW302dsAvvHnJEydkbWU= 2022-09-12T20:14:22.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jpnjfR6kW302dsAvvHnJEydkbWU=', 'Content-Length': '0'} 2022-09-12T20:14:22.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057492e9c2e02eede-00631f931e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.391 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:22.392 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:22.392 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:14:22.393 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:14:22.393 DEBUG:boto:Method: GET 2022-09-12T20:14:22.394 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:14:22.394 DEBUG:boto:Data: 2022-09-12T20:14:22.395 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.395 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.396 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.397 DEBUG:boto:Params: {} 2022-09-12T20:14:22.397 DEBUG:boto:Token: None 2022-09-12T20:14:22.398 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/ 2022-09-12T20:14:22.398 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q= 2022-09-12T20:14:22.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q=', 'Content-Length': '0'} 2022-09-12T20:14:22.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbf17a74057a2336-00631f931e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.402 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:22.403 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:22.404 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:22.404 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:14:22.405 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:14:22.405 DEBUG:boto:Method: GET 2022-09-12T20:14:22.406 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:14:22.406 DEBUG:boto:Data: 2022-09-12T20:14:22.407 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.407 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.408 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.408 DEBUG:boto:Params: {} 2022-09-12T20:14:22.409 DEBUG:boto:Token: None 2022-09-12T20:14:22.409 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/ 2022-09-12T20:14:22.410 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q= 2022-09-12T20:14:22.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q=', 'Content-Length': '0'} 2022-09-12T20:14:22.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb277542ea6c8f3d-00631f931e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.414 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:22.415 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:22.416 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:22.417 DEBUG:boto:Method: HEAD 2022-09-12T20:14:22.417 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:14:22.418 DEBUG:boto:Data: 2022-09-12T20:14:22.419 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.419 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.420 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.420 DEBUG:boto:Params: {} 2022-09-12T20:14:22.421 DEBUG:boto:Token: None 2022-09-12T20:14:22.421 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj23 2022-09-12T20:14:22.422 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rgx5sl0X+Vjh4f6uiqjo1m+crMo= 2022-09-12T20:14:22.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rgx5sl0X+Vjh4f6uiqjo1m+crMo=', 'Content-Length': '0'} 2022-09-12T20:14:22.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de561a07d43912dc-00631f931e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.425 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:22.426 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:22.426 DEBUG:boto:Method: HEAD 2022-09-12T20:14:22.427 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:14:22.427 DEBUG:boto:Data: 2022-09-12T20:14:22.428 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.428 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.429 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.429 DEBUG:boto:Params: {} 2022-09-12T20:14:22.430 DEBUG:boto:Token: None 2022-09-12T20:14:22.430 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj23 2022-09-12T20:14:22.431 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rgx5sl0X+Vjh4f6uiqjo1m+crMo= 2022-09-12T20:14:22.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rgx5sl0X+Vjh4f6uiqjo1m+crMo=', 'Content-Length': '0'} 2022-09-12T20:14:22.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000069d73ed7bd65396-00631f931e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:14:22.435 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:22.435 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:22.436 DEBUG:boto:Method: GET 2022-09-12T20:14:22.436 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:14:22.437 DEBUG:boto:Data: 2022-09-12T20:14:22.437 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.438 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.438 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.439 DEBUG:boto:Params: {} 2022-09-12T20:14:22.440 DEBUG:boto:Token: None 2022-09-12T20:14:22.440 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj23 2022-09-12T20:14:22.441 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zkvIVgr/LB9uSjfC+mFOghjEhsc= 2022-09-12T20:14:22.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zkvIVgr/LB9uSjfC+mFOghjEhsc=', 'Content-Length': '0'} 2022-09-12T20:14:22.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e007d2a6c311cf8f-00631f931e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.444 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:14:22.445 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:14:22.445 DEBUG:boto:Method: GET 2022-09-12T20:14:22.446 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:14:22.447 DEBUG:boto:Data: 2022-09-12T20:14:22.447 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.448 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.448 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.449 DEBUG:boto:Params: {} 2022-09-12T20:14:22.449 DEBUG:boto:Token: None 2022-09-12T20:14:22.450 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj23 2022-09-12T20:14:22.451 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zkvIVgr/LB9uSjfC+mFOghjEhsc= 2022-09-12T20:14:22.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zkvIVgr/LB9uSjfC+mFOghjEhsc=', 'Content-Length': '0'} 2022-09-12T20:14:22.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c121ac3a0ee0cd5b-00631f931e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.454 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:22.455 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:22.455 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:14:22.456 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:14:22.456 DEBUG:boto:Method: GET 2022-09-12T20:14:22.457 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:14:22.457 DEBUG:boto:Data: 2022-09-12T20:14:22.458 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.458 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.459 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.459 DEBUG:boto:Params: {} 2022-09-12T20:14:22.460 DEBUG:boto:Token: None 2022-09-12T20:14:22.460 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/ 2022-09-12T20:14:22.461 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q= 2022-09-12T20:14:22.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q=', 'Content-Length': '0'} 2022-09-12T20:14:22.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5a2bb6be6f58ac8-00631f931e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.465 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:22.466 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:22.466 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:22.467 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:14:22.467 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:14:22.468 DEBUG:boto:Method: GET 2022-09-12T20:14:22.468 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:14:22.469 DEBUG:boto:Data: 2022-09-12T20:14:22.469 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.470 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.470 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.471 DEBUG:boto:Params: {} 2022-09-12T20:14:22.471 DEBUG:boto:Token: None 2022-09-12T20:14:22.472 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/ 2022-09-12T20:14:22.472 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q= 2022-09-12T20:14:22.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q=', 'Content-Length': '0'} 2022-09-12T20:14:22.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e978d18ed02c404e-00631f931e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.476 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:22.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:14:22.477 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:22.478 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:22.479 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:22.479 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:22.480 DEBUG:boto:Method: GET 2022-09-12T20:14:22.480 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:22.481 DEBUG:boto:Data: 2022-09-12T20:14:22.481 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.482 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.482 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.483 DEBUG:boto:Params: {} 2022-09-12T20:14:22.483 DEBUG:boto:Token: None 2022-09-12T20:14:22.484 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:22.484 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:o5WS5AK8JBNihGAqH1a45UReM/8= 2022-09-12T20:14:22.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:o5WS5AK8JBNihGAqH1a45UReM/8=', 'Content-Length': '0'} 2022-09-12T20:14:22.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcb69d7d3f6431a9-00631f931e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.488 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:22.488 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:22.489 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:22.489 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:22.490 DEBUG:boto:Method: GET 2022-09-12T20:14:22.490 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:22.491 DEBUG:boto:Data: 2022-09-12T20:14:22.491 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.492 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.492 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.493 DEBUG:boto:Params: {} 2022-09-12T20:14:22.494 DEBUG:boto:Token: None 2022-09-12T20:14:22.494 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:22.495 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:o5WS5AK8JBNihGAqH1a45UReM/8= 2022-09-12T20:14:22.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:o5WS5AK8JBNihGAqH1a45UReM/8=', 'Content-Length': '0'} 2022-09-12T20:14:22.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d875264c0479647c-00631f931e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.498 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:22.499 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:22.499 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:22.500 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:22.500 DEBUG:boto:Method: HEAD 2022-09-12T20:14:22.501 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:22.501 DEBUG:boto:Data: 2022-09-12T20:14:22.502 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.502 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.503 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.503 DEBUG:boto:Params: {} 2022-09-12T20:14:22.504 DEBUG:boto:Token: None 2022-09-12T20:14:22.504 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:22.505 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NjN9Dabh/BUyAmZgH2mdtGU7KE4= 2022-09-12T20:14:22.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NjN9Dabh/BUyAmZgH2mdtGU7KE4=', 'Content-Length': '0'} 2022-09-12T20:14:22.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ff5ee7df621bc71-00631f931e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.508 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:22.509 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:22.509 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:22.510 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:22.510 DEBUG:boto:Method: HEAD 2022-09-12T20:14:22.511 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:22.511 DEBUG:boto:Data: 2022-09-12T20:14:22.512 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.512 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.513 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.513 DEBUG:boto:Params: {} 2022-09-12T20:14:22.514 DEBUG:boto:Token: None 2022-09-12T20:14:22.514 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:22.515 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NjN9Dabh/BUyAmZgH2mdtGU7KE4= 2022-09-12T20:14:22.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NjN9Dabh/BUyAmZgH2mdtGU7KE4=', 'Content-Length': '0'} 2022-09-12T20:14:22.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d5c05cd73922541-00631f931e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:14:22.518 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:22.519 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:22.520 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:22.520 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:22.521 DEBUG:boto:Method: GET 2022-09-12T20:14:22.521 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:22.522 DEBUG:boto:Data: 2022-09-12T20:14:22.522 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.523 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.523 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.524 DEBUG:boto:Params: {} 2022-09-12T20:14:22.525 DEBUG:boto:Token: None 2022-09-12T20:14:22.525 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:22.526 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:o5WS5AK8JBNihGAqH1a45UReM/8= 2022-09-12T20:14:22.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:o5WS5AK8JBNihGAqH1a45UReM/8=', 'Content-Length': '0'} 2022-09-12T20:14:22.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba02156a55927c45-00631f931e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.530 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:22.531 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:22.531 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:22.532 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:14:22.532 DEBUG:boto:Method: GET 2022-09-12T20:14:22.533 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:22.533 DEBUG:boto:Data: 2022-09-12T20:14:22.534 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.535 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.535 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.536 DEBUG:boto:Params: {} 2022-09-12T20:14:22.536 DEBUG:boto:Token: None 2022-09-12T20:14:22.537 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:14:22.538 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:o5WS5AK8JBNihGAqH1a45UReM/8= 2022-09-12T20:14:22.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:o5WS5AK8JBNihGAqH1a45UReM/8=', 'Content-Length': '0'} 2022-09-12T20:14:22.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eec390bf179d5b41-00631f931e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.541 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:22.542 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:22.542 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:14:22.543 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:14:22.543 DEBUG:boto:Method: GET 2022-09-12T20:14:22.544 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:14:22.545 DEBUG:boto:Data: 2022-09-12T20:14:22.545 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.546 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.546 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.547 DEBUG:boto:Params: {} 2022-09-12T20:14:22.547 DEBUG:boto:Token: None 2022-09-12T20:14:22.548 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/ 2022-09-12T20:14:22.549 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q= 2022-09-12T20:14:22.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q=', 'Content-Length': '0'} 2022-09-12T20:14:22.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aadb38eb9181b304-00631f931e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.553 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:22.554 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:22.554 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:22.555 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:14:22.555 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:14:22.556 DEBUG:boto:Method: GET 2022-09-12T20:14:22.556 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:14:22.557 DEBUG:boto:Data: 2022-09-12T20:14:22.557 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.558 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.559 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.559 DEBUG:boto:Params: {} 2022-09-12T20:14:22.560 DEBUG:boto:Token: None 2022-09-12T20:14:22.560 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/ 2022-09-12T20:14:22.561 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q= 2022-09-12T20:14:22.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q=', 'Content-Length': '0'} 2022-09-12T20:14:22.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed441131ea7d53f6-00631f931e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.565 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:22.566 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:22.566 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:22.567 DEBUG:boto:Method: HEAD 2022-09-12T20:14:22.568 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:14:22.569 DEBUG:boto:Data: 2022-09-12T20:14:22.570 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.571 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.571 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.572 DEBUG:boto:Params: {} 2022-09-12T20:14:22.572 DEBUG:boto:Token: None 2022-09-12T20:14:22.573 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj29 2022-09-12T20:14:22.574 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xS3q6hAmSQNRqoUhv1lpgGFPKT8= 2022-09-12T20:14:22.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xS3q6hAmSQNRqoUhv1lpgGFPKT8=', 'Content-Length': '0'} 2022-09-12T20:14:22.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4386c6377c9054f-00631f931e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.578 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:22.580 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:22.580 DEBUG:boto:Method: HEAD 2022-09-12T20:14:22.581 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:14:22.581 DEBUG:boto:Data: 2022-09-12T20:14:22.582 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.582 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.583 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.583 DEBUG:boto:Params: {} 2022-09-12T20:14:22.584 DEBUG:boto:Token: None 2022-09-12T20:14:22.584 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj29 2022-09-12T20:14:22.585 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xS3q6hAmSQNRqoUhv1lpgGFPKT8= 2022-09-12T20:14:22.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xS3q6hAmSQNRqoUhv1lpgGFPKT8=', 'Content-Length': '0'} 2022-09-12T20:14:22.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070e525bae48d5519-00631f931e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:14:22.589 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:22.590 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:22.590 DEBUG:boto:Method: GET 2022-09-12T20:14:22.591 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:14:22.591 DEBUG:boto:Data: 2022-09-12T20:14:22.592 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.593 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.593 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.594 DEBUG:boto:Params: {} 2022-09-12T20:14:22.594 DEBUG:boto:Token: None 2022-09-12T20:14:22.595 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj29 2022-09-12T20:14:22.596 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PNXtu8bEjllYE8OL+VUTWE+VUC0= 2022-09-12T20:14:22.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PNXtu8bEjllYE8OL+VUTWE+VUC0=', 'Content-Length': '0'} 2022-09-12T20:14:22.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a79390cbf3c2f80c-00631f931e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.599 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:14:22.600 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:14:22.600 DEBUG:boto:Method: GET 2022-09-12T20:14:22.601 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:14:22.602 DEBUG:boto:Data: 2022-09-12T20:14:22.602 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.603 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.603 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.604 DEBUG:boto:Params: {} 2022-09-12T20:14:22.604 DEBUG:boto:Token: None 2022-09-12T20:14:22.605 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj29 2022-09-12T20:14:22.606 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PNXtu8bEjllYE8OL+VUTWE+VUC0= 2022-09-12T20:14:22.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PNXtu8bEjllYE8OL+VUTWE+VUC0=', 'Content-Length': '0'} 2022-09-12T20:14:22.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058219ad0abf11469-00631f931e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.609 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:22.610 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:22.610 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:14:22.611 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:14:22.611 DEBUG:boto:Method: GET 2022-09-12T20:14:22.612 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:14:22.612 DEBUG:boto:Data: 2022-09-12T20:14:22.613 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.613 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.614 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.614 DEBUG:boto:Params: {} 2022-09-12T20:14:22.615 DEBUG:boto:Token: None 2022-09-12T20:14:22.616 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/ 2022-09-12T20:14:22.616 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q= 2022-09-12T20:14:22.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q=', 'Content-Length': '0'} 2022-09-12T20:14:22.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000461f14522c632439-00631f931e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.662 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:22.663 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:22.663 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:22.664 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:14:22.664 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:14:22.665 DEBUG:boto:Method: GET 2022-09-12T20:14:22.665 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:14:22.666 DEBUG:boto:Data: 2022-09-12T20:14:22.666 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.667 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.668 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.668 DEBUG:boto:Params: {} 2022-09-12T20:14:22.669 DEBUG:boto:Token: None 2022-09-12T20:14:22.669 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/ 2022-09-12T20:14:22.670 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q= 2022-09-12T20:14:22.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q=', 'Content-Length': '0'} 2022-09-12T20:14:22.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e4086ea23b10de8-00631f931e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.674 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:22.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-12T20:14:22.675 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:22.676 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:22.676 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:22.677 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:22.677 DEBUG:boto:Method: GET 2022-09-12T20:14:22.678 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:22.678 DEBUG:boto:Data: 2022-09-12T20:14:22.679 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.679 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.680 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.680 DEBUG:boto:Params: {} 2022-09-12T20:14:22.681 DEBUG:boto:Token: None 2022-09-12T20:14:22.682 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:22.682 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u56WhFyd8NPEJcqWwHT9rukBOh4= 2022-09-12T20:14:22.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u56WhFyd8NPEJcqWwHT9rukBOh4=', 'Content-Length': '0'} 2022-09-12T20:14:22.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000165be5b6fd1d647b-00631f931e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.686 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:22.686 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:22.687 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:22.687 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:22.688 DEBUG:boto:Method: GET 2022-09-12T20:14:22.688 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:22.689 DEBUG:boto:Data: 2022-09-12T20:14:22.690 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.690 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.691 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.691 DEBUG:boto:Params: {} 2022-09-12T20:14:22.692 DEBUG:boto:Token: None 2022-09-12T20:14:22.692 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:22.693 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u56WhFyd8NPEJcqWwHT9rukBOh4= 2022-09-12T20:14:22.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u56WhFyd8NPEJcqWwHT9rukBOh4=', 'Content-Length': '0'} 2022-09-12T20:14:22.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ecf58e3026e490e-00631f931e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.696 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:22.697 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:22.698 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:22.698 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:22.699 DEBUG:boto:Method: HEAD 2022-09-12T20:14:22.699 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:22.700 DEBUG:boto:Data: 2022-09-12T20:14:22.700 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.701 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.701 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.702 DEBUG:boto:Params: {} 2022-09-12T20:14:22.702 DEBUG:boto:Token: None 2022-09-12T20:14:22.703 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:22.703 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z7MuVJj4n518RUdeR9RQs/Cx6Ew= 2022-09-12T20:14:22.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z7MuVJj4n518RUdeR9RQs/Cx6Ew=', 'Content-Length': '0'} 2022-09-12T20:14:22.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dd47123b5503bc8-00631f931e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.707 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:22.707 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:22.708 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:22.708 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:22.709 DEBUG:boto:Method: HEAD 2022-09-12T20:14:22.709 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:22.710 DEBUG:boto:Data: 2022-09-12T20:14:22.710 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.711 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.711 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.712 DEBUG:boto:Params: {} 2022-09-12T20:14:22.712 DEBUG:boto:Token: None 2022-09-12T20:14:22.713 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:22.713 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z7MuVJj4n518RUdeR9RQs/Cx6Ew= 2022-09-12T20:14:22.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z7MuVJj4n518RUdeR9RQs/Cx6Ew=', 'Content-Length': '0'} 2022-09-12T20:14:22.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b676687555c5e05-00631f931e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-12T20:14:22.717 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:22.718 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:22.718 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:22.719 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:22.719 DEBUG:boto:Method: GET 2022-09-12T20:14:22.720 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:22.720 DEBUG:boto:Data: 2022-09-12T20:14:22.721 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.721 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.722 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.723 DEBUG:boto:Params: {} 2022-09-12T20:14:22.723 DEBUG:boto:Token: None 2022-09-12T20:14:22.724 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:22.724 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u56WhFyd8NPEJcqWwHT9rukBOh4= 2022-09-12T20:14:22.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u56WhFyd8NPEJcqWwHT9rukBOh4=', 'Content-Length': '0'} 2022-09-12T20:14:22.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000965dacffe69d67c7-00631f931e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.728 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:22.728 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:22.729 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:22.730 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:14:22.730 DEBUG:boto:Method: GET 2022-09-12T20:14:22.731 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:22.731 DEBUG:boto:Data: 2022-09-12T20:14:22.732 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.732 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.733 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.733 DEBUG:boto:Params: {} 2022-09-12T20:14:22.734 DEBUG:boto:Token: None 2022-09-12T20:14:22.734 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:14:22.735 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:u56WhFyd8NPEJcqWwHT9rukBOh4= 2022-09-12T20:14:22.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:u56WhFyd8NPEJcqWwHT9rukBOh4=', 'Content-Length': '0'} 2022-09-12T20:14:22.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002616f104bff7a32a-00631f931e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.739 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:22.739 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:22.740 DEBUG:boto:path=/jdpiak-75/?prefix=obj31 2022-09-12T20:14:22.740 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj31 2022-09-12T20:14:22.741 DEBUG:boto:Method: GET 2022-09-12T20:14:22.741 DEBUG:boto:Path: /jdpiak-75/?prefix=obj31 2022-09-12T20:14:22.742 DEBUG:boto:Data: 2022-09-12T20:14:22.742 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.743 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.743 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.744 DEBUG:boto:Params: {} 2022-09-12T20:14:22.745 DEBUG:boto:Token: None 2022-09-12T20:14:22.745 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/ 2022-09-12T20:14:22.746 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q= 2022-09-12T20:14:22.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q=', 'Content-Length': '0'} 2022-09-12T20:14:22.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026768511c45b164e-00631f931e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.750 DEBUG:boto:b'jdpiak-75obj311000falseobj312022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:22.751 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:22.751 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:22.752 DEBUG:boto:path=/jdpiak-75/?prefix=obj31 2022-09-12T20:14:22.752 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj31 2022-09-12T20:14:22.753 DEBUG:boto:Method: GET 2022-09-12T20:14:22.753 DEBUG:boto:Path: /jdpiak-75/?prefix=obj31 2022-09-12T20:14:22.754 DEBUG:boto:Data: 2022-09-12T20:14:22.754 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.755 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.755 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.756 DEBUG:boto:Params: {} 2022-09-12T20:14:22.757 DEBUG:boto:Token: None 2022-09-12T20:14:22.757 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/ 2022-09-12T20:14:22.758 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q= 2022-09-12T20:14:22.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q=', 'Content-Length': '0'} 2022-09-12T20:14:22.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ef15360cf6f1b34-00631f931e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.802 DEBUG:boto:b'jdpiak-75obj311000falseobj312022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:22.803 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:22.804 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:22.804 DEBUG:boto:Method: HEAD 2022-09-12T20:14:22.805 DEBUG:boto:Path: /jdpiak-75/obj31 2022-09-12T20:14:22.805 DEBUG:boto:Data: 2022-09-12T20:14:22.806 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.806 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.807 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.807 DEBUG:boto:Params: {} 2022-09-12T20:14:22.808 DEBUG:boto:Token: None 2022-09-12T20:14:22.808 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj31 2022-09-12T20:14:22.809 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cytCedTjHqjvQjB5dKwFm+FimR0= 2022-09-12T20:14:22.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cytCedTjHqjvQjB5dKwFm+FimR0=', 'Content-Length': '0'} 2022-09-12T20:14:22.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000976e95d35ca4f082-00631f931e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.812 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:22.813 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:22.813 DEBUG:boto:Method: HEAD 2022-09-12T20:14:22.814 DEBUG:boto:Path: /jdpiak-75/obj31 2022-09-12T20:14:22.815 DEBUG:boto:Data: 2022-09-12T20:14:22.815 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.816 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.816 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.817 DEBUG:boto:Params: {} 2022-09-12T20:14:22.817 DEBUG:boto:Token: None 2022-09-12T20:14:22.818 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj31 2022-09-12T20:14:22.818 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cytCedTjHqjvQjB5dKwFm+FimR0= 2022-09-12T20:14:22.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cytCedTjHqjvQjB5dKwFm+FimR0=', 'Content-Length': '0'} 2022-09-12T20:14:22.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039028b5e480e6297-00631f931e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-12T20:14:22.822 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:22.823 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:22.823 DEBUG:boto:Method: GET 2022-09-12T20:14:22.824 DEBUG:boto:Path: /jdpiak-75/obj31 2022-09-12T20:14:22.825 DEBUG:boto:Data: 2022-09-12T20:14:22.825 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.826 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.826 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.827 DEBUG:boto:Params: {} 2022-09-12T20:14:22.827 DEBUG:boto:Token: None 2022-09-12T20:14:22.828 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj31 2022-09-12T20:14:22.829 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J5UfYFReR4QeVLrSCa2O9tIIWgk= 2022-09-12T20:14:22.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J5UfYFReR4QeVLrSCa2O9tIIWgk=', 'Content-Length': '0'} 2022-09-12T20:14:22.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fcf416e20658c41-00631f931e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.832 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:14:22.833 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:14:22.833 DEBUG:boto:Method: GET 2022-09-12T20:14:22.834 DEBUG:boto:Path: /jdpiak-75/obj31 2022-09-12T20:14:22.834 DEBUG:boto:Data: 2022-09-12T20:14:22.835 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.835 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.836 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.836 DEBUG:boto:Params: {} 2022-09-12T20:14:22.837 DEBUG:boto:Token: None 2022-09-12T20:14:22.838 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj31 2022-09-12T20:14:22.838 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J5UfYFReR4QeVLrSCa2O9tIIWgk= 2022-09-12T20:14:22.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J5UfYFReR4QeVLrSCa2O9tIIWgk=', 'Content-Length': '0'} 2022-09-12T20:14:22.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5818c0e22205d00-00631f931e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.842 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:22.842 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:22.843 DEBUG:boto:path=/jdpiak-75/?prefix=obj31 2022-09-12T20:14:22.843 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj31 2022-09-12T20:14:22.844 DEBUG:boto:Method: GET 2022-09-12T20:14:22.844 DEBUG:boto:Path: /jdpiak-75/?prefix=obj31 2022-09-12T20:14:22.845 DEBUG:boto:Data: 2022-09-12T20:14:22.845 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.846 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.846 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.847 DEBUG:boto:Params: {} 2022-09-12T20:14:22.848 DEBUG:boto:Token: None 2022-09-12T20:14:22.848 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/ 2022-09-12T20:14:22.849 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q= 2022-09-12T20:14:22.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q=', 'Content-Length': '0'} 2022-09-12T20:14:22.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030b172b6e0ff8324-00631f931e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.853 DEBUG:boto:b'jdpiak-75obj311000falseobj312022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:22.854 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:22.855 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:22.855 DEBUG:boto:path=/jdpiak-75/?prefix=obj31 2022-09-12T20:14:22.856 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj31 2022-09-12T20:14:22.856 DEBUG:boto:Method: GET 2022-09-12T20:14:22.857 DEBUG:boto:Path: /jdpiak-75/?prefix=obj31 2022-09-12T20:14:22.857 DEBUG:boto:Data: 2022-09-12T20:14:22.858 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.859 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.859 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.860 DEBUG:boto:Params: {} 2022-09-12T20:14:22.860 DEBUG:boto:Token: None 2022-09-12T20:14:22.861 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/ 2022-09-12T20:14:22.861 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q= 2022-09-12T20:14:22.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q=', 'Content-Length': '0'} 2022-09-12T20:14:22.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000106dd682f4855d5e-00631f931e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.865 DEBUG:boto:b'jdpiak-75obj311000falseobj312022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:22.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:14:22.867 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:22.867 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:22.868 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:22.868 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:22.869 DEBUG:boto:Method: GET 2022-09-12T20:14:22.869 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:22.870 DEBUG:boto:Data: 2022-09-12T20:14:22.870 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.871 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.872 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.872 DEBUG:boto:Params: {} 2022-09-12T20:14:22.873 DEBUG:boto:Token: None 2022-09-12T20:14:22.873 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:22.874 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RkjpHbxnE6WUwmF4cKkFCV+XXZc= 2022-09-12T20:14:22.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RkjpHbxnE6WUwmF4cKkFCV+XXZc=', 'Content-Length': '0'} 2022-09-12T20:14:22.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014590b96f2abf52b-00631f931e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.878 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:22.878 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:22.879 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:22.879 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:22.880 DEBUG:boto:Method: GET 2022-09-12T20:14:22.880 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:22.881 DEBUG:boto:Data: 2022-09-12T20:14:22.881 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.882 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.882 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.883 DEBUG:boto:Params: {} 2022-09-12T20:14:22.883 DEBUG:boto:Token: None 2022-09-12T20:14:22.884 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:22.885 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RkjpHbxnE6WUwmF4cKkFCV+XXZc= 2022-09-12T20:14:22.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RkjpHbxnE6WUwmF4cKkFCV+XXZc=', 'Content-Length': '0'} 2022-09-12T20:14:22.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc22c6772da059ce-00631f931e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.888 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:22.889 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:22.890 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:22.890 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:22.891 DEBUG:boto:Method: HEAD 2022-09-12T20:14:22.891 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:22.892 DEBUG:boto:Data: 2022-09-12T20:14:22.892 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.893 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.893 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.894 DEBUG:boto:Params: {} 2022-09-12T20:14:22.894 DEBUG:boto:Token: None 2022-09-12T20:14:22.895 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:22.895 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PbCrEcGWcfUN9mk8A5wkAwrblzw= 2022-09-12T20:14:22.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PbCrEcGWcfUN9mk8A5wkAwrblzw=', 'Content-Length': '0'} 2022-09-12T20:14:22.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b33abeda5ea7bfb1-00631f931e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.899 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:22.899 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:22.900 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:22.900 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:22.901 DEBUG:boto:Method: HEAD 2022-09-12T20:14:22.902 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:22.902 DEBUG:boto:Data: 2022-09-12T20:14:22.903 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.903 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.904 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.904 DEBUG:boto:Params: {} 2022-09-12T20:14:22.905 DEBUG:boto:Token: None 2022-09-12T20:14:22.906 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:22.906 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PbCrEcGWcfUN9mk8A5wkAwrblzw= 2022-09-12T20:14:22.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PbCrEcGWcfUN9mk8A5wkAwrblzw=', 'Content-Length': '0'} 2022-09-12T20:14:22.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000482e4261095823ac-00631f931e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:14:22.911 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:22.911 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:22.912 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:22.913 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:22.913 DEBUG:boto:Method: GET 2022-09-12T20:14:22.914 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:22.915 DEBUG:boto:Data: 2022-09-12T20:14:22.915 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.916 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.916 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.917 DEBUG:boto:Params: {} 2022-09-12T20:14:22.918 DEBUG:boto:Token: None 2022-09-12T20:14:22.919 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:22.919 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RkjpHbxnE6WUwmF4cKkFCV+XXZc= 2022-09-12T20:14:22.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RkjpHbxnE6WUwmF4cKkFCV+XXZc=', 'Content-Length': '0'} 2022-09-12T20:14:22.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000089c95989dd09210-00631f931e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.966 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:22.967 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:22.967 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:22.968 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:14:22.969 DEBUG:boto:Method: GET 2022-09-12T20:14:22.969 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:22.970 DEBUG:boto:Data: 2022-09-12T20:14:22.971 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.972 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:22.972 DEBUG:boto:Port: 8000 2022-09-12T20:14:22.973 DEBUG:boto:Params: {} 2022-09-12T20:14:22.974 DEBUG:boto:Token: None 2022-09-12T20:14:22.975 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:14:22.976 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RkjpHbxnE6WUwmF4cKkFCV+XXZc= 2022-09-12T20:14:22.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RkjpHbxnE6WUwmF4cKkFCV+XXZc=', 'Content-Length': '0'} 2022-09-12T20:14:22.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003967672201aaf401-00631f931e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.980 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:22.980 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:22.981 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:14:22.981 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:14:22.982 DEBUG:boto:Method: GET 2022-09-12T20:14:22.983 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:14:22.984 DEBUG:boto:Data: 2022-09-12T20:14:22.984 DEBUG:boto:Headers: {} 2022-09-12T20:14:22.985 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:22.986 DEBUG:boto:Port: 8001 2022-09-12T20:14:22.987 DEBUG:boto:Params: {} 2022-09-12T20:14:22.988 DEBUG:boto:Token: None 2022-09-12T20:14:22.989 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:22 GMT /jdpiak-75/ 2022-09-12T20:14:22.990 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q= 2022-09-12T20:14:22.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5SoKO+lVScTu+kDFMGU0Lzqjy7Q=', 'Content-Length': '0'} 2022-09-12T20:14:22.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017d6f8cee57632f0-00631f931e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:22.995 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:22.996 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:22.997 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:22.997 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:14:22.998 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:14:22.999 DEBUG:boto:Method: GET 2022-09-12T20:14:22.999 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:14:23.000 DEBUG:boto:Data: 2022-09-12T20:14:23.001 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.002 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.002 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.003 DEBUG:boto:Params: {} 2022-09-12T20:14:23.004 DEBUG:boto:Token: None 2022-09-12T20:14:23.005 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/ 2022-09-12T20:14:23.006 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo= 2022-09-12T20:14:23.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo=', 'Content-Length': '0'} 2022-09-12T20:14:23.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a28aafa173f8db73-00631f931f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.011 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:23.012 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:23.012 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:23.013 DEBUG:boto:Method: HEAD 2022-09-12T20:14:23.014 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:14:23.014 DEBUG:boto:Data: 2022-09-12T20:14:23.015 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.016 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.017 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.017 DEBUG:boto:Params: {} 2022-09-12T20:14:23.018 DEBUG:boto:Token: None 2022-09-12T20:14:23.019 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj33 2022-09-12T20:14:23.020 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m/WP/Eerb10E1kxKeZ8l/4m+MTY= 2022-09-12T20:14:23.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m/WP/Eerb10E1kxKeZ8l/4m+MTY=', 'Content-Length': '0'} 2022-09-12T20:14:23.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076b9a5785158405b-00631f931f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.023 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:23.024 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:23.025 DEBUG:boto:Method: HEAD 2022-09-12T20:14:23.026 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:14:23.026 DEBUG:boto:Data: 2022-09-12T20:14:23.027 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.028 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.028 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.029 DEBUG:boto:Params: {} 2022-09-12T20:14:23.029 DEBUG:boto:Token: None 2022-09-12T20:14:23.030 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj33 2022-09-12T20:14:23.031 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m/WP/Eerb10E1kxKeZ8l/4m+MTY= 2022-09-12T20:14:23.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m/WP/Eerb10E1kxKeZ8l/4m+MTY=', 'Content-Length': '0'} 2022-09-12T20:14:23.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000509544ba2d526504-00631f931f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:14:23.036 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:23.037 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:23.038 DEBUG:boto:Method: GET 2022-09-12T20:14:23.039 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:14:23.040 DEBUG:boto:Data: 2022-09-12T20:14:23.041 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.042 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.043 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.044 DEBUG:boto:Params: {} 2022-09-12T20:14:23.045 DEBUG:boto:Token: None 2022-09-12T20:14:23.046 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj33 2022-09-12T20:14:23.047 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/o7+9H9LkNT62Lu9jpFN/mrDL3c= 2022-09-12T20:14:23.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/o7+9H9LkNT62Lu9jpFN/mrDL3c=', 'Content-Length': '0'} 2022-09-12T20:14:23.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008787dcb1b338804-00631f931f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.094 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:14:23.095 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:14:23.097 DEBUG:boto:Method: GET 2022-09-12T20:14:23.097 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:14:23.098 DEBUG:boto:Data: 2022-09-12T20:14:23.099 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.100 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.101 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.102 DEBUG:boto:Params: {} 2022-09-12T20:14:23.103 DEBUG:boto:Token: None 2022-09-12T20:14:23.104 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj33 2022-09-12T20:14:23.106 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/o7+9H9LkNT62Lu9jpFN/mrDL3c= 2022-09-12T20:14:23.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/o7+9H9LkNT62Lu9jpFN/mrDL3c=', 'Content-Length': '0'} 2022-09-12T20:14:23.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ea6e69ec444753d-00631f931f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.110 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:23.111 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:23.112 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:14:23.113 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:14:23.114 DEBUG:boto:Method: GET 2022-09-12T20:14:23.115 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:14:23.116 DEBUG:boto:Data: 2022-09-12T20:14:23.117 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.118 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.119 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.120 DEBUG:boto:Params: {} 2022-09-12T20:14:23.121 DEBUG:boto:Token: None 2022-09-12T20:14:23.122 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/ 2022-09-12T20:14:23.123 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo= 2022-09-12T20:14:23.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo=', 'Content-Length': '0'} 2022-09-12T20:14:23.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006efe9545dbfbb254-00631f931f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.129 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:23.130 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:23.131 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:23.132 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:14:23.132 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:14:23.133 DEBUG:boto:Method: GET 2022-09-12T20:14:23.134 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:14:23.134 DEBUG:boto:Data: 2022-09-12T20:14:23.135 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.135 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.136 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.137 DEBUG:boto:Params: {} 2022-09-12T20:14:23.137 DEBUG:boto:Token: None 2022-09-12T20:14:23.138 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/ 2022-09-12T20:14:23.139 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo= 2022-09-12T20:14:23.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo=', 'Content-Length': '0'} 2022-09-12T20:14:23.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfcc6ec00dc75712-00631f931f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.144 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:23.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:14:23.145 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:23.146 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:23.146 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:23.147 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:23.148 DEBUG:boto:Method: GET 2022-09-12T20:14:23.148 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:23.149 DEBUG:boto:Data: 2022-09-12T20:14:23.150 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.151 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.151 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.152 DEBUG:boto:Params: {} 2022-09-12T20:14:23.153 DEBUG:boto:Token: None 2022-09-12T20:14:23.153 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:23.154 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KT8s7QMcqPkVXaR8OcMsK4Flo2k= 2022-09-12T20:14:23.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KT8s7QMcqPkVXaR8OcMsK4Flo2k=', 'Content-Length': '0'} 2022-09-12T20:14:23.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000637cb94e14e228d6-00631f931f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.158 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:23.158 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:23.159 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:23.160 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:23.160 DEBUG:boto:Method: GET 2022-09-12T20:14:23.161 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:23.162 DEBUG:boto:Data: 2022-09-12T20:14:23.162 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.163 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.164 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.164 DEBUG:boto:Params: {} 2022-09-12T20:14:23.165 DEBUG:boto:Token: None 2022-09-12T20:14:23.166 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:23.166 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KT8s7QMcqPkVXaR8OcMsK4Flo2k= 2022-09-12T20:14:23.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KT8s7QMcqPkVXaR8OcMsK4Flo2k=', 'Content-Length': '0'} 2022-09-12T20:14:23.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f31ccd680396856-00631f931f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.170 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:23.171 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:23.171 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:23.172 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:23.173 DEBUG:boto:Method: HEAD 2022-09-12T20:14:23.174 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:23.174 DEBUG:boto:Data: 2022-09-12T20:14:23.175 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.176 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.176 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.177 DEBUG:boto:Params: {} 2022-09-12T20:14:23.178 DEBUG:boto:Token: None 2022-09-12T20:14:23.179 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:23.179 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NraMXOmJvS61CHTthLAgca/t1sk= 2022-09-12T20:14:23.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NraMXOmJvS61CHTthLAgca/t1sk=', 'Content-Length': '0'} 2022-09-12T20:14:23.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000999734cbc5bc100e-00631f931f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.183 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:23.184 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:23.184 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:23.185 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:23.185 DEBUG:boto:Method: HEAD 2022-09-12T20:14:23.186 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:23.186 DEBUG:boto:Data: 2022-09-12T20:14:23.187 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.187 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.188 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.189 DEBUG:boto:Params: {} 2022-09-12T20:14:23.189 DEBUG:boto:Token: None 2022-09-12T20:14:23.190 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:23.190 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NraMXOmJvS61CHTthLAgca/t1sk= 2022-09-12T20:14:23.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NraMXOmJvS61CHTthLAgca/t1sk=', 'Content-Length': '0'} 2022-09-12T20:14:23.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a04d702d76cc259f-00631f931f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:14:23.194 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:23.195 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:23.195 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:23.196 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:23.196 DEBUG:boto:Method: GET 2022-09-12T20:14:23.197 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:23.198 DEBUG:boto:Data: 2022-09-12T20:14:23.198 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.199 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.199 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.200 DEBUG:boto:Params: {} 2022-09-12T20:14:23.200 DEBUG:boto:Token: None 2022-09-12T20:14:23.201 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:23.202 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KT8s7QMcqPkVXaR8OcMsK4Flo2k= 2022-09-12T20:14:23.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KT8s7QMcqPkVXaR8OcMsK4Flo2k=', 'Content-Length': '0'} 2022-09-12T20:14:23.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091ac00e3a02f0d39-00631f931f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.246 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:23.247 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:23.247 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:23.248 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:14:23.248 DEBUG:boto:Method: GET 2022-09-12T20:14:23.249 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:23.249 DEBUG:boto:Data: 2022-09-12T20:14:23.250 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.250 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.251 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.252 DEBUG:boto:Params: {} 2022-09-12T20:14:23.252 DEBUG:boto:Token: None 2022-09-12T20:14:23.253 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:14:23.253 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KT8s7QMcqPkVXaR8OcMsK4Flo2k= 2022-09-12T20:14:23.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KT8s7QMcqPkVXaR8OcMsK4Flo2k=', 'Content-Length': '0'} 2022-09-12T20:14:23.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f96453486db88f7-00631f931f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.257 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:23.258 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:23.258 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:14:23.259 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:14:23.259 DEBUG:boto:Method: GET 2022-09-12T20:14:23.260 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:14:23.260 DEBUG:boto:Data: 2022-09-12T20:14:23.261 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.261 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.262 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.263 DEBUG:boto:Params: {} 2022-09-12T20:14:23.263 DEBUG:boto:Token: None 2022-09-12T20:14:23.264 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/ 2022-09-12T20:14:23.265 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo= 2022-09-12T20:14:23.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo=', 'Content-Length': '0'} 2022-09-12T20:14:23.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015c3bc11f3d1eb2d-00631f931f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.269 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:23.270 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:23.271 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:23.271 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:14:23.272 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:14:23.272 DEBUG:boto:Method: GET 2022-09-12T20:14:23.273 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:14:23.273 DEBUG:boto:Data: 2022-09-12T20:14:23.274 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.274 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.275 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.275 DEBUG:boto:Params: {} 2022-09-12T20:14:23.276 DEBUG:boto:Token: None 2022-09-12T20:14:23.277 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/ 2022-09-12T20:14:23.277 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo= 2022-09-12T20:14:23.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo=', 'Content-Length': '0'} 2022-09-12T20:14:23.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088ba1bfa421f5c0d-00631f931f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.282 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:23.283 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:23.283 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:23.284 DEBUG:boto:Method: HEAD 2022-09-12T20:14:23.284 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:14:23.285 DEBUG:boto:Data: 2022-09-12T20:14:23.285 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.286 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.286 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.287 DEBUG:boto:Params: {} 2022-09-12T20:14:23.287 DEBUG:boto:Token: None 2022-09-12T20:14:23.288 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj34 2022-09-12T20:14:23.289 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4JFcVyl66Oy821MMYRpgd5wrwlg= 2022-09-12T20:14:23.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4JFcVyl66Oy821MMYRpgd5wrwlg=', 'Content-Length': '0'} 2022-09-12T20:14:23.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027d0757a2e600f66-00631f931f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.292 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:23.293 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:23.294 DEBUG:boto:Method: HEAD 2022-09-12T20:14:23.294 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:14:23.295 DEBUG:boto:Data: 2022-09-12T20:14:23.295 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.296 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.296 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.297 DEBUG:boto:Params: {} 2022-09-12T20:14:23.297 DEBUG:boto:Token: None 2022-09-12T20:14:23.298 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj34 2022-09-12T20:14:23.298 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4JFcVyl66Oy821MMYRpgd5wrwlg= 2022-09-12T20:14:23.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4JFcVyl66Oy821MMYRpgd5wrwlg=', 'Content-Length': '0'} 2022-09-12T20:14:23.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b31c353ab56758f7-00631f931f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:14:23.302 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:23.303 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:23.303 DEBUG:boto:Method: GET 2022-09-12T20:14:23.304 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:14:23.305 DEBUG:boto:Data: 2022-09-12T20:14:23.305 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.306 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.306 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.307 DEBUG:boto:Params: {} 2022-09-12T20:14:23.307 DEBUG:boto:Token: None 2022-09-12T20:14:23.308 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj34 2022-09-12T20:14:23.308 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HFQFpWdq14iNJVmmz866lJ5O7LI= 2022-09-12T20:14:23.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HFQFpWdq14iNJVmmz866lJ5O7LI=', 'Content-Length': '0'} 2022-09-12T20:14:23.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030958dc092c53046-00631f931f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.312 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:14:23.313 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:14:23.313 DEBUG:boto:Method: GET 2022-09-12T20:14:23.314 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:14:23.314 DEBUG:boto:Data: 2022-09-12T20:14:23.315 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.315 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.316 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.316 DEBUG:boto:Params: {} 2022-09-12T20:14:23.317 DEBUG:boto:Token: None 2022-09-12T20:14:23.318 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj34 2022-09-12T20:14:23.318 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HFQFpWdq14iNJVmmz866lJ5O7LI= 2022-09-12T20:14:23.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HFQFpWdq14iNJVmmz866lJ5O7LI=', 'Content-Length': '0'} 2022-09-12T20:14:23.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c7cc29293a1d65d-00631f931f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.322 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:23.323 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:23.323 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:14:23.324 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:14:23.324 DEBUG:boto:Method: GET 2022-09-12T20:14:23.325 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:14:23.325 DEBUG:boto:Data: 2022-09-12T20:14:23.326 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.326 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.327 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.327 DEBUG:boto:Params: {} 2022-09-12T20:14:23.328 DEBUG:boto:Token: None 2022-09-12T20:14:23.328 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/ 2022-09-12T20:14:23.329 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo= 2022-09-12T20:14:23.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo=', 'Content-Length': '0'} 2022-09-12T20:14:23.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000589f011dfb8a0e2b-00631f931f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.333 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:23.334 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:23.335 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:23.335 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:14:23.336 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:14:23.336 DEBUG:boto:Method: GET 2022-09-12T20:14:23.337 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:14:23.337 DEBUG:boto:Data: 2022-09-12T20:14:23.338 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.338 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.339 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.339 DEBUG:boto:Params: {} 2022-09-12T20:14:23.340 DEBUG:boto:Token: None 2022-09-12T20:14:23.341 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/ 2022-09-12T20:14:23.341 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo= 2022-09-12T20:14:23.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo=', 'Content-Length': '0'} 2022-09-12T20:14:23.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6b2ff2c8854e4aa-00631f931f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.345 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:23.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:14:23.347 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:23.347 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:23.348 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:23.348 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:23.349 DEBUG:boto:Method: GET 2022-09-12T20:14:23.349 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:23.350 DEBUG:boto:Data: 2022-09-12T20:14:23.350 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.351 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.352 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.352 DEBUG:boto:Params: {} 2022-09-12T20:14:23.353 DEBUG:boto:Token: None 2022-09-12T20:14:23.353 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:23.354 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oQpP5O4eHvM30R2801r57JcrXow= 2022-09-12T20:14:23.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oQpP5O4eHvM30R2801r57JcrXow=', 'Content-Length': '0'} 2022-09-12T20:14:23.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000238c86eb2fabf5e-00631f931f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.358 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:23.358 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:23.359 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:23.359 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:23.360 DEBUG:boto:Method: GET 2022-09-12T20:14:23.360 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:23.361 DEBUG:boto:Data: 2022-09-12T20:14:23.361 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.362 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.362 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.363 DEBUG:boto:Params: {} 2022-09-12T20:14:23.364 DEBUG:boto:Token: None 2022-09-12T20:14:23.364 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:23.365 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oQpP5O4eHvM30R2801r57JcrXow= 2022-09-12T20:14:23.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oQpP5O4eHvM30R2801r57JcrXow=', 'Content-Length': '0'} 2022-09-12T20:14:23.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee47ac8c0e074b04-00631f931f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.369 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:23.369 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:23.370 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:23.370 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:23.371 DEBUG:boto:Method: HEAD 2022-09-12T20:14:23.371 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:23.372 DEBUG:boto:Data: 2022-09-12T20:14:23.372 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.373 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.373 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.374 DEBUG:boto:Params: {} 2022-09-12T20:14:23.374 DEBUG:boto:Token: None 2022-09-12T20:14:23.375 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:23.376 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3Cff98hCyGl6ZamY7FdkPU23pGk= 2022-09-12T20:14:23.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3Cff98hCyGl6ZamY7FdkPU23pGk=', 'Content-Length': '0'} 2022-09-12T20:14:23.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3e2c191089cf336-00631f931f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.379 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:23.379 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:23.380 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:23.380 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:23.381 DEBUG:boto:Method: HEAD 2022-09-12T20:14:23.381 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:23.382 DEBUG:boto:Data: 2022-09-12T20:14:23.382 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.383 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.383 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.384 DEBUG:boto:Params: {} 2022-09-12T20:14:23.384 DEBUG:boto:Token: None 2022-09-12T20:14:23.385 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:23.385 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3Cff98hCyGl6ZamY7FdkPU23pGk= 2022-09-12T20:14:23.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3Cff98hCyGl6ZamY7FdkPU23pGk=', 'Content-Length': '0'} 2022-09-12T20:14:23.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b2bc57160b0d6e3-00631f931f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:14:23.389 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:23.390 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:23.391 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:23.391 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:23.392 DEBUG:boto:Method: GET 2022-09-12T20:14:23.392 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:23.393 DEBUG:boto:Data: 2022-09-12T20:14:23.393 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.394 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.394 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.395 DEBUG:boto:Params: {} 2022-09-12T20:14:23.395 DEBUG:boto:Token: None 2022-09-12T20:14:23.396 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:23.397 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oQpP5O4eHvM30R2801r57JcrXow= 2022-09-12T20:14:23.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oQpP5O4eHvM30R2801r57JcrXow=', 'Content-Length': '0'} 2022-09-12T20:14:23.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043f083c3b6f46d4b-00631f931f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.400 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:23.400 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:23.401 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:23.401 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:14:23.402 DEBUG:boto:Method: GET 2022-09-12T20:14:23.402 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:23.403 DEBUG:boto:Data: 2022-09-12T20:14:23.404 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.404 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.405 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.405 DEBUG:boto:Params: {} 2022-09-12T20:14:23.406 DEBUG:boto:Token: None 2022-09-12T20:14:23.406 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:14:23.407 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oQpP5O4eHvM30R2801r57JcrXow= 2022-09-12T20:14:23.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oQpP5O4eHvM30R2801r57JcrXow=', 'Content-Length': '0'} 2022-09-12T20:14:23.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029b2221834f0d100-00631f931f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.411 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:23.411 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:23.412 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:14:23.412 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:14:23.413 DEBUG:boto:Method: GET 2022-09-12T20:14:23.413 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:14:23.414 DEBUG:boto:Data: 2022-09-12T20:14:23.414 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.415 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.415 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.416 DEBUG:boto:Params: {} 2022-09-12T20:14:23.416 DEBUG:boto:Token: None 2022-09-12T20:14:23.417 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/ 2022-09-12T20:14:23.418 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo= 2022-09-12T20:14:23.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo=', 'Content-Length': '0'} 2022-09-12T20:14:23.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045197a82030f3d0d-00631f931f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.422 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:23.422 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:23.423 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:23.423 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:14:23.424 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:14:23.424 DEBUG:boto:Method: GET 2022-09-12T20:14:23.425 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:14:23.426 DEBUG:boto:Data: 2022-09-12T20:14:23.426 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.427 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.427 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.428 DEBUG:boto:Params: {} 2022-09-12T20:14:23.428 DEBUG:boto:Token: None 2022-09-12T20:14:23.429 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/ 2022-09-12T20:14:23.429 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo= 2022-09-12T20:14:23.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo=', 'Content-Length': '0'} 2022-09-12T20:14:23.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbbb2237a69dac01-00631f931f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.433 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:23.434 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:23.435 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:23.435 DEBUG:boto:Method: HEAD 2022-09-12T20:14:23.436 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:14:23.436 DEBUG:boto:Data: 2022-09-12T20:14:23.437 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.437 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.438 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.438 DEBUG:boto:Params: {} 2022-09-12T20:14:23.439 DEBUG:boto:Token: None 2022-09-12T20:14:23.439 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj38 2022-09-12T20:14:23.440 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A7AVKWPAYjMX1s0VGxkCXGjprn8= 2022-09-12T20:14:23.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A7AVKWPAYjMX1s0VGxkCXGjprn8=', 'Content-Length': '0'} 2022-09-12T20:14:23.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081c0f5693f5dfbfc-00631f931f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.443 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:23.444 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:23.444 DEBUG:boto:Method: HEAD 2022-09-12T20:14:23.445 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:14:23.445 DEBUG:boto:Data: 2022-09-12T20:14:23.446 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.446 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.447 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.447 DEBUG:boto:Params: {} 2022-09-12T20:14:23.448 DEBUG:boto:Token: None 2022-09-12T20:14:23.448 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj38 2022-09-12T20:14:23.449 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A7AVKWPAYjMX1s0VGxkCXGjprn8= 2022-09-12T20:14:23.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A7AVKWPAYjMX1s0VGxkCXGjprn8=', 'Content-Length': '0'} 2022-09-12T20:14:23.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031cd271e6052a0b2-00631f931f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:14:23.453 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:23.453 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:23.454 DEBUG:boto:Method: GET 2022-09-12T20:14:23.454 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:14:23.455 DEBUG:boto:Data: 2022-09-12T20:14:23.455 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.456 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.456 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.457 DEBUG:boto:Params: {} 2022-09-12T20:14:23.457 DEBUG:boto:Token: None 2022-09-12T20:14:23.458 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj38 2022-09-12T20:14:23.459 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nUkEKu+OmjIC3a5td72OUtzylDM= 2022-09-12T20:14:23.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nUkEKu+OmjIC3a5td72OUtzylDM=', 'Content-Length': '0'} 2022-09-12T20:14:23.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095cab9d409b39878-00631f931f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.506 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:14:23.507 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:14:23.507 DEBUG:boto:Method: GET 2022-09-12T20:14:23.508 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:14:23.508 DEBUG:boto:Data: 2022-09-12T20:14:23.509 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.509 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.510 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.510 DEBUG:boto:Params: {} 2022-09-12T20:14:23.511 DEBUG:boto:Token: None 2022-09-12T20:14:23.512 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj38 2022-09-12T20:14:23.512 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nUkEKu+OmjIC3a5td72OUtzylDM= 2022-09-12T20:14:23.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nUkEKu+OmjIC3a5td72OUtzylDM=', 'Content-Length': '0'} 2022-09-12T20:14:23.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019e99f2e89fd2b2b-00631f931f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.516 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:23.516 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:23.517 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:14:23.517 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:14:23.518 DEBUG:boto:Method: GET 2022-09-12T20:14:23.519 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:14:23.519 DEBUG:boto:Data: 2022-09-12T20:14:23.520 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.520 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.521 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.521 DEBUG:boto:Params: {} 2022-09-12T20:14:23.522 DEBUG:boto:Token: None 2022-09-12T20:14:23.522 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/ 2022-09-12T20:14:23.523 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo= 2022-09-12T20:14:23.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo=', 'Content-Length': '0'} 2022-09-12T20:14:23.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b017f70b2a71e05-00631f931f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.527 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:23.527 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:23.528 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:23.528 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:14:23.529 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:14:23.529 DEBUG:boto:Method: GET 2022-09-12T20:14:23.530 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:14:23.531 DEBUG:boto:Data: 2022-09-12T20:14:23.531 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.532 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.532 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.533 DEBUG:boto:Params: {} 2022-09-12T20:14:23.533 DEBUG:boto:Token: None 2022-09-12T20:14:23.534 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/ 2022-09-12T20:14:23.534 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo= 2022-09-12T20:14:23.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo=', 'Content-Length': '0'} 2022-09-12T20:14:23.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5ff6181eabb1129-00631f931f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.538 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:23.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:14:23.539 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:23.540 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:23.540 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:23.541 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:23.541 DEBUG:boto:Method: GET 2022-09-12T20:14:23.542 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:23.542 DEBUG:boto:Data: 2022-09-12T20:14:23.543 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.543 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.544 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.544 DEBUG:boto:Params: {} 2022-09-12T20:14:23.545 DEBUG:boto:Token: None 2022-09-12T20:14:23.546 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:23.546 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OrcRMshhela8FFSJw+Cj/79C2O0= 2022-09-12T20:14:23.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OrcRMshhela8FFSJw+Cj/79C2O0=', 'Content-Length': '0'} 2022-09-12T20:14:23.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cab4c68519dad1db-00631f931f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.549 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:23.550 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:23.551 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:23.551 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:23.552 DEBUG:boto:Method: GET 2022-09-12T20:14:23.552 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:23.553 DEBUG:boto:Data: 2022-09-12T20:14:23.553 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.554 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.554 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.555 DEBUG:boto:Params: {} 2022-09-12T20:14:23.555 DEBUG:boto:Token: None 2022-09-12T20:14:23.556 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:23.556 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OrcRMshhela8FFSJw+Cj/79C2O0= 2022-09-12T20:14:23.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OrcRMshhela8FFSJw+Cj/79C2O0=', 'Content-Length': '0'} 2022-09-12T20:14:23.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000972ecf2a278d0867-00631f931f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.560 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:23.561 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:23.561 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:23.562 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:23.562 DEBUG:boto:Method: HEAD 2022-09-12T20:14:23.563 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:23.563 DEBUG:boto:Data: 2022-09-12T20:14:23.564 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.564 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.565 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.565 DEBUG:boto:Params: {} 2022-09-12T20:14:23.566 DEBUG:boto:Token: None 2022-09-12T20:14:23.566 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:23.567 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YWvyjaFREib19ii04n0GzpilSlQ= 2022-09-12T20:14:23.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YWvyjaFREib19ii04n0GzpilSlQ=', 'Content-Length': '0'} 2022-09-12T20:14:23.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006179af1ef1cae68a-00631f931f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.570 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:23.571 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:23.571 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:23.572 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:23.572 DEBUG:boto:Method: HEAD 2022-09-12T20:14:23.573 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:23.573 DEBUG:boto:Data: 2022-09-12T20:14:23.574 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.574 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.575 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.575 DEBUG:boto:Params: {} 2022-09-12T20:14:23.576 DEBUG:boto:Token: None 2022-09-12T20:14:23.577 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:23.577 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YWvyjaFREib19ii04n0GzpilSlQ= 2022-09-12T20:14:23.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YWvyjaFREib19ii04n0GzpilSlQ=', 'Content-Length': '0'} 2022-09-12T20:14:23.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e37b5b01e13e51b-00631f931f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:14:23.581 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:23.581 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:23.582 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:23.582 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:23.583 DEBUG:boto:Method: GET 2022-09-12T20:14:23.583 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:23.584 DEBUG:boto:Data: 2022-09-12T20:14:23.584 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.585 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.585 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.586 DEBUG:boto:Params: {} 2022-09-12T20:14:23.586 DEBUG:boto:Token: None 2022-09-12T20:14:23.587 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:23.588 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OrcRMshhela8FFSJw+Cj/79C2O0= 2022-09-12T20:14:23.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OrcRMshhela8FFSJw+Cj/79C2O0=', 'Content-Length': '0'} 2022-09-12T20:14:23.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006172966bce1a29c5-00631f931f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.591 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:23.592 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:23.592 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:23.593 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:14:23.593 DEBUG:boto:Method: GET 2022-09-12T20:14:23.594 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:23.594 DEBUG:boto:Data: 2022-09-12T20:14:23.595 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.595 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.596 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.596 DEBUG:boto:Params: {} 2022-09-12T20:14:23.597 DEBUG:boto:Token: None 2022-09-12T20:14:23.597 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:14:23.598 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OrcRMshhela8FFSJw+Cj/79C2O0= 2022-09-12T20:14:23.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OrcRMshhela8FFSJw+Cj/79C2O0=', 'Content-Length': '0'} 2022-09-12T20:14:23.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cfbd1a21d11d463-00631f931f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.601 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:23.602 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:23.603 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:14:23.603 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:14:23.604 DEBUG:boto:Method: GET 2022-09-12T20:14:23.604 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:14:23.605 DEBUG:boto:Data: 2022-09-12T20:14:23.605 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.606 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.606 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.607 DEBUG:boto:Params: {} 2022-09-12T20:14:23.607 DEBUG:boto:Token: None 2022-09-12T20:14:23.608 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/ 2022-09-12T20:14:23.608 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo= 2022-09-12T20:14:23.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo=', 'Content-Length': '0'} 2022-09-12T20:14:23.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3f71a27d7d381e2-00631f931f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.654 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:23.655 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:23.655 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:23.656 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:14:23.656 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:14:23.657 DEBUG:boto:Method: GET 2022-09-12T20:14:23.657 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:14:23.658 DEBUG:boto:Data: 2022-09-12T20:14:23.658 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.659 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.659 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.660 DEBUG:boto:Params: {} 2022-09-12T20:14:23.661 DEBUG:boto:Token: None 2022-09-12T20:14:23.661 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/ 2022-09-12T20:14:23.662 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo= 2022-09-12T20:14:23.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo=', 'Content-Length': '0'} 2022-09-12T20:14:23.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba44a586f88eba46-00631f931f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.666 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:23.666 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:23.667 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:23.667 DEBUG:boto:Method: HEAD 2022-09-12T20:14:23.668 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:14:23.668 DEBUG:boto:Data: 2022-09-12T20:14:23.669 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.669 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.670 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.670 DEBUG:boto:Params: {} 2022-09-12T20:14:23.671 DEBUG:boto:Token: None 2022-09-12T20:14:23.671 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj44 2022-09-12T20:14:23.672 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TO4lYbaU/hEFu0oqe9oIkae14Ow= 2022-09-12T20:14:23.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TO4lYbaU/hEFu0oqe9oIkae14Ow=', 'Content-Length': '0'} 2022-09-12T20:14:23.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ce3874a4801a0e9-00631f931f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.675 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:23.676 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:23.676 DEBUG:boto:Method: HEAD 2022-09-12T20:14:23.677 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:14:23.677 DEBUG:boto:Data: 2022-09-12T20:14:23.678 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.678 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.679 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.679 DEBUG:boto:Params: {} 2022-09-12T20:14:23.680 DEBUG:boto:Token: None 2022-09-12T20:14:23.680 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj44 2022-09-12T20:14:23.681 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TO4lYbaU/hEFu0oqe9oIkae14Ow= 2022-09-12T20:14:23.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TO4lYbaU/hEFu0oqe9oIkae14Ow=', 'Content-Length': '0'} 2022-09-12T20:14:23.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9e8b344b1029ac5-00631f931f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:14:23.684 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:23.685 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:23.685 DEBUG:boto:Method: GET 2022-09-12T20:14:23.686 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:14:23.686 DEBUG:boto:Data: 2022-09-12T20:14:23.687 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.687 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.688 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.688 DEBUG:boto:Params: {} 2022-09-12T20:14:23.689 DEBUG:boto:Token: None 2022-09-12T20:14:23.690 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj44 2022-09-12T20:14:23.690 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7nNOcGBzI4VQO12iFgkbbybJKWE= 2022-09-12T20:14:23.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7nNOcGBzI4VQO12iFgkbbybJKWE=', 'Content-Length': '0'} 2022-09-12T20:14:23.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1f02a451510eec6-00631f931f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.693 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:14:23.694 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:14:23.695 DEBUG:boto:Method: GET 2022-09-12T20:14:23.695 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:14:23.696 DEBUG:boto:Data: 2022-09-12T20:14:23.696 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.697 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.697 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.698 DEBUG:boto:Params: {} 2022-09-12T20:14:23.698 DEBUG:boto:Token: None 2022-09-12T20:14:23.699 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj44 2022-09-12T20:14:23.699 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7nNOcGBzI4VQO12iFgkbbybJKWE= 2022-09-12T20:14:23.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7nNOcGBzI4VQO12iFgkbbybJKWE=', 'Content-Length': '0'} 2022-09-12T20:14:23.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033a2676407dac72a-00631f931f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.703 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:23.703 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:23.704 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:14:23.704 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:14:23.705 DEBUG:boto:Method: GET 2022-09-12T20:14:23.706 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:14:23.706 DEBUG:boto:Data: 2022-09-12T20:14:23.707 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.707 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.708 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.708 DEBUG:boto:Params: {} 2022-09-12T20:14:23.709 DEBUG:boto:Token: None 2022-09-12T20:14:23.709 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/ 2022-09-12T20:14:23.710 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo= 2022-09-12T20:14:23.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo=', 'Content-Length': '0'} 2022-09-12T20:14:23.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c330e6bc724d463-00631f931f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.714 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:23.714 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:23.715 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:23.715 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:14:23.716 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:14:23.716 DEBUG:boto:Method: GET 2022-09-12T20:14:23.717 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:14:23.718 DEBUG:boto:Data: 2022-09-12T20:14:23.718 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.719 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.719 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.720 DEBUG:boto:Params: {} 2022-09-12T20:14:23.720 DEBUG:boto:Token: None 2022-09-12T20:14:23.721 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/ 2022-09-12T20:14:23.721 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo= 2022-09-12T20:14:23.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo=', 'Content-Length': '0'} 2022-09-12T20:14:23.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072a07d4e42852203-00631f931f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.766 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:23.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:14:23.767 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:23.768 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:23.768 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:23.769 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:23.769 DEBUG:boto:Method: GET 2022-09-12T20:14:23.770 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:23.770 DEBUG:boto:Data: 2022-09-12T20:14:23.771 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.771 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.772 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.772 DEBUG:boto:Params: {} 2022-09-12T20:14:23.773 DEBUG:boto:Token: None 2022-09-12T20:14:23.774 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:23.774 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1irUvkiflhSfXIN0LH0D3Ybobv4= 2022-09-12T20:14:23.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1irUvkiflhSfXIN0LH0D3Ybobv4=', 'Content-Length': '0'} 2022-09-12T20:14:23.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003217ec81b8f19a42-00631f931f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.778 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:23.778 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:23.779 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:23.779 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:23.780 DEBUG:boto:Method: GET 2022-09-12T20:14:23.780 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:23.781 DEBUG:boto:Data: 2022-09-12T20:14:23.781 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.782 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.782 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.783 DEBUG:boto:Params: {} 2022-09-12T20:14:23.784 DEBUG:boto:Token: None 2022-09-12T20:14:23.784 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:23.785 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1irUvkiflhSfXIN0LH0D3Ybobv4= 2022-09-12T20:14:23.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1irUvkiflhSfXIN0LH0D3Ybobv4=', 'Content-Length': '0'} 2022-09-12T20:14:23.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e8994603ab2a65e-00631f931f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.788 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:23.789 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:23.789 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:23.790 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:23.790 DEBUG:boto:Method: HEAD 2022-09-12T20:14:23.791 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:23.791 DEBUG:boto:Data: 2022-09-12T20:14:23.792 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.792 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.793 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.793 DEBUG:boto:Params: {} 2022-09-12T20:14:23.794 DEBUG:boto:Token: None 2022-09-12T20:14:23.794 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:23.795 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gK+zZHWEBrMwLSCnTV+VZO62BMM= 2022-09-12T20:14:23.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gK+zZHWEBrMwLSCnTV+VZO62BMM=', 'Content-Length': '0'} 2022-09-12T20:14:23.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004596656c936124d6-00631f931f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.798 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:23.798 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:23.799 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:23.799 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:23.800 DEBUG:boto:Method: HEAD 2022-09-12T20:14:23.800 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:23.801 DEBUG:boto:Data: 2022-09-12T20:14:23.801 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.802 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.802 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.803 DEBUG:boto:Params: {} 2022-09-12T20:14:23.808 DEBUG:boto:Token: None 2022-09-12T20:14:23.809 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:23.810 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gK+zZHWEBrMwLSCnTV+VZO62BMM= 2022-09-12T20:14:23.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gK+zZHWEBrMwLSCnTV+VZO62BMM=', 'Content-Length': '0'} 2022-09-12T20:14:23.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006737e6baf13c152c-00631f931f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:14:23.814 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:23.814 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:23.815 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:23.815 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:23.816 DEBUG:boto:Method: GET 2022-09-12T20:14:23.816 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:23.817 DEBUG:boto:Data: 2022-09-12T20:14:23.817 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.818 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.818 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.819 DEBUG:boto:Params: {} 2022-09-12T20:14:23.819 DEBUG:boto:Token: None 2022-09-12T20:14:23.820 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:23.820 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1irUvkiflhSfXIN0LH0D3Ybobv4= 2022-09-12T20:14:23.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1irUvkiflhSfXIN0LH0D3Ybobv4=', 'Content-Length': '0'} 2022-09-12T20:14:23.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080141355cbb4a621-00631f931f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.824 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:23.825 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:23.825 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:23.826 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:14:23.826 DEBUG:boto:Method: GET 2022-09-12T20:14:23.827 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:23.827 DEBUG:boto:Data: 2022-09-12T20:14:23.828 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.828 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.829 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.829 DEBUG:boto:Params: {} 2022-09-12T20:14:23.830 DEBUG:boto:Token: None 2022-09-12T20:14:23.830 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:14:23.831 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1irUvkiflhSfXIN0LH0D3Ybobv4= 2022-09-12T20:14:23.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1irUvkiflhSfXIN0LH0D3Ybobv4=', 'Content-Length': '0'} 2022-09-12T20:14:23.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091ab2a52b224d44e-00631f931f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.834 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:23.835 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:23.835 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:14:23.836 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:14:23.836 DEBUG:boto:Method: GET 2022-09-12T20:14:23.837 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:14:23.837 DEBUG:boto:Data: 2022-09-12T20:14:23.838 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.838 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.839 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.839 DEBUG:boto:Params: {} 2022-09-12T20:14:23.840 DEBUG:boto:Token: None 2022-09-12T20:14:23.840 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/ 2022-09-12T20:14:23.841 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo= 2022-09-12T20:14:23.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo=', 'Content-Length': '0'} 2022-09-12T20:14:23.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094e1cbee14a2210d-00631f931f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.845 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:23.846 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:23.846 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:23.847 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:14:23.848 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:14:23.848 DEBUG:boto:Method: GET 2022-09-12T20:14:23.849 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:14:23.849 DEBUG:boto:Data: 2022-09-12T20:14:23.850 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.850 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.851 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.851 DEBUG:boto:Params: {} 2022-09-12T20:14:23.852 DEBUG:boto:Token: None 2022-09-12T20:14:23.852 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/ 2022-09-12T20:14:23.853 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo= 2022-09-12T20:14:23.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo=', 'Content-Length': '0'} 2022-09-12T20:14:23.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c70b9d167ac49e1-00631f931f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.857 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:23.857 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:23.858 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:23.858 DEBUG:boto:Method: HEAD 2022-09-12T20:14:23.859 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:14:23.859 DEBUG:boto:Data: 2022-09-12T20:14:23.860 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.860 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.861 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.861 DEBUG:boto:Params: {} 2022-09-12T20:14:23.862 DEBUG:boto:Token: None 2022-09-12T20:14:23.862 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj46 2022-09-12T20:14:23.863 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N8OR1hyoDT/enUtECNGfzUsL0Ks= 2022-09-12T20:14:23.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N8OR1hyoDT/enUtECNGfzUsL0Ks=', 'Content-Length': '0'} 2022-09-12T20:14:23.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e26243d4b9211345-00631f931f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.866 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:23.867 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:23.867 DEBUG:boto:Method: HEAD 2022-09-12T20:14:23.868 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:14:23.868 DEBUG:boto:Data: 2022-09-12T20:14:23.869 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.869 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.870 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.870 DEBUG:boto:Params: {} 2022-09-12T20:14:23.871 DEBUG:boto:Token: None 2022-09-12T20:14:23.871 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj46 2022-09-12T20:14:23.872 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N8OR1hyoDT/enUtECNGfzUsL0Ks= 2022-09-12T20:14:23.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N8OR1hyoDT/enUtECNGfzUsL0Ks=', 'Content-Length': '0'} 2022-09-12T20:14:23.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040f3460043258537-00631f931f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:14:23.875 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:23.876 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:23.876 DEBUG:boto:Method: GET 2022-09-12T20:14:23.877 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:14:23.877 DEBUG:boto:Data: 2022-09-12T20:14:23.878 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.878 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.879 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.879 DEBUG:boto:Params: {} 2022-09-12T20:14:23.880 DEBUG:boto:Token: None 2022-09-12T20:14:23.881 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj46 2022-09-12T20:14:23.881 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pzC6iBJQX+yd8lhZxoNB5r7ZRuM= 2022-09-12T20:14:23.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pzC6iBJQX+yd8lhZxoNB5r7ZRuM=', 'Content-Length': '0'} 2022-09-12T20:14:23.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6dc72a22a6619d3-00631f931f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.884 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:14:23.885 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:14:23.885 DEBUG:boto:Method: GET 2022-09-12T20:14:23.886 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:14:23.887 DEBUG:boto:Data: 2022-09-12T20:14:23.887 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.888 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.888 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.889 DEBUG:boto:Params: {} 2022-09-12T20:14:23.889 DEBUG:boto:Token: None 2022-09-12T20:14:23.890 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj46 2022-09-12T20:14:23.890 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pzC6iBJQX+yd8lhZxoNB5r7ZRuM= 2022-09-12T20:14:23.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pzC6iBJQX+yd8lhZxoNB5r7ZRuM=', 'Content-Length': '0'} 2022-09-12T20:14:23.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000597f20926c6bc43e-00631f931f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.894 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:23.894 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:23.895 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:14:23.895 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:14:23.896 DEBUG:boto:Method: GET 2022-09-12T20:14:23.896 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:14:23.897 DEBUG:boto:Data: 2022-09-12T20:14:23.897 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.898 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.898 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.899 DEBUG:boto:Params: {} 2022-09-12T20:14:23.900 DEBUG:boto:Token: None 2022-09-12T20:14:23.900 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/ 2022-09-12T20:14:23.901 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo= 2022-09-12T20:14:23.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo=', 'Content-Length': '0'} 2022-09-12T20:14:23.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082b25ea78e016485-00631f931f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.905 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:23.906 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:23.906 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:23.907 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:14:23.907 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:14:23.908 DEBUG:boto:Method: GET 2022-09-12T20:14:23.908 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:14:23.909 DEBUG:boto:Data: 2022-09-12T20:14:23.909 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.910 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.910 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.911 DEBUG:boto:Params: {} 2022-09-12T20:14:23.911 DEBUG:boto:Token: None 2022-09-12T20:14:23.912 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/ 2022-09-12T20:14:23.913 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo= 2022-09-12T20:14:23.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yjRZ10L/E7MfNXFb4XKZ2qAp7Mo=', 'Content-Length': '0'} 2022-09-12T20:14:23.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069ec7d6b228bb6d2-00631f931f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.916 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:23.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:14:23.918 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:23.918 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:23.919 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:23.919 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:23.920 DEBUG:boto:Method: GET 2022-09-12T20:14:23.920 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:23.921 DEBUG:boto:Data: 2022-09-12T20:14:23.921 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.922 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.922 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.923 DEBUG:boto:Params: {} 2022-09-12T20:14:23.923 DEBUG:boto:Token: None 2022-09-12T20:14:23.924 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:23.925 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aDIaGXRPW+XEfeB93J7ktLEp4hk= 2022-09-12T20:14:23.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aDIaGXRPW+XEfeB93J7ktLEp4hk=', 'Content-Length': '0'} 2022-09-12T20:14:23.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae74dff6bc66a0e3-00631f931f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.928 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:23.929 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:23.929 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:23.930 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:23.930 DEBUG:boto:Method: GET 2022-09-12T20:14:23.931 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:23.931 DEBUG:boto:Data: 2022-09-12T20:14:23.932 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.932 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.933 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.933 DEBUG:boto:Params: {} 2022-09-12T20:14:23.934 DEBUG:boto:Token: None 2022-09-12T20:14:23.934 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:23.935 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aDIaGXRPW+XEfeB93J7ktLEp4hk= 2022-09-12T20:14:23.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aDIaGXRPW+XEfeB93J7ktLEp4hk=', 'Content-Length': '0'} 2022-09-12T20:14:23.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c15497ffac3f0526-00631f931f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.939 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:23.939 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:23.940 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:23.940 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:23.941 DEBUG:boto:Method: HEAD 2022-09-12T20:14:23.941 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:23.942 DEBUG:boto:Data: 2022-09-12T20:14:23.942 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.943 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.943 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.944 DEBUG:boto:Params: {} 2022-09-12T20:14:23.944 DEBUG:boto:Token: None 2022-09-12T20:14:23.945 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:23.945 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+0IjyPcR2w1mXivyjtScKgJd790= 2022-09-12T20:14:23.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+0IjyPcR2w1mXivyjtScKgJd790=', 'Content-Length': '0'} 2022-09-12T20:14:23.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005087419e7dae6c83-00631f931f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.949 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:23.950 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:23.950 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:23.951 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:23.951 DEBUG:boto:Method: HEAD 2022-09-12T20:14:23.952 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:23.952 DEBUG:boto:Data: 2022-09-12T20:14:23.953 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.953 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.954 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.954 DEBUG:boto:Params: {} 2022-09-12T20:14:23.955 DEBUG:boto:Token: None 2022-09-12T20:14:23.955 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:23.956 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+0IjyPcR2w1mXivyjtScKgJd790= 2022-09-12T20:14:23.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+0IjyPcR2w1mXivyjtScKgJd790=', 'Content-Length': '0'} 2022-09-12T20:14:23.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b13809fee76c453-00631f931f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:14:23.960 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:23.961 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:23.961 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:23.962 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:23.962 DEBUG:boto:Method: GET 2022-09-12T20:14:23.963 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:23.963 DEBUG:boto:Data: 2022-09-12T20:14:23.964 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.964 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:23.965 DEBUG:boto:Port: 8001 2022-09-12T20:14:23.965 DEBUG:boto:Params: {} 2022-09-12T20:14:23.966 DEBUG:boto:Token: None 2022-09-12T20:14:23.967 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:23.967 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aDIaGXRPW+XEfeB93J7ktLEp4hk= 2022-09-12T20:14:23.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aDIaGXRPW+XEfeB93J7ktLEp4hk=', 'Content-Length': '0'} 2022-09-12T20:14:23.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000858d62ddf47bd82-00631f931f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:23.971 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:23.971 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:23.972 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:23.973 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:14:23.973 DEBUG:boto:Method: GET 2022-09-12T20:14:23.974 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:23.974 DEBUG:boto:Data: 2022-09-12T20:14:23.975 DEBUG:boto:Headers: {} 2022-09-12T20:14:23.975 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:23.976 DEBUG:boto:Port: 8000 2022-09-12T20:14:23.976 DEBUG:boto:Params: {} 2022-09-12T20:14:23.977 DEBUG:boto:Token: None 2022-09-12T20:14:23.977 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:23 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:14:23.978 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aDIaGXRPW+XEfeB93J7ktLEp4hk= 2022-09-12T20:14:23.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aDIaGXRPW+XEfeB93J7ktLEp4hk=', 'Content-Length': '0'} 2022-09-12T20:14:23.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000872beeb68ddcba72-00631f931f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.022 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:24.023 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:24.023 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:14:24.024 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:14:24.025 DEBUG:boto:Method: GET 2022-09-12T20:14:24.025 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:14:24.026 DEBUG:boto:Data: 2022-09-12T20:14:24.026 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.027 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.027 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.028 DEBUG:boto:Params: {} 2022-09-12T20:14:24.028 DEBUG:boto:Token: None 2022-09-12T20:14:24.029 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/ 2022-09-12T20:14:24.030 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o= 2022-09-12T20:14:24.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o=', 'Content-Length': '0'} 2022-09-12T20:14:24.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040fbf67cb7f83ebd-00631f9320-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.034 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:24.035 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:24.035 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:24.036 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:14:24.037 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:14:24.037 DEBUG:boto:Method: GET 2022-09-12T20:14:24.038 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:14:24.038 DEBUG:boto:Data: 2022-09-12T20:14:24.039 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.039 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.040 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.040 DEBUG:boto:Params: {} 2022-09-12T20:14:24.041 DEBUG:boto:Token: None 2022-09-12T20:14:24.041 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/ 2022-09-12T20:14:24.042 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o= 2022-09-12T20:14:24.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o=', 'Content-Length': '0'} 2022-09-12T20:14:24.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e501cf259f89ce60-00631f9320-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.046 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:24.047 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:24.048 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:24.048 DEBUG:boto:Method: HEAD 2022-09-12T20:14:24.049 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:14:24.049 DEBUG:boto:Data: 2022-09-12T20:14:24.050 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.050 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.051 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.051 DEBUG:boto:Params: {} 2022-09-12T20:14:24.052 DEBUG:boto:Token: None 2022-09-12T20:14:24.053 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj51 2022-09-12T20:14:24.053 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qDA7iYTktkZke5W8tnFNpjdtiJk= 2022-09-12T20:14:24.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qDA7iYTktkZke5W8tnFNpjdtiJk=', 'Content-Length': '0'} 2022-09-12T20:14:24.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004c2ec1d1f2b5dea-00631f9320-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.057 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:24.057 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:24.058 DEBUG:boto:Method: HEAD 2022-09-12T20:14:24.058 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:14:24.059 DEBUG:boto:Data: 2022-09-12T20:14:24.059 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.060 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.060 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.061 DEBUG:boto:Params: {} 2022-09-12T20:14:24.061 DEBUG:boto:Token: None 2022-09-12T20:14:24.062 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj51 2022-09-12T20:14:24.062 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qDA7iYTktkZke5W8tnFNpjdtiJk= 2022-09-12T20:14:24.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qDA7iYTktkZke5W8tnFNpjdtiJk=', 'Content-Length': '0'} 2022-09-12T20:14:24.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000504f0996a8c221f9-00631f9320-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:14:24.066 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:24.067 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:24.068 DEBUG:boto:Method: GET 2022-09-12T20:14:24.068 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:14:24.069 DEBUG:boto:Data: 2022-09-12T20:14:24.069 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.070 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.070 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.071 DEBUG:boto:Params: {} 2022-09-12T20:14:24.071 DEBUG:boto:Token: None 2022-09-12T20:14:24.072 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj51 2022-09-12T20:14:24.073 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nyvjc7RM01iz4B4Uv0c7Tx41aF0= 2022-09-12T20:14:24.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nyvjc7RM01iz4B4Uv0c7Tx41aF0=', 'Content-Length': '0'} 2022-09-12T20:14:24.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa318101a7572359-00631f9320-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.076 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:14:24.077 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:14:24.077 DEBUG:boto:Method: GET 2022-09-12T20:14:24.078 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:14:24.078 DEBUG:boto:Data: 2022-09-12T20:14:24.079 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.079 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.080 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.080 DEBUG:boto:Params: {} 2022-09-12T20:14:24.081 DEBUG:boto:Token: None 2022-09-12T20:14:24.082 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj51 2022-09-12T20:14:24.082 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nyvjc7RM01iz4B4Uv0c7Tx41aF0= 2022-09-12T20:14:24.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nyvjc7RM01iz4B4Uv0c7Tx41aF0=', 'Content-Length': '0'} 2022-09-12T20:14:24.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008f9acf05bab183a-00631f9320-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.086 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:24.086 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:24.087 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:14:24.087 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:14:24.088 DEBUG:boto:Method: GET 2022-09-12T20:14:24.089 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:14:24.089 DEBUG:boto:Data: 2022-09-12T20:14:24.090 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.090 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.091 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.091 DEBUG:boto:Params: {} 2022-09-12T20:14:24.092 DEBUG:boto:Token: None 2022-09-12T20:14:24.092 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/ 2022-09-12T20:14:24.093 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o= 2022-09-12T20:14:24.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o=', 'Content-Length': '0'} 2022-09-12T20:14:24.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080a0aefbe7ff96ad-00631f9320-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.097 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:24.098 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:24.099 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:24.099 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:14:24.100 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:14:24.100 DEBUG:boto:Method: GET 2022-09-12T20:14:24.101 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:14:24.101 DEBUG:boto:Data: 2022-09-12T20:14:24.102 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.102 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.103 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.103 DEBUG:boto:Params: {} 2022-09-12T20:14:24.104 DEBUG:boto:Token: None 2022-09-12T20:14:24.105 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/ 2022-09-12T20:14:24.105 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o= 2022-09-12T20:14:24.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o=', 'Content-Length': '0'} 2022-09-12T20:14:24.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084346b2e0a01d872-00631f9320-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.109 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:24.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:14:24.110 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:24.111 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:24.111 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:24.112 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:24.112 DEBUG:boto:Method: GET 2022-09-12T20:14:24.113 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:24.113 DEBUG:boto:Data: 2022-09-12T20:14:24.114 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.115 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.115 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.116 DEBUG:boto:Params: {} 2022-09-12T20:14:24.116 DEBUG:boto:Token: None 2022-09-12T20:14:24.117 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:24.117 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cg3d/99pX1ePd5wglc5obSCLYUM= 2022-09-12T20:14:24.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cg3d/99pX1ePd5wglc5obSCLYUM=', 'Content-Length': '0'} 2022-09-12T20:14:24.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c62a909f106c2b8-00631f9320-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.121 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:24.122 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:24.122 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:24.123 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:24.123 DEBUG:boto:Method: GET 2022-09-12T20:14:24.124 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:24.124 DEBUG:boto:Data: 2022-09-12T20:14:24.125 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.126 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.126 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.127 DEBUG:boto:Params: {} 2022-09-12T20:14:24.127 DEBUG:boto:Token: None 2022-09-12T20:14:24.128 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:24.128 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cg3d/99pX1ePd5wglc5obSCLYUM= 2022-09-12T20:14:24.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cg3d/99pX1ePd5wglc5obSCLYUM=', 'Content-Length': '0'} 2022-09-12T20:14:24.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017cca7c99b04f6ce-00631f9320-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.132 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:24.132 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:24.133 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:24.133 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:24.134 DEBUG:boto:Method: HEAD 2022-09-12T20:14:24.134 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:24.135 DEBUG:boto:Data: 2022-09-12T20:14:24.135 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.136 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.136 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.137 DEBUG:boto:Params: {} 2022-09-12T20:14:24.137 DEBUG:boto:Token: None 2022-09-12T20:14:24.138 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:24.139 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1a6vTRbscfM5uV9WFj0uE465Z6Q= 2022-09-12T20:14:24.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1a6vTRbscfM5uV9WFj0uE465Z6Q=', 'Content-Length': '0'} 2022-09-12T20:14:24.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e684821670e2dc97-00631f9320-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.142 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:24.143 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:24.143 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:24.144 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:24.145 DEBUG:boto:Method: HEAD 2022-09-12T20:14:24.145 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:24.146 DEBUG:boto:Data: 2022-09-12T20:14:24.146 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.147 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.147 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.148 DEBUG:boto:Params: {} 2022-09-12T20:14:24.148 DEBUG:boto:Token: None 2022-09-12T20:14:24.149 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:24.150 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1a6vTRbscfM5uV9WFj0uE465Z6Q= 2022-09-12T20:14:24.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1a6vTRbscfM5uV9WFj0uE465Z6Q=', 'Content-Length': '0'} 2022-09-12T20:14:24.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf76f11755c41c96-00631f9320-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:14:24.153 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:24.154 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:24.154 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:24.155 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:24.156 DEBUG:boto:Method: GET 2022-09-12T20:14:24.156 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:24.157 DEBUG:boto:Data: 2022-09-12T20:14:24.157 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.158 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.158 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.159 DEBUG:boto:Params: {} 2022-09-12T20:14:24.159 DEBUG:boto:Token: None 2022-09-12T20:14:24.160 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:24.161 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cg3d/99pX1ePd5wglc5obSCLYUM= 2022-09-12T20:14:24.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cg3d/99pX1ePd5wglc5obSCLYUM=', 'Content-Length': '0'} 2022-09-12T20:14:24.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c74f16e0159499ad-00631f9320-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.164 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:24.165 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:24.165 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:24.166 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:14:24.166 DEBUG:boto:Method: GET 2022-09-12T20:14:24.167 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:24.167 DEBUG:boto:Data: 2022-09-12T20:14:24.168 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.169 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.169 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.170 DEBUG:boto:Params: {} 2022-09-12T20:14:24.170 DEBUG:boto:Token: None 2022-09-12T20:14:24.171 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:14:24.172 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cg3d/99pX1ePd5wglc5obSCLYUM= 2022-09-12T20:14:24.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cg3d/99pX1ePd5wglc5obSCLYUM=', 'Content-Length': '0'} 2022-09-12T20:14:24.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000780f4a538775fa07-00631f9320-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.175 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:24.176 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:24.176 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:14:24.177 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:14:24.177 DEBUG:boto:Method: GET 2022-09-12T20:14:24.178 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:14:24.178 DEBUG:boto:Data: 2022-09-12T20:14:24.179 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.180 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.180 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.181 DEBUG:boto:Params: {} 2022-09-12T20:14:24.182 DEBUG:boto:Token: None 2022-09-12T20:14:24.182 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/ 2022-09-12T20:14:24.183 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o= 2022-09-12T20:14:24.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o=', 'Content-Length': '0'} 2022-09-12T20:14:24.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ff5f107176febb2-00631f9320-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.187 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:24.188 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:24.188 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:24.189 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:14:24.189 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:14:24.190 DEBUG:boto:Method: GET 2022-09-12T20:14:24.191 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:14:24.191 DEBUG:boto:Data: 2022-09-12T20:14:24.192 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.192 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.193 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.194 DEBUG:boto:Params: {} 2022-09-12T20:14:24.194 DEBUG:boto:Token: None 2022-09-12T20:14:24.195 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/ 2022-09-12T20:14:24.195 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o= 2022-09-12T20:14:24.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o=', 'Content-Length': '0'} 2022-09-12T20:14:24.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9375f451fc589e3-00631f9320-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.242 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:24.243 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:24.244 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:24.244 DEBUG:boto:Method: HEAD 2022-09-12T20:14:24.245 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:14:24.245 DEBUG:boto:Data: 2022-09-12T20:14:24.246 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.246 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.247 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.247 DEBUG:boto:Params: {} 2022-09-12T20:14:24.248 DEBUG:boto:Token: None 2022-09-12T20:14:24.249 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj55 2022-09-12T20:14:24.249 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OdXYNBsKkP9LTL2N7N7h5nFwXWU= 2022-09-12T20:14:24.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OdXYNBsKkP9LTL2N7N7h5nFwXWU=', 'Content-Length': '0'} 2022-09-12T20:14:24.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035219fe01088908b-00631f9320-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.253 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:24.253 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:24.254 DEBUG:boto:Method: HEAD 2022-09-12T20:14:24.254 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:14:24.255 DEBUG:boto:Data: 2022-09-12T20:14:24.256 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.256 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.257 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.257 DEBUG:boto:Params: {} 2022-09-12T20:14:24.258 DEBUG:boto:Token: None 2022-09-12T20:14:24.258 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj55 2022-09-12T20:14:24.259 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OdXYNBsKkP9LTL2N7N7h5nFwXWU= 2022-09-12T20:14:24.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OdXYNBsKkP9LTL2N7N7h5nFwXWU=', 'Content-Length': '0'} 2022-09-12T20:14:24.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e43123e61b96ceaf-00631f9320-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:14:24.263 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:24.263 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:24.264 DEBUG:boto:Method: GET 2022-09-12T20:14:24.265 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:14:24.265 DEBUG:boto:Data: 2022-09-12T20:14:24.266 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.266 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.267 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.268 DEBUG:boto:Params: {} 2022-09-12T20:14:24.268 DEBUG:boto:Token: None 2022-09-12T20:14:24.269 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj55 2022-09-12T20:14:24.269 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S4M0x67SL/cFKNpJryvBMirjqls= 2022-09-12T20:14:24.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S4M0x67SL/cFKNpJryvBMirjqls=', 'Content-Length': '0'} 2022-09-12T20:14:24.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a7241607c7b93c2-00631f9320-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.273 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:14:24.274 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:14:24.274 DEBUG:boto:Method: GET 2022-09-12T20:14:24.275 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:14:24.275 DEBUG:boto:Data: 2022-09-12T20:14:24.276 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.276 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.277 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.277 DEBUG:boto:Params: {} 2022-09-12T20:14:24.278 DEBUG:boto:Token: None 2022-09-12T20:14:24.279 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj55 2022-09-12T20:14:24.279 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S4M0x67SL/cFKNpJryvBMirjqls= 2022-09-12T20:14:24.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S4M0x67SL/cFKNpJryvBMirjqls=', 'Content-Length': '0'} 2022-09-12T20:14:24.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afad1ec65ecd3a44-00631f9320-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.283 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:24.283 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:24.284 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:14:24.284 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:14:24.285 DEBUG:boto:Method: GET 2022-09-12T20:14:24.285 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:14:24.286 DEBUG:boto:Data: 2022-09-12T20:14:24.286 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.287 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.287 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.288 DEBUG:boto:Params: {} 2022-09-12T20:14:24.288 DEBUG:boto:Token: None 2022-09-12T20:14:24.289 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/ 2022-09-12T20:14:24.290 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o= 2022-09-12T20:14:24.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o=', 'Content-Length': '0'} 2022-09-12T20:14:24.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000976bf6dd048ad9be-00631f9320-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.294 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:24.295 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:24.295 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:24.296 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:14:24.296 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:14:24.297 DEBUG:boto:Method: GET 2022-09-12T20:14:24.297 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:14:24.298 DEBUG:boto:Data: 2022-09-12T20:14:24.298 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.299 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.299 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.300 DEBUG:boto:Params: {} 2022-09-12T20:14:24.301 DEBUG:boto:Token: None 2022-09-12T20:14:24.301 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/ 2022-09-12T20:14:24.302 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o= 2022-09-12T20:14:24.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o=', 'Content-Length': '0'} 2022-09-12T20:14:24.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c809c8cd54984fee-00631f9320-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.306 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:24.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:14:24.307 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:24.308 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:24.308 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:24.309 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:24.309 DEBUG:boto:Method: GET 2022-09-12T20:14:24.310 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:24.311 DEBUG:boto:Data: 2022-09-12T20:14:24.311 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.312 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.312 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.313 DEBUG:boto:Params: {} 2022-09-12T20:14:24.313 DEBUG:boto:Token: None 2022-09-12T20:14:24.314 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:24.314 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XM8g84NdPflOnZudazzEhTjG1so= 2022-09-12T20:14:24.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XM8g84NdPflOnZudazzEhTjG1so=', 'Content-Length': '0'} 2022-09-12T20:14:24.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1bfd4a0eb455196-00631f9320-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.318 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:24.318 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:24.319 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:24.320 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:24.320 DEBUG:boto:Method: GET 2022-09-12T20:14:24.321 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:24.321 DEBUG:boto:Data: 2022-09-12T20:14:24.322 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.322 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.323 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.323 DEBUG:boto:Params: {} 2022-09-12T20:14:24.324 DEBUG:boto:Token: None 2022-09-12T20:14:24.324 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:24.325 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XM8g84NdPflOnZudazzEhTjG1so= 2022-09-12T20:14:24.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XM8g84NdPflOnZudazzEhTjG1so=', 'Content-Length': '0'} 2022-09-12T20:14:24.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066f5dfc09493e0ff-00631f9320-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.328 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:24.329 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:24.330 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:24.330 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:24.331 DEBUG:boto:Method: HEAD 2022-09-12T20:14:24.331 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:24.332 DEBUG:boto:Data: 2022-09-12T20:14:24.332 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.333 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.333 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.334 DEBUG:boto:Params: {} 2022-09-12T20:14:24.335 DEBUG:boto:Token: None 2022-09-12T20:14:24.335 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:24.336 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:w36pc3Sv/70kgSJdyUgQut5RbH4= 2022-09-12T20:14:24.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:w36pc3Sv/70kgSJdyUgQut5RbH4=', 'Content-Length': '0'} 2022-09-12T20:14:24.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9a17a4f6a4768ca-00631f9320-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.339 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:24.339 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:24.340 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:24.340 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:24.341 DEBUG:boto:Method: HEAD 2022-09-12T20:14:24.342 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:24.342 DEBUG:boto:Data: 2022-09-12T20:14:24.343 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.343 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.344 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.344 DEBUG:boto:Params: {} 2022-09-12T20:14:24.345 DEBUG:boto:Token: None 2022-09-12T20:14:24.345 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:24.346 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:w36pc3Sv/70kgSJdyUgQut5RbH4= 2022-09-12T20:14:24.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:w36pc3Sv/70kgSJdyUgQut5RbH4=', 'Content-Length': '0'} 2022-09-12T20:14:24.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028e5c76fe77aa39f-00631f9320-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:14:24.350 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:24.350 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:24.351 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:24.351 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:24.352 DEBUG:boto:Method: GET 2022-09-12T20:14:24.353 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:24.353 DEBUG:boto:Data: 2022-09-12T20:14:24.354 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.354 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.355 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.356 DEBUG:boto:Params: {} 2022-09-12T20:14:24.356 DEBUG:boto:Token: None 2022-09-12T20:14:24.357 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:24.357 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XM8g84NdPflOnZudazzEhTjG1so= 2022-09-12T20:14:24.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XM8g84NdPflOnZudazzEhTjG1so=', 'Content-Length': '0'} 2022-09-12T20:14:24.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070ed914cb607e1b0-00631f9320-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.402 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:24.403 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:24.403 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:24.404 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:14:24.405 DEBUG:boto:Method: GET 2022-09-12T20:14:24.405 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:24.406 DEBUG:boto:Data: 2022-09-12T20:14:24.406 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.407 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.407 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.408 DEBUG:boto:Params: {} 2022-09-12T20:14:24.408 DEBUG:boto:Token: None 2022-09-12T20:14:24.409 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:14:24.410 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XM8g84NdPflOnZudazzEhTjG1so= 2022-09-12T20:14:24.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XM8g84NdPflOnZudazzEhTjG1so=', 'Content-Length': '0'} 2022-09-12T20:14:24.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f07ff19ffda48113-00631f9320-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.413 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:24.414 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:24.415 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:14:24.415 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:14:24.416 DEBUG:boto:Method: GET 2022-09-12T20:14:24.416 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:14:24.417 DEBUG:boto:Data: 2022-09-12T20:14:24.417 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.418 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.418 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.419 DEBUG:boto:Params: {} 2022-09-12T20:14:24.419 DEBUG:boto:Token: None 2022-09-12T20:14:24.420 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/ 2022-09-12T20:14:24.421 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o= 2022-09-12T20:14:24.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o=', 'Content-Length': '0'} 2022-09-12T20:14:24.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d097b149eb96072-00631f9320-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.425 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:24.426 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:24.426 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:24.427 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:14:24.427 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:14:24.428 DEBUG:boto:Method: GET 2022-09-12T20:14:24.428 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:14:24.429 DEBUG:boto:Data: 2022-09-12T20:14:24.429 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.430 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.430 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.431 DEBUG:boto:Params: {} 2022-09-12T20:14:24.432 DEBUG:boto:Token: None 2022-09-12T20:14:24.432 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/ 2022-09-12T20:14:24.433 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o= 2022-09-12T20:14:24.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o=', 'Content-Length': '0'} 2022-09-12T20:14:24.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d93fa5505e016a19-00631f9320-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.437 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:24.438 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:24.438 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:24.439 DEBUG:boto:Method: HEAD 2022-09-12T20:14:24.440 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:14:24.440 DEBUG:boto:Data: 2022-09-12T20:14:24.441 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.441 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.442 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.442 DEBUG:boto:Params: {} 2022-09-12T20:14:24.443 DEBUG:boto:Token: None 2022-09-12T20:14:24.443 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj57 2022-09-12T20:14:24.444 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ooaR5Kahr7z95z2zNk+Wj80Vu/o= 2022-09-12T20:14:24.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ooaR5Kahr7z95z2zNk+Wj80Vu/o=', 'Content-Length': '0'} 2022-09-12T20:14:24.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edf9aae98ca4aa3b-00631f9320-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.447 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:24.448 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:24.448 DEBUG:boto:Method: HEAD 2022-09-12T20:14:24.449 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:14:24.450 DEBUG:boto:Data: 2022-09-12T20:14:24.450 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.451 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.451 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.452 DEBUG:boto:Params: {} 2022-09-12T20:14:24.452 DEBUG:boto:Token: None 2022-09-12T20:14:24.453 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj57 2022-09-12T20:14:24.453 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ooaR5Kahr7z95z2zNk+Wj80Vu/o= 2022-09-12T20:14:24.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ooaR5Kahr7z95z2zNk+Wj80Vu/o=', 'Content-Length': '0'} 2022-09-12T20:14:24.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070ee38cc0bee8b0c-00631f9320-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:14:24.457 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:24.458 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:24.458 DEBUG:boto:Method: GET 2022-09-12T20:14:24.459 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:14:24.459 DEBUG:boto:Data: 2022-09-12T20:14:24.460 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.461 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.461 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.462 DEBUG:boto:Params: {} 2022-09-12T20:14:24.462 DEBUG:boto:Token: None 2022-09-12T20:14:24.463 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj57 2022-09-12T20:14:24.464 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8WiW+H+2x8KuRy7SOdBiykVGjP8= 2022-09-12T20:14:24.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8WiW+H+2x8KuRy7SOdBiykVGjP8=', 'Content-Length': '0'} 2022-09-12T20:14:24.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f88b37f415e3d99b-00631f9320-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.467 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:14:24.468 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:14:24.468 DEBUG:boto:Method: GET 2022-09-12T20:14:24.469 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:14:24.469 DEBUG:boto:Data: 2022-09-12T20:14:24.470 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.470 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.471 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.471 DEBUG:boto:Params: {} 2022-09-12T20:14:24.472 DEBUG:boto:Token: None 2022-09-12T20:14:24.473 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj57 2022-09-12T20:14:24.473 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8WiW+H+2x8KuRy7SOdBiykVGjP8= 2022-09-12T20:14:24.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8WiW+H+2x8KuRy7SOdBiykVGjP8=', 'Content-Length': '0'} 2022-09-12T20:14:24.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007075e2e1552eb548-00631f9320-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.477 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:24.477 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:24.478 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:14:24.478 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:14:24.479 DEBUG:boto:Method: GET 2022-09-12T20:14:24.479 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:14:24.480 DEBUG:boto:Data: 2022-09-12T20:14:24.480 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.481 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.481 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.482 DEBUG:boto:Params: {} 2022-09-12T20:14:24.483 DEBUG:boto:Token: None 2022-09-12T20:14:24.483 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/ 2022-09-12T20:14:24.484 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o= 2022-09-12T20:14:24.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o=', 'Content-Length': '0'} 2022-09-12T20:14:24.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d75430305314e53-00631f9320-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.530 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:24.531 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:24.531 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:24.532 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:14:24.533 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:14:24.533 DEBUG:boto:Method: GET 2022-09-12T20:14:24.534 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:14:24.534 DEBUG:boto:Data: 2022-09-12T20:14:24.535 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.535 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.536 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.537 DEBUG:boto:Params: {} 2022-09-12T20:14:24.537 DEBUG:boto:Token: None 2022-09-12T20:14:24.538 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/ 2022-09-12T20:14:24.539 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o= 2022-09-12T20:14:24.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o=', 'Content-Length': '0'} 2022-09-12T20:14:24.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076f68a848c279a11-00631f9320-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.543 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:24.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:14:24.544 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:24.546 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:24.547 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:24.548 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:24.550 DEBUG:boto:Method: GET 2022-09-12T20:14:24.551 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:24.553 DEBUG:boto:Data: 2022-09-12T20:14:24.554 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.555 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.556 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.557 DEBUG:boto:Params: {} 2022-09-12T20:14:24.557 DEBUG:boto:Token: None 2022-09-12T20:14:24.558 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:24.559 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yarlLi09i+8ejy6Ui31qWKIDM6o= 2022-09-12T20:14:24.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yarlLi09i+8ejy6Ui31qWKIDM6o=', 'Content-Length': '0'} 2022-09-12T20:14:24.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062e03922520e977d-00631f9320-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.563 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:24.563 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:24.564 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:24.564 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:24.565 DEBUG:boto:Method: GET 2022-09-12T20:14:24.565 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:24.566 DEBUG:boto:Data: 2022-09-12T20:14:24.566 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.567 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.567 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.568 DEBUG:boto:Params: {} 2022-09-12T20:14:24.569 DEBUG:boto:Token: None 2022-09-12T20:14:24.569 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:24.570 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yarlLi09i+8ejy6Ui31qWKIDM6o= 2022-09-12T20:14:24.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yarlLi09i+8ejy6Ui31qWKIDM6o=', 'Content-Length': '0'} 2022-09-12T20:14:24.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cded7ee30e983a55-00631f9320-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.614 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:24.615 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:24.615 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:24.616 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:24.616 DEBUG:boto:Method: HEAD 2022-09-12T20:14:24.617 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:24.617 DEBUG:boto:Data: 2022-09-12T20:14:24.618 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.618 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.619 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.619 DEBUG:boto:Params: {} 2022-09-12T20:14:24.620 DEBUG:boto:Token: None 2022-09-12T20:14:24.620 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:24.621 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XELxKvnnGidsObB06gmfLGvPNe8= 2022-09-12T20:14:24.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XELxKvnnGidsObB06gmfLGvPNe8=', 'Content-Length': '0'} 2022-09-12T20:14:24.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e413b5e503ca7f47-00631f9320-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.625 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:24.625 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:24.626 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:24.626 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:24.627 DEBUG:boto:Method: HEAD 2022-09-12T20:14:24.627 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:24.628 DEBUG:boto:Data: 2022-09-12T20:14:24.628 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.629 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.629 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.630 DEBUG:boto:Params: {} 2022-09-12T20:14:24.630 DEBUG:boto:Token: None 2022-09-12T20:14:24.631 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:24.631 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XELxKvnnGidsObB06gmfLGvPNe8= 2022-09-12T20:14:24.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XELxKvnnGidsObB06gmfLGvPNe8=', 'Content-Length': '0'} 2022-09-12T20:14:24.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075c02cd3607ef0a9-00631f9320-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:14:24.635 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:24.636 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:24.636 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:24.637 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:24.637 DEBUG:boto:Method: GET 2022-09-12T20:14:24.638 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:24.638 DEBUG:boto:Data: 2022-09-12T20:14:24.639 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.640 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.640 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.641 DEBUG:boto:Params: {} 2022-09-12T20:14:24.641 DEBUG:boto:Token: None 2022-09-12T20:14:24.642 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:24.642 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yarlLi09i+8ejy6Ui31qWKIDM6o= 2022-09-12T20:14:24.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yarlLi09i+8ejy6Ui31qWKIDM6o=', 'Content-Length': '0'} 2022-09-12T20:14:24.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f6062cb046c2a2a-00631f9320-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.646 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:24.646 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:24.647 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:24.648 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:14:24.648 DEBUG:boto:Method: GET 2022-09-12T20:14:24.649 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:24.649 DEBUG:boto:Data: 2022-09-12T20:14:24.650 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.650 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.651 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.651 DEBUG:boto:Params: {} 2022-09-12T20:14:24.652 DEBUG:boto:Token: None 2022-09-12T20:14:24.652 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:14:24.653 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yarlLi09i+8ejy6Ui31qWKIDM6o= 2022-09-12T20:14:24.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yarlLi09i+8ejy6Ui31qWKIDM6o=', 'Content-Length': '0'} 2022-09-12T20:14:24.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061db175c664a8135-00631f9320-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.657 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:24.657 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:24.658 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:14:24.658 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:14:24.659 DEBUG:boto:Method: GET 2022-09-12T20:14:24.659 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:14:24.660 DEBUG:boto:Data: 2022-09-12T20:14:24.660 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.661 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.661 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.662 DEBUG:boto:Params: {} 2022-09-12T20:14:24.663 DEBUG:boto:Token: None 2022-09-12T20:14:24.663 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/ 2022-09-12T20:14:24.664 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o= 2022-09-12T20:14:24.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o=', 'Content-Length': '0'} 2022-09-12T20:14:24.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000812243fadcb325e8-00631f9320-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.668 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:24.669 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:24.669 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:24.670 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:14:24.670 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:14:24.671 DEBUG:boto:Method: GET 2022-09-12T20:14:24.671 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:14:24.672 DEBUG:boto:Data: 2022-09-12T20:14:24.672 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.673 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.673 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.674 DEBUG:boto:Params: {} 2022-09-12T20:14:24.674 DEBUG:boto:Token: None 2022-09-12T20:14:24.675 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/ 2022-09-12T20:14:24.675 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o= 2022-09-12T20:14:24.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o=', 'Content-Length': '0'} 2022-09-12T20:14:24.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036dfddd26b3991ee-00631f9320-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.679 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:24.680 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:24.680 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:24.681 DEBUG:boto:Method: HEAD 2022-09-12T20:14:24.681 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:14:24.682 DEBUG:boto:Data: 2022-09-12T20:14:24.682 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.683 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.683 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.684 DEBUG:boto:Params: {} 2022-09-12T20:14:24.684 DEBUG:boto:Token: None 2022-09-12T20:14:24.685 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj58 2022-09-12T20:14:24.685 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MAmaIOiOgIC2K+TMvg/WkCcoeDI= 2022-09-12T20:14:24.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MAmaIOiOgIC2K+TMvg/WkCcoeDI=', 'Content-Length': '0'} 2022-09-12T20:14:24.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecc6213d544bd948-00631f9320-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.689 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:24.689 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:24.690 DEBUG:boto:Method: HEAD 2022-09-12T20:14:24.690 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:14:24.691 DEBUG:boto:Data: 2022-09-12T20:14:24.691 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.692 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.692 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.693 DEBUG:boto:Params: {} 2022-09-12T20:14:24.693 DEBUG:boto:Token: None 2022-09-12T20:14:24.694 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj58 2022-09-12T20:14:24.694 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MAmaIOiOgIC2K+TMvg/WkCcoeDI= 2022-09-12T20:14:24.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MAmaIOiOgIC2K+TMvg/WkCcoeDI=', 'Content-Length': '0'} 2022-09-12T20:14:24.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f3c8f23e4c11daf-00631f9320-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:14:24.698 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:24.699 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:24.699 DEBUG:boto:Method: GET 2022-09-12T20:14:24.700 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:14:24.700 DEBUG:boto:Data: 2022-09-12T20:14:24.701 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.701 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.702 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.702 DEBUG:boto:Params: {} 2022-09-12T20:14:24.703 DEBUG:boto:Token: None 2022-09-12T20:14:24.703 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj58 2022-09-12T20:14:24.704 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ItIr0FZOMhjLn2Gl87m9Jhq6uKw= 2022-09-12T20:14:24.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ItIr0FZOMhjLn2Gl87m9Jhq6uKw=', 'Content-Length': '0'} 2022-09-12T20:14:24.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b31475023bccdf1a-00631f9320-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.707 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:14:24.708 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:14:24.709 DEBUG:boto:Method: GET 2022-09-12T20:14:24.709 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:14:24.710 DEBUG:boto:Data: 2022-09-12T20:14:24.710 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.711 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.711 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.712 DEBUG:boto:Params: {} 2022-09-12T20:14:24.712 DEBUG:boto:Token: None 2022-09-12T20:14:24.713 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj58 2022-09-12T20:14:24.714 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ItIr0FZOMhjLn2Gl87m9Jhq6uKw= 2022-09-12T20:14:24.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ItIr0FZOMhjLn2Gl87m9Jhq6uKw=', 'Content-Length': '0'} 2022-09-12T20:14:24.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ec776e81782d0ef-00631f9320-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.758 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:24.759 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:24.759 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:14:24.760 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:14:24.760 DEBUG:boto:Method: GET 2022-09-12T20:14:24.761 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:14:24.761 DEBUG:boto:Data: 2022-09-12T20:14:24.762 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.763 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.763 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.764 DEBUG:boto:Params: {} 2022-09-12T20:14:24.764 DEBUG:boto:Token: None 2022-09-12T20:14:24.765 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/ 2022-09-12T20:14:24.765 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o= 2022-09-12T20:14:24.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o=', 'Content-Length': '0'} 2022-09-12T20:14:24.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3b07f3882221975-00631f9320-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.769 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:24.770 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:24.771 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:24.771 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:14:24.772 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:14:24.772 DEBUG:boto:Method: GET 2022-09-12T20:14:24.773 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:14:24.773 DEBUG:boto:Data: 2022-09-12T20:14:24.774 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.774 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.775 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.775 DEBUG:boto:Params: {} 2022-09-12T20:14:24.776 DEBUG:boto:Token: None 2022-09-12T20:14:24.777 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/ 2022-09-12T20:14:24.777 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o= 2022-09-12T20:14:24.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o=', 'Content-Length': '0'} 2022-09-12T20:14:24.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000815e1c0b0d900c6b-00631f9320-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.781 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:24.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-12T20:14:24.783 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:24.783 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:24.784 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:24.784 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:24.785 DEBUG:boto:Method: GET 2022-09-12T20:14:24.785 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:24.786 DEBUG:boto:Data: 2022-09-12T20:14:24.786 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.787 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.787 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.788 DEBUG:boto:Params: {} 2022-09-12T20:14:24.788 DEBUG:boto:Token: None 2022-09-12T20:14:24.789 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:24.789 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:veM/wWcJsNxyXQZDBbuJBwEObr8= 2022-09-12T20:14:24.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:veM/wWcJsNxyXQZDBbuJBwEObr8=', 'Content-Length': '0'} 2022-09-12T20:14:24.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ccfa684963d973e-00631f9320-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.793 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:24.794 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:24.794 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:24.795 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:24.795 DEBUG:boto:Method: GET 2022-09-12T20:14:24.796 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:24.796 DEBUG:boto:Data: 2022-09-12T20:14:24.797 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.797 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.798 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.798 DEBUG:boto:Params: {} 2022-09-12T20:14:24.799 DEBUG:boto:Token: None 2022-09-12T20:14:24.799 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:24.800 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:veM/wWcJsNxyXQZDBbuJBwEObr8= 2022-09-12T20:14:24.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:veM/wWcJsNxyXQZDBbuJBwEObr8=', 'Content-Length': '0'} 2022-09-12T20:14:24.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e34009717567565-00631f9320-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.804 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:24.804 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:24.805 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:24.805 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:24.806 DEBUG:boto:Method: HEAD 2022-09-12T20:14:24.806 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:24.807 DEBUG:boto:Data: 2022-09-12T20:14:24.807 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.808 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.808 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.809 DEBUG:boto:Params: {} 2022-09-12T20:14:24.810 DEBUG:boto:Token: None 2022-09-12T20:14:24.810 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:24.811 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zejrhcvIYxAQ0mZnqfyNfdjq5pA= 2022-09-12T20:14:24.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zejrhcvIYxAQ0mZnqfyNfdjq5pA=', 'Content-Length': '0'} 2022-09-12T20:14:24.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002def58fa41b858ad-00631f9320-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.814 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:24.815 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:24.815 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:24.816 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:24.816 DEBUG:boto:Method: HEAD 2022-09-12T20:14:24.817 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:24.817 DEBUG:boto:Data: 2022-09-12T20:14:24.818 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.818 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.819 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.820 DEBUG:boto:Params: {} 2022-09-12T20:14:24.820 DEBUG:boto:Token: None 2022-09-12T20:14:24.821 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:24.821 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zejrhcvIYxAQ0mZnqfyNfdjq5pA= 2022-09-12T20:14:24.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zejrhcvIYxAQ0mZnqfyNfdjq5pA=', 'Content-Length': '0'} 2022-09-12T20:14:24.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c959f5d33cf03a0-00631f9320-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-12T20:14:24.825 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:24.826 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:24.826 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:24.827 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:24.827 DEBUG:boto:Method: GET 2022-09-12T20:14:24.828 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:24.829 DEBUG:boto:Data: 2022-09-12T20:14:24.829 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.830 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.830 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.831 DEBUG:boto:Params: {} 2022-09-12T20:14:24.832 DEBUG:boto:Token: None 2022-09-12T20:14:24.832 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:24.833 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:veM/wWcJsNxyXQZDBbuJBwEObr8= 2022-09-12T20:14:24.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:veM/wWcJsNxyXQZDBbuJBwEObr8=', 'Content-Length': '0'} 2022-09-12T20:14:24.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081f5f8b16f5ec702-00631f9320-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.837 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:24.837 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:24.838 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:24.839 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:14:24.839 DEBUG:boto:Method: GET 2022-09-12T20:14:24.840 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:24.840 DEBUG:boto:Data: 2022-09-12T20:14:24.841 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.842 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.842 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.843 DEBUG:boto:Params: {} 2022-09-12T20:14:24.844 DEBUG:boto:Token: None 2022-09-12T20:14:24.844 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:14:24.845 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:veM/wWcJsNxyXQZDBbuJBwEObr8= 2022-09-12T20:14:24.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:veM/wWcJsNxyXQZDBbuJBwEObr8=', 'Content-Length': '0'} 2022-09-12T20:14:24.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f9b4d8e7b95aef1-00631f9320-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.849 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:24.849 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:24.850 DEBUG:boto:path=/jdpiak-75/?prefix=obj62 2022-09-12T20:14:24.850 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj62 2022-09-12T20:14:24.851 DEBUG:boto:Method: GET 2022-09-12T20:14:24.852 DEBUG:boto:Path: /jdpiak-75/?prefix=obj62 2022-09-12T20:14:24.852 DEBUG:boto:Data: 2022-09-12T20:14:24.853 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.854 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.854 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.855 DEBUG:boto:Params: {} 2022-09-12T20:14:24.856 DEBUG:boto:Token: None 2022-09-12T20:14:24.856 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/ 2022-09-12T20:14:24.857 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o= 2022-09-12T20:14:24.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o=', 'Content-Length': '0'} 2022-09-12T20:14:24.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5760b003bc3eb80-00631f9320-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.862 DEBUG:boto:b'jdpiak-75obj621000falseobj622022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:24.862 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:24.863 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:24.864 DEBUG:boto:path=/jdpiak-75/?prefix=obj62 2022-09-12T20:14:24.864 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj62 2022-09-12T20:14:24.865 DEBUG:boto:Method: GET 2022-09-12T20:14:24.865 DEBUG:boto:Path: /jdpiak-75/?prefix=obj62 2022-09-12T20:14:24.866 DEBUG:boto:Data: 2022-09-12T20:14:24.867 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.867 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.868 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.869 DEBUG:boto:Params: {} 2022-09-12T20:14:24.869 DEBUG:boto:Token: None 2022-09-12T20:14:24.870 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/ 2022-09-12T20:14:24.871 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o= 2022-09-12T20:14:24.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o=', 'Content-Length': '0'} 2022-09-12T20:14:24.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062944629dec531da-00631f9320-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.875 DEBUG:boto:b'jdpiak-75obj621000falseobj622022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:24.876 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:24.876 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:24.877 DEBUG:boto:Method: HEAD 2022-09-12T20:14:24.878 DEBUG:boto:Path: /jdpiak-75/obj62 2022-09-12T20:14:24.878 DEBUG:boto:Data: 2022-09-12T20:14:24.879 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.879 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.880 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.880 DEBUG:boto:Params: {} 2022-09-12T20:14:24.881 DEBUG:boto:Token: None 2022-09-12T20:14:24.881 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj62 2022-09-12T20:14:24.882 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4MFDehZ2XQ2ICo91jcIwLLg8/3A= 2022-09-12T20:14:24.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4MFDehZ2XQ2ICo91jcIwLLg8/3A=', 'Content-Length': '0'} 2022-09-12T20:14:24.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae36c907e4b7f433-00631f9320-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.886 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:24.886 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:24.887 DEBUG:boto:Method: HEAD 2022-09-12T20:14:24.888 DEBUG:boto:Path: /jdpiak-75/obj62 2022-09-12T20:14:24.888 DEBUG:boto:Data: 2022-09-12T20:14:24.889 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.890 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.890 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.891 DEBUG:boto:Params: {} 2022-09-12T20:14:24.891 DEBUG:boto:Token: None 2022-09-12T20:14:24.892 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj62 2022-09-12T20:14:24.893 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4MFDehZ2XQ2ICo91jcIwLLg8/3A= 2022-09-12T20:14:24.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4MFDehZ2XQ2ICo91jcIwLLg8/3A=', 'Content-Length': '0'} 2022-09-12T20:14:24.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf3156a7a39e27ee-00631f9320-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-12T20:14:24.897 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:24.898 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:24.898 DEBUG:boto:Method: GET 2022-09-12T20:14:24.899 DEBUG:boto:Path: /jdpiak-75/obj62 2022-09-12T20:14:24.900 DEBUG:boto:Data: 2022-09-12T20:14:24.900 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.901 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.901 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.902 DEBUG:boto:Params: {} 2022-09-12T20:14:24.903 DEBUG:boto:Token: None 2022-09-12T20:14:24.903 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj62 2022-09-12T20:14:24.904 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ymn8bNvg2QMvkTM7VHQ0J/b+CCA= 2022-09-12T20:14:24.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ymn8bNvg2QMvkTM7VHQ0J/b+CCA=', 'Content-Length': '0'} 2022-09-12T20:14:24.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9d6632fccbcb5a0-00631f9320-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.908 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:14:24.908 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:14:24.909 DEBUG:boto:Method: GET 2022-09-12T20:14:24.910 DEBUG:boto:Path: /jdpiak-75/obj62 2022-09-12T20:14:24.910 DEBUG:boto:Data: 2022-09-12T20:14:24.911 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.911 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.912 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.913 DEBUG:boto:Params: {} 2022-09-12T20:14:24.913 DEBUG:boto:Token: None 2022-09-12T20:14:24.914 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj62 2022-09-12T20:14:24.915 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ymn8bNvg2QMvkTM7VHQ0J/b+CCA= 2022-09-12T20:14:24.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ymn8bNvg2QMvkTM7VHQ0J/b+CCA=', 'Content-Length': '0'} 2022-09-12T20:14:24.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043cf25a99a27eba5-00631f9320-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.958 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:24.959 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:24.959 DEBUG:boto:path=/jdpiak-75/?prefix=obj62 2022-09-12T20:14:24.960 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj62 2022-09-12T20:14:24.961 DEBUG:boto:Method: GET 2022-09-12T20:14:24.961 DEBUG:boto:Path: /jdpiak-75/?prefix=obj62 2022-09-12T20:14:24.962 DEBUG:boto:Data: 2022-09-12T20:14:24.962 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.963 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.964 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.964 DEBUG:boto:Params: {} 2022-09-12T20:14:24.965 DEBUG:boto:Token: None 2022-09-12T20:14:24.965 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/ 2022-09-12T20:14:24.966 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o= 2022-09-12T20:14:24.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o=', 'Content-Length': '0'} 2022-09-12T20:14:24.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c476dedf9e6c251d-00631f9320-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.970 DEBUG:boto:b'jdpiak-75obj621000falseobj622022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:24.971 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:24.972 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:24.972 DEBUG:boto:path=/jdpiak-75/?prefix=obj62 2022-09-12T20:14:24.973 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj62 2022-09-12T20:14:24.974 DEBUG:boto:Method: GET 2022-09-12T20:14:24.974 DEBUG:boto:Path: /jdpiak-75/?prefix=obj62 2022-09-12T20:14:24.975 DEBUG:boto:Data: 2022-09-12T20:14:24.976 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.976 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:24.977 DEBUG:boto:Port: 8000 2022-09-12T20:14:24.978 DEBUG:boto:Params: {} 2022-09-12T20:14:24.978 DEBUG:boto:Token: None 2022-09-12T20:14:24.979 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/ 2022-09-12T20:14:24.979 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o= 2022-09-12T20:14:24.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IKaFrVMt5Tl66nl7Iup1q82CX0o=', 'Content-Length': '0'} 2022-09-12T20:14:24.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fcc2fd7cd8210c3-00631f9320-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.983 DEBUG:boto:b'jdpiak-75obj621000falseobj622022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:24.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:14:24.985 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:24.985 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:24.986 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:24.987 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:24.987 DEBUG:boto:Method: GET 2022-09-12T20:14:24.988 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:24.989 DEBUG:boto:Data: 2022-09-12T20:14:24.989 DEBUG:boto:Headers: {} 2022-09-12T20:14:24.990 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:24.990 DEBUG:boto:Port: 8001 2022-09-12T20:14:24.991 DEBUG:boto:Params: {} 2022-09-12T20:14:24.992 DEBUG:boto:Token: None 2022-09-12T20:14:24.992 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:24 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:24.993 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jfGJtI/IvljtZ1WrVo+l8s2/c1Q= 2022-09-12T20:14:24.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jfGJtI/IvljtZ1WrVo+l8s2/c1Q=', 'Content-Length': '0'} 2022-09-12T20:14:24.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c61242f699922fd-00631f9320-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:24.997 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:24.997 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:24.998 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:24.998 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:24.999 DEBUG:boto:Method: GET 2022-09-12T20:14:25.000 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:25.000 DEBUG:boto:Data: 2022-09-12T20:14:25.001 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.001 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.002 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.002 DEBUG:boto:Params: {} 2022-09-12T20:14:25.003 DEBUG:boto:Token: None 2022-09-12T20:14:25.004 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:25.004 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9oVAgdL8imMFRciYLYFeZ/eHDGo= 2022-09-12T20:14:25.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9oVAgdL8imMFRciYLYFeZ/eHDGo=', 'Content-Length': '0'} 2022-09-12T20:14:25.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046857ca6b1608a8f-00631f9321-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.008 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:25.009 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:25.009 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:25.010 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:25.011 DEBUG:boto:Method: HEAD 2022-09-12T20:14:25.011 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:25.012 DEBUG:boto:Data: 2022-09-12T20:14:25.012 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.013 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.014 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.014 DEBUG:boto:Params: {} 2022-09-12T20:14:25.015 DEBUG:boto:Token: None 2022-09-12T20:14:25.016 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:25.016 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m4JAOhJBSPYaqtO1YdiL9x9BaWs= 2022-09-12T20:14:25.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m4JAOhJBSPYaqtO1YdiL9x9BaWs=', 'Content-Length': '0'} 2022-09-12T20:14:25.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a64945c70fe5b847-00631f9321-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.020 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:25.020 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:25.021 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:25.021 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:25.022 DEBUG:boto:Method: HEAD 2022-09-12T20:14:25.022 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:25.023 DEBUG:boto:Data: 2022-09-12T20:14:25.023 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.024 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.024 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.025 DEBUG:boto:Params: {} 2022-09-12T20:14:25.025 DEBUG:boto:Token: None 2022-09-12T20:14:25.026 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:25.026 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m4JAOhJBSPYaqtO1YdiL9x9BaWs= 2022-09-12T20:14:25.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m4JAOhJBSPYaqtO1YdiL9x9BaWs=', 'Content-Length': '0'} 2022-09-12T20:14:25.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df11125bb757d7e4-00631f9321-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:14:25.030 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:25.031 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:25.031 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:25.032 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:25.032 DEBUG:boto:Method: GET 2022-09-12T20:14:25.033 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:25.033 DEBUG:boto:Data: 2022-09-12T20:14:25.034 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.034 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.035 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.035 DEBUG:boto:Params: {} 2022-09-12T20:14:25.036 DEBUG:boto:Token: None 2022-09-12T20:14:25.037 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:25.037 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9oVAgdL8imMFRciYLYFeZ/eHDGo= 2022-09-12T20:14:25.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9oVAgdL8imMFRciYLYFeZ/eHDGo=', 'Content-Length': '0'} 2022-09-12T20:14:25.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a827c98877a3d4a1-00631f9321-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.041 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:25.041 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:25.042 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:25.042 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:14:25.043 DEBUG:boto:Method: GET 2022-09-12T20:14:25.043 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:25.044 DEBUG:boto:Data: 2022-09-12T20:14:25.044 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.045 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.045 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.046 DEBUG:boto:Params: {} 2022-09-12T20:14:25.046 DEBUG:boto:Token: None 2022-09-12T20:14:25.047 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:14:25.048 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9oVAgdL8imMFRciYLYFeZ/eHDGo= 2022-09-12T20:14:25.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9oVAgdL8imMFRciYLYFeZ/eHDGo=', 'Content-Length': '0'} 2022-09-12T20:14:25.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad4ba65a4e01249b-00631f9321-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.051 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:25.051 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:25.052 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:14:25.052 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:14:25.053 DEBUG:boto:Method: GET 2022-09-12T20:14:25.053 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:14:25.054 DEBUG:boto:Data: 2022-09-12T20:14:25.055 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.055 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.056 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.056 DEBUG:boto:Params: {} 2022-09-12T20:14:25.057 DEBUG:boto:Token: None 2022-09-12T20:14:25.057 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/ 2022-09-12T20:14:25.058 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E= 2022-09-12T20:14:25.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E=', 'Content-Length': '0'} 2022-09-12T20:14:25.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077ae5eded2f54c77-00631f9321-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.062 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:25.063 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:25.063 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:25.064 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:14:25.064 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:14:25.065 DEBUG:boto:Method: GET 2022-09-12T20:14:25.065 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:14:25.066 DEBUG:boto:Data: 2022-09-12T20:14:25.066 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.067 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.067 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.068 DEBUG:boto:Params: {} 2022-09-12T20:14:25.068 DEBUG:boto:Token: None 2022-09-12T20:14:25.069 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/ 2022-09-12T20:14:25.070 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E= 2022-09-12T20:14:25.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E=', 'Content-Length': '0'} 2022-09-12T20:14:25.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c7c690e05ba3f91-00631f9321-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.114 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:25.115 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:25.115 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:25.116 DEBUG:boto:Method: HEAD 2022-09-12T20:14:25.117 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:14:25.117 DEBUG:boto:Data: 2022-09-12T20:14:25.118 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.118 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.119 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.119 DEBUG:boto:Params: {} 2022-09-12T20:14:25.120 DEBUG:boto:Token: None 2022-09-12T20:14:25.120 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj68 2022-09-12T20:14:25.121 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9tduEpdwuWDBpSiJ5UHSO05N5wg= 2022-09-12T20:14:25.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9tduEpdwuWDBpSiJ5UHSO05N5wg=', 'Content-Length': '0'} 2022-09-12T20:14:25.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009502a602edb9ea41-00631f9321-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.125 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:25.125 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:25.126 DEBUG:boto:Method: HEAD 2022-09-12T20:14:25.126 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:14:25.127 DEBUG:boto:Data: 2022-09-12T20:14:25.127 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.128 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.128 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.129 DEBUG:boto:Params: {} 2022-09-12T20:14:25.129 DEBUG:boto:Token: None 2022-09-12T20:14:25.130 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj68 2022-09-12T20:14:25.131 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9tduEpdwuWDBpSiJ5UHSO05N5wg= 2022-09-12T20:14:25.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9tduEpdwuWDBpSiJ5UHSO05N5wg=', 'Content-Length': '0'} 2022-09-12T20:14:25.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f13556ee04fe734-00631f9321-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:14:25.135 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:25.135 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:25.136 DEBUG:boto:Method: GET 2022-09-12T20:14:25.136 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:14:25.137 DEBUG:boto:Data: 2022-09-12T20:14:25.137 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.138 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.138 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.139 DEBUG:boto:Params: {} 2022-09-12T20:14:25.139 DEBUG:boto:Token: None 2022-09-12T20:14:25.140 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj68 2022-09-12T20:14:25.141 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GRuDvXBIioXWoU+PFDtLHjIH2Y8= 2022-09-12T20:14:25.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GRuDvXBIioXWoU+PFDtLHjIH2Y8=', 'Content-Length': '0'} 2022-09-12T20:14:25.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1fe7b57ccb0c544-00631f9321-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.144 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:14:25.145 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:14:25.145 DEBUG:boto:Method: GET 2022-09-12T20:14:25.146 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:14:25.146 DEBUG:boto:Data: 2022-09-12T20:14:25.147 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.147 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.148 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.149 DEBUG:boto:Params: {} 2022-09-12T20:14:25.149 DEBUG:boto:Token: None 2022-09-12T20:14:25.150 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj68 2022-09-12T20:14:25.150 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GRuDvXBIioXWoU+PFDtLHjIH2Y8= 2022-09-12T20:14:25.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GRuDvXBIioXWoU+PFDtLHjIH2Y8=', 'Content-Length': '0'} 2022-09-12T20:14:25.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cde72f92d255512-00631f9321-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.154 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:25.155 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:25.155 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:14:25.156 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:14:25.156 DEBUG:boto:Method: GET 2022-09-12T20:14:25.157 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:14:25.157 DEBUG:boto:Data: 2022-09-12T20:14:25.158 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.158 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.159 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.159 DEBUG:boto:Params: {} 2022-09-12T20:14:25.160 DEBUG:boto:Token: None 2022-09-12T20:14:25.161 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/ 2022-09-12T20:14:25.161 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E= 2022-09-12T20:14:25.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E=', 'Content-Length': '0'} 2022-09-12T20:14:25.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056c688dcad6e067d-00631f9321-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.165 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:25.166 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:25.166 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:25.167 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:14:25.167 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:14:25.168 DEBUG:boto:Method: GET 2022-09-12T20:14:25.169 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:14:25.169 DEBUG:boto:Data: 2022-09-12T20:14:25.170 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.170 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.171 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.171 DEBUG:boto:Params: {} 2022-09-12T20:14:25.172 DEBUG:boto:Token: None 2022-09-12T20:14:25.172 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/ 2022-09-12T20:14:25.173 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E= 2022-09-12T20:14:25.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E=', 'Content-Length': '0'} 2022-09-12T20:14:25.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000148898455567eb4d-00631f9321-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.177 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:25.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:14:25.178 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:25.179 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:25.179 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:25.180 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:25.180 DEBUG:boto:Method: GET 2022-09-12T20:14:25.181 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:25.181 DEBUG:boto:Data: 2022-09-12T20:14:25.182 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.183 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.183 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.184 DEBUG:boto:Params: {} 2022-09-12T20:14:25.184 DEBUG:boto:Token: None 2022-09-12T20:14:25.185 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:25.185 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1///bsyNO5w9PY3iOYIqMY+O3g8= 2022-09-12T20:14:25.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1///bsyNO5w9PY3iOYIqMY+O3g8=', 'Content-Length': '0'} 2022-09-12T20:14:25.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbcf3d93766da8ae-00631f9321-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.189 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:25.190 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:25.190 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:25.191 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:25.191 DEBUG:boto:Method: GET 2022-09-12T20:14:25.192 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:25.192 DEBUG:boto:Data: 2022-09-12T20:14:25.193 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.193 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.194 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.194 DEBUG:boto:Params: {} 2022-09-12T20:14:25.195 DEBUG:boto:Token: None 2022-09-12T20:14:25.196 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:25.196 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1///bsyNO5w9PY3iOYIqMY+O3g8= 2022-09-12T20:14:25.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1///bsyNO5w9PY3iOYIqMY+O3g8=', 'Content-Length': '0'} 2022-09-12T20:14:25.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f95108c088f2092-00631f9321-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.199 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:25.200 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:25.201 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:25.201 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:25.202 DEBUG:boto:Method: HEAD 2022-09-12T20:14:25.202 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:25.203 DEBUG:boto:Data: 2022-09-12T20:14:25.203 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.204 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.204 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.205 DEBUG:boto:Params: {} 2022-09-12T20:14:25.205 DEBUG:boto:Token: None 2022-09-12T20:14:25.206 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:25.206 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HXK/KqY+eVNFicAcgZJkP1LK40E= 2022-09-12T20:14:25.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HXK/KqY+eVNFicAcgZJkP1LK40E=', 'Content-Length': '0'} 2022-09-12T20:14:25.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fcb75722deaa165-00631f9321-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.210 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:25.210 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:25.211 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:25.211 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:25.212 DEBUG:boto:Method: HEAD 2022-09-12T20:14:25.212 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:25.213 DEBUG:boto:Data: 2022-09-12T20:14:25.214 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.214 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.215 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.215 DEBUG:boto:Params: {} 2022-09-12T20:14:25.216 DEBUG:boto:Token: None 2022-09-12T20:14:25.216 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:25.217 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HXK/KqY+eVNFicAcgZJkP1LK40E= 2022-09-12T20:14:25.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HXK/KqY+eVNFicAcgZJkP1LK40E=', 'Content-Length': '0'} 2022-09-12T20:14:25.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000622be926d901c591-00631f9321-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:14:25.221 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:25.222 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:25.222 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:25.223 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:25.223 DEBUG:boto:Method: GET 2022-09-12T20:14:25.224 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:25.225 DEBUG:boto:Data: 2022-09-12T20:14:25.225 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.226 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.226 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.227 DEBUG:boto:Params: {} 2022-09-12T20:14:25.227 DEBUG:boto:Token: None 2022-09-12T20:14:25.228 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:25.228 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1///bsyNO5w9PY3iOYIqMY+O3g8= 2022-09-12T20:14:25.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1///bsyNO5w9PY3iOYIqMY+O3g8=', 'Content-Length': '0'} 2022-09-12T20:14:25.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec4eafc092b213ab-00631f9321-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.232 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:25.233 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:25.233 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:25.234 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:14:25.234 DEBUG:boto:Method: GET 2022-09-12T20:14:25.235 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:25.235 DEBUG:boto:Data: 2022-09-12T20:14:25.236 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.236 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.237 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.237 DEBUG:boto:Params: {} 2022-09-12T20:14:25.238 DEBUG:boto:Token: None 2022-09-12T20:14:25.238 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:14:25.239 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1///bsyNO5w9PY3iOYIqMY+O3g8= 2022-09-12T20:14:25.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1///bsyNO5w9PY3iOYIqMY+O3g8=', 'Content-Length': '0'} 2022-09-12T20:14:25.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db47d3fb24406a9d-00631f9321-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.243 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:25.243 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:25.244 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:14:25.244 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:14:25.245 DEBUG:boto:Method: GET 2022-09-12T20:14:25.245 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:14:25.246 DEBUG:boto:Data: 2022-09-12T20:14:25.246 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.247 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.247 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.248 DEBUG:boto:Params: {} 2022-09-12T20:14:25.248 DEBUG:boto:Token: None 2022-09-12T20:14:25.249 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/ 2022-09-12T20:14:25.250 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E= 2022-09-12T20:14:25.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E=', 'Content-Length': '0'} 2022-09-12T20:14:25.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b992390af777e9b7-00631f9321-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.254 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:25.255 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:25.255 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:25.256 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:14:25.256 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:14:25.257 DEBUG:boto:Method: GET 2022-09-12T20:14:25.258 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:14:25.258 DEBUG:boto:Data: 2022-09-12T20:14:25.259 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.259 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.260 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.260 DEBUG:boto:Params: {} 2022-09-12T20:14:25.261 DEBUG:boto:Token: None 2022-09-12T20:14:25.261 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/ 2022-09-12T20:14:25.262 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E= 2022-09-12T20:14:25.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E=', 'Content-Length': '0'} 2022-09-12T20:14:25.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c31862bc5f9616f2-00631f9321-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.306 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:25.307 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:25.307 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:25.308 DEBUG:boto:Method: HEAD 2022-09-12T20:14:25.309 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:14:25.310 DEBUG:boto:Data: 2022-09-12T20:14:25.310 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.311 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.311 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.312 DEBUG:boto:Params: {} 2022-09-12T20:14:25.313 DEBUG:boto:Token: None 2022-09-12T20:14:25.313 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj69 2022-09-12T20:14:25.314 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pz8WXEtMU3XQ8d+JFNFTs6PNyAg= 2022-09-12T20:14:25.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pz8WXEtMU3XQ8d+JFNFTs6PNyAg=', 'Content-Length': '0'} 2022-09-12T20:14:25.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cba697f91e1193f-00631f9321-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.317 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:25.318 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:25.319 DEBUG:boto:Method: HEAD 2022-09-12T20:14:25.319 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:14:25.320 DEBUG:boto:Data: 2022-09-12T20:14:25.320 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.321 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.321 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.322 DEBUG:boto:Params: {} 2022-09-12T20:14:25.322 DEBUG:boto:Token: None 2022-09-12T20:14:25.323 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj69 2022-09-12T20:14:25.324 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pz8WXEtMU3XQ8d+JFNFTs6PNyAg= 2022-09-12T20:14:25.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pz8WXEtMU3XQ8d+JFNFTs6PNyAg=', 'Content-Length': '0'} 2022-09-12T20:14:25.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000233fd9e75cf6d939-00631f9321-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:14:25.328 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:25.329 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:25.330 DEBUG:boto:Method: GET 2022-09-12T20:14:25.331 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:14:25.332 DEBUG:boto:Data: 2022-09-12T20:14:25.332 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.334 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.334 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.335 DEBUG:boto:Params: {} 2022-09-12T20:14:25.336 DEBUG:boto:Token: None 2022-09-12T20:14:25.337 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj69 2022-09-12T20:14:25.338 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3LpYuodCagmOzSU5v4ImM/p/e4c= 2022-09-12T20:14:25.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3LpYuodCagmOzSU5v4ImM/p/e4c=', 'Content-Length': '0'} 2022-09-12T20:14:25.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000894e84645e740aef-00631f9321-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.343 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:14:25.344 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:14:25.345 DEBUG:boto:Method: GET 2022-09-12T20:14:25.346 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:14:25.347 DEBUG:boto:Data: 2022-09-12T20:14:25.348 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.349 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.349 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.350 DEBUG:boto:Params: {} 2022-09-12T20:14:25.351 DEBUG:boto:Token: None 2022-09-12T20:14:25.352 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj69 2022-09-12T20:14:25.352 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3LpYuodCagmOzSU5v4ImM/p/e4c= 2022-09-12T20:14:25.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3LpYuodCagmOzSU5v4ImM/p/e4c=', 'Content-Length': '0'} 2022-09-12T20:14:25.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038ceec69cecb5235-00631f9321-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.356 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:25.356 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:25.357 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:14:25.357 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:14:25.358 DEBUG:boto:Method: GET 2022-09-12T20:14:25.359 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:14:25.359 DEBUG:boto:Data: 2022-09-12T20:14:25.360 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.360 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.361 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.361 DEBUG:boto:Params: {} 2022-09-12T20:14:25.362 DEBUG:boto:Token: None 2022-09-12T20:14:25.363 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/ 2022-09-12T20:14:25.363 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E= 2022-09-12T20:14:25.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E=', 'Content-Length': '0'} 2022-09-12T20:14:25.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c78202c8b304c496-00631f9321-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.367 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:25.368 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:25.369 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:25.369 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:14:25.370 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:14:25.370 DEBUG:boto:Method: GET 2022-09-12T20:14:25.371 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:14:25.371 DEBUG:boto:Data: 2022-09-12T20:14:25.372 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.373 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.373 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.374 DEBUG:boto:Params: {} 2022-09-12T20:14:25.374 DEBUG:boto:Token: None 2022-09-12T20:14:25.375 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/ 2022-09-12T20:14:25.375 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E= 2022-09-12T20:14:25.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E=', 'Content-Length': '0'} 2022-09-12T20:14:25.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3f52702d7d74a2e-00631f9321-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.379 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:25.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:14:25.381 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:25.381 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:25.382 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:25.382 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:25.383 DEBUG:boto:Method: GET 2022-09-12T20:14:25.383 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:25.384 DEBUG:boto:Data: 2022-09-12T20:14:25.385 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.385 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.386 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.386 DEBUG:boto:Params: {} 2022-09-12T20:14:25.387 DEBUG:boto:Token: None 2022-09-12T20:14:25.387 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:25.388 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:luKcHVfvI9TUIvOWFfnK0z2yURk= 2022-09-12T20:14:25.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:luKcHVfvI9TUIvOWFfnK0z2yURk=', 'Content-Length': '0'} 2022-09-12T20:14:25.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b943929ba1910051-00631f9321-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.434 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:25.435 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:25.435 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:25.436 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:25.436 DEBUG:boto:Method: GET 2022-09-12T20:14:25.437 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:25.437 DEBUG:boto:Data: 2022-09-12T20:14:25.438 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.438 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.439 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.440 DEBUG:boto:Params: {} 2022-09-12T20:14:25.440 DEBUG:boto:Token: None 2022-09-12T20:14:25.441 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:25.441 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:luKcHVfvI9TUIvOWFfnK0z2yURk= 2022-09-12T20:14:25.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:luKcHVfvI9TUIvOWFfnK0z2yURk=', 'Content-Length': '0'} 2022-09-12T20:14:25.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca27e8f4dbc215ab-00631f9321-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.445 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:25.446 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:25.446 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:25.447 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:25.447 DEBUG:boto:Method: HEAD 2022-09-12T20:14:25.448 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:25.448 DEBUG:boto:Data: 2022-09-12T20:14:25.449 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.449 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.450 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.450 DEBUG:boto:Params: {} 2022-09-12T20:14:25.451 DEBUG:boto:Token: None 2022-09-12T20:14:25.451 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:25.452 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nRu2wbf+dQddJ2VPJAk0ByY2KTo= 2022-09-12T20:14:25.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nRu2wbf+dQddJ2VPJAk0ByY2KTo=', 'Content-Length': '0'} 2022-09-12T20:14:25.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002813f6564cd3e343-00631f9321-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.455 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:25.456 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:25.456 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:25.457 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:25.457 DEBUG:boto:Method: HEAD 2022-09-12T20:14:25.458 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:25.458 DEBUG:boto:Data: 2022-09-12T20:14:25.459 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.460 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.460 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.461 DEBUG:boto:Params: {} 2022-09-12T20:14:25.462 DEBUG:boto:Token: None 2022-09-12T20:14:25.462 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:25.463 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nRu2wbf+dQddJ2VPJAk0ByY2KTo= 2022-09-12T20:14:25.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nRu2wbf+dQddJ2VPJAk0ByY2KTo=', 'Content-Length': '0'} 2022-09-12T20:14:25.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052c8a152988d34db-00631f9321-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:14:25.467 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:25.467 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:25.468 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:25.468 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:25.469 DEBUG:boto:Method: GET 2022-09-12T20:14:25.469 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:25.470 DEBUG:boto:Data: 2022-09-12T20:14:25.471 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.471 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.472 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.472 DEBUG:boto:Params: {} 2022-09-12T20:14:25.473 DEBUG:boto:Token: None 2022-09-12T20:14:25.473 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:25.474 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:luKcHVfvI9TUIvOWFfnK0z2yURk= 2022-09-12T20:14:25.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:luKcHVfvI9TUIvOWFfnK0z2yURk=', 'Content-Length': '0'} 2022-09-12T20:14:25.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d40a80caab0b16b-00631f9321-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.478 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:25.478 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:25.479 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:25.479 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:14:25.480 DEBUG:boto:Method: GET 2022-09-12T20:14:25.480 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:25.481 DEBUG:boto:Data: 2022-09-12T20:14:25.481 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.482 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.483 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.483 DEBUG:boto:Params: {} 2022-09-12T20:14:25.484 DEBUG:boto:Token: None 2022-09-12T20:14:25.484 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:14:25.485 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:luKcHVfvI9TUIvOWFfnK0z2yURk= 2022-09-12T20:14:25.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:luKcHVfvI9TUIvOWFfnK0z2yURk=', 'Content-Length': '0'} 2022-09-12T20:14:25.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd3cc2068adcdccc-00631f9321-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.489 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:25.490 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:25.490 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:14:25.491 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:14:25.491 DEBUG:boto:Method: GET 2022-09-12T20:14:25.492 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:14:25.492 DEBUG:boto:Data: 2022-09-12T20:14:25.493 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.493 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.494 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.494 DEBUG:boto:Params: {} 2022-09-12T20:14:25.495 DEBUG:boto:Token: None 2022-09-12T20:14:25.496 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/ 2022-09-12T20:14:25.496 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E= 2022-09-12T20:14:25.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E=', 'Content-Length': '0'} 2022-09-12T20:14:25.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f328a77c17884df-00631f9321-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.500 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:25.501 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:25.502 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:25.502 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:14:25.503 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:14:25.503 DEBUG:boto:Method: GET 2022-09-12T20:14:25.504 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:14:25.504 DEBUG:boto:Data: 2022-09-12T20:14:25.505 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.505 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.506 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.507 DEBUG:boto:Params: {} 2022-09-12T20:14:25.507 DEBUG:boto:Token: None 2022-09-12T20:14:25.508 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/ 2022-09-12T20:14:25.509 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E= 2022-09-12T20:14:25.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E=', 'Content-Length': '0'} 2022-09-12T20:14:25.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4b48054f9dc544c-00631f9321-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.513 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:25.514 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:25.514 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:25.515 DEBUG:boto:Method: HEAD 2022-09-12T20:14:25.515 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:14:25.516 DEBUG:boto:Data: 2022-09-12T20:14:25.516 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.517 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.517 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.518 DEBUG:boto:Params: {} 2022-09-12T20:14:25.519 DEBUG:boto:Token: None 2022-09-12T20:14:25.519 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj76 2022-09-12T20:14:25.520 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y067p6G8K8G8HwvjS45+GOhfyP0= 2022-09-12T20:14:25.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y067p6G8K8G8HwvjS45+GOhfyP0=', 'Content-Length': '0'} 2022-09-12T20:14:25.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea3c2badffc26cb2-00631f9321-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.524 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:25.524 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:25.525 DEBUG:boto:Method: HEAD 2022-09-12T20:14:25.525 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:14:25.526 DEBUG:boto:Data: 2022-09-12T20:14:25.527 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.527 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.528 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.528 DEBUG:boto:Params: {} 2022-09-12T20:14:25.529 DEBUG:boto:Token: None 2022-09-12T20:14:25.529 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj76 2022-09-12T20:14:25.530 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y067p6G8K8G8HwvjS45+GOhfyP0= 2022-09-12T20:14:25.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y067p6G8K8G8HwvjS45+GOhfyP0=', 'Content-Length': '0'} 2022-09-12T20:14:25.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba981051405138a5-00631f9321-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:14:25.534 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:25.534 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:25.535 DEBUG:boto:Method: GET 2022-09-12T20:14:25.536 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:14:25.536 DEBUG:boto:Data: 2022-09-12T20:14:25.537 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.537 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.538 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.539 DEBUG:boto:Params: {} 2022-09-12T20:14:25.539 DEBUG:boto:Token: None 2022-09-12T20:14:25.540 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj76 2022-09-12T20:14:25.541 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XjojgqT2xfWGapJJs9sRWYcd5VY= 2022-09-12T20:14:25.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XjojgqT2xfWGapJJs9sRWYcd5VY=', 'Content-Length': '0'} 2022-09-12T20:14:25.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddef440616fc7184-00631f9321-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.586 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:14:25.587 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:14:25.587 DEBUG:boto:Method: GET 2022-09-12T20:14:25.588 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:14:25.588 DEBUG:boto:Data: 2022-09-12T20:14:25.589 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.590 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.590 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.591 DEBUG:boto:Params: {} 2022-09-12T20:14:25.591 DEBUG:boto:Token: None 2022-09-12T20:14:25.592 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj76 2022-09-12T20:14:25.592 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XjojgqT2xfWGapJJs9sRWYcd5VY= 2022-09-12T20:14:25.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XjojgqT2xfWGapJJs9sRWYcd5VY=', 'Content-Length': '0'} 2022-09-12T20:14:25.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b90e7276e2c2c668-00631f9321-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.596 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:25.597 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:25.597 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:14:25.598 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:14:25.598 DEBUG:boto:Method: GET 2022-09-12T20:14:25.599 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:14:25.600 DEBUG:boto:Data: 2022-09-12T20:14:25.600 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.601 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.601 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.602 DEBUG:boto:Params: {} 2022-09-12T20:14:25.602 DEBUG:boto:Token: None 2022-09-12T20:14:25.603 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/ 2022-09-12T20:14:25.604 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E= 2022-09-12T20:14:25.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E=', 'Content-Length': '0'} 2022-09-12T20:14:25.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ced1ffb8be92b976-00631f9321-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.608 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:25.608 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:25.609 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:25.609 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:14:25.610 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:14:25.611 DEBUG:boto:Method: GET 2022-09-12T20:14:25.611 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:14:25.612 DEBUG:boto:Data: 2022-09-12T20:14:25.613 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.613 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.614 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.614 DEBUG:boto:Params: {} 2022-09-12T20:14:25.615 DEBUG:boto:Token: None 2022-09-12T20:14:25.616 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/ 2022-09-12T20:14:25.616 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E= 2022-09-12T20:14:25.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E=', 'Content-Length': '0'} 2022-09-12T20:14:25.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050d3fde958373b80-00631f9321-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.620 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:25.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:14:25.621 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:25.622 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:25.623 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:25.623 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:25.624 DEBUG:boto:Method: GET 2022-09-12T20:14:25.624 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:25.625 DEBUG:boto:Data: 2022-09-12T20:14:25.625 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.626 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.626 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.627 DEBUG:boto:Params: {} 2022-09-12T20:14:25.627 DEBUG:boto:Token: None 2022-09-12T20:14:25.628 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:25.629 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1S3l+7yGo3uzgMRBwWfBC5/EPvI= 2022-09-12T20:14:25.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1S3l+7yGo3uzgMRBwWfBC5/EPvI=', 'Content-Length': '0'} 2022-09-12T20:14:25.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbb85b0119dfe745-00631f9321-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.632 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:25.633 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:25.633 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:25.634 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:25.634 DEBUG:boto:Method: GET 2022-09-12T20:14:25.635 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:25.635 DEBUG:boto:Data: 2022-09-12T20:14:25.636 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.636 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.637 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.638 DEBUG:boto:Params: {} 2022-09-12T20:14:25.638 DEBUG:boto:Token: None 2022-09-12T20:14:25.639 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:25.639 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1S3l+7yGo3uzgMRBwWfBC5/EPvI= 2022-09-12T20:14:25.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1S3l+7yGo3uzgMRBwWfBC5/EPvI=', 'Content-Length': '0'} 2022-09-12T20:14:25.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b825c4d28ebbb5d-00631f9321-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.686 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:25.687 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:25.687 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:25.688 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:25.688 DEBUG:boto:Method: HEAD 2022-09-12T20:14:25.689 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:25.689 DEBUG:boto:Data: 2022-09-12T20:14:25.690 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.690 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.691 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.691 DEBUG:boto:Params: {} 2022-09-12T20:14:25.692 DEBUG:boto:Token: None 2022-09-12T20:14:25.692 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:25.693 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DqZF47LFsbjwb4TAOI6dxAfpQro= 2022-09-12T20:14:25.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DqZF47LFsbjwb4TAOI6dxAfpQro=', 'Content-Length': '0'} 2022-09-12T20:14:25.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007130e5783d90baae-00631f9321-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.697 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:25.697 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:25.698 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:25.698 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:25.699 DEBUG:boto:Method: HEAD 2022-09-12T20:14:25.699 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:25.700 DEBUG:boto:Data: 2022-09-12T20:14:25.700 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.701 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.701 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.702 DEBUG:boto:Params: {} 2022-09-12T20:14:25.702 DEBUG:boto:Token: None 2022-09-12T20:14:25.703 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:25.704 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DqZF47LFsbjwb4TAOI6dxAfpQro= 2022-09-12T20:14:25.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DqZF47LFsbjwb4TAOI6dxAfpQro=', 'Content-Length': '0'} 2022-09-12T20:14:25.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096293cb081fc4216-00631f9321-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:14:25.707 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:25.708 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:25.708 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:25.709 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:25.709 DEBUG:boto:Method: GET 2022-09-12T20:14:25.710 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:25.711 DEBUG:boto:Data: 2022-09-12T20:14:25.711 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.712 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.712 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.713 DEBUG:boto:Params: {} 2022-09-12T20:14:25.714 DEBUG:boto:Token: None 2022-09-12T20:14:25.714 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:25.715 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1S3l+7yGo3uzgMRBwWfBC5/EPvI= 2022-09-12T20:14:25.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1S3l+7yGo3uzgMRBwWfBC5/EPvI=', 'Content-Length': '0'} 2022-09-12T20:14:25.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057bf150fa504a73a-00631f9321-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.718 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:25.719 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:25.719 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:25.720 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:14:25.721 DEBUG:boto:Method: GET 2022-09-12T20:14:25.721 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:25.722 DEBUG:boto:Data: 2022-09-12T20:14:25.722 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.723 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.723 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.724 DEBUG:boto:Params: {} 2022-09-12T20:14:25.725 DEBUG:boto:Token: None 2022-09-12T20:14:25.725 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:14:25.726 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1S3l+7yGo3uzgMRBwWfBC5/EPvI= 2022-09-12T20:14:25.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1S3l+7yGo3uzgMRBwWfBC5/EPvI=', 'Content-Length': '0'} 2022-09-12T20:14:25.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b9a143812e488dc-00631f9321-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.730 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:25.730 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:25.731 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:14:25.731 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:14:25.732 DEBUG:boto:Method: GET 2022-09-12T20:14:25.732 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:14:25.733 DEBUG:boto:Data: 2022-09-12T20:14:25.733 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.734 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.734 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.735 DEBUG:boto:Params: {} 2022-09-12T20:14:25.735 DEBUG:boto:Token: None 2022-09-12T20:14:25.736 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/ 2022-09-12T20:14:25.737 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E= 2022-09-12T20:14:25.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E=', 'Content-Length': '0'} 2022-09-12T20:14:25.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045f2851152337841-00631f9321-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.741 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:25.742 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:25.742 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:25.743 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:14:25.743 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:14:25.744 DEBUG:boto:Method: GET 2022-09-12T20:14:25.744 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:14:25.745 DEBUG:boto:Data: 2022-09-12T20:14:25.745 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.746 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.747 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.747 DEBUG:boto:Params: {} 2022-09-12T20:14:25.748 DEBUG:boto:Token: None 2022-09-12T20:14:25.749 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/ 2022-09-12T20:14:25.749 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E= 2022-09-12T20:14:25.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E=', 'Content-Length': '0'} 2022-09-12T20:14:25.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008db7589e495b1639-00631f9321-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.753 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:25.754 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:25.754 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:25.755 DEBUG:boto:Method: HEAD 2022-09-12T20:14:25.756 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:14:25.756 DEBUG:boto:Data: 2022-09-12T20:14:25.757 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.757 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.758 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.758 DEBUG:boto:Params: {} 2022-09-12T20:14:25.759 DEBUG:boto:Token: None 2022-09-12T20:14:25.759 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj85 2022-09-12T20:14:25.760 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gV+sckbChqP9mJroPWGVWbLBBh4= 2022-09-12T20:14:25.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gV+sckbChqP9mJroPWGVWbLBBh4=', 'Content-Length': '0'} 2022-09-12T20:14:25.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038f58f061f2c691a-00631f9321-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.763 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:25.763 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:25.764 DEBUG:boto:Method: HEAD 2022-09-12T20:14:25.764 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:14:25.765 DEBUG:boto:Data: 2022-09-12T20:14:25.765 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.766 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.767 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.767 DEBUG:boto:Params: {} 2022-09-12T20:14:25.768 DEBUG:boto:Token: None 2022-09-12T20:14:25.768 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj85 2022-09-12T20:14:25.769 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gV+sckbChqP9mJroPWGVWbLBBh4= 2022-09-12T20:14:25.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gV+sckbChqP9mJroPWGVWbLBBh4=', 'Content-Length': '0'} 2022-09-12T20:14:25.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b507fb876a07d28c-00631f9321-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:14:25.773 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:25.774 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:25.774 DEBUG:boto:Method: GET 2022-09-12T20:14:25.775 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:14:25.775 DEBUG:boto:Data: 2022-09-12T20:14:25.776 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.776 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.777 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.778 DEBUG:boto:Params: {} 2022-09-12T20:14:25.778 DEBUG:boto:Token: None 2022-09-12T20:14:25.779 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj85 2022-09-12T20:14:25.779 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W57cSarynD80FsHj8cv7iWicWSc= 2022-09-12T20:14:25.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W57cSarynD80FsHj8cv7iWicWSc=', 'Content-Length': '0'} 2022-09-12T20:14:25.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093581ea52f29ad5f-00631f9321-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.826 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:14:25.827 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:14:25.828 DEBUG:boto:Method: GET 2022-09-12T20:14:25.829 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:14:25.830 DEBUG:boto:Data: 2022-09-12T20:14:25.831 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.832 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.833 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.833 DEBUG:boto:Params: {} 2022-09-12T20:14:25.835 DEBUG:boto:Token: None 2022-09-12T20:14:25.835 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj85 2022-09-12T20:14:25.837 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W57cSarynD80FsHj8cv7iWicWSc= 2022-09-12T20:14:25.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W57cSarynD80FsHj8cv7iWicWSc=', 'Content-Length': '0'} 2022-09-12T20:14:25.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000034011e3b9acdb3e-00631f9321-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.841 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:25.842 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:25.843 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:14:25.844 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:14:25.845 DEBUG:boto:Method: GET 2022-09-12T20:14:25.846 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:14:25.847 DEBUG:boto:Data: 2022-09-12T20:14:25.848 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.848 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.849 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.850 DEBUG:boto:Params: {} 2022-09-12T20:14:25.851 DEBUG:boto:Token: None 2022-09-12T20:14:25.852 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/ 2022-09-12T20:14:25.853 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E= 2022-09-12T20:14:25.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E=', 'Content-Length': '0'} 2022-09-12T20:14:25.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4f2f8dbe397ee83-00631f9321-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.858 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:25.860 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:25.861 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:25.861 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:14:25.862 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:14:25.863 DEBUG:boto:Method: GET 2022-09-12T20:14:25.864 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:14:25.865 DEBUG:boto:Data: 2022-09-12T20:14:25.866 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.867 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.868 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.868 DEBUG:boto:Params: {} 2022-09-12T20:14:25.869 DEBUG:boto:Token: None 2022-09-12T20:14:25.870 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/ 2022-09-12T20:14:25.871 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E= 2022-09-12T20:14:25.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E=', 'Content-Length': '0'} 2022-09-12T20:14:25.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6942e9a9ab25738-00631f9321-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.876 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:25.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:14:25.878 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:25.879 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:25.880 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:25.881 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:25.882 DEBUG:boto:Method: GET 2022-09-12T20:14:25.882 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:25.883 DEBUG:boto:Data: 2022-09-12T20:14:25.884 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.884 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.885 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.885 DEBUG:boto:Params: {} 2022-09-12T20:14:25.886 DEBUG:boto:Token: None 2022-09-12T20:14:25.886 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:25.887 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fLKHy9SaRjMYp2woPwrUYiNFKng= 2022-09-12T20:14:25.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fLKHy9SaRjMYp2woPwrUYiNFKng=', 'Content-Length': '0'} 2022-09-12T20:14:25.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060559771253e44b8-00631f9321-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.891 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:25.891 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:25.892 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:25.892 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:25.893 DEBUG:boto:Method: GET 2022-09-12T20:14:25.893 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:25.894 DEBUG:boto:Data: 2022-09-12T20:14:25.894 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.895 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.895 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.896 DEBUG:boto:Params: {} 2022-09-12T20:14:25.897 DEBUG:boto:Token: None 2022-09-12T20:14:25.897 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:25.898 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fLKHy9SaRjMYp2woPwrUYiNFKng= 2022-09-12T20:14:25.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fLKHy9SaRjMYp2woPwrUYiNFKng=', 'Content-Length': '0'} 2022-09-12T20:14:25.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c1099ab1b6f8c83-00631f9321-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.946 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:25.947 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:25.948 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:25.948 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:25.949 DEBUG:boto:Method: HEAD 2022-09-12T20:14:25.949 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:25.950 DEBUG:boto:Data: 2022-09-12T20:14:25.950 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.951 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.951 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.952 DEBUG:boto:Params: {} 2022-09-12T20:14:25.953 DEBUG:boto:Token: None 2022-09-12T20:14:25.953 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:25.954 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xlUEKIQnJbtt43MVOoamAbm77VI= 2022-09-12T20:14:25.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xlUEKIQnJbtt43MVOoamAbm77VI=', 'Content-Length': '0'} 2022-09-12T20:14:25.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053d678fbfbe9496f-00631f9321-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.957 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:25.958 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:25.958 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:25.959 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:25.960 DEBUG:boto:Method: HEAD 2022-09-12T20:14:25.960 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:25.961 DEBUG:boto:Data: 2022-09-12T20:14:25.961 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.962 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.962 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.963 DEBUG:boto:Params: {} 2022-09-12T20:14:25.963 DEBUG:boto:Token: None 2022-09-12T20:14:25.964 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:25.965 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xlUEKIQnJbtt43MVOoamAbm77VI= 2022-09-12T20:14:25.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xlUEKIQnJbtt43MVOoamAbm77VI=', 'Content-Length': '0'} 2022-09-12T20:14:25.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd097df38035d892-00631f9321-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:14:25.968 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:25.969 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:25.970 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:25.970 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:25.971 DEBUG:boto:Method: GET 2022-09-12T20:14:25.972 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:25.973 DEBUG:boto:Data: 2022-09-12T20:14:25.973 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.974 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.974 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.975 DEBUG:boto:Params: {} 2022-09-12T20:14:25.976 DEBUG:boto:Token: None 2022-09-12T20:14:25.976 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:25.977 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fLKHy9SaRjMYp2woPwrUYiNFKng= 2022-09-12T20:14:25.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fLKHy9SaRjMYp2woPwrUYiNFKng=', 'Content-Length': '0'} 2022-09-12T20:14:25.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087a4b3267d4ad34e-00631f9321-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.980 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:25.981 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:25.981 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:25.982 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:14:25.982 DEBUG:boto:Method: GET 2022-09-12T20:14:25.983 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:25.984 DEBUG:boto:Data: 2022-09-12T20:14:25.984 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.985 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:25.986 DEBUG:boto:Port: 8000 2022-09-12T20:14:25.986 DEBUG:boto:Params: {} 2022-09-12T20:14:25.987 DEBUG:boto:Token: None 2022-09-12T20:14:25.988 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:14:25.988 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fLKHy9SaRjMYp2woPwrUYiNFKng= 2022-09-12T20:14:25.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fLKHy9SaRjMYp2woPwrUYiNFKng=', 'Content-Length': '0'} 2022-09-12T20:14:25.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000594561a14f053f59-00631f9321-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:25.992 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:25.992 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:25.993 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:14:25.994 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:14:25.994 DEBUG:boto:Method: GET 2022-09-12T20:14:25.995 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:14:25.995 DEBUG:boto:Data: 2022-09-12T20:14:25.996 DEBUG:boto:Headers: {} 2022-09-12T20:14:25.996 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:25.997 DEBUG:boto:Port: 8001 2022-09-12T20:14:25.997 DEBUG:boto:Params: {} 2022-09-12T20:14:25.998 DEBUG:boto:Token: None 2022-09-12T20:14:25.998 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:25 GMT /jdpiak-75/ 2022-09-12T20:14:25.999 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E= 2022-09-12T20:14:26.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:25 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6scUGq1mmFvhDN4O964IFPiiu+E=', 'Content-Length': '0'} 2022-09-12T20:14:26.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfd83ea89c9569d3-00631f9322-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.003 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:26.004 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:26.005 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:26.005 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:14:26.006 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:14:26.006 DEBUG:boto:Method: GET 2022-09-12T20:14:26.007 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:14:26.007 DEBUG:boto:Data: 2022-09-12T20:14:26.008 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.008 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:26.009 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.010 DEBUG:boto:Params: {} 2022-09-12T20:14:26.010 DEBUG:boto:Token: None 2022-09-12T20:14:26.011 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/ 2022-09-12T20:14:26.011 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE= 2022-09-12T20:14:26.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE=', 'Content-Length': '0'} 2022-09-12T20:14:26.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f6c7d6e998e7e28-00631f9322-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.015 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:26.016 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:26.017 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:26.017 DEBUG:boto:Method: HEAD 2022-09-12T20:14:26.018 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:14:26.018 DEBUG:boto:Data: 2022-09-12T20:14:26.019 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.019 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:26.020 DEBUG:boto:Port: 8001 2022-09-12T20:14:26.020 DEBUG:boto:Params: {} 2022-09-12T20:14:26.021 DEBUG:boto:Token: None 2022-09-12T20:14:26.021 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj87 2022-09-12T20:14:26.022 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fevGUyt1C9Dhx50joo3lMxo7YTA= 2022-09-12T20:14:26.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fevGUyt1C9Dhx50joo3lMxo7YTA=', 'Content-Length': '0'} 2022-09-12T20:14:26.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084ea05672c6b98a6-00631f9322-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.025 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:26.026 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:26.026 DEBUG:boto:Method: HEAD 2022-09-12T20:14:26.027 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:14:26.027 DEBUG:boto:Data: 2022-09-12T20:14:26.028 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.028 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:26.029 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.029 DEBUG:boto:Params: {} 2022-09-12T20:14:26.030 DEBUG:boto:Token: None 2022-09-12T20:14:26.030 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj87 2022-09-12T20:14:26.031 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fevGUyt1C9Dhx50joo3lMxo7YTA= 2022-09-12T20:14:26.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fevGUyt1C9Dhx50joo3lMxo7YTA=', 'Content-Length': '0'} 2022-09-12T20:14:26.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af4feb37927a258e-00631f9322-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:14:26.035 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:26.035 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:26.036 DEBUG:boto:Method: GET 2022-09-12T20:14:26.036 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:14:26.037 DEBUG:boto:Data: 2022-09-12T20:14:26.037 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.038 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:26.038 DEBUG:boto:Port: 8001 2022-09-12T20:14:26.039 DEBUG:boto:Params: {} 2022-09-12T20:14:26.040 DEBUG:boto:Token: None 2022-09-12T20:14:26.040 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj87 2022-09-12T20:14:26.041 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oY26mFKp0mGrZ6Zq6wbWPVIHy34= 2022-09-12T20:14:26.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oY26mFKp0mGrZ6Zq6wbWPVIHy34=', 'Content-Length': '0'} 2022-09-12T20:14:26.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c45fdff1d352aa39-00631f9322-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.044 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:14:26.045 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:14:26.045 DEBUG:boto:Method: GET 2022-09-12T20:14:26.046 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:14:26.046 DEBUG:boto:Data: 2022-09-12T20:14:26.047 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.048 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:26.048 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.049 DEBUG:boto:Params: {} 2022-09-12T20:14:26.049 DEBUG:boto:Token: None 2022-09-12T20:14:26.050 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj87 2022-09-12T20:14:26.050 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oY26mFKp0mGrZ6Zq6wbWPVIHy34= 2022-09-12T20:14:26.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oY26mFKp0mGrZ6Zq6wbWPVIHy34=', 'Content-Length': '0'} 2022-09-12T20:14:26.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b30611715ab0e421-00631f9322-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.054 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:26.054 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:26.055 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:14:26.055 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:14:26.056 DEBUG:boto:Method: GET 2022-09-12T20:14:26.056 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:14:26.057 DEBUG:boto:Data: 2022-09-12T20:14:26.057 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.058 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:26.059 DEBUG:boto:Port: 8001 2022-09-12T20:14:26.059 DEBUG:boto:Params: {} 2022-09-12T20:14:26.060 DEBUG:boto:Token: None 2022-09-12T20:14:26.060 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/ 2022-09-12T20:14:26.061 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE= 2022-09-12T20:14:26.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE=', 'Content-Length': '0'} 2022-09-12T20:14:26.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c17f23ecfe260079-00631f9322-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.066 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:26.066 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:26.067 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:26.067 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:14:26.068 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:14:26.068 DEBUG:boto:Method: GET 2022-09-12T20:14:26.069 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:14:26.070 DEBUG:boto:Data: 2022-09-12T20:14:26.070 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.071 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:26.071 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.072 DEBUG:boto:Params: {} 2022-09-12T20:14:26.072 DEBUG:boto:Token: None 2022-09-12T20:14:26.073 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/ 2022-09-12T20:14:26.073 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE= 2022-09-12T20:14:26.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE=', 'Content-Length': '0'} 2022-09-12T20:14:26.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094d04abd1a76b5d1-00631f9322-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.078 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:26.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:14:26.079 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:26.079 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:26.080 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:26.081 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:26.081 DEBUG:boto:Method: GET 2022-09-12T20:14:26.082 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:26.082 DEBUG:boto:Data: 2022-09-12T20:14:26.083 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.083 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:26.084 DEBUG:boto:Port: 8001 2022-09-12T20:14:26.084 DEBUG:boto:Params: {} 2022-09-12T20:14:26.085 DEBUG:boto:Token: None 2022-09-12T20:14:26.085 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:26.086 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A43MkXMObCCGFGF9q6yKFYCvS9Q= 2022-09-12T20:14:26.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A43MkXMObCCGFGF9q6yKFYCvS9Q=', 'Content-Length': '0'} 2022-09-12T20:14:26.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061cd5712d8eb4f68-00631f9322-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.089 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:26.090 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:26.091 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:26.091 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:26.092 DEBUG:boto:Method: GET 2022-09-12T20:14:26.092 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:26.093 DEBUG:boto:Data: 2022-09-12T20:14:26.093 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.094 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:26.094 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.095 DEBUG:boto:Params: {} 2022-09-12T20:14:26.095 DEBUG:boto:Token: None 2022-09-12T20:14:26.096 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:26.097 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A43MkXMObCCGFGF9q6yKFYCvS9Q= 2022-09-12T20:14:26.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A43MkXMObCCGFGF9q6yKFYCvS9Q=', 'Content-Length': '0'} 2022-09-12T20:14:26.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc75f30bd57b3a90-00631f9322-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.100 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:26.100 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:26.101 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:26.101 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:26.102 DEBUG:boto:Method: HEAD 2022-09-12T20:14:26.102 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:26.103 DEBUG:boto:Data: 2022-09-12T20:14:26.103 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.104 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:26.104 DEBUG:boto:Port: 8001 2022-09-12T20:14:26.105 DEBUG:boto:Params: {} 2022-09-12T20:14:26.105 DEBUG:boto:Token: None 2022-09-12T20:14:26.106 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:26.107 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/tomfRsNQM2ruqndfe/oFRTQZrM= 2022-09-12T20:14:26.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/tomfRsNQM2ruqndfe/oFRTQZrM=', 'Content-Length': '0'} 2022-09-12T20:14:26.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006024bf4898830f51-00631f9322-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.110 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:26.111 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:26.111 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:26.112 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:26.112 DEBUG:boto:Method: HEAD 2022-09-12T20:14:26.113 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:26.113 DEBUG:boto:Data: 2022-09-12T20:14:26.114 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.114 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:26.115 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.115 DEBUG:boto:Params: {} 2022-09-12T20:14:26.116 DEBUG:boto:Token: None 2022-09-12T20:14:26.116 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:26.117 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/tomfRsNQM2ruqndfe/oFRTQZrM= 2022-09-12T20:14:26.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/tomfRsNQM2ruqndfe/oFRTQZrM=', 'Content-Length': '0'} 2022-09-12T20:14:26.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b1bd0afb1d5d756-00631f9322-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:14:26.121 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:26.122 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:26.122 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:26.123 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:26.123 DEBUG:boto:Method: GET 2022-09-12T20:14:26.124 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:26.124 DEBUG:boto:Data: 2022-09-12T20:14:26.125 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.125 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:26.126 DEBUG:boto:Port: 8001 2022-09-12T20:14:26.126 DEBUG:boto:Params: {} 2022-09-12T20:14:26.127 DEBUG:boto:Token: None 2022-09-12T20:14:26.127 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:26.128 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A43MkXMObCCGFGF9q6yKFYCvS9Q= 2022-09-12T20:14:26.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A43MkXMObCCGFGF9q6yKFYCvS9Q=', 'Content-Length': '0'} 2022-09-12T20:14:26.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec1891b651ae4690-00631f9322-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.132 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:26.132 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:26.133 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:26.133 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:14:26.134 DEBUG:boto:Method: GET 2022-09-12T20:14:26.135 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:26.135 DEBUG:boto:Data: 2022-09-12T20:14:26.136 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.136 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:26.137 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.137 DEBUG:boto:Params: {} 2022-09-12T20:14:26.138 DEBUG:boto:Token: None 2022-09-12T20:14:26.138 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:14:26.139 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A43MkXMObCCGFGF9q6yKFYCvS9Q= 2022-09-12T20:14:26.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A43MkXMObCCGFGF9q6yKFYCvS9Q=', 'Content-Length': '0'} 2022-09-12T20:14:26.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dba57603a93b9af2-00631f9322-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.143 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:26.143 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:26.144 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:14:26.144 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:14:26.145 DEBUG:boto:Method: GET 2022-09-12T20:14:26.145 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:14:26.146 DEBUG:boto:Data: 2022-09-12T20:14:26.146 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.147 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:26.147 DEBUG:boto:Port: 8001 2022-09-12T20:14:26.148 DEBUG:boto:Params: {} 2022-09-12T20:14:26.148 DEBUG:boto:Token: None 2022-09-12T20:14:26.149 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/ 2022-09-12T20:14:26.150 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE= 2022-09-12T20:14:26.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE=', 'Content-Length': '0'} 2022-09-12T20:14:26.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a69d0a8985cc2056-00631f9322-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.194 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:26.195 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:26.195 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:26.196 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:14:26.197 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:14:26.197 DEBUG:boto:Method: GET 2022-09-12T20:14:26.198 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:14:26.198 DEBUG:boto:Data: 2022-09-12T20:14:26.199 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.199 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:26.200 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.200 DEBUG:boto:Params: {} 2022-09-12T20:14:26.201 DEBUG:boto:Token: None 2022-09-12T20:14:26.202 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/ 2022-09-12T20:14:26.202 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE= 2022-09-12T20:14:26.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE=', 'Content-Length': '0'} 2022-09-12T20:14:26.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bf4f742cf58dbd2-00631f9322-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.206 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:26.207 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:26.208 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:26.208 DEBUG:boto:Method: HEAD 2022-09-12T20:14:26.209 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:14:26.209 DEBUG:boto:Data: 2022-09-12T20:14:26.210 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.210 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:26.211 DEBUG:boto:Port: 8001 2022-09-12T20:14:26.212 DEBUG:boto:Params: {} 2022-09-12T20:14:26.212 DEBUG:boto:Token: None 2022-09-12T20:14:26.213 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj92 2022-09-12T20:14:26.214 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IwjrtyY0cqZ/9rzoyDI/gKHJinc= 2022-09-12T20:14:26.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IwjrtyY0cqZ/9rzoyDI/gKHJinc=', 'Content-Length': '0'} 2022-09-12T20:14:26.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb71e4ca570385ad-00631f9322-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.217 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:26.217 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:26.218 DEBUG:boto:Method: HEAD 2022-09-12T20:14:26.218 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:14:26.219 DEBUG:boto:Data: 2022-09-12T20:14:26.219 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.220 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:26.220 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.221 DEBUG:boto:Params: {} 2022-09-12T20:14:26.221 DEBUG:boto:Token: None 2022-09-12T20:14:26.222 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj92 2022-09-12T20:14:26.223 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IwjrtyY0cqZ/9rzoyDI/gKHJinc= 2022-09-12T20:14:26.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IwjrtyY0cqZ/9rzoyDI/gKHJinc=', 'Content-Length': '0'} 2022-09-12T20:14:26.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008119d7a38ad34fba-00631f9322-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:14:26.227 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:26.227 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:26.228 DEBUG:boto:Method: GET 2022-09-12T20:14:26.229 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:14:26.229 DEBUG:boto:Data: 2022-09-12T20:14:26.230 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.230 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:26.231 DEBUG:boto:Port: 8001 2022-09-12T20:14:26.231 DEBUG:boto:Params: {} 2022-09-12T20:14:26.232 DEBUG:boto:Token: None 2022-09-12T20:14:26.232 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj92 2022-09-12T20:14:26.233 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lBYJ5gLVLTCfByZc/Y+YgFDwXmI= 2022-09-12T20:14:26.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lBYJ5gLVLTCfByZc/Y+YgFDwXmI=', 'Content-Length': '0'} 2022-09-12T20:14:26.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac479f50f33d3e51-00631f9322-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.237 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:14:26.237 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:14:26.238 DEBUG:boto:Method: GET 2022-09-12T20:14:26.238 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:14:26.239 DEBUG:boto:Data: 2022-09-12T20:14:26.239 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.240 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:26.240 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.241 DEBUG:boto:Params: {} 2022-09-12T20:14:26.242 DEBUG:boto:Token: None 2022-09-12T20:14:26.242 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj92 2022-09-12T20:14:26.243 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lBYJ5gLVLTCfByZc/Y+YgFDwXmI= 2022-09-12T20:14:26.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lBYJ5gLVLTCfByZc/Y+YgFDwXmI=', 'Content-Length': '0'} 2022-09-12T20:14:26.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8ab57b08f697521-00631f9322-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.286 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:26.287 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:26.287 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:14:26.288 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:14:26.288 DEBUG:boto:Method: GET 2022-09-12T20:14:26.289 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:14:26.289 DEBUG:boto:Data: 2022-09-12T20:14:26.290 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.291 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:26.291 DEBUG:boto:Port: 8001 2022-09-12T20:14:26.292 DEBUG:boto:Params: {} 2022-09-12T20:14:26.292 DEBUG:boto:Token: None 2022-09-12T20:14:26.293 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/ 2022-09-12T20:14:26.293 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE= 2022-09-12T20:14:26.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE=', 'Content-Length': '0'} 2022-09-12T20:14:26.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b5373f99a78d1f2-00631f9322-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.297 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:26.298 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:26.299 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:26.299 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:14:26.300 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:14:26.300 DEBUG:boto:Method: GET 2022-09-12T20:14:26.301 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:14:26.301 DEBUG:boto:Data: 2022-09-12T20:14:26.302 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.302 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:26.303 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.303 DEBUG:boto:Params: {} 2022-09-12T20:14:26.304 DEBUG:boto:Token: None 2022-09-12T20:14:26.304 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/ 2022-09-12T20:14:26.305 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE= 2022-09-12T20:14:26.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE=', 'Content-Length': '0'} 2022-09-12T20:14:26.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f67eba0819d7ad49-00631f9322-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.309 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:26.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-12T20:14:26.310 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:26.311 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:26.311 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:26.312 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:26.312 DEBUG:boto:Method: GET 2022-09-12T20:14:26.313 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:26.313 DEBUG:boto:Data: 2022-09-12T20:14:26.314 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.314 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:26.315 DEBUG:boto:Port: 8001 2022-09-12T20:14:26.316 DEBUG:boto:Params: {} 2022-09-12T20:14:26.316 DEBUG:boto:Token: None 2022-09-12T20:14:26.317 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:26.317 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tILMobnfKPx3hxhohZbxz1sV52Q= 2022-09-12T20:14:26.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tILMobnfKPx3hxhohZbxz1sV52Q=', 'Content-Length': '0'} 2022-09-12T20:14:26.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044e944c949a20a61-00631f9322-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.321 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:26.321 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:26.322 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:26.322 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:26.323 DEBUG:boto:Method: GET 2022-09-12T20:14:26.323 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:26.324 DEBUG:boto:Data: 2022-09-12T20:14:26.324 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.325 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:26.325 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.326 DEBUG:boto:Params: {} 2022-09-12T20:14:26.327 DEBUG:boto:Token: None 2022-09-12T20:14:26.327 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:26.328 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tILMobnfKPx3hxhohZbxz1sV52Q= 2022-09-12T20:14:26.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tILMobnfKPx3hxhohZbxz1sV52Q=', 'Content-Length': '0'} 2022-09-12T20:14:26.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000762a6f405f33eb98-00631f9322-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.331 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:26.332 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:26.332 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:26.333 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:26.333 DEBUG:boto:Method: HEAD 2022-09-12T20:14:26.334 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:26.334 DEBUG:boto:Data: 2022-09-12T20:14:26.335 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.335 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:26.336 DEBUG:boto:Port: 8001 2022-09-12T20:14:26.336 DEBUG:boto:Params: {} 2022-09-12T20:14:26.337 DEBUG:boto:Token: None 2022-09-12T20:14:26.337 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:26.338 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dVXNsdVnQs4JFcnAivjOVJdoNVs= 2022-09-12T20:14:26.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dVXNsdVnQs4JFcnAivjOVJdoNVs=', 'Content-Length': '0'} 2022-09-12T20:14:26.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005dd32d262f26e5d-00631f9322-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.341 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:26.342 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:26.342 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:26.343 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:26.343 DEBUG:boto:Method: HEAD 2022-09-12T20:14:26.343 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:26.344 DEBUG:boto:Data: 2022-09-12T20:14:26.344 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.345 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:26.345 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.346 DEBUG:boto:Params: {} 2022-09-12T20:14:26.346 DEBUG:boto:Token: None 2022-09-12T20:14:26.347 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:26.348 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dVXNsdVnQs4JFcnAivjOVJdoNVs= 2022-09-12T20:14:26.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dVXNsdVnQs4JFcnAivjOVJdoNVs=', 'Content-Length': '0'} 2022-09-12T20:14:26.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1f192e1ee019393-00631f9322-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-12T20:14:26.351 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:26.352 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:26.352 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:26.353 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:26.353 DEBUG:boto:Method: GET 2022-09-12T20:14:26.354 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:26.354 DEBUG:boto:Data: 2022-09-12T20:14:26.355 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.355 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:26.356 DEBUG:boto:Port: 8001 2022-09-12T20:14:26.356 DEBUG:boto:Params: {} 2022-09-12T20:14:26.357 DEBUG:boto:Token: None 2022-09-12T20:14:26.358 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:26.358 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tILMobnfKPx3hxhohZbxz1sV52Q= 2022-09-12T20:14:26.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tILMobnfKPx3hxhohZbxz1sV52Q=', 'Content-Length': '0'} 2022-09-12T20:14:26.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037abe7978f3434b1-00631f9322-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.362 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:26.362 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:26.363 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:26.363 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:14:26.364 DEBUG:boto:Method: GET 2022-09-12T20:14:26.364 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:26.365 DEBUG:boto:Data: 2022-09-12T20:14:26.365 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.366 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:26.366 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.367 DEBUG:boto:Params: {} 2022-09-12T20:14:26.367 DEBUG:boto:Token: None 2022-09-12T20:14:26.368 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:14:26.369 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tILMobnfKPx3hxhohZbxz1sV52Q= 2022-09-12T20:14:26.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tILMobnfKPx3hxhohZbxz1sV52Q=', 'Content-Length': '0'} 2022-09-12T20:14:26.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048b63937c3f541b8-00631f9322-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.414 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:26.415 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:26.415 DEBUG:boto:path=/jdpiak-75/?prefix=obj93 2022-09-12T20:14:26.416 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj93 2022-09-12T20:14:26.416 DEBUG:boto:Method: GET 2022-09-12T20:14:26.417 DEBUG:boto:Path: /jdpiak-75/?prefix=obj93 2022-09-12T20:14:26.417 DEBUG:boto:Data: 2022-09-12T20:14:26.418 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.419 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:26.419 DEBUG:boto:Port: 8001 2022-09-12T20:14:26.420 DEBUG:boto:Params: {} 2022-09-12T20:14:26.420 DEBUG:boto:Token: None 2022-09-12T20:14:26.421 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/ 2022-09-12T20:14:26.421 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE= 2022-09-12T20:14:26.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE=', 'Content-Length': '0'} 2022-09-12T20:14:26.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e28c65c354abede1-00631f9322-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.425 DEBUG:boto:b'jdpiak-75obj931000falseobj932022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:26.426 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:26.427 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:26.427 DEBUG:boto:path=/jdpiak-75/?prefix=obj93 2022-09-12T20:14:26.428 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj93 2022-09-12T20:14:26.429 DEBUG:boto:Method: GET 2022-09-12T20:14:26.429 DEBUG:boto:Path: /jdpiak-75/?prefix=obj93 2022-09-12T20:14:26.430 DEBUG:boto:Data: 2022-09-12T20:14:26.430 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.431 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:26.431 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.432 DEBUG:boto:Params: {} 2022-09-12T20:14:26.432 DEBUG:boto:Token: None 2022-09-12T20:14:26.433 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/ 2022-09-12T20:14:26.433 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE= 2022-09-12T20:14:26.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE=', 'Content-Length': '0'} 2022-09-12T20:14:26.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c771548ca51f7f4-00631f9322-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.438 DEBUG:boto:b'jdpiak-75obj931000falseobj932022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:26.438 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:26.439 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:26.440 DEBUG:boto:Method: HEAD 2022-09-12T20:14:26.440 DEBUG:boto:Path: /jdpiak-75/obj93 2022-09-12T20:14:26.441 DEBUG:boto:Data: 2022-09-12T20:14:26.441 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.442 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:26.442 DEBUG:boto:Port: 8001 2022-09-12T20:14:26.443 DEBUG:boto:Params: {} 2022-09-12T20:14:26.443 DEBUG:boto:Token: None 2022-09-12T20:14:26.444 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj93 2022-09-12T20:14:26.444 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yUgIc8mwi+161gQgE6qH5thN7Qs= 2022-09-12T20:14:26.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yUgIc8mwi+161gQgE6qH5thN7Qs=', 'Content-Length': '0'} 2022-09-12T20:14:26.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000404f081a07feee13-00631f9322-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.448 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:26.449 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:26.450 DEBUG:boto:Method: HEAD 2022-09-12T20:14:26.450 DEBUG:boto:Path: /jdpiak-75/obj93 2022-09-12T20:14:26.451 DEBUG:boto:Data: 2022-09-12T20:14:26.451 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.452 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:26.452 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.453 DEBUG:boto:Params: {} 2022-09-12T20:14:26.453 DEBUG:boto:Token: None 2022-09-12T20:14:26.454 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj93 2022-09-12T20:14:26.454 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yUgIc8mwi+161gQgE6qH5thN7Qs= 2022-09-12T20:14:26.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yUgIc8mwi+161gQgE6qH5thN7Qs=', 'Content-Length': '0'} 2022-09-12T20:14:26.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029e41e1257549577-00631f9322-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-12T20:14:26.459 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:26.459 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:26.460 DEBUG:boto:Method: GET 2022-09-12T20:14:26.460 DEBUG:boto:Path: /jdpiak-75/obj93 2022-09-12T20:14:26.461 DEBUG:boto:Data: 2022-09-12T20:14:26.462 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.462 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:26.463 DEBUG:boto:Port: 8001 2022-09-12T20:14:26.463 DEBUG:boto:Params: {} 2022-09-12T20:14:26.464 DEBUG:boto:Token: None 2022-09-12T20:14:26.464 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj93 2022-09-12T20:14:26.465 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hJLWw0OJmjiUUJMnQ0yuQyckIXI= 2022-09-12T20:14:26.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hJLWw0OJmjiUUJMnQ0yuQyckIXI=', 'Content-Length': '0'} 2022-09-12T20:14:26.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bd06920300a6313-00631f9322-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.468 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:14:26.469 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:14:26.469 DEBUG:boto:Method: GET 2022-09-12T20:14:26.470 DEBUG:boto:Path: /jdpiak-75/obj93 2022-09-12T20:14:26.471 DEBUG:boto:Data: 2022-09-12T20:14:26.471 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.472 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:26.472 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.473 DEBUG:boto:Params: {} 2022-09-12T20:14:26.473 DEBUG:boto:Token: None 2022-09-12T20:14:26.474 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj93 2022-09-12T20:14:26.475 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hJLWw0OJmjiUUJMnQ0yuQyckIXI= 2022-09-12T20:14:26.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hJLWw0OJmjiUUJMnQ0yuQyckIXI=', 'Content-Length': '0'} 2022-09-12T20:14:26.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bcbbbd22a031ba3-00631f9322-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.478 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:26.479 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:26.479 DEBUG:boto:path=/jdpiak-75/?prefix=obj93 2022-09-12T20:14:26.480 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj93 2022-09-12T20:14:26.480 DEBUG:boto:Method: GET 2022-09-12T20:14:26.481 DEBUG:boto:Path: /jdpiak-75/?prefix=obj93 2022-09-12T20:14:26.482 DEBUG:boto:Data: 2022-09-12T20:14:26.482 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.483 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:26.483 DEBUG:boto:Port: 8001 2022-09-12T20:14:26.484 DEBUG:boto:Params: {} 2022-09-12T20:14:26.484 DEBUG:boto:Token: None 2022-09-12T20:14:26.485 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/ 2022-09-12T20:14:26.486 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE= 2022-09-12T20:14:26.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE=', 'Content-Length': '0'} 2022-09-12T20:14:26.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbd64b89949c93d1-00631f9322-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.490 DEBUG:boto:b'jdpiak-75obj931000falseobj932022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:26.491 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:26.491 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:26.492 DEBUG:boto:path=/jdpiak-75/?prefix=obj93 2022-09-12T20:14:26.492 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj93 2022-09-12T20:14:26.493 DEBUG:boto:Method: GET 2022-09-12T20:14:26.493 DEBUG:boto:Path: /jdpiak-75/?prefix=obj93 2022-09-12T20:14:26.494 DEBUG:boto:Data: 2022-09-12T20:14:26.494 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.495 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:26.496 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.497 DEBUG:boto:Params: {} 2022-09-12T20:14:26.498 DEBUG:boto:Token: None 2022-09-12T20:14:26.499 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/ 2022-09-12T20:14:26.499 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE= 2022-09-12T20:14:26.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE=', 'Content-Length': '0'} 2022-09-12T20:14:26.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000361c927bc33886a5-00631f9322-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.503 DEBUG:boto:b'jdpiak-75obj931000falseobj932022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:26.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:14:26.505 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:26.505 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:26.506 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:26.506 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:26.507 DEBUG:boto:Method: GET 2022-09-12T20:14:26.507 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:26.508 DEBUG:boto:Data: 2022-09-12T20:14:26.508 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.509 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:26.509 DEBUG:boto:Port: 8001 2022-09-12T20:14:26.510 DEBUG:boto:Params: {} 2022-09-12T20:14:26.511 DEBUG:boto:Token: None 2022-09-12T20:14:26.511 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:26.512 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E0S8+JKcvsOOR3z6x7FWiz3oEyA= 2022-09-12T20:14:26.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E0S8+JKcvsOOR3z6x7FWiz3oEyA=', 'Content-Length': '0'} 2022-09-12T20:14:26.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e36a0a9500150979-00631f9322-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.515 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:26.516 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:26.517 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:26.517 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:26.518 DEBUG:boto:Method: GET 2022-09-12T20:14:26.518 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:26.519 DEBUG:boto:Data: 2022-09-12T20:14:26.519 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.520 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:26.520 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.521 DEBUG:boto:Params: {} 2022-09-12T20:14:26.521 DEBUG:boto:Token: None 2022-09-12T20:14:26.522 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:26.523 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E0S8+JKcvsOOR3z6x7FWiz3oEyA= 2022-09-12T20:14:26.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E0S8+JKcvsOOR3z6x7FWiz3oEyA=', 'Content-Length': '0'} 2022-09-12T20:14:26.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7ea85052be01486-00631f9322-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.526 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:26.527 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:26.528 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:26.528 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:26.529 DEBUG:boto:Method: HEAD 2022-09-12T20:14:26.529 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:26.530 DEBUG:boto:Data: 2022-09-12T20:14:26.530 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.531 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:26.531 DEBUG:boto:Port: 8001 2022-09-12T20:14:26.532 DEBUG:boto:Params: {} 2022-09-12T20:14:26.532 DEBUG:boto:Token: None 2022-09-12T20:14:26.533 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:26.533 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qrlNubs1My3DNAoW0NS74uwpESA= 2022-09-12T20:14:26.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qrlNubs1My3DNAoW0NS74uwpESA=', 'Content-Length': '0'} 2022-09-12T20:14:26.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b8ddcd1af01b907-00631f9322-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.537 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:26.537 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:26.538 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:26.538 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:26.539 DEBUG:boto:Method: HEAD 2022-09-12T20:14:26.539 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:26.540 DEBUG:boto:Data: 2022-09-12T20:14:26.540 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.541 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:26.541 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.542 DEBUG:boto:Params: {} 2022-09-12T20:14:26.543 DEBUG:boto:Token: None 2022-09-12T20:14:26.543 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:26.544 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qrlNubs1My3DNAoW0NS74uwpESA= 2022-09-12T20:14:26.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qrlNubs1My3DNAoW0NS74uwpESA=', 'Content-Length': '0'} 2022-09-12T20:14:26.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c97cc1b49532bad6-00631f9322-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:14:26.548 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:26.548 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:26.549 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:26.549 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:26.550 DEBUG:boto:Method: GET 2022-09-12T20:14:26.551 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:26.551 DEBUG:boto:Data: 2022-09-12T20:14:26.552 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.552 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:26.553 DEBUG:boto:Port: 8001 2022-09-12T20:14:26.553 DEBUG:boto:Params: {} 2022-09-12T20:14:26.554 DEBUG:boto:Token: None 2022-09-12T20:14:26.554 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:26.555 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E0S8+JKcvsOOR3z6x7FWiz3oEyA= 2022-09-12T20:14:26.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E0S8+JKcvsOOR3z6x7FWiz3oEyA=', 'Content-Length': '0'} 2022-09-12T20:14:26.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6086904cc2e604b-00631f9322-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.559 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:26.559 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:26.560 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:26.561 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:14:26.561 DEBUG:boto:Method: GET 2022-09-12T20:14:26.562 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:26.562 DEBUG:boto:Data: 2022-09-12T20:14:26.563 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.563 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:26.564 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.564 DEBUG:boto:Params: {} 2022-09-12T20:14:26.565 DEBUG:boto:Token: None 2022-09-12T20:14:26.566 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:14:26.566 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E0S8+JKcvsOOR3z6x7FWiz3oEyA= 2022-09-12T20:14:26.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E0S8+JKcvsOOR3z6x7FWiz3oEyA=', 'Content-Length': '0'} 2022-09-12T20:14:26.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b205e1ee7856f6f-00631f9322-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.610 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:26.611 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:26.612 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:14:26.612 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:14:26.613 DEBUG:boto:Method: GET 2022-09-12T20:14:26.613 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:14:26.614 DEBUG:boto:Data: 2022-09-12T20:14:26.614 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.615 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:26.615 DEBUG:boto:Port: 8001 2022-09-12T20:14:26.616 DEBUG:boto:Params: {} 2022-09-12T20:14:26.617 DEBUG:boto:Token: None 2022-09-12T20:14:26.617 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/ 2022-09-12T20:14:26.618 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE= 2022-09-12T20:14:26.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE=', 'Content-Length': '0'} 2022-09-12T20:14:26.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfc3ad5cb049d7d6-00631f9322-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.622 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:26.623 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:26.624 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:26.624 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:14:26.625 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:14:26.626 DEBUG:boto:Method: GET 2022-09-12T20:14:26.626 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:14:26.627 DEBUG:boto:Data: 2022-09-12T20:14:26.627 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.628 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:26.628 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.629 DEBUG:boto:Params: {} 2022-09-12T20:14:26.629 DEBUG:boto:Token: None 2022-09-12T20:14:26.630 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/ 2022-09-12T20:14:26.631 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE= 2022-09-12T20:14:26.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE=', 'Content-Length': '0'} 2022-09-12T20:14:26.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e9f11255fc1d875-00631f9322-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.635 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:26.636 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:26.636 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:26.637 DEBUG:boto:Method: HEAD 2022-09-12T20:14:26.637 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:14:26.638 DEBUG:boto:Data: 2022-09-12T20:14:26.638 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.639 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:26.639 DEBUG:boto:Port: 8001 2022-09-12T20:14:26.640 DEBUG:boto:Params: {} 2022-09-12T20:14:26.641 DEBUG:boto:Token: None 2022-09-12T20:14:26.641 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj95 2022-09-12T20:14:26.642 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EmDhObhozP98HPuNydzq9wTAAtc= 2022-09-12T20:14:26.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EmDhObhozP98HPuNydzq9wTAAtc=', 'Content-Length': '0'} 2022-09-12T20:14:26.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006238232f9e3d2738-00631f9322-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.645 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:26.646 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:26.646 DEBUG:boto:Method: HEAD 2022-09-12T20:14:26.647 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:14:26.647 DEBUG:boto:Data: 2022-09-12T20:14:26.648 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.648 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:26.649 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.649 DEBUG:boto:Params: {} 2022-09-12T20:14:26.650 DEBUG:boto:Token: None 2022-09-12T20:14:26.651 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj95 2022-09-12T20:14:26.651 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EmDhObhozP98HPuNydzq9wTAAtc= 2022-09-12T20:14:26.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EmDhObhozP98HPuNydzq9wTAAtc=', 'Content-Length': '0'} 2022-09-12T20:14:26.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4009f5519b9f784-00631f9322-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:14:26.655 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:26.656 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:26.656 DEBUG:boto:Method: GET 2022-09-12T20:14:26.657 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:14:26.657 DEBUG:boto:Data: 2022-09-12T20:14:26.658 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.659 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:26.659 DEBUG:boto:Port: 8001 2022-09-12T20:14:26.660 DEBUG:boto:Params: {} 2022-09-12T20:14:26.661 DEBUG:boto:Token: None 2022-09-12T20:14:26.661 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj95 2022-09-12T20:14:26.662 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Lb5KmZwPE8H0IxLdDlKbCR+5i5o= 2022-09-12T20:14:26.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Lb5KmZwPE8H0IxLdDlKbCR+5i5o=', 'Content-Length': '0'} 2022-09-12T20:14:26.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f6293dbe429a99e-00631f9322-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.665 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:14:26.666 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:14:26.666 DEBUG:boto:Method: GET 2022-09-12T20:14:26.667 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:14:26.673 DEBUG:boto:Data: 2022-09-12T20:14:26.673 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.674 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:26.675 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.675 DEBUG:boto:Params: {} 2022-09-12T20:14:26.676 DEBUG:boto:Token: None 2022-09-12T20:14:26.676 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj95 2022-09-12T20:14:26.677 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Lb5KmZwPE8H0IxLdDlKbCR+5i5o= 2022-09-12T20:14:26.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Lb5KmZwPE8H0IxLdDlKbCR+5i5o=', 'Content-Length': '0'} 2022-09-12T20:14:26.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005de46223823defaa-00631f9322-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.722 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:26.723 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:26.724 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:14:26.724 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:14:26.725 DEBUG:boto:Method: GET 2022-09-12T20:14:26.725 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:14:26.726 DEBUG:boto:Data: 2022-09-12T20:14:26.727 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.727 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:14:26.728 DEBUG:boto:Port: 8001 2022-09-12T20:14:26.728 DEBUG:boto:Params: {} 2022-09-12T20:14:26.729 DEBUG:boto:Token: None 2022-09-12T20:14:26.730 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/ 2022-09-12T20:14:26.731 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE= 2022-09-12T20:14:26.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE=', 'Content-Length': '0'} 2022-09-12T20:14:26.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f771a7f8afd2c8ed-00631f9322-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.735 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:26.736 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:26.737 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:26.737 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:14:26.738 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:14:26.739 DEBUG:boto:Method: GET 2022-09-12T20:14:26.739 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:14:26.740 DEBUG:boto:Data: 2022-09-12T20:14:26.740 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.741 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:14:26.741 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.742 DEBUG:boto:Params: {} 2022-09-12T20:14:26.743 DEBUG:boto:Token: None 2022-09-12T20:14:26.743 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/ 2022-09-12T20:14:26.744 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE= 2022-09-12T20:14:26.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FVBxLamUk7LlR4L9opZxX5jU8TE=', 'Content-Length': '0'} 2022-09-12T20:14:26.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030dddeaba914d57e-00631f9322-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.749 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:14:26.751 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-75 zones={test-zone3, test-zone2} 2022-09-12T20:14:26.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:14:26.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:14:26.754 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T20:14:26.816 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:14:26.816 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:14:26.877 INFO:teuthology.orchestra.run.smithi035.stdout:tenant: 2022-09-12T20:14:26.877 INFO:teuthology.orchestra.run.smithi035.stdout:bucket name: jdpiak-75 2022-09-12T20:14:26.892 INFO:teuthology.orchestra.run.smithi035.stdout:total entries: 30 2022-09-12T20:14:26.962 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:26.963 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:26.963 DEBUG:boto:Method: HEAD 2022-09-12T20:14:26.964 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:14:26.964 DEBUG:boto:Data: 2022-09-12T20:14:26.965 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.965 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:26.966 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.966 DEBUG:boto:Params: {} 2022-09-12T20:14:26.967 DEBUG:boto:Token: None 2022-09-12T20:14:26.967 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/ 2022-09-12T20:14:26.968 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7QutF8MRjHsR8yz/psyUgxky05s= 2022-09-12T20:14:26.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7QutF8MRjHsR8yz/psyUgxky05s=', 'Content-Length': '0'} 2022-09-12T20:14:26.972 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069d201d68da7004a-00631f9322-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.973 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:14:26.974 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:14:26.974 DEBUG:boto:Method: PUT 2022-09-12T20:14:26.975 DEBUG:boto:Path: /jdpiak-75/obj37 2022-09-12T20:14:26.975 DEBUG:boto:Data: 2022-09-12T20:14:26.976 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:14:26.976 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:26.977 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.977 DEBUG:boto:Params: {} 2022-09-12T20:14:26.978 DEBUG:boto:Token: None 2022-09-12T20:14:26.978 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/obj37 2022-09-12T20:14:26.979 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/ln36BqEEG1DksVQDFxznB69jc0= 2022-09-12T20:14:26.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/ln36BqEEG1DksVQDFxznB69jc0='} 2022-09-12T20:14:26.985 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fbc3983a5d73c4ca-00631f9322-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.986 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:26.986 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:26.987 DEBUG:boto:Method: HEAD 2022-09-12T20:14:26.987 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:14:26.988 DEBUG:boto:Data: 2022-09-12T20:14:26.988 DEBUG:boto:Headers: {} 2022-09-12T20:14:26.989 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:26.989 DEBUG:boto:Port: 8000 2022-09-12T20:14:26.990 DEBUG:boto:Params: {} 2022-09-12T20:14:26.990 DEBUG:boto:Token: None 2022-09-12T20:14:26.991 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:26 GMT /jdpiak-75/ 2022-09-12T20:14:26.991 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7QutF8MRjHsR8yz/psyUgxky05s= 2022-09-12T20:14:26.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:26 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7QutF8MRjHsR8yz/psyUgxky05s=', 'Content-Length': '0'} 2022-09-12T20:14:26.995 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3bf60b0a3cfae34-00631f9322-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:26.996 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:14:26.997 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:14:26.997 DEBUG:boto:Method: PUT 2022-09-12T20:14:26.998 DEBUG:boto:Path: /jdpiak-75/obj74 2022-09-12T20:14:26.999 DEBUG:boto:Data: 2022-09-12T20:14:26.999 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:14:27.000 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:27.001 DEBUG:boto:Port: 8000 2022-09-12T20:14:27.001 DEBUG:boto:Params: {} 2022-09-12T20:14:27.002 DEBUG:boto:Token: None 2022-09-12T20:14:27.003 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:14:27 GMT /jdpiak-75/obj74 2022-09-12T20:14:27.003 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F8ukXQtGLIpgNAHfcNvExOpKgsE= 2022-09-12T20:14:27.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:14:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F8ukXQtGLIpgNAHfcNvExOpKgsE='} 2022-09-12T20:14:27.046 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009bc03afee3c3f603-00631f9323-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:27.047 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:27.048 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:27.048 DEBUG:boto:Method: HEAD 2022-09-12T20:14:27.049 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:14:27.049 DEBUG:boto:Data: 2022-09-12T20:14:27.050 DEBUG:boto:Headers: {} 2022-09-12T20:14:27.050 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:27.051 DEBUG:boto:Port: 8000 2022-09-12T20:14:27.051 DEBUG:boto:Params: {} 2022-09-12T20:14:27.052 DEBUG:boto:Token: None 2022-09-12T20:14:27.053 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:27 GMT /jdpiak-75/ 2022-09-12T20:14:27.053 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:75zFk082xfWGdoBuvYR8Iv1gAuI= 2022-09-12T20:14:27.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:75zFk082xfWGdoBuvYR8Iv1gAuI=', 'Content-Length': '0'} 2022-09-12T20:14:27.058 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000bf958e98e937ded-00631f9323-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:27.059 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:14:27.059 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:14:27.060 DEBUG:boto:Method: PUT 2022-09-12T20:14:27.061 DEBUG:boto:Path: /jdpiak-75/obj111 2022-09-12T20:14:27.062 DEBUG:boto:Data: 2022-09-12T20:14:27.063 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:14:27.064 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:27.065 DEBUG:boto:Port: 8000 2022-09-12T20:14:27.065 DEBUG:boto:Params: {} 2022-09-12T20:14:27.066 DEBUG:boto:Token: None 2022-09-12T20:14:27.067 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:14:27 GMT /jdpiak-75/obj111 2022-09-12T20:14:27.068 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S/XjAsfyeDxpau1mRyXRYdfN8uI= 2022-09-12T20:14:27.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:14:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S/XjAsfyeDxpau1mRyXRYdfN8uI='} 2022-09-12T20:14:27.074 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000030ccf1e8fb08ddb4-00631f9323-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:27.075 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:27.075 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:27.076 DEBUG:boto:Method: HEAD 2022-09-12T20:14:27.076 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:14:27.077 DEBUG:boto:Data: 2022-09-12T20:14:27.077 DEBUG:boto:Headers: {} 2022-09-12T20:14:27.078 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:27.078 DEBUG:boto:Port: 8000 2022-09-12T20:14:27.079 DEBUG:boto:Params: {} 2022-09-12T20:14:27.079 DEBUG:boto:Token: None 2022-09-12T20:14:27.080 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:27 GMT /jdpiak-75/ 2022-09-12T20:14:27.081 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:75zFk082xfWGdoBuvYR8Iv1gAuI= 2022-09-12T20:14:27.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:75zFk082xfWGdoBuvYR8Iv1gAuI=', 'Content-Length': '0'} 2022-09-12T20:14:27.085 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000655b0912e1f6c70d-00631f9323-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:27.086 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:14:27.086 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:14:27.087 DEBUG:boto:Method: PUT 2022-09-12T20:14:27.087 DEBUG:boto:Path: /jdpiak-75/obj148 2022-09-12T20:14:27.088 DEBUG:boto:Data: 2022-09-12T20:14:27.088 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:14:27.089 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:27.089 DEBUG:boto:Port: 8000 2022-09-12T20:14:27.090 DEBUG:boto:Params: {} 2022-09-12T20:14:27.091 DEBUG:boto:Token: None 2022-09-12T20:14:27.091 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:14:27 GMT /jdpiak-75/obj148 2022-09-12T20:14:27.092 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KAPzQgMPvLq21ATa0I+5F6xFMMc= 2022-09-12T20:14:27.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:14:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KAPzQgMPvLq21ATa0I+5F6xFMMc='} 2022-09-12T20:14:27.098 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003178164c40204da7-00631f9323-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:27.099 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:14:27.100 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:14:27.101 DEBUG:boto:Method: HEAD 2022-09-12T20:14:27.101 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:14:27.102 DEBUG:boto:Data: 2022-09-12T20:14:27.103 DEBUG:boto:Headers: {} 2022-09-12T20:14:27.103 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:27.104 DEBUG:boto:Port: 8000 2022-09-12T20:14:27.104 DEBUG:boto:Params: {} 2022-09-12T20:14:27.105 DEBUG:boto:Token: None 2022-09-12T20:14:27.106 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:14:27 GMT /jdpiak-75/ 2022-09-12T20:14:27.106 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:75zFk082xfWGdoBuvYR8Iv1gAuI= 2022-09-12T20:14:27.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:14:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:75zFk082xfWGdoBuvYR8Iv1gAuI=', 'Content-Length': '0'} 2022-09-12T20:14:27.112 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc330cf9a668ce90-00631f9323-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:27.113 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:14:27.113 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:14:27.114 DEBUG:boto:Method: PUT 2022-09-12T20:14:27.114 DEBUG:boto:Path: /jdpiak-75/obj185 2022-09-12T20:14:27.115 DEBUG:boto:Data: 2022-09-12T20:14:27.115 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:14:27.116 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:14:27.116 DEBUG:boto:Port: 8000 2022-09-12T20:14:27.117 DEBUG:boto:Params: {} 2022-09-12T20:14:27.117 DEBUG:boto:Token: None 2022-09-12T20:14:27.118 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:14:27 GMT /jdpiak-75/obj185 2022-09-12T20:14:27.119 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+UBm0fP6yyV/9HBrQesupbsXn9I= 2022-09-12T20:14:27.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:14:27 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+UBm0fP6yyV/9HBrQesupbsXn9I='} 2022-09-12T20:14:27.162 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006eba72cfeaa759f2-00631f9323-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:14:27.163 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-75 2022-09-12T20:14:27.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:14:27.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:14:27.165 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T20:14:27.229 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:14:27.229 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:14:27.268 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:14:27.266+0000 7f7911b6e600 1 waiting to reach latest gen 6, on 5.. 2022-09-12T20:14:32.269 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:14:32.267+0000 7f7911b6e600 1 waiting to reach latest gen 6, on 5.. 2022-09-12T20:14:37.270 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:14:37.268+0000 7f7911b6e600 1 waiting to reach latest gen 6, on 5.. 2022-09-12T20:14:42.270 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:14:42.268+0000 7f7911b6e600 1 waiting to reach latest gen 6, on 5.. 2022-09-12T20:14:47.274 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:14:47.271+0000 7f7911b6e600 1 waiting for incremental sync to catch up: 2022-09-12T20:14:47.274 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:14:47.274 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , 00000000002.51.5, , , , , , 00000000002.93.5, , , , , 00000000002.49.5, , , , , , , , , , , , , , , 00000000001.88.5, , , , 00000000001.73.5] 2022-09-12T20:14:52.276 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:14:52.275+0000 7f7911b6e600 1 waiting for incremental sync to catch up: 2022-09-12T20:14:52.277 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:14:52.277 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , 00000000002.51.5, , , , , , 00000000002.93.5, , , , , 00000000002.49.5, , , , , , , , , , , , , , , 00000000001.88.5, , , , 00000000001.73.5] 2022-09-12T20:14:57.279 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:14:57.278+0000 7f7911b6e600 1 waiting for incremental sync to catch up: 2022-09-12T20:14:57.280 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:14:57.280 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , 00000000002.51.5, , , , , , 00000000002.93.5, , , , , 00000000002.49.5, , , , , , , , , , , , , , , 00000000001.88.5, , , , 00000000001.73.5] 2022-09-12T20:15:02.282 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:15:02.280+0000 7f7911b6e600 1 waiting for incremental sync to catch up: 2022-09-12T20:15:02.283 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:15:02.283 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , 00000000002.51.5, , , , , , 00000000002.93.5, , , , , 00000000002.49.5, , , , , , , , , , , , , , , 00000000001.88.5, , , , 00000000001.73.5] 2022-09-12T20:15:07.285 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:15:07.283+0000 7f7911b6e600 1 waiting for incremental sync to catch up: 2022-09-12T20:15:07.285 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , 00000000002.51.5, , , , , , , , , , , 00000000002.49.5, , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:15:07.286 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , 00000000002.51.5, , , , , , 00000000002.93.5, , , , , 00000000002.49.5, , , , , , , , , , , , , , , 00000000001.88.5, , , , 00000000001.73.5] 2022-09-12T20:15:12.288 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:15:12.286+0000 7f7911b6e600 1 waiting for incremental sync to catch up: 2022-09-12T20:15:12.288 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , 00000000002.51.5, , , , , , , , , , , 00000000002.49.5, , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:15:12.289 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , 00000000002.51.5, , , , , , 00000000002.93.5, , , , , 00000000002.49.5, , , , , , , , , , , , , , , 00000000001.88.5, , , , 00000000001.73.5] 2022-09-12T20:15:17.291 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:15:17.289+0000 7f7911b6e600 1 waiting for incremental sync to catch up: 2022-09-12T20:15:17.291 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , 00000000002.51.5, , , , , , , , , , , 00000000002.49.5, , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:15:17.291 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , 00000000002.51.5, , , , , , 00000000002.93.5, , , , , 00000000002.49.5, , , , , , , , , , , , , , , 00000000001.88.5, , , , 00000000001.73.5] 2022-09-12T20:15:22.293 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:15:22.291+0000 7f7911b6e600 1 waiting for incremental sync to catch up: 2022-09-12T20:15:22.294 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , 00000000002.51.5, , , , , , , , , , , 00000000002.49.5, , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:15:22.294 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , 00000000002.51.5, , , , , , 00000000002.93.5, , , , , 00000000002.49.5, , , , , , , , , , , , , , , 00000000001.88.5, , , , 00000000001.73.5] 2022-09-12T20:15:27.296 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:15:27.294+0000 7f7911b6e600 1 waiting for incremental sync to catch up: 2022-09-12T20:15:27.297 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , 00000000002.51.5, , , , , , , , , , , 00000000002.49.5, , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:15:27.297 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , 00000000002.51.5, , , , , , 00000000002.93.5, , , , , 00000000002.49.5, , , , , , , , , , , , , , , 00000000001.88.5, , , , 00000000001.73.5] 2022-09-12T20:15:32.299 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:15:32.297+0000 7f7911b6e600 1 waiting for incremental sync to catch up: 2022-09-12T20:15:32.300 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , 00000000002.51.5, , , , , , , , , , , 00000000002.49.5, , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:15:32.300 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , 00000000002.51.5, , , , , , 00000000002.93.5, , , , , 00000000002.49.5, , , , , , , , , , , , , , , 00000000001.88.5, , , , 00000000001.73.5] 2022-09-12T20:15:37.302 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:15:37.301+0000 7f7911b6e600 1 waiting for incremental sync to catch up: 2022-09-12T20:15:37.302 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , 00000000002.51.5, , , , , , , , , , , 00000000002.49.5, , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:15:37.303 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , 00000000002.51.5, , , , , , 00000000002.93.5, , , , , 00000000002.49.5, , , , , , , , , , , , , , , 00000000001.88.5, , , , 00000000001.73.5] 2022-09-12T20:15:42.305 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:15:42.304+0000 7f7911b6e600 1 waiting for incremental sync to catch up: 2022-09-12T20:15:42.305 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , 00000000002.51.5, , , , , , , , , , , 00000000002.49.5, , , , , , , , , , , , , , , , , , , ] 2022-09-12T20:15:42.306 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , 00000000002.51.5, , , , , , 00000000002.93.5, , , , , 00000000002.49.5, , , , , , , , , , , , , , , 00000000001.88.5, , , , 00000000001.73.5] 2022-09-12T20:15:47.308 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:15:47.306+0000 7f7911b6e600 1 bucket sync caught up with source: 2022-09-12T20:15:47.309 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , 00000000002.51.5, , , , , , 00000000002.93.5, , , , , 00000000002.49.5, , , , , , , , , , , , , , , 00000000001.88.5, , , , 00000000001.73.5] 2022-09-12T20:15:47.309 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , 00000000002.51.5, , , , , , 00000000002.93.5, , , , , 00000000002.49.5, , , , , , , , , , , , , , , 00000000001.88.5, , , , 00000000001.73.5] 2022-09-12T20:15:47.309 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:15:47.306+0000 7f7911b6e600 0 bucket checkpoint complete 2022-09-12T20:15:47.327 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-75 2022-09-12T20:15:47.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:15:47.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:15:47.329 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:15:47.396 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:15:47.396 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:15:47.438 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:15:47.438+0000 7f31ccda9600 1 bucket sync caught up with source: 2022-09-12T20:15:47.439 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , 00000000002.51.5, , , , , , 00000000002.93.5, , , , , 00000000002.49.5, , , , , , , , , , , , , , , 00000000001.88.5, , , , 00000000001.73.5] 2022-09-12T20:15:47.439 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , 00000000002.51.5, , , , , , 00000000002.93.5, , , , , 00000000002.49.5, , , , , , , , , , , , , , , 00000000001.88.5, , , , 00000000001.73.5] 2022-09-12T20:15:47.439 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:15:47.438+0000 7f31ccda9600 0 bucket checkpoint complete 2022-09-12T20:15:47.454 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-75 2022-09-12T20:15:47.455 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:15:47.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:15:47.456 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T20:15:47.520 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:15:47.520 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:15:47.568 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:15:47.566+0000 7f2a6d450600 1 bucket sync caught up with source: 2022-09-12T20:15:47.569 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , 00000000002.51.5, , , , , , 00000000002.93.5, , , , , 00000000002.49.5, , , , , , , , , , , , , , , 00000000001.88.5, , , , 00000000001.73.5] 2022-09-12T20:15:47.569 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , 00000000002.51.5, , , , , , 00000000002.93.5, , , , , 00000000002.49.5, , , , , , , , , , , , , , , 00000000001.88.5, , , , 00000000001.73.5] 2022-09-12T20:15:47.570 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:15:47.566+0000 7f2a6d450600 0 bucket checkpoint complete 2022-09-12T20:15:47.590 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-75 2022-09-12T20:15:47.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:15:47.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:15:47.592 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:15:47.655 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:15:47.655 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:15:47.695 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:15:47.695+0000 7fe4f6ee4600 1 waiting for incremental sync to catch up: 2022-09-12T20:15:47.696 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.648.5, 00000000002.837.5, 00000000002.643.5, 00000000005.359.5, 00000000004.935.5, 00000000001.434.5, 00000000001.1155.5, 00000000004.563.5, 00000000007.949.5, 00000000005.589.5] 2022-09-12T20:15:47.696 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.648.5, 00000000002.837.5, 00000000002.643.5, 00000000005.359.5, 00000000004.935.5, 00000000001.434.5, 00000000001.1155.5, 00000000004.563.5, 00000000007.949.5, 00000000006.591.5] 2022-09-12T20:15:52.697 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:15:52.697+0000 7fe4f6ee4600 1 waiting for incremental sync to catch up: 2022-09-12T20:15:52.698 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.648.5, 00000000002.837.5, 00000000002.643.5, 00000000005.359.5, 00000000004.935.5, 00000000001.434.5, 00000000001.1155.5, 00000000004.563.5, 00000000007.949.5, 00000000005.589.5] 2022-09-12T20:15:52.699 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.648.5, 00000000002.837.5, 00000000002.643.5, 00000000005.359.5, 00000000004.935.5, 00000000001.434.5, 00000000001.1155.5, 00000000004.563.5, 00000000007.949.5, 00000000006.591.5] 2022-09-12T20:15:57.698 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:15:57.697+0000 7fe4f6ee4600 1 bucket sync caught up with source: 2022-09-12T20:15:57.699 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.648.5, 00000000002.837.5, 00000000002.643.5, 00000000005.359.5, 00000000004.935.5, 00000000001.434.5, 00000000001.1155.5, 00000000004.563.5, 00000000007.949.5, 00000000006.591.5] 2022-09-12T20:15:57.699 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.648.5, 00000000002.837.5, 00000000002.643.5, 00000000005.359.5, 00000000004.935.5, 00000000001.434.5, 00000000001.1155.5, 00000000004.563.5, 00000000007.949.5, 00000000006.591.5] 2022-09-12T20:15:57.700 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:15:57.697+0000 7fe4f6ee4600 0 bucket checkpoint complete 2022-09-12T20:15:57.719 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-75 2022-09-12T20:15:57.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:15:57.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:15:57.723 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:15:57.787 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:15:57.787 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:15:57.841 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:15:57.840+0000 7f8f57191600 1 bucket sync caught up with source: 2022-09-12T20:15:57.842 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.648.5, 00000000002.837.5, 00000000002.643.5, 00000000005.359.5, 00000000004.935.5, 00000000001.434.5, 00000000001.1155.5, 00000000004.563.5, 00000000007.949.5, 00000000006.591.5] 2022-09-12T20:15:57.842 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.648.5, 00000000002.837.5, 00000000002.643.5, 00000000005.359.5, 00000000004.935.5, 00000000001.434.5, 00000000001.1155.5, 00000000004.563.5, 00000000007.949.5, 00000000006.591.5] 2022-09-12T20:15:57.842 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:15:57.840+0000 7f8f57191600 0 bucket checkpoint complete 2022-09-12T20:15:57.862 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-75 2022-09-12T20:15:57.863 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-75 2022-09-12T20:15:57.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:15:57.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:15:57.866 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:15:57.925 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:15:57.925 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:15:57.982 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:15:57.981+0000 7fc0153ea600 1 bucket sync caught up with source: 2022-09-12T20:15:57.983 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, 00000000003.304.5, 00000000002.337.5, 00000000002.287.5, 00000000005.356.5, 00000000004.551.5, 00000000001.338.5, 00000000001.466.5, 00000000004.332.5, 00000000007.563.5, 00000000006.504.5] 2022-09-12T20:15:57.983 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, 00000000003.304.5, 00000000002.337.5, 00000000002.287.5, 00000000005.356.5, 00000000004.551.5, 00000000001.338.5, 00000000001.466.5, 00000000004.332.5, 00000000007.563.5, 00000000006.504.5] 2022-09-12T20:15:57.983 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:15:57.981+0000 7fc0153ea600 0 bucket checkpoint complete 2022-09-12T20:15:57.997 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-75 2022-09-12T20:15:57.998 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:15:57.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:15:58.000 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-75 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T20:15:58.063 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:15:58.063 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:15:58.107 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:15:58.105+0000 7f0f4c4d2600 1 bucket sync caught up with source: 2022-09-12T20:15:58.108 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, 00000000003.304.5, 00000000002.337.5, 00000000002.287.5, 00000000005.356.5, 00000000004.551.5, 00000000001.338.5, 00000000001.466.5, 00000000004.332.5, 00000000007.563.5, 00000000006.504.5] 2022-09-12T20:15:58.108 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, 00000000003.304.5, 00000000002.337.5, 00000000002.287.5, 00000000005.356.5, 00000000004.551.5, 00000000001.338.5, 00000000001.466.5, 00000000004.332.5, 00000000007.563.5, 00000000006.504.5] 2022-09-12T20:15:58.108 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:15:58.105+0000 7f0f4c4d2600 0 bucket checkpoint complete 2022-09-12T20:15:58.122 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-75 2022-09-12T20:15:58.123 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-75 zones={test-zone2, test-zone1} 2022-09-12T20:15:58.124 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:58.124 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:58.125 DEBUG:boto:Method: HEAD 2022-09-12T20:15:58.125 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:15:58.126 DEBUG:boto:Data: 2022-09-12T20:15:58.126 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.127 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.127 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.128 DEBUG:boto:Params: {} 2022-09-12T20:15:58.129 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T20:15:58.129 DEBUG:boto:Token: None 2022-09-12T20:15:58.130 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/ 2022-09-12T20:15:58.131 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HNE+75YnnzHI2gO4beQkq1MUyuQ= 2022-09-12T20:15:58.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HNE+75YnnzHI2gO4beQkq1MUyuQ=', 'Content-Length': '0'} 2022-09-12T20:15:58.136 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008221ccc0698dc085-00631f937e-189d-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.137 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:58.138 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:58.138 DEBUG:boto:Method: HEAD 2022-09-12T20:15:58.139 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:15:58.139 DEBUG:boto:Data: 2022-09-12T20:15:58.140 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.141 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.141 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.142 DEBUG:boto:Params: {} 2022-09-12T20:15:58.142 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T20:15:58.143 DEBUG:boto:Token: None 2022-09-12T20:15:58.144 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/ 2022-09-12T20:15:58.144 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HNE+75YnnzHI2gO4beQkq1MUyuQ= 2022-09-12T20:15:58.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HNE+75YnnzHI2gO4beQkq1MUyuQ=', 'Content-Length': '0'} 2022-09-12T20:15:58.150 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031e7e505173a78ee-00631f937e-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.150 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T20:15:58.151 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:58.152 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:58.152 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:15:58.153 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:15:58.153 DEBUG:boto:Method: GET 2022-09-12T20:15:58.154 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:15:58.155 DEBUG:boto:Data: 2022-09-12T20:15:58.155 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.156 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.156 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.157 DEBUG:boto:Params: {} 2022-09-12T20:15:58.157 DEBUG:boto:Token: None 2022-09-12T20:15:58.158 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/?versions 2022-09-12T20:15:58.159 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZDZPpXx4WExtHIt5SqDOsChuXzk= 2022-09-12T20:15:58.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZDZPpXx4WExtHIt5SqDOsChuXzk=', 'Content-Length': '0'} 2022-09-12T20:15:58.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013988c26b9d0ce06-00631f937e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.164 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2022-09-12T20:14:27.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2022-09-12T20:14:27.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2022-09-12T20:14:26.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2022-09-12T20:14:27.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:58.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:15:58.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2022-09-12T20:15:58.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-12T20:15:58.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-12T20:15:58.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2022-09-12T20:15:58.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-12T20:15:58.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2022-09-12T20:15:58.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2022-09-12T20:15:58.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:15:58.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2022-09-12T20:15:58.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:15:58.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:15:58.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-12T20:15:58.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-12T20:15:58.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2022-09-12T20:15:58.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:15:58.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:15:58.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2022-09-12T20:15:58.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:15:58.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:15:58.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-12T20:15:58.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:15:58.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:15:58.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:15:58.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-12T20:15:58.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2022-09-12T20:15:58.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:15:58.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-12T20:15:58.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2022-09-12T20:15:58.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:15:58.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:15:58.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-12T20:15:58.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-12T20:15:58.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2022-09-12T20:15:58.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:15:58.184 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T20:15:58.185 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:58.185 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:58.186 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:15:58.186 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:15:58.187 DEBUG:boto:Method: GET 2022-09-12T20:15:58.187 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:15:58.188 DEBUG:boto:Data: 2022-09-12T20:15:58.189 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.189 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.190 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.190 DEBUG:boto:Params: {} 2022-09-12T20:15:58.191 DEBUG:boto:Token: None 2022-09-12T20:15:58.192 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/?versions 2022-09-12T20:15:58.192 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZDZPpXx4WExtHIt5SqDOsChuXzk= 2022-09-12T20:15:58.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZDZPpXx4WExtHIt5SqDOsChuXzk=', 'Content-Length': '0'} 2022-09-12T20:15:58.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce1ddb6edbee9eff-00631f937e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.198 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2022-09-12T20:14:27.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2022-09-12T20:14:27.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2022-09-12T20:14:26.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2022-09-12T20:14:27.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:58.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:15:58.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2022-09-12T20:15:58.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-12T20:15:58.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-12T20:15:58.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2022-09-12T20:15:58.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-12T20:15:58.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2022-09-12T20:15:58.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2022-09-12T20:15:58.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:15:58.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2022-09-12T20:15:58.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:15:58.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:15:58.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-12T20:15:58.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-12T20:15:58.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2022-09-12T20:15:58.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:15:58.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:15:58.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2022-09-12T20:15:58.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:15:58.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:15:58.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-12T20:15:58.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:15:58.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:15:58.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:15:58.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-12T20:15:58.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2022-09-12T20:15:58.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:15:58.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-12T20:15:58.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2022-09-12T20:15:58.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:15:58.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:15:58.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-12T20:15:58.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-12T20:15:58.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2022-09-12T20:15:58.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:15:58.217 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:58.217 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:58.218 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:15:58.219 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:15:58.219 DEBUG:boto:Method: GET 2022-09-12T20:15:58.220 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:15:58.220 DEBUG:boto:Data: 2022-09-12T20:15:58.221 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.221 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.222 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.222 DEBUG:boto:Params: {} 2022-09-12T20:15:58.223 DEBUG:boto:Token: None 2022-09-12T20:15:58.224 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/?versions 2022-09-12T20:15:58.224 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZDZPpXx4WExtHIt5SqDOsChuXzk= 2022-09-12T20:15:58.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZDZPpXx4WExtHIt5SqDOsChuXzk=', 'Content-Length': '0'} 2022-09-12T20:15:58.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e005e6f646e85d08-00631f937e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.228 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2022-09-12T20:14:27.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2022-09-12T20:14:27.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2022-09-12T20:14:26.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2022-09-12T20:14:27.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:58.231 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:58.231 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:58.232 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:15:58.232 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:15:58.233 DEBUG:boto:Method: GET 2022-09-12T20:15:58.233 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:15:58.234 DEBUG:boto:Data: 2022-09-12T20:15:58.234 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.235 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.235 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.236 DEBUG:boto:Params: {} 2022-09-12T20:15:58.236 DEBUG:boto:Token: None 2022-09-12T20:15:58.237 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/?versions 2022-09-12T20:15:58.238 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZDZPpXx4WExtHIt5SqDOsChuXzk= 2022-09-12T20:15:58.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZDZPpXx4WExtHIt5SqDOsChuXzk=', 'Content-Length': '0'} 2022-09-12T20:15:58.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7c76559a88323b7-00631f937e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.243 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2022-09-12T20:14:27.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2022-09-12T20:14:27.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2022-09-12T20:14:26.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2022-09-12T20:14:27.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:58.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:15:58.246 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:15:58.247 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:15:58.247 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:15:58.248 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:15:58.248 DEBUG:boto:Method: GET 2022-09-12T20:15:58.249 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:15:58.249 DEBUG:boto:Data: 2022-09-12T20:15:58.250 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.250 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.251 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.252 DEBUG:boto:Params: {} 2022-09-12T20:15:58.252 DEBUG:boto:Token: None 2022-09-12T20:15:58.253 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:15:58.254 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SFBzv7KC9R1m+TgvcsVQWdIGr7E= 2022-09-12T20:15:58.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SFBzv7KC9R1m+TgvcsVQWdIGr7E=', 'Content-Length': '0'} 2022-09-12T20:15:58.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000365f308aed5be43f-00631f937e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.257 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:15:58.258 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:15:58.259 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:15:58.259 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:15:58.260 DEBUG:boto:Method: GET 2022-09-12T20:15:58.260 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:15:58.261 DEBUG:boto:Data: 2022-09-12T20:15:58.262 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.262 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.263 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.263 DEBUG:boto:Params: {} 2022-09-12T20:15:58.264 DEBUG:boto:Token: None 2022-09-12T20:15:58.264 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:15:58.265 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SFBzv7KC9R1m+TgvcsVQWdIGr7E= 2022-09-12T20:15:58.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SFBzv7KC9R1m+TgvcsVQWdIGr7E=', 'Content-Length': '0'} 2022-09-12T20:15:58.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b358c38a43bbefa-00631f937e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.269 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:15:58.269 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:15:58.270 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:15:58.270 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:15:58.271 DEBUG:boto:Method: HEAD 2022-09-12T20:15:58.271 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:15:58.272 DEBUG:boto:Data: 2022-09-12T20:15:58.272 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.273 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.273 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.274 DEBUG:boto:Params: {} 2022-09-12T20:15:58.274 DEBUG:boto:Token: None 2022-09-12T20:15:58.275 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:15:58.275 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IkIYi4M+SwofH+57/JcYYRAxO6s= 2022-09-12T20:15:58.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IkIYi4M+SwofH+57/JcYYRAxO6s=', 'Content-Length': '0'} 2022-09-12T20:15:58.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049ddda7e60f1366e-00631f937e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.279 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:15:58.279 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:15:58.280 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:15:58.280 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:15:58.281 DEBUG:boto:Method: HEAD 2022-09-12T20:15:58.281 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:15:58.282 DEBUG:boto:Data: 2022-09-12T20:15:58.282 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.283 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.283 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.284 DEBUG:boto:Params: {} 2022-09-12T20:15:58.284 DEBUG:boto:Token: None 2022-09-12T20:15:58.285 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:15:58.286 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IkIYi4M+SwofH+57/JcYYRAxO6s= 2022-09-12T20:15:58.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IkIYi4M+SwofH+57/JcYYRAxO6s=', 'Content-Length': '0'} 2022-09-12T20:15:58.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc18056129b540cf-00631f937e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:15:58.290 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:15:58.291 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:15:58.291 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:15:58.292 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:15:58.292 DEBUG:boto:Method: GET 2022-09-12T20:15:58.293 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:15:58.293 DEBUG:boto:Data: 2022-09-12T20:15:58.294 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.294 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.295 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.295 DEBUG:boto:Params: {} 2022-09-12T20:15:58.296 DEBUG:boto:Token: None 2022-09-12T20:15:58.297 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:15:58.297 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SFBzv7KC9R1m+TgvcsVQWdIGr7E= 2022-09-12T20:15:58.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SFBzv7KC9R1m+TgvcsVQWdIGr7E=', 'Content-Length': '0'} 2022-09-12T20:15:58.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049198aa13353c71d-00631f937e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.300 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:15:58.301 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:15:58.301 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:15:58.302 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:15:58.303 DEBUG:boto:Method: GET 2022-09-12T20:15:58.303 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:15:58.304 DEBUG:boto:Data: 2022-09-12T20:15:58.304 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.305 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.305 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.306 DEBUG:boto:Params: {} 2022-09-12T20:15:58.306 DEBUG:boto:Token: None 2022-09-12T20:15:58.307 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:15:58.307 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SFBzv7KC9R1m+TgvcsVQWdIGr7E= 2022-09-12T20:15:58.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SFBzv7KC9R1m+TgvcsVQWdIGr7E=', 'Content-Length': '0'} 2022-09-12T20:15:58.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e84f2b4f2a0fb242-00631f937e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.311 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:58.312 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:58.312 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:15:58.313 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:15:58.313 DEBUG:boto:Method: GET 2022-09-12T20:15:58.314 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:15:58.314 DEBUG:boto:Data: 2022-09-12T20:15:58.315 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.315 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.316 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.316 DEBUG:boto:Params: {} 2022-09-12T20:15:58.317 DEBUG:boto:Token: None 2022-09-12T20:15:58.318 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/ 2022-09-12T20:15:58.318 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4dHe/m1FIw7ChRaNAGgtLkVCDco= 2022-09-12T20:15:58.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4dHe/m1FIw7ChRaNAGgtLkVCDco=', 'Content-Length': '0'} 2022-09-12T20:15:58.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe9d1967fd6d675d-00631f937e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.322 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:58.323 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:58.323 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:58.324 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:15:58.324 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:15:58.325 DEBUG:boto:Method: GET 2022-09-12T20:15:58.325 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:15:58.326 DEBUG:boto:Data: 2022-09-12T20:15:58.327 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.327 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.328 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.328 DEBUG:boto:Params: {} 2022-09-12T20:15:58.329 DEBUG:boto:Token: None 2022-09-12T20:15:58.329 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/ 2022-09-12T20:15:58.330 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4dHe/m1FIw7ChRaNAGgtLkVCDco= 2022-09-12T20:15:58.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4dHe/m1FIw7ChRaNAGgtLkVCDco=', 'Content-Length': '0'} 2022-09-12T20:15:58.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8519089248d4090-00631f937e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.336 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:58.336 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:15:58.337 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:15:58.338 DEBUG:boto:Method: HEAD 2022-09-12T20:15:58.338 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:15:58.339 DEBUG:boto:Data: 2022-09-12T20:15:58.339 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.340 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.341 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.341 DEBUG:boto:Params: {} 2022-09-12T20:15:58.342 DEBUG:boto:Token: None 2022-09-12T20:15:58.342 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj11 2022-09-12T20:15:58.343 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PqqYbVhZlcz6FHMx/4FoRZlazdY= 2022-09-12T20:15:58.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PqqYbVhZlcz6FHMx/4FoRZlazdY=', 'Content-Length': '0'} 2022-09-12T20:15:58.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091e2fbb5526d0426-00631f937e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.346 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:15:58.347 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:15:58.347 DEBUG:boto:Method: HEAD 2022-09-12T20:15:58.348 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:15:58.348 DEBUG:boto:Data: 2022-09-12T20:15:58.349 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.350 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.351 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.352 DEBUG:boto:Params: {} 2022-09-12T20:15:58.353 DEBUG:boto:Token: None 2022-09-12T20:15:58.354 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj11 2022-09-12T20:15:58.355 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PqqYbVhZlcz6FHMx/4FoRZlazdY= 2022-09-12T20:15:58.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PqqYbVhZlcz6FHMx/4FoRZlazdY=', 'Content-Length': '0'} 2022-09-12T20:15:58.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e5b307b302d000d-00631f937e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:15:58.359 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:15:58.360 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:15:58.360 DEBUG:boto:Method: GET 2022-09-12T20:15:58.361 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:15:58.361 DEBUG:boto:Data: 2022-09-12T20:15:58.362 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.363 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.363 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.364 DEBUG:boto:Params: {} 2022-09-12T20:15:58.364 DEBUG:boto:Token: None 2022-09-12T20:15:58.365 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj11 2022-09-12T20:15:58.366 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RrpqYCzQlE2HDbl1XltSt+nmc8s= 2022-09-12T20:15:58.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RrpqYCzQlE2HDbl1XltSt+nmc8s=', 'Content-Length': '0'} 2022-09-12T20:15:58.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b414a2fa9d60d6c-00631f937e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.369 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:15:58.370 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:15:58.370 DEBUG:boto:Method: GET 2022-09-12T20:15:58.371 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:15:58.371 DEBUG:boto:Data: 2022-09-12T20:15:58.372 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.372 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.373 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.373 DEBUG:boto:Params: {} 2022-09-12T20:15:58.374 DEBUG:boto:Token: None 2022-09-12T20:15:58.374 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj11 2022-09-12T20:15:58.375 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RrpqYCzQlE2HDbl1XltSt+nmc8s= 2022-09-12T20:15:58.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RrpqYCzQlE2HDbl1XltSt+nmc8s=', 'Content-Length': '0'} 2022-09-12T20:15:58.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000020c2511ce7aef4e-00631f937e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.418 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:58.419 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:58.420 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:15:58.421 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:15:58.422 DEBUG:boto:Method: GET 2022-09-12T20:15:58.423 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:15:58.424 DEBUG:boto:Data: 2022-09-12T20:15:58.425 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.425 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.426 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.427 DEBUG:boto:Params: {} 2022-09-12T20:15:58.428 DEBUG:boto:Token: None 2022-09-12T20:15:58.429 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/ 2022-09-12T20:15:58.430 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4dHe/m1FIw7ChRaNAGgtLkVCDco= 2022-09-12T20:15:58.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4dHe/m1FIw7ChRaNAGgtLkVCDco=', 'Content-Length': '0'} 2022-09-12T20:15:58.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000555f95a317350848-00631f937e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.435 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:58.437 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:58.438 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:58.439 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:15:58.440 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:15:58.441 DEBUG:boto:Method: GET 2022-09-12T20:15:58.441 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:15:58.442 DEBUG:boto:Data: 2022-09-12T20:15:58.442 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.443 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.443 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.444 DEBUG:boto:Params: {} 2022-09-12T20:15:58.444 DEBUG:boto:Token: None 2022-09-12T20:15:58.445 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/ 2022-09-12T20:15:58.445 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4dHe/m1FIw7ChRaNAGgtLkVCDco= 2022-09-12T20:15:58.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4dHe/m1FIw7ChRaNAGgtLkVCDco=', 'Content-Length': '0'} 2022-09-12T20:15:58.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9fc120ca01c58a9-00631f937e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.451 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:58.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2022-09-12T20:15:58.453 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:15:58.453 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:15:58.454 DEBUG:boto:path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:15:58.454 DEBUG:boto:auth_path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:15:58.455 DEBUG:boto:Method: GET 2022-09-12T20:15:58.455 DEBUG:boto:Path: /jdpiak-75/obj111?versionId=null 2022-09-12T20:15:58.456 DEBUG:boto:Data: 2022-09-12T20:15:58.456 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.457 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.458 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.458 DEBUG:boto:Params: {} 2022-09-12T20:15:58.459 DEBUG:boto:Token: None 2022-09-12T20:15:58.459 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj111?versionId=null 2022-09-12T20:15:58.460 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oA/dG8pX77OEpsprjo52bgBflzw= 2022-09-12T20:15:58.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oA/dG8pX77OEpsprjo52bgBflzw=', 'Content-Length': '0'} 2022-09-12T20:15:58.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e828ab5deb766371-00631f937e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.506 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:15:58.507 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:15:58.507 DEBUG:boto:path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:15:58.508 DEBUG:boto:auth_path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:15:58.508 DEBUG:boto:Method: GET 2022-09-12T20:15:58.509 DEBUG:boto:Path: /jdpiak-75/obj111?versionId=null 2022-09-12T20:15:58.509 DEBUG:boto:Data: 2022-09-12T20:15:58.510 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.510 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.511 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.511 DEBUG:boto:Params: {} 2022-09-12T20:15:58.512 DEBUG:boto:Token: None 2022-09-12T20:15:58.513 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj111?versionId=null 2022-09-12T20:15:58.513 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oA/dG8pX77OEpsprjo52bgBflzw= 2022-09-12T20:15:58.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oA/dG8pX77OEpsprjo52bgBflzw=', 'Content-Length': '0'} 2022-09-12T20:15:58.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057c90b4f40e8541f-00631f937e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.517 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:15:58.517 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:15:58.518 DEBUG:boto:path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:15:58.518 DEBUG:boto:auth_path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:15:58.519 DEBUG:boto:Method: HEAD 2022-09-12T20:15:58.519 DEBUG:boto:Path: /jdpiak-75/obj111?versionId=null 2022-09-12T20:15:58.520 DEBUG:boto:Data: 2022-09-12T20:15:58.520 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.521 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.521 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.522 DEBUG:boto:Params: {} 2022-09-12T20:15:58.522 DEBUG:boto:Token: None 2022-09-12T20:15:58.523 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj111?versionId=null 2022-09-12T20:15:58.524 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TwLo5U6M9QobRCsOx4jizMcaGLo= 2022-09-12T20:15:58.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TwLo5U6M9QobRCsOx4jizMcaGLo=', 'Content-Length': '0'} 2022-09-12T20:15:58.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f0f29bd5cbebcd8-00631f937e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.527 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:15:58.527 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:15:58.528 DEBUG:boto:path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:15:58.528 DEBUG:boto:auth_path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:15:58.529 DEBUG:boto:Method: HEAD 2022-09-12T20:15:58.529 DEBUG:boto:Path: /jdpiak-75/obj111?versionId=null 2022-09-12T20:15:58.530 DEBUG:boto:Data: 2022-09-12T20:15:58.530 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.531 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.531 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.532 DEBUG:boto:Params: {} 2022-09-12T20:15:58.532 DEBUG:boto:Token: None 2022-09-12T20:15:58.533 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj111?versionId=null 2022-09-12T20:15:58.534 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TwLo5U6M9QobRCsOx4jizMcaGLo= 2022-09-12T20:15:58.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TwLo5U6M9QobRCsOx4jizMcaGLo=', 'Content-Length': '0'} 2022-09-12T20:15:58.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f51d2fde49f27f60-00631f937e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2022-09-12T20:15:58.537 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:15:58.538 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:15:58.539 DEBUG:boto:path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:15:58.539 DEBUG:boto:auth_path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:15:58.540 DEBUG:boto:Method: GET 2022-09-12T20:15:58.540 DEBUG:boto:Path: /jdpiak-75/obj111?versionId=null 2022-09-12T20:15:58.541 DEBUG:boto:Data: 2022-09-12T20:15:58.541 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.542 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.542 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.543 DEBUG:boto:Params: {} 2022-09-12T20:15:58.543 DEBUG:boto:Token: None 2022-09-12T20:15:58.544 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj111?versionId=null 2022-09-12T20:15:58.545 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oA/dG8pX77OEpsprjo52bgBflzw= 2022-09-12T20:15:58.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oA/dG8pX77OEpsprjo52bgBflzw=', 'Content-Length': '0'} 2022-09-12T20:15:58.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaa3ce0925cbec6a-00631f937e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.548 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:15:58.549 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:15:58.550 DEBUG:boto:path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:15:58.551 DEBUG:boto:auth_path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:15:58.552 DEBUG:boto:Method: GET 2022-09-12T20:15:58.553 DEBUG:boto:Path: /jdpiak-75/obj111?versionId=null 2022-09-12T20:15:58.554 DEBUG:boto:Data: 2022-09-12T20:15:58.555 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.556 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.557 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.558 DEBUG:boto:Params: {} 2022-09-12T20:15:58.558 DEBUG:boto:Token: None 2022-09-12T20:15:58.559 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj111?versionId=null 2022-09-12T20:15:58.560 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oA/dG8pX77OEpsprjo52bgBflzw= 2022-09-12T20:15:58.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oA/dG8pX77OEpsprjo52bgBflzw=', 'Content-Length': '0'} 2022-09-12T20:15:58.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082e0ceec5ee0e601-00631f937e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.563 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:58.563 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:58.564 DEBUG:boto:path=/jdpiak-75/?prefix=obj111 2022-09-12T20:15:58.565 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj111 2022-09-12T20:15:58.565 DEBUG:boto:Method: GET 2022-09-12T20:15:58.566 DEBUG:boto:Path: /jdpiak-75/?prefix=obj111 2022-09-12T20:15:58.566 DEBUG:boto:Data: 2022-09-12T20:15:58.567 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.567 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.568 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.568 DEBUG:boto:Params: {} 2022-09-12T20:15:58.569 DEBUG:boto:Token: None 2022-09-12T20:15:58.569 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/ 2022-09-12T20:15:58.570 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4dHe/m1FIw7ChRaNAGgtLkVCDco= 2022-09-12T20:15:58.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4dHe/m1FIw7ChRaNAGgtLkVCDco=', 'Content-Length': '0'} 2022-09-12T20:15:58.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cb742a19414951f-00631f937e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.574 DEBUG:boto:b'jdpiak-75obj1111000falseobj1112022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:58.575 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:58.576 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:58.576 DEBUG:boto:path=/jdpiak-75/?prefix=obj111 2022-09-12T20:15:58.577 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj111 2022-09-12T20:15:58.577 DEBUG:boto:Method: GET 2022-09-12T20:15:58.578 DEBUG:boto:Path: /jdpiak-75/?prefix=obj111 2022-09-12T20:15:58.578 DEBUG:boto:Data: 2022-09-12T20:15:58.579 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.579 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.580 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.580 DEBUG:boto:Params: {} 2022-09-12T20:15:58.581 DEBUG:boto:Token: None 2022-09-12T20:15:58.582 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/ 2022-09-12T20:15:58.582 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4dHe/m1FIw7ChRaNAGgtLkVCDco= 2022-09-12T20:15:58.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4dHe/m1FIw7ChRaNAGgtLkVCDco=', 'Content-Length': '0'} 2022-09-12T20:15:58.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3ae0fd7626f54d2-00631f937e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.588 DEBUG:boto:b'jdpiak-75obj1111000falseobj1112022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:58.588 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:15:58.589 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:15:58.589 DEBUG:boto:Method: HEAD 2022-09-12T20:15:58.590 DEBUG:boto:Path: /jdpiak-75/obj111 2022-09-12T20:15:58.591 DEBUG:boto:Data: 2022-09-12T20:15:58.591 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.592 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.592 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.593 DEBUG:boto:Params: {} 2022-09-12T20:15:58.593 DEBUG:boto:Token: None 2022-09-12T20:15:58.594 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj111 2022-09-12T20:15:58.594 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Uu3R/m/DNLBWVIEGU0ds+eD0DxU= 2022-09-12T20:15:58.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Uu3R/m/DNLBWVIEGU0ds+eD0DxU=', 'Content-Length': '0'} 2022-09-12T20:15:58.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b201dbfa681d89b0-00631f937e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.597 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:15:58.598 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:15:58.598 DEBUG:boto:Method: HEAD 2022-09-12T20:15:58.599 DEBUG:boto:Path: /jdpiak-75/obj111 2022-09-12T20:15:58.599 DEBUG:boto:Data: 2022-09-12T20:15:58.600 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.600 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.601 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.601 DEBUG:boto:Params: {} 2022-09-12T20:15:58.602 DEBUG:boto:Token: None 2022-09-12T20:15:58.602 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj111 2022-09-12T20:15:58.603 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Uu3R/m/DNLBWVIEGU0ds+eD0DxU= 2022-09-12T20:15:58.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Uu3R/m/DNLBWVIEGU0ds+eD0DxU=', 'Content-Length': '0'} 2022-09-12T20:15:58.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8c34ab1f7b00bdf-00631f937e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2022-09-12T20:15:58.607 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:15:58.607 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:15:58.608 DEBUG:boto:Method: GET 2022-09-12T20:15:58.608 DEBUG:boto:Path: /jdpiak-75/obj111 2022-09-12T20:15:58.609 DEBUG:boto:Data: 2022-09-12T20:15:58.609 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.610 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.610 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.611 DEBUG:boto:Params: {} 2022-09-12T20:15:58.611 DEBUG:boto:Token: None 2022-09-12T20:15:58.612 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj111 2022-09-12T20:15:58.613 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rCU4iWe0O9WkcS3gqWxhZQogwHg= 2022-09-12T20:15:58.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rCU4iWe0O9WkcS3gqWxhZQogwHg=', 'Content-Length': '0'} 2022-09-12T20:15:58.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000549ca1d91c1a311c-00631f937e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.616 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:15:58.616 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:15:58.617 DEBUG:boto:Method: GET 2022-09-12T20:15:58.618 DEBUG:boto:Path: /jdpiak-75/obj111 2022-09-12T20:15:58.629 DEBUG:boto:Data: 2022-09-12T20:15:58.630 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.631 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.631 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.632 DEBUG:boto:Params: {} 2022-09-12T20:15:58.632 DEBUG:boto:Token: None 2022-09-12T20:15:58.633 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj111 2022-09-12T20:15:58.634 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rCU4iWe0O9WkcS3gqWxhZQogwHg= 2022-09-12T20:15:58.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rCU4iWe0O9WkcS3gqWxhZQogwHg=', 'Content-Length': '0'} 2022-09-12T20:15:58.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdcbde9c4fdbd252-00631f937e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.678 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:58.679 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:58.679 DEBUG:boto:path=/jdpiak-75/?prefix=obj111 2022-09-12T20:15:58.680 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj111 2022-09-12T20:15:58.680 DEBUG:boto:Method: GET 2022-09-12T20:15:58.681 DEBUG:boto:Path: /jdpiak-75/?prefix=obj111 2022-09-12T20:15:58.681 DEBUG:boto:Data: 2022-09-12T20:15:58.682 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.682 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.683 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.683 DEBUG:boto:Params: {} 2022-09-12T20:15:58.684 DEBUG:boto:Token: None 2022-09-12T20:15:58.685 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/ 2022-09-12T20:15:58.685 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4dHe/m1FIw7ChRaNAGgtLkVCDco= 2022-09-12T20:15:58.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4dHe/m1FIw7ChRaNAGgtLkVCDco=', 'Content-Length': '0'} 2022-09-12T20:15:58.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094030c40d8293127-00631f937e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.689 DEBUG:boto:b'jdpiak-75obj1111000falseobj1112022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:58.690 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:58.690 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:58.691 DEBUG:boto:path=/jdpiak-75/?prefix=obj111 2022-09-12T20:15:58.691 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj111 2022-09-12T20:15:58.692 DEBUG:boto:Method: GET 2022-09-12T20:15:58.692 DEBUG:boto:Path: /jdpiak-75/?prefix=obj111 2022-09-12T20:15:58.693 DEBUG:boto:Data: 2022-09-12T20:15:58.693 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.694 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.695 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.695 DEBUG:boto:Params: {} 2022-09-12T20:15:58.696 DEBUG:boto:Token: None 2022-09-12T20:15:58.696 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/ 2022-09-12T20:15:58.697 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4dHe/m1FIw7ChRaNAGgtLkVCDco= 2022-09-12T20:15:58.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4dHe/m1FIw7ChRaNAGgtLkVCDco=', 'Content-Length': '0'} 2022-09-12T20:15:58.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e1650901daea260-00631f937e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.702 DEBUG:boto:b'jdpiak-75obj1111000falseobj1112022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:58.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:15:58.704 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:15:58.704 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:15:58.705 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:15:58.705 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:15:58.706 DEBUG:boto:Method: GET 2022-09-12T20:15:58.706 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:15:58.707 DEBUG:boto:Data: 2022-09-12T20:15:58.707 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.708 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.708 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.709 DEBUG:boto:Params: {} 2022-09-12T20:15:58.710 DEBUG:boto:Token: None 2022-09-12T20:15:58.710 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:15:58.711 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dkoWQK2ifrD70oizjE3BNl3V7YQ= 2022-09-12T20:15:58.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dkoWQK2ifrD70oizjE3BNl3V7YQ=', 'Content-Length': '0'} 2022-09-12T20:15:58.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084aaf2edd3bc14c6-00631f937e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.714 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:15:58.715 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:15:58.715 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:15:58.716 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:15:58.716 DEBUG:boto:Method: GET 2022-09-12T20:15:58.717 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:15:58.717 DEBUG:boto:Data: 2022-09-12T20:15:58.718 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.719 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.719 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.720 DEBUG:boto:Params: {} 2022-09-12T20:15:58.720 DEBUG:boto:Token: None 2022-09-12T20:15:58.721 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:15:58.722 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dkoWQK2ifrD70oizjE3BNl3V7YQ= 2022-09-12T20:15:58.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dkoWQK2ifrD70oizjE3BNl3V7YQ=', 'Content-Length': '0'} 2022-09-12T20:15:58.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a340eaa4e8d8b5b1-00631f937e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.725 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:15:58.725 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:15:58.726 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:15:58.726 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:15:58.727 DEBUG:boto:Method: HEAD 2022-09-12T20:15:58.727 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:15:58.728 DEBUG:boto:Data: 2022-09-12T20:15:58.728 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.729 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.729 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.730 DEBUG:boto:Params: {} 2022-09-12T20:15:58.730 DEBUG:boto:Token: None 2022-09-12T20:15:58.731 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:15:58.731 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WPzcxXvRzgGpRCz8GPSpiX1ufQQ= 2022-09-12T20:15:58.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WPzcxXvRzgGpRCz8GPSpiX1ufQQ=', 'Content-Length': '0'} 2022-09-12T20:15:58.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da9a7c2dbe36e004-00631f937e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.734 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:15:58.735 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:15:58.735 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:15:58.736 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:15:58.736 DEBUG:boto:Method: HEAD 2022-09-12T20:15:58.737 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:15:58.737 DEBUG:boto:Data: 2022-09-12T20:15:58.738 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.738 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.739 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.739 DEBUG:boto:Params: {} 2022-09-12T20:15:58.740 DEBUG:boto:Token: None 2022-09-12T20:15:58.740 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:15:58.741 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WPzcxXvRzgGpRCz8GPSpiX1ufQQ= 2022-09-12T20:15:58.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WPzcxXvRzgGpRCz8GPSpiX1ufQQ=', 'Content-Length': '0'} 2022-09-12T20:15:58.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a7bf3f8b5ad5e34-00631f937e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:15:58.745 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:15:58.745 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:15:58.746 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:15:58.746 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:15:58.747 DEBUG:boto:Method: GET 2022-09-12T20:15:58.747 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:15:58.748 DEBUG:boto:Data: 2022-09-12T20:15:58.748 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.749 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.749 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.750 DEBUG:boto:Params: {} 2022-09-12T20:15:58.750 DEBUG:boto:Token: None 2022-09-12T20:15:58.751 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:15:58.752 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dkoWQK2ifrD70oizjE3BNl3V7YQ= 2022-09-12T20:15:58.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dkoWQK2ifrD70oizjE3BNl3V7YQ=', 'Content-Length': '0'} 2022-09-12T20:15:58.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7f8d344020cae25-00631f937e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.755 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:15:58.756 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:15:58.756 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:15:58.757 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:15:58.757 DEBUG:boto:Method: GET 2022-09-12T20:15:58.758 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:15:58.758 DEBUG:boto:Data: 2022-09-12T20:15:58.759 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.759 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.760 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.760 DEBUG:boto:Params: {} 2022-09-12T20:15:58.761 DEBUG:boto:Token: None 2022-09-12T20:15:58.762 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:15:58.763 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dkoWQK2ifrD70oizjE3BNl3V7YQ= 2022-09-12T20:15:58.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dkoWQK2ifrD70oizjE3BNl3V7YQ=', 'Content-Length': '0'} 2022-09-12T20:15:58.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068757e1cd180bd20-00631f937e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.767 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:58.768 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:58.768 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:15:58.769 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:15:58.769 DEBUG:boto:Method: GET 2022-09-12T20:15:58.770 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:15:58.770 DEBUG:boto:Data: 2022-09-12T20:15:58.771 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.771 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.772 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.773 DEBUG:boto:Params: {} 2022-09-12T20:15:58.773 DEBUG:boto:Token: None 2022-09-12T20:15:58.774 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/ 2022-09-12T20:15:58.774 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4dHe/m1FIw7ChRaNAGgtLkVCDco= 2022-09-12T20:15:58.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4dHe/m1FIw7ChRaNAGgtLkVCDco=', 'Content-Length': '0'} 2022-09-12T20:15:58.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067adece87883274c-00631f937e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.822 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:58.823 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:58.824 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:58.824 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:15:58.825 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:15:58.825 DEBUG:boto:Method: GET 2022-09-12T20:15:58.826 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:15:58.826 DEBUG:boto:Data: 2022-09-12T20:15:58.827 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.827 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.828 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.829 DEBUG:boto:Params: {} 2022-09-12T20:15:58.829 DEBUG:boto:Token: None 2022-09-12T20:15:58.830 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/ 2022-09-12T20:15:58.830 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4dHe/m1FIw7ChRaNAGgtLkVCDco= 2022-09-12T20:15:58.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4dHe/m1FIw7ChRaNAGgtLkVCDco=', 'Content-Length': '0'} 2022-09-12T20:15:58.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a2e6d67b367e282-00631f937e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.836 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:58.837 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:15:58.838 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:15:58.839 DEBUG:boto:Method: HEAD 2022-09-12T20:15:58.839 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:15:58.840 DEBUG:boto:Data: 2022-09-12T20:15:58.840 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.841 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.841 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.842 DEBUG:boto:Params: {} 2022-09-12T20:15:58.842 DEBUG:boto:Token: None 2022-09-12T20:15:58.843 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj115 2022-09-12T20:15:58.843 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NO/RlTCwxsnPAMWNg5XF+rM11NU= 2022-09-12T20:15:58.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NO/RlTCwxsnPAMWNg5XF+rM11NU=', 'Content-Length': '0'} 2022-09-12T20:15:58.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c41fe80b0505565-00631f937e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.847 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:15:58.847 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:15:58.848 DEBUG:boto:Method: HEAD 2022-09-12T20:15:58.848 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:15:58.849 DEBUG:boto:Data: 2022-09-12T20:15:58.849 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.850 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.850 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.851 DEBUG:boto:Params: {} 2022-09-12T20:15:58.852 DEBUG:boto:Token: None 2022-09-12T20:15:58.852 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj115 2022-09-12T20:15:58.853 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NO/RlTCwxsnPAMWNg5XF+rM11NU= 2022-09-12T20:15:58.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NO/RlTCwxsnPAMWNg5XF+rM11NU=', 'Content-Length': '0'} 2022-09-12T20:15:58.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aebe5666693e2d8f-00631f937e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:15:58.857 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:15:58.857 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:15:58.858 DEBUG:boto:Method: GET 2022-09-12T20:15:58.858 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:15:58.859 DEBUG:boto:Data: 2022-09-12T20:15:58.859 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.860 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.860 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.861 DEBUG:boto:Params: {} 2022-09-12T20:15:58.862 DEBUG:boto:Token: None 2022-09-12T20:15:58.862 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj115 2022-09-12T20:15:58.863 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7Vfw84I6wILiHVu/8KloCIfxH10= 2022-09-12T20:15:58.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7Vfw84I6wILiHVu/8KloCIfxH10=', 'Content-Length': '0'} 2022-09-12T20:15:58.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067ce0e0672bf1419-00631f937e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.866 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:15:58.867 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:15:58.868 DEBUG:boto:Method: GET 2022-09-12T20:15:58.868 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:15:58.869 DEBUG:boto:Data: 2022-09-12T20:15:58.869 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.870 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.870 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.871 DEBUG:boto:Params: {} 2022-09-12T20:15:58.871 DEBUG:boto:Token: None 2022-09-12T20:15:58.872 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj115 2022-09-12T20:15:58.873 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7Vfw84I6wILiHVu/8KloCIfxH10= 2022-09-12T20:15:58.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7Vfw84I6wILiHVu/8KloCIfxH10=', 'Content-Length': '0'} 2022-09-12T20:15:58.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018e72748fbec114b-00631f937e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.876 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:58.876 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:58.877 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:15:58.878 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:15:58.878 DEBUG:boto:Method: GET 2022-09-12T20:15:58.879 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:15:58.879 DEBUG:boto:Data: 2022-09-12T20:15:58.880 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.880 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.881 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.881 DEBUG:boto:Params: {} 2022-09-12T20:15:58.882 DEBUG:boto:Token: None 2022-09-12T20:15:58.883 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/ 2022-09-12T20:15:58.883 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4dHe/m1FIw7ChRaNAGgtLkVCDco= 2022-09-12T20:15:58.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4dHe/m1FIw7ChRaNAGgtLkVCDco=', 'Content-Length': '0'} 2022-09-12T20:15:58.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011aae9b7110087ab-00631f937e-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.887 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:58.888 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:58.889 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:58.889 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:15:58.890 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:15:58.890 DEBUG:boto:Method: GET 2022-09-12T20:15:58.891 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:15:58.892 DEBUG:boto:Data: 2022-09-12T20:15:58.892 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.893 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.893 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.894 DEBUG:boto:Params: {} 2022-09-12T20:15:58.894 DEBUG:boto:Token: None 2022-09-12T20:15:58.895 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/ 2022-09-12T20:15:58.895 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4dHe/m1FIw7ChRaNAGgtLkVCDco= 2022-09-12T20:15:58.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4dHe/m1FIw7ChRaNAGgtLkVCDco=', 'Content-Length': '0'} 2022-09-12T20:15:58.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000270fb4e5d41c6633-00631f937e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.901 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:58.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:15:58.902 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:15:58.903 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:15:58.903 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:15:58.904 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:15:58.904 DEBUG:boto:Method: GET 2022-09-12T20:15:58.905 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:15:58.906 DEBUG:boto:Data: 2022-09-12T20:15:58.906 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.907 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.907 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.908 DEBUG:boto:Params: {} 2022-09-12T20:15:58.908 DEBUG:boto:Token: None 2022-09-12T20:15:58.909 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:15:58.910 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kjQDcDW/r5XyF92wOstZ8TO8Z2o= 2022-09-12T20:15:58.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kjQDcDW/r5XyF92wOstZ8TO8Z2o=', 'Content-Length': '0'} 2022-09-12T20:15:58.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d57773215474d7a2-00631f937e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.913 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:15:58.914 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:15:58.914 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:15:58.915 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:15:58.915 DEBUG:boto:Method: GET 2022-09-12T20:15:58.916 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:15:58.916 DEBUG:boto:Data: 2022-09-12T20:15:58.917 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.917 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.918 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.919 DEBUG:boto:Params: {} 2022-09-12T20:15:58.919 DEBUG:boto:Token: None 2022-09-12T20:15:58.920 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:15:58.920 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kjQDcDW/r5XyF92wOstZ8TO8Z2o= 2022-09-12T20:15:58.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kjQDcDW/r5XyF92wOstZ8TO8Z2o=', 'Content-Length': '0'} 2022-09-12T20:15:58.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000346468ab06967b77-00631f937e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.966 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:15:58.967 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:15:58.967 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:15:58.968 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:15:58.968 DEBUG:boto:Method: HEAD 2022-09-12T20:15:58.969 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:15:58.969 DEBUG:boto:Data: 2022-09-12T20:15:58.970 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.971 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.971 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.972 DEBUG:boto:Params: {} 2022-09-12T20:15:58.972 DEBUG:boto:Token: None 2022-09-12T20:15:58.973 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:15:58.974 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SiSj2YPOIX6YvrNvIjWLswr9B2I= 2022-09-12T20:15:58.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SiSj2YPOIX6YvrNvIjWLswr9B2I=', 'Content-Length': '0'} 2022-09-12T20:15:58.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a552795efe0f915-00631f937e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.977 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:15:58.978 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:15:58.979 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:15:58.980 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:15:58.981 DEBUG:boto:Method: HEAD 2022-09-12T20:15:58.981 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:15:58.982 DEBUG:boto:Data: 2022-09-12T20:15:58.982 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.983 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:58.983 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.984 DEBUG:boto:Params: {} 2022-09-12T20:15:58.985 DEBUG:boto:Token: None 2022-09-12T20:15:58.985 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:15:58.986 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SiSj2YPOIX6YvrNvIjWLswr9B2I= 2022-09-12T20:15:58.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SiSj2YPOIX6YvrNvIjWLswr9B2I=', 'Content-Length': '0'} 2022-09-12T20:15:58.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbcb4d519158c1a5-00631f937e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:58.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:15:58.990 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:15:58.990 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:15:58.991 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:15:58.991 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:15:58.992 DEBUG:boto:Method: GET 2022-09-12T20:15:58.993 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:15:58.993 DEBUG:boto:Data: 2022-09-12T20:15:58.994 DEBUG:boto:Headers: {} 2022-09-12T20:15:58.995 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:58.995 DEBUG:boto:Port: 8000 2022-09-12T20:15:58.996 DEBUG:boto:Params: {} 2022-09-12T20:15:58.996 DEBUG:boto:Token: None 2022-09-12T20:15:58.997 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:58 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:15:58.998 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kjQDcDW/r5XyF92wOstZ8TO8Z2o= 2022-09-12T20:15:58.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:58 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kjQDcDW/r5XyF92wOstZ8TO8Z2o=', 'Content-Length': '0'} 2022-09-12T20:15:59.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8b133ca90bd6943-00631f937e-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.001 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:15:59.002 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:15:59.002 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:15:59.003 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:15:59.003 DEBUG:boto:Method: GET 2022-09-12T20:15:59.004 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:15:59.005 DEBUG:boto:Data: 2022-09-12T20:15:59.005 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.006 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.007 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.007 DEBUG:boto:Params: {} 2022-09-12T20:15:59.008 DEBUG:boto:Token: None 2022-09-12T20:15:59.009 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:15:59.009 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wKpSioqxWinOLeezsIythp/76I4= 2022-09-12T20:15:59.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wKpSioqxWinOLeezsIythp/76I4=', 'Content-Length': '0'} 2022-09-12T20:15:59.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a0d45f592a49ee6-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.013 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:59.014 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:59.014 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:15:59.015 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:15:59.015 DEBUG:boto:Method: GET 2022-09-12T20:15:59.016 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:15:59.017 DEBUG:boto:Data: 2022-09-12T20:15:59.017 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.018 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.019 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.019 DEBUG:boto:Params: {} 2022-09-12T20:15:59.020 DEBUG:boto:Token: None 2022-09-12T20:15:59.020 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/ 2022-09-12T20:15:59.021 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10= 2022-09-12T20:15:59.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10=', 'Content-Length': '0'} 2022-09-12T20:15:59.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d35183d9cc328b8-00631f937f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.025 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:59.026 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:59.026 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:59.027 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:15:59.027 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:15:59.028 DEBUG:boto:Method: GET 2022-09-12T20:15:59.028 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:15:59.029 DEBUG:boto:Data: 2022-09-12T20:15:59.029 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.030 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.030 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.031 DEBUG:boto:Params: {} 2022-09-12T20:15:59.032 DEBUG:boto:Token: None 2022-09-12T20:15:59.032 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/ 2022-09-12T20:15:59.033 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10= 2022-09-12T20:15:59.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10=', 'Content-Length': '0'} 2022-09-12T20:15:59.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af04e3e9180cdd10-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.039 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:59.039 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:15:59.040 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:15:59.040 DEBUG:boto:Method: HEAD 2022-09-12T20:15:59.041 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:15:59.041 DEBUG:boto:Data: 2022-09-12T20:15:59.042 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.042 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.043 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.043 DEBUG:boto:Params: {} 2022-09-12T20:15:59.044 DEBUG:boto:Token: None 2022-09-12T20:15:59.045 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj116 2022-09-12T20:15:59.045 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iBUF7tJSJPF2t8i9SeGLVs9vqbM= 2022-09-12T20:15:59.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iBUF7tJSJPF2t8i9SeGLVs9vqbM=', 'Content-Length': '0'} 2022-09-12T20:15:59.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e118b2ec34b6353e-00631f937f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.049 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:15:59.049 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:15:59.050 DEBUG:boto:Method: HEAD 2022-09-12T20:15:59.050 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:15:59.051 DEBUG:boto:Data: 2022-09-12T20:15:59.051 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.052 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.052 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.053 DEBUG:boto:Params: {} 2022-09-12T20:15:59.054 DEBUG:boto:Token: None 2022-09-12T20:15:59.054 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj116 2022-09-12T20:15:59.055 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iBUF7tJSJPF2t8i9SeGLVs9vqbM= 2022-09-12T20:15:59.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iBUF7tJSJPF2t8i9SeGLVs9vqbM=', 'Content-Length': '0'} 2022-09-12T20:15:59.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f688e0274e4a5ad-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:15:59.059 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:15:59.059 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:15:59.060 DEBUG:boto:Method: GET 2022-09-12T20:15:59.061 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:15:59.061 DEBUG:boto:Data: 2022-09-12T20:15:59.062 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.062 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.063 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.063 DEBUG:boto:Params: {} 2022-09-12T20:15:59.064 DEBUG:boto:Token: None 2022-09-12T20:15:59.065 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj116 2022-09-12T20:15:59.066 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DgzhnYYSFL2/4iQDJpVG2aAePSE= 2022-09-12T20:15:59.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DgzhnYYSFL2/4iQDJpVG2aAePSE=', 'Content-Length': '0'} 2022-09-12T20:15:59.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c16b802bebaca5f5-00631f937f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.069 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:15:59.070 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:15:59.070 DEBUG:boto:Method: GET 2022-09-12T20:15:59.071 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:15:59.072 DEBUG:boto:Data: 2022-09-12T20:15:59.072 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.073 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.073 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.074 DEBUG:boto:Params: {} 2022-09-12T20:15:59.074 DEBUG:boto:Token: None 2022-09-12T20:15:59.075 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj116 2022-09-12T20:15:59.076 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DgzhnYYSFL2/4iQDJpVG2aAePSE= 2022-09-12T20:15:59.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DgzhnYYSFL2/4iQDJpVG2aAePSE=', 'Content-Length': '0'} 2022-09-12T20:15:59.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008039d623294e1e2f-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.079 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:59.080 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:59.080 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:15:59.081 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:15:59.082 DEBUG:boto:Method: GET 2022-09-12T20:15:59.082 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:15:59.083 DEBUG:boto:Data: 2022-09-12T20:15:59.083 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.084 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.085 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.085 DEBUG:boto:Params: {} 2022-09-12T20:15:59.086 DEBUG:boto:Token: None 2022-09-12T20:15:59.086 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/ 2022-09-12T20:15:59.087 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10= 2022-09-12T20:15:59.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10=', 'Content-Length': '0'} 2022-09-12T20:15:59.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000495e5b9b7e285f4d-00631f937f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.091 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:59.092 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:59.092 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:59.093 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:15:59.093 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:15:59.094 DEBUG:boto:Method: GET 2022-09-12T20:15:59.095 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:15:59.095 DEBUG:boto:Data: 2022-09-12T20:15:59.096 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.096 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.097 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.098 DEBUG:boto:Params: {} 2022-09-12T20:15:59.098 DEBUG:boto:Token: None 2022-09-12T20:15:59.099 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/ 2022-09-12T20:15:59.099 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10= 2022-09-12T20:15:59.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10=', 'Content-Length': '0'} 2022-09-12T20:15:59.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a4cfd3a5fd87bd0-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.146 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:59.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-12T20:15:59.148 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:15:59.148 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:15:59.149 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:15:59.149 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:15:59.150 DEBUG:boto:Method: GET 2022-09-12T20:15:59.150 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:15:59.151 DEBUG:boto:Data: 2022-09-12T20:15:59.151 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.152 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.152 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.153 DEBUG:boto:Params: {} 2022-09-12T20:15:59.153 DEBUG:boto:Token: None 2022-09-12T20:15:59.154 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:15:59.155 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BFPy1GzTuGj37panMMiRRcRsEKw= 2022-09-12T20:15:59.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BFPy1GzTuGj37panMMiRRcRsEKw=', 'Content-Length': '0'} 2022-09-12T20:15:59.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce8ab2c6ad96a539-00631f937f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.158 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:15:59.159 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:15:59.159 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:15:59.160 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:15:59.160 DEBUG:boto:Method: GET 2022-09-12T20:15:59.161 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:15:59.161 DEBUG:boto:Data: 2022-09-12T20:15:59.162 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.162 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.163 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.163 DEBUG:boto:Params: {} 2022-09-12T20:15:59.164 DEBUG:boto:Token: None 2022-09-12T20:15:59.165 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:15:59.165 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BFPy1GzTuGj37panMMiRRcRsEKw= 2022-09-12T20:15:59.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BFPy1GzTuGj37panMMiRRcRsEKw=', 'Content-Length': '0'} 2022-09-12T20:15:59.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088bb119440142892-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.169 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:15:59.170 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:15:59.170 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:15:59.171 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:15:59.171 DEBUG:boto:Method: HEAD 2022-09-12T20:15:59.172 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:15:59.172 DEBUG:boto:Data: 2022-09-12T20:15:59.173 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.173 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.173 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.174 DEBUG:boto:Params: {} 2022-09-12T20:15:59.175 DEBUG:boto:Token: None 2022-09-12T20:15:59.175 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:15:59.176 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Arn5xUXjc0IAELSZzVR5+S+82AQ= 2022-09-12T20:15:59.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Arn5xUXjc0IAELSZzVR5+S+82AQ=', 'Content-Length': '0'} 2022-09-12T20:15:59.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007215c2cc1cfa703b-00631f937f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.179 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:15:59.179 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:15:59.180 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:15:59.180 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:15:59.181 DEBUG:boto:Method: HEAD 2022-09-12T20:15:59.181 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:15:59.182 DEBUG:boto:Data: 2022-09-12T20:15:59.182 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.183 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.183 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.184 DEBUG:boto:Params: {} 2022-09-12T20:15:59.184 DEBUG:boto:Token: None 2022-09-12T20:15:59.185 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:15:59.186 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Arn5xUXjc0IAELSZzVR5+S+82AQ= 2022-09-12T20:15:59.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Arn5xUXjc0IAELSZzVR5+S+82AQ=', 'Content-Length': '0'} 2022-09-12T20:15:59.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036d48a201c1f08c9-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-12T20:15:59.189 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:15:59.190 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:15:59.190 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:15:59.191 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:15:59.191 DEBUG:boto:Method: GET 2022-09-12T20:15:59.192 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:15:59.192 DEBUG:boto:Data: 2022-09-12T20:15:59.193 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.193 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.194 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.195 DEBUG:boto:Params: {} 2022-09-12T20:15:59.195 DEBUG:boto:Token: None 2022-09-12T20:15:59.196 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:15:59.196 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BFPy1GzTuGj37panMMiRRcRsEKw= 2022-09-12T20:15:59.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BFPy1GzTuGj37panMMiRRcRsEKw=', 'Content-Length': '0'} 2022-09-12T20:15:59.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d258660c09c90bb7-00631f937f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.242 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:15:59.243 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:15:59.243 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:15:59.244 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:15:59.244 DEBUG:boto:Method: GET 2022-09-12T20:15:59.245 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:15:59.245 DEBUG:boto:Data: 2022-09-12T20:15:59.246 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.246 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.247 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.248 DEBUG:boto:Params: {} 2022-09-12T20:15:59.248 DEBUG:boto:Token: None 2022-09-12T20:15:59.249 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:15:59.249 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BFPy1GzTuGj37panMMiRRcRsEKw= 2022-09-12T20:15:59.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BFPy1GzTuGj37panMMiRRcRsEKw=', 'Content-Length': '0'} 2022-09-12T20:15:59.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bf0612a953575bd-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.253 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:59.253 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:59.254 DEBUG:boto:path=/jdpiak-75/?prefix=obj124 2022-09-12T20:15:59.255 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj124 2022-09-12T20:15:59.256 DEBUG:boto:Method: GET 2022-09-12T20:15:59.256 DEBUG:boto:Path: /jdpiak-75/?prefix=obj124 2022-09-12T20:15:59.257 DEBUG:boto:Data: 2022-09-12T20:15:59.257 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.258 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.258 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.259 DEBUG:boto:Params: {} 2022-09-12T20:15:59.259 DEBUG:boto:Token: None 2022-09-12T20:15:59.260 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/ 2022-09-12T20:15:59.261 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10= 2022-09-12T20:15:59.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10=', 'Content-Length': '0'} 2022-09-12T20:15:59.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe4b67ce17ee842b-00631f937f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.264 DEBUG:boto:b'jdpiak-75obj1241000falseobj1242022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:59.265 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:59.266 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:59.266 DEBUG:boto:path=/jdpiak-75/?prefix=obj124 2022-09-12T20:15:59.267 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj124 2022-09-12T20:15:59.267 DEBUG:boto:Method: GET 2022-09-12T20:15:59.268 DEBUG:boto:Path: /jdpiak-75/?prefix=obj124 2022-09-12T20:15:59.268 DEBUG:boto:Data: 2022-09-12T20:15:59.269 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.269 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.270 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.271 DEBUG:boto:Params: {} 2022-09-12T20:15:59.271 DEBUG:boto:Token: None 2022-09-12T20:15:59.272 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/ 2022-09-12T20:15:59.272 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10= 2022-09-12T20:15:59.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10=', 'Content-Length': '0'} 2022-09-12T20:15:59.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cc5230fb1eeb861-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.278 DEBUG:boto:b'jdpiak-75obj1241000falseobj1242022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:59.279 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:15:59.279 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:15:59.280 DEBUG:boto:Method: HEAD 2022-09-12T20:15:59.280 DEBUG:boto:Path: /jdpiak-75/obj124 2022-09-12T20:15:59.281 DEBUG:boto:Data: 2022-09-12T20:15:59.281 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.282 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.282 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.283 DEBUG:boto:Params: {} 2022-09-12T20:15:59.283 DEBUG:boto:Token: None 2022-09-12T20:15:59.284 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj124 2022-09-12T20:15:59.285 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z+0HuI2j0YUTTDQ4Brd5PPbzEzg= 2022-09-12T20:15:59.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z+0HuI2j0YUTTDQ4Brd5PPbzEzg=', 'Content-Length': '0'} 2022-09-12T20:15:59.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e599fd602b3063a-00631f937f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.288 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:15:59.288 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:15:59.289 DEBUG:boto:Method: HEAD 2022-09-12T20:15:59.289 DEBUG:boto:Path: /jdpiak-75/obj124 2022-09-12T20:15:59.290 DEBUG:boto:Data: 2022-09-12T20:15:59.290 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.291 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.291 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.292 DEBUG:boto:Params: {} 2022-09-12T20:15:59.292 DEBUG:boto:Token: None 2022-09-12T20:15:59.293 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj124 2022-09-12T20:15:59.293 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:z+0HuI2j0YUTTDQ4Brd5PPbzEzg= 2022-09-12T20:15:59.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:z+0HuI2j0YUTTDQ4Brd5PPbzEzg=', 'Content-Length': '0'} 2022-09-12T20:15:59.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af444b601c52aab7-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-12T20:15:59.297 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:15:59.298 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:15:59.299 DEBUG:boto:Method: GET 2022-09-12T20:15:59.300 DEBUG:boto:Path: /jdpiak-75/obj124 2022-09-12T20:15:59.300 DEBUG:boto:Data: 2022-09-12T20:15:59.301 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.302 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.302 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.303 DEBUG:boto:Params: {} 2022-09-12T20:15:59.303 DEBUG:boto:Token: None 2022-09-12T20:15:59.304 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj124 2022-09-12T20:15:59.304 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HqSbaP3mDOKCv7V2pxehrWmfja4= 2022-09-12T20:15:59.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HqSbaP3mDOKCv7V2pxehrWmfja4=', 'Content-Length': '0'} 2022-09-12T20:15:59.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce4f2f05a7f6cc32-00631f937f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.308 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:15:59.308 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:15:59.309 DEBUG:boto:Method: GET 2022-09-12T20:15:59.310 DEBUG:boto:Path: /jdpiak-75/obj124 2022-09-12T20:15:59.310 DEBUG:boto:Data: 2022-09-12T20:15:59.311 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.312 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.312 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.313 DEBUG:boto:Params: {} 2022-09-12T20:15:59.313 DEBUG:boto:Token: None 2022-09-12T20:15:59.314 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj124 2022-09-12T20:15:59.314 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HqSbaP3mDOKCv7V2pxehrWmfja4= 2022-09-12T20:15:59.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HqSbaP3mDOKCv7V2pxehrWmfja4=', 'Content-Length': '0'} 2022-09-12T20:15:59.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056aea1486b0417f7-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.317 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:59.318 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:59.318 DEBUG:boto:path=/jdpiak-75/?prefix=obj124 2022-09-12T20:15:59.319 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj124 2022-09-12T20:15:59.320 DEBUG:boto:Method: GET 2022-09-12T20:15:59.320 DEBUG:boto:Path: /jdpiak-75/?prefix=obj124 2022-09-12T20:15:59.321 DEBUG:boto:Data: 2022-09-12T20:15:59.321 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.322 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.322 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.323 DEBUG:boto:Params: {} 2022-09-12T20:15:59.323 DEBUG:boto:Token: None 2022-09-12T20:15:59.324 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/ 2022-09-12T20:15:59.324 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10= 2022-09-12T20:15:59.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10=', 'Content-Length': '0'} 2022-09-12T20:15:59.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040c209f98332d7a4-00631f937f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.328 DEBUG:boto:b'jdpiak-75obj1241000falseobj1242022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:59.329 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:59.329 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:59.330 DEBUG:boto:path=/jdpiak-75/?prefix=obj124 2022-09-12T20:15:59.331 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj124 2022-09-12T20:15:59.331 DEBUG:boto:Method: GET 2022-09-12T20:15:59.332 DEBUG:boto:Path: /jdpiak-75/?prefix=obj124 2022-09-12T20:15:59.332 DEBUG:boto:Data: 2022-09-12T20:15:59.333 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.333 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.334 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.334 DEBUG:boto:Params: {} 2022-09-12T20:15:59.335 DEBUG:boto:Token: None 2022-09-12T20:15:59.335 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/ 2022-09-12T20:15:59.336 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10= 2022-09-12T20:15:59.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10=', 'Content-Length': '0'} 2022-09-12T20:15:59.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adbf01baf71bf076-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.342 DEBUG:boto:b'jdpiak-75obj1241000falseobj1242022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:59.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:15:59.343 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:15:59.344 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:15:59.344 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:15:59.345 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:15:59.345 DEBUG:boto:Method: GET 2022-09-12T20:15:59.346 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:15:59.346 DEBUG:boto:Data: 2022-09-12T20:15:59.347 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.347 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.348 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.348 DEBUG:boto:Params: {} 2022-09-12T20:15:59.349 DEBUG:boto:Token: None 2022-09-12T20:15:59.349 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:15:59.350 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FrGVVb4Hi6Pa98xrjwbRWrvJWG8= 2022-09-12T20:15:59.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FrGVVb4Hi6Pa98xrjwbRWrvJWG8=', 'Content-Length': '0'} 2022-09-12T20:15:59.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bd5f311073b5ab5-00631f937f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.354 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:15:59.354 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:15:59.355 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:15:59.355 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:15:59.356 DEBUG:boto:Method: GET 2022-09-12T20:15:59.356 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:15:59.357 DEBUG:boto:Data: 2022-09-12T20:15:59.357 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.358 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.358 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.359 DEBUG:boto:Params: {} 2022-09-12T20:15:59.359 DEBUG:boto:Token: None 2022-09-12T20:15:59.360 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:15:59.361 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FrGVVb4Hi6Pa98xrjwbRWrvJWG8= 2022-09-12T20:15:59.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FrGVVb4Hi6Pa98xrjwbRWrvJWG8=', 'Content-Length': '0'} 2022-09-12T20:15:59.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b772e240d107249-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.410 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:15:59.411 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:15:59.412 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:15:59.413 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:15:59.414 DEBUG:boto:Method: HEAD 2022-09-12T20:15:59.414 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:15:59.415 DEBUG:boto:Data: 2022-09-12T20:15:59.416 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.417 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.418 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.419 DEBUG:boto:Params: {} 2022-09-12T20:15:59.420 DEBUG:boto:Token: None 2022-09-12T20:15:59.421 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:15:59.422 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T5rHm1cxJMULfQHKrs3VWuFSj3o= 2022-09-12T20:15:59.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T5rHm1cxJMULfQHKrs3VWuFSj3o=', 'Content-Length': '0'} 2022-09-12T20:15:59.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e256c96dd62cfc73-00631f937f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.426 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:15:59.426 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:15:59.427 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:15:59.428 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:15:59.428 DEBUG:boto:Method: HEAD 2022-09-12T20:15:59.429 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:15:59.429 DEBUG:boto:Data: 2022-09-12T20:15:59.430 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.430 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.431 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.431 DEBUG:boto:Params: {} 2022-09-12T20:15:59.432 DEBUG:boto:Token: None 2022-09-12T20:15:59.433 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:15:59.433 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T5rHm1cxJMULfQHKrs3VWuFSj3o= 2022-09-12T20:15:59.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T5rHm1cxJMULfQHKrs3VWuFSj3o=', 'Content-Length': '0'} 2022-09-12T20:15:59.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000930259ecb2b4663-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:15:59.438 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:15:59.439 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:15:59.440 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:15:59.441 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:15:59.442 DEBUG:boto:Method: GET 2022-09-12T20:15:59.442 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:15:59.443 DEBUG:boto:Data: 2022-09-12T20:15:59.443 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.444 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.445 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.445 DEBUG:boto:Params: {} 2022-09-12T20:15:59.446 DEBUG:boto:Token: None 2022-09-12T20:15:59.447 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:15:59.447 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FrGVVb4Hi6Pa98xrjwbRWrvJWG8= 2022-09-12T20:15:59.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FrGVVb4Hi6Pa98xrjwbRWrvJWG8=', 'Content-Length': '0'} 2022-09-12T20:15:59.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdc157af49c5fda7-00631f937f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.451 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:15:59.451 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:15:59.452 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:15:59.452 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:15:59.453 DEBUG:boto:Method: GET 2022-09-12T20:15:59.454 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:15:59.454 DEBUG:boto:Data: 2022-09-12T20:15:59.455 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.455 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.456 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.456 DEBUG:boto:Params: {} 2022-09-12T20:15:59.457 DEBUG:boto:Token: None 2022-09-12T20:15:59.458 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:15:59.458 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FrGVVb4Hi6Pa98xrjwbRWrvJWG8= 2022-09-12T20:15:59.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FrGVVb4Hi6Pa98xrjwbRWrvJWG8=', 'Content-Length': '0'} 2022-09-12T20:15:59.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff433af4309e0025-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.462 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:59.463 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:59.463 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:15:59.464 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:15:59.464 DEBUG:boto:Method: GET 2022-09-12T20:15:59.465 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:15:59.465 DEBUG:boto:Data: 2022-09-12T20:15:59.466 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.466 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.467 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.468 DEBUG:boto:Params: {} 2022-09-12T20:15:59.468 DEBUG:boto:Token: None 2022-09-12T20:15:59.469 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/ 2022-09-12T20:15:59.470 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10= 2022-09-12T20:15:59.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10=', 'Content-Length': '0'} 2022-09-12T20:15:59.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4b338d871e7fe9f-00631f937f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.518 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:59.519 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:59.520 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:59.521 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:15:59.522 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:15:59.523 DEBUG:boto:Method: GET 2022-09-12T20:15:59.525 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:15:59.526 DEBUG:boto:Data: 2022-09-12T20:15:59.526 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.527 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.528 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.528 DEBUG:boto:Params: {} 2022-09-12T20:15:59.529 DEBUG:boto:Token: None 2022-09-12T20:15:59.529 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/ 2022-09-12T20:15:59.530 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10= 2022-09-12T20:15:59.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10=', 'Content-Length': '0'} 2022-09-12T20:15:59.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d44f9b1e1b7fbc7-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.537 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:59.538 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:15:59.538 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:15:59.539 DEBUG:boto:Method: HEAD 2022-09-12T20:15:59.539 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:15:59.540 DEBUG:boto:Data: 2022-09-12T20:15:59.541 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.541 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.542 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.542 DEBUG:boto:Params: {} 2022-09-12T20:15:59.543 DEBUG:boto:Token: None 2022-09-12T20:15:59.544 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj145 2022-09-12T20:15:59.544 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nHH2BwdyBYjUXeooOYRbTmQX2ls= 2022-09-12T20:15:59.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nHH2BwdyBYjUXeooOYRbTmQX2ls=', 'Content-Length': '0'} 2022-09-12T20:15:59.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000370bc1076d2d5e6d-00631f937f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.547 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:15:59.548 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:15:59.549 DEBUG:boto:Method: HEAD 2022-09-12T20:15:59.549 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:15:59.550 DEBUG:boto:Data: 2022-09-12T20:15:59.551 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.551 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.552 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.552 DEBUG:boto:Params: {} 2022-09-12T20:15:59.553 DEBUG:boto:Token: None 2022-09-12T20:15:59.553 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj145 2022-09-12T20:15:59.554 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nHH2BwdyBYjUXeooOYRbTmQX2ls= 2022-09-12T20:15:59.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nHH2BwdyBYjUXeooOYRbTmQX2ls=', 'Content-Length': '0'} 2022-09-12T20:15:59.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7c419cf77e76e2a-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:15:59.559 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:15:59.560 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:15:59.561 DEBUG:boto:Method: GET 2022-09-12T20:15:59.562 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:15:59.563 DEBUG:boto:Data: 2022-09-12T20:15:59.564 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.565 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.566 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.567 DEBUG:boto:Params: {} 2022-09-12T20:15:59.568 DEBUG:boto:Token: None 2022-09-12T20:15:59.568 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj145 2022-09-12T20:15:59.569 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sessf4kEatB/EjlRCRXztPgzfqo= 2022-09-12T20:15:59.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sessf4kEatB/EjlRCRXztPgzfqo=', 'Content-Length': '0'} 2022-09-12T20:15:59.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4243fd27f25dfe7-00631f937f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.573 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:15:59.573 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:15:59.574 DEBUG:boto:Method: GET 2022-09-12T20:15:59.575 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:15:59.575 DEBUG:boto:Data: 2022-09-12T20:15:59.576 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.576 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.577 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.577 DEBUG:boto:Params: {} 2022-09-12T20:15:59.578 DEBUG:boto:Token: None 2022-09-12T20:15:59.579 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj145 2022-09-12T20:15:59.579 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sessf4kEatB/EjlRCRXztPgzfqo= 2022-09-12T20:15:59.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sessf4kEatB/EjlRCRXztPgzfqo=', 'Content-Length': '0'} 2022-09-12T20:15:59.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f6ffc8844992a71-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.583 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:59.583 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:59.584 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:15:59.585 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:15:59.585 DEBUG:boto:Method: GET 2022-09-12T20:15:59.586 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:15:59.586 DEBUG:boto:Data: 2022-09-12T20:15:59.587 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.587 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.588 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.588 DEBUG:boto:Params: {} 2022-09-12T20:15:59.589 DEBUG:boto:Token: None 2022-09-12T20:15:59.590 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/ 2022-09-12T20:15:59.590 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10= 2022-09-12T20:15:59.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10=', 'Content-Length': '0'} 2022-09-12T20:15:59.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e5510ebcc410f93-00631f937f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.595 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:59.596 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:59.596 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:59.597 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:15:59.597 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:15:59.598 DEBUG:boto:Method: GET 2022-09-12T20:15:59.599 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:15:59.599 DEBUG:boto:Data: 2022-09-12T20:15:59.600 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.600 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.601 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.601 DEBUG:boto:Params: {} 2022-09-12T20:15:59.602 DEBUG:boto:Token: None 2022-09-12T20:15:59.603 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/ 2022-09-12T20:15:59.603 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10= 2022-09-12T20:15:59.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10=', 'Content-Length': '0'} 2022-09-12T20:15:59.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093124587dcdf13b6-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.654 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:59.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2022-09-12T20:15:59.656 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:15:59.656 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:15:59.657 DEBUG:boto:path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:15:59.657 DEBUG:boto:auth_path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:15:59.658 DEBUG:boto:Method: GET 2022-09-12T20:15:59.659 DEBUG:boto:Path: /jdpiak-75/obj148?versionId=null 2022-09-12T20:15:59.659 DEBUG:boto:Data: 2022-09-12T20:15:59.660 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.660 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.661 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.661 DEBUG:boto:Params: {} 2022-09-12T20:15:59.662 DEBUG:boto:Token: None 2022-09-12T20:15:59.663 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj148?versionId=null 2022-09-12T20:15:59.663 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WNA95mdl2+vQx5Vlv5FljmIJil4= 2022-09-12T20:15:59.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WNA95mdl2+vQx5Vlv5FljmIJil4=', 'Content-Length': '0'} 2022-09-12T20:15:59.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cabbdd3c3c8707c8-00631f937f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.667 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:15:59.667 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:15:59.668 DEBUG:boto:path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:15:59.668 DEBUG:boto:auth_path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:15:59.669 DEBUG:boto:Method: GET 2022-09-12T20:15:59.669 DEBUG:boto:Path: /jdpiak-75/obj148?versionId=null 2022-09-12T20:15:59.670 DEBUG:boto:Data: 2022-09-12T20:15:59.671 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.671 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.672 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.672 DEBUG:boto:Params: {} 2022-09-12T20:15:59.673 DEBUG:boto:Token: None 2022-09-12T20:15:59.673 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj148?versionId=null 2022-09-12T20:15:59.674 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WNA95mdl2+vQx5Vlv5FljmIJil4= 2022-09-12T20:15:59.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WNA95mdl2+vQx5Vlv5FljmIJil4=', 'Content-Length': '0'} 2022-09-12T20:15:59.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000743bec61c34a39cc-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.678 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:15:59.678 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:15:59.679 DEBUG:boto:path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:15:59.679 DEBUG:boto:auth_path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:15:59.680 DEBUG:boto:Method: HEAD 2022-09-12T20:15:59.681 DEBUG:boto:Path: /jdpiak-75/obj148?versionId=null 2022-09-12T20:15:59.681 DEBUG:boto:Data: 2022-09-12T20:15:59.682 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.682 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.683 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.683 DEBUG:boto:Params: {} 2022-09-12T20:15:59.684 DEBUG:boto:Token: None 2022-09-12T20:15:59.684 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj148?versionId=null 2022-09-12T20:15:59.685 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eDx4wK7Whyq0LuuAKSqejqajnWA= 2022-09-12T20:15:59.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eDx4wK7Whyq0LuuAKSqejqajnWA=', 'Content-Length': '0'} 2022-09-12T20:15:59.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008263acf33252a887-00631f937f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.688 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:15:59.689 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:15:59.689 DEBUG:boto:path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:15:59.690 DEBUG:boto:auth_path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:15:59.690 DEBUG:boto:Method: HEAD 2022-09-12T20:15:59.691 DEBUG:boto:Path: /jdpiak-75/obj148?versionId=null 2022-09-12T20:15:59.691 DEBUG:boto:Data: 2022-09-12T20:15:59.692 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.692 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.693 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.693 DEBUG:boto:Params: {} 2022-09-12T20:15:59.694 DEBUG:boto:Token: None 2022-09-12T20:15:59.695 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj148?versionId=null 2022-09-12T20:15:59.695 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eDx4wK7Whyq0LuuAKSqejqajnWA= 2022-09-12T20:15:59.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eDx4wK7Whyq0LuuAKSqejqajnWA=', 'Content-Length': '0'} 2022-09-12T20:15:59.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c973c668fa02747a-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2022-09-12T20:15:59.699 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:15:59.700 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:15:59.700 DEBUG:boto:path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:15:59.701 DEBUG:boto:auth_path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:15:59.701 DEBUG:boto:Method: GET 2022-09-12T20:15:59.702 DEBUG:boto:Path: /jdpiak-75/obj148?versionId=null 2022-09-12T20:15:59.703 DEBUG:boto:Data: 2022-09-12T20:15:59.703 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.704 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.704 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.705 DEBUG:boto:Params: {} 2022-09-12T20:15:59.705 DEBUG:boto:Token: None 2022-09-12T20:15:59.706 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj148?versionId=null 2022-09-12T20:15:59.707 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WNA95mdl2+vQx5Vlv5FljmIJil4= 2022-09-12T20:15:59.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WNA95mdl2+vQx5Vlv5FljmIJil4=', 'Content-Length': '0'} 2022-09-12T20:15:59.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f799d5604aabdf81-00631f937f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.754 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:15:59.755 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:15:59.755 DEBUG:boto:path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:15:59.756 DEBUG:boto:auth_path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:15:59.757 DEBUG:boto:Method: GET 2022-09-12T20:15:59.758 DEBUG:boto:Path: /jdpiak-75/obj148?versionId=null 2022-09-12T20:15:59.759 DEBUG:boto:Data: 2022-09-12T20:15:59.760 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.761 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.762 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.763 DEBUG:boto:Params: {} 2022-09-12T20:15:59.764 DEBUG:boto:Token: None 2022-09-12T20:15:59.765 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj148?versionId=null 2022-09-12T20:15:59.766 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WNA95mdl2+vQx5Vlv5FljmIJil4= 2022-09-12T20:15:59.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WNA95mdl2+vQx5Vlv5FljmIJil4=', 'Content-Length': '0'} 2022-09-12T20:15:59.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b870b2c2fc103fbe-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.771 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:59.772 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:59.773 DEBUG:boto:path=/jdpiak-75/?prefix=obj148 2022-09-12T20:15:59.774 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj148 2022-09-12T20:15:59.775 DEBUG:boto:Method: GET 2022-09-12T20:15:59.776 DEBUG:boto:Path: /jdpiak-75/?prefix=obj148 2022-09-12T20:15:59.777 DEBUG:boto:Data: 2022-09-12T20:15:59.778 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.779 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.780 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.781 DEBUG:boto:Params: {} 2022-09-12T20:15:59.782 DEBUG:boto:Token: None 2022-09-12T20:15:59.783 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/ 2022-09-12T20:15:59.784 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10= 2022-09-12T20:15:59.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10=', 'Content-Length': '0'} 2022-09-12T20:15:59.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d74771f7c9fdd0e-00631f937f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.789 DEBUG:boto:b'jdpiak-75obj1481000falseobj1482022-09-12T20:14:27.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:59.791 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:59.791 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:59.792 DEBUG:boto:path=/jdpiak-75/?prefix=obj148 2022-09-12T20:15:59.793 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj148 2022-09-12T20:15:59.794 DEBUG:boto:Method: GET 2022-09-12T20:15:59.795 DEBUG:boto:Path: /jdpiak-75/?prefix=obj148 2022-09-12T20:15:59.796 DEBUG:boto:Data: 2022-09-12T20:15:59.797 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.798 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.799 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.800 DEBUG:boto:Params: {} 2022-09-12T20:15:59.801 DEBUG:boto:Token: None 2022-09-12T20:15:59.802 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/ 2022-09-12T20:15:59.804 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10= 2022-09-12T20:15:59.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10=', 'Content-Length': '0'} 2022-09-12T20:15:59.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e9723992ee837e6-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.811 DEBUG:boto:b'jdpiak-75obj1481000falseobj1482022-09-12T20:14:27.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:59.813 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:15:59.814 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:15:59.815 DEBUG:boto:Method: HEAD 2022-09-12T20:15:59.816 DEBUG:boto:Path: /jdpiak-75/obj148 2022-09-12T20:15:59.817 DEBUG:boto:Data: 2022-09-12T20:15:59.817 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.818 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.819 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.820 DEBUG:boto:Params: {} 2022-09-12T20:15:59.821 DEBUG:boto:Token: None 2022-09-12T20:15:59.822 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj148 2022-09-12T20:15:59.823 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QL0oR5o9K4/dL+ANTTl3NBE/ZtQ= 2022-09-12T20:15:59.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QL0oR5o9K4/dL+ANTTl3NBE/ZtQ=', 'Content-Length': '0'} 2022-09-12T20:15:59.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7a19c20679f2027-00631f937f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.828 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:15:59.829 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:15:59.829 DEBUG:boto:Method: HEAD 2022-09-12T20:15:59.830 DEBUG:boto:Path: /jdpiak-75/obj148 2022-09-12T20:15:59.830 DEBUG:boto:Data: 2022-09-12T20:15:59.831 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.831 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.832 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.832 DEBUG:boto:Params: {} 2022-09-12T20:15:59.833 DEBUG:boto:Token: None 2022-09-12T20:15:59.833 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj148 2022-09-12T20:15:59.834 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QL0oR5o9K4/dL+ANTTl3NBE/ZtQ= 2022-09-12T20:15:59.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QL0oR5o9K4/dL+ANTTl3NBE/ZtQ=', 'Content-Length': '0'} 2022-09-12T20:15:59.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035d4466caf66dd05-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2022-09-12T20:15:59.838 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:15:59.839 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:15:59.839 DEBUG:boto:Method: GET 2022-09-12T20:15:59.840 DEBUG:boto:Path: /jdpiak-75/obj148 2022-09-12T20:15:59.841 DEBUG:boto:Data: 2022-09-12T20:15:59.842 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.842 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.843 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.844 DEBUG:boto:Params: {} 2022-09-12T20:15:59.845 DEBUG:boto:Token: None 2022-09-12T20:15:59.845 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj148 2022-09-12T20:15:59.846 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tf7DIa3HN+sA+uH9xKEzRUoOvrw= 2022-09-12T20:15:59.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tf7DIa3HN+sA+uH9xKEzRUoOvrw=', 'Content-Length': '0'} 2022-09-12T20:15:59.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a6a4a3d46b8b416-00631f937f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.850 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:15:59.851 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:15:59.852 DEBUG:boto:Method: GET 2022-09-12T20:15:59.854 DEBUG:boto:Path: /jdpiak-75/obj148 2022-09-12T20:15:59.855 DEBUG:boto:Data: 2022-09-12T20:15:59.856 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.857 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.858 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.859 DEBUG:boto:Params: {} 2022-09-12T20:15:59.860 DEBUG:boto:Token: None 2022-09-12T20:15:59.861 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj148 2022-09-12T20:15:59.862 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tf7DIa3HN+sA+uH9xKEzRUoOvrw= 2022-09-12T20:15:59.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tf7DIa3HN+sA+uH9xKEzRUoOvrw=', 'Content-Length': '0'} 2022-09-12T20:15:59.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ab6a0cd2d160e24-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.906 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:59.907 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:59.908 DEBUG:boto:path=/jdpiak-75/?prefix=obj148 2022-09-12T20:15:59.908 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj148 2022-09-12T20:15:59.909 DEBUG:boto:Method: GET 2022-09-12T20:15:59.909 DEBUG:boto:Path: /jdpiak-75/?prefix=obj148 2022-09-12T20:15:59.910 DEBUG:boto:Data: 2022-09-12T20:15:59.911 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.911 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.912 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.913 DEBUG:boto:Params: {} 2022-09-12T20:15:59.914 DEBUG:boto:Token: None 2022-09-12T20:15:59.914 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/ 2022-09-12T20:15:59.915 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10= 2022-09-12T20:15:59.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10=', 'Content-Length': '0'} 2022-09-12T20:15:59.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cffe0cdb20072e23-00631f937f-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.920 DEBUG:boto:b'jdpiak-75obj1481000falseobj1482022-09-12T20:14:27.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:59.920 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:15:59.921 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:15:59.922 DEBUG:boto:path=/jdpiak-75/?prefix=obj148 2022-09-12T20:15:59.922 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj148 2022-09-12T20:15:59.923 DEBUG:boto:Method: GET 2022-09-12T20:15:59.924 DEBUG:boto:Path: /jdpiak-75/?prefix=obj148 2022-09-12T20:15:59.924 DEBUG:boto:Data: 2022-09-12T20:15:59.925 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.926 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.926 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.927 DEBUG:boto:Params: {} 2022-09-12T20:15:59.928 DEBUG:boto:Token: None 2022-09-12T20:15:59.928 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/ 2022-09-12T20:15:59.929 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10= 2022-09-12T20:15:59.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/+3M1A4IfOEtI6NxIsudFEVlL10=', 'Content-Length': '0'} 2022-09-12T20:15:59.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e443334a5eb4d04-00631f937f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.935 DEBUG:boto:b'jdpiak-75obj1481000falseobj1482022-09-12T20:14:27.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:15:59.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-12T20:15:59.937 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:15:59.937 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:15:59.938 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:15:59.939 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:15:59.939 DEBUG:boto:Method: GET 2022-09-12T20:15:59.940 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:15:59.940 DEBUG:boto:Data: 2022-09-12T20:15:59.941 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.942 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:15:59.942 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.943 DEBUG:boto:Params: {} 2022-09-12T20:15:59.944 DEBUG:boto:Token: None 2022-09-12T20:15:59.945 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:15:59.945 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WFOpCtWJeBEj39g8vYKydTscC/w= 2022-09-12T20:15:59.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WFOpCtWJeBEj39g8vYKydTscC/w=', 'Content-Length': '0'} 2022-09-12T20:15:59.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a34a77f04ee9300e-00631f937f-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:15:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:15:59.990 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:15:59.991 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:15:59.992 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:15:59.992 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:15:59.993 DEBUG:boto:Method: GET 2022-09-12T20:15:59.994 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:15:59.995 DEBUG:boto:Data: 2022-09-12T20:15:59.995 DEBUG:boto:Headers: {} 2022-09-12T20:15:59.996 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:15:59.997 DEBUG:boto:Port: 8000 2022-09-12T20:15:59.998 DEBUG:boto:Params: {} 2022-09-12T20:15:59.999 DEBUG:boto:Token: None 2022-09-12T20:15:59.999 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:15:59 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:00.000 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WFOpCtWJeBEj39g8vYKydTscC/w= 2022-09-12T20:16:00.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:15:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WFOpCtWJeBEj39g8vYKydTscC/w=', 'Content-Length': '0'} 2022-09-12T20:16:00.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cccbdf2d864097f-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.005 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:00.005 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:00.006 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:00.007 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:00.007 DEBUG:boto:Method: HEAD 2022-09-12T20:16:00.008 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:00.009 DEBUG:boto:Data: 2022-09-12T20:16:00.009 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.010 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.011 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.011 DEBUG:boto:Params: {} 2022-09-12T20:16:00.012 DEBUG:boto:Token: None 2022-09-12T20:16:00.013 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:00.014 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xlrOk8jvbAxZVWNK7fKX1nLcS2c= 2022-09-12T20:16:00.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xlrOk8jvbAxZVWNK7fKX1nLcS2c=', 'Content-Length': '0'} 2022-09-12T20:16:00.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4cff00d0bf5c798-00631f9380-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.017 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:00.018 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:00.019 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:00.019 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:00.020 DEBUG:boto:Method: HEAD 2022-09-12T20:16:00.021 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:00.021 DEBUG:boto:Data: 2022-09-12T20:16:00.022 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.022 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.023 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.024 DEBUG:boto:Params: {} 2022-09-12T20:16:00.024 DEBUG:boto:Token: None 2022-09-12T20:16:00.025 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:00.026 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xlrOk8jvbAxZVWNK7fKX1nLcS2c= 2022-09-12T20:16:00.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xlrOk8jvbAxZVWNK7fKX1nLcS2c=', 'Content-Length': '0'} 2022-09-12T20:16:00.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087218fe33c5716c4-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-12T20:16:00.030 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:00.031 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:00.032 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:00.033 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:00.033 DEBUG:boto:Method: GET 2022-09-12T20:16:00.034 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:00.035 DEBUG:boto:Data: 2022-09-12T20:16:00.037 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.038 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.039 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.040 DEBUG:boto:Params: {} 2022-09-12T20:16:00.041 DEBUG:boto:Token: None 2022-09-12T20:16:00.042 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:00.043 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CSbZJhyZtdpjb/hBh8y5/P+qJCA= 2022-09-12T20:16:00.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CSbZJhyZtdpjb/hBh8y5/P+qJCA=', 'Content-Length': '0'} 2022-09-12T20:16:00.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bc941b724ce541b-00631f9380-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.048 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:00.049 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:00.050 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:00.051 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:00.052 DEBUG:boto:Method: GET 2022-09-12T20:16:00.053 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:00.054 DEBUG:boto:Data: 2022-09-12T20:16:00.055 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.056 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.057 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.059 DEBUG:boto:Params: {} 2022-09-12T20:16:00.060 DEBUG:boto:Token: None 2022-09-12T20:16:00.061 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:00.062 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CSbZJhyZtdpjb/hBh8y5/P+qJCA= 2022-09-12T20:16:00.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CSbZJhyZtdpjb/hBh8y5/P+qJCA=', 'Content-Length': '0'} 2022-09-12T20:16:00.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a33bf47c5d72d477-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.067 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:00.067 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:00.068 DEBUG:boto:path=/jdpiak-75/?prefix=obj155 2022-09-12T20:16:00.068 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj155 2022-09-12T20:16:00.069 DEBUG:boto:Method: GET 2022-09-12T20:16:00.070 DEBUG:boto:Path: /jdpiak-75/?prefix=obj155 2022-09-12T20:16:00.070 DEBUG:boto:Data: 2022-09-12T20:16:00.071 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.071 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.072 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.073 DEBUG:boto:Params: {} 2022-09-12T20:16:00.073 DEBUG:boto:Token: None 2022-09-12T20:16:00.074 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/ 2022-09-12T20:16:00.075 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI= 2022-09-12T20:16:00.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI=', 'Content-Length': '0'} 2022-09-12T20:16:00.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096891a682c2781a9-00631f9380-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.079 DEBUG:boto:b'jdpiak-75obj1551000falseobj1552022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:00.080 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:00.081 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:00.081 DEBUG:boto:path=/jdpiak-75/?prefix=obj155 2022-09-12T20:16:00.082 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj155 2022-09-12T20:16:00.082 DEBUG:boto:Method: GET 2022-09-12T20:16:00.083 DEBUG:boto:Path: /jdpiak-75/?prefix=obj155 2022-09-12T20:16:00.083 DEBUG:boto:Data: 2022-09-12T20:16:00.084 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.085 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.085 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.086 DEBUG:boto:Params: {} 2022-09-12T20:16:00.086 DEBUG:boto:Token: None 2022-09-12T20:16:00.087 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/ 2022-09-12T20:16:00.087 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI= 2022-09-12T20:16:00.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI=', 'Content-Length': '0'} 2022-09-12T20:16:00.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5e1974643a239c1-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.094 DEBUG:boto:b'jdpiak-75obj1551000falseobj1552022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:00.095 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:00.095 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:00.096 DEBUG:boto:Method: HEAD 2022-09-12T20:16:00.096 DEBUG:boto:Path: /jdpiak-75/obj155 2022-09-12T20:16:00.097 DEBUG:boto:Data: 2022-09-12T20:16:00.097 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.098 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.098 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.099 DEBUG:boto:Params: {} 2022-09-12T20:16:00.100 DEBUG:boto:Token: None 2022-09-12T20:16:00.100 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj155 2022-09-12T20:16:00.101 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UDeCoOnN3m+dVk86/X7EtM+1g3o= 2022-09-12T20:16:00.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UDeCoOnN3m+dVk86/X7EtM+1g3o=', 'Content-Length': '0'} 2022-09-12T20:16:00.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a02401dcb775601-00631f9380-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.104 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:00.105 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:00.105 DEBUG:boto:Method: HEAD 2022-09-12T20:16:00.106 DEBUG:boto:Path: /jdpiak-75/obj155 2022-09-12T20:16:00.106 DEBUG:boto:Data: 2022-09-12T20:16:00.107 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.107 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.108 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.108 DEBUG:boto:Params: {} 2022-09-12T20:16:00.109 DEBUG:boto:Token: None 2022-09-12T20:16:00.109 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj155 2022-09-12T20:16:00.110 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UDeCoOnN3m+dVk86/X7EtM+1g3o= 2022-09-12T20:16:00.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UDeCoOnN3m+dVk86/X7EtM+1g3o=', 'Content-Length': '0'} 2022-09-12T20:16:00.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001395ff75eb158fd9-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-12T20:16:00.116 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:00.116 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:00.117 DEBUG:boto:Method: GET 2022-09-12T20:16:00.117 DEBUG:boto:Path: /jdpiak-75/obj155 2022-09-12T20:16:00.118 DEBUG:boto:Data: 2022-09-12T20:16:00.118 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.119 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.119 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.120 DEBUG:boto:Params: {} 2022-09-12T20:16:00.120 DEBUG:boto:Token: None 2022-09-12T20:16:00.121 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj155 2022-09-12T20:16:00.122 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PeB2S+CsY6hRcccLUM103qtSKS0= 2022-09-12T20:16:00.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PeB2S+CsY6hRcccLUM103qtSKS0=', 'Content-Length': '0'} 2022-09-12T20:16:00.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d67359b28441b18e-00631f9380-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.170 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:00.171 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:00.171 DEBUG:boto:Method: GET 2022-09-12T20:16:00.172 DEBUG:boto:Path: /jdpiak-75/obj155 2022-09-12T20:16:00.173 DEBUG:boto:Data: 2022-09-12T20:16:00.173 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.174 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.174 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.175 DEBUG:boto:Params: {} 2022-09-12T20:16:00.175 DEBUG:boto:Token: None 2022-09-12T20:16:00.176 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj155 2022-09-12T20:16:00.177 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PeB2S+CsY6hRcccLUM103qtSKS0= 2022-09-12T20:16:00.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PeB2S+CsY6hRcccLUM103qtSKS0=', 'Content-Length': '0'} 2022-09-12T20:16:00.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b73c69b46190b4d1-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.180 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:00.181 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:00.181 DEBUG:boto:path=/jdpiak-75/?prefix=obj155 2022-09-12T20:16:00.182 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj155 2022-09-12T20:16:00.183 DEBUG:boto:Method: GET 2022-09-12T20:16:00.183 DEBUG:boto:Path: /jdpiak-75/?prefix=obj155 2022-09-12T20:16:00.184 DEBUG:boto:Data: 2022-09-12T20:16:00.184 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.185 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.185 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.186 DEBUG:boto:Params: {} 2022-09-12T20:16:00.186 DEBUG:boto:Token: None 2022-09-12T20:16:00.187 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/ 2022-09-12T20:16:00.188 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI= 2022-09-12T20:16:00.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI=', 'Content-Length': '0'} 2022-09-12T20:16:00.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df560057910c9703-00631f9380-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.192 DEBUG:boto:b'jdpiak-75obj1551000falseobj1552022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:00.193 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:00.193 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:00.194 DEBUG:boto:path=/jdpiak-75/?prefix=obj155 2022-09-12T20:16:00.194 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj155 2022-09-12T20:16:00.195 DEBUG:boto:Method: GET 2022-09-12T20:16:00.195 DEBUG:boto:Path: /jdpiak-75/?prefix=obj155 2022-09-12T20:16:00.196 DEBUG:boto:Data: 2022-09-12T20:16:00.197 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.197 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.198 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.199 DEBUG:boto:Params: {} 2022-09-12T20:16:00.199 DEBUG:boto:Token: None 2022-09-12T20:16:00.200 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/ 2022-09-12T20:16:00.200 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI= 2022-09-12T20:16:00.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI=', 'Content-Length': '0'} 2022-09-12T20:16:00.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd85f7fcf6a3896f-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.207 DEBUG:boto:b'jdpiak-75obj1551000falseobj1552022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:00.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:16:00.209 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:00.209 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:00.210 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:00.211 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:00.211 DEBUG:boto:Method: GET 2022-09-12T20:16:00.212 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:00.212 DEBUG:boto:Data: 2022-09-12T20:16:00.213 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.214 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.214 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.215 DEBUG:boto:Params: {} 2022-09-12T20:16:00.216 DEBUG:boto:Token: None 2022-09-12T20:16:00.216 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:00.217 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8/iwcBwMnTDKnyxAqKW9RPhWO98= 2022-09-12T20:16:00.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8/iwcBwMnTDKnyxAqKW9RPhWO98=', 'Content-Length': '0'} 2022-09-12T20:16:00.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d1843c4b3b38091-00631f9380-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.221 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:00.221 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:00.222 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:00.223 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:00.223 DEBUG:boto:Method: GET 2022-09-12T20:16:00.224 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:00.225 DEBUG:boto:Data: 2022-09-12T20:16:00.225 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.226 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.226 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.227 DEBUG:boto:Params: {} 2022-09-12T20:16:00.228 DEBUG:boto:Token: None 2022-09-12T20:16:00.228 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:00.229 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8/iwcBwMnTDKnyxAqKW9RPhWO98= 2022-09-12T20:16:00.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8/iwcBwMnTDKnyxAqKW9RPhWO98=', 'Content-Length': '0'} 2022-09-12T20:16:00.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa1fb55306f7b5b0-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.274 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:00.275 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:00.275 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:00.276 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:00.276 DEBUG:boto:Method: HEAD 2022-09-12T20:16:00.277 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:00.277 DEBUG:boto:Data: 2022-09-12T20:16:00.278 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.278 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.279 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.279 DEBUG:boto:Params: {} 2022-09-12T20:16:00.280 DEBUG:boto:Token: None 2022-09-12T20:16:00.281 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:00.281 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/S/MJT+y0pHgIghz8dxGiImrV2Y= 2022-09-12T20:16:00.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/S/MJT+y0pHgIghz8dxGiImrV2Y=', 'Content-Length': '0'} 2022-09-12T20:16:00.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003370b00e08aac61b-00631f9380-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.285 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:00.285 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:00.286 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:00.286 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:00.287 DEBUG:boto:Method: HEAD 2022-09-12T20:16:00.287 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:00.288 DEBUG:boto:Data: 2022-09-12T20:16:00.288 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.289 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.289 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.290 DEBUG:boto:Params: {} 2022-09-12T20:16:00.290 DEBUG:boto:Token: None 2022-09-12T20:16:00.291 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:00.291 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/S/MJT+y0pHgIghz8dxGiImrV2Y= 2022-09-12T20:16:00.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/S/MJT+y0pHgIghz8dxGiImrV2Y=', 'Content-Length': '0'} 2022-09-12T20:16:00.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004eb28c319d5b3220-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:16:00.295 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:00.296 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:00.297 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:00.297 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:00.298 DEBUG:boto:Method: GET 2022-09-12T20:16:00.298 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:00.299 DEBUG:boto:Data: 2022-09-12T20:16:00.299 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.300 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.300 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.301 DEBUG:boto:Params: {} 2022-09-12T20:16:00.301 DEBUG:boto:Token: None 2022-09-12T20:16:00.302 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:00.303 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8/iwcBwMnTDKnyxAqKW9RPhWO98= 2022-09-12T20:16:00.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8/iwcBwMnTDKnyxAqKW9RPhWO98=', 'Content-Length': '0'} 2022-09-12T20:16:00.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035db519a4c2860b3-00631f9380-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.306 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:00.307 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:00.307 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:00.308 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:00.308 DEBUG:boto:Method: GET 2022-09-12T20:16:00.309 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:00.309 DEBUG:boto:Data: 2022-09-12T20:16:00.310 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.310 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.311 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.311 DEBUG:boto:Params: {} 2022-09-12T20:16:00.312 DEBUG:boto:Token: None 2022-09-12T20:16:00.313 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:00.313 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8/iwcBwMnTDKnyxAqKW9RPhWO98= 2022-09-12T20:16:00.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8/iwcBwMnTDKnyxAqKW9RPhWO98=', 'Content-Length': '0'} 2022-09-12T20:16:00.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006516a19ecd40833f-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.317 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:00.318 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:00.319 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:16:00.319 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:16:00.320 DEBUG:boto:Method: GET 2022-09-12T20:16:00.320 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:16:00.321 DEBUG:boto:Data: 2022-09-12T20:16:00.321 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.322 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.322 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.323 DEBUG:boto:Params: {} 2022-09-12T20:16:00.323 DEBUG:boto:Token: None 2022-09-12T20:16:00.324 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/ 2022-09-12T20:16:00.325 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI= 2022-09-12T20:16:00.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI=', 'Content-Length': '0'} 2022-09-12T20:16:00.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ef8e697b1f385e4-00631f9380-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.329 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:00.330 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:00.330 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:00.331 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:16:00.331 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:16:00.332 DEBUG:boto:Method: GET 2022-09-12T20:16:00.332 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:16:00.333 DEBUG:boto:Data: 2022-09-12T20:16:00.334 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.334 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.335 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.335 DEBUG:boto:Params: {} 2022-09-12T20:16:00.336 DEBUG:boto:Token: None 2022-09-12T20:16:00.336 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/ 2022-09-12T20:16:00.337 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI= 2022-09-12T20:16:00.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI=', 'Content-Length': '0'} 2022-09-12T20:16:00.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057853b19f9c2c3e2-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.343 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:00.344 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:00.345 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:00.345 DEBUG:boto:Method: HEAD 2022-09-12T20:16:00.346 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:16:00.346 DEBUG:boto:Data: 2022-09-12T20:16:00.347 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.347 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.348 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.348 DEBUG:boto:Params: {} 2022-09-12T20:16:00.349 DEBUG:boto:Token: None 2022-09-12T20:16:00.349 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj17 2022-09-12T20:16:00.350 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sZCEyjKeASQ9SiwiwjOIAShAx9M= 2022-09-12T20:16:00.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sZCEyjKeASQ9SiwiwjOIAShAx9M=', 'Content-Length': '0'} 2022-09-12T20:16:00.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b218dd32b4638bf7-00631f9380-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.353 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:00.353 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:00.354 DEBUG:boto:Method: HEAD 2022-09-12T20:16:00.355 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:16:00.355 DEBUG:boto:Data: 2022-09-12T20:16:00.355 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.356 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.356 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.357 DEBUG:boto:Params: {} 2022-09-12T20:16:00.357 DEBUG:boto:Token: None 2022-09-12T20:16:00.358 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj17 2022-09-12T20:16:00.359 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sZCEyjKeASQ9SiwiwjOIAShAx9M= 2022-09-12T20:16:00.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sZCEyjKeASQ9SiwiwjOIAShAx9M=', 'Content-Length': '0'} 2022-09-12T20:16:00.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d91dfa2199ace160-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:16:00.363 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:00.363 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:00.364 DEBUG:boto:Method: GET 2022-09-12T20:16:00.364 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:16:00.365 DEBUG:boto:Data: 2022-09-12T20:16:00.365 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.366 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.366 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.367 DEBUG:boto:Params: {} 2022-09-12T20:16:00.367 DEBUG:boto:Token: None 2022-09-12T20:16:00.368 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj17 2022-09-12T20:16:00.369 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RYuTO/Meag4b/8+FobMIBjN6R9o= 2022-09-12T20:16:00.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RYuTO/Meag4b/8+FobMIBjN6R9o=', 'Content-Length': '0'} 2022-09-12T20:16:00.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c670961ff9612149-00631f9380-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.373 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:00.373 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:00.374 DEBUG:boto:Method: GET 2022-09-12T20:16:00.374 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:16:00.375 DEBUG:boto:Data: 2022-09-12T20:16:00.375 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.376 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.376 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.377 DEBUG:boto:Params: {} 2022-09-12T20:16:00.377 DEBUG:boto:Token: None 2022-09-12T20:16:00.378 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj17 2022-09-12T20:16:00.379 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RYuTO/Meag4b/8+FobMIBjN6R9o= 2022-09-12T20:16:00.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RYuTO/Meag4b/8+FobMIBjN6R9o=', 'Content-Length': '0'} 2022-09-12T20:16:00.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027d4b17a5a4d676a-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.430 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:00.431 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:00.431 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:16:00.432 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:16:00.432 DEBUG:boto:Method: GET 2022-09-12T20:16:00.433 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:16:00.433 DEBUG:boto:Data: 2022-09-12T20:16:00.434 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.435 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.435 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.436 DEBUG:boto:Params: {} 2022-09-12T20:16:00.436 DEBUG:boto:Token: None 2022-09-12T20:16:00.437 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/ 2022-09-12T20:16:00.438 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI= 2022-09-12T20:16:00.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI=', 'Content-Length': '0'} 2022-09-12T20:16:00.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089e04bcca1a1e794-00631f9380-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.442 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:00.443 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:00.443 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:00.444 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:16:00.445 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:16:00.445 DEBUG:boto:Method: GET 2022-09-12T20:16:00.446 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:16:00.446 DEBUG:boto:Data: 2022-09-12T20:16:00.447 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.447 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.448 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.448 DEBUG:boto:Params: {} 2022-09-12T20:16:00.449 DEBUG:boto:Token: None 2022-09-12T20:16:00.450 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/ 2022-09-12T20:16:00.450 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI= 2022-09-12T20:16:00.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI=', 'Content-Length': '0'} 2022-09-12T20:16:00.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000921e9be966a3ef6b-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.456 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:00.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2022-09-12T20:16:00.458 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:00.458 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:00.459 DEBUG:boto:path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:00.459 DEBUG:boto:auth_path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:00.460 DEBUG:boto:Method: GET 2022-09-12T20:16:00.460 DEBUG:boto:Path: /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:00.461 DEBUG:boto:Data: 2022-09-12T20:16:00.461 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.462 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.462 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.463 DEBUG:boto:Params: {} 2022-09-12T20:16:00.463 DEBUG:boto:Token: None 2022-09-12T20:16:00.464 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:00.465 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ajigEdud7XWqz5rORWUXl6/ySJQ= 2022-09-12T20:16:00.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ajigEdud7XWqz5rORWUXl6/ySJQ=', 'Content-Length': '0'} 2022-09-12T20:16:00.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce99d467f02c0278-00631f9380-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.469 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:00.469 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:00.470 DEBUG:boto:path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:00.470 DEBUG:boto:auth_path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:00.471 DEBUG:boto:Method: GET 2022-09-12T20:16:00.471 DEBUG:boto:Path: /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:00.472 DEBUG:boto:Data: 2022-09-12T20:16:00.472 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.473 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.473 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.474 DEBUG:boto:Params: {} 2022-09-12T20:16:00.474 DEBUG:boto:Token: None 2022-09-12T20:16:00.475 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:00.476 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ajigEdud7XWqz5rORWUXl6/ySJQ= 2022-09-12T20:16:00.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ajigEdud7XWqz5rORWUXl6/ySJQ=', 'Content-Length': '0'} 2022-09-12T20:16:00.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000936bbfe340594505-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.480 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:00.480 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:00.481 DEBUG:boto:path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:00.481 DEBUG:boto:auth_path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:00.482 DEBUG:boto:Method: HEAD 2022-09-12T20:16:00.482 DEBUG:boto:Path: /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:00.483 DEBUG:boto:Data: 2022-09-12T20:16:00.483 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.483 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.484 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.484 DEBUG:boto:Params: {} 2022-09-12T20:16:00.485 DEBUG:boto:Token: None 2022-09-12T20:16:00.486 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:00.486 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vF3FBr7nO6SwFpvWZVroup5xLg8= 2022-09-12T20:16:00.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vF3FBr7nO6SwFpvWZVroup5xLg8=', 'Content-Length': '0'} 2022-09-12T20:16:00.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b22b414822eb7ac1-00631f9380-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.489 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:00.490 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:00.491 DEBUG:boto:path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:00.491 DEBUG:boto:auth_path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:00.491 DEBUG:boto:Method: HEAD 2022-09-12T20:16:00.492 DEBUG:boto:Path: /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:00.492 DEBUG:boto:Data: 2022-09-12T20:16:00.493 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.493 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.494 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.494 DEBUG:boto:Params: {} 2022-09-12T20:16:00.495 DEBUG:boto:Token: None 2022-09-12T20:16:00.495 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:00.496 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vF3FBr7nO6SwFpvWZVroup5xLg8= 2022-09-12T20:16:00.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vF3FBr7nO6SwFpvWZVroup5xLg8=', 'Content-Length': '0'} 2022-09-12T20:16:00.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb91cf29552aec15-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2022-09-12T20:16:00.500 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:00.500 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:00.501 DEBUG:boto:path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:00.501 DEBUG:boto:auth_path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:00.502 DEBUG:boto:Method: GET 2022-09-12T20:16:00.502 DEBUG:boto:Path: /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:00.503 DEBUG:boto:Data: 2022-09-12T20:16:00.503 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.504 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.504 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.505 DEBUG:boto:Params: {} 2022-09-12T20:16:00.506 DEBUG:boto:Token: None 2022-09-12T20:16:00.506 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:00.507 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ajigEdud7XWqz5rORWUXl6/ySJQ= 2022-09-12T20:16:00.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ajigEdud7XWqz5rORWUXl6/ySJQ=', 'Content-Length': '0'} 2022-09-12T20:16:00.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fca6c3c6de6741d0-00631f9380-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.510 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:00.511 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:00.512 DEBUG:boto:path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:00.512 DEBUG:boto:auth_path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:00.513 DEBUG:boto:Method: GET 2022-09-12T20:16:00.513 DEBUG:boto:Path: /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:00.514 DEBUG:boto:Data: 2022-09-12T20:16:00.514 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.515 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.515 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.516 DEBUG:boto:Params: {} 2022-09-12T20:16:00.516 DEBUG:boto:Token: None 2022-09-12T20:16:00.517 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:00.518 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ajigEdud7XWqz5rORWUXl6/ySJQ= 2022-09-12T20:16:00.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ajigEdud7XWqz5rORWUXl6/ySJQ=', 'Content-Length': '0'} 2022-09-12T20:16:00.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003da7ad4d20278371-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.562 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:00.563 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:00.563 DEBUG:boto:path=/jdpiak-75/?prefix=obj185 2022-09-12T20:16:00.564 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj185 2022-09-12T20:16:00.564 DEBUG:boto:Method: GET 2022-09-12T20:16:00.565 DEBUG:boto:Path: /jdpiak-75/?prefix=obj185 2022-09-12T20:16:00.566 DEBUG:boto:Data: 2022-09-12T20:16:00.566 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.567 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.567 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.568 DEBUG:boto:Params: {} 2022-09-12T20:16:00.568 DEBUG:boto:Token: None 2022-09-12T20:16:00.569 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/ 2022-09-12T20:16:00.570 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI= 2022-09-12T20:16:00.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI=', 'Content-Length': '0'} 2022-09-12T20:16:00.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cd69a3486f3ccae-00631f9380-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.574 DEBUG:boto:b'jdpiak-75obj1851000falseobj1852022-09-12T20:14:27.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:00.575 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:00.575 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:00.576 DEBUG:boto:path=/jdpiak-75/?prefix=obj185 2022-09-12T20:16:00.576 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj185 2022-09-12T20:16:00.577 DEBUG:boto:Method: GET 2022-09-12T20:16:00.577 DEBUG:boto:Path: /jdpiak-75/?prefix=obj185 2022-09-12T20:16:00.578 DEBUG:boto:Data: 2022-09-12T20:16:00.578 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.579 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.579 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.580 DEBUG:boto:Params: {} 2022-09-12T20:16:00.580 DEBUG:boto:Token: None 2022-09-12T20:16:00.581 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/ 2022-09-12T20:16:00.582 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI= 2022-09-12T20:16:00.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI=', 'Content-Length': '0'} 2022-09-12T20:16:00.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e5f0442a6a74e3c-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.588 DEBUG:boto:b'jdpiak-75obj1851000falseobj1852022-09-12T20:14:27.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:00.589 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:00.590 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:00.590 DEBUG:boto:Method: HEAD 2022-09-12T20:16:00.591 DEBUG:boto:Path: /jdpiak-75/obj185 2022-09-12T20:16:00.591 DEBUG:boto:Data: 2022-09-12T20:16:00.592 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.592 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.593 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.593 DEBUG:boto:Params: {} 2022-09-12T20:16:00.594 DEBUG:boto:Token: None 2022-09-12T20:16:00.594 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj185 2022-09-12T20:16:00.595 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xkV86vX5V1AtVTnJl64KjjW2prc= 2022-09-12T20:16:00.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xkV86vX5V1AtVTnJl64KjjW2prc=', 'Content-Length': '0'} 2022-09-12T20:16:00.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d33e1617d546a49-00631f9380-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.598 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:00.599 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:00.599 DEBUG:boto:Method: HEAD 2022-09-12T20:16:00.600 DEBUG:boto:Path: /jdpiak-75/obj185 2022-09-12T20:16:00.601 DEBUG:boto:Data: 2022-09-12T20:16:00.601 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.602 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.602 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.603 DEBUG:boto:Params: {} 2022-09-12T20:16:00.603 DEBUG:boto:Token: None 2022-09-12T20:16:00.604 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj185 2022-09-12T20:16:00.604 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xkV86vX5V1AtVTnJl64KjjW2prc= 2022-09-12T20:16:00.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xkV86vX5V1AtVTnJl64KjjW2prc=', 'Content-Length': '0'} 2022-09-12T20:16:00.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007957596646010fd6-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2022-09-12T20:16:00.609 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:00.609 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:00.610 DEBUG:boto:Method: GET 2022-09-12T20:16:00.611 DEBUG:boto:Path: /jdpiak-75/obj185 2022-09-12T20:16:00.611 DEBUG:boto:Data: 2022-09-12T20:16:00.612 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.612 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.613 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.613 DEBUG:boto:Params: {} 2022-09-12T20:16:00.614 DEBUG:boto:Token: None 2022-09-12T20:16:00.615 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj185 2022-09-12T20:16:00.615 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vOL/5G+qbbpzmDePDo5Y9MEqWn4= 2022-09-12T20:16:00.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vOL/5G+qbbpzmDePDo5Y9MEqWn4=', 'Content-Length': '0'} 2022-09-12T20:16:00.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003bd3e93ebb9d69e-00631f9380-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.619 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:00.619 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:00.620 DEBUG:boto:Method: GET 2022-09-12T20:16:00.620 DEBUG:boto:Path: /jdpiak-75/obj185 2022-09-12T20:16:00.621 DEBUG:boto:Data: 2022-09-12T20:16:00.622 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.622 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.623 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.623 DEBUG:boto:Params: {} 2022-09-12T20:16:00.624 DEBUG:boto:Token: None 2022-09-12T20:16:00.624 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj185 2022-09-12T20:16:00.625 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vOL/5G+qbbpzmDePDo5Y9MEqWn4= 2022-09-12T20:16:00.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vOL/5G+qbbpzmDePDo5Y9MEqWn4=', 'Content-Length': '0'} 2022-09-12T20:16:00.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050f6f74dc7f1c031-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.629 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:00.630 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:00.630 DEBUG:boto:path=/jdpiak-75/?prefix=obj185 2022-09-12T20:16:00.631 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj185 2022-09-12T20:16:00.631 DEBUG:boto:Method: GET 2022-09-12T20:16:00.632 DEBUG:boto:Path: /jdpiak-75/?prefix=obj185 2022-09-12T20:16:00.632 DEBUG:boto:Data: 2022-09-12T20:16:00.633 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.633 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.634 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.634 DEBUG:boto:Params: {} 2022-09-12T20:16:00.645 DEBUG:boto:Token: None 2022-09-12T20:16:00.646 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/ 2022-09-12T20:16:00.647 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI= 2022-09-12T20:16:00.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI=', 'Content-Length': '0'} 2022-09-12T20:16:00.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e33af17ad89f381f-00631f9380-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.694 DEBUG:boto:b'jdpiak-75obj1851000falseobj1852022-09-12T20:14:27.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:00.695 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:00.696 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:00.696 DEBUG:boto:path=/jdpiak-75/?prefix=obj185 2022-09-12T20:16:00.697 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj185 2022-09-12T20:16:00.697 DEBUG:boto:Method: GET 2022-09-12T20:16:00.698 DEBUG:boto:Path: /jdpiak-75/?prefix=obj185 2022-09-12T20:16:00.699 DEBUG:boto:Data: 2022-09-12T20:16:00.699 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.700 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.700 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.701 DEBUG:boto:Params: {} 2022-09-12T20:16:00.701 DEBUG:boto:Token: None 2022-09-12T20:16:00.702 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/ 2022-09-12T20:16:00.703 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI= 2022-09-12T20:16:00.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI=', 'Content-Length': '0'} 2022-09-12T20:16:00.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005db09c5d75127239-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.709 DEBUG:boto:b'jdpiak-75obj1851000falseobj1852022-09-12T20:14:27.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:00.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:16:00.710 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:00.711 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:00.712 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:00.712 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:00.713 DEBUG:boto:Method: GET 2022-09-12T20:16:00.714 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:00.714 DEBUG:boto:Data: 2022-09-12T20:16:00.715 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.715 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.716 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.716 DEBUG:boto:Params: {} 2022-09-12T20:16:00.717 DEBUG:boto:Token: None 2022-09-12T20:16:00.718 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:00.718 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZqvcWIWO7RAvMvnScOQT6Qcf5Xw= 2022-09-12T20:16:00.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZqvcWIWO7RAvMvnScOQT6Qcf5Xw=', 'Content-Length': '0'} 2022-09-12T20:16:00.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009886fe61ab341653-00631f9380-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.722 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:00.722 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:00.723 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:00.724 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:00.724 DEBUG:boto:Method: GET 2022-09-12T20:16:00.725 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:00.725 DEBUG:boto:Data: 2022-09-12T20:16:00.726 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.726 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.727 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.728 DEBUG:boto:Params: {} 2022-09-12T20:16:00.728 DEBUG:boto:Token: None 2022-09-12T20:16:00.729 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:00.730 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZqvcWIWO7RAvMvnScOQT6Qcf5Xw= 2022-09-12T20:16:00.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZqvcWIWO7RAvMvnScOQT6Qcf5Xw=', 'Content-Length': '0'} 2022-09-12T20:16:00.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d3cc26bab0de745-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.778 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:00.779 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:00.780 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:00.780 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:00.781 DEBUG:boto:Method: HEAD 2022-09-12T20:16:00.781 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:00.782 DEBUG:boto:Data: 2022-09-12T20:16:00.782 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.783 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.783 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.784 DEBUG:boto:Params: {} 2022-09-12T20:16:00.784 DEBUG:boto:Token: None 2022-09-12T20:16:00.785 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:00.786 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nhl/QyZftCvr0hU8dw+kX9slbnw= 2022-09-12T20:16:00.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nhl/QyZftCvr0hU8dw+kX9slbnw=', 'Content-Length': '0'} 2022-09-12T20:16:00.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043920004be379ab4-00631f9380-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.789 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:00.790 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:00.791 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:00.791 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:00.792 DEBUG:boto:Method: HEAD 2022-09-12T20:16:00.792 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:00.793 DEBUG:boto:Data: 2022-09-12T20:16:00.793 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.794 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.794 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.795 DEBUG:boto:Params: {} 2022-09-12T20:16:00.796 DEBUG:boto:Token: None 2022-09-12T20:16:00.797 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:00.797 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nhl/QyZftCvr0hU8dw+kX9slbnw= 2022-09-12T20:16:00.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nhl/QyZftCvr0hU8dw+kX9slbnw=', 'Content-Length': '0'} 2022-09-12T20:16:00.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e29cab7dad28ad7-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:16:00.801 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:00.802 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:00.802 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:00.803 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:00.803 DEBUG:boto:Method: GET 2022-09-12T20:16:00.804 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:00.805 DEBUG:boto:Data: 2022-09-12T20:16:00.805 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.806 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.806 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.807 DEBUG:boto:Params: {} 2022-09-12T20:16:00.807 DEBUG:boto:Token: None 2022-09-12T20:16:00.808 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:00.809 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZqvcWIWO7RAvMvnScOQT6Qcf5Xw= 2022-09-12T20:16:00.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZqvcWIWO7RAvMvnScOQT6Qcf5Xw=', 'Content-Length': '0'} 2022-09-12T20:16:00.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051db1ade4816fa60-00631f9380-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.812 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:00.813 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:00.813 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:00.814 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:00.815 DEBUG:boto:Method: GET 2022-09-12T20:16:00.815 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:00.816 DEBUG:boto:Data: 2022-09-12T20:16:00.816 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.817 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.817 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.818 DEBUG:boto:Params: {} 2022-09-12T20:16:00.819 DEBUG:boto:Token: None 2022-09-12T20:16:00.819 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:00.820 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZqvcWIWO7RAvMvnScOQT6Qcf5Xw= 2022-09-12T20:16:00.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZqvcWIWO7RAvMvnScOQT6Qcf5Xw=', 'Content-Length': '0'} 2022-09-12T20:16:00.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007681e929bbdfa5f1-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.824 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:00.824 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:00.825 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:16:00.825 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:16:00.826 DEBUG:boto:Method: GET 2022-09-12T20:16:00.826 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:16:00.827 DEBUG:boto:Data: 2022-09-12T20:16:00.827 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.828 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.828 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.829 DEBUG:boto:Params: {} 2022-09-12T20:16:00.829 DEBUG:boto:Token: None 2022-09-12T20:16:00.830 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/ 2022-09-12T20:16:00.831 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI= 2022-09-12T20:16:00.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI=', 'Content-Length': '0'} 2022-09-12T20:16:00.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7c37938058c720d-00631f9380-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.834 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:00.835 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:00.836 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:00.836 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:16:00.837 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:16:00.837 DEBUG:boto:Method: GET 2022-09-12T20:16:00.838 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:16:00.838 DEBUG:boto:Data: 2022-09-12T20:16:00.839 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.839 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.840 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.840 DEBUG:boto:Params: {} 2022-09-12T20:16:00.841 DEBUG:boto:Token: None 2022-09-12T20:16:00.842 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/ 2022-09-12T20:16:00.842 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI= 2022-09-12T20:16:00.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI=', 'Content-Length': '0'} 2022-09-12T20:16:00.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a74cccaca8f22828-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.848 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:00.849 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:00.850 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:00.850 DEBUG:boto:Method: HEAD 2022-09-12T20:16:00.851 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:16:00.851 DEBUG:boto:Data: 2022-09-12T20:16:00.852 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.852 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.853 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.853 DEBUG:boto:Params: {} 2022-09-12T20:16:00.854 DEBUG:boto:Token: None 2022-09-12T20:16:00.854 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj19 2022-09-12T20:16:00.855 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MJt3HI+zom0fId7SFjpZIoKdVgE= 2022-09-12T20:16:00.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MJt3HI+zom0fId7SFjpZIoKdVgE=', 'Content-Length': '0'} 2022-09-12T20:16:00.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031340b305ac5c349-00631f9380-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.858 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:00.859 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:00.859 DEBUG:boto:Method: HEAD 2022-09-12T20:16:00.860 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:16:00.860 DEBUG:boto:Data: 2022-09-12T20:16:00.861 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.861 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.862 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.862 DEBUG:boto:Params: {} 2022-09-12T20:16:00.863 DEBUG:boto:Token: None 2022-09-12T20:16:00.863 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj19 2022-09-12T20:16:00.864 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MJt3HI+zom0fId7SFjpZIoKdVgE= 2022-09-12T20:16:00.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MJt3HI+zom0fId7SFjpZIoKdVgE=', 'Content-Length': '0'} 2022-09-12T20:16:00.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fee88471ab00768f-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:16:00.868 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:00.869 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:00.869 DEBUG:boto:Method: GET 2022-09-12T20:16:00.870 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:16:00.871 DEBUG:boto:Data: 2022-09-12T20:16:00.871 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.872 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.872 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.873 DEBUG:boto:Params: {} 2022-09-12T20:16:00.873 DEBUG:boto:Token: None 2022-09-12T20:16:00.874 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj19 2022-09-12T20:16:00.874 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:49Z5VgqKy1VUEx//oUhmcPnhSeM= 2022-09-12T20:16:00.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:49Z5VgqKy1VUEx//oUhmcPnhSeM=', 'Content-Length': '0'} 2022-09-12T20:16:00.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002253207875a57130-00631f9380-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.878 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:00.879 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:00.879 DEBUG:boto:Method: GET 2022-09-12T20:16:00.880 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:16:00.880 DEBUG:boto:Data: 2022-09-12T20:16:00.881 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.881 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.882 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.882 DEBUG:boto:Params: {} 2022-09-12T20:16:00.883 DEBUG:boto:Token: None 2022-09-12T20:16:00.884 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj19 2022-09-12T20:16:00.884 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:49Z5VgqKy1VUEx//oUhmcPnhSeM= 2022-09-12T20:16:00.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:49Z5VgqKy1VUEx//oUhmcPnhSeM=', 'Content-Length': '0'} 2022-09-12T20:16:00.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e87927b2b25991b0-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.934 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:00.935 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:00.936 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:16:00.936 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:16:00.937 DEBUG:boto:Method: GET 2022-09-12T20:16:00.937 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:16:00.938 DEBUG:boto:Data: 2022-09-12T20:16:00.938 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.939 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.939 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.940 DEBUG:boto:Params: {} 2022-09-12T20:16:00.941 DEBUG:boto:Token: None 2022-09-12T20:16:00.941 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/ 2022-09-12T20:16:00.942 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI= 2022-09-12T20:16:00.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI=', 'Content-Length': '0'} 2022-09-12T20:16:00.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c19b5352f28ca46c-00631f9380-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.946 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:00.947 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:00.948 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:00.948 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:16:00.949 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:16:00.949 DEBUG:boto:Method: GET 2022-09-12T20:16:00.950 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:16:00.950 DEBUG:boto:Data: 2022-09-12T20:16:00.951 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.951 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.952 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.952 DEBUG:boto:Params: {} 2022-09-12T20:16:00.953 DEBUG:boto:Token: None 2022-09-12T20:16:00.954 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/ 2022-09-12T20:16:00.954 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI= 2022-09-12T20:16:00.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xZBY+DHS2APL2WP5tw/TK8kPmoI=', 'Content-Length': '0'} 2022-09-12T20:16:00.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003abaf566a59f645-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.960 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:00.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:16:00.961 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:00.962 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:00.962 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:00.963 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:00.964 DEBUG:boto:Method: GET 2022-09-12T20:16:00.964 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:00.965 DEBUG:boto:Data: 2022-09-12T20:16:00.965 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.966 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.966 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.967 DEBUG:boto:Params: {} 2022-09-12T20:16:00.967 DEBUG:boto:Token: None 2022-09-12T20:16:00.968 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:00.968 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aeLHMVSPWuNgAUXPL8SbLw/ncXk= 2022-09-12T20:16:00.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aeLHMVSPWuNgAUXPL8SbLw/ncXk=', 'Content-Length': '0'} 2022-09-12T20:16:00.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9c2e10a8b80cbfa-00631f9380-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.972 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:00.973 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:00.973 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:00.974 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:00.974 DEBUG:boto:Method: GET 2022-09-12T20:16:00.975 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:00.975 DEBUG:boto:Data: 2022-09-12T20:16:00.976 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.976 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.977 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.977 DEBUG:boto:Params: {} 2022-09-12T20:16:00.978 DEBUG:boto:Token: None 2022-09-12T20:16:00.978 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:00.979 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aeLHMVSPWuNgAUXPL8SbLw/ncXk= 2022-09-12T20:16:00.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aeLHMVSPWuNgAUXPL8SbLw/ncXk=', 'Content-Length': '0'} 2022-09-12T20:16:00.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d31f4f038c7f0a8e-00631f9380-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.982 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:00.983 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:00.983 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:00.984 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:00.984 DEBUG:boto:Method: HEAD 2022-09-12T20:16:00.985 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:00.985 DEBUG:boto:Data: 2022-09-12T20:16:00.986 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.986 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:00.987 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.987 DEBUG:boto:Params: {} 2022-09-12T20:16:00.988 DEBUG:boto:Token: None 2022-09-12T20:16:00.988 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:00.989 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2qLNPP8c1WLOrMsu07e5hiofd1U= 2022-09-12T20:16:00.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2qLNPP8c1WLOrMsu07e5hiofd1U=', 'Content-Length': '0'} 2022-09-12T20:16:00.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080c32e9e1948a34a-00631f9380-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:00.992 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:00.993 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:00.993 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:00.994 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:00.994 DEBUG:boto:Method: HEAD 2022-09-12T20:16:00.995 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:00.995 DEBUG:boto:Data: 2022-09-12T20:16:00.996 DEBUG:boto:Headers: {} 2022-09-12T20:16:00.996 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:00.997 DEBUG:boto:Port: 8000 2022-09-12T20:16:00.997 DEBUG:boto:Params: {} 2022-09-12T20:16:00.998 DEBUG:boto:Token: None 2022-09-12T20:16:00.998 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:00 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:00.999 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2qLNPP8c1WLOrMsu07e5hiofd1U= 2022-09-12T20:16:01.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2qLNPP8c1WLOrMsu07e5hiofd1U=', 'Content-Length': '0'} 2022-09-12T20:16:01.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e2d168cab93c0e5-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:16:01.003 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:01.004 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:01.004 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:01.005 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:01.006 DEBUG:boto:Method: GET 2022-09-12T20:16:01.006 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:01.007 DEBUG:boto:Data: 2022-09-12T20:16:01.007 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.008 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.008 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.009 DEBUG:boto:Params: {} 2022-09-12T20:16:01.009 DEBUG:boto:Token: None 2022-09-12T20:16:01.010 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:01.010 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F81uJjyeybByohOV0bq1fvki0/s= 2022-09-12T20:16:01.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F81uJjyeybByohOV0bq1fvki0/s=', 'Content-Length': '0'} 2022-09-12T20:16:01.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026991362fd2bee81-00631f9381-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.014 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:01.015 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:01.015 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:01.016 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:01.016 DEBUG:boto:Method: GET 2022-09-12T20:16:01.017 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:01.017 DEBUG:boto:Data: 2022-09-12T20:16:01.018 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.018 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.019 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.019 DEBUG:boto:Params: {} 2022-09-12T20:16:01.020 DEBUG:boto:Token: None 2022-09-12T20:16:01.020 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:01.021 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F81uJjyeybByohOV0bq1fvki0/s= 2022-09-12T20:16:01.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F81uJjyeybByohOV0bq1fvki0/s=', 'Content-Length': '0'} 2022-09-12T20:16:01.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006318ee37671caec9-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.066 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:01.067 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:01.067 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:16:01.068 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:16:01.069 DEBUG:boto:Method: GET 2022-09-12T20:16:01.070 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:16:01.071 DEBUG:boto:Data: 2022-09-12T20:16:01.072 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.073 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.074 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.074 DEBUG:boto:Params: {} 2022-09-12T20:16:01.075 DEBUG:boto:Token: None 2022-09-12T20:16:01.076 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/ 2022-09-12T20:16:01.076 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A= 2022-09-12T20:16:01.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A=', 'Content-Length': '0'} 2022-09-12T20:16:01.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd30ed293984a057-00631f9381-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.080 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:01.081 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:01.082 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:01.082 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:16:01.083 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:16:01.083 DEBUG:boto:Method: GET 2022-09-12T20:16:01.084 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:16:01.085 DEBUG:boto:Data: 2022-09-12T20:16:01.086 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.086 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.087 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.087 DEBUG:boto:Params: {} 2022-09-12T20:16:01.088 DEBUG:boto:Token: None 2022-09-12T20:16:01.088 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/ 2022-09-12T20:16:01.089 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A= 2022-09-12T20:16:01.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A=', 'Content-Length': '0'} 2022-09-12T20:16:01.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4654073554336a6-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.095 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:01.096 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:01.097 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:01.098 DEBUG:boto:Method: HEAD 2022-09-12T20:16:01.098 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:16:01.099 DEBUG:boto:Data: 2022-09-12T20:16:01.099 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.100 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.100 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.101 DEBUG:boto:Params: {} 2022-09-12T20:16:01.101 DEBUG:boto:Token: None 2022-09-12T20:16:01.102 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj22 2022-09-12T20:16:01.102 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lOAidEm6jb9ZhJzM+U4YcQPvE/0= 2022-09-12T20:16:01.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lOAidEm6jb9ZhJzM+U4YcQPvE/0=', 'Content-Length': '0'} 2022-09-12T20:16:01.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000085dd300ee18cb85-00631f9381-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.106 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:01.106 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:01.107 DEBUG:boto:Method: HEAD 2022-09-12T20:16:01.107 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:16:01.108 DEBUG:boto:Data: 2022-09-12T20:16:01.108 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.109 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.109 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.110 DEBUG:boto:Params: {} 2022-09-12T20:16:01.110 DEBUG:boto:Token: None 2022-09-12T20:16:01.111 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj22 2022-09-12T20:16:01.111 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lOAidEm6jb9ZhJzM+U4YcQPvE/0= 2022-09-12T20:16:01.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lOAidEm6jb9ZhJzM+U4YcQPvE/0=', 'Content-Length': '0'} 2022-09-12T20:16:01.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009251a36a123adc44-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:16:01.115 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:01.116 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:01.117 DEBUG:boto:Method: GET 2022-09-12T20:16:01.117 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:16:01.118 DEBUG:boto:Data: 2022-09-12T20:16:01.118 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.119 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.119 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.120 DEBUG:boto:Params: {} 2022-09-12T20:16:01.120 DEBUG:boto:Token: None 2022-09-12T20:16:01.121 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj22 2022-09-12T20:16:01.122 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:beBePugRDXA61ole84eoAdrHZQ8= 2022-09-12T20:16:01.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:beBePugRDXA61ole84eoAdrHZQ8=', 'Content-Length': '0'} 2022-09-12T20:16:01.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057c317e733a1c009-00631f9381-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.166 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:01.167 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:01.167 DEBUG:boto:Method: GET 2022-09-12T20:16:01.168 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:16:01.169 DEBUG:boto:Data: 2022-09-12T20:16:01.170 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.170 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.171 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.171 DEBUG:boto:Params: {} 2022-09-12T20:16:01.172 DEBUG:boto:Token: None 2022-09-12T20:16:01.173 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj22 2022-09-12T20:16:01.173 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:beBePugRDXA61ole84eoAdrHZQ8= 2022-09-12T20:16:01.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:beBePugRDXA61ole84eoAdrHZQ8=', 'Content-Length': '0'} 2022-09-12T20:16:01.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe21a338ac433049-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.178 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:01.178 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:01.179 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:16:01.179 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:16:01.180 DEBUG:boto:Method: GET 2022-09-12T20:16:01.180 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:16:01.181 DEBUG:boto:Data: 2022-09-12T20:16:01.181 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.182 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.183 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.183 DEBUG:boto:Params: {} 2022-09-12T20:16:01.184 DEBUG:boto:Token: None 2022-09-12T20:16:01.184 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/ 2022-09-12T20:16:01.185 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A= 2022-09-12T20:16:01.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A=', 'Content-Length': '0'} 2022-09-12T20:16:01.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c02e68404cc6381d-00631f9381-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.189 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:01.190 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:01.190 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:01.191 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:16:01.191 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:16:01.192 DEBUG:boto:Method: GET 2022-09-12T20:16:01.192 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:16:01.193 DEBUG:boto:Data: 2022-09-12T20:16:01.193 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.194 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.194 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.195 DEBUG:boto:Params: {} 2022-09-12T20:16:01.196 DEBUG:boto:Token: None 2022-09-12T20:16:01.196 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/ 2022-09-12T20:16:01.197 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A= 2022-09-12T20:16:01.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A=', 'Content-Length': '0'} 2022-09-12T20:16:01.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092f966ed1bc569ce-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.202 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:01.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:16:01.204 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:01.204 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:01.205 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:01.205 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:01.206 DEBUG:boto:Method: GET 2022-09-12T20:16:01.206 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:01.207 DEBUG:boto:Data: 2022-09-12T20:16:01.207 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.208 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.208 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.209 DEBUG:boto:Params: {} 2022-09-12T20:16:01.210 DEBUG:boto:Token: None 2022-09-12T20:16:01.210 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:01.211 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:O2y74qx0k4zGlWHU/YZK8lR50z4= 2022-09-12T20:16:01.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:O2y74qx0k4zGlWHU/YZK8lR50z4=', 'Content-Length': '0'} 2022-09-12T20:16:01.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e18821435a629fd6-00631f9381-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.215 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:01.215 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:01.216 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:01.216 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:01.217 DEBUG:boto:Method: GET 2022-09-12T20:16:01.217 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:01.218 DEBUG:boto:Data: 2022-09-12T20:16:01.218 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.219 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.219 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.220 DEBUG:boto:Params: {} 2022-09-12T20:16:01.220 DEBUG:boto:Token: None 2022-09-12T20:16:01.221 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:01.222 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:O2y74qx0k4zGlWHU/YZK8lR50z4= 2022-09-12T20:16:01.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:O2y74qx0k4zGlWHU/YZK8lR50z4=', 'Content-Length': '0'} 2022-09-12T20:16:01.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079d4f76fb22f9b96-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.270 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:01.271 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:01.271 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:01.272 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:01.272 DEBUG:boto:Method: HEAD 2022-09-12T20:16:01.273 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:01.273 DEBUG:boto:Data: 2022-09-12T20:16:01.274 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.274 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.275 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.275 DEBUG:boto:Params: {} 2022-09-12T20:16:01.276 DEBUG:boto:Token: None 2022-09-12T20:16:01.277 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:01.277 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:v+/iuC7vjDOx0Lc0Gnukn0/XMNc= 2022-09-12T20:16:01.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:v+/iuC7vjDOx0Lc0Gnukn0/XMNc=', 'Content-Length': '0'} 2022-09-12T20:16:01.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000180876cc473d56ea-00631f9381-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.281 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:01.281 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:01.282 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:01.282 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:01.283 DEBUG:boto:Method: HEAD 2022-09-12T20:16:01.283 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:01.284 DEBUG:boto:Data: 2022-09-12T20:16:01.284 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.285 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.285 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.286 DEBUG:boto:Params: {} 2022-09-12T20:16:01.286 DEBUG:boto:Token: None 2022-09-12T20:16:01.287 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:01.288 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:v+/iuC7vjDOx0Lc0Gnukn0/XMNc= 2022-09-12T20:16:01.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:v+/iuC7vjDOx0Lc0Gnukn0/XMNc=', 'Content-Length': '0'} 2022-09-12T20:16:01.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a065db7cfcc4d933-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:16:01.291 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:01.292 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:01.293 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:01.293 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:01.294 DEBUG:boto:Method: GET 2022-09-12T20:16:01.294 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:01.295 DEBUG:boto:Data: 2022-09-12T20:16:01.295 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.296 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.296 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.297 DEBUG:boto:Params: {} 2022-09-12T20:16:01.297 DEBUG:boto:Token: None 2022-09-12T20:16:01.298 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:01.299 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:O2y74qx0k4zGlWHU/YZK8lR50z4= 2022-09-12T20:16:01.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:O2y74qx0k4zGlWHU/YZK8lR50z4=', 'Content-Length': '0'} 2022-09-12T20:16:01.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006413b7de84cbb0b6-00631f9381-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.302 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:01.303 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:01.303 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:01.304 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:01.304 DEBUG:boto:Method: GET 2022-09-12T20:16:01.305 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:01.305 DEBUG:boto:Data: 2022-09-12T20:16:01.306 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.306 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.307 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.307 DEBUG:boto:Params: {} 2022-09-12T20:16:01.308 DEBUG:boto:Token: None 2022-09-12T20:16:01.308 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:01.309 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:O2y74qx0k4zGlWHU/YZK8lR50z4= 2022-09-12T20:16:01.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:O2y74qx0k4zGlWHU/YZK8lR50z4=', 'Content-Length': '0'} 2022-09-12T20:16:01.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fa05246ae748ccd-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.312 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:01.313 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:01.313 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:16:01.314 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:16:01.314 DEBUG:boto:Method: GET 2022-09-12T20:16:01.315 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:16:01.315 DEBUG:boto:Data: 2022-09-12T20:16:01.316 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.317 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.317 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.318 DEBUG:boto:Params: {} 2022-09-12T20:16:01.318 DEBUG:boto:Token: None 2022-09-12T20:16:01.319 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/ 2022-09-12T20:16:01.319 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A= 2022-09-12T20:16:01.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A=', 'Content-Length': '0'} 2022-09-12T20:16:01.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000032113a5549d618d-00631f9381-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.323 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:01.324 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:01.325 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:01.325 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:16:01.326 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:16:01.326 DEBUG:boto:Method: GET 2022-09-12T20:16:01.327 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:16:01.327 DEBUG:boto:Data: 2022-09-12T20:16:01.328 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.328 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.329 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.329 DEBUG:boto:Params: {} 2022-09-12T20:16:01.330 DEBUG:boto:Token: None 2022-09-12T20:16:01.330 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/ 2022-09-12T20:16:01.331 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A= 2022-09-12T20:16:01.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A=', 'Content-Length': '0'} 2022-09-12T20:16:01.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000657e832d93b3e400-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.337 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:01.338 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:01.338 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:01.339 DEBUG:boto:Method: HEAD 2022-09-12T20:16:01.339 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:16:01.340 DEBUG:boto:Data: 2022-09-12T20:16:01.340 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.341 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.341 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.342 DEBUG:boto:Params: {} 2022-09-12T20:16:01.342 DEBUG:boto:Token: None 2022-09-12T20:16:01.343 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj23 2022-09-12T20:16:01.343 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/vFHYjb9KjAs8csMFnYHNSXqgNA= 2022-09-12T20:16:01.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/vFHYjb9KjAs8csMFnYHNSXqgNA=', 'Content-Length': '0'} 2022-09-12T20:16:01.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed9407db6cd35dc7-00631f9381-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.347 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:01.347 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:01.348 DEBUG:boto:Method: HEAD 2022-09-12T20:16:01.348 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:16:01.349 DEBUG:boto:Data: 2022-09-12T20:16:01.349 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.350 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.350 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.351 DEBUG:boto:Params: {} 2022-09-12T20:16:01.351 DEBUG:boto:Token: None 2022-09-12T20:16:01.352 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj23 2022-09-12T20:16:01.352 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/vFHYjb9KjAs8csMFnYHNSXqgNA= 2022-09-12T20:16:01.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/vFHYjb9KjAs8csMFnYHNSXqgNA=', 'Content-Length': '0'} 2022-09-12T20:16:01.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000beb4d1b02794bf53-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:16:01.356 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:01.357 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:01.357 DEBUG:boto:Method: GET 2022-09-12T20:16:01.358 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:16:01.358 DEBUG:boto:Data: 2022-09-12T20:16:01.359 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.360 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.360 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.361 DEBUG:boto:Params: {} 2022-09-12T20:16:01.361 DEBUG:boto:Token: None 2022-09-12T20:16:01.362 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj23 2022-09-12T20:16:01.362 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7093t73TegtXq7pfmoMTgQQSN+8= 2022-09-12T20:16:01.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7093t73TegtXq7pfmoMTgQQSN+8=', 'Content-Length': '0'} 2022-09-12T20:16:01.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af2126a5165e5b85-00631f9381-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.410 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:01.411 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:01.411 DEBUG:boto:Method: GET 2022-09-12T20:16:01.412 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:16:01.412 DEBUG:boto:Data: 2022-09-12T20:16:01.413 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.413 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.414 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.415 DEBUG:boto:Params: {} 2022-09-12T20:16:01.416 DEBUG:boto:Token: None 2022-09-12T20:16:01.416 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj23 2022-09-12T20:16:01.417 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7093t73TegtXq7pfmoMTgQQSN+8= 2022-09-12T20:16:01.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7093t73TegtXq7pfmoMTgQQSN+8=', 'Content-Length': '0'} 2022-09-12T20:16:01.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000985f82ba6b83a13f-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.421 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:01.421 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:01.422 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:16:01.422 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:16:01.423 DEBUG:boto:Method: GET 2022-09-12T20:16:01.424 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:16:01.424 DEBUG:boto:Data: 2022-09-12T20:16:01.425 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.425 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.426 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.426 DEBUG:boto:Params: {} 2022-09-12T20:16:01.427 DEBUG:boto:Token: None 2022-09-12T20:16:01.428 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/ 2022-09-12T20:16:01.428 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A= 2022-09-12T20:16:01.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A=', 'Content-Length': '0'} 2022-09-12T20:16:01.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1b0755dc9141cb0-00631f9381-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.432 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:01.433 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:01.434 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:01.434 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:16:01.435 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:16:01.435 DEBUG:boto:Method: GET 2022-09-12T20:16:01.436 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:16:01.436 DEBUG:boto:Data: 2022-09-12T20:16:01.437 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.437 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.438 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.439 DEBUG:boto:Params: {} 2022-09-12T20:16:01.439 DEBUG:boto:Token: None 2022-09-12T20:16:01.440 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/ 2022-09-12T20:16:01.440 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A= 2022-09-12T20:16:01.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A=', 'Content-Length': '0'} 2022-09-12T20:16:01.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087436ef09b45794e-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.447 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:01.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:16:01.448 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:01.449 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:01.449 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:01.450 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:01.450 DEBUG:boto:Method: GET 2022-09-12T20:16:01.451 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:01.451 DEBUG:boto:Data: 2022-09-12T20:16:01.452 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.452 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.453 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.453 DEBUG:boto:Params: {} 2022-09-12T20:16:01.454 DEBUG:boto:Token: None 2022-09-12T20:16:01.455 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:01.455 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lEnLojgdBpQTyLp078O9BMfwkSo= 2022-09-12T20:16:01.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lEnLojgdBpQTyLp078O9BMfwkSo=', 'Content-Length': '0'} 2022-09-12T20:16:01.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f7b0a41efd10e9b-00631f9381-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.459 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:01.459 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:01.460 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:01.461 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:01.461 DEBUG:boto:Method: GET 2022-09-12T20:16:01.462 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:01.462 DEBUG:boto:Data: 2022-09-12T20:16:01.463 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.463 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.464 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.464 DEBUG:boto:Params: {} 2022-09-12T20:16:01.465 DEBUG:boto:Token: None 2022-09-12T20:16:01.465 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:01.466 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lEnLojgdBpQTyLp078O9BMfwkSo= 2022-09-12T20:16:01.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lEnLojgdBpQTyLp078O9BMfwkSo=', 'Content-Length': '0'} 2022-09-12T20:16:01.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e7c7b0c62f554cf-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.470 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:01.470 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:01.471 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:01.471 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:01.472 DEBUG:boto:Method: HEAD 2022-09-12T20:16:01.472 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:01.473 DEBUG:boto:Data: 2022-09-12T20:16:01.473 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.474 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.474 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.475 DEBUG:boto:Params: {} 2022-09-12T20:16:01.475 DEBUG:boto:Token: None 2022-09-12T20:16:01.476 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:01.476 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gF5GdVHmklI0yvzfT63yJf3cQRc= 2022-09-12T20:16:01.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gF5GdVHmklI0yvzfT63yJf3cQRc=', 'Content-Length': '0'} 2022-09-12T20:16:01.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083fe6f5cda8ad92b-00631f9381-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.480 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:01.480 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:01.481 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:01.481 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:01.482 DEBUG:boto:Method: HEAD 2022-09-12T20:16:01.482 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:01.483 DEBUG:boto:Data: 2022-09-12T20:16:01.483 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.484 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.484 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.485 DEBUG:boto:Params: {} 2022-09-12T20:16:01.485 DEBUG:boto:Token: None 2022-09-12T20:16:01.486 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:01.487 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gF5GdVHmklI0yvzfT63yJf3cQRc= 2022-09-12T20:16:01.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gF5GdVHmklI0yvzfT63yJf3cQRc=', 'Content-Length': '0'} 2022-09-12T20:16:01.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b044d13c2b0439d-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:16:01.491 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:01.491 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:01.492 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:01.492 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:01.493 DEBUG:boto:Method: GET 2022-09-12T20:16:01.493 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:01.494 DEBUG:boto:Data: 2022-09-12T20:16:01.494 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.495 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.495 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.496 DEBUG:boto:Params: {} 2022-09-12T20:16:01.497 DEBUG:boto:Token: None 2022-09-12T20:16:01.497 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:01.498 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lEnLojgdBpQTyLp078O9BMfwkSo= 2022-09-12T20:16:01.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lEnLojgdBpQTyLp078O9BMfwkSo=', 'Content-Length': '0'} 2022-09-12T20:16:01.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067020df653be9d02-00631f9381-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.501 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:01.502 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:01.502 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:01.503 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:01.503 DEBUG:boto:Method: GET 2022-09-12T20:16:01.504 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:01.504 DEBUG:boto:Data: 2022-09-12T20:16:01.505 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.505 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.506 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.506 DEBUG:boto:Params: {} 2022-09-12T20:16:01.507 DEBUG:boto:Token: None 2022-09-12T20:16:01.508 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:01.508 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lEnLojgdBpQTyLp078O9BMfwkSo= 2022-09-12T20:16:01.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lEnLojgdBpQTyLp078O9BMfwkSo=', 'Content-Length': '0'} 2022-09-12T20:16:01.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003528bc03b5fd1efb-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.512 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:01.512 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:01.513 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:16:01.513 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:16:01.514 DEBUG:boto:Method: GET 2022-09-12T20:16:01.514 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:16:01.515 DEBUG:boto:Data: 2022-09-12T20:16:01.516 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.516 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.517 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.517 DEBUG:boto:Params: {} 2022-09-12T20:16:01.518 DEBUG:boto:Token: None 2022-09-12T20:16:01.518 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/ 2022-09-12T20:16:01.519 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A= 2022-09-12T20:16:01.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A=', 'Content-Length': '0'} 2022-09-12T20:16:01.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6aabe694cce219e-00631f9381-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.523 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:01.524 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:01.524 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:01.525 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:16:01.525 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:16:01.526 DEBUG:boto:Method: GET 2022-09-12T20:16:01.526 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:16:01.527 DEBUG:boto:Data: 2022-09-12T20:16:01.527 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.528 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.528 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.529 DEBUG:boto:Params: {} 2022-09-12T20:16:01.529 DEBUG:boto:Token: None 2022-09-12T20:16:01.530 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/ 2022-09-12T20:16:01.531 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A= 2022-09-12T20:16:01.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A=', 'Content-Length': '0'} 2022-09-12T20:16:01.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce3f7b659a81ffe2-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.537 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:01.537 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:01.538 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:01.538 DEBUG:boto:Method: HEAD 2022-09-12T20:16:01.539 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:16:01.539 DEBUG:boto:Data: 2022-09-12T20:16:01.540 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.540 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.541 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.541 DEBUG:boto:Params: {} 2022-09-12T20:16:01.542 DEBUG:boto:Token: None 2022-09-12T20:16:01.543 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj29 2022-09-12T20:16:01.543 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:j1uo4v+JW4KED4DlC47R5SlR/qs= 2022-09-12T20:16:01.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:j1uo4v+JW4KED4DlC47R5SlR/qs=', 'Content-Length': '0'} 2022-09-12T20:16:01.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b97a9c40db69393b-00631f9381-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.546 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:01.547 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:01.547 DEBUG:boto:Method: HEAD 2022-09-12T20:16:01.548 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:16:01.548 DEBUG:boto:Data: 2022-09-12T20:16:01.549 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.549 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.550 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.550 DEBUG:boto:Params: {} 2022-09-12T20:16:01.551 DEBUG:boto:Token: None 2022-09-12T20:16:01.552 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj29 2022-09-12T20:16:01.552 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:j1uo4v+JW4KED4DlC47R5SlR/qs= 2022-09-12T20:16:01.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:j1uo4v+JW4KED4DlC47R5SlR/qs=', 'Content-Length': '0'} 2022-09-12T20:16:01.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023b0e9dcd8d01724-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:16:01.556 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:01.557 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:01.557 DEBUG:boto:Method: GET 2022-09-12T20:16:01.558 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:16:01.559 DEBUG:boto:Data: 2022-09-12T20:16:01.559 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.560 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.560 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.561 DEBUG:boto:Params: {} 2022-09-12T20:16:01.561 DEBUG:boto:Token: None 2022-09-12T20:16:01.562 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj29 2022-09-12T20:16:01.562 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WQFFTP8TSm9AnR6FfrodT0MPkbg= 2022-09-12T20:16:01.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WQFFTP8TSm9AnR6FfrodT0MPkbg=', 'Content-Length': '0'} 2022-09-12T20:16:01.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8ad8a75d00784cb-00631f9381-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.606 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:01.607 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:01.608 DEBUG:boto:Method: GET 2022-09-12T20:16:01.608 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:16:01.609 DEBUG:boto:Data: 2022-09-12T20:16:01.609 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.610 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.611 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.611 DEBUG:boto:Params: {} 2022-09-12T20:16:01.612 DEBUG:boto:Token: None 2022-09-12T20:16:01.613 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj29 2022-09-12T20:16:01.614 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WQFFTP8TSm9AnR6FfrodT0MPkbg= 2022-09-12T20:16:01.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WQFFTP8TSm9AnR6FfrodT0MPkbg=', 'Content-Length': '0'} 2022-09-12T20:16:01.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093b9385f7b658d2c-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.617 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:01.618 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:01.619 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:16:01.619 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:16:01.620 DEBUG:boto:Method: GET 2022-09-12T20:16:01.620 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:16:01.621 DEBUG:boto:Data: 2022-09-12T20:16:01.621 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.622 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.622 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.623 DEBUG:boto:Params: {} 2022-09-12T20:16:01.624 DEBUG:boto:Token: None 2022-09-12T20:16:01.625 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/ 2022-09-12T20:16:01.625 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A= 2022-09-12T20:16:01.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A=', 'Content-Length': '0'} 2022-09-12T20:16:01.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e38ee8cde898998-00631f9381-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.629 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:01.630 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:01.631 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:01.631 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:16:01.632 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:16:01.632 DEBUG:boto:Method: GET 2022-09-12T20:16:01.633 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:16:01.634 DEBUG:boto:Data: 2022-09-12T20:16:01.635 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.636 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.636 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.637 DEBUG:boto:Params: {} 2022-09-12T20:16:01.637 DEBUG:boto:Token: None 2022-09-12T20:16:01.638 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/ 2022-09-12T20:16:01.639 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A= 2022-09-12T20:16:01.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A=', 'Content-Length': '0'} 2022-09-12T20:16:01.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038e75ca5574a3601-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.647 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:01.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-12T20:16:01.648 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:01.649 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:01.650 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:01.650 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:01.651 DEBUG:boto:Method: GET 2022-09-12T20:16:01.651 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:01.652 DEBUG:boto:Data: 2022-09-12T20:16:01.652 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.653 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.653 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.654 DEBUG:boto:Params: {} 2022-09-12T20:16:01.655 DEBUG:boto:Token: None 2022-09-12T20:16:01.656 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:01.657 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3sXBd0QIGQ+toITPx4FwrmR4lnU= 2022-09-12T20:16:01.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3sXBd0QIGQ+toITPx4FwrmR4lnU=', 'Content-Length': '0'} 2022-09-12T20:16:01.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edd7de3937212d93-00631f9381-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.661 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:01.661 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:01.662 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:01.663 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:01.663 DEBUG:boto:Method: GET 2022-09-12T20:16:01.664 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:01.664 DEBUG:boto:Data: 2022-09-12T20:16:01.665 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.666 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.666 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.667 DEBUG:boto:Params: {} 2022-09-12T20:16:01.668 DEBUG:boto:Token: None 2022-09-12T20:16:01.669 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:01.669 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3sXBd0QIGQ+toITPx4FwrmR4lnU= 2022-09-12T20:16:01.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3sXBd0QIGQ+toITPx4FwrmR4lnU=', 'Content-Length': '0'} 2022-09-12T20:16:01.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e80e9fbc8ffb6227-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.674 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:01.675 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:01.675 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:01.676 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:01.677 DEBUG:boto:Method: HEAD 2022-09-12T20:16:01.678 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:01.679 DEBUG:boto:Data: 2022-09-12T20:16:01.680 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.681 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.681 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.682 DEBUG:boto:Params: {} 2022-09-12T20:16:01.684 DEBUG:boto:Token: None 2022-09-12T20:16:01.685 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:01.686 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yPb3+HDBkz2QYCfmEyaQ9FC4g2M= 2022-09-12T20:16:01.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yPb3+HDBkz2QYCfmEyaQ9FC4g2M=', 'Content-Length': '0'} 2022-09-12T20:16:01.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ada6fa0e243c315b-00631f9381-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.691 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:01.692 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:01.693 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:01.693 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:01.694 DEBUG:boto:Method: HEAD 2022-09-12T20:16:01.695 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:01.696 DEBUG:boto:Data: 2022-09-12T20:16:01.697 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.698 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.699 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.700 DEBUG:boto:Params: {} 2022-09-12T20:16:01.701 DEBUG:boto:Token: None 2022-09-12T20:16:01.702 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:01.703 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yPb3+HDBkz2QYCfmEyaQ9FC4g2M= 2022-09-12T20:16:01.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yPb3+HDBkz2QYCfmEyaQ9FC4g2M=', 'Content-Length': '0'} 2022-09-12T20:16:01.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c72cdeab88ed7cdd-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-12T20:16:01.708 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:01.708 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:01.709 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:01.709 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:01.710 DEBUG:boto:Method: GET 2022-09-12T20:16:01.710 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:01.711 DEBUG:boto:Data: 2022-09-12T20:16:01.712 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.712 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.713 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.713 DEBUG:boto:Params: {} 2022-09-12T20:16:01.714 DEBUG:boto:Token: None 2022-09-12T20:16:01.714 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:01.715 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3sXBd0QIGQ+toITPx4FwrmR4lnU= 2022-09-12T20:16:01.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3sXBd0QIGQ+toITPx4FwrmR4lnU=', 'Content-Length': '0'} 2022-09-12T20:16:01.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b97f3e8c3bccc593-00631f9381-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.719 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:01.720 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:01.720 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:01.721 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:01.722 DEBUG:boto:Method: GET 2022-09-12T20:16:01.722 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:01.723 DEBUG:boto:Data: 2022-09-12T20:16:01.723 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.724 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.724 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.725 DEBUG:boto:Params: {} 2022-09-12T20:16:01.725 DEBUG:boto:Token: None 2022-09-12T20:16:01.726 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:01.727 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3sXBd0QIGQ+toITPx4FwrmR4lnU= 2022-09-12T20:16:01.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3sXBd0QIGQ+toITPx4FwrmR4lnU=', 'Content-Length': '0'} 2022-09-12T20:16:01.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3be1a18bc511bc9-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.731 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:01.731 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:01.732 DEBUG:boto:path=/jdpiak-75/?prefix=obj31 2022-09-12T20:16:01.732 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj31 2022-09-12T20:16:01.733 DEBUG:boto:Method: GET 2022-09-12T20:16:01.734 DEBUG:boto:Path: /jdpiak-75/?prefix=obj31 2022-09-12T20:16:01.734 DEBUG:boto:Data: 2022-09-12T20:16:01.735 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.735 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.736 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.736 DEBUG:boto:Params: {} 2022-09-12T20:16:01.737 DEBUG:boto:Token: None 2022-09-12T20:16:01.737 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/ 2022-09-12T20:16:01.738 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A= 2022-09-12T20:16:01.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A=', 'Content-Length': '0'} 2022-09-12T20:16:01.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6ee58eaf8695178-00631f9381-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.743 DEBUG:boto:b'jdpiak-75obj311000falseobj312022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:01.744 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:01.745 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:01.746 DEBUG:boto:path=/jdpiak-75/?prefix=obj31 2022-09-12T20:16:01.747 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj31 2022-09-12T20:16:01.748 DEBUG:boto:Method: GET 2022-09-12T20:16:01.748 DEBUG:boto:Path: /jdpiak-75/?prefix=obj31 2022-09-12T20:16:01.749 DEBUG:boto:Data: 2022-09-12T20:16:01.750 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.751 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.752 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.753 DEBUG:boto:Params: {} 2022-09-12T20:16:01.754 DEBUG:boto:Token: None 2022-09-12T20:16:01.755 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/ 2022-09-12T20:16:01.756 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A= 2022-09-12T20:16:01.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A=', 'Content-Length': '0'} 2022-09-12T20:16:01.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1fa611972139627-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.763 DEBUG:boto:b'jdpiak-75obj311000falseobj312022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:01.764 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:01.765 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:01.765 DEBUG:boto:Method: HEAD 2022-09-12T20:16:01.766 DEBUG:boto:Path: /jdpiak-75/obj31 2022-09-12T20:16:01.766 DEBUG:boto:Data: 2022-09-12T20:16:01.767 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.767 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.768 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.768 DEBUG:boto:Params: {} 2022-09-12T20:16:01.769 DEBUG:boto:Token: None 2022-09-12T20:16:01.770 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj31 2022-09-12T20:16:01.770 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:trGUIWs1Qf/ryZUX9febupW05+0= 2022-09-12T20:16:01.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:trGUIWs1Qf/ryZUX9febupW05+0=', 'Content-Length': '0'} 2022-09-12T20:16:01.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d487bb13e3faba45-00631f9381-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.774 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:01.774 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:01.775 DEBUG:boto:Method: HEAD 2022-09-12T20:16:01.775 DEBUG:boto:Path: /jdpiak-75/obj31 2022-09-12T20:16:01.776 DEBUG:boto:Data: 2022-09-12T20:16:01.776 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.777 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.777 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.778 DEBUG:boto:Params: {} 2022-09-12T20:16:01.778 DEBUG:boto:Token: None 2022-09-12T20:16:01.779 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj31 2022-09-12T20:16:01.779 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:trGUIWs1Qf/ryZUX9febupW05+0= 2022-09-12T20:16:01.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:trGUIWs1Qf/ryZUX9febupW05+0=', 'Content-Length': '0'} 2022-09-12T20:16:01.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b59bae00a90d551d-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-12T20:16:01.783 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:01.784 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:01.784 DEBUG:boto:Method: GET 2022-09-12T20:16:01.785 DEBUG:boto:Path: /jdpiak-75/obj31 2022-09-12T20:16:01.785 DEBUG:boto:Data: 2022-09-12T20:16:01.786 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.786 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.787 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.787 DEBUG:boto:Params: {} 2022-09-12T20:16:01.788 DEBUG:boto:Token: None 2022-09-12T20:16:01.788 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj31 2022-09-12T20:16:01.789 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tZpjFe+3HAZ49pK31rzR6+8osxc= 2022-09-12T20:16:01.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tZpjFe+3HAZ49pK31rzR6+8osxc=', 'Content-Length': '0'} 2022-09-12T20:16:01.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e230732facf65460-00631f9381-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.793 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:01.793 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:01.794 DEBUG:boto:Method: GET 2022-09-12T20:16:01.794 DEBUG:boto:Path: /jdpiak-75/obj31 2022-09-12T20:16:01.795 DEBUG:boto:Data: 2022-09-12T20:16:01.795 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.796 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.797 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.797 DEBUG:boto:Params: {} 2022-09-12T20:16:01.805 DEBUG:boto:Token: None 2022-09-12T20:16:01.806 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj31 2022-09-12T20:16:01.806 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tZpjFe+3HAZ49pK31rzR6+8osxc= 2022-09-12T20:16:01.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tZpjFe+3HAZ49pK31rzR6+8osxc=', 'Content-Length': '0'} 2022-09-12T20:16:01.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccd2a3157e5a874c-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.810 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:01.810 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:01.811 DEBUG:boto:path=/jdpiak-75/?prefix=obj31 2022-09-12T20:16:01.811 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj31 2022-09-12T20:16:01.812 DEBUG:boto:Method: GET 2022-09-12T20:16:01.812 DEBUG:boto:Path: /jdpiak-75/?prefix=obj31 2022-09-12T20:16:01.813 DEBUG:boto:Data: 2022-09-12T20:16:01.813 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.814 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.814 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.815 DEBUG:boto:Params: {} 2022-09-12T20:16:01.816 DEBUG:boto:Token: None 2022-09-12T20:16:01.816 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/ 2022-09-12T20:16:01.817 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A= 2022-09-12T20:16:01.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A=', 'Content-Length': '0'} 2022-09-12T20:16:01.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a4835c4cc651f9e-00631f9381-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.821 DEBUG:boto:b'jdpiak-75obj311000falseobj312022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:01.822 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:01.822 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:01.823 DEBUG:boto:path=/jdpiak-75/?prefix=obj31 2022-09-12T20:16:01.823 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj31 2022-09-12T20:16:01.824 DEBUG:boto:Method: GET 2022-09-12T20:16:01.824 DEBUG:boto:Path: /jdpiak-75/?prefix=obj31 2022-09-12T20:16:01.825 DEBUG:boto:Data: 2022-09-12T20:16:01.825 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.826 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.826 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.827 DEBUG:boto:Params: {} 2022-09-12T20:16:01.828 DEBUG:boto:Token: None 2022-09-12T20:16:01.828 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/ 2022-09-12T20:16:01.829 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A= 2022-09-12T20:16:01.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A=', 'Content-Length': '0'} 2022-09-12T20:16:01.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000335a075e69cc34c9-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.835 DEBUG:boto:b'jdpiak-75obj311000falseobj312022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:01.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:16:01.837 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:01.837 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:01.838 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:01.839 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:01.839 DEBUG:boto:Method: GET 2022-09-12T20:16:01.840 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:01.840 DEBUG:boto:Data: 2022-09-12T20:16:01.841 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.842 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.842 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.843 DEBUG:boto:Params: {} 2022-09-12T20:16:01.844 DEBUG:boto:Token: None 2022-09-12T20:16:01.844 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:01.845 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7Q4VXnuyExqdtGed5eG+ZR2cOHM= 2022-09-12T20:16:01.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7Q4VXnuyExqdtGed5eG+ZR2cOHM=', 'Content-Length': '0'} 2022-09-12T20:16:01.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020f475e01efc57d9-00631f9381-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.848 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:01.849 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:01.849 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:01.850 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:01.850 DEBUG:boto:Method: GET 2022-09-12T20:16:01.851 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:01.851 DEBUG:boto:Data: 2022-09-12T20:16:01.852 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.852 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.853 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.853 DEBUG:boto:Params: {} 2022-09-12T20:16:01.854 DEBUG:boto:Token: None 2022-09-12T20:16:01.855 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:01.855 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7Q4VXnuyExqdtGed5eG+ZR2cOHM= 2022-09-12T20:16:01.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7Q4VXnuyExqdtGed5eG+ZR2cOHM=', 'Content-Length': '0'} 2022-09-12T20:16:01.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfd688f5bc1c4b13-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.902 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:01.903 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:01.903 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:01.904 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:01.904 DEBUG:boto:Method: HEAD 2022-09-12T20:16:01.905 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:01.905 DEBUG:boto:Data: 2022-09-12T20:16:01.906 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.906 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.907 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.907 DEBUG:boto:Params: {} 2022-09-12T20:16:01.908 DEBUG:boto:Token: None 2022-09-12T20:16:01.909 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:01.909 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2CR0GR46NzJDnnqvOGCa3fUOgmg= 2022-09-12T20:16:01.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2CR0GR46NzJDnnqvOGCa3fUOgmg=', 'Content-Length': '0'} 2022-09-12T20:16:01.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000619ffd6a16ee2b05-00631f9381-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.913 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:01.913 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:01.914 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:01.914 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:01.915 DEBUG:boto:Method: HEAD 2022-09-12T20:16:01.915 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:01.916 DEBUG:boto:Data: 2022-09-12T20:16:01.916 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.917 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.918 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.919 DEBUG:boto:Params: {} 2022-09-12T20:16:01.919 DEBUG:boto:Token: None 2022-09-12T20:16:01.920 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:01.921 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2CR0GR46NzJDnnqvOGCa3fUOgmg= 2022-09-12T20:16:01.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2CR0GR46NzJDnnqvOGCa3fUOgmg=', 'Content-Length': '0'} 2022-09-12T20:16:01.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000959fb64bfa75fafc-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:16:01.925 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:01.925 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:01.926 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:01.927 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:01.927 DEBUG:boto:Method: GET 2022-09-12T20:16:01.928 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:01.928 DEBUG:boto:Data: 2022-09-12T20:16:01.929 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.929 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.930 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.930 DEBUG:boto:Params: {} 2022-09-12T20:16:01.931 DEBUG:boto:Token: None 2022-09-12T20:16:01.932 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:01.932 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7Q4VXnuyExqdtGed5eG+ZR2cOHM= 2022-09-12T20:16:01.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7Q4VXnuyExqdtGed5eG+ZR2cOHM=', 'Content-Length': '0'} 2022-09-12T20:16:01.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9003a32b7dc08c2-00631f9381-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.936 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:01.936 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:01.937 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:01.937 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:01.938 DEBUG:boto:Method: GET 2022-09-12T20:16:01.939 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:01.939 DEBUG:boto:Data: 2022-09-12T20:16:01.940 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.940 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:01.941 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.941 DEBUG:boto:Params: {} 2022-09-12T20:16:01.942 DEBUG:boto:Token: None 2022-09-12T20:16:01.942 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:01.943 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7Q4VXnuyExqdtGed5eG+ZR2cOHM= 2022-09-12T20:16:01.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7Q4VXnuyExqdtGed5eG+ZR2cOHM=', 'Content-Length': '0'} 2022-09-12T20:16:01.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2c00f34d34ffd65-00631f9381-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:01.947 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:01.947 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:01.948 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:16:01.948 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:16:01.949 DEBUG:boto:Method: GET 2022-09-12T20:16:01.949 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:16:01.950 DEBUG:boto:Data: 2022-09-12T20:16:01.950 DEBUG:boto:Headers: {} 2022-09-12T20:16:01.951 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:01.952 DEBUG:boto:Port: 8000 2022-09-12T20:16:01.952 DEBUG:boto:Params: {} 2022-09-12T20:16:01.953 DEBUG:boto:Token: None 2022-09-12T20:16:01.953 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:01 GMT /jdpiak-75/ 2022-09-12T20:16:01.954 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A= 2022-09-12T20:16:01.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7PhnsIC5DUbZV7HnlBgpHZMkA8A=', 'Content-Length': '0'} 2022-09-12T20:16:01.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f51198a6d85a015-00631f9381-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.002 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:02.003 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:02.004 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:02.004 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:16:02.005 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:16:02.005 DEBUG:boto:Method: GET 2022-09-12T20:16:02.006 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:16:02.006 DEBUG:boto:Data: 2022-09-12T20:16:02.007 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.007 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.008 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.008 DEBUG:boto:Params: {} 2022-09-12T20:16:02.009 DEBUG:boto:Token: None 2022-09-12T20:16:02.010 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/ 2022-09-12T20:16:02.010 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU= 2022-09-12T20:16:02.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU=', 'Content-Length': '0'} 2022-09-12T20:16:02.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca17cc0b984e3d9b-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.016 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:02.017 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:02.018 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:02.018 DEBUG:boto:Method: HEAD 2022-09-12T20:16:02.019 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:16:02.019 DEBUG:boto:Data: 2022-09-12T20:16:02.020 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.020 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.021 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.021 DEBUG:boto:Params: {} 2022-09-12T20:16:02.022 DEBUG:boto:Token: None 2022-09-12T20:16:02.022 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj33 2022-09-12T20:16:02.023 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5B7d3aehgp2K4+8Wq/n7lruENhM= 2022-09-12T20:16:02.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5B7d3aehgp2K4+8Wq/n7lruENhM=', 'Content-Length': '0'} 2022-09-12T20:16:02.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ccb187f203e607b-00631f9382-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.026 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:02.027 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:02.027 DEBUG:boto:Method: HEAD 2022-09-12T20:16:02.028 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:16:02.028 DEBUG:boto:Data: 2022-09-12T20:16:02.029 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.029 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.030 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.030 DEBUG:boto:Params: {} 2022-09-12T20:16:02.031 DEBUG:boto:Token: None 2022-09-12T20:16:02.032 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj33 2022-09-12T20:16:02.032 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5B7d3aehgp2K4+8Wq/n7lruENhM= 2022-09-12T20:16:02.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5B7d3aehgp2K4+8Wq/n7lruENhM=', 'Content-Length': '0'} 2022-09-12T20:16:02.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a4d09a406361f38-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:16:02.036 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:02.037 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:02.037 DEBUG:boto:Method: GET 2022-09-12T20:16:02.038 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:16:02.039 DEBUG:boto:Data: 2022-09-12T20:16:02.039 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.040 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.040 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.041 DEBUG:boto:Params: {} 2022-09-12T20:16:02.041 DEBUG:boto:Token: None 2022-09-12T20:16:02.042 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj33 2022-09-12T20:16:02.043 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QNEy2yzYEChhwmbUvYBPPbb3IjY= 2022-09-12T20:16:02.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QNEy2yzYEChhwmbUvYBPPbb3IjY=', 'Content-Length': '0'} 2022-09-12T20:16:02.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da2ab5e96b728244-00631f9382-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.047 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:02.048 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:02.049 DEBUG:boto:Method: GET 2022-09-12T20:16:02.049 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:16:02.050 DEBUG:boto:Data: 2022-09-12T20:16:02.051 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.051 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.052 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.052 DEBUG:boto:Params: {} 2022-09-12T20:16:02.053 DEBUG:boto:Token: None 2022-09-12T20:16:02.054 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj33 2022-09-12T20:16:02.054 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QNEy2yzYEChhwmbUvYBPPbb3IjY= 2022-09-12T20:16:02.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QNEy2yzYEChhwmbUvYBPPbb3IjY=', 'Content-Length': '0'} 2022-09-12T20:16:02.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0dbf7c82f63ba69-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.102 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:02.103 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:02.103 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:16:02.104 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:16:02.104 DEBUG:boto:Method: GET 2022-09-12T20:16:02.105 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:16:02.105 DEBUG:boto:Data: 2022-09-12T20:16:02.106 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.106 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.107 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.108 DEBUG:boto:Params: {} 2022-09-12T20:16:02.108 DEBUG:boto:Token: None 2022-09-12T20:16:02.109 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/ 2022-09-12T20:16:02.109 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU= 2022-09-12T20:16:02.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU=', 'Content-Length': '0'} 2022-09-12T20:16:02.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000082b88b1a388a1d6-00631f9382-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.114 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:02.115 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:02.115 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:02.116 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:16:02.116 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:16:02.117 DEBUG:boto:Method: GET 2022-09-12T20:16:02.117 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:16:02.118 DEBUG:boto:Data: 2022-09-12T20:16:02.118 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.119 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.119 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.120 DEBUG:boto:Params: {} 2022-09-12T20:16:02.121 DEBUG:boto:Token: None 2022-09-12T20:16:02.121 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/ 2022-09-12T20:16:02.122 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU= 2022-09-12T20:16:02.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU=', 'Content-Length': '0'} 2022-09-12T20:16:02.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000915ced5d12559487-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.129 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:02.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:16:02.130 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:02.131 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:02.131 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:02.132 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:02.132 DEBUG:boto:Method: GET 2022-09-12T20:16:02.133 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:02.134 DEBUG:boto:Data: 2022-09-12T20:16:02.134 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.135 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.135 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.136 DEBUG:boto:Params: {} 2022-09-12T20:16:02.137 DEBUG:boto:Token: None 2022-09-12T20:16:02.137 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:02.138 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bTCeYqSXw5lsReX5f1W+kLpMOs8= 2022-09-12T20:16:02.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bTCeYqSXw5lsReX5f1W+kLpMOs8=', 'Content-Length': '0'} 2022-09-12T20:16:02.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000799cd91aea7157ca-00631f9382-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.142 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:02.142 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:02.143 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:02.144 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:02.144 DEBUG:boto:Method: GET 2022-09-12T20:16:02.145 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:02.145 DEBUG:boto:Data: 2022-09-12T20:16:02.146 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.147 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.147 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.148 DEBUG:boto:Params: {} 2022-09-12T20:16:02.149 DEBUG:boto:Token: None 2022-09-12T20:16:02.150 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:02.150 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bTCeYqSXw5lsReX5f1W+kLpMOs8= 2022-09-12T20:16:02.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bTCeYqSXw5lsReX5f1W+kLpMOs8=', 'Content-Length': '0'} 2022-09-12T20:16:02.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008034117b34014d99-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.154 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:02.155 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:02.156 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:02.156 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:02.157 DEBUG:boto:Method: HEAD 2022-09-12T20:16:02.157 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:02.158 DEBUG:boto:Data: 2022-09-12T20:16:02.159 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.159 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.160 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.161 DEBUG:boto:Params: {} 2022-09-12T20:16:02.161 DEBUG:boto:Token: None 2022-09-12T20:16:02.162 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:02.163 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bo6c+eXfUYnyXe1EJJYqmo6pJO8= 2022-09-12T20:16:02.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bo6c+eXfUYnyXe1EJJYqmo6pJO8=', 'Content-Length': '0'} 2022-09-12T20:16:02.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014e88a0109d6be62-00631f9382-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.167 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:02.167 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:02.168 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:02.169 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:02.169 DEBUG:boto:Method: HEAD 2022-09-12T20:16:02.170 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:02.171 DEBUG:boto:Data: 2022-09-12T20:16:02.171 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.172 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.173 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.174 DEBUG:boto:Params: {} 2022-09-12T20:16:02.174 DEBUG:boto:Token: None 2022-09-12T20:16:02.175 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:02.176 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bo6c+eXfUYnyXe1EJJYqmo6pJO8= 2022-09-12T20:16:02.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bo6c+eXfUYnyXe1EJJYqmo6pJO8=', 'Content-Length': '0'} 2022-09-12T20:16:02.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071f64657f7311a93-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:16:02.180 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:02.181 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:02.182 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:02.182 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:02.183 DEBUG:boto:Method: GET 2022-09-12T20:16:02.183 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:02.184 DEBUG:boto:Data: 2022-09-12T20:16:02.185 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.185 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.186 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.187 DEBUG:boto:Params: {} 2022-09-12T20:16:02.187 DEBUG:boto:Token: None 2022-09-12T20:16:02.188 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:02.189 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bTCeYqSXw5lsReX5f1W+kLpMOs8= 2022-09-12T20:16:02.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bTCeYqSXw5lsReX5f1W+kLpMOs8=', 'Content-Length': '0'} 2022-09-12T20:16:02.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000291ca4728a6ece10-00631f9382-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.193 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:02.194 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:02.194 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:02.195 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:02.195 DEBUG:boto:Method: GET 2022-09-12T20:16:02.196 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:02.197 DEBUG:boto:Data: 2022-09-12T20:16:02.197 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.198 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.199 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.200 DEBUG:boto:Params: {} 2022-09-12T20:16:02.201 DEBUG:boto:Token: None 2022-09-12T20:16:02.202 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:02.203 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bTCeYqSXw5lsReX5f1W+kLpMOs8= 2022-09-12T20:16:02.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bTCeYqSXw5lsReX5f1W+kLpMOs8=', 'Content-Length': '0'} 2022-09-12T20:16:02.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000365bb9aa26866431-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.254 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:02.256 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:02.257 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:16:02.259 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:16:02.260 DEBUG:boto:Method: GET 2022-09-12T20:16:02.261 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:16:02.262 DEBUG:boto:Data: 2022-09-12T20:16:02.264 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.265 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.266 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.267 DEBUG:boto:Params: {} 2022-09-12T20:16:02.268 DEBUG:boto:Token: None 2022-09-12T20:16:02.270 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/ 2022-09-12T20:16:02.271 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU= 2022-09-12T20:16:02.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU=', 'Content-Length': '0'} 2022-09-12T20:16:02.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f86b4b66b69f1d4-00631f9382-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.277 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:02.279 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:02.280 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:02.281 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:16:02.282 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:16:02.284 DEBUG:boto:Method: GET 2022-09-12T20:16:02.285 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:16:02.286 DEBUG:boto:Data: 2022-09-12T20:16:02.287 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.288 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.289 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.290 DEBUG:boto:Params: {} 2022-09-12T20:16:02.291 DEBUG:boto:Token: None 2022-09-12T20:16:02.292 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/ 2022-09-12T20:16:02.293 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU= 2022-09-12T20:16:02.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU=', 'Content-Length': '0'} 2022-09-12T20:16:02.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000253a8450349fd89b-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.301 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:02.304 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:02.305 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:02.306 DEBUG:boto:Method: HEAD 2022-09-12T20:16:02.307 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:16:02.309 DEBUG:boto:Data: 2022-09-12T20:16:02.310 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.311 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.312 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.313 DEBUG:boto:Params: {} 2022-09-12T20:16:02.315 DEBUG:boto:Token: None 2022-09-12T20:16:02.316 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj34 2022-09-12T20:16:02.317 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JhGuzsZWGnugESln59GKx1/c7sw= 2022-09-12T20:16:02.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JhGuzsZWGnugESln59GKx1/c7sw=', 'Content-Length': '0'} 2022-09-12T20:16:02.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000704f6ab79a927afc-00631f9382-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.322 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:02.323 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:02.324 DEBUG:boto:Method: HEAD 2022-09-12T20:16:02.325 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:16:02.326 DEBUG:boto:Data: 2022-09-12T20:16:02.326 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.327 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.328 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.329 DEBUG:boto:Params: {} 2022-09-12T20:16:02.330 DEBUG:boto:Token: None 2022-09-12T20:16:02.331 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj34 2022-09-12T20:16:02.332 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JhGuzsZWGnugESln59GKx1/c7sw= 2022-09-12T20:16:02.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JhGuzsZWGnugESln59GKx1/c7sw=', 'Content-Length': '0'} 2022-09-12T20:16:02.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f3368b4537209ec-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:16:02.337 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:02.338 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:02.339 DEBUG:boto:Method: GET 2022-09-12T20:16:02.339 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:16:02.340 DEBUG:boto:Data: 2022-09-12T20:16:02.340 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.341 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.341 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.342 DEBUG:boto:Params: {} 2022-09-12T20:16:02.342 DEBUG:boto:Token: None 2022-09-12T20:16:02.343 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj34 2022-09-12T20:16:02.344 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ABwNJnC1LRpBy0Oy9ShhBxkbMY= 2022-09-12T20:16:02.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ABwNJnC1LRpBy0Oy9ShhBxkbMY=', 'Content-Length': '0'} 2022-09-12T20:16:02.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e59330fc2ea0da76-00631f9382-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.347 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:02.348 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:02.349 DEBUG:boto:Method: GET 2022-09-12T20:16:02.349 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:16:02.350 DEBUG:boto:Data: 2022-09-12T20:16:02.350 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.351 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.351 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.352 DEBUG:boto:Params: {} 2022-09-12T20:16:02.352 DEBUG:boto:Token: None 2022-09-12T20:16:02.353 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj34 2022-09-12T20:16:02.354 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ABwNJnC1LRpBy0Oy9ShhBxkbMY= 2022-09-12T20:16:02.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ABwNJnC1LRpBy0Oy9ShhBxkbMY=', 'Content-Length': '0'} 2022-09-12T20:16:02.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068d196040c5b1cea-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.357 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:02.358 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:02.358 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:16:02.359 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:16:02.359 DEBUG:boto:Method: GET 2022-09-12T20:16:02.360 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:16:02.360 DEBUG:boto:Data: 2022-09-12T20:16:02.361 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.361 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.362 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.362 DEBUG:boto:Params: {} 2022-09-12T20:16:02.363 DEBUG:boto:Token: None 2022-09-12T20:16:02.364 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/ 2022-09-12T20:16:02.364 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU= 2022-09-12T20:16:02.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU=', 'Content-Length': '0'} 2022-09-12T20:16:02.368 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c33beee19a93ca6e-00631f9382-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.368 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:02.369 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:02.370 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:02.370 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:16:02.371 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:16:02.371 DEBUG:boto:Method: GET 2022-09-12T20:16:02.372 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:16:02.372 DEBUG:boto:Data: 2022-09-12T20:16:02.373 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.374 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.374 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.375 DEBUG:boto:Params: {} 2022-09-12T20:16:02.375 DEBUG:boto:Token: None 2022-09-12T20:16:02.376 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/ 2022-09-12T20:16:02.377 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU= 2022-09-12T20:16:02.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU=', 'Content-Length': '0'} 2022-09-12T20:16:02.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006100b4b166ac79e7-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.383 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:02.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2022-09-12T20:16:02.384 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:02.385 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:02.385 DEBUG:boto:path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:02.386 DEBUG:boto:auth_path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:02.386 DEBUG:boto:Method: GET 2022-09-12T20:16:02.387 DEBUG:boto:Path: /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:02.387 DEBUG:boto:Data: 2022-09-12T20:16:02.388 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.388 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.389 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.389 DEBUG:boto:Params: {} 2022-09-12T20:16:02.390 DEBUG:boto:Token: None 2022-09-12T20:16:02.391 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:02.391 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oEJ/ygj6s1XHxTRX4GiTu2hFrT4= 2022-09-12T20:16:02.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oEJ/ygj6s1XHxTRX4GiTu2hFrT4=', 'Content-Length': '0'} 2022-09-12T20:16:02.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006554bb8426747056-00631f9382-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.438 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:02.439 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:02.440 DEBUG:boto:path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:02.440 DEBUG:boto:auth_path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:02.441 DEBUG:boto:Method: GET 2022-09-12T20:16:02.441 DEBUG:boto:Path: /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:02.442 DEBUG:boto:Data: 2022-09-12T20:16:02.442 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.443 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.443 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.444 DEBUG:boto:Params: {} 2022-09-12T20:16:02.444 DEBUG:boto:Token: None 2022-09-12T20:16:02.445 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:02.446 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oEJ/ygj6s1XHxTRX4GiTu2hFrT4= 2022-09-12T20:16:02.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oEJ/ygj6s1XHxTRX4GiTu2hFrT4=', 'Content-Length': '0'} 2022-09-12T20:16:02.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d50add962493f26-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.449 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:02.450 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:02.450 DEBUG:boto:path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:02.451 DEBUG:boto:auth_path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:02.451 DEBUG:boto:Method: HEAD 2022-09-12T20:16:02.452 DEBUG:boto:Path: /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:02.452 DEBUG:boto:Data: 2022-09-12T20:16:02.453 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.453 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.454 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.454 DEBUG:boto:Params: {} 2022-09-12T20:16:02.455 DEBUG:boto:Token: None 2022-09-12T20:16:02.456 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:02.457 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RwdSZyF1KtQqZqOQtZrmLMTh1R4= 2022-09-12T20:16:02.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RwdSZyF1KtQqZqOQtZrmLMTh1R4=', 'Content-Length': '0'} 2022-09-12T20:16:02.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8514f54431ba41e-00631f9382-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.460 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:02.461 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:02.461 DEBUG:boto:path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:02.462 DEBUG:boto:auth_path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:02.462 DEBUG:boto:Method: HEAD 2022-09-12T20:16:02.463 DEBUG:boto:Path: /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:02.463 DEBUG:boto:Data: 2022-09-12T20:16:02.464 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.464 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.465 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.465 DEBUG:boto:Params: {} 2022-09-12T20:16:02.466 DEBUG:boto:Token: None 2022-09-12T20:16:02.467 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:02.467 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RwdSZyF1KtQqZqOQtZrmLMTh1R4= 2022-09-12T20:16:02.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RwdSZyF1KtQqZqOQtZrmLMTh1R4=', 'Content-Length': '0'} 2022-09-12T20:16:02.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4dd20422f70c58e-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2022-09-12T20:16:02.471 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:02.472 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:02.472 DEBUG:boto:path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:02.473 DEBUG:boto:auth_path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:02.473 DEBUG:boto:Method: GET 2022-09-12T20:16:02.474 DEBUG:boto:Path: /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:02.474 DEBUG:boto:Data: 2022-09-12T20:16:02.475 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.475 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.476 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.476 DEBUG:boto:Params: {} 2022-09-12T20:16:02.477 DEBUG:boto:Token: None 2022-09-12T20:16:02.477 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:02.478 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oEJ/ygj6s1XHxTRX4GiTu2hFrT4= 2022-09-12T20:16:02.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oEJ/ygj6s1XHxTRX4GiTu2hFrT4=', 'Content-Length': '0'} 2022-09-12T20:16:02.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080dafff6d04b7077-00631f9382-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.481 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:02.482 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:02.482 DEBUG:boto:path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:02.483 DEBUG:boto:auth_path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:02.483 DEBUG:boto:Method: GET 2022-09-12T20:16:02.484 DEBUG:boto:Path: /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:02.484 DEBUG:boto:Data: 2022-09-12T20:16:02.485 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.485 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.486 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.486 DEBUG:boto:Params: {} 2022-09-12T20:16:02.487 DEBUG:boto:Token: None 2022-09-12T20:16:02.488 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:02.488 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oEJ/ygj6s1XHxTRX4GiTu2hFrT4= 2022-09-12T20:16:02.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oEJ/ygj6s1XHxTRX4GiTu2hFrT4=', 'Content-Length': '0'} 2022-09-12T20:16:02.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007967c58332132749-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.492 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:02.492 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:02.493 DEBUG:boto:path=/jdpiak-75/?prefix=obj37 2022-09-12T20:16:02.493 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj37 2022-09-12T20:16:02.494 DEBUG:boto:Method: GET 2022-09-12T20:16:02.494 DEBUG:boto:Path: /jdpiak-75/?prefix=obj37 2022-09-12T20:16:02.495 DEBUG:boto:Data: 2022-09-12T20:16:02.495 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.496 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.496 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.497 DEBUG:boto:Params: {} 2022-09-12T20:16:02.497 DEBUG:boto:Token: None 2022-09-12T20:16:02.498 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/ 2022-09-12T20:16:02.499 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU= 2022-09-12T20:16:02.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU=', 'Content-Length': '0'} 2022-09-12T20:16:02.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f189a659c1edede8-00631f9382-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.503 DEBUG:boto:b'jdpiak-75obj371000falseobj372022-09-12T20:14:26.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:02.503 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:02.504 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:02.504 DEBUG:boto:path=/jdpiak-75/?prefix=obj37 2022-09-12T20:16:02.505 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj37 2022-09-12T20:16:02.505 DEBUG:boto:Method: GET 2022-09-12T20:16:02.506 DEBUG:boto:Path: /jdpiak-75/?prefix=obj37 2022-09-12T20:16:02.507 DEBUG:boto:Data: 2022-09-12T20:16:02.507 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.508 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.508 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.509 DEBUG:boto:Params: {} 2022-09-12T20:16:02.509 DEBUG:boto:Token: None 2022-09-12T20:16:02.510 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/ 2022-09-12T20:16:02.510 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU= 2022-09-12T20:16:02.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU=', 'Content-Length': '0'} 2022-09-12T20:16:02.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f7bbf172c8f07b5-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.517 DEBUG:boto:b'jdpiak-75obj371000falseobj372022-09-12T20:14:26.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:02.517 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:02.518 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:02.519 DEBUG:boto:Method: HEAD 2022-09-12T20:16:02.519 DEBUG:boto:Path: /jdpiak-75/obj37 2022-09-12T20:16:02.520 DEBUG:boto:Data: 2022-09-12T20:16:02.520 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.521 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.521 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.521 DEBUG:boto:Params: {} 2022-09-12T20:16:02.522 DEBUG:boto:Token: None 2022-09-12T20:16:02.523 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj37 2022-09-12T20:16:02.523 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A7o/6pGnCM1ahUA841f5FL3qGTc= 2022-09-12T20:16:02.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A7o/6pGnCM1ahUA841f5FL3qGTc=', 'Content-Length': '0'} 2022-09-12T20:16:02.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3784fa1918a022c-00631f9382-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.526 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:02.527 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:02.528 DEBUG:boto:Method: HEAD 2022-09-12T20:16:02.528 DEBUG:boto:Path: /jdpiak-75/obj37 2022-09-12T20:16:02.528 DEBUG:boto:Data: 2022-09-12T20:16:02.529 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.529 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.530 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.530 DEBUG:boto:Params: {} 2022-09-12T20:16:02.531 DEBUG:boto:Token: None 2022-09-12T20:16:02.532 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj37 2022-09-12T20:16:02.532 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A7o/6pGnCM1ahUA841f5FL3qGTc= 2022-09-12T20:16:02.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A7o/6pGnCM1ahUA841f5FL3qGTc=', 'Content-Length': '0'} 2022-09-12T20:16:02.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078c14fdbd3c06db3-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2022-09-12T20:16:02.536 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:02.536 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:02.537 DEBUG:boto:Method: GET 2022-09-12T20:16:02.537 DEBUG:boto:Path: /jdpiak-75/obj37 2022-09-12T20:16:02.538 DEBUG:boto:Data: 2022-09-12T20:16:02.538 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.539 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.540 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.540 DEBUG:boto:Params: {} 2022-09-12T20:16:02.541 DEBUG:boto:Token: None 2022-09-12T20:16:02.541 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj37 2022-09-12T20:16:02.542 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B58Kbs92WliyV9775l2/xsY6E8c= 2022-09-12T20:16:02.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B58Kbs92WliyV9775l2/xsY6E8c=', 'Content-Length': '0'} 2022-09-12T20:16:02.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5ed8af384c882de-00631f9382-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.545 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:02.546 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:02.546 DEBUG:boto:Method: GET 2022-09-12T20:16:02.547 DEBUG:boto:Path: /jdpiak-75/obj37 2022-09-12T20:16:02.547 DEBUG:boto:Data: 2022-09-12T20:16:02.548 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.548 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.549 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.549 DEBUG:boto:Params: {} 2022-09-12T20:16:02.550 DEBUG:boto:Token: None 2022-09-12T20:16:02.551 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj37 2022-09-12T20:16:02.551 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B58Kbs92WliyV9775l2/xsY6E8c= 2022-09-12T20:16:02.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B58Kbs92WliyV9775l2/xsY6E8c=', 'Content-Length': '0'} 2022-09-12T20:16:02.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da7aab736d351a7a-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.598 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:02.599 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:02.599 DEBUG:boto:path=/jdpiak-75/?prefix=obj37 2022-09-12T20:16:02.600 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj37 2022-09-12T20:16:02.600 DEBUG:boto:Method: GET 2022-09-12T20:16:02.601 DEBUG:boto:Path: /jdpiak-75/?prefix=obj37 2022-09-12T20:16:02.601 DEBUG:boto:Data: 2022-09-12T20:16:02.602 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.602 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.603 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.603 DEBUG:boto:Params: {} 2022-09-12T20:16:02.604 DEBUG:boto:Token: None 2022-09-12T20:16:02.605 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/ 2022-09-12T20:16:02.605 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU= 2022-09-12T20:16:02.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU=', 'Content-Length': '0'} 2022-09-12T20:16:02.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083890c6426f7135e-00631f9382-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.609 DEBUG:boto:b'jdpiak-75obj371000falseobj372022-09-12T20:14:26.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:02.610 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:02.611 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:02.611 DEBUG:boto:path=/jdpiak-75/?prefix=obj37 2022-09-12T20:16:02.612 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj37 2022-09-12T20:16:02.612 DEBUG:boto:Method: GET 2022-09-12T20:16:02.613 DEBUG:boto:Path: /jdpiak-75/?prefix=obj37 2022-09-12T20:16:02.613 DEBUG:boto:Data: 2022-09-12T20:16:02.614 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.614 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.615 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.615 DEBUG:boto:Params: {} 2022-09-12T20:16:02.616 DEBUG:boto:Token: None 2022-09-12T20:16:02.616 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/ 2022-09-12T20:16:02.617 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU= 2022-09-12T20:16:02.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU=', 'Content-Length': '0'} 2022-09-12T20:16:02.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000775e69eed181aec5-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.623 DEBUG:boto:b'jdpiak-75obj371000falseobj372022-09-12T20:14:26.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:02.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:16:02.625 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:02.625 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:02.626 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:02.626 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:02.627 DEBUG:boto:Method: GET 2022-09-12T20:16:02.627 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:02.628 DEBUG:boto:Data: 2022-09-12T20:16:02.628 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.629 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.629 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.630 DEBUG:boto:Params: {} 2022-09-12T20:16:02.630 DEBUG:boto:Token: None 2022-09-12T20:16:02.631 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:02.632 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FtXHOuw7RUK5XIL0hb6/9tAT4Jw= 2022-09-12T20:16:02.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FtXHOuw7RUK5XIL0hb6/9tAT4Jw=', 'Content-Length': '0'} 2022-09-12T20:16:02.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007329a9e69bfdce7f-00631f9382-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.635 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:02.636 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:02.636 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:02.637 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:02.637 DEBUG:boto:Method: GET 2022-09-12T20:16:02.638 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:02.638 DEBUG:boto:Data: 2022-09-12T20:16:02.639 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.639 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.640 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.640 DEBUG:boto:Params: {} 2022-09-12T20:16:02.641 DEBUG:boto:Token: None 2022-09-12T20:16:02.642 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:02.642 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FtXHOuw7RUK5XIL0hb6/9tAT4Jw= 2022-09-12T20:16:02.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FtXHOuw7RUK5XIL0hb6/9tAT4Jw=', 'Content-Length': '0'} 2022-09-12T20:16:02.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000361f943e4dfd9061-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.646 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:02.647 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:02.647 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:02.648 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:02.648 DEBUG:boto:Method: HEAD 2022-09-12T20:16:02.649 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:02.649 DEBUG:boto:Data: 2022-09-12T20:16:02.650 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.657 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.658 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.659 DEBUG:boto:Params: {} 2022-09-12T20:16:02.659 DEBUG:boto:Token: None 2022-09-12T20:16:02.660 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:02.660 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q7TSoA0wONF7+8esSlGCnKbCqmI= 2022-09-12T20:16:02.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q7TSoA0wONF7+8esSlGCnKbCqmI=', 'Content-Length': '0'} 2022-09-12T20:16:02.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e12590e0ce027b78-00631f9382-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.664 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:02.664 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:02.665 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:02.665 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:02.666 DEBUG:boto:Method: HEAD 2022-09-12T20:16:02.666 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:02.667 DEBUG:boto:Data: 2022-09-12T20:16:02.667 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.668 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.668 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.669 DEBUG:boto:Params: {} 2022-09-12T20:16:02.669 DEBUG:boto:Token: None 2022-09-12T20:16:02.670 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:02.670 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q7TSoA0wONF7+8esSlGCnKbCqmI= 2022-09-12T20:16:02.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q7TSoA0wONF7+8esSlGCnKbCqmI=', 'Content-Length': '0'} 2022-09-12T20:16:02.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fd5f965327a70bd-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:16:02.674 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:02.675 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:02.675 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:02.676 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:02.676 DEBUG:boto:Method: GET 2022-09-12T20:16:02.677 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:02.677 DEBUG:boto:Data: 2022-09-12T20:16:02.678 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.679 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.679 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.680 DEBUG:boto:Params: {} 2022-09-12T20:16:02.680 DEBUG:boto:Token: None 2022-09-12T20:16:02.681 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:02.681 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FtXHOuw7RUK5XIL0hb6/9tAT4Jw= 2022-09-12T20:16:02.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FtXHOuw7RUK5XIL0hb6/9tAT4Jw=', 'Content-Length': '0'} 2022-09-12T20:16:02.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004747142ff9e1b595-00631f9382-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.726 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:02.727 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:02.728 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:02.728 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:02.729 DEBUG:boto:Method: GET 2022-09-12T20:16:02.729 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:02.730 DEBUG:boto:Data: 2022-09-12T20:16:02.730 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.731 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.731 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.732 DEBUG:boto:Params: {} 2022-09-12T20:16:02.732 DEBUG:boto:Token: None 2022-09-12T20:16:02.733 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:02.734 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FtXHOuw7RUK5XIL0hb6/9tAT4Jw= 2022-09-12T20:16:02.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FtXHOuw7RUK5XIL0hb6/9tAT4Jw=', 'Content-Length': '0'} 2022-09-12T20:16:02.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000579f442853ebcdae-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.738 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:02.738 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:02.739 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:16:02.739 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:16:02.740 DEBUG:boto:Method: GET 2022-09-12T20:16:02.740 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:16:02.741 DEBUG:boto:Data: 2022-09-12T20:16:02.741 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.742 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.742 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.743 DEBUG:boto:Params: {} 2022-09-12T20:16:02.744 DEBUG:boto:Token: None 2022-09-12T20:16:02.745 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/ 2022-09-12T20:16:02.746 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU= 2022-09-12T20:16:02.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU=', 'Content-Length': '0'} 2022-09-12T20:16:02.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007477b685ed031471-00631f9382-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.750 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:02.751 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:02.751 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:02.752 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:16:02.752 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:16:02.753 DEBUG:boto:Method: GET 2022-09-12T20:16:02.753 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:16:02.754 DEBUG:boto:Data: 2022-09-12T20:16:02.755 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.755 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.756 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.756 DEBUG:boto:Params: {} 2022-09-12T20:16:02.757 DEBUG:boto:Token: None 2022-09-12T20:16:02.758 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/ 2022-09-12T20:16:02.758 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU= 2022-09-12T20:16:02.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU=', 'Content-Length': '0'} 2022-09-12T20:16:02.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7df74beff594c99-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.764 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:02.765 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:02.766 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:02.766 DEBUG:boto:Method: HEAD 2022-09-12T20:16:02.767 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:16:02.767 DEBUG:boto:Data: 2022-09-12T20:16:02.768 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.768 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.769 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.769 DEBUG:boto:Params: {} 2022-09-12T20:16:02.770 DEBUG:boto:Token: None 2022-09-12T20:16:02.771 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj38 2022-09-12T20:16:02.771 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QWEnpg8LywdJfD0lTSbGw/FzNRc= 2022-09-12T20:16:02.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QWEnpg8LywdJfD0lTSbGw/FzNRc=', 'Content-Length': '0'} 2022-09-12T20:16:02.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007734648d449b3beb-00631f9382-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.775 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:02.775 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:02.776 DEBUG:boto:Method: HEAD 2022-09-12T20:16:02.776 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:16:02.777 DEBUG:boto:Data: 2022-09-12T20:16:02.777 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.778 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.778 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.779 DEBUG:boto:Params: {} 2022-09-12T20:16:02.780 DEBUG:boto:Token: None 2022-09-12T20:16:02.780 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj38 2022-09-12T20:16:02.781 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QWEnpg8LywdJfD0lTSbGw/FzNRc= 2022-09-12T20:16:02.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QWEnpg8LywdJfD0lTSbGw/FzNRc=', 'Content-Length': '0'} 2022-09-12T20:16:02.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5e7d69037d2bf0c-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:16:02.785 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:02.786 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:02.786 DEBUG:boto:Method: GET 2022-09-12T20:16:02.787 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:16:02.787 DEBUG:boto:Data: 2022-09-12T20:16:02.788 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.788 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.789 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.790 DEBUG:boto:Params: {} 2022-09-12T20:16:02.790 DEBUG:boto:Token: None 2022-09-12T20:16:02.791 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj38 2022-09-12T20:16:02.791 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7R1/xV4zR0IpHnAX2gv9ZaazymM= 2022-09-12T20:16:02.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7R1/xV4zR0IpHnAX2gv9ZaazymM=', 'Content-Length': '0'} 2022-09-12T20:16:02.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bfea3d1a5719822-00631f9382-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.795 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:02.796 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:02.796 DEBUG:boto:Method: GET 2022-09-12T20:16:02.797 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:16:02.797 DEBUG:boto:Data: 2022-09-12T20:16:02.798 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.799 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.799 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.800 DEBUG:boto:Params: {} 2022-09-12T20:16:02.800 DEBUG:boto:Token: None 2022-09-12T20:16:02.801 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj38 2022-09-12T20:16:02.801 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7R1/xV4zR0IpHnAX2gv9ZaazymM= 2022-09-12T20:16:02.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7R1/xV4zR0IpHnAX2gv9ZaazymM=', 'Content-Length': '0'} 2022-09-12T20:16:02.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cffab7c3ea85d07a-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.846 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:02.847 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:02.848 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:16:02.848 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:16:02.849 DEBUG:boto:Method: GET 2022-09-12T20:16:02.849 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:16:02.850 DEBUG:boto:Data: 2022-09-12T20:16:02.850 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.851 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.851 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.852 DEBUG:boto:Params: {} 2022-09-12T20:16:02.853 DEBUG:boto:Token: None 2022-09-12T20:16:02.853 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/ 2022-09-12T20:16:02.854 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU= 2022-09-12T20:16:02.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU=', 'Content-Length': '0'} 2022-09-12T20:16:02.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d741838c3f445966-00631f9382-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.858 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:02.859 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:02.859 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:02.860 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:16:02.860 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:16:02.861 DEBUG:boto:Method: GET 2022-09-12T20:16:02.861 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:16:02.862 DEBUG:boto:Data: 2022-09-12T20:16:02.862 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.863 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.863 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.864 DEBUG:boto:Params: {} 2022-09-12T20:16:02.864 DEBUG:boto:Token: None 2022-09-12T20:16:02.865 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/ 2022-09-12T20:16:02.866 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU= 2022-09-12T20:16:02.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU=', 'Content-Length': '0'} 2022-09-12T20:16:02.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca9e570b98536c25-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.871 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:02.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:16:02.873 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:02.873 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:02.874 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:02.874 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:02.875 DEBUG:boto:Method: GET 2022-09-12T20:16:02.876 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:02.876 DEBUG:boto:Data: 2022-09-12T20:16:02.877 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.877 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.878 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.878 DEBUG:boto:Params: {} 2022-09-12T20:16:02.879 DEBUG:boto:Token: None 2022-09-12T20:16:02.879 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:02.880 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:phECpVhjBpNdb+cEM5GUrIbsHTM= 2022-09-12T20:16:02.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:phECpVhjBpNdb+cEM5GUrIbsHTM=', 'Content-Length': '0'} 2022-09-12T20:16:02.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004848e8b23427aa31-00631f9382-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.883 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:02.884 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:02.885 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:02.885 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:02.886 DEBUG:boto:Method: GET 2022-09-12T20:16:02.886 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:02.887 DEBUG:boto:Data: 2022-09-12T20:16:02.887 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.888 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.888 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.889 DEBUG:boto:Params: {} 2022-09-12T20:16:02.889 DEBUG:boto:Token: None 2022-09-12T20:16:02.890 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:02.891 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:phECpVhjBpNdb+cEM5GUrIbsHTM= 2022-09-12T20:16:02.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:phECpVhjBpNdb+cEM5GUrIbsHTM=', 'Content-Length': '0'} 2022-09-12T20:16:02.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084165e03209508e5-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.894 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:02.895 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:02.895 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:02.896 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:02.896 DEBUG:boto:Method: HEAD 2022-09-12T20:16:02.897 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:02.897 DEBUG:boto:Data: 2022-09-12T20:16:02.898 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.898 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.899 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.899 DEBUG:boto:Params: {} 2022-09-12T20:16:02.900 DEBUG:boto:Token: None 2022-09-12T20:16:02.901 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:02.901 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dkcfSVV0GSowRIyKaS6TIiYgV+Y= 2022-09-12T20:16:02.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dkcfSVV0GSowRIyKaS6TIiYgV+Y=', 'Content-Length': '0'} 2022-09-12T20:16:02.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000664d972f46cc71c3-00631f9382-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.905 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:02.905 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:02.906 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:02.906 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:02.907 DEBUG:boto:Method: HEAD 2022-09-12T20:16:02.907 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:02.908 DEBUG:boto:Data: 2022-09-12T20:16:02.908 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.909 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.909 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.910 DEBUG:boto:Params: {} 2022-09-12T20:16:02.910 DEBUG:boto:Token: None 2022-09-12T20:16:02.911 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:02.912 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dkcfSVV0GSowRIyKaS6TIiYgV+Y= 2022-09-12T20:16:02.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dkcfSVV0GSowRIyKaS6TIiYgV+Y=', 'Content-Length': '0'} 2022-09-12T20:16:02.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cdcd260777ce2b5-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:16:02.916 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:02.916 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:02.917 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:02.917 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:02.918 DEBUG:boto:Method: GET 2022-09-12T20:16:02.918 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:02.919 DEBUG:boto:Data: 2022-09-12T20:16:02.919 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.920 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.920 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.921 DEBUG:boto:Params: {} 2022-09-12T20:16:02.922 DEBUG:boto:Token: None 2022-09-12T20:16:02.922 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:02.923 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:phECpVhjBpNdb+cEM5GUrIbsHTM= 2022-09-12T20:16:02.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:phECpVhjBpNdb+cEM5GUrIbsHTM=', 'Content-Length': '0'} 2022-09-12T20:16:02.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a032bb2a4489dab-00631f9382-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.966 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:02.967 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:02.967 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:02.968 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:02.968 DEBUG:boto:Method: GET 2022-09-12T20:16:02.969 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:02.969 DEBUG:boto:Data: 2022-09-12T20:16:02.970 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.971 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.971 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.972 DEBUG:boto:Params: {} 2022-09-12T20:16:02.972 DEBUG:boto:Token: None 2022-09-12T20:16:02.973 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:02.973 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:phECpVhjBpNdb+cEM5GUrIbsHTM= 2022-09-12T20:16:02.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:phECpVhjBpNdb+cEM5GUrIbsHTM=', 'Content-Length': '0'} 2022-09-12T20:16:02.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e572bfb60b6d728-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.977 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:02.978 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:02.978 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:16:02.979 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:16:02.979 DEBUG:boto:Method: GET 2022-09-12T20:16:02.980 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:16:02.980 DEBUG:boto:Data: 2022-09-12T20:16:02.981 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.981 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:02.982 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.982 DEBUG:boto:Params: {} 2022-09-12T20:16:02.983 DEBUG:boto:Token: None 2022-09-12T20:16:02.984 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/ 2022-09-12T20:16:02.984 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU= 2022-09-12T20:16:02.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU=', 'Content-Length': '0'} 2022-09-12T20:16:02.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000875cabfb0ccd7df1-00631f9382-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:02.988 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:02.989 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:02.990 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:02.990 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:16:02.991 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:16:02.991 DEBUG:boto:Method: GET 2022-09-12T20:16:02.992 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:16:02.992 DEBUG:boto:Data: 2022-09-12T20:16:02.993 DEBUG:boto:Headers: {} 2022-09-12T20:16:02.993 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:02.994 DEBUG:boto:Port: 8000 2022-09-12T20:16:02.994 DEBUG:boto:Params: {} 2022-09-12T20:16:02.995 DEBUG:boto:Token: None 2022-09-12T20:16:02.996 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:02 GMT /jdpiak-75/ 2022-09-12T20:16:02.996 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU= 2022-09-12T20:16:02.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XaSF7YuLo8mhz7fW4yz/4uxDogU=', 'Content-Length': '0'} 2022-09-12T20:16:03.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e4ce33cfa8236f3-00631f9382-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.003 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:03.003 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:03.004 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:03.004 DEBUG:boto:Method: HEAD 2022-09-12T20:16:03.005 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:16:03.006 DEBUG:boto:Data: 2022-09-12T20:16:03.006 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.007 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.007 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.008 DEBUG:boto:Params: {} 2022-09-12T20:16:03.008 DEBUG:boto:Token: None 2022-09-12T20:16:03.009 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj44 2022-09-12T20:16:03.009 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MgzrHW8DD9LoZMyyZLj+aalbqT0= 2022-09-12T20:16:03.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MgzrHW8DD9LoZMyyZLj+aalbqT0=', 'Content-Length': '0'} 2022-09-12T20:16:03.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bff48a4b9f77b00e-00631f9383-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.012 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:03.013 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:03.013 DEBUG:boto:Method: HEAD 2022-09-12T20:16:03.014 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:16:03.015 DEBUG:boto:Data: 2022-09-12T20:16:03.015 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.016 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.016 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.017 DEBUG:boto:Params: {} 2022-09-12T20:16:03.017 DEBUG:boto:Token: None 2022-09-12T20:16:03.018 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj44 2022-09-12T20:16:03.018 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MgzrHW8DD9LoZMyyZLj+aalbqT0= 2022-09-12T20:16:03.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MgzrHW8DD9LoZMyyZLj+aalbqT0=', 'Content-Length': '0'} 2022-09-12T20:16:03.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005234c304c30d495d-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:16:03.023 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:03.023 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:03.024 DEBUG:boto:Method: GET 2022-09-12T20:16:03.024 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:16:03.025 DEBUG:boto:Data: 2022-09-12T20:16:03.025 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.026 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.026 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.027 DEBUG:boto:Params: {} 2022-09-12T20:16:03.028 DEBUG:boto:Token: None 2022-09-12T20:16:03.028 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj44 2022-09-12T20:16:03.029 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UJ5wCZ4yoDoD7K+YP8GEaes9U3Y= 2022-09-12T20:16:03.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UJ5wCZ4yoDoD7K+YP8GEaes9U3Y=', 'Content-Length': '0'} 2022-09-12T20:16:03.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022421ef43cf474f7-00631f9383-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.032 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:03.033 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:03.034 DEBUG:boto:Method: GET 2022-09-12T20:16:03.034 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:16:03.035 DEBUG:boto:Data: 2022-09-12T20:16:03.035 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.036 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.036 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.037 DEBUG:boto:Params: {} 2022-09-12T20:16:03.037 DEBUG:boto:Token: None 2022-09-12T20:16:03.038 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj44 2022-09-12T20:16:03.039 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UJ5wCZ4yoDoD7K+YP8GEaes9U3Y= 2022-09-12T20:16:03.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UJ5wCZ4yoDoD7K+YP8GEaes9U3Y=', 'Content-Length': '0'} 2022-09-12T20:16:03.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b65548ea58faa07-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.082 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:03.084 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:03.085 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:16:03.086 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:16:03.087 DEBUG:boto:Method: GET 2022-09-12T20:16:03.088 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:16:03.089 DEBUG:boto:Data: 2022-09-12T20:16:03.090 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.091 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.093 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.094 DEBUG:boto:Params: {} 2022-09-12T20:16:03.095 DEBUG:boto:Token: None 2022-09-12T20:16:03.096 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/ 2022-09-12T20:16:03.097 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI= 2022-09-12T20:16:03.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI=', 'Content-Length': '0'} 2022-09-12T20:16:03.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b288d09f9e50b0ec-00631f9383-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.104 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:03.105 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:03.106 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:03.107 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:16:03.108 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:16:03.109 DEBUG:boto:Method: GET 2022-09-12T20:16:03.110 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:16:03.112 DEBUG:boto:Data: 2022-09-12T20:16:03.113 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.114 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.115 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.116 DEBUG:boto:Params: {} 2022-09-12T20:16:03.117 DEBUG:boto:Token: None 2022-09-12T20:16:03.118 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/ 2022-09-12T20:16:03.119 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI= 2022-09-12T20:16:03.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI=', 'Content-Length': '0'} 2022-09-12T20:16:03.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb9aaaa8800cef8a-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.128 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:03.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:16:03.129 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:03.130 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:03.131 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:03.132 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:03.133 DEBUG:boto:Method: GET 2022-09-12T20:16:03.133 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:03.134 DEBUG:boto:Data: 2022-09-12T20:16:03.135 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.136 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.137 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.138 DEBUG:boto:Params: {} 2022-09-12T20:16:03.139 DEBUG:boto:Token: None 2022-09-12T20:16:03.140 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:03.140 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zUJDLVHE4R2xckNMtDydyjM95F8= 2022-09-12T20:16:03.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zUJDLVHE4R2xckNMtDydyjM95F8=', 'Content-Length': '0'} 2022-09-12T20:16:03.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f340295a25b8cda9-00631f9383-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.144 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:03.145 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:03.145 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:03.146 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:03.146 DEBUG:boto:Method: GET 2022-09-12T20:16:03.147 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:03.148 DEBUG:boto:Data: 2022-09-12T20:16:03.148 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.149 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.150 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.151 DEBUG:boto:Params: {} 2022-09-12T20:16:03.151 DEBUG:boto:Token: None 2022-09-12T20:16:03.152 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:03.153 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zUJDLVHE4R2xckNMtDydyjM95F8= 2022-09-12T20:16:03.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zUJDLVHE4R2xckNMtDydyjM95F8=', 'Content-Length': '0'} 2022-09-12T20:16:03.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6128d617fb87a38-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.158 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:03.159 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:03.159 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:03.160 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:03.161 DEBUG:boto:Method: HEAD 2022-09-12T20:16:03.161 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:03.162 DEBUG:boto:Data: 2022-09-12T20:16:03.162 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.163 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.163 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.164 DEBUG:boto:Params: {} 2022-09-12T20:16:03.164 DEBUG:boto:Token: None 2022-09-12T20:16:03.165 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:03.166 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sR+PNYhec4bxSC1PcFR0vFFBU38= 2022-09-12T20:16:03.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sR+PNYhec4bxSC1PcFR0vFFBU38=', 'Content-Length': '0'} 2022-09-12T20:16:03.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c8578228b0ec060-00631f9383-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.170 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:03.170 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:03.171 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:03.171 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:03.172 DEBUG:boto:Method: HEAD 2022-09-12T20:16:03.173 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:03.173 DEBUG:boto:Data: 2022-09-12T20:16:03.174 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.174 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.175 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.176 DEBUG:boto:Params: {} 2022-09-12T20:16:03.177 DEBUG:boto:Token: None 2022-09-12T20:16:03.178 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:03.179 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sR+PNYhec4bxSC1PcFR0vFFBU38= 2022-09-12T20:16:03.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sR+PNYhec4bxSC1PcFR0vFFBU38=', 'Content-Length': '0'} 2022-09-12T20:16:03.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c93da928b96d6de-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:16:03.183 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:03.184 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:03.184 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:03.185 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:03.185 DEBUG:boto:Method: GET 2022-09-12T20:16:03.186 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:03.186 DEBUG:boto:Data: 2022-09-12T20:16:03.187 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.188 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.188 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.189 DEBUG:boto:Params: {} 2022-09-12T20:16:03.189 DEBUG:boto:Token: None 2022-09-12T20:16:03.190 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:03.191 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zUJDLVHE4R2xckNMtDydyjM95F8= 2022-09-12T20:16:03.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zUJDLVHE4R2xckNMtDydyjM95F8=', 'Content-Length': '0'} 2022-09-12T20:16:03.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c0ac130bbf96095-00631f9383-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.194 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:03.195 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:03.195 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:03.196 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:03.196 DEBUG:boto:Method: GET 2022-09-12T20:16:03.197 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:03.198 DEBUG:boto:Data: 2022-09-12T20:16:03.198 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.199 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.199 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.200 DEBUG:boto:Params: {} 2022-09-12T20:16:03.200 DEBUG:boto:Token: None 2022-09-12T20:16:03.201 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:03.202 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zUJDLVHE4R2xckNMtDydyjM95F8= 2022-09-12T20:16:03.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zUJDLVHE4R2xckNMtDydyjM95F8=', 'Content-Length': '0'} 2022-09-12T20:16:03.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006385f427d0aa783e-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.205 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:03.206 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:03.206 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:16:03.207 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:16:03.207 DEBUG:boto:Method: GET 2022-09-12T20:16:03.208 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:16:03.208 DEBUG:boto:Data: 2022-09-12T20:16:03.209 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.209 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.210 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.210 DEBUG:boto:Params: {} 2022-09-12T20:16:03.211 DEBUG:boto:Token: None 2022-09-12T20:16:03.212 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/ 2022-09-12T20:16:03.212 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI= 2022-09-12T20:16:03.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI=', 'Content-Length': '0'} 2022-09-12T20:16:03.216 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a877a09369c29f21-00631f9383-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.216 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:03.217 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:03.218 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:03.218 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:16:03.219 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:16:03.219 DEBUG:boto:Method: GET 2022-09-12T20:16:03.220 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:16:03.220 DEBUG:boto:Data: 2022-09-12T20:16:03.221 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.221 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.222 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.222 DEBUG:boto:Params: {} 2022-09-12T20:16:03.223 DEBUG:boto:Token: None 2022-09-12T20:16:03.224 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/ 2022-09-12T20:16:03.224 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI= 2022-09-12T20:16:03.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI=', 'Content-Length': '0'} 2022-09-12T20:16:03.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068cdcbea1ad76362-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.230 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:03.231 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:03.232 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:03.232 DEBUG:boto:Method: HEAD 2022-09-12T20:16:03.233 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:16:03.233 DEBUG:boto:Data: 2022-09-12T20:16:03.234 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.234 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.235 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.235 DEBUG:boto:Params: {} 2022-09-12T20:16:03.236 DEBUG:boto:Token: None 2022-09-12T20:16:03.236 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj46 2022-09-12T20:16:03.237 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l84uMkkKIUfj+8LsTvqOg9UbGJM= 2022-09-12T20:16:03.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l84uMkkKIUfj+8LsTvqOg9UbGJM=', 'Content-Length': '0'} 2022-09-12T20:16:03.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ad40aa0a2bb80a6-00631f9383-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.240 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:03.241 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:03.241 DEBUG:boto:Method: HEAD 2022-09-12T20:16:03.242 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:16:03.242 DEBUG:boto:Data: 2022-09-12T20:16:03.243 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.243 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.244 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.244 DEBUG:boto:Params: {} 2022-09-12T20:16:03.245 DEBUG:boto:Token: None 2022-09-12T20:16:03.245 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj46 2022-09-12T20:16:03.246 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:l84uMkkKIUfj+8LsTvqOg9UbGJM= 2022-09-12T20:16:03.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:l84uMkkKIUfj+8LsTvqOg9UbGJM=', 'Content-Length': '0'} 2022-09-12T20:16:03.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad38121c437c4ef5-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:16:03.250 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:03.251 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:03.251 DEBUG:boto:Method: GET 2022-09-12T20:16:03.252 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:16:03.252 DEBUG:boto:Data: 2022-09-12T20:16:03.253 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.253 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.254 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.255 DEBUG:boto:Params: {} 2022-09-12T20:16:03.255 DEBUG:boto:Token: None 2022-09-12T20:16:03.256 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj46 2022-09-12T20:16:03.256 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fUnv/Ost0YOyKLIJMty844ehOMc= 2022-09-12T20:16:03.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fUnv/Ost0YOyKLIJMty844ehOMc=', 'Content-Length': '0'} 2022-09-12T20:16:03.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b8e79f4df990454-00631f9383-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.302 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:03.303 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:03.303 DEBUG:boto:Method: GET 2022-09-12T20:16:03.304 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:16:03.304 DEBUG:boto:Data: 2022-09-12T20:16:03.305 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.306 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.306 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.307 DEBUG:boto:Params: {} 2022-09-12T20:16:03.307 DEBUG:boto:Token: None 2022-09-12T20:16:03.308 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj46 2022-09-12T20:16:03.309 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fUnv/Ost0YOyKLIJMty844ehOMc= 2022-09-12T20:16:03.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fUnv/Ost0YOyKLIJMty844ehOMc=', 'Content-Length': '0'} 2022-09-12T20:16:03.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e8ef3e67c23e24f-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.312 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:03.313 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:03.313 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:16:03.314 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:16:03.314 DEBUG:boto:Method: GET 2022-09-12T20:16:03.315 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:16:03.315 DEBUG:boto:Data: 2022-09-12T20:16:03.316 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.316 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.317 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.317 DEBUG:boto:Params: {} 2022-09-12T20:16:03.318 DEBUG:boto:Token: None 2022-09-12T20:16:03.319 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/ 2022-09-12T20:16:03.319 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI= 2022-09-12T20:16:03.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI=', 'Content-Length': '0'} 2022-09-12T20:16:03.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098b51f01533b231d-00631f9383-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.323 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:03.324 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:03.325 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:03.325 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:16:03.326 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:16:03.326 DEBUG:boto:Method: GET 2022-09-12T20:16:03.327 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:16:03.327 DEBUG:boto:Data: 2022-09-12T20:16:03.328 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.329 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.329 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.330 DEBUG:boto:Params: {} 2022-09-12T20:16:03.330 DEBUG:boto:Token: None 2022-09-12T20:16:03.331 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/ 2022-09-12T20:16:03.331 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI= 2022-09-12T20:16:03.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI=', 'Content-Length': '0'} 2022-09-12T20:16:03.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008049e4cd65735963-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.338 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:03.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:16:03.339 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:03.340 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:03.340 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:03.341 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:03.342 DEBUG:boto:Method: GET 2022-09-12T20:16:03.342 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:03.343 DEBUG:boto:Data: 2022-09-12T20:16:03.343 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.344 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.344 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.345 DEBUG:boto:Params: {} 2022-09-12T20:16:03.345 DEBUG:boto:Token: None 2022-09-12T20:16:03.346 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:03.347 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gA8t1W5EN1wOL3tdkhaq4jR5be8= 2022-09-12T20:16:03.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gA8t1W5EN1wOL3tdkhaq4jR5be8=', 'Content-Length': '0'} 2022-09-12T20:16:03.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf200f4a471aeffb-00631f9383-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.351 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:03.351 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:03.352 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:03.353 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:03.354 DEBUG:boto:Method: GET 2022-09-12T20:16:03.354 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:03.355 DEBUG:boto:Data: 2022-09-12T20:16:03.356 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.356 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.357 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.357 DEBUG:boto:Params: {} 2022-09-12T20:16:03.358 DEBUG:boto:Token: None 2022-09-12T20:16:03.358 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:03.359 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gA8t1W5EN1wOL3tdkhaq4jR5be8= 2022-09-12T20:16:03.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gA8t1W5EN1wOL3tdkhaq4jR5be8=', 'Content-Length': '0'} 2022-09-12T20:16:03.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a6ed60d5b93252a-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.363 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:03.364 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:03.364 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:03.365 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:03.365 DEBUG:boto:Method: HEAD 2022-09-12T20:16:03.366 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:03.366 DEBUG:boto:Data: 2022-09-12T20:16:03.367 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.367 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.368 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.368 DEBUG:boto:Params: {} 2022-09-12T20:16:03.369 DEBUG:boto:Token: None 2022-09-12T20:16:03.369 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:03.370 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zz3Exne9Wd0yUDr5yXYVosUWvK0= 2022-09-12T20:16:03.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zz3Exne9Wd0yUDr5yXYVosUWvK0=', 'Content-Length': '0'} 2022-09-12T20:16:03.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c4514548560832a-00631f9383-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.373 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:03.374 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:03.375 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:03.375 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:03.376 DEBUG:boto:Method: HEAD 2022-09-12T20:16:03.376 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:03.377 DEBUG:boto:Data: 2022-09-12T20:16:03.377 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.378 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.378 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.379 DEBUG:boto:Params: {} 2022-09-12T20:16:03.379 DEBUG:boto:Token: None 2022-09-12T20:16:03.380 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:03.381 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zz3Exne9Wd0yUDr5yXYVosUWvK0= 2022-09-12T20:16:03.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zz3Exne9Wd0yUDr5yXYVosUWvK0=', 'Content-Length': '0'} 2022-09-12T20:16:03.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3c667f0fe8d2441-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:16:03.385 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:03.386 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:03.386 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:03.387 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:03.387 DEBUG:boto:Method: GET 2022-09-12T20:16:03.388 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:03.389 DEBUG:boto:Data: 2022-09-12T20:16:03.389 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.390 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.390 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.391 DEBUG:boto:Params: {} 2022-09-12T20:16:03.391 DEBUG:boto:Token: None 2022-09-12T20:16:03.392 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:03.393 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gA8t1W5EN1wOL3tdkhaq4jR5be8= 2022-09-12T20:16:03.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gA8t1W5EN1wOL3tdkhaq4jR5be8=', 'Content-Length': '0'} 2022-09-12T20:16:03.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a12c4583be611505-00631f9383-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.396 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:03.397 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:03.397 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:03.398 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:03.398 DEBUG:boto:Method: GET 2022-09-12T20:16:03.399 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:03.400 DEBUG:boto:Data: 2022-09-12T20:16:03.400 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.401 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.401 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.402 DEBUG:boto:Params: {} 2022-09-12T20:16:03.402 DEBUG:boto:Token: None 2022-09-12T20:16:03.403 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:03.404 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gA8t1W5EN1wOL3tdkhaq4jR5be8= 2022-09-12T20:16:03.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gA8t1W5EN1wOL3tdkhaq4jR5be8=', 'Content-Length': '0'} 2022-09-12T20:16:03.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009863ac350df14e2c-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.450 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:03.451 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:03.451 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:16:03.452 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:16:03.452 DEBUG:boto:Method: GET 2022-09-12T20:16:03.453 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:16:03.454 DEBUG:boto:Data: 2022-09-12T20:16:03.454 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.455 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.455 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.456 DEBUG:boto:Params: {} 2022-09-12T20:16:03.456 DEBUG:boto:Token: None 2022-09-12T20:16:03.457 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/ 2022-09-12T20:16:03.458 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI= 2022-09-12T20:16:03.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI=', 'Content-Length': '0'} 2022-09-12T20:16:03.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000141cb605017182b9-00631f9383-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.462 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:03.463 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:03.463 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:03.464 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:16:03.464 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:16:03.465 DEBUG:boto:Method: GET 2022-09-12T20:16:03.465 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:16:03.466 DEBUG:boto:Data: 2022-09-12T20:16:03.466 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.467 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.467 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.468 DEBUG:boto:Params: {} 2022-09-12T20:16:03.469 DEBUG:boto:Token: None 2022-09-12T20:16:03.469 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/ 2022-09-12T20:16:03.470 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI= 2022-09-12T20:16:03.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI=', 'Content-Length': '0'} 2022-09-12T20:16:03.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047892da060016038-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.476 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:03.477 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:03.477 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:03.478 DEBUG:boto:Method: HEAD 2022-09-12T20:16:03.478 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:16:03.479 DEBUG:boto:Data: 2022-09-12T20:16:03.479 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.480 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.480 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.481 DEBUG:boto:Params: {} 2022-09-12T20:16:03.481 DEBUG:boto:Token: None 2022-09-12T20:16:03.482 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj51 2022-09-12T20:16:03.483 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7MFOyDg7bePpUUQW7+Ifa7Sh8Os= 2022-09-12T20:16:03.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7MFOyDg7bePpUUQW7+Ifa7Sh8Os=', 'Content-Length': '0'} 2022-09-12T20:16:03.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095d7e4834a59bdc1-00631f9383-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.486 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:03.486 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:03.487 DEBUG:boto:Method: HEAD 2022-09-12T20:16:03.488 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:16:03.488 DEBUG:boto:Data: 2022-09-12T20:16:03.489 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.489 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.490 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.490 DEBUG:boto:Params: {} 2022-09-12T20:16:03.491 DEBUG:boto:Token: None 2022-09-12T20:16:03.491 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj51 2022-09-12T20:16:03.492 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7MFOyDg7bePpUUQW7+Ifa7Sh8Os= 2022-09-12T20:16:03.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7MFOyDg7bePpUUQW7+Ifa7Sh8Os=', 'Content-Length': '0'} 2022-09-12T20:16:03.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c306b4944e5b4750-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:16:03.496 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:03.497 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:03.497 DEBUG:boto:Method: GET 2022-09-12T20:16:03.498 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:16:03.499 DEBUG:boto:Data: 2022-09-12T20:16:03.499 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.500 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.500 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.501 DEBUG:boto:Params: {} 2022-09-12T20:16:03.501 DEBUG:boto:Token: None 2022-09-12T20:16:03.502 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj51 2022-09-12T20:16:03.502 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FzAEzLPJUvLcL2evlmeAfHgnmFU= 2022-09-12T20:16:03.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FzAEzLPJUvLcL2evlmeAfHgnmFU=', 'Content-Length': '0'} 2022-09-12T20:16:03.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e752b8ae3276a21-00631f9383-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.506 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:03.506 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:03.507 DEBUG:boto:Method: GET 2022-09-12T20:16:03.508 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:16:03.508 DEBUG:boto:Data: 2022-09-12T20:16:03.509 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.509 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.510 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.510 DEBUG:boto:Params: {} 2022-09-12T20:16:03.511 DEBUG:boto:Token: None 2022-09-12T20:16:03.511 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj51 2022-09-12T20:16:03.512 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FzAEzLPJUvLcL2evlmeAfHgnmFU= 2022-09-12T20:16:03.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FzAEzLPJUvLcL2evlmeAfHgnmFU=', 'Content-Length': '0'} 2022-09-12T20:16:03.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4bd395f93113aaf-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.558 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:03.559 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:03.559 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:16:03.560 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:16:03.560 DEBUG:boto:Method: GET 2022-09-12T20:16:03.561 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:16:03.562 DEBUG:boto:Data: 2022-09-12T20:16:03.562 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.563 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.563 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.564 DEBUG:boto:Params: {} 2022-09-12T20:16:03.565 DEBUG:boto:Token: None 2022-09-12T20:16:03.565 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/ 2022-09-12T20:16:03.566 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI= 2022-09-12T20:16:03.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI=', 'Content-Length': '0'} 2022-09-12T20:16:03.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e4e91d257573bdc-00631f9383-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.570 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:03.571 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:03.572 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:03.572 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:16:03.573 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:16:03.573 DEBUG:boto:Method: GET 2022-09-12T20:16:03.574 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:16:03.574 DEBUG:boto:Data: 2022-09-12T20:16:03.575 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.575 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.576 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.577 DEBUG:boto:Params: {} 2022-09-12T20:16:03.578 DEBUG:boto:Token: None 2022-09-12T20:16:03.578 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/ 2022-09-12T20:16:03.579 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI= 2022-09-12T20:16:03.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI=', 'Content-Length': '0'} 2022-09-12T20:16:03.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9fd9c5e4a56fee8-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.585 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:03.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:16:03.587 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:03.587 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:03.588 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:03.588 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:03.589 DEBUG:boto:Method: GET 2022-09-12T20:16:03.589 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:03.590 DEBUG:boto:Data: 2022-09-12T20:16:03.590 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.591 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.591 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.592 DEBUG:boto:Params: {} 2022-09-12T20:16:03.593 DEBUG:boto:Token: None 2022-09-12T20:16:03.593 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:03.594 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wI58fcZX68I8+74suMkpmXraHLw= 2022-09-12T20:16:03.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wI58fcZX68I8+74suMkpmXraHLw=', 'Content-Length': '0'} 2022-09-12T20:16:03.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000011226be9912f150-00631f9383-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.597 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:03.598 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:03.598 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:03.599 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:03.600 DEBUG:boto:Method: GET 2022-09-12T20:16:03.600 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:03.601 DEBUG:boto:Data: 2022-09-12T20:16:03.601 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.602 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.602 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.603 DEBUG:boto:Params: {} 2022-09-12T20:16:03.603 DEBUG:boto:Token: None 2022-09-12T20:16:03.604 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:03.605 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wI58fcZX68I8+74suMkpmXraHLw= 2022-09-12T20:16:03.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wI58fcZX68I8+74suMkpmXraHLw=', 'Content-Length': '0'} 2022-09-12T20:16:03.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c99bc467fc8b5d59-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.608 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:03.609 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:03.610 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:03.610 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:03.611 DEBUG:boto:Method: HEAD 2022-09-12T20:16:03.611 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:03.612 DEBUG:boto:Data: 2022-09-12T20:16:03.612 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.613 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.613 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.614 DEBUG:boto:Params: {} 2022-09-12T20:16:03.614 DEBUG:boto:Token: None 2022-09-12T20:16:03.615 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:03.615 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:n7bZpQnzOafEJIov21RDEZp5t/w= 2022-09-12T20:16:03.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:n7bZpQnzOafEJIov21RDEZp5t/w=', 'Content-Length': '0'} 2022-09-12T20:16:03.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000278a945ed7fbeaab-00631f9383-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.619 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:03.619 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:03.620 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:03.620 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:03.621 DEBUG:boto:Method: HEAD 2022-09-12T20:16:03.621 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:03.622 DEBUG:boto:Data: 2022-09-12T20:16:03.623 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.623 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.624 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.624 DEBUG:boto:Params: {} 2022-09-12T20:16:03.625 DEBUG:boto:Token: None 2022-09-12T20:16:03.625 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:03.626 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:n7bZpQnzOafEJIov21RDEZp5t/w= 2022-09-12T20:16:03.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:n7bZpQnzOafEJIov21RDEZp5t/w=', 'Content-Length': '0'} 2022-09-12T20:16:03.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a7d8ba046be8f40-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:16:03.630 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:03.631 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:03.631 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:03.632 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:03.632 DEBUG:boto:Method: GET 2022-09-12T20:16:03.633 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:03.634 DEBUG:boto:Data: 2022-09-12T20:16:03.634 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.635 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.635 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.636 DEBUG:boto:Params: {} 2022-09-12T20:16:03.636 DEBUG:boto:Token: None 2022-09-12T20:16:03.637 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:03.637 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wI58fcZX68I8+74suMkpmXraHLw= 2022-09-12T20:16:03.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wI58fcZX68I8+74suMkpmXraHLw=', 'Content-Length': '0'} 2022-09-12T20:16:03.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8b4daac70314c9b-00631f9383-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.682 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:03.683 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:03.683 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:03.684 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:03.684 DEBUG:boto:Method: GET 2022-09-12T20:16:03.685 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:03.685 DEBUG:boto:Data: 2022-09-12T20:16:03.686 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.687 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.687 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.688 DEBUG:boto:Params: {} 2022-09-12T20:16:03.689 DEBUG:boto:Token: None 2022-09-12T20:16:03.689 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:03.690 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wI58fcZX68I8+74suMkpmXraHLw= 2022-09-12T20:16:03.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wI58fcZX68I8+74suMkpmXraHLw=', 'Content-Length': '0'} 2022-09-12T20:16:03.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077f347d7aa180fb8-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.694 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:03.694 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:03.695 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:16:03.695 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:16:03.696 DEBUG:boto:Method: GET 2022-09-12T20:16:03.696 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:16:03.697 DEBUG:boto:Data: 2022-09-12T20:16:03.697 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.698 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.698 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.699 DEBUG:boto:Params: {} 2022-09-12T20:16:03.699 DEBUG:boto:Token: None 2022-09-12T20:16:03.700 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/ 2022-09-12T20:16:03.701 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI= 2022-09-12T20:16:03.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI=', 'Content-Length': '0'} 2022-09-12T20:16:03.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000296c9c158d44cc80-00631f9383-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.705 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:03.705 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:03.706 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:03.706 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:16:03.707 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:16:03.708 DEBUG:boto:Method: GET 2022-09-12T20:16:03.708 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:16:03.709 DEBUG:boto:Data: 2022-09-12T20:16:03.709 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.710 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.710 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.711 DEBUG:boto:Params: {} 2022-09-12T20:16:03.711 DEBUG:boto:Token: None 2022-09-12T20:16:03.712 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/ 2022-09-12T20:16:03.712 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI= 2022-09-12T20:16:03.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI=', 'Content-Length': '0'} 2022-09-12T20:16:03.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e90f079c0f5bb55-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.719 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:03.720 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:03.720 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:03.721 DEBUG:boto:Method: HEAD 2022-09-12T20:16:03.722 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:16:03.723 DEBUG:boto:Data: 2022-09-12T20:16:03.724 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.725 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.726 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.727 DEBUG:boto:Params: {} 2022-09-12T20:16:03.728 DEBUG:boto:Token: None 2022-09-12T20:16:03.728 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj55 2022-09-12T20:16:03.729 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RaZx6UmcSAhaUuquVEk+A7EGqDI= 2022-09-12T20:16:03.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RaZx6UmcSAhaUuquVEk+A7EGqDI=', 'Content-Length': '0'} 2022-09-12T20:16:03.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1ba632ee0a4ce24-00631f9383-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.734 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:03.735 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:03.735 DEBUG:boto:Method: HEAD 2022-09-12T20:16:03.736 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:16:03.737 DEBUG:boto:Data: 2022-09-12T20:16:03.738 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.739 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.740 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.741 DEBUG:boto:Params: {} 2022-09-12T20:16:03.742 DEBUG:boto:Token: None 2022-09-12T20:16:03.742 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj55 2022-09-12T20:16:03.744 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RaZx6UmcSAhaUuquVEk+A7EGqDI= 2022-09-12T20:16:03.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RaZx6UmcSAhaUuquVEk+A7EGqDI=', 'Content-Length': '0'} 2022-09-12T20:16:03.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001975f43738115a4f-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:16:03.749 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:03.749 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:03.750 DEBUG:boto:Method: GET 2022-09-12T20:16:03.750 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:16:03.751 DEBUG:boto:Data: 2022-09-12T20:16:03.751 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.752 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.753 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.754 DEBUG:boto:Params: {} 2022-09-12T20:16:03.755 DEBUG:boto:Token: None 2022-09-12T20:16:03.756 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj55 2022-09-12T20:16:03.757 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dDscdgX4yx/KXs0TWUCFaFp+8dM= 2022-09-12T20:16:03.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dDscdgX4yx/KXs0TWUCFaFp+8dM=', 'Content-Length': '0'} 2022-09-12T20:16:03.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039d25180d68150e5-00631f9383-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.762 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:03.763 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:03.764 DEBUG:boto:Method: GET 2022-09-12T20:16:03.764 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:16:03.765 DEBUG:boto:Data: 2022-09-12T20:16:03.765 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.766 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.766 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.767 DEBUG:boto:Params: {} 2022-09-12T20:16:03.767 DEBUG:boto:Token: None 2022-09-12T20:16:03.768 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj55 2022-09-12T20:16:03.769 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dDscdgX4yx/KXs0TWUCFaFp+8dM= 2022-09-12T20:16:03.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dDscdgX4yx/KXs0TWUCFaFp+8dM=', 'Content-Length': '0'} 2022-09-12T20:16:03.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000380c703c1107e221-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.814 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:03.815 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:03.815 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:16:03.816 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:16:03.817 DEBUG:boto:Method: GET 2022-09-12T20:16:03.817 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:16:03.818 DEBUG:boto:Data: 2022-09-12T20:16:03.818 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.819 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.819 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.820 DEBUG:boto:Params: {} 2022-09-12T20:16:03.820 DEBUG:boto:Token: None 2022-09-12T20:16:03.821 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/ 2022-09-12T20:16:03.822 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI= 2022-09-12T20:16:03.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI=', 'Content-Length': '0'} 2022-09-12T20:16:03.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000787446cbd1d89ec4-00631f9383-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.826 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:03.827 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:03.828 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:03.828 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:16:03.829 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:16:03.830 DEBUG:boto:Method: GET 2022-09-12T20:16:03.830 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:16:03.831 DEBUG:boto:Data: 2022-09-12T20:16:03.831 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.832 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.833 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.833 DEBUG:boto:Params: {} 2022-09-12T20:16:03.834 DEBUG:boto:Token: None 2022-09-12T20:16:03.834 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/ 2022-09-12T20:16:03.835 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI= 2022-09-12T20:16:03.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI=', 'Content-Length': '0'} 2022-09-12T20:16:03.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000451432664da210ef-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.841 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:03.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:16:03.842 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:03.843 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:03.843 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:03.844 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:03.845 DEBUG:boto:Method: GET 2022-09-12T20:16:03.846 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:03.846 DEBUG:boto:Data: 2022-09-12T20:16:03.847 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.847 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.848 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.848 DEBUG:boto:Params: {} 2022-09-12T20:16:03.849 DEBUG:boto:Token: None 2022-09-12T20:16:03.849 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:03.850 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ihQdQJF86qvT8U2xMR7V9A3V078= 2022-09-12T20:16:03.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ihQdQJF86qvT8U2xMR7V9A3V078=', 'Content-Length': '0'} 2022-09-12T20:16:03.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001726557f4ed54ec0-00631f9383-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.855 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:03.856 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:03.857 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:03.858 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:03.859 DEBUG:boto:Method: GET 2022-09-12T20:16:03.860 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:03.861 DEBUG:boto:Data: 2022-09-12T20:16:03.861 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.862 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.863 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.864 DEBUG:boto:Params: {} 2022-09-12T20:16:03.864 DEBUG:boto:Token: None 2022-09-12T20:16:03.865 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:03.866 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ihQdQJF86qvT8U2xMR7V9A3V078= 2022-09-12T20:16:03.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ihQdQJF86qvT8U2xMR7V9A3V078=', 'Content-Length': '0'} 2022-09-12T20:16:03.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006355043cb7bfdd1e-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.869 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:03.870 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:03.870 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:03.871 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:03.871 DEBUG:boto:Method: HEAD 2022-09-12T20:16:03.872 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:03.872 DEBUG:boto:Data: 2022-09-12T20:16:03.873 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.874 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.874 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.875 DEBUG:boto:Params: {} 2022-09-12T20:16:03.875 DEBUG:boto:Token: None 2022-09-12T20:16:03.876 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:03.876 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VORneEX9yqNL9CElWp+V1HJJOQk= 2022-09-12T20:16:03.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VORneEX9yqNL9CElWp+V1HJJOQk=', 'Content-Length': '0'} 2022-09-12T20:16:03.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a2790338e3ac855-00631f9383-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.880 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:03.881 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:03.882 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:03.883 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:03.884 DEBUG:boto:Method: HEAD 2022-09-12T20:16:03.884 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:03.885 DEBUG:boto:Data: 2022-09-12T20:16:03.885 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.886 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.886 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.887 DEBUG:boto:Params: {} 2022-09-12T20:16:03.887 DEBUG:boto:Token: None 2022-09-12T20:16:03.888 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:03.888 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VORneEX9yqNL9CElWp+V1HJJOQk= 2022-09-12T20:16:03.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VORneEX9yqNL9CElWp+V1HJJOQk=', 'Content-Length': '0'} 2022-09-12T20:16:03.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa739d33eedfc90f-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:16:03.893 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:03.893 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:03.894 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:03.895 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:03.896 DEBUG:boto:Method: GET 2022-09-12T20:16:03.897 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:03.897 DEBUG:boto:Data: 2022-09-12T20:16:03.898 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.899 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.899 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.900 DEBUG:boto:Params: {} 2022-09-12T20:16:03.900 DEBUG:boto:Token: None 2022-09-12T20:16:03.901 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:03.902 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ihQdQJF86qvT8U2xMR7V9A3V078= 2022-09-12T20:16:03.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ihQdQJF86qvT8U2xMR7V9A3V078=', 'Content-Length': '0'} 2022-09-12T20:16:03.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe787670b371e6d2-00631f9383-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.905 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:03.906 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:03.906 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:03.907 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:03.907 DEBUG:boto:Method: GET 2022-09-12T20:16:03.908 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:03.908 DEBUG:boto:Data: 2022-09-12T20:16:03.909 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.909 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.910 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.910 DEBUG:boto:Params: {} 2022-09-12T20:16:03.911 DEBUG:boto:Token: None 2022-09-12T20:16:03.912 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:03.912 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ihQdQJF86qvT8U2xMR7V9A3V078= 2022-09-12T20:16:03.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ihQdQJF86qvT8U2xMR7V9A3V078=', 'Content-Length': '0'} 2022-09-12T20:16:03.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f3fe123d8ad342b-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.916 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:03.917 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:03.917 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:16:03.918 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:16:03.919 DEBUG:boto:Method: GET 2022-09-12T20:16:03.919 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:16:03.920 DEBUG:boto:Data: 2022-09-12T20:16:03.920 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.921 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.921 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.922 DEBUG:boto:Params: {} 2022-09-12T20:16:03.922 DEBUG:boto:Token: None 2022-09-12T20:16:03.923 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/ 2022-09-12T20:16:03.924 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI= 2022-09-12T20:16:03.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI=', 'Content-Length': '0'} 2022-09-12T20:16:03.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc03417e2377ca2c-00631f9383-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.928 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:03.928 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:03.929 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:03.930 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:16:03.930 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:16:03.931 DEBUG:boto:Method: GET 2022-09-12T20:16:03.931 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:16:03.932 DEBUG:boto:Data: 2022-09-12T20:16:03.932 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.933 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.933 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.934 DEBUG:boto:Params: {} 2022-09-12T20:16:03.934 DEBUG:boto:Token: None 2022-09-12T20:16:03.935 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/ 2022-09-12T20:16:03.936 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI= 2022-09-12T20:16:03.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H3huFwHgLhDySYlTNxQNaVKdTEI=', 'Content-Length': '0'} 2022-09-12T20:16:03.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc86199a9de0865f-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.942 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:03.943 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:03.944 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:03.944 DEBUG:boto:Method: HEAD 2022-09-12T20:16:03.945 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:16:03.945 DEBUG:boto:Data: 2022-09-12T20:16:03.946 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.946 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.947 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.948 DEBUG:boto:Params: {} 2022-09-12T20:16:03.948 DEBUG:boto:Token: None 2022-09-12T20:16:03.949 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj57 2022-09-12T20:16:03.949 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Gb2Fbmem0bVVnnEfQCYboZKpye4= 2022-09-12T20:16:03.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Gb2Fbmem0bVVnnEfQCYboZKpye4=', 'Content-Length': '0'} 2022-09-12T20:16:03.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006097d7b41f47215a-00631f9383-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.953 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:03.953 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:03.954 DEBUG:boto:Method: HEAD 2022-09-12T20:16:03.954 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:16:03.955 DEBUG:boto:Data: 2022-09-12T20:16:03.955 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.956 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.956 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.957 DEBUG:boto:Params: {} 2022-09-12T20:16:03.957 DEBUG:boto:Token: None 2022-09-12T20:16:03.958 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj57 2022-09-12T20:16:03.959 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Gb2Fbmem0bVVnnEfQCYboZKpye4= 2022-09-12T20:16:03.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Gb2Fbmem0bVVnnEfQCYboZKpye4=', 'Content-Length': '0'} 2022-09-12T20:16:03.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b54bdf763b53fc9-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:16:03.965 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:03.966 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:03.967 DEBUG:boto:Method: GET 2022-09-12T20:16:03.968 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:16:03.969 DEBUG:boto:Data: 2022-09-12T20:16:03.969 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.970 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:03.971 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.972 DEBUG:boto:Params: {} 2022-09-12T20:16:03.973 DEBUG:boto:Token: None 2022-09-12T20:16:03.974 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj57 2022-09-12T20:16:03.975 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0iw3Cu8NYzEuBscwODuxRsrXoaE= 2022-09-12T20:16:03.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0iw3Cu8NYzEuBscwODuxRsrXoaE=', 'Content-Length': '0'} 2022-09-12T20:16:03.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000312203fd1be053d3-00631f9383-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:03.980 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:03.981 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:03.982 DEBUG:boto:Method: GET 2022-09-12T20:16:03.983 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:16:03.984 DEBUG:boto:Data: 2022-09-12T20:16:03.985 DEBUG:boto:Headers: {} 2022-09-12T20:16:03.986 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:03.987 DEBUG:boto:Port: 8000 2022-09-12T20:16:03.988 DEBUG:boto:Params: {} 2022-09-12T20:16:03.989 DEBUG:boto:Token: None 2022-09-12T20:16:03.990 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:03 GMT /jdpiak-75/obj57 2022-09-12T20:16:03.991 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0iw3Cu8NYzEuBscwODuxRsrXoaE= 2022-09-12T20:16:03.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0iw3Cu8NYzEuBscwODuxRsrXoaE=', 'Content-Length': '0'} 2022-09-12T20:16:03.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087b257d37c80f45e-00631f9383-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.038 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:04.039 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:04.040 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:16:04.041 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:16:04.042 DEBUG:boto:Method: GET 2022-09-12T20:16:04.043 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:16:04.044 DEBUG:boto:Data: 2022-09-12T20:16:04.045 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.046 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:04.047 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.048 DEBUG:boto:Params: {} 2022-09-12T20:16:04.049 DEBUG:boto:Token: None 2022-09-12T20:16:04.051 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/ 2022-09-12T20:16:04.052 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c= 2022-09-12T20:16:04.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c=', 'Content-Length': '0'} 2022-09-12T20:16:04.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007772fa00dd2c8a0e-00631f9384-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.057 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:04.059 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:04.060 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:04.061 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:16:04.062 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:16:04.063 DEBUG:boto:Method: GET 2022-09-12T20:16:04.064 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:16:04.065 DEBUG:boto:Data: 2022-09-12T20:16:04.066 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.067 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:04.068 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.068 DEBUG:boto:Params: {} 2022-09-12T20:16:04.069 DEBUG:boto:Token: None 2022-09-12T20:16:04.069 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/ 2022-09-12T20:16:04.070 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c= 2022-09-12T20:16:04.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c=', 'Content-Length': '0'} 2022-09-12T20:16:04.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d9041bbfd56e660-00631f9384-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.076 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:04.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:16:04.077 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:04.078 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:04.079 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:04.079 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:04.080 DEBUG:boto:Method: GET 2022-09-12T20:16:04.080 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:04.081 DEBUG:boto:Data: 2022-09-12T20:16:04.081 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.082 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:04.082 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.083 DEBUG:boto:Params: {} 2022-09-12T20:16:04.083 DEBUG:boto:Token: None 2022-09-12T20:16:04.084 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:04.085 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cPHr4/uPoV/AUAy1ZMToSIxFO3Q= 2022-09-12T20:16:04.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cPHr4/uPoV/AUAy1ZMToSIxFO3Q=', 'Content-Length': '0'} 2022-09-12T20:16:04.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000808a3c35e4c7175a-00631f9384-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.088 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:04.089 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:04.089 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:04.090 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:04.090 DEBUG:boto:Method: GET 2022-09-12T20:16:04.091 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:04.091 DEBUG:boto:Data: 2022-09-12T20:16:04.092 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.092 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:04.093 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.093 DEBUG:boto:Params: {} 2022-09-12T20:16:04.094 DEBUG:boto:Token: None 2022-09-12T20:16:04.095 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:04.095 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cPHr4/uPoV/AUAy1ZMToSIxFO3Q= 2022-09-12T20:16:04.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cPHr4/uPoV/AUAy1ZMToSIxFO3Q=', 'Content-Length': '0'} 2022-09-12T20:16:04.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3467787e18cf0fc-00631f9384-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.099 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:04.100 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:04.100 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:04.101 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:04.101 DEBUG:boto:Method: HEAD 2022-09-12T20:16:04.102 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:04.102 DEBUG:boto:Data: 2022-09-12T20:16:04.103 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.103 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:04.104 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.104 DEBUG:boto:Params: {} 2022-09-12T20:16:04.105 DEBUG:boto:Token: None 2022-09-12T20:16:04.105 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:04.106 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pPl+UwQB1eSzyl9I3wvqlm+LgGk= 2022-09-12T20:16:04.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pPl+UwQB1eSzyl9I3wvqlm+LgGk=', 'Content-Length': '0'} 2022-09-12T20:16:04.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0219e791dd3826c-00631f9384-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.110 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:04.110 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:04.111 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:04.111 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:04.112 DEBUG:boto:Method: HEAD 2022-09-12T20:16:04.112 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:04.113 DEBUG:boto:Data: 2022-09-12T20:16:04.113 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.114 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:04.114 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.115 DEBUG:boto:Params: {} 2022-09-12T20:16:04.115 DEBUG:boto:Token: None 2022-09-12T20:16:04.116 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:04.116 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pPl+UwQB1eSzyl9I3wvqlm+LgGk= 2022-09-12T20:16:04.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pPl+UwQB1eSzyl9I3wvqlm+LgGk=', 'Content-Length': '0'} 2022-09-12T20:16:04.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b83e06b27f5e1145-00631f9384-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:16:04.121 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:04.121 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:04.122 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:04.122 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:04.123 DEBUG:boto:Method: GET 2022-09-12T20:16:04.123 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:04.124 DEBUG:boto:Data: 2022-09-12T20:16:04.124 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.125 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:04.126 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.126 DEBUG:boto:Params: {} 2022-09-12T20:16:04.127 DEBUG:boto:Token: None 2022-09-12T20:16:04.127 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:04.128 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cPHr4/uPoV/AUAy1ZMToSIxFO3Q= 2022-09-12T20:16:04.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cPHr4/uPoV/AUAy1ZMToSIxFO3Q=', 'Content-Length': '0'} 2022-09-12T20:16:04.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000126877c6c77dc90e-00631f9384-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.132 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:04.133 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:04.133 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:04.134 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:04.134 DEBUG:boto:Method: GET 2022-09-12T20:16:04.135 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:04.135 DEBUG:boto:Data: 2022-09-12T20:16:04.136 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.137 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:04.137 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.138 DEBUG:boto:Params: {} 2022-09-12T20:16:04.139 DEBUG:boto:Token: None 2022-09-12T20:16:04.139 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:04.140 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cPHr4/uPoV/AUAy1ZMToSIxFO3Q= 2022-09-12T20:16:04.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cPHr4/uPoV/AUAy1ZMToSIxFO3Q=', 'Content-Length': '0'} 2022-09-12T20:16:04.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc61f9fb574167be-00631f9384-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.186 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:04.187 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:04.187 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:16:04.188 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:16:04.189 DEBUG:boto:Method: GET 2022-09-12T20:16:04.189 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:16:04.190 DEBUG:boto:Data: 2022-09-12T20:16:04.190 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.191 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:04.191 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.192 DEBUG:boto:Params: {} 2022-09-12T20:16:04.193 DEBUG:boto:Token: None 2022-09-12T20:16:04.193 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/ 2022-09-12T20:16:04.194 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c= 2022-09-12T20:16:04.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c=', 'Content-Length': '0'} 2022-09-12T20:16:04.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e877ee07917c6541-00631f9384-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.198 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:04.199 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:04.199 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:04.200 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:16:04.200 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:16:04.201 DEBUG:boto:Method: GET 2022-09-12T20:16:04.201 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:16:04.202 DEBUG:boto:Data: 2022-09-12T20:16:04.203 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.203 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:04.204 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.204 DEBUG:boto:Params: {} 2022-09-12T20:16:04.205 DEBUG:boto:Token: None 2022-09-12T20:16:04.205 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/ 2022-09-12T20:16:04.206 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c= 2022-09-12T20:16:04.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c=', 'Content-Length': '0'} 2022-09-12T20:16:04.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d27f6c860d7559b-00631f9384-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.213 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:04.213 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:04.214 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:04.215 DEBUG:boto:Method: HEAD 2022-09-12T20:16:04.215 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:16:04.216 DEBUG:boto:Data: 2022-09-12T20:16:04.216 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.217 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:04.217 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.218 DEBUG:boto:Params: {} 2022-09-12T20:16:04.218 DEBUG:boto:Token: None 2022-09-12T20:16:04.219 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj58 2022-09-12T20:16:04.219 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2+v/7GRV1aS5T01zKY1Je6cc/0s= 2022-09-12T20:16:04.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2+v/7GRV1aS5T01zKY1Je6cc/0s=', 'Content-Length': '0'} 2022-09-12T20:16:04.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebf218ab969ea83a-00631f9384-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.223 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:04.223 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:04.224 DEBUG:boto:Method: HEAD 2022-09-12T20:16:04.224 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:16:04.225 DEBUG:boto:Data: 2022-09-12T20:16:04.225 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.226 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:04.226 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.227 DEBUG:boto:Params: {} 2022-09-12T20:16:04.227 DEBUG:boto:Token: None 2022-09-12T20:16:04.228 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj58 2022-09-12T20:16:04.229 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2+v/7GRV1aS5T01zKY1Je6cc/0s= 2022-09-12T20:16:04.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2+v/7GRV1aS5T01zKY1Je6cc/0s=', 'Content-Length': '0'} 2022-09-12T20:16:04.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000826b1eed03cc88d3-00631f9384-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:16:04.233 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:04.233 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:04.234 DEBUG:boto:Method: GET 2022-09-12T20:16:04.235 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:16:04.236 DEBUG:boto:Data: 2022-09-12T20:16:04.236 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.237 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:04.238 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.239 DEBUG:boto:Params: {} 2022-09-12T20:16:04.239 DEBUG:boto:Token: None 2022-09-12T20:16:04.240 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj58 2022-09-12T20:16:04.241 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:r6ViDmCWTyHJ+LH6l1RAka+v8HU= 2022-09-12T20:16:04.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:r6ViDmCWTyHJ+LH6l1RAka+v8HU=', 'Content-Length': '0'} 2022-09-12T20:16:04.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000385a31b671837f5a-00631f9384-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.290 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:04.291 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:04.292 DEBUG:boto:Method: GET 2022-09-12T20:16:04.292 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:16:04.293 DEBUG:boto:Data: 2022-09-12T20:16:04.294 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.295 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:04.295 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.296 DEBUG:boto:Params: {} 2022-09-12T20:16:04.297 DEBUG:boto:Token: None 2022-09-12T20:16:04.298 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj58 2022-09-12T20:16:04.299 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:r6ViDmCWTyHJ+LH6l1RAka+v8HU= 2022-09-12T20:16:04.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:r6ViDmCWTyHJ+LH6l1RAka+v8HU=', 'Content-Length': '0'} 2022-09-12T20:16:04.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094613695230a69cf-00631f9384-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.303 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:04.304 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:04.305 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:16:04.306 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:16:04.307 DEBUG:boto:Method: GET 2022-09-12T20:16:04.308 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:16:04.309 DEBUG:boto:Data: 2022-09-12T20:16:04.310 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.311 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:04.312 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.313 DEBUG:boto:Params: {} 2022-09-12T20:16:04.313 DEBUG:boto:Token: None 2022-09-12T20:16:04.314 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/ 2022-09-12T20:16:04.315 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c= 2022-09-12T20:16:04.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c=', 'Content-Length': '0'} 2022-09-12T20:16:04.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfa1da84d79e9ee9-00631f9384-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.320 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:04.321 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:04.322 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:04.322 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:16:04.323 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:16:04.323 DEBUG:boto:Method: GET 2022-09-12T20:16:04.324 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:16:04.325 DEBUG:boto:Data: 2022-09-12T20:16:04.326 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.326 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:04.327 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.328 DEBUG:boto:Params: {} 2022-09-12T20:16:04.329 DEBUG:boto:Token: None 2022-09-12T20:16:04.330 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/ 2022-09-12T20:16:04.331 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c= 2022-09-12T20:16:04.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c=', 'Content-Length': '0'} 2022-09-12T20:16:04.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d066332e89c87a2-00631f9384-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.336 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:04.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-12T20:16:04.338 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:04.339 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:04.340 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:04.340 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:04.341 DEBUG:boto:Method: GET 2022-09-12T20:16:04.342 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:04.342 DEBUG:boto:Data: 2022-09-12T20:16:04.343 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.344 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:04.344 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.345 DEBUG:boto:Params: {} 2022-09-12T20:16:04.346 DEBUG:boto:Token: None 2022-09-12T20:16:04.347 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:04.348 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lzq7vVBinuQlFZ4T5MmYWnAcx0w= 2022-09-12T20:16:04.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lzq7vVBinuQlFZ4T5MmYWnAcx0w=', 'Content-Length': '0'} 2022-09-12T20:16:04.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c248cd04e9f7e892-00631f9384-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.352 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:04.353 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:04.353 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:04.354 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:04.354 DEBUG:boto:Method: GET 2022-09-12T20:16:04.355 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:04.356 DEBUG:boto:Data: 2022-09-12T20:16:04.356 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.357 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:04.358 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.359 DEBUG:boto:Params: {} 2022-09-12T20:16:04.359 DEBUG:boto:Token: None 2022-09-12T20:16:04.360 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:04.361 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lzq7vVBinuQlFZ4T5MmYWnAcx0w= 2022-09-12T20:16:04.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lzq7vVBinuQlFZ4T5MmYWnAcx0w=', 'Content-Length': '0'} 2022-09-12T20:16:04.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0a64e728a39f692-00631f9384-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.406 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:04.407 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:04.407 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:04.408 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:04.409 DEBUG:boto:Method: HEAD 2022-09-12T20:16:04.409 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:04.410 DEBUG:boto:Data: 2022-09-12T20:16:04.411 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.411 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:04.412 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.413 DEBUG:boto:Params: {} 2022-09-12T20:16:04.413 DEBUG:boto:Token: None 2022-09-12T20:16:04.414 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:04.415 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VrGGrWabHzJE7a3egAzEBJnJ+/8= 2022-09-12T20:16:04.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VrGGrWabHzJE7a3egAzEBJnJ+/8=', 'Content-Length': '0'} 2022-09-12T20:16:04.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d010c91ebbec4ca-00631f9384-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.418 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:04.419 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:04.420 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:04.420 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:04.421 DEBUG:boto:Method: HEAD 2022-09-12T20:16:04.422 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:04.423 DEBUG:boto:Data: 2022-09-12T20:16:04.423 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.424 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:04.425 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.425 DEBUG:boto:Params: {} 2022-09-12T20:16:04.426 DEBUG:boto:Token: None 2022-09-12T20:16:04.427 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:04.428 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VrGGrWabHzJE7a3egAzEBJnJ+/8= 2022-09-12T20:16:04.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VrGGrWabHzJE7a3egAzEBJnJ+/8=', 'Content-Length': '0'} 2022-09-12T20:16:04.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005781788c0c1ddcd2-00631f9384-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-12T20:16:04.432 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:04.433 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:04.434 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:04.434 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:04.435 DEBUG:boto:Method: GET 2022-09-12T20:16:04.436 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:04.436 DEBUG:boto:Data: 2022-09-12T20:16:04.437 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.438 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:04.439 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.440 DEBUG:boto:Params: {} 2022-09-12T20:16:04.441 DEBUG:boto:Token: None 2022-09-12T20:16:04.442 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:04.443 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lzq7vVBinuQlFZ4T5MmYWnAcx0w= 2022-09-12T20:16:04.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lzq7vVBinuQlFZ4T5MmYWnAcx0w=', 'Content-Length': '0'} 2022-09-12T20:16:04.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029ea8728a0f5797f-00631f9384-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.447 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:04.447 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:04.448 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:04.448 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:04.449 DEBUG:boto:Method: GET 2022-09-12T20:16:04.449 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:04.450 DEBUG:boto:Data: 2022-09-12T20:16:04.451 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.451 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:04.452 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.452 DEBUG:boto:Params: {} 2022-09-12T20:16:04.453 DEBUG:boto:Token: None 2022-09-12T20:16:04.453 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:04.454 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lzq7vVBinuQlFZ4T5MmYWnAcx0w= 2022-09-12T20:16:04.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lzq7vVBinuQlFZ4T5MmYWnAcx0w=', 'Content-Length': '0'} 2022-09-12T20:16:04.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dee11f60a0252a25-00631f9384-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.458 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:04.459 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:04.459 DEBUG:boto:path=/jdpiak-75/?prefix=obj62 2022-09-12T20:16:04.460 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj62 2022-09-12T20:16:04.460 DEBUG:boto:Method: GET 2022-09-12T20:16:04.461 DEBUG:boto:Path: /jdpiak-75/?prefix=obj62 2022-09-12T20:16:04.461 DEBUG:boto:Data: 2022-09-12T20:16:04.462 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.462 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:04.463 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.463 DEBUG:boto:Params: {} 2022-09-12T20:16:04.464 DEBUG:boto:Token: None 2022-09-12T20:16:04.465 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/ 2022-09-12T20:16:04.465 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c= 2022-09-12T20:16:04.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c=', 'Content-Length': '0'} 2022-09-12T20:16:04.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000917098f028d16619-00631f9384-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.514 DEBUG:boto:b'jdpiak-75obj621000falseobj622022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:04.515 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:04.516 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:04.516 DEBUG:boto:path=/jdpiak-75/?prefix=obj62 2022-09-12T20:16:04.517 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj62 2022-09-12T20:16:04.517 DEBUG:boto:Method: GET 2022-09-12T20:16:04.518 DEBUG:boto:Path: /jdpiak-75/?prefix=obj62 2022-09-12T20:16:04.518 DEBUG:boto:Data: 2022-09-12T20:16:04.519 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.519 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:04.520 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.520 DEBUG:boto:Params: {} 2022-09-12T20:16:04.521 DEBUG:boto:Token: None 2022-09-12T20:16:04.522 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/ 2022-09-12T20:16:04.522 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c= 2022-09-12T20:16:04.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c=', 'Content-Length': '0'} 2022-09-12T20:16:04.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1a291c4899873b2-00631f9384-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.528 DEBUG:boto:b'jdpiak-75obj621000falseobj622022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:04.529 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:04.529 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:04.530 DEBUG:boto:Method: HEAD 2022-09-12T20:16:04.530 DEBUG:boto:Path: /jdpiak-75/obj62 2022-09-12T20:16:04.531 DEBUG:boto:Data: 2022-09-12T20:16:04.531 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.532 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:04.532 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.533 DEBUG:boto:Params: {} 2022-09-12T20:16:04.533 DEBUG:boto:Token: None 2022-09-12T20:16:04.534 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj62 2022-09-12T20:16:04.534 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5jZWai56v3aUAl09kZGUrJRM1QU= 2022-09-12T20:16:04.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5jZWai56v3aUAl09kZGUrJRM1QU=', 'Content-Length': '0'} 2022-09-12T20:16:04.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035e5bc61a3cf1343-00631f9384-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.538 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:04.538 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:04.539 DEBUG:boto:Method: HEAD 2022-09-12T20:16:04.540 DEBUG:boto:Path: /jdpiak-75/obj62 2022-09-12T20:16:04.540 DEBUG:boto:Data: 2022-09-12T20:16:04.541 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.541 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:04.542 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.542 DEBUG:boto:Params: {} 2022-09-12T20:16:04.543 DEBUG:boto:Token: None 2022-09-12T20:16:04.543 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj62 2022-09-12T20:16:04.544 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5jZWai56v3aUAl09kZGUrJRM1QU= 2022-09-12T20:16:04.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5jZWai56v3aUAl09kZGUrJRM1QU=', 'Content-Length': '0'} 2022-09-12T20:16:04.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000620758cc9891fd68-00631f9384-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-12T20:16:04.548 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:04.548 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:04.549 DEBUG:boto:Method: GET 2022-09-12T20:16:04.549 DEBUG:boto:Path: /jdpiak-75/obj62 2022-09-12T20:16:04.550 DEBUG:boto:Data: 2022-09-12T20:16:04.550 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.551 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:04.552 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.552 DEBUG:boto:Params: {} 2022-09-12T20:16:04.553 DEBUG:boto:Token: None 2022-09-12T20:16:04.553 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj62 2022-09-12T20:16:04.554 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7sGYrA+TXUjByHVIbvbr+iU9zI8= 2022-09-12T20:16:04.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7sGYrA+TXUjByHVIbvbr+iU9zI8=', 'Content-Length': '0'} 2022-09-12T20:16:04.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c8012f286b852f3-00631f9384-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.557 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:04.558 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:04.558 DEBUG:boto:Method: GET 2022-09-12T20:16:04.559 DEBUG:boto:Path: /jdpiak-75/obj62 2022-09-12T20:16:04.559 DEBUG:boto:Data: 2022-09-12T20:16:04.560 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.560 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:04.561 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.561 DEBUG:boto:Params: {} 2022-09-12T20:16:04.562 DEBUG:boto:Token: None 2022-09-12T20:16:04.563 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj62 2022-09-12T20:16:04.563 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7sGYrA+TXUjByHVIbvbr+iU9zI8= 2022-09-12T20:16:04.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7sGYrA+TXUjByHVIbvbr+iU9zI8=', 'Content-Length': '0'} 2022-09-12T20:16:04.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e995d2182fd4c4ab-00631f9384-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.614 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:04.615 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:04.615 DEBUG:boto:path=/jdpiak-75/?prefix=obj62 2022-09-12T20:16:04.616 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj62 2022-09-12T20:16:04.616 DEBUG:boto:Method: GET 2022-09-12T20:16:04.617 DEBUG:boto:Path: /jdpiak-75/?prefix=obj62 2022-09-12T20:16:04.617 DEBUG:boto:Data: 2022-09-12T20:16:04.618 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.619 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:04.619 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.620 DEBUG:boto:Params: {} 2022-09-12T20:16:04.620 DEBUG:boto:Token: None 2022-09-12T20:16:04.621 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/ 2022-09-12T20:16:04.621 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c= 2022-09-12T20:16:04.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c=', 'Content-Length': '0'} 2022-09-12T20:16:04.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000348d0863dc10521b-00631f9384-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.625 DEBUG:boto:b'jdpiak-75obj621000falseobj622022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:04.626 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:04.627 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:04.627 DEBUG:boto:path=/jdpiak-75/?prefix=obj62 2022-09-12T20:16:04.628 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj62 2022-09-12T20:16:04.628 DEBUG:boto:Method: GET 2022-09-12T20:16:04.629 DEBUG:boto:Path: /jdpiak-75/?prefix=obj62 2022-09-12T20:16:04.630 DEBUG:boto:Data: 2022-09-12T20:16:04.630 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.631 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:04.631 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.632 DEBUG:boto:Params: {} 2022-09-12T20:16:04.632 DEBUG:boto:Token: None 2022-09-12T20:16:04.633 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/ 2022-09-12T20:16:04.633 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c= 2022-09-12T20:16:04.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c=', 'Content-Length': '0'} 2022-09-12T20:16:04.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b40358e3bf7c44b-00631f9384-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.639 DEBUG:boto:b'jdpiak-75obj621000falseobj622022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:04.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:16:04.641 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:04.641 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:04.642 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:04.642 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:04.643 DEBUG:boto:Method: GET 2022-09-12T20:16:04.643 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:04.644 DEBUG:boto:Data: 2022-09-12T20:16:04.644 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.645 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:04.645 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.646 DEBUG:boto:Params: {} 2022-09-12T20:16:04.647 DEBUG:boto:Token: None 2022-09-12T20:16:04.647 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:04.648 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lriYfXdHqjLniQm9DilT49cjFxk= 2022-09-12T20:16:04.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lriYfXdHqjLniQm9DilT49cjFxk=', 'Content-Length': '0'} 2022-09-12T20:16:04.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb39a0079b8f278c-00631f9384-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.652 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:04.652 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:04.653 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:04.653 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:04.654 DEBUG:boto:Method: GET 2022-09-12T20:16:04.654 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:04.655 DEBUG:boto:Data: 2022-09-12T20:16:04.655 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.656 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:04.656 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.657 DEBUG:boto:Params: {} 2022-09-12T20:16:04.657 DEBUG:boto:Token: None 2022-09-12T20:16:04.658 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:04.659 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lriYfXdHqjLniQm9DilT49cjFxk= 2022-09-12T20:16:04.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lriYfXdHqjLniQm9DilT49cjFxk=', 'Content-Length': '0'} 2022-09-12T20:16:04.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068526421d10fc96e-00631f9384-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.662 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:04.663 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:04.663 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:04.664 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:04.664 DEBUG:boto:Method: HEAD 2022-09-12T20:16:04.665 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:04.665 DEBUG:boto:Data: 2022-09-12T20:16:04.666 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.678 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:04.678 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.679 DEBUG:boto:Params: {} 2022-09-12T20:16:04.680 DEBUG:boto:Token: None 2022-09-12T20:16:04.680 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:04.681 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kWyWkdrWYgTvy86GwHppWMcOXsQ= 2022-09-12T20:16:04.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kWyWkdrWYgTvy86GwHppWMcOXsQ=', 'Content-Length': '0'} 2022-09-12T20:16:04.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054e582b4fc3bc4ff-00631f9384-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.684 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:04.685 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:04.686 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:04.686 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:04.687 DEBUG:boto:Method: HEAD 2022-09-12T20:16:04.687 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:04.687 DEBUG:boto:Data: 2022-09-12T20:16:04.688 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.688 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:04.689 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.689 DEBUG:boto:Params: {} 2022-09-12T20:16:04.690 DEBUG:boto:Token: None 2022-09-12T20:16:04.691 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:04.691 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kWyWkdrWYgTvy86GwHppWMcOXsQ= 2022-09-12T20:16:04.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kWyWkdrWYgTvy86GwHppWMcOXsQ=', 'Content-Length': '0'} 2022-09-12T20:16:04.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d47605544edb3258-00631f9384-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:16:04.695 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:04.696 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:04.696 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:04.697 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:04.697 DEBUG:boto:Method: GET 2022-09-12T20:16:04.698 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:04.698 DEBUG:boto:Data: 2022-09-12T20:16:04.699 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.700 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:04.700 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.701 DEBUG:boto:Params: {} 2022-09-12T20:16:04.701 DEBUG:boto:Token: None 2022-09-12T20:16:04.702 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:04.702 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lriYfXdHqjLniQm9DilT49cjFxk= 2022-09-12T20:16:04.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lriYfXdHqjLniQm9DilT49cjFxk=', 'Content-Length': '0'} 2022-09-12T20:16:04.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003dee01c231b3cc8b-00631f9384-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.706 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:04.707 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:04.708 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:04.709 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:04.710 DEBUG:boto:Method: GET 2022-09-12T20:16:04.711 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:04.711 DEBUG:boto:Data: 2022-09-12T20:16:04.712 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.712 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:04.713 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.713 DEBUG:boto:Params: {} 2022-09-12T20:16:04.714 DEBUG:boto:Token: None 2022-09-12T20:16:04.714 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:04.715 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lriYfXdHqjLniQm9DilT49cjFxk= 2022-09-12T20:16:04.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lriYfXdHqjLniQm9DilT49cjFxk=', 'Content-Length': '0'} 2022-09-12T20:16:04.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6f5cc91aabea110-00631f9384-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.762 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:04.763 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:04.764 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:16:04.764 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:16:04.765 DEBUG:boto:Method: GET 2022-09-12T20:16:04.765 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:16:04.766 DEBUG:boto:Data: 2022-09-12T20:16:04.766 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.767 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:04.767 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.768 DEBUG:boto:Params: {} 2022-09-12T20:16:04.769 DEBUG:boto:Token: None 2022-09-12T20:16:04.769 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/ 2022-09-12T20:16:04.770 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c= 2022-09-12T20:16:04.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c=', 'Content-Length': '0'} 2022-09-12T20:16:04.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004843bee50e45adea-00631f9384-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.774 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:04.775 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:04.776 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:04.776 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:16:04.777 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:16:04.777 DEBUG:boto:Method: GET 2022-09-12T20:16:04.778 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:16:04.778 DEBUG:boto:Data: 2022-09-12T20:16:04.779 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.779 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:04.780 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.780 DEBUG:boto:Params: {} 2022-09-12T20:16:04.781 DEBUG:boto:Token: None 2022-09-12T20:16:04.782 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/ 2022-09-12T20:16:04.782 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c= 2022-09-12T20:16:04.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c=', 'Content-Length': '0'} 2022-09-12T20:16:04.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060b4fe27126117b6-00631f9384-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.788 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:04.789 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:04.790 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:04.790 DEBUG:boto:Method: HEAD 2022-09-12T20:16:04.791 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:16:04.791 DEBUG:boto:Data: 2022-09-12T20:16:04.792 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.792 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:04.793 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.794 DEBUG:boto:Params: {} 2022-09-12T20:16:04.794 DEBUG:boto:Token: None 2022-09-12T20:16:04.795 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj68 2022-09-12T20:16:04.795 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:miyYJzPVndgj/8mnX85qvC8bEAg= 2022-09-12T20:16:04.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:miyYJzPVndgj/8mnX85qvC8bEAg=', 'Content-Length': '0'} 2022-09-12T20:16:04.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d72b6394e6837544-00631f9384-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.799 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:04.800 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:04.801 DEBUG:boto:Method: HEAD 2022-09-12T20:16:04.802 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:16:04.803 DEBUG:boto:Data: 2022-09-12T20:16:04.804 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.805 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:04.805 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.806 DEBUG:boto:Params: {} 2022-09-12T20:16:04.807 DEBUG:boto:Token: None 2022-09-12T20:16:04.808 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj68 2022-09-12T20:16:04.809 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:miyYJzPVndgj/8mnX85qvC8bEAg= 2022-09-12T20:16:04.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:miyYJzPVndgj/8mnX85qvC8bEAg=', 'Content-Length': '0'} 2022-09-12T20:16:04.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083879697e74360bf-00631f9384-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:16:04.814 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:04.815 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:04.815 DEBUG:boto:Method: GET 2022-09-12T20:16:04.816 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:16:04.816 DEBUG:boto:Data: 2022-09-12T20:16:04.817 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.817 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:04.818 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.819 DEBUG:boto:Params: {} 2022-09-12T20:16:04.819 DEBUG:boto:Token: None 2022-09-12T20:16:04.820 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj68 2022-09-12T20:16:04.820 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LOFwgXne5yIeopVAdOdL2sNS95c= 2022-09-12T20:16:04.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LOFwgXne5yIeopVAdOdL2sNS95c=', 'Content-Length': '0'} 2022-09-12T20:16:04.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005886fba6a9d02bc-00631f9384-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.870 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:04.871 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:04.871 DEBUG:boto:Method: GET 2022-09-12T20:16:04.872 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:16:04.872 DEBUG:boto:Data: 2022-09-12T20:16:04.873 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.873 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:04.874 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.875 DEBUG:boto:Params: {} 2022-09-12T20:16:04.875 DEBUG:boto:Token: None 2022-09-12T20:16:04.876 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj68 2022-09-12T20:16:04.877 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LOFwgXne5yIeopVAdOdL2sNS95c= 2022-09-12T20:16:04.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LOFwgXne5yIeopVAdOdL2sNS95c=', 'Content-Length': '0'} 2022-09-12T20:16:04.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002e27babbac0c6ae-00631f9384-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.881 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:04.881 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:04.882 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:16:04.882 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:16:04.883 DEBUG:boto:Method: GET 2022-09-12T20:16:04.884 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:16:04.884 DEBUG:boto:Data: 2022-09-12T20:16:04.885 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.885 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:04.886 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.887 DEBUG:boto:Params: {} 2022-09-12T20:16:04.887 DEBUG:boto:Token: None 2022-09-12T20:16:04.888 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/ 2022-09-12T20:16:04.889 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c= 2022-09-12T20:16:04.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c=', 'Content-Length': '0'} 2022-09-12T20:16:04.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fc671993b80d24d-00631f9384-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.893 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:04.894 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:04.895 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:04.895 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:16:04.896 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:16:04.896 DEBUG:boto:Method: GET 2022-09-12T20:16:04.897 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:16:04.897 DEBUG:boto:Data: 2022-09-12T20:16:04.898 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.898 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:04.899 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.899 DEBUG:boto:Params: {} 2022-09-12T20:16:04.900 DEBUG:boto:Token: None 2022-09-12T20:16:04.901 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/ 2022-09-12T20:16:04.901 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c= 2022-09-12T20:16:04.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ktaaNgBnPip4aieMQtbLrVAuh2c=', 'Content-Length': '0'} 2022-09-12T20:16:04.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf58ae9150fa4fbb-00631f9384-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.909 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:04.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:16:04.911 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:04.912 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:04.913 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:04.914 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:04.914 DEBUG:boto:Method: GET 2022-09-12T20:16:04.915 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:04.916 DEBUG:boto:Data: 2022-09-12T20:16:04.916 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.917 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:04.917 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.918 DEBUG:boto:Params: {} 2022-09-12T20:16:04.919 DEBUG:boto:Token: None 2022-09-12T20:16:04.919 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:04.920 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hUjnSDW3LkVK5wfqeT+4vAAH7eM= 2022-09-12T20:16:04.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hUjnSDW3LkVK5wfqeT+4vAAH7eM=', 'Content-Length': '0'} 2022-09-12T20:16:04.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001de9cf514aad0264-00631f9384-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.924 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:04.925 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:04.925 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:04.926 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:04.926 DEBUG:boto:Method: GET 2022-09-12T20:16:04.927 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:04.927 DEBUG:boto:Data: 2022-09-12T20:16:04.928 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.929 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:04.929 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.930 DEBUG:boto:Params: {} 2022-09-12T20:16:04.930 DEBUG:boto:Token: None 2022-09-12T20:16:04.931 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:04.931 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hUjnSDW3LkVK5wfqeT+4vAAH7eM= 2022-09-12T20:16:04.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hUjnSDW3LkVK5wfqeT+4vAAH7eM=', 'Content-Length': '0'} 2022-09-12T20:16:04.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015b62ee080a6c650-00631f9384-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.978 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:04.979 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:04.980 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:04.980 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:04.981 DEBUG:boto:Method: HEAD 2022-09-12T20:16:04.981 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:04.982 DEBUG:boto:Data: 2022-09-12T20:16:04.982 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.983 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:04.983 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.984 DEBUG:boto:Params: {} 2022-09-12T20:16:04.984 DEBUG:boto:Token: None 2022-09-12T20:16:04.985 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:04.986 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7zyljEzB1ieedZx9Bib0BwLVZyc= 2022-09-12T20:16:04.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7zyljEzB1ieedZx9Bib0BwLVZyc=', 'Content-Length': '0'} 2022-09-12T20:16:04.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7b21372996fefcc-00631f9384-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:04.989 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:04.989 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:04.990 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:04.990 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:04.991 DEBUG:boto:Method: HEAD 2022-09-12T20:16:04.992 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:04.992 DEBUG:boto:Data: 2022-09-12T20:16:04.993 DEBUG:boto:Headers: {} 2022-09-12T20:16:04.993 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:04.994 DEBUG:boto:Port: 8000 2022-09-12T20:16:04.994 DEBUG:boto:Params: {} 2022-09-12T20:16:04.995 DEBUG:boto:Token: None 2022-09-12T20:16:04.995 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:04 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:04.996 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7zyljEzB1ieedZx9Bib0BwLVZyc= 2022-09-12T20:16:04.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:04 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7zyljEzB1ieedZx9Bib0BwLVZyc=', 'Content-Length': '0'} 2022-09-12T20:16:04.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e4de7dbc02b0abd-00631f9384-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:16:05.000 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:05.001 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:05.001 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:05.002 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:05.002 DEBUG:boto:Method: GET 2022-09-12T20:16:05.003 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:05.003 DEBUG:boto:Data: 2022-09-12T20:16:05.004 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.004 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.005 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.006 DEBUG:boto:Params: {} 2022-09-12T20:16:05.006 DEBUG:boto:Token: None 2022-09-12T20:16:05.007 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:05.007 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m7EXjUhMnyfS3kMsrCFDkPkNi40= 2022-09-12T20:16:05.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m7EXjUhMnyfS3kMsrCFDkPkNi40=', 'Content-Length': '0'} 2022-09-12T20:16:05.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000496b26668da0df08-00631f9385-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.011 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:05.011 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:05.012 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:05.012 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:05.013 DEBUG:boto:Method: GET 2022-09-12T20:16:05.014 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:05.014 DEBUG:boto:Data: 2022-09-12T20:16:05.015 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.015 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.016 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.016 DEBUG:boto:Params: {} 2022-09-12T20:16:05.017 DEBUG:boto:Token: None 2022-09-12T20:16:05.018 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:05.018 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m7EXjUhMnyfS3kMsrCFDkPkNi40= 2022-09-12T20:16:05.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m7EXjUhMnyfS3kMsrCFDkPkNi40=', 'Content-Length': '0'} 2022-09-12T20:16:05.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f38a6a743a9ada8-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.022 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:05.023 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:05.023 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:16:05.024 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:16:05.024 DEBUG:boto:Method: GET 2022-09-12T20:16:05.025 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:16:05.025 DEBUG:boto:Data: 2022-09-12T20:16:05.026 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.027 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.027 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.028 DEBUG:boto:Params: {} 2022-09-12T20:16:05.028 DEBUG:boto:Token: None 2022-09-12T20:16:05.029 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/ 2022-09-12T20:16:05.029 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY= 2022-09-12T20:16:05.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY=', 'Content-Length': '0'} 2022-09-12T20:16:05.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e756dd585263c2dd-00631f9385-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.033 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:05.034 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:05.035 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:05.035 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:16:05.036 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:16:05.036 DEBUG:boto:Method: GET 2022-09-12T20:16:05.037 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:16:05.038 DEBUG:boto:Data: 2022-09-12T20:16:05.038 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.039 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.039 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.040 DEBUG:boto:Params: {} 2022-09-12T20:16:05.040 DEBUG:boto:Token: None 2022-09-12T20:16:05.041 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/ 2022-09-12T20:16:05.041 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY= 2022-09-12T20:16:05.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY=', 'Content-Length': '0'} 2022-09-12T20:16:05.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fba57dae6d2a7433-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.048 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:05.049 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:05.049 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:05.050 DEBUG:boto:Method: HEAD 2022-09-12T20:16:05.050 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:16:05.051 DEBUG:boto:Data: 2022-09-12T20:16:05.051 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.052 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.052 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.053 DEBUG:boto:Params: {} 2022-09-12T20:16:05.053 DEBUG:boto:Token: None 2022-09-12T20:16:05.054 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj69 2022-09-12T20:16:05.054 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jvYQWAP/6yhmNzS6nWdvloxINz4= 2022-09-12T20:16:05.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jvYQWAP/6yhmNzS6nWdvloxINz4=', 'Content-Length': '0'} 2022-09-12T20:16:05.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7fd291b3b3ac4f5-00631f9385-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.057 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:05.058 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:05.058 DEBUG:boto:Method: HEAD 2022-09-12T20:16:05.059 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:16:05.060 DEBUG:boto:Data: 2022-09-12T20:16:05.060 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.061 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.061 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.062 DEBUG:boto:Params: {} 2022-09-12T20:16:05.062 DEBUG:boto:Token: None 2022-09-12T20:16:05.063 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj69 2022-09-12T20:16:05.064 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jvYQWAP/6yhmNzS6nWdvloxINz4= 2022-09-12T20:16:05.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jvYQWAP/6yhmNzS6nWdvloxINz4=', 'Content-Length': '0'} 2022-09-12T20:16:05.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000866a10f3215794a8-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:16:05.068 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:05.068 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:05.069 DEBUG:boto:Method: GET 2022-09-12T20:16:05.069 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:16:05.070 DEBUG:boto:Data: 2022-09-12T20:16:05.070 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.071 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.072 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.072 DEBUG:boto:Params: {} 2022-09-12T20:16:05.073 DEBUG:boto:Token: None 2022-09-12T20:16:05.073 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj69 2022-09-12T20:16:05.074 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H5t4gRhhojv4AP03BIdncHap0DE= 2022-09-12T20:16:05.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H5t4gRhhojv4AP03BIdncHap0DE=', 'Content-Length': '0'} 2022-09-12T20:16:05.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d838298a3d814fcf-00631f9385-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.078 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:05.078 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:05.079 DEBUG:boto:Method: GET 2022-09-12T20:16:05.079 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:16:05.080 DEBUG:boto:Data: 2022-09-12T20:16:05.080 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.081 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.081 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.082 DEBUG:boto:Params: {} 2022-09-12T20:16:05.083 DEBUG:boto:Token: None 2022-09-12T20:16:05.083 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj69 2022-09-12T20:16:05.084 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H5t4gRhhojv4AP03BIdncHap0DE= 2022-09-12T20:16:05.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H5t4gRhhojv4AP03BIdncHap0DE=', 'Content-Length': '0'} 2022-09-12T20:16:05.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038d406e0eccfdda0-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.130 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:05.131 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:05.131 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:16:05.132 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:16:05.132 DEBUG:boto:Method: GET 2022-09-12T20:16:05.133 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:16:05.133 DEBUG:boto:Data: 2022-09-12T20:16:05.134 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.134 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.135 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.135 DEBUG:boto:Params: {} 2022-09-12T20:16:05.136 DEBUG:boto:Token: None 2022-09-12T20:16:05.137 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/ 2022-09-12T20:16:05.137 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY= 2022-09-12T20:16:05.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY=', 'Content-Length': '0'} 2022-09-12T20:16:05.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dae2f4eb68f2bf95-00631f9385-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.141 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:05.142 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:05.143 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:05.143 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:16:05.144 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:16:05.144 DEBUG:boto:Method: GET 2022-09-12T20:16:05.145 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:16:05.145 DEBUG:boto:Data: 2022-09-12T20:16:05.146 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.146 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.147 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.147 DEBUG:boto:Params: {} 2022-09-12T20:16:05.148 DEBUG:boto:Token: None 2022-09-12T20:16:05.149 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/ 2022-09-12T20:16:05.150 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY= 2022-09-12T20:16:05.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY=', 'Content-Length': '0'} 2022-09-12T20:16:05.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9a093a553fb29ef-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.157 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:05.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2022-09-12T20:16:05.159 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:05.160 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:05.160 DEBUG:boto:path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:05.161 DEBUG:boto:auth_path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:05.161 DEBUG:boto:Method: GET 2022-09-12T20:16:05.162 DEBUG:boto:Path: /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:05.163 DEBUG:boto:Data: 2022-09-12T20:16:05.163 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.164 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.164 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.165 DEBUG:boto:Params: {} 2022-09-12T20:16:05.165 DEBUG:boto:Token: None 2022-09-12T20:16:05.166 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:05.167 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ukYAJj/XLjzol5GQ6l0s1gF9zVg= 2022-09-12T20:16:05.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ukYAJj/XLjzol5GQ6l0s1gF9zVg=', 'Content-Length': '0'} 2022-09-12T20:16:05.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009476edcc4612c671-00631f9385-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.170 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:05.171 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:05.171 DEBUG:boto:path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:05.172 DEBUG:boto:auth_path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:05.173 DEBUG:boto:Method: GET 2022-09-12T20:16:05.173 DEBUG:boto:Path: /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:05.174 DEBUG:boto:Data: 2022-09-12T20:16:05.174 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.175 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.175 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.176 DEBUG:boto:Params: {} 2022-09-12T20:16:05.177 DEBUG:boto:Token: None 2022-09-12T20:16:05.177 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:05.178 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ukYAJj/XLjzol5GQ6l0s1gF9zVg= 2022-09-12T20:16:05.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ukYAJj/XLjzol5GQ6l0s1gF9zVg=', 'Content-Length': '0'} 2022-09-12T20:16:05.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012bd509c4af9b79c-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.181 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:05.182 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:05.183 DEBUG:boto:path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:05.184 DEBUG:boto:auth_path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:05.184 DEBUG:boto:Method: HEAD 2022-09-12T20:16:05.185 DEBUG:boto:Path: /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:05.185 DEBUG:boto:Data: 2022-09-12T20:16:05.186 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.187 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.188 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.188 DEBUG:boto:Params: {} 2022-09-12T20:16:05.189 DEBUG:boto:Token: None 2022-09-12T20:16:05.190 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:05.190 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uasiUCIUpFtif1z56njNspaeNIw= 2022-09-12T20:16:05.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uasiUCIUpFtif1z56njNspaeNIw=', 'Content-Length': '0'} 2022-09-12T20:16:05.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7a1997a26428466-00631f9385-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.194 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:05.194 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:05.195 DEBUG:boto:path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:05.195 DEBUG:boto:auth_path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:05.196 DEBUG:boto:Method: HEAD 2022-09-12T20:16:05.196 DEBUG:boto:Path: /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:05.197 DEBUG:boto:Data: 2022-09-12T20:16:05.197 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.198 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.198 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.199 DEBUG:boto:Params: {} 2022-09-12T20:16:05.199 DEBUG:boto:Token: None 2022-09-12T20:16:05.200 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:05.200 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uasiUCIUpFtif1z56njNspaeNIw= 2022-09-12T20:16:05.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uasiUCIUpFtif1z56njNspaeNIw=', 'Content-Length': '0'} 2022-09-12T20:16:05.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc43574561239f7b-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2022-09-12T20:16:05.204 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:05.205 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:05.205 DEBUG:boto:path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:05.206 DEBUG:boto:auth_path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:05.207 DEBUG:boto:Method: GET 2022-09-12T20:16:05.207 DEBUG:boto:Path: /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:05.208 DEBUG:boto:Data: 2022-09-12T20:16:05.208 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.209 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.209 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.210 DEBUG:boto:Params: {} 2022-09-12T20:16:05.210 DEBUG:boto:Token: None 2022-09-12T20:16:05.211 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:05.212 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ukYAJj/XLjzol5GQ6l0s1gF9zVg= 2022-09-12T20:16:05.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ukYAJj/XLjzol5GQ6l0s1gF9zVg=', 'Content-Length': '0'} 2022-09-12T20:16:05.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009096ec48868bc64f-00631f9385-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.215 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:05.216 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:05.216 DEBUG:boto:path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:05.217 DEBUG:boto:auth_path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:05.217 DEBUG:boto:Method: GET 2022-09-12T20:16:05.218 DEBUG:boto:Path: /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:05.218 DEBUG:boto:Data: 2022-09-12T20:16:05.219 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.219 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.220 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.221 DEBUG:boto:Params: {} 2022-09-12T20:16:05.221 DEBUG:boto:Token: None 2022-09-12T20:16:05.222 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:05.222 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ukYAJj/XLjzol5GQ6l0s1gF9zVg= 2022-09-12T20:16:05.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ukYAJj/XLjzol5GQ6l0s1gF9zVg=', 'Content-Length': '0'} 2022-09-12T20:16:05.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000892c9d687b7451f4-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.226 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:05.227 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:05.227 DEBUG:boto:path=/jdpiak-75/?prefix=obj74 2022-09-12T20:16:05.228 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj74 2022-09-12T20:16:05.228 DEBUG:boto:Method: GET 2022-09-12T20:16:05.229 DEBUG:boto:Path: /jdpiak-75/?prefix=obj74 2022-09-12T20:16:05.229 DEBUG:boto:Data: 2022-09-12T20:16:05.230 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.230 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.231 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.231 DEBUG:boto:Params: {} 2022-09-12T20:16:05.232 DEBUG:boto:Token: None 2022-09-12T20:16:05.232 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/ 2022-09-12T20:16:05.233 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY= 2022-09-12T20:16:05.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY=', 'Content-Length': '0'} 2022-09-12T20:16:05.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ea77008cabcc7c2-00631f9385-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.278 DEBUG:boto:b'jdpiak-75obj741000falseobj742022-09-12T20:14:27.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:05.279 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:05.280 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:05.281 DEBUG:boto:path=/jdpiak-75/?prefix=obj74 2022-09-12T20:16:05.281 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj74 2022-09-12T20:16:05.282 DEBUG:boto:Method: GET 2022-09-12T20:16:05.282 DEBUG:boto:Path: /jdpiak-75/?prefix=obj74 2022-09-12T20:16:05.283 DEBUG:boto:Data: 2022-09-12T20:16:05.283 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.284 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.284 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.285 DEBUG:boto:Params: {} 2022-09-12T20:16:05.285 DEBUG:boto:Token: None 2022-09-12T20:16:05.286 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/ 2022-09-12T20:16:05.287 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY= 2022-09-12T20:16:05.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY=', 'Content-Length': '0'} 2022-09-12T20:16:05.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000470200e48d0a7efb-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.293 DEBUG:boto:b'jdpiak-75obj741000falseobj742022-09-12T20:14:27.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:05.294 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:05.295 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:05.295 DEBUG:boto:Method: HEAD 2022-09-12T20:16:05.296 DEBUG:boto:Path: /jdpiak-75/obj74 2022-09-12T20:16:05.296 DEBUG:boto:Data: 2022-09-12T20:16:05.297 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.297 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.298 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.298 DEBUG:boto:Params: {} 2022-09-12T20:16:05.299 DEBUG:boto:Token: None 2022-09-12T20:16:05.299 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj74 2022-09-12T20:16:05.300 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TYTyYQWC8CT/cJ07Gfig0GxjMx8= 2022-09-12T20:16:05.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TYTyYQWC8CT/cJ07Gfig0GxjMx8=', 'Content-Length': '0'} 2022-09-12T20:16:05.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ce686a1d188301f-00631f9385-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.304 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:05.304 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:05.305 DEBUG:boto:Method: HEAD 2022-09-12T20:16:05.306 DEBUG:boto:Path: /jdpiak-75/obj74 2022-09-12T20:16:05.306 DEBUG:boto:Data: 2022-09-12T20:16:05.307 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.307 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.308 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.308 DEBUG:boto:Params: {} 2022-09-12T20:16:05.309 DEBUG:boto:Token: None 2022-09-12T20:16:05.309 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj74 2022-09-12T20:16:05.310 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TYTyYQWC8CT/cJ07Gfig0GxjMx8= 2022-09-12T20:16:05.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TYTyYQWC8CT/cJ07Gfig0GxjMx8=', 'Content-Length': '0'} 2022-09-12T20:16:05.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002178b508718811f0-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2022-09-12T20:16:05.314 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:05.315 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:05.316 DEBUG:boto:Method: GET 2022-09-12T20:16:05.316 DEBUG:boto:Path: /jdpiak-75/obj74 2022-09-12T20:16:05.317 DEBUG:boto:Data: 2022-09-12T20:16:05.318 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.318 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.319 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.319 DEBUG:boto:Params: {} 2022-09-12T20:16:05.320 DEBUG:boto:Token: None 2022-09-12T20:16:05.321 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj74 2022-09-12T20:16:05.321 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bCmV+kKGnaQu4wTZ4yx845p18tY= 2022-09-12T20:16:05.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bCmV+kKGnaQu4wTZ4yx845p18tY=', 'Content-Length': '0'} 2022-09-12T20:16:05.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008566047543c77d31-00631f9385-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.325 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:05.325 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:05.326 DEBUG:boto:Method: GET 2022-09-12T20:16:05.326 DEBUG:boto:Path: /jdpiak-75/obj74 2022-09-12T20:16:05.327 DEBUG:boto:Data: 2022-09-12T20:16:05.328 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.328 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.329 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.329 DEBUG:boto:Params: {} 2022-09-12T20:16:05.330 DEBUG:boto:Token: None 2022-09-12T20:16:05.331 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj74 2022-09-12T20:16:05.331 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bCmV+kKGnaQu4wTZ4yx845p18tY= 2022-09-12T20:16:05.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bCmV+kKGnaQu4wTZ4yx845p18tY=', 'Content-Length': '0'} 2022-09-12T20:16:05.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3a62c8956d5afa6-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.378 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:05.379 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:05.379 DEBUG:boto:path=/jdpiak-75/?prefix=obj74 2022-09-12T20:16:05.380 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj74 2022-09-12T20:16:05.381 DEBUG:boto:Method: GET 2022-09-12T20:16:05.382 DEBUG:boto:Path: /jdpiak-75/?prefix=obj74 2022-09-12T20:16:05.382 DEBUG:boto:Data: 2022-09-12T20:16:05.383 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.383 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.384 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.384 DEBUG:boto:Params: {} 2022-09-12T20:16:05.385 DEBUG:boto:Token: None 2022-09-12T20:16:05.386 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/ 2022-09-12T20:16:05.386 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY= 2022-09-12T20:16:05.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY=', 'Content-Length': '0'} 2022-09-12T20:16:05.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7ce81490ddd27f3-00631f9385-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.391 DEBUG:boto:b'jdpiak-75obj741000falseobj742022-09-12T20:14:27.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:05.392 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:05.392 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:05.393 DEBUG:boto:path=/jdpiak-75/?prefix=obj74 2022-09-12T20:16:05.393 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj74 2022-09-12T20:16:05.394 DEBUG:boto:Method: GET 2022-09-12T20:16:05.395 DEBUG:boto:Path: /jdpiak-75/?prefix=obj74 2022-09-12T20:16:05.395 DEBUG:boto:Data: 2022-09-12T20:16:05.396 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.396 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.397 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.397 DEBUG:boto:Params: {} 2022-09-12T20:16:05.398 DEBUG:boto:Token: None 2022-09-12T20:16:05.399 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/ 2022-09-12T20:16:05.399 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY= 2022-09-12T20:16:05.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY=', 'Content-Length': '0'} 2022-09-12T20:16:05.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b058fd28836a47d-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.406 DEBUG:boto:b'jdpiak-75obj741000falseobj742022-09-12T20:14:27.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:05.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:16:05.407 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:05.408 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:05.408 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:05.409 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:05.410 DEBUG:boto:Method: GET 2022-09-12T20:16:05.410 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:05.411 DEBUG:boto:Data: 2022-09-12T20:16:05.412 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.412 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.413 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.413 DEBUG:boto:Params: {} 2022-09-12T20:16:05.414 DEBUG:boto:Token: None 2022-09-12T20:16:05.415 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:05.415 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2/n7NLpOODBAxr7/mjvfC73V6xk= 2022-09-12T20:16:05.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2/n7NLpOODBAxr7/mjvfC73V6xk=', 'Content-Length': '0'} 2022-09-12T20:16:05.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053f8e2c6a1fe7d9b-00631f9385-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.462 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:05.463 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:05.463 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:05.464 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:05.465 DEBUG:boto:Method: GET 2022-09-12T20:16:05.465 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:05.466 DEBUG:boto:Data: 2022-09-12T20:16:05.466 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.467 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.467 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.468 DEBUG:boto:Params: {} 2022-09-12T20:16:05.469 DEBUG:boto:Token: None 2022-09-12T20:16:05.469 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:05.470 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2/n7NLpOODBAxr7/mjvfC73V6xk= 2022-09-12T20:16:05.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2/n7NLpOODBAxr7/mjvfC73V6xk=', 'Content-Length': '0'} 2022-09-12T20:16:05.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3b90cf8652452fe-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.474 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:05.475 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:05.476 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:05.477 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:05.477 DEBUG:boto:Method: HEAD 2022-09-12T20:16:05.478 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:05.478 DEBUG:boto:Data: 2022-09-12T20:16:05.479 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.480 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.480 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.481 DEBUG:boto:Params: {} 2022-09-12T20:16:05.481 DEBUG:boto:Token: None 2022-09-12T20:16:05.482 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:05.482 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bgNcRnTdEFPizRtaAlMDA2Q7Yy4= 2022-09-12T20:16:05.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bgNcRnTdEFPizRtaAlMDA2Q7Yy4=', 'Content-Length': '0'} 2022-09-12T20:16:05.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081a9251488e085f7-00631f9385-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.486 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:05.487 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:05.488 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:05.488 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:05.489 DEBUG:boto:Method: HEAD 2022-09-12T20:16:05.489 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:05.490 DEBUG:boto:Data: 2022-09-12T20:16:05.491 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.491 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.492 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.492 DEBUG:boto:Params: {} 2022-09-12T20:16:05.493 DEBUG:boto:Token: None 2022-09-12T20:16:05.493 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:05.494 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bgNcRnTdEFPizRtaAlMDA2Q7Yy4= 2022-09-12T20:16:05.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bgNcRnTdEFPizRtaAlMDA2Q7Yy4=', 'Content-Length': '0'} 2022-09-12T20:16:05.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062be10a7cd783471-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:16:05.499 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:05.499 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:05.500 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:05.500 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:05.501 DEBUG:boto:Method: GET 2022-09-12T20:16:05.501 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:05.502 DEBUG:boto:Data: 2022-09-12T20:16:05.503 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.503 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.504 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.504 DEBUG:boto:Params: {} 2022-09-12T20:16:05.505 DEBUG:boto:Token: None 2022-09-12T20:16:05.506 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:05.506 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2/n7NLpOODBAxr7/mjvfC73V6xk= 2022-09-12T20:16:05.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2/n7NLpOODBAxr7/mjvfC73V6xk=', 'Content-Length': '0'} 2022-09-12T20:16:05.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f78fe26eb07a1128-00631f9385-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.510 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:05.511 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:05.511 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:05.512 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:05.512 DEBUG:boto:Method: GET 2022-09-12T20:16:05.513 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:05.513 DEBUG:boto:Data: 2022-09-12T20:16:05.514 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.516 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.517 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.517 DEBUG:boto:Params: {} 2022-09-12T20:16:05.519 DEBUG:boto:Token: None 2022-09-12T20:16:05.519 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:05.520 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2/n7NLpOODBAxr7/mjvfC73V6xk= 2022-09-12T20:16:05.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2/n7NLpOODBAxr7/mjvfC73V6xk=', 'Content-Length': '0'} 2022-09-12T20:16:05.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e93ac2167c54732-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.525 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:05.526 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:05.527 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:16:05.529 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:16:05.530 DEBUG:boto:Method: GET 2022-09-12T20:16:05.532 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:16:05.534 DEBUG:boto:Data: 2022-09-12T20:16:05.535 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.536 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.536 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.537 DEBUG:boto:Params: {} 2022-09-12T20:16:05.537 DEBUG:boto:Token: None 2022-09-12T20:16:05.538 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/ 2022-09-12T20:16:05.539 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY= 2022-09-12T20:16:05.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY=', 'Content-Length': '0'} 2022-09-12T20:16:05.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d66a55c764a54e97-00631f9385-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.543 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:05.544 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:05.544 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:05.545 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:16:05.545 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:16:05.546 DEBUG:boto:Method: GET 2022-09-12T20:16:05.546 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:16:05.547 DEBUG:boto:Data: 2022-09-12T20:16:05.547 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.548 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.548 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.549 DEBUG:boto:Params: {} 2022-09-12T20:16:05.550 DEBUG:boto:Token: None 2022-09-12T20:16:05.550 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/ 2022-09-12T20:16:05.551 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY= 2022-09-12T20:16:05.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY=', 'Content-Length': '0'} 2022-09-12T20:16:05.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e687f4983582b0e-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.557 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:05.558 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:05.559 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:05.559 DEBUG:boto:Method: HEAD 2022-09-12T20:16:05.560 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:16:05.561 DEBUG:boto:Data: 2022-09-12T20:16:05.562 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.563 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.564 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.565 DEBUG:boto:Params: {} 2022-09-12T20:16:05.566 DEBUG:boto:Token: None 2022-09-12T20:16:05.566 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj76 2022-09-12T20:16:05.567 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mDSW8a92mK987O6O0XfNDWWK24w= 2022-09-12T20:16:05.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mDSW8a92mK987O6O0XfNDWWK24w=', 'Content-Length': '0'} 2022-09-12T20:16:05.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f8870538eebe6b0-00631f9385-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.570 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:05.571 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:05.571 DEBUG:boto:Method: HEAD 2022-09-12T20:16:05.572 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:16:05.573 DEBUG:boto:Data: 2022-09-12T20:16:05.573 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.574 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.574 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.575 DEBUG:boto:Params: {} 2022-09-12T20:16:05.575 DEBUG:boto:Token: None 2022-09-12T20:16:05.576 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj76 2022-09-12T20:16:05.576 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mDSW8a92mK987O6O0XfNDWWK24w= 2022-09-12T20:16:05.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mDSW8a92mK987O6O0XfNDWWK24w=', 'Content-Length': '0'} 2022-09-12T20:16:05.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d58af454d5358b19-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:16:05.580 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:05.581 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:05.582 DEBUG:boto:Method: GET 2022-09-12T20:16:05.582 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:16:05.583 DEBUG:boto:Data: 2022-09-12T20:16:05.584 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.584 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.585 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.585 DEBUG:boto:Params: {} 2022-09-12T20:16:05.586 DEBUG:boto:Token: None 2022-09-12T20:16:05.587 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj76 2022-09-12T20:16:05.587 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:j7K/7pgJC6iPBo9kG+fPHX0M+mM= 2022-09-12T20:16:05.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:j7K/7pgJC6iPBo9kG+fPHX0M+mM=', 'Content-Length': '0'} 2022-09-12T20:16:05.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c50f8e36ec489b85-00631f9385-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.591 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:05.592 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:05.593 DEBUG:boto:Method: GET 2022-09-12T20:16:05.594 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:16:05.595 DEBUG:boto:Data: 2022-09-12T20:16:05.596 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.597 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.598 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.599 DEBUG:boto:Params: {} 2022-09-12T20:16:05.600 DEBUG:boto:Token: None 2022-09-12T20:16:05.600 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj76 2022-09-12T20:16:05.602 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:j7K/7pgJC6iPBo9kG+fPHX0M+mM= 2022-09-12T20:16:05.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:j7K/7pgJC6iPBo9kG+fPHX0M+mM=', 'Content-Length': '0'} 2022-09-12T20:16:05.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090ced251316a9e7d-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.646 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:05.647 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:05.648 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:16:05.649 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:16:05.649 DEBUG:boto:Method: GET 2022-09-12T20:16:05.650 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:16:05.650 DEBUG:boto:Data: 2022-09-12T20:16:05.651 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.651 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.652 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.652 DEBUG:boto:Params: {} 2022-09-12T20:16:05.653 DEBUG:boto:Token: None 2022-09-12T20:16:05.654 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/ 2022-09-12T20:16:05.654 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY= 2022-09-12T20:16:05.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY=', 'Content-Length': '0'} 2022-09-12T20:16:05.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033b55a06bf7e535c-00631f9385-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.659 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:05.660 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:05.660 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:05.661 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:16:05.661 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:16:05.662 DEBUG:boto:Method: GET 2022-09-12T20:16:05.662 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:16:05.663 DEBUG:boto:Data: 2022-09-12T20:16:05.663 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.664 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.664 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.665 DEBUG:boto:Params: {} 2022-09-12T20:16:05.665 DEBUG:boto:Token: None 2022-09-12T20:16:05.666 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/ 2022-09-12T20:16:05.667 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY= 2022-09-12T20:16:05.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY=', 'Content-Length': '0'} 2022-09-12T20:16:05.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008dd7f196811149d-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.672 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:05.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:16:05.674 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:05.674 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:05.675 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:05.676 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:05.676 DEBUG:boto:Method: GET 2022-09-12T20:16:05.677 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:05.677 DEBUG:boto:Data: 2022-09-12T20:16:05.678 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.679 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.679 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.680 DEBUG:boto:Params: {} 2022-09-12T20:16:05.680 DEBUG:boto:Token: None 2022-09-12T20:16:05.681 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:05.682 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SpItusxnSMUVFzdj1324FprFiNo= 2022-09-12T20:16:05.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SpItusxnSMUVFzdj1324FprFiNo=', 'Content-Length': '0'} 2022-09-12T20:16:05.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065fe9271c0881a2b-00631f9385-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.726 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:05.727 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:05.727 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:05.728 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:05.728 DEBUG:boto:Method: GET 2022-09-12T20:16:05.729 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:05.730 DEBUG:boto:Data: 2022-09-12T20:16:05.730 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.731 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.731 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.732 DEBUG:boto:Params: {} 2022-09-12T20:16:05.732 DEBUG:boto:Token: None 2022-09-12T20:16:05.733 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:05.734 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SpItusxnSMUVFzdj1324FprFiNo= 2022-09-12T20:16:05.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SpItusxnSMUVFzdj1324FprFiNo=', 'Content-Length': '0'} 2022-09-12T20:16:05.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000383bb405ce61d8b6-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.738 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:05.738 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:05.739 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:05.739 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:05.740 DEBUG:boto:Method: HEAD 2022-09-12T20:16:05.740 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:05.741 DEBUG:boto:Data: 2022-09-12T20:16:05.741 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.742 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.743 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.743 DEBUG:boto:Params: {} 2022-09-12T20:16:05.744 DEBUG:boto:Token: None 2022-09-12T20:16:05.745 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:05.745 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WJAOvyLDS/WCk+oD5KMEVjDF8Wg= 2022-09-12T20:16:05.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WJAOvyLDS/WCk+oD5KMEVjDF8Wg=', 'Content-Length': '0'} 2022-09-12T20:16:05.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2c5b50338c99004-00631f9385-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.751 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:05.752 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:05.752 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:05.753 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:05.754 DEBUG:boto:Method: HEAD 2022-09-12T20:16:05.755 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:05.756 DEBUG:boto:Data: 2022-09-12T20:16:05.756 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.757 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.757 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.758 DEBUG:boto:Params: {} 2022-09-12T20:16:05.759 DEBUG:boto:Token: None 2022-09-12T20:16:05.759 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:05.760 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WJAOvyLDS/WCk+oD5KMEVjDF8Wg= 2022-09-12T20:16:05.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WJAOvyLDS/WCk+oD5KMEVjDF8Wg=', 'Content-Length': '0'} 2022-09-12T20:16:05.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e9269fae28ec69f-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:16:05.764 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:05.764 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:05.765 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:05.765 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:05.766 DEBUG:boto:Method: GET 2022-09-12T20:16:05.767 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:05.767 DEBUG:boto:Data: 2022-09-12T20:16:05.768 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.768 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.769 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.769 DEBUG:boto:Params: {} 2022-09-12T20:16:05.770 DEBUG:boto:Token: None 2022-09-12T20:16:05.771 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:05.771 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SpItusxnSMUVFzdj1324FprFiNo= 2022-09-12T20:16:05.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SpItusxnSMUVFzdj1324FprFiNo=', 'Content-Length': '0'} 2022-09-12T20:16:05.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018bb52590149b3be-00631f9385-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.775 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:05.776 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:05.776 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:05.777 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:05.777 DEBUG:boto:Method: GET 2022-09-12T20:16:05.778 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:05.778 DEBUG:boto:Data: 2022-09-12T20:16:05.779 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.779 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.780 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.780 DEBUG:boto:Params: {} 2022-09-12T20:16:05.781 DEBUG:boto:Token: None 2022-09-12T20:16:05.782 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:05.783 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SpItusxnSMUVFzdj1324FprFiNo= 2022-09-12T20:16:05.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SpItusxnSMUVFzdj1324FprFiNo=', 'Content-Length': '0'} 2022-09-12T20:16:05.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fe6799948d9674d-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.787 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:05.787 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:05.788 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:16:05.789 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:16:05.789 DEBUG:boto:Method: GET 2022-09-12T20:16:05.790 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:16:05.790 DEBUG:boto:Data: 2022-09-12T20:16:05.791 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.791 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.792 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.792 DEBUG:boto:Params: {} 2022-09-12T20:16:05.793 DEBUG:boto:Token: None 2022-09-12T20:16:05.794 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/ 2022-09-12T20:16:05.794 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY= 2022-09-12T20:16:05.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY=', 'Content-Length': '0'} 2022-09-12T20:16:05.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004cb325c5d81e928-00631f9385-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.798 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:05.799 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:05.800 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:05.800 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:16:05.801 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:16:05.802 DEBUG:boto:Method: GET 2022-09-12T20:16:05.802 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:16:05.803 DEBUG:boto:Data: 2022-09-12T20:16:05.803 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.804 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.804 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.805 DEBUG:boto:Params: {} 2022-09-12T20:16:05.806 DEBUG:boto:Token: None 2022-09-12T20:16:05.806 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/ 2022-09-12T20:16:05.807 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY= 2022-09-12T20:16:05.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY=', 'Content-Length': '0'} 2022-09-12T20:16:05.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040e684d5b89cef61-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.854 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:05.855 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:05.856 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:05.857 DEBUG:boto:Method: HEAD 2022-09-12T20:16:05.857 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:16:05.858 DEBUG:boto:Data: 2022-09-12T20:16:05.858 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.859 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.860 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.860 DEBUG:boto:Params: {} 2022-09-12T20:16:05.861 DEBUG:boto:Token: None 2022-09-12T20:16:05.862 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj85 2022-09-12T20:16:05.862 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RKH12Gl4Zy3rWfipDDyToeQ+Lto= 2022-09-12T20:16:05.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RKH12Gl4Zy3rWfipDDyToeQ+Lto=', 'Content-Length': '0'} 2022-09-12T20:16:05.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000605838017e2a5e5a-00631f9385-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.866 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:05.867 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:05.867 DEBUG:boto:Method: HEAD 2022-09-12T20:16:05.868 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:16:05.868 DEBUG:boto:Data: 2022-09-12T20:16:05.869 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.870 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.870 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.871 DEBUG:boto:Params: {} 2022-09-12T20:16:05.871 DEBUG:boto:Token: None 2022-09-12T20:16:05.872 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj85 2022-09-12T20:16:05.873 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RKH12Gl4Zy3rWfipDDyToeQ+Lto= 2022-09-12T20:16:05.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RKH12Gl4Zy3rWfipDDyToeQ+Lto=', 'Content-Length': '0'} 2022-09-12T20:16:05.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e27d8063b10365f-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:16:05.877 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:05.878 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:05.879 DEBUG:boto:Method: GET 2022-09-12T20:16:05.880 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:16:05.880 DEBUG:boto:Data: 2022-09-12T20:16:05.881 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.881 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.882 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.882 DEBUG:boto:Params: {} 2022-09-12T20:16:05.883 DEBUG:boto:Token: None 2022-09-12T20:16:05.884 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj85 2022-09-12T20:16:05.884 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:97ddU8vS59zRoFeOIsR7dFGaByE= 2022-09-12T20:16:05.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:97ddU8vS59zRoFeOIsR7dFGaByE=', 'Content-Length': '0'} 2022-09-12T20:16:05.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089c07fdf53166c4c-00631f9385-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.889 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:05.890 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:05.891 DEBUG:boto:Method: GET 2022-09-12T20:16:05.892 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:16:05.893 DEBUG:boto:Data: 2022-09-12T20:16:05.894 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.895 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.896 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.897 DEBUG:boto:Params: {} 2022-09-12T20:16:05.898 DEBUG:boto:Token: None 2022-09-12T20:16:05.899 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj85 2022-09-12T20:16:05.900 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:97ddU8vS59zRoFeOIsR7dFGaByE= 2022-09-12T20:16:05.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:97ddU8vS59zRoFeOIsR7dFGaByE=', 'Content-Length': '0'} 2022-09-12T20:16:05.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f2ac50fe536956e-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.906 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:05.907 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:05.908 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:16:05.909 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:16:05.910 DEBUG:boto:Method: GET 2022-09-12T20:16:05.911 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:16:05.912 DEBUG:boto:Data: 2022-09-12T20:16:05.913 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.914 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.915 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.916 DEBUG:boto:Params: {} 2022-09-12T20:16:05.917 DEBUG:boto:Token: None 2022-09-12T20:16:05.918 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/ 2022-09-12T20:16:05.919 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY= 2022-09-12T20:16:05.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY=', 'Content-Length': '0'} 2022-09-12T20:16:05.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e14ca4fdd8992302-00631f9385-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.925 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:05.926 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:05.927 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:05.928 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:16:05.929 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:16:05.930 DEBUG:boto:Method: GET 2022-09-12T20:16:05.931 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:16:05.932 DEBUG:boto:Data: 2022-09-12T20:16:05.933 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.934 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:05.935 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.936 DEBUG:boto:Params: {} 2022-09-12T20:16:05.937 DEBUG:boto:Token: None 2022-09-12T20:16:05.938 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/ 2022-09-12T20:16:05.940 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY= 2022-09-12T20:16:05.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JHjWfc1Ttx+BnkvIgmjblbDXOFY=', 'Content-Length': '0'} 2022-09-12T20:16:05.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001b437bfb69f5f40-00631f9385-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:05.947 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:05.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:16:05.950 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:05.951 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:05.952 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:05.953 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:05.953 DEBUG:boto:Method: GET 2022-09-12T20:16:05.954 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:05.954 DEBUG:boto:Data: 2022-09-12T20:16:05.955 DEBUG:boto:Headers: {} 2022-09-12T20:16:05.955 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:05.956 DEBUG:boto:Port: 8000 2022-09-12T20:16:05.956 DEBUG:boto:Params: {} 2022-09-12T20:16:05.957 DEBUG:boto:Token: None 2022-09-12T20:16:05.958 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:05 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:05.958 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:r0Vg7n4IydEt2Dgzypt7YODdt5c= 2022-09-12T20:16:05.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:r0Vg7n4IydEt2Dgzypt7YODdt5c=', 'Content-Length': '0'} 2022-09-12T20:16:05.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000219cff4a8e0590cf-00631f9385-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.006 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:06.007 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:06.007 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:06.008 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:06.009 DEBUG:boto:Method: GET 2022-09-12T20:16:06.009 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:06.010 DEBUG:boto:Data: 2022-09-12T20:16:06.010 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.011 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.011 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.012 DEBUG:boto:Params: {} 2022-09-12T20:16:06.012 DEBUG:boto:Token: None 2022-09-12T20:16:06.013 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:06.014 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YJlWlOczmvQs1qQBtTuXXRYYUTQ= 2022-09-12T20:16:06.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YJlWlOczmvQs1qQBtTuXXRYYUTQ=', 'Content-Length': '0'} 2022-09-12T20:16:06.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d263d0eb64c64ede-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.018 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:06.018 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:06.019 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:06.019 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:06.020 DEBUG:boto:Method: HEAD 2022-09-12T20:16:06.020 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:06.021 DEBUG:boto:Data: 2022-09-12T20:16:06.022 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.022 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.023 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.023 DEBUG:boto:Params: {} 2022-09-12T20:16:06.024 DEBUG:boto:Token: None 2022-09-12T20:16:06.025 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:06.025 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:92pN35JVaieRJjLylhGb/ryvJQY= 2022-09-12T20:16:06.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:92pN35JVaieRJjLylhGb/ryvJQY=', 'Content-Length': '0'} 2022-09-12T20:16:06.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6ad0468b7487e74-00631f9386-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.029 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:06.029 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:06.030 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:06.030 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:06.031 DEBUG:boto:Method: HEAD 2022-09-12T20:16:06.031 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:06.032 DEBUG:boto:Data: 2022-09-12T20:16:06.032 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.033 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.034 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.034 DEBUG:boto:Params: {} 2022-09-12T20:16:06.035 DEBUG:boto:Token: None 2022-09-12T20:16:06.035 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:06.036 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:92pN35JVaieRJjLylhGb/ryvJQY= 2022-09-12T20:16:06.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:92pN35JVaieRJjLylhGb/ryvJQY=', 'Content-Length': '0'} 2022-09-12T20:16:06.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9e08da08364cd55-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:16:06.041 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:06.041 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:06.042 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:06.042 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:06.043 DEBUG:boto:Method: GET 2022-09-12T20:16:06.044 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:06.044 DEBUG:boto:Data: 2022-09-12T20:16:06.045 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.046 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.046 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.047 DEBUG:boto:Params: {} 2022-09-12T20:16:06.047 DEBUG:boto:Token: None 2022-09-12T20:16:06.048 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:06.049 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YJlWlOczmvQs1qQBtTuXXRYYUTQ= 2022-09-12T20:16:06.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YJlWlOczmvQs1qQBtTuXXRYYUTQ=', 'Content-Length': '0'} 2022-09-12T20:16:06.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c141211830cde11-00631f9386-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.052 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:06.053 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:06.053 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:06.054 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:06.055 DEBUG:boto:Method: GET 2022-09-12T20:16:06.056 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:06.056 DEBUG:boto:Data: 2022-09-12T20:16:06.057 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.057 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.058 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.058 DEBUG:boto:Params: {} 2022-09-12T20:16:06.059 DEBUG:boto:Token: None 2022-09-12T20:16:06.060 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:06.060 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YJlWlOczmvQs1qQBtTuXXRYYUTQ= 2022-09-12T20:16:06.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YJlWlOczmvQs1qQBtTuXXRYYUTQ=', 'Content-Length': '0'} 2022-09-12T20:16:06.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000366575472457f69d-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.064 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:06.065 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:06.066 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:16:06.066 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:16:06.067 DEBUG:boto:Method: GET 2022-09-12T20:16:06.067 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:16:06.068 DEBUG:boto:Data: 2022-09-12T20:16:06.068 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.069 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.069 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.070 DEBUG:boto:Params: {} 2022-09-12T20:16:06.071 DEBUG:boto:Token: None 2022-09-12T20:16:06.071 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/ 2022-09-12T20:16:06.072 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc= 2022-09-12T20:16:06.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc=', 'Content-Length': '0'} 2022-09-12T20:16:06.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebb1d901a37842d4-00631f9386-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.076 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:06.077 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:06.078 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:06.078 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:16:06.079 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:16:06.079 DEBUG:boto:Method: GET 2022-09-12T20:16:06.080 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:16:06.080 DEBUG:boto:Data: 2022-09-12T20:16:06.081 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.081 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.082 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.082 DEBUG:boto:Params: {} 2022-09-12T20:16:06.083 DEBUG:boto:Token: None 2022-09-12T20:16:06.084 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/ 2022-09-12T20:16:06.084 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc= 2022-09-12T20:16:06.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc=', 'Content-Length': '0'} 2022-09-12T20:16:06.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040f84c0e56b26f48-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.130 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:06.131 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:06.132 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:06.132 DEBUG:boto:Method: HEAD 2022-09-12T20:16:06.133 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:16:06.133 DEBUG:boto:Data: 2022-09-12T20:16:06.134 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.134 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.135 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.135 DEBUG:boto:Params: {} 2022-09-12T20:16:06.136 DEBUG:boto:Token: None 2022-09-12T20:16:06.136 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj87 2022-09-12T20:16:06.137 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:w3NCPf7I7b+uaWJSHat1v7SPzVM= 2022-09-12T20:16:06.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:w3NCPf7I7b+uaWJSHat1v7SPzVM=', 'Content-Length': '0'} 2022-09-12T20:16:06.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054904ca7c8f17de6-00631f9386-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.140 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:06.141 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:06.141 DEBUG:boto:Method: HEAD 2022-09-12T20:16:06.142 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:16:06.142 DEBUG:boto:Data: 2022-09-12T20:16:06.143 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.143 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.144 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.144 DEBUG:boto:Params: {} 2022-09-12T20:16:06.145 DEBUG:boto:Token: None 2022-09-12T20:16:06.146 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj87 2022-09-12T20:16:06.146 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:w3NCPf7I7b+uaWJSHat1v7SPzVM= 2022-09-12T20:16:06.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:w3NCPf7I7b+uaWJSHat1v7SPzVM=', 'Content-Length': '0'} 2022-09-12T20:16:06.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d537f4574438d2fd-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:16:06.150 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:06.151 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:06.151 DEBUG:boto:Method: GET 2022-09-12T20:16:06.152 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:16:06.152 DEBUG:boto:Data: 2022-09-12T20:16:06.153 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.153 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.154 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.154 DEBUG:boto:Params: {} 2022-09-12T20:16:06.155 DEBUG:boto:Token: None 2022-09-12T20:16:06.156 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj87 2022-09-12T20:16:06.156 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tDa1h9WDggtzTX7rCRH1+l1ZBUI= 2022-09-12T20:16:06.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tDa1h9WDggtzTX7rCRH1+l1ZBUI=', 'Content-Length': '0'} 2022-09-12T20:16:06.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000755640db0e2090e7-00631f9386-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.160 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:06.161 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:06.161 DEBUG:boto:Method: GET 2022-09-12T20:16:06.162 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:16:06.162 DEBUG:boto:Data: 2022-09-12T20:16:06.163 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.163 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.164 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.164 DEBUG:boto:Params: {} 2022-09-12T20:16:06.165 DEBUG:boto:Token: None 2022-09-12T20:16:06.165 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj87 2022-09-12T20:16:06.166 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tDa1h9WDggtzTX7rCRH1+l1ZBUI= 2022-09-12T20:16:06.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tDa1h9WDggtzTX7rCRH1+l1ZBUI=', 'Content-Length': '0'} 2022-09-12T20:16:06.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085ea94d73ffb600d-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.170 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:06.170 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:06.171 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:16:06.171 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:16:06.172 DEBUG:boto:Method: GET 2022-09-12T20:16:06.172 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:16:06.173 DEBUG:boto:Data: 2022-09-12T20:16:06.173 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.174 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.174 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.175 DEBUG:boto:Params: {} 2022-09-12T20:16:06.175 DEBUG:boto:Token: None 2022-09-12T20:16:06.176 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/ 2022-09-12T20:16:06.176 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc= 2022-09-12T20:16:06.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc=', 'Content-Length': '0'} 2022-09-12T20:16:06.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c37e6ab633320ba-00631f9386-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.180 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:06.181 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:06.182 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:06.182 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:16:06.183 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:16:06.183 DEBUG:boto:Method: GET 2022-09-12T20:16:06.184 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:16:06.184 DEBUG:boto:Data: 2022-09-12T20:16:06.185 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.185 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.186 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.186 DEBUG:boto:Params: {} 2022-09-12T20:16:06.187 DEBUG:boto:Token: None 2022-09-12T20:16:06.187 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/ 2022-09-12T20:16:06.188 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc= 2022-09-12T20:16:06.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc=', 'Content-Length': '0'} 2022-09-12T20:16:06.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a56388b7bdc79f5-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.194 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:06.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:16:06.195 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:06.196 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:06.196 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:06.197 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:06.197 DEBUG:boto:Method: GET 2022-09-12T20:16:06.198 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:06.198 DEBUG:boto:Data: 2022-09-12T20:16:06.199 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.200 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.200 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.201 DEBUG:boto:Params: {} 2022-09-12T20:16:06.201 DEBUG:boto:Token: None 2022-09-12T20:16:06.202 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:06.202 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TmtOHGz8GFKkU956QBX8MC1G/b0= 2022-09-12T20:16:06.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TmtOHGz8GFKkU956QBX8MC1G/b0=', 'Content-Length': '0'} 2022-09-12T20:16:06.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d2ad43b78fc7d53-00631f9386-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.206 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:06.207 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:06.207 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:06.208 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:06.208 DEBUG:boto:Method: GET 2022-09-12T20:16:06.209 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:06.209 DEBUG:boto:Data: 2022-09-12T20:16:06.210 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.210 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.211 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.211 DEBUG:boto:Params: {} 2022-09-12T20:16:06.212 DEBUG:boto:Token: None 2022-09-12T20:16:06.212 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:06.213 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TmtOHGz8GFKkU956QBX8MC1G/b0= 2022-09-12T20:16:06.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TmtOHGz8GFKkU956QBX8MC1G/b0=', 'Content-Length': '0'} 2022-09-12T20:16:06.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003605b23c87dc7df-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.217 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:06.217 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:06.218 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:06.218 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:06.219 DEBUG:boto:Method: HEAD 2022-09-12T20:16:06.219 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:06.220 DEBUG:boto:Data: 2022-09-12T20:16:06.220 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.221 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.221 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.222 DEBUG:boto:Params: {} 2022-09-12T20:16:06.222 DEBUG:boto:Token: None 2022-09-12T20:16:06.223 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:06.223 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ydgiq/ON1eEk+E7LkVjrPXR6rVM= 2022-09-12T20:16:06.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ydgiq/ON1eEk+E7LkVjrPXR6rVM=', 'Content-Length': '0'} 2022-09-12T20:16:06.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000432ae67d5e59e29f-00631f9386-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.227 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:06.227 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:06.228 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:06.228 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:06.229 DEBUG:boto:Method: HEAD 2022-09-12T20:16:06.229 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:06.230 DEBUG:boto:Data: 2022-09-12T20:16:06.230 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.231 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.232 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.232 DEBUG:boto:Params: {} 2022-09-12T20:16:06.233 DEBUG:boto:Token: None 2022-09-12T20:16:06.233 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:06.234 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ydgiq/ON1eEk+E7LkVjrPXR6rVM= 2022-09-12T20:16:06.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ydgiq/ON1eEk+E7LkVjrPXR6rVM=', 'Content-Length': '0'} 2022-09-12T20:16:06.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000262025d178a5cd76-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:16:06.238 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:06.238 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:06.239 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:06.239 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:06.240 DEBUG:boto:Method: GET 2022-09-12T20:16:06.240 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:06.241 DEBUG:boto:Data: 2022-09-12T20:16:06.241 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.242 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.242 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.243 DEBUG:boto:Params: {} 2022-09-12T20:16:06.243 DEBUG:boto:Token: None 2022-09-12T20:16:06.244 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:06.245 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TmtOHGz8GFKkU956QBX8MC1G/b0= 2022-09-12T20:16:06.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TmtOHGz8GFKkU956QBX8MC1G/b0=', 'Content-Length': '0'} 2022-09-12T20:16:06.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000986f59f20b393dda-00631f9386-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.249 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:06.249 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:06.250 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:06.250 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:06.251 DEBUG:boto:Method: GET 2022-09-12T20:16:06.251 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:06.252 DEBUG:boto:Data: 2022-09-12T20:16:06.252 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.253 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.253 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.254 DEBUG:boto:Params: {} 2022-09-12T20:16:06.254 DEBUG:boto:Token: None 2022-09-12T20:16:06.255 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:06.256 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TmtOHGz8GFKkU956QBX8MC1G/b0= 2022-09-12T20:16:06.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TmtOHGz8GFKkU956QBX8MC1G/b0=', 'Content-Length': '0'} 2022-09-12T20:16:06.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000711737e89253def3-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.260 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:06.260 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:06.261 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:16:06.261 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:16:06.262 DEBUG:boto:Method: GET 2022-09-12T20:16:06.262 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:16:06.263 DEBUG:boto:Data: 2022-09-12T20:16:06.264 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.264 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.265 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.266 DEBUG:boto:Params: {} 2022-09-12T20:16:06.266 DEBUG:boto:Token: None 2022-09-12T20:16:06.267 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/ 2022-09-12T20:16:06.267 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc= 2022-09-12T20:16:06.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc=', 'Content-Length': '0'} 2022-09-12T20:16:06.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044dc711eb59c9b8b-00631f9386-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.272 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:06.273 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:06.273 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:06.274 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:16:06.275 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:16:06.275 DEBUG:boto:Method: GET 2022-09-12T20:16:06.276 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:16:06.276 DEBUG:boto:Data: 2022-09-12T20:16:06.277 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.277 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.278 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.279 DEBUG:boto:Params: {} 2022-09-12T20:16:06.279 DEBUG:boto:Token: None 2022-09-12T20:16:06.280 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/ 2022-09-12T20:16:06.280 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc= 2022-09-12T20:16:06.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc=', 'Content-Length': '0'} 2022-09-12T20:16:06.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee3a3dd82a7f2eee-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.330 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:06.331 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:06.332 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:06.333 DEBUG:boto:Method: HEAD 2022-09-12T20:16:06.333 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:16:06.334 DEBUG:boto:Data: 2022-09-12T20:16:06.334 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.335 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.335 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.336 DEBUG:boto:Params: {} 2022-09-12T20:16:06.337 DEBUG:boto:Token: None 2022-09-12T20:16:06.337 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj92 2022-09-12T20:16:06.338 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EgG+hR1XDpU+KsrxnXCHUETU1GY= 2022-09-12T20:16:06.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EgG+hR1XDpU+KsrxnXCHUETU1GY=', 'Content-Length': '0'} 2022-09-12T20:16:06.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000964e81531ee140b5-00631f9386-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.342 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:06.342 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:06.343 DEBUG:boto:Method: HEAD 2022-09-12T20:16:06.344 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:16:06.345 DEBUG:boto:Data: 2022-09-12T20:16:06.346 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.347 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.348 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.349 DEBUG:boto:Params: {} 2022-09-12T20:16:06.350 DEBUG:boto:Token: None 2022-09-12T20:16:06.351 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj92 2022-09-12T20:16:06.352 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EgG+hR1XDpU+KsrxnXCHUETU1GY= 2022-09-12T20:16:06.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EgG+hR1XDpU+KsrxnXCHUETU1GY=', 'Content-Length': '0'} 2022-09-12T20:16:06.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7268e7d0c12e65e-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:16:06.357 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:06.358 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:06.359 DEBUG:boto:Method: GET 2022-09-12T20:16:06.360 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:16:06.361 DEBUG:boto:Data: 2022-09-12T20:16:06.362 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.363 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.364 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.365 DEBUG:boto:Params: {} 2022-09-12T20:16:06.366 DEBUG:boto:Token: None 2022-09-12T20:16:06.367 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj92 2022-09-12T20:16:06.368 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hIIVRTGRhZVtEoquoqvcLosxgZE= 2022-09-12T20:16:06.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hIIVRTGRhZVtEoquoqvcLosxgZE=', 'Content-Length': '0'} 2022-09-12T20:16:06.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056dd23e6c54cf578-00631f9386-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.373 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:06.374 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:06.375 DEBUG:boto:Method: GET 2022-09-12T20:16:06.376 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:16:06.377 DEBUG:boto:Data: 2022-09-12T20:16:06.378 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.379 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.379 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.380 DEBUG:boto:Params: {} 2022-09-12T20:16:06.381 DEBUG:boto:Token: None 2022-09-12T20:16:06.382 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj92 2022-09-12T20:16:06.384 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hIIVRTGRhZVtEoquoqvcLosxgZE= 2022-09-12T20:16:06.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hIIVRTGRhZVtEoquoqvcLosxgZE=', 'Content-Length': '0'} 2022-09-12T20:16:06.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc8c4264468873bc-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.388 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:06.389 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:06.390 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:16:06.391 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:16:06.391 DEBUG:boto:Method: GET 2022-09-12T20:16:06.392 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:16:06.393 DEBUG:boto:Data: 2022-09-12T20:16:06.394 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.395 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.396 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.397 DEBUG:boto:Params: {} 2022-09-12T20:16:06.398 DEBUG:boto:Token: None 2022-09-12T20:16:06.399 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/ 2022-09-12T20:16:06.399 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc= 2022-09-12T20:16:06.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc=', 'Content-Length': '0'} 2022-09-12T20:16:06.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c64f3787dc6c1c2-00631f9386-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.403 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:06.404 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:06.405 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:06.406 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:16:06.407 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:16:06.407 DEBUG:boto:Method: GET 2022-09-12T20:16:06.408 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:16:06.408 DEBUG:boto:Data: 2022-09-12T20:16:06.409 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.409 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.410 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.410 DEBUG:boto:Params: {} 2022-09-12T20:16:06.411 DEBUG:boto:Token: None 2022-09-12T20:16:06.411 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/ 2022-09-12T20:16:06.412 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc= 2022-09-12T20:16:06.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc=', 'Content-Length': '0'} 2022-09-12T20:16:06.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000312b46e65bcb27dc-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.418 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:06.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-12T20:16:06.419 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:06.420 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:06.420 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:06.421 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:06.421 DEBUG:boto:Method: GET 2022-09-12T20:16:06.422 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:06.422 DEBUG:boto:Data: 2022-09-12T20:16:06.423 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.423 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.424 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.424 DEBUG:boto:Params: {} 2022-09-12T20:16:06.425 DEBUG:boto:Token: None 2022-09-12T20:16:06.426 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:06.426 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:srS2h4Hv68iw65KejrtNuybPd8Q= 2022-09-12T20:16:06.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:srS2h4Hv68iw65KejrtNuybPd8Q=', 'Content-Length': '0'} 2022-09-12T20:16:06.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000956c766f31f7af8b-00631f9386-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.470 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:06.471 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:06.471 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:06.472 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:06.472 DEBUG:boto:Method: GET 2022-09-12T20:16:06.473 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:06.473 DEBUG:boto:Data: 2022-09-12T20:16:06.474 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.475 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.475 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.476 DEBUG:boto:Params: {} 2022-09-12T20:16:06.476 DEBUG:boto:Token: None 2022-09-12T20:16:06.477 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:06.477 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:srS2h4Hv68iw65KejrtNuybPd8Q= 2022-09-12T20:16:06.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:srS2h4Hv68iw65KejrtNuybPd8Q=', 'Content-Length': '0'} 2022-09-12T20:16:06.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5f596e8c53aa8c7-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.481 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:06.482 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:06.482 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:06.483 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:06.483 DEBUG:boto:Method: HEAD 2022-09-12T20:16:06.484 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:06.484 DEBUG:boto:Data: 2022-09-12T20:16:06.485 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.485 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.486 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.486 DEBUG:boto:Params: {} 2022-09-12T20:16:06.487 DEBUG:boto:Token: None 2022-09-12T20:16:06.487 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:06.488 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2o8TKmYe3lQ8LcMFyZLM/PyD1KM= 2022-09-12T20:16:06.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2o8TKmYe3lQ8LcMFyZLM/PyD1KM=', 'Content-Length': '0'} 2022-09-12T20:16:06.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fe66a0820dd6254-00631f9386-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.492 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:06.492 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:06.493 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:06.493 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:06.494 DEBUG:boto:Method: HEAD 2022-09-12T20:16:06.495 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:06.495 DEBUG:boto:Data: 2022-09-12T20:16:06.496 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.496 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.497 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.497 DEBUG:boto:Params: {} 2022-09-12T20:16:06.498 DEBUG:boto:Token: None 2022-09-12T20:16:06.499 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:06.499 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2o8TKmYe3lQ8LcMFyZLM/PyD1KM= 2022-09-12T20:16:06.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2o8TKmYe3lQ8LcMFyZLM/PyD1KM=', 'Content-Length': '0'} 2022-09-12T20:16:06.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddd1a081e3be5f98-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-12T20:16:06.504 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:06.504 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:06.505 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:06.505 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:06.506 DEBUG:boto:Method: GET 2022-09-12T20:16:06.506 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:06.507 DEBUG:boto:Data: 2022-09-12T20:16:06.507 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.508 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.508 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.509 DEBUG:boto:Params: {} 2022-09-12T20:16:06.510 DEBUG:boto:Token: None 2022-09-12T20:16:06.510 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:06.511 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:srS2h4Hv68iw65KejrtNuybPd8Q= 2022-09-12T20:16:06.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:srS2h4Hv68iw65KejrtNuybPd8Q=', 'Content-Length': '0'} 2022-09-12T20:16:06.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000505daa6ba9499e9a-00631f9386-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.514 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:06.515 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:06.515 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:06.516 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:06.517 DEBUG:boto:Method: GET 2022-09-12T20:16:06.517 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:06.518 DEBUG:boto:Data: 2022-09-12T20:16:06.518 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.519 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.519 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.520 DEBUG:boto:Params: {} 2022-09-12T20:16:06.520 DEBUG:boto:Token: None 2022-09-12T20:16:06.521 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:06.522 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:srS2h4Hv68iw65KejrtNuybPd8Q= 2022-09-12T20:16:06.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:srS2h4Hv68iw65KejrtNuybPd8Q=', 'Content-Length': '0'} 2022-09-12T20:16:06.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0f725847fa65819-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.566 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:06.567 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:06.568 DEBUG:boto:path=/jdpiak-75/?prefix=obj93 2022-09-12T20:16:06.569 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj93 2022-09-12T20:16:06.569 DEBUG:boto:Method: GET 2022-09-12T20:16:06.570 DEBUG:boto:Path: /jdpiak-75/?prefix=obj93 2022-09-12T20:16:06.570 DEBUG:boto:Data: 2022-09-12T20:16:06.571 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.571 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.572 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.572 DEBUG:boto:Params: {} 2022-09-12T20:16:06.573 DEBUG:boto:Token: None 2022-09-12T20:16:06.574 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/ 2022-09-12T20:16:06.574 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc= 2022-09-12T20:16:06.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc=', 'Content-Length': '0'} 2022-09-12T20:16:06.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c44352382e8bc5fd-00631f9386-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.579 DEBUG:boto:b'jdpiak-75obj931000falseobj932022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:06.580 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:06.580 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:06.581 DEBUG:boto:path=/jdpiak-75/?prefix=obj93 2022-09-12T20:16:06.581 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj93 2022-09-12T20:16:06.582 DEBUG:boto:Method: GET 2022-09-12T20:16:06.582 DEBUG:boto:Path: /jdpiak-75/?prefix=obj93 2022-09-12T20:16:06.583 DEBUG:boto:Data: 2022-09-12T20:16:06.584 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.584 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.585 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.586 DEBUG:boto:Params: {} 2022-09-12T20:16:06.586 DEBUG:boto:Token: None 2022-09-12T20:16:06.587 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/ 2022-09-12T20:16:06.587 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc= 2022-09-12T20:16:06.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc=', 'Content-Length': '0'} 2022-09-12T20:16:06.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077bfe88cd17b22d5-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.593 DEBUG:boto:b'jdpiak-75obj931000falseobj932022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:06.594 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:06.595 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:06.596 DEBUG:boto:Method: HEAD 2022-09-12T20:16:06.596 DEBUG:boto:Path: /jdpiak-75/obj93 2022-09-12T20:16:06.597 DEBUG:boto:Data: 2022-09-12T20:16:06.597 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.598 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.598 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.599 DEBUG:boto:Params: {} 2022-09-12T20:16:06.599 DEBUG:boto:Token: None 2022-09-12T20:16:06.600 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj93 2022-09-12T20:16:06.601 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iSpXxZFaW78w2Ia+tMKtSpFsxgQ= 2022-09-12T20:16:06.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iSpXxZFaW78w2Ia+tMKtSpFsxgQ=', 'Content-Length': '0'} 2022-09-12T20:16:06.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcef20b9bae216e0-00631f9386-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.604 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:06.605 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:06.605 DEBUG:boto:Method: HEAD 2022-09-12T20:16:06.606 DEBUG:boto:Path: /jdpiak-75/obj93 2022-09-12T20:16:06.606 DEBUG:boto:Data: 2022-09-12T20:16:06.607 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.607 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.608 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.608 DEBUG:boto:Params: {} 2022-09-12T20:16:06.609 DEBUG:boto:Token: None 2022-09-12T20:16:06.609 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj93 2022-09-12T20:16:06.610 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iSpXxZFaW78w2Ia+tMKtSpFsxgQ= 2022-09-12T20:16:06.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iSpXxZFaW78w2Ia+tMKtSpFsxgQ=', 'Content-Length': '0'} 2022-09-12T20:16:06.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e67bc763f6d2c88-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-12T20:16:06.614 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:06.615 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:06.615 DEBUG:boto:Method: GET 2022-09-12T20:16:06.616 DEBUG:boto:Path: /jdpiak-75/obj93 2022-09-12T20:16:06.616 DEBUG:boto:Data: 2022-09-12T20:16:06.617 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.617 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.618 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.618 DEBUG:boto:Params: {} 2022-09-12T20:16:06.619 DEBUG:boto:Token: None 2022-09-12T20:16:06.620 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj93 2022-09-12T20:16:06.620 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sjD7OwV9wysl7AFhFHoATzrFTkE= 2022-09-12T20:16:06.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sjD7OwV9wysl7AFhFHoATzrFTkE=', 'Content-Length': '0'} 2022-09-12T20:16:06.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1819cb402aa1ae2-00631f9386-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.670 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:06.671 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:06.671 DEBUG:boto:Method: GET 2022-09-12T20:16:06.672 DEBUG:boto:Path: /jdpiak-75/obj93 2022-09-12T20:16:06.672 DEBUG:boto:Data: 2022-09-12T20:16:06.673 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.674 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.674 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.675 DEBUG:boto:Params: {} 2022-09-12T20:16:06.675 DEBUG:boto:Token: None 2022-09-12T20:16:06.676 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj93 2022-09-12T20:16:06.677 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sjD7OwV9wysl7AFhFHoATzrFTkE= 2022-09-12T20:16:06.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sjD7OwV9wysl7AFhFHoATzrFTkE=', 'Content-Length': '0'} 2022-09-12T20:16:06.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b70ded3e0a7016e0-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.681 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:06.681 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:06.682 DEBUG:boto:path=/jdpiak-75/?prefix=obj93 2022-09-12T20:16:06.693 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj93 2022-09-12T20:16:06.694 DEBUG:boto:Method: GET 2022-09-12T20:16:06.695 DEBUG:boto:Path: /jdpiak-75/?prefix=obj93 2022-09-12T20:16:06.695 DEBUG:boto:Data: 2022-09-12T20:16:06.696 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.696 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.697 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.697 DEBUG:boto:Params: {} 2022-09-12T20:16:06.698 DEBUG:boto:Token: None 2022-09-12T20:16:06.699 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/ 2022-09-12T20:16:06.699 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc= 2022-09-12T20:16:06.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc=', 'Content-Length': '0'} 2022-09-12T20:16:06.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000337d19fee677b8d2-00631f9386-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.703 DEBUG:boto:b'jdpiak-75obj931000falseobj932022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:06.704 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:06.705 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:06.705 DEBUG:boto:path=/jdpiak-75/?prefix=obj93 2022-09-12T20:16:06.706 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj93 2022-09-12T20:16:06.707 DEBUG:boto:Method: GET 2022-09-12T20:16:06.707 DEBUG:boto:Path: /jdpiak-75/?prefix=obj93 2022-09-12T20:16:06.708 DEBUG:boto:Data: 2022-09-12T20:16:06.709 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.710 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.710 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.711 DEBUG:boto:Params: {} 2022-09-12T20:16:06.712 DEBUG:boto:Token: None 2022-09-12T20:16:06.713 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/ 2022-09-12T20:16:06.713 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc= 2022-09-12T20:16:06.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc=', 'Content-Length': '0'} 2022-09-12T20:16:06.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f68fe230665a0290-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.720 DEBUG:boto:b'jdpiak-75obj931000falseobj932022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:06.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:16:06.721 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:06.722 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:06.723 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:06.723 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:06.724 DEBUG:boto:Method: GET 2022-09-12T20:16:06.725 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:06.725 DEBUG:boto:Data: 2022-09-12T20:16:06.726 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.727 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.728 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.728 DEBUG:boto:Params: {} 2022-09-12T20:16:06.729 DEBUG:boto:Token: None 2022-09-12T20:16:06.730 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:06.731 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6LUStuFwzPG5wicSRJK+ujULAy8= 2022-09-12T20:16:06.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6LUStuFwzPG5wicSRJK+ujULAy8=', 'Content-Length': '0'} 2022-09-12T20:16:06.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6889d8a6c1ed614-00631f9386-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.735 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:06.736 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:06.736 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:06.737 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:06.738 DEBUG:boto:Method: GET 2022-09-12T20:16:06.739 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:06.739 DEBUG:boto:Data: 2022-09-12T20:16:06.740 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.741 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.742 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.742 DEBUG:boto:Params: {} 2022-09-12T20:16:06.743 DEBUG:boto:Token: None 2022-09-12T20:16:06.744 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:06.745 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6LUStuFwzPG5wicSRJK+ujULAy8= 2022-09-12T20:16:06.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6LUStuFwzPG5wicSRJK+ujULAy8=', 'Content-Length': '0'} 2022-09-12T20:16:06.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075a644f8975cda40-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.794 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:06.795 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:06.795 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:06.796 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:06.796 DEBUG:boto:Method: HEAD 2022-09-12T20:16:06.797 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:06.798 DEBUG:boto:Data: 2022-09-12T20:16:06.798 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.799 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.799 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.800 DEBUG:boto:Params: {} 2022-09-12T20:16:06.800 DEBUG:boto:Token: None 2022-09-12T20:16:06.801 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:06.802 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2pYT/TORzK0lbCnSqN8Xc5DpHp8= 2022-09-12T20:16:06.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2pYT/TORzK0lbCnSqN8Xc5DpHp8=', 'Content-Length': '0'} 2022-09-12T20:16:06.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008273bd36ef4cfa6-00631f9386-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.805 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:06.805 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:06.806 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:06.807 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:06.807 DEBUG:boto:Method: HEAD 2022-09-12T20:16:06.808 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:06.808 DEBUG:boto:Data: 2022-09-12T20:16:06.809 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.809 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.810 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.810 DEBUG:boto:Params: {} 2022-09-12T20:16:06.811 DEBUG:boto:Token: None 2022-09-12T20:16:06.812 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:06.813 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2pYT/TORzK0lbCnSqN8Xc5DpHp8= 2022-09-12T20:16:06.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2pYT/TORzK0lbCnSqN8Xc5DpHp8=', 'Content-Length': '0'} 2022-09-12T20:16:06.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfc037b14a12bfaf-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:16:06.817 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:06.818 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:06.819 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:06.820 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:06.821 DEBUG:boto:Method: GET 2022-09-12T20:16:06.822 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:06.823 DEBUG:boto:Data: 2022-09-12T20:16:06.824 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.825 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.826 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.827 DEBUG:boto:Params: {} 2022-09-12T20:16:06.828 DEBUG:boto:Token: None 2022-09-12T20:16:06.829 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:06.830 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6LUStuFwzPG5wicSRJK+ujULAy8= 2022-09-12T20:16:06.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6LUStuFwzPG5wicSRJK+ujULAy8=', 'Content-Length': '0'} 2022-09-12T20:16:06.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f32c29fde4680ad-00631f9386-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.835 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:06.836 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:06.837 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:06.838 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:06.839 DEBUG:boto:Method: GET 2022-09-12T20:16:06.840 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:06.841 DEBUG:boto:Data: 2022-09-12T20:16:06.842 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.843 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.844 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.845 DEBUG:boto:Params: {} 2022-09-12T20:16:06.846 DEBUG:boto:Token: None 2022-09-12T20:16:06.847 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:06.848 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6LUStuFwzPG5wicSRJK+ujULAy8= 2022-09-12T20:16:06.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6LUStuFwzPG5wicSRJK+ujULAy8=', 'Content-Length': '0'} 2022-09-12T20:16:06.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bed3eb511aaf8afd-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.853 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:06.854 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:06.855 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:16:06.856 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:16:06.857 DEBUG:boto:Method: GET 2022-09-12T20:16:06.857 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:16:06.858 DEBUG:boto:Data: 2022-09-12T20:16:06.859 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.859 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.860 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.861 DEBUG:boto:Params: {} 2022-09-12T20:16:06.861 DEBUG:boto:Token: None 2022-09-12T20:16:06.862 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/ 2022-09-12T20:16:06.863 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc= 2022-09-12T20:16:06.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc=', 'Content-Length': '0'} 2022-09-12T20:16:06.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c497bf787ae3d2d-00631f9386-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.910 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:06.911 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:06.912 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:06.913 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:16:06.914 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:16:06.915 DEBUG:boto:Method: GET 2022-09-12T20:16:06.916 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:16:06.917 DEBUG:boto:Data: 2022-09-12T20:16:06.917 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.918 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.919 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.919 DEBUG:boto:Params: {} 2022-09-12T20:16:06.920 DEBUG:boto:Token: None 2022-09-12T20:16:06.921 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/ 2022-09-12T20:16:06.922 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc= 2022-09-12T20:16:06.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc=', 'Content-Length': '0'} 2022-09-12T20:16:06.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd674d0c1c4c7a95-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.929 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:06.930 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:06.931 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:06.932 DEBUG:boto:Method: HEAD 2022-09-12T20:16:06.932 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:16:06.933 DEBUG:boto:Data: 2022-09-12T20:16:06.934 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.935 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.936 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.936 DEBUG:boto:Params: {} 2022-09-12T20:16:06.938 DEBUG:boto:Token: None 2022-09-12T20:16:06.939 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj95 2022-09-12T20:16:06.939 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:j/vT0q+VLIbiSQr92/If3R2U8GM= 2022-09-12T20:16:06.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:j/vT0q+VLIbiSQr92/If3R2U8GM=', 'Content-Length': '0'} 2022-09-12T20:16:06.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045b07e94f5cc2426-00631f9386-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.943 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:06.944 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:06.945 DEBUG:boto:Method: HEAD 2022-09-12T20:16:06.945 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:16:06.946 DEBUG:boto:Data: 2022-09-12T20:16:06.946 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.947 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.947 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.948 DEBUG:boto:Params: {} 2022-09-12T20:16:06.948 DEBUG:boto:Token: None 2022-09-12T20:16:06.949 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj95 2022-09-12T20:16:06.950 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:j/vT0q+VLIbiSQr92/If3R2U8GM= 2022-09-12T20:16:06.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:j/vT0q+VLIbiSQr92/If3R2U8GM=', 'Content-Length': '0'} 2022-09-12T20:16:06.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b6677a1dad8e5c5-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:16:06.954 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:06.955 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:06.955 DEBUG:boto:Method: GET 2022-09-12T20:16:06.956 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:16:06.956 DEBUG:boto:Data: 2022-09-12T20:16:06.957 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.958 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.958 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.959 DEBUG:boto:Params: {} 2022-09-12T20:16:06.959 DEBUG:boto:Token: None 2022-09-12T20:16:06.960 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj95 2022-09-12T20:16:06.960 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ldNWE2FxOO1+nyzsd3oFBBv7Byg= 2022-09-12T20:16:06.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ldNWE2FxOO1+nyzsd3oFBBv7Byg=', 'Content-Length': '0'} 2022-09-12T20:16:06.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007075922d2bcb039a-00631f9386-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.964 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:06.965 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:06.965 DEBUG:boto:Method: GET 2022-09-12T20:16:06.966 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:16:06.966 DEBUG:boto:Data: 2022-09-12T20:16:06.967 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.967 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.968 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.969 DEBUG:boto:Params: {} 2022-09-12T20:16:06.969 DEBUG:boto:Token: None 2022-09-12T20:16:06.970 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/obj95 2022-09-12T20:16:06.970 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ldNWE2FxOO1+nyzsd3oFBBv7Byg= 2022-09-12T20:16:06.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ldNWE2FxOO1+nyzsd3oFBBv7Byg=', 'Content-Length': '0'} 2022-09-12T20:16:06.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1d6b1ff2527a3e7-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.974 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:06.975 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:06.975 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:16:06.976 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:16:06.976 DEBUG:boto:Method: GET 2022-09-12T20:16:06.977 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:16:06.977 DEBUG:boto:Data: 2022-09-12T20:16:06.978 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.978 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:06.979 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.980 DEBUG:boto:Params: {} 2022-09-12T20:16:06.980 DEBUG:boto:Token: None 2022-09-12T20:16:06.981 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/ 2022-09-12T20:16:06.981 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc= 2022-09-12T20:16:06.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc=', 'Content-Length': '0'} 2022-09-12T20:16:06.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012bc97326be2e0f7-00631f9386-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:06.985 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:06.986 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:06.987 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:06.987 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:16:06.988 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:16:06.989 DEBUG:boto:Method: GET 2022-09-12T20:16:06.989 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:16:06.990 DEBUG:boto:Data: 2022-09-12T20:16:06.990 DEBUG:boto:Headers: {} 2022-09-12T20:16:06.991 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:06.991 DEBUG:boto:Port: 8000 2022-09-12T20:16:06.992 DEBUG:boto:Params: {} 2022-09-12T20:16:06.992 DEBUG:boto:Token: None 2022-09-12T20:16:06.993 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:06 GMT /jdpiak-75/ 2022-09-12T20:16:06.994 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc= 2022-09-12T20:16:06.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:06 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jyVR71N/yBiUJdG48g/riwlbOnc=', 'Content-Length': '0'} 2022-09-12T20:16:06.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ff1463972af8499-00631f9386-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.000 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:07.001 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-75 zones={test-zone2, test-zone1} 2022-09-12T20:16:07.001 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-75 zones={test-zone3, test-zone1} 2022-09-12T20:16:07.002 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:07.002 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:07.003 DEBUG:boto:Method: HEAD 2022-09-12T20:16:07.004 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:16:07.004 DEBUG:boto:Data: 2022-09-12T20:16:07.005 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.005 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.006 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.006 DEBUG:boto:Params: {} 2022-09-12T20:16:07.007 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T20:16:07.007 DEBUG:boto:Token: None 2022-09-12T20:16:07.008 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/ 2022-09-12T20:16:07.009 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Wzna86ZyjI43OwByUygbfe/hJSQ= 2022-09-12T20:16:07.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Wzna86ZyjI43OwByUygbfe/hJSQ=', 'Content-Length': '0'} 2022-09-12T20:16:07.013 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f569ec63a4990d20-00631f9387-1826-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.014 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:07.015 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:07.015 DEBUG:boto:Method: HEAD 2022-09-12T20:16:07.016 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:16:07.016 DEBUG:boto:Data: 2022-09-12T20:16:07.017 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.017 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.018 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.018 DEBUG:boto:Params: {} 2022-09-12T20:16:07.019 DEBUG:boto:Token: None 2022-09-12T20:16:07.019 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/ 2022-09-12T20:16:07.020 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Wzna86ZyjI43OwByUygbfe/hJSQ= 2022-09-12T20:16:07.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Wzna86ZyjI43OwByUygbfe/hJSQ=', 'Content-Length': '0'} 2022-09-12T20:16:07.025 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0704d9bf8310bdb-00631f9387-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.026 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T20:16:07.026 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:07.027 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:07.027 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:16:07.028 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:16:07.028 DEBUG:boto:Method: GET 2022-09-12T20:16:07.029 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:16:07.029 DEBUG:boto:Data: 2022-09-12T20:16:07.030 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.030 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.031 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.031 DEBUG:boto:Params: {} 2022-09-12T20:16:07.032 DEBUG:boto:Token: None 2022-09-12T20:16:07.033 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/?versions 2022-09-12T20:16:07.033 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NjnDx9KFudg0aMD67iOqqBfd8oA= 2022-09-12T20:16:07.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NjnDx9KFudg0aMD67iOqqBfd8oA=', 'Content-Length': '0'} 2022-09-12T20:16:07.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007860894e322c9fe9-00631f9387-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.038 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2022-09-12T20:14:27.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2022-09-12T20:14:27.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2022-09-12T20:14:26.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2022-09-12T20:14:27.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:07.040 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:16:07.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2022-09-12T20:16:07.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-12T20:16:07.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-12T20:16:07.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2022-09-12T20:16:07.043 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-12T20:16:07.043 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2022-09-12T20:16:07.044 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2022-09-12T20:16:07.044 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:16:07.045 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2022-09-12T20:16:07.045 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:16:07.046 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:16:07.046 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-12T20:16:07.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-12T20:16:07.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2022-09-12T20:16:07.048 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:16:07.048 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:16:07.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2022-09-12T20:16:07.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:16:07.050 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:16:07.050 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-12T20:16:07.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:16:07.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:16:07.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:16:07.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-12T20:16:07.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2022-09-12T20:16:07.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:16:07.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-12T20:16:07.054 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2022-09-12T20:16:07.054 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:16:07.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:16:07.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-12T20:16:07.056 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-12T20:16:07.056 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2022-09-12T20:16:07.057 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:16:07.057 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T20:16:07.058 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:07.059 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:07.059 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:16:07.060 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:16:07.060 DEBUG:boto:Method: GET 2022-09-12T20:16:07.061 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:16:07.061 DEBUG:boto:Data: 2022-09-12T20:16:07.062 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.062 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.063 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.063 DEBUG:boto:Params: {} 2022-09-12T20:16:07.064 DEBUG:boto:Token: None 2022-09-12T20:16:07.065 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/?versions 2022-09-12T20:16:07.065 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NjnDx9KFudg0aMD67iOqqBfd8oA= 2022-09-12T20:16:07.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NjnDx9KFudg0aMD67iOqqBfd8oA=', 'Content-Length': '0'} 2022-09-12T20:16:07.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a4f5bb836e256a8-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.071 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2022-09-12T20:14:27.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2022-09-12T20:14:27.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2022-09-12T20:14:26.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2022-09-12T20:14:27.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:07.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:16:07.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2022-09-12T20:16:07.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-12T20:16:07.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-12T20:16:07.076 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2022-09-12T20:16:07.076 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-12T20:16:07.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2022-09-12T20:16:07.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2022-09-12T20:16:07.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:16:07.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2022-09-12T20:16:07.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:16:07.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:16:07.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-12T20:16:07.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-12T20:16:07.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2022-09-12T20:16:07.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:16:07.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:16:07.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2022-09-12T20:16:07.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:16:07.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:16:07.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-12T20:16:07.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:16:07.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:16:07.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:16:07.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-12T20:16:07.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2022-09-12T20:16:07.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:16:07.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-12T20:16:07.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2022-09-12T20:16:07.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:16:07.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:16:07.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-12T20:16:07.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-12T20:16:07.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2022-09-12T20:16:07.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:16:07.091 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:07.092 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:07.092 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:16:07.093 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:16:07.093 DEBUG:boto:Method: GET 2022-09-12T20:16:07.094 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:16:07.094 DEBUG:boto:Data: 2022-09-12T20:16:07.095 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.095 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.096 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.096 DEBUG:boto:Params: {} 2022-09-12T20:16:07.097 DEBUG:boto:Token: None 2022-09-12T20:16:07.098 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/?versions 2022-09-12T20:16:07.098 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NjnDx9KFudg0aMD67iOqqBfd8oA= 2022-09-12T20:16:07.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NjnDx9KFudg0aMD67iOqqBfd8oA=', 'Content-Length': '0'} 2022-09-12T20:16:07.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d73307a5d1ad9536-00631f9387-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.103 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2022-09-12T20:14:27.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2022-09-12T20:14:27.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2022-09-12T20:14:26.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2022-09-12T20:14:27.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:07.105 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:07.106 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:07.106 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:16:07.107 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:16:07.107 DEBUG:boto:Method: GET 2022-09-12T20:16:07.108 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:16:07.108 DEBUG:boto:Data: 2022-09-12T20:16:07.109 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.109 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.110 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.111 DEBUG:boto:Params: {} 2022-09-12T20:16:07.111 DEBUG:boto:Token: None 2022-09-12T20:16:07.112 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/?versions 2022-09-12T20:16:07.112 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NjnDx9KFudg0aMD67iOqqBfd8oA= 2022-09-12T20:16:07.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NjnDx9KFudg0aMD67iOqqBfd8oA=', 'Content-Length': '0'} 2022-09-12T20:16:07.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd0fda9f837843d8-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.119 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2022-09-12T20:14:27.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2022-09-12T20:14:27.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2022-09-12T20:14:26.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2022-09-12T20:14:27.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:07.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:16:07.122 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:16:07.122 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:16:07.123 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:16:07.123 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:16:07.124 DEBUG:boto:Method: GET 2022-09-12T20:16:07.124 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:16:07.125 DEBUG:boto:Data: 2022-09-12T20:16:07.125 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.126 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.127 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.127 DEBUG:boto:Params: {} 2022-09-12T20:16:07.128 DEBUG:boto:Token: None 2022-09-12T20:16:07.128 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:16:07.129 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ObE0gEULsIWfBM+qRA4ozgOymJc= 2022-09-12T20:16:07.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ObE0gEULsIWfBM+qRA4ozgOymJc=', 'Content-Length': '0'} 2022-09-12T20:16:07.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fe1619dac1b8b07-00631f9387-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.133 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:16:07.134 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:16:07.134 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:16:07.135 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:16:07.135 DEBUG:boto:Method: GET 2022-09-12T20:16:07.136 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:16:07.136 DEBUG:boto:Data: 2022-09-12T20:16:07.137 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.137 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.138 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.139 DEBUG:boto:Params: {} 2022-09-12T20:16:07.139 DEBUG:boto:Token: None 2022-09-12T20:16:07.140 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:16:07.140 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ObE0gEULsIWfBM+qRA4ozgOymJc= 2022-09-12T20:16:07.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ObE0gEULsIWfBM+qRA4ozgOymJc=', 'Content-Length': '0'} 2022-09-12T20:16:07.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd7531d35c6304df-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.144 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:16:07.145 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:16:07.145 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:16:07.146 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:16:07.146 DEBUG:boto:Method: HEAD 2022-09-12T20:16:07.147 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:16:07.147 DEBUG:boto:Data: 2022-09-12T20:16:07.148 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.148 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.149 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.150 DEBUG:boto:Params: {} 2022-09-12T20:16:07.150 DEBUG:boto:Token: None 2022-09-12T20:16:07.151 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:16:07.151 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0n5FFlTFtPENLqIAsW2utLYIaI4= 2022-09-12T20:16:07.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0n5FFlTFtPENLqIAsW2utLYIaI4=', 'Content-Length': '0'} 2022-09-12T20:16:07.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5d06b07150742e3-00631f9387-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.155 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:16:07.155 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:16:07.156 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:16:07.156 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:16:07.157 DEBUG:boto:Method: HEAD 2022-09-12T20:16:07.157 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:16:07.158 DEBUG:boto:Data: 2022-09-12T20:16:07.158 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.159 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.160 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.160 DEBUG:boto:Params: {} 2022-09-12T20:16:07.161 DEBUG:boto:Token: None 2022-09-12T20:16:07.161 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:16:07.162 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0n5FFlTFtPENLqIAsW2utLYIaI4= 2022-09-12T20:16:07.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0n5FFlTFtPENLqIAsW2utLYIaI4=', 'Content-Length': '0'} 2022-09-12T20:16:07.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e4168ec1799687d-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:16:07.166 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:16:07.167 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:16:07.167 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:16:07.168 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:16:07.168 DEBUG:boto:Method: GET 2022-09-12T20:16:07.169 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:16:07.169 DEBUG:boto:Data: 2022-09-12T20:16:07.170 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.171 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.171 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.172 DEBUG:boto:Params: {} 2022-09-12T20:16:07.172 DEBUG:boto:Token: None 2022-09-12T20:16:07.173 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:16:07.173 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ObE0gEULsIWfBM+qRA4ozgOymJc= 2022-09-12T20:16:07.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ObE0gEULsIWfBM+qRA4ozgOymJc=', 'Content-Length': '0'} 2022-09-12T20:16:07.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000295bd3dff1f78c78-00631f9387-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.177 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:16:07.178 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:16:07.178 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:16:07.179 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:16:07.179 DEBUG:boto:Method: GET 2022-09-12T20:16:07.180 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:16:07.180 DEBUG:boto:Data: 2022-09-12T20:16:07.181 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.182 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.182 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.183 DEBUG:boto:Params: {} 2022-09-12T20:16:07.183 DEBUG:boto:Token: None 2022-09-12T20:16:07.184 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:16:07.184 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ObE0gEULsIWfBM+qRA4ozgOymJc= 2022-09-12T20:16:07.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ObE0gEULsIWfBM+qRA4ozgOymJc=', 'Content-Length': '0'} 2022-09-12T20:16:07.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032d6cab09bcf9ff6-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.188 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:07.189 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:07.189 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:16:07.190 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:16:07.190 DEBUG:boto:Method: GET 2022-09-12T20:16:07.191 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:16:07.191 DEBUG:boto:Data: 2022-09-12T20:16:07.192 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.193 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.193 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.194 DEBUG:boto:Params: {} 2022-09-12T20:16:07.194 DEBUG:boto:Token: None 2022-09-12T20:16:07.195 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/ 2022-09-12T20:16:07.196 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds= 2022-09-12T20:16:07.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds=', 'Content-Length': '0'} 2022-09-12T20:16:07.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a19ff38191e6eba-00631f9387-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.200 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:07.202 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:07.203 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:07.203 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:16:07.204 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:16:07.204 DEBUG:boto:Method: GET 2022-09-12T20:16:07.205 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:16:07.205 DEBUG:boto:Data: 2022-09-12T20:16:07.206 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.206 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.207 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.207 DEBUG:boto:Params: {} 2022-09-12T20:16:07.208 DEBUG:boto:Token: None 2022-09-12T20:16:07.209 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/ 2022-09-12T20:16:07.209 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds= 2022-09-12T20:16:07.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds=', 'Content-Length': '0'} 2022-09-12T20:16:07.216 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4ed176aff8842e2-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.216 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:07.217 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:16:07.218 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:16:07.219 DEBUG:boto:Method: HEAD 2022-09-12T20:16:07.219 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:16:07.220 DEBUG:boto:Data: 2022-09-12T20:16:07.220 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.221 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.221 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.222 DEBUG:boto:Params: {} 2022-09-12T20:16:07.222 DEBUG:boto:Token: None 2022-09-12T20:16:07.223 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj11 2022-09-12T20:16:07.223 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EYYL1nA+RoZRSf9Q3BA395Hm5eM= 2022-09-12T20:16:07.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EYYL1nA+RoZRSf9Q3BA395Hm5eM=', 'Content-Length': '0'} 2022-09-12T20:16:07.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088783e557aaa1cbd-00631f9387-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.227 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:16:07.228 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:16:07.228 DEBUG:boto:Method: HEAD 2022-09-12T20:16:07.229 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:16:07.229 DEBUG:boto:Data: 2022-09-12T20:16:07.230 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.230 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.231 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.231 DEBUG:boto:Params: {} 2022-09-12T20:16:07.232 DEBUG:boto:Token: None 2022-09-12T20:16:07.233 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj11 2022-09-12T20:16:07.233 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EYYL1nA+RoZRSf9Q3BA395Hm5eM= 2022-09-12T20:16:07.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EYYL1nA+RoZRSf9Q3BA395Hm5eM=', 'Content-Length': '0'} 2022-09-12T20:16:07.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e5af1eed019aa0f-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:16:07.238 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:16:07.238 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:16:07.239 DEBUG:boto:Method: GET 2022-09-12T20:16:07.239 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:16:07.240 DEBUG:boto:Data: 2022-09-12T20:16:07.240 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.241 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.242 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.242 DEBUG:boto:Params: {} 2022-09-12T20:16:07.243 DEBUG:boto:Token: None 2022-09-12T20:16:07.243 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj11 2022-09-12T20:16:07.244 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jF1o8PlM04vMPpYquNJX9F8eToY= 2022-09-12T20:16:07.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jF1o8PlM04vMPpYquNJX9F8eToY=', 'Content-Length': '0'} 2022-09-12T20:16:07.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c943e7e3ed4e12d7-00631f9387-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.290 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:16:07.291 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:16:07.291 DEBUG:boto:Method: GET 2022-09-12T20:16:07.292 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:16:07.293 DEBUG:boto:Data: 2022-09-12T20:16:07.293 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.294 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.294 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.295 DEBUG:boto:Params: {} 2022-09-12T20:16:07.295 DEBUG:boto:Token: None 2022-09-12T20:16:07.296 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj11 2022-09-12T20:16:07.297 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jF1o8PlM04vMPpYquNJX9F8eToY= 2022-09-12T20:16:07.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jF1o8PlM04vMPpYquNJX9F8eToY=', 'Content-Length': '0'} 2022-09-12T20:16:07.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aaebaa30003a8b9-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.300 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:07.301 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:07.301 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:16:07.302 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:16:07.302 DEBUG:boto:Method: GET 2022-09-12T20:16:07.303 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:16:07.303 DEBUG:boto:Data: 2022-09-12T20:16:07.304 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.305 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.305 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.306 DEBUG:boto:Params: {} 2022-09-12T20:16:07.306 DEBUG:boto:Token: None 2022-09-12T20:16:07.307 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/ 2022-09-12T20:16:07.307 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds= 2022-09-12T20:16:07.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds=', 'Content-Length': '0'} 2022-09-12T20:16:07.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb2ac2047ba83691-00631f9387-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.312 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:07.313 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:07.313 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:07.314 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:16:07.314 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:16:07.315 DEBUG:boto:Method: GET 2022-09-12T20:16:07.315 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:16:07.316 DEBUG:boto:Data: 2022-09-12T20:16:07.316 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.317 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.318 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.318 DEBUG:boto:Params: {} 2022-09-12T20:16:07.319 DEBUG:boto:Token: None 2022-09-12T20:16:07.319 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/ 2022-09-12T20:16:07.320 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds= 2022-09-12T20:16:07.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds=', 'Content-Length': '0'} 2022-09-12T20:16:07.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003464052b538f7a1c-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.326 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:07.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2022-09-12T20:16:07.328 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:16:07.328 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:16:07.329 DEBUG:boto:path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:16:07.330 DEBUG:boto:auth_path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:16:07.330 DEBUG:boto:Method: GET 2022-09-12T20:16:07.331 DEBUG:boto:Path: /jdpiak-75/obj111?versionId=null 2022-09-12T20:16:07.331 DEBUG:boto:Data: 2022-09-12T20:16:07.332 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.332 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.333 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.333 DEBUG:boto:Params: {} 2022-09-12T20:16:07.334 DEBUG:boto:Token: None 2022-09-12T20:16:07.335 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj111?versionId=null 2022-09-12T20:16:07.335 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jmPm62NnKJSx4FY7j/CsPaTgnI0= 2022-09-12T20:16:07.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jmPm62NnKJSx4FY7j/CsPaTgnI0=', 'Content-Length': '0'} 2022-09-12T20:16:07.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f56d5576645a25a2-00631f9387-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.339 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:16:07.340 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:16:07.340 DEBUG:boto:path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:16:07.341 DEBUG:boto:auth_path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:16:07.341 DEBUG:boto:Method: GET 2022-09-12T20:16:07.342 DEBUG:boto:Path: /jdpiak-75/obj111?versionId=null 2022-09-12T20:16:07.343 DEBUG:boto:Data: 2022-09-12T20:16:07.343 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.344 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.344 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.345 DEBUG:boto:Params: {} 2022-09-12T20:16:07.345 DEBUG:boto:Token: None 2022-09-12T20:16:07.346 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj111?versionId=null 2022-09-12T20:16:07.347 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jmPm62NnKJSx4FY7j/CsPaTgnI0= 2022-09-12T20:16:07.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jmPm62NnKJSx4FY7j/CsPaTgnI0=', 'Content-Length': '0'} 2022-09-12T20:16:07.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc7b9353b8206766-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.390 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:16:07.391 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:16:07.391 DEBUG:boto:path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:16:07.392 DEBUG:boto:auth_path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:16:07.392 DEBUG:boto:Method: HEAD 2022-09-12T20:16:07.393 DEBUG:boto:Path: /jdpiak-75/obj111?versionId=null 2022-09-12T20:16:07.393 DEBUG:boto:Data: 2022-09-12T20:16:07.394 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.394 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.395 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.395 DEBUG:boto:Params: {} 2022-09-12T20:16:07.396 DEBUG:boto:Token: None 2022-09-12T20:16:07.397 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj111?versionId=null 2022-09-12T20:16:07.397 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KHpNn+WbfUFCI7ue79ZW7uJE5HY= 2022-09-12T20:16:07.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KHpNn+WbfUFCI7ue79ZW7uJE5HY=', 'Content-Length': '0'} 2022-09-12T20:16:07.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067025fe0fdc18bf1-00631f9387-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.401 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:16:07.401 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:16:07.402 DEBUG:boto:path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:16:07.402 DEBUG:boto:auth_path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:16:07.403 DEBUG:boto:Method: HEAD 2022-09-12T20:16:07.403 DEBUG:boto:Path: /jdpiak-75/obj111?versionId=null 2022-09-12T20:16:07.404 DEBUG:boto:Data: 2022-09-12T20:16:07.404 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.405 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.406 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.406 DEBUG:boto:Params: {} 2022-09-12T20:16:07.407 DEBUG:boto:Token: None 2022-09-12T20:16:07.407 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj111?versionId=null 2022-09-12T20:16:07.408 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KHpNn+WbfUFCI7ue79ZW7uJE5HY= 2022-09-12T20:16:07.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KHpNn+WbfUFCI7ue79ZW7uJE5HY=', 'Content-Length': '0'} 2022-09-12T20:16:07.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002052ef4c486db46b-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2022-09-12T20:16:07.412 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:16:07.413 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:16:07.413 DEBUG:boto:path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:16:07.414 DEBUG:boto:auth_path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:16:07.415 DEBUG:boto:Method: GET 2022-09-12T20:16:07.415 DEBUG:boto:Path: /jdpiak-75/obj111?versionId=null 2022-09-12T20:16:07.416 DEBUG:boto:Data: 2022-09-12T20:16:07.416 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.417 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.418 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.418 DEBUG:boto:Params: {} 2022-09-12T20:16:07.419 DEBUG:boto:Token: None 2022-09-12T20:16:07.419 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj111?versionId=null 2022-09-12T20:16:07.420 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jmPm62NnKJSx4FY7j/CsPaTgnI0= 2022-09-12T20:16:07.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jmPm62NnKJSx4FY7j/CsPaTgnI0=', 'Content-Length': '0'} 2022-09-12T20:16:07.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076be2834218bfd12-00631f9387-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.424 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:16:07.425 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:16:07.425 DEBUG:boto:path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:16:07.426 DEBUG:boto:auth_path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:16:07.426 DEBUG:boto:Method: GET 2022-09-12T20:16:07.427 DEBUG:boto:Path: /jdpiak-75/obj111?versionId=null 2022-09-12T20:16:07.427 DEBUG:boto:Data: 2022-09-12T20:16:07.428 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.428 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.429 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.429 DEBUG:boto:Params: {} 2022-09-12T20:16:07.430 DEBUG:boto:Token: None 2022-09-12T20:16:07.431 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj111?versionId=null 2022-09-12T20:16:07.431 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jmPm62NnKJSx4FY7j/CsPaTgnI0= 2022-09-12T20:16:07.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jmPm62NnKJSx4FY7j/CsPaTgnI0=', 'Content-Length': '0'} 2022-09-12T20:16:07.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5d0b07698a3118f-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.435 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:07.436 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:07.436 DEBUG:boto:path=/jdpiak-75/?prefix=obj111 2022-09-12T20:16:07.437 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj111 2022-09-12T20:16:07.437 DEBUG:boto:Method: GET 2022-09-12T20:16:07.438 DEBUG:boto:Path: /jdpiak-75/?prefix=obj111 2022-09-12T20:16:07.439 DEBUG:boto:Data: 2022-09-12T20:16:07.439 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.440 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.440 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.441 DEBUG:boto:Params: {} 2022-09-12T20:16:07.441 DEBUG:boto:Token: None 2022-09-12T20:16:07.442 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/ 2022-09-12T20:16:07.443 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds= 2022-09-12T20:16:07.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds=', 'Content-Length': '0'} 2022-09-12T20:16:07.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086e5e93fa4d340c4-00631f9387-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.494 DEBUG:boto:b'jdpiak-75obj1111000falseobj1112022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:07.495 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:07.496 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:07.496 DEBUG:boto:path=/jdpiak-75/?prefix=obj111 2022-09-12T20:16:07.497 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj111 2022-09-12T20:16:07.497 DEBUG:boto:Method: GET 2022-09-12T20:16:07.498 DEBUG:boto:Path: /jdpiak-75/?prefix=obj111 2022-09-12T20:16:07.498 DEBUG:boto:Data: 2022-09-12T20:16:07.499 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.500 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.500 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.501 DEBUG:boto:Params: {} 2022-09-12T20:16:07.501 DEBUG:boto:Token: None 2022-09-12T20:16:07.502 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/ 2022-09-12T20:16:07.503 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds= 2022-09-12T20:16:07.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds=', 'Content-Length': '0'} 2022-09-12T20:16:07.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfa5ca4abc73ac73-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.509 DEBUG:boto:b'jdpiak-75obj1111000falseobj1112022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:07.510 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:16:07.511 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:16:07.512 DEBUG:boto:Method: HEAD 2022-09-12T20:16:07.513 DEBUG:boto:Path: /jdpiak-75/obj111 2022-09-12T20:16:07.513 DEBUG:boto:Data: 2022-09-12T20:16:07.514 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.515 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.515 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.516 DEBUG:boto:Params: {} 2022-09-12T20:16:07.517 DEBUG:boto:Token: None 2022-09-12T20:16:07.517 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj111 2022-09-12T20:16:07.518 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PVP7vvNDergJTT+Cu1Lbujhd/js= 2022-09-12T20:16:07.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PVP7vvNDergJTT+Cu1Lbujhd/js=', 'Content-Length': '0'} 2022-09-12T20:16:07.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dddd564ee528d3e6-00631f9387-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.522 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:16:07.522 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:16:07.523 DEBUG:boto:Method: HEAD 2022-09-12T20:16:07.523 DEBUG:boto:Path: /jdpiak-75/obj111 2022-09-12T20:16:07.524 DEBUG:boto:Data: 2022-09-12T20:16:07.524 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.525 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.526 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.526 DEBUG:boto:Params: {} 2022-09-12T20:16:07.527 DEBUG:boto:Token: None 2022-09-12T20:16:07.527 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj111 2022-09-12T20:16:07.528 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PVP7vvNDergJTT+Cu1Lbujhd/js= 2022-09-12T20:16:07.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PVP7vvNDergJTT+Cu1Lbujhd/js=', 'Content-Length': '0'} 2022-09-12T20:16:07.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5b3f9561e9fdc6d-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2022-09-12T20:16:07.532 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:16:07.532 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:16:07.533 DEBUG:boto:Method: GET 2022-09-12T20:16:07.534 DEBUG:boto:Path: /jdpiak-75/obj111 2022-09-12T20:16:07.534 DEBUG:boto:Data: 2022-09-12T20:16:07.535 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.535 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.536 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.536 DEBUG:boto:Params: {} 2022-09-12T20:16:07.537 DEBUG:boto:Token: None 2022-09-12T20:16:07.537 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj111 2022-09-12T20:16:07.538 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LzrEF2zWVlcmQmEMc17QycScGq8= 2022-09-12T20:16:07.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LzrEF2zWVlcmQmEMc17QycScGq8=', 'Content-Length': '0'} 2022-09-12T20:16:07.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc2a3f6042a825b7-00631f9387-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.542 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:16:07.542 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:16:07.543 DEBUG:boto:Method: GET 2022-09-12T20:16:07.544 DEBUG:boto:Path: /jdpiak-75/obj111 2022-09-12T20:16:07.544 DEBUG:boto:Data: 2022-09-12T20:16:07.545 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.545 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.546 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.546 DEBUG:boto:Params: {} 2022-09-12T20:16:07.547 DEBUG:boto:Token: None 2022-09-12T20:16:07.548 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj111 2022-09-12T20:16:07.548 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LzrEF2zWVlcmQmEMc17QycScGq8= 2022-09-12T20:16:07.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LzrEF2zWVlcmQmEMc17QycScGq8=', 'Content-Length': '0'} 2022-09-12T20:16:07.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022399759dd1932c5-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.594 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:07.595 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:07.595 DEBUG:boto:path=/jdpiak-75/?prefix=obj111 2022-09-12T20:16:07.596 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj111 2022-09-12T20:16:07.596 DEBUG:boto:Method: GET 2022-09-12T20:16:07.597 DEBUG:boto:Path: /jdpiak-75/?prefix=obj111 2022-09-12T20:16:07.597 DEBUG:boto:Data: 2022-09-12T20:16:07.598 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.599 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.599 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.600 DEBUG:boto:Params: {} 2022-09-12T20:16:07.600 DEBUG:boto:Token: None 2022-09-12T20:16:07.601 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/ 2022-09-12T20:16:07.602 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds= 2022-09-12T20:16:07.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds=', 'Content-Length': '0'} 2022-09-12T20:16:07.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062b9c86f9b46ac3d-00631f9387-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.606 DEBUG:boto:b'jdpiak-75obj1111000falseobj1112022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:07.607 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:07.607 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:07.608 DEBUG:boto:path=/jdpiak-75/?prefix=obj111 2022-09-12T20:16:07.608 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj111 2022-09-12T20:16:07.609 DEBUG:boto:Method: GET 2022-09-12T20:16:07.609 DEBUG:boto:Path: /jdpiak-75/?prefix=obj111 2022-09-12T20:16:07.610 DEBUG:boto:Data: 2022-09-12T20:16:07.610 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.611 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.611 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.612 DEBUG:boto:Params: {} 2022-09-12T20:16:07.613 DEBUG:boto:Token: None 2022-09-12T20:16:07.613 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/ 2022-09-12T20:16:07.614 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds= 2022-09-12T20:16:07.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds=', 'Content-Length': '0'} 2022-09-12T20:16:07.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d39ea0c55822f325-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.620 DEBUG:boto:b'jdpiak-75obj1111000falseobj1112022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:07.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:16:07.621 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:16:07.622 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:16:07.622 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:16:07.623 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:16:07.623 DEBUG:boto:Method: GET 2022-09-12T20:16:07.624 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:16:07.624 DEBUG:boto:Data: 2022-09-12T20:16:07.625 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.625 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.626 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.626 DEBUG:boto:Params: {} 2022-09-12T20:16:07.627 DEBUG:boto:Token: None 2022-09-12T20:16:07.628 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:16:07.628 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mYGSu4LnV7Qcy+80mE6oHSmmKeo= 2022-09-12T20:16:07.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mYGSu4LnV7Qcy+80mE6oHSmmKeo=', 'Content-Length': '0'} 2022-09-12T20:16:07.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2ceeb299228fb18-00631f9387-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.632 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:16:07.632 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:16:07.633 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:16:07.633 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:16:07.634 DEBUG:boto:Method: GET 2022-09-12T20:16:07.634 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:16:07.635 DEBUG:boto:Data: 2022-09-12T20:16:07.635 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.636 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.636 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.637 DEBUG:boto:Params: {} 2022-09-12T20:16:07.637 DEBUG:boto:Token: None 2022-09-12T20:16:07.638 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:16:07.639 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mYGSu4LnV7Qcy+80mE6oHSmmKeo= 2022-09-12T20:16:07.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mYGSu4LnV7Qcy+80mE6oHSmmKeo=', 'Content-Length': '0'} 2022-09-12T20:16:07.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ee16ebb48d45606-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.642 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:16:07.643 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:16:07.643 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:16:07.644 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:16:07.644 DEBUG:boto:Method: HEAD 2022-09-12T20:16:07.645 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:16:07.645 DEBUG:boto:Data: 2022-09-12T20:16:07.646 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.646 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.647 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.647 DEBUG:boto:Params: {} 2022-09-12T20:16:07.648 DEBUG:boto:Token: None 2022-09-12T20:16:07.648 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:16:07.649 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/URSksZrCKwnOeZUX1zUGhoMmK4= 2022-09-12T20:16:07.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/URSksZrCKwnOeZUX1zUGhoMmK4=', 'Content-Length': '0'} 2022-09-12T20:16:07.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3fb9e9708e68693-00631f9387-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.652 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:16:07.653 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:16:07.653 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:16:07.654 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:16:07.654 DEBUG:boto:Method: HEAD 2022-09-12T20:16:07.655 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:16:07.655 DEBUG:boto:Data: 2022-09-12T20:16:07.656 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.656 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.657 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.657 DEBUG:boto:Params: {} 2022-09-12T20:16:07.658 DEBUG:boto:Token: None 2022-09-12T20:16:07.658 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:16:07.659 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/URSksZrCKwnOeZUX1zUGhoMmK4= 2022-09-12T20:16:07.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/URSksZrCKwnOeZUX1zUGhoMmK4=', 'Content-Length': '0'} 2022-09-12T20:16:07.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f49e264ea3550b3d-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:16:07.663 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:16:07.663 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:16:07.664 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:16:07.664 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:16:07.665 DEBUG:boto:Method: GET 2022-09-12T20:16:07.665 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:16:07.666 DEBUG:boto:Data: 2022-09-12T20:16:07.666 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.667 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.667 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.668 DEBUG:boto:Params: {} 2022-09-12T20:16:07.669 DEBUG:boto:Token: None 2022-09-12T20:16:07.669 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:16:07.670 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mYGSu4LnV7Qcy+80mE6oHSmmKeo= 2022-09-12T20:16:07.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mYGSu4LnV7Qcy+80mE6oHSmmKeo=', 'Content-Length': '0'} 2022-09-12T20:16:07.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000463a4d0a2247b185-00631f9387-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.673 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:16:07.674 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:16:07.674 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:16:07.675 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:16:07.675 DEBUG:boto:Method: GET 2022-09-12T20:16:07.676 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:16:07.677 DEBUG:boto:Data: 2022-09-12T20:16:07.677 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.678 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.678 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.679 DEBUG:boto:Params: {} 2022-09-12T20:16:07.679 DEBUG:boto:Token: None 2022-09-12T20:16:07.680 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:16:07.681 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mYGSu4LnV7Qcy+80mE6oHSmmKeo= 2022-09-12T20:16:07.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mYGSu4LnV7Qcy+80mE6oHSmmKeo=', 'Content-Length': '0'} 2022-09-12T20:16:07.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000012b4477fa8b33e3-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.685 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:07.685 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:07.686 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:16:07.686 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:16:07.687 DEBUG:boto:Method: GET 2022-09-12T20:16:07.687 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:16:07.688 DEBUG:boto:Data: 2022-09-12T20:16:07.688 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.689 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.689 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.690 DEBUG:boto:Params: {} 2022-09-12T20:16:07.691 DEBUG:boto:Token: None 2022-09-12T20:16:07.691 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/ 2022-09-12T20:16:07.692 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds= 2022-09-12T20:16:07.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds=', 'Content-Length': '0'} 2022-09-12T20:16:07.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2971dfad4099477-00631f9387-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.696 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:07.697 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:07.698 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:07.698 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:16:07.699 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:16:07.699 DEBUG:boto:Method: GET 2022-09-12T20:16:07.700 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:16:07.700 DEBUG:boto:Data: 2022-09-12T20:16:07.701 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.701 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.702 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.702 DEBUG:boto:Params: {} 2022-09-12T20:16:07.703 DEBUG:boto:Token: None 2022-09-12T20:16:07.704 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/ 2022-09-12T20:16:07.704 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds= 2022-09-12T20:16:07.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds=', 'Content-Length': '0'} 2022-09-12T20:16:07.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f76451a7d8de143-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.710 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:07.711 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:16:07.712 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:16:07.712 DEBUG:boto:Method: HEAD 2022-09-12T20:16:07.713 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:16:07.713 DEBUG:boto:Data: 2022-09-12T20:16:07.714 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.714 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.715 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.715 DEBUG:boto:Params: {} 2022-09-12T20:16:07.716 DEBUG:boto:Token: None 2022-09-12T20:16:07.716 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj115 2022-09-12T20:16:07.717 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5pCi501QMN8Zqe3GeeIis7wdUcg= 2022-09-12T20:16:07.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5pCi501QMN8Zqe3GeeIis7wdUcg=', 'Content-Length': '0'} 2022-09-12T20:16:07.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094633ccfacd0263a-00631f9387-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.720 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:16:07.721 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:16:07.722 DEBUG:boto:Method: HEAD 2022-09-12T20:16:07.722 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:16:07.723 DEBUG:boto:Data: 2022-09-12T20:16:07.723 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.724 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.724 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.725 DEBUG:boto:Params: {} 2022-09-12T20:16:07.725 DEBUG:boto:Token: None 2022-09-12T20:16:07.726 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj115 2022-09-12T20:16:07.726 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5pCi501QMN8Zqe3GeeIis7wdUcg= 2022-09-12T20:16:07.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5pCi501QMN8Zqe3GeeIis7wdUcg=', 'Content-Length': '0'} 2022-09-12T20:16:07.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000065ef07fec834d2f-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:16:07.730 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:16:07.731 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:16:07.732 DEBUG:boto:Method: GET 2022-09-12T20:16:07.732 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:16:07.733 DEBUG:boto:Data: 2022-09-12T20:16:07.733 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.734 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.735 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.735 DEBUG:boto:Params: {} 2022-09-12T20:16:07.737 DEBUG:boto:Token: None 2022-09-12T20:16:07.737 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj115 2022-09-12T20:16:07.738 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XAkUruSOrJR827svreWgQ6jEdBM= 2022-09-12T20:16:07.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XAkUruSOrJR827svreWgQ6jEdBM=', 'Content-Length': '0'} 2022-09-12T20:16:07.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f9980a46f35282d-00631f9387-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.782 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:16:07.783 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:16:07.784 DEBUG:boto:Method: GET 2022-09-12T20:16:07.784 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:16:07.785 DEBUG:boto:Data: 2022-09-12T20:16:07.786 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.786 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.787 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.787 DEBUG:boto:Params: {} 2022-09-12T20:16:07.788 DEBUG:boto:Token: None 2022-09-12T20:16:07.789 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj115 2022-09-12T20:16:07.789 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XAkUruSOrJR827svreWgQ6jEdBM= 2022-09-12T20:16:07.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XAkUruSOrJR827svreWgQ6jEdBM=', 'Content-Length': '0'} 2022-09-12T20:16:07.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddc4c5d84c226e0c-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.793 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:07.793 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:07.794 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:16:07.795 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:16:07.795 DEBUG:boto:Method: GET 2022-09-12T20:16:07.796 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:16:07.796 DEBUG:boto:Data: 2022-09-12T20:16:07.797 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.797 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.798 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.799 DEBUG:boto:Params: {} 2022-09-12T20:16:07.799 DEBUG:boto:Token: None 2022-09-12T20:16:07.800 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/ 2022-09-12T20:16:07.800 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds= 2022-09-12T20:16:07.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds=', 'Content-Length': '0'} 2022-09-12T20:16:07.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5e22b974fb1e80c-00631f9387-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.805 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:07.805 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:07.806 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:07.807 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:16:07.807 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:16:07.808 DEBUG:boto:Method: GET 2022-09-12T20:16:07.808 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:16:07.809 DEBUG:boto:Data: 2022-09-12T20:16:07.809 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.810 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.810 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.811 DEBUG:boto:Params: {} 2022-09-12T20:16:07.812 DEBUG:boto:Token: None 2022-09-12T20:16:07.812 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/ 2022-09-12T20:16:07.813 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds= 2022-09-12T20:16:07.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds=', 'Content-Length': '0'} 2022-09-12T20:16:07.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d604fb94ea316416-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.819 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:07.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:16:07.820 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:16:07.821 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:16:07.821 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:16:07.822 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:16:07.823 DEBUG:boto:Method: GET 2022-09-12T20:16:07.823 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:16:07.824 DEBUG:boto:Data: 2022-09-12T20:16:07.824 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.825 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.826 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.826 DEBUG:boto:Params: {} 2022-09-12T20:16:07.827 DEBUG:boto:Token: None 2022-09-12T20:16:07.827 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:16:07.828 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S+89TOQbMvdG0DqlZEccD0GoqCY= 2022-09-12T20:16:07.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S+89TOQbMvdG0DqlZEccD0GoqCY=', 'Content-Length': '0'} 2022-09-12T20:16:07.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000942b8a3a1c54064a-00631f9387-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.832 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:16:07.832 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:16:07.833 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:16:07.833 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:16:07.834 DEBUG:boto:Method: GET 2022-09-12T20:16:07.835 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:16:07.835 DEBUG:boto:Data: 2022-09-12T20:16:07.836 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.836 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.837 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.837 DEBUG:boto:Params: {} 2022-09-12T20:16:07.838 DEBUG:boto:Token: None 2022-09-12T20:16:07.839 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:16:07.839 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S+89TOQbMvdG0DqlZEccD0GoqCY= 2022-09-12T20:16:07.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S+89TOQbMvdG0DqlZEccD0GoqCY=', 'Content-Length': '0'} 2022-09-12T20:16:07.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b3d2958abd6dc51-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.844 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:16:07.844 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:16:07.845 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:16:07.845 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:16:07.846 DEBUG:boto:Method: HEAD 2022-09-12T20:16:07.846 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:16:07.847 DEBUG:boto:Data: 2022-09-12T20:16:07.847 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.848 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.848 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.849 DEBUG:boto:Params: {} 2022-09-12T20:16:07.849 DEBUG:boto:Token: None 2022-09-12T20:16:07.850 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:16:07.851 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:a9Yp/aS60DrdPnTkoWhg8phu4As= 2022-09-12T20:16:07.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:a9Yp/aS60DrdPnTkoWhg8phu4As=', 'Content-Length': '0'} 2022-09-12T20:16:07.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010a069d3e48665fc-00631f9387-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.854 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:16:07.855 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:16:07.855 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:16:07.856 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:16:07.856 DEBUG:boto:Method: HEAD 2022-09-12T20:16:07.857 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:16:07.857 DEBUG:boto:Data: 2022-09-12T20:16:07.858 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.858 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.859 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.859 DEBUG:boto:Params: {} 2022-09-12T20:16:07.860 DEBUG:boto:Token: None 2022-09-12T20:16:07.861 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:16:07.861 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:a9Yp/aS60DrdPnTkoWhg8phu4As= 2022-09-12T20:16:07.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:a9Yp/aS60DrdPnTkoWhg8phu4As=', 'Content-Length': '0'} 2022-09-12T20:16:07.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ac1b986fdf16bc5-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:16:07.865 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:16:07.866 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:16:07.866 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:16:07.867 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:16:07.867 DEBUG:boto:Method: GET 2022-09-12T20:16:07.868 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:16:07.868 DEBUG:boto:Data: 2022-09-12T20:16:07.869 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.870 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.870 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.871 DEBUG:boto:Params: {} 2022-09-12T20:16:07.871 DEBUG:boto:Token: None 2022-09-12T20:16:07.872 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:16:07.873 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S+89TOQbMvdG0DqlZEccD0GoqCY= 2022-09-12T20:16:07.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S+89TOQbMvdG0DqlZEccD0GoqCY=', 'Content-Length': '0'} 2022-09-12T20:16:07.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018ffb01e11089df6-00631f9387-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.876 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:16:07.877 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:16:07.877 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:16:07.878 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:16:07.879 DEBUG:boto:Method: GET 2022-09-12T20:16:07.879 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:16:07.880 DEBUG:boto:Data: 2022-09-12T20:16:07.880 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.881 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.881 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.882 DEBUG:boto:Params: {} 2022-09-12T20:16:07.882 DEBUG:boto:Token: None 2022-09-12T20:16:07.883 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:16:07.884 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:S+89TOQbMvdG0DqlZEccD0GoqCY= 2022-09-12T20:16:07.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:S+89TOQbMvdG0DqlZEccD0GoqCY=', 'Content-Length': '0'} 2022-09-12T20:16:07.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006570a190f53d9735-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.888 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:07.889 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:07.889 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:16:07.890 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:16:07.890 DEBUG:boto:Method: GET 2022-09-12T20:16:07.891 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:16:07.891 DEBUG:boto:Data: 2022-09-12T20:16:07.892 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.892 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.893 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.894 DEBUG:boto:Params: {} 2022-09-12T20:16:07.894 DEBUG:boto:Token: None 2022-09-12T20:16:07.895 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/ 2022-09-12T20:16:07.895 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds= 2022-09-12T20:16:07.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds=', 'Content-Length': '0'} 2022-09-12T20:16:07.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000012f826ba77b05a-00631f9387-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.899 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:07.900 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:07.901 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:07.902 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:16:07.902 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:16:07.903 DEBUG:boto:Method: GET 2022-09-12T20:16:07.903 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:16:07.904 DEBUG:boto:Data: 2022-09-12T20:16:07.904 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.905 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.905 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.906 DEBUG:boto:Params: {} 2022-09-12T20:16:07.906 DEBUG:boto:Token: None 2022-09-12T20:16:07.907 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/ 2022-09-12T20:16:07.908 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds= 2022-09-12T20:16:07.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds=', 'Content-Length': '0'} 2022-09-12T20:16:07.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb39bcf3aa654e5c-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.954 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:07.955 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:16:07.956 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:16:07.956 DEBUG:boto:Method: HEAD 2022-09-12T20:16:07.957 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:16:07.957 DEBUG:boto:Data: 2022-09-12T20:16:07.958 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.958 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.959 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.959 DEBUG:boto:Params: {} 2022-09-12T20:16:07.960 DEBUG:boto:Token: None 2022-09-12T20:16:07.960 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj116 2022-09-12T20:16:07.961 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6DeiWuyZNg+ya8og3K0wKLqaWKA= 2022-09-12T20:16:07.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6DeiWuyZNg+ya8og3K0wKLqaWKA=', 'Content-Length': '0'} 2022-09-12T20:16:07.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb85b0e11e2c59d1-00631f9387-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.964 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:16:07.965 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:16:07.965 DEBUG:boto:Method: HEAD 2022-09-12T20:16:07.966 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:16:07.966 DEBUG:boto:Data: 2022-09-12T20:16:07.967 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.967 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.968 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.968 DEBUG:boto:Params: {} 2022-09-12T20:16:07.969 DEBUG:boto:Token: None 2022-09-12T20:16:07.970 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj116 2022-09-12T20:16:07.970 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6DeiWuyZNg+ya8og3K0wKLqaWKA= 2022-09-12T20:16:07.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6DeiWuyZNg+ya8og3K0wKLqaWKA=', 'Content-Length': '0'} 2022-09-12T20:16:07.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066d8d9903205455c-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:16:07.974 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:16:07.975 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:16:07.975 DEBUG:boto:Method: GET 2022-09-12T20:16:07.976 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:16:07.976 DEBUG:boto:Data: 2022-09-12T20:16:07.977 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.977 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.978 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.978 DEBUG:boto:Params: {} 2022-09-12T20:16:07.979 DEBUG:boto:Token: None 2022-09-12T20:16:07.979 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj116 2022-09-12T20:16:07.980 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:js+wcCnZiK2rcBb+ODu0C9rjXIM= 2022-09-12T20:16:07.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:js+wcCnZiK2rcBb+ODu0C9rjXIM=', 'Content-Length': '0'} 2022-09-12T20:16:07.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030c31eb8f12041ca-00631f9387-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.984 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:16:07.984 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:16:07.985 DEBUG:boto:Method: GET 2022-09-12T20:16:07.985 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:16:07.986 DEBUG:boto:Data: 2022-09-12T20:16:07.986 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.987 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:07.987 DEBUG:boto:Port: 8000 2022-09-12T20:16:07.988 DEBUG:boto:Params: {} 2022-09-12T20:16:07.988 DEBUG:boto:Token: None 2022-09-12T20:16:07.989 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/obj116 2022-09-12T20:16:07.989 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:js+wcCnZiK2rcBb+ODu0C9rjXIM= 2022-09-12T20:16:07.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:js+wcCnZiK2rcBb+ODu0C9rjXIM=', 'Content-Length': '0'} 2022-09-12T20:16:07.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045aa7a795ff4a007-00631f9387-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:07.993 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:07.994 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:07.994 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:16:07.995 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:16:07.995 DEBUG:boto:Method: GET 2022-09-12T20:16:07.996 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:16:07.996 DEBUG:boto:Data: 2022-09-12T20:16:07.997 DEBUG:boto:Headers: {} 2022-09-12T20:16:07.997 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:07.998 DEBUG:boto:Port: 8001 2022-09-12T20:16:07.998 DEBUG:boto:Params: {} 2022-09-12T20:16:07.999 DEBUG:boto:Token: None 2022-09-12T20:16:07.999 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:07 GMT /jdpiak-75/ 2022-09-12T20:16:08.000 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds= 2022-09-12T20:16:08.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:07 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:58RVN2msezd2Dw0dF9PJcyCpjds=', 'Content-Length': '0'} 2022-09-12T20:16:08.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046db52d636cac322-00631f9388-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.046 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:08.047 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:08.048 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:08.048 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:16:08.049 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:16:08.049 DEBUG:boto:Method: GET 2022-09-12T20:16:08.050 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:16:08.050 DEBUG:boto:Data: 2022-09-12T20:16:08.051 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.051 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.052 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.052 DEBUG:boto:Params: {} 2022-09-12T20:16:08.053 DEBUG:boto:Token: None 2022-09-12T20:16:08.054 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/ 2022-09-12T20:16:08.054 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws= 2022-09-12T20:16:08.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws=', 'Content-Length': '0'} 2022-09-12T20:16:08.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000faedc5f8350e3d7-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.060 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:08.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-12T20:16:08.061 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:16:08.062 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:16:08.063 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:16:08.063 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:16:08.064 DEBUG:boto:Method: GET 2022-09-12T20:16:08.064 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:16:08.065 DEBUG:boto:Data: 2022-09-12T20:16:08.065 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.066 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.066 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.067 DEBUG:boto:Params: {} 2022-09-12T20:16:08.068 DEBUG:boto:Token: None 2022-09-12T20:16:08.068 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:16:08.069 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MEMynicV07kXLZVGf75iRgm/Qi0= 2022-09-12T20:16:08.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MEMynicV07kXLZVGf75iRgm/Qi0=', 'Content-Length': '0'} 2022-09-12T20:16:08.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054b276c2ce5d7ad7-00631f9388-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.072 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:16:08.073 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:16:08.074 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:16:08.074 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:16:08.075 DEBUG:boto:Method: GET 2022-09-12T20:16:08.075 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:16:08.076 DEBUG:boto:Data: 2022-09-12T20:16:08.076 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.077 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.077 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.078 DEBUG:boto:Params: {} 2022-09-12T20:16:08.079 DEBUG:boto:Token: None 2022-09-12T20:16:08.079 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:16:08.080 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MEMynicV07kXLZVGf75iRgm/Qi0= 2022-09-12T20:16:08.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MEMynicV07kXLZVGf75iRgm/Qi0=', 'Content-Length': '0'} 2022-09-12T20:16:08.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035a28b3d2e3d7297-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.084 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:16:08.084 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:16:08.085 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:16:08.085 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:16:08.086 DEBUG:boto:Method: HEAD 2022-09-12T20:16:08.086 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:16:08.087 DEBUG:boto:Data: 2022-09-12T20:16:08.087 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.088 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.088 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.089 DEBUG:boto:Params: {} 2022-09-12T20:16:08.090 DEBUG:boto:Token: None 2022-09-12T20:16:08.090 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:16:08.091 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bSu5EfE3z5eLW+zESXRjg4Amd9c= 2022-09-12T20:16:08.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bSu5EfE3z5eLW+zESXRjg4Amd9c=', 'Content-Length': '0'} 2022-09-12T20:16:08.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000115822d99d4fb2cf-00631f9388-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.094 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:16:08.095 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:16:08.095 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:16:08.096 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:16:08.096 DEBUG:boto:Method: HEAD 2022-09-12T20:16:08.097 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:16:08.097 DEBUG:boto:Data: 2022-09-12T20:16:08.098 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.098 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.099 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.099 DEBUG:boto:Params: {} 2022-09-12T20:16:08.100 DEBUG:boto:Token: None 2022-09-12T20:16:08.101 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:16:08.101 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bSu5EfE3z5eLW+zESXRjg4Amd9c= 2022-09-12T20:16:08.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bSu5EfE3z5eLW+zESXRjg4Amd9c=', 'Content-Length': '0'} 2022-09-12T20:16:08.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050ea10842a1dabbf-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-12T20:16:08.105 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:16:08.106 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:16:08.106 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:16:08.107 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:16:08.107 DEBUG:boto:Method: GET 2022-09-12T20:16:08.108 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:16:08.108 DEBUG:boto:Data: 2022-09-12T20:16:08.109 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.109 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.110 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.110 DEBUG:boto:Params: {} 2022-09-12T20:16:08.111 DEBUG:boto:Token: None 2022-09-12T20:16:08.111 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:16:08.112 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MEMynicV07kXLZVGf75iRgm/Qi0= 2022-09-12T20:16:08.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MEMynicV07kXLZVGf75iRgm/Qi0=', 'Content-Length': '0'} 2022-09-12T20:16:08.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9553a0daf5bc3ef-00631f9388-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.116 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:16:08.116 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:16:08.117 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:16:08.117 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:16:08.118 DEBUG:boto:Method: GET 2022-09-12T20:16:08.118 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:16:08.119 DEBUG:boto:Data: 2022-09-12T20:16:08.119 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.120 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.120 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.121 DEBUG:boto:Params: {} 2022-09-12T20:16:08.122 DEBUG:boto:Token: None 2022-09-12T20:16:08.122 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:16:08.123 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MEMynicV07kXLZVGf75iRgm/Qi0= 2022-09-12T20:16:08.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MEMynicV07kXLZVGf75iRgm/Qi0=', 'Content-Length': '0'} 2022-09-12T20:16:08.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008656d36143461260-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.166 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:08.167 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:08.169 DEBUG:boto:path=/jdpiak-75/?prefix=obj124 2022-09-12T20:16:08.170 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj124 2022-09-12T20:16:08.170 DEBUG:boto:Method: GET 2022-09-12T20:16:08.171 DEBUG:boto:Path: /jdpiak-75/?prefix=obj124 2022-09-12T20:16:08.172 DEBUG:boto:Data: 2022-09-12T20:16:08.173 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.174 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.174 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.175 DEBUG:boto:Params: {} 2022-09-12T20:16:08.175 DEBUG:boto:Token: None 2022-09-12T20:16:08.176 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/ 2022-09-12T20:16:08.177 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws= 2022-09-12T20:16:08.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws=', 'Content-Length': '0'} 2022-09-12T20:16:08.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a63acd31a4025c81-00631f9388-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.182 DEBUG:boto:b'jdpiak-75obj1241000falseobj1242022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:08.183 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:08.183 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:08.184 DEBUG:boto:path=/jdpiak-75/?prefix=obj124 2022-09-12T20:16:08.185 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj124 2022-09-12T20:16:08.185 DEBUG:boto:Method: GET 2022-09-12T20:16:08.186 DEBUG:boto:Path: /jdpiak-75/?prefix=obj124 2022-09-12T20:16:08.186 DEBUG:boto:Data: 2022-09-12T20:16:08.187 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.188 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.188 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.189 DEBUG:boto:Params: {} 2022-09-12T20:16:08.189 DEBUG:boto:Token: None 2022-09-12T20:16:08.190 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/ 2022-09-12T20:16:08.191 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws= 2022-09-12T20:16:08.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws=', 'Content-Length': '0'} 2022-09-12T20:16:08.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000431c63e1ed528f1d-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.198 DEBUG:boto:b'jdpiak-75obj1241000falseobj1242022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:08.199 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:16:08.200 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:16:08.201 DEBUG:boto:Method: HEAD 2022-09-12T20:16:08.202 DEBUG:boto:Path: /jdpiak-75/obj124 2022-09-12T20:16:08.202 DEBUG:boto:Data: 2022-09-12T20:16:08.203 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.204 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.204 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.205 DEBUG:boto:Params: {} 2022-09-12T20:16:08.205 DEBUG:boto:Token: None 2022-09-12T20:16:08.206 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj124 2022-09-12T20:16:08.207 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wOjXsRLRX1JMTo3CK+1+Zzd/0M4= 2022-09-12T20:16:08.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wOjXsRLRX1JMTo3CK+1+Zzd/0M4=', 'Content-Length': '0'} 2022-09-12T20:16:08.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa29508ed42cf06b-00631f9388-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.212 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:16:08.213 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:16:08.214 DEBUG:boto:Method: HEAD 2022-09-12T20:16:08.215 DEBUG:boto:Path: /jdpiak-75/obj124 2022-09-12T20:16:08.215 DEBUG:boto:Data: 2022-09-12T20:16:08.216 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.216 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.217 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.217 DEBUG:boto:Params: {} 2022-09-12T20:16:08.218 DEBUG:boto:Token: None 2022-09-12T20:16:08.218 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj124 2022-09-12T20:16:08.219 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wOjXsRLRX1JMTo3CK+1+Zzd/0M4= 2022-09-12T20:16:08.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wOjXsRLRX1JMTo3CK+1+Zzd/0M4=', 'Content-Length': '0'} 2022-09-12T20:16:08.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9a8861807586a46-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-12T20:16:08.223 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:16:08.224 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:16:08.225 DEBUG:boto:Method: GET 2022-09-12T20:16:08.225 DEBUG:boto:Path: /jdpiak-75/obj124 2022-09-12T20:16:08.226 DEBUG:boto:Data: 2022-09-12T20:16:08.226 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.227 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.228 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.228 DEBUG:boto:Params: {} 2022-09-12T20:16:08.229 DEBUG:boto:Token: None 2022-09-12T20:16:08.229 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj124 2022-09-12T20:16:08.230 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hpSouaNQI/OORjTjJnVWQfhfKio= 2022-09-12T20:16:08.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hpSouaNQI/OORjTjJnVWQfhfKio=', 'Content-Length': '0'} 2022-09-12T20:16:08.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4946663234d0808-00631f9388-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.233 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:16:08.234 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:16:08.235 DEBUG:boto:Method: GET 2022-09-12T20:16:08.235 DEBUG:boto:Path: /jdpiak-75/obj124 2022-09-12T20:16:08.237 DEBUG:boto:Data: 2022-09-12T20:16:08.238 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.239 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.240 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.241 DEBUG:boto:Params: {} 2022-09-12T20:16:08.242 DEBUG:boto:Token: None 2022-09-12T20:16:08.243 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj124 2022-09-12T20:16:08.244 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hpSouaNQI/OORjTjJnVWQfhfKio= 2022-09-12T20:16:08.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hpSouaNQI/OORjTjJnVWQfhfKio=', 'Content-Length': '0'} 2022-09-12T20:16:08.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bc0dc89769adcd5-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.249 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:08.250 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:08.250 DEBUG:boto:path=/jdpiak-75/?prefix=obj124 2022-09-12T20:16:08.251 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj124 2022-09-12T20:16:08.252 DEBUG:boto:Method: GET 2022-09-12T20:16:08.252 DEBUG:boto:Path: /jdpiak-75/?prefix=obj124 2022-09-12T20:16:08.253 DEBUG:boto:Data: 2022-09-12T20:16:08.253 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.254 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.255 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.255 DEBUG:boto:Params: {} 2022-09-12T20:16:08.256 DEBUG:boto:Token: None 2022-09-12T20:16:08.256 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/ 2022-09-12T20:16:08.257 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws= 2022-09-12T20:16:08.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws=', 'Content-Length': '0'} 2022-09-12T20:16:08.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000622a4720029f9c2f-00631f9388-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.261 DEBUG:boto:b'jdpiak-75obj1241000falseobj1242022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:08.262 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:08.263 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:08.263 DEBUG:boto:path=/jdpiak-75/?prefix=obj124 2022-09-12T20:16:08.264 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj124 2022-09-12T20:16:08.264 DEBUG:boto:Method: GET 2022-09-12T20:16:08.265 DEBUG:boto:Path: /jdpiak-75/?prefix=obj124 2022-09-12T20:16:08.266 DEBUG:boto:Data: 2022-09-12T20:16:08.267 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.268 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.269 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.270 DEBUG:boto:Params: {} 2022-09-12T20:16:08.271 DEBUG:boto:Token: None 2022-09-12T20:16:08.272 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/ 2022-09-12T20:16:08.272 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws= 2022-09-12T20:16:08.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws=', 'Content-Length': '0'} 2022-09-12T20:16:08.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5ee231192cb7312-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.278 DEBUG:boto:b'jdpiak-75obj1241000falseobj1242022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:08.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:16:08.280 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:16:08.280 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:16:08.281 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:16:08.281 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:16:08.282 DEBUG:boto:Method: GET 2022-09-12T20:16:08.283 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:16:08.283 DEBUG:boto:Data: 2022-09-12T20:16:08.284 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.284 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.285 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.285 DEBUG:boto:Params: {} 2022-09-12T20:16:08.286 DEBUG:boto:Token: None 2022-09-12T20:16:08.287 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:16:08.287 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OBa6/YSNnEqMog3yLtprKkJxE9o= 2022-09-12T20:16:08.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OBa6/YSNnEqMog3yLtprKkJxE9o=', 'Content-Length': '0'} 2022-09-12T20:16:08.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ad583ae241ac987-00631f9388-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.292 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:16:08.293 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:16:08.294 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:16:08.295 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:16:08.296 DEBUG:boto:Method: GET 2022-09-12T20:16:08.297 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:16:08.298 DEBUG:boto:Data: 2022-09-12T20:16:08.298 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.299 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.300 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.300 DEBUG:boto:Params: {} 2022-09-12T20:16:08.301 DEBUG:boto:Token: None 2022-09-12T20:16:08.301 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:16:08.302 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OBa6/YSNnEqMog3yLtprKkJxE9o= 2022-09-12T20:16:08.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OBa6/YSNnEqMog3yLtprKkJxE9o=', 'Content-Length': '0'} 2022-09-12T20:16:08.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2d41d37bc47e567-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.346 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:16:08.347 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:16:08.347 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:16:08.348 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:16:08.348 DEBUG:boto:Method: HEAD 2022-09-12T20:16:08.349 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:16:08.349 DEBUG:boto:Data: 2022-09-12T20:16:08.350 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.350 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.351 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.351 DEBUG:boto:Params: {} 2022-09-12T20:16:08.352 DEBUG:boto:Token: None 2022-09-12T20:16:08.353 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:16:08.353 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OYE/lodnINs83Tlq6f1Kttg559E= 2022-09-12T20:16:08.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OYE/lodnINs83Tlq6f1Kttg559E=', 'Content-Length': '0'} 2022-09-12T20:16:08.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d36ec690cda50247-00631f9388-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.356 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:16:08.357 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:16:08.358 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:16:08.358 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:16:08.359 DEBUG:boto:Method: HEAD 2022-09-12T20:16:08.359 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:16:08.360 DEBUG:boto:Data: 2022-09-12T20:16:08.360 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.361 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.361 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.362 DEBUG:boto:Params: {} 2022-09-12T20:16:08.362 DEBUG:boto:Token: None 2022-09-12T20:16:08.363 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:16:08.363 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OYE/lodnINs83Tlq6f1Kttg559E= 2022-09-12T20:16:08.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OYE/lodnINs83Tlq6f1Kttg559E=', 'Content-Length': '0'} 2022-09-12T20:16:08.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfcd85b85cd986e2-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:16:08.367 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:16:08.368 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:16:08.368 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:16:08.369 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:16:08.369 DEBUG:boto:Method: GET 2022-09-12T20:16:08.370 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:16:08.371 DEBUG:boto:Data: 2022-09-12T20:16:08.371 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.372 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.372 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.373 DEBUG:boto:Params: {} 2022-09-12T20:16:08.374 DEBUG:boto:Token: None 2022-09-12T20:16:08.374 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:16:08.375 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OBa6/YSNnEqMog3yLtprKkJxE9o= 2022-09-12T20:16:08.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OBa6/YSNnEqMog3yLtprKkJxE9o=', 'Content-Length': '0'} 2022-09-12T20:16:08.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbfc445eedec3511-00631f9388-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.379 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:16:08.379 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:16:08.380 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:16:08.380 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:16:08.381 DEBUG:boto:Method: GET 2022-09-12T20:16:08.381 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:16:08.382 DEBUG:boto:Data: 2022-09-12T20:16:08.382 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.383 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.383 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.384 DEBUG:boto:Params: {} 2022-09-12T20:16:08.384 DEBUG:boto:Token: None 2022-09-12T20:16:08.385 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:16:08.386 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OBa6/YSNnEqMog3yLtprKkJxE9o= 2022-09-12T20:16:08.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OBa6/YSNnEqMog3yLtprKkJxE9o=', 'Content-Length': '0'} 2022-09-12T20:16:08.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000160f09229205a3bf-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.389 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:08.390 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:08.390 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:16:08.391 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:16:08.391 DEBUG:boto:Method: GET 2022-09-12T20:16:08.392 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:16:08.392 DEBUG:boto:Data: 2022-09-12T20:16:08.393 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.393 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.394 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.394 DEBUG:boto:Params: {} 2022-09-12T20:16:08.395 DEBUG:boto:Token: None 2022-09-12T20:16:08.396 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/ 2022-09-12T20:16:08.396 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws= 2022-09-12T20:16:08.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws=', 'Content-Length': '0'} 2022-09-12T20:16:08.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003792688a7bd1d3d5-00631f9388-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.442 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:08.443 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:08.444 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:08.444 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:16:08.445 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:16:08.445 DEBUG:boto:Method: GET 2022-09-12T20:16:08.446 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:16:08.446 DEBUG:boto:Data: 2022-09-12T20:16:08.447 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.447 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.448 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.448 DEBUG:boto:Params: {} 2022-09-12T20:16:08.449 DEBUG:boto:Token: None 2022-09-12T20:16:08.450 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/ 2022-09-12T20:16:08.450 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws= 2022-09-12T20:16:08.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws=', 'Content-Length': '0'} 2022-09-12T20:16:08.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004cc04e3c97a1ad4-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.456 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:08.457 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:16:08.457 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:16:08.458 DEBUG:boto:Method: HEAD 2022-09-12T20:16:08.458 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:16:08.459 DEBUG:boto:Data: 2022-09-12T20:16:08.459 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.460 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.460 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.461 DEBUG:boto:Params: {} 2022-09-12T20:16:08.461 DEBUG:boto:Token: None 2022-09-12T20:16:08.462 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj145 2022-09-12T20:16:08.462 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TMDfVQdwOgPR9r6yi3ZKGHebjfg= 2022-09-12T20:16:08.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TMDfVQdwOgPR9r6yi3ZKGHebjfg=', 'Content-Length': '0'} 2022-09-12T20:16:08.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c8d7d2e27f76711-00631f9388-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.466 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:16:08.466 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:16:08.467 DEBUG:boto:Method: HEAD 2022-09-12T20:16:08.467 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:16:08.468 DEBUG:boto:Data: 2022-09-12T20:16:08.468 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.469 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.469 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.470 DEBUG:boto:Params: {} 2022-09-12T20:16:08.470 DEBUG:boto:Token: None 2022-09-12T20:16:08.471 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj145 2022-09-12T20:16:08.472 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TMDfVQdwOgPR9r6yi3ZKGHebjfg= 2022-09-12T20:16:08.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TMDfVQdwOgPR9r6yi3ZKGHebjfg=', 'Content-Length': '0'} 2022-09-12T20:16:08.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001063ae3dbefa6a62-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:16:08.476 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:16:08.476 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:16:08.477 DEBUG:boto:Method: GET 2022-09-12T20:16:08.477 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:16:08.478 DEBUG:boto:Data: 2022-09-12T20:16:08.478 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.479 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.479 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.480 DEBUG:boto:Params: {} 2022-09-12T20:16:08.480 DEBUG:boto:Token: None 2022-09-12T20:16:08.481 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj145 2022-09-12T20:16:08.482 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ezmvfeq5JGeb1CW6wtgoP43irEo= 2022-09-12T20:16:08.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ezmvfeq5JGeb1CW6wtgoP43irEo=', 'Content-Length': '0'} 2022-09-12T20:16:08.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000511b4d8f6a9633c5-00631f9388-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.485 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:16:08.485 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:16:08.486 DEBUG:boto:Method: GET 2022-09-12T20:16:08.486 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:16:08.487 DEBUG:boto:Data: 2022-09-12T20:16:08.487 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.488 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.489 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.489 DEBUG:boto:Params: {} 2022-09-12T20:16:08.490 DEBUG:boto:Token: None 2022-09-12T20:16:08.490 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj145 2022-09-12T20:16:08.491 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ezmvfeq5JGeb1CW6wtgoP43irEo= 2022-09-12T20:16:08.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ezmvfeq5JGeb1CW6wtgoP43irEo=', 'Content-Length': '0'} 2022-09-12T20:16:08.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000490529893fc06e18-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.534 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:08.535 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:08.536 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:16:08.537 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:16:08.538 DEBUG:boto:Method: GET 2022-09-12T20:16:08.539 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:16:08.540 DEBUG:boto:Data: 2022-09-12T20:16:08.541 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.542 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.543 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.544 DEBUG:boto:Params: {} 2022-09-12T20:16:08.545 DEBUG:boto:Token: None 2022-09-12T20:16:08.546 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/ 2022-09-12T20:16:08.548 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws= 2022-09-12T20:16:08.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws=', 'Content-Length': '0'} 2022-09-12T20:16:08.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6ba0197609772da-00631f9388-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.552 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:08.553 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:08.553 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:08.554 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:16:08.555 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:16:08.555 DEBUG:boto:Method: GET 2022-09-12T20:16:08.556 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:16:08.556 DEBUG:boto:Data: 2022-09-12T20:16:08.557 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.557 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.558 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.558 DEBUG:boto:Params: {} 2022-09-12T20:16:08.559 DEBUG:boto:Token: None 2022-09-12T20:16:08.559 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/ 2022-09-12T20:16:08.560 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws= 2022-09-12T20:16:08.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws=', 'Content-Length': '0'} 2022-09-12T20:16:08.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f1d13e99f9943ad-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.566 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:08.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2022-09-12T20:16:08.567 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:16:08.568 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:16:08.568 DEBUG:boto:path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:16:08.569 DEBUG:boto:auth_path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:16:08.569 DEBUG:boto:Method: GET 2022-09-12T20:16:08.570 DEBUG:boto:Path: /jdpiak-75/obj148?versionId=null 2022-09-12T20:16:08.570 DEBUG:boto:Data: 2022-09-12T20:16:08.571 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.572 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.572 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.573 DEBUG:boto:Params: {} 2022-09-12T20:16:08.573 DEBUG:boto:Token: None 2022-09-12T20:16:08.574 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj148?versionId=null 2022-09-12T20:16:08.574 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nK2n3du6k64lTiC5epdyKFQiLh4= 2022-09-12T20:16:08.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nK2n3du6k64lTiC5epdyKFQiLh4=', 'Content-Length': '0'} 2022-09-12T20:16:08.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c3b379d769617d2-00631f9388-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.578 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:16:08.579 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:16:08.580 DEBUG:boto:path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:16:08.580 DEBUG:boto:auth_path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:16:08.581 DEBUG:boto:Method: GET 2022-09-12T20:16:08.581 DEBUG:boto:Path: /jdpiak-75/obj148?versionId=null 2022-09-12T20:16:08.582 DEBUG:boto:Data: 2022-09-12T20:16:08.582 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.583 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.583 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.584 DEBUG:boto:Params: {} 2022-09-12T20:16:08.585 DEBUG:boto:Token: None 2022-09-12T20:16:08.585 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj148?versionId=null 2022-09-12T20:16:08.586 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nK2n3du6k64lTiC5epdyKFQiLh4= 2022-09-12T20:16:08.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nK2n3du6k64lTiC5epdyKFQiLh4=', 'Content-Length': '0'} 2022-09-12T20:16:08.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d3da3545e66c9c0-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.590 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:16:08.590 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:16:08.591 DEBUG:boto:path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:16:08.591 DEBUG:boto:auth_path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:16:08.592 DEBUG:boto:Method: HEAD 2022-09-12T20:16:08.592 DEBUG:boto:Path: /jdpiak-75/obj148?versionId=null 2022-09-12T20:16:08.593 DEBUG:boto:Data: 2022-09-12T20:16:08.593 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.594 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.594 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.595 DEBUG:boto:Params: {} 2022-09-12T20:16:08.595 DEBUG:boto:Token: None 2022-09-12T20:16:08.596 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj148?versionId=null 2022-09-12T20:16:08.597 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:svRBPkMS5roat8mkgI3wnGf/d+w= 2022-09-12T20:16:08.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:svRBPkMS5roat8mkgI3wnGf/d+w=', 'Content-Length': '0'} 2022-09-12T20:16:08.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faad14c087aa5468-00631f9388-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.600 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:16:08.601 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:16:08.601 DEBUG:boto:path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:16:08.602 DEBUG:boto:auth_path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:16:08.602 DEBUG:boto:Method: HEAD 2022-09-12T20:16:08.603 DEBUG:boto:Path: /jdpiak-75/obj148?versionId=null 2022-09-12T20:16:08.603 DEBUG:boto:Data: 2022-09-12T20:16:08.604 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.604 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.605 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.605 DEBUG:boto:Params: {} 2022-09-12T20:16:08.606 DEBUG:boto:Token: None 2022-09-12T20:16:08.606 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj148?versionId=null 2022-09-12T20:16:08.607 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:svRBPkMS5roat8mkgI3wnGf/d+w= 2022-09-12T20:16:08.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:svRBPkMS5roat8mkgI3wnGf/d+w=', 'Content-Length': '0'} 2022-09-12T20:16:08.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000451afdf0c9c398a-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2022-09-12T20:16:08.611 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:16:08.611 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:16:08.612 DEBUG:boto:path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:16:08.613 DEBUG:boto:auth_path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:16:08.613 DEBUG:boto:Method: GET 2022-09-12T20:16:08.614 DEBUG:boto:Path: /jdpiak-75/obj148?versionId=null 2022-09-12T20:16:08.614 DEBUG:boto:Data: 2022-09-12T20:16:08.615 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.616 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.616 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.617 DEBUG:boto:Params: {} 2022-09-12T20:16:08.618 DEBUG:boto:Token: None 2022-09-12T20:16:08.619 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj148?versionId=null 2022-09-12T20:16:08.620 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nK2n3du6k64lTiC5epdyKFQiLh4= 2022-09-12T20:16:08.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nK2n3du6k64lTiC5epdyKFQiLh4=', 'Content-Length': '0'} 2022-09-12T20:16:08.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd72d28904efc61d-00631f9388-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.625 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:16:08.625 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:16:08.626 DEBUG:boto:path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:16:08.626 DEBUG:boto:auth_path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:16:08.627 DEBUG:boto:Method: GET 2022-09-12T20:16:08.627 DEBUG:boto:Path: /jdpiak-75/obj148?versionId=null 2022-09-12T20:16:08.628 DEBUG:boto:Data: 2022-09-12T20:16:08.629 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.629 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.630 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.630 DEBUG:boto:Params: {} 2022-09-12T20:16:08.631 DEBUG:boto:Token: None 2022-09-12T20:16:08.631 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj148?versionId=null 2022-09-12T20:16:08.632 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nK2n3du6k64lTiC5epdyKFQiLh4= 2022-09-12T20:16:08.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nK2n3du6k64lTiC5epdyKFQiLh4=', 'Content-Length': '0'} 2022-09-12T20:16:08.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005df2c65163aa6f05-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.636 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:08.636 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:08.637 DEBUG:boto:path=/jdpiak-75/?prefix=obj148 2022-09-12T20:16:08.637 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj148 2022-09-12T20:16:08.638 DEBUG:boto:Method: GET 2022-09-12T20:16:08.639 DEBUG:boto:Path: /jdpiak-75/?prefix=obj148 2022-09-12T20:16:08.639 DEBUG:boto:Data: 2022-09-12T20:16:08.640 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.640 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.641 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.641 DEBUG:boto:Params: {} 2022-09-12T20:16:08.642 DEBUG:boto:Token: None 2022-09-12T20:16:08.642 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/ 2022-09-12T20:16:08.643 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws= 2022-09-12T20:16:08.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws=', 'Content-Length': '0'} 2022-09-12T20:16:08.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001aac2151cacef4b6-00631f9388-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.647 DEBUG:boto:b'jdpiak-75obj1481000falseobj1482022-09-12T20:14:27.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:08.648 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:08.649 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:08.649 DEBUG:boto:path=/jdpiak-75/?prefix=obj148 2022-09-12T20:16:08.650 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj148 2022-09-12T20:16:08.651 DEBUG:boto:Method: GET 2022-09-12T20:16:08.651 DEBUG:boto:Path: /jdpiak-75/?prefix=obj148 2022-09-12T20:16:08.652 DEBUG:boto:Data: 2022-09-12T20:16:08.652 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.653 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.654 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.654 DEBUG:boto:Params: {} 2022-09-12T20:16:08.655 DEBUG:boto:Token: None 2022-09-12T20:16:08.655 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/ 2022-09-12T20:16:08.656 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws= 2022-09-12T20:16:08.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws=', 'Content-Length': '0'} 2022-09-12T20:16:08.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045c76f16b97fee2c-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.706 DEBUG:boto:b'jdpiak-75obj1481000falseobj1482022-09-12T20:14:27.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:08.707 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:16:08.708 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:16:08.708 DEBUG:boto:Method: HEAD 2022-09-12T20:16:08.709 DEBUG:boto:Path: /jdpiak-75/obj148 2022-09-12T20:16:08.710 DEBUG:boto:Data: 2022-09-12T20:16:08.710 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.711 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.711 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.712 DEBUG:boto:Params: {} 2022-09-12T20:16:08.712 DEBUG:boto:Token: None 2022-09-12T20:16:08.713 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj148 2022-09-12T20:16:08.714 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k1f0vTTebwQv+Dg5sW4XCNFlgek= 2022-09-12T20:16:08.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k1f0vTTebwQv+Dg5sW4XCNFlgek=', 'Content-Length': '0'} 2022-09-12T20:16:08.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f2ce98dfe7f8eac-00631f9388-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.718 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:16:08.718 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:16:08.719 DEBUG:boto:Method: HEAD 2022-09-12T20:16:08.719 DEBUG:boto:Path: /jdpiak-75/obj148 2022-09-12T20:16:08.720 DEBUG:boto:Data: 2022-09-12T20:16:08.720 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.721 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.722 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.722 DEBUG:boto:Params: {} 2022-09-12T20:16:08.723 DEBUG:boto:Token: None 2022-09-12T20:16:08.723 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj148 2022-09-12T20:16:08.724 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:k1f0vTTebwQv+Dg5sW4XCNFlgek= 2022-09-12T20:16:08.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:k1f0vTTebwQv+Dg5sW4XCNFlgek=', 'Content-Length': '0'} 2022-09-12T20:16:08.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000475dce72a1c39465-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2022-09-12T20:16:08.728 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:16:08.729 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:16:08.729 DEBUG:boto:Method: GET 2022-09-12T20:16:08.730 DEBUG:boto:Path: /jdpiak-75/obj148 2022-09-12T20:16:08.731 DEBUG:boto:Data: 2022-09-12T20:16:08.731 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.732 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.732 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.733 DEBUG:boto:Params: {} 2022-09-12T20:16:08.733 DEBUG:boto:Token: None 2022-09-12T20:16:08.734 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj148 2022-09-12T20:16:08.735 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4TP8GlHeUNdRBr9b9dS39nk5zoU= 2022-09-12T20:16:08.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4TP8GlHeUNdRBr9b9dS39nk5zoU=', 'Content-Length': '0'} 2022-09-12T20:16:08.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004339e5378f08f418-00631f9388-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.739 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:16:08.739 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:16:08.740 DEBUG:boto:Method: GET 2022-09-12T20:16:08.740 DEBUG:boto:Path: /jdpiak-75/obj148 2022-09-12T20:16:08.741 DEBUG:boto:Data: 2022-09-12T20:16:08.742 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.742 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.743 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.743 DEBUG:boto:Params: {} 2022-09-12T20:16:08.744 DEBUG:boto:Token: None 2022-09-12T20:16:08.745 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj148 2022-09-12T20:16:08.745 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4TP8GlHeUNdRBr9b9dS39nk5zoU= 2022-09-12T20:16:08.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4TP8GlHeUNdRBr9b9dS39nk5zoU=', 'Content-Length': '0'} 2022-09-12T20:16:08.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065fea548ca319151-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.749 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:08.749 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:08.750 DEBUG:boto:path=/jdpiak-75/?prefix=obj148 2022-09-12T20:16:08.751 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj148 2022-09-12T20:16:08.751 DEBUG:boto:Method: GET 2022-09-12T20:16:08.752 DEBUG:boto:Path: /jdpiak-75/?prefix=obj148 2022-09-12T20:16:08.752 DEBUG:boto:Data: 2022-09-12T20:16:08.753 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.753 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.754 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.754 DEBUG:boto:Params: {} 2022-09-12T20:16:08.755 DEBUG:boto:Token: None 2022-09-12T20:16:08.756 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/ 2022-09-12T20:16:08.756 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws= 2022-09-12T20:16:08.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws=', 'Content-Length': '0'} 2022-09-12T20:16:08.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000422372fe421881db-00631f9388-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.761 DEBUG:boto:b'jdpiak-75obj1481000falseobj1482022-09-12T20:14:27.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:08.762 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:08.762 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:08.763 DEBUG:boto:path=/jdpiak-75/?prefix=obj148 2022-09-12T20:16:08.763 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj148 2022-09-12T20:16:08.764 DEBUG:boto:Method: GET 2022-09-12T20:16:08.764 DEBUG:boto:Path: /jdpiak-75/?prefix=obj148 2022-09-12T20:16:08.765 DEBUG:boto:Data: 2022-09-12T20:16:08.765 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.766 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.767 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.767 DEBUG:boto:Params: {} 2022-09-12T20:16:08.768 DEBUG:boto:Token: None 2022-09-12T20:16:08.768 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/ 2022-09-12T20:16:08.769 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws= 2022-09-12T20:16:08.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws=', 'Content-Length': '0'} 2022-09-12T20:16:08.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a6a63553396eaf7-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.775 DEBUG:boto:b'jdpiak-75obj1481000falseobj1482022-09-12T20:14:27.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:08.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-12T20:16:08.777 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:08.777 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:08.778 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:08.779 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:08.779 DEBUG:boto:Method: GET 2022-09-12T20:16:08.780 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:08.780 DEBUG:boto:Data: 2022-09-12T20:16:08.781 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.781 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.782 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.783 DEBUG:boto:Params: {} 2022-09-12T20:16:08.783 DEBUG:boto:Token: None 2022-09-12T20:16:08.784 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:08.784 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NnAiqGLvqz8KVAEkLGuHUTM5XcA= 2022-09-12T20:16:08.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NnAiqGLvqz8KVAEkLGuHUTM5XcA=', 'Content-Length': '0'} 2022-09-12T20:16:08.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9143e0fdb9ff240-00631f9388-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.788 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:08.789 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:08.790 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:08.790 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:08.791 DEBUG:boto:Method: GET 2022-09-12T20:16:08.792 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:08.793 DEBUG:boto:Data: 2022-09-12T20:16:08.793 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.794 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.795 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.795 DEBUG:boto:Params: {} 2022-09-12T20:16:08.796 DEBUG:boto:Token: None 2022-09-12T20:16:08.796 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:08.797 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NnAiqGLvqz8KVAEkLGuHUTM5XcA= 2022-09-12T20:16:08.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NnAiqGLvqz8KVAEkLGuHUTM5XcA=', 'Content-Length': '0'} 2022-09-12T20:16:08.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b674eec8940067cc-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.846 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:08.847 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:08.847 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:08.848 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:08.848 DEBUG:boto:Method: HEAD 2022-09-12T20:16:08.849 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:08.849 DEBUG:boto:Data: 2022-09-12T20:16:08.850 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.850 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.851 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.852 DEBUG:boto:Params: {} 2022-09-12T20:16:08.853 DEBUG:boto:Token: None 2022-09-12T20:16:08.854 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:08.855 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F9jOV20BwqcjSDANFqBaNfJUOHM= 2022-09-12T20:16:08.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F9jOV20BwqcjSDANFqBaNfJUOHM=', 'Content-Length': '0'} 2022-09-12T20:16:08.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea9a56d74b10e1ec-00631f9388-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.860 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:08.861 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:08.862 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:08.862 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:08.863 DEBUG:boto:Method: HEAD 2022-09-12T20:16:08.863 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:08.864 DEBUG:boto:Data: 2022-09-12T20:16:08.865 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.865 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.866 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.866 DEBUG:boto:Params: {} 2022-09-12T20:16:08.867 DEBUG:boto:Token: None 2022-09-12T20:16:08.868 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:08.868 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F9jOV20BwqcjSDANFqBaNfJUOHM= 2022-09-12T20:16:08.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F9jOV20BwqcjSDANFqBaNfJUOHM=', 'Content-Length': '0'} 2022-09-12T20:16:08.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f8eed6ea2d07f0c-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-12T20:16:08.872 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:08.873 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:08.873 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:08.874 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:08.874 DEBUG:boto:Method: GET 2022-09-12T20:16:08.875 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:08.875 DEBUG:boto:Data: 2022-09-12T20:16:08.876 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.877 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.877 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.878 DEBUG:boto:Params: {} 2022-09-12T20:16:08.879 DEBUG:boto:Token: None 2022-09-12T20:16:08.879 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:08.880 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NnAiqGLvqz8KVAEkLGuHUTM5XcA= 2022-09-12T20:16:08.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NnAiqGLvqz8KVAEkLGuHUTM5XcA=', 'Content-Length': '0'} 2022-09-12T20:16:08.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8b833cea7c4b180-00631f9388-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.926 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:08.927 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:08.927 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:08.928 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:08.929 DEBUG:boto:Method: GET 2022-09-12T20:16:08.929 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:08.930 DEBUG:boto:Data: 2022-09-12T20:16:08.930 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.931 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.932 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.932 DEBUG:boto:Params: {} 2022-09-12T20:16:08.934 DEBUG:boto:Token: None 2022-09-12T20:16:08.935 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:08.936 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NnAiqGLvqz8KVAEkLGuHUTM5XcA= 2022-09-12T20:16:08.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NnAiqGLvqz8KVAEkLGuHUTM5XcA=', 'Content-Length': '0'} 2022-09-12T20:16:08.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f73e0f2b39b774d1-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.941 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:08.942 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:08.943 DEBUG:boto:path=/jdpiak-75/?prefix=obj155 2022-09-12T20:16:08.944 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj155 2022-09-12T20:16:08.945 DEBUG:boto:Method: GET 2022-09-12T20:16:08.946 DEBUG:boto:Path: /jdpiak-75/?prefix=obj155 2022-09-12T20:16:08.947 DEBUG:boto:Data: 2022-09-12T20:16:08.947 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.948 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.948 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.949 DEBUG:boto:Params: {} 2022-09-12T20:16:08.949 DEBUG:boto:Token: None 2022-09-12T20:16:08.950 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/ 2022-09-12T20:16:08.950 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws= 2022-09-12T20:16:08.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws=', 'Content-Length': '0'} 2022-09-12T20:16:08.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000658558c614faab9c-00631f9388-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.955 DEBUG:boto:b'jdpiak-75obj1551000falseobj1552022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:08.955 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:08.956 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:08.957 DEBUG:boto:path=/jdpiak-75/?prefix=obj155 2022-09-12T20:16:08.957 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj155 2022-09-12T20:16:08.958 DEBUG:boto:Method: GET 2022-09-12T20:16:08.958 DEBUG:boto:Path: /jdpiak-75/?prefix=obj155 2022-09-12T20:16:08.959 DEBUG:boto:Data: 2022-09-12T20:16:08.959 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.960 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.960 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.961 DEBUG:boto:Params: {} 2022-09-12T20:16:08.961 DEBUG:boto:Token: None 2022-09-12T20:16:08.962 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/ 2022-09-12T20:16:08.963 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws= 2022-09-12T20:16:08.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UZN5wYh238m3ozo0LvG0EgvcUws=', 'Content-Length': '0'} 2022-09-12T20:16:08.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c7144bf4eec8ebd-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.969 DEBUG:boto:b'jdpiak-75obj1551000falseobj1552022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:08.970 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:08.971 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:08.972 DEBUG:boto:Method: HEAD 2022-09-12T20:16:08.972 DEBUG:boto:Path: /jdpiak-75/obj155 2022-09-12T20:16:08.973 DEBUG:boto:Data: 2022-09-12T20:16:08.973 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.974 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.974 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.975 DEBUG:boto:Params: {} 2022-09-12T20:16:08.975 DEBUG:boto:Token: None 2022-09-12T20:16:08.976 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj155 2022-09-12T20:16:08.976 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ic8+czYQkih3mP13/mkjcuar8RE= 2022-09-12T20:16:08.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ic8+czYQkih3mP13/mkjcuar8RE=', 'Content-Length': '0'} 2022-09-12T20:16:08.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e2b95e8465011c6-00631f9388-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.981 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:08.982 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:08.983 DEBUG:boto:Method: HEAD 2022-09-12T20:16:08.984 DEBUG:boto:Path: /jdpiak-75/obj155 2022-09-12T20:16:08.985 DEBUG:boto:Data: 2022-09-12T20:16:08.986 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.987 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:08.987 DEBUG:boto:Port: 8000 2022-09-12T20:16:08.988 DEBUG:boto:Params: {} 2022-09-12T20:16:08.989 DEBUG:boto:Token: None 2022-09-12T20:16:08.989 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:08 GMT /jdpiak-75/obj155 2022-09-12T20:16:08.990 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ic8+czYQkih3mP13/mkjcuar8RE= 2022-09-12T20:16:08.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:08 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ic8+czYQkih3mP13/mkjcuar8RE=', 'Content-Length': '0'} 2022-09-12T20:16:08.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0c2b5309b02e39b-00631f9388-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:08.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-12T20:16:08.994 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:08.995 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:08.995 DEBUG:boto:Method: GET 2022-09-12T20:16:08.996 DEBUG:boto:Path: /jdpiak-75/obj155 2022-09-12T20:16:08.996 DEBUG:boto:Data: 2022-09-12T20:16:08.997 DEBUG:boto:Headers: {} 2022-09-12T20:16:08.997 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:08.998 DEBUG:boto:Port: 8001 2022-09-12T20:16:08.998 DEBUG:boto:Params: {} 2022-09-12T20:16:08.999 DEBUG:boto:Token: None 2022-09-12T20:16:09.000 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj155 2022-09-12T20:16:09.001 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TG9O0+ygbz1GVJqau1Ely81zIRI= 2022-09-12T20:16:09.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TG9O0+ygbz1GVJqau1Ely81zIRI=', 'Content-Length': '0'} 2022-09-12T20:16:09.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004671b82d0e1f92dc-00631f9389-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.005 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:09.005 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:09.006 DEBUG:boto:Method: GET 2022-09-12T20:16:09.007 DEBUG:boto:Path: /jdpiak-75/obj155 2022-09-12T20:16:09.007 DEBUG:boto:Data: 2022-09-12T20:16:09.008 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.008 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.009 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.009 DEBUG:boto:Params: {} 2022-09-12T20:16:09.010 DEBUG:boto:Token: None 2022-09-12T20:16:09.011 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj155 2022-09-12T20:16:09.011 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TG9O0+ygbz1GVJqau1Ely81zIRI= 2022-09-12T20:16:09.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TG9O0+ygbz1GVJqau1Ely81zIRI=', 'Content-Length': '0'} 2022-09-12T20:16:09.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed55a914f04c64a6-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.016 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:09.016 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:09.017 DEBUG:boto:path=/jdpiak-75/?prefix=obj155 2022-09-12T20:16:09.017 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj155 2022-09-12T20:16:09.018 DEBUG:boto:Method: GET 2022-09-12T20:16:09.019 DEBUG:boto:Path: /jdpiak-75/?prefix=obj155 2022-09-12T20:16:09.020 DEBUG:boto:Data: 2022-09-12T20:16:09.020 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.021 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.022 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.022 DEBUG:boto:Params: {} 2022-09-12T20:16:09.023 DEBUG:boto:Token: None 2022-09-12T20:16:09.024 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/ 2022-09-12T20:16:09.025 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg= 2022-09-12T20:16:09.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg=', 'Content-Length': '0'} 2022-09-12T20:16:09.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000548bae4cea932693-00631f9389-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.030 DEBUG:boto:b'jdpiak-75obj1551000falseobj1552022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:09.031 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:09.032 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:09.033 DEBUG:boto:path=/jdpiak-75/?prefix=obj155 2022-09-12T20:16:09.034 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj155 2022-09-12T20:16:09.035 DEBUG:boto:Method: GET 2022-09-12T20:16:09.036 DEBUG:boto:Path: /jdpiak-75/?prefix=obj155 2022-09-12T20:16:09.036 DEBUG:boto:Data: 2022-09-12T20:16:09.037 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.038 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.038 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.039 DEBUG:boto:Params: {} 2022-09-12T20:16:09.040 DEBUG:boto:Token: None 2022-09-12T20:16:09.041 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/ 2022-09-12T20:16:09.041 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg= 2022-09-12T20:16:09.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg=', 'Content-Length': '0'} 2022-09-12T20:16:09.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034b6e37cd3477280-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.090 DEBUG:boto:b'jdpiak-75obj1551000falseobj1552022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:09.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:16:09.093 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:09.094 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:09.094 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:09.095 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:09.096 DEBUG:boto:Method: GET 2022-09-12T20:16:09.097 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:09.098 DEBUG:boto:Data: 2022-09-12T20:16:09.099 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.100 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.101 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.102 DEBUG:boto:Params: {} 2022-09-12T20:16:09.103 DEBUG:boto:Token: None 2022-09-12T20:16:09.104 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:09.105 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HJjb+8cWl0EiTKph3juNEa1xilo= 2022-09-12T20:16:09.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HJjb+8cWl0EiTKph3juNEa1xilo=', 'Content-Length': '0'} 2022-09-12T20:16:09.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5e5567e6f9083d5-00631f9389-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.110 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:09.111 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:09.111 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:09.112 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:09.113 DEBUG:boto:Method: GET 2022-09-12T20:16:09.113 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:09.114 DEBUG:boto:Data: 2022-09-12T20:16:09.115 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.116 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.117 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.118 DEBUG:boto:Params: {} 2022-09-12T20:16:09.120 DEBUG:boto:Token: None 2022-09-12T20:16:09.121 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:09.122 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HJjb+8cWl0EiTKph3juNEa1xilo= 2022-09-12T20:16:09.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HJjb+8cWl0EiTKph3juNEa1xilo=', 'Content-Length': '0'} 2022-09-12T20:16:09.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000555275ecdc2fbb9f-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.127 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:09.128 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:09.129 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:09.130 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:09.131 DEBUG:boto:Method: HEAD 2022-09-12T20:16:09.132 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:09.133 DEBUG:boto:Data: 2022-09-12T20:16:09.134 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.135 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.136 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.137 DEBUG:boto:Params: {} 2022-09-12T20:16:09.138 DEBUG:boto:Token: None 2022-09-12T20:16:09.139 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:09.140 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uyRLQRG5zKM86fI9ETC/G9Lwhi8= 2022-09-12T20:16:09.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uyRLQRG5zKM86fI9ETC/G9Lwhi8=', 'Content-Length': '0'} 2022-09-12T20:16:09.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081139ebb59e7e2fe-00631f9389-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.145 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:09.146 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:09.147 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:09.148 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:09.148 DEBUG:boto:Method: HEAD 2022-09-12T20:16:09.149 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:09.150 DEBUG:boto:Data: 2022-09-12T20:16:09.151 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.152 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.153 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.154 DEBUG:boto:Params: {} 2022-09-12T20:16:09.155 DEBUG:boto:Token: None 2022-09-12T20:16:09.156 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:09.157 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uyRLQRG5zKM86fI9ETC/G9Lwhi8= 2022-09-12T20:16:09.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uyRLQRG5zKM86fI9ETC/G9Lwhi8=', 'Content-Length': '0'} 2022-09-12T20:16:09.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc546f1c2546821b-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:16:09.164 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:09.165 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:09.165 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:09.166 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:09.167 DEBUG:boto:Method: GET 2022-09-12T20:16:09.168 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:09.169 DEBUG:boto:Data: 2022-09-12T20:16:09.170 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.171 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.172 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.173 DEBUG:boto:Params: {} 2022-09-12T20:16:09.174 DEBUG:boto:Token: None 2022-09-12T20:16:09.175 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:09.175 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HJjb+8cWl0EiTKph3juNEa1xilo= 2022-09-12T20:16:09.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HJjb+8cWl0EiTKph3juNEa1xilo=', 'Content-Length': '0'} 2022-09-12T20:16:09.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039e6526118bc735f-00631f9389-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.179 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:09.180 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:09.180 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:09.181 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:09.181 DEBUG:boto:Method: GET 2022-09-12T20:16:09.182 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:09.182 DEBUG:boto:Data: 2022-09-12T20:16:09.183 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.183 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.184 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.184 DEBUG:boto:Params: {} 2022-09-12T20:16:09.185 DEBUG:boto:Token: None 2022-09-12T20:16:09.186 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:09.186 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HJjb+8cWl0EiTKph3juNEa1xilo= 2022-09-12T20:16:09.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HJjb+8cWl0EiTKph3juNEa1xilo=', 'Content-Length': '0'} 2022-09-12T20:16:09.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d989145a071303e-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.190 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:09.191 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:09.191 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:16:09.192 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:16:09.192 DEBUG:boto:Method: GET 2022-09-12T20:16:09.193 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:16:09.193 DEBUG:boto:Data: 2022-09-12T20:16:09.194 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.194 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.195 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.195 DEBUG:boto:Params: {} 2022-09-12T20:16:09.196 DEBUG:boto:Token: None 2022-09-12T20:16:09.196 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/ 2022-09-12T20:16:09.197 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg= 2022-09-12T20:16:09.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg=', 'Content-Length': '0'} 2022-09-12T20:16:09.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1d34091811ee8be-00631f9389-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.242 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:09.243 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:09.244 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:09.244 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:16:09.245 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:16:09.245 DEBUG:boto:Method: GET 2022-09-12T20:16:09.246 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:16:09.247 DEBUG:boto:Data: 2022-09-12T20:16:09.247 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.248 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.248 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.249 DEBUG:boto:Params: {} 2022-09-12T20:16:09.249 DEBUG:boto:Token: None 2022-09-12T20:16:09.250 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/ 2022-09-12T20:16:09.251 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg= 2022-09-12T20:16:09.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg=', 'Content-Length': '0'} 2022-09-12T20:16:09.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039f8c96810fb2ed2-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.257 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:09.258 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:09.258 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:09.259 DEBUG:boto:Method: HEAD 2022-09-12T20:16:09.259 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:16:09.260 DEBUG:boto:Data: 2022-09-12T20:16:09.260 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.261 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.261 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.262 DEBUG:boto:Params: {} 2022-09-12T20:16:09.263 DEBUG:boto:Token: None 2022-09-12T20:16:09.263 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj17 2022-09-12T20:16:09.264 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2B0S8gNs8RhTTF0udYduarSx8RI= 2022-09-12T20:16:09.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2B0S8gNs8RhTTF0udYduarSx8RI=', 'Content-Length': '0'} 2022-09-12T20:16:09.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035aaf20ffd48e2f8-00631f9389-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.268 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:09.268 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:09.269 DEBUG:boto:Method: HEAD 2022-09-12T20:16:09.269 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:16:09.270 DEBUG:boto:Data: 2022-09-12T20:16:09.270 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.271 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.271 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.272 DEBUG:boto:Params: {} 2022-09-12T20:16:09.272 DEBUG:boto:Token: None 2022-09-12T20:16:09.273 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj17 2022-09-12T20:16:09.273 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2B0S8gNs8RhTTF0udYduarSx8RI= 2022-09-12T20:16:09.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2B0S8gNs8RhTTF0udYduarSx8RI=', 'Content-Length': '0'} 2022-09-12T20:16:09.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008257ca7d9a011204-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:16:09.277 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:09.278 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:09.278 DEBUG:boto:Method: GET 2022-09-12T20:16:09.279 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:16:09.279 DEBUG:boto:Data: 2022-09-12T20:16:09.280 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.281 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.281 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.282 DEBUG:boto:Params: {} 2022-09-12T20:16:09.282 DEBUG:boto:Token: None 2022-09-12T20:16:09.283 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj17 2022-09-12T20:16:09.283 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Pyph62NiqaDTxF7QqKeRdh1OyxY= 2022-09-12T20:16:09.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Pyph62NiqaDTxF7QqKeRdh1OyxY=', 'Content-Length': '0'} 2022-09-12T20:16:09.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c036727dec56a66c-00631f9389-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.288 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:09.288 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:09.289 DEBUG:boto:Method: GET 2022-09-12T20:16:09.289 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:16:09.290 DEBUG:boto:Data: 2022-09-12T20:16:09.291 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.291 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.292 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.292 DEBUG:boto:Params: {} 2022-09-12T20:16:09.293 DEBUG:boto:Token: None 2022-09-12T20:16:09.293 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj17 2022-09-12T20:16:09.294 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Pyph62NiqaDTxF7QqKeRdh1OyxY= 2022-09-12T20:16:09.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Pyph62NiqaDTxF7QqKeRdh1OyxY=', 'Content-Length': '0'} 2022-09-12T20:16:09.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ca5e9224e43c329-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.298 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:09.299 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:09.299 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:16:09.300 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:16:09.300 DEBUG:boto:Method: GET 2022-09-12T20:16:09.301 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:16:09.301 DEBUG:boto:Data: 2022-09-12T20:16:09.302 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.303 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.303 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.304 DEBUG:boto:Params: {} 2022-09-12T20:16:09.304 DEBUG:boto:Token: None 2022-09-12T20:16:09.305 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/ 2022-09-12T20:16:09.305 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg= 2022-09-12T20:16:09.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg=', 'Content-Length': '0'} 2022-09-12T20:16:09.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000397bcd8277f3ed7b-00631f9389-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.354 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:09.355 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:09.356 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:09.356 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:16:09.357 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:16:09.357 DEBUG:boto:Method: GET 2022-09-12T20:16:09.358 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:16:09.359 DEBUG:boto:Data: 2022-09-12T20:16:09.359 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.360 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.360 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.361 DEBUG:boto:Params: {} 2022-09-12T20:16:09.362 DEBUG:boto:Token: None 2022-09-12T20:16:09.362 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/ 2022-09-12T20:16:09.363 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg= 2022-09-12T20:16:09.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg=', 'Content-Length': '0'} 2022-09-12T20:16:09.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8eed9654e4ce4c9-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.369 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:09.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2022-09-12T20:16:09.371 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:09.371 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:09.372 DEBUG:boto:path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:09.372 DEBUG:boto:auth_path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:09.373 DEBUG:boto:Method: GET 2022-09-12T20:16:09.373 DEBUG:boto:Path: /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:09.374 DEBUG:boto:Data: 2022-09-12T20:16:09.375 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.375 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.376 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.376 DEBUG:boto:Params: {} 2022-09-12T20:16:09.377 DEBUG:boto:Token: None 2022-09-12T20:16:09.378 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:09.378 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RGP8wdj8lNJ8+PNDbZwpC8Z+ARs= 2022-09-12T20:16:09.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RGP8wdj8lNJ8+PNDbZwpC8Z+ARs=', 'Content-Length': '0'} 2022-09-12T20:16:09.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006f22162691356bf-00631f9389-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.382 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:09.383 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:09.383 DEBUG:boto:path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:09.384 DEBUG:boto:auth_path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:09.385 DEBUG:boto:Method: GET 2022-09-12T20:16:09.385 DEBUG:boto:Path: /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:09.386 DEBUG:boto:Data: 2022-09-12T20:16:09.386 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.387 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.387 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.388 DEBUG:boto:Params: {} 2022-09-12T20:16:09.389 DEBUG:boto:Token: None 2022-09-12T20:16:09.389 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:09.390 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RGP8wdj8lNJ8+PNDbZwpC8Z+ARs= 2022-09-12T20:16:09.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RGP8wdj8lNJ8+PNDbZwpC8Z+ARs=', 'Content-Length': '0'} 2022-09-12T20:16:09.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce0a5eca5d8469de-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.434 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:09.435 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:09.436 DEBUG:boto:path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:09.436 DEBUG:boto:auth_path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:09.437 DEBUG:boto:Method: HEAD 2022-09-12T20:16:09.437 DEBUG:boto:Path: /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:09.438 DEBUG:boto:Data: 2022-09-12T20:16:09.438 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.439 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.439 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.440 DEBUG:boto:Params: {} 2022-09-12T20:16:09.441 DEBUG:boto:Token: None 2022-09-12T20:16:09.441 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:09.442 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WcqzFGRXJv4yuHunw2+cgWMstC4= 2022-09-12T20:16:09.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WcqzFGRXJv4yuHunw2+cgWMstC4=', 'Content-Length': '0'} 2022-09-12T20:16:09.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb21fe459cf47170-00631f9389-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.445 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:09.446 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:09.447 DEBUG:boto:path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:09.447 DEBUG:boto:auth_path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:09.448 DEBUG:boto:Method: HEAD 2022-09-12T20:16:09.448 DEBUG:boto:Path: /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:09.449 DEBUG:boto:Data: 2022-09-12T20:16:09.449 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.450 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.450 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.451 DEBUG:boto:Params: {} 2022-09-12T20:16:09.451 DEBUG:boto:Token: None 2022-09-12T20:16:09.452 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:09.453 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WcqzFGRXJv4yuHunw2+cgWMstC4= 2022-09-12T20:16:09.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WcqzFGRXJv4yuHunw2+cgWMstC4=', 'Content-Length': '0'} 2022-09-12T20:16:09.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079c5b25c5d6ca963-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2022-09-12T20:16:09.457 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:09.458 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:09.458 DEBUG:boto:path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:09.459 DEBUG:boto:auth_path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:09.459 DEBUG:boto:Method: GET 2022-09-12T20:16:09.460 DEBUG:boto:Path: /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:09.460 DEBUG:boto:Data: 2022-09-12T20:16:09.461 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.461 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.462 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.463 DEBUG:boto:Params: {} 2022-09-12T20:16:09.463 DEBUG:boto:Token: None 2022-09-12T20:16:09.464 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:09.464 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RGP8wdj8lNJ8+PNDbZwpC8Z+ARs= 2022-09-12T20:16:09.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RGP8wdj8lNJ8+PNDbZwpC8Z+ARs=', 'Content-Length': '0'} 2022-09-12T20:16:09.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f23fef1b3dd9fb12-00631f9389-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.468 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:09.469 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:09.469 DEBUG:boto:path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:09.470 DEBUG:boto:auth_path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:09.470 DEBUG:boto:Method: GET 2022-09-12T20:16:09.471 DEBUG:boto:Path: /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:09.472 DEBUG:boto:Data: 2022-09-12T20:16:09.472 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.473 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.473 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.474 DEBUG:boto:Params: {} 2022-09-12T20:16:09.475 DEBUG:boto:Token: None 2022-09-12T20:16:09.475 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:09.476 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RGP8wdj8lNJ8+PNDbZwpC8Z+ARs= 2022-09-12T20:16:09.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RGP8wdj8lNJ8+PNDbZwpC8Z+ARs=', 'Content-Length': '0'} 2022-09-12T20:16:09.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3dc5a169fcf49dd-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.480 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:09.480 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:09.481 DEBUG:boto:path=/jdpiak-75/?prefix=obj185 2022-09-12T20:16:09.481 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj185 2022-09-12T20:16:09.482 DEBUG:boto:Method: GET 2022-09-12T20:16:09.482 DEBUG:boto:Path: /jdpiak-75/?prefix=obj185 2022-09-12T20:16:09.483 DEBUG:boto:Data: 2022-09-12T20:16:09.483 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.484 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.485 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.485 DEBUG:boto:Params: {} 2022-09-12T20:16:09.486 DEBUG:boto:Token: None 2022-09-12T20:16:09.486 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/ 2022-09-12T20:16:09.487 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg= 2022-09-12T20:16:09.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg=', 'Content-Length': '0'} 2022-09-12T20:16:09.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fb319966fde1d3a-00631f9389-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.491 DEBUG:boto:b'jdpiak-75obj1851000falseobj1852022-09-12T20:14:27.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:09.492 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:09.492 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:09.493 DEBUG:boto:path=/jdpiak-75/?prefix=obj185 2022-09-12T20:16:09.494 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj185 2022-09-12T20:16:09.494 DEBUG:boto:Method: GET 2022-09-12T20:16:09.495 DEBUG:boto:Path: /jdpiak-75/?prefix=obj185 2022-09-12T20:16:09.495 DEBUG:boto:Data: 2022-09-12T20:16:09.496 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.496 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.497 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.497 DEBUG:boto:Params: {} 2022-09-12T20:16:09.498 DEBUG:boto:Token: None 2022-09-12T20:16:09.499 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/ 2022-09-12T20:16:09.499 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg= 2022-09-12T20:16:09.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg=', 'Content-Length': '0'} 2022-09-12T20:16:09.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dac05c25aadfadf-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.505 DEBUG:boto:b'jdpiak-75obj1851000falseobj1852022-09-12T20:14:27.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:09.506 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:09.507 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:09.507 DEBUG:boto:Method: HEAD 2022-09-12T20:16:09.508 DEBUG:boto:Path: /jdpiak-75/obj185 2022-09-12T20:16:09.508 DEBUG:boto:Data: 2022-09-12T20:16:09.509 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.509 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.510 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.510 DEBUG:boto:Params: {} 2022-09-12T20:16:09.511 DEBUG:boto:Token: None 2022-09-12T20:16:09.511 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj185 2022-09-12T20:16:09.512 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QqzWyqNLckb3TJPubDmbYUBAjdY= 2022-09-12T20:16:09.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QqzWyqNLckb3TJPubDmbYUBAjdY=', 'Content-Length': '0'} 2022-09-12T20:16:09.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010b6c8f0dda9bb75-00631f9389-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.515 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:09.516 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:09.516 DEBUG:boto:Method: HEAD 2022-09-12T20:16:09.517 DEBUG:boto:Path: /jdpiak-75/obj185 2022-09-12T20:16:09.517 DEBUG:boto:Data: 2022-09-12T20:16:09.518 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.519 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.519 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.520 DEBUG:boto:Params: {} 2022-09-12T20:16:09.520 DEBUG:boto:Token: None 2022-09-12T20:16:09.521 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj185 2022-09-12T20:16:09.521 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QqzWyqNLckb3TJPubDmbYUBAjdY= 2022-09-12T20:16:09.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QqzWyqNLckb3TJPubDmbYUBAjdY=', 'Content-Length': '0'} 2022-09-12T20:16:09.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceddf98ae2dfa7dc-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2022-09-12T20:16:09.525 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:09.526 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:09.527 DEBUG:boto:Method: GET 2022-09-12T20:16:09.527 DEBUG:boto:Path: /jdpiak-75/obj185 2022-09-12T20:16:09.528 DEBUG:boto:Data: 2022-09-12T20:16:09.528 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.529 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.529 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.530 DEBUG:boto:Params: {} 2022-09-12T20:16:09.531 DEBUG:boto:Token: None 2022-09-12T20:16:09.531 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj185 2022-09-12T20:16:09.532 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E3nUFy6hpi/qXf2cFaOs0HLk5MM= 2022-09-12T20:16:09.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E3nUFy6hpi/qXf2cFaOs0HLk5MM=', 'Content-Length': '0'} 2022-09-12T20:16:09.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fb070f5b2d05454-00631f9389-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.582 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:09.583 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:09.583 DEBUG:boto:Method: GET 2022-09-12T20:16:09.584 DEBUG:boto:Path: /jdpiak-75/obj185 2022-09-12T20:16:09.584 DEBUG:boto:Data: 2022-09-12T20:16:09.585 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.585 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.586 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.586 DEBUG:boto:Params: {} 2022-09-12T20:16:09.587 DEBUG:boto:Token: None 2022-09-12T20:16:09.588 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj185 2022-09-12T20:16:09.588 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E3nUFy6hpi/qXf2cFaOs0HLk5MM= 2022-09-12T20:16:09.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E3nUFy6hpi/qXf2cFaOs0HLk5MM=', 'Content-Length': '0'} 2022-09-12T20:16:09.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a78fab36f43a55f-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.592 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:09.593 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:09.593 DEBUG:boto:path=/jdpiak-75/?prefix=obj185 2022-09-12T20:16:09.594 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj185 2022-09-12T20:16:09.594 DEBUG:boto:Method: GET 2022-09-12T20:16:09.595 DEBUG:boto:Path: /jdpiak-75/?prefix=obj185 2022-09-12T20:16:09.595 DEBUG:boto:Data: 2022-09-12T20:16:09.596 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.596 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.597 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.598 DEBUG:boto:Params: {} 2022-09-12T20:16:09.598 DEBUG:boto:Token: None 2022-09-12T20:16:09.599 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/ 2022-09-12T20:16:09.599 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg= 2022-09-12T20:16:09.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg=', 'Content-Length': '0'} 2022-09-12T20:16:09.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090d4c5363e6e455b-00631f9389-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.603 DEBUG:boto:b'jdpiak-75obj1851000falseobj1852022-09-12T20:14:27.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:09.604 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:09.605 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:09.605 DEBUG:boto:path=/jdpiak-75/?prefix=obj185 2022-09-12T20:16:09.606 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj185 2022-09-12T20:16:09.607 DEBUG:boto:Method: GET 2022-09-12T20:16:09.607 DEBUG:boto:Path: /jdpiak-75/?prefix=obj185 2022-09-12T20:16:09.608 DEBUG:boto:Data: 2022-09-12T20:16:09.608 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.609 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.609 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.610 DEBUG:boto:Params: {} 2022-09-12T20:16:09.610 DEBUG:boto:Token: None 2022-09-12T20:16:09.611 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/ 2022-09-12T20:16:09.612 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg= 2022-09-12T20:16:09.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg=', 'Content-Length': '0'} 2022-09-12T20:16:09.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccd34c585528bdab-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.618 DEBUG:boto:b'jdpiak-75obj1851000falseobj1852022-09-12T20:14:27.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:09.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:16:09.619 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:09.619 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:09.620 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:09.621 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:09.621 DEBUG:boto:Method: GET 2022-09-12T20:16:09.622 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:09.622 DEBUG:boto:Data: 2022-09-12T20:16:09.623 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.623 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.624 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.625 DEBUG:boto:Params: {} 2022-09-12T20:16:09.625 DEBUG:boto:Token: None 2022-09-12T20:16:09.626 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:09.626 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LqEeWM76Wyvnsk+UC/HOyCCN0S4= 2022-09-12T20:16:09.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LqEeWM76Wyvnsk+UC/HOyCCN0S4=', 'Content-Length': '0'} 2022-09-12T20:16:09.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c420ba977118751-00631f9389-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.630 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:09.631 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:09.631 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:09.632 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:09.632 DEBUG:boto:Method: GET 2022-09-12T20:16:09.633 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:09.633 DEBUG:boto:Data: 2022-09-12T20:16:09.634 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.635 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.635 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.636 DEBUG:boto:Params: {} 2022-09-12T20:16:09.636 DEBUG:boto:Token: None 2022-09-12T20:16:09.637 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:09.638 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LqEeWM76Wyvnsk+UC/HOyCCN0S4= 2022-09-12T20:16:09.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LqEeWM76Wyvnsk+UC/HOyCCN0S4=', 'Content-Length': '0'} 2022-09-12T20:16:09.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006055decac9d5b117-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.642 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:09.642 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:09.643 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:09.643 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:09.644 DEBUG:boto:Method: HEAD 2022-09-12T20:16:09.644 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:09.645 DEBUG:boto:Data: 2022-09-12T20:16:09.645 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.646 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.646 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.647 DEBUG:boto:Params: {} 2022-09-12T20:16:09.647 DEBUG:boto:Token: None 2022-09-12T20:16:09.648 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:09.648 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m39+hwaF1tEo2zBMqVs7WYUD6UA= 2022-09-12T20:16:09.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m39+hwaF1tEo2zBMqVs7WYUD6UA=', 'Content-Length': '0'} 2022-09-12T20:16:09.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d34d20010baf2c84-00631f9389-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.652 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:09.652 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:09.653 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:09.653 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:09.654 DEBUG:boto:Method: HEAD 2022-09-12T20:16:09.655 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:09.655 DEBUG:boto:Data: 2022-09-12T20:16:09.656 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.656 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.657 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.657 DEBUG:boto:Params: {} 2022-09-12T20:16:09.658 DEBUG:boto:Token: None 2022-09-12T20:16:09.658 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:09.659 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m39+hwaF1tEo2zBMqVs7WYUD6UA= 2022-09-12T20:16:09.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m39+hwaF1tEo2zBMqVs7WYUD6UA=', 'Content-Length': '0'} 2022-09-12T20:16:09.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adae9045084bc249-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:16:09.663 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:09.663 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:09.664 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:09.664 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:09.665 DEBUG:boto:Method: GET 2022-09-12T20:16:09.666 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:09.666 DEBUG:boto:Data: 2022-09-12T20:16:09.667 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.667 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.668 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.668 DEBUG:boto:Params: {} 2022-09-12T20:16:09.669 DEBUG:boto:Token: None 2022-09-12T20:16:09.670 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:09.670 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LqEeWM76Wyvnsk+UC/HOyCCN0S4= 2022-09-12T20:16:09.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LqEeWM76Wyvnsk+UC/HOyCCN0S4=', 'Content-Length': '0'} 2022-09-12T20:16:09.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a87ff97431ac528-00631f9389-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.674 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:09.675 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:09.675 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:09.676 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:09.676 DEBUG:boto:Method: GET 2022-09-12T20:16:09.677 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:09.678 DEBUG:boto:Data: 2022-09-12T20:16:09.678 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.679 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.679 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.680 DEBUG:boto:Params: {} 2022-09-12T20:16:09.680 DEBUG:boto:Token: None 2022-09-12T20:16:09.681 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:09.682 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LqEeWM76Wyvnsk+UC/HOyCCN0S4= 2022-09-12T20:16:09.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LqEeWM76Wyvnsk+UC/HOyCCN0S4=', 'Content-Length': '0'} 2022-09-12T20:16:09.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c2add62b3af1d2a-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.685 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:09.686 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:09.687 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:16:09.687 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:16:09.688 DEBUG:boto:Method: GET 2022-09-12T20:16:09.688 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:16:09.689 DEBUG:boto:Data: 2022-09-12T20:16:09.689 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.690 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.690 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.691 DEBUG:boto:Params: {} 2022-09-12T20:16:09.692 DEBUG:boto:Token: None 2022-09-12T20:16:09.692 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/ 2022-09-12T20:16:09.693 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg= 2022-09-12T20:16:09.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg=', 'Content-Length': '0'} 2022-09-12T20:16:09.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9d8537e0b44fbf3-00631f9389-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.697 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:09.697 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:09.698 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:09.699 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:16:09.699 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:16:09.700 DEBUG:boto:Method: GET 2022-09-12T20:16:09.700 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:16:09.701 DEBUG:boto:Data: 2022-09-12T20:16:09.701 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.702 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.702 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.703 DEBUG:boto:Params: {} 2022-09-12T20:16:09.703 DEBUG:boto:Token: None 2022-09-12T20:16:09.704 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/ 2022-09-12T20:16:09.705 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg= 2022-09-12T20:16:09.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg=', 'Content-Length': '0'} 2022-09-12T20:16:09.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ec61686e2cb51e4-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.711 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:09.712 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:09.713 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:09.713 DEBUG:boto:Method: HEAD 2022-09-12T20:16:09.714 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:16:09.714 DEBUG:boto:Data: 2022-09-12T20:16:09.715 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.715 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.716 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.716 DEBUG:boto:Params: {} 2022-09-12T20:16:09.717 DEBUG:boto:Token: None 2022-09-12T20:16:09.717 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj19 2022-09-12T20:16:09.718 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fFgllxIzpdnDOXoiThLWe+/2d/A= 2022-09-12T20:16:09.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fFgllxIzpdnDOXoiThLWe+/2d/A=', 'Content-Length': '0'} 2022-09-12T20:16:09.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a5c5945e060fcec-00631f9389-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.721 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:09.722 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:09.723 DEBUG:boto:Method: HEAD 2022-09-12T20:16:09.723 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:16:09.724 DEBUG:boto:Data: 2022-09-12T20:16:09.724 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.725 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.725 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.726 DEBUG:boto:Params: {} 2022-09-12T20:16:09.726 DEBUG:boto:Token: None 2022-09-12T20:16:09.727 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj19 2022-09-12T20:16:09.727 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fFgllxIzpdnDOXoiThLWe+/2d/A= 2022-09-12T20:16:09.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fFgllxIzpdnDOXoiThLWe+/2d/A=', 'Content-Length': '0'} 2022-09-12T20:16:09.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066a84c0fcb939c85-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:16:09.731 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:09.732 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:09.732 DEBUG:boto:Method: GET 2022-09-12T20:16:09.733 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:16:09.733 DEBUG:boto:Data: 2022-09-12T20:16:09.734 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.735 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.735 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.736 DEBUG:boto:Params: {} 2022-09-12T20:16:09.736 DEBUG:boto:Token: None 2022-09-12T20:16:09.737 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj19 2022-09-12T20:16:09.737 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PKGGvBpRVBww2L8PDrRDiQdPgV0= 2022-09-12T20:16:09.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PKGGvBpRVBww2L8PDrRDiQdPgV0=', 'Content-Length': '0'} 2022-09-12T20:16:09.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cd3d01f78bc0fe3-00631f9389-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.741 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:09.741 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:09.742 DEBUG:boto:Method: GET 2022-09-12T20:16:09.743 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:16:09.743 DEBUG:boto:Data: 2022-09-12T20:16:09.744 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.744 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.745 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.745 DEBUG:boto:Params: {} 2022-09-12T20:16:09.746 DEBUG:boto:Token: None 2022-09-12T20:16:09.747 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj19 2022-09-12T20:16:09.747 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PKGGvBpRVBww2L8PDrRDiQdPgV0= 2022-09-12T20:16:09.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PKGGvBpRVBww2L8PDrRDiQdPgV0=', 'Content-Length': '0'} 2022-09-12T20:16:09.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099384c399949ddd3-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.790 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:09.791 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:09.792 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:16:09.794 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:16:09.794 DEBUG:boto:Method: GET 2022-09-12T20:16:09.795 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:16:09.796 DEBUG:boto:Data: 2022-09-12T20:16:09.797 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.797 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.798 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.799 DEBUG:boto:Params: {} 2022-09-12T20:16:09.799 DEBUG:boto:Token: None 2022-09-12T20:16:09.800 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/ 2022-09-12T20:16:09.801 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg= 2022-09-12T20:16:09.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg=', 'Content-Length': '0'} 2022-09-12T20:16:09.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad44dd87d347603b-00631f9389-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.805 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:09.806 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:09.806 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:09.807 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:16:09.807 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:16:09.808 DEBUG:boto:Method: GET 2022-09-12T20:16:09.808 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:16:09.809 DEBUG:boto:Data: 2022-09-12T20:16:09.809 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.810 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.811 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.811 DEBUG:boto:Params: {} 2022-09-12T20:16:09.812 DEBUG:boto:Token: None 2022-09-12T20:16:09.812 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/ 2022-09-12T20:16:09.813 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg= 2022-09-12T20:16:09.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg=', 'Content-Length': '0'} 2022-09-12T20:16:09.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d66a393e560542d-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.819 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:09.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:16:09.821 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:09.821 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:09.822 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:09.822 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:09.823 DEBUG:boto:Method: GET 2022-09-12T20:16:09.823 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:09.824 DEBUG:boto:Data: 2022-09-12T20:16:09.824 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.825 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.825 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.826 DEBUG:boto:Params: {} 2022-09-12T20:16:09.827 DEBUG:boto:Token: None 2022-09-12T20:16:09.827 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:09.828 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ygskohvNUuGPYI2l2cHYOvrfZLU= 2022-09-12T20:16:09.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ygskohvNUuGPYI2l2cHYOvrfZLU=', 'Content-Length': '0'} 2022-09-12T20:16:09.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000621100580c18ce45-00631f9389-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.832 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:09.832 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:09.833 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:09.833 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:09.834 DEBUG:boto:Method: GET 2022-09-12T20:16:09.834 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:09.835 DEBUG:boto:Data: 2022-09-12T20:16:09.835 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.836 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.837 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.837 DEBUG:boto:Params: {} 2022-09-12T20:16:09.838 DEBUG:boto:Token: None 2022-09-12T20:16:09.838 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:09.839 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ygskohvNUuGPYI2l2cHYOvrfZLU= 2022-09-12T20:16:09.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ygskohvNUuGPYI2l2cHYOvrfZLU=', 'Content-Length': '0'} 2022-09-12T20:16:09.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f8ede471b0ae721-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.843 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:09.844 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:09.844 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:09.845 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:09.846 DEBUG:boto:Method: HEAD 2022-09-12T20:16:09.846 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:09.847 DEBUG:boto:Data: 2022-09-12T20:16:09.847 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.848 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.848 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.849 DEBUG:boto:Params: {} 2022-09-12T20:16:09.849 DEBUG:boto:Token: None 2022-09-12T20:16:09.850 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:09.850 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yLUnCU7IvJFw+XuQtiDIZx2+YWg= 2022-09-12T20:16:09.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yLUnCU7IvJFw+XuQtiDIZx2+YWg=', 'Content-Length': '0'} 2022-09-12T20:16:09.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005b18454fc08070f-00631f9389-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.854 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:09.854 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:09.855 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:09.855 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:09.856 DEBUG:boto:Method: HEAD 2022-09-12T20:16:09.856 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:09.857 DEBUG:boto:Data: 2022-09-12T20:16:09.857 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.858 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.858 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.859 DEBUG:boto:Params: {} 2022-09-12T20:16:09.859 DEBUG:boto:Token: None 2022-09-12T20:16:09.860 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:09.861 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yLUnCU7IvJFw+XuQtiDIZx2+YWg= 2022-09-12T20:16:09.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yLUnCU7IvJFw+XuQtiDIZx2+YWg=', 'Content-Length': '0'} 2022-09-12T20:16:09.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd2161f000c792af-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:16:09.865 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:09.865 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:09.866 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:09.866 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:09.867 DEBUG:boto:Method: GET 2022-09-12T20:16:09.867 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:09.868 DEBUG:boto:Data: 2022-09-12T20:16:09.868 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.869 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.870 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.870 DEBUG:boto:Params: {} 2022-09-12T20:16:09.871 DEBUG:boto:Token: None 2022-09-12T20:16:09.871 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:09.872 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ygskohvNUuGPYI2l2cHYOvrfZLU= 2022-09-12T20:16:09.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ygskohvNUuGPYI2l2cHYOvrfZLU=', 'Content-Length': '0'} 2022-09-12T20:16:09.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a57b641df9ec156-00631f9389-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.918 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:09.919 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:09.920 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:09.921 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:09.922 DEBUG:boto:Method: GET 2022-09-12T20:16:09.923 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:09.923 DEBUG:boto:Data: 2022-09-12T20:16:09.924 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.925 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.926 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.927 DEBUG:boto:Params: {} 2022-09-12T20:16:09.928 DEBUG:boto:Token: None 2022-09-12T20:16:09.929 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:09.930 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ygskohvNUuGPYI2l2cHYOvrfZLU= 2022-09-12T20:16:09.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ygskohvNUuGPYI2l2cHYOvrfZLU=', 'Content-Length': '0'} 2022-09-12T20:16:09.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a3056e70e731959-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.935 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:09.935 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:09.936 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:16:09.937 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:16:09.938 DEBUG:boto:Method: GET 2022-09-12T20:16:09.939 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:16:09.940 DEBUG:boto:Data: 2022-09-12T20:16:09.941 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.941 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.942 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.943 DEBUG:boto:Params: {} 2022-09-12T20:16:09.944 DEBUG:boto:Token: None 2022-09-12T20:16:09.945 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/ 2022-09-12T20:16:09.946 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg= 2022-09-12T20:16:09.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg=', 'Content-Length': '0'} 2022-09-12T20:16:09.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d77b64dcee4bac66-00631f9389-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.951 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:09.953 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:09.954 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:09.955 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:16:09.955 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:16:09.956 DEBUG:boto:Method: GET 2022-09-12T20:16:09.957 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:16:09.958 DEBUG:boto:Data: 2022-09-12T20:16:09.959 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.960 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.961 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.962 DEBUG:boto:Params: {} 2022-09-12T20:16:09.963 DEBUG:boto:Token: None 2022-09-12T20:16:09.963 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/ 2022-09-12T20:16:09.964 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg= 2022-09-12T20:16:09.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WQ0BDcMrucvA/mbX9kCtVNFHLNg=', 'Content-Length': '0'} 2022-09-12T20:16:09.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ea379fd02ae1b35-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.970 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:09.970 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:09.971 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:09.971 DEBUG:boto:Method: HEAD 2022-09-12T20:16:09.972 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:16:09.972 DEBUG:boto:Data: 2022-09-12T20:16:09.973 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.973 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.974 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.974 DEBUG:boto:Params: {} 2022-09-12T20:16:09.975 DEBUG:boto:Token: None 2022-09-12T20:16:09.976 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj22 2022-09-12T20:16:09.976 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x5VoijrGTssmULJhmsXDkEcqHBk= 2022-09-12T20:16:09.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x5VoijrGTssmULJhmsXDkEcqHBk=', 'Content-Length': '0'} 2022-09-12T20:16:09.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a551be049de5d77-00631f9389-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.980 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:09.980 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:09.981 DEBUG:boto:Method: HEAD 2022-09-12T20:16:09.981 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:16:09.982 DEBUG:boto:Data: 2022-09-12T20:16:09.982 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.983 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:09.983 DEBUG:boto:Port: 8000 2022-09-12T20:16:09.984 DEBUG:boto:Params: {} 2022-09-12T20:16:09.985 DEBUG:boto:Token: None 2022-09-12T20:16:09.986 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj22 2022-09-12T20:16:09.987 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x5VoijrGTssmULJhmsXDkEcqHBk= 2022-09-12T20:16:09.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x5VoijrGTssmULJhmsXDkEcqHBk=', 'Content-Length': '0'} 2022-09-12T20:16:09.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c33294c14ecd6a9-00631f9389-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:09.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:16:09.991 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:09.992 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:09.992 DEBUG:boto:Method: GET 2022-09-12T20:16:09.993 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:16:09.993 DEBUG:boto:Data: 2022-09-12T20:16:09.994 DEBUG:boto:Headers: {} 2022-09-12T20:16:09.995 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:09.995 DEBUG:boto:Port: 8001 2022-09-12T20:16:09.996 DEBUG:boto:Params: {} 2022-09-12T20:16:09.997 DEBUG:boto:Token: None 2022-09-12T20:16:09.998 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:09 GMT /jdpiak-75/obj22 2022-09-12T20:16:09.999 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+YcMkFHJjS6XFSsETEq50S/IXbc= 2022-09-12T20:16:09.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:09 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+YcMkFHJjS6XFSsETEq50S/IXbc=', 'Content-Length': '0'} 2022-09-12T20:16:10.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a687192fc4f34e3b-00631f938a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.002 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:10.003 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:10.003 DEBUG:boto:Method: GET 2022-09-12T20:16:10.004 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:16:10.004 DEBUG:boto:Data: 2022-09-12T20:16:10.005 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.005 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.006 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.006 DEBUG:boto:Params: {} 2022-09-12T20:16:10.007 DEBUG:boto:Token: None 2022-09-12T20:16:10.008 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj22 2022-09-12T20:16:10.008 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FNZjOD1Es0boifyMH+nAhPIlmlI= 2022-09-12T20:16:10.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FNZjOD1Es0boifyMH+nAhPIlmlI=', 'Content-Length': '0'} 2022-09-12T20:16:10.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018f4e0f905e8ce39-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.012 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:10.013 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:10.013 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:16:10.014 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:16:10.014 DEBUG:boto:Method: GET 2022-09-12T20:16:10.015 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:16:10.016 DEBUG:boto:Data: 2022-09-12T20:16:10.016 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.017 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.018 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.018 DEBUG:boto:Params: {} 2022-09-12T20:16:10.019 DEBUG:boto:Token: None 2022-09-12T20:16:10.019 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/ 2022-09-12T20:16:10.020 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE= 2022-09-12T20:16:10.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE=', 'Content-Length': '0'} 2022-09-12T20:16:10.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de7590cbf717b3a4-00631f938a-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.025 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:10.026 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:10.026 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:10.027 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:16:10.028 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:16:10.028 DEBUG:boto:Method: GET 2022-09-12T20:16:10.029 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:16:10.029 DEBUG:boto:Data: 2022-09-12T20:16:10.030 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.030 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.031 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.031 DEBUG:boto:Params: {} 2022-09-12T20:16:10.032 DEBUG:boto:Token: None 2022-09-12T20:16:10.032 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/ 2022-09-12T20:16:10.033 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE= 2022-09-12T20:16:10.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE=', 'Content-Length': '0'} 2022-09-12T20:16:10.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054e218d0bc0f7d14-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.040 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:10.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:16:10.041 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:10.041 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:10.042 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:10.042 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:10.043 DEBUG:boto:Method: GET 2022-09-12T20:16:10.044 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:10.044 DEBUG:boto:Data: 2022-09-12T20:16:10.045 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.045 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.046 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.046 DEBUG:boto:Params: {} 2022-09-12T20:16:10.047 DEBUG:boto:Token: None 2022-09-12T20:16:10.047 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:10.048 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TWdUBbTy/i8gmh0eRmvroF8/fh0= 2022-09-12T20:16:10.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TWdUBbTy/i8gmh0eRmvroF8/fh0=', 'Content-Length': '0'} 2022-09-12T20:16:10.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cd786ec20586519-00631f938a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.052 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:10.052 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:10.053 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:10.053 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:10.054 DEBUG:boto:Method: GET 2022-09-12T20:16:10.054 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:10.055 DEBUG:boto:Data: 2022-09-12T20:16:10.055 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.056 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.057 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.057 DEBUG:boto:Params: {} 2022-09-12T20:16:10.058 DEBUG:boto:Token: None 2022-09-12T20:16:10.059 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:10.059 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TWdUBbTy/i8gmh0eRmvroF8/fh0= 2022-09-12T20:16:10.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TWdUBbTy/i8gmh0eRmvroF8/fh0=', 'Content-Length': '0'} 2022-09-12T20:16:10.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b6ca17184041d16-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.106 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:10.107 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:10.108 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:10.109 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:10.110 DEBUG:boto:Method: HEAD 2022-09-12T20:16:10.111 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:10.111 DEBUG:boto:Data: 2022-09-12T20:16:10.112 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.112 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.113 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.113 DEBUG:boto:Params: {} 2022-09-12T20:16:10.114 DEBUG:boto:Token: None 2022-09-12T20:16:10.114 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:10.115 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+tlLZzQlBw30sIle2qxB9wHXnvY= 2022-09-12T20:16:10.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+tlLZzQlBw30sIle2qxB9wHXnvY=', 'Content-Length': '0'} 2022-09-12T20:16:10.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f59f9de9ae0bfdd-00631f938a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.119 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:10.120 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:10.121 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:10.122 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:10.122 DEBUG:boto:Method: HEAD 2022-09-12T20:16:10.123 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:10.123 DEBUG:boto:Data: 2022-09-12T20:16:10.124 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.124 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.125 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.125 DEBUG:boto:Params: {} 2022-09-12T20:16:10.126 DEBUG:boto:Token: None 2022-09-12T20:16:10.127 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:10.127 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+tlLZzQlBw30sIle2qxB9wHXnvY= 2022-09-12T20:16:10.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+tlLZzQlBw30sIle2qxB9wHXnvY=', 'Content-Length': '0'} 2022-09-12T20:16:10.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c77a0f571f25513f-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:16:10.132 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:10.132 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:10.133 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:10.133 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:10.134 DEBUG:boto:Method: GET 2022-09-12T20:16:10.134 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:10.135 DEBUG:boto:Data: 2022-09-12T20:16:10.135 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.136 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.136 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.137 DEBUG:boto:Params: {} 2022-09-12T20:16:10.138 DEBUG:boto:Token: None 2022-09-12T20:16:10.138 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:10.139 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TWdUBbTy/i8gmh0eRmvroF8/fh0= 2022-09-12T20:16:10.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TWdUBbTy/i8gmh0eRmvroF8/fh0=', 'Content-Length': '0'} 2022-09-12T20:16:10.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c246f818797b9859-00631f938a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.143 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:10.144 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:10.144 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:10.145 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:10.145 DEBUG:boto:Method: GET 2022-09-12T20:16:10.146 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:10.147 DEBUG:boto:Data: 2022-09-12T20:16:10.147 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.148 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.148 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.149 DEBUG:boto:Params: {} 2022-09-12T20:16:10.150 DEBUG:boto:Token: None 2022-09-12T20:16:10.151 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:10.152 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TWdUBbTy/i8gmh0eRmvroF8/fh0= 2022-09-12T20:16:10.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TWdUBbTy/i8gmh0eRmvroF8/fh0=', 'Content-Length': '0'} 2022-09-12T20:16:10.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d451593d0045f0c-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.155 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:10.156 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:10.157 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:16:10.159 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:16:10.160 DEBUG:boto:Method: GET 2022-09-12T20:16:10.161 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:16:10.162 DEBUG:boto:Data: 2022-09-12T20:16:10.163 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.164 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.165 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.165 DEBUG:boto:Params: {} 2022-09-12T20:16:10.167 DEBUG:boto:Token: None 2022-09-12T20:16:10.168 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/ 2022-09-12T20:16:10.168 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE= 2022-09-12T20:16:10.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE=', 'Content-Length': '0'} 2022-09-12T20:16:10.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb1e06fbd444bc26-00631f938a-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.218 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:10.219 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:10.220 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:10.221 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:16:10.221 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:16:10.222 DEBUG:boto:Method: GET 2022-09-12T20:16:10.223 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:16:10.224 DEBUG:boto:Data: 2022-09-12T20:16:10.224 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.225 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.225 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.226 DEBUG:boto:Params: {} 2022-09-12T20:16:10.227 DEBUG:boto:Token: None 2022-09-12T20:16:10.227 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/ 2022-09-12T20:16:10.228 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE= 2022-09-12T20:16:10.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE=', 'Content-Length': '0'} 2022-09-12T20:16:10.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5159227a209e0f9-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.235 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:10.236 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:10.237 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:10.237 DEBUG:boto:Method: HEAD 2022-09-12T20:16:10.238 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:16:10.238 DEBUG:boto:Data: 2022-09-12T20:16:10.239 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.239 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.240 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.240 DEBUG:boto:Params: {} 2022-09-12T20:16:10.241 DEBUG:boto:Token: None 2022-09-12T20:16:10.242 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj23 2022-09-12T20:16:10.242 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xtZwFeloICL027xWpBpKg/PcmD0= 2022-09-12T20:16:10.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xtZwFeloICL027xWpBpKg/PcmD0=', 'Content-Length': '0'} 2022-09-12T20:16:10.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0cabd46b9787bff-00631f938a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.245 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:10.246 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:10.247 DEBUG:boto:Method: HEAD 2022-09-12T20:16:10.247 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:16:10.248 DEBUG:boto:Data: 2022-09-12T20:16:10.248 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.249 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.249 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.250 DEBUG:boto:Params: {} 2022-09-12T20:16:10.250 DEBUG:boto:Token: None 2022-09-12T20:16:10.251 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj23 2022-09-12T20:16:10.252 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xtZwFeloICL027xWpBpKg/PcmD0= 2022-09-12T20:16:10.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xtZwFeloICL027xWpBpKg/PcmD0=', 'Content-Length': '0'} 2022-09-12T20:16:10.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ee661e712b38dd4-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:16:10.256 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:10.256 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:10.257 DEBUG:boto:Method: GET 2022-09-12T20:16:10.257 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:16:10.258 DEBUG:boto:Data: 2022-09-12T20:16:10.259 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.259 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.260 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.260 DEBUG:boto:Params: {} 2022-09-12T20:16:10.261 DEBUG:boto:Token: None 2022-09-12T20:16:10.262 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj23 2022-09-12T20:16:10.262 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kgfANHQgdWS9WPJjVndsttsV8kg= 2022-09-12T20:16:10.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kgfANHQgdWS9WPJjVndsttsV8kg=', 'Content-Length': '0'} 2022-09-12T20:16:10.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000524ed59722175dab-00631f938a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.266 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:10.267 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:10.267 DEBUG:boto:Method: GET 2022-09-12T20:16:10.268 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:16:10.268 DEBUG:boto:Data: 2022-09-12T20:16:10.269 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.270 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.270 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.271 DEBUG:boto:Params: {} 2022-09-12T20:16:10.271 DEBUG:boto:Token: None 2022-09-12T20:16:10.272 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj23 2022-09-12T20:16:10.273 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kgfANHQgdWS9WPJjVndsttsV8kg= 2022-09-12T20:16:10.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kgfANHQgdWS9WPJjVndsttsV8kg=', 'Content-Length': '0'} 2022-09-12T20:16:10.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cb79b7ac1a341a9-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.318 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:10.319 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:10.319 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:16:10.320 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:16:10.320 DEBUG:boto:Method: GET 2022-09-12T20:16:10.321 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:16:10.321 DEBUG:boto:Data: 2022-09-12T20:16:10.322 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.323 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.323 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.324 DEBUG:boto:Params: {} 2022-09-12T20:16:10.324 DEBUG:boto:Token: None 2022-09-12T20:16:10.325 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/ 2022-09-12T20:16:10.325 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE= 2022-09-12T20:16:10.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE=', 'Content-Length': '0'} 2022-09-12T20:16:10.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ae86407d25e7691-00631f938a-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.329 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:10.330 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:10.331 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:10.331 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:16:10.332 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:16:10.333 DEBUG:boto:Method: GET 2022-09-12T20:16:10.333 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:16:10.334 DEBUG:boto:Data: 2022-09-12T20:16:10.334 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.335 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.335 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.336 DEBUG:boto:Params: {} 2022-09-12T20:16:10.336 DEBUG:boto:Token: None 2022-09-12T20:16:10.337 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/ 2022-09-12T20:16:10.338 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE= 2022-09-12T20:16:10.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE=', 'Content-Length': '0'} 2022-09-12T20:16:10.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9d8bc8681231461-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.343 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:10.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:16:10.345 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:10.345 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:10.346 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:10.347 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:10.347 DEBUG:boto:Method: GET 2022-09-12T20:16:10.348 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:10.348 DEBUG:boto:Data: 2022-09-12T20:16:10.349 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.349 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.350 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.350 DEBUG:boto:Params: {} 2022-09-12T20:16:10.351 DEBUG:boto:Token: None 2022-09-12T20:16:10.351 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:10.352 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BJteISQs1Vp1GhxYyvu7IjHNnl0= 2022-09-12T20:16:10.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BJteISQs1Vp1GhxYyvu7IjHNnl0=', 'Content-Length': '0'} 2022-09-12T20:16:10.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060e0f18a9ca15d26-00631f938a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.355 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:10.356 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:10.357 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:10.357 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:10.358 DEBUG:boto:Method: GET 2022-09-12T20:16:10.358 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:10.359 DEBUG:boto:Data: 2022-09-12T20:16:10.360 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.360 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.361 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.361 DEBUG:boto:Params: {} 2022-09-12T20:16:10.362 DEBUG:boto:Token: None 2022-09-12T20:16:10.363 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:10.363 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BJteISQs1Vp1GhxYyvu7IjHNnl0= 2022-09-12T20:16:10.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BJteISQs1Vp1GhxYyvu7IjHNnl0=', 'Content-Length': '0'} 2022-09-12T20:16:10.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c5dc31a4df26ae9-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.367 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:10.367 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:10.368 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:10.368 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:10.369 DEBUG:boto:Method: HEAD 2022-09-12T20:16:10.369 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:10.370 DEBUG:boto:Data: 2022-09-12T20:16:10.370 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.371 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.371 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.372 DEBUG:boto:Params: {} 2022-09-12T20:16:10.372 DEBUG:boto:Token: None 2022-09-12T20:16:10.373 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:10.374 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pwbOYmTmaAB9apWGyXw5q54hK70= 2022-09-12T20:16:10.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pwbOYmTmaAB9apWGyXw5q54hK70=', 'Content-Length': '0'} 2022-09-12T20:16:10.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000761e37e61b589be2-00631f938a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.377 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:10.377 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:10.378 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:10.378 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:10.379 DEBUG:boto:Method: HEAD 2022-09-12T20:16:10.379 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:10.380 DEBUG:boto:Data: 2022-09-12T20:16:10.380 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.381 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.381 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.382 DEBUG:boto:Params: {} 2022-09-12T20:16:10.382 DEBUG:boto:Token: None 2022-09-12T20:16:10.383 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:10.384 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pwbOYmTmaAB9apWGyXw5q54hK70= 2022-09-12T20:16:10.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pwbOYmTmaAB9apWGyXw5q54hK70=', 'Content-Length': '0'} 2022-09-12T20:16:10.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc1e19abe7d9d194-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:16:10.388 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:10.388 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:10.389 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:10.389 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:10.390 DEBUG:boto:Method: GET 2022-09-12T20:16:10.390 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:10.391 DEBUG:boto:Data: 2022-09-12T20:16:10.391 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.392 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.392 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.393 DEBUG:boto:Params: {} 2022-09-12T20:16:10.393 DEBUG:boto:Token: None 2022-09-12T20:16:10.394 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:10.394 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BJteISQs1Vp1GhxYyvu7IjHNnl0= 2022-09-12T20:16:10.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BJteISQs1Vp1GhxYyvu7IjHNnl0=', 'Content-Length': '0'} 2022-09-12T20:16:10.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ccaf6462eb1a310-00631f938a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.398 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:10.398 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:10.399 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:10.399 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:10.400 DEBUG:boto:Method: GET 2022-09-12T20:16:10.400 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:10.401 DEBUG:boto:Data: 2022-09-12T20:16:10.401 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.402 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.402 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.403 DEBUG:boto:Params: {} 2022-09-12T20:16:10.404 DEBUG:boto:Token: None 2022-09-12T20:16:10.404 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:10.405 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BJteISQs1Vp1GhxYyvu7IjHNnl0= 2022-09-12T20:16:10.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BJteISQs1Vp1GhxYyvu7IjHNnl0=', 'Content-Length': '0'} 2022-09-12T20:16:10.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d72bdaf3a3bb93bb-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.450 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:10.451 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:10.451 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:16:10.452 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:16:10.452 DEBUG:boto:Method: GET 2022-09-12T20:16:10.453 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:16:10.454 DEBUG:boto:Data: 2022-09-12T20:16:10.454 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.455 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.455 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.456 DEBUG:boto:Params: {} 2022-09-12T20:16:10.457 DEBUG:boto:Token: None 2022-09-12T20:16:10.457 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/ 2022-09-12T20:16:10.458 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE= 2022-09-12T20:16:10.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE=', 'Content-Length': '0'} 2022-09-12T20:16:10.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d282333279d90c83-00631f938a-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.462 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:10.463 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:10.464 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:10.465 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:16:10.465 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:16:10.466 DEBUG:boto:Method: GET 2022-09-12T20:16:10.467 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:16:10.467 DEBUG:boto:Data: 2022-09-12T20:16:10.468 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.468 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.469 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.469 DEBUG:boto:Params: {} 2022-09-12T20:16:10.470 DEBUG:boto:Token: None 2022-09-12T20:16:10.471 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/ 2022-09-12T20:16:10.472 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE= 2022-09-12T20:16:10.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE=', 'Content-Length': '0'} 2022-09-12T20:16:10.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d46c4bfed23e1a6-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.478 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:10.479 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:10.480 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:10.480 DEBUG:boto:Method: HEAD 2022-09-12T20:16:10.481 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:16:10.481 DEBUG:boto:Data: 2022-09-12T20:16:10.482 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.482 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.483 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.483 DEBUG:boto:Params: {} 2022-09-12T20:16:10.484 DEBUG:boto:Token: None 2022-09-12T20:16:10.484 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj29 2022-09-12T20:16:10.485 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F/DrCynls/Rc1p7F/4qpWFhxi0Q= 2022-09-12T20:16:10.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F/DrCynls/Rc1p7F/4qpWFhxi0Q=', 'Content-Length': '0'} 2022-09-12T20:16:10.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea46b71b6827c2e4-00631f938a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.489 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:10.490 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:10.491 DEBUG:boto:Method: HEAD 2022-09-12T20:16:10.492 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:16:10.493 DEBUG:boto:Data: 2022-09-12T20:16:10.493 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.494 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.495 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.496 DEBUG:boto:Params: {} 2022-09-12T20:16:10.497 DEBUG:boto:Token: None 2022-09-12T20:16:10.497 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj29 2022-09-12T20:16:10.498 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:F/DrCynls/Rc1p7F/4qpWFhxi0Q= 2022-09-12T20:16:10.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:F/DrCynls/Rc1p7F/4qpWFhxi0Q=', 'Content-Length': '0'} 2022-09-12T20:16:10.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f76f0cfa59c1bfd9-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:16:10.502 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:10.502 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:10.503 DEBUG:boto:Method: GET 2022-09-12T20:16:10.504 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:16:10.504 DEBUG:boto:Data: 2022-09-12T20:16:10.505 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.505 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.506 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.506 DEBUG:boto:Params: {} 2022-09-12T20:16:10.507 DEBUG:boto:Token: None 2022-09-12T20:16:10.507 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj29 2022-09-12T20:16:10.508 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L1bZPazsN/dYT6jf1EaFVNYaO4k= 2022-09-12T20:16:10.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L1bZPazsN/dYT6jf1EaFVNYaO4k=', 'Content-Length': '0'} 2022-09-12T20:16:10.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a413ee9c20df8295-00631f938a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.512 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:10.512 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:10.513 DEBUG:boto:Method: GET 2022-09-12T20:16:10.514 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:16:10.514 DEBUG:boto:Data: 2022-09-12T20:16:10.515 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.515 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.516 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.516 DEBUG:boto:Params: {} 2022-09-12T20:16:10.517 DEBUG:boto:Token: None 2022-09-12T20:16:10.518 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj29 2022-09-12T20:16:10.519 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L1bZPazsN/dYT6jf1EaFVNYaO4k= 2022-09-12T20:16:10.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L1bZPazsN/dYT6jf1EaFVNYaO4k=', 'Content-Length': '0'} 2022-09-12T20:16:10.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f0b9d96eb024154-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.523 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:10.523 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:10.524 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:16:10.524 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:16:10.525 DEBUG:boto:Method: GET 2022-09-12T20:16:10.526 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:16:10.526 DEBUG:boto:Data: 2022-09-12T20:16:10.527 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.527 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.528 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.528 DEBUG:boto:Params: {} 2022-09-12T20:16:10.529 DEBUG:boto:Token: None 2022-09-12T20:16:10.530 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/ 2022-09-12T20:16:10.531 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE= 2022-09-12T20:16:10.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE=', 'Content-Length': '0'} 2022-09-12T20:16:10.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043f26aa130a481d4-00631f938a-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.578 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:10.579 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:10.580 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:10.580 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:16:10.581 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:16:10.581 DEBUG:boto:Method: GET 2022-09-12T20:16:10.582 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:16:10.582 DEBUG:boto:Data: 2022-09-12T20:16:10.583 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.584 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.584 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.585 DEBUG:boto:Params: {} 2022-09-12T20:16:10.585 DEBUG:boto:Token: None 2022-09-12T20:16:10.586 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/ 2022-09-12T20:16:10.587 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE= 2022-09-12T20:16:10.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE=', 'Content-Length': '0'} 2022-09-12T20:16:10.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6a7fa53a1a9cec5-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.593 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:10.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-12T20:16:10.595 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:10.596 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:10.596 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:10.597 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:10.597 DEBUG:boto:Method: GET 2022-09-12T20:16:10.598 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:10.599 DEBUG:boto:Data: 2022-09-12T20:16:10.600 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.600 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.601 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.601 DEBUG:boto:Params: {} 2022-09-12T20:16:10.602 DEBUG:boto:Token: None 2022-09-12T20:16:10.603 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:10.603 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TPaAJcAeag2zVkhmYRjJbVWNfjc= 2022-09-12T20:16:10.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TPaAJcAeag2zVkhmYRjJbVWNfjc=', 'Content-Length': '0'} 2022-09-12T20:16:10.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8d38f91afa200fa-00631f938a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.607 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:10.608 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:10.608 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:10.609 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:10.610 DEBUG:boto:Method: GET 2022-09-12T20:16:10.610 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:10.611 DEBUG:boto:Data: 2022-09-12T20:16:10.611 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.612 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.612 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.613 DEBUG:boto:Params: {} 2022-09-12T20:16:10.613 DEBUG:boto:Token: None 2022-09-12T20:16:10.614 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:10.615 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TPaAJcAeag2zVkhmYRjJbVWNfjc= 2022-09-12T20:16:10.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TPaAJcAeag2zVkhmYRjJbVWNfjc=', 'Content-Length': '0'} 2022-09-12T20:16:10.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008cb16dd77683f32-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.658 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:10.659 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:10.659 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:10.660 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:10.660 DEBUG:boto:Method: HEAD 2022-09-12T20:16:10.661 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:10.661 DEBUG:boto:Data: 2022-09-12T20:16:10.662 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.662 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.663 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.663 DEBUG:boto:Params: {} 2022-09-12T20:16:10.664 DEBUG:boto:Token: None 2022-09-12T20:16:10.664 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:10.665 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ax/eG36d5ihEBRnZ19a0wK2dKvA= 2022-09-12T20:16:10.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ax/eG36d5ihEBRnZ19a0wK2dKvA=', 'Content-Length': '0'} 2022-09-12T20:16:10.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e87e0af4340dc724-00631f938a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.668 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:10.669 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:10.669 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:10.670 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:10.670 DEBUG:boto:Method: HEAD 2022-09-12T20:16:10.671 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:10.671 DEBUG:boto:Data: 2022-09-12T20:16:10.672 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.672 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.673 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.673 DEBUG:boto:Params: {} 2022-09-12T20:16:10.674 DEBUG:boto:Token: None 2022-09-12T20:16:10.675 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:10.675 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ax/eG36d5ihEBRnZ19a0wK2dKvA= 2022-09-12T20:16:10.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ax/eG36d5ihEBRnZ19a0wK2dKvA=', 'Content-Length': '0'} 2022-09-12T20:16:10.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a516c49bdf331df9-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-12T20:16:10.679 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:10.680 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:10.680 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:10.681 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:10.681 DEBUG:boto:Method: GET 2022-09-12T20:16:10.682 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:10.683 DEBUG:boto:Data: 2022-09-12T20:16:10.684 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.685 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.685 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.686 DEBUG:boto:Params: {} 2022-09-12T20:16:10.688 DEBUG:boto:Token: None 2022-09-12T20:16:10.688 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:10.689 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TPaAJcAeag2zVkhmYRjJbVWNfjc= 2022-09-12T20:16:10.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TPaAJcAeag2zVkhmYRjJbVWNfjc=', 'Content-Length': '0'} 2022-09-12T20:16:10.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074ecac3fe8904aad-00631f938a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.694 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:10.695 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:10.695 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:10.696 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:10.696 DEBUG:boto:Method: GET 2022-09-12T20:16:10.697 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:10.697 DEBUG:boto:Data: 2022-09-12T20:16:10.698 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.698 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.699 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.699 DEBUG:boto:Params: {} 2022-09-12T20:16:10.700 DEBUG:boto:Token: None 2022-09-12T20:16:10.701 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:10.701 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TPaAJcAeag2zVkhmYRjJbVWNfjc= 2022-09-12T20:16:10.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TPaAJcAeag2zVkhmYRjJbVWNfjc=', 'Content-Length': '0'} 2022-09-12T20:16:10.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074b4374e7466e824-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.705 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:10.706 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:10.706 DEBUG:boto:path=/jdpiak-75/?prefix=obj31 2022-09-12T20:16:10.707 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj31 2022-09-12T20:16:10.707 DEBUG:boto:Method: GET 2022-09-12T20:16:10.708 DEBUG:boto:Path: /jdpiak-75/?prefix=obj31 2022-09-12T20:16:10.708 DEBUG:boto:Data: 2022-09-12T20:16:10.709 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.709 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.710 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.710 DEBUG:boto:Params: {} 2022-09-12T20:16:10.711 DEBUG:boto:Token: None 2022-09-12T20:16:10.712 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/ 2022-09-12T20:16:10.712 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE= 2022-09-12T20:16:10.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE=', 'Content-Length': '0'} 2022-09-12T20:16:10.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c05ab38f841dabc6-00631f938a-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.726 DEBUG:boto:b'jdpiak-75obj311000falseobj312022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:10.727 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:10.727 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:10.728 DEBUG:boto:path=/jdpiak-75/?prefix=obj31 2022-09-12T20:16:10.728 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj31 2022-09-12T20:16:10.729 DEBUG:boto:Method: GET 2022-09-12T20:16:10.729 DEBUG:boto:Path: /jdpiak-75/?prefix=obj31 2022-09-12T20:16:10.730 DEBUG:boto:Data: 2022-09-12T20:16:10.731 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.731 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.732 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.732 DEBUG:boto:Params: {} 2022-09-12T20:16:10.733 DEBUG:boto:Token: None 2022-09-12T20:16:10.734 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/ 2022-09-12T20:16:10.734 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE= 2022-09-12T20:16:10.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE=', 'Content-Length': '0'} 2022-09-12T20:16:10.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0ef3d8791a7e8f0-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.740 DEBUG:boto:b'jdpiak-75obj311000falseobj312022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:10.741 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:10.742 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:10.743 DEBUG:boto:Method: HEAD 2022-09-12T20:16:10.743 DEBUG:boto:Path: /jdpiak-75/obj31 2022-09-12T20:16:10.744 DEBUG:boto:Data: 2022-09-12T20:16:10.744 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.745 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.745 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.746 DEBUG:boto:Params: {} 2022-09-12T20:16:10.746 DEBUG:boto:Token: None 2022-09-12T20:16:10.747 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj31 2022-09-12T20:16:10.748 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zMcByPY4E1IQt+zCr7SJRB1O0As= 2022-09-12T20:16:10.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zMcByPY4E1IQt+zCr7SJRB1O0As=', 'Content-Length': '0'} 2022-09-12T20:16:10.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001714a205bae0410f-00631f938a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.751 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:10.752 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:10.752 DEBUG:boto:Method: HEAD 2022-09-12T20:16:10.753 DEBUG:boto:Path: /jdpiak-75/obj31 2022-09-12T20:16:10.754 DEBUG:boto:Data: 2022-09-12T20:16:10.754 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.755 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.755 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.756 DEBUG:boto:Params: {} 2022-09-12T20:16:10.756 DEBUG:boto:Token: None 2022-09-12T20:16:10.757 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj31 2022-09-12T20:16:10.757 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zMcByPY4E1IQt+zCr7SJRB1O0As= 2022-09-12T20:16:10.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zMcByPY4E1IQt+zCr7SJRB1O0As=', 'Content-Length': '0'} 2022-09-12T20:16:10.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007defed762de3a84-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-12T20:16:10.761 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:10.762 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:10.763 DEBUG:boto:Method: GET 2022-09-12T20:16:10.763 DEBUG:boto:Path: /jdpiak-75/obj31 2022-09-12T20:16:10.764 DEBUG:boto:Data: 2022-09-12T20:16:10.764 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.765 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.765 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.766 DEBUG:boto:Params: {} 2022-09-12T20:16:10.767 DEBUG:boto:Token: None 2022-09-12T20:16:10.767 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj31 2022-09-12T20:16:10.768 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:n7QHB1VHPLkmhbB7imDywcbPGGw= 2022-09-12T20:16:10.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:n7QHB1VHPLkmhbB7imDywcbPGGw=', 'Content-Length': '0'} 2022-09-12T20:16:10.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c940684c220068e3-00631f938a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.814 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:10.815 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:10.815 DEBUG:boto:Method: GET 2022-09-12T20:16:10.816 DEBUG:boto:Path: /jdpiak-75/obj31 2022-09-12T20:16:10.816 DEBUG:boto:Data: 2022-09-12T20:16:10.817 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.818 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.818 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.819 DEBUG:boto:Params: {} 2022-09-12T20:16:10.820 DEBUG:boto:Token: None 2022-09-12T20:16:10.821 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj31 2022-09-12T20:16:10.821 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:n7QHB1VHPLkmhbB7imDywcbPGGw= 2022-09-12T20:16:10.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:n7QHB1VHPLkmhbB7imDywcbPGGw=', 'Content-Length': '0'} 2022-09-12T20:16:10.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076267cc9d8fd31d3-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.825 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:10.826 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:10.826 DEBUG:boto:path=/jdpiak-75/?prefix=obj31 2022-09-12T20:16:10.827 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj31 2022-09-12T20:16:10.827 DEBUG:boto:Method: GET 2022-09-12T20:16:10.828 DEBUG:boto:Path: /jdpiak-75/?prefix=obj31 2022-09-12T20:16:10.829 DEBUG:boto:Data: 2022-09-12T20:16:10.830 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.830 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.831 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.831 DEBUG:boto:Params: {} 2022-09-12T20:16:10.832 DEBUG:boto:Token: None 2022-09-12T20:16:10.832 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/ 2022-09-12T20:16:10.833 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE= 2022-09-12T20:16:10.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE=', 'Content-Length': '0'} 2022-09-12T20:16:10.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd11f9c01a032665-00631f938a-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.837 DEBUG:boto:b'jdpiak-75obj311000falseobj312022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:10.838 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:10.839 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:10.840 DEBUG:boto:path=/jdpiak-75/?prefix=obj31 2022-09-12T20:16:10.840 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj31 2022-09-12T20:16:10.841 DEBUG:boto:Method: GET 2022-09-12T20:16:10.841 DEBUG:boto:Path: /jdpiak-75/?prefix=obj31 2022-09-12T20:16:10.842 DEBUG:boto:Data: 2022-09-12T20:16:10.842 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.843 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.843 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.844 DEBUG:boto:Params: {} 2022-09-12T20:16:10.844 DEBUG:boto:Token: None 2022-09-12T20:16:10.845 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/ 2022-09-12T20:16:10.846 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE= 2022-09-12T20:16:10.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE=', 'Content-Length': '0'} 2022-09-12T20:16:10.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ba283e0267764fc-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.852 DEBUG:boto:b'jdpiak-75obj311000falseobj312022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:10.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:16:10.853 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:10.854 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:10.854 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:10.855 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:10.855 DEBUG:boto:Method: GET 2022-09-12T20:16:10.856 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:10.857 DEBUG:boto:Data: 2022-09-12T20:16:10.857 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.858 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.858 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.859 DEBUG:boto:Params: {} 2022-09-12T20:16:10.859 DEBUG:boto:Token: None 2022-09-12T20:16:10.860 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:10.861 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7OJBiPznqhvc0ixDy1eN9lkvSn8= 2022-09-12T20:16:10.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7OJBiPznqhvc0ixDy1eN9lkvSn8=', 'Content-Length': '0'} 2022-09-12T20:16:10.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000729dee7a72b15cb8-00631f938a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.864 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:10.865 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:10.866 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:10.866 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:10.867 DEBUG:boto:Method: GET 2022-09-12T20:16:10.867 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:10.868 DEBUG:boto:Data: 2022-09-12T20:16:10.868 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.869 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.869 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.870 DEBUG:boto:Params: {} 2022-09-12T20:16:10.871 DEBUG:boto:Token: None 2022-09-12T20:16:10.871 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:10.872 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7OJBiPznqhvc0ixDy1eN9lkvSn8= 2022-09-12T20:16:10.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7OJBiPznqhvc0ixDy1eN9lkvSn8=', 'Content-Length': '0'} 2022-09-12T20:16:10.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8473479d472b58a-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.918 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:10.919 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:10.919 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:10.920 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:10.920 DEBUG:boto:Method: HEAD 2022-09-12T20:16:10.921 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:10.921 DEBUG:boto:Data: 2022-09-12T20:16:10.922 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.922 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.923 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.923 DEBUG:boto:Params: {} 2022-09-12T20:16:10.924 DEBUG:boto:Token: None 2022-09-12T20:16:10.925 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:10.925 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1BCAkPQ1BJEw2kqRfTJKEIWP1do= 2022-09-12T20:16:10.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1BCAkPQ1BJEw2kqRfTJKEIWP1do=', 'Content-Length': '0'} 2022-09-12T20:16:10.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8ac1ef8759ce7f2-00631f938a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.929 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:10.930 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:10.930 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:10.931 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:10.931 DEBUG:boto:Method: HEAD 2022-09-12T20:16:10.932 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:10.932 DEBUG:boto:Data: 2022-09-12T20:16:10.933 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.934 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.934 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.935 DEBUG:boto:Params: {} 2022-09-12T20:16:10.935 DEBUG:boto:Token: None 2022-09-12T20:16:10.936 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:10.936 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1BCAkPQ1BJEw2kqRfTJKEIWP1do= 2022-09-12T20:16:10.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1BCAkPQ1BJEw2kqRfTJKEIWP1do=', 'Content-Length': '0'} 2022-09-12T20:16:10.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009de9f6c0dee8844f-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:16:10.940 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:10.941 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:10.942 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:10.942 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:10.943 DEBUG:boto:Method: GET 2022-09-12T20:16:10.943 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:10.944 DEBUG:boto:Data: 2022-09-12T20:16:10.944 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.945 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.945 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.946 DEBUG:boto:Params: {} 2022-09-12T20:16:10.947 DEBUG:boto:Token: None 2022-09-12T20:16:10.947 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:10.948 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7OJBiPznqhvc0ixDy1eN9lkvSn8= 2022-09-12T20:16:10.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7OJBiPznqhvc0ixDy1eN9lkvSn8=', 'Content-Length': '0'} 2022-09-12T20:16:10.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdcfb9b72a186c06-00631f938a-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.952 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:10.953 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:10.954 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:10.955 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:10.956 DEBUG:boto:Method: GET 2022-09-12T20:16:10.956 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:10.957 DEBUG:boto:Data: 2022-09-12T20:16:10.958 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.958 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.959 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.959 DEBUG:boto:Params: {} 2022-09-12T20:16:10.960 DEBUG:boto:Token: None 2022-09-12T20:16:10.961 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:10.961 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7OJBiPznqhvc0ixDy1eN9lkvSn8= 2022-09-12T20:16:10.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7OJBiPznqhvc0ixDy1eN9lkvSn8=', 'Content-Length': '0'} 2022-09-12T20:16:10.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c662087382ace7dd-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.965 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:10.965 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:10.966 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:16:10.967 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:16:10.967 DEBUG:boto:Method: GET 2022-09-12T20:16:10.968 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:16:10.968 DEBUG:boto:Data: 2022-09-12T20:16:10.969 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.969 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:10.970 DEBUG:boto:Port: 8001 2022-09-12T20:16:10.970 DEBUG:boto:Params: {} 2022-09-12T20:16:10.971 DEBUG:boto:Token: None 2022-09-12T20:16:10.972 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/ 2022-09-12T20:16:10.972 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE= 2022-09-12T20:16:10.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE=', 'Content-Length': '0'} 2022-09-12T20:16:10.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009614ca8005a6bd5-00631f938a-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.977 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:10.977 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:10.978 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:10.979 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:16:10.980 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:16:10.981 DEBUG:boto:Method: GET 2022-09-12T20:16:10.982 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:16:10.983 DEBUG:boto:Data: 2022-09-12T20:16:10.984 DEBUG:boto:Headers: {} 2022-09-12T20:16:10.985 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:10.986 DEBUG:boto:Port: 8000 2022-09-12T20:16:10.987 DEBUG:boto:Params: {} 2022-09-12T20:16:10.988 DEBUG:boto:Token: None 2022-09-12T20:16:10.989 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:10 GMT /jdpiak-75/ 2022-09-12T20:16:10.990 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE= 2022-09-12T20:16:10.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:10 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qCP0rjF8hKrBvIoJ6QUG7fPRxFE=', 'Content-Length': '0'} 2022-09-12T20:16:10.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9e9e8db2fc911f6-00631f938a-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:10.996 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:10.997 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:10.998 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:10.998 DEBUG:boto:Method: HEAD 2022-09-12T20:16:10.999 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:16:10.999 DEBUG:boto:Data: 2022-09-12T20:16:11.000 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.000 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.001 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.001 DEBUG:boto:Params: {} 2022-09-12T20:16:11.002 DEBUG:boto:Token: None 2022-09-12T20:16:11.002 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj33 2022-09-12T20:16:11.003 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Th76Ip0C7Smw8MjioQ2Wh0S40Tg= 2022-09-12T20:16:11.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Th76Ip0C7Smw8MjioQ2Wh0S40Tg=', 'Content-Length': '0'} 2022-09-12T20:16:11.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4162d2e678aa0da-00631f938b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.007 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:11.007 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:11.008 DEBUG:boto:Method: HEAD 2022-09-12T20:16:11.008 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:16:11.009 DEBUG:boto:Data: 2022-09-12T20:16:11.009 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.010 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.011 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.011 DEBUG:boto:Params: {} 2022-09-12T20:16:11.012 DEBUG:boto:Token: None 2022-09-12T20:16:11.013 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj33 2022-09-12T20:16:11.014 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Th76Ip0C7Smw8MjioQ2Wh0S40Tg= 2022-09-12T20:16:11.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Th76Ip0C7Smw8MjioQ2Wh0S40Tg=', 'Content-Length': '0'} 2022-09-12T20:16:11.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3c55121af9101e4-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:16:11.018 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:11.018 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:11.019 DEBUG:boto:Method: GET 2022-09-12T20:16:11.019 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:16:11.020 DEBUG:boto:Data: 2022-09-12T20:16:11.021 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.021 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.022 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.022 DEBUG:boto:Params: {} 2022-09-12T20:16:11.023 DEBUG:boto:Token: None 2022-09-12T20:16:11.024 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj33 2022-09-12T20:16:11.024 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lPsxXR8OchQjW6+mqV1MggrptA0= 2022-09-12T20:16:11.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lPsxXR8OchQjW6+mqV1MggrptA0=', 'Content-Length': '0'} 2022-09-12T20:16:11.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e38738c977294244-00631f938b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.028 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:11.029 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:11.030 DEBUG:boto:Method: GET 2022-09-12T20:16:11.030 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:16:11.031 DEBUG:boto:Data: 2022-09-12T20:16:11.032 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.032 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.033 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.033 DEBUG:boto:Params: {} 2022-09-12T20:16:11.034 DEBUG:boto:Token: None 2022-09-12T20:16:11.035 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj33 2022-09-12T20:16:11.035 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lPsxXR8OchQjW6+mqV1MggrptA0= 2022-09-12T20:16:11.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lPsxXR8OchQjW6+mqV1MggrptA0=', 'Content-Length': '0'} 2022-09-12T20:16:11.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa20895c53b48449-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.039 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:11.039 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:11.040 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:16:11.040 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:16:11.041 DEBUG:boto:Method: GET 2022-09-12T20:16:11.041 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:16:11.042 DEBUG:boto:Data: 2022-09-12T20:16:11.042 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.043 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.043 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.044 DEBUG:boto:Params: {} 2022-09-12T20:16:11.044 DEBUG:boto:Token: None 2022-09-12T20:16:11.045 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/ 2022-09-12T20:16:11.046 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg= 2022-09-12T20:16:11.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg=', 'Content-Length': '0'} 2022-09-12T20:16:11.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ff0aeba24fcb64b-00631f938b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.050 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:11.051 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:11.051 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:11.052 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:16:11.052 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:16:11.053 DEBUG:boto:Method: GET 2022-09-12T20:16:11.053 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:16:11.054 DEBUG:boto:Data: 2022-09-12T20:16:11.054 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.055 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.056 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.056 DEBUG:boto:Params: {} 2022-09-12T20:16:11.057 DEBUG:boto:Token: None 2022-09-12T20:16:11.058 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/ 2022-09-12T20:16:11.058 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg= 2022-09-12T20:16:11.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg=', 'Content-Length': '0'} 2022-09-12T20:16:11.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c9fb3ce5f2f00fa-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.065 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:11.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:16:11.067 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:11.068 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:11.069 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:11.069 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:11.070 DEBUG:boto:Method: GET 2022-09-12T20:16:11.070 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:11.071 DEBUG:boto:Data: 2022-09-12T20:16:11.071 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.072 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.072 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.073 DEBUG:boto:Params: {} 2022-09-12T20:16:11.074 DEBUG:boto:Token: None 2022-09-12T20:16:11.074 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:11.075 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TYryut58WsB8WvZoHTbpCFSOPGc= 2022-09-12T20:16:11.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TYryut58WsB8WvZoHTbpCFSOPGc=', 'Content-Length': '0'} 2022-09-12T20:16:11.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047d38eae852bda36-00631f938b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.079 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:11.079 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:11.080 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:11.080 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:11.081 DEBUG:boto:Method: GET 2022-09-12T20:16:11.081 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:11.082 DEBUG:boto:Data: 2022-09-12T20:16:11.082 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.083 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.084 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.084 DEBUG:boto:Params: {} 2022-09-12T20:16:11.085 DEBUG:boto:Token: None 2022-09-12T20:16:11.085 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:11.086 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TYryut58WsB8WvZoHTbpCFSOPGc= 2022-09-12T20:16:11.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TYryut58WsB8WvZoHTbpCFSOPGc=', 'Content-Length': '0'} 2022-09-12T20:16:11.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e7b2c6bb7583743-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.130 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:11.131 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:11.131 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:11.132 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:11.132 DEBUG:boto:Method: HEAD 2022-09-12T20:16:11.133 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:11.133 DEBUG:boto:Data: 2022-09-12T20:16:11.134 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.134 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.135 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.135 DEBUG:boto:Params: {} 2022-09-12T20:16:11.136 DEBUG:boto:Token: None 2022-09-12T20:16:11.137 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:11.137 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WkfaOlEWqWv+J3djDPDfZJ5HDwo= 2022-09-12T20:16:11.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WkfaOlEWqWv+J3djDPDfZJ5HDwo=', 'Content-Length': '0'} 2022-09-12T20:16:11.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afb7c6bfd5d674ee-00631f938b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.141 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:11.141 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:11.142 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:11.142 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:11.143 DEBUG:boto:Method: HEAD 2022-09-12T20:16:11.143 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:11.144 DEBUG:boto:Data: 2022-09-12T20:16:11.144 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.145 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.145 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.146 DEBUG:boto:Params: {} 2022-09-12T20:16:11.146 DEBUG:boto:Token: None 2022-09-12T20:16:11.147 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:11.147 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WkfaOlEWqWv+J3djDPDfZJ5HDwo= 2022-09-12T20:16:11.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WkfaOlEWqWv+J3djDPDfZJ5HDwo=', 'Content-Length': '0'} 2022-09-12T20:16:11.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cab68543ff12e0c-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:16:11.151 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:11.152 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:11.152 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:11.153 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:11.153 DEBUG:boto:Method: GET 2022-09-12T20:16:11.154 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:11.154 DEBUG:boto:Data: 2022-09-12T20:16:11.155 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.155 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.156 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.156 DEBUG:boto:Params: {} 2022-09-12T20:16:11.157 DEBUG:boto:Token: None 2022-09-12T20:16:11.158 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:11.158 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TYryut58WsB8WvZoHTbpCFSOPGc= 2022-09-12T20:16:11.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TYryut58WsB8WvZoHTbpCFSOPGc=', 'Content-Length': '0'} 2022-09-12T20:16:11.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000301f4cb0ed4b72d2-00631f938b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.162 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:11.163 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:11.163 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:11.164 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:11.164 DEBUG:boto:Method: GET 2022-09-12T20:16:11.165 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:11.165 DEBUG:boto:Data: 2022-09-12T20:16:11.166 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.166 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.167 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.167 DEBUG:boto:Params: {} 2022-09-12T20:16:11.168 DEBUG:boto:Token: None 2022-09-12T20:16:11.169 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:11.169 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TYryut58WsB8WvZoHTbpCFSOPGc= 2022-09-12T20:16:11.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TYryut58WsB8WvZoHTbpCFSOPGc=', 'Content-Length': '0'} 2022-09-12T20:16:11.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcec5720ccea85a3-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.173 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:11.173 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:11.174 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:16:11.174 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:16:11.175 DEBUG:boto:Method: GET 2022-09-12T20:16:11.175 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:16:11.176 DEBUG:boto:Data: 2022-09-12T20:16:11.176 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.177 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.178 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.178 DEBUG:boto:Params: {} 2022-09-12T20:16:11.179 DEBUG:boto:Token: None 2022-09-12T20:16:11.179 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/ 2022-09-12T20:16:11.180 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg= 2022-09-12T20:16:11.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg=', 'Content-Length': '0'} 2022-09-12T20:16:11.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf86850152f59c4a-00631f938b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.184 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:11.185 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:11.185 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:11.186 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:16:11.186 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:16:11.187 DEBUG:boto:Method: GET 2022-09-12T20:16:11.187 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:16:11.188 DEBUG:boto:Data: 2022-09-12T20:16:11.188 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.189 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.190 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.191 DEBUG:boto:Params: {} 2022-09-12T20:16:11.191 DEBUG:boto:Token: None 2022-09-12T20:16:11.192 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/ 2022-09-12T20:16:11.192 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg= 2022-09-12T20:16:11.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg=', 'Content-Length': '0'} 2022-09-12T20:16:11.198 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb41c3e7a845dadd-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.198 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:11.199 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:11.200 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:11.200 DEBUG:boto:Method: HEAD 2022-09-12T20:16:11.201 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:16:11.201 DEBUG:boto:Data: 2022-09-12T20:16:11.202 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.202 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.203 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.203 DEBUG:boto:Params: {} 2022-09-12T20:16:11.204 DEBUG:boto:Token: None 2022-09-12T20:16:11.205 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj34 2022-09-12T20:16:11.206 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Pagl7lMl89V0r/qFANg/QGsxEWo= 2022-09-12T20:16:11.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Pagl7lMl89V0r/qFANg/QGsxEWo=', 'Content-Length': '0'} 2022-09-12T20:16:11.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a750f0e6c8b1211-00631f938b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.209 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:11.210 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:11.210 DEBUG:boto:Method: HEAD 2022-09-12T20:16:11.211 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:16:11.211 DEBUG:boto:Data: 2022-09-12T20:16:11.212 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.213 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.214 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.214 DEBUG:boto:Params: {} 2022-09-12T20:16:11.215 DEBUG:boto:Token: None 2022-09-12T20:16:11.215 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj34 2022-09-12T20:16:11.216 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Pagl7lMl89V0r/qFANg/QGsxEWo= 2022-09-12T20:16:11.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Pagl7lMl89V0r/qFANg/QGsxEWo=', 'Content-Length': '0'} 2022-09-12T20:16:11.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038777aca765fdb40-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:16:11.220 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:11.221 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:11.221 DEBUG:boto:Method: GET 2022-09-12T20:16:11.222 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:16:11.222 DEBUG:boto:Data: 2022-09-12T20:16:11.223 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.223 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.224 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.224 DEBUG:boto:Params: {} 2022-09-12T20:16:11.225 DEBUG:boto:Token: None 2022-09-12T20:16:11.226 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj34 2022-09-12T20:16:11.226 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:35r8HBc3baaSGGxWIO263zK+mXI= 2022-09-12T20:16:11.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:35r8HBc3baaSGGxWIO263zK+mXI=', 'Content-Length': '0'} 2022-09-12T20:16:11.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ac297810fc89473-00631f938b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.230 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:11.231 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:11.231 DEBUG:boto:Method: GET 2022-09-12T20:16:11.232 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:16:11.233 DEBUG:boto:Data: 2022-09-12T20:16:11.233 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.234 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.234 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.235 DEBUG:boto:Params: {} 2022-09-12T20:16:11.235 DEBUG:boto:Token: None 2022-09-12T20:16:11.236 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj34 2022-09-12T20:16:11.237 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:35r8HBc3baaSGGxWIO263zK+mXI= 2022-09-12T20:16:11.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:35r8HBc3baaSGGxWIO263zK+mXI=', 'Content-Length': '0'} 2022-09-12T20:16:11.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b983d07413e93c0-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.282 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:11.283 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:11.283 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:16:11.284 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:16:11.284 DEBUG:boto:Method: GET 2022-09-12T20:16:11.285 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:16:11.285 DEBUG:boto:Data: 2022-09-12T20:16:11.286 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.287 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.287 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.288 DEBUG:boto:Params: {} 2022-09-12T20:16:11.288 DEBUG:boto:Token: None 2022-09-12T20:16:11.289 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/ 2022-09-12T20:16:11.290 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg= 2022-09-12T20:16:11.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg=', 'Content-Length': '0'} 2022-09-12T20:16:11.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bef95af74da3c31f-00631f938b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.295 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:11.295 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:11.296 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:11.297 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:16:11.297 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:16:11.298 DEBUG:boto:Method: GET 2022-09-12T20:16:11.299 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:16:11.299 DEBUG:boto:Data: 2022-09-12T20:16:11.300 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.301 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.301 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.302 DEBUG:boto:Params: {} 2022-09-12T20:16:11.303 DEBUG:boto:Token: None 2022-09-12T20:16:11.304 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/ 2022-09-12T20:16:11.304 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg= 2022-09-12T20:16:11.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg=', 'Content-Length': '0'} 2022-09-12T20:16:11.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd0b823808fa21e1-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.311 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:11.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2022-09-12T20:16:11.313 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:11.313 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:11.314 DEBUG:boto:path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:11.314 DEBUG:boto:auth_path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:11.315 DEBUG:boto:Method: GET 2022-09-12T20:16:11.315 DEBUG:boto:Path: /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:11.316 DEBUG:boto:Data: 2022-09-12T20:16:11.317 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.317 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.318 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.318 DEBUG:boto:Params: {} 2022-09-12T20:16:11.319 DEBUG:boto:Token: None 2022-09-12T20:16:11.320 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:11.321 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uIc2N3yF6N+CrNqpl8YUJqcuJhk= 2022-09-12T20:16:11.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uIc2N3yF6N+CrNqpl8YUJqcuJhk=', 'Content-Length': '0'} 2022-09-12T20:16:11.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000749ea97866976632-00631f938b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.366 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:11.367 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:11.367 DEBUG:boto:path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:11.368 DEBUG:boto:auth_path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:11.368 DEBUG:boto:Method: GET 2022-09-12T20:16:11.369 DEBUG:boto:Path: /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:11.370 DEBUG:boto:Data: 2022-09-12T20:16:11.370 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.371 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.372 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.372 DEBUG:boto:Params: {} 2022-09-12T20:16:11.373 DEBUG:boto:Token: None 2022-09-12T20:16:11.373 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:11.374 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uIc2N3yF6N+CrNqpl8YUJqcuJhk= 2022-09-12T20:16:11.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uIc2N3yF6N+CrNqpl8YUJqcuJhk=', 'Content-Length': '0'} 2022-09-12T20:16:11.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059fac697d61e92d4-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.379 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:11.379 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:11.380 DEBUG:boto:path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:11.380 DEBUG:boto:auth_path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:11.381 DEBUG:boto:Method: HEAD 2022-09-12T20:16:11.382 DEBUG:boto:Path: /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:11.382 DEBUG:boto:Data: 2022-09-12T20:16:11.383 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.383 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.384 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.384 DEBUG:boto:Params: {} 2022-09-12T20:16:11.385 DEBUG:boto:Token: None 2022-09-12T20:16:11.386 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:11.386 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:st52nrFC/BSGBvFtm8wEc6Y0O70= 2022-09-12T20:16:11.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:st52nrFC/BSGBvFtm8wEc6Y0O70=', 'Content-Length': '0'} 2022-09-12T20:16:11.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000751dd5b81efeaf8f-00631f938b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.390 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:11.391 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:11.391 DEBUG:boto:path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:11.392 DEBUG:boto:auth_path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:11.392 DEBUG:boto:Method: HEAD 2022-09-12T20:16:11.393 DEBUG:boto:Path: /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:11.393 DEBUG:boto:Data: 2022-09-12T20:16:11.394 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.394 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.395 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.395 DEBUG:boto:Params: {} 2022-09-12T20:16:11.396 DEBUG:boto:Token: None 2022-09-12T20:16:11.397 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:11.397 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:st52nrFC/BSGBvFtm8wEc6Y0O70= 2022-09-12T20:16:11.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:st52nrFC/BSGBvFtm8wEc6Y0O70=', 'Content-Length': '0'} 2022-09-12T20:16:11.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd0149f0cb3881b9-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2022-09-12T20:16:11.403 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:11.404 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:11.405 DEBUG:boto:path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:11.405 DEBUG:boto:auth_path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:11.406 DEBUG:boto:Method: GET 2022-09-12T20:16:11.407 DEBUG:boto:Path: /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:11.407 DEBUG:boto:Data: 2022-09-12T20:16:11.408 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.408 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.409 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.409 DEBUG:boto:Params: {} 2022-09-12T20:16:11.410 DEBUG:boto:Token: None 2022-09-12T20:16:11.411 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:11.411 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uIc2N3yF6N+CrNqpl8YUJqcuJhk= 2022-09-12T20:16:11.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uIc2N3yF6N+CrNqpl8YUJqcuJhk=', 'Content-Length': '0'} 2022-09-12T20:16:11.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007504c4d4c41ba4d8-00631f938b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.415 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:11.415 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:11.416 DEBUG:boto:path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:11.417 DEBUG:boto:auth_path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:11.417 DEBUG:boto:Method: GET 2022-09-12T20:16:11.418 DEBUG:boto:Path: /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:11.418 DEBUG:boto:Data: 2022-09-12T20:16:11.419 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.420 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.420 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.421 DEBUG:boto:Params: {} 2022-09-12T20:16:11.421 DEBUG:boto:Token: None 2022-09-12T20:16:11.422 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:11.422 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uIc2N3yF6N+CrNqpl8YUJqcuJhk= 2022-09-12T20:16:11.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uIc2N3yF6N+CrNqpl8YUJqcuJhk=', 'Content-Length': '0'} 2022-09-12T20:16:11.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ab593fdc3382ee1-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.426 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:11.427 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:11.427 DEBUG:boto:path=/jdpiak-75/?prefix=obj37 2022-09-12T20:16:11.428 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj37 2022-09-12T20:16:11.429 DEBUG:boto:Method: GET 2022-09-12T20:16:11.429 DEBUG:boto:Path: /jdpiak-75/?prefix=obj37 2022-09-12T20:16:11.430 DEBUG:boto:Data: 2022-09-12T20:16:11.430 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.431 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.431 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.432 DEBUG:boto:Params: {} 2022-09-12T20:16:11.433 DEBUG:boto:Token: None 2022-09-12T20:16:11.433 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/ 2022-09-12T20:16:11.434 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg= 2022-09-12T20:16:11.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg=', 'Content-Length': '0'} 2022-09-12T20:16:11.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6f60dbb2c2a6818-00631f938b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.438 DEBUG:boto:b'jdpiak-75obj371000falseobj372022-09-12T20:14:26.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:11.439 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:11.439 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:11.440 DEBUG:boto:path=/jdpiak-75/?prefix=obj37 2022-09-12T20:16:11.440 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj37 2022-09-12T20:16:11.441 DEBUG:boto:Method: GET 2022-09-12T20:16:11.442 DEBUG:boto:Path: /jdpiak-75/?prefix=obj37 2022-09-12T20:16:11.442 DEBUG:boto:Data: 2022-09-12T20:16:11.443 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.444 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.445 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.445 DEBUG:boto:Params: {} 2022-09-12T20:16:11.447 DEBUG:boto:Token: None 2022-09-12T20:16:11.448 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/ 2022-09-12T20:16:11.449 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg= 2022-09-12T20:16:11.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg=', 'Content-Length': '0'} 2022-09-12T20:16:11.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b308631d8b2783db-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.455 DEBUG:boto:b'jdpiak-75obj371000falseobj372022-09-12T20:14:26.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:11.456 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:11.457 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:11.457 DEBUG:boto:Method: HEAD 2022-09-12T20:16:11.458 DEBUG:boto:Path: /jdpiak-75/obj37 2022-09-12T20:16:11.458 DEBUG:boto:Data: 2022-09-12T20:16:11.459 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.459 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.460 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.461 DEBUG:boto:Params: {} 2022-09-12T20:16:11.461 DEBUG:boto:Token: None 2022-09-12T20:16:11.462 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj37 2022-09-12T20:16:11.462 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uRPwg7e1PsDJBG0+VQhUJfO9BV4= 2022-09-12T20:16:11.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uRPwg7e1PsDJBG0+VQhUJfO9BV4=', 'Content-Length': '0'} 2022-09-12T20:16:11.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003afeefce02687b5b-00631f938b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.466 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:11.466 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:11.467 DEBUG:boto:Method: HEAD 2022-09-12T20:16:11.467 DEBUG:boto:Path: /jdpiak-75/obj37 2022-09-12T20:16:11.468 DEBUG:boto:Data: 2022-09-12T20:16:11.468 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.469 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.469 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.470 DEBUG:boto:Params: {} 2022-09-12T20:16:11.470 DEBUG:boto:Token: None 2022-09-12T20:16:11.471 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj37 2022-09-12T20:16:11.472 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uRPwg7e1PsDJBG0+VQhUJfO9BV4= 2022-09-12T20:16:11.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uRPwg7e1PsDJBG0+VQhUJfO9BV4=', 'Content-Length': '0'} 2022-09-12T20:16:11.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1d42d72da996dc7-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2022-09-12T20:16:11.476 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:11.476 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:11.477 DEBUG:boto:Method: GET 2022-09-12T20:16:11.477 DEBUG:boto:Path: /jdpiak-75/obj37 2022-09-12T20:16:11.478 DEBUG:boto:Data: 2022-09-12T20:16:11.479 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.479 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.480 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.480 DEBUG:boto:Params: {} 2022-09-12T20:16:11.481 DEBUG:boto:Token: None 2022-09-12T20:16:11.481 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj37 2022-09-12T20:16:11.482 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1RbfVtgjs72F6HfRDUhfgMQdzEs= 2022-09-12T20:16:11.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1RbfVtgjs72F6HfRDUhfgMQdzEs=', 'Content-Length': '0'} 2022-09-12T20:16:11.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c77113b003c61df0-00631f938b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.486 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:11.486 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:11.487 DEBUG:boto:Method: GET 2022-09-12T20:16:11.487 DEBUG:boto:Path: /jdpiak-75/obj37 2022-09-12T20:16:11.488 DEBUG:boto:Data: 2022-09-12T20:16:11.488 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.489 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.489 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.490 DEBUG:boto:Params: {} 2022-09-12T20:16:11.490 DEBUG:boto:Token: None 2022-09-12T20:16:11.491 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj37 2022-09-12T20:16:11.492 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1RbfVtgjs72F6HfRDUhfgMQdzEs= 2022-09-12T20:16:11.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1RbfVtgjs72F6HfRDUhfgMQdzEs=', 'Content-Length': '0'} 2022-09-12T20:16:11.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5f1630d3d79a448-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.542 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:11.543 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:11.544 DEBUG:boto:path=/jdpiak-75/?prefix=obj37 2022-09-12T20:16:11.544 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj37 2022-09-12T20:16:11.545 DEBUG:boto:Method: GET 2022-09-12T20:16:11.545 DEBUG:boto:Path: /jdpiak-75/?prefix=obj37 2022-09-12T20:16:11.546 DEBUG:boto:Data: 2022-09-12T20:16:11.546 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.547 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.548 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.548 DEBUG:boto:Params: {} 2022-09-12T20:16:11.549 DEBUG:boto:Token: None 2022-09-12T20:16:11.549 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/ 2022-09-12T20:16:11.550 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg= 2022-09-12T20:16:11.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg=', 'Content-Length': '0'} 2022-09-12T20:16:11.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4284c68b2c29756-00631f938b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.554 DEBUG:boto:b'jdpiak-75obj371000falseobj372022-09-12T20:14:26.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:11.555 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:11.555 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:11.556 DEBUG:boto:path=/jdpiak-75/?prefix=obj37 2022-09-12T20:16:11.556 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj37 2022-09-12T20:16:11.557 DEBUG:boto:Method: GET 2022-09-12T20:16:11.557 DEBUG:boto:Path: /jdpiak-75/?prefix=obj37 2022-09-12T20:16:11.558 DEBUG:boto:Data: 2022-09-12T20:16:11.558 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.559 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.559 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.560 DEBUG:boto:Params: {} 2022-09-12T20:16:11.561 DEBUG:boto:Token: None 2022-09-12T20:16:11.561 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/ 2022-09-12T20:16:11.562 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg= 2022-09-12T20:16:11.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg=', 'Content-Length': '0'} 2022-09-12T20:16:11.567 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055273620ae09e67b-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.568 DEBUG:boto:b'jdpiak-75obj371000falseobj372022-09-12T20:14:26.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:11.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:16:11.569 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:11.570 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:11.570 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:11.571 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:11.571 DEBUG:boto:Method: GET 2022-09-12T20:16:11.572 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:11.572 DEBUG:boto:Data: 2022-09-12T20:16:11.573 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.573 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.574 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.574 DEBUG:boto:Params: {} 2022-09-12T20:16:11.575 DEBUG:boto:Token: None 2022-09-12T20:16:11.576 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:11.576 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8q/qeOFykndbpcz2w8tCPrxQmos= 2022-09-12T20:16:11.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8q/qeOFykndbpcz2w8tCPrxQmos=', 'Content-Length': '0'} 2022-09-12T20:16:11.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee96b80d17c3a19f-00631f938b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.580 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:11.581 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:11.581 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:11.582 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:11.583 DEBUG:boto:Method: GET 2022-09-12T20:16:11.583 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:11.584 DEBUG:boto:Data: 2022-09-12T20:16:11.584 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.585 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.585 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.586 DEBUG:boto:Params: {} 2022-09-12T20:16:11.587 DEBUG:boto:Token: None 2022-09-12T20:16:11.587 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:11.588 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8q/qeOFykndbpcz2w8tCPrxQmos= 2022-09-12T20:16:11.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8q/qeOFykndbpcz2w8tCPrxQmos=', 'Content-Length': '0'} 2022-09-12T20:16:11.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9085c19038ed955-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.592 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:11.592 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:11.593 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:11.593 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:11.594 DEBUG:boto:Method: HEAD 2022-09-12T20:16:11.594 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:11.595 DEBUG:boto:Data: 2022-09-12T20:16:11.595 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.596 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.596 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.597 DEBUG:boto:Params: {} 2022-09-12T20:16:11.597 DEBUG:boto:Token: None 2022-09-12T20:16:11.598 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:11.598 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2DdvdphZUKj/N82krT/V27mfZ3k= 2022-09-12T20:16:11.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2DdvdphZUKj/N82krT/V27mfZ3k=', 'Content-Length': '0'} 2022-09-12T20:16:11.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4c0e64e35a028e9-00631f938b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.602 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:11.602 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:11.603 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:11.603 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:11.604 DEBUG:boto:Method: HEAD 2022-09-12T20:16:11.605 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:11.605 DEBUG:boto:Data: 2022-09-12T20:16:11.605 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.606 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.606 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.607 DEBUG:boto:Params: {} 2022-09-12T20:16:11.608 DEBUG:boto:Token: None 2022-09-12T20:16:11.608 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:11.609 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2DdvdphZUKj/N82krT/V27mfZ3k= 2022-09-12T20:16:11.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2DdvdphZUKj/N82krT/V27mfZ3k=', 'Content-Length': '0'} 2022-09-12T20:16:11.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000284da33660f6bde4-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:16:11.613 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:11.613 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:11.614 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:11.614 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:11.615 DEBUG:boto:Method: GET 2022-09-12T20:16:11.615 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:11.616 DEBUG:boto:Data: 2022-09-12T20:16:11.616 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.617 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.618 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.618 DEBUG:boto:Params: {} 2022-09-12T20:16:11.619 DEBUG:boto:Token: None 2022-09-12T20:16:11.619 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:11.620 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8q/qeOFykndbpcz2w8tCPrxQmos= 2022-09-12T20:16:11.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8q/qeOFykndbpcz2w8tCPrxQmos=', 'Content-Length': '0'} 2022-09-12T20:16:11.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003eb361786abb99f1-00631f938b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.623 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:11.624 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:11.624 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:11.625 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:11.625 DEBUG:boto:Method: GET 2022-09-12T20:16:11.626 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:11.626 DEBUG:boto:Data: 2022-09-12T20:16:11.627 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.628 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.628 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.629 DEBUG:boto:Params: {} 2022-09-12T20:16:11.630 DEBUG:boto:Token: None 2022-09-12T20:16:11.630 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:11.631 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8q/qeOFykndbpcz2w8tCPrxQmos= 2022-09-12T20:16:11.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8q/qeOFykndbpcz2w8tCPrxQmos=', 'Content-Length': '0'} 2022-09-12T20:16:11.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068a0784b5bf2da3d-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.678 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:11.679 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:11.680 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:16:11.680 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:16:11.681 DEBUG:boto:Method: GET 2022-09-12T20:16:11.681 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:16:11.682 DEBUG:boto:Data: 2022-09-12T20:16:11.682 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.683 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.683 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.684 DEBUG:boto:Params: {} 2022-09-12T20:16:11.685 DEBUG:boto:Token: None 2022-09-12T20:16:11.685 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/ 2022-09-12T20:16:11.686 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg= 2022-09-12T20:16:11.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg=', 'Content-Length': '0'} 2022-09-12T20:16:11.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034b8a8133b634983-00631f938b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.690 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:11.691 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:11.692 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:11.692 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:16:11.693 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:16:11.693 DEBUG:boto:Method: GET 2022-09-12T20:16:11.694 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:16:11.694 DEBUG:boto:Data: 2022-09-12T20:16:11.695 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.695 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.696 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.696 DEBUG:boto:Params: {} 2022-09-12T20:16:11.697 DEBUG:boto:Token: None 2022-09-12T20:16:11.698 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/ 2022-09-12T20:16:11.698 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg= 2022-09-12T20:16:11.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg=', 'Content-Length': '0'} 2022-09-12T20:16:11.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb4d2f53238a35f3-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.705 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:11.706 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:11.706 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:11.707 DEBUG:boto:Method: HEAD 2022-09-12T20:16:11.707 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:16:11.708 DEBUG:boto:Data: 2022-09-12T20:16:11.708 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.709 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.709 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.710 DEBUG:boto:Params: {} 2022-09-12T20:16:11.710 DEBUG:boto:Token: None 2022-09-12T20:16:11.711 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj38 2022-09-12T20:16:11.711 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HBpOJJ/I2btq0jIYSiLmV5fFJPQ= 2022-09-12T20:16:11.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HBpOJJ/I2btq0jIYSiLmV5fFJPQ=', 'Content-Length': '0'} 2022-09-12T20:16:11.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ebcbd92f7926d43-00631f938b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.715 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:11.715 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:11.716 DEBUG:boto:Method: HEAD 2022-09-12T20:16:11.716 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:16:11.717 DEBUG:boto:Data: 2022-09-12T20:16:11.717 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.718 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.719 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.719 DEBUG:boto:Params: {} 2022-09-12T20:16:11.720 DEBUG:boto:Token: None 2022-09-12T20:16:11.720 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj38 2022-09-12T20:16:11.721 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HBpOJJ/I2btq0jIYSiLmV5fFJPQ= 2022-09-12T20:16:11.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HBpOJJ/I2btq0jIYSiLmV5fFJPQ=', 'Content-Length': '0'} 2022-09-12T20:16:11.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5d64a91981dc072-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:16:11.725 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:11.726 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:11.726 DEBUG:boto:Method: GET 2022-09-12T20:16:11.727 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:16:11.727 DEBUG:boto:Data: 2022-09-12T20:16:11.728 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.729 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.729 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.730 DEBUG:boto:Params: {} 2022-09-12T20:16:11.730 DEBUG:boto:Token: None 2022-09-12T20:16:11.731 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj38 2022-09-12T20:16:11.732 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B4RYNITaxHjALQqURROMCbVB+EU= 2022-09-12T20:16:11.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B4RYNITaxHjALQqURROMCbVB+EU=', 'Content-Length': '0'} 2022-09-12T20:16:11.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd41b67df5649d73-00631f938b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.778 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:11.779 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:11.779 DEBUG:boto:Method: GET 2022-09-12T20:16:11.780 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:16:11.781 DEBUG:boto:Data: 2022-09-12T20:16:11.781 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.782 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.782 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.783 DEBUG:boto:Params: {} 2022-09-12T20:16:11.783 DEBUG:boto:Token: None 2022-09-12T20:16:11.784 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj38 2022-09-12T20:16:11.785 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:B4RYNITaxHjALQqURROMCbVB+EU= 2022-09-12T20:16:11.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:B4RYNITaxHjALQqURROMCbVB+EU=', 'Content-Length': '0'} 2022-09-12T20:16:11.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bad102d0f4e120b-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.788 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:11.789 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:11.790 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:16:11.790 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:16:11.791 DEBUG:boto:Method: GET 2022-09-12T20:16:11.791 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:16:11.792 DEBUG:boto:Data: 2022-09-12T20:16:11.792 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.793 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.793 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.794 DEBUG:boto:Params: {} 2022-09-12T20:16:11.795 DEBUG:boto:Token: None 2022-09-12T20:16:11.795 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/ 2022-09-12T20:16:11.796 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg= 2022-09-12T20:16:11.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg=', 'Content-Length': '0'} 2022-09-12T20:16:11.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ff780e3e48156f0-00631f938b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.800 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:11.801 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:11.802 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:11.802 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:16:11.803 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:16:11.803 DEBUG:boto:Method: GET 2022-09-12T20:16:11.804 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:16:11.804 DEBUG:boto:Data: 2022-09-12T20:16:11.805 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.805 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.806 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.812 DEBUG:boto:Params: {} 2022-09-12T20:16:11.813 DEBUG:boto:Token: None 2022-09-12T20:16:11.814 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/ 2022-09-12T20:16:11.814 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg= 2022-09-12T20:16:11.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg=', 'Content-Length': '0'} 2022-09-12T20:16:11.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026e404fdec78169c-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.820 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:11.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:16:11.822 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:11.822 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:11.823 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:11.823 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:11.824 DEBUG:boto:Method: GET 2022-09-12T20:16:11.825 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:11.825 DEBUG:boto:Data: 2022-09-12T20:16:11.826 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.826 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.827 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.827 DEBUG:boto:Params: {} 2022-09-12T20:16:11.828 DEBUG:boto:Token: None 2022-09-12T20:16:11.829 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:11.829 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q3DTwsyO0oWnKipcutLVK740320= 2022-09-12T20:16:11.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q3DTwsyO0oWnKipcutLVK740320=', 'Content-Length': '0'} 2022-09-12T20:16:11.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd5734311aedde56-00631f938b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.833 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:11.833 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:11.834 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:11.835 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:11.835 DEBUG:boto:Method: GET 2022-09-12T20:16:11.836 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:11.837 DEBUG:boto:Data: 2022-09-12T20:16:11.837 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.838 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.838 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.839 DEBUG:boto:Params: {} 2022-09-12T20:16:11.839 DEBUG:boto:Token: None 2022-09-12T20:16:11.840 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:11.841 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q3DTwsyO0oWnKipcutLVK740320= 2022-09-12T20:16:11.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q3DTwsyO0oWnKipcutLVK740320=', 'Content-Length': '0'} 2022-09-12T20:16:11.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000341b74829c8f2df7-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.886 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:11.887 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:11.888 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:11.889 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:11.890 DEBUG:boto:Method: HEAD 2022-09-12T20:16:11.891 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:11.892 DEBUG:boto:Data: 2022-09-12T20:16:11.893 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.894 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.895 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.896 DEBUG:boto:Params: {} 2022-09-12T20:16:11.897 DEBUG:boto:Token: None 2022-09-12T20:16:11.898 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:11.899 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rpKxvdjdCwpJnUd8ytFQnG0Qleg= 2022-09-12T20:16:11.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rpKxvdjdCwpJnUd8ytFQnG0Qleg=', 'Content-Length': '0'} 2022-09-12T20:16:11.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb279259d9582fc5-00631f938b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.905 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:11.906 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:11.907 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:11.907 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:11.908 DEBUG:boto:Method: HEAD 2022-09-12T20:16:11.909 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:11.910 DEBUG:boto:Data: 2022-09-12T20:16:11.911 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.911 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.912 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.913 DEBUG:boto:Params: {} 2022-09-12T20:16:11.915 DEBUG:boto:Token: None 2022-09-12T20:16:11.916 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:11.917 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rpKxvdjdCwpJnUd8ytFQnG0Qleg= 2022-09-12T20:16:11.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rpKxvdjdCwpJnUd8ytFQnG0Qleg=', 'Content-Length': '0'} 2022-09-12T20:16:11.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fe5d6a3d547605e-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:16:11.923 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:11.924 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:11.924 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:11.925 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:11.925 DEBUG:boto:Method: GET 2022-09-12T20:16:11.926 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:11.927 DEBUG:boto:Data: 2022-09-12T20:16:11.928 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.929 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.929 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.930 DEBUG:boto:Params: {} 2022-09-12T20:16:11.930 DEBUG:boto:Token: None 2022-09-12T20:16:11.931 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:11.931 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q3DTwsyO0oWnKipcutLVK740320= 2022-09-12T20:16:11.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q3DTwsyO0oWnKipcutLVK740320=', 'Content-Length': '0'} 2022-09-12T20:16:11.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000617dcdd20eb1a0f2-00631f938b-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.935 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:11.936 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:11.936 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:11.937 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:11.937 DEBUG:boto:Method: GET 2022-09-12T20:16:11.938 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:11.939 DEBUG:boto:Data: 2022-09-12T20:16:11.939 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.940 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:11.940 DEBUG:boto:Port: 8000 2022-09-12T20:16:11.941 DEBUG:boto:Params: {} 2022-09-12T20:16:11.941 DEBUG:boto:Token: None 2022-09-12T20:16:11.942 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:11.942 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q3DTwsyO0oWnKipcutLVK740320= 2022-09-12T20:16:11.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q3DTwsyO0oWnKipcutLVK740320=', 'Content-Length': '0'} 2022-09-12T20:16:11.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee1520c9a1e65d17-00631f938b-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.946 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:11.946 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:11.947 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:16:11.948 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:16:11.948 DEBUG:boto:Method: GET 2022-09-12T20:16:11.949 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:16:11.949 DEBUG:boto:Data: 2022-09-12T20:16:11.950 DEBUG:boto:Headers: {} 2022-09-12T20:16:11.950 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:11.951 DEBUG:boto:Port: 8001 2022-09-12T20:16:11.952 DEBUG:boto:Params: {} 2022-09-12T20:16:11.952 DEBUG:boto:Token: None 2022-09-12T20:16:11.953 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:11 GMT /jdpiak-75/ 2022-09-12T20:16:11.953 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg= 2022-09-12T20:16:11.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:11 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eRGElaHMhuO/piWIDmR5KTi3bJg=', 'Content-Length': '0'} 2022-09-12T20:16:11.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f303ee2d4207ed0f-00631f938b-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:11.998 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:11.999 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:12.000 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:12.000 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:16:12.001 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:16:12.001 DEBUG:boto:Method: GET 2022-09-12T20:16:12.002 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:16:12.003 DEBUG:boto:Data: 2022-09-12T20:16:12.003 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.004 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.004 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.005 DEBUG:boto:Params: {} 2022-09-12T20:16:12.005 DEBUG:boto:Token: None 2022-09-12T20:16:12.006 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/ 2022-09-12T20:16:12.007 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA= 2022-09-12T20:16:12.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA=', 'Content-Length': '0'} 2022-09-12T20:16:12.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a13a82b98bf8a4e-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.013 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:12.014 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:12.014 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:12.015 DEBUG:boto:Method: HEAD 2022-09-12T20:16:12.015 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:16:12.016 DEBUG:boto:Data: 2022-09-12T20:16:12.016 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.017 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:12.017 DEBUG:boto:Port: 8001 2022-09-12T20:16:12.018 DEBUG:boto:Params: {} 2022-09-12T20:16:12.018 DEBUG:boto:Token: None 2022-09-12T20:16:12.019 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj44 2022-09-12T20:16:12.019 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ry8ussOOhcxNfMy6oMgwr9b2Rss= 2022-09-12T20:16:12.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ry8ussOOhcxNfMy6oMgwr9b2Rss=', 'Content-Length': '0'} 2022-09-12T20:16:12.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089f2f53433b6d612-00631f938c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.023 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:12.023 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:12.024 DEBUG:boto:Method: HEAD 2022-09-12T20:16:12.024 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:16:12.025 DEBUG:boto:Data: 2022-09-12T20:16:12.025 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.026 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.026 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.027 DEBUG:boto:Params: {} 2022-09-12T20:16:12.027 DEBUG:boto:Token: None 2022-09-12T20:16:12.028 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj44 2022-09-12T20:16:12.028 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ry8ussOOhcxNfMy6oMgwr9b2Rss= 2022-09-12T20:16:12.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ry8ussOOhcxNfMy6oMgwr9b2Rss=', 'Content-Length': '0'} 2022-09-12T20:16:12.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bfb48438c9fc923-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:16:12.032 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:12.033 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:12.033 DEBUG:boto:Method: GET 2022-09-12T20:16:12.034 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:16:12.035 DEBUG:boto:Data: 2022-09-12T20:16:12.035 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.036 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:12.036 DEBUG:boto:Port: 8001 2022-09-12T20:16:12.037 DEBUG:boto:Params: {} 2022-09-12T20:16:12.037 DEBUG:boto:Token: None 2022-09-12T20:16:12.038 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj44 2022-09-12T20:16:12.038 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ssTl0dNrF2rXfD221/FUOhVZnNc= 2022-09-12T20:16:12.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ssTl0dNrF2rXfD221/FUOhVZnNc=', 'Content-Length': '0'} 2022-09-12T20:16:12.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7ced7ce3a3c7b0b-00631f938c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.042 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:12.042 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:12.043 DEBUG:boto:Method: GET 2022-09-12T20:16:12.044 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:16:12.044 DEBUG:boto:Data: 2022-09-12T20:16:12.045 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.045 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.046 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.046 DEBUG:boto:Params: {} 2022-09-12T20:16:12.047 DEBUG:boto:Token: None 2022-09-12T20:16:12.047 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj44 2022-09-12T20:16:12.048 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ssTl0dNrF2rXfD221/FUOhVZnNc= 2022-09-12T20:16:12.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ssTl0dNrF2rXfD221/FUOhVZnNc=', 'Content-Length': '0'} 2022-09-12T20:16:12.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000406e8a12309c67e3-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.052 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:12.052 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:12.053 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:16:12.053 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:16:12.054 DEBUG:boto:Method: GET 2022-09-12T20:16:12.054 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:16:12.055 DEBUG:boto:Data: 2022-09-12T20:16:12.055 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.056 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:12.056 DEBUG:boto:Port: 8001 2022-09-12T20:16:12.057 DEBUG:boto:Params: {} 2022-09-12T20:16:12.057 DEBUG:boto:Token: None 2022-09-12T20:16:12.058 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/ 2022-09-12T20:16:12.059 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA= 2022-09-12T20:16:12.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA=', 'Content-Length': '0'} 2022-09-12T20:16:12.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aca256acf94a5346-00631f938c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.063 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:12.064 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:12.064 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:12.065 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:16:12.065 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:16:12.066 DEBUG:boto:Method: GET 2022-09-12T20:16:12.066 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:16:12.067 DEBUG:boto:Data: 2022-09-12T20:16:12.067 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.068 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.068 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.069 DEBUG:boto:Params: {} 2022-09-12T20:16:12.070 DEBUG:boto:Token: None 2022-09-12T20:16:12.070 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/ 2022-09-12T20:16:12.071 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA= 2022-09-12T20:16:12.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA=', 'Content-Length': '0'} 2022-09-12T20:16:12.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009a6599869ef3b21-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.118 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:12.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:16:12.120 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:12.120 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:12.121 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:12.121 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:12.122 DEBUG:boto:Method: GET 2022-09-12T20:16:12.122 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:12.123 DEBUG:boto:Data: 2022-09-12T20:16:12.123 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.124 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:12.124 DEBUG:boto:Port: 8001 2022-09-12T20:16:12.125 DEBUG:boto:Params: {} 2022-09-12T20:16:12.125 DEBUG:boto:Token: None 2022-09-12T20:16:12.126 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:12.127 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0xgwjsEnoH2LQvM8RnIwr2fwYlY= 2022-09-12T20:16:12.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0xgwjsEnoH2LQvM8RnIwr2fwYlY=', 'Content-Length': '0'} 2022-09-12T20:16:12.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046d4fde9f78172e4-00631f938c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.131 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:12.131 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:12.132 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:12.132 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:12.133 DEBUG:boto:Method: GET 2022-09-12T20:16:12.133 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:12.134 DEBUG:boto:Data: 2022-09-12T20:16:12.134 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.135 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.136 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.136 DEBUG:boto:Params: {} 2022-09-12T20:16:12.137 DEBUG:boto:Token: None 2022-09-12T20:16:12.138 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:12.138 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0xgwjsEnoH2LQvM8RnIwr2fwYlY= 2022-09-12T20:16:12.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0xgwjsEnoH2LQvM8RnIwr2fwYlY=', 'Content-Length': '0'} 2022-09-12T20:16:12.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a79be2bf381d986-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.142 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:12.142 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:12.143 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:12.143 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:12.144 DEBUG:boto:Method: HEAD 2022-09-12T20:16:12.144 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:12.145 DEBUG:boto:Data: 2022-09-12T20:16:12.145 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.146 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:12.146 DEBUG:boto:Port: 8001 2022-09-12T20:16:12.147 DEBUG:boto:Params: {} 2022-09-12T20:16:12.147 DEBUG:boto:Token: None 2022-09-12T20:16:12.148 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:12.149 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8ZCO6s6SMIMazrZVsACj1KyuWOw= 2022-09-12T20:16:12.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8ZCO6s6SMIMazrZVsACj1KyuWOw=', 'Content-Length': '0'} 2022-09-12T20:16:12.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abb479327c329d69-00631f938c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.152 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:12.153 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:12.153 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:12.154 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:12.154 DEBUG:boto:Method: HEAD 2022-09-12T20:16:12.155 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:12.155 DEBUG:boto:Data: 2022-09-12T20:16:12.156 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.156 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.157 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.157 DEBUG:boto:Params: {} 2022-09-12T20:16:12.158 DEBUG:boto:Token: None 2022-09-12T20:16:12.159 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:12.159 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8ZCO6s6SMIMazrZVsACj1KyuWOw= 2022-09-12T20:16:12.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8ZCO6s6SMIMazrZVsACj1KyuWOw=', 'Content-Length': '0'} 2022-09-12T20:16:12.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060de329eb42368eb-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:16:12.163 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:12.164 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:12.164 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:12.165 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:12.165 DEBUG:boto:Method: GET 2022-09-12T20:16:12.166 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:12.166 DEBUG:boto:Data: 2022-09-12T20:16:12.167 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.167 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:12.168 DEBUG:boto:Port: 8001 2022-09-12T20:16:12.168 DEBUG:boto:Params: {} 2022-09-12T20:16:12.169 DEBUG:boto:Token: None 2022-09-12T20:16:12.170 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:12.170 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0xgwjsEnoH2LQvM8RnIwr2fwYlY= 2022-09-12T20:16:12.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0xgwjsEnoH2LQvM8RnIwr2fwYlY=', 'Content-Length': '0'} 2022-09-12T20:16:12.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fc41ec20a70db25-00631f938c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.214 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:12.215 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:12.215 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:12.216 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:12.216 DEBUG:boto:Method: GET 2022-09-12T20:16:12.217 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:12.217 DEBUG:boto:Data: 2022-09-12T20:16:12.218 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.219 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.220 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.221 DEBUG:boto:Params: {} 2022-09-12T20:16:12.222 DEBUG:boto:Token: None 2022-09-12T20:16:12.222 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:12.223 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0xgwjsEnoH2LQvM8RnIwr2fwYlY= 2022-09-12T20:16:12.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0xgwjsEnoH2LQvM8RnIwr2fwYlY=', 'Content-Length': '0'} 2022-09-12T20:16:12.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000705ad58875884e72-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.227 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:12.227 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:12.228 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:16:12.228 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:16:12.229 DEBUG:boto:Method: GET 2022-09-12T20:16:12.230 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:16:12.231 DEBUG:boto:Data: 2022-09-12T20:16:12.232 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.232 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:12.233 DEBUG:boto:Port: 8001 2022-09-12T20:16:12.234 DEBUG:boto:Params: {} 2022-09-12T20:16:12.234 DEBUG:boto:Token: None 2022-09-12T20:16:12.235 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/ 2022-09-12T20:16:12.235 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA= 2022-09-12T20:16:12.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA=', 'Content-Length': '0'} 2022-09-12T20:16:12.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb92813ab07e603d-00631f938c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.240 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:12.240 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:12.241 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:12.241 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:16:12.242 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:16:12.243 DEBUG:boto:Method: GET 2022-09-12T20:16:12.243 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:16:12.244 DEBUG:boto:Data: 2022-09-12T20:16:12.244 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.245 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.245 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.246 DEBUG:boto:Params: {} 2022-09-12T20:16:12.246 DEBUG:boto:Token: None 2022-09-12T20:16:12.247 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/ 2022-09-12T20:16:12.247 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA= 2022-09-12T20:16:12.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA=', 'Content-Length': '0'} 2022-09-12T20:16:12.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000289be44582e0398a-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.254 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:12.255 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:12.255 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:12.256 DEBUG:boto:Method: HEAD 2022-09-12T20:16:12.257 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:16:12.257 DEBUG:boto:Data: 2022-09-12T20:16:12.258 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.259 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:12.259 DEBUG:boto:Port: 8001 2022-09-12T20:16:12.260 DEBUG:boto:Params: {} 2022-09-12T20:16:12.260 DEBUG:boto:Token: None 2022-09-12T20:16:12.261 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj46 2022-09-12T20:16:12.261 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/LyeM5xTDBu0MaGG7E9J3RmmTAs= 2022-09-12T20:16:12.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/LyeM5xTDBu0MaGG7E9J3RmmTAs=', 'Content-Length': '0'} 2022-09-12T20:16:12.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffaf61dca7ca3949-00631f938c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.265 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:12.265 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:12.266 DEBUG:boto:Method: HEAD 2022-09-12T20:16:12.266 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:16:12.267 DEBUG:boto:Data: 2022-09-12T20:16:12.267 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.268 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.268 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.269 DEBUG:boto:Params: {} 2022-09-12T20:16:12.269 DEBUG:boto:Token: None 2022-09-12T20:16:12.270 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj46 2022-09-12T20:16:12.271 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/LyeM5xTDBu0MaGG7E9J3RmmTAs= 2022-09-12T20:16:12.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/LyeM5xTDBu0MaGG7E9J3RmmTAs=', 'Content-Length': '0'} 2022-09-12T20:16:12.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000117e1a3a15df965b-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:16:12.274 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:12.275 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:12.275 DEBUG:boto:Method: GET 2022-09-12T20:16:12.276 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:16:12.276 DEBUG:boto:Data: 2022-09-12T20:16:12.277 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.278 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:12.278 DEBUG:boto:Port: 8001 2022-09-12T20:16:12.279 DEBUG:boto:Params: {} 2022-09-12T20:16:12.279 DEBUG:boto:Token: None 2022-09-12T20:16:12.280 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj46 2022-09-12T20:16:12.280 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KWtiedqgWQct+mlZ3LXugIp2Krk= 2022-09-12T20:16:12.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KWtiedqgWQct+mlZ3LXugIp2Krk=', 'Content-Length': '0'} 2022-09-12T20:16:12.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e81fc3965a480b97-00631f938c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.284 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:12.284 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:12.285 DEBUG:boto:Method: GET 2022-09-12T20:16:12.285 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:16:12.286 DEBUG:boto:Data: 2022-09-12T20:16:12.286 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.287 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.287 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.288 DEBUG:boto:Params: {} 2022-09-12T20:16:12.289 DEBUG:boto:Token: None 2022-09-12T20:16:12.289 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj46 2022-09-12T20:16:12.290 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KWtiedqgWQct+mlZ3LXugIp2Krk= 2022-09-12T20:16:12.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KWtiedqgWQct+mlZ3LXugIp2Krk=', 'Content-Length': '0'} 2022-09-12T20:16:12.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b49995544bd22e5d-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.338 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:12.339 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:12.340 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:16:12.341 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:16:12.342 DEBUG:boto:Method: GET 2022-09-12T20:16:12.343 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:16:12.344 DEBUG:boto:Data: 2022-09-12T20:16:12.345 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.345 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:12.346 DEBUG:boto:Port: 8001 2022-09-12T20:16:12.347 DEBUG:boto:Params: {} 2022-09-12T20:16:12.348 DEBUG:boto:Token: None 2022-09-12T20:16:12.349 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/ 2022-09-12T20:16:12.350 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA= 2022-09-12T20:16:12.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA=', 'Content-Length': '0'} 2022-09-12T20:16:12.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9042fffdfc94f17-00631f938c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.355 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:12.356 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:12.357 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:12.358 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:16:12.359 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:16:12.360 DEBUG:boto:Method: GET 2022-09-12T20:16:12.361 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:16:12.362 DEBUG:boto:Data: 2022-09-12T20:16:12.363 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.363 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.364 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.365 DEBUG:boto:Params: {} 2022-09-12T20:16:12.366 DEBUG:boto:Token: None 2022-09-12T20:16:12.367 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/ 2022-09-12T20:16:12.368 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA= 2022-09-12T20:16:12.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA=', 'Content-Length': '0'} 2022-09-12T20:16:12.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0fef0f4ff01b826-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.375 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:12.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:16:12.378 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:12.379 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:12.379 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:12.380 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:12.381 DEBUG:boto:Method: GET 2022-09-12T20:16:12.382 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:12.383 DEBUG:boto:Data: 2022-09-12T20:16:12.384 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.385 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:12.386 DEBUG:boto:Port: 8001 2022-09-12T20:16:12.387 DEBUG:boto:Params: {} 2022-09-12T20:16:12.388 DEBUG:boto:Token: None 2022-09-12T20:16:12.389 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:12.390 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UyIsQCtB3Jvv8AT8Io82bnn4CtQ= 2022-09-12T20:16:12.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UyIsQCtB3Jvv8AT8Io82bnn4CtQ=', 'Content-Length': '0'} 2022-09-12T20:16:12.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a3fe49f6876633d-00631f938c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.438 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:12.439 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:12.440 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:12.441 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:12.442 DEBUG:boto:Method: GET 2022-09-12T20:16:12.443 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:12.444 DEBUG:boto:Data: 2022-09-12T20:16:12.445 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.446 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.447 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.447 DEBUG:boto:Params: {} 2022-09-12T20:16:12.448 DEBUG:boto:Token: None 2022-09-12T20:16:12.449 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:12.450 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UyIsQCtB3Jvv8AT8Io82bnn4CtQ= 2022-09-12T20:16:12.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UyIsQCtB3Jvv8AT8Io82bnn4CtQ=', 'Content-Length': '0'} 2022-09-12T20:16:12.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037b0e7cd1fc1a4ad-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.456 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:12.457 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:12.457 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:12.458 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:12.459 DEBUG:boto:Method: HEAD 2022-09-12T20:16:12.460 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:12.461 DEBUG:boto:Data: 2022-09-12T20:16:12.462 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.463 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:12.463 DEBUG:boto:Port: 8001 2022-09-12T20:16:12.464 DEBUG:boto:Params: {} 2022-09-12T20:16:12.465 DEBUG:boto:Token: None 2022-09-12T20:16:12.466 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:12.467 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hLAXHRisfXDvWwnbYfbJZwKS1sY= 2022-09-12T20:16:12.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hLAXHRisfXDvWwnbYfbJZwKS1sY=', 'Content-Length': '0'} 2022-09-12T20:16:12.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016408702f3bde857-00631f938c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.472 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:12.473 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:12.474 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:12.474 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:12.475 DEBUG:boto:Method: HEAD 2022-09-12T20:16:12.476 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:12.477 DEBUG:boto:Data: 2022-09-12T20:16:12.478 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.479 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.480 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.480 DEBUG:boto:Params: {} 2022-09-12T20:16:12.481 DEBUG:boto:Token: None 2022-09-12T20:16:12.482 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:12.483 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hLAXHRisfXDvWwnbYfbJZwKS1sY= 2022-09-12T20:16:12.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hLAXHRisfXDvWwnbYfbJZwKS1sY=', 'Content-Length': '0'} 2022-09-12T20:16:12.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092f357b4803b19f3-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:16:12.489 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:12.490 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:12.491 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:12.492 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:12.492 DEBUG:boto:Method: GET 2022-09-12T20:16:12.493 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:12.495 DEBUG:boto:Data: 2022-09-12T20:16:12.496 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.497 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:12.498 DEBUG:boto:Port: 8001 2022-09-12T20:16:12.499 DEBUG:boto:Params: {} 2022-09-12T20:16:12.500 DEBUG:boto:Token: None 2022-09-12T20:16:12.501 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:12.502 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UyIsQCtB3Jvv8AT8Io82bnn4CtQ= 2022-09-12T20:16:12.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UyIsQCtB3Jvv8AT8Io82bnn4CtQ=', 'Content-Length': '0'} 2022-09-12T20:16:12.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a4a25532a89f842-00631f938c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.506 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:12.506 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:12.507 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:12.507 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:12.508 DEBUG:boto:Method: GET 2022-09-12T20:16:12.508 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:12.509 DEBUG:boto:Data: 2022-09-12T20:16:12.510 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.511 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.512 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.512 DEBUG:boto:Params: {} 2022-09-12T20:16:12.513 DEBUG:boto:Token: None 2022-09-12T20:16:12.514 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:12.515 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UyIsQCtB3Jvv8AT8Io82bnn4CtQ= 2022-09-12T20:16:12.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UyIsQCtB3Jvv8AT8Io82bnn4CtQ=', 'Content-Length': '0'} 2022-09-12T20:16:12.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066ef8be8d5d81c88-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.562 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:12.563 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:12.564 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:16:12.564 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:16:12.565 DEBUG:boto:Method: GET 2022-09-12T20:16:12.565 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:16:12.566 DEBUG:boto:Data: 2022-09-12T20:16:12.566 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.567 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:12.568 DEBUG:boto:Port: 8001 2022-09-12T20:16:12.568 DEBUG:boto:Params: {} 2022-09-12T20:16:12.569 DEBUG:boto:Token: None 2022-09-12T20:16:12.570 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/ 2022-09-12T20:16:12.570 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA= 2022-09-12T20:16:12.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA=', 'Content-Length': '0'} 2022-09-12T20:16:12.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd101646a21038c4-00631f938c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.575 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:12.576 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:12.576 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:12.577 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:16:12.577 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:16:12.578 DEBUG:boto:Method: GET 2022-09-12T20:16:12.578 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:16:12.579 DEBUG:boto:Data: 2022-09-12T20:16:12.579 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.580 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.580 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.581 DEBUG:boto:Params: {} 2022-09-12T20:16:12.582 DEBUG:boto:Token: None 2022-09-12T20:16:12.582 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/ 2022-09-12T20:16:12.583 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA= 2022-09-12T20:16:12.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA=', 'Content-Length': '0'} 2022-09-12T20:16:12.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080daf330c9be6d3b-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.589 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:12.590 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:12.590 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:12.591 DEBUG:boto:Method: HEAD 2022-09-12T20:16:12.591 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:16:12.592 DEBUG:boto:Data: 2022-09-12T20:16:12.593 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.593 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:12.594 DEBUG:boto:Port: 8001 2022-09-12T20:16:12.594 DEBUG:boto:Params: {} 2022-09-12T20:16:12.595 DEBUG:boto:Token: None 2022-09-12T20:16:12.595 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj51 2022-09-12T20:16:12.596 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EV7eINDz1m2TtqDHe8QRFSqX944= 2022-09-12T20:16:12.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EV7eINDz1m2TtqDHe8QRFSqX944=', 'Content-Length': '0'} 2022-09-12T20:16:12.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b66c910049bb991-00631f938c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.599 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:12.600 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:12.600 DEBUG:boto:Method: HEAD 2022-09-12T20:16:12.601 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:16:12.601 DEBUG:boto:Data: 2022-09-12T20:16:12.602 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.603 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.603 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.604 DEBUG:boto:Params: {} 2022-09-12T20:16:12.604 DEBUG:boto:Token: None 2022-09-12T20:16:12.605 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj51 2022-09-12T20:16:12.605 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EV7eINDz1m2TtqDHe8QRFSqX944= 2022-09-12T20:16:12.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EV7eINDz1m2TtqDHe8QRFSqX944=', 'Content-Length': '0'} 2022-09-12T20:16:12.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006962c8d783560922-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:16:12.609 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:12.610 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:12.610 DEBUG:boto:Method: GET 2022-09-12T20:16:12.611 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:16:12.611 DEBUG:boto:Data: 2022-09-12T20:16:12.612 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.613 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:12.613 DEBUG:boto:Port: 8001 2022-09-12T20:16:12.614 DEBUG:boto:Params: {} 2022-09-12T20:16:12.615 DEBUG:boto:Token: None 2022-09-12T20:16:12.616 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj51 2022-09-12T20:16:12.616 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WIND6YTymrUaS8qDrzQ2u75DsVI= 2022-09-12T20:16:12.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WIND6YTymrUaS8qDrzQ2u75DsVI=', 'Content-Length': '0'} 2022-09-12T20:16:12.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000163e81e4ea8a13a9-00631f938c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.662 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:12.663 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:12.664 DEBUG:boto:Method: GET 2022-09-12T20:16:12.664 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:16:12.665 DEBUG:boto:Data: 2022-09-12T20:16:12.665 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.666 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.667 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.667 DEBUG:boto:Params: {} 2022-09-12T20:16:12.668 DEBUG:boto:Token: None 2022-09-12T20:16:12.669 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj51 2022-09-12T20:16:12.670 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WIND6YTymrUaS8qDrzQ2u75DsVI= 2022-09-12T20:16:12.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WIND6YTymrUaS8qDrzQ2u75DsVI=', 'Content-Length': '0'} 2022-09-12T20:16:12.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053d7b5989e48e42d-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.673 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:12.674 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:12.674 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:16:12.675 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:16:12.675 DEBUG:boto:Method: GET 2022-09-12T20:16:12.676 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:16:12.677 DEBUG:boto:Data: 2022-09-12T20:16:12.677 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.678 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:12.678 DEBUG:boto:Port: 8001 2022-09-12T20:16:12.679 DEBUG:boto:Params: {} 2022-09-12T20:16:12.679 DEBUG:boto:Token: None 2022-09-12T20:16:12.680 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/ 2022-09-12T20:16:12.680 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA= 2022-09-12T20:16:12.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA=', 'Content-Length': '0'} 2022-09-12T20:16:12.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000faed4f81784c1374-00631f938c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.685 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:12.685 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:12.686 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:12.686 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:16:12.687 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:16:12.688 DEBUG:boto:Method: GET 2022-09-12T20:16:12.688 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:16:12.689 DEBUG:boto:Data: 2022-09-12T20:16:12.689 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.690 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.690 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.691 DEBUG:boto:Params: {} 2022-09-12T20:16:12.691 DEBUG:boto:Token: None 2022-09-12T20:16:12.692 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/ 2022-09-12T20:16:12.692 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA= 2022-09-12T20:16:12.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA=', 'Content-Length': '0'} 2022-09-12T20:16:12.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c344311e57004f8-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.698 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:12.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:16:12.699 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:12.700 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:12.700 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:12.701 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:12.702 DEBUG:boto:Method: GET 2022-09-12T20:16:12.702 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:12.703 DEBUG:boto:Data: 2022-09-12T20:16:12.703 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.704 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:12.704 DEBUG:boto:Port: 8001 2022-09-12T20:16:12.705 DEBUG:boto:Params: {} 2022-09-12T20:16:12.705 DEBUG:boto:Token: None 2022-09-12T20:16:12.706 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:12.707 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9rSK3MN1960kIeCXxNzZZUiipJw= 2022-09-12T20:16:12.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9rSK3MN1960kIeCXxNzZZUiipJw=', 'Content-Length': '0'} 2022-09-12T20:16:12.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029ed6159976f9eac-00631f938c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.710 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:12.711 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:12.712 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:12.712 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:12.713 DEBUG:boto:Method: GET 2022-09-12T20:16:12.713 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:12.714 DEBUG:boto:Data: 2022-09-12T20:16:12.714 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.715 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.716 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.716 DEBUG:boto:Params: {} 2022-09-12T20:16:12.717 DEBUG:boto:Token: None 2022-09-12T20:16:12.717 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:12.718 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9rSK3MN1960kIeCXxNzZZUiipJw= 2022-09-12T20:16:12.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9rSK3MN1960kIeCXxNzZZUiipJw=', 'Content-Length': '0'} 2022-09-12T20:16:12.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dad3d6852805c40a-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.722 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:12.722 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:12.723 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:12.723 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:12.724 DEBUG:boto:Method: HEAD 2022-09-12T20:16:12.724 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:12.725 DEBUG:boto:Data: 2022-09-12T20:16:12.725 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.726 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:12.727 DEBUG:boto:Port: 8001 2022-09-12T20:16:12.727 DEBUG:boto:Params: {} 2022-09-12T20:16:12.728 DEBUG:boto:Token: None 2022-09-12T20:16:12.728 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:12.729 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:daZwkSNkJeWora7FGp4ohZoJaaA= 2022-09-12T20:16:12.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:daZwkSNkJeWora7FGp4ohZoJaaA=', 'Content-Length': '0'} 2022-09-12T20:16:12.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055e38d1cc4fe0014-00631f938c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.732 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:12.733 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:12.733 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:12.734 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:12.744 DEBUG:boto:Method: HEAD 2022-09-12T20:16:12.745 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:12.745 DEBUG:boto:Data: 2022-09-12T20:16:12.746 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.746 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.747 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.747 DEBUG:boto:Params: {} 2022-09-12T20:16:12.748 DEBUG:boto:Token: None 2022-09-12T20:16:12.748 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:12.749 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:daZwkSNkJeWora7FGp4ohZoJaaA= 2022-09-12T20:16:12.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:daZwkSNkJeWora7FGp4ohZoJaaA=', 'Content-Length': '0'} 2022-09-12T20:16:12.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009081b80aa6aef24d-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:16:12.753 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:12.754 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:12.754 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:12.755 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:12.755 DEBUG:boto:Method: GET 2022-09-12T20:16:12.756 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:12.756 DEBUG:boto:Data: 2022-09-12T20:16:12.757 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.757 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:12.758 DEBUG:boto:Port: 8001 2022-09-12T20:16:12.758 DEBUG:boto:Params: {} 2022-09-12T20:16:12.759 DEBUG:boto:Token: None 2022-09-12T20:16:12.760 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:12.760 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9rSK3MN1960kIeCXxNzZZUiipJw= 2022-09-12T20:16:12.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9rSK3MN1960kIeCXxNzZZUiipJw=', 'Content-Length': '0'} 2022-09-12T20:16:12.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bddfaafa573ebb4-00631f938c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.764 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:12.764 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:12.765 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:12.765 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:12.766 DEBUG:boto:Method: GET 2022-09-12T20:16:12.766 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:12.767 DEBUG:boto:Data: 2022-09-12T20:16:12.768 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.768 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.769 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.769 DEBUG:boto:Params: {} 2022-09-12T20:16:12.770 DEBUG:boto:Token: None 2022-09-12T20:16:12.770 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:12.771 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9rSK3MN1960kIeCXxNzZZUiipJw= 2022-09-12T20:16:12.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9rSK3MN1960kIeCXxNzZZUiipJw=', 'Content-Length': '0'} 2022-09-12T20:16:12.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088836e09374ec12b-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.774 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:12.775 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:12.775 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:16:12.776 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:16:12.776 DEBUG:boto:Method: GET 2022-09-12T20:16:12.777 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:16:12.778 DEBUG:boto:Data: 2022-09-12T20:16:12.778 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.779 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:12.779 DEBUG:boto:Port: 8001 2022-09-12T20:16:12.780 DEBUG:boto:Params: {} 2022-09-12T20:16:12.780 DEBUG:boto:Token: None 2022-09-12T20:16:12.781 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/ 2022-09-12T20:16:12.782 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA= 2022-09-12T20:16:12.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA=', 'Content-Length': '0'} 2022-09-12T20:16:12.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d74061582a6fda5d-00631f938c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.786 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:12.787 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:12.787 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:12.788 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:16:12.788 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:16:12.789 DEBUG:boto:Method: GET 2022-09-12T20:16:12.789 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:16:12.790 DEBUG:boto:Data: 2022-09-12T20:16:12.791 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.791 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.792 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.792 DEBUG:boto:Params: {} 2022-09-12T20:16:12.793 DEBUG:boto:Token: None 2022-09-12T20:16:12.793 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/ 2022-09-12T20:16:12.794 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA= 2022-09-12T20:16:12.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA=', 'Content-Length': '0'} 2022-09-12T20:16:12.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012f7117369597afd-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.800 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:12.801 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:12.801 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:12.802 DEBUG:boto:Method: HEAD 2022-09-12T20:16:12.803 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:16:12.803 DEBUG:boto:Data: 2022-09-12T20:16:12.804 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.804 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:12.805 DEBUG:boto:Port: 8001 2022-09-12T20:16:12.805 DEBUG:boto:Params: {} 2022-09-12T20:16:12.806 DEBUG:boto:Token: None 2022-09-12T20:16:12.806 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj55 2022-09-12T20:16:12.807 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3hIFIFxMVs/Il1J+5DGl4g1DkGc= 2022-09-12T20:16:12.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3hIFIFxMVs/Il1J+5DGl4g1DkGc=', 'Content-Length': '0'} 2022-09-12T20:16:12.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010c2da6d8e197efe-00631f938c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.811 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:12.811 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:12.812 DEBUG:boto:Method: HEAD 2022-09-12T20:16:12.812 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:16:12.813 DEBUG:boto:Data: 2022-09-12T20:16:12.813 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.814 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.814 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.815 DEBUG:boto:Params: {} 2022-09-12T20:16:12.816 DEBUG:boto:Token: None 2022-09-12T20:16:12.816 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj55 2022-09-12T20:16:12.817 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3hIFIFxMVs/Il1J+5DGl4g1DkGc= 2022-09-12T20:16:12.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3hIFIFxMVs/Il1J+5DGl4g1DkGc=', 'Content-Length': '0'} 2022-09-12T20:16:12.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000861e99346d407bca-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:16:12.821 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:12.822 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:12.822 DEBUG:boto:Method: GET 2022-09-12T20:16:12.823 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:16:12.823 DEBUG:boto:Data: 2022-09-12T20:16:12.824 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.825 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:12.825 DEBUG:boto:Port: 8001 2022-09-12T20:16:12.826 DEBUG:boto:Params: {} 2022-09-12T20:16:12.826 DEBUG:boto:Token: None 2022-09-12T20:16:12.827 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj55 2022-09-12T20:16:12.827 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FSzHJbSp0/6PzhOg2oF9o5Bgnk4= 2022-09-12T20:16:12.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FSzHJbSp0/6PzhOg2oF9o5Bgnk4=', 'Content-Length': '0'} 2022-09-12T20:16:12.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006446702387c043dc-00631f938c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.831 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:12.831 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:12.832 DEBUG:boto:Method: GET 2022-09-12T20:16:12.833 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:16:12.833 DEBUG:boto:Data: 2022-09-12T20:16:12.834 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.834 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.835 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.835 DEBUG:boto:Params: {} 2022-09-12T20:16:12.836 DEBUG:boto:Token: None 2022-09-12T20:16:12.837 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj55 2022-09-12T20:16:12.837 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FSzHJbSp0/6PzhOg2oF9o5Bgnk4= 2022-09-12T20:16:12.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FSzHJbSp0/6PzhOg2oF9o5Bgnk4=', 'Content-Length': '0'} 2022-09-12T20:16:12.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c675658f70eb2609-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.886 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:12.887 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:12.887 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:16:12.888 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:16:12.888 DEBUG:boto:Method: GET 2022-09-12T20:16:12.889 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:16:12.889 DEBUG:boto:Data: 2022-09-12T20:16:12.890 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.891 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:12.891 DEBUG:boto:Port: 8001 2022-09-12T20:16:12.892 DEBUG:boto:Params: {} 2022-09-12T20:16:12.893 DEBUG:boto:Token: None 2022-09-12T20:16:12.894 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/ 2022-09-12T20:16:12.895 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA= 2022-09-12T20:16:12.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA=', 'Content-Length': '0'} 2022-09-12T20:16:12.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042a31a7939b2dcec-00631f938c-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.901 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:12.901 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:12.902 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:12.903 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:16:12.903 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:16:12.904 DEBUG:boto:Method: GET 2022-09-12T20:16:12.904 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:16:12.905 DEBUG:boto:Data: 2022-09-12T20:16:12.905 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.906 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.906 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.907 DEBUG:boto:Params: {} 2022-09-12T20:16:12.908 DEBUG:boto:Token: None 2022-09-12T20:16:12.908 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/ 2022-09-12T20:16:12.909 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA= 2022-09-12T20:16:12.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+XLgsFBAIhqoLnoZ+reE6Ae4FuA=', 'Content-Length': '0'} 2022-09-12T20:16:12.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af379d5f1b9ddb60-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.915 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:12.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:16:12.918 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:12.919 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:12.920 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:12.920 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:12.921 DEBUG:boto:Method: GET 2022-09-12T20:16:12.922 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:12.923 DEBUG:boto:Data: 2022-09-12T20:16:12.924 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.925 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:12.926 DEBUG:boto:Port: 8001 2022-09-12T20:16:12.927 DEBUG:boto:Params: {} 2022-09-12T20:16:12.928 DEBUG:boto:Token: None 2022-09-12T20:16:12.929 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:12.930 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D7eUKPBroRFn1cEFJ0v0+yOeXuM= 2022-09-12T20:16:12.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D7eUKPBroRFn1cEFJ0v0+yOeXuM=', 'Content-Length': '0'} 2022-09-12T20:16:12.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9a486067a2770b3-00631f938c-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.979 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:12.980 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:12.981 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:12.982 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:12.983 DEBUG:boto:Method: GET 2022-09-12T20:16:12.984 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:12.985 DEBUG:boto:Data: 2022-09-12T20:16:12.986 DEBUG:boto:Headers: {} 2022-09-12T20:16:12.987 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:12.988 DEBUG:boto:Port: 8000 2022-09-12T20:16:12.989 DEBUG:boto:Params: {} 2022-09-12T20:16:12.990 DEBUG:boto:Token: None 2022-09-12T20:16:12.991 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:12 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:12.992 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:D7eUKPBroRFn1cEFJ0v0+yOeXuM= 2022-09-12T20:16:12.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:12 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:D7eUKPBroRFn1cEFJ0v0+yOeXuM=', 'Content-Length': '0'} 2022-09-12T20:16:12.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cb1c85c44eb9f52-00631f938c-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:12.998 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:12.999 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:13.000 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:13.001 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:13.002 DEBUG:boto:Method: HEAD 2022-09-12T20:16:13.003 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:13.004 DEBUG:boto:Data: 2022-09-12T20:16:13.005 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.006 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.007 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.008 DEBUG:boto:Params: {} 2022-09-12T20:16:13.009 DEBUG:boto:Token: None 2022-09-12T20:16:13.010 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:13.011 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kRMQbYyI3iYXqJwRBTCZwLepYTg= 2022-09-12T20:16:13.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kRMQbYyI3iYXqJwRBTCZwLepYTg=', 'Content-Length': '0'} 2022-09-12T20:16:13.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006edb4ed3c4bcff3c-00631f938d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.015 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:13.015 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:13.016 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:13.016 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:13.017 DEBUG:boto:Method: HEAD 2022-09-12T20:16:13.017 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:13.018 DEBUG:boto:Data: 2022-09-12T20:16:13.018 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.019 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.019 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.020 DEBUG:boto:Params: {} 2022-09-12T20:16:13.021 DEBUG:boto:Token: None 2022-09-12T20:16:13.021 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:13.022 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kRMQbYyI3iYXqJwRBTCZwLepYTg= 2022-09-12T20:16:13.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kRMQbYyI3iYXqJwRBTCZwLepYTg=', 'Content-Length': '0'} 2022-09-12T20:16:13.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001da460032ea5eb91-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:16:13.026 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:13.027 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:13.027 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:13.028 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:13.028 DEBUG:boto:Method: GET 2022-09-12T20:16:13.029 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:13.029 DEBUG:boto:Data: 2022-09-12T20:16:13.030 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.030 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.031 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.031 DEBUG:boto:Params: {} 2022-09-12T20:16:13.032 DEBUG:boto:Token: None 2022-09-12T20:16:13.033 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:13.034 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zZJQqVGogg97Y3bheFw/MWWSXcE= 2022-09-12T20:16:13.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zZJQqVGogg97Y3bheFw/MWWSXcE=', 'Content-Length': '0'} 2022-09-12T20:16:13.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc7451d1bf818f52-00631f938d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.037 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:13.038 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:13.039 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:13.039 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:13.040 DEBUG:boto:Method: GET 2022-09-12T20:16:13.041 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:13.042 DEBUG:boto:Data: 2022-09-12T20:16:13.043 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.043 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.044 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.045 DEBUG:boto:Params: {} 2022-09-12T20:16:13.045 DEBUG:boto:Token: None 2022-09-12T20:16:13.046 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:13.047 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zZJQqVGogg97Y3bheFw/MWWSXcE= 2022-09-12T20:16:13.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zZJQqVGogg97Y3bheFw/MWWSXcE=', 'Content-Length': '0'} 2022-09-12T20:16:13.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fde898696b53ef5-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.051 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:13.052 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:13.052 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:16:13.053 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:16:13.053 DEBUG:boto:Method: GET 2022-09-12T20:16:13.054 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:16:13.054 DEBUG:boto:Data: 2022-09-12T20:16:13.055 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.055 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.056 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.057 DEBUG:boto:Params: {} 2022-09-12T20:16:13.057 DEBUG:boto:Token: None 2022-09-12T20:16:13.058 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/ 2022-09-12T20:16:13.058 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ= 2022-09-12T20:16:13.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ=', 'Content-Length': '0'} 2022-09-12T20:16:13.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ab0f414e40dfc28-00631f938d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.064 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:13.065 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:13.066 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:13.066 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:16:13.067 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:16:13.067 DEBUG:boto:Method: GET 2022-09-12T20:16:13.068 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:16:13.068 DEBUG:boto:Data: 2022-09-12T20:16:13.069 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.069 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.070 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.071 DEBUG:boto:Params: {} 2022-09-12T20:16:13.071 DEBUG:boto:Token: None 2022-09-12T20:16:13.072 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/ 2022-09-12T20:16:13.073 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ= 2022-09-12T20:16:13.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ=', 'Content-Length': '0'} 2022-09-12T20:16:13.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcc3d66e0759b8bb-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.079 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:13.079 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:13.080 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:13.081 DEBUG:boto:Method: HEAD 2022-09-12T20:16:13.081 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:16:13.082 DEBUG:boto:Data: 2022-09-12T20:16:13.082 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.083 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.083 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.084 DEBUG:boto:Params: {} 2022-09-12T20:16:13.084 DEBUG:boto:Token: None 2022-09-12T20:16:13.085 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj57 2022-09-12T20:16:13.086 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:w+nPnPZ8+3IZ4cVQlSHQ4a8ep6E= 2022-09-12T20:16:13.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:w+nPnPZ8+3IZ4cVQlSHQ4a8ep6E=', 'Content-Length': '0'} 2022-09-12T20:16:13.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000186616e8cb3c3507-00631f938d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.089 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:13.090 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:13.090 DEBUG:boto:Method: HEAD 2022-09-12T20:16:13.091 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:16:13.091 DEBUG:boto:Data: 2022-09-12T20:16:13.092 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.092 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.093 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.093 DEBUG:boto:Params: {} 2022-09-12T20:16:13.094 DEBUG:boto:Token: None 2022-09-12T20:16:13.095 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj57 2022-09-12T20:16:13.095 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:w+nPnPZ8+3IZ4cVQlSHQ4a8ep6E= 2022-09-12T20:16:13.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:w+nPnPZ8+3IZ4cVQlSHQ4a8ep6E=', 'Content-Length': '0'} 2022-09-12T20:16:13.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006341b8c75292420-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:16:13.099 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:13.100 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:13.100 DEBUG:boto:Method: GET 2022-09-12T20:16:13.101 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:16:13.101 DEBUG:boto:Data: 2022-09-12T20:16:13.102 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.102 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.103 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.103 DEBUG:boto:Params: {} 2022-09-12T20:16:13.104 DEBUG:boto:Token: None 2022-09-12T20:16:13.105 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj57 2022-09-12T20:16:13.105 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4DitBqTi3ExVj2lM/qdarmDvxGw= 2022-09-12T20:16:13.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4DitBqTi3ExVj2lM/qdarmDvxGw=', 'Content-Length': '0'} 2022-09-12T20:16:13.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003993a156303ca4dc-00631f938d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.150 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:13.151 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:13.151 DEBUG:boto:Method: GET 2022-09-12T20:16:13.152 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:16:13.153 DEBUG:boto:Data: 2022-09-12T20:16:13.153 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.154 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.154 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.155 DEBUG:boto:Params: {} 2022-09-12T20:16:13.155 DEBUG:boto:Token: None 2022-09-12T20:16:13.156 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj57 2022-09-12T20:16:13.157 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4DitBqTi3ExVj2lM/qdarmDvxGw= 2022-09-12T20:16:13.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4DitBqTi3ExVj2lM/qdarmDvxGw=', 'Content-Length': '0'} 2022-09-12T20:16:13.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaa689df0cb3b3b1-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.161 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:13.161 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:13.162 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:16:13.162 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:16:13.163 DEBUG:boto:Method: GET 2022-09-12T20:16:13.163 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:16:13.164 DEBUG:boto:Data: 2022-09-12T20:16:13.164 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.165 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.165 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.166 DEBUG:boto:Params: {} 2022-09-12T20:16:13.166 DEBUG:boto:Token: None 2022-09-12T20:16:13.167 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/ 2022-09-12T20:16:13.168 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ= 2022-09-12T20:16:13.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ=', 'Content-Length': '0'} 2022-09-12T20:16:13.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ba3b081c0a5f6c4-00631f938d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.172 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:13.172 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:13.173 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:13.174 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:16:13.174 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:16:13.175 DEBUG:boto:Method: GET 2022-09-12T20:16:13.175 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:16:13.176 DEBUG:boto:Data: 2022-09-12T20:16:13.176 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.177 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.177 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.178 DEBUG:boto:Params: {} 2022-09-12T20:16:13.178 DEBUG:boto:Token: None 2022-09-12T20:16:13.179 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/ 2022-09-12T20:16:13.179 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ= 2022-09-12T20:16:13.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ=', 'Content-Length': '0'} 2022-09-12T20:16:13.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6cc62d42daabbbe-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.185 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:13.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:16:13.186 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:13.187 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:13.187 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:13.188 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:13.188 DEBUG:boto:Method: GET 2022-09-12T20:16:13.189 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:13.189 DEBUG:boto:Data: 2022-09-12T20:16:13.190 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.190 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.191 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.191 DEBUG:boto:Params: {} 2022-09-12T20:16:13.192 DEBUG:boto:Token: None 2022-09-12T20:16:13.193 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:13.193 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KWDTnZ7UgJD/U8GMnTv/K7aSZ1s= 2022-09-12T20:16:13.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KWDTnZ7UgJD/U8GMnTv/K7aSZ1s=', 'Content-Length': '0'} 2022-09-12T20:16:13.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000063a24448d5dd156-00631f938d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.197 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:13.198 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:13.198 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:13.199 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:13.199 DEBUG:boto:Method: GET 2022-09-12T20:16:13.200 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:13.201 DEBUG:boto:Data: 2022-09-12T20:16:13.201 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.202 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.202 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.203 DEBUG:boto:Params: {} 2022-09-12T20:16:13.203 DEBUG:boto:Token: None 2022-09-12T20:16:13.204 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:13.204 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KWDTnZ7UgJD/U8GMnTv/K7aSZ1s= 2022-09-12T20:16:13.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KWDTnZ7UgJD/U8GMnTv/K7aSZ1s=', 'Content-Length': '0'} 2022-09-12T20:16:13.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4f134c01bba35e0-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.250 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:13.251 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:13.251 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:13.252 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:13.252 DEBUG:boto:Method: HEAD 2022-09-12T20:16:13.253 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:13.253 DEBUG:boto:Data: 2022-09-12T20:16:13.254 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.255 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.255 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.256 DEBUG:boto:Params: {} 2022-09-12T20:16:13.256 DEBUG:boto:Token: None 2022-09-12T20:16:13.257 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:13.258 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iX9H5H4M1U2elHt7dj4m8AWF0K0= 2022-09-12T20:16:13.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iX9H5H4M1U2elHt7dj4m8AWF0K0=', 'Content-Length': '0'} 2022-09-12T20:16:13.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fd91d55aec58d1e-00631f938d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.262 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:13.262 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:13.263 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:13.263 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:13.264 DEBUG:boto:Method: HEAD 2022-09-12T20:16:13.264 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:13.265 DEBUG:boto:Data: 2022-09-12T20:16:13.265 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.266 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.266 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.267 DEBUG:boto:Params: {} 2022-09-12T20:16:13.268 DEBUG:boto:Token: None 2022-09-12T20:16:13.268 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:13.269 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:iX9H5H4M1U2elHt7dj4m8AWF0K0= 2022-09-12T20:16:13.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:iX9H5H4M1U2elHt7dj4m8AWF0K0=', 'Content-Length': '0'} 2022-09-12T20:16:13.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5aceb2be055f433-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:16:13.273 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:13.274 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:13.274 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:13.275 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:13.275 DEBUG:boto:Method: GET 2022-09-12T20:16:13.276 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:13.276 DEBUG:boto:Data: 2022-09-12T20:16:13.277 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.277 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.278 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.279 DEBUG:boto:Params: {} 2022-09-12T20:16:13.279 DEBUG:boto:Token: None 2022-09-12T20:16:13.280 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:13.281 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KWDTnZ7UgJD/U8GMnTv/K7aSZ1s= 2022-09-12T20:16:13.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KWDTnZ7UgJD/U8GMnTv/K7aSZ1s=', 'Content-Length': '0'} 2022-09-12T20:16:13.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fb1519826046ace-00631f938d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.285 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:13.285 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:13.286 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:13.286 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:13.287 DEBUG:boto:Method: GET 2022-09-12T20:16:13.287 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:13.288 DEBUG:boto:Data: 2022-09-12T20:16:13.289 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.289 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.290 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.290 DEBUG:boto:Params: {} 2022-09-12T20:16:13.291 DEBUG:boto:Token: None 2022-09-12T20:16:13.292 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:13.292 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KWDTnZ7UgJD/U8GMnTv/K7aSZ1s= 2022-09-12T20:16:13.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KWDTnZ7UgJD/U8GMnTv/K7aSZ1s=', 'Content-Length': '0'} 2022-09-12T20:16:13.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f9c26f50b144c22-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.296 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:13.297 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:13.297 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:16:13.298 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:16:13.299 DEBUG:boto:Method: GET 2022-09-12T20:16:13.299 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:16:13.300 DEBUG:boto:Data: 2022-09-12T20:16:13.300 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.301 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.301 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.302 DEBUG:boto:Params: {} 2022-09-12T20:16:13.302 DEBUG:boto:Token: None 2022-09-12T20:16:13.303 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/ 2022-09-12T20:16:13.304 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ= 2022-09-12T20:16:13.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ=', 'Content-Length': '0'} 2022-09-12T20:16:13.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c964a09cb54f446-00631f938d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.308 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:13.309 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:13.310 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:13.310 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:16:13.311 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:16:13.311 DEBUG:boto:Method: GET 2022-09-12T20:16:13.312 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:16:13.313 DEBUG:boto:Data: 2022-09-12T20:16:13.313 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.314 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.315 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.315 DEBUG:boto:Params: {} 2022-09-12T20:16:13.316 DEBUG:boto:Token: None 2022-09-12T20:16:13.316 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/ 2022-09-12T20:16:13.317 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ= 2022-09-12T20:16:13.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ=', 'Content-Length': '0'} 2022-09-12T20:16:13.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ccd8ae9e180346c-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.325 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:13.326 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:13.327 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:13.328 DEBUG:boto:Method: HEAD 2022-09-12T20:16:13.329 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:16:13.330 DEBUG:boto:Data: 2022-09-12T20:16:13.331 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.332 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.333 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.334 DEBUG:boto:Params: {} 2022-09-12T20:16:13.335 DEBUG:boto:Token: None 2022-09-12T20:16:13.336 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj58 2022-09-12T20:16:13.337 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SEJnw+vbEpbV2/5r502gvf5jEV8= 2022-09-12T20:16:13.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SEJnw+vbEpbV2/5r502gvf5jEV8=', 'Content-Length': '0'} 2022-09-12T20:16:13.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000138dc29c31906acc-00631f938d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.341 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:13.342 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:13.343 DEBUG:boto:Method: HEAD 2022-09-12T20:16:13.343 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:16:13.344 DEBUG:boto:Data: 2022-09-12T20:16:13.344 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.345 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.345 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.346 DEBUG:boto:Params: {} 2022-09-12T20:16:13.347 DEBUG:boto:Token: None 2022-09-12T20:16:13.347 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj58 2022-09-12T20:16:13.348 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SEJnw+vbEpbV2/5r502gvf5jEV8= 2022-09-12T20:16:13.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SEJnw+vbEpbV2/5r502gvf5jEV8=', 'Content-Length': '0'} 2022-09-12T20:16:13.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a31152027ffbb0ad-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:16:13.353 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:13.353 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:13.354 DEBUG:boto:Method: GET 2022-09-12T20:16:13.354 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:16:13.355 DEBUG:boto:Data: 2022-09-12T20:16:13.355 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.356 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.356 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.357 DEBUG:boto:Params: {} 2022-09-12T20:16:13.358 DEBUG:boto:Token: None 2022-09-12T20:16:13.358 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj58 2022-09-12T20:16:13.359 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:luqiimGtCEt15WcilrOXgZkmDxQ= 2022-09-12T20:16:13.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:luqiimGtCEt15WcilrOXgZkmDxQ=', 'Content-Length': '0'} 2022-09-12T20:16:13.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faef5d6085fd32ec-00631f938d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.402 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:13.403 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:13.404 DEBUG:boto:Method: GET 2022-09-12T20:16:13.404 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:16:13.405 DEBUG:boto:Data: 2022-09-12T20:16:13.405 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.406 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.406 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.407 DEBUG:boto:Params: {} 2022-09-12T20:16:13.408 DEBUG:boto:Token: None 2022-09-12T20:16:13.408 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj58 2022-09-12T20:16:13.409 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:luqiimGtCEt15WcilrOXgZkmDxQ= 2022-09-12T20:16:13.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:luqiimGtCEt15WcilrOXgZkmDxQ=', 'Content-Length': '0'} 2022-09-12T20:16:13.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d76f682a48a13ce-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.413 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:13.414 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:13.415 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:16:13.416 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:16:13.417 DEBUG:boto:Method: GET 2022-09-12T20:16:13.418 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:16:13.419 DEBUG:boto:Data: 2022-09-12T20:16:13.420 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.420 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.421 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.421 DEBUG:boto:Params: {} 2022-09-12T20:16:13.422 DEBUG:boto:Token: None 2022-09-12T20:16:13.423 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/ 2022-09-12T20:16:13.424 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ= 2022-09-12T20:16:13.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ=', 'Content-Length': '0'} 2022-09-12T20:16:13.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7b366d7677fd101-00631f938d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.428 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:13.429 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:13.430 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:13.430 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:16:13.431 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:16:13.431 DEBUG:boto:Method: GET 2022-09-12T20:16:13.432 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:16:13.432 DEBUG:boto:Data: 2022-09-12T20:16:13.433 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.433 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.434 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.435 DEBUG:boto:Params: {} 2022-09-12T20:16:13.435 DEBUG:boto:Token: None 2022-09-12T20:16:13.436 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/ 2022-09-12T20:16:13.437 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ= 2022-09-12T20:16:13.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ=', 'Content-Length': '0'} 2022-09-12T20:16:13.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dfc5777abb4c5a1-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.443 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:13.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-12T20:16:13.444 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:13.445 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:13.445 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:13.446 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:13.446 DEBUG:boto:Method: GET 2022-09-12T20:16:13.447 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:13.447 DEBUG:boto:Data: 2022-09-12T20:16:13.448 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.449 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.449 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.450 DEBUG:boto:Params: {} 2022-09-12T20:16:13.451 DEBUG:boto:Token: None 2022-09-12T20:16:13.451 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:13.452 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fW7irddfjUaUCgY10RGoOhI/454= 2022-09-12T20:16:13.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fW7irddfjUaUCgY10RGoOhI/454=', 'Content-Length': '0'} 2022-09-12T20:16:13.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000729c071ef8c0a9c3-00631f938d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.456 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:13.456 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:13.457 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:13.457 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:13.458 DEBUG:boto:Method: GET 2022-09-12T20:16:13.459 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:13.460 DEBUG:boto:Data: 2022-09-12T20:16:13.460 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.461 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.462 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.463 DEBUG:boto:Params: {} 2022-09-12T20:16:13.465 DEBUG:boto:Token: None 2022-09-12T20:16:13.466 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:13.467 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fW7irddfjUaUCgY10RGoOhI/454= 2022-09-12T20:16:13.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fW7irddfjUaUCgY10RGoOhI/454=', 'Content-Length': '0'} 2022-09-12T20:16:13.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000661a8d3319ef9ab7-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.514 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:13.515 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:13.516 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:13.516 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:13.517 DEBUG:boto:Method: HEAD 2022-09-12T20:16:13.517 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:13.518 DEBUG:boto:Data: 2022-09-12T20:16:13.518 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.519 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.519 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.520 DEBUG:boto:Params: {} 2022-09-12T20:16:13.521 DEBUG:boto:Token: None 2022-09-12T20:16:13.521 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:13.522 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rQXgp3f881DqNAwZC6u7J9m1djA= 2022-09-12T20:16:13.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rQXgp3f881DqNAwZC6u7J9m1djA=', 'Content-Length': '0'} 2022-09-12T20:16:13.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009886b43405291b06-00631f938d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.525 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:13.526 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:13.527 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:13.527 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:13.528 DEBUG:boto:Method: HEAD 2022-09-12T20:16:13.528 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:13.529 DEBUG:boto:Data: 2022-09-12T20:16:13.529 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.530 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.531 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.531 DEBUG:boto:Params: {} 2022-09-12T20:16:13.532 DEBUG:boto:Token: None 2022-09-12T20:16:13.533 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:13.534 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rQXgp3f881DqNAwZC6u7J9m1djA= 2022-09-12T20:16:13.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rQXgp3f881DqNAwZC6u7J9m1djA=', 'Content-Length': '0'} 2022-09-12T20:16:13.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db988b520966ad70-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-12T20:16:13.538 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:13.538 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:13.539 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:13.540 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:13.540 DEBUG:boto:Method: GET 2022-09-12T20:16:13.541 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:13.541 DEBUG:boto:Data: 2022-09-12T20:16:13.542 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.542 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.543 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.543 DEBUG:boto:Params: {} 2022-09-12T20:16:13.544 DEBUG:boto:Token: None 2022-09-12T20:16:13.545 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:13.545 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fW7irddfjUaUCgY10RGoOhI/454= 2022-09-12T20:16:13.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fW7irddfjUaUCgY10RGoOhI/454=', 'Content-Length': '0'} 2022-09-12T20:16:13.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003dd69cc7b2446c8a-00631f938d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.549 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:13.549 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:13.550 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:13.551 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:13.551 DEBUG:boto:Method: GET 2022-09-12T20:16:13.552 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:13.552 DEBUG:boto:Data: 2022-09-12T20:16:13.553 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.553 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.554 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.555 DEBUG:boto:Params: {} 2022-09-12T20:16:13.555 DEBUG:boto:Token: None 2022-09-12T20:16:13.556 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:13.556 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:fW7irddfjUaUCgY10RGoOhI/454= 2022-09-12T20:16:13.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:fW7irddfjUaUCgY10RGoOhI/454=', 'Content-Length': '0'} 2022-09-12T20:16:13.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000093cf2ff5bf7477b-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.560 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:13.561 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:13.561 DEBUG:boto:path=/jdpiak-75/?prefix=obj62 2022-09-12T20:16:13.562 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj62 2022-09-12T20:16:13.562 DEBUG:boto:Method: GET 2022-09-12T20:16:13.563 DEBUG:boto:Path: /jdpiak-75/?prefix=obj62 2022-09-12T20:16:13.564 DEBUG:boto:Data: 2022-09-12T20:16:13.564 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.565 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.566 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.566 DEBUG:boto:Params: {} 2022-09-12T20:16:13.567 DEBUG:boto:Token: None 2022-09-12T20:16:13.567 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/ 2022-09-12T20:16:13.568 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ= 2022-09-12T20:16:13.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ=', 'Content-Length': '0'} 2022-09-12T20:16:13.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a754f420d77fb74-00631f938d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.572 DEBUG:boto:b'jdpiak-75obj621000falseobj622022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:13.573 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:13.574 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:13.574 DEBUG:boto:path=/jdpiak-75/?prefix=obj62 2022-09-12T20:16:13.575 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj62 2022-09-12T20:16:13.575 DEBUG:boto:Method: GET 2022-09-12T20:16:13.576 DEBUG:boto:Path: /jdpiak-75/?prefix=obj62 2022-09-12T20:16:13.577 DEBUG:boto:Data: 2022-09-12T20:16:13.577 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.578 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.579 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.579 DEBUG:boto:Params: {} 2022-09-12T20:16:13.580 DEBUG:boto:Token: None 2022-09-12T20:16:13.580 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/ 2022-09-12T20:16:13.581 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ= 2022-09-12T20:16:13.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ=', 'Content-Length': '0'} 2022-09-12T20:16:13.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046023a4360766ec5-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.587 DEBUG:boto:b'jdpiak-75obj621000falseobj622022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:13.588 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:13.589 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:13.589 DEBUG:boto:Method: HEAD 2022-09-12T20:16:13.590 DEBUG:boto:Path: /jdpiak-75/obj62 2022-09-12T20:16:13.591 DEBUG:boto:Data: 2022-09-12T20:16:13.591 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.592 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.592 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.593 DEBUG:boto:Params: {} 2022-09-12T20:16:13.593 DEBUG:boto:Token: None 2022-09-12T20:16:13.594 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj62 2022-09-12T20:16:13.595 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oNL6xcwkDrqZLK8z6IAiZmJ1o0s= 2022-09-12T20:16:13.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oNL6xcwkDrqZLK8z6IAiZmJ1o0s=', 'Content-Length': '0'} 2022-09-12T20:16:13.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6ce678836a833c8-00631f938d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.600 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:13.601 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:13.602 DEBUG:boto:Method: HEAD 2022-09-12T20:16:13.603 DEBUG:boto:Path: /jdpiak-75/obj62 2022-09-12T20:16:13.603 DEBUG:boto:Data: 2022-09-12T20:16:13.604 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.605 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.605 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.606 DEBUG:boto:Params: {} 2022-09-12T20:16:13.606 DEBUG:boto:Token: None 2022-09-12T20:16:13.607 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj62 2022-09-12T20:16:13.607 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:oNL6xcwkDrqZLK8z6IAiZmJ1o0s= 2022-09-12T20:16:13.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:oNL6xcwkDrqZLK8z6IAiZmJ1o0s=', 'Content-Length': '0'} 2022-09-12T20:16:13.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aec4d9b74052545a-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-12T20:16:13.612 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:13.612 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:13.613 DEBUG:boto:Method: GET 2022-09-12T20:16:13.613 DEBUG:boto:Path: /jdpiak-75/obj62 2022-09-12T20:16:13.614 DEBUG:boto:Data: 2022-09-12T20:16:13.614 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.615 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.615 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.616 DEBUG:boto:Params: {} 2022-09-12T20:16:13.617 DEBUG:boto:Token: None 2022-09-12T20:16:13.618 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj62 2022-09-12T20:16:13.619 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W5FbohTPI28/F19hb46/uCoXr2o= 2022-09-12T20:16:13.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W5FbohTPI28/F19hb46/uCoXr2o=', 'Content-Length': '0'} 2022-09-12T20:16:13.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cbea4e55d7e8091-00631f938d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.662 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:13.663 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:13.663 DEBUG:boto:Method: GET 2022-09-12T20:16:13.664 DEBUG:boto:Path: /jdpiak-75/obj62 2022-09-12T20:16:13.665 DEBUG:boto:Data: 2022-09-12T20:16:13.667 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.668 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.670 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.671 DEBUG:boto:Params: {} 2022-09-12T20:16:13.672 DEBUG:boto:Token: None 2022-09-12T20:16:13.674 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj62 2022-09-12T20:16:13.675 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W5FbohTPI28/F19hb46/uCoXr2o= 2022-09-12T20:16:13.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W5FbohTPI28/F19hb46/uCoXr2o=', 'Content-Length': '0'} 2022-09-12T20:16:13.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000063c6255b3c67542-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.682 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:13.683 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:13.684 DEBUG:boto:path=/jdpiak-75/?prefix=obj62 2022-09-12T20:16:13.685 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj62 2022-09-12T20:16:13.686 DEBUG:boto:Method: GET 2022-09-12T20:16:13.687 DEBUG:boto:Path: /jdpiak-75/?prefix=obj62 2022-09-12T20:16:13.688 DEBUG:boto:Data: 2022-09-12T20:16:13.689 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.691 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.692 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.693 DEBUG:boto:Params: {} 2022-09-12T20:16:13.694 DEBUG:boto:Token: None 2022-09-12T20:16:13.695 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/ 2022-09-12T20:16:13.696 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ= 2022-09-12T20:16:13.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ=', 'Content-Length': '0'} 2022-09-12T20:16:13.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9b8957b83dc237a-00631f938d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.702 DEBUG:boto:b'jdpiak-75obj621000falseobj622022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:13.704 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:13.705 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:13.706 DEBUG:boto:path=/jdpiak-75/?prefix=obj62 2022-09-12T20:16:13.707 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj62 2022-09-12T20:16:13.709 DEBUG:boto:Method: GET 2022-09-12T20:16:13.710 DEBUG:boto:Path: /jdpiak-75/?prefix=obj62 2022-09-12T20:16:13.711 DEBUG:boto:Data: 2022-09-12T20:16:13.712 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.713 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.714 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.715 DEBUG:boto:Params: {} 2022-09-12T20:16:13.716 DEBUG:boto:Token: None 2022-09-12T20:16:13.717 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/ 2022-09-12T20:16:13.719 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ= 2022-09-12T20:16:13.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ=', 'Content-Length': '0'} 2022-09-12T20:16:13.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000438d9e34fee62635-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.727 DEBUG:boto:b'jdpiak-75obj621000falseobj622022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:13.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:16:13.729 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:13.730 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:13.731 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:13.732 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:13.733 DEBUG:boto:Method: GET 2022-09-12T20:16:13.734 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:13.735 DEBUG:boto:Data: 2022-09-12T20:16:13.736 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.737 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.738 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.739 DEBUG:boto:Params: {} 2022-09-12T20:16:13.740 DEBUG:boto:Token: None 2022-09-12T20:16:13.741 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:13.743 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:n6hnGq+lfmnx1AP60TZ4w6qAzaI= 2022-09-12T20:16:13.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:n6hnGq+lfmnx1AP60TZ4w6qAzaI=', 'Content-Length': '0'} 2022-09-12T20:16:13.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000772581e0a24f2ed9-00631f938d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.748 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:13.748 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:13.749 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:13.749 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:13.750 DEBUG:boto:Method: GET 2022-09-12T20:16:13.750 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:13.751 DEBUG:boto:Data: 2022-09-12T20:16:13.751 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.752 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.753 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.753 DEBUG:boto:Params: {} 2022-09-12T20:16:13.754 DEBUG:boto:Token: None 2022-09-12T20:16:13.754 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:13.755 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:n6hnGq+lfmnx1AP60TZ4w6qAzaI= 2022-09-12T20:16:13.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:n6hnGq+lfmnx1AP60TZ4w6qAzaI=', 'Content-Length': '0'} 2022-09-12T20:16:13.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d12fb5d8e1dc02c9-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.798 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:13.799 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:13.800 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:13.801 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:13.802 DEBUG:boto:Method: HEAD 2022-09-12T20:16:13.803 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:13.804 DEBUG:boto:Data: 2022-09-12T20:16:13.805 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.806 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.807 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.808 DEBUG:boto:Params: {} 2022-09-12T20:16:13.809 DEBUG:boto:Token: None 2022-09-12T20:16:13.810 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:13.811 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2xvLZc+DBflY4CF48yPwNElOAVg= 2022-09-12T20:16:13.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2xvLZc+DBflY4CF48yPwNElOAVg=', 'Content-Length': '0'} 2022-09-12T20:16:13.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a1e8b250e097a51-00631f938d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.815 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:13.816 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:13.817 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:13.818 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:13.819 DEBUG:boto:Method: HEAD 2022-09-12T20:16:13.820 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:13.821 DEBUG:boto:Data: 2022-09-12T20:16:13.822 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.823 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.824 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.825 DEBUG:boto:Params: {} 2022-09-12T20:16:13.826 DEBUG:boto:Token: None 2022-09-12T20:16:13.827 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:13.828 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2xvLZc+DBflY4CF48yPwNElOAVg= 2022-09-12T20:16:13.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2xvLZc+DBflY4CF48yPwNElOAVg=', 'Content-Length': '0'} 2022-09-12T20:16:13.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092e1fd769a0949f5-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:16:13.835 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:13.836 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:13.837 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:13.838 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:13.839 DEBUG:boto:Method: GET 2022-09-12T20:16:13.840 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:13.841 DEBUG:boto:Data: 2022-09-12T20:16:13.842 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.843 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.844 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.845 DEBUG:boto:Params: {} 2022-09-12T20:16:13.846 DEBUG:boto:Token: None 2022-09-12T20:16:13.848 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:13.849 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:n6hnGq+lfmnx1AP60TZ4w6qAzaI= 2022-09-12T20:16:13.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:n6hnGq+lfmnx1AP60TZ4w6qAzaI=', 'Content-Length': '0'} 2022-09-12T20:16:13.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b140773db9a9b347-00631f938d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.854 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:13.855 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:13.856 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:13.857 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:13.858 DEBUG:boto:Method: GET 2022-09-12T20:16:13.859 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:13.860 DEBUG:boto:Data: 2022-09-12T20:16:13.861 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.862 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.862 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.863 DEBUG:boto:Params: {} 2022-09-12T20:16:13.864 DEBUG:boto:Token: None 2022-09-12T20:16:13.864 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:13.865 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:n6hnGq+lfmnx1AP60TZ4w6qAzaI= 2022-09-12T20:16:13.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:n6hnGq+lfmnx1AP60TZ4w6qAzaI=', 'Content-Length': '0'} 2022-09-12T20:16:13.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031727514dd004d16-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.869 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:13.869 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:13.870 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:16:13.871 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:16:13.871 DEBUG:boto:Method: GET 2022-09-12T20:16:13.872 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:16:13.873 DEBUG:boto:Data: 2022-09-12T20:16:13.873 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.874 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.874 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.875 DEBUG:boto:Params: {} 2022-09-12T20:16:13.875 DEBUG:boto:Token: None 2022-09-12T20:16:13.876 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/ 2022-09-12T20:16:13.877 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ= 2022-09-12T20:16:13.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ=', 'Content-Length': '0'} 2022-09-12T20:16:13.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003aa939000d163977-00631f938d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.881 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:13.882 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:13.882 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:13.883 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:16:13.883 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:16:13.884 DEBUG:boto:Method: GET 2022-09-12T20:16:13.884 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:16:13.885 DEBUG:boto:Data: 2022-09-12T20:16:13.886 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.886 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.887 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.888 DEBUG:boto:Params: {} 2022-09-12T20:16:13.888 DEBUG:boto:Token: None 2022-09-12T20:16:13.889 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/ 2022-09-12T20:16:13.889 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ= 2022-09-12T20:16:13.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ=', 'Content-Length': '0'} 2022-09-12T20:16:13.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c30ec63971b7d00a-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.896 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:13.897 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:13.897 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:13.898 DEBUG:boto:Method: HEAD 2022-09-12T20:16:13.899 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:16:13.899 DEBUG:boto:Data: 2022-09-12T20:16:13.900 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.900 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.901 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.901 DEBUG:boto:Params: {} 2022-09-12T20:16:13.902 DEBUG:boto:Token: None 2022-09-12T20:16:13.902 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj68 2022-09-12T20:16:13.903 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9SNH4N8TPzd5yLNaAYOm4HdIQfk= 2022-09-12T20:16:13.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9SNH4N8TPzd5yLNaAYOm4HdIQfk=', 'Content-Length': '0'} 2022-09-12T20:16:13.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e53dca2ce25e2ef-00631f938d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.906 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:13.907 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:13.907 DEBUG:boto:Method: HEAD 2022-09-12T20:16:13.908 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:16:13.908 DEBUG:boto:Data: 2022-09-12T20:16:13.909 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.909 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.910 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.910 DEBUG:boto:Params: {} 2022-09-12T20:16:13.911 DEBUG:boto:Token: None 2022-09-12T20:16:13.912 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj68 2022-09-12T20:16:13.912 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9SNH4N8TPzd5yLNaAYOm4HdIQfk= 2022-09-12T20:16:13.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9SNH4N8TPzd5yLNaAYOm4HdIQfk=', 'Content-Length': '0'} 2022-09-12T20:16:13.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dc1c3698103423b-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:16:13.916 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:13.917 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:13.917 DEBUG:boto:Method: GET 2022-09-12T20:16:13.918 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:16:13.918 DEBUG:boto:Data: 2022-09-12T20:16:13.919 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.919 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.920 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.920 DEBUG:boto:Params: {} 2022-09-12T20:16:13.921 DEBUG:boto:Token: None 2022-09-12T20:16:13.921 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj68 2022-09-12T20:16:13.922 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t/WeyeC7bF+as0Xf3qqPU+cAqrQ= 2022-09-12T20:16:13.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t/WeyeC7bF+as0Xf3qqPU+cAqrQ=', 'Content-Length': '0'} 2022-09-12T20:16:13.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000013e252628f971e8-00631f938d-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.926 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:13.926 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:13.927 DEBUG:boto:Method: GET 2022-09-12T20:16:13.927 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:16:13.928 DEBUG:boto:Data: 2022-09-12T20:16:13.928 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.929 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.929 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.930 DEBUG:boto:Params: {} 2022-09-12T20:16:13.931 DEBUG:boto:Token: None 2022-09-12T20:16:13.931 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/obj68 2022-09-12T20:16:13.932 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t/WeyeC7bF+as0Xf3qqPU+cAqrQ= 2022-09-12T20:16:13.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t/WeyeC7bF+as0Xf3qqPU+cAqrQ=', 'Content-Length': '0'} 2022-09-12T20:16:13.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fd45da3df4f664c-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.978 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:13.979 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:13.980 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:16:13.980 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:16:13.981 DEBUG:boto:Method: GET 2022-09-12T20:16:13.981 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:16:13.982 DEBUG:boto:Data: 2022-09-12T20:16:13.982 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.983 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:13.983 DEBUG:boto:Port: 8001 2022-09-12T20:16:13.984 DEBUG:boto:Params: {} 2022-09-12T20:16:13.984 DEBUG:boto:Token: None 2022-09-12T20:16:13.985 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/ 2022-09-12T20:16:13.986 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ= 2022-09-12T20:16:13.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ=', 'Content-Length': '0'} 2022-09-12T20:16:13.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007da5ae9a9a3f92ee-00631f938d-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:13.990 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:13.991 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:13.991 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:13.992 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:16:13.992 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:16:13.993 DEBUG:boto:Method: GET 2022-09-12T20:16:13.993 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:16:13.994 DEBUG:boto:Data: 2022-09-12T20:16:13.995 DEBUG:boto:Headers: {} 2022-09-12T20:16:13.995 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:13.996 DEBUG:boto:Port: 8000 2022-09-12T20:16:13.996 DEBUG:boto:Params: {} 2022-09-12T20:16:13.997 DEBUG:boto:Token: None 2022-09-12T20:16:13.997 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:13 GMT /jdpiak-75/ 2022-09-12T20:16:13.998 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ= 2022-09-12T20:16:13.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:13 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HjVMQjs9i++58omemOVckmvU3eQ=', 'Content-Length': '0'} 2022-09-12T20:16:14.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bc42443a5430eda-00631f938d-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.004 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:14.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:16:14.006 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:14.006 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:14.007 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:14.008 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:14.008 DEBUG:boto:Method: GET 2022-09-12T20:16:14.009 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:14.010 DEBUG:boto:Data: 2022-09-12T20:16:14.010 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.011 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.011 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.012 DEBUG:boto:Params: {} 2022-09-12T20:16:14.013 DEBUG:boto:Token: None 2022-09-12T20:16:14.013 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:14.014 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ns4iTBfgVchyfE7AVBqyyd6N368= 2022-09-12T20:16:14.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ns4iTBfgVchyfE7AVBqyyd6N368=', 'Content-Length': '0'} 2022-09-12T20:16:14.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b24222495eac49c8-00631f938e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.018 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:14.019 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:14.019 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:14.020 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:14.020 DEBUG:boto:Method: GET 2022-09-12T20:16:14.021 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:14.022 DEBUG:boto:Data: 2022-09-12T20:16:14.022 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.023 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.023 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.024 DEBUG:boto:Params: {} 2022-09-12T20:16:14.024 DEBUG:boto:Token: None 2022-09-12T20:16:14.025 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:14.026 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ns4iTBfgVchyfE7AVBqyyd6N368= 2022-09-12T20:16:14.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ns4iTBfgVchyfE7AVBqyyd6N368=', 'Content-Length': '0'} 2022-09-12T20:16:14.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006a9c92a4f04ec9f-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.030 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:14.031 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:14.032 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:14.032 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:14.033 DEBUG:boto:Method: HEAD 2022-09-12T20:16:14.033 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:14.034 DEBUG:boto:Data: 2022-09-12T20:16:14.035 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.035 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.036 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.037 DEBUG:boto:Params: {} 2022-09-12T20:16:14.037 DEBUG:boto:Token: None 2022-09-12T20:16:14.038 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:14.039 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NyrK24LPfzKh191x3D/SBecbj0Y= 2022-09-12T20:16:14.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NyrK24LPfzKh191x3D/SBecbj0Y=', 'Content-Length': '0'} 2022-09-12T20:16:14.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005fe387ad2634ab4-00631f938e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.043 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:14.043 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:14.044 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:14.044 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:14.045 DEBUG:boto:Method: HEAD 2022-09-12T20:16:14.045 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:14.046 DEBUG:boto:Data: 2022-09-12T20:16:14.047 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.047 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.048 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.048 DEBUG:boto:Params: {} 2022-09-12T20:16:14.049 DEBUG:boto:Token: None 2022-09-12T20:16:14.050 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:14.050 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NyrK24LPfzKh191x3D/SBecbj0Y= 2022-09-12T20:16:14.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NyrK24LPfzKh191x3D/SBecbj0Y=', 'Content-Length': '0'} 2022-09-12T20:16:14.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccc51ae70dc15e4d-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:16:14.055 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:14.055 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:14.056 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:14.056 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:14.057 DEBUG:boto:Method: GET 2022-09-12T20:16:14.057 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:14.058 DEBUG:boto:Data: 2022-09-12T20:16:14.058 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.059 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.059 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.060 DEBUG:boto:Params: {} 2022-09-12T20:16:14.061 DEBUG:boto:Token: None 2022-09-12T20:16:14.061 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:14.062 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ns4iTBfgVchyfE7AVBqyyd6N368= 2022-09-12T20:16:14.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ns4iTBfgVchyfE7AVBqyyd6N368=', 'Content-Length': '0'} 2022-09-12T20:16:14.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3e696cb7fd47f0b-00631f938e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.065 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:14.066 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:14.066 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:14.067 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:14.068 DEBUG:boto:Method: GET 2022-09-12T20:16:14.068 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:14.069 DEBUG:boto:Data: 2022-09-12T20:16:14.069 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.070 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.070 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.071 DEBUG:boto:Params: {} 2022-09-12T20:16:14.072 DEBUG:boto:Token: None 2022-09-12T20:16:14.072 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:14.073 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ns4iTBfgVchyfE7AVBqyyd6N368= 2022-09-12T20:16:14.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ns4iTBfgVchyfE7AVBqyyd6N368=', 'Content-Length': '0'} 2022-09-12T20:16:14.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5d2dfc5377a2b68-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.077 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:14.077 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:14.078 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:16:14.079 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:16:14.079 DEBUG:boto:Method: GET 2022-09-12T20:16:14.080 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:16:14.080 DEBUG:boto:Data: 2022-09-12T20:16:14.081 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.081 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.082 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.082 DEBUG:boto:Params: {} 2022-09-12T20:16:14.083 DEBUG:boto:Token: None 2022-09-12T20:16:14.084 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/ 2022-09-12T20:16:14.084 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA= 2022-09-12T20:16:14.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA=', 'Content-Length': '0'} 2022-09-12T20:16:14.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba02dd493c135fbc-00631f938e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.089 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:14.089 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:14.090 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:14.091 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:16:14.091 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:16:14.092 DEBUG:boto:Method: GET 2022-09-12T20:16:14.093 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:16:14.093 DEBUG:boto:Data: 2022-09-12T20:16:14.094 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.094 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.095 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.095 DEBUG:boto:Params: {} 2022-09-12T20:16:14.096 DEBUG:boto:Token: None 2022-09-12T20:16:14.097 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/ 2022-09-12T20:16:14.097 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA= 2022-09-12T20:16:14.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA=', 'Content-Length': '0'} 2022-09-12T20:16:14.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022dbe14817d79ddd-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.103 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:14.104 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:14.105 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:14.105 DEBUG:boto:Method: HEAD 2022-09-12T20:16:14.106 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:16:14.106 DEBUG:boto:Data: 2022-09-12T20:16:14.107 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.107 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.108 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.108 DEBUG:boto:Params: {} 2022-09-12T20:16:14.109 DEBUG:boto:Token: None 2022-09-12T20:16:14.109 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj69 2022-09-12T20:16:14.110 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ho22I2dCItc6jl1yvjs8Dik0bTw= 2022-09-12T20:16:14.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ho22I2dCItc6jl1yvjs8Dik0bTw=', 'Content-Length': '0'} 2022-09-12T20:16:14.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ba822c28865feed-00631f938e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.113 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:14.114 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:14.114 DEBUG:boto:Method: HEAD 2022-09-12T20:16:14.115 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:16:14.115 DEBUG:boto:Data: 2022-09-12T20:16:14.116 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.116 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.117 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.117 DEBUG:boto:Params: {} 2022-09-12T20:16:14.118 DEBUG:boto:Token: None 2022-09-12T20:16:14.118 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj69 2022-09-12T20:16:14.119 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ho22I2dCItc6jl1yvjs8Dik0bTw= 2022-09-12T20:16:14.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ho22I2dCItc6jl1yvjs8Dik0bTw=', 'Content-Length': '0'} 2022-09-12T20:16:14.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a628edaef98af33-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:16:14.123 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:14.123 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:14.124 DEBUG:boto:Method: GET 2022-09-12T20:16:14.124 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:16:14.125 DEBUG:boto:Data: 2022-09-12T20:16:14.125 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.126 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.126 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.127 DEBUG:boto:Params: {} 2022-09-12T20:16:14.127 DEBUG:boto:Token: None 2022-09-12T20:16:14.128 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj69 2022-09-12T20:16:14.129 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XMkTfWVNz7u8YAvlTk++btUpcb0= 2022-09-12T20:16:14.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XMkTfWVNz7u8YAvlTk++btUpcb0=', 'Content-Length': '0'} 2022-09-12T20:16:14.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7a0030975d28d61-00631f938e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.174 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:14.175 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:14.175 DEBUG:boto:Method: GET 2022-09-12T20:16:14.176 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:16:14.177 DEBUG:boto:Data: 2022-09-12T20:16:14.177 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.178 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.178 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.179 DEBUG:boto:Params: {} 2022-09-12T20:16:14.179 DEBUG:boto:Token: None 2022-09-12T20:16:14.180 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj69 2022-09-12T20:16:14.181 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XMkTfWVNz7u8YAvlTk++btUpcb0= 2022-09-12T20:16:14.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XMkTfWVNz7u8YAvlTk++btUpcb0=', 'Content-Length': '0'} 2022-09-12T20:16:14.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050bc0ba0904fd74d-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.185 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:14.185 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:14.186 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:16:14.186 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:16:14.187 DEBUG:boto:Method: GET 2022-09-12T20:16:14.187 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:16:14.188 DEBUG:boto:Data: 2022-09-12T20:16:14.188 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.189 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.189 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.190 DEBUG:boto:Params: {} 2022-09-12T20:16:14.191 DEBUG:boto:Token: None 2022-09-12T20:16:14.191 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/ 2022-09-12T20:16:14.192 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA= 2022-09-12T20:16:14.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA=', 'Content-Length': '0'} 2022-09-12T20:16:14.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2e63a3ac65c6edf-00631f938e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.196 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:14.197 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:14.197 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:14.198 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:16:14.198 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:16:14.199 DEBUG:boto:Method: GET 2022-09-12T20:16:14.199 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:16:14.200 DEBUG:boto:Data: 2022-09-12T20:16:14.200 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.201 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.201 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.202 DEBUG:boto:Params: {} 2022-09-12T20:16:14.203 DEBUG:boto:Token: None 2022-09-12T20:16:14.203 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/ 2022-09-12T20:16:14.204 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA= 2022-09-12T20:16:14.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA=', 'Content-Length': '0'} 2022-09-12T20:16:14.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f69552773060cc1c-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.210 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:14.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2022-09-12T20:16:14.211 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:14.212 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:14.212 DEBUG:boto:path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:14.213 DEBUG:boto:auth_path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:14.214 DEBUG:boto:Method: GET 2022-09-12T20:16:14.214 DEBUG:boto:Path: /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:14.215 DEBUG:boto:Data: 2022-09-12T20:16:14.216 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.216 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.217 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.217 DEBUG:boto:Params: {} 2022-09-12T20:16:14.218 DEBUG:boto:Token: None 2022-09-12T20:16:14.219 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:14.219 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2EIEMQuNe39XPkcE9PqfSaa27S4= 2022-09-12T20:16:14.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2EIEMQuNe39XPkcE9PqfSaa27S4=', 'Content-Length': '0'} 2022-09-12T20:16:14.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f09d9e6035c1d28-00631f938e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.223 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:14.224 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:14.224 DEBUG:boto:path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:14.225 DEBUG:boto:auth_path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:14.225 DEBUG:boto:Method: GET 2022-09-12T20:16:14.226 DEBUG:boto:Path: /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:14.227 DEBUG:boto:Data: 2022-09-12T20:16:14.227 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.228 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.228 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.229 DEBUG:boto:Params: {} 2022-09-12T20:16:14.229 DEBUG:boto:Token: None 2022-09-12T20:16:14.230 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:14.231 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2EIEMQuNe39XPkcE9PqfSaa27S4= 2022-09-12T20:16:14.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2EIEMQuNe39XPkcE9PqfSaa27S4=', 'Content-Length': '0'} 2022-09-12T20:16:14.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034fd97a415e77b0e-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.235 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:14.236 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:14.236 DEBUG:boto:path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:14.237 DEBUG:boto:auth_path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:14.237 DEBUG:boto:Method: HEAD 2022-09-12T20:16:14.238 DEBUG:boto:Path: /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:14.238 DEBUG:boto:Data: 2022-09-12T20:16:14.239 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.240 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.241 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.242 DEBUG:boto:Params: {} 2022-09-12T20:16:14.243 DEBUG:boto:Token: None 2022-09-12T20:16:14.244 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:14.246 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VLKZDQiIwAU9cvEFN2k7VR4/kFs= 2022-09-12T20:16:14.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VLKZDQiIwAU9cvEFN2k7VR4/kFs=', 'Content-Length': '0'} 2022-09-12T20:16:14.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000244efbfe69c879f5-00631f938e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.251 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:14.251 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:14.252 DEBUG:boto:path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:14.252 DEBUG:boto:auth_path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:14.253 DEBUG:boto:Method: HEAD 2022-09-12T20:16:14.253 DEBUG:boto:Path: /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:14.254 DEBUG:boto:Data: 2022-09-12T20:16:14.254 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.255 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.255 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.256 DEBUG:boto:Params: {} 2022-09-12T20:16:14.256 DEBUG:boto:Token: None 2022-09-12T20:16:14.257 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:14.257 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VLKZDQiIwAU9cvEFN2k7VR4/kFs= 2022-09-12T20:16:14.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VLKZDQiIwAU9cvEFN2k7VR4/kFs=', 'Content-Length': '0'} 2022-09-12T20:16:14.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a5809638a40ed85-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2022-09-12T20:16:14.262 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:14.263 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:14.263 DEBUG:boto:path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:14.264 DEBUG:boto:auth_path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:14.264 DEBUG:boto:Method: GET 2022-09-12T20:16:14.265 DEBUG:boto:Path: /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:14.265 DEBUG:boto:Data: 2022-09-12T20:16:14.266 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.266 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.267 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.268 DEBUG:boto:Params: {} 2022-09-12T20:16:14.268 DEBUG:boto:Token: None 2022-09-12T20:16:14.269 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:14.270 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2EIEMQuNe39XPkcE9PqfSaa27S4= 2022-09-12T20:16:14.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2EIEMQuNe39XPkcE9PqfSaa27S4=', 'Content-Length': '0'} 2022-09-12T20:16:14.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000077a45c13b0a361b-00631f938e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.314 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:14.315 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:14.315 DEBUG:boto:path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:14.316 DEBUG:boto:auth_path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:14.316 DEBUG:boto:Method: GET 2022-09-12T20:16:14.317 DEBUG:boto:Path: /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:14.317 DEBUG:boto:Data: 2022-09-12T20:16:14.318 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.319 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.319 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.320 DEBUG:boto:Params: {} 2022-09-12T20:16:14.320 DEBUG:boto:Token: None 2022-09-12T20:16:14.321 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:14.321 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2EIEMQuNe39XPkcE9PqfSaa27S4= 2022-09-12T20:16:14.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2EIEMQuNe39XPkcE9PqfSaa27S4=', 'Content-Length': '0'} 2022-09-12T20:16:14.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff337d129de6b866-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.325 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:14.326 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:14.327 DEBUG:boto:path=/jdpiak-75/?prefix=obj74 2022-09-12T20:16:14.327 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj74 2022-09-12T20:16:14.328 DEBUG:boto:Method: GET 2022-09-12T20:16:14.328 DEBUG:boto:Path: /jdpiak-75/?prefix=obj74 2022-09-12T20:16:14.329 DEBUG:boto:Data: 2022-09-12T20:16:14.329 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.330 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.330 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.331 DEBUG:boto:Params: {} 2022-09-12T20:16:14.331 DEBUG:boto:Token: None 2022-09-12T20:16:14.332 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/ 2022-09-12T20:16:14.332 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA= 2022-09-12T20:16:14.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA=', 'Content-Length': '0'} 2022-09-12T20:16:14.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdf530b500d01711-00631f938e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.337 DEBUG:boto:b'jdpiak-75obj741000falseobj742022-09-12T20:14:27.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:14.338 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:14.338 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:14.339 DEBUG:boto:path=/jdpiak-75/?prefix=obj74 2022-09-12T20:16:14.339 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj74 2022-09-12T20:16:14.340 DEBUG:boto:Method: GET 2022-09-12T20:16:14.340 DEBUG:boto:Path: /jdpiak-75/?prefix=obj74 2022-09-12T20:16:14.341 DEBUG:boto:Data: 2022-09-12T20:16:14.341 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.342 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.342 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.343 DEBUG:boto:Params: {} 2022-09-12T20:16:14.343 DEBUG:boto:Token: None 2022-09-12T20:16:14.344 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/ 2022-09-12T20:16:14.345 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA= 2022-09-12T20:16:14.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA=', 'Content-Length': '0'} 2022-09-12T20:16:14.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000540e7249b0aa4435-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.351 DEBUG:boto:b'jdpiak-75obj741000falseobj742022-09-12T20:14:27.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:14.351 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:14.352 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:14.352 DEBUG:boto:Method: HEAD 2022-09-12T20:16:14.353 DEBUG:boto:Path: /jdpiak-75/obj74 2022-09-12T20:16:14.353 DEBUG:boto:Data: 2022-09-12T20:16:14.354 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.354 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.355 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.355 DEBUG:boto:Params: {} 2022-09-12T20:16:14.356 DEBUG:boto:Token: None 2022-09-12T20:16:14.357 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj74 2022-09-12T20:16:14.357 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1yGVunO3tOHk1yMRtW8c1EEDCgA= 2022-09-12T20:16:14.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1yGVunO3tOHk1yMRtW8c1EEDCgA=', 'Content-Length': '0'} 2022-09-12T20:16:14.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb3ed4cda7126f56-00631f938e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.360 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:14.361 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:14.361 DEBUG:boto:Method: HEAD 2022-09-12T20:16:14.362 DEBUG:boto:Path: /jdpiak-75/obj74 2022-09-12T20:16:14.362 DEBUG:boto:Data: 2022-09-12T20:16:14.363 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.363 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.364 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.364 DEBUG:boto:Params: {} 2022-09-12T20:16:14.365 DEBUG:boto:Token: None 2022-09-12T20:16:14.366 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj74 2022-09-12T20:16:14.366 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1yGVunO3tOHk1yMRtW8c1EEDCgA= 2022-09-12T20:16:14.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1yGVunO3tOHk1yMRtW8c1EEDCgA=', 'Content-Length': '0'} 2022-09-12T20:16:14.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eea23d1c802f06a6-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2022-09-12T20:16:14.370 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:14.370 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:14.371 DEBUG:boto:Method: GET 2022-09-12T20:16:14.372 DEBUG:boto:Path: /jdpiak-75/obj74 2022-09-12T20:16:14.372 DEBUG:boto:Data: 2022-09-12T20:16:14.373 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.373 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.374 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.374 DEBUG:boto:Params: {} 2022-09-12T20:16:14.375 DEBUG:boto:Token: None 2022-09-12T20:16:14.375 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj74 2022-09-12T20:16:14.376 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ok+RRGlZSXxLiRiFCDR6gwUa8UQ= 2022-09-12T20:16:14.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ok+RRGlZSXxLiRiFCDR6gwUa8UQ=', 'Content-Length': '0'} 2022-09-12T20:16:14.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d392f63c5c7f8c7f-00631f938e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.380 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:14.380 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:14.381 DEBUG:boto:Method: GET 2022-09-12T20:16:14.381 DEBUG:boto:Path: /jdpiak-75/obj74 2022-09-12T20:16:14.382 DEBUG:boto:Data: 2022-09-12T20:16:14.382 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.383 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.383 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.384 DEBUG:boto:Params: {} 2022-09-12T20:16:14.384 DEBUG:boto:Token: None 2022-09-12T20:16:14.385 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj74 2022-09-12T20:16:14.386 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ok+RRGlZSXxLiRiFCDR6gwUa8UQ= 2022-09-12T20:16:14.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ok+RRGlZSXxLiRiFCDR6gwUa8UQ=', 'Content-Length': '0'} 2022-09-12T20:16:14.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007d13bba540bbb6e-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.430 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:14.431 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:14.431 DEBUG:boto:path=/jdpiak-75/?prefix=obj74 2022-09-12T20:16:14.432 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj74 2022-09-12T20:16:14.432 DEBUG:boto:Method: GET 2022-09-12T20:16:14.433 DEBUG:boto:Path: /jdpiak-75/?prefix=obj74 2022-09-12T20:16:14.433 DEBUG:boto:Data: 2022-09-12T20:16:14.434 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.434 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.435 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.436 DEBUG:boto:Params: {} 2022-09-12T20:16:14.436 DEBUG:boto:Token: None 2022-09-12T20:16:14.437 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/ 2022-09-12T20:16:14.438 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA= 2022-09-12T20:16:14.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA=', 'Content-Length': '0'} 2022-09-12T20:16:14.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b7e9772ba17fe18-00631f938e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.442 DEBUG:boto:b'jdpiak-75obj741000falseobj742022-09-12T20:14:27.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:14.443 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:14.444 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:14.444 DEBUG:boto:path=/jdpiak-75/?prefix=obj74 2022-09-12T20:16:14.445 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj74 2022-09-12T20:16:14.445 DEBUG:boto:Method: GET 2022-09-12T20:16:14.446 DEBUG:boto:Path: /jdpiak-75/?prefix=obj74 2022-09-12T20:16:14.447 DEBUG:boto:Data: 2022-09-12T20:16:14.447 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.448 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.448 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.449 DEBUG:boto:Params: {} 2022-09-12T20:16:14.449 DEBUG:boto:Token: None 2022-09-12T20:16:14.450 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/ 2022-09-12T20:16:14.451 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA= 2022-09-12T20:16:14.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA=', 'Content-Length': '0'} 2022-09-12T20:16:14.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e66fb1db445038ce-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.457 DEBUG:boto:b'jdpiak-75obj741000falseobj742022-09-12T20:14:27.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:14.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:16:14.458 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:14.459 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:14.459 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:14.460 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:14.460 DEBUG:boto:Method: GET 2022-09-12T20:16:14.461 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:14.461 DEBUG:boto:Data: 2022-09-12T20:16:14.462 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.462 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.463 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.463 DEBUG:boto:Params: {} 2022-09-12T20:16:14.464 DEBUG:boto:Token: None 2022-09-12T20:16:14.465 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:14.465 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lTuSCZ0PBJSzFyn/dCpyMi9LRcA= 2022-09-12T20:16:14.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lTuSCZ0PBJSzFyn/dCpyMi9LRcA=', 'Content-Length': '0'} 2022-09-12T20:16:14.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000947e70d7204d0701-00631f938e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.469 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:14.470 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:14.470 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:14.471 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:14.471 DEBUG:boto:Method: GET 2022-09-12T20:16:14.472 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:14.472 DEBUG:boto:Data: 2022-09-12T20:16:14.473 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.474 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.474 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.475 DEBUG:boto:Params: {} 2022-09-12T20:16:14.475 DEBUG:boto:Token: None 2022-09-12T20:16:14.476 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:14.477 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lTuSCZ0PBJSzFyn/dCpyMi9LRcA= 2022-09-12T20:16:14.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lTuSCZ0PBJSzFyn/dCpyMi9LRcA=', 'Content-Length': '0'} 2022-09-12T20:16:14.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b0ae46c840b42e7-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.481 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:14.481 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:14.482 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:14.482 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:14.483 DEBUG:boto:Method: HEAD 2022-09-12T20:16:14.483 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:14.484 DEBUG:boto:Data: 2022-09-12T20:16:14.484 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.485 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.485 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.486 DEBUG:boto:Params: {} 2022-09-12T20:16:14.486 DEBUG:boto:Token: None 2022-09-12T20:16:14.487 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:14.488 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ct2u43fYL+KWwL1GnxpNfJ8yHA4= 2022-09-12T20:16:14.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ct2u43fYL+KWwL1GnxpNfJ8yHA4=', 'Content-Length': '0'} 2022-09-12T20:16:14.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b06475a26556409-00631f938e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.491 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:14.491 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:14.492 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:14.492 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:14.493 DEBUG:boto:Method: HEAD 2022-09-12T20:16:14.494 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:14.494 DEBUG:boto:Data: 2022-09-12T20:16:14.495 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.495 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.496 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.496 DEBUG:boto:Params: {} 2022-09-12T20:16:14.497 DEBUG:boto:Token: None 2022-09-12T20:16:14.497 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:14.498 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ct2u43fYL+KWwL1GnxpNfJ8yHA4= 2022-09-12T20:16:14.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ct2u43fYL+KWwL1GnxpNfJ8yHA4=', 'Content-Length': '0'} 2022-09-12T20:16:14.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000998db642673d1fc5-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:16:14.502 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:14.503 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:14.503 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:14.504 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:14.504 DEBUG:boto:Method: GET 2022-09-12T20:16:14.505 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:14.505 DEBUG:boto:Data: 2022-09-12T20:16:14.506 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.506 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.507 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.507 DEBUG:boto:Params: {} 2022-09-12T20:16:14.508 DEBUG:boto:Token: None 2022-09-12T20:16:14.509 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:14.509 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lTuSCZ0PBJSzFyn/dCpyMi9LRcA= 2022-09-12T20:16:14.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lTuSCZ0PBJSzFyn/dCpyMi9LRcA=', 'Content-Length': '0'} 2022-09-12T20:16:14.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c0dc1e7e6358c8d-00631f938e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.513 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:14.513 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:14.514 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:14.514 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:14.515 DEBUG:boto:Method: GET 2022-09-12T20:16:14.516 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:14.516 DEBUG:boto:Data: 2022-09-12T20:16:14.517 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.517 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.518 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.518 DEBUG:boto:Params: {} 2022-09-12T20:16:14.519 DEBUG:boto:Token: None 2022-09-12T20:16:14.519 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:14.520 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lTuSCZ0PBJSzFyn/dCpyMi9LRcA= 2022-09-12T20:16:14.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lTuSCZ0PBJSzFyn/dCpyMi9LRcA=', 'Content-Length': '0'} 2022-09-12T20:16:14.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008752b2436f22685c-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.524 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:14.524 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:14.525 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:16:14.525 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:16:14.526 DEBUG:boto:Method: GET 2022-09-12T20:16:14.526 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:16:14.527 DEBUG:boto:Data: 2022-09-12T20:16:14.527 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.528 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.528 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.529 DEBUG:boto:Params: {} 2022-09-12T20:16:14.529 DEBUG:boto:Token: None 2022-09-12T20:16:14.530 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/ 2022-09-12T20:16:14.531 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA= 2022-09-12T20:16:14.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA=', 'Content-Length': '0'} 2022-09-12T20:16:14.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd94fbc309e9a742-00631f938e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.535 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:14.535 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:14.536 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:14.536 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:16:14.537 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:16:14.537 DEBUG:boto:Method: GET 2022-09-12T20:16:14.538 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:16:14.539 DEBUG:boto:Data: 2022-09-12T20:16:14.539 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.540 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.540 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.541 DEBUG:boto:Params: {} 2022-09-12T20:16:14.541 DEBUG:boto:Token: None 2022-09-12T20:16:14.542 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/ 2022-09-12T20:16:14.542 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA= 2022-09-12T20:16:14.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA=', 'Content-Length': '0'} 2022-09-12T20:16:14.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3bbdf52f798bc21-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.594 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:14.595 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:14.596 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:14.596 DEBUG:boto:Method: HEAD 2022-09-12T20:16:14.597 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:16:14.597 DEBUG:boto:Data: 2022-09-12T20:16:14.598 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.598 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.599 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.599 DEBUG:boto:Params: {} 2022-09-12T20:16:14.600 DEBUG:boto:Token: None 2022-09-12T20:16:14.600 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj76 2022-09-12T20:16:14.601 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pQrt1MZEXsiOHYiDj37BCLLm4GU= 2022-09-12T20:16:14.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pQrt1MZEXsiOHYiDj37BCLLm4GU=', 'Content-Length': '0'} 2022-09-12T20:16:14.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b97c5ce0419f246-00631f938e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.604 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:14.605 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:14.605 DEBUG:boto:Method: HEAD 2022-09-12T20:16:14.606 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:16:14.606 DEBUG:boto:Data: 2022-09-12T20:16:14.607 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.607 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.608 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.608 DEBUG:boto:Params: {} 2022-09-12T20:16:14.609 DEBUG:boto:Token: None 2022-09-12T20:16:14.610 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj76 2022-09-12T20:16:14.610 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pQrt1MZEXsiOHYiDj37BCLLm4GU= 2022-09-12T20:16:14.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pQrt1MZEXsiOHYiDj37BCLLm4GU=', 'Content-Length': '0'} 2022-09-12T20:16:14.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076f2402958dd8abf-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:16:14.615 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:14.615 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:14.616 DEBUG:boto:Method: GET 2022-09-12T20:16:14.616 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:16:14.617 DEBUG:boto:Data: 2022-09-12T20:16:14.618 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.619 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.620 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.621 DEBUG:boto:Params: {} 2022-09-12T20:16:14.621 DEBUG:boto:Token: None 2022-09-12T20:16:14.622 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj76 2022-09-12T20:16:14.623 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6qddq3apH363kRbuUlV/OWUFFBg= 2022-09-12T20:16:14.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6qddq3apH363kRbuUlV/OWUFFBg=', 'Content-Length': '0'} 2022-09-12T20:16:14.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000028c264af41b754b-00631f938e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.627 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:14.627 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:14.628 DEBUG:boto:Method: GET 2022-09-12T20:16:14.628 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:16:14.629 DEBUG:boto:Data: 2022-09-12T20:16:14.629 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.630 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.630 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.631 DEBUG:boto:Params: {} 2022-09-12T20:16:14.632 DEBUG:boto:Token: None 2022-09-12T20:16:14.632 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj76 2022-09-12T20:16:14.633 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6qddq3apH363kRbuUlV/OWUFFBg= 2022-09-12T20:16:14.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6qddq3apH363kRbuUlV/OWUFFBg=', 'Content-Length': '0'} 2022-09-12T20:16:14.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094a701486df0dc7a-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.637 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:14.638 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:14.638 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:16:14.639 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:16:14.639 DEBUG:boto:Method: GET 2022-09-12T20:16:14.640 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:16:14.641 DEBUG:boto:Data: 2022-09-12T20:16:14.641 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.642 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.643 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.643 DEBUG:boto:Params: {} 2022-09-12T20:16:14.644 DEBUG:boto:Token: None 2022-09-12T20:16:14.645 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/ 2022-09-12T20:16:14.646 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA= 2022-09-12T20:16:14.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA=', 'Content-Length': '0'} 2022-09-12T20:16:14.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082d4ecd3c6bf0160-00631f938e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.690 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:14.692 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:14.694 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:14.696 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:16:14.697 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:16:14.698 DEBUG:boto:Method: GET 2022-09-12T20:16:14.699 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:16:14.701 DEBUG:boto:Data: 2022-09-12T20:16:14.702 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.704 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.705 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.706 DEBUG:boto:Params: {} 2022-09-12T20:16:14.707 DEBUG:boto:Token: None 2022-09-12T20:16:14.709 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/ 2022-09-12T20:16:14.710 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA= 2022-09-12T20:16:14.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA=', 'Content-Length': '0'} 2022-09-12T20:16:14.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062fefc5e98dafc9a-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.719 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:14.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:16:14.722 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:14.723 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:14.724 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:14.725 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:14.727 DEBUG:boto:Method: GET 2022-09-12T20:16:14.728 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:14.729 DEBUG:boto:Data: 2022-09-12T20:16:14.730 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.731 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.732 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.733 DEBUG:boto:Params: {} 2022-09-12T20:16:14.733 DEBUG:boto:Token: None 2022-09-12T20:16:14.735 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:14.736 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+KeMpbovSggqDoouPOeEXL5YKmY= 2022-09-12T20:16:14.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+KeMpbovSggqDoouPOeEXL5YKmY=', 'Content-Length': '0'} 2022-09-12T20:16:14.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa687b7eb9682a0c-00631f938e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.749 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:14.750 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:14.751 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:14.760 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:14.761 DEBUG:boto:Method: GET 2022-09-12T20:16:14.762 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:14.763 DEBUG:boto:Data: 2022-09-12T20:16:14.764 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.765 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.766 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.767 DEBUG:boto:Params: {} 2022-09-12T20:16:14.768 DEBUG:boto:Token: None 2022-09-12T20:16:14.769 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:14.770 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+KeMpbovSggqDoouPOeEXL5YKmY= 2022-09-12T20:16:14.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+KeMpbovSggqDoouPOeEXL5YKmY=', 'Content-Length': '0'} 2022-09-12T20:16:14.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067a3a4303253b86c-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.775 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:14.776 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:14.777 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:14.778 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:14.778 DEBUG:boto:Method: HEAD 2022-09-12T20:16:14.779 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:14.779 DEBUG:boto:Data: 2022-09-12T20:16:14.780 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.780 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.781 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.782 DEBUG:boto:Params: {} 2022-09-12T20:16:14.783 DEBUG:boto:Token: None 2022-09-12T20:16:14.784 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:14.785 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m6YLNFDu4prpY2F/uk/ORRKrgS4= 2022-09-12T20:16:14.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m6YLNFDu4prpY2F/uk/ORRKrgS4=', 'Content-Length': '0'} 2022-09-12T20:16:14.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a4ef1a8207b8141-00631f938e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.789 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:14.789 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:14.790 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:14.790 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:14.791 DEBUG:boto:Method: HEAD 2022-09-12T20:16:14.791 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:14.792 DEBUG:boto:Data: 2022-09-12T20:16:14.793 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.793 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.794 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.795 DEBUG:boto:Params: {} 2022-09-12T20:16:14.795 DEBUG:boto:Token: None 2022-09-12T20:16:14.796 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:14.797 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m6YLNFDu4prpY2F/uk/ORRKrgS4= 2022-09-12T20:16:14.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m6YLNFDu4prpY2F/uk/ORRKrgS4=', 'Content-Length': '0'} 2022-09-12T20:16:14.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066d53b1556e084ca-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:16:14.803 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:14.804 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:14.804 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:14.805 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:14.806 DEBUG:boto:Method: GET 2022-09-12T20:16:14.807 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:14.807 DEBUG:boto:Data: 2022-09-12T20:16:14.808 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.809 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.809 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.810 DEBUG:boto:Params: {} 2022-09-12T20:16:14.811 DEBUG:boto:Token: None 2022-09-12T20:16:14.811 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:14.812 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+KeMpbovSggqDoouPOeEXL5YKmY= 2022-09-12T20:16:14.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+KeMpbovSggqDoouPOeEXL5YKmY=', 'Content-Length': '0'} 2022-09-12T20:16:14.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfec33065865e56e-00631f938e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.817 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:14.818 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:14.819 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:14.819 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:14.820 DEBUG:boto:Method: GET 2022-09-12T20:16:14.821 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:14.821 DEBUG:boto:Data: 2022-09-12T20:16:14.822 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.823 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.823 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.824 DEBUG:boto:Params: {} 2022-09-12T20:16:14.825 DEBUG:boto:Token: None 2022-09-12T20:16:14.825 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:14.826 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+KeMpbovSggqDoouPOeEXL5YKmY= 2022-09-12T20:16:14.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+KeMpbovSggqDoouPOeEXL5YKmY=', 'Content-Length': '0'} 2022-09-12T20:16:14.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3f04e00db6382fd-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.831 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:14.832 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:14.832 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:16:14.833 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:16:14.833 DEBUG:boto:Method: GET 2022-09-12T20:16:14.834 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:16:14.835 DEBUG:boto:Data: 2022-09-12T20:16:14.835 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.836 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.836 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.837 DEBUG:boto:Params: {} 2022-09-12T20:16:14.838 DEBUG:boto:Token: None 2022-09-12T20:16:14.839 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/ 2022-09-12T20:16:14.840 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA= 2022-09-12T20:16:14.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA=', 'Content-Length': '0'} 2022-09-12T20:16:14.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091a6caeef4115707-00631f938e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.846 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:14.847 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:14.848 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:14.849 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:16:14.850 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:16:14.850 DEBUG:boto:Method: GET 2022-09-12T20:16:14.851 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:16:14.852 DEBUG:boto:Data: 2022-09-12T20:16:14.853 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.853 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.854 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.854 DEBUG:boto:Params: {} 2022-09-12T20:16:14.855 DEBUG:boto:Token: None 2022-09-12T20:16:14.856 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/ 2022-09-12T20:16:14.856 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA= 2022-09-12T20:16:14.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA=', 'Content-Length': '0'} 2022-09-12T20:16:14.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000831f4e46ec2019b-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.906 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:14.908 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:14.909 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:14.910 DEBUG:boto:Method: HEAD 2022-09-12T20:16:14.911 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:16:14.912 DEBUG:boto:Data: 2022-09-12T20:16:14.913 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.913 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.914 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.915 DEBUG:boto:Params: {} 2022-09-12T20:16:14.916 DEBUG:boto:Token: None 2022-09-12T20:16:14.917 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj85 2022-09-12T20:16:14.918 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wyGCfhat2xhJgjGiDTEPBdQgG+g= 2022-09-12T20:16:14.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wyGCfhat2xhJgjGiDTEPBdQgG+g=', 'Content-Length': '0'} 2022-09-12T20:16:14.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e9f24be29cf3029-00631f938e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.921 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:14.922 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:14.923 DEBUG:boto:Method: HEAD 2022-09-12T20:16:14.923 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:16:14.924 DEBUG:boto:Data: 2022-09-12T20:16:14.925 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.925 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.926 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.926 DEBUG:boto:Params: {} 2022-09-12T20:16:14.927 DEBUG:boto:Token: None 2022-09-12T20:16:14.928 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj85 2022-09-12T20:16:14.929 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wyGCfhat2xhJgjGiDTEPBdQgG+g= 2022-09-12T20:16:14.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wyGCfhat2xhJgjGiDTEPBdQgG+g=', 'Content-Length': '0'} 2022-09-12T20:16:14.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7a1351d12cb0185-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:16:14.934 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:14.935 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:14.935 DEBUG:boto:Method: GET 2022-09-12T20:16:14.936 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:16:14.936 DEBUG:boto:Data: 2022-09-12T20:16:14.937 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.938 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.939 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.940 DEBUG:boto:Params: {} 2022-09-12T20:16:14.940 DEBUG:boto:Token: None 2022-09-12T20:16:14.941 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj85 2022-09-12T20:16:14.942 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OJFn1gYFFCyd6WoQK4xIEDsdquA= 2022-09-12T20:16:14.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OJFn1gYFFCyd6WoQK4xIEDsdquA=', 'Content-Length': '0'} 2022-09-12T20:16:14.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d62edbd9e42594f-00631f938e-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.948 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:14.948 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:14.949 DEBUG:boto:Method: GET 2022-09-12T20:16:14.950 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:16:14.950 DEBUG:boto:Data: 2022-09-12T20:16:14.951 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.952 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.952 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.953 DEBUG:boto:Params: {} 2022-09-12T20:16:14.954 DEBUG:boto:Token: None 2022-09-12T20:16:14.955 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/obj85 2022-09-12T20:16:14.955 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OJFn1gYFFCyd6WoQK4xIEDsdquA= 2022-09-12T20:16:14.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OJFn1gYFFCyd6WoQK4xIEDsdquA=', 'Content-Length': '0'} 2022-09-12T20:16:14.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e9abf5e08ab1b0c-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.960 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:14.960 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:14.961 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:16:14.961 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:16:14.963 DEBUG:boto:Method: GET 2022-09-12T20:16:14.966 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:16:14.969 DEBUG:boto:Data: 2022-09-12T20:16:14.971 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.972 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:14.973 DEBUG:boto:Port: 8001 2022-09-12T20:16:14.975 DEBUG:boto:Params: {} 2022-09-12T20:16:14.976 DEBUG:boto:Token: None 2022-09-12T20:16:14.977 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/ 2022-09-12T20:16:14.979 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA= 2022-09-12T20:16:14.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA=', 'Content-Length': '0'} 2022-09-12T20:16:14.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000730addfd50bea206-00631f938e-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:14.988 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:14.989 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:14.989 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:14.990 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:16:14.990 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:16:14.991 DEBUG:boto:Method: GET 2022-09-12T20:16:14.992 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:16:14.992 DEBUG:boto:Data: 2022-09-12T20:16:14.993 DEBUG:boto:Headers: {} 2022-09-12T20:16:14.993 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:14.994 DEBUG:boto:Port: 8000 2022-09-12T20:16:14.995 DEBUG:boto:Params: {} 2022-09-12T20:16:14.995 DEBUG:boto:Token: None 2022-09-12T20:16:14.996 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:14 GMT /jdpiak-75/ 2022-09-12T20:16:14.996 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA= 2022-09-12T20:16:14.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:14 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nrit44GRB+Vy05L2WMwzOcNVhIA=', 'Content-Length': '0'} 2022-09-12T20:16:15.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2014b7b1e028654-00631f938e-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.004 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:15.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:16:15.005 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:15.006 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:15.006 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:15.007 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:15.008 DEBUG:boto:Method: GET 2022-09-12T20:16:15.008 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:15.009 DEBUG:boto:Data: 2022-09-12T20:16:15.010 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.011 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.012 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.012 DEBUG:boto:Params: {} 2022-09-12T20:16:15.013 DEBUG:boto:Token: None 2022-09-12T20:16:15.014 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:15.015 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6lx0Ok0TRllWXgkFYSxmP7mUtT0= 2022-09-12T20:16:15.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6lx0Ok0TRllWXgkFYSxmP7mUtT0=', 'Content-Length': '0'} 2022-09-12T20:16:15.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075965833ec99dbe3-00631f938f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.022 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:15.022 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:15.023 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:15.023 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:15.024 DEBUG:boto:Method: GET 2022-09-12T20:16:15.025 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:15.026 DEBUG:boto:Data: 2022-09-12T20:16:15.026 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.027 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.027 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.028 DEBUG:boto:Params: {} 2022-09-12T20:16:15.029 DEBUG:boto:Token: None 2022-09-12T20:16:15.030 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:15.030 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6lx0Ok0TRllWXgkFYSxmP7mUtT0= 2022-09-12T20:16:15.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6lx0Ok0TRllWXgkFYSxmP7mUtT0=', 'Content-Length': '0'} 2022-09-12T20:16:15.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a86b43d512ba5483-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.078 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:15.079 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:15.080 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:15.080 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:15.081 DEBUG:boto:Method: HEAD 2022-09-12T20:16:15.081 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:15.082 DEBUG:boto:Data: 2022-09-12T20:16:15.082 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.083 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.084 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.084 DEBUG:boto:Params: {} 2022-09-12T20:16:15.085 DEBUG:boto:Token: None 2022-09-12T20:16:15.086 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:15.087 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zFrWqxVkg3oBp2K9jKbh6aMoNIY= 2022-09-12T20:16:15.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zFrWqxVkg3oBp2K9jKbh6aMoNIY=', 'Content-Length': '0'} 2022-09-12T20:16:15.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce7863b249d48d5f-00631f938f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.093 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:15.094 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:15.094 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:15.095 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:15.095 DEBUG:boto:Method: HEAD 2022-09-12T20:16:15.096 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:15.096 DEBUG:boto:Data: 2022-09-12T20:16:15.097 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.098 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.098 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.099 DEBUG:boto:Params: {} 2022-09-12T20:16:15.099 DEBUG:boto:Token: None 2022-09-12T20:16:15.100 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:15.101 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zFrWqxVkg3oBp2K9jKbh6aMoNIY= 2022-09-12T20:16:15.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zFrWqxVkg3oBp2K9jKbh6aMoNIY=', 'Content-Length': '0'} 2022-09-12T20:16:15.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ada311922323897f-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:16:15.105 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:15.106 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:15.106 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:15.107 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:15.107 DEBUG:boto:Method: GET 2022-09-12T20:16:15.108 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:15.109 DEBUG:boto:Data: 2022-09-12T20:16:15.109 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.110 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.110 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.111 DEBUG:boto:Params: {} 2022-09-12T20:16:15.113 DEBUG:boto:Token: None 2022-09-12T20:16:15.114 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:15.115 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6lx0Ok0TRllWXgkFYSxmP7mUtT0= 2022-09-12T20:16:15.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6lx0Ok0TRllWXgkFYSxmP7mUtT0=', 'Content-Length': '0'} 2022-09-12T20:16:15.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e94cf1464fb851ac-00631f938f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.163 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:15.164 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:15.164 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:15.165 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:15.165 DEBUG:boto:Method: GET 2022-09-12T20:16:15.166 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:15.167 DEBUG:boto:Data: 2022-09-12T20:16:15.167 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.168 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.168 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.169 DEBUG:boto:Params: {} 2022-09-12T20:16:15.169 DEBUG:boto:Token: None 2022-09-12T20:16:15.170 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:15.171 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6lx0Ok0TRllWXgkFYSxmP7mUtT0= 2022-09-12T20:16:15.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6lx0Ok0TRllWXgkFYSxmP7mUtT0=', 'Content-Length': '0'} 2022-09-12T20:16:15.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000487f74b02d8f1bd5-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.175 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:15.175 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:15.176 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:16:15.176 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:16:15.177 DEBUG:boto:Method: GET 2022-09-12T20:16:15.177 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:16:15.178 DEBUG:boto:Data: 2022-09-12T20:16:15.178 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.179 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.180 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.180 DEBUG:boto:Params: {} 2022-09-12T20:16:15.181 DEBUG:boto:Token: None 2022-09-12T20:16:15.181 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/ 2022-09-12T20:16:15.182 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs= 2022-09-12T20:16:15.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs=', 'Content-Length': '0'} 2022-09-12T20:16:15.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d47a996b27f5845d-00631f938f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.186 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:15.187 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:15.188 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:15.188 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:16:15.189 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:16:15.189 DEBUG:boto:Method: GET 2022-09-12T20:16:15.190 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:16:15.190 DEBUG:boto:Data: 2022-09-12T20:16:15.191 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.191 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.192 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.192 DEBUG:boto:Params: {} 2022-09-12T20:16:15.193 DEBUG:boto:Token: None 2022-09-12T20:16:15.194 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/ 2022-09-12T20:16:15.194 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs= 2022-09-12T20:16:15.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs=', 'Content-Length': '0'} 2022-09-12T20:16:15.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab715f61929733ba-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.202 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:15.202 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:15.203 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:15.203 DEBUG:boto:Method: HEAD 2022-09-12T20:16:15.204 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:16:15.204 DEBUG:boto:Data: 2022-09-12T20:16:15.205 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.205 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.206 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.206 DEBUG:boto:Params: {} 2022-09-12T20:16:15.207 DEBUG:boto:Token: None 2022-09-12T20:16:15.208 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj87 2022-09-12T20:16:15.208 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EXPSkX/qIV82xn4n0ZBFwurZpTU= 2022-09-12T20:16:15.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EXPSkX/qIV82xn4n0ZBFwurZpTU=', 'Content-Length': '0'} 2022-09-12T20:16:15.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072cb4990b56a34e8-00631f938f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.212 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:15.213 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:15.213 DEBUG:boto:Method: HEAD 2022-09-12T20:16:15.214 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:16:15.215 DEBUG:boto:Data: 2022-09-12T20:16:15.215 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.216 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.216 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.217 DEBUG:boto:Params: {} 2022-09-12T20:16:15.217 DEBUG:boto:Token: None 2022-09-12T20:16:15.218 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj87 2022-09-12T20:16:15.219 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EXPSkX/qIV82xn4n0ZBFwurZpTU= 2022-09-12T20:16:15.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EXPSkX/qIV82xn4n0ZBFwurZpTU=', 'Content-Length': '0'} 2022-09-12T20:16:15.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aa92d3f270e9dcb-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:16:15.226 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:15.227 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:15.228 DEBUG:boto:Method: GET 2022-09-12T20:16:15.228 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:16:15.229 DEBUG:boto:Data: 2022-09-12T20:16:15.229 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.230 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.230 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.231 DEBUG:boto:Params: {} 2022-09-12T20:16:15.232 DEBUG:boto:Token: None 2022-09-12T20:16:15.232 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj87 2022-09-12T20:16:15.233 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bogpp3Y7AuhbIYYG+MSffOq6DYQ= 2022-09-12T20:16:15.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bogpp3Y7AuhbIYYG+MSffOq6DYQ=', 'Content-Length': '0'} 2022-09-12T20:16:15.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023892a27f6394f4f-00631f938f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.238 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:15.239 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:15.240 DEBUG:boto:Method: GET 2022-09-12T20:16:15.240 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:16:15.241 DEBUG:boto:Data: 2022-09-12T20:16:15.242 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.242 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.243 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.243 DEBUG:boto:Params: {} 2022-09-12T20:16:15.244 DEBUG:boto:Token: None 2022-09-12T20:16:15.245 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj87 2022-09-12T20:16:15.247 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bogpp3Y7AuhbIYYG+MSffOq6DYQ= 2022-09-12T20:16:15.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bogpp3Y7AuhbIYYG+MSffOq6DYQ=', 'Content-Length': '0'} 2022-09-12T20:16:15.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e76e33624a29bfd9-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.295 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:15.296 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:15.297 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:16:15.298 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:16:15.299 DEBUG:boto:Method: GET 2022-09-12T20:16:15.300 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:16:15.302 DEBUG:boto:Data: 2022-09-12T20:16:15.303 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.303 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.304 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.305 DEBUG:boto:Params: {} 2022-09-12T20:16:15.305 DEBUG:boto:Token: None 2022-09-12T20:16:15.306 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/ 2022-09-12T20:16:15.308 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs= 2022-09-12T20:16:15.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs=', 'Content-Length': '0'} 2022-09-12T20:16:15.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041e86ccc7cdd62d0-00631f938f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.313 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:15.314 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:15.314 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:15.315 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:16:15.315 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:16:15.316 DEBUG:boto:Method: GET 2022-09-12T20:16:15.316 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:16:15.317 DEBUG:boto:Data: 2022-09-12T20:16:15.318 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.318 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.319 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.320 DEBUG:boto:Params: {} 2022-09-12T20:16:15.320 DEBUG:boto:Token: None 2022-09-12T20:16:15.321 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/ 2022-09-12T20:16:15.322 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs= 2022-09-12T20:16:15.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs=', 'Content-Length': '0'} 2022-09-12T20:16:15.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018c02c4dfe391395-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.328 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:15.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:16:15.330 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:15.330 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:15.331 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:15.332 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:15.332 DEBUG:boto:Method: GET 2022-09-12T20:16:15.333 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:15.333 DEBUG:boto:Data: 2022-09-12T20:16:15.334 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.334 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.335 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.336 DEBUG:boto:Params: {} 2022-09-12T20:16:15.336 DEBUG:boto:Token: None 2022-09-12T20:16:15.337 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:15.338 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cCnBXF862kvL7k5eGX8L3r6TCD0= 2022-09-12T20:16:15.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cCnBXF862kvL7k5eGX8L3r6TCD0=', 'Content-Length': '0'} 2022-09-12T20:16:15.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfa24df632d322a2-00631f938f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.342 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:15.342 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:15.343 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:15.344 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:15.344 DEBUG:boto:Method: GET 2022-09-12T20:16:15.345 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:15.345 DEBUG:boto:Data: 2022-09-12T20:16:15.346 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.347 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.348 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.348 DEBUG:boto:Params: {} 2022-09-12T20:16:15.349 DEBUG:boto:Token: None 2022-09-12T20:16:15.350 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:15.350 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cCnBXF862kvL7k5eGX8L3r6TCD0= 2022-09-12T20:16:15.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cCnBXF862kvL7k5eGX8L3r6TCD0=', 'Content-Length': '0'} 2022-09-12T20:16:15.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b004d3f659df2a7e-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.355 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:15.356 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:15.356 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:15.357 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:15.357 DEBUG:boto:Method: HEAD 2022-09-12T20:16:15.358 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:15.359 DEBUG:boto:Data: 2022-09-12T20:16:15.359 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.360 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.360 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.361 DEBUG:boto:Params: {} 2022-09-12T20:16:15.361 DEBUG:boto:Token: None 2022-09-12T20:16:15.362 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:15.362 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:M+Q6K86TDzUe/pDXvG3O1ZCRQ44= 2022-09-12T20:16:15.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:M+Q6K86TDzUe/pDXvG3O1ZCRQ44=', 'Content-Length': '0'} 2022-09-12T20:16:15.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001741c0d25eadcdda-00631f938f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.366 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:15.366 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:15.367 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:15.368 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:15.368 DEBUG:boto:Method: HEAD 2022-09-12T20:16:15.369 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:15.369 DEBUG:boto:Data: 2022-09-12T20:16:15.370 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.370 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.371 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.371 DEBUG:boto:Params: {} 2022-09-12T20:16:15.372 DEBUG:boto:Token: None 2022-09-12T20:16:15.373 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:15.374 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:M+Q6K86TDzUe/pDXvG3O1ZCRQ44= 2022-09-12T20:16:15.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:M+Q6K86TDzUe/pDXvG3O1ZCRQ44=', 'Content-Length': '0'} 2022-09-12T20:16:15.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000678c29f427fdaefc-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:16:15.378 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:15.379 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:15.380 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:15.380 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:15.381 DEBUG:boto:Method: GET 2022-09-12T20:16:15.381 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:15.382 DEBUG:boto:Data: 2022-09-12T20:16:15.383 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.383 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.384 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.385 DEBUG:boto:Params: {} 2022-09-12T20:16:15.386 DEBUG:boto:Token: None 2022-09-12T20:16:15.387 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:15.387 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cCnBXF862kvL7k5eGX8L3r6TCD0= 2022-09-12T20:16:15.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cCnBXF862kvL7k5eGX8L3r6TCD0=', 'Content-Length': '0'} 2022-09-12T20:16:15.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050542111c53f1e32-00631f938f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.391 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:15.392 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:15.393 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:15.393 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:15.394 DEBUG:boto:Method: GET 2022-09-12T20:16:15.394 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:15.395 DEBUG:boto:Data: 2022-09-12T20:16:15.395 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.396 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.396 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.397 DEBUG:boto:Params: {} 2022-09-12T20:16:15.398 DEBUG:boto:Token: None 2022-09-12T20:16:15.398 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:15.399 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cCnBXF862kvL7k5eGX8L3r6TCD0= 2022-09-12T20:16:15.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cCnBXF862kvL7k5eGX8L3r6TCD0=', 'Content-Length': '0'} 2022-09-12T20:16:15.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d3312db736c2253-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.446 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:15.447 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:15.448 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:16:15.449 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:16:15.450 DEBUG:boto:Method: GET 2022-09-12T20:16:15.451 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:16:15.452 DEBUG:boto:Data: 2022-09-12T20:16:15.453 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.454 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.455 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.456 DEBUG:boto:Params: {} 2022-09-12T20:16:15.457 DEBUG:boto:Token: None 2022-09-12T20:16:15.458 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/ 2022-09-12T20:16:15.459 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs= 2022-09-12T20:16:15.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs=', 'Content-Length': '0'} 2022-09-12T20:16:15.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003afe63c4ba9765c1-00631f938f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.464 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:15.465 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:15.466 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:15.466 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:16:15.467 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:16:15.467 DEBUG:boto:Method: GET 2022-09-12T20:16:15.468 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:16:15.468 DEBUG:boto:Data: 2022-09-12T20:16:15.469 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.469 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.470 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.470 DEBUG:boto:Params: {} 2022-09-12T20:16:15.471 DEBUG:boto:Token: None 2022-09-12T20:16:15.471 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/ 2022-09-12T20:16:15.472 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs= 2022-09-12T20:16:15.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs=', 'Content-Length': '0'} 2022-09-12T20:16:15.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000175ba2a524b80df3-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.478 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:15.479 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:15.480 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:15.480 DEBUG:boto:Method: HEAD 2022-09-12T20:16:15.481 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:16:15.481 DEBUG:boto:Data: 2022-09-12T20:16:15.482 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.482 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.483 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.483 DEBUG:boto:Params: {} 2022-09-12T20:16:15.484 DEBUG:boto:Token: None 2022-09-12T20:16:15.485 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj92 2022-09-12T20:16:15.485 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VB1zRv2uTuhPJHlsfxN72V9l8yQ= 2022-09-12T20:16:15.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VB1zRv2uTuhPJHlsfxN72V9l8yQ=', 'Content-Length': '0'} 2022-09-12T20:16:15.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2bc4203603551fa-00631f938f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.489 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:15.489 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:15.490 DEBUG:boto:Method: HEAD 2022-09-12T20:16:15.490 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:16:15.491 DEBUG:boto:Data: 2022-09-12T20:16:15.491 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.492 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.493 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.493 DEBUG:boto:Params: {} 2022-09-12T20:16:15.494 DEBUG:boto:Token: None 2022-09-12T20:16:15.494 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj92 2022-09-12T20:16:15.495 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:VB1zRv2uTuhPJHlsfxN72V9l8yQ= 2022-09-12T20:16:15.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:VB1zRv2uTuhPJHlsfxN72V9l8yQ=', 'Content-Length': '0'} 2022-09-12T20:16:15.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bbde97e3363a55c-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:16:15.499 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:15.499 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:15.500 DEBUG:boto:Method: GET 2022-09-12T20:16:15.500 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:16:15.501 DEBUG:boto:Data: 2022-09-12T20:16:15.501 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.502 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.502 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.503 DEBUG:boto:Params: {} 2022-09-12T20:16:15.504 DEBUG:boto:Token: None 2022-09-12T20:16:15.504 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj92 2022-09-12T20:16:15.505 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/LrlU9mp4OaIKPpJdKwYC1Lt3j0= 2022-09-12T20:16:15.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/LrlU9mp4OaIKPpJdKwYC1Lt3j0=', 'Content-Length': '0'} 2022-09-12T20:16:15.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a920acf69bfab62-00631f938f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.550 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:15.551 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:15.552 DEBUG:boto:Method: GET 2022-09-12T20:16:15.552 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:16:15.553 DEBUG:boto:Data: 2022-09-12T20:16:15.553 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.554 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.554 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.555 DEBUG:boto:Params: {} 2022-09-12T20:16:15.556 DEBUG:boto:Token: None 2022-09-12T20:16:15.556 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj92 2022-09-12T20:16:15.557 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/LrlU9mp4OaIKPpJdKwYC1Lt3j0= 2022-09-12T20:16:15.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/LrlU9mp4OaIKPpJdKwYC1Lt3j0=', 'Content-Length': '0'} 2022-09-12T20:16:15.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fce85dba03251b51-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.560 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:15.561 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:15.562 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:16:15.562 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:16:15.563 DEBUG:boto:Method: GET 2022-09-12T20:16:15.563 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:16:15.564 DEBUG:boto:Data: 2022-09-12T20:16:15.564 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.565 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.565 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.566 DEBUG:boto:Params: {} 2022-09-12T20:16:15.566 DEBUG:boto:Token: None 2022-09-12T20:16:15.567 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/ 2022-09-12T20:16:15.568 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs= 2022-09-12T20:16:15.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs=', 'Content-Length': '0'} 2022-09-12T20:16:15.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038d26e721b590088-00631f938f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.572 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:15.572 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:15.573 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:15.574 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:16:15.574 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:16:15.575 DEBUG:boto:Method: GET 2022-09-12T20:16:15.575 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:16:15.576 DEBUG:boto:Data: 2022-09-12T20:16:15.576 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.577 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.577 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.578 DEBUG:boto:Params: {} 2022-09-12T20:16:15.578 DEBUG:boto:Token: None 2022-09-12T20:16:15.579 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/ 2022-09-12T20:16:15.580 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs= 2022-09-12T20:16:15.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs=', 'Content-Length': '0'} 2022-09-12T20:16:15.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a980c8dfe60a7f9-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.586 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:15.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-12T20:16:15.587 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:15.588 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:15.588 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:15.589 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:15.589 DEBUG:boto:Method: GET 2022-09-12T20:16:15.590 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:15.590 DEBUG:boto:Data: 2022-09-12T20:16:15.591 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.592 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.592 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.593 DEBUG:boto:Params: {} 2022-09-12T20:16:15.593 DEBUG:boto:Token: None 2022-09-12T20:16:15.594 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:15.594 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rjw5NWwZD+i2cqM1ziLeEb5G27k= 2022-09-12T20:16:15.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rjw5NWwZD+i2cqM1ziLeEb5G27k=', 'Content-Length': '0'} 2022-09-12T20:16:15.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077c815bd02f7c2bd-00631f938f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.598 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:15.599 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:15.599 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:15.600 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:15.600 DEBUG:boto:Method: GET 2022-09-12T20:16:15.601 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:15.601 DEBUG:boto:Data: 2022-09-12T20:16:15.602 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.602 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.603 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.604 DEBUG:boto:Params: {} 2022-09-12T20:16:15.604 DEBUG:boto:Token: None 2022-09-12T20:16:15.605 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:15.605 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rjw5NWwZD+i2cqM1ziLeEb5G27k= 2022-09-12T20:16:15.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rjw5NWwZD+i2cqM1ziLeEb5G27k=', 'Content-Length': '0'} 2022-09-12T20:16:15.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d44f1461d64df7e-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.650 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:15.651 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:15.651 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:15.652 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:15.652 DEBUG:boto:Method: HEAD 2022-09-12T20:16:15.653 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:15.653 DEBUG:boto:Data: 2022-09-12T20:16:15.654 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.654 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.655 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.655 DEBUG:boto:Params: {} 2022-09-12T20:16:15.656 DEBUG:boto:Token: None 2022-09-12T20:16:15.656 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:15.657 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FFPQSOq0dqBgui0df27vZkNdBYQ= 2022-09-12T20:16:15.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FFPQSOq0dqBgui0df27vZkNdBYQ=', 'Content-Length': '0'} 2022-09-12T20:16:15.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090a970d305b8f5eb-00631f938f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.661 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:15.661 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:15.662 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:15.662 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:15.663 DEBUG:boto:Method: HEAD 2022-09-12T20:16:15.663 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:15.664 DEBUG:boto:Data: 2022-09-12T20:16:15.664 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.665 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.665 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.666 DEBUG:boto:Params: {} 2022-09-12T20:16:15.666 DEBUG:boto:Token: None 2022-09-12T20:16:15.667 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:15.667 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FFPQSOq0dqBgui0df27vZkNdBYQ= 2022-09-12T20:16:15.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FFPQSOq0dqBgui0df27vZkNdBYQ=', 'Content-Length': '0'} 2022-09-12T20:16:15.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000361aad534a688f6e-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-12T20:16:15.671 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:15.672 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:15.673 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:15.673 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:15.674 DEBUG:boto:Method: GET 2022-09-12T20:16:15.674 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:15.675 DEBUG:boto:Data: 2022-09-12T20:16:15.675 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.676 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.676 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.677 DEBUG:boto:Params: {} 2022-09-12T20:16:15.677 DEBUG:boto:Token: None 2022-09-12T20:16:15.678 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:15.679 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rjw5NWwZD+i2cqM1ziLeEb5G27k= 2022-09-12T20:16:15.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rjw5NWwZD+i2cqM1ziLeEb5G27k=', 'Content-Length': '0'} 2022-09-12T20:16:15.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000618cd523c3c87310-00631f938f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.682 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:15.683 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:15.683 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:15.684 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:15.684 DEBUG:boto:Method: GET 2022-09-12T20:16:15.685 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:15.685 DEBUG:boto:Data: 2022-09-12T20:16:15.686 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.687 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.687 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.688 DEBUG:boto:Params: {} 2022-09-12T20:16:15.688 DEBUG:boto:Token: None 2022-09-12T20:16:15.689 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:15.689 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rjw5NWwZD+i2cqM1ziLeEb5G27k= 2022-09-12T20:16:15.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rjw5NWwZD+i2cqM1ziLeEb5G27k=', 'Content-Length': '0'} 2022-09-12T20:16:15.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd1aac42cfdfea1f-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.693 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:15.694 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:15.694 DEBUG:boto:path=/jdpiak-75/?prefix=obj93 2022-09-12T20:16:15.695 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj93 2022-09-12T20:16:15.695 DEBUG:boto:Method: GET 2022-09-12T20:16:15.696 DEBUG:boto:Path: /jdpiak-75/?prefix=obj93 2022-09-12T20:16:15.696 DEBUG:boto:Data: 2022-09-12T20:16:15.697 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.697 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.698 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.699 DEBUG:boto:Params: {} 2022-09-12T20:16:15.699 DEBUG:boto:Token: None 2022-09-12T20:16:15.700 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/ 2022-09-12T20:16:15.700 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs= 2022-09-12T20:16:15.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs=', 'Content-Length': '0'} 2022-09-12T20:16:15.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbc5895ddc68a38f-00631f938f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.704 DEBUG:boto:b'jdpiak-75obj931000falseobj932022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:15.705 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:15.706 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:15.706 DEBUG:boto:path=/jdpiak-75/?prefix=obj93 2022-09-12T20:16:15.707 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj93 2022-09-12T20:16:15.707 DEBUG:boto:Method: GET 2022-09-12T20:16:15.708 DEBUG:boto:Path: /jdpiak-75/?prefix=obj93 2022-09-12T20:16:15.708 DEBUG:boto:Data: 2022-09-12T20:16:15.709 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.709 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.710 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.711 DEBUG:boto:Params: {} 2022-09-12T20:16:15.711 DEBUG:boto:Token: None 2022-09-12T20:16:15.712 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/ 2022-09-12T20:16:15.712 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs= 2022-09-12T20:16:15.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs=', 'Content-Length': '0'} 2022-09-12T20:16:15.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed3f2f96e9a1cc23-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.718 DEBUG:boto:b'jdpiak-75obj931000falseobj932022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:15.719 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:15.719 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:15.720 DEBUG:boto:Method: HEAD 2022-09-12T20:16:15.720 DEBUG:boto:Path: /jdpiak-75/obj93 2022-09-12T20:16:15.721 DEBUG:boto:Data: 2022-09-12T20:16:15.721 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.722 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.722 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.723 DEBUG:boto:Params: {} 2022-09-12T20:16:15.724 DEBUG:boto:Token: None 2022-09-12T20:16:15.724 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj93 2022-09-12T20:16:15.725 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BPdBZbN7X7l+NOBjSIg52j+918s= 2022-09-12T20:16:15.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BPdBZbN7X7l+NOBjSIg52j+918s=', 'Content-Length': '0'} 2022-09-12T20:16:15.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000298036639ae55120-00631f938f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.728 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:15.729 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:15.729 DEBUG:boto:Method: HEAD 2022-09-12T20:16:15.730 DEBUG:boto:Path: /jdpiak-75/obj93 2022-09-12T20:16:15.730 DEBUG:boto:Data: 2022-09-12T20:16:15.731 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.731 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.732 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.732 DEBUG:boto:Params: {} 2022-09-12T20:16:15.733 DEBUG:boto:Token: None 2022-09-12T20:16:15.734 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj93 2022-09-12T20:16:15.734 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BPdBZbN7X7l+NOBjSIg52j+918s= 2022-09-12T20:16:15.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BPdBZbN7X7l+NOBjSIg52j+918s=', 'Content-Length': '0'} 2022-09-12T20:16:15.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9abbc456cfeb7d1-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-12T20:16:15.738 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:15.739 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:15.739 DEBUG:boto:Method: GET 2022-09-12T20:16:15.740 DEBUG:boto:Path: /jdpiak-75/obj93 2022-09-12T20:16:15.740 DEBUG:boto:Data: 2022-09-12T20:16:15.741 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.741 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.742 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.743 DEBUG:boto:Params: {} 2022-09-12T20:16:15.743 DEBUG:boto:Token: None 2022-09-12T20:16:15.744 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj93 2022-09-12T20:16:15.745 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IVEU83V2OH31oLxT64rD8pyiqaw= 2022-09-12T20:16:15.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IVEU83V2OH31oLxT64rD8pyiqaw=', 'Content-Length': '0'} 2022-09-12T20:16:15.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e41438c0c6e6a53a-00631f938f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.790 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:15.791 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:15.791 DEBUG:boto:Method: GET 2022-09-12T20:16:15.792 DEBUG:boto:Path: /jdpiak-75/obj93 2022-09-12T20:16:15.792 DEBUG:boto:Data: 2022-09-12T20:16:15.793 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.793 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.794 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.795 DEBUG:boto:Params: {} 2022-09-12T20:16:15.796 DEBUG:boto:Token: None 2022-09-12T20:16:15.796 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj93 2022-09-12T20:16:15.797 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IVEU83V2OH31oLxT64rD8pyiqaw= 2022-09-12T20:16:15.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IVEU83V2OH31oLxT64rD8pyiqaw=', 'Content-Length': '0'} 2022-09-12T20:16:15.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093ba444a04cbfe76-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.801 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:15.801 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:15.802 DEBUG:boto:path=/jdpiak-75/?prefix=obj93 2022-09-12T20:16:15.802 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj93 2022-09-12T20:16:15.803 DEBUG:boto:Method: GET 2022-09-12T20:16:15.804 DEBUG:boto:Path: /jdpiak-75/?prefix=obj93 2022-09-12T20:16:15.804 DEBUG:boto:Data: 2022-09-12T20:16:15.805 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.805 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.806 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.807 DEBUG:boto:Params: {} 2022-09-12T20:16:15.807 DEBUG:boto:Token: None 2022-09-12T20:16:15.808 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/ 2022-09-12T20:16:15.808 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs= 2022-09-12T20:16:15.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs=', 'Content-Length': '0'} 2022-09-12T20:16:15.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000602ac9a9b191b261-00631f938f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.812 DEBUG:boto:b'jdpiak-75obj931000falseobj932022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:15.813 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:15.814 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:15.815 DEBUG:boto:path=/jdpiak-75/?prefix=obj93 2022-09-12T20:16:15.815 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj93 2022-09-12T20:16:15.816 DEBUG:boto:Method: GET 2022-09-12T20:16:15.816 DEBUG:boto:Path: /jdpiak-75/?prefix=obj93 2022-09-12T20:16:15.817 DEBUG:boto:Data: 2022-09-12T20:16:15.817 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.818 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.818 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.819 DEBUG:boto:Params: {} 2022-09-12T20:16:15.819 DEBUG:boto:Token: None 2022-09-12T20:16:15.820 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/ 2022-09-12T20:16:15.821 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs= 2022-09-12T20:16:15.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs=', 'Content-Length': '0'} 2022-09-12T20:16:15.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fee1627f02a3825-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.827 DEBUG:boto:b'jdpiak-75obj931000falseobj932022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:15.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:16:15.828 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:15.829 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:15.830 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:15.830 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:15.831 DEBUG:boto:Method: GET 2022-09-12T20:16:15.831 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:15.832 DEBUG:boto:Data: 2022-09-12T20:16:15.832 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.833 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.833 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.834 DEBUG:boto:Params: {} 2022-09-12T20:16:15.834 DEBUG:boto:Token: None 2022-09-12T20:16:15.835 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:15.836 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6Y000imVF9hUBvRWDgmq8nU5+fc= 2022-09-12T20:16:15.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6Y000imVF9hUBvRWDgmq8nU5+fc=', 'Content-Length': '0'} 2022-09-12T20:16:15.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c76d388f89872507-00631f938f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.839 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:15.840 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:15.841 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:15.841 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:15.842 DEBUG:boto:Method: GET 2022-09-12T20:16:15.842 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:15.843 DEBUG:boto:Data: 2022-09-12T20:16:15.843 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.844 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.844 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.845 DEBUG:boto:Params: {} 2022-09-12T20:16:15.846 DEBUG:boto:Token: None 2022-09-12T20:16:15.846 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:15.847 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6Y000imVF9hUBvRWDgmq8nU5+fc= 2022-09-12T20:16:15.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6Y000imVF9hUBvRWDgmq8nU5+fc=', 'Content-Length': '0'} 2022-09-12T20:16:15.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b9707bde29f26ee-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.850 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:15.851 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:15.851 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:15.852 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:15.852 DEBUG:boto:Method: HEAD 2022-09-12T20:16:15.853 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:15.853 DEBUG:boto:Data: 2022-09-12T20:16:15.854 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.854 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.855 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.855 DEBUG:boto:Params: {} 2022-09-12T20:16:15.856 DEBUG:boto:Token: None 2022-09-12T20:16:15.856 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:15.857 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dpd3wtHck5QGKl7F252m8v0r+Do= 2022-09-12T20:16:15.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dpd3wtHck5QGKl7F252m8v0r+Do=', 'Content-Length': '0'} 2022-09-12T20:16:15.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006de6b25fd1f868b9-00631f938f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.860 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:15.861 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:15.861 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:15.862 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:15.862 DEBUG:boto:Method: HEAD 2022-09-12T20:16:15.863 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:15.863 DEBUG:boto:Data: 2022-09-12T20:16:15.864 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.864 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.865 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.865 DEBUG:boto:Params: {} 2022-09-12T20:16:15.866 DEBUG:boto:Token: None 2022-09-12T20:16:15.866 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:15.867 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Dpd3wtHck5QGKl7F252m8v0r+Do= 2022-09-12T20:16:15.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Dpd3wtHck5QGKl7F252m8v0r+Do=', 'Content-Length': '0'} 2022-09-12T20:16:15.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cbcc01344db9a64-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:16:15.871 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:15.872 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:15.872 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:15.873 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:15.873 DEBUG:boto:Method: GET 2022-09-12T20:16:15.874 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:15.874 DEBUG:boto:Data: 2022-09-12T20:16:15.875 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.876 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.876 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.877 DEBUG:boto:Params: {} 2022-09-12T20:16:15.877 DEBUG:boto:Token: None 2022-09-12T20:16:15.878 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:15.878 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6Y000imVF9hUBvRWDgmq8nU5+fc= 2022-09-12T20:16:15.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6Y000imVF9hUBvRWDgmq8nU5+fc=', 'Content-Length': '0'} 2022-09-12T20:16:15.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003db639ca164016a2-00631f938f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.922 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:15.923 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:15.923 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:15.924 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:15.925 DEBUG:boto:Method: GET 2022-09-12T20:16:15.925 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:15.926 DEBUG:boto:Data: 2022-09-12T20:16:15.926 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.927 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.927 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.928 DEBUG:boto:Params: {} 2022-09-12T20:16:15.928 DEBUG:boto:Token: None 2022-09-12T20:16:15.929 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:15.930 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6Y000imVF9hUBvRWDgmq8nU5+fc= 2022-09-12T20:16:15.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6Y000imVF9hUBvRWDgmq8nU5+fc=', 'Content-Length': '0'} 2022-09-12T20:16:15.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000931a1c2961f7c2ed-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.933 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:15.934 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:15.935 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:16:15.935 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:16:15.936 DEBUG:boto:Method: GET 2022-09-12T20:16:15.936 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:16:15.937 DEBUG:boto:Data: 2022-09-12T20:16:15.937 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.938 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.938 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.939 DEBUG:boto:Params: {} 2022-09-12T20:16:15.940 DEBUG:boto:Token: None 2022-09-12T20:16:15.940 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/ 2022-09-12T20:16:15.941 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs= 2022-09-12T20:16:15.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs=', 'Content-Length': '0'} 2022-09-12T20:16:15.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e70e0e41fb5fa437-00631f938f-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.945 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:15.946 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:15.947 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:15.947 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:16:15.948 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:16:15.948 DEBUG:boto:Method: GET 2022-09-12T20:16:15.949 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:16:15.949 DEBUG:boto:Data: 2022-09-12T20:16:15.950 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.951 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.951 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.952 DEBUG:boto:Params: {} 2022-09-12T20:16:15.952 DEBUG:boto:Token: None 2022-09-12T20:16:15.953 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/ 2022-09-12T20:16:15.953 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs= 2022-09-12T20:16:15.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cLU5M+ZHvxqXW6U6ApT3/GC/RYs=', 'Content-Length': '0'} 2022-09-12T20:16:15.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea9b3b0dc01abd2e-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.959 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:15.960 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:15.961 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:15.961 DEBUG:boto:Method: HEAD 2022-09-12T20:16:15.962 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:16:15.962 DEBUG:boto:Data: 2022-09-12T20:16:15.963 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.963 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.964 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.964 DEBUG:boto:Params: {} 2022-09-12T20:16:15.965 DEBUG:boto:Token: None 2022-09-12T20:16:15.965 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj95 2022-09-12T20:16:15.966 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Brz5ZUVK1s+G8kw43vrL2oqWZ7w= 2022-09-12T20:16:15.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Brz5ZUVK1s+G8kw43vrL2oqWZ7w=', 'Content-Length': '0'} 2022-09-12T20:16:15.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000220990a6bbc9a9d6-00631f938f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.970 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:15.971 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:15.971 DEBUG:boto:Method: HEAD 2022-09-12T20:16:15.972 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:16:15.972 DEBUG:boto:Data: 2022-09-12T20:16:15.973 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.973 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.974 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.974 DEBUG:boto:Params: {} 2022-09-12T20:16:15.975 DEBUG:boto:Token: None 2022-09-12T20:16:15.975 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj95 2022-09-12T20:16:15.976 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Brz5ZUVK1s+G8kw43vrL2oqWZ7w= 2022-09-12T20:16:15.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Brz5ZUVK1s+G8kw43vrL2oqWZ7w=', 'Content-Length': '0'} 2022-09-12T20:16:15.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa91d7446f530373-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:16:15.980 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:15.980 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:15.981 DEBUG:boto:Method: GET 2022-09-12T20:16:15.982 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:16:15.982 DEBUG:boto:Data: 2022-09-12T20:16:15.983 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.983 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:15.984 DEBUG:boto:Port: 8001 2022-09-12T20:16:15.984 DEBUG:boto:Params: {} 2022-09-12T20:16:15.985 DEBUG:boto:Token: None 2022-09-12T20:16:15.986 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj95 2022-09-12T20:16:15.986 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pB/n9duteB4qNOe0q0jdpsBekwM= 2022-09-12T20:16:15.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pB/n9duteB4qNOe0q0jdpsBekwM=', 'Content-Length': '0'} 2022-09-12T20:16:15.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bd8b33430586b5c-00631f938f-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:15.990 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:15.991 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:15.991 DEBUG:boto:Method: GET 2022-09-12T20:16:15.992 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:16:15.993 DEBUG:boto:Data: 2022-09-12T20:16:15.994 DEBUG:boto:Headers: {} 2022-09-12T20:16:15.994 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:15.995 DEBUG:boto:Port: 8000 2022-09-12T20:16:15.995 DEBUG:boto:Params: {} 2022-09-12T20:16:15.996 DEBUG:boto:Token: None 2022-09-12T20:16:15.996 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:15 GMT /jdpiak-75/obj95 2022-09-12T20:16:15.997 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pB/n9duteB4qNOe0q0jdpsBekwM= 2022-09-12T20:16:15.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:15 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pB/n9duteB4qNOe0q0jdpsBekwM=', 'Content-Length': '0'} 2022-09-12T20:16:16.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dee917cb36917876-00631f938f-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.042 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:16.043 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:16.043 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:16:16.044 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:16:16.045 DEBUG:boto:Method: GET 2022-09-12T20:16:16.045 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:16:16.046 DEBUG:boto:Data: 2022-09-12T20:16:16.046 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.047 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.047 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.048 DEBUG:boto:Params: {} 2022-09-12T20:16:16.048 DEBUG:boto:Token: None 2022-09-12T20:16:16.049 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/ 2022-09-12T20:16:16.050 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE= 2022-09-12T20:16:16.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE=', 'Content-Length': '0'} 2022-09-12T20:16:16.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b24d73d72cad453a-00631f9390-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.054 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:16.055 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:16.056 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:16.056 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:16:16.057 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:16:16.057 DEBUG:boto:Method: GET 2022-09-12T20:16:16.058 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:16:16.058 DEBUG:boto:Data: 2022-09-12T20:16:16.059 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.060 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:16:16.060 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.061 DEBUG:boto:Params: {} 2022-09-12T20:16:16.062 DEBUG:boto:Token: None 2022-09-12T20:16:16.063 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/ 2022-09-12T20:16:16.063 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE= 2022-09-12T20:16:16.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE=', 'Content-Length': '0'} 2022-09-12T20:16:16.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce8a653385cdffbb-00631f9390-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.070 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:16.071 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-75 zones={test-zone3, test-zone1} 2022-09-12T20:16:16.072 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-75 zones={test-zone3, test-zone2} 2022-09-12T20:16:16.072 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:16.073 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:16.073 DEBUG:boto:Method: HEAD 2022-09-12T20:16:16.074 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:16:16.074 DEBUG:boto:Data: 2022-09-12T20:16:16.075 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.075 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.076 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.076 DEBUG:boto:Params: {} 2022-09-12T20:16:16.077 DEBUG:boto:Token: None 2022-09-12T20:16:16.077 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/ 2022-09-12T20:16:16.078 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c17g3sVkQB+dWTU2MG7g0ZJTOsM= 2022-09-12T20:16:16.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c17g3sVkQB+dWTU2MG7g0ZJTOsM=', 'Content-Length': '0'} 2022-09-12T20:16:16.081 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b03d0bf62ee0bde-00631f9390-1826-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.082 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:16.082 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:16.083 DEBUG:boto:Method: HEAD 2022-09-12T20:16:16.083 DEBUG:boto:Path: /jdpiak-75/ 2022-09-12T20:16:16.084 DEBUG:boto:Data: 2022-09-12T20:16:16.084 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.085 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.085 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.086 DEBUG:boto:Params: {} 2022-09-12T20:16:16.087 DEBUG:boto:Token: None 2022-09-12T20:16:16.087 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/ 2022-09-12T20:16:16.088 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c17g3sVkQB+dWTU2MG7g0ZJTOsM= 2022-09-12T20:16:16.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c17g3sVkQB+dWTU2MG7g0ZJTOsM=', 'Content-Length': '0'} 2022-09-12T20:16:16.091 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e623b0184970ed22-00631f9390-189d-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.092 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T20:16:16.092 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:16.093 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:16.094 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:16:16.094 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:16:16.095 DEBUG:boto:Method: GET 2022-09-12T20:16:16.095 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:16:16.096 DEBUG:boto:Data: 2022-09-12T20:16:16.096 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.097 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.097 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.098 DEBUG:boto:Params: {} 2022-09-12T20:16:16.098 DEBUG:boto:Token: None 2022-09-12T20:16:16.099 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/?versions 2022-09-12T20:16:16.099 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GJxiShrYyuFXiOFdtACJv58bnqA= 2022-09-12T20:16:16.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GJxiShrYyuFXiOFdtACJv58bnqA=', 'Content-Length': '0'} 2022-09-12T20:16:16.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093a9789ae7b690cf-00631f9390-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.104 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2022-09-12T20:14:27.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2022-09-12T20:14:27.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2022-09-12T20:14:26.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2022-09-12T20:14:27.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:16.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:16:16.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2022-09-12T20:16:16.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-12T20:16:16.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-12T20:16:16.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2022-09-12T20:16:16.109 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-12T20:16:16.109 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2022-09-12T20:16:16.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2022-09-12T20:16:16.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:16:16.111 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2022-09-12T20:16:16.111 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:16:16.112 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:16:16.112 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-12T20:16:16.113 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-12T20:16:16.113 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2022-09-12T20:16:16.114 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:16:16.114 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:16:16.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2022-09-12T20:16:16.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:16:16.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:16:16.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-12T20:16:16.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:16:16.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:16:16.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:16:16.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-12T20:16:16.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2022-09-12T20:16:16.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:16:16.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-12T20:16:16.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2022-09-12T20:16:16.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:16:16.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:16:16.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-12T20:16:16.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-12T20:16:16.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2022-09-12T20:16:16.123 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:16:16.123 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T20:16:16.124 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:16.124 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:16.125 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:16:16.125 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:16:16.126 DEBUG:boto:Method: GET 2022-09-12T20:16:16.126 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:16:16.127 DEBUG:boto:Data: 2022-09-12T20:16:16.127 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.128 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.128 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.129 DEBUG:boto:Params: {} 2022-09-12T20:16:16.130 DEBUG:boto:Token: None 2022-09-12T20:16:16.130 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/?versions 2022-09-12T20:16:16.131 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GJxiShrYyuFXiOFdtACJv58bnqA= 2022-09-12T20:16:16.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GJxiShrYyuFXiOFdtACJv58bnqA=', 'Content-Length': '0'} 2022-09-12T20:16:16.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5f8962f2eb82e04-00631f9390-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.135 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2022-09-12T20:14:27.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2022-09-12T20:14:27.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2022-09-12T20:14:26.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2022-09-12T20:14:27.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:16.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:16:16.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2022-09-12T20:16:16.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-12T20:16:16.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-12T20:16:16.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2022-09-12T20:16:16.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-12T20:16:16.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2022-09-12T20:16:16.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2022-09-12T20:16:16.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-12T20:16:16.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2022-09-12T20:16:16.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-12T20:16:16.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:16:16.144 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-12T20:16:16.144 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-12T20:16:16.144 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2022-09-12T20:16:16.145 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:16:16.145 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-12T20:16:16.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2022-09-12T20:16:16.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-12T20:16:16.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:16:16.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-12T20:16:16.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-12T20:16:16.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:16:16.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-12T20:16:16.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-12T20:16:16.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2022-09-12T20:16:16.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-12T20:16:16.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-12T20:16:16.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2022-09-12T20:16:16.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-12T20:16:16.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-12T20:16:16.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-12T20:16:16.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-12T20:16:16.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2022-09-12T20:16:16.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-12T20:16:16.154 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:16.155 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:16.156 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:16:16.156 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:16:16.157 DEBUG:boto:Method: GET 2022-09-12T20:16:16.157 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:16:16.158 DEBUG:boto:Data: 2022-09-12T20:16:16.158 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.159 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.160 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.160 DEBUG:boto:Params: {} 2022-09-12T20:16:16.161 DEBUG:boto:Token: None 2022-09-12T20:16:16.161 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/?versions 2022-09-12T20:16:16.162 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GJxiShrYyuFXiOFdtACJv58bnqA= 2022-09-12T20:16:16.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GJxiShrYyuFXiOFdtACJv58bnqA=', 'Content-Length': '0'} 2022-09-12T20:16:16.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2fe569983d8ed5f-00631f9390-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.167 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2022-09-12T20:14:27.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2022-09-12T20:14:27.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2022-09-12T20:14:26.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2022-09-12T20:14:27.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:16.170 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:16.170 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:16.171 DEBUG:boto:path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:16:16.171 DEBUG:boto:auth_path=/jdpiak-75/?versions&max-keys=999 2022-09-12T20:16:16.172 DEBUG:boto:Method: GET 2022-09-12T20:16:16.172 DEBUG:boto:Path: /jdpiak-75/?versions&max-keys=999 2022-09-12T20:16:16.173 DEBUG:boto:Data: 2022-09-12T20:16:16.173 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.174 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.174 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.175 DEBUG:boto:Params: {} 2022-09-12T20:16:16.175 DEBUG:boto:Token: None 2022-09-12T20:16:16.176 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/?versions 2022-09-12T20:16:16.177 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GJxiShrYyuFXiOFdtACJv58bnqA= 2022-09-12T20:16:16.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GJxiShrYyuFXiOFdtACJv58bnqA=', 'Content-Length': '0'} 2022-09-12T20:16:16.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025f670ec15061274-00631f9390-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.181 DEBUG:boto:b'jdpiak-75999falseobj11nulltrue2022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2022-09-12T20:14:27.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2022-09-12T20:14:27.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2022-09-12T20:14:26.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2022-09-12T20:14:27.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:16.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:16:16.183 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:16:16.184 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:16:16.185 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:16:16.186 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:16:16.187 DEBUG:boto:Method: GET 2022-09-12T20:16:16.187 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:16:16.188 DEBUG:boto:Data: 2022-09-12T20:16:16.189 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.190 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.190 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.191 DEBUG:boto:Params: {} 2022-09-12T20:16:16.192 DEBUG:boto:Token: None 2022-09-12T20:16:16.193 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:16:16.194 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L9cpC9P2TuoBor+cUJtvRzEY9zI= 2022-09-12T20:16:16.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L9cpC9P2TuoBor+cUJtvRzEY9zI=', 'Content-Length': '0'} 2022-09-12T20:16:16.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000514cecb45d5e23aa-00631f9390-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.201 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:16:16.202 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:16:16.203 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:16:16.204 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:16:16.205 DEBUG:boto:Method: GET 2022-09-12T20:16:16.206 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:16:16.207 DEBUG:boto:Data: 2022-09-12T20:16:16.207 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.208 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.209 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.210 DEBUG:boto:Params: {} 2022-09-12T20:16:16.211 DEBUG:boto:Token: None 2022-09-12T20:16:16.211 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:16:16.212 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L9cpC9P2TuoBor+cUJtvRzEY9zI= 2022-09-12T20:16:16.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L9cpC9P2TuoBor+cUJtvRzEY9zI=', 'Content-Length': '0'} 2022-09-12T20:16:16.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000242fc195e5de1e20-00631f9390-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.217 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:16:16.217 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:16:16.218 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:16:16.219 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:16:16.220 DEBUG:boto:Method: HEAD 2022-09-12T20:16:16.220 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:16:16.221 DEBUG:boto:Data: 2022-09-12T20:16:16.221 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.222 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.223 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.223 DEBUG:boto:Params: {} 2022-09-12T20:16:16.224 DEBUG:boto:Token: None 2022-09-12T20:16:16.225 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:16:16.226 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RaKR2xpHgtxyoKuUVyjGtXeIMnE= 2022-09-12T20:16:16.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RaKR2xpHgtxyoKuUVyjGtXeIMnE=', 'Content-Length': '0'} 2022-09-12T20:16:16.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7ac3371f61da2aa-00631f9390-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.230 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:16:16.231 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:16:16.231 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:16:16.232 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:16:16.233 DEBUG:boto:Method: HEAD 2022-09-12T20:16:16.233 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:16:16.234 DEBUG:boto:Data: 2022-09-12T20:16:16.234 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.235 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.235 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.236 DEBUG:boto:Params: {} 2022-09-12T20:16:16.237 DEBUG:boto:Token: None 2022-09-12T20:16:16.238 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:16:16.239 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RaKR2xpHgtxyoKuUVyjGtXeIMnE= 2022-09-12T20:16:16.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RaKR2xpHgtxyoKuUVyjGtXeIMnE=', 'Content-Length': '0'} 2022-09-12T20:16:16.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001205a8169aa66e23-00631f9390-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:16:16.244 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:16:16.244 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:16:16.245 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:16:16.246 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:16:16.246 DEBUG:boto:Method: GET 2022-09-12T20:16:16.247 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:16:16.247 DEBUG:boto:Data: 2022-09-12T20:16:16.248 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.249 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.249 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.250 DEBUG:boto:Params: {} 2022-09-12T20:16:16.251 DEBUG:boto:Token: None 2022-09-12T20:16:16.251 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:16:16.252 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L9cpC9P2TuoBor+cUJtvRzEY9zI= 2022-09-12T20:16:16.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L9cpC9P2TuoBor+cUJtvRzEY9zI=', 'Content-Length': '0'} 2022-09-12T20:16:16.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000248f55646a2183b3-00631f9390-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.298 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:16:16.299 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:16:16.300 DEBUG:boto:path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:16:16.300 DEBUG:boto:auth_path=/jdpiak-75/obj11?versionId=null 2022-09-12T20:16:16.301 DEBUG:boto:Method: GET 2022-09-12T20:16:16.301 DEBUG:boto:Path: /jdpiak-75/obj11?versionId=null 2022-09-12T20:16:16.302 DEBUG:boto:Data: 2022-09-12T20:16:16.303 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.303 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.304 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.305 DEBUG:boto:Params: {} 2022-09-12T20:16:16.305 DEBUG:boto:Token: None 2022-09-12T20:16:16.306 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj11?versionId=null 2022-09-12T20:16:16.307 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L9cpC9P2TuoBor+cUJtvRzEY9zI= 2022-09-12T20:16:16.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L9cpC9P2TuoBor+cUJtvRzEY9zI=', 'Content-Length': '0'} 2022-09-12T20:16:16.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b2873cd8cf3ff68-00631f9390-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.311 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:16.312 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:16.313 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:16:16.314 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:16:16.314 DEBUG:boto:Method: GET 2022-09-12T20:16:16.315 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:16:16.316 DEBUG:boto:Data: 2022-09-12T20:16:16.316 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.317 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.318 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.318 DEBUG:boto:Params: {} 2022-09-12T20:16:16.319 DEBUG:boto:Token: None 2022-09-12T20:16:16.320 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/ 2022-09-12T20:16:16.320 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE= 2022-09-12T20:16:16.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE=', 'Content-Length': '0'} 2022-09-12T20:16:16.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a373b92177325d08-00631f9390-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.325 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:16.326 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:16.327 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:16.328 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:16:16.329 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:16:16.329 DEBUG:boto:Method: GET 2022-09-12T20:16:16.330 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:16:16.331 DEBUG:boto:Data: 2022-09-12T20:16:16.331 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.332 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.333 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.333 DEBUG:boto:Params: {} 2022-09-12T20:16:16.334 DEBUG:boto:Token: None 2022-09-12T20:16:16.335 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/ 2022-09-12T20:16:16.335 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE= 2022-09-12T20:16:16.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE=', 'Content-Length': '0'} 2022-09-12T20:16:16.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2b109215edbdeba-00631f9390-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.339 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:16.340 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:16:16.341 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:16:16.341 DEBUG:boto:Method: HEAD 2022-09-12T20:16:16.342 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:16:16.343 DEBUG:boto:Data: 2022-09-12T20:16:16.343 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.344 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.344 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.345 DEBUG:boto:Params: {} 2022-09-12T20:16:16.345 DEBUG:boto:Token: None 2022-09-12T20:16:16.346 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj11 2022-09-12T20:16:16.346 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zIzkh4fcdXdG/nQEO0xz68jA2nE= 2022-09-12T20:16:16.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zIzkh4fcdXdG/nQEO0xz68jA2nE=', 'Content-Length': '0'} 2022-09-12T20:16:16.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9773175fac66920-00631f9390-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.350 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:16:16.350 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:16:16.351 DEBUG:boto:Method: HEAD 2022-09-12T20:16:16.351 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:16:16.352 DEBUG:boto:Data: 2022-09-12T20:16:16.352 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.353 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.353 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.354 DEBUG:boto:Params: {} 2022-09-12T20:16:16.354 DEBUG:boto:Token: None 2022-09-12T20:16:16.355 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj11 2022-09-12T20:16:16.355 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zIzkh4fcdXdG/nQEO0xz68jA2nE= 2022-09-12T20:16:16.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zIzkh4fcdXdG/nQEO0xz68jA2nE=', 'Content-Length': '0'} 2022-09-12T20:16:16.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad92bd4f434ce510-00631f9390-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:16:16.359 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:16:16.360 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:16:16.360 DEBUG:boto:Method: GET 2022-09-12T20:16:16.361 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:16:16.361 DEBUG:boto:Data: 2022-09-12T20:16:16.362 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.362 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.363 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.363 DEBUG:boto:Params: {} 2022-09-12T20:16:16.364 DEBUG:boto:Token: None 2022-09-12T20:16:16.364 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj11 2022-09-12T20:16:16.365 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IdHW4spQeN7xezRvFV+7XyQRLeA= 2022-09-12T20:16:16.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IdHW4spQeN7xezRvFV+7XyQRLeA=', 'Content-Length': '0'} 2022-09-12T20:16:16.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ecc146d7573aa0b-00631f9390-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.369 DEBUG:boto:path=/jdpiak-75/obj11 2022-09-12T20:16:16.369 DEBUG:boto:auth_path=/jdpiak-75/obj11 2022-09-12T20:16:16.370 DEBUG:boto:Method: GET 2022-09-12T20:16:16.370 DEBUG:boto:Path: /jdpiak-75/obj11 2022-09-12T20:16:16.371 DEBUG:boto:Data: 2022-09-12T20:16:16.371 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.372 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.372 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.373 DEBUG:boto:Params: {} 2022-09-12T20:16:16.374 DEBUG:boto:Token: None 2022-09-12T20:16:16.374 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj11 2022-09-12T20:16:16.375 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IdHW4spQeN7xezRvFV+7XyQRLeA= 2022-09-12T20:16:16.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IdHW4spQeN7xezRvFV+7XyQRLeA=', 'Content-Length': '0'} 2022-09-12T20:16:16.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007000e764d6ad4d53-00631f9390-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.418 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:16.419 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:16.420 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:16:16.420 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:16:16.421 DEBUG:boto:Method: GET 2022-09-12T20:16:16.421 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:16:16.422 DEBUG:boto:Data: 2022-09-12T20:16:16.422 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.423 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.423 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.424 DEBUG:boto:Params: {} 2022-09-12T20:16:16.424 DEBUG:boto:Token: None 2022-09-12T20:16:16.425 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/ 2022-09-12T20:16:16.426 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE= 2022-09-12T20:16:16.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE=', 'Content-Length': '0'} 2022-09-12T20:16:16.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ae6dfeeab55ce65-00631f9390-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.430 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:16.431 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:16.432 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:16.433 DEBUG:boto:path=/jdpiak-75/?prefix=obj11 2022-09-12T20:16:16.433 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj11 2022-09-12T20:16:16.434 DEBUG:boto:Method: GET 2022-09-12T20:16:16.434 DEBUG:boto:Path: /jdpiak-75/?prefix=obj11 2022-09-12T20:16:16.435 DEBUG:boto:Data: 2022-09-12T20:16:16.435 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.436 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.437 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.437 DEBUG:boto:Params: {} 2022-09-12T20:16:16.438 DEBUG:boto:Token: None 2022-09-12T20:16:16.438 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/ 2022-09-12T20:16:16.439 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE= 2022-09-12T20:16:16.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE=', 'Content-Length': '0'} 2022-09-12T20:16:16.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd2d5dd46624b2c1-00631f9390-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.443 DEBUG:boto:b'jdpiak-75obj111000falseobj112022-09-12T20:03:02.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:16.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2022-09-12T20:16:16.445 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:16:16.446 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:16:16.446 DEBUG:boto:path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:16:16.447 DEBUG:boto:auth_path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:16:16.447 DEBUG:boto:Method: GET 2022-09-12T20:16:16.448 DEBUG:boto:Path: /jdpiak-75/obj111?versionId=null 2022-09-12T20:16:16.448 DEBUG:boto:Data: 2022-09-12T20:16:16.449 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.449 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.450 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.451 DEBUG:boto:Params: {} 2022-09-12T20:16:16.451 DEBUG:boto:Token: None 2022-09-12T20:16:16.452 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj111?versionId=null 2022-09-12T20:16:16.453 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eju89cdh6HXAiOnxZW1wFNFKQeM= 2022-09-12T20:16:16.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eju89cdh6HXAiOnxZW1wFNFKQeM=', 'Content-Length': '0'} 2022-09-12T20:16:16.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3c1eb3c11706881-00631f9390-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.457 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:16:16.457 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:16:16.458 DEBUG:boto:path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:16:16.459 DEBUG:boto:auth_path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:16:16.459 DEBUG:boto:Method: GET 2022-09-12T20:16:16.460 DEBUG:boto:Path: /jdpiak-75/obj111?versionId=null 2022-09-12T20:16:16.460 DEBUG:boto:Data: 2022-09-12T20:16:16.461 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.461 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.462 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.463 DEBUG:boto:Params: {} 2022-09-12T20:16:16.463 DEBUG:boto:Token: None 2022-09-12T20:16:16.464 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj111?versionId=null 2022-09-12T20:16:16.465 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eju89cdh6HXAiOnxZW1wFNFKQeM= 2022-09-12T20:16:16.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eju89cdh6HXAiOnxZW1wFNFKQeM=', 'Content-Length': '0'} 2022-09-12T20:16:16.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b9be43fa0ddcd01-00631f9390-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.468 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:16:16.469 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:16:16.469 DEBUG:boto:path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:16:16.470 DEBUG:boto:auth_path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:16:16.471 DEBUG:boto:Method: HEAD 2022-09-12T20:16:16.471 DEBUG:boto:Path: /jdpiak-75/obj111?versionId=null 2022-09-12T20:16:16.472 DEBUG:boto:Data: 2022-09-12T20:16:16.473 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.473 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.474 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.474 DEBUG:boto:Params: {} 2022-09-12T20:16:16.475 DEBUG:boto:Token: None 2022-09-12T20:16:16.475 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj111?versionId=null 2022-09-12T20:16:16.476 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cg0+vDk3gXDAZQsDxEHKJZAp0ic= 2022-09-12T20:16:16.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cg0+vDk3gXDAZQsDxEHKJZAp0ic=', 'Content-Length': '0'} 2022-09-12T20:16:16.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d34c43c31abbc897-00631f9390-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.479 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:16:16.480 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:16:16.480 DEBUG:boto:path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:16:16.481 DEBUG:boto:auth_path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:16:16.481 DEBUG:boto:Method: HEAD 2022-09-12T20:16:16.482 DEBUG:boto:Path: /jdpiak-75/obj111?versionId=null 2022-09-12T20:16:16.482 DEBUG:boto:Data: 2022-09-12T20:16:16.483 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.483 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.484 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.484 DEBUG:boto:Params: {} 2022-09-12T20:16:16.485 DEBUG:boto:Token: None 2022-09-12T20:16:16.486 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj111?versionId=null 2022-09-12T20:16:16.487 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Cg0+vDk3gXDAZQsDxEHKJZAp0ic= 2022-09-12T20:16:16.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Cg0+vDk3gXDAZQsDxEHKJZAp0ic=', 'Content-Length': '0'} 2022-09-12T20:16:16.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000eb7617b97e4b0f1-00631f9390-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2022-09-12T20:16:16.491 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:16:16.492 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:16:16.492 DEBUG:boto:path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:16:16.493 DEBUG:boto:auth_path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:16:16.493 DEBUG:boto:Method: GET 2022-09-12T20:16:16.494 DEBUG:boto:Path: /jdpiak-75/obj111?versionId=null 2022-09-12T20:16:16.494 DEBUG:boto:Data: 2022-09-12T20:16:16.495 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.495 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.496 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.496 DEBUG:boto:Params: {} 2022-09-12T20:16:16.497 DEBUG:boto:Token: None 2022-09-12T20:16:16.498 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj111?versionId=null 2022-09-12T20:16:16.498 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eju89cdh6HXAiOnxZW1wFNFKQeM= 2022-09-12T20:16:16.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eju89cdh6HXAiOnxZW1wFNFKQeM=', 'Content-Length': '0'} 2022-09-12T20:16:16.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000982ef33384080f07-00631f9390-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.502 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:16:16.503 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:16:16.503 DEBUG:boto:path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:16:16.504 DEBUG:boto:auth_path=/jdpiak-75/obj111?versionId=null 2022-09-12T20:16:16.504 DEBUG:boto:Method: GET 2022-09-12T20:16:16.505 DEBUG:boto:Path: /jdpiak-75/obj111?versionId=null 2022-09-12T20:16:16.505 DEBUG:boto:Data: 2022-09-12T20:16:16.506 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.506 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.507 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.507 DEBUG:boto:Params: {} 2022-09-12T20:16:16.508 DEBUG:boto:Token: None 2022-09-12T20:16:16.509 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj111?versionId=null 2022-09-12T20:16:16.509 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eju89cdh6HXAiOnxZW1wFNFKQeM= 2022-09-12T20:16:16.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eju89cdh6HXAiOnxZW1wFNFKQeM=', 'Content-Length': '0'} 2022-09-12T20:16:16.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061a955da61e30b6e-00631f9390-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.513 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:16.513 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:16.514 DEBUG:boto:path=/jdpiak-75/?prefix=obj111 2022-09-12T20:16:16.514 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj111 2022-09-12T20:16:16.515 DEBUG:boto:Method: GET 2022-09-12T20:16:16.515 DEBUG:boto:Path: /jdpiak-75/?prefix=obj111 2022-09-12T20:16:16.516 DEBUG:boto:Data: 2022-09-12T20:16:16.516 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.517 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.518 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.518 DEBUG:boto:Params: {} 2022-09-12T20:16:16.519 DEBUG:boto:Token: None 2022-09-12T20:16:16.519 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/ 2022-09-12T20:16:16.520 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE= 2022-09-12T20:16:16.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE=', 'Content-Length': '0'} 2022-09-12T20:16:16.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000670f4f7c201b1fcc-00631f9390-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.524 DEBUG:boto:b'jdpiak-75obj1111000falseobj1112022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:16.525 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:16.525 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:16.526 DEBUG:boto:path=/jdpiak-75/?prefix=obj111 2022-09-12T20:16:16.527 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj111 2022-09-12T20:16:16.527 DEBUG:boto:Method: GET 2022-09-12T20:16:16.528 DEBUG:boto:Path: /jdpiak-75/?prefix=obj111 2022-09-12T20:16:16.528 DEBUG:boto:Data: 2022-09-12T20:16:16.529 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.529 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.530 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.530 DEBUG:boto:Params: {} 2022-09-12T20:16:16.531 DEBUG:boto:Token: None 2022-09-12T20:16:16.531 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/ 2022-09-12T20:16:16.532 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE= 2022-09-12T20:16:16.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE=', 'Content-Length': '0'} 2022-09-12T20:16:16.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f76dfbbb4d9f86a2-00631f9390-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.536 DEBUG:boto:b'jdpiak-75obj1111000falseobj1112022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:16.537 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:16:16.537 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:16:16.538 DEBUG:boto:Method: HEAD 2022-09-12T20:16:16.539 DEBUG:boto:Path: /jdpiak-75/obj111 2022-09-12T20:16:16.539 DEBUG:boto:Data: 2022-09-12T20:16:16.540 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.540 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.541 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.541 DEBUG:boto:Params: {} 2022-09-12T20:16:16.542 DEBUG:boto:Token: None 2022-09-12T20:16:16.542 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj111 2022-09-12T20:16:16.543 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CRub+yWsyxnjYtvv3Rt2bGd4/Rk= 2022-09-12T20:16:16.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CRub+yWsyxnjYtvv3Rt2bGd4/Rk=', 'Content-Length': '0'} 2022-09-12T20:16:16.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001af9be9c18bb8c0f-00631f9390-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.547 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:16:16.547 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:16:16.548 DEBUG:boto:Method: HEAD 2022-09-12T20:16:16.548 DEBUG:boto:Path: /jdpiak-75/obj111 2022-09-12T20:16:16.549 DEBUG:boto:Data: 2022-09-12T20:16:16.549 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.550 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.550 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.551 DEBUG:boto:Params: {} 2022-09-12T20:16:16.551 DEBUG:boto:Token: None 2022-09-12T20:16:16.552 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj111 2022-09-12T20:16:16.552 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:CRub+yWsyxnjYtvv3Rt2bGd4/Rk= 2022-09-12T20:16:16.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:CRub+yWsyxnjYtvv3Rt2bGd4/Rk=', 'Content-Length': '0'} 2022-09-12T20:16:16.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e3ce0d6a4a8cce1-00631f9390-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2022-09-12T20:16:16.556 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:16:16.557 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:16:16.557 DEBUG:boto:Method: GET 2022-09-12T20:16:16.558 DEBUG:boto:Path: /jdpiak-75/obj111 2022-09-12T20:16:16.558 DEBUG:boto:Data: 2022-09-12T20:16:16.559 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.559 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.560 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.560 DEBUG:boto:Params: {} 2022-09-12T20:16:16.561 DEBUG:boto:Token: None 2022-09-12T20:16:16.561 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj111 2022-09-12T20:16:16.562 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yTz69gOu2mKva1eaDzhm/un6Ul8= 2022-09-12T20:16:16.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yTz69gOu2mKva1eaDzhm/un6Ul8=', 'Content-Length': '0'} 2022-09-12T20:16:16.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b3a44b150002464-00631f9390-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.606 DEBUG:boto:path=/jdpiak-75/obj111 2022-09-12T20:16:16.607 DEBUG:boto:auth_path=/jdpiak-75/obj111 2022-09-12T20:16:16.607 DEBUG:boto:Method: GET 2022-09-12T20:16:16.608 DEBUG:boto:Path: /jdpiak-75/obj111 2022-09-12T20:16:16.608 DEBUG:boto:Data: 2022-09-12T20:16:16.609 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.609 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.610 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.610 DEBUG:boto:Params: {} 2022-09-12T20:16:16.611 DEBUG:boto:Token: None 2022-09-12T20:16:16.612 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj111 2022-09-12T20:16:16.612 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yTz69gOu2mKva1eaDzhm/un6Ul8= 2022-09-12T20:16:16.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yTz69gOu2mKva1eaDzhm/un6Ul8=', 'Content-Length': '0'} 2022-09-12T20:16:16.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003531dc7a643d24c5-00631f9390-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.616 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:16.616 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:16.617 DEBUG:boto:path=/jdpiak-75/?prefix=obj111 2022-09-12T20:16:16.618 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj111 2022-09-12T20:16:16.618 DEBUG:boto:Method: GET 2022-09-12T20:16:16.619 DEBUG:boto:Path: /jdpiak-75/?prefix=obj111 2022-09-12T20:16:16.619 DEBUG:boto:Data: 2022-09-12T20:16:16.620 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.620 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.621 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.621 DEBUG:boto:Params: {} 2022-09-12T20:16:16.622 DEBUG:boto:Token: None 2022-09-12T20:16:16.622 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/ 2022-09-12T20:16:16.623 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE= 2022-09-12T20:16:16.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE=', 'Content-Length': '0'} 2022-09-12T20:16:16.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ecbad7620a6362e-00631f9390-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.627 DEBUG:boto:b'jdpiak-75obj1111000falseobj1112022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:16.628 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:16.628 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:16.629 DEBUG:boto:path=/jdpiak-75/?prefix=obj111 2022-09-12T20:16:16.629 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj111 2022-09-12T20:16:16.630 DEBUG:boto:Method: GET 2022-09-12T20:16:16.630 DEBUG:boto:Path: /jdpiak-75/?prefix=obj111 2022-09-12T20:16:16.631 DEBUG:boto:Data: 2022-09-12T20:16:16.632 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.632 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.633 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.633 DEBUG:boto:Params: {} 2022-09-12T20:16:16.634 DEBUG:boto:Token: None 2022-09-12T20:16:16.634 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/ 2022-09-12T20:16:16.635 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE= 2022-09-12T20:16:16.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE=', 'Content-Length': '0'} 2022-09-12T20:16:16.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ceb28c751d8918a-00631f9390-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.639 DEBUG:boto:b'jdpiak-75obj1111000falseobj1112022-09-12T20:14:27.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:16.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:16:16.640 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:16:16.641 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:16:16.641 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:16:16.642 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:16:16.642 DEBUG:boto:Method: GET 2022-09-12T20:16:16.643 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:16:16.643 DEBUG:boto:Data: 2022-09-12T20:16:16.644 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.644 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.645 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.646 DEBUG:boto:Params: {} 2022-09-12T20:16:16.646 DEBUG:boto:Token: None 2022-09-12T20:16:16.647 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:16:16.647 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YMut7/Ys8kg4SmoAAGNpGiJtgQg= 2022-09-12T20:16:16.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YMut7/Ys8kg4SmoAAGNpGiJtgQg=', 'Content-Length': '0'} 2022-09-12T20:16:16.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c1a6737a274e342-00631f9390-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.651 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:16:16.652 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:16:16.652 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:16:16.653 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:16:16.653 DEBUG:boto:Method: GET 2022-09-12T20:16:16.654 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:16:16.654 DEBUG:boto:Data: 2022-09-12T20:16:16.655 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.655 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.656 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.656 DEBUG:boto:Params: {} 2022-09-12T20:16:16.657 DEBUG:boto:Token: None 2022-09-12T20:16:16.658 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:16:16.658 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YMut7/Ys8kg4SmoAAGNpGiJtgQg= 2022-09-12T20:16:16.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YMut7/Ys8kg4SmoAAGNpGiJtgQg=', 'Content-Length': '0'} 2022-09-12T20:16:16.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000301f5f0df6b6dc0-00631f9390-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.662 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:16:16.662 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:16:16.663 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:16:16.663 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:16:16.664 DEBUG:boto:Method: HEAD 2022-09-12T20:16:16.664 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:16:16.665 DEBUG:boto:Data: 2022-09-12T20:16:16.665 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.666 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.666 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.667 DEBUG:boto:Params: {} 2022-09-12T20:16:16.667 DEBUG:boto:Token: None 2022-09-12T20:16:16.668 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:16:16.668 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V/ARlgiQECjPYQapEzMiO78dq3g= 2022-09-12T20:16:16.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V/ARlgiQECjPYQapEzMiO78dq3g=', 'Content-Length': '0'} 2022-09-12T20:16:16.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acee5ecaf76c0640-00631f9390-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.672 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:16:16.672 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:16:16.673 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:16:16.673 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:16:16.674 DEBUG:boto:Method: HEAD 2022-09-12T20:16:16.674 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:16:16.675 DEBUG:boto:Data: 2022-09-12T20:16:16.675 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.676 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.676 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.677 DEBUG:boto:Params: {} 2022-09-12T20:16:16.677 DEBUG:boto:Token: None 2022-09-12T20:16:16.678 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:16:16.678 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V/ARlgiQECjPYQapEzMiO78dq3g= 2022-09-12T20:16:16.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V/ARlgiQECjPYQapEzMiO78dq3g=', 'Content-Length': '0'} 2022-09-12T20:16:16.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1332164e2e21fbd-00631f9390-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:16:16.682 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:16:16.683 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:16:16.684 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:16:16.684 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:16:16.685 DEBUG:boto:Method: GET 2022-09-12T20:16:16.685 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:16:16.686 DEBUG:boto:Data: 2022-09-12T20:16:16.686 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.687 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.687 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.688 DEBUG:boto:Params: {} 2022-09-12T20:16:16.688 DEBUG:boto:Token: None 2022-09-12T20:16:16.689 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:16:16.690 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YMut7/Ys8kg4SmoAAGNpGiJtgQg= 2022-09-12T20:16:16.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YMut7/Ys8kg4SmoAAGNpGiJtgQg=', 'Content-Length': '0'} 2022-09-12T20:16:16.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5e140aad73800c6-00631f9390-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.734 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:16:16.735 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:16:16.735 DEBUG:boto:path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:16:16.736 DEBUG:boto:auth_path=/jdpiak-75/obj115?versionId=null 2022-09-12T20:16:16.736 DEBUG:boto:Method: GET 2022-09-12T20:16:16.737 DEBUG:boto:Path: /jdpiak-75/obj115?versionId=null 2022-09-12T20:16:16.737 DEBUG:boto:Data: 2022-09-12T20:16:16.738 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.739 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.739 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.740 DEBUG:boto:Params: {} 2022-09-12T20:16:16.740 DEBUG:boto:Token: None 2022-09-12T20:16:16.741 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj115?versionId=null 2022-09-12T20:16:16.742 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YMut7/Ys8kg4SmoAAGNpGiJtgQg= 2022-09-12T20:16:16.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YMut7/Ys8kg4SmoAAGNpGiJtgQg=', 'Content-Length': '0'} 2022-09-12T20:16:16.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdaa4438f424b9ce-00631f9390-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.745 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:16.746 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:16.746 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:16:16.747 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:16:16.747 DEBUG:boto:Method: GET 2022-09-12T20:16:16.748 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:16:16.748 DEBUG:boto:Data: 2022-09-12T20:16:16.749 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.749 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.750 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.750 DEBUG:boto:Params: {} 2022-09-12T20:16:16.751 DEBUG:boto:Token: None 2022-09-12T20:16:16.751 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/ 2022-09-12T20:16:16.752 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE= 2022-09-12T20:16:16.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE=', 'Content-Length': '0'} 2022-09-12T20:16:16.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f089990bf12acee-00631f9390-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.756 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:16.757 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:16.757 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:16.758 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:16:16.758 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:16:16.759 DEBUG:boto:Method: GET 2022-09-12T20:16:16.759 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:16:16.760 DEBUG:boto:Data: 2022-09-12T20:16:16.760 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.761 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.761 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.762 DEBUG:boto:Params: {} 2022-09-12T20:16:16.770 DEBUG:boto:Token: None 2022-09-12T20:16:16.770 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/ 2022-09-12T20:16:16.771 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE= 2022-09-12T20:16:16.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE=', 'Content-Length': '0'} 2022-09-12T20:16:16.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1b5ec45b681442f-00631f9390-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.775 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:16.776 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:16:16.776 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:16:16.777 DEBUG:boto:Method: HEAD 2022-09-12T20:16:16.777 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:16:16.778 DEBUG:boto:Data: 2022-09-12T20:16:16.778 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.779 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.779 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.780 DEBUG:boto:Params: {} 2022-09-12T20:16:16.780 DEBUG:boto:Token: None 2022-09-12T20:16:16.781 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj115 2022-09-12T20:16:16.782 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/fby5RS4+2oe6IicdZ70/xOk8RA= 2022-09-12T20:16:16.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/fby5RS4+2oe6IicdZ70/xOk8RA=', 'Content-Length': '0'} 2022-09-12T20:16:16.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e07a1aa25ef05137-00631f9390-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.785 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:16:16.785 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:16:16.786 DEBUG:boto:Method: HEAD 2022-09-12T20:16:16.786 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:16:16.787 DEBUG:boto:Data: 2022-09-12T20:16:16.787 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.788 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.789 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.789 DEBUG:boto:Params: {} 2022-09-12T20:16:16.790 DEBUG:boto:Token: None 2022-09-12T20:16:16.792 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj115 2022-09-12T20:16:16.792 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/fby5RS4+2oe6IicdZ70/xOk8RA= 2022-09-12T20:16:16.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/fby5RS4+2oe6IicdZ70/xOk8RA=', 'Content-Length': '0'} 2022-09-12T20:16:16.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031b76ef5bcceab67-00631f9390-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-12T20:16:16.796 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:16:16.796 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:16:16.797 DEBUG:boto:Method: GET 2022-09-12T20:16:16.798 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:16:16.798 DEBUG:boto:Data: 2022-09-12T20:16:16.799 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.800 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.800 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.801 DEBUG:boto:Params: {} 2022-09-12T20:16:16.801 DEBUG:boto:Token: None 2022-09-12T20:16:16.802 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj115 2022-09-12T20:16:16.802 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sU0CLlyjaDveIQE5oiv9FFYGcA4= 2022-09-12T20:16:16.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sU0CLlyjaDveIQE5oiv9FFYGcA4=', 'Content-Length': '0'} 2022-09-12T20:16:16.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038aeeff0675a2a75-00631f9390-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.806 DEBUG:boto:path=/jdpiak-75/obj115 2022-09-12T20:16:16.807 DEBUG:boto:auth_path=/jdpiak-75/obj115 2022-09-12T20:16:16.808 DEBUG:boto:Method: GET 2022-09-12T20:16:16.808 DEBUG:boto:Path: /jdpiak-75/obj115 2022-09-12T20:16:16.809 DEBUG:boto:Data: 2022-09-12T20:16:16.809 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.810 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.811 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.811 DEBUG:boto:Params: {} 2022-09-12T20:16:16.812 DEBUG:boto:Token: None 2022-09-12T20:16:16.812 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj115 2022-09-12T20:16:16.813 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sU0CLlyjaDveIQE5oiv9FFYGcA4= 2022-09-12T20:16:16.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sU0CLlyjaDveIQE5oiv9FFYGcA4=', 'Content-Length': '0'} 2022-09-12T20:16:16.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000baaac4a157fe47d8-00631f9390-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.858 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:16.859 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:16.859 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:16:16.860 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:16:16.860 DEBUG:boto:Method: GET 2022-09-12T20:16:16.861 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:16:16.861 DEBUG:boto:Data: 2022-09-12T20:16:16.862 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.862 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.863 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.864 DEBUG:boto:Params: {} 2022-09-12T20:16:16.864 DEBUG:boto:Token: None 2022-09-12T20:16:16.865 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/ 2022-09-12T20:16:16.865 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE= 2022-09-12T20:16:16.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE=', 'Content-Length': '0'} 2022-09-12T20:16:16.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054d741832cc09822-00631f9390-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.870 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:16.870 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:16.871 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:16.871 DEBUG:boto:path=/jdpiak-75/?prefix=obj115 2022-09-12T20:16:16.872 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj115 2022-09-12T20:16:16.872 DEBUG:boto:Method: GET 2022-09-12T20:16:16.873 DEBUG:boto:Path: /jdpiak-75/?prefix=obj115 2022-09-12T20:16:16.874 DEBUG:boto:Data: 2022-09-12T20:16:16.874 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.875 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.875 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.876 DEBUG:boto:Params: {} 2022-09-12T20:16:16.876 DEBUG:boto:Token: None 2022-09-12T20:16:16.877 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/ 2022-09-12T20:16:16.878 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE= 2022-09-12T20:16:16.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE=', 'Content-Length': '0'} 2022-09-12T20:16:16.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fee46b699102223b-00631f9390-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.882 DEBUG:boto:b'jdpiak-75obj1151000falseobj1152022-09-12T20:08:38.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:16.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:16:16.883 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:16:16.884 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:16:16.884 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:16:16.885 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:16:16.885 DEBUG:boto:Method: GET 2022-09-12T20:16:16.886 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:16:16.886 DEBUG:boto:Data: 2022-09-12T20:16:16.887 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.887 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.888 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.888 DEBUG:boto:Params: {} 2022-09-12T20:16:16.889 DEBUG:boto:Token: None 2022-09-12T20:16:16.889 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:16:16.890 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3RaLAXXnuhA0lgFWuaFa61jJWsM= 2022-09-12T20:16:16.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3RaLAXXnuhA0lgFWuaFa61jJWsM=', 'Content-Length': '0'} 2022-09-12T20:16:16.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089c435fbba59a60d-00631f9390-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.894 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:16:16.894 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:16:16.895 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:16:16.895 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:16:16.896 DEBUG:boto:Method: GET 2022-09-12T20:16:16.896 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:16:16.897 DEBUG:boto:Data: 2022-09-12T20:16:16.897 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.898 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.899 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.899 DEBUG:boto:Params: {} 2022-09-12T20:16:16.900 DEBUG:boto:Token: None 2022-09-12T20:16:16.900 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:16:16.901 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3RaLAXXnuhA0lgFWuaFa61jJWsM= 2022-09-12T20:16:16.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3RaLAXXnuhA0lgFWuaFa61jJWsM=', 'Content-Length': '0'} 2022-09-12T20:16:16.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea4819331dc02430-00631f9390-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.904 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:16:16.905 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:16:16.905 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:16:16.906 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:16:16.906 DEBUG:boto:Method: HEAD 2022-09-12T20:16:16.907 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:16:16.908 DEBUG:boto:Data: 2022-09-12T20:16:16.908 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.909 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.909 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.909 DEBUG:boto:Params: {} 2022-09-12T20:16:16.910 DEBUG:boto:Token: None 2022-09-12T20:16:16.911 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:16:16.911 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uEBPaadKAfWmYiSb9x3+53g6e1Q= 2022-09-12T20:16:16.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uEBPaadKAfWmYiSb9x3+53g6e1Q=', 'Content-Length': '0'} 2022-09-12T20:16:16.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f21ad62bb783663-00631f9390-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.914 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:16:16.915 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:16:16.915 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:16:16.916 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:16:16.916 DEBUG:boto:Method: HEAD 2022-09-12T20:16:16.917 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:16:16.917 DEBUG:boto:Data: 2022-09-12T20:16:16.918 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.918 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.919 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.919 DEBUG:boto:Params: {} 2022-09-12T20:16:16.920 DEBUG:boto:Token: None 2022-09-12T20:16:16.921 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:16:16.921 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:uEBPaadKAfWmYiSb9x3+53g6e1Q= 2022-09-12T20:16:16.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:uEBPaadKAfWmYiSb9x3+53g6e1Q=', 'Content-Length': '0'} 2022-09-12T20:16:16.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac3d79aabf0a28a5-00631f9390-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:16:16.925 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:16:16.925 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:16:16.926 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:16:16.927 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:16:16.927 DEBUG:boto:Method: GET 2022-09-12T20:16:16.928 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:16:16.928 DEBUG:boto:Data: 2022-09-12T20:16:16.929 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.929 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.930 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.930 DEBUG:boto:Params: {} 2022-09-12T20:16:16.931 DEBUG:boto:Token: None 2022-09-12T20:16:16.931 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:16:16.932 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3RaLAXXnuhA0lgFWuaFa61jJWsM= 2022-09-12T20:16:16.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3RaLAXXnuhA0lgFWuaFa61jJWsM=', 'Content-Length': '0'} 2022-09-12T20:16:16.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000940bd9435b9cbb69-00631f9390-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.935 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:16:16.936 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:16:16.936 DEBUG:boto:path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:16:16.937 DEBUG:boto:auth_path=/jdpiak-75/obj116?versionId=null 2022-09-12T20:16:16.937 DEBUG:boto:Method: GET 2022-09-12T20:16:16.938 DEBUG:boto:Path: /jdpiak-75/obj116?versionId=null 2022-09-12T20:16:16.938 DEBUG:boto:Data: 2022-09-12T20:16:16.939 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.939 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:16.940 DEBUG:boto:Port: 8000 2022-09-12T20:16:16.940 DEBUG:boto:Params: {} 2022-09-12T20:16:16.941 DEBUG:boto:Token: None 2022-09-12T20:16:16.942 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/obj116?versionId=null 2022-09-12T20:16:16.942 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3RaLAXXnuhA0lgFWuaFa61jJWsM= 2022-09-12T20:16:16.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3RaLAXXnuhA0lgFWuaFa61jJWsM=', 'Content-Length': '0'} 2022-09-12T20:16:16.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4f317dbb37e46fc-00631f9390-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.946 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:16.946 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:16.947 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:16:16.947 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:16:16.948 DEBUG:boto:Method: GET 2022-09-12T20:16:16.948 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:16:16.949 DEBUG:boto:Data: 2022-09-12T20:16:16.949 DEBUG:boto:Headers: {} 2022-09-12T20:16:16.950 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:16.950 DEBUG:boto:Port: 8001 2022-09-12T20:16:16.951 DEBUG:boto:Params: {} 2022-09-12T20:16:16.951 DEBUG:boto:Token: None 2022-09-12T20:16:16.952 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:16 GMT /jdpiak-75/ 2022-09-12T20:16:16.952 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE= 2022-09-12T20:16:16.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V+dKKi+wGYKpzuIx1mbJzzLCxvE=', 'Content-Length': '0'} 2022-09-12T20:16:16.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d8407c378249f1e-00631f9390-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:16.998 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:16.999 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:17.000 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:17.000 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:16:17.001 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:16:17.001 DEBUG:boto:Method: GET 2022-09-12T20:16:17.002 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:16:17.002 DEBUG:boto:Data: 2022-09-12T20:16:17.003 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.003 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.004 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.004 DEBUG:boto:Params: {} 2022-09-12T20:16:17.005 DEBUG:boto:Token: None 2022-09-12T20:16:17.006 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/ 2022-09-12T20:16:17.006 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ= 2022-09-12T20:16:17.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ=', 'Content-Length': '0'} 2022-09-12T20:16:17.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2497e12a4c4ace9-00631f9391-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.011 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:17.013 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:16:17.014 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:16:17.016 DEBUG:boto:Method: HEAD 2022-09-12T20:16:17.017 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:16:17.018 DEBUG:boto:Data: 2022-09-12T20:16:17.019 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.019 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.020 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.020 DEBUG:boto:Params: {} 2022-09-12T20:16:17.021 DEBUG:boto:Token: None 2022-09-12T20:16:17.021 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj116 2022-09-12T20:16:17.022 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pSl2kY6mIa2/uqkpjOO48oqzXGY= 2022-09-12T20:16:17.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pSl2kY6mIa2/uqkpjOO48oqzXGY=', 'Content-Length': '0'} 2022-09-12T20:16:17.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c67ffefce07e4695-00631f9391-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.026 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:16:17.026 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:16:17.027 DEBUG:boto:Method: HEAD 2022-09-12T20:16:17.027 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:16:17.028 DEBUG:boto:Data: 2022-09-12T20:16:17.029 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.029 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.030 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.031 DEBUG:boto:Params: {} 2022-09-12T20:16:17.031 DEBUG:boto:Token: None 2022-09-12T20:16:17.032 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj116 2022-09-12T20:16:17.032 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pSl2kY6mIa2/uqkpjOO48oqzXGY= 2022-09-12T20:16:17.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pSl2kY6mIa2/uqkpjOO48oqzXGY=', 'Content-Length': '0'} 2022-09-12T20:16:17.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b08a62ce5892fd77-00631f9391-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-12T20:16:17.036 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:16:17.037 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:16:17.037 DEBUG:boto:Method: GET 2022-09-12T20:16:17.038 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:16:17.039 DEBUG:boto:Data: 2022-09-12T20:16:17.039 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.040 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.040 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.041 DEBUG:boto:Params: {} 2022-09-12T20:16:17.041 DEBUG:boto:Token: None 2022-09-12T20:16:17.042 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj116 2022-09-12T20:16:17.043 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WhC1MUz/cC4hLceVUCKcHhNqBo0= 2022-09-12T20:16:17.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WhC1MUz/cC4hLceVUCKcHhNqBo0=', 'Content-Length': '0'} 2022-09-12T20:16:17.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005252c7e7a83a9a74-00631f9391-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.047 DEBUG:boto:path=/jdpiak-75/obj116 2022-09-12T20:16:17.047 DEBUG:boto:auth_path=/jdpiak-75/obj116 2022-09-12T20:16:17.048 DEBUG:boto:Method: GET 2022-09-12T20:16:17.048 DEBUG:boto:Path: /jdpiak-75/obj116 2022-09-12T20:16:17.049 DEBUG:boto:Data: 2022-09-12T20:16:17.050 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.050 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.051 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.051 DEBUG:boto:Params: {} 2022-09-12T20:16:17.052 DEBUG:boto:Token: None 2022-09-12T20:16:17.052 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj116 2022-09-12T20:16:17.053 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WhC1MUz/cC4hLceVUCKcHhNqBo0= 2022-09-12T20:16:17.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WhC1MUz/cC4hLceVUCKcHhNqBo0=', 'Content-Length': '0'} 2022-09-12T20:16:17.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3fd488c235a68df-00631f9391-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.057 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:17.057 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:17.058 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:16:17.058 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:16:17.059 DEBUG:boto:Method: GET 2022-09-12T20:16:17.059 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:16:17.060 DEBUG:boto:Data: 2022-09-12T20:16:17.061 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.061 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.062 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.062 DEBUG:boto:Params: {} 2022-09-12T20:16:17.063 DEBUG:boto:Token: None 2022-09-12T20:16:17.064 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/ 2022-09-12T20:16:17.064 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ= 2022-09-12T20:16:17.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ=', 'Content-Length': '0'} 2022-09-12T20:16:17.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da22be6fd581cc9d-00631f9391-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.069 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:17.070 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:17.070 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:17.071 DEBUG:boto:path=/jdpiak-75/?prefix=obj116 2022-09-12T20:16:17.072 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj116 2022-09-12T20:16:17.072 DEBUG:boto:Method: GET 2022-09-12T20:16:17.073 DEBUG:boto:Path: /jdpiak-75/?prefix=obj116 2022-09-12T20:16:17.074 DEBUG:boto:Data: 2022-09-12T20:16:17.074 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.075 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.075 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.076 DEBUG:boto:Params: {} 2022-09-12T20:16:17.076 DEBUG:boto:Token: None 2022-09-12T20:16:17.077 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/ 2022-09-12T20:16:17.078 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ= 2022-09-12T20:16:17.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ=', 'Content-Length': '0'} 2022-09-12T20:16:17.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005a26f2a3fd6e791-00631f9391-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.082 DEBUG:boto:b'jdpiak-75obj1161000falseobj1162022-09-12T20:10:09.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:17.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-12T20:16:17.083 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:16:17.084 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:16:17.084 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:16:17.085 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:16:17.085 DEBUG:boto:Method: GET 2022-09-12T20:16:17.086 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:16:17.087 DEBUG:boto:Data: 2022-09-12T20:16:17.087 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.088 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.088 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.089 DEBUG:boto:Params: {} 2022-09-12T20:16:17.090 DEBUG:boto:Token: None 2022-09-12T20:16:17.090 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:16:17.091 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YvRlLNkKvoUXZDgW9qR9g6i16k0= 2022-09-12T20:16:17.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YvRlLNkKvoUXZDgW9qR9g6i16k0=', 'Content-Length': '0'} 2022-09-12T20:16:17.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004073d7a868d89744-00631f9391-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.095 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:16:17.095 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:16:17.096 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:16:17.096 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:16:17.097 DEBUG:boto:Method: GET 2022-09-12T20:16:17.098 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:16:17.098 DEBUG:boto:Data: 2022-09-12T20:16:17.099 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.099 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.100 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.100 DEBUG:boto:Params: {} 2022-09-12T20:16:17.101 DEBUG:boto:Token: None 2022-09-12T20:16:17.102 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:16:17.102 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YvRlLNkKvoUXZDgW9qR9g6i16k0= 2022-09-12T20:16:17.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YvRlLNkKvoUXZDgW9qR9g6i16k0=', 'Content-Length': '0'} 2022-09-12T20:16:17.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000242b4bb71c43304b-00631f9391-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.106 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:16:17.107 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:16:17.107 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:16:17.108 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:16:17.108 DEBUG:boto:Method: HEAD 2022-09-12T20:16:17.109 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:16:17.109 DEBUG:boto:Data: 2022-09-12T20:16:17.110 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.110 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.111 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.111 DEBUG:boto:Params: {} 2022-09-12T20:16:17.112 DEBUG:boto:Token: None 2022-09-12T20:16:17.113 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:16:17.113 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EYeMfBlPm7rO7r1SNDoEdM8JX4Q= 2022-09-12T20:16:17.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EYeMfBlPm7rO7r1SNDoEdM8JX4Q=', 'Content-Length': '0'} 2022-09-12T20:16:17.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c804ffeb606ada3b-00631f9391-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.117 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:16:17.117 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:16:17.118 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:16:17.118 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:16:17.119 DEBUG:boto:Method: HEAD 2022-09-12T20:16:17.119 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:16:17.120 DEBUG:boto:Data: 2022-09-12T20:16:17.120 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.121 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.121 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.122 DEBUG:boto:Params: {} 2022-09-12T20:16:17.122 DEBUG:boto:Token: None 2022-09-12T20:16:17.123 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:16:17.124 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EYeMfBlPm7rO7r1SNDoEdM8JX4Q= 2022-09-12T20:16:17.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EYeMfBlPm7rO7r1SNDoEdM8JX4Q=', 'Content-Length': '0'} 2022-09-12T20:16:17.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092a1d792b41aa9cb-00631f9391-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-12T20:16:17.127 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:16:17.128 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:16:17.129 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:16:17.129 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:16:17.130 DEBUG:boto:Method: GET 2022-09-12T20:16:17.131 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:16:17.131 DEBUG:boto:Data: 2022-09-12T20:16:17.132 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.132 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.133 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.133 DEBUG:boto:Params: {} 2022-09-12T20:16:17.134 DEBUG:boto:Token: None 2022-09-12T20:16:17.135 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:16:17.136 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YvRlLNkKvoUXZDgW9qR9g6i16k0= 2022-09-12T20:16:17.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YvRlLNkKvoUXZDgW9qR9g6i16k0=', 'Content-Length': '0'} 2022-09-12T20:16:17.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d54c2db7236d472c-00631f9391-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.182 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:16:17.183 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:16:17.183 DEBUG:boto:path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:16:17.184 DEBUG:boto:auth_path=/jdpiak-75/obj124?versionId=null 2022-09-12T20:16:17.184 DEBUG:boto:Method: GET 2022-09-12T20:16:17.185 DEBUG:boto:Path: /jdpiak-75/obj124?versionId=null 2022-09-12T20:16:17.185 DEBUG:boto:Data: 2022-09-12T20:16:17.186 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.186 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.187 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.188 DEBUG:boto:Params: {} 2022-09-12T20:16:17.188 DEBUG:boto:Token: None 2022-09-12T20:16:17.189 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj124?versionId=null 2022-09-12T20:16:17.189 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YvRlLNkKvoUXZDgW9qR9g6i16k0= 2022-09-12T20:16:17.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YvRlLNkKvoUXZDgW9qR9g6i16k0=', 'Content-Length': '0'} 2022-09-12T20:16:17.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064bea38eaccee245-00631f9391-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.193 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:17.194 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:17.194 DEBUG:boto:path=/jdpiak-75/?prefix=obj124 2022-09-12T20:16:17.195 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj124 2022-09-12T20:16:17.195 DEBUG:boto:Method: GET 2022-09-12T20:16:17.196 DEBUG:boto:Path: /jdpiak-75/?prefix=obj124 2022-09-12T20:16:17.197 DEBUG:boto:Data: 2022-09-12T20:16:17.197 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.198 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.198 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.199 DEBUG:boto:Params: {} 2022-09-12T20:16:17.199 DEBUG:boto:Token: None 2022-09-12T20:16:17.200 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/ 2022-09-12T20:16:17.200 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ= 2022-09-12T20:16:17.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ=', 'Content-Length': '0'} 2022-09-12T20:16:17.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fccb27ca0819b3d3-00631f9391-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.205 DEBUG:boto:b'jdpiak-75obj1241000falseobj1242022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:17.205 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:17.206 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:17.207 DEBUG:boto:path=/jdpiak-75/?prefix=obj124 2022-09-12T20:16:17.207 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj124 2022-09-12T20:16:17.208 DEBUG:boto:Method: GET 2022-09-12T20:16:17.208 DEBUG:boto:Path: /jdpiak-75/?prefix=obj124 2022-09-12T20:16:17.209 DEBUG:boto:Data: 2022-09-12T20:16:17.209 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.210 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.210 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.211 DEBUG:boto:Params: {} 2022-09-12T20:16:17.211 DEBUG:boto:Token: None 2022-09-12T20:16:17.212 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/ 2022-09-12T20:16:17.213 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ= 2022-09-12T20:16:17.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ=', 'Content-Length': '0'} 2022-09-12T20:16:17.216 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fec4b659ce309b7a-00631f9391-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.216 DEBUG:boto:b'jdpiak-75obj1241000falseobj1242022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:17.217 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:16:17.218 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:16:17.218 DEBUG:boto:Method: HEAD 2022-09-12T20:16:17.219 DEBUG:boto:Path: /jdpiak-75/obj124 2022-09-12T20:16:17.219 DEBUG:boto:Data: 2022-09-12T20:16:17.220 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.220 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.221 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.221 DEBUG:boto:Params: {} 2022-09-12T20:16:17.222 DEBUG:boto:Token: None 2022-09-12T20:16:17.222 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj124 2022-09-12T20:16:17.223 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G6CxVIq/S9GLEP+uGdsFMXJSi9M= 2022-09-12T20:16:17.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G6CxVIq/S9GLEP+uGdsFMXJSi9M=', 'Content-Length': '0'} 2022-09-12T20:16:17.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041c8b73339c557d0-00631f9391-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.226 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:16:17.227 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:16:17.227 DEBUG:boto:Method: HEAD 2022-09-12T20:16:17.228 DEBUG:boto:Path: /jdpiak-75/obj124 2022-09-12T20:16:17.228 DEBUG:boto:Data: 2022-09-12T20:16:17.229 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.229 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.230 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.230 DEBUG:boto:Params: {} 2022-09-12T20:16:17.231 DEBUG:boto:Token: None 2022-09-12T20:16:17.232 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj124 2022-09-12T20:16:17.232 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G6CxVIq/S9GLEP+uGdsFMXJSi9M= 2022-09-12T20:16:17.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G6CxVIq/S9GLEP+uGdsFMXJSi9M=', 'Content-Length': '0'} 2022-09-12T20:16:17.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013dfcbd5c3ffc3e5-00631f9391-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-12T20:16:17.238 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:16:17.239 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:16:17.239 DEBUG:boto:Method: GET 2022-09-12T20:16:17.240 DEBUG:boto:Path: /jdpiak-75/obj124 2022-09-12T20:16:17.241 DEBUG:boto:Data: 2022-09-12T20:16:17.241 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.242 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.242 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.243 DEBUG:boto:Params: {} 2022-09-12T20:16:17.244 DEBUG:boto:Token: None 2022-09-12T20:16:17.244 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj124 2022-09-12T20:16:17.245 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5l8sT7Z6RRaOSHaiNsWbLVzq+5U= 2022-09-12T20:16:17.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5l8sT7Z6RRaOSHaiNsWbLVzq+5U=', 'Content-Length': '0'} 2022-09-12T20:16:17.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085fa6bc7df8c9322-00631f9391-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.249 DEBUG:boto:path=/jdpiak-75/obj124 2022-09-12T20:16:17.250 DEBUG:boto:auth_path=/jdpiak-75/obj124 2022-09-12T20:16:17.250 DEBUG:boto:Method: GET 2022-09-12T20:16:17.251 DEBUG:boto:Path: /jdpiak-75/obj124 2022-09-12T20:16:17.251 DEBUG:boto:Data: 2022-09-12T20:16:17.252 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.252 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.253 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.253 DEBUG:boto:Params: {} 2022-09-12T20:16:17.254 DEBUG:boto:Token: None 2022-09-12T20:16:17.255 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj124 2022-09-12T20:16:17.255 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5l8sT7Z6RRaOSHaiNsWbLVzq+5U= 2022-09-12T20:16:17.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5l8sT7Z6RRaOSHaiNsWbLVzq+5U=', 'Content-Length': '0'} 2022-09-12T20:16:17.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084cfb1a3abf25788-00631f9391-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.259 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:17.259 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:17.260 DEBUG:boto:path=/jdpiak-75/?prefix=obj124 2022-09-12T20:16:17.260 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj124 2022-09-12T20:16:17.261 DEBUG:boto:Method: GET 2022-09-12T20:16:17.261 DEBUG:boto:Path: /jdpiak-75/?prefix=obj124 2022-09-12T20:16:17.262 DEBUG:boto:Data: 2022-09-12T20:16:17.263 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.263 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.264 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.264 DEBUG:boto:Params: {} 2022-09-12T20:16:17.265 DEBUG:boto:Token: None 2022-09-12T20:16:17.265 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/ 2022-09-12T20:16:17.266 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ= 2022-09-12T20:16:17.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ=', 'Content-Length': '0'} 2022-09-12T20:16:17.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5330aa9df398eba-00631f9391-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.270 DEBUG:boto:b'jdpiak-75obj1241000falseobj1242022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:17.271 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:17.271 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:17.272 DEBUG:boto:path=/jdpiak-75/?prefix=obj124 2022-09-12T20:16:17.272 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj124 2022-09-12T20:16:17.273 DEBUG:boto:Method: GET 2022-09-12T20:16:17.273 DEBUG:boto:Path: /jdpiak-75/?prefix=obj124 2022-09-12T20:16:17.274 DEBUG:boto:Data: 2022-09-12T20:16:17.275 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.275 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.276 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.276 DEBUG:boto:Params: {} 2022-09-12T20:16:17.277 DEBUG:boto:Token: None 2022-09-12T20:16:17.278 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/ 2022-09-12T20:16:17.278 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ= 2022-09-12T20:16:17.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ=', 'Content-Length': '0'} 2022-09-12T20:16:17.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013b16426f7873e1f-00631f9391-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.322 DEBUG:boto:b'jdpiak-75obj1241000falseobj1242022-09-12T20:12:24.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:17.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:16:17.324 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:16:17.325 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:16:17.325 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:16:17.326 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:16:17.326 DEBUG:boto:Method: GET 2022-09-12T20:16:17.327 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:16:17.328 DEBUG:boto:Data: 2022-09-12T20:16:17.328 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.329 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.329 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.330 DEBUG:boto:Params: {} 2022-09-12T20:16:17.330 DEBUG:boto:Token: None 2022-09-12T20:16:17.331 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:16:17.332 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:a/ZFkWLFzM0B/Z/dJondfr4KPk4= 2022-09-12T20:16:17.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:a/ZFkWLFzM0B/Z/dJondfr4KPk4=', 'Content-Length': '0'} 2022-09-12T20:16:17.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000537aed3808f0aa3-00631f9391-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.335 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:16:17.336 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:16:17.337 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:16:17.337 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:16:17.338 DEBUG:boto:Method: GET 2022-09-12T20:16:17.339 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:16:17.340 DEBUG:boto:Data: 2022-09-12T20:16:17.340 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.341 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.341 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.342 DEBUG:boto:Params: {} 2022-09-12T20:16:17.343 DEBUG:boto:Token: None 2022-09-12T20:16:17.343 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:16:17.344 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:a/ZFkWLFzM0B/Z/dJondfr4KPk4= 2022-09-12T20:16:17.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:a/ZFkWLFzM0B/Z/dJondfr4KPk4=', 'Content-Length': '0'} 2022-09-12T20:16:17.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008eb35ee6896bc00b-00631f9391-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.348 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:16:17.348 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:16:17.349 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:16:17.349 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:16:17.350 DEBUG:boto:Method: HEAD 2022-09-12T20:16:17.350 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:16:17.351 DEBUG:boto:Data: 2022-09-12T20:16:17.352 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.352 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.353 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.353 DEBUG:boto:Params: {} 2022-09-12T20:16:17.354 DEBUG:boto:Token: None 2022-09-12T20:16:17.354 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:16:17.355 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gm2t8DfeK4r4NRYUZ9bE3mVIAtY= 2022-09-12T20:16:17.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gm2t8DfeK4r4NRYUZ9bE3mVIAtY=', 'Content-Length': '0'} 2022-09-12T20:16:17.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a14de4befc948edb-00631f9391-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.358 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:16:17.359 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:16:17.359 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:16:17.360 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:16:17.360 DEBUG:boto:Method: HEAD 2022-09-12T20:16:17.361 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:16:17.361 DEBUG:boto:Data: 2022-09-12T20:16:17.362 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.362 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.363 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.363 DEBUG:boto:Params: {} 2022-09-12T20:16:17.364 DEBUG:boto:Token: None 2022-09-12T20:16:17.365 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:16:17.365 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gm2t8DfeK4r4NRYUZ9bE3mVIAtY= 2022-09-12T20:16:17.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gm2t8DfeK4r4NRYUZ9bE3mVIAtY=', 'Content-Length': '0'} 2022-09-12T20:16:17.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081724751658960bb-00631f9391-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:16:17.370 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:16:17.370 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:16:17.371 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:16:17.371 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:16:17.372 DEBUG:boto:Method: GET 2022-09-12T20:16:17.373 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:16:17.373 DEBUG:boto:Data: 2022-09-12T20:16:17.374 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.374 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.375 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.375 DEBUG:boto:Params: {} 2022-09-12T20:16:17.376 DEBUG:boto:Token: None 2022-09-12T20:16:17.377 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:16:17.377 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:a/ZFkWLFzM0B/Z/dJondfr4KPk4= 2022-09-12T20:16:17.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:a/ZFkWLFzM0B/Z/dJondfr4KPk4=', 'Content-Length': '0'} 2022-09-12T20:16:17.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b63320f3d4a72a9c-00631f9391-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.381 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:16:17.382 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:16:17.382 DEBUG:boto:path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:16:17.383 DEBUG:boto:auth_path=/jdpiak-75/obj145?versionId=null 2022-09-12T20:16:17.383 DEBUG:boto:Method: GET 2022-09-12T20:16:17.384 DEBUG:boto:Path: /jdpiak-75/obj145?versionId=null 2022-09-12T20:16:17.384 DEBUG:boto:Data: 2022-09-12T20:16:17.385 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.385 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.386 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.387 DEBUG:boto:Params: {} 2022-09-12T20:16:17.387 DEBUG:boto:Token: None 2022-09-12T20:16:17.388 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj145?versionId=null 2022-09-12T20:16:17.388 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:a/ZFkWLFzM0B/Z/dJondfr4KPk4= 2022-09-12T20:16:17.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:a/ZFkWLFzM0B/Z/dJondfr4KPk4=', 'Content-Length': '0'} 2022-09-12T20:16:17.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075eeb771bee21e2a-00631f9391-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.392 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:17.393 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:17.393 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:16:17.394 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:16:17.394 DEBUG:boto:Method: GET 2022-09-12T20:16:17.395 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:16:17.395 DEBUG:boto:Data: 2022-09-12T20:16:17.396 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.396 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.397 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.397 DEBUG:boto:Params: {} 2022-09-12T20:16:17.398 DEBUG:boto:Token: None 2022-09-12T20:16:17.399 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/ 2022-09-12T20:16:17.399 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ= 2022-09-12T20:16:17.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ=', 'Content-Length': '0'} 2022-09-12T20:16:17.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9fcd06a61be2e5f-00631f9391-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.403 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:17.404 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:17.405 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:17.405 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:16:17.406 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:16:17.406 DEBUG:boto:Method: GET 2022-09-12T20:16:17.407 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:16:17.408 DEBUG:boto:Data: 2022-09-12T20:16:17.408 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.409 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.409 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.410 DEBUG:boto:Params: {} 2022-09-12T20:16:17.411 DEBUG:boto:Token: None 2022-09-12T20:16:17.411 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/ 2022-09-12T20:16:17.412 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ= 2022-09-12T20:16:17.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ=', 'Content-Length': '0'} 2022-09-12T20:16:17.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078d98dfa428bfe1d-00631f9391-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.416 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:17.417 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:16:17.417 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:16:17.418 DEBUG:boto:Method: HEAD 2022-09-12T20:16:17.419 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:16:17.419 DEBUG:boto:Data: 2022-09-12T20:16:17.420 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.420 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.421 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.421 DEBUG:boto:Params: {} 2022-09-12T20:16:17.422 DEBUG:boto:Token: None 2022-09-12T20:16:17.422 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj145 2022-09-12T20:16:17.423 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2nv0whGEk+DPsd+P20gBK7H/1FI= 2022-09-12T20:16:17.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2nv0whGEk+DPsd+P20gBK7H/1FI=', 'Content-Length': '0'} 2022-09-12T20:16:17.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099db79e2c8d2cdfa-00631f9391-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.427 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:16:17.427 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:16:17.428 DEBUG:boto:Method: HEAD 2022-09-12T20:16:17.429 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:16:17.429 DEBUG:boto:Data: 2022-09-12T20:16:17.430 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.430 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.431 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.431 DEBUG:boto:Params: {} 2022-09-12T20:16:17.432 DEBUG:boto:Token: None 2022-09-12T20:16:17.432 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj145 2022-09-12T20:16:17.433 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2nv0whGEk+DPsd+P20gBK7H/1FI= 2022-09-12T20:16:17.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2nv0whGEk+DPsd+P20gBK7H/1FI=', 'Content-Length': '0'} 2022-09-12T20:16:17.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da98b440d464ce80-00631f9391-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-12T20:16:17.437 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:16:17.438 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:16:17.438 DEBUG:boto:Method: GET 2022-09-12T20:16:17.439 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:16:17.440 DEBUG:boto:Data: 2022-09-12T20:16:17.440 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.441 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.441 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.442 DEBUG:boto:Params: {} 2022-09-12T20:16:17.443 DEBUG:boto:Token: None 2022-09-12T20:16:17.443 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj145 2022-09-12T20:16:17.444 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XYrEh12C7g430chVrNpALi+y0HM= 2022-09-12T20:16:17.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XYrEh12C7g430chVrNpALi+y0HM=', 'Content-Length': '0'} 2022-09-12T20:16:17.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039bf26e48228ea14-00631f9391-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.490 DEBUG:boto:path=/jdpiak-75/obj145 2022-09-12T20:16:17.491 DEBUG:boto:auth_path=/jdpiak-75/obj145 2022-09-12T20:16:17.491 DEBUG:boto:Method: GET 2022-09-12T20:16:17.492 DEBUG:boto:Path: /jdpiak-75/obj145 2022-09-12T20:16:17.493 DEBUG:boto:Data: 2022-09-12T20:16:17.493 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.494 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.494 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.495 DEBUG:boto:Params: {} 2022-09-12T20:16:17.496 DEBUG:boto:Token: None 2022-09-12T20:16:17.496 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj145 2022-09-12T20:16:17.497 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XYrEh12C7g430chVrNpALi+y0HM= 2022-09-12T20:16:17.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XYrEh12C7g430chVrNpALi+y0HM=', 'Content-Length': '0'} 2022-09-12T20:16:17.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061364b9862624de4-00631f9391-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.500 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:17.501 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:17.502 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:16:17.502 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:16:17.503 DEBUG:boto:Method: GET 2022-09-12T20:16:17.503 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:16:17.504 DEBUG:boto:Data: 2022-09-12T20:16:17.504 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.505 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.505 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.506 DEBUG:boto:Params: {} 2022-09-12T20:16:17.507 DEBUG:boto:Token: None 2022-09-12T20:16:17.507 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/ 2022-09-12T20:16:17.508 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ= 2022-09-12T20:16:17.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ=', 'Content-Length': '0'} 2022-09-12T20:16:17.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0246e788f1360ca-00631f9391-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.512 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:17.513 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:17.514 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:17.515 DEBUG:boto:path=/jdpiak-75/?prefix=obj145 2022-09-12T20:16:17.515 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj145 2022-09-12T20:16:17.516 DEBUG:boto:Method: GET 2022-09-12T20:16:17.516 DEBUG:boto:Path: /jdpiak-75/?prefix=obj145 2022-09-12T20:16:17.517 DEBUG:boto:Data: 2022-09-12T20:16:17.517 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.518 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.518 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.519 DEBUG:boto:Params: {} 2022-09-12T20:16:17.520 DEBUG:boto:Token: None 2022-09-12T20:16:17.521 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/ 2022-09-12T20:16:17.521 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ= 2022-09-12T20:16:17.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ=', 'Content-Length': '0'} 2022-09-12T20:16:17.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003785285f3027465a-00631f9391-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.525 DEBUG:boto:b'jdpiak-75obj1451000falseobj1452022-09-12T20:10:09.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:17.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2022-09-12T20:16:17.527 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:16:17.527 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:16:17.528 DEBUG:boto:path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:16:17.528 DEBUG:boto:auth_path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:16:17.529 DEBUG:boto:Method: GET 2022-09-12T20:16:17.529 DEBUG:boto:Path: /jdpiak-75/obj148?versionId=null 2022-09-12T20:16:17.530 DEBUG:boto:Data: 2022-09-12T20:16:17.530 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.531 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.531 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.532 DEBUG:boto:Params: {} 2022-09-12T20:16:17.533 DEBUG:boto:Token: None 2022-09-12T20:16:17.533 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj148?versionId=null 2022-09-12T20:16:17.534 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y3++2hZ7/EIiHBEP1P5Z0btE7jE= 2022-09-12T20:16:17.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y3++2hZ7/EIiHBEP1P5Z0btE7jE=', 'Content-Length': '0'} 2022-09-12T20:16:17.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0727ab215e45639-00631f9391-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.538 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:16:17.538 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:16:17.539 DEBUG:boto:path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:16:17.539 DEBUG:boto:auth_path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:16:17.540 DEBUG:boto:Method: GET 2022-09-12T20:16:17.541 DEBUG:boto:Path: /jdpiak-75/obj148?versionId=null 2022-09-12T20:16:17.542 DEBUG:boto:Data: 2022-09-12T20:16:17.542 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.543 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.543 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.544 DEBUG:boto:Params: {} 2022-09-12T20:16:17.544 DEBUG:boto:Token: None 2022-09-12T20:16:17.545 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj148?versionId=null 2022-09-12T20:16:17.546 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y3++2hZ7/EIiHBEP1P5Z0btE7jE= 2022-09-12T20:16:17.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y3++2hZ7/EIiHBEP1P5Z0btE7jE=', 'Content-Length': '0'} 2022-09-12T20:16:17.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5dd2b4c9f1a6787-00631f9391-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.549 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:16:17.550 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:16:17.550 DEBUG:boto:path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:16:17.551 DEBUG:boto:auth_path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:16:17.552 DEBUG:boto:Method: HEAD 2022-09-12T20:16:17.552 DEBUG:boto:Path: /jdpiak-75/obj148?versionId=null 2022-09-12T20:16:17.553 DEBUG:boto:Data: 2022-09-12T20:16:17.553 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.554 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.554 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.555 DEBUG:boto:Params: {} 2022-09-12T20:16:17.555 DEBUG:boto:Token: None 2022-09-12T20:16:17.556 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj148?versionId=null 2022-09-12T20:16:17.556 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8X098+cHU87RYxtF5HE6M5a7BYM= 2022-09-12T20:16:17.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8X098+cHU87RYxtF5HE6M5a7BYM=', 'Content-Length': '0'} 2022-09-12T20:16:17.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7c0270c918a0c7e-00631f9391-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.560 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:16:17.560 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:16:17.561 DEBUG:boto:path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:16:17.561 DEBUG:boto:auth_path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:16:17.562 DEBUG:boto:Method: HEAD 2022-09-12T20:16:17.563 DEBUG:boto:Path: /jdpiak-75/obj148?versionId=null 2022-09-12T20:16:17.563 DEBUG:boto:Data: 2022-09-12T20:16:17.564 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.564 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.565 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.565 DEBUG:boto:Params: {} 2022-09-12T20:16:17.566 DEBUG:boto:Token: None 2022-09-12T20:16:17.566 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj148?versionId=null 2022-09-12T20:16:17.567 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:8X098+cHU87RYxtF5HE6M5a7BYM= 2022-09-12T20:16:17.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:8X098+cHU87RYxtF5HE6M5a7BYM=', 'Content-Length': '0'} 2022-09-12T20:16:17.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000856d0a55ddd1119d-00631f9391-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2022-09-12T20:16:17.571 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:16:17.571 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:16:17.572 DEBUG:boto:path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:16:17.572 DEBUG:boto:auth_path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:16:17.573 DEBUG:boto:Method: GET 2022-09-12T20:16:17.574 DEBUG:boto:Path: /jdpiak-75/obj148?versionId=null 2022-09-12T20:16:17.574 DEBUG:boto:Data: 2022-09-12T20:16:17.575 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.575 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.576 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.577 DEBUG:boto:Params: {} 2022-09-12T20:16:17.577 DEBUG:boto:Token: None 2022-09-12T20:16:17.578 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj148?versionId=null 2022-09-12T20:16:17.578 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y3++2hZ7/EIiHBEP1P5Z0btE7jE= 2022-09-12T20:16:17.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y3++2hZ7/EIiHBEP1P5Z0btE7jE=', 'Content-Length': '0'} 2022-09-12T20:16:17.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a04a709f6d289c90-00631f9391-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.582 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:16:17.582 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:16:17.583 DEBUG:boto:path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:16:17.583 DEBUG:boto:auth_path=/jdpiak-75/obj148?versionId=null 2022-09-12T20:16:17.584 DEBUG:boto:Method: GET 2022-09-12T20:16:17.585 DEBUG:boto:Path: /jdpiak-75/obj148?versionId=null 2022-09-12T20:16:17.585 DEBUG:boto:Data: 2022-09-12T20:16:17.586 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.586 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.587 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.587 DEBUG:boto:Params: {} 2022-09-12T20:16:17.588 DEBUG:boto:Token: None 2022-09-12T20:16:17.588 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj148?versionId=null 2022-09-12T20:16:17.589 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y3++2hZ7/EIiHBEP1P5Z0btE7jE= 2022-09-12T20:16:17.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y3++2hZ7/EIiHBEP1P5Z0btE7jE=', 'Content-Length': '0'} 2022-09-12T20:16:17.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001e5564ce7745b34-00631f9391-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.593 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:17.593 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:17.594 DEBUG:boto:path=/jdpiak-75/?prefix=obj148 2022-09-12T20:16:17.594 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj148 2022-09-12T20:16:17.595 DEBUG:boto:Method: GET 2022-09-12T20:16:17.596 DEBUG:boto:Path: /jdpiak-75/?prefix=obj148 2022-09-12T20:16:17.596 DEBUG:boto:Data: 2022-09-12T20:16:17.597 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.597 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.598 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.598 DEBUG:boto:Params: {} 2022-09-12T20:16:17.599 DEBUG:boto:Token: None 2022-09-12T20:16:17.599 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/ 2022-09-12T20:16:17.600 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ= 2022-09-12T20:16:17.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ=', 'Content-Length': '0'} 2022-09-12T20:16:17.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f85abb2fb8077cc2-00631f9391-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.604 DEBUG:boto:b'jdpiak-75obj1481000falseobj1482022-09-12T20:14:27.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:17.605 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:17.605 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:17.606 DEBUG:boto:path=/jdpiak-75/?prefix=obj148 2022-09-12T20:16:17.606 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj148 2022-09-12T20:16:17.607 DEBUG:boto:Method: GET 2022-09-12T20:16:17.608 DEBUG:boto:Path: /jdpiak-75/?prefix=obj148 2022-09-12T20:16:17.608 DEBUG:boto:Data: 2022-09-12T20:16:17.609 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.609 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.610 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.611 DEBUG:boto:Params: {} 2022-09-12T20:16:17.611 DEBUG:boto:Token: None 2022-09-12T20:16:17.612 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/ 2022-09-12T20:16:17.612 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ= 2022-09-12T20:16:17.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ=', 'Content-Length': '0'} 2022-09-12T20:16:17.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f18f9cdd50cc7a71-00631f9391-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.617 DEBUG:boto:b'jdpiak-75obj1481000falseobj1482022-09-12T20:14:27.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:17.618 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:16:17.619 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:16:17.620 DEBUG:boto:Method: HEAD 2022-09-12T20:16:17.621 DEBUG:boto:Path: /jdpiak-75/obj148 2022-09-12T20:16:17.621 DEBUG:boto:Data: 2022-09-12T20:16:17.622 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.622 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.623 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.623 DEBUG:boto:Params: {} 2022-09-12T20:16:17.624 DEBUG:boto:Token: None 2022-09-12T20:16:17.624 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj148 2022-09-12T20:16:17.625 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RuuByiTel6DR0QNM5sKM3rzRJhk= 2022-09-12T20:16:17.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RuuByiTel6DR0QNM5sKM3rzRJhk=', 'Content-Length': '0'} 2022-09-12T20:16:17.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004c7ccfb9fa639c0-00631f9391-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.628 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:16:17.629 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:16:17.629 DEBUG:boto:Method: HEAD 2022-09-12T20:16:17.630 DEBUG:boto:Path: /jdpiak-75/obj148 2022-09-12T20:16:17.631 DEBUG:boto:Data: 2022-09-12T20:16:17.631 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.632 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.632 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.633 DEBUG:boto:Params: {} 2022-09-12T20:16:17.634 DEBUG:boto:Token: None 2022-09-12T20:16:17.634 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj148 2022-09-12T20:16:17.635 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RuuByiTel6DR0QNM5sKM3rzRJhk= 2022-09-12T20:16:17.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RuuByiTel6DR0QNM5sKM3rzRJhk=', 'Content-Length': '0'} 2022-09-12T20:16:17.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b07f2f631a061dd8-00631f9391-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2022-09-12T20:16:17.639 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:16:17.639 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:16:17.640 DEBUG:boto:Method: GET 2022-09-12T20:16:17.640 DEBUG:boto:Path: /jdpiak-75/obj148 2022-09-12T20:16:17.641 DEBUG:boto:Data: 2022-09-12T20:16:17.641 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.642 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.642 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.643 DEBUG:boto:Params: {} 2022-09-12T20:16:17.644 DEBUG:boto:Token: None 2022-09-12T20:16:17.644 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj148 2022-09-12T20:16:17.645 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HcyLeUN7wMfnv0Pl9R7fUIu124g= 2022-09-12T20:16:17.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HcyLeUN7wMfnv0Pl9R7fUIu124g=', 'Content-Length': '0'} 2022-09-12T20:16:17.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008359c11165bfd42f-00631f9391-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.690 DEBUG:boto:path=/jdpiak-75/obj148 2022-09-12T20:16:17.691 DEBUG:boto:auth_path=/jdpiak-75/obj148 2022-09-12T20:16:17.691 DEBUG:boto:Method: GET 2022-09-12T20:16:17.692 DEBUG:boto:Path: /jdpiak-75/obj148 2022-09-12T20:16:17.693 DEBUG:boto:Data: 2022-09-12T20:16:17.693 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.694 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.695 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.695 DEBUG:boto:Params: {} 2022-09-12T20:16:17.696 DEBUG:boto:Token: None 2022-09-12T20:16:17.697 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj148 2022-09-12T20:16:17.698 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HcyLeUN7wMfnv0Pl9R7fUIu124g= 2022-09-12T20:16:17.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HcyLeUN7wMfnv0Pl9R7fUIu124g=', 'Content-Length': '0'} 2022-09-12T20:16:17.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000156fe108f27eff3d-00631f9391-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.703 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:17.704 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:17.705 DEBUG:boto:path=/jdpiak-75/?prefix=obj148 2022-09-12T20:16:17.705 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj148 2022-09-12T20:16:17.706 DEBUG:boto:Method: GET 2022-09-12T20:16:17.707 DEBUG:boto:Path: /jdpiak-75/?prefix=obj148 2022-09-12T20:16:17.708 DEBUG:boto:Data: 2022-09-12T20:16:17.709 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.710 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.711 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.712 DEBUG:boto:Params: {} 2022-09-12T20:16:17.712 DEBUG:boto:Token: None 2022-09-12T20:16:17.713 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/ 2022-09-12T20:16:17.714 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ= 2022-09-12T20:16:17.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ=', 'Content-Length': '0'} 2022-09-12T20:16:17.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008de7869ffed32759-00631f9391-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.719 DEBUG:boto:b'jdpiak-75obj1481000falseobj1482022-09-12T20:14:27.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:17.720 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:17.720 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:17.721 DEBUG:boto:path=/jdpiak-75/?prefix=obj148 2022-09-12T20:16:17.721 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj148 2022-09-12T20:16:17.722 DEBUG:boto:Method: GET 2022-09-12T20:16:17.723 DEBUG:boto:Path: /jdpiak-75/?prefix=obj148 2022-09-12T20:16:17.723 DEBUG:boto:Data: 2022-09-12T20:16:17.724 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.724 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.725 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.726 DEBUG:boto:Params: {} 2022-09-12T20:16:17.726 DEBUG:boto:Token: None 2022-09-12T20:16:17.727 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/ 2022-09-12T20:16:17.728 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ= 2022-09-12T20:16:17.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ=', 'Content-Length': '0'} 2022-09-12T20:16:17.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085267231a98eaaed-00631f9391-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.732 DEBUG:boto:b'jdpiak-75obj1481000falseobj1482022-09-12T20:14:27.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:17.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-12T20:16:17.733 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:17.734 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:17.734 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:17.735 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:17.735 DEBUG:boto:Method: GET 2022-09-12T20:16:17.736 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:17.736 DEBUG:boto:Data: 2022-09-12T20:16:17.737 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.738 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.738 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.739 DEBUG:boto:Params: {} 2022-09-12T20:16:17.739 DEBUG:boto:Token: None 2022-09-12T20:16:17.740 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:17.740 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nZ92A/PaHMUxj8tOhpnPuZrtDiE= 2022-09-12T20:16:17.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nZ92A/PaHMUxj8tOhpnPuZrtDiE=', 'Content-Length': '0'} 2022-09-12T20:16:17.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a037d27a60d9f6cb-00631f9391-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.744 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:17.745 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:17.745 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:17.746 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:17.746 DEBUG:boto:Method: GET 2022-09-12T20:16:17.747 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:17.747 DEBUG:boto:Data: 2022-09-12T20:16:17.748 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.749 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.749 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.750 DEBUG:boto:Params: {} 2022-09-12T20:16:17.750 DEBUG:boto:Token: None 2022-09-12T20:16:17.751 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:17.751 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nZ92A/PaHMUxj8tOhpnPuZrtDiE= 2022-09-12T20:16:17.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nZ92A/PaHMUxj8tOhpnPuZrtDiE=', 'Content-Length': '0'} 2022-09-12T20:16:17.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058fb0e4130df054d-00631f9391-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.755 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:17.755 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:17.756 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:17.756 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:17.757 DEBUG:boto:Method: HEAD 2022-09-12T20:16:17.757 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:17.758 DEBUG:boto:Data: 2022-09-12T20:16:17.758 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.759 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.759 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.760 DEBUG:boto:Params: {} 2022-09-12T20:16:17.760 DEBUG:boto:Token: None 2022-09-12T20:16:17.761 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:17.762 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/NF8ruxnDHbhwinHiC7AQE3p0Uw= 2022-09-12T20:16:17.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/NF8ruxnDHbhwinHiC7AQE3p0Uw=', 'Content-Length': '0'} 2022-09-12T20:16:17.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f26c2a6d6d22810-00631f9391-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.765 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:17.766 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:17.766 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:17.767 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:17.767 DEBUG:boto:Method: HEAD 2022-09-12T20:16:17.768 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:17.768 DEBUG:boto:Data: 2022-09-12T20:16:17.769 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.769 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.770 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.770 DEBUG:boto:Params: {} 2022-09-12T20:16:17.771 DEBUG:boto:Token: None 2022-09-12T20:16:17.771 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:17.772 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/NF8ruxnDHbhwinHiC7AQE3p0Uw= 2022-09-12T20:16:17.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/NF8ruxnDHbhwinHiC7AQE3p0Uw=', 'Content-Length': '0'} 2022-09-12T20:16:17.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064fd3b2cc67c9031-00631f9391-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-12T20:16:17.776 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:17.776 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:17.777 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:17.777 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:17.778 DEBUG:boto:Method: GET 2022-09-12T20:16:17.778 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:17.779 DEBUG:boto:Data: 2022-09-12T20:16:17.779 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.780 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.780 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.781 DEBUG:boto:Params: {} 2022-09-12T20:16:17.782 DEBUG:boto:Token: None 2022-09-12T20:16:17.782 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:17.783 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nZ92A/PaHMUxj8tOhpnPuZrtDiE= 2022-09-12T20:16:17.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nZ92A/PaHMUxj8tOhpnPuZrtDiE=', 'Content-Length': '0'} 2022-09-12T20:16:17.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094109e15cdb557c7-00631f9391-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.786 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:17.787 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:17.787 DEBUG:boto:path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:17.788 DEBUG:boto:auth_path=/jdpiak-75/obj155?versionId=null 2022-09-12T20:16:17.788 DEBUG:boto:Method: GET 2022-09-12T20:16:17.789 DEBUG:boto:Path: /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:17.790 DEBUG:boto:Data: 2022-09-12T20:16:17.790 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.791 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.791 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.792 DEBUG:boto:Params: {} 2022-09-12T20:16:17.792 DEBUG:boto:Token: None 2022-09-12T20:16:17.793 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj155?versionId=null 2022-09-12T20:16:17.794 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nZ92A/PaHMUxj8tOhpnPuZrtDiE= 2022-09-12T20:16:17.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nZ92A/PaHMUxj8tOhpnPuZrtDiE=', 'Content-Length': '0'} 2022-09-12T20:16:17.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f60a372ad932c1f0-00631f9391-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.842 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:17.843 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:17.843 DEBUG:boto:path=/jdpiak-75/?prefix=obj155 2022-09-12T20:16:17.844 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj155 2022-09-12T20:16:17.845 DEBUG:boto:Method: GET 2022-09-12T20:16:17.845 DEBUG:boto:Path: /jdpiak-75/?prefix=obj155 2022-09-12T20:16:17.846 DEBUG:boto:Data: 2022-09-12T20:16:17.846 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.847 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.847 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.848 DEBUG:boto:Params: {} 2022-09-12T20:16:17.849 DEBUG:boto:Token: None 2022-09-12T20:16:17.849 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/ 2022-09-12T20:16:17.850 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ= 2022-09-12T20:16:17.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ=', 'Content-Length': '0'} 2022-09-12T20:16:17.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008629a4f317745c66-00631f9391-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.854 DEBUG:boto:b'jdpiak-75obj1551000falseobj1552022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:17.855 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:17.856 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:17.856 DEBUG:boto:path=/jdpiak-75/?prefix=obj155 2022-09-12T20:16:17.857 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj155 2022-09-12T20:16:17.857 DEBUG:boto:Method: GET 2022-09-12T20:16:17.858 DEBUG:boto:Path: /jdpiak-75/?prefix=obj155 2022-09-12T20:16:17.858 DEBUG:boto:Data: 2022-09-12T20:16:17.859 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.859 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.860 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.861 DEBUG:boto:Params: {} 2022-09-12T20:16:17.861 DEBUG:boto:Token: None 2022-09-12T20:16:17.862 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/ 2022-09-12T20:16:17.862 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ= 2022-09-12T20:16:17.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ=', 'Content-Length': '0'} 2022-09-12T20:16:17.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000921674fd385deb2b-00631f9391-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.866 DEBUG:boto:b'jdpiak-75obj1551000falseobj1552022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:17.867 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:17.868 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:17.868 DEBUG:boto:Method: HEAD 2022-09-12T20:16:17.869 DEBUG:boto:Path: /jdpiak-75/obj155 2022-09-12T20:16:17.869 DEBUG:boto:Data: 2022-09-12T20:16:17.870 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.870 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.871 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.871 DEBUG:boto:Params: {} 2022-09-12T20:16:17.872 DEBUG:boto:Token: None 2022-09-12T20:16:17.872 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj155 2022-09-12T20:16:17.873 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Std3Ep+HpzoaZz6I3SU5LiR0ayM= 2022-09-12T20:16:17.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Std3Ep+HpzoaZz6I3SU5LiR0ayM=', 'Content-Length': '0'} 2022-09-12T20:16:17.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8007ca1cfee6b4a-00631f9391-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.876 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:17.877 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:17.877 DEBUG:boto:Method: HEAD 2022-09-12T20:16:17.878 DEBUG:boto:Path: /jdpiak-75/obj155 2022-09-12T20:16:17.878 DEBUG:boto:Data: 2022-09-12T20:16:17.879 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.879 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.880 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.880 DEBUG:boto:Params: {} 2022-09-12T20:16:17.881 DEBUG:boto:Token: None 2022-09-12T20:16:17.882 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj155 2022-09-12T20:16:17.882 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Std3Ep+HpzoaZz6I3SU5LiR0ayM= 2022-09-12T20:16:17.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Std3Ep+HpzoaZz6I3SU5LiR0ayM=', 'Content-Length': '0'} 2022-09-12T20:16:17.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048ec2f8bf09bb3dd-00631f9391-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-12T20:16:17.886 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:17.887 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:17.887 DEBUG:boto:Method: GET 2022-09-12T20:16:17.888 DEBUG:boto:Path: /jdpiak-75/obj155 2022-09-12T20:16:17.888 DEBUG:boto:Data: 2022-09-12T20:16:17.889 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.889 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.890 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.890 DEBUG:boto:Params: {} 2022-09-12T20:16:17.891 DEBUG:boto:Token: None 2022-09-12T20:16:17.892 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj155 2022-09-12T20:16:17.892 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YYxDCCeOnxm2IShW0fhwKvODJOM= 2022-09-12T20:16:17.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YYxDCCeOnxm2IShW0fhwKvODJOM=', 'Content-Length': '0'} 2022-09-12T20:16:17.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000904c82135caff3b1-00631f9391-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.896 DEBUG:boto:path=/jdpiak-75/obj155 2022-09-12T20:16:17.896 DEBUG:boto:auth_path=/jdpiak-75/obj155 2022-09-12T20:16:17.897 DEBUG:boto:Method: GET 2022-09-12T20:16:17.897 DEBUG:boto:Path: /jdpiak-75/obj155 2022-09-12T20:16:17.898 DEBUG:boto:Data: 2022-09-12T20:16:17.899 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.899 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.900 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.900 DEBUG:boto:Params: {} 2022-09-12T20:16:17.901 DEBUG:boto:Token: None 2022-09-12T20:16:17.901 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj155 2022-09-12T20:16:17.902 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YYxDCCeOnxm2IShW0fhwKvODJOM= 2022-09-12T20:16:17.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YYxDCCeOnxm2IShW0fhwKvODJOM=', 'Content-Length': '0'} 2022-09-12T20:16:17.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dec6f4d034ce330-00631f9391-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.906 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:17.906 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:17.907 DEBUG:boto:path=/jdpiak-75/?prefix=obj155 2022-09-12T20:16:17.907 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj155 2022-09-12T20:16:17.908 DEBUG:boto:Method: GET 2022-09-12T20:16:17.908 DEBUG:boto:Path: /jdpiak-75/?prefix=obj155 2022-09-12T20:16:17.909 DEBUG:boto:Data: 2022-09-12T20:16:17.909 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.910 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.910 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.911 DEBUG:boto:Params: {} 2022-09-12T20:16:17.911 DEBUG:boto:Token: None 2022-09-12T20:16:17.912 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/ 2022-09-12T20:16:17.913 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ= 2022-09-12T20:16:17.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ=', 'Content-Length': '0'} 2022-09-12T20:16:17.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d367a4aaa53c421e-00631f9391-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.958 DEBUG:boto:b'jdpiak-75obj1551000falseobj1552022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:17.959 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:17.960 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:17.960 DEBUG:boto:path=/jdpiak-75/?prefix=obj155 2022-09-12T20:16:17.961 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj155 2022-09-12T20:16:17.961 DEBUG:boto:Method: GET 2022-09-12T20:16:17.962 DEBUG:boto:Path: /jdpiak-75/?prefix=obj155 2022-09-12T20:16:17.963 DEBUG:boto:Data: 2022-09-12T20:16:17.963 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.964 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.964 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.965 DEBUG:boto:Params: {} 2022-09-12T20:16:17.965 DEBUG:boto:Token: None 2022-09-12T20:16:17.966 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/ 2022-09-12T20:16:17.966 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ= 2022-09-12T20:16:17.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QMRE40GoQxLyZSUReDaY1UR4DdQ=', 'Content-Length': '0'} 2022-09-12T20:16:17.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf3b2fc79091f928-00631f9391-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.970 DEBUG:boto:b'jdpiak-75obj1551000falseobj1552022-09-12T20:12:24.442Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:17.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:16:17.972 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:17.973 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:17.973 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:17.974 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:17.974 DEBUG:boto:Method: GET 2022-09-12T20:16:17.975 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:17.975 DEBUG:boto:Data: 2022-09-12T20:16:17.976 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.977 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.977 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.978 DEBUG:boto:Params: {} 2022-09-12T20:16:17.978 DEBUG:boto:Token: None 2022-09-12T20:16:17.979 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:17.980 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ba24srp7QOAuqkV+EHxXRehnnn8= 2022-09-12T20:16:17.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ba24srp7QOAuqkV+EHxXRehnnn8=', 'Content-Length': '0'} 2022-09-12T20:16:17.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020ae5245a67dfd68-00631f9391-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.983 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:17.984 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:17.984 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:17.985 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:17.985 DEBUG:boto:Method: GET 2022-09-12T20:16:17.986 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:17.987 DEBUG:boto:Data: 2022-09-12T20:16:17.987 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.988 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:17.988 DEBUG:boto:Port: 8000 2022-09-12T20:16:17.989 DEBUG:boto:Params: {} 2022-09-12T20:16:17.989 DEBUG:boto:Token: None 2022-09-12T20:16:17.990 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:17 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:17.990 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ba24srp7QOAuqkV+EHxXRehnnn8= 2022-09-12T20:16:17.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ba24srp7QOAuqkV+EHxXRehnnn8=', 'Content-Length': '0'} 2022-09-12T20:16:17.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed9ca5c95821ce9c-00631f9391-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:17.994 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:17.995 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:17.995 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:17.996 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:17.996 DEBUG:boto:Method: HEAD 2022-09-12T20:16:17.997 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:17.997 DEBUG:boto:Data: 2022-09-12T20:16:17.998 DEBUG:boto:Headers: {} 2022-09-12T20:16:17.998 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:17.999 DEBUG:boto:Port: 8001 2022-09-12T20:16:17.999 DEBUG:boto:Params: {} 2022-09-12T20:16:18.000 DEBUG:boto:Token: None 2022-09-12T20:16:18.000 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:18.001 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ljBIQE3HhrJuwLh45JVdpZut1qw= 2022-09-12T20:16:18.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ljBIQE3HhrJuwLh45JVdpZut1qw=', 'Content-Length': '0'} 2022-09-12T20:16:18.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038248ee8efb3473c-00631f9392-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.004 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:18.005 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:18.005 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:18.006 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:18.006 DEBUG:boto:Method: HEAD 2022-09-12T20:16:18.007 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:18.007 DEBUG:boto:Data: 2022-09-12T20:16:18.008 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.008 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.009 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.009 DEBUG:boto:Params: {} 2022-09-12T20:16:18.010 DEBUG:boto:Token: None 2022-09-12T20:16:18.010 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:18.011 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ljBIQE3HhrJuwLh45JVdpZut1qw= 2022-09-12T20:16:18.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ljBIQE3HhrJuwLh45JVdpZut1qw=', 'Content-Length': '0'} 2022-09-12T20:16:18.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9dbd882395a290e-00631f9392-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:16:18.015 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:18.016 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:18.016 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:18.017 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:18.017 DEBUG:boto:Method: GET 2022-09-12T20:16:18.018 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:18.019 DEBUG:boto:Data: 2022-09-12T20:16:18.019 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.020 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.020 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.021 DEBUG:boto:Params: {} 2022-09-12T20:16:18.021 DEBUG:boto:Token: None 2022-09-12T20:16:18.022 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:18.022 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J3wihDOn8s8EMzjMqmcQb0bRcwA= 2022-09-12T20:16:18.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J3wihDOn8s8EMzjMqmcQb0bRcwA=', 'Content-Length': '0'} 2022-09-12T20:16:18.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000523f53ffae4b6150-00631f9392-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.026 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:18.027 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:18.028 DEBUG:boto:path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:18.028 DEBUG:boto:auth_path=/jdpiak-75/obj17?versionId=null 2022-09-12T20:16:18.029 DEBUG:boto:Method: GET 2022-09-12T20:16:18.029 DEBUG:boto:Path: /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:18.030 DEBUG:boto:Data: 2022-09-12T20:16:18.030 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.031 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.031 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.032 DEBUG:boto:Params: {} 2022-09-12T20:16:18.032 DEBUG:boto:Token: None 2022-09-12T20:16:18.033 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj17?versionId=null 2022-09-12T20:16:18.034 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J3wihDOn8s8EMzjMqmcQb0bRcwA= 2022-09-12T20:16:18.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J3wihDOn8s8EMzjMqmcQb0bRcwA=', 'Content-Length': '0'} 2022-09-12T20:16:18.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c7c2a85a5a7f023-00631f9392-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.037 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:18.038 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:18.038 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:16:18.039 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:16:18.039 DEBUG:boto:Method: GET 2022-09-12T20:16:18.040 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:16:18.040 DEBUG:boto:Data: 2022-09-12T20:16:18.041 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.041 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.042 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.042 DEBUG:boto:Params: {} 2022-09-12T20:16:18.043 DEBUG:boto:Token: None 2022-09-12T20:16:18.044 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/ 2022-09-12T20:16:18.044 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk= 2022-09-12T20:16:18.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk=', 'Content-Length': '0'} 2022-09-12T20:16:18.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069446fddf308a464-00631f9392-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.049 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:18.049 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:18.050 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:18.051 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:16:18.051 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:16:18.052 DEBUG:boto:Method: GET 2022-09-12T20:16:18.052 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:16:18.053 DEBUG:boto:Data: 2022-09-12T20:16:18.053 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.054 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.054 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.055 DEBUG:boto:Params: {} 2022-09-12T20:16:18.055 DEBUG:boto:Token: None 2022-09-12T20:16:18.056 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/ 2022-09-12T20:16:18.057 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk= 2022-09-12T20:16:18.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk=', 'Content-Length': '0'} 2022-09-12T20:16:18.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc6757d557d57940-00631f9392-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.061 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:18.061 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:18.062 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:18.063 DEBUG:boto:Method: HEAD 2022-09-12T20:16:18.063 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:16:18.064 DEBUG:boto:Data: 2022-09-12T20:16:18.064 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.065 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.065 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.066 DEBUG:boto:Params: {} 2022-09-12T20:16:18.066 DEBUG:boto:Token: None 2022-09-12T20:16:18.067 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj17 2022-09-12T20:16:18.067 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wp003RqxtxR5KHSa0JaKrhhFn/g= 2022-09-12T20:16:18.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wp003RqxtxR5KHSa0JaKrhhFn/g=', 'Content-Length': '0'} 2022-09-12T20:16:18.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023029f8ef668f38b-00631f9392-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.071 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:18.071 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:18.072 DEBUG:boto:Method: HEAD 2022-09-12T20:16:18.072 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:16:18.073 DEBUG:boto:Data: 2022-09-12T20:16:18.073 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.074 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.075 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.075 DEBUG:boto:Params: {} 2022-09-12T20:16:18.076 DEBUG:boto:Token: None 2022-09-12T20:16:18.077 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj17 2022-09-12T20:16:18.077 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wp003RqxtxR5KHSa0JaKrhhFn/g= 2022-09-12T20:16:18.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wp003RqxtxR5KHSa0JaKrhhFn/g=', 'Content-Length': '0'} 2022-09-12T20:16:18.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018111964b4f06088-00631f9392-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-12T20:16:18.081 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:18.082 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:18.083 DEBUG:boto:Method: GET 2022-09-12T20:16:18.083 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:16:18.084 DEBUG:boto:Data: 2022-09-12T20:16:18.084 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.085 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.085 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.086 DEBUG:boto:Params: {} 2022-09-12T20:16:18.086 DEBUG:boto:Token: None 2022-09-12T20:16:18.087 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj17 2022-09-12T20:16:18.088 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+LK+k6FIPyvE4fzTjkaBMr4+AvA= 2022-09-12T20:16:18.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+LK+k6FIPyvE4fzTjkaBMr4+AvA=', 'Content-Length': '0'} 2022-09-12T20:16:18.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005ce6d34cf8db647-00631f9392-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.134 DEBUG:boto:path=/jdpiak-75/obj17 2022-09-12T20:16:18.135 DEBUG:boto:auth_path=/jdpiak-75/obj17 2022-09-12T20:16:18.136 DEBUG:boto:Method: GET 2022-09-12T20:16:18.138 DEBUG:boto:Path: /jdpiak-75/obj17 2022-09-12T20:16:18.139 DEBUG:boto:Data: 2022-09-12T20:16:18.141 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.142 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.143 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.145 DEBUG:boto:Params: {} 2022-09-12T20:16:18.146 DEBUG:boto:Token: None 2022-09-12T20:16:18.148 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj17 2022-09-12T20:16:18.149 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+LK+k6FIPyvE4fzTjkaBMr4+AvA= 2022-09-12T20:16:18.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+LK+k6FIPyvE4fzTjkaBMr4+AvA=', 'Content-Length': '0'} 2022-09-12T20:16:18.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae1de40fa05a9ac4-00631f9392-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.156 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:18.157 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:18.157 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:16:18.158 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:16:18.159 DEBUG:boto:Method: GET 2022-09-12T20:16:18.161 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:16:18.162 DEBUG:boto:Data: 2022-09-12T20:16:18.163 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.164 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.165 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.165 DEBUG:boto:Params: {} 2022-09-12T20:16:18.166 DEBUG:boto:Token: None 2022-09-12T20:16:18.167 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/ 2022-09-12T20:16:18.168 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk= 2022-09-12T20:16:18.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk=', 'Content-Length': '0'} 2022-09-12T20:16:18.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e85a2eb9500f1b6-00631f9392-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.173 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:18.174 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:18.175 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:18.175 DEBUG:boto:path=/jdpiak-75/?prefix=obj17 2022-09-12T20:16:18.176 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj17 2022-09-12T20:16:18.176 DEBUG:boto:Method: GET 2022-09-12T20:16:18.177 DEBUG:boto:Path: /jdpiak-75/?prefix=obj17 2022-09-12T20:16:18.178 DEBUG:boto:Data: 2022-09-12T20:16:18.178 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.179 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.180 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.181 DEBUG:boto:Params: {} 2022-09-12T20:16:18.182 DEBUG:boto:Token: None 2022-09-12T20:16:18.183 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/ 2022-09-12T20:16:18.183 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk= 2022-09-12T20:16:18.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk=', 'Content-Length': '0'} 2022-09-12T20:16:18.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060b14813ab227b8f-00631f9392-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.230 DEBUG:boto:b'jdpiak-75obj171000falseobj172022-09-12T20:03:28.498Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:18.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2022-09-12T20:16:18.233 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:18.234 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:18.235 DEBUG:boto:path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:18.236 DEBUG:boto:auth_path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:18.237 DEBUG:boto:Method: GET 2022-09-12T20:16:18.239 DEBUG:boto:Path: /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:18.240 DEBUG:boto:Data: 2022-09-12T20:16:18.241 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.242 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.243 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.245 DEBUG:boto:Params: {} 2022-09-12T20:16:18.246 DEBUG:boto:Token: None 2022-09-12T20:16:18.247 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:18.249 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bH9nn1IiggXPAbBMdIi4QX4cnjg= 2022-09-12T20:16:18.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bH9nn1IiggXPAbBMdIi4QX4cnjg=', 'Content-Length': '0'} 2022-09-12T20:16:18.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1778c92798163aa-00631f9392-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.255 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:18.256 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:18.257 DEBUG:boto:path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:18.258 DEBUG:boto:auth_path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:18.259 DEBUG:boto:Method: GET 2022-09-12T20:16:18.260 DEBUG:boto:Path: /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:18.261 DEBUG:boto:Data: 2022-09-12T20:16:18.262 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.263 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.264 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.265 DEBUG:boto:Params: {} 2022-09-12T20:16:18.267 DEBUG:boto:Token: None 2022-09-12T20:16:18.268 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:18.270 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bH9nn1IiggXPAbBMdIi4QX4cnjg= 2022-09-12T20:16:18.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bH9nn1IiggXPAbBMdIi4QX4cnjg=', 'Content-Length': '0'} 2022-09-12T20:16:18.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf742e946b6ed9b5-00631f9392-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.275 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:18.277 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:18.278 DEBUG:boto:path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:18.279 DEBUG:boto:auth_path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:18.280 DEBUG:boto:Method: HEAD 2022-09-12T20:16:18.281 DEBUG:boto:Path: /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:18.282 DEBUG:boto:Data: 2022-09-12T20:16:18.284 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.285 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.286 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.287 DEBUG:boto:Params: {} 2022-09-12T20:16:18.288 DEBUG:boto:Token: None 2022-09-12T20:16:18.290 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:18.291 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QoDqm1Qg2wTOx5oLHLZin2e1JSI= 2022-09-12T20:16:18.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QoDqm1Qg2wTOx5oLHLZin2e1JSI=', 'Content-Length': '0'} 2022-09-12T20:16:18.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffe233b237db21b4-00631f9392-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.297 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:18.298 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:18.299 DEBUG:boto:path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:18.301 DEBUG:boto:auth_path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:18.302 DEBUG:boto:Method: HEAD 2022-09-12T20:16:18.303 DEBUG:boto:Path: /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:18.305 DEBUG:boto:Data: 2022-09-12T20:16:18.306 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.307 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.309 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.310 DEBUG:boto:Params: {} 2022-09-12T20:16:18.311 DEBUG:boto:Token: None 2022-09-12T20:16:18.312 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:18.313 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QoDqm1Qg2wTOx5oLHLZin2e1JSI= 2022-09-12T20:16:18.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QoDqm1Qg2wTOx5oLHLZin2e1JSI=', 'Content-Length': '0'} 2022-09-12T20:16:18.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5ae379faad6c264-00631f9392-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2022-09-12T20:16:18.320 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:18.321 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:18.322 DEBUG:boto:path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:18.322 DEBUG:boto:auth_path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:18.323 DEBUG:boto:Method: GET 2022-09-12T20:16:18.324 DEBUG:boto:Path: /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:18.324 DEBUG:boto:Data: 2022-09-12T20:16:18.325 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.326 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.326 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.327 DEBUG:boto:Params: {} 2022-09-12T20:16:18.328 DEBUG:boto:Token: None 2022-09-12T20:16:18.328 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:18.329 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bH9nn1IiggXPAbBMdIi4QX4cnjg= 2022-09-12T20:16:18.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bH9nn1IiggXPAbBMdIi4QX4cnjg=', 'Content-Length': '0'} 2022-09-12T20:16:18.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c72924259f6ad8d-00631f9392-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.333 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:18.334 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:18.335 DEBUG:boto:path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:18.335 DEBUG:boto:auth_path=/jdpiak-75/obj185?versionId=null 2022-09-12T20:16:18.336 DEBUG:boto:Method: GET 2022-09-12T20:16:18.337 DEBUG:boto:Path: /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:18.337 DEBUG:boto:Data: 2022-09-12T20:16:18.338 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.339 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.339 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.340 DEBUG:boto:Params: {} 2022-09-12T20:16:18.341 DEBUG:boto:Token: None 2022-09-12T20:16:18.341 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj185?versionId=null 2022-09-12T20:16:18.342 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bH9nn1IiggXPAbBMdIi4QX4cnjg= 2022-09-12T20:16:18.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bH9nn1IiggXPAbBMdIi4QX4cnjg=', 'Content-Length': '0'} 2022-09-12T20:16:18.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0ac81e9d85b0e0d-00631f9392-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.346 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:18.347 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:18.348 DEBUG:boto:path=/jdpiak-75/?prefix=obj185 2022-09-12T20:16:18.348 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj185 2022-09-12T20:16:18.349 DEBUG:boto:Method: GET 2022-09-12T20:16:18.350 DEBUG:boto:Path: /jdpiak-75/?prefix=obj185 2022-09-12T20:16:18.350 DEBUG:boto:Data: 2022-09-12T20:16:18.351 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.352 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.352 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.353 DEBUG:boto:Params: {} 2022-09-12T20:16:18.354 DEBUG:boto:Token: None 2022-09-12T20:16:18.355 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/ 2022-09-12T20:16:18.355 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk= 2022-09-12T20:16:18.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk=', 'Content-Length': '0'} 2022-09-12T20:16:18.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7369cc86806b493-00631f9392-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.360 DEBUG:boto:b'jdpiak-75obj1851000falseobj1852022-09-12T20:14:27.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:18.361 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:18.361 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:18.362 DEBUG:boto:path=/jdpiak-75/?prefix=obj185 2022-09-12T20:16:18.363 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj185 2022-09-12T20:16:18.363 DEBUG:boto:Method: GET 2022-09-12T20:16:18.364 DEBUG:boto:Path: /jdpiak-75/?prefix=obj185 2022-09-12T20:16:18.365 DEBUG:boto:Data: 2022-09-12T20:16:18.365 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.366 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.367 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.367 DEBUG:boto:Params: {} 2022-09-12T20:16:18.368 DEBUG:boto:Token: None 2022-09-12T20:16:18.369 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/ 2022-09-12T20:16:18.370 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk= 2022-09-12T20:16:18.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk=', 'Content-Length': '0'} 2022-09-12T20:16:18.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b2cedf818c4b9ec-00631f9392-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.374 DEBUG:boto:b'jdpiak-75obj1851000falseobj1852022-09-12T20:14:27.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:18.375 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:18.375 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:18.376 DEBUG:boto:Method: HEAD 2022-09-12T20:16:18.376 DEBUG:boto:Path: /jdpiak-75/obj185 2022-09-12T20:16:18.377 DEBUG:boto:Data: 2022-09-12T20:16:18.377 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.378 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.379 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.379 DEBUG:boto:Params: {} 2022-09-12T20:16:18.380 DEBUG:boto:Token: None 2022-09-12T20:16:18.381 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj185 2022-09-12T20:16:18.381 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sxkE1iYOp7pwar/triLtYmIVdeQ= 2022-09-12T20:16:18.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sxkE1iYOp7pwar/triLtYmIVdeQ=', 'Content-Length': '0'} 2022-09-12T20:16:18.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a3f42ccd57a4df5-00631f9392-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.384 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:18.385 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:18.386 DEBUG:boto:Method: HEAD 2022-09-12T20:16:18.386 DEBUG:boto:Path: /jdpiak-75/obj185 2022-09-12T20:16:18.387 DEBUG:boto:Data: 2022-09-12T20:16:18.387 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.388 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.388 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.389 DEBUG:boto:Params: {} 2022-09-12T20:16:18.390 DEBUG:boto:Token: None 2022-09-12T20:16:18.390 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj185 2022-09-12T20:16:18.391 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sxkE1iYOp7pwar/triLtYmIVdeQ= 2022-09-12T20:16:18.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sxkE1iYOp7pwar/triLtYmIVdeQ=', 'Content-Length': '0'} 2022-09-12T20:16:18.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bfa742c1fd96d3f-00631f9392-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2022-09-12T20:16:18.394 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:18.395 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:18.396 DEBUG:boto:Method: GET 2022-09-12T20:16:18.396 DEBUG:boto:Path: /jdpiak-75/obj185 2022-09-12T20:16:18.397 DEBUG:boto:Data: 2022-09-12T20:16:18.397 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.398 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.398 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.399 DEBUG:boto:Params: {} 2022-09-12T20:16:18.399 DEBUG:boto:Token: None 2022-09-12T20:16:18.400 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj185 2022-09-12T20:16:18.401 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DSddywyYEjXecsXRXoTbgWErCF0= 2022-09-12T20:16:18.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DSddywyYEjXecsXRXoTbgWErCF0=', 'Content-Length': '0'} 2022-09-12T20:16:18.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006555761306218aad-00631f9392-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.446 DEBUG:boto:path=/jdpiak-75/obj185 2022-09-12T20:16:18.447 DEBUG:boto:auth_path=/jdpiak-75/obj185 2022-09-12T20:16:18.448 DEBUG:boto:Method: GET 2022-09-12T20:16:18.448 DEBUG:boto:Path: /jdpiak-75/obj185 2022-09-12T20:16:18.449 DEBUG:boto:Data: 2022-09-12T20:16:18.449 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.450 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.450 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.451 DEBUG:boto:Params: {} 2022-09-12T20:16:18.451 DEBUG:boto:Token: None 2022-09-12T20:16:18.452 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj185 2022-09-12T20:16:18.453 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DSddywyYEjXecsXRXoTbgWErCF0= 2022-09-12T20:16:18.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DSddywyYEjXecsXRXoTbgWErCF0=', 'Content-Length': '0'} 2022-09-12T20:16:18.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000938ac7ecaa6e32be-00631f9392-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.456 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:18.457 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:18.458 DEBUG:boto:path=/jdpiak-75/?prefix=obj185 2022-09-12T20:16:18.458 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj185 2022-09-12T20:16:18.459 DEBUG:boto:Method: GET 2022-09-12T20:16:18.459 DEBUG:boto:Path: /jdpiak-75/?prefix=obj185 2022-09-12T20:16:18.460 DEBUG:boto:Data: 2022-09-12T20:16:18.460 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.461 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.461 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.462 DEBUG:boto:Params: {} 2022-09-12T20:16:18.463 DEBUG:boto:Token: None 2022-09-12T20:16:18.463 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/ 2022-09-12T20:16:18.464 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk= 2022-09-12T20:16:18.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk=', 'Content-Length': '0'} 2022-09-12T20:16:18.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d43f5e5bc7b2cbf-00631f9392-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.468 DEBUG:boto:b'jdpiak-75obj1851000falseobj1852022-09-12T20:14:27.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:18.469 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:18.470 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:18.470 DEBUG:boto:path=/jdpiak-75/?prefix=obj185 2022-09-12T20:16:18.471 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj185 2022-09-12T20:16:18.471 DEBUG:boto:Method: GET 2022-09-12T20:16:18.472 DEBUG:boto:Path: /jdpiak-75/?prefix=obj185 2022-09-12T20:16:18.472 DEBUG:boto:Data: 2022-09-12T20:16:18.473 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.473 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.474 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.475 DEBUG:boto:Params: {} 2022-09-12T20:16:18.475 DEBUG:boto:Token: None 2022-09-12T20:16:18.476 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/ 2022-09-12T20:16:18.476 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk= 2022-09-12T20:16:18.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk=', 'Content-Length': '0'} 2022-09-12T20:16:18.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae3a306c239c031e-00631f9392-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.480 DEBUG:boto:b'jdpiak-75obj1851000falseobj1852022-09-12T20:14:27.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:18.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:16:18.482 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:18.482 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:18.483 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:18.483 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:18.484 DEBUG:boto:Method: GET 2022-09-12T20:16:18.485 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:18.485 DEBUG:boto:Data: 2022-09-12T20:16:18.486 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.486 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.487 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.487 DEBUG:boto:Params: {} 2022-09-12T20:16:18.488 DEBUG:boto:Token: None 2022-09-12T20:16:18.489 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:18.489 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:072qRKaZFvJKrxtQRQlUyaHtbkE= 2022-09-12T20:16:18.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:072qRKaZFvJKrxtQRQlUyaHtbkE=', 'Content-Length': '0'} 2022-09-12T20:16:18.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004531070696c5a8f2-00631f9392-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.493 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:18.493 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:18.494 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:18.494 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:18.495 DEBUG:boto:Method: GET 2022-09-12T20:16:18.496 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:18.496 DEBUG:boto:Data: 2022-09-12T20:16:18.497 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.497 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.498 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.498 DEBUG:boto:Params: {} 2022-09-12T20:16:18.499 DEBUG:boto:Token: None 2022-09-12T20:16:18.499 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:18.500 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:072qRKaZFvJKrxtQRQlUyaHtbkE= 2022-09-12T20:16:18.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:072qRKaZFvJKrxtQRQlUyaHtbkE=', 'Content-Length': '0'} 2022-09-12T20:16:18.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009894c3313ae528dc-00631f9392-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.504 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:18.504 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:18.505 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:18.505 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:18.506 DEBUG:boto:Method: HEAD 2022-09-12T20:16:18.506 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:18.507 DEBUG:boto:Data: 2022-09-12T20:16:18.507 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.508 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.508 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.509 DEBUG:boto:Params: {} 2022-09-12T20:16:18.509 DEBUG:boto:Token: None 2022-09-12T20:16:18.510 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:18.511 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+MuHQKLj6C0mlMzwmnVmlrGQX/k= 2022-09-12T20:16:18.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+MuHQKLj6C0mlMzwmnVmlrGQX/k=', 'Content-Length': '0'} 2022-09-12T20:16:18.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abf5e3b1889a6362-00631f9392-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.514 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:18.515 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:18.515 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:18.516 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:18.516 DEBUG:boto:Method: HEAD 2022-09-12T20:16:18.517 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:18.517 DEBUG:boto:Data: 2022-09-12T20:16:18.518 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.518 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.519 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.519 DEBUG:boto:Params: {} 2022-09-12T20:16:18.520 DEBUG:boto:Token: None 2022-09-12T20:16:18.521 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:18.521 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+MuHQKLj6C0mlMzwmnVmlrGQX/k= 2022-09-12T20:16:18.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+MuHQKLj6C0mlMzwmnVmlrGQX/k=', 'Content-Length': '0'} 2022-09-12T20:16:18.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a58d306b1ab08b4-00631f9392-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:16:18.525 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:18.525 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:18.526 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:18.527 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:18.527 DEBUG:boto:Method: GET 2022-09-12T20:16:18.528 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:18.528 DEBUG:boto:Data: 2022-09-12T20:16:18.529 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.529 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.530 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.530 DEBUG:boto:Params: {} 2022-09-12T20:16:18.531 DEBUG:boto:Token: None 2022-09-12T20:16:18.531 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:18.532 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:072qRKaZFvJKrxtQRQlUyaHtbkE= 2022-09-12T20:16:18.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:072qRKaZFvJKrxtQRQlUyaHtbkE=', 'Content-Length': '0'} 2022-09-12T20:16:18.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046922c1832d255eb-00631f9392-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.535 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:18.536 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:18.537 DEBUG:boto:path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:18.537 DEBUG:boto:auth_path=/jdpiak-75/obj19?versionId=null 2022-09-12T20:16:18.538 DEBUG:boto:Method: GET 2022-09-12T20:16:18.538 DEBUG:boto:Path: /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:18.539 DEBUG:boto:Data: 2022-09-12T20:16:18.539 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.540 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.540 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.541 DEBUG:boto:Params: {} 2022-09-12T20:16:18.541 DEBUG:boto:Token: None 2022-09-12T20:16:18.542 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj19?versionId=null 2022-09-12T20:16:18.543 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:072qRKaZFvJKrxtQRQlUyaHtbkE= 2022-09-12T20:16:18.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:072qRKaZFvJKrxtQRQlUyaHtbkE=', 'Content-Length': '0'} 2022-09-12T20:16:18.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f8b38789c250274-00631f9392-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.546 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:18.546 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:18.547 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:16:18.548 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:16:18.548 DEBUG:boto:Method: GET 2022-09-12T20:16:18.549 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:16:18.549 DEBUG:boto:Data: 2022-09-12T20:16:18.550 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.550 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.551 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.551 DEBUG:boto:Params: {} 2022-09-12T20:16:18.552 DEBUG:boto:Token: None 2022-09-12T20:16:18.552 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/ 2022-09-12T20:16:18.553 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk= 2022-09-12T20:16:18.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk=', 'Content-Length': '0'} 2022-09-12T20:16:18.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d912e5236adb674-00631f9392-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.598 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:18.599 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:18.600 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:18.600 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:16:18.601 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:16:18.601 DEBUG:boto:Method: GET 2022-09-12T20:16:18.602 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:16:18.602 DEBUG:boto:Data: 2022-09-12T20:16:18.603 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.603 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.604 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.604 DEBUG:boto:Params: {} 2022-09-12T20:16:18.605 DEBUG:boto:Token: None 2022-09-12T20:16:18.606 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/ 2022-09-12T20:16:18.606 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk= 2022-09-12T20:16:18.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk=', 'Content-Length': '0'} 2022-09-12T20:16:18.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004770d2c5e2798f1c-00631f9392-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.611 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:18.611 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:18.612 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:18.612 DEBUG:boto:Method: HEAD 2022-09-12T20:16:18.613 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:16:18.613 DEBUG:boto:Data: 2022-09-12T20:16:18.614 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.614 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.615 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.615 DEBUG:boto:Params: {} 2022-09-12T20:16:18.616 DEBUG:boto:Token: None 2022-09-12T20:16:18.617 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj19 2022-09-12T20:16:18.617 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:obzFyueBw2NUhBFHXkIgzAG/iGQ= 2022-09-12T20:16:18.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:obzFyueBw2NUhBFHXkIgzAG/iGQ=', 'Content-Length': '0'} 2022-09-12T20:16:18.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5f6361456db8b62-00631f9392-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.621 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:18.621 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:18.622 DEBUG:boto:Method: HEAD 2022-09-12T20:16:18.622 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:16:18.623 DEBUG:boto:Data: 2022-09-12T20:16:18.623 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.624 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.625 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.625 DEBUG:boto:Params: {} 2022-09-12T20:16:18.626 DEBUG:boto:Token: None 2022-09-12T20:16:18.627 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj19 2022-09-12T20:16:18.628 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:obzFyueBw2NUhBFHXkIgzAG/iGQ= 2022-09-12T20:16:18.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:obzFyueBw2NUhBFHXkIgzAG/iGQ=', 'Content-Length': '0'} 2022-09-12T20:16:18.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044f79773a2132f39-00631f9392-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-12T20:16:18.631 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:18.632 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:18.633 DEBUG:boto:Method: GET 2022-09-12T20:16:18.633 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:16:18.634 DEBUG:boto:Data: 2022-09-12T20:16:18.634 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.635 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.635 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.636 DEBUG:boto:Params: {} 2022-09-12T20:16:18.637 DEBUG:boto:Token: None 2022-09-12T20:16:18.637 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj19 2022-09-12T20:16:18.638 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SY9vdLWReebgTDYCPv0Rl7yZzlU= 2022-09-12T20:16:18.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SY9vdLWReebgTDYCPv0Rl7yZzlU=', 'Content-Length': '0'} 2022-09-12T20:16:18.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e043375c38c5ccc3-00631f9392-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.641 DEBUG:boto:path=/jdpiak-75/obj19 2022-09-12T20:16:18.642 DEBUG:boto:auth_path=/jdpiak-75/obj19 2022-09-12T20:16:18.642 DEBUG:boto:Method: GET 2022-09-12T20:16:18.643 DEBUG:boto:Path: /jdpiak-75/obj19 2022-09-12T20:16:18.644 DEBUG:boto:Data: 2022-09-12T20:16:18.644 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.645 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.645 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.646 DEBUG:boto:Params: {} 2022-09-12T20:16:18.647 DEBUG:boto:Token: None 2022-09-12T20:16:18.647 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj19 2022-09-12T20:16:18.648 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SY9vdLWReebgTDYCPv0Rl7yZzlU= 2022-09-12T20:16:18.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SY9vdLWReebgTDYCPv0Rl7yZzlU=', 'Content-Length': '0'} 2022-09-12T20:16:18.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000770e7c4ed3ce1e33-00631f9392-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.694 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:18.695 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:18.695 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:16:18.696 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:16:18.696 DEBUG:boto:Method: GET 2022-09-12T20:16:18.697 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:16:18.697 DEBUG:boto:Data: 2022-09-12T20:16:18.698 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.699 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.699 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.700 DEBUG:boto:Params: {} 2022-09-12T20:16:18.700 DEBUG:boto:Token: None 2022-09-12T20:16:18.701 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/ 2022-09-12T20:16:18.701 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk= 2022-09-12T20:16:18.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk=', 'Content-Length': '0'} 2022-09-12T20:16:18.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000462268bf618d52be-00631f9392-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.706 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:18.707 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:18.708 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:18.708 DEBUG:boto:path=/jdpiak-75/?prefix=obj19 2022-09-12T20:16:18.709 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj19 2022-09-12T20:16:18.709 DEBUG:boto:Method: GET 2022-09-12T20:16:18.710 DEBUG:boto:Path: /jdpiak-75/?prefix=obj19 2022-09-12T20:16:18.710 DEBUG:boto:Data: 2022-09-12T20:16:18.711 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.711 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.712 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.712 DEBUG:boto:Params: {} 2022-09-12T20:16:18.713 DEBUG:boto:Token: None 2022-09-12T20:16:18.714 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/ 2022-09-12T20:16:18.714 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk= 2022-09-12T20:16:18.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk=', 'Content-Length': '0'} 2022-09-12T20:16:18.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e283bebbe4b2cf2d-00631f9392-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.718 DEBUG:boto:b'jdpiak-75obj191000falseobj192022-09-12T20:06:16.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:18.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:16:18.720 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:18.720 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:18.721 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:18.721 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:18.722 DEBUG:boto:Method: GET 2022-09-12T20:16:18.722 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:18.723 DEBUG:boto:Data: 2022-09-12T20:16:18.723 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.724 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.724 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.725 DEBUG:boto:Params: {} 2022-09-12T20:16:18.726 DEBUG:boto:Token: None 2022-09-12T20:16:18.726 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:18.727 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/U4hj3F9WMmlJ++frDJy1O2M+eI= 2022-09-12T20:16:18.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/U4hj3F9WMmlJ++frDJy1O2M+eI=', 'Content-Length': '0'} 2022-09-12T20:16:18.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2d2e23f3abefb89-00631f9392-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.731 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:18.731 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:18.732 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:18.732 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:18.733 DEBUG:boto:Method: GET 2022-09-12T20:16:18.733 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:18.734 DEBUG:boto:Data: 2022-09-12T20:16:18.734 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.735 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.735 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.736 DEBUG:boto:Params: {} 2022-09-12T20:16:18.737 DEBUG:boto:Token: None 2022-09-12T20:16:18.737 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:18.738 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/U4hj3F9WMmlJ++frDJy1O2M+eI= 2022-09-12T20:16:18.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/U4hj3F9WMmlJ++frDJy1O2M+eI=', 'Content-Length': '0'} 2022-09-12T20:16:18.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb6f2e7812eeec60-00631f9392-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.742 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:18.743 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:18.744 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:18.745 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:18.746 DEBUG:boto:Method: HEAD 2022-09-12T20:16:18.746 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:18.747 DEBUG:boto:Data: 2022-09-12T20:16:18.747 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.748 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.748 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.749 DEBUG:boto:Params: {} 2022-09-12T20:16:18.749 DEBUG:boto:Token: None 2022-09-12T20:16:18.750 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:18.750 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nWcjVmdf7rQzjpfvWfggqtwWhtU= 2022-09-12T20:16:18.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nWcjVmdf7rQzjpfvWfggqtwWhtU=', 'Content-Length': '0'} 2022-09-12T20:16:18.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007775d022d191d5e5-00631f9392-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.754 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:18.754 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:18.755 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:18.756 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:18.756 DEBUG:boto:Method: HEAD 2022-09-12T20:16:18.757 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:18.757 DEBUG:boto:Data: 2022-09-12T20:16:18.758 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.758 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.759 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.759 DEBUG:boto:Params: {} 2022-09-12T20:16:18.760 DEBUG:boto:Token: None 2022-09-12T20:16:18.760 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:18.761 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:nWcjVmdf7rQzjpfvWfggqtwWhtU= 2022-09-12T20:16:18.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:nWcjVmdf7rQzjpfvWfggqtwWhtU=', 'Content-Length': '0'} 2022-09-12T20:16:18.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071ee957885759b70-00631f9392-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:16:18.765 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:18.765 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:18.766 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:18.766 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:18.767 DEBUG:boto:Method: GET 2022-09-12T20:16:18.768 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:18.768 DEBUG:boto:Data: 2022-09-12T20:16:18.769 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.769 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.770 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.770 DEBUG:boto:Params: {} 2022-09-12T20:16:18.771 DEBUG:boto:Token: None 2022-09-12T20:16:18.771 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:18.772 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/U4hj3F9WMmlJ++frDJy1O2M+eI= 2022-09-12T20:16:18.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/U4hj3F9WMmlJ++frDJy1O2M+eI=', 'Content-Length': '0'} 2022-09-12T20:16:18.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d029dcd8181fc472-00631f9392-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.776 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:18.776 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:18.777 DEBUG:boto:path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:18.777 DEBUG:boto:auth_path=/jdpiak-75/obj22?versionId=null 2022-09-12T20:16:18.778 DEBUG:boto:Method: GET 2022-09-12T20:16:18.778 DEBUG:boto:Path: /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:18.787 DEBUG:boto:Data: 2022-09-12T20:16:18.788 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.789 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.790 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.791 DEBUG:boto:Params: {} 2022-09-12T20:16:18.792 DEBUG:boto:Token: None 2022-09-12T20:16:18.793 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj22?versionId=null 2022-09-12T20:16:18.794 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/U4hj3F9WMmlJ++frDJy1O2M+eI= 2022-09-12T20:16:18.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/U4hj3F9WMmlJ++frDJy1O2M+eI=', 'Content-Length': '0'} 2022-09-12T20:16:18.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea7693267396ff43-00631f9392-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.797 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:18.798 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:18.799 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:16:18.799 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:16:18.800 DEBUG:boto:Method: GET 2022-09-12T20:16:18.800 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:16:18.801 DEBUG:boto:Data: 2022-09-12T20:16:18.801 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.802 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.802 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.803 DEBUG:boto:Params: {} 2022-09-12T20:16:18.803 DEBUG:boto:Token: None 2022-09-12T20:16:18.804 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/ 2022-09-12T20:16:18.805 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk= 2022-09-12T20:16:18.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk=', 'Content-Length': '0'} 2022-09-12T20:16:18.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d5f2aff30838c22-00631f9392-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.809 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:18.810 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:18.810 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:18.811 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:16:18.811 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:16:18.812 DEBUG:boto:Method: GET 2022-09-12T20:16:18.812 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:16:18.813 DEBUG:boto:Data: 2022-09-12T20:16:18.813 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.814 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.814 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.815 DEBUG:boto:Params: {} 2022-09-12T20:16:18.816 DEBUG:boto:Token: None 2022-09-12T20:16:18.816 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/ 2022-09-12T20:16:18.817 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk= 2022-09-12T20:16:18.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk=', 'Content-Length': '0'} 2022-09-12T20:16:18.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094449d2a78449169-00631f9392-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.820 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:18.821 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:18.822 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:18.822 DEBUG:boto:Method: HEAD 2022-09-12T20:16:18.823 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:16:18.823 DEBUG:boto:Data: 2022-09-12T20:16:18.824 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.824 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.825 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.825 DEBUG:boto:Params: {} 2022-09-12T20:16:18.826 DEBUG:boto:Token: None 2022-09-12T20:16:18.826 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj22 2022-09-12T20:16:18.827 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xJy7LKdtbBDeqbpVwUArsRSPr5Y= 2022-09-12T20:16:18.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xJy7LKdtbBDeqbpVwUArsRSPr5Y=', 'Content-Length': '0'} 2022-09-12T20:16:18.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031efdb382c83bcb4-00631f9392-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.830 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:18.831 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:18.832 DEBUG:boto:Method: HEAD 2022-09-12T20:16:18.832 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:16:18.833 DEBUG:boto:Data: 2022-09-12T20:16:18.833 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.834 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.835 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.835 DEBUG:boto:Params: {} 2022-09-12T20:16:18.836 DEBUG:boto:Token: None 2022-09-12T20:16:18.837 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj22 2022-09-12T20:16:18.838 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xJy7LKdtbBDeqbpVwUArsRSPr5Y= 2022-09-12T20:16:18.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xJy7LKdtbBDeqbpVwUArsRSPr5Y=', 'Content-Length': '0'} 2022-09-12T20:16:18.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000010bbf4050434b38-00631f9392-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:16:18.842 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:18.842 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:18.843 DEBUG:boto:Method: GET 2022-09-12T20:16:18.843 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:16:18.844 DEBUG:boto:Data: 2022-09-12T20:16:18.844 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.845 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.845 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.846 DEBUG:boto:Params: {} 2022-09-12T20:16:18.847 DEBUG:boto:Token: None 2022-09-12T20:16:18.847 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj22 2022-09-12T20:16:18.848 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:klKfbTchLQYEtSKsV3WotLLHIMw= 2022-09-12T20:16:18.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:klKfbTchLQYEtSKsV3WotLLHIMw=', 'Content-Length': '0'} 2022-09-12T20:16:18.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c24351769a32047-00631f9392-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.852 DEBUG:boto:path=/jdpiak-75/obj22 2022-09-12T20:16:18.853 DEBUG:boto:auth_path=/jdpiak-75/obj22 2022-09-12T20:16:18.854 DEBUG:boto:Method: GET 2022-09-12T20:16:18.855 DEBUG:boto:Path: /jdpiak-75/obj22 2022-09-12T20:16:18.855 DEBUG:boto:Data: 2022-09-12T20:16:18.856 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.856 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.857 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.858 DEBUG:boto:Params: {} 2022-09-12T20:16:18.858 DEBUG:boto:Token: None 2022-09-12T20:16:18.859 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj22 2022-09-12T20:16:18.859 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:klKfbTchLQYEtSKsV3WotLLHIMw= 2022-09-12T20:16:18.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:klKfbTchLQYEtSKsV3WotLLHIMw=', 'Content-Length': '0'} 2022-09-12T20:16:18.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000622524126033501d-00631f9392-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.863 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:18.863 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:18.864 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:16:18.865 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:16:18.865 DEBUG:boto:Method: GET 2022-09-12T20:16:18.866 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:16:18.866 DEBUG:boto:Data: 2022-09-12T20:16:18.867 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.867 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.868 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.868 DEBUG:boto:Params: {} 2022-09-12T20:16:18.869 DEBUG:boto:Token: None 2022-09-12T20:16:18.870 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/ 2022-09-12T20:16:18.870 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk= 2022-09-12T20:16:18.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk=', 'Content-Length': '0'} 2022-09-12T20:16:18.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002071c2771765bff9-00631f9392-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.914 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:18.915 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:18.916 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:18.916 DEBUG:boto:path=/jdpiak-75/?prefix=obj22 2022-09-12T20:16:18.917 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj22 2022-09-12T20:16:18.917 DEBUG:boto:Method: GET 2022-09-12T20:16:18.918 DEBUG:boto:Path: /jdpiak-75/?prefix=obj22 2022-09-12T20:16:18.919 DEBUG:boto:Data: 2022-09-12T20:16:18.919 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.920 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.920 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.921 DEBUG:boto:Params: {} 2022-09-12T20:16:18.921 DEBUG:boto:Token: None 2022-09-12T20:16:18.922 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/ 2022-09-12T20:16:18.922 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk= 2022-09-12T20:16:18.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WuuHUr9wLaF1Xog1tr2QGigmMdk=', 'Content-Length': '0'} 2022-09-12T20:16:18.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064b0cc38139d5fb6-00631f9392-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.926 DEBUG:boto:b'jdpiak-75obj221000falseobj222022-09-12T20:03:02.445Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:18.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:16:18.928 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:18.929 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:18.930 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:18.931 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:18.931 DEBUG:boto:Method: GET 2022-09-12T20:16:18.932 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:18.932 DEBUG:boto:Data: 2022-09-12T20:16:18.933 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.934 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.935 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.936 DEBUG:boto:Params: {} 2022-09-12T20:16:18.937 DEBUG:boto:Token: None 2022-09-12T20:16:18.937 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:18.938 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c7EKOJgIAqhftYwWADCIizmNRnA= 2022-09-12T20:16:18.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c7EKOJgIAqhftYwWADCIizmNRnA=', 'Content-Length': '0'} 2022-09-12T20:16:18.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006c5a52418a37100-00631f9392-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.942 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:18.942 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:18.943 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:18.944 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:18.944 DEBUG:boto:Method: GET 2022-09-12T20:16:18.945 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:18.945 DEBUG:boto:Data: 2022-09-12T20:16:18.946 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.946 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.947 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.947 DEBUG:boto:Params: {} 2022-09-12T20:16:18.948 DEBUG:boto:Token: None 2022-09-12T20:16:18.948 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:18.949 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c7EKOJgIAqhftYwWADCIizmNRnA= 2022-09-12T20:16:18.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c7EKOJgIAqhftYwWADCIizmNRnA=', 'Content-Length': '0'} 2022-09-12T20:16:18.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8321b85629515cf-00631f9392-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.952 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:18.953 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:18.953 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:18.954 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:18.954 DEBUG:boto:Method: HEAD 2022-09-12T20:16:18.955 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:18.956 DEBUG:boto:Data: 2022-09-12T20:16:18.956 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.957 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.958 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.958 DEBUG:boto:Params: {} 2022-09-12T20:16:18.959 DEBUG:boto:Token: None 2022-09-12T20:16:18.960 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:18.960 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hbN8t04BmMwUJ3y14rSVNdP40Hc= 2022-09-12T20:16:18.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hbN8t04BmMwUJ3y14rSVNdP40Hc=', 'Content-Length': '0'} 2022-09-12T20:16:18.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbb99f81004a9028-00631f9392-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.964 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:18.964 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:18.965 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:18.965 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:18.966 DEBUG:boto:Method: HEAD 2022-09-12T20:16:18.966 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:18.967 DEBUG:boto:Data: 2022-09-12T20:16:18.967 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.968 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.968 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.969 DEBUG:boto:Params: {} 2022-09-12T20:16:18.969 DEBUG:boto:Token: None 2022-09-12T20:16:18.970 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:18.970 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hbN8t04BmMwUJ3y14rSVNdP40Hc= 2022-09-12T20:16:18.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hbN8t04BmMwUJ3y14rSVNdP40Hc=', 'Content-Length': '0'} 2022-09-12T20:16:18.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ee9f8ad69e36e7b-00631f9392-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:16:18.974 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:18.975 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:18.976 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:18.977 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:18.978 DEBUG:boto:Method: GET 2022-09-12T20:16:18.979 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:18.980 DEBUG:boto:Data: 2022-09-12T20:16:18.980 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.981 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:18.981 DEBUG:boto:Port: 8001 2022-09-12T20:16:18.982 DEBUG:boto:Params: {} 2022-09-12T20:16:18.982 DEBUG:boto:Token: None 2022-09-12T20:16:18.983 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:18.984 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c7EKOJgIAqhftYwWADCIizmNRnA= 2022-09-12T20:16:18.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c7EKOJgIAqhftYwWADCIizmNRnA=', 'Content-Length': '0'} 2022-09-12T20:16:18.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000830a3b859d6cd70b-00631f9392-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:18.987 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:18.988 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:18.988 DEBUG:boto:path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:18.989 DEBUG:boto:auth_path=/jdpiak-75/obj23?versionId=null 2022-09-12T20:16:18.990 DEBUG:boto:Method: GET 2022-09-12T20:16:18.991 DEBUG:boto:Path: /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:18.992 DEBUG:boto:Data: 2022-09-12T20:16:18.993 DEBUG:boto:Headers: {} 2022-09-12T20:16:18.994 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:18.994 DEBUG:boto:Port: 8000 2022-09-12T20:16:18.995 DEBUG:boto:Params: {} 2022-09-12T20:16:18.995 DEBUG:boto:Token: None 2022-09-12T20:16:18.996 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:18 GMT /jdpiak-75/obj23?versionId=null 2022-09-12T20:16:18.997 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:c7EKOJgIAqhftYwWADCIizmNRnA= 2022-09-12T20:16:18.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:c7EKOJgIAqhftYwWADCIizmNRnA=', 'Content-Length': '0'} 2022-09-12T20:16:18.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cd0ccbe2d2316f6-00631f9392-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.000 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:19.001 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:19.001 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:16:19.002 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:16:19.002 DEBUG:boto:Method: GET 2022-09-12T20:16:19.003 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:16:19.003 DEBUG:boto:Data: 2022-09-12T20:16:19.004 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.004 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.005 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.005 DEBUG:boto:Params: {} 2022-09-12T20:16:19.006 DEBUG:boto:Token: None 2022-09-12T20:16:19.007 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/ 2022-09-12T20:16:19.007 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs= 2022-09-12T20:16:19.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs=', 'Content-Length': '0'} 2022-09-12T20:16:19.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7df4aa5e9c14fc7-00631f9393-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.054 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:19.055 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:19.056 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:19.056 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:16:19.057 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:16:19.057 DEBUG:boto:Method: GET 2022-09-12T20:16:19.058 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:16:19.059 DEBUG:boto:Data: 2022-09-12T20:16:19.059 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.060 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.060 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.061 DEBUG:boto:Params: {} 2022-09-12T20:16:19.061 DEBUG:boto:Token: None 2022-09-12T20:16:19.062 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/ 2022-09-12T20:16:19.063 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs= 2022-09-12T20:16:19.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs=', 'Content-Length': '0'} 2022-09-12T20:16:19.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba42215b2e324331-00631f9393-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.067 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:19.067 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:19.068 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:19.068 DEBUG:boto:Method: HEAD 2022-09-12T20:16:19.069 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:16:19.069 DEBUG:boto:Data: 2022-09-12T20:16:19.070 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.070 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.071 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.072 DEBUG:boto:Params: {} 2022-09-12T20:16:19.072 DEBUG:boto:Token: None 2022-09-12T20:16:19.073 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj23 2022-09-12T20:16:19.073 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yg19MLE3QaoE0rxE0fjk+HQ/u7g= 2022-09-12T20:16:19.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yg19MLE3QaoE0rxE0fjk+HQ/u7g=', 'Content-Length': '0'} 2022-09-12T20:16:19.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c446161d19969532-00631f9393-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.077 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:19.077 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:19.078 DEBUG:boto:Method: HEAD 2022-09-12T20:16:19.079 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:16:19.079 DEBUG:boto:Data: 2022-09-12T20:16:19.080 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.080 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.081 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.082 DEBUG:boto:Params: {} 2022-09-12T20:16:19.082 DEBUG:boto:Token: None 2022-09-12T20:16:19.083 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj23 2022-09-12T20:16:19.084 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yg19MLE3QaoE0rxE0fjk+HQ/u7g= 2022-09-12T20:16:19.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yg19MLE3QaoE0rxE0fjk+HQ/u7g=', 'Content-Length': '0'} 2022-09-12T20:16:19.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ad9411c2f1f2766-00631f9393-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-12T20:16:19.088 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:19.088 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:19.089 DEBUG:boto:Method: GET 2022-09-12T20:16:19.089 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:16:19.090 DEBUG:boto:Data: 2022-09-12T20:16:19.091 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.091 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.092 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.092 DEBUG:boto:Params: {} 2022-09-12T20:16:19.093 DEBUG:boto:Token: None 2022-09-12T20:16:19.094 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj23 2022-09-12T20:16:19.094 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x4zYIyL+BF9Bi0KrMt1RGvC+ccY= 2022-09-12T20:16:19.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x4zYIyL+BF9Bi0KrMt1RGvC+ccY=', 'Content-Length': '0'} 2022-09-12T20:16:19.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faa43c047e337a07-00631f9393-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.098 DEBUG:boto:path=/jdpiak-75/obj23 2022-09-12T20:16:19.098 DEBUG:boto:auth_path=/jdpiak-75/obj23 2022-09-12T20:16:19.099 DEBUG:boto:Method: GET 2022-09-12T20:16:19.100 DEBUG:boto:Path: /jdpiak-75/obj23 2022-09-12T20:16:19.100 DEBUG:boto:Data: 2022-09-12T20:16:19.101 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.101 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.102 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.102 DEBUG:boto:Params: {} 2022-09-12T20:16:19.103 DEBUG:boto:Token: None 2022-09-12T20:16:19.104 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj23 2022-09-12T20:16:19.104 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:x4zYIyL+BF9Bi0KrMt1RGvC+ccY= 2022-09-12T20:16:19.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:x4zYIyL+BF9Bi0KrMt1RGvC+ccY=', 'Content-Length': '0'} 2022-09-12T20:16:19.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2c70d43a956b84a-00631f9393-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.150 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:19.151 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:19.151 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:16:19.152 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:16:19.152 DEBUG:boto:Method: GET 2022-09-12T20:16:19.153 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:16:19.154 DEBUG:boto:Data: 2022-09-12T20:16:19.155 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.155 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.156 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.156 DEBUG:boto:Params: {} 2022-09-12T20:16:19.157 DEBUG:boto:Token: None 2022-09-12T20:16:19.158 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/ 2022-09-12T20:16:19.158 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs= 2022-09-12T20:16:19.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs=', 'Content-Length': '0'} 2022-09-12T20:16:19.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090c180b6dccf39ec-00631f9393-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.162 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:19.163 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:19.164 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:19.164 DEBUG:boto:path=/jdpiak-75/?prefix=obj23 2022-09-12T20:16:19.165 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj23 2022-09-12T20:16:19.166 DEBUG:boto:Method: GET 2022-09-12T20:16:19.166 DEBUG:boto:Path: /jdpiak-75/?prefix=obj23 2022-09-12T20:16:19.167 DEBUG:boto:Data: 2022-09-12T20:16:19.167 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.168 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.168 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.169 DEBUG:boto:Params: {} 2022-09-12T20:16:19.169 DEBUG:boto:Token: None 2022-09-12T20:16:19.170 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/ 2022-09-12T20:16:19.171 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs= 2022-09-12T20:16:19.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs=', 'Content-Length': '0'} 2022-09-12T20:16:19.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d771d9b609547e16-00631f9393-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.175 DEBUG:boto:b'jdpiak-75obj231000falseobj232022-09-12T20:08:38.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:19.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:16:19.176 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:19.177 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:19.177 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:19.178 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:19.178 DEBUG:boto:Method: GET 2022-09-12T20:16:19.179 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:19.180 DEBUG:boto:Data: 2022-09-12T20:16:19.180 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.181 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.181 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.182 DEBUG:boto:Params: {} 2022-09-12T20:16:19.183 DEBUG:boto:Token: None 2022-09-12T20:16:19.183 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:19.184 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DuaWCFOXZJ2oESAbZ59li8qe9vU= 2022-09-12T20:16:19.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DuaWCFOXZJ2oESAbZ59li8qe9vU=', 'Content-Length': '0'} 2022-09-12T20:16:19.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7bdb76f31b7de64-00631f9393-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.187 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:19.188 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:19.189 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:19.189 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:19.190 DEBUG:boto:Method: GET 2022-09-12T20:16:19.190 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:19.191 DEBUG:boto:Data: 2022-09-12T20:16:19.191 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.192 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.192 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.193 DEBUG:boto:Params: {} 2022-09-12T20:16:19.193 DEBUG:boto:Token: None 2022-09-12T20:16:19.194 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:19.195 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DuaWCFOXZJ2oESAbZ59li8qe9vU= 2022-09-12T20:16:19.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DuaWCFOXZJ2oESAbZ59li8qe9vU=', 'Content-Length': '0'} 2022-09-12T20:16:19.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003951d35970cea276-00631f9393-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.199 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:19.200 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:19.200 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:19.201 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:19.201 DEBUG:boto:Method: HEAD 2022-09-12T20:16:19.202 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:19.202 DEBUG:boto:Data: 2022-09-12T20:16:19.203 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.203 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.204 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.205 DEBUG:boto:Params: {} 2022-09-12T20:16:19.205 DEBUG:boto:Token: None 2022-09-12T20:16:19.206 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:19.206 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sHX2esOTSerimxaMqwpBl9fLA2g= 2022-09-12T20:16:19.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sHX2esOTSerimxaMqwpBl9fLA2g=', 'Content-Length': '0'} 2022-09-12T20:16:19.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dcd7db544bb9dc7-00631f9393-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.210 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:19.210 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:19.211 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:19.211 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:19.212 DEBUG:boto:Method: HEAD 2022-09-12T20:16:19.213 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:19.213 DEBUG:boto:Data: 2022-09-12T20:16:19.214 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.214 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.215 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.215 DEBUG:boto:Params: {} 2022-09-12T20:16:19.216 DEBUG:boto:Token: None 2022-09-12T20:16:19.216 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:19.217 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sHX2esOTSerimxaMqwpBl9fLA2g= 2022-09-12T20:16:19.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sHX2esOTSerimxaMqwpBl9fLA2g=', 'Content-Length': '0'} 2022-09-12T20:16:19.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e17064d7ae1b262b-00631f9393-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:16:19.221 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:19.221 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:19.222 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:19.222 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:19.223 DEBUG:boto:Method: GET 2022-09-12T20:16:19.223 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:19.224 DEBUG:boto:Data: 2022-09-12T20:16:19.225 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.225 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.226 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.226 DEBUG:boto:Params: {} 2022-09-12T20:16:19.227 DEBUG:boto:Token: None 2022-09-12T20:16:19.227 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:19.228 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DuaWCFOXZJ2oESAbZ59li8qe9vU= 2022-09-12T20:16:19.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DuaWCFOXZJ2oESAbZ59li8qe9vU=', 'Content-Length': '0'} 2022-09-12T20:16:19.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c472637151f6eba7-00631f9393-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.231 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:19.232 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:19.233 DEBUG:boto:path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:19.233 DEBUG:boto:auth_path=/jdpiak-75/obj29?versionId=null 2022-09-12T20:16:19.234 DEBUG:boto:Method: GET 2022-09-12T20:16:19.234 DEBUG:boto:Path: /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:19.235 DEBUG:boto:Data: 2022-09-12T20:16:19.235 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.236 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.236 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.237 DEBUG:boto:Params: {} 2022-09-12T20:16:19.237 DEBUG:boto:Token: None 2022-09-12T20:16:19.238 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj29?versionId=null 2022-09-12T20:16:19.239 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:DuaWCFOXZJ2oESAbZ59li8qe9vU= 2022-09-12T20:16:19.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:DuaWCFOXZJ2oESAbZ59li8qe9vU=', 'Content-Length': '0'} 2022-09-12T20:16:19.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f9007adb39158db-00631f9393-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.282 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:19.283 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:19.283 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:16:19.284 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:16:19.284 DEBUG:boto:Method: GET 2022-09-12T20:16:19.285 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:16:19.285 DEBUG:boto:Data: 2022-09-12T20:16:19.286 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.287 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.287 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.288 DEBUG:boto:Params: {} 2022-09-12T20:16:19.288 DEBUG:boto:Token: None 2022-09-12T20:16:19.289 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/ 2022-09-12T20:16:19.289 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs= 2022-09-12T20:16:19.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs=', 'Content-Length': '0'} 2022-09-12T20:16:19.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2c25c68a17af441-00631f9393-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.294 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:19.294 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:19.295 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:19.296 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:16:19.296 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:16:19.297 DEBUG:boto:Method: GET 2022-09-12T20:16:19.297 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:16:19.298 DEBUG:boto:Data: 2022-09-12T20:16:19.298 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.299 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.299 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.300 DEBUG:boto:Params: {} 2022-09-12T20:16:19.300 DEBUG:boto:Token: None 2022-09-12T20:16:19.301 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/ 2022-09-12T20:16:19.302 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs= 2022-09-12T20:16:19.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs=', 'Content-Length': '0'} 2022-09-12T20:16:19.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031a02edc50491770-00631f9393-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.306 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:19.306 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:19.307 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:19.308 DEBUG:boto:Method: HEAD 2022-09-12T20:16:19.308 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:16:19.309 DEBUG:boto:Data: 2022-09-12T20:16:19.309 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.310 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.310 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.311 DEBUG:boto:Params: {} 2022-09-12T20:16:19.311 DEBUG:boto:Token: None 2022-09-12T20:16:19.312 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj29 2022-09-12T20:16:19.312 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:a8sHTcbF57lglox56xYxMHXJMTY= 2022-09-12T20:16:19.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:a8sHTcbF57lglox56xYxMHXJMTY=', 'Content-Length': '0'} 2022-09-12T20:16:19.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d13940404670fb3e-00631f9393-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.316 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:19.316 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:19.317 DEBUG:boto:Method: HEAD 2022-09-12T20:16:19.317 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:16:19.318 DEBUG:boto:Data: 2022-09-12T20:16:19.318 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.319 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.319 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.320 DEBUG:boto:Params: {} 2022-09-12T20:16:19.320 DEBUG:boto:Token: None 2022-09-12T20:16:19.321 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj29 2022-09-12T20:16:19.322 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:a8sHTcbF57lglox56xYxMHXJMTY= 2022-09-12T20:16:19.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:a8sHTcbF57lglox56xYxMHXJMTY=', 'Content-Length': '0'} 2022-09-12T20:16:19.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003ebdc7053fb4e0b-00631f9393-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-12T20:16:19.326 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:19.327 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:19.327 DEBUG:boto:Method: GET 2022-09-12T20:16:19.328 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:16:19.328 DEBUG:boto:Data: 2022-09-12T20:16:19.329 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.330 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.330 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.331 DEBUG:boto:Params: {} 2022-09-12T20:16:19.331 DEBUG:boto:Token: None 2022-09-12T20:16:19.332 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj29 2022-09-12T20:16:19.332 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Hc1DfAU2jbGw+5iNqPEHxmlzbl0= 2022-09-12T20:16:19.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Hc1DfAU2jbGw+5iNqPEHxmlzbl0=', 'Content-Length': '0'} 2022-09-12T20:16:19.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f8d52b5702d2afa-00631f9393-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.378 DEBUG:boto:path=/jdpiak-75/obj29 2022-09-12T20:16:19.379 DEBUG:boto:auth_path=/jdpiak-75/obj29 2022-09-12T20:16:19.380 DEBUG:boto:Method: GET 2022-09-12T20:16:19.380 DEBUG:boto:Path: /jdpiak-75/obj29 2022-09-12T20:16:19.381 DEBUG:boto:Data: 2022-09-12T20:16:19.381 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.382 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.382 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.383 DEBUG:boto:Params: {} 2022-09-12T20:16:19.383 DEBUG:boto:Token: None 2022-09-12T20:16:19.384 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj29 2022-09-12T20:16:19.385 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Hc1DfAU2jbGw+5iNqPEHxmlzbl0= 2022-09-12T20:16:19.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Hc1DfAU2jbGw+5iNqPEHxmlzbl0=', 'Content-Length': '0'} 2022-09-12T20:16:19.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c44c803a63be189a-00631f9393-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.389 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:19.390 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:19.390 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:16:19.391 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:16:19.391 DEBUG:boto:Method: GET 2022-09-12T20:16:19.392 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:16:19.393 DEBUG:boto:Data: 2022-09-12T20:16:19.393 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.394 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.394 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.395 DEBUG:boto:Params: {} 2022-09-12T20:16:19.396 DEBUG:boto:Token: None 2022-09-12T20:16:19.396 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/ 2022-09-12T20:16:19.397 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs= 2022-09-12T20:16:19.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs=', 'Content-Length': '0'} 2022-09-12T20:16:19.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9aa221c5aa5d115-00631f9393-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.401 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:19.402 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:19.403 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:19.403 DEBUG:boto:path=/jdpiak-75/?prefix=obj29 2022-09-12T20:16:19.404 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj29 2022-09-12T20:16:19.405 DEBUG:boto:Method: GET 2022-09-12T20:16:19.405 DEBUG:boto:Path: /jdpiak-75/?prefix=obj29 2022-09-12T20:16:19.406 DEBUG:boto:Data: 2022-09-12T20:16:19.407 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.408 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.408 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.409 DEBUG:boto:Params: {} 2022-09-12T20:16:19.409 DEBUG:boto:Token: None 2022-09-12T20:16:19.410 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/ 2022-09-12T20:16:19.411 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs= 2022-09-12T20:16:19.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs=', 'Content-Length': '0'} 2022-09-12T20:16:19.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028637b6194e53b68-00631f9393-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.415 DEBUG:boto:b'jdpiak-75obj291000falseobj292022-09-12T20:10:08.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:19.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-12T20:16:19.416 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:19.417 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:19.417 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:19.418 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:19.418 DEBUG:boto:Method: GET 2022-09-12T20:16:19.419 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:19.420 DEBUG:boto:Data: 2022-09-12T20:16:19.420 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.421 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.422 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.422 DEBUG:boto:Params: {} 2022-09-12T20:16:19.423 DEBUG:boto:Token: None 2022-09-12T20:16:19.424 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:19.424 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UenzRrwxlI14rLFUHR72hknI3eg= 2022-09-12T20:16:19.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UenzRrwxlI14rLFUHR72hknI3eg=', 'Content-Length': '0'} 2022-09-12T20:16:19.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aef7a4774cc171b4-00631f9393-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.428 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:19.429 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:19.429 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:19.430 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:19.430 DEBUG:boto:Method: GET 2022-09-12T20:16:19.431 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:19.431 DEBUG:boto:Data: 2022-09-12T20:16:19.432 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.433 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.433 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.434 DEBUG:boto:Params: {} 2022-09-12T20:16:19.435 DEBUG:boto:Token: None 2022-09-12T20:16:19.436 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:19.436 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UenzRrwxlI14rLFUHR72hknI3eg= 2022-09-12T20:16:19.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UenzRrwxlI14rLFUHR72hknI3eg=', 'Content-Length': '0'} 2022-09-12T20:16:19.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095bb9e1aca25e9ef-00631f9393-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.440 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:19.441 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:19.441 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:19.442 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:19.443 DEBUG:boto:Method: HEAD 2022-09-12T20:16:19.443 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:19.444 DEBUG:boto:Data: 2022-09-12T20:16:19.445 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.445 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.446 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.446 DEBUG:boto:Params: {} 2022-09-12T20:16:19.447 DEBUG:boto:Token: None 2022-09-12T20:16:19.448 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:19.448 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t/8r1/cW5bv5xabU9FrwBVS1+20= 2022-09-12T20:16:19.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t/8r1/cW5bv5xabU9FrwBVS1+20=', 'Content-Length': '0'} 2022-09-12T20:16:19.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008eaa19b148539fb0-00631f9393-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.452 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:19.453 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:19.454 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:19.454 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:19.455 DEBUG:boto:Method: HEAD 2022-09-12T20:16:19.455 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:19.456 DEBUG:boto:Data: 2022-09-12T20:16:19.456 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.457 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.457 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.458 DEBUG:boto:Params: {} 2022-09-12T20:16:19.459 DEBUG:boto:Token: None 2022-09-12T20:16:19.459 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:19.460 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t/8r1/cW5bv5xabU9FrwBVS1+20= 2022-09-12T20:16:19.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t/8r1/cW5bv5xabU9FrwBVS1+20=', 'Content-Length': '0'} 2022-09-12T20:16:19.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000551e4a471ac8c28-00631f9393-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-12T20:16:19.464 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:19.464 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:19.465 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:19.466 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:19.466 DEBUG:boto:Method: GET 2022-09-12T20:16:19.467 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:19.467 DEBUG:boto:Data: 2022-09-12T20:16:19.468 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.468 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.469 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.470 DEBUG:boto:Params: {} 2022-09-12T20:16:19.470 DEBUG:boto:Token: None 2022-09-12T20:16:19.471 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:19.471 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UenzRrwxlI14rLFUHR72hknI3eg= 2022-09-12T20:16:19.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UenzRrwxlI14rLFUHR72hknI3eg=', 'Content-Length': '0'} 2022-09-12T20:16:19.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067cbf01ddcd433bc-00631f9393-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.475 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:19.476 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:19.476 DEBUG:boto:path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:19.477 DEBUG:boto:auth_path=/jdpiak-75/obj31?versionId=null 2022-09-12T20:16:19.477 DEBUG:boto:Method: GET 2022-09-12T20:16:19.478 DEBUG:boto:Path: /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:19.479 DEBUG:boto:Data: 2022-09-12T20:16:19.479 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.480 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.480 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.481 DEBUG:boto:Params: {} 2022-09-12T20:16:19.482 DEBUG:boto:Token: None 2022-09-12T20:16:19.482 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj31?versionId=null 2022-09-12T20:16:19.483 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:UenzRrwxlI14rLFUHR72hknI3eg= 2022-09-12T20:16:19.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:UenzRrwxlI14rLFUHR72hknI3eg=', 'Content-Length': '0'} 2022-09-12T20:16:19.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025d1370d7da67c7d-00631f9393-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.526 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:19.527 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:19.527 DEBUG:boto:path=/jdpiak-75/?prefix=obj31 2022-09-12T20:16:19.528 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj31 2022-09-12T20:16:19.528 DEBUG:boto:Method: GET 2022-09-12T20:16:19.529 DEBUG:boto:Path: /jdpiak-75/?prefix=obj31 2022-09-12T20:16:19.529 DEBUG:boto:Data: 2022-09-12T20:16:19.530 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.531 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.531 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.532 DEBUG:boto:Params: {} 2022-09-12T20:16:19.532 DEBUG:boto:Token: None 2022-09-12T20:16:19.533 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/ 2022-09-12T20:16:19.534 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs= 2022-09-12T20:16:19.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs=', 'Content-Length': '0'} 2022-09-12T20:16:19.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d873ecede3cc6d5-00631f9393-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.538 DEBUG:boto:b'jdpiak-75obj311000falseobj312022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:19.539 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:19.539 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:19.540 DEBUG:boto:path=/jdpiak-75/?prefix=obj31 2022-09-12T20:16:19.540 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj31 2022-09-12T20:16:19.541 DEBUG:boto:Method: GET 2022-09-12T20:16:19.542 DEBUG:boto:Path: /jdpiak-75/?prefix=obj31 2022-09-12T20:16:19.542 DEBUG:boto:Data: 2022-09-12T20:16:19.543 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.543 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.544 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.544 DEBUG:boto:Params: {} 2022-09-12T20:16:19.545 DEBUG:boto:Token: None 2022-09-12T20:16:19.545 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/ 2022-09-12T20:16:19.546 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs= 2022-09-12T20:16:19.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs=', 'Content-Length': '0'} 2022-09-12T20:16:19.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed9a5eadad15ec81-00631f9393-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.550 DEBUG:boto:b'jdpiak-75obj311000falseobj312022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:19.551 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:19.551 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:19.552 DEBUG:boto:Method: HEAD 2022-09-12T20:16:19.552 DEBUG:boto:Path: /jdpiak-75/obj31 2022-09-12T20:16:19.553 DEBUG:boto:Data: 2022-09-12T20:16:19.553 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.554 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.554 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.555 DEBUG:boto:Params: {} 2022-09-12T20:16:19.556 DEBUG:boto:Token: None 2022-09-12T20:16:19.556 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj31 2022-09-12T20:16:19.557 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:17/c5Iiq51HDzaZb6HKh4TPLdtY= 2022-09-12T20:16:19.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:17/c5Iiq51HDzaZb6HKh4TPLdtY=', 'Content-Length': '0'} 2022-09-12T20:16:19.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012e5de9c975cc3c5-00631f9393-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.560 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:19.561 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:19.561 DEBUG:boto:Method: HEAD 2022-09-12T20:16:19.562 DEBUG:boto:Path: /jdpiak-75/obj31 2022-09-12T20:16:19.562 DEBUG:boto:Data: 2022-09-12T20:16:19.563 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.564 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.564 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.565 DEBUG:boto:Params: {} 2022-09-12T20:16:19.565 DEBUG:boto:Token: None 2022-09-12T20:16:19.566 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj31 2022-09-12T20:16:19.566 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:17/c5Iiq51HDzaZb6HKh4TPLdtY= 2022-09-12T20:16:19.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:17/c5Iiq51HDzaZb6HKh4TPLdtY=', 'Content-Length': '0'} 2022-09-12T20:16:19.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db90072b2b50c6c7-00631f9393-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-12T20:16:19.570 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:19.571 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:19.572 DEBUG:boto:Method: GET 2022-09-12T20:16:19.572 DEBUG:boto:Path: /jdpiak-75/obj31 2022-09-12T20:16:19.573 DEBUG:boto:Data: 2022-09-12T20:16:19.574 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.574 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.575 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.575 DEBUG:boto:Params: {} 2022-09-12T20:16:19.576 DEBUG:boto:Token: None 2022-09-12T20:16:19.576 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj31 2022-09-12T20:16:19.577 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y9V9WuEnrmwEzWKxkuV/PWY/4do= 2022-09-12T20:16:19.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y9V9WuEnrmwEzWKxkuV/PWY/4do=', 'Content-Length': '0'} 2022-09-12T20:16:19.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a87d8850710d52c9-00631f9393-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.581 DEBUG:boto:path=/jdpiak-75/obj31 2022-09-12T20:16:19.581 DEBUG:boto:auth_path=/jdpiak-75/obj31 2022-09-12T20:16:19.582 DEBUG:boto:Method: GET 2022-09-12T20:16:19.583 DEBUG:boto:Path: /jdpiak-75/obj31 2022-09-12T20:16:19.583 DEBUG:boto:Data: 2022-09-12T20:16:19.584 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.584 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.585 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.585 DEBUG:boto:Params: {} 2022-09-12T20:16:19.586 DEBUG:boto:Token: None 2022-09-12T20:16:19.587 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj31 2022-09-12T20:16:19.587 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y9V9WuEnrmwEzWKxkuV/PWY/4do= 2022-09-12T20:16:19.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y9V9WuEnrmwEzWKxkuV/PWY/4do=', 'Content-Length': '0'} 2022-09-12T20:16:19.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008737b4cd97a1c224-00631f9393-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.591 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:19.591 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:19.592 DEBUG:boto:path=/jdpiak-75/?prefix=obj31 2022-09-12T20:16:19.593 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj31 2022-09-12T20:16:19.593 DEBUG:boto:Method: GET 2022-09-12T20:16:19.594 DEBUG:boto:Path: /jdpiak-75/?prefix=obj31 2022-09-12T20:16:19.594 DEBUG:boto:Data: 2022-09-12T20:16:19.595 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.596 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.596 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.597 DEBUG:boto:Params: {} 2022-09-12T20:16:19.597 DEBUG:boto:Token: None 2022-09-12T20:16:19.598 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/ 2022-09-12T20:16:19.598 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs= 2022-09-12T20:16:19.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs=', 'Content-Length': '0'} 2022-09-12T20:16:19.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d4d73b91a9bbfd4-00631f9393-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.603 DEBUG:boto:b'jdpiak-75obj311000falseobj312022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:19.603 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:19.604 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:19.604 DEBUG:boto:path=/jdpiak-75/?prefix=obj31 2022-09-12T20:16:19.605 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj31 2022-09-12T20:16:19.606 DEBUG:boto:Method: GET 2022-09-12T20:16:19.606 DEBUG:boto:Path: /jdpiak-75/?prefix=obj31 2022-09-12T20:16:19.607 DEBUG:boto:Data: 2022-09-12T20:16:19.607 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.608 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.608 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.609 DEBUG:boto:Params: {} 2022-09-12T20:16:19.609 DEBUG:boto:Token: None 2022-09-12T20:16:19.610 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/ 2022-09-12T20:16:19.610 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs= 2022-09-12T20:16:19.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs=', 'Content-Length': '0'} 2022-09-12T20:16:19.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2c76acf7950772c-00631f9393-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.614 DEBUG:boto:b'jdpiak-75obj311000falseobj312022-09-12T20:12:24.266Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:19.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:16:19.616 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:19.616 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:19.617 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:19.617 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:19.618 DEBUG:boto:Method: GET 2022-09-12T20:16:19.618 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:19.619 DEBUG:boto:Data: 2022-09-12T20:16:19.619 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.620 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.620 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.621 DEBUG:boto:Params: {} 2022-09-12T20:16:19.622 DEBUG:boto:Token: None 2022-09-12T20:16:19.622 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:19.623 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LCJ+OBbYhu/TEU5YO4VN+Ikgvhw= 2022-09-12T20:16:19.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LCJ+OBbYhu/TEU5YO4VN+Ikgvhw=', 'Content-Length': '0'} 2022-09-12T20:16:19.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047e2e7c66432e336-00631f9393-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.627 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:19.627 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:19.628 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:19.628 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:19.629 DEBUG:boto:Method: GET 2022-09-12T20:16:19.629 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:19.630 DEBUG:boto:Data: 2022-09-12T20:16:19.630 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.631 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.631 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.632 DEBUG:boto:Params: {} 2022-09-12T20:16:19.633 DEBUG:boto:Token: None 2022-09-12T20:16:19.633 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:19.634 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LCJ+OBbYhu/TEU5YO4VN+Ikgvhw= 2022-09-12T20:16:19.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LCJ+OBbYhu/TEU5YO4VN+Ikgvhw=', 'Content-Length': '0'} 2022-09-12T20:16:19.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fef5dfe784c3289-00631f9393-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.678 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:19.679 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:19.679 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:19.680 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:19.680 DEBUG:boto:Method: HEAD 2022-09-12T20:16:19.681 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:19.681 DEBUG:boto:Data: 2022-09-12T20:16:19.682 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.682 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.683 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.683 DEBUG:boto:Params: {} 2022-09-12T20:16:19.684 DEBUG:boto:Token: None 2022-09-12T20:16:19.685 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:19.685 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TgMnaCz5UerfG6+hL6wY+2UJwjg= 2022-09-12T20:16:19.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TgMnaCz5UerfG6+hL6wY+2UJwjg=', 'Content-Length': '0'} 2022-09-12T20:16:19.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c89798078ea126e9-00631f9393-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.688 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:19.689 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:19.690 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:19.690 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:19.691 DEBUG:boto:Method: HEAD 2022-09-12T20:16:19.691 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:19.692 DEBUG:boto:Data: 2022-09-12T20:16:19.692 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.692 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.693 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.693 DEBUG:boto:Params: {} 2022-09-12T20:16:19.694 DEBUG:boto:Token: None 2022-09-12T20:16:19.695 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:19.695 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TgMnaCz5UerfG6+hL6wY+2UJwjg= 2022-09-12T20:16:19.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TgMnaCz5UerfG6+hL6wY+2UJwjg=', 'Content-Length': '0'} 2022-09-12T20:16:19.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003f8b225690cb36a-00631f9393-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:16:19.699 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:19.699 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:19.700 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:19.700 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:19.701 DEBUG:boto:Method: GET 2022-09-12T20:16:19.701 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:19.702 DEBUG:boto:Data: 2022-09-12T20:16:19.703 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.703 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.704 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.704 DEBUG:boto:Params: {} 2022-09-12T20:16:19.705 DEBUG:boto:Token: None 2022-09-12T20:16:19.705 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:19.706 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LCJ+OBbYhu/TEU5YO4VN+Ikgvhw= 2022-09-12T20:16:19.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LCJ+OBbYhu/TEU5YO4VN+Ikgvhw=', 'Content-Length': '0'} 2022-09-12T20:16:19.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000849bf512a37fa269-00631f9393-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.709 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:19.710 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:19.711 DEBUG:boto:path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:19.711 DEBUG:boto:auth_path=/jdpiak-75/obj33?versionId=null 2022-09-12T20:16:19.712 DEBUG:boto:Method: GET 2022-09-12T20:16:19.712 DEBUG:boto:Path: /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:19.713 DEBUG:boto:Data: 2022-09-12T20:16:19.713 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.714 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.714 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.715 DEBUG:boto:Params: {} 2022-09-12T20:16:19.715 DEBUG:boto:Token: None 2022-09-12T20:16:19.716 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj33?versionId=null 2022-09-12T20:16:19.717 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:LCJ+OBbYhu/TEU5YO4VN+Ikgvhw= 2022-09-12T20:16:19.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:LCJ+OBbYhu/TEU5YO4VN+Ikgvhw=', 'Content-Length': '0'} 2022-09-12T20:16:19.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048656af5530d9aac-00631f9393-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.720 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:19.721 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:19.721 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:16:19.722 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:16:19.722 DEBUG:boto:Method: GET 2022-09-12T20:16:19.723 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:16:19.723 DEBUG:boto:Data: 2022-09-12T20:16:19.724 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.724 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.725 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.725 DEBUG:boto:Params: {} 2022-09-12T20:16:19.726 DEBUG:boto:Token: None 2022-09-12T20:16:19.726 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/ 2022-09-12T20:16:19.727 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs= 2022-09-12T20:16:19.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs=', 'Content-Length': '0'} 2022-09-12T20:16:19.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a7bcc2be143d51f-00631f9393-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.731 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:19.732 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:19.732 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:19.733 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:16:19.733 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:16:19.734 DEBUG:boto:Method: GET 2022-09-12T20:16:19.734 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:16:19.735 DEBUG:boto:Data: 2022-09-12T20:16:19.735 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.736 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.736 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.737 DEBUG:boto:Params: {} 2022-09-12T20:16:19.738 DEBUG:boto:Token: None 2022-09-12T20:16:19.738 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/ 2022-09-12T20:16:19.739 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs= 2022-09-12T20:16:19.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs=', 'Content-Length': '0'} 2022-09-12T20:16:19.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1f99c4c0d24c54d-00631f9393-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.743 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:19.743 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:19.744 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:19.744 DEBUG:boto:Method: HEAD 2022-09-12T20:16:19.745 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:16:19.745 DEBUG:boto:Data: 2022-09-12T20:16:19.746 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.746 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.747 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.747 DEBUG:boto:Params: {} 2022-09-12T20:16:19.748 DEBUG:boto:Token: None 2022-09-12T20:16:19.748 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj33 2022-09-12T20:16:19.749 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A+FQ+ZI3vhmauB23kDvqXgMI30M= 2022-09-12T20:16:19.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A+FQ+ZI3vhmauB23kDvqXgMI30M=', 'Content-Length': '0'} 2022-09-12T20:16:19.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bda149aa595a406-00631f9393-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.752 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:19.753 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:19.753 DEBUG:boto:Method: HEAD 2022-09-12T20:16:19.754 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:16:19.754 DEBUG:boto:Data: 2022-09-12T20:16:19.755 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.755 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.756 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.756 DEBUG:boto:Params: {} 2022-09-12T20:16:19.757 DEBUG:boto:Token: None 2022-09-12T20:16:19.758 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj33 2022-09-12T20:16:19.758 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A+FQ+ZI3vhmauB23kDvqXgMI30M= 2022-09-12T20:16:19.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A+FQ+ZI3vhmauB23kDvqXgMI30M=', 'Content-Length': '0'} 2022-09-12T20:16:19.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060916f6ad60f5f94-00631f9393-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:16:19.762 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:19.762 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:19.763 DEBUG:boto:Method: GET 2022-09-12T20:16:19.764 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:16:19.764 DEBUG:boto:Data: 2022-09-12T20:16:19.765 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.765 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.766 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.766 DEBUG:boto:Params: {} 2022-09-12T20:16:19.767 DEBUG:boto:Token: None 2022-09-12T20:16:19.767 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj33 2022-09-12T20:16:19.768 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9xXjpCS2VqpHKjR96P74VbPd0bM= 2022-09-12T20:16:19.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9xXjpCS2VqpHKjR96P74VbPd0bM=', 'Content-Length': '0'} 2022-09-12T20:16:19.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8e008f2f559a9a7-00631f9393-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.814 DEBUG:boto:path=/jdpiak-75/obj33 2022-09-12T20:16:19.815 DEBUG:boto:auth_path=/jdpiak-75/obj33 2022-09-12T20:16:19.815 DEBUG:boto:Method: GET 2022-09-12T20:16:19.816 DEBUG:boto:Path: /jdpiak-75/obj33 2022-09-12T20:16:19.817 DEBUG:boto:Data: 2022-09-12T20:16:19.817 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.818 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.818 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.819 DEBUG:boto:Params: {} 2022-09-12T20:16:19.819 DEBUG:boto:Token: None 2022-09-12T20:16:19.820 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj33 2022-09-12T20:16:19.821 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9xXjpCS2VqpHKjR96P74VbPd0bM= 2022-09-12T20:16:19.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9xXjpCS2VqpHKjR96P74VbPd0bM=', 'Content-Length': '0'} 2022-09-12T20:16:19.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057ed0612b4a0db47-00631f9393-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.825 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:19.825 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:19.826 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:16:19.826 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:16:19.827 DEBUG:boto:Method: GET 2022-09-12T20:16:19.827 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:16:19.828 DEBUG:boto:Data: 2022-09-12T20:16:19.828 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.829 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.829 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.830 DEBUG:boto:Params: {} 2022-09-12T20:16:19.831 DEBUG:boto:Token: None 2022-09-12T20:16:19.831 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/ 2022-09-12T20:16:19.832 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs= 2022-09-12T20:16:19.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs=', 'Content-Length': '0'} 2022-09-12T20:16:19.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000015c98a246df47df-00631f9393-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.836 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:19.837 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:19.838 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:19.838 DEBUG:boto:path=/jdpiak-75/?prefix=obj33 2022-09-12T20:16:19.839 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj33 2022-09-12T20:16:19.839 DEBUG:boto:Method: GET 2022-09-12T20:16:19.840 DEBUG:boto:Path: /jdpiak-75/?prefix=obj33 2022-09-12T20:16:19.840 DEBUG:boto:Data: 2022-09-12T20:16:19.841 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.841 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.842 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.843 DEBUG:boto:Params: {} 2022-09-12T20:16:19.843 DEBUG:boto:Token: None 2022-09-12T20:16:19.844 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/ 2022-09-12T20:16:19.844 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs= 2022-09-12T20:16:19.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E7rmRlaEfdrCqp301wAOKsM+9Cs=', 'Content-Length': '0'} 2022-09-12T20:16:19.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b3faa10946e49ed-00631f9393-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.848 DEBUG:boto:b'jdpiak-75obj331000falseobj332022-09-12T20:03:02.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:19.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:16:19.850 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:19.851 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:19.852 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:19.852 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:19.853 DEBUG:boto:Method: GET 2022-09-12T20:16:19.853 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:19.854 DEBUG:boto:Data: 2022-09-12T20:16:19.854 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.855 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.855 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.856 DEBUG:boto:Params: {} 2022-09-12T20:16:19.856 DEBUG:boto:Token: None 2022-09-12T20:16:19.857 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:19.858 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HuQRMwmm8bveUgcWevHm2wDKyNY= 2022-09-12T20:16:19.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HuQRMwmm8bveUgcWevHm2wDKyNY=', 'Content-Length': '0'} 2022-09-12T20:16:19.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8a641d013bfd46d-00631f9393-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.861 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:19.862 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:19.863 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:19.863 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:19.864 DEBUG:boto:Method: GET 2022-09-12T20:16:19.865 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:19.865 DEBUG:boto:Data: 2022-09-12T20:16:19.866 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.866 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.867 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.867 DEBUG:boto:Params: {} 2022-09-12T20:16:19.868 DEBUG:boto:Token: None 2022-09-12T20:16:19.869 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:19.869 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HuQRMwmm8bveUgcWevHm2wDKyNY= 2022-09-12T20:16:19.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HuQRMwmm8bveUgcWevHm2wDKyNY=', 'Content-Length': '0'} 2022-09-12T20:16:19.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c892cbff18d5d2ef-00631f9393-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.873 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:19.873 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:19.874 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:19.874 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:19.875 DEBUG:boto:Method: HEAD 2022-09-12T20:16:19.875 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:19.876 DEBUG:boto:Data: 2022-09-12T20:16:19.876 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.877 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.877 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.878 DEBUG:boto:Params: {} 2022-09-12T20:16:19.878 DEBUG:boto:Token: None 2022-09-12T20:16:19.879 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:19.879 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H6b5/oMDDSzS850UWzeAuYSEHk4= 2022-09-12T20:16:19.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H6b5/oMDDSzS850UWzeAuYSEHk4=', 'Content-Length': '0'} 2022-09-12T20:16:19.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2b71556d8db7b3f-00631f9393-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.883 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:19.883 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:19.884 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:19.884 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:19.885 DEBUG:boto:Method: HEAD 2022-09-12T20:16:19.885 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:19.886 DEBUG:boto:Data: 2022-09-12T20:16:19.886 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.887 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.887 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.888 DEBUG:boto:Params: {} 2022-09-12T20:16:19.888 DEBUG:boto:Token: None 2022-09-12T20:16:19.889 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:19.889 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:H6b5/oMDDSzS850UWzeAuYSEHk4= 2022-09-12T20:16:19.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:H6b5/oMDDSzS850UWzeAuYSEHk4=', 'Content-Length': '0'} 2022-09-12T20:16:19.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088f8aa97572eecaf-00631f9393-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:16:19.894 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:19.894 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:19.895 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:19.895 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:19.896 DEBUG:boto:Method: GET 2022-09-12T20:16:19.896 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:19.897 DEBUG:boto:Data: 2022-09-12T20:16:19.897 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.898 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:19.898 DEBUG:boto:Port: 8001 2022-09-12T20:16:19.899 DEBUG:boto:Params: {} 2022-09-12T20:16:19.900 DEBUG:boto:Token: None 2022-09-12T20:16:19.900 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:19.901 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HuQRMwmm8bveUgcWevHm2wDKyNY= 2022-09-12T20:16:19.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HuQRMwmm8bveUgcWevHm2wDKyNY=', 'Content-Length': '0'} 2022-09-12T20:16:19.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba96b8300fd49a74-00631f9393-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.946 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:19.947 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:19.948 DEBUG:boto:path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:19.948 DEBUG:boto:auth_path=/jdpiak-75/obj34?versionId=null 2022-09-12T20:16:19.949 DEBUG:boto:Method: GET 2022-09-12T20:16:19.949 DEBUG:boto:Path: /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:19.950 DEBUG:boto:Data: 2022-09-12T20:16:19.950 DEBUG:boto:Headers: {} 2022-09-12T20:16:19.951 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:19.951 DEBUG:boto:Port: 8000 2022-09-12T20:16:19.952 DEBUG:boto:Params: {} 2022-09-12T20:16:19.953 DEBUG:boto:Token: None 2022-09-12T20:16:19.953 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:19 GMT /jdpiak-75/obj34?versionId=null 2022-09-12T20:16:19.954 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HuQRMwmm8bveUgcWevHm2wDKyNY= 2022-09-12T20:16:19.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HuQRMwmm8bveUgcWevHm2wDKyNY=', 'Content-Length': '0'} 2022-09-12T20:16:19.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da3e5fcc01108551-00631f9393-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:19.958 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:19.958 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:19.959 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:16:19.959 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:16:19.960 DEBUG:boto:Method: GET 2022-09-12T20:16:19.961 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:16:19.961 DEBUG:boto:Data: 2022-09-12T20:16:20.014 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.015 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:20.016 DEBUG:boto:Port: 8001 2022-09-12T20:16:20.017 DEBUG:boto:Params: {} 2022-09-12T20:16:20.018 DEBUG:boto:Token: None 2022-09-12T20:16:20.019 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/ 2022-09-12T20:16:20.021 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY= 2022-09-12T20:16:20.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY=', 'Content-Length': '0'} 2022-09-12T20:16:20.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd7bb4bf13430d68-00631f9394-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.029 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:20.031 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:20.032 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:20.033 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:16:20.033 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:16:20.034 DEBUG:boto:Method: GET 2022-09-12T20:16:20.035 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:16:20.036 DEBUG:boto:Data: 2022-09-12T20:16:20.037 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.038 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:20.039 DEBUG:boto:Port: 8000 2022-09-12T20:16:20.040 DEBUG:boto:Params: {} 2022-09-12T20:16:20.041 DEBUG:boto:Token: None 2022-09-12T20:16:20.043 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/ 2022-09-12T20:16:20.044 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY= 2022-09-12T20:16:20.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY=', 'Content-Length': '0'} 2022-09-12T20:16:20.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1174ed5be1760c3-00631f9394-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.054 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:20.056 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:20.056 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:20.057 DEBUG:boto:Method: HEAD 2022-09-12T20:16:20.057 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:16:20.058 DEBUG:boto:Data: 2022-09-12T20:16:20.059 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.059 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:20.060 DEBUG:boto:Port: 8001 2022-09-12T20:16:20.060 DEBUG:boto:Params: {} 2022-09-12T20:16:20.061 DEBUG:boto:Token: None 2022-09-12T20:16:20.062 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj34 2022-09-12T20:16:20.063 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qz/3Aw0E+k7CbMgbj7jOt6r7fQ8= 2022-09-12T20:16:20.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qz/3Aw0E+k7CbMgbj7jOt6r7fQ8=', 'Content-Length': '0'} 2022-09-12T20:16:20.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000771ccf122c4c1545-00631f9394-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.072 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:20.072 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:20.073 DEBUG:boto:Method: HEAD 2022-09-12T20:16:20.073 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:16:20.074 DEBUG:boto:Data: 2022-09-12T20:16:20.075 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.075 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:20.076 DEBUG:boto:Port: 8000 2022-09-12T20:16:20.076 DEBUG:boto:Params: {} 2022-09-12T20:16:20.077 DEBUG:boto:Token: None 2022-09-12T20:16:20.077 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj34 2022-09-12T20:16:20.078 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qz/3Aw0E+k7CbMgbj7jOt6r7fQ8= 2022-09-12T20:16:20.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qz/3Aw0E+k7CbMgbj7jOt6r7fQ8=', 'Content-Length': '0'} 2022-09-12T20:16:20.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098ee111787925fb3-00631f9394-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-12T20:16:20.083 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:20.084 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:20.084 DEBUG:boto:Method: GET 2022-09-12T20:16:20.085 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:16:20.085 DEBUG:boto:Data: 2022-09-12T20:16:20.086 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.087 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:20.087 DEBUG:boto:Port: 8001 2022-09-12T20:16:20.088 DEBUG:boto:Params: {} 2022-09-12T20:16:20.089 DEBUG:boto:Token: None 2022-09-12T20:16:20.089 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj34 2022-09-12T20:16:20.090 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gtkupg2fY/Ypp/OE3Gtvq1m8Yfg= 2022-09-12T20:16:20.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gtkupg2fY/Ypp/OE3Gtvq1m8Yfg=', 'Content-Length': '0'} 2022-09-12T20:16:20.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000904ade450563ee7e-00631f9394-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.103 DEBUG:boto:path=/jdpiak-75/obj34 2022-09-12T20:16:20.104 DEBUG:boto:auth_path=/jdpiak-75/obj34 2022-09-12T20:16:20.104 DEBUG:boto:Method: GET 2022-09-12T20:16:20.105 DEBUG:boto:Path: /jdpiak-75/obj34 2022-09-12T20:16:20.106 DEBUG:boto:Data: 2022-09-12T20:16:20.107 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.107 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:20.108 DEBUG:boto:Port: 8000 2022-09-12T20:16:20.108 DEBUG:boto:Params: {} 2022-09-12T20:16:20.109 DEBUG:boto:Token: None 2022-09-12T20:16:20.110 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj34 2022-09-12T20:16:20.110 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gtkupg2fY/Ypp/OE3Gtvq1m8Yfg= 2022-09-12T20:16:20.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gtkupg2fY/Ypp/OE3Gtvq1m8Yfg=', 'Content-Length': '0'} 2022-09-12T20:16:20.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f13096939531d1d-00631f9394-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.159 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:20.159 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:20.160 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:16:20.161 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:16:20.161 DEBUG:boto:Method: GET 2022-09-12T20:16:20.162 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:16:20.163 DEBUG:boto:Data: 2022-09-12T20:16:20.163 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.164 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:20.164 DEBUG:boto:Port: 8001 2022-09-12T20:16:20.165 DEBUG:boto:Params: {} 2022-09-12T20:16:20.166 DEBUG:boto:Token: None 2022-09-12T20:16:20.167 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/ 2022-09-12T20:16:20.168 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY= 2022-09-12T20:16:20.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY=', 'Content-Length': '0'} 2022-09-12T20:16:20.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aaedd0719a6b11d3-00631f9394-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.179 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:20.181 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:20.182 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:20.183 DEBUG:boto:path=/jdpiak-75/?prefix=obj34 2022-09-12T20:16:20.184 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj34 2022-09-12T20:16:20.185 DEBUG:boto:Method: GET 2022-09-12T20:16:20.186 DEBUG:boto:Path: /jdpiak-75/?prefix=obj34 2022-09-12T20:16:20.187 DEBUG:boto:Data: 2022-09-12T20:16:20.189 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.191 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:20.193 DEBUG:boto:Port: 8000 2022-09-12T20:16:20.193 DEBUG:boto:Params: {} 2022-09-12T20:16:20.194 DEBUG:boto:Token: None 2022-09-12T20:16:20.195 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/ 2022-09-12T20:16:20.196 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY= 2022-09-12T20:16:20.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY=', 'Content-Length': '0'} 2022-09-12T20:16:20.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa1d30d4bc70ef42-00631f9394-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.201 DEBUG:boto:b'jdpiak-75obj341000falseobj342022-09-12T20:03:28.524Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:20.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2022-09-12T20:16:20.203 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:20.204 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:20.205 DEBUG:boto:path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:20.206 DEBUG:boto:auth_path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:20.206 DEBUG:boto:Method: GET 2022-09-12T20:16:20.207 DEBUG:boto:Path: /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:20.208 DEBUG:boto:Data: 2022-09-12T20:16:20.209 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.210 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:20.210 DEBUG:boto:Port: 8001 2022-09-12T20:16:20.211 DEBUG:boto:Params: {} 2022-09-12T20:16:20.212 DEBUG:boto:Token: None 2022-09-12T20:16:20.213 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:20.214 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FHMWn774sEFo2e5odST3A2Uw86I= 2022-09-12T20:16:20.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FHMWn774sEFo2e5odST3A2Uw86I=', 'Content-Length': '0'} 2022-09-12T20:16:20.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007be86258bb81c011-00631f9394-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.261 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:20.262 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:20.263 DEBUG:boto:path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:20.263 DEBUG:boto:auth_path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:20.264 DEBUG:boto:Method: GET 2022-09-12T20:16:20.264 DEBUG:boto:Path: /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:20.265 DEBUG:boto:Data: 2022-09-12T20:16:20.266 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.266 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:20.267 DEBUG:boto:Port: 8000 2022-09-12T20:16:20.267 DEBUG:boto:Params: {} 2022-09-12T20:16:20.268 DEBUG:boto:Token: None 2022-09-12T20:16:20.269 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:20.270 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FHMWn774sEFo2e5odST3A2Uw86I= 2022-09-12T20:16:20.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FHMWn774sEFo2e5odST3A2Uw86I=', 'Content-Length': '0'} 2022-09-12T20:16:20.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd215f843f8a3f95-00631f9394-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.276 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:20.277 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:20.277 DEBUG:boto:path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:20.278 DEBUG:boto:auth_path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:20.279 DEBUG:boto:Method: HEAD 2022-09-12T20:16:20.280 DEBUG:boto:Path: /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:20.282 DEBUG:boto:Data: 2022-09-12T20:16:20.283 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.284 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:20.285 DEBUG:boto:Port: 8001 2022-09-12T20:16:20.286 DEBUG:boto:Params: {} 2022-09-12T20:16:20.287 DEBUG:boto:Token: None 2022-09-12T20:16:20.288 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:20.289 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sE9ZcGPp6LFHHTDH/t840Z8ZtPY= 2022-09-12T20:16:20.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sE9ZcGPp6LFHHTDH/t840Z8ZtPY=', 'Content-Length': '0'} 2022-09-12T20:16:20.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ada142b335c1a007-00631f9394-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.295 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:20.296 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:20.297 DEBUG:boto:path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:20.298 DEBUG:boto:auth_path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:20.298 DEBUG:boto:Method: HEAD 2022-09-12T20:16:20.299 DEBUG:boto:Path: /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:20.299 DEBUG:boto:Data: 2022-09-12T20:16:20.300 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.301 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:20.301 DEBUG:boto:Port: 8000 2022-09-12T20:16:20.302 DEBUG:boto:Params: {} 2022-09-12T20:16:20.303 DEBUG:boto:Token: None 2022-09-12T20:16:20.303 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:20.304 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sE9ZcGPp6LFHHTDH/t840Z8ZtPY= 2022-09-12T20:16:20.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sE9ZcGPp6LFHHTDH/t840Z8ZtPY=', 'Content-Length': '0'} 2022-09-12T20:16:20.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000085a307590de6cae-00631f9394-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2022-09-12T20:16:20.314 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:20.315 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:20.315 DEBUG:boto:path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:20.316 DEBUG:boto:auth_path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:20.317 DEBUG:boto:Method: GET 2022-09-12T20:16:20.317 DEBUG:boto:Path: /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:20.318 DEBUG:boto:Data: 2022-09-12T20:16:20.318 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.319 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:20.319 DEBUG:boto:Port: 8001 2022-09-12T20:16:20.320 DEBUG:boto:Params: {} 2022-09-12T20:16:20.321 DEBUG:boto:Token: None 2022-09-12T20:16:20.321 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:20.322 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FHMWn774sEFo2e5odST3A2Uw86I= 2022-09-12T20:16:20.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FHMWn774sEFo2e5odST3A2Uw86I=', 'Content-Length': '0'} 2022-09-12T20:16:20.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7f32d1b7c62d5e9-00631f9394-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.329 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:20.329 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:20.330 DEBUG:boto:path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:20.330 DEBUG:boto:auth_path=/jdpiak-75/obj37?versionId=null 2022-09-12T20:16:20.331 DEBUG:boto:Method: GET 2022-09-12T20:16:20.332 DEBUG:boto:Path: /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:20.332 DEBUG:boto:Data: 2022-09-12T20:16:20.333 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.334 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:20.334 DEBUG:boto:Port: 8000 2022-09-12T20:16:20.335 DEBUG:boto:Params: {} 2022-09-12T20:16:20.335 DEBUG:boto:Token: None 2022-09-12T20:16:20.336 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj37?versionId=null 2022-09-12T20:16:20.337 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FHMWn774sEFo2e5odST3A2Uw86I= 2022-09-12T20:16:20.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FHMWn774sEFo2e5odST3A2Uw86I=', 'Content-Length': '0'} 2022-09-12T20:16:20.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8189b4d8e98fb0c-00631f9394-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.347 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:20.348 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:20.348 DEBUG:boto:path=/jdpiak-75/?prefix=obj37 2022-09-12T20:16:20.349 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj37 2022-09-12T20:16:20.350 DEBUG:boto:Method: GET 2022-09-12T20:16:20.350 DEBUG:boto:Path: /jdpiak-75/?prefix=obj37 2022-09-12T20:16:20.351 DEBUG:boto:Data: 2022-09-12T20:16:20.352 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.352 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:20.353 DEBUG:boto:Port: 8001 2022-09-12T20:16:20.353 DEBUG:boto:Params: {} 2022-09-12T20:16:20.354 DEBUG:boto:Token: None 2022-09-12T20:16:20.355 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/ 2022-09-12T20:16:20.356 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY= 2022-09-12T20:16:20.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY=', 'Content-Length': '0'} 2022-09-12T20:16:20.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bf6e4ce8c75862d-00631f9394-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.364 DEBUG:boto:b'jdpiak-75obj371000falseobj372022-09-12T20:14:26.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:20.365 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:20.366 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:20.366 DEBUG:boto:path=/jdpiak-75/?prefix=obj37 2022-09-12T20:16:20.367 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj37 2022-09-12T20:16:20.368 DEBUG:boto:Method: GET 2022-09-12T20:16:20.368 DEBUG:boto:Path: /jdpiak-75/?prefix=obj37 2022-09-12T20:16:20.369 DEBUG:boto:Data: 2022-09-12T20:16:20.369 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.370 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:20.370 DEBUG:boto:Port: 8000 2022-09-12T20:16:20.371 DEBUG:boto:Params: {} 2022-09-12T20:16:20.371 DEBUG:boto:Token: None 2022-09-12T20:16:20.372 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/ 2022-09-12T20:16:20.373 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY= 2022-09-12T20:16:20.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY=', 'Content-Length': '0'} 2022-09-12T20:16:20.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000128cb986df9e0ab3-00631f9394-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.426 DEBUG:boto:b'jdpiak-75obj371000falseobj372022-09-12T20:14:26.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:20.427 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:20.428 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:20.428 DEBUG:boto:Method: HEAD 2022-09-12T20:16:20.429 DEBUG:boto:Path: /jdpiak-75/obj37 2022-09-12T20:16:20.429 DEBUG:boto:Data: 2022-09-12T20:16:20.430 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.431 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:20.431 DEBUG:boto:Port: 8001 2022-09-12T20:16:20.432 DEBUG:boto:Params: {} 2022-09-12T20:16:20.433 DEBUG:boto:Token: None 2022-09-12T20:16:20.433 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj37 2022-09-12T20:16:20.434 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qUQ39xSi1tYr+85+1SU8lpsLwU8= 2022-09-12T20:16:20.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qUQ39xSi1tYr+85+1SU8lpsLwU8=', 'Content-Length': '0'} 2022-09-12T20:16:20.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be14ee06a7dd3330-00631f9394-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.439 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:20.440 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:20.440 DEBUG:boto:Method: HEAD 2022-09-12T20:16:20.441 DEBUG:boto:Path: /jdpiak-75/obj37 2022-09-12T20:16:20.442 DEBUG:boto:Data: 2022-09-12T20:16:20.442 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.443 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:20.444 DEBUG:boto:Port: 8000 2022-09-12T20:16:20.445 DEBUG:boto:Params: {} 2022-09-12T20:16:20.445 DEBUG:boto:Token: None 2022-09-12T20:16:20.446 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj37 2022-09-12T20:16:20.447 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qUQ39xSi1tYr+85+1SU8lpsLwU8= 2022-09-12T20:16:20.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qUQ39xSi1tYr+85+1SU8lpsLwU8=', 'Content-Length': '0'} 2022-09-12T20:16:20.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be8d028ebd14bafb-00631f9394-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2022-09-12T20:16:20.452 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:20.453 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:20.453 DEBUG:boto:Method: GET 2022-09-12T20:16:20.454 DEBUG:boto:Path: /jdpiak-75/obj37 2022-09-12T20:16:20.454 DEBUG:boto:Data: 2022-09-12T20:16:20.455 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.456 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:20.456 DEBUG:boto:Port: 8001 2022-09-12T20:16:20.457 DEBUG:boto:Params: {} 2022-09-12T20:16:20.458 DEBUG:boto:Token: None 2022-09-12T20:16:20.458 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj37 2022-09-12T20:16:20.459 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lUwTRwIF9wED5ZytzqIhrl7fmrs= 2022-09-12T20:16:20.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lUwTRwIF9wED5ZytzqIhrl7fmrs=', 'Content-Length': '0'} 2022-09-12T20:16:20.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009932aa8935ed9ad-00631f9394-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.464 DEBUG:boto:path=/jdpiak-75/obj37 2022-09-12T20:16:20.464 DEBUG:boto:auth_path=/jdpiak-75/obj37 2022-09-12T20:16:20.465 DEBUG:boto:Method: GET 2022-09-12T20:16:20.466 DEBUG:boto:Path: /jdpiak-75/obj37 2022-09-12T20:16:20.466 DEBUG:boto:Data: 2022-09-12T20:16:20.467 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.468 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:20.468 DEBUG:boto:Port: 8000 2022-09-12T20:16:20.469 DEBUG:boto:Params: {} 2022-09-12T20:16:20.469 DEBUG:boto:Token: None 2022-09-12T20:16:20.470 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj37 2022-09-12T20:16:20.471 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lUwTRwIF9wED5ZytzqIhrl7fmrs= 2022-09-12T20:16:20.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lUwTRwIF9wED5ZytzqIhrl7fmrs=', 'Content-Length': '0'} 2022-09-12T20:16:20.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000081b87edee845ec3-00631f9394-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.475 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:20.476 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:20.476 DEBUG:boto:path=/jdpiak-75/?prefix=obj37 2022-09-12T20:16:20.477 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj37 2022-09-12T20:16:20.477 DEBUG:boto:Method: GET 2022-09-12T20:16:20.478 DEBUG:boto:Path: /jdpiak-75/?prefix=obj37 2022-09-12T20:16:20.478 DEBUG:boto:Data: 2022-09-12T20:16:20.479 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.480 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:20.480 DEBUG:boto:Port: 8001 2022-09-12T20:16:20.481 DEBUG:boto:Params: {} 2022-09-12T20:16:20.482 DEBUG:boto:Token: None 2022-09-12T20:16:20.482 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/ 2022-09-12T20:16:20.483 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY= 2022-09-12T20:16:20.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY=', 'Content-Length': '0'} 2022-09-12T20:16:20.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d449e4fd2d8d125f-00631f9394-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.530 DEBUG:boto:b'jdpiak-75obj371000falseobj372022-09-12T20:14:26.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:20.531 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:20.532 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:20.532 DEBUG:boto:path=/jdpiak-75/?prefix=obj37 2022-09-12T20:16:20.533 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj37 2022-09-12T20:16:20.533 DEBUG:boto:Method: GET 2022-09-12T20:16:20.534 DEBUG:boto:Path: /jdpiak-75/?prefix=obj37 2022-09-12T20:16:20.535 DEBUG:boto:Data: 2022-09-12T20:16:20.535 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.536 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:20.536 DEBUG:boto:Port: 8000 2022-09-12T20:16:20.537 DEBUG:boto:Params: {} 2022-09-12T20:16:20.538 DEBUG:boto:Token: None 2022-09-12T20:16:20.538 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/ 2022-09-12T20:16:20.539 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY= 2022-09-12T20:16:20.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY=', 'Content-Length': '0'} 2022-09-12T20:16:20.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f96042fe581f6aff-00631f9394-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.543 DEBUG:boto:b'jdpiak-75obj371000falseobj372022-09-12T20:14:26.981Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:20.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:16:20.545 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:20.546 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:20.546 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:20.547 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:20.548 DEBUG:boto:Method: GET 2022-09-12T20:16:20.548 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:20.549 DEBUG:boto:Data: 2022-09-12T20:16:20.549 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.550 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:20.551 DEBUG:boto:Port: 8001 2022-09-12T20:16:20.551 DEBUG:boto:Params: {} 2022-09-12T20:16:20.552 DEBUG:boto:Token: None 2022-09-12T20:16:20.553 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:20.553 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:d5StDjq2sIbOFMTTpACUb7SeYFM= 2022-09-12T20:16:20.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:d5StDjq2sIbOFMTTpACUb7SeYFM=', 'Content-Length': '0'} 2022-09-12T20:16:20.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6629910acd6ba2f-00631f9394-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.557 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:20.558 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:20.559 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:20.560 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:20.561 DEBUG:boto:Method: GET 2022-09-12T20:16:20.561 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:20.562 DEBUG:boto:Data: 2022-09-12T20:16:20.562 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.563 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:20.564 DEBUG:boto:Port: 8000 2022-09-12T20:16:20.564 DEBUG:boto:Params: {} 2022-09-12T20:16:20.565 DEBUG:boto:Token: None 2022-09-12T20:16:20.566 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:20.567 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:d5StDjq2sIbOFMTTpACUb7SeYFM= 2022-09-12T20:16:20.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:d5StDjq2sIbOFMTTpACUb7SeYFM=', 'Content-Length': '0'} 2022-09-12T20:16:20.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e626c0315b905f5f-00631f9394-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.610 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:20.611 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:20.611 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:20.612 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:20.612 DEBUG:boto:Method: HEAD 2022-09-12T20:16:20.613 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:20.613 DEBUG:boto:Data: 2022-09-12T20:16:20.614 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.615 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:20.615 DEBUG:boto:Port: 8001 2022-09-12T20:16:20.616 DEBUG:boto:Params: {} 2022-09-12T20:16:20.617 DEBUG:boto:Token: None 2022-09-12T20:16:20.618 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:20.618 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p1KJFbRfGr+DQw4j55q8nrp+y2M= 2022-09-12T20:16:20.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p1KJFbRfGr+DQw4j55q8nrp+y2M=', 'Content-Length': '0'} 2022-09-12T20:16:20.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000648f5b9e6d54050f-00631f9394-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.623 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:20.623 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:20.624 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:20.624 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:20.625 DEBUG:boto:Method: HEAD 2022-09-12T20:16:20.626 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:20.626 DEBUG:boto:Data: 2022-09-12T20:16:20.627 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.628 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:20.628 DEBUG:boto:Port: 8000 2022-09-12T20:16:20.629 DEBUG:boto:Params: {} 2022-09-12T20:16:20.630 DEBUG:boto:Token: None 2022-09-12T20:16:20.630 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:20.631 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p1KJFbRfGr+DQw4j55q8nrp+y2M= 2022-09-12T20:16:20.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p1KJFbRfGr+DQw4j55q8nrp+y2M=', 'Content-Length': '0'} 2022-09-12T20:16:20.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000602c76ad0d967fe6-00631f9394-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:16:20.635 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:20.636 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:20.637 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:20.637 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:20.638 DEBUG:boto:Method: GET 2022-09-12T20:16:20.639 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:20.639 DEBUG:boto:Data: 2022-09-12T20:16:20.640 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.640 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:20.641 DEBUG:boto:Port: 8001 2022-09-12T20:16:20.643 DEBUG:boto:Params: {} 2022-09-12T20:16:20.644 DEBUG:boto:Token: None 2022-09-12T20:16:20.645 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:20.646 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:d5StDjq2sIbOFMTTpACUb7SeYFM= 2022-09-12T20:16:20.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:d5StDjq2sIbOFMTTpACUb7SeYFM=', 'Content-Length': '0'} 2022-09-12T20:16:20.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f715ac0f035338ea-00631f9394-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.650 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:20.651 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:20.652 DEBUG:boto:path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:20.652 DEBUG:boto:auth_path=/jdpiak-75/obj38?versionId=null 2022-09-12T20:16:20.653 DEBUG:boto:Method: GET 2022-09-12T20:16:20.654 DEBUG:boto:Path: /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:20.654 DEBUG:boto:Data: 2022-09-12T20:16:20.655 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.655 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:20.656 DEBUG:boto:Port: 8000 2022-09-12T20:16:20.657 DEBUG:boto:Params: {} 2022-09-12T20:16:20.657 DEBUG:boto:Token: None 2022-09-12T20:16:20.658 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj38?versionId=null 2022-09-12T20:16:20.659 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:d5StDjq2sIbOFMTTpACUb7SeYFM= 2022-09-12T20:16:20.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:d5StDjq2sIbOFMTTpACUb7SeYFM=', 'Content-Length': '0'} 2022-09-12T20:16:20.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c92524a826a44c4-00631f9394-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.663 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:20.664 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:20.664 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:16:20.665 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:16:20.666 DEBUG:boto:Method: GET 2022-09-12T20:16:20.666 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:16:20.667 DEBUG:boto:Data: 2022-09-12T20:16:20.668 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.668 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:20.669 DEBUG:boto:Port: 8001 2022-09-12T20:16:20.670 DEBUG:boto:Params: {} 2022-09-12T20:16:20.670 DEBUG:boto:Token: None 2022-09-12T20:16:20.671 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/ 2022-09-12T20:16:20.672 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY= 2022-09-12T20:16:20.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY=', 'Content-Length': '0'} 2022-09-12T20:16:20.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc3d7bcd0023123e-00631f9394-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.676 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:20.677 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:20.678 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:20.679 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:16:20.679 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:16:20.680 DEBUG:boto:Method: GET 2022-09-12T20:16:20.681 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:16:20.681 DEBUG:boto:Data: 2022-09-12T20:16:20.682 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.683 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:20.683 DEBUG:boto:Port: 8000 2022-09-12T20:16:20.684 DEBUG:boto:Params: {} 2022-09-12T20:16:20.685 DEBUG:boto:Token: None 2022-09-12T20:16:20.686 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/ 2022-09-12T20:16:20.686 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY= 2022-09-12T20:16:20.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY=', 'Content-Length': '0'} 2022-09-12T20:16:20.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005493e1441f064701-00631f9394-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.691 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:20.691 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:20.692 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:20.692 DEBUG:boto:Method: HEAD 2022-09-12T20:16:20.693 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:16:20.693 DEBUG:boto:Data: 2022-09-12T20:16:20.694 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.695 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:20.695 DEBUG:boto:Port: 8001 2022-09-12T20:16:20.696 DEBUG:boto:Params: {} 2022-09-12T20:16:20.696 DEBUG:boto:Token: None 2022-09-12T20:16:20.697 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj38 2022-09-12T20:16:20.697 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RjXUopk7yOEp3kto9f08nKpmaz8= 2022-09-12T20:16:20.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RjXUopk7yOEp3kto9f08nKpmaz8=', 'Content-Length': '0'} 2022-09-12T20:16:20.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3fefcdf631deadd-00631f9394-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.701 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:20.701 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:20.702 DEBUG:boto:Method: HEAD 2022-09-12T20:16:20.703 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:16:20.703 DEBUG:boto:Data: 2022-09-12T20:16:20.704 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.704 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:20.705 DEBUG:boto:Port: 8000 2022-09-12T20:16:20.705 DEBUG:boto:Params: {} 2022-09-12T20:16:20.706 DEBUG:boto:Token: None 2022-09-12T20:16:20.706 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj38 2022-09-12T20:16:20.707 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RjXUopk7yOEp3kto9f08nKpmaz8= 2022-09-12T20:16:20.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RjXUopk7yOEp3kto9f08nKpmaz8=', 'Content-Length': '0'} 2022-09-12T20:16:20.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019c511ab4b023a28-00631f9394-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-12T20:16:20.711 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:20.711 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:20.712 DEBUG:boto:Method: GET 2022-09-12T20:16:20.713 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:16:20.713 DEBUG:boto:Data: 2022-09-12T20:16:20.714 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.715 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:20.715 DEBUG:boto:Port: 8001 2022-09-12T20:16:20.716 DEBUG:boto:Params: {} 2022-09-12T20:16:20.716 DEBUG:boto:Token: None 2022-09-12T20:16:20.717 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj38 2022-09-12T20:16:20.717 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0nOcTE4lXX1Dg8t6+d/ZiFduuj8= 2022-09-12T20:16:20.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0nOcTE4lXX1Dg8t6+d/ZiFduuj8=', 'Content-Length': '0'} 2022-09-12T20:16:20.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089adf590d0f44608-00631f9394-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.721 DEBUG:boto:path=/jdpiak-75/obj38 2022-09-12T20:16:20.722 DEBUG:boto:auth_path=/jdpiak-75/obj38 2022-09-12T20:16:20.722 DEBUG:boto:Method: GET 2022-09-12T20:16:20.723 DEBUG:boto:Path: /jdpiak-75/obj38 2022-09-12T20:16:20.724 DEBUG:boto:Data: 2022-09-12T20:16:20.724 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.725 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:20.725 DEBUG:boto:Port: 8000 2022-09-12T20:16:20.726 DEBUG:boto:Params: {} 2022-09-12T20:16:20.726 DEBUG:boto:Token: None 2022-09-12T20:16:20.727 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj38 2022-09-12T20:16:20.728 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0nOcTE4lXX1Dg8t6+d/ZiFduuj8= 2022-09-12T20:16:20.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0nOcTE4lXX1Dg8t6+d/ZiFduuj8=', 'Content-Length': '0'} 2022-09-12T20:16:20.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0b819832a481ae9-00631f9394-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.774 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:20.775 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:20.776 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:16:20.776 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:16:20.777 DEBUG:boto:Method: GET 2022-09-12T20:16:20.777 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:16:20.778 DEBUG:boto:Data: 2022-09-12T20:16:20.778 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.779 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:20.780 DEBUG:boto:Port: 8001 2022-09-12T20:16:20.780 DEBUG:boto:Params: {} 2022-09-12T20:16:20.781 DEBUG:boto:Token: None 2022-09-12T20:16:20.781 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/ 2022-09-12T20:16:20.782 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY= 2022-09-12T20:16:20.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY=', 'Content-Length': '0'} 2022-09-12T20:16:20.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7a6f0b86373be95-00631f9394-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.786 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:20.787 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:20.788 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:20.788 DEBUG:boto:path=/jdpiak-75/?prefix=obj38 2022-09-12T20:16:20.789 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj38 2022-09-12T20:16:20.789 DEBUG:boto:Method: GET 2022-09-12T20:16:20.790 DEBUG:boto:Path: /jdpiak-75/?prefix=obj38 2022-09-12T20:16:20.791 DEBUG:boto:Data: 2022-09-12T20:16:20.791 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.792 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:20.792 DEBUG:boto:Port: 8000 2022-09-12T20:16:20.793 DEBUG:boto:Params: {} 2022-09-12T20:16:20.793 DEBUG:boto:Token: None 2022-09-12T20:16:20.794 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/ 2022-09-12T20:16:20.794 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY= 2022-09-12T20:16:20.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY=', 'Content-Length': '0'} 2022-09-12T20:16:20.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000daf2d6a60e0c6f0e-00631f9394-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.806 DEBUG:boto:b'jdpiak-75obj381000falseobj382022-09-12T20:06:16.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:20.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:16:20.807 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:20.808 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:20.809 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:20.809 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:20.810 DEBUG:boto:Method: GET 2022-09-12T20:16:20.810 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:20.811 DEBUG:boto:Data: 2022-09-12T20:16:20.811 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.812 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:20.812 DEBUG:boto:Port: 8001 2022-09-12T20:16:20.813 DEBUG:boto:Params: {} 2022-09-12T20:16:20.814 DEBUG:boto:Token: None 2022-09-12T20:16:20.814 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:20.815 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:myWPPJR9uoVzFh7U+Poh4U3TilY= 2022-09-12T20:16:20.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:myWPPJR9uoVzFh7U+Poh4U3TilY=', 'Content-Length': '0'} 2022-09-12T20:16:20.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082a54203548eef6a-00631f9394-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.818 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:20.819 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:20.820 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:20.820 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:20.821 DEBUG:boto:Method: GET 2022-09-12T20:16:20.821 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:20.822 DEBUG:boto:Data: 2022-09-12T20:16:20.822 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.823 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:20.824 DEBUG:boto:Port: 8000 2022-09-12T20:16:20.824 DEBUG:boto:Params: {} 2022-09-12T20:16:20.825 DEBUG:boto:Token: None 2022-09-12T20:16:20.825 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:20.826 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:myWPPJR9uoVzFh7U+Poh4U3TilY= 2022-09-12T20:16:20.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:myWPPJR9uoVzFh7U+Poh4U3TilY=', 'Content-Length': '0'} 2022-09-12T20:16:20.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075c62b13f35570d4-00631f9394-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.830 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:20.830 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:20.831 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:20.831 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:20.832 DEBUG:boto:Method: HEAD 2022-09-12T20:16:20.832 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:20.833 DEBUG:boto:Data: 2022-09-12T20:16:20.834 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.834 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:20.835 DEBUG:boto:Port: 8001 2022-09-12T20:16:20.835 DEBUG:boto:Params: {} 2022-09-12T20:16:20.836 DEBUG:boto:Token: None 2022-09-12T20:16:20.837 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:20.837 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sJSiOMrZ0o1VWOngTvSFrDIVT98= 2022-09-12T20:16:20.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sJSiOMrZ0o1VWOngTvSFrDIVT98=', 'Content-Length': '0'} 2022-09-12T20:16:20.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8ed2e76935980c0-00631f9394-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.840 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:20.841 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:20.841 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:20.842 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:20.842 DEBUG:boto:Method: HEAD 2022-09-12T20:16:20.843 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:20.844 DEBUG:boto:Data: 2022-09-12T20:16:20.844 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.845 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:20.845 DEBUG:boto:Port: 8000 2022-09-12T20:16:20.846 DEBUG:boto:Params: {} 2022-09-12T20:16:20.846 DEBUG:boto:Token: None 2022-09-12T20:16:20.847 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:20.848 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sJSiOMrZ0o1VWOngTvSFrDIVT98= 2022-09-12T20:16:20.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sJSiOMrZ0o1VWOngTvSFrDIVT98=', 'Content-Length': '0'} 2022-09-12T20:16:20.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000010a13bbf502e969-00631f9394-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:16:20.851 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:20.852 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:20.853 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:20.853 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:20.854 DEBUG:boto:Method: GET 2022-09-12T20:16:20.854 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:20.855 DEBUG:boto:Data: 2022-09-12T20:16:20.856 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.856 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:20.857 DEBUG:boto:Port: 8001 2022-09-12T20:16:20.857 DEBUG:boto:Params: {} 2022-09-12T20:16:20.858 DEBUG:boto:Token: None 2022-09-12T20:16:20.859 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:20.859 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:myWPPJR9uoVzFh7U+Poh4U3TilY= 2022-09-12T20:16:20.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:myWPPJR9uoVzFh7U+Poh4U3TilY=', 'Content-Length': '0'} 2022-09-12T20:16:20.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e57dd8343c06d890-00631f9394-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.863 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:20.863 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:20.864 DEBUG:boto:path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:20.865 DEBUG:boto:auth_path=/jdpiak-75/obj44?versionId=null 2022-09-12T20:16:20.865 DEBUG:boto:Method: GET 2022-09-12T20:16:20.866 DEBUG:boto:Path: /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:20.866 DEBUG:boto:Data: 2022-09-12T20:16:20.867 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.868 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:20.868 DEBUG:boto:Port: 8000 2022-09-12T20:16:20.869 DEBUG:boto:Params: {} 2022-09-12T20:16:20.869 DEBUG:boto:Token: None 2022-09-12T20:16:20.870 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj44?versionId=null 2022-09-12T20:16:20.871 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:myWPPJR9uoVzFh7U+Poh4U3TilY= 2022-09-12T20:16:20.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:myWPPJR9uoVzFh7U+Poh4U3TilY=', 'Content-Length': '0'} 2022-09-12T20:16:20.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031cf4ab9f9297e27-00631f9394-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.874 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:20.875 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:20.875 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:16:20.876 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:16:20.877 DEBUG:boto:Method: GET 2022-09-12T20:16:20.878 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:16:20.878 DEBUG:boto:Data: 2022-09-12T20:16:20.879 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.879 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:20.880 DEBUG:boto:Port: 8001 2022-09-12T20:16:20.880 DEBUG:boto:Params: {} 2022-09-12T20:16:20.881 DEBUG:boto:Token: None 2022-09-12T20:16:20.881 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/ 2022-09-12T20:16:20.882 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY= 2022-09-12T20:16:20.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY=', 'Content-Length': '0'} 2022-09-12T20:16:20.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5599d020742d3e8-00631f9394-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.926 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:20.927 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:20.928 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:20.928 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:16:20.929 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:16:20.929 DEBUG:boto:Method: GET 2022-09-12T20:16:20.930 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:16:20.930 DEBUG:boto:Data: 2022-09-12T20:16:20.931 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.931 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:20.932 DEBUG:boto:Port: 8000 2022-09-12T20:16:20.933 DEBUG:boto:Params: {} 2022-09-12T20:16:20.933 DEBUG:boto:Token: None 2022-09-12T20:16:20.934 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/ 2022-09-12T20:16:20.935 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY= 2022-09-12T20:16:20.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4t153g0nYcO4M00/rOE9gROtXiY=', 'Content-Length': '0'} 2022-09-12T20:16:20.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab553ce0a4cd3753-00631f9394-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.939 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:20.940 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:20.940 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:20.941 DEBUG:boto:Method: HEAD 2022-09-12T20:16:20.941 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:16:20.942 DEBUG:boto:Data: 2022-09-12T20:16:20.942 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.943 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:20.943 DEBUG:boto:Port: 8001 2022-09-12T20:16:20.944 DEBUG:boto:Params: {} 2022-09-12T20:16:20.945 DEBUG:boto:Token: None 2022-09-12T20:16:20.945 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj44 2022-09-12T20:16:20.946 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BH8ZpGHVljuXhXXzUgBm/fK4c4c= 2022-09-12T20:16:20.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BH8ZpGHVljuXhXXzUgBm/fK4c4c=', 'Content-Length': '0'} 2022-09-12T20:16:20.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000959d09f41b7f2189-00631f9394-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.949 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:20.950 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:20.950 DEBUG:boto:Method: HEAD 2022-09-12T20:16:20.951 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:16:20.951 DEBUG:boto:Data: 2022-09-12T20:16:20.952 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.952 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:20.953 DEBUG:boto:Port: 8000 2022-09-12T20:16:20.953 DEBUG:boto:Params: {} 2022-09-12T20:16:20.954 DEBUG:boto:Token: None 2022-09-12T20:16:20.955 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj44 2022-09-12T20:16:20.955 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BH8ZpGHVljuXhXXzUgBm/fK4c4c= 2022-09-12T20:16:20.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BH8ZpGHVljuXhXXzUgBm/fK4c4c=', 'Content-Length': '0'} 2022-09-12T20:16:20.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c8ee687e1469a41-00631f9394-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:16:20.959 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:20.959 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:20.960 DEBUG:boto:Method: GET 2022-09-12T20:16:20.961 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:16:20.961 DEBUG:boto:Data: 2022-09-12T20:16:20.962 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.963 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:20.963 DEBUG:boto:Port: 8001 2022-09-12T20:16:20.964 DEBUG:boto:Params: {} 2022-09-12T20:16:20.964 DEBUG:boto:Token: None 2022-09-12T20:16:20.965 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj44 2022-09-12T20:16:20.965 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:i4kFVDZtKzjg6fw+YyESzfas0FE= 2022-09-12T20:16:20.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:i4kFVDZtKzjg6fw+YyESzfas0FE=', 'Content-Length': '0'} 2022-09-12T20:16:20.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdc6c6b733287975-00631f9394-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:20.969 DEBUG:boto:path=/jdpiak-75/obj44 2022-09-12T20:16:20.969 DEBUG:boto:auth_path=/jdpiak-75/obj44 2022-09-12T20:16:20.970 DEBUG:boto:Method: GET 2022-09-12T20:16:20.971 DEBUG:boto:Path: /jdpiak-75/obj44 2022-09-12T20:16:20.971 DEBUG:boto:Data: 2022-09-12T20:16:20.972 DEBUG:boto:Headers: {} 2022-09-12T20:16:20.972 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:20.973 DEBUG:boto:Port: 8000 2022-09-12T20:16:20.973 DEBUG:boto:Params: {} 2022-09-12T20:16:20.974 DEBUG:boto:Token: None 2022-09-12T20:16:20.974 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:20 GMT /jdpiak-75/obj44 2022-09-12T20:16:20.975 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:i4kFVDZtKzjg6fw+YyESzfas0FE= 2022-09-12T20:16:20.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:20 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:i4kFVDZtKzjg6fw+YyESzfas0FE=', 'Content-Length': '0'} 2022-09-12T20:16:20.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f6eaa97d749cbe3-00631f9394-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.018 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:21.019 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:21.019 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:16:21.020 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:16:21.020 DEBUG:boto:Method: GET 2022-09-12T20:16:21.021 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:16:21.021 DEBUG:boto:Data: 2022-09-12T20:16:21.022 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.023 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.023 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.024 DEBUG:boto:Params: {} 2022-09-12T20:16:21.024 DEBUG:boto:Token: None 2022-09-12T20:16:21.025 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/ 2022-09-12T20:16:21.025 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E= 2022-09-12T20:16:21.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E=', 'Content-Length': '0'} 2022-09-12T20:16:21.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010790da834fb3d1b-00631f9395-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.029 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:21.030 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:21.031 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:21.031 DEBUG:boto:path=/jdpiak-75/?prefix=obj44 2022-09-12T20:16:21.032 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj44 2022-09-12T20:16:21.032 DEBUG:boto:Method: GET 2022-09-12T20:16:21.033 DEBUG:boto:Path: /jdpiak-75/?prefix=obj44 2022-09-12T20:16:21.033 DEBUG:boto:Data: 2022-09-12T20:16:21.034 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.034 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.035 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.036 DEBUG:boto:Params: {} 2022-09-12T20:16:21.036 DEBUG:boto:Token: None 2022-09-12T20:16:21.037 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/ 2022-09-12T20:16:21.037 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E= 2022-09-12T20:16:21.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E=', 'Content-Length': '0'} 2022-09-12T20:16:21.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000108e9fbe44f8fb33-00631f9395-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.041 DEBUG:boto:b'jdpiak-75obj441000falseobj442022-09-12T20:03:02.545Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:21.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:16:21.043 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:21.043 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:21.044 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:21.044 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:21.045 DEBUG:boto:Method: GET 2022-09-12T20:16:21.045 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:21.046 DEBUG:boto:Data: 2022-09-12T20:16:21.046 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.047 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.047 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.048 DEBUG:boto:Params: {} 2022-09-12T20:16:21.048 DEBUG:boto:Token: None 2022-09-12T20:16:21.049 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:21.050 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MnT+1ERjP/UyplxDnOFTC0+aVQI= 2022-09-12T20:16:21.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MnT+1ERjP/UyplxDnOFTC0+aVQI=', 'Content-Length': '0'} 2022-09-12T20:16:21.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005456ca964ac2f01c-00631f9395-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.053 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:21.054 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:21.054 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:21.055 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:21.055 DEBUG:boto:Method: GET 2022-09-12T20:16:21.056 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:21.056 DEBUG:boto:Data: 2022-09-12T20:16:21.057 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.058 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.058 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.059 DEBUG:boto:Params: {} 2022-09-12T20:16:21.059 DEBUG:boto:Token: None 2022-09-12T20:16:21.060 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:21.061 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MnT+1ERjP/UyplxDnOFTC0+aVQI= 2022-09-12T20:16:21.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MnT+1ERjP/UyplxDnOFTC0+aVQI=', 'Content-Length': '0'} 2022-09-12T20:16:21.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003af80dcdfac52754-00631f9395-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.065 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:21.065 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:21.066 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:21.066 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:21.067 DEBUG:boto:Method: HEAD 2022-09-12T20:16:21.067 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:21.068 DEBUG:boto:Data: 2022-09-12T20:16:21.068 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.069 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.069 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.070 DEBUG:boto:Params: {} 2022-09-12T20:16:21.070 DEBUG:boto:Token: None 2022-09-12T20:16:21.071 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:21.071 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ilY/E6H1IRslxF29c8B13tcufdw= 2022-09-12T20:16:21.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ilY/E6H1IRslxF29c8B13tcufdw=', 'Content-Length': '0'} 2022-09-12T20:16:21.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032ccf9d093169f2c-00631f9395-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.075 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:21.075 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:21.076 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:21.076 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:21.077 DEBUG:boto:Method: HEAD 2022-09-12T20:16:21.077 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:21.078 DEBUG:boto:Data: 2022-09-12T20:16:21.078 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.079 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.079 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.080 DEBUG:boto:Params: {} 2022-09-12T20:16:21.080 DEBUG:boto:Token: None 2022-09-12T20:16:21.081 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:21.081 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ilY/E6H1IRslxF29c8B13tcufdw= 2022-09-12T20:16:21.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ilY/E6H1IRslxF29c8B13tcufdw=', 'Content-Length': '0'} 2022-09-12T20:16:21.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000815e149989bed11c-00631f9395-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:16:21.085 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:21.086 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:21.086 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:21.087 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:21.087 DEBUG:boto:Method: GET 2022-09-12T20:16:21.088 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:21.089 DEBUG:boto:Data: 2022-09-12T20:16:21.089 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.090 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.090 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.091 DEBUG:boto:Params: {} 2022-09-12T20:16:21.091 DEBUG:boto:Token: None 2022-09-12T20:16:21.092 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:21.092 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MnT+1ERjP/UyplxDnOFTC0+aVQI= 2022-09-12T20:16:21.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MnT+1ERjP/UyplxDnOFTC0+aVQI=', 'Content-Length': '0'} 2022-09-12T20:16:21.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004735273cc8729c9f-00631f9395-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.096 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:21.097 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:21.097 DEBUG:boto:path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:21.098 DEBUG:boto:auth_path=/jdpiak-75/obj46?versionId=null 2022-09-12T20:16:21.098 DEBUG:boto:Method: GET 2022-09-12T20:16:21.099 DEBUG:boto:Path: /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:21.099 DEBUG:boto:Data: 2022-09-12T20:16:21.100 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.100 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.101 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.101 DEBUG:boto:Params: {} 2022-09-12T20:16:21.102 DEBUG:boto:Token: None 2022-09-12T20:16:21.103 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj46?versionId=null 2022-09-12T20:16:21.103 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MnT+1ERjP/UyplxDnOFTC0+aVQI= 2022-09-12T20:16:21.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MnT+1ERjP/UyplxDnOFTC0+aVQI=', 'Content-Length': '0'} 2022-09-12T20:16:21.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047660aca5b05cc38-00631f9395-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.107 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:21.108 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:21.108 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:16:21.109 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:16:21.109 DEBUG:boto:Method: GET 2022-09-12T20:16:21.110 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:16:21.110 DEBUG:boto:Data: 2022-09-12T20:16:21.111 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.112 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.112 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.113 DEBUG:boto:Params: {} 2022-09-12T20:16:21.114 DEBUG:boto:Token: None 2022-09-12T20:16:21.115 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/ 2022-09-12T20:16:21.115 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E= 2022-09-12T20:16:21.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E=', 'Content-Length': '0'} 2022-09-12T20:16:21.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe707732998564de-00631f9395-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.119 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:21.120 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:21.121 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:21.121 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:16:21.122 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:16:21.123 DEBUG:boto:Method: GET 2022-09-12T20:16:21.123 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:16:21.124 DEBUG:boto:Data: 2022-09-12T20:16:21.124 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.125 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.125 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.126 DEBUG:boto:Params: {} 2022-09-12T20:16:21.126 DEBUG:boto:Token: None 2022-09-12T20:16:21.127 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/ 2022-09-12T20:16:21.128 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E= 2022-09-12T20:16:21.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E=', 'Content-Length': '0'} 2022-09-12T20:16:21.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5e5611ad5025f1b-00631f9395-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.132 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:21.132 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:21.133 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:21.133 DEBUG:boto:Method: HEAD 2022-09-12T20:16:21.134 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:16:21.135 DEBUG:boto:Data: 2022-09-12T20:16:21.135 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.136 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.136 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.137 DEBUG:boto:Params: {} 2022-09-12T20:16:21.137 DEBUG:boto:Token: None 2022-09-12T20:16:21.138 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj46 2022-09-12T20:16:21.138 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A7qPTvOQZx+Zyt49Cy0rG50tZoQ= 2022-09-12T20:16:21.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A7qPTvOQZx+Zyt49Cy0rG50tZoQ=', 'Content-Length': '0'} 2022-09-12T20:16:21.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2c38dd69e1e77a5-00631f9395-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.142 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:21.142 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:21.143 DEBUG:boto:Method: HEAD 2022-09-12T20:16:21.143 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:16:21.144 DEBUG:boto:Data: 2022-09-12T20:16:21.144 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.145 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.145 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.146 DEBUG:boto:Params: {} 2022-09-12T20:16:21.147 DEBUG:boto:Token: None 2022-09-12T20:16:21.147 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj46 2022-09-12T20:16:21.148 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A7qPTvOQZx+Zyt49Cy0rG50tZoQ= 2022-09-12T20:16:21.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A7qPTvOQZx+Zyt49Cy0rG50tZoQ=', 'Content-Length': '0'} 2022-09-12T20:16:21.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016b6232cc4a68af4-00631f9395-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-12T20:16:21.151 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:21.152 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:21.153 DEBUG:boto:Method: GET 2022-09-12T20:16:21.153 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:16:21.154 DEBUG:boto:Data: 2022-09-12T20:16:21.154 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.155 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.155 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.156 DEBUG:boto:Params: {} 2022-09-12T20:16:21.157 DEBUG:boto:Token: None 2022-09-12T20:16:21.157 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj46 2022-09-12T20:16:21.158 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+8A09EAxK1J0+48Ms2bK5eOG4O4= 2022-09-12T20:16:21.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+8A09EAxK1J0+48Ms2bK5eOG4O4=', 'Content-Length': '0'} 2022-09-12T20:16:21.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f486fff944067794-00631f9395-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.202 DEBUG:boto:path=/jdpiak-75/obj46 2022-09-12T20:16:21.203 DEBUG:boto:auth_path=/jdpiak-75/obj46 2022-09-12T20:16:21.203 DEBUG:boto:Method: GET 2022-09-12T20:16:21.204 DEBUG:boto:Path: /jdpiak-75/obj46 2022-09-12T20:16:21.204 DEBUG:boto:Data: 2022-09-12T20:16:21.205 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.205 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.206 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.207 DEBUG:boto:Params: {} 2022-09-12T20:16:21.207 DEBUG:boto:Token: None 2022-09-12T20:16:21.208 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj46 2022-09-12T20:16:21.209 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+8A09EAxK1J0+48Ms2bK5eOG4O4= 2022-09-12T20:16:21.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+8A09EAxK1J0+48Ms2bK5eOG4O4=', 'Content-Length': '0'} 2022-09-12T20:16:21.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000843bdf76dcaa8eee-00631f9395-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.212 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:21.213 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:21.214 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:16:21.214 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:16:21.215 DEBUG:boto:Method: GET 2022-09-12T20:16:21.215 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:16:21.216 DEBUG:boto:Data: 2022-09-12T20:16:21.216 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.217 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.217 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.218 DEBUG:boto:Params: {} 2022-09-12T20:16:21.219 DEBUG:boto:Token: None 2022-09-12T20:16:21.220 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/ 2022-09-12T20:16:21.220 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E= 2022-09-12T20:16:21.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E=', 'Content-Length': '0'} 2022-09-12T20:16:21.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bdbb7af7c8452a4-00631f9395-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.225 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:21.226 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:21.226 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:21.227 DEBUG:boto:path=/jdpiak-75/?prefix=obj46 2022-09-12T20:16:21.228 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj46 2022-09-12T20:16:21.228 DEBUG:boto:Method: GET 2022-09-12T20:16:21.229 DEBUG:boto:Path: /jdpiak-75/?prefix=obj46 2022-09-12T20:16:21.230 DEBUG:boto:Data: 2022-09-12T20:16:21.230 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.231 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.231 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.232 DEBUG:boto:Params: {} 2022-09-12T20:16:21.232 DEBUG:boto:Token: None 2022-09-12T20:16:21.233 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/ 2022-09-12T20:16:21.234 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E= 2022-09-12T20:16:21.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E=', 'Content-Length': '0'} 2022-09-12T20:16:21.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c67a597bfb2badeb-00631f9395-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.237 DEBUG:boto:b'jdpiak-75obj461000falseobj462022-09-12T20:08:38.594Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:21.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:16:21.239 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:21.239 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:21.240 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:21.240 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:21.241 DEBUG:boto:Method: GET 2022-09-12T20:16:21.241 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:21.242 DEBUG:boto:Data: 2022-09-12T20:16:21.243 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.243 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.244 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.244 DEBUG:boto:Params: {} 2022-09-12T20:16:21.245 DEBUG:boto:Token: None 2022-09-12T20:16:21.245 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:21.246 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fejub6pFbG8hR5Dr8AeeqyUUiYc= 2022-09-12T20:16:21.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fejub6pFbG8hR5Dr8AeeqyUUiYc=', 'Content-Length': '0'} 2022-09-12T20:16:21.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056683835cee540d6-00631f9395-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.250 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:21.250 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:21.251 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:21.251 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:21.252 DEBUG:boto:Method: GET 2022-09-12T20:16:21.252 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:21.253 DEBUG:boto:Data: 2022-09-12T20:16:21.254 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.254 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.255 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.255 DEBUG:boto:Params: {} 2022-09-12T20:16:21.256 DEBUG:boto:Token: None 2022-09-12T20:16:21.256 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:21.257 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fejub6pFbG8hR5Dr8AeeqyUUiYc= 2022-09-12T20:16:21.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fejub6pFbG8hR5Dr8AeeqyUUiYc=', 'Content-Length': '0'} 2022-09-12T20:16:21.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000376f1286dc972c2e-00631f9395-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.261 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:21.261 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:21.262 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:21.262 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:21.263 DEBUG:boto:Method: HEAD 2022-09-12T20:16:21.263 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:21.264 DEBUG:boto:Data: 2022-09-12T20:16:21.264 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.265 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.265 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.266 DEBUG:boto:Params: {} 2022-09-12T20:16:21.266 DEBUG:boto:Token: None 2022-09-12T20:16:21.267 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:21.267 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NkpJBl5No2DuDG34/hwS9i1qgK4= 2022-09-12T20:16:21.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NkpJBl5No2DuDG34/hwS9i1qgK4=', 'Content-Length': '0'} 2022-09-12T20:16:21.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000344a279d8fb5c4f5-00631f9395-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.271 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:21.271 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:21.272 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:21.272 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:21.273 DEBUG:boto:Method: HEAD 2022-09-12T20:16:21.273 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:21.274 DEBUG:boto:Data: 2022-09-12T20:16:21.274 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.275 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.275 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.276 DEBUG:boto:Params: {} 2022-09-12T20:16:21.276 DEBUG:boto:Token: None 2022-09-12T20:16:21.277 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:21.278 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NkpJBl5No2DuDG34/hwS9i1qgK4= 2022-09-12T20:16:21.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NkpJBl5No2DuDG34/hwS9i1qgK4=', 'Content-Length': '0'} 2022-09-12T20:16:21.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5b82af8653caefb-00631f9395-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:16:21.281 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:21.282 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:21.283 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:21.283 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:21.284 DEBUG:boto:Method: GET 2022-09-12T20:16:21.284 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:21.285 DEBUG:boto:Data: 2022-09-12T20:16:21.285 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.286 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.286 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.287 DEBUG:boto:Params: {} 2022-09-12T20:16:21.287 DEBUG:boto:Token: None 2022-09-12T20:16:21.288 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:21.289 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fejub6pFbG8hR5Dr8AeeqyUUiYc= 2022-09-12T20:16:21.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fejub6pFbG8hR5Dr8AeeqyUUiYc=', 'Content-Length': '0'} 2022-09-12T20:16:21.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082d2db0e370dcdc7-00631f9395-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.292 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:21.293 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:21.293 DEBUG:boto:path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:21.294 DEBUG:boto:auth_path=/jdpiak-75/obj51?versionId=null 2022-09-12T20:16:21.294 DEBUG:boto:Method: GET 2022-09-12T20:16:21.295 DEBUG:boto:Path: /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:21.295 DEBUG:boto:Data: 2022-09-12T20:16:21.296 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.297 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.298 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.299 DEBUG:boto:Params: {} 2022-09-12T20:16:21.300 DEBUG:boto:Token: None 2022-09-12T20:16:21.300 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj51?versionId=null 2022-09-12T20:16:21.301 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fejub6pFbG8hR5Dr8AeeqyUUiYc= 2022-09-12T20:16:21.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fejub6pFbG8hR5Dr8AeeqyUUiYc=', 'Content-Length': '0'} 2022-09-12T20:16:21.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aee7b77521dea76a-00631f9395-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.305 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:21.305 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:21.306 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:16:21.306 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:16:21.307 DEBUG:boto:Method: GET 2022-09-12T20:16:21.307 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:16:21.308 DEBUG:boto:Data: 2022-09-12T20:16:21.309 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.309 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.310 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.310 DEBUG:boto:Params: {} 2022-09-12T20:16:21.311 DEBUG:boto:Token: None 2022-09-12T20:16:21.311 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/ 2022-09-12T20:16:21.312 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E= 2022-09-12T20:16:21.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E=', 'Content-Length': '0'} 2022-09-12T20:16:21.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008db8c2db832102dc-00631f9395-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.316 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:21.317 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:21.318 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:21.318 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:16:21.319 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:16:21.319 DEBUG:boto:Method: GET 2022-09-12T20:16:21.320 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:16:21.320 DEBUG:boto:Data: 2022-09-12T20:16:21.321 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.321 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.322 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.322 DEBUG:boto:Params: {} 2022-09-12T20:16:21.323 DEBUG:boto:Token: None 2022-09-12T20:16:21.324 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/ 2022-09-12T20:16:21.324 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E= 2022-09-12T20:16:21.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E=', 'Content-Length': '0'} 2022-09-12T20:16:21.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a85a869cfd1abf7-00631f9395-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.329 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:21.330 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:21.330 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:21.331 DEBUG:boto:Method: HEAD 2022-09-12T20:16:21.331 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:16:21.332 DEBUG:boto:Data: 2022-09-12T20:16:21.332 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.333 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.333 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.334 DEBUG:boto:Params: {} 2022-09-12T20:16:21.334 DEBUG:boto:Token: None 2022-09-12T20:16:21.335 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj51 2022-09-12T20:16:21.336 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BDR9v7WqX1mOmWT9v5k7HPrLSyQ= 2022-09-12T20:16:21.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BDR9v7WqX1mOmWT9v5k7HPrLSyQ=', 'Content-Length': '0'} 2022-09-12T20:16:21.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006edb433a3ecba24-00631f9395-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.339 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:21.339 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:21.340 DEBUG:boto:Method: HEAD 2022-09-12T20:16:21.340 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:16:21.341 DEBUG:boto:Data: 2022-09-12T20:16:21.341 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.342 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.342 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.343 DEBUG:boto:Params: {} 2022-09-12T20:16:21.344 DEBUG:boto:Token: None 2022-09-12T20:16:21.344 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj51 2022-09-12T20:16:21.345 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BDR9v7WqX1mOmWT9v5k7HPrLSyQ= 2022-09-12T20:16:21.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BDR9v7WqX1mOmWT9v5k7HPrLSyQ=', 'Content-Length': '0'} 2022-09-12T20:16:21.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9b3994075590967-00631f9395-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-12T20:16:21.348 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:21.349 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:21.350 DEBUG:boto:Method: GET 2022-09-12T20:16:21.350 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:16:21.351 DEBUG:boto:Data: 2022-09-12T20:16:21.351 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.352 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.352 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.353 DEBUG:boto:Params: {} 2022-09-12T20:16:21.353 DEBUG:boto:Token: None 2022-09-12T20:16:21.354 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj51 2022-09-12T20:16:21.355 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dJ8+smTgGh9yM+JaX6QVI3Lih6k= 2022-09-12T20:16:21.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dJ8+smTgGh9yM+JaX6QVI3Lih6k=', 'Content-Length': '0'} 2022-09-12T20:16:21.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000641ca8101a4a738e-00631f9395-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.398 DEBUG:boto:path=/jdpiak-75/obj51 2022-09-12T20:16:21.399 DEBUG:boto:auth_path=/jdpiak-75/obj51 2022-09-12T20:16:21.399 DEBUG:boto:Method: GET 2022-09-12T20:16:21.400 DEBUG:boto:Path: /jdpiak-75/obj51 2022-09-12T20:16:21.400 DEBUG:boto:Data: 2022-09-12T20:16:21.401 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.402 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.402 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.403 DEBUG:boto:Params: {} 2022-09-12T20:16:21.403 DEBUG:boto:Token: None 2022-09-12T20:16:21.404 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj51 2022-09-12T20:16:21.404 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dJ8+smTgGh9yM+JaX6QVI3Lih6k= 2022-09-12T20:16:21.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dJ8+smTgGh9yM+JaX6QVI3Lih6k=', 'Content-Length': '0'} 2022-09-12T20:16:21.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000621b5288d72361d4-00631f9395-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.409 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:21.410 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:21.410 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:16:21.411 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:16:21.411 DEBUG:boto:Method: GET 2022-09-12T20:16:21.412 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:16:21.412 DEBUG:boto:Data: 2022-09-12T20:16:21.413 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.413 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.414 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.414 DEBUG:boto:Params: {} 2022-09-12T20:16:21.415 DEBUG:boto:Token: None 2022-09-12T20:16:21.416 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/ 2022-09-12T20:16:21.416 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E= 2022-09-12T20:16:21.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E=', 'Content-Length': '0'} 2022-09-12T20:16:21.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfca6fae47ae09fb-00631f9395-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.420 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:21.421 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:21.422 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:21.422 DEBUG:boto:path=/jdpiak-75/?prefix=obj51 2022-09-12T20:16:21.423 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj51 2022-09-12T20:16:21.424 DEBUG:boto:Method: GET 2022-09-12T20:16:21.425 DEBUG:boto:Path: /jdpiak-75/?prefix=obj51 2022-09-12T20:16:21.425 DEBUG:boto:Data: 2022-09-12T20:16:21.426 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.426 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.427 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.427 DEBUG:boto:Params: {} 2022-09-12T20:16:21.428 DEBUG:boto:Token: None 2022-09-12T20:16:21.428 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/ 2022-09-12T20:16:21.429 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E= 2022-09-12T20:16:21.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E=', 'Content-Length': '0'} 2022-09-12T20:16:21.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073f7413b0bdbcb3f-00631f9395-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.433 DEBUG:boto:b'jdpiak-75obj511000falseobj512022-09-12T20:03:28.556Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:21.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:16:21.434 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:21.435 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:21.435 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:21.436 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:21.436 DEBUG:boto:Method: GET 2022-09-12T20:16:21.437 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:21.438 DEBUG:boto:Data: 2022-09-12T20:16:21.438 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.439 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.439 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.440 DEBUG:boto:Params: {} 2022-09-12T20:16:21.440 DEBUG:boto:Token: None 2022-09-12T20:16:21.441 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:21.441 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HPL4Q0TjsScL571krLnNzoLoGF8= 2022-09-12T20:16:21.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HPL4Q0TjsScL571krLnNzoLoGF8=', 'Content-Length': '0'} 2022-09-12T20:16:21.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b66b8983c2771476-00631f9395-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.445 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:21.446 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:21.446 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:21.447 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:21.447 DEBUG:boto:Method: GET 2022-09-12T20:16:21.448 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:21.448 DEBUG:boto:Data: 2022-09-12T20:16:21.449 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.449 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.450 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.451 DEBUG:boto:Params: {} 2022-09-12T20:16:21.451 DEBUG:boto:Token: None 2022-09-12T20:16:21.452 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:21.452 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HPL4Q0TjsScL571krLnNzoLoGF8= 2022-09-12T20:16:21.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HPL4Q0TjsScL571krLnNzoLoGF8=', 'Content-Length': '0'} 2022-09-12T20:16:21.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc8c0ceaeb1b6fa4-00631f9395-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.456 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:21.456 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:21.457 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:21.457 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:21.458 DEBUG:boto:Method: HEAD 2022-09-12T20:16:21.458 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:21.459 DEBUG:boto:Data: 2022-09-12T20:16:21.459 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.460 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.460 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.461 DEBUG:boto:Params: {} 2022-09-12T20:16:21.461 DEBUG:boto:Token: None 2022-09-12T20:16:21.462 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:21.462 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aA/WZYXK7GsucGV9Hpq7OWRV6dQ= 2022-09-12T20:16:21.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aA/WZYXK7GsucGV9Hpq7OWRV6dQ=', 'Content-Length': '0'} 2022-09-12T20:16:21.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1983effbe41f95b-00631f9395-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.466 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:21.466 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:21.467 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:21.467 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:21.468 DEBUG:boto:Method: HEAD 2022-09-12T20:16:21.468 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:21.469 DEBUG:boto:Data: 2022-09-12T20:16:21.469 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.470 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.470 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.471 DEBUG:boto:Params: {} 2022-09-12T20:16:21.471 DEBUG:boto:Token: None 2022-09-12T20:16:21.472 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:21.472 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:aA/WZYXK7GsucGV9Hpq7OWRV6dQ= 2022-09-12T20:16:21.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:aA/WZYXK7GsucGV9Hpq7OWRV6dQ=', 'Content-Length': '0'} 2022-09-12T20:16:21.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de3cccc65a603953-00631f9395-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:16:21.476 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:21.477 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:21.477 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:21.478 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:21.478 DEBUG:boto:Method: GET 2022-09-12T20:16:21.479 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:21.479 DEBUG:boto:Data: 2022-09-12T20:16:21.480 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.480 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.481 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.481 DEBUG:boto:Params: {} 2022-09-12T20:16:21.482 DEBUG:boto:Token: None 2022-09-12T20:16:21.483 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:21.483 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HPL4Q0TjsScL571krLnNzoLoGF8= 2022-09-12T20:16:21.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HPL4Q0TjsScL571krLnNzoLoGF8=', 'Content-Length': '0'} 2022-09-12T20:16:21.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088b5ce980d47d7b5-00631f9395-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.487 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:21.487 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:21.488 DEBUG:boto:path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:21.489 DEBUG:boto:auth_path=/jdpiak-75/obj55?versionId=null 2022-09-12T20:16:21.489 DEBUG:boto:Method: GET 2022-09-12T20:16:21.490 DEBUG:boto:Path: /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:21.490 DEBUG:boto:Data: 2022-09-12T20:16:21.491 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.491 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.492 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.492 DEBUG:boto:Params: {} 2022-09-12T20:16:21.493 DEBUG:boto:Token: None 2022-09-12T20:16:21.493 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj55?versionId=null 2022-09-12T20:16:21.494 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:HPL4Q0TjsScL571krLnNzoLoGF8= 2022-09-12T20:16:21.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:HPL4Q0TjsScL571krLnNzoLoGF8=', 'Content-Length': '0'} 2022-09-12T20:16:21.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026e66488240b5282-00631f9395-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.498 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:21.498 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:21.499 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:16:21.499 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:16:21.500 DEBUG:boto:Method: GET 2022-09-12T20:16:21.500 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:16:21.501 DEBUG:boto:Data: 2022-09-12T20:16:21.502 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.502 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.503 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.503 DEBUG:boto:Params: {} 2022-09-12T20:16:21.504 DEBUG:boto:Token: None 2022-09-12T20:16:21.504 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/ 2022-09-12T20:16:21.505 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E= 2022-09-12T20:16:21.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E=', 'Content-Length': '0'} 2022-09-12T20:16:21.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb4f97f55fc75ac9-00631f9395-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.550 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:21.551 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:21.552 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:21.552 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:16:21.553 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:16:21.553 DEBUG:boto:Method: GET 2022-09-12T20:16:21.554 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:16:21.555 DEBUG:boto:Data: 2022-09-12T20:16:21.555 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.556 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.556 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.557 DEBUG:boto:Params: {} 2022-09-12T20:16:21.557 DEBUG:boto:Token: None 2022-09-12T20:16:21.558 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/ 2022-09-12T20:16:21.558 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E= 2022-09-12T20:16:21.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E=', 'Content-Length': '0'} 2022-09-12T20:16:21.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfa50d48db8503de-00631f9395-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.563 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:21.564 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:21.564 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:21.565 DEBUG:boto:Method: HEAD 2022-09-12T20:16:21.565 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:16:21.566 DEBUG:boto:Data: 2022-09-12T20:16:21.566 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.567 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.567 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.568 DEBUG:boto:Params: {} 2022-09-12T20:16:21.568 DEBUG:boto:Token: None 2022-09-12T20:16:21.569 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj55 2022-09-12T20:16:21.570 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Rc7S0HxiA2lLjYzsBpjOZoD4d2U= 2022-09-12T20:16:21.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Rc7S0HxiA2lLjYzsBpjOZoD4d2U=', 'Content-Length': '0'} 2022-09-12T20:16:21.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ff1e4a043eafcc8-00631f9395-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.573 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:21.574 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:21.574 DEBUG:boto:Method: HEAD 2022-09-12T20:16:21.575 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:16:21.575 DEBUG:boto:Data: 2022-09-12T20:16:21.576 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.576 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.577 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.577 DEBUG:boto:Params: {} 2022-09-12T20:16:21.578 DEBUG:boto:Token: None 2022-09-12T20:16:21.579 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj55 2022-09-12T20:16:21.579 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Rc7S0HxiA2lLjYzsBpjOZoD4d2U= 2022-09-12T20:16:21.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Rc7S0HxiA2lLjYzsBpjOZoD4d2U=', 'Content-Length': '0'} 2022-09-12T20:16:21.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fa5c3b659124e8c-00631f9395-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:16:21.583 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:21.584 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:21.584 DEBUG:boto:Method: GET 2022-09-12T20:16:21.585 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:16:21.585 DEBUG:boto:Data: 2022-09-12T20:16:21.586 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.586 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.587 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.587 DEBUG:boto:Params: {} 2022-09-12T20:16:21.588 DEBUG:boto:Token: None 2022-09-12T20:16:21.588 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj55 2022-09-12T20:16:21.589 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gww/GqwJAV9YU0jRhZLoaGzbU0I= 2022-09-12T20:16:21.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gww/GqwJAV9YU0jRhZLoaGzbU0I=', 'Content-Length': '0'} 2022-09-12T20:16:21.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000623072c02cbb1875-00631f9395-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.592 DEBUG:boto:path=/jdpiak-75/obj55 2022-09-12T20:16:21.593 DEBUG:boto:auth_path=/jdpiak-75/obj55 2022-09-12T20:16:21.594 DEBUG:boto:Method: GET 2022-09-12T20:16:21.594 DEBUG:boto:Path: /jdpiak-75/obj55 2022-09-12T20:16:21.595 DEBUG:boto:Data: 2022-09-12T20:16:21.595 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.596 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.596 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.597 DEBUG:boto:Params: {} 2022-09-12T20:16:21.597 DEBUG:boto:Token: None 2022-09-12T20:16:21.598 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj55 2022-09-12T20:16:21.599 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gww/GqwJAV9YU0jRhZLoaGzbU0I= 2022-09-12T20:16:21.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gww/GqwJAV9YU0jRhZLoaGzbU0I=', 'Content-Length': '0'} 2022-09-12T20:16:21.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b5b2c73b01aa3a7-00631f9395-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.602 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:21.603 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:21.603 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:16:21.604 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:16:21.605 DEBUG:boto:Method: GET 2022-09-12T20:16:21.605 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:16:21.606 DEBUG:boto:Data: 2022-09-12T20:16:21.606 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.607 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.607 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.608 DEBUG:boto:Params: {} 2022-09-12T20:16:21.608 DEBUG:boto:Token: None 2022-09-12T20:16:21.609 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/ 2022-09-12T20:16:21.610 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E= 2022-09-12T20:16:21.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E=', 'Content-Length': '0'} 2022-09-12T20:16:21.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a72f72e59b991e7-00631f9395-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.614 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:21.615 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:21.615 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:21.616 DEBUG:boto:path=/jdpiak-75/?prefix=obj55 2022-09-12T20:16:21.616 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj55 2022-09-12T20:16:21.617 DEBUG:boto:Method: GET 2022-09-12T20:16:21.618 DEBUG:boto:Path: /jdpiak-75/?prefix=obj55 2022-09-12T20:16:21.618 DEBUG:boto:Data: 2022-09-12T20:16:21.619 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.619 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.620 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.620 DEBUG:boto:Params: {} 2022-09-12T20:16:21.621 DEBUG:boto:Token: None 2022-09-12T20:16:21.622 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/ 2022-09-12T20:16:21.622 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E= 2022-09-12T20:16:21.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E=', 'Content-Length': '0'} 2022-09-12T20:16:21.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4488fc7be8faded-00631f9395-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.626 DEBUG:boto:b'jdpiak-75obj551000falseobj552022-09-12T20:03:02.577Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:21.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:16:21.627 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:21.628 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:21.628 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:21.629 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:21.629 DEBUG:boto:Method: GET 2022-09-12T20:16:21.630 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:21.631 DEBUG:boto:Data: 2022-09-12T20:16:21.631 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.632 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.632 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.633 DEBUG:boto:Params: {} 2022-09-12T20:16:21.634 DEBUG:boto:Token: None 2022-09-12T20:16:21.634 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:21.635 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/J7mX6MHsCpz7NokPxmGV1JSaxo= 2022-09-12T20:16:21.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/J7mX6MHsCpz7NokPxmGV1JSaxo=', 'Content-Length': '0'} 2022-09-12T20:16:21.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7addda8f0b24709-00631f9395-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.639 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:21.639 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:21.640 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:21.641 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:21.641 DEBUG:boto:Method: GET 2022-09-12T20:16:21.642 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:21.642 DEBUG:boto:Data: 2022-09-12T20:16:21.643 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.643 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.644 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.644 DEBUG:boto:Params: {} 2022-09-12T20:16:21.645 DEBUG:boto:Token: None 2022-09-12T20:16:21.646 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:21.646 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/J7mX6MHsCpz7NokPxmGV1JSaxo= 2022-09-12T20:16:21.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/J7mX6MHsCpz7NokPxmGV1JSaxo=', 'Content-Length': '0'} 2022-09-12T20:16:21.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000854d73b7aa9cadd1-00631f9395-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.650 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:21.650 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:21.651 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:21.651 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:21.652 DEBUG:boto:Method: HEAD 2022-09-12T20:16:21.652 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:21.653 DEBUG:boto:Data: 2022-09-12T20:16:21.653 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.654 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.654 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.655 DEBUG:boto:Params: {} 2022-09-12T20:16:21.655 DEBUG:boto:Token: None 2022-09-12T20:16:21.656 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:21.656 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Rdpb6gw/kMT6ZblV5CJPF7r7uaY= 2022-09-12T20:16:21.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Rdpb6gw/kMT6ZblV5CJPF7r7uaY=', 'Content-Length': '0'} 2022-09-12T20:16:21.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091b5ecd036c74520-00631f9395-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.660 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:21.660 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:21.661 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:21.661 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:21.662 DEBUG:boto:Method: HEAD 2022-09-12T20:16:21.662 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:21.663 DEBUG:boto:Data: 2022-09-12T20:16:21.663 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.664 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.664 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.665 DEBUG:boto:Params: {} 2022-09-12T20:16:21.665 DEBUG:boto:Token: None 2022-09-12T20:16:21.666 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:21.666 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Rdpb6gw/kMT6ZblV5CJPF7r7uaY= 2022-09-12T20:16:21.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Rdpb6gw/kMT6ZblV5CJPF7r7uaY=', 'Content-Length': '0'} 2022-09-12T20:16:21.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004710cd0a23c8b261-00631f9395-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:16:21.670 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:21.671 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:21.671 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:21.672 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:21.672 DEBUG:boto:Method: GET 2022-09-12T20:16:21.673 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:21.674 DEBUG:boto:Data: 2022-09-12T20:16:21.674 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.675 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.675 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.676 DEBUG:boto:Params: {} 2022-09-12T20:16:21.676 DEBUG:boto:Token: None 2022-09-12T20:16:21.677 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:21.678 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/J7mX6MHsCpz7NokPxmGV1JSaxo= 2022-09-12T20:16:21.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/J7mX6MHsCpz7NokPxmGV1JSaxo=', 'Content-Length': '0'} 2022-09-12T20:16:21.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000203464ec9ecad631-00631f9395-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.681 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:21.682 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:21.683 DEBUG:boto:path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:21.683 DEBUG:boto:auth_path=/jdpiak-75/obj57?versionId=null 2022-09-12T20:16:21.684 DEBUG:boto:Method: GET 2022-09-12T20:16:21.684 DEBUG:boto:Path: /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:21.685 DEBUG:boto:Data: 2022-09-12T20:16:21.685 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.686 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.686 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.687 DEBUG:boto:Params: {} 2022-09-12T20:16:21.688 DEBUG:boto:Token: None 2022-09-12T20:16:21.689 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj57?versionId=null 2022-09-12T20:16:21.689 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/J7mX6MHsCpz7NokPxmGV1JSaxo= 2022-09-12T20:16:21.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/J7mX6MHsCpz7NokPxmGV1JSaxo=', 'Content-Length': '0'} 2022-09-12T20:16:21.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c1f65b6322235f0-00631f9395-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.693 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:21.693 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:21.694 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:16:21.694 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:16:21.695 DEBUG:boto:Method: GET 2022-09-12T20:16:21.695 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:16:21.696 DEBUG:boto:Data: 2022-09-12T20:16:21.697 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.697 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.698 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.698 DEBUG:boto:Params: {} 2022-09-12T20:16:21.699 DEBUG:boto:Token: None 2022-09-12T20:16:21.699 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/ 2022-09-12T20:16:21.700 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E= 2022-09-12T20:16:21.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E=', 'Content-Length': '0'} 2022-09-12T20:16:21.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d9f295e822eb771-00631f9395-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.704 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:21.705 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:21.705 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:21.706 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:16:21.707 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:16:21.707 DEBUG:boto:Method: GET 2022-09-12T20:16:21.708 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:16:21.708 DEBUG:boto:Data: 2022-09-12T20:16:21.709 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.709 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.710 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.711 DEBUG:boto:Params: {} 2022-09-12T20:16:21.711 DEBUG:boto:Token: None 2022-09-12T20:16:21.712 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/ 2022-09-12T20:16:21.712 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E= 2022-09-12T20:16:21.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E=', 'Content-Length': '0'} 2022-09-12T20:16:21.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009074334637d603f2-00631f9395-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.716 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:21.717 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:21.718 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:21.718 DEBUG:boto:Method: HEAD 2022-09-12T20:16:21.719 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:16:21.719 DEBUG:boto:Data: 2022-09-12T20:16:21.720 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.720 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.721 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.721 DEBUG:boto:Params: {} 2022-09-12T20:16:21.722 DEBUG:boto:Token: None 2022-09-12T20:16:21.723 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj57 2022-09-12T20:16:21.723 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yF8ZnMvTNyoSTO8R7KVljfaDCAs= 2022-09-12T20:16:21.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yF8ZnMvTNyoSTO8R7KVljfaDCAs=', 'Content-Length': '0'} 2022-09-12T20:16:21.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002536e859dac59fd6-00631f9395-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.727 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:21.727 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:21.728 DEBUG:boto:Method: HEAD 2022-09-12T20:16:21.728 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:16:21.729 DEBUG:boto:Data: 2022-09-12T20:16:21.730 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.730 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.731 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.731 DEBUG:boto:Params: {} 2022-09-12T20:16:21.732 DEBUG:boto:Token: None 2022-09-12T20:16:21.732 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj57 2022-09-12T20:16:21.733 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yF8ZnMvTNyoSTO8R7KVljfaDCAs= 2022-09-12T20:16:21.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yF8ZnMvTNyoSTO8R7KVljfaDCAs=', 'Content-Length': '0'} 2022-09-12T20:16:21.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef8f728f271a1232-00631f9395-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-12T20:16:21.737 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:21.738 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:21.738 DEBUG:boto:Method: GET 2022-09-12T20:16:21.739 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:16:21.739 DEBUG:boto:Data: 2022-09-12T20:16:21.740 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.740 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.741 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.742 DEBUG:boto:Params: {} 2022-09-12T20:16:21.742 DEBUG:boto:Token: None 2022-09-12T20:16:21.743 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj57 2022-09-12T20:16:21.744 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/nWNpLxRJGVYF8X8HHTAhjWoHE8= 2022-09-12T20:16:21.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/nWNpLxRJGVYF8X8HHTAhjWoHE8=', 'Content-Length': '0'} 2022-09-12T20:16:21.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c9b0e609d2e2a08-00631f9395-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.790 DEBUG:boto:path=/jdpiak-75/obj57 2022-09-12T20:16:21.791 DEBUG:boto:auth_path=/jdpiak-75/obj57 2022-09-12T20:16:21.791 DEBUG:boto:Method: GET 2022-09-12T20:16:21.792 DEBUG:boto:Path: /jdpiak-75/obj57 2022-09-12T20:16:21.792 DEBUG:boto:Data: 2022-09-12T20:16:21.793 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.793 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.794 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.795 DEBUG:boto:Params: {} 2022-09-12T20:16:21.795 DEBUG:boto:Token: None 2022-09-12T20:16:21.796 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj57 2022-09-12T20:16:21.796 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/nWNpLxRJGVYF8X8HHTAhjWoHE8= 2022-09-12T20:16:21.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/nWNpLxRJGVYF8X8HHTAhjWoHE8=', 'Content-Length': '0'} 2022-09-12T20:16:21.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b079193cfaee44d6-00631f9395-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.800 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:21.801 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:21.801 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:16:21.802 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:16:21.802 DEBUG:boto:Method: GET 2022-09-12T20:16:21.803 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:16:21.803 DEBUG:boto:Data: 2022-09-12T20:16:21.804 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.804 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.805 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.805 DEBUG:boto:Params: {} 2022-09-12T20:16:21.806 DEBUG:boto:Token: None 2022-09-12T20:16:21.807 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/ 2022-09-12T20:16:21.807 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E= 2022-09-12T20:16:21.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E=', 'Content-Length': '0'} 2022-09-12T20:16:21.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e31cf6d978b4e5c-00631f9395-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.811 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:21.812 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:21.823 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:21.823 DEBUG:boto:path=/jdpiak-75/?prefix=obj57 2022-09-12T20:16:21.824 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj57 2022-09-12T20:16:21.824 DEBUG:boto:Method: GET 2022-09-12T20:16:21.825 DEBUG:boto:Path: /jdpiak-75/?prefix=obj57 2022-09-12T20:16:21.825 DEBUG:boto:Data: 2022-09-12T20:16:21.826 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.826 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.827 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.827 DEBUG:boto:Params: {} 2022-09-12T20:16:21.828 DEBUG:boto:Token: None 2022-09-12T20:16:21.829 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/ 2022-09-12T20:16:21.829 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E= 2022-09-12T20:16:21.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E=', 'Content-Length': '0'} 2022-09-12T20:16:21.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1de99a992fd8f41-00631f9395-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.834 DEBUG:boto:b'jdpiak-75obj571000falseobj572022-09-12T20:06:16.885Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:21.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:16:21.835 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:21.836 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:21.836 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:21.837 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:21.837 DEBUG:boto:Method: GET 2022-09-12T20:16:21.838 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:21.838 DEBUG:boto:Data: 2022-09-12T20:16:21.839 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.839 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.840 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.840 DEBUG:boto:Params: {} 2022-09-12T20:16:21.841 DEBUG:boto:Token: None 2022-09-12T20:16:21.842 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:21.842 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QbpYT+kv11T5Q/1C6hiCkF87Z0s= 2022-09-12T20:16:21.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QbpYT+kv11T5Q/1C6hiCkF87Z0s=', 'Content-Length': '0'} 2022-09-12T20:16:21.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000978fdedeeea4f077-00631f9395-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.846 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:21.846 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:21.847 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:21.847 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:21.848 DEBUG:boto:Method: GET 2022-09-12T20:16:21.848 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:21.849 DEBUG:boto:Data: 2022-09-12T20:16:21.850 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.850 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.851 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.851 DEBUG:boto:Params: {} 2022-09-12T20:16:21.852 DEBUG:boto:Token: None 2022-09-12T20:16:21.852 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:21.853 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QbpYT+kv11T5Q/1C6hiCkF87Z0s= 2022-09-12T20:16:21.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QbpYT+kv11T5Q/1C6hiCkF87Z0s=', 'Content-Length': '0'} 2022-09-12T20:16:21.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac79558ef1dda670-00631f9395-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.856 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:21.857 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:21.857 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:21.858 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:21.858 DEBUG:boto:Method: HEAD 2022-09-12T20:16:21.859 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:21.859 DEBUG:boto:Data: 2022-09-12T20:16:21.860 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.860 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.861 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.862 DEBUG:boto:Params: {} 2022-09-12T20:16:21.862 DEBUG:boto:Token: None 2022-09-12T20:16:21.863 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:21.863 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t3ki4I/0p8I3/rxSTbeeqFx2Y2g= 2022-09-12T20:16:21.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t3ki4I/0p8I3/rxSTbeeqFx2Y2g=', 'Content-Length': '0'} 2022-09-12T20:16:21.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce76c8c4f7163c36-00631f9395-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.866 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:21.867 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:21.867 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:21.868 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:21.868 DEBUG:boto:Method: HEAD 2022-09-12T20:16:21.869 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:21.869 DEBUG:boto:Data: 2022-09-12T20:16:21.870 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.870 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.871 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.871 DEBUG:boto:Params: {} 2022-09-12T20:16:21.872 DEBUG:boto:Token: None 2022-09-12T20:16:21.873 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:21.873 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:t3ki4I/0p8I3/rxSTbeeqFx2Y2g= 2022-09-12T20:16:21.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:t3ki4I/0p8I3/rxSTbeeqFx2Y2g=', 'Content-Length': '0'} 2022-09-12T20:16:21.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7abfc9059f860fb-00631f9395-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:16:21.877 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:21.877 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:21.878 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:21.879 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:21.879 DEBUG:boto:Method: GET 2022-09-12T20:16:21.880 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:21.880 DEBUG:boto:Data: 2022-09-12T20:16:21.881 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.881 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.882 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.883 DEBUG:boto:Params: {} 2022-09-12T20:16:21.883 DEBUG:boto:Token: None 2022-09-12T20:16:21.884 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:21.885 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QbpYT+kv11T5Q/1C6hiCkF87Z0s= 2022-09-12T20:16:21.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QbpYT+kv11T5Q/1C6hiCkF87Z0s=', 'Content-Length': '0'} 2022-09-12T20:16:21.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001808156a154bc8f0-00631f9395-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.888 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:21.889 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:21.889 DEBUG:boto:path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:21.890 DEBUG:boto:auth_path=/jdpiak-75/obj58?versionId=null 2022-09-12T20:16:21.890 DEBUG:boto:Method: GET 2022-09-12T20:16:21.891 DEBUG:boto:Path: /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:21.891 DEBUG:boto:Data: 2022-09-12T20:16:21.892 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.892 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.893 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.893 DEBUG:boto:Params: {} 2022-09-12T20:16:21.894 DEBUG:boto:Token: None 2022-09-12T20:16:21.895 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj58?versionId=null 2022-09-12T20:16:21.895 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QbpYT+kv11T5Q/1C6hiCkF87Z0s= 2022-09-12T20:16:21.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QbpYT+kv11T5Q/1C6hiCkF87Z0s=', 'Content-Length': '0'} 2022-09-12T20:16:21.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b232e93d9039d74-00631f9395-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.899 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:21.900 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:21.900 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:16:21.901 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:16:21.901 DEBUG:boto:Method: GET 2022-09-12T20:16:21.902 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:16:21.902 DEBUG:boto:Data: 2022-09-12T20:16:21.903 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.903 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.904 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.904 DEBUG:boto:Params: {} 2022-09-12T20:16:21.905 DEBUG:boto:Token: None 2022-09-12T20:16:21.905 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/ 2022-09-12T20:16:21.906 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E= 2022-09-12T20:16:21.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E=', 'Content-Length': '0'} 2022-09-12T20:16:21.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091984240ffeccaac-00631f9395-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.910 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:21.911 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:21.912 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:21.912 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:16:21.913 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:16:21.914 DEBUG:boto:Method: GET 2022-09-12T20:16:21.915 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:16:21.916 DEBUG:boto:Data: 2022-09-12T20:16:21.917 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.918 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.919 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.920 DEBUG:boto:Params: {} 2022-09-12T20:16:21.921 DEBUG:boto:Token: None 2022-09-12T20:16:21.922 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/ 2022-09-12T20:16:21.923 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E= 2022-09-12T20:16:21.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KoZQLEtB/DvN/N2Gcmw23xRyw/E=', 'Content-Length': '0'} 2022-09-12T20:16:21.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ada5a38cf4952d10-00631f9395-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.970 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:21.971 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:21.972 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:21.972 DEBUG:boto:Method: HEAD 2022-09-12T20:16:21.973 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:16:21.973 DEBUG:boto:Data: 2022-09-12T20:16:21.974 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.974 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.975 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.975 DEBUG:boto:Params: {} 2022-09-12T20:16:21.976 DEBUG:boto:Token: None 2022-09-12T20:16:21.977 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj58 2022-09-12T20:16:21.977 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rhAWIh95W6VcWuV9uNA1kUWexao= 2022-09-12T20:16:21.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rhAWIh95W6VcWuV9uNA1kUWexao=', 'Content-Length': '0'} 2022-09-12T20:16:21.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c76ca2d63cc0816-00631f9395-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.980 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:21.981 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:21.982 DEBUG:boto:Method: HEAD 2022-09-12T20:16:21.982 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:16:21.983 DEBUG:boto:Data: 2022-09-12T20:16:21.983 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.984 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:21.984 DEBUG:boto:Port: 8000 2022-09-12T20:16:21.985 DEBUG:boto:Params: {} 2022-09-12T20:16:21.985 DEBUG:boto:Token: None 2022-09-12T20:16:21.986 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj58 2022-09-12T20:16:21.986 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rhAWIh95W6VcWuV9uNA1kUWexao= 2022-09-12T20:16:21.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rhAWIh95W6VcWuV9uNA1kUWexao=', 'Content-Length': '0'} 2022-09-12T20:16:21.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8e50b7387152ae9-00631f9395-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:21.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-12T20:16:21.990 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:21.991 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:21.991 DEBUG:boto:Method: GET 2022-09-12T20:16:21.992 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:16:21.993 DEBUG:boto:Data: 2022-09-12T20:16:21.993 DEBUG:boto:Headers: {} 2022-09-12T20:16:21.994 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:21.994 DEBUG:boto:Port: 8001 2022-09-12T20:16:21.995 DEBUG:boto:Params: {} 2022-09-12T20:16:21.995 DEBUG:boto:Token: None 2022-09-12T20:16:21.996 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:21 GMT /jdpiak-75/obj58 2022-09-12T20:16:21.997 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9jCQaayuKsW8AQ1VretHxLL6Xbs= 2022-09-12T20:16:21.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:21 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9jCQaayuKsW8AQ1VretHxLL6Xbs=', 'Content-Length': '0'} 2022-09-12T20:16:21.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006473d291ce3d01d6-00631f9395-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.000 DEBUG:boto:path=/jdpiak-75/obj58 2022-09-12T20:16:22.001 DEBUG:boto:auth_path=/jdpiak-75/obj58 2022-09-12T20:16:22.001 DEBUG:boto:Method: GET 2022-09-12T20:16:22.002 DEBUG:boto:Path: /jdpiak-75/obj58 2022-09-12T20:16:22.002 DEBUG:boto:Data: 2022-09-12T20:16:22.003 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.003 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.004 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.004 DEBUG:boto:Params: {} 2022-09-12T20:16:22.005 DEBUG:boto:Token: None 2022-09-12T20:16:22.005 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj58 2022-09-12T20:16:22.006 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TErAcq28itLAXAl0teOaHRZGgfk= 2022-09-12T20:16:22.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TErAcq28itLAXAl0teOaHRZGgfk=', 'Content-Length': '0'} 2022-09-12T20:16:22.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db69e3fa9c48c91b-00631f9396-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.010 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:22.010 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:22.011 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:16:22.011 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:16:22.012 DEBUG:boto:Method: GET 2022-09-12T20:16:22.012 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:16:22.013 DEBUG:boto:Data: 2022-09-12T20:16:22.013 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.014 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.014 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.015 DEBUG:boto:Params: {} 2022-09-12T20:16:22.015 DEBUG:boto:Token: None 2022-09-12T20:16:22.016 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/ 2022-09-12T20:16:22.017 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM= 2022-09-12T20:16:22.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM=', 'Content-Length': '0'} 2022-09-12T20:16:22.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cf813af51c8de98-00631f9396-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.021 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:22.022 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:22.023 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:22.023 DEBUG:boto:path=/jdpiak-75/?prefix=obj58 2022-09-12T20:16:22.024 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj58 2022-09-12T20:16:22.024 DEBUG:boto:Method: GET 2022-09-12T20:16:22.025 DEBUG:boto:Path: /jdpiak-75/?prefix=obj58 2022-09-12T20:16:22.025 DEBUG:boto:Data: 2022-09-12T20:16:22.026 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.027 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.027 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.028 DEBUG:boto:Params: {} 2022-09-12T20:16:22.028 DEBUG:boto:Token: None 2022-09-12T20:16:22.029 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/ 2022-09-12T20:16:22.030 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM= 2022-09-12T20:16:22.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM=', 'Content-Length': '0'} 2022-09-12T20:16:22.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ca7de4d6f48ac6a-00631f9396-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.034 DEBUG:boto:b'jdpiak-75obj581000falseobj582022-09-12T20:10:08.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:22.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-12T20:16:22.035 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:22.036 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:22.036 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:22.037 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:22.037 DEBUG:boto:Method: GET 2022-09-12T20:16:22.038 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:22.039 DEBUG:boto:Data: 2022-09-12T20:16:22.039 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.040 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.040 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.041 DEBUG:boto:Params: {} 2022-09-12T20:16:22.041 DEBUG:boto:Token: None 2022-09-12T20:16:22.042 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:22.043 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:I38XZ7v5cTRpZuxmjnIN+2z/I9U= 2022-09-12T20:16:22.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:I38XZ7v5cTRpZuxmjnIN+2z/I9U=', 'Content-Length': '0'} 2022-09-12T20:16:22.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d09e395445163667-00631f9396-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.046 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:22.047 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:22.047 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:22.048 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:22.048 DEBUG:boto:Method: GET 2022-09-12T20:16:22.049 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:22.050 DEBUG:boto:Data: 2022-09-12T20:16:22.050 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.051 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.051 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.052 DEBUG:boto:Params: {} 2022-09-12T20:16:22.053 DEBUG:boto:Token: None 2022-09-12T20:16:22.053 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:22.054 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:I38XZ7v5cTRpZuxmjnIN+2z/I9U= 2022-09-12T20:16:22.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:I38XZ7v5cTRpZuxmjnIN+2z/I9U=', 'Content-Length': '0'} 2022-09-12T20:16:22.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000015dd49ae8eac735-00631f9396-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.058 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:22.058 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:22.059 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:22.059 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:22.060 DEBUG:boto:Method: HEAD 2022-09-12T20:16:22.060 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:22.061 DEBUG:boto:Data: 2022-09-12T20:16:22.061 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.062 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.062 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.063 DEBUG:boto:Params: {} 2022-09-12T20:16:22.064 DEBUG:boto:Token: None 2022-09-12T20:16:22.064 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:22.065 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/aPFlynPsABrnVKT263et3WRAHk= 2022-09-12T20:16:22.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/aPFlynPsABrnVKT263et3WRAHk=', 'Content-Length': '0'} 2022-09-12T20:16:22.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9fd7ae2592b2f1f-00631f9396-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.068 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:22.068 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:22.069 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:22.070 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:22.070 DEBUG:boto:Method: HEAD 2022-09-12T20:16:22.071 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:22.071 DEBUG:boto:Data: 2022-09-12T20:16:22.072 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.072 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.073 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.073 DEBUG:boto:Params: {} 2022-09-12T20:16:22.074 DEBUG:boto:Token: None 2022-09-12T20:16:22.074 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:22.075 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/aPFlynPsABrnVKT263et3WRAHk= 2022-09-12T20:16:22.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/aPFlynPsABrnVKT263et3WRAHk=', 'Content-Length': '0'} 2022-09-12T20:16:22.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f5a434e608a5aa7-00631f9396-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-12T20:16:22.079 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:22.080 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:22.081 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:22.081 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:22.082 DEBUG:boto:Method: GET 2022-09-12T20:16:22.082 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:22.083 DEBUG:boto:Data: 2022-09-12T20:16:22.083 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.084 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.084 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.085 DEBUG:boto:Params: {} 2022-09-12T20:16:22.085 DEBUG:boto:Token: None 2022-09-12T20:16:22.086 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:22.087 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:I38XZ7v5cTRpZuxmjnIN+2z/I9U= 2022-09-12T20:16:22.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:I38XZ7v5cTRpZuxmjnIN+2z/I9U=', 'Content-Length': '0'} 2022-09-12T20:16:22.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037191912451672a9-00631f9396-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.090 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:22.091 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:22.091 DEBUG:boto:path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:22.092 DEBUG:boto:auth_path=/jdpiak-75/obj62?versionId=null 2022-09-12T20:16:22.092 DEBUG:boto:Method: GET 2022-09-12T20:16:22.093 DEBUG:boto:Path: /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:22.094 DEBUG:boto:Data: 2022-09-12T20:16:22.094 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.095 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.095 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.096 DEBUG:boto:Params: {} 2022-09-12T20:16:22.096 DEBUG:boto:Token: None 2022-09-12T20:16:22.097 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj62?versionId=null 2022-09-12T20:16:22.097 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:I38XZ7v5cTRpZuxmjnIN+2z/I9U= 2022-09-12T20:16:22.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:I38XZ7v5cTRpZuxmjnIN+2z/I9U=', 'Content-Length': '0'} 2022-09-12T20:16:22.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e19f6e67cdbbfd7-00631f9396-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.101 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:22.102 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:22.102 DEBUG:boto:path=/jdpiak-75/?prefix=obj62 2022-09-12T20:16:22.103 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj62 2022-09-12T20:16:22.103 DEBUG:boto:Method: GET 2022-09-12T20:16:22.104 DEBUG:boto:Path: /jdpiak-75/?prefix=obj62 2022-09-12T20:16:22.105 DEBUG:boto:Data: 2022-09-12T20:16:22.105 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.106 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.106 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.107 DEBUG:boto:Params: {} 2022-09-12T20:16:22.107 DEBUG:boto:Token: None 2022-09-12T20:16:22.108 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/ 2022-09-12T20:16:22.109 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM= 2022-09-12T20:16:22.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM=', 'Content-Length': '0'} 2022-09-12T20:16:22.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a48a33fa48b083d-00631f9396-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.113 DEBUG:boto:b'jdpiak-75obj621000falseobj622022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:22.114 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:22.115 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:22.115 DEBUG:boto:path=/jdpiak-75/?prefix=obj62 2022-09-12T20:16:22.116 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj62 2022-09-12T20:16:22.116 DEBUG:boto:Method: GET 2022-09-12T20:16:22.117 DEBUG:boto:Path: /jdpiak-75/?prefix=obj62 2022-09-12T20:16:22.118 DEBUG:boto:Data: 2022-09-12T20:16:22.119 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.119 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.120 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.120 DEBUG:boto:Params: {} 2022-09-12T20:16:22.121 DEBUG:boto:Token: None 2022-09-12T20:16:22.121 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/ 2022-09-12T20:16:22.122 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM= 2022-09-12T20:16:22.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM=', 'Content-Length': '0'} 2022-09-12T20:16:22.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000827fc07b186ee5f4-00631f9396-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.126 DEBUG:boto:b'jdpiak-75obj621000falseobj622022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:22.127 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:22.128 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:22.128 DEBUG:boto:Method: HEAD 2022-09-12T20:16:22.129 DEBUG:boto:Path: /jdpiak-75/obj62 2022-09-12T20:16:22.129 DEBUG:boto:Data: 2022-09-12T20:16:22.130 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.130 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.131 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.131 DEBUG:boto:Params: {} 2022-09-12T20:16:22.132 DEBUG:boto:Token: None 2022-09-12T20:16:22.133 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj62 2022-09-12T20:16:22.133 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nz4F1VL2wqS4857nuhjz/kJ5QPA= 2022-09-12T20:16:22.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nz4F1VL2wqS4857nuhjz/kJ5QPA=', 'Content-Length': '0'} 2022-09-12T20:16:22.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016211307524caf04-00631f9396-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.137 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:22.137 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:22.138 DEBUG:boto:Method: HEAD 2022-09-12T20:16:22.139 DEBUG:boto:Path: /jdpiak-75/obj62 2022-09-12T20:16:22.139 DEBUG:boto:Data: 2022-09-12T20:16:22.140 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.140 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.141 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.141 DEBUG:boto:Params: {} 2022-09-12T20:16:22.142 DEBUG:boto:Token: None 2022-09-12T20:16:22.142 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj62 2022-09-12T20:16:22.143 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nz4F1VL2wqS4857nuhjz/kJ5QPA= 2022-09-12T20:16:22.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nz4F1VL2wqS4857nuhjz/kJ5QPA=', 'Content-Length': '0'} 2022-09-12T20:16:22.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e07a8ba798e6cd4f-00631f9396-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-12T20:16:22.147 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:22.148 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:22.148 DEBUG:boto:Method: GET 2022-09-12T20:16:22.149 DEBUG:boto:Path: /jdpiak-75/obj62 2022-09-12T20:16:22.150 DEBUG:boto:Data: 2022-09-12T20:16:22.150 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.151 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.151 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.152 DEBUG:boto:Params: {} 2022-09-12T20:16:22.152 DEBUG:boto:Token: None 2022-09-12T20:16:22.153 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj62 2022-09-12T20:16:22.153 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zt8K7Vr1+7zGDnptgplWFdw3DyE= 2022-09-12T20:16:22.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zt8K7Vr1+7zGDnptgplWFdw3DyE=', 'Content-Length': '0'} 2022-09-12T20:16:22.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095fcc4e533d4c03b-00631f9396-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.157 DEBUG:boto:path=/jdpiak-75/obj62 2022-09-12T20:16:22.158 DEBUG:boto:auth_path=/jdpiak-75/obj62 2022-09-12T20:16:22.158 DEBUG:boto:Method: GET 2022-09-12T20:16:22.159 DEBUG:boto:Path: /jdpiak-75/obj62 2022-09-12T20:16:22.159 DEBUG:boto:Data: 2022-09-12T20:16:22.160 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.160 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.161 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.161 DEBUG:boto:Params: {} 2022-09-12T20:16:22.162 DEBUG:boto:Token: None 2022-09-12T20:16:22.163 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj62 2022-09-12T20:16:22.163 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zt8K7Vr1+7zGDnptgplWFdw3DyE= 2022-09-12T20:16:22.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zt8K7Vr1+7zGDnptgplWFdw3DyE=', 'Content-Length': '0'} 2022-09-12T20:16:22.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047ff8be11473e27c-00631f9396-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.167 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:22.168 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:22.168 DEBUG:boto:path=/jdpiak-75/?prefix=obj62 2022-09-12T20:16:22.169 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj62 2022-09-12T20:16:22.169 DEBUG:boto:Method: GET 2022-09-12T20:16:22.170 DEBUG:boto:Path: /jdpiak-75/?prefix=obj62 2022-09-12T20:16:22.170 DEBUG:boto:Data: 2022-09-12T20:16:22.171 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.171 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.172 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.172 DEBUG:boto:Params: {} 2022-09-12T20:16:22.173 DEBUG:boto:Token: None 2022-09-12T20:16:22.174 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/ 2022-09-12T20:16:22.174 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM= 2022-09-12T20:16:22.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM=', 'Content-Length': '0'} 2022-09-12T20:16:22.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f1cd014bcd04d3a-00631f9396-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.178 DEBUG:boto:b'jdpiak-75obj621000falseobj622022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:22.179 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:22.180 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:22.180 DEBUG:boto:path=/jdpiak-75/?prefix=obj62 2022-09-12T20:16:22.181 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj62 2022-09-12T20:16:22.181 DEBUG:boto:Method: GET 2022-09-12T20:16:22.182 DEBUG:boto:Path: /jdpiak-75/?prefix=obj62 2022-09-12T20:16:22.182 DEBUG:boto:Data: 2022-09-12T20:16:22.183 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.183 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.184 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.184 DEBUG:boto:Params: {} 2022-09-12T20:16:22.185 DEBUG:boto:Token: None 2022-09-12T20:16:22.185 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/ 2022-09-12T20:16:22.186 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM= 2022-09-12T20:16:22.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM=', 'Content-Length': '0'} 2022-09-12T20:16:22.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d34b2abb8a9bb5eb-00631f9396-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.230 DEBUG:boto:b'jdpiak-75obj621000falseobj622022-09-12T20:12:24.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:22.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:16:22.232 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:22.232 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:22.233 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:22.233 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:22.234 DEBUG:boto:Method: GET 2022-09-12T20:16:22.234 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:22.235 DEBUG:boto:Data: 2022-09-12T20:16:22.235 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.236 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.236 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.237 DEBUG:boto:Params: {} 2022-09-12T20:16:22.238 DEBUG:boto:Token: None 2022-09-12T20:16:22.238 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:22.239 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gCXFe3ZAk0536ny0IAUGYChovy8= 2022-09-12T20:16:22.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gCXFe3ZAk0536ny0IAUGYChovy8=', 'Content-Length': '0'} 2022-09-12T20:16:22.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f029d413d3a15243-00631f9396-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.243 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:22.243 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:22.244 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:22.244 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:22.245 DEBUG:boto:Method: GET 2022-09-12T20:16:22.245 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:22.246 DEBUG:boto:Data: 2022-09-12T20:16:22.246 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.247 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.247 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.248 DEBUG:boto:Params: {} 2022-09-12T20:16:22.249 DEBUG:boto:Token: None 2022-09-12T20:16:22.249 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:22.250 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gCXFe3ZAk0536ny0IAUGYChovy8= 2022-09-12T20:16:22.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gCXFe3ZAk0536ny0IAUGYChovy8=', 'Content-Length': '0'} 2022-09-12T20:16:22.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001000c4bf04e6ee46-00631f9396-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.253 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:22.254 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:22.255 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:22.255 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:22.256 DEBUG:boto:Method: HEAD 2022-09-12T20:16:22.256 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:22.257 DEBUG:boto:Data: 2022-09-12T20:16:22.257 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.258 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.258 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.259 DEBUG:boto:Params: {} 2022-09-12T20:16:22.259 DEBUG:boto:Token: None 2022-09-12T20:16:22.260 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:22.261 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WLfz0U/U/MC3MrUjcazK4cxDFoA= 2022-09-12T20:16:22.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WLfz0U/U/MC3MrUjcazK4cxDFoA=', 'Content-Length': '0'} 2022-09-12T20:16:22.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d7610188e5de282-00631f9396-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.264 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:22.265 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:22.265 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:22.266 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:22.266 DEBUG:boto:Method: HEAD 2022-09-12T20:16:22.267 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:22.267 DEBUG:boto:Data: 2022-09-12T20:16:22.268 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.268 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.269 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.269 DEBUG:boto:Params: {} 2022-09-12T20:16:22.270 DEBUG:boto:Token: None 2022-09-12T20:16:22.271 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:22.271 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:WLfz0U/U/MC3MrUjcazK4cxDFoA= 2022-09-12T20:16:22.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:WLfz0U/U/MC3MrUjcazK4cxDFoA=', 'Content-Length': '0'} 2022-09-12T20:16:22.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094bd023536c66813-00631f9396-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:16:22.276 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:22.277 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:22.277 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:22.278 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:22.278 DEBUG:boto:Method: GET 2022-09-12T20:16:22.279 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:22.280 DEBUG:boto:Data: 2022-09-12T20:16:22.280 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.281 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.281 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.282 DEBUG:boto:Params: {} 2022-09-12T20:16:22.282 DEBUG:boto:Token: None 2022-09-12T20:16:22.283 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:22.284 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gCXFe3ZAk0536ny0IAUGYChovy8= 2022-09-12T20:16:22.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gCXFe3ZAk0536ny0IAUGYChovy8=', 'Content-Length': '0'} 2022-09-12T20:16:22.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075c8b4cf408c338a-00631f9396-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.330 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:22.331 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:22.331 DEBUG:boto:path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:22.332 DEBUG:boto:auth_path=/jdpiak-75/obj68?versionId=null 2022-09-12T20:16:22.332 DEBUG:boto:Method: GET 2022-09-12T20:16:22.333 DEBUG:boto:Path: /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:22.333 DEBUG:boto:Data: 2022-09-12T20:16:22.334 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.335 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.335 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.336 DEBUG:boto:Params: {} 2022-09-12T20:16:22.336 DEBUG:boto:Token: None 2022-09-12T20:16:22.337 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj68?versionId=null 2022-09-12T20:16:22.338 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gCXFe3ZAk0536ny0IAUGYChovy8= 2022-09-12T20:16:22.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gCXFe3ZAk0536ny0IAUGYChovy8=', 'Content-Length': '0'} 2022-09-12T20:16:22.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea16538f173a1695-00631f9396-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.341 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:22.342 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:22.343 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:16:22.343 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:16:22.344 DEBUG:boto:Method: GET 2022-09-12T20:16:22.344 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:16:22.345 DEBUG:boto:Data: 2022-09-12T20:16:22.346 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.346 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.347 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.347 DEBUG:boto:Params: {} 2022-09-12T20:16:22.348 DEBUG:boto:Token: None 2022-09-12T20:16:22.348 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/ 2022-09-12T20:16:22.349 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM= 2022-09-12T20:16:22.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM=', 'Content-Length': '0'} 2022-09-12T20:16:22.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c466e0956aac8403-00631f9396-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.353 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:22.354 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:22.355 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:22.355 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:16:22.356 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:16:22.356 DEBUG:boto:Method: GET 2022-09-12T20:16:22.357 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:16:22.358 DEBUG:boto:Data: 2022-09-12T20:16:22.358 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.359 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.359 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.360 DEBUG:boto:Params: {} 2022-09-12T20:16:22.360 DEBUG:boto:Token: None 2022-09-12T20:16:22.361 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/ 2022-09-12T20:16:22.362 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM= 2022-09-12T20:16:22.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM=', 'Content-Length': '0'} 2022-09-12T20:16:22.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce8f9825b0608fb4-00631f9396-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.366 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:22.367 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:22.367 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:22.368 DEBUG:boto:Method: HEAD 2022-09-12T20:16:22.368 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:16:22.369 DEBUG:boto:Data: 2022-09-12T20:16:22.369 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.370 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.371 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.371 DEBUG:boto:Params: {} 2022-09-12T20:16:22.372 DEBUG:boto:Token: None 2022-09-12T20:16:22.372 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj68 2022-09-12T20:16:22.373 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:klhLpy7woCkGOzeAjhhQocbzVM4= 2022-09-12T20:16:22.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:klhLpy7woCkGOzeAjhhQocbzVM4=', 'Content-Length': '0'} 2022-09-12T20:16:22.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055573975cd9f5c3e-00631f9396-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.378 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:22.378 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:22.379 DEBUG:boto:Method: HEAD 2022-09-12T20:16:22.379 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:16:22.380 DEBUG:boto:Data: 2022-09-12T20:16:22.380 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.381 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.381 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.382 DEBUG:boto:Params: {} 2022-09-12T20:16:22.382 DEBUG:boto:Token: None 2022-09-12T20:16:22.383 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj68 2022-09-12T20:16:22.383 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:klhLpy7woCkGOzeAjhhQocbzVM4= 2022-09-12T20:16:22.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:klhLpy7woCkGOzeAjhhQocbzVM4=', 'Content-Length': '0'} 2022-09-12T20:16:22.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6d5f5f7b6b9785c-00631f9396-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-12T20:16:22.387 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:22.388 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:22.389 DEBUG:boto:Method: GET 2022-09-12T20:16:22.390 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:16:22.390 DEBUG:boto:Data: 2022-09-12T20:16:22.391 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.392 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.392 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.393 DEBUG:boto:Params: {} 2022-09-12T20:16:22.394 DEBUG:boto:Token: None 2022-09-12T20:16:22.395 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj68 2022-09-12T20:16:22.396 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:I24eQzz2hx8Nq2J60p/shq3H1uY= 2022-09-12T20:16:22.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:I24eQzz2hx8Nq2J60p/shq3H1uY=', 'Content-Length': '0'} 2022-09-12T20:16:22.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4426c6718afd774-00631f9396-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.399 DEBUG:boto:path=/jdpiak-75/obj68 2022-09-12T20:16:22.400 DEBUG:boto:auth_path=/jdpiak-75/obj68 2022-09-12T20:16:22.400 DEBUG:boto:Method: GET 2022-09-12T20:16:22.401 DEBUG:boto:Path: /jdpiak-75/obj68 2022-09-12T20:16:22.402 DEBUG:boto:Data: 2022-09-12T20:16:22.402 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.403 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.404 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.405 DEBUG:boto:Params: {} 2022-09-12T20:16:22.406 DEBUG:boto:Token: None 2022-09-12T20:16:22.407 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj68 2022-09-12T20:16:22.408 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:I24eQzz2hx8Nq2J60p/shq3H1uY= 2022-09-12T20:16:22.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:I24eQzz2hx8Nq2J60p/shq3H1uY=', 'Content-Length': '0'} 2022-09-12T20:16:22.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb3ef55dd7c4d46d-00631f9396-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.414 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:22.415 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:22.415 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:16:22.416 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:16:22.416 DEBUG:boto:Method: GET 2022-09-12T20:16:22.417 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:16:22.418 DEBUG:boto:Data: 2022-09-12T20:16:22.418 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.419 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.420 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.420 DEBUG:boto:Params: {} 2022-09-12T20:16:22.421 DEBUG:boto:Token: None 2022-09-12T20:16:22.422 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/ 2022-09-12T20:16:22.422 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM= 2022-09-12T20:16:22.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM=', 'Content-Length': '0'} 2022-09-12T20:16:22.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000437903fd6e06e9ef-00631f9396-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.427 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:22.428 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:22.428 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:22.429 DEBUG:boto:path=/jdpiak-75/?prefix=obj68 2022-09-12T20:16:22.429 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj68 2022-09-12T20:16:22.430 DEBUG:boto:Method: GET 2022-09-12T20:16:22.431 DEBUG:boto:Path: /jdpiak-75/?prefix=obj68 2022-09-12T20:16:22.431 DEBUG:boto:Data: 2022-09-12T20:16:22.432 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.432 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.433 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.434 DEBUG:boto:Params: {} 2022-09-12T20:16:22.434 DEBUG:boto:Token: None 2022-09-12T20:16:22.435 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/ 2022-09-12T20:16:22.435 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM= 2022-09-12T20:16:22.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM=', 'Content-Length': '0'} 2022-09-12T20:16:22.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002c13700aaff6c1e-00631f9396-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.441 DEBUG:boto:b'jdpiak-75obj681000falseobj682022-09-12T20:03:28.622Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:22.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:16:22.442 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:22.443 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:22.444 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:22.444 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:22.446 DEBUG:boto:Method: GET 2022-09-12T20:16:22.446 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:22.447 DEBUG:boto:Data: 2022-09-12T20:16:22.448 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.448 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.449 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.449 DEBUG:boto:Params: {} 2022-09-12T20:16:22.450 DEBUG:boto:Token: None 2022-09-12T20:16:22.451 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:22.452 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sGv8aY9IngLueZiISr5mm8V0iPQ= 2022-09-12T20:16:22.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sGv8aY9IngLueZiISr5mm8V0iPQ=', 'Content-Length': '0'} 2022-09-12T20:16:22.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abe81b0276ecd3be-00631f9396-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.498 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:22.499 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:22.500 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:22.501 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:22.501 DEBUG:boto:Method: GET 2022-09-12T20:16:22.502 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:22.503 DEBUG:boto:Data: 2022-09-12T20:16:22.504 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.505 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.506 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.507 DEBUG:boto:Params: {} 2022-09-12T20:16:22.508 DEBUG:boto:Token: None 2022-09-12T20:16:22.509 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:22.510 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sGv8aY9IngLueZiISr5mm8V0iPQ= 2022-09-12T20:16:22.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sGv8aY9IngLueZiISr5mm8V0iPQ=', 'Content-Length': '0'} 2022-09-12T20:16:22.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad573a27bdfbc4e8-00631f9396-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.515 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:22.515 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:22.516 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:22.517 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:22.517 DEBUG:boto:Method: HEAD 2022-09-12T20:16:22.518 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:22.518 DEBUG:boto:Data: 2022-09-12T20:16:22.519 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.520 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.521 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.522 DEBUG:boto:Params: {} 2022-09-12T20:16:22.523 DEBUG:boto:Token: None 2022-09-12T20:16:22.523 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:22.524 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eymHyjRP3GpGMIjcteYLk5Y5Ijs= 2022-09-12T20:16:22.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eymHyjRP3GpGMIjcteYLk5Y5Ijs=', 'Content-Length': '0'} 2022-09-12T20:16:22.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa84f148d1e8948d-00631f9396-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.529 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:22.529 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:22.530 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:22.531 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:22.531 DEBUG:boto:Method: HEAD 2022-09-12T20:16:22.532 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:22.533 DEBUG:boto:Data: 2022-09-12T20:16:22.534 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.534 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.535 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.536 DEBUG:boto:Params: {} 2022-09-12T20:16:22.537 DEBUG:boto:Token: None 2022-09-12T20:16:22.538 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:22.539 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eymHyjRP3GpGMIjcteYLk5Y5Ijs= 2022-09-12T20:16:22.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eymHyjRP3GpGMIjcteYLk5Y5Ijs=', 'Content-Length': '0'} 2022-09-12T20:16:22.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aed928e4aab58016-00631f9396-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:16:22.544 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:22.545 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:22.545 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:22.546 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:22.547 DEBUG:boto:Method: GET 2022-09-12T20:16:22.548 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:22.549 DEBUG:boto:Data: 2022-09-12T20:16:22.549 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.550 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.551 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.551 DEBUG:boto:Params: {} 2022-09-12T20:16:22.552 DEBUG:boto:Token: None 2022-09-12T20:16:22.552 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:22.553 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sGv8aY9IngLueZiISr5mm8V0iPQ= 2022-09-12T20:16:22.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sGv8aY9IngLueZiISr5mm8V0iPQ=', 'Content-Length': '0'} 2022-09-12T20:16:22.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1d73ab00b41ca7b-00631f9396-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.557 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:22.558 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:22.559 DEBUG:boto:path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:22.559 DEBUG:boto:auth_path=/jdpiak-75/obj69?versionId=null 2022-09-12T20:16:22.560 DEBUG:boto:Method: GET 2022-09-12T20:16:22.561 DEBUG:boto:Path: /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:22.561 DEBUG:boto:Data: 2022-09-12T20:16:22.562 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.563 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.564 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.564 DEBUG:boto:Params: {} 2022-09-12T20:16:22.565 DEBUG:boto:Token: None 2022-09-12T20:16:22.566 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj69?versionId=null 2022-09-12T20:16:22.567 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sGv8aY9IngLueZiISr5mm8V0iPQ= 2022-09-12T20:16:22.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sGv8aY9IngLueZiISr5mm8V0iPQ=', 'Content-Length': '0'} 2022-09-12T20:16:22.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004884593e2fedd6e6-00631f9396-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.571 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:22.571 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:22.572 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:16:22.572 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:16:22.573 DEBUG:boto:Method: GET 2022-09-12T20:16:22.574 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:16:22.574 DEBUG:boto:Data: 2022-09-12T20:16:22.575 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.575 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.576 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.576 DEBUG:boto:Params: {} 2022-09-12T20:16:22.577 DEBUG:boto:Token: None 2022-09-12T20:16:22.577 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/ 2022-09-12T20:16:22.578 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM= 2022-09-12T20:16:22.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM=', 'Content-Length': '0'} 2022-09-12T20:16:22.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007febd60da8b6d581-00631f9396-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.583 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:22.583 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:22.584 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:22.585 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:16:22.585 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:16:22.586 DEBUG:boto:Method: GET 2022-09-12T20:16:22.586 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:16:22.587 DEBUG:boto:Data: 2022-09-12T20:16:22.587 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.588 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.588 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.589 DEBUG:boto:Params: {} 2022-09-12T20:16:22.589 DEBUG:boto:Token: None 2022-09-12T20:16:22.590 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/ 2022-09-12T20:16:22.591 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM= 2022-09-12T20:16:22.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM=', 'Content-Length': '0'} 2022-09-12T20:16:22.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dad151535d22ad93-00631f9396-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.638 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:22.639 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:22.640 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:22.640 DEBUG:boto:Method: HEAD 2022-09-12T20:16:22.641 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:16:22.641 DEBUG:boto:Data: 2022-09-12T20:16:22.642 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.642 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.643 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.643 DEBUG:boto:Params: {} 2022-09-12T20:16:22.644 DEBUG:boto:Token: None 2022-09-12T20:16:22.645 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj69 2022-09-12T20:16:22.645 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:99r0DRErLH9ubtPnZ8R9Q70pGOo= 2022-09-12T20:16:22.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:99r0DRErLH9ubtPnZ8R9Q70pGOo=', 'Content-Length': '0'} 2022-09-12T20:16:22.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d84e8f433a086260-00631f9396-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.649 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:22.649 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:22.650 DEBUG:boto:Method: HEAD 2022-09-12T20:16:22.650 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:16:22.651 DEBUG:boto:Data: 2022-09-12T20:16:22.652 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.652 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.653 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.653 DEBUG:boto:Params: {} 2022-09-12T20:16:22.654 DEBUG:boto:Token: None 2022-09-12T20:16:22.654 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj69 2022-09-12T20:16:22.655 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:99r0DRErLH9ubtPnZ8R9Q70pGOo= 2022-09-12T20:16:22.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:99r0DRErLH9ubtPnZ8R9Q70pGOo=', 'Content-Length': '0'} 2022-09-12T20:16:22.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002054090658e95230-00631f9396-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-12T20:16:22.659 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:22.660 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:22.660 DEBUG:boto:Method: GET 2022-09-12T20:16:22.661 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:16:22.661 DEBUG:boto:Data: 2022-09-12T20:16:22.662 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.662 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.663 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.663 DEBUG:boto:Params: {} 2022-09-12T20:16:22.664 DEBUG:boto:Token: None 2022-09-12T20:16:22.664 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj69 2022-09-12T20:16:22.665 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y3VszGzOzrAy27JQWUpuMH1qAa0= 2022-09-12T20:16:22.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y3VszGzOzrAy27JQWUpuMH1qAa0=', 'Content-Length': '0'} 2022-09-12T20:16:22.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c31014079cf1f2e-00631f9396-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.669 DEBUG:boto:path=/jdpiak-75/obj69 2022-09-12T20:16:22.669 DEBUG:boto:auth_path=/jdpiak-75/obj69 2022-09-12T20:16:22.670 DEBUG:boto:Method: GET 2022-09-12T20:16:22.670 DEBUG:boto:Path: /jdpiak-75/obj69 2022-09-12T20:16:22.671 DEBUG:boto:Data: 2022-09-12T20:16:22.671 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.672 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.672 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.673 DEBUG:boto:Params: {} 2022-09-12T20:16:22.674 DEBUG:boto:Token: None 2022-09-12T20:16:22.674 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj69 2022-09-12T20:16:22.675 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y3VszGzOzrAy27JQWUpuMH1qAa0= 2022-09-12T20:16:22.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y3VszGzOzrAy27JQWUpuMH1qAa0=', 'Content-Length': '0'} 2022-09-12T20:16:22.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002703879595675d07-00631f9396-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.679 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:22.679 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:22.680 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:16:22.680 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:16:22.681 DEBUG:boto:Method: GET 2022-09-12T20:16:22.682 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:16:22.682 DEBUG:boto:Data: 2022-09-12T20:16:22.683 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.683 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.684 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.684 DEBUG:boto:Params: {} 2022-09-12T20:16:22.685 DEBUG:boto:Token: None 2022-09-12T20:16:22.685 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/ 2022-09-12T20:16:22.686 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM= 2022-09-12T20:16:22.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM=', 'Content-Length': '0'} 2022-09-12T20:16:22.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f9864fd7dbb3b6c-00631f9396-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.690 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:22.691 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:22.692 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:22.693 DEBUG:boto:path=/jdpiak-75/?prefix=obj69 2022-09-12T20:16:22.693 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj69 2022-09-12T20:16:22.694 DEBUG:boto:Method: GET 2022-09-12T20:16:22.694 DEBUG:boto:Path: /jdpiak-75/?prefix=obj69 2022-09-12T20:16:22.695 DEBUG:boto:Data: 2022-09-12T20:16:22.695 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.696 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.696 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.697 DEBUG:boto:Params: {} 2022-09-12T20:16:22.698 DEBUG:boto:Token: None 2022-09-12T20:16:22.698 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/ 2022-09-12T20:16:22.699 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM= 2022-09-12T20:16:22.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM=', 'Content-Length': '0'} 2022-09-12T20:16:22.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fe0eafe4f459202-00631f9396-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.702 DEBUG:boto:b'jdpiak-75obj691000falseobj692022-09-12T20:08:38.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:22.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2022-09-12T20:16:22.704 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:22.704 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:22.705 DEBUG:boto:path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:22.705 DEBUG:boto:auth_path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:22.706 DEBUG:boto:Method: GET 2022-09-12T20:16:22.707 DEBUG:boto:Path: /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:22.707 DEBUG:boto:Data: 2022-09-12T20:16:22.708 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.708 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.709 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.709 DEBUG:boto:Params: {} 2022-09-12T20:16:22.710 DEBUG:boto:Token: None 2022-09-12T20:16:22.710 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:22.711 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FsxYZCf7/uAzn4l55MVcoHsTsYo= 2022-09-12T20:16:22.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FsxYZCf7/uAzn4l55MVcoHsTsYo=', 'Content-Length': '0'} 2022-09-12T20:16:22.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc6776efa7c8a3f1-00631f9396-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.715 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:22.715 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:22.716 DEBUG:boto:path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:22.716 DEBUG:boto:auth_path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:22.717 DEBUG:boto:Method: GET 2022-09-12T20:16:22.717 DEBUG:boto:Path: /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:22.718 DEBUG:boto:Data: 2022-09-12T20:16:22.718 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.719 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.720 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.720 DEBUG:boto:Params: {} 2022-09-12T20:16:22.721 DEBUG:boto:Token: None 2022-09-12T20:16:22.721 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:22.722 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FsxYZCf7/uAzn4l55MVcoHsTsYo= 2022-09-12T20:16:22.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FsxYZCf7/uAzn4l55MVcoHsTsYo=', 'Content-Length': '0'} 2022-09-12T20:16:22.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f82f774007c3517-00631f9396-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.726 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:22.726 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:22.727 DEBUG:boto:path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:22.727 DEBUG:boto:auth_path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:22.728 DEBUG:boto:Method: HEAD 2022-09-12T20:16:22.728 DEBUG:boto:Path: /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:22.729 DEBUG:boto:Data: 2022-09-12T20:16:22.729 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.730 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.730 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.731 DEBUG:boto:Params: {} 2022-09-12T20:16:22.731 DEBUG:boto:Token: None 2022-09-12T20:16:22.732 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:22.733 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6ZIW87RFCnWXlHlweuuvVRCPGjA= 2022-09-12T20:16:22.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6ZIW87RFCnWXlHlweuuvVRCPGjA=', 'Content-Length': '0'} 2022-09-12T20:16:22.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000289106cb7066c63b-00631f9396-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.736 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:22.737 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:22.737 DEBUG:boto:path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:22.738 DEBUG:boto:auth_path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:22.738 DEBUG:boto:Method: HEAD 2022-09-12T20:16:22.739 DEBUG:boto:Path: /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:22.739 DEBUG:boto:Data: 2022-09-12T20:16:22.740 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.740 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.741 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.741 DEBUG:boto:Params: {} 2022-09-12T20:16:22.742 DEBUG:boto:Token: None 2022-09-12T20:16:22.742 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:22.743 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:6ZIW87RFCnWXlHlweuuvVRCPGjA= 2022-09-12T20:16:22.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:6ZIW87RFCnWXlHlweuuvVRCPGjA=', 'Content-Length': '0'} 2022-09-12T20:16:22.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d35dac20882bcc30-00631f9396-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2022-09-12T20:16:22.747 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:22.747 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:22.748 DEBUG:boto:path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:22.748 DEBUG:boto:auth_path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:22.749 DEBUG:boto:Method: GET 2022-09-12T20:16:22.750 DEBUG:boto:Path: /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:22.750 DEBUG:boto:Data: 2022-09-12T20:16:22.751 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.751 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.752 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.752 DEBUG:boto:Params: {} 2022-09-12T20:16:22.753 DEBUG:boto:Token: None 2022-09-12T20:16:22.754 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:22.754 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FsxYZCf7/uAzn4l55MVcoHsTsYo= 2022-09-12T20:16:22.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FsxYZCf7/uAzn4l55MVcoHsTsYo=', 'Content-Length': '0'} 2022-09-12T20:16:22.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000879664f9a727333d-00631f9396-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.758 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:22.759 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:22.759 DEBUG:boto:path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:22.760 DEBUG:boto:auth_path=/jdpiak-75/obj74?versionId=null 2022-09-12T20:16:22.760 DEBUG:boto:Method: GET 2022-09-12T20:16:22.761 DEBUG:boto:Path: /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:22.761 DEBUG:boto:Data: 2022-09-12T20:16:22.762 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.762 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.763 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.764 DEBUG:boto:Params: {} 2022-09-12T20:16:22.764 DEBUG:boto:Token: None 2022-09-12T20:16:22.765 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj74?versionId=null 2022-09-12T20:16:22.766 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FsxYZCf7/uAzn4l55MVcoHsTsYo= 2022-09-12T20:16:22.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FsxYZCf7/uAzn4l55MVcoHsTsYo=', 'Content-Length': '0'} 2022-09-12T20:16:22.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4dba268eb9c182d-00631f9396-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.769 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:22.770 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:22.771 DEBUG:boto:path=/jdpiak-75/?prefix=obj74 2022-09-12T20:16:22.771 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj74 2022-09-12T20:16:22.772 DEBUG:boto:Method: GET 2022-09-12T20:16:22.772 DEBUG:boto:Path: /jdpiak-75/?prefix=obj74 2022-09-12T20:16:22.773 DEBUG:boto:Data: 2022-09-12T20:16:22.773 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.774 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.774 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.775 DEBUG:boto:Params: {} 2022-09-12T20:16:22.775 DEBUG:boto:Token: None 2022-09-12T20:16:22.776 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/ 2022-09-12T20:16:22.777 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM= 2022-09-12T20:16:22.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM=', 'Content-Length': '0'} 2022-09-12T20:16:22.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8ba99ecc2b73e1f-00631f9396-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.781 DEBUG:boto:b'jdpiak-75obj741000falseobj742022-09-12T20:14:27.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:22.782 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:22.782 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:22.783 DEBUG:boto:path=/jdpiak-75/?prefix=obj74 2022-09-12T20:16:22.783 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj74 2022-09-12T20:16:22.784 DEBUG:boto:Method: GET 2022-09-12T20:16:22.784 DEBUG:boto:Path: /jdpiak-75/?prefix=obj74 2022-09-12T20:16:22.785 DEBUG:boto:Data: 2022-09-12T20:16:22.785 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.786 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.786 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.787 DEBUG:boto:Params: {} 2022-09-12T20:16:22.787 DEBUG:boto:Token: None 2022-09-12T20:16:22.788 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/ 2022-09-12T20:16:22.789 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM= 2022-09-12T20:16:22.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM=', 'Content-Length': '0'} 2022-09-12T20:16:22.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007aa954ff1b73166b-00631f9396-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.793 DEBUG:boto:b'jdpiak-75obj741000falseobj742022-09-12T20:14:27.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:22.794 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:22.794 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:22.795 DEBUG:boto:Method: HEAD 2022-09-12T20:16:22.795 DEBUG:boto:Path: /jdpiak-75/obj74 2022-09-12T20:16:22.796 DEBUG:boto:Data: 2022-09-12T20:16:22.796 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.797 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.797 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.798 DEBUG:boto:Params: {} 2022-09-12T20:16:22.798 DEBUG:boto:Token: None 2022-09-12T20:16:22.799 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj74 2022-09-12T20:16:22.799 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3DJALhpYzILOhiwM3uGb182MUb4= 2022-09-12T20:16:22.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3DJALhpYzILOhiwM3uGb182MUb4=', 'Content-Length': '0'} 2022-09-12T20:16:22.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022ab6e203ee8dc1b-00631f9396-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.803 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:22.803 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:22.804 DEBUG:boto:Method: HEAD 2022-09-12T20:16:22.804 DEBUG:boto:Path: /jdpiak-75/obj74 2022-09-12T20:16:22.805 DEBUG:boto:Data: 2022-09-12T20:16:22.805 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.806 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.807 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.807 DEBUG:boto:Params: {} 2022-09-12T20:16:22.808 DEBUG:boto:Token: None 2022-09-12T20:16:22.808 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj74 2022-09-12T20:16:22.809 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3DJALhpYzILOhiwM3uGb182MUb4= 2022-09-12T20:16:22.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3DJALhpYzILOhiwM3uGb182MUb4=', 'Content-Length': '0'} 2022-09-12T20:16:22.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099666d08c169887e-00631f9396-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2022-09-12T20:16:22.813 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:22.813 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:22.814 DEBUG:boto:Method: GET 2022-09-12T20:16:22.824 DEBUG:boto:Path: /jdpiak-75/obj74 2022-09-12T20:16:22.825 DEBUG:boto:Data: 2022-09-12T20:16:22.825 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.826 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.826 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.827 DEBUG:boto:Params: {} 2022-09-12T20:16:22.828 DEBUG:boto:Token: None 2022-09-12T20:16:22.829 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj74 2022-09-12T20:16:22.829 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vZVjNYYpy1J3cOH7dO3ty5u+ZfE= 2022-09-12T20:16:22.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vZVjNYYpy1J3cOH7dO3ty5u+ZfE=', 'Content-Length': '0'} 2022-09-12T20:16:22.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d400f5a8c5ffe7cd-00631f9396-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.878 DEBUG:boto:path=/jdpiak-75/obj74 2022-09-12T20:16:22.879 DEBUG:boto:auth_path=/jdpiak-75/obj74 2022-09-12T20:16:22.879 DEBUG:boto:Method: GET 2022-09-12T20:16:22.880 DEBUG:boto:Path: /jdpiak-75/obj74 2022-09-12T20:16:22.881 DEBUG:boto:Data: 2022-09-12T20:16:22.881 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.882 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.882 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.883 DEBUG:boto:Params: {} 2022-09-12T20:16:22.883 DEBUG:boto:Token: None 2022-09-12T20:16:22.884 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj74 2022-09-12T20:16:22.884 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vZVjNYYpy1J3cOH7dO3ty5u+ZfE= 2022-09-12T20:16:22.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vZVjNYYpy1J3cOH7dO3ty5u+ZfE=', 'Content-Length': '0'} 2022-09-12T20:16:22.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:14:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043b313a3b4d1a3cc-00631f9396-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.888 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:22.889 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:22.890 DEBUG:boto:path=/jdpiak-75/?prefix=obj74 2022-09-12T20:16:22.890 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj74 2022-09-12T20:16:22.891 DEBUG:boto:Method: GET 2022-09-12T20:16:22.891 DEBUG:boto:Path: /jdpiak-75/?prefix=obj74 2022-09-12T20:16:22.892 DEBUG:boto:Data: 2022-09-12T20:16:22.892 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.893 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.893 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.894 DEBUG:boto:Params: {} 2022-09-12T20:16:22.894 DEBUG:boto:Token: None 2022-09-12T20:16:22.895 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/ 2022-09-12T20:16:22.896 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM= 2022-09-12T20:16:22.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM=', 'Content-Length': '0'} 2022-09-12T20:16:22.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066d5f9353861da0b-00631f9396-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.900 DEBUG:boto:b'jdpiak-75obj741000falseobj742022-09-12T20:14:27.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:22.901 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:22.901 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:22.902 DEBUG:boto:path=/jdpiak-75/?prefix=obj74 2022-09-12T20:16:22.902 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj74 2022-09-12T20:16:22.903 DEBUG:boto:Method: GET 2022-09-12T20:16:22.903 DEBUG:boto:Path: /jdpiak-75/?prefix=obj74 2022-09-12T20:16:22.904 DEBUG:boto:Data: 2022-09-12T20:16:22.904 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.905 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.905 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.906 DEBUG:boto:Params: {} 2022-09-12T20:16:22.907 DEBUG:boto:Token: None 2022-09-12T20:16:22.907 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/ 2022-09-12T20:16:22.908 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM= 2022-09-12T20:16:22.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cfASVkR+D6pGij0UNdh+Y3DGPlM=', 'Content-Length': '0'} 2022-09-12T20:16:22.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000569fdb24d75b2a7-00631f9396-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.912 DEBUG:boto:b'jdpiak-75obj741000falseobj742022-09-12T20:14:27.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:22.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:16:22.914 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:22.915 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:22.916 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:22.916 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:22.917 DEBUG:boto:Method: GET 2022-09-12T20:16:22.917 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:22.918 DEBUG:boto:Data: 2022-09-12T20:16:22.919 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.919 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.920 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.920 DEBUG:boto:Params: {} 2022-09-12T20:16:22.921 DEBUG:boto:Token: None 2022-09-12T20:16:22.921 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:22.922 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BzjQrNwwyrANN5NShvmdu/u7ZkI= 2022-09-12T20:16:22.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BzjQrNwwyrANN5NShvmdu/u7ZkI=', 'Content-Length': '0'} 2022-09-12T20:16:22.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002353ab7506ed165a-00631f9396-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.926 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:22.926 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:22.927 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:22.927 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:22.928 DEBUG:boto:Method: GET 2022-09-12T20:16:22.928 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:22.929 DEBUG:boto:Data: 2022-09-12T20:16:22.930 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.930 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.931 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.931 DEBUG:boto:Params: {} 2022-09-12T20:16:22.932 DEBUG:boto:Token: None 2022-09-12T20:16:22.932 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:22.933 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BzjQrNwwyrANN5NShvmdu/u7ZkI= 2022-09-12T20:16:22.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BzjQrNwwyrANN5NShvmdu/u7ZkI=', 'Content-Length': '0'} 2022-09-12T20:16:22.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000299d219bef9f3913-00631f9396-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.978 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:22.979 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:22.979 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:22.980 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:22.980 DEBUG:boto:Method: HEAD 2022-09-12T20:16:22.981 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:22.981 DEBUG:boto:Data: 2022-09-12T20:16:22.982 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.982 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:22.983 DEBUG:boto:Port: 8001 2022-09-12T20:16:22.983 DEBUG:boto:Params: {} 2022-09-12T20:16:22.984 DEBUG:boto:Token: None 2022-09-12T20:16:22.985 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:22.985 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RoTZJ9B0wiBVzEFkEjnzllu/U8M= 2022-09-12T20:16:22.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RoTZJ9B0wiBVzEFkEjnzllu/U8M=', 'Content-Length': '0'} 2022-09-12T20:16:22.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0895debd2c7ac7d-00631f9396-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:22.989 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:22.989 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:22.990 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:22.990 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:22.991 DEBUG:boto:Method: HEAD 2022-09-12T20:16:22.992 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:22.993 DEBUG:boto:Data: 2022-09-12T20:16:22.993 DEBUG:boto:Headers: {} 2022-09-12T20:16:22.994 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:22.994 DEBUG:boto:Port: 8000 2022-09-12T20:16:22.995 DEBUG:boto:Params: {} 2022-09-12T20:16:22.995 DEBUG:boto:Token: None 2022-09-12T20:16:22.996 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:22 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:22.997 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:RoTZJ9B0wiBVzEFkEjnzllu/U8M= 2022-09-12T20:16:22.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:22 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:RoTZJ9B0wiBVzEFkEjnzllu/U8M=', 'Content-Length': '0'} 2022-09-12T20:16:22.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a207554696c46d44-00631f9396-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:16:23.001 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:23.001 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:23.002 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:23.002 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:23.003 DEBUG:boto:Method: GET 2022-09-12T20:16:23.003 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:23.004 DEBUG:boto:Data: 2022-09-12T20:16:23.004 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.005 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.006 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.007 DEBUG:boto:Params: {} 2022-09-12T20:16:23.007 DEBUG:boto:Token: None 2022-09-12T20:16:23.008 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:23.009 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:woefSN1M0HvgvBL38NjCbayPrHo= 2022-09-12T20:16:23.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:woefSN1M0HvgvBL38NjCbayPrHo=', 'Content-Length': '0'} 2022-09-12T20:16:23.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e42a55969e0c7d2d-00631f9397-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.012 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:23.013 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:23.013 DEBUG:boto:path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:23.014 DEBUG:boto:auth_path=/jdpiak-75/obj76?versionId=null 2022-09-12T20:16:23.015 DEBUG:boto:Method: GET 2022-09-12T20:16:23.015 DEBUG:boto:Path: /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:23.016 DEBUG:boto:Data: 2022-09-12T20:16:23.017 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.017 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.018 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.019 DEBUG:boto:Params: {} 2022-09-12T20:16:23.019 DEBUG:boto:Token: None 2022-09-12T20:16:23.020 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj76?versionId=null 2022-09-12T20:16:23.020 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:woefSN1M0HvgvBL38NjCbayPrHo= 2022-09-12T20:16:23.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:woefSN1M0HvgvBL38NjCbayPrHo=', 'Content-Length': '0'} 2022-09-12T20:16:23.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017d8472190eb0e4a-00631f9397-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.024 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:23.025 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:23.025 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:16:23.026 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:16:23.026 DEBUG:boto:Method: GET 2022-09-12T20:16:23.027 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:16:23.028 DEBUG:boto:Data: 2022-09-12T20:16:23.028 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.029 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.030 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.030 DEBUG:boto:Params: {} 2022-09-12T20:16:23.031 DEBUG:boto:Token: None 2022-09-12T20:16:23.031 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/ 2022-09-12T20:16:23.032 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM= 2022-09-12T20:16:23.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM=', 'Content-Length': '0'} 2022-09-12T20:16:23.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a8ea3294abada64-00631f9397-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.036 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:23.037 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:23.038 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:23.038 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:16:23.039 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:16:23.040 DEBUG:boto:Method: GET 2022-09-12T20:16:23.040 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:16:23.041 DEBUG:boto:Data: 2022-09-12T20:16:23.041 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.042 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.043 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.043 DEBUG:boto:Params: {} 2022-09-12T20:16:23.044 DEBUG:boto:Token: None 2022-09-12T20:16:23.044 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/ 2022-09-12T20:16:23.045 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM= 2022-09-12T20:16:23.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM=', 'Content-Length': '0'} 2022-09-12T20:16:23.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048919894fb2bab1a-00631f9397-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.049 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:23.050 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:23.050 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:23.051 DEBUG:boto:Method: HEAD 2022-09-12T20:16:23.051 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:16:23.052 DEBUG:boto:Data: 2022-09-12T20:16:23.052 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.053 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.053 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.054 DEBUG:boto:Params: {} 2022-09-12T20:16:23.055 DEBUG:boto:Token: None 2022-09-12T20:16:23.055 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj76 2022-09-12T20:16:23.056 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yuAxmw4E3uy6rDkE4nPfOEUHQeo= 2022-09-12T20:16:23.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yuAxmw4E3uy6rDkE4nPfOEUHQeo=', 'Content-Length': '0'} 2022-09-12T20:16:23.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cf5cc0b267843c6-00631f9397-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.059 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:23.060 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:23.060 DEBUG:boto:Method: HEAD 2022-09-12T20:16:23.061 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:16:23.061 DEBUG:boto:Data: 2022-09-12T20:16:23.062 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.062 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.063 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.063 DEBUG:boto:Params: {} 2022-09-12T20:16:23.064 DEBUG:boto:Token: None 2022-09-12T20:16:23.064 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj76 2022-09-12T20:16:23.065 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yuAxmw4E3uy6rDkE4nPfOEUHQeo= 2022-09-12T20:16:23.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yuAxmw4E3uy6rDkE4nPfOEUHQeo=', 'Content-Length': '0'} 2022-09-12T20:16:23.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca14336c402a80fc-00631f9397-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-12T20:16:23.069 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:23.069 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:23.070 DEBUG:boto:Method: GET 2022-09-12T20:16:23.071 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:16:23.071 DEBUG:boto:Data: 2022-09-12T20:16:23.072 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.072 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.073 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.073 DEBUG:boto:Params: {} 2022-09-12T20:16:23.074 DEBUG:boto:Token: None 2022-09-12T20:16:23.075 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj76 2022-09-12T20:16:23.075 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ROLRU69du582f9LpsqSEpvOJZ0o= 2022-09-12T20:16:23.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ROLRU69du582f9LpsqSEpvOJZ0o=', 'Content-Length': '0'} 2022-09-12T20:16:23.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab33bb8cf5f01a8e-00631f9397-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.122 DEBUG:boto:path=/jdpiak-75/obj76 2022-09-12T20:16:23.123 DEBUG:boto:auth_path=/jdpiak-75/obj76 2022-09-12T20:16:23.124 DEBUG:boto:Method: GET 2022-09-12T20:16:23.125 DEBUG:boto:Path: /jdpiak-75/obj76 2022-09-12T20:16:23.125 DEBUG:boto:Data: 2022-09-12T20:16:23.126 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.126 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.127 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.127 DEBUG:boto:Params: {} 2022-09-12T20:16:23.128 DEBUG:boto:Token: None 2022-09-12T20:16:23.129 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj76 2022-09-12T20:16:23.129 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ROLRU69du582f9LpsqSEpvOJZ0o= 2022-09-12T20:16:23.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ROLRU69du582f9LpsqSEpvOJZ0o=', 'Content-Length': '0'} 2022-09-12T20:16:23.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b958f7601a506d0e-00631f9397-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.133 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:23.134 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:23.134 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:16:23.135 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:16:23.136 DEBUG:boto:Method: GET 2022-09-12T20:16:23.136 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:16:23.137 DEBUG:boto:Data: 2022-09-12T20:16:23.137 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.138 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.138 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.139 DEBUG:boto:Params: {} 2022-09-12T20:16:23.140 DEBUG:boto:Token: None 2022-09-12T20:16:23.140 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/ 2022-09-12T20:16:23.141 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM= 2022-09-12T20:16:23.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM=', 'Content-Length': '0'} 2022-09-12T20:16:23.144 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004371bf62b3d205bf-00631f9397-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.145 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:23.146 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:23.147 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:23.147 DEBUG:boto:path=/jdpiak-75/?prefix=obj76 2022-09-12T20:16:23.148 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj76 2022-09-12T20:16:23.148 DEBUG:boto:Method: GET 2022-09-12T20:16:23.149 DEBUG:boto:Path: /jdpiak-75/?prefix=obj76 2022-09-12T20:16:23.149 DEBUG:boto:Data: 2022-09-12T20:16:23.150 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.151 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.151 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.152 DEBUG:boto:Params: {} 2022-09-12T20:16:23.152 DEBUG:boto:Token: None 2022-09-12T20:16:23.153 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/ 2022-09-12T20:16:23.153 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM= 2022-09-12T20:16:23.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM=', 'Content-Length': '0'} 2022-09-12T20:16:23.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007479d0ce01118543-00631f9397-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.157 DEBUG:boto:b'jdpiak-75obj761000falseobj762022-09-12T20:06:16.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:23.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:16:23.159 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:23.159 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:23.160 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:23.160 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:23.161 DEBUG:boto:Method: GET 2022-09-12T20:16:23.161 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:23.162 DEBUG:boto:Data: 2022-09-12T20:16:23.163 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.163 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.164 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.164 DEBUG:boto:Params: {} 2022-09-12T20:16:23.165 DEBUG:boto:Token: None 2022-09-12T20:16:23.165 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:23.166 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ldhgnHPmiwdkYKGA6KH82ZSRuC4= 2022-09-12T20:16:23.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ldhgnHPmiwdkYKGA6KH82ZSRuC4=', 'Content-Length': '0'} 2022-09-12T20:16:23.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2e8276e834bb624-00631f9397-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.170 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:23.170 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:23.171 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:23.172 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:23.172 DEBUG:boto:Method: GET 2022-09-12T20:16:23.173 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:23.173 DEBUG:boto:Data: 2022-09-12T20:16:23.174 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.174 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.175 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.175 DEBUG:boto:Params: {} 2022-09-12T20:16:23.176 DEBUG:boto:Token: None 2022-09-12T20:16:23.177 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:23.177 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ldhgnHPmiwdkYKGA6KH82ZSRuC4= 2022-09-12T20:16:23.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ldhgnHPmiwdkYKGA6KH82ZSRuC4=', 'Content-Length': '0'} 2022-09-12T20:16:23.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d946fca8a1d2f336-00631f9397-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.181 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:23.182 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:23.182 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:23.183 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:23.183 DEBUG:boto:Method: HEAD 2022-09-12T20:16:23.184 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:23.184 DEBUG:boto:Data: 2022-09-12T20:16:23.185 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.185 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.186 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.186 DEBUG:boto:Params: {} 2022-09-12T20:16:23.187 DEBUG:boto:Token: None 2022-09-12T20:16:23.188 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:23.188 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mTgY+khoMxKgn8QmZwCIbof+vmI= 2022-09-12T20:16:23.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mTgY+khoMxKgn8QmZwCIbof+vmI=', 'Content-Length': '0'} 2022-09-12T20:16:23.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee9bd191b54bcf71-00631f9397-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.192 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:23.192 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:23.193 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:23.194 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:23.194 DEBUG:boto:Method: HEAD 2022-09-12T20:16:23.195 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:23.195 DEBUG:boto:Data: 2022-09-12T20:16:23.196 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.196 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.197 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.198 DEBUG:boto:Params: {} 2022-09-12T20:16:23.198 DEBUG:boto:Token: None 2022-09-12T20:16:23.199 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:23.199 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mTgY+khoMxKgn8QmZwCIbof+vmI= 2022-09-12T20:16:23.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mTgY+khoMxKgn8QmZwCIbof+vmI=', 'Content-Length': '0'} 2022-09-12T20:16:23.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007371d85da0a2222f-00631f9397-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:16:23.203 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:23.204 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:23.205 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:23.205 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:23.206 DEBUG:boto:Method: GET 2022-09-12T20:16:23.206 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:23.207 DEBUG:boto:Data: 2022-09-12T20:16:23.207 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.208 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.209 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.209 DEBUG:boto:Params: {} 2022-09-12T20:16:23.210 DEBUG:boto:Token: None 2022-09-12T20:16:23.211 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:23.212 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ldhgnHPmiwdkYKGA6KH82ZSRuC4= 2022-09-12T20:16:23.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ldhgnHPmiwdkYKGA6KH82ZSRuC4=', 'Content-Length': '0'} 2022-09-12T20:16:23.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a879a2d0f17f4de-00631f9397-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.216 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:23.216 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:23.217 DEBUG:boto:path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:23.218 DEBUG:boto:auth_path=/jdpiak-75/obj85?versionId=null 2022-09-12T20:16:23.218 DEBUG:boto:Method: GET 2022-09-12T20:16:23.219 DEBUG:boto:Path: /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:23.219 DEBUG:boto:Data: 2022-09-12T20:16:23.220 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.220 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.221 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.221 DEBUG:boto:Params: {} 2022-09-12T20:16:23.222 DEBUG:boto:Token: None 2022-09-12T20:16:23.223 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj85?versionId=null 2022-09-12T20:16:23.224 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ldhgnHPmiwdkYKGA6KH82ZSRuC4= 2022-09-12T20:16:23.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ldhgnHPmiwdkYKGA6KH82ZSRuC4=', 'Content-Length': '0'} 2022-09-12T20:16:23.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca5206fb754c1737-00631f9397-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.228 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:23.228 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:23.229 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:16:23.229 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:16:23.230 DEBUG:boto:Method: GET 2022-09-12T20:16:23.231 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:16:23.231 DEBUG:boto:Data: 2022-09-12T20:16:23.232 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.232 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.233 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.233 DEBUG:boto:Params: {} 2022-09-12T20:16:23.234 DEBUG:boto:Token: None 2022-09-12T20:16:23.234 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/ 2022-09-12T20:16:23.235 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM= 2022-09-12T20:16:23.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM=', 'Content-Length': '0'} 2022-09-12T20:16:23.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000453c186543dc09ce-00631f9397-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.282 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:23.283 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:23.284 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:23.284 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:16:23.285 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:16:23.285 DEBUG:boto:Method: GET 2022-09-12T20:16:23.286 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:16:23.287 DEBUG:boto:Data: 2022-09-12T20:16:23.287 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.288 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.288 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.289 DEBUG:boto:Params: {} 2022-09-12T20:16:23.290 DEBUG:boto:Token: None 2022-09-12T20:16:23.290 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/ 2022-09-12T20:16:23.291 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM= 2022-09-12T20:16:23.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM=', 'Content-Length': '0'} 2022-09-12T20:16:23.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d51435856f3e3424-00631f9397-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.296 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:23.297 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:23.297 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:23.298 DEBUG:boto:Method: HEAD 2022-09-12T20:16:23.298 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:16:23.299 DEBUG:boto:Data: 2022-09-12T20:16:23.299 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.300 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.300 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.301 DEBUG:boto:Params: {} 2022-09-12T20:16:23.301 DEBUG:boto:Token: None 2022-09-12T20:16:23.302 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj85 2022-09-12T20:16:23.304 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V6V4XgbvZiNK+dQ3ZJxoDVIn3Io= 2022-09-12T20:16:23.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V6V4XgbvZiNK+dQ3ZJxoDVIn3Io=', 'Content-Length': '0'} 2022-09-12T20:16:23.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9722c0de78fa7a3-00631f9397-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.309 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:23.309 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:23.310 DEBUG:boto:Method: HEAD 2022-09-12T20:16:23.310 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:16:23.312 DEBUG:boto:Data: 2022-09-12T20:16:23.312 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.313 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.314 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.314 DEBUG:boto:Params: {} 2022-09-12T20:16:23.315 DEBUG:boto:Token: None 2022-09-12T20:16:23.316 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj85 2022-09-12T20:16:23.316 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:V6V4XgbvZiNK+dQ3ZJxoDVIn3Io= 2022-09-12T20:16:23.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:V6V4XgbvZiNK+dQ3ZJxoDVIn3Io=', 'Content-Length': '0'} 2022-09-12T20:16:23.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8bf834f679174c4-00631f9397-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-12T20:16:23.321 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:23.322 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:23.322 DEBUG:boto:Method: GET 2022-09-12T20:16:23.323 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:16:23.324 DEBUG:boto:Data: 2022-09-12T20:16:23.324 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.325 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.325 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.326 DEBUG:boto:Params: {} 2022-09-12T20:16:23.326 DEBUG:boto:Token: None 2022-09-12T20:16:23.327 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj85 2022-09-12T20:16:23.328 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KW2N1dxHEIeFEsCHdyn0Su0Dpzo= 2022-09-12T20:16:23.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KW2N1dxHEIeFEsCHdyn0Su0Dpzo=', 'Content-Length': '0'} 2022-09-12T20:16:23.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cdbb395a468bf25-00631f9397-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.331 DEBUG:boto:path=/jdpiak-75/obj85 2022-09-12T20:16:23.332 DEBUG:boto:auth_path=/jdpiak-75/obj85 2022-09-12T20:16:23.332 DEBUG:boto:Method: GET 2022-09-12T20:16:23.333 DEBUG:boto:Path: /jdpiak-75/obj85 2022-09-12T20:16:23.333 DEBUG:boto:Data: 2022-09-12T20:16:23.334 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.334 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.335 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.336 DEBUG:boto:Params: {} 2022-09-12T20:16:23.336 DEBUG:boto:Token: None 2022-09-12T20:16:23.337 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj85 2022-09-12T20:16:23.337 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:KW2N1dxHEIeFEsCHdyn0Su0Dpzo= 2022-09-12T20:16:23.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:KW2N1dxHEIeFEsCHdyn0Su0Dpzo=', 'Content-Length': '0'} 2022-09-12T20:16:23.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:03:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edf3f02c1c6db6ee-00631f9397-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.382 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:23.383 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:23.383 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:16:23.384 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:16:23.384 DEBUG:boto:Method: GET 2022-09-12T20:16:23.385 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:16:23.386 DEBUG:boto:Data: 2022-09-12T20:16:23.386 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.387 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.387 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.388 DEBUG:boto:Params: {} 2022-09-12T20:16:23.388 DEBUG:boto:Token: None 2022-09-12T20:16:23.389 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/ 2022-09-12T20:16:23.390 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM= 2022-09-12T20:16:23.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM=', 'Content-Length': '0'} 2022-09-12T20:16:23.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030d54348bf14837d-00631f9397-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.394 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:23.395 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:23.395 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:23.396 DEBUG:boto:path=/jdpiak-75/?prefix=obj85 2022-09-12T20:16:23.397 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj85 2022-09-12T20:16:23.397 DEBUG:boto:Method: GET 2022-09-12T20:16:23.398 DEBUG:boto:Path: /jdpiak-75/?prefix=obj85 2022-09-12T20:16:23.398 DEBUG:boto:Data: 2022-09-12T20:16:23.399 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.399 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.400 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.400 DEBUG:boto:Params: {} 2022-09-12T20:16:23.401 DEBUG:boto:Token: None 2022-09-12T20:16:23.401 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/ 2022-09-12T20:16:23.402 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM= 2022-09-12T20:16:23.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM=', 'Content-Length': '0'} 2022-09-12T20:16:23.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c16c7427e8cd20ce-00631f9397-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.406 DEBUG:boto:b'jdpiak-75obj851000falseobj852022-09-12T20:03:28.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:23.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:16:23.408 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:23.408 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:23.409 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:23.409 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:23.410 DEBUG:boto:Method: GET 2022-09-12T20:16:23.411 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:23.411 DEBUG:boto:Data: 2022-09-12T20:16:23.412 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.412 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.413 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.413 DEBUG:boto:Params: {} 2022-09-12T20:16:23.414 DEBUG:boto:Token: None 2022-09-12T20:16:23.415 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:23.415 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/otmx9INZ2c2DKJx8Zk5ipXd3c4= 2022-09-12T20:16:23.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/otmx9INZ2c2DKJx8Zk5ipXd3c4=', 'Content-Length': '0'} 2022-09-12T20:16:23.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004acbd282eba5362f-00631f9397-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.420 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:23.421 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:23.422 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:23.423 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:23.423 DEBUG:boto:Method: GET 2022-09-12T20:16:23.424 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:23.425 DEBUG:boto:Data: 2022-09-12T20:16:23.426 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.426 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.427 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.428 DEBUG:boto:Params: {} 2022-09-12T20:16:23.428 DEBUG:boto:Token: None 2022-09-12T20:16:23.429 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:23.429 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/otmx9INZ2c2DKJx8Zk5ipXd3c4= 2022-09-12T20:16:23.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/otmx9INZ2c2DKJx8Zk5ipXd3c4=', 'Content-Length': '0'} 2022-09-12T20:16:23.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000313128a26003eb1-00631f9397-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.433 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:23.434 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:23.435 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:23.435 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:23.436 DEBUG:boto:Method: HEAD 2022-09-12T20:16:23.436 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:23.437 DEBUG:boto:Data: 2022-09-12T20:16:23.437 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.438 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.438 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.439 DEBUG:boto:Params: {} 2022-09-12T20:16:23.439 DEBUG:boto:Token: None 2022-09-12T20:16:23.440 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:23.441 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+167ARXKdUW4EIT+IJjNmFggYsY= 2022-09-12T20:16:23.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+167ARXKdUW4EIT+IJjNmFggYsY=', 'Content-Length': '0'} 2022-09-12T20:16:23.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059b82844abd85592-00631f9397-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.444 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:23.445 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:23.445 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:23.446 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:23.446 DEBUG:boto:Method: HEAD 2022-09-12T20:16:23.447 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:23.447 DEBUG:boto:Data: 2022-09-12T20:16:23.448 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.448 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.449 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.449 DEBUG:boto:Params: {} 2022-09-12T20:16:23.450 DEBUG:boto:Token: None 2022-09-12T20:16:23.451 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:23.451 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+167ARXKdUW4EIT+IJjNmFggYsY= 2022-09-12T20:16:23.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+167ARXKdUW4EIT+IJjNmFggYsY=', 'Content-Length': '0'} 2022-09-12T20:16:23.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d504323638ebd23e-00631f9397-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:16:23.456 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:23.456 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:23.457 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:23.457 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:23.458 DEBUG:boto:Method: GET 2022-09-12T20:16:23.459 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:23.459 DEBUG:boto:Data: 2022-09-12T20:16:23.460 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.460 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.461 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.461 DEBUG:boto:Params: {} 2022-09-12T20:16:23.462 DEBUG:boto:Token: None 2022-09-12T20:16:23.463 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:23.463 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/otmx9INZ2c2DKJx8Zk5ipXd3c4= 2022-09-12T20:16:23.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/otmx9INZ2c2DKJx8Zk5ipXd3c4=', 'Content-Length': '0'} 2022-09-12T20:16:23.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000556021f8df2d9b25-00631f9397-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.467 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:23.468 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:23.468 DEBUG:boto:path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:23.469 DEBUG:boto:auth_path=/jdpiak-75/obj87?versionId=null 2022-09-12T20:16:23.469 DEBUG:boto:Method: GET 2022-09-12T20:16:23.470 DEBUG:boto:Path: /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:23.470 DEBUG:boto:Data: 2022-09-12T20:16:23.471 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.471 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.472 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.472 DEBUG:boto:Params: {} 2022-09-12T20:16:23.473 DEBUG:boto:Token: None 2022-09-12T20:16:23.474 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj87?versionId=null 2022-09-12T20:16:23.474 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:/otmx9INZ2c2DKJx8Zk5ipXd3c4= 2022-09-12T20:16:23.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:/otmx9INZ2c2DKJx8Zk5ipXd3c4=', 'Content-Length': '0'} 2022-09-12T20:16:23.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6edcc71705de5b9-00631f9397-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.518 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:23.519 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:23.520 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:16:23.520 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:16:23.521 DEBUG:boto:Method: GET 2022-09-12T20:16:23.521 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:16:23.522 DEBUG:boto:Data: 2022-09-12T20:16:23.522 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.523 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.523 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.524 DEBUG:boto:Params: {} 2022-09-12T20:16:23.525 DEBUG:boto:Token: None 2022-09-12T20:16:23.525 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/ 2022-09-12T20:16:23.526 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM= 2022-09-12T20:16:23.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM=', 'Content-Length': '0'} 2022-09-12T20:16:23.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000016fcdb1a0938cee-00631f9397-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.530 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:23.531 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:23.532 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:23.532 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:16:23.533 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:16:23.533 DEBUG:boto:Method: GET 2022-09-12T20:16:23.534 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:16:23.534 DEBUG:boto:Data: 2022-09-12T20:16:23.535 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.535 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.536 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.536 DEBUG:boto:Params: {} 2022-09-12T20:16:23.537 DEBUG:boto:Token: None 2022-09-12T20:16:23.538 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/ 2022-09-12T20:16:23.538 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM= 2022-09-12T20:16:23.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM=', 'Content-Length': '0'} 2022-09-12T20:16:23.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f27e28b2a76a97f-00631f9397-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.543 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:23.544 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:23.544 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:23.545 DEBUG:boto:Method: HEAD 2022-09-12T20:16:23.545 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:16:23.546 DEBUG:boto:Data: 2022-09-12T20:16:23.547 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.547 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.548 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.549 DEBUG:boto:Params: {} 2022-09-12T20:16:23.549 DEBUG:boto:Token: None 2022-09-12T20:16:23.550 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj87 2022-09-12T20:16:23.551 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MH+fwiJKtNBZeftP4eb3+fHuRfI= 2022-09-12T20:16:23.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MH+fwiJKtNBZeftP4eb3+fHuRfI=', 'Content-Length': '0'} 2022-09-12T20:16:23.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e80d6f53f3e2467-00631f9397-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.554 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:23.555 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:23.555 DEBUG:boto:Method: HEAD 2022-09-12T20:16:23.556 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:16:23.557 DEBUG:boto:Data: 2022-09-12T20:16:23.558 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.558 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.559 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.559 DEBUG:boto:Params: {} 2022-09-12T20:16:23.560 DEBUG:boto:Token: None 2022-09-12T20:16:23.560 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj87 2022-09-12T20:16:23.561 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:MH+fwiJKtNBZeftP4eb3+fHuRfI= 2022-09-12T20:16:23.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:MH+fwiJKtNBZeftP4eb3+fHuRfI=', 'Content-Length': '0'} 2022-09-12T20:16:23.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068a27c0564705da9-00631f9397-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-12T20:16:23.565 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:23.566 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:23.566 DEBUG:boto:Method: GET 2022-09-12T20:16:23.567 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:16:23.567 DEBUG:boto:Data: 2022-09-12T20:16:23.568 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.569 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.569 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.570 DEBUG:boto:Params: {} 2022-09-12T20:16:23.570 DEBUG:boto:Token: None 2022-09-12T20:16:23.571 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj87 2022-09-12T20:16:23.572 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vhIC1K/KZhuPf4VQgPOwxbIRaiU= 2022-09-12T20:16:23.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vhIC1K/KZhuPf4VQgPOwxbIRaiU=', 'Content-Length': '0'} 2022-09-12T20:16:23.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d1d57164e0e0c4b-00631f9397-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.576 DEBUG:boto:path=/jdpiak-75/obj87 2022-09-12T20:16:23.576 DEBUG:boto:auth_path=/jdpiak-75/obj87 2022-09-12T20:16:23.577 DEBUG:boto:Method: GET 2022-09-12T20:16:23.577 DEBUG:boto:Path: /jdpiak-75/obj87 2022-09-12T20:16:23.578 DEBUG:boto:Data: 2022-09-12T20:16:23.578 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.579 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.580 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.580 DEBUG:boto:Params: {} 2022-09-12T20:16:23.581 DEBUG:boto:Token: None 2022-09-12T20:16:23.581 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj87 2022-09-12T20:16:23.582 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vhIC1K/KZhuPf4VQgPOwxbIRaiU= 2022-09-12T20:16:23.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vhIC1K/KZhuPf4VQgPOwxbIRaiU=', 'Content-Length': '0'} 2022-09-12T20:16:23.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:10:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002527790370566eeb-00631f9397-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.586 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:23.587 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:23.587 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:16:23.588 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:16:23.589 DEBUG:boto:Method: GET 2022-09-12T20:16:23.589 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:16:23.590 DEBUG:boto:Data: 2022-09-12T20:16:23.590 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.591 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.591 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.592 DEBUG:boto:Params: {} 2022-09-12T20:16:23.593 DEBUG:boto:Token: None 2022-09-12T20:16:23.594 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/ 2022-09-12T20:16:23.594 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM= 2022-09-12T20:16:23.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM=', 'Content-Length': '0'} 2022-09-12T20:16:23.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014a901a074aff0bc-00631f9397-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.638 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:23.639 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:23.640 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:23.640 DEBUG:boto:path=/jdpiak-75/?prefix=obj87 2022-09-12T20:16:23.641 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj87 2022-09-12T20:16:23.642 DEBUG:boto:Method: GET 2022-09-12T20:16:23.642 DEBUG:boto:Path: /jdpiak-75/?prefix=obj87 2022-09-12T20:16:23.643 DEBUG:boto:Data: 2022-09-12T20:16:23.643 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.644 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.644 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.645 DEBUG:boto:Params: {} 2022-09-12T20:16:23.646 DEBUG:boto:Token: None 2022-09-12T20:16:23.646 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/ 2022-09-12T20:16:23.647 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM= 2022-09-12T20:16:23.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM=', 'Content-Length': '0'} 2022-09-12T20:16:23.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055cefbb5b8a44339-00631f9397-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.651 DEBUG:boto:b'jdpiak-75obj871000falseobj872022-09-12T20:10:09.041Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:23.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:16:23.652 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:23.653 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:23.654 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:23.654 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:23.655 DEBUG:boto:Method: GET 2022-09-12T20:16:23.656 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:23.656 DEBUG:boto:Data: 2022-09-12T20:16:23.657 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.657 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.658 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.658 DEBUG:boto:Params: {} 2022-09-12T20:16:23.659 DEBUG:boto:Token: None 2022-09-12T20:16:23.660 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:23.660 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GOCeavh/W58HIi2AaJu9lizZjXc= 2022-09-12T20:16:23.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GOCeavh/W58HIi2AaJu9lizZjXc=', 'Content-Length': '0'} 2022-09-12T20:16:23.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b4b97821262b5d1-00631f9397-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.665 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:23.665 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:23.666 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:23.666 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:23.667 DEBUG:boto:Method: GET 2022-09-12T20:16:23.668 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:23.668 DEBUG:boto:Data: 2022-09-12T20:16:23.669 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.669 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.670 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.671 DEBUG:boto:Params: {} 2022-09-12T20:16:23.671 DEBUG:boto:Token: None 2022-09-12T20:16:23.672 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:23.672 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GOCeavh/W58HIi2AaJu9lizZjXc= 2022-09-12T20:16:23.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GOCeavh/W58HIi2AaJu9lizZjXc=', 'Content-Length': '0'} 2022-09-12T20:16:23.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c5532f529b4681e-00631f9397-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.718 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:23.719 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:23.719 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:23.720 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:23.720 DEBUG:boto:Method: HEAD 2022-09-12T20:16:23.721 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:23.721 DEBUG:boto:Data: 2022-09-12T20:16:23.722 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.723 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.723 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.724 DEBUG:boto:Params: {} 2022-09-12T20:16:23.724 DEBUG:boto:Token: None 2022-09-12T20:16:23.725 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:23.726 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1eMS/05/QvACVvL9/9Z7vJEcxds= 2022-09-12T20:16:23.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1eMS/05/QvACVvL9/9Z7vJEcxds=', 'Content-Length': '0'} 2022-09-12T20:16:23.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bb22b34ad9cfa61-00631f9397-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.729 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:23.730 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:23.730 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:23.731 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:23.731 DEBUG:boto:Method: HEAD 2022-09-12T20:16:23.732 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:23.732 DEBUG:boto:Data: 2022-09-12T20:16:23.733 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.733 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.734 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.734 DEBUG:boto:Params: {} 2022-09-12T20:16:23.735 DEBUG:boto:Token: None 2022-09-12T20:16:23.736 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:23.736 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1eMS/05/QvACVvL9/9Z7vJEcxds= 2022-09-12T20:16:23.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1eMS/05/QvACVvL9/9Z7vJEcxds=', 'Content-Length': '0'} 2022-09-12T20:16:23.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bacc2762f72dd869-00631f9397-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:16:23.740 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:23.740 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:23.741 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:23.741 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:23.742 DEBUG:boto:Method: GET 2022-09-12T20:16:23.743 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:23.743 DEBUG:boto:Data: 2022-09-12T20:16:23.744 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.744 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.745 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.745 DEBUG:boto:Params: {} 2022-09-12T20:16:23.746 DEBUG:boto:Token: None 2022-09-12T20:16:23.746 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:23.747 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GOCeavh/W58HIi2AaJu9lizZjXc= 2022-09-12T20:16:23.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GOCeavh/W58HIi2AaJu9lizZjXc=', 'Content-Length': '0'} 2022-09-12T20:16:23.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdb06136b6be60b9-00631f9397-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.751 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:23.751 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:23.752 DEBUG:boto:path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:23.752 DEBUG:boto:auth_path=/jdpiak-75/obj92?versionId=null 2022-09-12T20:16:23.753 DEBUG:boto:Method: GET 2022-09-12T20:16:23.753 DEBUG:boto:Path: /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:23.754 DEBUG:boto:Data: 2022-09-12T20:16:23.754 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.755 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.755 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.756 DEBUG:boto:Params: {} 2022-09-12T20:16:23.757 DEBUG:boto:Token: None 2022-09-12T20:16:23.757 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj92?versionId=null 2022-09-12T20:16:23.758 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GOCeavh/W58HIi2AaJu9lizZjXc= 2022-09-12T20:16:23.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GOCeavh/W58HIi2AaJu9lizZjXc=', 'Content-Length': '0'} 2022-09-12T20:16:23.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005be0e48123f1b6c6-00631f9397-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.761 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:23.762 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:23.762 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:16:23.763 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:16:23.763 DEBUG:boto:Method: GET 2022-09-12T20:16:23.764 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:16:23.765 DEBUG:boto:Data: 2022-09-12T20:16:23.765 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.766 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.766 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.767 DEBUG:boto:Params: {} 2022-09-12T20:16:23.767 DEBUG:boto:Token: None 2022-09-12T20:16:23.768 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/ 2022-09-12T20:16:23.768 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM= 2022-09-12T20:16:23.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM=', 'Content-Length': '0'} 2022-09-12T20:16:23.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051a86acbc9569d53-00631f9397-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.772 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:23.773 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:23.774 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:23.775 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:16:23.775 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:16:23.776 DEBUG:boto:Method: GET 2022-09-12T20:16:23.776 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:16:23.777 DEBUG:boto:Data: 2022-09-12T20:16:23.777 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.778 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.779 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.779 DEBUG:boto:Params: {} 2022-09-12T20:16:23.780 DEBUG:boto:Token: None 2022-09-12T20:16:23.780 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/ 2022-09-12T20:16:23.781 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM= 2022-09-12T20:16:23.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM=', 'Content-Length': '0'} 2022-09-12T20:16:23.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b52c20d773928bd1-00631f9397-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.785 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:23.786 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:23.787 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:23.787 DEBUG:boto:Method: HEAD 2022-09-12T20:16:23.788 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:16:23.788 DEBUG:boto:Data: 2022-09-12T20:16:23.789 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.789 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.790 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.790 DEBUG:boto:Params: {} 2022-09-12T20:16:23.791 DEBUG:boto:Token: None 2022-09-12T20:16:23.791 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj92 2022-09-12T20:16:23.792 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hlfOEZHkGcpQ9lw1oT1vRBm3ayc= 2022-09-12T20:16:23.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hlfOEZHkGcpQ9lw1oT1vRBm3ayc=', 'Content-Length': '0'} 2022-09-12T20:16:23.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096d6c77002265ca3-00631f9397-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.795 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:23.796 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:23.797 DEBUG:boto:Method: HEAD 2022-09-12T20:16:23.797 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:16:23.798 DEBUG:boto:Data: 2022-09-12T20:16:23.798 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.799 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.799 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.800 DEBUG:boto:Params: {} 2022-09-12T20:16:23.800 DEBUG:boto:Token: None 2022-09-12T20:16:23.801 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj92 2022-09-12T20:16:23.801 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:hlfOEZHkGcpQ9lw1oT1vRBm3ayc= 2022-09-12T20:16:23.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:hlfOEZHkGcpQ9lw1oT1vRBm3ayc=', 'Content-Length': '0'} 2022-09-12T20:16:23.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9c02ac787733fe3-00631f9397-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-12T20:16:23.806 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:23.807 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:23.807 DEBUG:boto:Method: GET 2022-09-12T20:16:23.808 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:16:23.808 DEBUG:boto:Data: 2022-09-12T20:16:23.809 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.809 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.810 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.811 DEBUG:boto:Params: {} 2022-09-12T20:16:23.811 DEBUG:boto:Token: None 2022-09-12T20:16:23.812 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj92 2022-09-12T20:16:23.812 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:++J2oMp9jBk1slKlI3AdhIVsONU= 2022-09-12T20:16:23.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:++J2oMp9jBk1slKlI3AdhIVsONU=', 'Content-Length': '0'} 2022-09-12T20:16:23.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076bb8f86ccab540e-00631f9397-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.858 DEBUG:boto:path=/jdpiak-75/obj92 2022-09-12T20:16:23.859 DEBUG:boto:auth_path=/jdpiak-75/obj92 2022-09-12T20:16:23.860 DEBUG:boto:Method: GET 2022-09-12T20:16:23.861 DEBUG:boto:Path: /jdpiak-75/obj92 2022-09-12T20:16:23.862 DEBUG:boto:Data: 2022-09-12T20:16:23.863 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.864 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.865 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.866 DEBUG:boto:Params: {} 2022-09-12T20:16:23.867 DEBUG:boto:Token: None 2022-09-12T20:16:23.869 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj92 2022-09-12T20:16:23.870 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:++J2oMp9jBk1slKlI3AdhIVsONU= 2022-09-12T20:16:23.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:++J2oMp9jBk1slKlI3AdhIVsONU=', 'Content-Length': '0'} 2022-09-12T20:16:23.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:08:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8be408ba208c519-00631f9397-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.875 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:23.876 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:23.877 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:16:23.878 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:16:23.879 DEBUG:boto:Method: GET 2022-09-12T20:16:23.880 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:16:23.881 DEBUG:boto:Data: 2022-09-12T20:16:23.882 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.883 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.884 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.885 DEBUG:boto:Params: {} 2022-09-12T20:16:23.886 DEBUG:boto:Token: None 2022-09-12T20:16:23.887 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/ 2022-09-12T20:16:23.888 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM= 2022-09-12T20:16:23.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM=', 'Content-Length': '0'} 2022-09-12T20:16:23.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000909335e39f235e19-00631f9397-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.893 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:23.894 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:23.895 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:23.896 DEBUG:boto:path=/jdpiak-75/?prefix=obj92 2022-09-12T20:16:23.896 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj92 2022-09-12T20:16:23.897 DEBUG:boto:Method: GET 2022-09-12T20:16:23.897 DEBUG:boto:Path: /jdpiak-75/?prefix=obj92 2022-09-12T20:16:23.898 DEBUG:boto:Data: 2022-09-12T20:16:23.898 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.899 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.899 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.900 DEBUG:boto:Params: {} 2022-09-12T20:16:23.900 DEBUG:boto:Token: None 2022-09-12T20:16:23.901 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/ 2022-09-12T20:16:23.902 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM= 2022-09-12T20:16:23.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM=', 'Content-Length': '0'} 2022-09-12T20:16:23.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000588d899d8656c174-00631f9397-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.906 DEBUG:boto:b'jdpiak-75obj921000falseobj922022-09-12T20:08:38.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:23.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-12T20:16:23.907 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:23.908 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:23.909 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:23.909 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:23.910 DEBUG:boto:Method: GET 2022-09-12T20:16:23.910 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:23.911 DEBUG:boto:Data: 2022-09-12T20:16:23.911 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.912 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.912 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.913 DEBUG:boto:Params: {} 2022-09-12T20:16:23.913 DEBUG:boto:Token: None 2022-09-12T20:16:23.914 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:23.915 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zdnGoU3fxDtLXWSzEKBOsOZEhUM= 2022-09-12T20:16:23.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zdnGoU3fxDtLXWSzEKBOsOZEhUM=', 'Content-Length': '0'} 2022-09-12T20:16:23.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6d037be04b9b357-00631f9397-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.918 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:23.919 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:23.920 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:23.920 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:23.921 DEBUG:boto:Method: GET 2022-09-12T20:16:23.922 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:23.922 DEBUG:boto:Data: 2022-09-12T20:16:23.923 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.923 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.924 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.924 DEBUG:boto:Params: {} 2022-09-12T20:16:23.925 DEBUG:boto:Token: None 2022-09-12T20:16:23.926 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:23.926 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zdnGoU3fxDtLXWSzEKBOsOZEhUM= 2022-09-12T20:16:23.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zdnGoU3fxDtLXWSzEKBOsOZEhUM=', 'Content-Length': '0'} 2022-09-12T20:16:23.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000087bd30c50ca6fd1-00631f9397-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.930 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:23.931 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:23.931 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:23.932 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:23.932 DEBUG:boto:Method: HEAD 2022-09-12T20:16:23.933 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:23.933 DEBUG:boto:Data: 2022-09-12T20:16:23.934 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.935 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.935 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.936 DEBUG:boto:Params: {} 2022-09-12T20:16:23.936 DEBUG:boto:Token: None 2022-09-12T20:16:23.937 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:23.938 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qpIJ+l+mNQmYbrzYXjx8jjAJouU= 2022-09-12T20:16:23.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qpIJ+l+mNQmYbrzYXjx8jjAJouU=', 'Content-Length': '0'} 2022-09-12T20:16:23.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006713faae050f3245-00631f9397-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.941 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:23.942 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:23.942 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:23.943 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:23.943 DEBUG:boto:Method: HEAD 2022-09-12T20:16:23.944 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:23.944 DEBUG:boto:Data: 2022-09-12T20:16:23.945 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.945 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.946 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.947 DEBUG:boto:Params: {} 2022-09-12T20:16:23.947 DEBUG:boto:Token: None 2022-09-12T20:16:23.948 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:23.949 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qpIJ+l+mNQmYbrzYXjx8jjAJouU= 2022-09-12T20:16:23.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qpIJ+l+mNQmYbrzYXjx8jjAJouU=', 'Content-Length': '0'} 2022-09-12T20:16:23.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023aafa7f4d9c0d97-00631f9397-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-12T20:16:23.953 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:23.953 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:23.954 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:23.955 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:23.955 DEBUG:boto:Method: GET 2022-09-12T20:16:23.956 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:23.956 DEBUG:boto:Data: 2022-09-12T20:16:23.957 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.957 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.958 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.958 DEBUG:boto:Params: {} 2022-09-12T20:16:23.959 DEBUG:boto:Token: None 2022-09-12T20:16:23.960 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:23.960 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zdnGoU3fxDtLXWSzEKBOsOZEhUM= 2022-09-12T20:16:23.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zdnGoU3fxDtLXWSzEKBOsOZEhUM=', 'Content-Length': '0'} 2022-09-12T20:16:23.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063bce66c7e85cf69-00631f9397-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.964 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:23.965 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:23.965 DEBUG:boto:path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:23.966 DEBUG:boto:auth_path=/jdpiak-75/obj93?versionId=null 2022-09-12T20:16:23.966 DEBUG:boto:Method: GET 2022-09-12T20:16:23.967 DEBUG:boto:Path: /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:23.967 DEBUG:boto:Data: 2022-09-12T20:16:23.968 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.968 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.969 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.969 DEBUG:boto:Params: {} 2022-09-12T20:16:23.970 DEBUG:boto:Token: None 2022-09-12T20:16:23.971 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/obj93?versionId=null 2022-09-12T20:16:23.971 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:zdnGoU3fxDtLXWSzEKBOsOZEhUM= 2022-09-12T20:16:23.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:zdnGoU3fxDtLXWSzEKBOsOZEhUM=', 'Content-Length': '0'} 2022-09-12T20:16:23.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9f2cf6080511f51-00631f9397-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.975 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:23.975 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:23.976 DEBUG:boto:path=/jdpiak-75/?prefix=obj93 2022-09-12T20:16:23.976 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj93 2022-09-12T20:16:23.977 DEBUG:boto:Method: GET 2022-09-12T20:16:23.978 DEBUG:boto:Path: /jdpiak-75/?prefix=obj93 2022-09-12T20:16:23.978 DEBUG:boto:Data: 2022-09-12T20:16:23.979 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.979 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:23.980 DEBUG:boto:Port: 8001 2022-09-12T20:16:23.980 DEBUG:boto:Params: {} 2022-09-12T20:16:23.981 DEBUG:boto:Token: None 2022-09-12T20:16:23.981 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/ 2022-09-12T20:16:23.982 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM= 2022-09-12T20:16:23.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM=', 'Content-Length': '0'} 2022-09-12T20:16:23.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9385a0ec0622e74-00631f9397-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:23.986 DEBUG:boto:b'jdpiak-75obj931000falseobj932022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:23.987 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:23.988 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:23.988 DEBUG:boto:path=/jdpiak-75/?prefix=obj93 2022-09-12T20:16:23.989 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj93 2022-09-12T20:16:23.989 DEBUG:boto:Method: GET 2022-09-12T20:16:23.990 DEBUG:boto:Path: /jdpiak-75/?prefix=obj93 2022-09-12T20:16:23.990 DEBUG:boto:Data: 2022-09-12T20:16:23.991 DEBUG:boto:Headers: {} 2022-09-12T20:16:23.991 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:23.992 DEBUG:boto:Port: 8000 2022-09-12T20:16:23.993 DEBUG:boto:Params: {} 2022-09-12T20:16:23.993 DEBUG:boto:Token: None 2022-09-12T20:16:23.994 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:23 GMT /jdpiak-75/ 2022-09-12T20:16:23.994 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM= 2022-09-12T20:16:23.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:23 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bUeC+lm1f+8VDXaWmhPJlurXMMM=', 'Content-Length': '0'} 2022-09-12T20:16:23.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6a377e2919ceb36-00631f9397-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:24.038 DEBUG:boto:b'jdpiak-75obj931000falseobj932022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:24.040 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:24.041 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:24.042 DEBUG:boto:Method: HEAD 2022-09-12T20:16:24.043 DEBUG:boto:Path: /jdpiak-75/obj93 2022-09-12T20:16:24.044 DEBUG:boto:Data: 2022-09-12T20:16:24.045 DEBUG:boto:Headers: {} 2022-09-12T20:16:24.045 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:24.046 DEBUG:boto:Port: 8001 2022-09-12T20:16:24.047 DEBUG:boto:Params: {} 2022-09-12T20:16:24.048 DEBUG:boto:Token: None 2022-09-12T20:16:24.049 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:24 GMT /jdpiak-75/obj93 2022-09-12T20:16:24.050 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L9sfHgbB0ptk+pEcVoyt51deQ1g= 2022-09-12T20:16:24.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L9sfHgbB0ptk+pEcVoyt51deQ1g=', 'Content-Length': '0'} 2022-09-12T20:16:24.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073e990c6069aea5f-00631f9398-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:24.055 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:24.056 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:24.057 DEBUG:boto:Method: HEAD 2022-09-12T20:16:24.058 DEBUG:boto:Path: /jdpiak-75/obj93 2022-09-12T20:16:24.059 DEBUG:boto:Data: 2022-09-12T20:16:24.060 DEBUG:boto:Headers: {} 2022-09-12T20:16:24.061 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:24.062 DEBUG:boto:Port: 8000 2022-09-12T20:16:24.063 DEBUG:boto:Params: {} 2022-09-12T20:16:24.064 DEBUG:boto:Token: None 2022-09-12T20:16:24.065 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:24 GMT /jdpiak-75/obj93 2022-09-12T20:16:24.066 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:L9sfHgbB0ptk+pEcVoyt51deQ1g= 2022-09-12T20:16:24.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:L9sfHgbB0ptk+pEcVoyt51deQ1g=', 'Content-Length': '0'} 2022-09-12T20:16:24.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ebe90c027291ccd-00631f9398-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:24.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-12T20:16:24.072 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:24.073 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:24.074 DEBUG:boto:Method: GET 2022-09-12T20:16:24.075 DEBUG:boto:Path: /jdpiak-75/obj93 2022-09-12T20:16:24.076 DEBUG:boto:Data: 2022-09-12T20:16:24.077 DEBUG:boto:Headers: {} 2022-09-12T20:16:24.077 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:24.078 DEBUG:boto:Port: 8001 2022-09-12T20:16:24.079 DEBUG:boto:Params: {} 2022-09-12T20:16:24.079 DEBUG:boto:Token: None 2022-09-12T20:16:24.080 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:24 GMT /jdpiak-75/obj93 2022-09-12T20:16:24.080 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:o/1jhOVwB0OCH1DUnGNMv8EwZqI= 2022-09-12T20:16:24.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:o/1jhOVwB0OCH1DUnGNMv8EwZqI=', 'Content-Length': '0'} 2022-09-12T20:16:24.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000924ee046279307d9-00631f9398-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:24.084 DEBUG:boto:path=/jdpiak-75/obj93 2022-09-12T20:16:24.085 DEBUG:boto:auth_path=/jdpiak-75/obj93 2022-09-12T20:16:24.085 DEBUG:boto:Method: GET 2022-09-12T20:16:24.086 DEBUG:boto:Path: /jdpiak-75/obj93 2022-09-12T20:16:24.086 DEBUG:boto:Data: 2022-09-12T20:16:24.087 DEBUG:boto:Headers: {} 2022-09-12T20:16:24.087 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:24.088 DEBUG:boto:Port: 8000 2022-09-12T20:16:24.088 DEBUG:boto:Params: {} 2022-09-12T20:16:24.089 DEBUG:boto:Token: None 2022-09-12T20:16:24.090 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:24 GMT /jdpiak-75/obj93 2022-09-12T20:16:24.090 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:o/1jhOVwB0OCH1DUnGNMv8EwZqI= 2022-09-12T20:16:24.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:o/1jhOVwB0OCH1DUnGNMv8EwZqI=', 'Content-Length': '0'} 2022-09-12T20:16:24.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:12:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbc5c0f3f8e97053-00631f9398-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:24.094 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:24.095 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:24.095 DEBUG:boto:path=/jdpiak-75/?prefix=obj93 2022-09-12T20:16:24.096 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj93 2022-09-12T20:16:24.097 DEBUG:boto:Method: GET 2022-09-12T20:16:24.097 DEBUG:boto:Path: /jdpiak-75/?prefix=obj93 2022-09-12T20:16:24.098 DEBUG:boto:Data: 2022-09-12T20:16:24.098 DEBUG:boto:Headers: {} 2022-09-12T20:16:24.099 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:24.099 DEBUG:boto:Port: 8001 2022-09-12T20:16:24.100 DEBUG:boto:Params: {} 2022-09-12T20:16:24.100 DEBUG:boto:Token: None 2022-09-12T20:16:24.101 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:24 GMT /jdpiak-75/ 2022-09-12T20:16:24.101 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+ZKWP/CfCPrYYeANnW6vTqb9ru4= 2022-09-12T20:16:24.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+ZKWP/CfCPrYYeANnW6vTqb9ru4=', 'Content-Length': '0'} 2022-09-12T20:16:24.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d37b3b1db0e29870-00631f9398-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:24.106 DEBUG:boto:b'jdpiak-75obj931000falseobj932022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:24.107 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:24.107 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:24.108 DEBUG:boto:path=/jdpiak-75/?prefix=obj93 2022-09-12T20:16:24.109 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj93 2022-09-12T20:16:24.109 DEBUG:boto:Method: GET 2022-09-12T20:16:24.110 DEBUG:boto:Path: /jdpiak-75/?prefix=obj93 2022-09-12T20:16:24.110 DEBUG:boto:Data: 2022-09-12T20:16:24.111 DEBUG:boto:Headers: {} 2022-09-12T20:16:24.111 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:24.112 DEBUG:boto:Port: 8000 2022-09-12T20:16:24.112 DEBUG:boto:Params: {} 2022-09-12T20:16:24.113 DEBUG:boto:Token: None 2022-09-12T20:16:24.113 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:24 GMT /jdpiak-75/ 2022-09-12T20:16:24.114 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+ZKWP/CfCPrYYeANnW6vTqb9ru4= 2022-09-12T20:16:24.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+ZKWP/CfCPrYYeANnW6vTqb9ru4=', 'Content-Length': '0'} 2022-09-12T20:16:24.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da882907d85e64f5-00631f9398-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:24.118 DEBUG:boto:b'jdpiak-75obj931000falseobj932022-09-12T20:12:24.352Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:24.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:16:24.120 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:24.121 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:24.121 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:24.122 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:24.122 DEBUG:boto:Method: GET 2022-09-12T20:16:24.123 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:24.123 DEBUG:boto:Data: 2022-09-12T20:16:24.124 DEBUG:boto:Headers: {} 2022-09-12T20:16:24.125 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:24.126 DEBUG:boto:Port: 8001 2022-09-12T20:16:24.127 DEBUG:boto:Params: {} 2022-09-12T20:16:24.128 DEBUG:boto:Token: None 2022-09-12T20:16:24.129 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:24 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:24.130 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Lxmh6hBtifR7iffV7PqP5hI8Q4U= 2022-09-12T20:16:24.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Lxmh6hBtifR7iffV7PqP5hI8Q4U=', 'Content-Length': '0'} 2022-09-12T20:16:24.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3b91604c9c8040c-00631f9398-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:24.136 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:24.136 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:24.137 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:24.138 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:24.138 DEBUG:boto:Method: GET 2022-09-12T20:16:24.139 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:24.139 DEBUG:boto:Data: 2022-09-12T20:16:24.140 DEBUG:boto:Headers: {} 2022-09-12T20:16:24.141 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:24.141 DEBUG:boto:Port: 8000 2022-09-12T20:16:24.142 DEBUG:boto:Params: {} 2022-09-12T20:16:24.142 DEBUG:boto:Token: None 2022-09-12T20:16:24.143 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:24 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:24.143 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Lxmh6hBtifR7iffV7PqP5hI8Q4U= 2022-09-12T20:16:24.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Lxmh6hBtifR7iffV7PqP5hI8Q4U=', 'Content-Length': '0'} 2022-09-12T20:16:24.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4682445d0acc26a-00631f9398-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:24.147 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:24.148 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:24.148 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:24.149 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:24.149 DEBUG:boto:Method: HEAD 2022-09-12T20:16:24.150 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:24.150 DEBUG:boto:Data: 2022-09-12T20:16:24.151 DEBUG:boto:Headers: {} 2022-09-12T20:16:24.151 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:24.152 DEBUG:boto:Port: 8001 2022-09-12T20:16:24.153 DEBUG:boto:Params: {} 2022-09-12T20:16:24.153 DEBUG:boto:Token: None 2022-09-12T20:16:24.154 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:24 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:24.155 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:URWQNlHXJTbWbtWPTEuyj2iRgbA= 2022-09-12T20:16:24.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:URWQNlHXJTbWbtWPTEuyj2iRgbA=', 'Content-Length': '0'} 2022-09-12T20:16:24.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd09f2af7c136872-00631f9398-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:24.159 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:24.160 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:24.160 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:24.161 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:24.161 DEBUG:boto:Method: HEAD 2022-09-12T20:16:24.162 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:24.162 DEBUG:boto:Data: 2022-09-12T20:16:24.163 DEBUG:boto:Headers: {} 2022-09-12T20:16:24.164 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:24.165 DEBUG:boto:Port: 8000 2022-09-12T20:16:24.165 DEBUG:boto:Params: {} 2022-09-12T20:16:24.166 DEBUG:boto:Token: None 2022-09-12T20:16:24.166 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:24 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:24.167 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:URWQNlHXJTbWbtWPTEuyj2iRgbA= 2022-09-12T20:16:24.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:URWQNlHXJTbWbtWPTEuyj2iRgbA=', 'Content-Length': '0'} 2022-09-12T20:16:24.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac3d505629d7f8b8-00631f9398-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:24.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:16:24.171 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:24.172 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:24.173 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:24.173 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:24.174 DEBUG:boto:Method: GET 2022-09-12T20:16:24.175 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:24.175 DEBUG:boto:Data: 2022-09-12T20:16:24.176 DEBUG:boto:Headers: {} 2022-09-12T20:16:24.177 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:24.177 DEBUG:boto:Port: 8001 2022-09-12T20:16:24.178 DEBUG:boto:Params: {} 2022-09-12T20:16:24.179 DEBUG:boto:Token: None 2022-09-12T20:16:24.180 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:24 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:24.181 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Lxmh6hBtifR7iffV7PqP5hI8Q4U= 2022-09-12T20:16:24.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Lxmh6hBtifR7iffV7PqP5hI8Q4U=', 'Content-Length': '0'} 2022-09-12T20:16:24.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e56a48fdb4204dd-00631f9398-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:24.226 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:24.227 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:24.228 DEBUG:boto:path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:24.228 DEBUG:boto:auth_path=/jdpiak-75/obj95?versionId=null 2022-09-12T20:16:24.229 DEBUG:boto:Method: GET 2022-09-12T20:16:24.229 DEBUG:boto:Path: /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:24.230 DEBUG:boto:Data: 2022-09-12T20:16:24.230 DEBUG:boto:Headers: {} 2022-09-12T20:16:24.231 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:24.231 DEBUG:boto:Port: 8000 2022-09-12T20:16:24.232 DEBUG:boto:Params: {} 2022-09-12T20:16:24.233 DEBUG:boto:Token: None 2022-09-12T20:16:24.233 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:24 GMT /jdpiak-75/obj95?versionId=null 2022-09-12T20:16:24.234 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Lxmh6hBtifR7iffV7PqP5hI8Q4U= 2022-09-12T20:16:24.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Lxmh6hBtifR7iffV7PqP5hI8Q4U=', 'Content-Length': '0'} 2022-09-12T20:16:24.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009672071bd69bc7f4-00631f9398-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:24.238 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:24.239 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:24.240 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:16:24.240 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:16:24.241 DEBUG:boto:Method: GET 2022-09-12T20:16:24.241 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:16:24.242 DEBUG:boto:Data: 2022-09-12T20:16:24.242 DEBUG:boto:Headers: {} 2022-09-12T20:16:24.243 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:24.243 DEBUG:boto:Port: 8001 2022-09-12T20:16:24.244 DEBUG:boto:Params: {} 2022-09-12T20:16:24.245 DEBUG:boto:Token: None 2022-09-12T20:16:24.245 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:24 GMT /jdpiak-75/ 2022-09-12T20:16:24.246 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+ZKWP/CfCPrYYeANnW6vTqb9ru4= 2022-09-12T20:16:24.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+ZKWP/CfCPrYYeANnW6vTqb9ru4=', 'Content-Length': '0'} 2022-09-12T20:16:24.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f951fd5106577e49-00631f9398-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:24.251 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:24.252 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:24.252 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:24.253 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:16:24.253 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:16:24.254 DEBUG:boto:Method: GET 2022-09-12T20:16:24.256 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:16:24.256 DEBUG:boto:Data: 2022-09-12T20:16:24.257 DEBUG:boto:Headers: {} 2022-09-12T20:16:24.258 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:24.259 DEBUG:boto:Port: 8000 2022-09-12T20:16:24.259 DEBUG:boto:Params: {} 2022-09-12T20:16:24.261 DEBUG:boto:Token: None 2022-09-12T20:16:24.262 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:24 GMT /jdpiak-75/ 2022-09-12T20:16:24.263 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+ZKWP/CfCPrYYeANnW6vTqb9ru4= 2022-09-12T20:16:24.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+ZKWP/CfCPrYYeANnW6vTqb9ru4=', 'Content-Length': '0'} 2022-09-12T20:16:24.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0360e4138eeb76f-00631f9398-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:24.267 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:24.268 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:24.269 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:24.269 DEBUG:boto:Method: HEAD 2022-09-12T20:16:24.270 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:16:24.270 DEBUG:boto:Data: 2022-09-12T20:16:24.271 DEBUG:boto:Headers: {} 2022-09-12T20:16:24.272 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:24.273 DEBUG:boto:Port: 8001 2022-09-12T20:16:24.274 DEBUG:boto:Params: {} 2022-09-12T20:16:24.275 DEBUG:boto:Token: None 2022-09-12T20:16:24.276 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:24 GMT /jdpiak-75/obj95 2022-09-12T20:16:24.277 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m7s+nKN85IAFpeXd3+bB5fFSmOQ= 2022-09-12T20:16:24.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m7s+nKN85IAFpeXd3+bB5fFSmOQ=', 'Content-Length': '0'} 2022-09-12T20:16:24.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041396625b9de00d6-00631f9398-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:24.282 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:24.283 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:24.283 DEBUG:boto:Method: HEAD 2022-09-12T20:16:24.284 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:16:24.285 DEBUG:boto:Data: 2022-09-12T20:16:24.285 DEBUG:boto:Headers: {} 2022-09-12T20:16:24.286 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:24.286 DEBUG:boto:Port: 8000 2022-09-12T20:16:24.287 DEBUG:boto:Params: {} 2022-09-12T20:16:24.288 DEBUG:boto:Token: None 2022-09-12T20:16:24.289 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:16:24 GMT /jdpiak-75/obj95 2022-09-12T20:16:24.289 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:m7s+nKN85IAFpeXd3+bB5fFSmOQ= 2022-09-12T20:16:24.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:m7s+nKN85IAFpeXd3+bB5fFSmOQ=', 'Content-Length': '0'} 2022-09-12T20:16:24.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000343cc47e321e40f2-00631f9398-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:24.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-12T20:16:24.293 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:24.294 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:24.295 DEBUG:boto:Method: GET 2022-09-12T20:16:24.295 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:16:24.296 DEBUG:boto:Data: 2022-09-12T20:16:24.296 DEBUG:boto:Headers: {} 2022-09-12T20:16:24.297 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:24.297 DEBUG:boto:Port: 8001 2022-09-12T20:16:24.298 DEBUG:boto:Params: {} 2022-09-12T20:16:24.298 DEBUG:boto:Token: None 2022-09-12T20:16:24.299 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:24 GMT /jdpiak-75/obj95 2022-09-12T20:16:24.299 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SiUTenD6FdIkvg9geuSQO/9bWjo= 2022-09-12T20:16:24.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SiUTenD6FdIkvg9geuSQO/9bWjo=', 'Content-Length': '0'} 2022-09-12T20:16:24.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041e7f8a7c14ce11d-00631f9398-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:24.303 DEBUG:boto:path=/jdpiak-75/obj95 2022-09-12T20:16:24.304 DEBUG:boto:auth_path=/jdpiak-75/obj95 2022-09-12T20:16:24.304 DEBUG:boto:Method: GET 2022-09-12T20:16:24.305 DEBUG:boto:Path: /jdpiak-75/obj95 2022-09-12T20:16:24.305 DEBUG:boto:Data: 2022-09-12T20:16:24.306 DEBUG:boto:Headers: {} 2022-09-12T20:16:24.306 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:24.307 DEBUG:boto:Port: 8000 2022-09-12T20:16:24.307 DEBUG:boto:Params: {} 2022-09-12T20:16:24.308 DEBUG:boto:Token: None 2022-09-12T20:16:24.308 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:24 GMT /jdpiak-75/obj95 2022-09-12T20:16:24.309 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SiUTenD6FdIkvg9geuSQO/9bWjo= 2022-09-12T20:16:24.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SiUTenD6FdIkvg9geuSQO/9bWjo=', 'Content-Length': '0'} 2022-09-12T20:16:24.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:06:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092360126e635abca-00631f9398-189d-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:16:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:24.313 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:24.314 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:24.314 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:16:24.315 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:16:24.315 DEBUG:boto:Method: GET 2022-09-12T20:16:24.316 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:16:24.316 DEBUG:boto:Data: 2022-09-12T20:16:24.317 DEBUG:boto:Headers: {} 2022-09-12T20:16:24.317 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:16:24.318 DEBUG:boto:Port: 8001 2022-09-12T20:16:24.318 DEBUG:boto:Params: {} 2022-09-12T20:16:24.319 DEBUG:boto:Token: None 2022-09-12T20:16:24.319 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:24 GMT /jdpiak-75/ 2022-09-12T20:16:24.320 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+ZKWP/CfCPrYYeANnW6vTqb9ru4= 2022-09-12T20:16:24.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+ZKWP/CfCPrYYeANnW6vTqb9ru4=', 'Content-Length': '0'} 2022-09-12T20:16:24.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098c353a89c47d071-00631f9398-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:24.366 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:24.367 DEBUG:boto:path=/jdpiak-75/ 2022-09-12T20:16:24.368 DEBUG:boto:auth_path=/jdpiak-75/ 2022-09-12T20:16:24.368 DEBUG:boto:path=/jdpiak-75/?prefix=obj95 2022-09-12T20:16:24.369 DEBUG:boto:auth_path=/jdpiak-75/?prefix=obj95 2022-09-12T20:16:24.369 DEBUG:boto:Method: GET 2022-09-12T20:16:24.370 DEBUG:boto:Path: /jdpiak-75/?prefix=obj95 2022-09-12T20:16:24.370 DEBUG:boto:Data: 2022-09-12T20:16:24.371 DEBUG:boto:Headers: {} 2022-09-12T20:16:24.372 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:24.372 DEBUG:boto:Port: 8000 2022-09-12T20:16:24.373 DEBUG:boto:Params: {} 2022-09-12T20:16:24.373 DEBUG:boto:Token: None 2022-09-12T20:16:24.374 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:16:24 GMT /jdpiak-75/ 2022-09-12T20:16:24.375 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+ZKWP/CfCPrYYeANnW6vTqb9ru4= 2022-09-12T20:16:24.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:16:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+ZKWP/CfCPrYYeANnW6vTqb9ru4=', 'Content-Length': '0'} 2022-09-12T20:16:24.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcc6c27c46fdc2e8-00631f9398-189d-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:16:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:24.379 DEBUG:boto:b'jdpiak-75obj951000falseobj952022-09-12T20:06:16.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:16:24.380 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-75 zones={test-zone3, test-zone2} 2022-09-12T20:16:24.381 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2022-09-12T20:16:24.382 INFO:teuthology.orchestra.run:waiting for 300 2022-09-12T20:16:24.383 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T20:16:24.381+0000 7fb7d5182700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 144065) UID: 0 2022-09-12T20:16:24.383 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T20:16:24.381+0000 7fb7d93ed600 -1 shutting down 2022-09-12T20:16:25.324 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T20:16:25.323+0000 7f790c812700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16) 2022-09-12T20:16:28.437 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2022-09-12T20:16:30.480 INFO:tasks.rgw.c2.client.0:Stopped 2022-09-12T20:16:30.481 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'list', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--max-entries', '10000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:16:30.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'list', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--max-entries', '10000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:16:30.482 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket list --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --max-entries 10000 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:16:30.543 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:30.543 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:30.591 INFO:teuthology.orchestra.run.smithi138.stdout:[ 2022-09-12T20:16:30.591 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.592 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj11", 2022-09-12T20:16:30.592 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.592 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.593 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.593 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1 2022-09-12T20:16:30.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.594 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.594 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.594 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.595 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.595 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 16, 2022-09-12T20:16:30.595 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:03:02.418388Z", 2022-09-12T20:16:30.596 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.596 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.596 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.596 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.597 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.597 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.597 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.597 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.598 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_aiN1ULdYFaJSwtR4J0baP5hZys9S9a3", 2022-09-12T20:16:30.598 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.598 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.599 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.600 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj111", 2022-09-12T20:16:30.600 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.600 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.600 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.601 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 5 2022-09-12T20:16:30.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.601 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.601 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.602 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.602 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.602 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 3, 2022-09-12T20:16:30.602 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:14:27.070655Z", 2022-09-12T20:16:30.603 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.603 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.604 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.604 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.606 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.606 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.606 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.606 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.607 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_zZ4LQXj2-yJrYSQprP8tLgQbPR_ncqa", 2022-09-12T20:16:30.607 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.608 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.608 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.608 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.609 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj115", 2022-09-12T20:16:30.609 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.609 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.609 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.610 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 4 2022-09-12T20:16:30.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.610 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.611 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.611 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.611 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.611 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 6, 2022-09-12T20:16:30.612 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:08:38.713245Z", 2022-09-12T20:16:30.612 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.612 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.612 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.613 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.613 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.613 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.614 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.614 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.615 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.615 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_b35tj5GYv44bQ-XhxM2VZV2VhLvEKvg", 2022-09-12T20:16:30.616 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.616 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.616 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.617 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.617 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.617 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj116", 2022-09-12T20:16:30.618 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.618 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.618 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.618 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 2 2022-09-12T20:16:30.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.619 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.619 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.619 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.620 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.620 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 3, 2022-09-12T20:16:30.620 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:10:09.066316Z", 2022-09-12T20:16:30.620 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.621 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.621 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.621 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.622 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.622 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.622 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.622 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.623 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_owYACU7i0b3wC-fQPhzESOEcyrufp_R", 2022-09-12T20:16:30.623 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.624 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.624 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.625 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.625 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj124", 2022-09-12T20:16:30.625 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.626 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.626 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.626 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 11 2022-09-12T20:16:30.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.627 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.627 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.627 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.627 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.628 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 16, 2022-09-12T20:16:30.628 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:12:24.415221Z", 2022-09-12T20:16:30.628 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.628 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.629 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.629 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.630 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.630 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.630 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.631 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.632 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_05CY_dRfWqUcAdZyL_rNwl2RBXjEx4V", 2022-09-12T20:16:30.632 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.632 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.633 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.634 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj145", 2022-09-12T20:16:30.634 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.634 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.635 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.635 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 2 2022-09-12T20:16:30.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.635 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.636 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.636 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.637 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.637 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 3, 2022-09-12T20:16:30.637 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:10:09.126236Z", 2022-09-12T20:16:30.638 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.638 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.638 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.638 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.639 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.639 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.639 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.640 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.640 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_boeyjy-FrBNbdkEISHlX4KvGERBeJx7", 2022-09-12T20:16:30.641 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.641 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.641 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.642 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj148", 2022-09-12T20:16:30.643 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.643 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.644 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.644 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 5 2022-09-12T20:16:30.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.645 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.645 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.646 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.646 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.646 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 16, 2022-09-12T20:16:30.646 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:14:27.094848Z", 2022-09-12T20:16:30.647 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.647 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.647 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.647 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.648 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.648 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.648 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.648 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.649 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_A_kt06WjBKRa_DoctesJFPKvwOitBfy", 2022-09-12T20:16:30.649 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.650 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.650 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.651 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj155", 2022-09-12T20:16:30.651 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.651 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.651 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.652 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 4 2022-09-12T20:16:30.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.653 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.653 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.654 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.654 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.654 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 3, 2022-09-12T20:16:30.655 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:12:24.442347Z", 2022-09-12T20:16:30.655 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.655 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.655 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.656 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.656 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.656 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.656 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.657 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.657 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Hx9J1yV3krALI9gkT2uslCZSQDIfU9i", 2022-09-12T20:16:30.658 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.658 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.659 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.659 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj17", 2022-09-12T20:16:30.660 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.660 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.660 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.660 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 3 2022-09-12T20:16:30.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.661 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.661 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.662 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.662 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.663 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 16, 2022-09-12T20:16:30.663 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:03:28.498996Z", 2022-09-12T20:16:30.664 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.664 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.665 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.665 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.665 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.666 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.666 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.666 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.667 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_R06shpElnwUMjSDL2Z3AFGBqIujbTpy", 2022-09-12T20:16:30.667 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.668 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.668 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.669 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj185", 2022-09-12T20:16:30.669 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.670 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.670 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.670 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 4 2022-09-12T20:16:30.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.671 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.672 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.672 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.673 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.673 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 16, 2022-09-12T20:16:30.674 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:14:27.121737Z", 2022-09-12T20:16:30.674 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.675 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.675 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.676 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.676 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.676 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.677 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.677 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.678 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_NOgp4yOTg_pM2oObVW8X8lkl_OJJTpn", 2022-09-12T20:16:30.678 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.679 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.679 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.680 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj19", 2022-09-12T20:16:30.681 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.681 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.681 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.681 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 5 2022-09-12T20:16:30.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.682 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.683 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.683 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.684 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.684 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 16, 2022-09-12T20:16:30.684 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:06:16.800579Z", 2022-09-12T20:16:30.684 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.685 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.685 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.685 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.686 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.686 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.686 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.687 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.687 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_I_aKcazHQ_d9RdJKpka46V9LLSA5zOg", 2022-09-12T20:16:30.688 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.688 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.688 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.689 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj22", 2022-09-12T20:16:30.689 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.689 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.690 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.690 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 6 2022-09-12T20:16:30.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.691 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.691 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.691 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.692 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.692 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 5, 2022-09-12T20:16:30.692 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:03:02.445885Z", 2022-09-12T20:16:30.693 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.693 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.693 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.694 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.694 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.694 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.694 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.695 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.695 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_aSzSzwxcN3RZQR63TdFhiQKyCpx8RRG", 2022-09-12T20:16:30.695 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.696 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.696 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.697 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj23", 2022-09-12T20:16:30.697 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.697 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.698 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.698 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 6 2022-09-12T20:16:30.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.698 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.699 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.699 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.699 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.699 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 5, 2022-09-12T20:16:30.700 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:08:38.567385Z", 2022-09-12T20:16:30.700 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.701 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.701 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.701 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.702 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.702 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.702 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.703 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.703 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_bce4aSW3MEqkFs3YcfAn-_h0wZJe9Yb", 2022-09-12T20:16:30.703 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.704 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.704 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.705 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj29", 2022-09-12T20:16:30.705 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.705 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.705 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.706 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 4 2022-09-12T20:16:30.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.706 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.707 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.707 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.707 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.707 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 3, 2022-09-12T20:16:30.708 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:10:08.950509Z", 2022-09-12T20:16:30.708 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.708 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.708 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.709 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.709 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.709 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.709 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.710 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.710 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_NCGJgKAOcf_shu5GTvn72QFAsS2JgKp", 2022-09-12T20:16:30.711 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.711 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.711 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.712 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj31", 2022-09-12T20:16:30.713 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.713 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.713 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.714 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 6 2022-09-12T20:16:30.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.714 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.715 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.715 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.715 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.716 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 16, 2022-09-12T20:16:30.716 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:12:24.266585Z", 2022-09-12T20:16:30.716 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.717 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.717 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.717 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.717 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.718 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.718 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.718 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.719 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.719 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_o_w2gfzWTDEv6PvBkJmcXb1UZDDL1rM", 2022-09-12T20:16:30.719 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.719 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.720 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.721 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj33", 2022-09-12T20:16:30.721 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.722 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.722 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.722 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 3 2022-09-12T20:16:30.723 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.723 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.723 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.723 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.724 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.724 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 3, 2022-09-12T20:16:30.724 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:03:02.515258Z", 2022-09-12T20:16:30.724 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.725 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.725 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.725 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.726 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.726 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.726 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.726 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.727 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.727 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_3gvjcMfcTe8NBjQ4VV1VD8Ah0p-KMwU", 2022-09-12T20:16:30.727 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.727 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.728 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.728 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj34", 2022-09-12T20:16:30.729 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.729 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.729 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.730 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 21 2022-09-12T20:16:30.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.731 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.731 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.731 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.732 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.732 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 6, 2022-09-12T20:16:30.732 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:03:28.524596Z", 2022-09-12T20:16:30.732 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.733 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.733 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.733 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.733 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.734 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.734 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.734 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.735 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_1_GbF-lumUjuqMhuig5CH9rLPU6iIEV", 2022-09-12T20:16:30.735 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.735 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.735 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.736 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj37", 2022-09-12T20:16:30.737 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.737 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.737 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.738 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 2 2022-09-12T20:16:30.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.738 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.738 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.739 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.739 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.739 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 6, 2022-09-12T20:16:30.739 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:14:26.981706Z", 2022-09-12T20:16:30.740 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.740 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.740 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.741 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.741 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.741 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.742 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.742 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.743 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_qhitjmS-3NjH_JCFx2o0kqk0CmB3uGT", 2022-09-12T20:16:30.743 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.743 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.744 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.745 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj38", 2022-09-12T20:16:30.745 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.745 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.745 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.746 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 4 2022-09-12T20:16:30.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.746 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.746 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.747 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.747 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.747 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 6, 2022-09-12T20:16:30.747 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:06:16.824591Z", 2022-09-12T20:16:30.748 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.748 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.748 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.749 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.749 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.749 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.750 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.750 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.750 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.751 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_50K7Uwi8oKpFnJWf0x3D3ap8FjFLIl3", 2022-09-12T20:16:30.751 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.751 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.751 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.752 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.752 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.752 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj44", 2022-09-12T20:16:30.752 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.753 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.753 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.753 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 3 2022-09-12T20:16:30.753 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.754 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.754 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.754 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.755 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.755 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 16, 2022-09-12T20:16:30.755 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:03:02.545371Z", 2022-09-12T20:16:30.755 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.756 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.756 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.756 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.757 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.757 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.757 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.758 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.758 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.758 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_JGEt_yq8UBPGmKFcYOWfuBrpTeayJBc", 2022-09-12T20:16:30.759 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.759 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.759 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.760 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.760 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.760 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj46", 2022-09-12T20:16:30.761 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.761 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.761 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.761 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1 2022-09-12T20:16:30.762 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.762 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.762 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.762 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.763 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.763 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 16, 2022-09-12T20:16:30.763 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:08:38.594488Z", 2022-09-12T20:16:30.763 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.764 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.764 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.764 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.764 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.765 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.765 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.765 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.765 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.766 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_Faa78yyFkZfd5jPBJRLQlnv2vayxcyp", 2022-09-12T20:16:30.766 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.766 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.767 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.767 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.767 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.767 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj51", 2022-09-12T20:16:30.768 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.768 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.768 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.769 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 4 2022-09-12T20:16:30.769 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.769 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.770 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.770 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.771 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.771 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 6, 2022-09-12T20:16:30.772 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:03:28.556115Z", 2022-09-12T20:16:30.772 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.772 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.772 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.773 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.773 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.773 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.773 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.774 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.774 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.774 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_vSCW-pu7BtvJ2xdYTxm6mE0UHg9z_Lu", 2022-09-12T20:16:30.774 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.775 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.775 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.775 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.775 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.776 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj55", 2022-09-12T20:16:30.776 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.776 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.776 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.777 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 3 2022-09-12T20:16:30.777 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.777 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.777 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.778 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.778 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.778 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 5, 2022-09-12T20:16:30.779 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:03:02.577714Z", 2022-09-12T20:16:30.779 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.780 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.780 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.781 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.781 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.781 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.781 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.782 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.782 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.782 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_pLxluwcnjIsogiVYyNDYG-SsLt0eD5T", 2022-09-12T20:16:30.783 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.783 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.783 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.783 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.784 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.784 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj57", 2022-09-12T20:16:30.784 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.785 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.785 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.785 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 3 2022-09-12T20:16:30.786 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.786 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.786 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.787 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.787 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.787 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 6, 2022-09-12T20:16:30.787 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:06:16.885756Z", 2022-09-12T20:16:30.788 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.788 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.788 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.789 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.789 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.789 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.790 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.790 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.790 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.791 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_8TPnBCe9SXROyyGzg0d2-w3_Pd7p3H6", 2022-09-12T20:16:30.791 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.791 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.791 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.792 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.792 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.792 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj58", 2022-09-12T20:16:30.792 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.793 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.793 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.793 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 18 2022-09-12T20:16:30.793 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.794 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.794 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.794 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.794 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.795 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 3, 2022-09-12T20:16:30.795 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:10:08.976956Z", 2022-09-12T20:16:30.795 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.795 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.796 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.796 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.796 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.796 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.797 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.797 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.797 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.797 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_85HlSfOydCwpOhWexuoAE0L8jZ16OBk", 2022-09-12T20:16:30.798 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.798 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.799 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.799 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.799 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.800 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj62", 2022-09-12T20:16:30.800 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.800 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.801 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.801 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 4 2022-09-12T20:16:30.801 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.801 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.802 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.802 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.802 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.802 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 5, 2022-09-12T20:16:30.803 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:12:24.328717Z", 2022-09-12T20:16:30.803 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.803 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.803 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.804 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.804 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.804 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.804 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.805 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.805 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.805 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_cyEKtSTbn_licZYVClnnSu3YFmKBaoJ", 2022-09-12T20:16:30.805 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.806 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.806 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.806 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.806 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.807 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj68", 2022-09-12T20:16:30.807 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.807 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.808 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.808 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 9 2022-09-12T20:16:30.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.809 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.809 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.809 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.810 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.810 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 16, 2022-09-12T20:16:30.810 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:03:28.622387Z", 2022-09-12T20:16:30.810 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.811 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.811 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.811 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.811 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.812 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.812 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.812 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.812 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.813 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_S6BSE93Avs3mYpeC8PBHTzwf_RwKCG-", 2022-09-12T20:16:30.813 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.813 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.813 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.814 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.814 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.814 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj69", 2022-09-12T20:16:30.814 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.815 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.815 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.815 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 8 2022-09-12T20:16:30.815 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.816 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.816 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.816 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.817 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.817 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 3, 2022-09-12T20:16:30.817 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:08:38.624845Z", 2022-09-12T20:16:30.818 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.818 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.818 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.819 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.819 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.819 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.820 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.820 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.821 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_MZn84D8FrrPtUdzU8v8n9NnCqlHY8ej", 2022-09-12T20:16:30.821 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.821 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.821 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.822 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj74", 2022-09-12T20:16:30.822 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.823 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.823 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.823 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 6 2022-09-12T20:16:30.823 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.824 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.824 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.824 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.824 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.825 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 16, 2022-09-12T20:16:30.825 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:14:27.006386Z", 2022-09-12T20:16:30.825 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.825 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.826 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.826 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.826 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.827 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.827 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.827 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.828 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_cKbu5VfGz03JWP68gqQ4r49kBYvyit3", 2022-09-12T20:16:30.828 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.828 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.828 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.829 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj76", 2022-09-12T20:16:30.830 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.830 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.830 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.831 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 8 2022-09-12T20:16:30.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.831 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.832 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.832 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.832 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.833 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 16, 2022-09-12T20:16:30.833 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:06:16.908179Z", 2022-09-12T20:16:30.833 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.833 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.834 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.834 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.834 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.834 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.835 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.835 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.835 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_MtIiwjqfUlF-gCTqEO03eJFxiIX5nnp", 2022-09-12T20:16:30.836 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.836 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.836 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.837 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj85", 2022-09-12T20:16:30.837 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.837 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.838 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.838 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 17 2022-09-12T20:16:30.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.839 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.839 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.840 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.840 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.840 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 3, 2022-09-12T20:16:30.840 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:03:28.644849Z", 2022-09-12T20:16:30.841 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.841 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.841 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.841 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.842 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.842 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.842 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.842 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.843 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_48ND4kg7sp_eT0qixfWwy7n_ju-vUrp", 2022-09-12T20:16:30.843 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.843 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.844 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.844 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.844 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.844 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj87", 2022-09-12T20:16:30.845 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.845 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.845 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.846 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 12 2022-09-12T20:16:30.846 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.846 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.847 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.847 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.849 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.849 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 3, 2022-09-12T20:16:30.849 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:10:09.041774Z", 2022-09-12T20:16:30.850 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.850 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.850 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.851 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.851 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.851 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.851 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.852 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.852 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.852 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_ocFt56Y8Jydr_h4MVnsa1tUQuQFZ0By", 2022-09-12T20:16:30.852 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.853 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.853 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.853 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.853 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.854 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj92", 2022-09-12T20:16:30.854 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.854 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.854 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.855 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 12 2022-09-12T20:16:30.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.855 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.855 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.856 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.856 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.856 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 16, 2022-09-12T20:16:30.857 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:08:38.685314Z", 2022-09-12T20:16:30.857 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.857 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.858 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.858 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.858 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.859 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.859 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.859 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.860 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_OfPCAvZRsdOntu6dIMMaDM5ObyLH9se", 2022-09-12T20:16:30.860 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.860 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.861 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.861 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.862 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj93", 2022-09-12T20:16:30.862 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.862 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.862 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.863 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 3 2022-09-12T20:16:30.863 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.863 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.863 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.864 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.864 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.864 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 5, 2022-09-12T20:16:30.865 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:12:24.352743Z", 2022-09-12T20:16:30.865 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.865 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.865 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.866 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.866 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.866 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.866 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.867 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.867 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.867 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_HUxxoQRGFZfZZsKWX0dXviESn4ISXam", 2022-09-12T20:16:30.868 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.868 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.868 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.869 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.869 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:16:30.869 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "obj95", 2022-09-12T20:16:30.869 INFO:teuthology.orchestra.run.smithi138.stdout: "instance": "", 2022-09-12T20:16:30.870 INFO:teuthology.orchestra.run.smithi138.stdout: "ver": { 2022-09-12T20:16:30.870 INFO:teuthology.orchestra.run.smithi138.stdout: "pool": 9, 2022-09-12T20:16:30.870 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1 2022-09-12T20:16:30.871 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.871 INFO:teuthology.orchestra.run.smithi138.stdout: "locator": "", 2022-09-12T20:16:30.871 INFO:teuthology.orchestra.run.smithi138.stdout: "exists": true, 2022-09-12T20:16:30.871 INFO:teuthology.orchestra.run.smithi138.stdout: "meta": { 2022-09-12T20:16:30.872 INFO:teuthology.orchestra.run.smithi138.stdout: "category": 1, 2022-09-12T20:16:30.872 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 3, 2022-09-12T20:16:30.872 INFO:teuthology.orchestra.run.smithi138.stdout: "mtime": "2022-09-12T20:06:16.969390Z", 2022-09-12T20:16:30.872 INFO:teuthology.orchestra.run.smithi138.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-12T20:16:30.873 INFO:teuthology.orchestra.run.smithi138.stdout: "storage_class": "", 2022-09-12T20:16:30.873 INFO:teuthology.orchestra.run.smithi138.stdout: "owner": "rgw-multisite-test-user", 2022-09-12T20:16:30.873 INFO:teuthology.orchestra.run.smithi138.stdout: "owner_display_name": "Multisite Test User", 2022-09-12T20:16:30.874 INFO:teuthology.orchestra.run.smithi138.stdout: "content_type": "application/octet-stream", 2022-09-12T20:16:30.874 INFO:teuthology.orchestra.run.smithi138.stdout: "accounted_size": 3, 2022-09-12T20:16:30.874 INFO:teuthology.orchestra.run.smithi138.stdout: "user_data": "", 2022-09-12T20:16:30.884 INFO:teuthology.orchestra.run.smithi138.stdout: "appendable": false 2022-09-12T20:16:30.884 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:16:30.885 INFO:teuthology.orchestra.run.smithi138.stdout: "tag": "_SkGUI9ZHHtnC1wAfniWkQ6JmRY5FiO6", 2022-09-12T20:16:30.885 INFO:teuthology.orchestra.run.smithi138.stdout: "flags": 0, 2022-09-12T20:16:30.885 INFO:teuthology.orchestra.run.smithi138.stdout: "pending_map": [], 2022-09-12T20:16:30.885 INFO:teuthology.orchestra.run.smithi138.stdout: "versioned_epoch": 0 2022-09-12T20:16:30.886 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:16:30.886 INFO:teuthology.orchestra.run.smithi138.stdout:] 2022-09-12T20:16:30.888 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:30.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:30.889 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj11 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:30.942 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:30.944 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:31.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj111', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:31.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj111', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:31.036 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj111 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:31.087 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:31.088 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:31.205 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj115', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:31.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj115', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:31.206 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj115 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:31.258 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:31.258 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:31.384 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj116', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:31.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj116', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:31.386 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj116 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:31.437 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:31.437 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:31.541 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj124', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:31.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj124', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:31.542 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj124 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:31.593 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:31.594 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:31.705 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj145', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:31.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj145', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:31.706 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj145 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:31.758 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:31.759 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:31.880 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj148', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:31.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj148', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:31.882 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj148 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:31.934 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:31.935 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:32.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj155', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:32.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj155', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:32.052 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj155 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:32.103 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:32.104 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:32.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:32.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:32.215 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj17 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:32.266 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:32.267 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:32.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj185', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:32.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj185', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:32.366 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj185 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:32.417 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:32.417 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:32.538 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:32.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:32.540 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj19 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:32.593 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:32.594 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:32.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:32.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:32.721 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj22 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:32.772 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:32.773 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:32.901 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:32.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:32.904 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj23 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:32.956 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:32.957 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:33.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj29', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:33.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj29', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:33.064 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj29 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:33.115 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:33.116 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:33.231 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj31', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:33.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj31', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:33.233 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj31 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:33.286 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:33.286 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:33.396 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj33', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:33.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj33', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:33.398 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj33 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:33.449 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:33.449 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:33.565 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj34', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:33.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj34', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:33.566 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj34 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:33.618 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:33.619 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:33.632 INFO:tasks.rgw.c1.client.0.smithi035.stdout:2022-09-12T20:16:33.631+0000 7f29e0f49700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-16) 2022-09-12T20:16:33.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj37', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:33.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj37', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:33.735 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj37 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:33.769 INFO:tasks.rgw.c1.client.0.smithi035.stdout:2022-09-12T20:16:33.769+0000 7f29e0f49700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5) 2022-09-12T20:16:33.786 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:33.786 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:33.825 INFO:tasks.rgw.c1.client.0.smithi035.stdout:2022-09-12T20:16:33.825+0000 7f29e0f49700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-16) 2022-09-12T20:16:33.903 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj38', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:33.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj38', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:33.905 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj38 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:33.912 INFO:tasks.rgw.c1.client.0.smithi035.stdout:2022-09-12T20:16:33.912+0000 7f29e0f49700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16) 2022-09-12T20:16:33.958 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:33.958 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:34.042 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2022-09-12T20:16:34.063 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:34.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:34.065 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj44 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:34.116 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:34.116 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:34.254 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj46', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:34.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj46', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:34.256 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj46 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:34.308 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:34.309 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:34.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj51', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:34.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj51', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:34.416 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj51 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:34.468 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:34.469 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:34.592 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj55', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:34.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj55', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:34.593 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj55 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:34.646 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:34.647 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:34.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj57', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:34.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj57', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:34.742 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj57 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:34.793 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:34.794 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:34.905 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj58', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:34.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj58', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:34.917 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj58 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:34.968 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:34.969 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:35.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:35.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:35.093 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj62 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:35.144 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:35.144 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:35.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj68', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:35.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj68', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:35.266 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj68 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:35.317 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:35.317 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:35.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:35.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:35.427 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj69 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:35.479 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:35.479 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:35.583 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj74', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:35.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj74', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:35.585 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj74 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:35.636 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:35.637 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:35.763 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj76', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:35.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj76', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:35.766 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj76 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:35.818 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:35.818 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:35.931 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:35.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:35.932 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj85 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:35.985 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:35.986 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:36.089 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj87', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:36.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj87', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:36.091 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj87 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:36.152 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:36.152 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:36.258 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj92', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:36.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj92', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:36.260 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj92 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:36.312 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:36.313 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:36.434 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj93', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:36.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj93', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:36.435 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj93 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:36.487 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:36.487 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:36.594 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj95', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:36.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--object', 'obj95', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:36.596 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --object obj95 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:36.647 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:36.648 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:36.757 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'trim', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:36.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'trim', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:36.759 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog trim --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:36.808 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:36.808 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:36.884 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:36.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:36.886 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:36.939 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:36.940 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:37.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:37.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'run', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-75', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:16:37.004 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync run --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-75 --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:16:37.056 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:16:37.057 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:16:37.144 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.142+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: sync status for source bucket: init. lease is: not taken. stop indications is: 0 2022-09-12T20:16:37.150 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.148+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: waiting on bucket lease 2022-09-12T20:16:37.151 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.149+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: status after acquiring the lock is: init 2022-09-12T20:16:37.173 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.171+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: listing bucket for full sync 2022-09-12T20:16:37.181 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj11 2022-09-12T20:16:37.182 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj11]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj11 op=0 op_state=1 2022-09-12T20:16:37.182 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj11]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj11[0] 2022-09-12T20:16:37.182 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj111 2022-09-12T20:16:37.183 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj111]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj111 op=0 op_state=1 2022-09-12T20:16:37.183 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 0 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj111]: : injecting data sync error on key=obj111 2022-09-12T20:16:37.183 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 10 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2022-09-12T20:16:37.184 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 0 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj111]: ERROR: failed to sync object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj111 2022-09-12T20:16:37.184 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj115 2022-09-12T20:16:37.184 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj115]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj115 op=0 op_state=1 2022-09-12T20:16:37.185 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj115]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj115[0] 2022-09-12T20:16:37.185 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj116 2022-09-12T20:16:37.185 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj116]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj116 op=0 op_state=1 2022-09-12T20:16:37.185 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj116]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj116[0] 2022-09-12T20:16:37.186 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj124 2022-09-12T20:16:37.186 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj124]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj124 op=0 op_state=1 2022-09-12T20:16:37.186 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj124]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj124[0] 2022-09-12T20:16:37.187 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj145 2022-09-12T20:16:37.187 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj145]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj145 op=0 op_state=1 2022-09-12T20:16:37.187 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj145]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj145[0] 2022-09-12T20:16:37.187 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj148 2022-09-12T20:16:37.188 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj148]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj148 op=0 op_state=1 2022-09-12T20:16:37.188 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj148]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj148[0] 2022-09-12T20:16:37.188 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj155 2022-09-12T20:16:37.189 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj155]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj155 op=0 op_state=1 2022-09-12T20:16:37.189 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 0 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj155]: : injecting data sync error on key=obj155 2022-09-12T20:16:37.189 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 10 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2022-09-12T20:16:37.189 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 0 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj155]: ERROR: failed to sync object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj155 2022-09-12T20:16:37.190 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj17 2022-09-12T20:16:37.190 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj17]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj17 op=0 op_state=1 2022-09-12T20:16:37.190 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj17]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj17[0] 2022-09-12T20:16:37.191 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj185 2022-09-12T20:16:37.191 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj185]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj185 op=0 op_state=1 2022-09-12T20:16:37.191 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj185]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj185[0] 2022-09-12T20:16:37.192 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj19 2022-09-12T20:16:37.192 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj19]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj19 op=0 op_state=1 2022-09-12T20:16:37.192 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj19]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj19[0] 2022-09-12T20:16:37.193 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj22 2022-09-12T20:16:37.193 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj22]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj22 op=0 op_state=1 2022-09-12T20:16:37.193 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 0 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj22]: : injecting data sync error on key=obj22 2022-09-12T20:16:37.193 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 10 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj22]: failed, retcode=-5 ((5) Input/output error) 2022-09-12T20:16:37.194 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 0 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj22]: ERROR: failed to sync object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj22 2022-09-12T20:16:37.194 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj23 2022-09-12T20:16:37.194 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj23]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj23 op=0 op_state=1 2022-09-12T20:16:37.195 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj23]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj23[0] 2022-09-12T20:16:37.195 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj29 2022-09-12T20:16:37.195 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj29]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj29 op=0 op_state=1 2022-09-12T20:16:37.195 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 0 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj29]: : injecting data sync error on key=obj29 2022-09-12T20:16:37.196 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 10 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2022-09-12T20:16:37.196 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 0 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj29]: ERROR: failed to sync object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj29 2022-09-12T20:16:37.196 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj31 2022-09-12T20:16:37.197 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj31]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj31 op=0 op_state=1 2022-09-12T20:16:37.197 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj31]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj31[0] 2022-09-12T20:16:37.197 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj33 2022-09-12T20:16:37.197 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj33]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj33 op=0 op_state=1 2022-09-12T20:16:37.198 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj33]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj33[0] 2022-09-12T20:16:37.198 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj34 2022-09-12T20:16:37.198 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj34]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj34 op=0 op_state=1 2022-09-12T20:16:37.199 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 0 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj34]: : injecting data sync error on key=obj34 2022-09-12T20:16:37.199 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 10 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj34]: failed, retcode=-5 ((5) Input/output error) 2022-09-12T20:16:37.199 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 0 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj34]: ERROR: failed to sync object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj34 2022-09-12T20:16:37.199 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj37 2022-09-12T20:16:37.200 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj37]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj37 op=0 op_state=1 2022-09-12T20:16:37.200 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.177+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj37]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj37[0] 2022-09-12T20:16:37.200 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.178+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj38 2022-09-12T20:16:37.200 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.178+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj38]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj38 op=0 op_state=1 2022-09-12T20:16:37.201 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.178+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj38]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj38[0] 2022-09-12T20:16:37.201 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.178+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj44 2022-09-12T20:16:37.201 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.178+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj44]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj44 op=0 op_state=1 2022-09-12T20:16:37.202 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.178+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj44]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj44[0] 2022-09-12T20:16:37.202 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.178+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj46 2022-09-12T20:16:37.202 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.178+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj46]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj46 op=0 op_state=1 2022-09-12T20:16:37.202 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.178+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj46]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj46[0] 2022-09-12T20:16:37.203 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.180+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: ERROR: full sync failed. error: -5 2022-09-12T20:16:37.204 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.202+0000 7fec92bfb600 10 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj115]: success 2022-09-12T20:16:37.204 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.202+0000 7fec92bfb600 10 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj33]: success 2022-09-12T20:16:37.205 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.203+0000 7fec92bfb600 10 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj116]: success 2022-09-12T20:16:37.205 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.203+0000 7fec92bfb600 10 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj17]: success 2022-09-12T20:16:37.206 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.203+0000 7fec92bfb600 10 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj124]: success 2022-09-12T20:16:37.206 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.204+0000 7fec92bfb600 10 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj44]: success 2022-09-12T20:16:37.207 INFO:teuthology.orchestra.run.smithi138.stderr:*** Caught signal (Segmentation fault) ** 2022-09-12T20:16:37.207 INFO:teuthology.orchestra.run.smithi138.stderr: in thread 7fec92bfb600 thread_name:radosgw-admin 2022-09-12T20:16:37.220 INFO:teuthology.orchestra.run.smithi138.stderr: ceph version 17.0.0-14883-g10f7d25d (10f7d25d6a0f340676cd251eab5a787ba01ace95) quincy (dev) 2022-09-12T20:16:37.220 INFO:teuthology.orchestra.run.smithi138.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fec9a3c1cf0] 2022-09-12T20:16:37.221 INFO:teuthology.orchestra.run.smithi138.stderr: 2: (tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned int, int)+0x93) [0x7fec9b40f523] 2022-09-12T20:16:37.221 INFO:teuthology.orchestra.run.smithi138.stderr: 3: (tcmalloc::ThreadCache::Scavenge()+0x4e) [0x7fec9b40f86e] 2022-09-12T20:16:37.221 INFO:teuthology.orchestra.run.smithi138.stderr: 4: (RGWBucketInfo::~RGWBucketInfo()+0x2ba) [0x564bb6ed588a] 2022-09-12T20:16:37.221 INFO:teuthology.orchestra.run.smithi138.stderr: 5: (RGWAsyncFetchRemoteObj::~RGWAsyncFetchRemoteObj()+0xf9) [0x564bb7113919] 2022-09-12T20:16:37.222 INFO:teuthology.orchestra.run.smithi138.stderr: 6: (ceph::common::RefCountedObject::put() const+0x1ba) [0x7fec997ed46a] 2022-09-12T20:16:37.222 INFO:teuthology.orchestra.run.smithi138.stderr: 7: (RGWFetchRemoteObjCR::request_cleanup()+0x64) [0x564bb6fb8564] 2022-09-12T20:16:37.222 INFO:teuthology.orchestra.run.smithi138.stderr: 8: (RGWSimpleCoroutine::operate(DoutPrefixProvider const*)+0x17c) [0x564bb710070c] 2022-09-12T20:16:37.223 INFO:teuthology.orchestra.run.smithi138.stderr: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x18f) [0x564bb7100f7f] 2022-09-12T20:16:37.223 INFO:teuthology.orchestra.run.smithi138.stderr: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x521) [0x564bb7101751] 2022-09-12T20:16:37.223 INFO:teuthology.orchestra.run.smithi138.stderr: 11: (RGWBucketPipeSyncStatusManager::run(DoutPrefixProvider const*)+0xaca) [0x564bb6fb1baa] 2022-09-12T20:16:37.223 INFO:teuthology.orchestra.run.smithi138.stderr: 12: main() 2022-09-12T20:16:37.224 INFO:teuthology.orchestra.run.smithi138.stderr: 13: __libc_start_main() 2022-09-12T20:16:37.224 INFO:teuthology.orchestra.run.smithi138.stderr: 14: _start() 2022-09-12T20:16:37.224 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:16:37.207+0000 7fec92bfb600 -1 *** Caught signal (Segmentation fault) ** 2022-09-12T20:16:37.224 INFO:teuthology.orchestra.run.smithi138.stderr: in thread 7fec92bfb600 thread_name:radosgw-admin 2022-09-12T20:16:37.225 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-09-12T20:16:37.225 INFO:teuthology.orchestra.run.smithi138.stderr: ceph version 17.0.0-14883-g10f7d25d (10f7d25d6a0f340676cd251eab5a787ba01ace95) quincy (dev) 2022-09-12T20:16:37.225 INFO:teuthology.orchestra.run.smithi138.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fec9a3c1cf0] 2022-09-12T20:16:37.225 INFO:teuthology.orchestra.run.smithi138.stderr: 2: (tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned int, int)+0x93) [0x7fec9b40f523] 2022-09-12T20:16:37.226 INFO:teuthology.orchestra.run.smithi138.stderr: 3: (tcmalloc::ThreadCache::Scavenge()+0x4e) [0x7fec9b40f86e] 2022-09-12T20:16:37.226 INFO:teuthology.orchestra.run.smithi138.stderr: 4: (RGWBucketInfo::~RGWBucketInfo()+0x2ba) [0x564bb6ed588a] 2022-09-12T20:16:37.226 INFO:teuthology.orchestra.run.smithi138.stderr: 5: (RGWAsyncFetchRemoteObj::~RGWAsyncFetchRemoteObj()+0xf9) [0x564bb7113919] 2022-09-12T20:16:37.227 INFO:teuthology.orchestra.run.smithi138.stderr: 6: (ceph::common::RefCountedObject::put() const+0x1ba) [0x7fec997ed46a] 2022-09-12T20:16:37.227 INFO:teuthology.orchestra.run.smithi138.stderr: 7: (RGWFetchRemoteObjCR::request_cleanup()+0x64) [0x564bb6fb8564] 2022-09-12T20:16:37.227 INFO:teuthology.orchestra.run.smithi138.stderr: 8: (RGWSimpleCoroutine::operate(DoutPrefixProvider const*)+0x17c) [0x564bb710070c] 2022-09-12T20:16:37.228 INFO:teuthology.orchestra.run.smithi138.stderr: 9: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x18f) [0x564bb7100f7f] 2022-09-12T20:16:37.228 INFO:teuthology.orchestra.run.smithi138.stderr: 10: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x521) [0x564bb7101751] 2022-09-12T20:16:37.228 INFO:teuthology.orchestra.run.smithi138.stderr: 11: (RGWBucketPipeSyncStatusManager::run(DoutPrefixProvider const*)+0xaca) [0x564bb6fb1baa] 2022-09-12T20:16:37.229 INFO:teuthology.orchestra.run.smithi138.stderr: 12: main() 2022-09-12T20:16:37.229 INFO:teuthology.orchestra.run.smithi138.stderr: 13: __libc_start_main() 2022-09-12T20:16:37.229 INFO:teuthology.orchestra.run.smithi138.stderr: 14: _start() 2022-09-12T20:16:37.230 INFO:teuthology.orchestra.run.smithi138.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-09-12T20:16:37.230 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-09-12T20:16:37.230 INFO:teuthology.orchestra.run.smithi138.stderr:--- begin dump of recent events --- 2022-09-12T20:16:37.230 INFO:teuthology.orchestra.run.smithi138.stderr: -499> 2022-09-12T20:16:37.055+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6972bd0) adding auth protocol: cephx 2022-09-12T20:16:37.231 INFO:teuthology.orchestra.run.smithi138.stderr: -498> 2022-09-12T20:16:37.055+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6972bd0) adding auth protocol: cephx 2022-09-12T20:16:37.231 INFO:teuthology.orchestra.run.smithi138.stderr: -497> 2022-09-12T20:16:37.055+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6972bd0) adding auth protocol: cephx 2022-09-12T20:16:37.231 INFO:teuthology.orchestra.run.smithi138.stderr: -496> 2022-09-12T20:16:37.055+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6972bd0) adding con mode: secure 2022-09-12T20:16:37.232 INFO:teuthology.orchestra.run.smithi138.stderr: -495> 2022-09-12T20:16:37.055+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6972bd0) adding con mode: crc 2022-09-12T20:16:37.232 INFO:teuthology.orchestra.run.smithi138.stderr: -494> 2022-09-12T20:16:37.055+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6972bd0) adding con mode: secure 2022-09-12T20:16:37.232 INFO:teuthology.orchestra.run.smithi138.stderr: -493> 2022-09-12T20:16:37.055+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6972bd0) adding con mode: crc 2022-09-12T20:16:37.232 INFO:teuthology.orchestra.run.smithi138.stderr: -492> 2022-09-12T20:16:37.055+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6972bd0) adding con mode: secure 2022-09-12T20:16:37.233 INFO:teuthology.orchestra.run.smithi138.stderr: -491> 2022-09-12T20:16:37.055+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6972bd0) adding con mode: crc 2022-09-12T20:16:37.233 INFO:teuthology.orchestra.run.smithi138.stderr: -490> 2022-09-12T20:16:37.055+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6972bd0) adding con mode: crc 2022-09-12T20:16:37.233 INFO:teuthology.orchestra.run.smithi138.stderr: -489> 2022-09-12T20:16:37.055+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6972bd0) adding con mode: secure 2022-09-12T20:16:37.233 INFO:teuthology.orchestra.run.smithi138.stderr: -488> 2022-09-12T20:16:37.055+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6972bd0) adding con mode: crc 2022-09-12T20:16:37.234 INFO:teuthology.orchestra.run.smithi138.stderr: -487> 2022-09-12T20:16:37.055+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6972bd0) adding con mode: secure 2022-09-12T20:16:37.234 INFO:teuthology.orchestra.run.smithi138.stderr: -486> 2022-09-12T20:16:37.055+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6972bd0) adding con mode: crc 2022-09-12T20:16:37.234 INFO:teuthology.orchestra.run.smithi138.stderr: -485> 2022-09-12T20:16:37.055+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6972bd0) adding con mode: secure 2022-09-12T20:16:37.234 INFO:teuthology.orchestra.run.smithi138.stderr: -484> 2022-09-12T20:16:37.055+0000 7fec92bfb600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-12T20:16:37.235 INFO:teuthology.orchestra.run.smithi138.stderr: -483> 2022-09-12T20:16:37.055+0000 7fec92bfb600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-12T20:16:37.235 INFO:teuthology.orchestra.run.smithi138.stderr: -482> 2022-09-12T20:16:37.055+0000 7fec92bfb600 10 monclient: _reopen_session rank -1 2022-09-12T20:16:37.235 INFO:teuthology.orchestra.run.smithi138.stderr: -481> 2022-09-12T20:16:37.055+0000 7fec92bfb600 10 monclient: _add_conns ranks=[0,1] 2022-09-12T20:16:37.235 INFO:teuthology.orchestra.run.smithi138.stderr: -480> 2022-09-12T20:16:37.055+0000 7fec92bfb600 10 monclient(hunting): picked mon.noname-a-legacy con 0x564bb994e800 addr v1:172.21.15.138:6789/0 2022-09-12T20:16:37.236 INFO:teuthology.orchestra.run.smithi138.stderr: -479> 2022-09-12T20:16:37.055+0000 7fec92bfb600 10 monclient(hunting): picked mon.noname-a con 0x564bb994ec00 addr v2:172.21.15.138:3300/0 2022-09-12T20:16:37.236 INFO:teuthology.orchestra.run.smithi138.stderr: -478> 2022-09-12T20:16:37.055+0000 7fec92bfb600 10 monclient(hunting): start opening mon connection 2022-09-12T20:16:37.236 INFO:teuthology.orchestra.run.smithi138.stderr: -477> 2022-09-12T20:16:37.055+0000 7fec92bfb600 10 monclient(hunting): _renew_subs 2022-09-12T20:16:37.236 INFO:teuthology.orchestra.run.smithi138.stderr: -476> 2022-09-12T20:16:37.055+0000 7fec92bfb600 10 monclient(hunting): authenticate will time out at 6431.644288s 2022-09-12T20:16:37.237 INFO:teuthology.orchestra.run.smithi138.stderr: -475> 2022-09-12T20:16:37.055+0000 7fec903a6700 10 monclient(hunting): get_auth_request con 0x564bb994ec00 auth_method 0 2022-09-12T20:16:37.237 INFO:teuthology.orchestra.run.smithi138.stderr: -474> 2022-09-12T20:16:37.055+0000 7fec903a6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2022-09-12T20:16:37.237 INFO:teuthology.orchestra.run.smithi138.stderr: -473> 2022-09-12T20:16:37.055+0000 7fec903a6700 10 monclient(hunting): _init_auth method 2 2022-09-12T20:16:37.237 INFO:teuthology.orchestra.run.smithi138.stderr: -472> 2022-09-12T20:16:37.055+0000 7fec903a6700 10 monclient(hunting): _init_auth creating new auth 2022-09-12T20:16:37.238 INFO:teuthology.orchestra.run.smithi138.stderr: -471> 2022-09-12T20:16:37.055+0000 7fec903a6700 10 monclient(hunting): handle_auth_reply_more payload 9 2022-09-12T20:16:37.238 INFO:teuthology.orchestra.run.smithi138.stderr: -470> 2022-09-12T20:16:37.055+0000 7fec903a6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2022-09-12T20:16:37.238 INFO:teuthology.orchestra.run.smithi138.stderr: -469> 2022-09-12T20:16:37.055+0000 7fec903a6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2022-09-12T20:16:37.238 INFO:teuthology.orchestra.run.smithi138.stderr: -468> 2022-09-12T20:16:37.055+0000 7fec903a6700 10 monclient(hunting): handle_auth_done global_id 6726 payload 274 2022-09-12T20:16:37.239 INFO:teuthology.orchestra.run.smithi138.stderr: -467> 2022-09-12T20:16:37.055+0000 7fec903a6700 10 monclient: _finish_hunting 0 2022-09-12T20:16:37.239 INFO:teuthology.orchestra.run.smithi138.stderr: -466> 2022-09-12T20:16:37.055+0000 7fec903a6700 1 monclient: found mon.noname-a 2022-09-12T20:16:37.239 INFO:teuthology.orchestra.run.smithi138.stderr: -465> 2022-09-12T20:16:37.055+0000 7fec903a6700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2022-09-12T20:16:37.239 INFO:teuthology.orchestra.run.smithi138.stderr: -464> 2022-09-12T20:16:37.055+0000 7fec8fba5700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2022-09-12T20:16:37.240 INFO:teuthology.orchestra.run.smithi138.stderr: -463> 2022-09-12T20:16:37.055+0000 7fec8fba5700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-12T20:16:37.240 INFO:teuthology.orchestra.run.smithi138.stderr: -462> 2022-09-12T20:16:37.055+0000 7fec8fba5700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2022-09-12T20:16:37.240 INFO:teuthology.orchestra.run.smithi138.stderr: -461> 2022-09-12T20:16:37.055+0000 7fec8fba5700 10 monclient: dump: 2022-09-12T20:16:37.241 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2022-09-12T20:16:37.241 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 27700e6f-3189-467a-882c-63786db1882c 2022-09-12T20:16:37.241 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2022-09-12T18:45:24.531591+0000 2022-09-12T20:16:37.241 INFO:teuthology.orchestra.run.smithi138.stderr:created 2022-09-12T18:45:24.531591+0000 2022-09-12T20:16:37.242 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 17 (quincy) 2022-09-12T20:16:37.242 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2022-09-12T20:16:37.242 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2022-09-12T20:16:37.243 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-09-12T20:16:37.243 INFO:teuthology.orchestra.run.smithi138.stderr: -460> 2022-09-12T20:16:37.055+0000 7fec8fba5700 10 monclient: _finish_auth 0 2022-09-12T20:16:37.243 INFO:teuthology.orchestra.run.smithi138.stderr: -459> 2022-09-12T20:16:37.055+0000 7fec8fba5700 10 monclient: _check_auth_tickets 2022-09-12T20:16:37.244 INFO:teuthology.orchestra.run.smithi138.stderr: -458> 2022-09-12T20:16:37.055+0000 7fec92bfb600 5 monclient: authenticate success, global_id 6726 2022-09-12T20:16:37.244 INFO:teuthology.orchestra.run.smithi138.stderr: -457> 2022-09-12T20:16:37.055+0000 7fec8fba5700 10 monclient: handle_config config(0 keys) v1 2022-09-12T20:16:37.244 INFO:teuthology.orchestra.run.smithi138.stderr: -456> 2022-09-12T20:16:37.055+0000 7fec8fba5700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-12T20:16:37.245 INFO:teuthology.orchestra.run.smithi138.stderr: -455> 2022-09-12T20:16:37.055+0000 7fec8fba5700 10 monclient: got monmap 1 from mon.a (according to old e1) 2022-09-12T20:16:37.245 INFO:teuthology.orchestra.run.smithi138.stderr: -454> 2022-09-12T20:16:37.055+0000 7fec8fba5700 10 monclient: dump: 2022-09-12T20:16:37.245 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2022-09-12T20:16:37.245 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 27700e6f-3189-467a-882c-63786db1882c 2022-09-12T20:16:37.246 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2022-09-12T18:45:24.531591+0000 2022-09-12T20:16:37.246 INFO:teuthology.orchestra.run.smithi138.stderr:created 2022-09-12T18:45:24.531591+0000 2022-09-12T20:16:37.246 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 17 (quincy) 2022-09-12T20:16:37.247 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2022-09-12T20:16:37.247 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2022-09-12T20:16:37.247 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-09-12T20:16:37.247 INFO:teuthology.orchestra.run.smithi138.stderr: -453> 2022-09-12T20:16:37.055+0000 7fec92bfb600 10 monclient: get_monmap_and_config success 2022-09-12T20:16:37.248 INFO:teuthology.orchestra.run.smithi138.stderr: -452> 2022-09-12T20:16:37.055+0000 7fec92bfb600 4 set_mon_vals no callback set 2022-09-12T20:16:37.248 INFO:teuthology.orchestra.run.smithi138.stderr: -451> 2022-09-12T20:16:37.056+0000 7fec92bfb600 10 monclient: shutdown 2022-09-12T20:16:37.248 INFO:teuthology.orchestra.run.smithi138.stderr: -450> 2022-09-12T20:16:37.056+0000 7fec92bfb600 5 asok(0x564bb99e2000) init /var/run/ceph/c2-client.0.151814.asok 2022-09-12T20:16:37.249 INFO:teuthology.orchestra.run.smithi138.stderr: -449> 2022-09-12T20:16:37.056+0000 7fec92bfb600 5 asok(0x564bb99e2000) bind_and_listen /var/run/ceph/c2-client.0.151814.asok 2022-09-12T20:16:37.249 INFO:teuthology.orchestra.run.smithi138.stderr: -448> 2022-09-12T20:16:37.056+0000 7fec92bfb600 5 asok(0x564bb99e2000) register_command 0 hook 0x564bb999f358 2022-09-12T20:16:37.249 INFO:teuthology.orchestra.run.smithi138.stderr: -447> 2022-09-12T20:16:37.056+0000 7fec92bfb600 5 asok(0x564bb99e2000) register_command version hook 0x564bb999f358 2022-09-12T20:16:37.249 INFO:teuthology.orchestra.run.smithi138.stderr: -446> 2022-09-12T20:16:37.056+0000 7fec92bfb600 5 asok(0x564bb99e2000) register_command git_version hook 0x564bb999f358 2022-09-12T20:16:37.250 INFO:teuthology.orchestra.run.smithi138.stderr: -445> 2022-09-12T20:16:37.056+0000 7fec92bfb600 5 asok(0x564bb99e2000) register_command help hook 0x564bb9942bf0 2022-09-12T20:16:37.250 INFO:teuthology.orchestra.run.smithi138.stderr: -444> 2022-09-12T20:16:37.056+0000 7fec92bfb600 5 asok(0x564bb99e2000) register_command get_command_descriptions hook 0x564bb9942b50 2022-09-12T20:16:37.250 INFO:teuthology.orchestra.run.smithi138.stderr: -443> 2022-09-12T20:16:37.056+0000 7fec8fba5700 5 asok(0x564bb99e2000) entry start 2022-09-12T20:16:37.250 INFO:teuthology.orchestra.run.smithi138.stderr: -442> 2022-09-12T20:16:37.056+0000 7fec92bfb600 10 monclient: get_monmap_and_config 2022-09-12T20:16:37.251 INFO:teuthology.orchestra.run.smithi138.stderr: -441> 2022-09-12T20:16:37.056+0000 7fec92bfb600 10 monclient: build_initial_monmap 2022-09-12T20:16:37.251 INFO:teuthology.orchestra.run.smithi138.stderr: -440> 2022-09-12T20:16:37.056+0000 7fec92bfb600 1 build_initial for_mkfs: 0 2022-09-12T20:16:37.251 INFO:teuthology.orchestra.run.smithi138.stderr: -439> 2022-09-12T20:16:37.056+0000 7fec92bfb600 10 monclient: monmap: 2022-09-12T20:16:37.251 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2022-09-12T20:16:37.252 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2022-09-12T20:16:37.252 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2022-09-12T20:16:37.252 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2022-09-12T20:16:37.252 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2022-09-12T20:16:37.253 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2022-09-12T20:16:37.253 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2022-09-12T20:16:37.253 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-09-12T20:16:37.253 INFO:teuthology.orchestra.run.smithi138.stderr: -438> 2022-09-12T20:16:37.056+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding auth protocol: cephx 2022-09-12T20:16:37.254 INFO:teuthology.orchestra.run.smithi138.stderr: -437> 2022-09-12T20:16:37.056+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding auth protocol: cephx 2022-09-12T20:16:37.254 INFO:teuthology.orchestra.run.smithi138.stderr: -436> 2022-09-12T20:16:37.056+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding auth protocol: cephx 2022-09-12T20:16:37.254 INFO:teuthology.orchestra.run.smithi138.stderr: -435> 2022-09-12T20:16:37.056+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding con mode: secure 2022-09-12T20:16:37.255 INFO:teuthology.orchestra.run.smithi138.stderr: -434> 2022-09-12T20:16:37.056+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding con mode: crc 2022-09-12T20:16:37.255 INFO:teuthology.orchestra.run.smithi138.stderr: -433> 2022-09-12T20:16:37.056+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding con mode: secure 2022-09-12T20:16:37.255 INFO:teuthology.orchestra.run.smithi138.stderr: -432> 2022-09-12T20:16:37.056+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding con mode: crc 2022-09-12T20:16:37.255 INFO:teuthology.orchestra.run.smithi138.stderr: -431> 2022-09-12T20:16:37.056+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding con mode: secure 2022-09-12T20:16:37.256 INFO:teuthology.orchestra.run.smithi138.stderr: -430> 2022-09-12T20:16:37.056+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding con mode: crc 2022-09-12T20:16:37.256 INFO:teuthology.orchestra.run.smithi138.stderr: -429> 2022-09-12T20:16:37.057+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding con mode: crc 2022-09-12T20:16:37.256 INFO:teuthology.orchestra.run.smithi138.stderr: -428> 2022-09-12T20:16:37.057+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding con mode: secure 2022-09-12T20:16:37.256 INFO:teuthology.orchestra.run.smithi138.stderr: -427> 2022-09-12T20:16:37.057+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding con mode: crc 2022-09-12T20:16:37.257 INFO:teuthology.orchestra.run.smithi138.stderr: -426> 2022-09-12T20:16:37.057+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding con mode: secure 2022-09-12T20:16:37.257 INFO:teuthology.orchestra.run.smithi138.stderr: -425> 2022-09-12T20:16:37.057+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding con mode: crc 2022-09-12T20:16:37.257 INFO:teuthology.orchestra.run.smithi138.stderr: -424> 2022-09-12T20:16:37.057+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding con mode: secure 2022-09-12T20:16:37.257 INFO:teuthology.orchestra.run.smithi138.stderr: -423> 2022-09-12T20:16:37.057+0000 7fec92bfb600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-12T20:16:37.258 INFO:teuthology.orchestra.run.smithi138.stderr: -422> 2022-09-12T20:16:37.057+0000 7fec92bfb600 10 monclient: init 2022-09-12T20:16:37.258 INFO:teuthology.orchestra.run.smithi138.stderr: -421> 2022-09-12T20:16:37.057+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973510) adding auth protocol: cephx 2022-09-12T20:16:37.258 INFO:teuthology.orchestra.run.smithi138.stderr: -420> 2022-09-12T20:16:37.057+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973510) adding auth protocol: cephx 2022-09-12T20:16:37.259 INFO:teuthology.orchestra.run.smithi138.stderr: -419> 2022-09-12T20:16:37.057+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973510) adding auth protocol: cephx 2022-09-12T20:16:37.259 INFO:teuthology.orchestra.run.smithi138.stderr: -418> 2022-09-12T20:16:37.057+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973510) adding con mode: secure 2022-09-12T20:16:37.259 INFO:teuthology.orchestra.run.smithi138.stderr: -417> 2022-09-12T20:16:37.057+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973510) adding con mode: crc 2022-09-12T20:16:37.260 INFO:teuthology.orchestra.run.smithi138.stderr: -416> 2022-09-12T20:16:37.057+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973510) adding con mode: secure 2022-09-12T20:16:37.260 INFO:teuthology.orchestra.run.smithi138.stderr: -415> 2022-09-12T20:16:37.057+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973510) adding con mode: crc 2022-09-12T20:16:37.260 INFO:teuthology.orchestra.run.smithi138.stderr: -414> 2022-09-12T20:16:37.057+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973510) adding con mode: secure 2022-09-12T20:16:37.260 INFO:teuthology.orchestra.run.smithi138.stderr: -413> 2022-09-12T20:16:37.057+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973510) adding con mode: crc 2022-09-12T20:16:37.261 INFO:teuthology.orchestra.run.smithi138.stderr: -412> 2022-09-12T20:16:37.057+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973510) adding con mode: crc 2022-09-12T20:16:37.261 INFO:teuthology.orchestra.run.smithi138.stderr: -411> 2022-09-12T20:16:37.057+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973510) adding con mode: secure 2022-09-12T20:16:37.261 INFO:teuthology.orchestra.run.smithi138.stderr: -410> 2022-09-12T20:16:37.057+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973510) adding con mode: crc 2022-09-12T20:16:37.261 INFO:teuthology.orchestra.run.smithi138.stderr: -409> 2022-09-12T20:16:37.057+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973510) adding con mode: secure 2022-09-12T20:16:37.262 INFO:teuthology.orchestra.run.smithi138.stderr: -408> 2022-09-12T20:16:37.057+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973510) adding con mode: crc 2022-09-12T20:16:37.262 INFO:teuthology.orchestra.run.smithi138.stderr: -407> 2022-09-12T20:16:37.057+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973510) adding con mode: secure 2022-09-12T20:16:37.262 INFO:teuthology.orchestra.run.smithi138.stderr: -406> 2022-09-12T20:16:37.057+0000 7fec92bfb600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-12T20:16:37.262 INFO:teuthology.orchestra.run.smithi138.stderr: -405> 2022-09-12T20:16:37.057+0000 7fec92bfb600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-12T20:16:37.263 INFO:teuthology.orchestra.run.smithi138.stderr: -404> 2022-09-12T20:16:37.057+0000 7fec92bfb600 10 monclient: _reopen_session rank -1 2022-09-12T20:16:37.263 INFO:teuthology.orchestra.run.smithi138.stderr: -403> 2022-09-12T20:16:37.057+0000 7fec92bfb600 10 monclient: _add_conns ranks=[0] 2022-09-12T20:16:37.263 INFO:teuthology.orchestra.run.smithi138.stderr: -402> 2022-09-12T20:16:37.057+0000 7fec92bfb600 10 monclient(hunting): picked mon.noname-a con 0x564bb994e800 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2022-09-12T20:16:37.263 INFO:teuthology.orchestra.run.smithi138.stderr: -401> 2022-09-12T20:16:37.057+0000 7fec92bfb600 10 monclient(hunting): start opening mon connection 2022-09-12T20:16:37.264 INFO:teuthology.orchestra.run.smithi138.stderr: -400> 2022-09-12T20:16:37.057+0000 7fec92bfb600 10 monclient(hunting): _renew_subs 2022-09-12T20:16:37.264 INFO:teuthology.orchestra.run.smithi138.stderr: -399> 2022-09-12T20:16:37.057+0000 7fec92bfb600 10 monclient(hunting): authenticate will time out at 6431.646229s 2022-09-12T20:16:37.264 INFO:teuthology.orchestra.run.smithi138.stderr: -398> 2022-09-12T20:16:37.057+0000 7fec90ba7700 10 monclient(hunting): get_auth_request con 0x564bb994e800 auth_method 0 2022-09-12T20:16:37.264 INFO:teuthology.orchestra.run.smithi138.stderr: -397> 2022-09-12T20:16:37.057+0000 7fec90ba7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2022-09-12T20:16:37.265 INFO:teuthology.orchestra.run.smithi138.stderr: -396> 2022-09-12T20:16:37.057+0000 7fec90ba7700 10 monclient(hunting): _init_auth method 2 2022-09-12T20:16:37.265 INFO:teuthology.orchestra.run.smithi138.stderr: -395> 2022-09-12T20:16:37.057+0000 7fec90ba7700 10 monclient(hunting): _init_auth creating new auth 2022-09-12T20:16:37.265 INFO:teuthology.orchestra.run.smithi138.stderr: -394> 2022-09-12T20:16:37.057+0000 7fec90ba7700 10 monclient(hunting): handle_auth_reply_more payload 9 2022-09-12T20:16:37.266 INFO:teuthology.orchestra.run.smithi138.stderr: -393> 2022-09-12T20:16:37.057+0000 7fec90ba7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2022-09-12T20:16:37.266 INFO:teuthology.orchestra.run.smithi138.stderr: -392> 2022-09-12T20:16:37.057+0000 7fec90ba7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2022-09-12T20:16:37.266 INFO:teuthology.orchestra.run.smithi138.stderr: -391> 2022-09-12T20:16:37.057+0000 7fec90ba7700 10 monclient(hunting): handle_auth_done global_id 6728 payload 274 2022-09-12T20:16:37.266 INFO:teuthology.orchestra.run.smithi138.stderr: -390> 2022-09-12T20:16:37.057+0000 7fec90ba7700 10 monclient: _finish_hunting 0 2022-09-12T20:16:37.267 INFO:teuthology.orchestra.run.smithi138.stderr: -389> 2022-09-12T20:16:37.057+0000 7fec90ba7700 1 monclient: found mon.noname-a 2022-09-12T20:16:37.267 INFO:teuthology.orchestra.run.smithi138.stderr: -388> 2022-09-12T20:16:37.057+0000 7fec90ba7700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2022-09-12T20:16:37.267 INFO:teuthology.orchestra.run.smithi138.stderr: -387> 2022-09-12T20:16:37.057+0000 7fec8eba3700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-12T20:16:37.268 INFO:teuthology.orchestra.run.smithi138.stderr: -386> 2022-09-12T20:16:37.057+0000 7fec8eba3700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2022-09-12T20:16:37.268 INFO:teuthology.orchestra.run.smithi138.stderr: -385> 2022-09-12T20:16:37.057+0000 7fec8eba3700 10 monclient: dump: 2022-09-12T20:16:37.268 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2022-09-12T20:16:37.269 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 27700e6f-3189-467a-882c-63786db1882c 2022-09-12T20:16:37.269 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2022-09-12T18:45:24.531591+0000 2022-09-12T20:16:37.269 INFO:teuthology.orchestra.run.smithi138.stderr:created 2022-09-12T18:45:24.531591+0000 2022-09-12T20:16:37.269 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 17 (quincy) 2022-09-12T20:16:37.270 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2022-09-12T20:16:37.270 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2022-09-12T20:16:37.270 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-09-12T20:16:37.270 INFO:teuthology.orchestra.run.smithi138.stderr: -384> 2022-09-12T20:16:37.057+0000 7fec8eba3700 10 monclient: _finish_auth 0 2022-09-12T20:16:37.271 INFO:teuthology.orchestra.run.smithi138.stderr: -383> 2022-09-12T20:16:37.057+0000 7fec8eba3700 10 monclient: _check_auth_tickets 2022-09-12T20:16:37.271 INFO:teuthology.orchestra.run.smithi138.stderr: -382> 2022-09-12T20:16:37.057+0000 7fec8eba3700 10 monclient: handle_config config(0 keys) v1 2022-09-12T20:16:37.271 INFO:teuthology.orchestra.run.smithi138.stderr: -381> 2022-09-12T20:16:37.057+0000 7fec8eba3700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-12T20:16:37.272 INFO:teuthology.orchestra.run.smithi138.stderr: -380> 2022-09-12T20:16:37.057+0000 7fec8eba3700 10 monclient: got monmap 1 from mon.a (according to old e1) 2022-09-12T20:16:37.272 INFO:teuthology.orchestra.run.smithi138.stderr: -379> 2022-09-12T20:16:37.057+0000 7fec8eba3700 10 monclient: dump: 2022-09-12T20:16:37.272 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2022-09-12T20:16:37.272 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 27700e6f-3189-467a-882c-63786db1882c 2022-09-12T20:16:37.273 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2022-09-12T18:45:24.531591+0000 2022-09-12T20:16:37.273 INFO:teuthology.orchestra.run.smithi138.stderr:created 2022-09-12T18:45:24.531591+0000 2022-09-12T20:16:37.273 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 17 (quincy) 2022-09-12T20:16:37.273 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2022-09-12T20:16:37.274 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2022-09-12T20:16:37.274 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-09-12T20:16:37.274 INFO:teuthology.orchestra.run.smithi138.stderr: -378> 2022-09-12T20:16:37.057+0000 7fec92bfb600 5 monclient: authenticate success, global_id 6728 2022-09-12T20:16:37.274 INFO:teuthology.orchestra.run.smithi138.stderr: -377> 2022-09-12T20:16:37.057+0000 7fec92bfb600 10 monclient: get_monmap_and_config success 2022-09-12T20:16:37.275 INFO:teuthology.orchestra.run.smithi138.stderr: -376> 2022-09-12T20:16:37.057+0000 7fec92bfb600 4 set_mon_vals no callback set 2022-09-12T20:16:37.275 INFO:teuthology.orchestra.run.smithi138.stderr: -375> 2022-09-12T20:16:37.058+0000 7fec92bfb600 10 monclient: shutdown 2022-09-12T20:16:37.275 INFO:teuthology.orchestra.run.smithi138.stderr: -374> 2022-09-12T20:16:37.058+0000 7fec92bfb600 10 monclient: build_initial_monmap 2022-09-12T20:16:37.276 INFO:teuthology.orchestra.run.smithi138.stderr: -373> 2022-09-12T20:16:37.058+0000 7fec92bfb600 1 build_initial for_mkfs: 0 2022-09-12T20:16:37.276 INFO:teuthology.orchestra.run.smithi138.stderr: -372> 2022-09-12T20:16:37.058+0000 7fec92bfb600 10 monclient: monmap: 2022-09-12T20:16:37.276 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2022-09-12T20:16:37.277 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2022-09-12T20:16:37.277 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2022-09-12T20:16:37.277 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2022-09-12T20:16:37.278 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2022-09-12T20:16:37.278 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2022-09-12T20:16:37.278 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2022-09-12T20:16:37.278 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-09-12T20:16:37.279 INFO:teuthology.orchestra.run.smithi138.stderr: -371> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding auth protocol: cephx 2022-09-12T20:16:37.279 INFO:teuthology.orchestra.run.smithi138.stderr: -370> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding auth protocol: cephx 2022-09-12T20:16:37.279 INFO:teuthology.orchestra.run.smithi138.stderr: -369> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding auth protocol: cephx 2022-09-12T20:16:37.279 INFO:teuthology.orchestra.run.smithi138.stderr: -368> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding con mode: secure 2022-09-12T20:16:37.280 INFO:teuthology.orchestra.run.smithi138.stderr: -367> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding con mode: crc 2022-09-12T20:16:37.280 INFO:teuthology.orchestra.run.smithi138.stderr: -366> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding con mode: secure 2022-09-12T20:16:37.280 INFO:teuthology.orchestra.run.smithi138.stderr: -365> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding con mode: crc 2022-09-12T20:16:37.280 INFO:teuthology.orchestra.run.smithi138.stderr: -364> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding con mode: secure 2022-09-12T20:16:37.281 INFO:teuthology.orchestra.run.smithi138.stderr: -363> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding con mode: crc 2022-09-12T20:16:37.281 INFO:teuthology.orchestra.run.smithi138.stderr: -362> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding con mode: crc 2022-09-12T20:16:37.281 INFO:teuthology.orchestra.run.smithi138.stderr: -361> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding con mode: secure 2022-09-12T20:16:37.281 INFO:teuthology.orchestra.run.smithi138.stderr: -360> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding con mode: crc 2022-09-12T20:16:37.282 INFO:teuthology.orchestra.run.smithi138.stderr: -359> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding con mode: secure 2022-09-12T20:16:37.282 INFO:teuthology.orchestra.run.smithi138.stderr: -358> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding con mode: crc 2022-09-12T20:16:37.282 INFO:teuthology.orchestra.run.smithi138.stderr: -357> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bba6aea40) adding con mode: secure 2022-09-12T20:16:37.283 INFO:teuthology.orchestra.run.smithi138.stderr: -356> 2022-09-12T20:16:37.058+0000 7fec92bfb600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-12T20:16:37.283 INFO:teuthology.orchestra.run.smithi138.stderr: -355> 2022-09-12T20:16:37.058+0000 7fec92bfb600 1 librados: starting msgr at 2022-09-12T20:16:37.283 INFO:teuthology.orchestra.run.smithi138.stderr: -354> 2022-09-12T20:16:37.058+0000 7fec92bfb600 1 librados: starting objecter 2022-09-12T20:16:37.283 INFO:teuthology.orchestra.run.smithi138.stderr: -353> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 asok(0x564bb99e2000) register_command objecter_requests hook 0x564bb9942b90 2022-09-12T20:16:37.284 INFO:teuthology.orchestra.run.smithi138.stderr: -352> 2022-09-12T20:16:37.058+0000 7fec92bfb600 1 librados: setting wanted keys 2022-09-12T20:16:37.284 INFO:teuthology.orchestra.run.smithi138.stderr: -351> 2022-09-12T20:16:37.058+0000 7fec92bfb600 1 librados: calling monclient init 2022-09-12T20:16:37.284 INFO:teuthology.orchestra.run.smithi138.stderr: -350> 2022-09-12T20:16:37.058+0000 7fec92bfb600 10 monclient: init 2022-09-12T20:16:37.284 INFO:teuthology.orchestra.run.smithi138.stderr: -349> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bb99aa0d8) adding auth protocol: cephx 2022-09-12T20:16:37.285 INFO:teuthology.orchestra.run.smithi138.stderr: -348> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bb99aa0d8) adding auth protocol: cephx 2022-09-12T20:16:37.285 INFO:teuthology.orchestra.run.smithi138.stderr: -347> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bb99aa0d8) adding auth protocol: cephx 2022-09-12T20:16:37.285 INFO:teuthology.orchestra.run.smithi138.stderr: -346> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bb99aa0d8) adding con mode: secure 2022-09-12T20:16:37.286 INFO:teuthology.orchestra.run.smithi138.stderr: -345> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bb99aa0d8) adding con mode: crc 2022-09-12T20:16:37.286 INFO:teuthology.orchestra.run.smithi138.stderr: -344> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bb99aa0d8) adding con mode: secure 2022-09-12T20:16:37.286 INFO:teuthology.orchestra.run.smithi138.stderr: -343> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bb99aa0d8) adding con mode: crc 2022-09-12T20:16:37.287 INFO:teuthology.orchestra.run.smithi138.stderr: -342> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bb99aa0d8) adding con mode: secure 2022-09-12T20:16:37.287 INFO:teuthology.orchestra.run.smithi138.stderr: -341> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bb99aa0d8) adding con mode: crc 2022-09-12T20:16:37.287 INFO:teuthology.orchestra.run.smithi138.stderr: -340> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bb99aa0d8) adding con mode: crc 2022-09-12T20:16:37.288 INFO:teuthology.orchestra.run.smithi138.stderr: -339> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bb99aa0d8) adding con mode: secure 2022-09-12T20:16:37.288 INFO:teuthology.orchestra.run.smithi138.stderr: -338> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bb99aa0d8) adding con mode: crc 2022-09-12T20:16:37.288 INFO:teuthology.orchestra.run.smithi138.stderr: -337> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bb99aa0d8) adding con mode: secure 2022-09-12T20:16:37.288 INFO:teuthology.orchestra.run.smithi138.stderr: -336> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bb99aa0d8) adding con mode: crc 2022-09-12T20:16:37.289 INFO:teuthology.orchestra.run.smithi138.stderr: -335> 2022-09-12T20:16:37.058+0000 7fec92bfb600 5 AuthRegistry(0x564bb99aa0d8) adding con mode: secure 2022-09-12T20:16:37.289 INFO:teuthology.orchestra.run.smithi138.stderr: -334> 2022-09-12T20:16:37.058+0000 7fec92bfb600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-12T20:16:37.289 INFO:teuthology.orchestra.run.smithi138.stderr: -333> 2022-09-12T20:16:37.058+0000 7fec92bfb600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-12T20:16:37.289 INFO:teuthology.orchestra.run.smithi138.stderr: -332> 2022-09-12T20:16:37.058+0000 7fec92bfb600 10 monclient: _reopen_session rank -1 2022-09-12T20:16:37.290 INFO:teuthology.orchestra.run.smithi138.stderr: -331> 2022-09-12T20:16:37.058+0000 7fec92bfb600 10 monclient: _add_conns ranks=[0] 2022-09-12T20:16:37.290 INFO:teuthology.orchestra.run.smithi138.stderr: -330> 2022-09-12T20:16:37.058+0000 7fec92bfb600 10 monclient(hunting): picked mon.noname-a con 0x564bb994e800 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2022-09-12T20:16:37.290 INFO:teuthology.orchestra.run.smithi138.stderr: -329> 2022-09-12T20:16:37.058+0000 7fec92bfb600 10 monclient(hunting): start opening mon connection 2022-09-12T20:16:37.290 INFO:teuthology.orchestra.run.smithi138.stderr: -328> 2022-09-12T20:16:37.058+0000 7fec92bfb600 10 monclient(hunting): _renew_subs 2022-09-12T20:16:37.291 INFO:teuthology.orchestra.run.smithi138.stderr: -327> 2022-09-12T20:16:37.058+0000 7fec92bfb600 10 monclient(hunting): authenticate will time out at 6431.647662s 2022-09-12T20:16:37.291 INFO:teuthology.orchestra.run.smithi138.stderr: -326> 2022-09-12T20:16:37.058+0000 7fec90ba7700 10 monclient(hunting): get_auth_request con 0x564bb994e800 auth_method 0 2022-09-12T20:16:37.291 INFO:teuthology.orchestra.run.smithi138.stderr: -325> 2022-09-12T20:16:37.058+0000 7fec90ba7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2022-09-12T20:16:37.291 INFO:teuthology.orchestra.run.smithi138.stderr: -324> 2022-09-12T20:16:37.058+0000 7fec90ba7700 10 monclient(hunting): _init_auth method 2 2022-09-12T20:16:37.292 INFO:teuthology.orchestra.run.smithi138.stderr: -323> 2022-09-12T20:16:37.058+0000 7fec90ba7700 10 monclient(hunting): _init_auth creating new auth 2022-09-12T20:16:37.292 INFO:teuthology.orchestra.run.smithi138.stderr: -322> 2022-09-12T20:16:37.058+0000 7fec90ba7700 10 monclient(hunting): handle_auth_reply_more payload 9 2022-09-12T20:16:37.292 INFO:teuthology.orchestra.run.smithi138.stderr: -321> 2022-09-12T20:16:37.058+0000 7fec90ba7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2022-09-12T20:16:37.292 INFO:teuthology.orchestra.run.smithi138.stderr: -320> 2022-09-12T20:16:37.058+0000 7fec90ba7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2022-09-12T20:16:37.293 INFO:teuthology.orchestra.run.smithi138.stderr: -319> 2022-09-12T20:16:37.058+0000 7fec90ba7700 10 monclient(hunting): handle_auth_done global_id 6729 payload 824 2022-09-12T20:16:37.293 INFO:teuthology.orchestra.run.smithi138.stderr: -318> 2022-09-12T20:16:37.058+0000 7fec90ba7700 10 monclient: _finish_hunting 0 2022-09-12T20:16:37.293 INFO:teuthology.orchestra.run.smithi138.stderr: -317> 2022-09-12T20:16:37.058+0000 7fec90ba7700 1 monclient: found mon.noname-a 2022-09-12T20:16:37.293 INFO:teuthology.orchestra.run.smithi138.stderr: -316> 2022-09-12T20:16:37.058+0000 7fec90ba7700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2022-09-12T20:16:37.294 INFO:teuthology.orchestra.run.smithi138.stderr: -315> 2022-09-12T20:16:37.059+0000 7fec8d3a0700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-12T20:16:37.294 INFO:teuthology.orchestra.run.smithi138.stderr: -314> 2022-09-12T20:16:37.059+0000 7fec8d3a0700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2022-09-12T20:16:37.294 INFO:teuthology.orchestra.run.smithi138.stderr: -313> 2022-09-12T20:16:37.059+0000 7fec8d3a0700 10 monclient: dump: 2022-09-12T20:16:37.295 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2022-09-12T20:16:37.295 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 27700e6f-3189-467a-882c-63786db1882c 2022-09-12T20:16:37.295 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2022-09-12T18:45:24.531591+0000 2022-09-12T20:16:37.295 INFO:teuthology.orchestra.run.smithi138.stderr:created 2022-09-12T18:45:24.531591+0000 2022-09-12T20:16:37.296 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 17 (quincy) 2022-09-12T20:16:37.296 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2022-09-12T20:16:37.296 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2022-09-12T20:16:37.296 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-09-12T20:16:37.297 INFO:teuthology.orchestra.run.smithi138.stderr: -312> 2022-09-12T20:16:37.059+0000 7fec8d3a0700 10 monclient: _finish_auth 0 2022-09-12T20:16:37.297 INFO:teuthology.orchestra.run.smithi138.stderr: -311> 2022-09-12T20:16:37.059+0000 7fec8d3a0700 10 monclient: _check_auth_tickets 2022-09-12T20:16:37.297 INFO:teuthology.orchestra.run.smithi138.stderr: -310> 2022-09-12T20:16:37.059+0000 7fec92bfb600 5 monclient: authenticate success, global_id 6729 2022-09-12T20:16:37.297 INFO:teuthology.orchestra.run.smithi138.stderr: -309> 2022-09-12T20:16:37.059+0000 7fec92bfb600 10 monclient: _renew_subs 2022-09-12T20:16:37.298 INFO:teuthology.orchestra.run.smithi138.stderr: -308> 2022-09-12T20:16:37.059+0000 7fec92bfb600 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2022-09-12T20:16:37.298 INFO:teuthology.orchestra.run.smithi138.stderr: -307> 2022-09-12T20:16:37.059+0000 7fec8d3a0700 10 monclient: handle_config config(0 keys) v1 2022-09-12T20:16:37.298 INFO:teuthology.orchestra.run.smithi138.stderr: -306> 2022-09-12T20:16:37.059+0000 7fec8d3a0700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-12T20:16:37.298 INFO:teuthology.orchestra.run.smithi138.stderr: -305> 2022-09-12T20:16:37.059+0000 7fec8d3a0700 10 monclient: got monmap 1 from mon.a (according to old e1) 2022-09-12T20:16:37.299 INFO:teuthology.orchestra.run.smithi138.stderr: -304> 2022-09-12T20:16:37.059+0000 7fec8eba3700 4 set_mon_vals no callback set 2022-09-12T20:16:37.299 INFO:teuthology.orchestra.run.smithi138.stderr: -303> 2022-09-12T20:16:37.059+0000 7fec8d3a0700 10 monclient: dump: 2022-09-12T20:16:37.299 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2022-09-12T20:16:37.299 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 27700e6f-3189-467a-882c-63786db1882c 2022-09-12T20:16:37.300 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2022-09-12T18:45:24.531591+0000 2022-09-12T20:16:37.300 INFO:teuthology.orchestra.run.smithi138.stderr:created 2022-09-12T18:45:24.531591+0000 2022-09-12T20:16:37.300 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 17 (quincy) 2022-09-12T20:16:37.300 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2022-09-12T20:16:37.301 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2022-09-12T20:16:37.301 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-09-12T20:16:37.301 INFO:teuthology.orchestra.run.smithi138.stderr: -302> 2022-09-12T20:16:37.059+0000 7fec92bfb600 10 monclient: _renew_subs 2022-09-12T20:16:37.302 INFO:teuthology.orchestra.run.smithi138.stderr: -301> 2022-09-12T20:16:37.059+0000 7fec92bfb600 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2022-09-12T20:16:37.302 INFO:teuthology.orchestra.run.smithi138.stderr: -300> 2022-09-12T20:16:37.059+0000 7fec92bfb600 1 librados: init done 2022-09-12T20:16:37.302 INFO:teuthology.orchestra.run.smithi138.stderr: -299> 2022-09-12T20:16:37.060+0000 7fec8d3a0700 4 mgrc handle_mgr_map Got map version 5 2022-09-12T20:16:37.302 INFO:teuthology.orchestra.run.smithi138.stderr: -298> 2022-09-12T20:16:37.060+0000 7fec8d3a0700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.138:6824/101383,v1:172.21.15.138:6825/101383] 2022-09-12T20:16:37.303 INFO:teuthology.orchestra.run.smithi138.stderr: -297> 2022-09-12T20:16:37.060+0000 7fec8d3a0700 4 mgrc reconnect Starting new session with [v2:172.21.15.138:6824/101383,v1:172.21.15.138:6825/101383] 2022-09-12T20:16:37.303 INFO:teuthology.orchestra.run.smithi138.stderr: -296> 2022-09-12T20:16:37.060+0000 7fec903a6700 10 monclient: get_auth_request con 0x564bb994f400 auth_method 0 2022-09-12T20:16:37.303 INFO:teuthology.orchestra.run.smithi138.stderr: -295> 2022-09-12T20:16:37.060+0000 7fec913a8700 10 monclient: get_auth_request con 0x564bb994ec00 auth_method 0 2022-09-12T20:16:37.303 INFO:teuthology.orchestra.run.smithi138.stderr: -294> 2022-09-12T20:16:37.061+0000 7fec90ba7700 10 monclient: get_auth_request con 0x564bb994fc00 auth_method 0 2022-09-12T20:16:37.304 INFO:teuthology.orchestra.run.smithi138.stderr: -293> 2022-09-12T20:16:37.062+0000 7fec903a6700 10 monclient: get_auth_request con 0x564bb9c6a800 auth_method 0 2022-09-12T20:16:37.304 INFO:teuthology.orchestra.run.smithi138.stderr: -292> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973f20) adding auth protocol: cephx 2022-09-12T20:16:37.304 INFO:teuthology.orchestra.run.smithi138.stderr: -291> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973f20) adding auth protocol: cephx 2022-09-12T20:16:37.304 INFO:teuthology.orchestra.run.smithi138.stderr: -290> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973f20) adding auth protocol: cephx 2022-09-12T20:16:37.305 INFO:teuthology.orchestra.run.smithi138.stderr: -289> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973f20) adding con mode: secure 2022-09-12T20:16:37.305 INFO:teuthology.orchestra.run.smithi138.stderr: -288> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973f20) adding con mode: crc 2022-09-12T20:16:37.305 INFO:teuthology.orchestra.run.smithi138.stderr: -287> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973f20) adding con mode: secure 2022-09-12T20:16:37.305 INFO:teuthology.orchestra.run.smithi138.stderr: -286> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973f20) adding con mode: crc 2022-09-12T20:16:37.306 INFO:teuthology.orchestra.run.smithi138.stderr: -285> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973f20) adding con mode: secure 2022-09-12T20:16:37.306 INFO:teuthology.orchestra.run.smithi138.stderr: -284> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973f20) adding con mode: crc 2022-09-12T20:16:37.306 INFO:teuthology.orchestra.run.smithi138.stderr: -283> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973f20) adding con mode: crc 2022-09-12T20:16:37.306 INFO:teuthology.orchestra.run.smithi138.stderr: -282> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973f20) adding con mode: secure 2022-09-12T20:16:37.307 INFO:teuthology.orchestra.run.smithi138.stderr: -281> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973f20) adding con mode: crc 2022-09-12T20:16:37.307 INFO:teuthology.orchestra.run.smithi138.stderr: -280> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973f20) adding con mode: secure 2022-09-12T20:16:37.307 INFO:teuthology.orchestra.run.smithi138.stderr: -279> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973f20) adding con mode: crc 2022-09-12T20:16:37.308 INFO:teuthology.orchestra.run.smithi138.stderr: -278> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973f20) adding con mode: secure 2022-09-12T20:16:37.308 INFO:teuthology.orchestra.run.smithi138.stderr: -277> 2022-09-12T20:16:37.090+0000 7fec92bfb600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-12T20:16:37.308 INFO:teuthology.orchestra.run.smithi138.stderr: -276> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 asok(0x564bb99e2000) register_command cache list hook 0x564bb9942b60 2022-09-12T20:16:37.308 INFO:teuthology.orchestra.run.smithi138.stderr: -275> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 asok(0x564bb99e2000) register_command cache inspect hook 0x564bb9942b60 2022-09-12T20:16:37.309 INFO:teuthology.orchestra.run.smithi138.stderr: -274> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 asok(0x564bb99e2000) register_command cache erase hook 0x564bb9942b60 2022-09-12T20:16:37.309 INFO:teuthology.orchestra.run.smithi138.stderr: -273> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 asok(0x564bb99e2000) register_command cache zap hook 0x564bb9942b60 2022-09-12T20:16:37.309 INFO:teuthology.orchestra.run.smithi138.stderr: -272> 2022-09-12T20:16:37.090+0000 7fec92bfb600 10 monclient: get_monmap_and_config 2022-09-12T20:16:37.309 INFO:teuthology.orchestra.run.smithi138.stderr: -271> 2022-09-12T20:16:37.090+0000 7fec92bfb600 10 monclient: build_initial_monmap 2022-09-12T20:16:37.310 INFO:teuthology.orchestra.run.smithi138.stderr: -270> 2022-09-12T20:16:37.090+0000 7fec92bfb600 1 build_initial for_mkfs: 0 2022-09-12T20:16:37.310 INFO:teuthology.orchestra.run.smithi138.stderr: -269> 2022-09-12T20:16:37.090+0000 7fec92bfb600 10 monclient: monmap: 2022-09-12T20:16:37.310 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2022-09-12T20:16:37.310 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2022-09-12T20:16:37.311 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2022-09-12T20:16:37.311 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2022-09-12T20:16:37.311 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2022-09-12T20:16:37.311 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2022-09-12T20:16:37.312 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2022-09-12T20:16:37.312 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-09-12T20:16:37.312 INFO:teuthology.orchestra.run.smithi138.stderr: -268> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding auth protocol: cephx 2022-09-12T20:16:37.312 INFO:teuthology.orchestra.run.smithi138.stderr: -267> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding auth protocol: cephx 2022-09-12T20:16:37.313 INFO:teuthology.orchestra.run.smithi138.stderr: -266> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding auth protocol: cephx 2022-09-12T20:16:37.313 INFO:teuthology.orchestra.run.smithi138.stderr: -265> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding con mode: secure 2022-09-12T20:16:37.313 INFO:teuthology.orchestra.run.smithi138.stderr: -264> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding con mode: crc 2022-09-12T20:16:37.313 INFO:teuthology.orchestra.run.smithi138.stderr: -263> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding con mode: secure 2022-09-12T20:16:37.314 INFO:teuthology.orchestra.run.smithi138.stderr: -262> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding con mode: crc 2022-09-12T20:16:37.314 INFO:teuthology.orchestra.run.smithi138.stderr: -261> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding con mode: secure 2022-09-12T20:16:37.314 INFO:teuthology.orchestra.run.smithi138.stderr: -260> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding con mode: crc 2022-09-12T20:16:37.314 INFO:teuthology.orchestra.run.smithi138.stderr: -259> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding con mode: crc 2022-09-12T20:16:37.315 INFO:teuthology.orchestra.run.smithi138.stderr: -258> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding con mode: secure 2022-09-12T20:16:37.315 INFO:teuthology.orchestra.run.smithi138.stderr: -257> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding con mode: crc 2022-09-12T20:16:37.315 INFO:teuthology.orchestra.run.smithi138.stderr: -256> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding con mode: secure 2022-09-12T20:16:37.315 INFO:teuthology.orchestra.run.smithi138.stderr: -255> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding con mode: crc 2022-09-12T20:16:37.316 INFO:teuthology.orchestra.run.smithi138.stderr: -254> 2022-09-12T20:16:37.090+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding con mode: secure 2022-09-12T20:16:37.316 INFO:teuthology.orchestra.run.smithi138.stderr: -253> 2022-09-12T20:16:37.091+0000 7fec92bfb600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-12T20:16:37.316 INFO:teuthology.orchestra.run.smithi138.stderr: -252> 2022-09-12T20:16:37.091+0000 7fec92bfb600 10 monclient: init 2022-09-12T20:16:37.317 INFO:teuthology.orchestra.run.smithi138.stderr: -251> 2022-09-12T20:16:37.091+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973b50) adding auth protocol: cephx 2022-09-12T20:16:37.317 INFO:teuthology.orchestra.run.smithi138.stderr: -250> 2022-09-12T20:16:37.091+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973b50) adding auth protocol: cephx 2022-09-12T20:16:37.317 INFO:teuthology.orchestra.run.smithi138.stderr: -249> 2022-09-12T20:16:37.091+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973b50) adding auth protocol: cephx 2022-09-12T20:16:37.317 INFO:teuthology.orchestra.run.smithi138.stderr: -248> 2022-09-12T20:16:37.091+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973b50) adding con mode: secure 2022-09-12T20:16:37.318 INFO:teuthology.orchestra.run.smithi138.stderr: -247> 2022-09-12T20:16:37.091+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973b50) adding con mode: crc 2022-09-12T20:16:37.318 INFO:teuthology.orchestra.run.smithi138.stderr: -246> 2022-09-12T20:16:37.091+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973b50) adding con mode: secure 2022-09-12T20:16:37.318 INFO:teuthology.orchestra.run.smithi138.stderr: -245> 2022-09-12T20:16:37.091+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973b50) adding con mode: crc 2022-09-12T20:16:37.318 INFO:teuthology.orchestra.run.smithi138.stderr: -244> 2022-09-12T20:16:37.091+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973b50) adding con mode: secure 2022-09-12T20:16:37.319 INFO:teuthology.orchestra.run.smithi138.stderr: -243> 2022-09-12T20:16:37.091+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973b50) adding con mode: crc 2022-09-12T20:16:37.319 INFO:teuthology.orchestra.run.smithi138.stderr: -242> 2022-09-12T20:16:37.091+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973b50) adding con mode: crc 2022-09-12T20:16:37.319 INFO:teuthology.orchestra.run.smithi138.stderr: -241> 2022-09-12T20:16:37.091+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973b50) adding con mode: secure 2022-09-12T20:16:37.319 INFO:teuthology.orchestra.run.smithi138.stderr: -240> 2022-09-12T20:16:37.091+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973b50) adding con mode: crc 2022-09-12T20:16:37.320 INFO:teuthology.orchestra.run.smithi138.stderr: -239> 2022-09-12T20:16:37.091+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973b50) adding con mode: secure 2022-09-12T20:16:37.320 INFO:teuthology.orchestra.run.smithi138.stderr: -238> 2022-09-12T20:16:37.091+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973b50) adding con mode: crc 2022-09-12T20:16:37.320 INFO:teuthology.orchestra.run.smithi138.stderr: -237> 2022-09-12T20:16:37.091+0000 7fec92bfb600 5 AuthRegistry(0x7ffec6973b50) adding con mode: secure 2022-09-12T20:16:37.320 INFO:teuthology.orchestra.run.smithi138.stderr: -236> 2022-09-12T20:16:37.091+0000 7fec92bfb600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-12T20:16:37.321 INFO:teuthology.orchestra.run.smithi138.stderr: -235> 2022-09-12T20:16:37.091+0000 7fec92bfb600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-12T20:16:37.321 INFO:teuthology.orchestra.run.smithi138.stderr: -234> 2022-09-12T20:16:37.091+0000 7fec92bfb600 10 monclient: _reopen_session rank -1 2022-09-12T20:16:37.321 INFO:teuthology.orchestra.run.smithi138.stderr: -233> 2022-09-12T20:16:37.091+0000 7fec92bfb600 10 monclient: _add_conns ranks=[0] 2022-09-12T20:16:37.321 INFO:teuthology.orchestra.run.smithi138.stderr: -232> 2022-09-12T20:16:37.091+0000 7fec92bfb600 10 monclient(hunting): picked mon.noname-a con 0x564bb994f800 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2022-09-12T20:16:37.322 INFO:teuthology.orchestra.run.smithi138.stderr: -231> 2022-09-12T20:16:37.091+0000 7fec92bfb600 10 monclient(hunting): start opening mon connection 2022-09-12T20:16:37.322 INFO:teuthology.orchestra.run.smithi138.stderr: -230> 2022-09-12T20:16:37.091+0000 7fec92bfb600 10 monclient(hunting): _renew_subs 2022-09-12T20:16:37.322 INFO:teuthology.orchestra.run.smithi138.stderr: -229> 2022-09-12T20:16:37.091+0000 7fec92bfb600 10 monclient(hunting): authenticate will time out at 6431.680356s 2022-09-12T20:16:37.323 INFO:teuthology.orchestra.run.smithi138.stderr: -228> 2022-09-12T20:16:37.091+0000 7fec90ba7700 10 monclient(hunting): get_auth_request con 0x564bb994f800 auth_method 0 2022-09-12T20:16:37.323 INFO:teuthology.orchestra.run.smithi138.stderr: -227> 2022-09-12T20:16:37.091+0000 7fec90ba7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2022-09-12T20:16:37.323 INFO:teuthology.orchestra.run.smithi138.stderr: -226> 2022-09-12T20:16:37.091+0000 7fec90ba7700 10 monclient(hunting): _init_auth method 2 2022-09-12T20:16:37.323 INFO:teuthology.orchestra.run.smithi138.stderr: -225> 2022-09-12T20:16:37.091+0000 7fec90ba7700 10 monclient(hunting): _init_auth creating new auth 2022-09-12T20:16:37.324 INFO:teuthology.orchestra.run.smithi138.stderr: -224> 2022-09-12T20:16:37.091+0000 7fec90ba7700 10 monclient(hunting): handle_auth_reply_more payload 9 2022-09-12T20:16:37.324 INFO:teuthology.orchestra.run.smithi138.stderr: -223> 2022-09-12T20:16:37.091+0000 7fec90ba7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2022-09-12T20:16:37.324 INFO:teuthology.orchestra.run.smithi138.stderr: -222> 2022-09-12T20:16:37.091+0000 7fec90ba7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2022-09-12T20:16:37.324 INFO:teuthology.orchestra.run.smithi138.stderr: -221> 2022-09-12T20:16:37.091+0000 7fec90ba7700 10 monclient(hunting): handle_auth_done global_id 6730 payload 274 2022-09-12T20:16:37.325 INFO:teuthology.orchestra.run.smithi138.stderr: -220> 2022-09-12T20:16:37.091+0000 7fec90ba7700 10 monclient: _finish_hunting 0 2022-09-12T20:16:37.325 INFO:teuthology.orchestra.run.smithi138.stderr: -219> 2022-09-12T20:16:37.091+0000 7fec90ba7700 1 monclient: found mon.noname-a 2022-09-12T20:16:37.325 INFO:teuthology.orchestra.run.smithi138.stderr: -218> 2022-09-12T20:16:37.091+0000 7fec90ba7700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2022-09-12T20:16:37.325 INFO:teuthology.orchestra.run.smithi138.stderr: -217> 2022-09-12T20:16:37.091+0000 7fec7ab7b700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-12T20:16:37.326 INFO:teuthology.orchestra.run.smithi138.stderr: -216> 2022-09-12T20:16:37.091+0000 7fec7ab7b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2022-09-12T20:16:37.326 INFO:teuthology.orchestra.run.smithi138.stderr: -215> 2022-09-12T20:16:37.091+0000 7fec7ab7b700 10 monclient: dump: 2022-09-12T20:16:37.326 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2022-09-12T20:16:37.326 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 27700e6f-3189-467a-882c-63786db1882c 2022-09-12T20:16:37.327 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2022-09-12T18:45:24.531591+0000 2022-09-12T20:16:37.327 INFO:teuthology.orchestra.run.smithi138.stderr:created 2022-09-12T18:45:24.531591+0000 2022-09-12T20:16:37.327 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 17 (quincy) 2022-09-12T20:16:37.327 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2022-09-12T20:16:37.328 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2022-09-12T20:16:37.328 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-09-12T20:16:37.328 INFO:teuthology.orchestra.run.smithi138.stderr: -214> 2022-09-12T20:16:37.091+0000 7fec7ab7b700 10 monclient: _finish_auth 0 2022-09-12T20:16:37.328 INFO:teuthology.orchestra.run.smithi138.stderr: -213> 2022-09-12T20:16:37.091+0000 7fec7ab7b700 10 monclient: _check_auth_tickets 2022-09-12T20:16:37.329 INFO:teuthology.orchestra.run.smithi138.stderr: -212> 2022-09-12T20:16:37.091+0000 7fec92bfb600 5 monclient: authenticate success, global_id 6730 2022-09-12T20:16:37.329 INFO:teuthology.orchestra.run.smithi138.stderr: -211> 2022-09-12T20:16:37.091+0000 7fec7ab7b700 10 monclient: handle_config config(0 keys) v1 2022-09-12T20:16:37.329 INFO:teuthology.orchestra.run.smithi138.stderr: -210> 2022-09-12T20:16:37.091+0000 7fec7ab7b700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-12T20:16:37.329 INFO:teuthology.orchestra.run.smithi138.stderr: -209> 2022-09-12T20:16:37.091+0000 7fec7ab7b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2022-09-12T20:16:37.330 INFO:teuthology.orchestra.run.smithi138.stderr: -208> 2022-09-12T20:16:37.091+0000 7fec7ab7b700 10 monclient: dump: 2022-09-12T20:16:37.330 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2022-09-12T20:16:37.330 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 27700e6f-3189-467a-882c-63786db1882c 2022-09-12T20:16:37.331 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2022-09-12T18:45:24.531591+0000 2022-09-12T20:16:37.331 INFO:teuthology.orchestra.run.smithi138.stderr:created 2022-09-12T18:45:24.531591+0000 2022-09-12T20:16:37.331 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 17 (quincy) 2022-09-12T20:16:37.331 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2022-09-12T20:16:37.332 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2022-09-12T20:16:37.332 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-09-12T20:16:37.332 INFO:teuthology.orchestra.run.smithi138.stderr: -207> 2022-09-12T20:16:37.092+0000 7fec92bfb600 10 monclient: get_monmap_and_config success 2022-09-12T20:16:37.332 INFO:teuthology.orchestra.run.smithi138.stderr: -206> 2022-09-12T20:16:37.092+0000 7fec92bfb600 4 set_mon_vals no callback set 2022-09-12T20:16:37.333 INFO:teuthology.orchestra.run.smithi138.stderr: -205> 2022-09-12T20:16:37.092+0000 7fec92bfb600 10 monclient: shutdown 2022-09-12T20:16:37.333 INFO:teuthology.orchestra.run.smithi138.stderr: -204> 2022-09-12T20:16:37.092+0000 7fec92bfb600 10 monclient: build_initial_monmap 2022-09-12T20:16:37.333 INFO:teuthology.orchestra.run.smithi138.stderr: -203> 2022-09-12T20:16:37.092+0000 7fec92bfb600 1 build_initial for_mkfs: 0 2022-09-12T20:16:37.333 INFO:teuthology.orchestra.run.smithi138.stderr: -202> 2022-09-12T20:16:37.092+0000 7fec92bfb600 10 monclient: monmap: 2022-09-12T20:16:37.334 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2022-09-12T20:16:37.334 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2022-09-12T20:16:37.334 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2022-09-12T20:16:37.334 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2022-09-12T20:16:37.335 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2022-09-12T20:16:37.335 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2022-09-12T20:16:37.335 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2022-09-12T20:16:37.335 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-09-12T20:16:37.336 INFO:teuthology.orchestra.run.smithi138.stderr: -201> 2022-09-12T20:16:37.092+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding auth protocol: cephx 2022-09-12T20:16:37.336 INFO:teuthology.orchestra.run.smithi138.stderr: -200> 2022-09-12T20:16:37.092+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding auth protocol: cephx 2022-09-12T20:16:37.336 INFO:teuthology.orchestra.run.smithi138.stderr: -199> 2022-09-12T20:16:37.092+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding auth protocol: cephx 2022-09-12T20:16:37.336 INFO:teuthology.orchestra.run.smithi138.stderr: -198> 2022-09-12T20:16:37.092+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding con mode: secure 2022-09-12T20:16:37.337 INFO:teuthology.orchestra.run.smithi138.stderr: -197> 2022-09-12T20:16:37.092+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding con mode: crc 2022-09-12T20:16:37.337 INFO:teuthology.orchestra.run.smithi138.stderr: -196> 2022-09-12T20:16:37.092+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding con mode: secure 2022-09-12T20:16:37.337 INFO:teuthology.orchestra.run.smithi138.stderr: -195> 2022-09-12T20:16:37.092+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding con mode: crc 2022-09-12T20:16:37.337 INFO:teuthology.orchestra.run.smithi138.stderr: -194> 2022-09-12T20:16:37.092+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding con mode: secure 2022-09-12T20:16:37.338 INFO:teuthology.orchestra.run.smithi138.stderr: -193> 2022-09-12T20:16:37.092+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding con mode: crc 2022-09-12T20:16:37.338 INFO:teuthology.orchestra.run.smithi138.stderr: -192> 2022-09-12T20:16:37.092+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding con mode: crc 2022-09-12T20:16:37.338 INFO:teuthology.orchestra.run.smithi138.stderr: -191> 2022-09-12T20:16:37.092+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding con mode: secure 2022-09-12T20:16:37.338 INFO:teuthology.orchestra.run.smithi138.stderr: -190> 2022-09-12T20:16:37.092+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding con mode: crc 2022-09-12T20:16:37.339 INFO:teuthology.orchestra.run.smithi138.stderr: -189> 2022-09-12T20:16:37.092+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding con mode: secure 2022-09-12T20:16:37.339 INFO:teuthology.orchestra.run.smithi138.stderr: -188> 2022-09-12T20:16:37.092+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding con mode: crc 2022-09-12T20:16:37.339 INFO:teuthology.orchestra.run.smithi138.stderr: -187> 2022-09-12T20:16:37.092+0000 7fec92bfb600 5 AuthRegistry(0x564bba6ae140) adding con mode: secure 2022-09-12T20:16:37.339 INFO:teuthology.orchestra.run.smithi138.stderr: -186> 2022-09-12T20:16:37.092+0000 7fec92bfb600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-12T20:16:37.340 INFO:teuthology.orchestra.run.smithi138.stderr: -185> 2022-09-12T20:16:37.092+0000 7fec92bfb600 1 librados: starting msgr at 2022-09-12T20:16:37.340 INFO:teuthology.orchestra.run.smithi138.stderr: -184> 2022-09-12T20:16:37.092+0000 7fec92bfb600 1 librados: starting objecter 2022-09-12T20:16:37.340 INFO:teuthology.orchestra.run.smithi138.stderr: -183> 2022-09-12T20:16:37.092+0000 7fec92bfb600 5 asok(0x564bb99e2000) register_command objecter_requests cmddesc objecter_requests hook 0x564bb9942750 EEXIST 2022-09-12T20:16:37.341 INFO:teuthology.orchestra.run.smithi138.stderr: -182> 2022-09-12T20:16:37.093+0000 7fec92bfb600 1 librados: setting wanted keys 2022-09-12T20:16:37.341 INFO:teuthology.orchestra.run.smithi138.stderr: -181> 2022-09-12T20:16:37.093+0000 7fec92bfb600 1 librados: calling monclient init 2022-09-12T20:16:37.341 INFO:teuthology.orchestra.run.smithi138.stderr: -180> 2022-09-12T20:16:37.093+0000 7fec92bfb600 10 monclient: init 2022-09-12T20:16:37.341 INFO:teuthology.orchestra.run.smithi138.stderr: -179> 2022-09-12T20:16:37.093+0000 7fec92bfb600 5 AuthRegistry(0x564bb9c7d0d8) adding auth protocol: cephx 2022-09-12T20:16:37.342 INFO:teuthology.orchestra.run.smithi138.stderr: -178> 2022-09-12T20:16:37.093+0000 7fec92bfb600 5 AuthRegistry(0x564bb9c7d0d8) adding auth protocol: cephx 2022-09-12T20:16:37.342 INFO:teuthology.orchestra.run.smithi138.stderr: -177> 2022-09-12T20:16:37.093+0000 7fec92bfb600 5 AuthRegistry(0x564bb9c7d0d8) adding auth protocol: cephx 2022-09-12T20:16:37.342 INFO:teuthology.orchestra.run.smithi138.stderr: -176> 2022-09-12T20:16:37.093+0000 7fec92bfb600 5 AuthRegistry(0x564bb9c7d0d8) adding con mode: secure 2022-09-12T20:16:37.342 INFO:teuthology.orchestra.run.smithi138.stderr: -175> 2022-09-12T20:16:37.093+0000 7fec92bfb600 5 AuthRegistry(0x564bb9c7d0d8) adding con mode: crc 2022-09-12T20:16:37.343 INFO:teuthology.orchestra.run.smithi138.stderr: -174> 2022-09-12T20:16:37.093+0000 7fec92bfb600 5 AuthRegistry(0x564bb9c7d0d8) adding con mode: secure 2022-09-12T20:16:37.343 INFO:teuthology.orchestra.run.smithi138.stderr: -173> 2022-09-12T20:16:37.093+0000 7fec92bfb600 5 AuthRegistry(0x564bb9c7d0d8) adding con mode: crc 2022-09-12T20:16:37.343 INFO:teuthology.orchestra.run.smithi138.stderr: -172> 2022-09-12T20:16:37.093+0000 7fec92bfb600 5 AuthRegistry(0x564bb9c7d0d8) adding con mode: secure 2022-09-12T20:16:37.343 INFO:teuthology.orchestra.run.smithi138.stderr: -171> 2022-09-12T20:16:37.093+0000 7fec92bfb600 5 AuthRegistry(0x564bb9c7d0d8) adding con mode: crc 2022-09-12T20:16:37.344 INFO:teuthology.orchestra.run.smithi138.stderr: -170> 2022-09-12T20:16:37.093+0000 7fec92bfb600 5 AuthRegistry(0x564bb9c7d0d8) adding con mode: crc 2022-09-12T20:16:37.344 INFO:teuthology.orchestra.run.smithi138.stderr: -169> 2022-09-12T20:16:37.093+0000 7fec92bfb600 5 AuthRegistry(0x564bb9c7d0d8) adding con mode: secure 2022-09-12T20:16:37.344 INFO:teuthology.orchestra.run.smithi138.stderr: -168> 2022-09-12T20:16:37.093+0000 7fec92bfb600 5 AuthRegistry(0x564bb9c7d0d8) adding con mode: crc 2022-09-12T20:16:37.344 INFO:teuthology.orchestra.run.smithi138.stderr: -167> 2022-09-12T20:16:37.093+0000 7fec92bfb600 5 AuthRegistry(0x564bb9c7d0d8) adding con mode: secure 2022-09-12T20:16:37.345 INFO:teuthology.orchestra.run.smithi138.stderr: -166> 2022-09-12T20:16:37.093+0000 7fec92bfb600 5 AuthRegistry(0x564bb9c7d0d8) adding con mode: crc 2022-09-12T20:16:37.345 INFO:teuthology.orchestra.run.smithi138.stderr: -165> 2022-09-12T20:16:37.093+0000 7fec92bfb600 5 AuthRegistry(0x564bb9c7d0d8) adding con mode: secure 2022-09-12T20:16:37.345 INFO:teuthology.orchestra.run.smithi138.stderr: -164> 2022-09-12T20:16:37.093+0000 7fec92bfb600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-12T20:16:37.345 INFO:teuthology.orchestra.run.smithi138.stderr: -163> 2022-09-12T20:16:37.093+0000 7fec92bfb600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-12T20:16:37.346 INFO:teuthology.orchestra.run.smithi138.stderr: -162> 2022-09-12T20:16:37.093+0000 7fec92bfb600 10 monclient: _reopen_session rank -1 2022-09-12T20:16:37.346 INFO:teuthology.orchestra.run.smithi138.stderr: -161> 2022-09-12T20:16:37.093+0000 7fec92bfb600 10 monclient: _add_conns ranks=[0] 2022-09-12T20:16:37.346 INFO:teuthology.orchestra.run.smithi138.stderr: -160> 2022-09-12T20:16:37.093+0000 7fec92bfb600 10 monclient(hunting): picked mon.noname-a con 0x564bb994f800 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2022-09-12T20:16:37.346 INFO:teuthology.orchestra.run.smithi138.stderr: -159> 2022-09-12T20:16:37.093+0000 7fec92bfb600 10 monclient(hunting): start opening mon connection 2022-09-12T20:16:37.347 INFO:teuthology.orchestra.run.smithi138.stderr: -158> 2022-09-12T20:16:37.093+0000 7fec92bfb600 10 monclient(hunting): _renew_subs 2022-09-12T20:16:37.347 INFO:teuthology.orchestra.run.smithi138.stderr: -157> 2022-09-12T20:16:37.093+0000 7fec92bfb600 10 monclient(hunting): authenticate will time out at 6431.682287s 2022-09-12T20:16:37.347 INFO:teuthology.orchestra.run.smithi138.stderr: -156> 2022-09-12T20:16:37.093+0000 7fec90ba7700 10 monclient(hunting): get_auth_request con 0x564bb994f800 auth_method 0 2022-09-12T20:16:37.348 INFO:teuthology.orchestra.run.smithi138.stderr: -155> 2022-09-12T20:16:37.093+0000 7fec90ba7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2022-09-12T20:16:37.348 INFO:teuthology.orchestra.run.smithi138.stderr: -154> 2022-09-12T20:16:37.093+0000 7fec90ba7700 10 monclient(hunting): _init_auth method 2 2022-09-12T20:16:37.348 INFO:teuthology.orchestra.run.smithi138.stderr: -153> 2022-09-12T20:16:37.093+0000 7fec90ba7700 10 monclient(hunting): _init_auth creating new auth 2022-09-12T20:16:37.348 INFO:teuthology.orchestra.run.smithi138.stderr: -152> 2022-09-12T20:16:37.093+0000 7fec90ba7700 10 monclient(hunting): handle_auth_reply_more payload 9 2022-09-12T20:16:37.349 INFO:teuthology.orchestra.run.smithi138.stderr: -151> 2022-09-12T20:16:37.093+0000 7fec90ba7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2022-09-12T20:16:37.349 INFO:teuthology.orchestra.run.smithi138.stderr: -150> 2022-09-12T20:16:37.093+0000 7fec90ba7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2022-09-12T20:16:37.349 INFO:teuthology.orchestra.run.smithi138.stderr: -149> 2022-09-12T20:16:37.093+0000 7fec90ba7700 10 monclient(hunting): handle_auth_done global_id 6731 payload 824 2022-09-12T20:16:37.349 INFO:teuthology.orchestra.run.smithi138.stderr: -148> 2022-09-12T20:16:37.093+0000 7fec90ba7700 10 monclient: _finish_hunting 0 2022-09-12T20:16:37.350 INFO:teuthology.orchestra.run.smithi138.stderr: -147> 2022-09-12T20:16:37.093+0000 7fec90ba7700 1 monclient: found mon.noname-a 2022-09-12T20:16:37.350 INFO:teuthology.orchestra.run.smithi138.stderr: -146> 2022-09-12T20:16:37.093+0000 7fec90ba7700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2022-09-12T20:16:37.350 INFO:teuthology.orchestra.run.smithi138.stderr: -145> 2022-09-12T20:16:37.093+0000 7fec79378700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-12T20:16:37.350 INFO:teuthology.orchestra.run.smithi138.stderr: -144> 2022-09-12T20:16:37.093+0000 7fec79378700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2022-09-12T20:16:37.351 INFO:teuthology.orchestra.run.smithi138.stderr: -143> 2022-09-12T20:16:37.093+0000 7fec79378700 10 monclient: dump: 2022-09-12T20:16:37.351 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2022-09-12T20:16:37.351 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 27700e6f-3189-467a-882c-63786db1882c 2022-09-12T20:16:37.351 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2022-09-12T18:45:24.531591+0000 2022-09-12T20:16:37.352 INFO:teuthology.orchestra.run.smithi138.stderr:created 2022-09-12T18:45:24.531591+0000 2022-09-12T20:16:37.352 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 17 (quincy) 2022-09-12T20:16:37.352 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2022-09-12T20:16:37.352 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2022-09-12T20:16:37.353 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-09-12T20:16:37.353 INFO:teuthology.orchestra.run.smithi138.stderr: -142> 2022-09-12T20:16:37.093+0000 7fec79378700 10 monclient: _finish_auth 0 2022-09-12T20:16:37.353 INFO:teuthology.orchestra.run.smithi138.stderr: -141> 2022-09-12T20:16:37.093+0000 7fec79378700 10 monclient: _check_auth_tickets 2022-09-12T20:16:37.353 INFO:teuthology.orchestra.run.smithi138.stderr: -140> 2022-09-12T20:16:37.093+0000 7fec92bfb600 5 monclient: authenticate success, global_id 6731 2022-09-12T20:16:37.354 INFO:teuthology.orchestra.run.smithi138.stderr: -139> 2022-09-12T20:16:37.093+0000 7fec92bfb600 10 monclient: _renew_subs 2022-09-12T20:16:37.354 INFO:teuthology.orchestra.run.smithi138.stderr: -138> 2022-09-12T20:16:37.093+0000 7fec92bfb600 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2022-09-12T20:16:37.354 INFO:teuthology.orchestra.run.smithi138.stderr: -137> 2022-09-12T20:16:37.093+0000 7fec79378700 10 monclient: handle_config config(0 keys) v1 2022-09-12T20:16:37.355 INFO:teuthology.orchestra.run.smithi138.stderr: -136> 2022-09-12T20:16:37.093+0000 7fec79378700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-12T20:16:37.355 INFO:teuthology.orchestra.run.smithi138.stderr: -135> 2022-09-12T20:16:37.093+0000 7fec79378700 10 monclient: got monmap 1 from mon.a (according to old e1) 2022-09-12T20:16:37.355 INFO:teuthology.orchestra.run.smithi138.stderr: -134> 2022-09-12T20:16:37.093+0000 7fec7ab7b700 4 set_mon_vals no callback set 2022-09-12T20:16:37.355 INFO:teuthology.orchestra.run.smithi138.stderr: -133> 2022-09-12T20:16:37.093+0000 7fec79378700 10 monclient: dump: 2022-09-12T20:16:37.356 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2022-09-12T20:16:37.356 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 27700e6f-3189-467a-882c-63786db1882c 2022-09-12T20:16:37.356 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2022-09-12T18:45:24.531591+0000 2022-09-12T20:16:37.356 INFO:teuthology.orchestra.run.smithi138.stderr:created 2022-09-12T18:45:24.531591+0000 2022-09-12T20:16:37.357 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 17 (quincy) 2022-09-12T20:16:37.357 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2022-09-12T20:16:37.357 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2022-09-12T20:16:37.357 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-09-12T20:16:37.358 INFO:teuthology.orchestra.run.smithi138.stderr: -132> 2022-09-12T20:16:37.093+0000 7fec92bfb600 10 monclient: _renew_subs 2022-09-12T20:16:37.358 INFO:teuthology.orchestra.run.smithi138.stderr: -131> 2022-09-12T20:16:37.093+0000 7fec92bfb600 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2022-09-12T20:16:37.358 INFO:teuthology.orchestra.run.smithi138.stderr: -130> 2022-09-12T20:16:37.093+0000 7fec92bfb600 1 librados: init done 2022-09-12T20:16:37.358 INFO:teuthology.orchestra.run.smithi138.stderr: -129> 2022-09-12T20:16:37.093+0000 7fec92bfb600 5 asok(0x564bb99e2000) register_command cr dump hook 0x564bb9abedd8 2022-09-12T20:16:37.359 INFO:teuthology.orchestra.run.smithi138.stderr: -128> 2022-09-12T20:16:37.094+0000 7fec79378700 4 mgrc handle_mgr_map Got map version 5 2022-09-12T20:16:37.359 INFO:teuthology.orchestra.run.smithi138.stderr: -127> 2022-09-12T20:16:37.094+0000 7fec79378700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.138:6824/101383,v1:172.21.15.138:6825/101383] 2022-09-12T20:16:37.359 INFO:teuthology.orchestra.run.smithi138.stderr: -126> 2022-09-12T20:16:37.094+0000 7fec79378700 4 mgrc reconnect Starting new session with [v2:172.21.15.138:6824/101383,v1:172.21.15.138:6825/101383] 2022-09-12T20:16:37.359 INFO:teuthology.orchestra.run.smithi138.stderr: -125> 2022-09-12T20:16:37.094+0000 7fec903a6700 10 monclient: get_auth_request con 0x564bb9cb4c00 auth_method 0 2022-09-12T20:16:37.360 INFO:teuthology.orchestra.run.smithi138.stderr: -124> 2022-09-12T20:16:37.095+0000 7fec913a8700 10 monclient: get_auth_request con 0x564bb9c6ac00 auth_method 0 2022-09-12T20:16:37.360 INFO:teuthology.orchestra.run.smithi138.stderr: -123> 2022-09-12T20:16:37.096+0000 7fec90ba7700 10 monclient: get_auth_request con 0x564bb9d62000 auth_method 0 2022-09-12T20:16:37.360 INFO:teuthology.orchestra.run.smithi138.stderr: -122> 2022-09-12T20:16:37.100+0000 7fec903a6700 10 monclient: get_auth_request con 0x564bb9d62c00 auth_method 0 2022-09-12T20:16:37.360 INFO:teuthology.orchestra.run.smithi138.stderr: -121> 2022-09-12T20:16:37.139+0000 7fec92bfb600 5 asok(0x564bb99e2000) register_command sync trace show hook 0x564bb9977420 2022-09-12T20:16:37.361 INFO:teuthology.orchestra.run.smithi138.stderr: -120> 2022-09-12T20:16:37.139+0000 7fec92bfb600 5 asok(0x564bb99e2000) register_command sync trace history hook 0x564bb9977420 2022-09-12T20:16:37.361 INFO:teuthology.orchestra.run.smithi138.stderr: -119> 2022-09-12T20:16:37.139+0000 7fec92bfb600 5 asok(0x564bb99e2000) register_command sync trace active hook 0x564bb9977420 2022-09-12T20:16:37.361 INFO:teuthology.orchestra.run.smithi138.stderr: -118> 2022-09-12T20:16:37.139+0000 7fec92bfb600 5 asok(0x564bb99e2000) register_command sync trace active_short hook 0x564bb9977420 2022-09-12T20:16:37.361 INFO:teuthology.orchestra.run.smithi138.stderr: -117> 2022-09-12T20:16:37.142+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: sync status for source bucket: init. lease is: not taken. stop indications is: 0 2022-09-12T20:16:37.362 INFO:teuthology.orchestra.run.smithi138.stderr: -116> 2022-09-12T20:16:37.148+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: waiting on bucket lease 2022-09-12T20:16:37.362 INFO:teuthology.orchestra.run.smithi138.stderr: -115> 2022-09-12T20:16:37.149+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: status after acquiring the lock is: init 2022-09-12T20:16:37.362 INFO:teuthology.orchestra.run.smithi138.stderr: -114> 2022-09-12T20:16:37.171+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: listing bucket for full sync 2022-09-12T20:16:37.362 INFO:teuthology.orchestra.run.smithi138.stderr: -113> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj11 2022-09-12T20:16:37.363 INFO:teuthology.orchestra.run.smithi138.stderr: -112> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj11]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj11 op=0 op_state=1 2022-09-12T20:16:37.363 INFO:teuthology.orchestra.run.smithi138.stderr: -111> 2022-09-12T20:16:37.177+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj11]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj11[0] 2022-09-12T20:16:37.363 INFO:teuthology.orchestra.run.smithi138.stderr: -110> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj111 2022-09-12T20:16:37.364 INFO:teuthology.orchestra.run.smithi138.stderr: -109> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj111]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj111 op=0 op_state=1 2022-09-12T20:16:37.364 INFO:teuthology.orchestra.run.smithi138.stderr: -108> 2022-09-12T20:16:37.177+0000 7fec92bfb600 0 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj111]: : injecting data sync error on key=obj111 2022-09-12T20:16:37.364 INFO:teuthology.orchestra.run.smithi138.stderr: -107> 2022-09-12T20:16:37.177+0000 7fec92bfb600 10 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2022-09-12T20:16:37.364 INFO:teuthology.orchestra.run.smithi138.stderr: -106> 2022-09-12T20:16:37.177+0000 7fec92bfb600 0 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj111]: ERROR: failed to sync object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj111 2022-09-12T20:16:37.365 INFO:teuthology.orchestra.run.smithi138.stderr: -105> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj115 2022-09-12T20:16:37.365 INFO:teuthology.orchestra.run.smithi138.stderr: -104> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj115]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj115 op=0 op_state=1 2022-09-12T20:16:37.365 INFO:teuthology.orchestra.run.smithi138.stderr: -103> 2022-09-12T20:16:37.177+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj115]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj115[0] 2022-09-12T20:16:37.365 INFO:teuthology.orchestra.run.smithi138.stderr: -102> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj116 2022-09-12T20:16:37.366 INFO:teuthology.orchestra.run.smithi138.stderr: -101> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj116]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj116 op=0 op_state=1 2022-09-12T20:16:37.366 INFO:teuthology.orchestra.run.smithi138.stderr: -100> 2022-09-12T20:16:37.177+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj116]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj116[0] 2022-09-12T20:16:37.366 INFO:teuthology.orchestra.run.smithi138.stderr: -99> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj124 2022-09-12T20:16:37.366 INFO:teuthology.orchestra.run.smithi138.stderr: -98> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj124]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj124 op=0 op_state=1 2022-09-12T20:16:37.367 INFO:teuthology.orchestra.run.smithi138.stderr: -97> 2022-09-12T20:16:37.177+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj124]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj124[0] 2022-09-12T20:16:37.367 INFO:teuthology.orchestra.run.smithi138.stderr: -96> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj145 2022-09-12T20:16:37.367 INFO:teuthology.orchestra.run.smithi138.stderr: -95> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj145]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj145 op=0 op_state=1 2022-09-12T20:16:37.368 INFO:teuthology.orchestra.run.smithi138.stderr: -94> 2022-09-12T20:16:37.177+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj145]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj145[0] 2022-09-12T20:16:37.368 INFO:teuthology.orchestra.run.smithi138.stderr: -93> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj148 2022-09-12T20:16:37.368 INFO:teuthology.orchestra.run.smithi138.stderr: -92> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj148]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj148 op=0 op_state=1 2022-09-12T20:16:37.368 INFO:teuthology.orchestra.run.smithi138.stderr: -91> 2022-09-12T20:16:37.177+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj148]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj148[0] 2022-09-12T20:16:37.369 INFO:teuthology.orchestra.run.smithi138.stderr: -90> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj155 2022-09-12T20:16:37.369 INFO:teuthology.orchestra.run.smithi138.stderr: -89> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj155]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj155 op=0 op_state=1 2022-09-12T20:16:37.369 INFO:teuthology.orchestra.run.smithi138.stderr: -88> 2022-09-12T20:16:37.177+0000 7fec92bfb600 0 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj155]: : injecting data sync error on key=obj155 2022-09-12T20:16:37.369 INFO:teuthology.orchestra.run.smithi138.stderr: -87> 2022-09-12T20:16:37.177+0000 7fec92bfb600 10 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2022-09-12T20:16:37.370 INFO:teuthology.orchestra.run.smithi138.stderr: -86> 2022-09-12T20:16:37.177+0000 7fec92bfb600 0 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj155]: ERROR: failed to sync object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj155 2022-09-12T20:16:37.370 INFO:teuthology.orchestra.run.smithi138.stderr: -85> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj17 2022-09-12T20:16:37.370 INFO:teuthology.orchestra.run.smithi138.stderr: -84> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj17]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj17 op=0 op_state=1 2022-09-12T20:16:37.370 INFO:teuthology.orchestra.run.smithi138.stderr: -83> 2022-09-12T20:16:37.177+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj17]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj17[0] 2022-09-12T20:16:37.371 INFO:teuthology.orchestra.run.smithi138.stderr: -82> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj185 2022-09-12T20:16:37.371 INFO:teuthology.orchestra.run.smithi138.stderr: -81> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj185]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj185 op=0 op_state=1 2022-09-12T20:16:37.371 INFO:teuthology.orchestra.run.smithi138.stderr: -80> 2022-09-12T20:16:37.177+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj185]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj185[0] 2022-09-12T20:16:37.372 INFO:teuthology.orchestra.run.smithi138.stderr: -79> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj19 2022-09-12T20:16:37.372 INFO:teuthology.orchestra.run.smithi138.stderr: -78> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj19]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj19 op=0 op_state=1 2022-09-12T20:16:37.372 INFO:teuthology.orchestra.run.smithi138.stderr: -77> 2022-09-12T20:16:37.177+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj19]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj19[0] 2022-09-12T20:16:37.372 INFO:teuthology.orchestra.run.smithi138.stderr: -76> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj22 2022-09-12T20:16:37.373 INFO:teuthology.orchestra.run.smithi138.stderr: -75> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj22]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj22 op=0 op_state=1 2022-09-12T20:16:37.373 INFO:teuthology.orchestra.run.smithi138.stderr: -74> 2022-09-12T20:16:37.177+0000 7fec92bfb600 0 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj22]: : injecting data sync error on key=obj22 2022-09-12T20:16:37.373 INFO:teuthology.orchestra.run.smithi138.stderr: -73> 2022-09-12T20:16:37.177+0000 7fec92bfb600 10 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj22]: failed, retcode=-5 ((5) Input/output error) 2022-09-12T20:16:37.373 INFO:teuthology.orchestra.run.smithi138.stderr: -72> 2022-09-12T20:16:37.177+0000 7fec92bfb600 0 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj22]: ERROR: failed to sync object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj22 2022-09-12T20:16:37.374 INFO:teuthology.orchestra.run.smithi138.stderr: -71> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj23 2022-09-12T20:16:37.374 INFO:teuthology.orchestra.run.smithi138.stderr: -70> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj23]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj23 op=0 op_state=1 2022-09-12T20:16:37.374 INFO:teuthology.orchestra.run.smithi138.stderr: -69> 2022-09-12T20:16:37.177+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj23]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj23[0] 2022-09-12T20:16:37.374 INFO:teuthology.orchestra.run.smithi138.stderr: -68> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj29 2022-09-12T20:16:37.375 INFO:teuthology.orchestra.run.smithi138.stderr: -67> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj29]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj29 op=0 op_state=1 2022-09-12T20:16:37.375 INFO:teuthology.orchestra.run.smithi138.stderr: -66> 2022-09-12T20:16:37.177+0000 7fec92bfb600 0 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj29]: : injecting data sync error on key=obj29 2022-09-12T20:16:37.375 INFO:teuthology.orchestra.run.smithi138.stderr: -65> 2022-09-12T20:16:37.177+0000 7fec92bfb600 10 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2022-09-12T20:16:37.375 INFO:teuthology.orchestra.run.smithi138.stderr: -64> 2022-09-12T20:16:37.177+0000 7fec92bfb600 0 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj29]: ERROR: failed to sync object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj29 2022-09-12T20:16:37.376 INFO:teuthology.orchestra.run.smithi138.stderr: -63> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj31 2022-09-12T20:16:37.376 INFO:teuthology.orchestra.run.smithi138.stderr: -62> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj31]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj31 op=0 op_state=1 2022-09-12T20:16:37.376 INFO:teuthology.orchestra.run.smithi138.stderr: -61> 2022-09-12T20:16:37.177+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj31]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj31[0] 2022-09-12T20:16:37.377 INFO:teuthology.orchestra.run.smithi138.stderr: -60> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]: [full sync] syncing object: jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0/obj33 2022-09-12T20:16:37.377 INFO:teuthology.orchestra.run.smithi138.stderr: -59> 2022-09-12T20:16:37.177+0000 7fec92bfb600 20 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj33]: bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b=bucket sync single entry (source_zone=fb29b4fb-1f78-4524-ab80-6a5e9e01ab47) b= log_entry=obj33 op=0 op_state=1 2022-09-12T20:16:37.377 INFO:teuthology.orchestra.run.smithi138.stderr: -58> 2022-09-12T20:16:37.177+0000 7fec92bfb600 5 RGW-SYNC:bucket[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26<-jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:full_sync[jdpiak-75:fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26:0]:entry[obj33]: bucket sync: sync obj: fb29b4fb-1f78-4524-ab80-6a5e9e01ab47/:jdpiak-75[fb29b4fb-1f78-4524-ab80-6a5e9e01ab47.5570.26])/obj33[0] 2022-09-12T20:16:37.424 DEBUG:teuthology.orchestra.run:got remote process result: None 2022-09-12T20:16:37.424 INFO:tasks.rgw.c2.client.0:Restarting daemon 2022-09-12T20:16:37.425 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2022-09-12T20:16:37.428 INFO:tasks.rgw.c2.client.0:Started 2022-09-12T20:16:37.429 DEBUG:teuthology.orchestra.run.smithi138:> curl http://smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:37.450 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-12T20:16:37.451 INFO:teuthology.orchestra.run.smithi138.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-12T20:16:37.451 INFO:teuthology.orchestra.run.smithi138.stderr: Dload Upload Total Spent Left Speed 2022-09-12T20:16:37.452 INFO:teuthology.orchestra.run.smithi138.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi138.front.sepia.ceph.com port 8000: Connection refused 2022-09-12T20:16:38.453 DEBUG:teuthology.orchestra.run.smithi138:> curl http://smithi138.front.sepia.ceph.com:8000 2022-09-12T20:16:38.478 INFO:teuthology.orchestra.run.smithi138.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-12T20:16:38.479 INFO:teuthology.orchestra.run.smithi138.stderr: Dload Upload Total Spent Left Speed 2022-09-12T20:16:38.480 INFO:teuthology.orchestra.run.smithi138.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-12T20:16:38.481 INFO:teuthology.orchestra.run.smithi138.stdout:anonymous 2022-09-12T20:16:38.594 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... FAIL 2022-09-12T20:16:38.597 INFO:rgw_multi.tests:create role zone=test-zone1 name=roles-jdpiak-1 2022-09-12T20:16:38.598 DEBUG:boto:Method: POST 2022-09-12T20:16:38.599 DEBUG:boto:Path: / 2022-09-12T20:16:38.599 DEBUG:boto:Data: 2022-09-12T20:16:38.600 DEBUG:boto:Headers: {} 2022-09-12T20:16:38.600 DEBUG:boto:Host: smithi035.front.sepia.ceph.com 2022-09-12T20:16:38.601 DEBUG:boto:Port: 8000 2022-09-12T20:16:38.601 DEBUG:boto:Params: {'RoleName': 'roles-jdpiak-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2022-09-12T20:16:38.602 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T20:16:38.603 DEBUG:boto:Token: None 2022-09-12T20:16:38.604 DEBUG:boto:CanonicalRequest: POST / host:smithi035.front.sepia.ceph.com:8000 x-amz-date:20220912T201638Z host;x-amz-date b5e03dd23d579c9deaf550a94f384e7197b83214cb72367dbaa8d8cd019483fd 2022-09-12T20:16:38.604 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220912T201638Z 20220912/front/smithi035/aws4_request 2eeabda3438fe09ae64d4dea6ee7720c1446d27476285cda0d78b18b9411297b 2022-09-12T20:16:38.605 DEBUG:boto:Signature: 3674446162329457612029e7cc872fd803757f3a73ca6ec579225471ededae6e 2022-09-12T20:16:38.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220912T201638Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=FUCN3RT6SAOJRSRI0ZIX/20220912/front/smithi035/aws4_request,SignedHeaders=host;x-amz-date,Signature=3674446162329457612029e7cc872fd803757f3a73ca6ec579225471ededae6e', 'Host': 'smithi035.front.sepia.ceph.com:8000'} 2022-09-12T20:16:38.619 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000230f9f2120f289cc-00631f93a6-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '731'), ('Date', 'Mon, 12 Sep 2022 20:16:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:16:38.620 DEBUG:boto:b'a58d672c-a5ab-491a-ad02-9ea7f50aa831roles-jdpiak-1/arn:aws:iam:::role/roles-jdpiak-12022-09-12T20:16:38.610Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx00000230f9f2120f289cc-00631f93a6-15c0-test-zone1' 2022-09-12T20:16:38.622 INFO:rgw_multi.tests:create role zone=test-zone2 name=roles-jdpiak-2 2022-09-12T20:16:38.622 DEBUG:boto:Method: POST 2022-09-12T20:16:38.623 DEBUG:boto:Path: / 2022-09-12T20:16:38.623 DEBUG:boto:Data: 2022-09-12T20:16:38.624 DEBUG:boto:Headers: {} 2022-09-12T20:16:38.624 DEBUG:boto:Host: smithi138.front.sepia.ceph.com 2022-09-12T20:16:38.625 DEBUG:boto:Port: 8000 2022-09-12T20:16:38.626 DEBUG:boto:Params: {'RoleName': 'roles-jdpiak-2', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2022-09-12T20:16:38.627 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T20:16:38.627 DEBUG:boto:Token: None 2022-09-12T20:16:38.628 DEBUG:boto:CanonicalRequest: POST / host:smithi138.front.sepia.ceph.com:8000 x-amz-date:20220912T201638Z host;x-amz-date 4d2a5b5fe1061e86d3272673ed5dbde12aa299af099c2c4b3b8ea0f425fbc7e5 2022-09-12T20:16:38.629 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220912T201638Z 20220912/front/smithi138/aws4_request 30a411dc94cdbb23835d516c592265374e03ff5ad03a2e9c9130a5d71b2f11b0 2022-09-12T20:16:38.629 DEBUG:boto:Signature: 6d1ed41575a68ea188f235e365d152bae5bb14e62721a00e20ada7493dda4682 2022-09-12T20:16:38.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220912T201638Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=FUCN3RT6SAOJRSRI0ZIX/20220912/front/smithi138/aws4_request,SignedHeaders=host;x-amz-date,Signature=6d1ed41575a68ea188f235e365d152bae5bb14e62721a00e20ada7493dda4682', 'Host': 'smithi138.front.sepia.ceph.com:8000'} 2022-09-12T20:17:07.082 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f9edf0d30107687d-00631f93a6-1a4f-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '731'), ('Date', 'Mon, 12 Sep 2022 20:17:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:17:07.083 DEBUG:boto:b'a8613d90-ee66-435f-a686-60e0bcd19f1droles-jdpiak-2/arn:aws:iam:::role/roles-jdpiak-22022-09-12T20:16:38.644Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx00000f9edf0d30107687d-00631f93a6-1a4f-test-zone2' 2022-09-12T20:17:07.085 INFO:rgw_multi.tests:create role zone=test-zone3 name=roles-jdpiak-3 2022-09-12T20:17:07.085 DEBUG:boto:Method: POST 2022-09-12T20:17:07.086 DEBUG:boto:Path: / 2022-09-12T20:17:07.086 DEBUG:boto:Data: 2022-09-12T20:17:07.087 DEBUG:boto:Headers: {} 2022-09-12T20:17:07.087 DEBUG:boto:Host: smithi035.front.sepia.ceph.com 2022-09-12T20:17:07.088 DEBUG:boto:Port: 8001 2022-09-12T20:17:07.088 DEBUG:boto:Params: {'RoleName': 'roles-jdpiak-3', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2022-09-12T20:17:07.089 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T20:17:07.089 DEBUG:boto:Token: None 2022-09-12T20:17:07.090 DEBUG:boto:CanonicalRequest: POST / host:smithi035.front.sepia.ceph.com:8001 x-amz-date:20220912T201707Z host;x-amz-date 9445b939f1493239895a08b44e227ba0234807f4979ef8007cb21c17d98cbba4 2022-09-12T20:17:07.091 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220912T201707Z 20220912/front/smithi035/aws4_request 5825bb944510bdbcf2d56bd313aded070aafbb398dfbfb1c97145f6cf6e5c9f5 2022-09-12T20:17:07.091 DEBUG:boto:Signature: 159f0121f53f4619a8f6a5a3ff127023f2139d3ad9a37b70d7b825c4a3c34626 2022-09-12T20:17:07.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220912T201707Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=FUCN3RT6SAOJRSRI0ZIX/20220912/front/smithi035/aws4_request,SignedHeaders=host;x-amz-date,Signature=159f0121f53f4619a8f6a5a3ff127023f2139d3ad9a37b70d7b825c4a3c34626', 'Host': 'smithi035.front.sepia.ceph.com:8001'} 2022-09-12T20:17:07.114 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006a4e097329664986-00631f93c3-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '731'), ('Date', 'Mon, 12 Sep 2022 20:17:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:17:07.116 DEBUG:boto:b'a6d8cb45-8742-400c-8567-58f7f2ea3702roles-jdpiak-3/arn:aws:iam:::role/roles-jdpiak-32022-09-12T20:17:07.108Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx000006a4e097329664986-00631f93c3-1826-test-zone3' 2022-09-12T20:17:07.118 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:17:07.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:17:07.120 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:17:07.178 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:17:07.179 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:17:07.216 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=38ca1f57-1358-496c-a99b-df49ee365a4f 2022-09-12T20:17:07.230 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T20:17:07.230 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:17:07.231 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T20:17:07.231 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T20:17:07.231 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:17:07.231 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:17:07.232 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T20:17:07.232 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T20:17:07.232 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:17:07.233 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:17:07.233 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663013827.102653_882.1", 2022-09-12T20:17:07.233 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:17:07.102653Z" 2022-09-12T20:17:07.234 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:17:07.234 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:17:07.234 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:17:07.235 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:16:38.638998Z" 2022-09-12T20:17:07.235 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:17:07.236 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T20:17:07.238 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:17:07.239 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T20:17:07.239 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:17:07.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:17:07.241 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:17:07.298 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:17:07.299 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:17:07.336 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:17:07.334+0000 7fec2f784600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:17:07.338 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:17:07.335+0000 7fec2f784600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:17:07.356 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:17:07.357 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:17:07.357 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:17:07.358 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:17:07.358 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:17:07.358 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:17:07.358 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:17:07.359 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:07.359 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:17:07.360 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:07.360 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:17:07.360 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:07.360 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:07.361 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:17:07.361 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:07.361 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:17:07.361 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:07.362 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:17:07.362 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:07.362 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:07.362 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:07.363 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:07.363 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:17:07.363 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:07.364 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:07.364 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:17:07.364 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:07.365 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:07.365 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:07.366 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:17:07.366 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:07.367 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:07.367 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:07.368 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:07.368 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:17:07.369 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:07.369 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:07.370 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013666.904810_865.1", 2022-09-12T20:17:07.370 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:07.371 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:07.372 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:07.372 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:14:26.904810Z", 2022-09-12T20:17:07.372 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:17:07.373 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:07.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:07.374 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:07.375 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:17:07.375 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:07.375 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:07.376 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T20:17:07.376 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:07.377 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:07.377 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:07.377 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T20:17:07.378 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:07.378 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:07.378 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:07.378 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:17:07.379 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:07.379 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:17:07.380 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:17:07.380 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:17:07.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:07.381 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:17:07Z" 2022-09-12T20:17:07.382 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:17:07.383 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663013666.904810_865.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:14:26.904810Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:17:07Z" } 2022-09-12T20:17:07.384 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663013666.904810_865.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:14:26.904810Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T20:17:07.385 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:17:07.385 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663013666.904810_865.1', 3: ''} 2022-09-12T20:17:07.386 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663013827.102653_882.1 target=1_1663013666.904810_865.1, shard=3 master=1_1663013798.638998_440.1 target= 2022-09-12T20:17:12.387 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:17:12.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:17:12.389 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:17:12.446 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:17:12.447 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:17:12.491 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:17:12.490+0000 7f9152b76600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:17:12.492 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:17:12.491+0000 7f9152b76600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:17:12.506 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:17:12.506 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:17:12.507 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:17:12.507 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:17:12.507 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:17:12.508 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:17:12.508 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:17:12.508 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:12.508 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:17:12.509 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:12.509 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:17:12.509 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:12.509 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:12.510 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:17:12.510 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:12.510 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:17:12.510 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:12.511 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:17:12.511 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:12.511 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:12.512 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:12.512 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:12.512 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:17:12.512 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:12.513 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:12.513 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:17:12.513 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:12.513 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:12.514 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:12.514 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:17:12.514 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:12.514 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:12.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:12.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:12.515 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:17:12.515 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:12.516 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:12.516 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013666.904810_865.1", 2022-09-12T20:17:12.516 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:12.517 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:12.517 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:12.517 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:14:26.904810Z", 2022-09-12T20:17:12.518 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:17:12.518 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:12.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:12.518 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:12.519 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:17:12.519 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:12.519 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:12.520 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T20:17:12.520 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:12.520 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:12.520 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:12.521 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T20:17:12.521 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:12.521 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:12.521 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:12.522 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:17:12.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:12.522 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:17:12.522 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:17:12.523 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:17:12.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:12.523 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:17:12Z" 2022-09-12T20:17:12.523 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:17:12.525 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663013666.904810_865.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:14:26.904810Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:17:12Z" } 2022-09-12T20:17:12.525 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663013666.904810_865.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:14:26.904810Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T20:17:12.526 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:17:12.526 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663013666.904810_865.1', 3: ''} 2022-09-12T20:17:12.527 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663013827.102653_882.1 target=1_1663013666.904810_865.1, shard=3 master=1_1663013798.638998_440.1 target= 2022-09-12T20:17:17.529 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:17:17.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:17:17.531 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:17:17.589 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:17:17.589 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:17:17.631 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:17:17.629+0000 7fd800d8b600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:17:17.632 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:17:17.630+0000 7fd800d8b600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:17:17.652 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:17:17.652 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:17:17.653 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:17:17.653 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:17:17.653 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:17:17.653 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:17:17.654 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:17:17.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:17.654 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:17:17.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:17.655 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:17:17.655 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:17.655 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:17.656 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:17:17.656 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:17.656 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:17:17.656 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:17.657 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:17:17.657 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:17.657 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:17.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:17.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:17.658 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:17:17.659 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:17.659 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:17.659 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:17:17.659 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:17.660 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:17.660 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:17.660 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:17:17.660 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:17.661 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:17.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:17.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:17.661 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:17:17.662 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:17.662 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:17.662 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013666.904810_865.1", 2022-09-12T20:17:17.663 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:17.663 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:17.663 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:17.663 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:14:26.904810Z", 2022-09-12T20:17:17.664 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:17:17.664 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:17.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:17.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:17.666 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:17:17.666 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:17.667 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:17.667 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T20:17:17.668 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:17.668 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:17.669 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:17.669 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T20:17:17.669 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:17.670 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:17.670 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:17.671 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:17:17.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:17.672 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:17:17.672 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:17:17.673 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:17:17.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:17.674 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:17:17Z" 2022-09-12T20:17:17.674 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:17:17.675 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663013666.904810_865.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:14:26.904810Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:17:17Z" } 2022-09-12T20:17:17.677 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663013666.904810_865.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:14:26.904810Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T20:17:17.678 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:17:17.678 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663013666.904810_865.1', 3: ''} 2022-09-12T20:17:17.679 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663013827.102653_882.1 target=1_1663013666.904810_865.1, shard=3 master=1_1663013798.638998_440.1 target= 2022-09-12T20:17:22.680 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:17:22.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:17:22.682 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:17:22.739 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:17:22.739 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:17:22.784 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:17:22.782+0000 7fc5b7750600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:17:22.785 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:17:22.783+0000 7fc5b7750600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:17:22.800 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:17:22.801 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:17:22.801 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:17:22.802 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:17:22.802 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:17:22.802 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:17:22.803 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:17:22.803 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:22.803 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:17:22.803 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:22.804 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:17:22.804 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:22.804 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:22.804 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:17:22.805 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:22.805 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:17:22.805 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:22.805 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:17:22.806 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:22.806 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:22.806 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:22.806 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:22.807 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:17:22.807 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:22.807 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:22.807 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:17:22.807 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:22.808 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:22.808 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:22.808 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:17:22.808 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:22.809 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:22.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:22.809 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:22.810 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:17:22.810 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:22.811 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:22.811 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013666.904810_865.1", 2022-09-12T20:17:22.811 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:22.811 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:22.812 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:22.812 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:14:26.904810Z", 2022-09-12T20:17:22.812 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:17:22.812 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:22.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:22.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:22.813 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:17:22.813 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:22.814 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:22.814 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T20:17:22.814 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:22.814 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:22.815 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:22.815 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T20:17:22.815 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:22.816 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:22.816 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:22.816 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:17:22.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:22.817 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:17:22.817 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:17:22.818 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:17:22.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:22.818 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:17:22Z" 2022-09-12T20:17:22.818 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:17:22.820 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663013666.904810_865.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:14:26.904810Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:17:22Z" } 2022-09-12T20:17:22.821 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663013666.904810_865.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:14:26.904810Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T20:17:22.821 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:17:22.822 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663013666.904810_865.1', 3: ''} 2022-09-12T20:17:22.822 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663013827.102653_882.1 target=1_1663013666.904810_865.1, shard=3 master=1_1663013798.638998_440.1 target= 2022-09-12T20:17:27.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:17:27.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:17:27.825 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:17:27.882 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:17:27.882 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:17:27.915 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:17:27.913+0000 7f2fb3daf600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:17:27.916 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:17:27.914+0000 7f2fb3daf600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:17:27.930 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:17:27.930 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:17:27.931 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:17:27.931 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:17:27.931 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:17:27.932 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:17:27.932 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:17:27.932 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:27.933 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:17:27.933 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:27.933 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:17:27.934 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:27.934 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:27.935 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:17:27.935 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:27.936 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:17:27.936 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:27.937 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:17:27.937 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:27.937 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:27.937 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:27.938 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:27.938 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:17:27.938 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:27.938 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:27.939 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:17:27.939 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:27.939 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:27.939 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:27.940 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:17:27.940 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:27.940 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:27.940 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:27.941 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:27.941 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:17:27.941 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:27.942 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:27.942 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013666.904810_865.1", 2022-09-12T20:17:27.942 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:27.942 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:27.943 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:27.943 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:14:26.904810Z", 2022-09-12T20:17:27.943 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:17:27.943 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:27.944 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:27.944 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:27.944 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:17:27.945 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:27.945 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:27.946 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T20:17:27.946 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:27.946 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:27.947 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:27.947 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T20:17:27.947 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:27.948 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:27.948 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:27.949 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:17:27.949 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:27.949 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:17:27.950 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:17:27.950 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:17:27.950 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:27.951 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:17:27Z" 2022-09-12T20:17:27.951 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:17:27.952 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663013666.904810_865.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:14:26.904810Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:17:27Z" } 2022-09-12T20:17:27.953 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663013666.904810_865.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:14:26.904810Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T20:17:27.953 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:17:27.954 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663013666.904810_865.1', 3: ''} 2022-09-12T20:17:27.955 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663013827.102653_882.1 target=1_1663013666.904810_865.1, shard=3 master=1_1663013798.638998_440.1 target= 2022-09-12T20:17:32.956 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:17:32.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:17:32.957 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:17:33.016 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:17:33.017 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:17:33.061 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:17:33.059+0000 7f1f79b02600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:17:33.061 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:17:33.060+0000 7f1f79b02600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:17:33.080 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:17:33.080 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:17:33.081 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:17:33.081 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:17:33.081 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:17:33.081 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:17:33.082 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:17:33.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:33.082 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:17:33.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:33.083 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:17:33.083 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:33.084 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:33.084 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:17:33.084 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:33.084 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:17:33.085 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:33.085 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:17:33.085 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:33.085 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:33.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:33.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:33.086 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:17:33.086 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:33.087 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:33.087 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:17:33.087 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:33.088 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:33.088 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:33.089 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:17:33.089 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:33.090 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:33.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:33.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:33.091 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:17:33.092 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:33.092 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:33.092 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013666.904810_865.1", 2022-09-12T20:17:33.093 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:33.093 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:33.093 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:33.094 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:14:26.904810Z", 2022-09-12T20:17:33.094 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:17:33.094 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:33.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:33.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:33.095 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:17:33.095 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:33.095 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:33.096 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T20:17:33.096 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:33.096 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:33.096 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:33.097 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T20:17:33.097 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:33.097 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:33.098 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:33.098 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:17:33.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:33.099 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:17:33.099 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:17:33.099 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:17:33.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:33.100 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:17:33Z" 2022-09-12T20:17:33.100 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:17:33.101 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663013666.904810_865.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:14:26.904810Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:17:33Z" } 2022-09-12T20:17:33.102 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663013666.904810_865.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:14:26.904810Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T20:17:33.102 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:17:33.103 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663013666.904810_865.1', 3: ''} 2022-09-12T20:17:33.103 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663013827.102653_882.1 target=1_1663013666.904810_865.1, shard=3 master=1_1663013798.638998_440.1 target= 2022-09-12T20:17:38.104 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:17:38.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:17:38.106 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:17:38.164 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:17:38.165 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:17:38.198 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:17:38.196+0000 7fe05428a600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:17:38.198 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:17:38.197+0000 7fe05428a600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:17:38.214 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:17:38.214 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:17:38.215 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:17:38.215 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:17:38.216 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:17:38.216 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:17:38.216 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:17:38.217 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:38.217 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:17:38.217 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:38.218 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:17:38.218 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:38.218 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:38.219 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:17:38.219 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:38.219 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:17:38.219 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:38.220 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:17:38.220 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:38.220 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:38.221 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:38.221 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:38.221 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:17:38.222 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:38.222 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:38.222 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:17:38.222 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:38.223 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:38.223 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:38.223 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:17:38.224 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:38.224 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:38.224 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:38.224 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:38.225 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:17:38.225 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:38.226 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:38.226 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013827.098611_881.1", 2022-09-12T20:17:38.226 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:38.226 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:38.227 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:38.227 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:17:07.098611Z", 2022-09-12T20:17:38.227 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:17:38.228 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:38.228 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:38.228 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:38.228 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:17:38.229 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:38.229 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:38.229 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T20:17:38.229 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:38.230 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:38.230 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:38.230 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T20:17:38.231 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:38.231 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:38.231 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:38.231 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:17:38.232 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:38.232 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:17:38.232 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:17:38.232 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:17:38.233 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:38.233 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:17:38Z" 2022-09-12T20:17:38.233 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:17:38.235 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663013827.098611_881.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:17:07.098611Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:17:38Z" } 2022-09-12T20:17:38.236 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663013827.098611_881.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:17:07.098611Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T20:17:38.236 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:17:38.237 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663013827.098611_881.1', 3: ''} 2022-09-12T20:17:38.237 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663013827.102653_882.1 target=1_1663013827.098611_881.1, shard=3 master=1_1663013798.638998_440.1 target= 2022-09-12T20:17:43.239 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:17:43.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:17:43.241 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:17:43.298 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:17:43.299 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:17:43.341 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:17:43.339+0000 7f579b48f600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:17:43.342 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:17:43.340+0000 7f579b48f600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:17:43.358 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:17:43.359 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:17:43.359 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:17:43.359 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:17:43.359 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:17:43.360 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:17:43.360 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:17:43.360 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:43.360 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:17:43.361 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:43.361 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:17:43.361 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:43.361 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:43.362 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:17:43.362 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:43.362 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:17:43.362 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:43.363 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:17:43.363 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:43.363 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:43.364 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:43.364 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:43.364 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:17:43.364 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:43.365 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:43.365 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:17:43.365 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:43.365 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:43.366 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:43.366 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:17:43.366 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:43.366 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:43.367 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:43.367 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:43.367 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:17:43.367 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:43.368 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:43.368 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013827.102653_882.1", 2022-09-12T20:17:43.368 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:43.368 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:43.369 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:43.369 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:17:07.102653Z", 2022-09-12T20:17:43.369 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:17:43.369 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:43.370 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:43.370 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:43.370 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:17:43.370 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:43.371 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:43.371 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T20:17:43.371 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:43.371 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:43.372 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:43.372 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T20:17:43.372 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:43.372 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:43.373 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:43.373 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:17:43.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:43.373 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:17:43.374 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:17:43.374 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:17:43.374 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:43.375 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:17:43Z" 2022-09-12T20:17:43.375 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:17:43.376 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663013827.102653_882.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:17:07.102653Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:17:43Z" } 2022-09-12T20:17:43.377 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663013827.102653_882.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:17:07.102653Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T20:17:43.377 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:17:43.378 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: ''} 2022-09-12T20:17:43.378 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1663013798.638998_440.1 target= 2022-09-12T20:17:48.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:17:48.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:17:48.382 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:17:48.448 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:17:48.449 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:17:48.484 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:17:48.481+0000 7fa7437c7600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:17:48.485 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:17:48.482+0000 7fa7437c7600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:17:48.500 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:17:48.501 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:17:48.502 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:17:48.502 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:17:48.503 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:17:48.503 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:17:48.503 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:17:48.504 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:48.504 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:17:48.504 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:48.504 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:17:48.505 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:48.505 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:48.506 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:17:48.507 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:48.507 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:17:48.507 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:48.507 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:17:48.508 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:48.508 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:48.509 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:48.509 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:48.509 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:17:48.510 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:48.510 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:48.511 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:17:48.511 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:48.512 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:48.512 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:48.513 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:17:48.513 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:48.514 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:48.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:48.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:48.515 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:17:48.515 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:48.515 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:48.516 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013827.102653_882.1", 2022-09-12T20:17:48.516 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:48.517 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:48.517 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:48.517 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:17:07.102653Z", 2022-09-12T20:17:48.518 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:17:48.518 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:48.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:48.519 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:48.519 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:17:48.520 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:48.520 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:48.520 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T20:17:48.521 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:48.521 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:48.522 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:48.522 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T20:17:48.523 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:48.523 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:48.523 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:48.524 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:17:48.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:48.525 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:17:48.525 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:17:48.525 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:17:48.526 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:48.526 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:17:48Z" 2022-09-12T20:17:48.527 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:17:48.527 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663013827.102653_882.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:17:07.102653Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:17:48Z" } 2022-09-12T20:17:48.529 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663013827.102653_882.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:17:07.102653Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T20:17:48.529 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:17:48.530 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: ''} 2022-09-12T20:17:48.531 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1663013798.638998_440.1 target= 2022-09-12T20:17:53.537 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:17:53.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:17:53.539 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:17:53.596 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:17:53.597 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:17:53.639 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:17:53.636+0000 7fd2c427c600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:17:53.639 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:17:53.637+0000 7fd2c427c600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:17:53.658 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:17:53.659 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:17:53.659 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:17:53.659 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:17:53.659 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:17:53.660 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:17:53.660 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:17:53.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:53.660 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:17:53.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:53.661 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:17:53.661 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:53.661 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:53.662 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:17:53.662 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:53.662 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:17:53.663 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:53.663 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:17:53.663 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:53.663 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:53.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:53.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:53.664 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:17:53.664 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:53.665 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:53.665 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:17:53.665 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:53.665 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:53.666 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:53.666 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:17:53.666 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:53.667 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:53.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:53.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:53.667 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:17:53.668 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:53.668 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:53.668 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013827.102653_882.1", 2022-09-12T20:17:53.668 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:53.669 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:53.669 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:53.669 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:17:07.102653Z", 2022-09-12T20:17:53.669 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:17:53.670 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:53.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:53.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:53.670 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:17:53.671 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:53.671 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:53.671 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T20:17:53.671 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:53.672 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:53.672 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:53.672 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T20:17:53.672 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:53.673 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:53.673 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:53.673 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:17:53.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:53.674 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:17:53.674 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:17:53.674 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:17:53.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:53.675 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:17:53Z" 2022-09-12T20:17:53.675 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:17:53.676 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663013827.102653_882.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:17:07.102653Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:17:53Z" } 2022-09-12T20:17:53.677 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663013827.102653_882.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:17:07.102653Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T20:17:53.677 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:17:53.678 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: ''} 2022-09-12T20:17:53.678 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1663013798.638998_440.1 target= 2022-09-12T20:17:58.679 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:17:58.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:17:58.681 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:17:58.741 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:17:58.741 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:17:58.778 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:17:58.776+0000 7fbbe7941600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:17:58.779 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:17:58.777+0000 7fbbe7941600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:17:58.793 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:17:58.794 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:17:58.794 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:17:58.795 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:17:58.795 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:17:58.796 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:17:58.796 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:17:58.797 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:58.797 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:17:58.798 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:58.798 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:17:58.799 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:58.799 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:58.800 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:17:58.800 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:58.801 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:17:58.801 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:58.802 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:17:58.802 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:58.802 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:58.803 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:58.803 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:58.803 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:17:58.804 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:58.804 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:58.804 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:17:58.805 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:58.806 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:58.806 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:58.807 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:17:58.807 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:58.808 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:58.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:58.809 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:58.809 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:17:58.810 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:58.810 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:58.811 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013827.102653_882.1", 2022-09-12T20:17:58.811 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:58.812 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:58.812 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:58.813 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:17:07.102653Z", 2022-09-12T20:17:58.813 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:17:58.814 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:58.814 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:58.815 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:17:58.815 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:17:58.816 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:17:58.816 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:17:58.816 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T20:17:58.816 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:17:58.817 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:17:58.817 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:17:58.817 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T20:17:58.818 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:17:58.818 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:58.819 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:17:58.820 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:17:58.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:58.820 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:17:58.821 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:17:58.822 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:17:58.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:17:58.823 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:17:58Z" 2022-09-12T20:17:58.823 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:17:58.825 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663013827.102653_882.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:17:07.102653Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:17:58Z" } 2022-09-12T20:17:58.827 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663013827.102653_882.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:17:07.102653Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T20:17:58.828 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:17:58.829 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: ''} 2022-09-12T20:17:58.829 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1663013798.638998_440.1 target= 2022-09-12T20:18:03.831 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:18:03.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:18:03.833 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:18:03.891 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:18:03.891 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:18:03.922 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:18:03.920+0000 7f55d9615600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:18:03.924 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:18:03.922+0000 7f55d9615600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:18:03.941 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:18:03.942 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:18:03.942 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:18:03.943 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:18:03.943 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:18:03.943 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:18:03.943 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:18:03.944 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:03.944 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:18:03.944 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:03.944 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:18:03.945 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:03.945 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:03.945 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:18:03.945 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:03.946 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:18:03.946 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:03.946 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:18:03.947 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:18:03.947 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:03.947 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:03.947 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:03.948 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:18:03.948 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:03.948 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:03.948 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:18:03.949 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:03.949 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:03.949 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:03.949 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:18:03.950 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:18:03.950 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:03.951 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:03.951 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:03.951 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:18:03.952 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:03.952 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:03.952 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013827.102653_882.1", 2022-09-12T20:18:03.952 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:03.953 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:03.953 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:03.953 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:17:07.102653Z", 2022-09-12T20:18:03.953 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:18:03.954 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:03.954 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:03.954 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:03.955 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:18:03.955 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:03.955 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:03.955 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T20:18:03.956 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:03.956 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:03.956 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:03.956 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T20:18:03.957 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:18:03.957 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:03.957 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:03.958 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:18:03.958 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:03.958 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:18:03.958 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:18:03.959 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:18:03.959 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:03.959 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:18:03Z" 2022-09-12T20:18:03.959 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:18:03.961 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663013827.102653_882.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:17:07.102653Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:18:03Z" } 2022-09-12T20:18:03.962 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663013827.102653_882.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:17:07.102653Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T20:18:03.962 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:18:03.963 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: ''} 2022-09-12T20:18:03.963 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1663013798.638998_440.1 target= 2022-09-12T20:18:08.964 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:18:08.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:18:08.966 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:18:09.025 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:18:09.058 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:18:09.069 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:18:09.067+0000 7fe8e7c58600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:18:09.070 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:18:09.068+0000 7fe8e7c58600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:18:09.086 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:18:09.087 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:18:09.087 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:18:09.087 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:18:09.088 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:18:09.088 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:18:09.088 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:18:09.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:09.089 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:18:09.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:09.089 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:18:09.089 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:09.090 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:09.090 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:18:09.090 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:09.090 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:18:09.091 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:09.091 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:18:09.091 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:18:09.092 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:09.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:09.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:09.093 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:18:09.094 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:09.094 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:09.094 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:18:09.095 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:09.095 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:09.096 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:09.096 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:18:09.097 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:18:09.097 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:09.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:09.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:09.098 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:18:09.098 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:09.099 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:09.099 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013827.102653_882.1", 2022-09-12T20:18:09.099 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:09.099 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:09.100 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:09.100 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:17:07.102653Z", 2022-09-12T20:18:09.100 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:18:09.100 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:09.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:09.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:09.101 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:18:09.101 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:09.101 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:09.102 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T20:18:09.102 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:09.102 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:09.102 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:09.103 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T20:18:09.103 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:18:09.103 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:09.104 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:09.104 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:18:09.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:09.104 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:18:09.105 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:18:09.105 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:18:09.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:09.106 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:18:09Z" 2022-09-12T20:18:09.106 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:18:09.107 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663013827.102653_882.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:17:07.102653Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:18:09Z" } 2022-09-12T20:18:09.108 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663013827.102653_882.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:17:07.102653Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T20:18:09.108 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:18:09.109 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: ''} 2022-09-12T20:18:09.109 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1663013798.638998_440.1 target= 2022-09-12T20:18:14.111 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:18:14.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:18:14.112 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:18:14.172 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:18:14.172 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:18:14.203 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:18:14.201+0000 7f26577bf600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:18:14.204 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:18:14.202+0000 7f26577bf600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:18:14.218 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:18:14.219 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:18:14.219 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:18:14.219 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:18:14.220 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:18:14.220 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:18:14.220 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:18:14.221 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:14.221 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:18:14.221 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:14.222 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:18:14.222 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:14.223 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:14.223 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:18:14.223 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:14.223 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:18:14.224 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:14.224 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:18:14.225 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:18:14.225 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:14.225 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:14.225 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:14.226 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:18:14.226 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:14.226 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:14.227 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:18:14.227 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:14.227 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:14.227 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:14.228 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:18:14.228 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:18:14.228 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:14.229 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:14.229 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:14.229 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:18:14.230 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:14.230 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:14.230 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013827.102653_882.1", 2022-09-12T20:18:14.231 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:14.231 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:14.231 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:14.232 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:17:07.102653Z", 2022-09-12T20:18:14.232 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:18:14.232 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:14.232 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:14.233 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:14.233 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:18:14.233 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:14.233 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:14.234 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008580.904048_86.1", 2022-09-12T20:18:14.234 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:14.234 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:14.235 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:14.235 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:49:40.904048Z", 2022-09-12T20:18:14.235 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:18:14.235 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:14.236 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:14.236 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:18:14.236 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:14.236 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:18:14.237 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:18:14.237 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:18:14.237 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:14.238 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:18:14Z" 2022-09-12T20:18:14.238 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:18:14.239 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663013827.102653_882.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:17:07.102653Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663008580.904048_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T18:49:40.904048Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:18:14Z" } 2022-09-12T20:18:14.240 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663013827.102653_882.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:17:07.102653Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663008580.904048_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T18:49:40.904048Z', 'realm_epoch': 2}}] 2022-09-12T20:18:14.241 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:18:14.241 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: ''} 2022-09-12T20:18:14.242 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1663013798.638998_440.1 target= 2022-09-12T20:18:19.243 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:18:19.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:18:19.245 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:18:19.303 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:18:19.304 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:18:19.346 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:18:19.341+0000 7f10f2755600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:18:19.347 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:18:19.342+0000 7f10f2755600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:18:19.367 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:18:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:18:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:18:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:18:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:18:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:18:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:18:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:18:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:19.372 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:18:19.372 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:19.373 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:19.374 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:18:19.374 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:19.375 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:18:19.375 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:19.376 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:18:19.376 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:18:19.376 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:19.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:19.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:19.377 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:18:19.378 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:19.378 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:19.379 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:18:19.379 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:19.379 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:19.380 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:19.380 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:18:19.381 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:18:19.381 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:19.382 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:19.382 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:19.383 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:18:19.384 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:19.384 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:19.384 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013827.102653_882.1", 2022-09-12T20:18:19.385 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:19.385 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:19.385 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:19.386 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:17:07.102653Z", 2022-09-12T20:18:19.386 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:18:19.386 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:18:19.388 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:19.388 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:19.388 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013798.610384_437.1", 2022-09-12T20:18:19.388 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:19.389 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:19.389 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:19.390 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:16:38.610384Z", 2022-09-12T20:18:19.390 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:18:19.391 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:19.391 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:19.392 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:18:19.392 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:19.393 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:18:19.394 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:18:19.395 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:18:19.396 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:19.396 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:18:19Z" 2022-09-12T20:18:19.397 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:18:19.399 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663013827.102653_882.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:17:07.102653Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.610384_437.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.610384Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:18:19Z" } 2022-09-12T20:18:19.400 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663013827.102653_882.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:17:07.102653Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.610384_437.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.610384Z', 'realm_epoch': 4}}] 2022-09-12T20:18:19.401 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:18:19.401 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: '1_1663013798.610384_437.1'} 2022-09-12T20:18:19.402 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1663013798.638998_440.1 target=1_1663013798.610384_437.1 2022-09-12T20:18:24.403 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:18:24.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:18:24.405 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:18:24.462 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:18:24.463 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:18:24.498 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:18:24.496+0000 7f9f823f3600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:18:24.499 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:18:24.498+0000 7f9f823f3600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:18:24.521 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:18:24.521 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:18:24.521 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:18:24.522 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:18:24.522 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:18:24.522 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:18:24.523 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:18:24.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:24.523 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:18:24.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:24.524 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:18:24.524 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:24.525 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:24.525 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:18:24.526 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:24.526 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:18:24.526 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:24.527 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:18:24.527 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:18:24.527 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:24.527 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:24.528 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:24.528 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:18:24.528 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:24.528 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:24.529 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:18:24.529 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:24.529 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:24.529 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:24.530 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:18:24.530 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:18:24.530 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:24.530 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:24.531 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:24.531 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:18:24.531 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:24.531 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:24.532 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013827.102653_882.1", 2022-09-12T20:18:24.532 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:24.532 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:24.532 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:24.533 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:17:07.102653Z", 2022-09-12T20:18:24.533 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:18:24.533 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:24.534 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:24.534 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:24.535 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:18:24.535 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:24.536 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:24.536 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:18:24.536 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:24.537 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:24.537 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:24.537 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:18:24.537 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:18:24.538 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:24.538 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:24.538 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:18:24.538 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:24.539 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:18:24.539 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:18:24.539 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:18:24.539 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:24.540 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:18:24Z" 2022-09-12T20:18:24.540 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:18:24.541 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663013827.102653_882.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:17:07.102653Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:18:24Z" } 2022-09-12T20:18:24.542 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663013827.102653_882.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:17:07.102653Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:18:24.542 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:18:24.543 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:18:24.544 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T20:18:24.545 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T20:18:24.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:18:24.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:18:24.547 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:18:24.605 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:18:24.605 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:18:24.638 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:18:24.638+0000 7f9ed9bb2600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:18:24.639 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:18:24.639+0000 7f9ed9bb2600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:18:24.654 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T20:18:24.654 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T20:18:24.654 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T20:18:24.654 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T20:18:24.655 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T20:18:24.655 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:18:24.655 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T20:18:24.655 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:18:24.656 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T20:18:24.656 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:18:24.656 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T20:18:24.656 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:18:24.657 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T20:18:24.657 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:18:24.657 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:18:24.657 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T20:18:24.658 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:18:24.658 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:18:24.658 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T20:18:24.658 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:18:24.659 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:18:24.659 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:18:24.659 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T20:18:24.660 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:18:24.660 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T20:18:24.660 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T20:18:24.660 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:18:24.661 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:18:24.661 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:18:24.661 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T20:18:24.661 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T20:18:24.662 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:18:24.662 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:18:24.662 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:18:24.662 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T20:18:24.663 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:18:24.663 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T20:18:24.663 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663013827.102653_882.1", 2022-09-12T20:18:24.664 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:18:24.664 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:18:24.665 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:18:24.665 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T20:17:07.102653Z", 2022-09-12T20:18:24.665 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T20:18:24.666 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:18:24.666 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:18:24.666 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:18:24.666 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T20:18:24.667 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:18:24.667 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T20:18:24.667 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:18:24.667 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:18:24.668 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:18:24.668 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:18:24.668 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:18:24.668 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T20:18:24.669 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:18:24.669 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:18:24.669 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T20:18:24.669 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:18:24.670 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T20:18:24.670 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T20:18:24.670 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T20:18:24.670 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:18:24.671 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T20:18:24Z" 2022-09-12T20:18:24.671 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T20:18:24.672 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663013827.102653_882.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:17:07.102653Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:18:24Z" } 2022-09-12T20:18:24.673 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663013827.102653_882.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:17:07.102653Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:18:24.673 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:18:24.674 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:18:24.674 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T20:18:24.675 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T20:18:24.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:18:24.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:18:24.677 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:18:24.727 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:18:24.728 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:18:24.757 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:18:24.755+0000 7f6e8b515600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:18:24.758 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:18:24.756+0000 7f6e8b515600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:18:24.775 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:18:24.776 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:18:24.776 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:18:24.776 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:18:24.777 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:18:24.777 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:18:24.777 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:18:24.778 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:24.778 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:18:24.778 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:24.779 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:18:24.779 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:24.779 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:24.779 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:18:24.780 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:24.780 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:18:24.780 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:24.781 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:18:24.781 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:18:24.781 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:24.781 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:24.782 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:24.782 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:18:24.782 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:24.782 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:24.783 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:18:24.783 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:24.783 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:24.784 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:24.784 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:18:24.784 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:18:24.784 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:24.785 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:24.785 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:24.785 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:18:24.785 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:24.786 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:24.786 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013827.102653_882.1", 2022-09-12T20:18:24.786 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:24.787 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:24.787 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:24.788 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:17:07.102653Z", 2022-09-12T20:18:24.788 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:18:24.788 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:24.789 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:24.790 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:24.790 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:18:24.791 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:24.791 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:24.791 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:18:24.791 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:24.792 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:24.792 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:24.792 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:18:24.793 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:18:24.793 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:24.793 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:24.794 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:18:24.794 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:24.794 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:18:24.794 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:18:24.795 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:18:24.795 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:24.795 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:18:24Z" 2022-09-12T20:18:24.795 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:18:24.797 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663013827.102653_882.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:17:07.102653Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:18:24Z" } 2022-09-12T20:18:24.797 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663013827.102653_882.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:17:07.102653Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:18:24.798 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:18:24.799 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663013827.102653_882.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:18:24.799 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T20:18:24.800 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-jdpiak-1 zones={test-zone2, test-zone1} 2022-09-12T20:18:24.800 DEBUG:boto:Method: POST 2022-09-12T20:18:24.801 DEBUG:boto:Path: / 2022-09-12T20:18:24.801 DEBUG:boto:Data: 2022-09-12T20:18:24.802 DEBUG:boto:Headers: {} 2022-09-12T20:18:24.803 DEBUG:boto:Host: smithi138.front.sepia.ceph.com 2022-09-12T20:18:24.803 DEBUG:boto:Port: 8000 2022-09-12T20:18:24.804 DEBUG:boto:Params: {'RoleName': 'roles-jdpiak-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2022-09-12T20:18:24.805 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T20:18:24.806 DEBUG:boto:Token: None 2022-09-12T20:18:24.806 DEBUG:boto:CanonicalRequest: POST / host:smithi138.front.sepia.ceph.com:8000 x-amz-date:20220912T201824Z host;x-amz-date f7faac2c8b534039190ef6b746c72ae7f7accccf70fb080b8680f0d821ce527c 2022-09-12T20:18:24.807 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220912T201824Z 20220912/front/smithi138/aws4_request b4d86768420465b3f559c02e16c0da1fa60678cfcbe6fb84d163003717c9d321 2022-09-12T20:18:24.808 DEBUG:boto:Signature: b1f06759a2bd338613ad0e0b696018ab96fe14ada0c0f869b461a7cb1898f5ba 2022-09-12T20:18:24.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220912T201824Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=FUCN3RT6SAOJRSRI0ZIX/20220912/front/smithi138/aws4_request,SignedHeaders=host;x-amz-date,Signature=b1f06759a2bd338613ad0e0b696018ab96fe14ada0c0f869b461a7cb1898f5ba', 'Host': 'smithi138.front.sepia.ceph.com:8000'} 2022-09-12T20:18:24.812 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000051f31dd5988c3ade-00631f9410-1a4f-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Mon, 12 Sep 2022 20:18:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:18:24.813 DEBUG:boto:b'tx0000051f31dd5988c3ade-00631f9410-1a4f-test-zone2a58d672c-a5ab-491a-ad02-9ea7f50aa831roles-jdpiak-1/arn:aws:iam:::role/roles-jdpiak-12022-09-12T20:16:38.610Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2022-09-12T20:18:24.814 DEBUG:boto:Method: POST 2022-09-12T20:18:24.815 DEBUG:boto:Path: / 2022-09-12T20:18:24.815 DEBUG:boto:Data: 2022-09-12T20:18:24.816 DEBUG:boto:Headers: {} 2022-09-12T20:18:24.816 DEBUG:boto:Host: smithi035.front.sepia.ceph.com 2022-09-12T20:18:24.817 DEBUG:boto:Port: 8000 2022-09-12T20:18:24.817 DEBUG:boto:Params: {'RoleName': 'roles-jdpiak-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2022-09-12T20:18:24.818 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T20:18:24.819 DEBUG:boto:Token: None 2022-09-12T20:18:24.819 DEBUG:boto:CanonicalRequest: POST / host:smithi035.front.sepia.ceph.com:8000 x-amz-date:20220912T201824Z host;x-amz-date f7faac2c8b534039190ef6b746c72ae7f7accccf70fb080b8680f0d821ce527c 2022-09-12T20:18:24.820 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220912T201824Z 20220912/front/smithi035/aws4_request 3bcad546466fd9d40290459011e3cb2ad6c8d366413b546fb5f8a00782aa2067 2022-09-12T20:18:24.821 DEBUG:boto:Signature: 872d6b475931a13d86a82bb238b3fdeb5977e6c5edd7f90334cf46f6de1d8881 2022-09-12T20:18:24.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220912T201824Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=FUCN3RT6SAOJRSRI0ZIX/20220912/front/smithi035/aws4_request,SignedHeaders=host;x-amz-date,Signature=872d6b475931a13d86a82bb238b3fdeb5977e6c5edd7f90334cf46f6de1d8881', 'Host': 'smithi035.front.sepia.ceph.com:8000'} 2022-09-12T20:18:24.824 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000af41ec46e4f9627b-00631f9410-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Mon, 12 Sep 2022 20:18:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:18:24.825 DEBUG:boto:b'tx00000af41ec46e4f9627b-00631f9410-15c0-test-zone1a58d672c-a5ab-491a-ad02-9ea7f50aa831roles-jdpiak-1/arn:aws:iam:::role/roles-jdpiak-12022-09-12T20:16:38.610Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2022-09-12T20:18:24.826 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-jdpiak-1 2022-09-12T20:18:24.827 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-jdpiak-1 zones={test-zone2, test-zone1} 2022-09-12T20:18:24.827 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-jdpiak-1 zones={test-zone3, test-zone1} 2022-09-12T20:18:24.828 DEBUG:boto:Method: POST 2022-09-12T20:18:24.828 DEBUG:boto:Path: / 2022-09-12T20:18:24.829 DEBUG:boto:Data: 2022-09-12T20:18:24.829 DEBUG:boto:Headers: {} 2022-09-12T20:18:24.830 DEBUG:boto:Host: smithi035.front.sepia.ceph.com 2022-09-12T20:18:24.830 DEBUG:boto:Port: 8001 2022-09-12T20:18:24.831 DEBUG:boto:Params: {'RoleName': 'roles-jdpiak-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2022-09-12T20:18:24.832 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T20:18:24.832 DEBUG:boto:Token: None 2022-09-12T20:18:24.833 DEBUG:boto:CanonicalRequest: POST / host:smithi035.front.sepia.ceph.com:8001 x-amz-date:20220912T201824Z host;x-amz-date f7faac2c8b534039190ef6b746c72ae7f7accccf70fb080b8680f0d821ce527c 2022-09-12T20:18:24.834 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220912T201824Z 20220912/front/smithi035/aws4_request fd7aa37ce3071210f523244021cdbe83ea3a9cae3ee72f737598e5c301fe9ac8 2022-09-12T20:18:24.834 DEBUG:boto:Signature: 9476769f16c34493204b0b365dbb0eadab1fc778f3c8bd56b2505b02c94cc950 2022-09-12T20:18:24.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220912T201824Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=FUCN3RT6SAOJRSRI0ZIX/20220912/front/smithi035/aws4_request,SignedHeaders=host;x-amz-date,Signature=9476769f16c34493204b0b365dbb0eadab1fc778f3c8bd56b2505b02c94cc950', 'Host': 'smithi035.front.sepia.ceph.com:8001'} 2022-09-12T20:18:24.838 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a10c00c2b1de721a-00631f9410-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Mon, 12 Sep 2022 20:18:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:18:24.839 DEBUG:boto:b'tx00000a10c00c2b1de721a-00631f9410-1826-test-zone3a58d672c-a5ab-491a-ad02-9ea7f50aa831roles-jdpiak-1/arn:aws:iam:::role/roles-jdpiak-12022-09-12T20:16:38.765Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2022-09-12T20:18:24.840 DEBUG:boto:Method: POST 2022-09-12T20:18:24.840 DEBUG:boto:Path: / 2022-09-12T20:18:24.841 DEBUG:boto:Data: 2022-09-12T20:18:24.841 DEBUG:boto:Headers: {} 2022-09-12T20:18:24.842 DEBUG:boto:Host: smithi035.front.sepia.ceph.com 2022-09-12T20:18:24.842 DEBUG:boto:Port: 8000 2022-09-12T20:18:24.843 DEBUG:boto:Params: {'RoleName': 'roles-jdpiak-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2022-09-12T20:18:24.843 DEBUG:boto:Token: None 2022-09-12T20:18:24.844 DEBUG:boto:CanonicalRequest: POST / host:smithi035.front.sepia.ceph.com:8000 x-amz-date:20220912T201824Z host;x-amz-date f7faac2c8b534039190ef6b746c72ae7f7accccf70fb080b8680f0d821ce527c 2022-09-12T20:18:24.845 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220912T201824Z 20220912/front/smithi035/aws4_request 3bcad546466fd9d40290459011e3cb2ad6c8d366413b546fb5f8a00782aa2067 2022-09-12T20:18:24.845 DEBUG:boto:Signature: 872d6b475931a13d86a82bb238b3fdeb5977e6c5edd7f90334cf46f6de1d8881 2022-09-12T20:18:24.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220912T201824Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=FUCN3RT6SAOJRSRI0ZIX/20220912/front/smithi035/aws4_request,SignedHeaders=host;x-amz-date,Signature=872d6b475931a13d86a82bb238b3fdeb5977e6c5edd7f90334cf46f6de1d8881', 'Host': 'smithi035.front.sepia.ceph.com:8000'} 2022-09-12T20:18:24.847 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000047de6180122dff50-00631f9410-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Mon, 12 Sep 2022 20:18:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:18:24.848 DEBUG:boto:b'tx0000047de6180122dff50-00631f9410-15c0-test-zone1a58d672c-a5ab-491a-ad02-9ea7f50aa831roles-jdpiak-1/arn:aws:iam:::role/roles-jdpiak-12022-09-12T20:16:38.610Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2022-09-12T20:18:24.849 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-jdpiak-1 2022-09-12T20:18:24.850 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-jdpiak-1 zones={test-zone3, test-zone1} 2022-09-12T20:18:24.850 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-jdpiak-2 zones={test-zone1, test-zone2} 2022-09-12T20:18:24.851 DEBUG:boto:Method: POST 2022-09-12T20:18:24.851 DEBUG:boto:Path: / 2022-09-12T20:18:24.852 DEBUG:boto:Data: 2022-09-12T20:18:24.853 DEBUG:boto:Headers: {} 2022-09-12T20:18:24.853 DEBUG:boto:Host: smithi035.front.sepia.ceph.com 2022-09-12T20:18:24.854 DEBUG:boto:Port: 8000 2022-09-12T20:18:24.854 DEBUG:boto:Params: {'RoleName': 'roles-jdpiak-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2022-09-12T20:18:24.856 DEBUG:boto:Token: None 2022-09-12T20:18:24.857 DEBUG:boto:CanonicalRequest: POST / host:smithi035.front.sepia.ceph.com:8000 x-amz-date:20220912T201824Z host;x-amz-date 79ee0f15148f75043e14c8d171dd68840aaf705793fa22e32b7239ae27bf5661 2022-09-12T20:18:24.859 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220912T201824Z 20220912/front/smithi035/aws4_request 436e3f24707ed20d499db7667a9690459561d88b95b037efd60a4662d9155531 2022-09-12T20:18:24.860 DEBUG:boto:Signature: e9f7ea50123ca1924880b18cbb5318c8e8a81707817da8d57522110be2ec4e84 2022-09-12T20:18:24.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220912T201824Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=FUCN3RT6SAOJRSRI0ZIX/20220912/front/smithi035/aws4_request,SignedHeaders=host;x-amz-date,Signature=e9f7ea50123ca1924880b18cbb5318c8e8a81707817da8d57522110be2ec4e84', 'Host': 'smithi035.front.sepia.ceph.com:8000'} 2022-09-12T20:18:24.864 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000fce923ec68cd513-00631f9410-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Mon, 12 Sep 2022 20:18:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:18:24.864 DEBUG:boto:b'tx000000fce923ec68cd513-00631f9410-15c0-test-zone1a8613d90-ee66-435f-a686-60e0bcd19f1droles-jdpiak-2/arn:aws:iam:::role/roles-jdpiak-22022-09-12T20:16:38.635Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2022-09-12T20:18:24.865 DEBUG:boto:Method: POST 2022-09-12T20:18:24.866 DEBUG:boto:Path: / 2022-09-12T20:18:24.867 DEBUG:boto:Data: 2022-09-12T20:18:24.867 DEBUG:boto:Headers: {} 2022-09-12T20:18:24.868 DEBUG:boto:Host: smithi138.front.sepia.ceph.com 2022-09-12T20:18:24.869 DEBUG:boto:Port: 8000 2022-09-12T20:18:24.869 DEBUG:boto:Params: {'RoleName': 'roles-jdpiak-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2022-09-12T20:18:24.870 DEBUG:boto:Token: None 2022-09-12T20:18:24.870 DEBUG:boto:CanonicalRequest: POST / host:smithi138.front.sepia.ceph.com:8000 x-amz-date:20220912T201824Z host;x-amz-date 79ee0f15148f75043e14c8d171dd68840aaf705793fa22e32b7239ae27bf5661 2022-09-12T20:18:24.871 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220912T201824Z 20220912/front/smithi138/aws4_request 700af93c90834295abe18fd50f0e1e228041be024581cc65d39bdb9a946151e8 2022-09-12T20:18:24.872 DEBUG:boto:Signature: 76178b4b221f4ab6e1b4f1b086fceecde4063445943151b9c4ed13aca03cd60e 2022-09-12T20:18:24.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220912T201824Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=FUCN3RT6SAOJRSRI0ZIX/20220912/front/smithi138/aws4_request,SignedHeaders=host;x-amz-date,Signature=76178b4b221f4ab6e1b4f1b086fceecde4063445943151b9c4ed13aca03cd60e', 'Host': 'smithi138.front.sepia.ceph.com:8000'} 2022-09-12T20:18:24.874 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000646f2c7febe9ca9d-00631f9410-1a4f-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Mon, 12 Sep 2022 20:18:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:18:24.875 DEBUG:boto:b'tx00000646f2c7febe9ca9d-00631f9410-1a4f-test-zone2a8613d90-ee66-435f-a686-60e0bcd19f1droles-jdpiak-2/arn:aws:iam:::role/roles-jdpiak-22022-09-12T20:16:38.635Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2022-09-12T20:18:24.876 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-jdpiak-2 2022-09-12T20:18:24.877 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-jdpiak-2 zones={test-zone1, test-zone2} 2022-09-12T20:18:24.877 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-jdpiak-2 zones={test-zone3, test-zone2} 2022-09-12T20:18:24.878 DEBUG:boto:Method: POST 2022-09-12T20:18:24.878 DEBUG:boto:Path: / 2022-09-12T20:18:24.879 DEBUG:boto:Data: 2022-09-12T20:18:24.879 DEBUG:boto:Headers: {} 2022-09-12T20:18:24.880 DEBUG:boto:Host: smithi035.front.sepia.ceph.com 2022-09-12T20:18:24.880 DEBUG:boto:Port: 8001 2022-09-12T20:18:24.881 DEBUG:boto:Params: {'RoleName': 'roles-jdpiak-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2022-09-12T20:18:24.882 DEBUG:boto:Token: None 2022-09-12T20:18:24.882 DEBUG:boto:CanonicalRequest: POST / host:smithi035.front.sepia.ceph.com:8001 x-amz-date:20220912T201824Z host;x-amz-date 79ee0f15148f75043e14c8d171dd68840aaf705793fa22e32b7239ae27bf5661 2022-09-12T20:18:24.883 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220912T201824Z 20220912/front/smithi035/aws4_request 4a3f763b615ae84f66fbab4175dd464b0cb6cc44efee8e61d7719b87eac2f97e 2022-09-12T20:18:24.884 DEBUG:boto:Signature: 40f8fe8a945501c846695963fe2201daf3c791a4a41079cc9b3f0de54de739ff 2022-09-12T20:18:24.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220912T201824Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=FUCN3RT6SAOJRSRI0ZIX/20220912/front/smithi035/aws4_request,SignedHeaders=host;x-amz-date,Signature=40f8fe8a945501c846695963fe2201daf3c791a4a41079cc9b3f0de54de739ff', 'Host': 'smithi035.front.sepia.ceph.com:8001'} 2022-09-12T20:18:24.886 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007e9ed62461805e2c-00631f9410-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Mon, 12 Sep 2022 20:18:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:18:24.887 DEBUG:boto:b'tx000007e9ed62461805e2c-00631f9410-1826-test-zone3a8613d90-ee66-435f-a686-60e0bcd19f1droles-jdpiak-2/arn:aws:iam:::role/roles-jdpiak-22022-09-12T20:16:38.725Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2022-09-12T20:18:24.888 DEBUG:boto:Method: POST 2022-09-12T20:18:24.889 DEBUG:boto:Path: / 2022-09-12T20:18:24.889 DEBUG:boto:Data: 2022-09-12T20:18:24.890 DEBUG:boto:Headers: {} 2022-09-12T20:18:24.891 DEBUG:boto:Host: smithi138.front.sepia.ceph.com 2022-09-12T20:18:24.891 DEBUG:boto:Port: 8000 2022-09-12T20:18:24.892 DEBUG:boto:Params: {'RoleName': 'roles-jdpiak-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2022-09-12T20:18:24.892 DEBUG:boto:Token: None 2022-09-12T20:18:24.893 DEBUG:boto:CanonicalRequest: POST / host:smithi138.front.sepia.ceph.com:8000 x-amz-date:20220912T201824Z host;x-amz-date 79ee0f15148f75043e14c8d171dd68840aaf705793fa22e32b7239ae27bf5661 2022-09-12T20:18:24.893 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220912T201824Z 20220912/front/smithi138/aws4_request 700af93c90834295abe18fd50f0e1e228041be024581cc65d39bdb9a946151e8 2022-09-12T20:18:24.894 DEBUG:boto:Signature: 76178b4b221f4ab6e1b4f1b086fceecde4063445943151b9c4ed13aca03cd60e 2022-09-12T20:18:24.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220912T201824Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=FUCN3RT6SAOJRSRI0ZIX/20220912/front/smithi138/aws4_request,SignedHeaders=host;x-amz-date,Signature=76178b4b221f4ab6e1b4f1b086fceecde4063445943151b9c4ed13aca03cd60e', 'Host': 'smithi138.front.sepia.ceph.com:8000'} 2022-09-12T20:18:24.896 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a80e9c720e4a4886-00631f9410-1a4f-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Mon, 12 Sep 2022 20:18:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:18:24.897 DEBUG:boto:b'tx00000a80e9c720e4a4886-00631f9410-1a4f-test-zone2a8613d90-ee66-435f-a686-60e0bcd19f1droles-jdpiak-2/arn:aws:iam:::role/roles-jdpiak-22022-09-12T20:16:38.635Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2022-09-12T20:18:24.898 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-jdpiak-2 2022-09-12T20:18:24.899 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-jdpiak-2 zones={test-zone3, test-zone2} 2022-09-12T20:18:24.899 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-jdpiak-3 zones={test-zone1, test-zone3} 2022-09-12T20:18:24.900 DEBUG:boto:Method: POST 2022-09-12T20:18:24.900 DEBUG:boto:Path: / 2022-09-12T20:18:24.901 DEBUG:boto:Data: 2022-09-12T20:18:24.901 DEBUG:boto:Headers: {} 2022-09-12T20:18:24.902 DEBUG:boto:Host: smithi035.front.sepia.ceph.com 2022-09-12T20:18:24.902 DEBUG:boto:Port: 8000 2022-09-12T20:18:24.903 DEBUG:boto:Params: {'RoleName': 'roles-jdpiak-3', 'Action': 'GetRole', 'Version': '2010-05-08'} 2022-09-12T20:18:24.903 DEBUG:boto:Token: None 2022-09-12T20:18:24.904 DEBUG:boto:CanonicalRequest: POST / host:smithi035.front.sepia.ceph.com:8000 x-amz-date:20220912T201824Z host;x-amz-date 168023e9ce2613828f46a5c510c5e419a25661f70078e18340b559469b9d5fbe 2022-09-12T20:18:24.905 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220912T201824Z 20220912/front/smithi035/aws4_request bcfedbfa8f41a6c23a91d4259486efd74ad4998c740a3ba9a220393a24ad4fef 2022-09-12T20:18:24.905 DEBUG:boto:Signature: 4e0869f3820bea6444e0213ec59ac3c895766a96ca82640d583925e1078a04d0 2022-09-12T20:18:24.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220912T201824Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=FUCN3RT6SAOJRSRI0ZIX/20220912/front/smithi035/aws4_request,SignedHeaders=host;x-amz-date,Signature=4e0869f3820bea6444e0213ec59ac3c895766a96ca82640d583925e1078a04d0', 'Host': 'smithi035.front.sepia.ceph.com:8000'} 2022-09-12T20:18:24.908 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002482217da4c18f85-00631f9410-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '718'), ('Date', 'Mon, 12 Sep 2022 20:18:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:18:24.908 DEBUG:boto:b'tx000002482217da4c18f85-00631f9410-15c0-test-zone1a6d8cb45-8742-400c-8567-58f7f2ea3702roles-jdpiak-3/arn:aws:iam:::role/roles-jdpiak-32022-09-12T20:17:07.98Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2022-09-12T20:18:24.909 DEBUG:boto:Method: POST 2022-09-12T20:18:24.910 DEBUG:boto:Path: / 2022-09-12T20:18:24.911 DEBUG:boto:Data: 2022-09-12T20:18:24.911 DEBUG:boto:Headers: {} 2022-09-12T20:18:24.912 DEBUG:boto:Host: smithi035.front.sepia.ceph.com 2022-09-12T20:18:24.912 DEBUG:boto:Port: 8001 2022-09-12T20:18:24.913 DEBUG:boto:Params: {'RoleName': 'roles-jdpiak-3', 'Action': 'GetRole', 'Version': '2010-05-08'} 2022-09-12T20:18:24.913 DEBUG:boto:Token: None 2022-09-12T20:18:24.914 DEBUG:boto:CanonicalRequest: POST / host:smithi035.front.sepia.ceph.com:8001 x-amz-date:20220912T201824Z host;x-amz-date 168023e9ce2613828f46a5c510c5e419a25661f70078e18340b559469b9d5fbe 2022-09-12T20:18:24.915 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220912T201824Z 20220912/front/smithi035/aws4_request bd21ccef2335d14ac0250e67654e1a6111e0fd8ff88cff891bde8ec30e1b4b1b 2022-09-12T20:18:24.916 DEBUG:boto:Signature: 6fd97b97ef9345b9022da056bda46e4e1e6a7c81ed3dc577b9ef475453e5c3e9 2022-09-12T20:18:24.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220912T201824Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=FUCN3RT6SAOJRSRI0ZIX/20220912/front/smithi035/aws4_request,SignedHeaders=host;x-amz-date,Signature=6fd97b97ef9345b9022da056bda46e4e1e6a7c81ed3dc577b9ef475453e5c3e9', 'Host': 'smithi035.front.sepia.ceph.com:8001'} 2022-09-12T20:18:24.920 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fc98d7433a59993f-00631f9410-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '718'), ('Date', 'Mon, 12 Sep 2022 20:18:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:18:24.921 DEBUG:boto:b'tx00000fc98d7433a59993f-00631f9410-1826-test-zone3a6d8cb45-8742-400c-8567-58f7f2ea3702roles-jdpiak-3/arn:aws:iam:::role/roles-jdpiak-32022-09-12T20:17:07.98Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2022-09-12T20:18:24.922 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-jdpiak-3 2022-09-12T20:18:24.923 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-jdpiak-3 zones={test-zone1, test-zone3} 2022-09-12T20:18:24.923 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-jdpiak-3 zones={test-zone2, test-zone3} 2022-09-12T20:18:24.924 DEBUG:boto:Method: POST 2022-09-12T20:18:24.924 DEBUG:boto:Path: / 2022-09-12T20:18:24.925 DEBUG:boto:Data: 2022-09-12T20:18:24.926 DEBUG:boto:Headers: {} 2022-09-12T20:18:24.926 DEBUG:boto:Host: smithi138.front.sepia.ceph.com 2022-09-12T20:18:24.927 DEBUG:boto:Port: 8000 2022-09-12T20:18:24.928 DEBUG:boto:Params: {'RoleName': 'roles-jdpiak-3', 'Action': 'GetRole', 'Version': '2010-05-08'} 2022-09-12T20:18:24.929 DEBUG:boto:Token: None 2022-09-12T20:18:24.929 DEBUG:boto:CanonicalRequest: POST / host:smithi138.front.sepia.ceph.com:8000 x-amz-date:20220912T201824Z host;x-amz-date 168023e9ce2613828f46a5c510c5e419a25661f70078e18340b559469b9d5fbe 2022-09-12T20:18:24.930 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220912T201824Z 20220912/front/smithi138/aws4_request d089419943a6e34f11f5bae0525efa75eab2bd81511841f6c395c906c752d140 2022-09-12T20:18:24.931 DEBUG:boto:Signature: 11284e1b7ef7278bc140b17aad9085bf2ada25375216ca15d59b010d47bc54e2 2022-09-12T20:18:24.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220912T201824Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=FUCN3RT6SAOJRSRI0ZIX/20220912/front/smithi138/aws4_request,SignedHeaders=host;x-amz-date,Signature=11284e1b7ef7278bc140b17aad9085bf2ada25375216ca15d59b010d47bc54e2', 'Host': 'smithi138.front.sepia.ceph.com:8000'} 2022-09-12T20:18:24.934 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f88070b08eceb58d-00631f9410-1a4f-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Mon, 12 Sep 2022 20:18:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:18:24.934 DEBUG:boto:b'tx00000f88070b08eceb58d-00631f9410-1a4f-test-zone2a6d8cb45-8742-400c-8567-58f7f2ea3702roles-jdpiak-3/arn:aws:iam:::role/roles-jdpiak-32022-09-12T20:17:38.196Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2022-09-12T20:18:24.935 DEBUG:boto:Method: POST 2022-09-12T20:18:24.936 DEBUG:boto:Path: / 2022-09-12T20:18:24.936 DEBUG:boto:Data: 2022-09-12T20:18:24.937 DEBUG:boto:Headers: {} 2022-09-12T20:18:24.937 DEBUG:boto:Host: smithi035.front.sepia.ceph.com 2022-09-12T20:18:24.938 DEBUG:boto:Port: 8001 2022-09-12T20:18:24.939 DEBUG:boto:Params: {'RoleName': 'roles-jdpiak-3', 'Action': 'GetRole', 'Version': '2010-05-08'} 2022-09-12T20:18:24.940 DEBUG:boto:Token: None 2022-09-12T20:18:24.941 DEBUG:boto:CanonicalRequest: POST / host:smithi035.front.sepia.ceph.com:8001 x-amz-date:20220912T201824Z host;x-amz-date 168023e9ce2613828f46a5c510c5e419a25661f70078e18340b559469b9d5fbe 2022-09-12T20:18:24.941 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220912T201824Z 20220912/front/smithi035/aws4_request bd21ccef2335d14ac0250e67654e1a6111e0fd8ff88cff891bde8ec30e1b4b1b 2022-09-12T20:18:24.942 DEBUG:boto:Signature: 6fd97b97ef9345b9022da056bda46e4e1e6a7c81ed3dc577b9ef475453e5c3e9 2022-09-12T20:18:24.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220912T201824Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=FUCN3RT6SAOJRSRI0ZIX/20220912/front/smithi035/aws4_request,SignedHeaders=host;x-amz-date,Signature=6fd97b97ef9345b9022da056bda46e4e1e6a7c81ed3dc577b9ef475453e5c3e9', 'Host': 'smithi035.front.sepia.ceph.com:8001'} 2022-09-12T20:18:24.944 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000a5ec2ee7d24b0ba-00631f9410-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '718'), ('Date', 'Mon, 12 Sep 2022 20:18:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:18:24.945 DEBUG:boto:b'tx000000a5ec2ee7d24b0ba-00631f9410-1826-test-zone3a6d8cb45-8742-400c-8567-58f7f2ea3702roles-jdpiak-3/arn:aws:iam:::role/roles-jdpiak-32022-09-12T20:17:07.98Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2022-09-12T20:18:24.946 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-jdpiak-3 2022-09-12T20:18:24.946 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-jdpiak-3 zones={test-zone2, test-zone3} 2022-09-12T20:18:24.947 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ok 2022-09-12T20:18:24.949 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:18:24.949 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:18:24.950 DEBUG:boto:Method: PUT 2022-09-12T20:18:24.951 DEBUG:boto:Path: /jdpiak-76/ 2022-09-12T20:18:24.952 DEBUG:boto:Data: 2022-09-12T20:18:24.952 DEBUG:boto:Headers: {} 2022-09-12T20:18:24.953 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:18:24.953 DEBUG:boto:Port: 8000 2022-09-12T20:18:24.954 DEBUG:boto:Params: {} 2022-09-12T20:18:24.954 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T20:18:24.955 DEBUG:boto:Token: None 2022-09-12T20:18:24.956 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 20:18:24 GMT /jdpiak-76/ 2022-09-12T20:18:24.956 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:icWG9tT9ery8JEtToKG4f6mpwHs= 2022-09-12T20:18:24.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:18:24 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:icWG9tT9ery8JEtToKG4f6mpwHs=', 'Content-Length': '0'} 2022-09-12T20:18:24.982 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000014cbef3e5e607a8c-00631f9410-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:18:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:18:24.983 DEBUG:rgw_multi.tests:created bucket=jdpiak-76 2022-09-12T20:18:24.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:18:24.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:18:24.984 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:18:25.038 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:18:25.038 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:18:25.083 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=38ca1f57-1358-496c-a99b-df49ee365a4f 2022-09-12T20:18:25.100 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T20:18:25.100 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:18:25.100 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T20:18:25.101 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T20:18:25.101 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:18:25.101 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:18:25.102 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T20:18:25.102 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T20:18:25.102 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:18:25.102 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:18:25.103 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663013904.978431_888.1", 2022-09-12T20:18:25.103 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:18:24.978431Z" 2022-09-12T20:18:25.103 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:18:25.103 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:18:25.104 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:18:25.104 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:16:38.638998Z" 2022-09-12T20:18:25.104 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:18:25.104 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T20:18:25.106 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1663013904.978431_888.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:18:25.106 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T20:18:25.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:18:25.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:18:25.108 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:18:25.158 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:18:25.158 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:18:25.212 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:18:25.210+0000 7f333d050600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:18:25.212 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:18:25.211+0000 7f333d050600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:18:25.244 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:18:25.244 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:18:25.244 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:18:25.245 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:18:25.245 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:18:25.245 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:18:25.246 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:18:25.246 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:25.246 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:18:25.246 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:25.247 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:18:25.247 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:25.247 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:25.247 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:18:25.248 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:25.248 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:18:25.248 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:25.249 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:18:25.249 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:18:25.249 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:25.250 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:25.250 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:25.250 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:18:25.251 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:25.251 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:25.251 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:18:25.251 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:25.252 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:25.252 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:25.252 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:18:25.252 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:18:25.253 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:25.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:25.253 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:25.253 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:18:25.254 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:25.254 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:25.254 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013904.969436_885.1", 2022-09-12T20:18:25.255 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:25.255 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:25.255 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:25.255 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:18:24.969436Z", 2022-09-12T20:18:25.256 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:18:25.256 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:25.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:25.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:25.257 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:18:25.257 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:25.257 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:25.257 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:18:25.258 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:25.258 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:25.258 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:25.258 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:18:25.259 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:18:25.259 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:25.259 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:25.259 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:18:25.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:25.260 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:18:25.260 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:18:25.261 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:18:25.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:25.261 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:18:25Z" 2022-09-12T20:18:25.261 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:18:25.263 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663013904.969436_885.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:18:24.969436Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:18:25Z" } 2022-09-12T20:18:25.263 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663013904.969436_885.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:18:24.969436Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:18:25.264 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663013904.978431_888.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:18:25.264 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663013904.969436_885.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:18:25.265 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663013904.978431_888.1 target=1_1663013904.969436_885.1 2022-09-12T20:18:30.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:18:30.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:18:30.267 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:18:30.323 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:18:30.324 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:18:30.356 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:18:30.354+0000 7f8ff1b20600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:18:30.357 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:18:30.355+0000 7f8ff1b20600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:18:30.372 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:18:30.372 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:18:30.372 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:18:30.372 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:18:30.373 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:18:30.373 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:18:30.373 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:18:30.374 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:30.374 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:18:30.374 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:30.374 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:18:30.375 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:30.375 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:30.375 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:18:30.375 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:30.376 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:18:30.376 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:30.376 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:18:30.376 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:18:30.377 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:30.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:30.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:30.377 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:18:30.378 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:30.378 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:30.378 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:18:30.378 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:30.379 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:30.379 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:30.379 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:18:30.379 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:18:30.380 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:30.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:30.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:30.380 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:18:30.381 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:30.381 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:30.381 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013904.978431_888.1", 2022-09-12T20:18:30.381 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:30.382 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:30.382 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:30.382 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:18:24.978431Z", 2022-09-12T20:18:30.382 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:18:30.383 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:30.383 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:30.383 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:30.383 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:18:30.384 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:30.384 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:30.384 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:18:30.384 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:30.384 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:30.385 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:30.385 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:18:30.385 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:18:30.385 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:30.386 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:30.386 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:18:30.386 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:30.386 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:18:30.387 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:18:30.387 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:18:30.387 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:30.387 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:18:30Z" 2022-09-12T20:18:30.388 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:18:30.389 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663013904.978431_888.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:18:24.978431Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:18:30Z" } 2022-09-12T20:18:30.390 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663013904.978431_888.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:18:24.978431Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:18:30.390 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663013904.978431_888.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:18:30.391 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663013904.978431_888.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:18:30.391 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T20:18:30.392 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T20:18:30.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:18:30.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:18:30.393 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:18:30.452 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:18:30.452 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:18:30.490 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:18:30.489+0000 7ff6751bb600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:18:30.491 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:18:30.490+0000 7ff6751bb600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:18:30.507 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T20:18:30.507 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T20:18:30.508 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T20:18:30.508 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T20:18:30.508 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T20:18:30.508 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:18:30.509 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T20:18:30.509 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:18:30.509 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T20:18:30.509 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:18:30.510 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T20:18:30.510 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:18:30.510 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T20:18:30.510 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:18:30.511 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:18:30.511 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T20:18:30.511 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:18:30.511 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:18:30.512 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T20:18:30.512 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:18:30.512 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:18:30.512 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:18:30.513 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T20:18:30.513 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:18:30.513 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T20:18:30.513 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T20:18:30.514 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:18:30.514 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:18:30.514 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:18:30.514 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T20:18:30.515 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T20:18:30.515 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:18:30.515 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:18:30.515 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:18:30.516 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T20:18:30.516 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:18:30.516 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T20:18:30.516 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663013904.978431_888.1", 2022-09-12T20:18:30.517 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:18:30.517 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:18:30.517 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:18:30.518 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T20:18:24.978431Z", 2022-09-12T20:18:30.518 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T20:18:30.518 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:18:30.518 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:18:30.519 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:18:30.519 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T20:18:30.519 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:18:30.519 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T20:18:30.520 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:18:30.520 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:18:30.520 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:18:30.520 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:18:30.521 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:18:30.521 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T20:18:30.521 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:18:30.521 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:18:30.522 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T20:18:30.522 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:18:30.522 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T20:18:30.522 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T20:18:30.523 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T20:18:30.523 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:18:30.523 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T20:18:30Z" 2022-09-12T20:18:30.523 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T20:18:30.525 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663013904.978431_888.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:18:24.978431Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:18:30Z" } 2022-09-12T20:18:30.525 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663013904.978431_888.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:18:24.978431Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:18:30.526 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663013904.978431_888.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:18:30.526 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663013904.978431_888.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:18:30.527 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T20:18:30.527 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T20:18:30.528 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:18:30.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:18:30.529 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:18:30.581 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:18:30.581 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:18:30.611 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:18:30.609+0000 7fdc7d16d600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:18:30.613 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:18:30.611+0000 7fdc7d16d600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:18:30.632 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:18:30.632 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:18:30.632 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:18:30.633 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:18:30.633 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:18:30.633 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:18:30.634 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:18:30.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:30.634 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:18:30.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:30.635 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:18:30.635 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:30.635 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:30.636 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:18:30.636 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:30.636 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:18:30.636 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:30.637 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:18:30.637 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:18:30.637 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:30.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:30.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:30.638 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:18:30.638 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:30.638 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:30.639 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:18:30.639 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:30.639 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:30.639 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:30.640 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:18:30.640 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:18:30.640 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:30.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:30.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:30.641 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:18:30.641 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:30.641 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:30.642 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013904.978431_888.1", 2022-09-12T20:18:30.642 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:30.642 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:30.642 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:30.643 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:18:24.978431Z", 2022-09-12T20:18:30.643 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:18:30.643 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:30.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:30.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:18:30.644 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:18:30.644 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:18:30.644 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:18:30.644 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:18:30.645 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:18:30.645 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:18:30.645 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:18:30.645 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:18:30.646 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:18:30.646 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:30.646 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:18:30.646 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:18:30.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:30.647 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:18:30.647 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:18:30.648 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:18:30.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:18:30.648 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:18:30Z" 2022-09-12T20:18:30.648 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:18:30.650 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663013904.978431_888.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:18:24.978431Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:18:30Z" } 2022-09-12T20:18:30.650 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663013904.978431_888.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:18:24.978431Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:18:30.651 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663013904.978431_888.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:18:30.651 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663013904.978431_888.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:18:30.652 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T20:18:30.652 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2022-09-12T20:18:30.653 INFO:teuthology.orchestra.run:waiting for 300 2022-09-12T20:18:30.654 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T20:18:30.651+0000 7f0ce345b700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 151964) UID: 0 2022-09-12T20:18:30.654 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T20:18:30.651+0000 7f0ce76c6600 -1 shutting down 2022-09-12T20:18:33.779 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2022-09-12T20:18:36.755 INFO:tasks.rgw.c2.client.0:Stopped 2022-09-12T20:18:36.757 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:18:36.758 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:18:36.759 DEBUG:boto:Method: HEAD 2022-09-12T20:18:36.759 DEBUG:boto:Path: /jdpiak-76/ 2022-09-12T20:18:36.760 DEBUG:boto:Data: 2022-09-12T20:18:36.761 DEBUG:boto:Headers: {} 2022-09-12T20:18:36.762 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:18:36.763 DEBUG:boto:Port: 8000 2022-09-12T20:18:36.764 DEBUG:boto:Params: {} 2022-09-12T20:18:36.765 DEBUG:boto:Token: None 2022-09-12T20:18:36.766 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:18:36 GMT /jdpiak-76/ 2022-09-12T20:18:36.767 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bTFd1o/poumpy+R/Mg/GPnefXMM= 2022-09-12T20:18:36.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:18:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bTFd1o/poumpy+R/Mg/GPnefXMM=', 'Content-Length': '0'} 2022-09-12T20:18:36.771 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a61acf5fbbf3400-00631f941c-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:18:36.772 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:18:36.773 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:18:36.774 DEBUG:boto:Method: PUT 2022-09-12T20:18:36.775 DEBUG:boto:Path: /jdpiak-76/obj11 2022-09-12T20:18:36.776 DEBUG:boto:Data: 2022-09-12T20:18:36.777 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:18:36.777 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:18:36.778 DEBUG:boto:Port: 8000 2022-09-12T20:18:36.779 DEBUG:boto:Params: {} 2022-09-12T20:18:36.780 DEBUG:boto:Token: None 2022-09-12T20:18:36.781 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:18:36 GMT /jdpiak-76/obj11 2022-09-12T20:18:36.782 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:55ZVBv8kARs8pCESPsYMFBUCdYk= 2022-09-12T20:18:36.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:18:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:55ZVBv8kARs8pCESPsYMFBUCdYk='} 2022-09-12T20:18:36.792 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000089fb5f760485532f-00631f941c-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:18:36.793 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:18:36.794 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:18:36.794 DEBUG:boto:Method: HEAD 2022-09-12T20:18:36.795 DEBUG:boto:Path: /jdpiak-76/ 2022-09-12T20:18:36.795 DEBUG:boto:Data: 2022-09-12T20:18:36.796 DEBUG:boto:Headers: {} 2022-09-12T20:18:36.797 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:18:36.798 DEBUG:boto:Port: 8000 2022-09-12T20:18:36.799 DEBUG:boto:Params: {} 2022-09-12T20:18:36.799 DEBUG:boto:Token: None 2022-09-12T20:18:36.800 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:18:36 GMT /jdpiak-76/ 2022-09-12T20:18:36.801 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bTFd1o/poumpy+R/Mg/GPnefXMM= 2022-09-12T20:18:36.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:18:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bTFd1o/poumpy+R/Mg/GPnefXMM=', 'Content-Length': '0'} 2022-09-12T20:18:36.804 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002a39e32b53fb5a09-00631f941c-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:18:36.805 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:18:36.805 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:18:36.806 DEBUG:boto:Method: PUT 2022-09-12T20:18:36.807 DEBUG:boto:Path: /jdpiak-76/obj22 2022-09-12T20:18:36.807 DEBUG:boto:Data: 2022-09-12T20:18:36.808 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:18:36.808 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:18:36.808 DEBUG:boto:Port: 8000 2022-09-12T20:18:36.809 DEBUG:boto:Params: {} 2022-09-12T20:18:36.810 DEBUG:boto:Token: None 2022-09-12T20:18:36.810 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:18:36 GMT /jdpiak-76/obj22 2022-09-12T20:18:36.811 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pDG7nsih23+lkfPyQbtMlmPd7po= 2022-09-12T20:18:36.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:18:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pDG7nsih23+lkfPyQbtMlmPd7po='} 2022-09-12T20:18:36.817 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000296f9cc8977e9941-00631f941c-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:18:36.818 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:18:36.818 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:18:36.819 DEBUG:boto:Method: HEAD 2022-09-12T20:18:36.819 DEBUG:boto:Path: /jdpiak-76/ 2022-09-12T20:18:36.820 DEBUG:boto:Data: 2022-09-12T20:18:36.820 DEBUG:boto:Headers: {} 2022-09-12T20:18:36.821 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:18:36.821 DEBUG:boto:Port: 8000 2022-09-12T20:18:36.822 DEBUG:boto:Params: {} 2022-09-12T20:18:36.823 DEBUG:boto:Token: None 2022-09-12T20:18:36.824 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:18:36 GMT /jdpiak-76/ 2022-09-12T20:18:36.824 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bTFd1o/poumpy+R/Mg/GPnefXMM= 2022-09-12T20:18:36.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:18:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bTFd1o/poumpy+R/Mg/GPnefXMM=', 'Content-Length': '0'} 2022-09-12T20:18:36.828 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac7378977019205f-00631f941c-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:18:36.829 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:18:36.829 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:18:36.830 DEBUG:boto:Method: PUT 2022-09-12T20:18:36.830 DEBUG:boto:Path: /jdpiak-76/obj33 2022-09-12T20:18:36.831 DEBUG:boto:Data: 2022-09-12T20:18:36.831 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:18:36.832 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:18:36.832 DEBUG:boto:Port: 8000 2022-09-12T20:18:36.833 DEBUG:boto:Params: {} 2022-09-12T20:18:36.833 DEBUG:boto:Token: None 2022-09-12T20:18:36.834 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:18:36 GMT /jdpiak-76/obj33 2022-09-12T20:18:36.835 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:25q2/bM5+r6cTyEnvd6JJ/e9sKw= 2022-09-12T20:18:36.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:18:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:25q2/bM5+r6cTyEnvd6JJ/e9sKw='} 2022-09-12T20:18:36.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009cd650808127730f-00631f941c-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:18:36.879 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:18:36.880 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:18:36.880 DEBUG:boto:Method: HEAD 2022-09-12T20:18:36.881 DEBUG:boto:Path: /jdpiak-76/ 2022-09-12T20:18:36.881 DEBUG:boto:Data: 2022-09-12T20:18:36.882 DEBUG:boto:Headers: {} 2022-09-12T20:18:36.882 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:18:36.883 DEBUG:boto:Port: 8000 2022-09-12T20:18:36.883 DEBUG:boto:Params: {} 2022-09-12T20:18:36.884 DEBUG:boto:Token: None 2022-09-12T20:18:36.884 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:18:36 GMT /jdpiak-76/ 2022-09-12T20:18:36.885 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bTFd1o/poumpy+R/Mg/GPnefXMM= 2022-09-12T20:18:36.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:18:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bTFd1o/poumpy+R/Mg/GPnefXMM=', 'Content-Length': '0'} 2022-09-12T20:18:36.888 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000048d7e4838858a9d3-00631f941c-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:18:36.889 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:18:36.890 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:18:36.890 DEBUG:boto:Method: PUT 2022-09-12T20:18:36.891 DEBUG:boto:Path: /jdpiak-76/obj44 2022-09-12T20:18:36.891 DEBUG:boto:Data: 2022-09-12T20:18:36.892 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:18:36.892 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:18:36.893 DEBUG:boto:Port: 8000 2022-09-12T20:18:36.893 DEBUG:boto:Params: {} 2022-09-12T20:18:36.894 DEBUG:boto:Token: None 2022-09-12T20:18:36.895 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:18:36 GMT /jdpiak-76/obj44 2022-09-12T20:18:36.895 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:tC+e98/oylTqY4Kht1GRDij1194= 2022-09-12T20:18:36.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:18:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:tC+e98/oylTqY4Kht1GRDij1194='} 2022-09-12T20:18:36.903 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009ef7cbd1c0efcf4e-00631f941c-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:18:36.904 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:18:36.904 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:18:36.905 DEBUG:boto:Method: HEAD 2022-09-12T20:18:36.905 DEBUG:boto:Path: /jdpiak-76/ 2022-09-12T20:18:36.906 DEBUG:boto:Data: 2022-09-12T20:18:36.906 DEBUG:boto:Headers: {} 2022-09-12T20:18:36.907 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:18:36.907 DEBUG:boto:Port: 8000 2022-09-12T20:18:36.908 DEBUG:boto:Params: {} 2022-09-12T20:18:36.909 DEBUG:boto:Token: None 2022-09-12T20:18:36.909 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:18:36 GMT /jdpiak-76/ 2022-09-12T20:18:36.910 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bTFd1o/poumpy+R/Mg/GPnefXMM= 2022-09-12T20:18:36.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:18:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bTFd1o/poumpy+R/Mg/GPnefXMM=', 'Content-Length': '0'} 2022-09-12T20:18:36.913 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020732ff6d2d7a555-00631f941c-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:18:36.914 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:18:36.914 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:18:36.915 DEBUG:boto:Method: PUT 2022-09-12T20:18:36.915 DEBUG:boto:Path: /jdpiak-76/obj55 2022-09-12T20:18:36.916 DEBUG:boto:Data: 2022-09-12T20:18:36.917 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:18:36.917 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:18:36.918 DEBUG:boto:Port: 8000 2022-09-12T20:18:36.918 DEBUG:boto:Params: {} 2022-09-12T20:18:36.919 DEBUG:boto:Token: None 2022-09-12T20:18:36.920 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:18:36 GMT /jdpiak-76/obj55 2022-09-12T20:18:36.920 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xfKfvoX2zqOOv9UOZneF/Eq1aqA= 2022-09-12T20:18:36.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:18:36 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xfKfvoX2zqOOv9UOZneF/Eq1aqA='} 2022-09-12T20:18:36.927 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b6d386451702585-00631f941c-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:18:36.927 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:18:36.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:18:36.928 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:18:36.987 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:18:36.987 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:18:37.027 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:18:37.025+0000 7fbedbda8600 10 RGW-SYNC:data:init_data_sync_status: took lease 2022-09-12T20:18:37.055 INFO:tasks.rgw.c2.client.0:Restarting daemon 2022-09-12T20:18:37.056 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2022-09-12T20:18:37.058 INFO:tasks.rgw.c2.client.0:Started 2022-09-12T20:18:37.058 DEBUG:teuthology.orchestra.run.smithi138:> curl http://smithi138.front.sepia.ceph.com:8000 2022-09-12T20:18:37.080 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-12T20:18:37.081 INFO:teuthology.orchestra.run.smithi138.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-12T20:18:37.081 INFO:teuthology.orchestra.run.smithi138.stderr: Dload Upload Total Spent Left Speed 2022-09-12T20:18:37.082 INFO:teuthology.orchestra.run.smithi138.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi138.front.sepia.ceph.com port 8000: Connection refused 2022-09-12T20:18:38.082 DEBUG:teuthology.orchestra.run.smithi138:> curl http://smithi138.front.sepia.ceph.com:8000 2022-09-12T20:18:38.105 INFO:teuthology.orchestra.run.smithi138.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-12T20:18:38.106 INFO:teuthology.orchestra.run.smithi138.stderr: Dload Upload Total Spent Left Speed 2022-09-12T20:18:38.106 INFO:teuthology.orchestra.run.smithi138.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-12T20:18:38.107 INFO:teuthology.orchestra.run.smithi138.stdout:anonymous 2022-09-12T20:18:38.108 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-76 2022-09-12T20:18:38.109 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-76', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:18:38.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-76', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:18:38.110 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-76 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T20:18:38.203 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:18:38.203 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:18:38.236 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:18:38.235+0000 7f45b633e600 1 waiting to reach incremental sync.. 2022-09-12T20:18:39.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2022-09-12T20:18:43.237 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:18:43.236+0000 7f45b633e600 1 waiting to reach incremental sync.. 2022-09-12T20:18:48.238 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:18:48.236+0000 7f45b633e600 1 waiting to reach incremental sync.. 2022-09-12T20:18:53.237 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:18:53.235+0000 7f45b633e600 1 waiting to reach incremental sync.. 2022-09-12T20:18:58.238 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:18:58.236+0000 7f45b633e600 1 waiting to reach incremental sync.. 2022-09-12T20:19:03.238 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:19:03.236+0000 7f45b633e600 1 waiting to reach incremental sync.. 2022-09-12T20:19:08.239 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:19:08.237+0000 7f45b633e600 1 waiting to reach incremental sync.. 2022-09-12T20:19:13.239 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:19:13.237+0000 7f45b633e600 1 waiting to reach incremental sync.. 2022-09-12T20:19:18.240 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:19:18.238+0000 7f45b633e600 1 waiting to reach incremental sync.. 2022-09-12T20:19:23.240 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:19:23.238+0000 7f45b633e600 1 waiting to reach incremental sync.. 2022-09-12T20:19:28.241 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:19:28.240+0000 7f45b633e600 1 waiting to reach incremental sync.. 2022-09-12T20:19:33.242 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:19:33.241+0000 7f45b633e600 1 waiting to reach incremental sync.. 2022-09-12T20:19:38.242 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:19:38.240+0000 7f45b633e600 1 waiting to reach incremental sync.. 2022-09-12T20:19:43.243 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:19:43.241+0000 7f45b633e600 1 waiting to reach incremental sync.. 2022-09-12T20:19:48.243 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:19:48.241+0000 7f45b633e600 1 waiting to reach incremental sync.. 2022-09-12T20:19:53.244 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:19:53.242+0000 7f45b633e600 1 waiting to reach incremental sync.. 2022-09-12T20:19:58.245 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:19:58.242+0000 7f45b633e600 1 waiting to reach incremental sync.. 2022-09-12T20:20:03.245 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:20:03.243+0000 7f45b633e600 1 waiting to reach incremental sync.. 2022-09-12T20:20:08.245 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:20:08.243+0000 7f45b633e600 1 waiting to reach incremental sync.. 2022-09-12T20:20:13.248 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:20:13.246+0000 7f45b633e600 1 bucket sync caught up with source: 2022-09-12T20:20:13.248 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000001.67.5, 00000000002.56.5, 00000000002.92.5] 2022-09-12T20:20:13.248 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000001.67.5, 00000000002.56.5, 00000000002.92.5] 2022-09-12T20:20:13.249 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:20:13.246+0000 7f45b633e600 0 bucket checkpoint complete 2022-09-12T20:20:13.275 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=jdpiak-76 2022-09-12T20:20:13.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-76', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:20:13.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-76', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:20:13.278 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-76 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:20:13.346 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:20:13.346 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:20:13.394 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:20:13.392+0000 7f2c92738600 1 bucket sync caught up with source: 2022-09-12T20:20:13.394 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000001.67.5, 00000000002.56.5, 00000000002.92.5] 2022-09-12T20:20:13.394 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000001.67.5, 00000000002.56.5, 00000000002.92.5] 2022-09-12T20:20:13.395 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:20:13.392+0000 7f2c92738600 0 bucket checkpoint complete 2022-09-12T20:20:13.412 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=jdpiak-76 2022-09-12T20:20:13.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-76', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:20:13.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-76', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:20:13.414 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-76 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T20:20:13.474 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:20:13.474 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:20:13.511 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:20:13.509+0000 7f43f1f56600 1 bucket sync caught up with source: 2022-09-12T20:20:13.511 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000001.67.5, 00000000002.56.5, 00000000002.92.5] 2022-09-12T20:20:13.511 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000001.67.5, 00000000002.56.5, 00000000002.92.5] 2022-09-12T20:20:13.512 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:20:13.509+0000 7f43f1f56600 0 bucket checkpoint complete 2022-09-12T20:20:13.529 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=jdpiak-76 2022-09-12T20:20:13.529 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-76', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:20:13.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-76', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:20:13.531 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-76 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:20:13.588 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:20:13.589 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:20:13.637 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:20:13.636+0000 7fb337f51600 1 bucket sync caught up with source: 2022-09-12T20:20:13.638 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000001.928.5, 00000000002.302.5, 00000000002.904.5] 2022-09-12T20:20:13.638 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000001.928.5, 00000000002.302.5, 00000000002.904.5] 2022-09-12T20:20:13.638 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:20:13.636+0000 7fb337f51600 0 bucket checkpoint complete 2022-09-12T20:20:13.651 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=jdpiak-76 2022-09-12T20:20:13.652 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-76', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:20:13.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-76', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:20:13.653 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-76 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:20:13.711 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:20:13.711 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:20:13.750 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:20:13.748+0000 7f13b9cc5600 1 bucket sync caught up with source: 2022-09-12T20:20:13.751 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000001.928.5, 00000000002.302.5, 00000000002.904.5] 2022-09-12T20:20:13.751 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000001.928.5, 00000000002.302.5, 00000000002.904.5] 2022-09-12T20:20:13.751 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:20:13.748+0000 7f13b9cc5600 0 bucket checkpoint complete 2022-09-12T20:20:13.763 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=jdpiak-76 2022-09-12T20:20:13.763 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=jdpiak-76 2022-09-12T20:20:13.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-76', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:20:13.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-76', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:20:13.765 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket jdpiak-76 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-12T20:20:13.823 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:20:13.824 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:20:13.875 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:20:13.873+0000 7f0302852600 1 bucket sync caught up with source: 2022-09-12T20:20:13.875 INFO:teuthology.orchestra.run.smithi035.stderr: local status: [, , , , , , , , 00000000001.394.5, 00000000002.133.5, 00000000002.437.5] 2022-09-12T20:20:13.876 INFO:teuthology.orchestra.run.smithi035.stderr: remote markers: [, , , , , , , , 00000000001.394.5, 00000000002.133.5, 00000000002.437.5] 2022-09-12T20:20:13.876 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:20:13.873+0000 7f0302852600 0 bucket checkpoint complete 2022-09-12T20:20:13.889 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=jdpiak-76 2022-09-12T20:20:13.890 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-76', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:20:13.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-76', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:20:13.892 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-76 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T20:20:13.949 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:20:13.950 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:20:13.995 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:20:13.993+0000 7f1dc02e2600 1 waiting to reach incremental sync.. 2022-09-12T20:20:18.996 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:20:18.995+0000 7f1dc02e2600 1 waiting to reach incremental sync.. 2022-09-12T20:20:23.996 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:20:23.995+0000 7f1dc02e2600 1 waiting to reach incremental sync.. 2022-09-12T20:20:28.997 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:20:28.995+0000 7f1dc02e2600 1 waiting to reach incremental sync.. 2022-09-12T20:20:33.998 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:20:33.996+0000 7f1dc02e2600 1 waiting to reach incremental sync.. 2022-09-12T20:20:38.999 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:20:38.997+0000 7f1dc02e2600 1 waiting to reach incremental sync.. 2022-09-12T20:20:43.999 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:20:43.997+0000 7f1dc02e2600 1 waiting to reach incremental sync.. 2022-09-12T20:20:49.000 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:20:48.998+0000 7f1dc02e2600 1 waiting to reach incremental sync.. 2022-09-12T20:20:54.000 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:20:53.998+0000 7f1dc02e2600 1 waiting to reach incremental sync.. 2022-09-12T20:20:59.001 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:20:58.999+0000 7f1dc02e2600 1 waiting to reach incremental sync.. 2022-09-12T20:21:04.002 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:21:04.001+0000 7f1dc02e2600 1 waiting to reach incremental sync.. 2022-09-12T20:21:09.002 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:21:09.001+0000 7f1dc02e2600 1 waiting to reach incremental sync.. 2022-09-12T20:21:14.003 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:21:14.001+0000 7f1dc02e2600 1 waiting to reach incremental sync.. 2022-09-12T20:21:19.004 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:21:19.001+0000 7f1dc02e2600 1 waiting to reach incremental sync.. 2022-09-12T20:21:24.004 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:21:24.002+0000 7f1dc02e2600 1 waiting to reach incremental sync.. 2022-09-12T20:21:29.005 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:21:29.003+0000 7f1dc02e2600 1 waiting to reach incremental sync.. 2022-09-12T20:21:34.005 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:21:34.003+0000 7f1dc02e2600 1 waiting to reach incremental sync.. 2022-09-12T20:21:39.006 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:21:39.004+0000 7f1dc02e2600 1 waiting to reach incremental sync.. 2022-09-12T20:21:44.006 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:21:44.005+0000 7f1dc02e2600 1 waiting to reach incremental sync.. 2022-09-12T20:21:49.007 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:21:49.006+0000 7f1dc02e2600 1 waiting to reach incremental sync.. 2022-09-12T20:21:54.008 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:21:54.007+0000 7f1dc02e2600 1 waiting to reach incremental sync.. 2022-09-12T20:21:59.010 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:21:59.008+0000 7f1dc02e2600 1 bucket sync caught up with source: 2022-09-12T20:21:59.011 INFO:teuthology.orchestra.run.smithi138.stderr: local status: [, , , , , , , , 00000000001.394.5, 00000000002.133.5, 00000000002.437.5] 2022-09-12T20:21:59.011 INFO:teuthology.orchestra.run.smithi138.stderr: remote markers: [, , , , , , , , 00000000001.394.5, 00000000002.133.5, 00000000002.437.5] 2022-09-12T20:21:59.012 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:21:59.008+0000 7f1dc02e2600 0 bucket checkpoint complete 2022-09-12T20:21:59.031 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=jdpiak-76 2022-09-12T20:21:59.033 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-76 zones={test-zone2, test-zone1} 2022-09-12T20:21:59.034 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:21:59.036 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:21:59.037 DEBUG:boto:Method: HEAD 2022-09-12T20:21:59.038 DEBUG:boto:Path: /jdpiak-76/ 2022-09-12T20:21:59.039 DEBUG:boto:Data: 2022-09-12T20:21:59.040 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.041 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:21:59.042 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.043 DEBUG:boto:Params: {} 2022-09-12T20:21:59.046 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T20:21:59.047 DEBUG:boto:Token: None 2022-09-12T20:21:59.049 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/ 2022-09-12T20:21:59.050 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FSZqEDSZPG98IM6xluJYcUzGq7U= 2022-09-12T20:21:59.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FSZqEDSZPG98IM6xluJYcUzGq7U=', 'Content-Length': '0'} 2022-09-12T20:21:59.061 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000139936ef116683d6-00631f94e7-1ad5-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.062 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:21:59.063 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:21:59.065 DEBUG:boto:Method: HEAD 2022-09-12T20:21:59.066 DEBUG:boto:Path: /jdpiak-76/ 2022-09-12T20:21:59.067 DEBUG:boto:Data: 2022-09-12T20:21:59.068 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.069 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:21:59.070 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.071 DEBUG:boto:Params: {} 2022-09-12T20:21:59.072 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T20:21:59.073 DEBUG:boto:Token: None 2022-09-12T20:21:59.074 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/ 2022-09-12T20:21:59.075 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FSZqEDSZPG98IM6xluJYcUzGq7U= 2022-09-12T20:21:59.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FSZqEDSZPG98IM6xluJYcUzGq7U=', 'Content-Length': '0'} 2022-09-12T20:21:59.081 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d81d38f46c9b0257-00631f94e7-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.083 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T20:21:59.084 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:21:59.085 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:21:59.086 DEBUG:boto:path=/jdpiak-76/?versions&max-keys=999 2022-09-12T20:21:59.087 DEBUG:boto:auth_path=/jdpiak-76/?versions&max-keys=999 2022-09-12T20:21:59.088 DEBUG:boto:Method: GET 2022-09-12T20:21:59.089 DEBUG:boto:Path: /jdpiak-76/?versions&max-keys=999 2022-09-12T20:21:59.090 DEBUG:boto:Data: 2022-09-12T20:21:59.092 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.093 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:21:59.093 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.094 DEBUG:boto:Params: {} 2022-09-12T20:21:59.094 DEBUG:boto:Token: None 2022-09-12T20:21:59.095 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/?versions 2022-09-12T20:21:59.096 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yBZC/UMvLIgV/RtvcSPuAIPaeDA= 2022-09-12T20:21:59.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yBZC/UMvLIgV/RtvcSPuAIPaeDA=', 'Content-Length': '0'} 2022-09-12T20:21:59.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e80f3753ddb978cd-00631f94e7-1ad5-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.100 DEBUG:boto:b'jdpiak-76999falseobj11nulltrue2022-09-12T20:18:36.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:18:36.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:18:36.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:18:36.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:18:36.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:21:59.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:21:59.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:21:59.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:21:59.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:21:59.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:21:59.104 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T20:21:59.105 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:21:59.106 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:21:59.106 DEBUG:boto:path=/jdpiak-76/?versions&max-keys=999 2022-09-12T20:21:59.107 DEBUG:boto:auth_path=/jdpiak-76/?versions&max-keys=999 2022-09-12T20:21:59.107 DEBUG:boto:Method: GET 2022-09-12T20:21:59.108 DEBUG:boto:Path: /jdpiak-76/?versions&max-keys=999 2022-09-12T20:21:59.108 DEBUG:boto:Data: 2022-09-12T20:21:59.109 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.109 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:21:59.110 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.110 DEBUG:boto:Params: {} 2022-09-12T20:21:59.111 DEBUG:boto:Token: None 2022-09-12T20:21:59.111 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/?versions 2022-09-12T20:21:59.112 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yBZC/UMvLIgV/RtvcSPuAIPaeDA= 2022-09-12T20:21:59.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yBZC/UMvLIgV/RtvcSPuAIPaeDA=', 'Content-Length': '0'} 2022-09-12T20:21:59.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001692db612cb6a77d-00631f94e7-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.116 DEBUG:boto:b'jdpiak-76999falseobj11nulltrue2022-09-12T20:18:36.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:18:36.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:18:36.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:18:36.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:18:36.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:21:59.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:21:59.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:21:59.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:21:59.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:21:59.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:21:59.120 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:21:59.121 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:21:59.121 DEBUG:boto:path=/jdpiak-76/?versions&max-keys=999 2022-09-12T20:21:59.122 DEBUG:boto:auth_path=/jdpiak-76/?versions&max-keys=999 2022-09-12T20:21:59.123 DEBUG:boto:Method: GET 2022-09-12T20:21:59.123 DEBUG:boto:Path: /jdpiak-76/?versions&max-keys=999 2022-09-12T20:21:59.124 DEBUG:boto:Data: 2022-09-12T20:21:59.124 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.125 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:21:59.125 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.126 DEBUG:boto:Params: {} 2022-09-12T20:21:59.126 DEBUG:boto:Token: None 2022-09-12T20:21:59.127 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/?versions 2022-09-12T20:21:59.128 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yBZC/UMvLIgV/RtvcSPuAIPaeDA= 2022-09-12T20:21:59.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yBZC/UMvLIgV/RtvcSPuAIPaeDA=', 'Content-Length': '0'} 2022-09-12T20:21:59.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cd1a0bd19dad515-00631f94e7-1ad5-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.133 DEBUG:boto:b'jdpiak-76999falseobj11nulltrue2022-09-12T20:18:36.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:18:36.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:18:36.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:18:36.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:18:36.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:21:59.134 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:21:59.134 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:21:59.135 DEBUG:boto:path=/jdpiak-76/?versions&max-keys=999 2022-09-12T20:21:59.135 DEBUG:boto:auth_path=/jdpiak-76/?versions&max-keys=999 2022-09-12T20:21:59.136 DEBUG:boto:Method: GET 2022-09-12T20:21:59.136 DEBUG:boto:Path: /jdpiak-76/?versions&max-keys=999 2022-09-12T20:21:59.137 DEBUG:boto:Data: 2022-09-12T20:21:59.138 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.138 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:21:59.139 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.139 DEBUG:boto:Params: {} 2022-09-12T20:21:59.140 DEBUG:boto:Token: None 2022-09-12T20:21:59.141 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/?versions 2022-09-12T20:21:59.141 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yBZC/UMvLIgV/RtvcSPuAIPaeDA= 2022-09-12T20:21:59.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yBZC/UMvLIgV/RtvcSPuAIPaeDA=', 'Content-Length': '0'} 2022-09-12T20:21:59.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045a0ad21d90b7e32-00631f94e7-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.146 DEBUG:boto:b'jdpiak-76999falseobj11nulltrue2022-09-12T20:18:36.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:18:36.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:18:36.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:18:36.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:18:36.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:21:59.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:21:59.148 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:21:59.148 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:21:59.149 DEBUG:boto:path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:21:59.150 DEBUG:boto:auth_path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:21:59.150 DEBUG:boto:Method: GET 2022-09-12T20:21:59.151 DEBUG:boto:Path: /jdpiak-76/obj11?versionId=null 2022-09-12T20:21:59.151 DEBUG:boto:Data: 2022-09-12T20:21:59.152 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.152 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:21:59.153 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.154 DEBUG:boto:Params: {} 2022-09-12T20:21:59.154 DEBUG:boto:Token: None 2022-09-12T20:21:59.155 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj11?versionId=null 2022-09-12T20:21:59.156 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AIYSgQM9AdiYm75ZKGO+2xlb7OI= 2022-09-12T20:21:59.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AIYSgQM9AdiYm75ZKGO+2xlb7OI=', 'Content-Length': '0'} 2022-09-12T20:21:59.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076934ee427f57d6c-00631f94e7-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.160 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:21:59.161 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:21:59.161 DEBUG:boto:path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:21:59.162 DEBUG:boto:auth_path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:21:59.162 DEBUG:boto:Method: GET 2022-09-12T20:21:59.163 DEBUG:boto:Path: /jdpiak-76/obj11?versionId=null 2022-09-12T20:21:59.163 DEBUG:boto:Data: 2022-09-12T20:21:59.164 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.164 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:21:59.165 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.166 DEBUG:boto:Params: {} 2022-09-12T20:21:59.166 DEBUG:boto:Token: None 2022-09-12T20:21:59.167 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj11?versionId=null 2022-09-12T20:21:59.167 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AIYSgQM9AdiYm75ZKGO+2xlb7OI= 2022-09-12T20:21:59.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AIYSgQM9AdiYm75ZKGO+2xlb7OI=', 'Content-Length': '0'} 2022-09-12T20:21:59.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4573de605e800c7-00631f94e7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.171 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:21:59.172 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:21:59.172 DEBUG:boto:path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:21:59.173 DEBUG:boto:auth_path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:21:59.174 DEBUG:boto:Method: HEAD 2022-09-12T20:21:59.174 DEBUG:boto:Path: /jdpiak-76/obj11?versionId=null 2022-09-12T20:21:59.175 DEBUG:boto:Data: 2022-09-12T20:21:59.175 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.176 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:21:59.176 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.177 DEBUG:boto:Params: {} 2022-09-12T20:21:59.177 DEBUG:boto:Token: None 2022-09-12T20:21:59.178 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj11?versionId=null 2022-09-12T20:21:59.179 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ipf5VRLYsa45rSPfX+A11eD+sSc= 2022-09-12T20:21:59.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ipf5VRLYsa45rSPfX+A11eD+sSc=', 'Content-Length': '0'} 2022-09-12T20:21:59.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000678d176e21c7c241-00631f94e7-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.182 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:21:59.183 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:21:59.183 DEBUG:boto:path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:21:59.184 DEBUG:boto:auth_path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:21:59.184 DEBUG:boto:Method: HEAD 2022-09-12T20:21:59.185 DEBUG:boto:Path: /jdpiak-76/obj11?versionId=null 2022-09-12T20:21:59.185 DEBUG:boto:Data: 2022-09-12T20:21:59.186 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.186 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:21:59.187 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.187 DEBUG:boto:Params: {} 2022-09-12T20:21:59.188 DEBUG:boto:Token: None 2022-09-12T20:21:59.188 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj11?versionId=null 2022-09-12T20:21:59.189 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Ipf5VRLYsa45rSPfX+A11eD+sSc= 2022-09-12T20:21:59.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Ipf5VRLYsa45rSPfX+A11eD+sSc=', 'Content-Length': '0'} 2022-09-12T20:21:59.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079e850cf1e834fc5-00631f94e7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:21:59.193 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:21:59.193 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:21:59.194 DEBUG:boto:path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:21:59.194 DEBUG:boto:auth_path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:21:59.195 DEBUG:boto:Method: GET 2022-09-12T20:21:59.195 DEBUG:boto:Path: /jdpiak-76/obj11?versionId=null 2022-09-12T20:21:59.196 DEBUG:boto:Data: 2022-09-12T20:21:59.196 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.197 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:21:59.198 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.198 DEBUG:boto:Params: {} 2022-09-12T20:21:59.199 DEBUG:boto:Token: None 2022-09-12T20:21:59.200 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj11?versionId=null 2022-09-12T20:21:59.200 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AIYSgQM9AdiYm75ZKGO+2xlb7OI= 2022-09-12T20:21:59.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AIYSgQM9AdiYm75ZKGO+2xlb7OI=', 'Content-Length': '0'} 2022-09-12T20:21:59.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7f894a821f0a03a-00631f94e7-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.246 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:21:59.247 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:21:59.248 DEBUG:boto:path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:21:59.248 DEBUG:boto:auth_path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:21:59.249 DEBUG:boto:Method: GET 2022-09-12T20:21:59.250 DEBUG:boto:Path: /jdpiak-76/obj11?versionId=null 2022-09-12T20:21:59.250 DEBUG:boto:Data: 2022-09-12T20:21:59.251 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.252 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:21:59.252 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.253 DEBUG:boto:Params: {} 2022-09-12T20:21:59.254 DEBUG:boto:Token: None 2022-09-12T20:21:59.255 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj11?versionId=null 2022-09-12T20:21:59.255 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:AIYSgQM9AdiYm75ZKGO+2xlb7OI= 2022-09-12T20:21:59.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:AIYSgQM9AdiYm75ZKGO+2xlb7OI=', 'Content-Length': '0'} 2022-09-12T20:21:59.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf41cd12f4f8d1dc-00631f94e7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.260 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:21:59.261 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:21:59.261 DEBUG:boto:path=/jdpiak-76/?prefix=obj11 2022-09-12T20:21:59.262 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj11 2022-09-12T20:21:59.263 DEBUG:boto:Method: GET 2022-09-12T20:21:59.263 DEBUG:boto:Path: /jdpiak-76/?prefix=obj11 2022-09-12T20:21:59.264 DEBUG:boto:Data: 2022-09-12T20:21:59.265 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.265 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:21:59.266 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.266 DEBUG:boto:Params: {} 2022-09-12T20:21:59.267 DEBUG:boto:Token: None 2022-09-12T20:21:59.268 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/ 2022-09-12T20:21:59.268 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Bixlod2SW1wncDTy5dHDU30CKto= 2022-09-12T20:21:59.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Bixlod2SW1wncDTy5dHDU30CKto=', 'Content-Length': '0'} 2022-09-12T20:21:59.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000582a50bba38ceb60-00631f94e7-1ad5-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.273 DEBUG:boto:b'jdpiak-76obj111000falseobj112022-09-12T20:18:36.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:21:59.275 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:21:59.276 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:21:59.276 DEBUG:boto:path=/jdpiak-76/?prefix=obj11 2022-09-12T20:21:59.277 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj11 2022-09-12T20:21:59.278 DEBUG:boto:Method: GET 2022-09-12T20:21:59.278 DEBUG:boto:Path: /jdpiak-76/?prefix=obj11 2022-09-12T20:21:59.279 DEBUG:boto:Data: 2022-09-12T20:21:59.279 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.280 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:21:59.280 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.281 DEBUG:boto:Params: {} 2022-09-12T20:21:59.281 DEBUG:boto:Token: None 2022-09-12T20:21:59.282 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/ 2022-09-12T20:21:59.283 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Bixlod2SW1wncDTy5dHDU30CKto= 2022-09-12T20:21:59.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Bixlod2SW1wncDTy5dHDU30CKto=', 'Content-Length': '0'} 2022-09-12T20:21:59.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e7d5dde574754ec-00631f94e7-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.330 DEBUG:boto:b'jdpiak-76obj111000falseobj112022-09-12T20:18:36.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:21:59.332 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:21:59.333 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:21:59.333 DEBUG:boto:Method: HEAD 2022-09-12T20:21:59.335 DEBUG:boto:Path: /jdpiak-76/obj11 2022-09-12T20:21:59.336 DEBUG:boto:Data: 2022-09-12T20:21:59.336 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.337 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:21:59.338 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.339 DEBUG:boto:Params: {} 2022-09-12T20:21:59.340 DEBUG:boto:Token: None 2022-09-12T20:21:59.341 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj11 2022-09-12T20:21:59.342 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T5CbZolVbgpT8X/EG4LhzUeqyW8= 2022-09-12T20:21:59.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T5CbZolVbgpT8X/EG4LhzUeqyW8=', 'Content-Length': '0'} 2022-09-12T20:21:59.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000666fedc2e306f309-00631f94e7-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.348 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:21:59.349 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:21:59.349 DEBUG:boto:Method: HEAD 2022-09-12T20:21:59.350 DEBUG:boto:Path: /jdpiak-76/obj11 2022-09-12T20:21:59.351 DEBUG:boto:Data: 2022-09-12T20:21:59.352 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.353 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:21:59.354 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.355 DEBUG:boto:Params: {} 2022-09-12T20:21:59.355 DEBUG:boto:Token: None 2022-09-12T20:21:59.356 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj11 2022-09-12T20:21:59.356 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:T5CbZolVbgpT8X/EG4LhzUeqyW8= 2022-09-12T20:21:59.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:T5CbZolVbgpT8X/EG4LhzUeqyW8=', 'Content-Length': '0'} 2022-09-12T20:21:59.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8f59153a05dcfc9-00631f94e7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:21:59.361 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:21:59.362 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:21:59.363 DEBUG:boto:Method: GET 2022-09-12T20:21:59.363 DEBUG:boto:Path: /jdpiak-76/obj11 2022-09-12T20:21:59.364 DEBUG:boto:Data: 2022-09-12T20:21:59.364 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.365 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:21:59.365 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.366 DEBUG:boto:Params: {} 2022-09-12T20:21:59.367 DEBUG:boto:Token: None 2022-09-12T20:21:59.367 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj11 2022-09-12T20:21:59.368 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s2Gt4rmHLSCWjoE8OdS5BZ1uHxI= 2022-09-12T20:21:59.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s2Gt4rmHLSCWjoE8OdS5BZ1uHxI=', 'Content-Length': '0'} 2022-09-12T20:21:59.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8a6ed164b96dd31-00631f94e7-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.372 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:21:59.373 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:21:59.373 DEBUG:boto:Method: GET 2022-09-12T20:21:59.374 DEBUG:boto:Path: /jdpiak-76/obj11 2022-09-12T20:21:59.375 DEBUG:boto:Data: 2022-09-12T20:21:59.376 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.377 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:21:59.377 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.378 DEBUG:boto:Params: {} 2022-09-12T20:21:59.378 DEBUG:boto:Token: None 2022-09-12T20:21:59.379 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj11 2022-09-12T20:21:59.379 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:s2Gt4rmHLSCWjoE8OdS5BZ1uHxI= 2022-09-12T20:21:59.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:s2Gt4rmHLSCWjoE8OdS5BZ1uHxI=', 'Content-Length': '0'} 2022-09-12T20:21:59.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fb11726e80a5a7d-00631f94e7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.383 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:21:59.384 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:21:59.384 DEBUG:boto:path=/jdpiak-76/?prefix=obj11 2022-09-12T20:21:59.385 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj11 2022-09-12T20:21:59.385 DEBUG:boto:Method: GET 2022-09-12T20:21:59.386 DEBUG:boto:Path: /jdpiak-76/?prefix=obj11 2022-09-12T20:21:59.386 DEBUG:boto:Data: 2022-09-12T20:21:59.387 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.387 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:21:59.388 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.388 DEBUG:boto:Params: {} 2022-09-12T20:21:59.389 DEBUG:boto:Token: None 2022-09-12T20:21:59.390 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/ 2022-09-12T20:21:59.390 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Bixlod2SW1wncDTy5dHDU30CKto= 2022-09-12T20:21:59.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Bixlod2SW1wncDTy5dHDU30CKto=', 'Content-Length': '0'} 2022-09-12T20:21:59.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad9323bcf7d4cadd-00631f94e7-1ad5-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.395 DEBUG:boto:b'jdpiak-76obj111000falseobj112022-09-12T20:18:36.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:21:59.396 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:21:59.397 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:21:59.397 DEBUG:boto:path=/jdpiak-76/?prefix=obj11 2022-09-12T20:21:59.398 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj11 2022-09-12T20:21:59.398 DEBUG:boto:Method: GET 2022-09-12T20:21:59.399 DEBUG:boto:Path: /jdpiak-76/?prefix=obj11 2022-09-12T20:21:59.400 DEBUG:boto:Data: 2022-09-12T20:21:59.401 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.402 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:21:59.403 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.404 DEBUG:boto:Params: {} 2022-09-12T20:21:59.405 DEBUG:boto:Token: None 2022-09-12T20:21:59.406 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/ 2022-09-12T20:21:59.407 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Bixlod2SW1wncDTy5dHDU30CKto= 2022-09-12T20:21:59.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Bixlod2SW1wncDTy5dHDU30CKto=', 'Content-Length': '0'} 2022-09-12T20:21:59.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ea537b18743a78b-00631f94e7-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.413 DEBUG:boto:b'jdpiak-76obj111000falseobj112022-09-12T20:18:36.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:21:59.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:21:59.416 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:21:59.417 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:21:59.417 DEBUG:boto:path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:21:59.418 DEBUG:boto:auth_path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:21:59.419 DEBUG:boto:Method: GET 2022-09-12T20:21:59.419 DEBUG:boto:Path: /jdpiak-76/obj22?versionId=null 2022-09-12T20:21:59.420 DEBUG:boto:Data: 2022-09-12T20:21:59.421 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.422 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:21:59.423 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.424 DEBUG:boto:Params: {} 2022-09-12T20:21:59.424 DEBUG:boto:Token: None 2022-09-12T20:21:59.425 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj22?versionId=null 2022-09-12T20:21:59.426 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:d9YEwS53cSS3SGUWCTuOtsybwAA= 2022-09-12T20:21:59.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:d9YEwS53cSS3SGUWCTuOtsybwAA=', 'Content-Length': '0'} 2022-09-12T20:21:59.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041c07d1e81ac105b-00631f94e7-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.470 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:21:59.471 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:21:59.472 DEBUG:boto:path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:21:59.472 DEBUG:boto:auth_path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:21:59.473 DEBUG:boto:Method: GET 2022-09-12T20:21:59.473 DEBUG:boto:Path: /jdpiak-76/obj22?versionId=null 2022-09-12T20:21:59.474 DEBUG:boto:Data: 2022-09-12T20:21:59.474 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.475 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:21:59.475 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.476 DEBUG:boto:Params: {} 2022-09-12T20:21:59.476 DEBUG:boto:Token: None 2022-09-12T20:21:59.477 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj22?versionId=null 2022-09-12T20:21:59.478 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:d9YEwS53cSS3SGUWCTuOtsybwAA= 2022-09-12T20:21:59.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:d9YEwS53cSS3SGUWCTuOtsybwAA=', 'Content-Length': '0'} 2022-09-12T20:21:59.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1a6f5da4a15aaab-00631f94e7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.482 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:21:59.482 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:21:59.483 DEBUG:boto:path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:21:59.483 DEBUG:boto:auth_path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:21:59.484 DEBUG:boto:Method: HEAD 2022-09-12T20:21:59.484 DEBUG:boto:Path: /jdpiak-76/obj22?versionId=null 2022-09-12T20:21:59.485 DEBUG:boto:Data: 2022-09-12T20:21:59.485 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.486 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:21:59.486 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.487 DEBUG:boto:Params: {} 2022-09-12T20:21:59.487 DEBUG:boto:Token: None 2022-09-12T20:21:59.488 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj22?versionId=null 2022-09-12T20:21:59.488 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1A7BfuN6HDTf8HetEOFxDo6E+c0= 2022-09-12T20:21:59.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1A7BfuN6HDTf8HetEOFxDo6E+c0=', 'Content-Length': '0'} 2022-09-12T20:21:59.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5c9d2bab32f10a7-00631f94e7-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.492 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:21:59.493 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:21:59.493 DEBUG:boto:path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:21:59.494 DEBUG:boto:auth_path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:21:59.494 DEBUG:boto:Method: HEAD 2022-09-12T20:21:59.495 DEBUG:boto:Path: /jdpiak-76/obj22?versionId=null 2022-09-12T20:21:59.495 DEBUG:boto:Data: 2022-09-12T20:21:59.496 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.496 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:21:59.497 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.497 DEBUG:boto:Params: {} 2022-09-12T20:21:59.498 DEBUG:boto:Token: None 2022-09-12T20:21:59.498 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj22?versionId=null 2022-09-12T20:21:59.499 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1A7BfuN6HDTf8HetEOFxDo6E+c0= 2022-09-12T20:21:59.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1A7BfuN6HDTf8HetEOFxDo6E+c0=', 'Content-Length': '0'} 2022-09-12T20:21:59.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c3ac4d53bb78b6a-00631f94e7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:21:59.504 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:21:59.504 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:21:59.505 DEBUG:boto:path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:21:59.505 DEBUG:boto:auth_path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:21:59.506 DEBUG:boto:Method: GET 2022-09-12T20:21:59.506 DEBUG:boto:Path: /jdpiak-76/obj22?versionId=null 2022-09-12T20:21:59.507 DEBUG:boto:Data: 2022-09-12T20:21:59.508 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.508 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:21:59.509 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.509 DEBUG:boto:Params: {} 2022-09-12T20:21:59.510 DEBUG:boto:Token: None 2022-09-12T20:21:59.511 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj22?versionId=null 2022-09-12T20:21:59.511 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:d9YEwS53cSS3SGUWCTuOtsybwAA= 2022-09-12T20:21:59.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:d9YEwS53cSS3SGUWCTuOtsybwAA=', 'Content-Length': '0'} 2022-09-12T20:21:59.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1a2014e1f83041f-00631f94e7-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.515 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:21:59.516 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:21:59.517 DEBUG:boto:path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:21:59.517 DEBUG:boto:auth_path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:21:59.518 DEBUG:boto:Method: GET 2022-09-12T20:21:59.519 DEBUG:boto:Path: /jdpiak-76/obj22?versionId=null 2022-09-12T20:21:59.519 DEBUG:boto:Data: 2022-09-12T20:21:59.520 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.520 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:21:59.521 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.521 DEBUG:boto:Params: {} 2022-09-12T20:21:59.522 DEBUG:boto:Token: None 2022-09-12T20:21:59.523 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj22?versionId=null 2022-09-12T20:21:59.523 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:d9YEwS53cSS3SGUWCTuOtsybwAA= 2022-09-12T20:21:59.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:d9YEwS53cSS3SGUWCTuOtsybwAA=', 'Content-Length': '0'} 2022-09-12T20:21:59.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002232cf89e2e4009e-00631f94e7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.570 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:21:59.571 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:21:59.571 DEBUG:boto:path=/jdpiak-76/?prefix=obj22 2022-09-12T20:21:59.572 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj22 2022-09-12T20:21:59.573 DEBUG:boto:Method: GET 2022-09-12T20:21:59.573 DEBUG:boto:Path: /jdpiak-76/?prefix=obj22 2022-09-12T20:21:59.574 DEBUG:boto:Data: 2022-09-12T20:21:59.574 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.575 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:21:59.575 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.576 DEBUG:boto:Params: {} 2022-09-12T20:21:59.577 DEBUG:boto:Token: None 2022-09-12T20:21:59.577 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/ 2022-09-12T20:21:59.578 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Bixlod2SW1wncDTy5dHDU30CKto= 2022-09-12T20:21:59.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Bixlod2SW1wncDTy5dHDU30CKto=', 'Content-Length': '0'} 2022-09-12T20:21:59.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8a8f4ee763036c1-00631f94e7-1ad5-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.582 DEBUG:boto:b'jdpiak-76obj221000falseobj222022-09-12T20:18:36.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:21:59.584 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:21:59.584 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:21:59.585 DEBUG:boto:path=/jdpiak-76/?prefix=obj22 2022-09-12T20:21:59.585 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj22 2022-09-12T20:21:59.586 DEBUG:boto:Method: GET 2022-09-12T20:21:59.586 DEBUG:boto:Path: /jdpiak-76/?prefix=obj22 2022-09-12T20:21:59.587 DEBUG:boto:Data: 2022-09-12T20:21:59.587 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.588 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:21:59.589 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.589 DEBUG:boto:Params: {} 2022-09-12T20:21:59.590 DEBUG:boto:Token: None 2022-09-12T20:21:59.590 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/ 2022-09-12T20:21:59.591 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Bixlod2SW1wncDTy5dHDU30CKto= 2022-09-12T20:21:59.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Bixlod2SW1wncDTy5dHDU30CKto=', 'Content-Length': '0'} 2022-09-12T20:21:59.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060e560fe62f1af0d-00631f94e7-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.596 DEBUG:boto:b'jdpiak-76obj221000falseobj222022-09-12T20:18:36.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:21:59.597 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:21:59.597 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:21:59.598 DEBUG:boto:Method: HEAD 2022-09-12T20:21:59.599 DEBUG:boto:Path: /jdpiak-76/obj22 2022-09-12T20:21:59.599 DEBUG:boto:Data: 2022-09-12T20:21:59.600 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.601 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:21:59.601 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.602 DEBUG:boto:Params: {} 2022-09-12T20:21:59.602 DEBUG:boto:Token: None 2022-09-12T20:21:59.603 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj22 2022-09-12T20:21:59.604 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OHCYaZtHDsctc0aNLpBXus3nrsM= 2022-09-12T20:21:59.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OHCYaZtHDsctc0aNLpBXus3nrsM=', 'Content-Length': '0'} 2022-09-12T20:21:59.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080105b77638aa44c-00631f94e7-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.607 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:21:59.608 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:21:59.608 DEBUG:boto:Method: HEAD 2022-09-12T20:21:59.609 DEBUG:boto:Path: /jdpiak-76/obj22 2022-09-12T20:21:59.609 DEBUG:boto:Data: 2022-09-12T20:21:59.610 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.610 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:21:59.611 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.611 DEBUG:boto:Params: {} 2022-09-12T20:21:59.612 DEBUG:boto:Token: None 2022-09-12T20:21:59.613 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj22 2022-09-12T20:21:59.613 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OHCYaZtHDsctc0aNLpBXus3nrsM= 2022-09-12T20:21:59.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OHCYaZtHDsctc0aNLpBXus3nrsM=', 'Content-Length': '0'} 2022-09-12T20:21:59.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edcf1b55b8de6d47-00631f94e7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:21:59.617 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:21:59.618 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:21:59.619 DEBUG:boto:Method: GET 2022-09-12T20:21:59.619 DEBUG:boto:Path: /jdpiak-76/obj22 2022-09-12T20:21:59.620 DEBUG:boto:Data: 2022-09-12T20:21:59.620 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.621 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:21:59.621 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.622 DEBUG:boto:Params: {} 2022-09-12T20:21:59.623 DEBUG:boto:Token: None 2022-09-12T20:21:59.623 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj22 2022-09-12T20:21:59.624 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QyoedoFZDUF8miGnTS/pMMKTy0o= 2022-09-12T20:21:59.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QyoedoFZDUF8miGnTS/pMMKTy0o=', 'Content-Length': '0'} 2022-09-12T20:21:59.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a7aac5b4daabce3-00631f94e7-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.628 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:21:59.629 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:21:59.630 DEBUG:boto:Method: GET 2022-09-12T20:21:59.630 DEBUG:boto:Path: /jdpiak-76/obj22 2022-09-12T20:21:59.631 DEBUG:boto:Data: 2022-09-12T20:21:59.631 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.632 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:21:59.632 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.633 DEBUG:boto:Params: {} 2022-09-12T20:21:59.633 DEBUG:boto:Token: None 2022-09-12T20:21:59.634 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj22 2022-09-12T20:21:59.635 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:QyoedoFZDUF8miGnTS/pMMKTy0o= 2022-09-12T20:21:59.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:QyoedoFZDUF8miGnTS/pMMKTy0o=', 'Content-Length': '0'} 2022-09-12T20:21:59.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035a4df9660135f53-00631f94e7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.639 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:21:59.640 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:21:59.640 DEBUG:boto:path=/jdpiak-76/?prefix=obj22 2022-09-12T20:21:59.641 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj22 2022-09-12T20:21:59.641 DEBUG:boto:Method: GET 2022-09-12T20:21:59.642 DEBUG:boto:Path: /jdpiak-76/?prefix=obj22 2022-09-12T20:21:59.642 DEBUG:boto:Data: 2022-09-12T20:21:59.643 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.643 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:21:59.644 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.644 DEBUG:boto:Params: {} 2022-09-12T20:21:59.645 DEBUG:boto:Token: None 2022-09-12T20:21:59.646 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/ 2022-09-12T20:21:59.646 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Bixlod2SW1wncDTy5dHDU30CKto= 2022-09-12T20:21:59.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Bixlod2SW1wncDTy5dHDU30CKto=', 'Content-Length': '0'} 2022-09-12T20:21:59.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045762df10dace882-00631f94e7-1ad5-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.694 DEBUG:boto:b'jdpiak-76obj221000falseobj222022-09-12T20:18:36.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:21:59.695 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:21:59.696 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:21:59.696 DEBUG:boto:path=/jdpiak-76/?prefix=obj22 2022-09-12T20:21:59.697 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj22 2022-09-12T20:21:59.697 DEBUG:boto:Method: GET 2022-09-12T20:21:59.698 DEBUG:boto:Path: /jdpiak-76/?prefix=obj22 2022-09-12T20:21:59.699 DEBUG:boto:Data: 2022-09-12T20:21:59.699 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.700 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:21:59.701 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.702 DEBUG:boto:Params: {} 2022-09-12T20:21:59.702 DEBUG:boto:Token: None 2022-09-12T20:21:59.703 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/ 2022-09-12T20:21:59.704 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Bixlod2SW1wncDTy5dHDU30CKto= 2022-09-12T20:21:59.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Bixlod2SW1wncDTy5dHDU30CKto=', 'Content-Length': '0'} 2022-09-12T20:21:59.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afbe2ccc9de85a11-00631f94e7-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.708 DEBUG:boto:b'jdpiak-76obj221000falseobj222022-09-12T20:18:36.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:21:59.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:21:59.709 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:21:59.710 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:21:59.711 DEBUG:boto:path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:21:59.711 DEBUG:boto:auth_path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:21:59.712 DEBUG:boto:Method: GET 2022-09-12T20:21:59.713 DEBUG:boto:Path: /jdpiak-76/obj33?versionId=null 2022-09-12T20:21:59.713 DEBUG:boto:Data: 2022-09-12T20:21:59.714 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.715 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:21:59.715 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.716 DEBUG:boto:Params: {} 2022-09-12T20:21:59.716 DEBUG:boto:Token: None 2022-09-12T20:21:59.717 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj33?versionId=null 2022-09-12T20:21:59.718 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GLRRiGTRCDnTbhR47zpbvhjHKfI= 2022-09-12T20:21:59.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GLRRiGTRCDnTbhR47zpbvhjHKfI=', 'Content-Length': '0'} 2022-09-12T20:21:59.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6eda1e6ba8d6eb9-00631f94e7-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.722 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:21:59.723 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:21:59.724 DEBUG:boto:path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:21:59.725 DEBUG:boto:auth_path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:21:59.726 DEBUG:boto:Method: GET 2022-09-12T20:21:59.727 DEBUG:boto:Path: /jdpiak-76/obj33?versionId=null 2022-09-12T20:21:59.727 DEBUG:boto:Data: 2022-09-12T20:21:59.728 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.729 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:21:59.730 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.731 DEBUG:boto:Params: {} 2022-09-12T20:21:59.732 DEBUG:boto:Token: None 2022-09-12T20:21:59.733 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj33?versionId=null 2022-09-12T20:21:59.734 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GLRRiGTRCDnTbhR47zpbvhjHKfI= 2022-09-12T20:21:59.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GLRRiGTRCDnTbhR47zpbvhjHKfI=', 'Content-Length': '0'} 2022-09-12T20:21:59.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002468d07428f9d01-00631f94e7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.782 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:21:59.783 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:21:59.784 DEBUG:boto:path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:21:59.784 DEBUG:boto:auth_path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:21:59.785 DEBUG:boto:Method: HEAD 2022-09-12T20:21:59.785 DEBUG:boto:Path: /jdpiak-76/obj33?versionId=null 2022-09-12T20:21:59.785 DEBUG:boto:Data: 2022-09-12T20:21:59.786 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.787 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:21:59.787 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.788 DEBUG:boto:Params: {} 2022-09-12T20:21:59.788 DEBUG:boto:Token: None 2022-09-12T20:21:59.789 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj33?versionId=null 2022-09-12T20:21:59.789 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dP2I5/M6O2WD8IC8BdmqCYkr2GA= 2022-09-12T20:21:59.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dP2I5/M6O2WD8IC8BdmqCYkr2GA=', 'Content-Length': '0'} 2022-09-12T20:21:59.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050e936b7ba6b1121-00631f94e7-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.794 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:21:59.795 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:21:59.796 DEBUG:boto:path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:21:59.796 DEBUG:boto:auth_path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:21:59.797 DEBUG:boto:Method: HEAD 2022-09-12T20:21:59.798 DEBUG:boto:Path: /jdpiak-76/obj33?versionId=null 2022-09-12T20:21:59.799 DEBUG:boto:Data: 2022-09-12T20:21:59.800 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.801 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:21:59.801 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.802 DEBUG:boto:Params: {} 2022-09-12T20:21:59.802 DEBUG:boto:Token: None 2022-09-12T20:21:59.803 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj33?versionId=null 2022-09-12T20:21:59.804 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dP2I5/M6O2WD8IC8BdmqCYkr2GA= 2022-09-12T20:21:59.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dP2I5/M6O2WD8IC8BdmqCYkr2GA=', 'Content-Length': '0'} 2022-09-12T20:21:59.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000115f1a0cafd24094-00631f94e7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:21:59.808 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:21:59.809 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:21:59.809 DEBUG:boto:path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:21:59.810 DEBUG:boto:auth_path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:21:59.810 DEBUG:boto:Method: GET 2022-09-12T20:21:59.811 DEBUG:boto:Path: /jdpiak-76/obj33?versionId=null 2022-09-12T20:21:59.812 DEBUG:boto:Data: 2022-09-12T20:21:59.813 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.813 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:21:59.814 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.814 DEBUG:boto:Params: {} 2022-09-12T20:21:59.815 DEBUG:boto:Token: None 2022-09-12T20:21:59.816 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj33?versionId=null 2022-09-12T20:21:59.816 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GLRRiGTRCDnTbhR47zpbvhjHKfI= 2022-09-12T20:21:59.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GLRRiGTRCDnTbhR47zpbvhjHKfI=', 'Content-Length': '0'} 2022-09-12T20:21:59.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f4ffb11fa9b3828-00631f94e7-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.820 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:21:59.821 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:21:59.822 DEBUG:boto:path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:21:59.823 DEBUG:boto:auth_path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:21:59.823 DEBUG:boto:Method: GET 2022-09-12T20:21:59.824 DEBUG:boto:Path: /jdpiak-76/obj33?versionId=null 2022-09-12T20:21:59.825 DEBUG:boto:Data: 2022-09-12T20:21:59.826 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.826 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:21:59.827 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.827 DEBUG:boto:Params: {} 2022-09-12T20:21:59.828 DEBUG:boto:Token: None 2022-09-12T20:21:59.829 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj33?versionId=null 2022-09-12T20:21:59.829 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GLRRiGTRCDnTbhR47zpbvhjHKfI= 2022-09-12T20:21:59.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GLRRiGTRCDnTbhR47zpbvhjHKfI=', 'Content-Length': '0'} 2022-09-12T20:21:59.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a743756d5e997143-00631f94e7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.833 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:21:59.834 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:21:59.834 DEBUG:boto:path=/jdpiak-76/?prefix=obj33 2022-09-12T20:21:59.835 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj33 2022-09-12T20:21:59.836 DEBUG:boto:Method: GET 2022-09-12T20:21:59.837 DEBUG:boto:Path: /jdpiak-76/?prefix=obj33 2022-09-12T20:21:59.838 DEBUG:boto:Data: 2022-09-12T20:21:59.839 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.840 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:21:59.841 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.842 DEBUG:boto:Params: {} 2022-09-12T20:21:59.843 DEBUG:boto:Token: None 2022-09-12T20:21:59.844 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/ 2022-09-12T20:21:59.845 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Bixlod2SW1wncDTy5dHDU30CKto= 2022-09-12T20:21:59.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Bixlod2SW1wncDTy5dHDU30CKto=', 'Content-Length': '0'} 2022-09-12T20:21:59.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000943932969d6e0695-00631f94e7-1ad5-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.851 DEBUG:boto:b'jdpiak-76obj331000falseobj332022-09-12T20:18:36.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:21:59.852 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:21:59.853 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:21:59.854 DEBUG:boto:path=/jdpiak-76/?prefix=obj33 2022-09-12T20:21:59.855 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj33 2022-09-12T20:21:59.856 DEBUG:boto:Method: GET 2022-09-12T20:21:59.857 DEBUG:boto:Path: /jdpiak-76/?prefix=obj33 2022-09-12T20:21:59.858 DEBUG:boto:Data: 2022-09-12T20:21:59.859 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.860 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:21:59.861 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.862 DEBUG:boto:Params: {} 2022-09-12T20:21:59.863 DEBUG:boto:Token: None 2022-09-12T20:21:59.863 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/ 2022-09-12T20:21:59.864 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Bixlod2SW1wncDTy5dHDU30CKto= 2022-09-12T20:21:59.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Bixlod2SW1wncDTy5dHDU30CKto=', 'Content-Length': '0'} 2022-09-12T20:21:59.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e45999955b81606e-00631f94e7-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.869 DEBUG:boto:b'jdpiak-76obj331000falseobj332022-09-12T20:18:36.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:21:59.869 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:21:59.870 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:21:59.871 DEBUG:boto:Method: HEAD 2022-09-12T20:21:59.871 DEBUG:boto:Path: /jdpiak-76/obj33 2022-09-12T20:21:59.872 DEBUG:boto:Data: 2022-09-12T20:21:59.872 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.873 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:21:59.873 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.874 DEBUG:boto:Params: {} 2022-09-12T20:21:59.874 DEBUG:boto:Token: None 2022-09-12T20:21:59.875 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj33 2022-09-12T20:21:59.875 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ChJc+Gj/7ZIRqJIMK8Gh00NWa6c= 2022-09-12T20:21:59.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ChJc+Gj/7ZIRqJIMK8Gh00NWa6c=', 'Content-Length': '0'} 2022-09-12T20:21:59.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004db25538b729e21f-00631f94e7-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.879 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:21:59.880 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:21:59.880 DEBUG:boto:Method: HEAD 2022-09-12T20:21:59.881 DEBUG:boto:Path: /jdpiak-76/obj33 2022-09-12T20:21:59.881 DEBUG:boto:Data: 2022-09-12T20:21:59.882 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.882 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:21:59.883 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.883 DEBUG:boto:Params: {} 2022-09-12T20:21:59.884 DEBUG:boto:Token: None 2022-09-12T20:21:59.884 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj33 2022-09-12T20:21:59.885 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ChJc+Gj/7ZIRqJIMK8Gh00NWa6c= 2022-09-12T20:21:59.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ChJc+Gj/7ZIRqJIMK8Gh00NWa6c=', 'Content-Length': '0'} 2022-09-12T20:21:59.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d52be8d796dafff0-00631f94e7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:21:59.890 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:21:59.890 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:21:59.891 DEBUG:boto:Method: GET 2022-09-12T20:21:59.891 DEBUG:boto:Path: /jdpiak-76/obj33 2022-09-12T20:21:59.892 DEBUG:boto:Data: 2022-09-12T20:21:59.892 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.893 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:21:59.894 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.894 DEBUG:boto:Params: {} 2022-09-12T20:21:59.895 DEBUG:boto:Token: None 2022-09-12T20:21:59.896 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj33 2022-09-12T20:21:59.896 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gw1KfEuetODNOzXWK6aTUUsmlaI= 2022-09-12T20:21:59.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gw1KfEuetODNOzXWK6aTUUsmlaI=', 'Content-Length': '0'} 2022-09-12T20:21:59.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db963171d4bb12ae-00631f94e7-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.900 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:21:59.901 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:21:59.901 DEBUG:boto:Method: GET 2022-09-12T20:21:59.902 DEBUG:boto:Path: /jdpiak-76/obj33 2022-09-12T20:21:59.902 DEBUG:boto:Data: 2022-09-12T20:21:59.903 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.903 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:21:59.904 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.905 DEBUG:boto:Params: {} 2022-09-12T20:21:59.905 DEBUG:boto:Token: None 2022-09-12T20:21:59.906 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/obj33 2022-09-12T20:21:59.907 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gw1KfEuetODNOzXWK6aTUUsmlaI= 2022-09-12T20:21:59.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gw1KfEuetODNOzXWK6aTUUsmlaI=', 'Content-Length': '0'} 2022-09-12T20:21:59.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb604b85d07ee8a8-00631f94e7-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.954 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:21:59.955 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:21:59.957 DEBUG:boto:path=/jdpiak-76/?prefix=obj33 2022-09-12T20:21:59.957 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj33 2022-09-12T20:21:59.959 DEBUG:boto:Method: GET 2022-09-12T20:21:59.960 DEBUG:boto:Path: /jdpiak-76/?prefix=obj33 2022-09-12T20:21:59.961 DEBUG:boto:Data: 2022-09-12T20:21:59.961 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.962 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:21:59.963 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.964 DEBUG:boto:Params: {} 2022-09-12T20:21:59.965 DEBUG:boto:Token: None 2022-09-12T20:21:59.967 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/ 2022-09-12T20:21:59.968 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Bixlod2SW1wncDTy5dHDU30CKto= 2022-09-12T20:21:59.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Bixlod2SW1wncDTy5dHDU30CKto=', 'Content-Length': '0'} 2022-09-12T20:21:59.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000391bd7a34165bda3-00631f94e7-1ad5-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.973 DEBUG:boto:b'jdpiak-76obj331000falseobj332022-09-12T20:18:36.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:21:59.975 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:21:59.976 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:21:59.977 DEBUG:boto:path=/jdpiak-76/?prefix=obj33 2022-09-12T20:21:59.978 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj33 2022-09-12T20:21:59.979 DEBUG:boto:Method: GET 2022-09-12T20:21:59.980 DEBUG:boto:Path: /jdpiak-76/?prefix=obj33 2022-09-12T20:21:59.980 DEBUG:boto:Data: 2022-09-12T20:21:59.981 DEBUG:boto:Headers: {} 2022-09-12T20:21:59.982 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:21:59.983 DEBUG:boto:Port: 8000 2022-09-12T20:21:59.984 DEBUG:boto:Params: {} 2022-09-12T20:21:59.985 DEBUG:boto:Token: None 2022-09-12T20:21:59.986 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:21:59 GMT /jdpiak-76/ 2022-09-12T20:21:59.988 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Bixlod2SW1wncDTy5dHDU30CKto= 2022-09-12T20:21:59.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:21:59 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Bixlod2SW1wncDTy5dHDU30CKto=', 'Content-Length': '0'} 2022-09-12T20:21:59.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8c2fd032fd9bed0-00631f94e7-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:21:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:21:59.993 DEBUG:boto:b'jdpiak-76obj331000falseobj332022-09-12T20:18:36.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:21:59.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:21:59.995 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:21:59.996 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:21:59.997 DEBUG:boto:path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:21:59.998 DEBUG:boto:auth_path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:21:59.998 DEBUG:boto:Method: GET 2022-09-12T20:21:59.999 DEBUG:boto:Path: /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:00.000 DEBUG:boto:Data: 2022-09-12T20:22:00.001 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.001 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:00.002 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.003 DEBUG:boto:Params: {} 2022-09-12T20:22:00.004 DEBUG:boto:Token: None 2022-09-12T20:22:00.004 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:00.005 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kj37jUCpz4uygQ4YHxv96Va8rQ0= 2022-09-12T20:22:00.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kj37jUCpz4uygQ4YHxv96Va8rQ0=', 'Content-Length': '0'} 2022-09-12T20:22:00.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ea01d36a93cac7f-00631f94e8-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.010 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:00.010 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:00.011 DEBUG:boto:path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:00.012 DEBUG:boto:auth_path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:00.012 DEBUG:boto:Method: GET 2022-09-12T20:22:00.013 DEBUG:boto:Path: /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:00.013 DEBUG:boto:Data: 2022-09-12T20:22:00.014 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.014 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.015 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.016 DEBUG:boto:Params: {} 2022-09-12T20:22:00.016 DEBUG:boto:Token: None 2022-09-12T20:22:00.017 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:00.018 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kj37jUCpz4uygQ4YHxv96Va8rQ0= 2022-09-12T20:22:00.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kj37jUCpz4uygQ4YHxv96Va8rQ0=', 'Content-Length': '0'} 2022-09-12T20:22:00.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000414aaf5b9753cc84-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.022 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:00.023 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:00.023 DEBUG:boto:path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:00.024 DEBUG:boto:auth_path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:00.025 DEBUG:boto:Method: HEAD 2022-09-12T20:22:00.025 DEBUG:boto:Path: /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:00.026 DEBUG:boto:Data: 2022-09-12T20:22:00.027 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.027 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:00.028 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.028 DEBUG:boto:Params: {} 2022-09-12T20:22:00.029 DEBUG:boto:Token: None 2022-09-12T20:22:00.030 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:00.031 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:asvqsybPPMwz0HCjzlkDp8AvzOk= 2022-09-12T20:22:00.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:asvqsybPPMwz0HCjzlkDp8AvzOk=', 'Content-Length': '0'} 2022-09-12T20:22:00.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ff18b583620a170-00631f94e8-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.035 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:00.035 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:00.036 DEBUG:boto:path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:00.037 DEBUG:boto:auth_path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:00.037 DEBUG:boto:Method: HEAD 2022-09-12T20:22:00.038 DEBUG:boto:Path: /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:00.038 DEBUG:boto:Data: 2022-09-12T20:22:00.039 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.040 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.041 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.041 DEBUG:boto:Params: {} 2022-09-12T20:22:00.042 DEBUG:boto:Token: None 2022-09-12T20:22:00.043 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:00.044 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:asvqsybPPMwz0HCjzlkDp8AvzOk= 2022-09-12T20:22:00.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:asvqsybPPMwz0HCjzlkDp8AvzOk=', 'Content-Length': '0'} 2022-09-12T20:22:00.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003feca4c812bb0053-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:22:00.051 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:00.052 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:00.052 DEBUG:boto:path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:00.053 DEBUG:boto:auth_path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:00.053 DEBUG:boto:Method: GET 2022-09-12T20:22:00.054 DEBUG:boto:Path: /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:00.055 DEBUG:boto:Data: 2022-09-12T20:22:00.055 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.056 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:00.057 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.057 DEBUG:boto:Params: {} 2022-09-12T20:22:00.058 DEBUG:boto:Token: None 2022-09-12T20:22:00.059 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:00.059 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kj37jUCpz4uygQ4YHxv96Va8rQ0= 2022-09-12T20:22:00.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kj37jUCpz4uygQ4YHxv96Va8rQ0=', 'Content-Length': '0'} 2022-09-12T20:22:00.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c6dd47a6b7c86e8-00631f94e8-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.063 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:00.064 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:00.065 DEBUG:boto:path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:00.065 DEBUG:boto:auth_path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:00.066 DEBUG:boto:Method: GET 2022-09-12T20:22:00.066 DEBUG:boto:Path: /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:00.067 DEBUG:boto:Data: 2022-09-12T20:22:00.067 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.068 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.068 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.069 DEBUG:boto:Params: {} 2022-09-12T20:22:00.070 DEBUG:boto:Token: None 2022-09-12T20:22:00.070 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:00.071 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kj37jUCpz4uygQ4YHxv96Va8rQ0= 2022-09-12T20:22:00.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kj37jUCpz4uygQ4YHxv96Va8rQ0=', 'Content-Length': '0'} 2022-09-12T20:22:00.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c12ad99ec3cbd6e-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.075 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:00.075 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:00.076 DEBUG:boto:path=/jdpiak-76/?prefix=obj44 2022-09-12T20:22:00.077 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj44 2022-09-12T20:22:00.077 DEBUG:boto:Method: GET 2022-09-12T20:22:00.078 DEBUG:boto:Path: /jdpiak-76/?prefix=obj44 2022-09-12T20:22:00.078 DEBUG:boto:Data: 2022-09-12T20:22:00.079 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.079 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:00.080 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.080 DEBUG:boto:Params: {} 2022-09-12T20:22:00.081 DEBUG:boto:Token: None 2022-09-12T20:22:00.081 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/ 2022-09-12T20:22:00.082 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8= 2022-09-12T20:22:00.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8=', 'Content-Length': '0'} 2022-09-12T20:22:00.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000713057b374dbf0bb-00631f94e8-1ad5-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.126 DEBUG:boto:b'jdpiak-76obj441000falseobj442022-09-12T20:18:36.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:00.127 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:00.128 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:00.128 DEBUG:boto:path=/jdpiak-76/?prefix=obj44 2022-09-12T20:22:00.129 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj44 2022-09-12T20:22:00.129 DEBUG:boto:Method: GET 2022-09-12T20:22:00.130 DEBUG:boto:Path: /jdpiak-76/?prefix=obj44 2022-09-12T20:22:00.130 DEBUG:boto:Data: 2022-09-12T20:22:00.131 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.131 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.132 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.132 DEBUG:boto:Params: {} 2022-09-12T20:22:00.133 DEBUG:boto:Token: None 2022-09-12T20:22:00.134 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/ 2022-09-12T20:22:00.134 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8= 2022-09-12T20:22:00.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8=', 'Content-Length': '0'} 2022-09-12T20:22:00.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007564fba71bf8f3ee-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.139 DEBUG:boto:b'jdpiak-76obj441000falseobj442022-09-12T20:18:36.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:00.140 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:00.141 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:00.141 DEBUG:boto:Method: HEAD 2022-09-12T20:22:00.142 DEBUG:boto:Path: /jdpiak-76/obj44 2022-09-12T20:22:00.142 DEBUG:boto:Data: 2022-09-12T20:22:00.143 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.144 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:00.144 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.145 DEBUG:boto:Params: {} 2022-09-12T20:22:00.145 DEBUG:boto:Token: None 2022-09-12T20:22:00.146 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj44 2022-09-12T20:22:00.146 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Kra4EWMJ1R1qAScBiAIFQba05Fg= 2022-09-12T20:22:00.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Kra4EWMJ1R1qAScBiAIFQba05Fg=', 'Content-Length': '0'} 2022-09-12T20:22:00.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea677700b3d16a37-00631f94e8-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.150 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:00.151 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:00.151 DEBUG:boto:Method: HEAD 2022-09-12T20:22:00.152 DEBUG:boto:Path: /jdpiak-76/obj44 2022-09-12T20:22:00.153 DEBUG:boto:Data: 2022-09-12T20:22:00.153 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.154 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.154 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.155 DEBUG:boto:Params: {} 2022-09-12T20:22:00.155 DEBUG:boto:Token: None 2022-09-12T20:22:00.156 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj44 2022-09-12T20:22:00.156 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Kra4EWMJ1R1qAScBiAIFQba05Fg= 2022-09-12T20:22:00.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Kra4EWMJ1R1qAScBiAIFQba05Fg=', 'Content-Length': '0'} 2022-09-12T20:22:00.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083ff17fd3105b911-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:22:00.161 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:00.162 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:00.163 DEBUG:boto:Method: GET 2022-09-12T20:22:00.163 DEBUG:boto:Path: /jdpiak-76/obj44 2022-09-12T20:22:00.164 DEBUG:boto:Data: 2022-09-12T20:22:00.164 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.165 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:00.165 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.166 DEBUG:boto:Params: {} 2022-09-12T20:22:00.167 DEBUG:boto:Token: None 2022-09-12T20:22:00.167 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj44 2022-09-12T20:22:00.168 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E0iLCml+g68ht7l+wTpzlPmpGAY= 2022-09-12T20:22:00.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E0iLCml+g68ht7l+wTpzlPmpGAY=', 'Content-Length': '0'} 2022-09-12T20:22:00.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009549873dc7783698-00631f94e8-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.172 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:00.173 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:00.173 DEBUG:boto:Method: GET 2022-09-12T20:22:00.174 DEBUG:boto:Path: /jdpiak-76/obj44 2022-09-12T20:22:00.174 DEBUG:boto:Data: 2022-09-12T20:22:00.175 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.175 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.176 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.177 DEBUG:boto:Params: {} 2022-09-12T20:22:00.177 DEBUG:boto:Token: None 2022-09-12T20:22:00.178 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj44 2022-09-12T20:22:00.178 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:E0iLCml+g68ht7l+wTpzlPmpGAY= 2022-09-12T20:22:00.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:E0iLCml+g68ht7l+wTpzlPmpGAY=', 'Content-Length': '0'} 2022-09-12T20:22:00.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0af4d1c2724501a-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.182 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:00.183 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:00.183 DEBUG:boto:path=/jdpiak-76/?prefix=obj44 2022-09-12T20:22:00.184 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj44 2022-09-12T20:22:00.185 DEBUG:boto:Method: GET 2022-09-12T20:22:00.185 DEBUG:boto:Path: /jdpiak-76/?prefix=obj44 2022-09-12T20:22:00.186 DEBUG:boto:Data: 2022-09-12T20:22:00.186 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.187 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:00.187 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.188 DEBUG:boto:Params: {} 2022-09-12T20:22:00.188 DEBUG:boto:Token: None 2022-09-12T20:22:00.189 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/ 2022-09-12T20:22:00.190 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8= 2022-09-12T20:22:00.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8=', 'Content-Length': '0'} 2022-09-12T20:22:00.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb49e652143beab6-00631f94e8-1ad5-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.194 DEBUG:boto:b'jdpiak-76obj441000falseobj442022-09-12T20:18:36.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:00.195 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:00.195 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:00.196 DEBUG:boto:path=/jdpiak-76/?prefix=obj44 2022-09-12T20:22:00.196 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj44 2022-09-12T20:22:00.197 DEBUG:boto:Method: GET 2022-09-12T20:22:00.197 DEBUG:boto:Path: /jdpiak-76/?prefix=obj44 2022-09-12T20:22:00.198 DEBUG:boto:Data: 2022-09-12T20:22:00.198 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.199 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.200 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.200 DEBUG:boto:Params: {} 2022-09-12T20:22:00.201 DEBUG:boto:Token: None 2022-09-12T20:22:00.201 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/ 2022-09-12T20:22:00.202 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8= 2022-09-12T20:22:00.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8=', 'Content-Length': '0'} 2022-09-12T20:22:00.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb8727b63e676055-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.207 DEBUG:boto:b'jdpiak-76obj441000falseobj442022-09-12T20:18:36.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:00.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:22:00.208 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:00.209 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:00.209 DEBUG:boto:path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:00.210 DEBUG:boto:auth_path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:00.210 DEBUG:boto:Method: GET 2022-09-12T20:22:00.211 DEBUG:boto:Path: /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:00.211 DEBUG:boto:Data: 2022-09-12T20:22:00.212 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.212 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:00.213 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.214 DEBUG:boto:Params: {} 2022-09-12T20:22:00.214 DEBUG:boto:Token: None 2022-09-12T20:22:00.215 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:00.215 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N/+lLaxxjBRoi4DyYwwr1fMJZ60= 2022-09-12T20:22:00.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N/+lLaxxjBRoi4DyYwwr1fMJZ60=', 'Content-Length': '0'} 2022-09-12T20:22:00.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036b623ba04ef3d2e-00631f94e8-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.219 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:00.220 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:00.220 DEBUG:boto:path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:00.221 DEBUG:boto:auth_path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:00.222 DEBUG:boto:Method: GET 2022-09-12T20:22:00.222 DEBUG:boto:Path: /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:00.223 DEBUG:boto:Data: 2022-09-12T20:22:00.223 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.224 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.224 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.225 DEBUG:boto:Params: {} 2022-09-12T20:22:00.225 DEBUG:boto:Token: None 2022-09-12T20:22:00.226 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:00.227 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N/+lLaxxjBRoi4DyYwwr1fMJZ60= 2022-09-12T20:22:00.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N/+lLaxxjBRoi4DyYwwr1fMJZ60=', 'Content-Length': '0'} 2022-09-12T20:22:00.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000134c67cd2d343db3-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.270 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:00.271 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:00.271 DEBUG:boto:path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:00.272 DEBUG:boto:auth_path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:00.273 DEBUG:boto:Method: HEAD 2022-09-12T20:22:00.274 DEBUG:boto:Path: /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:00.275 DEBUG:boto:Data: 2022-09-12T20:22:00.275 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.276 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:00.276 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.277 DEBUG:boto:Params: {} 2022-09-12T20:22:00.278 DEBUG:boto:Token: None 2022-09-12T20:22:00.279 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:00.279 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bOpE8TsvluRClDqTL+K8cPmwros= 2022-09-12T20:22:00.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bOpE8TsvluRClDqTL+K8cPmwros=', 'Content-Length': '0'} 2022-09-12T20:22:00.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008651b9b125a7957b-00631f94e8-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.283 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:00.284 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:00.284 DEBUG:boto:path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:00.285 DEBUG:boto:auth_path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:00.286 DEBUG:boto:Method: HEAD 2022-09-12T20:22:00.286 DEBUG:boto:Path: /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:00.287 DEBUG:boto:Data: 2022-09-12T20:22:00.287 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.288 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.288 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.289 DEBUG:boto:Params: {} 2022-09-12T20:22:00.290 DEBUG:boto:Token: None 2022-09-12T20:22:00.290 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:00.291 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bOpE8TsvluRClDqTL+K8cPmwros= 2022-09-12T20:22:00.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bOpE8TsvluRClDqTL+K8cPmwros=', 'Content-Length': '0'} 2022-09-12T20:22:00.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e988e51e8ef87ec-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:22:00.295 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:00.296 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:00.297 DEBUG:boto:path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:00.297 DEBUG:boto:auth_path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:00.298 DEBUG:boto:Method: GET 2022-09-12T20:22:00.298 DEBUG:boto:Path: /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:00.299 DEBUG:boto:Data: 2022-09-12T20:22:00.300 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.300 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:00.301 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.301 DEBUG:boto:Params: {} 2022-09-12T20:22:00.302 DEBUG:boto:Token: None 2022-09-12T20:22:00.303 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:00.303 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N/+lLaxxjBRoi4DyYwwr1fMJZ60= 2022-09-12T20:22:00.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N/+lLaxxjBRoi4DyYwwr1fMJZ60=', 'Content-Length': '0'} 2022-09-12T20:22:00.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddd266bec968e43b-00631f94e8-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.350 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:00.351 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:00.352 DEBUG:boto:path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:00.352 DEBUG:boto:auth_path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:00.353 DEBUG:boto:Method: GET 2022-09-12T20:22:00.353 DEBUG:boto:Path: /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:00.354 DEBUG:boto:Data: 2022-09-12T20:22:00.354 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.355 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.355 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.356 DEBUG:boto:Params: {} 2022-09-12T20:22:00.357 DEBUG:boto:Token: None 2022-09-12T20:22:00.359 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:00.360 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:N/+lLaxxjBRoi4DyYwwr1fMJZ60= 2022-09-12T20:22:00.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:N/+lLaxxjBRoi4DyYwwr1fMJZ60=', 'Content-Length': '0'} 2022-09-12T20:22:00.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094025ba67d9714fd-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.365 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:00.365 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:00.366 DEBUG:boto:path=/jdpiak-76/?prefix=obj55 2022-09-12T20:22:00.366 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj55 2022-09-12T20:22:00.367 DEBUG:boto:Method: GET 2022-09-12T20:22:00.367 DEBUG:boto:Path: /jdpiak-76/?prefix=obj55 2022-09-12T20:22:00.368 DEBUG:boto:Data: 2022-09-12T20:22:00.369 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.370 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:00.371 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.371 DEBUG:boto:Params: {} 2022-09-12T20:22:00.372 DEBUG:boto:Token: None 2022-09-12T20:22:00.372 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/ 2022-09-12T20:22:00.373 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8= 2022-09-12T20:22:00.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8=', 'Content-Length': '0'} 2022-09-12T20:22:00.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f91b74490e9661b-00631f94e8-1ad5-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.378 DEBUG:boto:b'jdpiak-76obj551000falseobj552022-09-12T20:18:36.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:00.379 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:00.379 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:00.380 DEBUG:boto:path=/jdpiak-76/?prefix=obj55 2022-09-12T20:22:00.381 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj55 2022-09-12T20:22:00.381 DEBUG:boto:Method: GET 2022-09-12T20:22:00.382 DEBUG:boto:Path: /jdpiak-76/?prefix=obj55 2022-09-12T20:22:00.383 DEBUG:boto:Data: 2022-09-12T20:22:00.383 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.384 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.384 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.385 DEBUG:boto:Params: {} 2022-09-12T20:22:00.385 DEBUG:boto:Token: None 2022-09-12T20:22:00.386 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/ 2022-09-12T20:22:00.387 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8= 2022-09-12T20:22:00.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8=', 'Content-Length': '0'} 2022-09-12T20:22:00.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023ea79d05e1d4e42-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.392 DEBUG:boto:b'jdpiak-76obj551000falseobj552022-09-12T20:18:36.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:00.392 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:00.393 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:00.393 DEBUG:boto:Method: HEAD 2022-09-12T20:22:00.394 DEBUG:boto:Path: /jdpiak-76/obj55 2022-09-12T20:22:00.395 DEBUG:boto:Data: 2022-09-12T20:22:00.395 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.396 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:00.396 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.397 DEBUG:boto:Params: {} 2022-09-12T20:22:00.398 DEBUG:boto:Token: None 2022-09-12T20:22:00.398 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj55 2022-09-12T20:22:00.399 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4AoVBn9zRy9rb2Kg7vB7w7ptEIg= 2022-09-12T20:22:00.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4AoVBn9zRy9rb2Kg7vB7w7ptEIg=', 'Content-Length': '0'} 2022-09-12T20:22:00.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bd6b8cf8ca55156-00631f94e8-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.403 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:00.404 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:00.404 DEBUG:boto:Method: HEAD 2022-09-12T20:22:00.405 DEBUG:boto:Path: /jdpiak-76/obj55 2022-09-12T20:22:00.405 DEBUG:boto:Data: 2022-09-12T20:22:00.406 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.407 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.407 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.408 DEBUG:boto:Params: {} 2022-09-12T20:22:00.408 DEBUG:boto:Token: None 2022-09-12T20:22:00.409 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj55 2022-09-12T20:22:00.409 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4AoVBn9zRy9rb2Kg7vB7w7ptEIg= 2022-09-12T20:22:00.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4AoVBn9zRy9rb2Kg7vB7w7ptEIg=', 'Content-Length': '0'} 2022-09-12T20:22:00.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8bfca6eec0b11f6-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:22:00.414 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:00.415 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:00.415 DEBUG:boto:Method: GET 2022-09-12T20:22:00.416 DEBUG:boto:Path: /jdpiak-76/obj55 2022-09-12T20:22:00.416 DEBUG:boto:Data: 2022-09-12T20:22:00.417 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.418 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:00.418 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.419 DEBUG:boto:Params: {} 2022-09-12T20:22:00.419 DEBUG:boto:Token: None 2022-09-12T20:22:00.420 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj55 2022-09-12T20:22:00.421 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Me666xN97RC9bttGAu0Pwkh45H8= 2022-09-12T20:22:00.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Me666xN97RC9bttGAu0Pwkh45H8=', 'Content-Length': '0'} 2022-09-12T20:22:00.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000858da9ee18ea3d54-00631f94e8-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.425 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:00.425 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:00.426 DEBUG:boto:Method: GET 2022-09-12T20:22:00.426 DEBUG:boto:Path: /jdpiak-76/obj55 2022-09-12T20:22:00.427 DEBUG:boto:Data: 2022-09-12T20:22:00.428 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.428 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.429 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.429 DEBUG:boto:Params: {} 2022-09-12T20:22:00.430 DEBUG:boto:Token: None 2022-09-12T20:22:00.431 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj55 2022-09-12T20:22:00.431 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Me666xN97RC9bttGAu0Pwkh45H8= 2022-09-12T20:22:00.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Me666xN97RC9bttGAu0Pwkh45H8=', 'Content-Length': '0'} 2022-09-12T20:22:00.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067b26531c35b476a-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.435 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:00.436 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:00.436 DEBUG:boto:path=/jdpiak-76/?prefix=obj55 2022-09-12T20:22:00.437 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj55 2022-09-12T20:22:00.437 DEBUG:boto:Method: GET 2022-09-12T20:22:00.438 DEBUG:boto:Path: /jdpiak-76/?prefix=obj55 2022-09-12T20:22:00.439 DEBUG:boto:Data: 2022-09-12T20:22:00.439 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.440 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:00.440 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.441 DEBUG:boto:Params: {} 2022-09-12T20:22:00.442 DEBUG:boto:Token: None 2022-09-12T20:22:00.442 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/ 2022-09-12T20:22:00.443 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8= 2022-09-12T20:22:00.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8=', 'Content-Length': '0'} 2022-09-12T20:22:00.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000017f2c3c82678c1-00631f94e8-1ad5-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.448 DEBUG:boto:b'jdpiak-76obj551000falseobj552022-09-12T20:18:36.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:00.450 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:00.451 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:00.452 DEBUG:boto:path=/jdpiak-76/?prefix=obj55 2022-09-12T20:22:00.453 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj55 2022-09-12T20:22:00.454 DEBUG:boto:Method: GET 2022-09-12T20:22:00.455 DEBUG:boto:Path: /jdpiak-76/?prefix=obj55 2022-09-12T20:22:00.456 DEBUG:boto:Data: 2022-09-12T20:22:00.457 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.458 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.459 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.460 DEBUG:boto:Params: {} 2022-09-12T20:22:00.461 DEBUG:boto:Token: None 2022-09-12T20:22:00.463 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/ 2022-09-12T20:22:00.464 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8= 2022-09-12T20:22:00.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8=', 'Content-Length': '0'} 2022-09-12T20:22:00.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008752ad5bec9e86f1-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.510 DEBUG:boto:b'jdpiak-76obj551000falseobj552022-09-12T20:18:36.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:00.511 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-76 zones={test-zone2, test-zone1} 2022-09-12T20:22:00.512 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-76 zones={test-zone3, test-zone1} 2022-09-12T20:22:00.512 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:00.513 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:00.514 DEBUG:boto:Method: HEAD 2022-09-12T20:22:00.514 DEBUG:boto:Path: /jdpiak-76/ 2022-09-12T20:22:00.515 DEBUG:boto:Data: 2022-09-12T20:22:00.515 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.516 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:00.517 DEBUG:boto:Port: 8001 2022-09-12T20:22:00.517 DEBUG:boto:Params: {} 2022-09-12T20:22:00.518 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-12T20:22:00.519 DEBUG:boto:Token: None 2022-09-12T20:22:00.520 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/ 2022-09-12T20:22:00.520 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f6BnZzgR86PYwbdpAVPpPzFPABs= 2022-09-12T20:22:00.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f6BnZzgR86PYwbdpAVPpPzFPABs=', 'Content-Length': '0'} 2022-09-12T20:22:00.525 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dfe9c001d6d5fcbe-00631f94e8-1826-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.526 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:00.526 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:00.527 DEBUG:boto:Method: HEAD 2022-09-12T20:22:00.528 DEBUG:boto:Path: /jdpiak-76/ 2022-09-12T20:22:00.528 DEBUG:boto:Data: 2022-09-12T20:22:00.529 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.530 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.530 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.531 DEBUG:boto:Params: {} 2022-09-12T20:22:00.531 DEBUG:boto:Token: None 2022-09-12T20:22:00.532 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/ 2022-09-12T20:22:00.533 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:f6BnZzgR86PYwbdpAVPpPzFPABs= 2022-09-12T20:22:00.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:f6BnZzgR86PYwbdpAVPpPzFPABs=', 'Content-Length': '0'} 2022-09-12T20:22:00.536 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd558a9880a8e8fa-00631f94e8-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.537 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T20:22:00.538 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:00.539 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:00.539 DEBUG:boto:path=/jdpiak-76/?versions&max-keys=999 2022-09-12T20:22:00.540 DEBUG:boto:auth_path=/jdpiak-76/?versions&max-keys=999 2022-09-12T20:22:00.540 DEBUG:boto:Method: GET 2022-09-12T20:22:00.541 DEBUG:boto:Path: /jdpiak-76/?versions&max-keys=999 2022-09-12T20:22:00.541 DEBUG:boto:Data: 2022-09-12T20:22:00.542 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.542 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:00.543 DEBUG:boto:Port: 8001 2022-09-12T20:22:00.544 DEBUG:boto:Params: {} 2022-09-12T20:22:00.544 DEBUG:boto:Token: None 2022-09-12T20:22:00.545 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/?versions 2022-09-12T20:22:00.545 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Os0zP81PekQi8wcEH7s4YwZ3b+A= 2022-09-12T20:22:00.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Os0zP81PekQi8wcEH7s4YwZ3b+A=', 'Content-Length': '0'} 2022-09-12T20:22:00.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f52b70f799611ba8-00631f94e8-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.550 DEBUG:boto:b'jdpiak-76999falseobj11nulltrue2022-09-12T20:18:36.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:18:36.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:18:36.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:18:36.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:18:36.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:00.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:22:00.552 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:22:00.552 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:22:00.554 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:22:00.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:22:00.556 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T20:22:00.557 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:00.558 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:00.559 DEBUG:boto:path=/jdpiak-76/?versions&max-keys=999 2022-09-12T20:22:00.560 DEBUG:boto:auth_path=/jdpiak-76/?versions&max-keys=999 2022-09-12T20:22:00.561 DEBUG:boto:Method: GET 2022-09-12T20:22:00.562 DEBUG:boto:Path: /jdpiak-76/?versions&max-keys=999 2022-09-12T20:22:00.563 DEBUG:boto:Data: 2022-09-12T20:22:00.564 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.565 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.566 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.567 DEBUG:boto:Params: {} 2022-09-12T20:22:00.568 DEBUG:boto:Token: None 2022-09-12T20:22:00.570 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/?versions 2022-09-12T20:22:00.571 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Os0zP81PekQi8wcEH7s4YwZ3b+A= 2022-09-12T20:22:00.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Os0zP81PekQi8wcEH7s4YwZ3b+A=', 'Content-Length': '0'} 2022-09-12T20:22:00.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b33b27ed149f9db-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.577 DEBUG:boto:b'jdpiak-76999falseobj11nulltrue2022-09-12T20:18:36.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:18:36.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:18:36.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:18:36.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:18:36.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:00.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:22:00.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:22:00.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:22:00.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:22:00.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:22:00.583 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:00.584 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:00.585 DEBUG:boto:path=/jdpiak-76/?versions&max-keys=999 2022-09-12T20:22:00.586 DEBUG:boto:auth_path=/jdpiak-76/?versions&max-keys=999 2022-09-12T20:22:00.587 DEBUG:boto:Method: GET 2022-09-12T20:22:00.588 DEBUG:boto:Path: /jdpiak-76/?versions&max-keys=999 2022-09-12T20:22:00.589 DEBUG:boto:Data: 2022-09-12T20:22:00.590 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.591 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:00.592 DEBUG:boto:Port: 8001 2022-09-12T20:22:00.592 DEBUG:boto:Params: {} 2022-09-12T20:22:00.593 DEBUG:boto:Token: None 2022-09-12T20:22:00.594 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/?versions 2022-09-12T20:22:00.595 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Os0zP81PekQi8wcEH7s4YwZ3b+A= 2022-09-12T20:22:00.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Os0zP81PekQi8wcEH7s4YwZ3b+A=', 'Content-Length': '0'} 2022-09-12T20:22:00.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1dd820cc39808fc-00631f94e8-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.599 DEBUG:boto:b'jdpiak-76999falseobj11nulltrue2022-09-12T20:18:36.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:18:36.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:18:36.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:18:36.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:18:36.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:00.602 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:00.602 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:00.603 DEBUG:boto:path=/jdpiak-76/?versions&max-keys=999 2022-09-12T20:22:00.603 DEBUG:boto:auth_path=/jdpiak-76/?versions&max-keys=999 2022-09-12T20:22:00.604 DEBUG:boto:Method: GET 2022-09-12T20:22:00.604 DEBUG:boto:Path: /jdpiak-76/?versions&max-keys=999 2022-09-12T20:22:00.605 DEBUG:boto:Data: 2022-09-12T20:22:00.605 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.606 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.606 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.607 DEBUG:boto:Params: {} 2022-09-12T20:22:00.608 DEBUG:boto:Token: None 2022-09-12T20:22:00.608 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/?versions 2022-09-12T20:22:00.609 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Os0zP81PekQi8wcEH7s4YwZ3b+A= 2022-09-12T20:22:00.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Os0zP81PekQi8wcEH7s4YwZ3b+A=', 'Content-Length': '0'} 2022-09-12T20:22:00.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccded193bdaa71c4-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.613 DEBUG:boto:b'jdpiak-76999falseobj11nulltrue2022-09-12T20:18:36.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:18:36.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:18:36.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:18:36.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:18:36.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:00.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:22:00.615 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:22:00.615 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:22:00.616 DEBUG:boto:path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:22:00.616 DEBUG:boto:auth_path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:22:00.617 DEBUG:boto:Method: GET 2022-09-12T20:22:00.618 DEBUG:boto:Path: /jdpiak-76/obj11?versionId=null 2022-09-12T20:22:00.618 DEBUG:boto:Data: 2022-09-12T20:22:00.619 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.619 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:00.620 DEBUG:boto:Port: 8001 2022-09-12T20:22:00.620 DEBUG:boto:Params: {} 2022-09-12T20:22:00.621 DEBUG:boto:Token: None 2022-09-12T20:22:00.621 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj11?versionId=null 2022-09-12T20:22:00.622 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SAM7nfcTCF7ho7z036TLyxbo5a0= 2022-09-12T20:22:00.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SAM7nfcTCF7ho7z036TLyxbo5a0=', 'Content-Length': '0'} 2022-09-12T20:22:00.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ab4579cc0f769e5-00631f94e8-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.626 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:22:00.627 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:22:00.627 DEBUG:boto:path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:22:00.628 DEBUG:boto:auth_path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:22:00.628 DEBUG:boto:Method: GET 2022-09-12T20:22:00.629 DEBUG:boto:Path: /jdpiak-76/obj11?versionId=null 2022-09-12T20:22:00.629 DEBUG:boto:Data: 2022-09-12T20:22:00.630 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.630 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.631 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.631 DEBUG:boto:Params: {} 2022-09-12T20:22:00.632 DEBUG:boto:Token: None 2022-09-12T20:22:00.633 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj11?versionId=null 2022-09-12T20:22:00.633 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SAM7nfcTCF7ho7z036TLyxbo5a0= 2022-09-12T20:22:00.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SAM7nfcTCF7ho7z036TLyxbo5a0=', 'Content-Length': '0'} 2022-09-12T20:22:00.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000df761f420a27c81-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.637 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:22:00.638 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:22:00.638 DEBUG:boto:path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:22:00.639 DEBUG:boto:auth_path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:22:00.639 DEBUG:boto:Method: HEAD 2022-09-12T20:22:00.640 DEBUG:boto:Path: /jdpiak-76/obj11?versionId=null 2022-09-12T20:22:00.640 DEBUG:boto:Data: 2022-09-12T20:22:00.641 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.641 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:00.642 DEBUG:boto:Port: 8001 2022-09-12T20:22:00.642 DEBUG:boto:Params: {} 2022-09-12T20:22:00.643 DEBUG:boto:Token: None 2022-09-12T20:22:00.643 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj11?versionId=null 2022-09-12T20:22:00.644 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZCGsUvO2bzY2zR+S8h/ZBYqVREc= 2022-09-12T20:22:00.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZCGsUvO2bzY2zR+S8h/ZBYqVREc=', 'Content-Length': '0'} 2022-09-12T20:22:00.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052967bede0ebf81b-00631f94e8-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.647 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:22:00.648 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:22:00.648 DEBUG:boto:path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:22:00.649 DEBUG:boto:auth_path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:22:00.649 DEBUG:boto:Method: HEAD 2022-09-12T20:22:00.650 DEBUG:boto:Path: /jdpiak-76/obj11?versionId=null 2022-09-12T20:22:00.650 DEBUG:boto:Data: 2022-09-12T20:22:00.651 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.651 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.652 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.652 DEBUG:boto:Params: {} 2022-09-12T20:22:00.653 DEBUG:boto:Token: None 2022-09-12T20:22:00.654 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj11?versionId=null 2022-09-12T20:22:00.654 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZCGsUvO2bzY2zR+S8h/ZBYqVREc= 2022-09-12T20:22:00.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZCGsUvO2bzY2zR+S8h/ZBYqVREc=', 'Content-Length': '0'} 2022-09-12T20:22:00.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001990d1527efa1538-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:22:00.658 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:22:00.659 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:22:00.659 DEBUG:boto:path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:22:00.660 DEBUG:boto:auth_path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:22:00.660 DEBUG:boto:Method: GET 2022-09-12T20:22:00.661 DEBUG:boto:Path: /jdpiak-76/obj11?versionId=null 2022-09-12T20:22:00.662 DEBUG:boto:Data: 2022-09-12T20:22:00.662 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.663 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:00.663 DEBUG:boto:Port: 8001 2022-09-12T20:22:00.664 DEBUG:boto:Params: {} 2022-09-12T20:22:00.665 DEBUG:boto:Token: None 2022-09-12T20:22:00.666 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj11?versionId=null 2022-09-12T20:22:00.666 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SAM7nfcTCF7ho7z036TLyxbo5a0= 2022-09-12T20:22:00.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SAM7nfcTCF7ho7z036TLyxbo5a0=', 'Content-Length': '0'} 2022-09-12T20:22:00.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054aa65e92b690dca-00631f94e8-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.670 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:22:00.670 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:22:00.671 DEBUG:boto:path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:22:00.671 DEBUG:boto:auth_path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:22:00.672 DEBUG:boto:Method: GET 2022-09-12T20:22:00.673 DEBUG:boto:Path: /jdpiak-76/obj11?versionId=null 2022-09-12T20:22:00.673 DEBUG:boto:Data: 2022-09-12T20:22:00.674 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.675 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.675 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.676 DEBUG:boto:Params: {} 2022-09-12T20:22:00.677 DEBUG:boto:Token: None 2022-09-12T20:22:00.677 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj11?versionId=null 2022-09-12T20:22:00.678 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:SAM7nfcTCF7ho7z036TLyxbo5a0= 2022-09-12T20:22:00.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:SAM7nfcTCF7ho7z036TLyxbo5a0=', 'Content-Length': '0'} 2022-09-12T20:22:00.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a67488233f5bfb0b-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.682 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:00.682 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:00.683 DEBUG:boto:path=/jdpiak-76/?prefix=obj11 2022-09-12T20:22:00.683 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj11 2022-09-12T20:22:00.684 DEBUG:boto:Method: GET 2022-09-12T20:22:00.685 DEBUG:boto:Path: /jdpiak-76/?prefix=obj11 2022-09-12T20:22:00.685 DEBUG:boto:Data: 2022-09-12T20:22:00.686 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.687 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:00.687 DEBUG:boto:Port: 8001 2022-09-12T20:22:00.688 DEBUG:boto:Params: {} 2022-09-12T20:22:00.688 DEBUG:boto:Token: None 2022-09-12T20:22:00.689 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/ 2022-09-12T20:22:00.690 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8= 2022-09-12T20:22:00.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8=', 'Content-Length': '0'} 2022-09-12T20:22:00.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a28c25b933d34665-00631f94e8-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.694 DEBUG:boto:b'jdpiak-76obj111000falseobj112022-09-12T20:18:36.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:00.695 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:00.695 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:00.696 DEBUG:boto:path=/jdpiak-76/?prefix=obj11 2022-09-12T20:22:00.697 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj11 2022-09-12T20:22:00.697 DEBUG:boto:Method: GET 2022-09-12T20:22:00.698 DEBUG:boto:Path: /jdpiak-76/?prefix=obj11 2022-09-12T20:22:00.698 DEBUG:boto:Data: 2022-09-12T20:22:00.699 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.699 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.700 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.701 DEBUG:boto:Params: {} 2022-09-12T20:22:00.701 DEBUG:boto:Token: None 2022-09-12T20:22:00.702 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/ 2022-09-12T20:22:00.702 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8= 2022-09-12T20:22:00.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8=', 'Content-Length': '0'} 2022-09-12T20:22:00.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e294ff5f3dc32902-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.707 DEBUG:boto:b'jdpiak-76obj111000falseobj112022-09-12T20:18:36.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:00.708 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:22:00.709 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:22:00.709 DEBUG:boto:Method: HEAD 2022-09-12T20:22:00.710 DEBUG:boto:Path: /jdpiak-76/obj11 2022-09-12T20:22:00.710 DEBUG:boto:Data: 2022-09-12T20:22:00.711 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.711 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:00.712 DEBUG:boto:Port: 8001 2022-09-12T20:22:00.712 DEBUG:boto:Params: {} 2022-09-12T20:22:00.713 DEBUG:boto:Token: None 2022-09-12T20:22:00.714 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj11 2022-09-12T20:22:00.714 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PFXRICWcoh5vxOw59ZXATJLYgH4= 2022-09-12T20:22:00.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PFXRICWcoh5vxOw59ZXATJLYgH4=', 'Content-Length': '0'} 2022-09-12T20:22:00.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041f3c269ccf3793d-00631f94e8-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.718 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:22:00.719 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:22:00.719 DEBUG:boto:Method: HEAD 2022-09-12T20:22:00.720 DEBUG:boto:Path: /jdpiak-76/obj11 2022-09-12T20:22:00.720 DEBUG:boto:Data: 2022-09-12T20:22:00.721 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.721 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.722 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.722 DEBUG:boto:Params: {} 2022-09-12T20:22:00.723 DEBUG:boto:Token: None 2022-09-12T20:22:00.723 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj11 2022-09-12T20:22:00.724 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PFXRICWcoh5vxOw59ZXATJLYgH4= 2022-09-12T20:22:00.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PFXRICWcoh5vxOw59ZXATJLYgH4=', 'Content-Length': '0'} 2022-09-12T20:22:00.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017ba55afb5b62edd-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:22:00.728 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:22:00.729 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:22:00.729 DEBUG:boto:Method: GET 2022-09-12T20:22:00.730 DEBUG:boto:Path: /jdpiak-76/obj11 2022-09-12T20:22:00.731 DEBUG:boto:Data: 2022-09-12T20:22:00.731 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.732 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:00.732 DEBUG:boto:Port: 8001 2022-09-12T20:22:00.733 DEBUG:boto:Params: {} 2022-09-12T20:22:00.733 DEBUG:boto:Token: None 2022-09-12T20:22:00.734 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj11 2022-09-12T20:22:00.735 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BAz0cfMV79QKsLnZhZ6EEuSa6wM= 2022-09-12T20:22:00.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BAz0cfMV79QKsLnZhZ6EEuSa6wM=', 'Content-Length': '0'} 2022-09-12T20:22:00.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071ee9e3689f15c5c-00631f94e8-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.739 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:22:00.739 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:22:00.740 DEBUG:boto:Method: GET 2022-09-12T20:22:00.740 DEBUG:boto:Path: /jdpiak-76/obj11 2022-09-12T20:22:00.741 DEBUG:boto:Data: 2022-09-12T20:22:00.741 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.742 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.742 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.743 DEBUG:boto:Params: {} 2022-09-12T20:22:00.744 DEBUG:boto:Token: None 2022-09-12T20:22:00.744 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj11 2022-09-12T20:22:00.745 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BAz0cfMV79QKsLnZhZ6EEuSa6wM= 2022-09-12T20:22:00.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BAz0cfMV79QKsLnZhZ6EEuSa6wM=', 'Content-Length': '0'} 2022-09-12T20:22:00.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001579a6e37590fff0-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.790 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:00.791 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:00.792 DEBUG:boto:path=/jdpiak-76/?prefix=obj11 2022-09-12T20:22:00.792 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj11 2022-09-12T20:22:00.793 DEBUG:boto:Method: GET 2022-09-12T20:22:00.793 DEBUG:boto:Path: /jdpiak-76/?prefix=obj11 2022-09-12T20:22:00.794 DEBUG:boto:Data: 2022-09-12T20:22:00.794 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.795 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:00.795 DEBUG:boto:Port: 8001 2022-09-12T20:22:00.796 DEBUG:boto:Params: {} 2022-09-12T20:22:00.796 DEBUG:boto:Token: None 2022-09-12T20:22:00.797 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/ 2022-09-12T20:22:00.798 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8= 2022-09-12T20:22:00.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8=', 'Content-Length': '0'} 2022-09-12T20:22:00.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006868b0394071ced0-00631f94e8-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.802 DEBUG:boto:b'jdpiak-76obj111000falseobj112022-09-12T20:18:36.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:00.803 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:00.803 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:00.804 DEBUG:boto:path=/jdpiak-76/?prefix=obj11 2022-09-12T20:22:00.804 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj11 2022-09-12T20:22:00.805 DEBUG:boto:Method: GET 2022-09-12T20:22:00.805 DEBUG:boto:Path: /jdpiak-76/?prefix=obj11 2022-09-12T20:22:00.806 DEBUG:boto:Data: 2022-09-12T20:22:00.806 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.807 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.807 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.808 DEBUG:boto:Params: {} 2022-09-12T20:22:00.808 DEBUG:boto:Token: None 2022-09-12T20:22:00.809 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/ 2022-09-12T20:22:00.810 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8= 2022-09-12T20:22:00.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8=', 'Content-Length': '0'} 2022-09-12T20:22:00.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005851526d24b13d57-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.814 DEBUG:boto:b'jdpiak-76obj111000falseobj112022-09-12T20:18:36.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:00.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:22:00.816 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:22:00.816 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:22:00.817 DEBUG:boto:path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:22:00.818 DEBUG:boto:auth_path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:22:00.818 DEBUG:boto:Method: GET 2022-09-12T20:22:00.819 DEBUG:boto:Path: /jdpiak-76/obj22?versionId=null 2022-09-12T20:22:00.819 DEBUG:boto:Data: 2022-09-12T20:22:00.820 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.820 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:00.821 DEBUG:boto:Port: 8001 2022-09-12T20:22:00.821 DEBUG:boto:Params: {} 2022-09-12T20:22:00.822 DEBUG:boto:Token: None 2022-09-12T20:22:00.823 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj22?versionId=null 2022-09-12T20:22:00.824 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:v9YufbV2we0hoouUdiPeNu5UCKI= 2022-09-12T20:22:00.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:v9YufbV2we0hoouUdiPeNu5UCKI=', 'Content-Length': '0'} 2022-09-12T20:22:00.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b952f65e06101c74-00631f94e8-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.828 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:22:00.829 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:22:00.829 DEBUG:boto:path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:22:00.830 DEBUG:boto:auth_path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:22:00.830 DEBUG:boto:Method: GET 2022-09-12T20:22:00.831 DEBUG:boto:Path: /jdpiak-76/obj22?versionId=null 2022-09-12T20:22:00.831 DEBUG:boto:Data: 2022-09-12T20:22:00.832 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.832 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.833 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.833 DEBUG:boto:Params: {} 2022-09-12T20:22:00.834 DEBUG:boto:Token: None 2022-09-12T20:22:00.835 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj22?versionId=null 2022-09-12T20:22:00.835 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:v9YufbV2we0hoouUdiPeNu5UCKI= 2022-09-12T20:22:00.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:v9YufbV2we0hoouUdiPeNu5UCKI=', 'Content-Length': '0'} 2022-09-12T20:22:00.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bafcad9059ddea42-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.839 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:22:00.840 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:22:00.840 DEBUG:boto:path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:22:00.841 DEBUG:boto:auth_path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:22:00.841 DEBUG:boto:Method: HEAD 2022-09-12T20:22:00.842 DEBUG:boto:Path: /jdpiak-76/obj22?versionId=null 2022-09-12T20:22:00.842 DEBUG:boto:Data: 2022-09-12T20:22:00.843 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.843 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:00.844 DEBUG:boto:Port: 8001 2022-09-12T20:22:00.844 DEBUG:boto:Params: {} 2022-09-12T20:22:00.845 DEBUG:boto:Token: None 2022-09-12T20:22:00.845 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj22?versionId=null 2022-09-12T20:22:00.846 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mCVrqNAPpDWT1+UDQyOi7fq1kak= 2022-09-12T20:22:00.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mCVrqNAPpDWT1+UDQyOi7fq1kak=', 'Content-Length': '0'} 2022-09-12T20:22:00.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5f6b04a958880fe-00631f94e8-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.849 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:22:00.850 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:22:00.850 DEBUG:boto:path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:22:00.851 DEBUG:boto:auth_path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:22:00.851 DEBUG:boto:Method: HEAD 2022-09-12T20:22:00.852 DEBUG:boto:Path: /jdpiak-76/obj22?versionId=null 2022-09-12T20:22:00.852 DEBUG:boto:Data: 2022-09-12T20:22:00.853 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.853 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.854 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.854 DEBUG:boto:Params: {} 2022-09-12T20:22:00.855 DEBUG:boto:Token: None 2022-09-12T20:22:00.856 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj22?versionId=null 2022-09-12T20:22:00.856 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mCVrqNAPpDWT1+UDQyOi7fq1kak= 2022-09-12T20:22:00.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mCVrqNAPpDWT1+UDQyOi7fq1kak=', 'Content-Length': '0'} 2022-09-12T20:22:00.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bbd943767a692d4-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:22:00.860 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:22:00.861 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:22:00.861 DEBUG:boto:path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:22:00.862 DEBUG:boto:auth_path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:22:00.862 DEBUG:boto:Method: GET 2022-09-12T20:22:00.863 DEBUG:boto:Path: /jdpiak-76/obj22?versionId=null 2022-09-12T20:22:00.863 DEBUG:boto:Data: 2022-09-12T20:22:00.864 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.865 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:00.865 DEBUG:boto:Port: 8001 2022-09-12T20:22:00.866 DEBUG:boto:Params: {} 2022-09-12T20:22:00.867 DEBUG:boto:Token: None 2022-09-12T20:22:00.867 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj22?versionId=null 2022-09-12T20:22:00.868 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:v9YufbV2we0hoouUdiPeNu5UCKI= 2022-09-12T20:22:00.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:v9YufbV2we0hoouUdiPeNu5UCKI=', 'Content-Length': '0'} 2022-09-12T20:22:00.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009db936cdde97aedf-00631f94e8-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.914 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:22:00.916 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:22:00.916 DEBUG:boto:path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:22:00.917 DEBUG:boto:auth_path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:22:00.918 DEBUG:boto:Method: GET 2022-09-12T20:22:00.919 DEBUG:boto:Path: /jdpiak-76/obj22?versionId=null 2022-09-12T20:22:00.920 DEBUG:boto:Data: 2022-09-12T20:22:00.920 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.921 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.921 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.922 DEBUG:boto:Params: {} 2022-09-12T20:22:00.922 DEBUG:boto:Token: None 2022-09-12T20:22:00.923 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/obj22?versionId=null 2022-09-12T20:22:00.924 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:v9YufbV2we0hoouUdiPeNu5UCKI= 2022-09-12T20:22:00.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:v9YufbV2we0hoouUdiPeNu5UCKI=', 'Content-Length': '0'} 2022-09-12T20:22:00.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090bb8d8bcfa7ee78-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.927 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:00.928 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:00.929 DEBUG:boto:path=/jdpiak-76/?prefix=obj22 2022-09-12T20:22:00.929 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj22 2022-09-12T20:22:00.930 DEBUG:boto:Method: GET 2022-09-12T20:22:00.930 DEBUG:boto:Path: /jdpiak-76/?prefix=obj22 2022-09-12T20:22:00.931 DEBUG:boto:Data: 2022-09-12T20:22:00.931 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.932 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:00.932 DEBUG:boto:Port: 8001 2022-09-12T20:22:00.933 DEBUG:boto:Params: {} 2022-09-12T20:22:00.934 DEBUG:boto:Token: None 2022-09-12T20:22:00.934 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/ 2022-09-12T20:22:00.935 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8= 2022-09-12T20:22:00.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8=', 'Content-Length': '0'} 2022-09-12T20:22:00.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001de7f55857999568-00631f94e8-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:00.941 DEBUG:boto:b'jdpiak-76obj221000falseobj222022-09-12T20:18:36.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:00.942 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:00.943 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:00.945 DEBUG:boto:path=/jdpiak-76/?prefix=obj22 2022-09-12T20:22:00.946 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj22 2022-09-12T20:22:00.947 DEBUG:boto:Method: GET 2022-09-12T20:22:00.947 DEBUG:boto:Path: /jdpiak-76/?prefix=obj22 2022-09-12T20:22:00.948 DEBUG:boto:Data: 2022-09-12T20:22:00.948 DEBUG:boto:Headers: {} 2022-09-12T20:22:00.949 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:00.950 DEBUG:boto:Port: 8000 2022-09-12T20:22:00.951 DEBUG:boto:Params: {} 2022-09-12T20:22:00.953 DEBUG:boto:Token: None 2022-09-12T20:22:00.954 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:00 GMT /jdpiak-76/ 2022-09-12T20:22:00.955 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8= 2022-09-12T20:22:00.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:00 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:q0gd9CpUfanD6k0gvRhaERYM7B8=', 'Content-Length': '0'} 2022-09-12T20:22:00.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d642d8c6a443a924-00631f94e8-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.002 DEBUG:boto:b'jdpiak-76obj221000falseobj222022-09-12T20:18:36.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:01.004 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:22:01.005 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:22:01.006 DEBUG:boto:Method: HEAD 2022-09-12T20:22:01.007 DEBUG:boto:Path: /jdpiak-76/obj22 2022-09-12T20:22:01.008 DEBUG:boto:Data: 2022-09-12T20:22:01.009 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.009 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.010 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.011 DEBUG:boto:Params: {} 2022-09-12T20:22:01.012 DEBUG:boto:Token: None 2022-09-12T20:22:01.013 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj22 2022-09-12T20:22:01.014 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nr5VFDquM5w67t6k6CCu1eVLkH0= 2022-09-12T20:22:01.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nr5VFDquM5w67t6k6CCu1eVLkH0=', 'Content-Length': '0'} 2022-09-12T20:22:01.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee406437d1e966b5-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.019 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:22:01.020 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:22:01.021 DEBUG:boto:Method: HEAD 2022-09-12T20:22:01.022 DEBUG:boto:Path: /jdpiak-76/obj22 2022-09-12T20:22:01.023 DEBUG:boto:Data: 2022-09-12T20:22:01.024 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.025 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:01.026 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.027 DEBUG:boto:Params: {} 2022-09-12T20:22:01.028 DEBUG:boto:Token: None 2022-09-12T20:22:01.029 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj22 2022-09-12T20:22:01.030 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Nr5VFDquM5w67t6k6CCu1eVLkH0= 2022-09-12T20:22:01.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Nr5VFDquM5w67t6k6CCu1eVLkH0=', 'Content-Length': '0'} 2022-09-12T20:22:01.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ba1d9f14f51ff7d-00631f94e9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:22:01.036 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:22:01.037 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:22:01.038 DEBUG:boto:Method: GET 2022-09-12T20:22:01.038 DEBUG:boto:Path: /jdpiak-76/obj22 2022-09-12T20:22:01.039 DEBUG:boto:Data: 2022-09-12T20:22:01.039 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.040 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.041 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.042 DEBUG:boto:Params: {} 2022-09-12T20:22:01.043 DEBUG:boto:Token: None 2022-09-12T20:22:01.043 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj22 2022-09-12T20:22:01.044 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TiXVlkEpY9c85fy+3vkF1HDAp1I= 2022-09-12T20:22:01.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TiXVlkEpY9c85fy+3vkF1HDAp1I=', 'Content-Length': '0'} 2022-09-12T20:22:01.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006317907d33c16904-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.048 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:22:01.048 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:22:01.049 DEBUG:boto:Method: GET 2022-09-12T20:22:01.049 DEBUG:boto:Path: /jdpiak-76/obj22 2022-09-12T20:22:01.050 DEBUG:boto:Data: 2022-09-12T20:22:01.051 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.051 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:01.052 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.052 DEBUG:boto:Params: {} 2022-09-12T20:22:01.053 DEBUG:boto:Token: None 2022-09-12T20:22:01.054 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj22 2022-09-12T20:22:01.054 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TiXVlkEpY9c85fy+3vkF1HDAp1I= 2022-09-12T20:22:01.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TiXVlkEpY9c85fy+3vkF1HDAp1I=', 'Content-Length': '0'} 2022-09-12T20:22:01.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051d0ea1e8e32a2d5-00631f94e9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.058 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:01.059 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:01.059 DEBUG:boto:path=/jdpiak-76/?prefix=obj22 2022-09-12T20:22:01.060 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj22 2022-09-12T20:22:01.060 DEBUG:boto:Method: GET 2022-09-12T20:22:01.061 DEBUG:boto:Path: /jdpiak-76/?prefix=obj22 2022-09-12T20:22:01.061 DEBUG:boto:Data: 2022-09-12T20:22:01.062 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.062 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.063 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.063 DEBUG:boto:Params: {} 2022-09-12T20:22:01.064 DEBUG:boto:Token: None 2022-09-12T20:22:01.065 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/ 2022-09-12T20:22:01.065 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58= 2022-09-12T20:22:01.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58=', 'Content-Length': '0'} 2022-09-12T20:22:01.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063c891327a487eb1-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.114 DEBUG:boto:b'jdpiak-76obj221000falseobj222022-09-12T20:18:36.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:01.115 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:01.116 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:01.117 DEBUG:boto:path=/jdpiak-76/?prefix=obj22 2022-09-12T20:22:01.117 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj22 2022-09-12T20:22:01.118 DEBUG:boto:Method: GET 2022-09-12T20:22:01.118 DEBUG:boto:Path: /jdpiak-76/?prefix=obj22 2022-09-12T20:22:01.119 DEBUG:boto:Data: 2022-09-12T20:22:01.119 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.120 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:01.120 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.121 DEBUG:boto:Params: {} 2022-09-12T20:22:01.122 DEBUG:boto:Token: None 2022-09-12T20:22:01.122 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/ 2022-09-12T20:22:01.123 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58= 2022-09-12T20:22:01.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58=', 'Content-Length': '0'} 2022-09-12T20:22:01.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef99939358eceb0a-00631f94e9-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.128 DEBUG:boto:b'jdpiak-76obj221000falseobj222022-09-12T20:18:36.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:01.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:22:01.129 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:22:01.130 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:22:01.130 DEBUG:boto:path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:22:01.131 DEBUG:boto:auth_path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:22:01.131 DEBUG:boto:Method: GET 2022-09-12T20:22:01.132 DEBUG:boto:Path: /jdpiak-76/obj33?versionId=null 2022-09-12T20:22:01.132 DEBUG:boto:Data: 2022-09-12T20:22:01.133 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.133 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.134 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.135 DEBUG:boto:Params: {} 2022-09-12T20:22:01.135 DEBUG:boto:Token: None 2022-09-12T20:22:01.136 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj33?versionId=null 2022-09-12T20:22:01.136 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OtzIzhmWVQSQJls/iwkv1sUbHOU= 2022-09-12T20:22:01.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OtzIzhmWVQSQJls/iwkv1sUbHOU=', 'Content-Length': '0'} 2022-09-12T20:22:01.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7fa84738034c180-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.141 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:22:01.141 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:22:01.142 DEBUG:boto:path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:22:01.142 DEBUG:boto:auth_path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:22:01.143 DEBUG:boto:Method: GET 2022-09-12T20:22:01.144 DEBUG:boto:Path: /jdpiak-76/obj33?versionId=null 2022-09-12T20:22:01.144 DEBUG:boto:Data: 2022-09-12T20:22:01.145 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.145 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:01.146 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.146 DEBUG:boto:Params: {} 2022-09-12T20:22:01.147 DEBUG:boto:Token: None 2022-09-12T20:22:01.148 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj33?versionId=null 2022-09-12T20:22:01.148 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OtzIzhmWVQSQJls/iwkv1sUbHOU= 2022-09-12T20:22:01.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OtzIzhmWVQSQJls/iwkv1sUbHOU=', 'Content-Length': '0'} 2022-09-12T20:22:01.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007345b119fbc73dff-00631f94e9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.194 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:22:01.195 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:22:01.195 DEBUG:boto:path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:22:01.196 DEBUG:boto:auth_path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:22:01.196 DEBUG:boto:Method: HEAD 2022-09-12T20:22:01.197 DEBUG:boto:Path: /jdpiak-76/obj33?versionId=null 2022-09-12T20:22:01.198 DEBUG:boto:Data: 2022-09-12T20:22:01.198 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.199 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.199 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.200 DEBUG:boto:Params: {} 2022-09-12T20:22:01.200 DEBUG:boto:Token: None 2022-09-12T20:22:01.201 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj33?versionId=null 2022-09-12T20:22:01.202 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IRpeStAtnJHByi0r653lAIXZJBA= 2022-09-12T20:22:01.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IRpeStAtnJHByi0r653lAIXZJBA=', 'Content-Length': '0'} 2022-09-12T20:22:01.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce552162b4b4721a-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.205 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:22:01.206 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:22:01.206 DEBUG:boto:path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:22:01.207 DEBUG:boto:auth_path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:22:01.207 DEBUG:boto:Method: HEAD 2022-09-12T20:22:01.208 DEBUG:boto:Path: /jdpiak-76/obj33?versionId=null 2022-09-12T20:22:01.208 DEBUG:boto:Data: 2022-09-12T20:22:01.209 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.209 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:01.210 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.211 DEBUG:boto:Params: {} 2022-09-12T20:22:01.211 DEBUG:boto:Token: None 2022-09-12T20:22:01.212 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj33?versionId=null 2022-09-12T20:22:01.212 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IRpeStAtnJHByi0r653lAIXZJBA= 2022-09-12T20:22:01.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IRpeStAtnJHByi0r653lAIXZJBA=', 'Content-Length': '0'} 2022-09-12T20:22:01.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000a850bcd1f80e4f-00631f94e9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:22:01.217 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:22:01.217 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:22:01.218 DEBUG:boto:path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:22:01.218 DEBUG:boto:auth_path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:22:01.219 DEBUG:boto:Method: GET 2022-09-12T20:22:01.220 DEBUG:boto:Path: /jdpiak-76/obj33?versionId=null 2022-09-12T20:22:01.220 DEBUG:boto:Data: 2022-09-12T20:22:01.221 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.221 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.222 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.222 DEBUG:boto:Params: {} 2022-09-12T20:22:01.223 DEBUG:boto:Token: None 2022-09-12T20:22:01.224 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj33?versionId=null 2022-09-12T20:22:01.224 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OtzIzhmWVQSQJls/iwkv1sUbHOU= 2022-09-12T20:22:01.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OtzIzhmWVQSQJls/iwkv1sUbHOU=', 'Content-Length': '0'} 2022-09-12T20:22:01.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f0ff7276618727e-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.228 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:22:01.229 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:22:01.229 DEBUG:boto:path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:22:01.230 DEBUG:boto:auth_path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:22:01.230 DEBUG:boto:Method: GET 2022-09-12T20:22:01.231 DEBUG:boto:Path: /jdpiak-76/obj33?versionId=null 2022-09-12T20:22:01.231 DEBUG:boto:Data: 2022-09-12T20:22:01.232 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.232 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:01.233 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.233 DEBUG:boto:Params: {} 2022-09-12T20:22:01.234 DEBUG:boto:Token: None 2022-09-12T20:22:01.235 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj33?versionId=null 2022-09-12T20:22:01.235 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OtzIzhmWVQSQJls/iwkv1sUbHOU= 2022-09-12T20:22:01.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OtzIzhmWVQSQJls/iwkv1sUbHOU=', 'Content-Length': '0'} 2022-09-12T20:22:01.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f87dff9e4e470ba-00631f94e9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.239 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:01.239 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:01.240 DEBUG:boto:path=/jdpiak-76/?prefix=obj33 2022-09-12T20:22:01.240 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj33 2022-09-12T20:22:01.241 DEBUG:boto:Method: GET 2022-09-12T20:22:01.242 DEBUG:boto:Path: /jdpiak-76/?prefix=obj33 2022-09-12T20:22:01.242 DEBUG:boto:Data: 2022-09-12T20:22:01.243 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.243 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.244 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.245 DEBUG:boto:Params: {} 2022-09-12T20:22:01.246 DEBUG:boto:Token: None 2022-09-12T20:22:01.246 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/ 2022-09-12T20:22:01.247 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58= 2022-09-12T20:22:01.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58=', 'Content-Length': '0'} 2022-09-12T20:22:01.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000494bb2baec58acd5-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.251 DEBUG:boto:b'jdpiak-76obj331000falseobj332022-09-12T20:18:36.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:01.252 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:01.253 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:01.253 DEBUG:boto:path=/jdpiak-76/?prefix=obj33 2022-09-12T20:22:01.254 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj33 2022-09-12T20:22:01.254 DEBUG:boto:Method: GET 2022-09-12T20:22:01.255 DEBUG:boto:Path: /jdpiak-76/?prefix=obj33 2022-09-12T20:22:01.256 DEBUG:boto:Data: 2022-09-12T20:22:01.257 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.258 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:01.258 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.259 DEBUG:boto:Params: {} 2022-09-12T20:22:01.259 DEBUG:boto:Token: None 2022-09-12T20:22:01.260 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/ 2022-09-12T20:22:01.260 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58= 2022-09-12T20:22:01.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58=', 'Content-Length': '0'} 2022-09-12T20:22:01.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034f08d3965cdf8b8-00631f94e9-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.265 DEBUG:boto:b'jdpiak-76obj331000falseobj332022-09-12T20:18:36.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:01.265 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:22:01.266 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:22:01.267 DEBUG:boto:Method: HEAD 2022-09-12T20:22:01.268 DEBUG:boto:Path: /jdpiak-76/obj33 2022-09-12T20:22:01.268 DEBUG:boto:Data: 2022-09-12T20:22:01.269 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.269 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.270 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.271 DEBUG:boto:Params: {} 2022-09-12T20:22:01.271 DEBUG:boto:Token: None 2022-09-12T20:22:01.272 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj33 2022-09-12T20:22:01.272 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fc37T9reWvVW/BiH7ni3fACGFWc= 2022-09-12T20:22:01.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fc37T9reWvVW/BiH7ni3fACGFWc=', 'Content-Length': '0'} 2022-09-12T20:22:01.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025fef11ef1153a3b-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.276 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:22:01.276 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:22:01.277 DEBUG:boto:Method: HEAD 2022-09-12T20:22:01.277 DEBUG:boto:Path: /jdpiak-76/obj33 2022-09-12T20:22:01.278 DEBUG:boto:Data: 2022-09-12T20:22:01.278 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.279 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:01.279 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.280 DEBUG:boto:Params: {} 2022-09-12T20:22:01.280 DEBUG:boto:Token: None 2022-09-12T20:22:01.281 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj33 2022-09-12T20:22:01.282 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Fc37T9reWvVW/BiH7ni3fACGFWc= 2022-09-12T20:22:01.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Fc37T9reWvVW/BiH7ni3fACGFWc=', 'Content-Length': '0'} 2022-09-12T20:22:01.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000865ba8dd3dea3c3d-00631f94e9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:22:01.286 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:22:01.286 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:22:01.287 DEBUG:boto:Method: GET 2022-09-12T20:22:01.287 DEBUG:boto:Path: /jdpiak-76/obj33 2022-09-12T20:22:01.288 DEBUG:boto:Data: 2022-09-12T20:22:01.288 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.289 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.289 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.290 DEBUG:boto:Params: {} 2022-09-12T20:22:01.290 DEBUG:boto:Token: None 2022-09-12T20:22:01.291 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj33 2022-09-12T20:22:01.292 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gq7yWY1RK/5deo4sTLOBfF9FkXE= 2022-09-12T20:22:01.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gq7yWY1RK/5deo4sTLOBfF9FkXE=', 'Content-Length': '0'} 2022-09-12T20:22:01.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039405a567ad51e86-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.295 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:22:01.296 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:22:01.296 DEBUG:boto:Method: GET 2022-09-12T20:22:01.297 DEBUG:boto:Path: /jdpiak-76/obj33 2022-09-12T20:22:01.297 DEBUG:boto:Data: 2022-09-12T20:22:01.298 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.299 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:01.299 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.300 DEBUG:boto:Params: {} 2022-09-12T20:22:01.300 DEBUG:boto:Token: None 2022-09-12T20:22:01.301 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj33 2022-09-12T20:22:01.301 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gq7yWY1RK/5deo4sTLOBfF9FkXE= 2022-09-12T20:22:01.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gq7yWY1RK/5deo4sTLOBfF9FkXE=', 'Content-Length': '0'} 2022-09-12T20:22:01.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e605664740fbd8a-00631f94e9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.305 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:01.306 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:01.306 DEBUG:boto:path=/jdpiak-76/?prefix=obj33 2022-09-12T20:22:01.307 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj33 2022-09-12T20:22:01.307 DEBUG:boto:Method: GET 2022-09-12T20:22:01.308 DEBUG:boto:Path: /jdpiak-76/?prefix=obj33 2022-09-12T20:22:01.308 DEBUG:boto:Data: 2022-09-12T20:22:01.309 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.309 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.310 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.310 DEBUG:boto:Params: {} 2022-09-12T20:22:01.311 DEBUG:boto:Token: None 2022-09-12T20:22:01.312 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/ 2022-09-12T20:22:01.312 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58= 2022-09-12T20:22:01.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58=', 'Content-Length': '0'} 2022-09-12T20:22:01.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052979dc675918a31-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.362 DEBUG:boto:b'jdpiak-76obj331000falseobj332022-09-12T20:18:36.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:01.363 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:01.364 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:01.365 DEBUG:boto:path=/jdpiak-76/?prefix=obj33 2022-09-12T20:22:01.365 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj33 2022-09-12T20:22:01.366 DEBUG:boto:Method: GET 2022-09-12T20:22:01.366 DEBUG:boto:Path: /jdpiak-76/?prefix=obj33 2022-09-12T20:22:01.367 DEBUG:boto:Data: 2022-09-12T20:22:01.368 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.368 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:01.369 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.369 DEBUG:boto:Params: {} 2022-09-12T20:22:01.370 DEBUG:boto:Token: None 2022-09-12T20:22:01.371 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/ 2022-09-12T20:22:01.371 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58= 2022-09-12T20:22:01.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58=', 'Content-Length': '0'} 2022-09-12T20:22:01.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6d755f20b3c76b0-00631f94e9-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.375 DEBUG:boto:b'jdpiak-76obj331000falseobj332022-09-12T20:18:36.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:01.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:22:01.377 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:01.377 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:01.378 DEBUG:boto:path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:01.378 DEBUG:boto:auth_path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:01.379 DEBUG:boto:Method: GET 2022-09-12T20:22:01.379 DEBUG:boto:Path: /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:01.380 DEBUG:boto:Data: 2022-09-12T20:22:01.381 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.381 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.382 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.382 DEBUG:boto:Params: {} 2022-09-12T20:22:01.383 DEBUG:boto:Token: None 2022-09-12T20:22:01.383 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:01.384 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BOOhnUadc+58eCbUd+ZUEupRlRw= 2022-09-12T20:22:01.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BOOhnUadc+58eCbUd+ZUEupRlRw=', 'Content-Length': '0'} 2022-09-12T20:22:01.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000072a846e540edacb-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.388 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:01.389 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:01.390 DEBUG:boto:path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:01.390 DEBUG:boto:auth_path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:01.391 DEBUG:boto:Method: GET 2022-09-12T20:22:01.391 DEBUG:boto:Path: /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:01.392 DEBUG:boto:Data: 2022-09-12T20:22:01.392 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.393 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:01.393 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.394 DEBUG:boto:Params: {} 2022-09-12T20:22:01.395 DEBUG:boto:Token: None 2022-09-12T20:22:01.395 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:01.396 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BOOhnUadc+58eCbUd+ZUEupRlRw= 2022-09-12T20:22:01.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BOOhnUadc+58eCbUd+ZUEupRlRw=', 'Content-Length': '0'} 2022-09-12T20:22:01.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb5766c160db85c9-00631f94e9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.442 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:01.443 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:01.443 DEBUG:boto:path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:01.444 DEBUG:boto:auth_path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:01.444 DEBUG:boto:Method: HEAD 2022-09-12T20:22:01.445 DEBUG:boto:Path: /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:01.445 DEBUG:boto:Data: 2022-09-12T20:22:01.446 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.446 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.447 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.447 DEBUG:boto:Params: {} 2022-09-12T20:22:01.448 DEBUG:boto:Token: None 2022-09-12T20:22:01.449 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:01.449 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TM5OX10kX6kFH5mbRT4Qw6rlHv4= 2022-09-12T20:22:01.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TM5OX10kX6kFH5mbRT4Qw6rlHv4=', 'Content-Length': '0'} 2022-09-12T20:22:01.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dbf8b94ddcd614c-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.453 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:01.453 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:01.454 DEBUG:boto:path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:01.454 DEBUG:boto:auth_path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:01.455 DEBUG:boto:Method: HEAD 2022-09-12T20:22:01.455 DEBUG:boto:Path: /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:01.456 DEBUG:boto:Data: 2022-09-12T20:22:01.456 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.457 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:01.457 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.458 DEBUG:boto:Params: {} 2022-09-12T20:22:01.458 DEBUG:boto:Token: None 2022-09-12T20:22:01.459 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:01.459 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:TM5OX10kX6kFH5mbRT4Qw6rlHv4= 2022-09-12T20:22:01.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:TM5OX10kX6kFH5mbRT4Qw6rlHv4=', 'Content-Length': '0'} 2022-09-12T20:22:01.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2def0f3c9c14732-00631f94e9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:22:01.463 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:01.464 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:01.464 DEBUG:boto:path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:01.465 DEBUG:boto:auth_path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:01.465 DEBUG:boto:Method: GET 2022-09-12T20:22:01.466 DEBUG:boto:Path: /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:01.466 DEBUG:boto:Data: 2022-09-12T20:22:01.467 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.468 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.468 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.469 DEBUG:boto:Params: {} 2022-09-12T20:22:01.469 DEBUG:boto:Token: None 2022-09-12T20:22:01.470 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:01.470 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BOOhnUadc+58eCbUd+ZUEupRlRw= 2022-09-12T20:22:01.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BOOhnUadc+58eCbUd+ZUEupRlRw=', 'Content-Length': '0'} 2022-09-12T20:22:01.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f648fdbd0af4de64-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.474 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:01.475 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:01.475 DEBUG:boto:path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:01.476 DEBUG:boto:auth_path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:01.476 DEBUG:boto:Method: GET 2022-09-12T20:22:01.477 DEBUG:boto:Path: /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:01.477 DEBUG:boto:Data: 2022-09-12T20:22:01.478 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.478 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:01.479 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.480 DEBUG:boto:Params: {} 2022-09-12T20:22:01.480 DEBUG:boto:Token: None 2022-09-12T20:22:01.481 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:01.482 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BOOhnUadc+58eCbUd+ZUEupRlRw= 2022-09-12T20:22:01.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BOOhnUadc+58eCbUd+ZUEupRlRw=', 'Content-Length': '0'} 2022-09-12T20:22:01.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099a0f8f2c7eeb85e-00631f94e9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.486 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:01.486 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:01.487 DEBUG:boto:path=/jdpiak-76/?prefix=obj44 2022-09-12T20:22:01.487 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj44 2022-09-12T20:22:01.488 DEBUG:boto:Method: GET 2022-09-12T20:22:01.488 DEBUG:boto:Path: /jdpiak-76/?prefix=obj44 2022-09-12T20:22:01.489 DEBUG:boto:Data: 2022-09-12T20:22:01.490 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.490 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.491 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.491 DEBUG:boto:Params: {} 2022-09-12T20:22:01.492 DEBUG:boto:Token: None 2022-09-12T20:22:01.492 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/ 2022-09-12T20:22:01.493 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58= 2022-09-12T20:22:01.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58=', 'Content-Length': '0'} 2022-09-12T20:22:01.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cac6ba324ebcc8ef-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.497 DEBUG:boto:b'jdpiak-76obj441000falseobj442022-09-12T20:18:36.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:01.498 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:01.499 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:01.499 DEBUG:boto:path=/jdpiak-76/?prefix=obj44 2022-09-12T20:22:01.500 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj44 2022-09-12T20:22:01.500 DEBUG:boto:Method: GET 2022-09-12T20:22:01.501 DEBUG:boto:Path: /jdpiak-76/?prefix=obj44 2022-09-12T20:22:01.502 DEBUG:boto:Data: 2022-09-12T20:22:01.502 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.503 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:01.503 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.504 DEBUG:boto:Params: {} 2022-09-12T20:22:01.504 DEBUG:boto:Token: None 2022-09-12T20:22:01.505 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/ 2022-09-12T20:22:01.506 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58= 2022-09-12T20:22:01.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58=', 'Content-Length': '0'} 2022-09-12T20:22:01.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed236199a72f883f-00631f94e9-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.510 DEBUG:boto:b'jdpiak-76obj441000falseobj442022-09-12T20:18:36.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:01.511 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:01.511 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:01.512 DEBUG:boto:Method: HEAD 2022-09-12T20:22:01.512 DEBUG:boto:Path: /jdpiak-76/obj44 2022-09-12T20:22:01.513 DEBUG:boto:Data: 2022-09-12T20:22:01.513 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.514 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.514 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.515 DEBUG:boto:Params: {} 2022-09-12T20:22:01.516 DEBUG:boto:Token: None 2022-09-12T20:22:01.516 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj44 2022-09-12T20:22:01.517 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sOXSbt4UNVEJT0CTO1ufIRaA47Y= 2022-09-12T20:22:01.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sOXSbt4UNVEJT0CTO1ufIRaA47Y=', 'Content-Length': '0'} 2022-09-12T20:22:01.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b0d3638349103a4-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.520 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:01.521 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:01.521 DEBUG:boto:Method: HEAD 2022-09-12T20:22:01.522 DEBUG:boto:Path: /jdpiak-76/obj44 2022-09-12T20:22:01.522 DEBUG:boto:Data: 2022-09-12T20:22:01.523 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.524 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:01.524 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.525 DEBUG:boto:Params: {} 2022-09-12T20:22:01.525 DEBUG:boto:Token: None 2022-09-12T20:22:01.526 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj44 2022-09-12T20:22:01.527 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:sOXSbt4UNVEJT0CTO1ufIRaA47Y= 2022-09-12T20:22:01.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:sOXSbt4UNVEJT0CTO1ufIRaA47Y=', 'Content-Length': '0'} 2022-09-12T20:22:01.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000791759ae73a5fc64-00631f94e9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:22:01.531 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:01.531 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:01.532 DEBUG:boto:Method: GET 2022-09-12T20:22:01.532 DEBUG:boto:Path: /jdpiak-76/obj44 2022-09-12T20:22:01.533 DEBUG:boto:Data: 2022-09-12T20:22:01.533 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.534 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.535 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.535 DEBUG:boto:Params: {} 2022-09-12T20:22:01.536 DEBUG:boto:Token: None 2022-09-12T20:22:01.536 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj44 2022-09-12T20:22:01.537 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EYPYDNWe0jeALEmUG5uQw1WDkGM= 2022-09-12T20:22:01.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EYPYDNWe0jeALEmUG5uQw1WDkGM=', 'Content-Length': '0'} 2022-09-12T20:22:01.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d553c62d08a3bb50-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.541 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:01.541 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:01.542 DEBUG:boto:Method: GET 2022-09-12T20:22:01.542 DEBUG:boto:Path: /jdpiak-76/obj44 2022-09-12T20:22:01.543 DEBUG:boto:Data: 2022-09-12T20:22:01.543 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.544 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:01.544 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.545 DEBUG:boto:Params: {} 2022-09-12T20:22:01.545 DEBUG:boto:Token: None 2022-09-12T20:22:01.546 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj44 2022-09-12T20:22:01.547 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EYPYDNWe0jeALEmUG5uQw1WDkGM= 2022-09-12T20:22:01.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EYPYDNWe0jeALEmUG5uQw1WDkGM=', 'Content-Length': '0'} 2022-09-12T20:22:01.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004afe766964278a84-00631f94e9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.551 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:01.551 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:01.552 DEBUG:boto:path=/jdpiak-76/?prefix=obj44 2022-09-12T20:22:01.552 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj44 2022-09-12T20:22:01.553 DEBUG:boto:Method: GET 2022-09-12T20:22:01.553 DEBUG:boto:Path: /jdpiak-76/?prefix=obj44 2022-09-12T20:22:01.554 DEBUG:boto:Data: 2022-09-12T20:22:01.554 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.555 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.556 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.556 DEBUG:boto:Params: {} 2022-09-12T20:22:01.557 DEBUG:boto:Token: None 2022-09-12T20:22:01.558 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/ 2022-09-12T20:22:01.558 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58= 2022-09-12T20:22:01.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58=', 'Content-Length': '0'} 2022-09-12T20:22:01.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c721141e7cdec90f-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.574 DEBUG:boto:b'jdpiak-76obj441000falseobj442022-09-12T20:18:36.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:01.575 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:01.575 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:01.576 DEBUG:boto:path=/jdpiak-76/?prefix=obj44 2022-09-12T20:22:01.577 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj44 2022-09-12T20:22:01.577 DEBUG:boto:Method: GET 2022-09-12T20:22:01.578 DEBUG:boto:Path: /jdpiak-76/?prefix=obj44 2022-09-12T20:22:01.578 DEBUG:boto:Data: 2022-09-12T20:22:01.579 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.580 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:01.580 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.581 DEBUG:boto:Params: {} 2022-09-12T20:22:01.581 DEBUG:boto:Token: None 2022-09-12T20:22:01.582 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/ 2022-09-12T20:22:01.582 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58= 2022-09-12T20:22:01.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58=', 'Content-Length': '0'} 2022-09-12T20:22:01.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d389eb1b393071ec-00631f94e9-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.630 DEBUG:boto:b'jdpiak-76obj441000falseobj442022-09-12T20:18:36.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:01.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:22:01.632 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:01.632 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:01.633 DEBUG:boto:path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:01.633 DEBUG:boto:auth_path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:01.634 DEBUG:boto:Method: GET 2022-09-12T20:22:01.634 DEBUG:boto:Path: /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:01.635 DEBUG:boto:Data: 2022-09-12T20:22:01.635 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.636 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.636 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.637 DEBUG:boto:Params: {} 2022-09-12T20:22:01.638 DEBUG:boto:Token: None 2022-09-12T20:22:01.639 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:01.639 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p7rraL3RtQ9qdviWDp83jzYzljA= 2022-09-12T20:22:01.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p7rraL3RtQ9qdviWDp83jzYzljA=', 'Content-Length': '0'} 2022-09-12T20:22:01.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000914345c96a2aa94a-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.643 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:01.644 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:01.644 DEBUG:boto:path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:01.645 DEBUG:boto:auth_path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:01.645 DEBUG:boto:Method: GET 2022-09-12T20:22:01.646 DEBUG:boto:Path: /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:01.646 DEBUG:boto:Data: 2022-09-12T20:22:01.647 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.648 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:01.648 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.649 DEBUG:boto:Params: {} 2022-09-12T20:22:01.649 DEBUG:boto:Token: None 2022-09-12T20:22:01.650 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:01.651 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p7rraL3RtQ9qdviWDp83jzYzljA= 2022-09-12T20:22:01.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p7rraL3RtQ9qdviWDp83jzYzljA=', 'Content-Length': '0'} 2022-09-12T20:22:01.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000274b7c63fc79cc82-00631f94e9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.655 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:01.655 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:01.656 DEBUG:boto:path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:01.656 DEBUG:boto:auth_path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:01.657 DEBUG:boto:Method: HEAD 2022-09-12T20:22:01.657 DEBUG:boto:Path: /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:01.658 DEBUG:boto:Data: 2022-09-12T20:22:01.658 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.659 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.659 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.660 DEBUG:boto:Params: {} 2022-09-12T20:22:01.660 DEBUG:boto:Token: None 2022-09-12T20:22:01.661 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:01.661 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PBSnlbCgywL0yfTNMaCyslsndOE= 2022-09-12T20:22:01.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PBSnlbCgywL0yfTNMaCyslsndOE=', 'Content-Length': '0'} 2022-09-12T20:22:01.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000459f395b1a617350-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.665 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:01.665 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:01.666 DEBUG:boto:path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:01.666 DEBUG:boto:auth_path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:01.667 DEBUG:boto:Method: HEAD 2022-09-12T20:22:01.667 DEBUG:boto:Path: /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:01.668 DEBUG:boto:Data: 2022-09-12T20:22:01.668 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.669 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:01.669 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.670 DEBUG:boto:Params: {} 2022-09-12T20:22:01.670 DEBUG:boto:Token: None 2022-09-12T20:22:01.671 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:01.672 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:PBSnlbCgywL0yfTNMaCyslsndOE= 2022-09-12T20:22:01.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:PBSnlbCgywL0yfTNMaCyslsndOE=', 'Content-Length': '0'} 2022-09-12T20:22:01.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dd4a67b90aa0481-00631f94e9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:22:01.676 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:01.677 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:01.677 DEBUG:boto:path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:01.678 DEBUG:boto:auth_path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:01.678 DEBUG:boto:Method: GET 2022-09-12T20:22:01.679 DEBUG:boto:Path: /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:01.679 DEBUG:boto:Data: 2022-09-12T20:22:01.680 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.681 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.681 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.682 DEBUG:boto:Params: {} 2022-09-12T20:22:01.682 DEBUG:boto:Token: None 2022-09-12T20:22:01.683 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:01.683 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p7rraL3RtQ9qdviWDp83jzYzljA= 2022-09-12T20:22:01.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p7rraL3RtQ9qdviWDp83jzYzljA=', 'Content-Length': '0'} 2022-09-12T20:22:01.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082a550cf5c259a62-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.687 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:01.687 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:01.688 DEBUG:boto:path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:01.689 DEBUG:boto:auth_path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:01.689 DEBUG:boto:Method: GET 2022-09-12T20:22:01.690 DEBUG:boto:Path: /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:01.690 DEBUG:boto:Data: 2022-09-12T20:22:01.691 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.692 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:01.692 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.693 DEBUG:boto:Params: {} 2022-09-12T20:22:01.693 DEBUG:boto:Token: None 2022-09-12T20:22:01.694 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:01.694 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:p7rraL3RtQ9qdviWDp83jzYzljA= 2022-09-12T20:22:01.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:p7rraL3RtQ9qdviWDp83jzYzljA=', 'Content-Length': '0'} 2022-09-12T20:22:01.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f4d157c5e494a05-00631f94e9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.738 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:01.739 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:01.740 DEBUG:boto:path=/jdpiak-76/?prefix=obj55 2022-09-12T20:22:01.741 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj55 2022-09-12T20:22:01.741 DEBUG:boto:Method: GET 2022-09-12T20:22:01.742 DEBUG:boto:Path: /jdpiak-76/?prefix=obj55 2022-09-12T20:22:01.743 DEBUG:boto:Data: 2022-09-12T20:22:01.743 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.744 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.745 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.745 DEBUG:boto:Params: {} 2022-09-12T20:22:01.746 DEBUG:boto:Token: None 2022-09-12T20:22:01.747 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/ 2022-09-12T20:22:01.747 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58= 2022-09-12T20:22:01.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58=', 'Content-Length': '0'} 2022-09-12T20:22:01.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d55a8288fd15fa8-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.752 DEBUG:boto:b'jdpiak-76obj551000falseobj552022-09-12T20:18:36.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:01.752 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:01.753 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:01.754 DEBUG:boto:path=/jdpiak-76/?prefix=obj55 2022-09-12T20:22:01.754 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj55 2022-09-12T20:22:01.755 DEBUG:boto:Method: GET 2022-09-12T20:22:01.755 DEBUG:boto:Path: /jdpiak-76/?prefix=obj55 2022-09-12T20:22:01.756 DEBUG:boto:Data: 2022-09-12T20:22:01.756 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.757 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:01.758 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.758 DEBUG:boto:Params: {} 2022-09-12T20:22:01.759 DEBUG:boto:Token: None 2022-09-12T20:22:01.760 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/ 2022-09-12T20:22:01.761 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58= 2022-09-12T20:22:01.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58=', 'Content-Length': '0'} 2022-09-12T20:22:01.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000baf122bb6e7cc9a4-00631f94e9-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.765 DEBUG:boto:b'jdpiak-76obj551000falseobj552022-09-12T20:18:36.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:01.766 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:01.766 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:01.767 DEBUG:boto:Method: HEAD 2022-09-12T20:22:01.767 DEBUG:boto:Path: /jdpiak-76/obj55 2022-09-12T20:22:01.768 DEBUG:boto:Data: 2022-09-12T20:22:01.768 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.769 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.769 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.770 DEBUG:boto:Params: {} 2022-09-12T20:22:01.770 DEBUG:boto:Token: None 2022-09-12T20:22:01.771 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj55 2022-09-12T20:22:01.772 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A59ODgBUvXTJ4adPLR2SIiwXt+A= 2022-09-12T20:22:01.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A59ODgBUvXTJ4adPLR2SIiwXt+A=', 'Content-Length': '0'} 2022-09-12T20:22:01.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6110f5d5bcf1810-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.775 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:01.775 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:01.776 DEBUG:boto:Method: HEAD 2022-09-12T20:22:01.776 DEBUG:boto:Path: /jdpiak-76/obj55 2022-09-12T20:22:01.777 DEBUG:boto:Data: 2022-09-12T20:22:01.778 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.778 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:01.779 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.779 DEBUG:boto:Params: {} 2022-09-12T20:22:01.780 DEBUG:boto:Token: None 2022-09-12T20:22:01.781 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj55 2022-09-12T20:22:01.781 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:A59ODgBUvXTJ4adPLR2SIiwXt+A= 2022-09-12T20:22:01.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:A59ODgBUvXTJ4adPLR2SIiwXt+A=', 'Content-Length': '0'} 2022-09-12T20:22:01.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000785900f55539ab27-00631f94e9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:22:01.785 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:01.786 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:01.787 DEBUG:boto:Method: GET 2022-09-12T20:22:01.787 DEBUG:boto:Path: /jdpiak-76/obj55 2022-09-12T20:22:01.788 DEBUG:boto:Data: 2022-09-12T20:22:01.788 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.789 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.789 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.790 DEBUG:boto:Params: {} 2022-09-12T20:22:01.790 DEBUG:boto:Token: None 2022-09-12T20:22:01.791 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj55 2022-09-12T20:22:01.791 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Xg50W8AgsxZLWEh/S/xe7du+cZM= 2022-09-12T20:22:01.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Xg50W8AgsxZLWEh/S/xe7du+cZM=', 'Content-Length': '0'} 2022-09-12T20:22:01.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c811f7dd070f34f-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.838 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:01.839 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:01.839 DEBUG:boto:Method: GET 2022-09-12T20:22:01.840 DEBUG:boto:Path: /jdpiak-76/obj55 2022-09-12T20:22:01.840 DEBUG:boto:Data: 2022-09-12T20:22:01.841 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.841 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:01.842 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.842 DEBUG:boto:Params: {} 2022-09-12T20:22:01.843 DEBUG:boto:Token: None 2022-09-12T20:22:01.844 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj55 2022-09-12T20:22:01.844 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Xg50W8AgsxZLWEh/S/xe7du+cZM= 2022-09-12T20:22:01.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Xg50W8AgsxZLWEh/S/xe7du+cZM=', 'Content-Length': '0'} 2022-09-12T20:22:01.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000176ca57fde18b0ad-00631f94e9-15c0-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.848 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:01.849 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:01.849 DEBUG:boto:path=/jdpiak-76/?prefix=obj55 2022-09-12T20:22:01.850 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj55 2022-09-12T20:22:01.850 DEBUG:boto:Method: GET 2022-09-12T20:22:01.851 DEBUG:boto:Path: /jdpiak-76/?prefix=obj55 2022-09-12T20:22:01.851 DEBUG:boto:Data: 2022-09-12T20:22:01.852 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.852 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.853 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.853 DEBUG:boto:Params: {} 2022-09-12T20:22:01.854 DEBUG:boto:Token: None 2022-09-12T20:22:01.855 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/ 2022-09-12T20:22:01.855 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58= 2022-09-12T20:22:01.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58=', 'Content-Length': '0'} 2022-09-12T20:22:01.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2f914ca31a72bc0-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.860 DEBUG:boto:b'jdpiak-76obj551000falseobj552022-09-12T20:18:36.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:01.860 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:01.861 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:01.861 DEBUG:boto:path=/jdpiak-76/?prefix=obj55 2022-09-12T20:22:01.862 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj55 2022-09-12T20:22:01.863 DEBUG:boto:Method: GET 2022-09-12T20:22:01.863 DEBUG:boto:Path: /jdpiak-76/?prefix=obj55 2022-09-12T20:22:01.864 DEBUG:boto:Data: 2022-09-12T20:22:01.865 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.865 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:01.866 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.867 DEBUG:boto:Params: {} 2022-09-12T20:22:01.867 DEBUG:boto:Token: None 2022-09-12T20:22:01.868 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/ 2022-09-12T20:22:01.868 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58= 2022-09-12T20:22:01.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:jBJsNZ2D5PAQndB/TMYxoiaEY58=', 'Content-Length': '0'} 2022-09-12T20:22:01.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be4e311e85bf9eb5-00631f94e9-15c0-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.873 DEBUG:boto:b'jdpiak-76obj551000falseobj552022-09-12T20:18:36.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:01.873 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-76 zones={test-zone3, test-zone1} 2022-09-12T20:22:01.874 INFO:tasks.rgw_multi.zone_rados:comparing bucket=jdpiak-76 zones={test-zone3, test-zone2} 2022-09-12T20:22:01.875 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:01.875 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:01.876 DEBUG:boto:Method: HEAD 2022-09-12T20:22:01.876 DEBUG:boto:Path: /jdpiak-76/ 2022-09-12T20:22:01.877 DEBUG:boto:Data: 2022-09-12T20:22:01.877 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.878 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.878 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.879 DEBUG:boto:Params: {} 2022-09-12T20:22:01.879 DEBUG:boto:Token: None 2022-09-12T20:22:01.880 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/ 2022-09-12T20:22:01.880 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BqaBZ9fSd5+Q4bRaJGZRCrkUVoI= 2022-09-12T20:22:01.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BqaBZ9fSd5+Q4bRaJGZRCrkUVoI=', 'Content-Length': '0'} 2022-09-12T20:22:01.884 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2c33821322a74cd-00631f94e9-1826-test-zone3'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.884 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:01.885 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:01.885 DEBUG:boto:Method: HEAD 2022-09-12T20:22:01.886 DEBUG:boto:Path: /jdpiak-76/ 2022-09-12T20:22:01.886 DEBUG:boto:Data: 2022-09-12T20:22:01.887 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.887 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:01.888 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.888 DEBUG:boto:Params: {} 2022-09-12T20:22:01.889 DEBUG:boto:Token: None 2022-09-12T20:22:01.890 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/ 2022-09-12T20:22:01.890 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:BqaBZ9fSd5+Q4bRaJGZRCrkUVoI= 2022-09-12T20:22:01.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:BqaBZ9fSd5+Q4bRaJGZRCrkUVoI=', 'Content-Length': '0'} 2022-09-12T20:22:01.894 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ba397ef140a8c44-00631f94e9-1ad5-test-zone2'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.894 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-12T20:22:01.895 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:01.895 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:01.896 DEBUG:boto:path=/jdpiak-76/?versions&max-keys=999 2022-09-12T20:22:01.896 DEBUG:boto:auth_path=/jdpiak-76/?versions&max-keys=999 2022-09-12T20:22:01.897 DEBUG:boto:Method: GET 2022-09-12T20:22:01.897 DEBUG:boto:Path: /jdpiak-76/?versions&max-keys=999 2022-09-12T20:22:01.898 DEBUG:boto:Data: 2022-09-12T20:22:01.898 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.899 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.899 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.900 DEBUG:boto:Params: {} 2022-09-12T20:22:01.900 DEBUG:boto:Token: None 2022-09-12T20:22:01.901 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/?versions 2022-09-12T20:22:01.902 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qLhvBhAD/zbrEPVR51VLY7jOUSg= 2022-09-12T20:22:01.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qLhvBhAD/zbrEPVR51VLY7jOUSg=', 'Content-Length': '0'} 2022-09-12T20:22:01.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000311ec12befbe1066-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.906 DEBUG:boto:b'jdpiak-76999falseobj11nulltrue2022-09-12T20:18:36.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:18:36.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:18:36.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:18:36.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:18:36.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:01.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:22:01.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:22:01.908 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:22:01.908 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:22:01.909 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:22:01.909 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-12T20:22:01.910 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:01.910 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:01.911 DEBUG:boto:path=/jdpiak-76/?versions&max-keys=999 2022-09-12T20:22:01.911 DEBUG:boto:auth_path=/jdpiak-76/?versions&max-keys=999 2022-09-12T20:22:01.912 DEBUG:boto:Method: GET 2022-09-12T20:22:01.912 DEBUG:boto:Path: /jdpiak-76/?versions&max-keys=999 2022-09-12T20:22:01.913 DEBUG:boto:Data: 2022-09-12T20:22:01.913 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.914 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:01.914 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.915 DEBUG:boto:Params: {} 2022-09-12T20:22:01.916 DEBUG:boto:Token: None 2022-09-12T20:22:01.916 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/?versions 2022-09-12T20:22:01.917 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qLhvBhAD/zbrEPVR51VLY7jOUSg= 2022-09-12T20:22:01.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qLhvBhAD/zbrEPVR51VLY7jOUSg=', 'Content-Length': '0'} 2022-09-12T20:22:01.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d181d61c001b9f7b-00631f94e9-1ad5-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.922 DEBUG:boto:b'jdpiak-76999falseobj11nulltrue2022-09-12T20:18:36.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:18:36.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:18:36.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:18:36.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:18:36.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:01.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-12T20:22:01.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-12T20:22:01.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-12T20:22:01.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-12T20:22:01.925 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-12T20:22:01.925 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:01.926 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:01.927 DEBUG:boto:path=/jdpiak-76/?versions&max-keys=999 2022-09-12T20:22:01.927 DEBUG:boto:auth_path=/jdpiak-76/?versions&max-keys=999 2022-09-12T20:22:01.928 DEBUG:boto:Method: GET 2022-09-12T20:22:01.928 DEBUG:boto:Path: /jdpiak-76/?versions&max-keys=999 2022-09-12T20:22:01.929 DEBUG:boto:Data: 2022-09-12T20:22:01.929 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.930 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.930 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.931 DEBUG:boto:Params: {} 2022-09-12T20:22:01.931 DEBUG:boto:Token: None 2022-09-12T20:22:01.932 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/?versions 2022-09-12T20:22:01.932 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qLhvBhAD/zbrEPVR51VLY7jOUSg= 2022-09-12T20:22:01.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qLhvBhAD/zbrEPVR51VLY7jOUSg=', 'Content-Length': '0'} 2022-09-12T20:22:01.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a62f928061ba90a6-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.937 DEBUG:boto:b'jdpiak-76999falseobj11nulltrue2022-09-12T20:18:36.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:18:36.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:18:36.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:18:36.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:18:36.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:01.938 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:01.938 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:01.939 DEBUG:boto:path=/jdpiak-76/?versions&max-keys=999 2022-09-12T20:22:01.939 DEBUG:boto:auth_path=/jdpiak-76/?versions&max-keys=999 2022-09-12T20:22:01.940 DEBUG:boto:Method: GET 2022-09-12T20:22:01.940 DEBUG:boto:Path: /jdpiak-76/?versions&max-keys=999 2022-09-12T20:22:01.941 DEBUG:boto:Data: 2022-09-12T20:22:01.941 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.942 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:01.942 DEBUG:boto:Port: 8000 2022-09-12T20:22:01.943 DEBUG:boto:Params: {} 2022-09-12T20:22:01.943 DEBUG:boto:Token: None 2022-09-12T20:22:01.944 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/?versions 2022-09-12T20:22:01.944 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:qLhvBhAD/zbrEPVR51VLY7jOUSg= 2022-09-12T20:22:01.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:qLhvBhAD/zbrEPVR51VLY7jOUSg=', 'Content-Length': '0'} 2022-09-12T20:22:01.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee67fe27a1217498-00631f94e9-1ad5-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:01.948 DEBUG:boto:b'jdpiak-76999falseobj11nulltrue2022-09-12T20:18:36.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-12T20:18:36.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-12T20:18:36.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-12T20:18:36.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-12T20:18:36.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:01.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:22:01.950 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:22:01.951 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:22:01.951 DEBUG:boto:path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:22:01.952 DEBUG:boto:auth_path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:22:01.952 DEBUG:boto:Method: GET 2022-09-12T20:22:01.953 DEBUG:boto:Path: /jdpiak-76/obj11?versionId=null 2022-09-12T20:22:01.953 DEBUG:boto:Data: 2022-09-12T20:22:01.954 DEBUG:boto:Headers: {} 2022-09-12T20:22:01.954 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:01.955 DEBUG:boto:Port: 8001 2022-09-12T20:22:01.955 DEBUG:boto:Params: {} 2022-09-12T20:22:01.956 DEBUG:boto:Token: None 2022-09-12T20:22:01.956 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:01 GMT /jdpiak-76/obj11?versionId=null 2022-09-12T20:22:01.957 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:dRqbeGTy1ZNL9OuYVOKM4zbuZIo= 2022-09-12T20:22:01.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:01 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:dRqbeGTy1ZNL9OuYVOKM4zbuZIo=', 'Content-Length': '0'} 2022-09-12T20:22:01.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d6ba596f6da7210-00631f94e9-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.002 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:22:02.003 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:22:02.004 DEBUG:boto:path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:22:02.004 DEBUG:boto:auth_path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:22:02.005 DEBUG:boto:Method: GET 2022-09-12T20:22:02.005 DEBUG:boto:Path: /jdpiak-76/obj11?versionId=null 2022-09-12T20:22:02.006 DEBUG:boto:Data: 2022-09-12T20:22:02.007 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.007 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.008 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.008 DEBUG:boto:Params: {} 2022-09-12T20:22:02.009 DEBUG:boto:Token: None 2022-09-12T20:22:02.010 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj11?versionId=null 2022-09-12T20:22:02.011 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Sz5kPLzaNmrTQMUDkX5Vmf5zze4= 2022-09-12T20:22:02.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Sz5kPLzaNmrTQMUDkX5Vmf5zze4=', 'Content-Length': '0'} 2022-09-12T20:22:02.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbfe7afb108213d1-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.015 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:22:02.015 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:22:02.016 DEBUG:boto:path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:22:02.017 DEBUG:boto:auth_path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:22:02.017 DEBUG:boto:Method: HEAD 2022-09-12T20:22:02.018 DEBUG:boto:Path: /jdpiak-76/obj11?versionId=null 2022-09-12T20:22:02.018 DEBUG:boto:Data: 2022-09-12T20:22:02.019 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.019 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.020 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.020 DEBUG:boto:Params: {} 2022-09-12T20:22:02.021 DEBUG:boto:Token: None 2022-09-12T20:22:02.022 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj11?versionId=null 2022-09-12T20:22:02.022 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5R+Bp/34e8Xit0IY1s9Sx8OoLKw= 2022-09-12T20:22:02.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5R+Bp/34e8Xit0IY1s9Sx8OoLKw=', 'Content-Length': '0'} 2022-09-12T20:22:02.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bafb75ecba2b7a3-00631f94ea-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.027 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:22:02.027 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:22:02.028 DEBUG:boto:path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:22:02.028 DEBUG:boto:auth_path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:22:02.029 DEBUG:boto:Method: HEAD 2022-09-12T20:22:02.030 DEBUG:boto:Path: /jdpiak-76/obj11?versionId=null 2022-09-12T20:22:02.030 DEBUG:boto:Data: 2022-09-12T20:22:02.031 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.031 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.032 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.032 DEBUG:boto:Params: {} 2022-09-12T20:22:02.033 DEBUG:boto:Token: None 2022-09-12T20:22:02.034 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj11?versionId=null 2022-09-12T20:22:02.034 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5R+Bp/34e8Xit0IY1s9Sx8OoLKw= 2022-09-12T20:22:02.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5R+Bp/34e8Xit0IY1s9Sx8OoLKw=', 'Content-Length': '0'} 2022-09-12T20:22:02.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5b6595db6746b49-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:22:02.039 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:22:02.039 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:22:02.040 DEBUG:boto:path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:22:02.040 DEBUG:boto:auth_path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:22:02.041 DEBUG:boto:Method: GET 2022-09-12T20:22:02.041 DEBUG:boto:Path: /jdpiak-76/obj11?versionId=null 2022-09-12T20:22:02.042 DEBUG:boto:Data: 2022-09-12T20:22:02.042 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.043 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.044 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.044 DEBUG:boto:Params: {} 2022-09-12T20:22:02.045 DEBUG:boto:Token: None 2022-09-12T20:22:02.045 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj11?versionId=null 2022-09-12T20:22:02.046 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Sz5kPLzaNmrTQMUDkX5Vmf5zze4= 2022-09-12T20:22:02.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Sz5kPLzaNmrTQMUDkX5Vmf5zze4=', 'Content-Length': '0'} 2022-09-12T20:22:02.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000432695c427cbe221-00631f94ea-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.050 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:22:02.050 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:22:02.051 DEBUG:boto:path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:22:02.051 DEBUG:boto:auth_path=/jdpiak-76/obj11?versionId=null 2022-09-12T20:22:02.052 DEBUG:boto:Method: GET 2022-09-12T20:22:02.053 DEBUG:boto:Path: /jdpiak-76/obj11?versionId=null 2022-09-12T20:22:02.053 DEBUG:boto:Data: 2022-09-12T20:22:02.054 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.055 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.055 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.056 DEBUG:boto:Params: {} 2022-09-12T20:22:02.057 DEBUG:boto:Token: None 2022-09-12T20:22:02.057 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj11?versionId=null 2022-09-12T20:22:02.058 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Sz5kPLzaNmrTQMUDkX5Vmf5zze4= 2022-09-12T20:22:02.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Sz5kPLzaNmrTQMUDkX5Vmf5zze4=', 'Content-Length': '0'} 2022-09-12T20:22:02.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa5be42a645586b9-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.102 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:02.103 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:02.104 DEBUG:boto:path=/jdpiak-76/?prefix=obj11 2022-09-12T20:22:02.105 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj11 2022-09-12T20:22:02.105 DEBUG:boto:Method: GET 2022-09-12T20:22:02.106 DEBUG:boto:Path: /jdpiak-76/?prefix=obj11 2022-09-12T20:22:02.106 DEBUG:boto:Data: 2022-09-12T20:22:02.107 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.107 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.108 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.108 DEBUG:boto:Params: {} 2022-09-12T20:22:02.109 DEBUG:boto:Token: None 2022-09-12T20:22:02.110 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/ 2022-09-12T20:22:02.110 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0= 2022-09-12T20:22:02.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0=', 'Content-Length': '0'} 2022-09-12T20:22:02.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e78eb8a8f7e5337-00631f94ea-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.115 DEBUG:boto:b'jdpiak-76obj111000falseobj112022-09-12T20:18:36.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:02.115 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:02.116 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:02.117 DEBUG:boto:path=/jdpiak-76/?prefix=obj11 2022-09-12T20:22:02.117 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj11 2022-09-12T20:22:02.118 DEBUG:boto:Method: GET 2022-09-12T20:22:02.118 DEBUG:boto:Path: /jdpiak-76/?prefix=obj11 2022-09-12T20:22:02.119 DEBUG:boto:Data: 2022-09-12T20:22:02.119 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.120 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.120 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.121 DEBUG:boto:Params: {} 2022-09-12T20:22:02.122 DEBUG:boto:Token: None 2022-09-12T20:22:02.122 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/ 2022-09-12T20:22:02.123 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0= 2022-09-12T20:22:02.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0=', 'Content-Length': '0'} 2022-09-12T20:22:02.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d06560aa9dc89708-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.127 DEBUG:boto:b'jdpiak-76obj111000falseobj112022-09-12T20:18:36.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:02.128 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:22:02.129 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:22:02.129 DEBUG:boto:Method: HEAD 2022-09-12T20:22:02.130 DEBUG:boto:Path: /jdpiak-76/obj11 2022-09-12T20:22:02.131 DEBUG:boto:Data: 2022-09-12T20:22:02.131 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.132 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.133 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.133 DEBUG:boto:Params: {} 2022-09-12T20:22:02.134 DEBUG:boto:Token: None 2022-09-12T20:22:02.135 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj11 2022-09-12T20:22:02.136 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Od02l8K8H0mJI/wrqg9Js3FxBDU= 2022-09-12T20:22:02.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Od02l8K8H0mJI/wrqg9Js3FxBDU=', 'Content-Length': '0'} 2022-09-12T20:22:02.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a05f5e4ba8d49603-00631f94ea-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.140 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:22:02.140 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:22:02.141 DEBUG:boto:Method: HEAD 2022-09-12T20:22:02.142 DEBUG:boto:Path: /jdpiak-76/obj11 2022-09-12T20:22:02.142 DEBUG:boto:Data: 2022-09-12T20:22:02.143 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.144 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.145 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.145 DEBUG:boto:Params: {} 2022-09-12T20:22:02.146 DEBUG:boto:Token: None 2022-09-12T20:22:02.147 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj11 2022-09-12T20:22:02.148 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Od02l8K8H0mJI/wrqg9Js3FxBDU= 2022-09-12T20:22:02.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Od02l8K8H0mJI/wrqg9Js3FxBDU=', 'Content-Length': '0'} 2022-09-12T20:22:02.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067e6201f4af6d8c6-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-12T20:22:02.153 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:22:02.153 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:22:02.154 DEBUG:boto:Method: GET 2022-09-12T20:22:02.155 DEBUG:boto:Path: /jdpiak-76/obj11 2022-09-12T20:22:02.156 DEBUG:boto:Data: 2022-09-12T20:22:02.156 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.157 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.158 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.159 DEBUG:boto:Params: {} 2022-09-12T20:22:02.160 DEBUG:boto:Token: None 2022-09-12T20:22:02.161 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj11 2022-09-12T20:22:02.162 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5qmEOcJoFYzKdZcNDkRM1mal404= 2022-09-12T20:22:02.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5qmEOcJoFYzKdZcNDkRM1mal404=', 'Content-Length': '0'} 2022-09-12T20:22:02.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057ee5075de1104e6-00631f94ea-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.210 DEBUG:boto:path=/jdpiak-76/obj11 2022-09-12T20:22:02.211 DEBUG:boto:auth_path=/jdpiak-76/obj11 2022-09-12T20:22:02.212 DEBUG:boto:Method: GET 2022-09-12T20:22:02.213 DEBUG:boto:Path: /jdpiak-76/obj11 2022-09-12T20:22:02.214 DEBUG:boto:Data: 2022-09-12T20:22:02.214 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.215 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.216 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.217 DEBUG:boto:Params: {} 2022-09-12T20:22:02.217 DEBUG:boto:Token: None 2022-09-12T20:22:02.218 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj11 2022-09-12T20:22:02.219 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:5qmEOcJoFYzKdZcNDkRM1mal404= 2022-09-12T20:22:02.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:5qmEOcJoFYzKdZcNDkRM1mal404=', 'Content-Length': '0'} 2022-09-12T20:22:02.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aeba79fc6e5638e7-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.224 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:02.224 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:02.225 DEBUG:boto:path=/jdpiak-76/?prefix=obj11 2022-09-12T20:22:02.225 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj11 2022-09-12T20:22:02.226 DEBUG:boto:Method: GET 2022-09-12T20:22:02.227 DEBUG:boto:Path: /jdpiak-76/?prefix=obj11 2022-09-12T20:22:02.227 DEBUG:boto:Data: 2022-09-12T20:22:02.228 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.229 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.230 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.230 DEBUG:boto:Params: {} 2022-09-12T20:22:02.231 DEBUG:boto:Token: None 2022-09-12T20:22:02.232 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/ 2022-09-12T20:22:02.233 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0= 2022-09-12T20:22:02.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0=', 'Content-Length': '0'} 2022-09-12T20:22:02.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f38a6b87bb0d7942-00631f94ea-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.238 DEBUG:boto:b'jdpiak-76obj111000falseobj112022-09-12T20:18:36.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:02.239 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:02.239 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:02.240 DEBUG:boto:path=/jdpiak-76/?prefix=obj11 2022-09-12T20:22:02.240 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj11 2022-09-12T20:22:02.241 DEBUG:boto:Method: GET 2022-09-12T20:22:02.241 DEBUG:boto:Path: /jdpiak-76/?prefix=obj11 2022-09-12T20:22:02.242 DEBUG:boto:Data: 2022-09-12T20:22:02.243 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.244 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.244 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.245 DEBUG:boto:Params: {} 2022-09-12T20:22:02.245 DEBUG:boto:Token: None 2022-09-12T20:22:02.246 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/ 2022-09-12T20:22:02.247 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0= 2022-09-12T20:22:02.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0=', 'Content-Length': '0'} 2022-09-12T20:22:02.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052a67d64164bc647-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.294 DEBUG:boto:b'jdpiak-76obj111000falseobj112022-09-12T20:18:36.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:02.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:22:02.296 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:22:02.296 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:22:02.297 DEBUG:boto:path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:22:02.297 DEBUG:boto:auth_path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:22:02.298 DEBUG:boto:Method: GET 2022-09-12T20:22:02.298 DEBUG:boto:Path: /jdpiak-76/obj22?versionId=null 2022-09-12T20:22:02.299 DEBUG:boto:Data: 2022-09-12T20:22:02.299 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.300 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.301 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.301 DEBUG:boto:Params: {} 2022-09-12T20:22:02.302 DEBUG:boto:Token: None 2022-09-12T20:22:02.304 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj22?versionId=null 2022-09-12T20:22:02.304 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gA22FVf/88PHQ5EV+xKuIJZiOVY= 2022-09-12T20:22:02.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gA22FVf/88PHQ5EV+xKuIJZiOVY=', 'Content-Length': '0'} 2022-09-12T20:22:02.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035b2f7809a630705-00631f94ea-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.308 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:22:02.309 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:22:02.309 DEBUG:boto:path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:22:02.310 DEBUG:boto:auth_path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:22:02.311 DEBUG:boto:Method: GET 2022-09-12T20:22:02.311 DEBUG:boto:Path: /jdpiak-76/obj22?versionId=null 2022-09-12T20:22:02.312 DEBUG:boto:Data: 2022-09-12T20:22:02.312 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.313 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.314 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.314 DEBUG:boto:Params: {} 2022-09-12T20:22:02.315 DEBUG:boto:Token: None 2022-09-12T20:22:02.315 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj22?versionId=null 2022-09-12T20:22:02.316 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gA22FVf/88PHQ5EV+xKuIJZiOVY= 2022-09-12T20:22:02.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gA22FVf/88PHQ5EV+xKuIJZiOVY=', 'Content-Length': '0'} 2022-09-12T20:22:02.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7957e3ad0d21bad-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.320 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:22:02.321 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:22:02.322 DEBUG:boto:path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:22:02.323 DEBUG:boto:auth_path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:22:02.324 DEBUG:boto:Method: HEAD 2022-09-12T20:22:02.325 DEBUG:boto:Path: /jdpiak-76/obj22?versionId=null 2022-09-12T20:22:02.326 DEBUG:boto:Data: 2022-09-12T20:22:02.327 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.328 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.329 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.330 DEBUG:boto:Params: {} 2022-09-12T20:22:02.331 DEBUG:boto:Token: None 2022-09-12T20:22:02.331 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj22?versionId=null 2022-09-12T20:22:02.332 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:juxOj4B48JZdE5TfZISVquNLQXk= 2022-09-12T20:22:02.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:juxOj4B48JZdE5TfZISVquNLQXk=', 'Content-Length': '0'} 2022-09-12T20:22:02.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067c2d2f0b043bdb1-00631f94ea-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.336 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:22:02.336 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:22:02.337 DEBUG:boto:path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:22:02.337 DEBUG:boto:auth_path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:22:02.338 DEBUG:boto:Method: HEAD 2022-09-12T20:22:02.338 DEBUG:boto:Path: /jdpiak-76/obj22?versionId=null 2022-09-12T20:22:02.339 DEBUG:boto:Data: 2022-09-12T20:22:02.339 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.340 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.340 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.341 DEBUG:boto:Params: {} 2022-09-12T20:22:02.342 DEBUG:boto:Token: None 2022-09-12T20:22:02.342 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj22?versionId=null 2022-09-12T20:22:02.343 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:juxOj4B48JZdE5TfZISVquNLQXk= 2022-09-12T20:22:02.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:juxOj4B48JZdE5TfZISVquNLQXk=', 'Content-Length': '0'} 2022-09-12T20:22:02.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c89130e8461a543-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:22:02.348 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:22:02.349 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:22:02.349 DEBUG:boto:path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:22:02.350 DEBUG:boto:auth_path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:22:02.351 DEBUG:boto:Method: GET 2022-09-12T20:22:02.352 DEBUG:boto:Path: /jdpiak-76/obj22?versionId=null 2022-09-12T20:22:02.353 DEBUG:boto:Data: 2022-09-12T20:22:02.354 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.355 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.356 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.357 DEBUG:boto:Params: {} 2022-09-12T20:22:02.358 DEBUG:boto:Token: None 2022-09-12T20:22:02.359 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj22?versionId=null 2022-09-12T20:22:02.360 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gA22FVf/88PHQ5EV+xKuIJZiOVY= 2022-09-12T20:22:02.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gA22FVf/88PHQ5EV+xKuIJZiOVY=', 'Content-Length': '0'} 2022-09-12T20:22:02.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051a5f50d065fd71a-00631f94ea-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.364 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:22:02.365 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:22:02.365 DEBUG:boto:path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:22:02.366 DEBUG:boto:auth_path=/jdpiak-76/obj22?versionId=null 2022-09-12T20:22:02.366 DEBUG:boto:Method: GET 2022-09-12T20:22:02.367 DEBUG:boto:Path: /jdpiak-76/obj22?versionId=null 2022-09-12T20:22:02.367 DEBUG:boto:Data: 2022-09-12T20:22:02.368 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.368 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.369 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.369 DEBUG:boto:Params: {} 2022-09-12T20:22:02.370 DEBUG:boto:Token: None 2022-09-12T20:22:02.371 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj22?versionId=null 2022-09-12T20:22:02.371 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:gA22FVf/88PHQ5EV+xKuIJZiOVY= 2022-09-12T20:22:02.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:gA22FVf/88PHQ5EV+xKuIJZiOVY=', 'Content-Length': '0'} 2022-09-12T20:22:02.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e0d1f0fc6b2463a-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.375 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:02.376 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:02.376 DEBUG:boto:path=/jdpiak-76/?prefix=obj22 2022-09-12T20:22:02.377 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj22 2022-09-12T20:22:02.377 DEBUG:boto:Method: GET 2022-09-12T20:22:02.378 DEBUG:boto:Path: /jdpiak-76/?prefix=obj22 2022-09-12T20:22:02.378 DEBUG:boto:Data: 2022-09-12T20:22:02.379 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.380 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.380 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.381 DEBUG:boto:Params: {} 2022-09-12T20:22:02.381 DEBUG:boto:Token: None 2022-09-12T20:22:02.382 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/ 2022-09-12T20:22:02.383 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0= 2022-09-12T20:22:02.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0=', 'Content-Length': '0'} 2022-09-12T20:22:02.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000834a55ccfd670678-00631f94ea-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.430 DEBUG:boto:b'jdpiak-76obj221000falseobj222022-09-12T20:18:36.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:02.431 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:02.432 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:02.432 DEBUG:boto:path=/jdpiak-76/?prefix=obj22 2022-09-12T20:22:02.433 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj22 2022-09-12T20:22:02.433 DEBUG:boto:Method: GET 2022-09-12T20:22:02.434 DEBUG:boto:Path: /jdpiak-76/?prefix=obj22 2022-09-12T20:22:02.434 DEBUG:boto:Data: 2022-09-12T20:22:02.435 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.435 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.436 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.437 DEBUG:boto:Params: {} 2022-09-12T20:22:02.437 DEBUG:boto:Token: None 2022-09-12T20:22:02.438 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/ 2022-09-12T20:22:02.438 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0= 2022-09-12T20:22:02.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0=', 'Content-Length': '0'} 2022-09-12T20:22:02.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000559131910856e547-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.443 DEBUG:boto:b'jdpiak-76obj221000falseobj222022-09-12T20:18:36.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:02.443 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:22:02.444 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:22:02.444 DEBUG:boto:Method: HEAD 2022-09-12T20:22:02.445 DEBUG:boto:Path: /jdpiak-76/obj22 2022-09-12T20:22:02.445 DEBUG:boto:Data: 2022-09-12T20:22:02.446 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.446 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.447 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.447 DEBUG:boto:Params: {} 2022-09-12T20:22:02.448 DEBUG:boto:Token: None 2022-09-12T20:22:02.449 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj22 2022-09-12T20:22:02.449 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XbepmiA1WUi5kJoQAuab6gLzMdE= 2022-09-12T20:22:02.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XbepmiA1WUi5kJoQAuab6gLzMdE=', 'Content-Length': '0'} 2022-09-12T20:22:02.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ee73675bfb48175-00631f94ea-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.452 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:22:02.453 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:22:02.453 DEBUG:boto:Method: HEAD 2022-09-12T20:22:02.454 DEBUG:boto:Path: /jdpiak-76/obj22 2022-09-12T20:22:02.455 DEBUG:boto:Data: 2022-09-12T20:22:02.455 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.456 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.456 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.456 DEBUG:boto:Params: {} 2022-09-12T20:22:02.457 DEBUG:boto:Token: None 2022-09-12T20:22:02.458 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj22 2022-09-12T20:22:02.458 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:XbepmiA1WUi5kJoQAuab6gLzMdE= 2022-09-12T20:22:02.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:XbepmiA1WUi5kJoQAuab6gLzMdE=', 'Content-Length': '0'} 2022-09-12T20:22:02.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088c0043149a8a0f9-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-12T20:22:02.462 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:22:02.462 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:22:02.463 DEBUG:boto:Method: GET 2022-09-12T20:22:02.464 DEBUG:boto:Path: /jdpiak-76/obj22 2022-09-12T20:22:02.464 DEBUG:boto:Data: 2022-09-12T20:22:02.465 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.465 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.466 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.466 DEBUG:boto:Params: {} 2022-09-12T20:22:02.467 DEBUG:boto:Token: None 2022-09-12T20:22:02.467 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj22 2022-09-12T20:22:02.468 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FA0QQlsFpx3Qqxhsos69xnU8dMI= 2022-09-12T20:22:02.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FA0QQlsFpx3Qqxhsos69xnU8dMI=', 'Content-Length': '0'} 2022-09-12T20:22:02.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de3fb5bad86cad83-00631f94ea-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.472 DEBUG:boto:path=/jdpiak-76/obj22 2022-09-12T20:22:02.472 DEBUG:boto:auth_path=/jdpiak-76/obj22 2022-09-12T20:22:02.473 DEBUG:boto:Method: GET 2022-09-12T20:22:02.473 DEBUG:boto:Path: /jdpiak-76/obj22 2022-09-12T20:22:02.474 DEBUG:boto:Data: 2022-09-12T20:22:02.474 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.475 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.475 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.476 DEBUG:boto:Params: {} 2022-09-12T20:22:02.477 DEBUG:boto:Token: None 2022-09-12T20:22:02.477 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj22 2022-09-12T20:22:02.478 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FA0QQlsFpx3Qqxhsos69xnU8dMI= 2022-09-12T20:22:02.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FA0QQlsFpx3Qqxhsos69xnU8dMI=', 'Content-Length': '0'} 2022-09-12T20:22:02.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8b4ceeb8a29b0ef-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.522 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:02.523 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:02.523 DEBUG:boto:path=/jdpiak-76/?prefix=obj22 2022-09-12T20:22:02.524 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj22 2022-09-12T20:22:02.524 DEBUG:boto:Method: GET 2022-09-12T20:22:02.525 DEBUG:boto:Path: /jdpiak-76/?prefix=obj22 2022-09-12T20:22:02.525 DEBUG:boto:Data: 2022-09-12T20:22:02.526 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.527 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.527 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.528 DEBUG:boto:Params: {} 2022-09-12T20:22:02.528 DEBUG:boto:Token: None 2022-09-12T20:22:02.529 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/ 2022-09-12T20:22:02.529 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0= 2022-09-12T20:22:02.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0=', 'Content-Length': '0'} 2022-09-12T20:22:02.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003debdbb5179c25c7-00631f94ea-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.534 DEBUG:boto:b'jdpiak-76obj221000falseobj222022-09-12T20:18:36.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:02.535 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:02.535 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:02.536 DEBUG:boto:path=/jdpiak-76/?prefix=obj22 2022-09-12T20:22:02.536 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj22 2022-09-12T20:22:02.537 DEBUG:boto:Method: GET 2022-09-12T20:22:02.537 DEBUG:boto:Path: /jdpiak-76/?prefix=obj22 2022-09-12T20:22:02.538 DEBUG:boto:Data: 2022-09-12T20:22:02.538 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.539 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.539 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.540 DEBUG:boto:Params: {} 2022-09-12T20:22:02.541 DEBUG:boto:Token: None 2022-09-12T20:22:02.541 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/ 2022-09-12T20:22:02.542 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0= 2022-09-12T20:22:02.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0=', 'Content-Length': '0'} 2022-09-12T20:22:02.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061138a348f607c7e-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.546 DEBUG:boto:b'jdpiak-76obj221000falseobj222022-09-12T20:18:36.814Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:02.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:22:02.547 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:22:02.548 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:22:02.549 DEBUG:boto:path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:22:02.549 DEBUG:boto:auth_path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:22:02.550 DEBUG:boto:Method: GET 2022-09-12T20:22:02.550 DEBUG:boto:Path: /jdpiak-76/obj33?versionId=null 2022-09-12T20:22:02.551 DEBUG:boto:Data: 2022-09-12T20:22:02.551 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.552 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.552 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.553 DEBUG:boto:Params: {} 2022-09-12T20:22:02.554 DEBUG:boto:Token: None 2022-09-12T20:22:02.554 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj33?versionId=null 2022-09-12T20:22:02.555 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:78srmIXD6k9LNHPl+yIfxGR0224= 2022-09-12T20:22:02.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:78srmIXD6k9LNHPl+yIfxGR0224=', 'Content-Length': '0'} 2022-09-12T20:22:02.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9b6043dd2ccc632-00631f94ea-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.558 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:22:02.559 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:22:02.560 DEBUG:boto:path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:22:02.560 DEBUG:boto:auth_path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:22:02.561 DEBUG:boto:Method: GET 2022-09-12T20:22:02.561 DEBUG:boto:Path: /jdpiak-76/obj33?versionId=null 2022-09-12T20:22:02.562 DEBUG:boto:Data: 2022-09-12T20:22:02.562 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.563 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.563 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.564 DEBUG:boto:Params: {} 2022-09-12T20:22:02.565 DEBUG:boto:Token: None 2022-09-12T20:22:02.565 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj33?versionId=null 2022-09-12T20:22:02.566 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:78srmIXD6k9LNHPl+yIfxGR0224= 2022-09-12T20:22:02.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:78srmIXD6k9LNHPl+yIfxGR0224=', 'Content-Length': '0'} 2022-09-12T20:22:02.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8f8ef4fda6eeea4-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.569 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:22:02.570 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:22:02.570 DEBUG:boto:path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:22:02.571 DEBUG:boto:auth_path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:22:02.572 DEBUG:boto:Method: HEAD 2022-09-12T20:22:02.572 DEBUG:boto:Path: /jdpiak-76/obj33?versionId=null 2022-09-12T20:22:02.573 DEBUG:boto:Data: 2022-09-12T20:22:02.573 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.574 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.574 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.575 DEBUG:boto:Params: {} 2022-09-12T20:22:02.575 DEBUG:boto:Token: None 2022-09-12T20:22:02.576 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj33?versionId=null 2022-09-12T20:22:02.577 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EOdHhW5nrFXMi1Fa+eGDog5SEG0= 2022-09-12T20:22:02.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EOdHhW5nrFXMi1Fa+eGDog5SEG0=', 'Content-Length': '0'} 2022-09-12T20:22:02.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b418b2eeb2cacc9-00631f94ea-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.580 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:22:02.581 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:22:02.581 DEBUG:boto:path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:22:02.582 DEBUG:boto:auth_path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:22:02.583 DEBUG:boto:Method: HEAD 2022-09-12T20:22:02.583 DEBUG:boto:Path: /jdpiak-76/obj33?versionId=null 2022-09-12T20:22:02.584 DEBUG:boto:Data: 2022-09-12T20:22:02.585 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.586 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.587 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.588 DEBUG:boto:Params: {} 2022-09-12T20:22:02.590 DEBUG:boto:Token: None 2022-09-12T20:22:02.591 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj33?versionId=null 2022-09-12T20:22:02.592 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:EOdHhW5nrFXMi1Fa+eGDog5SEG0= 2022-09-12T20:22:02.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:EOdHhW5nrFXMi1Fa+eGDog5SEG0=', 'Content-Length': '0'} 2022-09-12T20:22:02.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fb22ca3f65a8e38-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:22:02.597 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:22:02.597 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:22:02.598 DEBUG:boto:path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:22:02.598 DEBUG:boto:auth_path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:22:02.599 DEBUG:boto:Method: GET 2022-09-12T20:22:02.600 DEBUG:boto:Path: /jdpiak-76/obj33?versionId=null 2022-09-12T20:22:02.600 DEBUG:boto:Data: 2022-09-12T20:22:02.601 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.601 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.602 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.602 DEBUG:boto:Params: {} 2022-09-12T20:22:02.603 DEBUG:boto:Token: None 2022-09-12T20:22:02.603 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj33?versionId=null 2022-09-12T20:22:02.604 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:78srmIXD6k9LNHPl+yIfxGR0224= 2022-09-12T20:22:02.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:78srmIXD6k9LNHPl+yIfxGR0224=', 'Content-Length': '0'} 2022-09-12T20:22:02.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061cf463a56dba755-00631f94ea-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.608 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:22:02.609 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:22:02.609 DEBUG:boto:path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:22:02.610 DEBUG:boto:auth_path=/jdpiak-76/obj33?versionId=null 2022-09-12T20:22:02.611 DEBUG:boto:Method: GET 2022-09-12T20:22:02.611 DEBUG:boto:Path: /jdpiak-76/obj33?versionId=null 2022-09-12T20:22:02.612 DEBUG:boto:Data: 2022-09-12T20:22:02.612 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.613 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.614 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.614 DEBUG:boto:Params: {} 2022-09-12T20:22:02.615 DEBUG:boto:Token: None 2022-09-12T20:22:02.616 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj33?versionId=null 2022-09-12T20:22:02.616 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:78srmIXD6k9LNHPl+yIfxGR0224= 2022-09-12T20:22:02.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:78srmIXD6k9LNHPl+yIfxGR0224=', 'Content-Length': '0'} 2022-09-12T20:22:02.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8b2c595deccb098-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.620 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:02.621 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:02.622 DEBUG:boto:path=/jdpiak-76/?prefix=obj33 2022-09-12T20:22:02.623 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj33 2022-09-12T20:22:02.623 DEBUG:boto:Method: GET 2022-09-12T20:22:02.624 DEBUG:boto:Path: /jdpiak-76/?prefix=obj33 2022-09-12T20:22:02.624 DEBUG:boto:Data: 2022-09-12T20:22:02.625 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.625 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.626 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.627 DEBUG:boto:Params: {} 2022-09-12T20:22:02.627 DEBUG:boto:Token: None 2022-09-12T20:22:02.628 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/ 2022-09-12T20:22:02.628 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0= 2022-09-12T20:22:02.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0=', 'Content-Length': '0'} 2022-09-12T20:22:02.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0878be943ca64cb-00631f94ea-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.633 DEBUG:boto:b'jdpiak-76obj331000falseobj332022-09-12T20:18:36.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:02.634 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:02.634 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:02.635 DEBUG:boto:path=/jdpiak-76/?prefix=obj33 2022-09-12T20:22:02.635 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj33 2022-09-12T20:22:02.636 DEBUG:boto:Method: GET 2022-09-12T20:22:02.636 DEBUG:boto:Path: /jdpiak-76/?prefix=obj33 2022-09-12T20:22:02.637 DEBUG:boto:Data: 2022-09-12T20:22:02.637 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.638 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.639 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.639 DEBUG:boto:Params: {} 2022-09-12T20:22:02.640 DEBUG:boto:Token: None 2022-09-12T20:22:02.640 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/ 2022-09-12T20:22:02.641 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0= 2022-09-12T20:22:02.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0=', 'Content-Length': '0'} 2022-09-12T20:22:02.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dbb187d359d5add-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.645 DEBUG:boto:b'jdpiak-76obj331000falseobj332022-09-12T20:18:36.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:02.646 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:22:02.646 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:22:02.647 DEBUG:boto:Method: HEAD 2022-09-12T20:22:02.647 DEBUG:boto:Path: /jdpiak-76/obj33 2022-09-12T20:22:02.648 DEBUG:boto:Data: 2022-09-12T20:22:02.648 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.649 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.649 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.650 DEBUG:boto:Params: {} 2022-09-12T20:22:02.651 DEBUG:boto:Token: None 2022-09-12T20:22:02.651 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj33 2022-09-12T20:22:02.652 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:285XfK8w5QadyrxQCce9zWVtCr0= 2022-09-12T20:22:02.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:285XfK8w5QadyrxQCce9zWVtCr0=', 'Content-Length': '0'} 2022-09-12T20:22:02.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f43ac6c8accb4c3-00631f94ea-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.655 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:22:02.656 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:22:02.656 DEBUG:boto:Method: HEAD 2022-09-12T20:22:02.657 DEBUG:boto:Path: /jdpiak-76/obj33 2022-09-12T20:22:02.657 DEBUG:boto:Data: 2022-09-12T20:22:02.658 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.658 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.659 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.659 DEBUG:boto:Params: {} 2022-09-12T20:22:02.660 DEBUG:boto:Token: None 2022-09-12T20:22:02.660 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj33 2022-09-12T20:22:02.661 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:285XfK8w5QadyrxQCce9zWVtCr0= 2022-09-12T20:22:02.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:285XfK8w5QadyrxQCce9zWVtCr0=', 'Content-Length': '0'} 2022-09-12T20:22:02.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adc1e063efe0c853-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-12T20:22:02.667 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:22:02.668 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:22:02.669 DEBUG:boto:Method: GET 2022-09-12T20:22:02.670 DEBUG:boto:Path: /jdpiak-76/obj33 2022-09-12T20:22:02.671 DEBUG:boto:Data: 2022-09-12T20:22:02.672 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.673 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.673 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.674 DEBUG:boto:Params: {} 2022-09-12T20:22:02.675 DEBUG:boto:Token: None 2022-09-12T20:22:02.675 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj33 2022-09-12T20:22:02.676 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Gu8kNxktb+qMnnZ+3nzW7JLsk0= 2022-09-12T20:22:02.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Gu8kNxktb+qMnnZ+3nzW7JLsk0=', 'Content-Length': '0'} 2022-09-12T20:22:02.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002636931ef9100e74-00631f94ea-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.722 DEBUG:boto:path=/jdpiak-76/obj33 2022-09-12T20:22:02.723 DEBUG:boto:auth_path=/jdpiak-76/obj33 2022-09-12T20:22:02.724 DEBUG:boto:Method: GET 2022-09-12T20:22:02.724 DEBUG:boto:Path: /jdpiak-76/obj33 2022-09-12T20:22:02.725 DEBUG:boto:Data: 2022-09-12T20:22:02.725 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.726 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.727 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.727 DEBUG:boto:Params: {} 2022-09-12T20:22:02.728 DEBUG:boto:Token: None 2022-09-12T20:22:02.729 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj33 2022-09-12T20:22:02.729 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:0Gu8kNxktb+qMnnZ+3nzW7JLsk0= 2022-09-12T20:22:02.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:0Gu8kNxktb+qMnnZ+3nzW7JLsk0=', 'Content-Length': '0'} 2022-09-12T20:22:02.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6c2b267d59540a8-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.733 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:02.734 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:02.734 DEBUG:boto:path=/jdpiak-76/?prefix=obj33 2022-09-12T20:22:02.735 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj33 2022-09-12T20:22:02.735 DEBUG:boto:Method: GET 2022-09-12T20:22:02.736 DEBUG:boto:Path: /jdpiak-76/?prefix=obj33 2022-09-12T20:22:02.736 DEBUG:boto:Data: 2022-09-12T20:22:02.737 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.737 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.738 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.739 DEBUG:boto:Params: {} 2022-09-12T20:22:02.739 DEBUG:boto:Token: None 2022-09-12T20:22:02.740 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/ 2022-09-12T20:22:02.741 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0= 2022-09-12T20:22:02.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0=', 'Content-Length': '0'} 2022-09-12T20:22:02.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cfaf2340a7b60e0-00631f94ea-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.745 DEBUG:boto:b'jdpiak-76obj331000falseobj332022-09-12T20:18:36.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:02.746 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:02.747 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:02.747 DEBUG:boto:path=/jdpiak-76/?prefix=obj33 2022-09-12T20:22:02.748 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj33 2022-09-12T20:22:02.748 DEBUG:boto:Method: GET 2022-09-12T20:22:02.749 DEBUG:boto:Path: /jdpiak-76/?prefix=obj33 2022-09-12T20:22:02.749 DEBUG:boto:Data: 2022-09-12T20:22:02.750 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.750 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.751 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.751 DEBUG:boto:Params: {} 2022-09-12T20:22:02.752 DEBUG:boto:Token: None 2022-09-12T20:22:02.753 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/ 2022-09-12T20:22:02.753 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0= 2022-09-12T20:22:02.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0=', 'Content-Length': '0'} 2022-09-12T20:22:02.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045f1d047757c1982-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.757 DEBUG:boto:b'jdpiak-76obj331000falseobj332022-09-12T20:18:36.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:02.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:22:02.759 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:02.759 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:02.760 DEBUG:boto:path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:02.760 DEBUG:boto:auth_path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:02.761 DEBUG:boto:Method: GET 2022-09-12T20:22:02.761 DEBUG:boto:Path: /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:02.762 DEBUG:boto:Data: 2022-09-12T20:22:02.762 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.763 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.764 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.765 DEBUG:boto:Params: {} 2022-09-12T20:22:02.766 DEBUG:boto:Token: None 2022-09-12T20:22:02.767 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:02.767 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FD5z6EAGLHA40QhzbHLFWmUNaA4= 2022-09-12T20:22:02.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FD5z6EAGLHA40QhzbHLFWmUNaA4=', 'Content-Length': '0'} 2022-09-12T20:22:02.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043a22425855eef4f-00631f94ea-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.774 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:02.775 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:02.775 DEBUG:boto:path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:02.776 DEBUG:boto:auth_path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:02.777 DEBUG:boto:Method: GET 2022-09-12T20:22:02.778 DEBUG:boto:Path: /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:02.778 DEBUG:boto:Data: 2022-09-12T20:22:02.779 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.779 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.780 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.781 DEBUG:boto:Params: {} 2022-09-12T20:22:02.782 DEBUG:boto:Token: None 2022-09-12T20:22:02.782 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:02.783 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FD5z6EAGLHA40QhzbHLFWmUNaA4= 2022-09-12T20:22:02.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FD5z6EAGLHA40QhzbHLFWmUNaA4=', 'Content-Length': '0'} 2022-09-12T20:22:02.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6ba1402ea0b10ab-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.791 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:02.793 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:02.793 DEBUG:boto:path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:02.794 DEBUG:boto:auth_path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:02.796 DEBUG:boto:Method: HEAD 2022-09-12T20:22:02.797 DEBUG:boto:Path: /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:02.798 DEBUG:boto:Data: 2022-09-12T20:22:02.800 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.800 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.802 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.802 DEBUG:boto:Params: {} 2022-09-12T20:22:02.804 DEBUG:boto:Token: None 2022-09-12T20:22:02.805 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:02.806 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yq/dGTAhKYXvqDvm2r1+jK6aDiY= 2022-09-12T20:22:02.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yq/dGTAhKYXvqDvm2r1+jK6aDiY=', 'Content-Length': '0'} 2022-09-12T20:22:02.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000703beafc1a929c35-00631f94ea-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.810 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:02.812 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:02.813 DEBUG:boto:path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:02.814 DEBUG:boto:auth_path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:02.815 DEBUG:boto:Method: HEAD 2022-09-12T20:22:02.817 DEBUG:boto:Path: /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:02.819 DEBUG:boto:Data: 2022-09-12T20:22:02.820 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.821 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.823 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.826 DEBUG:boto:Params: {} 2022-09-12T20:22:02.827 DEBUG:boto:Token: None 2022-09-12T20:22:02.828 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:02.829 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:yq/dGTAhKYXvqDvm2r1+jK6aDiY= 2022-09-12T20:22:02.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:yq/dGTAhKYXvqDvm2r1+jK6aDiY=', 'Content-Length': '0'} 2022-09-12T20:22:02.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4ee3be935899b5e-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:22:02.837 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:02.837 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:02.838 DEBUG:boto:path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:02.839 DEBUG:boto:auth_path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:02.840 DEBUG:boto:Method: GET 2022-09-12T20:22:02.841 DEBUG:boto:Path: /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:02.841 DEBUG:boto:Data: 2022-09-12T20:22:02.842 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.842 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.843 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.844 DEBUG:boto:Params: {} 2022-09-12T20:22:02.844 DEBUG:boto:Token: None 2022-09-12T20:22:02.845 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:02.846 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FD5z6EAGLHA40QhzbHLFWmUNaA4= 2022-09-12T20:22:02.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FD5z6EAGLHA40QhzbHLFWmUNaA4=', 'Content-Length': '0'} 2022-09-12T20:22:02.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096213819e551eb0d-00631f94ea-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.855 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:02.856 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:02.857 DEBUG:boto:path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:02.857 DEBUG:boto:auth_path=/jdpiak-76/obj44?versionId=null 2022-09-12T20:22:02.858 DEBUG:boto:Method: GET 2022-09-12T20:22:02.858 DEBUG:boto:Path: /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:02.859 DEBUG:boto:Data: 2022-09-12T20:22:02.860 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.860 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.861 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.861 DEBUG:boto:Params: {} 2022-09-12T20:22:02.862 DEBUG:boto:Token: None 2022-09-12T20:22:02.863 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj44?versionId=null 2022-09-12T20:22:02.863 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FD5z6EAGLHA40QhzbHLFWmUNaA4= 2022-09-12T20:22:02.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FD5z6EAGLHA40QhzbHLFWmUNaA4=', 'Content-Length': '0'} 2022-09-12T20:22:02.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003c8cc05d8720f9e-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.911 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:02.912 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:02.913 DEBUG:boto:path=/jdpiak-76/?prefix=obj44 2022-09-12T20:22:02.914 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj44 2022-09-12T20:22:02.914 DEBUG:boto:Method: GET 2022-09-12T20:22:02.915 DEBUG:boto:Path: /jdpiak-76/?prefix=obj44 2022-09-12T20:22:02.916 DEBUG:boto:Data: 2022-09-12T20:22:02.917 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.917 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.918 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.919 DEBUG:boto:Params: {} 2022-09-12T20:22:02.919 DEBUG:boto:Token: None 2022-09-12T20:22:02.920 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/ 2022-09-12T20:22:02.921 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0= 2022-09-12T20:22:02.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0=', 'Content-Length': '0'} 2022-09-12T20:22:02.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d9999b0880e1b4e-00631f94ea-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.937 DEBUG:boto:b'jdpiak-76obj441000falseobj442022-09-12T20:18:36.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:02.939 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:02.941 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:02.942 DEBUG:boto:path=/jdpiak-76/?prefix=obj44 2022-09-12T20:22:02.943 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj44 2022-09-12T20:22:02.944 DEBUG:boto:Method: GET 2022-09-12T20:22:02.945 DEBUG:boto:Path: /jdpiak-76/?prefix=obj44 2022-09-12T20:22:02.946 DEBUG:boto:Data: 2022-09-12T20:22:02.947 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.948 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.950 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.951 DEBUG:boto:Params: {} 2022-09-12T20:22:02.952 DEBUG:boto:Token: None 2022-09-12T20:22:02.954 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/ 2022-09-12T20:22:02.955 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0= 2022-09-12T20:22:02.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:537fj/4ic1r3l8YnVRU2ZXoD9V0=', 'Content-Length': '0'} 2022-09-12T20:22:02.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aab8808d43200c0f-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.961 DEBUG:boto:b'jdpiak-76obj441000falseobj442022-09-12T20:18:36.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:02.962 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:02.963 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:02.964 DEBUG:boto:Method: HEAD 2022-09-12T20:22:02.964 DEBUG:boto:Path: /jdpiak-76/obj44 2022-09-12T20:22:02.965 DEBUG:boto:Data: 2022-09-12T20:22:02.965 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.966 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.966 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.967 DEBUG:boto:Params: {} 2022-09-12T20:22:02.968 DEBUG:boto:Token: None 2022-09-12T20:22:02.968 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj44 2022-09-12T20:22:02.970 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mSMliCLRzWVWRV22RAyEtJ9xwJA= 2022-09-12T20:22:02.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mSMliCLRzWVWRV22RAyEtJ9xwJA=', 'Content-Length': '0'} 2022-09-12T20:22:02.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c30245b008548d64-00631f94ea-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.976 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:02.977 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:02.977 DEBUG:boto:Method: HEAD 2022-09-12T20:22:02.978 DEBUG:boto:Path: /jdpiak-76/obj44 2022-09-12T20:22:02.979 DEBUG:boto:Data: 2022-09-12T20:22:02.980 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.981 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:02.982 DEBUG:boto:Port: 8000 2022-09-12T20:22:02.982 DEBUG:boto:Params: {} 2022-09-12T20:22:02.983 DEBUG:boto:Token: None 2022-09-12T20:22:02.984 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj44 2022-09-12T20:22:02.985 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mSMliCLRzWVWRV22RAyEtJ9xwJA= 2022-09-12T20:22:02.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mSMliCLRzWVWRV22RAyEtJ9xwJA=', 'Content-Length': '0'} 2022-09-12T20:22:02.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000753d86305f755c05-00631f94ea-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:02.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-12T20:22:02.992 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:02.993 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:02.993 DEBUG:boto:Method: GET 2022-09-12T20:22:02.994 DEBUG:boto:Path: /jdpiak-76/obj44 2022-09-12T20:22:02.994 DEBUG:boto:Data: 2022-09-12T20:22:02.995 DEBUG:boto:Headers: {} 2022-09-12T20:22:02.996 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:02.996 DEBUG:boto:Port: 8001 2022-09-12T20:22:02.997 DEBUG:boto:Params: {} 2022-09-12T20:22:02.998 DEBUG:boto:Token: None 2022-09-12T20:22:02.999 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:02 GMT /jdpiak-76/obj44 2022-09-12T20:22:03.000 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NGcoC1t9kztaRhWjAMndjnQFLVg= 2022-09-12T20:22:03.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:02 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NGcoC1t9kztaRhWjAMndjnQFLVg=', 'Content-Length': '0'} 2022-09-12T20:22:03.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bd0596c87fdb339-00631f94eb-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:03.059 DEBUG:boto:path=/jdpiak-76/obj44 2022-09-12T20:22:03.060 DEBUG:boto:auth_path=/jdpiak-76/obj44 2022-09-12T20:22:03.061 DEBUG:boto:Method: GET 2022-09-12T20:22:03.062 DEBUG:boto:Path: /jdpiak-76/obj44 2022-09-12T20:22:03.063 DEBUG:boto:Data: 2022-09-12T20:22:03.064 DEBUG:boto:Headers: {} 2022-09-12T20:22:03.065 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:03.066 DEBUG:boto:Port: 8000 2022-09-12T20:22:03.067 DEBUG:boto:Params: {} 2022-09-12T20:22:03.068 DEBUG:boto:Token: None 2022-09-12T20:22:03.069 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:03 GMT /jdpiak-76/obj44 2022-09-12T20:22:03.069 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bWF60+sMrnq/WI+YfKMUR/TIxK0= 2022-09-12T20:22:03.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bWF60+sMrnq/WI+YfKMUR/TIxK0=', 'Content-Length': '0'} 2022-09-12T20:22:03.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b84b45244a267447-00631f94eb-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:03.078 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:03.079 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:03.080 DEBUG:boto:path=/jdpiak-76/?prefix=obj44 2022-09-12T20:22:03.080 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj44 2022-09-12T20:22:03.081 DEBUG:boto:Method: GET 2022-09-12T20:22:03.081 DEBUG:boto:Path: /jdpiak-76/?prefix=obj44 2022-09-12T20:22:03.082 DEBUG:boto:Data: 2022-09-12T20:22:03.083 DEBUG:boto:Headers: {} 2022-09-12T20:22:03.083 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:03.084 DEBUG:boto:Port: 8001 2022-09-12T20:22:03.084 DEBUG:boto:Params: {} 2022-09-12T20:22:03.085 DEBUG:boto:Token: None 2022-09-12T20:22:03.086 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:03 GMT /jdpiak-76/ 2022-09-12T20:22:03.087 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eIKsVLFct4M3JU07ljMr8SBXGvQ= 2022-09-12T20:22:03.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eIKsVLFct4M3JU07ljMr8SBXGvQ=', 'Content-Length': '0'} 2022-09-12T20:22:03.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000671970e0d21f3463-00631f94eb-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:03.097 DEBUG:boto:b'jdpiak-76obj441000falseobj442022-09-12T20:18:36.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:03.098 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:03.100 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:03.101 DEBUG:boto:path=/jdpiak-76/?prefix=obj44 2022-09-12T20:22:03.102 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj44 2022-09-12T20:22:03.103 DEBUG:boto:Method: GET 2022-09-12T20:22:03.104 DEBUG:boto:Path: /jdpiak-76/?prefix=obj44 2022-09-12T20:22:03.105 DEBUG:boto:Data: 2022-09-12T20:22:03.106 DEBUG:boto:Headers: {} 2022-09-12T20:22:03.107 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:03.108 DEBUG:boto:Port: 8000 2022-09-12T20:22:03.109 DEBUG:boto:Params: {} 2022-09-12T20:22:03.110 DEBUG:boto:Token: None 2022-09-12T20:22:03.111 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:03 GMT /jdpiak-76/ 2022-09-12T20:22:03.113 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eIKsVLFct4M3JU07ljMr8SBXGvQ= 2022-09-12T20:22:03.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eIKsVLFct4M3JU07ljMr8SBXGvQ=', 'Content-Length': '0'} 2022-09-12T20:22:03.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061509d8b61ed573d-00631f94eb-1ad5-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:03.120 DEBUG:boto:b'jdpiak-76obj441000falseobj442022-09-12T20:18:36.898Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:03.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:22:03.122 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:03.123 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:03.124 DEBUG:boto:path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:03.125 DEBUG:boto:auth_path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:03.126 DEBUG:boto:Method: GET 2022-09-12T20:22:03.127 DEBUG:boto:Path: /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:03.128 DEBUG:boto:Data: 2022-09-12T20:22:03.129 DEBUG:boto:Headers: {} 2022-09-12T20:22:03.130 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:03.131 DEBUG:boto:Port: 8001 2022-09-12T20:22:03.132 DEBUG:boto:Params: {} 2022-09-12T20:22:03.133 DEBUG:boto:Token: None 2022-09-12T20:22:03.134 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:03 GMT /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:03.135 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W9mibMnbMUnLhfhVEPUdDz2QgWk= 2022-09-12T20:22:03.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W9mibMnbMUnLhfhVEPUdDz2QgWk=', 'Content-Length': '0'} 2022-09-12T20:22:03.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069eeb677c66199f1-00631f94eb-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:03.142 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:03.143 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:03.144 DEBUG:boto:path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:03.145 DEBUG:boto:auth_path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:03.146 DEBUG:boto:Method: GET 2022-09-12T20:22:03.147 DEBUG:boto:Path: /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:03.148 DEBUG:boto:Data: 2022-09-12T20:22:03.149 DEBUG:boto:Headers: {} 2022-09-12T20:22:03.150 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:03.151 DEBUG:boto:Port: 8000 2022-09-12T20:22:03.152 DEBUG:boto:Params: {} 2022-09-12T20:22:03.153 DEBUG:boto:Token: None 2022-09-12T20:22:03.155 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:03 GMT /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:03.156 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W9mibMnbMUnLhfhVEPUdDz2QgWk= 2022-09-12T20:22:03.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W9mibMnbMUnLhfhVEPUdDz2QgWk=', 'Content-Length': '0'} 2022-09-12T20:22:03.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebd522683c3aee19-00631f94eb-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:03.202 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:03.204 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:03.204 DEBUG:boto:path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:03.205 DEBUG:boto:auth_path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:03.206 DEBUG:boto:Method: HEAD 2022-09-12T20:22:03.207 DEBUG:boto:Path: /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:03.208 DEBUG:boto:Data: 2022-09-12T20:22:03.208 DEBUG:boto:Headers: {} 2022-09-12T20:22:03.209 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:03.210 DEBUG:boto:Port: 8001 2022-09-12T20:22:03.211 DEBUG:boto:Params: {} 2022-09-12T20:22:03.212 DEBUG:boto:Token: None 2022-09-12T20:22:03.213 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:03 GMT /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:03.213 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:o9uHGJC5FeXAhR24FPrsexYJJGI= 2022-09-12T20:22:03.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:o9uHGJC5FeXAhR24FPrsexYJJGI=', 'Content-Length': '0'} 2022-09-12T20:22:03.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e87c157cb2e4c67-00631f94eb-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:03.219 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:03.220 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:03.220 DEBUG:boto:path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:03.221 DEBUG:boto:auth_path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:03.221 DEBUG:boto:Method: HEAD 2022-09-12T20:22:03.222 DEBUG:boto:Path: /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:03.222 DEBUG:boto:Data: 2022-09-12T20:22:03.223 DEBUG:boto:Headers: {} 2022-09-12T20:22:03.224 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:03.225 DEBUG:boto:Port: 8000 2022-09-12T20:22:03.226 DEBUG:boto:Params: {} 2022-09-12T20:22:03.227 DEBUG:boto:Token: None 2022-09-12T20:22:03.228 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:03 GMT /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:03.229 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:o9uHGJC5FeXAhR24FPrsexYJJGI= 2022-09-12T20:22:03.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:o9uHGJC5FeXAhR24FPrsexYJJGI=', 'Content-Length': '0'} 2022-09-12T20:22:03.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ace06c34d046c91-00631f94eb-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:03.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:22:03.237 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:03.237 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:03.238 DEBUG:boto:path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:03.239 DEBUG:boto:auth_path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:03.239 DEBUG:boto:Method: GET 2022-09-12T20:22:03.240 DEBUG:boto:Path: /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:03.240 DEBUG:boto:Data: 2022-09-12T20:22:03.241 DEBUG:boto:Headers: {} 2022-09-12T20:22:03.241 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:03.242 DEBUG:boto:Port: 8001 2022-09-12T20:22:03.242 DEBUG:boto:Params: {} 2022-09-12T20:22:03.243 DEBUG:boto:Token: None 2022-09-12T20:22:03.244 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:03 GMT /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:03.244 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W9mibMnbMUnLhfhVEPUdDz2QgWk= 2022-09-12T20:22:03.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W9mibMnbMUnLhfhVEPUdDz2QgWk=', 'Content-Length': '0'} 2022-09-12T20:22:03.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003826a2d90080da9e-00631f94eb-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:03.296 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:03.297 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:03.297 DEBUG:boto:path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:03.298 DEBUG:boto:auth_path=/jdpiak-76/obj55?versionId=null 2022-09-12T20:22:03.299 DEBUG:boto:Method: GET 2022-09-12T20:22:03.299 DEBUG:boto:Path: /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:03.300 DEBUG:boto:Data: 2022-09-12T20:22:03.301 DEBUG:boto:Headers: {} 2022-09-12T20:22:03.302 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:03.302 DEBUG:boto:Port: 8000 2022-09-12T20:22:03.303 DEBUG:boto:Params: {} 2022-09-12T20:22:03.304 DEBUG:boto:Token: None 2022-09-12T20:22:03.304 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:03 GMT /jdpiak-76/obj55?versionId=null 2022-09-12T20:22:03.305 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:W9mibMnbMUnLhfhVEPUdDz2QgWk= 2022-09-12T20:22:03.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:W9mibMnbMUnLhfhVEPUdDz2QgWk=', 'Content-Length': '0'} 2022-09-12T20:22:03.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7613c7b84121b9c-00631f94eb-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:03.311 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:03.312 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:03.313 DEBUG:boto:path=/jdpiak-76/?prefix=obj55 2022-09-12T20:22:03.313 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj55 2022-09-12T20:22:03.315 DEBUG:boto:Method: GET 2022-09-12T20:22:03.315 DEBUG:boto:Path: /jdpiak-76/?prefix=obj55 2022-09-12T20:22:03.316 DEBUG:boto:Data: 2022-09-12T20:22:03.317 DEBUG:boto:Headers: {} 2022-09-12T20:22:03.318 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:03.319 DEBUG:boto:Port: 8001 2022-09-12T20:22:03.320 DEBUG:boto:Params: {} 2022-09-12T20:22:03.321 DEBUG:boto:Token: None 2022-09-12T20:22:03.321 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:03 GMT /jdpiak-76/ 2022-09-12T20:22:03.322 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eIKsVLFct4M3JU07ljMr8SBXGvQ= 2022-09-12T20:22:03.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eIKsVLFct4M3JU07ljMr8SBXGvQ=', 'Content-Length': '0'} 2022-09-12T20:22:03.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f677b14d7e57b7d9-00631f94eb-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:03.328 DEBUG:boto:b'jdpiak-76obj551000falseobj552022-09-12T20:18:36.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:03.329 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:03.330 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:03.330 DEBUG:boto:path=/jdpiak-76/?prefix=obj55 2022-09-12T20:22:03.331 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj55 2022-09-12T20:22:03.332 DEBUG:boto:Method: GET 2022-09-12T20:22:03.332 DEBUG:boto:Path: /jdpiak-76/?prefix=obj55 2022-09-12T20:22:03.333 DEBUG:boto:Data: 2022-09-12T20:22:03.333 DEBUG:boto:Headers: {} 2022-09-12T20:22:03.334 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:03.334 DEBUG:boto:Port: 8000 2022-09-12T20:22:03.335 DEBUG:boto:Params: {} 2022-09-12T20:22:03.336 DEBUG:boto:Token: None 2022-09-12T20:22:03.336 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:03 GMT /jdpiak-76/ 2022-09-12T20:22:03.337 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eIKsVLFct4M3JU07ljMr8SBXGvQ= 2022-09-12T20:22:03.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eIKsVLFct4M3JU07ljMr8SBXGvQ=', 'Content-Length': '0'} 2022-09-12T20:22:03.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b840c62ce8251a31-00631f94eb-1ad5-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:03.343 DEBUG:boto:b'jdpiak-76obj551000falseobj552022-09-12T20:18:36.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:03.344 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:03.344 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:03.345 DEBUG:boto:Method: HEAD 2022-09-12T20:22:03.345 DEBUG:boto:Path: /jdpiak-76/obj55 2022-09-12T20:22:03.346 DEBUG:boto:Data: 2022-09-12T20:22:03.346 DEBUG:boto:Headers: {} 2022-09-12T20:22:03.347 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:03.348 DEBUG:boto:Port: 8001 2022-09-12T20:22:03.348 DEBUG:boto:Params: {} 2022-09-12T20:22:03.349 DEBUG:boto:Token: None 2022-09-12T20:22:03.350 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:03 GMT /jdpiak-76/obj55 2022-09-12T20:22:03.350 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J6IaVjKPwWSei34owPQxqXT1jNo= 2022-09-12T20:22:03.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J6IaVjKPwWSei34owPQxqXT1jNo=', 'Content-Length': '0'} 2022-09-12T20:22:03.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003643b36ed464660b-00631f94eb-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:03.354 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:03.355 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:03.356 DEBUG:boto:Method: HEAD 2022-09-12T20:22:03.356 DEBUG:boto:Path: /jdpiak-76/obj55 2022-09-12T20:22:03.357 DEBUG:boto:Data: 2022-09-12T20:22:03.358 DEBUG:boto:Headers: {} 2022-09-12T20:22:03.358 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:03.359 DEBUG:boto:Port: 8000 2022-09-12T20:22:03.359 DEBUG:boto:Params: {} 2022-09-12T20:22:03.360 DEBUG:boto:Token: None 2022-09-12T20:22:03.360 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:03 GMT /jdpiak-76/obj55 2022-09-12T20:22:03.361 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:J6IaVjKPwWSei34owPQxqXT1jNo= 2022-09-12T20:22:03.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:J6IaVjKPwWSei34owPQxqXT1jNo=', 'Content-Length': '0'} 2022-09-12T20:22:03.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d03704ed57705b37-00631f94eb-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:03.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-12T20:22:03.368 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:03.369 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:03.370 DEBUG:boto:Method: GET 2022-09-12T20:22:03.370 DEBUG:boto:Path: /jdpiak-76/obj55 2022-09-12T20:22:03.371 DEBUG:boto:Data: 2022-09-12T20:22:03.372 DEBUG:boto:Headers: {} 2022-09-12T20:22:03.373 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:03.373 DEBUG:boto:Port: 8001 2022-09-12T20:22:03.374 DEBUG:boto:Params: {} 2022-09-12T20:22:03.375 DEBUG:boto:Token: None 2022-09-12T20:22:03.376 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:03 GMT /jdpiak-76/obj55 2022-09-12T20:22:03.377 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y7OD5QYfyHbida74Y3WcUB/s6Pg= 2022-09-12T20:22:03.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y7OD5QYfyHbida74Y3WcUB/s6Pg=', 'Content-Length': '0'} 2022-09-12T20:22:03.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043b9ced3a88d4fb3-00631f94eb-1826-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:03.384 DEBUG:boto:path=/jdpiak-76/obj55 2022-09-12T20:22:03.384 DEBUG:boto:auth_path=/jdpiak-76/obj55 2022-09-12T20:22:03.385 DEBUG:boto:Method: GET 2022-09-12T20:22:03.386 DEBUG:boto:Path: /jdpiak-76/obj55 2022-09-12T20:22:03.386 DEBUG:boto:Data: 2022-09-12T20:22:03.387 DEBUG:boto:Headers: {} 2022-09-12T20:22:03.387 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:03.388 DEBUG:boto:Port: 8000 2022-09-12T20:22:03.388 DEBUG:boto:Params: {} 2022-09-12T20:22:03.389 DEBUG:boto:Token: None 2022-09-12T20:22:03.390 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:03 GMT /jdpiak-76/obj55 2022-09-12T20:22:03.391 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Y7OD5QYfyHbida74Y3WcUB/s6Pg= 2022-09-12T20:22:03.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Y7OD5QYfyHbida74Y3WcUB/s6Pg=', 'Content-Length': '0'} 2022-09-12T20:22:03.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Mon, 12 Sep 2022 20:18:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe8224642a41c9bf-00631f94eb-1ad5-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Mon, 12 Sep 2022 20:22:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:03.398 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:03.399 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:03.400 DEBUG:boto:path=/jdpiak-76/?prefix=obj55 2022-09-12T20:22:03.401 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj55 2022-09-12T20:22:03.402 DEBUG:boto:Method: GET 2022-09-12T20:22:03.403 DEBUG:boto:Path: /jdpiak-76/?prefix=obj55 2022-09-12T20:22:03.404 DEBUG:boto:Data: 2022-09-12T20:22:03.405 DEBUG:boto:Headers: {} 2022-09-12T20:22:03.406 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8001 2022-09-12T20:22:03.407 DEBUG:boto:Port: 8001 2022-09-12T20:22:03.408 DEBUG:boto:Params: {} 2022-09-12T20:22:03.409 DEBUG:boto:Token: None 2022-09-12T20:22:03.410 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:03 GMT /jdpiak-76/ 2022-09-12T20:22:03.411 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eIKsVLFct4M3JU07ljMr8SBXGvQ= 2022-09-12T20:22:03.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eIKsVLFct4M3JU07ljMr8SBXGvQ=', 'Content-Length': '0'} 2022-09-12T20:22:03.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003200d00bdeb2fe0e-00631f94eb-1826-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:03.459 DEBUG:boto:b'jdpiak-76obj551000falseobj552022-09-12T20:18:36.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:03.460 DEBUG:boto:path=/jdpiak-76/ 2022-09-12T20:22:03.461 DEBUG:boto:auth_path=/jdpiak-76/ 2022-09-12T20:22:03.462 DEBUG:boto:path=/jdpiak-76/?prefix=obj55 2022-09-12T20:22:03.463 DEBUG:boto:auth_path=/jdpiak-76/?prefix=obj55 2022-09-12T20:22:03.464 DEBUG:boto:Method: GET 2022-09-12T20:22:03.465 DEBUG:boto:Path: /jdpiak-76/?prefix=obj55 2022-09-12T20:22:03.466 DEBUG:boto:Data: 2022-09-12T20:22:03.467 DEBUG:boto:Headers: {} 2022-09-12T20:22:03.467 DEBUG:boto:Host: smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:03.468 DEBUG:boto:Port: 8000 2022-09-12T20:22:03.468 DEBUG:boto:Params: {} 2022-09-12T20:22:03.469 DEBUG:boto:Token: None 2022-09-12T20:22:03.470 DEBUG:boto:StringToSign: GET Mon, 12 Sep 2022 20:22:03 GMT /jdpiak-76/ 2022-09-12T20:22:03.470 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eIKsVLFct4M3JU07ljMr8SBXGvQ= 2022-09-12T20:22:03.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:03 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eIKsVLFct4M3JU07ljMr8SBXGvQ=', 'Content-Length': '0'} 2022-09-12T20:22:03.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025ae1d627d5e8feb-00631f94eb-1ad5-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Mon, 12 Sep 2022 20:22:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:03.478 DEBUG:boto:b'jdpiak-76obj551000falseobj552022-09-12T20:18:36.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-12T20:22:03.479 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=jdpiak-76 zones={test-zone3, test-zone2} 2022-09-12T20:22:03.479 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2022-09-12T20:22:03.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:03.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:03.482 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:22:03.545 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:03.546 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:03.611 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T20:22:03.611 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:03.612 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000010110", 2022-09-12T20:22:03.612 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:14:26.943083Z" 2022-09-12T20:22:03.612 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:03.613 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:03.613 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000018007", 2022-09-12T20:22:03.613 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:18:36.902390Z" 2022-09-12T20:22:03.614 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:03.614 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:03.614 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000018425", 2022-09-12T20:22:03.614 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:18:58.490510Z" 2022-09-12T20:22:03.615 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:03.615 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:03.615 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000012531", 2022-09-12T20:22:03.616 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:18:58.492086Z" 2022-09-12T20:22:03.616 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:03.616 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T20:22:03.617 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000010110', 1: '00000000000000000000:00000000000000018007', 2: '00000000000000000000:00000000000000018425', 3: '00000000000000000000:00000000000000012531'} 2022-09-12T20:22:03.618 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2022-09-12T20:22:03.619 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:03.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:03.620 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:22:03.680 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:03.681 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:03.740 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:22:03.741 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:22:03.741 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:22:03.741 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:22:03.742 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:22:03.742 INFO:teuthology.orchestra.run.smithi138.stdout: "instance_id": 11889344883753774101 2022-09-12T20:22:03.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:03.742 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:22:03.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:03.743 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:22:03.743 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:22:03.743 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T20:22:03.744 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000010110", 2022-09-12T20:22:03.744 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:22:03.744 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 9, 2022-09-12T20:22:03.745 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:22:03.745 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:14:26.943083Z" 2022-09-12T20:22:03.745 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:03.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:03.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:03.746 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:22:03.746 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:22:03.747 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T20:22:03.747 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000018007", 2022-09-12T20:22:03.747 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:22:03.748 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 67, 2022-09-12T20:22:03.748 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:22:03.748 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:18:36.902390Z" 2022-09-12T20:22:03.748 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:03.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:03.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:03.749 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:22:03.749 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:22:03.750 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T20:22:03.750 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000018425", 2022-09-12T20:22:03.750 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:22:03.750 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:22:03.751 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:22:03.751 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:18:58.490510Z" 2022-09-12T20:22:03.751 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:03.751 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:03.752 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:03.752 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:22:03.752 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:22:03.753 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T20:22:03.753 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000012531", 2022-09-12T20:22:03.753 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:22:03.753 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:22:03.754 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:22:03.754 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:18:58.492086Z" 2022-09-12T20:22:03.754 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:03.754 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:03.755 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:22:03.755 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:03.755 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:22:03.755 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 76, 2022-09-12T20:22:03.756 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 76 2022-09-12T20:22:03.756 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:03.756 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:22:03Z" 2022-09-12T20:22:03.756 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:22:03.758 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11889344883753774101 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010110", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2022-09-12T20:14:26.943083Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018007", "next_step_marker": "", "total_entries": 67, "pos": 0, "timestamp": "2022-09-12T20:18:36.902390Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018425", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:18:58.490510Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012531", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:18:58.492086Z" } } ] }, "full_sync": { "total": 76, "complete": 76 }, "current_time": "2022-09-12T20:22:03Z" } 2022-09-12T20:22:03.758 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010110', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2022-09-12T20:14:26.943083Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018007', 'next_step_marker': '', 'total_entries': 67, 'pos': 0, 'timestamp': '2022-09-12T20:18:36.902390Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018425', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:18:58.490510Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012531', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:18:58.492086Z'}}] 2022-09-12T20:22:03.759 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000010110', 1: '00000000000000000000:00000000000000018007', 2: '00000000000000000000:00000000000000018425', 3: '00000000000000000000:00000000000000012531'} 2022-09-12T20:22:03.760 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000010110', 1: '00000000000000000000:00000000000000018007', 2: '00000000000000000000:00000000000000018425', 3: '00000000000000000000:00000000000000012531'} 2022-09-12T20:22:03.760 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2022-09-12T20:22:03.761 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone3 2022-09-12T20:22:03.761 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:03.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:03.762 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:22:03.814 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:03.815 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:03.867 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T20:22:03.867 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:03.867 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000010110", 2022-09-12T20:22:03.868 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:14:26.943083Z" 2022-09-12T20:22:03.868 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:03.868 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:03.869 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000018007", 2022-09-12T20:22:03.869 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:18:36.902390Z" 2022-09-12T20:22:03.869 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:03.869 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:03.870 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000018425", 2022-09-12T20:22:03.870 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:18:58.490510Z" 2022-09-12T20:22:03.870 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:03.870 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:03.871 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000012531", 2022-09-12T20:22:03.871 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:18:58.492086Z" 2022-09-12T20:22:03.871 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:03.872 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T20:22:03.873 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000010110', 1: '00000000000000000000:00000000000000018007', 2: '00000000000000000000:00000000000000018425', 3: '00000000000000000000:00000000000000012531'} 2022-09-12T20:22:03.873 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2022-09-12T20:22:03.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:03.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:03.875 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:22:03.927 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:03.928 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:03.981 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T20:22:03.982 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T20:22:03.982 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T20:22:03.982 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T20:22:03.983 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T20:22:03.983 INFO:teuthology.orchestra.run.smithi035.stdout: "instance_id": 16681684026229800507 2022-09-12T20:22:03.983 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:03.984 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T20:22:03.984 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:03.984 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T20:22:03.984 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:22:03.985 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T20:22:03.985 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000010110", 2022-09-12T20:22:03.985 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:22:03.986 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:22:03.986 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:22:03.986 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T20:14:26.943083Z" 2022-09-12T20:22:03.987 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:03.987 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:03.987 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:03.988 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T20:22:03.988 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:22:03.988 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T20:22:03.989 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000018007", 2022-09-12T20:22:03.989 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:22:03.989 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:22:03.989 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:22:03.990 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T20:18:36.902390Z" 2022-09-12T20:22:03.990 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:03.990 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:03.991 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:03.991 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T20:22:03.991 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:22:03.991 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T20:22:03.992 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000018425", 2022-09-12T20:22:03.992 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:22:03.992 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:22:03.992 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:22:03.993 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T20:18:58.490510Z" 2022-09-12T20:22:03.993 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:03.993 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:03.994 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:03.994 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T20:22:03.994 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:22:03.995 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T20:22:03.995 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000012531", 2022-09-12T20:22:03.995 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:22:03.996 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:22:03.996 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:22:03.996 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T20:18:58.492086Z" 2022-09-12T20:22:03.997 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:03.997 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:03.997 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T20:22:03.997 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:03.998 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T20:22:03.998 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 0, 2022-09-12T20:22:03.998 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 0 2022-09-12T20:22:03.998 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:03.999 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T20:22:03Z" 2022-09-12T20:22:03.999 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T20:22:04.000 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16681684026229800507 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010110", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:14:26.943083Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018007", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:18:36.902390Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018425", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:18:58.490510Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012531", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:18:58.492086Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T20:22:03Z" } 2022-09-12T20:22:04.001 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010110', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:14:26.943083Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018007', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:18:36.902390Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018425', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:18:58.490510Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012531', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:18:58.492086Z'}}] 2022-09-12T20:22:04.002 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000010110', 1: '00000000000000000000:00000000000000018007', 2: '00000000000000000000:00000000000000018425', 3: '00000000000000000000:00000000000000012531'} 2022-09-12T20:22:04.002 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000010110', 1: '00000000000000000000:00000000000000018007', 2: '00000000000000000000:00000000000000018425', 3: '00000000000000000000:00000000000000012531'} 2022-09-12T20:22:04.003 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2022-09-12T20:22:04.003 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone4 2022-09-12T20:22:04.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:04.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:04.005 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:22:04.056 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:04.057 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:04.106 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T20:22:04.107 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:04.107 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000010110", 2022-09-12T20:22:04.107 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:14:26.943083Z" 2022-09-12T20:22:04.108 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:04.108 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:04.108 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000018007", 2022-09-12T20:22:04.109 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:18:36.902390Z" 2022-09-12T20:22:04.109 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:04.109 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:04.109 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000018425", 2022-09-12T20:22:04.110 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:18:58.490510Z" 2022-09-12T20:22:04.110 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:04.110 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:04.110 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000012531", 2022-09-12T20:22:04.111 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:18:58.492086Z" 2022-09-12T20:22:04.111 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:04.111 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T20:22:04.113 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000010110', 1: '00000000000000000000:00000000000000018007', 2: '00000000000000000000:00000000000000018425', 3: '00000000000000000000:00000000000000012531'} 2022-09-12T20:22:04.113 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2022-09-12T20:22:04.114 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:04.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:04.115 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:22:04.169 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:04.169 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:04.212 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:22:04.213 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:22:04.213 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:22:04.213 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:22:04.214 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:22:04.214 INFO:teuthology.orchestra.run.smithi138.stdout: "instance_id": 14449053848353038014 2022-09-12T20:22:04.214 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:04.215 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:22:04.215 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:04.215 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:22:04.215 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:22:04.216 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T20:22:04.216 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000010110", 2022-09-12T20:22:04.216 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:22:04.216 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:22:04.217 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:22:04.217 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:14:26.943083Z" 2022-09-12T20:22:04.217 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:04.217 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:04.218 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:04.218 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:22:04.218 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:22:04.218 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T20:22:04.219 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000018007", 2022-09-12T20:22:04.219 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:22:04.219 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:22:04.219 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:22:04.220 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:18:36.902390Z" 2022-09-12T20:22:04.220 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:04.220 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:04.220 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:04.221 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:22:04.221 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:22:04.221 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T20:22:04.221 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000018425", 2022-09-12T20:22:04.222 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:22:04.222 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:22:04.222 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:22:04.222 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:18:58.490510Z" 2022-09-12T20:22:04.223 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:04.223 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:04.223 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:04.223 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:22:04.224 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:22:04.224 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T20:22:04.224 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000012531", 2022-09-12T20:22:04.224 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:22:04.225 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:22:04.225 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:22:04.225 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:18:58.492086Z" 2022-09-12T20:22:04.225 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:04.226 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:04.226 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:22:04.226 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:04.226 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:22:04.227 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 0, 2022-09-12T20:22:04.227 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 0 2022-09-12T20:22:04.227 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:04.228 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:22:04Z" 2022-09-12T20:22:04.228 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:22:04.229 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14449053848353038014 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010110", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:14:26.943083Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018007", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:18:36.902390Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018425", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:18:58.490510Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012531", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:18:58.492086Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T20:22:04Z" } 2022-09-12T20:22:04.230 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010110', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:14:26.943083Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018007', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:18:36.902390Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018425', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:18:58.490510Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012531', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:18:58.492086Z'}}] 2022-09-12T20:22:04.231 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000010110', 1: '00000000000000000000:00000000000000018007', 2: '00000000000000000000:00000000000000018425', 3: '00000000000000000000:00000000000000012531'} 2022-09-12T20:22:04.231 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000010110', 1: '00000000000000000000:00000000000000018007', 2: '00000000000000000000:00000000000000018425', 3: '00000000000000000000:00000000000000012531'} 2022-09-12T20:22:04.232 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2022-09-12T20:22:04.232 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2022-09-12T20:22:04.232 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:04.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:04.234 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:22:04.282 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:04.282 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:04.336 INFO:teuthology.orchestra.run.smithi138.stdout:[ 2022-09-12T20:22:04.337 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:04.337 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000006287", 2022-09-12T20:22:04.338 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T20:20:13.353912Z" 2022-09-12T20:22:04.338 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:04.338 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:04.339 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000013754", 2022-09-12T20:22:04.339 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T20:20:27.180071Z" 2022-09-12T20:22:04.339 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:04.339 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:04.340 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000016435", 2022-09-12T20:22:04.340 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T20:20:27.181912Z" 2022-09-12T20:22:04.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:04.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:04.341 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000008712", 2022-09-12T20:22:04.341 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T20:20:27.183709Z" 2022-09-12T20:22:04.341 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:04.342 INFO:teuthology.orchestra.run.smithi138.stdout:] 2022-09-12T20:22:04.343 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000006287', 1: '00000000000000000000:00000000000000013754', 2: '00000000000000000000:00000000000000016435', 3: '00000000000000000000:00000000000000008712'} 2022-09-12T20:22:04.343 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2022-09-12T20:22:04.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:04.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:04.345 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:22:04.399 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:04.399 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:04.450 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T20:22:04.450 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T20:22:04.450 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T20:22:04.451 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T20:22:04.451 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T20:22:04.451 INFO:teuthology.orchestra.run.smithi035.stdout: "instance_id": 2819386190429732693 2022-09-12T20:22:04.452 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:04.452 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T20:22:04.452 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:04.453 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T20:22:04.453 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:22:04.453 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T20:22:04.454 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006287", 2022-09-12T20:22:04.454 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:22:04.454 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:22:04.454 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:22:04.455 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T20:20:13.353912Z" 2022-09-12T20:22:04.455 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:04.455 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:04.455 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:04.456 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T20:22:04.456 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:22:04.456 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T20:22:04.457 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000013754", 2022-09-12T20:22:04.457 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:22:04.457 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:22:04.458 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:22:04.458 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T20:20:27.180071Z" 2022-09-12T20:22:04.458 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:04.459 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:04.459 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:04.459 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T20:22:04.459 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:22:04.460 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T20:22:04.460 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000016435", 2022-09-12T20:22:04.460 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:22:04.461 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:22:04.461 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:22:04.461 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T20:20:27.181912Z" 2022-09-12T20:22:04.462 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:04.462 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:04.462 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:04.462 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T20:22:04.463 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:22:04.463 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T20:22:04.463 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000008712", 2022-09-12T20:22:04.463 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:22:04.464 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:22:04.464 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:22:04.464 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T20:20:27.183709Z" 2022-09-12T20:22:04.464 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:04.465 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:04.465 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T20:22:04.465 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:04.465 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T20:22:04.466 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 0, 2022-09-12T20:22:04.466 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 0 2022-09-12T20:22:04.466 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:04.467 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T20:22:04Z" 2022-09-12T20:22:04.467 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T20:22:04.468 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 2819386190429732693 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006287", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:20:13.353912Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013754", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:20:27.180071Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000016435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:20:27.181912Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008712", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:20:27.183709Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T20:22:04Z" } 2022-09-12T20:22:04.469 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006287', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:20:13.353912Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013754', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:20:27.180071Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000016435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:20:27.181912Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008712', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:20:27.183709Z'}}] 2022-09-12T20:22:04.470 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000006287', 1: '00000000000000000000:00000000000000013754', 2: '00000000000000000000:00000000000000016435', 3: '00000000000000000000:00000000000000008712'} 2022-09-12T20:22:04.470 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000006287', 1: '00000000000000000000:00000000000000013754', 2: '00000000000000000000:00000000000000016435', 3: '00000000000000000000:00000000000000008712'} 2022-09-12T20:22:04.471 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2022-09-12T20:22:04.471 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone3 2022-09-12T20:22:04.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:04.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:04.473 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:22:04.521 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:04.522 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:04.584 INFO:teuthology.orchestra.run.smithi138.stdout:[ 2022-09-12T20:22:04.584 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:04.585 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000006287", 2022-09-12T20:22:04.585 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T20:20:13.353912Z" 2022-09-12T20:22:04.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:04.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:04.586 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000013754", 2022-09-12T20:22:04.586 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T20:20:27.180071Z" 2022-09-12T20:22:04.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:04.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:04.587 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000016435", 2022-09-12T20:22:04.587 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T20:20:27.181912Z" 2022-09-12T20:22:04.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:04.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:04.588 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000008712", 2022-09-12T20:22:04.588 INFO:teuthology.orchestra.run.smithi138.stdout: "last_update": "2022-09-12T20:20:27.183709Z" 2022-09-12T20:22:04.588 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:04.588 INFO:teuthology.orchestra.run.smithi138.stdout:] 2022-09-12T20:22:04.590 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000006287', 1: '00000000000000000000:00000000000000013754', 2: '00000000000000000000:00000000000000016435', 3: '00000000000000000000:00000000000000008712'} 2022-09-12T20:22:04.590 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2022-09-12T20:22:04.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:04.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:04.592 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:22:04.645 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:04.646 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:04.699 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T20:22:04.699 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T20:22:04.699 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T20:22:04.700 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T20:22:04.700 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T20:22:04.700 INFO:teuthology.orchestra.run.smithi035.stdout: "instance_id": 16602476873656756125 2022-09-12T20:22:04.700 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:04.701 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T20:22:04.701 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:04.701 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T20:22:04.701 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:22:04.702 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T20:22:04.702 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006287", 2022-09-12T20:22:04.702 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:22:04.703 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:22:04.703 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:22:04.703 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T20:20:13.353912Z" 2022-09-12T20:22:04.703 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:04.704 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:04.704 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:04.704 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T20:22:04.704 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:22:04.705 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T20:22:04.705 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000013754", 2022-09-12T20:22:04.705 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:22:04.705 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:22:04.706 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:22:04.706 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T20:20:27.180071Z" 2022-09-12T20:22:04.706 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:04.706 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:04.707 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:04.707 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T20:22:04.707 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:22:04.707 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T20:22:04.708 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000016435", 2022-09-12T20:22:04.708 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:22:04.708 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:22:04.708 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:22:04.709 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T20:20:27.181912Z" 2022-09-12T20:22:04.709 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:04.709 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:04.709 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:04.710 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T20:22:04.710 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:22:04.710 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T20:22:04.710 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000008712", 2022-09-12T20:22:04.711 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:22:04.711 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:22:04.711 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:22:04.711 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T20:20:27.183709Z" 2022-09-12T20:22:04.712 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:04.712 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:04.712 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T20:22:04.712 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:04.713 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T20:22:04.713 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 0, 2022-09-12T20:22:04.713 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 0 2022-09-12T20:22:04.713 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:04.714 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T20:22:04Z" 2022-09-12T20:22:04.714 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T20:22:04.715 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16602476873656756125 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006287", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:20:13.353912Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013754", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:20:27.180071Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000016435", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:20:27.181912Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008712", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:20:27.183709Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T20:22:04Z" } 2022-09-12T20:22:04.716 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006287', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:20:13.353912Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013754', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:20:27.180071Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000016435', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:20:27.181912Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008712', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:20:27.183709Z'}}] 2022-09-12T20:22:04.716 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000006287', 1: '00000000000000000000:00000000000000013754', 2: '00000000000000000000:00000000000000016435', 3: '00000000000000000000:00000000000000008712'} 2022-09-12T20:22:04.717 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000006287', 1: '00000000000000000000:00000000000000013754', 2: '00000000000000000000:00000000000000016435', 3: '00000000000000000000:00000000000000008712'} 2022-09-12T20:22:04.717 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2022-09-12T20:22:04.718 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone4 2022-09-12T20:22:04.718 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone1 2022-09-12T20:22:04.719 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:04.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:04.720 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:22:04.769 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:04.770 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:04.818 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T20:22:04.819 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:04.819 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000005441", 2022-09-12T20:22:04.819 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:12:44.392629Z" 2022-09-12T20:22:04.820 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:04.820 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:04.820 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000011074", 2022-09-12T20:22:04.820 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:18:44.299895Z" 2022-09-12T20:22:04.821 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:04.821 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:04.821 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000012487", 2022-09-12T20:22:04.821 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:18:59.031000Z" 2022-09-12T20:22:04.822 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:04.822 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:04.822 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006025", 2022-09-12T20:22:04.822 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:18:59.032755Z" 2022-09-12T20:22:04.823 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:04.823 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T20:22:04.824 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000005441', 1: '00000000000000000000:00000000000000011074', 2: '00000000000000000000:00000000000000012487', 3: '00000000000000000000:00000000000000006025'} 2022-09-12T20:22:04.825 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2022-09-12T20:22:04.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:04.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:04.827 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:22:04.879 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:04.879 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:04.930 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T20:22:04.930 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T20:22:04.931 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T20:22:04.931 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T20:22:04.932 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T20:22:04.932 INFO:teuthology.orchestra.run.smithi035.stdout: "instance_id": 9390097840922254989 2022-09-12T20:22:04.932 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:04.933 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T20:22:04.933 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:04.934 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T20:22:04.935 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:22:04.935 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T20:22:04.936 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000005441", 2022-09-12T20:22:04.936 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:22:04.936 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:22:04.936 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:22:04.937 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T20:12:44.392629Z" 2022-09-12T20:22:04.937 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:04.937 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:04.937 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:04.938 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T20:22:04.938 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:22:04.938 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T20:22:04.938 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000011074", 2022-09-12T20:22:04.939 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:22:04.939 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:22:04.939 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:22:04.940 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T20:18:44.299895Z" 2022-09-12T20:22:04.940 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:04.940 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:04.941 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:04.941 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T20:22:04.941 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:22:04.941 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T20:22:04.942 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000012487", 2022-09-12T20:22:04.942 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:22:04.942 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:22:04.942 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:22:04.943 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T20:18:59.031000Z" 2022-09-12T20:22:04.943 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:04.943 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:04.943 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:04.944 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T20:22:04.944 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:22:04.944 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "incremental-sync", 2022-09-12T20:22:04.944 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006025", 2022-09-12T20:22:04.945 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:22:04.945 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:22:04.945 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:22:04.946 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T20:18:59.032755Z" 2022-09-12T20:22:04.946 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:04.946 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:04.946 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T20:22:04.947 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:04.947 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T20:22:04.947 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 0, 2022-09-12T20:22:04.948 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 0 2022-09-12T20:22:04.948 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:04.948 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T20:22:04Z" 2022-09-12T20:22:04.948 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T20:22:04.950 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 9390097840922254989 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005441", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:12:44.392629Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011074", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:18:44.299895Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012487", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:18:59.031000Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006025", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:18:59.032755Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T20:22:04Z" } 2022-09-12T20:22:04.951 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005441', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:12:44.392629Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011074', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:18:44.299895Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012487', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:18:59.031000Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006025', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:18:59.032755Z'}}] 2022-09-12T20:22:04.952 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005441', 1: '00000000000000000000:00000000000000011074', 2: '00000000000000000000:00000000000000012487', 3: '00000000000000000000:00000000000000006025'} 2022-09-12T20:22:04.952 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005441', 1: '00000000000000000000:00000000000000011074', 2: '00000000000000000000:00000000000000012487', 3: '00000000000000000000:00000000000000006025'} 2022-09-12T20:22:04.953 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2022-09-12T20:22:04.953 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone2 2022-09-12T20:22:04.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:04.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:04.955 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:22:05.005 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:05.006 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:05.060 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T20:22:05.061 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:05.061 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000005441", 2022-09-12T20:22:05.061 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:12:44.392629Z" 2022-09-12T20:22:05.062 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:05.062 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:05.062 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000011074", 2022-09-12T20:22:05.062 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:18:44.299895Z" 2022-09-12T20:22:05.063 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:05.063 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:05.063 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000012487", 2022-09-12T20:22:05.064 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:18:59.031000Z" 2022-09-12T20:22:05.064 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:05.064 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:05.064 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "00000000000000000000:00000000000000006025", 2022-09-12T20:22:05.065 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:18:59.032755Z" 2022-09-12T20:22:05.065 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:05.065 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T20:22:05.066 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000005441', 1: '00000000000000000000:00000000000000011074', 2: '00000000000000000000:00000000000000012487', 3: '00000000000000000000:00000000000000006025'} 2022-09-12T20:22:05.067 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2022-09-12T20:22:05.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:05.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:05.069 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:22:05.120 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:05.120 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:05.192 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:22:05.192 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:22:05.193 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:22:05.193 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:22:05.193 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:22:05.193 INFO:teuthology.orchestra.run.smithi138.stdout: "instance_id": 15445976120917781840 2022-09-12T20:22:05.194 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:05.194 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:22:05.195 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:05.195 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:22:05.195 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:22:05.196 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T20:22:05.196 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000005441", 2022-09-12T20:22:05.196 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:22:05.196 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:22:05.197 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:22:05.197 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:12:44.392629Z" 2022-09-12T20:22:05.197 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:05.197 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:05.198 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:05.198 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:22:05.198 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:22:05.198 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T20:22:05.199 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000011074", 2022-09-12T20:22:05.199 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:22:05.199 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:22:05.199 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:22:05.200 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:18:44.299895Z" 2022-09-12T20:22:05.200 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:05.200 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:05.200 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:05.201 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:22:05.201 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:22:05.201 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T20:22:05.201 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000012487", 2022-09-12T20:22:05.202 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:22:05.202 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:22:05.202 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:22:05.203 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:18:59.031000Z" 2022-09-12T20:22:05.203 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:05.203 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:05.203 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:05.204 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:22:05.204 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:22:05.204 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "incremental-sync", 2022-09-12T20:22:05.204 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "00000000000000000000:00000000000000006025", 2022-09-12T20:22:05.205 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:22:05.205 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:22:05.206 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:22:05.206 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:18:59.032755Z" 2022-09-12T20:22:05.206 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:05.206 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:05.207 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:22:05.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:05.207 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:22:05.207 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 0, 2022-09-12T20:22:05.208 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 0 2022-09-12T20:22:05.208 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:05.208 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:22:05Z" 2022-09-12T20:22:05.208 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:22:05.210 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 15445976120917781840 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005441", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:12:44.392629Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011074", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:18:44.299895Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012487", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:18:59.031000Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006025", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:18:59.032755Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-12T20:22:05Z" } 2022-09-12T20:22:05.210 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005441', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:12:44.392629Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011074', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:18:44.299895Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012487', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:18:59.031000Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006025', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:18:59.032755Z'}}] 2022-09-12T20:22:05.211 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005441', 1: '00000000000000000000:00000000000000011074', 2: '00000000000000000000:00000000000000012487', 3: '00000000000000000000:00000000000000006025'} 2022-09-12T20:22:05.211 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005441', 1: '00000000000000000000:00000000000000011074', 2: '00000000000000000000:00000000000000012487', 3: '00000000000000000000:00000000000000006025'} 2022-09-12T20:22:05.212 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2022-09-12T20:22:05.212 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone4 2022-09-12T20:22:05.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... ok 2022-09-12T20:22:05.216 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:05.216 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:05.217 DEBUG:boto:Method: PUT 2022-09-12T20:22:05.217 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:05.218 DEBUG:boto:Data: 2022-09-12T20:22:05.218 DEBUG:boto:Headers: {} 2022-09-12T20:22:05.219 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:05.219 DEBUG:boto:Port: 8000 2022-09-12T20:22:05.220 DEBUG:boto:Params: {} 2022-09-12T20:22:05.220 DEBUG:boto:Token: None 2022-09-12T20:22:05.221 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 20:22:05 GMT /jdpiak-77/ 2022-09-12T20:22:05.221 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:v3Ue9fMbdGvD8Cg65ioR/1dQNss= 2022-09-12T20:22:05.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:05 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:v3Ue9fMbdGvD8Cg65ioR/1dQNss=', 'Content-Length': '0'} 2022-09-12T20:22:05.241 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c41b98c1cf288613-00631f94ed-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:05.242 DEBUG:rgw_multi.tests:created bucket=jdpiak-77 2022-09-12T20:22:05.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:05.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:05.244 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:22:05.296 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:05.297 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:05.333 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=38ca1f57-1358-496c-a99b-df49ee365a4f 2022-09-12T20:22:05.348 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T20:22:05.348 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:05.348 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T20:22:05.348 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T20:22:05.349 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:05.349 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:05.349 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T20:22:05.350 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T20:22:05.350 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:05.351 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:05.351 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663014125.239690_900.1", 2022-09-12T20:22:05.351 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:22:05.239690Z" 2022-09-12T20:22:05.352 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:05.352 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:05.352 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:22:05.353 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:16:38.638998Z" 2022-09-12T20:22:05.353 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:05.353 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T20:22:05.354 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1663014125.239690_900.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:22:05.355 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T20:22:05.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:05.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:05.357 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:22:05.407 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:05.408 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:05.444 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:22:05.442+0000 7f0a2edd6600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:22:05.445 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:22:05.443+0000 7f0a2edd6600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:22:05.464 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:22:05.464 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:22:05.465 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:22:05.465 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:22:05.465 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:22:05.466 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:22:05.466 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:22:05.467 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:05.468 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:22:05.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:05.469 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:22:05.469 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:22:05.470 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:22:05.471 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:22:05.471 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:22:05.472 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:22:05.472 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:22:05.473 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:22:05.473 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:22:05.474 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:05.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:05.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:05.475 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:22:05.476 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:22:05.476 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:22:05.477 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:22:05.478 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:22:05.478 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:22:05.479 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:22:05.479 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:22:05.480 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:22:05.480 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:05.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:05.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:05.481 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:22:05.482 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:22:05.482 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:22:05.482 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663014125.239690_900.1", 2022-09-12T20:22:05.482 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:22:05.483 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:22:05.483 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:22:05.483 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:22:05.239690Z", 2022-09-12T20:22:05.484 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:22:05.484 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:05.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:05.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:05.485 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:22:05.485 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:22:05.485 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:22:05.485 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:22:05.486 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:22:05.486 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:22:05.486 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:22:05.486 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:22:05.487 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:22:05.487 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:05.487 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:05.487 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:22:05.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:05.488 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:22:05.488 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:22:05.488 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:22:05.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:05.489 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:22:05Z" 2022-09-12T20:22:05.489 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:22:05.490 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663014125.239690_900.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:22:05.239690Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:22:05Z" } 2022-09-12T20:22:05.491 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663014125.239690_900.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:22:05.239690Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:22:05.492 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663014125.239690_900.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:22:05.492 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663014125.239690_900.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:22:05.493 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-12T20:22:05.493 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-12T20:22:05.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:05.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:05.495 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:22:05.547 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:05.547 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:05.580 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:22:05.578+0000 7fdef59a9600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:22:05.581 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T20:22:05.580+0000 7fdef59a9600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:22:05.605 INFO:teuthology.orchestra.run.smithi035.stdout:{ 2022-09-12T20:22:05.606 INFO:teuthology.orchestra.run.smithi035.stdout: "sync_status": { 2022-09-12T20:22:05.606 INFO:teuthology.orchestra.run.smithi035.stdout: "info": { 2022-09-12T20:22:05.606 INFO:teuthology.orchestra.run.smithi035.stdout: "status": "sync", 2022-09-12T20:22:05.607 INFO:teuthology.orchestra.run.smithi035.stdout: "num_shards": 4, 2022-09-12T20:22:05.607 INFO:teuthology.orchestra.run.smithi035.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:22:05.607 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T20:22:05.607 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:05.608 INFO:teuthology.orchestra.run.smithi035.stdout: "markers": [ 2022-09-12T20:22:05.608 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:05.608 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 0, 2022-09-12T20:22:05.608 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:22:05.609 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T20:22:05.609 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:22:05.609 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:22:05.609 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 1, 2022-09-12T20:22:05.610 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:22:05.610 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:22:05.610 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T20:22:05.611 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:05.611 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:05.611 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:05.611 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 1, 2022-09-12T20:22:05.612 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:22:05.612 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T20:22:05.612 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T20:22:05.612 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:22:05.613 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:22:05.613 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:22:05.613 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "0.000000", 2022-09-12T20:22:05.614 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 2 2022-09-12T20:22:05.614 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:05.614 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:05.614 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:05.615 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 2, 2022-09-12T20:22:05.615 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:22:05.615 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T20:22:05.615 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663014125.239690_900.1", 2022-09-12T20:22:05.616 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:22:05.616 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:22:05.616 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:22:05.616 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T20:22:05.239690Z", 2022-09-12T20:22:05.617 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T20:22:05.617 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:05.617 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:05.617 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:22:05.618 INFO:teuthology.orchestra.run.smithi035.stdout: "key": 3, 2022-09-12T20:22:05.618 INFO:teuthology.orchestra.run.smithi035.stdout: "val": { 2022-09-12T20:22:05.618 INFO:teuthology.orchestra.run.smithi035.stdout: "state": 1, 2022-09-12T20:22:05.618 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:22:05.619 INFO:teuthology.orchestra.run.smithi035.stdout: "next_step_marker": "", 2022-09-12T20:22:05.619 INFO:teuthology.orchestra.run.smithi035.stdout: "total_entries": 0, 2022-09-12T20:22:05.619 INFO:teuthology.orchestra.run.smithi035.stdout: "pos": 0, 2022-09-12T20:22:05.619 INFO:teuthology.orchestra.run.smithi035.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:22:05.620 INFO:teuthology.orchestra.run.smithi035.stdout: "realm_epoch": 4 2022-09-12T20:22:05.620 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:05.620 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:22:05.620 INFO:teuthology.orchestra.run.smithi035.stdout: ] 2022-09-12T20:22:05.621 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:05.621 INFO:teuthology.orchestra.run.smithi035.stdout: "full_sync": { 2022-09-12T20:22:05.621 INFO:teuthology.orchestra.run.smithi035.stdout: "total": 1, 2022-09-12T20:22:05.621 INFO:teuthology.orchestra.run.smithi035.stdout: "complete": 1 2022-09-12T20:22:05.622 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:22:05.622 INFO:teuthology.orchestra.run.smithi035.stdout: "current_time": "2022-09-12T20:22:05Z" 2022-09-12T20:22:05.622 INFO:teuthology.orchestra.run.smithi035.stdout:} 2022-09-12T20:22:05.623 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663014125.239690_900.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:22:05.239690Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:22:05Z" } 2022-09-12T20:22:05.623 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663014125.239690_900.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:22:05.239690Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:22:05.624 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663014125.239690_900.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:22:05.624 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663014125.239690_900.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:22:05.625 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-12T20:22:05.625 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-12T20:22:05.626 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:05.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:05.627 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:22:05.677 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:05.677 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:05.719 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:22:05.718+0000 7f2ee9953600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:22:05.720 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:22:05.719+0000 7f2ee9953600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:22:05.737 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:22:05.738 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:22:05.738 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:22:05.738 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:22:05.739 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:22:05.739 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:22:05.739 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:22:05.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:05.740 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:22:05.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:05.740 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:22:05.740 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:22:05.741 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:22:05.741 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:22:05.741 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:22:05.741 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:22:05.742 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:22:05.742 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:22:05.742 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:22:05.743 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:05.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:05.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:05.743 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:22:05.744 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:22:05.744 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:22:05.744 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:22:05.744 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:22:05.745 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:22:05.745 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:22:05.745 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:22:05.746 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:22:05.746 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:05.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:05.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:05.747 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:22:05.747 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:22:05.747 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:22:05.747 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663014125.233171_897.1", 2022-09-12T20:22:05.748 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:22:05.748 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:22:05.748 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:22:05.748 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:22:05.233171Z", 2022-09-12T20:22:05.749 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:22:05.749 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:05.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:05.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:05.750 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:22:05.750 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:22:05.750 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:22:05.750 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:22:05.751 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:22:05.751 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:22:05.751 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:22:05.752 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:22:05.752 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:22:05.752 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:05.752 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:05.753 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:22:05.753 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:05.753 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:22:05.753 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:22:05.754 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:22:05.754 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:05.754 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:22:05Z" 2022-09-12T20:22:05.754 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:22:05.756 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663014125.233171_897.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:22:05.233171Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:22:05Z" } 2022-09-12T20:22:05.756 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663014125.233171_897.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:22:05.233171Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:22:05.757 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663014125.239690_900.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:22:05.757 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663014125.233171_897.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:22:05.758 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1663014125.239690_900.1 target=1_1663014125.233171_897.1 2022-09-12T20:22:10.760 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:10.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:22:10.761 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:22:10.818 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:10.819 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:10.857 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:22:10.855+0000 7f761ba88600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:22:10.858 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:22:10.856+0000 7f761ba88600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:22:10.874 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:22:10.874 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:22:10.875 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:22:10.875 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:22:10.875 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:22:10.876 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:22:10.876 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:22:10.877 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:10.877 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:22:10.877 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:10.877 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:22:10.878 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:22:10.878 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:22:10.878 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:22:10.878 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:22:10.879 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:22:10.879 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:22:10.879 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:22:10.879 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:22:10.880 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:10.880 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:10.880 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:10.880 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:22:10.881 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:22:10.881 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:22:10.881 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:22:10.881 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:22:10.882 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:22:10.882 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:22:10.882 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:22:10.882 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:22:10.883 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:10.883 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:10.883 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:10.883 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:22:10.884 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:22:10.884 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:22:10.884 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663014125.239690_900.1", 2022-09-12T20:22:10.884 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:22:10.885 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:22:10.885 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:22:10.885 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:22:05.239690Z", 2022-09-12T20:22:10.885 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:22:10.886 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:10.886 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:10.886 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:22:10.886 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:22:10.887 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:22:10.887 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:22:10.887 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:22:10.887 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:22:10.888 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:22:10.888 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:22:10.888 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:22:10.888 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:22:10.889 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:10.889 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:22:10.889 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:22:10.889 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:10.890 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:22:10.890 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:22:10.890 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:22:10.890 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:22:10.891 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:22:10Z" 2022-09-12T20:22:10.891 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:22:10.892 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663014125.239690_900.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:22:05.239690Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:22:10Z" } 2022-09-12T20:22:10.893 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663014125.239690_900.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:22:05.239690Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:22:10.893 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663014125.239690_900.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:22:10.894 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663014125.239690_900.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:22:10.894 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-12T20:22:10.895 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2022-09-12T20:22:10.895 INFO:teuthology.orchestra.run:waiting for 300 2022-09-12T20:22:10.896 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T20:22:10.894+0000 7f5305681700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 154754) UID: 0 2022-09-12T20:22:10.897 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T20:22:10.894+0000 7f53098ec600 -1 shutting down 2022-09-12T20:22:16.261 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2022-09-12T20:22:16.997 INFO:tasks.rgw.c2.client.0:Stopped 2022-09-12T20:22:16.998 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:16.999 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:16.999 DEBUG:boto:Method: HEAD 2022-09-12T20:22:17.000 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:17.000 DEBUG:boto:Data: 2022-09-12T20:22:17.001 DEBUG:boto:Headers: {} 2022-09-12T20:22:17.001 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.002 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.002 DEBUG:boto:Params: {} 2022-09-12T20:22:17.003 DEBUG:boto:Token: None 2022-09-12T20:22:17.004 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/ 2022-09-12T20:22:17.004 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0= 2022-09-12T20:22:17.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0=', 'Content-Length': '0'} 2022-09-12T20:22:17.008 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eaeb9489f51f50fe-00631f94f9-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.009 DEBUG:boto:path=/jdpiak-77/obj11 2022-09-12T20:22:17.009 DEBUG:boto:auth_path=/jdpiak-77/obj11 2022-09-12T20:22:17.010 DEBUG:boto:Method: PUT 2022-09-12T20:22:17.010 DEBUG:boto:Path: /jdpiak-77/obj11 2022-09-12T20:22:17.011 DEBUG:boto:Data: 2022-09-12T20:22:17.011 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:17.012 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.012 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.013 DEBUG:boto:Params: {} 2022-09-12T20:22:17.013 DEBUG:boto:Token: None 2022-09-12T20:22:17.014 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/obj11 2022-09-12T20:22:17.015 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9X/irkePlYe0SgseewOqNXaq91Y= 2022-09-12T20:22:17.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9X/irkePlYe0SgseewOqNXaq91Y='} 2022-09-12T20:22:17.025 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006486e1ac1479ef13-00631f94f9-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.027 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:17.029 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:17.031 DEBUG:boto:Method: HEAD 2022-09-12T20:22:17.032 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:17.032 DEBUG:boto:Data: 2022-09-12T20:22:17.034 DEBUG:boto:Headers: {} 2022-09-12T20:22:17.036 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.038 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.040 DEBUG:boto:Params: {} 2022-09-12T20:22:17.041 DEBUG:boto:Token: None 2022-09-12T20:22:17.043 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/ 2022-09-12T20:22:17.044 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0= 2022-09-12T20:22:17.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0=', 'Content-Length': '0'} 2022-09-12T20:22:17.048 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ebd864d3e83909e-00631f94f9-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.050 DEBUG:boto:path=/jdpiak-77/obj22 2022-09-12T20:22:17.051 DEBUG:boto:auth_path=/jdpiak-77/obj22 2022-09-12T20:22:17.052 DEBUG:boto:Method: PUT 2022-09-12T20:22:17.053 DEBUG:boto:Path: /jdpiak-77/obj22 2022-09-12T20:22:17.053 DEBUG:boto:Data: 2022-09-12T20:22:17.054 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:17.054 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.055 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.056 DEBUG:boto:Params: {} 2022-09-12T20:22:17.057 DEBUG:boto:Token: None 2022-09-12T20:22:17.058 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/obj22 2022-09-12T20:22:17.059 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:znsa7Q66mtbuQ4IF3ZM7UuaPHLU= 2022-09-12T20:22:17.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:znsa7Q66mtbuQ4IF3ZM7UuaPHLU='} 2022-09-12T20:22:17.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c46263c79d834818-00631f94f9-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.107 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:17.108 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:17.109 DEBUG:boto:Method: HEAD 2022-09-12T20:22:17.109 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:17.110 DEBUG:boto:Data: 2022-09-12T20:22:17.110 DEBUG:boto:Headers: {} 2022-09-12T20:22:17.111 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.111 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.112 DEBUG:boto:Params: {} 2022-09-12T20:22:17.112 DEBUG:boto:Token: None 2022-09-12T20:22:17.113 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/ 2022-09-12T20:22:17.114 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0= 2022-09-12T20:22:17.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0=', 'Content-Length': '0'} 2022-09-12T20:22:17.117 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2c17a69ed95094f-00631f94f9-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.118 DEBUG:boto:path=/jdpiak-77/obj33 2022-09-12T20:22:17.119 DEBUG:boto:auth_path=/jdpiak-77/obj33 2022-09-12T20:22:17.119 DEBUG:boto:Method: PUT 2022-09-12T20:22:17.120 DEBUG:boto:Path: /jdpiak-77/obj33 2022-09-12T20:22:17.120 DEBUG:boto:Data: 2022-09-12T20:22:17.121 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:17.121 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.122 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.122 DEBUG:boto:Params: {} 2022-09-12T20:22:17.123 DEBUG:boto:Token: None 2022-09-12T20:22:17.123 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/obj33 2022-09-12T20:22:17.124 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NQJC7/ey8Y5jnZTiX4FaZunXSXw= 2022-09-12T20:22:17.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NQJC7/ey8Y5jnZTiX4FaZunXSXw='} 2022-09-12T20:22:17.133 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000267a065410837b0b-00631f94f9-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.134 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:17.134 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:17.135 DEBUG:boto:Method: HEAD 2022-09-12T20:22:17.136 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:17.137 DEBUG:boto:Data: 2022-09-12T20:22:17.137 DEBUG:boto:Headers: {} 2022-09-12T20:22:17.138 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.138 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.139 DEBUG:boto:Params: {} 2022-09-12T20:22:17.140 DEBUG:boto:Token: None 2022-09-12T20:22:17.140 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/ 2022-09-12T20:22:17.141 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0= 2022-09-12T20:22:17.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0=', 'Content-Length': '0'} 2022-09-12T20:22:17.144 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083fd8e536721efeb-00631f94f9-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.145 DEBUG:boto:path=/jdpiak-77/obj44 2022-09-12T20:22:17.146 DEBUG:boto:auth_path=/jdpiak-77/obj44 2022-09-12T20:22:17.147 DEBUG:boto:Method: PUT 2022-09-12T20:22:17.148 DEBUG:boto:Path: /jdpiak-77/obj44 2022-09-12T20:22:17.148 DEBUG:boto:Data: 2022-09-12T20:22:17.149 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:17.149 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.150 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.150 DEBUG:boto:Params: {} 2022-09-12T20:22:17.151 DEBUG:boto:Token: None 2022-09-12T20:22:17.152 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/obj44 2022-09-12T20:22:17.152 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:awG+pr7Ft62O09VeCDSDPmqYyQo= 2022-09-12T20:22:17.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:awG+pr7Ft62O09VeCDSDPmqYyQo='} 2022-09-12T20:22:17.160 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087df314c7cc0bc86-00631f94f9-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.161 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:17.162 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:17.162 DEBUG:boto:Method: HEAD 2022-09-12T20:22:17.163 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:17.163 DEBUG:boto:Data: 2022-09-12T20:22:17.164 DEBUG:boto:Headers: {} 2022-09-12T20:22:17.164 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.165 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.166 DEBUG:boto:Params: {} 2022-09-12T20:22:17.167 DEBUG:boto:Token: None 2022-09-12T20:22:17.167 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/ 2022-09-12T20:22:17.168 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0= 2022-09-12T20:22:17.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0=', 'Content-Length': '0'} 2022-09-12T20:22:17.171 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b501b7253339e832-00631f94f9-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.172 DEBUG:boto:path=/jdpiak-77/obj55 2022-09-12T20:22:17.173 DEBUG:boto:auth_path=/jdpiak-77/obj55 2022-09-12T20:22:17.174 DEBUG:boto:Method: PUT 2022-09-12T20:22:17.175 DEBUG:boto:Path: /jdpiak-77/obj55 2022-09-12T20:22:17.176 DEBUG:boto:Data: 2022-09-12T20:22:17.177 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:17.178 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.179 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.180 DEBUG:boto:Params: {} 2022-09-12T20:22:17.180 DEBUG:boto:Token: None 2022-09-12T20:22:17.181 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/obj55 2022-09-12T20:22:17.182 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:pIWxBI+UucoCJXR897FSYTOXqBI= 2022-09-12T20:22:17.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:pIWxBI+UucoCJXR897FSYTOXqBI='} 2022-09-12T20:22:17.226 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000013d5b79d18bcc060-00631f94f9-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-77', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:22:17.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-77', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:22:17.229 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-77 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T20:22:17.289 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:17.289 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:17.338 INFO:teuthology.orchestra.run.smithi035.stdout:tenant: 2022-09-12T20:22:17.339 INFO:teuthology.orchestra.run.smithi035.stdout:bucket name: jdpiak-77 2022-09-12T20:22:17.343 INFO:teuthology.orchestra.run.smithi035.stdout:total entries: 5 2022-09-12T20:22:17.401 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:17.402 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:17.403 DEBUG:boto:Method: HEAD 2022-09-12T20:22:17.404 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:17.404 DEBUG:boto:Data: 2022-09-12T20:22:17.405 DEBUG:boto:Headers: {} 2022-09-12T20:22:17.405 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.406 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.406 DEBUG:boto:Params: {} 2022-09-12T20:22:17.407 DEBUG:boto:Token: None 2022-09-12T20:22:17.407 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/ 2022-09-12T20:22:17.408 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0= 2022-09-12T20:22:17.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0=', 'Content-Length': '0'} 2022-09-12T20:22:17.411 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e87efd243062671-00631f94f9-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.412 DEBUG:boto:path=/jdpiak-77/obj17 2022-09-12T20:22:17.412 DEBUG:boto:auth_path=/jdpiak-77/obj17 2022-09-12T20:22:17.413 DEBUG:boto:Method: PUT 2022-09-12T20:22:17.413 DEBUG:boto:Path: /jdpiak-77/obj17 2022-09-12T20:22:17.414 DEBUG:boto:Data: 2022-09-12T20:22:17.414 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:17.415 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.415 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.416 DEBUG:boto:Params: {} 2022-09-12T20:22:17.417 DEBUG:boto:Token: None 2022-09-12T20:22:17.417 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/obj17 2022-09-12T20:22:17.418 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:STutpJUFwenBqlIPY5WFwaVhakk= 2022-09-12T20:22:17.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:STutpJUFwenBqlIPY5WFwaVhakk='} 2022-09-12T20:22:17.431 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000676bc9810d46f49d-00631f94f9-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.432 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:17.433 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:17.434 DEBUG:boto:Method: HEAD 2022-09-12T20:22:17.434 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:17.435 DEBUG:boto:Data: 2022-09-12T20:22:17.436 DEBUG:boto:Headers: {} 2022-09-12T20:22:17.436 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.437 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.437 DEBUG:boto:Params: {} 2022-09-12T20:22:17.438 DEBUG:boto:Token: None 2022-09-12T20:22:17.438 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/ 2022-09-12T20:22:17.439 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0= 2022-09-12T20:22:17.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0=', 'Content-Length': '0'} 2022-09-12T20:22:17.442 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e6477978a3d584ac-00631f94f9-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.443 DEBUG:boto:path=/jdpiak-77/obj34 2022-09-12T20:22:17.443 DEBUG:boto:auth_path=/jdpiak-77/obj34 2022-09-12T20:22:17.444 DEBUG:boto:Method: PUT 2022-09-12T20:22:17.444 DEBUG:boto:Path: /jdpiak-77/obj34 2022-09-12T20:22:17.445 DEBUG:boto:Data: 2022-09-12T20:22:17.445 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:17.446 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.446 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.447 DEBUG:boto:Params: {} 2022-09-12T20:22:17.448 DEBUG:boto:Token: None 2022-09-12T20:22:17.448 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/obj34 2022-09-12T20:22:17.449 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:lUpSWxqH35KaG0o5EYl+bhY3PWk= 2022-09-12T20:22:17.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:lUpSWxqH35KaG0o5EYl+bhY3PWk='} 2022-09-12T20:22:17.494 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b96ae14c4a3210fe-00631f94f9-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.495 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:17.496 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:17.496 DEBUG:boto:Method: HEAD 2022-09-12T20:22:17.497 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:17.497 DEBUG:boto:Data: 2022-09-12T20:22:17.498 DEBUG:boto:Headers: {} 2022-09-12T20:22:17.498 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.499 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.499 DEBUG:boto:Params: {} 2022-09-12T20:22:17.500 DEBUG:boto:Token: None 2022-09-12T20:22:17.500 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/ 2022-09-12T20:22:17.501 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0= 2022-09-12T20:22:17.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0=', 'Content-Length': '0'} 2022-09-12T20:22:17.504 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce87dd48b98574ca-00631f94f9-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.505 DEBUG:boto:path=/jdpiak-77/obj51 2022-09-12T20:22:17.505 DEBUG:boto:auth_path=/jdpiak-77/obj51 2022-09-12T20:22:17.506 DEBUG:boto:Method: PUT 2022-09-12T20:22:17.507 DEBUG:boto:Path: /jdpiak-77/obj51 2022-09-12T20:22:17.507 DEBUG:boto:Data: 2022-09-12T20:22:17.507 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:17.508 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.508 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.509 DEBUG:boto:Params: {} 2022-09-12T20:22:17.510 DEBUG:boto:Token: None 2022-09-12T20:22:17.510 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/obj51 2022-09-12T20:22:17.511 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kKwYgRM497o36ifEf95riws9Zgo= 2022-09-12T20:22:17.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kKwYgRM497o36ifEf95riws9Zgo='} 2022-09-12T20:22:17.520 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4eb563d192be194-00631f94f9-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.520 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:17.521 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:17.521 DEBUG:boto:Method: HEAD 2022-09-12T20:22:17.522 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:17.522 DEBUG:boto:Data: 2022-09-12T20:22:17.523 DEBUG:boto:Headers: {} 2022-09-12T20:22:17.523 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.524 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.524 DEBUG:boto:Params: {} 2022-09-12T20:22:17.525 DEBUG:boto:Token: None 2022-09-12T20:22:17.526 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/ 2022-09-12T20:22:17.526 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0= 2022-09-12T20:22:17.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0=', 'Content-Length': '0'} 2022-09-12T20:22:17.530 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005715b0e3c5cb2dd4-00631f94f9-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.530 DEBUG:boto:path=/jdpiak-77/obj68 2022-09-12T20:22:17.531 DEBUG:boto:auth_path=/jdpiak-77/obj68 2022-09-12T20:22:17.531 DEBUG:boto:Method: PUT 2022-09-12T20:22:17.532 DEBUG:boto:Path: /jdpiak-77/obj68 2022-09-12T20:22:17.532 DEBUG:boto:Data: 2022-09-12T20:22:17.533 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:17.533 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.534 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.534 DEBUG:boto:Params: {} 2022-09-12T20:22:17.535 DEBUG:boto:Token: None 2022-09-12T20:22:17.535 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/obj68 2022-09-12T20:22:17.536 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:vKXHRqpFAlxBq2Q4Z/MxiqunO64= 2022-09-12T20:22:17.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:vKXHRqpFAlxBq2Q4Z/MxiqunO64='} 2022-09-12T20:22:17.543 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ded32183f331b134-00631f94f9-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.544 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:17.545 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:17.545 DEBUG:boto:Method: HEAD 2022-09-12T20:22:17.546 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:17.546 DEBUG:boto:Data: 2022-09-12T20:22:17.547 DEBUG:boto:Headers: {} 2022-09-12T20:22:17.547 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.548 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.548 DEBUG:boto:Params: {} 2022-09-12T20:22:17.549 DEBUG:boto:Token: None 2022-09-12T20:22:17.549 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/ 2022-09-12T20:22:17.550 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0= 2022-09-12T20:22:17.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0=', 'Content-Length': '0'} 2022-09-12T20:22:17.553 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fad685c7487b9599-00631f94f9-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.554 DEBUG:boto:path=/jdpiak-77/obj85 2022-09-12T20:22:17.554 DEBUG:boto:auth_path=/jdpiak-77/obj85 2022-09-12T20:22:17.555 DEBUG:boto:Method: PUT 2022-09-12T20:22:17.555 DEBUG:boto:Path: /jdpiak-77/obj85 2022-09-12T20:22:17.556 DEBUG:boto:Data: 2022-09-12T20:22:17.556 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:17.557 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.557 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.558 DEBUG:boto:Params: {} 2022-09-12T20:22:17.558 DEBUG:boto:Token: None 2022-09-12T20:22:17.559 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/obj85 2022-09-12T20:22:17.560 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:G8FkHjW+rsFSt8P8W9iwtYVPIGA= 2022-09-12T20:22:17.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:G8FkHjW+rsFSt8P8W9iwtYVPIGA='} 2022-09-12T20:22:17.602 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000089e1a047503f027e-00631f94f9-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-77', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:22:17.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-77', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:22:17.604 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-77 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T20:22:17.660 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:17.660 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:17.719 INFO:teuthology.orchestra.run.smithi035.stdout:tenant: 2022-09-12T20:22:17.719 INFO:teuthology.orchestra.run.smithi035.stdout:bucket name: jdpiak-77 2022-09-12T20:22:17.727 INFO:teuthology.orchestra.run.smithi035.stdout:total entries: 10 2022-09-12T20:22:17.775 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:17.776 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:17.776 DEBUG:boto:Method: HEAD 2022-09-12T20:22:17.777 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:17.777 DEBUG:boto:Data: 2022-09-12T20:22:17.778 DEBUG:boto:Headers: {} 2022-09-12T20:22:17.778 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.779 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.779 DEBUG:boto:Params: {} 2022-09-12T20:22:17.780 DEBUG:boto:Token: None 2022-09-12T20:22:17.781 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/ 2022-09-12T20:22:17.781 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0= 2022-09-12T20:22:17.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0=', 'Content-Length': '0'} 2022-09-12T20:22:17.785 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b738acdbf6d954b8-00631f94f9-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.786 DEBUG:boto:path=/jdpiak-77/obj19 2022-09-12T20:22:17.786 DEBUG:boto:auth_path=/jdpiak-77/obj19 2022-09-12T20:22:17.787 DEBUG:boto:Method: PUT 2022-09-12T20:22:17.787 DEBUG:boto:Path: /jdpiak-77/obj19 2022-09-12T20:22:17.788 DEBUG:boto:Data: 2022-09-12T20:22:17.788 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:17.789 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.789 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.790 DEBUG:boto:Params: {} 2022-09-12T20:22:17.790 DEBUG:boto:Token: None 2022-09-12T20:22:17.791 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/obj19 2022-09-12T20:22:17.792 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:FqsgJPme0pfzZ9V0dlthdzICHIE= 2022-09-12T20:22:17.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:FqsgJPme0pfzZ9V0dlthdzICHIE='} 2022-09-12T20:22:17.798 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008cd7623d6d8a3a49-00631f94f9-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.799 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:17.799 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:17.800 DEBUG:boto:Method: HEAD 2022-09-12T20:22:17.800 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:17.801 DEBUG:boto:Data: 2022-09-12T20:22:17.801 DEBUG:boto:Headers: {} 2022-09-12T20:22:17.802 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.802 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.803 DEBUG:boto:Params: {} 2022-09-12T20:22:17.803 DEBUG:boto:Token: None 2022-09-12T20:22:17.804 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/ 2022-09-12T20:22:17.804 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0= 2022-09-12T20:22:17.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0=', 'Content-Length': '0'} 2022-09-12T20:22:17.808 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a774cc9d2c7a3556-00631f94f9-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.808 DEBUG:boto:path=/jdpiak-77/obj38 2022-09-12T20:22:17.809 DEBUG:boto:auth_path=/jdpiak-77/obj38 2022-09-12T20:22:17.809 DEBUG:boto:Method: PUT 2022-09-12T20:22:17.810 DEBUG:boto:Path: /jdpiak-77/obj38 2022-09-12T20:22:17.811 DEBUG:boto:Data: 2022-09-12T20:22:17.811 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:17.812 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.812 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.812 DEBUG:boto:Params: {} 2022-09-12T20:22:17.813 DEBUG:boto:Token: None 2022-09-12T20:22:17.814 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/obj38 2022-09-12T20:22:17.814 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:cw4mYT6xU0wnVoiRm6M6e3zx7xI= 2022-09-12T20:22:17.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:cw4mYT6xU0wnVoiRm6M6e3zx7xI='} 2022-09-12T20:22:17.820 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000028157a0f13aba1bf-00631f94f9-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.821 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:17.822 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:17.822 DEBUG:boto:Method: HEAD 2022-09-12T20:22:17.823 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:17.823 DEBUG:boto:Data: 2022-09-12T20:22:17.824 DEBUG:boto:Headers: {} 2022-09-12T20:22:17.824 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.825 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.825 DEBUG:boto:Params: {} 2022-09-12T20:22:17.826 DEBUG:boto:Token: None 2022-09-12T20:22:17.826 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/ 2022-09-12T20:22:17.827 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0= 2022-09-12T20:22:17.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0=', 'Content-Length': '0'} 2022-09-12T20:22:17.830 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081427bc0628d55a0-00631f94f9-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.831 DEBUG:boto:path=/jdpiak-77/obj57 2022-09-12T20:22:17.831 DEBUG:boto:auth_path=/jdpiak-77/obj57 2022-09-12T20:22:17.832 DEBUG:boto:Method: PUT 2022-09-12T20:22:17.832 DEBUG:boto:Path: /jdpiak-77/obj57 2022-09-12T20:22:17.833 DEBUG:boto:Data: 2022-09-12T20:22:17.833 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:17.834 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.834 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.835 DEBUG:boto:Params: {} 2022-09-12T20:22:17.835 DEBUG:boto:Token: None 2022-09-12T20:22:17.836 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/obj57 2022-09-12T20:22:17.837 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:YhKQI7dfmdrS1A1Nx3qaRq4EJYM= 2022-09-12T20:22:17.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:YhKQI7dfmdrS1A1Nx3qaRq4EJYM='} 2022-09-12T20:22:17.882 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008488e0bfda954957-00631f94f9-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.883 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:17.884 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:17.884 DEBUG:boto:Method: HEAD 2022-09-12T20:22:17.885 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:17.885 DEBUG:boto:Data: 2022-09-12T20:22:17.886 DEBUG:boto:Headers: {} 2022-09-12T20:22:17.886 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.887 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.887 DEBUG:boto:Params: {} 2022-09-12T20:22:17.888 DEBUG:boto:Token: None 2022-09-12T20:22:17.888 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/ 2022-09-12T20:22:17.889 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0= 2022-09-12T20:22:17.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0=', 'Content-Length': '0'} 2022-09-12T20:22:17.892 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad889987900f76db-00631f94f9-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.893 DEBUG:boto:path=/jdpiak-77/obj76 2022-09-12T20:22:17.893 DEBUG:boto:auth_path=/jdpiak-77/obj76 2022-09-12T20:22:17.894 DEBUG:boto:Method: PUT 2022-09-12T20:22:17.895 DEBUG:boto:Path: /jdpiak-77/obj76 2022-09-12T20:22:17.895 DEBUG:boto:Data: 2022-09-12T20:22:17.896 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:17.896 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.896 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.897 DEBUG:boto:Params: {} 2022-09-12T20:22:17.898 DEBUG:boto:Token: None 2022-09-12T20:22:17.898 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/obj76 2022-09-12T20:22:17.899 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:NaGuMqCAzC03MWaP3aK/wD/b8so= 2022-09-12T20:22:17.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:NaGuMqCAzC03MWaP3aK/wD/b8so='} 2022-09-12T20:22:17.909 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000adf8695067b72df-00631f94f9-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.910 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:17.910 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:17.911 DEBUG:boto:Method: HEAD 2022-09-12T20:22:17.911 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:17.912 DEBUG:boto:Data: 2022-09-12T20:22:17.912 DEBUG:boto:Headers: {} 2022-09-12T20:22:17.913 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.913 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.914 DEBUG:boto:Params: {} 2022-09-12T20:22:17.914 DEBUG:boto:Token: None 2022-09-12T20:22:17.915 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/ 2022-09-12T20:22:17.915 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0= 2022-09-12T20:22:17.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:xC3AOrL26h4E2mWonlq5aCHFxm0=', 'Content-Length': '0'} 2022-09-12T20:22:17.919 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af0e24c356e96bc3-00631f94f9-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.920 DEBUG:boto:path=/jdpiak-77/obj95 2022-09-12T20:22:17.920 DEBUG:boto:auth_path=/jdpiak-77/obj95 2022-09-12T20:22:17.921 DEBUG:boto:Method: PUT 2022-09-12T20:22:17.921 DEBUG:boto:Path: /jdpiak-77/obj95 2022-09-12T20:22:17.922 DEBUG:boto:Data: 2022-09-12T20:22:17.922 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:17.923 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:17.924 DEBUG:boto:Port: 8000 2022-09-12T20:22:17.924 DEBUG:boto:Params: {} 2022-09-12T20:22:17.925 DEBUG:boto:Token: None 2022-09-12T20:22:17.925 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:17 GMT /jdpiak-77/obj95 2022-09-12T20:22:17.926 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:X87b6UsDueieUozTqf9W5QNnLAM= 2022-09-12T20:22:17.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:17 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:X87b6UsDueieUozTqf9W5QNnLAM='} 2022-09-12T20:22:17.970 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d3bcca3cd02dc1b-00631f94f9-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:17.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-77', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:22:17.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-77', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:22:17.972 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-77 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T20:22:18.026 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:18.027 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:18.097 INFO:teuthology.orchestra.run.smithi035.stdout:tenant: 2022-09-12T20:22:18.098 INFO:teuthology.orchestra.run.smithi035.stdout:bucket name: jdpiak-77 2022-09-12T20:22:18.106 INFO:teuthology.orchestra.run.smithi035.stdout:total entries: 15 2022-09-12T20:22:18.168 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:18.168 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:18.169 DEBUG:boto:Method: HEAD 2022-09-12T20:22:18.169 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:18.170 DEBUG:boto:Data: 2022-09-12T20:22:18.170 DEBUG:boto:Headers: {} 2022-09-12T20:22:18.171 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:18.171 DEBUG:boto:Port: 8000 2022-09-12T20:22:18.172 DEBUG:boto:Params: {} 2022-09-12T20:22:18.172 DEBUG:boto:Token: None 2022-09-12T20:22:18.173 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:18 GMT /jdpiak-77/ 2022-09-12T20:22:18.174 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7L7Vt87FsrqyoCy0RzfCZotMPU4= 2022-09-12T20:22:18.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7L7Vt87FsrqyoCy0RzfCZotMPU4=', 'Content-Length': '0'} 2022-09-12T20:22:18.177 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000181e0247d3d7db33-00631f94fa-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:18.178 DEBUG:boto:path=/jdpiak-77/obj23 2022-09-12T20:22:18.178 DEBUG:boto:auth_path=/jdpiak-77/obj23 2022-09-12T20:22:18.179 DEBUG:boto:Method: PUT 2022-09-12T20:22:18.179 DEBUG:boto:Path: /jdpiak-77/obj23 2022-09-12T20:22:18.180 DEBUG:boto:Data: 2022-09-12T20:22:18.180 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:18.181 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:18.181 DEBUG:boto:Port: 8000 2022-09-12T20:22:18.182 DEBUG:boto:Params: {} 2022-09-12T20:22:18.182 DEBUG:boto:Token: None 2022-09-12T20:22:18.183 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:18 GMT /jdpiak-77/obj23 2022-09-12T20:22:18.184 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:o5dP3PfLpCqAJcVd7MElqS57NAM= 2022-09-12T20:22:18.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:o5dP3PfLpCqAJcVd7MElqS57NAM='} 2022-09-12T20:22:18.191 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c68811c1046e40eb-00631f94fa-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:18.192 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:18.193 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:18.193 DEBUG:boto:Method: HEAD 2022-09-12T20:22:18.194 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:18.194 DEBUG:boto:Data: 2022-09-12T20:22:18.195 DEBUG:boto:Headers: {} 2022-09-12T20:22:18.195 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:18.195 DEBUG:boto:Port: 8000 2022-09-12T20:22:18.197 DEBUG:boto:Params: {} 2022-09-12T20:22:18.198 DEBUG:boto:Token: None 2022-09-12T20:22:18.198 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:18 GMT /jdpiak-77/ 2022-09-12T20:22:18.199 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7L7Vt87FsrqyoCy0RzfCZotMPU4= 2022-09-12T20:22:18.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7L7Vt87FsrqyoCy0RzfCZotMPU4=', 'Content-Length': '0'} 2022-09-12T20:22:18.203 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000071ca2b01d874318a-00631f94fa-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:18.203 DEBUG:boto:path=/jdpiak-77/obj46 2022-09-12T20:22:18.204 DEBUG:boto:auth_path=/jdpiak-77/obj46 2022-09-12T20:22:18.204 DEBUG:boto:Method: PUT 2022-09-12T20:22:18.205 DEBUG:boto:Path: /jdpiak-77/obj46 2022-09-12T20:22:18.205 DEBUG:boto:Data: 2022-09-12T20:22:18.206 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:18.207 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:18.207 DEBUG:boto:Port: 8000 2022-09-12T20:22:18.208 DEBUG:boto:Params: {} 2022-09-12T20:22:18.208 DEBUG:boto:Token: None 2022-09-12T20:22:18.209 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:18 GMT /jdpiak-77/obj46 2022-09-12T20:22:18.209 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:Zi4YqDjLJ9R6TWQz+sRwwdzN8tQ= 2022-09-12T20:22:18.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:Zi4YqDjLJ9R6TWQz+sRwwdzN8tQ='} 2022-09-12T20:22:18.216 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a2190ec9ade3c7f-00631f94fa-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:18.217 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:18.218 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:18.218 DEBUG:boto:Method: HEAD 2022-09-12T20:22:18.219 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:18.219 DEBUG:boto:Data: 2022-09-12T20:22:18.220 DEBUG:boto:Headers: {} 2022-09-12T20:22:18.220 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:18.221 DEBUG:boto:Port: 8000 2022-09-12T20:22:18.221 DEBUG:boto:Params: {} 2022-09-12T20:22:18.222 DEBUG:boto:Token: None 2022-09-12T20:22:18.223 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:18 GMT /jdpiak-77/ 2022-09-12T20:22:18.223 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7L7Vt87FsrqyoCy0RzfCZotMPU4= 2022-09-12T20:22:18.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7L7Vt87FsrqyoCy0RzfCZotMPU4=', 'Content-Length': '0'} 2022-09-12T20:22:18.227 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4e6fab6dd129296-00631f94fa-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:18.228 DEBUG:boto:path=/jdpiak-77/obj69 2022-09-12T20:22:18.228 DEBUG:boto:auth_path=/jdpiak-77/obj69 2022-09-12T20:22:18.229 DEBUG:boto:Method: PUT 2022-09-12T20:22:18.229 DEBUG:boto:Path: /jdpiak-77/obj69 2022-09-12T20:22:18.230 DEBUG:boto:Data: 2022-09-12T20:22:18.231 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:18.231 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:18.232 DEBUG:boto:Port: 8000 2022-09-12T20:22:18.232 DEBUG:boto:Params: {} 2022-09-12T20:22:18.234 DEBUG:boto:Token: None 2022-09-12T20:22:18.237 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:18 GMT /jdpiak-77/obj69 2022-09-12T20:22:18.238 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:eT8TuXFf9ehERzebJZuV9fuP9OE= 2022-09-12T20:22:18.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:eT8TuXFf9ehERzebJZuV9fuP9OE='} 2022-09-12T20:22:18.282 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed83533ddb928faf-00631f94fa-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:18.283 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:18.284 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:18.284 DEBUG:boto:Method: HEAD 2022-09-12T20:22:18.285 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:18.285 DEBUG:boto:Data: 2022-09-12T20:22:18.286 DEBUG:boto:Headers: {} 2022-09-12T20:22:18.286 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:18.287 DEBUG:boto:Port: 8000 2022-09-12T20:22:18.287 DEBUG:boto:Params: {} 2022-09-12T20:22:18.288 DEBUG:boto:Token: None 2022-09-12T20:22:18.288 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:18 GMT /jdpiak-77/ 2022-09-12T20:22:18.289 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7L7Vt87FsrqyoCy0RzfCZotMPU4= 2022-09-12T20:22:18.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7L7Vt87FsrqyoCy0RzfCZotMPU4=', 'Content-Length': '0'} 2022-09-12T20:22:18.293 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073de81d75ac48de3-00631f94fa-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:18.294 DEBUG:boto:path=/jdpiak-77/obj92 2022-09-12T20:22:18.295 DEBUG:boto:auth_path=/jdpiak-77/obj92 2022-09-12T20:22:18.295 DEBUG:boto:Method: PUT 2022-09-12T20:22:18.296 DEBUG:boto:Path: /jdpiak-77/obj92 2022-09-12T20:22:18.296 DEBUG:boto:Data: 2022-09-12T20:22:18.297 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:18.297 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:18.298 DEBUG:boto:Port: 8000 2022-09-12T20:22:18.298 DEBUG:boto:Params: {} 2022-09-12T20:22:18.299 DEBUG:boto:Token: None 2022-09-12T20:22:18.299 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:18 GMT /jdpiak-77/obj92 2022-09-12T20:22:18.300 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rs483hw44OhekzTK1f14FcG0kz8= 2022-09-12T20:22:18.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rs483hw44OhekzTK1f14FcG0kz8='} 2022-09-12T20:22:18.307 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a401a1dcbe4f54a7-00631f94fa-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:18.308 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:18.308 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:18.309 DEBUG:boto:Method: HEAD 2022-09-12T20:22:18.309 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:18.310 DEBUG:boto:Data: 2022-09-12T20:22:18.310 DEBUG:boto:Headers: {} 2022-09-12T20:22:18.311 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:18.311 DEBUG:boto:Port: 8000 2022-09-12T20:22:18.312 DEBUG:boto:Params: {} 2022-09-12T20:22:18.312 DEBUG:boto:Token: None 2022-09-12T20:22:18.313 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:18 GMT /jdpiak-77/ 2022-09-12T20:22:18.314 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7L7Vt87FsrqyoCy0RzfCZotMPU4= 2022-09-12T20:22:18.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7L7Vt87FsrqyoCy0RzfCZotMPU4=', 'Content-Length': '0'} 2022-09-12T20:22:18.318 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6728b501a69a194-00631f94fa-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:18.318 DEBUG:boto:path=/jdpiak-77/obj115 2022-09-12T20:22:18.319 DEBUG:boto:auth_path=/jdpiak-77/obj115 2022-09-12T20:22:18.320 DEBUG:boto:Method: PUT 2022-09-12T20:22:18.320 DEBUG:boto:Path: /jdpiak-77/obj115 2022-09-12T20:22:18.321 DEBUG:boto:Data: 2022-09-12T20:22:18.321 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:18.322 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:18.322 DEBUG:boto:Port: 8000 2022-09-12T20:22:18.323 DEBUG:boto:Params: {} 2022-09-12T20:22:18.323 DEBUG:boto:Token: None 2022-09-12T20:22:18.324 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:18 GMT /jdpiak-77/obj115 2022-09-12T20:22:18.325 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:GRGicNM3OV6Cfrs35t+r8XluvyQ= 2022-09-12T20:22:18.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:GRGicNM3OV6Cfrs35t+r8XluvyQ='} 2022-09-12T20:22:18.334 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000053681f01e3f858ba-00631f94fa-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:18.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-77', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:22:18.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-77', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:22:18.336 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-77 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T20:22:18.391 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:18.392 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:18.452 INFO:teuthology.orchestra.run.smithi035.stdout:tenant: 2022-09-12T20:22:18.453 INFO:teuthology.orchestra.run.smithi035.stdout:bucket name: jdpiak-77 2022-09-12T20:22:18.462 INFO:teuthology.orchestra.run.smithi035.stdout:total entries: 20 2022-09-12T20:22:18.548 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:18.549 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:18.550 DEBUG:boto:Method: HEAD 2022-09-12T20:22:18.550 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:18.551 DEBUG:boto:Data: 2022-09-12T20:22:18.551 DEBUG:boto:Headers: {} 2022-09-12T20:22:18.552 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:18.552 DEBUG:boto:Port: 8000 2022-09-12T20:22:18.553 DEBUG:boto:Params: {} 2022-09-12T20:22:18.553 DEBUG:boto:Token: None 2022-09-12T20:22:18.554 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:18 GMT /jdpiak-77/ 2022-09-12T20:22:18.554 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7L7Vt87FsrqyoCy0RzfCZotMPU4= 2022-09-12T20:22:18.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7L7Vt87FsrqyoCy0RzfCZotMPU4=', 'Content-Length': '0'} 2022-09-12T20:22:18.558 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d45baac94ae7e50f-00631f94fa-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:18.559 DEBUG:boto:path=/jdpiak-77/obj29 2022-09-12T20:22:18.560 DEBUG:boto:auth_path=/jdpiak-77/obj29 2022-09-12T20:22:18.560 DEBUG:boto:Method: PUT 2022-09-12T20:22:18.561 DEBUG:boto:Path: /jdpiak-77/obj29 2022-09-12T20:22:18.562 DEBUG:boto:Data: 2022-09-12T20:22:18.562 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:18.563 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:18.563 DEBUG:boto:Port: 8000 2022-09-12T20:22:18.564 DEBUG:boto:Params: {} 2022-09-12T20:22:18.564 DEBUG:boto:Token: None 2022-09-12T20:22:18.565 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:18 GMT /jdpiak-77/obj29 2022-09-12T20:22:18.565 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:IYsrpKLvq5AmG0MskF8MM/vBbOQ= 2022-09-12T20:22:18.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:IYsrpKLvq5AmG0MskF8MM/vBbOQ='} 2022-09-12T20:22:18.610 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f17fb7a529e84e8b-00631f94fa-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:18.611 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:18.611 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:18.612 DEBUG:boto:Method: HEAD 2022-09-12T20:22:18.612 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:18.613 DEBUG:boto:Data: 2022-09-12T20:22:18.613 DEBUG:boto:Headers: {} 2022-09-12T20:22:18.614 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:18.615 DEBUG:boto:Port: 8000 2022-09-12T20:22:18.615 DEBUG:boto:Params: {} 2022-09-12T20:22:18.616 DEBUG:boto:Token: None 2022-09-12T20:22:18.616 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:18 GMT /jdpiak-77/ 2022-09-12T20:22:18.617 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7L7Vt87FsrqyoCy0RzfCZotMPU4= 2022-09-12T20:22:18.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7L7Vt87FsrqyoCy0RzfCZotMPU4=', 'Content-Length': '0'} 2022-09-12T20:22:18.623 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e8a2c56e8a1a6b6-00631f94fa-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:18.624 DEBUG:boto:path=/jdpiak-77/obj58 2022-09-12T20:22:18.624 DEBUG:boto:auth_path=/jdpiak-77/obj58 2022-09-12T20:22:18.625 DEBUG:boto:Method: PUT 2022-09-12T20:22:18.625 DEBUG:boto:Path: /jdpiak-77/obj58 2022-09-12T20:22:18.626 DEBUG:boto:Data: 2022-09-12T20:22:18.626 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:18.627 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:18.627 DEBUG:boto:Port: 8000 2022-09-12T20:22:18.628 DEBUG:boto:Params: {} 2022-09-12T20:22:18.629 DEBUG:boto:Token: None 2022-09-12T20:22:18.629 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:18 GMT /jdpiak-77/obj58 2022-09-12T20:22:18.630 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:wfWOxoAS6SZnAPxZYpJoXK7PRSw= 2022-09-12T20:22:18.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:wfWOxoAS6SZnAPxZYpJoXK7PRSw='} 2022-09-12T20:22:18.636 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af4af1e88576a7b3-00631f94fa-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:18.637 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:18.638 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:18.638 DEBUG:boto:Method: HEAD 2022-09-12T20:22:18.639 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:18.639 DEBUG:boto:Data: 2022-09-12T20:22:18.640 DEBUG:boto:Headers: {} 2022-09-12T20:22:18.640 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:18.641 DEBUG:boto:Port: 8000 2022-09-12T20:22:18.641 DEBUG:boto:Params: {} 2022-09-12T20:22:18.642 DEBUG:boto:Token: None 2022-09-12T20:22:18.642 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:18 GMT /jdpiak-77/ 2022-09-12T20:22:18.643 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7L7Vt87FsrqyoCy0RzfCZotMPU4= 2022-09-12T20:22:18.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7L7Vt87FsrqyoCy0RzfCZotMPU4=', 'Content-Length': '0'} 2022-09-12T20:22:18.647 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000231ac177d6bcba58-00631f94fa-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:18.648 DEBUG:boto:path=/jdpiak-77/obj87 2022-09-12T20:22:18.648 DEBUG:boto:auth_path=/jdpiak-77/obj87 2022-09-12T20:22:18.649 DEBUG:boto:Method: PUT 2022-09-12T20:22:18.649 DEBUG:boto:Path: /jdpiak-77/obj87 2022-09-12T20:22:18.650 DEBUG:boto:Data: 2022-09-12T20:22:18.650 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:18.651 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:18.651 DEBUG:boto:Port: 8000 2022-09-12T20:22:18.652 DEBUG:boto:Params: {} 2022-09-12T20:22:18.652 DEBUG:boto:Token: None 2022-09-12T20:22:18.653 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:18 GMT /jdpiak-77/obj87 2022-09-12T20:22:18.653 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ruw/mBz0a8Qjjx02a5bv+zs90qE= 2022-09-12T20:22:18.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ruw/mBz0a8Qjjx02a5bv+zs90qE='} 2022-09-12T20:22:18.698 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c3ac656fc7c9b39-00631f94fa-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:18.699 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:18.700 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:18.700 DEBUG:boto:Method: HEAD 2022-09-12T20:22:18.701 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:18.702 DEBUG:boto:Data: 2022-09-12T20:22:18.702 DEBUG:boto:Headers: {} 2022-09-12T20:22:18.703 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:18.703 DEBUG:boto:Port: 8000 2022-09-12T20:22:18.704 DEBUG:boto:Params: {} 2022-09-12T20:22:18.704 DEBUG:boto:Token: None 2022-09-12T20:22:18.705 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:18 GMT /jdpiak-77/ 2022-09-12T20:22:18.705 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7L7Vt87FsrqyoCy0RzfCZotMPU4= 2022-09-12T20:22:18.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7L7Vt87FsrqyoCy0RzfCZotMPU4=', 'Content-Length': '0'} 2022-09-12T20:22:18.709 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2004e3afc21816a-00631f94fa-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:18.710 DEBUG:boto:path=/jdpiak-77/obj116 2022-09-12T20:22:18.711 DEBUG:boto:auth_path=/jdpiak-77/obj116 2022-09-12T20:22:18.711 DEBUG:boto:Method: PUT 2022-09-12T20:22:18.712 DEBUG:boto:Path: /jdpiak-77/obj116 2022-09-12T20:22:18.712 DEBUG:boto:Data: 2022-09-12T20:22:18.712 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:18.713 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:18.713 DEBUG:boto:Port: 8000 2022-09-12T20:22:18.714 DEBUG:boto:Params: {} 2022-09-12T20:22:18.714 DEBUG:boto:Token: None 2022-09-12T20:22:18.715 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:18 GMT /jdpiak-77/obj116 2022-09-12T20:22:18.716 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:ZgQBVnaZl65NVmdlWxRL04uBkeM= 2022-09-12T20:22:18.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:ZgQBVnaZl65NVmdlWxRL04uBkeM='} 2022-09-12T20:22:18.722 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006e5970412a4d6f38-00631f94fa-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:18.723 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:18.723 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:18.724 DEBUG:boto:Method: HEAD 2022-09-12T20:22:18.725 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:18.725 DEBUG:boto:Data: 2022-09-12T20:22:18.725 DEBUG:boto:Headers: {} 2022-09-12T20:22:18.726 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:18.726 DEBUG:boto:Port: 8000 2022-09-12T20:22:18.727 DEBUG:boto:Params: {} 2022-09-12T20:22:18.727 DEBUG:boto:Token: None 2022-09-12T20:22:18.728 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:18 GMT /jdpiak-77/ 2022-09-12T20:22:18.729 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7L7Vt87FsrqyoCy0RzfCZotMPU4= 2022-09-12T20:22:18.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7L7Vt87FsrqyoCy0RzfCZotMPU4=', 'Content-Length': '0'} 2022-09-12T20:22:18.733 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050716dd3cbfe9464-00631f94fa-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:18.733 DEBUG:boto:path=/jdpiak-77/obj145 2022-09-12T20:22:18.734 DEBUG:boto:auth_path=/jdpiak-77/obj145 2022-09-12T20:22:18.734 DEBUG:boto:Method: PUT 2022-09-12T20:22:18.735 DEBUG:boto:Path: /jdpiak-77/obj145 2022-09-12T20:22:18.735 DEBUG:boto:Data: 2022-09-12T20:22:18.736 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:18.736 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:18.737 DEBUG:boto:Port: 8000 2022-09-12T20:22:18.737 DEBUG:boto:Params: {} 2022-09-12T20:22:18.738 DEBUG:boto:Token: None 2022-09-12T20:22:18.739 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:18 GMT /jdpiak-77/obj145 2022-09-12T20:22:18.739 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:mnPBK3R7bQkp9L3VcnbsNljDzTk= 2022-09-12T20:22:18.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:mnPBK3R7bQkp9L3VcnbsNljDzTk='} 2022-09-12T20:22:18.746 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008ac9b9a95853c166-00631f94fa-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:18.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-77', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:22:18.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-77', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:22:18.748 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-77 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T20:22:18.803 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:18.804 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:18.858 INFO:teuthology.orchestra.run.smithi035.stdout:tenant: 2022-09-12T20:22:18.858 INFO:teuthology.orchestra.run.smithi035.stdout:bucket name: jdpiak-77 2022-09-12T20:22:18.871 INFO:teuthology.orchestra.run.smithi035.stdout:total entries: 25 2022-09-12T20:22:18.978 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:18.978 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:18.979 DEBUG:boto:Method: HEAD 2022-09-12T20:22:18.979 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:18.980 DEBUG:boto:Data: 2022-09-12T20:22:18.980 DEBUG:boto:Headers: {} 2022-09-12T20:22:18.981 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:18.981 DEBUG:boto:Port: 8000 2022-09-12T20:22:18.982 DEBUG:boto:Params: {} 2022-09-12T20:22:18.982 DEBUG:boto:Token: None 2022-09-12T20:22:18.983 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:18 GMT /jdpiak-77/ 2022-09-12T20:22:18.984 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:7L7Vt87FsrqyoCy0RzfCZotMPU4= 2022-09-12T20:22:18.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:7L7Vt87FsrqyoCy0RzfCZotMPU4=', 'Content-Length': '0'} 2022-09-12T20:22:18.988 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ebe6d0114d40a97-00631f94fa-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:18.988 DEBUG:boto:path=/jdpiak-77/obj31 2022-09-12T20:22:18.989 DEBUG:boto:auth_path=/jdpiak-77/obj31 2022-09-12T20:22:18.990 DEBUG:boto:Method: PUT 2022-09-12T20:22:18.990 DEBUG:boto:Path: /jdpiak-77/obj31 2022-09-12T20:22:18.991 DEBUG:boto:Data: 2022-09-12T20:22:18.991 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:18.992 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:18.992 DEBUG:boto:Port: 8000 2022-09-12T20:22:18.993 DEBUG:boto:Params: {} 2022-09-12T20:22:18.993 DEBUG:boto:Token: None 2022-09-12T20:22:18.994 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:18 GMT /jdpiak-77/obj31 2022-09-12T20:22:18.994 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1FGQw0ttaq1OXBFmLIKwOP0jP+U= 2022-09-12T20:22:18.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:18 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1FGQw0ttaq1OXBFmLIKwOP0jP+U='} 2022-09-12T20:22:19.038 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005cb0bd17bbdb569c-00631f94fa-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:19.039 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:19.040 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:19.040 DEBUG:boto:Method: HEAD 2022-09-12T20:22:19.041 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:19.041 DEBUG:boto:Data: 2022-09-12T20:22:19.042 DEBUG:boto:Headers: {} 2022-09-12T20:22:19.042 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:19.043 DEBUG:boto:Port: 8000 2022-09-12T20:22:19.043 DEBUG:boto:Params: {} 2022-09-12T20:22:19.044 DEBUG:boto:Token: None 2022-09-12T20:22:19.044 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:19 GMT /jdpiak-77/ 2022-09-12T20:22:19.045 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3XVUPKDOIXtZ5UNFKK/jG2lxJ0k= 2022-09-12T20:22:19.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3XVUPKDOIXtZ5UNFKK/jG2lxJ0k=', 'Content-Length': '0'} 2022-09-12T20:22:19.049 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8714480a6cf4f34-00631f94fb-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:19.049 DEBUG:boto:path=/jdpiak-77/obj62 2022-09-12T20:22:19.050 DEBUG:boto:auth_path=/jdpiak-77/obj62 2022-09-12T20:22:19.050 DEBUG:boto:Method: PUT 2022-09-12T20:22:19.051 DEBUG:boto:Path: /jdpiak-77/obj62 2022-09-12T20:22:19.051 DEBUG:boto:Data: 2022-09-12T20:22:19.052 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:19.052 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:19.053 DEBUG:boto:Port: 8000 2022-09-12T20:22:19.053 DEBUG:boto:Params: {} 2022-09-12T20:22:19.054 DEBUG:boto:Token: None 2022-09-12T20:22:19.055 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:19 GMT /jdpiak-77/obj62 2022-09-12T20:22:19.055 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:4ejTB9yoelsmIuYrd92e+pFH/z0= 2022-09-12T20:22:19.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:4ejTB9yoelsmIuYrd92e+pFH/z0='} 2022-09-12T20:22:19.061 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b008320e13fe63cc-00631f94fb-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:19.062 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:19.062 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:19.063 DEBUG:boto:Method: HEAD 2022-09-12T20:22:19.063 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:19.064 DEBUG:boto:Data: 2022-09-12T20:22:19.064 DEBUG:boto:Headers: {} 2022-09-12T20:22:19.065 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:19.065 DEBUG:boto:Port: 8000 2022-09-12T20:22:19.066 DEBUG:boto:Params: {} 2022-09-12T20:22:19.066 DEBUG:boto:Token: None 2022-09-12T20:22:19.067 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:19 GMT /jdpiak-77/ 2022-09-12T20:22:19.068 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3XVUPKDOIXtZ5UNFKK/jG2lxJ0k= 2022-09-12T20:22:19.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3XVUPKDOIXtZ5UNFKK/jG2lxJ0k=', 'Content-Length': '0'} 2022-09-12T20:22:19.072 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009faa8265e80652cd-00631f94fb-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:19.072 DEBUG:boto:path=/jdpiak-77/obj93 2022-09-12T20:22:19.073 DEBUG:boto:auth_path=/jdpiak-77/obj93 2022-09-12T20:22:19.074 DEBUG:boto:Method: PUT 2022-09-12T20:22:19.074 DEBUG:boto:Path: /jdpiak-77/obj93 2022-09-12T20:22:19.075 DEBUG:boto:Data: 2022-09-12T20:22:19.075 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:19.076 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:19.076 DEBUG:boto:Port: 8000 2022-09-12T20:22:19.077 DEBUG:boto:Params: {} 2022-09-12T20:22:19.077 DEBUG:boto:Token: None 2022-09-12T20:22:19.078 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:19 GMT /jdpiak-77/obj93 2022-09-12T20:22:19.079 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:1rwHwHnUg2eWMQ/3Ct53cRxg9co= 2022-09-12T20:22:19.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:1rwHwHnUg2eWMQ/3Ct53cRxg9co='} 2022-09-12T20:22:19.122 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000049c6ec331e5fb203-00631f94fb-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:19.123 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:19.124 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:19.124 DEBUG:boto:Method: HEAD 2022-09-12T20:22:19.125 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:19.125 DEBUG:boto:Data: 2022-09-12T20:22:19.126 DEBUG:boto:Headers: {} 2022-09-12T20:22:19.126 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:19.127 DEBUG:boto:Port: 8000 2022-09-12T20:22:19.128 DEBUG:boto:Params: {} 2022-09-12T20:22:19.128 DEBUG:boto:Token: None 2022-09-12T20:22:19.129 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:19 GMT /jdpiak-77/ 2022-09-12T20:22:19.129 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3XVUPKDOIXtZ5UNFKK/jG2lxJ0k= 2022-09-12T20:22:19.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3XVUPKDOIXtZ5UNFKK/jG2lxJ0k=', 'Content-Length': '0'} 2022-09-12T20:22:19.134 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbd903b2d9a2789d-00631f94fb-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:19.134 DEBUG:boto:path=/jdpiak-77/obj124 2022-09-12T20:22:19.135 DEBUG:boto:auth_path=/jdpiak-77/obj124 2022-09-12T20:22:19.135 DEBUG:boto:Method: PUT 2022-09-12T20:22:19.136 DEBUG:boto:Path: /jdpiak-77/obj124 2022-09-12T20:22:19.137 DEBUG:boto:Data: 2022-09-12T20:22:19.137 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:19.138 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:19.138 DEBUG:boto:Port: 8000 2022-09-12T20:22:19.139 DEBUG:boto:Params: {} 2022-09-12T20:22:19.139 DEBUG:boto:Token: None 2022-09-12T20:22:19.140 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:19 GMT /jdpiak-77/obj124 2022-09-12T20:22:19.140 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:JZ8WONjXe1lH6nbizMUOMuwxBiw= 2022-09-12T20:22:19.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:JZ8WONjXe1lH6nbizMUOMuwxBiw='} 2022-09-12T20:22:19.147 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009d991a7066cba337-00631f94fb-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:19.147 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:19.148 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:19.148 DEBUG:boto:Method: HEAD 2022-09-12T20:22:19.149 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:19.149 DEBUG:boto:Data: 2022-09-12T20:22:19.150 DEBUG:boto:Headers: {} 2022-09-12T20:22:19.151 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:19.151 DEBUG:boto:Port: 8000 2022-09-12T20:22:19.152 DEBUG:boto:Params: {} 2022-09-12T20:22:19.152 DEBUG:boto:Token: None 2022-09-12T20:22:19.153 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:19 GMT /jdpiak-77/ 2022-09-12T20:22:19.153 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3XVUPKDOIXtZ5UNFKK/jG2lxJ0k= 2022-09-12T20:22:19.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3XVUPKDOIXtZ5UNFKK/jG2lxJ0k=', 'Content-Length': '0'} 2022-09-12T20:22:19.158 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f4028dd5d758a3b-00631f94fb-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:19.158 DEBUG:boto:path=/jdpiak-77/obj155 2022-09-12T20:22:19.159 DEBUG:boto:auth_path=/jdpiak-77/obj155 2022-09-12T20:22:19.160 DEBUG:boto:Method: PUT 2022-09-12T20:22:19.160 DEBUG:boto:Path: /jdpiak-77/obj155 2022-09-12T20:22:19.161 DEBUG:boto:Data: 2022-09-12T20:22:19.161 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:19.162 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:19.162 DEBUG:boto:Port: 8000 2022-09-12T20:22:19.163 DEBUG:boto:Params: {} 2022-09-12T20:22:19.163 DEBUG:boto:Token: None 2022-09-12T20:22:19.164 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:19 GMT /jdpiak-77/obj155 2022-09-12T20:22:19.164 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9VwhBRVNmaeSCukxCTlT6AMCbII= 2022-09-12T20:22:19.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9VwhBRVNmaeSCukxCTlT6AMCbII='} 2022-09-12T20:22:19.171 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000541e4b4ed14c8eb7-00631f94fb-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:19.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-77', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:22:19.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'jdpiak-77', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:22:19.172 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket jdpiak-77 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-12T20:22:19.227 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:19.227 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:19.282 INFO:teuthology.orchestra.run.smithi035.stdout:tenant: 2022-09-12T20:22:19.283 INFO:teuthology.orchestra.run.smithi035.stdout:bucket name: jdpiak-77 2022-09-12T20:22:19.296 INFO:teuthology.orchestra.run.smithi035.stdout:total entries: 30 2022-09-12T20:22:19.400 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:19.400 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:19.401 DEBUG:boto:Method: HEAD 2022-09-12T20:22:19.401 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:19.402 DEBUG:boto:Data: 2022-09-12T20:22:19.402 DEBUG:boto:Headers: {} 2022-09-12T20:22:19.403 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:19.403 DEBUG:boto:Port: 8000 2022-09-12T20:22:19.404 DEBUG:boto:Params: {} 2022-09-12T20:22:19.404 DEBUG:boto:Token: None 2022-09-12T20:22:19.405 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:19 GMT /jdpiak-77/ 2022-09-12T20:22:19.405 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3XVUPKDOIXtZ5UNFKK/jG2lxJ0k= 2022-09-12T20:22:19.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3XVUPKDOIXtZ5UNFKK/jG2lxJ0k=', 'Content-Length': '0'} 2022-09-12T20:22:19.410 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000956aa21d41ca59d5-00631f94fb-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:19.411 DEBUG:boto:path=/jdpiak-77/obj37 2022-09-12T20:22:19.412 DEBUG:boto:auth_path=/jdpiak-77/obj37 2022-09-12T20:22:19.412 DEBUG:boto:Method: PUT 2022-09-12T20:22:19.413 DEBUG:boto:Path: /jdpiak-77/obj37 2022-09-12T20:22:19.413 DEBUG:boto:Data: 2022-09-12T20:22:19.414 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:19.414 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:19.415 DEBUG:boto:Port: 8000 2022-09-12T20:22:19.415 DEBUG:boto:Params: {} 2022-09-12T20:22:19.416 DEBUG:boto:Token: None 2022-09-12T20:22:19.417 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:19 GMT /jdpiak-77/obj37 2022-09-12T20:22:19.417 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:OsTex4SvBXh/0YSVoyMULVSBUwQ= 2022-09-12T20:22:19.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:OsTex4SvBXh/0YSVoyMULVSBUwQ='} 2022-09-12T20:22:19.462 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3c292116700a7ab-00631f94fb-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:19.463 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:19.464 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:19.464 DEBUG:boto:Method: HEAD 2022-09-12T20:22:19.465 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:19.465 DEBUG:boto:Data: 2022-09-12T20:22:19.466 DEBUG:boto:Headers: {} 2022-09-12T20:22:19.466 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:19.467 DEBUG:boto:Port: 8000 2022-09-12T20:22:19.467 DEBUG:boto:Params: {} 2022-09-12T20:22:19.468 DEBUG:boto:Token: None 2022-09-12T20:22:19.468 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:19 GMT /jdpiak-77/ 2022-09-12T20:22:19.469 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3XVUPKDOIXtZ5UNFKK/jG2lxJ0k= 2022-09-12T20:22:19.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3XVUPKDOIXtZ5UNFKK/jG2lxJ0k=', 'Content-Length': '0'} 2022-09-12T20:22:19.473 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d120d6cb416ba053-00631f94fb-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:19.473 DEBUG:boto:path=/jdpiak-77/obj74 2022-09-12T20:22:19.474 DEBUG:boto:auth_path=/jdpiak-77/obj74 2022-09-12T20:22:19.475 DEBUG:boto:Method: PUT 2022-09-12T20:22:19.475 DEBUG:boto:Path: /jdpiak-77/obj74 2022-09-12T20:22:19.475 DEBUG:boto:Data: 2022-09-12T20:22:19.476 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:19.477 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:19.477 DEBUG:boto:Port: 8000 2022-09-12T20:22:19.478 DEBUG:boto:Params: {} 2022-09-12T20:22:19.478 DEBUG:boto:Token: None 2022-09-12T20:22:19.479 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:19 GMT /jdpiak-77/obj74 2022-09-12T20:22:19.479 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:rZXA6sPalAdjF/SgZe+pzd4dkHI= 2022-09-12T20:22:19.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:rZXA6sPalAdjF/SgZe+pzd4dkHI='} 2022-09-12T20:22:19.486 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed199ac4239ccdfe-00631f94fb-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:19.486 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:19.487 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:19.487 DEBUG:boto:Method: HEAD 2022-09-12T20:22:19.488 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:19.488 DEBUG:boto:Data: 2022-09-12T20:22:19.489 DEBUG:boto:Headers: {} 2022-09-12T20:22:19.489 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:19.490 DEBUG:boto:Port: 8000 2022-09-12T20:22:19.490 DEBUG:boto:Params: {} 2022-09-12T20:22:19.491 DEBUG:boto:Token: None 2022-09-12T20:22:19.491 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:19 GMT /jdpiak-77/ 2022-09-12T20:22:19.492 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3XVUPKDOIXtZ5UNFKK/jG2lxJ0k= 2022-09-12T20:22:19.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3XVUPKDOIXtZ5UNFKK/jG2lxJ0k=', 'Content-Length': '0'} 2022-09-12T20:22:19.496 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e78cb7fbf0dc451-00631f94fb-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:19.497 DEBUG:boto:path=/jdpiak-77/obj111 2022-09-12T20:22:19.497 DEBUG:boto:auth_path=/jdpiak-77/obj111 2022-09-12T20:22:19.498 DEBUG:boto:Method: PUT 2022-09-12T20:22:19.498 DEBUG:boto:Path: /jdpiak-77/obj111 2022-09-12T20:22:19.499 DEBUG:boto:Data: 2022-09-12T20:22:19.499 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:19.500 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:19.500 DEBUG:boto:Port: 8000 2022-09-12T20:22:19.501 DEBUG:boto:Params: {} 2022-09-12T20:22:19.501 DEBUG:boto:Token: None 2022-09-12T20:22:19.502 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:19 GMT /jdpiak-77/obj111 2022-09-12T20:22:19.502 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:kytENKV69M3xS7/2/uL3MDwmzKY= 2022-09-12T20:22:19.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:kytENKV69M3xS7/2/uL3MDwmzKY='} 2022-09-12T20:22:19.546 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b60ba7e1164971f5-00631f94fb-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:19.547 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:19.548 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:19.548 DEBUG:boto:Method: HEAD 2022-09-12T20:22:19.549 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:19.549 DEBUG:boto:Data: 2022-09-12T20:22:19.550 DEBUG:boto:Headers: {} 2022-09-12T20:22:19.550 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:19.551 DEBUG:boto:Port: 8000 2022-09-12T20:22:19.551 DEBUG:boto:Params: {} 2022-09-12T20:22:19.552 DEBUG:boto:Token: None 2022-09-12T20:22:19.552 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:19 GMT /jdpiak-77/ 2022-09-12T20:22:19.553 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3XVUPKDOIXtZ5UNFKK/jG2lxJ0k= 2022-09-12T20:22:19.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3XVUPKDOIXtZ5UNFKK/jG2lxJ0k=', 'Content-Length': '0'} 2022-09-12T20:22:19.557 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037f5f67c8a3bb22f-00631f94fb-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:19.558 DEBUG:boto:path=/jdpiak-77/obj148 2022-09-12T20:22:19.558 DEBUG:boto:auth_path=/jdpiak-77/obj148 2022-09-12T20:22:19.559 DEBUG:boto:Method: PUT 2022-09-12T20:22:19.559 DEBUG:boto:Path: /jdpiak-77/obj148 2022-09-12T20:22:19.560 DEBUG:boto:Data: 2022-09-12T20:22:19.560 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:19.561 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:19.561 DEBUG:boto:Port: 8000 2022-09-12T20:22:19.562 DEBUG:boto:Params: {} 2022-09-12T20:22:19.562 DEBUG:boto:Token: None 2022-09-12T20:22:19.563 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:19 GMT /jdpiak-77/obj148 2022-09-12T20:22:19.564 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:egBcg4qWDXlzBlS8Kf/nqSLpAVY= 2022-09-12T20:22:19.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:egBcg4qWDXlzBlS8Kf/nqSLpAVY='} 2022-09-12T20:22:19.572 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008ecb74bfe3905d83-00631f94fb-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:19.573 DEBUG:boto:path=/jdpiak-77/ 2022-09-12T20:22:19.573 DEBUG:boto:auth_path=/jdpiak-77/ 2022-09-12T20:22:19.574 DEBUG:boto:Method: HEAD 2022-09-12T20:22:19.574 DEBUG:boto:Path: /jdpiak-77/ 2022-09-12T20:22:19.575 DEBUG:boto:Data: 2022-09-12T20:22:19.575 DEBUG:boto:Headers: {} 2022-09-12T20:22:19.576 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:19.576 DEBUG:boto:Port: 8000 2022-09-12T20:22:19.577 DEBUG:boto:Params: {} 2022-09-12T20:22:19.577 DEBUG:boto:Token: None 2022-09-12T20:22:19.578 DEBUG:boto:StringToSign: HEAD Mon, 12 Sep 2022 20:22:19 GMT /jdpiak-77/ 2022-09-12T20:22:19.578 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:3XVUPKDOIXtZ5UNFKK/jG2lxJ0k= 2022-09-12T20:22:19.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:22:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:3XVUPKDOIXtZ5UNFKK/jG2lxJ0k=', 'Content-Length': '0'} 2022-09-12T20:22:19.583 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc2e88d2d57ae7eb-00631f94fb-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:22:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:19.584 DEBUG:boto:path=/jdpiak-77/obj185 2022-09-12T20:22:19.584 DEBUG:boto:auth_path=/jdpiak-77/obj185 2022-09-12T20:22:19.585 DEBUG:boto:Method: PUT 2022-09-12T20:22:19.585 DEBUG:boto:Path: /jdpiak-77/obj185 2022-09-12T20:22:19.586 DEBUG:boto:Data: 2022-09-12T20:22:19.586 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-12T20:22:19.587 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:22:19.587 DEBUG:boto:Port: 8000 2022-09-12T20:22:19.588 DEBUG:boto:Params: {} 2022-09-12T20:22:19.588 DEBUG:boto:Token: None 2022-09-12T20:22:19.589 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Mon, 12 Sep 2022 20:22:19 GMT /jdpiak-77/obj185 2022-09-12T20:22:19.589 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:bxqna4V1vbTTkxO39fqqJiYzY3M= 2022-09-12T20:22:19.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Mon, 12 Sep 2022 20:22:19 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:bxqna4V1vbTTkxO39fqqJiYzY3M='} 2022-09-12T20:22:19.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e2fc66afc5853831-00631f94fb-15c0-test-zone1'), ('Date', 'Mon, 12 Sep 2022 20:22:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:22:19.635 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:22:19.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-12T20:22:19.636 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-12T20:22:19.697 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:19.697 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:19.749 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:22:19.747+0000 7f5bb7e9d600 10 RGW-SYNC:data:init_data_sync_status: took lease 2022-09-12T20:22:19.781 INFO:tasks.rgw.c2.client.0:Restarting daemon 2022-09-12T20:22:19.782 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2022-09-12T20:22:19.785 INFO:tasks.rgw.c2.client.0:Started 2022-09-12T20:22:19.785 DEBUG:teuthology.orchestra.run.smithi138:> curl http://smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:19.807 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-12T20:22:19.808 INFO:teuthology.orchestra.run.smithi138.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-12T20:22:19.808 INFO:teuthology.orchestra.run.smithi138.stderr: Dload Upload Total Spent Left Speed 2022-09-12T20:22:19.808 INFO:teuthology.orchestra.run.smithi138.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi138.front.sepia.ceph.com port 8000: Connection refused 2022-09-12T20:22:20.810 DEBUG:teuthology.orchestra.run.smithi138:> curl http://smithi138.front.sepia.ceph.com:8000 2022-09-12T20:22:20.834 INFO:teuthology.orchestra.run.smithi138.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-12T20:22:20.835 INFO:teuthology.orchestra.run.smithi138.stderr: Dload Upload Total Spent Left Speed 2022-09-12T20:22:20.835 INFO:teuthology.orchestra.run.smithi138.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-12T20:22:20.837 INFO:teuthology.orchestra.run.smithi138.stdout:anonymous 2022-09-12T20:22:20.837 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=jdpiak-77 2022-09-12T20:22:20.838 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-77', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:22:20.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'jdpiak-77', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-12T20:22:20.840 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket jdpiak-77 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-12T20:22:20.934 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:22:20.934 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:22:20.995 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:22:20.993+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:22:21.968 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2022-09-12T20:22:24.466 INFO:tasks.rgw.c1.client.1.smithi035.stdout:*** Caught signal (Segmentation fault) ** 2022-09-12T20:22:24.466 INFO:tasks.rgw.c1.client.1.smithi035.stdout: in thread 7f790e015700 thread_name:data-sync 2022-09-12T20:22:24.467 INFO:tasks.rgw.c1.client.1.smithi035.stdout: ceph version 17.0.0-14883-g10f7d25d (10f7d25d6a0f340676cd251eab5a787ba01ace95) quincy (dev) 2022-09-12T20:22:24.467 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7936d23cf0] 2022-09-12T20:22:24.467 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 2: /lib64/libc.so.6(+0xc96e0) [0x7f7935b436e0] 2022-09-12T20:22:24.467 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 3: (std::_Rb_tree::marker_entry>, std::_Select1st::marker_entry> >, std::less, std::allocator::marker_entry> > >::find(rgw_obj_key const&)+0x6f) [0x55cfe6de9d6f] 2022-09-12T20:22:24.468 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 4: (RGWBucketSyncSingleEntryCR::operate(DoutPrefixProvider const*)+0x8bc) [0x55cfe6e1356c] 2022-09-12T20:22:24.468 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x18f) [0x55cfe693a57f] 2022-09-12T20:22:24.468 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x521) [0x55cfe693ad51] 2022-09-12T20:22:24.468 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0x91) [0x55cfe693bd61] 2022-09-12T20:22:24.469 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x20a) [0x55cfe6db93fa] 2022-09-12T20:22:24.469 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 9: (RGWDataSyncProcessorThread::process(DoutPrefixProvider const*)+0x58) [0x55cfe6a187a8] 2022-09-12T20:22:24.469 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x55cfe69e2693] 2022-09-12T20:22:24.470 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 11: /lib64/libpthread.so.0(+0x81ca) [0x7f7936d191ca] 2022-09-12T20:22:24.470 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 12: clone() 2022-09-12T20:22:24.470 INFO:tasks.rgw.c1.client.1.smithi035.stdout:2022-09-12T20:22:24.462+0000 7f790e015700 -1 *** Caught signal (Segmentation fault) ** 2022-09-12T20:22:24.470 INFO:tasks.rgw.c1.client.1.smithi035.stdout: in thread 7f790e015700 thread_name:data-sync 2022-09-12T20:22:24.471 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 2022-09-12T20:22:24.471 INFO:tasks.rgw.c1.client.1.smithi035.stdout: ceph version 17.0.0-14883-g10f7d25d (10f7d25d6a0f340676cd251eab5a787ba01ace95) quincy (dev) 2022-09-12T20:22:24.471 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7936d23cf0] 2022-09-12T20:22:24.471 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 2: /lib64/libc.so.6(+0xc96e0) [0x7f7935b436e0] 2022-09-12T20:22:24.472 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 3: (std::_Rb_tree::marker_entry>, std::_Select1st::marker_entry> >, std::less, std::allocator::marker_entry> > >::find(rgw_obj_key const&)+0x6f) [0x55cfe6de9d6f] 2022-09-12T20:22:24.472 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 4: (RGWBucketSyncSingleEntryCR::operate(DoutPrefixProvider const*)+0x8bc) [0x55cfe6e1356c] 2022-09-12T20:22:24.472 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x18f) [0x55cfe693a57f] 2022-09-12T20:22:24.472 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x521) [0x55cfe693ad51] 2022-09-12T20:22:24.473 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0x91) [0x55cfe693bd61] 2022-09-12T20:22:24.473 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x20a) [0x55cfe6db93fa] 2022-09-12T20:22:24.473 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 9: (RGWDataSyncProcessorThread::process(DoutPrefixProvider const*)+0x58) [0x55cfe6a187a8] 2022-09-12T20:22:24.473 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x55cfe69e2693] 2022-09-12T20:22:24.473 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 11: /lib64/libpthread.so.0(+0x81ca) [0x7f7936d191ca] 2022-09-12T20:22:24.474 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 12: clone() 2022-09-12T20:22:24.474 INFO:tasks.rgw.c1.client.1.smithi035.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-09-12T20:22:24.474 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 2022-09-12T20:22:24.482 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 0> 2022-09-12T20:22:24.462+0000 7f790e015700 -1 *** Caught signal (Segmentation fault) ** 2022-09-12T20:22:24.483 INFO:tasks.rgw.c1.client.1.smithi035.stdout: in thread 7f790e015700 thread_name:data-sync 2022-09-12T20:22:24.483 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 2022-09-12T20:22:24.483 INFO:tasks.rgw.c1.client.1.smithi035.stdout: ceph version 17.0.0-14883-g10f7d25d (10f7d25d6a0f340676cd251eab5a787ba01ace95) quincy (dev) 2022-09-12T20:22:24.483 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7936d23cf0] 2022-09-12T20:22:24.484 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 2: /lib64/libc.so.6(+0xc96e0) [0x7f7935b436e0] 2022-09-12T20:22:24.484 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 3: (std::_Rb_tree::marker_entry>, std::_Select1st::marker_entry> >, std::less, std::allocator::marker_entry> > >::find(rgw_obj_key const&)+0x6f) [0x55cfe6de9d6f] 2022-09-12T20:22:24.484 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 4: (RGWBucketSyncSingleEntryCR::operate(DoutPrefixProvider const*)+0x8bc) [0x55cfe6e1356c] 2022-09-12T20:22:24.485 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x18f) [0x55cfe693a57f] 2022-09-12T20:22:24.485 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x521) [0x55cfe693ad51] 2022-09-12T20:22:24.485 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0x91) [0x55cfe693bd61] 2022-09-12T20:22:24.485 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x20a) [0x55cfe6db93fa] 2022-09-12T20:22:24.486 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 9: (RGWDataSyncProcessorThread::process(DoutPrefixProvider const*)+0x58) [0x55cfe6a187a8] 2022-09-12T20:22:24.486 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x55cfe69e2693] 2022-09-12T20:22:24.486 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 11: /lib64/libpthread.so.0(+0x81ca) [0x7f7936d191ca] 2022-09-12T20:22:24.487 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 12: clone() 2022-09-12T20:22:24.487 INFO:tasks.rgw.c1.client.1.smithi035.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-09-12T20:22:24.487 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 2022-09-12T20:22:24.493 INFO:tasks.rgw.c1.client.1.smithi035.stdout: -462> 2022-09-12T20:22:24.462+0000 7f790e015700 -1 *** Caught signal (Segmentation fault) ** 2022-09-12T20:22:24.493 INFO:tasks.rgw.c1.client.1.smithi035.stdout: in thread 7f790e015700 thread_name:data-sync 2022-09-12T20:22:24.494 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 2022-09-12T20:22:24.494 INFO:tasks.rgw.c1.client.1.smithi035.stdout: ceph version 17.0.0-14883-g10f7d25d (10f7d25d6a0f340676cd251eab5a787ba01ace95) quincy (dev) 2022-09-12T20:22:24.494 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7936d23cf0] 2022-09-12T20:22:24.495 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 2: /lib64/libc.so.6(+0xc96e0) [0x7f7935b436e0] 2022-09-12T20:22:24.495 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 3: (std::_Rb_tree::marker_entry>, std::_Select1st::marker_entry> >, std::less, std::allocator::marker_entry> > >::find(rgw_obj_key const&)+0x6f) [0x55cfe6de9d6f] 2022-09-12T20:22:24.495 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 4: (RGWBucketSyncSingleEntryCR::operate(DoutPrefixProvider const*)+0x8bc) [0x55cfe6e1356c] 2022-09-12T20:22:24.495 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x18f) [0x55cfe693a57f] 2022-09-12T20:22:24.496 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x521) [0x55cfe693ad51] 2022-09-12T20:22:24.496 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0x91) [0x55cfe693bd61] 2022-09-12T20:22:24.496 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x20a) [0x55cfe6db93fa] 2022-09-12T20:22:24.497 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 9: (RGWDataSyncProcessorThread::process(DoutPrefixProvider const*)+0x58) [0x55cfe6a187a8] 2022-09-12T20:22:24.497 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x55cfe69e2693] 2022-09-12T20:22:24.497 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 11: /lib64/libpthread.so.0(+0x81ca) [0x7f7936d191ca] 2022-09-12T20:22:24.497 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 12: clone() 2022-09-12T20:22:24.498 INFO:tasks.rgw.c1.client.1.smithi035.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-09-12T20:22:24.498 INFO:tasks.rgw.c1.client.1.smithi035.stdout: 2022-09-12T20:22:24.586 INFO:tasks.rgw.c1.client.1.smithi035.stderr:daemon-helper: command crashed with signal 11 2022-09-12T20:22:25.995 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:22:25.994+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:22:27.878 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2022-09-12T20:22:30.996 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:22:30.994+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:22:33.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~6s 2022-09-12T20:22:35.996 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:22:35.995+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:22:39.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~11s 2022-09-12T20:22:40.996 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:22:40.996+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:22:44.014 INFO:tasks.rgw.c1.client.0.smithi035.stdout:2022-09-12T20:22:44.013+0000 7f29e274c700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5) 2022-09-12T20:22:44.088 INFO:tasks.rgw.c1.client.0.smithi035.stdout:2022-09-12T20:22:44.086+0000 7f29e274c700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-16) 2022-09-12T20:22:44.213 INFO:tasks.rgw.c1.client.0.smithi035.stdout:2022-09-12T20:22:44.211+0000 7f29e274c700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-16) 2022-09-12T20:22:44.293 INFO:tasks.rgw.c1.client.0.smithi035.stdout:2022-09-12T20:22:44.292+0000 7f29e274c700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16) 2022-09-12T20:22:44.697 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~17s 2022-09-12T20:22:45.997 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:22:45.995+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:22:50.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~22s 2022-09-12T20:22:50.998 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:22:50.996+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:22:55.911 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~28s 2022-09-12T20:22:56.001 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:22:55.996+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:23:00.999 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:23:00.997+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:23:01.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~34s 2022-09-12T20:23:05.999 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:23:05.998+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:23:07.123 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~39s 2022-09-12T20:23:11.000 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:23:10.998+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:23:12.730 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~45s 2022-09-12T20:23:16.000 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:23:15.999+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:23:18.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~50s 2022-09-12T20:23:21.001 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:23:21.000+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:23:23.942 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~56s 2022-09-12T20:23:24.534 INFO:tasks.rgw.c2.client.0.smithi138.stdout:*** Caught signal (Segmentation fault) ** 2022-09-12T20:23:24.535 INFO:tasks.rgw.c2.client.0.smithi138.stdout: in thread 7f0832339700 thread_name:data-sync 2022-09-12T20:23:24.537 INFO:tasks.rgw.c2.client.0.smithi138.stdout: ceph version 17.0.0-14883-g10f7d25d (10f7d25d6a0f340676cd251eab5a787ba01ace95) quincy (dev) 2022-09-12T20:23:24.538 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f085d04bcf0] 2022-09-12T20:23:24.538 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 2: (RGWBucketSyncSingleEntryCR::operate(DoutPrefixProvider const*)+0x870) [0x55795457a520] 2022-09-12T20:23:24.539 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 3: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x18f) [0x5579540a157f] 2022-09-12T20:23:24.539 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 4: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x521) [0x5579540a1d51] 2022-09-12T20:23:24.540 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 5: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0x91) [0x5579540a2d61] 2022-09-12T20:23:24.541 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 6: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x20a) [0x5579545203fa] 2022-09-12T20:23:24.541 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 7: (RGWDataSyncProcessorThread::process(DoutPrefixProvider const*)+0x58) [0x55795417f7a8] 2022-09-12T20:23:24.541 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 8: (RGWRadosThread::Worker::entry()+0xb3) [0x557954149693] 2022-09-12T20:23:24.542 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f085d0411ca] 2022-09-12T20:23:24.542 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 10: clone() 2022-09-12T20:23:24.543 INFO:tasks.rgw.c2.client.0.smithi138.stdout:2022-09-12T20:23:24.537+0000 7f0832339700 -1 *** Caught signal (Segmentation fault) ** 2022-09-12T20:23:24.543 INFO:tasks.rgw.c2.client.0.smithi138.stdout: in thread 7f0832339700 thread_name:data-sync 2022-09-12T20:23:24.544 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 2022-09-12T20:23:24.544 INFO:tasks.rgw.c2.client.0.smithi138.stdout: ceph version 17.0.0-14883-g10f7d25d (10f7d25d6a0f340676cd251eab5a787ba01ace95) quincy (dev) 2022-09-12T20:23:24.545 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f085d04bcf0] 2022-09-12T20:23:24.545 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 2: (RGWBucketSyncSingleEntryCR::operate(DoutPrefixProvider const*)+0x870) [0x55795457a520] 2022-09-12T20:23:24.546 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 3: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x18f) [0x5579540a157f] 2022-09-12T20:23:24.546 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 4: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x521) [0x5579540a1d51] 2022-09-12T20:23:24.547 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 5: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0x91) [0x5579540a2d61] 2022-09-12T20:23:24.547 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 6: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x20a) [0x5579545203fa] 2022-09-12T20:23:24.548 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 7: (RGWDataSyncProcessorThread::process(DoutPrefixProvider const*)+0x58) [0x55795417f7a8] 2022-09-12T20:23:24.548 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 8: (RGWRadosThread::Worker::entry()+0xb3) [0x557954149693] 2022-09-12T20:23:24.549 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f085d0411ca] 2022-09-12T20:23:24.549 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 10: clone() 2022-09-12T20:23:24.550 INFO:tasks.rgw.c2.client.0.smithi138.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-09-12T20:23:24.550 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 2022-09-12T20:23:24.552 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 0> 2022-09-12T20:23:24.537+0000 7f0832339700 -1 *** Caught signal (Segmentation fault) ** 2022-09-12T20:23:24.552 INFO:tasks.rgw.c2.client.0.smithi138.stdout: in thread 7f0832339700 thread_name:data-sync 2022-09-12T20:23:24.553 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 2022-09-12T20:23:24.553 INFO:tasks.rgw.c2.client.0.smithi138.stdout: ceph version 17.0.0-14883-g10f7d25d (10f7d25d6a0f340676cd251eab5a787ba01ace95) quincy (dev) 2022-09-12T20:23:24.554 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f085d04bcf0] 2022-09-12T20:23:24.554 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 2: (RGWBucketSyncSingleEntryCR::operate(DoutPrefixProvider const*)+0x870) [0x55795457a520] 2022-09-12T20:23:24.555 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 3: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x18f) [0x5579540a157f] 2022-09-12T20:23:24.555 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 4: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x521) [0x5579540a1d51] 2022-09-12T20:23:24.556 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 5: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0x91) [0x5579540a2d61] 2022-09-12T20:23:24.556 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 6: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x20a) [0x5579545203fa] 2022-09-12T20:23:24.556 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 7: (RGWDataSyncProcessorThread::process(DoutPrefixProvider const*)+0x58) [0x55795417f7a8] 2022-09-12T20:23:24.557 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 8: (RGWRadosThread::Worker::entry()+0xb3) [0x557954149693] 2022-09-12T20:23:24.557 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f085d0411ca] 2022-09-12T20:23:24.558 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 10: clone() 2022-09-12T20:23:24.558 INFO:tasks.rgw.c2.client.0.smithi138.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-09-12T20:23:24.559 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 2022-09-12T20:23:24.564 INFO:tasks.rgw.c2.client.0.smithi138.stdout: -57> 2022-09-12T20:23:24.537+0000 7f0832339700 -1 *** Caught signal (Segmentation fault) ** 2022-09-12T20:23:24.564 INFO:tasks.rgw.c2.client.0.smithi138.stdout: in thread 7f0832339700 thread_name:data-sync 2022-09-12T20:23:24.565 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 2022-09-12T20:23:24.565 INFO:tasks.rgw.c2.client.0.smithi138.stdout: ceph version 17.0.0-14883-g10f7d25d (10f7d25d6a0f340676cd251eab5a787ba01ace95) quincy (dev) 2022-09-12T20:23:24.565 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f085d04bcf0] 2022-09-12T20:23:24.566 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 2: (RGWBucketSyncSingleEntryCR::operate(DoutPrefixProvider const*)+0x870) [0x55795457a520] 2022-09-12T20:23:24.566 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 3: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x18f) [0x5579540a157f] 2022-09-12T20:23:24.566 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 4: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x521) [0x5579540a1d51] 2022-09-12T20:23:24.567 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 5: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0x91) [0x5579540a2d61] 2022-09-12T20:23:24.567 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 6: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x20a) [0x5579545203fa] 2022-09-12T20:23:24.567 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 7: (RGWDataSyncProcessorThread::process(DoutPrefixProvider const*)+0x58) [0x55795417f7a8] 2022-09-12T20:23:24.568 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 8: (RGWRadosThread::Worker::entry()+0xb3) [0x557954149693] 2022-09-12T20:23:24.568 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f085d0411ca] 2022-09-12T20:23:24.569 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 10: clone() 2022-09-12T20:23:24.569 INFO:tasks.rgw.c2.client.0.smithi138.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-09-12T20:23:24.570 INFO:tasks.rgw.c2.client.0.smithi138.stdout: 2022-09-12T20:23:24.729 INFO:tasks.rgw.c2.client.0.smithi138.stderr:daemon-helper: command crashed with signal 11 2022-09-12T20:23:26.002 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:23:26.001+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:23:29.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~62s 2022-09-12T20:23:30.343 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2022-09-12T20:23:31.002 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:23:31.001+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:23:35.155 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~67s 2022-09-12T20:23:35.949 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~6s 2022-09-12T20:23:36.003 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:23:36.001+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:23:40.762 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~73s 2022-09-12T20:23:41.004 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:23:41.002+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:23:41.555 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~11s 2022-09-12T20:23:46.005 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:23:46.003+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:23:46.368 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~78s 2022-09-12T20:23:47.161 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~17s 2022-09-12T20:23:51.005 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:23:51.003+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:23:51.975 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~84s 2022-09-12T20:23:52.767 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~22s 2022-09-12T20:23:56.006 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:23:56.004+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:23:57.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~90s 2022-09-12T20:23:58.373 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~28s 2022-09-12T20:24:01.007 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:24:01.005+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:24:03.187 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~95s 2022-09-12T20:24:03.980 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~34s 2022-09-12T20:24:06.007 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:24:06.006+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:24:08.793 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~101s 2022-09-12T20:24:09.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~39s 2022-09-12T20:24:11.008 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:24:11.007+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:24:14.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~107s 2022-09-12T20:24:15.191 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~45s 2022-09-12T20:24:16.008 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:24:16.008+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:24:20.006 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~112s 2022-09-12T20:24:20.798 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~50s 2022-09-12T20:24:21.009 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:24:21.007+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:24:25.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~118s 2022-09-12T20:24:26.009 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:24:26.008+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:24:26.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~56s 2022-09-12T20:24:31.011 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:24:31.008+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:24:31.224 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~123s 2022-09-12T20:24:32.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~62s 2022-09-12T20:24:36.011 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:24:36.009+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:24:36.831 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~129s 2022-09-12T20:24:37.615 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~67s 2022-09-12T20:24:41.011 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:24:41.010+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:24:42.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~135s 2022-09-12T20:24:43.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~73s 2022-09-12T20:24:46.012 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:24:46.010+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:24:48.044 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~140s 2022-09-12T20:24:48.827 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~78s 2022-09-12T20:24:51.013 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:24:51.011+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:24:53.650 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~146s 2022-09-12T20:24:54.433 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~84s 2022-09-12T20:24:56.013 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:24:56.013+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:24:59.257 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~151s 2022-09-12T20:25:00.039 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~90s 2022-09-12T20:25:01.014 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:25:01.013+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:25:04.862 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~157s 2022-09-12T20:25:05.644 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~95s 2022-09-12T20:25:06.015 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:25:06.013+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:25:10.468 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~163s 2022-09-12T20:25:11.015 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:25:11.014+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:25:11.250 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~101s 2022-09-12T20:25:16.016 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:25:16.014+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:25:16.075 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~168s 2022-09-12T20:25:16.856 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~107s 2022-09-12T20:25:21.016 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:25:21.015+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:25:21.682 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~174s 2022-09-12T20:25:22.462 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~112s 2022-09-12T20:25:26.017 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:25:26.015+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:25:27.288 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~179s 2022-09-12T20:25:28.067 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~118s 2022-09-12T20:25:31.018 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:25:31.016+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:25:32.894 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~185s 2022-09-12T20:25:33.673 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~123s 2022-09-12T20:25:36.018 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:25:36.017+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:25:38.500 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~191s 2022-09-12T20:25:39.279 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~129s 2022-09-12T20:25:41.019 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:25:41.017+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:25:44.107 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~196s 2022-09-12T20:25:44.886 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~135s 2022-09-12T20:25:46.020 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:25:46.019+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:25:49.713 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~202s 2022-09-12T20:25:50.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~140s 2022-09-12T20:25:51.020 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:25:51.019+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:25:55.320 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~207s 2022-09-12T20:25:56.021 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:25:56.019+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:25:56.099 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~146s 2022-09-12T20:26:00.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~213s 2022-09-12T20:26:01.022 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:26:01.020+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:26:01.706 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~151s 2022-09-12T20:26:06.022 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:26:06.020+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:26:06.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~219s 2022-09-12T20:26:07.312 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~157s 2022-09-12T20:26:11.022 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:26:11.020+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:26:12.137 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~224s 2022-09-12T20:26:12.917 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~163s 2022-09-12T20:26:16.023 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:26:16.021+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:26:17.743 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~230s 2022-09-12T20:26:18.524 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~168s 2022-09-12T20:26:21.023 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:26:21.022+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:26:23.349 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~235s 2022-09-12T20:26:24.129 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~174s 2022-09-12T20:26:26.024 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:26:26.022+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:26:28.956 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~241s 2022-09-12T20:26:29.735 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~179s 2022-09-12T20:26:31.025 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:26:31.024+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:26:34.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~247s 2022-09-12T20:26:35.340 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~185s 2022-09-12T20:26:36.025 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:26:36.025+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:26:40.168 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~252s 2022-09-12T20:26:40.946 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~191s 2022-09-12T20:26:41.026 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:26:41.024+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:26:45.773 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~258s 2022-09-12T20:26:46.026 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:26:46.025+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:26:46.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~196s 2022-09-12T20:26:51.027 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:26:51.026+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:26:51.379 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~264s 2022-09-12T20:26:52.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~202s 2022-09-12T20:26:56.035 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:26:56.026+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:26:56.984 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~269s 2022-09-12T20:26:57.763 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~207s 2022-09-12T20:27:01.028 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:27:01.027+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:27:02.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~275s 2022-09-12T20:27:03.369 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~213s 2022-09-12T20:27:06.029 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:27:06.027+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:27:08.196 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~280s 2022-09-12T20:27:08.976 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~219s 2022-09-12T20:27:11.029 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:27:11.028+0000 7fa3ab005600 1 waiting to reach incremental sync.. 2022-09-12T20:27:13.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~286s 2022-09-12T20:27:14.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~224s 2022-09-12T20:27:16.030 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:27:16.030+0000 7fa3ab005600 -1 bucket checkpoint timed out waiting to reach incremental sync 2022-09-12T20:27:16.031 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:27:16.030+0000 7fa3ab005600 0 bucket sync checkpoint failed: (110) Connection timed out 2022-09-12T20:27:16.031 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:27:16.030+0000 7fa3ab005600 -1 bucket sync checkpoint failed: (110) Connection timed out 2022-09-12T20:27:16.053 DEBUG:teuthology.orchestra.run:got remote process result: 110 2022-09-12T20:27:16.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... FAIL 2022-09-12T20:27:16.063 DEBUG:boto:path=/jdpiak-78/ 2022-09-12T20:27:16.064 DEBUG:boto:auth_path=/jdpiak-78/ 2022-09-12T20:27:16.064 DEBUG:boto:Method: PUT 2022-09-12T20:27:16.065 DEBUG:boto:Path: /jdpiak-78/ 2022-09-12T20:27:16.065 DEBUG:boto:Data: 2022-09-12T20:27:16.066 DEBUG:boto:Headers: {} 2022-09-12T20:27:16.066 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T20:27:16.067 DEBUG:boto:Port: 8000 2022-09-12T20:27:16.067 DEBUG:boto:Params: {} 2022-09-12T20:27:16.069 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T20:27:16.069 DEBUG:boto:Token: None 2022-09-12T20:27:16.070 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 20:27:16 GMT /jdpiak-78/ 2022-09-12T20:27:16.071 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:9D3PBZ9tS3ho11+ky3n9ZxGz50s= 2022-09-12T20:27:16.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 20:27:16 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:9D3PBZ9tS3ho11+ky3n9ZxGz50s=', 'Content-Length': '0'} 2022-09-12T20:27:16.100 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000427a2417b7c7c013-00631f9624-15c0-test-zone1'), ('Content-Length', '0'), ('Date', 'Mon, 12 Sep 2022 20:27:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-12T20:27:16.101 DEBUG:rgw_multi.tests:created bucket=jdpiak-78 2022-09-12T20:27:16.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:27:16.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:27:16.104 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:27:16.163 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:27:16.164 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:27:16.199 INFO:teuthology.orchestra.run.smithi035.stderr:No --period given, using current period=38ca1f57-1358-496c-a99b-df49ee365a4f 2022-09-12T20:27:16.213 INFO:teuthology.orchestra.run.smithi035.stdout:[ 2022-09-12T20:27:16.213 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:27:16.213 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T20:27:16.214 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T20:27:16.214 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:27:16.215 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:27:16.215 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "", 2022-09-12T20:27:16.215 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "0.000000" 2022-09-12T20:27:16.215 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:27:16.216 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:27:16.216 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663014436.097832_1531.1", 2022-09-12T20:27:16.216 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:27:16.097832Z" 2022-09-12T20:27:16.216 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2022-09-12T20:27:16.217 INFO:teuthology.orchestra.run.smithi035.stdout: { 2022-09-12T20:27:16.217 INFO:teuthology.orchestra.run.smithi035.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:27:16.217 INFO:teuthology.orchestra.run.smithi035.stdout: "last_update": "2022-09-12T20:16:38.638998Z" 2022-09-12T20:27:16.217 INFO:teuthology.orchestra.run.smithi035.stdout: } 2022-09-12T20:27:16.218 INFO:teuthology.orchestra.run.smithi035.stdout:] 2022-09-12T20:27:16.219 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1663014436.097832_1531.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:27:16.219 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-12T20:27:16.220 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:27:16.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:27:16.221 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:27:16.281 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:27:16.282 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:27:16.320 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:27:16.320+0000 7f5cde4c8600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:27:16.321 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:27:16.321+0000 7f5cde4c8600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:27:16.335 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:27:16.335 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:27:16.336 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:27:16.336 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:27:16.336 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:27:16.337 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:27:16.337 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:16.337 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:16.337 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:27:16.338 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:16.338 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:27:16.338 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:16.339 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:16.339 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:27:16.339 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:16.339 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:27:16.340 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:16.340 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:27:16.340 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:27:16.340 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:16.341 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:16.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:16.341 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:27:16.341 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:16.342 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:16.342 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:27:16.343 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:16.343 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:16.343 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:16.343 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:27:16.344 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:27:16.344 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:16.344 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:16.345 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:16.345 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:27:16.345 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:16.345 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:16.346 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663014125.239690_900.1", 2022-09-12T20:27:16.346 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:16.346 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:16.346 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:16.347 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:22:05.239690Z", 2022-09-12T20:27:16.347 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:16.347 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:16.348 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:16.348 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:16.348 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:27:16.348 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:16.349 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:16.349 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:27:16.349 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:16.349 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:16.350 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:16.350 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:27:16.350 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:16.350 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:16.351 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:16.351 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:27:16.351 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:16.351 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:27:16.352 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:27:16.352 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:27:16.352 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:16.352 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:27:16Z" 2022-09-12T20:27:16.353 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:27:16.354 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663014125.239690_900.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:22:05.239690Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:27:16Z" } 2022-09-12T20:27:16.355 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663014125.239690_900.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:22:05.239690Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:27:16.355 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663014436.097832_1531.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:27:16.356 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663014125.239690_900.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:27:16.356 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663014436.097832_1531.1 target=1_1663014125.239690_900.1 2022-09-12T20:27:19.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~292s 2022-09-12T20:27:20.187 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~230s 2022-09-12T20:27:21.358 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:27:21.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:27:21.359 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:27:21.415 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:27:21.416 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:27:21.446 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:27:21.446+0000 7fa5bce0f600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:27:21.447 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:27:21.447+0000 7fa5bce0f600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:27:21.464 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:27:21.465 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:27:21.465 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:27:21.465 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:27:21.465 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:27:21.466 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:27:21.466 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:21.466 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:21.467 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:27:21.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:21.468 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:27:21.468 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:21.468 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:21.469 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:27:21.469 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:21.469 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:27:21.469 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:21.470 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:27:21.470 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:27:21.470 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:21.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:21.471 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:21.471 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:27:21.471 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:21.471 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:21.472 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:27:21.472 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:21.472 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:21.472 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:21.473 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:27:21.473 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:27:21.473 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:21.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:21.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:21.474 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:27:21.475 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:21.475 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:21.475 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663014125.239690_900.1", 2022-09-12T20:27:21.475 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:21.476 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:21.476 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:21.476 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:22:05.239690Z", 2022-09-12T20:27:21.477 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:21.477 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:21.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:21.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:21.478 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:27:21.478 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:21.479 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:21.479 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:27:21.479 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:21.479 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:21.480 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:21.480 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:27:21.480 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:21.480 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:21.481 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:21.481 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:27:21.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:21.481 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:27:21.482 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:27:21.482 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:27:21.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:21.482 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:27:21Z" 2022-09-12T20:27:21.483 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:27:21.484 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663014125.239690_900.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:22:05.239690Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:27:21Z" } 2022-09-12T20:27:21.485 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663014125.239690_900.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:22:05.239690Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:27:21.485 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663014436.097832_1531.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:27:21.486 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663014125.239690_900.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:27:21.487 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663014436.097832_1531.1 target=1_1663014125.239690_900.1 2022-09-12T20:27:25.013 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~297s 2022-09-12T20:27:25.793 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~235s 2022-09-12T20:27:26.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:27:26.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:27:26.490 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:27:26.547 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:27:26.547 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:27:26.589 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:27:26.587+0000 7ff7091e4600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:27:26.590 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:27:26.588+0000 7ff7091e4600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:27:26.605 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:27:26.605 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:27:26.605 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:27:26.606 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:27:26.606 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:27:26.606 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:27:26.607 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:26.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:26.607 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:27:26.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:26.608 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:27:26.608 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:26.608 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:26.608 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:27:26.609 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:26.609 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:27:26.609 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:26.609 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:27:26.610 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:27:26.610 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:26.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:26.610 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:26.611 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:27:26.611 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:26.611 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:26.612 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:27:26.612 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:26.612 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:26.613 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:26.613 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:27:26.613 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:27:26.613 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:26.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:26.614 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:26.614 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:27:26.614 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:26.615 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:26.615 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663014125.239690_900.1", 2022-09-12T20:27:26.615 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:26.615 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:26.616 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:26.616 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:22:05.239690Z", 2022-09-12T20:27:26.616 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:26.616 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:26.617 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:26.617 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:26.617 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:27:26.617 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:26.618 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:26.618 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:27:26.618 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:26.618 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:26.619 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:26.619 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:27:26.619 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:26.619 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:26.620 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:26.620 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:27:26.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:26.620 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:27:26.621 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:27:26.621 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:27:26.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:26.621 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:27:26Z" 2022-09-12T20:27:26.622 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:27:26.623 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663014125.239690_900.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:22:05.239690Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:27:26Z" } 2022-09-12T20:27:26.623 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663014125.239690_900.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:22:05.239690Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:27:26.624 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663014436.097832_1531.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:27:26.624 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663014125.239690_900.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:27:26.625 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663014436.097832_1531.1 target=1_1663014125.239690_900.1 2022-09-12T20:27:30.619 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~303s 2022-09-12T20:27:30.620 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2022-09-12T20:27:31.226 INFO:tasks.ceph.c1.osd.0:Sent signal 15 2022-09-12T20:27:31.230 INFO:tasks.ceph.c1.osd.1:Sent signal 15 2022-09-12T20:27:31.231 INFO:tasks.ceph.c1.osd.2:Sent signal 15 2022-09-12T20:27:31.231 INFO:tasks.ceph.c1.mon.a:Sent signal 15 2022-09-12T20:27:31.232 INFO:tasks.rgw.c1.client.0:Sent signal 15 2022-09-12T20:27:31.233 INFO:tasks.ceph.c1.mgr.x:Sent signal 15 2022-09-12T20:27:31.235 INFO:tasks.ceph.c1.osd.0.smithi035.stderr:2022-09-12T20:27:31.226+0000 7f45ac5b9700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 0 (PID: 101687) UID: 0 2022-09-12T20:27:31.235 INFO:tasks.ceph.c1.osd.0.smithi035.stderr:2022-09-12T20:27:31.226+0000 7f45ac5b9700 -1 osd.0 58 *** Got signal Terminated *** 2022-09-12T20:27:31.236 INFO:tasks.ceph.c1.osd.0.smithi035.stderr:2022-09-12T20:27:31.226+0000 7f45ac5b9700 -1 osd.0 58 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-09-12T20:27:31.237 INFO:tasks.ceph.c1.osd.1.smithi035.stderr:2022-09-12T20:27:31.231+0000 7f4c8535a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 1 (PID: 101690) UID: 0 2022-09-12T20:27:31.237 INFO:tasks.ceph.c1.osd.1.smithi035.stderr:2022-09-12T20:27:31.231+0000 7f4c8535a700 -1 osd.1 58 *** Got signal Terminated *** 2022-09-12T20:27:31.238 INFO:tasks.ceph.c1.osd.1.smithi035.stderr:2022-09-12T20:27:31.231+0000 7f4c8535a700 -1 osd.1 58 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-09-12T20:27:31.239 INFO:tasks.ceph.c1.osd.2.smithi035.stderr:2022-09-12T20:27:31.231+0000 7f6383e8b700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 2 (PID: 101693) UID: 0 2022-09-12T20:27:31.240 INFO:tasks.ceph.c1.osd.2.smithi035.stderr:2022-09-12T20:27:31.231+0000 7f6383e8b700 -1 osd.2 58 *** Got signal Terminated *** 2022-09-12T20:27:31.240 INFO:tasks.ceph.c1.osd.2.smithi035.stderr:2022-09-12T20:27:31.231+0000 7f6383e8b700 -1 osd.2 58 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-09-12T20:27:31.241 INFO:tasks.ceph.c1.mon.a.smithi035.stderr:2022-09-12T20:27:31.232+0000 7f1eecc38700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster c1 -i a (PID: 101277) UID: 0 2022-09-12T20:27:31.242 INFO:tasks.ceph.c1.mon.a.smithi035.stderr:2022-09-12T20:27:31.232+0000 7f1eecc38700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2022-09-12T20:27:31.243 INFO:tasks.rgw.c1.client.0.smithi035.stdout:2022-09-12T20:27:31.233+0000 7f2a0178a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 109173) UID: 0 2022-09-12T20:27:31.243 INFO:tasks.rgw.c1.client.0.smithi035.stdout:2022-09-12T20:27:31.233+0000 7f2a059f5600 -1 shutting down 2022-09-12T20:27:31.244 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T20:27:31.234+0000 7f267102e700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster c1 -i x (PID: 101280) UID: 0 2022-09-12T20:27:31.245 INFO:tasks.ceph.c1.mgr.x.smithi035.stderr:2022-09-12T20:27:31.234+0000 7f267102e700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2022-09-12T20:27:31.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~241s 2022-09-12T20:27:31.627 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:27:31.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:27:31.628 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:27:31.686 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:27:31.686 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:27:31.716 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:27:31.715+0000 7f252a750600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:27:31.717 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:27:31.716+0000 7f252a750600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:27:31.732 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:27:31.732 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:27:31.733 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:27:31.733 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:27:31.733 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:27:31.733 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:27:31.734 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:31.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:31.734 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:27:31.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:31.735 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:27:31.735 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:31.735 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:31.736 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:27:31.736 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:31.736 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:27:31.736 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:31.737 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:27:31.737 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:27:31.737 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:31.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:31.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:31.738 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:27:31.738 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:31.739 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:31.739 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:27:31.739 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:31.739 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:31.740 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:31.740 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:27:31.740 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:27:31.740 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:31.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:31.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:31.741 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:27:31.742 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:31.742 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:31.742 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663014125.239690_900.1", 2022-09-12T20:27:31.742 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:31.743 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:31.743 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:31.743 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:22:05.239690Z", 2022-09-12T20:27:31.743 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:31.744 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:31.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:31.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:31.744 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:27:31.745 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:31.745 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:31.745 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:27:31.745 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:31.746 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:31.746 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:31.746 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:27:31.747 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:31.747 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:31.747 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:31.747 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:27:31.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:31.748 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:27:31.748 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:27:31.748 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:27:31.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:31.749 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:27:31Z" 2022-09-12T20:27:31.749 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:27:31.751 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663014125.239690_900.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:22:05.239690Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:27:31Z" } 2022-09-12T20:27:31.751 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663014125.239690_900.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:22:05.239690Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:27:31.752 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663014436.097832_1531.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:27:31.752 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663014125.239690_900.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:27:31.753 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663014436.097832_1531.1 target=1_1663014125.239690_900.1 2022-09-12T20:27:33.966 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T20:27:33.965+0000 7fe8bd190700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16) 2022-09-12T20:27:34.096 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T20:27:34.094+0000 7fe8bd190700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-11) 2022-09-12T20:27:34.166 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T20:27:34.164+0000 7fe8bd190700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-11) 2022-09-12T20:27:34.245 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T20:27:34.244+0000 7fe8bd190700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11)ERROR: collect() returned error (ret=-11) 2022-09-12T20:27:36.754 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:27:36.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:27:36.756 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:27:36.813 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:27:36.814 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:27:36.854 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:27:36.852+0000 7f741b017600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:27:36.855 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:27:36.854+0000 7f741b017600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:27:36.871 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:27:36.872 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:27:36.872 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:27:36.872 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:27:36.873 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:27:36.873 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:27:36.873 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:36.874 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:36.874 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:27:36.874 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:36.874 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:27:36.875 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:36.875 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:36.875 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:27:36.875 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:36.876 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:27:36.876 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:36.876 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:27:36.876 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:27:36.877 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:36.877 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:36.877 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:36.877 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:27:36.878 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:36.878 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:36.879 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:27:36.879 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:36.879 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:36.879 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:36.880 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:27:36.880 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:27:36.880 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:36.880 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:36.881 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:36.881 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:27:36.881 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:36.881 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:36.882 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663014125.239690_900.1", 2022-09-12T20:27:36.882 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:36.882 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:36.882 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:36.883 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:22:05.239690Z", 2022-09-12T20:27:36.883 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:36.883 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:36.883 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:36.884 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:36.884 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:27:36.884 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:36.884 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:36.885 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:27:36.885 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:36.885 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:36.885 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:36.886 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:27:36.886 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:36.886 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:36.886 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:36.887 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:27:36.887 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:36.887 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:27:36.887 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:27:36.888 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:27:36.888 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:36.888 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:27:36Z" 2022-09-12T20:27:36.888 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:27:36.890 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663014125.239690_900.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:22:05.239690Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:27:36Z" } 2022-09-12T20:27:36.890 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663014125.239690_900.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:22:05.239690Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:27:36.891 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663014436.097832_1531.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:27:36.891 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663014125.239690_900.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:27:36.892 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663014436.097832_1531.1 target=1_1663014125.239690_900.1 2022-09-12T20:27:37.005 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~247s 2022-09-12T20:27:41.893 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:27:41.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:27:41.895 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:27:41.950 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:27:41.951 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:27:41.985 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:27:41.983+0000 7f1bfeac8600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:27:41.986 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:27:41.985+0000 7f1bfeac8600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:27:42.005 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:27:42.005 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:27:42.005 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:27:42.005 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:27:42.006 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:27:42.006 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:27:42.006 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:42.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:42.007 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:27:42.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:42.007 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:27:42.008 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:42.008 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:42.008 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:27:42.008 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:42.009 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:27:42.009 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:42.009 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:27:42.009 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:27:42.010 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:42.010 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:42.010 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:42.011 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:27:42.011 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:42.011 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:42.011 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:27:42.012 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:42.012 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:42.012 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:42.012 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:27:42.013 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:27:42.013 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:42.013 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:42.013 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:42.014 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:27:42.014 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:42.014 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:42.014 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663014125.239690_900.1", 2022-09-12T20:27:42.015 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:42.015 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:42.015 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:42.015 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:22:05.239690Z", 2022-09-12T20:27:42.016 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:42.016 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:42.016 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:42.017 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:42.017 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:27:42.017 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:42.017 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:42.018 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:27:42.018 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:42.018 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:42.018 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:42.019 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:27:42.019 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:42.019 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:42.020 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:42.020 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:27:42.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:42.020 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:27:42.021 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:27:42.021 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:27:42.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:42.021 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:27:41Z" 2022-09-12T20:27:42.022 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:27:42.023 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663014125.239690_900.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:22:05.239690Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:27:41Z" } 2022-09-12T20:27:42.024 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663014125.239690_900.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:22:05.239690Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:27:42.025 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663014436.097832_1531.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:27:42.025 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663014125.239690_900.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:27:42.026 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663014436.097832_1531.1 target=1_1663014125.239690_900.1 2022-09-12T20:27:42.626 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~252s 2022-09-12T20:27:47.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:27:47.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:27:47.028 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:27:47.085 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:27:47.086 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:27:47.128 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:27:47.127+0000 7fb3e47a9600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:27:47.129 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:27:47.128+0000 7fb3e47a9600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:27:47.150 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:27:47.150 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:27:47.150 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:27:47.151 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:27:47.151 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:27:47.151 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:27:47.151 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:47.152 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:47.152 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:27:47.152 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:47.152 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:27:47.153 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:47.153 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:47.153 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:27:47.153 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:47.154 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:27:47.154 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:47.154 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:27:47.155 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:27:47.155 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:47.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:47.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:47.156 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:27:47.156 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:47.156 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:47.156 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:27:47.157 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:47.157 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:47.157 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:47.157 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:27:47.158 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:27:47.158 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:47.158 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:47.158 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:47.159 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:27:47.159 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:47.159 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:47.159 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663014125.239690_900.1", 2022-09-12T20:27:47.160 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:47.160 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:47.160 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:47.160 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:22:05.239690Z", 2022-09-12T20:27:47.161 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:47.161 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:47.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:47.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:47.162 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:27:47.162 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:47.162 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:47.162 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:27:47.163 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:47.163 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:47.163 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:47.164 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:27:47.164 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:47.164 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:47.164 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:47.165 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:27:47.165 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:47.165 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:27:47.165 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:27:47.166 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:27:47.166 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:47.166 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:27:47Z" 2022-09-12T20:27:47.166 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:27:47.168 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663014125.239690_900.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:22:05.239690Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:27:47Z" } 2022-09-12T20:27:47.168 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663014125.239690_900.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:22:05.239690Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:27:47.169 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663014436.097832_1531.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:27:47.169 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663014125.239690_900.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:27:47.170 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663014436.097832_1531.1 target=1_1663014125.239690_900.1 2022-09-12T20:27:48.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~258s 2022-09-12T20:27:52.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:27:52.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:27:52.172 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:27:52.230 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:27:52.230 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:27:52.272 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:27:52.271+0000 7f3122392600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:27:52.273 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:27:52.272+0000 7f3122392600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:27:52.290 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:27:52.290 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:27:52.290 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:27:52.291 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:27:52.291 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:27:52.291 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:27:52.291 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:52.292 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:52.292 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:27:52.292 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:52.292 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:27:52.293 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:52.293 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:52.293 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:27:52.293 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:52.294 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:27:52.294 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:52.294 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:27:52.294 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:27:52.295 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:52.295 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:52.296 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:52.296 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:27:52.296 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:52.296 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:52.297 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:27:52.297 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:52.297 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:52.297 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:52.298 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:27:52.298 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:27:52.298 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:52.298 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:52.299 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:52.299 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:27:52.299 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:52.299 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:52.300 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663014125.239690_900.1", 2022-09-12T20:27:52.300 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:52.301 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:52.301 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:52.302 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:22:05.239690Z", 2022-09-12T20:27:52.302 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:52.302 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:52.303 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:52.303 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:52.304 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:27:52.304 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:52.304 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:52.305 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:27:52.305 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:52.306 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:52.306 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:52.306 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:27:52.307 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:52.307 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:52.308 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:52.308 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:27:52.309 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:52.309 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:27:52.309 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:27:52.310 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:27:52.310 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:52.311 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:27:52Z" 2022-09-12T20:27:52.311 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:27:52.313 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663014125.239690_900.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:22:05.239690Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:27:52Z" } 2022-09-12T20:27:52.314 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663014125.239690_900.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:22:05.239690Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:27:52.315 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663014436.097832_1531.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:27:52.316 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663014125.239690_900.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:27:52.317 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663014436.097832_1531.1 target=1_1663014125.239690_900.1 2022-09-12T20:27:53.837 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~263s 2022-09-12T20:27:57.318 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:27:57.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:27:57.320 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:27:57.376 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:27:57.377 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:27:57.424 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:27:57.423+0000 7ff766cb2600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:27:57.425 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:27:57.424+0000 7ff766cb2600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:27:57.443 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:27:57.444 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:27:57.444 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:27:57.444 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:27:57.445 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:27:57.445 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:27:57.445 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:57.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:57.446 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:27:57.446 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:57.446 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:27:57.447 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:57.447 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:57.447 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:27:57.447 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:57.448 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:27:57.448 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:57.448 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:27:57.448 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:27:57.449 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:57.449 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:57.449 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:57.449 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:27:57.450 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:57.450 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:57.450 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:27:57.451 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:57.451 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:57.451 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:57.451 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:27:57.452 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:27:57.452 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:57.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:57.452 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:57.453 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:27:57.453 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:57.453 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:57.453 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663014125.239690_900.1", 2022-09-12T20:27:57.454 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:57.454 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:57.454 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:57.454 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:22:05.239690Z", 2022-09-12T20:27:57.455 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:57.455 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:57.455 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:57.455 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:27:57.456 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:27:57.456 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:27:57.456 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:27:57.456 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:27:57.457 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:27:57.457 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:27:57.457 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:27:57.457 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:27:57.458 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:27:57.458 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:57.458 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:27:57.459 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:27:57.459 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:57.459 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:27:57.459 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:27:57.460 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:27:57.460 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:27:57.460 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:27:57Z" 2022-09-12T20:27:57.460 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:27:57.461 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663014125.239690_900.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:22:05.239690Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:27:57Z" } 2022-09-12T20:27:57.462 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663014125.239690_900.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:22:05.239690Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:27:57.463 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663014436.097832_1531.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:27:57.463 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663014125.239690_900.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:27:57.464 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663014436.097832_1531.1 target=1_1663014125.239690_900.1 2022-09-12T20:27:59.443 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~269s 2022-09-12T20:28:02.465 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:28:02.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:28:02.467 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:28:02.523 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:28:02.524 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:28:02.567 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:28:02.566+0000 7f53f056d600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:28:02.568 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:28:02.568+0000 7f53f056d600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:28:02.582 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:28:02.583 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:28:02.583 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:28:02.583 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:28:02.584 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:28:02.584 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:28:02.584 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:28:02.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:02.585 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:28:02.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:02.585 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:28:02.585 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:02.586 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:02.586 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:28:02.586 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:02.587 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:28:02.587 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:02.587 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:28:02.588 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:28:02.588 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:02.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:02.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:02.589 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:28:02.589 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:02.589 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:02.589 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:28:02.590 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:02.590 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:28:02.590 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:02.591 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:28:02.591 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:28:02.591 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:02.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:02.592 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:02.592 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:28:02.592 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:02.592 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:02.593 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663014125.239690_900.1", 2022-09-12T20:28:02.593 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:02.593 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:28:02.593 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:02.594 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:22:05.239690Z", 2022-09-12T20:28:02.594 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:28:02.594 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:02.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:02.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:02.595 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:28:02.595 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:02.595 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:02.596 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:28:02.596 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:02.596 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:28:02.596 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:02.597 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:28:02.597 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:28:02.597 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:02.598 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:02.598 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:28:02.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:02.599 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:28:02.599 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:28:02.599 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:28:02.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:02.600 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:28:02Z" 2022-09-12T20:28:02.600 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:28:02.601 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663014125.239690_900.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:22:05.239690Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:28:02Z" } 2022-09-12T20:28:02.602 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663014125.239690_900.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:22:05.239690Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:28:02.603 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663014436.097832_1531.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:28:02.603 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663014125.239690_900.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:28:02.604 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663014436.097832_1531.1 target=1_1663014125.239690_900.1 2022-09-12T20:28:05.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~275s 2022-09-12T20:28:07.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:28:07.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:28:07.607 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:28:07.664 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:28:07.664 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:28:07.696 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:28:07.696+0000 7ffa1d4aa600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:28:07.697 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:28:07.697+0000 7ffa1d4aa600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:28:07.711 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:28:07.712 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:28:07.712 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:28:07.712 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:28:07.713 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:28:07.713 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:28:07.713 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:28:07.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:07.714 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:28:07.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:07.715 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:28:07.715 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:07.715 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:07.715 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:28:07.716 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:07.716 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:28:07.716 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:07.716 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:28:07.717 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:28:07.717 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:07.717 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:07.718 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:07.718 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:28:07.718 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:07.718 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:07.719 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:28:07.719 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:07.719 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:28:07.719 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:07.720 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:28:07.720 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:28:07.720 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:07.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:07.721 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:07.721 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:28:07.721 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:07.721 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:07.722 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663014125.239690_900.1", 2022-09-12T20:28:07.722 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:07.722 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:28:07.722 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:07.723 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:22:05.239690Z", 2022-09-12T20:28:07.723 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:28:07.723 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:07.723 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:07.724 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:07.724 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:28:07.724 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:07.724 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:07.725 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:28:07.725 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:07.725 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:28:07.725 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:07.726 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:28:07.726 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:28:07.726 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:07.726 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:07.727 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:28:07.727 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:07.727 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:28:07.728 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:28:07.728 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:28:07.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:07.728 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:28:07Z" 2022-09-12T20:28:07.729 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:28:07.730 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663014125.239690_900.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:22:05.239690Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:28:07Z" } 2022-09-12T20:28:07.731 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663014125.239690_900.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:22:05.239690Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:28:07.731 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663014436.097832_1531.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:28:07.732 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663014125.239690_900.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:28:07.732 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663014436.097832_1531.1 target=1_1663014125.239690_900.1 2022-09-12T20:28:10.652 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~280s 2022-09-12T20:28:12.734 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:28:12.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:28:12.735 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:28:12.792 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:28:12.793 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:28:12.839 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:28:12.837+0000 7fb55205b600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:28:12.840 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:28:12.838+0000 7fb55205b600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:28:12.857 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:28:12.858 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:28:12.858 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:28:12.858 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:28:12.859 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:28:12.859 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:28:12.859 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:28:12.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:12.860 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:28:12.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:12.860 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:28:12.861 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:12.861 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:12.861 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:28:12.861 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:12.862 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:28:12.862 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:12.862 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:28:12.863 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:28:12.863 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:12.863 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:12.863 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:12.864 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:28:12.864 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:12.864 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:12.864 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:28:12.865 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:12.865 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:28:12.865 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:12.865 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:28:12.866 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:28:12.866 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:12.866 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:12.866 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:12.867 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:28:12.867 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:12.867 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:12.867 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663014125.239690_900.1", 2022-09-12T20:28:12.868 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:12.868 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:28:12.868 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:12.868 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:22:05.239690Z", 2022-09-12T20:28:12.869 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:28:12.869 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:12.869 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:12.869 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:12.870 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:28:12.870 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:12.870 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:12.871 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:28:12.871 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:12.871 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:28:12.872 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:12.872 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:28:12.872 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:28:12.872 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:12.873 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:12.873 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:28:12.873 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:12.873 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:28:12.874 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:28:12.874 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:28:12.874 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:12.874 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:28:12Z" 2022-09-12T20:28:12.875 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:28:12.876 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663014125.239690_900.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:22:05.239690Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:28:12Z" } 2022-09-12T20:28:12.877 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663014125.239690_900.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:22:05.239690Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:28:12.877 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663014436.097832_1531.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:28:12.878 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663014125.239690_900.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:28:12.878 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663014436.097832_1531.1 target=1_1663014125.239690_900.1 2022-09-12T20:28:16.258 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~286s 2022-09-12T20:28:17.879 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:28:17.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:28:17.881 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:28:17.939 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:28:17.940 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:28:17.983 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:28:17.982+0000 7fed14532600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:28:17.986 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:28:17.983+0000 7fed14532600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:28:18.000 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:28:18.001 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:28:18.001 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:28:18.001 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:28:18.002 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:28:18.002 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:28:18.003 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:28:18.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:18.004 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:28:18.004 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:18.005 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:28:18.005 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:18.006 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:18.006 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:28:18.006 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:18.007 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:28:18.007 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:18.008 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:28:18.008 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:28:18.009 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:18.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:18.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:18.010 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:28:18.010 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:18.011 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:18.011 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:28:18.012 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:18.012 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:28:18.013 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:18.013 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:28:18.013 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:28:18.014 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:18.015 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:18.015 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:18.016 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:28:18.016 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:18.016 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:18.017 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663014125.239690_900.1", 2022-09-12T20:28:18.017 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:18.018 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:28:18.018 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:18.019 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:22:05.239690Z", 2022-09-12T20:28:18.019 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:28:18.020 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:18.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:18.021 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:18.021 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:28:18.022 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:18.022 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:18.023 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:28:18.023 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:18.023 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:28:18.024 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:18.024 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:28:18.025 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:28:18.025 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:18.026 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:18.026 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:28:18.026 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:18.027 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:28:18.027 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:28:18.028 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:28:18.028 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:18.029 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:28:17Z" 2022-09-12T20:28:18.029 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:28:18.031 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663014125.239690_900.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:22:05.239690Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:28:17Z" } 2022-09-12T20:28:18.041 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663014125.239690_900.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:22:05.239690Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:28:18.043 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663014436.097832_1531.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:28:18.044 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663014125.239690_900.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:28:18.044 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663014436.097832_1531.1 target=1_1663014125.239690_900.1 2022-09-12T20:28:21.863 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~292s 2022-09-12T20:28:23.048 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:28:23.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:28:23.050 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:28:23.107 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:28:23.108 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:28:23.156 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:28:23.155+0000 7fb2c7953600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:28:23.158 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:28:23.156+0000 7fb2c7953600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:28:23.176 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:28:23.177 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:28:23.178 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:28:23.178 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:28:23.179 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:28:23.179 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:28:23.180 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:28:23.180 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:23.181 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:28:23.181 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:23.182 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:28:23.183 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:23.183 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:23.184 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:28:23.184 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:23.185 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:28:23.185 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:23.186 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:28:23.186 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:28:23.187 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:23.187 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:23.188 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:23.188 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:28:23.189 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:23.189 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:23.190 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:28:23.190 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:23.191 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:28:23.191 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:23.192 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:28:23.192 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:28:23.193 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:23.193 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:23.194 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:23.194 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:28:23.195 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:23.196 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:23.196 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663014125.239690_900.1", 2022-09-12T20:28:23.197 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:23.197 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:28:23.197 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:23.198 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:22:05.239690Z", 2022-09-12T20:28:23.199 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:28:23.199 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:23.200 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:23.200 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:23.201 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:28:23.201 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:23.202 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:23.202 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:28:23.203 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:23.203 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:28:23.204 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:23.204 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:28:23.205 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:28:23.205 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:23.206 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:23.206 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:28:23.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:23.207 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:28:23.208 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:28:23.209 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:28:23.209 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:23.210 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:28:23Z" 2022-09-12T20:28:23.211 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:28:23.213 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663014125.239690_900.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:22:05.239690Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:28:23Z" } 2022-09-12T20:28:23.214 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663014125.239690_900.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:22:05.239690Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:28:23.215 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663014436.097832_1531.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:28:23.216 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663014125.239690_900.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:28:23.217 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663014436.097832_1531.1 target=1_1663014125.239690_900.1 2022-09-12T20:28:27.469 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~297s 2022-09-12T20:28:28.219 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:28:28.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:28:28.221 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:28:28.279 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:28:28.280 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:28:28.332 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:28:28.331+0000 7efcf8630600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:28:28.333 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:28:28.332+0000 7efcf8630600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:28:28.353 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:28:28.354 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:28:28.354 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:28:28.354 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:28:28.355 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:28:28.355 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:28:28.355 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:28:28.355 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:28.356 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:28:28.356 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:28.356 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:28:28.356 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:28.357 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:28.357 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:28:28.357 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:28.358 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:28:28.358 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:28.358 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:28:28.359 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:28:28.359 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:28.359 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:28.360 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:28.360 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:28:28.360 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:28.360 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:28.361 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:28:28.361 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:28.361 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:28:28.361 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:28.362 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:28:28.362 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:28:28.362 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:28.363 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:28.363 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:28.363 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:28:28.363 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:28.364 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:28.364 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663014125.239690_900.1", 2022-09-12T20:28:28.364 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:28.364 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:28:28.365 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:28.365 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:22:05.239690Z", 2022-09-12T20:28:28.365 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:28:28.366 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:28.366 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:28.366 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:28.366 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:28:28.367 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:28.367 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:28.367 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:28:28.368 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:28.368 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:28:28.368 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:28.368 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:28:28.369 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:28:28.369 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:28.369 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:28.370 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:28:28.370 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:28.370 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:28:28.370 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:28:28.371 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:28:28.371 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:28.371 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:28:28Z" 2022-09-12T20:28:28.372 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:28:28.373 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663014125.239690_900.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:22:05.239690Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:28:28Z" } 2022-09-12T20:28:28.374 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663014125.239690_900.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:22:05.239690Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:28:28.374 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663014436.097832_1531.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:28:28.375 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663014125.239690_900.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:28:28.375 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663014436.097832_1531.1 target=1_1663014125.239690_900.1 2022-09-12T20:28:33.075 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~303s 2022-09-12T20:28:33.076 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2022-09-12T20:28:33.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:28:33.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:28:33.378 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:28:33.437 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:28:33.437 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:28:33.470 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:28:33.468+0000 7ff3bbe87600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:28:33.471 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:28:33.469+0000 7ff3bbe87600 20 RGW-SYNC:meta: read sync status 2022-09-12T20:28:33.487 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2022-09-12T20:28:33.488 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_status": { 2022-09-12T20:28:33.488 INFO:teuthology.orchestra.run.smithi138.stdout: "info": { 2022-09-12T20:28:33.488 INFO:teuthology.orchestra.run.smithi138.stdout: "status": "sync", 2022-09-12T20:28:33.488 INFO:teuthology.orchestra.run.smithi138.stdout: "num_shards": 4, 2022-09-12T20:28:33.489 INFO:teuthology.orchestra.run.smithi138.stdout: "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", 2022-09-12T20:28:33.489 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:28:33.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:33.489 INFO:teuthology.orchestra.run.smithi138.stdout: "markers": [ 2022-09-12T20:28:33.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:33.490 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 0, 2022-09-12T20:28:33.490 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:33.491 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:33.491 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663008447.976965_53.1", 2022-09-12T20:28:33.491 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:33.491 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 1, 2022-09-12T20:28:33.492 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:33.492 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T18:47:27.976965Z", 2022-09-12T20:28:33.492 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:28:33.492 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:33.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:33.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:33.493 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 1, 2022-09-12T20:28:33.493 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:33.494 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:33.494 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "", 2022-09-12T20:28:33.494 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:33.495 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:28:33.495 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:33.495 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "0.000000", 2022-09-12T20:28:33.495 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 2 2022-09-12T20:28:33.496 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:33.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:33.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:33.496 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 2, 2022-09-12T20:28:33.497 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:33.497 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:33.497 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663014125.239690_900.1", 2022-09-12T20:28:33.497 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:33.498 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:28:33.498 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:33.498 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:22:05.239690Z", 2022-09-12T20:28:33.498 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:28:33.499 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:33.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:33.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2022-09-12T20:28:33.500 INFO:teuthology.orchestra.run.smithi138.stdout: "key": 3, 2022-09-12T20:28:33.500 INFO:teuthology.orchestra.run.smithi138.stdout: "val": { 2022-09-12T20:28:33.500 INFO:teuthology.orchestra.run.smithi138.stdout: "state": 1, 2022-09-12T20:28:33.500 INFO:teuthology.orchestra.run.smithi138.stdout: "marker": "1_1663013798.638998_440.1", 2022-09-12T20:28:33.500 INFO:teuthology.orchestra.run.smithi138.stdout: "next_step_marker": "", 2022-09-12T20:28:33.501 INFO:teuthology.orchestra.run.smithi138.stdout: "total_entries": 0, 2022-09-12T20:28:33.501 INFO:teuthology.orchestra.run.smithi138.stdout: "pos": 0, 2022-09-12T20:28:33.501 INFO:teuthology.orchestra.run.smithi138.stdout: "timestamp": "2022-09-12T20:16:38.638998Z", 2022-09-12T20:28:33.501 INFO:teuthology.orchestra.run.smithi138.stdout: "realm_epoch": 4 2022-09-12T20:28:33.502 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:33.502 INFO:teuthology.orchestra.run.smithi138.stdout: } 2022-09-12T20:28:33.502 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2022-09-12T20:28:33.503 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:33.503 INFO:teuthology.orchestra.run.smithi138.stdout: "full_sync": { 2022-09-12T20:28:33.503 INFO:teuthology.orchestra.run.smithi138.stdout: "total": 1, 2022-09-12T20:28:33.503 INFO:teuthology.orchestra.run.smithi138.stdout: "complete": 1 2022-09-12T20:28:33.504 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2022-09-12T20:28:33.504 INFO:teuthology.orchestra.run.smithi138.stdout: "current_time": "2022-09-12T20:28:33Z" 2022-09-12T20:28:33.504 INFO:teuthology.orchestra.run.smithi138.stdout:} 2022-09-12T20:28:33.506 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "38ca1f57-1358-496c-a99b-df49ee365a4f", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1663008447.976965_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-12T18:47:27.976965Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1663014125.239690_900.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:22:05.239690Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1663013798.638998_440.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-12T20:16:38.638998Z", "realm_epoch": 4 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-12T20:28:33Z" } 2022-09-12T20:28:33.507 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1663008447.976965_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-12T18:47:27.976965Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1663014125.239690_900.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:22:05.239690Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1663013798.638998_440.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-12T20:16:38.638998Z', 'realm_epoch': 4}}] 2022-09-12T20:28:33.507 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1663014436.097832_1531.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:28:33.508 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1663014125.239690_900.1', 3: '1_1663013798.638998_440.1'} 2022-09-12T20:28:33.508 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1663014436.097832_1531.1 target=1_1663014125.239690_900.1 2022-09-12T20:28:33.681 INFO:tasks.ceph.c2.osd.0:Sent signal 15 2022-09-12T20:28:33.682 INFO:tasks.ceph.c2.osd.1:Sent signal 15 2022-09-12T20:28:33.683 INFO:tasks.ceph.c2.osd.2:Sent signal 15 2022-09-12T20:28:33.684 INFO:tasks.ceph.c2.mon.a:Sent signal 15 2022-09-12T20:28:33.684 INFO:tasks.rgw.c2.client.1:Sent signal 15 2022-09-12T20:28:33.685 INFO:tasks.ceph.c2.mgr.x:Sent signal 15 2022-09-12T20:28:33.686 INFO:tasks.ceph.c2.osd.0.smithi138.stderr:2022-09-12T20:28:33.681+0000 7f0041f79700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 0 (PID: 101785) UID: 0 2022-09-12T20:28:33.686 INFO:tasks.ceph.c2.osd.0.smithi138.stderr:2022-09-12T20:28:33.681+0000 7f0041f79700 -1 osd.0 60 *** Got signal Terminated *** 2022-09-12T20:28:33.687 INFO:tasks.ceph.c2.osd.0.smithi138.stderr:2022-09-12T20:28:33.681+0000 7f0041f79700 -1 osd.0 60 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-09-12T20:28:33.687 INFO:tasks.ceph.c2.osd.1.smithi138.stderr:2022-09-12T20:28:33.682+0000 7fddd1dbb700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 1 (PID: 101788) UID: 0 2022-09-12T20:28:33.688 INFO:tasks.ceph.c2.osd.1.smithi138.stderr:2022-09-12T20:28:33.682+0000 7fddd1dbb700 -1 osd.1 60 *** Got signal Terminated *** 2022-09-12T20:28:33.688 INFO:tasks.ceph.c2.osd.1.smithi138.stderr:2022-09-12T20:28:33.682+0000 7fddd1dbb700 -1 osd.1 60 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-09-12T20:28:33.689 INFO:tasks.ceph.c2.osd.2.smithi138.stderr:2022-09-12T20:28:33.682+0000 7f7afb885700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 2 (PID: 101793) UID: 0 2022-09-12T20:28:33.689 INFO:tasks.ceph.c2.osd.2.smithi138.stderr:2022-09-12T20:28:33.682+0000 7f7afb885700 -1 osd.2 60 *** Got signal Terminated *** 2022-09-12T20:28:33.689 INFO:tasks.ceph.c2.osd.2.smithi138.stderr:2022-09-12T20:28:33.682+0000 7f7afb885700 -1 osd.2 60 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-09-12T20:28:33.690 INFO:tasks.ceph.c2.mon.a.smithi138.stderr:2022-09-12T20:28:33.683+0000 7f2da9a53700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster c2 -i a (PID: 101375) UID: 0 2022-09-12T20:28:33.690 INFO:tasks.ceph.c2.mon.a.smithi138.stderr:2022-09-12T20:28:33.683+0000 7f2da9a53700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2022-09-12T20:28:33.691 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T20:28:33.684+0000 7fe8db1cc700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 110325) UID: 0 2022-09-12T20:28:33.691 INFO:tasks.rgw.c2.client.1.smithi138.stdout:2022-09-12T20:28:33.684+0000 7fe8df437600 -1 shutting down 2022-09-12T20:28:38.510 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:28:38.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T20:28:38.511 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T20:28:38.568 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setuser ceph since I am not root 2022-09-12T20:28:38.569 INFO:teuthology.orchestra.run.smithi138.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T20:33:38.572 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:33:38.568+0000 7fb61487f600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T20:38:38.573 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:38:38.569+0000 7fb61487f600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T20:43:38.575 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:43:38.570+0000 7fb61487f600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T20:48:38.576 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:48:38.571+0000 7fb61487f600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T20:53:38.575 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:53:38.570+0000 7fb61487f600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T20:58:38.576 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T20:58:38.571+0000 7fb61487f600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T21:03:38.577 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T21:03:38.572+0000 7fb61487f600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T21:08:38.576 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T21:08:38.572+0000 7fb61487f600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T21:13:38.577 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T21:13:38.573+0000 7fb61487f600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T21:18:38.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-12T21:18:39.640 INFO:teuthology.orchestra.run.smithi138.stderr:2022-09-12T21:18:38.574+0000 7fb61487f600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T21:18:39.641 INFO:teuthology.orchestra.run.smithi138.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-12T21:18:39.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... FAIL 2022-09-12T21:18:39.652 DEBUG:boto:path=/jdpiak-79/ 2022-09-12T21:18:39.663 DEBUG:boto:auth_path=/jdpiak-79/ 2022-09-12T21:18:39.664 DEBUG:boto:Method: PUT 2022-09-12T21:18:39.665 DEBUG:boto:Path: /jdpiak-79/ 2022-09-12T21:18:39.667 DEBUG:boto:Data: 2022-09-12T21:18:39.668 DEBUG:boto:Headers: {} 2022-09-12T21:18:39.670 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T21:18:39.671 DEBUG:boto:Port: 8000 2022-09-12T21:18:39.672 DEBUG:boto:Params: {} 2022-09-12T21:18:39.675 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T21:18:39.676 DEBUG:boto:Token: None 2022-09-12T21:18:39.678 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 21:18:39 GMT /jdpiak-79/ 2022-09-12T21:18:39.680 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+T681rRTifEBA7rkseiohiQW3ns= 2022-09-12T21:18:39.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 21:18:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+T681rRTifEBA7rkseiohiQW3ns=', 'Content-Length': '0'} 2022-09-12T21:18:39.688 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-12T21:18:39.697 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T21:18:40.573 DEBUG:boto:Token: None 2022-09-12T21:18:40.574 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 21:18:39 GMT /jdpiak-79/ 2022-09-12T21:18:40.575 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+T681rRTifEBA7rkseiohiQW3ns= 2022-09-12T21:18:40.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 21:18:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+T681rRTifEBA7rkseiohiQW3ns=', 'Content-Length': '0'} 2022-09-12T21:18:40.579 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-12T21:18:40.581 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T21:18:42.263 DEBUG:boto:Token: None 2022-09-12T21:18:42.264 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 21:18:39 GMT /jdpiak-79/ 2022-09-12T21:18:42.265 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+T681rRTifEBA7rkseiohiQW3ns= 2022-09-12T21:18:42.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 21:18:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+T681rRTifEBA7rkseiohiQW3ns=', 'Content-Length': '0'} 2022-09-12T21:18:42.272 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-12T21:18:42.273 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T21:18:45.409 DEBUG:boto:Token: None 2022-09-12T21:18:45.409 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 21:18:39 GMT /jdpiak-79/ 2022-09-12T21:18:45.410 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+T681rRTifEBA7rkseiohiQW3ns= 2022-09-12T21:18:45.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 21:18:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+T681rRTifEBA7rkseiohiQW3ns=', 'Content-Length': '0'} 2022-09-12T21:18:45.413 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-12T21:18:45.413 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T21:18:51.582 DEBUG:boto:Token: None 2022-09-12T21:18:51.584 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 21:18:39 GMT /jdpiak-79/ 2022-09-12T21:18:51.585 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+T681rRTifEBA7rkseiohiQW3ns= 2022-09-12T21:18:51.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 21:18:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+T681rRTifEBA7rkseiohiQW3ns=', 'Content-Length': '0'} 2022-09-12T21:18:51.592 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-12T21:18:51.593 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T21:18:58.666 DEBUG:boto:Token: None 2022-09-12T21:18:58.667 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 21:18:39 GMT /jdpiak-79/ 2022-09-12T21:18:58.668 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+T681rRTifEBA7rkseiohiQW3ns= 2022-09-12T21:18:58.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 21:18:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+T681rRTifEBA7rkseiohiQW3ns=', 'Content-Length': '0'} 2022-09-12T21:18:58.673 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-12T21:18:58.674 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T21:19:18.360 DEBUG:boto:Token: None 2022-09-12T21:19:18.362 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 21:18:39 GMT /jdpiak-79/ 2022-09-12T21:19:18.363 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:+T681rRTifEBA7rkseiohiQW3ns= 2022-09-12T21:19:18.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 21:18:39 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:+T681rRTifEBA7rkseiohiQW3ns=', 'Content-Length': '0'} 2022-09-12T21:19:18.368 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-12T21:19:18.369 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T21:19:42.090 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... ERROR 2022-09-12T21:19:42.093 DEBUG:boto:path=/jdpiak-80/ 2022-09-12T21:19:42.094 DEBUG:boto:auth_path=/jdpiak-80/ 2022-09-12T21:19:42.094 DEBUG:boto:Method: PUT 2022-09-12T21:19:42.095 DEBUG:boto:Path: /jdpiak-80/ 2022-09-12T21:19:42.095 DEBUG:boto:Data: 2022-09-12T21:19:42.096 DEBUG:boto:Headers: {} 2022-09-12T21:19:42.096 DEBUG:boto:Host: smithi035.front.sepia.ceph.com:8000 2022-09-12T21:19:42.097 DEBUG:boto:Port: 8000 2022-09-12T21:19:42.097 DEBUG:boto:Params: {} 2022-09-12T21:19:42.098 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T21:19:42.099 DEBUG:boto:Token: None 2022-09-12T21:19:42.099 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 21:19:42 GMT /jdpiak-80/ 2022-09-12T21:19:42.100 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2e2qi5pQyVFhj9VvXXmj9s/K1rQ= 2022-09-12T21:19:42.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 21:19:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2e2qi5pQyVFhj9VvXXmj9s/K1rQ=', 'Content-Length': '0'} 2022-09-12T21:19:42.102 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-12T21:19:42.103 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T21:19:42.484 DEBUG:boto:Token: None 2022-09-12T21:19:42.485 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 21:19:42 GMT /jdpiak-80/ 2022-09-12T21:19:42.485 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2e2qi5pQyVFhj9VvXXmj9s/K1rQ= 2022-09-12T21:19:42.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 21:19:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2e2qi5pQyVFhj9VvXXmj9s/K1rQ=', 'Content-Length': '0'} 2022-09-12T21:19:42.487 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-12T21:19:42.488 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T21:19:44.151 DEBUG:boto:Token: None 2022-09-12T21:19:44.153 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 21:19:42 GMT /jdpiak-80/ 2022-09-12T21:19:44.154 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2e2qi5pQyVFhj9VvXXmj9s/K1rQ= 2022-09-12T21:19:44.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 21:19:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2e2qi5pQyVFhj9VvXXmj9s/K1rQ=', 'Content-Length': '0'} 2022-09-12T21:19:44.163 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-12T21:19:44.166 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T21:19:46.850 DEBUG:boto:Token: None 2022-09-12T21:19:46.851 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 21:19:42 GMT /jdpiak-80/ 2022-09-12T21:19:46.852 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2e2qi5pQyVFhj9VvXXmj9s/K1rQ= 2022-09-12T21:19:46.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 21:19:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2e2qi5pQyVFhj9VvXXmj9s/K1rQ=', 'Content-Length': '0'} 2022-09-12T21:19:46.856 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-12T21:19:46.857 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T21:19:48.331 DEBUG:boto:Token: None 2022-09-12T21:19:48.333 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 21:19:42 GMT /jdpiak-80/ 2022-09-12T21:19:48.334 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2e2qi5pQyVFhj9VvXXmj9s/K1rQ= 2022-09-12T21:19:48.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 21:19:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2e2qi5pQyVFhj9VvXXmj9s/K1rQ=', 'Content-Length': '0'} 2022-09-12T21:19:48.338 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-12T21:19:48.339 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T21:19:58.681 DEBUG:boto:Token: None 2022-09-12T21:19:58.683 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 21:19:42 GMT /jdpiak-80/ 2022-09-12T21:19:58.684 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2e2qi5pQyVFhj9VvXXmj9s/K1rQ= 2022-09-12T21:19:58.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 21:19:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2e2qi5pQyVFhj9VvXXmj9s/K1rQ=', 'Content-Length': '0'} 2022-09-12T21:19:58.686 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-12T21:19:58.687 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T21:20:16.530 DEBUG:boto:Token: None 2022-09-12T21:20:16.531 DEBUG:boto:StringToSign: PUT Mon, 12 Sep 2022 21:19:42 GMT /jdpiak-80/ 2022-09-12T21:20:16.531 DEBUG:boto:Signature: AWS FUCN3RT6SAOJRSRI0ZIX:2e2qi5pQyVFhj9VvXXmj9s/K1rQ= 2022-09-12T21:20:16.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Mon, 12 Sep 2022 21:19:42 GMT', 'Authorization': 'AWS FUCN3RT6SAOJRSRI0ZIX:2e2qi5pQyVFhj9VvXXmj9s/K1rQ=', 'Content-Length': '0'} 2022-09-12T21:20:16.534 DEBUG:boto:encountered ConnectionRefusedError exception, reconnecting 2022-09-12T21:20:16.535 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-12T21:21:04.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... ERROR 2022-09-12T21:21:04.415 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2022-09-12T21:21:04.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T21:21:04.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T21:21:04.423 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T21:21:04.478 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T21:21:04.479 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T21:26:04.490 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T21:26:04.478+0000 7fb11079c600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T21:31:04.483 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T21:31:04.478+0000 7fb11079c600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T21:36:04.483 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T21:36:04.479+0000 7fb11079c600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T21:41:04.489 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T21:41:04.479+0000 7fb11079c600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T21:46:04.484 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T21:46:04.480+0000 7fb11079c600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T21:51:04.491 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T21:51:04.481+0000 7fb11079c600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T21:56:04.486 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T21:56:04.481+0000 7fb11079c600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T22:01:04.487 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T22:01:04.482+0000 7fb11079c600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T22:06:04.488 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T22:06:04.482+0000 7fb11079c600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T22:11:04.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-12T22:11:04.572 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T22:11:04.483+0000 7fb11079c600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T22:11:04.573 INFO:teuthology.orchestra.run.smithi035.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-12T22:11:04.580 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... FAIL 2022-09-12T22:11:04.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T22:11:04.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T22:11:04.589 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T22:11:04.647 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T22:11:04.647 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T22:16:04.669 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T22:16:04.647+0000 7fa60abcd600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T22:21:04.677 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T22:21:04.648+0000 7fa60abcd600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T22:26:04.666 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T22:26:04.649+0000 7fa60abcd600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T22:31:04.667 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T22:31:04.649+0000 7fa60abcd600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T22:36:04.659 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T22:36:04.649+0000 7fa60abcd600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T22:41:04.657 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T22:41:04.650+0000 7fa60abcd600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T22:46:04.655 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T22:46:04.650+0000 7fa60abcd600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T22:51:04.654 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T22:51:04.651+0000 7fa60abcd600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T22:56:04.679 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T22:56:04.651+0000 7fa60abcd600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T23:01:04.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-12T23:01:04.722 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T23:01:04.652+0000 7fa60abcd600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T23:01:04.737 INFO:teuthology.orchestra.run.smithi035.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-12T23:01:04.749 INFO:tasks.rgw_multisite_tests:test s3 records fetching ... FAIL 2022-09-12T23:01:04.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T23:01:04.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T23:01:04.770 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T23:01:04.834 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T23:01:04.834 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T23:06:04.839 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T23:06:04.834+0000 7f774f55f600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T23:11:04.839 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T23:11:04.834+0000 7f774f55f600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T23:16:04.841 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T23:16:04.834+0000 7f774f55f600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T23:21:04.844 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T23:21:04.834+0000 7f774f55f600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T23:26:04.839 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T23:26:04.835+0000 7f774f55f600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T23:31:04.864 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T23:31:04.836+0000 7f774f55f600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T23:36:04.842 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T23:36:04.837+0000 7f774f55f600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T23:41:04.843 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T23:41:04.837+0000 7f774f55f600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T23:46:04.845 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T23:46:04.837+0000 7f774f55f600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T23:51:04.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-12T23:51:04.941 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T23:51:04.837+0000 7f774f55f600 0 monclient(hunting): authenticate timed out after 300 2022-09-12T23:51:04.948 INFO:teuthology.orchestra.run.smithi035.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-12T23:51:04.951 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete ... FAIL 2022-09-12T23:51:04.953 INFO:tasks.rgw_multisite_tests:test s3 notification filter on master ... ok 2022-09-12T23:51:04.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests_ps.test_object_timing ... ok 2022-09-12T23:51:04.954 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification ... ok 2022-09-12T23:51:04.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T23:51:04.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-12T23:51:04.957 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-12T23:51:05.028 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-12T23:51:05.029 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-12T23:56:05.033 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-12T23:56:05.027+0000 7ff97c165600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T00:01:05.052 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T00:01:05.028+0000 7ff97c165600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T00:06:05.035 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T00:06:05.028+0000 7ff97c165600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T00:11:05.036 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T00:11:05.030+0000 7ff97c165600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T00:16:05.043 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T00:16:05.031+0000 7ff97c165600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T00:21:05.035 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T00:21:05.031+0000 7ff97c165600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T00:26:05.037 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T00:26:05.032+0000 7ff97c165600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T00:31:05.035 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T00:31:05.032+0000 7ff97c165600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T00:36:05.043 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T00:36:05.033+0000 7ff97c165600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T00:41:05.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-13T00:41:05.069 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T00:41:05.033+0000 7ff97c165600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T00:41:05.070 INFO:teuthology.orchestra.run.smithi035.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-13T00:41:05.076 INFO:tasks.rgw_multisite_tests:test set/get/delete of topic ... FAIL 2022-09-13T00:41:05.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T00:41:05.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T00:41:05.083 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T00:41:05.170 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-13T00:41:05.171 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T00:46:05.175 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T00:46:05.170+0000 7febbe02c600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T00:51:05.174 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T00:51:05.171+0000 7febbe02c600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T00:56:05.177 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T00:56:05.171+0000 7febbe02c600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T01:01:05.186 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T01:01:05.172+0000 7febbe02c600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T01:06:05.176 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T01:06:05.172+0000 7febbe02c600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T01:11:05.178 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T01:11:05.173+0000 7febbe02c600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T01:16:05.177 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T01:16:05.175+0000 7febbe02c600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T01:21:05.178 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T01:21:05.175+0000 7febbe02c600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T01:26:05.179 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T01:26:05.175+0000 7febbe02c600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T01:31:05.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-13T01:31:05.203 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T01:31:05.175+0000 7febbe02c600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T01:31:05.203 INFO:teuthology.orchestra.run.smithi035.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-13T01:31:05.206 INFO:tasks.rgw_multisite_tests:test set topic with endpoint ... FAIL 2022-09-13T01:31:05.208 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T01:31:05.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T01:31:05.210 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T01:31:05.266 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-13T01:31:05.267 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T01:36:05.270 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T01:36:05.268+0000 7f6e8cfaa600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T01:41:05.274 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T01:41:05.268+0000 7f6e8cfaa600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T01:46:05.272 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T01:46:05.269+0000 7f6e8cfaa600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T01:51:05.273 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T01:51:05.270+0000 7f6e8cfaa600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T01:56:05.274 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T01:56:05.269+0000 7f6e8cfaa600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T02:01:05.273 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T02:01:05.270+0000 7f6e8cfaa600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T02:06:05.275 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T02:06:05.270+0000 7f6e8cfaa600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T02:11:05.275 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T02:11:05.272+0000 7f6e8cfaa600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T02:16:05.276 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T02:16:05.273+0000 7f6e8cfaa600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T02:21:05.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-13T02:21:05.345 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T02:21:05.273+0000 7f6e8cfaa600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T02:21:05.346 INFO:teuthology.orchestra.run.smithi035.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-13T02:21:05.348 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification ... FAIL 2022-09-13T02:21:05.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T02:21:05.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T02:21:05.355 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T02:21:05.417 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-13T02:21:05.418 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T02:26:05.421 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T02:26:05.418+0000 7f7681daf600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T02:31:05.424 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T02:31:05.418+0000 7f7681daf600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T02:36:05.423 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T02:36:05.419+0000 7f7681daf600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T02:41:05.427 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T02:41:05.420+0000 7f7681daf600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T02:46:05.424 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T02:46:05.420+0000 7f7681daf600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T02:51:05.424 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T02:51:05.421+0000 7f7681daf600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T02:56:05.427 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T02:56:05.420+0000 7f7681daf600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T03:01:05.425 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T03:01:05.421+0000 7f7681daf600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T03:06:05.426 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T03:06:05.421+0000 7f7681daf600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T03:11:05.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-13T03:11:05.480 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T03:11:05.421+0000 7f7681daf600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T03:11:05.481 INFO:teuthology.orchestra.run.smithi035.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-13T03:11:05.483 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification on specific events ... FAIL 2022-09-13T03:11:05.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T03:11:05.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T03:11:05.487 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T03:11:05.545 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-13T03:11:05.546 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T03:16:05.549 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T03:16:05.544+0000 7fab02090600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T03:21:05.550 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T03:21:05.545+0000 7fab02090600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T03:26:05.551 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T03:26:05.545+0000 7fab02090600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T03:31:05.551 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T03:31:05.546+0000 7fab02090600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T03:36:05.554 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T03:36:05.547+0000 7fab02090600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T03:41:05.552 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T03:41:05.548+0000 7fab02090600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T03:46:05.554 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T03:46:05.549+0000 7fab02090600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T03:51:05.554 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T03:51:05.549+0000 7fab02090600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T03:56:05.557 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T03:56:05.549+0000 7fab02090600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T04:01:05.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-13T04:01:05.609 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T04:01:05.550+0000 7fab02090600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T04:01:05.610 INFO:teuthology.orchestra.run.smithi035.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-13T04:01:05.612 INFO:tasks.rgw_multisite_tests:test set/get/delete of subscription ... FAIL 2022-09-13T04:01:05.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T04:01:05.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T04:01:05.617 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T04:01:05.678 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-13T04:01:05.678 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T04:06:05.683 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T04:06:05.677+0000 7fbb73520600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T04:11:05.682 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T04:11:05.677+0000 7fbb73520600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T04:16:05.683 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T04:16:05.679+0000 7fbb73520600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T04:21:05.687 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T04:21:05.678+0000 7fbb73520600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T04:26:05.684 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T04:26:05.679+0000 7fbb73520600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T04:31:05.685 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T04:31:05.680+0000 7fbb73520600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T04:36:05.686 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T04:36:05.681+0000 7fbb73520600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T04:41:05.686 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T04:41:05.680+0000 7fbb73520600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T04:46:05.685 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T04:46:05.681+0000 7fbb73520600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T04:51:05.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-13T04:51:05.726 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T04:51:05.682+0000 7fbb73520600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T04:51:05.727 INFO:teuthology.orchestra.run.smithi035.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-13T04:51:05.731 INFO:tasks.rgw_multisite_tests:test radosgw-admin commands ... FAIL 2022-09-13T04:51:05.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T04:51:05.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T04:51:05.737 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T04:51:05.798 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-13T04:51:05.799 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T04:56:05.803 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T04:56:05.799+0000 7fde2b6e2600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T05:01:05.804 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T05:01:05.798+0000 7fde2b6e2600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T05:06:05.805 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T05:06:05.799+0000 7fde2b6e2600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T05:11:05.804 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T05:11:05.800+0000 7fde2b6e2600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T05:16:05.806 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T05:16:05.801+0000 7fde2b6e2600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T05:21:05.804 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T05:21:05.801+0000 7fde2b6e2600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T05:26:05.806 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T05:26:05.802+0000 7fde2b6e2600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T05:31:05.807 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T05:31:05.801+0000 7fde2b6e2600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T05:36:05.809 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T05:36:05.802+0000 7fde2b6e2600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T05:41:05.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-13T05:41:05.881 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T05:41:05.803+0000 7fde2b6e2600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T05:41:05.882 INFO:teuthology.orchestra.run.smithi035.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-13T05:41:05.886 INFO:tasks.rgw_multisite_tests:test that events are only sent on incremental sync ... FAIL 2022-09-13T05:41:05.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T05:41:05.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T05:41:05.892 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T05:41:05.956 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-13T05:41:05.957 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T05:46:05.957 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T05:46:05.951+0000 7f3eb69b8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T05:51:05.958 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T05:51:05.952+0000 7f3eb69b8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T05:56:05.957 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T05:56:05.953+0000 7f3eb69b8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T06:01:05.957 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T06:01:05.953+0000 7f3eb69b8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T06:06:05.959 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T06:06:05.954+0000 7f3eb69b8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T06:11:05.962 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T06:11:05.955+0000 7f3eb69b8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T06:16:05.960 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T06:16:05.956+0000 7f3eb69b8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T06:21:05.960 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T06:21:05.955+0000 7f3eb69b8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T06:26:05.961 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T06:26:05.956+0000 7f3eb69b8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T06:31:05.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-13T06:31:05.982 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T06:31:05.956+0000 7f3eb69b8600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T06:31:05.982 INFO:teuthology.orchestra.run.smithi035.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-13T06:31:05.988 INFO:tasks.rgw_multisite_tests:test subscriptions for different events ... FAIL 2022-09-13T06:31:05.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T06:31:05.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-13T06:31:05.992 DEBUG:teuthology.orchestra.run.smithi035:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-13T06:31:06.048 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setuser ceph since I am not root 2022-09-13T06:31:06.049 INFO:teuthology.orchestra.run.smithi035.stderr:ignoring --setgroup ceph since I am not root 2022-09-13T06:36:06.053 INFO:teuthology.orchestra.run.smithi035.stderr:2022-09-13T06:36:06.048+0000 7fc30c158600 0 monclient(hunting): authenticate timed out after 300 2022-09-13T06:36:13.725 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-09-13T06:36:13.730 DEBUG:teuthology.task.console_log:Killing console logger for smithi035 2022-09-13T06:36:13.732 DEBUG:teuthology.task.console_log:Killing console logger for smithi138 2022-09-13T06:36:13.733 DEBUG:teuthology.exit:Finished running handlers